{"record_id":"1912.02516v3","paper":{"title":"Local convergence of tensor methods","authors":["Doikov, Nikita","Nesterov, Yurii"],"arxiv_id":"1912.02516","version":"v3","journal":"Mathematical Programming","abstract_url":"https://arxiv.org/abs/1912.02516v3","pdf_url":"https://arxiv.org/pdf/1912.02516v3","doi":"10.1007/s10107-020-01606-x"},"context":{"raw_tex":"{\\bf Notations and generalities.} In what follows, we\r\ndenote by $\\E$ a finite-dimensional real vector space, and\r\nby $\\E^*$ its dual spaced composed by linear functions on\r\n$\\E$. For such a function $s \\in \\E^*$, we denote by $\\la\r\ns, x \\ra$ its value at $x \\in \\E$. Using a self-adjoint\r\npositive-definite operator $B: \\E \\to \\E^*$ (notation $B =\r\nB^* \\succ 0$), we can endow these spaces with mutually\r\nconjugate Euclidean norms:\r\n$$\r\n\\ba{rcl}\r\n\\| x \\| & = & \\la B x, x \\ra^{1/2}, \\quad x \\in \\E, \\quad\r\n\\| g \\|_* \\; = \\; \\la g, B^{-1} g \\ra^{1/2}, \\quad g \\in\r\n\\E^*.\r\n\\ea\r\n$$\r\n\r\nFor a smooth function $f: \\dom f \\to \\R$ with convex and\r\nopen domain $\\dom f  \\subseteq \\E$, denote by $\\nabla\r\nf(x)$ its gradient, and by $\\nabla^2 f(x)$ its Hessian\r\nevaluated at point $x \\in \\dom f \\subseteq \\E$. Note that\r\n$$\r\n\\ba{rcl}\r\n\\nabla f(x) & \\in & \\E^*, \\quad \\nabla^2 f(x) h \\; \\in \\;\r\n\\E^*, \\quad x \\in \\dom f, \\; h \\in \\E.\r\n\\ea\r\n$$\r\nFor non-differentiable convex function $f(\\cdot)$, we\r\ndenote by $\\partial f(x) \\subset \\E^*$ its subdifferential\r\nat the point $x \\in \\dom f$.\r\n\r\nIn what follows, we often work with directional\r\nderivatives. For $p \\geq 1$, denote by\r\n$$\r\n\\ba{c}\r\nD^p f(x)[h_1, \\dots, h_p]\r\n\\ea\r\n$$\r\nthe directional derivative of function $f$ at $x$ along\r\ndirections $h_i \\in \\E$, $i = 1, \\dots, p$. If all\r\ndirections $h_1, \\dots, h_p$ are the same, we apply a\r\nsimpler notation\r\n$$\r\n\\ba{c}\r\nD^p f(x)[h]^p, \\quad h \\in \\E.\r\n\\ea\r\n$$\r\n\r\nNote that $D^p f(x)[ \\cdot]$ is a {\\em symmetric\r\n\t$p$-linear form}. Its {\\em norm} is defined in the\r\nstandard way:\r\n\\beq\\label{eq-DNorm1}\r\n\\ba{rcl}\r\n\\| D^pf(x) \\| & = & \\max\\limits_{h_1, \\dots, h_p \\in \\E}\r\n\\left\\{ D^p f(x)[h_1, \\dots, h_p ]: \\; \\| h_i \\| \\leq 1,\r\n\\, i = 1,\r\n\\dots, p \\right\\}\\\\\r\n\\\\\r\n& = & \\max\\limits_{h \\in \\E} \\left\\{ \\Big| D^p\r\nf(x)[h]^p\\Big|: \\; \\| h \\| \\leq 1 \\right\\}\r\n\\ea\r\n\\eeq\r\n(for the last equation see, for example, Appendix 1 in\r\n\\cite{nesterov1994interior}). Similarly, we define\r\n\\beq\\label{eq-DNorm2}\r\n\\ba{rcl}\r\n\\| D^pf(x) - D^pf(y) \\| & = & \\max\\limits_{h \\in \\E}\r\n\\left\\{ \\Big| D^p f(x)[h]^p - D^pf(y)[h]^p\\Big|: \\; \\| h\r\n\\| \\leq 1 \\right\\}.\r\n\\ea\r\n\\eeq\r\n\r\nIn particular, for any $x \\in \\dom f$ and $h_1, h_2 \\in\r\n\\E$, we have\r\n$$\r\n\\ba{rcl}\r\nDf(x)[h_1] & = & \\la \\nabla f(x), h_1 \\ra, \\quad\r\nD^2f(x)[h_1, h_2] \\; = \\; \\la \\nabla^2 f(x) h_1, h_2 \\ra.\r\n\\ea\r\n$$\r\nThus, for the Hessian, our definition corresponds to a\r\n{\\em spectral norm} of the self-adjoint linear operator\r\n(maximal module of all eigenvalues computed with respect\r\nto $B \\succ 0$).\r\n\n\nFinally, the Taylor approximation of function $f(\\cdot)$\r\nat $x \\in \\dom f$ is defined as follows:\r\n$$\r\n\\ba{rcl}\r\nf(x+h) & = & \\Omega_p(f, x; x + h)  + o(\\|h\\|^p), \\quad x+h \\in\r\n\\dom\r\nf,\\\\\r\n\\\\\r\n\\Omega_p(f,x;y) & \\Def & f(x) + \\sum\\limits_{k=1}^p {1\r\n\t\\over k!} D^k f(x)[y-x]^k, \\quad y \\in \\E.\r\n\\ea\r\n$$\r\nConsequently, for all $y \\in \\E$ we have\r\n\\beq\\label{eq-PGrad}\r\n\\ba{rcl}\r\n\\nabla \\Omega_p(f,x;y) & = & \\sum\\limits_{k=1}^p {1 \\over\r\n\t(k-1)!} D^k f(x)[y-x]^{k-1},\r\n\\ea\r\n\\eeq\r\n\n\nIn this paper, we consider the following {\\em composite}\r\nconvex minimization problem\r\n\\beq\\label{prob-Main}\r\n\\min\\limits_{x \\in \\dom h} \\Big\\{ F(x) = f(x) + h(x)\r\n\\Big\\},\r\n\\eeq\r\nwhere $h: \\E \\to \\R \\cup \\{+\\infty\\}$ is a {\\em simple} proper\r\nclosed convex function and $f \\in C^{p,p}(\\dom h)$ for a\r\ncertain $p \\geq 2$. In other words, we assume that the\r\n$p$th derivative of function $f$ is Lipschitz continuous:\r\n\\beq\\label{eq-Lip}\r\n\\ba{rcl}\r\n\\| D^p f(x) - D^p f(y) \\| & \\leq & L_p \\| x - y \\|,\r\n\\quad x, y \\in \\dom h.\r\n\\ea\r\n\\eeq\r\n\r\nAssuming that $L_{p} < +\\infty$, by the standard\r\nintegration arguments we can bound the residual between\r\nfunction value and its Taylor approximation:\r\n\\beq\\label{eq-BoundF}\r\n\\ba{rcl}\r\n| f(y) - \\Omega_p(f,x;y) | & \\leq & {L_{p} \\over (p+1)!}\r\n\\| y - x \\|^{p+1}, \\quad x, y \\in \\dom h.\r\n\\ea\r\n\\eeq\r\nApplying the same reasoning to functions $\\la \\nabla\r\nf(\\cdot), h \\ra$ and $\\la \\nabla^2 f(\\cdot) h, h \\ra$ with\r\ndirection $h \\in \\E$ being fixed, we get the following\r\nguarantees:\r\n\\beq\\label{eq-BoundG}\r\n\\ba{rcl}\r\n\\| \\nabla f(y) - \\nabla \\Omega_p(f,x;y) \\|_* & \\leq & {L_p\r\n\t\\over p!} \\| y - x \\|^{p},\r\n\\ea\r\n\\eeq\r\n\\beq\\label{eq-BoundH}\r\n\\ba{rcl}\r\n\\| \\nabla^2 f(y) - \\nabla^2 \\Omega_p(f,x;y) \\| & \\leq &\r\n{L_p \\over (p-1)!} \\| y - x \\|^{p-1},\r\n\\ea\r\n\\eeq\r\nwhich are valid for all $x, y \\in \\dom h$.\r\n\n\nLet us define now one step of the {\\em Regularized\r\n\tComposite Tensor Method} (RCTM) of degree $p \\geq 2$:\r\n\\beq\\label{def-RCTS}\r\n\\ba{rcl}\r\nT & \\equiv & T_H(x) \\; \\Def \\; \\arg\\min\\limits_{y \\in \\E}\r\n\\left\\{ \\Omega_p(f,x;y) + {H \\over (p+1)!} \\| y - x\r\n\\|^{p+1} + h(y) \\right\\}.\r\n\\ea\r\n\\eeq\r\nIt can be shown that for\r\n\\beq\\label{eq-H}\r\n\\ba{c}\r\n\\mbox{\\fbox{\\rule[-2mm]{0cm}{6mm}$H \\; \\geq \\; p L_p$}}\r\n\\ea\r\n\\eeq\r\nthe auxiliary optimization problem in (\\ref{def-RCTS}) is\r\n{\\em convex} (see Theorem 1 in \\cite{nesterov2019implementable}).\r\nThis condition\r\nis crucial for implementability of our methods and we\r\nalways assume it to be satisfied.\r\n\r\nLet us write down the first-order optimality condition for\r\nthe auxiliary optimization problem in (\\ref{def-RCTS}):\r\n\\beq\\label{eq-OptC}\r\n\\ba{rcl}\r\n\\la \\nabla \\Omega_p(f,x;T) + {H \\over p!} \\| T - x\r\n\\|^{p-1}B(T-x), y - T \\ra + h(y) & \\geq & h(T),\r\n\\ea\r\n\\eeq\r\nfor all $y \\in \\dom h$.\r\nIn other words, for vector\r\n\\beq\\label{def-HP}\r\n\\ba{rcl}\r\nh'(T) & \\Def & - \\left( \\nabla \\Omega_p(f,x;T) + {H \\over\r\n\tp!} \\| T - x \\|^{p-1}B(T-x) \\right)\r\n\\ea\r\n\\eeq\r\nwe have $h'(T) \\stackrel{(\\ref{eq-OptC})}{\\in} \\partial\r\nh(T)$. This fact explains our notation\r\n\\beq\\label{def-FP}\r\n\\ba{rcl}\r\nF'(T) & \\Def & \\nabla f(T) + h'(T) \\; \\in \\partial F(T).\r\n\\ea\r\n\\eeq\r\n\r\nLet us present some properties of the point $T = T_H(x)$.\r\nFirst of all, we need some bounds for the norm of vector\r\n$F'(T)$. Note that\r\n\\beq\\label{eq-N3}\r\n\\ba{rcl}\r\n\\Big\\| F'(T) + {H \\over p!} \\| T - x \\|^{p-1}B(T-x)\r\n\\Big\\|_* & \\refEQ{def-HP} & \\Big\\| \\nabla f(T) - \\nabla\r\n\\Omega_p(f,x;T) \\Big\\|_*\\\\\r\n\\\\\r\n& \\refLE{eq-BoundG} & {L_p \\over p!} \\| T - x \\|^p.\r\n\\ea\r\n\\eeq\r\nConsequently,\r\n\\beq\\label{eq-NewG}\r\n\\ba{rcl}\r\n\\| F'(T) \\|_* & \\leq & {L_p+H \\over p!} \\| T - x \\|^p.\r\n\\ea\r\n\\eeq\r\n\n\nThe main goal of this paper consists in analyzing the\r\nlocal behavior of the {\\em Regularized Composite Tensor\r\n\tMethod} (RCTM):\r\n\\beq\\label{met-RCTM}\r\n\\ba{rcl}\r\nx_0 \\; \\in \\; \\dom h, \\quad x_{k+1} & = & T_H(x_k), \\quad\r\nk \\geq 0,\r\n\\ea\r\n\\eeq\r\nas applied to the problem (\\ref{prob-Main}). In order to\r\nprove local superlinear convergence of this scheme, we\r\nneed one more assumption.\r\n\r\n\\begin{assumption}\\label{assumption-Uni}\r\n\tThe objective in problem\r\n\t(\\ref{prob-Main}) is uniformly convex of degree $q \\geq 2$.\r\n\tThus, for all $x, y \\in \\dom h$ and for all $G_x \\in \\partial F(x), G_y \\in \\partial F(y)$,\r\n\tit holds:\r\n\t\\beq\\label{eq-UniC}\r\n\t\\ba{rcl}\r\n\t\\la G_x - G_y, x - y \\ra & \\geq & \\sigma_q\r\n\t\\| x - y \\|^q, \r\n\t\\ea\r\n\t\\eeq\r\n\tfor certain $\\sigma_q > 0$.\r\n\\end{assumption}\r\n\n\nLet us study now the local convergence of the method\r\n(\\ref{met-RCTM}) in terms of the norm of gradient. For any\r\n$x \\in \\dom h$ denote\r\n\\beq\\label{def-GNorm}\r\n\\ba{rcl}\r\n\\eta(x) & \\Def & \\min\\limits_{g \\in \\partial h(x)} \\| \\nabla\r\nf(x) + g \\|_*.\r\n\\ea\r\n\\eeq\r\nIf $\\partial h(x) = \\emptyset$, we set $\\eta(x) =\r\n+\\infty$.\r\n","units":[{"raw_tex":"{\\bf Notations and generalities.} In what follows, we\r\ndenote by $\\E$ a finite-dimensional real vector space, and\r\nby $\\E^*$ its dual spaced composed by linear functions on\r\n$\\E$. For such a function $s \\in \\E^*$, we denote by $\\la\r\ns, x \\ra$ its value at $x \\in \\E$. Using a self-adjoint\r\npositive-definite operator $B: \\E \\to \\E^*$ (notation $B =\r\nB^* \\succ 0$), we can endow these spaces with mutually\r\nconjugate Euclidean norms:\r\n$$\r\n\\ba{rcl}\r\n\\| x \\| & = & \\la B x, x \\ra^{1/2}, \\quad x \\in \\E, \\quad\r\n\\| g \\|_* \\; = \\; \\la g, B^{-1} g \\ra^{1/2}, \\quad g \\in\r\n\\E^*.\r\n\\ea\r\n$$\r\n\r\nFor a smooth function $f: \\dom f \\to \\R$ with convex and\r\nopen domain $\\dom f  \\subseteq \\E$, denote by $\\nabla\r\nf(x)$ its gradient, and by $\\nabla^2 f(x)$ its Hessian\r\nevaluated at point $x \\in \\dom f \\subseteq \\E$. Note that\r\n$$\r\n\\ba{rcl}\r\n\\nabla f(x) & \\in & \\E^*, \\quad \\nabla^2 f(x) h \\; \\in \\;\r\n\\E^*, \\quad x \\in \\dom f, \\; h \\in \\E.\r\n\\ea\r\n$$\r\nFor non-differentiable convex function $f(\\cdot)$, we\r\ndenote by $\\partial f(x) \\subset \\E^*$ its subdifferential\r\nat the point $x \\in \\dom f$.\r\n\r\nIn what follows, we often work with directional\r\nderivatives. For $p \\geq 1$, denote by\r\n$$\r\n\\ba{c}\r\nD^p f(x)[h_1, \\dots, h_p]\r\n\\ea\r\n$$\r\nthe directional derivative of function $f$ at $x$ along\r\ndirections $h_i \\in \\E$, $i = 1, \\dots, p$. If all\r\ndirections $h_1, \\dots, h_p$ are the same, we apply a\r\nsimpler notation\r\n$$\r\n\\ba{c}\r\nD^p f(x)[h]^p, \\quad h \\in \\E.\r\n\\ea\r\n$$\r\n\r\nNote that $D^p f(x)[ \\cdot]$ is a {\\em symmetric\r\n\t$p$-linear form}. Its {\\em norm} is defined in the\r\nstandard way:\r\n\\beq\\label{eq-DNorm1}\r\n\\ba{rcl}\r\n\\| D^pf(x) \\| & = & \\max\\limits_{h_1, \\dots, h_p \\in \\E}\r\n\\left\\{ D^p f(x)[h_1, \\dots, h_p ]: \\; \\| h_i \\| \\leq 1,\r\n\\, i = 1,\r\n\\dots, p \\right\\}\\\\\r\n\\\\\r\n& = & \\max\\limits_{h \\in \\E} \\left\\{ \\Big| D^p\r\nf(x)[h]^p\\Big|: \\; \\| h \\| \\leq 1 \\right\\}\r\n\\ea\r\n\\eeq\r\n(for the last equation see, for example, Appendix 1 in\r\n\\cite{nesterov1994interior}). Similarly, we define\r\n\\beq\\label{eq-DNorm2}\r\n\\ba{rcl}\r\n\\| D^pf(x) - D^pf(y) \\| & = & \\max\\limits_{h \\in \\E}\r\n\\left\\{ \\Big| D^p f(x)[h]^p - D^pf(y)[h]^p\\Big|: \\; \\| h\r\n\\| \\leq 1 \\right\\}.\r\n\\ea\r\n\\eeq\r\n\r\nIn particular, for any $x \\in \\dom f$ and $h_1, h_2 \\in\r\n\\E$, we have\r\n$$\r\n\\ba{rcl}\r\nDf(x)[h_1] & = & \\la \\nabla f(x), h_1 \\ra, \\quad\r\nD^2f(x)[h_1, h_2] \\; = \\; \\la \\nabla^2 f(x) h_1, h_2 \\ra.\r\n\\ea\r\n$$\r\nThus, for the Hessian, our definition corresponds to a\r\n{\\em spectral norm} of the self-adjoint linear operator\r\n(maximal module of all eigenvalues computed with respect\r\nto $B \\succ 0$).\r\n","start":9634,"end":12173},{"raw_tex":"Finally, the Taylor approximation of function $f(\\cdot)$\r\nat $x \\in \\dom f$ is defined as follows:\r\n$$\r\n\\ba{rcl}\r\nf(x+h) & = & \\Omega_p(f, x; x + h)  + o(\\|h\\|^p), \\quad x+h \\in\r\n\\dom\r\nf,\\\\\r\n\\\\\r\n\\Omega_p(f,x;y) & \\Def & f(x) + \\sum\\limits_{k=1}^p {1\r\n\t\\over k!} D^k f(x)[y-x]^k, \\quad y \\in \\E.\r\n\\ea\r\n$$\r\nConsequently, for all $y \\in \\E$ we have\r\n\\beq\\label{eq-PGrad}\r\n\\ba{rcl}\r\n\\nabla \\Omega_p(f,x;y) & = & \\sum\\limits_{k=1}^p {1 \\over\r\n\t(k-1)!} D^k f(x)[y-x]^{k-1},\r\n\\ea\r\n\\eeq\r\n","start":12175,"end":12655},{"raw_tex":"In this paper, we consider the following {\\em composite}\r\nconvex minimization problem\r\n\\beq\\label{prob-Main}\r\n\\min\\limits_{x \\in \\dom h} \\Big\\{ F(x) = f(x) + h(x)\r\n\\Big\\},\r\n\\eeq\r\nwhere $h: \\E \\to \\R \\cup \\{+\\infty\\}$ is a {\\em simple} proper\r\nclosed convex function and $f \\in C^{p,p}(\\dom h)$ for a\r\ncertain $p \\geq 2$. In other words, we assume that the\r\n$p$th derivative of function $f$ is Lipschitz continuous:\r\n\\beq\\label{eq-Lip}\r\n\\ba{rcl}\r\n\\| D^p f(x) - D^p f(y) \\| & \\leq & L_p \\| x - y \\|,\r\n\\quad x, y \\in \\dom h.\r\n\\ea\r\n\\eeq\r\n\r\nAssuming that $L_{p} < +\\infty$, by the standard\r\nintegration arguments we can bound the residual between\r\nfunction value and its Taylor approximation:\r\n\\beq\\label{eq-BoundF}\r\n\\ba{rcl}\r\n| f(y) - \\Omega_p(f,x;y) | & \\leq & {L_{p} \\over (p+1)!}\r\n\\| y - x \\|^{p+1}, \\quad x, y \\in \\dom h.\r\n\\ea\r\n\\eeq\r\nApplying the same reasoning to functions $\\la \\nabla\r\nf(\\cdot), h \\ra$ and $\\la \\nabla^2 f(\\cdot) h, h \\ra$ with\r\ndirection $h \\in \\E$ being fixed, we get the following\r\nguarantees:\r\n\\beq\\label{eq-BoundG}\r\n\\ba{rcl}\r\n\\| \\nabla f(y) - \\nabla \\Omega_p(f,x;y) \\|_* & \\leq & {L_p\r\n\t\\over p!} \\| y - x \\|^{p},\r\n\\ea\r\n\\eeq\r\n\\beq\\label{eq-BoundH}\r\n\\ba{rcl}\r\n\\| \\nabla^2 f(y) - \\nabla^2 \\Omega_p(f,x;y) \\| & \\leq &\r\n{L_p \\over (p-1)!} \\| y - x \\|^{p-1},\r\n\\ea\r\n\\eeq\r\nwhich are valid for all $x, y \\in \\dom h$.\r\n","start":12847,"end":14181},{"raw_tex":"Let us define now one step of the {\\em Regularized\r\n\tComposite Tensor Method} (RCTM) of degree $p \\geq 2$:\r\n\\beq\\label{def-RCTS}\r\n\\ba{rcl}\r\nT & \\equiv & T_H(x) \\; \\Def \\; \\arg\\min\\limits_{y \\in \\E}\r\n\\left\\{ \\Omega_p(f,x;y) + {H \\over (p+1)!} \\| y - x\r\n\\|^{p+1} + h(y) \\right\\}.\r\n\\ea\r\n\\eeq\r\nIt can be shown that for\r\n\\beq\\label{eq-H}\r\n\\ba{c}\r\n\\mbox{\\fbox{\\rule[-2mm]{0cm}{6mm}$H \\; \\geq \\; p L_p$}}\r\n\\ea\r\n\\eeq\r\nthe auxiliary optimization problem in (\\ref{def-RCTS}) is\r\n{\\em convex} (see Theorem 1 in \\cite{nesterov2019implementable}).\r\nThis condition\r\nis crucial for implementability of our methods and we\r\nalways assume it to be satisfied.\r\n\r\nLet us write down the first-order optimality condition for\r\nthe auxiliary optimization problem in (\\ref{def-RCTS}):\r\n\\beq\\label{eq-OptC}\r\n\\ba{rcl}\r\n\\la \\nabla \\Omega_p(f,x;T) + {H \\over p!} \\| T - x\r\n\\|^{p-1}B(T-x), y - T \\ra + h(y) & \\geq & h(T),\r\n\\ea\r\n\\eeq\r\nfor all $y \\in \\dom h$.\r\nIn other words, for vector\r\n\\beq\\label{def-HP}\r\n\\ba{rcl}\r\nh'(T) & \\Def & - \\left( \\nabla \\Omega_p(f,x;T) + {H \\over\r\n\tp!} \\| T - x \\|^{p-1}B(T-x) \\right)\r\n\\ea\r\n\\eeq\r\nwe have $h'(T) \\stackrel{(\\ref{eq-OptC})}{\\in} \\partial\r\nh(T)$. This fact explains our notation\r\n\\beq\\label{def-FP}\r\n\\ba{rcl}\r\nF'(T) & \\Def & \\nabla f(T) + h'(T) \\; \\in \\partial F(T).\r\n\\ea\r\n\\eeq\r\n\r\nLet us present some properties of the point $T = T_H(x)$.\r\nFirst of all, we need some bounds for the norm of vector\r\n$F'(T)$. Note that\r\n\\beq\\label{eq-N3}\r\n\\ba{rcl}\r\n\\Big\\| F'(T) + {H \\over p!} \\| T - x \\|^{p-1}B(T-x)\r\n\\Big\\|_* & \\refEQ{def-HP} & \\Big\\| \\nabla f(T) - \\nabla\r\n\\Omega_p(f,x;T) \\Big\\|_*\\\\\r\n\\\\\r\n& \\refLE{eq-BoundG} & {L_p \\over p!} \\| T - x \\|^p.\r\n\\ea\r\n\\eeq\r\nConsequently,\r\n\\beq\\label{eq-NewG}\r\n\\ba{rcl}\r\n\\| F'(T) \\|_* & \\leq & {L_p+H \\over p!} \\| T - x \\|^p.\r\n\\ea\r\n\\eeq\r\n","start":14183,"end":15961},{"raw_tex":"The main goal of this paper consists in analyzing the\r\nlocal behavior of the {\\em Regularized Composite Tensor\r\n\tMethod} (RCTM):\r\n\\beq\\label{met-RCTM}\r\n\\ba{rcl}\r\nx_0 \\; \\in \\; \\dom h, \\quad x_{k+1} & = & T_H(x_k), \\quad\r\nk \\geq 0,\r\n\\ea\r\n\\eeq\r\nas applied to the problem (\\ref{prob-Main}). In order to\r\nprove local superlinear convergence of this scheme, we\r\nneed one more assumption.\r\n\r\n\\begin{assumption}\\label{assumption-Uni}\r\n\tThe objective in problem\r\n\t(\\ref{prob-Main}) is uniformly convex of degree $q \\geq 2$.\r\n\tThus, for all $x, y \\in \\dom h$ and for all $G_x \\in \\partial F(x), G_y \\in \\partial F(y)$,\r\n\tit holds:\r\n\t\\beq\\label{eq-UniC}\r\n\t\\ba{rcl}\r\n\t\\la G_x - G_y, x - y \\ra & \\geq & \\sigma_q\r\n\t\\| x - y \\|^q, \r\n\t\\ea\r\n\t\\eeq\r\n\tfor certain $\\sigma_q > 0$.\r\n\\end{assumption}\r\n","start":18444,"end":19224},{"raw_tex":"Let us study now the local convergence of the method\r\n(\\ref{met-RCTM}) in terms of the norm of gradient. For any\r\n$x \\in \\dom h$ denote\r\n\\beq\\label{def-GNorm}\r\n\\ba{rcl}\r\n\\eta(x) & \\Def & \\min\\limits_{g \\in \\partial h(x)} \\| \\nabla\r\nf(x) + g \\|_*.\r\n\\ea\r\n\\eeq\r\nIf $\\partial h(x) = \\emptyset$, we set $\\eta(x) =\r\n+\\infty$.\r\n","start":21357,"end":21678}]},"statement":{"raw_tex":"\\BT\\label{th-RateG}\r\nFor any $k \\geq 0$ we have\r\n\\beq\\label{eq-RateG}\r\n\\ba{rcl}\r\n\\eta(x_{k+1})\r\n& \\, \\leq \\, &\r\n\\|F'(x_{k + 1}) \\|_{*}\r\n\\; \\leq \\;\r\n{L_p + H \\over p!} \\left[ {1 \\over\r\n\t\\sigma_q} \\, \\eta(x_k) \\right]^{p \\over q-1}.\r\n\\ea\r\n\\eeq\r\n\\ET","units":[{"raw_tex":"\\BT\\label{th-RateG}\r\nFor any $k \\geq 0$ we have\r\n\\beq\\label{eq-RateG}\r\n\\ba{rcl}\r\n\\eta(x_{k+1})\r\n& \\, \\leq \\, &\r\n\\|F'(x_{k + 1}) \\|_{*}\r\n\\; \\leq \\;\r\n{L_p + H \\over p!} \\left[ {1 \\over\r\n\t\\sigma_q} \\, \\eta(x_k) \\right]^{p \\over q-1}.\r\n\\ea\r\n\\eeq\r\n\\ET","start":21680,"end":21926}]},"proof":{"raw_tex":"\\BL\\label{lm-DecF}\r\nLet $\\beta > 1$ and $H = \\beta L_p$. Then\r\n\\beq\\label{eq-DecFB}\r\n\\ba{rcl}\r\n\\la F'(T), x - T \\ra & \\geq & \\left( {p! \\over (p+1)L_p}\r\n\\right)^{1 \\over p} \\cdot \\| F'(T) \\|_*^{p+1 \\over p}\r\n\\cdot {(\\beta^2 - 1)^{p-1 \\over 2p} \\over \\beta} \\cdot {p\r\n\t\\over (p^2-1)^{p-1 \\over 2p}}.\r\n\\ea\r\n\\eeq\r\nIn particular, if $\\beta = p$, then\r\n\\beq\\label{eq-DecF}\r\n\\ba{rcl}\r\n\\la F'(T), x - T \\ra & \\geq & \\left( {p! \\over (p+1)L_p}\r\n\\right)^{1 \\over p} \\cdot \\| F'(T) \\|_*^{p+1 \\over p}.\r\n\\ea\r\n\\eeq\r\n\\EL\n\n\\proof\r\nDenote $r = \\| T  - x \\|$, $h = {H \\over p!}$, and $l =\r\n{L_p \\over p!}$. Then inequality (\\ref{eq-N3}) can be\r\nwritten as follows:\r\n$$\r\n\\ba{rcl}\r\n\\| F'(T) + h r^{p-1} B(T-x) \\|^2_* & \\leq & l^2 r^{2p}.\r\n\\ea\r\n$$\r\nThis means that\r\n\\beq\\label{eq-N4}\r\n\\ba{rcl}\r\n\\la F'(T), x - T \\ra & \\geq & {1 \\over 2 h r^{p-1}} \\|\r\nF'(T) \\|_*^2 + {r^{2p} (h^2 - l^2) \\over 2h r^{p-1}}.\r\n\\ea\r\n\\eeq\r\nDenote\r\n$$\r\n\\ba{rcl}\r\na & = & {1 \\over 2h} \\| F'(T) \\|_*^2, \\quad b \\; = \\; {h^2\r\n\t- l^2 \\over 2h}, \\quad \\tau \\; = \\; r^{p-1}, \\quad \\alpha\r\n\\; = \\; {p+1 \\over p-1}.\r\n\\ea\r\n$$\r\nThen inequality (\\ref{eq-N4}) can be rewritten as follows:\r\n$$\r\n\\ba{rcl}\r\n\\la F'(T) , x - T \\ra & \\geq & {a \\over \\tau} + b\r\n\\tau^{\\alpha} \\; \\geq \\; \\min\\limits_{t > 0} \\left\\{ {a\r\n\t\\over t} + b t^{\\alpha} \\right\\} \\; = \\; (1+\\alpha)\r\n\\left({a \\over \\alpha} \\right)^{\\alpha \\over 1 + \\alpha}\r\nb^{1 \\over 1 + \\alpha}.\r\n\\ea\r\n$$\r\nTaking into account that $1+\\alpha = {2p \\over p-1}$ and\r\n${\\alpha \\over 1 + \\alpha} = {p + 1 \\over 2p}$, and using the\r\nactual meaning of $a$, $b$, and $\\alpha$, we get\r\n$$\r\n\\ba{rcl}\r\n\\la F'(T), x - T \\ra & \\geq & {2 p \\over p-1} \\cdot { \\|\r\n\tF'(T) \\|_*^{p+1 \\over p} \\over (2h)^{p+1 \\over 2p}} \\cdot\r\n{(p-1)^{p+1 \\over 2p} \\over (p+1)^{p+1 \\over 2p}} \\cdot\r\n{(h^2 - l^2)^{p-1 \\over 2p} \\over (2h)^{p-1 \\over 2p}}\\\\\r\n\\\\\r\n& = & \\| F'(T) \\|_*^{p+1 \\over p} \\cdot {(h^2 - l^2)^{p-1\r\n\t\t\\over 2p} \\over h} \\cdot {p \\over (p+1)^{p+1 \\over 2p}\r\n\t(p-1)^{p-1 \\over 2p}}\\\\\r\n\\\\\r\n& = & \\| F'(T) \\|_*^{p+1 \\over p} \\cdot {(h^2 - l^2)^{p-1\r\n\t\t\\over 2p} \\over h} \\cdot {p \\over (p^2-1)^{p-1 \\over 2p}\r\n\t(p+1)^{1 \\over p}}.\r\n\\ea\r\n$$\r\nIt remains to note that\r\n$$\r\n\\ba{rcl}\r\n{(h^2 - l^2)^{p-1 \\over 2p} \\over h} & = & {(H^2 -\r\n\tL_p^2)^{p-1 \\over 2p} \\over H} \\cdot (p!)^{1 \\over p} \\; =\r\n\\; {(\\beta^2 - 1)^{p-1 \\over 2p} \\over \\beta} \\cdot\r\n\\left({p! \\over L_p} \\right)^{1 \\over p}.\r\n\\ea\r\n$$\r\n\\qed\n\n\\proof\r\nIndeed, in view of inequality \\eqref{eq-UniC}, we have\r\n$$\r\n\\ba{rcl}\r\n\\la \\nabla f(x_k) + g_k, x_{k} - x_{k + 1} \\ra\r\n& \\geq &\r\n\\la F'(x_{k + 1}), x_k - x_{k + 1} \\ra + \\sigma_q \\|x_k - x_{k + 1}\\|^q \\\\\r\n\\\\\r\n& \\refGE{eq-DecFB} & \\sigma_q \\|x_k - x_{k + 1}\\|^{q},\r\n\\ea\r\n$$\r\nwhere $g_k$ is an arbitrary vector from $\\partial h(x_k)$.\r\nTherefore, we conclude that\r\n$$\r\n\\ba{rcl}\r\n\\eta(x_k)  & \\geq &  \\sigma_q \\| x_k -\r\nx_{k+1} \\|^{q-1}.\r\n\\ea\r\n$$\r\nIt remains to use inequality\r\n(\\ref{eq-NewG}).\r\n\\qed","units":[{"raw_tex":"\\BL\\label{lm-DecF}\r\nLet $\\beta > 1$ and $H = \\beta L_p$. Then\r\n\\beq\\label{eq-DecFB}\r\n\\ba{rcl}\r\n\\la F'(T), x - T \\ra & \\geq & \\left( {p! \\over (p+1)L_p}\r\n\\right)^{1 \\over p} \\cdot \\| F'(T) \\|_*^{p+1 \\over p}\r\n\\cdot {(\\beta^2 - 1)^{p-1 \\over 2p} \\over \\beta} \\cdot {p\r\n\t\\over (p^2-1)^{p-1 \\over 2p}}.\r\n\\ea\r\n\\eeq\r\nIn particular, if $\\beta = p$, then\r\n\\beq\\label{eq-DecF}\r\n\\ba{rcl}\r\n\\la F'(T), x - T \\ra & \\geq & \\left( {p! \\over (p+1)L_p}\r\n\\right)^{1 \\over p} \\cdot \\| F'(T) \\|_*^{p+1 \\over p}.\r\n\\ea\r\n\\eeq\r\n\\EL","start":16003,"end":16510},{"raw_tex":"\\proof\r\nDenote $r = \\| T  - x \\|$, $h = {H \\over p!}$, and $l =\r\n{L_p \\over p!}$. Then inequality (\\ref{eq-N3}) can be\r\nwritten as follows:\r\n$$\r\n\\ba{rcl}\r\n\\| F'(T) + h r^{p-1} B(T-x) \\|^2_* & \\leq & l^2 r^{2p}.\r\n\\ea\r\n$$\r\nThis means that\r\n\\beq\\label{eq-N4}\r\n\\ba{rcl}\r\n\\la F'(T), x - T \\ra & \\geq & {1 \\over 2 h r^{p-1}} \\|\r\nF'(T) \\|_*^2 + {r^{2p} (h^2 - l^2) \\over 2h r^{p-1}}.\r\n\\ea\r\n\\eeq\r\nDenote\r\n$$\r\n\\ba{rcl}\r\na & = & {1 \\over 2h} \\| F'(T) \\|_*^2, \\quad b \\; = \\; {h^2\r\n\t- l^2 \\over 2h}, \\quad \\tau \\; = \\; r^{p-1}, \\quad \\alpha\r\n\\; = \\; {p+1 \\over p-1}.\r\n\\ea\r\n$$\r\nThen inequality (\\ref{eq-N4}) can be rewritten as follows:\r\n$$\r\n\\ba{rcl}\r\n\\la F'(T) , x - T \\ra & \\geq & {a \\over \\tau} + b\r\n\\tau^{\\alpha} \\; \\geq \\; \\min\\limits_{t > 0} \\left\\{ {a\r\n\t\\over t} + b t^{\\alpha} \\right\\} \\; = \\; (1+\\alpha)\r\n\\left({a \\over \\alpha} \\right)^{\\alpha \\over 1 + \\alpha}\r\nb^{1 \\over 1 + \\alpha}.\r\n\\ea\r\n$$\r\nTaking into account that $1+\\alpha = {2p \\over p-1}$ and\r\n${\\alpha \\over 1 + \\alpha} = {p + 1 \\over 2p}$, and using the\r\nactual meaning of $a$, $b$, and $\\alpha$, we get\r\n$$\r\n\\ba{rcl}\r\n\\la F'(T), x - T \\ra & \\geq & {2 p \\over p-1} \\cdot { \\|\r\n\tF'(T) \\|_*^{p+1 \\over p} \\over (2h)^{p+1 \\over 2p}} \\cdot\r\n{(p-1)^{p+1 \\over 2p} \\over (p+1)^{p+1 \\over 2p}} \\cdot\r\n{(h^2 - l^2)^{p-1 \\over 2p} \\over (2h)^{p-1 \\over 2p}}\\\\\r\n\\\\\r\n& = & \\| F'(T) \\|_*^{p+1 \\over p} \\cdot {(h^2 - l^2)^{p-1\r\n\t\t\\over 2p} \\over h} \\cdot {p \\over (p+1)^{p+1 \\over 2p}\r\n\t(p-1)^{p-1 \\over 2p}}\\\\\r\n\\\\\r\n& = & \\| F'(T) \\|_*^{p+1 \\over p} \\cdot {(h^2 - l^2)^{p-1\r\n\t\t\\over 2p} \\over h} \\cdot {p \\over (p^2-1)^{p-1 \\over 2p}\r\n\t(p+1)^{1 \\over p}}.\r\n\\ea\r\n$$\r\nIt remains to note that\r\n$$\r\n\\ba{rcl}\r\n{(h^2 - l^2)^{p-1 \\over 2p} \\over h} & = & {(H^2 -\r\n\tL_p^2)^{p-1 \\over 2p} \\over H} \\cdot (p!)^{1 \\over p} \\; =\r\n\\; {(\\beta^2 - 1)^{p-1 \\over 2p} \\over \\beta} \\cdot\r\n\\left({p! \\over L_p} \\right)^{1 \\over p}.\r\n\\ea\r\n$$\r\n\\qed","start":16512,"end":18386},{"raw_tex":"\\proof\r\nIndeed, in view of inequality \\eqref{eq-UniC}, we have\r\n$$\r\n\\ba{rcl}\r\n\\la \\nabla f(x_k) + g_k, x_{k} - x_{k + 1} \\ra\r\n& \\geq &\r\n\\la F'(x_{k + 1}), x_k - x_{k + 1} \\ra + \\sigma_q \\|x_k - x_{k + 1}\\|^q \\\\\r\n\\\\\r\n& \\refGE{eq-DecFB} & \\sigma_q \\|x_k - x_{k + 1}\\|^{q},\r\n\\ea\r\n$$\r\nwhere $g_k$ is an arbitrary vector from $\\partial h(x_k)$.\r\nTherefore, we conclude that\r\n$$\r\n\\ba{rcl}\r\n\\eta(x_k)  & \\geq &  \\sigma_q \\| x_k -\r\nx_{k+1} \\|^{q-1}.\r\n\\ea\r\n$$\r\nIt remains to use inequality\r\n(\\ref{eq-NewG}).\r\n\\qed","start":21928,"end":22432}]},"nodes":[{"id":"S01","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"title":"空间、对偶配对与范数","source_units":[{"raw_tex":"{\\bf Notations and generalities.} In what follows, we\r\ndenote by $\\E$ a finite-dimensional real vector space, and\r\nby $\\E^*$ its dual spaced composed by linear functions on\r\n$\\E$. For such a function $s \\in \\E^*$, we denote by $\\la\r\ns, x \\ra$ its value at $x \\in \\E$. Using a self-adjoint\r\npositive-definite operator $B: \\E \\to \\E^*$ (notation $B =\r\nB^* \\succ 0$), we can endow these spaces with mutually\r\nconjugate Euclidean norms:\r\n$$\r\n\\ba{rcl}\r\n\\| x \\| & = & \\la B x, x \\ra^{1/2}, \\quad x \\in \\E, \\quad\r\n\\| g \\|_* \\; = \\; \\la g, B^{-1} g \\ra^{1/2}, \\quad g \\in\r\n\\E^*.\r\n\\ea\r\n$$\r\n\r\n","start":9634,"end":10217}]},{"id":"S02","semantic_type":"definition","kind":"definition","labels":["eq-DNorm1","eq-DNorm2"],"declared_external":false,"title":"导数、次微分与张量范数","source_units":[{"raw_tex":"For a smooth function $f: \\dom f \\to \\R$ with convex and\r\nopen domain $\\dom f  \\subseteq \\E$, denote by $\\nabla\r\nf(x)$ its gradient, and by $\\nabla^2 f(x)$ its Hessian\r\nevaluated at point $x \\in \\dom f \\subseteq \\E$. Note that\r\n$$\r\n\\ba{rcl}\r\n\\nabla f(x) & \\in & \\E^*, \\quad \\nabla^2 f(x) h \\; \\in \\;\r\n\\E^*, \\quad x \\in \\dom f, \\; h \\in \\E.\r\n\\ea\r\n$$\r\nFor non-differentiable convex function $f(\\cdot)$, we\r\ndenote by $\\partial f(x) \\subset \\E^*$ its subdifferential\r\nat the point $x \\in \\dom f$.\r\n\r\nIn what follows, we often work with directional\r\nderivatives. For $p \\geq 1$, denote by\r\n$$\r\n\\ba{c}\r\nD^p f(x)[h_1, \\dots, h_p]\r\n\\ea\r\n$$\r\nthe directional derivative of function $f$ at $x$ along\r\ndirections $h_i \\in \\E$, $i = 1, \\dots, p$. If all\r\ndirections $h_1, \\dots, h_p$ are the same, we apply a\r\nsimpler notation\r\n$$\r\n\\ba{c}\r\nD^p f(x)[h]^p, \\quad h \\in \\E.\r\n\\ea\r\n$$\r\n\r\nNote that $D^p f(x)[ \\cdot]$ is a {\\em symmetric\r\n\t$p$-linear form}. Its {\\em norm} is defined in the\r\nstandard way:\r\n\\beq\\label{eq-DNorm1}\r\n\\ba{rcl}\r\n\\| D^pf(x) \\| & = & \\max\\limits_{h_1, \\dots, h_p \\in \\E}\r\n\\left\\{ D^p f(x)[h_1, \\dots, h_p ]: \\; \\| h_i \\| \\leq 1,\r\n\\, i = 1,\r\n\\dots, p \\right\\}\\\\\r\n\\\\\r\n& = & \\max\\limits_{h \\in \\E} \\left\\{ \\Big| D^p\r\nf(x)[h]^p\\Big|: \\; \\| h \\| \\leq 1 \\right\\}\r\n\\ea\r\n\\eeq\r\n(for the last equation see, for example, Appendix 1 in\r\n\\cite{nesterov1994interior}). Similarly, we define\r\n\\beq\\label{eq-DNorm2}\r\n\\ba{rcl}\r\n\\| D^pf(x) - D^pf(y) \\| & = & \\max\\limits_{h \\in \\E}\r\n\\left\\{ \\Big| D^p f(x)[h]^p - D^pf(y)[h]^p\\Big|: \\; \\| h\r\n\\| \\leq 1 \\right\\}.\r\n\\ea\r\n\\eeq\r\n\r\nIn particular, for any $x \\in \\dom f$ and $h_1, h_2 \\in\r\n\\E$, we have\r\n$$\r\n\\ba{rcl}\r\nDf(x)[h_1] & = & \\la \\nabla f(x), h_1 \\ra, \\quad\r\nD^2f(x)[h_1, h_2] \\; = \\; \\la \\nabla^2 f(x) h_1, h_2 \\ra.\r\n\\ea\r\n$$\r\nThus, for the Hessian, our definition corresponds to a\r\n{\\em spectral norm} of the self-adjoint linear operator\r\n(maximal module of all eigenvalues computed with respect\r\nto $B \\succ 0$).\r\n","start":10217,"end":12173}]},{"id":"S03","semantic_type":"definition","kind":"definition","labels":["eq-PGrad"],"declared_external":false,"title":"Taylor 模型及其梯度","source_units":[{"raw_tex":"Finally, the Taylor approximation of function $f(\\cdot)$\r\nat $x \\in \\dom f$ is defined as follows:\r\n$$\r\n\\ba{rcl}\r\nf(x+h) & = & \\Omega_p(f, x; x + h)  + o(\\|h\\|^p), \\quad x+h \\in\r\n\\dom\r\nf,\\\\\r\n\\\\\r\n\\Omega_p(f,x;y) & \\Def & f(x) + \\sum\\limits_{k=1}^p {1\r\n\t\\over k!} D^k f(x)[y-x]^k, \\quad y \\in \\E.\r\n\\ea\r\n$$\r\nConsequently, for all $y \\in \\E$ we have\r\n\\beq\\label{eq-PGrad}\r\n\\ba{rcl}\r\n\\nabla \\Omega_p(f,x;y) & = & \\sum\\limits_{k=1}^p {1 \\over\r\n\t(k-1)!} D^k f(x)[y-x]^{k-1},\r\n\\ea\r\n\\eeq\r\n","start":12175,"end":12655}]},{"id":"S04","semantic_type":"problem_setting","kind":"problem_setting","labels":["prob-Main"],"declared_external":false,"title":"复合凸优化问题与研究对象","source_units":[{"raw_tex":"In this paper, we consider the following {\\em composite}\r\nconvex minimization problem\r\n\\beq\\label{prob-Main}\r\n\\min\\limits_{x \\in \\dom h} \\Big\\{ F(x) = f(x) + h(x)\r\n\\Big\\},\r\n\\eeq\r\nwhere $h: \\E \\to \\R \\cup \\{+\\infty\\}$ is a {\\em simple} proper\r\nclosed convex function and $f \\in C^{p,p}(\\dom h)$ for a\r\ncertain $p \\geq 2$. ","start":12847,"end":13168}]},{"id":"S05","semantic_type":"assumption","kind":"assumption","labels":["eq-Lip"],"declared_external":false,"title":"p 阶导数 Lipschitz 连续","source_units":[{"raw_tex":"In other words, we assume that the\r\n$p$th derivative of function $f$ is Lipschitz continuous:\r\n\\beq\\label{eq-Lip}\r\n\\ba{rcl}\r\n\\| D^p f(x) - D^p f(y) \\| & \\leq & L_p \\| x - y \\|,\r\n\\quad x, y \\in \\dom h.\r\n\\ea\r\n\\eeq","start":13168,"end":13379}]},{"id":"S06","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq-BoundF","eq-BoundG","eq-BoundH"],"declared_external":false,"title":"Taylor 函数值、梯度与 Hessian 余项界","source_units":[{"raw_tex":"\r\n\r\nAssuming that $L_{p} < +\\infty$, by the standard\r\nintegration arguments we can bound the residual between\r\nfunction value and its Taylor approximation:\r\n\\beq\\label{eq-BoundF}\r\n\\ba{rcl}\r\n| f(y) - \\Omega_p(f,x;y) | & \\leq & {L_{p} \\over (p+1)!}\r\n\\| y - x \\|^{p+1}, \\quad x, y \\in \\dom h.\r\n\\ea\r\n\\eeq\r\nApplying the same reasoning to functions $\\la \\nabla\r\nf(\\cdot), h \\ra$ and $\\la \\nabla^2 f(\\cdot) h, h \\ra$ with\r\ndirection $h \\in \\E$ being fixed, we get the following\r\nguarantees:\r\n\\beq\\label{eq-BoundG}\r\n\\ba{rcl}\r\n\\| \\nabla f(y) - \\nabla \\Omega_p(f,x;y) \\|_* & \\leq & {L_p\r\n\t\\over p!} \\| y - x \\|^{p},\r\n\\ea\r\n\\eeq\r\n\\beq\\label{eq-BoundH}\r\n\\ba{rcl}\r\n\\| \\nabla^2 f(y) - \\nabla^2 \\Omega_p(f,x;y) \\| & \\leq &\r\n{L_p \\over (p-1)!} \\| y - x \\|^{p-1},\r\n\\ea\r\n\\eeq\r\nwhich are valid for all $x, y \\in \\dom h$.\r\n","start":13379,"end":14181}]},{"id":"S07","semantic_type":"definition","kind":"definition","labels":["def-RCTS"],"declared_external":false,"title":"正则化张量步 T_H","source_units":[{"raw_tex":"Let us define now one step of the {\\em Regularized\r\n\tComposite Tensor Method} (RCTM) of degree $p \\geq 2$:\r\n\\beq\\label{def-RCTS}\r\n\\ba{rcl}\r\nT & \\equiv & T_H(x) \\; \\Def \\; \\arg\\min\\limits_{y \\in \\E}\r\n\\left\\{ \\Omega_p(f,x;y) + {H \\over (p+1)!} \\| y - x\r\n\\|^{p+1} + h(y) \\right\\}.\r\n\\ea\r\n\\eeq","start":14183,"end":14471}]},{"id":"S08","semantic_type":"assumption","kind":"assumption","labels":["eq-H"],"declared_external":false,"title":"正则化条件 H≥pL_p","source_units":[{"raw_tex":"\r\nIt can be shown that for\r\n\\beq\\label{eq-H}\r\n\\ba{c}\r\n\\mbox{\\fbox{\\rule[-2mm]{0cm}{6mm}$H \\; \\geq \\; p L_p$}}\r\n\\ea\r\n\\eeq\r\nthe auxiliary optimization problem in (\\ref{def-RCTS}) is\r\n{\\em convex} (see Theorem 1 in \\cite{nesterov2019implementable}).\r\nThis condition\r\nis crucial for implementability of our methods and we\r\nalways assume it to be satisfied.\r\n\r\n","start":14471,"end":14827}]},{"id":"S09","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["Convexity of the regularized Taylor model: Theorem 1 of nesterov2019implementable, cited at lines 424–434.","Symmetric multilinear-form norm identity: Appendix 1 of nesterov1994interior, cited at lines 304–326.","Standard Taylor remainder estimates, convex subdifferential calculus, first-order optimality, dual-norm Cauchy–Schwarz, and elementary scalar minimization."],"title":"正则化 Taylor 模型的凸性","source_units":[{"raw_tex":"\r\nthe auxiliary optimization problem in (\\ref{def-RCTS}) is\r\n{\\em convex} (see Theorem 1 in \\cite{nesterov2019implementable}).\r\nThis condition\r\nis crucial for implementability of our methods and we\r\nalways assume it to be satisfied.\r\n\r\n","start":14591,"end":14827}]},{"id":"S10","semantic_type":"proof_step","kind":"proof_step","labels":["eq-OptC"],"declared_external":false,"title":"子问题一阶最优性条件","source_units":[{"raw_tex":"Let us write down the first-order optimality condition for\r\nthe auxiliary optimization problem in (\\ref{def-RCTS}):\r\n\\beq\\label{eq-OptC}\r\n\\ba{rcl}\r\n\\la \\nabla \\Omega_p(f,x;T) + {H \\over p!} \\| T - x\r\n\\|^{p-1}B(T-x), y - T \\ra + h(y) & \\geq & h(T),\r\n\\ea\r\n\\eeq\r\nfor all $y \\in \\dom h$.\r\n","start":14827,"end":15112}]},{"id":"S11","semantic_type":"definition","kind":"definition","labels":["def-HP"],"declared_external":false,"title":"选定的 h 次梯度 h′(T)","source_units":[{"raw_tex":"In other words, for vector\r\n\\beq\\label{def-HP}\r\n\\ba{rcl}\r\nh'(T) & \\Def & - \\left( \\nabla \\Omega_p(f,x;T) + {H \\over\r\n\tp!} \\| T - x \\|^{p-1}B(T-x) \\right)\r\n\\ea\r\n\\eeq\r\nwe have $h'(T) \\stackrel{(\\ref{eq-OptC})}{\\in} \\partial\r\nh(T)$. ","start":15112,"end":15342}]},{"id":"S12","semantic_type":"definition","kind":"definition","labels":["def-FP"],"declared_external":false,"title":"选定的 F 次梯度 F′(T)","source_units":[{"raw_tex":"This fact explains our notation\r\n\\beq\\label{def-FP}\r\n\\ba{rcl}\r\nF'(T) & \\Def & \\nabla f(T) + h'(T) \\; \\in \\partial F(T).\r\n\\ea\r\n\\eeq","start":15342,"end":15472}]},{"id":"S13","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq-N3"],"declared_external":false,"title":"梯度模型误差界","source_units":[{"raw_tex":"\r\n\r\nLet us present some properties of the point $T = T_H(x)$.\r\nFirst of all, we need some bounds for the norm of vector\r\n$F'(T)$. Note that\r\n\\beq\\label{eq-N3}\r\n\\ba{rcl}\r\n\\Big\\| F'(T) + {H \\over p!} \\| T - x \\|^{p-1}B(T-x)\r\n\\Big\\|_* & \\refEQ{def-HP} & \\Big\\| \\nabla f(T) - \\nabla\r\n\\Omega_p(f,x;T) \\Big\\|_*\\\\\r\n\\\\\r\n& \\refLE{eq-BoundG} & {L_p \\over p!} \\| T - x \\|^p.\r\n\\ea\r\n\\eeq","start":15472,"end":15846}]},{"id":"S14","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq-NewG"],"declared_external":false,"title":"选定次梯度范数上界","source_units":[{"raw_tex":"\r\nConsequently,\r\n\\beq\\label{eq-NewG}\r\n\\ba{rcl}\r\n\\| F'(T) \\|_* & \\leq & {L_p+H \\over p!} \\| T - x \\|^p.\r\n\\ea\r\n\\eeq\r\n","start":15846,"end":15961}]},{"id":"S15","semantic_type":"algorithm","kind":"algorithm","labels":["met-RCTM"],"declared_external":false,"title":"RCTM 迭代规则","source_units":[{"raw_tex":"The main goal of this paper consists in analyzing the\r\nlocal behavior of the {\\em Regularized Composite Tensor\r\n\tMethod} (RCTM):\r\n\\beq\\label{met-RCTM}\r\n\\ba{rcl}\r\nx_0 \\; \\in \\; \\dom h, \\quad x_{k+1} & = & T_H(x_k), \\quad\r\nk \\geq 0,\r\n\\ea\r\n\\eeq","start":18444,"end":18685}]},{"id":"S16","semantic_type":"assumption","kind":"assumption","labels":["assumption-Uni","eq-UniC"],"declared_external":false,"title":"目标 F 的 q 次一致凸性","source_units":[{"raw_tex":"\r\nas applied to the problem (\\ref{prob-Main}). In order to\r\nprove local superlinear convergence of this scheme, we\r\nneed one more assumption.\r\n\r\n\\begin{assumption}\\label{assumption-Uni}\r\n\tThe objective in problem\r\n\t(\\ref{prob-Main}) is uniformly convex of degree $q \\geq 2$.\r\n\tThus, for all $x, y \\in \\dom h$ and for all $G_x \\in \\partial F(x), G_y \\in \\partial F(y)$,\r\n\tit holds:\r\n\t\\beq\\label{eq-UniC}\r\n\t\\ba{rcl}\r\n\t\\la G_x - G_y, x - y \\ra & \\geq & \\sigma_q\r\n\t\\| x - y \\|^q, \r\n\t\\ea\r\n\t\\eeq\r\n\tfor certain $\\sigma_q > 0$.\r\n\\end{assumption}\r\n","start":18685,"end":19224}]},{"id":"S17","semantic_type":"definition","kind":"definition","labels":["def-GNorm"],"declared_external":false,"title":"最小次梯度范数 η","source_units":[{"raw_tex":"Let us study now the local convergence of the method\r\n(\\ref{met-RCTM}) in terms of the norm of gradient. For any\r\n$x \\in \\dom h$ denote\r\n\\beq\\label{def-GNorm}\r\n\\ba{rcl}\r\n\\eta(x) & \\Def & \\min\\limits_{g \\in \\partial h(x)} \\| \\nabla\r\nf(x) + g \\|_*.\r\n\\ea\r\n\\eeq\r\nIf $\\partial h(x) = \\emptyset$, we set $\\eta(x) =\r\n+\\infty$.\r\n","start":21357,"end":21678}]},{"id":"S18","semantic_type":"theorem","kind":"theorem","labels":["th-RateG","eq-RateG"],"declared_external":false,"title":"定理 2：最小次梯度范数递推","source_units":[{"raw_tex":"\\BT\\label{th-RateG}\r\nFor any $k \\geq 0$ we have\r\n\\beq\\label{eq-RateG}\r\n\\ba{rcl}\r\n\\eta(x_{k+1})\r\n& \\, \\leq \\, &\r\n\\|F'(x_{k + 1}) \\|_{*}\r\n\\; \\leq \\;\r\n{L_p + H \\over p!} \\left[ {1 \\over\r\n\t\\sigma_q} \\, \\eta(x_k) \\right]^{p \\over q-1}.\r\n\\ea\r\n\\eeq\r\n\\ET","start":21680,"end":21926}]},{"id":"S19","semantic_type":"lemma","kind":"lemma","labels":["lm-DecF","eq-DecFB","eq-DecF"],"declared_external":false,"title":"引理 1：完整下降估计","source_units":[{"raw_tex":"\\BL\\label{lm-DecF}\r\nLet $\\beta > 1$ and $H = \\beta L_p$. Then\r\n\\beq\\label{eq-DecFB}\r\n\\ba{rcl}\r\n\\la F'(T), x - T \\ra & \\geq & \\left( {p! \\over (p+1)L_p}\r\n\\right)^{1 \\over p} \\cdot \\| F'(T) \\|_*^{p+1 \\over p}\r\n\\cdot {(\\beta^2 - 1)^{p-1 \\over 2p} \\over \\beta} \\cdot {p\r\n\t\\over (p^2-1)^{p-1 \\over 2p}}.\r\n\\ea\r\n\\eeq\r\nIn particular, if $\\beta = p$, then\r\n\\beq\\label{eq-DecF}\r\n\\ba{rcl}\r\n\\la F'(T), x - T \\ra & \\geq & \\left( {p! \\over (p+1)L_p}\r\n\\right)^{1 \\over p} \\cdot \\| F'(T) \\|_*^{p+1 \\over p}.\r\n\\ea\r\n\\eeq\r\n\\EL","start":16003,"end":16510}]},{"id":"S20","semantic_type":"proof","kind":"proof","labels":["eq-N4"],"declared_external":false,"title":"下降引理的完整作者证明","source_units":[{"raw_tex":"\\proof\r\nDenote $r = \\| T  - x \\|$, $h = {H \\over p!}$, and $l =\r\n{L_p \\over p!}$. Then inequality (\\ref{eq-N3}) can be\r\nwritten as follows:\r\n$$\r\n\\ba{rcl}\r\n\\| F'(T) + h r^{p-1} B(T-x) \\|^2_* & \\leq & l^2 r^{2p}.\r\n\\ea\r\n$$\r\nThis means that\r\n\\beq\\label{eq-N4}\r\n\\ba{rcl}\r\n\\la F'(T), x - T \\ra & \\geq & {1 \\over 2 h r^{p-1}} \\|\r\nF'(T) \\|_*^2 + {r^{2p} (h^2 - l^2) \\over 2h r^{p-1}}.\r\n\\ea\r\n\\eeq\r\nDenote\r\n$$\r\n\\ba{rcl}\r\na & = & {1 \\over 2h} \\| F'(T) \\|_*^2, \\quad b \\; = \\; {h^2\r\n\t- l^2 \\over 2h}, \\quad \\tau \\; = \\; r^{p-1}, \\quad \\alpha\r\n\\; = \\; {p+1 \\over p-1}.\r\n\\ea\r\n$$\r\nThen inequality (\\ref{eq-N4}) can be rewritten as follows:\r\n$$\r\n\\ba{rcl}\r\n\\la F'(T) , x - T \\ra & \\geq & {a \\over \\tau} + b\r\n\\tau^{\\alpha} \\; \\geq \\; \\min\\limits_{t > 0} \\left\\{ {a\r\n\t\\over t} + b t^{\\alpha} \\right\\} \\; = \\; (1+\\alpha)\r\n\\left({a \\over \\alpha} \\right)^{\\alpha \\over 1 + \\alpha}\r\nb^{1 \\over 1 + \\alpha}.\r\n\\ea\r\n$$\r\nTaking into account that $1+\\alpha = {2p \\over p-1}$ and\r\n${\\alpha \\over 1 + \\alpha} = {p + 1 \\over 2p}$, and using the\r\nactual meaning of $a$, $b$, and $\\alpha$, we get\r\n$$\r\n\\ba{rcl}\r\n\\la F'(T), x - T \\ra & \\geq & {2 p \\over p-1} \\cdot { \\|\r\n\tF'(T) \\|_*^{p+1 \\over p} \\over (2h)^{p+1 \\over 2p}} \\cdot\r\n{(p-1)^{p+1 \\over 2p} \\over (p+1)^{p+1 \\over 2p}} \\cdot\r\n{(h^2 - l^2)^{p-1 \\over 2p} \\over (2h)^{p-1 \\over 2p}}\\\\\r\n\\\\\r\n& = & \\| F'(T) \\|_*^{p+1 \\over p} \\cdot {(h^2 - l^2)^{p-1\r\n\t\t\\over 2p} \\over h} \\cdot {p \\over (p+1)^{p+1 \\over 2p}\r\n\t(p-1)^{p-1 \\over 2p}}\\\\\r\n\\\\\r\n& = & \\| F'(T) \\|_*^{p+1 \\over p} \\cdot {(h^2 - l^2)^{p-1\r\n\t\t\\over 2p} \\over h} \\cdot {p \\over (p^2-1)^{p-1 \\over 2p}\r\n\t(p+1)^{1 \\over p}}.\r\n\\ea\r\n$$\r\nIt remains to note that\r\n$$\r\n\\ba{rcl}\r\n{(h^2 - l^2)^{p-1 \\over 2p} \\over h} & = & {(H^2 -\r\n\tL_p^2)^{p-1 \\over 2p} \\over H} \\cdot (p!)^{1 \\over p} \\; =\r\n\\; {(\\beta^2 - 1)^{p-1 \\over 2p} \\over \\beta} \\cdot\r\n\\left({p! \\over L_p} \\right)^{1 \\over p}.\r\n\\ea\r\n$$\r\n\\qed","start":16512,"end":18386}]},{"id":"S21","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"title":"平方范数形式与局部标量记号","source_units":[{"raw_tex":"\\proof\r\nDenote $r = \\| T  - x \\|$, $h = {H \\over p!}$, and $l =\r\n{L_p \\over p!}$. Then inequality (\\ref{eq-N3}) can be\r\nwritten as follows:\r\n$$\r\n\\ba{rcl}\r\n\\| F'(T) + h r^{p-1} B(T-x) \\|^2_* & \\leq & l^2 r^{2p}.\r\n\\ea\r\n$$\r\n","start":16512,"end":16733}]},{"id":"S22","semantic_type":"proof_step","kind":"proof_step","labels":["eq-N4"],"declared_external":false,"title":"展开得到下降内积下界","source_units":[{"raw_tex":"This means that\r\n\\beq\\label{eq-N4}\r\n\\ba{rcl}\r\n\\la F'(T), x - T \\ra & \\geq & {1 \\over 2 h r^{p-1}} \\|\r\nF'(T) \\|_*^2 + {r^{2p} (h^2 - l^2) \\over 2h r^{p-1}}.\r\n\\ea\r\n\\eeq\r\n","start":16733,"end":16901}]},{"id":"S23","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"title":"标量重参数化","source_units":[{"raw_tex":"Denote\r\n$$\r\n\\ba{rcl}\r\na & = & {1 \\over 2h} \\| F'(T) \\|_*^2, \\quad b \\; = \\; {h^2\r\n\t- l^2 \\over 2h}, \\quad \\tau \\; = \\; r^{p-1}, \\quad \\alpha\r\n\\; = \\; {p+1 \\over p-1}.\r\n\\ea\r\n$$\r\n","start":16901,"end":17078}]},{"id":"S24","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"title":"计算标量极小值","source_units":[{"raw_tex":"Then inequality (\\ref{eq-N4}) can be rewritten as follows:\r\n$$\r\n\\ba{rcl}\r\n\\la F'(T) , x - T \\ra & \\geq & {a \\over \\tau} + b\r\n\\tau^{\\alpha} \\; \\geq \\; \\min\\limits_{t > 0} \\left\\{ {a\r\n\t\\over t} + b t^{\\alpha} \\right\\} \\; = \\; (1+\\alpha)\r\n\\left({a \\over \\alpha} \\right)^{\\alpha \\over 1 + \\alpha}\r\nb^{1 \\over 1 + \\alpha}.\r\n\\ea\r\n$$\r\n","start":17078,"end":17406}]},{"id":"S25","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"title":"代入并整理全部常数","source_units":[{"raw_tex":"Taking into account that $1+\\alpha = {2p \\over p-1}$ and\r\n${\\alpha \\over 1 + \\alpha} = {p + 1 \\over 2p}$, and using the\r\nactual meaning of $a$, $b$, and $\\alpha$, we get\r\n$$\r\n\\ba{rcl}\r\n\\la F'(T), x - T \\ra & \\geq & {2 p \\over p-1} \\cdot { \\|\r\n\tF'(T) \\|_*^{p+1 \\over p} \\over (2h)^{p+1 \\over 2p}} \\cdot\r\n{(p-1)^{p+1 \\over 2p} \\over (p+1)^{p+1 \\over 2p}} \\cdot\r\n{(h^2 - l^2)^{p-1 \\over 2p} \\over (2h)^{p-1 \\over 2p}}\\\\\r\n\\\\\r\n& = & \\| F'(T) \\|_*^{p+1 \\over p} \\cdot {(h^2 - l^2)^{p-1\r\n\t\t\\over 2p} \\over h} \\cdot {p \\over (p+1)^{p+1 \\over 2p}\r\n\t(p-1)^{p-1 \\over 2p}}\\\\\r\n\\\\\r\n& = & \\| F'(T) \\|_*^{p+1 \\over p} \\cdot {(h^2 - l^2)^{p-1\r\n\t\t\\over 2p} \\over h} \\cdot {p \\over (p^2-1)^{p-1 \\over 2p}\r\n\t(p+1)^{1 \\over p}}.\r\n\\ea\r\n$$\r\nIt remains to note that\r\n$$\r\n\\ba{rcl}\r\n{(h^2 - l^2)^{p-1 \\over 2p} \\over h} & = & {(H^2 -\r\n\tL_p^2)^{p-1 \\over 2p} \\over H} \\cdot (p!)^{1 \\over p} \\; =\r\n\\; {(\\beta^2 - 1)^{p-1 \\over 2p} \\over \\beta} \\cdot\r\n\\left({p! \\over L_p} \\right)^{1 \\over p}.\r\n\\ea\r\n$$\r\n\\qed","start":17406,"end":18386}]},{"id":"S26","semantic_type":"assumption","kind":"assumption","labels":["lm-DecF"],"declared_external":false,"title":"下降引理局部参数","source_units":[{"raw_tex":"\\BL\\label{lm-DecF}\r\nLet $\\beta > 1$ and $H = \\beta L_p$. Then","start":16003,"end":16064}]},{"id":"S27","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\proof\r\nIndeed, in view of inequality \\eqref{eq-UniC}, we have\r\n$$\r\n\\ba{rcl}\r\n\\la \\nabla f(x_k) + g_k, x_{k} - x_{k + 1} \\ra\r\n& \\geq &\r\n\\la F'(x_{k + 1}), x_k - x_{k + 1} \\ra + \\sigma_q \\|x_k - x_{k + 1}\\|^q \\\\\r\n\\\\\r\n& \\refGE{eq-DecFB} & \\sigma_q \\|x_k - x_{k + 1}\\|^{q},\r\n\\ea\r\n$$\r\nwhere $g_k$ is an arbitrary vector from $\\partial h(x_k)$.\r\nTherefore, we conclude that\r\n$$\r\n\\ba{rcl}\r\n\\eta(x_k)  & \\geq &  \\sigma_q \\| x_k -\r\nx_{k+1} \\|^{q-1}.\r\n\\ea\r\n$$\r\nIt remains to use inequality\r\n(\\ref{eq-NewG}).\r\n\\qed","start":21928,"end":22432}]},{"id":"S28","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"title":"一致凸性与下降性给出内积下界","source_units":[{"raw_tex":"\\proof\r\nIndeed, in view of inequality \\eqref{eq-UniC}, we have\r\n$$\r\n\\ba{rcl}\r\n\\la \\nabla f(x_k) + g_k, x_{k} - x_{k + 1} \\ra\r\n& \\geq &\r\n\\la F'(x_{k + 1}), x_k - x_{k + 1} \\ra + \\sigma_q \\|x_k - x_{k + 1}\\|^q \\\\\r\n\\\\\r\n& \\refGE{eq-DecFB} & \\sigma_q \\|x_k - x_{k + 1}\\|^{q},\r\n\\ea\r\n$$","start":21928,"end":22207}]},{"id":"S29","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"title":"主证明实际使用的下降子事实","source_units":[{"raw_tex":"$$\r\n\\ba{rcl}\r\n\\la \\nabla f(x_k) + g_k, x_{k} - x_{k + 1} \\ra\r\n& \\geq &\r\n\\la F'(x_{k + 1}), x_k - x_{k + 1} \\ra + \\sigma_q \\|x_k - x_{k + 1}\\|^q \\\\\r\n\\\\\r\n& \\refGE{eq-DecFB} & \\sigma_q \\|x_k - x_{k + 1}\\|^{q},\r\n\\ea\r\n$$","start":21992,"end":22207}]},{"id":"S30","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"title":"用 η 控制迭代步长","source_units":[{"raw_tex":"\r\nwhere $g_k$ is an arbitrary vector from $\\partial h(x_k)$.\r\nTherefore, we conclude that\r\n$$\r\n\\ba{rcl}\r\n\\eta(x_k)  & \\geq &  \\sigma_q \\| x_k -\r\nx_{k+1} \\|^{q-1}.\r\n\\ea\r\n$$\r\n","start":22207,"end":22380}]},{"id":"S31","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"title":"合并次梯度范数界得到递推","source_units":[{"raw_tex":"It remains to use inequality\r\n(\\ref{eq-NewG}).\r\n\\qed","start":22380,"end":22432}]},{"id":"S32","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"title":"定理之后的超线性收敛解释","source_units":[{"raw_tex":"As we can see, the condition for superlinear convergence\r\nof the method (\\ref{met-RCTM}) in terms of the norm of the\r\ngradient is the same as in Corollary \\ref{cor-LocF}: we\r\nneed to have ${p \\over q-1} > 1$, that is $p > q-1$.\r\nMoreover, the local rate of convergence has the same order\r\nas that for the residual of the function value.\r\n","start":22436,"end":22774}]},{"id":"S33","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["Convexity of the regularized Taylor model: Theorem 1 of nesterov2019implementable, cited at lines 424–434.","Symmetric multilinear-form norm identity: Appendix 1 of nesterov1994interior, cited at lines 304–326.","Standard Taylor remainder estimates, convex subdifferential calculus, first-order optimality, dual-norm Cauchy–Schwarz, and elementary scalar minimization."],"title":"对偶范数 Cauchy–Schwarz 不等式","source_units":[]},{"id":"S34","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["Convexity of the regularized Taylor model: Theorem 1 of nesterov2019implementable, cited at lines 424–434.","Symmetric multilinear-form norm identity: Appendix 1 of nesterov1994interior, cited at lines 304–326.","Standard Taylor remainder estimates, convex subdifferential calculus, first-order optimality, dual-norm Cauchy–Schwarz, and elementary scalar minimization."],"title":"对称多线性形式范数恒等式","source_units":[{"raw_tex":"(for the last equation see, for example, Appendix 1 in\r\n\\cite{nesterov1994interior}). ","start":11494,"end":11580}]},{"id":"S35","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["Convexity of the regularized Taylor model: Theorem 1 of nesterov2019implementable, cited at lines 424–434.","Symmetric multilinear-form norm identity: Appendix 1 of nesterov1994interior, cited at lines 304–326.","Standard Taylor remainder estimates, convex subdifferential calculus, first-order optimality, dual-norm Cauchy–Schwarz, and elementary scalar minimization."],"title":"标准微积分、凸次微分与标量最小化","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S18","edges":[{"source_id":"S01","target_id":"S02","kind":"supports"},{"source_id":"S34","target_id":"S02","kind":"supports"},{"source_id":"S02","target_id":"S03","kind":"supports"},{"source_id":"S04","target_id":"S06","kind":"supports"},{"source_id":"S05","target_id":"S06","kind":"supports"},{"source_id":"S03","target_id":"S06","kind":"supports"},{"source_id":"S35","target_id":"S06","kind":"supports"},{"source_id":"S04","target_id":"S07","kind":"supports"},{"source_id":"S03","target_id":"S07","kind":"supports"},{"source_id":"S07","target_id":"S10","kind":"supports"},{"source_id":"S08","target_id":"S10","kind":"supports"},{"source_id":"S09","target_id":"S10","kind":"supports"},{"source_id":"S35","target_id":"S10","kind":"supports"},{"source_id":"S07","target_id":"S11","kind":"supports"},{"source_id":"S03","target_id":"S11","kind":"supports"},{"source_id":"S10","target_id":"S11","kind":"supports"},{"source_id":"S04","target_id":"S12","kind":"supports"},{"source_id":"S11","target_id":"S12","kind":"supports"},{"source_id":"S35","target_id":"S12","kind":"supports"},{"source_id":"S06","target_id":"S13","kind":"supports"},{"source_id":"S11","target_id":"S13","kind":"supports"},{"source_id":"S12","target_id":"S13","kind":"supports"},{"source_id":"S13","target_id":"S14","kind":"supports"},{"source_id":"S01","target_id":"S14","kind":"supports"},{"source_id":"S07","target_id":"S15","kind":"supports"},{"source_id":"S04","target_id":"S15","kind":"supports"},{"source_id":"S01","target_id":"S16","kind":"supports"},{"source_id":"S04","target_id":"S16","kind":"supports"},{"source_id":"S04","target_id":"S17","kind":"supports"},{"source_id":"S02","target_id":"S17","kind":"supports"},{"source_id":"S01","target_id":"S17","kind":"supports"},{"source_id":"S13","target_id":"S21","kind":"supports"},{"source_id":"S26","target_id":"S21","kind":"supports"},{"source_id":"S01","target_id":"S21","kind":"supports"},{"source_id":"S21","target_id":"S22","kind":"supports"},{"source_id":"S22","target_id":"S23","kind":"supports"},{"source_id":"S26","target_id":"S23","kind":"supports"},{"source_id":"S23","target_id":"S24","kind":"supports"},{"source_id":"S35","target_id":"S24","kind":"supports"},{"source_id":"S24","target_id":"S25","kind":"supports"},{"source_id":"S26","target_id":"S25","kind":"supports"},{"source_id":"S21","target_id":"S20","kind":"supports"},{"source_id":"S22","target_id":"S20","kind":"supports"},{"source_id":"S23","target_id":"S20","kind":"supports"},{"source_id":"S24","target_id":"S20","kind":"supports"},{"source_id":"S25","target_id":"S20","kind":"supports"},{"source_id":"S20","target_id":"S19","kind":"supports"},{"source_id":"S26","target_id":"S19","kind":"supports"},{"source_id":"S19","target_id":"S29","kind":"supports"},{"source_id":"S08","target_id":"S29","kind":"supports"},{"source_id":"S04","target_id":"S29","kind":"supports"},{"source_id":"S16","target_id":"S28","kind":"supports"},{"source_id":"S12","target_id":"S28","kind":"supports"},{"source_id":"S29","target_id":"S28","kind":"supports"},{"source_id":"S15","target_id":"S28","kind":"supports"},{"source_id":"S28","target_id":"S30","kind":"supports"},{"source_id":"S17","target_id":"S30","kind":"supports"},{"source_id":"S33","target_id":"S30","kind":"supports"},{"source_id":"S30","target_id":"S31","kind":"supports"},{"source_id":"S14","target_id":"S31","kind":"supports"},{"source_id":"S17","target_id":"S31","kind":"supports"},{"source_id":"S15","target_id":"S31","kind":"supports"},{"source_id":"S28","target_id":"S27","kind":"supports"},{"source_id":"S30","target_id":"S27","kind":"supports"},{"source_id":"S31","target_id":"S27","kind":"supports"},{"source_id":"S27","target_id":"S18","kind":"supports"},{"source_id":"S18","target_id":"S32","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S18","edges":[{"source_id":"S01","target_id":"S03","kind":"supports"},{"source_id":"S04","target_id":"S14","kind":"supports"},{"source_id":"S05","target_id":"S14","kind":"supports"},{"source_id":"S03","target_id":"S14","kind":"supports"},{"source_id":"S04","target_id":"S07","kind":"supports"},{"source_id":"S03","target_id":"S07","kind":"supports"},{"source_id":"S07","target_id":"S12","kind":"supports"},{"source_id":"S08","target_id":"S12","kind":"supports"},{"source_id":"S03","target_id":"S12","kind":"supports"},{"source_id":"S04","target_id":"S12","kind":"supports"},{"source_id":"S12","target_id":"S14","kind":"supports"},{"source_id":"S01","target_id":"S14","kind":"supports"},{"source_id":"S07","target_id":"S15","kind":"supports"},{"source_id":"S04","target_id":"S15","kind":"supports"},{"source_id":"S01","target_id":"S16","kind":"supports"},{"source_id":"S04","target_id":"S16","kind":"supports"},{"source_id":"S04","target_id":"S17","kind":"supports"},{"source_id":"S03","target_id":"S17","kind":"supports"},{"source_id":"S01","target_id":"S17","kind":"supports"},{"source_id":"S14","target_id":"S19","kind":"supports"},{"source_id":"S01","target_id":"S19","kind":"supports"},{"source_id":"S03","target_id":"S19","kind":"supports"},{"source_id":"S19","target_id":"S29","kind":"supports"},{"source_id":"S08","target_id":"S29","kind":"supports"},{"source_id":"S04","target_id":"S29","kind":"supports"},{"source_id":"S16","target_id":"S28","kind":"supports"},{"source_id":"S12","target_id":"S28","kind":"supports"},{"source_id":"S29","target_id":"S28","kind":"supports"},{"source_id":"S15","target_id":"S28","kind":"supports"},{"source_id":"S28","target_id":"S30","kind":"supports"},{"source_id":"S17","target_id":"S30","kind":"supports"},{"source_id":"S01","target_id":"S30","kind":"supports"},{"source_id":"S30","target_id":"S31","kind":"supports"},{"source_id":"S14","target_id":"S31","kind":"supports"},{"source_id":"S17","target_id":"S31","kind":"supports"},{"source_id":"S15","target_id":"S31","kind":"supports"},{"source_id":"S28","target_id":"S27","kind":"supports"},{"source_id":"S30","target_id":"S27","kind":"supports"},{"source_id":"S31","target_id":"S27","kind":"supports"},{"source_id":"S27","target_id":"S18","kind":"supports"}],"nodes":[{"id":"S01","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S01","S33"],"declared_external":false,"title":"范数、配对与对偶范数不等式","source_units":[{"raw_tex":"{\\bf Notations and generalities.} In what follows, we\r\ndenote by $\\E$ a finite-dimensional real vector space, and\r\nby $\\E^*$ its dual spaced composed by linear functions on\r\n$\\E$. For such a function $s \\in \\E^*$, we denote by $\\la\r\ns, x \\ra$ its value at $x \\in \\E$. Using a self-adjoint\r\npositive-definite operator $B: \\E \\to \\E^*$ (notation $B =\r\nB^* \\succ 0$), we can endow these spaces with mutually\r\nconjugate Euclidean norms:\r\n$$\r\n\\ba{rcl}\r\n\\| x \\| & = & \\la B x, x \\ra^{1/2}, \\quad x \\in \\E, \\quad\r\n\\| g \\|_* \\; = \\; \\la g, B^{-1} g \\ra^{1/2}, \\quad g \\in\r\n\\E^*.\r\n\\ea\r\n$$\r\n\r\n","start":9634,"end":10217}]},{"id":"S03","semantic_type":"definition","kind":"definition","labels":["eq-PGrad"],"member_ids":["S03","S02","S34","S35"],"declared_external":false,"title":"Taylor 模型、导数与标准微积分","source_units":[{"raw_tex":"For a smooth function $f: \\dom f \\to \\R$ with convex and\r\nopen domain $\\dom f  \\subseteq \\E$, denote by $\\nabla\r\nf(x)$ its gradient, and by $\\nabla^2 f(x)$ its Hessian\r\nevaluated at point $x \\in \\dom f \\subseteq \\E$. Note that\r\n$$\r\n\\ba{rcl}\r\n\\nabla f(x) & \\in & \\E^*, \\quad \\nabla^2 f(x) h \\; \\in \\;\r\n\\E^*, \\quad x \\in \\dom f, \\; h \\in \\E.\r\n\\ea\r\n$$\r\nFor non-differentiable convex function $f(\\cdot)$, we\r\ndenote by $\\partial f(x) \\subset \\E^*$ its subdifferential\r\nat the point $x \\in \\dom f$.\r\n\r\nIn what follows, we often work with directional\r\nderivatives. For $p \\geq 1$, denote by\r\n$$\r\n\\ba{c}\r\nD^p f(x)[h_1, \\dots, h_p]\r\n\\ea\r\n$$\r\nthe directional derivative of function $f$ at $x$ along\r\ndirections $h_i \\in \\E$, $i = 1, \\dots, p$. If all\r\ndirections $h_1, \\dots, h_p$ are the same, we apply a\r\nsimpler notation\r\n$$\r\n\\ba{c}\r\nD^p f(x)[h]^p, \\quad h \\in \\E.\r\n\\ea\r\n$$\r\n\r\nNote that $D^p f(x)[ \\cdot]$ is a {\\em symmetric\r\n\t$p$-linear form}. Its {\\em norm} is defined in the\r\nstandard way:\r\n\\beq\\label{eq-DNorm1}\r\n\\ba{rcl}\r\n\\| D^pf(x) \\| & = & \\max\\limits_{h_1, \\dots, h_p \\in \\E}\r\n\\left\\{ D^p f(x)[h_1, \\dots, h_p ]: \\; \\| h_i \\| \\leq 1,\r\n\\, i = 1,\r\n\\dots, p \\right\\}\\\\\r\n\\\\\r\n& = & \\max\\limits_{h \\in \\E} \\left\\{ \\Big| D^p\r\nf(x)[h]^p\\Big|: \\; \\| h \\| \\leq 1 \\right\\}\r\n\\ea\r\n\\eeq\r\n(for the last equation see, for example, Appendix 1 in\r\n\\cite{nesterov1994interior}). Similarly, we define\r\n\\beq\\label{eq-DNorm2}\r\n\\ba{rcl}\r\n\\| D^pf(x) - D^pf(y) \\| & = & \\max\\limits_{h \\in \\E}\r\n\\left\\{ \\Big| D^p f(x)[h]^p - D^pf(y)[h]^p\\Big|: \\; \\| h\r\n\\| \\leq 1 \\right\\}.\r\n\\ea\r\n\\eeq\r\n\r\nIn particular, for any $x \\in \\dom f$ and $h_1, h_2 \\in\r\n\\E$, we have\r\n$$\r\n\\ba{rcl}\r\nDf(x)[h_1] & = & \\la \\nabla f(x), h_1 \\ra, \\quad\r\nD^2f(x)[h_1, h_2] \\; = \\; \\la \\nabla^2 f(x) h_1, h_2 \\ra.\r\n\\ea\r\n$$\r\nThus, for the Hessian, our definition corresponds to a\r\n{\\em spectral norm} of the self-adjoint linear operator\r\n(maximal module of all eigenvalues computed with respect\r\nto $B \\succ 0$).\r\n","start":10217,"end":12173},{"raw_tex":"(for the last equation see, for example, Appendix 1 in\r\n\\cite{nesterov1994interior}). ","start":11494,"end":11580},{"raw_tex":"Finally, the Taylor approximation of function $f(\\cdot)$\r\nat $x \\in \\dom f$ is defined as follows:\r\n$$\r\n\\ba{rcl}\r\nf(x+h) & = & \\Omega_p(f, x; x + h)  + o(\\|h\\|^p), \\quad x+h \\in\r\n\\dom\r\nf,\\\\\r\n\\\\\r\n\\Omega_p(f,x;y) & \\Def & f(x) + \\sum\\limits_{k=1}^p {1\r\n\t\\over k!} D^k f(x)[y-x]^k, \\quad y \\in \\E.\r\n\\ea\r\n$$\r\nConsequently, for all $y \\in \\E$ we have\r\n\\beq\\label{eq-PGrad}\r\n\\ba{rcl}\r\n\\nabla \\Omega_p(f,x;y) & = & \\sum\\limits_{k=1}^p {1 \\over\r\n\t(k-1)!} D^k f(x)[y-x]^{k-1},\r\n\\ea\r\n\\eeq\r\n","start":12175,"end":12655}]},{"id":"S04","semantic_type":"problem_setting","kind":"problem_setting","labels":["prob-Main"],"member_ids":["S04"],"declared_external":false,"title":"复合凸优化问题与研究对象","source_units":[{"raw_tex":"In this paper, we consider the following {\\em composite}\r\nconvex minimization problem\r\n\\beq\\label{prob-Main}\r\n\\min\\limits_{x \\in \\dom h} \\Big\\{ F(x) = f(x) + h(x)\r\n\\Big\\},\r\n\\eeq\r\nwhere $h: \\E \\to \\R \\cup \\{+\\infty\\}$ is a {\\em simple} proper\r\nclosed convex function and $f \\in C^{p,p}(\\dom h)$ for a\r\ncertain $p \\geq 2$. ","start":12847,"end":13168}]},{"id":"S05","semantic_type":"assumption","kind":"assumption","labels":["eq-Lip"],"member_ids":["S05"],"declared_external":false,"title":"p 阶导数 Lipschitz 连续","source_units":[{"raw_tex":"In other words, we assume that the\r\n$p$th derivative of function $f$ is Lipschitz continuous:\r\n\\beq\\label{eq-Lip}\r\n\\ba{rcl}\r\n\\| D^p f(x) - D^p f(y) \\| & \\leq & L_p \\| x - y \\|,\r\n\\quad x, y \\in \\dom h.\r\n\\ea\r\n\\eeq","start":13168,"end":13379}]},{"id":"S07","semantic_type":"definition","kind":"definition","labels":["def-RCTS"],"member_ids":["S07"],"declared_external":false,"title":"正则化张量步 T_H","source_units":[{"raw_tex":"Let us define now one step of the {\\em Regularized\r\n\tComposite Tensor Method} (RCTM) of degree $p \\geq 2$:\r\n\\beq\\label{def-RCTS}\r\n\\ba{rcl}\r\nT & \\equiv & T_H(x) \\; \\Def \\; \\arg\\min\\limits_{y \\in \\E}\r\n\\left\\{ \\Omega_p(f,x;y) + {H \\over (p+1)!} \\| y - x\r\n\\|^{p+1} + h(y) \\right\\}.\r\n\\ea\r\n\\eeq","start":14183,"end":14471}]},{"id":"S08","semantic_type":"assumption","kind":"assumption","labels":["eq-H"],"member_ids":["S08","S09"],"declared_external":false,"title":"正则化条件 H≥pL_p","source_units":[{"raw_tex":"\r\nIt can be shown that for\r\n\\beq\\label{eq-H}\r\n\\ba{c}\r\n\\mbox{\\fbox{\\rule[-2mm]{0cm}{6mm}$H \\; \\geq \\; p L_p$}}\r\n\\ea\r\n\\eeq\r\nthe auxiliary optimization problem in (\\ref{def-RCTS}) is\r\n{\\em convex} (see Theorem 1 in \\cite{nesterov2019implementable}).\r\nThis condition\r\nis crucial for implementability of our methods and we\r\nalways assume it to be satisfied.\r\n\r\n","start":14471,"end":14827},{"raw_tex":"\r\nthe auxiliary optimization problem in (\\ref{def-RCTS}) is\r\n{\\em convex} (see Theorem 1 in \\cite{nesterov2019implementable}).\r\nThis condition\r\nis crucial for implementability of our methods and we\r\nalways assume it to be satisfied.\r\n\r\n","start":14591,"end":14827}]},{"id":"S12","semantic_type":"definition","kind":"definition","labels":["def-FP"],"member_ids":["S12","S10","S11"],"declared_external":false,"title":"最优性与所选次梯度的定义","source_units":[{"raw_tex":"Let us write down the first-order optimality condition for\r\nthe auxiliary optimization problem in (\\ref{def-RCTS}):\r\n\\beq\\label{eq-OptC}\r\n\\ba{rcl}\r\n\\la \\nabla \\Omega_p(f,x;T) + {H \\over p!} \\| T - x\r\n\\|^{p-1}B(T-x), y - T \\ra + h(y) & \\geq & h(T),\r\n\\ea\r\n\\eeq\r\nfor all $y \\in \\dom h$.\r\n","start":14827,"end":15112},{"raw_tex":"In other words, for vector\r\n\\beq\\label{def-HP}\r\n\\ba{rcl}\r\nh'(T) & \\Def & - \\left( \\nabla \\Omega_p(f,x;T) + {H \\over\r\n\tp!} \\| T - x \\|^{p-1}B(T-x) \\right)\r\n\\ea\r\n\\eeq\r\nwe have $h'(T) \\stackrel{(\\ref{eq-OptC})}{\\in} \\partial\r\nh(T)$. ","start":15112,"end":15342},{"raw_tex":"This fact explains our notation\r\n\\beq\\label{def-FP}\r\n\\ba{rcl}\r\nF'(T) & \\Def & \\nabla f(T) + h'(T) \\; \\in \\partial F(T).\r\n\\ea\r\n\\eeq","start":15342,"end":15472}]},{"id":"S14","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq-NewG"],"member_ids":["S14","S06","S13"],"declared_external":false,"title":"Taylor 误差及次梯度范数界","source_units":[{"raw_tex":"\r\n\r\nAssuming that $L_{p} < +\\infty$, by the standard\r\nintegration arguments we can bound the residual between\r\nfunction value and its Taylor approximation:\r\n\\beq\\label{eq-BoundF}\r\n\\ba{rcl}\r\n| f(y) - \\Omega_p(f,x;y) | & \\leq & {L_{p} \\over (p+1)!}\r\n\\| y - x \\|^{p+1}, \\quad x, y \\in \\dom h.\r\n\\ea\r\n\\eeq\r\nApplying the same reasoning to functions $\\la \\nabla\r\nf(\\cdot), h \\ra$ and $\\la \\nabla^2 f(\\cdot) h, h \\ra$ with\r\ndirection $h \\in \\E$ being fixed, we get the following\r\nguarantees:\r\n\\beq\\label{eq-BoundG}\r\n\\ba{rcl}\r\n\\| \\nabla f(y) - \\nabla \\Omega_p(f,x;y) \\|_* & \\leq & {L_p\r\n\t\\over p!} \\| y - x \\|^{p},\r\n\\ea\r\n\\eeq\r\n\\beq\\label{eq-BoundH}\r\n\\ba{rcl}\r\n\\| \\nabla^2 f(y) - \\nabla^2 \\Omega_p(f,x;y) \\| & \\leq &\r\n{L_p \\over (p-1)!} \\| y - x \\|^{p-1},\r\n\\ea\r\n\\eeq\r\nwhich are valid for all $x, y \\in \\dom h$.\r\n","start":13379,"end":14181},{"raw_tex":"\r\n\r\nLet us present some properties of the point $T = T_H(x)$.\r\nFirst of all, we need some bounds for the norm of vector\r\n$F'(T)$. Note that\r\n\\beq\\label{eq-N3}\r\n\\ba{rcl}\r\n\\Big\\| F'(T) + {H \\over p!} \\| T - x \\|^{p-1}B(T-x)\r\n\\Big\\|_* & \\refEQ{def-HP} & \\Big\\| \\nabla f(T) - \\nabla\r\n\\Omega_p(f,x;T) \\Big\\|_*\\\\\r\n\\\\\r\n& \\refLE{eq-BoundG} & {L_p \\over p!} \\| T - x \\|^p.\r\n\\ea\r\n\\eeq","start":15472,"end":15846},{"raw_tex":"\r\nConsequently,\r\n\\beq\\label{eq-NewG}\r\n\\ba{rcl}\r\n\\| F'(T) \\|_* & \\leq & {L_p+H \\over p!} \\| T - x \\|^p.\r\n\\ea\r\n\\eeq\r\n","start":15846,"end":15961}]},{"id":"S15","semantic_type":"algorithm","kind":"algorithm","labels":["met-RCTM"],"member_ids":["S15"],"declared_external":false,"title":"RCTM 迭代规则","source_units":[{"raw_tex":"The main goal of this paper consists in analyzing the\r\nlocal behavior of the {\\em Regularized Composite Tensor\r\n\tMethod} (RCTM):\r\n\\beq\\label{met-RCTM}\r\n\\ba{rcl}\r\nx_0 \\; \\in \\; \\dom h, \\quad x_{k+1} & = & T_H(x_k), \\quad\r\nk \\geq 0,\r\n\\ea\r\n\\eeq","start":18444,"end":18685}]},{"id":"S16","semantic_type":"assumption","kind":"assumption","labels":["assumption-Uni","eq-UniC"],"member_ids":["S16"],"declared_external":false,"title":"目标 F 的 q 次一致凸性","source_units":[{"raw_tex":"\r\nas applied to the problem (\\ref{prob-Main}). In order to\r\nprove local superlinear convergence of this scheme, we\r\nneed one more assumption.\r\n\r\n\\begin{assumption}\\label{assumption-Uni}\r\n\tThe objective in problem\r\n\t(\\ref{prob-Main}) is uniformly convex of degree $q \\geq 2$.\r\n\tThus, for all $x, y \\in \\dom h$ and for all $G_x \\in \\partial F(x), G_y \\in \\partial F(y)$,\r\n\tit holds:\r\n\t\\beq\\label{eq-UniC}\r\n\t\\ba{rcl}\r\n\t\\la G_x - G_y, x - y \\ra & \\geq & \\sigma_q\r\n\t\\| x - y \\|^q, \r\n\t\\ea\r\n\t\\eeq\r\n\tfor certain $\\sigma_q > 0$.\r\n\\end{assumption}\r\n","start":18685,"end":19224}]},{"id":"S17","semantic_type":"definition","kind":"definition","labels":["def-GNorm"],"member_ids":["S17"],"declared_external":false,"title":"最小次梯度范数 η","source_units":[{"raw_tex":"Let us study now the local convergence of the method\r\n(\\ref{met-RCTM}) in terms of the norm of gradient. For any\r\n$x \\in \\dom h$ denote\r\n\\beq\\label{def-GNorm}\r\n\\ba{rcl}\r\n\\eta(x) & \\Def & \\min\\limits_{g \\in \\partial h(x)} \\| \\nabla\r\nf(x) + g \\|_*.\r\n\\ea\r\n\\eeq\r\nIf $\\partial h(x) = \\emptyset$, we set $\\eta(x) =\r\n+\\infty$.\r\n","start":21357,"end":21678}]},{"id":"S19","semantic_type":"lemma","kind":"lemma","labels":["lm-DecF","eq-DecFB","eq-DecF"],"member_ids":["S19","S20","S21","S22","S23","S24","S25","S26"],"declared_external":false,"title":"完整下降引理及作者证明","source_units":[{"raw_tex":"\\BL\\label{lm-DecF}\r\nLet $\\beta > 1$ and $H = \\beta L_p$. Then","start":16003,"end":16064},{"raw_tex":"\\BL\\label{lm-DecF}\r\nLet $\\beta > 1$ and $H = \\beta L_p$. Then\r\n\\beq\\label{eq-DecFB}\r\n\\ba{rcl}\r\n\\la F'(T), x - T \\ra & \\geq & \\left( {p! \\over (p+1)L_p}\r\n\\right)^{1 \\over p} \\cdot \\| F'(T) \\|_*^{p+1 \\over p}\r\n\\cdot {(\\beta^2 - 1)^{p-1 \\over 2p} \\over \\beta} \\cdot {p\r\n\t\\over (p^2-1)^{p-1 \\over 2p}}.\r\n\\ea\r\n\\eeq\r\nIn particular, if $\\beta = p$, then\r\n\\beq\\label{eq-DecF}\r\n\\ba{rcl}\r\n\\la F'(T), x - T \\ra & \\geq & \\left( {p! \\over (p+1)L_p}\r\n\\right)^{1 \\over p} \\cdot \\| F'(T) \\|_*^{p+1 \\over p}.\r\n\\ea\r\n\\eeq\r\n\\EL","start":16003,"end":16510},{"raw_tex":"\\proof\r\nDenote $r = \\| T  - x \\|$, $h = {H \\over p!}$, and $l =\r\n{L_p \\over p!}$. Then inequality (\\ref{eq-N3}) can be\r\nwritten as follows:\r\n$$\r\n\\ba{rcl}\r\n\\| F'(T) + h r^{p-1} B(T-x) \\|^2_* & \\leq & l^2 r^{2p}.\r\n\\ea\r\n$$\r\n","start":16512,"end":16733},{"raw_tex":"\\proof\r\nDenote $r = \\| T  - x \\|$, $h = {H \\over p!}$, and $l =\r\n{L_p \\over p!}$. Then inequality (\\ref{eq-N3}) can be\r\nwritten as follows:\r\n$$\r\n\\ba{rcl}\r\n\\| F'(T) + h r^{p-1} B(T-x) \\|^2_* & \\leq & l^2 r^{2p}.\r\n\\ea\r\n$$\r\nThis means that\r\n\\beq\\label{eq-N4}\r\n\\ba{rcl}\r\n\\la F'(T), x - T \\ra & \\geq & {1 \\over 2 h r^{p-1}} \\|\r\nF'(T) \\|_*^2 + {r^{2p} (h^2 - l^2) \\over 2h r^{p-1}}.\r\n\\ea\r\n\\eeq\r\nDenote\r\n$$\r\n\\ba{rcl}\r\na & = & {1 \\over 2h} \\| F'(T) \\|_*^2, \\quad b \\; = \\; {h^2\r\n\t- l^2 \\over 2h}, \\quad \\tau \\; = \\; r^{p-1}, \\quad \\alpha\r\n\\; = \\; {p+1 \\over p-1}.\r\n\\ea\r\n$$\r\nThen inequality (\\ref{eq-N4}) can be rewritten as follows:\r\n$$\r\n\\ba{rcl}\r\n\\la F'(T) , x - T \\ra & \\geq & {a \\over \\tau} + b\r\n\\tau^{\\alpha} \\; \\geq \\; \\min\\limits_{t > 0} \\left\\{ {a\r\n\t\\over t} + b t^{\\alpha} \\right\\} \\; = \\; (1+\\alpha)\r\n\\left({a \\over \\alpha} \\right)^{\\alpha \\over 1 + \\alpha}\r\nb^{1 \\over 1 + \\alpha}.\r\n\\ea\r\n$$\r\nTaking into account that $1+\\alpha = {2p \\over p-1}$ and\r\n${\\alpha \\over 1 + \\alpha} = {p + 1 \\over 2p}$, and using the\r\nactual meaning of $a$, $b$, and $\\alpha$, we get\r\n$$\r\n\\ba{rcl}\r\n\\la F'(T), x - T \\ra & \\geq & {2 p \\over p-1} \\cdot { \\|\r\n\tF'(T) \\|_*^{p+1 \\over p} \\over (2h)^{p+1 \\over 2p}} \\cdot\r\n{(p-1)^{p+1 \\over 2p} \\over (p+1)^{p+1 \\over 2p}} \\cdot\r\n{(h^2 - l^2)^{p-1 \\over 2p} \\over (2h)^{p-1 \\over 2p}}\\\\\r\n\\\\\r\n& = & \\| F'(T) \\|_*^{p+1 \\over p} \\cdot {(h^2 - l^2)^{p-1\r\n\t\t\\over 2p} \\over h} \\cdot {p \\over (p+1)^{p+1 \\over 2p}\r\n\t(p-1)^{p-1 \\over 2p}}\\\\\r\n\\\\\r\n& = & \\| F'(T) \\|_*^{p+1 \\over p} \\cdot {(h^2 - l^2)^{p-1\r\n\t\t\\over 2p} \\over h} \\cdot {p \\over (p^2-1)^{p-1 \\over 2p}\r\n\t(p+1)^{1 \\over p}}.\r\n\\ea\r\n$$\r\nIt remains to note that\r\n$$\r\n\\ba{rcl}\r\n{(h^2 - l^2)^{p-1 \\over 2p} \\over h} & = & {(H^2 -\r\n\tL_p^2)^{p-1 \\over 2p} \\over H} \\cdot (p!)^{1 \\over p} \\; =\r\n\\; {(\\beta^2 - 1)^{p-1 \\over 2p} \\over \\beta} \\cdot\r\n\\left({p! \\over L_p} \\right)^{1 \\over p}.\r\n\\ea\r\n$$\r\n\\qed","start":16512,"end":18386},{"raw_tex":"This means that\r\n\\beq\\label{eq-N4}\r\n\\ba{rcl}\r\n\\la F'(T), x - T \\ra & \\geq & {1 \\over 2 h r^{p-1}} \\|\r\nF'(T) \\|_*^2 + {r^{2p} (h^2 - l^2) \\over 2h r^{p-1}}.\r\n\\ea\r\n\\eeq\r\n","start":16733,"end":16901},{"raw_tex":"Denote\r\n$$\r\n\\ba{rcl}\r\na & = & {1 \\over 2h} \\| F'(T) \\|_*^2, \\quad b \\; = \\; {h^2\r\n\t- l^2 \\over 2h}, \\quad \\tau \\; = \\; r^{p-1}, \\quad \\alpha\r\n\\; = \\; {p+1 \\over p-1}.\r\n\\ea\r\n$$\r\n","start":16901,"end":17078},{"raw_tex":"Then inequality (\\ref{eq-N4}) can be rewritten as follows:\r\n$$\r\n\\ba{rcl}\r\n\\la F'(T) , x - T \\ra & \\geq & {a \\over \\tau} + b\r\n\\tau^{\\alpha} \\; \\geq \\; \\min\\limits_{t > 0} \\left\\{ {a\r\n\t\\over t} + b t^{\\alpha} \\right\\} \\; = \\; (1+\\alpha)\r\n\\left({a \\over \\alpha} \\right)^{\\alpha \\over 1 + \\alpha}\r\nb^{1 \\over 1 + \\alpha}.\r\n\\ea\r\n$$\r\n","start":17078,"end":17406},{"raw_tex":"Taking into account that $1+\\alpha = {2p \\over p-1}$ and\r\n${\\alpha \\over 1 + \\alpha} = {p + 1 \\over 2p}$, and using the\r\nactual meaning of $a$, $b$, and $\\alpha$, we get\r\n$$\r\n\\ba{rcl}\r\n\\la F'(T), x - T \\ra & \\geq & {2 p \\over p-1} \\cdot { \\|\r\n\tF'(T) \\|_*^{p+1 \\over p} \\over (2h)^{p+1 \\over 2p}} \\cdot\r\n{(p-1)^{p+1 \\over 2p} \\over (p+1)^{p+1 \\over 2p}} \\cdot\r\n{(h^2 - l^2)^{p-1 \\over 2p} \\over (2h)^{p-1 \\over 2p}}\\\\\r\n\\\\\r\n& = & \\| F'(T) \\|_*^{p+1 \\over p} \\cdot {(h^2 - l^2)^{p-1\r\n\t\t\\over 2p} \\over h} \\cdot {p \\over (p+1)^{p+1 \\over 2p}\r\n\t(p-1)^{p-1 \\over 2p}}\\\\\r\n\\\\\r\n& = & \\| F'(T) \\|_*^{p+1 \\over p} \\cdot {(h^2 - l^2)^{p-1\r\n\t\t\\over 2p} \\over h} \\cdot {p \\over (p^2-1)^{p-1 \\over 2p}\r\n\t(p+1)^{1 \\over p}}.\r\n\\ea\r\n$$\r\nIt remains to note that\r\n$$\r\n\\ba{rcl}\r\n{(h^2 - l^2)^{p-1 \\over 2p} \\over h} & = & {(H^2 -\r\n\tL_p^2)^{p-1 \\over 2p} \\over H} \\cdot (p!)^{1 \\over p} \\; =\r\n\\; {(\\beta^2 - 1)^{p-1 \\over 2p} \\over \\beta} \\cdot\r\n\\left({p! \\over L_p} \\right)^{1 \\over p}.\r\n\\ea\r\n$$\r\n\\qed","start":17406,"end":18386}]},{"id":"S29","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S29"],"declared_external":false,"title":"主证明实际使用的下降子事实","source_units":[{"raw_tex":"$$\r\n\\ba{rcl}\r\n\\la \\nabla f(x_k) + g_k, x_{k} - x_{k + 1} \\ra\r\n& \\geq &\r\n\\la F'(x_{k + 1}), x_k - x_{k + 1} \\ra + \\sigma_q \\|x_k - x_{k + 1}\\|^q \\\\\r\n\\\\\r\n& \\refGE{eq-DecFB} & \\sigma_q \\|x_k - x_{k + 1}\\|^{q},\r\n\\ea\r\n$$","start":21992,"end":22207}]},{"id":"S28","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S28"],"declared_external":false,"title":"一致凸性与下降性给出内积下界","source_units":[{"raw_tex":"\\proof\r\nIndeed, in view of inequality \\eqref{eq-UniC}, we have\r\n$$\r\n\\ba{rcl}\r\n\\la \\nabla f(x_k) + g_k, x_{k} - x_{k + 1} \\ra\r\n& \\geq &\r\n\\la F'(x_{k + 1}), x_k - x_{k + 1} \\ra + \\sigma_q \\|x_k - x_{k + 1}\\|^q \\\\\r\n\\\\\r\n& \\refGE{eq-DecFB} & \\sigma_q \\|x_k - x_{k + 1}\\|^{q},\r\n\\ea\r\n$$","start":21928,"end":22207}]},{"id":"S30","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S30"],"declared_external":false,"title":"用 η 控制迭代步长","source_units":[{"raw_tex":"\r\nwhere $g_k$ is an arbitrary vector from $\\partial h(x_k)$.\r\nTherefore, we conclude that\r\n$$\r\n\\ba{rcl}\r\n\\eta(x_k)  & \\geq &  \\sigma_q \\| x_k -\r\nx_{k+1} \\|^{q-1}.\r\n\\ea\r\n$$\r\n","start":22207,"end":22380}]},{"id":"S31","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S31"],"declared_external":false,"title":"合并次梯度范数界得到递推","source_units":[{"raw_tex":"It remains to use inequality\r\n(\\ref{eq-NewG}).\r\n\\qed","start":22380,"end":22432}]},{"id":"S27","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S27"],"declared_external":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\proof\r\nIndeed, in view of inequality \\eqref{eq-UniC}, we have\r\n$$\r\n\\ba{rcl}\r\n\\la \\nabla f(x_k) + g_k, x_{k} - x_{k + 1} \\ra\r\n& \\geq &\r\n\\la F'(x_{k + 1}), x_k - x_{k + 1} \\ra + \\sigma_q \\|x_k - x_{k + 1}\\|^q \\\\\r\n\\\\\r\n& \\refGE{eq-DecFB} & \\sigma_q \\|x_k - x_{k + 1}\\|^{q},\r\n\\ea\r\n$$\r\nwhere $g_k$ is an arbitrary vector from $\\partial h(x_k)$.\r\nTherefore, we conclude that\r\n$$\r\n\\ba{rcl}\r\n\\eta(x_k)  & \\geq &  \\sigma_q \\| x_k -\r\nx_{k+1} \\|^{q-1}.\r\n\\ea\r\n$$\r\nIt remains to use inequality\r\n(\\ref{eq-NewG}).\r\n\\qed","start":21928,"end":22432}]},{"id":"S18","semantic_type":"theorem","kind":"theorem","labels":["th-RateG","eq-RateG"],"member_ids":["S18"],"declared_external":false,"title":"定理 2：最小次梯度范数递推","source_units":[{"raw_tex":"\\BT\\label{th-RateG}\r\nFor any $k \\geq 0$ we have\r\n\\beq\\label{eq-RateG}\r\n\\ba{rcl}\r\n\\eta(x_{k+1})\r\n& \\, \\leq \\, &\r\n\\|F'(x_{k + 1}) \\|_{*}\r\n\\; \\leq \\;\r\n{L_p + H \\over p!} \\left[ {1 \\over\r\n\t\\sigma_q} \\, \\eta(x_k) \\right]^{p \\over q-1}.\r\n\\ea\r\n\\eeq\r\n\\ET","start":21680,"end":21926}]}],"collapse_map":{"S01":"S01","S03":"S03","S04":"S04","S05":"S05","S07":"S07","S08":"S08","S12":"S12","S14":"S14","S15":"S15","S16":"S16","S17":"S17","S19":"S19","S29":"S29","S28":"S28","S30":"S30","S31":"S31","S27":"S27","S18":"S18","S02":"S03","S06":"S14","S09":"S08","S10":"S12","S11":"S12","S13":"S14","S20":"S19","S21":"S19","S22":"S19","S23":"S19","S24":"S19","S25":"S19","S26":"S19","S33":"S01","S34":"S03","S35":"S03"}}}}
{"record_id":"1912.07481v4","paper":{"title":"On Lower Iteration Complexity Bounds for the Saddle Point Problems","authors":["Zhang, Junyu","Hong, Mingyi","Zhang, Shuzhong"],"arxiv_id":"1912.07481","version":"v4","journal":"Mathematical Programming","abstract_url":"https://arxiv.org/abs/1912.07481v4","pdf_url":"https://arxiv.org/pdf/1912.07481v4","doi":"10.1007/s10107-021-01660-z"},"context":{"raw_tex":"\\begin{definition}[Bilinear  class $\\cB(L_{xy},\\mu_x,\\mu_y)$]\r\n\t\\label{defn:bilinear-class}\r\n\tIn this special class, the problems are written as\r\n\t\\begin{equation}\r\n\t\\label{prob:bilinear}\r\n\t\\min_x \\max_y  F(x,y):=f(x) + x^\\top Ay - g(y),\r\n\t\\end{equation}\r\n%\tWe require\r\nwhere $f(x)$ and $g(y)$ are both lower semi-continuous with $f(x)$ being $\\mu_x$-strongly convex and $g(y)$ being $\\mu_y$-strongly convex. The coupling matrix $A$ satisfies $\\|A\\|_2\\leq L_{xy}$.\r\n\\end{definition}\r\nFor this special model class $\\cB(L_{xy},\\mu_x,\\mu_y)$, %subclass of problems,\r\nwe assume the availability of the following prox-operations:\r\n\\begin{equation}\r\n\\label{defn:prox-fg}\r\n\\prox_{\\gamma f}(v):=\\argmin_x f(x) + \\frac{1}{2\\gamma}\\|x-v\\|^2 \\quad \\mbox{ and } \\quad\\prox_{\\sigma g}(u):=\\argmin_y g(y) + \\frac{1}{2\\sigma}\\|y-u\\|^2.\r\n\\end{equation}\r\n\n\n\\subsection{Primal function, dual function, and the duality gap}\r\nFirst, we define $\\Phi(\\cdot)$ to be the primal function and $\\Psi(\\cdot)$ to be the dual function of the saddle point problem $\\min_x \\max_y F(x,y)$, respectively, with the following definitions\r\n\\begin{equation}\r\n\\label{defn:primal-func}\r\n\\Phi(x):=\\max_y F(x,y)\\qquad\\mbox{ and }\\qquad\\Psi(y):=\\min_x F(x,y).\r\n\\end{equation}\r\nAs the maximum of a class of $\\mu_x$-strongly convex function, we know $\\Phi(x)$ is a $\\mu_x$-strongly convex function. Similarly, $\\Psi(y)$ is a $\\mu_y$-strongly concave function. We define the duality gap as\r\n$$\\Delta(x,y) := \\max_{y'} F(x,y')-\\min_{x'} F(x',y) = \\Phi(x)-\\Psi(y).$$\r\nSuppose the unique solution of this min-max problem is $(x^*,y^*)$. By the strong duality theorem, we know for any $x$ and $y$ it holds that \r\n$$\\Phi(x)\\geq\\min_{x'}\\Phi(x') = \\Phi(x^*) =F(x^*,y^*) = \\Psi(y^*) = \\max_{y'}\\Psi(y')\\geq \\Psi(y).$$\r\nTogether with the $\\mu_x$-strong convexity of $\\Phi$ and the $\\mu_y$-strong concavity of $\\Psi$, we further have \r\n\\begin{equation}\r\n\\label{defn:primal-gap-LB}\r\n\\Delta(x,y)= \\Phi(x)-\\Phi(x^*) + \\Psi(y^*) - \\Psi(y)\\geq\\frac{\\mu_x}{2}\\|x-x^*\\|^2 + \\frac{\\mu_y}{2}\\|y-y^*\\|^2.\r\n\\end{equation}\r\n\r\nNow, suppose that $(\\tilde x_k,\\tilde y_k)$ is the approximate solution generated after $k$ iterations of an algorithm. Our aim %main strategy in the later discussion \r\nis to lower bound the distance between $(\\tilde x_k,\\tilde y_k)$ and $(x^*,y^*)$. By \\eqref{defn:primal-gap-LB}, this would construct a lower iteration complexity bound in terms of the duality gap as well.\r\n\n\n\\begin{definition}[Proximal algorithm class]\r\n\\label{defn:Prox-AlgoClass}\r\nIn each iteration, the iterate sequence $\\{(x^{k},y^{k})\\}_{k=0,1,...}$ are generated so that  $(x^{k},y^{k})\\in\\cH_x^{k} \\times \\cH_y^{k}$. These subspaces are generated with $\\cH_x^{0} = \\mathrm{Span}\\{x^0\\}, \\cH_y^{0} = \\mathrm{Span}\\{y^0\\}$ and\r\n\\begin{equation}\r\n\\label{eqn:Prox-AlgoClass}\\begin{cases}\r\n\\cH_x^{k+1} := \\mathrm{Span}\\{x^i,\\prox_{\\gamma_i f}(\\hat{x}^i-\\gamma_iA\\tilde{y}^i)~\\,: \\forall \\hat{x}^i\\in\\cH_x^{i},~ \\tilde{y}^i\\in\\cH_y^i,~ 0\\leq i\\leq k\\}\\\\\r\n\\cH_y^{k+1} := \\mathrm{Span}\\{y^i,\\prox_{\\sigma_i g}(\\hat{y}^i+\\sigma_iA^\\top\\tilde{x}^i): \\forall \\tilde{x}^i\\in\\cH_x^{i},~ \\hat{y}^i\\in\\cH_y^i,~ 0\\leq i\\leq k\\}.\r\n\\end{cases}\r\n\\end{equation}\r\n\\end{definition}\r\nRemark that when applying the proximal oracles, it is not necessary to use the most recent iterate $x^k$ as the proximal center. Neither is it necessary to use the gradients of the coupling term (namely the $A^\\top x$ and $Ay$ terms) at the current iterate. Instead, the algorithm class allows the usage of the combination of {\\it any}\\/ points in the historical search space. We shall also remark that the algorithm class in Definition~\\ref{defn:Prox-AlgoClass} does not necessarily need to update $x$ and $y$ at the same time, because setting $x^{k+1}=x^k$ or $y^{k+1} = y^k$ also satisfies Definition \\ref{defn:Prox-AlgoClass}. Thus this algorithm class also includes the methods that alternatingly update $x$ and $y$. Below is a sample algorithm in this class.\r\n\n\n\\subsection{General deterministic algorithm classes without linear span structure}\r\nAlthough all the reviewed first-order methods satisfy the \\emph{linear span} property in the proximal algorithm class in Definition \\ref{defn:Prox-AlgoClass} and the pure first-order algorithm class in Definition \\ref{defn:Pure-AlgoClass}, this does not exclude the possibility of the deriving an algorithm that does not satisfy the linear span property. Therefore, we also define the general deterministic proximal algorithm class and the general deterministic pure first-order algorithm class as follows, whose iteration complexity lower bound can be generalized from their linear span counterpart through the technique of \\emph{adversary rotation}. \r\n\\begin{definition}[General proximal algorithm class]\r\n\t\\label{defn:Det-Prox-AlgoClass}\r\n\tConsider the problem \\eqref{prob:bilinear} in the problem class $\\cB(L_{xy},\\mu_x,\\mu_y)$, denote $\\theta = (L_{xy},\\mu_x,\\mu_y)$ as the corresponding problem parameters. Let algorithm $\\cA$ belong to the general  proximal algorithm class. Then $\\cA$ consists of a sequence of deterministic mappings $\\{(\\cA_x^1, \\cA_y^1,\\cA_u^1,\\cA_v^1),(\\cA_x^2, \\cA_y^2,\\cA_u^2,\\cA_v^2)...\\}$ such that the iterate sequence $\\{(x^{k},y^{k})\\}_{k=0,1,...}$ and the output sequence $\\{(\\tilde x^{k},\\tilde y^{k})\\}_{k=0,1,...}$ are generated by \r\n\t\\begin{equation}\r\n\t\\label{eqn:Det-Prox-AlgoClass}\\begin{cases}\r\n\t(x^k,\\tilde x^k) := \\cA_x^k\\left(\\theta; x^0,Ay^0,...,x^{k-1},Ay^{k-1}; \\mathbf{prox}_{\\gamma_k f}(u^k) \\right),\\\\\r\n\t(y^k,\\tilde y^k) := \\cA_y^k\\left(\\theta; y^0,A^\\top x^0,...,y^{k-1},A^\\top x^{k-1}; \\mathbf{prox}_{\\sigma_k g}(v^k) \\right),\r\n\t\\end{cases}\r\n\t\\end{equation} \r\n\twhere $u^k = \\cA^k_u(\\theta; x^0,Ay^0,...,x^{k-1},Ay^{k-1})$, $v^k = \\cA^k_v(\\theta; y^0,A^\\top x^0,...,y^{k-1},A^\\top x^{k-1})$, and \r\n\t$(x^0,y^0)$ is any given initial solution. \r\n\\end{definition}\r\nOne remark is that the input of the proximal mapping $\\mathbf{prox}_{\\gamma_kf}(\\cdot)$ is  constructed with other inputs to the $\\cA_x^k$, i.e., there could be another deterministic mapping $\\cA_u^k$ to generate a vector  $u^k = \\cA^k_u(\\theta; x^0,Ay^0,...,x^{k-1},Ay^{k-1})$ and then $\\mathbf{prox}_{\\gamma_kf}(u^k)$ is passed to the mapping $\\cA_x^k$. This $\\cA^k_u$ does not need to be linear. The situation for $v^k$ and $\\cA^k_v$ is similar. Similar to the general proximal algorithm class, the general pure first-order algorithm class is defined as follows. \r\n\n\nLet us construct the following bilinearly coupled min-max saddle point problem:\r\n\\begin{equation}\r\n\\label{prob:worst-proximal}\r\n\\min_x\\max_y F(x,y):=\\frac{\\mu_x}{2}\\|x\\|^2 + \\frac{L_{xy}}{2}x^\\top Ay - \\frac{\\mu_y}{2}\\|y\\|^2 - b^\\top y\r\n\\end{equation}\r\nwhere $b$ is a vector to be determined later, and the coupling matrix $A$ (hence $A^2$ and $A^4$) is defined as follows: %. Its polynomials $A^2$ and $A^4$ are also shown bellow for later reference.\r\n\\begin{equation}\r\n\\label{defn:A}\r\nA = \\begin{pmatrix}\r\n& &    &     & 1\\\\\r\n& &    &  1 & -1\\\\\r\n& & 1 & -1&\\\\\r\n& \\udots &\\udots & & \\\\\r\n1& -1 & & &\r\n\\end{pmatrix}, \\,\\,\r\nA^2 = \\begin{pmatrix}\r\n1   & -1 &            &               & \\\\\r\n-1 & 2   & -1        &              & \\\\\r\n& -1 &\\ddots & \\ddots & \\\\\r\n&      &\\ddots & 2           &-1\\\\\r\n&      &              & -1         & 2\r\n\\end{pmatrix},\r\n\\,\\,\r\nA^4 = \\begin{pmatrix}\r\n2   & -3 &     1      &               &  & &\\\\\r\n-3 & 6   & -4        &     1         & & & \\\\\r\n1& -4 &6 & -4 &1 \\\\\r\n&  \\ddots     &\\ddots & \\ddots  &\\ddots & \\ddots & \\\\\r\n& & 1& -4 &6 & -4 &1\\\\\r\n& & & 1& -4 &6 & -4 \\\\\r\n& & & & 1&-4&5\r\n\\end{pmatrix}.\r\n\\end{equation}\r\nNote that $A^\\top =A$ and $\\|A\\|_2 \\leq 2$. Therefore \\eqref{prob:worst-proximal} is an instance in the problem class $\\cB(L_{xy},\\mu_x,\\mu_y)$. It is worth noting that the example \\eqref{prob:worst-proximal} is the same as that in Proposition 2 of \\cite{LowB-Quadratic}, which is a parallel work focused on pure first-order algorithm class. Here, we use the same example to elaborate the lower bound of the proximal methods over the general bilinear coupling class $\\cB(L_{xy},\\mu_x,\\mu_y)$, as well as a warmup for the discussion of more complex problem class $\\cF(L_{x},L_{y},L_{xy},\\mu_x,\\mu_y)$. \r\n\r\nDenote $e_i$ to be the $i$-th unit vector, which has $1$ at the $i$-th component and 0 elsewhere. Then by direct calculation, one can check that $A^2$ satisfies the following \\emph{zero-chain property} (see Chapter 2 of  \\cite{nesterov2018lectures}).\r\n\r\n\n\nFor problem \\eqref{prob:worst-proximal}, the proximal operators in \\eqref{eqn:Prox-AlgoClass} can be written explicitly:\r\n\\begin{eqnarray}\r\n\\label{defn:prox-closedForm-x}\r\n\\prox_{\\gamma_i f}(\\hat{x}_i-\\gamma_iA\\tilde{y}_i) & = & \\argmin_x \\frac{\\mu_x}{2}\\|x\\|^2 + \\frac{1}{2\\gamma_i}\\left\\|x-(\\hat{x}_i-\\frac{\\gamma_iL_{xy}}{2}A\\tilde{y}_i)\\right\\|^2\\\\\r\n& = & \\frac{1}{1+\\gamma_i\\mu_x}\\hat{x}_i - \\frac{\\gamma_iL_{xy}}{2(1+\\gamma_i\\mu_x)}A\\tilde{y}_i\\nonumber\\\\\r\n& \\in & \\mathrm{Span}\\{\\hat{x}_i,A\\tilde{y}_i\\}.\\nonumber\r\n\\end{eqnarray}\r\nSimilarly, for the $y$ block, we also have\r\n\\begin{eqnarray}\r\n\\label{defn:prox-closedForm-y}\r\n\\prox_{\\sigma_i g}(\\hat{y}_i+\\sigma_iA^\\top\\tilde{x}_i)\r\n%& = & \\argmin_y \\frac{\\mu_y}{2}\\|y\\|^2 + b^Ty + \\frac{1}{2\\sigma_i}\\left\\|y-(\\hat{y}_i+\\frac{\\sigma_iL_{xy}}{2}A^\\top \\tilde{x}_i)\\right\\|^2\\\\\r\n=  \\frac{\\hat{y}_i-\\sigma_i b}{1+\\sigma_i\\mu_y} +  \\frac{\\sigma_iL_{xy}}{2(1+\\sigma_i\\mu_x)}A\\tilde{x}_i \\in \\mathrm{Span}\\{\\hat{y}_i,A\\tilde{x}_i,b\\}.\r\n\\end{eqnarray}\r\nLet us assume the initial point to be $x^0 = y^0 = 0$ ($\\cH_x^0 = \\cH_y^0 = \\{0\\}$) without loss of generality.\r\n\n\nLet us lower bound the dual gap, which is upper bounded by the whole duality gap. To achieve this, let us first write down the dual function of problem \\eqref{prob:worst-proximal} as\r\n\\begin{equation}\r\n\\label{defn:dual-function-prox}\r\n\\Psi(y) = \\min_{x} F(x,y) = -\\frac{1}{2}y^\\top \\left(\\frac{L_{xy}^2}{4\\mu_x}\\cdot A^2 + \\mu_y\\cdot I\\right)y - b^\\top y . \r\n\\end{equation}\r\nFor this $\\mu_y$-strongly concave dual function, we can characterize the optimal solution $y^*$ directly by its KKT condition $\\nabla \\Psi(y^*) = 0$. However, the exact solution $y^*$ does not have a simple and clear form, so we choose to  characterize it by an approximate solution $\\hat y^*$.\r\n\n\nConsider the bilinear problem class $\\cB(L_{xy},\\mu_x,\\mu_y)$ and the corresponding worst case problem \\eqref{prob:worst-proximal} with $F(x,y) = \\frac{\\mu_x}{2}\\|x\\|^2 + \\frac{L_{xy}}{2}x^\\top A y - \\frac{\\mu_y}{2}\\|y\\|^2 - b^\\top y$,\r\nwhere $A$ and $b$ are defined in accordance with Theorem \\ref{thm:proximal}. We define the orthogonally rotated problem as \r\n\\begin{equation}\r\n\\label{prob:rotate}\r\n\\min_x\\max_y F_{U,V}(x,y) : = F(Ux,Vy) = \\frac{\\mu_x}{2}\\|x\\|^2 + \\frac{L_{xy}}{2}x^\\top U^\\top AV y - \\frac{\\mu_y}{2}\\|y\\|^2 - b^\\top Vy,\r\n\\end{equation}\r\nwhere $U,V$ are two orthogonal matrices. Therefore, it is clear that $F_{U,V}\\in\\cB(L_{xy},\\mu_x,\\mu_y)$. Let $(x^*, y^*)$ be the saddle point of $F(x,y)$, then it is clear that the saddle point of $F_{U,V}(x,y)$ is $(\\bar x^*,\\bar y^*) = (U^\\top  x^*,V^\\top y^*).$ Consequently, the lower bound for the general proximal algorithm class is characterized by the following theorem. \r\n\n\n\\section{Proof of Theorem \\ref{theorem:gen-prox}}\r\n\\label{appdx:gen-prox} \r\n\tBefore proceeding the proof, let us first quote a lemma from \\cite{YYXu}. \r\n\t\\begin{lemma}[Lemma 3.1, \\cite{YYXu}] \r\n\t\t\\label{lemma:yyx} \r\n\t\tLet $\\cX\\subsetneqq\\bar{\\cX}\\subseteqq\\R^{p}$ be two linear subspaces. Then for any $\\bar x\\in\\R^p$, there exists an orthogonal matrix $\\Gamma\\in\\R^{p\\times p}$ s.t. $\\Gamma x = x, \\forall x\\in\\cX$ and $\\Gamma \\bar x\\in\\bar{\\cX}$.\r\n\t\\end{lemma}\r\n\tNote that for an orthogonal matrix $\\Gamma$, if $\\Gamma x = x$, then we also have $\\Gamma^\\top x = x$.  Now let us start our proof of Theorem \\ref{theorem:gen-prox}.\r\n\r\n","units":[{"raw_tex":"\\begin{definition}[Bilinear  class $\\cB(L_{xy},\\mu_x,\\mu_y)$]\r\n\t\\label{defn:bilinear-class}\r\n\tIn this special class, the problems are written as\r\n\t\\begin{equation}\r\n\t\\label{prob:bilinear}\r\n\t\\min_x \\max_y  F(x,y):=f(x) + x^\\top Ay - g(y),\r\n\t\\end{equation}\r\n%\tWe require\r\nwhere $f(x)$ and $g(y)$ are both lower semi-continuous with $f(x)$ being $\\mu_x$-strongly convex and $g(y)$ being $\\mu_y$-strongly convex. The coupling matrix $A$ satisfies $\\|A\\|_2\\leq L_{xy}$.\r\n\\end{definition}\r\nFor this special model class $\\cB(L_{xy},\\mu_x,\\mu_y)$, %subclass of problems,\r\nwe assume the availability of the following prox-operations:\r\n\\begin{equation}\r\n\\label{defn:prox-fg}\r\n\\prox_{\\gamma f}(v):=\\argmin_x f(x) + \\frac{1}{2\\gamma}\\|x-v\\|^2 \\quad \\mbox{ and } \\quad\\prox_{\\sigma g}(u):=\\argmin_y g(y) + \\frac{1}{2\\sigma}\\|y-u\\|^2.\r\n\\end{equation}\r\n","start":9313,"end":10151},{"raw_tex":"\\subsection{Primal function, dual function, and the duality gap}\r\nFirst, we define $\\Phi(\\cdot)$ to be the primal function and $\\Psi(\\cdot)$ to be the dual function of the saddle point problem $\\min_x \\max_y F(x,y)$, respectively, with the following definitions\r\n\\begin{equation}\r\n\\label{defn:primal-func}\r\n\\Phi(x):=\\max_y F(x,y)\\qquad\\mbox{ and }\\qquad\\Psi(y):=\\min_x F(x,y).\r\n\\end{equation}\r\nAs the maximum of a class of $\\mu_x$-strongly convex function, we know $\\Phi(x)$ is a $\\mu_x$-strongly convex function. Similarly, $\\Psi(y)$ is a $\\mu_y$-strongly concave function. We define the duality gap as\r\n$$\\Delta(x,y) := \\max_{y'} F(x,y')-\\min_{x'} F(x',y) = \\Phi(x)-\\Psi(y).$$\r\nSuppose the unique solution of this min-max problem is $(x^*,y^*)$. By the strong duality theorem, we know for any $x$ and $y$ it holds that \r\n$$\\Phi(x)\\geq\\min_{x'}\\Phi(x') = \\Phi(x^*) =F(x^*,y^*) = \\Psi(y^*) = \\max_{y'}\\Psi(y')\\geq \\Psi(y).$$\r\nTogether with the $\\mu_x$-strong convexity of $\\Phi$ and the $\\mu_y$-strong concavity of $\\Psi$, we further have \r\n\\begin{equation}\r\n\\label{defn:primal-gap-LB}\r\n\\Delta(x,y)= \\Phi(x)-\\Phi(x^*) + \\Psi(y^*) - \\Psi(y)\\geq\\frac{\\mu_x}{2}\\|x-x^*\\|^2 + \\frac{\\mu_y}{2}\\|y-y^*\\|^2.\r\n\\end{equation}\r\n\r\nNow, suppose that $(\\tilde x_k,\\tilde y_k)$ is the approximate solution generated after $k$ iterations of an algorithm. Our aim %main strategy in the later discussion \r\nis to lower bound the distance between $(\\tilde x_k,\\tilde y_k)$ and $(x^*,y^*)$. By \\eqref{defn:primal-gap-LB}, this would construct a lower iteration complexity bound in terms of the duality gap as well.\r\n","start":22208,"end":23803},{"raw_tex":"\\begin{definition}[Proximal algorithm class]\r\n\\label{defn:Prox-AlgoClass}\r\nIn each iteration, the iterate sequence $\\{(x^{k},y^{k})\\}_{k=0,1,...}$ are generated so that  $(x^{k},y^{k})\\in\\cH_x^{k} \\times \\cH_y^{k}$. These subspaces are generated with $\\cH_x^{0} = \\mathrm{Span}\\{x^0\\}, \\cH_y^{0} = \\mathrm{Span}\\{y^0\\}$ and\r\n\\begin{equation}\r\n\\label{eqn:Prox-AlgoClass}\\begin{cases}\r\n\\cH_x^{k+1} := \\mathrm{Span}\\{x^i,\\prox_{\\gamma_i f}(\\hat{x}^i-\\gamma_iA\\tilde{y}^i)~\\,: \\forall \\hat{x}^i\\in\\cH_x^{i},~ \\tilde{y}^i\\in\\cH_y^i,~ 0\\leq i\\leq k\\}\\\\\r\n\\cH_y^{k+1} := \\mathrm{Span}\\{y^i,\\prox_{\\sigma_i g}(\\hat{y}^i+\\sigma_iA^\\top\\tilde{x}^i): \\forall \\tilde{x}^i\\in\\cH_x^{i},~ \\hat{y}^i\\in\\cH_y^i,~ 0\\leq i\\leq k\\}.\r\n\\end{cases}\r\n\\end{equation}\r\n\\end{definition}\r\nRemark that when applying the proximal oracles, it is not necessary to use the most recent iterate $x^k$ as the proximal center. Neither is it necessary to use the gradients of the coupling term (namely the $A^\\top x$ and $Ay$ terms) at the current iterate. Instead, the algorithm class allows the usage of the combination of {\\it any}\\/ points in the historical search space. We shall also remark that the algorithm class in Definition~\\ref{defn:Prox-AlgoClass} does not necessarily need to update $x$ and $y$ at the same time, because setting $x^{k+1}=x^k$ or $y^{k+1} = y^k$ also satisfies Definition \\ref{defn:Prox-AlgoClass}. Thus this algorithm class also includes the methods that alternatingly update $x$ and $y$. Below is a sample algorithm in this class.\r\n","start":24321,"end":25847},{"raw_tex":"\\subsection{General deterministic algorithm classes without linear span structure}\r\nAlthough all the reviewed first-order methods satisfy the \\emph{linear span} property in the proximal algorithm class in Definition \\ref{defn:Prox-AlgoClass} and the pure first-order algorithm class in Definition \\ref{defn:Pure-AlgoClass}, this does not exclude the possibility of the deriving an algorithm that does not satisfy the linear span property. Therefore, we also define the general deterministic proximal algorithm class and the general deterministic pure first-order algorithm class as follows, whose iteration complexity lower bound can be generalized from their linear span counterpart through the technique of \\emph{adversary rotation}. \r\n\\begin{definition}[General proximal algorithm class]\r\n\t\\label{defn:Det-Prox-AlgoClass}\r\n\tConsider the problem \\eqref{prob:bilinear} in the problem class $\\cB(L_{xy},\\mu_x,\\mu_y)$, denote $\\theta = (L_{xy},\\mu_x,\\mu_y)$ as the corresponding problem parameters. Let algorithm $\\cA$ belong to the general  proximal algorithm class. Then $\\cA$ consists of a sequence of deterministic mappings $\\{(\\cA_x^1, \\cA_y^1,\\cA_u^1,\\cA_v^1),(\\cA_x^2, \\cA_y^2,\\cA_u^2,\\cA_v^2)...\\}$ such that the iterate sequence $\\{(x^{k},y^{k})\\}_{k=0,1,...}$ and the output sequence $\\{(\\tilde x^{k},\\tilde y^{k})\\}_{k=0,1,...}$ are generated by \r\n\t\\begin{equation}\r\n\t\\label{eqn:Det-Prox-AlgoClass}\\begin{cases}\r\n\t(x^k,\\tilde x^k) := \\cA_x^k\\left(\\theta; x^0,Ay^0,...,x^{k-1},Ay^{k-1}; \\mathbf{prox}_{\\gamma_k f}(u^k) \\right),\\\\\r\n\t(y^k,\\tilde y^k) := \\cA_y^k\\left(\\theta; y^0,A^\\top x^0,...,y^{k-1},A^\\top x^{k-1}; \\mathbf{prox}_{\\sigma_k g}(v^k) \\right),\r\n\t\\end{cases}\r\n\t\\end{equation} \r\n\twhere $u^k = \\cA^k_u(\\theta; x^0,Ay^0,...,x^{k-1},Ay^{k-1})$, $v^k = \\cA^k_v(\\theta; y^0,A^\\top x^0,...,y^{k-1},A^\\top x^{k-1})$, and \r\n\t$(x^0,y^0)$ is any given initial solution. \r\n\\end{definition}\r\nOne remark is that the input of the proximal mapping $\\mathbf{prox}_{\\gamma_kf}(\\cdot)$ is  constructed with other inputs to the $\\cA_x^k$, i.e., there could be another deterministic mapping $\\cA_u^k$ to generate a vector  $u^k = \\cA^k_u(\\theta; x^0,Ay^0,...,x^{k-1},Ay^{k-1})$ and then $\\mathbf{prox}_{\\gamma_kf}(u^k)$ is passed to the mapping $\\cA_x^k$. This $\\cA^k_u$ does not need to be linear. The situation for $v^k$ and $\\cA^k_v$ is similar. Similar to the general proximal algorithm class, the general pure first-order algorithm class is defined as follows. \r\n","start":32650,"end":35118},{"raw_tex":"Let us construct the following bilinearly coupled min-max saddle point problem:\r\n\\begin{equation}\r\n\\label{prob:worst-proximal}\r\n\\min_x\\max_y F(x,y):=\\frac{\\mu_x}{2}\\|x\\|^2 + \\frac{L_{xy}}{2}x^\\top Ay - \\frac{\\mu_y}{2}\\|y\\|^2 - b^\\top y\r\n\\end{equation}\r\nwhere $b$ is a vector to be determined later, and the coupling matrix $A$ (hence $A^2$ and $A^4$) is defined as follows: %. Its polynomials $A^2$ and $A^4$ are also shown bellow for later reference.\r\n\\begin{equation}\r\n\\label{defn:A}\r\nA = \\begin{pmatrix}\r\n& &    &     & 1\\\\\r\n& &    &  1 & -1\\\\\r\n& & 1 & -1&\\\\\r\n& \\udots &\\udots & & \\\\\r\n1& -1 & & &\r\n\\end{pmatrix}, \\,\\,\r\nA^2 = \\begin{pmatrix}\r\n1   & -1 &            &               & \\\\\r\n-1 & 2   & -1        &              & \\\\\r\n& -1 &\\ddots & \\ddots & \\\\\r\n&      &\\ddots & 2           &-1\\\\\r\n&      &              & -1         & 2\r\n\\end{pmatrix},\r\n\\,\\,\r\nA^4 = \\begin{pmatrix}\r\n2   & -3 &     1      &               &  & &\\\\\r\n-3 & 6   & -4        &     1         & & & \\\\\r\n1& -4 &6 & -4 &1 \\\\\r\n&  \\ddots     &\\ddots & \\ddots  &\\ddots & \\ddots & \\\\\r\n& & 1& -4 &6 & -4 &1\\\\\r\n& & & 1& -4 &6 & -4 \\\\\r\n& & & & 1&-4&5\r\n\\end{pmatrix}.\r\n\\end{equation}\r\nNote that $A^\\top =A$ and $\\|A\\|_2 \\leq 2$. Therefore \\eqref{prob:worst-proximal} is an instance in the problem class $\\cB(L_{xy},\\mu_x,\\mu_y)$. It is worth noting that the example \\eqref{prob:worst-proximal} is the same as that in Proposition 2 of \\cite{LowB-Quadratic}, which is a parallel work focused on pure first-order algorithm class. Here, we use the same example to elaborate the lower bound of the proximal methods over the general bilinear coupling class $\\cB(L_{xy},\\mu_x,\\mu_y)$, as well as a warmup for the discussion of more complex problem class $\\cF(L_{x},L_{y},L_{xy},\\mu_x,\\mu_y)$. \r\n\r\nDenote $e_i$ to be the $i$-th unit vector, which has $1$ at the $i$-th component and 0 elsewhere. Then by direct calculation, one can check that $A^2$ satisfies the following \\emph{zero-chain property} (see Chapter 2 of  \\cite{nesterov2018lectures}).\r\n\r\n","start":37175,"end":39181},{"raw_tex":"For problem \\eqref{prob:worst-proximal}, the proximal operators in \\eqref{eqn:Prox-AlgoClass} can be written explicitly:\r\n\\begin{eqnarray}\r\n\\label{defn:prox-closedForm-x}\r\n\\prox_{\\gamma_i f}(\\hat{x}_i-\\gamma_iA\\tilde{y}_i) & = & \\argmin_x \\frac{\\mu_x}{2}\\|x\\|^2 + \\frac{1}{2\\gamma_i}\\left\\|x-(\\hat{x}_i-\\frac{\\gamma_iL_{xy}}{2}A\\tilde{y}_i)\\right\\|^2\\\\\r\n& = & \\frac{1}{1+\\gamma_i\\mu_x}\\hat{x}_i - \\frac{\\gamma_iL_{xy}}{2(1+\\gamma_i\\mu_x)}A\\tilde{y}_i\\nonumber\\\\\r\n& \\in & \\mathrm{Span}\\{\\hat{x}_i,A\\tilde{y}_i\\}.\\nonumber\r\n\\end{eqnarray}\r\nSimilarly, for the $y$ block, we also have\r\n\\begin{eqnarray}\r\n\\label{defn:prox-closedForm-y}\r\n\\prox_{\\sigma_i g}(\\hat{y}_i+\\sigma_iA^\\top\\tilde{x}_i)\r\n%& = & \\argmin_y \\frac{\\mu_y}{2}\\|y\\|^2 + b^Ty + \\frac{1}{2\\sigma_i}\\left\\|y-(\\hat{y}_i+\\frac{\\sigma_iL_{xy}}{2}A^\\top \\tilde{x}_i)\\right\\|^2\\\\\r\n=  \\frac{\\hat{y}_i-\\sigma_i b}{1+\\sigma_i\\mu_y} +  \\frac{\\sigma_iL_{xy}}{2(1+\\sigma_i\\mu_x)}A\\tilde{x}_i \\in \\mathrm{Span}\\{\\hat{y}_i,A\\tilde{x}_i,b\\}.\r\n\\end{eqnarray}\r\nLet us assume the initial point to be $x^0 = y^0 = 0$ ($\\cH_x^0 = \\cH_y^0 = \\{0\\}$) without loss of generality.\r\n","start":39569,"end":40685},{"raw_tex":"Let us lower bound the dual gap, which is upper bounded by the whole duality gap. To achieve this, let us first write down the dual function of problem \\eqref{prob:worst-proximal} as\r\n\\begin{equation}\r\n\\label{defn:dual-function-prox}\r\n\\Psi(y) = \\min_{x} F(x,y) = -\\frac{1}{2}y^\\top \\left(\\frac{L_{xy}^2}{4\\mu_x}\\cdot A^2 + \\mu_y\\cdot I\\right)y - b^\\top y . \r\n\\end{equation}\r\nFor this $\\mu_y$-strongly concave dual function, we can characterize the optimal solution $y^*$ directly by its KKT condition $\\nabla \\Psi(y^*) = 0$. However, the exact solution $y^*$ does not have a simple and clear form, so we choose to  characterize it by an approximate solution $\\hat y^*$.\r\n","start":42258,"end":42929},{"raw_tex":"Consider the bilinear problem class $\\cB(L_{xy},\\mu_x,\\mu_y)$ and the corresponding worst case problem \\eqref{prob:worst-proximal} with $F(x,y) = \\frac{\\mu_x}{2}\\|x\\|^2 + \\frac{L_{xy}}{2}x^\\top A y - \\frac{\\mu_y}{2}\\|y\\|^2 - b^\\top y$,\r\nwhere $A$ and $b$ are defined in accordance with Theorem \\ref{thm:proximal}. We define the orthogonally rotated problem as \r\n\\begin{equation}\r\n\\label{prob:rotate}\r\n\\min_x\\max_y F_{U,V}(x,y) : = F(Ux,Vy) = \\frac{\\mu_x}{2}\\|x\\|^2 + \\frac{L_{xy}}{2}x^\\top U^\\top AV y - \\frac{\\mu_y}{2}\\|y\\|^2 - b^\\top Vy,\r\n\\end{equation}\r\nwhere $U,V$ are two orthogonal matrices. Therefore, it is clear that $F_{U,V}\\in\\cB(L_{xy},\\mu_x,\\mu_y)$. Let $(x^*, y^*)$ be the saddle point of $F(x,y)$, then it is clear that the saddle point of $F_{U,V}(x,y)$ is $(\\bar x^*,\\bar y^*) = (U^\\top  x^*,V^\\top y^*).$ Consequently, the lower bound for the general proximal algorithm class is characterized by the following theorem. \r\n","start":49395,"end":50334},{"raw_tex":"\\section{Proof of Theorem \\ref{theorem:gen-prox}}\r\n\\label{appdx:gen-prox} \r\n\tBefore proceeding the proof, let us first quote a lemma from \\cite{YYXu}. \r\n\t\\begin{lemma}[Lemma 3.1, \\cite{YYXu}] \r\n\t\t\\label{lemma:yyx} \r\n\t\tLet $\\cX\\subsetneqq\\bar{\\cX}\\subseteqq\\R^{p}$ be two linear subspaces. Then for any $\\bar x\\in\\R^p$, there exists an orthogonal matrix $\\Gamma\\in\\R^{p\\times p}$ s.t. $\\Gamma x = x, \\forall x\\in\\cX$ and $\\Gamma \\bar x\\in\\bar{\\cX}$.\r\n\t\\end{lemma}\r\n\tNote that for an orthogonal matrix $\\Gamma$, if $\\Gamma x = x$, then we also have $\\Gamma^\\top x = x$.  Now let us start our proof of Theorem \\ref{theorem:gen-prox}.\r\n\r\n","start":79022,"end":79656}]},"statement":{"raw_tex":"\\begin{theorem}\r\n\\label{theorem:gen-prox}\r\nLet $\\cA$ be any algorithm from the general proximal algorithm class decribed in Definition \\ref{defn:Det-Prox-AlgoClass}. We assume the dimension $n$ is sufficiently large for simplicity. For any integer $k$, then there exist orthogonal matrices $U,V$ s.t. $F_{U,V}\\in\\cB(L_{xy},\\mu_x,\\mu_y)$, when applying $\\cA$ to $F_{U,V}$ with initial solution $(x^0,y^0) = (0,0)$, the iterates and output satisfies\r\n$$\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1}\\qquad\\mbox{and}\\qquad(\\tilde x^k,\\tilde y^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1},$$\r\nwhere $\\cH_x^i,\\cH_y^i$ are defined by Lemma \\ref{lemma:subspace-prox}. Consequenty, by Theorem \\ref{thm:proximal}, \r\n\\begin{eqnarray*}\r\n\\|\\tilde y^k-V^\\top  y^*\\|^2\\geq \\frac{q^{4k+2}}{16}\\|y^*-y^0\\|^2\r\n\\end{eqnarray*}\r\nwhere $q$ is given in Theorem \\ref{thm:proximal}. As a result, it takes $\\Omega\\left(\\sqrt{\\frac{L^2_{xy}}{\\mu_x\\mu_y}}\\cdot\\log\\left(\\frac{1}{\\epsilon}\\right)\\right)$ iterations to output a solution with $O(\\epsilon)$ duality gap.\r\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\r\n\\label{theorem:gen-prox}\r\nLet $\\cA$ be any algorithm from the general proximal algorithm class decribed in Definition \\ref{defn:Det-Prox-AlgoClass}. We assume the dimension $n$ is sufficiently large for simplicity. For any integer $k$, then there exist orthogonal matrices $U,V$ s.t. $F_{U,V}\\in\\cB(L_{xy},\\mu_x,\\mu_y)$, when applying $\\cA$ to $F_{U,V}$ with initial solution $(x^0,y^0) = (0,0)$, the iterates and output satisfies\r\n$$\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1}\\qquad\\mbox{and}\\qquad(\\tilde x^k,\\tilde y^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1},$$\r\nwhere $\\cH_x^i,\\cH_y^i$ are defined by Lemma \\ref{lemma:subspace-prox}. Consequenty, by Theorem \\ref{thm:proximal}, \r\n\\begin{eqnarray*}\r\n\\|\\tilde y^k-V^\\top  y^*\\|^2\\geq \\frac{q^{4k+2}}{16}\\|y^*-y^0\\|^2\r\n\\end{eqnarray*}\r\nwhere $q$ is given in Theorem \\ref{thm:proximal}. As a result, it takes $\\Omega\\left(\\sqrt{\\frac{L^2_{xy}}{\\mu_x\\mu_y}}\\cdot\\log\\left(\\frac{1}{\\epsilon}\\right)\\right)$ iterations to output a solution with $O(\\epsilon)$ duality gap.\r\n\\end{theorem}","start":50334,"end":51429}]},"proof":{"raw_tex":"\\begin{proposition}[Zero-chain property]\r\n\t\\label{prop:zero-chain}\r\n\tFor any vector $v\\in\\R^n$, if $v\\in\\mathrm{Span}\\{e_i:i\\leq k\\}$ for some $1\\leq k\\leq n-1$,  then $A^2v\\in\\mathrm{Span}\\{e_i:i\\leq k+1\\}$.\r\n\\end{proposition}\n\nDirectly substituting \\eqref{defn:prox-closedForm-x} and \\ref{defn:prox-closedForm-y} into Definition \\eqref{defn:Prox-AlgoClass} yields\r\n\\begin{equation*}\r\n\\begin{cases}\r\n\\cH_x^1 \\subseteq \\mathrm{Span}\\{0\\}\\\\\r\n\\cH_y^1 \\subseteq \\mathrm{Span}\\{b\\}\r\n\\end{cases}\r\n\\quad\r\n\\begin{cases}\r\n\\cH_x^2 \\subseteq \\mathrm{Span}\\{Ab\\}\\\\\r\n\\cH_y^2 \\subseteq \\mathrm{Span}\\{b\\}\r\n\\end{cases}\r\n\\quad\r\n\\begin{cases}\r\n\\cH_x^3 \\subseteq \\mathrm{Span}\\{Ab\\}\\\\\r\n\\cH_y^3 \\subseteq \\mathrm{Span}\\{b,A^2b\\}\r\n\\end{cases}\r\n\\quad\r\n\\begin{cases}\r\n\\cH_x^4 \\subseteq \\mathrm{Span}\\{Ab, A^3b\\}\\\\\r\n\\cH_y^4 \\subseteq \\mathrm{Span}\\{b,A^2b\\}\r\n\\end{cases}\r\n\\ldots\r\n\\end{equation*}\r\nWe formally summarize this observation below: %in the Lemma \\ref{lemma:subspace-prox}.\r\n\\begin{lemma}\r\n\t\\label{lemma:subspace-prox}\r\n\tFor problem \\eqref{prob:worst-proximal}, for any $k \\in \\mathbb{N}$, if the iterates are generated so that  $(x_k,y_k)\\in\\cH_x^k\\times\\cH_y^k$, with $\\cH_x^k$ and $\\cH_y^k$ defined by \\eqref{defn:Prox-AlgoClass}, then based on \\eqref{defn:prox-closedForm-x} and \\eqref{defn:prox-closedForm-y} the search subspaces satisfy\r\n\t$$\\cH_x^k \\subseteq \\begin{cases}\r\n\t\\{0\\}, &  k = 1\\\\\r\n\t\\mathrm{Span}\\left\\{A^{2i}(Ab): 0\\leq i\\leq \\left\\lfloor \\frac{k}{2}\\right\\rfloor -1 \\right\\}, & k\\geq 2\r\n\t\\end{cases} \\quad \\mbox{ and }\\quad\r\n\t\\cH_y^k \\subseteq \\mathrm{Span}\\left\\{A^{2i}b: 0\\leq i\\leq \\left\\lceil \\frac{k}{2}\\right\\rceil -1 \\right\\}.$$\r\n\\end{lemma}\n\n\\begin{lemma}[Approximate optimal solution]\r\n\t\\label{lemma:apxsolu-proximal}\r\n\tLet us assign the value of $b$ as $b := -\\frac{L_{xy}^2}{4\\mu_x}e_1$. Denote $\\alpha := \\frac{4\\mu_x\\mu_y}{L_{xy}^2}$, and let $q = \\half\\left((2+\\alpha) - \\sqrt{(2+\\alpha)^2 - 4}\\right)\\in(0,1)$\r\n\tbe the smallest root of the quadratic equation $1 - (2+\\alpha)q + q^2 = 0$. Then, an approximate optimal solution $\\hat y^*$ can be constructed as\r\n\t\\begin{equation}\r\n\t\\hat y^*_{i} = \\frac{q^i}{1-q} \\quad\\mbox{ for } \\quad i = 1,2,...,n.\r\n\t\\end{equation}\r\n\tThe approximation error can be bounded by\r\n\t\\begin{equation}\r\n\t\\label{eqn:apxsolu-proximal}\r\n\t\\|\\hat y^* - y^*\\| \\leq  \\frac{q^{n+1}}{\\alpha(1-q)},\r\n\t\\end{equation}\r\n\twhere $\\hat y^*_i$ is the $i$-th element of $\\hat y^*$. Note that $q<1$ and the lower bound is dimension-independent, hence we are free to choose $n$ to make the approximation error arbitrarily small.\r\n\\end{lemma}\n\n\\begin{proof}\r\n\tFirst, let us substitute the value of $b$ into the KKT system $\\nabla \\Psi(y^*)=0$, by slight rearranging and scaling the terms, we get\r\n\t\\begin{equation*}\r\n\t\\left(A^2 + \\frac{4\\mu_x\\mu_y}{L_{xy}^2} I\\right)y^* = - \\frac{4\\mu_x}{L_{xy}^2}b.\r\n\t\\end{equation*}\r\n\tUsing the definition of $\\alpha$ and $b$, the equation becomes\r\n\t$$(A^2+\\alpha I)y^* = e_1.$$\r\n\tSubstituting the formula of $A^2$ in \\eqref{defn:A}, we expand the above equation as\r\n\t\\begin{eqnarray}\r\n\t\\label{defn:KKT-proximal}\r\n\t\\begin{cases}\r\n\t(1+\\alpha)y_1^* - y_2^* = 1\\\\\r\n\t-y_1^* + (2+\\alpha)y_2^* - y_3^* = 0\\\\\r\n\t\\qquad\\qquad\\quad\\vdots\\\\\r\n\t-y_{n-2}^* + (2+\\alpha)y_{n-1}^* - y_n^* = 0\\\\\r\n\t-y_{n-1}^* + (2+\\alpha)y_n^* = 0.\r\n\t\\end{cases}\r\n\t\\end{eqnarray}\r\n\tBy direct calculation, we can check that $\\hat y^*$ satisfies the first $n-1$ equations of the KKT system \\eqref{defn:KKT-proximal}. The last equation, however, is violated, but with a residual of size $q^{n+1}/(1-q)$.  In details,\r\n\t$$\\begin{cases}\r\n\t(A^2 + \\alpha\\cdot I)\\hat y^* = e_1 + \\frac{q^{n+1}}{1-q}\\cdot e_n\\\\\r\n\t(A^2 + \\alpha\\cdot I)y^* = e_1.\r\n\t\\end{cases}$$\r\n\tThis indicates that $\\hat y^* - y^* = \\frac{q^{n+1}}{1-q}\\cdot(A^2 + \\alpha I)^{-1} e_n$. Note that $\\alpha^{-1}I\\succeq (A^2 + \\alpha I)^{-1}\\succ 0$, we have the approximation error bounded by \\eqref{eqn:apxsolu-proximal}.\r\n\\end{proof}\n\n Note that in Lemma \\ref{lemma:apxsolu-proximal}, we have chosen  $b\\propto e_1$. By the zero-chain property in Proposition \\ref{prop:zero-chain} and Lemma \\ref{lemma:subspace-prox},  we can verify that the subspaces $\\cH_y^{2k-1}$ and $\\cH_y^{2k}$ satisfy\r\n\\begin{equation}\r\n\\label{eqn:subspace-prox}\r\n\\cH_y^{2k-1}, \\cH_y^{2k} \\subseteq \\mathrm{Span}\\{b,A^2b,...,A^{2(k-1)}b\\} =\\mathrm{Span}\\{e_1, e_2, ...,e_k\\}.\r\n\\end{equation}\r\nThis implies that for both $y^{2k}$ and $y^{2k-1}$, the only possible nonzero elements are the first $k$ ones, which again implies that the lower bound of $\\|y^{2k}-y^*\\|^2$ and $\\|y^{2k-1}-y^*\\|^2$ will be similar. For simplicity, we only discuss this lower bound  for $y^{2k}$. The counterpart for $y^{2k-1}$ can be obtained in a similar way.  Therefore, we have the following estimations.\r\n\n\n\\begin{lemma}\r\n\t\\label{lemma:lowerbound-distance-prox}\r\n\tAssume $k\\leq \\frac{n}{2}$ and $n\\geq2\\log_q\\left(\\frac{\\alpha}{4\\sqrt{2}}\\right)$. Then\r\n\t\\begin{equation}\r\n\t\\label{eqn:lowB-dis-prox}\r\n\t\\|y^{2k}-y^*\\|^2\\geq \\frac{q^{2k}}{16}\\|y^0-y^*\\|^2\r\n\t\\end{equation}\r\n\twhere $y^0 = 0$ is the initial solution.\r\n\\end{lemma}\n\n\\begin{proof}\r\n\tBy the subspace characterization \\eqref{eqn:subspace-prox}, we have\r\n\t$$\\|y^{2k}-\\hat y^*\\| \\geq \\sqrt{\\sum^n_{j = k+1} (\\hat y^*_j)^2} = \\frac{q^{k}}{1-q}\\sqrt{q^2 + q^4 + \\cdots + q^{2(n-k)}}\\geq \\frac{q^k}{\\sqrt{2}}\\|\\hat y^*\\|= \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|,$$\r\n\twhere the last inequality is due to the fact that $q< 1$, $k\\leq \\frac{n}{2}$, and $y^0 = 0$.\r\n If we choose $n$ to be large enough, then $\\hat y^*$ and $y^*$ can be made arbitrarily close to each other. Hence we can transform the above inequality to  \\eqref{eqn:lowB-dis-prox}. More details of this derivation can be found in Appendix \\ref{appendix-lowerbound-distance-prox}.\r\n\\end{proof}\n\nUsing Lemma \\ref{lemma:lowerbound-distance-prox} and \\eqref{defn:primal-gap-LB}, it is then straightforward to lower bound the duality gap by\r\n$$\\Delta(x^{2k},y^{2k})\\geq q^{2k}\\cdot\\frac{\\mu_y\\|y^*-y^0\\|^2}{32}.$$\r\n\n\n\\begin{theorem}\r\n\t\\label{thm:proximal}\r\n Let the positive parameters $\\mu_x,\\mu_y>0$ and $L_{xy}> 0$ be given. For any integer $k$, there exists a problem instance from $\\cB(L_{xy},\\mu_x,\\mu_y)$ of form \\eqref{prob:worst-proximal}, with $n\\geq \\max\\left\\{2\\log_q\\left(\\frac{\\mu_x\\mu_y}{\\sqrt{2}L_{xy}^2}\\right), 4k\\right\\}$, where $A\\in\\mathbb{R}^{n\\times n}$ as defined in \\eqref{defn:A}, and $b = -\\frac{L_{xy}^2}{4\\mu_x}e_1$. For such a problem instance, any approximate solution $(\\tilde x^k, \\tilde y^k)\\in\\cH_x^k\\times\\cH_y^k$ generated by the proximal algorithm class under the linear span assumption \\eqref{eqn:Prox-AlgoClass}  satisfies\r\n\t\\begin{eqnarray}\r\n\t\\max_y F(\\tilde x^{k},y) - \\min_{x} F(x, \\tilde y^k)\\geq q^{k}\\cdot\\frac{\\mu_y\\|y^*-y^0\\|^2}{32} \\quad\\mbox{ and }\\quad \\|\\tilde y^k-y^*\\|^2\\geq q^{k}\\cdot\\frac{\\|y^*-y^0\\|^2}{16},\r\n\t\\end{eqnarray}\r\n\twhere $q = 1+\\frac{2\\mu_x\\mu_y}{L_{xy}^2} - 2\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}$.\r\n\\end{theorem}\n\n\\begin{proposition}\r\n\t\\label{remark:complexity-proximal}\r\n\tUnder the same set of assumptions of Theorem \\ref{thm:proximal}, if we require the duality gap to be bounded by $\\epsilon$, the number of iterations needed is at least\r\n\t\\begin{equation}\r\n\t\\label{eqn:lowerbound-proximal-1stAlgo}\r\n\tk\\geq \\ln\\left(\\frac{\\mu_y\\|y^*-y^0\\|^2}{32\\epsilon}\\right)/\\ln(q^{-1}) = \\Omega\\left(\\sqrt{\\frac{L_{xy}^2}{\\mu_x\\mu_y}}\\cdot\\ln\\left(\\frac{1}{\\epsilon}\\right)\\right).\r\n\t\\end{equation}\r\n\\end{proposition}\n\nFor the proof of this theorem, we only need to construct the orthogonal matrices $U,V$ such that when the algorithm $\\cA$ is applied to $F_{U,V}$, the subspace inclusion argument $\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1}$ and $(\\tilde x^k,\\tilde y^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1}$ holds. As a result, \r\n\\begin{eqnarray*}\r\n\t\\|\\tilde y^k-V^\\top  y^*\\|^2 = \\|V\\tilde y^k- y^*\\|^2\\geq\\min_{y\\in\\cH_y^{4k+1}}\\|y-y^*\\|^2\\geq \\frac{q^{4k+2}}{16}\\|y^*-y^0\\|^2.\r\n\\end{eqnarray*}\r\nWith this argument, the latter results follow directly from the discussion of Theorem \\ref{thm:proximal}. The proof of this theorem is presented in Appendix \\ref{appdx:gen-prox}.\r\n\n\n\\section{Proof of Lemma \\ref{lemma:lowerbound-distance-prox}}\r\n\\label{appendix-lowerbound-distance-prox}\r\nBy the subspace characterization \\eqref{eqn:subspace-prox}, we have\r\n$$\\|y^{2k}-\\hat y^*\\| \\geq \\sqrt{\\sum^n_{j = k+1} (\\hat y^*_j)^2} = \\frac{q^{k}}{1-q}\\sqrt{q^2 + q^4 + \\cdots + q^{2(n-k)}}\\geq \\frac{q^k}{\\sqrt{2}}\\|\\hat y^*\\|= \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|,$$\r\nwhere the last inequality is due to the fact that $q\\leq 1, k\\leq \\frac{n}{2}$ and $y^0 = 0$. Note that by Lemma \\ref{lemma:apxsolu-proximal}, if we require $n\\geq 2\\log_q\\left(\\frac{\\alpha}{4\\sqrt{2}}\\right)$, then we can guarantee that \\begin{equation}\r\n\\label{eqn:err-n-proximal}\r\n\\|\\hat y^* - y^*\\| \\leq \\frac{q^{n+1}}{\\alpha(1-q)}\\leq  \\frac{q^{\\frac{n}{2}}}{\\alpha}\\cdot q^k\\cdot\\frac{q}{(1-q)} \\leq \\frac{1}{4}\\cdot \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|\\quad\\mbox{ for }\\quad \\forall 1\\leq k\\leq n/2,\r\n\\end{equation}\r\nwhere the last inequality is due to $\\frac{q^{\\frac{n}{2}}}{\\alpha}\\leq\\frac{1}{4\\sqrt{2}}$ and $q/(1-q)\\leq \\|y^0-\\hat y^*\\|$. Therefore, we have\r\n\\begin{eqnarray}\r\n\\label{eqn:dist-lower-proximal}\r\n\\|y^{2k}-y^*\\|^2 & \\geq & (\\|y^{2k}-\\hat y^*\\|-\\|\\hat y^*- y^*\\|)^2\\\\\r\n& \\geq & \\|y^{2k}-\\hat y^*\\|^2 - 2\\|y^{2k}-\\hat y^*\\|\\|\\hat y^*- y^*\\|\\nonumber\\\\\r\n& \\geq & \\min_t\\left\\{t^2 - 2\\|\\hat y^*- y^*\\|t: t\\geq \\delta_k:= \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|\\right\\}\\nonumber\\\\\r\n& = & \\delta_k(\\delta_k - 2\\|\\hat y^*- y^*\\|)\\nonumber\\\\\r\n& \\geq & \\frac{1}{2}\\delta_k^2 = \\frac{q^{2k}}{4}\\|y^0-\\hat y^*\\|^2, \\nonumber\r\n\\end{eqnarray}\r\nwhere the fourth line is due to that $d(t^2 - 2\\|\\hat y^*-y^*\\|t)/dt = 2(t-\\|\\hat y^*-y^*\\|)\\geq0$ when $t\\geq\\delta_k$. Hence the quadratic function is monotonically increasing in the considered interval. In addition, we also have\r\n$$\\|y^0-y^*\\| \\leq \\|y^0-\\hat y^*\\| + \\|\\hat y^*-y^*\\|\\leq \\|y^0-\\hat y^*\\| + \\frac{q^n}{\\alpha}\\cdot\\frac{q}{1-q}\\leq (1 + q^n/\\alpha)\\|y^0-\\hat y^*\\|\\leq 2\\|y^0-\\hat y^*\\|,$$\r\nwhere the third inequality is due to that $\\|y^0-\\hat y^*\\|\\geq \\hat y^*_1 = q/(1-q)$. For the last inequality, if $\\alpha\\geq 1$, then $q^n/\\alpha<1$; if $\\alpha\\leq 1$, then  $q^n/\\alpha\\leq \\alpha/32\\leq1$ since $n\\geq 2\\log_q\\left(\\frac{\\alpha}{4\\sqrt{2}}\\right)$. Combining the above two inequalities,  the desired bound \\eqref{eqn:lowB-dis-prox} follows.\r\n\\section{Proof of Proposition \\ref{remark:complexity-proximal}}\r\n\\label{appendix:r-complexity-proximal}\r\nHere we only prove the last inequality of \\eqref{eqn:lowerbound-proximal-1stAlgo}. Due to the fact that $(\\ln(1+z))^{-1}\\geq 1/z$ for $\\forall z>0$, we know\r\n\\begin{eqnarray*}\r\n(\\ln(q^{-1}))^{-1} & = & (\\ln(1 + (1-q)/q))^{-1} \\geq \\frac{q}{1-q}\\\\\r\n& = &   \\frac{1+\\frac{2\\mu_x\\mu_y}{L_{xy}^2} - 2\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}}{2\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}-\\frac{2\\mu_x\\mu_y}{L_{xy}^2}}\\\\\r\n& = & \\frac{\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}-\\frac{\\mu_x\\mu_y}{L_{xy}^2}}{\\frac{2\\mu_x\\mu_y}{L_{xy}^2}}\\\\\r\n& = &  \\half\\sqrt{\\frac{L_{xy}^2}{\\mu_x\\mu_y} + 1} - \\frac{1}{2},\\\\\r\n& = & \\Omega\\left(\\sqrt{\\frac{L_{xy}^2}{\\mu_x\\mu_y}}\\right)\r\n\\end{eqnarray*}\r\nwhich completes the proof.\r\n\r\n\n\n\\begin{proof}\r\n\t\tTo prove this theorem, we only need to show\r\n\t\t$$\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1}\\qquad\\mbox{and}\\qquad(\\tilde x^k,\\tilde y^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1}.$$\r\n\t\tWe separate the proof into two parts.\r\n\t\t\r\n\t\t\\textbf{Part I. There exist orthogonal matrices $\\hat U$, $\\hat V$ s.t. when $\\cA$ is applied to the rotated instance $F_{\\hat U,\\hat V}$, $\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq \\hat U^\\top\\cH_x^{4k-1}\\times \\hat V^\\top\\cH_y^{4k-1}.$}\r\n\t\r\n\t\tLet $\\theta = (L_{xy},\\mu_x,\\mu_y)$ be the set of algorithmic parameters. To prove the result, let us construct the worst-case function $F_{U,V}$ in a recursive way. \t \\\\\r\n\t\t\\textbf{Case $k = 1$:} Let us define $U_0 = V_0 = I$. When $\\cA$ is applied to the function $F_{U_0,V_0}\\in\\cB(L_{xy}, \\mu_x, \\mu_y)$, the iterate sequence is \r\n\t\t$(x_{0}^0, y_{0}^0) = (0, 0)$ and \r\n\t\t$$\\begin{cases}\r\n\t\tu^1_0 = \\cA_u^1(\\theta; x_0^0, U_0^\\top A V_0 y_0^0), \\qquad (x^1_0, \\tilde x^1_0) = \\cA_x^1(\\theta; x^0_0,  U_0^\\top A V_0 y_0^0, \\mathbf{prox}_{\\gamma_1f}(u^1_0)),\\\\\r\n\t\tv^1_0 = \\cA_v^1(\\theta; y_0^0, V^\\top_0 A U_0x_0^0),\\qquad\r\n\t\t(y^1_0, \\tilde y^1_0) = \\cA_y^1(\\theta; y^0_0, V^\\top_0 A U_0x_0^0, \\mathbf{prox}_{\\sigma_1g}(v^1_0)).\r\n\t\t\\end{cases}$$\r\n\t\tBy Lemma \\ref{lemma:yyx}, there exists orthogonal matrices $\\Gamma_x^0$ and $\\Gamma_y^0$ such that \r\n\t\t$\\Gamma_x^0x^1_0\\in\\cH_x^3 = \\mathrm{Span}\\{Ab\\}$, $\\Gamma_y^0y^1_0\\in\\cH_y^3=\\mathrm{Span}\\{b, A^2b\\}$, and $\\Gamma_y^0b = (\\Gamma_y^0)^\\top b = b.$ That is \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-1}\r\n\t\tx_0^1\\in U_1^\\top\\cH_x^3,\\qquad\\mbox{and}\\qquad y_0^1\\in V_1^\\top\\cH_y^3, \\quad V_1 b = V_1^\\top b = b,\r\n\t\t\\end{equation}\t\r\n\t\twhere $U_1 = U_0\\Gamma_x^0$ and $V_1 = V_0\\Gamma_y^0$. \r\n\t\r\n\t\tNow we prove that when we apply the algorithm $\\cA$ to $F_{U_1,V_1}$, the generated iterates $\\{(x^0_1, y^0_1), (x^1_1, y^1_1)\\}$ satisfy that\r\n\t\t$(x^0_1, y^0_1) = (0, 0)$ and  $(x^1_1, y^1_1) = (x^1_0, y^1_0)$. That is, the first two iterates generated by $\\cA$ is completely the same for $F_{U_0,V_0}$ and $F_{U_1,V_1}$. The reason is because \r\n\t\t$u^1_1 = \\cA_u^1(\\theta; x^0_1, U_1^\\top AV_1y^0_1) = \\cA_u^1(\\theta; 0, 0) = \\cA_u^1(\\theta; x^0_0, U_0^\\top AV_0y^0_0)  = u^1_0$, \r\n\t\ttherefore \r\n\t\t\\begin{eqnarray*}\r\n\t\t\t(x^1_1, \\tilde x^1_1) & = & \\cA_x^1(\\theta; x^0_1, U_1^\\top A V_1 y_1^0, \\mathbf{prox}_{\\gamma_1f}(u^1_1)) \\\\\r\n\t\t\t& = & \\cA_x^1(\\theta; 0, 0, \\mathbf{prox}_{\\gamma_1f}(u^1_1)) \\\\\r\n\t\t\t& = & \\cA_x^1(\\theta; x^0_0, U_0^\\top A V_0 y_0^0, \\mathbf{prox}_{\\gamma_1f}(u^1_0)) \\\\\r\n\t\t\t& = & (x^1_0, \\tilde x^1_0).\r\n\t\t\\end{eqnarray*}\r\n\t\tThrough similar argument, we know $(y^1_1,\\tilde y^1_1) = (y^1_0,\\tilde y^1_0)$. Therefore, \\eqref{thm:gen-prox-1} indicates that \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-2}\r\n\t\tx_1^1\\in U_1^\\top\\cH_x^3,\\qquad\\mbox{and}\\qquad y_1^1\\in V_1^\\top\\cH_y^3, \\quad V_1 b = V_1^\\top b = b\\in V_1^\\top\\cH_y^3.\r\n\t\t\\end{equation}\r\n\t\t\\textbf{Case $k=2$.} For the ease of the readers to follow, we perform one extra step of discussion for $k=2$, before presenting the construction on general $k$. \r\n\t    \r\n\t\tFor the problem instance $F_{U_1,V_1}$, the iterates generated by $\\cA$ are $(x_{1}^0, y_{1}^0) = (0, 0)$ and \r\n\t\t$$\\begin{cases}\r\n\t\tu^1_1 = \\cA_u^1(\\theta; x_1^0, U_1^\\top A V_1 y_1^0), \\qquad (x^1_1, \\tilde x^1_1) = \\cA_x^1(\\theta; x^0_1,  U_1^\\top A V_1 y_1^0, \\mathbf{prox}_{\\gamma_1f}(u^1_1)),\\\\\r\n\t\tv^1_1 = \\cA_v^1(\\theta; y_1^0, V^\\top_1 A U_1x_1^0),\\qquad\r\n\t\t(y^1_1, \\tilde y^1_1) = \\cA_y^1(\\theta; y^0_1, V^\\top_1 A U_1x_1^0, \\mathbf{prox}_{\\sigma_1g}(v^1_1)).\\\\\r\n\t\t\\end{cases}$$\r\n\t\t$$\\begin{cases}\r\n\t\tu^2_1 = \\cA_u^2(\\theta; x_1^0, U_1^\\top A V_1 y_1^0, x_1^1, U_1^\\top A V_1 y_1^1), \\quad (x^2_1, \\tilde x^2_1) = \\cA_x^2(\\theta; x^0_1,  U_1^\\top A V_1 y_1^0, x_1^1, U_1^\\top A V_1 y_1^1, \\mathbf{prox}_{\\gamma_2f}(u^2_1)),\\\\\r\n\t\tv^2_1 = \\cA_v^2(\\theta; y_1^0, V^\\top_1 A U_1x_1^0, y_1^1, V^\\top_1 A U_1x_1^1),\\quad\r\n\t\t(y^2_1, \\tilde y^2_1) = \\cA_y^2(\\theta; y^0_1, V^\\top_1 A U_1x_1^0, y_1^1, V^\\top_1 A U_1x_1^1, \\mathbf{prox}_{\\sigma_2g}(v^2_1)).\\\\\r\n\t\t\\end{cases}$$\r\n\t\tNote that  $x_1^1 \\in U_1^\\top\\cH_x^3 \\subsetneqq U_1^\\top\\cH_x^5\\subsetneqq U_1^\\top\\cH_x^7$ and $\\{y_1^1,b\\}\\subsetneqq V_1^\\top\\cH_y^3\\subsetneqq V_1^\\top\\cH_y^5\\subsetneqq V_1^\\top\\cH_y^7$. Therefore, there exist orthogonal matrices $\\Gamma_x^1$ and $\\Gamma_y^1$ such that \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-3}\r\n\t\t\\begin{cases}\r\n\t\t\\Gamma_x^1x = (\\Gamma_x^1)^\\top x = x, \\,\\,\\forall x\\in U_1^\\top\\cH_x^5,\\,\\,\\,\\, \\Gamma_x^1 x_1^2\\in U_1^\\top\\cH_x^7,\\\\\r\n\t\t\\Gamma_y^1y \\,= (\\Gamma_y^1)^\\top y = y, \\,\\,\\,\\forall y\\in V_1^\\top\\cH_y^5,\\,\\,\\,\\, \\Gamma_y^1 y_1^2\\in V_1^\\top\\cH_y^7.\r\n\t\t\\end{cases}\r\n\t\t\\end{equation}\r\n\t\tNow, let us define \r\n\t\t$$U_2 = U_1\\Gamma_x^1\\qquad\\mbox{and}\\qquad V_2 = V_1\\Gamma_y^1.$$\r\n\t\tNow we prove that if $\\cA$ is applied to $F_{U_2,V_2}$, the generated iterates $\\{(x_{2}^0, y_{2}^0), (x_{2}^1, y_{2}^1), (x_{2}^2, y_{2}^2)\\}$ satisfy $(x_{2}^0, y_{2}^0) = (0, 0)$, $(x_{2}^1, y_{2}^1) = (x_{1}^1, y_{1}^1)$, and  $(x_{2}^2, y_{2}^2) = (x_{1}^2, y_{1}^2)$. The argument for $(x_{2}^1, y_{2}^1) = (x_{1}^1, y_{1}^1)$ is almost the same as  that of the case $k=1$. We only provide the proof for $(x_{2}^2, y_{2}^2) = (x_{1}^2, y_{1}^2)$.\r\n\t\r\n\tNext, we need to show $u_2^2 = u_1^2$, which can be proved by arguing that all the inputs to $\\cA_u^2$ are the same for both $u_2^2$ and $u_1^2$. First, it is straightforward that $x_1^0 = 0 = x^0_2, U_1^\\top AV_1 y_1^0 = 0 = U_2^\\top AV_2 y_2^0$. By previous argument $x_2^1 = x_1^1$.  Finally, consider the last input $U_2^\\top AV_2 y_2^1$, because $y_2^1 = y_1^1\\in V_1^\\top\\cH_y^3\\subsetneqq V_1^\\top\\cH_y^5$, we have $\\Gamma_y^1 y_2^1 = y_2^1 = y_1^1\\in V_1^\\top\\cH_y^3.$ Then $V_2y_2^1 = V_1\\Gamma_y^1y_2^1\\in V_1V_1^\\top\\cH_y^3 = \\cH_y^3.$ Therefore $U_1^\\top AV_2y_2^1\\in U_1^\\top A\\cH_y^3 = U_1^\\top\\cH_x^5$ and \r\n\t\t$$U_2^\\top AV_2y_2^1 = \\Gamma_x^1U_1^\\top AV_2y_2^1 = U_1^\\top AV_2y_2^1 = U_1^\\top AV_1\\Gamma_y^1y_2^1 = U_1^\\top AV_1y_1^1.$$ \r\n\t\tConsequently, \r\n\t\t\\begin{eqnarray*}\r\n\t\t\tu_2^2  = \\cA_u^2(\\theta; x_2^0, U_2^\\top AV_2 y_2^0, x_2^1, U_2^\\top AV_2 y_2^1)\r\n\t\t\t=\\cA_u^2(\\theta; x_1^0, U_1^\\top AV_1 y_1^0, x_1^1, U_1^\\top AV_1 y_1^1)\r\n\t\t\t= u_2^1\r\n\t\t\\end{eqnarray*}\r\n\t\tand \r\n\t\t\\begin{eqnarray*}\r\n\t\t\t(x_2^2,\\tilde x_2^2)  & = & \\cA_x^2(\\theta; x_2^0, U_2^\\top AV_2 y_2^0, x_2^1, U_2^\\top AV_2 y_2^1,\\mathbf{prox}_{\\gamma_2f}(u^2_2))\\\\\r\n\t\t\t& = &\\cA_x^2(\\theta; x_1^0, U_1^\\top AV_1 y_1^0, x_1^1, U_1^\\top AV_1 y_1^1,\\mathbf{prox}_{\\gamma_2f}(u^2_1))\\\\\r\n\t\t\t& = & (x_1^2,\\tilde x_1^2).\r\n\t\t\\end{eqnarray*}\r\n\t\tThrough a similar argument, we have $(y_2^2,\\tilde y_2^2) = (y_1^2,\\tilde y_1^2)$. By \\eqref{thm:gen-prox-2} and \\eqref{thm:gen-prox-3}, we have \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-4}\r\n\t\t\\{x_2^0,x_2^1,x_2^2\\}\\in U_2^\\top\\cH_x^7\\qquad\\mbox{and}\\qquad\\{b,y_2^0,y_2^1,y_2^2\\}\\in V_2^\\top\\cH_y^7.\r\n\t\t\\end{equation}\r\n\t\t\\textbf{Case $k$.} Suppose we already have orthogonal matrices $U_{k-1}, V_{k-1}$, such that when $\\cA$ is applied to $F_{U_{k-1},V_{k-1}}$, we have\r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-5}\r\n\t\t\\{x_{k-1}^0,x_{k-1}^1,\\cdots, x_{k-1}^{k-1}\\}\\in U_{k-1}^\\top\\cH_x^{4k-5}\\qquad\\mbox{and}\\qquad\\{b,y_{k-1}^0,y_{k-1}^1,\\cdots,y_{k-1}^{k-1}\\}\\in V_{k-1}^\\top\\cH_y^{4k-5}.\r\n\t\t\\end{equation}\r\n\t\tAgain, by Lemma \\ref{lemma:yyx}, there exist orthogonal matrices $\\Gamma_x^{k-1}$ and $\\Gamma_y^{k-1}$, such that \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-6}\r\n\t\t\\begin{cases}\r\n\t\t\\Gamma_x^{k-1}x = (\\Gamma_x^{k-1})^\\top x = x, \\,\\,\\forall x\\in U_{k-1}^\\top\\cH_x^{4k-3},\\,\\,\\,\\, \\Gamma_x^{k-1} x_{k-1}^{k}\\in U_{k-1}^\\top\\cH_x^{4k-1},\\\\\r\n\t\t\\Gamma_y^{k-1}y \\,= (\\Gamma_y^{k-1})^\\top y = y, \\,\\,\\,\\forall y\\in V_{k-1}^\\top\\cH_y^{4k-3},\\,\\,\\,\\, \\Gamma_y^{k-1} y_{k-1}^k\\in V_{k-1}^\\top\\cH_y^{4k-1}.\r\n\t\t\\end{cases}\r\n\t\t\\end{equation}\r\n\t\tNow we define that \r\n\t\t$$U_k = U_{k-1}\\Gamma_x^{k-1} \\qquad\\mbox{and}\\qquad V_k = V_{k-1}\\Gamma_y^{k-1}.$$\r\n\t\tTherefore, similar to our previous discussion, we only need to argue that when $\\cA$ is applied to $F_{U_k,V_k}$, the generated iterates $\\{(x_k^0,y_k^0), (x_k^1,y_k^1),\\cdots,(x_k^k,y_k^k)\\}$ satisfy $(x_k^i,y_k^i) = (x_{k-1}^i,y_{k-1}^i)$ for $i = 0,1,...,k$. We prove this argument by induction. First, it is straightforward that $(x_k^0,y_k^0) = (0,0) = (x_{k-1}^0,y_{k-1}^0)$. Suppose $(x_k^i,y_k^i) = (x_{k-1}^i,y_{k-1}^i)$ holds for $i = 0,1,...,j-1\\leq k-1$, now we prove $(x_k^j,y_k^j) = (x_{k-1}^j,y_{k-1}^j)$, which is almost identical to the case $k=2$.\r\n\t\r\n\tFor any $i\\in\\{0,1,...,j-1\\}$, let us show $U_{k-1}^\\top AV_{k-1} y_{k-1}^i = U_k^\\top AV_k y_k^i$. Because $y_k^i = y_{k-1}^i\\in V_{k-1}^\\top\\cH_y^{4k-5}\\subsetneqq V_{k-1}^\\top\\cH_y^{4k-3}$, we have $\\Gamma_y^{k-1} y_k^i = y_k^i = y_{k-1}^i\\in V_{k-1}^\\top\\cH_y^{4k-5}.$ Then $V_ky_k^i = V_{k-1}\\Gamma_y^{k-1}y_k^i\\in V_{k-1}V_{k-1}^\\top\\cH_y^{4k-5} = \\cH_y^{4k-5}.$ Therefore $U_{k-1}^\\top AV_{k}y_k^i\\in U_{k-1}^\\top A\\cH_y^{4k-5} = U_{k-1}^\\top\\cH_x^{4k-3}$ and \r\n\t\t$$U_k^\\top AV_ky_k^i = (\\Gamma_x^{k-1})^\\top U_{k-1}^\\top AV_ky_k^i = U_{k-1}^\\top AV_{k}y_k^i = U_{k-1}^\\top AV_{k-1}\\Gamma_y^{k-1}y_k^i = U_{k-1}^\\top AV_{k-1}y_{k-1}^i,$$ \r\n\t\tfor $0\\leq i\\leq j-1$. Consequently, \r\n\t\t\\begin{eqnarray*}\r\n\t\t\tu_k^i  & = & \\cA_u^i(\\theta; x_k^0, U_k^\\top AV_k y_k^0,..., x_k^{i-1}, U_k^\\top AV_k y_k^{i-1})\\\\\r\n\t\t\t& = & \\cA_u^i(\\theta; x_{k-1}^0, U_{k-1}^\\top AV_{k-1} y_{k-1}^0,..., x_{k-1}^{i-1}, U_{k-1}^\\top AV_{k-1} y_{k-1}^{i-1})\\\\\r\n\t\t\t& = & u_{k-1}^i\r\n\t\t\\end{eqnarray*}\r\n\t\tand \r\n\t\t\\begin{eqnarray*}\r\n\t\t\t(x_k^i,\\tilde x_k^i)  & = & \\cA_x^i(\\theta; x_k^0, U_k^\\top AV_k y_k^0,..., x_k^{i-1}, U_k^\\top AV_k y_k^{i-1},\\mathbf{prox}_{\\gamma_if}(u^i_k))\\\\\r\n\t\t\t& = &\\cA_x^2(\\theta; x_{k-1}^0, U_{k-1}^\\top AV_{k-1} y_{k-1}^0,..., x_{k-1}^{i-1}, U_{k-1}^\\top AV_{k-1} y_{k-1}^{i-1},\\mathbf{prox}_{\\gamma_if}(u^i_{k-1}))\\\\\r\n\t\t\t& = & (x_{k-1}^i,\\tilde x_{k-1}^i).\r\n\t\t\\end{eqnarray*}\r\n\t\tThrough a similar argument, we have $(y_k^i,\\tilde y_k^i) = (y_{k-1}^i,\\tilde y_{k-1}^i)$. By induction, we know $(y_k^i,\\tilde y_k^i) = (y_{k-1}^i,\\tilde y_{k-1}^i)$ for $i = 0,1,...,k$. Consequently, we have \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-7}\r\n\t\t\\{x_{k}^0,x_{k}^1,\\cdots, x_{k}^{k}\\}\\in U_{k}^\\top\\cH_x^{4k-1}\\qquad\\mbox{and}\\qquad\\{b,y_{k}^0,y_{k}^1,\\cdots,y_{k}^{k}\\}\\in V_{k}^\\top\\cH_y^{4k-1}.\r\n\t\t\\end{equation}\r\n\t    By setting $\\hat U = U_k$ and $\\hat V = V_k$, we prove the result for Part I.\\vspace{0.5cm}\\\\       \r\n       \t\\textbf{Part II. There exist orthogonal matrices $U$, $V$ such that when $\\cA$ is applied to the rotated instance $F_{U,V}$, $\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1},$ and $(\\tilde{x}^k,\\tilde{y}^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1}$}.\r\n       \r\n       Given the result of Part I, and let $\\{(x_k^0, y_k^0),...,(x_k^k, y_k^k)\\}$ and $(\\tilde x_k^k, \\tilde y_k^k)$ be generated by $\\cA$ when applied to $F_{\\hat U,\\hat V} = F_{U_k,V_k}$. Therefore, by Lemma \\ref{lemma:yyx}, there exist orthogonal matrices $P,Q$ such that \r\n       \\begin{equation}\r\n       \\label{thm:gen-prox-8}\r\n       \\begin{cases}\r\n       Px = P^\\top x = x, \\,\\,\\forall x\\in U_{k}^\\top\\cH_x^{4k-1},\\,\\,\\,\\, P \\tilde x_{k}^{k}\\in U_{k}^\\top\\cH_x^{4k+1},\\\\\r\n       Qy \\,=Q^\\top y = y, \\,\\,\\forall y\\in V_{k}^\\top\\cH_y^{4k-1},\\,\\,\\,\\, Q\\tilde y_{k}^k\\in V_{k}^\\top\\cH_y^{4k+1}.\r\n       \\end{cases}\r\n       \\end{equation}\r\n       Define $U = U_kP$, and $V = V_kQ$. Let $\\{(x^0,y^0),...,(x^k,y^k)\\}$ and the output $(\\tilde x^k,\\tilde y^k)$ be generated by $\\cA$ when applied to $F_{{U,V}}$. Then following the same line of argument of Case $k$, Part I, we have \r\n       $$(x^i,y^i) = (x^i_k,y^i_k), \\,\\,\\mbox{for}\\,\\, i = 0,1,...,k\\qquad\\mbox{and}\\qquad(\\tilde x^k,\\tilde y^k) = (\\tilde x^k_k,\\tilde y^k_k).$$\r\n       Therefore, combining \\eqref{thm:gen-prox-8}, we complete the proof of Part II.  \r\n\\end{proof}","units":[{"raw_tex":"\\begin{proposition}[Zero-chain property]\r\n\t\\label{prop:zero-chain}\r\n\tFor any vector $v\\in\\R^n$, if $v\\in\\mathrm{Span}\\{e_i:i\\leq k\\}$ for some $1\\leq k\\leq n-1$,  then $A^2v\\in\\mathrm{Span}\\{e_i:i\\leq k+1\\}$.\r\n\\end{proposition}","start":39181,"end":39408},{"raw_tex":"Directly substituting \\eqref{defn:prox-closedForm-x} and \\ref{defn:prox-closedForm-y} into Definition \\eqref{defn:Prox-AlgoClass} yields\r\n\\begin{equation*}\r\n\\begin{cases}\r\n\\cH_x^1 \\subseteq \\mathrm{Span}\\{0\\}\\\\\r\n\\cH_y^1 \\subseteq \\mathrm{Span}\\{b\\}\r\n\\end{cases}\r\n\\quad\r\n\\begin{cases}\r\n\\cH_x^2 \\subseteq \\mathrm{Span}\\{Ab\\}\\\\\r\n\\cH_y^2 \\subseteq \\mathrm{Span}\\{b\\}\r\n\\end{cases}\r\n\\quad\r\n\\begin{cases}\r\n\\cH_x^3 \\subseteq \\mathrm{Span}\\{Ab\\}\\\\\r\n\\cH_y^3 \\subseteq \\mathrm{Span}\\{b,A^2b\\}\r\n\\end{cases}\r\n\\quad\r\n\\begin{cases}\r\n\\cH_x^4 \\subseteq \\mathrm{Span}\\{Ab, A^3b\\}\\\\\r\n\\cH_y^4 \\subseteq \\mathrm{Span}\\{b,A^2b\\}\r\n\\end{cases}\r\n\\ldots\r\n\\end{equation*}\r\nWe formally summarize this observation below: %in the Lemma \\ref{lemma:subspace-prox}.\r\n\\begin{lemma}\r\n\t\\label{lemma:subspace-prox}\r\n\tFor problem \\eqref{prob:worst-proximal}, for any $k \\in \\mathbb{N}$, if the iterates are generated so that  $(x_k,y_k)\\in\\cH_x^k\\times\\cH_y^k$, with $\\cH_x^k$ and $\\cH_y^k$ defined by \\eqref{defn:Prox-AlgoClass}, then based on \\eqref{defn:prox-closedForm-x} and \\eqref{defn:prox-closedForm-y} the search subspaces satisfy\r\n\t$$\\cH_x^k \\subseteq \\begin{cases}\r\n\t\\{0\\}, &  k = 1\\\\\r\n\t\\mathrm{Span}\\left\\{A^{2i}(Ab): 0\\leq i\\leq \\left\\lfloor \\frac{k}{2}\\right\\rfloor -1 \\right\\}, & k\\geq 2\r\n\t\\end{cases} \\quad \\mbox{ and }\\quad\r\n\t\\cH_y^k \\subseteq \\mathrm{Span}\\left\\{A^{2i}b: 0\\leq i\\leq \\left\\lceil \\frac{k}{2}\\right\\rceil -1 \\right\\}.$$\r\n\\end{lemma}","start":40734,"end":42161},{"raw_tex":"\\begin{lemma}[Approximate optimal solution]\r\n\t\\label{lemma:apxsolu-proximal}\r\n\tLet us assign the value of $b$ as $b := -\\frac{L_{xy}^2}{4\\mu_x}e_1$. Denote $\\alpha := \\frac{4\\mu_x\\mu_y}{L_{xy}^2}$, and let $q = \\half\\left((2+\\alpha) - \\sqrt{(2+\\alpha)^2 - 4}\\right)\\in(0,1)$\r\n\tbe the smallest root of the quadratic equation $1 - (2+\\alpha)q + q^2 = 0$. Then, an approximate optimal solution $\\hat y^*$ can be constructed as\r\n\t\\begin{equation}\r\n\t\\hat y^*_{i} = \\frac{q^i}{1-q} \\quad\\mbox{ for } \\quad i = 1,2,...,n.\r\n\t\\end{equation}\r\n\tThe approximation error can be bounded by\r\n\t\\begin{equation}\r\n\t\\label{eqn:apxsolu-proximal}\r\n\t\\|\\hat y^* - y^*\\| \\leq  \\frac{q^{n+1}}{\\alpha(1-q)},\r\n\t\\end{equation}\r\n\twhere $\\hat y^*_i$ is the $i$-th element of $\\hat y^*$. Note that $q<1$ and the lower bound is dimension-independent, hence we are free to choose $n$ to make the approximation error arbitrarily small.\r\n\\end{lemma}","start":42931,"end":43845},{"raw_tex":"\\begin{proof}\r\n\tFirst, let us substitute the value of $b$ into the KKT system $\\nabla \\Psi(y^*)=0$, by slight rearranging and scaling the terms, we get\r\n\t\\begin{equation*}\r\n\t\\left(A^2 + \\frac{4\\mu_x\\mu_y}{L_{xy}^2} I\\right)y^* = - \\frac{4\\mu_x}{L_{xy}^2}b.\r\n\t\\end{equation*}\r\n\tUsing the definition of $\\alpha$ and $b$, the equation becomes\r\n\t$$(A^2+\\alpha I)y^* = e_1.$$\r\n\tSubstituting the formula of $A^2$ in \\eqref{defn:A}, we expand the above equation as\r\n\t\\begin{eqnarray}\r\n\t\\label{defn:KKT-proximal}\r\n\t\\begin{cases}\r\n\t(1+\\alpha)y_1^* - y_2^* = 1\\\\\r\n\t-y_1^* + (2+\\alpha)y_2^* - y_3^* = 0\\\\\r\n\t\\qquad\\qquad\\quad\\vdots\\\\\r\n\t-y_{n-2}^* + (2+\\alpha)y_{n-1}^* - y_n^* = 0\\\\\r\n\t-y_{n-1}^* + (2+\\alpha)y_n^* = 0.\r\n\t\\end{cases}\r\n\t\\end{eqnarray}\r\n\tBy direct calculation, we can check that $\\hat y^*$ satisfies the first $n-1$ equations of the KKT system \\eqref{defn:KKT-proximal}. The last equation, however, is violated, but with a residual of size $q^{n+1}/(1-q)$.  In details,\r\n\t$$\\begin{cases}\r\n\t(A^2 + \\alpha\\cdot I)\\hat y^* = e_1 + \\frac{q^{n+1}}{1-q}\\cdot e_n\\\\\r\n\t(A^2 + \\alpha\\cdot I)y^* = e_1.\r\n\t\\end{cases}$$\r\n\tThis indicates that $\\hat y^* - y^* = \\frac{q^{n+1}}{1-q}\\cdot(A^2 + \\alpha I)^{-1} e_n$. Note that $\\alpha^{-1}I\\succeq (A^2 + \\alpha I)^{-1}\\succ 0$, we have the approximation error bounded by \\eqref{eqn:apxsolu-proximal}.\r\n\\end{proof}","start":43847,"end":45197},{"raw_tex":" Note that in Lemma \\ref{lemma:apxsolu-proximal}, we have chosen  $b\\propto e_1$. By the zero-chain property in Proposition \\ref{prop:zero-chain} and Lemma \\ref{lemma:subspace-prox},  we can verify that the subspaces $\\cH_y^{2k-1}$ and $\\cH_y^{2k}$ satisfy\r\n\\begin{equation}\r\n\\label{eqn:subspace-prox}\r\n\\cH_y^{2k-1}, \\cH_y^{2k} \\subseteq \\mathrm{Span}\\{b,A^2b,...,A^{2(k-1)}b\\} =\\mathrm{Span}\\{e_1, e_2, ...,e_k\\}.\r\n\\end{equation}\r\nThis implies that for both $y^{2k}$ and $y^{2k-1}$, the only possible nonzero elements are the first $k$ ones, which again implies that the lower bound of $\\|y^{2k}-y^*\\|^2$ and $\\|y^{2k-1}-y^*\\|^2$ will be similar. For simplicity, we only discuss this lower bound  for $y^{2k}$. The counterpart for $y^{2k-1}$ can be obtained in a similar way.  Therefore, we have the following estimations.\r\n","start":45203,"end":46028},{"raw_tex":"\\begin{lemma}\r\n\t\\label{lemma:lowerbound-distance-prox}\r\n\tAssume $k\\leq \\frac{n}{2}$ and $n\\geq2\\log_q\\left(\\frac{\\alpha}{4\\sqrt{2}}\\right)$. Then\r\n\t\\begin{equation}\r\n\t\\label{eqn:lowB-dis-prox}\r\n\t\\|y^{2k}-y^*\\|^2\\geq \\frac{q^{2k}}{16}\\|y^0-y^*\\|^2\r\n\t\\end{equation}\r\n\twhere $y^0 = 0$ is the initial solution.\r\n\\end{lemma}","start":46030,"end":46349},{"raw_tex":"\\begin{proof}\r\n\tBy the subspace characterization \\eqref{eqn:subspace-prox}, we have\r\n\t$$\\|y^{2k}-\\hat y^*\\| \\geq \\sqrt{\\sum^n_{j = k+1} (\\hat y^*_j)^2} = \\frac{q^{k}}{1-q}\\sqrt{q^2 + q^4 + \\cdots + q^{2(n-k)}}\\geq \\frac{q^k}{\\sqrt{2}}\\|\\hat y^*\\|= \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|,$$\r\n\twhere the last inequality is due to the fact that $q< 1$, $k\\leq \\frac{n}{2}$, and $y^0 = 0$.\r\n If we choose $n$ to be large enough, then $\\hat y^*$ and $y^*$ can be made arbitrarily close to each other. Hence we can transform the above inequality to  \\eqref{eqn:lowB-dis-prox}. More details of this derivation can be found in Appendix \\ref{appendix-lowerbound-distance-prox}.\r\n\\end{proof}","start":46351,"end":47030},{"raw_tex":"Using Lemma \\ref{lemma:lowerbound-distance-prox} and \\eqref{defn:primal-gap-LB}, it is then straightforward to lower bound the duality gap by\r\n$$\\Delta(x^{2k},y^{2k})\\geq q^{2k}\\cdot\\frac{\\mu_y\\|y^*-y^0\\|^2}{32}.$$\r\n","start":47034,"end":47250},{"raw_tex":"\\begin{theorem}\r\n\t\\label{thm:proximal}\r\n Let the positive parameters $\\mu_x,\\mu_y>0$ and $L_{xy}> 0$ be given. For any integer $k$, there exists a problem instance from $\\cB(L_{xy},\\mu_x,\\mu_y)$ of form \\eqref{prob:worst-proximal}, with $n\\geq \\max\\left\\{2\\log_q\\left(\\frac{\\mu_x\\mu_y}{\\sqrt{2}L_{xy}^2}\\right), 4k\\right\\}$, where $A\\in\\mathbb{R}^{n\\times n}$ as defined in \\eqref{defn:A}, and $b = -\\frac{L_{xy}^2}{4\\mu_x}e_1$. For such a problem instance, any approximate solution $(\\tilde x^k, \\tilde y^k)\\in\\cH_x^k\\times\\cH_y^k$ generated by the proximal algorithm class under the linear span assumption \\eqref{eqn:Prox-AlgoClass}  satisfies\r\n\t\\begin{eqnarray}\r\n\t\\max_y F(\\tilde x^{k},y) - \\min_{x} F(x, \\tilde y^k)\\geq q^{k}\\cdot\\frac{\\mu_y\\|y^*-y^0\\|^2}{32} \\quad\\mbox{ and }\\quad \\|\\tilde y^k-y^*\\|^2\\geq q^{k}\\cdot\\frac{\\|y^*-y^0\\|^2}{16},\r\n\t\\end{eqnarray}\r\n\twhere $q = 1+\\frac{2\\mu_x\\mu_y}{L_{xy}^2} - 2\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}$.\r\n\\end{theorem}","start":47304,"end":48312},{"raw_tex":"\\begin{proposition}\r\n\t\\label{remark:complexity-proximal}\r\n\tUnder the same set of assumptions of Theorem \\ref{thm:proximal}, if we require the duality gap to be bounded by $\\epsilon$, the number of iterations needed is at least\r\n\t\\begin{equation}\r\n\t\\label{eqn:lowerbound-proximal-1stAlgo}\r\n\tk\\geq \\ln\\left(\\frac{\\mu_y\\|y^*-y^0\\|^2}{32\\epsilon}\\right)/\\ln(q^{-1}) = \\Omega\\left(\\sqrt{\\frac{L_{xy}^2}{\\mu_x\\mu_y}}\\cdot\\ln\\left(\\frac{1}{\\epsilon}\\right)\\right).\r\n\t\\end{equation}\r\n\\end{proposition}","start":48314,"end":48807},{"raw_tex":"For the proof of this theorem, we only need to construct the orthogonal matrices $U,V$ such that when the algorithm $\\cA$ is applied to $F_{U,V}$, the subspace inclusion argument $\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1}$ and $(\\tilde x^k,\\tilde y^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1}$ holds. As a result, \r\n\\begin{eqnarray*}\r\n\t\\|\\tilde y^k-V^\\top  y^*\\|^2 = \\|V\\tilde y^k- y^*\\|^2\\geq\\min_{y\\in\\cH_y^{4k+1}}\\|y-y^*\\|^2\\geq \\frac{q^{4k+2}}{16}\\|y^*-y^0\\|^2.\r\n\\end{eqnarray*}\r\nWith this argument, the latter results follow directly from the discussion of Theorem \\ref{thm:proximal}. The proof of this theorem is presented in Appendix \\ref{appdx:gen-prox}.\r\n","start":51431,"end":52140},{"raw_tex":"\\section{Proof of Lemma \\ref{lemma:lowerbound-distance-prox}}\r\n\\label{appendix-lowerbound-distance-prox}\r\nBy the subspace characterization \\eqref{eqn:subspace-prox}, we have\r\n$$\\|y^{2k}-\\hat y^*\\| \\geq \\sqrt{\\sum^n_{j = k+1} (\\hat y^*_j)^2} = \\frac{q^{k}}{1-q}\\sqrt{q^2 + q^4 + \\cdots + q^{2(n-k)}}\\geq \\frac{q^k}{\\sqrt{2}}\\|\\hat y^*\\|= \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|,$$\r\nwhere the last inequality is due to the fact that $q\\leq 1, k\\leq \\frac{n}{2}$ and $y^0 = 0$. Note that by Lemma \\ref{lemma:apxsolu-proximal}, if we require $n\\geq 2\\log_q\\left(\\frac{\\alpha}{4\\sqrt{2}}\\right)$, then we can guarantee that \\begin{equation}\r\n\\label{eqn:err-n-proximal}\r\n\\|\\hat y^* - y^*\\| \\leq \\frac{q^{n+1}}{\\alpha(1-q)}\\leq  \\frac{q^{\\frac{n}{2}}}{\\alpha}\\cdot q^k\\cdot\\frac{q}{(1-q)} \\leq \\frac{1}{4}\\cdot \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|\\quad\\mbox{ for }\\quad \\forall 1\\leq k\\leq n/2,\r\n\\end{equation}\r\nwhere the last inequality is due to $\\frac{q^{\\frac{n}{2}}}{\\alpha}\\leq\\frac{1}{4\\sqrt{2}}$ and $q/(1-q)\\leq \\|y^0-\\hat y^*\\|$. Therefore, we have\r\n\\begin{eqnarray}\r\n\\label{eqn:dist-lower-proximal}\r\n\\|y^{2k}-y^*\\|^2 & \\geq & (\\|y^{2k}-\\hat y^*\\|-\\|\\hat y^*- y^*\\|)^2\\\\\r\n& \\geq & \\|y^{2k}-\\hat y^*\\|^2 - 2\\|y^{2k}-\\hat y^*\\|\\|\\hat y^*- y^*\\|\\nonumber\\\\\r\n& \\geq & \\min_t\\left\\{t^2 - 2\\|\\hat y^*- y^*\\|t: t\\geq \\delta_k:= \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|\\right\\}\\nonumber\\\\\r\n& = & \\delta_k(\\delta_k - 2\\|\\hat y^*- y^*\\|)\\nonumber\\\\\r\n& \\geq & \\frac{1}{2}\\delta_k^2 = \\frac{q^{2k}}{4}\\|y^0-\\hat y^*\\|^2, \\nonumber\r\n\\end{eqnarray}\r\nwhere the fourth line is due to that $d(t^2 - 2\\|\\hat y^*-y^*\\|t)/dt = 2(t-\\|\\hat y^*-y^*\\|)\\geq0$ when $t\\geq\\delta_k$. Hence the quadratic function is monotonically increasing in the considered interval. In addition, we also have\r\n$$\\|y^0-y^*\\| \\leq \\|y^0-\\hat y^*\\| + \\|\\hat y^*-y^*\\|\\leq \\|y^0-\\hat y^*\\| + \\frac{q^n}{\\alpha}\\cdot\\frac{q}{1-q}\\leq (1 + q^n/\\alpha)\\|y^0-\\hat y^*\\|\\leq 2\\|y^0-\\hat y^*\\|,$$\r\nwhere the third inequality is due to that $\\|y^0-\\hat y^*\\|\\geq \\hat y^*_1 = q/(1-q)$. For the last inequality, if $\\alpha\\geq 1$, then $q^n/\\alpha<1$; if $\\alpha\\leq 1$, then  $q^n/\\alpha\\leq \\alpha/32\\leq1$ since $n\\geq 2\\log_q\\left(\\frac{\\alpha}{4\\sqrt{2}}\\right)$. Combining the above two inequalities,  the desired bound \\eqref{eqn:lowB-dis-prox} follows.\r\n\\section{Proof of Proposition \\ref{remark:complexity-proximal}}\r\n\\label{appendix:r-complexity-proximal}\r\nHere we only prove the last inequality of \\eqref{eqn:lowerbound-proximal-1stAlgo}. Due to the fact that $(\\ln(1+z))^{-1}\\geq 1/z$ for $\\forall z>0$, we know\r\n\\begin{eqnarray*}\r\n(\\ln(q^{-1}))^{-1} & = & (\\ln(1 + (1-q)/q))^{-1} \\geq \\frac{q}{1-q}\\\\\r\n& = &   \\frac{1+\\frac{2\\mu_x\\mu_y}{L_{xy}^2} - 2\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}}{2\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}-\\frac{2\\mu_x\\mu_y}{L_{xy}^2}}\\\\\r\n& = & \\frac{\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}-\\frac{\\mu_x\\mu_y}{L_{xy}^2}}{\\frac{2\\mu_x\\mu_y}{L_{xy}^2}}\\\\\r\n& = &  \\half\\sqrt{\\frac{L_{xy}^2}{\\mu_x\\mu_y} + 1} - \\frac{1}{2},\\\\\r\n& = & \\Omega\\left(\\sqrt{\\frac{L_{xy}^2}{\\mu_x\\mu_y}}\\right)\r\n\\end{eqnarray*}\r\nwhich completes the proof.\r\n\r\n","start":75790,"end":79022},{"raw_tex":"\\begin{proof}\r\n\t\tTo prove this theorem, we only need to show\r\n\t\t$$\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1}\\qquad\\mbox{and}\\qquad(\\tilde x^k,\\tilde y^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1}.$$\r\n\t\tWe separate the proof into two parts.\r\n\t\t\r\n\t\t\\textbf{Part I. There exist orthogonal matrices $\\hat U$, $\\hat V$ s.t. when $\\cA$ is applied to the rotated instance $F_{\\hat U,\\hat V}$, $\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq \\hat U^\\top\\cH_x^{4k-1}\\times \\hat V^\\top\\cH_y^{4k-1}.$}\r\n\t\r\n\t\tLet $\\theta = (L_{xy},\\mu_x,\\mu_y)$ be the set of algorithmic parameters. To prove the result, let us construct the worst-case function $F_{U,V}$ in a recursive way. \t \\\\\r\n\t\t\\textbf{Case $k = 1$:} Let us define $U_0 = V_0 = I$. When $\\cA$ is applied to the function $F_{U_0,V_0}\\in\\cB(L_{xy}, \\mu_x, \\mu_y)$, the iterate sequence is \r\n\t\t$(x_{0}^0, y_{0}^0) = (0, 0)$ and \r\n\t\t$$\\begin{cases}\r\n\t\tu^1_0 = \\cA_u^1(\\theta; x_0^0, U_0^\\top A V_0 y_0^0), \\qquad (x^1_0, \\tilde x^1_0) = \\cA_x^1(\\theta; x^0_0,  U_0^\\top A V_0 y_0^0, \\mathbf{prox}_{\\gamma_1f}(u^1_0)),\\\\\r\n\t\tv^1_0 = \\cA_v^1(\\theta; y_0^0, V^\\top_0 A U_0x_0^0),\\qquad\r\n\t\t(y^1_0, \\tilde y^1_0) = \\cA_y^1(\\theta; y^0_0, V^\\top_0 A U_0x_0^0, \\mathbf{prox}_{\\sigma_1g}(v^1_0)).\r\n\t\t\\end{cases}$$\r\n\t\tBy Lemma \\ref{lemma:yyx}, there exists orthogonal matrices $\\Gamma_x^0$ and $\\Gamma_y^0$ such that \r\n\t\t$\\Gamma_x^0x^1_0\\in\\cH_x^3 = \\mathrm{Span}\\{Ab\\}$, $\\Gamma_y^0y^1_0\\in\\cH_y^3=\\mathrm{Span}\\{b, A^2b\\}$, and $\\Gamma_y^0b = (\\Gamma_y^0)^\\top b = b.$ That is \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-1}\r\n\t\tx_0^1\\in U_1^\\top\\cH_x^3,\\qquad\\mbox{and}\\qquad y_0^1\\in V_1^\\top\\cH_y^3, \\quad V_1 b = V_1^\\top b = b,\r\n\t\t\\end{equation}\t\r\n\t\twhere $U_1 = U_0\\Gamma_x^0$ and $V_1 = V_0\\Gamma_y^0$. \r\n\t\r\n\t\tNow we prove that when we apply the algorithm $\\cA$ to $F_{U_1,V_1}$, the generated iterates $\\{(x^0_1, y^0_1), (x^1_1, y^1_1)\\}$ satisfy that\r\n\t\t$(x^0_1, y^0_1) = (0, 0)$ and  $(x^1_1, y^1_1) = (x^1_0, y^1_0)$. That is, the first two iterates generated by $\\cA$ is completely the same for $F_{U_0,V_0}$ and $F_{U_1,V_1}$. The reason is because \r\n\t\t$u^1_1 = \\cA_u^1(\\theta; x^0_1, U_1^\\top AV_1y^0_1) = \\cA_u^1(\\theta; 0, 0) = \\cA_u^1(\\theta; x^0_0, U_0^\\top AV_0y^0_0)  = u^1_0$, \r\n\t\ttherefore \r\n\t\t\\begin{eqnarray*}\r\n\t\t\t(x^1_1, \\tilde x^1_1) & = & \\cA_x^1(\\theta; x^0_1, U_1^\\top A V_1 y_1^0, \\mathbf{prox}_{\\gamma_1f}(u^1_1)) \\\\\r\n\t\t\t& = & \\cA_x^1(\\theta; 0, 0, \\mathbf{prox}_{\\gamma_1f}(u^1_1)) \\\\\r\n\t\t\t& = & \\cA_x^1(\\theta; x^0_0, U_0^\\top A V_0 y_0^0, \\mathbf{prox}_{\\gamma_1f}(u^1_0)) \\\\\r\n\t\t\t& = & (x^1_0, \\tilde x^1_0).\r\n\t\t\\end{eqnarray*}\r\n\t\tThrough similar argument, we know $(y^1_1,\\tilde y^1_1) = (y^1_0,\\tilde y^1_0)$. Therefore, \\eqref{thm:gen-prox-1} indicates that \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-2}\r\n\t\tx_1^1\\in U_1^\\top\\cH_x^3,\\qquad\\mbox{and}\\qquad y_1^1\\in V_1^\\top\\cH_y^3, \\quad V_1 b = V_1^\\top b = b\\in V_1^\\top\\cH_y^3.\r\n\t\t\\end{equation}\r\n\t\t\\textbf{Case $k=2$.} For the ease of the readers to follow, we perform one extra step of discussion for $k=2$, before presenting the construction on general $k$. \r\n\t    \r\n\t\tFor the problem instance $F_{U_1,V_1}$, the iterates generated by $\\cA$ are $(x_{1}^0, y_{1}^0) = (0, 0)$ and \r\n\t\t$$\\begin{cases}\r\n\t\tu^1_1 = \\cA_u^1(\\theta; x_1^0, U_1^\\top A V_1 y_1^0), \\qquad (x^1_1, \\tilde x^1_1) = \\cA_x^1(\\theta; x^0_1,  U_1^\\top A V_1 y_1^0, \\mathbf{prox}_{\\gamma_1f}(u^1_1)),\\\\\r\n\t\tv^1_1 = \\cA_v^1(\\theta; y_1^0, V^\\top_1 A U_1x_1^0),\\qquad\r\n\t\t(y^1_1, \\tilde y^1_1) = \\cA_y^1(\\theta; y^0_1, V^\\top_1 A U_1x_1^0, \\mathbf{prox}_{\\sigma_1g}(v^1_1)).\\\\\r\n\t\t\\end{cases}$$\r\n\t\t$$\\begin{cases}\r\n\t\tu^2_1 = \\cA_u^2(\\theta; x_1^0, U_1^\\top A V_1 y_1^0, x_1^1, U_1^\\top A V_1 y_1^1), \\quad (x^2_1, \\tilde x^2_1) = \\cA_x^2(\\theta; x^0_1,  U_1^\\top A V_1 y_1^0, x_1^1, U_1^\\top A V_1 y_1^1, \\mathbf{prox}_{\\gamma_2f}(u^2_1)),\\\\\r\n\t\tv^2_1 = \\cA_v^2(\\theta; y_1^0, V^\\top_1 A U_1x_1^0, y_1^1, V^\\top_1 A U_1x_1^1),\\quad\r\n\t\t(y^2_1, \\tilde y^2_1) = \\cA_y^2(\\theta; y^0_1, V^\\top_1 A U_1x_1^0, y_1^1, V^\\top_1 A U_1x_1^1, \\mathbf{prox}_{\\sigma_2g}(v^2_1)).\\\\\r\n\t\t\\end{cases}$$\r\n\t\tNote that  $x_1^1 \\in U_1^\\top\\cH_x^3 \\subsetneqq U_1^\\top\\cH_x^5\\subsetneqq U_1^\\top\\cH_x^7$ and $\\{y_1^1,b\\}\\subsetneqq V_1^\\top\\cH_y^3\\subsetneqq V_1^\\top\\cH_y^5\\subsetneqq V_1^\\top\\cH_y^7$. Therefore, there exist orthogonal matrices $\\Gamma_x^1$ and $\\Gamma_y^1$ such that \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-3}\r\n\t\t\\begin{cases}\r\n\t\t\\Gamma_x^1x = (\\Gamma_x^1)^\\top x = x, \\,\\,\\forall x\\in U_1^\\top\\cH_x^5,\\,\\,\\,\\, \\Gamma_x^1 x_1^2\\in U_1^\\top\\cH_x^7,\\\\\r\n\t\t\\Gamma_y^1y \\,= (\\Gamma_y^1)^\\top y = y, \\,\\,\\,\\forall y\\in V_1^\\top\\cH_y^5,\\,\\,\\,\\, \\Gamma_y^1 y_1^2\\in V_1^\\top\\cH_y^7.\r\n\t\t\\end{cases}\r\n\t\t\\end{equation}\r\n\t\tNow, let us define \r\n\t\t$$U_2 = U_1\\Gamma_x^1\\qquad\\mbox{and}\\qquad V_2 = V_1\\Gamma_y^1.$$\r\n\t\tNow we prove that if $\\cA$ is applied to $F_{U_2,V_2}$, the generated iterates $\\{(x_{2}^0, y_{2}^0), (x_{2}^1, y_{2}^1), (x_{2}^2, y_{2}^2)\\}$ satisfy $(x_{2}^0, y_{2}^0) = (0, 0)$, $(x_{2}^1, y_{2}^1) = (x_{1}^1, y_{1}^1)$, and  $(x_{2}^2, y_{2}^2) = (x_{1}^2, y_{1}^2)$. The argument for $(x_{2}^1, y_{2}^1) = (x_{1}^1, y_{1}^1)$ is almost the same as  that of the case $k=1$. We only provide the proof for $(x_{2}^2, y_{2}^2) = (x_{1}^2, y_{1}^2)$.\r\n\t\r\n\tNext, we need to show $u_2^2 = u_1^2$, which can be proved by arguing that all the inputs to $\\cA_u^2$ are the same for both $u_2^2$ and $u_1^2$. First, it is straightforward that $x_1^0 = 0 = x^0_2, U_1^\\top AV_1 y_1^0 = 0 = U_2^\\top AV_2 y_2^0$. By previous argument $x_2^1 = x_1^1$.  Finally, consider the last input $U_2^\\top AV_2 y_2^1$, because $y_2^1 = y_1^1\\in V_1^\\top\\cH_y^3\\subsetneqq V_1^\\top\\cH_y^5$, we have $\\Gamma_y^1 y_2^1 = y_2^1 = y_1^1\\in V_1^\\top\\cH_y^3.$ Then $V_2y_2^1 = V_1\\Gamma_y^1y_2^1\\in V_1V_1^\\top\\cH_y^3 = \\cH_y^3.$ Therefore $U_1^\\top AV_2y_2^1\\in U_1^\\top A\\cH_y^3 = U_1^\\top\\cH_x^5$ and \r\n\t\t$$U_2^\\top AV_2y_2^1 = \\Gamma_x^1U_1^\\top AV_2y_2^1 = U_1^\\top AV_2y_2^1 = U_1^\\top AV_1\\Gamma_y^1y_2^1 = U_1^\\top AV_1y_1^1.$$ \r\n\t\tConsequently, \r\n\t\t\\begin{eqnarray*}\r\n\t\t\tu_2^2  = \\cA_u^2(\\theta; x_2^0, U_2^\\top AV_2 y_2^0, x_2^1, U_2^\\top AV_2 y_2^1)\r\n\t\t\t=\\cA_u^2(\\theta; x_1^0, U_1^\\top AV_1 y_1^0, x_1^1, U_1^\\top AV_1 y_1^1)\r\n\t\t\t= u_2^1\r\n\t\t\\end{eqnarray*}\r\n\t\tand \r\n\t\t\\begin{eqnarray*}\r\n\t\t\t(x_2^2,\\tilde x_2^2)  & = & \\cA_x^2(\\theta; x_2^0, U_2^\\top AV_2 y_2^0, x_2^1, U_2^\\top AV_2 y_2^1,\\mathbf{prox}_{\\gamma_2f}(u^2_2))\\\\\r\n\t\t\t& = &\\cA_x^2(\\theta; x_1^0, U_1^\\top AV_1 y_1^0, x_1^1, U_1^\\top AV_1 y_1^1,\\mathbf{prox}_{\\gamma_2f}(u^2_1))\\\\\r\n\t\t\t& = & (x_1^2,\\tilde x_1^2).\r\n\t\t\\end{eqnarray*}\r\n\t\tThrough a similar argument, we have $(y_2^2,\\tilde y_2^2) = (y_1^2,\\tilde y_1^2)$. By \\eqref{thm:gen-prox-2} and \\eqref{thm:gen-prox-3}, we have \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-4}\r\n\t\t\\{x_2^0,x_2^1,x_2^2\\}\\in U_2^\\top\\cH_x^7\\qquad\\mbox{and}\\qquad\\{b,y_2^0,y_2^1,y_2^2\\}\\in V_2^\\top\\cH_y^7.\r\n\t\t\\end{equation}\r\n\t\t\\textbf{Case $k$.} Suppose we already have orthogonal matrices $U_{k-1}, V_{k-1}$, such that when $\\cA$ is applied to $F_{U_{k-1},V_{k-1}}$, we have\r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-5}\r\n\t\t\\{x_{k-1}^0,x_{k-1}^1,\\cdots, x_{k-1}^{k-1}\\}\\in U_{k-1}^\\top\\cH_x^{4k-5}\\qquad\\mbox{and}\\qquad\\{b,y_{k-1}^0,y_{k-1}^1,\\cdots,y_{k-1}^{k-1}\\}\\in V_{k-1}^\\top\\cH_y^{4k-5}.\r\n\t\t\\end{equation}\r\n\t\tAgain, by Lemma \\ref{lemma:yyx}, there exist orthogonal matrices $\\Gamma_x^{k-1}$ and $\\Gamma_y^{k-1}$, such that \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-6}\r\n\t\t\\begin{cases}\r\n\t\t\\Gamma_x^{k-1}x = (\\Gamma_x^{k-1})^\\top x = x, \\,\\,\\forall x\\in U_{k-1}^\\top\\cH_x^{4k-3},\\,\\,\\,\\, \\Gamma_x^{k-1} x_{k-1}^{k}\\in U_{k-1}^\\top\\cH_x^{4k-1},\\\\\r\n\t\t\\Gamma_y^{k-1}y \\,= (\\Gamma_y^{k-1})^\\top y = y, \\,\\,\\,\\forall y\\in V_{k-1}^\\top\\cH_y^{4k-3},\\,\\,\\,\\, \\Gamma_y^{k-1} y_{k-1}^k\\in V_{k-1}^\\top\\cH_y^{4k-1}.\r\n\t\t\\end{cases}\r\n\t\t\\end{equation}\r\n\t\tNow we define that \r\n\t\t$$U_k = U_{k-1}\\Gamma_x^{k-1} \\qquad\\mbox{and}\\qquad V_k = V_{k-1}\\Gamma_y^{k-1}.$$\r\n\t\tTherefore, similar to our previous discussion, we only need to argue that when $\\cA$ is applied to $F_{U_k,V_k}$, the generated iterates $\\{(x_k^0,y_k^0), (x_k^1,y_k^1),\\cdots,(x_k^k,y_k^k)\\}$ satisfy $(x_k^i,y_k^i) = (x_{k-1}^i,y_{k-1}^i)$ for $i = 0,1,...,k$. We prove this argument by induction. First, it is straightforward that $(x_k^0,y_k^0) = (0,0) = (x_{k-1}^0,y_{k-1}^0)$. Suppose $(x_k^i,y_k^i) = (x_{k-1}^i,y_{k-1}^i)$ holds for $i = 0,1,...,j-1\\leq k-1$, now we prove $(x_k^j,y_k^j) = (x_{k-1}^j,y_{k-1}^j)$, which is almost identical to the case $k=2$.\r\n\t\r\n\tFor any $i\\in\\{0,1,...,j-1\\}$, let us show $U_{k-1}^\\top AV_{k-1} y_{k-1}^i = U_k^\\top AV_k y_k^i$. Because $y_k^i = y_{k-1}^i\\in V_{k-1}^\\top\\cH_y^{4k-5}\\subsetneqq V_{k-1}^\\top\\cH_y^{4k-3}$, we have $\\Gamma_y^{k-1} y_k^i = y_k^i = y_{k-1}^i\\in V_{k-1}^\\top\\cH_y^{4k-5}.$ Then $V_ky_k^i = V_{k-1}\\Gamma_y^{k-1}y_k^i\\in V_{k-1}V_{k-1}^\\top\\cH_y^{4k-5} = \\cH_y^{4k-5}.$ Therefore $U_{k-1}^\\top AV_{k}y_k^i\\in U_{k-1}^\\top A\\cH_y^{4k-5} = U_{k-1}^\\top\\cH_x^{4k-3}$ and \r\n\t\t$$U_k^\\top AV_ky_k^i = (\\Gamma_x^{k-1})^\\top U_{k-1}^\\top AV_ky_k^i = U_{k-1}^\\top AV_{k}y_k^i = U_{k-1}^\\top AV_{k-1}\\Gamma_y^{k-1}y_k^i = U_{k-1}^\\top AV_{k-1}y_{k-1}^i,$$ \r\n\t\tfor $0\\leq i\\leq j-1$. Consequently, \r\n\t\t\\begin{eqnarray*}\r\n\t\t\tu_k^i  & = & \\cA_u^i(\\theta; x_k^0, U_k^\\top AV_k y_k^0,..., x_k^{i-1}, U_k^\\top AV_k y_k^{i-1})\\\\\r\n\t\t\t& = & \\cA_u^i(\\theta; x_{k-1}^0, U_{k-1}^\\top AV_{k-1} y_{k-1}^0,..., x_{k-1}^{i-1}, U_{k-1}^\\top AV_{k-1} y_{k-1}^{i-1})\\\\\r\n\t\t\t& = & u_{k-1}^i\r\n\t\t\\end{eqnarray*}\r\n\t\tand \r\n\t\t\\begin{eqnarray*}\r\n\t\t\t(x_k^i,\\tilde x_k^i)  & = & \\cA_x^i(\\theta; x_k^0, U_k^\\top AV_k y_k^0,..., x_k^{i-1}, U_k^\\top AV_k y_k^{i-1},\\mathbf{prox}_{\\gamma_if}(u^i_k))\\\\\r\n\t\t\t& = &\\cA_x^2(\\theta; x_{k-1}^0, U_{k-1}^\\top AV_{k-1} y_{k-1}^0,..., x_{k-1}^{i-1}, U_{k-1}^\\top AV_{k-1} y_{k-1}^{i-1},\\mathbf{prox}_{\\gamma_if}(u^i_{k-1}))\\\\\r\n\t\t\t& = & (x_{k-1}^i,\\tilde x_{k-1}^i).\r\n\t\t\\end{eqnarray*}\r\n\t\tThrough a similar argument, we have $(y_k^i,\\tilde y_k^i) = (y_{k-1}^i,\\tilde y_{k-1}^i)$. By induction, we know $(y_k^i,\\tilde y_k^i) = (y_{k-1}^i,\\tilde y_{k-1}^i)$ for $i = 0,1,...,k$. Consequently, we have \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-7}\r\n\t\t\\{x_{k}^0,x_{k}^1,\\cdots, x_{k}^{k}\\}\\in U_{k}^\\top\\cH_x^{4k-1}\\qquad\\mbox{and}\\qquad\\{b,y_{k}^0,y_{k}^1,\\cdots,y_{k}^{k}\\}\\in V_{k}^\\top\\cH_y^{4k-1}.\r\n\t\t\\end{equation}\r\n\t    By setting $\\hat U = U_k$ and $\\hat V = V_k$, we prove the result for Part I.\\vspace{0.5cm}\\\\       \r\n       \t\\textbf{Part II. There exist orthogonal matrices $U$, $V$ such that when $\\cA$ is applied to the rotated instance $F_{U,V}$, $\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1},$ and $(\\tilde{x}^k,\\tilde{y}^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1}$}.\r\n       \r\n       Given the result of Part I, and let $\\{(x_k^0, y_k^0),...,(x_k^k, y_k^k)\\}$ and $(\\tilde x_k^k, \\tilde y_k^k)$ be generated by $\\cA$ when applied to $F_{\\hat U,\\hat V} = F_{U_k,V_k}$. Therefore, by Lemma \\ref{lemma:yyx}, there exist orthogonal matrices $P,Q$ such that \r\n       \\begin{equation}\r\n       \\label{thm:gen-prox-8}\r\n       \\begin{cases}\r\n       Px = P^\\top x = x, \\,\\,\\forall x\\in U_{k}^\\top\\cH_x^{4k-1},\\,\\,\\,\\, P \\tilde x_{k}^{k}\\in U_{k}^\\top\\cH_x^{4k+1},\\\\\r\n       Qy \\,=Q^\\top y = y, \\,\\,\\forall y\\in V_{k}^\\top\\cH_y^{4k-1},\\,\\,\\,\\, Q\\tilde y_{k}^k\\in V_{k}^\\top\\cH_y^{4k+1}.\r\n       \\end{cases}\r\n       \\end{equation}\r\n       Define $U = U_kP$, and $V = V_kQ$. Let $\\{(x^0,y^0),...,(x^k,y^k)\\}$ and the output $(\\tilde x^k,\\tilde y^k)$ be generated by $\\cA$ when applied to $F_{{U,V}}$. Then following the same line of argument of Case $k$, Part I, we have \r\n       $$(x^i,y^i) = (x^i_k,y^i_k), \\,\\,\\mbox{for}\\,\\, i = 0,1,...,k\\qquad\\mbox{and}\\qquad(\\tilde x^k,\\tilde y^k) = (\\tilde x^k_k,\\tilde y^k_k).$$\r\n       Therefore, combining \\eqref{thm:gen-prox-8}, we complete the proof of Part II.  \r\n\\end{proof}","start":79656,"end":91544}]},"nodes":[{"id":"S001","semantic_type":"definition","kind":"definition","labels":["defn:bilinear-class","prob:bilinear"],"declared_external":false,"source_only":false,"title":"双线性问题类","source_units":[{"raw_tex":"\\begin{definition}[Bilinear  class $\\cB(L_{xy},\\mu_x,\\mu_y)$]\r\n\t\\label{defn:bilinear-class}\r\n\tIn this special class, the problems are written as\r\n\t\\begin{equation}\r\n\t\\label{prob:bilinear}\r\n\t\\min_x \\max_y  F(x,y):=f(x) + x^\\top Ay - g(y),\r\n\t\\end{equation}\r\n%\tWe require\r\nwhere $f(x)$ and $g(y)$ are both lower semi-continuous with $f(x)$ being $\\mu_x$-strongly convex and $g(y)$ being $\\mu_y$-strongly convex. The coupling matrix $A$ satisfies $\\|A\\|_2\\leq L_{xy}$.\r\n\\end{definition}\r\n","start":9313,"end":9797}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":["defn:prox-fg"],"declared_external":false,"source_only":false,"title":"近端预言机","source_units":[{"raw_tex":"For this special model class $\\cB(L_{xy},\\mu_x,\\mu_y)$, %subclass of problems,\r\nwe assume the availability of the following prox-operations:\r\n\\begin{equation}\r\n\\label{defn:prox-fg}\r\n\\prox_{\\gamma f}(v):=\\argmin_x f(x) + \\frac{1}{2\\gamma}\\|x-v\\|^2 \\quad \\mbox{ and } \\quad\\prox_{\\sigma g}(u):=\\argmin_y g(y) + \\frac{1}{2\\sigma}\\|y-u\\|^2.\r\n\\end{equation}\r\n","start":9797,"end":10151}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":["defn:primal-func"],"declared_external":false,"source_only":false,"title":"原始函数与对偶函数","source_units":[{"raw_tex":"\\subsection{Primal function, dual function, and the duality gap}\r\nFirst, we define $\\Phi(\\cdot)$ to be the primal function and $\\Psi(\\cdot)$ to be the dual function of the saddle point problem $\\min_x \\max_y F(x,y)$, respectively, with the following definitions\r\n\\begin{equation}\r\n\\label{defn:primal-func}\r\n\\Phi(x):=\\max_y F(x,y)\\qquad\\mbox{ and }\\qquad\\Psi(y):=\\min_x F(x,y).\r\n\\end{equation}\r\n","start":22208,"end":22602}]},{"id":"S004","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"值函数的强凸凹性","source_units":[{"raw_tex":"As the maximum of a class of $\\mu_x$-strongly convex function, we know $\\Phi(x)$ is a $\\mu_x$-strongly convex function. Similarly, $\\Psi(y)$ is a $\\mu_y$-strongly concave function. We define the duality gap as\r\n","start":22602,"end":22813}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"对偶间隙","source_units":[{"raw_tex":"$$\\Delta(x,y) := \\max_{y'} F(x,y')-\\min_{x'} F(x',y) = \\Phi(x)-\\Psi(y).$$\r\n","start":22813,"end":22888}]},{"id":"S006","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"鞍点处的值相等","source_units":[{"raw_tex":"Suppose the unique solution of this min-max problem is $(x^*,y^*)$. By the strong duality theorem, we know for any $x$ and $y$ it holds that \r\n$$\\Phi(x)\\geq\\min_{x'}\\Phi(x') = \\Phi(x^*) =F(x^*,y^*) = \\Psi(y^*) = \\max_{y'}\\Psi(y')\\geq \\Psi(y).$$\r\n","start":22888,"end":23134}]},{"id":"S007","semantic_type":"derived_fact","kind":"derived_fact","labels":["defn:primal-gap-LB"],"declared_external":false,"source_only":false,"title":"间隙控制解距离","source_units":[{"raw_tex":"Together with the $\\mu_x$-strong convexity of $\\Phi$ and the $\\mu_y$-strong concavity of $\\Psi$, we further have \r\n\\begin{equation}\r\n\\label{defn:primal-gap-LB}\r\n\\Delta(x,y)= \\Phi(x)-\\Phi(x^*) + \\Psi(y^*) - \\Psi(y)\\geq\\frac{\\mu_x}{2}\\|x-x^*\\|^2 + \\frac{\\mu_y}{2}\\|y-y^*\\|^2.\r\n\\end{equation}\r\n\r\n","start":23134,"end":23427}]},{"id":"S008","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"距离下界的用途","source_units":[{"raw_tex":"Now, suppose that $(\\tilde x_k,\\tilde y_k)$ is the approximate solution generated after $k$ iterations of an algorithm. Our aim %main strategy in the later discussion \r\nis to lower bound the distance between $(\\tilde x_k,\\tilde y_k)$ and $(x^*,y^*)$. By \\eqref{defn:primal-gap-LB}, this would construct a lower iteration complexity bound in terms of the duality gap as well.\r\n","start":23427,"end":23803}]},{"id":"S009","semantic_type":"algorithm","kind":"algorithm","labels":["defn:Prox-AlgoClass","eqn:Prox-AlgoClass"],"declared_external":false,"source_only":false,"title":"线性跨度近端算法类","source_units":[{"raw_tex":"\\begin{definition}[Proximal algorithm class]\r\n\\label{defn:Prox-AlgoClass}\r\nIn each iteration, the iterate sequence $\\{(x^{k},y^{k})\\}_{k=0,1,...}$ are generated so that  $(x^{k},y^{k})\\in\\cH_x^{k} \\times \\cH_y^{k}$. These subspaces are generated with $\\cH_x^{0} = \\mathrm{Span}\\{x^0\\}, \\cH_y^{0} = \\mathrm{Span}\\{y^0\\}$ and\r\n\\begin{equation}\r\n\\label{eqn:Prox-AlgoClass}\\begin{cases}\r\n\\cH_x^{k+1} := \\mathrm{Span}\\{x^i,\\prox_{\\gamma_i f}(\\hat{x}^i-\\gamma_iA\\tilde{y}^i)~\\,: \\forall \\hat{x}^i\\in\\cH_x^{i},~ \\tilde{y}^i\\in\\cH_y^i,~ 0\\leq i\\leq k\\}\\\\\r\n\\cH_y^{k+1} := \\mathrm{Span}\\{y^i,\\prox_{\\sigma_i g}(\\hat{y}^i+\\sigma_iA^\\top\\tilde{x}^i): \\forall \\tilde{x}^i\\in\\cH_x^{i},~ \\hat{y}^i\\in\\cH_y^i,~ 0\\leq i\\leq k\\}.\r\n\\end{cases}\r\n\\end{equation}\r\n\\end{definition}\r\n","start":24321,"end":25081}]},{"id":"S010","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"历史点与交替更新","source_units":[{"raw_tex":"Remark that when applying the proximal oracles, it is not necessary to use the most recent iterate $x^k$ as the proximal center. Neither is it necessary to use the gradients of the coupling term (namely the $A^\\top x$ and $Ay$ terms) at the current iterate. Instead, the algorithm class allows the usage of the combination of {\\it any}\\/ points in the historical search space. We shall also remark that the algorithm class in Definition~\\ref{defn:Prox-AlgoClass} does not necessarily need to update $x$ and $y$ at the same time, because setting $x^{k+1}=x^k$ or $y^{k+1} = y^k$ also satisfies Definition \\ref{defn:Prox-AlgoClass}. Thus this algorithm class also includes the methods that alternatingly update $x$ and $y$. Below is a sample algorithm in this class.\r\n","start":25081,"end":25847}]},{"id":"S011","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"去除跨度限制的动机","source_units":[{"raw_tex":"\\subsection{General deterministic algorithm classes without linear span structure}\r\nAlthough all the reviewed first-order methods satisfy the \\emph{linear span} property in the proximal algorithm class in Definition \\ref{defn:Prox-AlgoClass} and the pure first-order algorithm class in Definition \\ref{defn:Pure-AlgoClass}, this does not exclude the possibility of the deriving an algorithm that does not satisfy the linear span property. Therefore, we also define the general deterministic proximal algorithm class and the general deterministic pure first-order algorithm class as follows, whose iteration complexity lower bound can be generalized from their linear span counterpart through the technique of \\emph{adversary rotation}. \r\n","start":32650,"end":33388}]},{"id":"S012","semantic_type":"algorithm","kind":"algorithm","labels":["defn:Det-Prox-AlgoClass","eqn:Det-Prox-AlgoClass"],"declared_external":false,"source_only":false,"title":"一般确定性近端算法","source_units":[{"raw_tex":"\\begin{definition}[General proximal algorithm class]\r\n\t\\label{defn:Det-Prox-AlgoClass}\r\n\tConsider the problem \\eqref{prob:bilinear} in the problem class $\\cB(L_{xy},\\mu_x,\\mu_y)$, denote $\\theta = (L_{xy},\\mu_x,\\mu_y)$ as the corresponding problem parameters. Let algorithm $\\cA$ belong to the general  proximal algorithm class. Then $\\cA$ consists of a sequence of deterministic mappings $\\{(\\cA_x^1, \\cA_y^1,\\cA_u^1,\\cA_v^1),(\\cA_x^2, \\cA_y^2,\\cA_u^2,\\cA_v^2)...\\}$ such that the iterate sequence $\\{(x^{k},y^{k})\\}_{k=0,1,...}$ and the output sequence $\\{(\\tilde x^{k},\\tilde y^{k})\\}_{k=0,1,...}$ are generated by \r\n\t\\begin{equation}\r\n\t\\label{eqn:Det-Prox-AlgoClass}\\begin{cases}\r\n\t(x^k,\\tilde x^k) := \\cA_x^k\\left(\\theta; x^0,Ay^0,...,x^{k-1},Ay^{k-1}; \\mathbf{prox}_{\\gamma_k f}(u^k) \\right),\\\\\r\n\t(y^k,\\tilde y^k) := \\cA_y^k\\left(\\theta; y^0,A^\\top x^0,...,y^{k-1},A^\\top x^{k-1}; \\mathbf{prox}_{\\sigma_k g}(v^k) \\right),\r\n\t\\end{cases}\r\n\t\\end{equation} \r\n\twhere $u^k = \\cA^k_u(\\theta; x^0,Ay^0,...,x^{k-1},Ay^{k-1})$, $v^k = \\cA^k_v(\\theta; y^0,A^\\top x^0,...,y^{k-1},A^\\top x^{k-1})$, and \r\n\t$(x^0,y^0)$ is any given initial solution. \r\n\\end{definition}\r\n","start":33388,"end":34550}]},{"id":"S013","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"近端输入可非线性生成","source_units":[{"raw_tex":"One remark is that the input of the proximal mapping $\\mathbf{prox}_{\\gamma_kf}(\\cdot)$ is  constructed with other inputs to the $\\cA_x^k$, i.e., there could be another deterministic mapping $\\cA_u^k$ to generate a vector  $u^k = \\cA^k_u(\\theta; x^0,Ay^0,...,x^{k-1},Ay^{k-1})$ and then $\\mathbf{prox}_{\\gamma_kf}(u^k)$ is passed to the mapping $\\cA_x^k$. This $\\cA^k_u$ does not need to be linear. The situation for $v^k$ and $\\cA^k_v$ is similar. Similar to the general proximal algorithm class, the general pure first-order algorithm class is defined as follows. \r\n","start":34550,"end":35118}]},{"id":"S014","semantic_type":"problem_setting","kind":"problem_setting","labels":["prob:worst-proximal"],"declared_external":false,"source_only":false,"title":"二次最坏情形实例","source_units":[{"raw_tex":"Let us construct the following bilinearly coupled min-max saddle point problem:\r\n\\begin{equation}\r\n\\label{prob:worst-proximal}\r\n\\min_x\\max_y F(x,y):=\\frac{\\mu_x}{2}\\|x\\|^2 + \\frac{L_{xy}}{2}x^\\top Ay - \\frac{\\mu_y}{2}\\|y\\|^2 - b^\\top y\r\n\\end{equation}\r\n","start":37175,"end":37428}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"反对角差分矩阵","source_units":[{"raw_tex":"where $b$ is a vector to be determined later, and the coupling matrix $A$ (hence $A^2$ and $A^4$) is defined as follows: %. Its polynomials $A^2$ and $A^4$ are also shown bellow for later reference.\r\n\\begin{equation}\r\n\\label{defn:A}\r\nA = \\begin{pmatrix}\r\n& &    &     & 1\\\\\r\n& &    &  1 & -1\\\\\r\n& & 1 & -1&\\\\\r\n& \\udots &\\udots & & \\\\\r\n1& -1 & & &\r\n\\end{pmatrix}, \\,\\,\r\n","start":37428,"end":37797}]},{"id":"S016","semantic_type":"derived_fact","kind":"derived_fact","labels":["defn:A"],"declared_external":false,"source_only":false,"title":"平方矩阵的三对角形式","source_units":[{"raw_tex":"A^2 = \\begin{pmatrix}\r\n1   & -1 &            &               & \\\\\r\n-1 & 2   & -1        &              & \\\\\r\n& -1 &\\ddots & \\ddots & \\\\\r\n&      &\\ddots & 2           &-1\\\\\r\n&      &              & -1         & 2\r\n\\end{pmatrix},\r\n","start":37797,"end":38026}]},{"id":"S017","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"四次幂矩阵","source_units":[{"raw_tex":"\\,\\,\r\nA^4 = \\begin{pmatrix}\r\n2   & -3 &     1      &               &  & &\\\\\r\n-3 & 6   & -4        &     1         & & & \\\\\r\n1& -4 &6 & -4 &1 \\\\\r\n&  \\ddots     &\\ddots & \\ddots  &\\ddots & \\ddots & \\\\\r\n& & 1& -4 &6 & -4 &1\\\\\r\n& & & 1& -4 &6 & -4 \\\\\r\n& & & & 1&-4&5\r\n\\end{pmatrix}.\r\n\\end{equation}\r\n","start":38026,"end":38322}]},{"id":"S018","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"矩阵范数与实例归属","source_units":[{"raw_tex":"Note that $A^\\top =A$ and $\\|A\\|_2 \\leq 2$. Therefore \\eqref{prob:worst-proximal} is an instance in the problem class $\\cB(L_{xy},\\mu_x,\\mu_y)$. It is worth noting that the example \\eqref{prob:worst-proximal} is the same as that in Proposition 2 of \\cite{LowB-Quadratic}, which is a parallel work focused on pure first-order algorithm class. Here, we use the same example to elaborate the lower bound of the proximal methods over the general bilinear coupling class $\\cB(L_{xy},\\mu_x,\\mu_y)$, as well as a warmup for the discussion of more complex problem class $\\cF(L_{x},L_{y},L_{xy},\\mu_x,\\mu_y)$. \r\n","start":38322,"end":38925}]},{"id":"S019","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"坐标向量与直接计算说明","source_units":[{"raw_tex":"\r\nDenote $e_i$ to be the $i$-th unit vector, which has $1$ at the $i$-th component and 0 elsewhere. Then by direct calculation, one can check that $A^2$ satisfies the following \\emph{zero-chain property} (see Chapter 2 of  \\cite{nesterov2018lectures}).\r\n\r\n","start":38925,"end":39181}]},{"id":"S020","semantic_type":"proposition","kind":"proposition","labels":["prop:zero-chain"],"declared_external":false,"source_only":false,"title":"平方矩阵的零链性质","source_units":[{"raw_tex":"\\begin{proposition}[Zero-chain property]\r\n\t\\label{prop:zero-chain}\r\n\tFor any vector $v\\in\\R^n$, if $v\\in\\mathrm{Span}\\{e_i:i\\leq k\\}$ for some $1\\leq k\\leq n-1$,  then $A^2v\\in\\mathrm{Span}\\{e_i:i\\leq k+1\\}$.\r\n\\end{proposition}","start":39181,"end":39408}]},{"id":"S021","semantic_type":"derived_fact","kind":"derived_fact","labels":["defn:prox-closedForm-x"],"declared_external":false,"source_only":false,"title":"原始块近端闭式","source_units":[{"raw_tex":"For problem \\eqref{prob:worst-proximal}, the proximal operators in \\eqref{eqn:Prox-AlgoClass} can be written explicitly:\r\n\\begin{eqnarray}\r\n\\label{defn:prox-closedForm-x}\r\n\\prox_{\\gamma_i f}(\\hat{x}_i-\\gamma_iA\\tilde{y}_i) & = & \\argmin_x \\frac{\\mu_x}{2}\\|x\\|^2 + \\frac{1}{2\\gamma_i}\\left\\|x-(\\hat{x}_i-\\frac{\\gamma_iL_{xy}}{2}A\\tilde{y}_i)\\right\\|^2\\\\\r\n& = & \\frac{1}{1+\\gamma_i\\mu_x}\\hat{x}_i - \\frac{\\gamma_iL_{xy}}{2(1+\\gamma_i\\mu_x)}A\\tilde{y}_i\\nonumber\\\\\r\n& \\in & \\mathrm{Span}\\{\\hat{x}_i,A\\tilde{y}_i\\}.\\nonumber\r\n\\end{eqnarray}\r\n","start":39569,"end":40107}]},{"id":"S022","semantic_type":"derived_fact","kind":"derived_fact","labels":["defn:prox-closedForm-y"],"declared_external":false,"source_only":false,"title":"对偶块近端闭式","source_units":[{"raw_tex":"Similarly, for the $y$ block, we also have\r\n\\begin{eqnarray}\r\n\\label{defn:prox-closedForm-y}\r\n\\prox_{\\sigma_i g}(\\hat{y}_i+\\sigma_iA^\\top\\tilde{x}_i)\r\n%& = & \\argmin_y \\frac{\\mu_y}{2}\\|y\\|^2 + b^Ty + \\frac{1}{2\\sigma_i}\\left\\|y-(\\hat{y}_i+\\frac{\\sigma_iL_{xy}}{2}A^\\top \\tilde{x}_i)\\right\\|^2\\\\\r\n=  \\frac{\\hat{y}_i-\\sigma_i b}{1+\\sigma_i\\mu_y} +  \\frac{\\sigma_iL_{xy}}{2(1+\\sigma_i\\mu_x)}A\\tilde{x}_i \\in \\mathrm{Span}\\{\\hat{y}_i,A\\tilde{x}_i,b\\}.\r\n\\end{eqnarray}\r\n","start":40107,"end":40572}]},{"id":"S023","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"零初始点","source_units":[{"raw_tex":"Let us assume the initial point to be $x^0 = y^0 = 0$ ($\\cH_x^0 = \\cH_y^0 = \\{0\\}$) without loss of generality.\r\n","start":40572,"end":40685}]},{"id":"S024","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"搜索空间的正文推导","source_units":[{"raw_tex":"Directly substituting \\eqref{defn:prox-closedForm-x} and \\ref{defn:prox-closedForm-y} into Definition \\eqref{defn:Prox-AlgoClass} yields\r\n\\begin{equation*}\r\n\\begin{cases}\r\n\\cH_x^1 \\subseteq \\mathrm{Span}\\{0\\}\\\\\r\n\\cH_y^1 \\subseteq \\mathrm{Span}\\{b\\}\r\n\\end{cases}\r\n\\quad\r\n\\begin{cases}\r\n\\cH_x^2 \\subseteq \\mathrm{Span}\\{Ab\\}\\\\\r\n\\cH_y^2 \\subseteq \\mathrm{Span}\\{b\\}\r\n\\end{cases}\r\n\\quad\r\n\\begin{cases}\r\n\\cH_x^3 \\subseteq \\mathrm{Span}\\{Ab\\}\\\\\r\n\\cH_y^3 \\subseteq \\mathrm{Span}\\{b,A^2b\\}\r\n\\end{cases}\r\n\\quad\r\n\\begin{cases}\r\n\\cH_x^4 \\subseteq \\mathrm{Span}\\{Ab, A^3b\\}\\\\\r\n\\cH_y^4 \\subseteq \\mathrm{Span}\\{b,A^2b\\}\r\n\\end{cases}\r\n\\ldots\r\n\\end{equation*}\r\nWe formally summarize this observation below: %in the Lemma \\ref{lemma:subspace-prox}.\r\n","start":40734,"end":41468}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"前四轮的交错扩张","source_units":[{"raw_tex":"Directly substituting \\eqref{defn:prox-closedForm-x} and \\ref{defn:prox-closedForm-y} into Definition \\eqref{defn:Prox-AlgoClass} yields\r\n\\begin{equation*}\r\n\\begin{cases}\r\n\\cH_x^1 \\subseteq \\mathrm{Span}\\{0\\}\\\\\r\n\\cH_y^1 \\subseteq \\mathrm{Span}\\{b\\}\r\n\\end{cases}\r\n\\quad\r\n\\begin{cases}\r\n\\cH_x^2 \\subseteq \\mathrm{Span}\\{Ab\\}\\\\\r\n\\cH_y^2 \\subseteq \\mathrm{Span}\\{b\\}\r\n\\end{cases}\r\n\\quad\r\n\\begin{cases}\r\n\\cH_x^3 \\subseteq \\mathrm{Span}\\{Ab\\}\\\\\r\n\\cH_y^3 \\subseteq \\mathrm{Span}\\{b,A^2b\\}\r\n\\end{cases}\r\n\\quad\r\n\\begin{cases}\r\n\\cH_x^4 \\subseteq \\mathrm{Span}\\{Ab, A^3b\\}\\\\\r\n\\cH_y^4 \\subseteq \\mathrm{Span}\\{b,A^2b\\}\r\n\\end{cases}\r\n\\ldots\r\n\\end{equation*}\r\n","start":40734,"end":41380}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","labels":["lemma:subspace-prox"],"declared_external":false,"source_only":false,"title":"近端搜索空间引理","source_units":[{"raw_tex":"\\begin{lemma}\r\n\t\\label{lemma:subspace-prox}\r\n\tFor problem \\eqref{prob:worst-proximal}, for any $k \\in \\mathbb{N}$, if the iterates are generated so that  $(x_k,y_k)\\in\\cH_x^k\\times\\cH_y^k$, with $\\cH_x^k$ and $\\cH_y^k$ defined by \\eqref{defn:Prox-AlgoClass}, then based on \\eqref{defn:prox-closedForm-x} and \\eqref{defn:prox-closedForm-y} the search subspaces satisfy\r\n\t$$\\cH_x^k \\subseteq \\begin{cases}\r\n\t\\{0\\}, &  k = 1\\\\\r\n\t\\mathrm{Span}\\left\\{A^{2i}(Ab): 0\\leq i\\leq \\left\\lfloor \\frac{k}{2}\\right\\rfloor -1 \\right\\}, & k\\geq 2\r\n\t\\end{cases} \\quad \\mbox{ and }\\quad\r\n\t\\cH_y^k \\subseteq \\mathrm{Span}\\left\\{A^{2i}b: 0\\leq i\\leq \\left\\lceil \\frac{k}{2}\\right\\rceil -1 \\right\\}.$$\r\n\\end{lemma}","start":41468,"end":42161}]},{"id":"S027","semantic_type":"derived_fact","kind":"derived_fact","labels":["defn:dual-function-prox"],"declared_external":false,"source_only":false,"title":"实例的对偶二次函数","source_units":[{"raw_tex":"Let us lower bound the dual gap, which is upper bounded by the whole duality gap. To achieve this, let us first write down the dual function of problem \\eqref{prob:worst-proximal} as\r\n\\begin{equation}\r\n\\label{defn:dual-function-prox}\r\n\\Psi(y) = \\min_{x} F(x,y) = -\\frac{1}{2}y^\\top \\left(\\frac{L_{xy}^2}{4\\mu_x}\\cdot A^2 + \\mu_y\\cdot I\\right)y - b^\\top y . \r\n\\end{equation}\r\n","start":42258,"end":42633}]},{"id":"S028","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"驻点刻画与近似路线","source_units":[{"raw_tex":"For this $\\mu_y$-strongly concave dual function, we can characterize the optimal solution $y^*$ directly by its KKT condition $\\nabla \\Psi(y^*) = 0$. However, the exact solution $y^*$ does not have a simple and clear form, so we choose to  characterize it by an approximate solution $\\hat y^*$.\r\n","start":42633,"end":42929}]},{"id":"S029","semantic_type":"lemma","kind":"lemma","labels":["eqn:apxsolu-proximal","lemma:apxsolu-proximal"],"declared_external":false,"source_only":false,"title":"几何型近似最优解","source_units":[{"raw_tex":"\\begin{lemma}[Approximate optimal solution]\r\n\t\\label{lemma:apxsolu-proximal}\r\n\tLet us assign the value of $b$ as $b := -\\frac{L_{xy}^2}{4\\mu_x}e_1$. Denote $\\alpha := \\frac{4\\mu_x\\mu_y}{L_{xy}^2}$, and let $q = \\half\\left((2+\\alpha) - \\sqrt{(2+\\alpha)^2 - 4}\\right)\\in(0,1)$\r\n\tbe the smallest root of the quadratic equation $1 - (2+\\alpha)q + q^2 = 0$. Then, an approximate optimal solution $\\hat y^*$ can be constructed as\r\n\t\\begin{equation}\r\n\t\\hat y^*_{i} = \\frac{q^i}{1-q} \\quad\\mbox{ for } \\quad i = 1,2,...,n.\r\n\t\\end{equation}\r\n\tThe approximation error can be bounded by\r\n\t\\begin{equation}\r\n\t\\label{eqn:apxsolu-proximal}\r\n\t\\|\\hat y^* - y^*\\| \\leq  \\frac{q^{n+1}}{\\alpha(1-q)},\r\n\t\\end{equation}\r\n\twhere $\\hat y^*_i$ is the $i$-th element of $\\hat y^*$. Note that $q<1$ and the lower bound is dimension-independent, hence we are free to choose $n$ to make the approximation error arbitrarily small.\r\n\\end{lemma}","start":42931,"end":43845}]},{"id":"S030","semantic_type":"definition","kind":"definition","labels":["lemma:apxsolu-proximal"],"declared_external":false,"source_only":false,"title":"参数与收缩因子","source_units":[{"raw_tex":"\tLet us assign the value of $b$ as $b := -\\frac{L_{xy}^2}{4\\mu_x}e_1$. Denote $\\alpha := \\frac{4\\mu_x\\mu_y}{L_{xy}^2}$, and let $q = \\half\\left((2+\\alpha) - \\sqrt{(2+\\alpha)^2 - 4}\\right)\\in(0,1)$\r\n\tbe the smallest root of the quadratic equation $1 - (2+\\alpha)q + q^2 = 0$. Then, an approximate optimal solution $\\hat y^*$ can be constructed as\r\n","start":43009,"end":43356}]},{"id":"S031","semantic_type":"definition","kind":"definition","labels":["lemma:apxsolu-proximal"],"declared_external":false,"source_only":false,"title":"线性项的指定","source_units":[{"raw_tex":"\tLet us assign the value of $b$ as $b := -\\frac{L_{xy}^2}{4\\mu_x}e_1$. Denote $\\alpha := \\frac{4\\mu_x\\mu_y}{L_{xy}^2}$, and let $q = \\half\\left((2+\\alpha) - \\sqrt{(2+\\alpha)^2 - 4}\\right)\\in(0,1)$\r\n","start":43009,"end":43207}]},{"id":"S032","semantic_type":"definition","kind":"definition","labels":["lemma:apxsolu-proximal"],"declared_external":false,"source_only":false,"title":"几何坐标近似向量","source_units":[{"raw_tex":"\t\\begin{equation}\r\n\t\\hat y^*_{i} = \\frac{q^i}{1-q} \\quad\\mbox{ for } \\quad i = 1,2,...,n.\r\n\t\\end{equation}\r\n","start":43356,"end":43464}]},{"id":"S033","semantic_type":"proof","kind":"proof","labels":["defn:KKT-proximal"],"declared_external":false,"source_only":false,"title":"近似解引理完整证明","source_units":[{"raw_tex":"\\begin{proof}\r\n\tFirst, let us substitute the value of $b$ into the KKT system $\\nabla \\Psi(y^*)=0$, by slight rearranging and scaling the terms, we get\r\n\t\\begin{equation*}\r\n\t\\left(A^2 + \\frac{4\\mu_x\\mu_y}{L_{xy}^2} I\\right)y^* = - \\frac{4\\mu_x}{L_{xy}^2}b.\r\n\t\\end{equation*}\r\n\tUsing the definition of $\\alpha$ and $b$, the equation becomes\r\n\t$$(A^2+\\alpha I)y^* = e_1.$$\r\n\tSubstituting the formula of $A^2$ in \\eqref{defn:A}, we expand the above equation as\r\n\t\\begin{eqnarray}\r\n\t\\label{defn:KKT-proximal}\r\n\t\\begin{cases}\r\n\t(1+\\alpha)y_1^* - y_2^* = 1\\\\\r\n\t-y_1^* + (2+\\alpha)y_2^* - y_3^* = 0\\\\\r\n\t\\qquad\\qquad\\quad\\vdots\\\\\r\n\t-y_{n-2}^* + (2+\\alpha)y_{n-1}^* - y_n^* = 0\\\\\r\n\t-y_{n-1}^* + (2+\\alpha)y_n^* = 0.\r\n\t\\end{cases}\r\n\t\\end{eqnarray}\r\n\tBy direct calculation, we can check that $\\hat y^*$ satisfies the first $n-1$ equations of the KKT system \\eqref{defn:KKT-proximal}. The last equation, however, is violated, but with a residual of size $q^{n+1}/(1-q)$.  In details,\r\n\t$$\\begin{cases}\r\n\t(A^2 + \\alpha\\cdot I)\\hat y^* = e_1 + \\frac{q^{n+1}}{1-q}\\cdot e_n\\\\\r\n\t(A^2 + \\alpha\\cdot I)y^* = e_1.\r\n\t\\end{cases}$$\r\n\tThis indicates that $\\hat y^* - y^* = \\frac{q^{n+1}}{1-q}\\cdot(A^2 + \\alpha I)^{-1} e_n$. Note that $\\alpha^{-1}I\\succeq (A^2 + \\alpha I)^{-1}\\succ 0$, we have the approximation error bounded by \\eqref{eqn:apxsolu-proximal}.\r\n\\end{proof}","start":43847,"end":45197}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"缩放对偶驻点方程","source_units":[{"raw_tex":"\\begin{proof}\r\n\tFirst, let us substitute the value of $b$ into the KKT system $\\nabla \\Psi(y^*)=0$, by slight rearranging and scaling the terms, we get\r\n\t\\begin{equation*}\r\n\t\\left(A^2 + \\frac{4\\mu_x\\mu_y}{L_{xy}^2} I\\right)y^* = - \\frac{4\\mu_x}{L_{xy}^2}b.\r\n\t\\end{equation*}\r\n\tUsing the definition of $\\alpha$ and $b$, the equation becomes\r\n\t$$(A^2+\\alpha I)y^* = e_1.$$\r\n","start":43847,"end":44219}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":["defn:KKT-proximal"],"declared_external":false,"source_only":false,"title":"展开三对角递推","source_units":[{"raw_tex":"\tSubstituting the formula of $A^2$ in \\eqref{defn:A}, we expand the above equation as\r\n\t\\begin{eqnarray}\r\n\t\\label{defn:KKT-proximal}\r\n\t\\begin{cases}\r\n\t(1+\\alpha)y_1^* - y_2^* = 1\\\\\r\n\t-y_1^* + (2+\\alpha)y_2^* - y_3^* = 0\\\\\r\n\t\\qquad\\qquad\\quad\\vdots\\\\\r\n\t-y_{n-2}^* + (2+\\alpha)y_{n-1}^* - y_n^* = 0\\\\\r\n\t-y_{n-1}^* + (2+\\alpha)y_n^* = 0.\r\n\t\\end{cases}\r\n\t\\end{eqnarray}\r\n","start":44219,"end":44586}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"仅末坐标存在残差","source_units":[{"raw_tex":"\tBy direct calculation, we can check that $\\hat y^*$ satisfies the first $n-1$ equations of the KKT system \\eqref{defn:KKT-proximal}. The last equation, however, is violated, but with a residual of size $q^{n+1}/(1-q)$.  In details,\r\n\t$$\\begin{cases}\r\n\t(A^2 + \\alpha\\cdot I)\\hat y^* = e_1 + \\frac{q^{n+1}}{1-q}\\cdot e_n\\\\\r\n\t(A^2 + \\alpha\\cdot I)y^* = e_1.\r\n\t\\end{cases}$$\r\n","start":44586,"end":44959}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"逆矩阵控制近似误差","source_units":[{"raw_tex":"\tThis indicates that $\\hat y^* - y^* = \\frac{q^{n+1}}{1-q}\\cdot(A^2 + \\alpha I)^{-1} e_n$. Note that $\\alpha^{-1}I\\succeq (A^2 + \\alpha I)^{-1}\\succ 0$, we have the approximation error bounded by \\eqref{eqn:apxsolu-proximal}.\r\n\\end{proof}","start":44959,"end":45197}]},{"id":"S038","semantic_type":"proof","kind":"proof","labels":["eqn:subspace-prox"],"declared_external":false,"source_only":false,"title":"坐标支撑的正文推导","source_units":[{"raw_tex":" Note that in Lemma \\ref{lemma:apxsolu-proximal}, we have chosen  $b\\propto e_1$. By the zero-chain property in Proposition \\ref{prop:zero-chain} and Lemma \\ref{lemma:subspace-prox},  we can verify that the subspaces $\\cH_y^{2k-1}$ and $\\cH_y^{2k}$ satisfy\r\n\\begin{equation}\r\n\\label{eqn:subspace-prox}\r\n\\cH_y^{2k-1}, \\cH_y^{2k} \\subseteq \\mathrm{Span}\\{b,A^2b,...,A^{2(k-1)}b\\} =\\mathrm{Span}\\{e_1, e_2, ...,e_k\\}.\r\n\\end{equation}\r\nThis implies that for both $y^{2k}$ and $y^{2k-1}$, the only possible nonzero elements are the first $k$ ones, which again implies that the lower bound of $\\|y^{2k}-y^*\\|^2$ and $\\|y^{2k-1}-y^*\\|^2$ will be similar. For simplicity, we only discuss this lower bound  for $y^{2k}$. The counterpart for $y^{2k-1}$ can be obtained in a similar way.  Therefore, we have the following estimations.\r\n","start":45203,"end":46028}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:subspace-prox"],"declared_external":false,"source_only":false,"title":"对偶空间的坐标支撑","source_units":[{"raw_tex":" Note that in Lemma \\ref{lemma:apxsolu-proximal}, we have chosen  $b\\propto e_1$. By the zero-chain property in Proposition \\ref{prop:zero-chain} and Lemma \\ref{lemma:subspace-prox},  we can verify that the subspaces $\\cH_y^{2k-1}$ and $\\cH_y^{2k}$ satisfy\r\n\\begin{equation}\r\n\\label{eqn:subspace-prox}\r\n\\cH_y^{2k-1}, \\cH_y^{2k} \\subseteq \\mathrm{Span}\\{b,A^2b,...,A^{2(k-1)}b\\} =\\mathrm{Span}\\{e_1, e_2, ...,e_k\\}.\r\n\\end{equation}\r\n","start":45203,"end":45635}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"奇偶轮次共享支撑限制","source_units":[{"raw_tex":"This implies that for both $y^{2k}$ and $y^{2k-1}$, the only possible nonzero elements are the first $k$ ones, which again implies that the lower bound of $\\|y^{2k}-y^*\\|^2$ and $\\|y^{2k-1}-y^*\\|^2$ will be similar. For simplicity, we only discuss this lower bound  for $y^{2k}$. The counterpart for $y^{2k-1}$ can be obtained in a similar way.  Therefore, we have the following estimations.\r\n","start":45635,"end":46028}]},{"id":"S041","semantic_type":"lemma","kind":"lemma","labels":["eqn:lowB-dis-prox","lemma:lowerbound-distance-prox"],"declared_external":false,"source_only":false,"title":"对偶解距离下界","source_units":[{"raw_tex":"\\begin{lemma}\r\n\t\\label{lemma:lowerbound-distance-prox}\r\n\tAssume $k\\leq \\frac{n}{2}$ and $n\\geq2\\log_q\\left(\\frac{\\alpha}{4\\sqrt{2}}\\right)$. Then\r\n\t\\begin{equation}\r\n\t\\label{eqn:lowB-dis-prox}\r\n\t\\|y^{2k}-y^*\\|^2\\geq \\frac{q^{2k}}{16}\\|y^0-y^*\\|^2\r\n\t\\end{equation}\r\n\twhere $y^0 = 0$ is the initial solution.\r\n\\end{lemma}","start":46030,"end":46349}]},{"id":"S042","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"距离引理的正文证明","source_units":[{"raw_tex":"\\begin{proof}\r\n\tBy the subspace characterization \\eqref{eqn:subspace-prox}, we have\r\n\t$$\\|y^{2k}-\\hat y^*\\| \\geq \\sqrt{\\sum^n_{j = k+1} (\\hat y^*_j)^2} = \\frac{q^{k}}{1-q}\\sqrt{q^2 + q^4 + \\cdots + q^{2(n-k)}}\\geq \\frac{q^k}{\\sqrt{2}}\\|\\hat y^*\\|= \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|,$$\r\n\twhere the last inequality is due to the fact that $q< 1$, $k\\leq \\frac{n}{2}$, and $y^0 = 0$.\r\n If we choose $n$ to be large enough, then $\\hat y^*$ and $y^*$ can be made arbitrarily close to each other. Hence we can transform the above inequality to  \\eqref{eqn:lowB-dis-prox}. More details of this derivation can be found in Appendix \\ref{appendix-lowerbound-distance-prox}.\r\n\\end{proof}","start":46351,"end":47030}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"未访问坐标的尾部范数","source_units":[{"raw_tex":"\tBy the subspace characterization \\eqref{eqn:subspace-prox}, we have\r\n\t$$\\|y^{2k}-\\hat y^*\\| \\geq \\sqrt{\\sum^n_{j = k+1} (\\hat y^*_j)^2} = \\frac{q^{k}}{1-q}\\sqrt{q^2 + q^4 + \\cdots + q^{2(n-k)}}\\geq \\frac{q^k}{\\sqrt{2}}\\|\\hat y^*\\|= \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|,$$\r\n\twhere the last inequality is due to the fact that $q< 1$, $k\\leq \\frac{n}{2}$, and $y^0 = 0$.\r\n","start":46366,"end":46736}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"大维数转移到真解","source_units":[{"raw_tex":" If we choose $n$ to be large enough, then $\\hat y^*$ and $y^*$ can be made arbitrarily close to each other. Hence we can transform the above inequality to  \\eqref{eqn:lowB-dis-prox}. More details of this derivation can be found in Appendix \\ref{appendix-lowerbound-distance-prox}.\r\n\\end{proof}","start":46736,"end":47030}]},{"id":"S045","semantic_type":"proof","kind":"proof","labels":["appendix-lowerbound-distance-prox","eqn:dist-lower-proximal","eqn:err-n-proximal"],"declared_external":false,"source_only":false,"title":"距离引理的附录完整证明","source_units":[{"raw_tex":"\\section{Proof of Lemma \\ref{lemma:lowerbound-distance-prox}}\r\n\\label{appendix-lowerbound-distance-prox}\r\nBy the subspace characterization \\eqref{eqn:subspace-prox}, we have\r\n$$\\|y^{2k}-\\hat y^*\\| \\geq \\sqrt{\\sum^n_{j = k+1} (\\hat y^*_j)^2} = \\frac{q^{k}}{1-q}\\sqrt{q^2 + q^4 + \\cdots + q^{2(n-k)}}\\geq \\frac{q^k}{\\sqrt{2}}\\|\\hat y^*\\|= \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|,$$\r\nwhere the last inequality is due to the fact that $q\\leq 1, k\\leq \\frac{n}{2}$ and $y^0 = 0$. Note that by Lemma \\ref{lemma:apxsolu-proximal}, if we require $n\\geq 2\\log_q\\left(\\frac{\\alpha}{4\\sqrt{2}}\\right)$, then we can guarantee that \\begin{equation}\r\n\\label{eqn:err-n-proximal}\r\n\\|\\hat y^* - y^*\\| \\leq \\frac{q^{n+1}}{\\alpha(1-q)}\\leq  \\frac{q^{\\frac{n}{2}}}{\\alpha}\\cdot q^k\\cdot\\frac{q}{(1-q)} \\leq \\frac{1}{4}\\cdot \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|\\quad\\mbox{ for }\\quad \\forall 1\\leq k\\leq n/2,\r\n\\end{equation}\r\nwhere the last inequality is due to $\\frac{q^{\\frac{n}{2}}}{\\alpha}\\leq\\frac{1}{4\\sqrt{2}}$ and $q/(1-q)\\leq \\|y^0-\\hat y^*\\|$. Therefore, we have\r\n\\begin{eqnarray}\r\n\\label{eqn:dist-lower-proximal}\r\n\\|y^{2k}-y^*\\|^2 & \\geq & (\\|y^{2k}-\\hat y^*\\|-\\|\\hat y^*- y^*\\|)^2\\\\\r\n& \\geq & \\|y^{2k}-\\hat y^*\\|^2 - 2\\|y^{2k}-\\hat y^*\\|\\|\\hat y^*- y^*\\|\\nonumber\\\\\r\n& \\geq & \\min_t\\left\\{t^2 - 2\\|\\hat y^*- y^*\\|t: t\\geq \\delta_k:= \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|\\right\\}\\nonumber\\\\\r\n& = & \\delta_k(\\delta_k - 2\\|\\hat y^*- y^*\\|)\\nonumber\\\\\r\n& \\geq & \\frac{1}{2}\\delta_k^2 = \\frac{q^{2k}}{4}\\|y^0-\\hat y^*\\|^2, \\nonumber\r\n\\end{eqnarray}\r\nwhere the fourth line is due to that $d(t^2 - 2\\|\\hat y^*-y^*\\|t)/dt = 2(t-\\|\\hat y^*-y^*\\|)\\geq0$ when $t\\geq\\delta_k$. Hence the quadratic function is monotonically increasing in the considered interval. In addition, we also have\r\n$$\\|y^0-y^*\\| \\leq \\|y^0-\\hat y^*\\| + \\|\\hat y^*-y^*\\|\\leq \\|y^0-\\hat y^*\\| + \\frac{q^n}{\\alpha}\\cdot\\frac{q}{1-q}\\leq (1 + q^n/\\alpha)\\|y^0-\\hat y^*\\|\\leq 2\\|y^0-\\hat y^*\\|,$$\r\nwhere the third inequality is due to that $\\|y^0-\\hat y^*\\|\\geq \\hat y^*_1 = q/(1-q)$. For the last inequality, if $\\alpha\\geq 1$, then $q^n/\\alpha<1$; if $\\alpha\\leq 1$, then  $q^n/\\alpha\\leq \\alpha/32\\leq1$ since $n\\geq 2\\log_q\\left(\\frac{\\alpha}{4\\sqrt{2}}\\right)$. Combining the above two inequalities,  the desired bound \\eqref{eqn:lowB-dis-prox} follows.\r\n","start":75790,"end":78095}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["appendix-lowerbound-distance-prox"],"declared_external":false,"source_only":false,"title":"附录重述尾部下界","source_units":[{"raw_tex":"\\section{Proof of Lemma \\ref{lemma:lowerbound-distance-prox}}\r\n\\label{appendix-lowerbound-distance-prox}\r\nBy the subspace characterization \\eqref{eqn:subspace-prox}, we have\r\n$$\\|y^{2k}-\\hat y^*\\| \\geq \\sqrt{\\sum^n_{j = k+1} (\\hat y^*_j)^2} = \\frac{q^{k}}{1-q}\\sqrt{q^2 + q^4 + \\cdots + q^{2(n-k)}}\\geq \\frac{q^k}{\\sqrt{2}}\\|\\hat y^*\\|= \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|,$$\r\nwhere the last inequality is due to the fact that $q\\leq 1, k\\leq \\frac{n}{2}$ and $y^0 = 0$. Note that by Lemma \\ref{lemma:apxsolu-proximal}, if we require $n\\geq 2\\log_q\\left(\\frac{\\alpha}{4\\sqrt{2}}\\right)$, then we can guarantee that \\begin{equation}\r\n","start":75790,"end":76424}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":["appendix-lowerbound-distance-prox","eqn:err-n-proximal"],"declared_external":false,"source_only":false,"title":"近似误差压至尾部四分之一","source_units":[{"raw_tex":"where the last inequality is due to the fact that $q\\leq 1, k\\leq \\frac{n}{2}$ and $y^0 = 0$. Note that by Lemma \\ref{lemma:apxsolu-proximal}, if we require $n\\geq 2\\log_q\\left(\\frac{\\alpha}{4\\sqrt{2}}\\right)$, then we can guarantee that \\begin{equation}\r\n\\label{eqn:err-n-proximal}\r\n\\|\\hat y^* - y^*\\| \\leq \\frac{q^{n+1}}{\\alpha(1-q)}\\leq  \\frac{q^{\\frac{n}{2}}}{\\alpha}\\cdot q^k\\cdot\\frac{q}{(1-q)} \\leq \\frac{1}{4}\\cdot \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|\\quad\\mbox{ for }\\quad \\forall 1\\leq k\\leq n/2,\r\n\\end{equation}\r\nwhere the last inequality is due to $\\frac{q^{\\frac{n}{2}}}{\\alpha}\\leq\\frac{1}{4\\sqrt{2}}$ and $q/(1-q)\\leq \\|y^0-\\hat y^*\\|$. Therefore, we have\r\n","start":76168,"end":76840}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":["appendix-lowerbound-distance-prox","eqn:dist-lower-proximal"],"declared_external":false,"source_only":false,"title":"三角不等式与单调二次式","source_units":[{"raw_tex":"\\begin{eqnarray}\r\n\\label{eqn:dist-lower-proximal}\r\n\\|y^{2k}-y^*\\|^2 & \\geq & (\\|y^{2k}-\\hat y^*\\|-\\|\\hat y^*- y^*\\|)^2\\\\\r\n& \\geq & \\|y^{2k}-\\hat y^*\\|^2 - 2\\|y^{2k}-\\hat y^*\\|\\|\\hat y^*- y^*\\|\\nonumber\\\\\r\n& \\geq & \\min_t\\left\\{t^2 - 2\\|\\hat y^*- y^*\\|t: t\\geq \\delta_k:= \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|\\right\\}\\nonumber\\\\\r\n& = & \\delta_k(\\delta_k - 2\\|\\hat y^*- y^*\\|)\\nonumber\\\\\r\n& \\geq & \\frac{1}{2}\\delta_k^2 = \\frac{q^{2k}}{4}\\|y^0-\\hat y^*\\|^2, \\nonumber\r\n\\end{eqnarray}\r\nwhere the fourth line is due to that $d(t^2 - 2\\|\\hat y^*-y^*\\|t)/dt = 2(t-\\|\\hat y^*-y^*\\|)\\geq0$ when $t\\geq\\delta_k$. Hence the quadratic function is monotonically increasing in the considered interval. In addition, we also have\r\n","start":76840,"end":77555}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["appendix-lowerbound-distance-prox"],"declared_external":false,"source_only":false,"title":"比较两种初始距离","source_units":[{"raw_tex":"$$\\|y^0-y^*\\| \\leq \\|y^0-\\hat y^*\\| + \\|\\hat y^*-y^*\\|\\leq \\|y^0-\\hat y^*\\| + \\frac{q^n}{\\alpha}\\cdot\\frac{q}{1-q}\\leq (1 + q^n/\\alpha)\\|y^0-\\hat y^*\\|\\leq 2\\|y^0-\\hat y^*\\|,$$\r\nwhere the third inequality is due to that $\\|y^0-\\hat y^*\\|\\geq \\hat y^*_1 = q/(1-q)$. For the last inequality, if $\\alpha\\geq 1$, then $q^n/\\alpha<1$; if $\\alpha\\leq 1$, then  $q^n/\\alpha\\leq \\alpha/32\\leq1$ since $n\\geq 2\\log_q\\left(\\frac{\\alpha}{4\\sqrt{2}}\\right)$. Combining the above two inequalities,  the desired bound \\eqref{eqn:lowB-dis-prox} follows.\r\n","start":77555,"end":78095}]},{"id":"S050","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"间隙下界的正文推导","source_units":[{"raw_tex":"Using Lemma \\ref{lemma:lowerbound-distance-prox} and \\eqref{defn:primal-gap-LB}, it is then straightforward to lower bound the duality gap by\r\n$$\\Delta(x^{2k},y^{2k})\\geq q^{2k}\\cdot\\frac{\\mu_y\\|y^*-y^0\\|^2}{32}.$$\r\n","start":47034,"end":47250}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"丢弃原始距离项","source_units":[{"raw_tex":"Using Lemma \\ref{lemma:lowerbound-distance-prox} and \\eqref{defn:primal-gap-LB}, it is then straightforward to lower bound the duality gap by\r\n$$\\Delta(x^{2k},y^{2k})\\geq q^{2k}\\cdot\\frac{\\mu_y\\|y^*-y^0\\|^2}{32}.$$\r\n","start":47034,"end":47250}]},{"id":"S052","semantic_type":"theorem","kind":"theorem","labels":["thm:proximal"],"declared_external":false,"source_only":false,"title":"线性跨度近端下界","source_units":[{"raw_tex":"\\begin{theorem}\r\n\t\\label{thm:proximal}\r\n Let the positive parameters $\\mu_x,\\mu_y>0$ and $L_{xy}> 0$ be given. For any integer $k$, there exists a problem instance from $\\cB(L_{xy},\\mu_x,\\mu_y)$ of form \\eqref{prob:worst-proximal}, with $n\\geq \\max\\left\\{2\\log_q\\left(\\frac{\\mu_x\\mu_y}{\\sqrt{2}L_{xy}^2}\\right), 4k\\right\\}$, where $A\\in\\mathbb{R}^{n\\times n}$ as defined in \\eqref{defn:A}, and $b = -\\frac{L_{xy}^2}{4\\mu_x}e_1$. For such a problem instance, any approximate solution $(\\tilde x^k, \\tilde y^k)\\in\\cH_x^k\\times\\cH_y^k$ generated by the proximal algorithm class under the linear span assumption \\eqref{eqn:Prox-AlgoClass}  satisfies\r\n\t\\begin{eqnarray}\r\n\t\\max_y F(\\tilde x^{k},y) - \\min_{x} F(x, \\tilde y^k)\\geq q^{k}\\cdot\\frac{\\mu_y\\|y^*-y^0\\|^2}{32} \\quad\\mbox{ and }\\quad \\|\\tilde y^k-y^*\\|^2\\geq q^{k}\\cdot\\frac{\\|y^*-y^0\\|^2}{16},\r\n\t\\end{eqnarray}\r\n\twhere $q = 1+\\frac{2\\mu_x\\mu_y}{L_{xy}^2} - 2\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}$.\r\n\\end{theorem}","start":47304,"end":48312}]},{"id":"S053","semantic_type":"proposition","kind":"proposition","labels":["eqn:lowerbound-proximal-1stAlgo","remark:complexity-proximal"],"declared_external":false,"source_only":false,"title":"收缩下界转为迭代复杂度","source_units":[{"raw_tex":"\\begin{proposition}\r\n\t\\label{remark:complexity-proximal}\r\n\tUnder the same set of assumptions of Theorem \\ref{thm:proximal}, if we require the duality gap to be bounded by $\\epsilon$, the number of iterations needed is at least\r\n\t\\begin{equation}\r\n\t\\label{eqn:lowerbound-proximal-1stAlgo}\r\n\tk\\geq \\ln\\left(\\frac{\\mu_y\\|y^*-y^0\\|^2}{32\\epsilon}\\right)/\\ln(q^{-1}) = \\Omega\\left(\\sqrt{\\frac{L_{xy}^2}{\\mu_x\\mu_y}}\\cdot\\ln\\left(\\frac{1}{\\epsilon}\\right)\\right).\r\n\t\\end{equation}\r\n\\end{proposition}","start":48314,"end":48807}]},{"id":"S054","semantic_type":"proof","kind":"proof","labels":["appdx:gen-prox","appendix:r-complexity-proximal"],"declared_external":false,"source_only":false,"title":"复杂度命题的附录证明","source_units":[{"raw_tex":"\\section{Proof of Proposition \\ref{remark:complexity-proximal}}\r\n\\label{appendix:r-complexity-proximal}\r\nHere we only prove the last inequality of \\eqref{eqn:lowerbound-proximal-1stAlgo}. Due to the fact that $(\\ln(1+z))^{-1}\\geq 1/z$ for $\\forall z>0$, we know\r\n\\begin{eqnarray*}\r\n(\\ln(q^{-1}))^{-1} & = & (\\ln(1 + (1-q)/q))^{-1} \\geq \\frac{q}{1-q}\\\\\r\n& = &   \\frac{1+\\frac{2\\mu_x\\mu_y}{L_{xy}^2} - 2\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}}{2\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}-\\frac{2\\mu_x\\mu_y}{L_{xy}^2}}\\\\\r\n& = & \\frac{\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}-\\frac{\\mu_x\\mu_y}{L_{xy}^2}}{\\frac{2\\mu_x\\mu_y}{L_{xy}^2}}\\\\\r\n& = &  \\half\\sqrt{\\frac{L_{xy}^2}{\\mu_x\\mu_y} + 1} - \\frac{1}{2},\\\\\r\n& = & \\Omega\\left(\\sqrt{\\frac{L_{xy}^2}{\\mu_x\\mu_y}}\\right)\r\n\\end{eqnarray*}\r\nwhich completes the proof.\r\n\r\n","start":78095,"end":79022}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["appdx:gen-prox","appendix:r-complexity-proximal"],"declared_external":false,"source_only":false,"title":"对数倒数的初等下界","source_units":[{"raw_tex":"Here we only prove the last inequality of \\eqref{eqn:lowerbound-proximal-1stAlgo}. Due to the fact that $(\\ln(1+z))^{-1}\\geq 1/z$ for $\\forall z>0$, we know\r\n\\begin{eqnarray*}\r\n(\\ln(q^{-1}))^{-1} & = & (\\ln(1 + (1-q)/q))^{-1} \\geq \\frac{q}{1-q}\\\\\r\n","start":78200,"end":78448}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":["appdx:gen-prox","appendix:r-complexity-proximal"],"declared_external":false,"source_only":false,"title":"显式代入收缩因子","source_units":[{"raw_tex":"& = &   \\frac{1+\\frac{2\\mu_x\\mu_y}{L_{xy}^2} - 2\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}}{2\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}-\\frac{2\\mu_x\\mu_y}{L_{xy}^2}}\\\\\r\n& = & \\frac{\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}-\\frac{\\mu_x\\mu_y}{L_{xy}^2}}{\\frac{2\\mu_x\\mu_y}{L_{xy}^2}}\\\\\r\n& = &  \\half\\sqrt{\\frac{L_{xy}^2}{\\mu_x\\mu_y} + 1} - \\frac{1}{2},\\\\\r\n& = & \\Omega\\left(\\sqrt{\\frac{L_{xy}^2}{\\mu_x\\mu_y}}\\right)\r\n\\end{eqnarray*}\r\n","start":78448,"end":78992}]},{"id":"S057","semantic_type":"definition","kind":"definition","labels":["prob:rotate"],"declared_external":false,"source_only":false,"title":"正交旋转后的实例","source_units":[{"raw_tex":"Consider the bilinear problem class $\\cB(L_{xy},\\mu_x,\\mu_y)$ and the corresponding worst case problem \\eqref{prob:worst-proximal} with $F(x,y) = \\frac{\\mu_x}{2}\\|x\\|^2 + \\frac{L_{xy}}{2}x^\\top A y - \\frac{\\mu_y}{2}\\|y\\|^2 - b^\\top y$,\r\nwhere $A$ and $b$ are defined in accordance with Theorem \\ref{thm:proximal}. We define the orthogonally rotated problem as \r\n\\begin{equation}\r\n\\label{prob:rotate}\r\n\\min_x\\max_y F_{U,V}(x,y) : = F(Ux,Vy) = \\frac{\\mu_x}{2}\\|x\\|^2 + \\frac{L_{xy}}{2}x^\\top U^\\top AV y - \\frac{\\mu_y}{2}\\|y\\|^2 - b^\\top Vy,\r\n\\end{equation}\r\n","start":49395,"end":49952}]},{"id":"S058","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"旋转保持问题类","source_units":[{"raw_tex":"where $U,V$ are two orthogonal matrices. Therefore, it is clear that $F_{U,V}\\in\\cB(L_{xy},\\mu_x,\\mu_y)$. Let $(x^*, y^*)$ be the saddle point of $F(x,y)$, then it is clear that the saddle point of $F_{U,V}(x,y)$ is $(\\bar x^*,\\bar y^*) = (U^\\top  x^*,V^\\top y^*).$ Consequently, the lower bound for the general proximal algorithm class is characterized by the following theorem. \r\n","start":49952,"end":50334}]},{"id":"S059","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"鞍点随旋转变换","source_units":[{"raw_tex":"where $U,V$ are two orthogonal matrices. Therefore, it is clear that $F_{U,V}\\in\\cB(L_{xy},\\mu_x,\\mu_y)$. Let $(x^*, y^*)$ be the saddle point of $F(x,y)$, then it is clear that the saddle point of $F_{U,V}(x,y)$ is $(\\bar x^*,\\bar y^*) = (U^\\top  x^*,V^\\top y^*).$ Consequently, the lower bound for the general proximal algorithm class is characterized by the following theorem. \r\n","start":49952,"end":50334}]},{"id":"S060","semantic_type":"lemma","kind":"lemma","labels":["appdx:gen-prox","lemma:yyx"],"declared_external":true,"declaration_text":["原始先决条件列表明确允许的 YYXu 文献 Lemma 3.1；原文 B0146 在第956–959行重述，其证明未提供。"],"source_only":false,"title":"保持子空间的旋转引理","source_units":[{"raw_tex":"\tBefore proceeding the proof, let us first quote a lemma from \\cite{YYXu}. \r\n\t\\begin{lemma}[Lemma 3.1, \\cite{YYXu}] \r\n\t\t\\label{lemma:yyx} \r\n\t\tLet $\\cX\\subsetneqq\\bar{\\cX}\\subseteqq\\R^{p}$ be two linear subspaces. Then for any $\\bar x\\in\\R^p$, there exists an orthogonal matrix $\\Gamma\\in\\R^{p\\times p}$ s.t. $\\Gamma x = x, \\forall x\\in\\cX$ and $\\Gamma \\bar x\\in\\bar{\\cX}$.\r\n\t\\end{lemma}\r\n","start":79098,"end":79486}]},{"id":"S061","semantic_type":"derived_fact","kind":"derived_fact","labels":["appdx:gen-prox"],"declared_external":false,"source_only":false,"title":"转置也固定相同向量","source_units":[{"raw_tex":"\tNote that for an orthogonal matrix $\\Gamma$, if $\\Gamma x = x$, then we also have $\\Gamma^\\top x = x$.  Now let us start our proof of Theorem \\ref{theorem:gen-prox}.\r\n\r\n","start":79486,"end":79656}]},{"id":"S062","semantic_type":"notation","kind":"notation","labels":["appdx:gen-prox"],"declared_external":false,"source_only":false,"title":"附录证明的归属标题","source_units":[{"raw_tex":"\\section{Proof of Theorem \\ref{theorem:gen-prox}}\r\n\\label{appdx:gen-prox} \r\n","start":79022,"end":79098}]},{"id":"S063","semantic_type":"theorem","kind":"theorem","labels":["theorem:gen-prox"],"declared_external":false,"source_only":false,"title":"一般近端算法的复杂度下界","source_units":[{"raw_tex":"\\begin{theorem}\r\n\\label{theorem:gen-prox}\r\nLet $\\cA$ be any algorithm from the general proximal algorithm class decribed in Definition \\ref{defn:Det-Prox-AlgoClass}. We assume the dimension $n$ is sufficiently large for simplicity. For any integer $k$, then there exist orthogonal matrices $U,V$ s.t. $F_{U,V}\\in\\cB(L_{xy},\\mu_x,\\mu_y)$, when applying $\\cA$ to $F_{U,V}$ with initial solution $(x^0,y^0) = (0,0)$, the iterates and output satisfies\r\n$$\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1}\\qquad\\mbox{and}\\qquad(\\tilde x^k,\\tilde y^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1},$$\r\nwhere $\\cH_x^i,\\cH_y^i$ are defined by Lemma \\ref{lemma:subspace-prox}. Consequenty, by Theorem \\ref{thm:proximal}, \r\n\\begin{eqnarray*}\r\n\\|\\tilde y^k-V^\\top  y^*\\|^2\\geq \\frac{q^{4k+2}}{16}\\|y^*-y^0\\|^2\r\n\\end{eqnarray*}\r\nwhere $q$ is given in Theorem \\ref{thm:proximal}. As a result, it takes $\\Omega\\left(\\sqrt{\\frac{L^2_{xy}}{\\mu_x\\mu_y}}\\cdot\\log\\left(\\frac{1}{\\epsilon}\\right)\\right)$ iterations to output a solution with $O(\\epsilon)$ duality gap.\r\n\\end{theorem}","start":50334,"end":51429}]},{"id":"S064","semantic_type":"proof","kind":"proof","labels":["thm:gen-prox-1","thm:gen-prox-2","thm:gen-prox-3","thm:gen-prox-4","thm:gen-prox-5","thm:gen-prox-6","thm:gen-prox-7","thm:gen-prox-8"],"declared_external":false,"source_only":false,"title":"主定理完整作者证明","source_units":[{"raw_tex":"For the proof of this theorem, we only need to construct the orthogonal matrices $U,V$ such that when the algorithm $\\cA$ is applied to $F_{U,V}$, the subspace inclusion argument $\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1}$ and $(\\tilde x^k,\\tilde y^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1}$ holds. As a result, \r\n\\begin{eqnarray*}\r\n\t\\|\\tilde y^k-V^\\top  y^*\\|^2 = \\|V\\tilde y^k- y^*\\|^2\\geq\\min_{y\\in\\cH_y^{4k+1}}\\|y-y^*\\|^2\\geq \\frac{q^{4k+2}}{16}\\|y^*-y^0\\|^2.\r\n\\end{eqnarray*}\r\nWith this argument, the latter results follow directly from the discussion of Theorem \\ref{thm:proximal}. The proof of this theorem is presented in Appendix \\ref{appdx:gen-prox}.\r\n","start":51431,"end":52140},{"raw_tex":"\\begin{proof}\r\n\t\tTo prove this theorem, we only need to show\r\n\t\t$$\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1}\\qquad\\mbox{and}\\qquad(\\tilde x^k,\\tilde y^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1}.$$\r\n\t\tWe separate the proof into two parts.\r\n\t\t\r\n\t\t\\textbf{Part I. There exist orthogonal matrices $\\hat U$, $\\hat V$ s.t. when $\\cA$ is applied to the rotated instance $F_{\\hat U,\\hat V}$, $\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq \\hat U^\\top\\cH_x^{4k-1}\\times \\hat V^\\top\\cH_y^{4k-1}.$}\r\n\t\r\n\t\tLet $\\theta = (L_{xy},\\mu_x,\\mu_y)$ be the set of algorithmic parameters. To prove the result, let us construct the worst-case function $F_{U,V}$ in a recursive way. \t \\\\\r\n\t\t\\textbf{Case $k = 1$:} Let us define $U_0 = V_0 = I$. When $\\cA$ is applied to the function $F_{U_0,V_0}\\in\\cB(L_{xy}, \\mu_x, \\mu_y)$, the iterate sequence is \r\n\t\t$(x_{0}^0, y_{0}^0) = (0, 0)$ and \r\n\t\t$$\\begin{cases}\r\n\t\tu^1_0 = \\cA_u^1(\\theta; x_0^0, U_0^\\top A V_0 y_0^0), \\qquad (x^1_0, \\tilde x^1_0) = \\cA_x^1(\\theta; x^0_0,  U_0^\\top A V_0 y_0^0, \\mathbf{prox}_{\\gamma_1f}(u^1_0)),\\\\\r\n\t\tv^1_0 = \\cA_v^1(\\theta; y_0^0, V^\\top_0 A U_0x_0^0),\\qquad\r\n\t\t(y^1_0, \\tilde y^1_0) = \\cA_y^1(\\theta; y^0_0, V^\\top_0 A U_0x_0^0, \\mathbf{prox}_{\\sigma_1g}(v^1_0)).\r\n\t\t\\end{cases}$$\r\n\t\tBy Lemma \\ref{lemma:yyx}, there exists orthogonal matrices $\\Gamma_x^0$ and $\\Gamma_y^0$ such that \r\n\t\t$\\Gamma_x^0x^1_0\\in\\cH_x^3 = \\mathrm{Span}\\{Ab\\}$, $\\Gamma_y^0y^1_0\\in\\cH_y^3=\\mathrm{Span}\\{b, A^2b\\}$, and $\\Gamma_y^0b = (\\Gamma_y^0)^\\top b = b.$ That is \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-1}\r\n\t\tx_0^1\\in U_1^\\top\\cH_x^3,\\qquad\\mbox{and}\\qquad y_0^1\\in V_1^\\top\\cH_y^3, \\quad V_1 b = V_1^\\top b = b,\r\n\t\t\\end{equation}\t\r\n\t\twhere $U_1 = U_0\\Gamma_x^0$ and $V_1 = V_0\\Gamma_y^0$. \r\n\t\r\n\t\tNow we prove that when we apply the algorithm $\\cA$ to $F_{U_1,V_1}$, the generated iterates $\\{(x^0_1, y^0_1), (x^1_1, y^1_1)\\}$ satisfy that\r\n\t\t$(x^0_1, y^0_1) = (0, 0)$ and  $(x^1_1, y^1_1) = (x^1_0, y^1_0)$. That is, the first two iterates generated by $\\cA$ is completely the same for $F_{U_0,V_0}$ and $F_{U_1,V_1}$. The reason is because \r\n\t\t$u^1_1 = \\cA_u^1(\\theta; x^0_1, U_1^\\top AV_1y^0_1) = \\cA_u^1(\\theta; 0, 0) = \\cA_u^1(\\theta; x^0_0, U_0^\\top AV_0y^0_0)  = u^1_0$, \r\n\t\ttherefore \r\n\t\t\\begin{eqnarray*}\r\n\t\t\t(x^1_1, \\tilde x^1_1) & = & \\cA_x^1(\\theta; x^0_1, U_1^\\top A V_1 y_1^0, \\mathbf{prox}_{\\gamma_1f}(u^1_1)) \\\\\r\n\t\t\t& = & \\cA_x^1(\\theta; 0, 0, \\mathbf{prox}_{\\gamma_1f}(u^1_1)) \\\\\r\n\t\t\t& = & \\cA_x^1(\\theta; x^0_0, U_0^\\top A V_0 y_0^0, \\mathbf{prox}_{\\gamma_1f}(u^1_0)) \\\\\r\n\t\t\t& = & (x^1_0, \\tilde x^1_0).\r\n\t\t\\end{eqnarray*}\r\n\t\tThrough similar argument, we know $(y^1_1,\\tilde y^1_1) = (y^1_0,\\tilde y^1_0)$. Therefore, \\eqref{thm:gen-prox-1} indicates that \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-2}\r\n\t\tx_1^1\\in U_1^\\top\\cH_x^3,\\qquad\\mbox{and}\\qquad y_1^1\\in V_1^\\top\\cH_y^3, \\quad V_1 b = V_1^\\top b = b\\in V_1^\\top\\cH_y^3.\r\n\t\t\\end{equation}\r\n\t\t\\textbf{Case $k=2$.} For the ease of the readers to follow, we perform one extra step of discussion for $k=2$, before presenting the construction on general $k$. \r\n\t    \r\n\t\tFor the problem instance $F_{U_1,V_1}$, the iterates generated by $\\cA$ are $(x_{1}^0, y_{1}^0) = (0, 0)$ and \r\n\t\t$$\\begin{cases}\r\n\t\tu^1_1 = \\cA_u^1(\\theta; x_1^0, U_1^\\top A V_1 y_1^0), \\qquad (x^1_1, \\tilde x^1_1) = \\cA_x^1(\\theta; x^0_1,  U_1^\\top A V_1 y_1^0, \\mathbf{prox}_{\\gamma_1f}(u^1_1)),\\\\\r\n\t\tv^1_1 = \\cA_v^1(\\theta; y_1^0, V^\\top_1 A U_1x_1^0),\\qquad\r\n\t\t(y^1_1, \\tilde y^1_1) = \\cA_y^1(\\theta; y^0_1, V^\\top_1 A U_1x_1^0, \\mathbf{prox}_{\\sigma_1g}(v^1_1)).\\\\\r\n\t\t\\end{cases}$$\r\n\t\t$$\\begin{cases}\r\n\t\tu^2_1 = \\cA_u^2(\\theta; x_1^0, U_1^\\top A V_1 y_1^0, x_1^1, U_1^\\top A V_1 y_1^1), \\quad (x^2_1, \\tilde x^2_1) = \\cA_x^2(\\theta; x^0_1,  U_1^\\top A V_1 y_1^0, x_1^1, U_1^\\top A V_1 y_1^1, \\mathbf{prox}_{\\gamma_2f}(u^2_1)),\\\\\r\n\t\tv^2_1 = \\cA_v^2(\\theta; y_1^0, V^\\top_1 A U_1x_1^0, y_1^1, V^\\top_1 A U_1x_1^1),\\quad\r\n\t\t(y^2_1, \\tilde y^2_1) = \\cA_y^2(\\theta; y^0_1, V^\\top_1 A U_1x_1^0, y_1^1, V^\\top_1 A U_1x_1^1, \\mathbf{prox}_{\\sigma_2g}(v^2_1)).\\\\\r\n\t\t\\end{cases}$$\r\n\t\tNote that  $x_1^1 \\in U_1^\\top\\cH_x^3 \\subsetneqq U_1^\\top\\cH_x^5\\subsetneqq U_1^\\top\\cH_x^7$ and $\\{y_1^1,b\\}\\subsetneqq V_1^\\top\\cH_y^3\\subsetneqq V_1^\\top\\cH_y^5\\subsetneqq V_1^\\top\\cH_y^7$. Therefore, there exist orthogonal matrices $\\Gamma_x^1$ and $\\Gamma_y^1$ such that \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-3}\r\n\t\t\\begin{cases}\r\n\t\t\\Gamma_x^1x = (\\Gamma_x^1)^\\top x = x, \\,\\,\\forall x\\in U_1^\\top\\cH_x^5,\\,\\,\\,\\, \\Gamma_x^1 x_1^2\\in U_1^\\top\\cH_x^7,\\\\\r\n\t\t\\Gamma_y^1y \\,= (\\Gamma_y^1)^\\top y = y, \\,\\,\\,\\forall y\\in V_1^\\top\\cH_y^5,\\,\\,\\,\\, \\Gamma_y^1 y_1^2\\in V_1^\\top\\cH_y^7.\r\n\t\t\\end{cases}\r\n\t\t\\end{equation}\r\n\t\tNow, let us define \r\n\t\t$$U_2 = U_1\\Gamma_x^1\\qquad\\mbox{and}\\qquad V_2 = V_1\\Gamma_y^1.$$\r\n\t\tNow we prove that if $\\cA$ is applied to $F_{U_2,V_2}$, the generated iterates $\\{(x_{2}^0, y_{2}^0), (x_{2}^1, y_{2}^1), (x_{2}^2, y_{2}^2)\\}$ satisfy $(x_{2}^0, y_{2}^0) = (0, 0)$, $(x_{2}^1, y_{2}^1) = (x_{1}^1, y_{1}^1)$, and  $(x_{2}^2, y_{2}^2) = (x_{1}^2, y_{1}^2)$. The argument for $(x_{2}^1, y_{2}^1) = (x_{1}^1, y_{1}^1)$ is almost the same as  that of the case $k=1$. We only provide the proof for $(x_{2}^2, y_{2}^2) = (x_{1}^2, y_{1}^2)$.\r\n\t\r\n\tNext, we need to show $u_2^2 = u_1^2$, which can be proved by arguing that all the inputs to $\\cA_u^2$ are the same for both $u_2^2$ and $u_1^2$. First, it is straightforward that $x_1^0 = 0 = x^0_2, U_1^\\top AV_1 y_1^0 = 0 = U_2^\\top AV_2 y_2^0$. By previous argument $x_2^1 = x_1^1$.  Finally, consider the last input $U_2^\\top AV_2 y_2^1$, because $y_2^1 = y_1^1\\in V_1^\\top\\cH_y^3\\subsetneqq V_1^\\top\\cH_y^5$, we have $\\Gamma_y^1 y_2^1 = y_2^1 = y_1^1\\in V_1^\\top\\cH_y^3.$ Then $V_2y_2^1 = V_1\\Gamma_y^1y_2^1\\in V_1V_1^\\top\\cH_y^3 = \\cH_y^3.$ Therefore $U_1^\\top AV_2y_2^1\\in U_1^\\top A\\cH_y^3 = U_1^\\top\\cH_x^5$ and \r\n\t\t$$U_2^\\top AV_2y_2^1 = \\Gamma_x^1U_1^\\top AV_2y_2^1 = U_1^\\top AV_2y_2^1 = U_1^\\top AV_1\\Gamma_y^1y_2^1 = U_1^\\top AV_1y_1^1.$$ \r\n\t\tConsequently, \r\n\t\t\\begin{eqnarray*}\r\n\t\t\tu_2^2  = \\cA_u^2(\\theta; x_2^0, U_2^\\top AV_2 y_2^0, x_2^1, U_2^\\top AV_2 y_2^1)\r\n\t\t\t=\\cA_u^2(\\theta; x_1^0, U_1^\\top AV_1 y_1^0, x_1^1, U_1^\\top AV_1 y_1^1)\r\n\t\t\t= u_2^1\r\n\t\t\\end{eqnarray*}\r\n\t\tand \r\n\t\t\\begin{eqnarray*}\r\n\t\t\t(x_2^2,\\tilde x_2^2)  & = & \\cA_x^2(\\theta; x_2^0, U_2^\\top AV_2 y_2^0, x_2^1, U_2^\\top AV_2 y_2^1,\\mathbf{prox}_{\\gamma_2f}(u^2_2))\\\\\r\n\t\t\t& = &\\cA_x^2(\\theta; x_1^0, U_1^\\top AV_1 y_1^0, x_1^1, U_1^\\top AV_1 y_1^1,\\mathbf{prox}_{\\gamma_2f}(u^2_1))\\\\\r\n\t\t\t& = & (x_1^2,\\tilde x_1^2).\r\n\t\t\\end{eqnarray*}\r\n\t\tThrough a similar argument, we have $(y_2^2,\\tilde y_2^2) = (y_1^2,\\tilde y_1^2)$. By \\eqref{thm:gen-prox-2} and \\eqref{thm:gen-prox-3}, we have \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-4}\r\n\t\t\\{x_2^0,x_2^1,x_2^2\\}\\in U_2^\\top\\cH_x^7\\qquad\\mbox{and}\\qquad\\{b,y_2^0,y_2^1,y_2^2\\}\\in V_2^\\top\\cH_y^7.\r\n\t\t\\end{equation}\r\n\t\t\\textbf{Case $k$.} Suppose we already have orthogonal matrices $U_{k-1}, V_{k-1}$, such that when $\\cA$ is applied to $F_{U_{k-1},V_{k-1}}$, we have\r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-5}\r\n\t\t\\{x_{k-1}^0,x_{k-1}^1,\\cdots, x_{k-1}^{k-1}\\}\\in U_{k-1}^\\top\\cH_x^{4k-5}\\qquad\\mbox{and}\\qquad\\{b,y_{k-1}^0,y_{k-1}^1,\\cdots,y_{k-1}^{k-1}\\}\\in V_{k-1}^\\top\\cH_y^{4k-5}.\r\n\t\t\\end{equation}\r\n\t\tAgain, by Lemma \\ref{lemma:yyx}, there exist orthogonal matrices $\\Gamma_x^{k-1}$ and $\\Gamma_y^{k-1}$, such that \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-6}\r\n\t\t\\begin{cases}\r\n\t\t\\Gamma_x^{k-1}x = (\\Gamma_x^{k-1})^\\top x = x, \\,\\,\\forall x\\in U_{k-1}^\\top\\cH_x^{4k-3},\\,\\,\\,\\, \\Gamma_x^{k-1} x_{k-1}^{k}\\in U_{k-1}^\\top\\cH_x^{4k-1},\\\\\r\n\t\t\\Gamma_y^{k-1}y \\,= (\\Gamma_y^{k-1})^\\top y = y, \\,\\,\\,\\forall y\\in V_{k-1}^\\top\\cH_y^{4k-3},\\,\\,\\,\\, \\Gamma_y^{k-1} y_{k-1}^k\\in V_{k-1}^\\top\\cH_y^{4k-1}.\r\n\t\t\\end{cases}\r\n\t\t\\end{equation}\r\n\t\tNow we define that \r\n\t\t$$U_k = U_{k-1}\\Gamma_x^{k-1} \\qquad\\mbox{and}\\qquad V_k = V_{k-1}\\Gamma_y^{k-1}.$$\r\n\t\tTherefore, similar to our previous discussion, we only need to argue that when $\\cA$ is applied to $F_{U_k,V_k}$, the generated iterates $\\{(x_k^0,y_k^0), (x_k^1,y_k^1),\\cdots,(x_k^k,y_k^k)\\}$ satisfy $(x_k^i,y_k^i) = (x_{k-1}^i,y_{k-1}^i)$ for $i = 0,1,...,k$. We prove this argument by induction. First, it is straightforward that $(x_k^0,y_k^0) = (0,0) = (x_{k-1}^0,y_{k-1}^0)$. Suppose $(x_k^i,y_k^i) = (x_{k-1}^i,y_{k-1}^i)$ holds for $i = 0,1,...,j-1\\leq k-1$, now we prove $(x_k^j,y_k^j) = (x_{k-1}^j,y_{k-1}^j)$, which is almost identical to the case $k=2$.\r\n\t\r\n\tFor any $i\\in\\{0,1,...,j-1\\}$, let us show $U_{k-1}^\\top AV_{k-1} y_{k-1}^i = U_k^\\top AV_k y_k^i$. Because $y_k^i = y_{k-1}^i\\in V_{k-1}^\\top\\cH_y^{4k-5}\\subsetneqq V_{k-1}^\\top\\cH_y^{4k-3}$, we have $\\Gamma_y^{k-1} y_k^i = y_k^i = y_{k-1}^i\\in V_{k-1}^\\top\\cH_y^{4k-5}.$ Then $V_ky_k^i = V_{k-1}\\Gamma_y^{k-1}y_k^i\\in V_{k-1}V_{k-1}^\\top\\cH_y^{4k-5} = \\cH_y^{4k-5}.$ Therefore $U_{k-1}^\\top AV_{k}y_k^i\\in U_{k-1}^\\top A\\cH_y^{4k-5} = U_{k-1}^\\top\\cH_x^{4k-3}$ and \r\n\t\t$$U_k^\\top AV_ky_k^i = (\\Gamma_x^{k-1})^\\top U_{k-1}^\\top AV_ky_k^i = U_{k-1}^\\top AV_{k}y_k^i = U_{k-1}^\\top AV_{k-1}\\Gamma_y^{k-1}y_k^i = U_{k-1}^\\top AV_{k-1}y_{k-1}^i,$$ \r\n\t\tfor $0\\leq i\\leq j-1$. Consequently, \r\n\t\t\\begin{eqnarray*}\r\n\t\t\tu_k^i  & = & \\cA_u^i(\\theta; x_k^0, U_k^\\top AV_k y_k^0,..., x_k^{i-1}, U_k^\\top AV_k y_k^{i-1})\\\\\r\n\t\t\t& = & \\cA_u^i(\\theta; x_{k-1}^0, U_{k-1}^\\top AV_{k-1} y_{k-1}^0,..., x_{k-1}^{i-1}, U_{k-1}^\\top AV_{k-1} y_{k-1}^{i-1})\\\\\r\n\t\t\t& = & u_{k-1}^i\r\n\t\t\\end{eqnarray*}\r\n\t\tand \r\n\t\t\\begin{eqnarray*}\r\n\t\t\t(x_k^i,\\tilde x_k^i)  & = & \\cA_x^i(\\theta; x_k^0, U_k^\\top AV_k y_k^0,..., x_k^{i-1}, U_k^\\top AV_k y_k^{i-1},\\mathbf{prox}_{\\gamma_if}(u^i_k))\\\\\r\n\t\t\t& = &\\cA_x^2(\\theta; x_{k-1}^0, U_{k-1}^\\top AV_{k-1} y_{k-1}^0,..., x_{k-1}^{i-1}, U_{k-1}^\\top AV_{k-1} y_{k-1}^{i-1},\\mathbf{prox}_{\\gamma_if}(u^i_{k-1}))\\\\\r\n\t\t\t& = & (x_{k-1}^i,\\tilde x_{k-1}^i).\r\n\t\t\\end{eqnarray*}\r\n\t\tThrough a similar argument, we have $(y_k^i,\\tilde y_k^i) = (y_{k-1}^i,\\tilde y_{k-1}^i)$. By induction, we know $(y_k^i,\\tilde y_k^i) = (y_{k-1}^i,\\tilde y_{k-1}^i)$ for $i = 0,1,...,k$. Consequently, we have \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-7}\r\n\t\t\\{x_{k}^0,x_{k}^1,\\cdots, x_{k}^{k}\\}\\in U_{k}^\\top\\cH_x^{4k-1}\\qquad\\mbox{and}\\qquad\\{b,y_{k}^0,y_{k}^1,\\cdots,y_{k}^{k}\\}\\in V_{k}^\\top\\cH_y^{4k-1}.\r\n\t\t\\end{equation}\r\n\t    By setting $\\hat U = U_k$ and $\\hat V = V_k$, we prove the result for Part I.\\vspace{0.5cm}\\\\       \r\n       \t\\textbf{Part II. There exist orthogonal matrices $U$, $V$ such that when $\\cA$ is applied to the rotated instance $F_{U,V}$, $\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1},$ and $(\\tilde{x}^k,\\tilde{y}^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1}$}.\r\n       \r\n       Given the result of Part I, and let $\\{(x_k^0, y_k^0),...,(x_k^k, y_k^k)\\}$ and $(\\tilde x_k^k, \\tilde y_k^k)$ be generated by $\\cA$ when applied to $F_{\\hat U,\\hat V} = F_{U_k,V_k}$. Therefore, by Lemma \\ref{lemma:yyx}, there exist orthogonal matrices $P,Q$ such that \r\n       \\begin{equation}\r\n       \\label{thm:gen-prox-8}\r\n       \\begin{cases}\r\n       Px = P^\\top x = x, \\,\\,\\forall x\\in U_{k}^\\top\\cH_x^{4k-1},\\,\\,\\,\\, P \\tilde x_{k}^{k}\\in U_{k}^\\top\\cH_x^{4k+1},\\\\\r\n       Qy \\,=Q^\\top y = y, \\,\\,\\forall y\\in V_{k}^\\top\\cH_y^{4k-1},\\,\\,\\,\\, Q\\tilde y_{k}^k\\in V_{k}^\\top\\cH_y^{4k+1}.\r\n       \\end{cases}\r\n       \\end{equation}\r\n       Define $U = U_kP$, and $V = V_kQ$. Let $\\{(x^0,y^0),...,(x^k,y^k)\\}$ and the output $(\\tilde x^k,\\tilde y^k)$ be generated by $\\cA$ when applied to $F_{{U,V}}$. Then following the same line of argument of Case $k$, Part I, we have \r\n       $$(x^i,y^i) = (x^i_k,y^i_k), \\,\\,\\mbox{for}\\,\\, i = 0,1,...,k\\qquad\\mbox{and}\\qquad(\\tilde x^k,\\tilde y^k) = (\\tilde x^k_k,\\tilde y^k_k).$$\r\n       Therefore, combining \\eqref{thm:gen-prox-8}, we complete the proof of Part II.  \r\n\\end{proof}","start":79656,"end":91544}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"旋回距离并转为复杂度","source_units":[{"raw_tex":"For the proof of this theorem, we only need to construct the orthogonal matrices $U,V$ such that when the algorithm $\\cA$ is applied to $F_{U,V}$, the subspace inclusion argument $\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1}$ and $(\\tilde x^k,\\tilde y^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1}$ holds. As a result, \r\n\\begin{eqnarray*}\r\n\t\\|\\tilde y^k-V^\\top  y^*\\|^2 = \\|V\\tilde y^k- y^*\\|^2\\geq\\min_{y\\in\\cH_y^{4k+1}}\\|y-y^*\\|^2\\geq \\frac{q^{4k+2}}{16}\\|y^*-y^0\\|^2.\r\n\\end{eqnarray*}\r\nWith this argument, the latter results follow directly from the discussion of Theorem \\ref{thm:proximal}. The proof of this theorem is presented in Appendix \\ref{appdx:gen-prox}.\r\n","start":51431,"end":52140}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"构造目标与两部分安排","source_units":[{"raw_tex":"\\begin{proof}\r\n\t\tTo prove this theorem, we only need to show\r\n\t\t$$\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1}\\qquad\\mbox{and}\\qquad(\\tilde x^k,\\tilde y^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1}.$$\r\n\t\tWe separate the proof into two parts.\r\n\t\t\r\n\t\t\\textbf{Part I. There exist orthogonal matrices $\\hat U$, $\\hat V$ s.t. when $\\cA$ is applied to the rotated instance $F_{\\hat U,\\hat V}$, $\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq \\hat U^\\top\\cH_x^{4k-1}\\times \\hat V^\\top\\cH_y^{4k-1}.$}\r\n\t\r\n\t\tLet $\\theta = (L_{xy},\\mu_x,\\mu_y)$ be the set of algorithmic parameters. To prove the result, let us construct the worst-case function $F_{U,V}$ in a recursive way. \t \\\\\r\n","start":79656,"end":80358}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"第一轮的原始运行记录","source_units":[{"raw_tex":"\t\t\\textbf{Case $k = 1$:} Let us define $U_0 = V_0 = I$. When $\\cA$ is applied to the function $F_{U_0,V_0}\\in\\cB(L_{xy}, \\mu_x, \\mu_y)$, the iterate sequence is \r\n\t\t$(x_{0}^0, y_{0}^0) = (0, 0)$ and \r\n\t\t$$\\begin{cases}\r\n\t\tu^1_0 = \\cA_u^1(\\theta; x_0^0, U_0^\\top A V_0 y_0^0), \\qquad (x^1_0, \\tilde x^1_0) = \\cA_x^1(\\theta; x^0_0,  U_0^\\top A V_0 y_0^0, \\mathbf{prox}_{\\gamma_1f}(u^1_0)),\\\\\r\n\t\tv^1_0 = \\cA_v^1(\\theta; y_0^0, V^\\top_0 A U_0x_0^0),\\qquad\r\n\t\t(y^1_0, \\tilde y^1_0) = \\cA_y^1(\\theta; y^0_0, V^\\top_0 A U_0x_0^0, \\mathbf{prox}_{\\sigma_1g}(v^1_0)).\r\n\t\t\\end{cases}$$\r\n","start":80358,"end":80934}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":["thm:gen-prox-1"],"declared_external":false,"source_only":false,"title":"第一轮的对抗旋转","source_units":[{"raw_tex":"\t\tBy Lemma \\ref{lemma:yyx}, there exists orthogonal matrices $\\Gamma_x^0$ and $\\Gamma_y^0$ such that \r\n\t\t$\\Gamma_x^0x^1_0\\in\\cH_x^3 = \\mathrm{Span}\\{Ab\\}$, $\\Gamma_y^0y^1_0\\in\\cH_y^3=\\mathrm{Span}\\{b, A^2b\\}$, and $\\Gamma_y^0b = (\\Gamma_y^0)^\\top b = b.$ That is \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-1}\r\n\t\tx_0^1\\in U_1^\\top\\cH_x^3,\\qquad\\mbox{and}\\qquad y_0^1\\in V_1^\\top\\cH_y^3, \\quad V_1 b = V_1^\\top b = b,\r\n\t\t\\end{equation}\t\r\n\t\twhere $U_1 = U_0\\Gamma_x^0$ and $V_1 = V_0\\Gamma_y^0$. \r\n","start":80934,"end":81430}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":["thm:gen-prox-2"],"declared_external":false,"source_only":false,"title":"第一轮运行保持不变","source_units":[{"raw_tex":"\t\r\n\t\tNow we prove that when we apply the algorithm $\\cA$ to $F_{U_1,V_1}$, the generated iterates $\\{(x^0_1, y^0_1), (x^1_1, y^1_1)\\}$ satisfy that\r\n\t\t$(x^0_1, y^0_1) = (0, 0)$ and  $(x^1_1, y^1_1) = (x^1_0, y^1_0)$. That is, the first two iterates generated by $\\cA$ is completely the same for $F_{U_0,V_0}$ and $F_{U_1,V_1}$. The reason is because \r\n\t\t$u^1_1 = \\cA_u^1(\\theta; x^0_1, U_1^\\top AV_1y^0_1) = \\cA_u^1(\\theta; 0, 0) = \\cA_u^1(\\theta; x^0_0, U_0^\\top AV_0y^0_0)  = u^1_0$, \r\n\t\ttherefore \r\n\t\t\\begin{eqnarray*}\r\n\t\t\t(x^1_1, \\tilde x^1_1) & = & \\cA_x^1(\\theta; x^0_1, U_1^\\top A V_1 y_1^0, \\mathbf{prox}_{\\gamma_1f}(u^1_1)) \\\\\r\n\t\t\t& = & \\cA_x^1(\\theta; 0, 0, \\mathbf{prox}_{\\gamma_1f}(u^1_1)) \\\\\r\n\t\t\t& = & \\cA_x^1(\\theta; x^0_0, U_0^\\top A V_0 y_0^0, \\mathbf{prox}_{\\gamma_1f}(u^1_0)) \\\\\r\n\t\t\t& = & (x^1_0, \\tilde x^1_0).\r\n\t\t\\end{eqnarray*}\r\n\t\tThrough similar argument, we know $(y^1_1,\\tilde y^1_1) = (y^1_0,\\tilde y^1_0)$. Therefore, \\eqref{thm:gen-prox-1} indicates that \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-2}\r\n\t\tx_1^1\\in U_1^\\top\\cH_x^3,\\qquad\\mbox{and}\\qquad y_1^1\\in V_1^\\top\\cH_y^3, \\quad V_1 b = V_1^\\top b = b\\in V_1^\\top\\cH_y^3.\r\n\t\t\\end{equation}\r\n","start":81430,"end":82604}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"第二轮的运行记录","source_units":[{"raw_tex":"\t\t\\textbf{Case $k=2$.} For the ease of the readers to follow, we perform one extra step of discussion for $k=2$, before presenting the construction on general $k$. \r\n\t    \r\n\t\tFor the problem instance $F_{U_1,V_1}$, the iterates generated by $\\cA$ are $(x_{1}^0, y_{1}^0) = (0, 0)$ and \r\n\t\t$$\\begin{cases}\r\n\t\tu^1_1 = \\cA_u^1(\\theta; x_1^0, U_1^\\top A V_1 y_1^0), \\qquad (x^1_1, \\tilde x^1_1) = \\cA_x^1(\\theta; x^0_1,  U_1^\\top A V_1 y_1^0, \\mathbf{prox}_{\\gamma_1f}(u^1_1)),\\\\\r\n\t\tv^1_1 = \\cA_v^1(\\theta; y_1^0, V^\\top_1 A U_1x_1^0),\\qquad\r\n\t\t(y^1_1, \\tilde y^1_1) = \\cA_y^1(\\theta; y^0_1, V^\\top_1 A U_1x_1^0, \\mathbf{prox}_{\\sigma_1g}(v^1_1)).\\\\\r\n\t\t\\end{cases}$$\r\n\t\t$$\\begin{cases}\r\n\t\tu^2_1 = \\cA_u^2(\\theta; x_1^0, U_1^\\top A V_1 y_1^0, x_1^1, U_1^\\top A V_1 y_1^1), \\quad (x^2_1, \\tilde x^2_1) = \\cA_x^2(\\theta; x^0_1,  U_1^\\top A V_1 y_1^0, x_1^1, U_1^\\top A V_1 y_1^1, \\mathbf{prox}_{\\gamma_2f}(u^2_1)),\\\\\r\n\t\tv^2_1 = \\cA_v^2(\\theta; y_1^0, V^\\top_1 A U_1x_1^0, y_1^1, V^\\top_1 A U_1x_1^1),\\quad\r\n\t\t(y^2_1, \\tilde y^2_1) = \\cA_y^2(\\theta; y^0_1, V^\\top_1 A U_1x_1^0, y_1^1, V^\\top_1 A U_1x_1^1, \\mathbf{prox}_{\\sigma_2g}(v^2_1)).\\\\\r\n\t\t\\end{cases}$$\r\n","start":82604,"end":83757}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":["thm:gen-prox-3"],"declared_external":false,"source_only":false,"title":"固定五阶空间旋入七阶","source_units":[{"raw_tex":"\t\tNote that  $x_1^1 \\in U_1^\\top\\cH_x^3 \\subsetneqq U_1^\\top\\cH_x^5\\subsetneqq U_1^\\top\\cH_x^7$ and $\\{y_1^1,b\\}\\subsetneqq V_1^\\top\\cH_y^3\\subsetneqq V_1^\\top\\cH_y^5\\subsetneqq V_1^\\top\\cH_y^7$. Therefore, there exist orthogonal matrices $\\Gamma_x^1$ and $\\Gamma_y^1$ such that \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-3}\r\n\t\t\\begin{cases}\r\n\t\t\\Gamma_x^1x = (\\Gamma_x^1)^\\top x = x, \\,\\,\\forall x\\in U_1^\\top\\cH_x^5,\\,\\,\\,\\, \\Gamma_x^1 x_1^2\\in U_1^\\top\\cH_x^7,\\\\\r\n\t\t\\Gamma_y^1y \\,= (\\Gamma_y^1)^\\top y = y, \\,\\,\\,\\forall y\\in V_1^\\top\\cH_y^5,\\,\\,\\,\\, \\Gamma_y^1 y_1^2\\in V_1^\\top\\cH_y^7.\r\n\t\t\\end{cases}\r\n\t\t\\end{equation}\r\n\t\tNow, let us define \r\n\t\t$$U_2 = U_1\\Gamma_x^1\\qquad\\mbox{and}\\qquad V_2 = V_1\\Gamma_y^1.$$\r\n","start":83757,"end":84475}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"第二轮历史信息保持","source_units":[{"raw_tex":"\t\tNow we prove that if $\\cA$ is applied to $F_{U_2,V_2}$, the generated iterates $\\{(x_{2}^0, y_{2}^0), (x_{2}^1, y_{2}^1), (x_{2}^2, y_{2}^2)\\}$ satisfy $(x_{2}^0, y_{2}^0) = (0, 0)$, $(x_{2}^1, y_{2}^1) = (x_{1}^1, y_{1}^1)$, and  $(x_{2}^2, y_{2}^2) = (x_{1}^2, y_{1}^2)$. The argument for $(x_{2}^1, y_{2}^1) = (x_{1}^1, y_{1}^1)$ is almost the same as  that of the case $k=1$. We only provide the proof for $(x_{2}^2, y_{2}^2) = (x_{1}^2, y_{1}^2)$.\r\n\t\r\n\tNext, we need to show $u_2^2 = u_1^2$, which can be proved by arguing that all the inputs to $\\cA_u^2$ are the same for both $u_2^2$ and $u_1^2$. First, it is straightforward that $x_1^0 = 0 = x^0_2, U_1^\\top AV_1 y_1^0 = 0 = U_2^\\top AV_2 y_2^0$. By previous argument $x_2^1 = x_1^1$.  Finally, consider the last input $U_2^\\top AV_2 y_2^1$, because $y_2^1 = y_1^1\\in V_1^\\top\\cH_y^3\\subsetneqq V_1^\\top\\cH_y^5$, we have $\\Gamma_y^1 y_2^1 = y_2^1 = y_1^1\\in V_1^\\top\\cH_y^3.$ Then $V_2y_2^1 = V_1\\Gamma_y^1y_2^1\\in V_1V_1^\\top\\cH_y^3 = \\cH_y^3.$ Therefore $U_1^\\top AV_2y_2^1\\in U_1^\\top A\\cH_y^3 = U_1^\\top\\cH_x^5$ and \r\n\t\t$$U_2^\\top AV_2y_2^1 = \\Gamma_x^1U_1^\\top AV_2y_2^1 = U_1^\\top AV_2y_2^1 = U_1^\\top AV_1\\Gamma_y^1y_2^1 = U_1^\\top AV_1y_1^1.$$ \r\n\t\tConsequently, \r\n\t\t\\begin{eqnarray*}\r\n\t\t\tu_2^2  = \\cA_u^2(\\theta; x_2^0, U_2^\\top AV_2 y_2^0, x_2^1, U_2^\\top AV_2 y_2^1)\r\n\t\t\t=\\cA_u^2(\\theta; x_1^0, U_1^\\top AV_1 y_1^0, x_1^1, U_1^\\top AV_1 y_1^1)\r\n\t\t\t= u_2^1\r\n\t\t\\end{eqnarray*}\r\n\t\tand \r\n\t\t\\begin{eqnarray*}\r\n\t\t\t(x_2^2,\\tilde x_2^2)  & = & \\cA_x^2(\\theta; x_2^0, U_2^\\top AV_2 y_2^0, x_2^1, U_2^\\top AV_2 y_2^1,\\mathbf{prox}_{\\gamma_2f}(u^2_2))\\\\\r\n\t\t\t& = &\\cA_x^2(\\theta; x_1^0, U_1^\\top AV_1 y_1^0, x_1^1, U_1^\\top AV_1 y_1^1,\\mathbf{prox}_{\\gamma_2f}(u^2_1))\\\\\r\n\t\t\t& = & (x_1^2,\\tilde x_1^2).\r\n\t\t\\end{eqnarray*}\r\n","start":84475,"end":86257}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":["thm:gen-prox-4"],"declared_external":false,"source_only":false,"title":"第二轮子空间结论","source_units":[{"raw_tex":"\t\tThrough a similar argument, we have $(y_2^2,\\tilde y_2^2) = (y_1^2,\\tilde y_1^2)$. By \\eqref{thm:gen-prox-2} and \\eqref{thm:gen-prox-3}, we have \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-4}\r\n\t\t\\{x_2^0,x_2^1,x_2^2\\}\\in U_2^\\top\\cH_x^7\\qquad\\mbox{and}\\qquad\\{b,y_2^0,y_2^1,y_2^2\\}\\in V_2^\\top\\cH_y^7.\r\n\t\t\\end{equation}\r\n","start":86257,"end":86579}]},{"id":"S074","semantic_type":"assumption","kind":"assumption","labels":["thm:gen-prox-5"],"declared_external":false,"source_only":false,"title":"外层归纳假设","source_units":[{"raw_tex":"\t\t\\textbf{Case $k$.} Suppose we already have orthogonal matrices $U_{k-1}, V_{k-1}$, such that when $\\cA$ is applied to $F_{U_{k-1},V_{k-1}}$, we have\r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-5}\r\n\t\t\\{x_{k-1}^0,x_{k-1}^1,\\cdots, x_{k-1}^{k-1}\\}\\in U_{k-1}^\\top\\cH_x^{4k-5}\\qquad\\mbox{and}\\qquad\\{b,y_{k-1}^0,y_{k-1}^1,\\cdots,y_{k-1}^{k-1}\\}\\in V_{k-1}^\\top\\cH_y^{4k-5}.\r\n\t\t\\end{equation}\r\n","start":86579,"end":86969}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":["thm:gen-prox-6"],"declared_external":false,"source_only":false,"title":"一般轮次的旋转扩张","source_units":[{"raw_tex":"\t\tAgain, by Lemma \\ref{lemma:yyx}, there exist orthogonal matrices $\\Gamma_x^{k-1}$ and $\\Gamma_y^{k-1}$, such that \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-6}\r\n\t\t\\begin{cases}\r\n\t\t\\Gamma_x^{k-1}x = (\\Gamma_x^{k-1})^\\top x = x, \\,\\,\\forall x\\in U_{k-1}^\\top\\cH_x^{4k-3},\\,\\,\\,\\, \\Gamma_x^{k-1} x_{k-1}^{k}\\in U_{k-1}^\\top\\cH_x^{4k-1},\\\\\r\n\t\t\\Gamma_y^{k-1}y \\,= (\\Gamma_y^{k-1})^\\top y = y, \\,\\,\\,\\forall y\\in V_{k-1}^\\top\\cH_y^{4k-3},\\,\\,\\,\\, \\Gamma_y^{k-1} y_{k-1}^k\\in V_{k-1}^\\top\\cH_y^{4k-1}.\r\n\t\t\\end{cases}\r\n\t\t\\end{equation}\r\n\t\tNow we define that \r\n\t\t$$U_k = U_{k-1}\\Gamma_x^{k-1} \\qquad\\mbox{and}\\qquad V_k = V_{k-1}\\Gamma_y^{k-1}.$$\r\n","start":86969,"end":87611}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"逐轮归纳运行不变","source_units":[{"raw_tex":"\t\tTherefore, similar to our previous discussion, we only need to argue that when $\\cA$ is applied to $F_{U_k,V_k}$, the generated iterates $\\{(x_k^0,y_k^0), (x_k^1,y_k^1),\\cdots,(x_k^k,y_k^k)\\}$ satisfy $(x_k^i,y_k^i) = (x_{k-1}^i,y_{k-1}^i)$ for $i = 0,1,...,k$. We prove this argument by induction. First, it is straightforward that $(x_k^0,y_k^0) = (0,0) = (x_{k-1}^0,y_{k-1}^0)$. Suppose $(x_k^i,y_k^i) = (x_{k-1}^i,y_{k-1}^i)$ holds for $i = 0,1,...,j-1\\leq k-1$, now we prove $(x_k^j,y_k^j) = (x_{k-1}^j,y_{k-1}^j)$, which is almost identical to the case $k=2$.\r\n\t\r\n\tFor any $i\\in\\{0,1,...,j-1\\}$, let us show $U_{k-1}^\\top AV_{k-1} y_{k-1}^i = U_k^\\top AV_k y_k^i$. Because $y_k^i = y_{k-1}^i\\in V_{k-1}^\\top\\cH_y^{4k-5}\\subsetneqq V_{k-1}^\\top\\cH_y^{4k-3}$, we have $\\Gamma_y^{k-1} y_k^i = y_k^i = y_{k-1}^i\\in V_{k-1}^\\top\\cH_y^{4k-5}.$ Then $V_ky_k^i = V_{k-1}\\Gamma_y^{k-1}y_k^i\\in V_{k-1}V_{k-1}^\\top\\cH_y^{4k-5} = \\cH_y^{4k-5}.$ Therefore $U_{k-1}^\\top AV_{k}y_k^i\\in U_{k-1}^\\top A\\cH_y^{4k-5} = U_{k-1}^\\top\\cH_x^{4k-3}$ and \r\n\t\t$$U_k^\\top AV_ky_k^i = (\\Gamma_x^{k-1})^\\top U_{k-1}^\\top AV_ky_k^i = U_{k-1}^\\top AV_{k}y_k^i = U_{k-1}^\\top AV_{k-1}\\Gamma_y^{k-1}y_k^i = U_{k-1}^\\top AV_{k-1}y_{k-1}^i,$$ \r\n\t\tfor $0\\leq i\\leq j-1$. Consequently, \r\n\t\t\\begin{eqnarray*}\r\n\t\t\tu_k^i  & = & \\cA_u^i(\\theta; x_k^0, U_k^\\top AV_k y_k^0,..., x_k^{i-1}, U_k^\\top AV_k y_k^{i-1})\\\\\r\n\t\t\t& = & \\cA_u^i(\\theta; x_{k-1}^0, U_{k-1}^\\top AV_{k-1} y_{k-1}^0,..., x_{k-1}^{i-1}, U_{k-1}^\\top AV_{k-1} y_{k-1}^{i-1})\\\\\r\n\t\t\t& = & u_{k-1}^i\r\n\t\t\\end{eqnarray*}\r\n\t\tand \r\n\t\t\\begin{eqnarray*}\r\n\t\t\t(x_k^i,\\tilde x_k^i)  & = & \\cA_x^i(\\theta; x_k^0, U_k^\\top AV_k y_k^0,..., x_k^{i-1}, U_k^\\top AV_k y_k^{i-1},\\mathbf{prox}_{\\gamma_if}(u^i_k))\\\\\r\n\t\t\t& = &\\cA_x^2(\\theta; x_{k-1}^0, U_{k-1}^\\top AV_{k-1} y_{k-1}^0,..., x_{k-1}^{i-1}, U_{k-1}^\\top AV_{k-1} y_{k-1}^{i-1},\\mathbf{prox}_{\\gamma_if}(u^i_{k-1}))\\\\\r\n\t\t\t& = & (x_{k-1}^i,\\tilde x_{k-1}^i).\r\n\t\t\\end{eqnarray*}\r\n","start":87611,"end":89566}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","labels":["thm:gen-prox-7"],"declared_external":false,"source_only":false,"title":"完成迭代序列的旋转构造","source_units":[{"raw_tex":"\t\tThrough a similar argument, we have $(y_k^i,\\tilde y_k^i) = (y_{k-1}^i,\\tilde y_{k-1}^i)$. By induction, we know $(y_k^i,\\tilde y_k^i) = (y_{k-1}^i,\\tilde y_{k-1}^i)$ for $i = 0,1,...,k$. Consequently, we have \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-7}\r\n\t\t\\{x_{k}^0,x_{k}^1,\\cdots, x_{k}^{k}\\}\\in U_{k}^\\top\\cH_x^{4k-1}\\qquad\\mbox{and}\\qquad\\{b,y_{k}^0,y_{k}^1,\\cdots,y_{k}^{k}\\}\\in V_{k}^\\top\\cH_y^{4k-1}.\r\n\t\t\\end{equation}\r\n\t    By setting $\\hat U = U_k$ and $\\hat V = V_k$, we prove the result for Part I.\\vspace{0.5cm}\\\\       \r\n","start":89566,"end":90105}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","labels":["thm:gen-prox-8"],"declared_external":false,"source_only":false,"title":"额外旋转控制输出","source_units":[{"raw_tex":"       \t\\textbf{Part II. There exist orthogonal matrices $U$, $V$ such that when $\\cA$ is applied to the rotated instance $F_{U,V}$, $\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1},$ and $(\\tilde{x}^k,\\tilde{y}^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1}$}.\r\n       \r\n       Given the result of Part I, and let $\\{(x_k^0, y_k^0),...,(x_k^k, y_k^k)\\}$ and $(\\tilde x_k^k, \\tilde y_k^k)$ be generated by $\\cA$ when applied to $F_{\\hat U,\\hat V} = F_{U_k,V_k}$. Therefore, by Lemma \\ref{lemma:yyx}, there exist orthogonal matrices $P,Q$ such that \r\n       \\begin{equation}\r\n       \\label{thm:gen-prox-8}\r\n       \\begin{cases}\r\n       Px = P^\\top x = x, \\,\\,\\forall x\\in U_{k}^\\top\\cH_x^{4k-1},\\,\\,\\,\\, P \\tilde x_{k}^{k}\\in U_{k}^\\top\\cH_x^{4k+1},\\\\\r\n       Qy \\,=Q^\\top y = y, \\,\\,\\forall y\\in V_{k}^\\top\\cH_y^{4k-1},\\,\\,\\,\\, Q\\tilde y_{k}^k\\in V_{k}^\\top\\cH_y^{4k+1}.\r\n       \\end{cases}\r\n       \\end{equation}\r\n","start":90105,"end":91056}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"最终实例保持运行与输出","source_units":[{"raw_tex":"       Define $U = U_kP$, and $V = V_kQ$. Let $\\{(x^0,y^0),...,(x^k,y^k)\\}$ and the output $(\\tilde x^k,\\tilde y^k)$ be generated by $\\cA$ when applied to $F_{{U,V}}$. Then following the same line of argument of Case $k$, Part I, we have \r\n       $$(x^i,y^i) = (x^i_k,y^i_k), \\,\\,\\mbox{for}\\,\\, i = 0,1,...,k\\qquad\\mbox{and}\\qquad(\\tilde x^k,\\tilde y^k) = (\\tilde x^k_k,\\tilde y^k_k).$$\r\n       Therefore, combining \\eqref{thm:gen-prox-8}, we complete the proof of Part II.  \r\n\\end{proof}","start":91056,"end":91544}]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件列表所列：第301–308行使用的标准强凸性和鞍点对偶性事实；未提供外部证明。"],"source_only":false,"title":"强凸凹与鞍点标准事实","source_units":[]},{"id":"S081","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件列表所列：正定矩阵的初等谱界；对应第577行的使用。"],"source_only":false,"title":"正定逆矩阵的谱界","source_units":[]},{"id":"S082","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件列表所列：初等几何级数；对应第599、924行的使用。"],"source_only":false,"title":"几何级数尾部比较","source_units":[]},{"id":"S083","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件列表所列：初等对数不等式；原文第943行明确陈述所用形式。"],"source_only":false,"title":"初等对数不等式","source_units":[]},{"id":"S084","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；原文第637、650、960行直接使用的正交矩阵基本性质，未给出独立外部来源。"],"source_only":false,"title":"正交变换的基本不变性","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S063","edges":[{"source_id":"S001","target_id":"S009","kind":"supports"},{"source_id":"S002","target_id":"S009","kind":"supports"},{"source_id":"S001","target_id":"S012","kind":"supports"},{"source_id":"S002","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S080","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S006","kind":"supports"},{"source_id":"S080","target_id":"S006","kind":"supports"},{"source_id":"S004","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S080","target_id":"S007","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S001","target_id":"S018","kind":"supports"},{"source_id":"S014","target_id":"S018","kind":"supports"},{"source_id":"S015","target_id":"S018","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S016","target_id":"S020","kind":"supports"},{"source_id":"S002","target_id":"S021","kind":"supports"},{"source_id":"S009","target_id":"S021","kind":"supports"},{"source_id":"S014","target_id":"S021","kind":"supports"},{"source_id":"S002","target_id":"S022","kind":"supports"},{"source_id":"S009","target_id":"S022","kind":"supports"},{"source_id":"S014","target_id":"S022","kind":"supports"},{"source_id":"S018","target_id":"S022","kind":"supports"},{"source_id":"S009","target_id":"S025","kind":"supports"},{"source_id":"S021","target_id":"S025","kind":"supports"},{"source_id":"S022","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S026","kind":"supports"},{"source_id":"S003","target_id":"S027","kind":"supports"},{"source_id":"S014","target_id":"S027","kind":"supports"},{"source_id":"S018","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S080","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S034","kind":"supports"},{"source_id":"S028","target_id":"S034","kind":"supports"},{"source_id":"S030","target_id":"S034","kind":"supports"},{"source_id":"S031","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S016","target_id":"S035","kind":"supports"},{"source_id":"S030","target_id":"S036","kind":"supports"},{"source_id":"S032","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S034","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S081","target_id":"S037","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S035","target_id":"S033","kind":"supports"},{"source_id":"S036","target_id":"S033","kind":"supports"},{"source_id":"S037","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S029","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S032","target_id":"S029","kind":"supports"},{"source_id":"S020","target_id":"S039","kind":"supports"},{"source_id":"S026","target_id":"S039","kind":"supports"},{"source_id":"S031","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S039","target_id":"S043","kind":"supports"},{"source_id":"S032","target_id":"S043","kind":"supports"},{"source_id":"S023","target_id":"S043","kind":"supports"},{"source_id":"S082","target_id":"S043","kind":"supports"},{"source_id":"S029","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S044","target_id":"S042","kind":"supports"},{"source_id":"S039","target_id":"S046","kind":"supports"},{"source_id":"S032","target_id":"S046","kind":"supports"},{"source_id":"S023","target_id":"S046","kind":"supports"},{"source_id":"S082","target_id":"S046","kind":"supports"},{"source_id":"S029","target_id":"S047","kind":"supports"},{"source_id":"S030","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S029","target_id":"S049","kind":"supports"},{"source_id":"S030","target_id":"S049","kind":"supports"},{"source_id":"S032","target_id":"S049","kind":"supports"},{"source_id":"S023","target_id":"S049","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S048","target_id":"S045","kind":"supports"},{"source_id":"S049","target_id":"S045","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S045","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S051","kind":"supports"},{"source_id":"S007","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S018","target_id":"S052","kind":"supports"},{"source_id":"S038","target_id":"S052","kind":"supports"},{"source_id":"S041","target_id":"S052","kind":"supports"},{"source_id":"S050","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S030","target_id":"S055","kind":"supports"},{"source_id":"S083","target_id":"S055","kind":"supports"},{"source_id":"S030","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S054","kind":"supports"},{"source_id":"S056","target_id":"S054","kind":"supports"},{"source_id":"S014","target_id":"S057","kind":"supports"},{"source_id":"S052","target_id":"S057","kind":"supports"},{"source_id":"S001","target_id":"S058","kind":"supports"},{"source_id":"S018","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S084","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S059","kind":"supports"},{"source_id":"S084","target_id":"S059","kind":"supports"},{"source_id":"S084","target_id":"S061","kind":"supports"},{"source_id":"S012","target_id":"S063","kind":"supports"},{"source_id":"S026","target_id":"S063","kind":"supports"},{"source_id":"S057","target_id":"S063","kind":"supports"},{"source_id":"S052","target_id":"S063","kind":"supports"},{"source_id":"S012","target_id":"S067","kind":"supports"},{"source_id":"S057","target_id":"S067","kind":"supports"},{"source_id":"S023","target_id":"S067","kind":"supports"},{"source_id":"S060","target_id":"S068","kind":"supports"},{"source_id":"S026","target_id":"S068","kind":"supports"},{"source_id":"S031","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S012","target_id":"S069","kind":"supports"},{"source_id":"S014","target_id":"S069","kind":"supports"},{"source_id":"S067","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S021","target_id":"S069","kind":"supports"},{"source_id":"S022","target_id":"S069","kind":"supports"},{"source_id":"S012","target_id":"S070","kind":"supports"},{"source_id":"S069","target_id":"S070","kind":"supports"},{"source_id":"S060","target_id":"S071","kind":"supports"},{"source_id":"S061","target_id":"S071","kind":"supports"},{"source_id":"S026","target_id":"S071","kind":"supports"},{"source_id":"S069","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S012","target_id":"S072","kind":"supports"},{"source_id":"S026","target_id":"S072","kind":"supports"},{"source_id":"S069","target_id":"S072","kind":"supports"},{"source_id":"S070","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S021","target_id":"S072","kind":"supports"},{"source_id":"S022","target_id":"S072","kind":"supports"},{"source_id":"S069","target_id":"S073","kind":"supports"},{"source_id":"S071","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S060","target_id":"S075","kind":"supports"},{"source_id":"S061","target_id":"S075","kind":"supports"},{"source_id":"S026","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S012","target_id":"S076","kind":"supports"},{"source_id":"S026","target_id":"S076","kind":"supports"},{"source_id":"S074","target_id":"S076","kind":"supports"},{"source_id":"S075","target_id":"S076","kind":"supports"},{"source_id":"S021","target_id":"S076","kind":"supports"},{"source_id":"S022","target_id":"S076","kind":"supports"},{"source_id":"S074","target_id":"S077","kind":"supports"},{"source_id":"S075","target_id":"S077","kind":"supports"},{"source_id":"S076","target_id":"S077","kind":"supports"},{"source_id":"S060","target_id":"S078","kind":"supports"},{"source_id":"S061","target_id":"S078","kind":"supports"},{"source_id":"S026","target_id":"S078","kind":"supports"},{"source_id":"S077","target_id":"S078","kind":"supports"},{"source_id":"S012","target_id":"S079","kind":"supports"},{"source_id":"S026","target_id":"S079","kind":"supports"},{"source_id":"S077","target_id":"S079","kind":"supports"},{"source_id":"S078","target_id":"S079","kind":"supports"},{"source_id":"S076","target_id":"S079","kind":"supports"},{"source_id":"S021","target_id":"S079","kind":"supports"},{"source_id":"S022","target_id":"S079","kind":"supports"},{"source_id":"S079","target_id":"S065","kind":"supports"},{"source_id":"S059","target_id":"S065","kind":"supports"},{"source_id":"S026","target_id":"S065","kind":"supports"},{"source_id":"S052","target_id":"S065","kind":"supports"},{"source_id":"S084","target_id":"S065","kind":"supports"},{"source_id":"S007","target_id":"S065","kind":"supports"},{"source_id":"S053","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S064","kind":"supports"},{"source_id":"S067","target_id":"S064","kind":"supports"},{"source_id":"S068","target_id":"S064","kind":"supports"},{"source_id":"S069","target_id":"S064","kind":"supports"},{"source_id":"S070","target_id":"S064","kind":"supports"},{"source_id":"S071","target_id":"S064","kind":"supports"},{"source_id":"S072","target_id":"S064","kind":"supports"},{"source_id":"S073","target_id":"S064","kind":"supports"},{"source_id":"S074","target_id":"S064","kind":"supports"},{"source_id":"S075","target_id":"S064","kind":"supports"},{"source_id":"S076","target_id":"S064","kind":"supports"},{"source_id":"S077","target_id":"S064","kind":"supports"},{"source_id":"S078","target_id":"S064","kind":"supports"},{"source_id":"S079","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S063","kind":"supports"},{"source_id":"S058","target_id":"S063","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S063","edges":[{"source_id":"S001","target_id":"S009","kind":"supports"},{"source_id":"S002","target_id":"S009","kind":"supports"},{"source_id":"S001","target_id":"S012","kind":"supports"},{"source_id":"S002","target_id":"S012","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S080","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S006","kind":"supports"},{"source_id":"S080","target_id":"S006","kind":"supports"},{"source_id":"S004","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S080","target_id":"S007","kind":"supports"},{"source_id":"S001","target_id":"S018","kind":"supports"},{"source_id":"S014","target_id":"S018","kind":"supports"},{"source_id":"S015","target_id":"S018","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S016","target_id":"S020","kind":"supports"},{"source_id":"S002","target_id":"S021","kind":"supports"},{"source_id":"S009","target_id":"S021","kind":"supports"},{"source_id":"S014","target_id":"S021","kind":"supports"},{"source_id":"S002","target_id":"S022","kind":"supports"},{"source_id":"S009","target_id":"S022","kind":"supports"},{"source_id":"S014","target_id":"S022","kind":"supports"},{"source_id":"S018","target_id":"S022","kind":"supports"},{"source_id":"S009","target_id":"S026","kind":"supports"},{"source_id":"S021","target_id":"S026","kind":"supports"},{"source_id":"S022","target_id":"S026","kind":"supports"},{"source_id":"S023","target_id":"S026","kind":"supports"},{"source_id":"S003","target_id":"S027","kind":"supports"},{"source_id":"S014","target_id":"S027","kind":"supports"},{"source_id":"S018","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S080","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S016","target_id":"S029","kind":"supports"},{"source_id":"S032","target_id":"S029","kind":"supports"},{"source_id":"S081","target_id":"S029","kind":"supports"},{"source_id":"S020","target_id":"S039","kind":"supports"},{"source_id":"S026","target_id":"S039","kind":"supports"},{"source_id":"S031","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S052","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S032","target_id":"S041","kind":"supports"},{"source_id":"S023","target_id":"S041","kind":"supports"},{"source_id":"S082","target_id":"S041","kind":"supports"},{"source_id":"S029","target_id":"S041","kind":"supports"},{"source_id":"S030","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S052","kind":"supports"},{"source_id":"S007","target_id":"S052","kind":"supports"},{"source_id":"S018","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S030","target_id":"S053","kind":"supports"},{"source_id":"S083","target_id":"S053","kind":"supports"},{"source_id":"S014","target_id":"S057","kind":"supports"},{"source_id":"S052","target_id":"S057","kind":"supports"},{"source_id":"S001","target_id":"S058","kind":"supports"},{"source_id":"S018","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S084","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S059","kind":"supports"},{"source_id":"S084","target_id":"S059","kind":"supports"},{"source_id":"S084","target_id":"S061","kind":"supports"},{"source_id":"S012","target_id":"S063","kind":"supports"},{"source_id":"S026","target_id":"S063","kind":"supports"},{"source_id":"S057","target_id":"S063","kind":"supports"},{"source_id":"S052","target_id":"S063","kind":"supports"},{"source_id":"S012","target_id":"S067","kind":"supports"},{"source_id":"S057","target_id":"S067","kind":"supports"},{"source_id":"S023","target_id":"S067","kind":"supports"},{"source_id":"S060","target_id":"S068","kind":"supports"},{"source_id":"S026","target_id":"S068","kind":"supports"},{"source_id":"S031","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S012","target_id":"S069","kind":"supports"},{"source_id":"S014","target_id":"S069","kind":"supports"},{"source_id":"S067","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S021","target_id":"S069","kind":"supports"},{"source_id":"S022","target_id":"S069","kind":"supports"},{"source_id":"S012","target_id":"S070","kind":"supports"},{"source_id":"S069","target_id":"S070","kind":"supports"},{"source_id":"S060","target_id":"S071","kind":"supports"},{"source_id":"S061","target_id":"S071","kind":"supports"},{"source_id":"S026","target_id":"S071","kind":"supports"},{"source_id":"S069","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S012","target_id":"S072","kind":"supports"},{"source_id":"S026","target_id":"S072","kind":"supports"},{"source_id":"S069","target_id":"S072","kind":"supports"},{"source_id":"S070","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S021","target_id":"S072","kind":"supports"},{"source_id":"S022","target_id":"S072","kind":"supports"},{"source_id":"S069","target_id":"S073","kind":"supports"},{"source_id":"S071","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S060","target_id":"S075","kind":"supports"},{"source_id":"S061","target_id":"S075","kind":"supports"},{"source_id":"S026","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S012","target_id":"S076","kind":"supports"},{"source_id":"S026","target_id":"S076","kind":"supports"},{"source_id":"S074","target_id":"S076","kind":"supports"},{"source_id":"S075","target_id":"S076","kind":"supports"},{"source_id":"S021","target_id":"S076","kind":"supports"},{"source_id":"S022","target_id":"S076","kind":"supports"},{"source_id":"S074","target_id":"S077","kind":"supports"},{"source_id":"S075","target_id":"S077","kind":"supports"},{"source_id":"S076","target_id":"S077","kind":"supports"},{"source_id":"S060","target_id":"S078","kind":"supports"},{"source_id":"S061","target_id":"S078","kind":"supports"},{"source_id":"S026","target_id":"S078","kind":"supports"},{"source_id":"S077","target_id":"S078","kind":"supports"},{"source_id":"S012","target_id":"S079","kind":"supports"},{"source_id":"S026","target_id":"S079","kind":"supports"},{"source_id":"S077","target_id":"S079","kind":"supports"},{"source_id":"S078","target_id":"S079","kind":"supports"},{"source_id":"S076","target_id":"S079","kind":"supports"},{"source_id":"S021","target_id":"S079","kind":"supports"},{"source_id":"S022","target_id":"S079","kind":"supports"},{"source_id":"S079","target_id":"S065","kind":"supports"},{"source_id":"S059","target_id":"S065","kind":"supports"},{"source_id":"S026","target_id":"S065","kind":"supports"},{"source_id":"S052","target_id":"S065","kind":"supports"},{"source_id":"S084","target_id":"S065","kind":"supports"},{"source_id":"S007","target_id":"S065","kind":"supports"},{"source_id":"S053","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S064","kind":"supports"},{"source_id":"S067","target_id":"S064","kind":"supports"},{"source_id":"S068","target_id":"S064","kind":"supports"},{"source_id":"S069","target_id":"S064","kind":"supports"},{"source_id":"S070","target_id":"S064","kind":"supports"},{"source_id":"S071","target_id":"S064","kind":"supports"},{"source_id":"S072","target_id":"S064","kind":"supports"},{"source_id":"S073","target_id":"S064","kind":"supports"},{"source_id":"S074","target_id":"S064","kind":"supports"},{"source_id":"S075","target_id":"S064","kind":"supports"},{"source_id":"S076","target_id":"S064","kind":"supports"},{"source_id":"S077","target_id":"S064","kind":"supports"},{"source_id":"S078","target_id":"S064","kind":"supports"},{"source_id":"S079","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S063","kind":"supports"},{"source_id":"S058","target_id":"S063","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"definition","kind":"definition","labels":["defn:bilinear-class","prob:bilinear"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"双线性问题类","source_units":[{"raw_tex":"\\begin{definition}[Bilinear  class $\\cB(L_{xy},\\mu_x,\\mu_y)$]\r\n\t\\label{defn:bilinear-class}\r\n\tIn this special class, the problems are written as\r\n\t\\begin{equation}\r\n\t\\label{prob:bilinear}\r\n\t\\min_x \\max_y  F(x,y):=f(x) + x^\\top Ay - g(y),\r\n\t\\end{equation}\r\n%\tWe require\r\nwhere $f(x)$ and $g(y)$ are both lower semi-continuous with $f(x)$ being $\\mu_x$-strongly convex and $g(y)$ being $\\mu_y$-strongly convex. The coupling matrix $A$ satisfies $\\|A\\|_2\\leq L_{xy}$.\r\n\\end{definition}\r\n","start":9313,"end":9797}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":["defn:prox-fg"],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"近端预言机","source_units":[{"raw_tex":"For this special model class $\\cB(L_{xy},\\mu_x,\\mu_y)$, %subclass of problems,\r\nwe assume the availability of the following prox-operations:\r\n\\begin{equation}\r\n\\label{defn:prox-fg}\r\n\\prox_{\\gamma f}(v):=\\argmin_x f(x) + \\frac{1}{2\\gamma}\\|x-v\\|^2 \\quad \\mbox{ and } \\quad\\prox_{\\sigma g}(u):=\\argmin_y g(y) + \\frac{1}{2\\sigma}\\|y-u\\|^2.\r\n\\end{equation}\r\n","start":9797,"end":10151}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":["defn:primal-func"],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"原始函数与对偶函数","source_units":[{"raw_tex":"\\subsection{Primal function, dual function, and the duality gap}\r\nFirst, we define $\\Phi(\\cdot)$ to be the primal function and $\\Psi(\\cdot)$ to be the dual function of the saddle point problem $\\min_x \\max_y F(x,y)$, respectively, with the following definitions\r\n\\begin{equation}\r\n\\label{defn:primal-func}\r\n\\Phi(x):=\\max_y F(x,y)\\qquad\\mbox{ and }\\qquad\\Psi(y):=\\min_x F(x,y).\r\n\\end{equation}\r\n","start":22208,"end":22602}]},{"id":"S004","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"值函数的强凸凹性","source_units":[{"raw_tex":"As the maximum of a class of $\\mu_x$-strongly convex function, we know $\\Phi(x)$ is a $\\mu_x$-strongly convex function. Similarly, $\\Psi(y)$ is a $\\mu_y$-strongly concave function. We define the duality gap as\r\n","start":22602,"end":22813}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"对偶间隙","source_units":[{"raw_tex":"$$\\Delta(x,y) := \\max_{y'} F(x,y')-\\min_{x'} F(x',y) = \\Phi(x)-\\Psi(y).$$\r\n","start":22813,"end":22888}]},{"id":"S006","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"鞍点处的值相等","source_units":[{"raw_tex":"Suppose the unique solution of this min-max problem is $(x^*,y^*)$. By the strong duality theorem, we know for any $x$ and $y$ it holds that \r\n$$\\Phi(x)\\geq\\min_{x'}\\Phi(x') = \\Phi(x^*) =F(x^*,y^*) = \\Psi(y^*) = \\max_{y'}\\Psi(y')\\geq \\Psi(y).$$\r\n","start":22888,"end":23134}]},{"id":"S007","semantic_type":"derived_fact","kind":"derived_fact","labels":["defn:primal-gap-LB"],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"间隙控制解距离","source_units":[{"raw_tex":"Together with the $\\mu_x$-strong convexity of $\\Phi$ and the $\\mu_y$-strong concavity of $\\Psi$, we further have \r\n\\begin{equation}\r\n\\label{defn:primal-gap-LB}\r\n\\Delta(x,y)= \\Phi(x)-\\Phi(x^*) + \\Psi(y^*) - \\Psi(y)\\geq\\frac{\\mu_x}{2}\\|x-x^*\\|^2 + \\frac{\\mu_y}{2}\\|y-y^*\\|^2.\r\n\\end{equation}\r\n\r\n","start":23134,"end":23427}]},{"id":"S009","semantic_type":"algorithm","kind":"algorithm","labels":["defn:Prox-AlgoClass","eqn:Prox-AlgoClass"],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"线性跨度近端算法类","source_units":[{"raw_tex":"\\begin{definition}[Proximal algorithm class]\r\n\\label{defn:Prox-AlgoClass}\r\nIn each iteration, the iterate sequence $\\{(x^{k},y^{k})\\}_{k=0,1,...}$ are generated so that  $(x^{k},y^{k})\\in\\cH_x^{k} \\times \\cH_y^{k}$. These subspaces are generated with $\\cH_x^{0} = \\mathrm{Span}\\{x^0\\}, \\cH_y^{0} = \\mathrm{Span}\\{y^0\\}$ and\r\n\\begin{equation}\r\n\\label{eqn:Prox-AlgoClass}\\begin{cases}\r\n\\cH_x^{k+1} := \\mathrm{Span}\\{x^i,\\prox_{\\gamma_i f}(\\hat{x}^i-\\gamma_iA\\tilde{y}^i)~\\,: \\forall \\hat{x}^i\\in\\cH_x^{i},~ \\tilde{y}^i\\in\\cH_y^i,~ 0\\leq i\\leq k\\}\\\\\r\n\\cH_y^{k+1} := \\mathrm{Span}\\{y^i,\\prox_{\\sigma_i g}(\\hat{y}^i+\\sigma_iA^\\top\\tilde{x}^i): \\forall \\tilde{x}^i\\in\\cH_x^{i},~ \\hat{y}^i\\in\\cH_y^i,~ 0\\leq i\\leq k\\}.\r\n\\end{cases}\r\n\\end{equation}\r\n\\end{definition}\r\n","start":24321,"end":25081}]},{"id":"S011","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"去除跨度限制的动机","source_units":[{"raw_tex":"\\subsection{General deterministic algorithm classes without linear span structure}\r\nAlthough all the reviewed first-order methods satisfy the \\emph{linear span} property in the proximal algorithm class in Definition \\ref{defn:Prox-AlgoClass} and the pure first-order algorithm class in Definition \\ref{defn:Pure-AlgoClass}, this does not exclude the possibility of the deriving an algorithm that does not satisfy the linear span property. Therefore, we also define the general deterministic proximal algorithm class and the general deterministic pure first-order algorithm class as follows, whose iteration complexity lower bound can be generalized from their linear span counterpart through the technique of \\emph{adversary rotation}. \r\n","start":32650,"end":33388}]},{"id":"S012","semantic_type":"algorithm","kind":"algorithm","labels":["defn:Det-Prox-AlgoClass","eqn:Det-Prox-AlgoClass"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"一般确定性近端算法","source_units":[{"raw_tex":"\\begin{definition}[General proximal algorithm class]\r\n\t\\label{defn:Det-Prox-AlgoClass}\r\n\tConsider the problem \\eqref{prob:bilinear} in the problem class $\\cB(L_{xy},\\mu_x,\\mu_y)$, denote $\\theta = (L_{xy},\\mu_x,\\mu_y)$ as the corresponding problem parameters. Let algorithm $\\cA$ belong to the general  proximal algorithm class. Then $\\cA$ consists of a sequence of deterministic mappings $\\{(\\cA_x^1, \\cA_y^1,\\cA_u^1,\\cA_v^1),(\\cA_x^2, \\cA_y^2,\\cA_u^2,\\cA_v^2)...\\}$ such that the iterate sequence $\\{(x^{k},y^{k})\\}_{k=0,1,...}$ and the output sequence $\\{(\\tilde x^{k},\\tilde y^{k})\\}_{k=0,1,...}$ are generated by \r\n\t\\begin{equation}\r\n\t\\label{eqn:Det-Prox-AlgoClass}\\begin{cases}\r\n\t(x^k,\\tilde x^k) := \\cA_x^k\\left(\\theta; x^0,Ay^0,...,x^{k-1},Ay^{k-1}; \\mathbf{prox}_{\\gamma_k f}(u^k) \\right),\\\\\r\n\t(y^k,\\tilde y^k) := \\cA_y^k\\left(\\theta; y^0,A^\\top x^0,...,y^{k-1},A^\\top x^{k-1}; \\mathbf{prox}_{\\sigma_k g}(v^k) \\right),\r\n\t\\end{cases}\r\n\t\\end{equation} \r\n\twhere $u^k = \\cA^k_u(\\theta; x^0,Ay^0,...,x^{k-1},Ay^{k-1})$, $v^k = \\cA^k_v(\\theta; y^0,A^\\top x^0,...,y^{k-1},A^\\top x^{k-1})$, and \r\n\t$(x^0,y^0)$ is any given initial solution. \r\n\\end{definition}\r\n","start":33388,"end":34550}]},{"id":"S014","semantic_type":"problem_setting","kind":"problem_setting","labels":["prob:worst-proximal"],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"二次最坏情形实例","source_units":[{"raw_tex":"Let us construct the following bilinearly coupled min-max saddle point problem:\r\n\\begin{equation}\r\n\\label{prob:worst-proximal}\r\n\\min_x\\max_y F(x,y):=\\frac{\\mu_x}{2}\\|x\\|^2 + \\frac{L_{xy}}{2}x^\\top Ay - \\frac{\\mu_y}{2}\\|y\\|^2 - b^\\top y\r\n\\end{equation}\r\n","start":37175,"end":37428}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"反对角差分矩阵","source_units":[{"raw_tex":"where $b$ is a vector to be determined later, and the coupling matrix $A$ (hence $A^2$ and $A^4$) is defined as follows: %. Its polynomials $A^2$ and $A^4$ are also shown bellow for later reference.\r\n\\begin{equation}\r\n\\label{defn:A}\r\nA = \\begin{pmatrix}\r\n& &    &     & 1\\\\\r\n& &    &  1 & -1\\\\\r\n& & 1 & -1&\\\\\r\n& \\udots &\\udots & & \\\\\r\n1& -1 & & &\r\n\\end{pmatrix}, \\,\\,\r\n","start":37428,"end":37797}]},{"id":"S016","semantic_type":"derived_fact","kind":"derived_fact","labels":["defn:A"],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"平方矩阵的三对角形式","source_units":[{"raw_tex":"A^2 = \\begin{pmatrix}\r\n1   & -1 &            &               & \\\\\r\n-1 & 2   & -1        &              & \\\\\r\n& -1 &\\ddots & \\ddots & \\\\\r\n&      &\\ddots & 2           &-1\\\\\r\n&      &              & -1         & 2\r\n\\end{pmatrix},\r\n","start":37797,"end":38026}]},{"id":"S017","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"四次幂矩阵","source_units":[{"raw_tex":"\\,\\,\r\nA^4 = \\begin{pmatrix}\r\n2   & -3 &     1      &               &  & &\\\\\r\n-3 & 6   & -4        &     1         & & & \\\\\r\n1& -4 &6 & -4 &1 \\\\\r\n&  \\ddots     &\\ddots & \\ddots  &\\ddots & \\ddots & \\\\\r\n& & 1& -4 &6 & -4 &1\\\\\r\n& & & 1& -4 &6 & -4 \\\\\r\n& & & & 1&-4&5\r\n\\end{pmatrix}.\r\n\\end{equation}\r\n","start":38026,"end":38322}]},{"id":"S018","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"矩阵范数与实例归属","source_units":[{"raw_tex":"Note that $A^\\top =A$ and $\\|A\\|_2 \\leq 2$. Therefore \\eqref{prob:worst-proximal} is an instance in the problem class $\\cB(L_{xy},\\mu_x,\\mu_y)$. It is worth noting that the example \\eqref{prob:worst-proximal} is the same as that in Proposition 2 of \\cite{LowB-Quadratic}, which is a parallel work focused on pure first-order algorithm class. Here, we use the same example to elaborate the lower bound of the proximal methods over the general bilinear coupling class $\\cB(L_{xy},\\mu_x,\\mu_y)$, as well as a warmup for the discussion of more complex problem class $\\cF(L_{x},L_{y},L_{xy},\\mu_x,\\mu_y)$. \r\n","start":38322,"end":38925}]},{"id":"S019","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"坐标向量与直接计算说明","source_units":[{"raw_tex":"\r\nDenote $e_i$ to be the $i$-th unit vector, which has $1$ at the $i$-th component and 0 elsewhere. Then by direct calculation, one can check that $A^2$ satisfies the following \\emph{zero-chain property} (see Chapter 2 of  \\cite{nesterov2018lectures}).\r\n\r\n","start":38925,"end":39181}]},{"id":"S020","semantic_type":"proposition","kind":"proposition","labels":["prop:zero-chain"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"平方矩阵的零链性质","source_units":[{"raw_tex":"\\begin{proposition}[Zero-chain property]\r\n\t\\label{prop:zero-chain}\r\n\tFor any vector $v\\in\\R^n$, if $v\\in\\mathrm{Span}\\{e_i:i\\leq k\\}$ for some $1\\leq k\\leq n-1$,  then $A^2v\\in\\mathrm{Span}\\{e_i:i\\leq k+1\\}$.\r\n\\end{proposition}","start":39181,"end":39408}]},{"id":"S021","semantic_type":"derived_fact","kind":"derived_fact","labels":["defn:prox-closedForm-x"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"原始块近端闭式","source_units":[{"raw_tex":"For problem \\eqref{prob:worst-proximal}, the proximal operators in \\eqref{eqn:Prox-AlgoClass} can be written explicitly:\r\n\\begin{eqnarray}\r\n\\label{defn:prox-closedForm-x}\r\n\\prox_{\\gamma_i f}(\\hat{x}_i-\\gamma_iA\\tilde{y}_i) & = & \\argmin_x \\frac{\\mu_x}{2}\\|x\\|^2 + \\frac{1}{2\\gamma_i}\\left\\|x-(\\hat{x}_i-\\frac{\\gamma_iL_{xy}}{2}A\\tilde{y}_i)\\right\\|^2\\\\\r\n& = & \\frac{1}{1+\\gamma_i\\mu_x}\\hat{x}_i - \\frac{\\gamma_iL_{xy}}{2(1+\\gamma_i\\mu_x)}A\\tilde{y}_i\\nonumber\\\\\r\n& \\in & \\mathrm{Span}\\{\\hat{x}_i,A\\tilde{y}_i\\}.\\nonumber\r\n\\end{eqnarray}\r\n","start":39569,"end":40107}]},{"id":"S022","semantic_type":"derived_fact","kind":"derived_fact","labels":["defn:prox-closedForm-y"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"对偶块近端闭式","source_units":[{"raw_tex":"Similarly, for the $y$ block, we also have\r\n\\begin{eqnarray}\r\n\\label{defn:prox-closedForm-y}\r\n\\prox_{\\sigma_i g}(\\hat{y}_i+\\sigma_iA^\\top\\tilde{x}_i)\r\n%& = & \\argmin_y \\frac{\\mu_y}{2}\\|y\\|^2 + b^Ty + \\frac{1}{2\\sigma_i}\\left\\|y-(\\hat{y}_i+\\frac{\\sigma_iL_{xy}}{2}A^\\top \\tilde{x}_i)\\right\\|^2\\\\\r\n=  \\frac{\\hat{y}_i-\\sigma_i b}{1+\\sigma_i\\mu_y} +  \\frac{\\sigma_iL_{xy}}{2(1+\\sigma_i\\mu_x)}A\\tilde{x}_i \\in \\mathrm{Span}\\{\\hat{y}_i,A\\tilde{x}_i,b\\}.\r\n\\end{eqnarray}\r\n","start":40107,"end":40572}]},{"id":"S023","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"零初始点","source_units":[{"raw_tex":"Let us assume the initial point to be $x^0 = y^0 = 0$ ($\\cH_x^0 = \\cH_y^0 = \\{0\\}$) without loss of generality.\r\n","start":40572,"end":40685}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","labels":["lemma:subspace-prox"],"member_ids":["S024","S025","S026"],"declared_external":false,"source_only":false,"title":"近端搜索空间引理","source_units":[{"raw_tex":"Directly substituting \\eqref{defn:prox-closedForm-x} and \\ref{defn:prox-closedForm-y} into Definition \\eqref{defn:Prox-AlgoClass} yields\r\n\\begin{equation*}\r\n\\begin{cases}\r\n\\cH_x^1 \\subseteq \\mathrm{Span}\\{0\\}\\\\\r\n\\cH_y^1 \\subseteq \\mathrm{Span}\\{b\\}\r\n\\end{cases}\r\n\\quad\r\n\\begin{cases}\r\n\\cH_x^2 \\subseteq \\mathrm{Span}\\{Ab\\}\\\\\r\n\\cH_y^2 \\subseteq \\mathrm{Span}\\{b\\}\r\n\\end{cases}\r\n\\quad\r\n\\begin{cases}\r\n\\cH_x^3 \\subseteq \\mathrm{Span}\\{Ab\\}\\\\\r\n\\cH_y^3 \\subseteq \\mathrm{Span}\\{b,A^2b\\}\r\n\\end{cases}\r\n\\quad\r\n\\begin{cases}\r\n\\cH_x^4 \\subseteq \\mathrm{Span}\\{Ab, A^3b\\}\\\\\r\n\\cH_y^4 \\subseteq \\mathrm{Span}\\{b,A^2b\\}\r\n\\end{cases}\r\n\\ldots\r\n\\end{equation*}\r\n","start":40734,"end":41380},{"raw_tex":"Directly substituting \\eqref{defn:prox-closedForm-x} and \\ref{defn:prox-closedForm-y} into Definition \\eqref{defn:Prox-AlgoClass} yields\r\n\\begin{equation*}\r\n\\begin{cases}\r\n\\cH_x^1 \\subseteq \\mathrm{Span}\\{0\\}\\\\\r\n\\cH_y^1 \\subseteq \\mathrm{Span}\\{b\\}\r\n\\end{cases}\r\n\\quad\r\n\\begin{cases}\r\n\\cH_x^2 \\subseteq \\mathrm{Span}\\{Ab\\}\\\\\r\n\\cH_y^2 \\subseteq \\mathrm{Span}\\{b\\}\r\n\\end{cases}\r\n\\quad\r\n\\begin{cases}\r\n\\cH_x^3 \\subseteq \\mathrm{Span}\\{Ab\\}\\\\\r\n\\cH_y^3 \\subseteq \\mathrm{Span}\\{b,A^2b\\}\r\n\\end{cases}\r\n\\quad\r\n\\begin{cases}\r\n\\cH_x^4 \\subseteq \\mathrm{Span}\\{Ab, A^3b\\}\\\\\r\n\\cH_y^4 \\subseteq \\mathrm{Span}\\{b,A^2b\\}\r\n\\end{cases}\r\n\\ldots\r\n\\end{equation*}\r\nWe formally summarize this observation below: %in the Lemma \\ref{lemma:subspace-prox}.\r\n","start":40734,"end":41468},{"raw_tex":"\\begin{lemma}\r\n\t\\label{lemma:subspace-prox}\r\n\tFor problem \\eqref{prob:worst-proximal}, for any $k \\in \\mathbb{N}$, if the iterates are generated so that  $(x_k,y_k)\\in\\cH_x^k\\times\\cH_y^k$, with $\\cH_x^k$ and $\\cH_y^k$ defined by \\eqref{defn:Prox-AlgoClass}, then based on \\eqref{defn:prox-closedForm-x} and \\eqref{defn:prox-closedForm-y} the search subspaces satisfy\r\n\t$$\\cH_x^k \\subseteq \\begin{cases}\r\n\t\\{0\\}, &  k = 1\\\\\r\n\t\\mathrm{Span}\\left\\{A^{2i}(Ab): 0\\leq i\\leq \\left\\lfloor \\frac{k}{2}\\right\\rfloor -1 \\right\\}, & k\\geq 2\r\n\t\\end{cases} \\quad \\mbox{ and }\\quad\r\n\t\\cH_y^k \\subseteq \\mathrm{Span}\\left\\{A^{2i}b: 0\\leq i\\leq \\left\\lceil \\frac{k}{2}\\right\\rceil -1 \\right\\}.$$\r\n\\end{lemma}","start":41468,"end":42161}]},{"id":"S027","semantic_type":"derived_fact","kind":"derived_fact","labels":["defn:dual-function-prox"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"实例的对偶二次函数","source_units":[{"raw_tex":"Let us lower bound the dual gap, which is upper bounded by the whole duality gap. To achieve this, let us first write down the dual function of problem \\eqref{prob:worst-proximal} as\r\n\\begin{equation}\r\n\\label{defn:dual-function-prox}\r\n\\Psi(y) = \\min_{x} F(x,y) = -\\frac{1}{2}y^\\top \\left(\\frac{L_{xy}^2}{4\\mu_x}\\cdot A^2 + \\mu_y\\cdot I\\right)y - b^\\top y . \r\n\\end{equation}\r\n","start":42258,"end":42633}]},{"id":"S028","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"驻点刻画与近似路线","source_units":[{"raw_tex":"For this $\\mu_y$-strongly concave dual function, we can characterize the optimal solution $y^*$ directly by its KKT condition $\\nabla \\Psi(y^*) = 0$. However, the exact solution $y^*$ does not have a simple and clear form, so we choose to  characterize it by an approximate solution $\\hat y^*$.\r\n","start":42633,"end":42929}]},{"id":"S029","semantic_type":"lemma","kind":"lemma","labels":["eqn:apxsolu-proximal","lemma:apxsolu-proximal"],"member_ids":["S029","S033","S034","S035","S036","S037"],"declared_external":false,"source_only":false,"title":"几何型近似最优解","source_units":[{"raw_tex":"\\begin{lemma}[Approximate optimal solution]\r\n\t\\label{lemma:apxsolu-proximal}\r\n\tLet us assign the value of $b$ as $b := -\\frac{L_{xy}^2}{4\\mu_x}e_1$. Denote $\\alpha := \\frac{4\\mu_x\\mu_y}{L_{xy}^2}$, and let $q = \\half\\left((2+\\alpha) - \\sqrt{(2+\\alpha)^2 - 4}\\right)\\in(0,1)$\r\n\tbe the smallest root of the quadratic equation $1 - (2+\\alpha)q + q^2 = 0$. Then, an approximate optimal solution $\\hat y^*$ can be constructed as\r\n\t\\begin{equation}\r\n\t\\hat y^*_{i} = \\frac{q^i}{1-q} \\quad\\mbox{ for } \\quad i = 1,2,...,n.\r\n\t\\end{equation}\r\n\tThe approximation error can be bounded by\r\n\t\\begin{equation}\r\n\t\\label{eqn:apxsolu-proximal}\r\n\t\\|\\hat y^* - y^*\\| \\leq  \\frac{q^{n+1}}{\\alpha(1-q)},\r\n\t\\end{equation}\r\n\twhere $\\hat y^*_i$ is the $i$-th element of $\\hat y^*$. Note that $q<1$ and the lower bound is dimension-independent, hence we are free to choose $n$ to make the approximation error arbitrarily small.\r\n\\end{lemma}","start":42931,"end":43845},{"raw_tex":"\\begin{proof}\r\n\tFirst, let us substitute the value of $b$ into the KKT system $\\nabla \\Psi(y^*)=0$, by slight rearranging and scaling the terms, we get\r\n\t\\begin{equation*}\r\n\t\\left(A^2 + \\frac{4\\mu_x\\mu_y}{L_{xy}^2} I\\right)y^* = - \\frac{4\\mu_x}{L_{xy}^2}b.\r\n\t\\end{equation*}\r\n\tUsing the definition of $\\alpha$ and $b$, the equation becomes\r\n\t$$(A^2+\\alpha I)y^* = e_1.$$\r\n","start":43847,"end":44219},{"raw_tex":"\\begin{proof}\r\n\tFirst, let us substitute the value of $b$ into the KKT system $\\nabla \\Psi(y^*)=0$, by slight rearranging and scaling the terms, we get\r\n\t\\begin{equation*}\r\n\t\\left(A^2 + \\frac{4\\mu_x\\mu_y}{L_{xy}^2} I\\right)y^* = - \\frac{4\\mu_x}{L_{xy}^2}b.\r\n\t\\end{equation*}\r\n\tUsing the definition of $\\alpha$ and $b$, the equation becomes\r\n\t$$(A^2+\\alpha I)y^* = e_1.$$\r\n\tSubstituting the formula of $A^2$ in \\eqref{defn:A}, we expand the above equation as\r\n\t\\begin{eqnarray}\r\n\t\\label{defn:KKT-proximal}\r\n\t\\begin{cases}\r\n\t(1+\\alpha)y_1^* - y_2^* = 1\\\\\r\n\t-y_1^* + (2+\\alpha)y_2^* - y_3^* = 0\\\\\r\n\t\\qquad\\qquad\\quad\\vdots\\\\\r\n\t-y_{n-2}^* + (2+\\alpha)y_{n-1}^* - y_n^* = 0\\\\\r\n\t-y_{n-1}^* + (2+\\alpha)y_n^* = 0.\r\n\t\\end{cases}\r\n\t\\end{eqnarray}\r\n\tBy direct calculation, we can check that $\\hat y^*$ satisfies the first $n-1$ equations of the KKT system \\eqref{defn:KKT-proximal}. The last equation, however, is violated, but with a residual of size $q^{n+1}/(1-q)$.  In details,\r\n\t$$\\begin{cases}\r\n\t(A^2 + \\alpha\\cdot I)\\hat y^* = e_1 + \\frac{q^{n+1}}{1-q}\\cdot e_n\\\\\r\n\t(A^2 + \\alpha\\cdot I)y^* = e_1.\r\n\t\\end{cases}$$\r\n\tThis indicates that $\\hat y^* - y^* = \\frac{q^{n+1}}{1-q}\\cdot(A^2 + \\alpha I)^{-1} e_n$. Note that $\\alpha^{-1}I\\succeq (A^2 + \\alpha I)^{-1}\\succ 0$, we have the approximation error bounded by \\eqref{eqn:apxsolu-proximal}.\r\n\\end{proof}","start":43847,"end":45197},{"raw_tex":"\tSubstituting the formula of $A^2$ in \\eqref{defn:A}, we expand the above equation as\r\n\t\\begin{eqnarray}\r\n\t\\label{defn:KKT-proximal}\r\n\t\\begin{cases}\r\n\t(1+\\alpha)y_1^* - y_2^* = 1\\\\\r\n\t-y_1^* + (2+\\alpha)y_2^* - y_3^* = 0\\\\\r\n\t\\qquad\\qquad\\quad\\vdots\\\\\r\n\t-y_{n-2}^* + (2+\\alpha)y_{n-1}^* - y_n^* = 0\\\\\r\n\t-y_{n-1}^* + (2+\\alpha)y_n^* = 0.\r\n\t\\end{cases}\r\n\t\\end{eqnarray}\r\n","start":44219,"end":44586},{"raw_tex":"\tBy direct calculation, we can check that $\\hat y^*$ satisfies the first $n-1$ equations of the KKT system \\eqref{defn:KKT-proximal}. The last equation, however, is violated, but with a residual of size $q^{n+1}/(1-q)$.  In details,\r\n\t$$\\begin{cases}\r\n\t(A^2 + \\alpha\\cdot I)\\hat y^* = e_1 + \\frac{q^{n+1}}{1-q}\\cdot e_n\\\\\r\n\t(A^2 + \\alpha\\cdot I)y^* = e_1.\r\n\t\\end{cases}$$\r\n","start":44586,"end":44959},{"raw_tex":"\tThis indicates that $\\hat y^* - y^* = \\frac{q^{n+1}}{1-q}\\cdot(A^2 + \\alpha I)^{-1} e_n$. Note that $\\alpha^{-1}I\\succeq (A^2 + \\alpha I)^{-1}\\succ 0$, we have the approximation error bounded by \\eqref{eqn:apxsolu-proximal}.\r\n\\end{proof}","start":44959,"end":45197}]},{"id":"S030","semantic_type":"definition","kind":"definition","labels":["lemma:apxsolu-proximal"],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"参数与收缩因子","source_units":[{"raw_tex":"\tLet us assign the value of $b$ as $b := -\\frac{L_{xy}^2}{4\\mu_x}e_1$. Denote $\\alpha := \\frac{4\\mu_x\\mu_y}{L_{xy}^2}$, and let $q = \\half\\left((2+\\alpha) - \\sqrt{(2+\\alpha)^2 - 4}\\right)\\in(0,1)$\r\n\tbe the smallest root of the quadratic equation $1 - (2+\\alpha)q + q^2 = 0$. Then, an approximate optimal solution $\\hat y^*$ can be constructed as\r\n","start":43009,"end":43356}]},{"id":"S031","semantic_type":"definition","kind":"definition","labels":["lemma:apxsolu-proximal"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"线性项的指定","source_units":[{"raw_tex":"\tLet us assign the value of $b$ as $b := -\\frac{L_{xy}^2}{4\\mu_x}e_1$. Denote $\\alpha := \\frac{4\\mu_x\\mu_y}{L_{xy}^2}$, and let $q = \\half\\left((2+\\alpha) - \\sqrt{(2+\\alpha)^2 - 4}\\right)\\in(0,1)$\r\n","start":43009,"end":43207}]},{"id":"S032","semantic_type":"definition","kind":"definition","labels":["lemma:apxsolu-proximal"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"几何坐标近似向量","source_units":[{"raw_tex":"\t\\begin{equation}\r\n\t\\hat y^*_{i} = \\frac{q^i}{1-q} \\quad\\mbox{ for } \\quad i = 1,2,...,n.\r\n\t\\end{equation}\r\n","start":43356,"end":43464}]},{"id":"S052","semantic_type":"theorem","kind":"theorem","labels":["thm:proximal"],"member_ids":["S038","S040","S050","S051","S052"],"declared_external":false,"source_only":false,"title":"线性跨度近端下界","source_units":[{"raw_tex":" Note that in Lemma \\ref{lemma:apxsolu-proximal}, we have chosen  $b\\propto e_1$. By the zero-chain property in Proposition \\ref{prop:zero-chain} and Lemma \\ref{lemma:subspace-prox},  we can verify that the subspaces $\\cH_y^{2k-1}$ and $\\cH_y^{2k}$ satisfy\r\n\\begin{equation}\r\n\\label{eqn:subspace-prox}\r\n\\cH_y^{2k-1}, \\cH_y^{2k} \\subseteq \\mathrm{Span}\\{b,A^2b,...,A^{2(k-1)}b\\} =\\mathrm{Span}\\{e_1, e_2, ...,e_k\\}.\r\n\\end{equation}\r\nThis implies that for both $y^{2k}$ and $y^{2k-1}$, the only possible nonzero elements are the first $k$ ones, which again implies that the lower bound of $\\|y^{2k}-y^*\\|^2$ and $\\|y^{2k-1}-y^*\\|^2$ will be similar. For simplicity, we only discuss this lower bound  for $y^{2k}$. The counterpart for $y^{2k-1}$ can be obtained in a similar way.  Therefore, we have the following estimations.\r\n","start":45203,"end":46028},{"raw_tex":"This implies that for both $y^{2k}$ and $y^{2k-1}$, the only possible nonzero elements are the first $k$ ones, which again implies that the lower bound of $\\|y^{2k}-y^*\\|^2$ and $\\|y^{2k-1}-y^*\\|^2$ will be similar. For simplicity, we only discuss this lower bound  for $y^{2k}$. The counterpart for $y^{2k-1}$ can be obtained in a similar way.  Therefore, we have the following estimations.\r\n","start":45635,"end":46028},{"raw_tex":"Using Lemma \\ref{lemma:lowerbound-distance-prox} and \\eqref{defn:primal-gap-LB}, it is then straightforward to lower bound the duality gap by\r\n$$\\Delta(x^{2k},y^{2k})\\geq q^{2k}\\cdot\\frac{\\mu_y\\|y^*-y^0\\|^2}{32}.$$\r\n","start":47034,"end":47250},{"raw_tex":"\\begin{theorem}\r\n\t\\label{thm:proximal}\r\n Let the positive parameters $\\mu_x,\\mu_y>0$ and $L_{xy}> 0$ be given. For any integer $k$, there exists a problem instance from $\\cB(L_{xy},\\mu_x,\\mu_y)$ of form \\eqref{prob:worst-proximal}, with $n\\geq \\max\\left\\{2\\log_q\\left(\\frac{\\mu_x\\mu_y}{\\sqrt{2}L_{xy}^2}\\right), 4k\\right\\}$, where $A\\in\\mathbb{R}^{n\\times n}$ as defined in \\eqref{defn:A}, and $b = -\\frac{L_{xy}^2}{4\\mu_x}e_1$. For such a problem instance, any approximate solution $(\\tilde x^k, \\tilde y^k)\\in\\cH_x^k\\times\\cH_y^k$ generated by the proximal algorithm class under the linear span assumption \\eqref{eqn:Prox-AlgoClass}  satisfies\r\n\t\\begin{eqnarray}\r\n\t\\max_y F(\\tilde x^{k},y) - \\min_{x} F(x, \\tilde y^k)\\geq q^{k}\\cdot\\frac{\\mu_y\\|y^*-y^0\\|^2}{32} \\quad\\mbox{ and }\\quad \\|\\tilde y^k-y^*\\|^2\\geq q^{k}\\cdot\\frac{\\|y^*-y^0\\|^2}{16},\r\n\t\\end{eqnarray}\r\n\twhere $q = 1+\\frac{2\\mu_x\\mu_y}{L_{xy}^2} - 2\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}$.\r\n\\end{theorem}","start":47304,"end":48312}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:subspace-prox"],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"对偶空间的坐标支撑","source_units":[{"raw_tex":" Note that in Lemma \\ref{lemma:apxsolu-proximal}, we have chosen  $b\\propto e_1$. By the zero-chain property in Proposition \\ref{prop:zero-chain} and Lemma \\ref{lemma:subspace-prox},  we can verify that the subspaces $\\cH_y^{2k-1}$ and $\\cH_y^{2k}$ satisfy\r\n\\begin{equation}\r\n\\label{eqn:subspace-prox}\r\n\\cH_y^{2k-1}, \\cH_y^{2k} \\subseteq \\mathrm{Span}\\{b,A^2b,...,A^{2(k-1)}b\\} =\\mathrm{Span}\\{e_1, e_2, ...,e_k\\}.\r\n\\end{equation}\r\n","start":45203,"end":45635}]},{"id":"S041","semantic_type":"lemma","kind":"lemma","labels":["eqn:lowB-dis-prox","lemma:lowerbound-distance-prox"],"member_ids":["S041","S042","S043","S044","S045","S046","S047","S048","S049"],"declared_external":false,"source_only":false,"title":"对偶解距离下界","source_units":[{"raw_tex":"\\begin{lemma}\r\n\t\\label{lemma:lowerbound-distance-prox}\r\n\tAssume $k\\leq \\frac{n}{2}$ and $n\\geq2\\log_q\\left(\\frac{\\alpha}{4\\sqrt{2}}\\right)$. Then\r\n\t\\begin{equation}\r\n\t\\label{eqn:lowB-dis-prox}\r\n\t\\|y^{2k}-y^*\\|^2\\geq \\frac{q^{2k}}{16}\\|y^0-y^*\\|^2\r\n\t\\end{equation}\r\n\twhere $y^0 = 0$ is the initial solution.\r\n\\end{lemma}","start":46030,"end":46349},{"raw_tex":"\\begin{proof}\r\n\tBy the subspace characterization \\eqref{eqn:subspace-prox}, we have\r\n\t$$\\|y^{2k}-\\hat y^*\\| \\geq \\sqrt{\\sum^n_{j = k+1} (\\hat y^*_j)^2} = \\frac{q^{k}}{1-q}\\sqrt{q^2 + q^4 + \\cdots + q^{2(n-k)}}\\geq \\frac{q^k}{\\sqrt{2}}\\|\\hat y^*\\|= \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|,$$\r\n\twhere the last inequality is due to the fact that $q< 1$, $k\\leq \\frac{n}{2}$, and $y^0 = 0$.\r\n If we choose $n$ to be large enough, then $\\hat y^*$ and $y^*$ can be made arbitrarily close to each other. Hence we can transform the above inequality to  \\eqref{eqn:lowB-dis-prox}. More details of this derivation can be found in Appendix \\ref{appendix-lowerbound-distance-prox}.\r\n\\end{proof}","start":46351,"end":47030},{"raw_tex":"\tBy the subspace characterization \\eqref{eqn:subspace-prox}, we have\r\n\t$$\\|y^{2k}-\\hat y^*\\| \\geq \\sqrt{\\sum^n_{j = k+1} (\\hat y^*_j)^2} = \\frac{q^{k}}{1-q}\\sqrt{q^2 + q^4 + \\cdots + q^{2(n-k)}}\\geq \\frac{q^k}{\\sqrt{2}}\\|\\hat y^*\\|= \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|,$$\r\n\twhere the last inequality is due to the fact that $q< 1$, $k\\leq \\frac{n}{2}$, and $y^0 = 0$.\r\n","start":46366,"end":46736},{"raw_tex":" If we choose $n$ to be large enough, then $\\hat y^*$ and $y^*$ can be made arbitrarily close to each other. Hence we can transform the above inequality to  \\eqref{eqn:lowB-dis-prox}. More details of this derivation can be found in Appendix \\ref{appendix-lowerbound-distance-prox}.\r\n\\end{proof}","start":46736,"end":47030},{"raw_tex":"\\section{Proof of Lemma \\ref{lemma:lowerbound-distance-prox}}\r\n\\label{appendix-lowerbound-distance-prox}\r\nBy the subspace characterization \\eqref{eqn:subspace-prox}, we have\r\n$$\\|y^{2k}-\\hat y^*\\| \\geq \\sqrt{\\sum^n_{j = k+1} (\\hat y^*_j)^2} = \\frac{q^{k}}{1-q}\\sqrt{q^2 + q^4 + \\cdots + q^{2(n-k)}}\\geq \\frac{q^k}{\\sqrt{2}}\\|\\hat y^*\\|= \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|,$$\r\nwhere the last inequality is due to the fact that $q\\leq 1, k\\leq \\frac{n}{2}$ and $y^0 = 0$. Note that by Lemma \\ref{lemma:apxsolu-proximal}, if we require $n\\geq 2\\log_q\\left(\\frac{\\alpha}{4\\sqrt{2}}\\right)$, then we can guarantee that \\begin{equation}\r\n","start":75790,"end":76424},{"raw_tex":"\\section{Proof of Lemma \\ref{lemma:lowerbound-distance-prox}}\r\n\\label{appendix-lowerbound-distance-prox}\r\nBy the subspace characterization \\eqref{eqn:subspace-prox}, we have\r\n$$\\|y^{2k}-\\hat y^*\\| \\geq \\sqrt{\\sum^n_{j = k+1} (\\hat y^*_j)^2} = \\frac{q^{k}}{1-q}\\sqrt{q^2 + q^4 + \\cdots + q^{2(n-k)}}\\geq \\frac{q^k}{\\sqrt{2}}\\|\\hat y^*\\|= \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|,$$\r\nwhere the last inequality is due to the fact that $q\\leq 1, k\\leq \\frac{n}{2}$ and $y^0 = 0$. Note that by Lemma \\ref{lemma:apxsolu-proximal}, if we require $n\\geq 2\\log_q\\left(\\frac{\\alpha}{4\\sqrt{2}}\\right)$, then we can guarantee that \\begin{equation}\r\n\\label{eqn:err-n-proximal}\r\n\\|\\hat y^* - y^*\\| \\leq \\frac{q^{n+1}}{\\alpha(1-q)}\\leq  \\frac{q^{\\frac{n}{2}}}{\\alpha}\\cdot q^k\\cdot\\frac{q}{(1-q)} \\leq \\frac{1}{4}\\cdot \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|\\quad\\mbox{ for }\\quad \\forall 1\\leq k\\leq n/2,\r\n\\end{equation}\r\nwhere the last inequality is due to $\\frac{q^{\\frac{n}{2}}}{\\alpha}\\leq\\frac{1}{4\\sqrt{2}}$ and $q/(1-q)\\leq \\|y^0-\\hat y^*\\|$. Therefore, we have\r\n\\begin{eqnarray}\r\n\\label{eqn:dist-lower-proximal}\r\n\\|y^{2k}-y^*\\|^2 & \\geq & (\\|y^{2k}-\\hat y^*\\|-\\|\\hat y^*- y^*\\|)^2\\\\\r\n& \\geq & \\|y^{2k}-\\hat y^*\\|^2 - 2\\|y^{2k}-\\hat y^*\\|\\|\\hat y^*- y^*\\|\\nonumber\\\\\r\n& \\geq & \\min_t\\left\\{t^2 - 2\\|\\hat y^*- y^*\\|t: t\\geq \\delta_k:= \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|\\right\\}\\nonumber\\\\\r\n& = & \\delta_k(\\delta_k - 2\\|\\hat y^*- y^*\\|)\\nonumber\\\\\r\n& \\geq & \\frac{1}{2}\\delta_k^2 = \\frac{q^{2k}}{4}\\|y^0-\\hat y^*\\|^2, \\nonumber\r\n\\end{eqnarray}\r\nwhere the fourth line is due to that $d(t^2 - 2\\|\\hat y^*-y^*\\|t)/dt = 2(t-\\|\\hat y^*-y^*\\|)\\geq0$ when $t\\geq\\delta_k$. Hence the quadratic function is monotonically increasing in the considered interval. In addition, we also have\r\n$$\\|y^0-y^*\\| \\leq \\|y^0-\\hat y^*\\| + \\|\\hat y^*-y^*\\|\\leq \\|y^0-\\hat y^*\\| + \\frac{q^n}{\\alpha}\\cdot\\frac{q}{1-q}\\leq (1 + q^n/\\alpha)\\|y^0-\\hat y^*\\|\\leq 2\\|y^0-\\hat y^*\\|,$$\r\nwhere the third inequality is due to that $\\|y^0-\\hat y^*\\|\\geq \\hat y^*_1 = q/(1-q)$. For the last inequality, if $\\alpha\\geq 1$, then $q^n/\\alpha<1$; if $\\alpha\\leq 1$, then  $q^n/\\alpha\\leq \\alpha/32\\leq1$ since $n\\geq 2\\log_q\\left(\\frac{\\alpha}{4\\sqrt{2}}\\right)$. Combining the above two inequalities,  the desired bound \\eqref{eqn:lowB-dis-prox} follows.\r\n","start":75790,"end":78095},{"raw_tex":"where the last inequality is due to the fact that $q\\leq 1, k\\leq \\frac{n}{2}$ and $y^0 = 0$. Note that by Lemma \\ref{lemma:apxsolu-proximal}, if we require $n\\geq 2\\log_q\\left(\\frac{\\alpha}{4\\sqrt{2}}\\right)$, then we can guarantee that \\begin{equation}\r\n\\label{eqn:err-n-proximal}\r\n\\|\\hat y^* - y^*\\| \\leq \\frac{q^{n+1}}{\\alpha(1-q)}\\leq  \\frac{q^{\\frac{n}{2}}}{\\alpha}\\cdot q^k\\cdot\\frac{q}{(1-q)} \\leq \\frac{1}{4}\\cdot \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|\\quad\\mbox{ for }\\quad \\forall 1\\leq k\\leq n/2,\r\n\\end{equation}\r\nwhere the last inequality is due to $\\frac{q^{\\frac{n}{2}}}{\\alpha}\\leq\\frac{1}{4\\sqrt{2}}$ and $q/(1-q)\\leq \\|y^0-\\hat y^*\\|$. Therefore, we have\r\n","start":76168,"end":76840},{"raw_tex":"\\begin{eqnarray}\r\n\\label{eqn:dist-lower-proximal}\r\n\\|y^{2k}-y^*\\|^2 & \\geq & (\\|y^{2k}-\\hat y^*\\|-\\|\\hat y^*- y^*\\|)^2\\\\\r\n& \\geq & \\|y^{2k}-\\hat y^*\\|^2 - 2\\|y^{2k}-\\hat y^*\\|\\|\\hat y^*- y^*\\|\\nonumber\\\\\r\n& \\geq & \\min_t\\left\\{t^2 - 2\\|\\hat y^*- y^*\\|t: t\\geq \\delta_k:= \\frac{q^k}{\\sqrt{2}}\\|y^0-\\hat y^*\\|\\right\\}\\nonumber\\\\\r\n& = & \\delta_k(\\delta_k - 2\\|\\hat y^*- y^*\\|)\\nonumber\\\\\r\n& \\geq & \\frac{1}{2}\\delta_k^2 = \\frac{q^{2k}}{4}\\|y^0-\\hat y^*\\|^2, \\nonumber\r\n\\end{eqnarray}\r\nwhere the fourth line is due to that $d(t^2 - 2\\|\\hat y^*-y^*\\|t)/dt = 2(t-\\|\\hat y^*-y^*\\|)\\geq0$ when $t\\geq\\delta_k$. Hence the quadratic function is monotonically increasing in the considered interval. In addition, we also have\r\n","start":76840,"end":77555},{"raw_tex":"$$\\|y^0-y^*\\| \\leq \\|y^0-\\hat y^*\\| + \\|\\hat y^*-y^*\\|\\leq \\|y^0-\\hat y^*\\| + \\frac{q^n}{\\alpha}\\cdot\\frac{q}{1-q}\\leq (1 + q^n/\\alpha)\\|y^0-\\hat y^*\\|\\leq 2\\|y^0-\\hat y^*\\|,$$\r\nwhere the third inequality is due to that $\\|y^0-\\hat y^*\\|\\geq \\hat y^*_1 = q/(1-q)$. For the last inequality, if $\\alpha\\geq 1$, then $q^n/\\alpha<1$; if $\\alpha\\leq 1$, then  $q^n/\\alpha\\leq \\alpha/32\\leq1$ since $n\\geq 2\\log_q\\left(\\frac{\\alpha}{4\\sqrt{2}}\\right)$. Combining the above two inequalities,  the desired bound \\eqref{eqn:lowB-dis-prox} follows.\r\n","start":77555,"end":78095}]},{"id":"S053","semantic_type":"proposition","kind":"proposition","labels":["eqn:lowerbound-proximal-1stAlgo","remark:complexity-proximal"],"member_ids":["S053","S054","S055","S056"],"declared_external":false,"source_only":false,"title":"收缩下界转为迭代复杂度","source_units":[{"raw_tex":"\\begin{proposition}\r\n\t\\label{remark:complexity-proximal}\r\n\tUnder the same set of assumptions of Theorem \\ref{thm:proximal}, if we require the duality gap to be bounded by $\\epsilon$, the number of iterations needed is at least\r\n\t\\begin{equation}\r\n\t\\label{eqn:lowerbound-proximal-1stAlgo}\r\n\tk\\geq \\ln\\left(\\frac{\\mu_y\\|y^*-y^0\\|^2}{32\\epsilon}\\right)/\\ln(q^{-1}) = \\Omega\\left(\\sqrt{\\frac{L_{xy}^2}{\\mu_x\\mu_y}}\\cdot\\ln\\left(\\frac{1}{\\epsilon}\\right)\\right).\r\n\t\\end{equation}\r\n\\end{proposition}","start":48314,"end":48807},{"raw_tex":"\\section{Proof of Proposition \\ref{remark:complexity-proximal}}\r\n\\label{appendix:r-complexity-proximal}\r\nHere we only prove the last inequality of \\eqref{eqn:lowerbound-proximal-1stAlgo}. Due to the fact that $(\\ln(1+z))^{-1}\\geq 1/z$ for $\\forall z>0$, we know\r\n\\begin{eqnarray*}\r\n(\\ln(q^{-1}))^{-1} & = & (\\ln(1 + (1-q)/q))^{-1} \\geq \\frac{q}{1-q}\\\\\r\n& = &   \\frac{1+\\frac{2\\mu_x\\mu_y}{L_{xy}^2} - 2\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}}{2\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}-\\frac{2\\mu_x\\mu_y}{L_{xy}^2}}\\\\\r\n& = & \\frac{\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}-\\frac{\\mu_x\\mu_y}{L_{xy}^2}}{\\frac{2\\mu_x\\mu_y}{L_{xy}^2}}\\\\\r\n& = &  \\half\\sqrt{\\frac{L_{xy}^2}{\\mu_x\\mu_y} + 1} - \\frac{1}{2},\\\\\r\n& = & \\Omega\\left(\\sqrt{\\frac{L_{xy}^2}{\\mu_x\\mu_y}}\\right)\r\n\\end{eqnarray*}\r\nwhich completes the proof.\r\n\r\n","start":78095,"end":79022},{"raw_tex":"Here we only prove the last inequality of \\eqref{eqn:lowerbound-proximal-1stAlgo}. Due to the fact that $(\\ln(1+z))^{-1}\\geq 1/z$ for $\\forall z>0$, we know\r\n\\begin{eqnarray*}\r\n(\\ln(q^{-1}))^{-1} & = & (\\ln(1 + (1-q)/q))^{-1} \\geq \\frac{q}{1-q}\\\\\r\n","start":78200,"end":78448},{"raw_tex":"& = &   \\frac{1+\\frac{2\\mu_x\\mu_y}{L_{xy}^2} - 2\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}}{2\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}-\\frac{2\\mu_x\\mu_y}{L_{xy}^2}}\\\\\r\n& = & \\frac{\\sqrt{\\left(\\frac{\\mu_x\\mu_y}{L_{xy}^2}\\right)^2 + \\frac{\\mu_x\\mu_y}{L_{xy}^2}}-\\frac{\\mu_x\\mu_y}{L_{xy}^2}}{\\frac{2\\mu_x\\mu_y}{L_{xy}^2}}\\\\\r\n& = &  \\half\\sqrt{\\frac{L_{xy}^2}{\\mu_x\\mu_y} + 1} - \\frac{1}{2},\\\\\r\n& = & \\Omega\\left(\\sqrt{\\frac{L_{xy}^2}{\\mu_x\\mu_y}}\\right)\r\n\\end{eqnarray*}\r\n","start":78448,"end":78992}]},{"id":"S057","semantic_type":"definition","kind":"definition","labels":["prob:rotate"],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"正交旋转后的实例","source_units":[{"raw_tex":"Consider the bilinear problem class $\\cB(L_{xy},\\mu_x,\\mu_y)$ and the corresponding worst case problem \\eqref{prob:worst-proximal} with $F(x,y) = \\frac{\\mu_x}{2}\\|x\\|^2 + \\frac{L_{xy}}{2}x^\\top A y - \\frac{\\mu_y}{2}\\|y\\|^2 - b^\\top y$,\r\nwhere $A$ and $b$ are defined in accordance with Theorem \\ref{thm:proximal}. We define the orthogonally rotated problem as \r\n\\begin{equation}\r\n\\label{prob:rotate}\r\n\\min_x\\max_y F_{U,V}(x,y) : = F(Ux,Vy) = \\frac{\\mu_x}{2}\\|x\\|^2 + \\frac{L_{xy}}{2}x^\\top U^\\top AV y - \\frac{\\mu_y}{2}\\|y\\|^2 - b^\\top Vy,\r\n\\end{equation}\r\n","start":49395,"end":49952}]},{"id":"S058","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"旋转保持问题类","source_units":[{"raw_tex":"where $U,V$ are two orthogonal matrices. Therefore, it is clear that $F_{U,V}\\in\\cB(L_{xy},\\mu_x,\\mu_y)$. Let $(x^*, y^*)$ be the saddle point of $F(x,y)$, then it is clear that the saddle point of $F_{U,V}(x,y)$ is $(\\bar x^*,\\bar y^*) = (U^\\top  x^*,V^\\top y^*).$ Consequently, the lower bound for the general proximal algorithm class is characterized by the following theorem. \r\n","start":49952,"end":50334}]},{"id":"S059","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"鞍点随旋转变换","source_units":[{"raw_tex":"where $U,V$ are two orthogonal matrices. Therefore, it is clear that $F_{U,V}\\in\\cB(L_{xy},\\mu_x,\\mu_y)$. Let $(x^*, y^*)$ be the saddle point of $F(x,y)$, then it is clear that the saddle point of $F_{U,V}(x,y)$ is $(\\bar x^*,\\bar y^*) = (U^\\top  x^*,V^\\top y^*).$ Consequently, the lower bound for the general proximal algorithm class is characterized by the following theorem. \r\n","start":49952,"end":50334}]},{"id":"S060","semantic_type":"lemma","kind":"lemma","labels":["appdx:gen-prox","lemma:yyx"],"member_ids":["S060"],"declared_external":true,"declaration_text":["原始先决条件列表明确允许的 YYXu 文献 Lemma 3.1；原文 B0146 在第956–959行重述，其证明未提供。"],"source_only":false,"title":"保持子空间的旋转引理","source_units":[{"raw_tex":"\tBefore proceeding the proof, let us first quote a lemma from \\cite{YYXu}. \r\n\t\\begin{lemma}[Lemma 3.1, \\cite{YYXu}] \r\n\t\t\\label{lemma:yyx} \r\n\t\tLet $\\cX\\subsetneqq\\bar{\\cX}\\subseteqq\\R^{p}$ be two linear subspaces. Then for any $\\bar x\\in\\R^p$, there exists an orthogonal matrix $\\Gamma\\in\\R^{p\\times p}$ s.t. $\\Gamma x = x, \\forall x\\in\\cX$ and $\\Gamma \\bar x\\in\\bar{\\cX}$.\r\n\t\\end{lemma}\r\n","start":79098,"end":79486}]},{"id":"S061","semantic_type":"derived_fact","kind":"derived_fact","labels":["appdx:gen-prox"],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"转置也固定相同向量","source_units":[{"raw_tex":"\tNote that for an orthogonal matrix $\\Gamma$, if $\\Gamma x = x$, then we also have $\\Gamma^\\top x = x$.  Now let us start our proof of Theorem \\ref{theorem:gen-prox}.\r\n\r\n","start":79486,"end":79656}]},{"id":"S062","semantic_type":"notation","kind":"notation","labels":["appdx:gen-prox"],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"附录证明的归属标题","source_units":[{"raw_tex":"\\section{Proof of Theorem \\ref{theorem:gen-prox}}\r\n\\label{appdx:gen-prox} \r\n","start":79022,"end":79098}]},{"id":"S063","semantic_type":"theorem","kind":"theorem","labels":["theorem:gen-prox"],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"一般近端算法的复杂度下界","source_units":[{"raw_tex":"\\begin{theorem}\r\n\\label{theorem:gen-prox}\r\nLet $\\cA$ be any algorithm from the general proximal algorithm class decribed in Definition \\ref{defn:Det-Prox-AlgoClass}. We assume the dimension $n$ is sufficiently large for simplicity. For any integer $k$, then there exist orthogonal matrices $U,V$ s.t. $F_{U,V}\\in\\cB(L_{xy},\\mu_x,\\mu_y)$, when applying $\\cA$ to $F_{U,V}$ with initial solution $(x^0,y^0) = (0,0)$, the iterates and output satisfies\r\n$$\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1}\\qquad\\mbox{and}\\qquad(\\tilde x^k,\\tilde y^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1},$$\r\nwhere $\\cH_x^i,\\cH_y^i$ are defined by Lemma \\ref{lemma:subspace-prox}. Consequenty, by Theorem \\ref{thm:proximal}, \r\n\\begin{eqnarray*}\r\n\\|\\tilde y^k-V^\\top  y^*\\|^2\\geq \\frac{q^{4k+2}}{16}\\|y^*-y^0\\|^2\r\n\\end{eqnarray*}\r\nwhere $q$ is given in Theorem \\ref{thm:proximal}. As a result, it takes $\\Omega\\left(\\sqrt{\\frac{L^2_{xy}}{\\mu_x\\mu_y}}\\cdot\\log\\left(\\frac{1}{\\epsilon}\\right)\\right)$ iterations to output a solution with $O(\\epsilon)$ duality gap.\r\n\\end{theorem}","start":50334,"end":51429}]},{"id":"S064","semantic_type":"proof","kind":"proof","labels":["thm:gen-prox-1","thm:gen-prox-2","thm:gen-prox-3","thm:gen-prox-4","thm:gen-prox-5","thm:gen-prox-6","thm:gen-prox-7","thm:gen-prox-8"],"member_ids":["S064"],"declared_external":false,"source_only":false,"title":"主定理完整作者证明","source_units":[{"raw_tex":"For the proof of this theorem, we only need to construct the orthogonal matrices $U,V$ such that when the algorithm $\\cA$ is applied to $F_{U,V}$, the subspace inclusion argument $\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1}$ and $(\\tilde x^k,\\tilde y^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1}$ holds. As a result, \r\n\\begin{eqnarray*}\r\n\t\\|\\tilde y^k-V^\\top  y^*\\|^2 = \\|V\\tilde y^k- y^*\\|^2\\geq\\min_{y\\in\\cH_y^{4k+1}}\\|y-y^*\\|^2\\geq \\frac{q^{4k+2}}{16}\\|y^*-y^0\\|^2.\r\n\\end{eqnarray*}\r\nWith this argument, the latter results follow directly from the discussion of Theorem \\ref{thm:proximal}. The proof of this theorem is presented in Appendix \\ref{appdx:gen-prox}.\r\n","start":51431,"end":52140},{"raw_tex":"\\begin{proof}\r\n\t\tTo prove this theorem, we only need to show\r\n\t\t$$\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1}\\qquad\\mbox{and}\\qquad(\\tilde x^k,\\tilde y^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1}.$$\r\n\t\tWe separate the proof into two parts.\r\n\t\t\r\n\t\t\\textbf{Part I. There exist orthogonal matrices $\\hat U$, $\\hat V$ s.t. when $\\cA$ is applied to the rotated instance $F_{\\hat U,\\hat V}$, $\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq \\hat U^\\top\\cH_x^{4k-1}\\times \\hat V^\\top\\cH_y^{4k-1}.$}\r\n\t\r\n\t\tLet $\\theta = (L_{xy},\\mu_x,\\mu_y)$ be the set of algorithmic parameters. To prove the result, let us construct the worst-case function $F_{U,V}$ in a recursive way. \t \\\\\r\n\t\t\\textbf{Case $k = 1$:} Let us define $U_0 = V_0 = I$. When $\\cA$ is applied to the function $F_{U_0,V_0}\\in\\cB(L_{xy}, \\mu_x, \\mu_y)$, the iterate sequence is \r\n\t\t$(x_{0}^0, y_{0}^0) = (0, 0)$ and \r\n\t\t$$\\begin{cases}\r\n\t\tu^1_0 = \\cA_u^1(\\theta; x_0^0, U_0^\\top A V_0 y_0^0), \\qquad (x^1_0, \\tilde x^1_0) = \\cA_x^1(\\theta; x^0_0,  U_0^\\top A V_0 y_0^0, \\mathbf{prox}_{\\gamma_1f}(u^1_0)),\\\\\r\n\t\tv^1_0 = \\cA_v^1(\\theta; y_0^0, V^\\top_0 A U_0x_0^0),\\qquad\r\n\t\t(y^1_0, \\tilde y^1_0) = \\cA_y^1(\\theta; y^0_0, V^\\top_0 A U_0x_0^0, \\mathbf{prox}_{\\sigma_1g}(v^1_0)).\r\n\t\t\\end{cases}$$\r\n\t\tBy Lemma \\ref{lemma:yyx}, there exists orthogonal matrices $\\Gamma_x^0$ and $\\Gamma_y^0$ such that \r\n\t\t$\\Gamma_x^0x^1_0\\in\\cH_x^3 = \\mathrm{Span}\\{Ab\\}$, $\\Gamma_y^0y^1_0\\in\\cH_y^3=\\mathrm{Span}\\{b, A^2b\\}$, and $\\Gamma_y^0b = (\\Gamma_y^0)^\\top b = b.$ That is \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-1}\r\n\t\tx_0^1\\in U_1^\\top\\cH_x^3,\\qquad\\mbox{and}\\qquad y_0^1\\in V_1^\\top\\cH_y^3, \\quad V_1 b = V_1^\\top b = b,\r\n\t\t\\end{equation}\t\r\n\t\twhere $U_1 = U_0\\Gamma_x^0$ and $V_1 = V_0\\Gamma_y^0$. \r\n\t\r\n\t\tNow we prove that when we apply the algorithm $\\cA$ to $F_{U_1,V_1}$, the generated iterates $\\{(x^0_1, y^0_1), (x^1_1, y^1_1)\\}$ satisfy that\r\n\t\t$(x^0_1, y^0_1) = (0, 0)$ and  $(x^1_1, y^1_1) = (x^1_0, y^1_0)$. That is, the first two iterates generated by $\\cA$ is completely the same for $F_{U_0,V_0}$ and $F_{U_1,V_1}$. The reason is because \r\n\t\t$u^1_1 = \\cA_u^1(\\theta; x^0_1, U_1^\\top AV_1y^0_1) = \\cA_u^1(\\theta; 0, 0) = \\cA_u^1(\\theta; x^0_0, U_0^\\top AV_0y^0_0)  = u^1_0$, \r\n\t\ttherefore \r\n\t\t\\begin{eqnarray*}\r\n\t\t\t(x^1_1, \\tilde x^1_1) & = & \\cA_x^1(\\theta; x^0_1, U_1^\\top A V_1 y_1^0, \\mathbf{prox}_{\\gamma_1f}(u^1_1)) \\\\\r\n\t\t\t& = & \\cA_x^1(\\theta; 0, 0, \\mathbf{prox}_{\\gamma_1f}(u^1_1)) \\\\\r\n\t\t\t& = & \\cA_x^1(\\theta; x^0_0, U_0^\\top A V_0 y_0^0, \\mathbf{prox}_{\\gamma_1f}(u^1_0)) \\\\\r\n\t\t\t& = & (x^1_0, \\tilde x^1_0).\r\n\t\t\\end{eqnarray*}\r\n\t\tThrough similar argument, we know $(y^1_1,\\tilde y^1_1) = (y^1_0,\\tilde y^1_0)$. Therefore, \\eqref{thm:gen-prox-1} indicates that \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-2}\r\n\t\tx_1^1\\in U_1^\\top\\cH_x^3,\\qquad\\mbox{and}\\qquad y_1^1\\in V_1^\\top\\cH_y^3, \\quad V_1 b = V_1^\\top b = b\\in V_1^\\top\\cH_y^3.\r\n\t\t\\end{equation}\r\n\t\t\\textbf{Case $k=2$.} For the ease of the readers to follow, we perform one extra step of discussion for $k=2$, before presenting the construction on general $k$. \r\n\t    \r\n\t\tFor the problem instance $F_{U_1,V_1}$, the iterates generated by $\\cA$ are $(x_{1}^0, y_{1}^0) = (0, 0)$ and \r\n\t\t$$\\begin{cases}\r\n\t\tu^1_1 = \\cA_u^1(\\theta; x_1^0, U_1^\\top A V_1 y_1^0), \\qquad (x^1_1, \\tilde x^1_1) = \\cA_x^1(\\theta; x^0_1,  U_1^\\top A V_1 y_1^0, \\mathbf{prox}_{\\gamma_1f}(u^1_1)),\\\\\r\n\t\tv^1_1 = \\cA_v^1(\\theta; y_1^0, V^\\top_1 A U_1x_1^0),\\qquad\r\n\t\t(y^1_1, \\tilde y^1_1) = \\cA_y^1(\\theta; y^0_1, V^\\top_1 A U_1x_1^0, \\mathbf{prox}_{\\sigma_1g}(v^1_1)).\\\\\r\n\t\t\\end{cases}$$\r\n\t\t$$\\begin{cases}\r\n\t\tu^2_1 = \\cA_u^2(\\theta; x_1^0, U_1^\\top A V_1 y_1^0, x_1^1, U_1^\\top A V_1 y_1^1), \\quad (x^2_1, \\tilde x^2_1) = \\cA_x^2(\\theta; x^0_1,  U_1^\\top A V_1 y_1^0, x_1^1, U_1^\\top A V_1 y_1^1, \\mathbf{prox}_{\\gamma_2f}(u^2_1)),\\\\\r\n\t\tv^2_1 = \\cA_v^2(\\theta; y_1^0, V^\\top_1 A U_1x_1^0, y_1^1, V^\\top_1 A U_1x_1^1),\\quad\r\n\t\t(y^2_1, \\tilde y^2_1) = \\cA_y^2(\\theta; y^0_1, V^\\top_1 A U_1x_1^0, y_1^1, V^\\top_1 A U_1x_1^1, \\mathbf{prox}_{\\sigma_2g}(v^2_1)).\\\\\r\n\t\t\\end{cases}$$\r\n\t\tNote that  $x_1^1 \\in U_1^\\top\\cH_x^3 \\subsetneqq U_1^\\top\\cH_x^5\\subsetneqq U_1^\\top\\cH_x^7$ and $\\{y_1^1,b\\}\\subsetneqq V_1^\\top\\cH_y^3\\subsetneqq V_1^\\top\\cH_y^5\\subsetneqq V_1^\\top\\cH_y^7$. Therefore, there exist orthogonal matrices $\\Gamma_x^1$ and $\\Gamma_y^1$ such that \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-3}\r\n\t\t\\begin{cases}\r\n\t\t\\Gamma_x^1x = (\\Gamma_x^1)^\\top x = x, \\,\\,\\forall x\\in U_1^\\top\\cH_x^5,\\,\\,\\,\\, \\Gamma_x^1 x_1^2\\in U_1^\\top\\cH_x^7,\\\\\r\n\t\t\\Gamma_y^1y \\,= (\\Gamma_y^1)^\\top y = y, \\,\\,\\,\\forall y\\in V_1^\\top\\cH_y^5,\\,\\,\\,\\, \\Gamma_y^1 y_1^2\\in V_1^\\top\\cH_y^7.\r\n\t\t\\end{cases}\r\n\t\t\\end{equation}\r\n\t\tNow, let us define \r\n\t\t$$U_2 = U_1\\Gamma_x^1\\qquad\\mbox{and}\\qquad V_2 = V_1\\Gamma_y^1.$$\r\n\t\tNow we prove that if $\\cA$ is applied to $F_{U_2,V_2}$, the generated iterates $\\{(x_{2}^0, y_{2}^0), (x_{2}^1, y_{2}^1), (x_{2}^2, y_{2}^2)\\}$ satisfy $(x_{2}^0, y_{2}^0) = (0, 0)$, $(x_{2}^1, y_{2}^1) = (x_{1}^1, y_{1}^1)$, and  $(x_{2}^2, y_{2}^2) = (x_{1}^2, y_{1}^2)$. The argument for $(x_{2}^1, y_{2}^1) = (x_{1}^1, y_{1}^1)$ is almost the same as  that of the case $k=1$. We only provide the proof for $(x_{2}^2, y_{2}^2) = (x_{1}^2, y_{1}^2)$.\r\n\t\r\n\tNext, we need to show $u_2^2 = u_1^2$, which can be proved by arguing that all the inputs to $\\cA_u^2$ are the same for both $u_2^2$ and $u_1^2$. First, it is straightforward that $x_1^0 = 0 = x^0_2, U_1^\\top AV_1 y_1^0 = 0 = U_2^\\top AV_2 y_2^0$. By previous argument $x_2^1 = x_1^1$.  Finally, consider the last input $U_2^\\top AV_2 y_2^1$, because $y_2^1 = y_1^1\\in V_1^\\top\\cH_y^3\\subsetneqq V_1^\\top\\cH_y^5$, we have $\\Gamma_y^1 y_2^1 = y_2^1 = y_1^1\\in V_1^\\top\\cH_y^3.$ Then $V_2y_2^1 = V_1\\Gamma_y^1y_2^1\\in V_1V_1^\\top\\cH_y^3 = \\cH_y^3.$ Therefore $U_1^\\top AV_2y_2^1\\in U_1^\\top A\\cH_y^3 = U_1^\\top\\cH_x^5$ and \r\n\t\t$$U_2^\\top AV_2y_2^1 = \\Gamma_x^1U_1^\\top AV_2y_2^1 = U_1^\\top AV_2y_2^1 = U_1^\\top AV_1\\Gamma_y^1y_2^1 = U_1^\\top AV_1y_1^1.$$ \r\n\t\tConsequently, \r\n\t\t\\begin{eqnarray*}\r\n\t\t\tu_2^2  = \\cA_u^2(\\theta; x_2^0, U_2^\\top AV_2 y_2^0, x_2^1, U_2^\\top AV_2 y_2^1)\r\n\t\t\t=\\cA_u^2(\\theta; x_1^0, U_1^\\top AV_1 y_1^0, x_1^1, U_1^\\top AV_1 y_1^1)\r\n\t\t\t= u_2^1\r\n\t\t\\end{eqnarray*}\r\n\t\tand \r\n\t\t\\begin{eqnarray*}\r\n\t\t\t(x_2^2,\\tilde x_2^2)  & = & \\cA_x^2(\\theta; x_2^0, U_2^\\top AV_2 y_2^0, x_2^1, U_2^\\top AV_2 y_2^1,\\mathbf{prox}_{\\gamma_2f}(u^2_2))\\\\\r\n\t\t\t& = &\\cA_x^2(\\theta; x_1^0, U_1^\\top AV_1 y_1^0, x_1^1, U_1^\\top AV_1 y_1^1,\\mathbf{prox}_{\\gamma_2f}(u^2_1))\\\\\r\n\t\t\t& = & (x_1^2,\\tilde x_1^2).\r\n\t\t\\end{eqnarray*}\r\n\t\tThrough a similar argument, we have $(y_2^2,\\tilde y_2^2) = (y_1^2,\\tilde y_1^2)$. By \\eqref{thm:gen-prox-2} and \\eqref{thm:gen-prox-3}, we have \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-4}\r\n\t\t\\{x_2^0,x_2^1,x_2^2\\}\\in U_2^\\top\\cH_x^7\\qquad\\mbox{and}\\qquad\\{b,y_2^0,y_2^1,y_2^2\\}\\in V_2^\\top\\cH_y^7.\r\n\t\t\\end{equation}\r\n\t\t\\textbf{Case $k$.} Suppose we already have orthogonal matrices $U_{k-1}, V_{k-1}$, such that when $\\cA$ is applied to $F_{U_{k-1},V_{k-1}}$, we have\r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-5}\r\n\t\t\\{x_{k-1}^0,x_{k-1}^1,\\cdots, x_{k-1}^{k-1}\\}\\in U_{k-1}^\\top\\cH_x^{4k-5}\\qquad\\mbox{and}\\qquad\\{b,y_{k-1}^0,y_{k-1}^1,\\cdots,y_{k-1}^{k-1}\\}\\in V_{k-1}^\\top\\cH_y^{4k-5}.\r\n\t\t\\end{equation}\r\n\t\tAgain, by Lemma \\ref{lemma:yyx}, there exist orthogonal matrices $\\Gamma_x^{k-1}$ and $\\Gamma_y^{k-1}$, such that \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-6}\r\n\t\t\\begin{cases}\r\n\t\t\\Gamma_x^{k-1}x = (\\Gamma_x^{k-1})^\\top x = x, \\,\\,\\forall x\\in U_{k-1}^\\top\\cH_x^{4k-3},\\,\\,\\,\\, \\Gamma_x^{k-1} x_{k-1}^{k}\\in U_{k-1}^\\top\\cH_x^{4k-1},\\\\\r\n\t\t\\Gamma_y^{k-1}y \\,= (\\Gamma_y^{k-1})^\\top y = y, \\,\\,\\,\\forall y\\in V_{k-1}^\\top\\cH_y^{4k-3},\\,\\,\\,\\, \\Gamma_y^{k-1} y_{k-1}^k\\in V_{k-1}^\\top\\cH_y^{4k-1}.\r\n\t\t\\end{cases}\r\n\t\t\\end{equation}\r\n\t\tNow we define that \r\n\t\t$$U_k = U_{k-1}\\Gamma_x^{k-1} \\qquad\\mbox{and}\\qquad V_k = V_{k-1}\\Gamma_y^{k-1}.$$\r\n\t\tTherefore, similar to our previous discussion, we only need to argue that when $\\cA$ is applied to $F_{U_k,V_k}$, the generated iterates $\\{(x_k^0,y_k^0), (x_k^1,y_k^1),\\cdots,(x_k^k,y_k^k)\\}$ satisfy $(x_k^i,y_k^i) = (x_{k-1}^i,y_{k-1}^i)$ for $i = 0,1,...,k$. We prove this argument by induction. First, it is straightforward that $(x_k^0,y_k^0) = (0,0) = (x_{k-1}^0,y_{k-1}^0)$. Suppose $(x_k^i,y_k^i) = (x_{k-1}^i,y_{k-1}^i)$ holds for $i = 0,1,...,j-1\\leq k-1$, now we prove $(x_k^j,y_k^j) = (x_{k-1}^j,y_{k-1}^j)$, which is almost identical to the case $k=2$.\r\n\t\r\n\tFor any $i\\in\\{0,1,...,j-1\\}$, let us show $U_{k-1}^\\top AV_{k-1} y_{k-1}^i = U_k^\\top AV_k y_k^i$. Because $y_k^i = y_{k-1}^i\\in V_{k-1}^\\top\\cH_y^{4k-5}\\subsetneqq V_{k-1}^\\top\\cH_y^{4k-3}$, we have $\\Gamma_y^{k-1} y_k^i = y_k^i = y_{k-1}^i\\in V_{k-1}^\\top\\cH_y^{4k-5}.$ Then $V_ky_k^i = V_{k-1}\\Gamma_y^{k-1}y_k^i\\in V_{k-1}V_{k-1}^\\top\\cH_y^{4k-5} = \\cH_y^{4k-5}.$ Therefore $U_{k-1}^\\top AV_{k}y_k^i\\in U_{k-1}^\\top A\\cH_y^{4k-5} = U_{k-1}^\\top\\cH_x^{4k-3}$ and \r\n\t\t$$U_k^\\top AV_ky_k^i = (\\Gamma_x^{k-1})^\\top U_{k-1}^\\top AV_ky_k^i = U_{k-1}^\\top AV_{k}y_k^i = U_{k-1}^\\top AV_{k-1}\\Gamma_y^{k-1}y_k^i = U_{k-1}^\\top AV_{k-1}y_{k-1}^i,$$ \r\n\t\tfor $0\\leq i\\leq j-1$. Consequently, \r\n\t\t\\begin{eqnarray*}\r\n\t\t\tu_k^i  & = & \\cA_u^i(\\theta; x_k^0, U_k^\\top AV_k y_k^0,..., x_k^{i-1}, U_k^\\top AV_k y_k^{i-1})\\\\\r\n\t\t\t& = & \\cA_u^i(\\theta; x_{k-1}^0, U_{k-1}^\\top AV_{k-1} y_{k-1}^0,..., x_{k-1}^{i-1}, U_{k-1}^\\top AV_{k-1} y_{k-1}^{i-1})\\\\\r\n\t\t\t& = & u_{k-1}^i\r\n\t\t\\end{eqnarray*}\r\n\t\tand \r\n\t\t\\begin{eqnarray*}\r\n\t\t\t(x_k^i,\\tilde x_k^i)  & = & \\cA_x^i(\\theta; x_k^0, U_k^\\top AV_k y_k^0,..., x_k^{i-1}, U_k^\\top AV_k y_k^{i-1},\\mathbf{prox}_{\\gamma_if}(u^i_k))\\\\\r\n\t\t\t& = &\\cA_x^2(\\theta; x_{k-1}^0, U_{k-1}^\\top AV_{k-1} y_{k-1}^0,..., x_{k-1}^{i-1}, U_{k-1}^\\top AV_{k-1} y_{k-1}^{i-1},\\mathbf{prox}_{\\gamma_if}(u^i_{k-1}))\\\\\r\n\t\t\t& = & (x_{k-1}^i,\\tilde x_{k-1}^i).\r\n\t\t\\end{eqnarray*}\r\n\t\tThrough a similar argument, we have $(y_k^i,\\tilde y_k^i) = (y_{k-1}^i,\\tilde y_{k-1}^i)$. By induction, we know $(y_k^i,\\tilde y_k^i) = (y_{k-1}^i,\\tilde y_{k-1}^i)$ for $i = 0,1,...,k$. Consequently, we have \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-7}\r\n\t\t\\{x_{k}^0,x_{k}^1,\\cdots, x_{k}^{k}\\}\\in U_{k}^\\top\\cH_x^{4k-1}\\qquad\\mbox{and}\\qquad\\{b,y_{k}^0,y_{k}^1,\\cdots,y_{k}^{k}\\}\\in V_{k}^\\top\\cH_y^{4k-1}.\r\n\t\t\\end{equation}\r\n\t    By setting $\\hat U = U_k$ and $\\hat V = V_k$, we prove the result for Part I.\\vspace{0.5cm}\\\\       \r\n       \t\\textbf{Part II. There exist orthogonal matrices $U$, $V$ such that when $\\cA$ is applied to the rotated instance $F_{U,V}$, $\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1},$ and $(\\tilde{x}^k,\\tilde{y}^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1}$}.\r\n       \r\n       Given the result of Part I, and let $\\{(x_k^0, y_k^0),...,(x_k^k, y_k^k)\\}$ and $(\\tilde x_k^k, \\tilde y_k^k)$ be generated by $\\cA$ when applied to $F_{\\hat U,\\hat V} = F_{U_k,V_k}$. Therefore, by Lemma \\ref{lemma:yyx}, there exist orthogonal matrices $P,Q$ such that \r\n       \\begin{equation}\r\n       \\label{thm:gen-prox-8}\r\n       \\begin{cases}\r\n       Px = P^\\top x = x, \\,\\,\\forall x\\in U_{k}^\\top\\cH_x^{4k-1},\\,\\,\\,\\, P \\tilde x_{k}^{k}\\in U_{k}^\\top\\cH_x^{4k+1},\\\\\r\n       Qy \\,=Q^\\top y = y, \\,\\,\\forall y\\in V_{k}^\\top\\cH_y^{4k-1},\\,\\,\\,\\, Q\\tilde y_{k}^k\\in V_{k}^\\top\\cH_y^{4k+1}.\r\n       \\end{cases}\r\n       \\end{equation}\r\n       Define $U = U_kP$, and $V = V_kQ$. Let $\\{(x^0,y^0),...,(x^k,y^k)\\}$ and the output $(\\tilde x^k,\\tilde y^k)$ be generated by $\\cA$ when applied to $F_{{U,V}}$. Then following the same line of argument of Case $k$, Part I, we have \r\n       $$(x^i,y^i) = (x^i_k,y^i_k), \\,\\,\\mbox{for}\\,\\, i = 0,1,...,k\\qquad\\mbox{and}\\qquad(\\tilde x^k,\\tilde y^k) = (\\tilde x^k_k,\\tilde y^k_k).$$\r\n       Therefore, combining \\eqref{thm:gen-prox-8}, we complete the proof of Part II.  \r\n\\end{proof}","start":79656,"end":91544}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S065"],"declared_external":false,"source_only":false,"title":"旋回距离并转为复杂度","source_units":[{"raw_tex":"For the proof of this theorem, we only need to construct the orthogonal matrices $U,V$ such that when the algorithm $\\cA$ is applied to $F_{U,V}$, the subspace inclusion argument $\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1}$ and $(\\tilde x^k,\\tilde y^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1}$ holds. As a result, \r\n\\begin{eqnarray*}\r\n\t\\|\\tilde y^k-V^\\top  y^*\\|^2 = \\|V\\tilde y^k- y^*\\|^2\\geq\\min_{y\\in\\cH_y^{4k+1}}\\|y-y^*\\|^2\\geq \\frac{q^{4k+2}}{16}\\|y^*-y^0\\|^2.\r\n\\end{eqnarray*}\r\nWith this argument, the latter results follow directly from the discussion of Theorem \\ref{thm:proximal}. The proof of this theorem is presented in Appendix \\ref{appdx:gen-prox}.\r\n","start":51431,"end":52140}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S066"],"declared_external":false,"source_only":false,"title":"构造目标与两部分安排","source_units":[{"raw_tex":"\\begin{proof}\r\n\t\tTo prove this theorem, we only need to show\r\n\t\t$$\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1}\\qquad\\mbox{and}\\qquad(\\tilde x^k,\\tilde y^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1}.$$\r\n\t\tWe separate the proof into two parts.\r\n\t\t\r\n\t\t\\textbf{Part I. There exist orthogonal matrices $\\hat U$, $\\hat V$ s.t. when $\\cA$ is applied to the rotated instance $F_{\\hat U,\\hat V}$, $\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq \\hat U^\\top\\cH_x^{4k-1}\\times \\hat V^\\top\\cH_y^{4k-1}.$}\r\n\t\r\n\t\tLet $\\theta = (L_{xy},\\mu_x,\\mu_y)$ be the set of algorithmic parameters. To prove the result, let us construct the worst-case function $F_{U,V}$ in a recursive way. \t \\\\\r\n","start":79656,"end":80358}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S067"],"declared_external":false,"source_only":false,"title":"第一轮的原始运行记录","source_units":[{"raw_tex":"\t\t\\textbf{Case $k = 1$:} Let us define $U_0 = V_0 = I$. When $\\cA$ is applied to the function $F_{U_0,V_0}\\in\\cB(L_{xy}, \\mu_x, \\mu_y)$, the iterate sequence is \r\n\t\t$(x_{0}^0, y_{0}^0) = (0, 0)$ and \r\n\t\t$$\\begin{cases}\r\n\t\tu^1_0 = \\cA_u^1(\\theta; x_0^0, U_0^\\top A V_0 y_0^0), \\qquad (x^1_0, \\tilde x^1_0) = \\cA_x^1(\\theta; x^0_0,  U_0^\\top A V_0 y_0^0, \\mathbf{prox}_{\\gamma_1f}(u^1_0)),\\\\\r\n\t\tv^1_0 = \\cA_v^1(\\theta; y_0^0, V^\\top_0 A U_0x_0^0),\\qquad\r\n\t\t(y^1_0, \\tilde y^1_0) = \\cA_y^1(\\theta; y^0_0, V^\\top_0 A U_0x_0^0, \\mathbf{prox}_{\\sigma_1g}(v^1_0)).\r\n\t\t\\end{cases}$$\r\n","start":80358,"end":80934}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":["thm:gen-prox-1"],"member_ids":["S068"],"declared_external":false,"source_only":false,"title":"第一轮的对抗旋转","source_units":[{"raw_tex":"\t\tBy Lemma \\ref{lemma:yyx}, there exists orthogonal matrices $\\Gamma_x^0$ and $\\Gamma_y^0$ such that \r\n\t\t$\\Gamma_x^0x^1_0\\in\\cH_x^3 = \\mathrm{Span}\\{Ab\\}$, $\\Gamma_y^0y^1_0\\in\\cH_y^3=\\mathrm{Span}\\{b, A^2b\\}$, and $\\Gamma_y^0b = (\\Gamma_y^0)^\\top b = b.$ That is \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-1}\r\n\t\tx_0^1\\in U_1^\\top\\cH_x^3,\\qquad\\mbox{and}\\qquad y_0^1\\in V_1^\\top\\cH_y^3, \\quad V_1 b = V_1^\\top b = b,\r\n\t\t\\end{equation}\t\r\n\t\twhere $U_1 = U_0\\Gamma_x^0$ and $V_1 = V_0\\Gamma_y^0$. \r\n","start":80934,"end":81430}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":["thm:gen-prox-2"],"member_ids":["S069"],"declared_external":false,"source_only":false,"title":"第一轮运行保持不变","source_units":[{"raw_tex":"\t\r\n\t\tNow we prove that when we apply the algorithm $\\cA$ to $F_{U_1,V_1}$, the generated iterates $\\{(x^0_1, y^0_1), (x^1_1, y^1_1)\\}$ satisfy that\r\n\t\t$(x^0_1, y^0_1) = (0, 0)$ and  $(x^1_1, y^1_1) = (x^1_0, y^1_0)$. That is, the first two iterates generated by $\\cA$ is completely the same for $F_{U_0,V_0}$ and $F_{U_1,V_1}$. The reason is because \r\n\t\t$u^1_1 = \\cA_u^1(\\theta; x^0_1, U_1^\\top AV_1y^0_1) = \\cA_u^1(\\theta; 0, 0) = \\cA_u^1(\\theta; x^0_0, U_0^\\top AV_0y^0_0)  = u^1_0$, \r\n\t\ttherefore \r\n\t\t\\begin{eqnarray*}\r\n\t\t\t(x^1_1, \\tilde x^1_1) & = & \\cA_x^1(\\theta; x^0_1, U_1^\\top A V_1 y_1^0, \\mathbf{prox}_{\\gamma_1f}(u^1_1)) \\\\\r\n\t\t\t& = & \\cA_x^1(\\theta; 0, 0, \\mathbf{prox}_{\\gamma_1f}(u^1_1)) \\\\\r\n\t\t\t& = & \\cA_x^1(\\theta; x^0_0, U_0^\\top A V_0 y_0^0, \\mathbf{prox}_{\\gamma_1f}(u^1_0)) \\\\\r\n\t\t\t& = & (x^1_0, \\tilde x^1_0).\r\n\t\t\\end{eqnarray*}\r\n\t\tThrough similar argument, we know $(y^1_1,\\tilde y^1_1) = (y^1_0,\\tilde y^1_0)$. Therefore, \\eqref{thm:gen-prox-1} indicates that \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-2}\r\n\t\tx_1^1\\in U_1^\\top\\cH_x^3,\\qquad\\mbox{and}\\qquad y_1^1\\in V_1^\\top\\cH_y^3, \\quad V_1 b = V_1^\\top b = b\\in V_1^\\top\\cH_y^3.\r\n\t\t\\end{equation}\r\n","start":81430,"end":82604}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S070"],"declared_external":false,"source_only":false,"title":"第二轮的运行记录","source_units":[{"raw_tex":"\t\t\\textbf{Case $k=2$.} For the ease of the readers to follow, we perform one extra step of discussion for $k=2$, before presenting the construction on general $k$. \r\n\t    \r\n\t\tFor the problem instance $F_{U_1,V_1}$, the iterates generated by $\\cA$ are $(x_{1}^0, y_{1}^0) = (0, 0)$ and \r\n\t\t$$\\begin{cases}\r\n\t\tu^1_1 = \\cA_u^1(\\theta; x_1^0, U_1^\\top A V_1 y_1^0), \\qquad (x^1_1, \\tilde x^1_1) = \\cA_x^1(\\theta; x^0_1,  U_1^\\top A V_1 y_1^0, \\mathbf{prox}_{\\gamma_1f}(u^1_1)),\\\\\r\n\t\tv^1_1 = \\cA_v^1(\\theta; y_1^0, V^\\top_1 A U_1x_1^0),\\qquad\r\n\t\t(y^1_1, \\tilde y^1_1) = \\cA_y^1(\\theta; y^0_1, V^\\top_1 A U_1x_1^0, \\mathbf{prox}_{\\sigma_1g}(v^1_1)).\\\\\r\n\t\t\\end{cases}$$\r\n\t\t$$\\begin{cases}\r\n\t\tu^2_1 = \\cA_u^2(\\theta; x_1^0, U_1^\\top A V_1 y_1^0, x_1^1, U_1^\\top A V_1 y_1^1), \\quad (x^2_1, \\tilde x^2_1) = \\cA_x^2(\\theta; x^0_1,  U_1^\\top A V_1 y_1^0, x_1^1, U_1^\\top A V_1 y_1^1, \\mathbf{prox}_{\\gamma_2f}(u^2_1)),\\\\\r\n\t\tv^2_1 = \\cA_v^2(\\theta; y_1^0, V^\\top_1 A U_1x_1^0, y_1^1, V^\\top_1 A U_1x_1^1),\\quad\r\n\t\t(y^2_1, \\tilde y^2_1) = \\cA_y^2(\\theta; y^0_1, V^\\top_1 A U_1x_1^0, y_1^1, V^\\top_1 A U_1x_1^1, \\mathbf{prox}_{\\sigma_2g}(v^2_1)).\\\\\r\n\t\t\\end{cases}$$\r\n","start":82604,"end":83757}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":["thm:gen-prox-3"],"member_ids":["S071"],"declared_external":false,"source_only":false,"title":"固定五阶空间旋入七阶","source_units":[{"raw_tex":"\t\tNote that  $x_1^1 \\in U_1^\\top\\cH_x^3 \\subsetneqq U_1^\\top\\cH_x^5\\subsetneqq U_1^\\top\\cH_x^7$ and $\\{y_1^1,b\\}\\subsetneqq V_1^\\top\\cH_y^3\\subsetneqq V_1^\\top\\cH_y^5\\subsetneqq V_1^\\top\\cH_y^7$. Therefore, there exist orthogonal matrices $\\Gamma_x^1$ and $\\Gamma_y^1$ such that \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-3}\r\n\t\t\\begin{cases}\r\n\t\t\\Gamma_x^1x = (\\Gamma_x^1)^\\top x = x, \\,\\,\\forall x\\in U_1^\\top\\cH_x^5,\\,\\,\\,\\, \\Gamma_x^1 x_1^2\\in U_1^\\top\\cH_x^7,\\\\\r\n\t\t\\Gamma_y^1y \\,= (\\Gamma_y^1)^\\top y = y, \\,\\,\\,\\forall y\\in V_1^\\top\\cH_y^5,\\,\\,\\,\\, \\Gamma_y^1 y_1^2\\in V_1^\\top\\cH_y^7.\r\n\t\t\\end{cases}\r\n\t\t\\end{equation}\r\n\t\tNow, let us define \r\n\t\t$$U_2 = U_1\\Gamma_x^1\\qquad\\mbox{and}\\qquad V_2 = V_1\\Gamma_y^1.$$\r\n","start":83757,"end":84475}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S072"],"declared_external":false,"source_only":false,"title":"第二轮历史信息保持","source_units":[{"raw_tex":"\t\tNow we prove that if $\\cA$ is applied to $F_{U_2,V_2}$, the generated iterates $\\{(x_{2}^0, y_{2}^0), (x_{2}^1, y_{2}^1), (x_{2}^2, y_{2}^2)\\}$ satisfy $(x_{2}^0, y_{2}^0) = (0, 0)$, $(x_{2}^1, y_{2}^1) = (x_{1}^1, y_{1}^1)$, and  $(x_{2}^2, y_{2}^2) = (x_{1}^2, y_{1}^2)$. The argument for $(x_{2}^1, y_{2}^1) = (x_{1}^1, y_{1}^1)$ is almost the same as  that of the case $k=1$. We only provide the proof for $(x_{2}^2, y_{2}^2) = (x_{1}^2, y_{1}^2)$.\r\n\t\r\n\tNext, we need to show $u_2^2 = u_1^2$, which can be proved by arguing that all the inputs to $\\cA_u^2$ are the same for both $u_2^2$ and $u_1^2$. First, it is straightforward that $x_1^0 = 0 = x^0_2, U_1^\\top AV_1 y_1^0 = 0 = U_2^\\top AV_2 y_2^0$. By previous argument $x_2^1 = x_1^1$.  Finally, consider the last input $U_2^\\top AV_2 y_2^1$, because $y_2^1 = y_1^1\\in V_1^\\top\\cH_y^3\\subsetneqq V_1^\\top\\cH_y^5$, we have $\\Gamma_y^1 y_2^1 = y_2^1 = y_1^1\\in V_1^\\top\\cH_y^3.$ Then $V_2y_2^1 = V_1\\Gamma_y^1y_2^1\\in V_1V_1^\\top\\cH_y^3 = \\cH_y^3.$ Therefore $U_1^\\top AV_2y_2^1\\in U_1^\\top A\\cH_y^3 = U_1^\\top\\cH_x^5$ and \r\n\t\t$$U_2^\\top AV_2y_2^1 = \\Gamma_x^1U_1^\\top AV_2y_2^1 = U_1^\\top AV_2y_2^1 = U_1^\\top AV_1\\Gamma_y^1y_2^1 = U_1^\\top AV_1y_1^1.$$ \r\n\t\tConsequently, \r\n\t\t\\begin{eqnarray*}\r\n\t\t\tu_2^2  = \\cA_u^2(\\theta; x_2^0, U_2^\\top AV_2 y_2^0, x_2^1, U_2^\\top AV_2 y_2^1)\r\n\t\t\t=\\cA_u^2(\\theta; x_1^0, U_1^\\top AV_1 y_1^0, x_1^1, U_1^\\top AV_1 y_1^1)\r\n\t\t\t= u_2^1\r\n\t\t\\end{eqnarray*}\r\n\t\tand \r\n\t\t\\begin{eqnarray*}\r\n\t\t\t(x_2^2,\\tilde x_2^2)  & = & \\cA_x^2(\\theta; x_2^0, U_2^\\top AV_2 y_2^0, x_2^1, U_2^\\top AV_2 y_2^1,\\mathbf{prox}_{\\gamma_2f}(u^2_2))\\\\\r\n\t\t\t& = &\\cA_x^2(\\theta; x_1^0, U_1^\\top AV_1 y_1^0, x_1^1, U_1^\\top AV_1 y_1^1,\\mathbf{prox}_{\\gamma_2f}(u^2_1))\\\\\r\n\t\t\t& = & (x_1^2,\\tilde x_1^2).\r\n\t\t\\end{eqnarray*}\r\n","start":84475,"end":86257}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":["thm:gen-prox-4"],"member_ids":["S073"],"declared_external":false,"source_only":false,"title":"第二轮子空间结论","source_units":[{"raw_tex":"\t\tThrough a similar argument, we have $(y_2^2,\\tilde y_2^2) = (y_1^2,\\tilde y_1^2)$. By \\eqref{thm:gen-prox-2} and \\eqref{thm:gen-prox-3}, we have \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-4}\r\n\t\t\\{x_2^0,x_2^1,x_2^2\\}\\in U_2^\\top\\cH_x^7\\qquad\\mbox{and}\\qquad\\{b,y_2^0,y_2^1,y_2^2\\}\\in V_2^\\top\\cH_y^7.\r\n\t\t\\end{equation}\r\n","start":86257,"end":86579}]},{"id":"S074","semantic_type":"assumption","kind":"assumption","labels":["thm:gen-prox-5"],"member_ids":["S074"],"declared_external":false,"source_only":false,"title":"外层归纳假设","source_units":[{"raw_tex":"\t\t\\textbf{Case $k$.} Suppose we already have orthogonal matrices $U_{k-1}, V_{k-1}$, such that when $\\cA$ is applied to $F_{U_{k-1},V_{k-1}}$, we have\r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-5}\r\n\t\t\\{x_{k-1}^0,x_{k-1}^1,\\cdots, x_{k-1}^{k-1}\\}\\in U_{k-1}^\\top\\cH_x^{4k-5}\\qquad\\mbox{and}\\qquad\\{b,y_{k-1}^0,y_{k-1}^1,\\cdots,y_{k-1}^{k-1}\\}\\in V_{k-1}^\\top\\cH_y^{4k-5}.\r\n\t\t\\end{equation}\r\n","start":86579,"end":86969}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":["thm:gen-prox-6"],"member_ids":["S075"],"declared_external":false,"source_only":false,"title":"一般轮次的旋转扩张","source_units":[{"raw_tex":"\t\tAgain, by Lemma \\ref{lemma:yyx}, there exist orthogonal matrices $\\Gamma_x^{k-1}$ and $\\Gamma_y^{k-1}$, such that \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-6}\r\n\t\t\\begin{cases}\r\n\t\t\\Gamma_x^{k-1}x = (\\Gamma_x^{k-1})^\\top x = x, \\,\\,\\forall x\\in U_{k-1}^\\top\\cH_x^{4k-3},\\,\\,\\,\\, \\Gamma_x^{k-1} x_{k-1}^{k}\\in U_{k-1}^\\top\\cH_x^{4k-1},\\\\\r\n\t\t\\Gamma_y^{k-1}y \\,= (\\Gamma_y^{k-1})^\\top y = y, \\,\\,\\,\\forall y\\in V_{k-1}^\\top\\cH_y^{4k-3},\\,\\,\\,\\, \\Gamma_y^{k-1} y_{k-1}^k\\in V_{k-1}^\\top\\cH_y^{4k-1}.\r\n\t\t\\end{cases}\r\n\t\t\\end{equation}\r\n\t\tNow we define that \r\n\t\t$$U_k = U_{k-1}\\Gamma_x^{k-1} \\qquad\\mbox{and}\\qquad V_k = V_{k-1}\\Gamma_y^{k-1}.$$\r\n","start":86969,"end":87611}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S076"],"declared_external":false,"source_only":false,"title":"逐轮归纳运行不变","source_units":[{"raw_tex":"\t\tTherefore, similar to our previous discussion, we only need to argue that when $\\cA$ is applied to $F_{U_k,V_k}$, the generated iterates $\\{(x_k^0,y_k^0), (x_k^1,y_k^1),\\cdots,(x_k^k,y_k^k)\\}$ satisfy $(x_k^i,y_k^i) = (x_{k-1}^i,y_{k-1}^i)$ for $i = 0,1,...,k$. We prove this argument by induction. First, it is straightforward that $(x_k^0,y_k^0) = (0,0) = (x_{k-1}^0,y_{k-1}^0)$. Suppose $(x_k^i,y_k^i) = (x_{k-1}^i,y_{k-1}^i)$ holds for $i = 0,1,...,j-1\\leq k-1$, now we prove $(x_k^j,y_k^j) = (x_{k-1}^j,y_{k-1}^j)$, which is almost identical to the case $k=2$.\r\n\t\r\n\tFor any $i\\in\\{0,1,...,j-1\\}$, let us show $U_{k-1}^\\top AV_{k-1} y_{k-1}^i = U_k^\\top AV_k y_k^i$. Because $y_k^i = y_{k-1}^i\\in V_{k-1}^\\top\\cH_y^{4k-5}\\subsetneqq V_{k-1}^\\top\\cH_y^{4k-3}$, we have $\\Gamma_y^{k-1} y_k^i = y_k^i = y_{k-1}^i\\in V_{k-1}^\\top\\cH_y^{4k-5}.$ Then $V_ky_k^i = V_{k-1}\\Gamma_y^{k-1}y_k^i\\in V_{k-1}V_{k-1}^\\top\\cH_y^{4k-5} = \\cH_y^{4k-5}.$ Therefore $U_{k-1}^\\top AV_{k}y_k^i\\in U_{k-1}^\\top A\\cH_y^{4k-5} = U_{k-1}^\\top\\cH_x^{4k-3}$ and \r\n\t\t$$U_k^\\top AV_ky_k^i = (\\Gamma_x^{k-1})^\\top U_{k-1}^\\top AV_ky_k^i = U_{k-1}^\\top AV_{k}y_k^i = U_{k-1}^\\top AV_{k-1}\\Gamma_y^{k-1}y_k^i = U_{k-1}^\\top AV_{k-1}y_{k-1}^i,$$ \r\n\t\tfor $0\\leq i\\leq j-1$. Consequently, \r\n\t\t\\begin{eqnarray*}\r\n\t\t\tu_k^i  & = & \\cA_u^i(\\theta; x_k^0, U_k^\\top AV_k y_k^0,..., x_k^{i-1}, U_k^\\top AV_k y_k^{i-1})\\\\\r\n\t\t\t& = & \\cA_u^i(\\theta; x_{k-1}^0, U_{k-1}^\\top AV_{k-1} y_{k-1}^0,..., x_{k-1}^{i-1}, U_{k-1}^\\top AV_{k-1} y_{k-1}^{i-1})\\\\\r\n\t\t\t& = & u_{k-1}^i\r\n\t\t\\end{eqnarray*}\r\n\t\tand \r\n\t\t\\begin{eqnarray*}\r\n\t\t\t(x_k^i,\\tilde x_k^i)  & = & \\cA_x^i(\\theta; x_k^0, U_k^\\top AV_k y_k^0,..., x_k^{i-1}, U_k^\\top AV_k y_k^{i-1},\\mathbf{prox}_{\\gamma_if}(u^i_k))\\\\\r\n\t\t\t& = &\\cA_x^2(\\theta; x_{k-1}^0, U_{k-1}^\\top AV_{k-1} y_{k-1}^0,..., x_{k-1}^{i-1}, U_{k-1}^\\top AV_{k-1} y_{k-1}^{i-1},\\mathbf{prox}_{\\gamma_if}(u^i_{k-1}))\\\\\r\n\t\t\t& = & (x_{k-1}^i,\\tilde x_{k-1}^i).\r\n\t\t\\end{eqnarray*}\r\n","start":87611,"end":89566}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","labels":["thm:gen-prox-7"],"member_ids":["S077"],"declared_external":false,"source_only":false,"title":"完成迭代序列的旋转构造","source_units":[{"raw_tex":"\t\tThrough a similar argument, we have $(y_k^i,\\tilde y_k^i) = (y_{k-1}^i,\\tilde y_{k-1}^i)$. By induction, we know $(y_k^i,\\tilde y_k^i) = (y_{k-1}^i,\\tilde y_{k-1}^i)$ for $i = 0,1,...,k$. Consequently, we have \r\n\t\t\\begin{equation}\r\n\t\t\\label{thm:gen-prox-7}\r\n\t\t\\{x_{k}^0,x_{k}^1,\\cdots, x_{k}^{k}\\}\\in U_{k}^\\top\\cH_x^{4k-1}\\qquad\\mbox{and}\\qquad\\{b,y_{k}^0,y_{k}^1,\\cdots,y_{k}^{k}\\}\\in V_{k}^\\top\\cH_y^{4k-1}.\r\n\t\t\\end{equation}\r\n\t    By setting $\\hat U = U_k$ and $\\hat V = V_k$, we prove the result for Part I.\\vspace{0.5cm}\\\\       \r\n","start":89566,"end":90105}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","labels":["thm:gen-prox-8"],"member_ids":["S078"],"declared_external":false,"source_only":false,"title":"额外旋转控制输出","source_units":[{"raw_tex":"       \t\\textbf{Part II. There exist orthogonal matrices $U$, $V$ such that when $\\cA$ is applied to the rotated instance $F_{U,V}$, $\\{(x^0,y^0),...,(x^k,y^k)\\}\\subseteq U^\\top\\cH_x^{4k-1}\\times V^\\top\\cH_y^{4k-1},$ and $(\\tilde{x}^k,\\tilde{y}^k)\\in U^\\top\\cH_x^{4k+1}\\times V^\\top\\cH_y^{4k+1}$}.\r\n       \r\n       Given the result of Part I, and let $\\{(x_k^0, y_k^0),...,(x_k^k, y_k^k)\\}$ and $(\\tilde x_k^k, \\tilde y_k^k)$ be generated by $\\cA$ when applied to $F_{\\hat U,\\hat V} = F_{U_k,V_k}$. Therefore, by Lemma \\ref{lemma:yyx}, there exist orthogonal matrices $P,Q$ such that \r\n       \\begin{equation}\r\n       \\label{thm:gen-prox-8}\r\n       \\begin{cases}\r\n       Px = P^\\top x = x, \\,\\,\\forall x\\in U_{k}^\\top\\cH_x^{4k-1},\\,\\,\\,\\, P \\tilde x_{k}^{k}\\in U_{k}^\\top\\cH_x^{4k+1},\\\\\r\n       Qy \\,=Q^\\top y = y, \\,\\,\\forall y\\in V_{k}^\\top\\cH_y^{4k-1},\\,\\,\\,\\, Q\\tilde y_{k}^k\\in V_{k}^\\top\\cH_y^{4k+1}.\r\n       \\end{cases}\r\n       \\end{equation}\r\n","start":90105,"end":91056}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S079"],"declared_external":false,"source_only":false,"title":"最终实例保持运行与输出","source_units":[{"raw_tex":"       Define $U = U_kP$, and $V = V_kQ$. Let $\\{(x^0,y^0),...,(x^k,y^k)\\}$ and the output $(\\tilde x^k,\\tilde y^k)$ be generated by $\\cA$ when applied to $F_{{U,V}}$. Then following the same line of argument of Case $k$, Part I, we have \r\n       $$(x^i,y^i) = (x^i_k,y^i_k), \\,\\,\\mbox{for}\\,\\, i = 0,1,...,k\\qquad\\mbox{and}\\qquad(\\tilde x^k,\\tilde y^k) = (\\tilde x^k_k,\\tilde y^k_k).$$\r\n       Therefore, combining \\eqref{thm:gen-prox-8}, we complete the proof of Part II.  \r\n\\end{proof}","start":91056,"end":91544}]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S080"],"declared_external":true,"declaration_text":["原始先决条件列表所列：第301–308行使用的标准强凸性和鞍点对偶性事实；未提供外部证明。"],"source_only":false,"title":"强凸凹与鞍点标准事实","source_units":[]},{"id":"S081","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S081"],"declared_external":true,"declaration_text":["原始先决条件列表所列：正定矩阵的初等谱界；对应第577行的使用。"],"source_only":false,"title":"正定逆矩阵的谱界","source_units":[]},{"id":"S082","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S082"],"declared_external":true,"declaration_text":["原始先决条件列表所列：初等几何级数；对应第599、924行的使用。"],"source_only":false,"title":"几何级数尾部比较","source_units":[]},{"id":"S083","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S083"],"declared_external":true,"declaration_text":["原始先决条件列表所列：初等对数不等式；原文第943行明确陈述所用形式。"],"source_only":false,"title":"初等对数不等式","source_units":[]},{"id":"S084","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S084"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；原文第637、650、960行直接使用的正交矩阵基本性质，未给出独立外部来源。"],"source_only":false,"title":"正交变换的基本不变性","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S009":"S009","S011":"S011","S012":"S012","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S026","S025":"S026","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S029","S034":"S029","S035":"S029","S036":"S029","S037":"S029","S038":"S052","S039":"S039","S040":"S052","S041":"S041","S042":"S041","S043":"S041","S044":"S041","S045":"S041","S046":"S041","S047":"S041","S048":"S041","S049":"S041","S050":"S052","S051":"S052","S052":"S052","S053":"S053","S054":"S053","S055":"S053","S056":"S053","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076","S077":"S077","S078":"S078","S079":"S079","S080":"S080","S081":"S081","S082":"S082","S083":"S083","S084":"S084"}}}}
{"record_id":"2309.16530v1","paper":{"title":"Acceleration by Stepsize Hedging II: Silver Stepsize Schedule for Smooth Convex Optimization","authors":["Altschuler, Jason M.","Parrilo, Pablo A."],"arxiv_id":"2309.16530","version":"v1","journal":"Mathematical Programming","abstract_url":"https://arxiv.org/abs/2309.16530v1","pdf_url":"https://arxiv.org/pdf/2309.16530v1","doi":"10.1007/s10107-024-02164-2"},"context":{"raw_tex":"We revisit the classical problem of smooth convex optimization: solve $\\min_{x \\in \\R^d} f(x)$ where $f$ is convex and $M$-smooth (i.e., its gradient is $M$-Lipschitz). A celebrated result is that with a prudent choice of stepsizes $\\{\\alpha_t\\}$, the gradient descent algorithm (GD) \n\\begin{align}\n\tx_{t+1} = x_t - \\frac{\\alpha_t}{M} \\nabla f(x_t) \n\\end{align}\nsolves such a convex optimization problem to arbitrary accuracy from any initialization $x_0$. How quickly does GD converge? The mainstream approach (see e.g., the textbooks \\citep{BertsekasNonlinear,Nocedal,nesterov-survey,vishnoi2021algorithms,bubeck-book,BNO,lan2020first,bertsekas2015convex} among many others) is to use a constant stepsize schedule $\\alpha_t \\equiv \\bar{\\alpha} \\in (0,2)$ since this ensures\n\\begin{align}\n\tf(x_n) - f^* \\leq \\frac{cM\\|x_0 - x^*\\|^2}{n} \n\t\\label{eq:rate-std}\n\\end{align}\nwhere $x^*$ denotes any minimizer of $f$, $f^* := f(x^*)$ denotes the corresponding minimal value, and $c$ is a small constant, e.g., $c = \\tfrac{1}{4}$ for $\\bar{\\alpha} = 1$~\\citep{DT14}.\n\n\n\\par In this setting, the Silver Stepsize Schedule is particularly simple. For any integer $n = 2^{k}-1$, we recursively construct the schedule $h_{2n+1}$ of length $2n+1$ from the schedule $h_n$ of length $n$ via\n\\begin{align}\n\th_{2n+1} := [h_n, \\; 1 + \\rho^{k-1}, \\; h_n]\n\t\\label{eq:steps-recursive}\n\\end{align}\nwhere $\\rho := 1 + \\sqrt{2}$ denotes the silver ratio, and $h_1 := [\\sqrt{2}]$. This results in the simple pattern $[\\sqrt{2}, \\; 2, \\; \\sqrt{2}, \\; 1+\\sqrt{2}, \\dots]$ as depicted in Figure~\\ref{fig:silverstepsize}. This schedule is exactly the Silver Stepsize Schedule from~\\citep{alt23hedging1} in the limit that the strong convexity parameter vanishes (see Remark~\\ref{rem:steps-limit} for details), and bears similarities to those in~\\citep{gupta22,grimmer23,GrimmerShuWang}; see \\S\\ref{ssec:rel}.\n\n\nWe show that these stepsizes yield an improved convergence rate~\\eqref{eq:rate-std} where $\\tfrac{c}{n}$ is replaced by\n\\begin{align}\n\tr_k := \\frac{1}{1 + \\sqrt{4\\rho^{2k} - 3}} \\leq\n\t\\frac{1}{2 \\rho^{\\log_2 n}}\n\t= \\frac{1}{2n^{\\log_2 \\rho}}\n\t\\approx \\frac{1}{2 n^{1.2716}}\n\t\\,.\n\t\\label{eq:r-asymptotics}\n\\end{align}\nThis bound gives the correct asymptotic scaling of $r_k$ since the inequality is asymptotically tight.\n\n\n\\paragraph*{Indexing.} Throughout, the horizon is $n = 2^k-1$. This correspondence between $n \\in \\{1,3,7,15,\\dots\\}$ and $k \\in \\{1,2,3,4,\\dots\\}$ allows re-indexing in a way that simplifies notation for the recursion.\n\n\n\\paragraph*{Rescaling.} For simplicity, we normalize $M = 1$, i.e., $\\| \\nabla f(x) - \\nabla f(y) \\| \\leq \\|x-y\\|$ for all $x,y$. This is without loss of generality since if $h$ is $M$-smooth and convex,  then $f := h/M$ is $1$-smooth and convex, thus the result we establish $f_n - f^* \\leq r_k\\|x_0 - x^*\\|^2$ implies $h_n - h^* \\leq r_k M \\|x_0 - x^*\\|^2$. Note that running GD on $f$ simply amounts to rescaling the Silver Stepsize Schedules for $h$ by $1/M$.\n\n\n The Silver Stepsize Schedule is defined recursively in~\\eqref{eq:steps-recursive}. Here we mention an equivalent direct expression and make several remarks. Let $\\bin(t)$ denote the $2$-adic valuation of $t$, i.e., the smallest non-negative integer $i$ such that $2^i$ is in the binary expansion of $t$. For example, $\\bin(1) = 0$, $\\bin(2) = 1$, $\\bin(3) = 0$, $\\bin(4) = 2$, etc.\n\n\n\\begin{defin}[Silver Stepsize Schedule for smooth convex optimization]\n\tFor $t \\in \\{0,1, 2,\\dots\\}$, the $t$-th stepsize of the Silver Stepsize Schedule is\n\t\t\\begin{align}\n\t\t\\alpha_t\n\t\t:=\n\t\t1 + \\rho^{\\bin(t+1)-1}\\,.\n\t\t\\label{eq:steps}\n\t\\end{align}\n\\end{defin}\n\n\n\\par Briefly, the idea behind multi-step descent is that it is essential to capture how different iterations affect other iterations' progress. We do this by exploiting long-range consistency conditions between the iterates along GD's trajectory, as encoded by the co-coercivities \n\\begin{align}\n\tQ_{ij} := 2(f_i - f_j) + 2\\langle g_j, x_j - x_i \\rangle - \\|g_j - g_i\\|^2\\,.\n\t\\label{eq:cocoercivity}\n\\end{align}\nThe significance of these co-coercivities is that the constraints $\\{Q_{ij} \\geq 0\\}_{i \\neq j \\in \\{0,1,\\dots,n,*\\}}$ are necessary and sufficient for the existence of a $1$-smooth convex function $f$ satisfying $f_i = f(x_i)$ and $g_i = \\nabla f(x_i)$ for each $i \\in \\{0,1,\\dots,n,*\\}$~\\citep{pesto}. In other words, the co-coercivity conditions $\\{Q_{ij} \\geq 0 \\}_{i \\neq j \\in \\{0,1,\\dots,n,*\\}}$ generate all possible long-range consistency constraints on the objective function $f$. Or, said another way, the co-coercivity conditions generate all possible valid inequalities with which one can prove convergence rates for GD. For a further discussion, see~\\citep[\\S2.2]{alt23hedging1}.\n\n\\par Concretely, to prove Theorem~\\ref{thm:main}, we exhibit explicit non-negative multipliers $\\lam_{ij}$ satisfying\n\\begin{align}\n\t\\sum_{i \\neq j \\in \\{0, \\dots, n,*\\}} \\lam_{ij} Q_{ij}  = \n\t\\|x_0 - x^*\\|^2 - \\|x_n - c_k g_n - x^*\\|^2\n\t+ \\frac{f^* - f_n}{r_k}\\,,\n\t\\label{eq:cert}\n\\end{align}\nwhere we use the shorthand $c_k := \\frac{1}{2r_k}$. Since $\\sum_{ij}\\lam_{ij} Q_{ij} \\geq 0$ for any $1$-smooth convex function, and since $\\|x_n - c_k g_n - x^*\\|^2 \\geq 0$ trivially as it is a square, this immediately implies the desired rate \n\\begin{align}\n\tf_n - f^* \\leq r_k \\|x_0 - x^*\\|^2\\,.\n\t\\label{eq:cert-cor}\n\\end{align}\n\n\n\\begin{example}[$n=1$]\\label{ex:n=1}\n\tFor $n=1$, the identity~\\eqref{eq:cert} is\n\t\\begin{align*}\n\t\t\\sum_{i \\neq j \\in \\{0, 1, *\\}} \\lam_{ij} Q_{ij} \n\t\t=\n\t\t\\|x_0 - x^*\\|^2 - \\|x_1 - c_1 g_1 - x^*\\|^2 + \\frac{f^* - f_1}{r_1}\\,.\n\t\\end{align*}\n\tHere, $x_1 = x_0 - \\sqrt{2} g_0$, $r_1 \n\t= \\frac{1}{1+\\sqrt{4\\rho^2-3}}\n\t\\approx 0.1816$, $c_1 = \\frac{1}{2r_1} \\approx 2.7535$, and \n\t\\begin{align*}\n\t\t\\begin{bmatrix}\n\t\t\t\\lam_{00} & \\lam_{01} & \\lam_{0*} \\\\\n\t\t\t\\lam_{10} & \\lam_{11} & \\lam_{1*} \\\\\n\t\t\t\\lam_{*0} & \\lam_{*1} & \\lam_{**}\n\t\t\\end{bmatrix}\n\t\t=\n\t\t\\begin{bmatrix}\n\t\t\t0\n\t\t\t&\n\t\t\t1+\\sqrt{2}\n\t\t\t&\n\t\t\t0\n\t\t\t\\\\\n\t\t\t1\n\t\t\t&\n\t\t\t0\n\t\t\t&\n\t\t\t\\sqrt{2}\n\t\t\t\\\\\n\t\t\t\\sqrt{2}\n\t\t\t&\n\t\t\t\\frac{1}{2r_1}\n\t\t\t&\n\t\t\t0\n\t\t\\end{bmatrix}\n\t\t\\,.\n\t\\end{align*}\n\\end{example}\n\n\\par For larger horizons $n$, we construct $\\lam_{ij}$ via the recursive gluing technique of~\\citep{alt23hedging1}. See Figure~\\ref{fig:gluing}. Below, we say that the multipliers $\\{\\sig_{ij}\\}_{i,j \\in \\{0,\\dots,n,*\\}}$ satisfy the $*$-sparsity property if $\\sig_{i*} = 0$ for all $i < n$. This is satisfied by construction and simplifies part of the proof (isolated in Lemma~\\ref{lem:opt}). \n\n\nThe rest of the section is dedicated to proving Theorem~\\ref{thm:cert}. We first isolate two helper lemmas in \\S\\ref{ssec:gluing:helper}, and then we combine them to prove the result in \\S\\ref{ssec:gluing:pf}.  For notational simplicity, henceforth we assume $x^* = 0$; this is without loss of generality after translating. \n\n\\subsection{Helper lemmas}\\label{ssec:gluing:helper}\n\nHere we provide three helper lemmas for the proof of Theorem~\\ref{thm:cert}. The first lemma explicitly computes all multipliers involving $x^*$ for the $n$-step certificate. \n\n\nThe next two lemmas are more substantial. These help us verify~\\eqref{eq:cert:2n+1}---which consists of a linear form in all $2n+3$ function values and a quadratic form in all $2n+3$ gradients and iterates. Na\\\"ively verifying such an identity requires checking $\\Theta(n)$ coefficients for the linear form and $\\Theta(n^2)$ coefficients for the quadratic form. The following two lemmas show that due to the recursive construction of the Silver Stepsize Schedule and the gluing, these forms only affect the indices $n,2n+1,*$. This reduces verifying the rate certificate~\\eqref{eq:cert:2n+1} to checking only $\\Theta(1)$ coefficients, as detailed below in \\S\\ref{ssec:gluing:pf}. Below, for shorthand, let $F_{ij} := 2(f_i - f_j)$ and $P_{ij} := 2\\langle g_j, x_j - x_i \\rangle - \\|g_i - g_j\\|^2$  denote the linear and quadratic components of $Q_{ij}$, respectively. For bookkeeping purposes, we use $3$-dimensional vectors and $4 \\times 4$ matrices to denote the coefficients of these forms. \n\n\nThe vectors $e, s, \\ell$ in Lemma~\\ref{lem:succinct-linear} are defined as follows:\n\\begin{align*}\n\t\\small\n\te := \\begin{bmatrix} \n\t\t\\frac{1}{r_k} \\\\\n\t\t-\\frac{1}{r_{k+1}} + \\frac{1+2\\rho}{r_k} \\\\\n\t\t\\frac{1}{r_{k+1}} - \\frac{4\\rho}{r_k}\n\t\\end{bmatrix}\n\t\\,,\n\t\\qquad\n\ts := 2\\rho(\\rho^{k} - 1) \\begin{bmatrix}\n\t\t1 \\\\\n\t\t1 \\\\\n\t\t-2\n\t\\end{bmatrix}\n\t\\,,\n\t\\qquad\n\t\\ell := \\begin{bmatrix}\n\t\t\\frac{1}{r_k} - 2\\rho(\\rho^k - 1)\\\\\n\t\t-\\frac{1}{r_{k+1}} + \\frac{1+2\\rho}{r_k} - 2\\rho(\\rho^k - 1) \\\\\n\t\t\\frac{1}{r_{k+1}} - \\frac{4\\rho}{r_k} + 4\\rho(\\rho^k - 1)\n\t\\end{bmatrix}\\,.\n\\end{align*}\n\n\nThe matrices in Lemma~\\ref{lem:succinct} are defined as follows. For brevity, we write $\\sim$ for the entries below the diagonal since these coefficient matrices are symmetric.\n\\begin{align*}\n\t\\small\n\tE :=\n\t\\begin{bmatrix}\n\t\t-2\\rho\n\t\t&\n\t\t(1 + 2\\rho) (1 + \\rho^{k-1}) -\\frac{1}{2r_k} \n\t\t&\n\t\t0\n\t\t&\n\t\t0\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\frac{1}{4r_k^2} - (1 + 2\\rho) (1+\\rho^{k-1})^2\n\t\t&\n\t\t0\n\t\t&\n\t\t0 \n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t2\\rho \n\t\t&\n\t\t\\frac{1}{2r_{k+1}} - \\frac{1+2\\rho}{2r_k}\n\t\t\\\\\n\t\t\\sim & \\sim & \\sim &\n\t\t\\frac{1+2\\rho}{4r_k^2} - \\frac{1}{4r_{k+1}^2}\n\t\\end{bmatrix}\n\\end{align*}\n\\begin{align*}\n\t\\small\n\tS := \n\t\\begin{bmatrix}\n\t\t0\n\t\t&\n\t\t\\Delta_{2n+1,n} + \\Delta_{*,n}\n\t\t&\n\t\t0\n\t\t&\n\t\t-\\Delta_{n,2n+1}\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t-\\left( \\Delta_{n,*} + \\Delta_{*,n} + \\Delta_{2n+1,n} + \\Delta_{n,2n+1} \\right)\n\t\t&\n\t\t-\\Delta_{2n+1,n}\n\t\t&\n\t\t\\Delta_{n,2n+1} + \\Delta_{2n+1,n}\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t0\n\t\t&\n\t\t\\Delta_{n,2n+1} + \\Delta_{*,2n+1}\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t- \\left( \\Delta_{n,2n+1} + \\Delta_{2n+1,n}  + \\Delta_{2n+1,*} + \\Delta_{*,2n+1}\\right)\n\t\\end{bmatrix}\n\\end{align*}\n\\begin{align*}\n\t\\small\n\tL :=  \\rho \\begin{bmatrix}\n\t\t-2\n\t\t&\n\t\t2 + \\rho^{k-1} \n\t\t&\n\t\t0\n\t\t&\n\t\t1\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t-1 - \\rho^k - 2 \\rho^{k-1} \n\t\t&\n\t\t\\rho^{k-1}\n\t\t&\n\t\t- 1 - \\rho^{k-1}\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t2\n\t\t&\n\t\t-1\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t1 - \\rho^k\n\t\\end{bmatrix}\n\\end{align*}\n","units":[{"raw_tex":"We revisit the classical problem of smooth convex optimization: solve $\\min_{x \\in \\R^d} f(x)$ where $f$ is convex and $M$-smooth (i.e., its gradient is $M$-Lipschitz). A celebrated result is that with a prudent choice of stepsizes $\\{\\alpha_t\\}$, the gradient descent algorithm (GD) \n\\begin{align}\n\tx_{t+1} = x_t - \\frac{\\alpha_t}{M} \\nabla f(x_t) \n\\end{align}\nsolves such a convex optimization problem to arbitrary accuracy from any initialization $x_0$. How quickly does GD converge? The mainstream approach (see e.g., the textbooks \\citep{BertsekasNonlinear,Nocedal,nesterov-survey,vishnoi2021algorithms,bubeck-book,BNO,lan2020first,bertsekas2015convex} among many others) is to use a constant stepsize schedule $\\alpha_t \\equiv \\bar{\\alpha} \\in (0,2)$ since this ensures\n\\begin{align}\n\tf(x_n) - f^* \\leq \\frac{cM\\|x_0 - x^*\\|^2}{n} \n\t\\label{eq:rate-std}\n\\end{align}\nwhere $x^*$ denotes any minimizer of $f$, $f^* := f(x^*)$ denotes the corresponding minimal value, and $c$ is a small constant, e.g., $c = \\tfrac{1}{4}$ for $\\bar{\\alpha} = 1$~\\citep{DT14}.\n","start":6414,"end":7475},{"raw_tex":"\\par In this setting, the Silver Stepsize Schedule is particularly simple. For any integer $n = 2^{k}-1$, we recursively construct the schedule $h_{2n+1}$ of length $2n+1$ from the schedule $h_n$ of length $n$ via\n\\begin{align}\n\th_{2n+1} := [h_n, \\; 1 + \\rho^{k-1}, \\; h_n]\n\t\\label{eq:steps-recursive}\n\\end{align}\nwhere $\\rho := 1 + \\sqrt{2}$ denotes the silver ratio, and $h_1 := [\\sqrt{2}]$. This results in the simple pattern $[\\sqrt{2}, \\; 2, \\; \\sqrt{2}, \\; 1+\\sqrt{2}, \\dots]$ as depicted in Figure~\\ref{fig:silverstepsize}. This schedule is exactly the Silver Stepsize Schedule from~\\citep{alt23hedging1} in the limit that the strong convexity parameter vanishes (see Remark~\\ref{rem:steps-limit} for details), and bears similarities to those in~\\citep{gupta22,grimmer23,GrimmerShuWang}; see \\S\\ref{ssec:rel}.\n","start":9151,"end":9968},{"raw_tex":"We show that these stepsizes yield an improved convergence rate~\\eqref{eq:rate-std} where $\\tfrac{c}{n}$ is replaced by\n\\begin{align}\n\tr_k := \\frac{1}{1 + \\sqrt{4\\rho^{2k} - 3}} \\leq\n\t\\frac{1}{2 \\rho^{\\log_2 n}}\n\t= \\frac{1}{2n^{\\log_2 \\rho}}\n\t\\approx \\frac{1}{2 n^{1.2716}}\n\t\\,.\n\t\\label{eq:r-asymptotics}\n\\end{align}\nThis bound gives the correct asymptotic scaling of $r_k$ since the inequality is asymptotically tight.\n","start":9970,"end":10390},{"raw_tex":"\\paragraph*{Indexing.} Throughout, the horizon is $n = 2^k-1$. This correspondence between $n \\in \\{1,3,7,15,\\dots\\}$ and $k \\in \\{1,2,3,4,\\dots\\}$ allows re-indexing in a way that simplifies notation for the recursion.\n\n\n\\paragraph*{Rescaling.} For simplicity, we normalize $M = 1$, i.e., $\\| \\nabla f(x) - \\nabla f(y) \\| \\leq \\|x-y\\|$ for all $x,y$. This is without loss of generality since if $h$ is $M$-smooth and convex,  then $f := h/M$ is $1$-smooth and convex, thus the result we establish $f_n - f^* \\leq r_k\\|x_0 - x^*\\|^2$ implies $h_n - h^* \\leq r_k M \\|x_0 - x^*\\|^2$. Note that running GD on $f$ simply amounts to rescaling the Silver Stepsize Schedules for $h$ by $1/M$.\n","start":16612,"end":17298},{"raw_tex":" The Silver Stepsize Schedule is defined recursively in~\\eqref{eq:steps-recursive}. Here we mention an equivalent direct expression and make several remarks. Let $\\bin(t)$ denote the $2$-adic valuation of $t$, i.e., the smallest non-negative integer $i$ such that $2^i$ is in the binary expansion of $t$. For example, $\\bin(1) = 0$, $\\bin(2) = 1$, $\\bin(3) = 0$, $\\bin(4) = 2$, etc.\n\n\n\\begin{defin}[Silver Stepsize Schedule for smooth convex optimization]\n\tFor $t \\in \\{0,1, 2,\\dots\\}$, the $t$-th stepsize of the Silver Stepsize Schedule is\n\t\t\\begin{align}\n\t\t\\alpha_t\n\t\t:=\n\t\t1 + \\rho^{\\bin(t+1)-1}\\,.\n\t\t\\label{eq:steps}\n\t\\end{align}\n\\end{defin}\n","start":17362,"end":18008},{"raw_tex":"\\par Briefly, the idea behind multi-step descent is that it is essential to capture how different iterations affect other iterations' progress. We do this by exploiting long-range consistency conditions between the iterates along GD's trajectory, as encoded by the co-coercivities \n\\begin{align}\n\tQ_{ij} := 2(f_i - f_j) + 2\\langle g_j, x_j - x_i \\rangle - \\|g_j - g_i\\|^2\\,.\n\t\\label{eq:cocoercivity}\n\\end{align}\nThe significance of these co-coercivities is that the constraints $\\{Q_{ij} \\geq 0\\}_{i \\neq j \\in \\{0,1,\\dots,n,*\\}}$ are necessary and sufficient for the existence of a $1$-smooth convex function $f$ satisfying $f_i = f(x_i)$ and $g_i = \\nabla f(x_i)$ for each $i \\in \\{0,1,\\dots,n,*\\}$~\\citep{pesto}. In other words, the co-coercivity conditions $\\{Q_{ij} \\geq 0 \\}_{i \\neq j \\in \\{0,1,\\dots,n,*\\}}$ generate all possible long-range consistency constraints on the objective function $f$. Or, said another way, the co-coercivity conditions generate all possible valid inequalities with which one can prove convergence rates for GD. For a further discussion, see~\\citep[\\S2.2]{alt23hedging1}.\n\n\\par Concretely, to prove Theorem~\\ref{thm:main}, we exhibit explicit non-negative multipliers $\\lam_{ij}$ satisfying\n\\begin{align}\n\t\\sum_{i \\neq j \\in \\{0, \\dots, n,*\\}} \\lam_{ij} Q_{ij}  = \n\t\\|x_0 - x^*\\|^2 - \\|x_n - c_k g_n - x^*\\|^2\n\t+ \\frac{f^* - f_n}{r_k}\\,,\n\t\\label{eq:cert}\n\\end{align}\nwhere we use the shorthand $c_k := \\frac{1}{2r_k}$. Since $\\sum_{ij}\\lam_{ij} Q_{ij} \\geq 0$ for any $1$-smooth convex function, and since $\\|x_n - c_k g_n - x^*\\|^2 \\geq 0$ trivially as it is a square, this immediately implies the desired rate \n\\begin{align}\n\tf_n - f^* \\leq r_k \\|x_0 - x^*\\|^2\\,.\n\t\\label{eq:cert-cor}\n\\end{align}\n","start":20197,"end":21930},{"raw_tex":"\\begin{example}[$n=1$]\\label{ex:n=1}\n\tFor $n=1$, the identity~\\eqref{eq:cert} is\n\t\\begin{align*}\n\t\t\\sum_{i \\neq j \\in \\{0, 1, *\\}} \\lam_{ij} Q_{ij} \n\t\t=\n\t\t\\|x_0 - x^*\\|^2 - \\|x_1 - c_1 g_1 - x^*\\|^2 + \\frac{f^* - f_1}{r_1}\\,.\n\t\\end{align*}\n\tHere, $x_1 = x_0 - \\sqrt{2} g_0$, $r_1 \n\t= \\frac{1}{1+\\sqrt{4\\rho^2-3}}\n\t\\approx 0.1816$, $c_1 = \\frac{1}{2r_1} \\approx 2.7535$, and \n\t\\begin{align*}\n\t\t\\begin{bmatrix}\n\t\t\t\\lam_{00} & \\lam_{01} & \\lam_{0*} \\\\\n\t\t\t\\lam_{10} & \\lam_{11} & \\lam_{1*} \\\\\n\t\t\t\\lam_{*0} & \\lam_{*1} & \\lam_{**}\n\t\t\\end{bmatrix}\n\t\t=\n\t\t\\begin{bmatrix}\n\t\t\t0\n\t\t\t&\n\t\t\t1+\\sqrt{2}\n\t\t\t&\n\t\t\t0\n\t\t\t\\\\\n\t\t\t1\n\t\t\t&\n\t\t\t0\n\t\t\t&\n\t\t\t\\sqrt{2}\n\t\t\t\\\\\n\t\t\t\\sqrt{2}\n\t\t\t&\n\t\t\t\\frac{1}{2r_1}\n\t\t\t&\n\t\t\t0\n\t\t\\end{bmatrix}\n\t\t\\,.\n\t\\end{align*}\n\\end{example}","start":22940,"end":23676},{"raw_tex":"\\par For larger horizons $n$, we construct $\\lam_{ij}$ via the recursive gluing technique of~\\citep{alt23hedging1}. See Figure~\\ref{fig:gluing}. Below, we say that the multipliers $\\{\\sig_{ij}\\}_{i,j \\in \\{0,\\dots,n,*\\}}$ satisfy the $*$-sparsity property if $\\sig_{i*} = 0$ for all $i < n$. This is satisfied by construction and simplifies part of the proof (isolated in Lemma~\\ref{lem:opt}). \n","start":23680,"end":24075},{"raw_tex":"The rest of the section is dedicated to proving Theorem~\\ref{thm:cert}. We first isolate two helper lemmas in \\S\\ref{ssec:gluing:helper}, and then we combine them to prove the result in \\S\\ref{ssec:gluing:pf}.  For notational simplicity, henceforth we assume $x^* = 0$; this is without loss of generality after translating. \n\n\\subsection{Helper lemmas}\\label{ssec:gluing:helper}\n\nHere we provide three helper lemmas for the proof of Theorem~\\ref{thm:cert}. The first lemma explicitly computes all multipliers involving $x^*$ for the $n$-step certificate. \n","start":27659,"end":28215},{"raw_tex":"The next two lemmas are more substantial. These help us verify~\\eqref{eq:cert:2n+1}---which consists of a linear form in all $2n+3$ function values and a quadratic form in all $2n+3$ gradients and iterates. Na\\\"ively verifying such an identity requires checking $\\Theta(n)$ coefficients for the linear form and $\\Theta(n^2)$ coefficients for the quadratic form. The following two lemmas show that due to the recursive construction of the Silver Stepsize Schedule and the gluing, these forms only affect the indices $n,2n+1,*$. This reduces verifying the rate certificate~\\eqref{eq:cert:2n+1} to checking only $\\Theta(1)$ coefficients, as detailed below in \\S\\ref{ssec:gluing:pf}. Below, for shorthand, let $F_{ij} := 2(f_i - f_j)$ and $P_{ij} := 2\\langle g_j, x_j - x_i \\rangle - \\|g_i - g_j\\|^2$  denote the linear and quadratic components of $Q_{ij}$, respectively. For bookkeeping purposes, we use $3$-dimensional vectors and $4 \\times 4$ matrices to denote the coefficients of these forms. \n","start":28912,"end":29907},{"raw_tex":"The vectors $e, s, \\ell$ in Lemma~\\ref{lem:succinct-linear} are defined as follows:\n\\begin{align*}\n\t\\small\n\te := \\begin{bmatrix} \n\t\t\\frac{1}{r_k} \\\\\n\t\t-\\frac{1}{r_{k+1}} + \\frac{1+2\\rho}{r_k} \\\\\n\t\t\\frac{1}{r_{k+1}} - \\frac{4\\rho}{r_k}\n\t\\end{bmatrix}\n\t\\,,\n\t\\qquad\n\ts := 2\\rho(\\rho^{k} - 1) \\begin{bmatrix}\n\t\t1 \\\\\n\t\t1 \\\\\n\t\t-2\n\t\\end{bmatrix}\n\t\\,,\n\t\\qquad\n\t\\ell := \\begin{bmatrix}\n\t\t\\frac{1}{r_k} - 2\\rho(\\rho^k - 1)\\\\\n\t\t-\\frac{1}{r_{k+1}} + \\frac{1+2\\rho}{r_k} - 2\\rho(\\rho^k - 1) \\\\\n\t\t\\frac{1}{r_{k+1}} - \\frac{4\\rho}{r_k} + 4\\rho(\\rho^k - 1)\n\t\\end{bmatrix}\\,.\n\\end{align*}\n","start":33033,"end":33605},{"raw_tex":"The matrices in Lemma~\\ref{lem:succinct} are defined as follows. For brevity, we write $\\sim$ for the entries below the diagonal since these coefficient matrices are symmetric.\n\\begin{align*}\n\t\\small\n\tE :=\n\t\\begin{bmatrix}\n\t\t-2\\rho\n\t\t&\n\t\t(1 + 2\\rho) (1 + \\rho^{k-1}) -\\frac{1}{2r_k} \n\t\t&\n\t\t0\n\t\t&\n\t\t0\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\frac{1}{4r_k^2} - (1 + 2\\rho) (1+\\rho^{k-1})^2\n\t\t&\n\t\t0\n\t\t&\n\t\t0 \n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t2\\rho \n\t\t&\n\t\t\\frac{1}{2r_{k+1}} - \\frac{1+2\\rho}{2r_k}\n\t\t\\\\\n\t\t\\sim & \\sim & \\sim &\n\t\t\\frac{1+2\\rho}{4r_k^2} - \\frac{1}{4r_{k+1}^2}\n\t\\end{bmatrix}\n\\end{align*}\n\\begin{align*}\n\t\\small\n\tS := \n\t\\begin{bmatrix}\n\t\t0\n\t\t&\n\t\t\\Delta_{2n+1,n} + \\Delta_{*,n}\n\t\t&\n\t\t0\n\t\t&\n\t\t-\\Delta_{n,2n+1}\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t-\\left( \\Delta_{n,*} + \\Delta_{*,n} + \\Delta_{2n+1,n} + \\Delta_{n,2n+1} \\right)\n\t\t&\n\t\t-\\Delta_{2n+1,n}\n\t\t&\n\t\t\\Delta_{n,2n+1} + \\Delta_{2n+1,n}\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t0\n\t\t&\n\t\t\\Delta_{n,2n+1} + \\Delta_{*,2n+1}\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t- \\left( \\Delta_{n,2n+1} + \\Delta_{2n+1,n}  + \\Delta_{2n+1,*} + \\Delta_{*,2n+1}\\right)\n\t\\end{bmatrix}\n\\end{align*}\n\\begin{align*}\n\t\\small\n\tL :=  \\rho \\begin{bmatrix}\n\t\t-2\n\t\t&\n\t\t2 + \\rho^{k-1} \n\t\t&\n\t\t0\n\t\t&\n\t\t1\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t-1 - \\rho^k - 2 \\rho^{k-1} \n\t\t&\n\t\t\\rho^{k-1}\n\t\t&\n\t\t- 1 - \\rho^{k-1}\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t2\n\t\t&\n\t\t-1\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t1 - \\rho^k\n\t\\end{bmatrix}\n\\end{align*}\n","start":33694,"end":35070}]},"statement":{"raw_tex":"\\begin{theorem}[Main result]\\label{thm:main}\n\tFor any horizon $n = 2^k - 1$, any dimension $d$, any $M$-smooth convex function $f : \\R^d \\to \\R$, and any initialization $x_0 \\in \\R^d$, \n\t\\begin{align}\n\t\tf(x_n) - f^* \\leq r_k\\, M\\|x_0 - x^*\\|^2 \\,,\n\t\\end{align}\n\twhere $x^*$ denotes any minimizer of $f$, and $x_n$ denotes the output of $n$ steps of GD using the Silver Stepsize Schedule. In particular, in order to achieve error $f(x_n) - f^* \\leq \\eps$, it suffices to run GD for\n\t\\begin{align*}\n\t\tn\n\t\t\\leq\n\t\t\\left( \\frac{M \\|x_0 - x^*\\|^2}{2 \\eps} \\right)^{\\log_{\\rho} 2} \n\t\t\\approx\n\t\t\\left( \\frac{M\\|x_0 - x^*\\|2}{2 \\eps} \\right)^{0.7864}\n\t\t\\;\\; \\text{iterations}.\n\t\\end{align*}\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}[Main result]\\label{thm:main}\n\tFor any horizon $n = 2^k - 1$, any dimension $d$, any $M$-smooth convex function $f : \\R^d \\to \\R$, and any initialization $x_0 \\in \\R^d$, \n\t\\begin{align}\n\t\tf(x_n) - f^* \\leq r_k\\, M\\|x_0 - x^*\\|^2 \\,,\n\t\\end{align}\n\twhere $x^*$ denotes any minimizer of $f$, and $x_n$ denotes the output of $n$ steps of GD using the Silver Stepsize Schedule. In particular, in order to achieve error $f(x_n) - f^* \\leq \\eps$, it suffices to run GD for\n\t\\begin{align*}\n\t\tn\n\t\t\\leq\n\t\t\\left( \\frac{M \\|x_0 - x^*\\|^2}{2 \\eps} \\right)^{\\log_{\\rho} 2} \n\t\t\\approx\n\t\t\\left( \\frac{M\\|x_0 - x^*\\|2}{2 \\eps} \\right)^{0.7864}\n\t\t\\;\\; \\text{iterations}.\n\t\\end{align*}\n\\end{theorem}","start":10393,"end":11088}]},"proof":{"raw_tex":"\\begin{lemma}[Sum of Silver Stepsizes]\\label{lem:sum-steps}\n\t$\\sum_{t=0}^{n-1} \\alpha_t = \\rho^k - 1$ for any $k \\in \\N$. \n\\end{lemma}\n\n\\begin{proof}\n\tThe base case $k=1$ is trivial. The inductive step follows from the recursion~\\eqref{eq:steps-recursive}.\n\\end{proof}\n\n\\begin{theorem}[Recursive gluing]\\label{thm:cert}\n\tLet $n = 2^k-1$. Suppose $\\{\\sig_{ij}\\}_{i,j \\in \\{0,\\dots,n,*\\}} $ satisfies $*$-sparsity and certifies the $n$-step rate, i.e.,\n\t\\begin{align}\n\t\t\\sum_{i \\neq j \\in \\{0, \\dots, n,*\\}} \\sig_{ij} Q_{ij} \n\t\t=\n\t\t\\|x_0 - x^*\\|^2 - \\|x_n - c_k g_n - x^*\\|^2  + \\frac{f^* - f_n}{r_k}\\ .\n\t\t\\label{eq:cert:n}\n\t\\end{align}\n\tThen there exists $\\{\\lam_{ij}\\}_{i, j \\in \\{0,\\dots,2n+1,*\\}}$ that satisfies $*$-sparsity and certifies the $2n+1$-step rate, i.e.,\n\t\\begin{align}\n\t\t\\sum_{i \\neq j \\in \\{0, \\dots, 2n+1,*\\}} \\lam_{ij} Q_{ij} \n\t\t=\n\t\t\\|x_0 - x^*\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1} - x^*\\|^2 + \\frac{f^* - f_{2n+1}}{r_{k+1}}\\ .\n\t\t\\label{eq:cert:2n+1}\n\t\\end{align}\n\tMoreover, this certificate is explicitly given by\n\t\\begin{align}\n\t\t\\lam_{ij} = \\underbrace{\\Theta_{ij}}_{\\text{gluing}}\n\t\t+\n\t\t\\underbrace{\\Xi_{ij}}_{\\text{rank-one correction}}\n\t\t+\n\t\t\\underbrace{\\Delta_{ij}}_{\\text{sparse correction}}\n\t\t\\label{eq:lambda}\n\t\\end{align}\n\tThe ``gluing component'' $\\Theta$ is defined as \n\t\\begin{align}\n\t\t\\Theta_{ij} := \\underbrace{\\sig_{i,j} \\cdot \\mathds{1}_{i,j \\in \\{0,\\dots,n,*\\}}}_{\\text{recurrence for first $n$ steps}}\n\t\t\\;\\;+\\;\\;\n\t\t\\underbrace{ (1 + 2\\rho)\\; \\sig_{i-n-1,j-n-1} \\cdot \\mathds{1}_{i,j \\in \\{n+1,\\dots,2n+1,*\\}}}_{\\text{recurrence for last $n$ steps}}\\,.\n\t\t\\label{eq:Theta}\n\t\\end{align}\n\tThe ``rank-one correction'' $\\Xi$ is zero except the entries $\\{\\Xi_{ij}\\}_{i \\in \\{n,2n+1,*\\},\\; j \\in \\{n+1, \\dots, 2n\\}}$ which are\n\t\\begin{align}\n\t\t\\begin{bmatrix}\n\t\t\t\\Xi_{n,n+1} & \\Xi_{n,n+2} & \\cdots & \\Xi_{n,2n} \\\\\n\t\t\t\\Xi_{2n+1,n+1} & \\Xi_{2n+1,n+2} & \\cdots & \\Xi_{2n+1,2n} \\\\\n\t\t\t\\Xi_{*,n+1} & \\Xi_{*,n+2} & \\cdots & \\Xi_{*,2n}\n\t\t\\end{bmatrix}\n\t\t:=\n\t\t\\rho \\begin{bmatrix}\n\t\t\t1 \\\\\n\t\t\t1 \\\\\n\t\t\t-2\n\t\t\\end{bmatrix} \\begin{bmatrix}\n\t\t\t\\alpha_{n+1} & \\alpha_{n+2} & \\cdots & \\alpha_{2n}\n\t\t\\end{bmatrix}\n\t\t\\label{eq:Xi}\n\t\\end{align}\n\tThe ``sparse correction'' $\\Delta$ is zero except the entries $\\{\\Delta_{ij}\\}_{i \\neq j \\in \\{n,2n+1,*\\}}$ which are\n\t\\begin{align}\n\t\t\\begin{bmatrix}\n\t\t\t\\Delta_{n,n}\n\t\t\t&\n\t\t\t\\Delta_{n,2n+1}\n\t\t\t&\n\t\t\t\\Delta_{n,*} \n\t\t\t\\\\\n\t\t\t\\Delta_{2n+1,n}\n\t\t\t&\n\t\t\t\\Delta_{2n+1,2n+1} \n\t\t\t&\n\t\t\t\\Delta_{2n+1,*} \n\t\t\t\\\\\n\t\t\t\\Delta_{*,n}\n\t\t\t&\n\t\t\t\\Delta_{*,2n+1} \n\t\t\t&\n\t\t\t\\Delta_{*,*}\n\t\t\\end{bmatrix}\n\t\t:=\n\t\t\\begin{bmatrix}\n\t\t\t0 \n\t\t\t&\n\t\t\t\\rho\n\t\t\t&\n\t\t\t1 - \\rho^k\n\t\t\t\\\\\n\t\t\t\\rho^k\n\t\t\t&\n\t\t\t0 \n\t\t\t&\n\t\t\t2\\rho - \\sqrt{2} \\rho^{k+1}\n\t\t\t\\\\\n\t\t\t1 + \\rho^{k-1} - \\frac{1}{2r_k}\n\t\t\t&\n\t\t\t\\frac{1}{2r_{k+1}} - \\frac{1+2\\rho}{2r_k}\n\t\t\t&\n\t\t\t0\n\t\t\\end{bmatrix}\n\t\t\\label{eq:Del}\n\t\\end{align}\n\\end{theorem}\n\n\\begin{proof}[Proof of Theorem~\\ref{thm:main}]\n\tIt suffices to prove~\\eqref{eq:cert}; we prove this by induction. The base case $n=1$ is Example~\\ref{ex:n=1}. The inductive step is Theorem~\\ref{thm:cert}. \n\\end{proof}\n\n\\begin{lemma}[Multipliers involving $x^*$]\\label{lem:opt} \n\t$\\sig_{n*} = \\rho^k - 1$, $\\sig_{*n} = \\frac{1}{2r_k}$, and $\\sig_{*t} = \\alpha_t$ for $t \\in \\{0,\\dots,n-1\\}$. \n\\end{lemma}\n\n\\begin{proof}\n\tExpand both sides of the identity~\\eqref{eq:cert:n} using the $*$-sparsity assumption, the definition of the co-coercivities, and the definition of GD, i.e., $x_t = x_0 - \\sum_{s=0}^{t-1} \\alpha_s g_s$. Matching coefficients for the $\\langle x_0, g_t \\rangle$ terms yields the claimed formulas for $\\sig_{*t}$. Matching coefficients for the $f^*$ term gives the identity $\\sig_{n*} = \\sum_{j=0}^n \\sig_{*j}-  \\tfrac{1}{2r_k}$. This equals $\\rho^k - 1$ by Lemma~\\ref{lem:sum-steps}. \n\\end{proof}\n\n\\begin{lemma}[Succinct linear forms]\\label{lem:succinct-linear}\n\tLet $u := [f_n, f_{2n+1}, f^*]^T$, and let $e, s, \\ell \\in \\R^3$ be the vectors defined in Appendix~\\ref{app:succinct-linear}. \n\t\\begin{itemize}\n\t\t\\item \\underline{Gluing error.} $\\frac{f^* - f_{2n+1}}{r_{k+1}} - \\sum_{ij} \\Theta_{ij} F_{ij} = \\langle e, u \\rangle$\n\t\t\\item \\underline{Sparse correction.} $\\sum_{ij} \\Delta_{ij} F_{ij} = \\langle s, u \\rangle$\n\t\t\\item \\underline{Rank-one correction.} $\\sum_{ij} \\Xi_{ij} F_{ij} = \\langle \\ell, u \\rangle$\n\t\\end{itemize}\n\\end{lemma}\n\n\\begin{proof}\n\tExpand the definition of co-coercivities, simplify the rank-one correction using Lemma~\\ref{lem:sum-steps}, and simplify the sparse correction using the Pell recurrence $\\rho^{k+1} = 2\\rho^k + \\rho^{k-1}$.\n\\end{proof}\n\n\\begin{lemma}[Succinct quadratic forms]\\label{lem:succinct} Let $v := [x_n,g_n,x_{2n+1},g_{2n+1}]^T$, and let $E$, $S$, $L$ be the $4 \\times 4$ matrices defined in Appendix~\\ref{app:succinct}.\n\t\\begin{itemize}\n\t\t\\item \\underline{Gluing error.} \n\t\t$\\|x_0\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1}\\|^2\n\t\t- \\sum_{ij} \\Theta_{ij} P_{ij} = \\langle E, vv^T \\rangle$\n\t\t\\item \\underline{Sparse correction.} $\\sum_{ij} \\Delta_{ij} P_{ij} = \\langle S, vv^T \\rangle$\n\t\t\\item \\underline{Rank-one correction.} $\\sum_{ij} \\Xi_{ij} P_{ij} = \\langle L, vv^T \\rangle$\n\t\\end{itemize}\n\\end{lemma}\n\n\\begin{proof}\n\tDeferred to Appendix~\\ref{app:succinct} for brevity.\n\\end{proof}\n\n\\subsection{Proof of recursive gluing (Theorem~\\ref{thm:cert})}\\label{ssec:gluing:pf}\n\n\\paragraph*{Non-negativity.} We verify $\\lam_{ij} \\geq 0$ for all $i \\neq j \\in \\{0, \\dots, 2n+1,*\\}$. For nearly all entries, this is obvious since $\\lam_{ij}$ is constructed by adding and multiplying non-negative numbers. For the remaining entries where either $\\Xi$ or $\\Delta$ is negative, compute the corresponding entry of $\\lam$ by summing the corrections and using Lemma~\\ref{lem:opt}. This gives $\\lam_{n*} = 0$, $\\lam_{2n+1,*} = \\rho^{k+1} - 1$, $\\lam_{*,n} = \\rho^{k-1} + 1$, $\\lam_{*,2n+1} = \\tfrac{1}{2r_{k+1}}$, and $\\lam_{*,t} = \\alpha_t$ for all $t \\in \\{n+1,\\dots,2n\\}$. All these entries are clearly non-negative.\n\n\\paragraph*{Rate certificate.} The identity~\\eqref{eq:cert:2n+1} has two components: a linear form in the function values and a quadratic form in the iterates and gradients. For the linear form, it suffices to verify $e-s-\\ell = 0$ by Lemma~\\ref{lem:succinct-linear}, where $e,s,\\ell \\in \\R^3$ are the vectors defined in Appendix~\\ref{app:succinct-linear}. This is obvious by inspection. For the quadratic form, it suffices to verify $E - S - L = 0$, where $E,S,L \\in \\R^{4 \\times 4}$ are the matrices in Lemma~\\ref{lem:succinct} defined in Appendix~\\ref{app:succinct}. By plugging in the explicit values for $r_k$ and $\\Delta$, this is straightforward to check by hand. For brevity, we provide a simple Mathematica script that verifies these identities at the URL~\\citep{MathematicaURL}.\n \n\n\n\t \n\t \n\t \\appendix\n\t \n\n\n\\begin{proof}[Proof of Lemma~\\ref{lem:succinct}]\n\t\\underline{Gluing component.} \n\tRecall that $h_{2n+1} = [h_n, \\alpha_n, h_n]$ by the recursive construction~\\eqref{eq:steps-recursive} of the Silver Stepsize Schedule. Thus, since $\\sig$ certifies the $n$-step rate, we have\n\t\\begin{align*}\n\t\t\\sum_{i \\neq j \\in \\{0, \\dots, n,*\\}} \\sig_{ij} P_{ij} \n\t\t&=\n\t\t\\|x_0\\|^2 - \\|x_n - c_k g_n\\|^2\n\t\t\\\\ \n\t\t\\sum_{i \\neq j \\in \\{n+1, \\dots, 2n,*\\}} \\sig_{ij} P_{ij} \n\t\t&=\n\t\t\\|x_{n+1}\\|^2 - \\|x_{2n+1} - c_k g_{2n+1}\\|^2\n\t\\end{align*}\n\tBy definition, $\\sum_{ij} \\Theta_{ij} P_{ij}$ is the former plus $(1 + 2\\rho)$ times the latter. \n\tThus the gluing error $\\|x_0\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1}\\|^2 - \\sum_{ij} \\Theta_{ij} P_{ij}$ is equal to\n\t\\begin{align*}\n\t\t(1+2\\rho) \\|x_{2n+1} - c_k g_{2n+1}\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1}\\|^2 + \\|x_n - c_k g_n\\|^2 - (1 + 2\\rho) \\|x_{n+1}\\|^2\n\t\\end{align*}\n\tThe key point is that after \n\texpanding $x_{n+1} = x_n - \\alpha_n g_n$, this is a quadratic form in only the $4$ variables $x_n, g_n, x_{2n+1},g_{2n+1}$. Collecting terms, substituting $c_k = \\tfrac{1}{2r_k}$, and using the definition of the Silver Stepsize $\\alpha_n = 1 + \\rho^{k-1}$ yields $\\langle E, vv^T \\rangle$, as desired.\n\t\n\t\\par \\underline{Sparse component.} Expand the definition of the co-coercivity and collect terms. Note also that this equals the matrix $S$ in Part~I~\\citep{alt23hedging1}, in the limit that the strong convexity tends to $0$.\n\t\n\t\\par \\underline{Low-rank component.}  Expanding the co-coercivities, using the identity $x_{2n+1} - x_{n+1}= - \\sum_{t=n+1}^{2n} \\alpha_t g_t$ by definition of GD, and using the identity $\\sum_{t=n+1}^{2n} \\alpha_t = \\sum_{t=0}^{n-1} \\alpha_t = \\rho^k - 1$ from Lemma~\\ref{lem:sum-steps}, \n\t\\begin{align*}\n\t\t\\sum_{t=n+1}^{2n} \\Big( \\Xi_{n,t} P_{n,t} - \\Xi_{*,t} P_{*,t} \\Big)\n\t\t=\n\t\t\\rho \\sum_{t=n+1}^{2n} \\alpha_t \\Big(\n\t\t2 \\langle g_t, g_n - x_n \\rangle - \\|g_n\\|^2\n\t\t\\Big)\n\t\t=\n\t\t2\\rho \\langle x_{2n+1} - x_{n+1}, x_n - g_n \\rangle - \\rho(\\rho^k - 1 ) \\|g_n\\|^2 \\,.\n\t\\end{align*}\n\tAn identical calculation yields\n\t\\begin{align*}\n\t\t\\sum_{t=n+1}^{2n} \\Big( \\Xi_{2n+1,t} P_{2n+1,t} - \\Xi_{*,t} P_{*,t} \\Big)\n\t\t&=\n\t\t2\\rho \\langle x_{2n+1} - x_{n+1}, x_{2n+1} - g_{2n+1} \\rangle - \\rho(\\rho^k - 1 ) \\|g_{2n+1}\\|^2 \\,.\n\t\\end{align*}\n\tNow sum these two displays, use the definition of GD to expand $x_{n+1} = x_n - \\alpha_n g_n$, and use the definition of the Silver Stepsize $\\alpha_n = 1 + \\rho^{k-1}$. This gives $\\langle L, vv^T \\rangle$, as desired.\n\\end{proof}","units":[{"raw_tex":"\\begin{lemma}[Sum of Silver Stepsizes]\\label{lem:sum-steps}\n\t$\\sum_{t=0}^{n-1} \\alpha_t = \\rho^k - 1$ for any $k \\in \\N$. \n\\end{lemma}","start":19217,"end":19351},{"raw_tex":"\\begin{proof}\n\tThe base case $k=1$ is trivial. The inductive step follows from the recursion~\\eqref{eq:steps-recursive}.\n\\end{proof}","start":19352,"end":19484},{"raw_tex":"\\begin{theorem}[Recursive gluing]\\label{thm:cert}\n\tLet $n = 2^k-1$. Suppose $\\{\\sig_{ij}\\}_{i,j \\in \\{0,\\dots,n,*\\}} $ satisfies $*$-sparsity and certifies the $n$-step rate, i.e.,\n\t\\begin{align}\n\t\t\\sum_{i \\neq j \\in \\{0, \\dots, n,*\\}} \\sig_{ij} Q_{ij} \n\t\t=\n\t\t\\|x_0 - x^*\\|^2 - \\|x_n - c_k g_n - x^*\\|^2  + \\frac{f^* - f_n}{r_k}\\ .\n\t\t\\label{eq:cert:n}\n\t\\end{align}\n\tThen there exists $\\{\\lam_{ij}\\}_{i, j \\in \\{0,\\dots,2n+1,*\\}}$ that satisfies $*$-sparsity and certifies the $2n+1$-step rate, i.e.,\n\t\\begin{align}\n\t\t\\sum_{i \\neq j \\in \\{0, \\dots, 2n+1,*\\}} \\lam_{ij} Q_{ij} \n\t\t=\n\t\t\\|x_0 - x^*\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1} - x^*\\|^2 + \\frac{f^* - f_{2n+1}}{r_{k+1}}\\ .\n\t\t\\label{eq:cert:2n+1}\n\t\\end{align}\n\tMoreover, this certificate is explicitly given by\n\t\\begin{align}\n\t\t\\lam_{ij} = \\underbrace{\\Theta_{ij}}_{\\text{gluing}}\n\t\t+\n\t\t\\underbrace{\\Xi_{ij}}_{\\text{rank-one correction}}\n\t\t+\n\t\t\\underbrace{\\Delta_{ij}}_{\\text{sparse correction}}\n\t\t\\label{eq:lambda}\n\t\\end{align}\n\tThe ``gluing component'' $\\Theta$ is defined as \n\t\\begin{align}\n\t\t\\Theta_{ij} := \\underbrace{\\sig_{i,j} \\cdot \\mathds{1}_{i,j \\in \\{0,\\dots,n,*\\}}}_{\\text{recurrence for first $n$ steps}}\n\t\t\\;\\;+\\;\\;\n\t\t\\underbrace{ (1 + 2\\rho)\\; \\sig_{i-n-1,j-n-1} \\cdot \\mathds{1}_{i,j \\in \\{n+1,\\dots,2n+1,*\\}}}_{\\text{recurrence for last $n$ steps}}\\,.\n\t\t\\label{eq:Theta}\n\t\\end{align}\n\tThe ``rank-one correction'' $\\Xi$ is zero except the entries $\\{\\Xi_{ij}\\}_{i \\in \\{n,2n+1,*\\},\\; j \\in \\{n+1, \\dots, 2n\\}}$ which are\n\t\\begin{align}\n\t\t\\begin{bmatrix}\n\t\t\t\\Xi_{n,n+1} & \\Xi_{n,n+2} & \\cdots & \\Xi_{n,2n} \\\\\n\t\t\t\\Xi_{2n+1,n+1} & \\Xi_{2n+1,n+2} & \\cdots & \\Xi_{2n+1,2n} \\\\\n\t\t\t\\Xi_{*,n+1} & \\Xi_{*,n+2} & \\cdots & \\Xi_{*,2n}\n\t\t\\end{bmatrix}\n\t\t:=\n\t\t\\rho \\begin{bmatrix}\n\t\t\t1 \\\\\n\t\t\t1 \\\\\n\t\t\t-2\n\t\t\\end{bmatrix} \\begin{bmatrix}\n\t\t\t\\alpha_{n+1} & \\alpha_{n+2} & \\cdots & \\alpha_{2n}\n\t\t\\end{bmatrix}\n\t\t\\label{eq:Xi}\n\t\\end{align}\n\tThe ``sparse correction'' $\\Delta$ is zero except the entries $\\{\\Delta_{ij}\\}_{i \\neq j \\in \\{n,2n+1,*\\}}$ which are\n\t\\begin{align}\n\t\t\\begin{bmatrix}\n\t\t\t\\Delta_{n,n}\n\t\t\t&\n\t\t\t\\Delta_{n,2n+1}\n\t\t\t&\n\t\t\t\\Delta_{n,*} \n\t\t\t\\\\\n\t\t\t\\Delta_{2n+1,n}\n\t\t\t&\n\t\t\t\\Delta_{2n+1,2n+1} \n\t\t\t&\n\t\t\t\\Delta_{2n+1,*} \n\t\t\t\\\\\n\t\t\t\\Delta_{*,n}\n\t\t\t&\n\t\t\t\\Delta_{*,2n+1} \n\t\t\t&\n\t\t\t\\Delta_{*,*}\n\t\t\\end{bmatrix}\n\t\t:=\n\t\t\\begin{bmatrix}\n\t\t\t0 \n\t\t\t&\n\t\t\t\\rho\n\t\t\t&\n\t\t\t1 - \\rho^k\n\t\t\t\\\\\n\t\t\t\\rho^k\n\t\t\t&\n\t\t\t0 \n\t\t\t&\n\t\t\t2\\rho - \\sqrt{2} \\rho^{k+1}\n\t\t\t\\\\\n\t\t\t1 + \\rho^{k-1} - \\frac{1}{2r_k}\n\t\t\t&\n\t\t\t\\frac{1}{2r_{k+1}} - \\frac{1+2\\rho}{2r_k}\n\t\t\t&\n\t\t\t0\n\t\t\\end{bmatrix}\n\t\t\\label{eq:Del}\n\t\\end{align}\n\\end{theorem}","start":24614,"end":27161},{"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm:main}]\n\tIt suffices to prove~\\eqref{eq:cert}; we prove this by induction. The base case $n=1$ is Example~\\ref{ex:n=1}. The inductive step is Theorem~\\ref{thm:cert}. \n\\end{proof}","start":27439,"end":27656},{"raw_tex":"\\begin{lemma}[Multipliers involving $x^*$]\\label{lem:opt} \n\t$\\sig_{n*} = \\rho^k - 1$, $\\sig_{*n} = \\frac{1}{2r_k}$, and $\\sig_{*t} = \\alpha_t$ for $t \\in \\{0,\\dots,n-1\\}$. \n\\end{lemma}","start":28216,"end":28400},{"raw_tex":"\\begin{proof}\n\tExpand both sides of the identity~\\eqref{eq:cert:n} using the $*$-sparsity assumption, the definition of the co-coercivities, and the definition of GD, i.e., $x_t = x_0 - \\sum_{s=0}^{t-1} \\alpha_s g_s$. Matching coefficients for the $\\langle x_0, g_t \\rangle$ terms yields the claimed formulas for $\\sig_{*t}$. Matching coefficients for the $f^*$ term gives the identity $\\sig_{n*} = \\sum_{j=0}^n \\sig_{*j}-  \\tfrac{1}{2r_k}$. This equals $\\rho^k - 1$ by Lemma~\\ref{lem:sum-steps}. \n\\end{proof}","start":28401,"end":28910},{"raw_tex":"\\begin{lemma}[Succinct linear forms]\\label{lem:succinct-linear}\n\tLet $u := [f_n, f_{2n+1}, f^*]^T$, and let $e, s, \\ell \\in \\R^3$ be the vectors defined in Appendix~\\ref{app:succinct-linear}. \n\t\\begin{itemize}\n\t\t\\item \\underline{Gluing error.} $\\frac{f^* - f_{2n+1}}{r_{k+1}} - \\sum_{ij} \\Theta_{ij} F_{ij} = \\langle e, u \\rangle$\n\t\t\\item \\underline{Sparse correction.} $\\sum_{ij} \\Delta_{ij} F_{ij} = \\langle s, u \\rangle$\n\t\t\\item \\underline{Rank-one correction.} $\\sum_{ij} \\Xi_{ij} F_{ij} = \\langle \\ell, u \\rangle$\n\t\\end{itemize}\n\\end{lemma}","start":29910,"end":30455},{"raw_tex":"\\begin{proof}\n\tExpand the definition of co-coercivities, simplify the rank-one correction using Lemma~\\ref{lem:sum-steps}, and simplify the sparse correction using the Pell recurrence $\\rho^{k+1} = 2\\rho^k + \\rho^{k-1}$.\n\\end{proof}","start":30456,"end":30688},{"raw_tex":"\\begin{lemma}[Succinct quadratic forms]\\label{lem:succinct} Let $v := [x_n,g_n,x_{2n+1},g_{2n+1}]^T$, and let $E$, $S$, $L$ be the $4 \\times 4$ matrices defined in Appendix~\\ref{app:succinct}.\n\t\\begin{itemize}\n\t\t\\item \\underline{Gluing error.} \n\t\t$\\|x_0\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1}\\|^2\n\t\t- \\sum_{ij} \\Theta_{ij} P_{ij} = \\langle E, vv^T \\rangle$\n\t\t\\item \\underline{Sparse correction.} $\\sum_{ij} \\Delta_{ij} P_{ij} = \\langle S, vv^T \\rangle$\n\t\t\\item \\underline{Rank-one correction.} $\\sum_{ij} \\Xi_{ij} P_{ij} = \\langle L, vv^T \\rangle$\n\t\\end{itemize}\n\\end{lemma}","start":30690,"end":31261},{"raw_tex":"\\begin{proof}\n\tDeferred to Appendix~\\ref{app:succinct} for brevity.\n\\end{proof}","start":31262,"end":31341},{"raw_tex":"\\subsection{Proof of recursive gluing (Theorem~\\ref{thm:cert})}\\label{ssec:gluing:pf}\n\n\\paragraph*{Non-negativity.} We verify $\\lam_{ij} \\geq 0$ for all $i \\neq j \\in \\{0, \\dots, 2n+1,*\\}$. For nearly all entries, this is obvious since $\\lam_{ij}$ is constructed by adding and multiplying non-negative numbers. For the remaining entries where either $\\Xi$ or $\\Delta$ is negative, compute the corresponding entry of $\\lam$ by summing the corrections and using Lemma~\\ref{lem:opt}. This gives $\\lam_{n*} = 0$, $\\lam_{2n+1,*} = \\rho^{k+1} - 1$, $\\lam_{*,n} = \\rho^{k-1} + 1$, $\\lam_{*,2n+1} = \\tfrac{1}{2r_{k+1}}$, and $\\lam_{*,t} = \\alpha_t$ for all $t \\in \\{n+1,\\dots,2n\\}$. All these entries are clearly non-negative.\n\n\\paragraph*{Rate certificate.} The identity~\\eqref{eq:cert:2n+1} has two components: a linear form in the function values and a quadratic form in the iterates and gradients. For the linear form, it suffices to verify $e-s-\\ell = 0$ by Lemma~\\ref{lem:succinct-linear}, where $e,s,\\ell \\in \\R^3$ are the vectors defined in Appendix~\\ref{app:succinct-linear}. This is obvious by inspection. For the quadratic form, it suffices to verify $E - S - L = 0$, where $E,S,L \\in \\R^{4 \\times 4}$ are the matrices in Lemma~\\ref{lem:succinct} defined in Appendix~\\ref{app:succinct}. By plugging in the explicit values for $r_k$ and $\\Delta$, this is straightforward to check by hand. For brevity, we provide a simple Mathematica script that verifies these identities at the URL~\\citep{MathematicaURL}.\n \n\n\n\t \n\t \n\t \\appendix\n\t \n","start":31347,"end":32881},{"raw_tex":"\\begin{proof}[Proof of Lemma~\\ref{lem:succinct}]\n\t\\underline{Gluing component.} \n\tRecall that $h_{2n+1} = [h_n, \\alpha_n, h_n]$ by the recursive construction~\\eqref{eq:steps-recursive} of the Silver Stepsize Schedule. Thus, since $\\sig$ certifies the $n$-step rate, we have\n\t\\begin{align*}\n\t\t\\sum_{i \\neq j \\in \\{0, \\dots, n,*\\}} \\sig_{ij} P_{ij} \n\t\t&=\n\t\t\\|x_0\\|^2 - \\|x_n - c_k g_n\\|^2\n\t\t\\\\ \n\t\t\\sum_{i \\neq j \\in \\{n+1, \\dots, 2n,*\\}} \\sig_{ij} P_{ij} \n\t\t&=\n\t\t\\|x_{n+1}\\|^2 - \\|x_{2n+1} - c_k g_{2n+1}\\|^2\n\t\\end{align*}\n\tBy definition, $\\sum_{ij} \\Theta_{ij} P_{ij}$ is the former plus $(1 + 2\\rho)$ times the latter. \n\tThus the gluing error $\\|x_0\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1}\\|^2 - \\sum_{ij} \\Theta_{ij} P_{ij}$ is equal to\n\t\\begin{align*}\n\t\t(1+2\\rho) \\|x_{2n+1} - c_k g_{2n+1}\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1}\\|^2 + \\|x_n - c_k g_n\\|^2 - (1 + 2\\rho) \\|x_{n+1}\\|^2\n\t\\end{align*}\n\tThe key point is that after \n\texpanding $x_{n+1} = x_n - \\alpha_n g_n$, this is a quadratic form in only the $4$ variables $x_n, g_n, x_{2n+1},g_{2n+1}$. Collecting terms, substituting $c_k = \\tfrac{1}{2r_k}$, and using the definition of the Silver Stepsize $\\alpha_n = 1 + \\rho^{k-1}$ yields $\\langle E, vv^T \\rangle$, as desired.\n\t\n\t\\par \\underline{Sparse component.} Expand the definition of the co-coercivity and collect terms. Note also that this equals the matrix $S$ in Part~I~\\citep{alt23hedging1}, in the limit that the strong convexity tends to $0$.\n\t\n\t\\par \\underline{Low-rank component.}  Expanding the co-coercivities, using the identity $x_{2n+1} - x_{n+1}= - \\sum_{t=n+1}^{2n} \\alpha_t g_t$ by definition of GD, and using the identity $\\sum_{t=n+1}^{2n} \\alpha_t = \\sum_{t=0}^{n-1} \\alpha_t = \\rho^k - 1$ from Lemma~\\ref{lem:sum-steps}, \n\t\\begin{align*}\n\t\t\\sum_{t=n+1}^{2n} \\Big( \\Xi_{n,t} P_{n,t} - \\Xi_{*,t} P_{*,t} \\Big)\n\t\t=\n\t\t\\rho \\sum_{t=n+1}^{2n} \\alpha_t \\Big(\n\t\t2 \\langle g_t, g_n - x_n \\rangle - \\|g_n\\|^2\n\t\t\\Big)\n\t\t=\n\t\t2\\rho \\langle x_{2n+1} - x_{n+1}, x_n - g_n \\rangle - \\rho(\\rho^k - 1 ) \\|g_n\\|^2 \\,.\n\t\\end{align*}\n\tAn identical calculation yields\n\t\\begin{align*}\n\t\t\\sum_{t=n+1}^{2n} \\Big( \\Xi_{2n+1,t} P_{2n+1,t} - \\Xi_{*,t} P_{*,t} \\Big)\n\t\t&=\n\t\t2\\rho \\langle x_{2n+1} - x_{n+1}, x_{2n+1} - g_{2n+1} \\rangle - \\rho(\\rho^k - 1 ) \\|g_{2n+1}\\|^2 \\,.\n\t\\end{align*}\n\tNow sum these two displays, use the definition of GD to expand $x_{n+1} = x_n - \\alpha_n g_n$, and use the definition of the Silver Stepsize $\\alpha_n = 1 + \\rho^{k-1}$. This gives $\\langle L, vv^T \\rangle$, as desired.\n\\end{proof}","start":35071,"end":37587}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"光滑凸优化问题","source_units":[{"raw_tex":"We revisit the classical problem of smooth convex optimization: solve $\\min_{x \\in \\R^d} f(x)$ where $f$ is convex and $M$-smooth (i.e., its gradient is $M$-Lipschitz). A celebrated result is that with a prudent choice of stepsizes $\\{\\alpha_t\\}$, the gradient descent algorithm (GD) \n","start":6414,"end":6699}]},{"id":"S002","semantic_type":"algorithm","kind":"algorithm","labels":[],"declared_external":false,"source_only":false,"title":"梯度下降更新","source_units":[{"raw_tex":"\\begin{align}\n\tx_{t+1} = x_t - \\frac{\\alpha_t}{M} \\nabla f(x_t) \n\\end{align}\n","start":6699,"end":6776}]},{"id":"S003","semantic_type":"external_fact","kind":"external_fact","labels":["eq:rate-std"],"declared_external":true,"declaration_text":["原文第217行列举的优化教材；这里只保留来源背景，不作为所选证明的输入。"],"source_only":false,"title":"常步长背景界","source_units":[{"raw_tex":"solves such a convex optimization problem to arbitrary accuracy from any initialization $x_0$. How quickly does GD converge? The mainstream approach (see e.g., the textbooks \\citep{BertsekasNonlinear,Nocedal,nesterov-survey,vishnoi2021algorithms,bubeck-book,BNO,lan2020first,bertsekas2015convex} among many others) is to use a constant stepsize schedule $\\alpha_t \\equiv \\bar{\\alpha} \\in (0,2)$ since this ensures\n\\begin{align}\n\tf(x_n) - f^* \\leq \\frac{cM\\|x_0 - x^*\\|^2}{n} \n\t\\label{eq:rate-std}\n\\end{align}\n","start":6776,"end":7285}]},{"id":"S004","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"最优点与最优值","source_units":[{"raw_tex":"where $x^*$ denotes any minimizer of $f$, $f^* := f(x^*)$ denotes the corresponding minimal value, and $c$ is a small constant, e.g., $c = \\tfrac{1}{4}$ for $\\bar{\\alpha} = 1$~\\citep{DT14}.\n","start":7285,"end":7475}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":["eq:steps-recursive"],"declared_external":false,"source_only":false,"title":"银步长递归","source_units":[{"raw_tex":"\\par In this setting, the Silver Stepsize Schedule is particularly simple. For any integer $n = 2^{k}-1$, we recursively construct the schedule $h_{2n+1}$ of length $2n+1$ from the schedule $h_n$ of length $n$ via\n\\begin{align}\n\th_{2n+1} := [h_n, \\; 1 + \\rho^{k-1}, \\; h_n]\n\t\\label{eq:steps-recursive}\n\\end{align}\nwhere $\\rho := 1 + \\sqrt{2}$ denotes the silver ratio, and $h_1 := [\\sqrt{2}]$. This results in the simple pattern $[\\sqrt{2}, \\; 2, \\; \\sqrt{2}, \\; 1+\\sqrt{2}, \\dots]$ as depicted in Figure~\\ref{fig:silverstepsize}. This schedule is exactly the Silver Stepsize Schedule from~\\citep{alt23hedging1} in the limit that the strong convexity parameter vanishes (see Remark~\\ref{rem:steps-limit} for details), and bears similarities to those in~\\citep{gupta22,grimmer23,GrimmerShuWang}; see \\S\\ref{ssec:rel}.\n","start":9151,"end":9968}]},{"id":"S006","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"步长来源说明","source_units":[{"raw_tex":"where $\\rho := 1 + \\sqrt{2}$ denotes the silver ratio, and $h_1 := [\\sqrt{2}]$. This results in the simple pattern $[\\sqrt{2}, \\; 2, \\; \\sqrt{2}, \\; 1+\\sqrt{2}, \\dots]$ as depicted in Figure~\\ref{fig:silverstepsize}. This schedule is exactly the Silver Stepsize Schedule from~\\citep{alt23hedging1} in the limit that the strong convexity parameter vanishes (see Remark~\\ref{rem:steps-limit} for details), and bears similarities to those in~\\citep{gupta22,grimmer23,GrimmerShuWang}; see \\S\\ref{ssec:rel}.\n","start":9465,"end":9968}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":["eq:r-asymptotics"],"declared_external":false,"source_only":false,"title":"速率参数","source_units":[{"raw_tex":"We show that these stepsizes yield an improved convergence rate~\\eqref{eq:rate-std} where $\\tfrac{c}{n}$ is replaced by\n\\begin{align}\n\tr_k := \\frac{1}{1 + \\sqrt{4\\rho^{2k} - 3}} \\leq\n\t\\frac{1}{2 \\rho^{\\log_2 n}}\n\t= \\frac{1}{2n^{\\log_2 \\rho}}\n\t\\approx \\frac{1}{2 n^{1.2716}}\n\t\\,.\n\t\\label{eq:r-asymptotics}\n\\end{align}\n","start":9970,"end":10287}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"速率渐近比较","source_units":[{"raw_tex":"\tr_k := \\frac{1}{1 + \\sqrt{4\\rho^{2k} - 3}} \\leq\n\t\\frac{1}{2 \\rho^{\\log_2 n}}\n\t= \\frac{1}{2n^{\\log_2 \\rho}}\n\t\\approx \\frac{1}{2 n^{1.2716}}\n\t\\,.\n\t\\label{eq:r-asymptotics}\n\\end{align}\nThis bound gives the correct asymptotic scaling of $r_k$ since the inequality is asymptotically tight.\n","start":10104,"end":10390}]},{"id":"S009","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"终止时刻索引","source_units":[{"raw_tex":"\\paragraph*{Indexing.} Throughout, the horizon is $n = 2^k-1$. This correspondence between $n \\in \\{1,3,7,15,\\dots\\}$ and $k \\in \\{1,2,3,4,\\dots\\}$ allows re-indexing in a way that simplifies notation for the recursion.\n\n\n","start":16612,"end":16834}]},{"id":"S010","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"单位光滑性归一化","source_units":[{"raw_tex":"\\paragraph*{Rescaling.} For simplicity, we normalize $M = 1$, i.e., $\\| \\nabla f(x) - \\nabla f(y) \\| \\leq \\|x-y\\|$ for all $x,y$. This is without loss of generality since if $h$ is $M$-smooth and convex,  then $f := h/M$ is $1$-smooth and convex, thus the result we establish $f_n - f^* \\leq r_k\\|x_0 - x^*\\|^2$ implies $h_n - h^* \\leq r_k M \\|x_0 - x^*\\|^2$. Note that running GD on $f$ simply amounts to rescaling the Silver Stepsize Schedules for $h$ by $1/M$.\n","start":16834,"end":17298}]},{"id":"S011","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"二进赋值记号","source_units":[{"raw_tex":" The Silver Stepsize Schedule is defined recursively in~\\eqref{eq:steps-recursive}. Here we mention an equivalent direct expression and make several remarks. Let $\\bin(t)$ denote the $2$-adic valuation of $t$, i.e., the smallest non-negative integer $i$ such that $2^i$ is in the binary expansion of $t$. For example, $\\bin(1) = 0$, $\\bin(2) = 1$, $\\bin(3) = 0$, $\\bin(4) = 2$, etc.\n\n\n","start":17362,"end":17747}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["eq:steps"],"declared_external":false,"source_only":false,"title":"显式银步长","source_units":[{"raw_tex":"\\begin{defin}[Silver Stepsize Schedule for smooth convex optimization]\n\tFor $t \\in \\{0,1, 2,\\dots\\}$, the $t$-th stepsize of the Silver Stepsize Schedule is\n\t\t\\begin{align}\n\t\t\\alpha_t\n\t\t:=\n\t\t1 + \\rho^{\\bin(t+1)-1}\\,.\n\t\t\\label{eq:steps}\n\t\\end{align}\n\\end{defin}\n","start":17747,"end":18008}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["eq:cocoercivity"],"declared_external":false,"source_only":false,"title":"余强制性量","source_units":[{"raw_tex":"\\par Briefly, the idea behind multi-step descent is that it is essential to capture how different iterations affect other iterations' progress. We do this by exploiting long-range consistency conditions between the iterates along GD's trajectory, as encoded by the co-coercivities \n\\begin{align}\n\tQ_{ij} := 2(f_i - f_j) + 2\\langle g_j, x_j - x_i \\rangle - \\|g_j - g_i\\|^2\\,.\n\t\\label{eq:cocoercivity}\n\\end{align}\n","start":20197,"end":20609}]},{"id":"S014","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisite 列表第一项；原文第384行 citep{pesto}。只需要其中余强制性不等式的必要性。"],"source_only":false,"title":"光滑凸插值必要性","source_units":[{"raw_tex":"The significance of these co-coercivities is that the constraints $\\{Q_{ij} \\geq 0\\}_{i \\neq j \\in \\{0,1,\\dots,n,*\\}}$ are necessary and sufficient for the existence of a $1$-smooth convex function $f$ satisfying $f_i = f(x_i)$ and $g_i = \\nabla f(x_i)$ for each $i \\in \\{0,1,\\dots,n,*\\}$~\\citep{pesto}. In other words, the co-coercivity conditions $\\{Q_{ij} \\geq 0 \\}_{i \\neq j \\in \\{0,1,\\dots,n,*\\}}$ generate all possible long-range consistency constraints on the objective function $f$. Or, said another way, the co-coercivity conditions generate all possible valid inequalities with which one can prove convergence rates for GD. For a further discussion, see~\\citep[\\S2.2]{alt23hedging1}.\n\n","start":20609,"end":21304}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":["eq:cert"],"declared_external":false,"source_only":false,"title":"多步速率证书","source_units":[{"raw_tex":"\\par Concretely, to prove Theorem~\\ref{thm:main}, we exhibit explicit non-negative multipliers $\\lam_{ij}$ satisfying\n\\begin{align}\n\t\\sum_{i \\neq j \\in \\{0, \\dots, n,*\\}} \\lam_{ij} Q_{ij}  = \n\t\\|x_0 - x^*\\|^2 - \\|x_n - c_k g_n - x^*\\|^2\n\t+ \\frac{f^* - f_n}{r_k}\\,,\n\t\\label{eq:cert}\n\\end{align}\n","start":21304,"end":21598}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"终端平移系数","source_units":[{"raw_tex":"where we use the shorthand $c_k := \\frac{1}{2r_k}$. Since $\\sum_{ij}\\lam_{ij} Q_{ij} \\geq 0$ for any $1$-smooth convex function, and since $\\|x_n - c_k g_n - x^*\\|^2 \\geq 0$ trivially as it is a square, this immediately implies the desired rate \n","start":21598,"end":21844}]},{"id":"S017","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:cert-cor"],"declared_external":false,"source_only":false,"title":"证书推出误差界","source_units":[{"raw_tex":"where we use the shorthand $c_k := \\frac{1}{2r_k}$. Since $\\sum_{ij}\\lam_{ij} Q_{ij} \\geq 0$ for any $1$-smooth convex function, and since $\\|x_n - c_k g_n - x^*\\|^2 \\geq 0$ trivially as it is a square, this immediately implies the desired rate \n\\begin{align}\n\tf_n - f^* \\leq r_k \\|x_0 - x^*\\|^2\\,.\n\t\\label{eq:cert-cor}\n\\end{align}\n","start":21598,"end":21930}]},{"id":"S018","semantic_type":"proposition","kind":"proposition","labels":["ex:n=1"],"declared_external":false,"source_only":false,"title":"一步基础证书","source_units":[{"raw_tex":"\\begin{example}[$n=1$]\\label{ex:n=1}\n\tFor $n=1$, the identity~\\eqref{eq:cert} is\n\t\\begin{align*}\n\t\t\\sum_{i \\neq j \\in \\{0, 1, *\\}} \\lam_{ij} Q_{ij} \n\t\t=\n\t\t\\|x_0 - x^*\\|^2 - \\|x_1 - c_1 g_1 - x^*\\|^2 + \\frac{f^* - f_1}{r_1}\\,.\n\t\\end{align*}\n\tHere, $x_1 = x_0 - \\sqrt{2} g_0$, $r_1 \n\t= \\frac{1}{1+\\sqrt{4\\rho^2-3}}\n\t\\approx 0.1816$, $c_1 = \\frac{1}{2r_1} \\approx 2.7535$, and \n\t\\begin{align*}\n\t\t\\begin{bmatrix}\n\t\t\t\\lam_{00} & \\lam_{01} & \\lam_{0*} \\\\\n\t\t\t\\lam_{10} & \\lam_{11} & \\lam_{1*} \\\\\n\t\t\t\\lam_{*0} & \\lam_{*1} & \\lam_{**}\n\t\t\\end{bmatrix}\n\t\t=\n\t\t\\begin{bmatrix}\n\t\t\t0\n\t\t\t&\n\t\t\t1+\\sqrt{2}\n\t\t\t&\n\t\t\t0\n\t\t\t\\\\\n\t\t\t1\n\t\t\t&\n\t\t\t0\n\t\t\t&\n\t\t\t\\sqrt{2}\n\t\t\t\\\\\n\t\t\t\\sqrt{2}\n\t\t\t&\n\t\t\t\\frac{1}{2r_1}\n\t\t\t&\n\t\t\t0\n\t\t\\end{bmatrix}\n\t\t\\,.\n\t\\end{align*}\n\\end{example}","start":22940,"end":23676}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":["ex:n=1"],"declared_external":false,"source_only":false,"title":"基础乘子数据","source_units":[{"raw_tex":"\tHere, $x_1 = x_0 - \\sqrt{2} g_0$, $r_1 \n\t= \\frac{1}{1+\\sqrt{4\\rho^2-3}}\n\t\\approx 0.1816$, $c_1 = \\frac{1}{2r_1} \\approx 2.7535$, and \n\t\\begin{align*}\n\t\t\\begin{bmatrix}\n\t\t\t\\lam_{00} & \\lam_{01} & \\lam_{0*} \\\\\n\t\t\t\\lam_{10} & \\lam_{11} & \\lam_{1*} \\\\\n\t\t\t\\lam_{*0} & \\lam_{*1} & \\lam_{**}\n\t\t\\end{bmatrix}\n\t\t=\n\t\t\\begin{bmatrix}\n\t\t\t0\n\t\t\t&\n\t\t\t1+\\sqrt{2}\n\t\t\t&\n\t\t\t0\n\t\t\t\\\\\n\t\t\t1\n\t\t\t&\n\t\t\t0\n\t\t\t&\n\t\t\t\\sqrt{2}\n\t\t\t\\\\\n\t\t\t\\sqrt{2}\n\t\t\t&\n\t\t\t\\frac{1}{2r_1}\n\t\t\t&\n\t\t\t0\n\t\t\\end{bmatrix}\n\t\t\\,.\n\t\\end{align*}\n\\end{example}","start":23180,"end":23676}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"星号稀疏性","source_units":[{"raw_tex":"\\par For larger horizons $n$, we construct $\\lam_{ij}$ via the recursive gluing technique of~\\citep{alt23hedging1}. See Figure~\\ref{fig:gluing}. Below, we say that the multipliers $\\{\\sig_{ij}\\}_{i,j \\in \\{0,\\dots,n,*\\}}$ satisfy the $*$-sparsity property if $\\sig_{i*} = 0$ for all $i < n$. This is satisfied by construction and simplifies part of the proof (isolated in Lemma~\\ref{lem:opt}). \n","start":23680,"end":24075}]},{"id":"S021","semantic_type":"theorem","kind":"theorem","labels":["thm:main"],"declared_external":false,"source_only":false,"title":"银步长收敛主定理","source_units":[{"raw_tex":"\\begin{theorem}[Main result]\\label{thm:main}\n\tFor any horizon $n = 2^k - 1$, any dimension $d$, any $M$-smooth convex function $f : \\R^d \\to \\R$, and any initialization $x_0 \\in \\R^d$, \n\t\\begin{align}\n\t\tf(x_n) - f^* \\leq r_k\\, M\\|x_0 - x^*\\|^2 \\,,\n\t\\end{align}\n\twhere $x^*$ denotes any minimizer of $f$, and $x_n$ denotes the output of $n$ steps of GD using the Silver Stepsize Schedule. In particular, in order to achieve error $f(x_n) - f^* \\leq \\eps$, it suffices to run GD for\n\t\\begin{align*}\n\t\tn\n\t\t\\leq\n\t\t\\left( \\frac{M \\|x_0 - x^*\\|^2}{2 \\eps} \\right)^{\\log_{\\rho} 2} \n\t\t\\approx\n\t\t\\left( \\frac{M\\|x_0 - x^*\\|2}{2 \\eps} \\right)^{0.7864}\n\t\t\\;\\; \\text{iterations}.\n\t\\end{align*}\n\\end{theorem}","start":10393,"end":11088}]},{"id":"S022","semantic_type":"lemma","kind":"lemma","labels":["lem:sum-steps"],"declared_external":false,"source_only":false,"title":"银步长之和","source_units":[{"raw_tex":"\\begin{lemma}[Sum of Silver Stepsizes]\\label{lem:sum-steps}\n\t$\\sum_{t=0}^{n-1} \\alpha_t = \\rho^k - 1$ for any $k \\in \\N$. \n\\end{lemma}","start":19217,"end":19351}]},{"id":"S023","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"步长求和作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\tThe base case $k=1$ is trivial. The inductive step follows from the recursion~\\eqref{eq:steps-recursive}.\n\\end{proof}","start":19352,"end":19484}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"求和归纳基础","source_units":[{"raw_tex":"\tThe base case $k=1$ is trivial. The inductive step follows from the recursion~\\eqref{eq:steps-recursive}.\n","start":19366,"end":19473}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"求和递归步骤","source_units":[{"raw_tex":"\tThe base case $k=1$ is trivial. The inductive step follows from the recursion~\\eqref{eq:steps-recursive}.\n","start":19366,"end":19473}]},{"id":"S026","semantic_type":"theorem","kind":"theorem","labels":["eq:Del","eq:Theta","eq:Xi","eq:cert:2n+1","eq:cert:n","eq:lambda","thm:cert"],"declared_external":false,"source_only":false,"title":"递归拼接定理","source_units":[{"raw_tex":"\\begin{theorem}[Recursive gluing]\\label{thm:cert}\n\tLet $n = 2^k-1$. Suppose $\\{\\sig_{ij}\\}_{i,j \\in \\{0,\\dots,n,*\\}} $ satisfies $*$-sparsity and certifies the $n$-step rate, i.e.,\n\t\\begin{align}\n\t\t\\sum_{i \\neq j \\in \\{0, \\dots, n,*\\}} \\sig_{ij} Q_{ij} \n\t\t=\n\t\t\\|x_0 - x^*\\|^2 - \\|x_n - c_k g_n - x^*\\|^2  + \\frac{f^* - f_n}{r_k}\\ .\n\t\t\\label{eq:cert:n}\n\t\\end{align}\n\tThen there exists $\\{\\lam_{ij}\\}_{i, j \\in \\{0,\\dots,2n+1,*\\}}$ that satisfies $*$-sparsity and certifies the $2n+1$-step rate, i.e.,\n\t\\begin{align}\n\t\t\\sum_{i \\neq j \\in \\{0, \\dots, 2n+1,*\\}} \\lam_{ij} Q_{ij} \n\t\t=\n\t\t\\|x_0 - x^*\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1} - x^*\\|^2 + \\frac{f^* - f_{2n+1}}{r_{k+1}}\\ .\n\t\t\\label{eq:cert:2n+1}\n\t\\end{align}\n\tMoreover, this certificate is explicitly given by\n\t\\begin{align}\n\t\t\\lam_{ij} = \\underbrace{\\Theta_{ij}}_{\\text{gluing}}\n\t\t+\n\t\t\\underbrace{\\Xi_{ij}}_{\\text{rank-one correction}}\n\t\t+\n\t\t\\underbrace{\\Delta_{ij}}_{\\text{sparse correction}}\n\t\t\\label{eq:lambda}\n\t\\end{align}\n\tThe ``gluing component'' $\\Theta$ is defined as \n\t\\begin{align}\n\t\t\\Theta_{ij} := \\underbrace{\\sig_{i,j} \\cdot \\mathds{1}_{i,j \\in \\{0,\\dots,n,*\\}}}_{\\text{recurrence for first $n$ steps}}\n\t\t\\;\\;+\\;\\;\n\t\t\\underbrace{ (1 + 2\\rho)\\; \\sig_{i-n-1,j-n-1} \\cdot \\mathds{1}_{i,j \\in \\{n+1,\\dots,2n+1,*\\}}}_{\\text{recurrence for last $n$ steps}}\\,.\n\t\t\\label{eq:Theta}\n\t\\end{align}\n\tThe ``rank-one correction'' $\\Xi$ is zero except the entries $\\{\\Xi_{ij}\\}_{i \\in \\{n,2n+1,*\\},\\; j \\in \\{n+1, \\dots, 2n\\}}$ which are\n\t\\begin{align}\n\t\t\\begin{bmatrix}\n\t\t\t\\Xi_{n,n+1} & \\Xi_{n,n+2} & \\cdots & \\Xi_{n,2n} \\\\\n\t\t\t\\Xi_{2n+1,n+1} & \\Xi_{2n+1,n+2} & \\cdots & \\Xi_{2n+1,2n} \\\\\n\t\t\t\\Xi_{*,n+1} & \\Xi_{*,n+2} & \\cdots & \\Xi_{*,2n}\n\t\t\\end{bmatrix}\n\t\t:=\n\t\t\\rho \\begin{bmatrix}\n\t\t\t1 \\\\\n\t\t\t1 \\\\\n\t\t\t-2\n\t\t\\end{bmatrix} \\begin{bmatrix}\n\t\t\t\\alpha_{n+1} & \\alpha_{n+2} & \\cdots & \\alpha_{2n}\n\t\t\\end{bmatrix}\n\t\t\\label{eq:Xi}\n\t\\end{align}\n\tThe ``sparse correction'' $\\Delta$ is zero except the entries $\\{\\Delta_{ij}\\}_{i \\neq j \\in \\{n,2n+1,*\\}}$ which are\n\t\\begin{align}\n\t\t\\begin{bmatrix}\n\t\t\t\\Delta_{n,n}\n\t\t\t&\n\t\t\t\\Delta_{n,2n+1}\n\t\t\t&\n\t\t\t\\Delta_{n,*} \n\t\t\t\\\\\n\t\t\t\\Delta_{2n+1,n}\n\t\t\t&\n\t\t\t\\Delta_{2n+1,2n+1} \n\t\t\t&\n\t\t\t\\Delta_{2n+1,*} \n\t\t\t\\\\\n\t\t\t\\Delta_{*,n}\n\t\t\t&\n\t\t\t\\Delta_{*,2n+1} \n\t\t\t&\n\t\t\t\\Delta_{*,*}\n\t\t\\end{bmatrix}\n\t\t:=\n\t\t\\begin{bmatrix}\n\t\t\t0 \n\t\t\t&\n\t\t\t\\rho\n\t\t\t&\n\t\t\t1 - \\rho^k\n\t\t\t\\\\\n\t\t\t\\rho^k\n\t\t\t&\n\t\t\t0 \n\t\t\t&\n\t\t\t2\\rho - \\sqrt{2} \\rho^{k+1}\n\t\t\t\\\\\n\t\t\t1 + \\rho^{k-1} - \\frac{1}{2r_k}\n\t\t\t&\n\t\t\t\\frac{1}{2r_{k+1}} - \\frac{1+2\\rho}{2r_k}\n\t\t\t&\n\t\t\t0\n\t\t\\end{bmatrix}\n\t\t\\label{eq:Del}\n\t\\end{align}\n\\end{theorem}","start":24614,"end":27161}]},{"id":"S027","semantic_type":"assumption","kind":"assumption","labels":["eq:cert:n","thm:cert"],"declared_external":false,"source_only":false,"title":"拼接输入证书假设","source_units":[{"raw_tex":"\\begin{theorem}[Recursive gluing]\\label{thm:cert}\n\tLet $n = 2^k-1$. Suppose $\\{\\sig_{ij}\\}_{i,j \\in \\{0,\\dots,n,*\\}} $ satisfies $*$-sparsity and certifies the $n$-step rate, i.e.,\n\t\\begin{align}\n\t\t\\sum_{i \\neq j \\in \\{0, \\dots, n,*\\}} \\sig_{ij} Q_{ij} \n\t\t=\n\t\t\\|x_0 - x^*\\|^2 - \\|x_n - c_k g_n - x^*\\|^2  + \\frac{f^* - f_n}{r_k}\\ .\n\t\t\\label{eq:cert:n}\n\t\\end{align}\n","start":24614,"end":24979}]},{"id":"S028","semantic_type":"definition","kind":"definition","labels":["eq:cert:2n+1","thm:cert"],"declared_external":false,"source_only":false,"title":"拼接目标证书形式","source_units":[{"raw_tex":"\tThen there exists $\\{\\lam_{ij}\\}_{i, j \\in \\{0,\\dots,2n+1,*\\}}$ that satisfies $*$-sparsity and certifies the $2n+1$-step rate, i.e.,\n\t\\begin{align}\n\t\t\\sum_{i \\neq j \\in \\{0, \\dots, 2n+1,*\\}} \\lam_{ij} Q_{ij} \n\t\t=\n\t\t\\|x_0 - x^*\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1} - x^*\\|^2 + \\frac{f^* - f_{2n+1}}{r_{k+1}}\\ .\n\t\t\\label{eq:cert:2n+1}\n\t\\end{align}\n","start":24979,"end":25326}]},{"id":"S029","semantic_type":"definition","kind":"definition","labels":["eq:lambda","thm:cert"],"declared_external":false,"source_only":false,"title":"拼接乘子分解","source_units":[{"raw_tex":"\tMoreover, this certificate is explicitly given by\n\t\\begin{align}\n\t\t\\lam_{ij} = \\underbrace{\\Theta_{ij}}_{\\text{gluing}}\n\t\t+\n\t\t\\underbrace{\\Xi_{ij}}_{\\text{rank-one correction}}\n\t\t+\n\t\t\\underbrace{\\Delta_{ij}}_{\\text{sparse correction}}\n\t\t\\label{eq:lambda}\n\t\\end{align}\n","start":25326,"end":25595}]},{"id":"S030","semantic_type":"definition","kind":"definition","labels":["eq:Theta","thm:cert"],"declared_external":false,"source_only":false,"title":"两段拼接分量","source_units":[{"raw_tex":"\tThe ``gluing component'' $\\Theta$ is defined as \n\t\\begin{align}\n\t\t\\Theta_{ij} := \\underbrace{\\sig_{i,j} \\cdot \\mathds{1}_{i,j \\in \\{0,\\dots,n,*\\}}}_{\\text{recurrence for first $n$ steps}}\n\t\t\\;\\;+\\;\\;\n\t\t\\underbrace{ (1 + 2\\rho)\\; \\sig_{i-n-1,j-n-1} \\cdot \\mathds{1}_{i,j \\in \\{n+1,\\dots,2n+1,*\\}}}_{\\text{recurrence for last $n$ steps}}\\,.\n\t\t\\label{eq:Theta}\n\t\\end{align}\n","start":25595,"end":25967}]},{"id":"S031","semantic_type":"definition","kind":"definition","labels":["eq:Xi","thm:cert"],"declared_external":false,"source_only":false,"title":"秩一修正","source_units":[{"raw_tex":"\tThe ``rank-one correction'' $\\Xi$ is zero except the entries $\\{\\Xi_{ij}\\}_{i \\in \\{n,2n+1,*\\},\\; j \\in \\{n+1, \\dots, 2n\\}}$ which are\n\t\\begin{align}\n\t\t\\begin{bmatrix}\n\t\t\t\\Xi_{n,n+1} & \\Xi_{n,n+2} & \\cdots & \\Xi_{n,2n} \\\\\n\t\t\t\\Xi_{2n+1,n+1} & \\Xi_{2n+1,n+2} & \\cdots & \\Xi_{2n+1,2n} \\\\\n\t\t\t\\Xi_{*,n+1} & \\Xi_{*,n+2} & \\cdots & \\Xi_{*,2n}\n\t\t\\end{bmatrix}\n\t\t:=\n\t\t\\rho \\begin{bmatrix}\n\t\t\t1 \\\\\n\t\t\t1 \\\\\n\t\t\t-2\n\t\t\\end{bmatrix} \\begin{bmatrix}\n\t\t\t\\alpha_{n+1} & \\alpha_{n+2} & \\cdots & \\alpha_{2n}\n\t\t\\end{bmatrix}\n\t\t\\label{eq:Xi}\n\t\\end{align}\n","start":25967,"end":26501}]},{"id":"S032","semantic_type":"definition","kind":"definition","labels":["eq:Del","thm:cert"],"declared_external":false,"source_only":false,"title":"稀疏修正","source_units":[{"raw_tex":"\tThe ``sparse correction'' $\\Delta$ is zero except the entries $\\{\\Delta_{ij}\\}_{i \\neq j \\in \\{n,2n+1,*\\}}$ which are\n\t\\begin{align}\n\t\t\\begin{bmatrix}\n\t\t\t\\Delta_{n,n}\n\t\t\t&\n\t\t\t\\Delta_{n,2n+1}\n\t\t\t&\n\t\t\t\\Delta_{n,*} \n\t\t\t\\\\\n\t\t\t\\Delta_{2n+1,n}\n\t\t\t&\n\t\t\t\\Delta_{2n+1,2n+1} \n\t\t\t&\n\t\t\t\\Delta_{2n+1,*} \n\t\t\t\\\\\n\t\t\t\\Delta_{*,n}\n\t\t\t&\n\t\t\t\\Delta_{*,2n+1} \n\t\t\t&\n\t\t\t\\Delta_{*,*}\n\t\t\\end{bmatrix}\n\t\t:=\n\t\t\\begin{bmatrix}\n\t\t\t0 \n\t\t\t&\n\t\t\t\\rho\n\t\t\t&\n\t\t\t1 - \\rho^k\n\t\t\t\\\\\n\t\t\t\\rho^k\n\t\t\t&\n\t\t\t0 \n\t\t\t&\n\t\t\t2\\rho - \\sqrt{2} \\rho^{k+1}\n\t\t\t\\\\\n\t\t\t1 + \\rho^{k-1} - \\frac{1}{2r_k}\n\t\t\t&\n\t\t\t\\frac{1}{2r_{k+1}} - \\frac{1+2\\rho}{2r_k}\n\t\t\t&\n\t\t\t0\n\t\t\\end{bmatrix}\n\t\t\\label{eq:Del}\n\t\\end{align}\n","start":26501,"end":27148}]},{"id":"S033","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"主定理作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm:main}]\n\tIt suffices to prove~\\eqref{eq:cert}; we prove this by induction. The base case $n=1$ is Example~\\ref{ex:n=1}. The inductive step is Theorem~\\ref{thm:cert}. \n\\end{proof}","start":27439,"end":27656}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"主证明归纳基础","source_units":[{"raw_tex":"\tIt suffices to prove~\\eqref{eq:cert}; we prove this by induction. The base case $n=1$ is Example~\\ref{ex:n=1}. The inductive step is Theorem~\\ref{thm:cert}. \n","start":27486,"end":27645}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"主证明递归推广","source_units":[{"raw_tex":"\tIt suffices to prove~\\eqref{eq:cert}; we prove this by induction. The base case $n=1$ is Example~\\ref{ex:n=1}. The inductive step is Theorem~\\ref{thm:cert}. \n","start":27486,"end":27645}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"主证明归约到证书","source_units":[{"raw_tex":"\tIt suffices to prove~\\eqref{eq:cert}; we prove this by induction. The base case $n=1$ is Example~\\ref{ex:n=1}. The inductive step is Theorem~\\ref{thm:cert}. \n","start":27486,"end":27645}]},{"id":"S037","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"最优点平移归一化","source_units":[{"raw_tex":"The rest of the section is dedicated to proving Theorem~\\ref{thm:cert}. We first isolate two helper lemmas in \\S\\ref{ssec:gluing:helper}, and then we combine them to prove the result in \\S\\ref{ssec:gluing:pf}.  For notational simplicity, henceforth we assume $x^* = 0$; this is without loss of generality after translating. \n\n","start":27659,"end":27985}]},{"id":"S038","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"辅助引理路线说明","source_units":[{"raw_tex":"\\subsection{Helper lemmas}\\label{ssec:gluing:helper}\n\nHere we provide three helper lemmas for the proof of Theorem~\\ref{thm:cert}. The first lemma explicitly computes all multipliers involving $x^*$ for the $n$-step certificate. \n","start":27985,"end":28215}]},{"id":"S039","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"证书线性二次分解","source_units":[{"raw_tex":"The next two lemmas are more substantial. These help us verify~\\eqref{eq:cert:2n+1}---which consists of a linear form in all $2n+3$ function values and a quadratic form in all $2n+3$ gradients and iterates. Na\\\"ively verifying such an identity requires checking $\\Theta(n)$ coefficients for the linear form and $\\Theta(n^2)$ coefficients for the quadratic form. The following two lemmas show that due to the recursive construction of the Silver Stepsize Schedule and the gluing, these forms only affect the indices $n,2n+1,*$. This reduces verifying the rate certificate~\\eqref{eq:cert:2n+1} to checking only $\\Theta(1)$ coefficients, as detailed below in \\S\\ref{ssec:gluing:pf}. Below, for shorthand, let $F_{ij} := 2(f_i - f_j)$ and $P_{ij} := 2\\langle g_j, x_j - x_i \\rangle - \\|g_i - g_j\\|^2$  denote the linear and quadratic components of $Q_{ij}$, respectively. For bookkeeping purposes, we use $3$-dimensional vectors and $4 \\times 4$ matrices to denote the coefficients of these forms. \n","start":28912,"end":29907}]},{"id":"S040","semantic_type":"lemma","kind":"lemma","labels":["lem:opt"],"declared_external":false,"source_only":false,"title":"最优点相关乘子","source_units":[{"raw_tex":"\\begin{lemma}[Multipliers involving $x^*$]\\label{lem:opt} \n\t$\\sig_{n*} = \\rho^k - 1$, $\\sig_{*n} = \\frac{1}{2r_k}$, and $\\sig_{*t} = \\alpha_t$ for $t \\in \\{0,\\dots,n-1\\}$. \n\\end{lemma}","start":28216,"end":28400}]},{"id":"S041","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"最优点乘子作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\tExpand both sides of the identity~\\eqref{eq:cert:n} using the $*$-sparsity assumption, the definition of the co-coercivities, and the definition of GD, i.e., $x_t = x_0 - \\sum_{s=0}^{t-1} \\alpha_s g_s$. Matching coefficients for the $\\langle x_0, g_t \\rangle$ terms yields the claimed formulas for $\\sig_{*t}$. Matching coefficients for the $f^*$ term gives the identity $\\sig_{n*} = \\sum_{j=0}^n \\sig_{*j}-  \\tfrac{1}{2r_k}$. This equals $\\rho^k - 1$ by Lemma~\\ref{lem:sum-steps}. \n\\end{proof}","start":28401,"end":28910}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"比较初始点内积系数","source_units":[{"raw_tex":"\tExpand both sides of the identity~\\eqref{eq:cert:n} using the $*$-sparsity assumption, the definition of the co-coercivities, and the definition of GD, i.e., $x_t = x_0 - \\sum_{s=0}^{t-1} \\alpha_s g_s$. Matching coefficients for the $\\langle x_0, g_t \\rangle$ terms yields the claimed formulas for $\\sig_{*t}$. Matching coefficients for the $f^*$ term gives the identity $\\sig_{n*} = \\sum_{j=0}^n \\sig_{*j}-  \\tfrac{1}{2r_k}$. This equals $\\rho^k - 1$ by Lemma~\\ref{lem:sum-steps}. \n","start":28415,"end":28899}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"比较最优值系数","source_units":[{"raw_tex":"\tExpand both sides of the identity~\\eqref{eq:cert:n} using the $*$-sparsity assumption, the definition of the co-coercivities, and the definition of GD, i.e., $x_t = x_0 - \\sum_{s=0}^{t-1} \\alpha_s g_s$. Matching coefficients for the $\\langle x_0, g_t \\rangle$ terms yields the claimed formulas for $\\sig_{*t}$. Matching coefficients for the $f^*$ term gives the identity $\\sig_{n*} = \\sum_{j=0}^n \\sig_{*j}-  \\tfrac{1}{2r_k}$. This equals $\\rho^k - 1$ by Lemma~\\ref{lem:sum-steps}. \n","start":28415,"end":28899}]},{"id":"S044","semantic_type":"lemma","kind":"lemma","labels":["lem:succinct-linear"],"declared_external":false,"source_only":false,"title":"简洁线性形式","source_units":[{"raw_tex":"\\begin{lemma}[Succinct linear forms]\\label{lem:succinct-linear}\n\tLet $u := [f_n, f_{2n+1}, f^*]^T$, and let $e, s, \\ell \\in \\R^3$ be the vectors defined in Appendix~\\ref{app:succinct-linear}. \n\t\\begin{itemize}\n\t\t\\item \\underline{Gluing error.} $\\frac{f^* - f_{2n+1}}{r_{k+1}} - \\sum_{ij} \\Theta_{ij} F_{ij} = \\langle e, u \\rangle$\n\t\t\\item \\underline{Sparse correction.} $\\sum_{ij} \\Delta_{ij} F_{ij} = \\langle s, u \\rangle$\n\t\t\\item \\underline{Rank-one correction.} $\\sum_{ij} \\Xi_{ij} F_{ij} = \\langle \\ell, u \\rangle$\n\t\\end{itemize}\n\\end{lemma}","start":29910,"end":30455}]},{"id":"S045","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"线性形式作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\tExpand the definition of co-coercivities, simplify the rank-one correction using Lemma~\\ref{lem:sum-steps}, and simplify the sparse correction using the Pell recurrence $\\rho^{k+1} = 2\\rho^k + \\rho^{k-1}$.\n\\end{proof}","start":30456,"end":30688}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"展开线性系数","source_units":[{"raw_tex":"\tExpand the definition of co-coercivities, simplify the rank-one correction using Lemma~\\ref{lem:sum-steps}, and simplify the sparse correction using the Pell recurrence $\\rho^{k+1} = 2\\rho^k + \\rho^{k-1}$.\n","start":30470,"end":30677}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"秩一线性项求和","source_units":[{"raw_tex":"\tExpand the definition of co-coercivities, simplify the rank-one correction using Lemma~\\ref{lem:sum-steps}, and simplify the sparse correction using the Pell recurrence $\\rho^{k+1} = 2\\rho^k + \\rho^{k-1}$.\n","start":30470,"end":30677}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"稀疏线性项化简","source_units":[{"raw_tex":"\tExpand the definition of co-coercivities, simplify the rank-one correction using Lemma~\\ref{lem:sum-steps}, and simplify the sparse correction using the Pell recurrence $\\rho^{k+1} = 2\\rho^k + \\rho^{k-1}$.\n","start":30470,"end":30677}]},{"id":"S049","semantic_type":"lemma","kind":"lemma","labels":["lem:succinct"],"declared_external":false,"source_only":false,"title":"简洁二次形式","source_units":[{"raw_tex":"\\begin{lemma}[Succinct quadratic forms]\\label{lem:succinct} Let $v := [x_n,g_n,x_{2n+1},g_{2n+1}]^T$, and let $E$, $S$, $L$ be the $4 \\times 4$ matrices defined in Appendix~\\ref{app:succinct}.\n\t\\begin{itemize}\n\t\t\\item \\underline{Gluing error.} \n\t\t$\\|x_0\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1}\\|^2\n\t\t- \\sum_{ij} \\Theta_{ij} P_{ij} = \\langle E, vv^T \\rangle$\n\t\t\\item \\underline{Sparse correction.} $\\sum_{ij} \\Delta_{ij} P_{ij} = \\langle S, vv^T \\rangle$\n\t\t\\item \\underline{Rank-one correction.} $\\sum_{ij} \\Xi_{ij} P_{ij} = \\langle L, vv^T \\rangle$\n\t\\end{itemize}\n\\end{lemma}","start":30690,"end":31261}]},{"id":"S050","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"二次形式完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\tDeferred to Appendix~\\ref{app:succinct} for brevity.\n\\end{proof}","start":31262,"end":31341},{"raw_tex":"\\begin{proof}[Proof of Lemma~\\ref{lem:succinct}]\n\t\\underline{Gluing component.} \n\tRecall that $h_{2n+1} = [h_n, \\alpha_n, h_n]$ by the recursive construction~\\eqref{eq:steps-recursive} of the Silver Stepsize Schedule. Thus, since $\\sig$ certifies the $n$-step rate, we have\n\t\\begin{align*}\n\t\t\\sum_{i \\neq j \\in \\{0, \\dots, n,*\\}} \\sig_{ij} P_{ij} \n\t\t&=\n\t\t\\|x_0\\|^2 - \\|x_n - c_k g_n\\|^2\n\t\t\\\\ \n\t\t\\sum_{i \\neq j \\in \\{n+1, \\dots, 2n,*\\}} \\sig_{ij} P_{ij} \n\t\t&=\n\t\t\\|x_{n+1}\\|^2 - \\|x_{2n+1} - c_k g_{2n+1}\\|^2\n\t\\end{align*}\n\tBy definition, $\\sum_{ij} \\Theta_{ij} P_{ij}$ is the former plus $(1 + 2\\rho)$ times the latter. \n\tThus the gluing error $\\|x_0\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1}\\|^2 - \\sum_{ij} \\Theta_{ij} P_{ij}$ is equal to\n\t\\begin{align*}\n\t\t(1+2\\rho) \\|x_{2n+1} - c_k g_{2n+1}\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1}\\|^2 + \\|x_n - c_k g_n\\|^2 - (1 + 2\\rho) \\|x_{n+1}\\|^2\n\t\\end{align*}\n\tThe key point is that after \n\texpanding $x_{n+1} = x_n - \\alpha_n g_n$, this is a quadratic form in only the $4$ variables $x_n, g_n, x_{2n+1},g_{2n+1}$. Collecting terms, substituting $c_k = \\tfrac{1}{2r_k}$, and using the definition of the Silver Stepsize $\\alpha_n = 1 + \\rho^{k-1}$ yields $\\langle E, vv^T \\rangle$, as desired.\n\t\n\t\\par \\underline{Sparse component.} Expand the definition of the co-coercivity and collect terms. Note also that this equals the matrix $S$ in Part~I~\\citep{alt23hedging1}, in the limit that the strong convexity tends to $0$.\n\t\n\t\\par \\underline{Low-rank component.}  Expanding the co-coercivities, using the identity $x_{2n+1} - x_{n+1}= - \\sum_{t=n+1}^{2n} \\alpha_t g_t$ by definition of GD, and using the identity $\\sum_{t=n+1}^{2n} \\alpha_t = \\sum_{t=0}^{n-1} \\alpha_t = \\rho^k - 1$ from Lemma~\\ref{lem:sum-steps}, \n\t\\begin{align*}\n\t\t\\sum_{t=n+1}^{2n} \\Big( \\Xi_{n,t} P_{n,t} - \\Xi_{*,t} P_{*,t} \\Big)\n\t\t=\n\t\t\\rho \\sum_{t=n+1}^{2n} \\alpha_t \\Big(\n\t\t2 \\langle g_t, g_n - x_n \\rangle - \\|g_n\\|^2\n\t\t\\Big)\n\t\t=\n\t\t2\\rho \\langle x_{2n+1} - x_{n+1}, x_n - g_n \\rangle - \\rho(\\rho^k - 1 ) \\|g_n\\|^2 \\,.\n\t\\end{align*}\n\tAn identical calculation yields\n\t\\begin{align*}\n\t\t\\sum_{t=n+1}^{2n} \\Big( \\Xi_{2n+1,t} P_{2n+1,t} - \\Xi_{*,t} P_{*,t} \\Big)\n\t\t&=\n\t\t2\\rho \\langle x_{2n+1} - x_{n+1}, x_{2n+1} - g_{2n+1} \\rangle - \\rho(\\rho^k - 1 ) \\|g_{2n+1}\\|^2 \\,.\n\t\\end{align*}\n\tNow sum these two displays, use the definition of GD to expand $x_{n+1} = x_n - \\alpha_n g_n$, and use the definition of the Silver Stepsize $\\alpha_n = 1 + \\rho^{k-1}$. This gives $\\langle L, vv^T \\rangle$, as desired.\n\\end{proof}","start":35071,"end":37587}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"二次证明延期指引","source_units":[{"raw_tex":"\\begin{proof}\n\tDeferred to Appendix~\\ref{app:succinct} for brevity.\n\\end{proof}","start":31262,"end":31341}]},{"id":"S052","semantic_type":"proof","kind":"proof","labels":["ssec:gluing:pf"],"declared_external":false,"source_only":false,"title":"递归拼接作者证明","source_units":[{"raw_tex":"\\subsection{Proof of recursive gluing (Theorem~\\ref{thm:cert})}\\label{ssec:gluing:pf}\n\n\\paragraph*{Non-negativity.} We verify $\\lam_{ij} \\geq 0$ for all $i \\neq j \\in \\{0, \\dots, 2n+1,*\\}$. For nearly all entries, this is obvious since $\\lam_{ij}$ is constructed by adding and multiplying non-negative numbers. For the remaining entries where either $\\Xi$ or $\\Delta$ is negative, compute the corresponding entry of $\\lam$ by summing the corrections and using Lemma~\\ref{lem:opt}. This gives $\\lam_{n*} = 0$, $\\lam_{2n+1,*} = \\rho^{k+1} - 1$, $\\lam_{*,n} = \\rho^{k-1} + 1$, $\\lam_{*,2n+1} = \\tfrac{1}{2r_{k+1}}$, and $\\lam_{*,t} = \\alpha_t$ for all $t \\in \\{n+1,\\dots,2n\\}$. All these entries are clearly non-negative.\n\n\\paragraph*{Rate certificate.} The identity~\\eqref{eq:cert:2n+1} has two components: a linear form in the function values and a quadratic form in the iterates and gradients. For the linear form, it suffices to verify $e-s-\\ell = 0$ by Lemma~\\ref{lem:succinct-linear}, where $e,s,\\ell \\in \\R^3$ are the vectors defined in Appendix~\\ref{app:succinct-linear}. This is obvious by inspection. For the quadratic form, it suffices to verify $E - S - L = 0$, where $E,S,L \\in \\R^{4 \\times 4}$ are the matrices in Lemma~\\ref{lem:succinct} defined in Appendix~\\ref{app:succinct}. By plugging in the explicit values for $r_k$ and $\\Delta$, this is straightforward to check by hand. For brevity, we provide a simple Mathematica script that verifies these identities at the URL~\\citep{MathematicaURL}.\n \n\n\n\t \n\t \n\t \\appendix\n\t \n","start":31347,"end":32881}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":["ssec:gluing:pf"],"declared_external":false,"source_only":false,"title":"乘子非负与零结构","source_units":[{"raw_tex":"\\paragraph*{Non-negativity.} We verify $\\lam_{ij} \\geq 0$ for all $i \\neq j \\in \\{0, \\dots, 2n+1,*\\}$. For nearly all entries, this is obvious since $\\lam_{ij}$ is constructed by adding and multiplying non-negative numbers. For the remaining entries where either $\\Xi$ or $\\Delta$ is negative, compute the corresponding entry of $\\lam$ by summing the corrections and using Lemma~\\ref{lem:opt}. This gives $\\lam_{n*} = 0$, $\\lam_{2n+1,*} = \\rho^{k+1} - 1$, $\\lam_{*,n} = \\rho^{k-1} + 1$, $\\lam_{*,2n+1} = \\tfrac{1}{2r_{k+1}}$, and $\\lam_{*,t} = \\alpha_t$ for all $t \\in \\{n+1,\\dots,2n\\}$. All these entries are clearly non-negative.\n\n","start":31434,"end":32067}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":["ssec:gluing:pf"],"declared_external":false,"source_only":false,"title":"线性证书消去","source_units":[{"raw_tex":"\\paragraph*{Rate certificate.} The identity~\\eqref{eq:cert:2n+1} has two components: a linear form in the function values and a quadratic form in the iterates and gradients. For the linear form, it suffices to verify $e-s-\\ell = 0$ by Lemma~\\ref{lem:succinct-linear}, where $e,s,\\ell \\in \\R^3$ are the vectors defined in Appendix~\\ref{app:succinct-linear}. This is obvious by inspection. For the quadratic form, it suffices to verify $E - S - L = 0$, where $E,S,L \\in \\R^{4 \\times 4}$ are the matrices in Lemma~\\ref{lem:succinct} defined in Appendix~\\ref{app:succinct}. By plugging in the explicit values for $r_k$ and $\\Delta$, this is straightforward to check by hand. For brevity, we provide a simple Mathematica script that verifies these identities at the URL~\\citep{MathematicaURL}.\n","start":32067,"end":32856}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["ssec:gluing:pf"],"declared_external":false,"source_only":false,"title":"二次证书消去","source_units":[{"raw_tex":"\\paragraph*{Rate certificate.} The identity~\\eqref{eq:cert:2n+1} has two components: a linear form in the function values and a quadratic form in the iterates and gradients. For the linear form, it suffices to verify $e-s-\\ell = 0$ by Lemma~\\ref{lem:succinct-linear}, where $e,s,\\ell \\in \\R^3$ are the vectors defined in Appendix~\\ref{app:succinct-linear}. This is obvious by inspection. For the quadratic form, it suffices to verify $E - S - L = 0$, where $E,S,L \\in \\R^{4 \\times 4}$ are the matrices in Lemma~\\ref{lem:succinct} defined in Appendix~\\ref{app:succinct}. By plugging in the explicit values for $r_k$ and $\\Delta$, this is straightforward to check by hand. For brevity, we provide a simple Mathematica script that verifies these identities at the URL~\\citep{MathematicaURL}.\n","start":32067,"end":32856}]},{"id":"S056","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"对称矩阵省略约定","source_units":[{"raw_tex":"The matrices in Lemma~\\ref{lem:succinct} are defined as follows. For brevity, we write $\\sim$ for the entries below the diagonal since these coefficient matrices are symmetric.\n\\begin{align*}\n\t\\small\n","start":33694,"end":33894}]},{"id":"S057","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"拼接误差矩阵","source_units":[{"raw_tex":"\tE :=\n\t\\begin{bmatrix}\n\t\t-2\\rho\n\t\t&\n\t\t(1 + 2\\rho) (1 + \\rho^{k-1}) -\\frac{1}{2r_k} \n\t\t&\n\t\t0\n\t\t&\n\t\t0\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\frac{1}{4r_k^2} - (1 + 2\\rho) (1+\\rho^{k-1})^2\n\t\t&\n\t\t0\n\t\t&\n\t\t0 \n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t2\\rho \n\t\t&\n\t\t\\frac{1}{2r_{k+1}} - \\frac{1+2\\rho}{2r_k}\n\t\t\\\\\n\t\t\\sim & \\sim & \\sim &\n\t\t\\frac{1+2\\rho}{4r_k^2} - \\frac{1}{4r_{k+1}^2}\n\t\\end{bmatrix}\n\\end{align*}\n","start":33894,"end":34265}]},{"id":"S058","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"稀疏修正矩阵","source_units":[{"raw_tex":"\\begin{align*}\n\t\\small\n\tS := \n\t\\begin{bmatrix}\n\t\t0\n\t\t&\n\t\t\\Delta_{2n+1,n} + \\Delta_{*,n}\n\t\t&\n\t\t0\n\t\t&\n\t\t-\\Delta_{n,2n+1}\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t-\\left( \\Delta_{n,*} + \\Delta_{*,n} + \\Delta_{2n+1,n} + \\Delta_{n,2n+1} \\right)\n\t\t&\n\t\t-\\Delta_{2n+1,n}\n\t\t&\n\t\t\\Delta_{n,2n+1} + \\Delta_{2n+1,n}\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t0\n\t\t&\n\t\t\\Delta_{n,2n+1} + \\Delta_{*,2n+1}\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t- \\left( \\Delta_{n,2n+1} + \\Delta_{2n+1,n}  + \\Delta_{2n+1,*} + \\Delta_{*,2n+1}\\right)\n\t\\end{bmatrix}\n\\end{align*}\n","start":34265,"end":34771}]},{"id":"S059","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"秩一修正矩阵","source_units":[{"raw_tex":"\\begin{align*}\n\t\\small\n\tL :=  \\rho \\begin{bmatrix}\n\t\t-2\n\t\t&\n\t\t2 + \\rho^{k-1} \n\t\t&\n\t\t0\n\t\t&\n\t\t1\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t-1 - \\rho^k - 2 \\rho^{k-1} \n\t\t&\n\t\t\\rho^{k-1}\n\t\t&\n\t\t- 1 - \\rho^{k-1}\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t2\n\t\t&\n\t\t-1\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t1 - \\rho^k\n\t\\end{bmatrix}\n\\end{align*}\n","start":34771,"end":35070}]},{"id":"S060","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"线性误差向量","source_units":[{"raw_tex":"The vectors $e, s, \\ell$ in Lemma~\\ref{lem:succinct-linear} are defined as follows:\n\\begin{align*}\n\t\\small\n\te := \\begin{bmatrix} \n\t\t\\frac{1}{r_k} \\\\\n\t\t-\\frac{1}{r_{k+1}} + \\frac{1+2\\rho}{r_k} \\\\\n\t\t\\frac{1}{r_{k+1}} - \\frac{4\\rho}{r_k}\n\t\\end{bmatrix}\n\t\\,,\n\t\\qquad\n","start":33033,"end":33296}]},{"id":"S061","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"原文向量 s","source_units":[{"raw_tex":"\ts := 2\\rho(\\rho^{k} - 1) \\begin{bmatrix}\n\t\t1 \\\\\n\t\t1 \\\\\n\t\t-2\n\t\\end{bmatrix}\n\t\\,,\n\t\\qquad\n","start":33296,"end":33385}]},{"id":"S062","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"原文向量 ell","source_units":[{"raw_tex":"\t\\ell := \\begin{bmatrix}\n\t\t\\frac{1}{r_k} - 2\\rho(\\rho^k - 1)\\\\\n\t\t-\\frac{1}{r_{k+1}} + \\frac{1+2\\rho}{r_k} - 2\\rho(\\rho^k - 1) \\\\\n\t\t\\frac{1}{r_{k+1}} - \\frac{4\\rho}{r_k} + 4\\rho(\\rho^k - 1)\n\t\\end{bmatrix}\\,.\n\\end{align*}\n","start":33385,"end":33605}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"复用两段输入证书","source_units":[{"raw_tex":"\t\\underline{Gluing component.} \n\tRecall that $h_{2n+1} = [h_n, \\alpha_n, h_n]$ by the recursive construction~\\eqref{eq:steps-recursive} of the Silver Stepsize Schedule. Thus, since $\\sig$ certifies the $n$-step rate, we have\n\t\\begin{align*}\n\t\t\\sum_{i \\neq j \\in \\{0, \\dots, n,*\\}} \\sig_{ij} P_{ij} \n\t\t&=\n\t\t\\|x_0\\|^2 - \\|x_n - c_k g_n\\|^2\n\t\t\\\\ \n\t\t\\sum_{i \\neq j \\in \\{n+1, \\dots, 2n,*\\}} \\sig_{ij} P_{ij} \n\t\t&=\n\t\t\\|x_{n+1}\\|^2 - \\|x_{2n+1} - c_k g_{2n+1}\\|^2\n\t\\end{align*}\n","start":35120,"end":35592}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"拼接误差压缩为 E","source_units":[{"raw_tex":"\tBy definition, $\\sum_{ij} \\Theta_{ij} P_{ij}$ is the former plus $(1 + 2\\rho)$ times the latter. \n\tThus the gluing error $\\|x_0\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1}\\|^2 - \\sum_{ij} \\Theta_{ij} P_{ij}$ is equal to\n\t\\begin{align*}\n\t\t(1+2\\rho) \\|x_{2n+1} - c_k g_{2n+1}\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1}\\|^2 + \\|x_n - c_k g_n\\|^2 - (1 + 2\\rho) \\|x_{n+1}\\|^2\n\t\\end{align*}\n\tThe key point is that after \n\texpanding $x_{n+1} = x_n - \\alpha_n g_n$, this is a quadratic form in only the $4$ variables $x_n, g_n, x_{2n+1},g_{2n+1}$. Collecting terms, substituting $c_k = \\tfrac{1}{2r_k}$, and using the definition of the Silver Stepsize $\\alpha_n = 1 + \\rho^{k-1}$ yields $\\langle E, vv^T \\rangle$, as desired.\n","start":35592,"end":36296}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"稀疏二次项压缩为 S","source_units":[{"raw_tex":"\t\n\t\\par \\underline{Sparse component.} Expand the definition of the co-coercivity and collect terms. Note also that this equals the matrix $S$ in Part~I~\\citep{alt23hedging1}, in the limit that the strong convexity tends to $0$.\n\t\n","start":36296,"end":36526}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"低秩首组求和","source_units":[{"raw_tex":"\t\\par \\underline{Low-rank component.}  Expanding the co-coercivities, using the identity $x_{2n+1} - x_{n+1}= - \\sum_{t=n+1}^{2n} \\alpha_t g_t$ by definition of GD, and using the identity $\\sum_{t=n+1}^{2n} \\alpha_t = \\sum_{t=0}^{n-1} \\alpha_t = \\rho^k - 1$ from Lemma~\\ref{lem:sum-steps}, \n\t\\begin{align*}\n\t\t\\sum_{t=n+1}^{2n} \\Big( \\Xi_{n,t} P_{n,t} - \\Xi_{*,t} P_{*,t} \\Big)\n\t\t=\n\t\t\\rho \\sum_{t=n+1}^{2n} \\alpha_t \\Big(\n\t\t2 \\langle g_t, g_n - x_n \\rangle - \\|g_n\\|^2\n\t\t\\Big)\n\t\t=\n\t\t2\\rho \\langle x_{2n+1} - x_{n+1}, x_n - g_n \\rangle - \\rho(\\rho^k - 1 ) \\|g_n\\|^2 \\,.\n\t\\end{align*}\n","start":36526,"end":37108}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"低秩次组求和","source_units":[{"raw_tex":"\tAn identical calculation yields\n\t\\begin{align*}\n\t\t\\sum_{t=n+1}^{2n} \\Big( \\Xi_{2n+1,t} P_{2n+1,t} - \\Xi_{*,t} P_{*,t} \\Big)\n\t\t&=\n\t\t2\\rho \\langle x_{2n+1} - x_{n+1}, x_{2n+1} - g_{2n+1} \\rangle - \\rho(\\rho^k - 1 ) \\|g_{2n+1}\\|^2 \\,.\n\t\\end{align*}\n","start":37108,"end":37355}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"低秩合并得到 L","source_units":[{"raw_tex":"\tNow sum these two displays, use the definition of GD to expand $x_{n+1} = x_n - \\alpha_n g_n$, and use the definition of the Silver Stepsize $\\alpha_n = 1 + \\rho^{k-1}$. This gives $\\langle L, vv^T \\rangle$, as desired.\n\\end{proof}","start":37355,"end":37587}]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisite 列表第二项：Standard first-order optimality at an unconstrained differentiable minimizer。"],"source_only":false,"title":"最小点一阶最优性","source_units":[]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisite 列表第三项：Elementary coefficient matching and algebraic identities for the silver ratio。"],"source_only":false,"title":"系数匹配与银比代数","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S021","edges":[{"source_id":"S001","target_id":"S021","kind":"supports"},{"source_id":"S002","target_id":"S021","kind":"supports"},{"source_id":"S004","target_id":"S021","kind":"supports"},{"source_id":"S009","target_id":"S021","kind":"supports"},{"source_id":"S012","target_id":"S021","kind":"supports"},{"source_id":"S007","target_id":"S021","kind":"supports"},{"source_id":"S005","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S009","target_id":"S008","kind":"supports"},{"source_id":"S005","target_id":"S008","kind":"supports"},{"source_id":"S001","target_id":"S010","kind":"supports"},{"source_id":"S002","target_id":"S010","kind":"supports"},{"source_id":"S001","target_id":"S014","kind":"supports"},{"source_id":"S010","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S007","target_id":"S016","kind":"supports"},{"source_id":"S013","target_id":"S015","kind":"supports"},{"source_id":"S004","target_id":"S015","kind":"supports"},{"source_id":"S007","target_id":"S015","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S014","target_id":"S017","kind":"supports"},{"source_id":"S007","target_id":"S017","kind":"supports"},{"source_id":"S015","target_id":"S018","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S005","target_id":"S024","kind":"supports"},{"source_id":"S005","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S009","target_id":"S027","kind":"supports"},{"source_id":"S020","target_id":"S027","kind":"supports"},{"source_id":"S015","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S032","target_id":"S029","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S029","target_id":"S026","kind":"supports"},{"source_id":"S018","target_id":"S034","kind":"supports"},{"source_id":"S026","target_id":"S035","kind":"supports"},{"source_id":"S009","target_id":"S035","kind":"supports"},{"source_id":"S017","target_id":"S036","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S035","target_id":"S033","kind":"supports"},{"source_id":"S036","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S021","kind":"supports"},{"source_id":"S010","target_id":"S021","kind":"supports"},{"source_id":"S008","target_id":"S021","kind":"supports"},{"source_id":"S027","target_id":"S042","kind":"supports"},{"source_id":"S020","target_id":"S042","kind":"supports"},{"source_id":"S013","target_id":"S042","kind":"supports"},{"source_id":"S002","target_id":"S042","kind":"supports"},{"source_id":"S037","target_id":"S042","kind":"supports"},{"source_id":"S069","target_id":"S042","kind":"supports"},{"source_id":"S070","target_id":"S042","kind":"supports"},{"source_id":"S027","target_id":"S043","kind":"supports"},{"source_id":"S020","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S022","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S043","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S013","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S044","kind":"supports"},{"source_id":"S030","target_id":"S044","kind":"supports"},{"source_id":"S031","target_id":"S044","kind":"supports"},{"source_id":"S032","target_id":"S044","kind":"supports"},{"source_id":"S060","target_id":"S044","kind":"supports"},{"source_id":"S061","target_id":"S044","kind":"supports"},{"source_id":"S062","target_id":"S044","kind":"supports"},{"source_id":"S039","target_id":"S046","kind":"supports"},{"source_id":"S030","target_id":"S046","kind":"supports"},{"source_id":"S027","target_id":"S046","kind":"supports"},{"source_id":"S031","target_id":"S047","kind":"supports"},{"source_id":"S039","target_id":"S047","kind":"supports"},{"source_id":"S022","target_id":"S047","kind":"supports"},{"source_id":"S005","target_id":"S047","kind":"supports"},{"source_id":"S032","target_id":"S048","kind":"supports"},{"source_id":"S039","target_id":"S048","kind":"supports"},{"source_id":"S070","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S048","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S039","target_id":"S049","kind":"supports"},{"source_id":"S037","target_id":"S049","kind":"supports"},{"source_id":"S056","target_id":"S049","kind":"supports"},{"source_id":"S057","target_id":"S049","kind":"supports"},{"source_id":"S058","target_id":"S049","kind":"supports"},{"source_id":"S059","target_id":"S049","kind":"supports"},{"source_id":"S005","target_id":"S063","kind":"supports"},{"source_id":"S027","target_id":"S063","kind":"supports"},{"source_id":"S039","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S030","target_id":"S064","kind":"supports"},{"source_id":"S002","target_id":"S064","kind":"supports"},{"source_id":"S005","target_id":"S064","kind":"supports"},{"source_id":"S016","target_id":"S064","kind":"supports"},{"source_id":"S057","target_id":"S064","kind":"supports"},{"source_id":"S032","target_id":"S065","kind":"supports"},{"source_id":"S039","target_id":"S065","kind":"supports"},{"source_id":"S058","target_id":"S065","kind":"supports"},{"source_id":"S031","target_id":"S066","kind":"supports"},{"source_id":"S039","target_id":"S066","kind":"supports"},{"source_id":"S002","target_id":"S066","kind":"supports"},{"source_id":"S022","target_id":"S066","kind":"supports"},{"source_id":"S005","target_id":"S066","kind":"supports"},{"source_id":"S031","target_id":"S067","kind":"supports"},{"source_id":"S039","target_id":"S067","kind":"supports"},{"source_id":"S002","target_id":"S067","kind":"supports"},{"source_id":"S022","target_id":"S067","kind":"supports"},{"source_id":"S005","target_id":"S067","kind":"supports"},{"source_id":"S037","target_id":"S065","kind":"supports"},{"source_id":"S069","target_id":"S065","kind":"supports"},{"source_id":"S037","target_id":"S066","kind":"supports"},{"source_id":"S069","target_id":"S066","kind":"supports"},{"source_id":"S037","target_id":"S067","kind":"supports"},{"source_id":"S069","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S002","target_id":"S068","kind":"supports"},{"source_id":"S005","target_id":"S068","kind":"supports"},{"source_id":"S059","target_id":"S068","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S063","target_id":"S050","kind":"supports"},{"source_id":"S064","target_id":"S050","kind":"supports"},{"source_id":"S065","target_id":"S050","kind":"supports"},{"source_id":"S066","target_id":"S050","kind":"supports"},{"source_id":"S067","target_id":"S050","kind":"supports"},{"source_id":"S068","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S029","target_id":"S053","kind":"supports"},{"source_id":"S030","target_id":"S053","kind":"supports"},{"source_id":"S031","target_id":"S053","kind":"supports"},{"source_id":"S032","target_id":"S053","kind":"supports"},{"source_id":"S040","target_id":"S053","kind":"supports"},{"source_id":"S020","target_id":"S053","kind":"supports"},{"source_id":"S027","target_id":"S053","kind":"supports"},{"source_id":"S044","target_id":"S054","kind":"supports"},{"source_id":"S060","target_id":"S054","kind":"supports"},{"source_id":"S061","target_id":"S054","kind":"supports"},{"source_id":"S062","target_id":"S054","kind":"supports"},{"source_id":"S049","target_id":"S055","kind":"supports"},{"source_id":"S057","target_id":"S055","kind":"supports"},{"source_id":"S058","target_id":"S055","kind":"supports"},{"source_id":"S059","target_id":"S055","kind":"supports"},{"source_id":"S007","target_id":"S055","kind":"supports"},{"source_id":"S032","target_id":"S055","kind":"supports"},{"source_id":"S039","target_id":"S052","kind":"supports"},{"source_id":"S029","target_id":"S052","kind":"supports"},{"source_id":"S028","target_id":"S052","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S055","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S026","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S056","target_id":"S058","kind":"supports"},{"source_id":"S032","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S059","kind":"supports"},{"source_id":"S005","target_id":"S059","kind":"supports"},{"source_id":"S007","target_id":"S060","kind":"supports"},{"source_id":"S005","target_id":"S061","kind":"supports"},{"source_id":"S009","target_id":"S061","kind":"supports"},{"source_id":"S007","target_id":"S062","kind":"supports"},{"source_id":"S005","target_id":"S062","kind":"supports"},{"source_id":"S009","target_id":"S062","kind":"supports"},{"source_id":"S070","target_id":"S054","kind":"supports"},{"source_id":"S070","target_id":"S055","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S021","edges":[{"source_id":"S001","target_id":"S021","kind":"supports"},{"source_id":"S002","target_id":"S021","kind":"supports"},{"source_id":"S004","target_id":"S021","kind":"supports"},{"source_id":"S009","target_id":"S021","kind":"supports"},{"source_id":"S012","target_id":"S021","kind":"supports"},{"source_id":"S007","target_id":"S021","kind":"supports"},{"source_id":"S005","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S009","target_id":"S008","kind":"supports"},{"source_id":"S005","target_id":"S008","kind":"supports"},{"source_id":"S001","target_id":"S010","kind":"supports"},{"source_id":"S002","target_id":"S010","kind":"supports"},{"source_id":"S001","target_id":"S014","kind":"supports"},{"source_id":"S010","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S007","target_id":"S016","kind":"supports"},{"source_id":"S013","target_id":"S015","kind":"supports"},{"source_id":"S004","target_id":"S015","kind":"supports"},{"source_id":"S007","target_id":"S015","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S014","target_id":"S017","kind":"supports"},{"source_id":"S007","target_id":"S017","kind":"supports"},{"source_id":"S015","target_id":"S018","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S005","target_id":"S022","kind":"supports"},{"source_id":"S009","target_id":"S027","kind":"supports"},{"source_id":"S020","target_id":"S027","kind":"supports"},{"source_id":"S015","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S032","target_id":"S029","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S029","target_id":"S026","kind":"supports"},{"source_id":"S018","target_id":"S034","kind":"supports"},{"source_id":"S026","target_id":"S035","kind":"supports"},{"source_id":"S009","target_id":"S035","kind":"supports"},{"source_id":"S017","target_id":"S036","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S035","target_id":"S033","kind":"supports"},{"source_id":"S036","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S021","kind":"supports"},{"source_id":"S010","target_id":"S021","kind":"supports"},{"source_id":"S008","target_id":"S021","kind":"supports"},{"source_id":"S027","target_id":"S040","kind":"supports"},{"source_id":"S020","target_id":"S040","kind":"supports"},{"source_id":"S013","target_id":"S040","kind":"supports"},{"source_id":"S002","target_id":"S040","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S069","target_id":"S040","kind":"supports"},{"source_id":"S070","target_id":"S040","kind":"supports"},{"source_id":"S022","target_id":"S040","kind":"supports"},{"source_id":"S013","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S044","kind":"supports"},{"source_id":"S030","target_id":"S044","kind":"supports"},{"source_id":"S031","target_id":"S044","kind":"supports"},{"source_id":"S032","target_id":"S044","kind":"supports"},{"source_id":"S060","target_id":"S044","kind":"supports"},{"source_id":"S061","target_id":"S044","kind":"supports"},{"source_id":"S062","target_id":"S044","kind":"supports"},{"source_id":"S027","target_id":"S044","kind":"supports"},{"source_id":"S022","target_id":"S044","kind":"supports"},{"source_id":"S005","target_id":"S044","kind":"supports"},{"source_id":"S070","target_id":"S044","kind":"supports"},{"source_id":"S039","target_id":"S049","kind":"supports"},{"source_id":"S037","target_id":"S049","kind":"supports"},{"source_id":"S056","target_id":"S049","kind":"supports"},{"source_id":"S057","target_id":"S049","kind":"supports"},{"source_id":"S058","target_id":"S049","kind":"supports"},{"source_id":"S059","target_id":"S049","kind":"supports"},{"source_id":"S005","target_id":"S049","kind":"supports"},{"source_id":"S027","target_id":"S049","kind":"supports"},{"source_id":"S030","target_id":"S049","kind":"supports"},{"source_id":"S002","target_id":"S049","kind":"supports"},{"source_id":"S016","target_id":"S049","kind":"supports"},{"source_id":"S032","target_id":"S049","kind":"supports"},{"source_id":"S031","target_id":"S049","kind":"supports"},{"source_id":"S022","target_id":"S049","kind":"supports"},{"source_id":"S069","target_id":"S049","kind":"supports"},{"source_id":"S030","target_id":"S026","kind":"supports"},{"source_id":"S031","target_id":"S026","kind":"supports"},{"source_id":"S032","target_id":"S026","kind":"supports"},{"source_id":"S040","target_id":"S026","kind":"supports"},{"source_id":"S020","target_id":"S026","kind":"supports"},{"source_id":"S044","target_id":"S026","kind":"supports"},{"source_id":"S060","target_id":"S026","kind":"supports"},{"source_id":"S061","target_id":"S026","kind":"supports"},{"source_id":"S062","target_id":"S026","kind":"supports"},{"source_id":"S049","target_id":"S026","kind":"supports"},{"source_id":"S057","target_id":"S026","kind":"supports"},{"source_id":"S058","target_id":"S026","kind":"supports"},{"source_id":"S059","target_id":"S026","kind":"supports"},{"source_id":"S007","target_id":"S026","kind":"supports"},{"source_id":"S039","target_id":"S026","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S056","target_id":"S058","kind":"supports"},{"source_id":"S032","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S059","kind":"supports"},{"source_id":"S005","target_id":"S059","kind":"supports"},{"source_id":"S007","target_id":"S060","kind":"supports"},{"source_id":"S005","target_id":"S061","kind":"supports"},{"source_id":"S009","target_id":"S061","kind":"supports"},{"source_id":"S007","target_id":"S062","kind":"supports"},{"source_id":"S005","target_id":"S062","kind":"supports"},{"source_id":"S009","target_id":"S062","kind":"supports"},{"source_id":"S070","target_id":"S026","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"光滑凸优化问题","source_units":[{"raw_tex":"We revisit the classical problem of smooth convex optimization: solve $\\min_{x \\in \\R^d} f(x)$ where $f$ is convex and $M$-smooth (i.e., its gradient is $M$-Lipschitz). A celebrated result is that with a prudent choice of stepsizes $\\{\\alpha_t\\}$, the gradient descent algorithm (GD) \n","start":6414,"end":6699}]},{"id":"S002","semantic_type":"algorithm","kind":"algorithm","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"梯度下降更新","source_units":[{"raw_tex":"\\begin{align}\n\tx_{t+1} = x_t - \\frac{\\alpha_t}{M} \\nabla f(x_t) \n\\end{align}\n","start":6699,"end":6776}]},{"id":"S003","semantic_type":"external_fact","kind":"external_fact","labels":["eq:rate-std"],"member_ids":["S003"],"declared_external":true,"declaration_text":["原文第217行列举的优化教材；这里只保留来源背景，不作为所选证明的输入。"],"source_only":false,"title":"常步长背景界","source_units":[{"raw_tex":"solves such a convex optimization problem to arbitrary accuracy from any initialization $x_0$. How quickly does GD converge? The mainstream approach (see e.g., the textbooks \\citep{BertsekasNonlinear,Nocedal,nesterov-survey,vishnoi2021algorithms,bubeck-book,BNO,lan2020first,bertsekas2015convex} among many others) is to use a constant stepsize schedule $\\alpha_t \\equiv \\bar{\\alpha} \\in (0,2)$ since this ensures\n\\begin{align}\n\tf(x_n) - f^* \\leq \\frac{cM\\|x_0 - x^*\\|^2}{n} \n\t\\label{eq:rate-std}\n\\end{align}\n","start":6776,"end":7285}]},{"id":"S004","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"最优点与最优值","source_units":[{"raw_tex":"where $x^*$ denotes any minimizer of $f$, $f^* := f(x^*)$ denotes the corresponding minimal value, and $c$ is a small constant, e.g., $c = \\tfrac{1}{4}$ for $\\bar{\\alpha} = 1$~\\citep{DT14}.\n","start":7285,"end":7475}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":["eq:steps-recursive"],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"银步长递归","source_units":[{"raw_tex":"\\par In this setting, the Silver Stepsize Schedule is particularly simple. For any integer $n = 2^{k}-1$, we recursively construct the schedule $h_{2n+1}$ of length $2n+1$ from the schedule $h_n$ of length $n$ via\n\\begin{align}\n\th_{2n+1} := [h_n, \\; 1 + \\rho^{k-1}, \\; h_n]\n\t\\label{eq:steps-recursive}\n\\end{align}\nwhere $\\rho := 1 + \\sqrt{2}$ denotes the silver ratio, and $h_1 := [\\sqrt{2}]$. This results in the simple pattern $[\\sqrt{2}, \\; 2, \\; \\sqrt{2}, \\; 1+\\sqrt{2}, \\dots]$ as depicted in Figure~\\ref{fig:silverstepsize}. This schedule is exactly the Silver Stepsize Schedule from~\\citep{alt23hedging1} in the limit that the strong convexity parameter vanishes (see Remark~\\ref{rem:steps-limit} for details), and bears similarities to those in~\\citep{gupta22,grimmer23,GrimmerShuWang}; see \\S\\ref{ssec:rel}.\n","start":9151,"end":9968}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":["eq:r-asymptotics"],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"速率参数","source_units":[{"raw_tex":"We show that these stepsizes yield an improved convergence rate~\\eqref{eq:rate-std} where $\\tfrac{c}{n}$ is replaced by\n\\begin{align}\n\tr_k := \\frac{1}{1 + \\sqrt{4\\rho^{2k} - 3}} \\leq\n\t\\frac{1}{2 \\rho^{\\log_2 n}}\n\t= \\frac{1}{2n^{\\log_2 \\rho}}\n\t\\approx \\frac{1}{2 n^{1.2716}}\n\t\\,.\n\t\\label{eq:r-asymptotics}\n\\end{align}\n","start":9970,"end":10287}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"速率渐近比较","source_units":[{"raw_tex":"\tr_k := \\frac{1}{1 + \\sqrt{4\\rho^{2k} - 3}} \\leq\n\t\\frac{1}{2 \\rho^{\\log_2 n}}\n\t= \\frac{1}{2n^{\\log_2 \\rho}}\n\t\\approx \\frac{1}{2 n^{1.2716}}\n\t\\,.\n\t\\label{eq:r-asymptotics}\n\\end{align}\nThis bound gives the correct asymptotic scaling of $r_k$ since the inequality is asymptotically tight.\n","start":10104,"end":10390}]},{"id":"S009","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"终止时刻索引","source_units":[{"raw_tex":"\\paragraph*{Indexing.} Throughout, the horizon is $n = 2^k-1$. This correspondence between $n \\in \\{1,3,7,15,\\dots\\}$ and $k \\in \\{1,2,3,4,\\dots\\}$ allows re-indexing in a way that simplifies notation for the recursion.\n\n\n","start":16612,"end":16834}]},{"id":"S010","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"单位光滑性归一化","source_units":[{"raw_tex":"\\paragraph*{Rescaling.} For simplicity, we normalize $M = 1$, i.e., $\\| \\nabla f(x) - \\nabla f(y) \\| \\leq \\|x-y\\|$ for all $x,y$. This is without loss of generality since if $h$ is $M$-smooth and convex,  then $f := h/M$ is $1$-smooth and convex, thus the result we establish $f_n - f^* \\leq r_k\\|x_0 - x^*\\|^2$ implies $h_n - h^* \\leq r_k M \\|x_0 - x^*\\|^2$. Note that running GD on $f$ simply amounts to rescaling the Silver Stepsize Schedules for $h$ by $1/M$.\n","start":16834,"end":17298}]},{"id":"S011","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"二进赋值记号","source_units":[{"raw_tex":" The Silver Stepsize Schedule is defined recursively in~\\eqref{eq:steps-recursive}. Here we mention an equivalent direct expression and make several remarks. Let $\\bin(t)$ denote the $2$-adic valuation of $t$, i.e., the smallest non-negative integer $i$ such that $2^i$ is in the binary expansion of $t$. For example, $\\bin(1) = 0$, $\\bin(2) = 1$, $\\bin(3) = 0$, $\\bin(4) = 2$, etc.\n\n\n","start":17362,"end":17747}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["eq:steps"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"显式银步长","source_units":[{"raw_tex":"\\begin{defin}[Silver Stepsize Schedule for smooth convex optimization]\n\tFor $t \\in \\{0,1, 2,\\dots\\}$, the $t$-th stepsize of the Silver Stepsize Schedule is\n\t\t\\begin{align}\n\t\t\\alpha_t\n\t\t:=\n\t\t1 + \\rho^{\\bin(t+1)-1}\\,.\n\t\t\\label{eq:steps}\n\t\\end{align}\n\\end{defin}\n","start":17747,"end":18008}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["eq:cocoercivity"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"余强制性量","source_units":[{"raw_tex":"\\par Briefly, the idea behind multi-step descent is that it is essential to capture how different iterations affect other iterations' progress. We do this by exploiting long-range consistency conditions between the iterates along GD's trajectory, as encoded by the co-coercivities \n\\begin{align}\n\tQ_{ij} := 2(f_i - f_j) + 2\\langle g_j, x_j - x_i \\rangle - \\|g_j - g_i\\|^2\\,.\n\t\\label{eq:cocoercivity}\n\\end{align}\n","start":20197,"end":20609}]},{"id":"S014","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S014"],"declared_external":true,"declaration_text":["原始 prerequisite 列表第一项；原文第384行 citep{pesto}。只需要其中余强制性不等式的必要性。"],"source_only":false,"title":"光滑凸插值必要性","source_units":[{"raw_tex":"The significance of these co-coercivities is that the constraints $\\{Q_{ij} \\geq 0\\}_{i \\neq j \\in \\{0,1,\\dots,n,*\\}}$ are necessary and sufficient for the existence of a $1$-smooth convex function $f$ satisfying $f_i = f(x_i)$ and $g_i = \\nabla f(x_i)$ for each $i \\in \\{0,1,\\dots,n,*\\}$~\\citep{pesto}. In other words, the co-coercivity conditions $\\{Q_{ij} \\geq 0 \\}_{i \\neq j \\in \\{0,1,\\dots,n,*\\}}$ generate all possible long-range consistency constraints on the objective function $f$. Or, said another way, the co-coercivity conditions generate all possible valid inequalities with which one can prove convergence rates for GD. For a further discussion, see~\\citep[\\S2.2]{alt23hedging1}.\n\n","start":20609,"end":21304}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":["eq:cert"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"多步速率证书","source_units":[{"raw_tex":"\\par Concretely, to prove Theorem~\\ref{thm:main}, we exhibit explicit non-negative multipliers $\\lam_{ij}$ satisfying\n\\begin{align}\n\t\\sum_{i \\neq j \\in \\{0, \\dots, n,*\\}} \\lam_{ij} Q_{ij}  = \n\t\\|x_0 - x^*\\|^2 - \\|x_n - c_k g_n - x^*\\|^2\n\t+ \\frac{f^* - f_n}{r_k}\\,,\n\t\\label{eq:cert}\n\\end{align}\n","start":21304,"end":21598}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"终端平移系数","source_units":[{"raw_tex":"where we use the shorthand $c_k := \\frac{1}{2r_k}$. Since $\\sum_{ij}\\lam_{ij} Q_{ij} \\geq 0$ for any $1$-smooth convex function, and since $\\|x_n - c_k g_n - x^*\\|^2 \\geq 0$ trivially as it is a square, this immediately implies the desired rate \n","start":21598,"end":21844}]},{"id":"S017","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:cert-cor"],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"证书推出误差界","source_units":[{"raw_tex":"where we use the shorthand $c_k := \\frac{1}{2r_k}$. Since $\\sum_{ij}\\lam_{ij} Q_{ij} \\geq 0$ for any $1$-smooth convex function, and since $\\|x_n - c_k g_n - x^*\\|^2 \\geq 0$ trivially as it is a square, this immediately implies the desired rate \n\\begin{align}\n\tf_n - f^* \\leq r_k \\|x_0 - x^*\\|^2\\,.\n\t\\label{eq:cert-cor}\n\\end{align}\n","start":21598,"end":21930}]},{"id":"S018","semantic_type":"proposition","kind":"proposition","labels":["ex:n=1"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"一步基础证书","source_units":[{"raw_tex":"\\begin{example}[$n=1$]\\label{ex:n=1}\n\tFor $n=1$, the identity~\\eqref{eq:cert} is\n\t\\begin{align*}\n\t\t\\sum_{i \\neq j \\in \\{0, 1, *\\}} \\lam_{ij} Q_{ij} \n\t\t=\n\t\t\\|x_0 - x^*\\|^2 - \\|x_1 - c_1 g_1 - x^*\\|^2 + \\frac{f^* - f_1}{r_1}\\,.\n\t\\end{align*}\n\tHere, $x_1 = x_0 - \\sqrt{2} g_0$, $r_1 \n\t= \\frac{1}{1+\\sqrt{4\\rho^2-3}}\n\t\\approx 0.1816$, $c_1 = \\frac{1}{2r_1} \\approx 2.7535$, and \n\t\\begin{align*}\n\t\t\\begin{bmatrix}\n\t\t\t\\lam_{00} & \\lam_{01} & \\lam_{0*} \\\\\n\t\t\t\\lam_{10} & \\lam_{11} & \\lam_{1*} \\\\\n\t\t\t\\lam_{*0} & \\lam_{*1} & \\lam_{**}\n\t\t\\end{bmatrix}\n\t\t=\n\t\t\\begin{bmatrix}\n\t\t\t0\n\t\t\t&\n\t\t\t1+\\sqrt{2}\n\t\t\t&\n\t\t\t0\n\t\t\t\\\\\n\t\t\t1\n\t\t\t&\n\t\t\t0\n\t\t\t&\n\t\t\t\\sqrt{2}\n\t\t\t\\\\\n\t\t\t\\sqrt{2}\n\t\t\t&\n\t\t\t\\frac{1}{2r_1}\n\t\t\t&\n\t\t\t0\n\t\t\\end{bmatrix}\n\t\t\\,.\n\t\\end{align*}\n\\end{example}","start":22940,"end":23676}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":["ex:n=1"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"基础乘子数据","source_units":[{"raw_tex":"\tHere, $x_1 = x_0 - \\sqrt{2} g_0$, $r_1 \n\t= \\frac{1}{1+\\sqrt{4\\rho^2-3}}\n\t\\approx 0.1816$, $c_1 = \\frac{1}{2r_1} \\approx 2.7535$, and \n\t\\begin{align*}\n\t\t\\begin{bmatrix}\n\t\t\t\\lam_{00} & \\lam_{01} & \\lam_{0*} \\\\\n\t\t\t\\lam_{10} & \\lam_{11} & \\lam_{1*} \\\\\n\t\t\t\\lam_{*0} & \\lam_{*1} & \\lam_{**}\n\t\t\\end{bmatrix}\n\t\t=\n\t\t\\begin{bmatrix}\n\t\t\t0\n\t\t\t&\n\t\t\t1+\\sqrt{2}\n\t\t\t&\n\t\t\t0\n\t\t\t\\\\\n\t\t\t1\n\t\t\t&\n\t\t\t0\n\t\t\t&\n\t\t\t\\sqrt{2}\n\t\t\t\\\\\n\t\t\t\\sqrt{2}\n\t\t\t&\n\t\t\t\\frac{1}{2r_1}\n\t\t\t&\n\t\t\t0\n\t\t\\end{bmatrix}\n\t\t\\,.\n\t\\end{align*}\n\\end{example}","start":23180,"end":23676}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"星号稀疏性","source_units":[{"raw_tex":"\\par For larger horizons $n$, we construct $\\lam_{ij}$ via the recursive gluing technique of~\\citep{alt23hedging1}. See Figure~\\ref{fig:gluing}. Below, we say that the multipliers $\\{\\sig_{ij}\\}_{i,j \\in \\{0,\\dots,n,*\\}}$ satisfy the $*$-sparsity property if $\\sig_{i*} = 0$ for all $i < n$. This is satisfied by construction and simplifies part of the proof (isolated in Lemma~\\ref{lem:opt}). \n","start":23680,"end":24075}]},{"id":"S021","semantic_type":"theorem","kind":"theorem","labels":["thm:main"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"银步长收敛主定理","source_units":[{"raw_tex":"\\begin{theorem}[Main result]\\label{thm:main}\n\tFor any horizon $n = 2^k - 1$, any dimension $d$, any $M$-smooth convex function $f : \\R^d \\to \\R$, and any initialization $x_0 \\in \\R^d$, \n\t\\begin{align}\n\t\tf(x_n) - f^* \\leq r_k\\, M\\|x_0 - x^*\\|^2 \\,,\n\t\\end{align}\n\twhere $x^*$ denotes any minimizer of $f$, and $x_n$ denotes the output of $n$ steps of GD using the Silver Stepsize Schedule. In particular, in order to achieve error $f(x_n) - f^* \\leq \\eps$, it suffices to run GD for\n\t\\begin{align*}\n\t\tn\n\t\t\\leq\n\t\t\\left( \\frac{M \\|x_0 - x^*\\|^2}{2 \\eps} \\right)^{\\log_{\\rho} 2} \n\t\t\\approx\n\t\t\\left( \\frac{M\\|x_0 - x^*\\|2}{2 \\eps} \\right)^{0.7864}\n\t\t\\;\\; \\text{iterations}.\n\t\\end{align*}\n\\end{theorem}","start":10393,"end":11088}]},{"id":"S022","semantic_type":"lemma","kind":"lemma","labels":["lem:sum-steps"],"member_ids":["S022","S023","S024","S025"],"declared_external":false,"source_only":false,"title":"银步长之和","source_units":[{"raw_tex":"\\begin{lemma}[Sum of Silver Stepsizes]\\label{lem:sum-steps}\n\t$\\sum_{t=0}^{n-1} \\alpha_t = \\rho^k - 1$ for any $k \\in \\N$. \n\\end{lemma}","start":19217,"end":19351},{"raw_tex":"\\begin{proof}\n\tThe base case $k=1$ is trivial. The inductive step follows from the recursion~\\eqref{eq:steps-recursive}.\n\\end{proof}","start":19352,"end":19484},{"raw_tex":"\tThe base case $k=1$ is trivial. The inductive step follows from the recursion~\\eqref{eq:steps-recursive}.\n","start":19366,"end":19473}]},{"id":"S026","semantic_type":"theorem","kind":"theorem","labels":["eq:Del","eq:Theta","eq:Xi","eq:cert:2n+1","eq:cert:n","eq:lambda","thm:cert"],"member_ids":["S026","S052","S053","S054","S055"],"declared_external":false,"source_only":false,"title":"递归拼接定理","source_units":[{"raw_tex":"\\begin{theorem}[Recursive gluing]\\label{thm:cert}\n\tLet $n = 2^k-1$. Suppose $\\{\\sig_{ij}\\}_{i,j \\in \\{0,\\dots,n,*\\}} $ satisfies $*$-sparsity and certifies the $n$-step rate, i.e.,\n\t\\begin{align}\n\t\t\\sum_{i \\neq j \\in \\{0, \\dots, n,*\\}} \\sig_{ij} Q_{ij} \n\t\t=\n\t\t\\|x_0 - x^*\\|^2 - \\|x_n - c_k g_n - x^*\\|^2  + \\frac{f^* - f_n}{r_k}\\ .\n\t\t\\label{eq:cert:n}\n\t\\end{align}\n\tThen there exists $\\{\\lam_{ij}\\}_{i, j \\in \\{0,\\dots,2n+1,*\\}}$ that satisfies $*$-sparsity and certifies the $2n+1$-step rate, i.e.,\n\t\\begin{align}\n\t\t\\sum_{i \\neq j \\in \\{0, \\dots, 2n+1,*\\}} \\lam_{ij} Q_{ij} \n\t\t=\n\t\t\\|x_0 - x^*\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1} - x^*\\|^2 + \\frac{f^* - f_{2n+1}}{r_{k+1}}\\ .\n\t\t\\label{eq:cert:2n+1}\n\t\\end{align}\n\tMoreover, this certificate is explicitly given by\n\t\\begin{align}\n\t\t\\lam_{ij} = \\underbrace{\\Theta_{ij}}_{\\text{gluing}}\n\t\t+\n\t\t\\underbrace{\\Xi_{ij}}_{\\text{rank-one correction}}\n\t\t+\n\t\t\\underbrace{\\Delta_{ij}}_{\\text{sparse correction}}\n\t\t\\label{eq:lambda}\n\t\\end{align}\n\tThe ``gluing component'' $\\Theta$ is defined as \n\t\\begin{align}\n\t\t\\Theta_{ij} := \\underbrace{\\sig_{i,j} \\cdot \\mathds{1}_{i,j \\in \\{0,\\dots,n,*\\}}}_{\\text{recurrence for first $n$ steps}}\n\t\t\\;\\;+\\;\\;\n\t\t\\underbrace{ (1 + 2\\rho)\\; \\sig_{i-n-1,j-n-1} \\cdot \\mathds{1}_{i,j \\in \\{n+1,\\dots,2n+1,*\\}}}_{\\text{recurrence for last $n$ steps}}\\,.\n\t\t\\label{eq:Theta}\n\t\\end{align}\n\tThe ``rank-one correction'' $\\Xi$ is zero except the entries $\\{\\Xi_{ij}\\}_{i \\in \\{n,2n+1,*\\},\\; j \\in \\{n+1, \\dots, 2n\\}}$ which are\n\t\\begin{align}\n\t\t\\begin{bmatrix}\n\t\t\t\\Xi_{n,n+1} & \\Xi_{n,n+2} & \\cdots & \\Xi_{n,2n} \\\\\n\t\t\t\\Xi_{2n+1,n+1} & \\Xi_{2n+1,n+2} & \\cdots & \\Xi_{2n+1,2n} \\\\\n\t\t\t\\Xi_{*,n+1} & \\Xi_{*,n+2} & \\cdots & \\Xi_{*,2n}\n\t\t\\end{bmatrix}\n\t\t:=\n\t\t\\rho \\begin{bmatrix}\n\t\t\t1 \\\\\n\t\t\t1 \\\\\n\t\t\t-2\n\t\t\\end{bmatrix} \\begin{bmatrix}\n\t\t\t\\alpha_{n+1} & \\alpha_{n+2} & \\cdots & \\alpha_{2n}\n\t\t\\end{bmatrix}\n\t\t\\label{eq:Xi}\n\t\\end{align}\n\tThe ``sparse correction'' $\\Delta$ is zero except the entries $\\{\\Delta_{ij}\\}_{i \\neq j \\in \\{n,2n+1,*\\}}$ which are\n\t\\begin{align}\n\t\t\\begin{bmatrix}\n\t\t\t\\Delta_{n,n}\n\t\t\t&\n\t\t\t\\Delta_{n,2n+1}\n\t\t\t&\n\t\t\t\\Delta_{n,*} \n\t\t\t\\\\\n\t\t\t\\Delta_{2n+1,n}\n\t\t\t&\n\t\t\t\\Delta_{2n+1,2n+1} \n\t\t\t&\n\t\t\t\\Delta_{2n+1,*} \n\t\t\t\\\\\n\t\t\t\\Delta_{*,n}\n\t\t\t&\n\t\t\t\\Delta_{*,2n+1} \n\t\t\t&\n\t\t\t\\Delta_{*,*}\n\t\t\\end{bmatrix}\n\t\t:=\n\t\t\\begin{bmatrix}\n\t\t\t0 \n\t\t\t&\n\t\t\t\\rho\n\t\t\t&\n\t\t\t1 - \\rho^k\n\t\t\t\\\\\n\t\t\t\\rho^k\n\t\t\t&\n\t\t\t0 \n\t\t\t&\n\t\t\t2\\rho - \\sqrt{2} \\rho^{k+1}\n\t\t\t\\\\\n\t\t\t1 + \\rho^{k-1} - \\frac{1}{2r_k}\n\t\t\t&\n\t\t\t\\frac{1}{2r_{k+1}} - \\frac{1+2\\rho}{2r_k}\n\t\t\t&\n\t\t\t0\n\t\t\\end{bmatrix}\n\t\t\\label{eq:Del}\n\t\\end{align}\n\\end{theorem}","start":24614,"end":27161},{"raw_tex":"\\subsection{Proof of recursive gluing (Theorem~\\ref{thm:cert})}\\label{ssec:gluing:pf}\n\n\\paragraph*{Non-negativity.} We verify $\\lam_{ij} \\geq 0$ for all $i \\neq j \\in \\{0, \\dots, 2n+1,*\\}$. For nearly all entries, this is obvious since $\\lam_{ij}$ is constructed by adding and multiplying non-negative numbers. For the remaining entries where either $\\Xi$ or $\\Delta$ is negative, compute the corresponding entry of $\\lam$ by summing the corrections and using Lemma~\\ref{lem:opt}. This gives $\\lam_{n*} = 0$, $\\lam_{2n+1,*} = \\rho^{k+1} - 1$, $\\lam_{*,n} = \\rho^{k-1} + 1$, $\\lam_{*,2n+1} = \\tfrac{1}{2r_{k+1}}$, and $\\lam_{*,t} = \\alpha_t$ for all $t \\in \\{n+1,\\dots,2n\\}$. All these entries are clearly non-negative.\n\n\\paragraph*{Rate certificate.} The identity~\\eqref{eq:cert:2n+1} has two components: a linear form in the function values and a quadratic form in the iterates and gradients. For the linear form, it suffices to verify $e-s-\\ell = 0$ by Lemma~\\ref{lem:succinct-linear}, where $e,s,\\ell \\in \\R^3$ are the vectors defined in Appendix~\\ref{app:succinct-linear}. This is obvious by inspection. For the quadratic form, it suffices to verify $E - S - L = 0$, where $E,S,L \\in \\R^{4 \\times 4}$ are the matrices in Lemma~\\ref{lem:succinct} defined in Appendix~\\ref{app:succinct}. By plugging in the explicit values for $r_k$ and $\\Delta$, this is straightforward to check by hand. For brevity, we provide a simple Mathematica script that verifies these identities at the URL~\\citep{MathematicaURL}.\n \n\n\n\t \n\t \n\t \\appendix\n\t \n","start":31347,"end":32881},{"raw_tex":"\\paragraph*{Non-negativity.} We verify $\\lam_{ij} \\geq 0$ for all $i \\neq j \\in \\{0, \\dots, 2n+1,*\\}$. For nearly all entries, this is obvious since $\\lam_{ij}$ is constructed by adding and multiplying non-negative numbers. For the remaining entries where either $\\Xi$ or $\\Delta$ is negative, compute the corresponding entry of $\\lam$ by summing the corrections and using Lemma~\\ref{lem:opt}. This gives $\\lam_{n*} = 0$, $\\lam_{2n+1,*} = \\rho^{k+1} - 1$, $\\lam_{*,n} = \\rho^{k-1} + 1$, $\\lam_{*,2n+1} = \\tfrac{1}{2r_{k+1}}$, and $\\lam_{*,t} = \\alpha_t$ for all $t \\in \\{n+1,\\dots,2n\\}$. All these entries are clearly non-negative.\n\n","start":31434,"end":32067},{"raw_tex":"\\paragraph*{Rate certificate.} The identity~\\eqref{eq:cert:2n+1} has two components: a linear form in the function values and a quadratic form in the iterates and gradients. For the linear form, it suffices to verify $e-s-\\ell = 0$ by Lemma~\\ref{lem:succinct-linear}, where $e,s,\\ell \\in \\R^3$ are the vectors defined in Appendix~\\ref{app:succinct-linear}. This is obvious by inspection. For the quadratic form, it suffices to verify $E - S - L = 0$, where $E,S,L \\in \\R^{4 \\times 4}$ are the matrices in Lemma~\\ref{lem:succinct} defined in Appendix~\\ref{app:succinct}. By plugging in the explicit values for $r_k$ and $\\Delta$, this is straightforward to check by hand. For brevity, we provide a simple Mathematica script that verifies these identities at the URL~\\citep{MathematicaURL}.\n","start":32067,"end":32856}]},{"id":"S027","semantic_type":"assumption","kind":"assumption","labels":["eq:cert:n","thm:cert"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"拼接输入证书假设","source_units":[{"raw_tex":"\\begin{theorem}[Recursive gluing]\\label{thm:cert}\n\tLet $n = 2^k-1$. Suppose $\\{\\sig_{ij}\\}_{i,j \\in \\{0,\\dots,n,*\\}} $ satisfies $*$-sparsity and certifies the $n$-step rate, i.e.,\n\t\\begin{align}\n\t\t\\sum_{i \\neq j \\in \\{0, \\dots, n,*\\}} \\sig_{ij} Q_{ij} \n\t\t=\n\t\t\\|x_0 - x^*\\|^2 - \\|x_n - c_k g_n - x^*\\|^2  + \\frac{f^* - f_n}{r_k}\\ .\n\t\t\\label{eq:cert:n}\n\t\\end{align}\n","start":24614,"end":24979}]},{"id":"S028","semantic_type":"definition","kind":"definition","labels":["eq:cert:2n+1","thm:cert"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"拼接目标证书形式","source_units":[{"raw_tex":"\tThen there exists $\\{\\lam_{ij}\\}_{i, j \\in \\{0,\\dots,2n+1,*\\}}$ that satisfies $*$-sparsity and certifies the $2n+1$-step rate, i.e.,\n\t\\begin{align}\n\t\t\\sum_{i \\neq j \\in \\{0, \\dots, 2n+1,*\\}} \\lam_{ij} Q_{ij} \n\t\t=\n\t\t\\|x_0 - x^*\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1} - x^*\\|^2 + \\frac{f^* - f_{2n+1}}{r_{k+1}}\\ .\n\t\t\\label{eq:cert:2n+1}\n\t\\end{align}\n","start":24979,"end":25326}]},{"id":"S029","semantic_type":"definition","kind":"definition","labels":["eq:lambda","thm:cert"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"拼接乘子分解","source_units":[{"raw_tex":"\tMoreover, this certificate is explicitly given by\n\t\\begin{align}\n\t\t\\lam_{ij} = \\underbrace{\\Theta_{ij}}_{\\text{gluing}}\n\t\t+\n\t\t\\underbrace{\\Xi_{ij}}_{\\text{rank-one correction}}\n\t\t+\n\t\t\\underbrace{\\Delta_{ij}}_{\\text{sparse correction}}\n\t\t\\label{eq:lambda}\n\t\\end{align}\n","start":25326,"end":25595}]},{"id":"S030","semantic_type":"definition","kind":"definition","labels":["eq:Theta","thm:cert"],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"两段拼接分量","source_units":[{"raw_tex":"\tThe ``gluing component'' $\\Theta$ is defined as \n\t\\begin{align}\n\t\t\\Theta_{ij} := \\underbrace{\\sig_{i,j} \\cdot \\mathds{1}_{i,j \\in \\{0,\\dots,n,*\\}}}_{\\text{recurrence for first $n$ steps}}\n\t\t\\;\\;+\\;\\;\n\t\t\\underbrace{ (1 + 2\\rho)\\; \\sig_{i-n-1,j-n-1} \\cdot \\mathds{1}_{i,j \\in \\{n+1,\\dots,2n+1,*\\}}}_{\\text{recurrence for last $n$ steps}}\\,.\n\t\t\\label{eq:Theta}\n\t\\end{align}\n","start":25595,"end":25967}]},{"id":"S031","semantic_type":"definition","kind":"definition","labels":["eq:Xi","thm:cert"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"秩一修正","source_units":[{"raw_tex":"\tThe ``rank-one correction'' $\\Xi$ is zero except the entries $\\{\\Xi_{ij}\\}_{i \\in \\{n,2n+1,*\\},\\; j \\in \\{n+1, \\dots, 2n\\}}$ which are\n\t\\begin{align}\n\t\t\\begin{bmatrix}\n\t\t\t\\Xi_{n,n+1} & \\Xi_{n,n+2} & \\cdots & \\Xi_{n,2n} \\\\\n\t\t\t\\Xi_{2n+1,n+1} & \\Xi_{2n+1,n+2} & \\cdots & \\Xi_{2n+1,2n} \\\\\n\t\t\t\\Xi_{*,n+1} & \\Xi_{*,n+2} & \\cdots & \\Xi_{*,2n}\n\t\t\\end{bmatrix}\n\t\t:=\n\t\t\\rho \\begin{bmatrix}\n\t\t\t1 \\\\\n\t\t\t1 \\\\\n\t\t\t-2\n\t\t\\end{bmatrix} \\begin{bmatrix}\n\t\t\t\\alpha_{n+1} & \\alpha_{n+2} & \\cdots & \\alpha_{2n}\n\t\t\\end{bmatrix}\n\t\t\\label{eq:Xi}\n\t\\end{align}\n","start":25967,"end":26501}]},{"id":"S032","semantic_type":"definition","kind":"definition","labels":["eq:Del","thm:cert"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"稀疏修正","source_units":[{"raw_tex":"\tThe ``sparse correction'' $\\Delta$ is zero except the entries $\\{\\Delta_{ij}\\}_{i \\neq j \\in \\{n,2n+1,*\\}}$ which are\n\t\\begin{align}\n\t\t\\begin{bmatrix}\n\t\t\t\\Delta_{n,n}\n\t\t\t&\n\t\t\t\\Delta_{n,2n+1}\n\t\t\t&\n\t\t\t\\Delta_{n,*} \n\t\t\t\\\\\n\t\t\t\\Delta_{2n+1,n}\n\t\t\t&\n\t\t\t\\Delta_{2n+1,2n+1} \n\t\t\t&\n\t\t\t\\Delta_{2n+1,*} \n\t\t\t\\\\\n\t\t\t\\Delta_{*,n}\n\t\t\t&\n\t\t\t\\Delta_{*,2n+1} \n\t\t\t&\n\t\t\t\\Delta_{*,*}\n\t\t\\end{bmatrix}\n\t\t:=\n\t\t\\begin{bmatrix}\n\t\t\t0 \n\t\t\t&\n\t\t\t\\rho\n\t\t\t&\n\t\t\t1 - \\rho^k\n\t\t\t\\\\\n\t\t\t\\rho^k\n\t\t\t&\n\t\t\t0 \n\t\t\t&\n\t\t\t2\\rho - \\sqrt{2} \\rho^{k+1}\n\t\t\t\\\\\n\t\t\t1 + \\rho^{k-1} - \\frac{1}{2r_k}\n\t\t\t&\n\t\t\t\\frac{1}{2r_{k+1}} - \\frac{1+2\\rho}{2r_k}\n\t\t\t&\n\t\t\t0\n\t\t\\end{bmatrix}\n\t\t\\label{eq:Del}\n\t\\end{align}\n","start":26501,"end":27148}]},{"id":"S033","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"主定理作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm:main}]\n\tIt suffices to prove~\\eqref{eq:cert}; we prove this by induction. The base case $n=1$ is Example~\\ref{ex:n=1}. The inductive step is Theorem~\\ref{thm:cert}. \n\\end{proof}","start":27439,"end":27656}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"主证明归纳基础","source_units":[{"raw_tex":"\tIt suffices to prove~\\eqref{eq:cert}; we prove this by induction. The base case $n=1$ is Example~\\ref{ex:n=1}. The inductive step is Theorem~\\ref{thm:cert}. \n","start":27486,"end":27645}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"主证明递归推广","source_units":[{"raw_tex":"\tIt suffices to prove~\\eqref{eq:cert}; we prove this by induction. The base case $n=1$ is Example~\\ref{ex:n=1}. The inductive step is Theorem~\\ref{thm:cert}. \n","start":27486,"end":27645}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"主证明归约到证书","source_units":[{"raw_tex":"\tIt suffices to prove~\\eqref{eq:cert}; we prove this by induction. The base case $n=1$ is Example~\\ref{ex:n=1}. The inductive step is Theorem~\\ref{thm:cert}. \n","start":27486,"end":27645}]},{"id":"S037","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"最优点平移归一化","source_units":[{"raw_tex":"The rest of the section is dedicated to proving Theorem~\\ref{thm:cert}. We first isolate two helper lemmas in \\S\\ref{ssec:gluing:helper}, and then we combine them to prove the result in \\S\\ref{ssec:gluing:pf}.  For notational simplicity, henceforth we assume $x^* = 0$; this is without loss of generality after translating. \n\n","start":27659,"end":27985}]},{"id":"S039","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"证书线性二次分解","source_units":[{"raw_tex":"The next two lemmas are more substantial. These help us verify~\\eqref{eq:cert:2n+1}---which consists of a linear form in all $2n+3$ function values and a quadratic form in all $2n+3$ gradients and iterates. Na\\\"ively verifying such an identity requires checking $\\Theta(n)$ coefficients for the linear form and $\\Theta(n^2)$ coefficients for the quadratic form. The following two lemmas show that due to the recursive construction of the Silver Stepsize Schedule and the gluing, these forms only affect the indices $n,2n+1,*$. This reduces verifying the rate certificate~\\eqref{eq:cert:2n+1} to checking only $\\Theta(1)$ coefficients, as detailed below in \\S\\ref{ssec:gluing:pf}. Below, for shorthand, let $F_{ij} := 2(f_i - f_j)$ and $P_{ij} := 2\\langle g_j, x_j - x_i \\rangle - \\|g_i - g_j\\|^2$  denote the linear and quadratic components of $Q_{ij}$, respectively. For bookkeeping purposes, we use $3$-dimensional vectors and $4 \\times 4$ matrices to denote the coefficients of these forms. \n","start":28912,"end":29907}]},{"id":"S040","semantic_type":"lemma","kind":"lemma","labels":["lem:opt"],"member_ids":["S040","S041","S042","S043"],"declared_external":false,"source_only":false,"title":"最优点相关乘子","source_units":[{"raw_tex":"\\begin{lemma}[Multipliers involving $x^*$]\\label{lem:opt} \n\t$\\sig_{n*} = \\rho^k - 1$, $\\sig_{*n} = \\frac{1}{2r_k}$, and $\\sig_{*t} = \\alpha_t$ for $t \\in \\{0,\\dots,n-1\\}$. \n\\end{lemma}","start":28216,"end":28400},{"raw_tex":"\\begin{proof}\n\tExpand both sides of the identity~\\eqref{eq:cert:n} using the $*$-sparsity assumption, the definition of the co-coercivities, and the definition of GD, i.e., $x_t = x_0 - \\sum_{s=0}^{t-1} \\alpha_s g_s$. Matching coefficients for the $\\langle x_0, g_t \\rangle$ terms yields the claimed formulas for $\\sig_{*t}$. Matching coefficients for the $f^*$ term gives the identity $\\sig_{n*} = \\sum_{j=0}^n \\sig_{*j}-  \\tfrac{1}{2r_k}$. This equals $\\rho^k - 1$ by Lemma~\\ref{lem:sum-steps}. \n\\end{proof}","start":28401,"end":28910},{"raw_tex":"\tExpand both sides of the identity~\\eqref{eq:cert:n} using the $*$-sparsity assumption, the definition of the co-coercivities, and the definition of GD, i.e., $x_t = x_0 - \\sum_{s=0}^{t-1} \\alpha_s g_s$. Matching coefficients for the $\\langle x_0, g_t \\rangle$ terms yields the claimed formulas for $\\sig_{*t}$. Matching coefficients for the $f^*$ term gives the identity $\\sig_{n*} = \\sum_{j=0}^n \\sig_{*j}-  \\tfrac{1}{2r_k}$. This equals $\\rho^k - 1$ by Lemma~\\ref{lem:sum-steps}. \n","start":28415,"end":28899}]},{"id":"S044","semantic_type":"lemma","kind":"lemma","labels":["lem:succinct-linear"],"member_ids":["S044","S045","S046","S047","S048"],"declared_external":false,"source_only":false,"title":"简洁线性形式","source_units":[{"raw_tex":"\\begin{lemma}[Succinct linear forms]\\label{lem:succinct-linear}\n\tLet $u := [f_n, f_{2n+1}, f^*]^T$, and let $e, s, \\ell \\in \\R^3$ be the vectors defined in Appendix~\\ref{app:succinct-linear}. \n\t\\begin{itemize}\n\t\t\\item \\underline{Gluing error.} $\\frac{f^* - f_{2n+1}}{r_{k+1}} - \\sum_{ij} \\Theta_{ij} F_{ij} = \\langle e, u \\rangle$\n\t\t\\item \\underline{Sparse correction.} $\\sum_{ij} \\Delta_{ij} F_{ij} = \\langle s, u \\rangle$\n\t\t\\item \\underline{Rank-one correction.} $\\sum_{ij} \\Xi_{ij} F_{ij} = \\langle \\ell, u \\rangle$\n\t\\end{itemize}\n\\end{lemma}","start":29910,"end":30455},{"raw_tex":"\\begin{proof}\n\tExpand the definition of co-coercivities, simplify the rank-one correction using Lemma~\\ref{lem:sum-steps}, and simplify the sparse correction using the Pell recurrence $\\rho^{k+1} = 2\\rho^k + \\rho^{k-1}$.\n\\end{proof}","start":30456,"end":30688},{"raw_tex":"\tExpand the definition of co-coercivities, simplify the rank-one correction using Lemma~\\ref{lem:sum-steps}, and simplify the sparse correction using the Pell recurrence $\\rho^{k+1} = 2\\rho^k + \\rho^{k-1}$.\n","start":30470,"end":30677}]},{"id":"S049","semantic_type":"lemma","kind":"lemma","labels":["lem:succinct"],"member_ids":["S049","S050","S051","S063","S064","S065","S066","S067","S068"],"declared_external":false,"source_only":false,"title":"简洁二次形式","source_units":[{"raw_tex":"\\begin{lemma}[Succinct quadratic forms]\\label{lem:succinct} Let $v := [x_n,g_n,x_{2n+1},g_{2n+1}]^T$, and let $E$, $S$, $L$ be the $4 \\times 4$ matrices defined in Appendix~\\ref{app:succinct}.\n\t\\begin{itemize}\n\t\t\\item \\underline{Gluing error.} \n\t\t$\\|x_0\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1}\\|^2\n\t\t- \\sum_{ij} \\Theta_{ij} P_{ij} = \\langle E, vv^T \\rangle$\n\t\t\\item \\underline{Sparse correction.} $\\sum_{ij} \\Delta_{ij} P_{ij} = \\langle S, vv^T \\rangle$\n\t\t\\item \\underline{Rank-one correction.} $\\sum_{ij} \\Xi_{ij} P_{ij} = \\langle L, vv^T \\rangle$\n\t\\end{itemize}\n\\end{lemma}","start":30690,"end":31261},{"raw_tex":"\\begin{proof}\n\tDeferred to Appendix~\\ref{app:succinct} for brevity.\n\\end{proof}","start":31262,"end":31341},{"raw_tex":"\\begin{proof}[Proof of Lemma~\\ref{lem:succinct}]\n\t\\underline{Gluing component.} \n\tRecall that $h_{2n+1} = [h_n, \\alpha_n, h_n]$ by the recursive construction~\\eqref{eq:steps-recursive} of the Silver Stepsize Schedule. Thus, since $\\sig$ certifies the $n$-step rate, we have\n\t\\begin{align*}\n\t\t\\sum_{i \\neq j \\in \\{0, \\dots, n,*\\}} \\sig_{ij} P_{ij} \n\t\t&=\n\t\t\\|x_0\\|^2 - \\|x_n - c_k g_n\\|^2\n\t\t\\\\ \n\t\t\\sum_{i \\neq j \\in \\{n+1, \\dots, 2n,*\\}} \\sig_{ij} P_{ij} \n\t\t&=\n\t\t\\|x_{n+1}\\|^2 - \\|x_{2n+1} - c_k g_{2n+1}\\|^2\n\t\\end{align*}\n\tBy definition, $\\sum_{ij} \\Theta_{ij} P_{ij}$ is the former plus $(1 + 2\\rho)$ times the latter. \n\tThus the gluing error $\\|x_0\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1}\\|^2 - \\sum_{ij} \\Theta_{ij} P_{ij}$ is equal to\n\t\\begin{align*}\n\t\t(1+2\\rho) \\|x_{2n+1} - c_k g_{2n+1}\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1}\\|^2 + \\|x_n - c_k g_n\\|^2 - (1 + 2\\rho) \\|x_{n+1}\\|^2\n\t\\end{align*}\n\tThe key point is that after \n\texpanding $x_{n+1} = x_n - \\alpha_n g_n$, this is a quadratic form in only the $4$ variables $x_n, g_n, x_{2n+1},g_{2n+1}$. Collecting terms, substituting $c_k = \\tfrac{1}{2r_k}$, and using the definition of the Silver Stepsize $\\alpha_n = 1 + \\rho^{k-1}$ yields $\\langle E, vv^T \\rangle$, as desired.\n\t\n\t\\par \\underline{Sparse component.} Expand the definition of the co-coercivity and collect terms. Note also that this equals the matrix $S$ in Part~I~\\citep{alt23hedging1}, in the limit that the strong convexity tends to $0$.\n\t\n\t\\par \\underline{Low-rank component.}  Expanding the co-coercivities, using the identity $x_{2n+1} - x_{n+1}= - \\sum_{t=n+1}^{2n} \\alpha_t g_t$ by definition of GD, and using the identity $\\sum_{t=n+1}^{2n} \\alpha_t = \\sum_{t=0}^{n-1} \\alpha_t = \\rho^k - 1$ from Lemma~\\ref{lem:sum-steps}, \n\t\\begin{align*}\n\t\t\\sum_{t=n+1}^{2n} \\Big( \\Xi_{n,t} P_{n,t} - \\Xi_{*,t} P_{*,t} \\Big)\n\t\t=\n\t\t\\rho \\sum_{t=n+1}^{2n} \\alpha_t \\Big(\n\t\t2 \\langle g_t, g_n - x_n \\rangle - \\|g_n\\|^2\n\t\t\\Big)\n\t\t=\n\t\t2\\rho \\langle x_{2n+1} - x_{n+1}, x_n - g_n \\rangle - \\rho(\\rho^k - 1 ) \\|g_n\\|^2 \\,.\n\t\\end{align*}\n\tAn identical calculation yields\n\t\\begin{align*}\n\t\t\\sum_{t=n+1}^{2n} \\Big( \\Xi_{2n+1,t} P_{2n+1,t} - \\Xi_{*,t} P_{*,t} \\Big)\n\t\t&=\n\t\t2\\rho \\langle x_{2n+1} - x_{n+1}, x_{2n+1} - g_{2n+1} \\rangle - \\rho(\\rho^k - 1 ) \\|g_{2n+1}\\|^2 \\,.\n\t\\end{align*}\n\tNow sum these two displays, use the definition of GD to expand $x_{n+1} = x_n - \\alpha_n g_n$, and use the definition of the Silver Stepsize $\\alpha_n = 1 + \\rho^{k-1}$. This gives $\\langle L, vv^T \\rangle$, as desired.\n\\end{proof}","start":35071,"end":37587},{"raw_tex":"\t\\underline{Gluing component.} \n\tRecall that $h_{2n+1} = [h_n, \\alpha_n, h_n]$ by the recursive construction~\\eqref{eq:steps-recursive} of the Silver Stepsize Schedule. Thus, since $\\sig$ certifies the $n$-step rate, we have\n\t\\begin{align*}\n\t\t\\sum_{i \\neq j \\in \\{0, \\dots, n,*\\}} \\sig_{ij} P_{ij} \n\t\t&=\n\t\t\\|x_0\\|^2 - \\|x_n - c_k g_n\\|^2\n\t\t\\\\ \n\t\t\\sum_{i \\neq j \\in \\{n+1, \\dots, 2n,*\\}} \\sig_{ij} P_{ij} \n\t\t&=\n\t\t\\|x_{n+1}\\|^2 - \\|x_{2n+1} - c_k g_{2n+1}\\|^2\n\t\\end{align*}\n","start":35120,"end":35592},{"raw_tex":"\tBy definition, $\\sum_{ij} \\Theta_{ij} P_{ij}$ is the former plus $(1 + 2\\rho)$ times the latter. \n\tThus the gluing error $\\|x_0\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1}\\|^2 - \\sum_{ij} \\Theta_{ij} P_{ij}$ is equal to\n\t\\begin{align*}\n\t\t(1+2\\rho) \\|x_{2n+1} - c_k g_{2n+1}\\|^2 - \\|x_{2n+1} - c_{k+1} g_{2n+1}\\|^2 + \\|x_n - c_k g_n\\|^2 - (1 + 2\\rho) \\|x_{n+1}\\|^2\n\t\\end{align*}\n\tThe key point is that after \n\texpanding $x_{n+1} = x_n - \\alpha_n g_n$, this is a quadratic form in only the $4$ variables $x_n, g_n, x_{2n+1},g_{2n+1}$. Collecting terms, substituting $c_k = \\tfrac{1}{2r_k}$, and using the definition of the Silver Stepsize $\\alpha_n = 1 + \\rho^{k-1}$ yields $\\langle E, vv^T \\rangle$, as desired.\n","start":35592,"end":36296},{"raw_tex":"\t\n\t\\par \\underline{Sparse component.} Expand the definition of the co-coercivity and collect terms. Note also that this equals the matrix $S$ in Part~I~\\citep{alt23hedging1}, in the limit that the strong convexity tends to $0$.\n\t\n","start":36296,"end":36526},{"raw_tex":"\t\\par \\underline{Low-rank component.}  Expanding the co-coercivities, using the identity $x_{2n+1} - x_{n+1}= - \\sum_{t=n+1}^{2n} \\alpha_t g_t$ by definition of GD, and using the identity $\\sum_{t=n+1}^{2n} \\alpha_t = \\sum_{t=0}^{n-1} \\alpha_t = \\rho^k - 1$ from Lemma~\\ref{lem:sum-steps}, \n\t\\begin{align*}\n\t\t\\sum_{t=n+1}^{2n} \\Big( \\Xi_{n,t} P_{n,t} - \\Xi_{*,t} P_{*,t} \\Big)\n\t\t=\n\t\t\\rho \\sum_{t=n+1}^{2n} \\alpha_t \\Big(\n\t\t2 \\langle g_t, g_n - x_n \\rangle - \\|g_n\\|^2\n\t\t\\Big)\n\t\t=\n\t\t2\\rho \\langle x_{2n+1} - x_{n+1}, x_n - g_n \\rangle - \\rho(\\rho^k - 1 ) \\|g_n\\|^2 \\,.\n\t\\end{align*}\n","start":36526,"end":37108},{"raw_tex":"\tAn identical calculation yields\n\t\\begin{align*}\n\t\t\\sum_{t=n+1}^{2n} \\Big( \\Xi_{2n+1,t} P_{2n+1,t} - \\Xi_{*,t} P_{*,t} \\Big)\n\t\t&=\n\t\t2\\rho \\langle x_{2n+1} - x_{n+1}, x_{2n+1} - g_{2n+1} \\rangle - \\rho(\\rho^k - 1 ) \\|g_{2n+1}\\|^2 \\,.\n\t\\end{align*}\n","start":37108,"end":37355},{"raw_tex":"\tNow sum these two displays, use the definition of GD to expand $x_{n+1} = x_n - \\alpha_n g_n$, and use the definition of the Silver Stepsize $\\alpha_n = 1 + \\rho^{k-1}$. This gives $\\langle L, vv^T \\rangle$, as desired.\n\\end{proof}","start":37355,"end":37587}]},{"id":"S056","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"对称矩阵省略约定","source_units":[{"raw_tex":"The matrices in Lemma~\\ref{lem:succinct} are defined as follows. For brevity, we write $\\sim$ for the entries below the diagonal since these coefficient matrices are symmetric.\n\\begin{align*}\n\t\\small\n","start":33694,"end":33894}]},{"id":"S057","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"拼接误差矩阵","source_units":[{"raw_tex":"\tE :=\n\t\\begin{bmatrix}\n\t\t-2\\rho\n\t\t&\n\t\t(1 + 2\\rho) (1 + \\rho^{k-1}) -\\frac{1}{2r_k} \n\t\t&\n\t\t0\n\t\t&\n\t\t0\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\frac{1}{4r_k^2} - (1 + 2\\rho) (1+\\rho^{k-1})^2\n\t\t&\n\t\t0\n\t\t&\n\t\t0 \n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t2\\rho \n\t\t&\n\t\t\\frac{1}{2r_{k+1}} - \\frac{1+2\\rho}{2r_k}\n\t\t\\\\\n\t\t\\sim & \\sim & \\sim &\n\t\t\\frac{1+2\\rho}{4r_k^2} - \\frac{1}{4r_{k+1}^2}\n\t\\end{bmatrix}\n\\end{align*}\n","start":33894,"end":34265}]},{"id":"S058","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"稀疏修正矩阵","source_units":[{"raw_tex":"\\begin{align*}\n\t\\small\n\tS := \n\t\\begin{bmatrix}\n\t\t0\n\t\t&\n\t\t\\Delta_{2n+1,n} + \\Delta_{*,n}\n\t\t&\n\t\t0\n\t\t&\n\t\t-\\Delta_{n,2n+1}\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t-\\left( \\Delta_{n,*} + \\Delta_{*,n} + \\Delta_{2n+1,n} + \\Delta_{n,2n+1} \\right)\n\t\t&\n\t\t-\\Delta_{2n+1,n}\n\t\t&\n\t\t\\Delta_{n,2n+1} + \\Delta_{2n+1,n}\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t0\n\t\t&\n\t\t\\Delta_{n,2n+1} + \\Delta_{*,2n+1}\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t- \\left( \\Delta_{n,2n+1} + \\Delta_{2n+1,n}  + \\Delta_{2n+1,*} + \\Delta_{*,2n+1}\\right)\n\t\\end{bmatrix}\n\\end{align*}\n","start":34265,"end":34771}]},{"id":"S059","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"秩一修正矩阵","source_units":[{"raw_tex":"\\begin{align*}\n\t\\small\n\tL :=  \\rho \\begin{bmatrix}\n\t\t-2\n\t\t&\n\t\t2 + \\rho^{k-1} \n\t\t&\n\t\t0\n\t\t&\n\t\t1\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t-1 - \\rho^k - 2 \\rho^{k-1} \n\t\t&\n\t\t\\rho^{k-1}\n\t\t&\n\t\t- 1 - \\rho^{k-1}\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t2\n\t\t&\n\t\t-1\n\t\t\\\\\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t\\sim\n\t\t&\n\t\t1 - \\rho^k\n\t\\end{bmatrix}\n\\end{align*}\n","start":34771,"end":35070}]},{"id":"S060","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"线性误差向量","source_units":[{"raw_tex":"The vectors $e, s, \\ell$ in Lemma~\\ref{lem:succinct-linear} are defined as follows:\n\\begin{align*}\n\t\\small\n\te := \\begin{bmatrix} \n\t\t\\frac{1}{r_k} \\\\\n\t\t-\\frac{1}{r_{k+1}} + \\frac{1+2\\rho}{r_k} \\\\\n\t\t\\frac{1}{r_{k+1}} - \\frac{4\\rho}{r_k}\n\t\\end{bmatrix}\n\t\\,,\n\t\\qquad\n","start":33033,"end":33296}]},{"id":"S061","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"原文向量 s","source_units":[{"raw_tex":"\ts := 2\\rho(\\rho^{k} - 1) \\begin{bmatrix}\n\t\t1 \\\\\n\t\t1 \\\\\n\t\t-2\n\t\\end{bmatrix}\n\t\\,,\n\t\\qquad\n","start":33296,"end":33385}]},{"id":"S062","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"原文向量 ell","source_units":[{"raw_tex":"\t\\ell := \\begin{bmatrix}\n\t\t\\frac{1}{r_k} - 2\\rho(\\rho^k - 1)\\\\\n\t\t-\\frac{1}{r_{k+1}} + \\frac{1+2\\rho}{r_k} - 2\\rho(\\rho^k - 1) \\\\\n\t\t\\frac{1}{r_{k+1}} - \\frac{4\\rho}{r_k} + 4\\rho(\\rho^k - 1)\n\t\\end{bmatrix}\\,.\n\\end{align*}\n","start":33385,"end":33605}]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S069"],"declared_external":true,"declaration_text":["原始 prerequisite 列表第二项：Standard first-order optimality at an unconstrained differentiable minimizer。"],"source_only":false,"title":"最小点一阶最优性","source_units":[]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S070"],"declared_external":true,"declaration_text":["原始 prerequisite 列表第三项：Elementary coefficient matching and algebraic identities for the silver ratio。"],"source_only":false,"title":"系数匹配与银比代数","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S022","S024":"S022","S025":"S022","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S039":"S039","S040":"S040","S041":"S040","S042":"S040","S043":"S040","S044":"S044","S045":"S044","S046":"S044","S047":"S044","S048":"S044","S049":"S049","S050":"S049","S051":"S049","S052":"S026","S053":"S026","S054":"S026","S055":"S026","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S049","S064":"S049","S065":"S049","S066":"S049","S067":"S049","S068":"S049","S069":"S069","S070":"S070"}}}}
{"record_id":"1510.08560v4","paper":{"title":"Why Random Reshuffling Beats Stochastic Gradient Descent","authors":["Gürbüzbalaban, Mert","Ozdaglar, Asuman","Parrilo, Pablo"],"arxiv_id":"1510.08560","version":"v4","journal":"Mathematical Programming","abstract_url":"https://arxiv.org/abs/1510.08560v4","pdf_url":"https://arxiv.org/pdf/1510.08560v4","doi":"10.1007/s10107-019-01440-w"},"context":{"raw_tex":"We consider the following unconstrained optimization problem where the objective function is the sum of a large number of component functions: \n\\beq\\label{pbm-multi-agent}\n\\min f(x) := \\sum_{i=1}^m f_i(x) \\quad \\mbox{s.t.} \\quad x \\in \\R^n\n\\eeq\nwith $f_i:\\R^n \\to \\R$. This problem arises in many contexts and applications including regression or more generally parameter estimation problems (where $f_i(x)$ is the loss function representing the error between the output and the prediction of a parametric model)~\\cite{Boyd2011AdmmBook,bertsekas2011incremental,bertsekas1997hybrid, Bertsekas1996incremental}, minimization of an expected value of a function (where the expectation is taken over a finite probability distribution or approximated by an $m$-sample average)~\\cite{BottouLecun2005, Leroux2012sgd}, machine learning~\\cite{Jordan13DistLearningApi, Dickenstein2014, Leroux2012sgd}, or distributed optimization over networks~\\cite{RamNedicVeer2007, NedicOzdaglar09,Nedic2007rate}.\n\n\nAn alternative popular approach that works well in practice is following a mixed approach between SGD and IG, sampling the functions randomly but not allowing repetitions, that is sampling the component functions at each iteration \\textit{without-replacement}, or equivalently picking a random order at each cycle. Specifically, at each cycle $k$, we draw a permutation $\\sigma_k $ of $\\{1,2,\\dots,m\\}$ independently and uniformly at random over the set of all permutations \\beq \\Gamma = \\big\\{ \\sigma ~:~ \\sigma \\mbox{ is a permutation of } \\{1,2,\\dots,m\\} \\big\\} \\label{def-Gamma-perm-set}\\eeq \nand process the functions with this order: %$\\sigma_k = (\\sigma_k(1), \\sigma_k(2),\\dots, \\sigma_k(m)$:\n\\begin{equation}\\label{inner-update-wo} x_{i}^k :={x_{i-1}^k - \\alpha_k \\nabla f_{\\sigma_k(i)} (x_{i-1}^k)} , \\qquad i=1,2,\\dots,m,\n   \\end{equation} \n where $\\alpha_k>0$ is a stepsize. We set $x_0^{k+1} = x_{m}^k$ as before and refer to $\\{x_0^k\\}$ as the {\\it outer iterates}. This method is called the \\textit{Random Reshuffling} (RR) method~\\cite[Section 2.1]{Bertsekas15Book} and will be the focus of this paper.\n\n\n\\noindent \\textbf{Notation}: We study the point-wise dominance of stochastic sequences by deterministic sequences and use the following notation. Let $x_k = x_k(\\omega)$ be a stochastic real-valued sequence (where $\\omega$ can be thought as the source of randomness) and $y_k$ be a real-valued deterministic sequence. We write $ x_k = \\bigO(y_k)  \\iff \\exists h>0, \\exists k_0 \\quad \\mbox{such that} \\quad  |x_k| \\leq  h |y_k| \\quad \\forall k\\geq k_0, \\forall \\omega,$ where $h$ and $k_0$ are independent of $\\omega$ (Note that the requirement is that this inequality holds for all $\\omega$, not just for almost all $\\omega$). When $x_k$ is non-negative for every $\\omega$, given another deterministic positive sequence $z_k$, we also introduce the inequality version of this definition: \n\t$x_k \\leq y_k + o(z_k) \\iff \\forall \\varepsilon>0, \\exists k_0(\\varepsilon) \\quad \\mbox{such that}\\\t\\quad z_k^{-1} |x_k(\\omega) - y_k| \\leq \\varepsilon, \\quad \\forall k\\geq k_0(\\varepsilon), \\forall \\omega$\nwhere $k_0$ depends on $\\varepsilon$ but is independent of $\\omega$.\tWhen $x_k$ is deterministic, these definitions reduce to the standard definitions of $\\bigO(\\cdot)$ and $o(\\cdot)$ for deterministic sequences. For random $x_k$, the only difference is that we require the constants to be independent of the choice of $\\omega$. For example, if $x_k$ is uniformly distributed over $[0,10]$, we write $x_k = \\bigO(1)$. Throughout the paper, $\\| \\cdot \\|$ denotes the vector or matrix 2-norm (maximum singular value). %The notation $\\E_{\\sigma}X(\\sigma)$ denotes the expectation with respect to a random permutation $\\sigma$ of a random variable $X(\\sigma)$.\n\n\nWe consider solving problem (\\ref{pbm-multi-agent}) with RR method with iterations given in (\\ref{inner-update-wo}). Throughout we assume the following:  \n\\begin{assumption}\\label{assump-sum-is-str-cvx} The sum function $f(x)=\\sum_{i=1}^m f_i(x)$ is strongly convex, i.e., there exists a constant $c>0$ such that the function $f(x) - \\frac{c}{2} \\| x\\| ^2$ is convex on $\\R^n$.\\footnote{Such functions arise naturally in support vector machines and other regularized learning algorithms or regression problems (see e.g. \\cite{Leroux2012sgd, Rakhlin:2011, Hogwild11})}.\n\\end{assumption}\n\n\nNote that this assumption is on the sum function $f$, it does not require the convexity of the individual component functions $f_i$. A consequence of this assumption is that there exists a unique optimal solution to \\eqref{pbm-multi-agent} which we denote by $x^*$. Another consequence is that the Hessian at the optimal solution is invertible since\n\t \\beq \\Hs := \\nabla^2 f(x^*) \\succeq c I_n \\succ 0\n\t\t\\label{def-Hs}\t \n\t \\eeq \nwhere $I_n$\tis the $n\\times n$ identity matrix. \n\n\nTo analyze RR, we view it as a gradient method with random gradient errors and rewrite \nthe outer iterations \\eqref{inner-update-wo} as \n   \\beq \\frac{x_0^k - x_{0}^{k+1}}{\\alpha_k} = \\nabla f(x_0^k) + E_k \\label{averaged-iter-randomized}\n   \\eeq\nwhere \n   \\beq E_k := \\sum_{i=1}^{m} \\bigg( \\nabla f_{\\ski}(x_{i-1}^k) -  \\nabla f_{\\ski} (x_0^k) \\bigg) \\label{def-Ek}\n   \\eeq\nis the cumulative gradient errors associated with the cycle $k$. \n\n\nA key idea in our analysis is to use a recent upper bound for the convergence rate of cyclic incremental gradient method (see  \\cite{Gur2015IncGrad}), which can be generalized to hold for any fixed deterministic order. This bound implies an almost sure upper bound (in fact one that holds for all sample paths) on the distance of the outer iterates $x_0^k$ generated by RR from the optimal solution $x^*$ denoted by\n\t$\\dist_k  =\\| x_0^k - x^*\\|$\n(see Section \\ref{quad-conv}). Crucially, this result implies an upper bound in expected distance which is asymptotically $m$ times smaller than the almost sure guarantees on the distance of the iterates.\n\nIn analyzing RR, we will also consider the \\textit{average} of the outer iterate sequence given by\\footnote{It is well known that computing this (moving) average can be done efficiently in a dynamic manner by storing a vector of length $n$.} \n\t$ \\bar{x}_{k} := \\frac{\\sum_{j=0}^{k-1} x_0^j}{k}. \\label{simple-average}\n\t$\n We also consider averaging only the most recent iterates, i.e. at iteration $k$, averaging the last $qk$ iterates\nfor some constant $q\\in (0,1]$:\n\t$$\\bar{x}_{q,k} := \\frac{\\sum_{j=(1-q)k }^{k-1} x_0^j}{qk}, \\quad 0 < q\\leq 1. $$ \nThe generated sequence is referred to as the \\textit{$q$-suffix average} of the sequence $x_0^k$. For SGD, it was shown that $q$-suffix averaging with $0<q<1$ leads to better performance then averaging (which corresponds to the $q=1$ case by definition), improving the convergence rate in the suboptimality of the function value from $\\log k/k$ to $1/k$~\\cite{Rakhlin:2011, Shamir:2012tj}. This is in line with our results in Section~\\ref{sec:quad-case} which show faster rate for the $0<q<1$ case. The parameter $q$ can be thought as a measure of how much memory one uses during the averaging process. We define the \\textit{$q$-suffix average} of the stepsize in a similar way:\n \\beq \\bar{\\alpha}_{q,k} =\\frac{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}\n{qk}, \\quad 0<q\\leq 1.\\label{def-averaged-stepsize}\n\\eeq\n\n\nLet $f_i: \\R^n \\to \\R$ be a quadratic function of the form\n    \\beq f_i(x) = \\frac{1}{2} x_i^T P_i x - q_i^T x + r_i, \\quad i=1,2,\\dots, m,\n    \t\\label{def-fi}\n    \\eeq\nwhere $P_i $ is a symmetric $n\\times n$ matrix, $q_i \\in \\R^n$ is a column vector and $r_i$ is a scalar. Note that $f_i$ has Lipschitz gradients, i.e., \n    $$ \\| \\nabla f_i(y) - \\nabla f_i (z) \\| \\leq L_i \\| y - z\\|, \\quad \\forall y, z \\in \\R^n, $$\nwhere $L_i = \\| P_i \\|$. It follows from the triangle inequality that $f$ has Lipschitz gradients with Lipschitz constant at most\n\t\\beq  L := \\sum_{i=1}^m L_i .\n\t\t\\label{def-L}\n\t\\eeq\nMoreover, Assumption \\ref{assump-sum-is-str-cvx} implies that the Hessian matrix of the sum satisfies \n$\\nabla^2 f(x) = \\sum_{i=1}^m \\nabla^2 f_i(x) = \\sum_{i=1}^m P_i \\ge cI_n >0.$\n\n\n\\begin{theorem}\\cite{Gur2015IncGrad} \nLet Assumption \\ref{assump-sum-is-str-cvx} hold. Let $f_i(x)$ be a quadratic function of the form $f_i(x) = \\frac{1}{2} x_i^T P_i x - q_i^T x + r_i$\nwhere $P_i $ is a symmetric $n\\times n$ matrix, $q_i \\in \\R^n$ is a column vector and $r_i$ is a scalar for $i=1,2,\\dots,m$. Suppose Assumption \\ref{assump-sum-is-str-cvx}  holds. Consider the iterates $\\{x_0^k\\}$ generated by the iterations \\eqref{inner-update-wo} with a fixed order $\\sigma$ and stepsize $\\alpha_k = R/(k+1)^s$ where $R>0$ and  $s \\in (1/2,1)$. Then\\footnote{The original result in \\cite{Gur2015IncGrad} was stated for $\\sigma=\\{1,2,\\dots,m\\}$ but here we translate this result into an arbitrary permutation $\\sigma$ of $\\{1,2,\\dots,m\\}$ by noting that processing the set of functions $\\{f_1, f_2, \\dots, f_m\\}$ with order $\\sigma$ is equivalent to processing the permuted functions $\\{f_{\\s_1}, f_{\\s_2}, \\dots, f_{\\s_m}\\}$ with order $\\{1,2,\\dots,m\\}$.\n},\n         % \\beq \\dist_k \\leq \\frac{R \\| \\mu(\\sigma)\\| }{c}\\frac{1}{k^s} + o(\\frac{1}{k^s}) \\quad \\mbox{where} \\quad  \\mu(\\sigma) =  \\underset{1\\leq i < j\\leq m}{\\sum} P_{\\sigma(j)} \\nabla f_{\\sigma(i)}(x^*). %\\label{def-Msigma}% \\leq Lm \\Gs\n         % \\eeq\n    \t\\begin{align} \\dist_k &\\leq \\frac{R \\| \\mu(\\sigma)\\| }{c}\\frac{1}{k^s} + o(\\frac{1}{k^s})  &\\mbox{if}& \\quad 1/2<s<1, \\label{rate-prob-one-no-average}\\\\\n           \t\t\t  \\dist_k  &\\leq \\frac{R^2 \\|  \\mu({\\sigma}) \\|}{ Rc - 1} \\frac{1}{k} + o(\\frac{1}{k})  &\\mbox{if}& \\quad s=1 \\mbox{ and } Rc>1,\n\t  %\\label{def-Msigma}% \\leq Lm \\Gs\n    \t\\end{align}\nwhere $c$ is the strong convexity constant of the sum function $f(x)$ and\n\\beq\n\t\\mu(\\sigma) =  - \\underset{1\\leq i < j\\leq m}{\\sum} P_{\\sigma(j)} \\nabla f_{\\sigma(i)}(x^*). \\label{def-Msigma}\n\\eeq  \t\n\t\\label{theo-rate-quadratics} \n\\end{theorem}\n\nWe next provide some upper bounds on $\\mu(\\sigma)$. We define \n\t\\beqa \\Gs :&=& \\underset{1\\leq i \\leq m}{\\sup} \\| \\nabla f_i(x^*)\\|, \n\t\\label{def-Gs} \\\\\n%\\eeq\n%and \n%\\beq\n\t M_\\Gamma :&=&~ \\sup_{\\sigma \\in \\Gamma} \\|\\mu(\\sigma)\\|.\n\t \\label{def-M-gamma}\t\n\\eeqa\nUsing  $L_i = \\| P_i \\|$ for each $i$, it follows from the triangle inequality that\n\t\\beq  \\| \\mu(\\sigma) \\| \\leq M_\\Gamma  \\leq \\sup_{\\sigma \\in \\Gamma}  \\underset{1\\leq i < j\\leq m}{\\sum} L_{\\sigma(j)} \\Gs \\leq \\sum_{j=1}^m  (j-1) L_{\\sigma(j)} \\Gs \\leq Lm\\Gs\n\t\t\\label{def-M-gamma-bound}\n\t\\eeq\nwhere $L$ is the Lipschitz constant of the gradient of $f$ defined by \\eqref{def-L}. By replacing $\\mu(\\sigma)$ by $M_\\Gamma$ in Theorem \\ref{theo-rate-quadratics} one can get an upper bound on the worst-case convergence rate that applies to any choice of fixed order $\\sigma$. Using a similar argument along the lines of the proof of Theorem \\ref{theo-rate-quadratics} on the convergence rate of IG, it is straightforward to show that RR never performs any slower than this worst-case convergence rate which is the subject of the next result. The idea is to bound the stochastic $\\dist_k$ sequence from above point-wise. The proof is a simple exercise and is omitted due to space considerations. %left to the reader. % by a reasoning along the lines of the proof of Theorem \\ref{theo-rate-quadratics}, the details of the proof is skipped due to space considerations.\n\n\nOne way to get better constants is to consider convergence in expectation, a weaker notion of convergence compared to almost sure convergence. In the next theorem, we show that $M_\\Gamma$ can be improved to a typically much smaller constant $\\|\\bar{\\mu}\\|$ where \\beq \n\t\\bar{\\mu} := \\E \\big(\\mu(\\sigma_1)\\big) =\n\t\t\t\\frac{\\sum_{\\sigma \\in \\Gamma} \\mu(\\sigma)}{|\\Gamma|}. \\label{def-mu-bar}\n\\eeq can be thought as a measure of average performance over the choice of random permutations. \n\n\n\\begin{remark} A consequence of Lemma \\ref{lem-computing-mu-bar} proved in the Appendix is that \\beq \n  \t\\bar{\\mu} =  \\frac{1}{2} \\sum_{i=1}^m P_i \\nabla f_i(x^*)\n\\label{mu-bar-alternative}.\n\\eeq\nwhere $\\bar{\\mu}$ is defined by \\eqref{def-mu-bar}. By the triangle inequality, \n\t$\\| \\bar{\\mu} \\| \\leq \\sum_{i=1}^m L_i \\Gs = L\\Gs $\nwhere $\\Gs$ is defined by \\eqref{def-Gs}. This upper bound is $m$ times smaller than the previous upper bound on $M_\\Gamma$ in \\eqref{def-M-gamma-bound}. \\rev{As an example, consider $s= 1$ with $Rc = 2$. In this case, $L = \\bigO(m)$, $c = \\bigO(m)$, $R = \\bigO(1/m)$ and $\\| \\bar{\\mu}\\| = \\bigO(m)$. We obtain from \\eqref{exp-dist-s-1} that $\\E \\left(\\dist_k\\right) = O(\\frac{1}{mk}) + o(1/k)$. Note that, the performance guarantee for IG from Corollary \\ref{coro-deter-decay-rate-dist} is $\\dist_k = \\bigO(1/k) + o(1/k)$ which is also worse than RR by a factor of $O(m)$. For a fair comparison with the SGD method, we consider running the SGD iterations for $j = km$ iterations so that both RR and SGD methods have access to the same number of component gradients. In this case, expected distance to suboptimality for SGD with the recommended $O(1/j)$ stepsize is $\\bigO(\\frac{1}{\\sqrt{j}})$ where the hidden constants are independent of $m$ (see e.g. \\cite{PolyakJuditsky92,Moulines:2011vy}) which is worse than the $\\bigO(1/j)$ guarantees for RR when $j$ is sufficiently large. These bounds show that when $m$ is small and $k$ is large, IG could outperform SGD in theory, however SGD (and RR) are more suitable for applications when $m$ is large and will admit better bounds compared to IG if $m$ is large enough for a given $k$ fixed\\footnote{We note however that SGD upper bounds are in expectation whereas IG results are deterministic which is a stronger notion of convergence.}}  %This requires $k$ cycles, i.e. $j=km$ evaluations of the gradients of the component functions. \n% \tThe quantity \n% \t$$ Q = L/c$$\n% is called the condition number and is a well-known measure of the difficulty of the optimization problem \\eqref{pbm-multi-agent}. \n\\end{remark}\n\n\nThe first lemma is on characterizing what is the worst-case distance of the all the inner iterates of RR to the optimal solution $x^*$. This quantity we want to upper bound is a random variable, but the upper bounds we obtain are deterministic holding for every sample path. This lemma is based on Corollary \\ref{coro-deter-decay-rate-dist} and uses the fact that the distance between the inner iterates are on the order of the stepsize.\n","units":[{"raw_tex":"We consider the following unconstrained optimization problem where the objective function is the sum of a large number of component functions: \n\\beq\\label{pbm-multi-agent}\n\\min f(x) := \\sum_{i=1}^m f_i(x) \\quad \\mbox{s.t.} \\quad x \\in \\R^n\n\\eeq\nwith $f_i:\\R^n \\to \\R$. This problem arises in many contexts and applications including regression or more generally parameter estimation problems (where $f_i(x)$ is the loss function representing the error between the output and the prediction of a parametric model)~\\cite{Boyd2011AdmmBook,bertsekas2011incremental,bertsekas1997hybrid, Bertsekas1996incremental}, minimization of an expected value of a function (where the expectation is taken over a finite probability distribution or approximated by an $m$-sample average)~\\cite{BottouLecun2005, Leroux2012sgd}, machine learning~\\cite{Jordan13DistLearningApi, Dickenstein2014, Leroux2012sgd}, or distributed optimization over networks~\\cite{RamNedicVeer2007, NedicOzdaglar09,Nedic2007rate}.\n","start":9279,"end":10267},{"raw_tex":"An alternative popular approach that works well in practice is following a mixed approach between SGD and IG, sampling the functions randomly but not allowing repetitions, that is sampling the component functions at each iteration \\textit{without-replacement}, or equivalently picking a random order at each cycle. Specifically, at each cycle $k$, we draw a permutation $\\sigma_k $ of $\\{1,2,\\dots,m\\}$ independently and uniformly at random over the set of all permutations \\beq \\Gamma = \\big\\{ \\sigma ~:~ \\sigma \\mbox{ is a permutation of } \\{1,2,\\dots,m\\} \\big\\} \\label{def-Gamma-perm-set}\\eeq \nand process the functions with this order: %$\\sigma_k = (\\sigma_k(1), \\sigma_k(2),\\dots, \\sigma_k(m)$:\n\\begin{equation}\\label{inner-update-wo} x_{i}^k :={x_{i-1}^k - \\alpha_k \\nabla f_{\\sigma_k(i)} (x_{i-1}^k)} , \\qquad i=1,2,\\dots,m,\n   \\end{equation} \n where $\\alpha_k>0$ is a stepsize. We set $x_0^{k+1} = x_{m}^k$ as before and refer to $\\{x_0^k\\}$ as the {\\it outer iterates}. This method is called the \\textit{Random Reshuffling} (RR) method~\\cite[Section 2.1]{Bertsekas15Book} and will be the focus of this paper.\n","start":13220,"end":14338},{"raw_tex":"\\noindent \\textbf{Notation}: We study the point-wise dominance of stochastic sequences by deterministic sequences and use the following notation. Let $x_k = x_k(\\omega)$ be a stochastic real-valued sequence (where $\\omega$ can be thought as the source of randomness) and $y_k$ be a real-valued deterministic sequence. We write $ x_k = \\bigO(y_k)  \\iff \\exists h>0, \\exists k_0 \\quad \\mbox{such that} \\quad  |x_k| \\leq  h |y_k| \\quad \\forall k\\geq k_0, \\forall \\omega,$ where $h$ and $k_0$ are independent of $\\omega$ (Note that the requirement is that this inequality holds for all $\\omega$, not just for almost all $\\omega$). When $x_k$ is non-negative for every $\\omega$, given another deterministic positive sequence $z_k$, we also introduce the inequality version of this definition: \n\t$x_k \\leq y_k + o(z_k) \\iff \\forall \\varepsilon>0, \\exists k_0(\\varepsilon) \\quad \\mbox{such that}\\\t\\quad z_k^{-1} |x_k(\\omega) - y_k| \\leq \\varepsilon, \\quad \\forall k\\geq k_0(\\varepsilon), \\forall \\omega$\nwhere $k_0$ depends on $\\varepsilon$ but is independent of $\\omega$.\tWhen $x_k$ is deterministic, these definitions reduce to the standard definitions of $\\bigO(\\cdot)$ and $o(\\cdot)$ for deterministic sequences. For random $x_k$, the only difference is that we require the constants to be independent of the choice of $\\omega$. For example, if $x_k$ is uniformly distributed over $[0,10]$, we write $x_k = \\bigO(1)$. Throughout the paper, $\\| \\cdot \\|$ denotes the vector or matrix 2-norm (maximum singular value). %The notation $\\E_{\\sigma}X(\\sigma)$ denotes the expectation with respect to a random permutation $\\sigma$ of a random variable $X(\\sigma)$.\n","start":30505,"end":32159},{"raw_tex":"We consider solving problem (\\ref{pbm-multi-agent}) with RR method with iterations given in (\\ref{inner-update-wo}). Throughout we assume the following:  \n\\begin{assumption}\\label{assump-sum-is-str-cvx} The sum function $f(x)=\\sum_{i=1}^m f_i(x)$ is strongly convex, i.e., there exists a constant $c>0$ such that the function $f(x) - \\frac{c}{2} \\| x\\| ^2$ is convex on $\\R^n$.\\footnote{Such functions arise naturally in support vector machines and other regularized learning algorithms or regression problems (see e.g. \\cite{Leroux2012sgd, Rakhlin:2011, Hogwild11})}.\n\\end{assumption}\n\n\nNote that this assumption is on the sum function $f$, it does not require the convexity of the individual component functions $f_i$. A consequence of this assumption is that there exists a unique optimal solution to \\eqref{pbm-multi-agent} which we denote by $x^*$. Another consequence is that the Hessian at the optimal solution is invertible since\n\t \\beq \\Hs := \\nabla^2 f(x^*) \\succeq c I_n \\succ 0\n\t\t\\label{def-Hs}\t \n\t \\eeq \nwhere $I_n$\tis the $n\\times n$ identity matrix. \n\n\nTo analyze RR, we view it as a gradient method with random gradient errors and rewrite \nthe outer iterations \\eqref{inner-update-wo} as \n   \\beq \\frac{x_0^k - x_{0}^{k+1}}{\\alpha_k} = \\nabla f(x_0^k) + E_k \\label{averaged-iter-randomized}\n   \\eeq\nwhere \n   \\beq E_k := \\sum_{i=1}^{m} \\bigg( \\nabla f_{\\ski}(x_{i-1}^k) -  \\nabla f_{\\ski} (x_0^k) \\bigg) \\label{def-Ek}\n   \\eeq\nis the cumulative gradient errors associated with the cycle $k$. \n","start":32369,"end":33878},{"raw_tex":"A key idea in our analysis is to use a recent upper bound for the convergence rate of cyclic incremental gradient method (see  \\cite{Gur2015IncGrad}), which can be generalized to hold for any fixed deterministic order. This bound implies an almost sure upper bound (in fact one that holds for all sample paths) on the distance of the outer iterates $x_0^k$ generated by RR from the optimal solution $x^*$ denoted by\n\t$\\dist_k  =\\| x_0^k - x^*\\|$\n(see Section \\ref{quad-conv}). Crucially, this result implies an upper bound in expected distance which is asymptotically $m$ times smaller than the almost sure guarantees on the distance of the iterates.\n\nIn analyzing RR, we will also consider the \\textit{average} of the outer iterate sequence given by\\footnote{It is well known that computing this (moving) average can be done efficiently in a dynamic manner by storing a vector of length $n$.} \n\t$ \\bar{x}_{k} := \\frac{\\sum_{j=0}^{k-1} x_0^j}{k}. \\label{simple-average}\n\t$\n We also consider averaging only the most recent iterates, i.e. at iteration $k$, averaging the last $qk$ iterates\nfor some constant $q\\in (0,1]$:\n\t$$\\bar{x}_{q,k} := \\frac{\\sum_{j=(1-q)k }^{k-1} x_0^j}{qk}, \\quad 0 < q\\leq 1. $$ \nThe generated sequence is referred to as the \\textit{$q$-suffix average} of the sequence $x_0^k$. For SGD, it was shown that $q$-suffix averaging with $0<q<1$ leads to better performance then averaging (which corresponds to the $q=1$ case by definition), improving the convergence rate in the suboptimality of the function value from $\\log k/k$ to $1/k$~\\cite{Rakhlin:2011, Shamir:2012tj}. This is in line with our results in Section~\\ref{sec:quad-case} which show faster rate for the $0<q<1$ case. The parameter $q$ can be thought as a measure of how much memory one uses during the averaging process. We define the \\textit{$q$-suffix average} of the stepsize in a similar way:\n \\beq \\bar{\\alpha}_{q,k} =\\frac{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}\n{qk}, \\quad 0<q\\leq 1.\\label{def-averaged-stepsize}\n\\eeq\n","start":35073,"end":37077},{"raw_tex":"Let $f_i: \\R^n \\to \\R$ be a quadratic function of the form\n    \\beq f_i(x) = \\frac{1}{2} x_i^T P_i x - q_i^T x + r_i, \\quad i=1,2,\\dots, m,\n    \t\\label{def-fi}\n    \\eeq\nwhere $P_i $ is a symmetric $n\\times n$ matrix, $q_i \\in \\R^n$ is a column vector and $r_i$ is a scalar. Note that $f_i$ has Lipschitz gradients, i.e., \n    $$ \\| \\nabla f_i(y) - \\nabla f_i (z) \\| \\leq L_i \\| y - z\\|, \\quad \\forall y, z \\in \\R^n, $$\nwhere $L_i = \\| P_i \\|$. It follows from the triangle inequality that $f$ has Lipschitz gradients with Lipschitz constant at most\n\t\\beq  L := \\sum_{i=1}^m L_i .\n\t\t\\label{def-L}\n\t\\eeq\nMoreover, Assumption \\ref{assump-sum-is-str-cvx} implies that the Hessian matrix of the sum satisfies \n$\\nabla^2 f(x) = \\sum_{i=1}^m \\nabla^2 f_i(x) = \\sum_{i=1}^m P_i \\ge cI_n >0.$\n","start":48674,"end":49458},{"raw_tex":"\\begin{theorem}\\cite{Gur2015IncGrad} \nLet Assumption \\ref{assump-sum-is-str-cvx} hold. Let $f_i(x)$ be a quadratic function of the form $f_i(x) = \\frac{1}{2} x_i^T P_i x - q_i^T x + r_i$\nwhere $P_i $ is a symmetric $n\\times n$ matrix, $q_i \\in \\R^n$ is a column vector and $r_i$ is a scalar for $i=1,2,\\dots,m$. Suppose Assumption \\ref{assump-sum-is-str-cvx}  holds. Consider the iterates $\\{x_0^k\\}$ generated by the iterations \\eqref{inner-update-wo} with a fixed order $\\sigma$ and stepsize $\\alpha_k = R/(k+1)^s$ where $R>0$ and  $s \\in (1/2,1)$. Then\\footnote{The original result in \\cite{Gur2015IncGrad} was stated for $\\sigma=\\{1,2,\\dots,m\\}$ but here we translate this result into an arbitrary permutation $\\sigma$ of $\\{1,2,\\dots,m\\}$ by noting that processing the set of functions $\\{f_1, f_2, \\dots, f_m\\}$ with order $\\sigma$ is equivalent to processing the permuted functions $\\{f_{\\s_1}, f_{\\s_2}, \\dots, f_{\\s_m}\\}$ with order $\\{1,2,\\dots,m\\}$.\n},\n         % \\beq \\dist_k \\leq \\frac{R \\| \\mu(\\sigma)\\| }{c}\\frac{1}{k^s} + o(\\frac{1}{k^s}) \\quad \\mbox{where} \\quad  \\mu(\\sigma) =  \\underset{1\\leq i < j\\leq m}{\\sum} P_{\\sigma(j)} \\nabla f_{\\sigma(i)}(x^*). %\\label{def-Msigma}% \\leq Lm \\Gs\n         % \\eeq\n    \t\\begin{align} \\dist_k &\\leq \\frac{R \\| \\mu(\\sigma)\\| }{c}\\frac{1}{k^s} + o(\\frac{1}{k^s})  &\\mbox{if}& \\quad 1/2<s<1, \\label{rate-prob-one-no-average}\\\\\n           \t\t\t  \\dist_k  &\\leq \\frac{R^2 \\|  \\mu({\\sigma}) \\|}{ Rc - 1} \\frac{1}{k} + o(\\frac{1}{k})  &\\mbox{if}& \\quad s=1 \\mbox{ and } Rc>1,\n\t  %\\label{def-Msigma}% \\leq Lm \\Gs\n    \t\\end{align}\nwhere $c$ is the strong convexity constant of the sum function $f(x)$ and\n\\beq\n\t\\mu(\\sigma) =  - \\underset{1\\leq i < j\\leq m}{\\sum} P_{\\sigma(j)} \\nabla f_{\\sigma(i)}(x^*). \\label{def-Msigma}\n\\eeq  \t\n\t\\label{theo-rate-quadratics} \n\\end{theorem}","start":49907,"end":51726},{"raw_tex":"We next provide some upper bounds on $\\mu(\\sigma)$. We define \n\t\\beqa \\Gs :&=& \\underset{1\\leq i \\leq m}{\\sup} \\| \\nabla f_i(x^*)\\|, \n\t\\label{def-Gs} \\\\\n%\\eeq\n%and \n%\\beq\n\t M_\\Gamma :&=&~ \\sup_{\\sigma \\in \\Gamma} \\|\\mu(\\sigma)\\|.\n\t \\label{def-M-gamma}\t\n\\eeqa\nUsing  $L_i = \\| P_i \\|$ for each $i$, it follows from the triangle inequality that\n\t\\beq  \\| \\mu(\\sigma) \\| \\leq M_\\Gamma  \\leq \\sup_{\\sigma \\in \\Gamma}  \\underset{1\\leq i < j\\leq m}{\\sum} L_{\\sigma(j)} \\Gs \\leq \\sum_{j=1}^m  (j-1) L_{\\sigma(j)} \\Gs \\leq Lm\\Gs\n\t\t\\label{def-M-gamma-bound}\n\t\\eeq\nwhere $L$ is the Lipschitz constant of the gradient of $f$ defined by \\eqref{def-L}. By replacing $\\mu(\\sigma)$ by $M_\\Gamma$ in Theorem \\ref{theo-rate-quadratics} one can get an upper bound on the worst-case convergence rate that applies to any choice of fixed order $\\sigma$. Using a similar argument along the lines of the proof of Theorem \\ref{theo-rate-quadratics} on the convergence rate of IG, it is straightforward to show that RR never performs any slower than this worst-case convergence rate which is the subject of the next result. The idea is to bound the stochastic $\\dist_k$ sequence from above point-wise. The proof is a simple exercise and is omitted due to space considerations. %left to the reader. % by a reasoning along the lines of the proof of Theorem \\ref{theo-rate-quadratics}, the details of the proof is skipped due to space considerations.\n","start":52449,"end":53872},{"raw_tex":"One way to get better constants is to consider convergence in expectation, a weaker notion of convergence compared to almost sure convergence. In the next theorem, we show that $M_\\Gamma$ can be improved to a typically much smaller constant $\\|\\bar{\\mu}\\|$ where \\beq \n\t\\bar{\\mu} := \\E \\big(\\mu(\\sigma_1)\\big) =\n\t\t\t\\frac{\\sum_{\\sigma \\in \\Gamma} \\mu(\\sigma)}{|\\Gamma|}. \\label{def-mu-bar}\n\\eeq can be thought as a measure of average performance over the choice of random permutations. \n","start":57279,"end":57765},{"raw_tex":"\\begin{remark} A consequence of Lemma \\ref{lem-computing-mu-bar} proved in the Appendix is that \\beq \n  \t\\bar{\\mu} =  \\frac{1}{2} \\sum_{i=1}^m P_i \\nabla f_i(x^*)\n\\label{mu-bar-alternative}.\n\\eeq\nwhere $\\bar{\\mu}$ is defined by \\eqref{def-mu-bar}. By the triangle inequality, \n\t$\\| \\bar{\\mu} \\| \\leq \\sum_{i=1}^m L_i \\Gs = L\\Gs $\nwhere $\\Gs$ is defined by \\eqref{def-Gs}. This upper bound is $m$ times smaller than the previous upper bound on $M_\\Gamma$ in \\eqref{def-M-gamma-bound}. \\rev{As an example, consider $s= 1$ with $Rc = 2$. In this case, $L = \\bigO(m)$, $c = \\bigO(m)$, $R = \\bigO(1/m)$ and $\\| \\bar{\\mu}\\| = \\bigO(m)$. We obtain from \\eqref{exp-dist-s-1} that $\\E \\left(\\dist_k\\right) = O(\\frac{1}{mk}) + o(1/k)$. Note that, the performance guarantee for IG from Corollary \\ref{coro-deter-decay-rate-dist} is $\\dist_k = \\bigO(1/k) + o(1/k)$ which is also worse than RR by a factor of $O(m)$. For a fair comparison with the SGD method, we consider running the SGD iterations for $j = km$ iterations so that both RR and SGD methods have access to the same number of component gradients. In this case, expected distance to suboptimality for SGD with the recommended $O(1/j)$ stepsize is $\\bigO(\\frac{1}{\\sqrt{j}})$ where the hidden constants are independent of $m$ (see e.g. \\cite{PolyakJuditsky92,Moulines:2011vy}) which is worse than the $\\bigO(1/j)$ guarantees for RR when $j$ is sufficiently large. These bounds show that when $m$ is small and $k$ is large, IG could outperform SGD in theory, however SGD (and RR) are more suitable for applications when $m$ is large and will admit better bounds compared to IG if $m$ is large enough for a given $k$ fixed\\footnote{We note however that SGD upper bounds are in expectation whereas IG results are deterministic which is a stronger notion of convergence.}}  %This requires $k$ cycles, i.e. $j=km$ evaluations of the gradients of the component functions. \n% \tThe quantity \n% \t$$ Q = L/c$$\n% is called the condition number and is a well-known measure of the difficulty of the optimization problem \\eqref{pbm-multi-agent}. \n\\end{remark}\n","start":58784,"end":60878},{"raw_tex":"The first lemma is on characterizing what is the worst-case distance of the all the inner iterates of RR to the optimal solution $x^*$. This quantity we want to upper bound is a random variable, but the upper bounds we obtain are deterministic holding for every sample path. This lemma is based on Corollary \\ref{coro-deter-decay-rate-dist} and uses the fact that the distance between the inner iterates are on the order of the stepsize.\n","start":114560,"end":114998}]},"statement":{"raw_tex":"\\begin{theorem}\\label{thm-averaged-rate-quad}  Let $f_i(x)$ be a quadratic function of the form\n\t$$f_i(x) = \\frac{1}{2} x_i^T P_i x - q_i^T x + r_i$$ \nwhere $P_i $ is a symmetric $n\\times n$ matrix, $q_i \\in \\R^n$ is a column vector and $r_i$ is a scalar for $i=1,2,\\dots,m$. Consider the $q$-suffix averages $\\xqk$ of the RR iterates generated by the iterations \\eqref{inner-update-wo} with stepsize $\\alpha_k = \\frac{R}{(k+1)^s}$ where $R>0$ and $s \\in (\\frac{1}{2},1)$. Suppose that Assumption \\ref{assump-sum-is-str-cvx} holds. Then the following statements are true:\n\\begin{enumerate}\n\\item [$(i)$] For any $0 < q \\leq 1$, the $q$-suffix averaged stepsize $\\aqk$ defined in \\eqref{def-averaged-stepsize} satisfies\n    \\beq\\aqk = \\frac{\\aqs}{k^s} + \\bigO(\\frac{1}{k}) \\quad \\mbox{where} \\quad \\aqs = \\frac{1-(1-q)^{1-s}}{q(1-s)} R.\\label{def-aqk}\n    \\eeq\n \\item [$(ii)$] For any $0 < q \\leq 1$, we have \n    \\beq \\lim_{k \\to \\infty} \\frac{\\bar{x}_{q,k} - x^*}{\\aqk} =  -H_*^{-1} \\bar{\\mu}\\quad a.s. \\label{limit-main-result}\n    \\eeq\nwhere $\\bar{\\mu}$ is given by \\eqref{mu-bar-alternative}, i.e., the normalized error $(\\bar{x}_{q,k} - x^*)/\\aqk$  converges to the constant vector $-H_*^{-1} \\bar{\\mu}$ almost surely where $\\Hs=\\sum_{i=1}^m P_i$ is the Hessian matrix at the optimal solution and $\\bar{\\mu}$ is given by \\eqref{mu-bar-alternative}.\n% \\beq      \n%     \\bar{\\mu} =  \\frac{1}{2} \\sum_{i=1}^m P_i \\nabla f_i(x^*). \n%    \\eeq\nThen, from part $(i)$,  \n \\beq \\lim_{k \\to \\infty} k^s (\\bar{x}_{q,k} - x^*) =  -\\aqs H_*^{-1} \\bar{\\mu} \\quad a.s.\\label{lim-rate-aver-quad}\n \\eeq\nHence, the $q$-suffix averaged iterates $\\xqk$ converge to the optimal solution $x^*$ with rate $1/k^s$ almost surely. \n%where $\\Hs=\\sum_{i=1}^m P_i$ is the Hessian matrix at the optimal solution, \n%\\beq      \n%    \\aqs =  \\frac{1-(1-q)^{1-s}}{q} R \\quad \\mbox{and} \\quad \\bar{\\mu} =  \\frac{1}{2} \\sum_{i=1}^m P_i \\nabla f_i(x^*). \n%   \\eeq\n   \n \\item [$(iii)$] With probability at least $1-\\delta$, we have\n\t $$ \\bar{x}_{q,k} -  x^*  =  b_{q,k} + \\bigO \\bigg(\\frac{\\sqrt{\\log(1/\\delta)}}{k} \\bigg) +  \\begin{cases} \\bigO \\big(\\frac{\\log k}{k} \\big)  & \\mbox{if} \\quad  q=1 \\\\ \\bigO(\\frac{1}{k}\\big) & \\mbox{if} \\quad  0<q<1, \\end{cases} $$ where\n\t \\beq b_{q,k} = -\\bar{\\alpha}_{q,k} H_*^{-1}  \\bar{\\mu} \n\t \t\t\\label{def-bqk}\n\t \\eeq \nis deterministic, $\\bar{\\mu}$ is given by $\\eqref{mu-bar-alternative}$ and $\\bar{\\alpha}_{q,k}$ is the averaged stepsize defined in \\eqref{def-averaged-stepsize}. The constants hidden by $\\bigO(\\cdot)$ depend only on $\\Gs, L, m, R, c, q$ and $s$.\n\t \\item [$(iv)$] Let \\beq  \\hat{b}_{q,k} = -\\bar{\\alpha}_{q,k}\\bigg[\\sum_{i=1}^{m} P_{\\ski}\\bigg]^{-1} \\sum_{i=1}^{m} P_{\\ski} \\nabla f_{\\ski} (x_{i-1}^k)/2. \\label{def-hat-bqk}\n\t \\eeq\nwhere $\\bar{\\alpha}_{q,k}$ is the averaged stepsize defined in \\eqref{def-averaged-stepsize}. Then, \t \n\t $\\hat{b}_{q,k}  = b_{q,k} + \\bigO(\\alpha_k^2). $\nIt follows from part $(ii)$ that with probability at least $1-\\delta$,\n\t  $$ (\\bar{x}_{q,k} - \\hat{b}_{q,k}) -  x^*  = \\bigO \\bigg(\\frac{\\sqrt{\\log(1/\\delta)}}{k} \\bigg) +  \\begin{cases} \\bigO \\big(\\frac{\\log k}{k} \\big)  & \\mbox{if} \\quad  q=1 \\\\ \\bigO(\\frac{1}{k}\\big) & \\mbox{if} \\quad  0<q<1. \\end{cases} $$ \n\\end{enumerate} \n \n \\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\\label{thm-averaged-rate-quad}  Let $f_i(x)$ be a quadratic function of the form\n\t$$f_i(x) = \\frac{1}{2} x_i^T P_i x - q_i^T x + r_i$$ \nwhere $P_i $ is a symmetric $n\\times n$ matrix, $q_i \\in \\R^n$ is a column vector and $r_i$ is a scalar for $i=1,2,\\dots,m$. Consider the $q$-suffix averages $\\xqk$ of the RR iterates generated by the iterations \\eqref{inner-update-wo} with stepsize $\\alpha_k = \\frac{R}{(k+1)^s}$ where $R>0$ and $s \\in (\\frac{1}{2},1)$. Suppose that Assumption \\ref{assump-sum-is-str-cvx} holds. Then the following statements are true:\n\\begin{enumerate}\n\\item [$(i)$] For any $0 < q \\leq 1$, the $q$-suffix averaged stepsize $\\aqk$ defined in \\eqref{def-averaged-stepsize} satisfies\n    \\beq\\aqk = \\frac{\\aqs}{k^s} + \\bigO(\\frac{1}{k}) \\quad \\mbox{where} \\quad \\aqs = \\frac{1-(1-q)^{1-s}}{q(1-s)} R.\\label{def-aqk}\n    \\eeq\n \\item [$(ii)$] For any $0 < q \\leq 1$, we have \n    \\beq \\lim_{k \\to \\infty} \\frac{\\bar{x}_{q,k} - x^*}{\\aqk} =  -H_*^{-1} \\bar{\\mu}\\quad a.s. \\label{limit-main-result}\n    \\eeq\nwhere $\\bar{\\mu}$ is given by \\eqref{mu-bar-alternative}, i.e., the normalized error $(\\bar{x}_{q,k} - x^*)/\\aqk$  converges to the constant vector $-H_*^{-1} \\bar{\\mu}$ almost surely where $\\Hs=\\sum_{i=1}^m P_i$ is the Hessian matrix at the optimal solution and $\\bar{\\mu}$ is given by \\eqref{mu-bar-alternative}.\n% \\beq      \n%     \\bar{\\mu} =  \\frac{1}{2} \\sum_{i=1}^m P_i \\nabla f_i(x^*). \n%    \\eeq\nThen, from part $(i)$,  \n \\beq \\lim_{k \\to \\infty} k^s (\\bar{x}_{q,k} - x^*) =  -\\aqs H_*^{-1} \\bar{\\mu} \\quad a.s.\\label{lim-rate-aver-quad}\n \\eeq\nHence, the $q$-suffix averaged iterates $\\xqk$ converge to the optimal solution $x^*$ with rate $1/k^s$ almost surely. \n%where $\\Hs=\\sum_{i=1}^m P_i$ is the Hessian matrix at the optimal solution, \n%\\beq      \n%    \\aqs =  \\frac{1-(1-q)^{1-s}}{q} R \\quad \\mbox{and} \\quad \\bar{\\mu} =  \\frac{1}{2} \\sum_{i=1}^m P_i \\nabla f_i(x^*). \n%   \\eeq\n   \n \\item [$(iii)$] With probability at least $1-\\delta$, we have\n\t $$ \\bar{x}_{q,k} -  x^*  =  b_{q,k} + \\bigO \\bigg(\\frac{\\sqrt{\\log(1/\\delta)}}{k} \\bigg) +  \\begin{cases} \\bigO \\big(\\frac{\\log k}{k} \\big)  & \\mbox{if} \\quad  q=1 \\\\ \\bigO(\\frac{1}{k}\\big) & \\mbox{if} \\quad  0<q<1, \\end{cases} $$ where\n\t \\beq b_{q,k} = -\\bar{\\alpha}_{q,k} H_*^{-1}  \\bar{\\mu} \n\t \t\t\\label{def-bqk}\n\t \\eeq \nis deterministic, $\\bar{\\mu}$ is given by $\\eqref{mu-bar-alternative}$ and $\\bar{\\alpha}_{q,k}$ is the averaged stepsize defined in \\eqref{def-averaged-stepsize}. The constants hidden by $\\bigO(\\cdot)$ depend only on $\\Gs, L, m, R, c, q$ and $s$.\n\t \\item [$(iv)$] Let \\beq  \\hat{b}_{q,k} = -\\bar{\\alpha}_{q,k}\\bigg[\\sum_{i=1}^{m} P_{\\ski}\\bigg]^{-1} \\sum_{i=1}^{m} P_{\\ski} \\nabla f_{\\ski} (x_{i-1}^k)/2. \\label{def-hat-bqk}\n\t \\eeq\nwhere $\\bar{\\alpha}_{q,k}$ is the averaged stepsize defined in \\eqref{def-averaged-stepsize}. Then, \t \n\t $\\hat{b}_{q,k}  = b_{q,k} + \\bigO(\\alpha_k^2). $\nIt follows from part $(ii)$ that with probability at least $1-\\delta$,\n\t  $$ (\\bar{x}_{q,k} - \\hat{b}_{q,k}) -  x^*  = \\bigO \\bigg(\\frac{\\sqrt{\\log(1/\\delta)}}{k} \\bigg) +  \\begin{cases} \\bigO \\big(\\frac{\\log k}{k} \\big)  & \\mbox{if} \\quad  q=1 \\\\ \\bigO(\\frac{1}{k}\\big) & \\mbox{if} \\quad  0<q<1. \\end{cases} $$ \n\\end{enumerate} \n \n \\end{theorem}","start":65780,"end":69036}]},"proof":{"raw_tex":"\\begin{corollary}\\label{coro-deter-decay-rate-dist} Under the setting of Theorem \\ref{theo-rate-quadratics}, if $\\sigma$ is sampled uniformly at each cycle instead of being kept fixed, then\n\t\t% \\beqs \\  \\dist_k \\leq \\frac{R M_\\Gamma}{c}\\frac{1}{k^s} + o(\\frac{1}{k^s})%, \\quad M_\\Gamma = \\sup_{\\sigma \\in \\Gamma} \\mu(\\sigma)\n  %        \\eeqs \n         \\begin{align} \\dist_k &\\leq \\frac{R M_\\Gamma }{c}\\frac{1}{k^s} + o(\\frac{1}{k^s})  &\\mbox{if}& \\quad 1/2<s<1,\\\\\n           \t\t\t  \\dist_k  &\\leq \\frac{R^2 M_\\Gamma}{ Rc - 1} \\frac{1}{k} + o(\\frac{1}{k})  &\\mbox{if}& \\quad s=1 \\mbox{ and } Rc>1,\n\t  %\\label{def-Msigma}% \\leq Lm \\Gs\n    \t\\end{align}\nwith probability one where $M_\\Gamma$ is deterministic and is defined by \\eqref{def-M-gamma}.\n%the right-hand side is a deterministic sequence and         \n%  \t    $$M_\\Gamma \\leq Lm \\Gs, \\quad \\Gs = \\underset{1\\leq i \\leq m}{\\sup} \\| \\nabla f_i(x^*)\\|.$\n\\end{corollary}\n\n\\begin{proof} \n\t\\begin{itemize} \n\t\\item [$(i)$] As the stepsize sequence is monotonically decreasing, we have the bounds \n   \\beqas \\int_{(1-q)k}^k  \\frac{R}{(x+2)^s} dx \\leq  \\sum_{j=(1-q)k}^k \\alpha_j = \\sum_{j=(1-q)k}^k \\frac{R}{(k+1)^s} %&\\leq& R + \\sum_{j=(1-q)k + 1}^k \\frac{R}{(k+1)^s} \\\\\n   \t\t\t &\\leq& R + \\int_{(1-q)k}^{k-1}  \\frac{R}{(x+1)^s} dx. \n   %1 + \\int_{(1-q)k}^k  \\frac{R}{(x+1)^s} dx. \\\\\n   \\eeqas\nDividing each term by $qk$, after a straightforward integration we obtain\n  $$ \\aqk = \\frac{ k^{1-s} - \\big((1-q)k+1\\big)^{1-s} + \\bigO(1)}{(1-s)qk} R  = \\frac{a_q(s)}{k^s} + \\bigO(\\frac{1}{k}). \n   $$\n which completes the proof.  \n \t\t\t\\item [$(ii)$] Taking the $q$-suffix averages of both sides of \\eqref{averaged-iter-randomized}, we obtain \n\\beq \n\tI_{q,k} := \\frac{\\sum_{j=(1-q)k}^{k-1} {(x_0^j - x_{0}^{j+1})}{\\alpha_j^{-1}}}{qk} = \\frac{ \\sum_{j=(1-q)k}^{k-1}   \\nabla f(x_0^j)+ E_j}{qk}. \\label{averaged-iter-random-2}\n   \\eeq \nAs $f$ is a quadratic, the first order Taylor series for the gradient of $f$ is exact:\n  \\beq \\nabla f(x_0^j) = H_* (x_0^j-x^*).\n  \t\t\\label{grad-as-hessian-prod-quadratics}\n  \\eeq\n Therefore, \\eqref{averaged-iter-random-2} becomes  $\nI_{q,k} \n= \\frac{ \\sum_{j=(1-q)k}^{k-1}   H_* (x_0^j - x^*)   + E_j }{qk} $\n%\\label{averaged-iter-random-3} \nwhich is equivalent to\n\\beqa\n                    I_{q,k} &=& H_*(\\bar{x}_{q,k} - x^*) + \\frac{\\sum_{j=(1-q)k}^{k-1} E_j}{qk} = H_*(\\bar{x}_{q,k} - x^*) + \\bar{\\alpha}_{q,k}  Y_{q,k}\\label{iterate-eq-randomized}\n   \\eeqa \nwhere $Y_{q,k}$ is defined as\n    \\beq Y_{q,k} := \\frac{1}{\\bar{\\alpha}_{q,k}} \\frac{\\sum_{j=(1-q)k}^{k-1} E_j }{qk } =  \\frac{\\sum_{j=(1-q)k}^{k-1} E_j}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}\t\n\t\t\\label{def-Yqk}    \n     \\eeq\nand can be interpreted as the ($q$-suffix) averaged gradient error sequence $E_j$  normalized by the ($q$-suffix) averaged stepsize sequence $\\alpha_j$. Since $H_*$ is invertible by the strong convexity of $f$ (see \\eqref{def-Hs}), we can rewrite \\eqref{iterate-eq-randomized} as\n\\beqa  \\bar{x}_{q,k} - x^* &=& \n                     - H_*^{-1} \\aqk \\Yqk  + H_*^{-1} \\Iqk  \n            = - H_*^{-1} \\bar{\\alpha}_{q,k} Y_{q,k}  +  \\begin{cases} \\bigO(\\frac{1}{k})  & \\mbox{if } \\quad 0<q<1 \\\\\n\\bigO( \\frac{\\log k}{k})  & \\mbox{if } \\quad q = 1.\n\\end{cases} \\label{xk-averaged-asymptotics-2}\n              %           &=& \\bigO(\\frac{\\log k}{k}) + \\bigO(\\frac{1}{k}) - H_*^{-1} \\bigg(   \\frac{\\sum_{j=0}^{k-1} E_j}{\\sum_{j=0}^{k-1}\\alpha_j}\\bigg) \\frac{\\sum_{j=0}^{k-1}\\alpha_j}{k}   \\nonumber \n   \\eeqa              \nwhere we used the inequality $\\| H_*^{-1}\\| \\leq 1/c $ implied by \\eqref{def-Hs} and Lemma \\ref{lem-I-lk} from the appendix to provide an upper bound for the second term in the first equality. Note that, as a consequence of Lemma \\ref{lem-I-lk}, $\\bigO(\\cdot)$ notation above hides a constant that depends only on the parameters $\\Gs, L,c, m, R, s, q$ and also $\\dist_0$ when $q=1$. Then, dividing both sides of \\eqref{xk-averaged-asymptotics-2} by $\\aqk$, taking limits as $k$ goes to infinity,  using part $(i)$ on the asymptotic behavior of $\\aqk$ and the fact that $Y_{q,k} \\to \\bar{\\mu}$ a.s. from Lemma \\ref{lem-martingale-variance}, we obtain the claimed result.%$$ \\lim_{k \\to \\infty} k^s (\\bar{x}_{q,k} - x^*) = a_q(s) H_*^{-1} \\bar{\\mu}. $$\n%This completes the proof of part $(i)$.\n%Taking expectations, \n%\n%$$ \\E (\\bar{x}_k - x^*)  = \\bigO(\\frac{\\log k}{k}) -  H_*^{-1} \\bigg(   \\frac{\\sum_{j=0}^{k-1} \\alpha_j \\E v(u^j) }{k}\\bigg) + \\bigO(\\alpha_k^2).  $$\n%\n%\n%which implies\n%$$ \\E (\\bar{x}_k - x^*)  = \\bigO(\\frac{\\log k}{k}) -  b_* \\bigg(   \\frac{\\sum_{j=0}^{k-1} \\alpha_j  }{k}\\bigg) + \\bigO(\\alpha_k^2).  $$ \n%$ b_* = H_*^{-1}\\frac{\\sum_{r=1}^m \\nabla^2 f_r(x^*) \\nabla f_r(x^*)}{2}.$\n%Suppose we can compute an approximation $b_k$ to $b_*$ satisfying\n%   $b_k = b_* + \\bigO(\\alpha_k).$\n%Then,    \n%$$ \\E \\bigg(\\bar{x}_k + b_k \\bigg(   \\frac{\\sum_{j=0}^{k-1} \\alpha_j  }{k}\\bigg)  - x^*\\bigg)  = \\bigO(\\frac{\\log k}{k})  + \\bigO(\\alpha_k^2).  $$ \n%\n%NEED TO LOOK AT THE VARIANCE TERM AS WELL. USE OLD MARTINGALE APPROACH FOR BOUNDING THE VA\n\n \\item [$(iii)$] By parts $(i)$ and $(iii)$ of Lemma \\ref{lem-martingale-variance} from the appendix that relates the gradient error sequence $E_j$ to a sequence of i.i.d. variables $\\mu(\\sigma_j)$, for $0<q\\leq 1$,  \n\t\t\\beq Y_{q,k} = \\frac{\\sum_{j=(1-q)k}^{k-1} E_j}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j} =  \\frac{\\sum_{i=(1-q)k}^{k-1} \\alpha_j \\mu(\\sigma_j) + \\bigO(\\alpha_j^2)}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}. \t \n\t\t\t\\label{Yqk-helper}  \n     \\eeq \nWe first give a proof for $q=1$, the proof for the remaining $q \\in (0,1)$ case will be similar. Assume $q=1$. Plugging $q=1$ and \\eqref{Yqk-helper} into \\eqref{xk-averaged-asymptotics-2}, we obtain\n     \\beqa \\bar{x}_{1,k} - x^* \n     &=&\\bigO(\\frac{\\log k}{k})  - H_*^{-1} \\aonek \\Yonek \\nonumber \\\\\n     &=& \\bigO(\\frac{\\log k}{k}) - H_*^{-1} \\bigg(   \\frac{\\sum_{j=0}^{k-1} \\alpha_j \\big(\\mu(\\s_j) - \\bar{\\mu}\\big)}{k} +  \\frac{\\sum_{j=0}^{k-1}  \\alpha_j \\bar{\\mu} + \\bigO(\\alpha_j^2) }{k}  \\bigg) \\nonumber \\\\\n     \t\t\t\t\t\t &=& b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{\\sum_{j=0}^{k-1}  \\alpha_j (\\mu(\\s_j) - \\bar{\\mu})}{k} - H_*^{-1} \\sum_{j=0}^{k-1}  \\frac{\\bigO(\\alpha_j^2)}{k} \\nonumber \\\\     \n     \t\t\t\t\t\t    &=& b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{\\sum_{j=0}^{k-1}  \\alpha_j (\\mu(\\s_j) - \\bar{\\mu})}{k} \\label{xqk-high-proba} %\\\\     \t\t\t\t\t\t    \n     \t\t\t\t\t\t%    &=& b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{M_{1,k}}{k}  \\label{xk-biased-high-probability}\n     \\eeqa    \nwhere $\\bonek$ is defined by \\eqref{def-bqk} and we used in the last step the fact that for $s>1/2$ \n\t\\beq \\sum_{j=0}^\\infty \\alpha_j^2 = \\sum_{j=1}^\\infty \\frac{R^2}{j^{2s}} = R^2 \\zeta(2s) < \\infty\n\t\t\\label{square-summable-step}\t\n\t\\eeq \nwhere $\\zeta(\\cdot)$ is the Riemann-Zeta function. We now study the asymptotic behavior of the last summation term in \\eqref{xqk-high-proba} by introducing the process\n\t$S_{1,k} = \\sum_{j=0}^{k-1} Z_j$, where $Z_j := \\alpha_j (\\mu(\\s_j) - \\bar{\\mu})$ and $k\\geq 0$ with the convention that $S_{1,0} = 0$. Equipped with this definition, \\eqref{xqk-high-proba} becomes\n  \\beq \n  \t\t \\xonek - \\xs = b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{S_{1,k}}{k}.  \\label{xk-biased-high-probability}\n  \\eeq\t\t  \nThe random variables $Z_j$ are independent, centered and have an identical distribution up to the scaling factor $\\alpha_j$. Therefore,  $S_{1,k}$ is a sum of  centered random variables satisfying:\n    \\beqa \\begin{scriptsize} \\| S_{1,k} - S_{1,k-1} \\| =\\big \\|\\alpha_{k-1}\\big(\\mu(\\sigma_{k-1})-\\bar{\\mu}\\big)\\big\\| \n    \\leq \\gamma_{k-1}:=\\alpha_{k-1}LmG_* \\label{marting-diff-as-bound}\n    \\end{scriptsize}\n    \\eeqa \nwhere we used \\eqref{def-mu-sk-upper-bound} in the last inequality (see also Lemma \\ref{lem-computing-mu-bar}). Then, by the Azuma-Hoeffding inequality, for every $t>0$, \n  \\beqas \\P \\bigg( \\big\\|\\frac{S_{1,k}}{k}\\big\\| > \\frac{t}{k}\\bigg) &\\leq& 2 \\exp \\bigg( - \\frac{t^2}{2\\sum_{j=0}^{k-1} \\gamma_j^2} \\bigg) =  2 \\exp \\big( - \\frac{t^2}{ \\beta }\\big) \n  \\eeqas\nwhere $\\beta = 2\\sum_{j=0}^{\\infty} \\gamma_j^2 < \\infty$ as $\\alpha_j$ is square-summable (see \\eqref{square-summable-step}). Note that $\\beta$ depends only on $\\Gs, L,m$ and the stepsize parameters $R$ and $s$. It is easy to see that selecting $t \\geq t_\\delta = \\sqrt{\\beta  \\log(2/\\delta)}$ makes the right-hand side $\\leq \\delta$.  Therefore for any $\\delta>0$, with probability at least $1-\\delta$, \n\t\t\\beq \\bigl \\| \\frac{S_{1,k}}{k} \\bigr \\| \\leq \\frac{\\sqrt{\\beta  \\log(2/\\delta)}}{k}\n\t\t\t\\label{average-sum-bound-whp}\n\t\t\\eeq\nwhich if inserted into the expression \\eqref{xk-biased-high-probability} completes the proof for the $q=1$ case. For  $0<q<1$ case, the same line of reasoning applies except that we replace $b_{1,k}$ with $b_{q,k}$ and we can improve the $\\bigO( \\log k /k)$ term in the expression \\eqref{xk-biased-high-probability} to $\\bigO(1/k)$, this is justified by \\eqref{xk-averaged-asymptotics-2}. Then, this leads to \n\t\t \\beq \n  \t\t \\xqk - \\xs = b_{q,k}  +  \\bigO(\\frac{1}{k})  - H_*^{-1}   \\frac{S_{q,k}}{qk}  \\label{xk-biased-high-probability-3}\n  \\eeq\nwhere $S_{q,k} := \\sum_{j=(1-q)k}^{k-1} Z_j = S_{1,k} - S_{1,(1-q)k}$ is the $q$-suffix cumulative sum (cumulative sum of the last $qk$ terms) of the sequence $Z_{k}$. Then using \\eqref{average-sum-bound-whp}, with probability at least $1-\\delta$,\n\\beq \\bigl \\| \\frac{S_{q,k}}{k} \\bigr \\| \\leq \\| \\frac{S_{1,k}}{k} \\bigr \\|  + \\| \\frac{S_{1,(1-q)k}}{k} \\bigr \\|  \\leq \\frac{2 t_\\delta}{k}.\n\t\t\t\\label{average-sum-bound-whp-3}\n\t\t\\eeq\nPlugging this high probability bound into \\eqref{xk-biased-high-probability-3}, we conclude. \t\t\n\t\\item [$(iv)$] By Lemma \\ref{lem-random-tail-bound}, we have $\\underset{1\\leq i < m}{\\max} \\|x_{i-1}^k - x^*\\| = \\bigO(\\alpha_k)$. Therefore,\n\t \\beqa \\scriptsize \\| \\nabla f_{\\ski} (x_{i-1}^k)  - \\nabla f_{\\ski} (x^*) \\|  = \\bigO(\\alpha_k) \\label{grad-diff-quadratic}\n\t \\eeqa\nfor any $i=1,2,\\dots,m$. As a consequence, \t\n\t\\beqas \\hat{b}_{q,k} &=& -\\bar{\\alpha}_{q,k} H_*^{-1} \\sum_{i=1}^{m} P_{\\ski}  \\bigg(\\nabla f_{\\ski} (x^*) + \\bigO(\\alpha_k)\\bigg)  \\\\\n\t                         &=& - \\bar{\\alpha}_{q,k} H_*^{-1} \\sum_{j=1}^{m} P_{j} \\nabla f_j (x^*)   + \\bigO(\\alpha_k^2)   \n%\t                         &=& -\\bigg( H_*^{-1} + \\bigO(\\alpha_k) \\bigg) \\sum_{i=1}^{m} \\nabla^2 f_{i} (x_*) \\nabla f_i (x^*)  \\bar{\\alpha}_{q,k}+ \\bigO(\\alpha_k^2) +  o(\\alpha_k^2) \\\\\n\t                         = b_{q,k} + \\bigO(\\alpha_k^2)\t                         \t                        \t                         \n\t\\eeqas\t\t \nwhere in the second equality we use the fact that $\\bar{\\alpha}_{q,k}= \\bigO(1/k^s) = \\bigO(\\alpha_k)$ implied by part $(i)$.\n%\t\t\\item [$(ii)$]  By taking norm square of both sides and expectations in \\eqref{xk-averaged-asymptotics} and applying Cauchy-Schwartz gives\n%\t\t\t\\beqas \\E \\| \\bar{x}_k - x^*\\|^2 &\\leq& 4\\bigO(\\frac{\\log k}{k}) + 4\\bigO(\\frac{1}{k}) + 4 \\E \\bigg\\|  H_*^{-1} \\bigg(   \\frac{\\sum_{j=0}^{k-1} E_j}{k}\\bigg)\\bigg\\|^2 + 4\\bigO(\\alpha_k^2)   \\\\\n%\t\t\t                                              &\\leq& 4\\bigO(\\frac{\\log k}{k}) + 4\\bigO(\\frac{1}{k}) + \\frac{4}{c^2} \\E \\bigg\\|   \\frac{\\sum_{j=0}^{k-1} E_j}{k}\\bigg\\|^2 + 4\\bigO(\\alpha_k^2) \n%\t\t\t\\eeqas\n%We then decompose the term inside the expectation as\n%    \\beqas \\frac{\\sum_{j=0}^{k-1} E_j}{k} &=& \\frac{\\sum_{j=0}^{k-1} \\alpha_j v(u^j) - \\alpha_j \\bar{\\mu}}{k} + \\frac{\\sum_{j=0}^{k-1} \\alpha_j \\bar{\\mu} + \\bigO(\\alpha_j^2)}{k} \\\\\n%               &=& \\frac{\\sum_{j=0}^{k-1} \\alpha_j v(u^j) - \\alpha_j \\bar{\\mu}}{\\sigma_* (\\sum_{j=0}^{k-1}\\alpha_j^2)^{1/2}} \\frac{\\sigma_* (\\sum_{j=0}^{k-1}\\alpha_j^2)^{1/2}}{k} + \\frac{\\sum_{j=0}^{k-1} \\alpha_j \\bar{\\mu} + \\bigO(\\alpha_j^2)}{k}\n%    \\eeqas\n\\end{itemize}\n\\end{proof}\n\n\\begin{lemma}\\label{lem-random-tail-bound}Under the conditions of Theorem \\ref{thm-averaged-rate-quad} we have \n    $\\underset{0 \\leq i < m}{\\max} \\| x_i^k - x^*\\| =  \\bigO(\\frac{1}{k^s}).$\nwhere $\\bigO(\\cdot)$ hides a constant that depends only on $\\Gs, L, m,c$ and $R$. \n\\end{lemma}\n\n\\begin{proof} By Corollary \\ref{coro-deter-decay-rate-dist}, \n   \\beq \\| x_0^k - x^* \\|  = \\bigO(\\frac{1}{k^s}).\n   \t\t\\label{upper-bound-cycle-begin}\n   \\eeq  \nwhere $\\bigO(\\cdot)$ hides a constant that depends only on $\\Gs, L, m,R$ and $c$.  We have also for any $0\\leq i< m$ and $k\\geq 0$,\n   \\beqas \\| x_i^k - x^*\\|  &\\leq& \\| x_0^k - x^*\\| + \\| x_i^k - x_0^k \\|  = \\| x_0^k - x^*\\| + i\\alpha_k \\max_{\\ell=1,\\dots,i} \\|\\nabla f_{\\sigma_k(\\ell)} (x_{\\ell-1}^k)\\| \\\\\n   &\\leq& \\| x_0^k - x^*\\| + (m-1)\\frac{R}{(k+1)^s} \\big(  G_* + \\max_{\\ell=1,\\dots,i} \\|\\nabla f_{\\sigma_k(\\ell)} (x_{\\ell-1}^k) - \\nabla f_{\\sigma_k(\\ell)} (x^*)\\| \\big) \\\\\n   &\\leq& \\| x_0^k - x^*\\| + (m-1)\\frac{R}{(k+1)^s} \\big(  G_* + L \\max_{\\ell=1,\\dots,i} \\|x_{\\ell-1}^k - x^*\\| \\big). \n   \\eeqas\nwhere we used the $L$-Lipschitzness of the gradient of $f$ where $L$ is given by \\ref{def-L}. Using \\eqref{upper-bound-cycle-begin} and applying this inequality inductively for $i=0,1,2,\\dots, m-1$ we conclude.\n\\end{proof}\n\n\\begin{lemma}\\label{lem-I-lk} Under the conditions of Theorem \\ref{thm-averaged-rate-quad}, consider the sequence\n\\beq I_{q,k} = \\frac{\\sum_{j=(1-q)k}^{k-1} {(x_0^j - x_{0}^{j+1})}{\\alpha_j^{-1}}}{qk}, \\quad  0 < q \\leq 1. \\label{def-I-qk}\n\\eeq\nThen, \n    $ I_{q, k} = \\begin{cases}  \\bigO \\big( \\frac{\\log k}{k}  \\big) \n     &  \\mbox{if} \\quad q=1, \\\\  \n    \t\t\t\\bigO \\big( \\frac{1}{k}\\big) \t & \\mbox{if} \\quad 0<q<1.  \n    \\end{cases}  \n%    &\\leq& \\bigO \\big( \\frac{\\log k}{k}\\big) \\\\\n%               I_{qk, k} &\\leq& \\bigO \\big( \\frac{1}{k}\\big) \\quad \\mbox{for all} \\quad 0<q<1.\n   $\nIn the former case, $\\bigO(\\cdot)$ hides a constant that depends only on $\\Gs, L,m,  c, R, s,q$ and $\\dist_0$. In the latter case, the same dependency on the constants occurs except that the dependency on $\\dist_0$ can be removed. \n\\end{lemma}\n\n\\begin{proof}  It follows from integration by parts that for any $\\ell<k$, \n\\beq -\\sum_{j=\\ell}^{k-1}{(x_0^j - x_{0}^{j+1})}{\\alpha_j^{-1}}  =  \\alpha_k^{-1} (x_0^k - x^*) - \\alpha_\\ell^{-1} (x_0^\\ell - x^*) - \\sum_{j=\\ell}^{k-1} (x_0^{j+1} - x^*)(\\alpha_{j+1}^{-1}-\\alpha_j^{-1}). \\label{sum-of-aver-iters}\n\\eeq\nNext, we investigate the asymptotic behavior of the terms on the right-hand side. A consequence of Corollary \\ref{coro-deter-decay-rate-dist} and the inequality \\ref{def-M-gamma} is that  \n\t\\beq \\alpha_k^{-1} \\| x_0^k -x^*\\| = \\frac{(k+1)^s}{R} \\| x_0^k -x^*\\| \\leq \\frac{LmG_*}{c} + o(1) = \\bigO(1)\n\t\t\\label{step-times-dist-bd} \t\n\t\\eeq\nand therefore\n\\beqas \n\t| \\alpha_{k+1}^{-1} - \\alpha_{k}^{-1} |  \\| x_0^k -x^*\\| &=& \\frac{(k+2)^s - (k+1)^s} {(k+1)^s} \\alpha_k^{-1}\\| x_0^k -x^*\\|  \n\t = \\bigg( \\big(1 + \\frac{1}{k+1} \\big)^s -1 \\bigg) \\alpha_k^{-1}\\| x_0^k -x^*\\| \\\\\n\t  &\\leq &  \\frac{s}{k+1}\\alpha_k^{-1}\\| x_0^k -x^*\\|  \n\t  \\leq \\frac{sLmG_*}{c} \\frac{1}{k+1} + o(\\frac{1}{k+1}) = \\bigO(\\frac{1}{k+1})\n\\eeqas\nwhere $\\bigO(\\cdot)$ hides a constant that depends only on $L,\\Gs, c, m$ and $s$. Then, setting  $\\ell = (1-q)k$ in \\eqref{sum-of-aver-iters}, it follows that \n\\beqa  \\big\\| \\sum_{j=\\ell}^{k-1} {(x_0^j - x_0^{j+1})}{\\alpha_j^{-1}} \\big\\| &\\leq&  \\|\\alpha_k^{-1} (x_0^k - x^*)\\| + \\| \\alpha_{(1-q)k}^{-1} (x_0^{(1-q)k}- x^*)\\| \\\\\n  && + \\sum_{j=(1-q)k}^{k-1} \\| x_0^{j+1} - x^*\\| |\\alpha_{j+1}^{-1}-\\alpha_j^{-1}|.  \\nonumber\\\\\n       &=& \\bigO(1) + \\| \\alpha_{(1-q)k}^{-1} (x_0^{(1-q)k} - x^*)\\| + \\bigO \\bigg( \\sum_{j=(1-q)k}^{k-1} \\frac{1}{j+1}\\bigg).  \\label{weighted-iter-diff-bound} %\\\\\n       %&& + o \\bigg( \\sum_{j=\\ell}^{k-1} \\frac{1}{(j)^s} \\|(j+1)^s - j^s) \\|\\bigg). \\nonumber \\\\\n       %&=& \\bigO(1)  + \\| \\alpha_\\ell^{-1} (x_0^\\ell - x^*)\\| + \\bigO \\bigg( \\sum_{j=\\ell}^{k-1} \\big((1+\\frac{1}{j+1})^s - 1\\big) \\bigg) \\nonumber \\\\\n       %&& + o \\bigg( \\sum_{j=\\ell}^{k-1} \\big((1+\\frac{1}{j})^s - 1\\big) \\bigg). \\nonumber \\\\\n     % &=& \\bigO(1) + \\| \\alpha_\\ell^{-1} (x_0^\\ell - x^*)\\| + \\bigO \\bigg( \\sum_{j=\\ell}^{k-1} \\frac{s}{j+1} \\bigg) \n       %\\\\\n      %&\\leq& \\bigO(1) + \\bigO(\\frac{1}{k^{2s}} ) + \\bigO \\bigg( \\sum_{j=\\max(\\ell,1)}^{k-1} \\frac{s}{j} \\bigg) \\label{weighted-iter-diff-bound}\n\\eeqa\nWe also have\n  \\beq  \\| \\alpha_{(1-q)k}^{-1} (x_0^{(1-q)k} - x^*)\\|  = \\begin{cases} \\alpha_0^{-1} \\dist_0 & \\mbox{if} \\quad q = 1, \\\\ \\bigO(1)  & \\mbox{if} \\quad 0<q<1, \\end{cases} \\label{init-averaged-iter-bound}\n  \\eeq\nwhere the second part follows from \\eqref{step-times-dist-bd} with similar constants for the $\\bigO(\\cdot)$ term. As the sequence $\\frac{1}{j+1}$ is monotonically decreasing, for any $k>0$ we have the bounds \n\\beq  \\sum_{j=(1-q)k}^{k-1} \\frac{1}{j+1} \\leq \\frac{1}{(1-q)k + 1} + \\int_{(1-q)k}^{k-1} \\frac{1}{x+1} dx \\leq \\begin{cases} 1 + \\log k & \\mbox{if } \\quad  q=1, \\\\\n 1 + \\log(\\frac{1}{1-q})  & \\mbox{if } \\quad  0 <q <1.\n\\end{cases}  \\label{tail-suffix-averaging}\n\\eeq\nNote that when $q=1$ this bound grows with $k$ logarithmically whereas for $q<1$ it does not grow with $k$. \nThen, combining \\eqref{weighted-iter-diff-bound}, \\eqref{init-averaged-iter-bound} and \\eqref{tail-suffix-averaging}  we obtain\n  $$ \\| \\Iqk \\| \\leq \\frac{\\big\\| \\sum_{j=\\ell}^{k-1} {(x_0^j - x_0^{j+1})}{\\alpha_j^{-1}} \\big\\|}{qk} = \\begin{cases}  \\bigO \\big(  \\frac{\\log k}{k}\\big) \n     &  \\mbox{if} \\quad q=1 \\\\  \n    \t\t\t\\bigO \\big( \\frac{1}{k}\\big) \t & \\mbox{if} \\quad 0<q<1  \n    \\end{cases}  $$\nas desired which completes the proof.    \n\\end{proof}\n\n\\begin{lemma}\\label{lem-computing-mu-bar} Let $\\sigma$ be a random permutation of $\\{1,2,\\dots,m\\}$ sampled uniformly over the set of all permutations $\\Gamma$ defined by \\eqref{def-Gamma-perm-set} and $\\mu(\\sigma)$ be the vector defined by \\eqref{def-Msigma} that depends on $\\sigma$. Then, \\beq\n\t\\bar{\\mu} = \\E_{\\sigma}\\big(\\mu({\\sigma})\\big) = \\frac{1}{2} \\sum_{i=1}^m P_i \\nabla f_i(x^*)\n\\eeq\nwhere $\\E_{\\sigma}$ denotes the expectation with respect to the random permutation $\\sigma$ and $\\bar{mu}$ is defined by \\eqref{def-mu-bar}.\n\\end{lemma}\n\n\\begin{proof} For any $i \\neq \\ell$, the joint distribution of $(\\sigma(i), \\sigma(\\ell))$ is uniform over the set of all (ordered) pairs from $\\{1,2,\\dots,m\\}$. Therefore, for any $i \\neq \\ell$, \n    \\beqas\n    \\E_\\sigma \\big[ P_{\\sigma(i)} \\nabla f_{\\sigma(\\ell) }(x^*) \\big] &=& \\sum_{i=1}^m \\sum_{i\\neq j, j=1}^m \\frac{P_i\\nabla f_j(x^*)}{m(m-1)} \\\\\n    &=& \\frac{\\sum_{i=1}^m P_i  \\sum_{j=1}^m \\nabla f_j(x^*) - \\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)} \n    =  - \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)}\n    \\eeqas \nwhere we used the fact that $\\nabla f(x^*) = \\sum_{j=1}^m \\nabla f_j(x^*)=0$ by the first order optimality condition. Then, by taking the expectation of \\eqref{def-v-sigma-k}, we obtain \n\\beqas \\E_\\sigma (\\mu(\\sigma)) &=& - \\sum_{i=1}^{m} \\sum_{\\ell=0}^{i-1} \\E \\big[ P_{\\sigma(i)} \\nabla f_{\\sigma(\\ell)}(x^*) \\big] \n        =  \\sum_{i=1}^{m} \\sum_{\\ell=0}^{i-1}  \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)} = \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{2}.\n\\eeqas\t \nwhich completes the proof.\n\\end{proof}\n\n\\begin{lemma} \\label{lem-martingale-variance} Under the conditions of Theorem \\ref{thm-averaged-rate-quad},  the following statements are true: \n\\begin{itemize} \n   \\item [$(i)$] We have  \\beq E_k = \\alpha_k \\mu(\\sigma_k)  + \\bigO(\\alpha_k^2), \\quad k\\geq 0,  \\label{Ek-expression} \\eeq \nwhere $E_k$ is the gradient error defined by \\eqref{def-Ek}, $\\bigO(\\cdot)$ hides a constant that depends only on $\\Gs, L,m, R$ and $c$ and\n   \\beq  \\mu(\\sk) = - \\sum_{i=1}^{m} P_{\\ski} \\sum_{\\ell=1}^{i-1} \\nabla f_{\\skl}(x^*). \n\t\t\\label{def-v-sigma-k}   \n   \\eeq\nis a sequence of i.i.d. variables where the function $\\mu(\\cdot)$ is defined by \\eqref{def-Msigma}. \n%    \\item [$(ii)$]  It holds that   \n%    \\beq\n%           \\|\\mu(\\s_k)\\| \\leq LmG_* \n%           \t\t\\label{v-inf-norm}\n%     \\eeq\n% where $L=\\sum_i \\|P_i\\|$ and $\\Gs$ is defined by \\eqref{def-Gs}. \n% Furthermore,    \n%     \\beq      \n%        \\E v(\\sk) = \\bar{\\mu} =   \\frac{1}{2} \\sum_{i=1}^m P_i \\nabla f_i(x^*). % \\label{def-mu-star}\n%    \\eeq\n   %\\item [$(iii)$] $\\sigma_*^2 := var(v(u^k)) <\\infty$.\n   \\item [$(ii)$] For any $0 <q\\leq 1 $,  $\\lim_{k \\to \\infty} Y_{q,k}=  \\bar{\\mu}$ a.s. where \n    $ Y_{q,k} = \\frac{\\sum_{i=(1-q)k}^{k-1} E_j}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}.$\n   \\item [$(iii)$] It holds that \n       \\beq  \\| \\mu(\\sigma_k) \\| \\leq Lm G_*.\t\n       \t\t\\label{def-mu-sk-upper-bound}\n       \\eeq\t\n\\end{itemize}  \n   \n\n\\end{lemma}\n\n\\begin{proof} \n\n\t\\begin{itemize} \n\t\t\t\\item [$(i)$] As component functions are quadratics, \\eqref{def-Ek} becomes\n\\beqas E_k &=& \\sum_{i=1}^{m} P_{\\ski}(x_{i-1}^k - x_0^k) = - \\sum_{i=1}^{m}  P_{\\ski}  \\alpha_k \\sum_{\\ell=1}^{i-1} \\nabla f_{\\skl } (x_{\\ell-1}^k). \n \\eeqas\nwhere we can substitute \n \\beq  \\nabla f_{ \\skl } (x_{\\ell-1}^k) =  \\nabla f_{ \\skl } (x^*)  + P_{\\skl} (x_{\\ell-1}^k - x^*).\n \t\\label{quadratic-only-one}\n \\eeq\nThen an application of Lemma \\ref{lem-random-tail-bound}  proves directly the desired result.\n%  \t\t\t\\item [$(ii)$] For the first part, the inequality \\eqref{v-inf-norm} is just a consequence of the triangle inequality applied to the definition \\eqref{def-v-sigma-k} with $L_i = \\| P_i \\|$ and $L=\\sum_{i=1}^m L_i$. For the second part, note that for any $i \\neq \\ell$, the joint distribution of $(\\ski, \\skl)$ is uniform over the set of all (ordered) pairs from $\\{1,2,\\dots,m\\}$. Therefore, for any $i \\neq \\ell$, \n%     \\beqas\n%     \\E \\big[ P_{\\ski} \\nabla f_{\\skl }(x^*) \\big] &=& \\sum_{i=1}^m \\sum_{i\\neq j, j=1}^m \\frac{P_i\\nabla f_j(x^*)}{m(m-1)} \\\\\n%     &=& \\frac{\\sum_{i=1}^m P_i  \\sum_{j=1}^m \\nabla f_j(x^*) - \\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)} \\\\\n%     &=&  - \\frac{\\sum_{j=1}^m P_j \\nabla f_i(x^*)}{m(m-1)}\n%     \\eeqas \n% where we used the fact that $\\nabla f(x^*) = \\sum_{j=1}^m \\nabla f_j(x^*)=0$ by the first order optimality condition. Then, by taking the expectation of \\eqref{def-v-sigma-k}, we obtain \n% \\beqas \\E v(k) &=& - \\sum_{i=1}^{m} \\sum_{\\ell=0}^{i-1} \\E \\big[ P_{\\ski} \\nabla f_{\\skl}(x^*) \\big] \\\\\n%         &=& - \\sum_{i=1}^{m} \\sum_{\\ell=0}^{i-1} - \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)} \\\\\n%         &=&  \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{2}.\n% \\eeqas\t \n% which completes the proof.\n%           \\item [$(iii)$] All the gradients and Hessians of the component functions at the global optimum $x^*$ is finite and there are finitely many component functions, therefore all the moments of $v(u^j)$ exists and is finite including the second moment. Therefore, the variance $\\sigma_*^2$ is bounded.\\footnote{Note that $\\sigma_*^2$ could be computed explicitly by a similar calculation to the one in the proof of part $(ii)$.} This completes the proof.\n   \t\t\t\\item [$(ii)$] We introduce the normalized gradient error sequence $Y_j = E_j / \\alpha_j$. By part $(i)$, $Y_j = \\mu(\\sigma_j) + \\bigO(\\alpha_j)$ where $\\mu(\\sigma_j)$ is a sequence of i.i.d. variables. By the strong law of large numbers, we have\n   \t\t\t     \\beq \\lim_{k \\to \\infty} \\frac{\\sum_{j=0}^{k-1} \\mu(\\sigma_j)}{k} = \\E \\mu(\\sigma_j) = \\bar{\\mu} \\quad \\mbox{a.s.}\\label{lim-aver-vuj}  \\eeq\nwhere the last equality is by the definition of $\\bar{\\mu}$. Therefore,   \t\t\t\n    \t\t\t \\beqas \\lim_{k \\to \\infty} \\frac{\\sum_{j=0}^{k-1} Y_j}{k} = \\lim_{k \\to \\infty} \\bigg(\\frac{\\sum_{j=0}^{k-1} \\mu(\\sigma_j)}{k} + \\frac{\\sum_{j=0}^{k-1} \\bigO(\\alpha_j)}{k}\\bigg) = \\bar{\\mu} \\quad \\mbox{a.s.} \\eeqas\nwhere we used the fact that the second term is negligible as $\\sum_{j=0}^{k-1} \\alpha_j / k = \\bigO(k^{-s}) \\to 0$. \nAs the average of the sequence $Y_j$ converges almost surely, one can show that this implies almost sure convergence of a weighted average of the sequence $Y_j$ as well as long as weights satisfy certain conditions as $k \\to \\infty$. In particular, as the sequence $\\{\\alpha_j\\}$ is monotonically decreasing and is non-summable, by \\cite[Theorem 1]{Etemadi:2006ko}, \n       \\beq \\lim_{k \\to \\infty} Y_{1,k} = \\lim_{k \\to \\infty}\\frac{\\sum_{j=0}^{k-1} \\alpha_j Y_j}{\\sum_{j=0}^{k-1} \\alpha_j} =  \\lim_{k \\to \\infty} \\frac{\\sum_{j=0}^{k-1} E_j}{\\sum_{j=0}^{k-1} \\alpha_j} = \\bar{\\mu} \\quad \\mbox{a.s.}\\label{lim-average-mu-star}\n       \\eeq \nThis completes the proof for $q=1$. For $0<q<1$, by the definition of $Y_{q,k}$, we can write $Y_{1,k} = (1-w_k) Y_{q,k} + w_k Y_{1, (1-q)k}$ where the non-negative weights $w_k$ satisfy    \n$$w_k = \\frac{\\sum_{j=0}^{(1-q)k - 1 } \\alpha_j}{\\sum_{j=0}^{k - 1 } \\alpha_j } \\to_{k \\to \\infty} (1-q)^{1-s} < 1.$$     \n    As both $Y_{1,k}$ and $Y_{1, (1-q)k}$ go to $\\bar{\\mu}$ a.s. by \\eqref{lim-average-mu-star}, it follows that \n$$\\lim_{k\\to \\infty} Y_{q,k} = \\lim_{k\\to \\infty}  \\frac{Y_{1,k} - w_k Y_{1,(1-q)k}}{1-w_k} = \\bar{\\mu}\\quad \\mbox{a.s}$$ as well for any $0<q<1$. This completes the proof.\n\t\\item [$(iii)$] This is a direct consequence of the triangle inequality applied to the definition \\eqref{def-v-sigma-k} with $L_i = \\|P_i\\|$ and $L=\\sum_{i=1}^m L_i$. \n\\end{itemize}\n\\end{proof}","units":[{"raw_tex":"\\begin{corollary}\\label{coro-deter-decay-rate-dist} Under the setting of Theorem \\ref{theo-rate-quadratics}, if $\\sigma$ is sampled uniformly at each cycle instead of being kept fixed, then\n\t\t% \\beqs \\  \\dist_k \\leq \\frac{R M_\\Gamma}{c}\\frac{1}{k^s} + o(\\frac{1}{k^s})%, \\quad M_\\Gamma = \\sup_{\\sigma \\in \\Gamma} \\mu(\\sigma)\n  %        \\eeqs \n         \\begin{align} \\dist_k &\\leq \\frac{R M_\\Gamma }{c}\\frac{1}{k^s} + o(\\frac{1}{k^s})  &\\mbox{if}& \\quad 1/2<s<1,\\\\\n           \t\t\t  \\dist_k  &\\leq \\frac{R^2 M_\\Gamma}{ Rc - 1} \\frac{1}{k} + o(\\frac{1}{k})  &\\mbox{if}& \\quad s=1 \\mbox{ and } Rc>1,\n\t  %\\label{def-Msigma}% \\leq Lm \\Gs\n    \t\\end{align}\nwith probability one where $M_\\Gamma$ is deterministic and is defined by \\eqref{def-M-gamma}.\n%the right-hand side is a deterministic sequence and         \n%  \t    $$M_\\Gamma \\leq Lm \\Gs, \\quad \\Gs = \\underset{1\\leq i \\leq m}{\\sup} \\| \\nabla f_i(x^*)\\|.$\n\\end{corollary}","start":55773,"end":56691},{"raw_tex":"\\begin{proof} \n\t\\begin{itemize} \n\t\\item [$(i)$] As the stepsize sequence is monotonically decreasing, we have the bounds \n   \\beqas \\int_{(1-q)k}^k  \\frac{R}{(x+2)^s} dx \\leq  \\sum_{j=(1-q)k}^k \\alpha_j = \\sum_{j=(1-q)k}^k \\frac{R}{(k+1)^s} %&\\leq& R + \\sum_{j=(1-q)k + 1}^k \\frac{R}{(k+1)^s} \\\\\n   \t\t\t &\\leq& R + \\int_{(1-q)k}^{k-1}  \\frac{R}{(x+1)^s} dx. \n   %1 + \\int_{(1-q)k}^k  \\frac{R}{(x+1)^s} dx. \\\\\n   \\eeqas\nDividing each term by $qk$, after a straightforward integration we obtain\n  $$ \\aqk = \\frac{ k^{1-s} - \\big((1-q)k+1\\big)^{1-s} + \\bigO(1)}{(1-s)qk} R  = \\frac{a_q(s)}{k^s} + \\bigO(\\frac{1}{k}). \n   $$\n which completes the proof.  \n \t\t\t\\item [$(ii)$] Taking the $q$-suffix averages of both sides of \\eqref{averaged-iter-randomized}, we obtain \n\\beq \n\tI_{q,k} := \\frac{\\sum_{j=(1-q)k}^{k-1} {(x_0^j - x_{0}^{j+1})}{\\alpha_j^{-1}}}{qk} = \\frac{ \\sum_{j=(1-q)k}^{k-1}   \\nabla f(x_0^j)+ E_j}{qk}. \\label{averaged-iter-random-2}\n   \\eeq \nAs $f$ is a quadratic, the first order Taylor series for the gradient of $f$ is exact:\n  \\beq \\nabla f(x_0^j) = H_* (x_0^j-x^*).\n  \t\t\\label{grad-as-hessian-prod-quadratics}\n  \\eeq\n Therefore, \\eqref{averaged-iter-random-2} becomes  $\nI_{q,k} \n= \\frac{ \\sum_{j=(1-q)k}^{k-1}   H_* (x_0^j - x^*)   + E_j }{qk} $\n%\\label{averaged-iter-random-3} \nwhich is equivalent to\n\\beqa\n                    I_{q,k} &=& H_*(\\bar{x}_{q,k} - x^*) + \\frac{\\sum_{j=(1-q)k}^{k-1} E_j}{qk} = H_*(\\bar{x}_{q,k} - x^*) + \\bar{\\alpha}_{q,k}  Y_{q,k}\\label{iterate-eq-randomized}\n   \\eeqa \nwhere $Y_{q,k}$ is defined as\n    \\beq Y_{q,k} := \\frac{1}{\\bar{\\alpha}_{q,k}} \\frac{\\sum_{j=(1-q)k}^{k-1} E_j }{qk } =  \\frac{\\sum_{j=(1-q)k}^{k-1} E_j}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}\t\n\t\t\\label{def-Yqk}    \n     \\eeq\nand can be interpreted as the ($q$-suffix) averaged gradient error sequence $E_j$  normalized by the ($q$-suffix) averaged stepsize sequence $\\alpha_j$. Since $H_*$ is invertible by the strong convexity of $f$ (see \\eqref{def-Hs}), we can rewrite \\eqref{iterate-eq-randomized} as\n\\beqa  \\bar{x}_{q,k} - x^* &=& \n                     - H_*^{-1} \\aqk \\Yqk  + H_*^{-1} \\Iqk  \n            = - H_*^{-1} \\bar{\\alpha}_{q,k} Y_{q,k}  +  \\begin{cases} \\bigO(\\frac{1}{k})  & \\mbox{if } \\quad 0<q<1 \\\\\n\\bigO( \\frac{\\log k}{k})  & \\mbox{if } \\quad q = 1.\n\\end{cases} \\label{xk-averaged-asymptotics-2}\n              %           &=& \\bigO(\\frac{\\log k}{k}) + \\bigO(\\frac{1}{k}) - H_*^{-1} \\bigg(   \\frac{\\sum_{j=0}^{k-1} E_j}{\\sum_{j=0}^{k-1}\\alpha_j}\\bigg) \\frac{\\sum_{j=0}^{k-1}\\alpha_j}{k}   \\nonumber \n   \\eeqa              \nwhere we used the inequality $\\| H_*^{-1}\\| \\leq 1/c $ implied by \\eqref{def-Hs} and Lemma \\ref{lem-I-lk} from the appendix to provide an upper bound for the second term in the first equality. Note that, as a consequence of Lemma \\ref{lem-I-lk}, $\\bigO(\\cdot)$ notation above hides a constant that depends only on the parameters $\\Gs, L,c, m, R, s, q$ and also $\\dist_0$ when $q=1$. Then, dividing both sides of \\eqref{xk-averaged-asymptotics-2} by $\\aqk$, taking limits as $k$ goes to infinity,  using part $(i)$ on the asymptotic behavior of $\\aqk$ and the fact that $Y_{q,k} \\to \\bar{\\mu}$ a.s. from Lemma \\ref{lem-martingale-variance}, we obtain the claimed result.%$$ \\lim_{k \\to \\infty} k^s (\\bar{x}_{q,k} - x^*) = a_q(s) H_*^{-1} \\bar{\\mu}. $$\n%This completes the proof of part $(i)$.\n%Taking expectations, \n%\n%$$ \\E (\\bar{x}_k - x^*)  = \\bigO(\\frac{\\log k}{k}) -  H_*^{-1} \\bigg(   \\frac{\\sum_{j=0}^{k-1} \\alpha_j \\E v(u^j) }{k}\\bigg) + \\bigO(\\alpha_k^2).  $$\n%\n%\n%which implies\n%$$ \\E (\\bar{x}_k - x^*)  = \\bigO(\\frac{\\log k}{k}) -  b_* \\bigg(   \\frac{\\sum_{j=0}^{k-1} \\alpha_j  }{k}\\bigg) + \\bigO(\\alpha_k^2).  $$ \n%$ b_* = H_*^{-1}\\frac{\\sum_{r=1}^m \\nabla^2 f_r(x^*) \\nabla f_r(x^*)}{2}.$\n%Suppose we can compute an approximation $b_k$ to $b_*$ satisfying\n%   $b_k = b_* + \\bigO(\\alpha_k).$\n%Then,    \n%$$ \\E \\bigg(\\bar{x}_k + b_k \\bigg(   \\frac{\\sum_{j=0}^{k-1} \\alpha_j  }{k}\\bigg)  - x^*\\bigg)  = \\bigO(\\frac{\\log k}{k})  + \\bigO(\\alpha_k^2).  $$ \n%\n%NEED TO LOOK AT THE VARIANCE TERM AS WELL. USE OLD MARTINGALE APPROACH FOR BOUNDING THE VA\n\n \\item [$(iii)$] By parts $(i)$ and $(iii)$ of Lemma \\ref{lem-martingale-variance} from the appendix that relates the gradient error sequence $E_j$ to a sequence of i.i.d. variables $\\mu(\\sigma_j)$, for $0<q\\leq 1$,  \n\t\t\\beq Y_{q,k} = \\frac{\\sum_{j=(1-q)k}^{k-1} E_j}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j} =  \\frac{\\sum_{i=(1-q)k}^{k-1} \\alpha_j \\mu(\\sigma_j) + \\bigO(\\alpha_j^2)}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}. \t \n\t\t\t\\label{Yqk-helper}  \n     \\eeq \nWe first give a proof for $q=1$, the proof for the remaining $q \\in (0,1)$ case will be similar. Assume $q=1$. Plugging $q=1$ and \\eqref{Yqk-helper} into \\eqref{xk-averaged-asymptotics-2}, we obtain\n     \\beqa \\bar{x}_{1,k} - x^* \n     &=&\\bigO(\\frac{\\log k}{k})  - H_*^{-1} \\aonek \\Yonek \\nonumber \\\\\n     &=& \\bigO(\\frac{\\log k}{k}) - H_*^{-1} \\bigg(   \\frac{\\sum_{j=0}^{k-1} \\alpha_j \\big(\\mu(\\s_j) - \\bar{\\mu}\\big)}{k} +  \\frac{\\sum_{j=0}^{k-1}  \\alpha_j \\bar{\\mu} + \\bigO(\\alpha_j^2) }{k}  \\bigg) \\nonumber \\\\\n     \t\t\t\t\t\t &=& b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{\\sum_{j=0}^{k-1}  \\alpha_j (\\mu(\\s_j) - \\bar{\\mu})}{k} - H_*^{-1} \\sum_{j=0}^{k-1}  \\frac{\\bigO(\\alpha_j^2)}{k} \\nonumber \\\\     \n     \t\t\t\t\t\t    &=& b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{\\sum_{j=0}^{k-1}  \\alpha_j (\\mu(\\s_j) - \\bar{\\mu})}{k} \\label{xqk-high-proba} %\\\\     \t\t\t\t\t\t    \n     \t\t\t\t\t\t%    &=& b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{M_{1,k}}{k}  \\label{xk-biased-high-probability}\n     \\eeqa    \nwhere $\\bonek$ is defined by \\eqref{def-bqk} and we used in the last step the fact that for $s>1/2$ \n\t\\beq \\sum_{j=0}^\\infty \\alpha_j^2 = \\sum_{j=1}^\\infty \\frac{R^2}{j^{2s}} = R^2 \\zeta(2s) < \\infty\n\t\t\\label{square-summable-step}\t\n\t\\eeq \nwhere $\\zeta(\\cdot)$ is the Riemann-Zeta function. We now study the asymptotic behavior of the last summation term in \\eqref{xqk-high-proba} by introducing the process\n\t$S_{1,k} = \\sum_{j=0}^{k-1} Z_j$, where $Z_j := \\alpha_j (\\mu(\\s_j) - \\bar{\\mu})$ and $k\\geq 0$ with the convention that $S_{1,0} = 0$. Equipped with this definition, \\eqref{xqk-high-proba} becomes\n  \\beq \n  \t\t \\xonek - \\xs = b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{S_{1,k}}{k}.  \\label{xk-biased-high-probability}\n  \\eeq\t\t  \nThe random variables $Z_j$ are independent, centered and have an identical distribution up to the scaling factor $\\alpha_j$. Therefore,  $S_{1,k}$ is a sum of  centered random variables satisfying:\n    \\beqa \\begin{scriptsize} \\| S_{1,k} - S_{1,k-1} \\| =\\big \\|\\alpha_{k-1}\\big(\\mu(\\sigma_{k-1})-\\bar{\\mu}\\big)\\big\\| \n    \\leq \\gamma_{k-1}:=\\alpha_{k-1}LmG_* \\label{marting-diff-as-bound}\n    \\end{scriptsize}\n    \\eeqa \nwhere we used \\eqref{def-mu-sk-upper-bound} in the last inequality (see also Lemma \\ref{lem-computing-mu-bar}). Then, by the Azuma-Hoeffding inequality, for every $t>0$, \n  \\beqas \\P \\bigg( \\big\\|\\frac{S_{1,k}}{k}\\big\\| > \\frac{t}{k}\\bigg) &\\leq& 2 \\exp \\bigg( - \\frac{t^2}{2\\sum_{j=0}^{k-1} \\gamma_j^2} \\bigg) =  2 \\exp \\big( - \\frac{t^2}{ \\beta }\\big) \n  \\eeqas\nwhere $\\beta = 2\\sum_{j=0}^{\\infty} \\gamma_j^2 < \\infty$ as $\\alpha_j$ is square-summable (see \\eqref{square-summable-step}). Note that $\\beta$ depends only on $\\Gs, L,m$ and the stepsize parameters $R$ and $s$. It is easy to see that selecting $t \\geq t_\\delta = \\sqrt{\\beta  \\log(2/\\delta)}$ makes the right-hand side $\\leq \\delta$.  Therefore for any $\\delta>0$, with probability at least $1-\\delta$, \n\t\t\\beq \\bigl \\| \\frac{S_{1,k}}{k} \\bigr \\| \\leq \\frac{\\sqrt{\\beta  \\log(2/\\delta)}}{k}\n\t\t\t\\label{average-sum-bound-whp}\n\t\t\\eeq\nwhich if inserted into the expression \\eqref{xk-biased-high-probability} completes the proof for the $q=1$ case. For  $0<q<1$ case, the same line of reasoning applies except that we replace $b_{1,k}$ with $b_{q,k}$ and we can improve the $\\bigO( \\log k /k)$ term in the expression \\eqref{xk-biased-high-probability} to $\\bigO(1/k)$, this is justified by \\eqref{xk-averaged-asymptotics-2}. Then, this leads to \n\t\t \\beq \n  \t\t \\xqk - \\xs = b_{q,k}  +  \\bigO(\\frac{1}{k})  - H_*^{-1}   \\frac{S_{q,k}}{qk}  \\label{xk-biased-high-probability-3}\n  \\eeq\nwhere $S_{q,k} := \\sum_{j=(1-q)k}^{k-1} Z_j = S_{1,k} - S_{1,(1-q)k}$ is the $q$-suffix cumulative sum (cumulative sum of the last $qk$ terms) of the sequence $Z_{k}$. Then using \\eqref{average-sum-bound-whp}, with probability at least $1-\\delta$,\n\\beq \\bigl \\| \\frac{S_{q,k}}{k} \\bigr \\| \\leq \\| \\frac{S_{1,k}}{k} \\bigr \\|  + \\| \\frac{S_{1,(1-q)k}}{k} \\bigr \\|  \\leq \\frac{2 t_\\delta}{k}.\n\t\t\t\\label{average-sum-bound-whp-3}\n\t\t\\eeq\nPlugging this high probability bound into \\eqref{xk-biased-high-probability-3}, we conclude. \t\t\n\t\\item [$(iv)$] By Lemma \\ref{lem-random-tail-bound}, we have $\\underset{1\\leq i < m}{\\max} \\|x_{i-1}^k - x^*\\| = \\bigO(\\alpha_k)$. Therefore,\n\t \\beqa \\scriptsize \\| \\nabla f_{\\ski} (x_{i-1}^k)  - \\nabla f_{\\ski} (x^*) \\|  = \\bigO(\\alpha_k) \\label{grad-diff-quadratic}\n\t \\eeqa\nfor any $i=1,2,\\dots,m$. As a consequence, \t\n\t\\beqas \\hat{b}_{q,k} &=& -\\bar{\\alpha}_{q,k} H_*^{-1} \\sum_{i=1}^{m} P_{\\ski}  \\bigg(\\nabla f_{\\ski} (x^*) + \\bigO(\\alpha_k)\\bigg)  \\\\\n\t                         &=& - \\bar{\\alpha}_{q,k} H_*^{-1} \\sum_{j=1}^{m} P_{j} \\nabla f_j (x^*)   + \\bigO(\\alpha_k^2)   \n%\t                         &=& -\\bigg( H_*^{-1} + \\bigO(\\alpha_k) \\bigg) \\sum_{i=1}^{m} \\nabla^2 f_{i} (x_*) \\nabla f_i (x^*)  \\bar{\\alpha}_{q,k}+ \\bigO(\\alpha_k^2) +  o(\\alpha_k^2) \\\\\n\t                         = b_{q,k} + \\bigO(\\alpha_k^2)\t                         \t                        \t                         \n\t\\eeqas\t\t \nwhere in the second equality we use the fact that $\\bar{\\alpha}_{q,k}= \\bigO(1/k^s) = \\bigO(\\alpha_k)$ implied by part $(i)$.\n%\t\t\\item [$(ii)$]  By taking norm square of both sides and expectations in \\eqref{xk-averaged-asymptotics} and applying Cauchy-Schwartz gives\n%\t\t\t\\beqas \\E \\| \\bar{x}_k - x^*\\|^2 &\\leq& 4\\bigO(\\frac{\\log k}{k}) + 4\\bigO(\\frac{1}{k}) + 4 \\E \\bigg\\|  H_*^{-1} \\bigg(   \\frac{\\sum_{j=0}^{k-1} E_j}{k}\\bigg)\\bigg\\|^2 + 4\\bigO(\\alpha_k^2)   \\\\\n%\t\t\t                                              &\\leq& 4\\bigO(\\frac{\\log k}{k}) + 4\\bigO(\\frac{1}{k}) + \\frac{4}{c^2} \\E \\bigg\\|   \\frac{\\sum_{j=0}^{k-1} E_j}{k}\\bigg\\|^2 + 4\\bigO(\\alpha_k^2) \n%\t\t\t\\eeqas\n%We then decompose the term inside the expectation as\n%    \\beqas \\frac{\\sum_{j=0}^{k-1} E_j}{k} &=& \\frac{\\sum_{j=0}^{k-1} \\alpha_j v(u^j) - \\alpha_j \\bar{\\mu}}{k} + \\frac{\\sum_{j=0}^{k-1} \\alpha_j \\bar{\\mu} + \\bigO(\\alpha_j^2)}{k} \\\\\n%               &=& \\frac{\\sum_{j=0}^{k-1} \\alpha_j v(u^j) - \\alpha_j \\bar{\\mu}}{\\sigma_* (\\sum_{j=0}^{k-1}\\alpha_j^2)^{1/2}} \\frac{\\sigma_* (\\sum_{j=0}^{k-1}\\alpha_j^2)^{1/2}}{k} + \\frac{\\sum_{j=0}^{k-1} \\alpha_j \\bar{\\mu} + \\bigO(\\alpha_j^2)}{k}\n%    \\eeqas\n\\end{itemize}\n\\end{proof}","start":69040,"end":79865},{"raw_tex":"\\begin{lemma}\\label{lem-random-tail-bound}Under the conditions of Theorem \\ref{thm-averaged-rate-quad} we have \n    $\\underset{0 \\leq i < m}{\\max} \\| x_i^k - x^*\\| =  \\bigO(\\frac{1}{k^s}).$\nwhere $\\bigO(\\cdot)$ hides a constant that depends only on $\\Gs, L, m,c$ and $R$. \n\\end{lemma}","start":115000,"end":115284},{"raw_tex":"\\begin{proof} By Corollary \\ref{coro-deter-decay-rate-dist}, \n   \\beq \\| x_0^k - x^* \\|  = \\bigO(\\frac{1}{k^s}).\n   \t\t\\label{upper-bound-cycle-begin}\n   \\eeq  \nwhere $\\bigO(\\cdot)$ hides a constant that depends only on $\\Gs, L, m,R$ and $c$.  We have also for any $0\\leq i< m$ and $k\\geq 0$,\n   \\beqas \\| x_i^k - x^*\\|  &\\leq& \\| x_0^k - x^*\\| + \\| x_i^k - x_0^k \\|  = \\| x_0^k - x^*\\| + i\\alpha_k \\max_{\\ell=1,\\dots,i} \\|\\nabla f_{\\sigma_k(\\ell)} (x_{\\ell-1}^k)\\| \\\\\n   &\\leq& \\| x_0^k - x^*\\| + (m-1)\\frac{R}{(k+1)^s} \\big(  G_* + \\max_{\\ell=1,\\dots,i} \\|\\nabla f_{\\sigma_k(\\ell)} (x_{\\ell-1}^k) - \\nabla f_{\\sigma_k(\\ell)} (x^*)\\| \\big) \\\\\n   &\\leq& \\| x_0^k - x^*\\| + (m-1)\\frac{R}{(k+1)^s} \\big(  G_* + L \\max_{\\ell=1,\\dots,i} \\|x_{\\ell-1}^k - x^*\\| \\big). \n   \\eeqas\nwhere we used the $L$-Lipschitzness of the gradient of $f$ where $L$ is given by \\ref{def-L}. Using \\eqref{upper-bound-cycle-begin} and applying this inequality inductively for $i=0,1,2,\\dots, m-1$ we conclude.\n\\end{proof}","start":115286,"end":116281},{"raw_tex":"\\begin{lemma}\\label{lem-I-lk} Under the conditions of Theorem \\ref{thm-averaged-rate-quad}, consider the sequence\n\\beq I_{q,k} = \\frac{\\sum_{j=(1-q)k}^{k-1} {(x_0^j - x_{0}^{j+1})}{\\alpha_j^{-1}}}{qk}, \\quad  0 < q \\leq 1. \\label{def-I-qk}\n\\eeq\nThen, \n    $ I_{q, k} = \\begin{cases}  \\bigO \\big( \\frac{\\log k}{k}  \\big) \n     &  \\mbox{if} \\quad q=1, \\\\  \n    \t\t\t\\bigO \\big( \\frac{1}{k}\\big) \t & \\mbox{if} \\quad 0<q<1.  \n    \\end{cases}  \n%    &\\leq& \\bigO \\big( \\frac{\\log k}{k}\\big) \\\\\n%               I_{qk, k} &\\leq& \\bigO \\big( \\frac{1}{k}\\big) \\quad \\mbox{for all} \\quad 0<q<1.\n   $\nIn the former case, $\\bigO(\\cdot)$ hides a constant that depends only on $\\Gs, L,m,  c, R, s,q$ and $\\dist_0$. In the latter case, the same dependency on the constants occurs except that the dependency on $\\dist_0$ can be removed. \n\\end{lemma}","start":116644,"end":117475},{"raw_tex":"\\begin{proof}  It follows from integration by parts that for any $\\ell<k$, \n\\beq -\\sum_{j=\\ell}^{k-1}{(x_0^j - x_{0}^{j+1})}{\\alpha_j^{-1}}  =  \\alpha_k^{-1} (x_0^k - x^*) - \\alpha_\\ell^{-1} (x_0^\\ell - x^*) - \\sum_{j=\\ell}^{k-1} (x_0^{j+1} - x^*)(\\alpha_{j+1}^{-1}-\\alpha_j^{-1}). \\label{sum-of-aver-iters}\n\\eeq\nNext, we investigate the asymptotic behavior of the terms on the right-hand side. A consequence of Corollary \\ref{coro-deter-decay-rate-dist} and the inequality \\ref{def-M-gamma} is that  \n\t\\beq \\alpha_k^{-1} \\| x_0^k -x^*\\| = \\frac{(k+1)^s}{R} \\| x_0^k -x^*\\| \\leq \\frac{LmG_*}{c} + o(1) = \\bigO(1)\n\t\t\\label{step-times-dist-bd} \t\n\t\\eeq\nand therefore\n\\beqas \n\t| \\alpha_{k+1}^{-1} - \\alpha_{k}^{-1} |  \\| x_0^k -x^*\\| &=& \\frac{(k+2)^s - (k+1)^s} {(k+1)^s} \\alpha_k^{-1}\\| x_0^k -x^*\\|  \n\t = \\bigg( \\big(1 + \\frac{1}{k+1} \\big)^s -1 \\bigg) \\alpha_k^{-1}\\| x_0^k -x^*\\| \\\\\n\t  &\\leq &  \\frac{s}{k+1}\\alpha_k^{-1}\\| x_0^k -x^*\\|  \n\t  \\leq \\frac{sLmG_*}{c} \\frac{1}{k+1} + o(\\frac{1}{k+1}) = \\bigO(\\frac{1}{k+1})\n\\eeqas\nwhere $\\bigO(\\cdot)$ hides a constant that depends only on $L,\\Gs, c, m$ and $s$. Then, setting  $\\ell = (1-q)k$ in \\eqref{sum-of-aver-iters}, it follows that \n\\beqa  \\big\\| \\sum_{j=\\ell}^{k-1} {(x_0^j - x_0^{j+1})}{\\alpha_j^{-1}} \\big\\| &\\leq&  \\|\\alpha_k^{-1} (x_0^k - x^*)\\| + \\| \\alpha_{(1-q)k}^{-1} (x_0^{(1-q)k}- x^*)\\| \\\\\n  && + \\sum_{j=(1-q)k}^{k-1} \\| x_0^{j+1} - x^*\\| |\\alpha_{j+1}^{-1}-\\alpha_j^{-1}|.  \\nonumber\\\\\n       &=& \\bigO(1) + \\| \\alpha_{(1-q)k}^{-1} (x_0^{(1-q)k} - x^*)\\| + \\bigO \\bigg( \\sum_{j=(1-q)k}^{k-1} \\frac{1}{j+1}\\bigg).  \\label{weighted-iter-diff-bound} %\\\\\n       %&& + o \\bigg( \\sum_{j=\\ell}^{k-1} \\frac{1}{(j)^s} \\|(j+1)^s - j^s) \\|\\bigg). \\nonumber \\\\\n       %&=& \\bigO(1)  + \\| \\alpha_\\ell^{-1} (x_0^\\ell - x^*)\\| + \\bigO \\bigg( \\sum_{j=\\ell}^{k-1} \\big((1+\\frac{1}{j+1})^s - 1\\big) \\bigg) \\nonumber \\\\\n       %&& + o \\bigg( \\sum_{j=\\ell}^{k-1} \\big((1+\\frac{1}{j})^s - 1\\big) \\bigg). \\nonumber \\\\\n     % &=& \\bigO(1) + \\| \\alpha_\\ell^{-1} (x_0^\\ell - x^*)\\| + \\bigO \\bigg( \\sum_{j=\\ell}^{k-1} \\frac{s}{j+1} \\bigg) \n       %\\\\\n      %&\\leq& \\bigO(1) + \\bigO(\\frac{1}{k^{2s}} ) + \\bigO \\bigg( \\sum_{j=\\max(\\ell,1)}^{k-1} \\frac{s}{j} \\bigg) \\label{weighted-iter-diff-bound}\n\\eeqa\nWe also have\n  \\beq  \\| \\alpha_{(1-q)k}^{-1} (x_0^{(1-q)k} - x^*)\\|  = \\begin{cases} \\alpha_0^{-1} \\dist_0 & \\mbox{if} \\quad q = 1, \\\\ \\bigO(1)  & \\mbox{if} \\quad 0<q<1, \\end{cases} \\label{init-averaged-iter-bound}\n  \\eeq\nwhere the second part follows from \\eqref{step-times-dist-bd} with similar constants for the $\\bigO(\\cdot)$ term. As the sequence $\\frac{1}{j+1}$ is monotonically decreasing, for any $k>0$ we have the bounds \n\\beq  \\sum_{j=(1-q)k}^{k-1} \\frac{1}{j+1} \\leq \\frac{1}{(1-q)k + 1} + \\int_{(1-q)k}^{k-1} \\frac{1}{x+1} dx \\leq \\begin{cases} 1 + \\log k & \\mbox{if } \\quad  q=1, \\\\\n 1 + \\log(\\frac{1}{1-q})  & \\mbox{if } \\quad  0 <q <1.\n\\end{cases}  \\label{tail-suffix-averaging}\n\\eeq\nNote that when $q=1$ this bound grows with $k$ logarithmically whereas for $q<1$ it does not grow with $k$. \nThen, combining \\eqref{weighted-iter-diff-bound}, \\eqref{init-averaged-iter-bound} and \\eqref{tail-suffix-averaging}  we obtain\n  $$ \\| \\Iqk \\| \\leq \\frac{\\big\\| \\sum_{j=\\ell}^{k-1} {(x_0^j - x_0^{j+1})}{\\alpha_j^{-1}} \\big\\|}{qk} = \\begin{cases}  \\bigO \\big(  \\frac{\\log k}{k}\\big) \n     &  \\mbox{if} \\quad q=1 \\\\  \n    \t\t\t\\bigO \\big( \\frac{1}{k}\\big) \t & \\mbox{if} \\quad 0<q<1  \n    \\end{cases}  $$\nas desired which completes the proof.    \n\\end{proof}","start":117612,"end":121119},{"raw_tex":"\\begin{lemma}\\label{lem-computing-mu-bar} Let $\\sigma$ be a random permutation of $\\{1,2,\\dots,m\\}$ sampled uniformly over the set of all permutations $\\Gamma$ defined by \\eqref{def-Gamma-perm-set} and $\\mu(\\sigma)$ be the vector defined by \\eqref{def-Msigma} that depends on $\\sigma$. Then, \\beq\n\t\\bar{\\mu} = \\E_{\\sigma}\\big(\\mu({\\sigma})\\big) = \\frac{1}{2} \\sum_{i=1}^m P_i \\nabla f_i(x^*)\n\\eeq\nwhere $\\E_{\\sigma}$ denotes the expectation with respect to the random permutation $\\sigma$ and $\\bar{mu}$ is defined by \\eqref{def-mu-bar}.\n\\end{lemma}","start":121121,"end":121670},{"raw_tex":"\\begin{proof} For any $i \\neq \\ell$, the joint distribution of $(\\sigma(i), \\sigma(\\ell))$ is uniform over the set of all (ordered) pairs from $\\{1,2,\\dots,m\\}$. Therefore, for any $i \\neq \\ell$, \n    \\beqas\n    \\E_\\sigma \\big[ P_{\\sigma(i)} \\nabla f_{\\sigma(\\ell) }(x^*) \\big] &=& \\sum_{i=1}^m \\sum_{i\\neq j, j=1}^m \\frac{P_i\\nabla f_j(x^*)}{m(m-1)} \\\\\n    &=& \\frac{\\sum_{i=1}^m P_i  \\sum_{j=1}^m \\nabla f_j(x^*) - \\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)} \n    =  - \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)}\n    \\eeqas \nwhere we used the fact that $\\nabla f(x^*) = \\sum_{j=1}^m \\nabla f_j(x^*)=0$ by the first order optimality condition. Then, by taking the expectation of \\eqref{def-v-sigma-k}, we obtain \n\\beqas \\E_\\sigma (\\mu(\\sigma)) &=& - \\sum_{i=1}^{m} \\sum_{\\ell=0}^{i-1} \\E \\big[ P_{\\sigma(i)} \\nabla f_{\\sigma(\\ell)}(x^*) \\big] \n        =  \\sum_{i=1}^{m} \\sum_{\\ell=0}^{i-1}  \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)} = \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{2}.\n\\eeqas\t \nwhich completes the proof.\n\\end{proof}","start":121671,"end":122706},{"raw_tex":"\\begin{lemma} \\label{lem-martingale-variance} Under the conditions of Theorem \\ref{thm-averaged-rate-quad},  the following statements are true: \n\\begin{itemize} \n   \\item [$(i)$] We have  \\beq E_k = \\alpha_k \\mu(\\sigma_k)  + \\bigO(\\alpha_k^2), \\quad k\\geq 0,  \\label{Ek-expression} \\eeq \nwhere $E_k$ is the gradient error defined by \\eqref{def-Ek}, $\\bigO(\\cdot)$ hides a constant that depends only on $\\Gs, L,m, R$ and $c$ and\n   \\beq  \\mu(\\sk) = - \\sum_{i=1}^{m} P_{\\ski} \\sum_{\\ell=1}^{i-1} \\nabla f_{\\skl}(x^*). \n\t\t\\label{def-v-sigma-k}   \n   \\eeq\nis a sequence of i.i.d. variables where the function $\\mu(\\cdot)$ is defined by \\eqref{def-Msigma}. \n%    \\item [$(ii)$]  It holds that   \n%    \\beq\n%           \\|\\mu(\\s_k)\\| \\leq LmG_* \n%           \t\t\\label{v-inf-norm}\n%     \\eeq\n% where $L=\\sum_i \\|P_i\\|$ and $\\Gs$ is defined by \\eqref{def-Gs}. \n% Furthermore,    \n%     \\beq      \n%        \\E v(\\sk) = \\bar{\\mu} =   \\frac{1}{2} \\sum_{i=1}^m P_i \\nabla f_i(x^*). % \\label{def-mu-star}\n%    \\eeq\n   %\\item [$(iii)$] $\\sigma_*^2 := var(v(u^k)) <\\infty$.\n   \\item [$(ii)$] For any $0 <q\\leq 1 $,  $\\lim_{k \\to \\infty} Y_{q,k}=  \\bar{\\mu}$ a.s. where \n    $ Y_{q,k} = \\frac{\\sum_{i=(1-q)k}^{k-1} E_j}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}.$\n   \\item [$(iii)$] It holds that \n       \\beq  \\| \\mu(\\sigma_k) \\| \\leq Lm G_*.\t\n       \t\t\\label{def-mu-sk-upper-bound}\n       \\eeq\t\n\\end{itemize}  \n   \n\n\\end{lemma}","start":122707,"end":124108},{"raw_tex":"\\begin{proof} \n\n\t\\begin{itemize} \n\t\t\t\\item [$(i)$] As component functions are quadratics, \\eqref{def-Ek} becomes\n\\beqas E_k &=& \\sum_{i=1}^{m} P_{\\ski}(x_{i-1}^k - x_0^k) = - \\sum_{i=1}^{m}  P_{\\ski}  \\alpha_k \\sum_{\\ell=1}^{i-1} \\nabla f_{\\skl } (x_{\\ell-1}^k). \n \\eeqas\nwhere we can substitute \n \\beq  \\nabla f_{ \\skl } (x_{\\ell-1}^k) =  \\nabla f_{ \\skl } (x^*)  + P_{\\skl} (x_{\\ell-1}^k - x^*).\n \t\\label{quadratic-only-one}\n \\eeq\nThen an application of Lemma \\ref{lem-random-tail-bound}  proves directly the desired result.\n%  \t\t\t\\item [$(ii)$] For the first part, the inequality \\eqref{v-inf-norm} is just a consequence of the triangle inequality applied to the definition \\eqref{def-v-sigma-k} with $L_i = \\| P_i \\|$ and $L=\\sum_{i=1}^m L_i$. For the second part, note that for any $i \\neq \\ell$, the joint distribution of $(\\ski, \\skl)$ is uniform over the set of all (ordered) pairs from $\\{1,2,\\dots,m\\}$. Therefore, for any $i \\neq \\ell$, \n%     \\beqas\n%     \\E \\big[ P_{\\ski} \\nabla f_{\\skl }(x^*) \\big] &=& \\sum_{i=1}^m \\sum_{i\\neq j, j=1}^m \\frac{P_i\\nabla f_j(x^*)}{m(m-1)} \\\\\n%     &=& \\frac{\\sum_{i=1}^m P_i  \\sum_{j=1}^m \\nabla f_j(x^*) - \\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)} \\\\\n%     &=&  - \\frac{\\sum_{j=1}^m P_j \\nabla f_i(x^*)}{m(m-1)}\n%     \\eeqas \n% where we used the fact that $\\nabla f(x^*) = \\sum_{j=1}^m \\nabla f_j(x^*)=0$ by the first order optimality condition. Then, by taking the expectation of \\eqref{def-v-sigma-k}, we obtain \n% \\beqas \\E v(k) &=& - \\sum_{i=1}^{m} \\sum_{\\ell=0}^{i-1} \\E \\big[ P_{\\ski} \\nabla f_{\\skl}(x^*) \\big] \\\\\n%         &=& - \\sum_{i=1}^{m} \\sum_{\\ell=0}^{i-1} - \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)} \\\\\n%         &=&  \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{2}.\n% \\eeqas\t \n% which completes the proof.\n%           \\item [$(iii)$] All the gradients and Hessians of the component functions at the global optimum $x^*$ is finite and there are finitely many component functions, therefore all the moments of $v(u^j)$ exists and is finite including the second moment. Therefore, the variance $\\sigma_*^2$ is bounded.\\footnote{Note that $\\sigma_*^2$ could be computed explicitly by a similar calculation to the one in the proof of part $(ii)$.} This completes the proof.\n   \t\t\t\\item [$(ii)$] We introduce the normalized gradient error sequence $Y_j = E_j / \\alpha_j$. By part $(i)$, $Y_j = \\mu(\\sigma_j) + \\bigO(\\alpha_j)$ where $\\mu(\\sigma_j)$ is a sequence of i.i.d. variables. By the strong law of large numbers, we have\n   \t\t\t     \\beq \\lim_{k \\to \\infty} \\frac{\\sum_{j=0}^{k-1} \\mu(\\sigma_j)}{k} = \\E \\mu(\\sigma_j) = \\bar{\\mu} \\quad \\mbox{a.s.}\\label{lim-aver-vuj}  \\eeq\nwhere the last equality is by the definition of $\\bar{\\mu}$. Therefore,   \t\t\t\n    \t\t\t \\beqas \\lim_{k \\to \\infty} \\frac{\\sum_{j=0}^{k-1} Y_j}{k} = \\lim_{k \\to \\infty} \\bigg(\\frac{\\sum_{j=0}^{k-1} \\mu(\\sigma_j)}{k} + \\frac{\\sum_{j=0}^{k-1} \\bigO(\\alpha_j)}{k}\\bigg) = \\bar{\\mu} \\quad \\mbox{a.s.} \\eeqas\nwhere we used the fact that the second term is negligible as $\\sum_{j=0}^{k-1} \\alpha_j / k = \\bigO(k^{-s}) \\to 0$. \nAs the average of the sequence $Y_j$ converges almost surely, one can show that this implies almost sure convergence of a weighted average of the sequence $Y_j$ as well as long as weights satisfy certain conditions as $k \\to \\infty$. In particular, as the sequence $\\{\\alpha_j\\}$ is monotonically decreasing and is non-summable, by \\cite[Theorem 1]{Etemadi:2006ko}, \n       \\beq \\lim_{k \\to \\infty} Y_{1,k} = \\lim_{k \\to \\infty}\\frac{\\sum_{j=0}^{k-1} \\alpha_j Y_j}{\\sum_{j=0}^{k-1} \\alpha_j} =  \\lim_{k \\to \\infty} \\frac{\\sum_{j=0}^{k-1} E_j}{\\sum_{j=0}^{k-1} \\alpha_j} = \\bar{\\mu} \\quad \\mbox{a.s.}\\label{lim-average-mu-star}\n       \\eeq \nThis completes the proof for $q=1$. For $0<q<1$, by the definition of $Y_{q,k}$, we can write $Y_{1,k} = (1-w_k) Y_{q,k} + w_k Y_{1, (1-q)k}$ where the non-negative weights $w_k$ satisfy    \n$$w_k = \\frac{\\sum_{j=0}^{(1-q)k - 1 } \\alpha_j}{\\sum_{j=0}^{k - 1 } \\alpha_j } \\to_{k \\to \\infty} (1-q)^{1-s} < 1.$$     \n    As both $Y_{1,k}$ and $Y_{1, (1-q)k}$ go to $\\bar{\\mu}$ a.s. by \\eqref{lim-average-mu-star}, it follows that \n$$\\lim_{k\\to \\infty} Y_{q,k} = \\lim_{k\\to \\infty}  \\frac{Y_{1,k} - w_k Y_{1,(1-q)k}}{1-w_k} = \\bar{\\mu}\\quad \\mbox{a.s}$$ as well for any $0<q<1$. This completes the proof.\n\t\\item [$(iii)$] This is a direct consequence of the triangle inequality applied to the definition \\eqref{def-v-sigma-k} with $L_i = \\|P_i\\|$ and $L=\\sum_{i=1}^m L_i$. \n\\end{itemize}\n\\end{proof}","start":124109,"end":128603}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["pbm-multi-agent"],"declared_external":false,"source_only":false,"title":"有限和无约束优化","source_units":[{"raw_tex":"We consider the following unconstrained optimization problem where the objective function is the sum of a large number of component functions: \n\\beq\\label{pbm-multi-agent}\n\\min f(x) := \\sum_{i=1}^m f_i(x) \\quad \\mbox{s.t.} \\quad x \\in \\R^n\n\\eeq\nwith $f_i:\\R^n \\to \\R$. This problem arises in many contexts and applications including regression or more generally parameter estimation problems (where $f_i(x)$ is the loss function representing the error between the output and the prediction of a parametric model)~\\cite{Boyd2011AdmmBook,bertsekas2011incremental,bertsekas1997hybrid, Bertsekas1996incremental}, minimization of an expected value of a function (where the expectation is taken over a finite probability distribution or approximated by an $m$-sample average)~\\cite{BottouLecun2005, Leroux2012sgd}, machine learning~\\cite{Jordan13DistLearningApi, Dickenstein2014, Leroux2012sgd}, or distributed optimization over networks~\\cite{RamNedicVeer2007, NedicOzdaglar09,Nedic2007rate}.\n","start":9279,"end":10267}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":["def-Gamma-perm-set"],"declared_external":false,"source_only":false,"title":"独立均匀随机排列","source_units":[{"raw_tex":"An alternative popular approach that works well in practice is following a mixed approach between SGD and IG, sampling the functions randomly but not allowing repetitions, that is sampling the component functions at each iteration \\textit{without-replacement}, or equivalently picking a random order at each cycle. Specifically, at each cycle $k$, we draw a permutation $\\sigma_k $ of $\\{1,2,\\dots,m\\}$ independently and uniformly at random over the set of all permutations \\beq \\Gamma = \\big\\{ \\sigma ~:~ \\sigma \\mbox{ is a permutation of } \\{1,2,\\dots,m\\} \\big\\} \\label{def-Gamma-perm-set}\\eeq \n","start":13220,"end":13817}]},{"id":"S003","semantic_type":"algorithm","kind":"algorithm","labels":["inner-update-wo"],"declared_external":false,"source_only":false,"title":"随机重排迭代","source_units":[{"raw_tex":"and process the functions with this order: %$\\sigma_k = (\\sigma_k(1), \\sigma_k(2),\\dots, \\sigma_k(m)$:\n\\begin{equation}\\label{inner-update-wo} x_{i}^k :={x_{i-1}^k - \\alpha_k \\nabla f_{\\sigma_k(i)} (x_{i-1}^k)} , \\qquad i=1,2,\\dots,m,\n   \\end{equation} \n where $\\alpha_k>0$ is a stepsize. We set $x_0^{k+1} = x_{m}^k$ as before and refer to $\\{x_0^k\\}$ as the {\\it outer iterates}. This method is called the \\textit{Random Reshuffling} (RR) method~\\cite[Section 2.1]{Bertsekas15Book} and will be the focus of this paper.\n","start":13817,"end":14338}]},{"id":"S004","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"逐路径一致渐近记号","source_units":[{"raw_tex":"\\noindent \\textbf{Notation}: We study the point-wise dominance of stochastic sequences by deterministic sequences and use the following notation. Let $x_k = x_k(\\omega)$ be a stochastic real-valued sequence (where $\\omega$ can be thought as the source of randomness) and $y_k$ be a real-valued deterministic sequence. We write $ x_k = \\bigO(y_k)  \\iff \\exists h>0, \\exists k_0 \\quad \\mbox{such that} \\quad  |x_k| \\leq  h |y_k| \\quad \\forall k\\geq k_0, \\forall \\omega,$ where $h$ and $k_0$ are independent of $\\omega$ (Note that the requirement is that this inequality holds for all $\\omega$, not just for almost all $\\omega$). When $x_k$ is non-negative for every $\\omega$, given another deterministic positive sequence $z_k$, we also introduce the inequality version of this definition: \n\t$x_k \\leq y_k + o(z_k) \\iff \\forall \\varepsilon>0, \\exists k_0(\\varepsilon) \\quad \\mbox{such that}\\\t\\quad z_k^{-1} |x_k(\\omega) - y_k| \\leq \\varepsilon, \\quad \\forall k\\geq k_0(\\varepsilon), \\forall \\omega$\nwhere $k_0$ depends on $\\varepsilon$ but is independent of $\\omega$.\tWhen $x_k$ is deterministic, these definitions reduce to the standard definitions of $\\bigO(\\cdot)$ and $o(\\cdot)$ for deterministic sequences. For random $x_k$, the only difference is that we require the constants to be independent of the choice of $\\omega$. For example, if $x_k$ is uniformly distributed over $[0,10]$, we write $x_k = \\bigO(1)$. Throughout the paper, $\\| \\cdot \\|$ denotes the vector or matrix 2-norm (maximum singular value). %The notation $\\E_{\\sigma}X(\\sigma)$ denotes the expectation with respect to a random permutation $\\sigma$ of a random variable $X(\\sigma)$.\n","start":30505,"end":32159}]},{"id":"S005","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"欧氏范数与谱范数","source_units":[{"raw_tex":"where $k_0$ depends on $\\varepsilon$ but is independent of $\\omega$.\tWhen $x_k$ is deterministic, these definitions reduce to the standard definitions of $\\bigO(\\cdot)$ and $o(\\cdot)$ for deterministic sequences. For random $x_k$, the only difference is that we require the constants to be independent of the choice of $\\omega$. For example, if $x_k$ is uniformly distributed over $[0,10]$, we write $x_k = \\bigO(1)$. Throughout the paper, $\\| \\cdot \\|$ denotes the vector or matrix 2-norm (maximum singular value). %The notation $\\E_{\\sigma}X(\\sigma)$ denotes the expectation with respect to a random permutation $\\sigma$ of a random variable $X(\\sigma)$.\n","start":31502,"end":32159}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","labels":["assump-sum-is-str-cvx"],"declared_external":false,"source_only":false,"title":"总目标强凸","source_units":[{"raw_tex":"We consider solving problem (\\ref{pbm-multi-agent}) with RR method with iterations given in (\\ref{inner-update-wo}). Throughout we assume the following:  \n\\begin{assumption}\\label{assump-sum-is-str-cvx} The sum function $f(x)=\\sum_{i=1}^m f_i(x)$ is strongly convex, i.e., there exists a constant $c>0$ such that the function $f(x) - \\frac{c}{2} \\| x\\| ^2$ is convex on $\\R^n$.\\footnote{Such functions arise naturally in support vector machines and other regularized learning algorithms or regression problems (see e.g. \\cite{Leroux2012sgd, Rakhlin:2011, Hogwild11})}.\n\\end{assumption}\n\n\n","start":32369,"end":32957}]},{"id":"S007","semantic_type":"derived_fact","kind":"derived_fact","labels":["def-Hs"],"declared_external":false,"source_only":false,"title":"唯一最优解与可逆曲率","source_units":[{"raw_tex":"Note that this assumption is on the sum function $f$, it does not require the convexity of the individual component functions $f_i$. A consequence of this assumption is that there exists a unique optimal solution to \\eqref{pbm-multi-agent} which we denote by $x^*$. Another consequence is that the Hessian at the optimal solution is invertible since\n\t \\beq \\Hs := \\nabla^2 f(x^*) \\succeq c I_n \\succ 0\n\t\t\\label{def-Hs}\t \n\t \\eeq \nwhere $I_n$\tis the $n\\times n$ identity matrix. \n\n\n","start":32957,"end":33437}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":["def-Ek"],"declared_external":false,"source_only":false,"title":"周期累积梯度误差","source_units":[{"raw_tex":"where \n   \\beq E_k := \\sum_{i=1}^{m} \\bigg( \\nabla f_{\\ski}(x_{i-1}^k) -  \\nabla f_{\\ski} (x_0^k) \\bigg) \\label{def-Ek}\n   \\eeq\nis the cumulative gradient errors associated with the cycle $k$. \n","start":33684,"end":33878}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","labels":["averaged-iter-randomized"],"declared_external":false,"source_only":false,"title":"带误差梯度形式","source_units":[{"raw_tex":"To analyze RR, we view it as a gradient method with random gradient errors and rewrite \nthe outer iterations \\eqref{inner-update-wo} as \n   \\beq \\frac{x_0^k - x_{0}^{k+1}}{\\alpha_k} = \\nabla f(x_0^k) + E_k \\label{averaged-iter-randomized}\n   \\eeq\n","start":33437,"end":33684}]},{"id":"S010","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"外层距离记号","source_units":[{"raw_tex":"A key idea in our analysis is to use a recent upper bound for the convergence rate of cyclic incremental gradient method (see  \\cite{Gur2015IncGrad}), which can be generalized to hold for any fixed deterministic order. This bound implies an almost sure upper bound (in fact one that holds for all sample paths) on the distance of the outer iterates $x_0^k$ generated by RR from the optimal solution $x^*$ denoted by\n\t$\\dist_k  =\\| x_0^k - x^*\\|$\n(see Section \\ref{quad-conv}). Crucially, this result implies an upper bound in expected distance which is asymptotically $m$ times smaller than the almost sure guarantees on the distance of the iterates.\n\n","start":35073,"end":35725}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"外层简单平均","source_units":[{"raw_tex":"In analyzing RR, we will also consider the \\textit{average} of the outer iterate sequence given by\\footnote{It is well known that computing this (moving) average can be done efficiently in a dynamic manner by storing a vector of length $n$.} \n\t$ \\bar{x}_{k} := \\frac{\\sum_{j=0}^{k-1} x_0^j}{k}. \\label{simple-average}\n\t$\n","start":35725,"end":36046}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"迭代后缀平均","source_units":[{"raw_tex":" We also consider averaging only the most recent iterates, i.e. at iteration $k$, averaging the last $qk$ iterates\nfor some constant $q\\in (0,1]$:\n\t$$\\bar{x}_{q,k} := \\frac{\\sum_{j=(1-q)k }^{k-1} x_0^j}{qk}, \\quad 0 < q\\leq 1. $$ \nThe generated sequence is referred to as the \\textit{$q$-suffix average} of the sequence $x_0^k$. For SGD, it was shown that $q$-suffix averaging with $0<q<1$ leads to better performance then averaging (which corresponds to the $q=1$ case by definition), improving the convergence rate in the suboptimality of the function value from $\\log k/k$ to $1/k$~\\cite{Rakhlin:2011, Shamir:2012tj}. This is in line with our results in Section~\\ref{sec:quad-case} which show faster rate for the $0<q<1$ case. The parameter $q$ can be thought as a measure of how much memory one uses during the averaging process. We define the \\textit{$q$-suffix average} of the stepsize in a similar way:\n","start":36046,"end":36956}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["def-averaged-stepsize"],"declared_external":false,"source_only":false,"title":"步长后缀平均","source_units":[{"raw_tex":" \\beq \\bar{\\alpha}_{q,k} =\\frac{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}\n{qk}, \\quad 0<q\\leq 1.\\label{def-averaged-stepsize}\n\\eeq\n","start":36956,"end":37077}]},{"id":"S014","semantic_type":"problem_setting","kind":"problem_setting","labels":["def-fi"],"declared_external":false,"source_only":false,"title":"对称二次分量模型","source_units":[{"raw_tex":"Let $f_i: \\R^n \\to \\R$ be a quadratic function of the form\n    \\beq f_i(x) = \\frac{1}{2} x_i^T P_i x - q_i^T x + r_i, \\quad i=1,2,\\dots, m,\n    \t\\label{def-fi}\n    \\eeq\nwhere $P_i $ is a symmetric $n\\times n$ matrix, $q_i \\in \\R^n$ is a column vector and $r_i$ is a scalar. Note that $f_i$ has Lipschitz gradients, i.e., \n","start":48674,"end":48996}]},{"id":"S015","semantic_type":"derived_fact","kind":"derived_fact","labels":["def-L"],"declared_external":false,"source_only":false,"title":"分量梯度与总光滑界","source_units":[{"raw_tex":"    $$ \\| \\nabla f_i(y) - \\nabla f_i (z) \\| \\leq L_i \\| y - z\\|, \\quad \\forall y, z \\in \\R^n, $$\nwhere $L_i = \\| P_i \\|$. It follows from the triangle inequality that $f$ has Lipschitz gradients with Lipschitz constant at most\n\t\\beq  L := \\sum_{i=1}^m L_i .\n\t\t\\label{def-L}\n\t\\eeq\n","start":48996,"end":49276}]},{"id":"S016","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"常 Hessian 的正定下界","source_units":[{"raw_tex":"Moreover, Assumption \\ref{assump-sum-is-str-cvx} implies that the Hessian matrix of the sum satisfies \n$\\nabla^2 f(x) = \\sum_{i=1}^m \\nabla^2 f_i(x) = \\sum_{i=1}^m P_i \\ge cI_n >0.$\n","start":49276,"end":49458}]},{"id":"S017","semantic_type":"theorem","kind":"theorem","labels":["def-Msigma","rate-prob-one-no-average","theo-rate-quadratics"],"declared_external":true,"declaration_text":["原始前提清单允许 Gur2015IncGrad 的确定性 IG 收敛定理，文中重述为 B0031；完整外部证明未提供。"],"source_only":false,"title":"固定次序 IG 收敛界","source_units":[{"raw_tex":"\\begin{theorem}\\cite{Gur2015IncGrad} \nLet Assumption \\ref{assump-sum-is-str-cvx} hold. Let $f_i(x)$ be a quadratic function of the form $f_i(x) = \\frac{1}{2} x_i^T P_i x - q_i^T x + r_i$\nwhere $P_i $ is a symmetric $n\\times n$ matrix, $q_i \\in \\R^n$ is a column vector and $r_i$ is a scalar for $i=1,2,\\dots,m$. Suppose Assumption \\ref{assump-sum-is-str-cvx}  holds. Consider the iterates $\\{x_0^k\\}$ generated by the iterations \\eqref{inner-update-wo} with a fixed order $\\sigma$ and stepsize $\\alpha_k = R/(k+1)^s$ where $R>0$ and  $s \\in (1/2,1)$. Then\\footnote{The original result in \\cite{Gur2015IncGrad} was stated for $\\sigma=\\{1,2,\\dots,m\\}$ but here we translate this result into an arbitrary permutation $\\sigma$ of $\\{1,2,\\dots,m\\}$ by noting that processing the set of functions $\\{f_1, f_2, \\dots, f_m\\}$ with order $\\sigma$ is equivalent to processing the permuted functions $\\{f_{\\s_1}, f_{\\s_2}, \\dots, f_{\\s_m}\\}$ with order $\\{1,2,\\dots,m\\}$.\n},\n         % \\beq \\dist_k \\leq \\frac{R \\| \\mu(\\sigma)\\| }{c}\\frac{1}{k^s} + o(\\frac{1}{k^s}) \\quad \\mbox{where} \\quad  \\mu(\\sigma) =  \\underset{1\\leq i < j\\leq m}{\\sum} P_{\\sigma(j)} \\nabla f_{\\sigma(i)}(x^*). %\\label{def-Msigma}% \\leq Lm \\Gs\n         % \\eeq\n    \t\\begin{align} \\dist_k &\\leq \\frac{R \\| \\mu(\\sigma)\\| }{c}\\frac{1}{k^s} + o(\\frac{1}{k^s})  &\\mbox{if}& \\quad 1/2<s<1, \\label{rate-prob-one-no-average}\\\\\n           \t\t\t  \\dist_k  &\\leq \\frac{R^2 \\|  \\mu({\\sigma}) \\|}{ Rc - 1} \\frac{1}{k} + o(\\frac{1}{k})  &\\mbox{if}& \\quad s=1 \\mbox{ and } Rc>1,\n\t  %\\label{def-Msigma}% \\leq Lm \\Gs\n    \t\\end{align}\nwhere $c$ is the strong convexity constant of the sum function $f(x)$ and\n\\beq\n\t\\mu(\\sigma) =  - \\underset{1\\leq i < j\\leq m}{\\sum} P_{\\sigma(j)} \\nabla f_{\\sigma(i)}(x^*). \\label{def-Msigma}\n\\eeq  \t\n\t\\label{theo-rate-quadratics} \n\\end{theorem}","start":49907,"end":51726}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":["def-Msigma","theo-rate-quadratics"],"declared_external":false,"source_only":false,"title":"排列误差向量","source_units":[{"raw_tex":"where $c$ is the strong convexity constant of the sum function $f(x)$ and\n\\beq\n\t\\mu(\\sigma) =  - \\underset{1\\leq i < j\\leq m}{\\sum} P_{\\sigma(j)} \\nabla f_{\\sigma(i)}(x^*). \\label{def-Msigma}\n\\eeq  \t\n","start":51482,"end":51682}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":["def-Gs","def-M-gamma"],"declared_external":false,"source_only":false,"title":"最优点梯度与最坏排列常数","source_units":[{"raw_tex":"We next provide some upper bounds on $\\mu(\\sigma)$. We define \n\t\\beqa \\Gs :&=& \\underset{1\\leq i \\leq m}{\\sup} \\| \\nabla f_i(x^*)\\|, \n\t\\label{def-Gs} \\\\\n%\\eeq\n%and \n%\\beq\n\t M_\\Gamma :&=&~ \\sup_{\\sigma \\in \\Gamma} \\|\\mu(\\sigma)\\|.\n\t \\label{def-M-gamma}\t\n\\eeqa\n","start":52449,"end":52708}]},{"id":"S020","semantic_type":"derived_fact","kind":"derived_fact","labels":["def-M-gamma-bound"],"declared_external":false,"source_only":false,"title":"排列误差的统一上界","source_units":[{"raw_tex":"Using  $L_i = \\| P_i \\|$ for each $i$, it follows from the triangle inequality that\n\t\\beq  \\| \\mu(\\sigma) \\| \\leq M_\\Gamma  \\leq \\sup_{\\sigma \\in \\Gamma}  \\underset{1\\leq i < j\\leq m}{\\sum} L_{\\sigma(j)} \\Gs \\leq \\sum_{j=1}^m  (j-1) L_{\\sigma(j)} \\Gs \\leq Lm\\Gs\n\t\t\\label{def-M-gamma-bound}\n\t\\eeq\n","start":52708,"end":53004}]},{"id":"S021","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"RR 最坏界的省略证明说明","source_units":[{"raw_tex":"where $L$ is the Lipschitz constant of the gradient of $f$ defined by \\eqref{def-L}. By replacing $\\mu(\\sigma)$ by $M_\\Gamma$ in Theorem \\ref{theo-rate-quadratics} one can get an upper bound on the worst-case convergence rate that applies to any choice of fixed order $\\sigma$. Using a similar argument along the lines of the proof of Theorem \\ref{theo-rate-quadratics} on the convergence rate of IG, it is straightforward to show that RR never performs any slower than this worst-case convergence rate which is the subject of the next result. The idea is to bound the stochastic $\\dist_k$ sequence from above point-wise. The proof is a simple exercise and is omitted due to space considerations. %left to the reader. % by a reasoning along the lines of the proof of Theorem \\ref{theo-rate-quadratics}, the details of the proof is skipped due to space considerations.\n","start":53004,"end":53872}]},{"id":"S022","semantic_type":"definition","kind":"definition","labels":["def-mu-bar"],"declared_external":false,"source_only":false,"title":"平均排列误差","source_units":[{"raw_tex":"One way to get better constants is to consider convergence in expectation, a weaker notion of convergence compared to almost sure convergence. In the next theorem, we show that $M_\\Gamma$ can be improved to a typically much smaller constant $\\|\\bar{\\mu}\\|$ where \\beq \n\t\\bar{\\mu} := \\E \\big(\\mu(\\sigma_1)\\big) =\n\t\t\t\\frac{\\sum_{\\sigma \\in \\Gamma} \\mu(\\sigma)}{|\\Gamma|}. \\label{def-mu-bar}\n\\eeq can be thought as a measure of average performance over the choice of random permutations. \n","start":57279,"end":57765}]},{"id":"S023","semantic_type":"derived_fact","kind":"derived_fact","labels":["mu-bar-alternative"],"declared_external":false,"source_only":false,"title":"平均误差的分量公式","source_units":[{"raw_tex":"\\begin{remark} A consequence of Lemma \\ref{lem-computing-mu-bar} proved in the Appendix is that \\beq \n  \t\\bar{\\mu} =  \\frac{1}{2} \\sum_{i=1}^m P_i \\nabla f_i(x^*)\n\\label{mu-bar-alternative}.\n\\eeq\nwhere $\\bar{\\mu}$ is defined by \\eqref{def-mu-bar}. By the triangle inequality, \n","start":58784,"end":59061}]},{"id":"S024","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"平均偏差的量级界","source_units":[{"raw_tex":"\t$\\| \\bar{\\mu} \\| \\leq \\sum_{i=1}^m L_i \\Gs = L\\Gs $\n","start":59061,"end":59114}]},{"id":"S025","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"期望界与计算预算比较","source_units":[{"raw_tex":"where $\\Gs$ is defined by \\eqref{def-Gs}. This upper bound is $m$ times smaller than the previous upper bound on $M_\\Gamma$ in \\eqref{def-M-gamma-bound}. \\rev{As an example, consider $s= 1$ with $Rc = 2$. In this case, $L = \\bigO(m)$, $c = \\bigO(m)$, $R = \\bigO(1/m)$ and $\\| \\bar{\\mu}\\| = \\bigO(m)$. We obtain from \\eqref{exp-dist-s-1} that $\\E \\left(\\dist_k\\right) = O(\\frac{1}{mk}) + o(1/k)$. Note that, the performance guarantee for IG from Corollary \\ref{coro-deter-decay-rate-dist} is $\\dist_k = \\bigO(1/k) + o(1/k)$ which is also worse than RR by a factor of $O(m)$. For a fair comparison with the SGD method, we consider running the SGD iterations for $j = km$ iterations so that both RR and SGD methods have access to the same number of component gradients. In this case, expected distance to suboptimality for SGD with the recommended $O(1/j)$ stepsize is $\\bigO(\\frac{1}{\\sqrt{j}})$ where the hidden constants are independent of $m$ (see e.g. \\cite{PolyakJuditsky92,Moulines:2011vy}) which is worse than the $\\bigO(1/j)$ guarantees for RR when $j$ is sufficiently large. These bounds show that when $m$ is small and $k$ is large, IG could outperform SGD in theory, however SGD (and RR) are more suitable for applications when $m$ is large and will admit better bounds compared to IG if $m$ is large enough for a given $k$ fixed\\footnote{We note however that SGD upper bounds are in expectation whereas IG results are deterministic which is a stronger notion of convergence.}}  %This requires $k$ cycles, i.e. $j=km$ evaluations of the gradients of the component functions. \n% \tThe quantity \n% \t$$ Q = L/c$$\n% is called the condition number and is a well-known measure of the difficulty of the optimization problem \\eqref{pbm-multi-agent}. \n\\end{remark}\n","start":59114,"end":60878}]},{"id":"S026","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"附录界的逐路径意图","source_units":[{"raw_tex":"The first lemma is on characterizing what is the worst-case distance of the all the inner iterates of RR to the optimal solution $x^*$. This quantity we want to upper bound is a random variable, but the upper bounds we obtain are deterministic holding for every sample path. This lemma is based on Corollary \\ref{coro-deter-decay-rate-dist} and uses the fact that the distance between the inner iterates are on the order of the stepsize.\n","start":114560,"end":114998}]},{"id":"S027","semantic_type":"assumption","kind":"assumption","labels":["thm-averaged-rate-quad"],"declared_external":false,"source_only":false,"title":"步长与后缀参数","source_units":[{"raw_tex":"where $P_i $ is a symmetric $n\\times n$ matrix, $q_i \\in \\R^n$ is a column vector and $r_i$ is a scalar for $i=1,2,\\dots,m$. Consider the $q$-suffix averages $\\xqk$ of the RR iterates generated by the iterations \\eqref{inner-update-wo} with stepsize $\\alpha_k = \\frac{R}{(k+1)^s}$ where $R>0$ and $s \\in (\\frac{1}{2},1)$. Suppose that Assumption \\ref{assump-sum-is-str-cvx} holds. Then the following statements are true:\n\\begin{enumerate}\n\\item [$(i)$] For any $0 < q \\leq 1$, the $q$-suffix averaged stepsize $\\aqk$ defined in \\eqref{def-averaged-stepsize} satisfies\n","start":65931,"end":66499}]},{"id":"S028","semantic_type":"theorem","kind":"theorem","labels":["def-aqk","def-bqk","def-hat-bqk","lim-rate-aver-quad","limit-main-result","thm-averaged-rate-quad"],"declared_external":false,"source_only":false,"title":"平均 RR 渐近率与去偏差","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm-averaged-rate-quad}  Let $f_i(x)$ be a quadratic function of the form\n\t$$f_i(x) = \\frac{1}{2} x_i^T P_i x - q_i^T x + r_i$$ \nwhere $P_i $ is a symmetric $n\\times n$ matrix, $q_i \\in \\R^n$ is a column vector and $r_i$ is a scalar for $i=1,2,\\dots,m$. Consider the $q$-suffix averages $\\xqk$ of the RR iterates generated by the iterations \\eqref{inner-update-wo} with stepsize $\\alpha_k = \\frac{R}{(k+1)^s}$ where $R>0$ and $s \\in (\\frac{1}{2},1)$. Suppose that Assumption \\ref{assump-sum-is-str-cvx} holds. Then the following statements are true:\n\\begin{enumerate}\n\\item [$(i)$] For any $0 < q \\leq 1$, the $q$-suffix averaged stepsize $\\aqk$ defined in \\eqref{def-averaged-stepsize} satisfies\n    \\beq\\aqk = \\frac{\\aqs}{k^s} + \\bigO(\\frac{1}{k}) \\quad \\mbox{where} \\quad \\aqs = \\frac{1-(1-q)^{1-s}}{q(1-s)} R.\\label{def-aqk}\n    \\eeq\n \\item [$(ii)$] For any $0 < q \\leq 1$, we have \n    \\beq \\lim_{k \\to \\infty} \\frac{\\bar{x}_{q,k} - x^*}{\\aqk} =  -H_*^{-1} \\bar{\\mu}\\quad a.s. \\label{limit-main-result}\n    \\eeq\nwhere $\\bar{\\mu}$ is given by \\eqref{mu-bar-alternative}, i.e., the normalized error $(\\bar{x}_{q,k} - x^*)/\\aqk$  converges to the constant vector $-H_*^{-1} \\bar{\\mu}$ almost surely where $\\Hs=\\sum_{i=1}^m P_i$ is the Hessian matrix at the optimal solution and $\\bar{\\mu}$ is given by \\eqref{mu-bar-alternative}.\n% \\beq      \n%     \\bar{\\mu} =  \\frac{1}{2} \\sum_{i=1}^m P_i \\nabla f_i(x^*). \n%    \\eeq\nThen, from part $(i)$,  \n \\beq \\lim_{k \\to \\infty} k^s (\\bar{x}_{q,k} - x^*) =  -\\aqs H_*^{-1} \\bar{\\mu} \\quad a.s.\\label{lim-rate-aver-quad}\n \\eeq\nHence, the $q$-suffix averaged iterates $\\xqk$ converge to the optimal solution $x^*$ with rate $1/k^s$ almost surely. \n%where $\\Hs=\\sum_{i=1}^m P_i$ is the Hessian matrix at the optimal solution, \n%\\beq      \n%    \\aqs =  \\frac{1-(1-q)^{1-s}}{q} R \\quad \\mbox{and} \\quad \\bar{\\mu} =  \\frac{1}{2} \\sum_{i=1}^m P_i \\nabla f_i(x^*). \n%   \\eeq\n   \n \\item [$(iii)$] With probability at least $1-\\delta$, we have\n\t $$ \\bar{x}_{q,k} -  x^*  =  b_{q,k} + \\bigO \\bigg(\\frac{\\sqrt{\\log(1/\\delta)}}{k} \\bigg) +  \\begin{cases} \\bigO \\big(\\frac{\\log k}{k} \\big)  & \\mbox{if} \\quad  q=1 \\\\ \\bigO(\\frac{1}{k}\\big) & \\mbox{if} \\quad  0<q<1, \\end{cases} $$ where\n\t \\beq b_{q,k} = -\\bar{\\alpha}_{q,k} H_*^{-1}  \\bar{\\mu} \n\t \t\t\\label{def-bqk}\n\t \\eeq \nis deterministic, $\\bar{\\mu}$ is given by $\\eqref{mu-bar-alternative}$ and $\\bar{\\alpha}_{q,k}$ is the averaged stepsize defined in \\eqref{def-averaged-stepsize}. The constants hidden by $\\bigO(\\cdot)$ depend only on $\\Gs, L, m, R, c, q$ and $s$.\n\t \\item [$(iv)$] Let \\beq  \\hat{b}_{q,k} = -\\bar{\\alpha}_{q,k}\\bigg[\\sum_{i=1}^{m} P_{\\ski}\\bigg]^{-1} \\sum_{i=1}^{m} P_{\\ski} \\nabla f_{\\ski} (x_{i-1}^k)/2. \\label{def-hat-bqk}\n\t \\eeq\nwhere $\\bar{\\alpha}_{q,k}$ is the averaged stepsize defined in \\eqref{def-averaged-stepsize}. Then, \t \n\t $\\hat{b}_{q,k}  = b_{q,k} + \\bigO(\\alpha_k^2). $\nIt follows from part $(ii)$ that with probability at least $1-\\delta$,\n\t  $$ (\\bar{x}_{q,k} - \\hat{b}_{q,k}) -  x^*  = \\bigO \\bigg(\\frac{\\sqrt{\\log(1/\\delta)}}{k} \\bigg) +  \\begin{cases} \\bigO \\big(\\frac{\\log k}{k} \\big)  & \\mbox{if} \\quad  q=1 \\\\ \\bigO(\\frac{1}{k}\\big) & \\mbox{if} \\quad  0<q<1. \\end{cases} $$ \n\\end{enumerate} \n \n \\end{theorem}","start":65780,"end":69036}]},{"id":"S029","semantic_type":"definition","kind":"definition","labels":["def-bqk","thm-averaged-rate-quad"],"declared_external":false,"source_only":false,"title":"确定性偏差项","source_units":[{"raw_tex":"\t \\beq b_{q,k} = -\\bar{\\alpha}_{q,k} H_*^{-1}  \\bar{\\mu} \n\t \t\t\\label{def-bqk}\n\t \\eeq \nis deterministic, $\\bar{\\mu}$ is given by $\\eqref{mu-bar-alternative}$ and $\\bar{\\alpha}_{q,k}$ is the averaged stepsize defined in \\eqref{def-averaged-stepsize}. The constants hidden by $\\bigO(\\cdot)$ depend only on $\\Gs, L, m, R, c, q$ and $s$.\n","start":68018,"end":68351}]},{"id":"S030","semantic_type":"definition","kind":"definition","labels":["def-hat-bqk","thm-averaged-rate-quad"],"declared_external":false,"source_only":false,"title":"周期内偏差估计","source_units":[{"raw_tex":"\t \\item [$(iv)$] Let \\beq  \\hat{b}_{q,k} = -\\bar{\\alpha}_{q,k}\\bigg[\\sum_{i=1}^{m} P_{\\ski}\\bigg]^{-1} \\sum_{i=1}^{m} P_{\\ski} \\nabla f_{\\ski} (x_{i-1}^k)/2. \\label{def-hat-bqk}\n\t \\eeq\nwhere $\\bar{\\alpha}_{q,k}$ is the averaged stepsize defined in \\eqref{def-averaged-stepsize}. Then, \t \n","start":68351,"end":68639}]},{"id":"S031","semantic_type":"corollary","kind":"corollary","labels":["coro-deter-decay-rate-dist"],"declared_external":false,"source_only":false,"title":"RR 最坏路径距离界","source_units":[{"raw_tex":"\\begin{corollary}\\label{coro-deter-decay-rate-dist} Under the setting of Theorem \\ref{theo-rate-quadratics}, if $\\sigma$ is sampled uniformly at each cycle instead of being kept fixed, then\n\t\t% \\beqs \\  \\dist_k \\leq \\frac{R M_\\Gamma}{c}\\frac{1}{k^s} + o(\\frac{1}{k^s})%, \\quad M_\\Gamma = \\sup_{\\sigma \\in \\Gamma} \\mu(\\sigma)\n  %        \\eeqs \n         \\begin{align} \\dist_k &\\leq \\frac{R M_\\Gamma }{c}\\frac{1}{k^s} + o(\\frac{1}{k^s})  &\\mbox{if}& \\quad 1/2<s<1,\\\\\n           \t\t\t  \\dist_k  &\\leq \\frac{R^2 M_\\Gamma}{ Rc - 1} \\frac{1}{k} + o(\\frac{1}{k})  &\\mbox{if}& \\quad s=1 \\mbox{ and } Rc>1,\n\t  %\\label{def-Msigma}% \\leq Lm \\Gs\n    \t\\end{align}\nwith probability one where $M_\\Gamma$ is deterministic and is defined by \\eqref{def-M-gamma}.\n%the right-hand side is a deterministic sequence and         \n%  \t    $$M_\\Gamma \\leq Lm \\Gs, \\quad \\Gs = \\underset{1\\leq i \\leq m}{\\sup} \\| \\nabla f_i(x^*)\\|.$\n\\end{corollary}","start":55773,"end":56691}]},{"id":"S032","semantic_type":"proof","kind":"proof","labels":["Yqk-helper","average-sum-bound-whp","average-sum-bound-whp-3","averaged-iter-random-2","def-Yqk","grad-as-hessian-prod-quadratics","grad-diff-quadratic","iterate-eq-randomized","marting-diff-as-bound","square-summable-step","xk-averaged-asymptotics-2","xk-biased-high-probability","xk-biased-high-probability-3","xqk-high-proba"],"declared_external":false,"source_only":false,"title":"主定理完整作者证明","source_units":[{"raw_tex":"\\begin{proof} \n\t\\begin{itemize} \n\t\\item [$(i)$] As the stepsize sequence is monotonically decreasing, we have the bounds \n   \\beqas \\int_{(1-q)k}^k  \\frac{R}{(x+2)^s} dx \\leq  \\sum_{j=(1-q)k}^k \\alpha_j = \\sum_{j=(1-q)k}^k \\frac{R}{(k+1)^s} %&\\leq& R + \\sum_{j=(1-q)k + 1}^k \\frac{R}{(k+1)^s} \\\\\n   \t\t\t &\\leq& R + \\int_{(1-q)k}^{k-1}  \\frac{R}{(x+1)^s} dx. \n   %1 + \\int_{(1-q)k}^k  \\frac{R}{(x+1)^s} dx. \\\\\n   \\eeqas\nDividing each term by $qk$, after a straightforward integration we obtain\n  $$ \\aqk = \\frac{ k^{1-s} - \\big((1-q)k+1\\big)^{1-s} + \\bigO(1)}{(1-s)qk} R  = \\frac{a_q(s)}{k^s} + \\bigO(\\frac{1}{k}). \n   $$\n which completes the proof.  \n \t\t\t\\item [$(ii)$] Taking the $q$-suffix averages of both sides of \\eqref{averaged-iter-randomized}, we obtain \n\\beq \n\tI_{q,k} := \\frac{\\sum_{j=(1-q)k}^{k-1} {(x_0^j - x_{0}^{j+1})}{\\alpha_j^{-1}}}{qk} = \\frac{ \\sum_{j=(1-q)k}^{k-1}   \\nabla f(x_0^j)+ E_j}{qk}. \\label{averaged-iter-random-2}\n   \\eeq \nAs $f$ is a quadratic, the first order Taylor series for the gradient of $f$ is exact:\n  \\beq \\nabla f(x_0^j) = H_* (x_0^j-x^*).\n  \t\t\\label{grad-as-hessian-prod-quadratics}\n  \\eeq\n Therefore, \\eqref{averaged-iter-random-2} becomes  $\nI_{q,k} \n= \\frac{ \\sum_{j=(1-q)k}^{k-1}   H_* (x_0^j - x^*)   + E_j }{qk} $\n%\\label{averaged-iter-random-3} \nwhich is equivalent to\n\\beqa\n                    I_{q,k} &=& H_*(\\bar{x}_{q,k} - x^*) + \\frac{\\sum_{j=(1-q)k}^{k-1} E_j}{qk} = H_*(\\bar{x}_{q,k} - x^*) + \\bar{\\alpha}_{q,k}  Y_{q,k}\\label{iterate-eq-randomized}\n   \\eeqa \nwhere $Y_{q,k}$ is defined as\n    \\beq Y_{q,k} := \\frac{1}{\\bar{\\alpha}_{q,k}} \\frac{\\sum_{j=(1-q)k}^{k-1} E_j }{qk } =  \\frac{\\sum_{j=(1-q)k}^{k-1} E_j}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}\t\n\t\t\\label{def-Yqk}    \n     \\eeq\nand can be interpreted as the ($q$-suffix) averaged gradient error sequence $E_j$  normalized by the ($q$-suffix) averaged stepsize sequence $\\alpha_j$. Since $H_*$ is invertible by the strong convexity of $f$ (see \\eqref{def-Hs}), we can rewrite \\eqref{iterate-eq-randomized} as\n\\beqa  \\bar{x}_{q,k} - x^* &=& \n                     - H_*^{-1} \\aqk \\Yqk  + H_*^{-1} \\Iqk  \n            = - H_*^{-1} \\bar{\\alpha}_{q,k} Y_{q,k}  +  \\begin{cases} \\bigO(\\frac{1}{k})  & \\mbox{if } \\quad 0<q<1 \\\\\n\\bigO( \\frac{\\log k}{k})  & \\mbox{if } \\quad q = 1.\n\\end{cases} \\label{xk-averaged-asymptotics-2}\n              %           &=& \\bigO(\\frac{\\log k}{k}) + \\bigO(\\frac{1}{k}) - H_*^{-1} \\bigg(   \\frac{\\sum_{j=0}^{k-1} E_j}{\\sum_{j=0}^{k-1}\\alpha_j}\\bigg) \\frac{\\sum_{j=0}^{k-1}\\alpha_j}{k}   \\nonumber \n   \\eeqa              \nwhere we used the inequality $\\| H_*^{-1}\\| \\leq 1/c $ implied by \\eqref{def-Hs} and Lemma \\ref{lem-I-lk} from the appendix to provide an upper bound for the second term in the first equality. Note that, as a consequence of Lemma \\ref{lem-I-lk}, $\\bigO(\\cdot)$ notation above hides a constant that depends only on the parameters $\\Gs, L,c, m, R, s, q$ and also $\\dist_0$ when $q=1$. Then, dividing both sides of \\eqref{xk-averaged-asymptotics-2} by $\\aqk$, taking limits as $k$ goes to infinity,  using part $(i)$ on the asymptotic behavior of $\\aqk$ and the fact that $Y_{q,k} \\to \\bar{\\mu}$ a.s. from Lemma \\ref{lem-martingale-variance}, we obtain the claimed result.%$$ \\lim_{k \\to \\infty} k^s (\\bar{x}_{q,k} - x^*) = a_q(s) H_*^{-1} \\bar{\\mu}. $$\n%This completes the proof of part $(i)$.\n%Taking expectations, \n%\n%$$ \\E (\\bar{x}_k - x^*)  = \\bigO(\\frac{\\log k}{k}) -  H_*^{-1} \\bigg(   \\frac{\\sum_{j=0}^{k-1} \\alpha_j \\E v(u^j) }{k}\\bigg) + \\bigO(\\alpha_k^2).  $$\n%\n%\n%which implies\n%$$ \\E (\\bar{x}_k - x^*)  = \\bigO(\\frac{\\log k}{k}) -  b_* \\bigg(   \\frac{\\sum_{j=0}^{k-1} \\alpha_j  }{k}\\bigg) + \\bigO(\\alpha_k^2).  $$ \n%$ b_* = H_*^{-1}\\frac{\\sum_{r=1}^m \\nabla^2 f_r(x^*) \\nabla f_r(x^*)}{2}.$\n%Suppose we can compute an approximation $b_k$ to $b_*$ satisfying\n%   $b_k = b_* + \\bigO(\\alpha_k).$\n%Then,    \n%$$ \\E \\bigg(\\bar{x}_k + b_k \\bigg(   \\frac{\\sum_{j=0}^{k-1} \\alpha_j  }{k}\\bigg)  - x^*\\bigg)  = \\bigO(\\frac{\\log k}{k})  + \\bigO(\\alpha_k^2).  $$ \n%\n%NEED TO LOOK AT THE VARIANCE TERM AS WELL. USE OLD MARTINGALE APPROACH FOR BOUNDING THE VA\n\n \\item [$(iii)$] By parts $(i)$ and $(iii)$ of Lemma \\ref{lem-martingale-variance} from the appendix that relates the gradient error sequence $E_j$ to a sequence of i.i.d. variables $\\mu(\\sigma_j)$, for $0<q\\leq 1$,  \n\t\t\\beq Y_{q,k} = \\frac{\\sum_{j=(1-q)k}^{k-1} E_j}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j} =  \\frac{\\sum_{i=(1-q)k}^{k-1} \\alpha_j \\mu(\\sigma_j) + \\bigO(\\alpha_j^2)}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}. \t \n\t\t\t\\label{Yqk-helper}  \n     \\eeq \nWe first give a proof for $q=1$, the proof for the remaining $q \\in (0,1)$ case will be similar. Assume $q=1$. Plugging $q=1$ and \\eqref{Yqk-helper} into \\eqref{xk-averaged-asymptotics-2}, we obtain\n     \\beqa \\bar{x}_{1,k} - x^* \n     &=&\\bigO(\\frac{\\log k}{k})  - H_*^{-1} \\aonek \\Yonek \\nonumber \\\\\n     &=& \\bigO(\\frac{\\log k}{k}) - H_*^{-1} \\bigg(   \\frac{\\sum_{j=0}^{k-1} \\alpha_j \\big(\\mu(\\s_j) - \\bar{\\mu}\\big)}{k} +  \\frac{\\sum_{j=0}^{k-1}  \\alpha_j \\bar{\\mu} + \\bigO(\\alpha_j^2) }{k}  \\bigg) \\nonumber \\\\\n     \t\t\t\t\t\t &=& b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{\\sum_{j=0}^{k-1}  \\alpha_j (\\mu(\\s_j) - \\bar{\\mu})}{k} - H_*^{-1} \\sum_{j=0}^{k-1}  \\frac{\\bigO(\\alpha_j^2)}{k} \\nonumber \\\\     \n     \t\t\t\t\t\t    &=& b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{\\sum_{j=0}^{k-1}  \\alpha_j (\\mu(\\s_j) - \\bar{\\mu})}{k} \\label{xqk-high-proba} %\\\\     \t\t\t\t\t\t    \n     \t\t\t\t\t\t%    &=& b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{M_{1,k}}{k}  \\label{xk-biased-high-probability}\n     \\eeqa    \nwhere $\\bonek$ is defined by \\eqref{def-bqk} and we used in the last step the fact that for $s>1/2$ \n\t\\beq \\sum_{j=0}^\\infty \\alpha_j^2 = \\sum_{j=1}^\\infty \\frac{R^2}{j^{2s}} = R^2 \\zeta(2s) < \\infty\n\t\t\\label{square-summable-step}\t\n\t\\eeq \nwhere $\\zeta(\\cdot)$ is the Riemann-Zeta function. We now study the asymptotic behavior of the last summation term in \\eqref{xqk-high-proba} by introducing the process\n\t$S_{1,k} = \\sum_{j=0}^{k-1} Z_j$, where $Z_j := \\alpha_j (\\mu(\\s_j) - \\bar{\\mu})$ and $k\\geq 0$ with the convention that $S_{1,0} = 0$. Equipped with this definition, \\eqref{xqk-high-proba} becomes\n  \\beq \n  \t\t \\xonek - \\xs = b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{S_{1,k}}{k}.  \\label{xk-biased-high-probability}\n  \\eeq\t\t  \nThe random variables $Z_j$ are independent, centered and have an identical distribution up to the scaling factor $\\alpha_j$. Therefore,  $S_{1,k}$ is a sum of  centered random variables satisfying:\n    \\beqa \\begin{scriptsize} \\| S_{1,k} - S_{1,k-1} \\| =\\big \\|\\alpha_{k-1}\\big(\\mu(\\sigma_{k-1})-\\bar{\\mu}\\big)\\big\\| \n    \\leq \\gamma_{k-1}:=\\alpha_{k-1}LmG_* \\label{marting-diff-as-bound}\n    \\end{scriptsize}\n    \\eeqa \nwhere we used \\eqref{def-mu-sk-upper-bound} in the last inequality (see also Lemma \\ref{lem-computing-mu-bar}). Then, by the Azuma-Hoeffding inequality, for every $t>0$, \n  \\beqas \\P \\bigg( \\big\\|\\frac{S_{1,k}}{k}\\big\\| > \\frac{t}{k}\\bigg) &\\leq& 2 \\exp \\bigg( - \\frac{t^2}{2\\sum_{j=0}^{k-1} \\gamma_j^2} \\bigg) =  2 \\exp \\big( - \\frac{t^2}{ \\beta }\\big) \n  \\eeqas\nwhere $\\beta = 2\\sum_{j=0}^{\\infty} \\gamma_j^2 < \\infty$ as $\\alpha_j$ is square-summable (see \\eqref{square-summable-step}). Note that $\\beta$ depends only on $\\Gs, L,m$ and the stepsize parameters $R$ and $s$. It is easy to see that selecting $t \\geq t_\\delta = \\sqrt{\\beta  \\log(2/\\delta)}$ makes the right-hand side $\\leq \\delta$.  Therefore for any $\\delta>0$, with probability at least $1-\\delta$, \n\t\t\\beq \\bigl \\| \\frac{S_{1,k}}{k} \\bigr \\| \\leq \\frac{\\sqrt{\\beta  \\log(2/\\delta)}}{k}\n\t\t\t\\label{average-sum-bound-whp}\n\t\t\\eeq\nwhich if inserted into the expression \\eqref{xk-biased-high-probability} completes the proof for the $q=1$ case. For  $0<q<1$ case, the same line of reasoning applies except that we replace $b_{1,k}$ with $b_{q,k}$ and we can improve the $\\bigO( \\log k /k)$ term in the expression \\eqref{xk-biased-high-probability} to $\\bigO(1/k)$, this is justified by \\eqref{xk-averaged-asymptotics-2}. Then, this leads to \n\t\t \\beq \n  \t\t \\xqk - \\xs = b_{q,k}  +  \\bigO(\\frac{1}{k})  - H_*^{-1}   \\frac{S_{q,k}}{qk}  \\label{xk-biased-high-probability-3}\n  \\eeq\nwhere $S_{q,k} := \\sum_{j=(1-q)k}^{k-1} Z_j = S_{1,k} - S_{1,(1-q)k}$ is the $q$-suffix cumulative sum (cumulative sum of the last $qk$ terms) of the sequence $Z_{k}$. Then using \\eqref{average-sum-bound-whp}, with probability at least $1-\\delta$,\n\\beq \\bigl \\| \\frac{S_{q,k}}{k} \\bigr \\| \\leq \\| \\frac{S_{1,k}}{k} \\bigr \\|  + \\| \\frac{S_{1,(1-q)k}}{k} \\bigr \\|  \\leq \\frac{2 t_\\delta}{k}.\n\t\t\t\\label{average-sum-bound-whp-3}\n\t\t\\eeq\nPlugging this high probability bound into \\eqref{xk-biased-high-probability-3}, we conclude. \t\t\n\t\\item [$(iv)$] By Lemma \\ref{lem-random-tail-bound}, we have $\\underset{1\\leq i < m}{\\max} \\|x_{i-1}^k - x^*\\| = \\bigO(\\alpha_k)$. Therefore,\n\t \\beqa \\scriptsize \\| \\nabla f_{\\ski} (x_{i-1}^k)  - \\nabla f_{\\ski} (x^*) \\|  = \\bigO(\\alpha_k) \\label{grad-diff-quadratic}\n\t \\eeqa\nfor any $i=1,2,\\dots,m$. As a consequence, \t\n\t\\beqas \\hat{b}_{q,k} &=& -\\bar{\\alpha}_{q,k} H_*^{-1} \\sum_{i=1}^{m} P_{\\ski}  \\bigg(\\nabla f_{\\ski} (x^*) + \\bigO(\\alpha_k)\\bigg)  \\\\\n\t                         &=& - \\bar{\\alpha}_{q,k} H_*^{-1} \\sum_{j=1}^{m} P_{j} \\nabla f_j (x^*)   + \\bigO(\\alpha_k^2)   \n%\t                         &=& -\\bigg( H_*^{-1} + \\bigO(\\alpha_k) \\bigg) \\sum_{i=1}^{m} \\nabla^2 f_{i} (x_*) \\nabla f_i (x^*)  \\bar{\\alpha}_{q,k}+ \\bigO(\\alpha_k^2) +  o(\\alpha_k^2) \\\\\n\t                         = b_{q,k} + \\bigO(\\alpha_k^2)\t                         \t                        \t                         \n\t\\eeqas\t\t \nwhere in the second equality we use the fact that $\\bar{\\alpha}_{q,k}= \\bigO(1/k^s) = \\bigO(\\alpha_k)$ implied by part $(i)$.\n%\t\t\\item [$(ii)$]  By taking norm square of both sides and expectations in \\eqref{xk-averaged-asymptotics} and applying Cauchy-Schwartz gives\n%\t\t\t\\beqas \\E \\| \\bar{x}_k - x^*\\|^2 &\\leq& 4\\bigO(\\frac{\\log k}{k}) + 4\\bigO(\\frac{1}{k}) + 4 \\E \\bigg\\|  H_*^{-1} \\bigg(   \\frac{\\sum_{j=0}^{k-1} E_j}{k}\\bigg)\\bigg\\|^2 + 4\\bigO(\\alpha_k^2)   \\\\\n%\t\t\t                                              &\\leq& 4\\bigO(\\frac{\\log k}{k}) + 4\\bigO(\\frac{1}{k}) + \\frac{4}{c^2} \\E \\bigg\\|   \\frac{\\sum_{j=0}^{k-1} E_j}{k}\\bigg\\|^2 + 4\\bigO(\\alpha_k^2) \n%\t\t\t\\eeqas\n%We then decompose the term inside the expectation as\n%    \\beqas \\frac{\\sum_{j=0}^{k-1} E_j}{k} &=& \\frac{\\sum_{j=0}^{k-1} \\alpha_j v(u^j) - \\alpha_j \\bar{\\mu}}{k} + \\frac{\\sum_{j=0}^{k-1} \\alpha_j \\bar{\\mu} + \\bigO(\\alpha_j^2)}{k} \\\\\n%               &=& \\frac{\\sum_{j=0}^{k-1} \\alpha_j v(u^j) - \\alpha_j \\bar{\\mu}}{\\sigma_* (\\sum_{j=0}^{k-1}\\alpha_j^2)^{1/2}} \\frac{\\sigma_* (\\sum_{j=0}^{k-1}\\alpha_j^2)^{1/2}}{k} + \\frac{\\sum_{j=0}^{k-1} \\alpha_j \\bar{\\mu} + \\bigO(\\alpha_j^2)}{k}\n%    \\eeqas\n\\end{itemize}\n\\end{proof}","start":69040,"end":79865}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"积分比较平均步长","source_units":[{"raw_tex":"\\begin{proof} \n\t\\begin{itemize} \n\t\\item [$(i)$] As the stepsize sequence is monotonically decreasing, we have the bounds \n   \\beqas \\int_{(1-q)k}^k  \\frac{R}{(x+2)^s} dx \\leq  \\sum_{j=(1-q)k}^k \\alpha_j = \\sum_{j=(1-q)k}^k \\frac{R}{(k+1)^s} %&\\leq& R + \\sum_{j=(1-q)k + 1}^k \\frac{R}{(k+1)^s} \\\\\n   \t\t\t &\\leq& R + \\int_{(1-q)k}^{k-1}  \\frac{R}{(x+1)^s} dx. \n   %1 + \\int_{(1-q)k}^k  \\frac{R}{(x+1)^s} dx. \\\\\n   \\eeqas\nDividing each term by $qk$, after a straightforward integration we obtain\n  $$ \\aqk = \\frac{ k^{1-s} - \\big((1-q)k+1\\big)^{1-s} + \\bigO(1)}{(1-s)qk} R  = \\frac{a_q(s)}{k^s} + \\bigO(\\frac{1}{k}). \n   $$\n which completes the proof.  \n","start":69040,"end":69690}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":["averaged-iter-random-2"],"declared_external":false,"source_only":false,"title":"平均周期梯度等式","source_units":[{"raw_tex":" \t\t\t\\item [$(ii)$] Taking the $q$-suffix averages of both sides of \\eqref{averaged-iter-randomized}, we obtain \n\\beq \n\tI_{q,k} := \\frac{\\sum_{j=(1-q)k}^{k-1} {(x_0^j - x_{0}^{j+1})}{\\alpha_j^{-1}}}{qk} = \\frac{ \\sum_{j=(1-q)k}^{k-1}   \\nabla f(x_0^j)+ E_j}{qk}. \\label{averaged-iter-random-2}\n   \\eeq \n","start":69690,"end":69992}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":["grad-as-hessian-prod-quadratics"],"declared_external":false,"source_only":false,"title":"二次梯度精确线性化","source_units":[{"raw_tex":"As $f$ is a quadratic, the first order Taylor series for the gradient of $f$ is exact:\n  \\beq \\nabla f(x_0^j) = H_* (x_0^j-x^*).\n  \t\t\\label{grad-as-hessian-prod-quadratics}\n  \\eeq\n","start":69992,"end":70172}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":["def-Yqk","iterate-eq-randomized"],"declared_external":false,"source_only":false,"title":"平均误差的曲率分解","source_units":[{"raw_tex":" Therefore, \\eqref{averaged-iter-random-2} becomes  $\nI_{q,k} \n= \\frac{ \\sum_{j=(1-q)k}^{k-1}   H_* (x_0^j - x^*)   + E_j }{qk} $\n%\\label{averaged-iter-random-3} \nwhich is equivalent to\n\\beqa\n                    I_{q,k} &=& H_*(\\bar{x}_{q,k} - x^*) + \\frac{\\sum_{j=(1-q)k}^{k-1} E_j}{qk} = H_*(\\bar{x}_{q,k} - x^*) + \\bar{\\alpha}_{q,k}  Y_{q,k}\\label{iterate-eq-randomized}\n   \\eeqa \nwhere $Y_{q,k}$ is defined as\n    \\beq Y_{q,k} := \\frac{1}{\\bar{\\alpha}_{q,k}} \\frac{\\sum_{j=(1-q)k}^{k-1} E_j }{qk } =  \\frac{\\sum_{j=(1-q)k}^{k-1} E_j}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}\t\n\t\t\\label{def-Yqk}    \n     \\eeq\nand can be interpreted as the ($q$-suffix) averaged gradient error sequence $E_j$  normalized by the ($q$-suffix) averaged stepsize sequence $\\alpha_j$. Since $H_*$ is invertible by the strong convexity of $f$ (see \\eqref{def-Hs}), we can rewrite \\eqref{iterate-eq-randomized} as\n","start":70172,"end":71055}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":["xk-averaged-asymptotics-2"],"declared_external":false,"source_only":false,"title":"控制平均位移余项","source_units":[{"raw_tex":"\\beqa  \\bar{x}_{q,k} - x^* &=& \n                     - H_*^{-1} \\aqk \\Yqk  + H_*^{-1} \\Iqk  \n            = - H_*^{-1} \\bar{\\alpha}_{q,k} Y_{q,k}  +  \\begin{cases} \\bigO(\\frac{1}{k})  & \\mbox{if } \\quad 0<q<1 \\\\\n\\bigO( \\frac{\\log k}{k})  & \\mbox{if } \\quad q = 1.\n\\end{cases} \\label{xk-averaged-asymptotics-2}\n              %           &=& \\bigO(\\frac{\\log k}{k}) + \\bigO(\\frac{1}{k}) - H_*^{-1} \\bigg(   \\frac{\\sum_{j=0}^{k-1} E_j}{\\sum_{j=0}^{k-1}\\alpha_j}\\bigg) \\frac{\\sum_{j=0}^{k-1}\\alpha_j}{k}   \\nonumber \n   \\eeqa              \nwhere we used the inequality $\\| H_*^{-1}\\| \\leq 1/c $ implied by \\eqref{def-Hs} and Lemma \\ref{lem-I-lk} from the appendix to provide an upper bound for the second term in the first equality. Note that, as a consequence of Lemma \\ref{lem-I-lk}, $\\bigO(\\cdot)$ notation above hides a constant that depends only on the parameters $\\Gs, L,c, m, R, s, q$ and also $\\dist_0$ when $q=1$. Then, dividing both sides of \\eqref{xk-averaged-asymptotics-2} by $\\aqk$, taking limits as $k$ goes to infinity,  using part $(i)$ on the asymptotic behavior of $\\aqk$ and the fact that $Y_{q,k} \\to \\bar{\\mu}$ a.s. from Lemma \\ref{lem-martingale-variance}, we obtain the claimed result.%$$ \\lim_{k \\to \\infty} k^s (\\bar{x}_{q,k} - x^*) = a_q(s) H_*^{-1} \\bar{\\mu}. $$\n","start":71055,"end":72341}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"取得几乎必然极限","source_units":[{"raw_tex":"where we used the inequality $\\| H_*^{-1}\\| \\leq 1/c $ implied by \\eqref{def-Hs} and Lemma \\ref{lem-I-lk} from the appendix to provide an upper bound for the second term in the first equality. Note that, as a consequence of Lemma \\ref{lem-I-lk}, $\\bigO(\\cdot)$ notation above hides a constant that depends only on the parameters $\\Gs, L,c, m, R, s, q$ and also $\\dist_0$ when $q=1$. Then, dividing both sides of \\eqref{xk-averaged-asymptotics-2} by $\\aqk$, taking limits as $k$ goes to infinity,  using part $(i)$ on the asymptotic behavior of $\\aqk$ and the fact that $Y_{q,k} \\to \\bar{\\mu}$ a.s. from Lemma \\ref{lem-martingale-variance}, we obtain the claimed result.%$$ \\lim_{k \\to \\infty} k^s (\\bar{x}_{q,k} - x^*) = a_q(s) H_*^{-1} \\bar{\\mu}. $$\n","start":71590,"end":72341}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":["Yqk-helper"],"declared_external":false,"source_only":false,"title":"代入独立误差主项","source_units":[{"raw_tex":" \\item [$(iii)$] By parts $(i)$ and $(iii)$ of Lemma \\ref{lem-martingale-variance} from the appendix that relates the gradient error sequence $E_j$ to a sequence of i.i.d. variables $\\mu(\\sigma_j)$, for $0<q\\leq 1$,  \n\t\t\\beq Y_{q,k} = \\frac{\\sum_{j=(1-q)k}^{k-1} E_j}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j} =  \\frac{\\sum_{i=(1-q)k}^{k-1} \\alpha_j \\mu(\\sigma_j) + \\bigO(\\alpha_j^2)}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}. \t \n\t\t\t\\label{Yqk-helper}  \n     \\eeq \n","start":73148,"end":73594}]},{"id":"S040","semantic_type":"derived_fact","kind":"derived_fact","labels":["square-summable-step"],"declared_external":false,"source_only":false,"title":"平方步长可求和","source_units":[{"raw_tex":"\t\\beq \\sum_{j=0}^\\infty \\alpha_j^2 = \\sum_{j=1}^\\infty \\frac{R^2}{j^{2s}} = R^2 \\zeta(2s) < \\infty\n\t\t\\label{square-summable-step}\t\n\t\\eeq \n","start":74723,"end":74861}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":["xqk-high-proba"],"declared_external":false,"source_only":false,"title":"全平均的偏差与随机余项","source_units":[{"raw_tex":"We first give a proof for $q=1$, the proof for the remaining $q \\in (0,1)$ case will be similar. Assume $q=1$. Plugging $q=1$ and \\eqref{Yqk-helper} into \\eqref{xk-averaged-asymptotics-2}, we obtain\n     \\beqa \\bar{x}_{1,k} - x^* \n     &=&\\bigO(\\frac{\\log k}{k})  - H_*^{-1} \\aonek \\Yonek \\nonumber \\\\\n     &=& \\bigO(\\frac{\\log k}{k}) - H_*^{-1} \\bigg(   \\frac{\\sum_{j=0}^{k-1} \\alpha_j \\big(\\mu(\\s_j) - \\bar{\\mu}\\big)}{k} +  \\frac{\\sum_{j=0}^{k-1}  \\alpha_j \\bar{\\mu} + \\bigO(\\alpha_j^2) }{k}  \\bigg) \\nonumber \\\\\n     \t\t\t\t\t\t &=& b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{\\sum_{j=0}^{k-1}  \\alpha_j (\\mu(\\s_j) - \\bar{\\mu})}{k} - H_*^{-1} \\sum_{j=0}^{k-1}  \\frac{\\bigO(\\alpha_j^2)}{k} \\nonumber \\\\     \n     \t\t\t\t\t\t    &=& b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{\\sum_{j=0}^{k-1}  \\alpha_j (\\mu(\\s_j) - \\bar{\\mu})}{k} \\label{xqk-high-proba} %\\\\     \t\t\t\t\t\t    \n     \t\t\t\t\t\t%    &=& b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{M_{1,k}}{k}  \\label{xk-biased-high-probability}\n     \\eeqa    \nwhere $\\bonek$ is defined by \\eqref{def-bqk} and we used in the last step the fact that for $s>1/2$ \n","start":73594,"end":74723}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":["xk-biased-high-probability"],"declared_external":false,"source_only":false,"title":"中心化加权排列和","source_units":[{"raw_tex":"where $\\zeta(\\cdot)$ is the Riemann-Zeta function. We now study the asymptotic behavior of the last summation term in \\eqref{xqk-high-proba} by introducing the process\n\t$S_{1,k} = \\sum_{j=0}^{k-1} Z_j$, where $Z_j := \\alpha_j (\\mu(\\s_j) - \\bar{\\mu})$ and $k\\geq 0$ with the convention that $S_{1,0} = 0$. Equipped with this definition, \\eqref{xqk-high-proba} becomes\n  \\beq \n  \t\t \\xonek - \\xs = b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{S_{1,k}}{k}.  \\label{xk-biased-high-probability}\n  \\eeq\t\t  \nThe random variables $Z_j$ are independent, centered and have an identical distribution up to the scaling factor $\\alpha_j$. Therefore,  $S_{1,k}$ is a sum of  centered random variables satisfying:\n","start":74861,"end":75570}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":["marting-diff-as-bound"],"declared_external":false,"source_only":false,"title":"声称中心化增量界","source_units":[{"raw_tex":"    \\beqa \\begin{scriptsize} \\| S_{1,k} - S_{1,k-1} \\| =\\big \\|\\alpha_{k-1}\\big(\\mu(\\sigma_{k-1})-\\bar{\\mu}\\big)\\big\\| \n    \\leq \\gamma_{k-1}:=\\alpha_{k-1}LmG_* \\label{marting-diff-as-bound}\n    \\end{scriptsize}\n    \\eeqa \nwhere we used \\eqref{def-mu-sk-upper-bound} in the last inequality (see also Lemma \\ref{lem-computing-mu-bar}). Then, by the Azuma-Hoeffding inequality, for every $t>0$, \n","start":75570,"end":75964}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":["average-sum-bound-whp"],"declared_external":false,"source_only":false,"title":"全平均误差的浓缩界","source_units":[{"raw_tex":"where we used \\eqref{def-mu-sk-upper-bound} in the last inequality (see also Lemma \\ref{lem-computing-mu-bar}). Then, by the Azuma-Hoeffding inequality, for every $t>0$, \n  \\beqas \\P \\bigg( \\big\\|\\frac{S_{1,k}}{k}\\big\\| > \\frac{t}{k}\\bigg) &\\leq& 2 \\exp \\bigg( - \\frac{t^2}{2\\sum_{j=0}^{k-1} \\gamma_j^2} \\bigg) =  2 \\exp \\big( - \\frac{t^2}{ \\beta }\\big) \n  \\eeqas\nwhere $\\beta = 2\\sum_{j=0}^{\\infty} \\gamma_j^2 < \\infty$ as $\\alpha_j$ is square-summable (see \\eqref{square-summable-step}). Note that $\\beta$ depends only on $\\Gs, L,m$ and the stepsize parameters $R$ and $s$. It is easy to see that selecting $t \\geq t_\\delta = \\sqrt{\\beta  \\log(2/\\delta)}$ makes the right-hand side $\\leq \\delta$.  Therefore for any $\\delta>0$, with probability at least $1-\\delta$, \n\t\t\\beq \\bigl \\| \\frac{S_{1,k}}{k} \\bigr \\| \\leq \\frac{\\sqrt{\\beta  \\log(2/\\delta)}}{k}\n\t\t\t\\label{average-sum-bound-whp}\n\t\t\\eeq\nwhich if inserted into the expression \\eqref{xk-biased-high-probability} completes the proof for the $q=1$ case. For  $0<q<1$ case, the same line of reasoning applies except that we replace $b_{1,k}$ with $b_{q,k}$ and we can improve the $\\bigO( \\log k /k)$ term in the expression \\eqref{xk-biased-high-probability} to $\\bigO(1/k)$, this is justified by \\eqref{xk-averaged-asymptotics-2}. Then, this leads to \n","start":75793,"end":77099}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["average-sum-bound-whp-3","xk-biased-high-probability-3"],"declared_external":false,"source_only":false,"title":"后缀随机和与高概率界","source_units":[{"raw_tex":"which if inserted into the expression \\eqref{xk-biased-high-probability} completes the proof for the $q=1$ case. For  $0<q<1$ case, the same line of reasoning applies except that we replace $b_{1,k}$ with $b_{q,k}$ and we can improve the $\\bigO( \\log k /k)$ term in the expression \\eqref{xk-biased-high-probability} to $\\bigO(1/k)$, this is justified by \\eqref{xk-averaged-asymptotics-2}. Then, this leads to \n\t\t \\beq \n  \t\t \\xqk - \\xs = b_{q,k}  +  \\bigO(\\frac{1}{k})  - H_*^{-1}   \\frac{S_{q,k}}{qk}  \\label{xk-biased-high-probability-3}\n  \\eeq\nwhere $S_{q,k} := \\sum_{j=(1-q)k}^{k-1} Z_j = S_{1,k} - S_{1,(1-q)k}$ is the $q$-suffix cumulative sum (cumulative sum of the last $qk$ terms) of the sequence $Z_{k}$. Then using \\eqref{average-sum-bound-whp}, with probability at least $1-\\delta$,\n\\beq \\bigl \\| \\frac{S_{q,k}}{k} \\bigr \\| \\leq \\| \\frac{S_{1,k}}{k} \\bigr \\|  + \\| \\frac{S_{1,(1-q)k}}{k} \\bigr \\|  \\leq \\frac{2 t_\\delta}{k}.\n\t\t\t\\label{average-sum-bound-whp-3}\n\t\t\\eeq\nPlugging this high probability bound into \\eqref{xk-biased-high-probability-3}, we conclude. \t\t\n","start":76689,"end":77763}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["grad-diff-quadratic"],"declared_external":false,"source_only":false,"title":"内层梯度接近最优点梯度","source_units":[{"raw_tex":"\t\\item [$(iv)$] By Lemma \\ref{lem-random-tail-bound}, we have $\\underset{1\\leq i < m}{\\max} \\|x_{i-1}^k - x^*\\| = \\bigO(\\alpha_k)$. Therefore,\n\t \\beqa \\scriptsize \\| \\nabla f_{\\ski} (x_{i-1}^k)  - \\nabla f_{\\ski} (x^*) \\|  = \\bigO(\\alpha_k) \\label{grad-diff-quadratic}\n\t \\eeqa\nfor any $i=1,2,\\dots,m$. As a consequence, \t\n","start":77763,"end":78085}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"替换梯度估计偏差","source_units":[{"raw_tex":"\t\\beqas \\hat{b}_{q,k} &=& -\\bar{\\alpha}_{q,k} H_*^{-1} \\sum_{i=1}^{m} P_{\\ski}  \\bigg(\\nabla f_{\\ski} (x^*) + \\bigO(\\alpha_k)\\bigg)  \\\\\n\t                         &=& - \\bar{\\alpha}_{q,k} H_*^{-1} \\sum_{j=1}^{m} P_{j} \\nabla f_j (x^*)   + \\bigO(\\alpha_k^2)   \n%\t                         &=& -\\bigg( H_*^{-1} + \\bigO(\\alpha_k) \\bigg) \\sum_{i=1}^{m} \\nabla^2 f_{i} (x_*) \\nabla f_i (x^*)  \\bar{\\alpha}_{q,k}+ \\bigO(\\alpha_k^2) +  o(\\alpha_k^2) \\\\\n\t                         = b_{q,k} + \\bigO(\\alpha_k^2)\t                         \t                        \t                         \n\t\\eeqas\t\t \nwhere in the second equality we use the fact that $\\bar{\\alpha}_{q,k}= \\bigO(1/k^s) = \\bigO(\\alpha_k)$ implied by part $(i)$.\n","start":78085,"end":78799}]},{"id":"S048","semantic_type":"lemma","kind":"lemma","labels":["lem-random-tail-bound"],"declared_external":false,"source_only":false,"title":"所有内层迭代距离界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem-random-tail-bound}Under the conditions of Theorem \\ref{thm-averaged-rate-quad} we have \n    $\\underset{0 \\leq i < m}{\\max} \\| x_i^k - x^*\\| =  \\bigO(\\frac{1}{k^s}).$\nwhere $\\bigO(\\cdot)$ hides a constant that depends only on $\\Gs, L, m,c$ and $R$. \n\\end{lemma}","start":115000,"end":115284}]},{"id":"S049","semantic_type":"proof","kind":"proof","labels":["upper-bound-cycle-begin"],"declared_external":false,"source_only":false,"title":"内层距离引理完整证明","source_units":[{"raw_tex":"\\begin{proof} By Corollary \\ref{coro-deter-decay-rate-dist}, \n   \\beq \\| x_0^k - x^* \\|  = \\bigO(\\frac{1}{k^s}).\n   \t\t\\label{upper-bound-cycle-begin}\n   \\eeq  \nwhere $\\bigO(\\cdot)$ hides a constant that depends only on $\\Gs, L, m,R$ and $c$.  We have also for any $0\\leq i< m$ and $k\\geq 0$,\n   \\beqas \\| x_i^k - x^*\\|  &\\leq& \\| x_0^k - x^*\\| + \\| x_i^k - x_0^k \\|  = \\| x_0^k - x^*\\| + i\\alpha_k \\max_{\\ell=1,\\dots,i} \\|\\nabla f_{\\sigma_k(\\ell)} (x_{\\ell-1}^k)\\| \\\\\n   &\\leq& \\| x_0^k - x^*\\| + (m-1)\\frac{R}{(k+1)^s} \\big(  G_* + \\max_{\\ell=1,\\dots,i} \\|\\nabla f_{\\sigma_k(\\ell)} (x_{\\ell-1}^k) - \\nabla f_{\\sigma_k(\\ell)} (x^*)\\| \\big) \\\\\n   &\\leq& \\| x_0^k - x^*\\| + (m-1)\\frac{R}{(k+1)^s} \\big(  G_* + L \\max_{\\ell=1,\\dots,i} \\|x_{\\ell-1}^k - x^*\\| \\big). \n   \\eeqas\nwhere we used the $L$-Lipschitzness of the gradient of $f$ where $L$ is given by \\ref{def-L}. Using \\eqref{upper-bound-cycle-begin} and applying this inequality inductively for $i=0,1,2,\\dots, m-1$ we conclude.\n\\end{proof}","start":115286,"end":116281}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":["upper-bound-cycle-begin"],"declared_external":false,"source_only":false,"title":"控制周期起点距离","source_units":[{"raw_tex":"\\begin{proof} By Corollary \\ref{coro-deter-decay-rate-dist}, \n   \\beq \\| x_0^k - x^* \\|  = \\bigO(\\frac{1}{k^s}).\n   \t\t\\label{upper-bound-cycle-begin}\n   \\eeq  \nwhere $\\bigO(\\cdot)$ hides a constant that depends only on $\\Gs, L, m,R$ and $c$.  We have also for any $0\\leq i< m$ and $k\\geq 0$,\n","start":115286,"end":115578}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"内层位移递推与归纳","source_units":[{"raw_tex":"   \\beqas \\| x_i^k - x^*\\|  &\\leq& \\| x_0^k - x^*\\| + \\| x_i^k - x_0^k \\|  = \\| x_0^k - x^*\\| + i\\alpha_k \\max_{\\ell=1,\\dots,i} \\|\\nabla f_{\\sigma_k(\\ell)} (x_{\\ell-1}^k)\\| \\\\\n   &\\leq& \\| x_0^k - x^*\\| + (m-1)\\frac{R}{(k+1)^s} \\big(  G_* + \\max_{\\ell=1,\\dots,i} \\|\\nabla f_{\\sigma_k(\\ell)} (x_{\\ell-1}^k) - \\nabla f_{\\sigma_k(\\ell)} (x^*)\\| \\big) \\\\\n   &\\leq& \\| x_0^k - x^*\\| + (m-1)\\frac{R}{(k+1)^s} \\big(  G_* + L \\max_{\\ell=1,\\dots,i} \\|x_{\\ell-1}^k - x^*\\| \\big). \n   \\eeqas\nwhere we used the $L$-Lipschitzness of the gradient of $f$ where $L$ is given by \\ref{def-L}. Using \\eqref{upper-bound-cycle-begin} and applying this inequality inductively for $i=0,1,2,\\dots, m-1$ we conclude.\n\\end{proof}","start":115578,"end":116281}]},{"id":"S052","semantic_type":"lemma","kind":"lemma","labels":["def-I-qk","lem-I-lk"],"declared_external":false,"source_only":false,"title":"平均归一化位移界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem-I-lk} Under the conditions of Theorem \\ref{thm-averaged-rate-quad}, consider the sequence\n\\beq I_{q,k} = \\frac{\\sum_{j=(1-q)k}^{k-1} {(x_0^j - x_{0}^{j+1})}{\\alpha_j^{-1}}}{qk}, \\quad  0 < q \\leq 1. \\label{def-I-qk}\n\\eeq\nThen, \n    $ I_{q, k} = \\begin{cases}  \\bigO \\big( \\frac{\\log k}{k}  \\big) \n     &  \\mbox{if} \\quad q=1, \\\\  \n    \t\t\t\\bigO \\big( \\frac{1}{k}\\big) \t & \\mbox{if} \\quad 0<q<1.  \n    \\end{cases}  \n%    &\\leq& \\bigO \\big( \\frac{\\log k}{k}\\big) \\\\\n%               I_{qk, k} &\\leq& \\bigO \\big( \\frac{1}{k}\\big) \\quad \\mbox{for all} \\quad 0<q<1.\n   $\nIn the former case, $\\bigO(\\cdot)$ hides a constant that depends only on $\\Gs, L,m,  c, R, s,q$ and $\\dist_0$. In the latter case, the same dependency on the constants occurs except that the dependency on $\\dist_0$ can be removed. \n\\end{lemma}","start":116644,"end":117475}]},{"id":"S053","semantic_type":"definition","kind":"definition","labels":["def-I-qk","lem-I-lk"],"declared_external":false,"source_only":false,"title":"归一化外层平均位移","source_units":[{"raw_tex":"\\beq I_{q,k} = \\frac{\\sum_{j=(1-q)k}^{k-1} {(x_0^j - x_{0}^{j+1})}{\\alpha_j^{-1}}}{qk}, \\quad  0 < q \\leq 1. \\label{def-I-qk}\n\\eeq\n","start":116758,"end":116889}]},{"id":"S054","semantic_type":"proof","kind":"proof","labels":["init-averaged-iter-bound","step-times-dist-bd","sum-of-aver-iters","tail-suffix-averaging","weighted-iter-diff-bound"],"declared_external":false,"source_only":false,"title":"平均位移引理完整证明","source_units":[{"raw_tex":"\\begin{proof}  It follows from integration by parts that for any $\\ell<k$, \n\\beq -\\sum_{j=\\ell}^{k-1}{(x_0^j - x_{0}^{j+1})}{\\alpha_j^{-1}}  =  \\alpha_k^{-1} (x_0^k - x^*) - \\alpha_\\ell^{-1} (x_0^\\ell - x^*) - \\sum_{j=\\ell}^{k-1} (x_0^{j+1} - x^*)(\\alpha_{j+1}^{-1}-\\alpha_j^{-1}). \\label{sum-of-aver-iters}\n\\eeq\nNext, we investigate the asymptotic behavior of the terms on the right-hand side. A consequence of Corollary \\ref{coro-deter-decay-rate-dist} and the inequality \\ref{def-M-gamma} is that  \n\t\\beq \\alpha_k^{-1} \\| x_0^k -x^*\\| = \\frac{(k+1)^s}{R} \\| x_0^k -x^*\\| \\leq \\frac{LmG_*}{c} + o(1) = \\bigO(1)\n\t\t\\label{step-times-dist-bd} \t\n\t\\eeq\nand therefore\n\\beqas \n\t| \\alpha_{k+1}^{-1} - \\alpha_{k}^{-1} |  \\| x_0^k -x^*\\| &=& \\frac{(k+2)^s - (k+1)^s} {(k+1)^s} \\alpha_k^{-1}\\| x_0^k -x^*\\|  \n\t = \\bigg( \\big(1 + \\frac{1}{k+1} \\big)^s -1 \\bigg) \\alpha_k^{-1}\\| x_0^k -x^*\\| \\\\\n\t  &\\leq &  \\frac{s}{k+1}\\alpha_k^{-1}\\| x_0^k -x^*\\|  \n\t  \\leq \\frac{sLmG_*}{c} \\frac{1}{k+1} + o(\\frac{1}{k+1}) = \\bigO(\\frac{1}{k+1})\n\\eeqas\nwhere $\\bigO(\\cdot)$ hides a constant that depends only on $L,\\Gs, c, m$ and $s$. Then, setting  $\\ell = (1-q)k$ in \\eqref{sum-of-aver-iters}, it follows that \n\\beqa  \\big\\| \\sum_{j=\\ell}^{k-1} {(x_0^j - x_0^{j+1})}{\\alpha_j^{-1}} \\big\\| &\\leq&  \\|\\alpha_k^{-1} (x_0^k - x^*)\\| + \\| \\alpha_{(1-q)k}^{-1} (x_0^{(1-q)k}- x^*)\\| \\\\\n  && + \\sum_{j=(1-q)k}^{k-1} \\| x_0^{j+1} - x^*\\| |\\alpha_{j+1}^{-1}-\\alpha_j^{-1}|.  \\nonumber\\\\\n       &=& \\bigO(1) + \\| \\alpha_{(1-q)k}^{-1} (x_0^{(1-q)k} - x^*)\\| + \\bigO \\bigg( \\sum_{j=(1-q)k}^{k-1} \\frac{1}{j+1}\\bigg).  \\label{weighted-iter-diff-bound} %\\\\\n       %&& + o \\bigg( \\sum_{j=\\ell}^{k-1} \\frac{1}{(j)^s} \\|(j+1)^s - j^s) \\|\\bigg). \\nonumber \\\\\n       %&=& \\bigO(1)  + \\| \\alpha_\\ell^{-1} (x_0^\\ell - x^*)\\| + \\bigO \\bigg( \\sum_{j=\\ell}^{k-1} \\big((1+\\frac{1}{j+1})^s - 1\\big) \\bigg) \\nonumber \\\\\n       %&& + o \\bigg( \\sum_{j=\\ell}^{k-1} \\big((1+\\frac{1}{j})^s - 1\\big) \\bigg). \\nonumber \\\\\n     % &=& \\bigO(1) + \\| \\alpha_\\ell^{-1} (x_0^\\ell - x^*)\\| + \\bigO \\bigg( \\sum_{j=\\ell}^{k-1} \\frac{s}{j+1} \\bigg) \n       %\\\\\n      %&\\leq& \\bigO(1) + \\bigO(\\frac{1}{k^{2s}} ) + \\bigO \\bigg( \\sum_{j=\\max(\\ell,1)}^{k-1} \\frac{s}{j} \\bigg) \\label{weighted-iter-diff-bound}\n\\eeqa\nWe also have\n  \\beq  \\| \\alpha_{(1-q)k}^{-1} (x_0^{(1-q)k} - x^*)\\|  = \\begin{cases} \\alpha_0^{-1} \\dist_0 & \\mbox{if} \\quad q = 1, \\\\ \\bigO(1)  & \\mbox{if} \\quad 0<q<1, \\end{cases} \\label{init-averaged-iter-bound}\n  \\eeq\nwhere the second part follows from \\eqref{step-times-dist-bd} with similar constants for the $\\bigO(\\cdot)$ term. As the sequence $\\frac{1}{j+1}$ is monotonically decreasing, for any $k>0$ we have the bounds \n\\beq  \\sum_{j=(1-q)k}^{k-1} \\frac{1}{j+1} \\leq \\frac{1}{(1-q)k + 1} + \\int_{(1-q)k}^{k-1} \\frac{1}{x+1} dx \\leq \\begin{cases} 1 + \\log k & \\mbox{if } \\quad  q=1, \\\\\n 1 + \\log(\\frac{1}{1-q})  & \\mbox{if } \\quad  0 <q <1.\n\\end{cases}  \\label{tail-suffix-averaging}\n\\eeq\nNote that when $q=1$ this bound grows with $k$ logarithmically whereas for $q<1$ it does not grow with $k$. \nThen, combining \\eqref{weighted-iter-diff-bound}, \\eqref{init-averaged-iter-bound} and \\eqref{tail-suffix-averaging}  we obtain\n  $$ \\| \\Iqk \\| \\leq \\frac{\\big\\| \\sum_{j=\\ell}^{k-1} {(x_0^j - x_0^{j+1})}{\\alpha_j^{-1}} \\big\\|}{qk} = \\begin{cases}  \\bigO \\big(  \\frac{\\log k}{k}\\big) \n     &  \\mbox{if} \\quad q=1 \\\\  \n    \t\t\t\\bigO \\big( \\frac{1}{k}\\big) \t & \\mbox{if} \\quad 0<q<1  \n    \\end{cases}  $$\nas desired which completes the proof.    \n\\end{proof}","start":117612,"end":121119}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["sum-of-aver-iters"],"declared_external":false,"source_only":false,"title":"离散分部求和","source_units":[{"raw_tex":"\\begin{proof}  It follows from integration by parts that for any $\\ell<k$, \n\\beq -\\sum_{j=\\ell}^{k-1}{(x_0^j - x_{0}^{j+1})}{\\alpha_j^{-1}}  =  \\alpha_k^{-1} (x_0^k - x^*) - \\alpha_\\ell^{-1} (x_0^\\ell - x^*) - \\sum_{j=\\ell}^{k-1} (x_0^{j+1} - x^*)(\\alpha_{j+1}^{-1}-\\alpha_j^{-1}). \\label{sum-of-aver-iters}\n\\eeq\n","start":117612,"end":117925}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":["step-times-dist-bd"],"declared_external":false,"source_only":false,"title":"步长归一化距离有界","source_units":[{"raw_tex":"Next, we investigate the asymptotic behavior of the terms on the right-hand side. A consequence of Corollary \\ref{coro-deter-decay-rate-dist} and the inequality \\ref{def-M-gamma} is that  \n\t\\beq \\alpha_k^{-1} \\| x_0^k -x^*\\| = \\frac{(k+1)^s}{R} \\| x_0^k -x^*\\| \\leq \\frac{LmG_*}{c} + o(1) = \\bigO(1)\n\t\t\\label{step-times-dist-bd} \t\n\t\\eeq\n","start":117925,"end":118262}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"逆步长增量的调和控制","source_units":[{"raw_tex":"and therefore\n\\beqas \n\t| \\alpha_{k+1}^{-1} - \\alpha_{k}^{-1} |  \\| x_0^k -x^*\\| &=& \\frac{(k+2)^s - (k+1)^s} {(k+1)^s} \\alpha_k^{-1}\\| x_0^k -x^*\\|  \n\t = \\bigg( \\big(1 + \\frac{1}{k+1} \\big)^s -1 \\bigg) \\alpha_k^{-1}\\| x_0^k -x^*\\| \\\\\n\t  &\\leq &  \\frac{s}{k+1}\\alpha_k^{-1}\\| x_0^k -x^*\\|  \n\t  \\leq \\frac{sLmG_*}{c} \\frac{1}{k+1} + o(\\frac{1}{k+1}) = \\bigO(\\frac{1}{k+1})\n\\eeqas\n","start":118262,"end":118640}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":["init-averaged-iter-bound","weighted-iter-diff-bound"],"declared_external":false,"source_only":false,"title":"后缀端点与位移和","source_units":[{"raw_tex":"where $\\bigO(\\cdot)$ hides a constant that depends only on $L,\\Gs, c, m$ and $s$. Then, setting  $\\ell = (1-q)k$ in \\eqref{sum-of-aver-iters}, it follows that \n\\beqa  \\big\\| \\sum_{j=\\ell}^{k-1} {(x_0^j - x_0^{j+1})}{\\alpha_j^{-1}} \\big\\| &\\leq&  \\|\\alpha_k^{-1} (x_0^k - x^*)\\| + \\| \\alpha_{(1-q)k}^{-1} (x_0^{(1-q)k}- x^*)\\| \\\\\n  && + \\sum_{j=(1-q)k}^{k-1} \\| x_0^{j+1} - x^*\\| |\\alpha_{j+1}^{-1}-\\alpha_j^{-1}|.  \\nonumber\\\\\n       &=& \\bigO(1) + \\| \\alpha_{(1-q)k}^{-1} (x_0^{(1-q)k} - x^*)\\| + \\bigO \\bigg( \\sum_{j=(1-q)k}^{k-1} \\frac{1}{j+1}\\bigg).  \\label{weighted-iter-diff-bound} %\\\\\n       %&& + o \\bigg( \\sum_{j=\\ell}^{k-1} \\frac{1}{(j)^s} \\|(j+1)^s - j^s) \\|\\bigg). \\nonumber \\\\\n       %&=& \\bigO(1)  + \\| \\alpha_\\ell^{-1} (x_0^\\ell - x^*)\\| + \\bigO \\bigg( \\sum_{j=\\ell}^{k-1} \\big((1+\\frac{1}{j+1})^s - 1\\big) \\bigg) \\nonumber \\\\\n       %&& + o \\bigg( \\sum_{j=\\ell}^{k-1} \\big((1+\\frac{1}{j})^s - 1\\big) \\bigg). \\nonumber \\\\\n     % &=& \\bigO(1) + \\| \\alpha_\\ell^{-1} (x_0^\\ell - x^*)\\| + \\bigO \\bigg( \\sum_{j=\\ell}^{k-1} \\frac{s}{j+1} \\bigg) \n       %\\\\\n      %&\\leq& \\bigO(1) + \\bigO(\\frac{1}{k^{2s}} ) + \\bigO \\bigg( \\sum_{j=\\max(\\ell,1)}^{k-1} \\frac{s}{j} \\bigg) \\label{weighted-iter-diff-bound}\n\\eeqa\nWe also have\n  \\beq  \\| \\alpha_{(1-q)k}^{-1} (x_0^{(1-q)k} - x^*)\\|  = \\begin{cases} \\alpha_0^{-1} \\dist_0 & \\mbox{if} \\quad q = 1, \\\\ \\bigO(1)  & \\mbox{if} \\quad 0<q<1, \\end{cases} \\label{init-averaged-iter-bound}\n  \\eeq\nwhere the second part follows from \\eqref{step-times-dist-bd} with similar constants for the $\\bigO(\\cdot)$ term. As the sequence $\\frac{1}{j+1}$ is monotonically decreasing, for any $k>0$ we have the bounds \n","start":118640,"end":120288}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":["tail-suffix-averaging"],"declared_external":false,"source_only":false,"title":"后缀调和和估计","source_units":[{"raw_tex":"\\beq  \\sum_{j=(1-q)k}^{k-1} \\frac{1}{j+1} \\leq \\frac{1}{(1-q)k + 1} + \\int_{(1-q)k}^{k-1} \\frac{1}{x+1} dx \\leq \\begin{cases} 1 + \\log k & \\mbox{if } \\quad  q=1, \\\\\n 1 + \\log(\\frac{1}{1-q})  & \\mbox{if } \\quad  0 <q <1.\n\\end{cases}  \\label{tail-suffix-averaging}\n\\eeq\nNote that when $q=1$ this bound grows with $k$ logarithmically whereas for $q<1$ it does not grow with $k$. \n","start":120288,"end":120665}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"除以样本数完成位移界","source_units":[{"raw_tex":"Then, combining \\eqref{weighted-iter-diff-bound}, \\eqref{init-averaged-iter-bound} and \\eqref{tail-suffix-averaging}  we obtain\n  $$ \\| \\Iqk \\| \\leq \\frac{\\big\\| \\sum_{j=\\ell}^{k-1} {(x_0^j - x_0^{j+1})}{\\alpha_j^{-1}} \\big\\|}{qk} = \\begin{cases}  \\bigO \\big(  \\frac{\\log k}{k}\\big) \n     &  \\mbox{if} \\quad q=1 \\\\  \n    \t\t\t\\bigO \\big( \\frac{1}{k}\\big) \t & \\mbox{if} \\quad 0<q<1  \n    \\end{cases}  $$\nas desired which completes the proof.    \n\\end{proof}","start":120665,"end":121119}]},{"id":"S061","semantic_type":"lemma","kind":"lemma","labels":["lem-computing-mu-bar"],"declared_external":false,"source_only":false,"title":"均匀排列偏差均值","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem-computing-mu-bar} Let $\\sigma$ be a random permutation of $\\{1,2,\\dots,m\\}$ sampled uniformly over the set of all permutations $\\Gamma$ defined by \\eqref{def-Gamma-perm-set} and $\\mu(\\sigma)$ be the vector defined by \\eqref{def-Msigma} that depends on $\\sigma$. Then, \\beq\n\t\\bar{\\mu} = \\E_{\\sigma}\\big(\\mu({\\sigma})\\big) = \\frac{1}{2} \\sum_{i=1}^m P_i \\nabla f_i(x^*)\n\\eeq\nwhere $\\E_{\\sigma}$ denotes the expectation with respect to the random permutation $\\sigma$ and $\\bar{mu}$ is defined by \\eqref{def-mu-bar}.\n\\end{lemma}","start":121121,"end":121670}]},{"id":"S062","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"排列均值引理完整证明","source_units":[{"raw_tex":"\\begin{proof} For any $i \\neq \\ell$, the joint distribution of $(\\sigma(i), \\sigma(\\ell))$ is uniform over the set of all (ordered) pairs from $\\{1,2,\\dots,m\\}$. Therefore, for any $i \\neq \\ell$, \n    \\beqas\n    \\E_\\sigma \\big[ P_{\\sigma(i)} \\nabla f_{\\sigma(\\ell) }(x^*) \\big] &=& \\sum_{i=1}^m \\sum_{i\\neq j, j=1}^m \\frac{P_i\\nabla f_j(x^*)}{m(m-1)} \\\\\n    &=& \\frac{\\sum_{i=1}^m P_i  \\sum_{j=1}^m \\nabla f_j(x^*) - \\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)} \n    =  - \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)}\n    \\eeqas \nwhere we used the fact that $\\nabla f(x^*) = \\sum_{j=1}^m \\nabla f_j(x^*)=0$ by the first order optimality condition. Then, by taking the expectation of \\eqref{def-v-sigma-k}, we obtain \n\\beqas \\E_\\sigma (\\mu(\\sigma)) &=& - \\sum_{i=1}^{m} \\sum_{\\ell=0}^{i-1} \\E \\big[ P_{\\sigma(i)} \\nabla f_{\\sigma(\\ell)}(x^*) \\big] \n        =  \\sum_{i=1}^{m} \\sum_{\\ell=0}^{i-1}  \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)} = \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{2}.\n\\eeqas\t \nwhich completes the proof.\n\\end{proof}","start":121671,"end":122706}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"不同排列位置的联合期望","source_units":[{"raw_tex":"\\begin{proof} For any $i \\neq \\ell$, the joint distribution of $(\\sigma(i), \\sigma(\\ell))$ is uniform over the set of all (ordered) pairs from $\\{1,2,\\dots,m\\}$. Therefore, for any $i \\neq \\ell$, \n    \\beqas\n    \\E_\\sigma \\big[ P_{\\sigma(i)} \\nabla f_{\\sigma(\\ell) }(x^*) \\big] &=& \\sum_{i=1}^m \\sum_{i\\neq j, j=1}^m \\frac{P_i\\nabla f_j(x^*)}{m(m-1)} \\\\\n    &=& \\frac{\\sum_{i=1}^m P_i  \\sum_{j=1}^m \\nabla f_j(x^*) - \\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)} \n    =  - \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)}\n    \\eeqas \n","start":121671,"end":122200}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"最优性消去与对数求和","source_units":[{"raw_tex":"where we used the fact that $\\nabla f(x^*) = \\sum_{j=1}^m \\nabla f_j(x^*)=0$ by the first order optimality condition. Then, by taking the expectation of \\eqref{def-v-sigma-k}, we obtain \n\\beqas \\E_\\sigma (\\mu(\\sigma)) &=& - \\sum_{i=1}^{m} \\sum_{\\ell=0}^{i-1} \\E \\big[ P_{\\sigma(i)} \\nabla f_{\\sigma(\\ell)}(x^*) \\big] \n        =  \\sum_{i=1}^{m} \\sum_{\\ell=0}^{i-1}  \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)} = \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{2}.\n\\eeqas\t \nwhich completes the proof.\n\\end{proof}","start":122200,"end":122706}]},{"id":"S065","semantic_type":"lemma","kind":"lemma","labels":["Ek-expression","def-mu-sk-upper-bound","def-v-sigma-k","lem-martingale-variance"],"declared_external":false,"source_only":false,"title":"误差解耦及加权平均极限","source_units":[{"raw_tex":"\\begin{lemma} \\label{lem-martingale-variance} Under the conditions of Theorem \\ref{thm-averaged-rate-quad},  the following statements are true: \n\\begin{itemize} \n   \\item [$(i)$] We have  \\beq E_k = \\alpha_k \\mu(\\sigma_k)  + \\bigO(\\alpha_k^2), \\quad k\\geq 0,  \\label{Ek-expression} \\eeq \nwhere $E_k$ is the gradient error defined by \\eqref{def-Ek}, $\\bigO(\\cdot)$ hides a constant that depends only on $\\Gs, L,m, R$ and $c$ and\n   \\beq  \\mu(\\sk) = - \\sum_{i=1}^{m} P_{\\ski} \\sum_{\\ell=1}^{i-1} \\nabla f_{\\skl}(x^*). \n\t\t\\label{def-v-sigma-k}   \n   \\eeq\nis a sequence of i.i.d. variables where the function $\\mu(\\cdot)$ is defined by \\eqref{def-Msigma}. \n%    \\item [$(ii)$]  It holds that   \n%    \\beq\n%           \\|\\mu(\\s_k)\\| \\leq LmG_* \n%           \t\t\\label{v-inf-norm}\n%     \\eeq\n% where $L=\\sum_i \\|P_i\\|$ and $\\Gs$ is defined by \\eqref{def-Gs}. \n% Furthermore,    \n%     \\beq      \n%        \\E v(\\sk) = \\bar{\\mu} =   \\frac{1}{2} \\sum_{i=1}^m P_i \\nabla f_i(x^*). % \\label{def-mu-star}\n%    \\eeq\n   %\\item [$(iii)$] $\\sigma_*^2 := var(v(u^k)) <\\infty$.\n   \\item [$(ii)$] For any $0 <q\\leq 1 $,  $\\lim_{k \\to \\infty} Y_{q,k}=  \\bar{\\mu}$ a.s. where \n    $ Y_{q,k} = \\frac{\\sum_{i=(1-q)k}^{k-1} E_j}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}.$\n   \\item [$(iii)$] It holds that \n       \\beq  \\| \\mu(\\sigma_k) \\| \\leq Lm G_*.\t\n       \t\t\\label{def-mu-sk-upper-bound}\n       \\eeq\t\n\\end{itemize}  \n   \n\n\\end{lemma}","start":122707,"end":124108}]},{"id":"S066","semantic_type":"definition","kind":"definition","labels":["def-v-sigma-k","lem-martingale-variance"],"declared_external":false,"source_only":false,"title":"排列误差的嵌套和表示","source_units":[{"raw_tex":"   \\beq  \\mu(\\sk) = - \\sum_{i=1}^{m} P_{\\ski} \\sum_{\\ell=1}^{i-1} \\nabla f_{\\skl}(x^*). \n\t\t\\label{def-v-sigma-k}   \n   \\eeq\nis a sequence of i.i.d. variables where the function $\\mu(\\cdot)$ is defined by \\eqref{def-Msigma}. \n","start":123135,"end":123360}]},{"id":"S067","semantic_type":"lemma","kind":"lemma","labels":["Ek-expression","lem-martingale-variance"],"declared_external":false,"source_only":false,"title":"周期误差二阶解耦","source_units":[{"raw_tex":"\\begin{lemma} \\label{lem-martingale-variance} Under the conditions of Theorem \\ref{thm-averaged-rate-quad},  the following statements are true: \n\\begin{itemize} \n   \\item [$(i)$] We have  \\beq E_k = \\alpha_k \\mu(\\sigma_k)  + \\bigO(\\alpha_k^2), \\quad k\\geq 0,  \\label{Ek-expression} \\eeq \nwhere $E_k$ is the gradient error defined by \\eqref{def-Ek}, $\\bigO(\\cdot)$ hides a constant that depends only on $\\Gs, L,m, R$ and $c$ and\n","start":122707,"end":123135}]},{"id":"S068","semantic_type":"lemma","kind":"lemma","labels":["lem-martingale-variance"],"declared_external":false,"source_only":false,"title":"归一化后缀误差极限","source_units":[{"raw_tex":"   \\item [$(ii)$] For any $0 <q\\leq 1 $,  $\\lim_{k \\to \\infty} Y_{q,k}=  \\bar{\\mu}$ a.s. where \n    $ Y_{q,k} = \\frac{\\sum_{i=(1-q)k}^{k-1} E_j}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}.$\n","start":123764,"end":123943}]},{"id":"S069","semantic_type":"lemma","kind":"lemma","labels":["def-mu-sk-upper-bound","lem-martingale-variance"],"declared_external":false,"source_only":false,"title":"独立排列误差有界","source_units":[{"raw_tex":"   \\item [$(iii)$] It holds that \n       \\beq  \\| \\mu(\\sigma_k) \\| \\leq Lm G_*.\t\n       \t\t\\label{def-mu-sk-upper-bound}\n       \\eeq\t\n\\end{itemize}  \n   \n\n\\end{lemma}","start":123943,"end":124108}]},{"id":"S070","semantic_type":"proof","kind":"proof","labels":["lim-aver-vuj","lim-average-mu-star","quadratic-only-one"],"declared_external":false,"source_only":false,"title":"误差解耦引理完整证明","source_units":[{"raw_tex":"\\begin{proof} \n\n\t\\begin{itemize} \n\t\t\t\\item [$(i)$] As component functions are quadratics, \\eqref{def-Ek} becomes\n\\beqas E_k &=& \\sum_{i=1}^{m} P_{\\ski}(x_{i-1}^k - x_0^k) = - \\sum_{i=1}^{m}  P_{\\ski}  \\alpha_k \\sum_{\\ell=1}^{i-1} \\nabla f_{\\skl } (x_{\\ell-1}^k). \n \\eeqas\nwhere we can substitute \n \\beq  \\nabla f_{ \\skl } (x_{\\ell-1}^k) =  \\nabla f_{ \\skl } (x^*)  + P_{\\skl} (x_{\\ell-1}^k - x^*).\n \t\\label{quadratic-only-one}\n \\eeq\nThen an application of Lemma \\ref{lem-random-tail-bound}  proves directly the desired result.\n%  \t\t\t\\item [$(ii)$] For the first part, the inequality \\eqref{v-inf-norm} is just a consequence of the triangle inequality applied to the definition \\eqref{def-v-sigma-k} with $L_i = \\| P_i \\|$ and $L=\\sum_{i=1}^m L_i$. For the second part, note that for any $i \\neq \\ell$, the joint distribution of $(\\ski, \\skl)$ is uniform over the set of all (ordered) pairs from $\\{1,2,\\dots,m\\}$. Therefore, for any $i \\neq \\ell$, \n%     \\beqas\n%     \\E \\big[ P_{\\ski} \\nabla f_{\\skl }(x^*) \\big] &=& \\sum_{i=1}^m \\sum_{i\\neq j, j=1}^m \\frac{P_i\\nabla f_j(x^*)}{m(m-1)} \\\\\n%     &=& \\frac{\\sum_{i=1}^m P_i  \\sum_{j=1}^m \\nabla f_j(x^*) - \\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)} \\\\\n%     &=&  - \\frac{\\sum_{j=1}^m P_j \\nabla f_i(x^*)}{m(m-1)}\n%     \\eeqas \n% where we used the fact that $\\nabla f(x^*) = \\sum_{j=1}^m \\nabla f_j(x^*)=0$ by the first order optimality condition. Then, by taking the expectation of \\eqref{def-v-sigma-k}, we obtain \n% \\beqas \\E v(k) &=& - \\sum_{i=1}^{m} \\sum_{\\ell=0}^{i-1} \\E \\big[ P_{\\ski} \\nabla f_{\\skl}(x^*) \\big] \\\\\n%         &=& - \\sum_{i=1}^{m} \\sum_{\\ell=0}^{i-1} - \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)} \\\\\n%         &=&  \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{2}.\n% \\eeqas\t \n% which completes the proof.\n%           \\item [$(iii)$] All the gradients and Hessians of the component functions at the global optimum $x^*$ is finite and there are finitely many component functions, therefore all the moments of $v(u^j)$ exists and is finite including the second moment. Therefore, the variance $\\sigma_*^2$ is bounded.\\footnote{Note that $\\sigma_*^2$ could be computed explicitly by a similar calculation to the one in the proof of part $(ii)$.} This completes the proof.\n   \t\t\t\\item [$(ii)$] We introduce the normalized gradient error sequence $Y_j = E_j / \\alpha_j$. By part $(i)$, $Y_j = \\mu(\\sigma_j) + \\bigO(\\alpha_j)$ where $\\mu(\\sigma_j)$ is a sequence of i.i.d. variables. By the strong law of large numbers, we have\n   \t\t\t     \\beq \\lim_{k \\to \\infty} \\frac{\\sum_{j=0}^{k-1} \\mu(\\sigma_j)}{k} = \\E \\mu(\\sigma_j) = \\bar{\\mu} \\quad \\mbox{a.s.}\\label{lim-aver-vuj}  \\eeq\nwhere the last equality is by the definition of $\\bar{\\mu}$. Therefore,   \t\t\t\n    \t\t\t \\beqas \\lim_{k \\to \\infty} \\frac{\\sum_{j=0}^{k-1} Y_j}{k} = \\lim_{k \\to \\infty} \\bigg(\\frac{\\sum_{j=0}^{k-1} \\mu(\\sigma_j)}{k} + \\frac{\\sum_{j=0}^{k-1} \\bigO(\\alpha_j)}{k}\\bigg) = \\bar{\\mu} \\quad \\mbox{a.s.} \\eeqas\nwhere we used the fact that the second term is negligible as $\\sum_{j=0}^{k-1} \\alpha_j / k = \\bigO(k^{-s}) \\to 0$. \nAs the average of the sequence $Y_j$ converges almost surely, one can show that this implies almost sure convergence of a weighted average of the sequence $Y_j$ as well as long as weights satisfy certain conditions as $k \\to \\infty$. In particular, as the sequence $\\{\\alpha_j\\}$ is monotonically decreasing and is non-summable, by \\cite[Theorem 1]{Etemadi:2006ko}, \n       \\beq \\lim_{k \\to \\infty} Y_{1,k} = \\lim_{k \\to \\infty}\\frac{\\sum_{j=0}^{k-1} \\alpha_j Y_j}{\\sum_{j=0}^{k-1} \\alpha_j} =  \\lim_{k \\to \\infty} \\frac{\\sum_{j=0}^{k-1} E_j}{\\sum_{j=0}^{k-1} \\alpha_j} = \\bar{\\mu} \\quad \\mbox{a.s.}\\label{lim-average-mu-star}\n       \\eeq \nThis completes the proof for $q=1$. For $0<q<1$, by the definition of $Y_{q,k}$, we can write $Y_{1,k} = (1-w_k) Y_{q,k} + w_k Y_{1, (1-q)k}$ where the non-negative weights $w_k$ satisfy    \n$$w_k = \\frac{\\sum_{j=0}^{(1-q)k - 1 } \\alpha_j}{\\sum_{j=0}^{k - 1 } \\alpha_j } \\to_{k \\to \\infty} (1-q)^{1-s} < 1.$$     \n    As both $Y_{1,k}$ and $Y_{1, (1-q)k}$ go to $\\bar{\\mu}$ a.s. by \\eqref{lim-average-mu-star}, it follows that \n$$\\lim_{k\\to \\infty} Y_{q,k} = \\lim_{k\\to \\infty}  \\frac{Y_{1,k} - w_k Y_{1,(1-q)k}}{1-w_k} = \\bar{\\mu}\\quad \\mbox{a.s}$$ as well for any $0<q<1$. This completes the proof.\n\t\\item [$(iii)$] This is a direct consequence of the triangle inequality applied to the definition \\eqref{def-v-sigma-k} with $L_i = \\|P_i\\|$ and $L=\\sum_{i=1}^m L_i$. \n\\end{itemize}\n\\end{proof}","start":124109,"end":128603}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":["quadratic-only-one"],"declared_external":false,"source_only":false,"title":"展开周期误差并替换内层点","source_units":[{"raw_tex":"\\begin{proof} \n\n\t\\begin{itemize} \n\t\t\t\\item [$(i)$] As component functions are quadratics, \\eqref{def-Ek} becomes\n\\beqas E_k &=& \\sum_{i=1}^{m} P_{\\ski}(x_{i-1}^k - x_0^k) = - \\sum_{i=1}^{m}  P_{\\ski}  \\alpha_k \\sum_{\\ell=1}^{i-1} \\nabla f_{\\skl } (x_{\\ell-1}^k). \n \\eeqas\nwhere we can substitute \n \\beq  \\nabla f_{ \\skl } (x_{\\ell-1}^k) =  \\nabla f_{ \\skl } (x^*)  + P_{\\skl} (x_{\\ell-1}^k - x^*).\n \t\\label{quadratic-only-one}\n \\eeq\nThen an application of Lemma \\ref{lem-random-tail-bound}  proves directly the desired result.\n","start":124109,"end":124636}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":["lim-aver-vuj"],"declared_external":false,"source_only":false,"title":"普通平均的强大数极限","source_units":[{"raw_tex":"   \t\t\t\\item [$(ii)$] We introduce the normalized gradient error sequence $Y_j = E_j / \\alpha_j$. By part $(i)$, $Y_j = \\mu(\\sigma_j) + \\bigO(\\alpha_j)$ where $\\mu(\\sigma_j)$ is a sequence of i.i.d. variables. By the strong law of large numbers, we have\n   \t\t\t     \\beq \\lim_{k \\to \\infty} \\frac{\\sum_{j=0}^{k-1} \\mu(\\sigma_j)}{k} = \\E \\mu(\\sigma_j) = \\bar{\\mu} \\quad \\mbox{a.s.}\\label{lim-aver-vuj}  \\eeq\nwhere the last equality is by the definition of $\\bar{\\mu}$. Therefore,   \t\t\t\n    \t\t\t \\beqas \\lim_{k \\to \\infty} \\frac{\\sum_{j=0}^{k-1} Y_j}{k} = \\lim_{k \\to \\infty} \\bigg(\\frac{\\sum_{j=0}^{k-1} \\mu(\\sigma_j)}{k} + \\frac{\\sum_{j=0}^{k-1} \\bigO(\\alpha_j)}{k}\\bigg) = \\bar{\\mu} \\quad \\mbox{a.s.} \\eeqas\nwhere we used the fact that the second term is negligible as $\\sum_{j=0}^{k-1} \\alpha_j / k = \\bigO(k^{-s}) \\to 0$. \n","start":126345,"end":127168}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":["lim-average-mu-star"],"declared_external":false,"source_only":false,"title":"应用加权平均收敛定理","source_units":[{"raw_tex":"As the average of the sequence $Y_j$ converges almost surely, one can show that this implies almost sure convergence of a weighted average of the sequence $Y_j$ as well as long as weights satisfy certain conditions as $k \\to \\infty$. In particular, as the sequence $\\{\\alpha_j\\}$ is monotonically decreasing and is non-summable, by \\cite[Theorem 1]{Etemadi:2006ko}, \n       \\beq \\lim_{k \\to \\infty} Y_{1,k} = \\lim_{k \\to \\infty}\\frac{\\sum_{j=0}^{k-1} \\alpha_j Y_j}{\\sum_{j=0}^{k-1} \\alpha_j} =  \\lim_{k \\to \\infty} \\frac{\\sum_{j=0}^{k-1} E_j}{\\sum_{j=0}^{k-1} \\alpha_j} = \\bar{\\mu} \\quad \\mbox{a.s.}\\label{lim-average-mu-star}\n       \\eeq \n","start":127168,"end":127808}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"由前缀极限获得后缀极限","source_units":[{"raw_tex":"This completes the proof for $q=1$. For $0<q<1$, by the definition of $Y_{q,k}$, we can write $Y_{1,k} = (1-w_k) Y_{q,k} + w_k Y_{1, (1-q)k}$ where the non-negative weights $w_k$ satisfy    \n$$w_k = \\frac{\\sum_{j=0}^{(1-q)k - 1 } \\alpha_j}{\\sum_{j=0}^{k - 1 } \\alpha_j } \\to_{k \\to \\infty} (1-q)^{1-s} < 1.$$     \n    As both $Y_{1,k}$ and $Y_{1, (1-q)k}$ go to $\\bar{\\mu}$ a.s. by \\eqref{lim-average-mu-star}, it follows that \n$$\\lim_{k\\to \\infty} Y_{q,k} = \\lim_{k\\to \\infty}  \\frac{Y_{1,k} - w_k Y_{1,(1-q)k}}{1-w_k} = \\bar{\\mu}\\quad \\mbox{a.s}$$ as well for any $0<q<1$. This completes the proof.\n","start":127808,"end":128409}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"三角不等式控制排列误差","source_units":[{"raw_tex":"\t\\item [$(iii)$] This is a direct consequence of the triangle inequality applied to the definition \\eqref{def-v-sigma-k} with $L_i = \\|P_i\\|$ and $L=\\sum_{i=1}^m L_i$. \n\\end{itemize}\n\\end{proof}","start":128409,"end":128603}]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始外部前提清单：标准一阶最优性事实；作者在 L1100 明确使用，在 L617 隐含使用。"],"source_only":false,"title":"无约束一阶最优性","source_units":[]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始外部前提清单：标准强凸性及谱范数事实。"],"source_only":false,"title":"强凸与谱范数事实","source_units":[]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始外部前提清单：summation by parts and integral bounds for series；幂函数增量界为标注者显式列出的隐含标准事实。"],"source_only":false,"title":"分部求和与级数积分界","source_units":[]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始外部前提清单：Strong law of large numbers for bounded i.i.d. random vectors；作者在 L1159–1160 调用。"],"source_only":false,"title":"有界独立向量强大数律","source_units":[]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始外部前提清单：Etemadi:2006ko, Theorem 1；活动引用位于 L1164。"],"source_only":false,"title":"Etemadi 加权平均结果","source_units":[]},{"id":"S081","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始外部前提清单：Azuma–Hoeffding concentration, with an applicable vector-norm formulation required for lines 683–689。"],"source_only":false,"title":"所需向量 Azuma–Hoeffding 界","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S028","edges":[{"source_id":"S001","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S014","target_id":"S007","kind":"supports"},{"source_id":"S077","target_id":"S007","kind":"supports"},{"source_id":"S002","target_id":"S009","kind":"supports"},{"source_id":"S003","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S005","target_id":"S015","kind":"supports"},{"source_id":"S006","target_id":"S016","kind":"supports"},{"source_id":"S014","target_id":"S016","kind":"supports"},{"source_id":"S077","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S015","target_id":"S020","kind":"supports"},{"source_id":"S017","target_id":"S021","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S017","target_id":"S031","kind":"supports"},{"source_id":"S002","target_id":"S031","kind":"supports"},{"source_id":"S019","target_id":"S031","kind":"supports"},{"source_id":"S002","target_id":"S022","kind":"supports"},{"source_id":"S018","target_id":"S022","kind":"supports"},{"source_id":"S061","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S019","target_id":"S024","kind":"supports"},{"source_id":"S015","target_id":"S024","kind":"supports"},{"source_id":"S003","target_id":"S028","kind":"supports"},{"source_id":"S006","target_id":"S028","kind":"supports"},{"source_id":"S012","target_id":"S028","kind":"supports"},{"source_id":"S013","target_id":"S028","kind":"supports"},{"source_id":"S014","target_id":"S028","kind":"supports"},{"source_id":"S016","target_id":"S028","kind":"supports"},{"source_id":"S023","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S013","target_id":"S029","kind":"supports"},{"source_id":"S007","target_id":"S029","kind":"supports"},{"source_id":"S023","target_id":"S029","kind":"supports"},{"source_id":"S003","target_id":"S030","kind":"supports"},{"source_id":"S013","target_id":"S030","kind":"supports"},{"source_id":"S014","target_id":"S030","kind":"supports"},{"source_id":"S016","target_id":"S030","kind":"supports"},{"source_id":"S013","target_id":"S033","kind":"supports"},{"source_id":"S027","target_id":"S033","kind":"supports"},{"source_id":"S078","target_id":"S033","kind":"supports"},{"source_id":"S009","target_id":"S034","kind":"supports"},{"source_id":"S012","target_id":"S034","kind":"supports"},{"source_id":"S053","target_id":"S034","kind":"supports"},{"source_id":"S014","target_id":"S035","kind":"supports"},{"source_id":"S016","target_id":"S035","kind":"supports"},{"source_id":"S007","target_id":"S035","kind":"supports"},{"source_id":"S076","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S012","target_id":"S036","kind":"supports"},{"source_id":"S013","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S007","target_id":"S037","kind":"supports"},{"source_id":"S052","target_id":"S037","kind":"supports"},{"source_id":"S077","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S033","target_id":"S038","kind":"supports"},{"source_id":"S068","target_id":"S038","kind":"supports"},{"source_id":"S027","target_id":"S038","kind":"supports"},{"source_id":"S067","target_id":"S039","kind":"supports"},{"source_id":"S069","target_id":"S039","kind":"supports"},{"source_id":"S036","target_id":"S039","kind":"supports"},{"source_id":"S027","target_id":"S040","kind":"supports"},{"source_id":"S078","target_id":"S040","kind":"supports"},{"source_id":"S037","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S022","target_id":"S041","kind":"supports"},{"source_id":"S029","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S002","target_id":"S042","kind":"supports"},{"source_id":"S018","target_id":"S042","kind":"supports"},{"source_id":"S022","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S069","target_id":"S043","kind":"supports"},{"source_id":"S061","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S040","target_id":"S044","kind":"supports"},{"source_id":"S081","target_id":"S044","kind":"supports"},{"source_id":"S037","target_id":"S045","kind":"supports"},{"source_id":"S039","target_id":"S045","kind":"supports"},{"source_id":"S040","target_id":"S045","kind":"supports"},{"source_id":"S042","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S029","target_id":"S045","kind":"supports"},{"source_id":"S012","target_id":"S045","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S015","target_id":"S046","kind":"supports"},{"source_id":"S027","target_id":"S046","kind":"supports"},{"source_id":"S030","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S033","target_id":"S047","kind":"supports"},{"source_id":"S016","target_id":"S047","kind":"supports"},{"source_id":"S023","target_id":"S047","kind":"supports"},{"source_id":"S029","target_id":"S047","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S032","kind":"supports"},{"source_id":"S035","target_id":"S032","kind":"supports"},{"source_id":"S036","target_id":"S032","kind":"supports"},{"source_id":"S037","target_id":"S032","kind":"supports"},{"source_id":"S038","target_id":"S032","kind":"supports"},{"source_id":"S039","target_id":"S032","kind":"supports"},{"source_id":"S040","target_id":"S032","kind":"supports"},{"source_id":"S041","target_id":"S032","kind":"supports"},{"source_id":"S042","target_id":"S032","kind":"supports"},{"source_id":"S043","target_id":"S032","kind":"supports"},{"source_id":"S044","target_id":"S032","kind":"supports"},{"source_id":"S045","target_id":"S032","kind":"supports"},{"source_id":"S046","target_id":"S032","kind":"supports"},{"source_id":"S047","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S028","kind":"supports"},{"source_id":"S003","target_id":"S048","kind":"supports"},{"source_id":"S006","target_id":"S048","kind":"supports"},{"source_id":"S014","target_id":"S048","kind":"supports"},{"source_id":"S027","target_id":"S048","kind":"supports"},{"source_id":"S004","target_id":"S048","kind":"supports"},{"source_id":"S031","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S003","target_id":"S051","kind":"supports"},{"source_id":"S019","target_id":"S051","kind":"supports"},{"source_id":"S015","target_id":"S051","kind":"supports"},{"source_id":"S027","target_id":"S051","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S051","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S048","kind":"supports"},{"source_id":"S003","target_id":"S052","kind":"supports"},{"source_id":"S006","target_id":"S052","kind":"supports"},{"source_id":"S014","target_id":"S052","kind":"supports"},{"source_id":"S027","target_id":"S052","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S004","target_id":"S052","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S078","target_id":"S055","kind":"supports"},{"source_id":"S031","target_id":"S056","kind":"supports"},{"source_id":"S019","target_id":"S056","kind":"supports"},{"source_id":"S020","target_id":"S056","kind":"supports"},{"source_id":"S027","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S027","target_id":"S057","kind":"supports"},{"source_id":"S078","target_id":"S057","kind":"supports"},{"source_id":"S055","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S012","target_id":"S058","kind":"supports"},{"source_id":"S010","target_id":"S058","kind":"supports"},{"source_id":"S027","target_id":"S059","kind":"supports"},{"source_id":"S078","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S060","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S053","target_id":"S060","kind":"supports"},{"source_id":"S055","target_id":"S054","kind":"supports"},{"source_id":"S056","target_id":"S054","kind":"supports"},{"source_id":"S057","target_id":"S054","kind":"supports"},{"source_id":"S058","target_id":"S054","kind":"supports"},{"source_id":"S059","target_id":"S054","kind":"supports"},{"source_id":"S060","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S002","target_id":"S061","kind":"supports"},{"source_id":"S018","target_id":"S061","kind":"supports"},{"source_id":"S022","target_id":"S061","kind":"supports"},{"source_id":"S002","target_id":"S063","kind":"supports"},{"source_id":"S014","target_id":"S063","kind":"supports"},{"source_id":"S007","target_id":"S063","kind":"supports"},{"source_id":"S076","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S076","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S064","kind":"supports"},{"source_id":"S022","target_id":"S064","kind":"supports"},{"source_id":"S063","target_id":"S062","kind":"supports"},{"source_id":"S064","target_id":"S062","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S018","target_id":"S066","kind":"supports"},{"source_id":"S002","target_id":"S066","kind":"supports"},{"source_id":"S003","target_id":"S067","kind":"supports"},{"source_id":"S006","target_id":"S067","kind":"supports"},{"source_id":"S014","target_id":"S067","kind":"supports"},{"source_id":"S027","target_id":"S067","kind":"supports"},{"source_id":"S008","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S003","target_id":"S071","kind":"supports"},{"source_id":"S008","target_id":"S071","kind":"supports"},{"source_id":"S014","target_id":"S071","kind":"supports"},{"source_id":"S048","target_id":"S071","kind":"supports"},{"source_id":"S027","target_id":"S071","kind":"supports"},{"source_id":"S066","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S075","kind":"supports"},{"source_id":"S015","target_id":"S075","kind":"supports"},{"source_id":"S019","target_id":"S075","kind":"supports"},{"source_id":"S075","target_id":"S069","kind":"supports"},{"source_id":"S067","target_id":"S072","kind":"supports"},{"source_id":"S066","target_id":"S072","kind":"supports"},{"source_id":"S069","target_id":"S072","kind":"supports"},{"source_id":"S022","target_id":"S072","kind":"supports"},{"source_id":"S027","target_id":"S072","kind":"supports"},{"source_id":"S079","target_id":"S072","kind":"supports"},{"source_id":"S078","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S027","target_id":"S073","kind":"supports"},{"source_id":"S080","target_id":"S073","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S036","target_id":"S074","kind":"supports"},{"source_id":"S027","target_id":"S074","kind":"supports"},{"source_id":"S078","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S068","kind":"supports"},{"source_id":"S074","target_id":"S068","kind":"supports"},{"source_id":"S071","target_id":"S070","kind":"supports"},{"source_id":"S072","target_id":"S070","kind":"supports"},{"source_id":"S073","target_id":"S070","kind":"supports"},{"source_id":"S074","target_id":"S070","kind":"supports"},{"source_id":"S075","target_id":"S070","kind":"supports"},{"source_id":"S070","target_id":"S065","kind":"supports"},{"source_id":"S067","target_id":"S065","kind":"supports"},{"source_id":"S068","target_id":"S065","kind":"supports"},{"source_id":"S069","target_id":"S065","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S028","edges":[{"source_id":"S001","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S014","target_id":"S007","kind":"supports"},{"source_id":"S077","target_id":"S007","kind":"supports"},{"source_id":"S002","target_id":"S009","kind":"supports"},{"source_id":"S003","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S005","target_id":"S015","kind":"supports"},{"source_id":"S006","target_id":"S016","kind":"supports"},{"source_id":"S014","target_id":"S016","kind":"supports"},{"source_id":"S077","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S015","target_id":"S020","kind":"supports"},{"source_id":"S017","target_id":"S031","kind":"supports"},{"source_id":"S002","target_id":"S031","kind":"supports"},{"source_id":"S019","target_id":"S031","kind":"supports"},{"source_id":"S002","target_id":"S022","kind":"supports"},{"source_id":"S018","target_id":"S022","kind":"supports"},{"source_id":"S061","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S019","target_id":"S024","kind":"supports"},{"source_id":"S015","target_id":"S024","kind":"supports"},{"source_id":"S003","target_id":"S028","kind":"supports"},{"source_id":"S006","target_id":"S028","kind":"supports"},{"source_id":"S012","target_id":"S028","kind":"supports"},{"source_id":"S013","target_id":"S028","kind":"supports"},{"source_id":"S014","target_id":"S028","kind":"supports"},{"source_id":"S016","target_id":"S028","kind":"supports"},{"source_id":"S023","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S013","target_id":"S029","kind":"supports"},{"source_id":"S007","target_id":"S029","kind":"supports"},{"source_id":"S023","target_id":"S029","kind":"supports"},{"source_id":"S003","target_id":"S030","kind":"supports"},{"source_id":"S013","target_id":"S030","kind":"supports"},{"source_id":"S014","target_id":"S030","kind":"supports"},{"source_id":"S016","target_id":"S030","kind":"supports"},{"source_id":"S013","target_id":"S033","kind":"supports"},{"source_id":"S027","target_id":"S033","kind":"supports"},{"source_id":"S078","target_id":"S033","kind":"supports"},{"source_id":"S009","target_id":"S034","kind":"supports"},{"source_id":"S012","target_id":"S034","kind":"supports"},{"source_id":"S053","target_id":"S034","kind":"supports"},{"source_id":"S014","target_id":"S035","kind":"supports"},{"source_id":"S016","target_id":"S035","kind":"supports"},{"source_id":"S007","target_id":"S035","kind":"supports"},{"source_id":"S076","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S012","target_id":"S036","kind":"supports"},{"source_id":"S013","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S007","target_id":"S037","kind":"supports"},{"source_id":"S052","target_id":"S037","kind":"supports"},{"source_id":"S077","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S033","target_id":"S038","kind":"supports"},{"source_id":"S068","target_id":"S038","kind":"supports"},{"source_id":"S027","target_id":"S038","kind":"supports"},{"source_id":"S067","target_id":"S039","kind":"supports"},{"source_id":"S069","target_id":"S039","kind":"supports"},{"source_id":"S036","target_id":"S039","kind":"supports"},{"source_id":"S027","target_id":"S040","kind":"supports"},{"source_id":"S078","target_id":"S040","kind":"supports"},{"source_id":"S037","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S022","target_id":"S041","kind":"supports"},{"source_id":"S029","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S002","target_id":"S042","kind":"supports"},{"source_id":"S018","target_id":"S042","kind":"supports"},{"source_id":"S022","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S069","target_id":"S043","kind":"supports"},{"source_id":"S061","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S040","target_id":"S044","kind":"supports"},{"source_id":"S081","target_id":"S044","kind":"supports"},{"source_id":"S037","target_id":"S045","kind":"supports"},{"source_id":"S039","target_id":"S045","kind":"supports"},{"source_id":"S040","target_id":"S045","kind":"supports"},{"source_id":"S042","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S029","target_id":"S045","kind":"supports"},{"source_id":"S012","target_id":"S045","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S015","target_id":"S046","kind":"supports"},{"source_id":"S027","target_id":"S046","kind":"supports"},{"source_id":"S030","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S033","target_id":"S047","kind":"supports"},{"source_id":"S016","target_id":"S047","kind":"supports"},{"source_id":"S023","target_id":"S047","kind":"supports"},{"source_id":"S029","target_id":"S047","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S032","kind":"supports"},{"source_id":"S035","target_id":"S032","kind":"supports"},{"source_id":"S036","target_id":"S032","kind":"supports"},{"source_id":"S037","target_id":"S032","kind":"supports"},{"source_id":"S038","target_id":"S032","kind":"supports"},{"source_id":"S039","target_id":"S032","kind":"supports"},{"source_id":"S040","target_id":"S032","kind":"supports"},{"source_id":"S041","target_id":"S032","kind":"supports"},{"source_id":"S042","target_id":"S032","kind":"supports"},{"source_id":"S043","target_id":"S032","kind":"supports"},{"source_id":"S044","target_id":"S032","kind":"supports"},{"source_id":"S045","target_id":"S032","kind":"supports"},{"source_id":"S046","target_id":"S032","kind":"supports"},{"source_id":"S047","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S028","kind":"supports"},{"source_id":"S003","target_id":"S048","kind":"supports"},{"source_id":"S006","target_id":"S048","kind":"supports"},{"source_id":"S014","target_id":"S048","kind":"supports"},{"source_id":"S027","target_id":"S048","kind":"supports"},{"source_id":"S004","target_id":"S048","kind":"supports"},{"source_id":"S031","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S003","target_id":"S051","kind":"supports"},{"source_id":"S019","target_id":"S051","kind":"supports"},{"source_id":"S015","target_id":"S051","kind":"supports"},{"source_id":"S027","target_id":"S051","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S051","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S048","kind":"supports"},{"source_id":"S003","target_id":"S052","kind":"supports"},{"source_id":"S006","target_id":"S052","kind":"supports"},{"source_id":"S014","target_id":"S052","kind":"supports"},{"source_id":"S027","target_id":"S052","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S004","target_id":"S052","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S078","target_id":"S055","kind":"supports"},{"source_id":"S031","target_id":"S056","kind":"supports"},{"source_id":"S019","target_id":"S056","kind":"supports"},{"source_id":"S020","target_id":"S056","kind":"supports"},{"source_id":"S027","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S027","target_id":"S057","kind":"supports"},{"source_id":"S078","target_id":"S057","kind":"supports"},{"source_id":"S055","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S012","target_id":"S058","kind":"supports"},{"source_id":"S010","target_id":"S058","kind":"supports"},{"source_id":"S027","target_id":"S059","kind":"supports"},{"source_id":"S078","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S060","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S053","target_id":"S060","kind":"supports"},{"source_id":"S055","target_id":"S054","kind":"supports"},{"source_id":"S056","target_id":"S054","kind":"supports"},{"source_id":"S057","target_id":"S054","kind":"supports"},{"source_id":"S058","target_id":"S054","kind":"supports"},{"source_id":"S059","target_id":"S054","kind":"supports"},{"source_id":"S060","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S002","target_id":"S061","kind":"supports"},{"source_id":"S018","target_id":"S061","kind":"supports"},{"source_id":"S022","target_id":"S061","kind":"supports"},{"source_id":"S002","target_id":"S063","kind":"supports"},{"source_id":"S014","target_id":"S063","kind":"supports"},{"source_id":"S007","target_id":"S063","kind":"supports"},{"source_id":"S076","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S076","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S064","kind":"supports"},{"source_id":"S022","target_id":"S064","kind":"supports"},{"source_id":"S063","target_id":"S062","kind":"supports"},{"source_id":"S064","target_id":"S062","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S018","target_id":"S066","kind":"supports"},{"source_id":"S002","target_id":"S066","kind":"supports"},{"source_id":"S003","target_id":"S067","kind":"supports"},{"source_id":"S006","target_id":"S067","kind":"supports"},{"source_id":"S014","target_id":"S067","kind":"supports"},{"source_id":"S027","target_id":"S067","kind":"supports"},{"source_id":"S008","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S003","target_id":"S071","kind":"supports"},{"source_id":"S008","target_id":"S071","kind":"supports"},{"source_id":"S014","target_id":"S071","kind":"supports"},{"source_id":"S048","target_id":"S071","kind":"supports"},{"source_id":"S027","target_id":"S071","kind":"supports"},{"source_id":"S066","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S075","kind":"supports"},{"source_id":"S015","target_id":"S075","kind":"supports"},{"source_id":"S019","target_id":"S075","kind":"supports"},{"source_id":"S075","target_id":"S069","kind":"supports"},{"source_id":"S067","target_id":"S072","kind":"supports"},{"source_id":"S066","target_id":"S072","kind":"supports"},{"source_id":"S069","target_id":"S072","kind":"supports"},{"source_id":"S022","target_id":"S072","kind":"supports"},{"source_id":"S027","target_id":"S072","kind":"supports"},{"source_id":"S079","target_id":"S072","kind":"supports"},{"source_id":"S078","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S027","target_id":"S073","kind":"supports"},{"source_id":"S080","target_id":"S073","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S036","target_id":"S074","kind":"supports"},{"source_id":"S027","target_id":"S074","kind":"supports"},{"source_id":"S078","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S068","kind":"supports"},{"source_id":"S074","target_id":"S068","kind":"supports"},{"source_id":"S071","target_id":"S070","kind":"supports"},{"source_id":"S072","target_id":"S070","kind":"supports"},{"source_id":"S073","target_id":"S070","kind":"supports"},{"source_id":"S074","target_id":"S070","kind":"supports"},{"source_id":"S075","target_id":"S070","kind":"supports"},{"source_id":"S070","target_id":"S065","kind":"supports"},{"source_id":"S067","target_id":"S065","kind":"supports"},{"source_id":"S068","target_id":"S065","kind":"supports"},{"source_id":"S069","target_id":"S065","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["pbm-multi-agent"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"有限和无约束优化","source_units":[{"raw_tex":"We consider the following unconstrained optimization problem where the objective function is the sum of a large number of component functions: \n\\beq\\label{pbm-multi-agent}\n\\min f(x) := \\sum_{i=1}^m f_i(x) \\quad \\mbox{s.t.} \\quad x \\in \\R^n\n\\eeq\nwith $f_i:\\R^n \\to \\R$. This problem arises in many contexts and applications including regression or more generally parameter estimation problems (where $f_i(x)$ is the loss function representing the error between the output and the prediction of a parametric model)~\\cite{Boyd2011AdmmBook,bertsekas2011incremental,bertsekas1997hybrid, Bertsekas1996incremental}, minimization of an expected value of a function (where the expectation is taken over a finite probability distribution or approximated by an $m$-sample average)~\\cite{BottouLecun2005, Leroux2012sgd}, machine learning~\\cite{Jordan13DistLearningApi, Dickenstein2014, Leroux2012sgd}, or distributed optimization over networks~\\cite{RamNedicVeer2007, NedicOzdaglar09,Nedic2007rate}.\n","start":9279,"end":10267}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":["def-Gamma-perm-set"],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"独立均匀随机排列","source_units":[{"raw_tex":"An alternative popular approach that works well in practice is following a mixed approach between SGD and IG, sampling the functions randomly but not allowing repetitions, that is sampling the component functions at each iteration \\textit{without-replacement}, or equivalently picking a random order at each cycle. Specifically, at each cycle $k$, we draw a permutation $\\sigma_k $ of $\\{1,2,\\dots,m\\}$ independently and uniformly at random over the set of all permutations \\beq \\Gamma = \\big\\{ \\sigma ~:~ \\sigma \\mbox{ is a permutation of } \\{1,2,\\dots,m\\} \\big\\} \\label{def-Gamma-perm-set}\\eeq \n","start":13220,"end":13817}]},{"id":"S003","semantic_type":"algorithm","kind":"algorithm","labels":["inner-update-wo"],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"随机重排迭代","source_units":[{"raw_tex":"and process the functions with this order: %$\\sigma_k = (\\sigma_k(1), \\sigma_k(2),\\dots, \\sigma_k(m)$:\n\\begin{equation}\\label{inner-update-wo} x_{i}^k :={x_{i-1}^k - \\alpha_k \\nabla f_{\\sigma_k(i)} (x_{i-1}^k)} , \\qquad i=1,2,\\dots,m,\n   \\end{equation} \n where $\\alpha_k>0$ is a stepsize. We set $x_0^{k+1} = x_{m}^k$ as before and refer to $\\{x_0^k\\}$ as the {\\it outer iterates}. This method is called the \\textit{Random Reshuffling} (RR) method~\\cite[Section 2.1]{Bertsekas15Book} and will be the focus of this paper.\n","start":13817,"end":14338}]},{"id":"S004","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"逐路径一致渐近记号","source_units":[{"raw_tex":"\\noindent \\textbf{Notation}: We study the point-wise dominance of stochastic sequences by deterministic sequences and use the following notation. Let $x_k = x_k(\\omega)$ be a stochastic real-valued sequence (where $\\omega$ can be thought as the source of randomness) and $y_k$ be a real-valued deterministic sequence. We write $ x_k = \\bigO(y_k)  \\iff \\exists h>0, \\exists k_0 \\quad \\mbox{such that} \\quad  |x_k| \\leq  h |y_k| \\quad \\forall k\\geq k_0, \\forall \\omega,$ where $h$ and $k_0$ are independent of $\\omega$ (Note that the requirement is that this inequality holds for all $\\omega$, not just for almost all $\\omega$). When $x_k$ is non-negative for every $\\omega$, given another deterministic positive sequence $z_k$, we also introduce the inequality version of this definition: \n\t$x_k \\leq y_k + o(z_k) \\iff \\forall \\varepsilon>0, \\exists k_0(\\varepsilon) \\quad \\mbox{such that}\\\t\\quad z_k^{-1} |x_k(\\omega) - y_k| \\leq \\varepsilon, \\quad \\forall k\\geq k_0(\\varepsilon), \\forall \\omega$\nwhere $k_0$ depends on $\\varepsilon$ but is independent of $\\omega$.\tWhen $x_k$ is deterministic, these definitions reduce to the standard definitions of $\\bigO(\\cdot)$ and $o(\\cdot)$ for deterministic sequences. For random $x_k$, the only difference is that we require the constants to be independent of the choice of $\\omega$. For example, if $x_k$ is uniformly distributed over $[0,10]$, we write $x_k = \\bigO(1)$. Throughout the paper, $\\| \\cdot \\|$ denotes the vector or matrix 2-norm (maximum singular value). %The notation $\\E_{\\sigma}X(\\sigma)$ denotes the expectation with respect to a random permutation $\\sigma$ of a random variable $X(\\sigma)$.\n","start":30505,"end":32159}]},{"id":"S005","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"欧氏范数与谱范数","source_units":[{"raw_tex":"where $k_0$ depends on $\\varepsilon$ but is independent of $\\omega$.\tWhen $x_k$ is deterministic, these definitions reduce to the standard definitions of $\\bigO(\\cdot)$ and $o(\\cdot)$ for deterministic sequences. For random $x_k$, the only difference is that we require the constants to be independent of the choice of $\\omega$. For example, if $x_k$ is uniformly distributed over $[0,10]$, we write $x_k = \\bigO(1)$. Throughout the paper, $\\| \\cdot \\|$ denotes the vector or matrix 2-norm (maximum singular value). %The notation $\\E_{\\sigma}X(\\sigma)$ denotes the expectation with respect to a random permutation $\\sigma$ of a random variable $X(\\sigma)$.\n","start":31502,"end":32159}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","labels":["assump-sum-is-str-cvx"],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"总目标强凸","source_units":[{"raw_tex":"We consider solving problem (\\ref{pbm-multi-agent}) with RR method with iterations given in (\\ref{inner-update-wo}). Throughout we assume the following:  \n\\begin{assumption}\\label{assump-sum-is-str-cvx} The sum function $f(x)=\\sum_{i=1}^m f_i(x)$ is strongly convex, i.e., there exists a constant $c>0$ such that the function $f(x) - \\frac{c}{2} \\| x\\| ^2$ is convex on $\\R^n$.\\footnote{Such functions arise naturally in support vector machines and other regularized learning algorithms or regression problems (see e.g. \\cite{Leroux2012sgd, Rakhlin:2011, Hogwild11})}.\n\\end{assumption}\n\n\n","start":32369,"end":32957}]},{"id":"S007","semantic_type":"derived_fact","kind":"derived_fact","labels":["def-Hs"],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"唯一最优解与可逆曲率","source_units":[{"raw_tex":"Note that this assumption is on the sum function $f$, it does not require the convexity of the individual component functions $f_i$. A consequence of this assumption is that there exists a unique optimal solution to \\eqref{pbm-multi-agent} which we denote by $x^*$. Another consequence is that the Hessian at the optimal solution is invertible since\n\t \\beq \\Hs := \\nabla^2 f(x^*) \\succeq c I_n \\succ 0\n\t\t\\label{def-Hs}\t \n\t \\eeq \nwhere $I_n$\tis the $n\\times n$ identity matrix. \n\n\n","start":32957,"end":33437}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":["def-Ek"],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"周期累积梯度误差","source_units":[{"raw_tex":"where \n   \\beq E_k := \\sum_{i=1}^{m} \\bigg( \\nabla f_{\\ski}(x_{i-1}^k) -  \\nabla f_{\\ski} (x_0^k) \\bigg) \\label{def-Ek}\n   \\eeq\nis the cumulative gradient errors associated with the cycle $k$. \n","start":33684,"end":33878}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","labels":["averaged-iter-randomized"],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"带误差梯度形式","source_units":[{"raw_tex":"To analyze RR, we view it as a gradient method with random gradient errors and rewrite \nthe outer iterations \\eqref{inner-update-wo} as \n   \\beq \\frac{x_0^k - x_{0}^{k+1}}{\\alpha_k} = \\nabla f(x_0^k) + E_k \\label{averaged-iter-randomized}\n   \\eeq\n","start":33437,"end":33684}]},{"id":"S010","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"外层距离记号","source_units":[{"raw_tex":"A key idea in our analysis is to use a recent upper bound for the convergence rate of cyclic incremental gradient method (see  \\cite{Gur2015IncGrad}), which can be generalized to hold for any fixed deterministic order. This bound implies an almost sure upper bound (in fact one that holds for all sample paths) on the distance of the outer iterates $x_0^k$ generated by RR from the optimal solution $x^*$ denoted by\n\t$\\dist_k  =\\| x_0^k - x^*\\|$\n(see Section \\ref{quad-conv}). Crucially, this result implies an upper bound in expected distance which is asymptotically $m$ times smaller than the almost sure guarantees on the distance of the iterates.\n\n","start":35073,"end":35725}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"外层简单平均","source_units":[{"raw_tex":"In analyzing RR, we will also consider the \\textit{average} of the outer iterate sequence given by\\footnote{It is well known that computing this (moving) average can be done efficiently in a dynamic manner by storing a vector of length $n$.} \n\t$ \\bar{x}_{k} := \\frac{\\sum_{j=0}^{k-1} x_0^j}{k}. \\label{simple-average}\n\t$\n","start":35725,"end":36046}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"迭代后缀平均","source_units":[{"raw_tex":" We also consider averaging only the most recent iterates, i.e. at iteration $k$, averaging the last $qk$ iterates\nfor some constant $q\\in (0,1]$:\n\t$$\\bar{x}_{q,k} := \\frac{\\sum_{j=(1-q)k }^{k-1} x_0^j}{qk}, \\quad 0 < q\\leq 1. $$ \nThe generated sequence is referred to as the \\textit{$q$-suffix average} of the sequence $x_0^k$. For SGD, it was shown that $q$-suffix averaging with $0<q<1$ leads to better performance then averaging (which corresponds to the $q=1$ case by definition), improving the convergence rate in the suboptimality of the function value from $\\log k/k$ to $1/k$~\\cite{Rakhlin:2011, Shamir:2012tj}. This is in line with our results in Section~\\ref{sec:quad-case} which show faster rate for the $0<q<1$ case. The parameter $q$ can be thought as a measure of how much memory one uses during the averaging process. We define the \\textit{$q$-suffix average} of the stepsize in a similar way:\n","start":36046,"end":36956}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["def-averaged-stepsize"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"步长后缀平均","source_units":[{"raw_tex":" \\beq \\bar{\\alpha}_{q,k} =\\frac{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}\n{qk}, \\quad 0<q\\leq 1.\\label{def-averaged-stepsize}\n\\eeq\n","start":36956,"end":37077}]},{"id":"S014","semantic_type":"problem_setting","kind":"problem_setting","labels":["def-fi"],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"对称二次分量模型","source_units":[{"raw_tex":"Let $f_i: \\R^n \\to \\R$ be a quadratic function of the form\n    \\beq f_i(x) = \\frac{1}{2} x_i^T P_i x - q_i^T x + r_i, \\quad i=1,2,\\dots, m,\n    \t\\label{def-fi}\n    \\eeq\nwhere $P_i $ is a symmetric $n\\times n$ matrix, $q_i \\in \\R^n$ is a column vector and $r_i$ is a scalar. Note that $f_i$ has Lipschitz gradients, i.e., \n","start":48674,"end":48996}]},{"id":"S015","semantic_type":"derived_fact","kind":"derived_fact","labels":["def-L"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"分量梯度与总光滑界","source_units":[{"raw_tex":"    $$ \\| \\nabla f_i(y) - \\nabla f_i (z) \\| \\leq L_i \\| y - z\\|, \\quad \\forall y, z \\in \\R^n, $$\nwhere $L_i = \\| P_i \\|$. It follows from the triangle inequality that $f$ has Lipschitz gradients with Lipschitz constant at most\n\t\\beq  L := \\sum_{i=1}^m L_i .\n\t\t\\label{def-L}\n\t\\eeq\n","start":48996,"end":49276}]},{"id":"S016","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"常 Hessian 的正定下界","source_units":[{"raw_tex":"Moreover, Assumption \\ref{assump-sum-is-str-cvx} implies that the Hessian matrix of the sum satisfies \n$\\nabla^2 f(x) = \\sum_{i=1}^m \\nabla^2 f_i(x) = \\sum_{i=1}^m P_i \\ge cI_n >0.$\n","start":49276,"end":49458}]},{"id":"S017","semantic_type":"theorem","kind":"theorem","labels":["def-Msigma","rate-prob-one-no-average","theo-rate-quadratics"],"member_ids":["S017"],"declared_external":true,"declaration_text":["原始前提清单允许 Gur2015IncGrad 的确定性 IG 收敛定理，文中重述为 B0031；完整外部证明未提供。"],"source_only":false,"title":"固定次序 IG 收敛界","source_units":[{"raw_tex":"\\begin{theorem}\\cite{Gur2015IncGrad} \nLet Assumption \\ref{assump-sum-is-str-cvx} hold. Let $f_i(x)$ be a quadratic function of the form $f_i(x) = \\frac{1}{2} x_i^T P_i x - q_i^T x + r_i$\nwhere $P_i $ is a symmetric $n\\times n$ matrix, $q_i \\in \\R^n$ is a column vector and $r_i$ is a scalar for $i=1,2,\\dots,m$. Suppose Assumption \\ref{assump-sum-is-str-cvx}  holds. Consider the iterates $\\{x_0^k\\}$ generated by the iterations \\eqref{inner-update-wo} with a fixed order $\\sigma$ and stepsize $\\alpha_k = R/(k+1)^s$ where $R>0$ and  $s \\in (1/2,1)$. Then\\footnote{The original result in \\cite{Gur2015IncGrad} was stated for $\\sigma=\\{1,2,\\dots,m\\}$ but here we translate this result into an arbitrary permutation $\\sigma$ of $\\{1,2,\\dots,m\\}$ by noting that processing the set of functions $\\{f_1, f_2, \\dots, f_m\\}$ with order $\\sigma$ is equivalent to processing the permuted functions $\\{f_{\\s_1}, f_{\\s_2}, \\dots, f_{\\s_m}\\}$ with order $\\{1,2,\\dots,m\\}$.\n},\n         % \\beq \\dist_k \\leq \\frac{R \\| \\mu(\\sigma)\\| }{c}\\frac{1}{k^s} + o(\\frac{1}{k^s}) \\quad \\mbox{where} \\quad  \\mu(\\sigma) =  \\underset{1\\leq i < j\\leq m}{\\sum} P_{\\sigma(j)} \\nabla f_{\\sigma(i)}(x^*). %\\label{def-Msigma}% \\leq Lm \\Gs\n         % \\eeq\n    \t\\begin{align} \\dist_k &\\leq \\frac{R \\| \\mu(\\sigma)\\| }{c}\\frac{1}{k^s} + o(\\frac{1}{k^s})  &\\mbox{if}& \\quad 1/2<s<1, \\label{rate-prob-one-no-average}\\\\\n           \t\t\t  \\dist_k  &\\leq \\frac{R^2 \\|  \\mu({\\sigma}) \\|}{ Rc - 1} \\frac{1}{k} + o(\\frac{1}{k})  &\\mbox{if}& \\quad s=1 \\mbox{ and } Rc>1,\n\t  %\\label{def-Msigma}% \\leq Lm \\Gs\n    \t\\end{align}\nwhere $c$ is the strong convexity constant of the sum function $f(x)$ and\n\\beq\n\t\\mu(\\sigma) =  - \\underset{1\\leq i < j\\leq m}{\\sum} P_{\\sigma(j)} \\nabla f_{\\sigma(i)}(x^*). \\label{def-Msigma}\n\\eeq  \t\n\t\\label{theo-rate-quadratics} \n\\end{theorem}","start":49907,"end":51726}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":["def-Msigma","theo-rate-quadratics"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"排列误差向量","source_units":[{"raw_tex":"where $c$ is the strong convexity constant of the sum function $f(x)$ and\n\\beq\n\t\\mu(\\sigma) =  - \\underset{1\\leq i < j\\leq m}{\\sum} P_{\\sigma(j)} \\nabla f_{\\sigma(i)}(x^*). \\label{def-Msigma}\n\\eeq  \t\n","start":51482,"end":51682}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":["def-Gs","def-M-gamma"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"最优点梯度与最坏排列常数","source_units":[{"raw_tex":"We next provide some upper bounds on $\\mu(\\sigma)$. We define \n\t\\beqa \\Gs :&=& \\underset{1\\leq i \\leq m}{\\sup} \\| \\nabla f_i(x^*)\\|, \n\t\\label{def-Gs} \\\\\n%\\eeq\n%and \n%\\beq\n\t M_\\Gamma :&=&~ \\sup_{\\sigma \\in \\Gamma} \\|\\mu(\\sigma)\\|.\n\t \\label{def-M-gamma}\t\n\\eeqa\n","start":52449,"end":52708}]},{"id":"S020","semantic_type":"derived_fact","kind":"derived_fact","labels":["def-M-gamma-bound"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"排列误差的统一上界","source_units":[{"raw_tex":"Using  $L_i = \\| P_i \\|$ for each $i$, it follows from the triangle inequality that\n\t\\beq  \\| \\mu(\\sigma) \\| \\leq M_\\Gamma  \\leq \\sup_{\\sigma \\in \\Gamma}  \\underset{1\\leq i < j\\leq m}{\\sum} L_{\\sigma(j)} \\Gs \\leq \\sum_{j=1}^m  (j-1) L_{\\sigma(j)} \\Gs \\leq Lm\\Gs\n\t\t\\label{def-M-gamma-bound}\n\t\\eeq\n","start":52708,"end":53004}]},{"id":"S022","semantic_type":"definition","kind":"definition","labels":["def-mu-bar"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"平均排列误差","source_units":[{"raw_tex":"One way to get better constants is to consider convergence in expectation, a weaker notion of convergence compared to almost sure convergence. In the next theorem, we show that $M_\\Gamma$ can be improved to a typically much smaller constant $\\|\\bar{\\mu}\\|$ where \\beq \n\t\\bar{\\mu} := \\E \\big(\\mu(\\sigma_1)\\big) =\n\t\t\t\\frac{\\sum_{\\sigma \\in \\Gamma} \\mu(\\sigma)}{|\\Gamma|}. \\label{def-mu-bar}\n\\eeq can be thought as a measure of average performance over the choice of random permutations. \n","start":57279,"end":57765}]},{"id":"S023","semantic_type":"derived_fact","kind":"derived_fact","labels":["mu-bar-alternative"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"平均误差的分量公式","source_units":[{"raw_tex":"\\begin{remark} A consequence of Lemma \\ref{lem-computing-mu-bar} proved in the Appendix is that \\beq \n  \t\\bar{\\mu} =  \\frac{1}{2} \\sum_{i=1}^m P_i \\nabla f_i(x^*)\n\\label{mu-bar-alternative}.\n\\eeq\nwhere $\\bar{\\mu}$ is defined by \\eqref{def-mu-bar}. By the triangle inequality, \n","start":58784,"end":59061}]},{"id":"S024","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"平均偏差的量级界","source_units":[{"raw_tex":"\t$\\| \\bar{\\mu} \\| \\leq \\sum_{i=1}^m L_i \\Gs = L\\Gs $\n","start":59061,"end":59114}]},{"id":"S027","semantic_type":"assumption","kind":"assumption","labels":["thm-averaged-rate-quad"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"步长与后缀参数","source_units":[{"raw_tex":"where $P_i $ is a symmetric $n\\times n$ matrix, $q_i \\in \\R^n$ is a column vector and $r_i$ is a scalar for $i=1,2,\\dots,m$. Consider the $q$-suffix averages $\\xqk$ of the RR iterates generated by the iterations \\eqref{inner-update-wo} with stepsize $\\alpha_k = \\frac{R}{(k+1)^s}$ where $R>0$ and $s \\in (\\frac{1}{2},1)$. Suppose that Assumption \\ref{assump-sum-is-str-cvx} holds. Then the following statements are true:\n\\begin{enumerate}\n\\item [$(i)$] For any $0 < q \\leq 1$, the $q$-suffix averaged stepsize $\\aqk$ defined in \\eqref{def-averaged-stepsize} satisfies\n","start":65931,"end":66499}]},{"id":"S028","semantic_type":"theorem","kind":"theorem","labels":["def-aqk","def-bqk","def-hat-bqk","lim-rate-aver-quad","limit-main-result","thm-averaged-rate-quad"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"平均 RR 渐近率与去偏差","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm-averaged-rate-quad}  Let $f_i(x)$ be a quadratic function of the form\n\t$$f_i(x) = \\frac{1}{2} x_i^T P_i x - q_i^T x + r_i$$ \nwhere $P_i $ is a symmetric $n\\times n$ matrix, $q_i \\in \\R^n$ is a column vector and $r_i$ is a scalar for $i=1,2,\\dots,m$. Consider the $q$-suffix averages $\\xqk$ of the RR iterates generated by the iterations \\eqref{inner-update-wo} with stepsize $\\alpha_k = \\frac{R}{(k+1)^s}$ where $R>0$ and $s \\in (\\frac{1}{2},1)$. Suppose that Assumption \\ref{assump-sum-is-str-cvx} holds. Then the following statements are true:\n\\begin{enumerate}\n\\item [$(i)$] For any $0 < q \\leq 1$, the $q$-suffix averaged stepsize $\\aqk$ defined in \\eqref{def-averaged-stepsize} satisfies\n    \\beq\\aqk = \\frac{\\aqs}{k^s} + \\bigO(\\frac{1}{k}) \\quad \\mbox{where} \\quad \\aqs = \\frac{1-(1-q)^{1-s}}{q(1-s)} R.\\label{def-aqk}\n    \\eeq\n \\item [$(ii)$] For any $0 < q \\leq 1$, we have \n    \\beq \\lim_{k \\to \\infty} \\frac{\\bar{x}_{q,k} - x^*}{\\aqk} =  -H_*^{-1} \\bar{\\mu}\\quad a.s. \\label{limit-main-result}\n    \\eeq\nwhere $\\bar{\\mu}$ is given by \\eqref{mu-bar-alternative}, i.e., the normalized error $(\\bar{x}_{q,k} - x^*)/\\aqk$  converges to the constant vector $-H_*^{-1} \\bar{\\mu}$ almost surely where $\\Hs=\\sum_{i=1}^m P_i$ is the Hessian matrix at the optimal solution and $\\bar{\\mu}$ is given by \\eqref{mu-bar-alternative}.\n% \\beq      \n%     \\bar{\\mu} =  \\frac{1}{2} \\sum_{i=1}^m P_i \\nabla f_i(x^*). \n%    \\eeq\nThen, from part $(i)$,  \n \\beq \\lim_{k \\to \\infty} k^s (\\bar{x}_{q,k} - x^*) =  -\\aqs H_*^{-1} \\bar{\\mu} \\quad a.s.\\label{lim-rate-aver-quad}\n \\eeq\nHence, the $q$-suffix averaged iterates $\\xqk$ converge to the optimal solution $x^*$ with rate $1/k^s$ almost surely. \n%where $\\Hs=\\sum_{i=1}^m P_i$ is the Hessian matrix at the optimal solution, \n%\\beq      \n%    \\aqs =  \\frac{1-(1-q)^{1-s}}{q} R \\quad \\mbox{and} \\quad \\bar{\\mu} =  \\frac{1}{2} \\sum_{i=1}^m P_i \\nabla f_i(x^*). \n%   \\eeq\n   \n \\item [$(iii)$] With probability at least $1-\\delta$, we have\n\t $$ \\bar{x}_{q,k} -  x^*  =  b_{q,k} + \\bigO \\bigg(\\frac{\\sqrt{\\log(1/\\delta)}}{k} \\bigg) +  \\begin{cases} \\bigO \\big(\\frac{\\log k}{k} \\big)  & \\mbox{if} \\quad  q=1 \\\\ \\bigO(\\frac{1}{k}\\big) & \\mbox{if} \\quad  0<q<1, \\end{cases} $$ where\n\t \\beq b_{q,k} = -\\bar{\\alpha}_{q,k} H_*^{-1}  \\bar{\\mu} \n\t \t\t\\label{def-bqk}\n\t \\eeq \nis deterministic, $\\bar{\\mu}$ is given by $\\eqref{mu-bar-alternative}$ and $\\bar{\\alpha}_{q,k}$ is the averaged stepsize defined in \\eqref{def-averaged-stepsize}. The constants hidden by $\\bigO(\\cdot)$ depend only on $\\Gs, L, m, R, c, q$ and $s$.\n\t \\item [$(iv)$] Let \\beq  \\hat{b}_{q,k} = -\\bar{\\alpha}_{q,k}\\bigg[\\sum_{i=1}^{m} P_{\\ski}\\bigg]^{-1} \\sum_{i=1}^{m} P_{\\ski} \\nabla f_{\\ski} (x_{i-1}^k)/2. \\label{def-hat-bqk}\n\t \\eeq\nwhere $\\bar{\\alpha}_{q,k}$ is the averaged stepsize defined in \\eqref{def-averaged-stepsize}. Then, \t \n\t $\\hat{b}_{q,k}  = b_{q,k} + \\bigO(\\alpha_k^2). $\nIt follows from part $(ii)$ that with probability at least $1-\\delta$,\n\t  $$ (\\bar{x}_{q,k} - \\hat{b}_{q,k}) -  x^*  = \\bigO \\bigg(\\frac{\\sqrt{\\log(1/\\delta)}}{k} \\bigg) +  \\begin{cases} \\bigO \\big(\\frac{\\log k}{k} \\big)  & \\mbox{if} \\quad  q=1 \\\\ \\bigO(\\frac{1}{k}\\big) & \\mbox{if} \\quad  0<q<1. \\end{cases} $$ \n\\end{enumerate} \n \n \\end{theorem}","start":65780,"end":69036}]},{"id":"S029","semantic_type":"definition","kind":"definition","labels":["def-bqk","thm-averaged-rate-quad"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"确定性偏差项","source_units":[{"raw_tex":"\t \\beq b_{q,k} = -\\bar{\\alpha}_{q,k} H_*^{-1}  \\bar{\\mu} \n\t \t\t\\label{def-bqk}\n\t \\eeq \nis deterministic, $\\bar{\\mu}$ is given by $\\eqref{mu-bar-alternative}$ and $\\bar{\\alpha}_{q,k}$ is the averaged stepsize defined in \\eqref{def-averaged-stepsize}. The constants hidden by $\\bigO(\\cdot)$ depend only on $\\Gs, L, m, R, c, q$ and $s$.\n","start":68018,"end":68351}]},{"id":"S030","semantic_type":"definition","kind":"definition","labels":["def-hat-bqk","thm-averaged-rate-quad"],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"周期内偏差估计","source_units":[{"raw_tex":"\t \\item [$(iv)$] Let \\beq  \\hat{b}_{q,k} = -\\bar{\\alpha}_{q,k}\\bigg[\\sum_{i=1}^{m} P_{\\ski}\\bigg]^{-1} \\sum_{i=1}^{m} P_{\\ski} \\nabla f_{\\ski} (x_{i-1}^k)/2. \\label{def-hat-bqk}\n\t \\eeq\nwhere $\\bar{\\alpha}_{q,k}$ is the averaged stepsize defined in \\eqref{def-averaged-stepsize}. Then, \t \n","start":68351,"end":68639}]},{"id":"S031","semantic_type":"corollary","kind":"corollary","labels":["coro-deter-decay-rate-dist"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"RR 最坏路径距离界","source_units":[{"raw_tex":"\\begin{corollary}\\label{coro-deter-decay-rate-dist} Under the setting of Theorem \\ref{theo-rate-quadratics}, if $\\sigma$ is sampled uniformly at each cycle instead of being kept fixed, then\n\t\t% \\beqs \\  \\dist_k \\leq \\frac{R M_\\Gamma}{c}\\frac{1}{k^s} + o(\\frac{1}{k^s})%, \\quad M_\\Gamma = \\sup_{\\sigma \\in \\Gamma} \\mu(\\sigma)\n  %        \\eeqs \n         \\begin{align} \\dist_k &\\leq \\frac{R M_\\Gamma }{c}\\frac{1}{k^s} + o(\\frac{1}{k^s})  &\\mbox{if}& \\quad 1/2<s<1,\\\\\n           \t\t\t  \\dist_k  &\\leq \\frac{R^2 M_\\Gamma}{ Rc - 1} \\frac{1}{k} + o(\\frac{1}{k})  &\\mbox{if}& \\quad s=1 \\mbox{ and } Rc>1,\n\t  %\\label{def-Msigma}% \\leq Lm \\Gs\n    \t\\end{align}\nwith probability one where $M_\\Gamma$ is deterministic and is defined by \\eqref{def-M-gamma}.\n%the right-hand side is a deterministic sequence and         \n%  \t    $$M_\\Gamma \\leq Lm \\Gs, \\quad \\Gs = \\underset{1\\leq i \\leq m}{\\sup} \\| \\nabla f_i(x^*)\\|.$\n\\end{corollary}","start":55773,"end":56691}]},{"id":"S032","semantic_type":"proof","kind":"proof","labels":["Yqk-helper","average-sum-bound-whp","average-sum-bound-whp-3","averaged-iter-random-2","def-Yqk","grad-as-hessian-prod-quadratics","grad-diff-quadratic","iterate-eq-randomized","marting-diff-as-bound","square-summable-step","xk-averaged-asymptotics-2","xk-biased-high-probability","xk-biased-high-probability-3","xqk-high-proba"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"主定理完整作者证明","source_units":[{"raw_tex":"\\begin{proof} \n\t\\begin{itemize} \n\t\\item [$(i)$] As the stepsize sequence is monotonically decreasing, we have the bounds \n   \\beqas \\int_{(1-q)k}^k  \\frac{R}{(x+2)^s} dx \\leq  \\sum_{j=(1-q)k}^k \\alpha_j = \\sum_{j=(1-q)k}^k \\frac{R}{(k+1)^s} %&\\leq& R + \\sum_{j=(1-q)k + 1}^k \\frac{R}{(k+1)^s} \\\\\n   \t\t\t &\\leq& R + \\int_{(1-q)k}^{k-1}  \\frac{R}{(x+1)^s} dx. \n   %1 + \\int_{(1-q)k}^k  \\frac{R}{(x+1)^s} dx. \\\\\n   \\eeqas\nDividing each term by $qk$, after a straightforward integration we obtain\n  $$ \\aqk = \\frac{ k^{1-s} - \\big((1-q)k+1\\big)^{1-s} + \\bigO(1)}{(1-s)qk} R  = \\frac{a_q(s)}{k^s} + \\bigO(\\frac{1}{k}). \n   $$\n which completes the proof.  \n \t\t\t\\item [$(ii)$] Taking the $q$-suffix averages of both sides of \\eqref{averaged-iter-randomized}, we obtain \n\\beq \n\tI_{q,k} := \\frac{\\sum_{j=(1-q)k}^{k-1} {(x_0^j - x_{0}^{j+1})}{\\alpha_j^{-1}}}{qk} = \\frac{ \\sum_{j=(1-q)k}^{k-1}   \\nabla f(x_0^j)+ E_j}{qk}. \\label{averaged-iter-random-2}\n   \\eeq \nAs $f$ is a quadratic, the first order Taylor series for the gradient of $f$ is exact:\n  \\beq \\nabla f(x_0^j) = H_* (x_0^j-x^*).\n  \t\t\\label{grad-as-hessian-prod-quadratics}\n  \\eeq\n Therefore, \\eqref{averaged-iter-random-2} becomes  $\nI_{q,k} \n= \\frac{ \\sum_{j=(1-q)k}^{k-1}   H_* (x_0^j - x^*)   + E_j }{qk} $\n%\\label{averaged-iter-random-3} \nwhich is equivalent to\n\\beqa\n                    I_{q,k} &=& H_*(\\bar{x}_{q,k} - x^*) + \\frac{\\sum_{j=(1-q)k}^{k-1} E_j}{qk} = H_*(\\bar{x}_{q,k} - x^*) + \\bar{\\alpha}_{q,k}  Y_{q,k}\\label{iterate-eq-randomized}\n   \\eeqa \nwhere $Y_{q,k}$ is defined as\n    \\beq Y_{q,k} := \\frac{1}{\\bar{\\alpha}_{q,k}} \\frac{\\sum_{j=(1-q)k}^{k-1} E_j }{qk } =  \\frac{\\sum_{j=(1-q)k}^{k-1} E_j}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}\t\n\t\t\\label{def-Yqk}    \n     \\eeq\nand can be interpreted as the ($q$-suffix) averaged gradient error sequence $E_j$  normalized by the ($q$-suffix) averaged stepsize sequence $\\alpha_j$. Since $H_*$ is invertible by the strong convexity of $f$ (see \\eqref{def-Hs}), we can rewrite \\eqref{iterate-eq-randomized} as\n\\beqa  \\bar{x}_{q,k} - x^* &=& \n                     - H_*^{-1} \\aqk \\Yqk  + H_*^{-1} \\Iqk  \n            = - H_*^{-1} \\bar{\\alpha}_{q,k} Y_{q,k}  +  \\begin{cases} \\bigO(\\frac{1}{k})  & \\mbox{if } \\quad 0<q<1 \\\\\n\\bigO( \\frac{\\log k}{k})  & \\mbox{if } \\quad q = 1.\n\\end{cases} \\label{xk-averaged-asymptotics-2}\n              %           &=& \\bigO(\\frac{\\log k}{k}) + \\bigO(\\frac{1}{k}) - H_*^{-1} \\bigg(   \\frac{\\sum_{j=0}^{k-1} E_j}{\\sum_{j=0}^{k-1}\\alpha_j}\\bigg) \\frac{\\sum_{j=0}^{k-1}\\alpha_j}{k}   \\nonumber \n   \\eeqa              \nwhere we used the inequality $\\| H_*^{-1}\\| \\leq 1/c $ implied by \\eqref{def-Hs} and Lemma \\ref{lem-I-lk} from the appendix to provide an upper bound for the second term in the first equality. Note that, as a consequence of Lemma \\ref{lem-I-lk}, $\\bigO(\\cdot)$ notation above hides a constant that depends only on the parameters $\\Gs, L,c, m, R, s, q$ and also $\\dist_0$ when $q=1$. Then, dividing both sides of \\eqref{xk-averaged-asymptotics-2} by $\\aqk$, taking limits as $k$ goes to infinity,  using part $(i)$ on the asymptotic behavior of $\\aqk$ and the fact that $Y_{q,k} \\to \\bar{\\mu}$ a.s. from Lemma \\ref{lem-martingale-variance}, we obtain the claimed result.%$$ \\lim_{k \\to \\infty} k^s (\\bar{x}_{q,k} - x^*) = a_q(s) H_*^{-1} \\bar{\\mu}. $$\n%This completes the proof of part $(i)$.\n%Taking expectations, \n%\n%$$ \\E (\\bar{x}_k - x^*)  = \\bigO(\\frac{\\log k}{k}) -  H_*^{-1} \\bigg(   \\frac{\\sum_{j=0}^{k-1} \\alpha_j \\E v(u^j) }{k}\\bigg) + \\bigO(\\alpha_k^2).  $$\n%\n%\n%which implies\n%$$ \\E (\\bar{x}_k - x^*)  = \\bigO(\\frac{\\log k}{k}) -  b_* \\bigg(   \\frac{\\sum_{j=0}^{k-1} \\alpha_j  }{k}\\bigg) + \\bigO(\\alpha_k^2).  $$ \n%$ b_* = H_*^{-1}\\frac{\\sum_{r=1}^m \\nabla^2 f_r(x^*) \\nabla f_r(x^*)}{2}.$\n%Suppose we can compute an approximation $b_k$ to $b_*$ satisfying\n%   $b_k = b_* + \\bigO(\\alpha_k).$\n%Then,    \n%$$ \\E \\bigg(\\bar{x}_k + b_k \\bigg(   \\frac{\\sum_{j=0}^{k-1} \\alpha_j  }{k}\\bigg)  - x^*\\bigg)  = \\bigO(\\frac{\\log k}{k})  + \\bigO(\\alpha_k^2).  $$ \n%\n%NEED TO LOOK AT THE VARIANCE TERM AS WELL. USE OLD MARTINGALE APPROACH FOR BOUNDING THE VA\n\n \\item [$(iii)$] By parts $(i)$ and $(iii)$ of Lemma \\ref{lem-martingale-variance} from the appendix that relates the gradient error sequence $E_j$ to a sequence of i.i.d. variables $\\mu(\\sigma_j)$, for $0<q\\leq 1$,  \n\t\t\\beq Y_{q,k} = \\frac{\\sum_{j=(1-q)k}^{k-1} E_j}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j} =  \\frac{\\sum_{i=(1-q)k}^{k-1} \\alpha_j \\mu(\\sigma_j) + \\bigO(\\alpha_j^2)}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}. \t \n\t\t\t\\label{Yqk-helper}  \n     \\eeq \nWe first give a proof for $q=1$, the proof for the remaining $q \\in (0,1)$ case will be similar. Assume $q=1$. Plugging $q=1$ and \\eqref{Yqk-helper} into \\eqref{xk-averaged-asymptotics-2}, we obtain\n     \\beqa \\bar{x}_{1,k} - x^* \n     &=&\\bigO(\\frac{\\log k}{k})  - H_*^{-1} \\aonek \\Yonek \\nonumber \\\\\n     &=& \\bigO(\\frac{\\log k}{k}) - H_*^{-1} \\bigg(   \\frac{\\sum_{j=0}^{k-1} \\alpha_j \\big(\\mu(\\s_j) - \\bar{\\mu}\\big)}{k} +  \\frac{\\sum_{j=0}^{k-1}  \\alpha_j \\bar{\\mu} + \\bigO(\\alpha_j^2) }{k}  \\bigg) \\nonumber \\\\\n     \t\t\t\t\t\t &=& b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{\\sum_{j=0}^{k-1}  \\alpha_j (\\mu(\\s_j) - \\bar{\\mu})}{k} - H_*^{-1} \\sum_{j=0}^{k-1}  \\frac{\\bigO(\\alpha_j^2)}{k} \\nonumber \\\\     \n     \t\t\t\t\t\t    &=& b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{\\sum_{j=0}^{k-1}  \\alpha_j (\\mu(\\s_j) - \\bar{\\mu})}{k} \\label{xqk-high-proba} %\\\\     \t\t\t\t\t\t    \n     \t\t\t\t\t\t%    &=& b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{M_{1,k}}{k}  \\label{xk-biased-high-probability}\n     \\eeqa    \nwhere $\\bonek$ is defined by \\eqref{def-bqk} and we used in the last step the fact that for $s>1/2$ \n\t\\beq \\sum_{j=0}^\\infty \\alpha_j^2 = \\sum_{j=1}^\\infty \\frac{R^2}{j^{2s}} = R^2 \\zeta(2s) < \\infty\n\t\t\\label{square-summable-step}\t\n\t\\eeq \nwhere $\\zeta(\\cdot)$ is the Riemann-Zeta function. We now study the asymptotic behavior of the last summation term in \\eqref{xqk-high-proba} by introducing the process\n\t$S_{1,k} = \\sum_{j=0}^{k-1} Z_j$, where $Z_j := \\alpha_j (\\mu(\\s_j) - \\bar{\\mu})$ and $k\\geq 0$ with the convention that $S_{1,0} = 0$. Equipped with this definition, \\eqref{xqk-high-proba} becomes\n  \\beq \n  \t\t \\xonek - \\xs = b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{S_{1,k}}{k}.  \\label{xk-biased-high-probability}\n  \\eeq\t\t  \nThe random variables $Z_j$ are independent, centered and have an identical distribution up to the scaling factor $\\alpha_j$. Therefore,  $S_{1,k}$ is a sum of  centered random variables satisfying:\n    \\beqa \\begin{scriptsize} \\| S_{1,k} - S_{1,k-1} \\| =\\big \\|\\alpha_{k-1}\\big(\\mu(\\sigma_{k-1})-\\bar{\\mu}\\big)\\big\\| \n    \\leq \\gamma_{k-1}:=\\alpha_{k-1}LmG_* \\label{marting-diff-as-bound}\n    \\end{scriptsize}\n    \\eeqa \nwhere we used \\eqref{def-mu-sk-upper-bound} in the last inequality (see also Lemma \\ref{lem-computing-mu-bar}). Then, by the Azuma-Hoeffding inequality, for every $t>0$, \n  \\beqas \\P \\bigg( \\big\\|\\frac{S_{1,k}}{k}\\big\\| > \\frac{t}{k}\\bigg) &\\leq& 2 \\exp \\bigg( - \\frac{t^2}{2\\sum_{j=0}^{k-1} \\gamma_j^2} \\bigg) =  2 \\exp \\big( - \\frac{t^2}{ \\beta }\\big) \n  \\eeqas\nwhere $\\beta = 2\\sum_{j=0}^{\\infty} \\gamma_j^2 < \\infty$ as $\\alpha_j$ is square-summable (see \\eqref{square-summable-step}). Note that $\\beta$ depends only on $\\Gs, L,m$ and the stepsize parameters $R$ and $s$. It is easy to see that selecting $t \\geq t_\\delta = \\sqrt{\\beta  \\log(2/\\delta)}$ makes the right-hand side $\\leq \\delta$.  Therefore for any $\\delta>0$, with probability at least $1-\\delta$, \n\t\t\\beq \\bigl \\| \\frac{S_{1,k}}{k} \\bigr \\| \\leq \\frac{\\sqrt{\\beta  \\log(2/\\delta)}}{k}\n\t\t\t\\label{average-sum-bound-whp}\n\t\t\\eeq\nwhich if inserted into the expression \\eqref{xk-biased-high-probability} completes the proof for the $q=1$ case. For  $0<q<1$ case, the same line of reasoning applies except that we replace $b_{1,k}$ with $b_{q,k}$ and we can improve the $\\bigO( \\log k /k)$ term in the expression \\eqref{xk-biased-high-probability} to $\\bigO(1/k)$, this is justified by \\eqref{xk-averaged-asymptotics-2}. Then, this leads to \n\t\t \\beq \n  \t\t \\xqk - \\xs = b_{q,k}  +  \\bigO(\\frac{1}{k})  - H_*^{-1}   \\frac{S_{q,k}}{qk}  \\label{xk-biased-high-probability-3}\n  \\eeq\nwhere $S_{q,k} := \\sum_{j=(1-q)k}^{k-1} Z_j = S_{1,k} - S_{1,(1-q)k}$ is the $q$-suffix cumulative sum (cumulative sum of the last $qk$ terms) of the sequence $Z_{k}$. Then using \\eqref{average-sum-bound-whp}, with probability at least $1-\\delta$,\n\\beq \\bigl \\| \\frac{S_{q,k}}{k} \\bigr \\| \\leq \\| \\frac{S_{1,k}}{k} \\bigr \\|  + \\| \\frac{S_{1,(1-q)k}}{k} \\bigr \\|  \\leq \\frac{2 t_\\delta}{k}.\n\t\t\t\\label{average-sum-bound-whp-3}\n\t\t\\eeq\nPlugging this high probability bound into \\eqref{xk-biased-high-probability-3}, we conclude. \t\t\n\t\\item [$(iv)$] By Lemma \\ref{lem-random-tail-bound}, we have $\\underset{1\\leq i < m}{\\max} \\|x_{i-1}^k - x^*\\| = \\bigO(\\alpha_k)$. Therefore,\n\t \\beqa \\scriptsize \\| \\nabla f_{\\ski} (x_{i-1}^k)  - \\nabla f_{\\ski} (x^*) \\|  = \\bigO(\\alpha_k) \\label{grad-diff-quadratic}\n\t \\eeqa\nfor any $i=1,2,\\dots,m$. As a consequence, \t\n\t\\beqas \\hat{b}_{q,k} &=& -\\bar{\\alpha}_{q,k} H_*^{-1} \\sum_{i=1}^{m} P_{\\ski}  \\bigg(\\nabla f_{\\ski} (x^*) + \\bigO(\\alpha_k)\\bigg)  \\\\\n\t                         &=& - \\bar{\\alpha}_{q,k} H_*^{-1} \\sum_{j=1}^{m} P_{j} \\nabla f_j (x^*)   + \\bigO(\\alpha_k^2)   \n%\t                         &=& -\\bigg( H_*^{-1} + \\bigO(\\alpha_k) \\bigg) \\sum_{i=1}^{m} \\nabla^2 f_{i} (x_*) \\nabla f_i (x^*)  \\bar{\\alpha}_{q,k}+ \\bigO(\\alpha_k^2) +  o(\\alpha_k^2) \\\\\n\t                         = b_{q,k} + \\bigO(\\alpha_k^2)\t                         \t                        \t                         \n\t\\eeqas\t\t \nwhere in the second equality we use the fact that $\\bar{\\alpha}_{q,k}= \\bigO(1/k^s) = \\bigO(\\alpha_k)$ implied by part $(i)$.\n%\t\t\\item [$(ii)$]  By taking norm square of both sides and expectations in \\eqref{xk-averaged-asymptotics} and applying Cauchy-Schwartz gives\n%\t\t\t\\beqas \\E \\| \\bar{x}_k - x^*\\|^2 &\\leq& 4\\bigO(\\frac{\\log k}{k}) + 4\\bigO(\\frac{1}{k}) + 4 \\E \\bigg\\|  H_*^{-1} \\bigg(   \\frac{\\sum_{j=0}^{k-1} E_j}{k}\\bigg)\\bigg\\|^2 + 4\\bigO(\\alpha_k^2)   \\\\\n%\t\t\t                                              &\\leq& 4\\bigO(\\frac{\\log k}{k}) + 4\\bigO(\\frac{1}{k}) + \\frac{4}{c^2} \\E \\bigg\\|   \\frac{\\sum_{j=0}^{k-1} E_j}{k}\\bigg\\|^2 + 4\\bigO(\\alpha_k^2) \n%\t\t\t\\eeqas\n%We then decompose the term inside the expectation as\n%    \\beqas \\frac{\\sum_{j=0}^{k-1} E_j}{k} &=& \\frac{\\sum_{j=0}^{k-1} \\alpha_j v(u^j) - \\alpha_j \\bar{\\mu}}{k} + \\frac{\\sum_{j=0}^{k-1} \\alpha_j \\bar{\\mu} + \\bigO(\\alpha_j^2)}{k} \\\\\n%               &=& \\frac{\\sum_{j=0}^{k-1} \\alpha_j v(u^j) - \\alpha_j \\bar{\\mu}}{\\sigma_* (\\sum_{j=0}^{k-1}\\alpha_j^2)^{1/2}} \\frac{\\sigma_* (\\sum_{j=0}^{k-1}\\alpha_j^2)^{1/2}}{k} + \\frac{\\sum_{j=0}^{k-1} \\alpha_j \\bar{\\mu} + \\bigO(\\alpha_j^2)}{k}\n%    \\eeqas\n\\end{itemize}\n\\end{proof}","start":69040,"end":79865}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"积分比较平均步长","source_units":[{"raw_tex":"\\begin{proof} \n\t\\begin{itemize} \n\t\\item [$(i)$] As the stepsize sequence is monotonically decreasing, we have the bounds \n   \\beqas \\int_{(1-q)k}^k  \\frac{R}{(x+2)^s} dx \\leq  \\sum_{j=(1-q)k}^k \\alpha_j = \\sum_{j=(1-q)k}^k \\frac{R}{(k+1)^s} %&\\leq& R + \\sum_{j=(1-q)k + 1}^k \\frac{R}{(k+1)^s} \\\\\n   \t\t\t &\\leq& R + \\int_{(1-q)k}^{k-1}  \\frac{R}{(x+1)^s} dx. \n   %1 + \\int_{(1-q)k}^k  \\frac{R}{(x+1)^s} dx. \\\\\n   \\eeqas\nDividing each term by $qk$, after a straightforward integration we obtain\n  $$ \\aqk = \\frac{ k^{1-s} - \\big((1-q)k+1\\big)^{1-s} + \\bigO(1)}{(1-s)qk} R  = \\frac{a_q(s)}{k^s} + \\bigO(\\frac{1}{k}). \n   $$\n which completes the proof.  \n","start":69040,"end":69690}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":["averaged-iter-random-2"],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"平均周期梯度等式","source_units":[{"raw_tex":" \t\t\t\\item [$(ii)$] Taking the $q$-suffix averages of both sides of \\eqref{averaged-iter-randomized}, we obtain \n\\beq \n\tI_{q,k} := \\frac{\\sum_{j=(1-q)k}^{k-1} {(x_0^j - x_{0}^{j+1})}{\\alpha_j^{-1}}}{qk} = \\frac{ \\sum_{j=(1-q)k}^{k-1}   \\nabla f(x_0^j)+ E_j}{qk}. \\label{averaged-iter-random-2}\n   \\eeq \n","start":69690,"end":69992}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":["grad-as-hessian-prod-quadratics"],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"二次梯度精确线性化","source_units":[{"raw_tex":"As $f$ is a quadratic, the first order Taylor series for the gradient of $f$ is exact:\n  \\beq \\nabla f(x_0^j) = H_* (x_0^j-x^*).\n  \t\t\\label{grad-as-hessian-prod-quadratics}\n  \\eeq\n","start":69992,"end":70172}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":["def-Yqk","iterate-eq-randomized"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"平均误差的曲率分解","source_units":[{"raw_tex":" Therefore, \\eqref{averaged-iter-random-2} becomes  $\nI_{q,k} \n= \\frac{ \\sum_{j=(1-q)k}^{k-1}   H_* (x_0^j - x^*)   + E_j }{qk} $\n%\\label{averaged-iter-random-3} \nwhich is equivalent to\n\\beqa\n                    I_{q,k} &=& H_*(\\bar{x}_{q,k} - x^*) + \\frac{\\sum_{j=(1-q)k}^{k-1} E_j}{qk} = H_*(\\bar{x}_{q,k} - x^*) + \\bar{\\alpha}_{q,k}  Y_{q,k}\\label{iterate-eq-randomized}\n   \\eeqa \nwhere $Y_{q,k}$ is defined as\n    \\beq Y_{q,k} := \\frac{1}{\\bar{\\alpha}_{q,k}} \\frac{\\sum_{j=(1-q)k}^{k-1} E_j }{qk } =  \\frac{\\sum_{j=(1-q)k}^{k-1} E_j}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}\t\n\t\t\\label{def-Yqk}    \n     \\eeq\nand can be interpreted as the ($q$-suffix) averaged gradient error sequence $E_j$  normalized by the ($q$-suffix) averaged stepsize sequence $\\alpha_j$. Since $H_*$ is invertible by the strong convexity of $f$ (see \\eqref{def-Hs}), we can rewrite \\eqref{iterate-eq-randomized} as\n","start":70172,"end":71055}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":["xk-averaged-asymptotics-2"],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"控制平均位移余项","source_units":[{"raw_tex":"\\beqa  \\bar{x}_{q,k} - x^* &=& \n                     - H_*^{-1} \\aqk \\Yqk  + H_*^{-1} \\Iqk  \n            = - H_*^{-1} \\bar{\\alpha}_{q,k} Y_{q,k}  +  \\begin{cases} \\bigO(\\frac{1}{k})  & \\mbox{if } \\quad 0<q<1 \\\\\n\\bigO( \\frac{\\log k}{k})  & \\mbox{if } \\quad q = 1.\n\\end{cases} \\label{xk-averaged-asymptotics-2}\n              %           &=& \\bigO(\\frac{\\log k}{k}) + \\bigO(\\frac{1}{k}) - H_*^{-1} \\bigg(   \\frac{\\sum_{j=0}^{k-1} E_j}{\\sum_{j=0}^{k-1}\\alpha_j}\\bigg) \\frac{\\sum_{j=0}^{k-1}\\alpha_j}{k}   \\nonumber \n   \\eeqa              \nwhere we used the inequality $\\| H_*^{-1}\\| \\leq 1/c $ implied by \\eqref{def-Hs} and Lemma \\ref{lem-I-lk} from the appendix to provide an upper bound for the second term in the first equality. Note that, as a consequence of Lemma \\ref{lem-I-lk}, $\\bigO(\\cdot)$ notation above hides a constant that depends only on the parameters $\\Gs, L,c, m, R, s, q$ and also $\\dist_0$ when $q=1$. Then, dividing both sides of \\eqref{xk-averaged-asymptotics-2} by $\\aqk$, taking limits as $k$ goes to infinity,  using part $(i)$ on the asymptotic behavior of $\\aqk$ and the fact that $Y_{q,k} \\to \\bar{\\mu}$ a.s. from Lemma \\ref{lem-martingale-variance}, we obtain the claimed result.%$$ \\lim_{k \\to \\infty} k^s (\\bar{x}_{q,k} - x^*) = a_q(s) H_*^{-1} \\bar{\\mu}. $$\n","start":71055,"end":72341}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"取得几乎必然极限","source_units":[{"raw_tex":"where we used the inequality $\\| H_*^{-1}\\| \\leq 1/c $ implied by \\eqref{def-Hs} and Lemma \\ref{lem-I-lk} from the appendix to provide an upper bound for the second term in the first equality. Note that, as a consequence of Lemma \\ref{lem-I-lk}, $\\bigO(\\cdot)$ notation above hides a constant that depends only on the parameters $\\Gs, L,c, m, R, s, q$ and also $\\dist_0$ when $q=1$. Then, dividing both sides of \\eqref{xk-averaged-asymptotics-2} by $\\aqk$, taking limits as $k$ goes to infinity,  using part $(i)$ on the asymptotic behavior of $\\aqk$ and the fact that $Y_{q,k} \\to \\bar{\\mu}$ a.s. from Lemma \\ref{lem-martingale-variance}, we obtain the claimed result.%$$ \\lim_{k \\to \\infty} k^s (\\bar{x}_{q,k} - x^*) = a_q(s) H_*^{-1} \\bar{\\mu}. $$\n","start":71590,"end":72341}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":["Yqk-helper"],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"代入独立误差主项","source_units":[{"raw_tex":" \\item [$(iii)$] By parts $(i)$ and $(iii)$ of Lemma \\ref{lem-martingale-variance} from the appendix that relates the gradient error sequence $E_j$ to a sequence of i.i.d. variables $\\mu(\\sigma_j)$, for $0<q\\leq 1$,  \n\t\t\\beq Y_{q,k} = \\frac{\\sum_{j=(1-q)k}^{k-1} E_j}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j} =  \\frac{\\sum_{i=(1-q)k}^{k-1} \\alpha_j \\mu(\\sigma_j) + \\bigO(\\alpha_j^2)}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}. \t \n\t\t\t\\label{Yqk-helper}  \n     \\eeq \n","start":73148,"end":73594}]},{"id":"S040","semantic_type":"derived_fact","kind":"derived_fact","labels":["square-summable-step"],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"平方步长可求和","source_units":[{"raw_tex":"\t\\beq \\sum_{j=0}^\\infty \\alpha_j^2 = \\sum_{j=1}^\\infty \\frac{R^2}{j^{2s}} = R^2 \\zeta(2s) < \\infty\n\t\t\\label{square-summable-step}\t\n\t\\eeq \n","start":74723,"end":74861}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":["xqk-high-proba"],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"全平均的偏差与随机余项","source_units":[{"raw_tex":"We first give a proof for $q=1$, the proof for the remaining $q \\in (0,1)$ case will be similar. Assume $q=1$. Plugging $q=1$ and \\eqref{Yqk-helper} into \\eqref{xk-averaged-asymptotics-2}, we obtain\n     \\beqa \\bar{x}_{1,k} - x^* \n     &=&\\bigO(\\frac{\\log k}{k})  - H_*^{-1} \\aonek \\Yonek \\nonumber \\\\\n     &=& \\bigO(\\frac{\\log k}{k}) - H_*^{-1} \\bigg(   \\frac{\\sum_{j=0}^{k-1} \\alpha_j \\big(\\mu(\\s_j) - \\bar{\\mu}\\big)}{k} +  \\frac{\\sum_{j=0}^{k-1}  \\alpha_j \\bar{\\mu} + \\bigO(\\alpha_j^2) }{k}  \\bigg) \\nonumber \\\\\n     \t\t\t\t\t\t &=& b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{\\sum_{j=0}^{k-1}  \\alpha_j (\\mu(\\s_j) - \\bar{\\mu})}{k} - H_*^{-1} \\sum_{j=0}^{k-1}  \\frac{\\bigO(\\alpha_j^2)}{k} \\nonumber \\\\     \n     \t\t\t\t\t\t    &=& b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{\\sum_{j=0}^{k-1}  \\alpha_j (\\mu(\\s_j) - \\bar{\\mu})}{k} \\label{xqk-high-proba} %\\\\     \t\t\t\t\t\t    \n     \t\t\t\t\t\t%    &=& b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{M_{1,k}}{k}  \\label{xk-biased-high-probability}\n     \\eeqa    \nwhere $\\bonek$ is defined by \\eqref{def-bqk} and we used in the last step the fact that for $s>1/2$ \n","start":73594,"end":74723}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":["xk-biased-high-probability"],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"中心化加权排列和","source_units":[{"raw_tex":"where $\\zeta(\\cdot)$ is the Riemann-Zeta function. We now study the asymptotic behavior of the last summation term in \\eqref{xqk-high-proba} by introducing the process\n\t$S_{1,k} = \\sum_{j=0}^{k-1} Z_j$, where $Z_j := \\alpha_j (\\mu(\\s_j) - \\bar{\\mu})$ and $k\\geq 0$ with the convention that $S_{1,0} = 0$. Equipped with this definition, \\eqref{xqk-high-proba} becomes\n  \\beq \n  \t\t \\xonek - \\xs = b_{1,k}  +  \\bigO(\\frac{\\log k}{k})  - H_*^{-1}   \\frac{S_{1,k}}{k}.  \\label{xk-biased-high-probability}\n  \\eeq\t\t  \nThe random variables $Z_j$ are independent, centered and have an identical distribution up to the scaling factor $\\alpha_j$. Therefore,  $S_{1,k}$ is a sum of  centered random variables satisfying:\n","start":74861,"end":75570}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":["marting-diff-as-bound"],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"声称中心化增量界","source_units":[{"raw_tex":"    \\beqa \\begin{scriptsize} \\| S_{1,k} - S_{1,k-1} \\| =\\big \\|\\alpha_{k-1}\\big(\\mu(\\sigma_{k-1})-\\bar{\\mu}\\big)\\big\\| \n    \\leq \\gamma_{k-1}:=\\alpha_{k-1}LmG_* \\label{marting-diff-as-bound}\n    \\end{scriptsize}\n    \\eeqa \nwhere we used \\eqref{def-mu-sk-upper-bound} in the last inequality (see also Lemma \\ref{lem-computing-mu-bar}). Then, by the Azuma-Hoeffding inequality, for every $t>0$, \n","start":75570,"end":75964}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":["average-sum-bound-whp"],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"全平均误差的浓缩界","source_units":[{"raw_tex":"where we used \\eqref{def-mu-sk-upper-bound} in the last inequality (see also Lemma \\ref{lem-computing-mu-bar}). Then, by the Azuma-Hoeffding inequality, for every $t>0$, \n  \\beqas \\P \\bigg( \\big\\|\\frac{S_{1,k}}{k}\\big\\| > \\frac{t}{k}\\bigg) &\\leq& 2 \\exp \\bigg( - \\frac{t^2}{2\\sum_{j=0}^{k-1} \\gamma_j^2} \\bigg) =  2 \\exp \\big( - \\frac{t^2}{ \\beta }\\big) \n  \\eeqas\nwhere $\\beta = 2\\sum_{j=0}^{\\infty} \\gamma_j^2 < \\infty$ as $\\alpha_j$ is square-summable (see \\eqref{square-summable-step}). Note that $\\beta$ depends only on $\\Gs, L,m$ and the stepsize parameters $R$ and $s$. It is easy to see that selecting $t \\geq t_\\delta = \\sqrt{\\beta  \\log(2/\\delta)}$ makes the right-hand side $\\leq \\delta$.  Therefore for any $\\delta>0$, with probability at least $1-\\delta$, \n\t\t\\beq \\bigl \\| \\frac{S_{1,k}}{k} \\bigr \\| \\leq \\frac{\\sqrt{\\beta  \\log(2/\\delta)}}{k}\n\t\t\t\\label{average-sum-bound-whp}\n\t\t\\eeq\nwhich if inserted into the expression \\eqref{xk-biased-high-probability} completes the proof for the $q=1$ case. For  $0<q<1$ case, the same line of reasoning applies except that we replace $b_{1,k}$ with $b_{q,k}$ and we can improve the $\\bigO( \\log k /k)$ term in the expression \\eqref{xk-biased-high-probability} to $\\bigO(1/k)$, this is justified by \\eqref{xk-averaged-asymptotics-2}. Then, this leads to \n","start":75793,"end":77099}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["average-sum-bound-whp-3","xk-biased-high-probability-3"],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"后缀随机和与高概率界","source_units":[{"raw_tex":"which if inserted into the expression \\eqref{xk-biased-high-probability} completes the proof for the $q=1$ case. For  $0<q<1$ case, the same line of reasoning applies except that we replace $b_{1,k}$ with $b_{q,k}$ and we can improve the $\\bigO( \\log k /k)$ term in the expression \\eqref{xk-biased-high-probability} to $\\bigO(1/k)$, this is justified by \\eqref{xk-averaged-asymptotics-2}. Then, this leads to \n\t\t \\beq \n  \t\t \\xqk - \\xs = b_{q,k}  +  \\bigO(\\frac{1}{k})  - H_*^{-1}   \\frac{S_{q,k}}{qk}  \\label{xk-biased-high-probability-3}\n  \\eeq\nwhere $S_{q,k} := \\sum_{j=(1-q)k}^{k-1} Z_j = S_{1,k} - S_{1,(1-q)k}$ is the $q$-suffix cumulative sum (cumulative sum of the last $qk$ terms) of the sequence $Z_{k}$. Then using \\eqref{average-sum-bound-whp}, with probability at least $1-\\delta$,\n\\beq \\bigl \\| \\frac{S_{q,k}}{k} \\bigr \\| \\leq \\| \\frac{S_{1,k}}{k} \\bigr \\|  + \\| \\frac{S_{1,(1-q)k}}{k} \\bigr \\|  \\leq \\frac{2 t_\\delta}{k}.\n\t\t\t\\label{average-sum-bound-whp-3}\n\t\t\\eeq\nPlugging this high probability bound into \\eqref{xk-biased-high-probability-3}, we conclude. \t\t\n","start":76689,"end":77763}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["grad-diff-quadratic"],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"内层梯度接近最优点梯度","source_units":[{"raw_tex":"\t\\item [$(iv)$] By Lemma \\ref{lem-random-tail-bound}, we have $\\underset{1\\leq i < m}{\\max} \\|x_{i-1}^k - x^*\\| = \\bigO(\\alpha_k)$. Therefore,\n\t \\beqa \\scriptsize \\| \\nabla f_{\\ski} (x_{i-1}^k)  - \\nabla f_{\\ski} (x^*) \\|  = \\bigO(\\alpha_k) \\label{grad-diff-quadratic}\n\t \\eeqa\nfor any $i=1,2,\\dots,m$. As a consequence, \t\n","start":77763,"end":78085}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"替换梯度估计偏差","source_units":[{"raw_tex":"\t\\beqas \\hat{b}_{q,k} &=& -\\bar{\\alpha}_{q,k} H_*^{-1} \\sum_{i=1}^{m} P_{\\ski}  \\bigg(\\nabla f_{\\ski} (x^*) + \\bigO(\\alpha_k)\\bigg)  \\\\\n\t                         &=& - \\bar{\\alpha}_{q,k} H_*^{-1} \\sum_{j=1}^{m} P_{j} \\nabla f_j (x^*)   + \\bigO(\\alpha_k^2)   \n%\t                         &=& -\\bigg( H_*^{-1} + \\bigO(\\alpha_k) \\bigg) \\sum_{i=1}^{m} \\nabla^2 f_{i} (x_*) \\nabla f_i (x^*)  \\bar{\\alpha}_{q,k}+ \\bigO(\\alpha_k^2) +  o(\\alpha_k^2) \\\\\n\t                         = b_{q,k} + \\bigO(\\alpha_k^2)\t                         \t                        \t                         \n\t\\eeqas\t\t \nwhere in the second equality we use the fact that $\\bar{\\alpha}_{q,k}= \\bigO(1/k^s) = \\bigO(\\alpha_k)$ implied by part $(i)$.\n","start":78085,"end":78799}]},{"id":"S048","semantic_type":"lemma","kind":"lemma","labels":["lem-random-tail-bound"],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"所有内层迭代距离界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem-random-tail-bound}Under the conditions of Theorem \\ref{thm-averaged-rate-quad} we have \n    $\\underset{0 \\leq i < m}{\\max} \\| x_i^k - x^*\\| =  \\bigO(\\frac{1}{k^s}).$\nwhere $\\bigO(\\cdot)$ hides a constant that depends only on $\\Gs, L, m,c$ and $R$. \n\\end{lemma}","start":115000,"end":115284}]},{"id":"S049","semantic_type":"proof","kind":"proof","labels":["upper-bound-cycle-begin"],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"内层距离引理完整证明","source_units":[{"raw_tex":"\\begin{proof} By Corollary \\ref{coro-deter-decay-rate-dist}, \n   \\beq \\| x_0^k - x^* \\|  = \\bigO(\\frac{1}{k^s}).\n   \t\t\\label{upper-bound-cycle-begin}\n   \\eeq  \nwhere $\\bigO(\\cdot)$ hides a constant that depends only on $\\Gs, L, m,R$ and $c$.  We have also for any $0\\leq i< m$ and $k\\geq 0$,\n   \\beqas \\| x_i^k - x^*\\|  &\\leq& \\| x_0^k - x^*\\| + \\| x_i^k - x_0^k \\|  = \\| x_0^k - x^*\\| + i\\alpha_k \\max_{\\ell=1,\\dots,i} \\|\\nabla f_{\\sigma_k(\\ell)} (x_{\\ell-1}^k)\\| \\\\\n   &\\leq& \\| x_0^k - x^*\\| + (m-1)\\frac{R}{(k+1)^s} \\big(  G_* + \\max_{\\ell=1,\\dots,i} \\|\\nabla f_{\\sigma_k(\\ell)} (x_{\\ell-1}^k) - \\nabla f_{\\sigma_k(\\ell)} (x^*)\\| \\big) \\\\\n   &\\leq& \\| x_0^k - x^*\\| + (m-1)\\frac{R}{(k+1)^s} \\big(  G_* + L \\max_{\\ell=1,\\dots,i} \\|x_{\\ell-1}^k - x^*\\| \\big). \n   \\eeqas\nwhere we used the $L$-Lipschitzness of the gradient of $f$ where $L$ is given by \\ref{def-L}. Using \\eqref{upper-bound-cycle-begin} and applying this inequality inductively for $i=0,1,2,\\dots, m-1$ we conclude.\n\\end{proof}","start":115286,"end":116281}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":["upper-bound-cycle-begin"],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"控制周期起点距离","source_units":[{"raw_tex":"\\begin{proof} By Corollary \\ref{coro-deter-decay-rate-dist}, \n   \\beq \\| x_0^k - x^* \\|  = \\bigO(\\frac{1}{k^s}).\n   \t\t\\label{upper-bound-cycle-begin}\n   \\eeq  \nwhere $\\bigO(\\cdot)$ hides a constant that depends only on $\\Gs, L, m,R$ and $c$.  We have also for any $0\\leq i< m$ and $k\\geq 0$,\n","start":115286,"end":115578}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"内层位移递推与归纳","source_units":[{"raw_tex":"   \\beqas \\| x_i^k - x^*\\|  &\\leq& \\| x_0^k - x^*\\| + \\| x_i^k - x_0^k \\|  = \\| x_0^k - x^*\\| + i\\alpha_k \\max_{\\ell=1,\\dots,i} \\|\\nabla f_{\\sigma_k(\\ell)} (x_{\\ell-1}^k)\\| \\\\\n   &\\leq& \\| x_0^k - x^*\\| + (m-1)\\frac{R}{(k+1)^s} \\big(  G_* + \\max_{\\ell=1,\\dots,i} \\|\\nabla f_{\\sigma_k(\\ell)} (x_{\\ell-1}^k) - \\nabla f_{\\sigma_k(\\ell)} (x^*)\\| \\big) \\\\\n   &\\leq& \\| x_0^k - x^*\\| + (m-1)\\frac{R}{(k+1)^s} \\big(  G_* + L \\max_{\\ell=1,\\dots,i} \\|x_{\\ell-1}^k - x^*\\| \\big). \n   \\eeqas\nwhere we used the $L$-Lipschitzness of the gradient of $f$ where $L$ is given by \\ref{def-L}. Using \\eqref{upper-bound-cycle-begin} and applying this inequality inductively for $i=0,1,2,\\dots, m-1$ we conclude.\n\\end{proof}","start":115578,"end":116281}]},{"id":"S052","semantic_type":"lemma","kind":"lemma","labels":["def-I-qk","lem-I-lk"],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"平均归一化位移界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem-I-lk} Under the conditions of Theorem \\ref{thm-averaged-rate-quad}, consider the sequence\n\\beq I_{q,k} = \\frac{\\sum_{j=(1-q)k}^{k-1} {(x_0^j - x_{0}^{j+1})}{\\alpha_j^{-1}}}{qk}, \\quad  0 < q \\leq 1. \\label{def-I-qk}\n\\eeq\nThen, \n    $ I_{q, k} = \\begin{cases}  \\bigO \\big( \\frac{\\log k}{k}  \\big) \n     &  \\mbox{if} \\quad q=1, \\\\  \n    \t\t\t\\bigO \\big( \\frac{1}{k}\\big) \t & \\mbox{if} \\quad 0<q<1.  \n    \\end{cases}  \n%    &\\leq& \\bigO \\big( \\frac{\\log k}{k}\\big) \\\\\n%               I_{qk, k} &\\leq& \\bigO \\big( \\frac{1}{k}\\big) \\quad \\mbox{for all} \\quad 0<q<1.\n   $\nIn the former case, $\\bigO(\\cdot)$ hides a constant that depends only on $\\Gs, L,m,  c, R, s,q$ and $\\dist_0$. In the latter case, the same dependency on the constants occurs except that the dependency on $\\dist_0$ can be removed. \n\\end{lemma}","start":116644,"end":117475}]},{"id":"S053","semantic_type":"definition","kind":"definition","labels":["def-I-qk","lem-I-lk"],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"归一化外层平均位移","source_units":[{"raw_tex":"\\beq I_{q,k} = \\frac{\\sum_{j=(1-q)k}^{k-1} {(x_0^j - x_{0}^{j+1})}{\\alpha_j^{-1}}}{qk}, \\quad  0 < q \\leq 1. \\label{def-I-qk}\n\\eeq\n","start":116758,"end":116889}]},{"id":"S054","semantic_type":"proof","kind":"proof","labels":["init-averaged-iter-bound","step-times-dist-bd","sum-of-aver-iters","tail-suffix-averaging","weighted-iter-diff-bound"],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"平均位移引理完整证明","source_units":[{"raw_tex":"\\begin{proof}  It follows from integration by parts that for any $\\ell<k$, \n\\beq -\\sum_{j=\\ell}^{k-1}{(x_0^j - x_{0}^{j+1})}{\\alpha_j^{-1}}  =  \\alpha_k^{-1} (x_0^k - x^*) - \\alpha_\\ell^{-1} (x_0^\\ell - x^*) - \\sum_{j=\\ell}^{k-1} (x_0^{j+1} - x^*)(\\alpha_{j+1}^{-1}-\\alpha_j^{-1}). \\label{sum-of-aver-iters}\n\\eeq\nNext, we investigate the asymptotic behavior of the terms on the right-hand side. A consequence of Corollary \\ref{coro-deter-decay-rate-dist} and the inequality \\ref{def-M-gamma} is that  \n\t\\beq \\alpha_k^{-1} \\| x_0^k -x^*\\| = \\frac{(k+1)^s}{R} \\| x_0^k -x^*\\| \\leq \\frac{LmG_*}{c} + o(1) = \\bigO(1)\n\t\t\\label{step-times-dist-bd} \t\n\t\\eeq\nand therefore\n\\beqas \n\t| \\alpha_{k+1}^{-1} - \\alpha_{k}^{-1} |  \\| x_0^k -x^*\\| &=& \\frac{(k+2)^s - (k+1)^s} {(k+1)^s} \\alpha_k^{-1}\\| x_0^k -x^*\\|  \n\t = \\bigg( \\big(1 + \\frac{1}{k+1} \\big)^s -1 \\bigg) \\alpha_k^{-1}\\| x_0^k -x^*\\| \\\\\n\t  &\\leq &  \\frac{s}{k+1}\\alpha_k^{-1}\\| x_0^k -x^*\\|  \n\t  \\leq \\frac{sLmG_*}{c} \\frac{1}{k+1} + o(\\frac{1}{k+1}) = \\bigO(\\frac{1}{k+1})\n\\eeqas\nwhere $\\bigO(\\cdot)$ hides a constant that depends only on $L,\\Gs, c, m$ and $s$. Then, setting  $\\ell = (1-q)k$ in \\eqref{sum-of-aver-iters}, it follows that \n\\beqa  \\big\\| \\sum_{j=\\ell}^{k-1} {(x_0^j - x_0^{j+1})}{\\alpha_j^{-1}} \\big\\| &\\leq&  \\|\\alpha_k^{-1} (x_0^k - x^*)\\| + \\| \\alpha_{(1-q)k}^{-1} (x_0^{(1-q)k}- x^*)\\| \\\\\n  && + \\sum_{j=(1-q)k}^{k-1} \\| x_0^{j+1} - x^*\\| |\\alpha_{j+1}^{-1}-\\alpha_j^{-1}|.  \\nonumber\\\\\n       &=& \\bigO(1) + \\| \\alpha_{(1-q)k}^{-1} (x_0^{(1-q)k} - x^*)\\| + \\bigO \\bigg( \\sum_{j=(1-q)k}^{k-1} \\frac{1}{j+1}\\bigg).  \\label{weighted-iter-diff-bound} %\\\\\n       %&& + o \\bigg( \\sum_{j=\\ell}^{k-1} \\frac{1}{(j)^s} \\|(j+1)^s - j^s) \\|\\bigg). \\nonumber \\\\\n       %&=& \\bigO(1)  + \\| \\alpha_\\ell^{-1} (x_0^\\ell - x^*)\\| + \\bigO \\bigg( \\sum_{j=\\ell}^{k-1} \\big((1+\\frac{1}{j+1})^s - 1\\big) \\bigg) \\nonumber \\\\\n       %&& + o \\bigg( \\sum_{j=\\ell}^{k-1} \\big((1+\\frac{1}{j})^s - 1\\big) \\bigg). \\nonumber \\\\\n     % &=& \\bigO(1) + \\| \\alpha_\\ell^{-1} (x_0^\\ell - x^*)\\| + \\bigO \\bigg( \\sum_{j=\\ell}^{k-1} \\frac{s}{j+1} \\bigg) \n       %\\\\\n      %&\\leq& \\bigO(1) + \\bigO(\\frac{1}{k^{2s}} ) + \\bigO \\bigg( \\sum_{j=\\max(\\ell,1)}^{k-1} \\frac{s}{j} \\bigg) \\label{weighted-iter-diff-bound}\n\\eeqa\nWe also have\n  \\beq  \\| \\alpha_{(1-q)k}^{-1} (x_0^{(1-q)k} - x^*)\\|  = \\begin{cases} \\alpha_0^{-1} \\dist_0 & \\mbox{if} \\quad q = 1, \\\\ \\bigO(1)  & \\mbox{if} \\quad 0<q<1, \\end{cases} \\label{init-averaged-iter-bound}\n  \\eeq\nwhere the second part follows from \\eqref{step-times-dist-bd} with similar constants for the $\\bigO(\\cdot)$ term. As the sequence $\\frac{1}{j+1}$ is monotonically decreasing, for any $k>0$ we have the bounds \n\\beq  \\sum_{j=(1-q)k}^{k-1} \\frac{1}{j+1} \\leq \\frac{1}{(1-q)k + 1} + \\int_{(1-q)k}^{k-1} \\frac{1}{x+1} dx \\leq \\begin{cases} 1 + \\log k & \\mbox{if } \\quad  q=1, \\\\\n 1 + \\log(\\frac{1}{1-q})  & \\mbox{if } \\quad  0 <q <1.\n\\end{cases}  \\label{tail-suffix-averaging}\n\\eeq\nNote that when $q=1$ this bound grows with $k$ logarithmically whereas for $q<1$ it does not grow with $k$. \nThen, combining \\eqref{weighted-iter-diff-bound}, \\eqref{init-averaged-iter-bound} and \\eqref{tail-suffix-averaging}  we obtain\n  $$ \\| \\Iqk \\| \\leq \\frac{\\big\\| \\sum_{j=\\ell}^{k-1} {(x_0^j - x_0^{j+1})}{\\alpha_j^{-1}} \\big\\|}{qk} = \\begin{cases}  \\bigO \\big(  \\frac{\\log k}{k}\\big) \n     &  \\mbox{if} \\quad q=1 \\\\  \n    \t\t\t\\bigO \\big( \\frac{1}{k}\\big) \t & \\mbox{if} \\quad 0<q<1  \n    \\end{cases}  $$\nas desired which completes the proof.    \n\\end{proof}","start":117612,"end":121119}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["sum-of-aver-iters"],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"离散分部求和","source_units":[{"raw_tex":"\\begin{proof}  It follows from integration by parts that for any $\\ell<k$, \n\\beq -\\sum_{j=\\ell}^{k-1}{(x_0^j - x_{0}^{j+1})}{\\alpha_j^{-1}}  =  \\alpha_k^{-1} (x_0^k - x^*) - \\alpha_\\ell^{-1} (x_0^\\ell - x^*) - \\sum_{j=\\ell}^{k-1} (x_0^{j+1} - x^*)(\\alpha_{j+1}^{-1}-\\alpha_j^{-1}). \\label{sum-of-aver-iters}\n\\eeq\n","start":117612,"end":117925}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":["step-times-dist-bd"],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"步长归一化距离有界","source_units":[{"raw_tex":"Next, we investigate the asymptotic behavior of the terms on the right-hand side. A consequence of Corollary \\ref{coro-deter-decay-rate-dist} and the inequality \\ref{def-M-gamma} is that  \n\t\\beq \\alpha_k^{-1} \\| x_0^k -x^*\\| = \\frac{(k+1)^s}{R} \\| x_0^k -x^*\\| \\leq \\frac{LmG_*}{c} + o(1) = \\bigO(1)\n\t\t\\label{step-times-dist-bd} \t\n\t\\eeq\n","start":117925,"end":118262}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"逆步长增量的调和控制","source_units":[{"raw_tex":"and therefore\n\\beqas \n\t| \\alpha_{k+1}^{-1} - \\alpha_{k}^{-1} |  \\| x_0^k -x^*\\| &=& \\frac{(k+2)^s - (k+1)^s} {(k+1)^s} \\alpha_k^{-1}\\| x_0^k -x^*\\|  \n\t = \\bigg( \\big(1 + \\frac{1}{k+1} \\big)^s -1 \\bigg) \\alpha_k^{-1}\\| x_0^k -x^*\\| \\\\\n\t  &\\leq &  \\frac{s}{k+1}\\alpha_k^{-1}\\| x_0^k -x^*\\|  \n\t  \\leq \\frac{sLmG_*}{c} \\frac{1}{k+1} + o(\\frac{1}{k+1}) = \\bigO(\\frac{1}{k+1})\n\\eeqas\n","start":118262,"end":118640}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":["init-averaged-iter-bound","weighted-iter-diff-bound"],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"后缀端点与位移和","source_units":[{"raw_tex":"where $\\bigO(\\cdot)$ hides a constant that depends only on $L,\\Gs, c, m$ and $s$. Then, setting  $\\ell = (1-q)k$ in \\eqref{sum-of-aver-iters}, it follows that \n\\beqa  \\big\\| \\sum_{j=\\ell}^{k-1} {(x_0^j - x_0^{j+1})}{\\alpha_j^{-1}} \\big\\| &\\leq&  \\|\\alpha_k^{-1} (x_0^k - x^*)\\| + \\| \\alpha_{(1-q)k}^{-1} (x_0^{(1-q)k}- x^*)\\| \\\\\n  && + \\sum_{j=(1-q)k}^{k-1} \\| x_0^{j+1} - x^*\\| |\\alpha_{j+1}^{-1}-\\alpha_j^{-1}|.  \\nonumber\\\\\n       &=& \\bigO(1) + \\| \\alpha_{(1-q)k}^{-1} (x_0^{(1-q)k} - x^*)\\| + \\bigO \\bigg( \\sum_{j=(1-q)k}^{k-1} \\frac{1}{j+1}\\bigg).  \\label{weighted-iter-diff-bound} %\\\\\n       %&& + o \\bigg( \\sum_{j=\\ell}^{k-1} \\frac{1}{(j)^s} \\|(j+1)^s - j^s) \\|\\bigg). \\nonumber \\\\\n       %&=& \\bigO(1)  + \\| \\alpha_\\ell^{-1} (x_0^\\ell - x^*)\\| + \\bigO \\bigg( \\sum_{j=\\ell}^{k-1} \\big((1+\\frac{1}{j+1})^s - 1\\big) \\bigg) \\nonumber \\\\\n       %&& + o \\bigg( \\sum_{j=\\ell}^{k-1} \\big((1+\\frac{1}{j})^s - 1\\big) \\bigg). \\nonumber \\\\\n     % &=& \\bigO(1) + \\| \\alpha_\\ell^{-1} (x_0^\\ell - x^*)\\| + \\bigO \\bigg( \\sum_{j=\\ell}^{k-1} \\frac{s}{j+1} \\bigg) \n       %\\\\\n      %&\\leq& \\bigO(1) + \\bigO(\\frac{1}{k^{2s}} ) + \\bigO \\bigg( \\sum_{j=\\max(\\ell,1)}^{k-1} \\frac{s}{j} \\bigg) \\label{weighted-iter-diff-bound}\n\\eeqa\nWe also have\n  \\beq  \\| \\alpha_{(1-q)k}^{-1} (x_0^{(1-q)k} - x^*)\\|  = \\begin{cases} \\alpha_0^{-1} \\dist_0 & \\mbox{if} \\quad q = 1, \\\\ \\bigO(1)  & \\mbox{if} \\quad 0<q<1, \\end{cases} \\label{init-averaged-iter-bound}\n  \\eeq\nwhere the second part follows from \\eqref{step-times-dist-bd} with similar constants for the $\\bigO(\\cdot)$ term. As the sequence $\\frac{1}{j+1}$ is monotonically decreasing, for any $k>0$ we have the bounds \n","start":118640,"end":120288}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":["tail-suffix-averaging"],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"后缀调和和估计","source_units":[{"raw_tex":"\\beq  \\sum_{j=(1-q)k}^{k-1} \\frac{1}{j+1} \\leq \\frac{1}{(1-q)k + 1} + \\int_{(1-q)k}^{k-1} \\frac{1}{x+1} dx \\leq \\begin{cases} 1 + \\log k & \\mbox{if } \\quad  q=1, \\\\\n 1 + \\log(\\frac{1}{1-q})  & \\mbox{if } \\quad  0 <q <1.\n\\end{cases}  \\label{tail-suffix-averaging}\n\\eeq\nNote that when $q=1$ this bound grows with $k$ logarithmically whereas for $q<1$ it does not grow with $k$. \n","start":120288,"end":120665}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"除以样本数完成位移界","source_units":[{"raw_tex":"Then, combining \\eqref{weighted-iter-diff-bound}, \\eqref{init-averaged-iter-bound} and \\eqref{tail-suffix-averaging}  we obtain\n  $$ \\| \\Iqk \\| \\leq \\frac{\\big\\| \\sum_{j=\\ell}^{k-1} {(x_0^j - x_0^{j+1})}{\\alpha_j^{-1}} \\big\\|}{qk} = \\begin{cases}  \\bigO \\big(  \\frac{\\log k}{k}\\big) \n     &  \\mbox{if} \\quad q=1 \\\\  \n    \t\t\t\\bigO \\big( \\frac{1}{k}\\big) \t & \\mbox{if} \\quad 0<q<1  \n    \\end{cases}  $$\nas desired which completes the proof.    \n\\end{proof}","start":120665,"end":121119}]},{"id":"S061","semantic_type":"lemma","kind":"lemma","labels":["lem-computing-mu-bar"],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"均匀排列偏差均值","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem-computing-mu-bar} Let $\\sigma$ be a random permutation of $\\{1,2,\\dots,m\\}$ sampled uniformly over the set of all permutations $\\Gamma$ defined by \\eqref{def-Gamma-perm-set} and $\\mu(\\sigma)$ be the vector defined by \\eqref{def-Msigma} that depends on $\\sigma$. Then, \\beq\n\t\\bar{\\mu} = \\E_{\\sigma}\\big(\\mu({\\sigma})\\big) = \\frac{1}{2} \\sum_{i=1}^m P_i \\nabla f_i(x^*)\n\\eeq\nwhere $\\E_{\\sigma}$ denotes the expectation with respect to the random permutation $\\sigma$ and $\\bar{mu}$ is defined by \\eqref{def-mu-bar}.\n\\end{lemma}","start":121121,"end":121670}]},{"id":"S062","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"排列均值引理完整证明","source_units":[{"raw_tex":"\\begin{proof} For any $i \\neq \\ell$, the joint distribution of $(\\sigma(i), \\sigma(\\ell))$ is uniform over the set of all (ordered) pairs from $\\{1,2,\\dots,m\\}$. Therefore, for any $i \\neq \\ell$, \n    \\beqas\n    \\E_\\sigma \\big[ P_{\\sigma(i)} \\nabla f_{\\sigma(\\ell) }(x^*) \\big] &=& \\sum_{i=1}^m \\sum_{i\\neq j, j=1}^m \\frac{P_i\\nabla f_j(x^*)}{m(m-1)} \\\\\n    &=& \\frac{\\sum_{i=1}^m P_i  \\sum_{j=1}^m \\nabla f_j(x^*) - \\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)} \n    =  - \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)}\n    \\eeqas \nwhere we used the fact that $\\nabla f(x^*) = \\sum_{j=1}^m \\nabla f_j(x^*)=0$ by the first order optimality condition. Then, by taking the expectation of \\eqref{def-v-sigma-k}, we obtain \n\\beqas \\E_\\sigma (\\mu(\\sigma)) &=& - \\sum_{i=1}^{m} \\sum_{\\ell=0}^{i-1} \\E \\big[ P_{\\sigma(i)} \\nabla f_{\\sigma(\\ell)}(x^*) \\big] \n        =  \\sum_{i=1}^{m} \\sum_{\\ell=0}^{i-1}  \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)} = \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{2}.\n\\eeqas\t \nwhich completes the proof.\n\\end{proof}","start":121671,"end":122706}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"不同排列位置的联合期望","source_units":[{"raw_tex":"\\begin{proof} For any $i \\neq \\ell$, the joint distribution of $(\\sigma(i), \\sigma(\\ell))$ is uniform over the set of all (ordered) pairs from $\\{1,2,\\dots,m\\}$. Therefore, for any $i \\neq \\ell$, \n    \\beqas\n    \\E_\\sigma \\big[ P_{\\sigma(i)} \\nabla f_{\\sigma(\\ell) }(x^*) \\big] &=& \\sum_{i=1}^m \\sum_{i\\neq j, j=1}^m \\frac{P_i\\nabla f_j(x^*)}{m(m-1)} \\\\\n    &=& \\frac{\\sum_{i=1}^m P_i  \\sum_{j=1}^m \\nabla f_j(x^*) - \\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)} \n    =  - \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)}\n    \\eeqas \n","start":121671,"end":122200}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S064"],"declared_external":false,"source_only":false,"title":"最优性消去与对数求和","source_units":[{"raw_tex":"where we used the fact that $\\nabla f(x^*) = \\sum_{j=1}^m \\nabla f_j(x^*)=0$ by the first order optimality condition. Then, by taking the expectation of \\eqref{def-v-sigma-k}, we obtain \n\\beqas \\E_\\sigma (\\mu(\\sigma)) &=& - \\sum_{i=1}^{m} \\sum_{\\ell=0}^{i-1} \\E \\big[ P_{\\sigma(i)} \\nabla f_{\\sigma(\\ell)}(x^*) \\big] \n        =  \\sum_{i=1}^{m} \\sum_{\\ell=0}^{i-1}  \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)} = \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{2}.\n\\eeqas\t \nwhich completes the proof.\n\\end{proof}","start":122200,"end":122706}]},{"id":"S065","semantic_type":"lemma","kind":"lemma","labels":["Ek-expression","def-mu-sk-upper-bound","def-v-sigma-k","lem-martingale-variance"],"member_ids":["S065"],"declared_external":false,"source_only":false,"title":"误差解耦及加权平均极限","source_units":[{"raw_tex":"\\begin{lemma} \\label{lem-martingale-variance} Under the conditions of Theorem \\ref{thm-averaged-rate-quad},  the following statements are true: \n\\begin{itemize} \n   \\item [$(i)$] We have  \\beq E_k = \\alpha_k \\mu(\\sigma_k)  + \\bigO(\\alpha_k^2), \\quad k\\geq 0,  \\label{Ek-expression} \\eeq \nwhere $E_k$ is the gradient error defined by \\eqref{def-Ek}, $\\bigO(\\cdot)$ hides a constant that depends only on $\\Gs, L,m, R$ and $c$ and\n   \\beq  \\mu(\\sk) = - \\sum_{i=1}^{m} P_{\\ski} \\sum_{\\ell=1}^{i-1} \\nabla f_{\\skl}(x^*). \n\t\t\\label{def-v-sigma-k}   \n   \\eeq\nis a sequence of i.i.d. variables where the function $\\mu(\\cdot)$ is defined by \\eqref{def-Msigma}. \n%    \\item [$(ii)$]  It holds that   \n%    \\beq\n%           \\|\\mu(\\s_k)\\| \\leq LmG_* \n%           \t\t\\label{v-inf-norm}\n%     \\eeq\n% where $L=\\sum_i \\|P_i\\|$ and $\\Gs$ is defined by \\eqref{def-Gs}. \n% Furthermore,    \n%     \\beq      \n%        \\E v(\\sk) = \\bar{\\mu} =   \\frac{1}{2} \\sum_{i=1}^m P_i \\nabla f_i(x^*). % \\label{def-mu-star}\n%    \\eeq\n   %\\item [$(iii)$] $\\sigma_*^2 := var(v(u^k)) <\\infty$.\n   \\item [$(ii)$] For any $0 <q\\leq 1 $,  $\\lim_{k \\to \\infty} Y_{q,k}=  \\bar{\\mu}$ a.s. where \n    $ Y_{q,k} = \\frac{\\sum_{i=(1-q)k}^{k-1} E_j}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}.$\n   \\item [$(iii)$] It holds that \n       \\beq  \\| \\mu(\\sigma_k) \\| \\leq Lm G_*.\t\n       \t\t\\label{def-mu-sk-upper-bound}\n       \\eeq\t\n\\end{itemize}  \n   \n\n\\end{lemma}","start":122707,"end":124108}]},{"id":"S066","semantic_type":"definition","kind":"definition","labels":["def-v-sigma-k","lem-martingale-variance"],"member_ids":["S066"],"declared_external":false,"source_only":false,"title":"排列误差的嵌套和表示","source_units":[{"raw_tex":"   \\beq  \\mu(\\sk) = - \\sum_{i=1}^{m} P_{\\ski} \\sum_{\\ell=1}^{i-1} \\nabla f_{\\skl}(x^*). \n\t\t\\label{def-v-sigma-k}   \n   \\eeq\nis a sequence of i.i.d. variables where the function $\\mu(\\cdot)$ is defined by \\eqref{def-Msigma}. \n","start":123135,"end":123360}]},{"id":"S067","semantic_type":"lemma","kind":"lemma","labels":["Ek-expression","lem-martingale-variance"],"member_ids":["S067"],"declared_external":false,"source_only":false,"title":"周期误差二阶解耦","source_units":[{"raw_tex":"\\begin{lemma} \\label{lem-martingale-variance} Under the conditions of Theorem \\ref{thm-averaged-rate-quad},  the following statements are true: \n\\begin{itemize} \n   \\item [$(i)$] We have  \\beq E_k = \\alpha_k \\mu(\\sigma_k)  + \\bigO(\\alpha_k^2), \\quad k\\geq 0,  \\label{Ek-expression} \\eeq \nwhere $E_k$ is the gradient error defined by \\eqref{def-Ek}, $\\bigO(\\cdot)$ hides a constant that depends only on $\\Gs, L,m, R$ and $c$ and\n","start":122707,"end":123135}]},{"id":"S068","semantic_type":"lemma","kind":"lemma","labels":["lem-martingale-variance"],"member_ids":["S068"],"declared_external":false,"source_only":false,"title":"归一化后缀误差极限","source_units":[{"raw_tex":"   \\item [$(ii)$] For any $0 <q\\leq 1 $,  $\\lim_{k \\to \\infty} Y_{q,k}=  \\bar{\\mu}$ a.s. where \n    $ Y_{q,k} = \\frac{\\sum_{i=(1-q)k}^{k-1} E_j}{\\sum_{j=(1-q)k}^{k-1} \\alpha_j}.$\n","start":123764,"end":123943}]},{"id":"S069","semantic_type":"lemma","kind":"lemma","labels":["def-mu-sk-upper-bound","lem-martingale-variance"],"member_ids":["S069"],"declared_external":false,"source_only":false,"title":"独立排列误差有界","source_units":[{"raw_tex":"   \\item [$(iii)$] It holds that \n       \\beq  \\| \\mu(\\sigma_k) \\| \\leq Lm G_*.\t\n       \t\t\\label{def-mu-sk-upper-bound}\n       \\eeq\t\n\\end{itemize}  \n   \n\n\\end{lemma}","start":123943,"end":124108}]},{"id":"S070","semantic_type":"proof","kind":"proof","labels":["lim-aver-vuj","lim-average-mu-star","quadratic-only-one"],"member_ids":["S070"],"declared_external":false,"source_only":false,"title":"误差解耦引理完整证明","source_units":[{"raw_tex":"\\begin{proof} \n\n\t\\begin{itemize} \n\t\t\t\\item [$(i)$] As component functions are quadratics, \\eqref{def-Ek} becomes\n\\beqas E_k &=& \\sum_{i=1}^{m} P_{\\ski}(x_{i-1}^k - x_0^k) = - \\sum_{i=1}^{m}  P_{\\ski}  \\alpha_k \\sum_{\\ell=1}^{i-1} \\nabla f_{\\skl } (x_{\\ell-1}^k). \n \\eeqas\nwhere we can substitute \n \\beq  \\nabla f_{ \\skl } (x_{\\ell-1}^k) =  \\nabla f_{ \\skl } (x^*)  + P_{\\skl} (x_{\\ell-1}^k - x^*).\n \t\\label{quadratic-only-one}\n \\eeq\nThen an application of Lemma \\ref{lem-random-tail-bound}  proves directly the desired result.\n%  \t\t\t\\item [$(ii)$] For the first part, the inequality \\eqref{v-inf-norm} is just a consequence of the triangle inequality applied to the definition \\eqref{def-v-sigma-k} with $L_i = \\| P_i \\|$ and $L=\\sum_{i=1}^m L_i$. For the second part, note that for any $i \\neq \\ell$, the joint distribution of $(\\ski, \\skl)$ is uniform over the set of all (ordered) pairs from $\\{1,2,\\dots,m\\}$. Therefore, for any $i \\neq \\ell$, \n%     \\beqas\n%     \\E \\big[ P_{\\ski} \\nabla f_{\\skl }(x^*) \\big] &=& \\sum_{i=1}^m \\sum_{i\\neq j, j=1}^m \\frac{P_i\\nabla f_j(x^*)}{m(m-1)} \\\\\n%     &=& \\frac{\\sum_{i=1}^m P_i  \\sum_{j=1}^m \\nabla f_j(x^*) - \\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)} \\\\\n%     &=&  - \\frac{\\sum_{j=1}^m P_j \\nabla f_i(x^*)}{m(m-1)}\n%     \\eeqas \n% where we used the fact that $\\nabla f(x^*) = \\sum_{j=1}^m \\nabla f_j(x^*)=0$ by the first order optimality condition. Then, by taking the expectation of \\eqref{def-v-sigma-k}, we obtain \n% \\beqas \\E v(k) &=& - \\sum_{i=1}^{m} \\sum_{\\ell=0}^{i-1} \\E \\big[ P_{\\ski} \\nabla f_{\\skl}(x^*) \\big] \\\\\n%         &=& - \\sum_{i=1}^{m} \\sum_{\\ell=0}^{i-1} - \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{m(m-1)} \\\\\n%         &=&  \\frac{\\sum_{j=1}^m P_j \\nabla f_j(x^*)}{2}.\n% \\eeqas\t \n% which completes the proof.\n%           \\item [$(iii)$] All the gradients and Hessians of the component functions at the global optimum $x^*$ is finite and there are finitely many component functions, therefore all the moments of $v(u^j)$ exists and is finite including the second moment. Therefore, the variance $\\sigma_*^2$ is bounded.\\footnote{Note that $\\sigma_*^2$ could be computed explicitly by a similar calculation to the one in the proof of part $(ii)$.} This completes the proof.\n   \t\t\t\\item [$(ii)$] We introduce the normalized gradient error sequence $Y_j = E_j / \\alpha_j$. By part $(i)$, $Y_j = \\mu(\\sigma_j) + \\bigO(\\alpha_j)$ where $\\mu(\\sigma_j)$ is a sequence of i.i.d. variables. By the strong law of large numbers, we have\n   \t\t\t     \\beq \\lim_{k \\to \\infty} \\frac{\\sum_{j=0}^{k-1} \\mu(\\sigma_j)}{k} = \\E \\mu(\\sigma_j) = \\bar{\\mu} \\quad \\mbox{a.s.}\\label{lim-aver-vuj}  \\eeq\nwhere the last equality is by the definition of $\\bar{\\mu}$. Therefore,   \t\t\t\n    \t\t\t \\beqas \\lim_{k \\to \\infty} \\frac{\\sum_{j=0}^{k-1} Y_j}{k} = \\lim_{k \\to \\infty} \\bigg(\\frac{\\sum_{j=0}^{k-1} \\mu(\\sigma_j)}{k} + \\frac{\\sum_{j=0}^{k-1} \\bigO(\\alpha_j)}{k}\\bigg) = \\bar{\\mu} \\quad \\mbox{a.s.} \\eeqas\nwhere we used the fact that the second term is negligible as $\\sum_{j=0}^{k-1} \\alpha_j / k = \\bigO(k^{-s}) \\to 0$. \nAs the average of the sequence $Y_j$ converges almost surely, one can show that this implies almost sure convergence of a weighted average of the sequence $Y_j$ as well as long as weights satisfy certain conditions as $k \\to \\infty$. In particular, as the sequence $\\{\\alpha_j\\}$ is monotonically decreasing and is non-summable, by \\cite[Theorem 1]{Etemadi:2006ko}, \n       \\beq \\lim_{k \\to \\infty} Y_{1,k} = \\lim_{k \\to \\infty}\\frac{\\sum_{j=0}^{k-1} \\alpha_j Y_j}{\\sum_{j=0}^{k-1} \\alpha_j} =  \\lim_{k \\to \\infty} \\frac{\\sum_{j=0}^{k-1} E_j}{\\sum_{j=0}^{k-1} \\alpha_j} = \\bar{\\mu} \\quad \\mbox{a.s.}\\label{lim-average-mu-star}\n       \\eeq \nThis completes the proof for $q=1$. For $0<q<1$, by the definition of $Y_{q,k}$, we can write $Y_{1,k} = (1-w_k) Y_{q,k} + w_k Y_{1, (1-q)k}$ where the non-negative weights $w_k$ satisfy    \n$$w_k = \\frac{\\sum_{j=0}^{(1-q)k - 1 } \\alpha_j}{\\sum_{j=0}^{k - 1 } \\alpha_j } \\to_{k \\to \\infty} (1-q)^{1-s} < 1.$$     \n    As both $Y_{1,k}$ and $Y_{1, (1-q)k}$ go to $\\bar{\\mu}$ a.s. by \\eqref{lim-average-mu-star}, it follows that \n$$\\lim_{k\\to \\infty} Y_{q,k} = \\lim_{k\\to \\infty}  \\frac{Y_{1,k} - w_k Y_{1,(1-q)k}}{1-w_k} = \\bar{\\mu}\\quad \\mbox{a.s}$$ as well for any $0<q<1$. This completes the proof.\n\t\\item [$(iii)$] This is a direct consequence of the triangle inequality applied to the definition \\eqref{def-v-sigma-k} with $L_i = \\|P_i\\|$ and $L=\\sum_{i=1}^m L_i$. \n\\end{itemize}\n\\end{proof}","start":124109,"end":128603}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":["quadratic-only-one"],"member_ids":["S071"],"declared_external":false,"source_only":false,"title":"展开周期误差并替换内层点","source_units":[{"raw_tex":"\\begin{proof} \n\n\t\\begin{itemize} \n\t\t\t\\item [$(i)$] As component functions are quadratics, \\eqref{def-Ek} becomes\n\\beqas E_k &=& \\sum_{i=1}^{m} P_{\\ski}(x_{i-1}^k - x_0^k) = - \\sum_{i=1}^{m}  P_{\\ski}  \\alpha_k \\sum_{\\ell=1}^{i-1} \\nabla f_{\\skl } (x_{\\ell-1}^k). \n \\eeqas\nwhere we can substitute \n \\beq  \\nabla f_{ \\skl } (x_{\\ell-1}^k) =  \\nabla f_{ \\skl } (x^*)  + P_{\\skl} (x_{\\ell-1}^k - x^*).\n \t\\label{quadratic-only-one}\n \\eeq\nThen an application of Lemma \\ref{lem-random-tail-bound}  proves directly the desired result.\n","start":124109,"end":124636}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":["lim-aver-vuj"],"member_ids":["S072"],"declared_external":false,"source_only":false,"title":"普通平均的强大数极限","source_units":[{"raw_tex":"   \t\t\t\\item [$(ii)$] We introduce the normalized gradient error sequence $Y_j = E_j / \\alpha_j$. By part $(i)$, $Y_j = \\mu(\\sigma_j) + \\bigO(\\alpha_j)$ where $\\mu(\\sigma_j)$ is a sequence of i.i.d. variables. By the strong law of large numbers, we have\n   \t\t\t     \\beq \\lim_{k \\to \\infty} \\frac{\\sum_{j=0}^{k-1} \\mu(\\sigma_j)}{k} = \\E \\mu(\\sigma_j) = \\bar{\\mu} \\quad \\mbox{a.s.}\\label{lim-aver-vuj}  \\eeq\nwhere the last equality is by the definition of $\\bar{\\mu}$. Therefore,   \t\t\t\n    \t\t\t \\beqas \\lim_{k \\to \\infty} \\frac{\\sum_{j=0}^{k-1} Y_j}{k} = \\lim_{k \\to \\infty} \\bigg(\\frac{\\sum_{j=0}^{k-1} \\mu(\\sigma_j)}{k} + \\frac{\\sum_{j=0}^{k-1} \\bigO(\\alpha_j)}{k}\\bigg) = \\bar{\\mu} \\quad \\mbox{a.s.} \\eeqas\nwhere we used the fact that the second term is negligible as $\\sum_{j=0}^{k-1} \\alpha_j / k = \\bigO(k^{-s}) \\to 0$. \n","start":126345,"end":127168}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":["lim-average-mu-star"],"member_ids":["S073"],"declared_external":false,"source_only":false,"title":"应用加权平均收敛定理","source_units":[{"raw_tex":"As the average of the sequence $Y_j$ converges almost surely, one can show that this implies almost sure convergence of a weighted average of the sequence $Y_j$ as well as long as weights satisfy certain conditions as $k \\to \\infty$. In particular, as the sequence $\\{\\alpha_j\\}$ is monotonically decreasing and is non-summable, by \\cite[Theorem 1]{Etemadi:2006ko}, \n       \\beq \\lim_{k \\to \\infty} Y_{1,k} = \\lim_{k \\to \\infty}\\frac{\\sum_{j=0}^{k-1} \\alpha_j Y_j}{\\sum_{j=0}^{k-1} \\alpha_j} =  \\lim_{k \\to \\infty} \\frac{\\sum_{j=0}^{k-1} E_j}{\\sum_{j=0}^{k-1} \\alpha_j} = \\bar{\\mu} \\quad \\mbox{a.s.}\\label{lim-average-mu-star}\n       \\eeq \n","start":127168,"end":127808}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S074"],"declared_external":false,"source_only":false,"title":"由前缀极限获得后缀极限","source_units":[{"raw_tex":"This completes the proof for $q=1$. For $0<q<1$, by the definition of $Y_{q,k}$, we can write $Y_{1,k} = (1-w_k) Y_{q,k} + w_k Y_{1, (1-q)k}$ where the non-negative weights $w_k$ satisfy    \n$$w_k = \\frac{\\sum_{j=0}^{(1-q)k - 1 } \\alpha_j}{\\sum_{j=0}^{k - 1 } \\alpha_j } \\to_{k \\to \\infty} (1-q)^{1-s} < 1.$$     \n    As both $Y_{1,k}$ and $Y_{1, (1-q)k}$ go to $\\bar{\\mu}$ a.s. by \\eqref{lim-average-mu-star}, it follows that \n$$\\lim_{k\\to \\infty} Y_{q,k} = \\lim_{k\\to \\infty}  \\frac{Y_{1,k} - w_k Y_{1,(1-q)k}}{1-w_k} = \\bar{\\mu}\\quad \\mbox{a.s}$$ as well for any $0<q<1$. This completes the proof.\n","start":127808,"end":128409}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S075"],"declared_external":false,"source_only":false,"title":"三角不等式控制排列误差","source_units":[{"raw_tex":"\t\\item [$(iii)$] This is a direct consequence of the triangle inequality applied to the definition \\eqref{def-v-sigma-k} with $L_i = \\|P_i\\|$ and $L=\\sum_{i=1}^m L_i$. \n\\end{itemize}\n\\end{proof}","start":128409,"end":128603}]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S076"],"declared_external":true,"declaration_text":["原始外部前提清单：标准一阶最优性事实；作者在 L1100 明确使用，在 L617 隐含使用。"],"source_only":false,"title":"无约束一阶最优性","source_units":[]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S077"],"declared_external":true,"declaration_text":["原始外部前提清单：标准强凸性及谱范数事实。"],"source_only":false,"title":"强凸与谱范数事实","source_units":[]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S078"],"declared_external":true,"declaration_text":["原始外部前提清单：summation by parts and integral bounds for series；幂函数增量界为标注者显式列出的隐含标准事实。"],"source_only":false,"title":"分部求和与级数积分界","source_units":[]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S079"],"declared_external":true,"declaration_text":["原始外部前提清单：Strong law of large numbers for bounded i.i.d. random vectors；作者在 L1159–1160 调用。"],"source_only":false,"title":"有界独立向量强大数律","source_units":[]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S080"],"declared_external":true,"declaration_text":["原始外部前提清单：Etemadi:2006ko, Theorem 1；活动引用位于 L1164。"],"source_only":false,"title":"Etemadi 加权平均结果","source_units":[]},{"id":"S081","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S081"],"declared_external":true,"declaration_text":["原始外部前提清单：Azuma–Hoeffding concentration, with an applicable vector-norm formulation required for lines 683–689。"],"source_only":false,"title":"所需向量 Azuma–Hoeffding 界","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S022":"S022","S023":"S023","S024":"S024","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076","S077":"S077","S078":"S078","S079":"S079","S080":"S080","S081":"S081"}}}}
{"record_id":"2006.02032v5","paper":{"title":"A Unified Single-loop Alternating Gradient Projection Algorithm for Nonconvex-Concave and Convex-Nonconcave Minimax Problems","authors":["Xu, Zi","Zhang, Huiling","Xu, Yang","Lan, Guanghui"],"arxiv_id":"2006.02032","version":"v5","journal":"Mathematical Programming","abstract_url":"https://arxiv.org/abs/2006.02032v5","pdf_url":"https://arxiv.org/pdf/2006.02032v5","doi":"10.1007/s10107-022-01919-z"},"context":{"raw_tex":"\tWe consider the following minimax optimization problem:\n\t\\begin{equation}\\label{problem:1}\n\t\t\\min \\limits_{x\\in \\mathcal{X}}\\max \\limits_{y\\in \\mathcal{Y}}f(x,y),\\tag{P}\n\t\\end{equation}\n\twhere $\\mathcal{X}\\subseteq \\mathbb{R}^n$ and $\\mathcal{Y}\\subseteq \\mathbb{R}^m$ are nonempty closed and bounded convex sets, and $f: \\mathcal{X} \\times \\mathcal{Y} \\rightarrow \\mathbb{R}$\n\tis a smooth function. This problem has attracted more  attention due to its wide applications in machine learning, signal processing, and many other research fields in recent years.\n\n\n\t{\\bfseries Notation}. For vectors, we use $\\|\\cdot\\|$ to denote  the $l_2$-norm. For a function $f(x,y):\\mathbb{R}^n\\times\\mathbb{R}^m\\rightarrow \\mathbb{R}$, we use $\\nabla_{x} f(x,y)$ (or $\\nabla_{y} f(x, y)$) to denote the partial gradient of $f$ with respect to the first variable (or the second variable) at point $(x, y)$.\n\tLet $\\mathcal{P}_{\\mathcal{X}}$ and $\\mathcal{P}_{\\mathcal{Y}}$ denote projections onto the sets $\\mathcal{X}$ and $\\mathcal{Y}$. Finally, we use the notation $\\mathcal{O} (\\cdot)$ to hide only absolute constants which do not depend on any problem parameter, and $\\tilde{\\mathcal{O}}(\\cdot)$ notation to hide only absolute constants and log factors.\n\n\nA continuously differentiable function $f(\\cdot)$ is called $\\theta$-strongly convex if there exists a constant $\\theta> 0$ such\nthat for any $x,y \\in \\mathcal{X}$,\n\\begin{equation}\n  f(y)\\ge f(x)+\\langle \\nabla f(x),y-x \\rangle+\\frac{\\theta}{2}\\|y-x\\|^2.\\label{scon}\n\\end{equation}\n\n\n\tInstead of the original function $f(x, y)$, at the $k$-th iteration, AGP uses the gradient of a regularized version of the original function, i.e.,\n\t\\begin{equation}\n\t\tf_k (x, y) = f(x, y) + \\tfrac{b_{k}}{2} \\|x\\|^2 -\\tfrac{c_{k}}{2}\\| y\\|^2,\\label{sec:2:1}\n\t\\end{equation}\n\twhere $b_{k} \\ge 0$ and $c_{k}\\ge 0$ are two regularization parameters.\n\tMore specifically, for a given pair $(x_k, y_k) \\in \\mathcal{X} \\times \\mathcal{Y}$, AGP\n\tminimizes a linearized approximation of $f_k(x, y_k)$ plus some regularized terms to update  $x_k$ as follows:\n\t\\begin{align}\n\t\tx_{k+1}&={\\arg\\min}_{x\\in \\mathcal{X}}\\langle  \\nabla _x f_k\\left( x_k,y_k \\right) ,x-x_k \\rangle  +\\tfrac{\\beta_k}{2}\\| x-x_k \\| ^2\\nonumber\\\\\n\t\t&= \\mathcal{P}_\\mathcal{X} \\left( x_k - \\tfrac{1}{\\beta_k}  \\nabla _xf ( x_k,y_k ) - \\tfrac{1}{\\beta_k} b_{k} x_k \\right), \\label{sec:2:2}\n\t\\end{align}\n\twhere  $\\mathcal{P}_\\mathcal{X}$ is the projection operator onto $\\mathcal{X}$ and $\\beta_k > 0$ denotes\n\ta stepsize parameter. Similarly, it updates $y_k$ by maximizing a linearized approximation of $f_k(x_{k+1}, y)$ minus some regularized terms, i.e.,\n\t\\begin{align}\n\t\ty_{k+1}&={\\arg\\max}_{y\\in \\mathcal{Y}}\\langle  \\nabla _yf_k\\left( x_{k+1},y_k \\right),y-y_k \\rangle  -\\tfrac{\\gamma_k}{2}\\| y-y_k \\| ^2\\nonumber\\\\\n\t\t&= \\mathcal{P}_\\mathcal{Y}\\left( y_k+\\tfrac{1}{\\gamma_k} \\nabla _y {f}( x_{k+1},y_k ) - \\tfrac{1}{\\gamma_k} c_k y_k \\right),\\label{sec:2:3}\n\t\\end{align}\n\twhere  $\\mathcal{P}_\\mathcal{Y}$ is the projection operator onto  $\\mathcal{Y}$ and $\\gamma_k >0$ is another stepsize parameter.\n\n\n\t\\begin{algorithm}%[H]\n\t\t\\caption{(An Alternating Gradient Projection (AGP) Algorithm)}\n\t\t\\label{alg:2.1}\n\t\t\\begin{algorithmic}\n\t\t\t\\STATE{\\textbf{Step 1}:Input $x_1,y_1,\\beta_1,\\gamma_1, b_1, c_1$; Set $k=1$.}\n\t\t\t%    \\FOR{$t = 1, ..., T$}\n\t\t\t\\STATE{\\textbf{Step 2}:Calculate $\\beta_k$ and $b_k$, and perform the following update for $x_k$:  \t\n\t\t\t\t\\qquad \\begin{equation}\\label{sec:2:4:agp:upx}\n\t\t\t\t\tx_{k+1}=\\mathcal{P}_\\mathcal{X} \\left( x_k - \\tfrac{1}{\\beta_k}  \\nabla _xf ( x_k,y_k ) - \\tfrac{1}{\\beta_k} b_k x_k \\right).\n\t\t\t\\end{equation}}\t\t\n\t\t\t\\STATE{\\textbf{Step 3}:Calculate $\\gamma_k$ and $c_k$, and perform the following update for $y_k$:\n\t\t\t\t\\qquad\t\\begin{equation}\\label{sec:2:5:agp:upy}\n\t\t\t\t\ty_{k+1}=\\mathcal{P}_\\mathcal{Y}\\left( y_k+\\tfrac{1}{\\gamma_k} \\nabla _y {f}( x_{k+1},y_k ) - \\tfrac{1}{\\gamma_k} c_k y_k \\right).\n\t\t\t\\end{equation}}\n\t\t\t\\STATE{\\textbf{Step 4}:If some stationary condition is satisfied, stop; otherwise, set $k=k+1, $ go to Step 2.}\n\t\t\t%    \\ENDFOR\n\t\t\\end{algorithmic}\n\t\\end{algorithm}\n\n\n\t\\begin{defi}\\label{sec:2:def1:G}\n\t\tAt each iteration of Algorithm \\ref{alg:2.1}, the stationarity gap for problem \\eqref{problem:1} w.r.t. $f(x,y)$ is defined as:\n\t\t\\begin{equation*}\n\t\t\t\\nabla G\\left( x_k,y_k \\right) :=\\left[\\begin{array}{c}\n\t\t\t\t{\\beta_k\\left(x_k-\\operatorname{\\mathcal{P}_\\mathcal{X}}(x_k-\\frac{1}{\\beta_k} \\nabla_{x} f(x_k ,y_k))\\right)} \\\\\n\t\t\t\t\n\t\t\t\t{\\gamma_k\\left(y_k-\\operatorname{\\mathcal{P}_\\mathcal{Y}}(y_k+\\frac{1}{\\gamma_k} \\nabla_{y} f(x_k ,y_k))\\right)}\n\t\t\t\\end{array}\\right].\n\t\t\\end{equation*}\n    \tWe denote $\\nabla {G}_k:= \\nabla {G}(x_k, y_k)$,\n\t\t$(\\nabla G_k)_{x}:= \\beta_k( x_k-\\operatorname{\\mathcal{P}_\\mathcal{X}}( x_k-\\frac{1}{\\beta_k}\\nabla _{x}f( x_k,y_k)))$,\n\t\tand $(\\nabla G_k)_{y}:=\\gamma_k( y_k-\\operatorname{\\mathcal{P}_\\mathcal{Y}}( y_k+ \\frac{1}{\\gamma_k} \\nabla _{y}f( x_k,y_k )) )$.\n\t\t%\t\t\\begin{align*}\n\t\t%\t\t\t(\\nabla G^r)_{x_i}&:= \\beta_i^r\\left( x_i^r-\\operatorname{Prox_{\\mathcal{X}_i}^{\\beta_i^r}}\\left( x_i^r+ \\frac{1}{\\beta_i^r} \\nabla _{x_i}f\\left( x^r,y^r \\right) \\right) \\right),\\\\\n\t\t%\t\t\t(\\nabla G^r)_{y}&:=\\rho^r\\left( y^r-\\operatorname{Prox_{\\mathcal{Y}}^{\\rho^r}}\\left( y^r+ \\frac{1}{\\rho^r} \\nabla _{y}f\\left( x^r,y^r \\right) \\right) \\right).\n\t\t%\t\t\\end{align*}\n\t\\end{defi}\n\t\n\tNote that the widely used metrics for convex-concave minimax problems such as the min-max value or the distance to the set of optimal solutions of the min-max problem are not applicable in nonconvex cases.\n\tFor the latter cases we call $(x_k,y_k)$ an $\\varepsilon$-stationary point of $f(x,y)$ if $ \\|\\nabla G(x_k,y_k)\\|\\leq \\varepsilon$ with $\\nabla G(x_k,y_k)$ being defined as in Definition \\ref{sec:2:def1:G}, which has been widely used as the optimality measure, e.g., \\cite{Lin2020,Lu}. In the absence of constraints, $ \\| \\nabla G(x_k,y_k)\\| \\leq \\varepsilon$ reduces to the standard condition $ \\|\\nabla_{x} f(x_k ,y_k)\\|\\leq \\varepsilon $ and $\\|\\nabla_{y} f(x_k ,y_k)\\|\\leq \\varepsilon$ for unconstrained problems. The vector  $\\nabla G(x_k,y_k)$ also refers to gradient mapping at $(x_k,y_k)$, see \\cite{Nes2013} for the details. \n\n\n\t\\begin{ass}\\label{sec:2:ass1:lip}\n\t\t$f\\left(x,y\\right)$ has Lipschitz continuous gradients, i.e., there exist positive scalars $L_{22}$, $L_{12}$, $L_{11}$, $L_{21}$ such that for any $x,\\bar{x}\\in \\mathcal{X}$, $y, \\bar{y}\\in \\mathcal{Y}$,\n\t\t\\begin{align*}\n\t\t\t\\| \\nabla_{x} f\\left(x, y\\right)-\\nabla_{x} f\\left(\\bar{x}, y\\right)\\| &\\leq L_{11}\\|x-\\bar{x}\\|,\\\\\n\t\t\t\\|\\nabla_{x} f\\left(x, y\\right)-\\nabla_{x} f\\left(x, \\bar{y}\\right)\\| &\\leq L_{21}\\|y-\\bar{y}\\|,\\\\\n\t\t\t\\|\\nabla_{y} f\\left(x, \\bar{y}\\right)-\\nabla_{y} f\\left(x, y\\right)\\| &\\leq L_{22}\\|\\bar{y}-y\\|,\\\\\n\t\t\t\\|\\nabla_{y} f\\left(x, y\\right)-\\nabla_{y} f\\left(\\bar{x}, y\\right)\\| &\\leq L_{12}\\|x-\\bar{x}\\|.\n\t\t\\end{align*}\n\t\tWe denote $L:=\\max\\{L_{11}, L_{22}, L_{12}, L_{21}\\}$.\n\t\\end{ass}\n\n\n\\begin{equation}\\label{subsec:3.1:1}\n\t\t\\beta_k=\\eta, \\gamma_k=\\tfrac{1}{\\rho}, \\ \\mbox{and} \\ b_k=c_k=0\n\t\\end{equation}\n\n\\begin{align}\n\t\tx_{k+1}&=\\mathcal{P}_\\mathcal{X} \\left( x_k - \\tfrac{1}{\\eta}  \\nabla _xf ( x_k,y_k )  \\right),\\label{subsec:3.1:2:x}\\\\\n\t\ty_{k+1}&=\\mathcal{P}_\\mathcal{Y}\\left( y_k+\\rho \\nabla _y {f}( x_{k+1},y_k ) \\right),\\label{subsec:3.1:3:y}\n\t\\end{align}\n\n\tWe are now ready to establish the iteration complexity for the AGP algorithm in the nonconvex-strongly concave setting.\n\tIn particular,\n\tletting $\\nabla G\\left( x_k,y_k \\right)$ be defined as  in Definition \\ref{sec:2:def1:G} and $\\varepsilon>0$ be\n\ta given target accuracy, we provide\n\ta bound on $T(\\varepsilon)$, the first iteration index to achieve an $\\varepsilon$-stationary point, i.e., $ \\| \\nabla G(x_k,y_k)\\| \\leq \\varepsilon$, which is equivalent to\n\t\\begin{equation}\\label{subsec:3.1:4:defT}\n\t\tT(\\varepsilon):=\\min\\{k \\mid \\|\\nabla G(x_k,y_k)\\|\\leq \\varepsilon \\}.\n\t\\end{equation}\n\n\n\\subsection{Strongly Convex-Nonconcave Setting}\\label{subsec4}\nIn this subsection, we analyze the iteration complexity of Algorithm \\ref{alg:2.1} for solving strongly convex-nonconcave minimax optimization problems \\eqref{problem:1}, i.e., $f(x,y)$ is $\\theta$-strongly convex w.r.t. $x$ for any fixed $y\\in \\mathcal{Y}$,\nand nonconcave  w.r.t. $y$ for any given $x\\in \\mathcal{X}$. Under this setting,   $\\forall k\\ge 1$, we set\n\\begin{equation}\n\\beta_k=\\tfrac{1}{\\zeta}, \\gamma_k=\\nu,  b_k=c_k=0,\\label{subsec:4.1:1:para}\n\\end{equation}\nin  Algorithm \\ref{alg:2.1}, and simplify the update for $x_k$ and $y_k$ as follows:\n\\begin{align}\n\tx_{k+1}&=\\mathcal{P}_\\mathcal{X}\\left( x_k- \\zeta \\nabla _x {f}( x_{k},y_k) \\right),\\label{subsec:4.1:2:x}\\\\\n\ty_{k+1}&=\\mathcal{P}_\\mathcal{Y} \\left(y_k + \\tfrac{1}{\\nu}  \\nabla _yf ( x_{k+1},y_k )  \\right)\\label{subsec:4.1:3:y}.\n\\end{align}\n","units":[{"raw_tex":"\tWe consider the following minimax optimization problem:\n\t\\begin{equation}\\label{problem:1}\n\t\t\\min \\limits_{x\\in \\mathcal{X}}\\max \\limits_{y\\in \\mathcal{Y}}f(x,y),\\tag{P}\n\t\\end{equation}\n\twhere $\\mathcal{X}\\subseteq \\mathbb{R}^n$ and $\\mathcal{Y}\\subseteq \\mathbb{R}^m$ are nonempty closed and bounded convex sets, and $f: \\mathcal{X} \\times \\mathcal{Y} \\rightarrow \\mathbb{R}$\n\tis a smooth function. This problem has attracted more  attention due to its wide applications in machine learning, signal processing, and many other research fields in recent years.\n","start":6800,"end":7361},{"raw_tex":"\t{\\bfseries Notation}. For vectors, we use $\\|\\cdot\\|$ to denote  the $l_2$-norm. For a function $f(x,y):\\mathbb{R}^n\\times\\mathbb{R}^m\\rightarrow \\mathbb{R}$, we use $\\nabla_{x} f(x,y)$ (or $\\nabla_{y} f(x, y)$) to denote the partial gradient of $f$ with respect to the first variable (or the second variable) at point $(x, y)$.\n\tLet $\\mathcal{P}_{\\mathcal{X}}$ and $\\mathcal{P}_{\\mathcal{Y}}$ denote projections onto the sets $\\mathcal{X}$ and $\\mathcal{Y}$. Finally, we use the notation $\\mathcal{O} (\\cdot)$ to hide only absolute constants which do not depend on any problem parameter, and $\\tilde{\\mathcal{O}}(\\cdot)$ notation to hide only absolute constants and log factors.\n","start":28706,"end":29387},{"raw_tex":"A continuously differentiable function $f(\\cdot)$ is called $\\theta$-strongly convex if there exists a constant $\\theta> 0$ such\nthat for any $x,y \\in \\mathcal{X}$,\n\\begin{equation}\n  f(y)\\ge f(x)+\\langle \\nabla f(x),y-x \\rangle+\\frac{\\theta}{2}\\|y-x\\|^2.\\label{scon}\n\\end{equation}\n","start":29414,"end":29697},{"raw_tex":"\tInstead of the original function $f(x, y)$, at the $k$-th iteration, AGP uses the gradient of a regularized version of the original function, i.e.,\n\t\\begin{equation}\n\t\tf_k (x, y) = f(x, y) + \\tfrac{b_{k}}{2} \\|x\\|^2 -\\tfrac{c_{k}}{2}\\| y\\|^2,\\label{sec:2:1}\n\t\\end{equation}\n\twhere $b_{k} \\ge 0$ and $c_{k}\\ge 0$ are two regularization parameters.\n\tMore specifically, for a given pair $(x_k, y_k) \\in \\mathcal{X} \\times \\mathcal{Y}$, AGP\n\tminimizes a linearized approximation of $f_k(x, y_k)$ plus some regularized terms to update  $x_k$ as follows:\n\t\\begin{align}\n\t\tx_{k+1}&={\\arg\\min}_{x\\in \\mathcal{X}}\\langle  \\nabla _x f_k\\left( x_k,y_k \\right) ,x-x_k \\rangle  +\\tfrac{\\beta_k}{2}\\| x-x_k \\| ^2\\nonumber\\\\\n\t\t&= \\mathcal{P}_\\mathcal{X} \\left( x_k - \\tfrac{1}{\\beta_k}  \\nabla _xf ( x_k,y_k ) - \\tfrac{1}{\\beta_k} b_{k} x_k \\right), \\label{sec:2:2}\n\t\\end{align}\n\twhere  $\\mathcal{P}_\\mathcal{X}$ is the projection operator onto $\\mathcal{X}$ and $\\beta_k > 0$ denotes\n\ta stepsize parameter. Similarly, it updates $y_k$ by maximizing a linearized approximation of $f_k(x_{k+1}, y)$ minus some regularized terms, i.e.,\n\t\\begin{align}\n\t\ty_{k+1}&={\\arg\\max}_{y\\in \\mathcal{Y}}\\langle  \\nabla _yf_k\\left( x_{k+1},y_k \\right),y-y_k \\rangle  -\\tfrac{\\gamma_k}{2}\\| y-y_k \\| ^2\\nonumber\\\\\n\t\t&= \\mathcal{P}_\\mathcal{Y}\\left( y_k+\\tfrac{1}{\\gamma_k} \\nabla _y {f}( x_{k+1},y_k ) - \\tfrac{1}{\\gamma_k} c_k y_k \\right),\\label{sec:2:3}\n\t\\end{align}\n\twhere  $\\mathcal{P}_\\mathcal{Y}$ is the projection operator onto  $\\mathcal{Y}$ and $\\gamma_k >0$ is another stepsize parameter.\n","start":31108,"end":32677},{"raw_tex":"\t\\begin{algorithm}%[H]\n\t\t\\caption{(An Alternating Gradient Projection (AGP) Algorithm)}\n\t\t\\label{alg:2.1}\n\t\t\\begin{algorithmic}\n\t\t\t\\STATE{\\textbf{Step 1}:Input $x_1,y_1,\\beta_1,\\gamma_1, b_1, c_1$; Set $k=1$.}\n\t\t\t%    \\FOR{$t = 1, ..., T$}\n\t\t\t\\STATE{\\textbf{Step 2}:Calculate $\\beta_k$ and $b_k$, and perform the following update for $x_k$:  \t\n\t\t\t\t\\qquad \\begin{equation}\\label{sec:2:4:agp:upx}\n\t\t\t\t\tx_{k+1}=\\mathcal{P}_\\mathcal{X} \\left( x_k - \\tfrac{1}{\\beta_k}  \\nabla _xf ( x_k,y_k ) - \\tfrac{1}{\\beta_k} b_k x_k \\right).\n\t\t\t\\end{equation}}\t\t\n\t\t\t\\STATE{\\textbf{Step 3}:Calculate $\\gamma_k$ and $c_k$, and perform the following update for $y_k$:\n\t\t\t\t\\qquad\t\\begin{equation}\\label{sec:2:5:agp:upy}\n\t\t\t\t\ty_{k+1}=\\mathcal{P}_\\mathcal{Y}\\left( y_k+\\tfrac{1}{\\gamma_k} \\nabla _y {f}( x_{k+1},y_k ) - \\tfrac{1}{\\gamma_k} c_k y_k \\right).\n\t\t\t\\end{equation}}\n\t\t\t\\STATE{\\textbf{Step 4}:If some stationary condition is satisfied, stop; otherwise, set $k=k+1, $ go to Step 2.}\n\t\t\t%    \\ENDFOR\n\t\t\\end{algorithmic}\n\t\\end{algorithm}\n","start":34199,"end":35221},{"raw_tex":"\t\\begin{defi}\\label{sec:2:def1:G}\n\t\tAt each iteration of Algorithm \\ref{alg:2.1}, the stationarity gap for problem \\eqref{problem:1} w.r.t. $f(x,y)$ is defined as:\n\t\t\\begin{equation*}\n\t\t\t\\nabla G\\left( x_k,y_k \\right) :=\\left[\\begin{array}{c}\n\t\t\t\t{\\beta_k\\left(x_k-\\operatorname{\\mathcal{P}_\\mathcal{X}}(x_k-\\frac{1}{\\beta_k} \\nabla_{x} f(x_k ,y_k))\\right)} \\\\\n\t\t\t\t\n\t\t\t\t{\\gamma_k\\left(y_k-\\operatorname{\\mathcal{P}_\\mathcal{Y}}(y_k+\\frac{1}{\\gamma_k} \\nabla_{y} f(x_k ,y_k))\\right)}\n\t\t\t\\end{array}\\right].\n\t\t\\end{equation*}\n    \tWe denote $\\nabla {G}_k:= \\nabla {G}(x_k, y_k)$,\n\t\t$(\\nabla G_k)_{x}:= \\beta_k( x_k-\\operatorname{\\mathcal{P}_\\mathcal{X}}( x_k-\\frac{1}{\\beta_k}\\nabla _{x}f( x_k,y_k)))$,\n\t\tand $(\\nabla G_k)_{y}:=\\gamma_k( y_k-\\operatorname{\\mathcal{P}_\\mathcal{Y}}( y_k+ \\frac{1}{\\gamma_k} \\nabla _{y}f( x_k,y_k )) )$.\n\t\t%\t\t\\begin{align*}\n\t\t%\t\t\t(\\nabla G^r)_{x_i}&:= \\beta_i^r\\left( x_i^r-\\operatorname{Prox_{\\mathcal{X}_i}^{\\beta_i^r}}\\left( x_i^r+ \\frac{1}{\\beta_i^r} \\nabla _{x_i}f\\left( x^r,y^r \\right) \\right) \\right),\\\\\n\t\t%\t\t\t(\\nabla G^r)_{y}&:=\\rho^r\\left( y^r-\\operatorname{Prox_{\\mathcal{Y}}^{\\rho^r}}\\left( y^r+ \\frac{1}{\\rho^r} \\nabla _{y}f\\left( x^r,y^r \\right) \\right) \\right).\n\t\t%\t\t\\end{align*}\n\t\\end{defi}\n\t\n\tNote that the widely used metrics for convex-concave minimax problems such as the min-max value or the distance to the set of optimal solutions of the min-max problem are not applicable in nonconvex cases.\n\tFor the latter cases we call $(x_k,y_k)$ an $\\varepsilon$-stationary point of $f(x,y)$ if $ \\|\\nabla G(x_k,y_k)\\|\\leq \\varepsilon$ with $\\nabla G(x_k,y_k)$ being defined as in Definition \\ref{sec:2:def1:G}, which has been widely used as the optimality measure, e.g., \\cite{Lin2020,Lu}. In the absence of constraints, $ \\| \\nabla G(x_k,y_k)\\| \\leq \\varepsilon$ reduces to the standard condition $ \\|\\nabla_{x} f(x_k ,y_k)\\|\\leq \\varepsilon $ and $\\|\\nabla_{y} f(x_k ,y_k)\\|\\leq \\varepsilon$ for unconstrained problems. The vector  $\\nabla G(x_k,y_k)$ also refers to gradient mapping at $(x_k,y_k)$, see \\cite{Nes2013} for the details. \n","start":36086,"end":38166},{"raw_tex":"\t\\begin{ass}\\label{sec:2:ass1:lip}\n\t\t$f\\left(x,y\\right)$ has Lipschitz continuous gradients, i.e., there exist positive scalars $L_{22}$, $L_{12}$, $L_{11}$, $L_{21}$ such that for any $x,\\bar{x}\\in \\mathcal{X}$, $y, \\bar{y}\\in \\mathcal{Y}$,\n\t\t\\begin{align*}\n\t\t\t\\| \\nabla_{x} f\\left(x, y\\right)-\\nabla_{x} f\\left(\\bar{x}, y\\right)\\| &\\leq L_{11}\\|x-\\bar{x}\\|,\\\\\n\t\t\t\\|\\nabla_{x} f\\left(x, y\\right)-\\nabla_{x} f\\left(x, \\bar{y}\\right)\\| &\\leq L_{21}\\|y-\\bar{y}\\|,\\\\\n\t\t\t\\|\\nabla_{y} f\\left(x, \\bar{y}\\right)-\\nabla_{y} f\\left(x, y\\right)\\| &\\leq L_{22}\\|\\bar{y}-y\\|,\\\\\n\t\t\t\\|\\nabla_{y} f\\left(x, y\\right)-\\nabla_{y} f\\left(\\bar{x}, y\\right)\\| &\\leq L_{12}\\|x-\\bar{x}\\|.\n\t\t\\end{align*}\n\t\tWe denote $L:=\\max\\{L_{11}, L_{22}, L_{12}, L_{21}\\}$.\n\t\\end{ass}\n","start":39184,"end":39933},{"raw_tex":"\\begin{equation}\\label{subsec:3.1:1}\n\t\t\\beta_k=\\eta, \\gamma_k=\\tfrac{1}{\\rho}, \\ \\mbox{and} \\ b_k=c_k=0\n\t\\end{equation}","start":43836,"end":43955},{"raw_tex":"\\begin{align}\n\t\tx_{k+1}&=\\mathcal{P}_\\mathcal{X} \\left( x_k - \\tfrac{1}{\\eta}  \\nabla _xf ( x_k,y_k )  \\right),\\label{subsec:3.1:2:x}\\\\\n\t\ty_{k+1}&=\\mathcal{P}_\\mathcal{Y}\\left( y_k+\\rho \\nabla _y {f}( x_{k+1},y_k ) \\right),\\label{subsec:3.1:3:y}\n\t\\end{align}","start":44042,"end":44300},{"raw_tex":"\tWe are now ready to establish the iteration complexity for the AGP algorithm in the nonconvex-strongly concave setting.\n\tIn particular,\n\tletting $\\nabla G\\left( x_k,y_k \\right)$ be defined as  in Definition \\ref{sec:2:def1:G} and $\\varepsilon>0$ be\n\ta given target accuracy, we provide\n\ta bound on $T(\\varepsilon)$, the first iteration index to achieve an $\\varepsilon$-stationary point, i.e., $ \\| \\nabla G(x_k,y_k)\\| \\leq \\varepsilon$, which is equivalent to\n\t\\begin{equation}\\label{subsec:3.1:4:defT}\n\t\tT(\\varepsilon):=\\min\\{k \\mid \\|\\nabla G(x_k,y_k)\\|\\leq \\varepsilon \\}.\n\t\\end{equation}\n","start":55574,"end":56168},{"raw_tex":"\\subsection{Strongly Convex-Nonconcave Setting}\\label{subsec4}\nIn this subsection, we analyze the iteration complexity of Algorithm \\ref{alg:2.1} for solving strongly convex-nonconcave minimax optimization problems \\eqref{problem:1}, i.e., $f(x,y)$ is $\\theta$-strongly convex w.r.t. $x$ for any fixed $y\\in \\mathcal{Y}$,\nand nonconcave  w.r.t. $y$ for any given $x\\in \\mathcal{X}$. Under this setting,   $\\forall k\\ge 1$, we set\n\\begin{equation}\n\\beta_k=\\tfrac{1}{\\zeta}, \\gamma_k=\\nu,  b_k=c_k=0,\\label{subsec:4.1:1:para}\n\\end{equation}\nin  Algorithm \\ref{alg:2.1}, and simplify the update for $x_k$ and $y_k$ as follows:\n\\begin{align}\n\tx_{k+1}&=\\mathcal{P}_\\mathcal{X}\\left( x_k- \\zeta \\nabla _x {f}( x_{k},y_k) \\right),\\label{subsec:4.1:2:x}\\\\\n\ty_{k+1}&=\\mathcal{P}_\\mathcal{Y} \\left(y_k + \\tfrac{1}{\\nu}  \\nabla _yf ( x_{k+1},y_k )  \\right)\\label{subsec:4.1:3:y}.\n\\end{align}\n","start":93416,"end":94297}]},"statement":{"raw_tex":"\\begin{thm}\\label{thm:4.1}\n\tSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{\\left(x_{k},y_{k}\\right)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}\n\twith parameter settings in \\eqref{subsec:4.1:1:para}.\n\tIf the relations $\\nu>L_{22}, \\nu>L_{21}^2\\zeta+\\frac{4L_{21}^2}{\\zeta \\theta^2},~\\zeta\\leq \\frac{\\theta}{4L_{11}^2}$ are satisfied, then $\\forall \\varepsilon>0$,  it holds that $$ T\\left( \\varepsilon \\right) \\le\n\t\\tfrac{\\overline{F}-\\hat{F}_1}{\\hat{d}_1\\varepsilon ^2},$$\n\twhere\n\t$\\hat{d}_1:=\\tfrac{\\min \\left\\{\\tfrac{\\nu}{2}-\\tfrac{\\zeta L_{21}^2}{2}-\\tfrac{2 L_{21}^2}{\\zeta \\theta^2},\\tfrac{3\\theta-\\zeta L_{11}^2}{2}+\\tfrac{\\theta-4\\zeta L_{11}^2}{2\\zeta\\theta} \\right\\}}{\\max \\left\\{ \\tfrac{1}{\\zeta ^2}+2L_{12}^{2},2\\nu^2 \\right\\}}%\\overline{\\hat{F}}=\\overline{f}-(\\tfrac{\\theta}{2}-\\tfrac{3}{\\zeta})\\sigma _x^2\n\t$\n\tand $\\overline{F}=\\bar{f}+(\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\sigma_x^2$ with $\\overline{f} := \\max_{(x,y) \\in \\mathcal{X} \\times \\mathcal{Y}} f(x,y)$ and $\\sigma _x= \\max\\{\\|x_1-x_2\\| \\mid \\forall x_1, x_2\\in \\mathcal{X}\\}$.\n\\end{thm}","units":[{"raw_tex":"\\begin{thm}\\label{thm:4.1}\n\tSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{\\left(x_{k},y_{k}\\right)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}\n\twith parameter settings in \\eqref{subsec:4.1:1:para}.\n\tIf the relations $\\nu>L_{22}, \\nu>L_{21}^2\\zeta+\\frac{4L_{21}^2}{\\zeta \\theta^2},~\\zeta\\leq \\frac{\\theta}{4L_{11}^2}$ are satisfied, then $\\forall \\varepsilon>0$,  it holds that $$ T\\left( \\varepsilon \\right) \\le\n\t\\tfrac{\\overline{F}-\\hat{F}_1}{\\hat{d}_1\\varepsilon ^2},$$\n\twhere\n\t$\\hat{d}_1:=\\tfrac{\\min \\left\\{\\tfrac{\\nu}{2}-\\tfrac{\\zeta L_{21}^2}{2}-\\tfrac{2 L_{21}^2}{\\zeta \\theta^2},\\tfrac{3\\theta-\\zeta L_{11}^2}{2}+\\tfrac{\\theta-4\\zeta L_{11}^2}{2\\zeta\\theta} \\right\\}}{\\max \\left\\{ \\tfrac{1}{\\zeta ^2}+2L_{12}^{2},2\\nu^2 \\right\\}}%\\overline{\\hat{F}}=\\overline{f}-(\\tfrac{\\theta}{2}-\\tfrac{3}{\\zeta})\\sigma _x^2\n\t$\n\tand $\\overline{F}=\\bar{f}+(\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\sigma_x^2$ with $\\overline{f} := \\max_{(x,y) \\in \\mathcal{X} \\times \\mathcal{Y}} f(x,y)$ and $\\sigma _x= \\max\\{\\|x_1-x_2\\| \\mid \\forall x_1, x_2\\in \\mathcal{X}\\}$.\n\\end{thm}","start":105682,"end":106802}]},"proof":{"raw_tex":"\\begin{lem}\\label{lem:2.1}\nSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{(x_k,y_k)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}. Then we have\n\\begin{align}\n  f\\left( x_{k+1},y_k \\right) -f\\left( x_k,y_k \\right) &\\le-\\left( \\beta_k -\\tfrac{L_{11}}{2} \\right) \\lVert x_{k+1}-x_k \\rVert ^2-\\frac{b_k}{2}\\left[\\|x_{k+1}\\|^2-\\|x_k\\|^2-\\|x_{k+1}-x_k\\|^2\\right], \\label{lem:2.1:1}\\\\\n  f(x_{k+1},y_{k+1})-f( x_{k+1},y_{k})\n\t\t\t&\\ge\\left( \\gamma_k -\\tfrac{L_{22}}{2} \\right) \\lVert y_{k+1}-y_k \\rVert ^2+\\frac{c_k}{2}\\left[\\|y_{k+1}\\|^2-\\|y_k\\|^2-\\|y_{k+1}-y_k\\|^2\\right].\\label{lem:2.1:2}\n\\end{align}\n\\end{lem}\n\n\\begin{proof}\n  Firstly, by the optimality condition for \\eqref{sec:2:4:agp:upx}, we have\n\t\t\\begin{equation}\\label{lem:2.1:3}\n\t\t\t\\langle  \\nabla _xf\\left( x_k,y_k \\right) +\\beta_k \\left( x_{k+1}-x_k \\right)+b_kx_k ,x_k-x_{k+1} \\rangle  \\ge 0.\n\t\t\\end{equation}\n\t\tBy Assumption \\ref{sec:2:ass1:lip}, the gradient of $f$ is Lipschitz continuous, implying that\n\t\t\\begin{equation}\n\t\t\tf\\left( x_{k+1},y_k \\right) -f\\left( x_k,y_k \\right)\\leq \\langle  \\nabla _xf\\left( x_k,y_k \\right) ,x_{k+1}-x_k \\rangle  +\\tfrac{L_{11}}{2}\\lVert x_{k+1}-x_k \\rVert ^2. \\label{lem:2.1:4}\n\t\t\\end{equation}\n\t\tAdding \\eqref{lem:2.1:3} and \\eqref{lem:2.1:4}, we can easily show that\n\t\t\\begin{align*}\n\t\t\tf\\left( x_{k+1},y_k \\right) -f\\left( x_k,y_k \\right) &\\le -\\left( \\beta_k -\\tfrac{L_{11}}{2} \\right) \\lVert x_{k+1}-x_k \\rVert ^2-b_k\\langle x_k, x_{k+1}-x_k\\rangle\\nonumber\\\\\n            &= -\\left( \\beta_k -\\tfrac{L_{11}}{2} \\right) \\lVert x_{k+1}-x_k \\rVert ^2-\\frac{b_k}{2}\\left[\\|x_{k+1}\\|^2-\\|x_k\\|^2-\\|x_{k+1}-x_k\\|^2\\right].\n\t\t\\end{align*}\nThis completes the proof of \\eqref{lem:2.1:1}. By the optimality condition for \\eqref{sec:2:5:agp:upy}, we have\n\t\t\\begin{equation}\\label{lem:2.1:6}\n\t\t\t\\langle  \\nabla _{y}f( x_{k+1},y_{k})-\\gamma_k( y_{k+1}-y_k)-c_ky_k ,y_k-y_{k+1} \\rangle  \\leq 0.\n\t\t\\end{equation}\n\t\tBy  Assumption \\ref{sec:2:ass1:lip}, the gradient of $f$ is Lipschitz continuous,  implying that\n\t\t\\begin{align}\\label{lem:2.1:7}\n\t\t\tf(x_{k+1},y_{k+1})-f( x_{k+1},y_{k})\n\t\t\t\\ge  \\langle \\nabla _{y}f(x_{k+1},y_{k}),y_{k+1}\n    -y_{k} \\rangle -\\frac{L_{22}}{2}\\| y_{k+1}-y_{k} \\|^2.\n\t\t\\end{align}\n\t\t\tThe result \\eqref{lem:2.1:2} then follows by adding \\eqref{lem:2.1:6} and \\eqref{lem:2.1:7}.\n\n\\end{proof}\n\n\\begin{lem}\\label{lem:3.2:blostr:y}\n\t\tSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{\\left(x_k,y_k\\right)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}\n\t\twith parameter settings in \\eqref{subsec:3.1:1}. If $\\eta > L_{11}$, then we have\n\t\t\\begin{align}\\label{lem:3.2:1}\n\t\t\t&f( x_{k+1},y_{k+1}) -f\\left( x_k,y_k \\right)  \\nonumber \\\\\n\t\t\t\\leq& -\\left(\\tfrac{\\eta}{2}-\\tfrac{L_{12}^{2}\\rho}{2}\\right)\\| x_{k+1}-x_{k} \\| ^2-\\left(\\tfrac{\\mu}{2}-\\tfrac{1}{\\rho}\\right)\\| y_{k+1}-y_k \\|^2\\nonumber \\\\\n\t\t\t&-\\left(\\mu-\\tfrac{1}{2\\rho}-\\tfrac{\\rho L_{22}^2}{2}\\right)\\|y_k-y_{k-1} \\|^2.\n\t\t\\end{align}\n\t\\end{lem}\n\n\\begin{proof}\n\t\tThe optimality condition for $y_k$ in \\eqref{subsec:3.1:3:y} implies that $\\forall y\\in \\mathcal{Y}$ and $\\forall k\\geq 1$,\n\t\t\\begin{equation}\\label{lem:3.2:2}\n\t\t\t\\langle \\nabla _yf(x_{k+1},y_k)-\\frac{1}{\\rho}(y_{k+1}-y_k),y-y_{k+1} \\rangle \\le 0.\n\t\t\\end{equation}\n\t\tBy choosing $y=y_k$ in \\eqref{lem:3.2:2}, we have\n\t\t\\begin{equation}\\label{lem:3.2:3}\n\t\t\t\\langle \\nabla _yf(x_{k+1},y_k)-\\frac{1}{\\rho}(y_{k+1}-y_k),y_k-y_{k+1} \\rangle \\le 0.\n\t\t\\end{equation}\n\t\tOn the other hand, by replacing $k$ with $k-1$ and choosing $y=y_{k+1}$ in \\eqref{lem:3.2:2}, we obtain\n\t\t\\begin{equation}\\label{lem:3.2:4}\n\t\t\t\\langle \\nabla _yf(x_{k},y_{k-1})-\\frac{1}{\\rho}(y_k-y_{k-1}),y_{k+1}-y_k \\rangle \\le 0,\n\t\t\\end{equation}\n\t\twhich, in view of the fact that $f\\left( x,y  \\right)$ is $\\mu $-strongly concave w.r.t. $y$ for any given $x\\in \\mathcal{X}$,\n\t\tthen implies that\n\t\t\\begin{align}\\label{lem:3.2:5}\n\t\t\t&f( x_{k+1},y_{k+1}) -f( x_{k+1},y_k ) \\nonumber\\\\\n\t\t\t\\leq &  \\langle  \\nabla _yf( x_{k+1},y_k ),y_{k+1}-y_k\\rangle  -\\frac{\\mu}{2}\\| y_{k+1}-y_k \\|^2\\  \\nonumber\\\\\n\t\t\t\\leq &\\langle  \\nabla _yf( x_{k+1},y_k ) -\\nabla _yf( x_k,y_{k-1}) ,y_{k+1}-y_k \\rangle   \\nonumber\\\\\n\t\t\t& +\\frac{1}{\\rho}\\langle  y_k-y_{k-1},y_{k+1}-y_k\\rangle  -\\frac{\\mu}{2}\\| y_{k+1}-y_k \\|^2.\n\t\t\\end{align}\n\t\tDenoting $v_{k+1}:=\\left(y_{k+1}-y_k\\right)-\\left(y_k-y_{k-1}\\right)$, we can write the first inner product term in the r.h.s.\n\t\tof \\eqref{lem:3.2:5} as\n\t\t\\begin{align}\\label{lem:3.2:6}\n\t\t\t&\\langle  \\nabla _y f ( x_{k+1},y_k ) -\\nabla _y f ( x_k,y_{k-1}) ,y_{k+1}-y_k \\rangle  \\nonumber\\\\\n\t\t\t=&\\langle  \\nabla _y f ( x_{k+1},y_k ) -\\nabla _y f \\left( x_k,y_{k} \\right) ,y_{k+1}-y_k \\rangle  \\nonumber\\\\\n\t\t\t& + \\langle  \\nabla _y f \\left( x_k,y_{k} \\right) -\\nabla _y f ( x_k,y_{k-1}) ,v_{k+1} \\rangle  \\nonumber\\\\\n\t\t\t&+\\langle  \\nabla _y f \\left( x_k,y_{k}\\right) -\\nabla _y f ( x_k,y_{k-1}) ,y_k-y_{k-1} \\rangle .\n\t\t\\end{align}\n\t\tNext, we estimate the three terms in the right hand side of \\eqref{lem:3.2:6} respectively.\n\t\tBy Assumption \\ref{sec:2:ass1:lip} and the Cauchy-Schwarz inequality, we can bound the first two terms according to\n\t\t\\begin{align}\\label{lem:3.2:7}\n\t\t\t&\\langle  \\nabla _yf( x_{k+1},y_k ) -\\nabla _yf\\left( x_k,y_{k} \\right) ,y_{k+1}-y_k \\rangle  \\nonumber\\\\\n\t\t\t\\leq &\\frac{L_{12}^{2}\\rho}{2}\\| x_{k+1}-x_k \\|^2+\\frac{1}{2\\rho}\\| y_{k+1}-y_k \\|^2,\n\t\t\\end{align}\n\t\tand\n\t\t\\begin{align}\\label{lem:3.2:8}\n\t\t\t&\\langle  \\nabla _yf\\left(  x_k,y_{k} \\right) -\\nabla _yf( x_k,y_{k-1}) ,v_{k+1} \\rangle  \\leq \\frac{\\rho L_{22}^{2}}{2}\\|y_k-y_{k-1} \\|^2+\\frac{1}{2\\rho}\\|v_{k+1} \\|^2.\n\t\t\\end{align}\n\t\tFor the third term, by $\\mu $-strongly-concavity  of $f$ with respect to $y$,\n\t\t\\begin{align}\\label{lem:3.2:9}\n\t\t\t\\quad \\langle  \\nabla _yf\\left( x_k,y_{k} \\right) -\\nabla _yf( x_k,y_{k-1}) ,y_k-y_{k-1} \\rangle  \\le -\\mu \\| y_k-y_{k-1} \\|^2 .\n\t\t\\end{align}\n\t\tMoreover, it can be easily checked that\n\t\t\\begin{align}\\label{lem:3.2:10:3points}\n\t\t\t&\\langle  y_k-y_{k-1},y_{k+1}-y_k \\rangle =\\frac{1}{2}\\| y_k-y_{k-1} \\|^2+\\frac{1}{2}\\| y_{k+1}-y_k \\|^2-\\frac{1}{2}\\| v_{k+1} \\|^2.\n\t\t\\end{align}\n\t\tPlugging \\eqref{lem:3.2:6}-\\eqref{lem:3.2:10:3points} into \\eqref{lem:3.2:5} and rearranging the terms, we conclude that\n\t\t\\begin{align}\\label{lem:3.2:11}\n\t\t\t&f( x_{k+1},y_{k+1}) -f( x_{k+1},y_{k})  \\nonumber \\\\\n\t\t\t\\leq& \\frac{L_{12}^{2}\\rho}{2}\\| x_{k+1}-x_k \\|^2-(\\mu-\\frac{1}{2\\rho}-\\frac{\\rho L_{22}^2}{2})\\| y_k-y_{k-1}\\|^2\\nonumber\\\\\n\t\t\t&-(\\frac{\\mu}{2}-\\frac{1}{\\rho})\\|y_{k+1}-y_k \\|^2.\n\t\t\\end{align}\nBy setting $\\beta_k=\\eta$, $b_k=0$ in \\eqref{lem:2.1:1} of Lemma \\ref{lem:2.1}  and the assumption $\\eta > L_{11}$, we have\n\t\t\\begin{equation}\\label{lem:3.1:1}\n\t\t\tf(x_{k+1},y_{k})-f(x_{k},y_k)\\le -\\frac{\\eta}{2}\\| x_{k+1}-x_k \\|^2.\n\t\t\\end{equation}\nThe proof is completed by combining \\eqref{lem:3.2:11} with \\eqref{lem:3.1:1}.\n\t\\end{proof}\n\n\\begin{lem}\\label{lem:4.2}\n\tSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{\\left(x_{k},y_{k}\\right)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}\n\twith parameter settings in \\eqref{subsec:4.1:1:para}. If $\\nu > L_{22}$, then we have\n\t\\begin{align}\\label{lem:4.2:1:conclu}\n\t\t&f( x_{k+1},y_{k+1}) -f( x_{k},y_{k})  \\nonumber \\\\\n\t\t\\geq& \\frac{\\nu}{2}\\| y_{k+1}-y_{k} \\| ^2-\\frac{L_{21}^{2}\\zeta}{2}\\| y_{k}-y_{k-1} \\|^2+(\\theta-\\frac{1}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2})\\| x_{k}-x_{k-1}\\|^2\\nonumber\\\\\n\t\t&+ (\\frac{\\theta}{2}-\\frac{1}{\\zeta})\\|x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\n\\end{lem}\n\n\\begin{proof}\n\tSimilar to \\eqref{lem:3.2:2}-\\eqref{lem:3.2:4} in the proof of Lemma \\ref{lem:3.2:blostr:y}, by the optimality condition for $x_k$ in \\eqref{subsec:4.1:2:x} implies that $\\forall x\\in \\mathcal{X}$ and $\\forall k\\geq 1$,\n\t\\begin{align}\n\t\t\\langle \\nabla _xf(x_{k},y_{k})+\\frac{1}{\\zeta}(x_{k+1}-x_{k}),x-x_{k+1} \\rangle &\\ge 0,\\label{lem:4.2:2:opty}\\\\\n\t\t\\langle \\nabla _xf(x_{k},y_{k})+\\frac{1}{\\zeta}(x_{k+1}-x_{k}) ,x_{k}-x_{k+1} \\rangle &\\ge 0,\\label{lem:4.2:3:nopty}\\\\\n\t\t\\langle \\nabla _xf(x_{k-1},y_{k-1})+\\frac{1}{\\zeta}(x_{k}-x_{k-1}),x_{k+1}-x_{k} \\rangle &\\ge 0,\\label{lem:4.2:4:opty:k}\n\t\\end{align}\nwhich, in view of the fact that $f\\left( x,y  \\right)$ is $\\theta $-strongly convex w.r.t. $x$ for any given $y\\in \\mathcal{Y}$,\n\tthen implies that\n\t\\begin{align}\\label{lem:4.2:5}\n\t\t&f( x_{k+1},y_{k}) -f( x_{k},y_{k} ) \\nonumber\\\\\n\t\t\\geq &  \\langle  \\nabla _xf( x_{k},y_{k} ),x_{k+1}-x_{k}\\rangle  +\\frac{\\theta}{2}\\| x_{k+1}-x_{k} \\|^2\\  \\nonumber\\\\\n\t\t\\geq &\\langle  \\nabla _xf( x_{k},y_{k} ) -\\nabla _xf( x_{k-1},y_{k-1}) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t& -\\frac{1}{\\zeta}\\langle  x_{k}-x_{k-1},x_{k+1}-x_{k} \\rangle +\\frac{\\theta}{2}\\| x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\n\tDenoting $m_{k+1}:=\\left(x_{k+1}-x_k\\right)-\\left(x_k-x_{k-1}\\right)$, by Assumption \\ref{sec:2:ass1:lip}, the Cauchy-Schwarz inequality and\n\tthe $\\theta $-strongly convexity  of $f$ w.r.t. $x$, we can estimate the first inner product term in the r.h.s.\n\tof \\eqref{lem:4.2:5} as\n\t\\begin{align}\n\t\t&\\langle  \\nabla _x f ( x_{k},y_{k} ) -\\nabla _x f ( x_{k-1},y_{k-1}) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t=&\\langle  \\nabla _x f ( x_{k},y_{k} ) -\\nabla _x f \\left( x_{k},y_{k-1} \\right) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t& + \\langle  \\nabla _x f \\left( x_{k},y_{k-1} \\right) -\\nabla _x f ( x_{k-1},y_{k-1}) ,m_{k+1} \\rangle  \\nonumber\\\\\n\t\t&+\\langle  \\nabla _x f \\left( x_{k},y_{k-1}\\right) -\\nabla _x f ( x_{k-1},y_{k-1}) ,x_{k}-x_{k-1} \\rangle \\label{lem:4.2:6}\\\\\n\t\\geq &-\\frac{L_{21}^{2}\\zeta}{2}\\| y_{k}-y_{k-1} \\|^2-\\frac{1}{2\\zeta}\\| x_{k+1}-x_k \\|^2-\\frac{\\zeta L_{11}^{2}}{2}\\|x_k-x_{k-1} \\|^2\\nonumber\\\\\n\t&-\\frac{1}{2\\zeta}\\|m_{k+1} \\|^2 +\t\\theta \\| x_k-x_{k-1} \\|^2.\\label{lem:4.2:7}\n\t\\end{align}\n\tMoreover, it can be easily checked that\n\\begin{align}\\label{lem:4.2:8:3points}\n\t\t&\\langle  x_k-x_{k-1},x_{k+1}-x_k \\rangle =\\frac{1}{2}\\| x_k-x_{k-1} \\|^2+\\frac{1}{2}\\| x_{k+1}-x_k \\|^2-\\frac{1}{2}\\| m_{k+1} \\|^2.\n\t\\end{align}\n\tPlugging \\eqref{lem:4.2:7} and \\eqref{lem:4.2:8:3points} into \\eqref{lem:4.2:5} and rearranging the terms, we conclude that\n\t\\begin{align}\\label{lem:4.2:9}\n\tf( x_{k+1},y_{k}) -f( x_{k},y_{k})\t\\geq& -\\frac{L_{21}^{2}\\zeta}{2}\\| y_{k}-y_{k-1} \\|^2+(\\theta-\\frac{1}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2})\\| x_{k}-x_{k-1}\\|^2\\nonumber\\\\\n\t\t&+ (\\frac{\\theta}{2}-\\frac{1}{\\zeta})\\|x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\nBy setting $\\gamma_k=\\nu$, $c_k=0$ in \\eqref{lem:2.1:2} of Lemma \\ref{lem:2.1}  and the assumption $\\nu > L_{22}$, we have\n\t\t\\begin{equation}\\label{lem:4.1:1:conclu}\n\t\t\tf( x_{k+1},y_{k+1}) -f( x_{k+1},y_k )\\ge \\frac{\\nu}{2}\\| y_{k+1}-y_k \\|^2.\n\t\t\\end{equation}\n\tThe proof is completed by combining \\eqref{lem:4.2:9} with \\eqref{lem:4.1:1:conclu}.\n\\end{proof}\n\n\\begin{lem}\\label{lem:4.3}\n\tSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{\\left(x_k,y_k\\right)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}\n\twith parameter settings in \\eqref{subsec:4.1:1:para}. Denote\n\t$$\\hat{f}_{k+1}:=f\\left( x_{k+1},y_{k+1} \\right),\\ \\hat{S}_{k+1}:=-\\frac{2}{\\zeta^2\\theta}\\lVert x_{k+1}-x_k \\rVert^2,$$\n\t$$\\hat{F}_{k+1}:=\\hat{f}_{k+1}+\\hat{S}_{k+1}+ (\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\lVert x_{k+1}-x_{k} \\rVert^2-(\\frac{L_{21}^2\\zeta}{2}+\\frac{2L_{21}^2}{\\theta^2\\zeta})\\|y_{k+1}-y_k\\|^2.$$\n\tIf $\\nu>L_{22}$,\n%\t\\begin{equation}\\label{lem:4.3:1}\n%, \\nu>L_{21}^2\\zeta+\\frac{4L_{21}^2}{\\zeta \\theta^2},~\\zeta\\leq \\frac{\\theta}{4L_{11}^2},\n%\t\\end{equation}\n\tthen $\\forall k \\geq 1$,\n\t\\begin{align}\\label{lem:4.3:2:conclus}\n\t\t\\hat{F}_{k+1}-\\hat{F}_k\\geq& \\left(\\frac{\\nu}{2}-\\frac{\\zeta L_{21}^2}{2}-\\frac{2L_{21}^2}{\\zeta \\theta^2}\\right)\\lVert y_{k+1}-y_k \\rVert ^2\\nonumber\\\\\n\t\t& +\\left(\\frac{3\\theta-\\zeta L_{11}^2}{2}+\\frac{\\theta-4\\zeta L_{11}^2}{2\\zeta\\theta}\\right)\\lVert x_{k+1}-x_k \\rVert ^2.\n\t\\end{align}\n\\end{lem}\n\n\\begin{proof}\nFirst by \\eqref{lem:4.2:3:nopty} and \\eqref{lem:4.2:4:opty:k}, we have\n\t\\begin{align}\\label{lem:4.3:3}\n\t\t\\frac{1}{\\zeta}\\langle m_{k+1}, x_k -x_{k+1} \\rangle\n\t\t\\ge \\langle \\nabla _xf( x_k,y_k ) -\\nabla _xf\\left( x_{k-1},y_{k-1} \\right),x_{k+1}-x_k \\rangle,\n\t\\end{align}\nwhich together with \\eqref{lem:4.2:6} then imply that\n%\t\\eqref{lem:4.3:3} then implies that\n\t\\begin{align}\n\t\t\\frac{1}{\\zeta}\\langle m_{k+1},x_k-x_{k+1}  \\rangle  \\geq&\\langle  \\nabla _xf( x_{k},y_{k} ) -\\nabla _xf\\left( x_{k},y_{k-1} \\right) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t& + \\langle  \\nabla _xf\\left( x_{k},y_{k-1} \\right) -\\nabla _xf\\left( x_{k-1},y_{k-1} \\right) ,m_{k+1} \\rangle  \\nonumber\\\\\n\t\t&+\\langle  \\nabla _xf\\left( x_k,y_{k-1} \\right) -\\nabla _xf\\left( x_{k-1},y_{k-1} \\right) ,x_{k}-x_{k-1} \\rangle \\label{lem:4.3:4}\\\\\n\t\t\\geq& -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_{k} \\|^2-\\frac{\\zeta L_{11}^2}{2}\\|x_{k}-x_{k-1} \\|^2\\nonumber\\\\\n\t\t&-\\frac{1}{2\\zeta}\\| m_{k+1}\\|^2+\\theta\\|x_{k}-x_{k-1} \\|^2,\\label{lem:4.3:5}\n\t\\end{align}\nwhere the second inequality is similar to the proof of \\eqref{lem:4.2:7} except that for the first term in the r.h.s., we use $\\langle  \\nabla _xf( x_{k},y_{k} ) -\\nabla _xf\\left( x_{k},y_{k-1} \\right) ,x_{k+1}-x_{k} \\rangle \\geq -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_k \\|^2$. By using the identity $\\frac{1}{\\zeta}\\langle m_{k+1},x_{k}-x_{k+1} \\rangle  =\\frac{1}{2\\zeta}\\|x_{k}-x_{k-1}\\|^2- \\frac{1}{2\\zeta}\\|x_{k+1}-x_{k}\\|^2-\\frac{1}{2\\zeta}\\| m_{k+1}\\| ^2$, we conclude from \\eqref{lem:4.3:5} that\n\t\\begin{align}\\label{lem:4.3:6}\n\t\t&\\frac{1}{2\\zeta}\\|x_{k}-x_{k-1}\\|^2- \\frac{1}{2\\zeta}\\|x_{k+1}-x_{k}\\|^2-\\frac{1}{2\\zeta}\\| m_{k+1}\\| ^2\\nonumber\\\\\n\t\t\\geq& -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_k \\|^2-\\frac{\\zeta L_{11}^2}{2}\\|x_{k}-x_{k-1} \\|^2\\nonumber\\\\\n\t\t&-\\frac{1}{2\\zeta}\\| m_{k+1}\\|^2+\\theta\\|x_{k}-x_{k-1} \\|^2.\n\t\\end{align}\n\tRearranging the terms of \\eqref{lem:4.3:6}, we have\n\t\\begin{align}\\label{lem:4.3:7}\n\t\t&\\frac{1}{2\\zeta}\\|x_{k}-x_{k-1}\\|^2- \\frac{1}{2\\zeta}\\|x_{k+1}-x_{k}\\|^2\\nonumber\\\\\n\t\t\\geq& -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_{k} \\|^2+\\left(\\theta-\\frac{\\zeta L_{11}^2}{2}\\right)\\|x_{k}-x_{k-1} \\|^2.\n\t\\end{align}\n\tMultiplying $\\frac{4}{\\zeta\\theta}$ on both sides of \\eqref{lem:4.3:7} and using the definition of $\\hat{S}_{k+1}$, we obtain\n\t\\begin{align*}\n\t\t\\hat{S}_{k+1}-\\hat{S}_k \\geq & -\\frac{2L_{21}^{2}}{\\theta^2\\zeta}\\| y_{k}-y_{k-1}\\|^2-\\frac{2}{\\zeta}\\| x_{k+1}-x_k \\|^2+\\left(\\frac{4}{\\zeta}-\\frac{2 L_{11}^2}{\\theta}\\right)\\|x_{k}-x_{k-1} \\|^2.\n\t\\end{align*}\n\tThe proof is completed by \\eqref{lem:4.2:1:conclu} in Lemma \\ref{lem:4.2} and the definition of $\\hat{F}_k$.\n\\end{proof}\n\n\\begin{proof}\n\tBy \\eqref{subsec:4.1:2:x}, we immediately obtain\n\t\\begin{equation}\n\t\t\\|(\\nabla G_k)_{x} \\|\\leq\\frac{1}{\\zeta}\\| x_{k+1}-x_{k} \\|.\\label{thm:4.1:1:gx}\n\t\\end{equation}\n\tOn the other hand, similar to the proof of \\eqref{thm:3.1:2:gy},  by \\eqref{subsec:4.1:3:y} and the triangle inequality, and the nonexpansiveness of the projection operator $\\operatorname{ \\mathcal{P}_\\mathcal{Y}}$, we conclude that\n\t\\begin{align}\\label{thm:4.1:2:gy}\n\t\t\\| (\\nabla G_k)_{y} \\|\\le& \\nu\\| y_{k+1}-y_k \\|+L_{12}\\|x_{k+1}-x_{k} \\|.\n\t\t\t\\end{align}\n%\t\t\\leq&\\nu\\|\\operatorname{ Prox_{\\mathcal{Y}_j}^{\\nu}}\\left( y_j^r+\\frac{1}{\\nu} \\nabla _{y_j}f\\left( x^{r+1} ,w_j^{r+1} \\right) \\right)-\\operatorname{ Prox_{\\mathcal{Y}_j}^{\\nu}}\\left( y_j^r+ \\frac{1}{\\nu} \\nabla _{y_j}f\\left( x^r,y^r \\right) \\right) \\|\\nonumber\\\\\n%\t\t&+\\nu\\| y_j^{r+1}-y_j^r \\|\\nonumber\\\\\n%\t\t\\leq& \\nu\\| y_j^{r+1}-y_j^r \\| +\\|\\nabla _{y_j}f\\left( x^{r+1} ,w_j^{r+1} \\right)-\\nabla _{y_j}f\\left( x^r,y^r \\right) \\|\\nonumber\\\\\n%\t\t\\leq & \\nu\\| y_j^{r+1}-y_j^r \\| +\\|\\nabla _{y_j}f\\left( x^{r+1} ,w_j^{r+1} \\right)-\\nabla _{y_j}f\\left( x^r,w_j^{r+1} \\right) \\|\\nonumber\\\\\n%\t\t& +\\|\\nabla _{y_j}f\\left( x^r ,w_j^{r+1} \\right)-\\nabla _{y_j}f\\left( x^r,y^r \\right) \\|\\nonumber\\\\\n%\t\t\\le& \\nu\\| y_j^{r+1}-y_j^r \\|+L_{12}\\|x^{r+1}-x^r \\|+L_{22}\\|w_j^{r+1}-y^r \\|\\nonumber\\\\\n%\twhere the second inequality holds by the nonexpansiveness of the proximal operator $\\operatorname{ Prox_{\\mathcal{Y}_j}^{\\nu}}$, the last inequality holds since that $\\| y_i^{r+1}-y_i^r \\|\\le\\| y^{r+1}-y^r \\|$ and $\\|w_j^{r+1}-y^r \\|\\le\\| y^{r+1}-y^r \\|$ by the definition of $y_j$ and $w_j^{r+1}$.\n\tBy combining \\eqref{thm:4.1:1:gx} and \\eqref{thm:4.1:2:gy}, and using the Cauchy-Schwarz inequality, we obtain\n\t\\begin{align}\\label{thm:4.1:3:g}\n\t\t\\| \\nabla G_k \\|^2\n\t\t\\leq 2\\nu^2\\| y_{k+1}-y_{k} \\|^2+\\left( \\frac{1}{\\zeta ^2}+2L_{12}^{2} \\right)\\|x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\n\tObserving that $\\hat{d}_1 > 0$. Multiplying both sides of \\eqref{thm:4.1:3:g} by $\\hat{d}_1$, and using \\eqref{lem:4.3:2:conclus} in Lemma \\ref{lem:4.3},\n\twe have\n\t\\begin{align}\\label{thm:4.1:4:d1}\n\t\t\\hat{d}_1\\|\\nabla G_k \\|^2\\le \\hat{F}_{k+1}-\\hat{F}_k.\n\t\\end{align}\n\tSumming up the above inequalities from $k=1$ to $k=T(\\varepsilon)$, we obtain\n\t\\begin{align}\\label{thm:4.1:5}\n\t\t\\sum_{k=1}^{T\\left( \\varepsilon \\right)}{\\hat{d}_1\\|\\nabla G_k \\| ^2}\\le \\hat{F}_{T\\left( \\varepsilon \\right)+1}-\\hat{F}_1.\n\t\\end{align}\n\tNote that by the definition of $\\hat{F}_{k+1}$ in Lemma \\ref{lem:4.3}, we have\n\t\\begin{align*}\n\t\t\\hat{F}_{T\\left( \\varepsilon \\right)+1}&\\le\n\t\t\\hat{f}_{T\\left( \\varepsilon \\right)+1}+(\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\|x_{T\\left( \\varepsilon \\right)+1}-x_{T\\left( \\varepsilon \\right)} \\|^2\\\\\n\t\t&\\le \\bar{f}+(\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\sigma_x^2 = \\bar{F},\n\t\\end{align*}\n\twhere the last inequality follows from the definitions of $\\bar{f}$ and  $\\sigma _x$, and $\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta} \\ge 0 $ due to\n\tthe selection of $\\zeta$.\n\tWe then conclude from \\eqref{thm:4.1:5} that $\\sum_{k=1}^{T\\left( \\varepsilon \\right)}{\\hat{d}_1\\| \\nabla G_k \\|^2}\\le \\bar{F}-\\hat{F}_1$ which, in view of the definition of $T(\\varepsilon)$, implies that $\\varepsilon ^2\\le (\\bar{F}-\\hat{F}_1)/(T( \\varepsilon )\\cdot \\hat{d}_1)$\n\tor equivalently, $T\\left( \\varepsilon \\right) \\le (\\bar{F}-\\hat{F}_1)/(\\hat{d}_1\\varepsilon ^2)$.\n\\end{proof}","units":[{"raw_tex":"\\begin{lem}\\label{lem:2.1}\nSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{(x_k,y_k)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}. Then we have\n\\begin{align}\n  f\\left( x_{k+1},y_k \\right) -f\\left( x_k,y_k \\right) &\\le-\\left( \\beta_k -\\tfrac{L_{11}}{2} \\right) \\lVert x_{k+1}-x_k \\rVert ^2-\\frac{b_k}{2}\\left[\\|x_{k+1}\\|^2-\\|x_k\\|^2-\\|x_{k+1}-x_k\\|^2\\right], \\label{lem:2.1:1}\\\\\n  f(x_{k+1},y_{k+1})-f( x_{k+1},y_{k})\n\t\t\t&\\ge\\left( \\gamma_k -\\tfrac{L_{22}}{2} \\right) \\lVert y_{k+1}-y_k \\rVert ^2+\\frac{c_k}{2}\\left[\\|y_{k+1}\\|^2-\\|y_k\\|^2-\\|y_{k+1}-y_k\\|^2\\right].\\label{lem:2.1:2}\n\\end{align}\n\\end{lem}","start":40128,"end":40751},{"raw_tex":"\\begin{proof}\n  Firstly, by the optimality condition for \\eqref{sec:2:4:agp:upx}, we have\n\t\t\\begin{equation}\\label{lem:2.1:3}\n\t\t\t\\langle  \\nabla _xf\\left( x_k,y_k \\right) +\\beta_k \\left( x_{k+1}-x_k \\right)+b_kx_k ,x_k-x_{k+1} \\rangle  \\ge 0.\n\t\t\\end{equation}\n\t\tBy Assumption \\ref{sec:2:ass1:lip}, the gradient of $f$ is Lipschitz continuous, implying that\n\t\t\\begin{equation}\n\t\t\tf\\left( x_{k+1},y_k \\right) -f\\left( x_k,y_k \\right)\\leq \\langle  \\nabla _xf\\left( x_k,y_k \\right) ,x_{k+1}-x_k \\rangle  +\\tfrac{L_{11}}{2}\\lVert x_{k+1}-x_k \\rVert ^2. \\label{lem:2.1:4}\n\t\t\\end{equation}\n\t\tAdding \\eqref{lem:2.1:3} and \\eqref{lem:2.1:4}, we can easily show that\n\t\t\\begin{align*}\n\t\t\tf\\left( x_{k+1},y_k \\right) -f\\left( x_k,y_k \\right) &\\le -\\left( \\beta_k -\\tfrac{L_{11}}{2} \\right) \\lVert x_{k+1}-x_k \\rVert ^2-b_k\\langle x_k, x_{k+1}-x_k\\rangle\\nonumber\\\\\n            &= -\\left( \\beta_k -\\tfrac{L_{11}}{2} \\right) \\lVert x_{k+1}-x_k \\rVert ^2-\\frac{b_k}{2}\\left[\\|x_{k+1}\\|^2-\\|x_k\\|^2-\\|x_{k+1}-x_k\\|^2\\right].\n\t\t\\end{align*}\nThis completes the proof of \\eqref{lem:2.1:1}. By the optimality condition for \\eqref{sec:2:5:agp:upy}, we have\n\t\t\\begin{equation}\\label{lem:2.1:6}\n\t\t\t\\langle  \\nabla _{y}f( x_{k+1},y_{k})-\\gamma_k( y_{k+1}-y_k)-c_ky_k ,y_k-y_{k+1} \\rangle  \\leq 0.\n\t\t\\end{equation}\n\t\tBy  Assumption \\ref{sec:2:ass1:lip}, the gradient of $f$ is Lipschitz continuous,  implying that\n\t\t\\begin{align}\\label{lem:2.1:7}\n\t\t\tf(x_{k+1},y_{k+1})-f( x_{k+1},y_{k})\n\t\t\t\\ge  \\langle \\nabla _{y}f(x_{k+1},y_{k}),y_{k+1}\n    -y_{k} \\rangle -\\frac{L_{22}}{2}\\| y_{k+1}-y_{k} \\|^2.\n\t\t\\end{align}\n\t\t\tThe result \\eqref{lem:2.1:2} then follows by adding \\eqref{lem:2.1:6} and \\eqref{lem:2.1:7}.\n\n\\end{proof}","start":40753,"end":42448},{"raw_tex":"\\begin{lem}\\label{lem:3.2:blostr:y}\n\t\tSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{\\left(x_k,y_k\\right)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}\n\t\twith parameter settings in \\eqref{subsec:3.1:1}. If $\\eta > L_{11}$, then we have\n\t\t\\begin{align}\\label{lem:3.2:1}\n\t\t\t&f( x_{k+1},y_{k+1}) -f\\left( x_k,y_k \\right)  \\nonumber \\\\\n\t\t\t\\leq& -\\left(\\tfrac{\\eta}{2}-\\tfrac{L_{12}^{2}\\rho}{2}\\right)\\| x_{k+1}-x_{k} \\| ^2-\\left(\\tfrac{\\mu}{2}-\\tfrac{1}{\\rho}\\right)\\| y_{k+1}-y_k \\|^2\\nonumber \\\\\n\t\t\t&-\\left(\\mu-\\tfrac{1}{2\\rho}-\\tfrac{\\rho L_{22}^2}{2}\\right)\\|y_k-y_{k-1} \\|^2.\n\t\t\\end{align}\n\t\\end{lem}","start":46965,"end":47586},{"raw_tex":"\\begin{proof}\n\t\tThe optimality condition for $y_k$ in \\eqref{subsec:3.1:3:y} implies that $\\forall y\\in \\mathcal{Y}$ and $\\forall k\\geq 1$,\n\t\t\\begin{equation}\\label{lem:3.2:2}\n\t\t\t\\langle \\nabla _yf(x_{k+1},y_k)-\\frac{1}{\\rho}(y_{k+1}-y_k),y-y_{k+1} \\rangle \\le 0.\n\t\t\\end{equation}\n\t\tBy choosing $y=y_k$ in \\eqref{lem:3.2:2}, we have\n\t\t\\begin{equation}\\label{lem:3.2:3}\n\t\t\t\\langle \\nabla _yf(x_{k+1},y_k)-\\frac{1}{\\rho}(y_{k+1}-y_k),y_k-y_{k+1} \\rangle \\le 0.\n\t\t\\end{equation}\n\t\tOn the other hand, by replacing $k$ with $k-1$ and choosing $y=y_{k+1}$ in \\eqref{lem:3.2:2}, we obtain\n\t\t\\begin{equation}\\label{lem:3.2:4}\n\t\t\t\\langle \\nabla _yf(x_{k},y_{k-1})-\\frac{1}{\\rho}(y_k-y_{k-1}),y_{k+1}-y_k \\rangle \\le 0,\n\t\t\\end{equation}\n\t\twhich, in view of the fact that $f\\left( x,y  \\right)$ is $\\mu $-strongly concave w.r.t. $y$ for any given $x\\in \\mathcal{X}$,\n\t\tthen implies that\n\t\t\\begin{align}\\label{lem:3.2:5}\n\t\t\t&f( x_{k+1},y_{k+1}) -f( x_{k+1},y_k ) \\nonumber\\\\\n\t\t\t\\leq &  \\langle  \\nabla _yf( x_{k+1},y_k ),y_{k+1}-y_k\\rangle  -\\frac{\\mu}{2}\\| y_{k+1}-y_k \\|^2\\  \\nonumber\\\\\n\t\t\t\\leq &\\langle  \\nabla _yf( x_{k+1},y_k ) -\\nabla _yf( x_k,y_{k-1}) ,y_{k+1}-y_k \\rangle   \\nonumber\\\\\n\t\t\t& +\\frac{1}{\\rho}\\langle  y_k-y_{k-1},y_{k+1}-y_k\\rangle  -\\frac{\\mu}{2}\\| y_{k+1}-y_k \\|^2.\n\t\t\\end{align}\n\t\tDenoting $v_{k+1}:=\\left(y_{k+1}-y_k\\right)-\\left(y_k-y_{k-1}\\right)$, we can write the first inner product term in the r.h.s.\n\t\tof \\eqref{lem:3.2:5} as\n\t\t\\begin{align}\\label{lem:3.2:6}\n\t\t\t&\\langle  \\nabla _y f ( x_{k+1},y_k ) -\\nabla _y f ( x_k,y_{k-1}) ,y_{k+1}-y_k \\rangle  \\nonumber\\\\\n\t\t\t=&\\langle  \\nabla _y f ( x_{k+1},y_k ) -\\nabla _y f \\left( x_k,y_{k} \\right) ,y_{k+1}-y_k \\rangle  \\nonumber\\\\\n\t\t\t& + \\langle  \\nabla _y f \\left( x_k,y_{k} \\right) -\\nabla _y f ( x_k,y_{k-1}) ,v_{k+1} \\rangle  \\nonumber\\\\\n\t\t\t&+\\langle  \\nabla _y f \\left( x_k,y_{k}\\right) -\\nabla _y f ( x_k,y_{k-1}) ,y_k-y_{k-1} \\rangle .\n\t\t\\end{align}\n\t\tNext, we estimate the three terms in the right hand side of \\eqref{lem:3.2:6} respectively.\n\t\tBy Assumption \\ref{sec:2:ass1:lip} and the Cauchy-Schwarz inequality, we can bound the first two terms according to\n\t\t\\begin{align}\\label{lem:3.2:7}\n\t\t\t&\\langle  \\nabla _yf( x_{k+1},y_k ) -\\nabla _yf\\left( x_k,y_{k} \\right) ,y_{k+1}-y_k \\rangle  \\nonumber\\\\\n\t\t\t\\leq &\\frac{L_{12}^{2}\\rho}{2}\\| x_{k+1}-x_k \\|^2+\\frac{1}{2\\rho}\\| y_{k+1}-y_k \\|^2,\n\t\t\\end{align}\n\t\tand\n\t\t\\begin{align}\\label{lem:3.2:8}\n\t\t\t&\\langle  \\nabla _yf\\left(  x_k,y_{k} \\right) -\\nabla _yf( x_k,y_{k-1}) ,v_{k+1} \\rangle  \\leq \\frac{\\rho L_{22}^{2}}{2}\\|y_k-y_{k-1} \\|^2+\\frac{1}{2\\rho}\\|v_{k+1} \\|^2.\n\t\t\\end{align}\n\t\tFor the third term, by $\\mu $-strongly-concavity  of $f$ with respect to $y$,\n\t\t\\begin{align}\\label{lem:3.2:9}\n\t\t\t\\quad \\langle  \\nabla _yf\\left( x_k,y_{k} \\right) -\\nabla _yf( x_k,y_{k-1}) ,y_k-y_{k-1} \\rangle  \\le -\\mu \\| y_k-y_{k-1} \\|^2 .\n\t\t\\end{align}\n\t\tMoreover, it can be easily checked that\n\t\t\\begin{align}\\label{lem:3.2:10:3points}\n\t\t\t&\\langle  y_k-y_{k-1},y_{k+1}-y_k \\rangle =\\frac{1}{2}\\| y_k-y_{k-1} \\|^2+\\frac{1}{2}\\| y_{k+1}-y_k \\|^2-\\frac{1}{2}\\| v_{k+1} \\|^2.\n\t\t\\end{align}\n\t\tPlugging \\eqref{lem:3.2:6}-\\eqref{lem:3.2:10:3points} into \\eqref{lem:3.2:5} and rearranging the terms, we conclude that\n\t\t\\begin{align}\\label{lem:3.2:11}\n\t\t\t&f( x_{k+1},y_{k+1}) -f( x_{k+1},y_{k})  \\nonumber \\\\\n\t\t\t\\leq& \\frac{L_{12}^{2}\\rho}{2}\\| x_{k+1}-x_k \\|^2-(\\mu-\\frac{1}{2\\rho}-\\frac{\\rho L_{22}^2}{2})\\| y_k-y_{k-1}\\|^2\\nonumber\\\\\n\t\t\t&-(\\frac{\\mu}{2}-\\frac{1}{\\rho})\\|y_{k+1}-y_k \\|^2.\n\t\t\\end{align}\nBy setting $\\beta_k=\\eta$, $b_k=0$ in \\eqref{lem:2.1:1} of Lemma \\ref{lem:2.1}  and the assumption $\\eta > L_{11}$, we have\n\t\t\\begin{equation}\\label{lem:3.1:1}\n\t\t\tf(x_{k+1},y_{k})-f(x_{k},y_k)\\le -\\frac{\\eta}{2}\\| x_{k+1}-x_k \\|^2.\n\t\t\\end{equation}\nThe proof is completed by combining \\eqref{lem:3.2:11} with \\eqref{lem:3.1:1}.\n\t\\end{proof}","start":47590,"end":51440},{"raw_tex":"\\begin{lem}\\label{lem:4.2}\n\tSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{\\left(x_{k},y_{k}\\right)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}\n\twith parameter settings in \\eqref{subsec:4.1:1:para}. If $\\nu > L_{22}$, then we have\n\t\\begin{align}\\label{lem:4.2:1:conclu}\n\t\t&f( x_{k+1},y_{k+1}) -f( x_{k},y_{k})  \\nonumber \\\\\n\t\t\\geq& \\frac{\\nu}{2}\\| y_{k+1}-y_{k} \\| ^2-\\frac{L_{21}^{2}\\zeta}{2}\\| y_{k}-y_{k-1} \\|^2+(\\theta-\\frac{1}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2})\\| x_{k}-x_{k-1}\\|^2\\nonumber\\\\\n\t\t&+ (\\frac{\\theta}{2}-\\frac{1}{\\zeta})\\|x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\n\\end{lem}","start":97011,"end":97613},{"raw_tex":"\\begin{proof}\n\tSimilar to \\eqref{lem:3.2:2}-\\eqref{lem:3.2:4} in the proof of Lemma \\ref{lem:3.2:blostr:y}, by the optimality condition for $x_k$ in \\eqref{subsec:4.1:2:x} implies that $\\forall x\\in \\mathcal{X}$ and $\\forall k\\geq 1$,\n\t\\begin{align}\n\t\t\\langle \\nabla _xf(x_{k},y_{k})+\\frac{1}{\\zeta}(x_{k+1}-x_{k}),x-x_{k+1} \\rangle &\\ge 0,\\label{lem:4.2:2:opty}\\\\\n\t\t\\langle \\nabla _xf(x_{k},y_{k})+\\frac{1}{\\zeta}(x_{k+1}-x_{k}) ,x_{k}-x_{k+1} \\rangle &\\ge 0,\\label{lem:4.2:3:nopty}\\\\\n\t\t\\langle \\nabla _xf(x_{k-1},y_{k-1})+\\frac{1}{\\zeta}(x_{k}-x_{k-1}),x_{k+1}-x_{k} \\rangle &\\ge 0,\\label{lem:4.2:4:opty:k}\n\t\\end{align}\nwhich, in view of the fact that $f\\left( x,y  \\right)$ is $\\theta $-strongly convex w.r.t. $x$ for any given $y\\in \\mathcal{Y}$,\n\tthen implies that\n\t\\begin{align}\\label{lem:4.2:5}\n\t\t&f( x_{k+1},y_{k}) -f( x_{k},y_{k} ) \\nonumber\\\\\n\t\t\\geq &  \\langle  \\nabla _xf( x_{k},y_{k} ),x_{k+1}-x_{k}\\rangle  +\\frac{\\theta}{2}\\| x_{k+1}-x_{k} \\|^2\\  \\nonumber\\\\\n\t\t\\geq &\\langle  \\nabla _xf( x_{k},y_{k} ) -\\nabla _xf( x_{k-1},y_{k-1}) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t& -\\frac{1}{\\zeta}\\langle  x_{k}-x_{k-1},x_{k+1}-x_{k} \\rangle +\\frac{\\theta}{2}\\| x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\n\tDenoting $m_{k+1}:=\\left(x_{k+1}-x_k\\right)-\\left(x_k-x_{k-1}\\right)$, by Assumption \\ref{sec:2:ass1:lip}, the Cauchy-Schwarz inequality and\n\tthe $\\theta $-strongly convexity  of $f$ w.r.t. $x$, we can estimate the first inner product term in the r.h.s.\n\tof \\eqref{lem:4.2:5} as\n\t\\begin{align}\n\t\t&\\langle  \\nabla _x f ( x_{k},y_{k} ) -\\nabla _x f ( x_{k-1},y_{k-1}) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t=&\\langle  \\nabla _x f ( x_{k},y_{k} ) -\\nabla _x f \\left( x_{k},y_{k-1} \\right) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t& + \\langle  \\nabla _x f \\left( x_{k},y_{k-1} \\right) -\\nabla _x f ( x_{k-1},y_{k-1}) ,m_{k+1} \\rangle  \\nonumber\\\\\n\t\t&+\\langle  \\nabla _x f \\left( x_{k},y_{k-1}\\right) -\\nabla _x f ( x_{k-1},y_{k-1}) ,x_{k}-x_{k-1} \\rangle \\label{lem:4.2:6}\\\\\n\t\\geq &-\\frac{L_{21}^{2}\\zeta}{2}\\| y_{k}-y_{k-1} \\|^2-\\frac{1}{2\\zeta}\\| x_{k+1}-x_k \\|^2-\\frac{\\zeta L_{11}^{2}}{2}\\|x_k-x_{k-1} \\|^2\\nonumber\\\\\n\t&-\\frac{1}{2\\zeta}\\|m_{k+1} \\|^2 +\t\\theta \\| x_k-x_{k-1} \\|^2.\\label{lem:4.2:7}\n\t\\end{align}\n\tMoreover, it can be easily checked that\n\\begin{align}\\label{lem:4.2:8:3points}\n\t\t&\\langle  x_k-x_{k-1},x_{k+1}-x_k \\rangle =\\frac{1}{2}\\| x_k-x_{k-1} \\|^2+\\frac{1}{2}\\| x_{k+1}-x_k \\|^2-\\frac{1}{2}\\| m_{k+1} \\|^2.\n\t\\end{align}\n\tPlugging \\eqref{lem:4.2:7} and \\eqref{lem:4.2:8:3points} into \\eqref{lem:4.2:5} and rearranging the terms, we conclude that\n\t\\begin{align}\\label{lem:4.2:9}\n\tf( x_{k+1},y_{k}) -f( x_{k},y_{k})\t\\geq& -\\frac{L_{21}^{2}\\zeta}{2}\\| y_{k}-y_{k-1} \\|^2+(\\theta-\\frac{1}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2})\\| x_{k}-x_{k-1}\\|^2\\nonumber\\\\\n\t\t&+ (\\frac{\\theta}{2}-\\frac{1}{\\zeta})\\|x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\nBy setting $\\gamma_k=\\nu$, $c_k=0$ in \\eqref{lem:2.1:2} of Lemma \\ref{lem:2.1}  and the assumption $\\nu > L_{22}$, we have\n\t\t\\begin{equation}\\label{lem:4.1:1:conclu}\n\t\t\tf( x_{k+1},y_{k+1}) -f( x_{k+1},y_k )\\ge \\frac{\\nu}{2}\\| y_{k+1}-y_k \\|^2.\n\t\t\\end{equation}\n\tThe proof is completed by combining \\eqref{lem:4.2:9} with \\eqref{lem:4.1:1:conclu}.\n\\end{proof}","start":97615,"end":100813},{"raw_tex":"\\begin{lem}\\label{lem:4.3}\n\tSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{\\left(x_k,y_k\\right)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}\n\twith parameter settings in \\eqref{subsec:4.1:1:para}. Denote\n\t$$\\hat{f}_{k+1}:=f\\left( x_{k+1},y_{k+1} \\right),\\ \\hat{S}_{k+1}:=-\\frac{2}{\\zeta^2\\theta}\\lVert x_{k+1}-x_k \\rVert^2,$$\n\t$$\\hat{F}_{k+1}:=\\hat{f}_{k+1}+\\hat{S}_{k+1}+ (\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\lVert x_{k+1}-x_{k} \\rVert^2-(\\frac{L_{21}^2\\zeta}{2}+\\frac{2L_{21}^2}{\\theta^2\\zeta})\\|y_{k+1}-y_k\\|^2.$$\n\tIf $\\nu>L_{22}$,\n%\t\\begin{equation}\\label{lem:4.3:1}\n%, \\nu>L_{21}^2\\zeta+\\frac{4L_{21}^2}{\\zeta \\theta^2},~\\zeta\\leq \\frac{\\theta}{4L_{11}^2},\n%\t\\end{equation}\n\tthen $\\forall k \\geq 1$,\n\t\\begin{align}\\label{lem:4.3:2:conclus}\n\t\t\\hat{F}_{k+1}-\\hat{F}_k\\geq& \\left(\\frac{\\nu}{2}-\\frac{\\zeta L_{21}^2}{2}-\\frac{2L_{21}^2}{\\zeta \\theta^2}\\right)\\lVert y_{k+1}-y_k \\rVert ^2\\nonumber\\\\\n\t\t& +\\left(\\frac{3\\theta-\\zeta L_{11}^2}{2}+\\frac{\\theta-4\\zeta L_{11}^2}{2\\zeta\\theta}\\right)\\lVert x_{k+1}-x_k \\rVert ^2.\n\t\\end{align}\n\\end{lem}","start":101244,"end":102349},{"raw_tex":"\\begin{proof}\nFirst by \\eqref{lem:4.2:3:nopty} and \\eqref{lem:4.2:4:opty:k}, we have\n\t\\begin{align}\\label{lem:4.3:3}\n\t\t\\frac{1}{\\zeta}\\langle m_{k+1}, x_k -x_{k+1} \\rangle\n\t\t\\ge \\langle \\nabla _xf( x_k,y_k ) -\\nabla _xf\\left( x_{k-1},y_{k-1} \\right),x_{k+1}-x_k \\rangle,\n\t\\end{align}\nwhich together with \\eqref{lem:4.2:6} then imply that\n%\t\\eqref{lem:4.3:3} then implies that\n\t\\begin{align}\n\t\t\\frac{1}{\\zeta}\\langle m_{k+1},x_k-x_{k+1}  \\rangle  \\geq&\\langle  \\nabla _xf( x_{k},y_{k} ) -\\nabla _xf\\left( x_{k},y_{k-1} \\right) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t& + \\langle  \\nabla _xf\\left( x_{k},y_{k-1} \\right) -\\nabla _xf\\left( x_{k-1},y_{k-1} \\right) ,m_{k+1} \\rangle  \\nonumber\\\\\n\t\t&+\\langle  \\nabla _xf\\left( x_k,y_{k-1} \\right) -\\nabla _xf\\left( x_{k-1},y_{k-1} \\right) ,x_{k}-x_{k-1} \\rangle \\label{lem:4.3:4}\\\\\n\t\t\\geq& -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_{k} \\|^2-\\frac{\\zeta L_{11}^2}{2}\\|x_{k}-x_{k-1} \\|^2\\nonumber\\\\\n\t\t&-\\frac{1}{2\\zeta}\\| m_{k+1}\\|^2+\\theta\\|x_{k}-x_{k-1} \\|^2,\\label{lem:4.3:5}\n\t\\end{align}\nwhere the second inequality is similar to the proof of \\eqref{lem:4.2:7} except that for the first term in the r.h.s., we use $\\langle  \\nabla _xf( x_{k},y_{k} ) -\\nabla _xf\\left( x_{k},y_{k-1} \\right) ,x_{k+1}-x_{k} \\rangle \\geq -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_k \\|^2$. By using the identity $\\frac{1}{\\zeta}\\langle m_{k+1},x_{k}-x_{k+1} \\rangle  =\\frac{1}{2\\zeta}\\|x_{k}-x_{k-1}\\|^2- \\frac{1}{2\\zeta}\\|x_{k+1}-x_{k}\\|^2-\\frac{1}{2\\zeta}\\| m_{k+1}\\| ^2$, we conclude from \\eqref{lem:4.3:5} that\n\t\\begin{align}\\label{lem:4.3:6}\n\t\t&\\frac{1}{2\\zeta}\\|x_{k}-x_{k-1}\\|^2- \\frac{1}{2\\zeta}\\|x_{k+1}-x_{k}\\|^2-\\frac{1}{2\\zeta}\\| m_{k+1}\\| ^2\\nonumber\\\\\n\t\t\\geq& -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_k \\|^2-\\frac{\\zeta L_{11}^2}{2}\\|x_{k}-x_{k-1} \\|^2\\nonumber\\\\\n\t\t&-\\frac{1}{2\\zeta}\\| m_{k+1}\\|^2+\\theta\\|x_{k}-x_{k-1} \\|^2.\n\t\\end{align}\n\tRearranging the terms of \\eqref{lem:4.3:6}, we have\n\t\\begin{align}\\label{lem:4.3:7}\n\t\t&\\frac{1}{2\\zeta}\\|x_{k}-x_{k-1}\\|^2- \\frac{1}{2\\zeta}\\|x_{k+1}-x_{k}\\|^2\\nonumber\\\\\n\t\t\\geq& -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_{k} \\|^2+\\left(\\theta-\\frac{\\zeta L_{11}^2}{2}\\right)\\|x_{k}-x_{k-1} \\|^2.\n\t\\end{align}\n\tMultiplying $\\frac{4}{\\zeta\\theta}$ on both sides of \\eqref{lem:4.3:7} and using the definition of $\\hat{S}_{k+1}$, we obtain\n\t\\begin{align*}\n\t\t\\hat{S}_{k+1}-\\hat{S}_k \\geq & -\\frac{2L_{21}^{2}}{\\theta^2\\zeta}\\| y_{k}-y_{k-1}\\|^2-\\frac{2}{\\zeta}\\| x_{k+1}-x_k \\|^2+\\left(\\frac{4}{\\zeta}-\\frac{2 L_{11}^2}{\\theta}\\right)\\|x_{k}-x_{k-1} \\|^2.\n\t\\end{align*}\n\tThe proof is completed by \\eqref{lem:4.2:1:conclu} in Lemma \\ref{lem:4.2} and the definition of $\\hat{F}_k$.\n\\end{proof}","start":102351,"end":105155},{"raw_tex":"\\begin{proof}\n\tBy \\eqref{subsec:4.1:2:x}, we immediately obtain\n\t\\begin{equation}\n\t\t\\|(\\nabla G_k)_{x} \\|\\leq\\frac{1}{\\zeta}\\| x_{k+1}-x_{k} \\|.\\label{thm:4.1:1:gx}\n\t\\end{equation}\n\tOn the other hand, similar to the proof of \\eqref{thm:3.1:2:gy},  by \\eqref{subsec:4.1:3:y} and the triangle inequality, and the nonexpansiveness of the projection operator $\\operatorname{ \\mathcal{P}_\\mathcal{Y}}$, we conclude that\n\t\\begin{align}\\label{thm:4.1:2:gy}\n\t\t\\| (\\nabla G_k)_{y} \\|\\le& \\nu\\| y_{k+1}-y_k \\|+L_{12}\\|x_{k+1}-x_{k} \\|.\n\t\t\t\\end{align}\n%\t\t\\leq&\\nu\\|\\operatorname{ Prox_{\\mathcal{Y}_j}^{\\nu}}\\left( y_j^r+\\frac{1}{\\nu} \\nabla _{y_j}f\\left( x^{r+1} ,w_j^{r+1} \\right) \\right)-\\operatorname{ Prox_{\\mathcal{Y}_j}^{\\nu}}\\left( y_j^r+ \\frac{1}{\\nu} \\nabla _{y_j}f\\left( x^r,y^r \\right) \\right) \\|\\nonumber\\\\\n%\t\t&+\\nu\\| y_j^{r+1}-y_j^r \\|\\nonumber\\\\\n%\t\t\\leq& \\nu\\| y_j^{r+1}-y_j^r \\| +\\|\\nabla _{y_j}f\\left( x^{r+1} ,w_j^{r+1} \\right)-\\nabla _{y_j}f\\left( x^r,y^r \\right) \\|\\nonumber\\\\\n%\t\t\\leq & \\nu\\| y_j^{r+1}-y_j^r \\| +\\|\\nabla _{y_j}f\\left( x^{r+1} ,w_j^{r+1} \\right)-\\nabla _{y_j}f\\left( x^r,w_j^{r+1} \\right) \\|\\nonumber\\\\\n%\t\t& +\\|\\nabla _{y_j}f\\left( x^r ,w_j^{r+1} \\right)-\\nabla _{y_j}f\\left( x^r,y^r \\right) \\|\\nonumber\\\\\n%\t\t\\le& \\nu\\| y_j^{r+1}-y_j^r \\|+L_{12}\\|x^{r+1}-x^r \\|+L_{22}\\|w_j^{r+1}-y^r \\|\\nonumber\\\\\n%\twhere the second inequality holds by the nonexpansiveness of the proximal operator $\\operatorname{ Prox_{\\mathcal{Y}_j}^{\\nu}}$, the last inequality holds since that $\\| y_i^{r+1}-y_i^r \\|\\le\\| y^{r+1}-y^r \\|$ and $\\|w_j^{r+1}-y^r \\|\\le\\| y^{r+1}-y^r \\|$ by the definition of $y_j$ and $w_j^{r+1}$.\n\tBy combining \\eqref{thm:4.1:1:gx} and \\eqref{thm:4.1:2:gy}, and using the Cauchy-Schwarz inequality, we obtain\n\t\\begin{align}\\label{thm:4.1:3:g}\n\t\t\\| \\nabla G_k \\|^2\n\t\t\\leq 2\\nu^2\\| y_{k+1}-y_{k} \\|^2+\\left( \\frac{1}{\\zeta ^2}+2L_{12}^{2} \\right)\\|x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\n\tObserving that $\\hat{d}_1 > 0$. Multiplying both sides of \\eqref{thm:4.1:3:g} by $\\hat{d}_1$, and using \\eqref{lem:4.3:2:conclus} in Lemma \\ref{lem:4.3},\n\twe have\n\t\\begin{align}\\label{thm:4.1:4:d1}\n\t\t\\hat{d}_1\\|\\nabla G_k \\|^2\\le \\hat{F}_{k+1}-\\hat{F}_k.\n\t\\end{align}\n\tSumming up the above inequalities from $k=1$ to $k=T(\\varepsilon)$, we obtain\n\t\\begin{align}\\label{thm:4.1:5}\n\t\t\\sum_{k=1}^{T\\left( \\varepsilon \\right)}{\\hat{d}_1\\|\\nabla G_k \\| ^2}\\le \\hat{F}_{T\\left( \\varepsilon \\right)+1}-\\hat{F}_1.\n\t\\end{align}\n\tNote that by the definition of $\\hat{F}_{k+1}$ in Lemma \\ref{lem:4.3}, we have\n\t\\begin{align*}\n\t\t\\hat{F}_{T\\left( \\varepsilon \\right)+1}&\\le\n\t\t\\hat{f}_{T\\left( \\varepsilon \\right)+1}+(\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\|x_{T\\left( \\varepsilon \\right)+1}-x_{T\\left( \\varepsilon \\right)} \\|^2\\\\\n\t\t&\\le \\bar{f}+(\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\sigma_x^2 = \\bar{F},\n\t\\end{align*}\n\twhere the last inequality follows from the definitions of $\\bar{f}$ and  $\\sigma _x$, and $\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta} \\ge 0 $ due to\n\tthe selection of $\\zeta$.\n\tWe then conclude from \\eqref{thm:4.1:5} that $\\sum_{k=1}^{T\\left( \\varepsilon \\right)}{\\hat{d}_1\\| \\nabla G_k \\|^2}\\le \\bar{F}-\\hat{F}_1$ which, in view of the definition of $T(\\varepsilon)$, implies that $\\varepsilon ^2\\le (\\bar{F}-\\hat{F}_1)/(T( \\varepsilon )\\cdot \\hat{d}_1)$\n\tor equivalently, $T\\left( \\varepsilon \\right) \\le (\\bar{F}-\\hat{F}_1)/(\\hat{d}_1\\varepsilon ^2)$.\n\\end{proof}","start":106804,"end":110287}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["problem:1"],"declared_external":false,"source_only":false,"title":"极小极大问题与可行域","source_units":[{"raw_tex":"\tWe consider the following minimax optimization problem:\n\t\\begin{equation}\\label{problem:1}\n\t\t\\min \\limits_{x\\in \\mathcal{X}}\\max \\limits_{y\\in \\mathcal{Y}}f(x,y),\\tag{P}\n\t\\end{equation}\n\twhere $\\mathcal{X}\\subseteq \\mathbb{R}^n$ and $\\mathcal{Y}\\subseteq \\mathbb{R}^m$ are nonempty closed and bounded convex sets, and $f: \\mathcal{X} \\times \\mathcal{Y} \\rightarrow \\mathbb{R}$\n\tis a smooth function. This problem has attracted more  attention due to its wide applications in machine learning, signal processing, and many other research fields in recent years.\n","start":6800,"end":7361}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"范数、梯度与投影记号","source_units":[{"raw_tex":"\t{\\bfseries Notation}. For vectors, we use $\\|\\cdot\\|$ to denote  the $l_2$-norm. For a function $f(x,y):\\mathbb{R}^n\\times\\mathbb{R}^m\\rightarrow \\mathbb{R}$, we use $\\nabla_{x} f(x,y)$ (or $\\nabla_{y} f(x, y)$) to denote the partial gradient of $f$ with respect to the first variable (or the second variable) at point $(x, y)$.\n\tLet $\\mathcal{P}_{\\mathcal{X}}$ and $\\mathcal{P}_{\\mathcal{Y}}$ denote projections onto the sets $\\mathcal{X}$ and $\\mathcal{Y}$. Finally, we use the notation $\\mathcal{O} (\\cdot)$ to hide only absolute constants which do not depend on any problem parameter, and $\\tilde{\\mathcal{O}}(\\cdot)$ notation to hide only absolute constants and log factors.\n","start":28706,"end":29387}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":["scon"],"declared_external":false,"source_only":false,"title":"强凸性定义","source_units":[{"raw_tex":"A continuously differentiable function $f(\\cdot)$ is called $\\theta$-strongly convex if there exists a constant $\\theta> 0$ such\nthat for any $x,y \\in \\mathcal{X}$,\n\\begin{equation}\n  f(y)\\ge f(x)+\\langle \\nabla f(x),y-x \\rangle+\\frac{\\theta}{2}\\|y-x\\|^2.\\label{scon}\n\\end{equation}\n","start":29414,"end":29697}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["sec:2:1"],"declared_external":false,"source_only":false,"title":"正则化目标","source_units":[{"raw_tex":"\tInstead of the original function $f(x, y)$, at the $k$-th iteration, AGP uses the gradient of a regularized version of the original function, i.e.,\n\t\\begin{equation}\n\t\tf_k (x, y) = f(x, y) + \\tfrac{b_{k}}{2} \\|x\\|^2 -\\tfrac{c_{k}}{2}\\| y\\|^2,\\label{sec:2:1}\n\t\\end{equation}\n\twhere $b_{k} \\ge 0$ and $c_{k}\\ge 0$ are two regularization parameters.\n","start":31108,"end":31456}]},{"id":"S005","semantic_type":"algorithm","kind":"algorithm","labels":["sec:2:2"],"declared_external":false,"source_only":false,"title":"原变量投影子问题","source_units":[{"raw_tex":"\tMore specifically, for a given pair $(x_k, y_k) \\in \\mathcal{X} \\times \\mathcal{Y}$, AGP\n\tminimizes a linearized approximation of $f_k(x, y_k)$ plus some regularized terms to update  $x_k$ as follows:\n\t\\begin{align}\n\t\tx_{k+1}&={\\arg\\min}_{x\\in \\mathcal{X}}\\langle  \\nabla _x f_k\\left( x_k,y_k \\right) ,x-x_k \\rangle  +\\tfrac{\\beta_k}{2}\\| x-x_k \\| ^2\\nonumber\\\\\n\t\t&= \\mathcal{P}_\\mathcal{X} \\left( x_k - \\tfrac{1}{\\beta_k}  \\nabla _xf ( x_k,y_k ) - \\tfrac{1}{\\beta_k} b_{k} x_k \\right), \\label{sec:2:2}\n\t\\end{align}\n\twhere  $\\mathcal{P}_\\mathcal{X}$ is the projection operator onto $\\mathcal{X}$ and $\\beta_k > 0$ denotes\n","start":31456,"end":32079}]},{"id":"S006","semantic_type":"algorithm","kind":"algorithm","labels":["sec:2:3"],"declared_external":false,"source_only":false,"title":"对偶变量投影子问题","source_units":[{"raw_tex":"\ta stepsize parameter. Similarly, it updates $y_k$ by maximizing a linearized approximation of $f_k(x_{k+1}, y)$ minus some regularized terms, i.e.,\n\t\\begin{align}\n\t\ty_{k+1}&={\\arg\\max}_{y\\in \\mathcal{Y}}\\langle  \\nabla _yf_k\\left( x_{k+1},y_k \\right),y-y_k \\rangle  -\\tfrac{\\gamma_k}{2}\\| y-y_k \\| ^2\\nonumber\\\\\n\t\t&= \\mathcal{P}_\\mathcal{Y}\\left( y_k+\\tfrac{1}{\\gamma_k} \\nabla _y {f}( x_{k+1},y_k ) - \\tfrac{1}{\\gamma_k} c_k y_k \\right),\\label{sec:2:3}\n\t\\end{align}\n\twhere  $\\mathcal{P}_\\mathcal{Y}$ is the projection operator onto  $\\mathcal{Y}$ and $\\gamma_k >0$ is another stepsize parameter.\n","start":32079,"end":32677}]},{"id":"S007","semantic_type":"algorithm","kind":"algorithm","labels":["alg:2.1","sec:2:4:agp:upx","sec:2:5:agp:upy"],"declared_external":false,"source_only":false,"title":"完整 AGP 算法","source_units":[{"raw_tex":"\t\\begin{algorithm}%[H]\n\t\t\\caption{(An Alternating Gradient Projection (AGP) Algorithm)}\n\t\t\\label{alg:2.1}\n\t\t\\begin{algorithmic}\n\t\t\t\\STATE{\\textbf{Step 1}:Input $x_1,y_1,\\beta_1,\\gamma_1, b_1, c_1$; Set $k=1$.}\n\t\t\t%    \\FOR{$t = 1, ..., T$}\n\t\t\t\\STATE{\\textbf{Step 2}:Calculate $\\beta_k$ and $b_k$, and perform the following update for $x_k$:  \t\n\t\t\t\t\\qquad \\begin{equation}\\label{sec:2:4:agp:upx}\n\t\t\t\t\tx_{k+1}=\\mathcal{P}_\\mathcal{X} \\left( x_k - \\tfrac{1}{\\beta_k}  \\nabla _xf ( x_k,y_k ) - \\tfrac{1}{\\beta_k} b_k x_k \\right).\n\t\t\t\\end{equation}}\t\t\n\t\t\t\\STATE{\\textbf{Step 3}:Calculate $\\gamma_k$ and $c_k$, and perform the following update for $y_k$:\n\t\t\t\t\\qquad\t\\begin{equation}\\label{sec:2:5:agp:upy}\n\t\t\t\t\ty_{k+1}=\\mathcal{P}_\\mathcal{Y}\\left( y_k+\\tfrac{1}{\\gamma_k} \\nabla _y {f}( x_{k+1},y_k ) - \\tfrac{1}{\\gamma_k} c_k y_k \\right).\n\t\t\t\\end{equation}}\n\t\t\t\\STATE{\\textbf{Step 4}:If some stationary condition is satisfied, stop; otherwise, set $k=k+1, $ go to Step 2.}\n\t\t\t%    \\ENDFOR\n\t\t\\end{algorithmic}\n\t\\end{algorithm}\n","start":34199,"end":35221}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":["sec:2:def1:G"],"declared_external":false,"source_only":false,"title":"驻点间隙与分量","source_units":[{"raw_tex":"\t\\begin{defi}\\label{sec:2:def1:G}\n\t\tAt each iteration of Algorithm \\ref{alg:2.1}, the stationarity gap for problem \\eqref{problem:1} w.r.t. $f(x,y)$ is defined as:\n\t\t\\begin{equation*}\n\t\t\t\\nabla G\\left( x_k,y_k \\right) :=\\left[\\begin{array}{c}\n\t\t\t\t{\\beta_k\\left(x_k-\\operatorname{\\mathcal{P}_\\mathcal{X}}(x_k-\\frac{1}{\\beta_k} \\nabla_{x} f(x_k ,y_k))\\right)} \\\\\n\t\t\t\t\n\t\t\t\t{\\gamma_k\\left(y_k-\\operatorname{\\mathcal{P}_\\mathcal{Y}}(y_k+\\frac{1}{\\gamma_k} \\nabla_{y} f(x_k ,y_k))\\right)}\n\t\t\t\\end{array}\\right].\n\t\t\\end{equation*}\n    \tWe denote $\\nabla {G}_k:= \\nabla {G}(x_k, y_k)$,\n\t\t$(\\nabla G_k)_{x}:= \\beta_k( x_k-\\operatorname{\\mathcal{P}_\\mathcal{X}}( x_k-\\frac{1}{\\beta_k}\\nabla _{x}f( x_k,y_k)))$,\n\t\tand $(\\nabla G_k)_{y}:=\\gamma_k( y_k-\\operatorname{\\mathcal{P}_\\mathcal{Y}}( y_k+ \\frac{1}{\\gamma_k} \\nabla _{y}f( x_k,y_k )) )$.\n\t\t%\t\t\\begin{align*}\n\t\t%\t\t\t(\\nabla G^r)_{x_i}&:= \\beta_i^r\\left( x_i^r-\\operatorname{Prox_{\\mathcal{X}_i}^{\\beta_i^r}}\\left( x_i^r+ \\frac{1}{\\beta_i^r} \\nabla _{x_i}f\\left( x^r,y^r \\right) \\right) \\right),\\\\\n\t\t%\t\t\t(\\nabla G^r)_{y}&:=\\rho^r\\left( y^r-\\operatorname{Prox_{\\mathcal{Y}}^{\\rho^r}}\\left( y^r+ \\frac{1}{\\rho^r} \\nabla _{y}f\\left( x^r,y^r \\right) \\right) \\right).\n\t\t%\t\t\\end{align*}\n\t\\end{defi}\n","start":36086,"end":37321}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"近似驻点判据","source_units":[{"raw_tex":"\t\n\tNote that the widely used metrics for convex-concave minimax problems such as the min-max value or the distance to the set of optimal solutions of the min-max problem are not applicable in nonconvex cases.\n\tFor the latter cases we call $(x_k,y_k)$ an $\\varepsilon$-stationary point of $f(x,y)$ if $ \\|\\nabla G(x_k,y_k)\\|\\leq \\varepsilon$ with $\\nabla G(x_k,y_k)$ being defined as in Definition \\ref{sec:2:def1:G}, which has been widely used as the optimality measure, e.g., \\cite{Lin2020,Lu}. In the absence of constraints, $ \\| \\nabla G(x_k,y_k)\\| \\leq \\varepsilon$ reduces to the standard condition $ \\|\\nabla_{x} f(x_k ,y_k)\\|\\leq \\varepsilon $ and $\\|\\nabla_{y} f(x_k ,y_k)\\|\\leq \\varepsilon$ for unconstrained problems. The vector  $\\nabla G(x_k,y_k)$ also refers to gradient mapping at $(x_k,y_k)$, see \\cite{Nes2013} for the details. \n","start":37321,"end":38166}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","labels":["sec:2:ass1:lip"],"declared_external":false,"source_only":false,"title":"分量梯度 Lipschitz 假设","source_units":[{"raw_tex":"\t\\begin{ass}\\label{sec:2:ass1:lip}\n\t\t$f\\left(x,y\\right)$ has Lipschitz continuous gradients, i.e., there exist positive scalars $L_{22}$, $L_{12}$, $L_{11}$, $L_{21}$ such that for any $x,\\bar{x}\\in \\mathcal{X}$, $y, \\bar{y}\\in \\mathcal{Y}$,\n\t\t\\begin{align*}\n\t\t\t\\| \\nabla_{x} f\\left(x, y\\right)-\\nabla_{x} f\\left(\\bar{x}, y\\right)\\| &\\leq L_{11}\\|x-\\bar{x}\\|,\\\\\n\t\t\t\\|\\nabla_{x} f\\left(x, y\\right)-\\nabla_{x} f\\left(x, \\bar{y}\\right)\\| &\\leq L_{21}\\|y-\\bar{y}\\|,\\\\\n\t\t\t\\|\\nabla_{y} f\\left(x, \\bar{y}\\right)-\\nabla_{y} f\\left(x, y\\right)\\| &\\leq L_{22}\\|\\bar{y}-y\\|,\\\\\n\t\t\t\\|\\nabla_{y} f\\left(x, y\\right)-\\nabla_{y} f\\left(\\bar{x}, y\\right)\\| &\\leq L_{12}\\|x-\\bar{x}\\|.\n\t\t\\end{align*}\n\t\tWe denote $L:=\\max\\{L_{11}, L_{22}, L_{12}, L_{21}\\}$.\n\t\\end{ass}\n","start":39184,"end":39933}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","labels":["subsec:3.1:1"],"declared_external":false,"source_only":false,"title":"强凹辅助分析参数","source_units":[{"raw_tex":"\\begin{equation}\\label{subsec:3.1:1}\n\t\t\\beta_k=\\eta, \\gamma_k=\\tfrac{1}{\\rho}, \\ \\mbox{and} \\ b_k=c_k=0\n\t\\end{equation}","start":43836,"end":43955}]},{"id":"S012","semantic_type":"algorithm","kind":"algorithm","labels":["subsec:3.1:2:x","subsec:3.1:3:y"],"declared_external":false,"source_only":false,"title":"强凹辅助分析更新","source_units":[{"raw_tex":"\\begin{align}\n\t\tx_{k+1}&=\\mathcal{P}_\\mathcal{X} \\left( x_k - \\tfrac{1}{\\eta}  \\nabla _xf ( x_k,y_k )  \\right),\\label{subsec:3.1:2:x}\\\\\n\t\ty_{k+1}&=\\mathcal{P}_\\mathcal{Y}\\left( y_k+\\rho \\nabla _y {f}( x_{k+1},y_k ) \\right),\\label{subsec:3.1:3:y}\n\t\\end{align}","start":44042,"end":44300}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["subsec:3.1:4:defT"],"declared_external":false,"source_only":false,"title":"首次达到精度的索引","source_units":[{"raw_tex":"\tWe are now ready to establish the iteration complexity for the AGP algorithm in the nonconvex-strongly concave setting.\n\tIn particular,\n\tletting $\\nabla G\\left( x_k,y_k \\right)$ be defined as  in Definition \\ref{sec:2:def1:G} and $\\varepsilon>0$ be\n\ta given target accuracy, we provide\n\ta bound on $T(\\varepsilon)$, the first iteration index to achieve an $\\varepsilon$-stationary point, i.e., $ \\| \\nabla G(x_k,y_k)\\| \\leq \\varepsilon$, which is equivalent to\n\t\\begin{equation}\\label{subsec:3.1:4:defT}\n\t\tT(\\varepsilon):=\\min\\{k \\mid \\|\\nabla G(x_k,y_k)\\|\\leq \\varepsilon \\}.\n\t\\end{equation}\n","start":55574,"end":56168}]},{"id":"S014","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"强凸非凹问题情形","source_units":[{"raw_tex":"\\subsection{Strongly Convex-Nonconcave Setting}\\label{subsec4}\nIn this subsection, we analyze the iteration complexity of Algorithm \\ref{alg:2.1} for solving strongly convex-nonconcave minimax optimization problems \\eqref{problem:1}, i.e., $f(x,y)$ is $\\theta$-strongly convex w.r.t. $x$ for any fixed $y\\in \\mathcal{Y}$,\nand nonconcave  w.r.t. $y$ for any given $x\\in \\mathcal{X}$. Under this setting,   $\\forall k\\ge 1$, we set\n","start":93416,"end":93846}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","labels":["subsec:4.1:1:para"],"declared_external":false,"source_only":false,"title":"强凸情形参数特化","source_units":[{"raw_tex":"\\begin{equation}\n\\beta_k=\\tfrac{1}{\\zeta}, \\gamma_k=\\nu,  b_k=c_k=0,\\label{subsec:4.1:1:para}\n\\end{equation}\nin  Algorithm \\ref{alg:2.1}, and simplify the update for $x_k$ and $y_k$ as follows:\n","start":93846,"end":94040}]},{"id":"S016","semantic_type":"algorithm","kind":"algorithm","labels":["subsec:4.1:2:x","subsec:4.1:3:y"],"declared_external":false,"source_only":false,"title":"强凸情形交替更新","source_units":[{"raw_tex":"\\begin{align}\n\tx_{k+1}&=\\mathcal{P}_\\mathcal{X}\\left( x_k- \\zeta \\nabla _x {f}( x_{k},y_k) \\right),\\label{subsec:4.1:2:x}\\\\\n\ty_{k+1}&=\\mathcal{P}_\\mathcal{Y} \\left(y_k + \\tfrac{1}{\\nu}  \\nabla _yf ( x_{k+1},y_k )  \\right)\\label{subsec:4.1:3:y}.\n\\end{align}\n","start":94040,"end":94297}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":true,"title":"辅助引理的强凹情形","source_units":[{"raw_tex":"\tIn this subsection, we analyze the iteration complexity of Algorithm \\ref{alg:2.1} for solving nonconvex-strongly concave minimax optimization problems, i.e., $f(x,y)$ is nonconvex w.r.t. $x$ for any fixed $y\\in \\mathcal{Y}$,\n\tand $\\mu$-strongly concave  w.r.t. $y$ for any given $x\\in \\mathcal{X}$. Under this setting,  $\\forall k\\ge 1$, we set\n","start":43488,"end":43835}]},{"id":"S018","semantic_type":"assumption","kind":"assumption","labels":["thm:4.1"],"declared_external":false,"source_only":false,"title":"主定理参数限制","source_units":[{"raw_tex":"\tSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{\\left(x_{k},y_{k}\\right)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}\n\twith parameter settings in \\eqref{subsec:4.1:1:para}.\n\tIf the relations $\\nu>L_{22}, \\nu>L_{21}^2\\zeta+\\frac{4L_{21}^2}{\\zeta \\theta^2},~\\zeta\\leq \\frac{\\theta}{4L_{11}^2}$ are satisfied, then $\\forall \\varepsilon>0$,  it holds that $$ T\\left( \\varepsilon \\right) \\le\n","start":105709,"end":106117}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":["thm:4.1"],"declared_external":false,"source_only":false,"title":"复杂度常数与势函数上界记号","source_units":[{"raw_tex":"\twhere\n\t$\\hat{d}_1:=\\tfrac{\\min \\left\\{\\tfrac{\\nu}{2}-\\tfrac{\\zeta L_{21}^2}{2}-\\tfrac{2 L_{21}^2}{\\zeta \\theta^2},\\tfrac{3\\theta-\\zeta L_{11}^2}{2}+\\tfrac{\\theta-4\\zeta L_{11}^2}{2\\zeta\\theta} \\right\\}}{\\max \\left\\{ \\tfrac{1}{\\zeta ^2}+2L_{12}^{2},2\\nu^2 \\right\\}}%\\overline{\\hat{F}}=\\overline{f}-(\\tfrac{\\theta}{2}-\\tfrac{3}{\\zeta})\\sigma _x^2\n\t$\n\tand $\\overline{F}=\\bar{f}+(\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\sigma_x^2$ with $\\overline{f} := \\max_{(x,y) \\in \\mathcal{X} \\times \\mathcal{Y}} f(x,y)$ and $\\sigma _x= \\max\\{\\|x_1-x_2\\| \\mid \\forall x_1, x_2\\in \\mathcal{X}\\}$.\n","start":106177,"end":106793}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","labels":["lem:2.1"],"declared_external":false,"source_only":false,"title":"通用原变量函数变化界","source_units":[{"raw_tex":"\\begin{lem}\\label{lem:2.1}\nSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{(x_k,y_k)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}. Then we have\n\\begin{align}\n  f\\left( x_{k+1},y_k \\right) -f\\left( x_k,y_k \\right) &\\le-\\left( \\beta_k -\\tfrac{L_{11}}{2} \\right) \\lVert x_{k+1}-x_k \\rVert ^2-\\frac{b_k}{2}\\left[\\|x_{k+1}\\|^2-\\|x_k\\|^2-\\|x_{k+1}-x_k\\|^2\\right], \\label{lem:2.1:1}\\\\\n","start":40128,"end":40526}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","labels":["lem:2.1"],"declared_external":false,"source_only":false,"title":"通用对偶变量函数变化界","source_units":[{"raw_tex":"  f(x_{k+1},y_{k+1})-f( x_{k+1},y_{k})\n\t\t\t&\\ge\\left( \\gamma_k -\\tfrac{L_{22}}{2} \\right) \\lVert y_{k+1}-y_k \\rVert ^2+\\frac{c_k}{2}\\left[\\|y_{k+1}\\|^2-\\|y_k\\|^2-\\|y_{k+1}-y_k\\|^2\\right].\\label{lem:2.1:2}\n\\end{align}\n\\end{lem}","start":40526,"end":40751}]},{"id":"S022","semantic_type":"proof","kind":"proof","labels":["lem:2.1:3","lem:2.1:4","lem:2.1:6","lem:2.1:7"],"declared_external":false,"source_only":false,"title":"通用函数变化引理完整证明","source_units":[{"raw_tex":"\\begin{proof}\n  Firstly, by the optimality condition for \\eqref{sec:2:4:agp:upx}, we have\n\t\t\\begin{equation}\\label{lem:2.1:3}\n\t\t\t\\langle  \\nabla _xf\\left( x_k,y_k \\right) +\\beta_k \\left( x_{k+1}-x_k \\right)+b_kx_k ,x_k-x_{k+1} \\rangle  \\ge 0.\n\t\t\\end{equation}\n\t\tBy Assumption \\ref{sec:2:ass1:lip}, the gradient of $f$ is Lipschitz continuous, implying that\n\t\t\\begin{equation}\n\t\t\tf\\left( x_{k+1},y_k \\right) -f\\left( x_k,y_k \\right)\\leq \\langle  \\nabla _xf\\left( x_k,y_k \\right) ,x_{k+1}-x_k \\rangle  +\\tfrac{L_{11}}{2}\\lVert x_{k+1}-x_k \\rVert ^2. \\label{lem:2.1:4}\n\t\t\\end{equation}\n\t\tAdding \\eqref{lem:2.1:3} and \\eqref{lem:2.1:4}, we can easily show that\n\t\t\\begin{align*}\n\t\t\tf\\left( x_{k+1},y_k \\right) -f\\left( x_k,y_k \\right) &\\le -\\left( \\beta_k -\\tfrac{L_{11}}{2} \\right) \\lVert x_{k+1}-x_k \\rVert ^2-b_k\\langle x_k, x_{k+1}-x_k\\rangle\\nonumber\\\\\n            &= -\\left( \\beta_k -\\tfrac{L_{11}}{2} \\right) \\lVert x_{k+1}-x_k \\rVert ^2-\\frac{b_k}{2}\\left[\\|x_{k+1}\\|^2-\\|x_k\\|^2-\\|x_{k+1}-x_k\\|^2\\right].\n\t\t\\end{align*}\nThis completes the proof of \\eqref{lem:2.1:1}. By the optimality condition for \\eqref{sec:2:5:agp:upy}, we have\n\t\t\\begin{equation}\\label{lem:2.1:6}\n\t\t\t\\langle  \\nabla _{y}f( x_{k+1},y_{k})-\\gamma_k( y_{k+1}-y_k)-c_ky_k ,y_k-y_{k+1} \\rangle  \\leq 0.\n\t\t\\end{equation}\n\t\tBy  Assumption \\ref{sec:2:ass1:lip}, the gradient of $f$ is Lipschitz continuous,  implying that\n\t\t\\begin{align}\\label{lem:2.1:7}\n\t\t\tf(x_{k+1},y_{k+1})-f( x_{k+1},y_{k})\n\t\t\t\\ge  \\langle \\nabla _{y}f(x_{k+1},y_{k}),y_{k+1}\n    -y_{k} \\rangle -\\frac{L_{22}}{2}\\| y_{k+1}-y_{k} \\|^2.\n\t\t\\end{align}\n\t\t\tThe result \\eqref{lem:2.1:2} then follows by adding \\eqref{lem:2.1:6} and \\eqref{lem:2.1:7}.\n\n\\end{proof}","start":40753,"end":42448}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","labels":["lem:2.1:3"],"declared_external":false,"source_only":false,"title":"原变量投影最优性","source_units":[{"raw_tex":"\\begin{proof}\n  Firstly, by the optimality condition for \\eqref{sec:2:4:agp:upx}, we have\n\t\t\\begin{equation}\\label{lem:2.1:3}\n\t\t\t\\langle  \\nabla _xf\\left( x_k,y_k \\right) +\\beta_k \\left( x_{k+1}-x_k \\right)+b_kx_k ,x_k-x_{k+1} \\rangle  \\ge 0.\n\t\t\\end{equation}\n","start":40753,"end":41013}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","labels":["lem:2.1:4"],"declared_external":false,"source_only":false,"title":"原变量光滑性上界","source_units":[{"raw_tex":"\t\tBy Assumption \\ref{sec:2:ass1:lip}, the gradient of $f$ is Lipschitz continuous, implying that\n\t\t\\begin{equation}\n\t\t\tf\\left( x_{k+1},y_k \\right) -f\\left( x_k,y_k \\right)\\leq \\langle  \\nabla _xf\\left( x_k,y_k \\right) ,x_{k+1}-x_k \\rangle  +\\tfrac{L_{11}}{2}\\lVert x_{k+1}-x_k \\rVert ^2. \\label{lem:2.1:4}\n\t\t\\end{equation}\n","start":41013,"end":41336}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"合成原变量下降界","source_units":[{"raw_tex":"\t\tAdding \\eqref{lem:2.1:3} and \\eqref{lem:2.1:4}, we can easily show that\n\t\t\\begin{align*}\n\t\t\tf\\left( x_{k+1},y_k \\right) -f\\left( x_k,y_k \\right) &\\le -\\left( \\beta_k -\\tfrac{L_{11}}{2} \\right) \\lVert x_{k+1}-x_k \\rVert ^2-b_k\\langle x_k, x_{k+1}-x_k\\rangle\\nonumber\\\\\n            &= -\\left( \\beta_k -\\tfrac{L_{11}}{2} \\right) \\lVert x_{k+1}-x_k \\rVert ^2-\\frac{b_k}{2}\\left[\\|x_{k+1}\\|^2-\\|x_k\\|^2-\\|x_{k+1}-x_k\\|^2\\right].\n\t\t\\end{align*}\nThis completes the proof of \\eqref{lem:2.1:1}. By the optimality condition for \\eqref{sec:2:5:agp:upy}, we have\n","start":41336,"end":41889}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","labels":["lem:2.1:6"],"declared_external":false,"source_only":false,"title":"对偶变量投影最优性","source_units":[{"raw_tex":"This completes the proof of \\eqref{lem:2.1:1}. By the optimality condition for \\eqref{sec:2:5:agp:upy}, we have\n\t\t\\begin{equation}\\label{lem:2.1:6}\n\t\t\t\\langle  \\nabla _{y}f( x_{k+1},y_{k})-\\gamma_k( y_{k+1}-y_k)-c_ky_k ,y_k-y_{k+1} \\rangle  \\leq 0.\n\t\t\\end{equation}\n","start":41777,"end":42043}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","labels":["lem:2.1:7"],"declared_external":false,"source_only":false,"title":"对偶变量光滑性下界","source_units":[{"raw_tex":"\t\tBy  Assumption \\ref{sec:2:ass1:lip}, the gradient of $f$ is Lipschitz continuous,  implying that\n\t\t\\begin{align}\\label{lem:2.1:7}\n\t\t\tf(x_{k+1},y_{k+1})-f( x_{k+1},y_{k})\n\t\t\t\\ge  \\langle \\nabla _{y}f(x_{k+1},y_{k}),y_{k+1}\n    -y_{k} \\rangle -\\frac{L_{22}}{2}\\| y_{k+1}-y_{k} \\|^2.\n\t\t\\end{align}\n","start":42043,"end":42340}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"合成对偶变量上升界","source_units":[{"raw_tex":"\t\t\tThe result \\eqref{lem:2.1:2} then follows by adding \\eqref{lem:2.1:6} and \\eqref{lem:2.1:7}.\n\n\\end{proof}","start":42340,"end":42448}]},{"id":"S029","semantic_type":"lemma","kind":"lemma","labels":["lem:3.2:1","lem:3.2:blostr:y"],"declared_external":false,"source_only":false,"title":"强凹情形函数变化递推","source_units":[{"raw_tex":"\\begin{lem}\\label{lem:3.2:blostr:y}\n\t\tSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{\\left(x_k,y_k\\right)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}\n\t\twith parameter settings in \\eqref{subsec:3.1:1}. If $\\eta > L_{11}$, then we have\n\t\t\\begin{align}\\label{lem:3.2:1}\n\t\t\t&f( x_{k+1},y_{k+1}) -f\\left( x_k,y_k \\right)  \\nonumber \\\\\n\t\t\t\\leq& -\\left(\\tfrac{\\eta}{2}-\\tfrac{L_{12}^{2}\\rho}{2}\\right)\\| x_{k+1}-x_{k} \\| ^2-\\left(\\tfrac{\\mu}{2}-\\tfrac{1}{\\rho}\\right)\\| y_{k+1}-y_k \\|^2\\nonumber \\\\\n\t\t\t&-\\left(\\mu-\\tfrac{1}{2\\rho}-\\tfrac{\\rho L_{22}^2}{2}\\right)\\|y_k-y_{k-1} \\|^2.\n\t\t\\end{align}\n\t\\end{lem}","start":46965,"end":47586}]},{"id":"S030","semantic_type":"proof","kind":"proof","labels":["lem:3.1:1","lem:3.2:10:3points","lem:3.2:11","lem:3.2:2","lem:3.2:3","lem:3.2:4","lem:3.2:5","lem:3.2:6","lem:3.2:7","lem:3.2:8","lem:3.2:9"],"declared_external":false,"source_only":false,"title":"强凹函数变化递推完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\t\tThe optimality condition for $y_k$ in \\eqref{subsec:3.1:3:y} implies that $\\forall y\\in \\mathcal{Y}$ and $\\forall k\\geq 1$,\n\t\t\\begin{equation}\\label{lem:3.2:2}\n\t\t\t\\langle \\nabla _yf(x_{k+1},y_k)-\\frac{1}{\\rho}(y_{k+1}-y_k),y-y_{k+1} \\rangle \\le 0.\n\t\t\\end{equation}\n\t\tBy choosing $y=y_k$ in \\eqref{lem:3.2:2}, we have\n\t\t\\begin{equation}\\label{lem:3.2:3}\n\t\t\t\\langle \\nabla _yf(x_{k+1},y_k)-\\frac{1}{\\rho}(y_{k+1}-y_k),y_k-y_{k+1} \\rangle \\le 0.\n\t\t\\end{equation}\n\t\tOn the other hand, by replacing $k$ with $k-1$ and choosing $y=y_{k+1}$ in \\eqref{lem:3.2:2}, we obtain\n\t\t\\begin{equation}\\label{lem:3.2:4}\n\t\t\t\\langle \\nabla _yf(x_{k},y_{k-1})-\\frac{1}{\\rho}(y_k-y_{k-1}),y_{k+1}-y_k \\rangle \\le 0,\n\t\t\\end{equation}\n\t\twhich, in view of the fact that $f\\left( x,y  \\right)$ is $\\mu $-strongly concave w.r.t. $y$ for any given $x\\in \\mathcal{X}$,\n\t\tthen implies that\n\t\t\\begin{align}\\label{lem:3.2:5}\n\t\t\t&f( x_{k+1},y_{k+1}) -f( x_{k+1},y_k ) \\nonumber\\\\\n\t\t\t\\leq &  \\langle  \\nabla _yf( x_{k+1},y_k ),y_{k+1}-y_k\\rangle  -\\frac{\\mu}{2}\\| y_{k+1}-y_k \\|^2\\  \\nonumber\\\\\n\t\t\t\\leq &\\langle  \\nabla _yf( x_{k+1},y_k ) -\\nabla _yf( x_k,y_{k-1}) ,y_{k+1}-y_k \\rangle   \\nonumber\\\\\n\t\t\t& +\\frac{1}{\\rho}\\langle  y_k-y_{k-1},y_{k+1}-y_k\\rangle  -\\frac{\\mu}{2}\\| y_{k+1}-y_k \\|^2.\n\t\t\\end{align}\n\t\tDenoting $v_{k+1}:=\\left(y_{k+1}-y_k\\right)-\\left(y_k-y_{k-1}\\right)$, we can write the first inner product term in the r.h.s.\n\t\tof \\eqref{lem:3.2:5} as\n\t\t\\begin{align}\\label{lem:3.2:6}\n\t\t\t&\\langle  \\nabla _y f ( x_{k+1},y_k ) -\\nabla _y f ( x_k,y_{k-1}) ,y_{k+1}-y_k \\rangle  \\nonumber\\\\\n\t\t\t=&\\langle  \\nabla _y f ( x_{k+1},y_k ) -\\nabla _y f \\left( x_k,y_{k} \\right) ,y_{k+1}-y_k \\rangle  \\nonumber\\\\\n\t\t\t& + \\langle  \\nabla _y f \\left( x_k,y_{k} \\right) -\\nabla _y f ( x_k,y_{k-1}) ,v_{k+1} \\rangle  \\nonumber\\\\\n\t\t\t&+\\langle  \\nabla _y f \\left( x_k,y_{k}\\right) -\\nabla _y f ( x_k,y_{k-1}) ,y_k-y_{k-1} \\rangle .\n\t\t\\end{align}\n\t\tNext, we estimate the three terms in the right hand side of \\eqref{lem:3.2:6} respectively.\n\t\tBy Assumption \\ref{sec:2:ass1:lip} and the Cauchy-Schwarz inequality, we can bound the first two terms according to\n\t\t\\begin{align}\\label{lem:3.2:7}\n\t\t\t&\\langle  \\nabla _yf( x_{k+1},y_k ) -\\nabla _yf\\left( x_k,y_{k} \\right) ,y_{k+1}-y_k \\rangle  \\nonumber\\\\\n\t\t\t\\leq &\\frac{L_{12}^{2}\\rho}{2}\\| x_{k+1}-x_k \\|^2+\\frac{1}{2\\rho}\\| y_{k+1}-y_k \\|^2,\n\t\t\\end{align}\n\t\tand\n\t\t\\begin{align}\\label{lem:3.2:8}\n\t\t\t&\\langle  \\nabla _yf\\left(  x_k,y_{k} \\right) -\\nabla _yf( x_k,y_{k-1}) ,v_{k+1} \\rangle  \\leq \\frac{\\rho L_{22}^{2}}{2}\\|y_k-y_{k-1} \\|^2+\\frac{1}{2\\rho}\\|v_{k+1} \\|^2.\n\t\t\\end{align}\n\t\tFor the third term, by $\\mu $-strongly-concavity  of $f$ with respect to $y$,\n\t\t\\begin{align}\\label{lem:3.2:9}\n\t\t\t\\quad \\langle  \\nabla _yf\\left( x_k,y_{k} \\right) -\\nabla _yf( x_k,y_{k-1}) ,y_k-y_{k-1} \\rangle  \\le -\\mu \\| y_k-y_{k-1} \\|^2 .\n\t\t\\end{align}\n\t\tMoreover, it can be easily checked that\n\t\t\\begin{align}\\label{lem:3.2:10:3points}\n\t\t\t&\\langle  y_k-y_{k-1},y_{k+1}-y_k \\rangle =\\frac{1}{2}\\| y_k-y_{k-1} \\|^2+\\frac{1}{2}\\| y_{k+1}-y_k \\|^2-\\frac{1}{2}\\| v_{k+1} \\|^2.\n\t\t\\end{align}\n\t\tPlugging \\eqref{lem:3.2:6}-\\eqref{lem:3.2:10:3points} into \\eqref{lem:3.2:5} and rearranging the terms, we conclude that\n\t\t\\begin{align}\\label{lem:3.2:11}\n\t\t\t&f( x_{k+1},y_{k+1}) -f( x_{k+1},y_{k})  \\nonumber \\\\\n\t\t\t\\leq& \\frac{L_{12}^{2}\\rho}{2}\\| x_{k+1}-x_k \\|^2-(\\mu-\\frac{1}{2\\rho}-\\frac{\\rho L_{22}^2}{2})\\| y_k-y_{k-1}\\|^2\\nonumber\\\\\n\t\t\t&-(\\frac{\\mu}{2}-\\frac{1}{\\rho})\\|y_{k+1}-y_k \\|^2.\n\t\t\\end{align}\nBy setting $\\beta_k=\\eta$, $b_k=0$ in \\eqref{lem:2.1:1} of Lemma \\ref{lem:2.1}  and the assumption $\\eta > L_{11}$, we have\n\t\t\\begin{equation}\\label{lem:3.1:1}\n\t\t\tf(x_{k+1},y_{k})-f(x_{k},y_k)\\le -\\frac{\\eta}{2}\\| x_{k+1}-x_k \\|^2.\n\t\t\\end{equation}\nThe proof is completed by combining \\eqref{lem:3.2:11} with \\eqref{lem:3.1:1}.\n\t\\end{proof}","start":47590,"end":51440}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":["lem:3.2:2","lem:3.2:3"],"declared_external":false,"source_only":false,"title":"当前对偶投影最优性","source_units":[{"raw_tex":"\\begin{proof}\n\t\tThe optimality condition for $y_k$ in \\eqref{subsec:3.1:3:y} implies that $\\forall y\\in \\mathcal{Y}$ and $\\forall k\\geq 1$,\n\t\t\\begin{equation}\\label{lem:3.2:2}\n\t\t\t\\langle \\nabla _yf(x_{k+1},y_k)-\\frac{1}{\\rho}(y_{k+1}-y_k),y-y_{k+1} \\rangle \\le 0.\n\t\t\\end{equation}\n\t\tBy choosing $y=y_k$ in \\eqref{lem:3.2:2}, we have\n\t\t\\begin{equation}\\label{lem:3.2:3}\n\t\t\t\\langle \\nabla _yf(x_{k+1},y_k)-\\frac{1}{\\rho}(y_{k+1}-y_k),y_k-y_{k+1} \\rangle \\le 0.\n\t\t\\end{equation}\n","start":47590,"end":48066}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":["lem:3.2:4"],"declared_external":false,"source_only":false,"title":"前次对偶投影最优性","source_units":[{"raw_tex":"\t\tOn the other hand, by replacing $k$ with $k-1$ and choosing $y=y_{k+1}$ in \\eqref{lem:3.2:2}, we obtain\n\t\t\\begin{equation}\\label{lem:3.2:4}\n\t\t\t\\langle \\nabla _yf(x_{k},y_{k-1})-\\frac{1}{\\rho}(y_k-y_{k-1}),y_{k+1}-y_k \\rangle \\le 0,\n\t\t\\end{equation}\n","start":48066,"end":48317}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":["lem:3.2:5"],"declared_external":false,"source_only":false,"title":"强凹性控制对偶函数变化","source_units":[{"raw_tex":"\t\twhich, in view of the fact that $f\\left( x,y  \\right)$ is $\\mu $-strongly concave w.r.t. $y$ for any given $x\\in \\mathcal{X}$,\n\t\tthen implies that\n\t\t\\begin{align}\\label{lem:3.2:5}\n\t\t\t&f( x_{k+1},y_{k+1}) -f( x_{k+1},y_k ) \\nonumber\\\\\n\t\t\t\\leq &  \\langle  \\nabla _yf( x_{k+1},y_k ),y_{k+1}-y_k\\rangle  -\\frac{\\mu}{2}\\| y_{k+1}-y_k \\|^2\\  \\nonumber\\\\\n\t\t\t\\leq &\\langle  \\nabla _yf( x_{k+1},y_k ) -\\nabla _yf( x_k,y_{k-1}) ,y_{k+1}-y_k \\rangle   \\nonumber\\\\\n\t\t\t& +\\frac{1}{\\rho}\\langle  y_k-y_{k-1},y_{k+1}-y_k\\rangle  -\\frac{\\mu}{2}\\| y_{k+1}-y_k \\|^2.\n\t\t\\end{align}\n","start":48317,"end":48882}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":["lem:3.2:6"],"declared_external":false,"source_only":false,"title":"对偶梯度差三项分解","source_units":[{"raw_tex":"\t\tDenoting $v_{k+1}:=\\left(y_{k+1}-y_k\\right)-\\left(y_k-y_{k-1}\\right)$, we can write the first inner product term in the r.h.s.\n\t\tof \\eqref{lem:3.2:5} as\n\t\t\\begin{align}\\label{lem:3.2:6}\n\t\t\t&\\langle  \\nabla _y f ( x_{k+1},y_k ) -\\nabla _y f ( x_k,y_{k-1}) ,y_{k+1}-y_k \\rangle  \\nonumber\\\\\n\t\t\t=&\\langle  \\nabla _y f ( x_{k+1},y_k ) -\\nabla _y f \\left( x_k,y_{k} \\right) ,y_{k+1}-y_k \\rangle  \\nonumber\\\\\n\t\t\t& + \\langle  \\nabla _y f \\left( x_k,y_{k} \\right) -\\nabla _y f ( x_k,y_{k-1}) ,v_{k+1} \\rangle  \\nonumber\\\\\n\t\t\t&+\\langle  \\nabla _y f \\left( x_k,y_{k}\\right) -\\nabla _y f ( x_k,y_{k-1}) ,y_k-y_{k-1} \\rangle .\n\t\t\\end{align}\n","start":48882,"end":49513}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":["lem:3.2:7","lem:3.2:8"],"declared_external":false,"source_only":false,"title":"对偶梯度差的光滑性估计","source_units":[{"raw_tex":"\t\tNext, we estimate the three terms in the right hand side of \\eqref{lem:3.2:6} respectively.\n\t\tBy Assumption \\ref{sec:2:ass1:lip} and the Cauchy-Schwarz inequality, we can bound the first two terms according to\n\t\t\\begin{align}\\label{lem:3.2:7}\n\t\t\t&\\langle  \\nabla _yf( x_{k+1},y_k ) -\\nabla _yf\\left( x_k,y_{k} \\right) ,y_{k+1}-y_k \\rangle  \\nonumber\\\\\n\t\t\t\\leq &\\frac{L_{12}^{2}\\rho}{2}\\| x_{k+1}-x_k \\|^2+\\frac{1}{2\\rho}\\| y_{k+1}-y_k \\|^2,\n\t\t\\end{align}\n\t\tand\n\t\t\\begin{align}\\label{lem:3.2:8}\n\t\t\t&\\langle  \\nabla _yf\\left(  x_k,y_{k} \\right) -\\nabla _yf( x_k,y_{k-1}) ,v_{k+1} \\rangle  \\leq \\frac{\\rho L_{22}^{2}}{2}\\|y_k-y_{k-1} \\|^2+\\frac{1}{2\\rho}\\|v_{k+1} \\|^2.\n\t\t\\end{align}\n","start":49513,"end":50196}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":["lem:3.2:9"],"declared_external":false,"source_only":false,"title":"强凹梯度的负单调性","source_units":[{"raw_tex":"\t\tFor the third term, by $\\mu $-strongly-concavity  of $f$ with respect to $y$,\n\t\t\\begin{align}\\label{lem:3.2:9}\n\t\t\t\\quad \\langle  \\nabla _yf\\left( x_k,y_{k} \\right) -\\nabla _yf( x_k,y_{k-1}) ,y_k-y_{k-1} \\rangle  \\le -\\mu \\| y_k-y_{k-1} \\|^2 .\n\t\t\\end{align}\n","start":50196,"end":50455}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":["lem:3.2:10:3points"],"declared_external":false,"source_only":false,"title":"对偶三点内积恒等式","source_units":[{"raw_tex":"\t\tMoreover, it can be easily checked that\n\t\t\\begin{align}\\label{lem:3.2:10:3points}\n\t\t\t&\\langle  y_k-y_{k-1},y_{k+1}-y_k \\rangle =\\frac{1}{2}\\| y_k-y_{k-1} \\|^2+\\frac{1}{2}\\| y_{k+1}-y_k \\|^2-\\frac{1}{2}\\| v_{k+1} \\|^2.\n\t\t\\end{align}\n","start":50455,"end":50689}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":["lem:3.2:11"],"declared_external":false,"source_only":false,"title":"合成强凹对偶变化界","source_units":[{"raw_tex":"\t\tPlugging \\eqref{lem:3.2:6}-\\eqref{lem:3.2:10:3points} into \\eqref{lem:3.2:5} and rearranging the terms, we conclude that\n\t\t\\begin{align}\\label{lem:3.2:11}\n\t\t\t&f( x_{k+1},y_{k+1}) -f( x_{k+1},y_{k})  \\nonumber \\\\\n\t\t\t\\leq& \\frac{L_{12}^{2}\\rho}{2}\\| x_{k+1}-x_k \\|^2-(\\mu-\\frac{1}{2\\rho}-\\frac{\\rho L_{22}^2}{2})\\| y_k-y_{k-1}\\|^2\\nonumber\\\\\n\t\t\t&-(\\frac{\\mu}{2}-\\frac{1}{\\rho})\\|y_{k+1}-y_k \\|^2.\n\t\t\\end{align}\n","start":50689,"end":51100}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":["lem:3.1:1"],"declared_external":false,"source_only":false,"title":"特化通用原变量下降界","source_units":[{"raw_tex":"By setting $\\beta_k=\\eta$, $b_k=0$ in \\eqref{lem:2.1:1} of Lemma \\ref{lem:2.1}  and the assumption $\\eta > L_{11}$, we have\n\t\t\\begin{equation}\\label{lem:3.1:1}\n\t\t\tf(x_{k+1},y_{k})-f(x_{k},y_k)\\le -\\frac{\\eta}{2}\\| x_{k+1}-x_k \\|^2.\n\t\t\\end{equation}\n","start":51100,"end":51349}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"完成强凹函数变化递推","source_units":[{"raw_tex":"The proof is completed by combining \\eqref{lem:3.2:11} with \\eqref{lem:3.1:1}.\n\t\\end{proof}","start":51349,"end":51440}]},{"id":"S041","semantic_type":"lemma","kind":"lemma","labels":["lem:4.2","lem:4.2:1:conclu"],"declared_external":false,"source_only":false,"title":"强凸情形函数变化递推","source_units":[{"raw_tex":"\\begin{lem}\\label{lem:4.2}\n\tSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{\\left(x_{k},y_{k}\\right)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}\n\twith parameter settings in \\eqref{subsec:4.1:1:para}. If $\\nu > L_{22}$, then we have\n\t\\begin{align}\\label{lem:4.2:1:conclu}\n\t\t&f( x_{k+1},y_{k+1}) -f( x_{k},y_{k})  \\nonumber \\\\\n\t\t\\geq& \\frac{\\nu}{2}\\| y_{k+1}-y_{k} \\| ^2-\\frac{L_{21}^{2}\\zeta}{2}\\| y_{k}-y_{k-1} \\|^2+(\\theta-\\frac{1}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2})\\| x_{k}-x_{k-1}\\|^2\\nonumber\\\\\n\t\t&+ (\\frac{\\theta}{2}-\\frac{1}{\\zeta})\\|x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\n\\end{lem}","start":97011,"end":97613}]},{"id":"S042","semantic_type":"proof","kind":"proof","labels":["lem:4.1:1:conclu","lem:4.2:2:opty","lem:4.2:3:nopty","lem:4.2:4:opty:k","lem:4.2:5","lem:4.2:6","lem:4.2:7","lem:4.2:8:3points","lem:4.2:9"],"declared_external":false,"source_only":false,"title":"强凸函数变化递推完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\tSimilar to \\eqref{lem:3.2:2}-\\eqref{lem:3.2:4} in the proof of Lemma \\ref{lem:3.2:blostr:y}, by the optimality condition for $x_k$ in \\eqref{subsec:4.1:2:x} implies that $\\forall x\\in \\mathcal{X}$ and $\\forall k\\geq 1$,\n\t\\begin{align}\n\t\t\\langle \\nabla _xf(x_{k},y_{k})+\\frac{1}{\\zeta}(x_{k+1}-x_{k}),x-x_{k+1} \\rangle &\\ge 0,\\label{lem:4.2:2:opty}\\\\\n\t\t\\langle \\nabla _xf(x_{k},y_{k})+\\frac{1}{\\zeta}(x_{k+1}-x_{k}) ,x_{k}-x_{k+1} \\rangle &\\ge 0,\\label{lem:4.2:3:nopty}\\\\\n\t\t\\langle \\nabla _xf(x_{k-1},y_{k-1})+\\frac{1}{\\zeta}(x_{k}-x_{k-1}),x_{k+1}-x_{k} \\rangle &\\ge 0,\\label{lem:4.2:4:opty:k}\n\t\\end{align}\nwhich, in view of the fact that $f\\left( x,y  \\right)$ is $\\theta $-strongly convex w.r.t. $x$ for any given $y\\in \\mathcal{Y}$,\n\tthen implies that\n\t\\begin{align}\\label{lem:4.2:5}\n\t\t&f( x_{k+1},y_{k}) -f( x_{k},y_{k} ) \\nonumber\\\\\n\t\t\\geq &  \\langle  \\nabla _xf( x_{k},y_{k} ),x_{k+1}-x_{k}\\rangle  +\\frac{\\theta}{2}\\| x_{k+1}-x_{k} \\|^2\\  \\nonumber\\\\\n\t\t\\geq &\\langle  \\nabla _xf( x_{k},y_{k} ) -\\nabla _xf( x_{k-1},y_{k-1}) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t& -\\frac{1}{\\zeta}\\langle  x_{k}-x_{k-1},x_{k+1}-x_{k} \\rangle +\\frac{\\theta}{2}\\| x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\n\tDenoting $m_{k+1}:=\\left(x_{k+1}-x_k\\right)-\\left(x_k-x_{k-1}\\right)$, by Assumption \\ref{sec:2:ass1:lip}, the Cauchy-Schwarz inequality and\n\tthe $\\theta $-strongly convexity  of $f$ w.r.t. $x$, we can estimate the first inner product term in the r.h.s.\n\tof \\eqref{lem:4.2:5} as\n\t\\begin{align}\n\t\t&\\langle  \\nabla _x f ( x_{k},y_{k} ) -\\nabla _x f ( x_{k-1},y_{k-1}) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t=&\\langle  \\nabla _x f ( x_{k},y_{k} ) -\\nabla _x f \\left( x_{k},y_{k-1} \\right) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t& + \\langle  \\nabla _x f \\left( x_{k},y_{k-1} \\right) -\\nabla _x f ( x_{k-1},y_{k-1}) ,m_{k+1} \\rangle  \\nonumber\\\\\n\t\t&+\\langle  \\nabla _x f \\left( x_{k},y_{k-1}\\right) -\\nabla _x f ( x_{k-1},y_{k-1}) ,x_{k}-x_{k-1} \\rangle \\label{lem:4.2:6}\\\\\n\t\\geq &-\\frac{L_{21}^{2}\\zeta}{2}\\| y_{k}-y_{k-1} \\|^2-\\frac{1}{2\\zeta}\\| x_{k+1}-x_k \\|^2-\\frac{\\zeta L_{11}^{2}}{2}\\|x_k-x_{k-1} \\|^2\\nonumber\\\\\n\t&-\\frac{1}{2\\zeta}\\|m_{k+1} \\|^2 +\t\\theta \\| x_k-x_{k-1} \\|^2.\\label{lem:4.2:7}\n\t\\end{align}\n\tMoreover, it can be easily checked that\n\\begin{align}\\label{lem:4.2:8:3points}\n\t\t&\\langle  x_k-x_{k-1},x_{k+1}-x_k \\rangle =\\frac{1}{2}\\| x_k-x_{k-1} \\|^2+\\frac{1}{2}\\| x_{k+1}-x_k \\|^2-\\frac{1}{2}\\| m_{k+1} \\|^2.\n\t\\end{align}\n\tPlugging \\eqref{lem:4.2:7} and \\eqref{lem:4.2:8:3points} into \\eqref{lem:4.2:5} and rearranging the terms, we conclude that\n\t\\begin{align}\\label{lem:4.2:9}\n\tf( x_{k+1},y_{k}) -f( x_{k},y_{k})\t\\geq& -\\frac{L_{21}^{2}\\zeta}{2}\\| y_{k}-y_{k-1} \\|^2+(\\theta-\\frac{1}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2})\\| x_{k}-x_{k-1}\\|^2\\nonumber\\\\\n\t\t&+ (\\frac{\\theta}{2}-\\frac{1}{\\zeta})\\|x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\nBy setting $\\gamma_k=\\nu$, $c_k=0$ in \\eqref{lem:2.1:2} of Lemma \\ref{lem:2.1}  and the assumption $\\nu > L_{22}$, we have\n\t\t\\begin{equation}\\label{lem:4.1:1:conclu}\n\t\t\tf( x_{k+1},y_{k+1}) -f( x_{k+1},y_k )\\ge \\frac{\\nu}{2}\\| y_{k+1}-y_k \\|^2.\n\t\t\\end{equation}\n\tThe proof is completed by combining \\eqref{lem:4.2:9} with \\eqref{lem:4.1:1:conclu}.\n\\end{proof}","start":97615,"end":100813}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":["lem:4.2:2:opty","lem:4.2:3:nopty","lem:4.2:4:opty:k"],"declared_external":false,"source_only":false,"title":"原变量两次投影最优性","source_units":[{"raw_tex":"\\begin{proof}\n\tSimilar to \\eqref{lem:3.2:2}-\\eqref{lem:3.2:4} in the proof of Lemma \\ref{lem:3.2:blostr:y}, by the optimality condition for $x_k$ in \\eqref{subsec:4.1:2:x} implies that $\\forall x\\in \\mathcal{X}$ and $\\forall k\\geq 1$,\n\t\\begin{align}\n\t\t\\langle \\nabla _xf(x_{k},y_{k})+\\frac{1}{\\zeta}(x_{k+1}-x_{k}),x-x_{k+1} \\rangle &\\ge 0,\\label{lem:4.2:2:opty}\\\\\n\t\t\\langle \\nabla _xf(x_{k},y_{k})+\\frac{1}{\\zeta}(x_{k+1}-x_{k}) ,x_{k}-x_{k+1} \\rangle &\\ge 0,\\label{lem:4.2:3:nopty}\\\\\n\t\t\\langle \\nabla _xf(x_{k-1},y_{k-1})+\\frac{1}{\\zeta}(x_{k}-x_{k-1}),x_{k+1}-x_{k} \\rangle &\\ge 0,\\label{lem:4.2:4:opty:k}\n\t\\end{align}\n","start":97615,"end":98237}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":["lem:4.2:5"],"declared_external":false,"source_only":false,"title":"强凸性控制原变量函数变化","source_units":[{"raw_tex":"which, in view of the fact that $f\\left( x,y  \\right)$ is $\\theta $-strongly convex w.r.t. $x$ for any given $y\\in \\mathcal{Y}$,\n\tthen implies that\n\t\\begin{align}\\label{lem:4.2:5}\n\t\t&f( x_{k+1},y_{k}) -f( x_{k},y_{k} ) \\nonumber\\\\\n\t\t\\geq &  \\langle  \\nabla _xf( x_{k},y_{k} ),x_{k+1}-x_{k}\\rangle  +\\frac{\\theta}{2}\\| x_{k+1}-x_{k} \\|^2\\  \\nonumber\\\\\n\t\t\\geq &\\langle  \\nabla _xf( x_{k},y_{k} ) -\\nabla _xf( x_{k-1},y_{k-1}) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t& -\\frac{1}{\\zeta}\\langle  x_{k}-x_{k-1},x_{k+1}-x_{k} \\rangle +\\frac{\\theta}{2}\\| x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\n","start":98237,"end":98815}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"原变量梯度差三项分解","source_units":[{"raw_tex":"\tDenoting $m_{k+1}:=\\left(x_{k+1}-x_k\\right)-\\left(x_k-x_{k-1}\\right)$, by Assumption \\ref{sec:2:ass1:lip}, the Cauchy-Schwarz inequality and\n\tthe $\\theta $-strongly convexity  of $f$ w.r.t. $x$, we can estimate the first inner product term in the r.h.s.\n\tof \\eqref{lem:4.2:5} as\n\t\\begin{align}\n\t\t&\\langle  \\nabla _x f ( x_{k},y_{k} ) -\\nabla _x f ( x_{k-1},y_{k-1}) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t=&\\langle  \\nabla _x f ( x_{k},y_{k} ) -\\nabla _x f \\left( x_{k},y_{k-1} \\right) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t& + \\langle  \\nabla _x f \\left( x_{k},y_{k-1} \\right) -\\nabla _x f ( x_{k-1},y_{k-1}) ,m_{k+1} \\rangle  \\nonumber\\\\\n\t\t&+\\langle  \\nabla _x f \\left( x_{k},y_{k-1}\\right) -\\nabla _x f ( x_{k-1},y_{k-1}) ,x_{k}-x_{k-1} \\rangle \\label{lem:4.2:6}\\\\\n","start":98815,"end":99583}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"梯度差内积下界","source_units":[{"raw_tex":"\t\\geq &-\\frac{L_{21}^{2}\\zeta}{2}\\| y_{k}-y_{k-1} \\|^2-\\frac{1}{2\\zeta}\\| x_{k+1}-x_k \\|^2-\\frac{\\zeta L_{11}^{2}}{2}\\|x_k-x_{k-1} \\|^2\\nonumber\\\\\n\t&-\\frac{1}{2\\zeta}\\|m_{k+1} \\|^2 +\t\\theta \\| x_k-x_{k-1} \\|^2.\\label{lem:4.2:7}\n\t\\end{align}\n","start":99583,"end":99824}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":["lem:4.2:8:3points"],"declared_external":false,"source_only":false,"title":"原变量三点内积恒等式","source_units":[{"raw_tex":"\tMoreover, it can be easily checked that\n\\begin{align}\\label{lem:4.2:8:3points}\n\t\t&\\langle  x_k-x_{k-1},x_{k+1}-x_k \\rangle =\\frac{1}{2}\\| x_k-x_{k-1} \\|^2+\\frac{1}{2}\\| x_{k+1}-x_k \\|^2-\\frac{1}{2}\\| m_{k+1} \\|^2.\n\t\\end{align}\n","start":99824,"end":100052}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":["lem:4.2:9"],"declared_external":false,"source_only":false,"title":"合成强凸原变量变化界","source_units":[{"raw_tex":"\tPlugging \\eqref{lem:4.2:7} and \\eqref{lem:4.2:8:3points} into \\eqref{lem:4.2:5} and rearranging the terms, we conclude that\n\t\\begin{align}\\label{lem:4.2:9}\n\tf( x_{k+1},y_{k}) -f( x_{k},y_{k})\t\\geq& -\\frac{L_{21}^{2}\\zeta}{2}\\| y_{k}-y_{k-1} \\|^2+(\\theta-\\frac{1}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2})\\| x_{k}-x_{k-1}\\|^2\\nonumber\\\\\n\t\t&+ (\\frac{\\theta}{2}-\\frac{1}{\\zeta})\\|x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\n","start":100052,"end":100455}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["lem:4.1:1:conclu"],"declared_external":false,"source_only":false,"title":"特化通用对偶上升界","source_units":[{"raw_tex":"By setting $\\gamma_k=\\nu$, $c_k=0$ in \\eqref{lem:2.1:2} of Lemma \\ref{lem:2.1}  and the assumption $\\nu > L_{22}$, we have\n\t\t\\begin{equation}\\label{lem:4.1:1:conclu}\n\t\t\tf( x_{k+1},y_{k+1}) -f( x_{k+1},y_k )\\ge \\frac{\\nu}{2}\\| y_{k+1}-y_k \\|^2.\n\t\t\\end{equation}\n","start":100455,"end":100716}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"完成强凸函数变化递推","source_units":[{"raw_tex":"\tThe proof is completed by combining \\eqref{lem:4.2:9} with \\eqref{lem:4.1:1:conclu}.\n\\end{proof}","start":100716,"end":100813}]},{"id":"S051","semantic_type":"definition","kind":"definition","labels":["lem:4.3"],"declared_external":false,"source_only":false,"title":"强凸情形势函数","source_units":[{"raw_tex":"\t$$\\hat{f}_{k+1}:=f\\left( x_{k+1},y_{k+1} \\right),\\ \\hat{S}_{k+1}:=-\\frac{2}{\\zeta^2\\theta}\\lVert x_{k+1}-x_k \\rVert^2,$$\n\t$$\\hat{F}_{k+1}:=\\hat{f}_{k+1}+\\hat{S}_{k+1}+ (\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\lVert x_{k+1}-x_{k} \\rVert^2-(\\frac{L_{21}^2\\zeta}{2}+\\frac{2L_{21}^2}{\\theta^2\\zeta})\\|y_{k+1}-y_k\\|^2.$$\n","start":101468,"end":101820}]},{"id":"S052","semantic_type":"lemma","kind":"lemma","labels":["lem:4.3","lem:4.3:2:conclus"],"declared_external":false,"source_only":false,"title":"势函数增量控制位移","source_units":[{"raw_tex":"\\begin{lem}\\label{lem:4.3}\n\tSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{\\left(x_k,y_k\\right)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}\n\twith parameter settings in \\eqref{subsec:4.1:1:para}. Denote\n\t$$\\hat{f}_{k+1}:=f\\left( x_{k+1},y_{k+1} \\right),\\ \\hat{S}_{k+1}:=-\\frac{2}{\\zeta^2\\theta}\\lVert x_{k+1}-x_k \\rVert^2,$$\n\t$$\\hat{F}_{k+1}:=\\hat{f}_{k+1}+\\hat{S}_{k+1}+ (\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\lVert x_{k+1}-x_{k} \\rVert^2-(\\frac{L_{21}^2\\zeta}{2}+\\frac{2L_{21}^2}{\\theta^2\\zeta})\\|y_{k+1}-y_k\\|^2.$$\n\tIf $\\nu>L_{22}$,\n%\t\\begin{equation}\\label{lem:4.3:1}\n%, \\nu>L_{21}^2\\zeta+\\frac{4L_{21}^2}{\\zeta \\theta^2},~\\zeta\\leq \\frac{\\theta}{4L_{11}^2},\n%\t\\end{equation}\n\tthen $\\forall k \\geq 1$,\n\t\\begin{align}\\label{lem:4.3:2:conclus}\n\t\t\\hat{F}_{k+1}-\\hat{F}_k\\geq& \\left(\\frac{\\nu}{2}-\\frac{\\zeta L_{21}^2}{2}-\\frac{2L_{21}^2}{\\zeta \\theta^2}\\right)\\lVert y_{k+1}-y_k \\rVert ^2\\nonumber\\\\\n\t\t& +\\left(\\frac{3\\theta-\\zeta L_{11}^2}{2}+\\frac{\\theta-4\\zeta L_{11}^2}{2\\zeta\\theta}\\right)\\lVert x_{k+1}-x_k \\rVert ^2.\n\t\\end{align}\n\\end{lem}","start":101244,"end":102349}]},{"id":"S053","semantic_type":"proof","kind":"proof","labels":["lem:4.3:3","lem:4.3:4","lem:4.3:5","lem:4.3:6","lem:4.3:7"],"declared_external":false,"source_only":false,"title":"势函数增量引理完整证明","source_units":[{"raw_tex":"\\begin{proof}\nFirst by \\eqref{lem:4.2:3:nopty} and \\eqref{lem:4.2:4:opty:k}, we have\n\t\\begin{align}\\label{lem:4.3:3}\n\t\t\\frac{1}{\\zeta}\\langle m_{k+1}, x_k -x_{k+1} \\rangle\n\t\t\\ge \\langle \\nabla _xf( x_k,y_k ) -\\nabla _xf\\left( x_{k-1},y_{k-1} \\right),x_{k+1}-x_k \\rangle,\n\t\\end{align}\nwhich together with \\eqref{lem:4.2:6} then imply that\n%\t\\eqref{lem:4.3:3} then implies that\n\t\\begin{align}\n\t\t\\frac{1}{\\zeta}\\langle m_{k+1},x_k-x_{k+1}  \\rangle  \\geq&\\langle  \\nabla _xf( x_{k},y_{k} ) -\\nabla _xf\\left( x_{k},y_{k-1} \\right) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t& + \\langle  \\nabla _xf\\left( x_{k},y_{k-1} \\right) -\\nabla _xf\\left( x_{k-1},y_{k-1} \\right) ,m_{k+1} \\rangle  \\nonumber\\\\\n\t\t&+\\langle  \\nabla _xf\\left( x_k,y_{k-1} \\right) -\\nabla _xf\\left( x_{k-1},y_{k-1} \\right) ,x_{k}-x_{k-1} \\rangle \\label{lem:4.3:4}\\\\\n\t\t\\geq& -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_{k} \\|^2-\\frac{\\zeta L_{11}^2}{2}\\|x_{k}-x_{k-1} \\|^2\\nonumber\\\\\n\t\t&-\\frac{1}{2\\zeta}\\| m_{k+1}\\|^2+\\theta\\|x_{k}-x_{k-1} \\|^2,\\label{lem:4.3:5}\n\t\\end{align}\nwhere the second inequality is similar to the proof of \\eqref{lem:4.2:7} except that for the first term in the r.h.s., we use $\\langle  \\nabla _xf( x_{k},y_{k} ) -\\nabla _xf\\left( x_{k},y_{k-1} \\right) ,x_{k+1}-x_{k} \\rangle \\geq -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_k \\|^2$. By using the identity $\\frac{1}{\\zeta}\\langle m_{k+1},x_{k}-x_{k+1} \\rangle  =\\frac{1}{2\\zeta}\\|x_{k}-x_{k-1}\\|^2- \\frac{1}{2\\zeta}\\|x_{k+1}-x_{k}\\|^2-\\frac{1}{2\\zeta}\\| m_{k+1}\\| ^2$, we conclude from \\eqref{lem:4.3:5} that\n\t\\begin{align}\\label{lem:4.3:6}\n\t\t&\\frac{1}{2\\zeta}\\|x_{k}-x_{k-1}\\|^2- \\frac{1}{2\\zeta}\\|x_{k+1}-x_{k}\\|^2-\\frac{1}{2\\zeta}\\| m_{k+1}\\| ^2\\nonumber\\\\\n\t\t\\geq& -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_k \\|^2-\\frac{\\zeta L_{11}^2}{2}\\|x_{k}-x_{k-1} \\|^2\\nonumber\\\\\n\t\t&-\\frac{1}{2\\zeta}\\| m_{k+1}\\|^2+\\theta\\|x_{k}-x_{k-1} \\|^2.\n\t\\end{align}\n\tRearranging the terms of \\eqref{lem:4.3:6}, we have\n\t\\begin{align}\\label{lem:4.3:7}\n\t\t&\\frac{1}{2\\zeta}\\|x_{k}-x_{k-1}\\|^2- \\frac{1}{2\\zeta}\\|x_{k+1}-x_{k}\\|^2\\nonumber\\\\\n\t\t\\geq& -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_{k} \\|^2+\\left(\\theta-\\frac{\\zeta L_{11}^2}{2}\\right)\\|x_{k}-x_{k-1} \\|^2.\n\t\\end{align}\n\tMultiplying $\\frac{4}{\\zeta\\theta}$ on both sides of \\eqref{lem:4.3:7} and using the definition of $\\hat{S}_{k+1}$, we obtain\n\t\\begin{align*}\n\t\t\\hat{S}_{k+1}-\\hat{S}_k \\geq & -\\frac{2L_{21}^{2}}{\\theta^2\\zeta}\\| y_{k}-y_{k-1}\\|^2-\\frac{2}{\\zeta}\\| x_{k+1}-x_k \\|^2+\\left(\\frac{4}{\\zeta}-\\frac{2 L_{11}^2}{\\theta}\\right)\\|x_{k}-x_{k-1} \\|^2.\n\t\\end{align*}\n\tThe proof is completed by \\eqref{lem:4.2:1:conclu} in Lemma \\ref{lem:4.2} and the definition of $\\hat{F}_k$.\n\\end{proof}","start":102351,"end":105155}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":["lem:4.3:3"],"declared_external":false,"source_only":false,"title":"合并相邻原变量最优性","source_units":[{"raw_tex":"\\begin{proof}\nFirst by \\eqref{lem:4.2:3:nopty} and \\eqref{lem:4.2:4:opty:k}, we have\n\t\\begin{align}\\label{lem:4.3:3}\n\t\t\\frac{1}{\\zeta}\\langle m_{k+1}, x_k -x_{k+1} \\rangle\n\t\t\\ge \\langle \\nabla _xf( x_k,y_k ) -\\nabla _xf\\left( x_{k-1},y_{k-1} \\right),x_{k+1}-x_k \\rangle,\n\t\\end{align}\n","start":102351,"end":102635}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["lem:4.3:4","lem:4.3:5"],"declared_external":false,"source_only":false,"title":"重新加权梯度差下界","source_units":[{"raw_tex":"which together with \\eqref{lem:4.2:6} then imply that\n%\t\\eqref{lem:4.3:3} then implies that\n\t\\begin{align}\n\t\t\\frac{1}{\\zeta}\\langle m_{k+1},x_k-x_{k+1}  \\rangle  \\geq&\\langle  \\nabla _xf( x_{k},y_{k} ) -\\nabla _xf\\left( x_{k},y_{k-1} \\right) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t& + \\langle  \\nabla _xf\\left( x_{k},y_{k-1} \\right) -\\nabla _xf\\left( x_{k-1},y_{k-1} \\right) ,m_{k+1} \\rangle  \\nonumber\\\\\n\t\t&+\\langle  \\nabla _xf\\left( x_k,y_{k-1} \\right) -\\nabla _xf\\left( x_{k-1},y_{k-1} \\right) ,x_{k}-x_{k-1} \\rangle \\label{lem:4.3:4}\\\\\n\t\t\\geq& -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_{k} \\|^2-\\frac{\\zeta L_{11}^2}{2}\\|x_{k}-x_{k-1} \\|^2\\nonumber\\\\\n\t\t&-\\frac{1}{2\\zeta}\\| m_{k+1}\\|^2+\\theta\\|x_{k}-x_{k-1} \\|^2,\\label{lem:4.3:5}\n\t\\end{align}\nwhere the second inequality is similar to the proof of \\eqref{lem:4.2:7} except that for the first term in the r.h.s., we use $\\langle  \\nabla _xf( x_{k},y_{k} ) -\\nabla _xf\\left( x_{k},y_{k-1} \\right) ,x_{k+1}-x_{k} \\rangle \\geq -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_k \\|^2$. By using the identity $\\frac{1}{\\zeta}\\langle m_{k+1},x_{k}-x_{k+1} \\rangle  =\\frac{1}{2\\zeta}\\|x_{k}-x_{k-1}\\|^2- \\frac{1}{2\\zeta}\\|x_{k+1}-x_{k}\\|^2-\\frac{1}{2\\zeta}\\| m_{k+1}\\| ^2$, we conclude from \\eqref{lem:4.3:5} that\n","start":102635,"end":103958}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":["lem:4.3:6"],"declared_external":false,"source_only":false,"title":"代入原变量范数恒等式","source_units":[{"raw_tex":"where the second inequality is similar to the proof of \\eqref{lem:4.2:7} except that for the first term in the r.h.s., we use $\\langle  \\nabla _xf( x_{k},y_{k} ) -\\nabla _xf\\left( x_{k},y_{k-1} \\right) ,x_{k+1}-x_{k} \\rangle \\geq -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_k \\|^2$. By using the identity $\\frac{1}{\\zeta}\\langle m_{k+1},x_{k}-x_{k+1} \\rangle  =\\frac{1}{2\\zeta}\\|x_{k}-x_{k-1}\\|^2- \\frac{1}{2\\zeta}\\|x_{k+1}-x_{k}\\|^2-\\frac{1}{2\\zeta}\\| m_{k+1}\\| ^2$, we conclude from \\eqref{lem:4.3:5} that\n\t\\begin{align}\\label{lem:4.3:6}\n\t\t&\\frac{1}{2\\zeta}\\|x_{k}-x_{k-1}\\|^2- \\frac{1}{2\\zeta}\\|x_{k+1}-x_{k}\\|^2-\\frac{1}{2\\zeta}\\| m_{k+1}\\| ^2\\nonumber\\\\\n\t\t\\geq& -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_k \\|^2-\\frac{\\zeta L_{11}^2}{2}\\|x_{k}-x_{k-1} \\|^2\\nonumber\\\\\n\t\t&-\\frac{1}{2\\zeta}\\| m_{k+1}\\|^2+\\theta\\|x_{k}-x_{k-1} \\|^2.\n\t\\end{align}\n","start":103417,"end":104333}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":["lem:4.3:7"],"declared_external":false,"source_only":false,"title":"整理相邻原变量位移差","source_units":[{"raw_tex":"\tRearranging the terms of \\eqref{lem:4.3:6}, we have\n\t\\begin{align}\\label{lem:4.3:7}\n\t\t&\\frac{1}{2\\zeta}\\|x_{k}-x_{k-1}\\|^2- \\frac{1}{2\\zeta}\\|x_{k+1}-x_{k}\\|^2\\nonumber\\\\\n\t\t\\geq& -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_{k} \\|^2+\\left(\\theta-\\frac{\\zeta L_{11}^2}{2}\\right)\\|x_{k}-x_{k-1} \\|^2.\n\t\\end{align}\n","start":104333,"end":104678}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"负位移项的增量下界","source_units":[{"raw_tex":"\tMultiplying $\\frac{4}{\\zeta\\theta}$ on both sides of \\eqref{lem:4.3:7} and using the definition of $\\hat{S}_{k+1}$, we obtain\n\t\\begin{align*}\n\t\t\\hat{S}_{k+1}-\\hat{S}_k \\geq & -\\frac{2L_{21}^{2}}{\\theta^2\\zeta}\\| y_{k}-y_{k-1}\\|^2-\\frac{2}{\\zeta}\\| x_{k+1}-x_k \\|^2+\\left(\\frac{4}{\\zeta}-\\frac{2 L_{11}^2}{\\theta}\\right)\\|x_{k}-x_{k-1} \\|^2.\n\t\\end{align*}\n","start":104678,"end":105034}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"合成势函数增量","source_units":[{"raw_tex":"\tThe proof is completed by \\eqref{lem:4.2:1:conclu} in Lemma \\ref{lem:4.2} and the definition of $\\hat{F}_k$.\n\\end{proof}","start":105034,"end":105155}]},{"id":"S060","semantic_type":"theorem","kind":"theorem","labels":["thm:4.1"],"declared_external":false,"source_only":false,"title":"强凸非凹 AGP 驻点复杂度","source_units":[{"raw_tex":"\\begin{thm}\\label{thm:4.1}\n\tSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{\\left(x_{k},y_{k}\\right)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}\n\twith parameter settings in \\eqref{subsec:4.1:1:para}.\n\tIf the relations $\\nu>L_{22}, \\nu>L_{21}^2\\zeta+\\frac{4L_{21}^2}{\\zeta \\theta^2},~\\zeta\\leq \\frac{\\theta}{4L_{11}^2}$ are satisfied, then $\\forall \\varepsilon>0$,  it holds that $$ T\\left( \\varepsilon \\right) \\le\n\t\\tfrac{\\overline{F}-\\hat{F}_1}{\\hat{d}_1\\varepsilon ^2},$$\n\twhere\n\t$\\hat{d}_1:=\\tfrac{\\min \\left\\{\\tfrac{\\nu}{2}-\\tfrac{\\zeta L_{21}^2}{2}-\\tfrac{2 L_{21}^2}{\\zeta \\theta^2},\\tfrac{3\\theta-\\zeta L_{11}^2}{2}+\\tfrac{\\theta-4\\zeta L_{11}^2}{2\\zeta\\theta} \\right\\}}{\\max \\left\\{ \\tfrac{1}{\\zeta ^2}+2L_{12}^{2},2\\nu^2 \\right\\}}%\\overline{\\hat{F}}=\\overline{f}-(\\tfrac{\\theta}{2}-\\tfrac{3}{\\zeta})\\sigma _x^2\n\t$\n\tand $\\overline{F}=\\bar{f}+(\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\sigma_x^2$ with $\\overline{f} := \\max_{(x,y) \\in \\mathcal{X} \\times \\mathcal{Y}} f(x,y)$ and $\\sigma _x= \\max\\{\\|x_1-x_2\\| \\mid \\forall x_1, x_2\\in \\mathcal{X}\\}$.\n\\end{thm}","start":105682,"end":106802}]},{"id":"S061","semantic_type":"proof","kind":"proof","labels":["thm:4.1:1:gx","thm:4.1:2:gy","thm:4.1:3:g","thm:4.1:4:d1","thm:4.1:5"],"declared_external":false,"source_only":false,"title":"主定理完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\tBy \\eqref{subsec:4.1:2:x}, we immediately obtain\n\t\\begin{equation}\n\t\t\\|(\\nabla G_k)_{x} \\|\\leq\\frac{1}{\\zeta}\\| x_{k+1}-x_{k} \\|.\\label{thm:4.1:1:gx}\n\t\\end{equation}\n\tOn the other hand, similar to the proof of \\eqref{thm:3.1:2:gy},  by \\eqref{subsec:4.1:3:y} and the triangle inequality, and the nonexpansiveness of the projection operator $\\operatorname{ \\mathcal{P}_\\mathcal{Y}}$, we conclude that\n\t\\begin{align}\\label{thm:4.1:2:gy}\n\t\t\\| (\\nabla G_k)_{y} \\|\\le& \\nu\\| y_{k+1}-y_k \\|+L_{12}\\|x_{k+1}-x_{k} \\|.\n\t\t\t\\end{align}\n%\t\t\\leq&\\nu\\|\\operatorname{ Prox_{\\mathcal{Y}_j}^{\\nu}}\\left( y_j^r+\\frac{1}{\\nu} \\nabla _{y_j}f\\left( x^{r+1} ,w_j^{r+1} \\right) \\right)-\\operatorname{ Prox_{\\mathcal{Y}_j}^{\\nu}}\\left( y_j^r+ \\frac{1}{\\nu} \\nabla _{y_j}f\\left( x^r,y^r \\right) \\right) \\|\\nonumber\\\\\n%\t\t&+\\nu\\| y_j^{r+1}-y_j^r \\|\\nonumber\\\\\n%\t\t\\leq& \\nu\\| y_j^{r+1}-y_j^r \\| +\\|\\nabla _{y_j}f\\left( x^{r+1} ,w_j^{r+1} \\right)-\\nabla _{y_j}f\\left( x^r,y^r \\right) \\|\\nonumber\\\\\n%\t\t\\leq & \\nu\\| y_j^{r+1}-y_j^r \\| +\\|\\nabla _{y_j}f\\left( x^{r+1} ,w_j^{r+1} \\right)-\\nabla _{y_j}f\\left( x^r,w_j^{r+1} \\right) \\|\\nonumber\\\\\n%\t\t& +\\|\\nabla _{y_j}f\\left( x^r ,w_j^{r+1} \\right)-\\nabla _{y_j}f\\left( x^r,y^r \\right) \\|\\nonumber\\\\\n%\t\t\\le& \\nu\\| y_j^{r+1}-y_j^r \\|+L_{12}\\|x^{r+1}-x^r \\|+L_{22}\\|w_j^{r+1}-y^r \\|\\nonumber\\\\\n%\twhere the second inequality holds by the nonexpansiveness of the proximal operator $\\operatorname{ Prox_{\\mathcal{Y}_j}^{\\nu}}$, the last inequality holds since that $\\| y_i^{r+1}-y_i^r \\|\\le\\| y^{r+1}-y^r \\|$ and $\\|w_j^{r+1}-y^r \\|\\le\\| y^{r+1}-y^r \\|$ by the definition of $y_j$ and $w_j^{r+1}$.\n\tBy combining \\eqref{thm:4.1:1:gx} and \\eqref{thm:4.1:2:gy}, and using the Cauchy-Schwarz inequality, we obtain\n\t\\begin{align}\\label{thm:4.1:3:g}\n\t\t\\| \\nabla G_k \\|^2\n\t\t\\leq 2\\nu^2\\| y_{k+1}-y_{k} \\|^2+\\left( \\frac{1}{\\zeta ^2}+2L_{12}^{2} \\right)\\|x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\n\tObserving that $\\hat{d}_1 > 0$. Multiplying both sides of \\eqref{thm:4.1:3:g} by $\\hat{d}_1$, and using \\eqref{lem:4.3:2:conclus} in Lemma \\ref{lem:4.3},\n\twe have\n\t\\begin{align}\\label{thm:4.1:4:d1}\n\t\t\\hat{d}_1\\|\\nabla G_k \\|^2\\le \\hat{F}_{k+1}-\\hat{F}_k.\n\t\\end{align}\n\tSumming up the above inequalities from $k=1$ to $k=T(\\varepsilon)$, we obtain\n\t\\begin{align}\\label{thm:4.1:5}\n\t\t\\sum_{k=1}^{T\\left( \\varepsilon \\right)}{\\hat{d}_1\\|\\nabla G_k \\| ^2}\\le \\hat{F}_{T\\left( \\varepsilon \\right)+1}-\\hat{F}_1.\n\t\\end{align}\n\tNote that by the definition of $\\hat{F}_{k+1}$ in Lemma \\ref{lem:4.3}, we have\n\t\\begin{align*}\n\t\t\\hat{F}_{T\\left( \\varepsilon \\right)+1}&\\le\n\t\t\\hat{f}_{T\\left( \\varepsilon \\right)+1}+(\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\|x_{T\\left( \\varepsilon \\right)+1}-x_{T\\left( \\varepsilon \\right)} \\|^2\\\\\n\t\t&\\le \\bar{f}+(\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\sigma_x^2 = \\bar{F},\n\t\\end{align*}\n\twhere the last inequality follows from the definitions of $\\bar{f}$ and  $\\sigma _x$, and $\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta} \\ge 0 $ due to\n\tthe selection of $\\zeta$.\n\tWe then conclude from \\eqref{thm:4.1:5} that $\\sum_{k=1}^{T\\left( \\varepsilon \\right)}{\\hat{d}_1\\| \\nabla G_k \\|^2}\\le \\bar{F}-\\hat{F}_1$ which, in view of the definition of $T(\\varepsilon)$, implies that $\\varepsilon ^2\\le (\\bar{F}-\\hat{F}_1)/(T( \\varepsilon )\\cdot \\hat{d}_1)$\n\tor equivalently, $T\\left( \\varepsilon \\right) \\le (\\bar{F}-\\hat{F}_1)/(\\hat{d}_1\\varepsilon ^2)$.\n\\end{proof}","start":106804,"end":110287}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":["thm:4.1:1:gx"],"declared_external":false,"source_only":false,"title":"原变量梯度映射位移界","source_units":[{"raw_tex":"\\begin{proof}\n\tBy \\eqref{subsec:4.1:2:x}, we immediately obtain\n\t\\begin{equation}\n\t\t\\|(\\nabla G_k)_{x} \\|\\leq\\frac{1}{\\zeta}\\| x_{k+1}-x_{k} \\|.\\label{thm:4.1:1:gx}\n\t\\end{equation}\n","start":106804,"end":106985}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":["thm:4.1:2:gy"],"declared_external":false,"source_only":false,"title":"对偶梯度映射位移界","source_units":[{"raw_tex":"\tOn the other hand, similar to the proof of \\eqref{thm:3.1:2:gy},  by \\eqref{subsec:4.1:3:y} and the triangle inequality, and the nonexpansiveness of the projection operator $\\operatorname{ \\mathcal{P}_\\mathcal{Y}}$, we conclude that\n\t\\begin{align}\\label{thm:4.1:2:gy}\n\t\t\\| (\\nabla G_k)_{y} \\|\\le& \\nu\\| y_{k+1}-y_k \\|+L_{12}\\|x_{k+1}-x_{k} \\|.\n\t\t\t\\end{align}\n","start":106985,"end":107345}]},{"id":"S064","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"非活动分块推导注释","source_units":[{"raw_tex":"%\t\t\\leq&\\nu\\|\\operatorname{ Prox_{\\mathcal{Y}_j}^{\\nu}}\\left( y_j^r+\\frac{1}{\\nu} \\nabla _{y_j}f\\left( x^{r+1} ,w_j^{r+1} \\right) \\right)-\\operatorname{ Prox_{\\mathcal{Y}_j}^{\\nu}}\\left( y_j^r+ \\frac{1}{\\nu} \\nabla _{y_j}f\\left( x^r,y^r \\right) \\right) \\|\\nonumber\\\\\n%\t\t&+\\nu\\| y_j^{r+1}-y_j^r \\|\\nonumber\\\\\n%\t\t\\leq& \\nu\\| y_j^{r+1}-y_j^r \\| +\\|\\nabla _{y_j}f\\left( x^{r+1} ,w_j^{r+1} \\right)-\\nabla _{y_j}f\\left( x^r,y^r \\right) \\|\\nonumber\\\\\n%\t\t\\leq & \\nu\\| y_j^{r+1}-y_j^r \\| +\\|\\nabla _{y_j}f\\left( x^{r+1} ,w_j^{r+1} \\right)-\\nabla _{y_j}f\\left( x^r,w_j^{r+1} \\right) \\|\\nonumber\\\\\n%\t\t& +\\|\\nabla _{y_j}f\\left( x^r ,w_j^{r+1} \\right)-\\nabla _{y_j}f\\left( x^r,y^r \\right) \\|\\nonumber\\\\\n%\t\t\\le& \\nu\\| y_j^{r+1}-y_j^r \\|+L_{12}\\|x^{r+1}-x^r \\|+L_{22}\\|w_j^{r+1}-y^r \\|\\nonumber\\\\\n%\twhere the second inequality holds by the nonexpansiveness of the proximal operator $\\operatorname{ Prox_{\\mathcal{Y}_j}^{\\nu}}$, the last inequality holds since that $\\| y_i^{r+1}-y_i^r \\|\\le\\| y^{r+1}-y^r \\|$ and $\\|w_j^{r+1}-y^r \\|\\le\\| y^{r+1}-y^r \\|$ by the definition of $y_j$ and $w_j^{r+1}$.\n","start":107345,"end":108428}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":["thm:4.1:3:g"],"declared_external":false,"source_only":false,"title":"完整梯度映射平方界","source_units":[{"raw_tex":"\tBy combining \\eqref{thm:4.1:1:gx} and \\eqref{thm:4.1:2:gy}, and using the Cauchy-Schwarz inequality, we obtain\n\t\\begin{align}\\label{thm:4.1:3:g}\n\t\t\\| \\nabla G_k \\|^2\n\t\t\\leq 2\\nu^2\\| y_{k+1}-y_{k} \\|^2+\\left( \\frac{1}{\\zeta ^2}+2L_{12}^{2} \\right)\\|x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\n","start":108428,"end":108710}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":["thm:4.1:4:d1"],"declared_external":false,"source_only":false,"title":"势增量控制驻点间隙","source_units":[{"raw_tex":"\tObserving that $\\hat{d}_1 > 0$. Multiplying both sides of \\eqref{thm:4.1:3:g} by $\\hat{d}_1$, and using \\eqref{lem:4.3:2:conclus} in Lemma \\ref{lem:4.3},\n\twe have\n\t\\begin{align}\\label{thm:4.1:4:d1}\n\t\t\\hat{d}_1\\|\\nabla G_k \\|^2\\le \\hat{F}_{k+1}-\\hat{F}_k.\n\t\\end{align}\n","start":108710,"end":108979}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":["thm:4.1:5"],"declared_external":false,"source_only":false,"title":"驻点间隙的望远镜求和","source_units":[{"raw_tex":"\tSumming up the above inequalities from $k=1$ to $k=T(\\varepsilon)$, we obtain\n\t\\begin{align}\\label{thm:4.1:5}\n\t\t\\sum_{k=1}^{T\\left( \\varepsilon \\right)}{\\hat{d}_1\\|\\nabla G_k \\| ^2}\\le \\hat{F}_{T\\left( \\varepsilon \\right)+1}-\\hat{F}_1.\n\t\\end{align}\n","start":108979,"end":109229}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"末端势函数的统一上界","source_units":[{"raw_tex":"\tNote that by the definition of $\\hat{F}_{k+1}$ in Lemma \\ref{lem:4.3}, we have\n\t\\begin{align*}\n\t\t\\hat{F}_{T\\left( \\varepsilon \\right)+1}&\\le\n\t\t\\hat{f}_{T\\left( \\varepsilon \\right)+1}+(\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\|x_{T\\left( \\varepsilon \\right)+1}-x_{T\\left( \\varepsilon \\right)} \\|^2\\\\\n\t\t&\\le \\bar{f}+(\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\sigma_x^2 = \\bar{F},\n\t\\end{align*}\n\twhere the last inequality follows from the definitions of $\\bar{f}$ and  $\\sigma _x$, and $\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta} \\ge 0 $ due to\n\tthe selection of $\\zeta$.\n","start":109229,"end":109897}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"作者的首次命中时间结论","source_units":[{"raw_tex":"\tWe then conclude from \\eqref{thm:4.1:5} that $\\sum_{k=1}^{T\\left( \\varepsilon \\right)}{\\hat{d}_1\\| \\nabla G_k \\|^2}\\le \\bar{F}-\\hat{F}_1$ which, in view of the definition of $T(\\varepsilon)$, implies that $\\varepsilon ^2\\le (\\bar{F}-\\hat{F}_1)/(T( \\varepsilon )\\cdot \\hat{d}_1)$\n\tor equivalently, $T\\left( \\varepsilon \\right) \\le (\\bar{F}-\\hat{F}_1)/(\\hat{d}_1\\varepsilon ^2)$.\n\\end{proof}","start":109897,"end":110287}]},{"id":"S070","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":true,"title":"作者的梯度复杂度解释","source_units":[{"raw_tex":"Theorem \\ref{thm:4.1} shows that the number of gradient evaluations performed by Algorithm \\ref{alg:2.1} to obtain an $\\varepsilon$-stationary point of $f$ is bounded by $\\mathcal{O}\\left(L^2 \\varepsilon ^{-2} \\right)$ under the strongly convex-nonconcave setting. To the best of our knowledge, this is the first theoretical guarantee that has been obtained in the literature for solving this class of minimax problems.\n","start":110289,"end":110709}]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第1项：Optimality characterization and nonexpansiveness of Euclidean projection onto a nonempty closed convex set."],"source_only":false,"title":"欧氏投影最优性刻画","source_units":[]},{"id":"S072","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第1项：Optimality characterization and nonexpansiveness of Euclidean projection onto a nonempty closed convex set."],"source_only":false,"title":"欧氏投影非扩张性","source_units":[]},{"id":"S073","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第2项：Smooth descent inequalities and strong monotonicity of gradients of strongly convex functions."],"source_only":false,"title":"光滑函数二次变化界","source_units":[]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第2项：Smooth descent inequalities and strong monotonicity of gradients of strongly convex functions."],"source_only":false,"title":"强凸梯度的强单调性","source_units":[]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第3项：Cauchy–Schwarz, Young's inequality, triangle inequality, and elementary telescoping identities."],"source_only":false,"title":"基本内积与求和工具","source_units":[]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第4项：Existence of extrema of continuous functions on compact sets."],"source_only":false,"title":"紧集上连续函数取得极值","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S060","edges":[{"source_id":"S001","target_id":"S007","kind":"supports"},{"source_id":"S002","target_id":"S007","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S013","kind":"supports"},{"source_id":"S009","target_id":"S013","kind":"supports"},{"source_id":"S003","target_id":"S014","kind":"supports"},{"source_id":"S005","target_id":"S015","kind":"supports"},{"source_id":"S006","target_id":"S015","kind":"supports"},{"source_id":"S007","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S007","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S007","target_id":"S023","kind":"supports"},{"source_id":"S071","target_id":"S023","kind":"supports"},{"source_id":"S010","target_id":"S024","kind":"supports"},{"source_id":"S001","target_id":"S024","kind":"supports"},{"source_id":"S073","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S075","target_id":"S025","kind":"supports"},{"source_id":"S007","target_id":"S026","kind":"supports"},{"source_id":"S071","target_id":"S026","kind":"supports"},{"source_id":"S010","target_id":"S027","kind":"supports"},{"source_id":"S001","target_id":"S027","kind":"supports"},{"source_id":"S073","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S075","target_id":"S028","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S022","kind":"supports"},{"source_id":"S026","target_id":"S022","kind":"supports"},{"source_id":"S027","target_id":"S022","kind":"supports"},{"source_id":"S028","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S012","target_id":"S031","kind":"supports"},{"source_id":"S071","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S017","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S010","target_id":"S035","kind":"supports"},{"source_id":"S075","target_id":"S035","kind":"supports"},{"source_id":"S017","target_id":"S036","kind":"supports"},{"source_id":"S074","target_id":"S036","kind":"supports"},{"source_id":"S034","target_id":"S037","kind":"supports"},{"source_id":"S075","target_id":"S037","kind":"supports"},{"source_id":"S033","target_id":"S038","kind":"supports"},{"source_id":"S034","target_id":"S038","kind":"supports"},{"source_id":"S035","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S020","target_id":"S039","kind":"supports"},{"source_id":"S011","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S030","kind":"supports"},{"source_id":"S033","target_id":"S030","kind":"supports"},{"source_id":"S034","target_id":"S030","kind":"supports"},{"source_id":"S035","target_id":"S030","kind":"supports"},{"source_id":"S036","target_id":"S030","kind":"supports"},{"source_id":"S037","target_id":"S030","kind":"supports"},{"source_id":"S038","target_id":"S030","kind":"supports"},{"source_id":"S039","target_id":"S030","kind":"supports"},{"source_id":"S040","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S016","target_id":"S043","kind":"supports"},{"source_id":"S071","target_id":"S043","kind":"supports"},{"source_id":"S003","target_id":"S044","kind":"supports"},{"source_id":"S014","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S010","target_id":"S046","kind":"supports"},{"source_id":"S014","target_id":"S046","kind":"supports"},{"source_id":"S074","target_id":"S046","kind":"supports"},{"source_id":"S075","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S047","kind":"supports"},{"source_id":"S075","target_id":"S047","kind":"supports"},{"source_id":"S044","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S021","target_id":"S049","kind":"supports"},{"source_id":"S015","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S044","target_id":"S042","kind":"supports"},{"source_id":"S045","target_id":"S042","kind":"supports"},{"source_id":"S046","target_id":"S042","kind":"supports"},{"source_id":"S047","target_id":"S042","kind":"supports"},{"source_id":"S048","target_id":"S042","kind":"supports"},{"source_id":"S049","target_id":"S042","kind":"supports"},{"source_id":"S050","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S043","target_id":"S054","kind":"supports"},{"source_id":"S045","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S045","target_id":"S055","kind":"supports"},{"source_id":"S046","target_id":"S055","kind":"supports"},{"source_id":"S010","target_id":"S055","kind":"supports"},{"source_id":"S014","target_id":"S055","kind":"supports"},{"source_id":"S075","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S045","target_id":"S056","kind":"supports"},{"source_id":"S075","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S051","target_id":"S058","kind":"supports"},{"source_id":"S003","target_id":"S058","kind":"supports"},{"source_id":"S015","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S041","target_id":"S059","kind":"supports"},{"source_id":"S051","target_id":"S059","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S055","target_id":"S053","kind":"supports"},{"source_id":"S056","target_id":"S053","kind":"supports"},{"source_id":"S057","target_id":"S053","kind":"supports"},{"source_id":"S058","target_id":"S053","kind":"supports"},{"source_id":"S059","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S008","target_id":"S062","kind":"supports"},{"source_id":"S015","target_id":"S062","kind":"supports"},{"source_id":"S016","target_id":"S062","kind":"supports"},{"source_id":"S008","target_id":"S063","kind":"supports"},{"source_id":"S015","target_id":"S063","kind":"supports"},{"source_id":"S016","target_id":"S063","kind":"supports"},{"source_id":"S010","target_id":"S063","kind":"supports"},{"source_id":"S072","target_id":"S063","kind":"supports"},{"source_id":"S075","target_id":"S063","kind":"supports"},{"source_id":"S062","target_id":"S065","kind":"supports"},{"source_id":"S063","target_id":"S065","kind":"supports"},{"source_id":"S075","target_id":"S065","kind":"supports"},{"source_id":"S018","target_id":"S066","kind":"supports"},{"source_id":"S019","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S052","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S075","target_id":"S067","kind":"supports"},{"source_id":"S051","target_id":"S068","kind":"supports"},{"source_id":"S018","target_id":"S068","kind":"supports"},{"source_id":"S019","target_id":"S068","kind":"supports"},{"source_id":"S001","target_id":"S068","kind":"supports"},{"source_id":"S076","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S013","target_id":"S069","kind":"supports"},{"source_id":"S019","target_id":"S069","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S063","target_id":"S061","kind":"supports"},{"source_id":"S065","target_id":"S061","kind":"supports"},{"source_id":"S066","target_id":"S061","kind":"supports"},{"source_id":"S067","target_id":"S061","kind":"supports"},{"source_id":"S068","target_id":"S061","kind":"supports"},{"source_id":"S069","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S070","kind":"supports"},{"source_id":"S007","target_id":"S070","kind":"supports"},{"source_id":"S009","target_id":"S070","kind":"supports"},{"source_id":"S002","target_id":"S070","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S060","edges":[{"source_id":"S001","target_id":"S007","kind":"supports"},{"source_id":"S002","target_id":"S007","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S013","kind":"supports"},{"source_id":"S009","target_id":"S013","kind":"supports"},{"source_id":"S003","target_id":"S014","kind":"supports"},{"source_id":"S005","target_id":"S015","kind":"supports"},{"source_id":"S006","target_id":"S015","kind":"supports"},{"source_id":"S007","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S007","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S007","target_id":"S023","kind":"supports"},{"source_id":"S071","target_id":"S023","kind":"supports"},{"source_id":"S010","target_id":"S024","kind":"supports"},{"source_id":"S001","target_id":"S024","kind":"supports"},{"source_id":"S073","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S075","target_id":"S025","kind":"supports"},{"source_id":"S007","target_id":"S026","kind":"supports"},{"source_id":"S071","target_id":"S026","kind":"supports"},{"source_id":"S010","target_id":"S027","kind":"supports"},{"source_id":"S001","target_id":"S027","kind":"supports"},{"source_id":"S073","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S075","target_id":"S028","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S022","kind":"supports"},{"source_id":"S026","target_id":"S022","kind":"supports"},{"source_id":"S027","target_id":"S022","kind":"supports"},{"source_id":"S028","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S012","target_id":"S031","kind":"supports"},{"source_id":"S071","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S017","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S010","target_id":"S035","kind":"supports"},{"source_id":"S075","target_id":"S035","kind":"supports"},{"source_id":"S017","target_id":"S036","kind":"supports"},{"source_id":"S074","target_id":"S036","kind":"supports"},{"source_id":"S034","target_id":"S037","kind":"supports"},{"source_id":"S075","target_id":"S037","kind":"supports"},{"source_id":"S033","target_id":"S038","kind":"supports"},{"source_id":"S034","target_id":"S038","kind":"supports"},{"source_id":"S035","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S020","target_id":"S039","kind":"supports"},{"source_id":"S011","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S030","kind":"supports"},{"source_id":"S033","target_id":"S030","kind":"supports"},{"source_id":"S034","target_id":"S030","kind":"supports"},{"source_id":"S035","target_id":"S030","kind":"supports"},{"source_id":"S036","target_id":"S030","kind":"supports"},{"source_id":"S037","target_id":"S030","kind":"supports"},{"source_id":"S038","target_id":"S030","kind":"supports"},{"source_id":"S039","target_id":"S030","kind":"supports"},{"source_id":"S040","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S016","target_id":"S043","kind":"supports"},{"source_id":"S071","target_id":"S043","kind":"supports"},{"source_id":"S003","target_id":"S044","kind":"supports"},{"source_id":"S014","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S010","target_id":"S046","kind":"supports"},{"source_id":"S014","target_id":"S046","kind":"supports"},{"source_id":"S074","target_id":"S046","kind":"supports"},{"source_id":"S075","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S047","kind":"supports"},{"source_id":"S075","target_id":"S047","kind":"supports"},{"source_id":"S044","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S021","target_id":"S049","kind":"supports"},{"source_id":"S015","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S044","target_id":"S042","kind":"supports"},{"source_id":"S045","target_id":"S042","kind":"supports"},{"source_id":"S046","target_id":"S042","kind":"supports"},{"source_id":"S047","target_id":"S042","kind":"supports"},{"source_id":"S048","target_id":"S042","kind":"supports"},{"source_id":"S049","target_id":"S042","kind":"supports"},{"source_id":"S050","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S043","target_id":"S054","kind":"supports"},{"source_id":"S045","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S045","target_id":"S055","kind":"supports"},{"source_id":"S046","target_id":"S055","kind":"supports"},{"source_id":"S010","target_id":"S055","kind":"supports"},{"source_id":"S014","target_id":"S055","kind":"supports"},{"source_id":"S075","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S045","target_id":"S056","kind":"supports"},{"source_id":"S075","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S051","target_id":"S058","kind":"supports"},{"source_id":"S003","target_id":"S058","kind":"supports"},{"source_id":"S015","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S041","target_id":"S059","kind":"supports"},{"source_id":"S051","target_id":"S059","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S055","target_id":"S053","kind":"supports"},{"source_id":"S056","target_id":"S053","kind":"supports"},{"source_id":"S057","target_id":"S053","kind":"supports"},{"source_id":"S058","target_id":"S053","kind":"supports"},{"source_id":"S059","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S008","target_id":"S062","kind":"supports"},{"source_id":"S015","target_id":"S062","kind":"supports"},{"source_id":"S016","target_id":"S062","kind":"supports"},{"source_id":"S008","target_id":"S063","kind":"supports"},{"source_id":"S015","target_id":"S063","kind":"supports"},{"source_id":"S016","target_id":"S063","kind":"supports"},{"source_id":"S010","target_id":"S063","kind":"supports"},{"source_id":"S072","target_id":"S063","kind":"supports"},{"source_id":"S075","target_id":"S063","kind":"supports"},{"source_id":"S062","target_id":"S065","kind":"supports"},{"source_id":"S063","target_id":"S065","kind":"supports"},{"source_id":"S075","target_id":"S065","kind":"supports"},{"source_id":"S018","target_id":"S066","kind":"supports"},{"source_id":"S019","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S052","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S075","target_id":"S067","kind":"supports"},{"source_id":"S051","target_id":"S068","kind":"supports"},{"source_id":"S018","target_id":"S068","kind":"supports"},{"source_id":"S019","target_id":"S068","kind":"supports"},{"source_id":"S001","target_id":"S068","kind":"supports"},{"source_id":"S076","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S013","target_id":"S069","kind":"supports"},{"source_id":"S019","target_id":"S069","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S063","target_id":"S061","kind":"supports"},{"source_id":"S065","target_id":"S061","kind":"supports"},{"source_id":"S066","target_id":"S061","kind":"supports"},{"source_id":"S067","target_id":"S061","kind":"supports"},{"source_id":"S068","target_id":"S061","kind":"supports"},{"source_id":"S069","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["problem:1"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"极小极大问题与可行域","source_units":[{"raw_tex":"\tWe consider the following minimax optimization problem:\n\t\\begin{equation}\\label{problem:1}\n\t\t\\min \\limits_{x\\in \\mathcal{X}}\\max \\limits_{y\\in \\mathcal{Y}}f(x,y),\\tag{P}\n\t\\end{equation}\n\twhere $\\mathcal{X}\\subseteq \\mathbb{R}^n$ and $\\mathcal{Y}\\subseteq \\mathbb{R}^m$ are nonempty closed and bounded convex sets, and $f: \\mathcal{X} \\times \\mathcal{Y} \\rightarrow \\mathbb{R}$\n\tis a smooth function. This problem has attracted more  attention due to its wide applications in machine learning, signal processing, and many other research fields in recent years.\n","start":6800,"end":7361}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"范数、梯度与投影记号","source_units":[{"raw_tex":"\t{\\bfseries Notation}. For vectors, we use $\\|\\cdot\\|$ to denote  the $l_2$-norm. For a function $f(x,y):\\mathbb{R}^n\\times\\mathbb{R}^m\\rightarrow \\mathbb{R}$, we use $\\nabla_{x} f(x,y)$ (or $\\nabla_{y} f(x, y)$) to denote the partial gradient of $f$ with respect to the first variable (or the second variable) at point $(x, y)$.\n\tLet $\\mathcal{P}_{\\mathcal{X}}$ and $\\mathcal{P}_{\\mathcal{Y}}$ denote projections onto the sets $\\mathcal{X}$ and $\\mathcal{Y}$. Finally, we use the notation $\\mathcal{O} (\\cdot)$ to hide only absolute constants which do not depend on any problem parameter, and $\\tilde{\\mathcal{O}}(\\cdot)$ notation to hide only absolute constants and log factors.\n","start":28706,"end":29387}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":["scon"],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"强凸性定义","source_units":[{"raw_tex":"A continuously differentiable function $f(\\cdot)$ is called $\\theta$-strongly convex if there exists a constant $\\theta> 0$ such\nthat for any $x,y \\in \\mathcal{X}$,\n\\begin{equation}\n  f(y)\\ge f(x)+\\langle \\nabla f(x),y-x \\rangle+\\frac{\\theta}{2}\\|y-x\\|^2.\\label{scon}\n\\end{equation}\n","start":29414,"end":29697}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["sec:2:1"],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"正则化目标","source_units":[{"raw_tex":"\tInstead of the original function $f(x, y)$, at the $k$-th iteration, AGP uses the gradient of a regularized version of the original function, i.e.,\n\t\\begin{equation}\n\t\tf_k (x, y) = f(x, y) + \\tfrac{b_{k}}{2} \\|x\\|^2 -\\tfrac{c_{k}}{2}\\| y\\|^2,\\label{sec:2:1}\n\t\\end{equation}\n\twhere $b_{k} \\ge 0$ and $c_{k}\\ge 0$ are two regularization parameters.\n","start":31108,"end":31456}]},{"id":"S005","semantic_type":"algorithm","kind":"algorithm","labels":["sec:2:2"],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"原变量投影子问题","source_units":[{"raw_tex":"\tMore specifically, for a given pair $(x_k, y_k) \\in \\mathcal{X} \\times \\mathcal{Y}$, AGP\n\tminimizes a linearized approximation of $f_k(x, y_k)$ plus some regularized terms to update  $x_k$ as follows:\n\t\\begin{align}\n\t\tx_{k+1}&={\\arg\\min}_{x\\in \\mathcal{X}}\\langle  \\nabla _x f_k\\left( x_k,y_k \\right) ,x-x_k \\rangle  +\\tfrac{\\beta_k}{2}\\| x-x_k \\| ^2\\nonumber\\\\\n\t\t&= \\mathcal{P}_\\mathcal{X} \\left( x_k - \\tfrac{1}{\\beta_k}  \\nabla _xf ( x_k,y_k ) - \\tfrac{1}{\\beta_k} b_{k} x_k \\right), \\label{sec:2:2}\n\t\\end{align}\n\twhere  $\\mathcal{P}_\\mathcal{X}$ is the projection operator onto $\\mathcal{X}$ and $\\beta_k > 0$ denotes\n","start":31456,"end":32079}]},{"id":"S006","semantic_type":"algorithm","kind":"algorithm","labels":["sec:2:3"],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"对偶变量投影子问题","source_units":[{"raw_tex":"\ta stepsize parameter. Similarly, it updates $y_k$ by maximizing a linearized approximation of $f_k(x_{k+1}, y)$ minus some regularized terms, i.e.,\n\t\\begin{align}\n\t\ty_{k+1}&={\\arg\\max}_{y\\in \\mathcal{Y}}\\langle  \\nabla _yf_k\\left( x_{k+1},y_k \\right),y-y_k \\rangle  -\\tfrac{\\gamma_k}{2}\\| y-y_k \\| ^2\\nonumber\\\\\n\t\t&= \\mathcal{P}_\\mathcal{Y}\\left( y_k+\\tfrac{1}{\\gamma_k} \\nabla _y {f}( x_{k+1},y_k ) - \\tfrac{1}{\\gamma_k} c_k y_k \\right),\\label{sec:2:3}\n\t\\end{align}\n\twhere  $\\mathcal{P}_\\mathcal{Y}$ is the projection operator onto  $\\mathcal{Y}$ and $\\gamma_k >0$ is another stepsize parameter.\n","start":32079,"end":32677}]},{"id":"S007","semantic_type":"algorithm","kind":"algorithm","labels":["alg:2.1","sec:2:4:agp:upx","sec:2:5:agp:upy"],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"完整 AGP 算法","source_units":[{"raw_tex":"\t\\begin{algorithm}%[H]\n\t\t\\caption{(An Alternating Gradient Projection (AGP) Algorithm)}\n\t\t\\label{alg:2.1}\n\t\t\\begin{algorithmic}\n\t\t\t\\STATE{\\textbf{Step 1}:Input $x_1,y_1,\\beta_1,\\gamma_1, b_1, c_1$; Set $k=1$.}\n\t\t\t%    \\FOR{$t = 1, ..., T$}\n\t\t\t\\STATE{\\textbf{Step 2}:Calculate $\\beta_k$ and $b_k$, and perform the following update for $x_k$:  \t\n\t\t\t\t\\qquad \\begin{equation}\\label{sec:2:4:agp:upx}\n\t\t\t\t\tx_{k+1}=\\mathcal{P}_\\mathcal{X} \\left( x_k - \\tfrac{1}{\\beta_k}  \\nabla _xf ( x_k,y_k ) - \\tfrac{1}{\\beta_k} b_k x_k \\right).\n\t\t\t\\end{equation}}\t\t\n\t\t\t\\STATE{\\textbf{Step 3}:Calculate $\\gamma_k$ and $c_k$, and perform the following update for $y_k$:\n\t\t\t\t\\qquad\t\\begin{equation}\\label{sec:2:5:agp:upy}\n\t\t\t\t\ty_{k+1}=\\mathcal{P}_\\mathcal{Y}\\left( y_k+\\tfrac{1}{\\gamma_k} \\nabla _y {f}( x_{k+1},y_k ) - \\tfrac{1}{\\gamma_k} c_k y_k \\right).\n\t\t\t\\end{equation}}\n\t\t\t\\STATE{\\textbf{Step 4}:If some stationary condition is satisfied, stop; otherwise, set $k=k+1, $ go to Step 2.}\n\t\t\t%    \\ENDFOR\n\t\t\\end{algorithmic}\n\t\\end{algorithm}\n","start":34199,"end":35221}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":["sec:2:def1:G"],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"驻点间隙与分量","source_units":[{"raw_tex":"\t\\begin{defi}\\label{sec:2:def1:G}\n\t\tAt each iteration of Algorithm \\ref{alg:2.1}, the stationarity gap for problem \\eqref{problem:1} w.r.t. $f(x,y)$ is defined as:\n\t\t\\begin{equation*}\n\t\t\t\\nabla G\\left( x_k,y_k \\right) :=\\left[\\begin{array}{c}\n\t\t\t\t{\\beta_k\\left(x_k-\\operatorname{\\mathcal{P}_\\mathcal{X}}(x_k-\\frac{1}{\\beta_k} \\nabla_{x} f(x_k ,y_k))\\right)} \\\\\n\t\t\t\t\n\t\t\t\t{\\gamma_k\\left(y_k-\\operatorname{\\mathcal{P}_\\mathcal{Y}}(y_k+\\frac{1}{\\gamma_k} \\nabla_{y} f(x_k ,y_k))\\right)}\n\t\t\t\\end{array}\\right].\n\t\t\\end{equation*}\n    \tWe denote $\\nabla {G}_k:= \\nabla {G}(x_k, y_k)$,\n\t\t$(\\nabla G_k)_{x}:= \\beta_k( x_k-\\operatorname{\\mathcal{P}_\\mathcal{X}}( x_k-\\frac{1}{\\beta_k}\\nabla _{x}f( x_k,y_k)))$,\n\t\tand $(\\nabla G_k)_{y}:=\\gamma_k( y_k-\\operatorname{\\mathcal{P}_\\mathcal{Y}}( y_k+ \\frac{1}{\\gamma_k} \\nabla _{y}f( x_k,y_k )) )$.\n\t\t%\t\t\\begin{align*}\n\t\t%\t\t\t(\\nabla G^r)_{x_i}&:= \\beta_i^r\\left( x_i^r-\\operatorname{Prox_{\\mathcal{X}_i}^{\\beta_i^r}}\\left( x_i^r+ \\frac{1}{\\beta_i^r} \\nabla _{x_i}f\\left( x^r,y^r \\right) \\right) \\right),\\\\\n\t\t%\t\t\t(\\nabla G^r)_{y}&:=\\rho^r\\left( y^r-\\operatorname{Prox_{\\mathcal{Y}}^{\\rho^r}}\\left( y^r+ \\frac{1}{\\rho^r} \\nabla _{y}f\\left( x^r,y^r \\right) \\right) \\right).\n\t\t%\t\t\\end{align*}\n\t\\end{defi}\n","start":36086,"end":37321}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"近似驻点判据","source_units":[{"raw_tex":"\t\n\tNote that the widely used metrics for convex-concave minimax problems such as the min-max value or the distance to the set of optimal solutions of the min-max problem are not applicable in nonconvex cases.\n\tFor the latter cases we call $(x_k,y_k)$ an $\\varepsilon$-stationary point of $f(x,y)$ if $ \\|\\nabla G(x_k,y_k)\\|\\leq \\varepsilon$ with $\\nabla G(x_k,y_k)$ being defined as in Definition \\ref{sec:2:def1:G}, which has been widely used as the optimality measure, e.g., \\cite{Lin2020,Lu}. In the absence of constraints, $ \\| \\nabla G(x_k,y_k)\\| \\leq \\varepsilon$ reduces to the standard condition $ \\|\\nabla_{x} f(x_k ,y_k)\\|\\leq \\varepsilon $ and $\\|\\nabla_{y} f(x_k ,y_k)\\|\\leq \\varepsilon$ for unconstrained problems. The vector  $\\nabla G(x_k,y_k)$ also refers to gradient mapping at $(x_k,y_k)$, see \\cite{Nes2013} for the details. \n","start":37321,"end":38166}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","labels":["sec:2:ass1:lip"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"分量梯度 Lipschitz 假设","source_units":[{"raw_tex":"\t\\begin{ass}\\label{sec:2:ass1:lip}\n\t\t$f\\left(x,y\\right)$ has Lipschitz continuous gradients, i.e., there exist positive scalars $L_{22}$, $L_{12}$, $L_{11}$, $L_{21}$ such that for any $x,\\bar{x}\\in \\mathcal{X}$, $y, \\bar{y}\\in \\mathcal{Y}$,\n\t\t\\begin{align*}\n\t\t\t\\| \\nabla_{x} f\\left(x, y\\right)-\\nabla_{x} f\\left(\\bar{x}, y\\right)\\| &\\leq L_{11}\\|x-\\bar{x}\\|,\\\\\n\t\t\t\\|\\nabla_{x} f\\left(x, y\\right)-\\nabla_{x} f\\left(x, \\bar{y}\\right)\\| &\\leq L_{21}\\|y-\\bar{y}\\|,\\\\\n\t\t\t\\|\\nabla_{y} f\\left(x, \\bar{y}\\right)-\\nabla_{y} f\\left(x, y\\right)\\| &\\leq L_{22}\\|\\bar{y}-y\\|,\\\\\n\t\t\t\\|\\nabla_{y} f\\left(x, y\\right)-\\nabla_{y} f\\left(\\bar{x}, y\\right)\\| &\\leq L_{12}\\|x-\\bar{x}\\|.\n\t\t\\end{align*}\n\t\tWe denote $L:=\\max\\{L_{11}, L_{22}, L_{12}, L_{21}\\}$.\n\t\\end{ass}\n","start":39184,"end":39933}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","labels":["subsec:3.1:1"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"强凹辅助分析参数","source_units":[{"raw_tex":"\\begin{equation}\\label{subsec:3.1:1}\n\t\t\\beta_k=\\eta, \\gamma_k=\\tfrac{1}{\\rho}, \\ \\mbox{and} \\ b_k=c_k=0\n\t\\end{equation}","start":43836,"end":43955}]},{"id":"S012","semantic_type":"algorithm","kind":"algorithm","labels":["subsec:3.1:2:x","subsec:3.1:3:y"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"强凹辅助分析更新","source_units":[{"raw_tex":"\\begin{align}\n\t\tx_{k+1}&=\\mathcal{P}_\\mathcal{X} \\left( x_k - \\tfrac{1}{\\eta}  \\nabla _xf ( x_k,y_k )  \\right),\\label{subsec:3.1:2:x}\\\\\n\t\ty_{k+1}&=\\mathcal{P}_\\mathcal{Y}\\left( y_k+\\rho \\nabla _y {f}( x_{k+1},y_k ) \\right),\\label{subsec:3.1:3:y}\n\t\\end{align}","start":44042,"end":44300}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["subsec:3.1:4:defT"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"首次达到精度的索引","source_units":[{"raw_tex":"\tWe are now ready to establish the iteration complexity for the AGP algorithm in the nonconvex-strongly concave setting.\n\tIn particular,\n\tletting $\\nabla G\\left( x_k,y_k \\right)$ be defined as  in Definition \\ref{sec:2:def1:G} and $\\varepsilon>0$ be\n\ta given target accuracy, we provide\n\ta bound on $T(\\varepsilon)$, the first iteration index to achieve an $\\varepsilon$-stationary point, i.e., $ \\| \\nabla G(x_k,y_k)\\| \\leq \\varepsilon$, which is equivalent to\n\t\\begin{equation}\\label{subsec:3.1:4:defT}\n\t\tT(\\varepsilon):=\\min\\{k \\mid \\|\\nabla G(x_k,y_k)\\|\\leq \\varepsilon \\}.\n\t\\end{equation}\n","start":55574,"end":56168}]},{"id":"S014","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"强凸非凹问题情形","source_units":[{"raw_tex":"\\subsection{Strongly Convex-Nonconcave Setting}\\label{subsec4}\nIn this subsection, we analyze the iteration complexity of Algorithm \\ref{alg:2.1} for solving strongly convex-nonconcave minimax optimization problems \\eqref{problem:1}, i.e., $f(x,y)$ is $\\theta$-strongly convex w.r.t. $x$ for any fixed $y\\in \\mathcal{Y}$,\nand nonconcave  w.r.t. $y$ for any given $x\\in \\mathcal{X}$. Under this setting,   $\\forall k\\ge 1$, we set\n","start":93416,"end":93846}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","labels":["subsec:4.1:1:para"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"强凸情形参数特化","source_units":[{"raw_tex":"\\begin{equation}\n\\beta_k=\\tfrac{1}{\\zeta}, \\gamma_k=\\nu,  b_k=c_k=0,\\label{subsec:4.1:1:para}\n\\end{equation}\nin  Algorithm \\ref{alg:2.1}, and simplify the update for $x_k$ and $y_k$ as follows:\n","start":93846,"end":94040}]},{"id":"S016","semantic_type":"algorithm","kind":"algorithm","labels":["subsec:4.1:2:x","subsec:4.1:3:y"],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"强凸情形交替更新","source_units":[{"raw_tex":"\\begin{align}\n\tx_{k+1}&=\\mathcal{P}_\\mathcal{X}\\left( x_k- \\zeta \\nabla _x {f}( x_{k},y_k) \\right),\\label{subsec:4.1:2:x}\\\\\n\ty_{k+1}&=\\mathcal{P}_\\mathcal{Y} \\left(y_k + \\tfrac{1}{\\nu}  \\nabla _yf ( x_{k+1},y_k )  \\right)\\label{subsec:4.1:3:y}.\n\\end{align}\n","start":94040,"end":94297}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S017"],"declared_external":false,"source_only":true,"title":"辅助引理的强凹情形","source_units":[{"raw_tex":"\tIn this subsection, we analyze the iteration complexity of Algorithm \\ref{alg:2.1} for solving nonconvex-strongly concave minimax optimization problems, i.e., $f(x,y)$ is nonconvex w.r.t. $x$ for any fixed $y\\in \\mathcal{Y}$,\n\tand $\\mu$-strongly concave  w.r.t. $y$ for any given $x\\in \\mathcal{X}$. Under this setting,  $\\forall k\\ge 1$, we set\n","start":43488,"end":43835}]},{"id":"S018","semantic_type":"assumption","kind":"assumption","labels":["thm:4.1"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"主定理参数限制","source_units":[{"raw_tex":"\tSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{\\left(x_{k},y_{k}\\right)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}\n\twith parameter settings in \\eqref{subsec:4.1:1:para}.\n\tIf the relations $\\nu>L_{22}, \\nu>L_{21}^2\\zeta+\\frac{4L_{21}^2}{\\zeta \\theta^2},~\\zeta\\leq \\frac{\\theta}{4L_{11}^2}$ are satisfied, then $\\forall \\varepsilon>0$,  it holds that $$ T\\left( \\varepsilon \\right) \\le\n","start":105709,"end":106117}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":["thm:4.1"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"复杂度常数与势函数上界记号","source_units":[{"raw_tex":"\twhere\n\t$\\hat{d}_1:=\\tfrac{\\min \\left\\{\\tfrac{\\nu}{2}-\\tfrac{\\zeta L_{21}^2}{2}-\\tfrac{2 L_{21}^2}{\\zeta \\theta^2},\\tfrac{3\\theta-\\zeta L_{11}^2}{2}+\\tfrac{\\theta-4\\zeta L_{11}^2}{2\\zeta\\theta} \\right\\}}{\\max \\left\\{ \\tfrac{1}{\\zeta ^2}+2L_{12}^{2},2\\nu^2 \\right\\}}%\\overline{\\hat{F}}=\\overline{f}-(\\tfrac{\\theta}{2}-\\tfrac{3}{\\zeta})\\sigma _x^2\n\t$\n\tand $\\overline{F}=\\bar{f}+(\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\sigma_x^2$ with $\\overline{f} := \\max_{(x,y) \\in \\mathcal{X} \\times \\mathcal{Y}} f(x,y)$ and $\\sigma _x= \\max\\{\\|x_1-x_2\\| \\mid \\forall x_1, x_2\\in \\mathcal{X}\\}$.\n","start":106177,"end":106793}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","labels":["lem:2.1"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"通用原变量函数变化界","source_units":[{"raw_tex":"\\begin{lem}\\label{lem:2.1}\nSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{(x_k,y_k)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}. Then we have\n\\begin{align}\n  f\\left( x_{k+1},y_k \\right) -f\\left( x_k,y_k \\right) &\\le-\\left( \\beta_k -\\tfrac{L_{11}}{2} \\right) \\lVert x_{k+1}-x_k \\rVert ^2-\\frac{b_k}{2}\\left[\\|x_{k+1}\\|^2-\\|x_k\\|^2-\\|x_{k+1}-x_k\\|^2\\right], \\label{lem:2.1:1}\\\\\n","start":40128,"end":40526}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","labels":["lem:2.1"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"通用对偶变量函数变化界","source_units":[{"raw_tex":"  f(x_{k+1},y_{k+1})-f( x_{k+1},y_{k})\n\t\t\t&\\ge\\left( \\gamma_k -\\tfrac{L_{22}}{2} \\right) \\lVert y_{k+1}-y_k \\rVert ^2+\\frac{c_k}{2}\\left[\\|y_{k+1}\\|^2-\\|y_k\\|^2-\\|y_{k+1}-y_k\\|^2\\right].\\label{lem:2.1:2}\n\\end{align}\n\\end{lem}","start":40526,"end":40751}]},{"id":"S022","semantic_type":"proof","kind":"proof","labels":["lem:2.1:3","lem:2.1:4","lem:2.1:6","lem:2.1:7"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"通用函数变化引理完整证明","source_units":[{"raw_tex":"\\begin{proof}\n  Firstly, by the optimality condition for \\eqref{sec:2:4:agp:upx}, we have\n\t\t\\begin{equation}\\label{lem:2.1:3}\n\t\t\t\\langle  \\nabla _xf\\left( x_k,y_k \\right) +\\beta_k \\left( x_{k+1}-x_k \\right)+b_kx_k ,x_k-x_{k+1} \\rangle  \\ge 0.\n\t\t\\end{equation}\n\t\tBy Assumption \\ref{sec:2:ass1:lip}, the gradient of $f$ is Lipschitz continuous, implying that\n\t\t\\begin{equation}\n\t\t\tf\\left( x_{k+1},y_k \\right) -f\\left( x_k,y_k \\right)\\leq \\langle  \\nabla _xf\\left( x_k,y_k \\right) ,x_{k+1}-x_k \\rangle  +\\tfrac{L_{11}}{2}\\lVert x_{k+1}-x_k \\rVert ^2. \\label{lem:2.1:4}\n\t\t\\end{equation}\n\t\tAdding \\eqref{lem:2.1:3} and \\eqref{lem:2.1:4}, we can easily show that\n\t\t\\begin{align*}\n\t\t\tf\\left( x_{k+1},y_k \\right) -f\\left( x_k,y_k \\right) &\\le -\\left( \\beta_k -\\tfrac{L_{11}}{2} \\right) \\lVert x_{k+1}-x_k \\rVert ^2-b_k\\langle x_k, x_{k+1}-x_k\\rangle\\nonumber\\\\\n            &= -\\left( \\beta_k -\\tfrac{L_{11}}{2} \\right) \\lVert x_{k+1}-x_k \\rVert ^2-\\frac{b_k}{2}\\left[\\|x_{k+1}\\|^2-\\|x_k\\|^2-\\|x_{k+1}-x_k\\|^2\\right].\n\t\t\\end{align*}\nThis completes the proof of \\eqref{lem:2.1:1}. By the optimality condition for \\eqref{sec:2:5:agp:upy}, we have\n\t\t\\begin{equation}\\label{lem:2.1:6}\n\t\t\t\\langle  \\nabla _{y}f( x_{k+1},y_{k})-\\gamma_k( y_{k+1}-y_k)-c_ky_k ,y_k-y_{k+1} \\rangle  \\leq 0.\n\t\t\\end{equation}\n\t\tBy  Assumption \\ref{sec:2:ass1:lip}, the gradient of $f$ is Lipschitz continuous,  implying that\n\t\t\\begin{align}\\label{lem:2.1:7}\n\t\t\tf(x_{k+1},y_{k+1})-f( x_{k+1},y_{k})\n\t\t\t\\ge  \\langle \\nabla _{y}f(x_{k+1},y_{k}),y_{k+1}\n    -y_{k} \\rangle -\\frac{L_{22}}{2}\\| y_{k+1}-y_{k} \\|^2.\n\t\t\\end{align}\n\t\t\tThe result \\eqref{lem:2.1:2} then follows by adding \\eqref{lem:2.1:6} and \\eqref{lem:2.1:7}.\n\n\\end{proof}","start":40753,"end":42448}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","labels":["lem:2.1:3"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"原变量投影最优性","source_units":[{"raw_tex":"\\begin{proof}\n  Firstly, by the optimality condition for \\eqref{sec:2:4:agp:upx}, we have\n\t\t\\begin{equation}\\label{lem:2.1:3}\n\t\t\t\\langle  \\nabla _xf\\left( x_k,y_k \\right) +\\beta_k \\left( x_{k+1}-x_k \\right)+b_kx_k ,x_k-x_{k+1} \\rangle  \\ge 0.\n\t\t\\end{equation}\n","start":40753,"end":41013}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","labels":["lem:2.1:4"],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"原变量光滑性上界","source_units":[{"raw_tex":"\t\tBy Assumption \\ref{sec:2:ass1:lip}, the gradient of $f$ is Lipschitz continuous, implying that\n\t\t\\begin{equation}\n\t\t\tf\\left( x_{k+1},y_k \\right) -f\\left( x_k,y_k \\right)\\leq \\langle  \\nabla _xf\\left( x_k,y_k \\right) ,x_{k+1}-x_k \\rangle  +\\tfrac{L_{11}}{2}\\lVert x_{k+1}-x_k \\rVert ^2. \\label{lem:2.1:4}\n\t\t\\end{equation}\n","start":41013,"end":41336}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"合成原变量下降界","source_units":[{"raw_tex":"\t\tAdding \\eqref{lem:2.1:3} and \\eqref{lem:2.1:4}, we can easily show that\n\t\t\\begin{align*}\n\t\t\tf\\left( x_{k+1},y_k \\right) -f\\left( x_k,y_k \\right) &\\le -\\left( \\beta_k -\\tfrac{L_{11}}{2} \\right) \\lVert x_{k+1}-x_k \\rVert ^2-b_k\\langle x_k, x_{k+1}-x_k\\rangle\\nonumber\\\\\n            &= -\\left( \\beta_k -\\tfrac{L_{11}}{2} \\right) \\lVert x_{k+1}-x_k \\rVert ^2-\\frac{b_k}{2}\\left[\\|x_{k+1}\\|^2-\\|x_k\\|^2-\\|x_{k+1}-x_k\\|^2\\right].\n\t\t\\end{align*}\nThis completes the proof of \\eqref{lem:2.1:1}. By the optimality condition for \\eqref{sec:2:5:agp:upy}, we have\n","start":41336,"end":41889}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","labels":["lem:2.1:6"],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"对偶变量投影最优性","source_units":[{"raw_tex":"This completes the proof of \\eqref{lem:2.1:1}. By the optimality condition for \\eqref{sec:2:5:agp:upy}, we have\n\t\t\\begin{equation}\\label{lem:2.1:6}\n\t\t\t\\langle  \\nabla _{y}f( x_{k+1},y_{k})-\\gamma_k( y_{k+1}-y_k)-c_ky_k ,y_k-y_{k+1} \\rangle  \\leq 0.\n\t\t\\end{equation}\n","start":41777,"end":42043}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","labels":["lem:2.1:7"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"对偶变量光滑性下界","source_units":[{"raw_tex":"\t\tBy  Assumption \\ref{sec:2:ass1:lip}, the gradient of $f$ is Lipschitz continuous,  implying that\n\t\t\\begin{align}\\label{lem:2.1:7}\n\t\t\tf(x_{k+1},y_{k+1})-f( x_{k+1},y_{k})\n\t\t\t\\ge  \\langle \\nabla _{y}f(x_{k+1},y_{k}),y_{k+1}\n    -y_{k} \\rangle -\\frac{L_{22}}{2}\\| y_{k+1}-y_{k} \\|^2.\n\t\t\\end{align}\n","start":42043,"end":42340}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"合成对偶变量上升界","source_units":[{"raw_tex":"\t\t\tThe result \\eqref{lem:2.1:2} then follows by adding \\eqref{lem:2.1:6} and \\eqref{lem:2.1:7}.\n\n\\end{proof}","start":42340,"end":42448}]},{"id":"S029","semantic_type":"lemma","kind":"lemma","labels":["lem:3.2:1","lem:3.2:blostr:y"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"强凹情形函数变化递推","source_units":[{"raw_tex":"\\begin{lem}\\label{lem:3.2:blostr:y}\n\t\tSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{\\left(x_k,y_k\\right)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}\n\t\twith parameter settings in \\eqref{subsec:3.1:1}. If $\\eta > L_{11}$, then we have\n\t\t\\begin{align}\\label{lem:3.2:1}\n\t\t\t&f( x_{k+1},y_{k+1}) -f\\left( x_k,y_k \\right)  \\nonumber \\\\\n\t\t\t\\leq& -\\left(\\tfrac{\\eta}{2}-\\tfrac{L_{12}^{2}\\rho}{2}\\right)\\| x_{k+1}-x_{k} \\| ^2-\\left(\\tfrac{\\mu}{2}-\\tfrac{1}{\\rho}\\right)\\| y_{k+1}-y_k \\|^2\\nonumber \\\\\n\t\t\t&-\\left(\\mu-\\tfrac{1}{2\\rho}-\\tfrac{\\rho L_{22}^2}{2}\\right)\\|y_k-y_{k-1} \\|^2.\n\t\t\\end{align}\n\t\\end{lem}","start":46965,"end":47586}]},{"id":"S030","semantic_type":"proof","kind":"proof","labels":["lem:3.1:1","lem:3.2:10:3points","lem:3.2:11","lem:3.2:2","lem:3.2:3","lem:3.2:4","lem:3.2:5","lem:3.2:6","lem:3.2:7","lem:3.2:8","lem:3.2:9"],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"强凹函数变化递推完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\t\tThe optimality condition for $y_k$ in \\eqref{subsec:3.1:3:y} implies that $\\forall y\\in \\mathcal{Y}$ and $\\forall k\\geq 1$,\n\t\t\\begin{equation}\\label{lem:3.2:2}\n\t\t\t\\langle \\nabla _yf(x_{k+1},y_k)-\\frac{1}{\\rho}(y_{k+1}-y_k),y-y_{k+1} \\rangle \\le 0.\n\t\t\\end{equation}\n\t\tBy choosing $y=y_k$ in \\eqref{lem:3.2:2}, we have\n\t\t\\begin{equation}\\label{lem:3.2:3}\n\t\t\t\\langle \\nabla _yf(x_{k+1},y_k)-\\frac{1}{\\rho}(y_{k+1}-y_k),y_k-y_{k+1} \\rangle \\le 0.\n\t\t\\end{equation}\n\t\tOn the other hand, by replacing $k$ with $k-1$ and choosing $y=y_{k+1}$ in \\eqref{lem:3.2:2}, we obtain\n\t\t\\begin{equation}\\label{lem:3.2:4}\n\t\t\t\\langle \\nabla _yf(x_{k},y_{k-1})-\\frac{1}{\\rho}(y_k-y_{k-1}),y_{k+1}-y_k \\rangle \\le 0,\n\t\t\\end{equation}\n\t\twhich, in view of the fact that $f\\left( x,y  \\right)$ is $\\mu $-strongly concave w.r.t. $y$ for any given $x\\in \\mathcal{X}$,\n\t\tthen implies that\n\t\t\\begin{align}\\label{lem:3.2:5}\n\t\t\t&f( x_{k+1},y_{k+1}) -f( x_{k+1},y_k ) \\nonumber\\\\\n\t\t\t\\leq &  \\langle  \\nabla _yf( x_{k+1},y_k ),y_{k+1}-y_k\\rangle  -\\frac{\\mu}{2}\\| y_{k+1}-y_k \\|^2\\  \\nonumber\\\\\n\t\t\t\\leq &\\langle  \\nabla _yf( x_{k+1},y_k ) -\\nabla _yf( x_k,y_{k-1}) ,y_{k+1}-y_k \\rangle   \\nonumber\\\\\n\t\t\t& +\\frac{1}{\\rho}\\langle  y_k-y_{k-1},y_{k+1}-y_k\\rangle  -\\frac{\\mu}{2}\\| y_{k+1}-y_k \\|^2.\n\t\t\\end{align}\n\t\tDenoting $v_{k+1}:=\\left(y_{k+1}-y_k\\right)-\\left(y_k-y_{k-1}\\right)$, we can write the first inner product term in the r.h.s.\n\t\tof \\eqref{lem:3.2:5} as\n\t\t\\begin{align}\\label{lem:3.2:6}\n\t\t\t&\\langle  \\nabla _y f ( x_{k+1},y_k ) -\\nabla _y f ( x_k,y_{k-1}) ,y_{k+1}-y_k \\rangle  \\nonumber\\\\\n\t\t\t=&\\langle  \\nabla _y f ( x_{k+1},y_k ) -\\nabla _y f \\left( x_k,y_{k} \\right) ,y_{k+1}-y_k \\rangle  \\nonumber\\\\\n\t\t\t& + \\langle  \\nabla _y f \\left( x_k,y_{k} \\right) -\\nabla _y f ( x_k,y_{k-1}) ,v_{k+1} \\rangle  \\nonumber\\\\\n\t\t\t&+\\langle  \\nabla _y f \\left( x_k,y_{k}\\right) -\\nabla _y f ( x_k,y_{k-1}) ,y_k-y_{k-1} \\rangle .\n\t\t\\end{align}\n\t\tNext, we estimate the three terms in the right hand side of \\eqref{lem:3.2:6} respectively.\n\t\tBy Assumption \\ref{sec:2:ass1:lip} and the Cauchy-Schwarz inequality, we can bound the first two terms according to\n\t\t\\begin{align}\\label{lem:3.2:7}\n\t\t\t&\\langle  \\nabla _yf( x_{k+1},y_k ) -\\nabla _yf\\left( x_k,y_{k} \\right) ,y_{k+1}-y_k \\rangle  \\nonumber\\\\\n\t\t\t\\leq &\\frac{L_{12}^{2}\\rho}{2}\\| x_{k+1}-x_k \\|^2+\\frac{1}{2\\rho}\\| y_{k+1}-y_k \\|^2,\n\t\t\\end{align}\n\t\tand\n\t\t\\begin{align}\\label{lem:3.2:8}\n\t\t\t&\\langle  \\nabla _yf\\left(  x_k,y_{k} \\right) -\\nabla _yf( x_k,y_{k-1}) ,v_{k+1} \\rangle  \\leq \\frac{\\rho L_{22}^{2}}{2}\\|y_k-y_{k-1} \\|^2+\\frac{1}{2\\rho}\\|v_{k+1} \\|^2.\n\t\t\\end{align}\n\t\tFor the third term, by $\\mu $-strongly-concavity  of $f$ with respect to $y$,\n\t\t\\begin{align}\\label{lem:3.2:9}\n\t\t\t\\quad \\langle  \\nabla _yf\\left( x_k,y_{k} \\right) -\\nabla _yf( x_k,y_{k-1}) ,y_k-y_{k-1} \\rangle  \\le -\\mu \\| y_k-y_{k-1} \\|^2 .\n\t\t\\end{align}\n\t\tMoreover, it can be easily checked that\n\t\t\\begin{align}\\label{lem:3.2:10:3points}\n\t\t\t&\\langle  y_k-y_{k-1},y_{k+1}-y_k \\rangle =\\frac{1}{2}\\| y_k-y_{k-1} \\|^2+\\frac{1}{2}\\| y_{k+1}-y_k \\|^2-\\frac{1}{2}\\| v_{k+1} \\|^2.\n\t\t\\end{align}\n\t\tPlugging \\eqref{lem:3.2:6}-\\eqref{lem:3.2:10:3points} into \\eqref{lem:3.2:5} and rearranging the terms, we conclude that\n\t\t\\begin{align}\\label{lem:3.2:11}\n\t\t\t&f( x_{k+1},y_{k+1}) -f( x_{k+1},y_{k})  \\nonumber \\\\\n\t\t\t\\leq& \\frac{L_{12}^{2}\\rho}{2}\\| x_{k+1}-x_k \\|^2-(\\mu-\\frac{1}{2\\rho}-\\frac{\\rho L_{22}^2}{2})\\| y_k-y_{k-1}\\|^2\\nonumber\\\\\n\t\t\t&-(\\frac{\\mu}{2}-\\frac{1}{\\rho})\\|y_{k+1}-y_k \\|^2.\n\t\t\\end{align}\nBy setting $\\beta_k=\\eta$, $b_k=0$ in \\eqref{lem:2.1:1} of Lemma \\ref{lem:2.1}  and the assumption $\\eta > L_{11}$, we have\n\t\t\\begin{equation}\\label{lem:3.1:1}\n\t\t\tf(x_{k+1},y_{k})-f(x_{k},y_k)\\le -\\frac{\\eta}{2}\\| x_{k+1}-x_k \\|^2.\n\t\t\\end{equation}\nThe proof is completed by combining \\eqref{lem:3.2:11} with \\eqref{lem:3.1:1}.\n\t\\end{proof}","start":47590,"end":51440}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":["lem:3.2:2","lem:3.2:3"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"当前对偶投影最优性","source_units":[{"raw_tex":"\\begin{proof}\n\t\tThe optimality condition for $y_k$ in \\eqref{subsec:3.1:3:y} implies that $\\forall y\\in \\mathcal{Y}$ and $\\forall k\\geq 1$,\n\t\t\\begin{equation}\\label{lem:3.2:2}\n\t\t\t\\langle \\nabla _yf(x_{k+1},y_k)-\\frac{1}{\\rho}(y_{k+1}-y_k),y-y_{k+1} \\rangle \\le 0.\n\t\t\\end{equation}\n\t\tBy choosing $y=y_k$ in \\eqref{lem:3.2:2}, we have\n\t\t\\begin{equation}\\label{lem:3.2:3}\n\t\t\t\\langle \\nabla _yf(x_{k+1},y_k)-\\frac{1}{\\rho}(y_{k+1}-y_k),y_k-y_{k+1} \\rangle \\le 0.\n\t\t\\end{equation}\n","start":47590,"end":48066}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":["lem:3.2:4"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"前次对偶投影最优性","source_units":[{"raw_tex":"\t\tOn the other hand, by replacing $k$ with $k-1$ and choosing $y=y_{k+1}$ in \\eqref{lem:3.2:2}, we obtain\n\t\t\\begin{equation}\\label{lem:3.2:4}\n\t\t\t\\langle \\nabla _yf(x_{k},y_{k-1})-\\frac{1}{\\rho}(y_k-y_{k-1}),y_{k+1}-y_k \\rangle \\le 0,\n\t\t\\end{equation}\n","start":48066,"end":48317}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":["lem:3.2:5"],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"强凹性控制对偶函数变化","source_units":[{"raw_tex":"\t\twhich, in view of the fact that $f\\left( x,y  \\right)$ is $\\mu $-strongly concave w.r.t. $y$ for any given $x\\in \\mathcal{X}$,\n\t\tthen implies that\n\t\t\\begin{align}\\label{lem:3.2:5}\n\t\t\t&f( x_{k+1},y_{k+1}) -f( x_{k+1},y_k ) \\nonumber\\\\\n\t\t\t\\leq &  \\langle  \\nabla _yf( x_{k+1},y_k ),y_{k+1}-y_k\\rangle  -\\frac{\\mu}{2}\\| y_{k+1}-y_k \\|^2\\  \\nonumber\\\\\n\t\t\t\\leq &\\langle  \\nabla _yf( x_{k+1},y_k ) -\\nabla _yf( x_k,y_{k-1}) ,y_{k+1}-y_k \\rangle   \\nonumber\\\\\n\t\t\t& +\\frac{1}{\\rho}\\langle  y_k-y_{k-1},y_{k+1}-y_k\\rangle  -\\frac{\\mu}{2}\\| y_{k+1}-y_k \\|^2.\n\t\t\\end{align}\n","start":48317,"end":48882}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":["lem:3.2:6"],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"对偶梯度差三项分解","source_units":[{"raw_tex":"\t\tDenoting $v_{k+1}:=\\left(y_{k+1}-y_k\\right)-\\left(y_k-y_{k-1}\\right)$, we can write the first inner product term in the r.h.s.\n\t\tof \\eqref{lem:3.2:5} as\n\t\t\\begin{align}\\label{lem:3.2:6}\n\t\t\t&\\langle  \\nabla _y f ( x_{k+1},y_k ) -\\nabla _y f ( x_k,y_{k-1}) ,y_{k+1}-y_k \\rangle  \\nonumber\\\\\n\t\t\t=&\\langle  \\nabla _y f ( x_{k+1},y_k ) -\\nabla _y f \\left( x_k,y_{k} \\right) ,y_{k+1}-y_k \\rangle  \\nonumber\\\\\n\t\t\t& + \\langle  \\nabla _y f \\left( x_k,y_{k} \\right) -\\nabla _y f ( x_k,y_{k-1}) ,v_{k+1} \\rangle  \\nonumber\\\\\n\t\t\t&+\\langle  \\nabla _y f \\left( x_k,y_{k}\\right) -\\nabla _y f ( x_k,y_{k-1}) ,y_k-y_{k-1} \\rangle .\n\t\t\\end{align}\n","start":48882,"end":49513}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":["lem:3.2:7","lem:3.2:8"],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"对偶梯度差的光滑性估计","source_units":[{"raw_tex":"\t\tNext, we estimate the three terms in the right hand side of \\eqref{lem:3.2:6} respectively.\n\t\tBy Assumption \\ref{sec:2:ass1:lip} and the Cauchy-Schwarz inequality, we can bound the first two terms according to\n\t\t\\begin{align}\\label{lem:3.2:7}\n\t\t\t&\\langle  \\nabla _yf( x_{k+1},y_k ) -\\nabla _yf\\left( x_k,y_{k} \\right) ,y_{k+1}-y_k \\rangle  \\nonumber\\\\\n\t\t\t\\leq &\\frac{L_{12}^{2}\\rho}{2}\\| x_{k+1}-x_k \\|^2+\\frac{1}{2\\rho}\\| y_{k+1}-y_k \\|^2,\n\t\t\\end{align}\n\t\tand\n\t\t\\begin{align}\\label{lem:3.2:8}\n\t\t\t&\\langle  \\nabla _yf\\left(  x_k,y_{k} \\right) -\\nabla _yf( x_k,y_{k-1}) ,v_{k+1} \\rangle  \\leq \\frac{\\rho L_{22}^{2}}{2}\\|y_k-y_{k-1} \\|^2+\\frac{1}{2\\rho}\\|v_{k+1} \\|^2.\n\t\t\\end{align}\n","start":49513,"end":50196}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":["lem:3.2:9"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"强凹梯度的负单调性","source_units":[{"raw_tex":"\t\tFor the third term, by $\\mu $-strongly-concavity  of $f$ with respect to $y$,\n\t\t\\begin{align}\\label{lem:3.2:9}\n\t\t\t\\quad \\langle  \\nabla _yf\\left( x_k,y_{k} \\right) -\\nabla _yf( x_k,y_{k-1}) ,y_k-y_{k-1} \\rangle  \\le -\\mu \\| y_k-y_{k-1} \\|^2 .\n\t\t\\end{align}\n","start":50196,"end":50455}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":["lem:3.2:10:3points"],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"对偶三点内积恒等式","source_units":[{"raw_tex":"\t\tMoreover, it can be easily checked that\n\t\t\\begin{align}\\label{lem:3.2:10:3points}\n\t\t\t&\\langle  y_k-y_{k-1},y_{k+1}-y_k \\rangle =\\frac{1}{2}\\| y_k-y_{k-1} \\|^2+\\frac{1}{2}\\| y_{k+1}-y_k \\|^2-\\frac{1}{2}\\| v_{k+1} \\|^2.\n\t\t\\end{align}\n","start":50455,"end":50689}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":["lem:3.2:11"],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"合成强凹对偶变化界","source_units":[{"raw_tex":"\t\tPlugging \\eqref{lem:3.2:6}-\\eqref{lem:3.2:10:3points} into \\eqref{lem:3.2:5} and rearranging the terms, we conclude that\n\t\t\\begin{align}\\label{lem:3.2:11}\n\t\t\t&f( x_{k+1},y_{k+1}) -f( x_{k+1},y_{k})  \\nonumber \\\\\n\t\t\t\\leq& \\frac{L_{12}^{2}\\rho}{2}\\| x_{k+1}-x_k \\|^2-(\\mu-\\frac{1}{2\\rho}-\\frac{\\rho L_{22}^2}{2})\\| y_k-y_{k-1}\\|^2\\nonumber\\\\\n\t\t\t&-(\\frac{\\mu}{2}-\\frac{1}{\\rho})\\|y_{k+1}-y_k \\|^2.\n\t\t\\end{align}\n","start":50689,"end":51100}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":["lem:3.1:1"],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"特化通用原变量下降界","source_units":[{"raw_tex":"By setting $\\beta_k=\\eta$, $b_k=0$ in \\eqref{lem:2.1:1} of Lemma \\ref{lem:2.1}  and the assumption $\\eta > L_{11}$, we have\n\t\t\\begin{equation}\\label{lem:3.1:1}\n\t\t\tf(x_{k+1},y_{k})-f(x_{k},y_k)\\le -\\frac{\\eta}{2}\\| x_{k+1}-x_k \\|^2.\n\t\t\\end{equation}\n","start":51100,"end":51349}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"完成强凹函数变化递推","source_units":[{"raw_tex":"The proof is completed by combining \\eqref{lem:3.2:11} with \\eqref{lem:3.1:1}.\n\t\\end{proof}","start":51349,"end":51440}]},{"id":"S041","semantic_type":"lemma","kind":"lemma","labels":["lem:4.2","lem:4.2:1:conclu"],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"强凸情形函数变化递推","source_units":[{"raw_tex":"\\begin{lem}\\label{lem:4.2}\n\tSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{\\left(x_{k},y_{k}\\right)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}\n\twith parameter settings in \\eqref{subsec:4.1:1:para}. If $\\nu > L_{22}$, then we have\n\t\\begin{align}\\label{lem:4.2:1:conclu}\n\t\t&f( x_{k+1},y_{k+1}) -f( x_{k},y_{k})  \\nonumber \\\\\n\t\t\\geq& \\frac{\\nu}{2}\\| y_{k+1}-y_{k} \\| ^2-\\frac{L_{21}^{2}\\zeta}{2}\\| y_{k}-y_{k-1} \\|^2+(\\theta-\\frac{1}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2})\\| x_{k}-x_{k-1}\\|^2\\nonumber\\\\\n\t\t&+ (\\frac{\\theta}{2}-\\frac{1}{\\zeta})\\|x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\n\\end{lem}","start":97011,"end":97613}]},{"id":"S042","semantic_type":"proof","kind":"proof","labels":["lem:4.1:1:conclu","lem:4.2:2:opty","lem:4.2:3:nopty","lem:4.2:4:opty:k","lem:4.2:5","lem:4.2:6","lem:4.2:7","lem:4.2:8:3points","lem:4.2:9"],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"强凸函数变化递推完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\tSimilar to \\eqref{lem:3.2:2}-\\eqref{lem:3.2:4} in the proof of Lemma \\ref{lem:3.2:blostr:y}, by the optimality condition for $x_k$ in \\eqref{subsec:4.1:2:x} implies that $\\forall x\\in \\mathcal{X}$ and $\\forall k\\geq 1$,\n\t\\begin{align}\n\t\t\\langle \\nabla _xf(x_{k},y_{k})+\\frac{1}{\\zeta}(x_{k+1}-x_{k}),x-x_{k+1} \\rangle &\\ge 0,\\label{lem:4.2:2:opty}\\\\\n\t\t\\langle \\nabla _xf(x_{k},y_{k})+\\frac{1}{\\zeta}(x_{k+1}-x_{k}) ,x_{k}-x_{k+1} \\rangle &\\ge 0,\\label{lem:4.2:3:nopty}\\\\\n\t\t\\langle \\nabla _xf(x_{k-1},y_{k-1})+\\frac{1}{\\zeta}(x_{k}-x_{k-1}),x_{k+1}-x_{k} \\rangle &\\ge 0,\\label{lem:4.2:4:opty:k}\n\t\\end{align}\nwhich, in view of the fact that $f\\left( x,y  \\right)$ is $\\theta $-strongly convex w.r.t. $x$ for any given $y\\in \\mathcal{Y}$,\n\tthen implies that\n\t\\begin{align}\\label{lem:4.2:5}\n\t\t&f( x_{k+1},y_{k}) -f( x_{k},y_{k} ) \\nonumber\\\\\n\t\t\\geq &  \\langle  \\nabla _xf( x_{k},y_{k} ),x_{k+1}-x_{k}\\rangle  +\\frac{\\theta}{2}\\| x_{k+1}-x_{k} \\|^2\\  \\nonumber\\\\\n\t\t\\geq &\\langle  \\nabla _xf( x_{k},y_{k} ) -\\nabla _xf( x_{k-1},y_{k-1}) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t& -\\frac{1}{\\zeta}\\langle  x_{k}-x_{k-1},x_{k+1}-x_{k} \\rangle +\\frac{\\theta}{2}\\| x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\n\tDenoting $m_{k+1}:=\\left(x_{k+1}-x_k\\right)-\\left(x_k-x_{k-1}\\right)$, by Assumption \\ref{sec:2:ass1:lip}, the Cauchy-Schwarz inequality and\n\tthe $\\theta $-strongly convexity  of $f$ w.r.t. $x$, we can estimate the first inner product term in the r.h.s.\n\tof \\eqref{lem:4.2:5} as\n\t\\begin{align}\n\t\t&\\langle  \\nabla _x f ( x_{k},y_{k} ) -\\nabla _x f ( x_{k-1},y_{k-1}) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t=&\\langle  \\nabla _x f ( x_{k},y_{k} ) -\\nabla _x f \\left( x_{k},y_{k-1} \\right) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t& + \\langle  \\nabla _x f \\left( x_{k},y_{k-1} \\right) -\\nabla _x f ( x_{k-1},y_{k-1}) ,m_{k+1} \\rangle  \\nonumber\\\\\n\t\t&+\\langle  \\nabla _x f \\left( x_{k},y_{k-1}\\right) -\\nabla _x f ( x_{k-1},y_{k-1}) ,x_{k}-x_{k-1} \\rangle \\label{lem:4.2:6}\\\\\n\t\\geq &-\\frac{L_{21}^{2}\\zeta}{2}\\| y_{k}-y_{k-1} \\|^2-\\frac{1}{2\\zeta}\\| x_{k+1}-x_k \\|^2-\\frac{\\zeta L_{11}^{2}}{2}\\|x_k-x_{k-1} \\|^2\\nonumber\\\\\n\t&-\\frac{1}{2\\zeta}\\|m_{k+1} \\|^2 +\t\\theta \\| x_k-x_{k-1} \\|^2.\\label{lem:4.2:7}\n\t\\end{align}\n\tMoreover, it can be easily checked that\n\\begin{align}\\label{lem:4.2:8:3points}\n\t\t&\\langle  x_k-x_{k-1},x_{k+1}-x_k \\rangle =\\frac{1}{2}\\| x_k-x_{k-1} \\|^2+\\frac{1}{2}\\| x_{k+1}-x_k \\|^2-\\frac{1}{2}\\| m_{k+1} \\|^2.\n\t\\end{align}\n\tPlugging \\eqref{lem:4.2:7} and \\eqref{lem:4.2:8:3points} into \\eqref{lem:4.2:5} and rearranging the terms, we conclude that\n\t\\begin{align}\\label{lem:4.2:9}\n\tf( x_{k+1},y_{k}) -f( x_{k},y_{k})\t\\geq& -\\frac{L_{21}^{2}\\zeta}{2}\\| y_{k}-y_{k-1} \\|^2+(\\theta-\\frac{1}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2})\\| x_{k}-x_{k-1}\\|^2\\nonumber\\\\\n\t\t&+ (\\frac{\\theta}{2}-\\frac{1}{\\zeta})\\|x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\nBy setting $\\gamma_k=\\nu$, $c_k=0$ in \\eqref{lem:2.1:2} of Lemma \\ref{lem:2.1}  and the assumption $\\nu > L_{22}$, we have\n\t\t\\begin{equation}\\label{lem:4.1:1:conclu}\n\t\t\tf( x_{k+1},y_{k+1}) -f( x_{k+1},y_k )\\ge \\frac{\\nu}{2}\\| y_{k+1}-y_k \\|^2.\n\t\t\\end{equation}\n\tThe proof is completed by combining \\eqref{lem:4.2:9} with \\eqref{lem:4.1:1:conclu}.\n\\end{proof}","start":97615,"end":100813}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":["lem:4.2:2:opty","lem:4.2:3:nopty","lem:4.2:4:opty:k"],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"原变量两次投影最优性","source_units":[{"raw_tex":"\\begin{proof}\n\tSimilar to \\eqref{lem:3.2:2}-\\eqref{lem:3.2:4} in the proof of Lemma \\ref{lem:3.2:blostr:y}, by the optimality condition for $x_k$ in \\eqref{subsec:4.1:2:x} implies that $\\forall x\\in \\mathcal{X}$ and $\\forall k\\geq 1$,\n\t\\begin{align}\n\t\t\\langle \\nabla _xf(x_{k},y_{k})+\\frac{1}{\\zeta}(x_{k+1}-x_{k}),x-x_{k+1} \\rangle &\\ge 0,\\label{lem:4.2:2:opty}\\\\\n\t\t\\langle \\nabla _xf(x_{k},y_{k})+\\frac{1}{\\zeta}(x_{k+1}-x_{k}) ,x_{k}-x_{k+1} \\rangle &\\ge 0,\\label{lem:4.2:3:nopty}\\\\\n\t\t\\langle \\nabla _xf(x_{k-1},y_{k-1})+\\frac{1}{\\zeta}(x_{k}-x_{k-1}),x_{k+1}-x_{k} \\rangle &\\ge 0,\\label{lem:4.2:4:opty:k}\n\t\\end{align}\n","start":97615,"end":98237}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":["lem:4.2:5"],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"强凸性控制原变量函数变化","source_units":[{"raw_tex":"which, in view of the fact that $f\\left( x,y  \\right)$ is $\\theta $-strongly convex w.r.t. $x$ for any given $y\\in \\mathcal{Y}$,\n\tthen implies that\n\t\\begin{align}\\label{lem:4.2:5}\n\t\t&f( x_{k+1},y_{k}) -f( x_{k},y_{k} ) \\nonumber\\\\\n\t\t\\geq &  \\langle  \\nabla _xf( x_{k},y_{k} ),x_{k+1}-x_{k}\\rangle  +\\frac{\\theta}{2}\\| x_{k+1}-x_{k} \\|^2\\  \\nonumber\\\\\n\t\t\\geq &\\langle  \\nabla _xf( x_{k},y_{k} ) -\\nabla _xf( x_{k-1},y_{k-1}) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t& -\\frac{1}{\\zeta}\\langle  x_{k}-x_{k-1},x_{k+1}-x_{k} \\rangle +\\frac{\\theta}{2}\\| x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\n","start":98237,"end":98815}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"原变量梯度差三项分解","source_units":[{"raw_tex":"\tDenoting $m_{k+1}:=\\left(x_{k+1}-x_k\\right)-\\left(x_k-x_{k-1}\\right)$, by Assumption \\ref{sec:2:ass1:lip}, the Cauchy-Schwarz inequality and\n\tthe $\\theta $-strongly convexity  of $f$ w.r.t. $x$, we can estimate the first inner product term in the r.h.s.\n\tof \\eqref{lem:4.2:5} as\n\t\\begin{align}\n\t\t&\\langle  \\nabla _x f ( x_{k},y_{k} ) -\\nabla _x f ( x_{k-1},y_{k-1}) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t=&\\langle  \\nabla _x f ( x_{k},y_{k} ) -\\nabla _x f \\left( x_{k},y_{k-1} \\right) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t& + \\langle  \\nabla _x f \\left( x_{k},y_{k-1} \\right) -\\nabla _x f ( x_{k-1},y_{k-1}) ,m_{k+1} \\rangle  \\nonumber\\\\\n\t\t&+\\langle  \\nabla _x f \\left( x_{k},y_{k-1}\\right) -\\nabla _x f ( x_{k-1},y_{k-1}) ,x_{k}-x_{k-1} \\rangle \\label{lem:4.2:6}\\\\\n","start":98815,"end":99583}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"梯度差内积下界","source_units":[{"raw_tex":"\t\\geq &-\\frac{L_{21}^{2}\\zeta}{2}\\| y_{k}-y_{k-1} \\|^2-\\frac{1}{2\\zeta}\\| x_{k+1}-x_k \\|^2-\\frac{\\zeta L_{11}^{2}}{2}\\|x_k-x_{k-1} \\|^2\\nonumber\\\\\n\t&-\\frac{1}{2\\zeta}\\|m_{k+1} \\|^2 +\t\\theta \\| x_k-x_{k-1} \\|^2.\\label{lem:4.2:7}\n\t\\end{align}\n","start":99583,"end":99824}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":["lem:4.2:8:3points"],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"原变量三点内积恒等式","source_units":[{"raw_tex":"\tMoreover, it can be easily checked that\n\\begin{align}\\label{lem:4.2:8:3points}\n\t\t&\\langle  x_k-x_{k-1},x_{k+1}-x_k \\rangle =\\frac{1}{2}\\| x_k-x_{k-1} \\|^2+\\frac{1}{2}\\| x_{k+1}-x_k \\|^2-\\frac{1}{2}\\| m_{k+1} \\|^2.\n\t\\end{align}\n","start":99824,"end":100052}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":["lem:4.2:9"],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"合成强凸原变量变化界","source_units":[{"raw_tex":"\tPlugging \\eqref{lem:4.2:7} and \\eqref{lem:4.2:8:3points} into \\eqref{lem:4.2:5} and rearranging the terms, we conclude that\n\t\\begin{align}\\label{lem:4.2:9}\n\tf( x_{k+1},y_{k}) -f( x_{k},y_{k})\t\\geq& -\\frac{L_{21}^{2}\\zeta}{2}\\| y_{k}-y_{k-1} \\|^2+(\\theta-\\frac{1}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2})\\| x_{k}-x_{k-1}\\|^2\\nonumber\\\\\n\t\t&+ (\\frac{\\theta}{2}-\\frac{1}{\\zeta})\\|x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\n","start":100052,"end":100455}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["lem:4.1:1:conclu"],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"特化通用对偶上升界","source_units":[{"raw_tex":"By setting $\\gamma_k=\\nu$, $c_k=0$ in \\eqref{lem:2.1:2} of Lemma \\ref{lem:2.1}  and the assumption $\\nu > L_{22}$, we have\n\t\t\\begin{equation}\\label{lem:4.1:1:conclu}\n\t\t\tf( x_{k+1},y_{k+1}) -f( x_{k+1},y_k )\\ge \\frac{\\nu}{2}\\| y_{k+1}-y_k \\|^2.\n\t\t\\end{equation}\n","start":100455,"end":100716}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"完成强凸函数变化递推","source_units":[{"raw_tex":"\tThe proof is completed by combining \\eqref{lem:4.2:9} with \\eqref{lem:4.1:1:conclu}.\n\\end{proof}","start":100716,"end":100813}]},{"id":"S051","semantic_type":"definition","kind":"definition","labels":["lem:4.3"],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"强凸情形势函数","source_units":[{"raw_tex":"\t$$\\hat{f}_{k+1}:=f\\left( x_{k+1},y_{k+1} \\right),\\ \\hat{S}_{k+1}:=-\\frac{2}{\\zeta^2\\theta}\\lVert x_{k+1}-x_k \\rVert^2,$$\n\t$$\\hat{F}_{k+1}:=\\hat{f}_{k+1}+\\hat{S}_{k+1}+ (\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\lVert x_{k+1}-x_{k} \\rVert^2-(\\frac{L_{21}^2\\zeta}{2}+\\frac{2L_{21}^2}{\\theta^2\\zeta})\\|y_{k+1}-y_k\\|^2.$$\n","start":101468,"end":101820}]},{"id":"S052","semantic_type":"lemma","kind":"lemma","labels":["lem:4.3","lem:4.3:2:conclus"],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"势函数增量控制位移","source_units":[{"raw_tex":"\\begin{lem}\\label{lem:4.3}\n\tSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{\\left(x_k,y_k\\right)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}\n\twith parameter settings in \\eqref{subsec:4.1:1:para}. Denote\n\t$$\\hat{f}_{k+1}:=f\\left( x_{k+1},y_{k+1} \\right),\\ \\hat{S}_{k+1}:=-\\frac{2}{\\zeta^2\\theta}\\lVert x_{k+1}-x_k \\rVert^2,$$\n\t$$\\hat{F}_{k+1}:=\\hat{f}_{k+1}+\\hat{S}_{k+1}+ (\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\lVert x_{k+1}-x_{k} \\rVert^2-(\\frac{L_{21}^2\\zeta}{2}+\\frac{2L_{21}^2}{\\theta^2\\zeta})\\|y_{k+1}-y_k\\|^2.$$\n\tIf $\\nu>L_{22}$,\n%\t\\begin{equation}\\label{lem:4.3:1}\n%, \\nu>L_{21}^2\\zeta+\\frac{4L_{21}^2}{\\zeta \\theta^2},~\\zeta\\leq \\frac{\\theta}{4L_{11}^2},\n%\t\\end{equation}\n\tthen $\\forall k \\geq 1$,\n\t\\begin{align}\\label{lem:4.3:2:conclus}\n\t\t\\hat{F}_{k+1}-\\hat{F}_k\\geq& \\left(\\frac{\\nu}{2}-\\frac{\\zeta L_{21}^2}{2}-\\frac{2L_{21}^2}{\\zeta \\theta^2}\\right)\\lVert y_{k+1}-y_k \\rVert ^2\\nonumber\\\\\n\t\t& +\\left(\\frac{3\\theta-\\zeta L_{11}^2}{2}+\\frac{\\theta-4\\zeta L_{11}^2}{2\\zeta\\theta}\\right)\\lVert x_{k+1}-x_k \\rVert ^2.\n\t\\end{align}\n\\end{lem}","start":101244,"end":102349}]},{"id":"S053","semantic_type":"proof","kind":"proof","labels":["lem:4.3:3","lem:4.3:4","lem:4.3:5","lem:4.3:6","lem:4.3:7"],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"势函数增量引理完整证明","source_units":[{"raw_tex":"\\begin{proof}\nFirst by \\eqref{lem:4.2:3:nopty} and \\eqref{lem:4.2:4:opty:k}, we have\n\t\\begin{align}\\label{lem:4.3:3}\n\t\t\\frac{1}{\\zeta}\\langle m_{k+1}, x_k -x_{k+1} \\rangle\n\t\t\\ge \\langle \\nabla _xf( x_k,y_k ) -\\nabla _xf\\left( x_{k-1},y_{k-1} \\right),x_{k+1}-x_k \\rangle,\n\t\\end{align}\nwhich together with \\eqref{lem:4.2:6} then imply that\n%\t\\eqref{lem:4.3:3} then implies that\n\t\\begin{align}\n\t\t\\frac{1}{\\zeta}\\langle m_{k+1},x_k-x_{k+1}  \\rangle  \\geq&\\langle  \\nabla _xf( x_{k},y_{k} ) -\\nabla _xf\\left( x_{k},y_{k-1} \\right) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t& + \\langle  \\nabla _xf\\left( x_{k},y_{k-1} \\right) -\\nabla _xf\\left( x_{k-1},y_{k-1} \\right) ,m_{k+1} \\rangle  \\nonumber\\\\\n\t\t&+\\langle  \\nabla _xf\\left( x_k,y_{k-1} \\right) -\\nabla _xf\\left( x_{k-1},y_{k-1} \\right) ,x_{k}-x_{k-1} \\rangle \\label{lem:4.3:4}\\\\\n\t\t\\geq& -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_{k} \\|^2-\\frac{\\zeta L_{11}^2}{2}\\|x_{k}-x_{k-1} \\|^2\\nonumber\\\\\n\t\t&-\\frac{1}{2\\zeta}\\| m_{k+1}\\|^2+\\theta\\|x_{k}-x_{k-1} \\|^2,\\label{lem:4.3:5}\n\t\\end{align}\nwhere the second inequality is similar to the proof of \\eqref{lem:4.2:7} except that for the first term in the r.h.s., we use $\\langle  \\nabla _xf( x_{k},y_{k} ) -\\nabla _xf\\left( x_{k},y_{k-1} \\right) ,x_{k+1}-x_{k} \\rangle \\geq -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_k \\|^2$. By using the identity $\\frac{1}{\\zeta}\\langle m_{k+1},x_{k}-x_{k+1} \\rangle  =\\frac{1}{2\\zeta}\\|x_{k}-x_{k-1}\\|^2- \\frac{1}{2\\zeta}\\|x_{k+1}-x_{k}\\|^2-\\frac{1}{2\\zeta}\\| m_{k+1}\\| ^2$, we conclude from \\eqref{lem:4.3:5} that\n\t\\begin{align}\\label{lem:4.3:6}\n\t\t&\\frac{1}{2\\zeta}\\|x_{k}-x_{k-1}\\|^2- \\frac{1}{2\\zeta}\\|x_{k+1}-x_{k}\\|^2-\\frac{1}{2\\zeta}\\| m_{k+1}\\| ^2\\nonumber\\\\\n\t\t\\geq& -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_k \\|^2-\\frac{\\zeta L_{11}^2}{2}\\|x_{k}-x_{k-1} \\|^2\\nonumber\\\\\n\t\t&-\\frac{1}{2\\zeta}\\| m_{k+1}\\|^2+\\theta\\|x_{k}-x_{k-1} \\|^2.\n\t\\end{align}\n\tRearranging the terms of \\eqref{lem:4.3:6}, we have\n\t\\begin{align}\\label{lem:4.3:7}\n\t\t&\\frac{1}{2\\zeta}\\|x_{k}-x_{k-1}\\|^2- \\frac{1}{2\\zeta}\\|x_{k+1}-x_{k}\\|^2\\nonumber\\\\\n\t\t\\geq& -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_{k} \\|^2+\\left(\\theta-\\frac{\\zeta L_{11}^2}{2}\\right)\\|x_{k}-x_{k-1} \\|^2.\n\t\\end{align}\n\tMultiplying $\\frac{4}{\\zeta\\theta}$ on both sides of \\eqref{lem:4.3:7} and using the definition of $\\hat{S}_{k+1}$, we obtain\n\t\\begin{align*}\n\t\t\\hat{S}_{k+1}-\\hat{S}_k \\geq & -\\frac{2L_{21}^{2}}{\\theta^2\\zeta}\\| y_{k}-y_{k-1}\\|^2-\\frac{2}{\\zeta}\\| x_{k+1}-x_k \\|^2+\\left(\\frac{4}{\\zeta}-\\frac{2 L_{11}^2}{\\theta}\\right)\\|x_{k}-x_{k-1} \\|^2.\n\t\\end{align*}\n\tThe proof is completed by \\eqref{lem:4.2:1:conclu} in Lemma \\ref{lem:4.2} and the definition of $\\hat{F}_k$.\n\\end{proof}","start":102351,"end":105155}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":["lem:4.3:3"],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"合并相邻原变量最优性","source_units":[{"raw_tex":"\\begin{proof}\nFirst by \\eqref{lem:4.2:3:nopty} and \\eqref{lem:4.2:4:opty:k}, we have\n\t\\begin{align}\\label{lem:4.3:3}\n\t\t\\frac{1}{\\zeta}\\langle m_{k+1}, x_k -x_{k+1} \\rangle\n\t\t\\ge \\langle \\nabla _xf( x_k,y_k ) -\\nabla _xf\\left( x_{k-1},y_{k-1} \\right),x_{k+1}-x_k \\rangle,\n\t\\end{align}\n","start":102351,"end":102635}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["lem:4.3:4","lem:4.3:5"],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"重新加权梯度差下界","source_units":[{"raw_tex":"which together with \\eqref{lem:4.2:6} then imply that\n%\t\\eqref{lem:4.3:3} then implies that\n\t\\begin{align}\n\t\t\\frac{1}{\\zeta}\\langle m_{k+1},x_k-x_{k+1}  \\rangle  \\geq&\\langle  \\nabla _xf( x_{k},y_{k} ) -\\nabla _xf\\left( x_{k},y_{k-1} \\right) ,x_{k+1}-x_{k} \\rangle  \\nonumber\\\\\n\t\t& + \\langle  \\nabla _xf\\left( x_{k},y_{k-1} \\right) -\\nabla _xf\\left( x_{k-1},y_{k-1} \\right) ,m_{k+1} \\rangle  \\nonumber\\\\\n\t\t&+\\langle  \\nabla _xf\\left( x_k,y_{k-1} \\right) -\\nabla _xf\\left( x_{k-1},y_{k-1} \\right) ,x_{k}-x_{k-1} \\rangle \\label{lem:4.3:4}\\\\\n\t\t\\geq& -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_{k} \\|^2-\\frac{\\zeta L_{11}^2}{2}\\|x_{k}-x_{k-1} \\|^2\\nonumber\\\\\n\t\t&-\\frac{1}{2\\zeta}\\| m_{k+1}\\|^2+\\theta\\|x_{k}-x_{k-1} \\|^2,\\label{lem:4.3:5}\n\t\\end{align}\nwhere the second inequality is similar to the proof of \\eqref{lem:4.2:7} except that for the first term in the r.h.s., we use $\\langle  \\nabla _xf( x_{k},y_{k} ) -\\nabla _xf\\left( x_{k},y_{k-1} \\right) ,x_{k+1}-x_{k} \\rangle \\geq -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_k \\|^2$. By using the identity $\\frac{1}{\\zeta}\\langle m_{k+1},x_{k}-x_{k+1} \\rangle  =\\frac{1}{2\\zeta}\\|x_{k}-x_{k-1}\\|^2- \\frac{1}{2\\zeta}\\|x_{k+1}-x_{k}\\|^2-\\frac{1}{2\\zeta}\\| m_{k+1}\\| ^2$, we conclude from \\eqref{lem:4.3:5} that\n","start":102635,"end":103958}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":["lem:4.3:6"],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"代入原变量范数恒等式","source_units":[{"raw_tex":"where the second inequality is similar to the proof of \\eqref{lem:4.2:7} except that for the first term in the r.h.s., we use $\\langle  \\nabla _xf( x_{k},y_{k} ) -\\nabla _xf\\left( x_{k},y_{k-1} \\right) ,x_{k+1}-x_{k} \\rangle \\geq -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_k \\|^2$. By using the identity $\\frac{1}{\\zeta}\\langle m_{k+1},x_{k}-x_{k+1} \\rangle  =\\frac{1}{2\\zeta}\\|x_{k}-x_{k-1}\\|^2- \\frac{1}{2\\zeta}\\|x_{k+1}-x_{k}\\|^2-\\frac{1}{2\\zeta}\\| m_{k+1}\\| ^2$, we conclude from \\eqref{lem:4.3:5} that\n\t\\begin{align}\\label{lem:4.3:6}\n\t\t&\\frac{1}{2\\zeta}\\|x_{k}-x_{k-1}\\|^2- \\frac{1}{2\\zeta}\\|x_{k+1}-x_{k}\\|^2-\\frac{1}{2\\zeta}\\| m_{k+1}\\| ^2\\nonumber\\\\\n\t\t\\geq& -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_k \\|^2-\\frac{\\zeta L_{11}^2}{2}\\|x_{k}-x_{k-1} \\|^2\\nonumber\\\\\n\t\t&-\\frac{1}{2\\zeta}\\| m_{k+1}\\|^2+\\theta\\|x_{k}-x_{k-1} \\|^2.\n\t\\end{align}\n","start":103417,"end":104333}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":["lem:4.3:7"],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"整理相邻原变量位移差","source_units":[{"raw_tex":"\tRearranging the terms of \\eqref{lem:4.3:6}, we have\n\t\\begin{align}\\label{lem:4.3:7}\n\t\t&\\frac{1}{2\\zeta}\\|x_{k}-x_{k-1}\\|^2- \\frac{1}{2\\zeta}\\|x_{k+1}-x_{k}\\|^2\\nonumber\\\\\n\t\t\\geq& -\\frac{L_{21}^{2}}{2 \\theta}\\|y_{k}-y_{k-1} \\|^2-\\frac{\\theta}{2}\\|x_{k+1}-x_{k} \\|^2+\\left(\\theta-\\frac{\\zeta L_{11}^2}{2}\\right)\\|x_{k}-x_{k-1} \\|^2.\n\t\\end{align}\n","start":104333,"end":104678}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"负位移项的增量下界","source_units":[{"raw_tex":"\tMultiplying $\\frac{4}{\\zeta\\theta}$ on both sides of \\eqref{lem:4.3:7} and using the definition of $\\hat{S}_{k+1}$, we obtain\n\t\\begin{align*}\n\t\t\\hat{S}_{k+1}-\\hat{S}_k \\geq & -\\frac{2L_{21}^{2}}{\\theta^2\\zeta}\\| y_{k}-y_{k-1}\\|^2-\\frac{2}{\\zeta}\\| x_{k+1}-x_k \\|^2+\\left(\\frac{4}{\\zeta}-\\frac{2 L_{11}^2}{\\theta}\\right)\\|x_{k}-x_{k-1} \\|^2.\n\t\\end{align*}\n","start":104678,"end":105034}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"合成势函数增量","source_units":[{"raw_tex":"\tThe proof is completed by \\eqref{lem:4.2:1:conclu} in Lemma \\ref{lem:4.2} and the definition of $\\hat{F}_k$.\n\\end{proof}","start":105034,"end":105155}]},{"id":"S060","semantic_type":"theorem","kind":"theorem","labels":["thm:4.1"],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"强凸非凹 AGP 驻点复杂度","source_units":[{"raw_tex":"\\begin{thm}\\label{thm:4.1}\n\tSuppose that Assumption \\ref{sec:2:ass1:lip} holds. Let $\\{\\left(x_{k},y_{k}\\right)\\}$ be a sequence generated by Algorithm \\ref{alg:2.1}\n\twith parameter settings in \\eqref{subsec:4.1:1:para}.\n\tIf the relations $\\nu>L_{22}, \\nu>L_{21}^2\\zeta+\\frac{4L_{21}^2}{\\zeta \\theta^2},~\\zeta\\leq \\frac{\\theta}{4L_{11}^2}$ are satisfied, then $\\forall \\varepsilon>0$,  it holds that $$ T\\left( \\varepsilon \\right) \\le\n\t\\tfrac{\\overline{F}-\\hat{F}_1}{\\hat{d}_1\\varepsilon ^2},$$\n\twhere\n\t$\\hat{d}_1:=\\tfrac{\\min \\left\\{\\tfrac{\\nu}{2}-\\tfrac{\\zeta L_{21}^2}{2}-\\tfrac{2 L_{21}^2}{\\zeta \\theta^2},\\tfrac{3\\theta-\\zeta L_{11}^2}{2}+\\tfrac{\\theta-4\\zeta L_{11}^2}{2\\zeta\\theta} \\right\\}}{\\max \\left\\{ \\tfrac{1}{\\zeta ^2}+2L_{12}^{2},2\\nu^2 \\right\\}}%\\overline{\\hat{F}}=\\overline{f}-(\\tfrac{\\theta}{2}-\\tfrac{3}{\\zeta})\\sigma _x^2\n\t$\n\tand $\\overline{F}=\\bar{f}+(\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\sigma_x^2$ with $\\overline{f} := \\max_{(x,y) \\in \\mathcal{X} \\times \\mathcal{Y}} f(x,y)$ and $\\sigma _x= \\max\\{\\|x_1-x_2\\| \\mid \\forall x_1, x_2\\in \\mathcal{X}\\}$.\n\\end{thm}","start":105682,"end":106802}]},{"id":"S061","semantic_type":"proof","kind":"proof","labels":["thm:4.1:1:gx","thm:4.1:2:gy","thm:4.1:3:g","thm:4.1:4:d1","thm:4.1:5"],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"主定理完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\tBy \\eqref{subsec:4.1:2:x}, we immediately obtain\n\t\\begin{equation}\n\t\t\\|(\\nabla G_k)_{x} \\|\\leq\\frac{1}{\\zeta}\\| x_{k+1}-x_{k} \\|.\\label{thm:4.1:1:gx}\n\t\\end{equation}\n\tOn the other hand, similar to the proof of \\eqref{thm:3.1:2:gy},  by \\eqref{subsec:4.1:3:y} and the triangle inequality, and the nonexpansiveness of the projection operator $\\operatorname{ \\mathcal{P}_\\mathcal{Y}}$, we conclude that\n\t\\begin{align}\\label{thm:4.1:2:gy}\n\t\t\\| (\\nabla G_k)_{y} \\|\\le& \\nu\\| y_{k+1}-y_k \\|+L_{12}\\|x_{k+1}-x_{k} \\|.\n\t\t\t\\end{align}\n%\t\t\\leq&\\nu\\|\\operatorname{ Prox_{\\mathcal{Y}_j}^{\\nu}}\\left( y_j^r+\\frac{1}{\\nu} \\nabla _{y_j}f\\left( x^{r+1} ,w_j^{r+1} \\right) \\right)-\\operatorname{ Prox_{\\mathcal{Y}_j}^{\\nu}}\\left( y_j^r+ \\frac{1}{\\nu} \\nabla _{y_j}f\\left( x^r,y^r \\right) \\right) \\|\\nonumber\\\\\n%\t\t&+\\nu\\| y_j^{r+1}-y_j^r \\|\\nonumber\\\\\n%\t\t\\leq& \\nu\\| y_j^{r+1}-y_j^r \\| +\\|\\nabla _{y_j}f\\left( x^{r+1} ,w_j^{r+1} \\right)-\\nabla _{y_j}f\\left( x^r,y^r \\right) \\|\\nonumber\\\\\n%\t\t\\leq & \\nu\\| y_j^{r+1}-y_j^r \\| +\\|\\nabla _{y_j}f\\left( x^{r+1} ,w_j^{r+1} \\right)-\\nabla _{y_j}f\\left( x^r,w_j^{r+1} \\right) \\|\\nonumber\\\\\n%\t\t& +\\|\\nabla _{y_j}f\\left( x^r ,w_j^{r+1} \\right)-\\nabla _{y_j}f\\left( x^r,y^r \\right) \\|\\nonumber\\\\\n%\t\t\\le& \\nu\\| y_j^{r+1}-y_j^r \\|+L_{12}\\|x^{r+1}-x^r \\|+L_{22}\\|w_j^{r+1}-y^r \\|\\nonumber\\\\\n%\twhere the second inequality holds by the nonexpansiveness of the proximal operator $\\operatorname{ Prox_{\\mathcal{Y}_j}^{\\nu}}$, the last inequality holds since that $\\| y_i^{r+1}-y_i^r \\|\\le\\| y^{r+1}-y^r \\|$ and $\\|w_j^{r+1}-y^r \\|\\le\\| y^{r+1}-y^r \\|$ by the definition of $y_j$ and $w_j^{r+1}$.\n\tBy combining \\eqref{thm:4.1:1:gx} and \\eqref{thm:4.1:2:gy}, and using the Cauchy-Schwarz inequality, we obtain\n\t\\begin{align}\\label{thm:4.1:3:g}\n\t\t\\| \\nabla G_k \\|^2\n\t\t\\leq 2\\nu^2\\| y_{k+1}-y_{k} \\|^2+\\left( \\frac{1}{\\zeta ^2}+2L_{12}^{2} \\right)\\|x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\n\tObserving that $\\hat{d}_1 > 0$. Multiplying both sides of \\eqref{thm:4.1:3:g} by $\\hat{d}_1$, and using \\eqref{lem:4.3:2:conclus} in Lemma \\ref{lem:4.3},\n\twe have\n\t\\begin{align}\\label{thm:4.1:4:d1}\n\t\t\\hat{d}_1\\|\\nabla G_k \\|^2\\le \\hat{F}_{k+1}-\\hat{F}_k.\n\t\\end{align}\n\tSumming up the above inequalities from $k=1$ to $k=T(\\varepsilon)$, we obtain\n\t\\begin{align}\\label{thm:4.1:5}\n\t\t\\sum_{k=1}^{T\\left( \\varepsilon \\right)}{\\hat{d}_1\\|\\nabla G_k \\| ^2}\\le \\hat{F}_{T\\left( \\varepsilon \\right)+1}-\\hat{F}_1.\n\t\\end{align}\n\tNote that by the definition of $\\hat{F}_{k+1}$ in Lemma \\ref{lem:4.3}, we have\n\t\\begin{align*}\n\t\t\\hat{F}_{T\\left( \\varepsilon \\right)+1}&\\le\n\t\t\\hat{f}_{T\\left( \\varepsilon \\right)+1}+(\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\|x_{T\\left( \\varepsilon \\right)+1}-x_{T\\left( \\varepsilon \\right)} \\|^2\\\\\n\t\t&\\le \\bar{f}+(\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\sigma_x^2 = \\bar{F},\n\t\\end{align*}\n\twhere the last inequality follows from the definitions of $\\bar{f}$ and  $\\sigma _x$, and $\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta} \\ge 0 $ due to\n\tthe selection of $\\zeta$.\n\tWe then conclude from \\eqref{thm:4.1:5} that $\\sum_{k=1}^{T\\left( \\varepsilon \\right)}{\\hat{d}_1\\| \\nabla G_k \\|^2}\\le \\bar{F}-\\hat{F}_1$ which, in view of the definition of $T(\\varepsilon)$, implies that $\\varepsilon ^2\\le (\\bar{F}-\\hat{F}_1)/(T( \\varepsilon )\\cdot \\hat{d}_1)$\n\tor equivalently, $T\\left( \\varepsilon \\right) \\le (\\bar{F}-\\hat{F}_1)/(\\hat{d}_1\\varepsilon ^2)$.\n\\end{proof}","start":106804,"end":110287}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":["thm:4.1:1:gx"],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"原变量梯度映射位移界","source_units":[{"raw_tex":"\\begin{proof}\n\tBy \\eqref{subsec:4.1:2:x}, we immediately obtain\n\t\\begin{equation}\n\t\t\\|(\\nabla G_k)_{x} \\|\\leq\\frac{1}{\\zeta}\\| x_{k+1}-x_{k} \\|.\\label{thm:4.1:1:gx}\n\t\\end{equation}\n","start":106804,"end":106985}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":["thm:4.1:2:gy"],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"对偶梯度映射位移界","source_units":[{"raw_tex":"\tOn the other hand, similar to the proof of \\eqref{thm:3.1:2:gy},  by \\eqref{subsec:4.1:3:y} and the triangle inequality, and the nonexpansiveness of the projection operator $\\operatorname{ \\mathcal{P}_\\mathcal{Y}}$, we conclude that\n\t\\begin{align}\\label{thm:4.1:2:gy}\n\t\t\\| (\\nabla G_k)_{y} \\|\\le& \\nu\\| y_{k+1}-y_k \\|+L_{12}\\|x_{k+1}-x_{k} \\|.\n\t\t\t\\end{align}\n","start":106985,"end":107345}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":["thm:4.1:3:g"],"member_ids":["S065"],"declared_external":false,"source_only":false,"title":"完整梯度映射平方界","source_units":[{"raw_tex":"\tBy combining \\eqref{thm:4.1:1:gx} and \\eqref{thm:4.1:2:gy}, and using the Cauchy-Schwarz inequality, we obtain\n\t\\begin{align}\\label{thm:4.1:3:g}\n\t\t\\| \\nabla G_k \\|^2\n\t\t\\leq 2\\nu^2\\| y_{k+1}-y_{k} \\|^2+\\left( \\frac{1}{\\zeta ^2}+2L_{12}^{2} \\right)\\|x_{k+1}-x_{k} \\|^2.\n\t\\end{align}\n","start":108428,"end":108710}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":["thm:4.1:4:d1"],"member_ids":["S066"],"declared_external":false,"source_only":false,"title":"势增量控制驻点间隙","source_units":[{"raw_tex":"\tObserving that $\\hat{d}_1 > 0$. Multiplying both sides of \\eqref{thm:4.1:3:g} by $\\hat{d}_1$, and using \\eqref{lem:4.3:2:conclus} in Lemma \\ref{lem:4.3},\n\twe have\n\t\\begin{align}\\label{thm:4.1:4:d1}\n\t\t\\hat{d}_1\\|\\nabla G_k \\|^2\\le \\hat{F}_{k+1}-\\hat{F}_k.\n\t\\end{align}\n","start":108710,"end":108979}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":["thm:4.1:5"],"member_ids":["S067"],"declared_external":false,"source_only":false,"title":"驻点间隙的望远镜求和","source_units":[{"raw_tex":"\tSumming up the above inequalities from $k=1$ to $k=T(\\varepsilon)$, we obtain\n\t\\begin{align}\\label{thm:4.1:5}\n\t\t\\sum_{k=1}^{T\\left( \\varepsilon \\right)}{\\hat{d}_1\\|\\nabla G_k \\| ^2}\\le \\hat{F}_{T\\left( \\varepsilon \\right)+1}-\\hat{F}_1.\n\t\\end{align}\n","start":108979,"end":109229}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S068"],"declared_external":false,"source_only":false,"title":"末端势函数的统一上界","source_units":[{"raw_tex":"\tNote that by the definition of $\\hat{F}_{k+1}$ in Lemma \\ref{lem:4.3}, we have\n\t\\begin{align*}\n\t\t\\hat{F}_{T\\left( \\varepsilon \\right)+1}&\\le\n\t\t\\hat{f}_{T\\left( \\varepsilon \\right)+1}+(\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\|x_{T\\left( \\varepsilon \\right)+1}-x_{T\\left( \\varepsilon \\right)} \\|^2\\\\\n\t\t&\\le \\bar{f}+(\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta})\\sigma_x^2 = \\bar{F},\n\t\\end{align*}\n\twhere the last inequality follows from the definitions of $\\bar{f}$ and  $\\sigma _x$, and $\\theta+\\frac{7}{2\\zeta}-\\frac{\\zeta L_{11}^2}{2}-\\frac{2L_{11}^2}{\\theta} \\ge 0 $ due to\n\tthe selection of $\\zeta$.\n","start":109229,"end":109897}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S069"],"declared_external":false,"source_only":false,"title":"作者的首次命中时间结论","source_units":[{"raw_tex":"\tWe then conclude from \\eqref{thm:4.1:5} that $\\sum_{k=1}^{T\\left( \\varepsilon \\right)}{\\hat{d}_1\\| \\nabla G_k \\|^2}\\le \\bar{F}-\\hat{F}_1$ which, in view of the definition of $T(\\varepsilon)$, implies that $\\varepsilon ^2\\le (\\bar{F}-\\hat{F}_1)/(T( \\varepsilon )\\cdot \\hat{d}_1)$\n\tor equivalently, $T\\left( \\varepsilon \\right) \\le (\\bar{F}-\\hat{F}_1)/(\\hat{d}_1\\varepsilon ^2)$.\n\\end{proof}","start":109897,"end":110287}]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S071"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第1项：Optimality characterization and nonexpansiveness of Euclidean projection onto a nonempty closed convex set."],"source_only":false,"title":"欧氏投影最优性刻画","source_units":[]},{"id":"S072","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S072"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第1项：Optimality characterization and nonexpansiveness of Euclidean projection onto a nonempty closed convex set."],"source_only":false,"title":"欧氏投影非扩张性","source_units":[]},{"id":"S073","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S073"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第2项：Smooth descent inequalities and strong monotonicity of gradients of strongly convex functions."],"source_only":false,"title":"光滑函数二次变化界","source_units":[]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S074"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第2项：Smooth descent inequalities and strong monotonicity of gradients of strongly convex functions."],"source_only":false,"title":"强凸梯度的强单调性","source_units":[]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S075"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第3项：Cauchy–Schwarz, Young's inequality, triangle inequality, and elementary telescoping identities."],"source_only":false,"title":"基本内积与求和工具","source_units":[]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S076"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第4项：Existence of extrema of continuous functions on compact sets."],"source_only":false,"title":"紧集上连续函数取得极值","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076"}}}}
{"record_id":"2303.03534v5","paper":{"title":"Global convergence of the gradient method for functions definable in o-minimal structures","authors":["Josz, Cédric"],"arxiv_id":"2303.03534","version":"v5","journal":"Mathematical Programming","abstract_url":"https://arxiv.org/abs/2303.03534v5","pdf_url":"https://arxiv.org/pdf/2303.03534v5","doi":"10.1007/s10107-023-01937-5"},"context":{"raw_tex":"Let $\\mathbb{N} := \\{0,1,2,\\hdots\\}$ and let $\\|\\cdot\\|$ be the induced norm of an inner product $\\langle \\cdot, \\cdot\\rangle$ on $\\mathbb{R}^n$. Throughout this paper, we fix an arbitrary o-minimal structure on the real field (see Definition \\ref{def:o-minimal}), and say that sets or functions are definable if they are definable in this structure. For readers not familiar with o-minimal structures, note that seemingly all objective functions used in practice are definable in some o-minimal structure. Low-rank matrix recovery (for e.g., matrix factorization, matrix completion, and matrix sensing) and deep neural networks are no exception \\cite[Section 6.2]{bolte2020conservative}.\n\n\nWe begin by stating some standard definitions. Let $B(a,r)$ and $S(a,r)$ respectively denote the closed ball and the sphere of center $a\\in \\mathbb{R}^n$ and radius $r \\geqslant 0$.\n\n\\begin{definition}\n\\label{def:lipschitz}\nA function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}^m$ is locally Lipschitz if for all $a \\in \\mathbb{R}^n$, there exist $r>0$ and $L>0$ such that \n\\begin{equation*}\n        \\forall x,y \\in B(a,r), ~~~ \\|f(x)-f(y)\\| \\leqslant L \\|x-y\\|.\n    \\end{equation*}\n\\end{definition}\n\n\\begin{definition}\n    \\label{def:Clarke}\n    \\cite[Chapter 2]{clarke1990}\n    Let $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz function. The Clarke subdifferential is the set-valued mapping $\\partial f:\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^n$ defined for all $x \\in \\mathbb{R}^n$ by\n    $\\partial f(x) := \\{ s \\in \\mathbb{R}^n : f^\\circ(x,h) \\geqslant \\langle s , h \\rangle, ~ \\forall h\\in \\mathbb{R}^n \\}$ where\n\\begin{equation*}\n    f^\\circ(x,h) := \\limsup_{\\tiny\\begin{array}{c} y\\rightarrow x \\\\\n    t \\searrow 0\n    \\end{array}\n    } \\frac{f(y+th)-f(y)}{t}.\n\\end{equation*}\n\\end{definition}\n\nWe say that $x \\in \\mathbb{R}^n$ is critical if $0 \\in \\partial f(x)$. If $f$ is continuously differentiable in a neighborhood of $x \\in \\mathbb{R}^n$, then $\\partial f(x) = \\{\\nabla f(x)\\}$ \\cite[2.2.4 Proposition p. 33]{clarke1990}.\n\n\n\\begin{definition}\n    \\label{def:absolute_continuity} \n    \\cite[Definition 1 p. 12]{aubin1984differential}\n    Given some real numbers $a\\leqslant b$, a function $x:[a,b]\\rightarrow\\mathbb{R}^n$ is absolutely continuous if for all $\\epsilon>0$, there exists $\\delta>0$ such that, for any finite collection of disjoint subintervals $[a_1,b_1],\\hdots,[a_m,b_m]$ of $[a,b]$ such that $\\sum_{i=1}^m b_i-a_i \\leqslant \\delta$, we have $\\sum_{i=1}^m \\|x(b_i) - x(a_i)\\| \\leqslant \\epsilon$.\n\\end{definition}\n\nBy virtue of \\cite[Theorem 20.8]{nielsen1997introduction}, $x:[a,b]\\rightarrow\\mathbb{R}^n$ is absolutely continuous if and only if it is differentiable almost everywhere on $(a,b)$, its derivative $x'(\\cdot)$ is Lebesgue integrable, and $x(t) - x(a) = \\int_a^t x'(\\tau)d\\tau$ for all $t\\in [a,b]$. Given a noncompact interval $I$ of $\\mathbb{R}$, $x:I\\rightarrow \\mathbb{R}^n$ is absolutely continuous if it is absolutely continuous on all compact subintervals of $I$. Given an interval $I$ of $\\mathbb{R}$, let $\\mathcal{A}(I,\\mathbb{R}^n)$ be the set of absolutely continuous functions defined from $I$ to $\\mathbb{R}^n$.\n\n\\begin{definition}\n\\label{def:trajectory}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz function. Let $(S,\\leqslant)$ be the set of absolutely continuous functions $x:[0,T)\\rightarrow\\mathbb{R}^n$ where $0<T\\leqslant \\infty$ such that  \n\\begin{equation*}\n    x'(t) \\in -\\partial f(x(t)), ~~~\\mathrm{for~a.e.}~t\\in (0,T),\n\\end{equation*}\nequipped with the partial order $\\leqslant$ defined by $x_1:[0,T_1)\\rightarrow\\mathbb{R}^n$ is less than or equal to $x_2:[0,T_2)\\rightarrow\\mathbb{R}^n$ if and only if $T_1\\leqslant T_2$ and $x_1(t) = x_2(t)$ for all $t\\in [0,T_1)$. \nWe refer to maximal elements of $(S,\\leqslant)$ as continuous subgradient trajectories.\n\\end{definition}\n\nWe say that a continuous subgradient trajectory is globally defined if it is of the form $x:[0,T)\\rightarrow\\mathbb{R}^n$ where $T=\\infty$. We refer to discrete subgradient trajectories as sequences $(x_k)_{k\\in \\mathbb{N}}$ in $\\mathbb{R}^n$ such that $x_{k+1} \\in x_k -\\alpha_k \\partial f(x_k)$ for all $k\\in \\mathbb{N}$ for some $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots>0$. When $f$ is continuously differentiable, we refer to subgradient trajectories as  gradient trajectories.\n\n\\begin{definition}\n\\label{def:bounded_trajectory}\nA continuous subgradient trajectory $x: [0,T) \\rightarrow \\mathbb{R}^n$ where $0< T\\leqslant \\infty$ is bounded if there exists $c>0$ such that $\\|x(t)\\| \\leqslant c$ for all $t\\in [0,T)$.\n\\end{definition}\n\n\n\\begin{remark}\n\\label{rem:tracking}\nIf $X_0$ is a singleton $\\{x_0\\}$ and $x:[0,\\hat{T})\\rightarrow \\mathbb{R}^n$ is a gradient trajectory initialized at $x_0$, then one may remove the assumption that $f$ is lower bounded in Proposition \\ref{prop:tracking} and instead require that $T \\in (0,\\hat{T})$. In that case, $\\sigma_T(X_0)=\\int_0^T \\|x'(t)\\|dt < \\infty$ since $\\nabla f$ is Lipschitz continuous on the bounded set $x([0,T])$.\n\\end{remark}\n\n\\begin{definition}\n\\label{def:o-minimal}\n\\cite[Definition p. 503-506]{van1996geometric}\nAn o-minimal structure on the real field is a sequence $S = (S_k)_{k \\in \\mathbb{N}}$ such that for all $k \\in \\mathbb{N}$:\n\\begin{enumerate}\n    \\item $S_k$ is a boolean algebra of subsets of $\\mathbb{R}^k$, with $\\mathbb{R}^k \\in S_k$;\n    \\item $S_k$ contains the diagonal $\\{(x_1,\\hdots,x_k) \\in \\mathbb{R}^k : x_i = x_j\\}$ for $1\\leqslant i<j \\leqslant k$;\n    \\item If $A\\in S_k$, then $A\\times \\mathbb{R}$ and $\\mathbb{R}\\times A$ belong to $S_{k+1}$;\n    \\item If $A \\in S_{k+1}$ and $\\pi:\\mathbb{R}^{k+1}\\rightarrow\\mathbb{R}^k$ is the projection onto the first $k$ coordinates, then $\\pi(A) \\in S_k$;\n    \\item $S_3$ contains the graphs of addition and multiplication;\n    \\item $S_1$ consists exactly of the finite unions of open intervals and singletons. \n\\end{enumerate}\n\\end{definition}\n\nA subset $A$ of $\\mathbb{R}^n$ is definable in an o-minimal structure $(S_k)_{k\\in\\mathbb{N}}$ if $A \\in S_k$. A function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ is definable in an o-minimal structure if its graph, that is to say $\\{ (x,t) \\in \\mathbb{R}^{n+1} : f(x)=t \\}$, is definable in that structure.\n\n\nWe next state the uniform Kurdyka-\\L{}ojasiewicz inequality. Given a subset of $S$ of $\\mathbb{R}^n$, let $\\mathring{S}$ and $\\overline{S}$ denote the interior and closure of $S$ in $\\mathbb{R}^n$ respectively. A function $\\psi:S \\rightarrow S$ is a homeomorphism if it is a continuous bijection and the inverse function $\\psi^{-1}$ is continuous. $\\psi:S \\rightarrow S$ is a diffeomorphism if $\\mathring{S} \\neq \\emptyset$, $\\psi$ is a homeomorphism, and both $\\psi$ and $\\psi^{-1}$ are continuously differentiable on $\\mathring{S}$. Given $x\\in \\mathbb{R}^n$, consider the distance of $x$ to $S$ defined by $d(x,S) := \\inf \\{ \\|x-y\\| : y \\in S \\}$. Given a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$, a real number $v$ is critical value of $f$ in $S$ if there exists $x \\in S$ such that $v = f(x)$ and $0\\in \\partial f(x)$. Given a set-valued mapping $F:\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^m$ and $y \\in \\mathbb{R}^m$, let $F^{-1}(y) := \\{ x \\in \\mathbb{R}^n : F(x) \\ni y \\}$.\n\n\nWe say that $\\psi$ in Proposition \\ref{prop:KL} is a desingularizing function of $f$ on $X$. In order to prove Proposition \\ref{prop:length_continuous}, we recall the following result.\n\n\nAn immediate consequence of Propositions \\ref{prop:length_continuous} and \\ref{prop:length_discrete} is that bounded continuous subgradient trajectories and bounded discrete gradient trajectories have finite length, as is well known. An easy fact that we will not prove here is that continuous subgradient trajectories of finite length converge to critical points of the objective function under the sole assumption that it is locally Lipschitz. The same holds for discrete subgradient trajectories if the step sizes are not summable.\n","units":[{"raw_tex":"Let $\\mathbb{N} := \\{0,1,2,\\hdots\\}$ and let $\\|\\cdot\\|$ be the induced norm of an inner product $\\langle \\cdot, \\cdot\\rangle$ on $\\mathbb{R}^n$. Throughout this paper, we fix an arbitrary o-minimal structure on the real field (see Definition \\ref{def:o-minimal}), and say that sets or functions are definable if they are definable in this structure. For readers not familiar with o-minimal structures, note that seemingly all objective functions used in practice are definable in some o-minimal structure. Low-rank matrix recovery (for e.g., matrix factorization, matrix completion, and matrix sensing) and deep neural networks are no exception \\cite[Section 6.2]{bolte2020conservative}.\n","start":4045,"end":4734},{"raw_tex":"We begin by stating some standard definitions. Let $B(a,r)$ and $S(a,r)$ respectively denote the closed ball and the sphere of center $a\\in \\mathbb{R}^n$ and radius $r \\geqslant 0$.\n\n\\begin{definition}\n\\label{def:lipschitz}\nA function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}^m$ is locally Lipschitz if for all $a \\in \\mathbb{R}^n$, there exist $r>0$ and $L>0$ such that \n\\begin{equation*}\n        \\forall x,y \\in B(a,r), ~~~ \\|f(x)-f(y)\\| \\leqslant L \\|x-y\\|.\n    \\end{equation*}\n\\end{definition}\n\n\\begin{definition}\n    \\label{def:Clarke}\n    \\cite[Chapter 2]{clarke1990}\n    Let $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz function. The Clarke subdifferential is the set-valued mapping $\\partial f:\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^n$ defined for all $x \\in \\mathbb{R}^n$ by\n    $\\partial f(x) := \\{ s \\in \\mathbb{R}^n : f^\\circ(x,h) \\geqslant \\langle s , h \\rangle, ~ \\forall h\\in \\mathbb{R}^n \\}$ where\n\\begin{equation*}\n    f^\\circ(x,h) := \\limsup_{\\tiny\\begin{array}{c} y\\rightarrow x \\\\\n    t \\searrow 0\n    \\end{array}\n    } \\frac{f(y+th)-f(y)}{t}.\n\\end{equation*}\n\\end{definition}\n\nWe say that $x \\in \\mathbb{R}^n$ is critical if $0 \\in \\partial f(x)$. If $f$ is continuously differentiable in a neighborhood of $x \\in \\mathbb{R}^n$, then $\\partial f(x) = \\{\\nabla f(x)\\}$ \\cite[2.2.4 Proposition p. 33]{clarke1990}.\n","start":26428,"end":27776},{"raw_tex":"\\begin{definition}\n    \\label{def:absolute_continuity} \n    \\cite[Definition 1 p. 12]{aubin1984differential}\n    Given some real numbers $a\\leqslant b$, a function $x:[a,b]\\rightarrow\\mathbb{R}^n$ is absolutely continuous if for all $\\epsilon>0$, there exists $\\delta>0$ such that, for any finite collection of disjoint subintervals $[a_1,b_1],\\hdots,[a_m,b_m]$ of $[a,b]$ such that $\\sum_{i=1}^m b_i-a_i \\leqslant \\delta$, we have $\\sum_{i=1}^m \\|x(b_i) - x(a_i)\\| \\leqslant \\epsilon$.\n\\end{definition}\n\nBy virtue of \\cite[Theorem 20.8]{nielsen1997introduction}, $x:[a,b]\\rightarrow\\mathbb{R}^n$ is absolutely continuous if and only if it is differentiable almost everywhere on $(a,b)$, its derivative $x'(\\cdot)$ is Lebesgue integrable, and $x(t) - x(a) = \\int_a^t x'(\\tau)d\\tau$ for all $t\\in [a,b]$. Given a noncompact interval $I$ of $\\mathbb{R}$, $x:I\\rightarrow \\mathbb{R}^n$ is absolutely continuous if it is absolutely continuous on all compact subintervals of $I$. Given an interval $I$ of $\\mathbb{R}$, let $\\mathcal{A}(I,\\mathbb{R}^n)$ be the set of absolutely continuous functions defined from $I$ to $\\mathbb{R}^n$.\n\n\\begin{definition}\n\\label{def:trajectory}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz function. Let $(S,\\leqslant)$ be the set of absolutely continuous functions $x:[0,T)\\rightarrow\\mathbb{R}^n$ where $0<T\\leqslant \\infty$ such that  \n\\begin{equation*}\n    x'(t) \\in -\\partial f(x(t)), ~~~\\mathrm{for~a.e.}~t\\in (0,T),\n\\end{equation*}\nequipped with the partial order $\\leqslant$ defined by $x_1:[0,T_1)\\rightarrow\\mathbb{R}^n$ is less than or equal to $x_2:[0,T_2)\\rightarrow\\mathbb{R}^n$ if and only if $T_1\\leqslant T_2$ and $x_1(t) = x_2(t)$ for all $t\\in [0,T_1)$. \nWe refer to maximal elements of $(S,\\leqslant)$ as continuous subgradient trajectories.\n\\end{definition}\n\nWe say that a continuous subgradient trajectory is globally defined if it is of the form $x:[0,T)\\rightarrow\\mathbb{R}^n$ where $T=\\infty$. We refer to discrete subgradient trajectories as sequences $(x_k)_{k\\in \\mathbb{N}}$ in $\\mathbb{R}^n$ such that $x_{k+1} \\in x_k -\\alpha_k \\partial f(x_k)$ for all $k\\in \\mathbb{N}$ for some $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots>0$. When $f$ is continuously differentiable, we refer to subgradient trajectories as  gradient trajectories.\n\n\\begin{definition}\n\\label{def:bounded_trajectory}\nA continuous subgradient trajectory $x: [0,T) \\rightarrow \\mathbb{R}^n$ where $0< T\\leqslant \\infty$ is bounded if there exists $c>0$ such that $\\|x(t)\\| \\leqslant c$ for all $t\\in [0,T)$.\n\\end{definition}\n","start":28557,"end":31120},{"raw_tex":"\\begin{remark}\n\\label{rem:tracking}\nIf $X_0$ is a singleton $\\{x_0\\}$ and $x:[0,\\hat{T})\\rightarrow \\mathbb{R}^n$ is a gradient trajectory initialized at $x_0$, then one may remove the assumption that $f$ is lower bounded in Proposition \\ref{prop:tracking} and instead require that $T \\in (0,\\hat{T})$. In that case, $\\sigma_T(X_0)=\\int_0^T \\|x'(t)\\|dt < \\infty$ since $\\nabla f$ is Lipschitz continuous on the bounded set $x([0,T])$.\n\\end{remark}","start":44646,"end":45093},{"raw_tex":"\\begin{definition}\n\\label{def:o-minimal}\n\\cite[Definition p. 503-506]{van1996geometric}\nAn o-minimal structure on the real field is a sequence $S = (S_k)_{k \\in \\mathbb{N}}$ such that for all $k \\in \\mathbb{N}$:\n\\begin{enumerate}\n    \\item $S_k$ is a boolean algebra of subsets of $\\mathbb{R}^k$, with $\\mathbb{R}^k \\in S_k$;\n    \\item $S_k$ contains the diagonal $\\{(x_1,\\hdots,x_k) \\in \\mathbb{R}^k : x_i = x_j\\}$ for $1\\leqslant i<j \\leqslant k$;\n    \\item If $A\\in S_k$, then $A\\times \\mathbb{R}$ and $\\mathbb{R}\\times A$ belong to $S_{k+1}$;\n    \\item If $A \\in S_{k+1}$ and $\\pi:\\mathbb{R}^{k+1}\\rightarrow\\mathbb{R}^k$ is the projection onto the first $k$ coordinates, then $\\pi(A) \\in S_k$;\n    \\item $S_3$ contains the graphs of addition and multiplication;\n    \\item $S_1$ consists exactly of the finite unions of open intervals and singletons. \n\\end{enumerate}\n\\end{definition}\n\nA subset $A$ of $\\mathbb{R}^n$ is definable in an o-minimal structure $(S_k)_{k\\in\\mathbb{N}}$ if $A \\in S_k$. A function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ is definable in an o-minimal structure if its graph, that is to say $\\{ (x,t) \\in \\mathbb{R}^{n+1} : f(x)=t \\}$, is definable in that structure.\n","start":46181,"end":47376},{"raw_tex":"We next state the uniform Kurdyka-\\L{}ojasiewicz inequality. Given a subset of $S$ of $\\mathbb{R}^n$, let $\\mathring{S}$ and $\\overline{S}$ denote the interior and closure of $S$ in $\\mathbb{R}^n$ respectively. A function $\\psi:S \\rightarrow S$ is a homeomorphism if it is a continuous bijection and the inverse function $\\psi^{-1}$ is continuous. $\\psi:S \\rightarrow S$ is a diffeomorphism if $\\mathring{S} \\neq \\emptyset$, $\\psi$ is a homeomorphism, and both $\\psi$ and $\\psi^{-1}$ are continuously differentiable on $\\mathring{S}$. Given $x\\in \\mathbb{R}^n$, consider the distance of $x$ to $S$ defined by $d(x,S) := \\inf \\{ \\|x-y\\| : y \\in S \\}$. Given a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$, a real number $v$ is critical value of $f$ in $S$ if there exists $x \\in S$ such that $v = f(x)$ and $0\\in \\partial f(x)$. Given a set-valued mapping $F:\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^m$ and $y \\in \\mathbb{R}^m$, let $F^{-1}(y) := \\{ x \\in \\mathbb{R}^n : F(x) \\ni y \\}$.\n","start":48364,"end":49369},{"raw_tex":"We say that $\\psi$ in Proposition \\ref{prop:KL} is a desingularizing function of $f$ on $X$. In order to prove Proposition \\ref{prop:length_continuous}, we recall the following result.\n","start":52714,"end":52899},{"raw_tex":"An immediate consequence of Propositions \\ref{prop:length_continuous} and \\ref{prop:length_discrete} is that bounded continuous subgradient trajectories and bounded discrete gradient trajectories have finite length, as is well known. An easy fact that we will not prove here is that continuous subgradient trajectories of finite length converge to critical points of the objective function under the sole assumption that it is locally Lipschitz. The same holds for discrete subgradient trajectories if the step sizes are not summable.\n","start":64646,"end":65181}]},"statement":{"raw_tex":"\\begin{theorem}\n\\label{thm:convergence}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a definable differentiable function whose gradient is locally Lipschitz. The following statements are equivalent:\n\\begin{enumerate}\n    \\item For all $x_0 \\in \\mathbb{R}^n$, there exist $\\bar{\\alpha}>0$ and $c>0$ such that for all $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$, the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by \n    \\begin{equation}\n    \\label{eq:gradient_method}\n        x_{k+1} = x_k - \\alpha_k \\nabla f(x_k), ~~~\\forall k \\in \\mathbb{N},\n    \\end{equation}\n    satisfies $\\|x_k\\|\\leqslant c$ for all $k \\in \\mathbb{N}$.\n    \\item For all $0< T \\leqslant \\infty$ and any differentiable function $x:[0,T)\\rightarrow\\mathbb{R}^n$ such that\n    \\begin{equation*}\n        x'(t) = -\\nabla f(x(t)), ~~~ \\forall t \\in (0,T),\n    \\end{equation*}\n    there exists $c>0$ such that $\\|x(t)\\| \\leqslant c$ for all $t\\in [0,T)$.\n    \\item For any bounded subset $X_0$ of $\\mathbb{R}^n$, there exist $\\bar{\\alpha}>0$ and $c>0$ such that, for all $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$ and $x_0 \\in X_0$, the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by \\eqref{eq:gradient_method} obeys \n    \\begin{equation*}\n        \\sum_{k=0}^{\\infty} \\|x_{k+1}-x_k\\| \\leqslant c.\n    \\end{equation*}\n\\end{enumerate}\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\n\\label{thm:convergence}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a definable differentiable function whose gradient is locally Lipschitz. The following statements are equivalent:\n\\begin{enumerate}\n    \\item For all $x_0 \\in \\mathbb{R}^n$, there exist $\\bar{\\alpha}>0$ and $c>0$ such that for all $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$, the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by \n    \\begin{equation}\n    \\label{eq:gradient_method}\n        x_{k+1} = x_k - \\alpha_k \\nabla f(x_k), ~~~\\forall k \\in \\mathbb{N},\n    \\end{equation}\n    satisfies $\\|x_k\\|\\leqslant c$ for all $k \\in \\mathbb{N}$.\n    \\item For all $0< T \\leqslant \\infty$ and any differentiable function $x:[0,T)\\rightarrow\\mathbb{R}^n$ such that\n    \\begin{equation*}\n        x'(t) = -\\nabla f(x(t)), ~~~ \\forall t \\in (0,T),\n    \\end{equation*}\n    there exists $c>0$ such that $\\|x(t)\\| \\leqslant c$ for all $t\\in [0,T)$.\n    \\item For any bounded subset $X_0$ of $\\mathbb{R}^n$, there exist $\\bar{\\alpha}>0$ and $c>0$ such that, for all $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$ and $x_0 \\in X_0$, the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by \\eqref{eq:gradient_method} obeys \n    \\begin{equation*}\n        \\sum_{k=0}^{\\infty} \\|x_{k+1}-x_k\\| \\leqslant c.\n    \\end{equation*}\n\\end{enumerate}\n\\end{theorem}","start":4735,"end":6096}]},"proof":{"raw_tex":"\\begin{lemma}\n\\label{lemma:uniform_length}\nIf a locally Lipschitz definable function has bounded continuous subgradient trajectories, then continuous subgradient trajectories initialized in a bounded set have uniformly bounded lengths.\n\\end{lemma}\n\n\\begin{proposition}\n\\label{prop:existence_trajectory}\nLocally Lipschitz functions have at least one continuous subgradient trajectory for every initial point.\n\\end{proposition}\n\n\\begin{proof}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz function and let $x_0\\in\\mathbb{R}^n$. Since $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d) p. 29]{clarke1990} with nonempty, compact and convex values \\cite[2.1.2 Proposition (a) p. 27]{clarke1990}. By virtue of \\cite[Theorem 3 p. 98]{aubin1984differential}, there exist $\\epsilon>0$ and an absolutely continuous function $x:[0,\\epsilon)\\rightarrow\\mathbb{R}^n$ such that  \n\\begin{equation*}\n    x'(t) \\in -\\partial f(x(t)), ~\\mathrm{for~a.e.}~ t\\in (0,\\epsilon),~~~ x(0) = x_0.\n\\end{equation*}\nLet $S_{x_0}$ be the set of functions in $S$ from Definition \\ref{def:trajectory} such that $x(0) = x_0$. Then $S_{x_0} \\neq \\emptyset$. Let $P$ be a nonempty totally ordered subset of $S_{x_0}$. Let $0<T \\leqslant\\infty$ be the supremum of the endpoints of the domains of all functions in $P$. Consider the function $x:[0,T)\\rightarrow\\mathbb{R}^n$ such that for all $\\bar{x}:[0,\\overline{T})\\rightarrow \\mathbb{R}^n$ in $P$ we have $x(t) := \\bar{x}(t)$ for all $t\\in [0,\\overline{T})$. It is well defined and it is an upper bound of $P$ in $S_{x_0}$. By virtue of Zorn's lemma \\cite[Corollary 2.5 p. 884]{lang2012algebra}, $S_{x_0}$ contains a maximal element. \\end{proof}\n\n\\begin{proposition}\n\\label{prop:bounded_global}\nBounded continuous subgradient trajectories are globally defined.\n\\end{proposition}\n\n\\begin{proof}\nLet $x:[0,T)\\rightarrow\\mathbb{R}^n$ be a bounded continuous subgradient trajectory of $f$ where $0<T \\leqslant \\infty$. We reason by contradiction and assume that $T<\\infty$. By Definition \\ref{def:bounded_trajectory}, there exists $c>0$ such that $\\|x(t)\\| \\leqslant c$ for all $t\\in [0,T)$. Consider a sequence $t_0,t_1,t_2,\\hdots \\in [0,T)$ converging to $T$. For all $k\\in \\mathbb{N}$, let $x_k:[0,T]\\rightarrow\\mathbb{R}^n$ be defined by $x_k(t) := x(t)$ for all $t\\in [0,t_k]$ and $x_k(t) = 0$ for all $t\\in (t_k,T]$. By construction, $x'_k$ converges pointwise almost everywhere to $x'$ on $(0,T)$. Let $L>0$ be a Lipschitz constant of $f$ on $B(0,c)$. By \\cite[2.1.2 Proposition (a) p. 27]{clarke1990}, for all $x\\in B(0,c)$ and $s\\in \\partial f(x)$, we have $\\|s\\| \\leqslant L$. For almost every $t\\in (0,T)$, we thus have $\\|x_k'(t)\\| \\leqslant \\|x'(t)\\| \\leqslant L$. By Lebesgue's dominated convergence theorem, $\\int_0^{t_k} x'(t)dt = \\int_0^T x'_k(t)dt \\rightarrow \\int_0^T x'(t)dt$. Since $x(\\cdot)$ is absolutely continuous on $[0,t_k]$, we have \n$x(t_k) - x(0) = \\int_0^{t_k} x'(t)dt$. Thus $x(\\cdot)$ can be extended to an absolutely continuous function on $[0,T]$ where \n$x(T) := x(0) + \\int_0^T x'(t)dt$. According to Proposition \\ref{prop:existence_trajectory}, $x(\\cdot)$ can further be extended to an absolutely continuous function on $[0,T+\\epsilon)$ for some $\\epsilon>0$ while satisfying the differential inclusion. This contradicts the maximality of $x(\\cdot)$. \n\\end{proof}\n\n\\begin{proposition}\n\\label{prop:tracking}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a lower bounded differentiable function with a locally Lipschitz gradient. Let $X_0$ be a bounded subset of $\\mathbb{R}^n$ and let $T>0$. For all $\\epsilon>0$, there exists $\\bar{\\alpha}>0$ such that for all $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$ and for all sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ such that $x_0 \\in X_0$ and $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$, the continuous gradient trajectory $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ of $f$ initialized at $x_0$ satisfies\n\\begin{equation}\n\\label{eq:tracking}\n   \\forall k\\in \\mathbb{N}^*,~~~ \\alpha_0+\\cdots+\\alpha_{k-1} \\leqslant T ~~~\\Longrightarrow~~~ \\|x_k - x(\\alpha_0+\\cdots+\\alpha_{k-1})\\| \\leqslant \\epsilon.\n\\end{equation}\n\\end{proposition}\n\n\\begin{proof}\nWithout loss of generality, we may assume that $X_0 \\neq \\emptyset$. Let $L>0$ and $M>0$ respectively denote Lipschitz constants of $f$ and $\\nabla f$ with respect to $\\|\\cdot\\|$ on the convex hull of $B(X_0,\\sigma_T(X_0)+1):= X_0 + B(0,\\sigma_T(X_0)+1)$ where \n\\begin{subequations}\n\\label{eq:sup_trajectory_T_smooth}\n\\begin{align}\n    \\sigma_T(X_0) := & \\sup\\limits_{x \\in \\mathcal{C}^1(\\mathbb{R}_+,\\mathbb{R}^n)} ~~ \\int_0^T \\|x'(t)\\|dt \\\\\n  & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx'(t) = - \\nabla f(x(t)), ~\\forall t > 0,\\\\[3mm] x(0) \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\nAbove, $\\mathcal{C}^1(\\mathbb{R}_+,\\mathbb{R}^n)$ is the set of continuous functions on $[0,\\infty )$ that are continuously differentiable on $(0,\\infty)$. Note that $\\sigma_T(X_0)>-\\infty$ since $f$ is lower bounded and differentiable with a locally Lipschitz gradient. For every $x_0\\in X_0$, there hence exists a unique globally defined continuous gradient trajectory of $f$ initialized at $x_0$ \\cite[Theorem 17.1.1]{attouch2014variational}. Also, $\\sigma_T(X_0)<\\infty$ since for any feasible point $x(\\cdot)$ of \\eqref{eq:sup_trajectory_T_smooth}, we have \n\\begin{subequations}\n\\begin{align}\n    \\int_0^T \\|x'(t)\\|dt & \\leqslant \\sqrt{T} \\sqrt{ \\int_0^T \\|x'(t)\\|^2 dt} \\label{eq:length_finite_smooth_a} \\\\\n    & = \\sqrt{T} \\sqrt{ -\\int_0^T \\langle \\nabla f(x(t)), x'(t) \\rangle dt} \\label{eq:length_finite_smooth_b} \\\\\n    & = \\sqrt{T} \\sqrt{ -\\int_0^T (f \\circ x)'(t) dt} \\label{eq:length_finite_smooth_c} \\\\\n    & = \\sqrt{T} \\sqrt{ f(x(0)) - f(x(T))} \\label{eq:length_finite_smooth_d} \\\\\n    & \\leqslant \\sqrt{T\\left(\\sup_{X_0} f - \\inf_{\\mathbb{R}^n} f\\right)} < \\infty. \\label{eq:length_finite_smooth_e}\n\\end{align}\n\\end{subequations}\nIndeed, \\eqref{eq:length_finite_smooth_a} is due to the Cauchy-Schwarz inequality. \\eqref{eq:length_finite_smooth_e} holds because $X_0$ is bounded and $f$ is lower bounded.\n\nIt is easy to check that $L$ and $ML$ are respectively Lipschitz and gradient Lipschitz constants on $[0,T]$ of all continuous gradient trajectories of $f$ initialized in $X_0$. Indeed, let $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ be a continuous gradient trajectory initialized in $X_0$. Since $x(t) \\in B(X_0,\\sigma_T(X_0))$ for all $t\\in [0,T]$, we have $\\|x'(t)\\| = \\|\\nabla f(x(t))\\| \\leqslant L$. By the mean value theorem \\cite[5.19 Theorem]{rudin1964principles}, for all $s,t \\in [0,T]$ we have $\\|x'(t)-x'(s)\\| = \\|\\nabla f(x(t)) - \\nabla f(x(s))\\| \\leqslant M \\|x(t)-x(s)\\| \\leqslant M L |t-s|$. As a byproduct, we get the Taylor bound\n\\begin{subequations}\n    \\begin{align}\n    \\|x(t)-x(s) - x'(s)(t-s)\\| & = \\left\\| \\int_s^t (x'(\\tau) - x'(s)) d\\tau \\right\\| \\\\\n    & \\leqslant \\left| \\int_s^t \\| x'(\\tau) - x'(s) \\| d\\tau \\right| \\\\\n    & \\leqslant \\left|\\int_s^t ML | \\tau - s | d\\tau \\right| \\\\\n    & = \\frac{ML}{2}(t-s)^2. \\label{eq:local_error}\n    \\end{align}\n\\end{subequations}\n\nLet $\\epsilon \\in (0,1)$ and let $\\bar{\\alpha} := 2\\epsilon e^{-MT}/(LMT)$. Consider $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$ and a sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$ and $x_0 \\in X_0$. Let $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ be the unique continuous gradient trajectory of $f$ initialized at $x_0$. Let $t_0:=0$ and $t_k := \\alpha_0+\\cdots+\\alpha_{k-1}$ when $k\\in \\mathbb{N}^*$ so that $\\|x_0 - x(t_0)\\| = 0 \\leqslant \\epsilon$. Assume that \\eqref{eq:tracking} holds up to some index $K$. For $k=0,\\hdots,K$, we have\n\\begin{subequations}\n    \\begin{align}\n        \\|x_{k+1} - x(t_{k+1})\\| = &  ~\\| x_k - \\alpha_k \\nabla f(x_k) - x(t_{k+1}) \\| \\label{local_a} \\\\[2mm]\n        \\leqslant  & ~ \\| x_k - \\alpha_k \\nabla f(x_k) - [x(t_k) - \\alpha_k \\nabla f(x(t_k))] \\|~+ \\label{local_b} \\\\[2mm] & ~ \\| x(t_k) - \\alpha_k \\nabla f(x(t_k)) - x(t_{k+1}) \\| \\label{local_c} \\\\[2mm]\n        \\leqslant & ~ \\|x_k - x(t_k)\\| + \\alpha_k \\|\\nabla f(x_k) - \\nabla f(x(t_k))\\| ~+ \\label{local_d} \\\\[2mm]\n        & ~ ML\\alpha_k^2/2 \\label{local_e} \\\\[2mm]\n        \\leqslant & ~ (1+\\alpha_k M)\\|x_k - x(t_k)\\| + ML\\alpha_k^2/2. \\label{local_f}\n    \\end{align}\n\\end{subequations}\nThe term in \\eqref{local_c} is equal to the local truncation error $\\|x(t_{k+1})  - x(t_k) - x'(t_k)(t_{k+1}-t_k) \\|$, and can hence be bounded above using \\eqref{eq:local_error}. In the second term of \\eqref{local_d}, we invoke a Lipschitz constant of $\\nabla f$ on the convex hull of $B(X_0,\\sigma_T(X_0)+1)$. Indeed, by the induction hypothesis and $\\epsilon < 1$, $x_k$ and $x(t_k)$ belong to $B(X_0,\\sigma_T(X_0)+\\epsilon)$. Hence \n\\begin{subequations}\n    \\begin{align}\n        \\|x_{K+1} - x(t_{K+1})\\| \\leqslant &~ \\prod_{k=0}^K (1+\\alpha_kM)\\|x_0-x(t_0)\\|~+ \\label{eq:global_a}\\\\\n        &~ \\sum_{k=0}^K \\frac{ML\\alpha_k^2}{2} \\prod_{l=k+1}^{K} (1+\\alpha_lM)\\label{eq:global_b}\\\\\n        \\leqslant &~ \\prod_{l=0}^{K} (1+\\alpha_lM) \\sum_{k=0}^K \\frac{ML\\alpha_k^2}{2}\\label{eq:global_c}\\\\\n        \\leqslant &~ \\left(1+\\frac{M}{K+1}\\sum_{k=0}^K \\alpha_k\\right)^{K+1} \\sum_{k=0}^K \\frac{ML\\bar{\\alpha}\\alpha_k}{2}\\label{eq:global_d}\\\\\n        \\leqslant &~ \\left(1+\\frac{MT}{K+1}\\right)^{K+1} \\frac{ML\\bar{\\alpha}T}{2}\\label{eq:global_e}\\\\\n        \\leqslant &~~ \\bar{\\alpha} LMTe^{MT}/2\\label{eq:global_f}\\\\\n        = &~~ \\epsilon.\\label{eq:global_g}\n    \\end{align}\n\\end{subequations}\nThe term in \\eqref{eq:global_a} is equal to zero because $x_0 = x(t_0)$. By convention, we set the product in \\eqref{eq:global_b} to be equal to one if the index set for $l$ is empty. We bound the partial products ranging from $k+1$ to $K$ inside the summation of \\eqref{eq:global_b} by the full product ranging from $0$ to $K$ in order to obtain \\eqref{eq:global_c}. We then use the inequality of arithmetic and geometric means and $\\alpha_k\\leqslant\\bar{\\alpha}$ in order to obtain \\eqref{eq:global_d}. In order to prove the induction step, we assume that $\\alpha_0+\\cdots+\\alpha_K \\leqslant T$, hence \\eqref{eq:global_e}. \\eqref{eq:global_f} follows from $\\ln(1+x) \\leqslant x$ for all $x>0$. Finally, \\eqref{eq:global_g} holds because $\\bar{\\alpha} = 2\\epsilon e^{-MT}/(LMT)$.\n\\end{proof}\n\n\\begin{proposition}\n\\label{prop:KL}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz definable function and let $X$ be a bounded subset of $\\mathbb{R}^n$. Let $V$ be the set of critical values of $f$ in $\\overline{X}$ if it is nonempty, otherwise let $V:=\\{0\\}$. There exists a concave definable diffeomorphism $\\psi:[0,\\infty)\\rightarrow[0,\\infty)$ such that\n\\begin{equation*}\n    \\forall x \\in X \\setminus (\\partial \\tilde{f})^{-1}(0), ~~~ d(0,\\partial (\\psi \\circ \\tilde{f})(x)) \\geqslant 1\n\\end{equation*}\nwhere $\\tilde{f}(x) := d(f(x),V)$ for all $x\\in \\mathbb{R}^n$.\n\\end{proposition}\n\n\\begin{proof}\nBy the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, the set $V$ is finite. In addition, by the Kurdyka-\\L{}ojasiewicz inequality \\cite[Corollary 15]{bolte2007clarke}, there exist $\\epsilon>0$ and a strictly increasing concave continuous definable function $\\psi:[0,\\epsilon)\\rightarrow \\mathbb{R}$ that is continuously differentiable on $(0,\\epsilon)$ with $\\psi(0) = 0$ such that, for all $x \\in X$ and $v \\in V$ satisfying $0 < | f(x)-v | < \\epsilon$, we have $d(0,\\partial (\\psi \\circ | f-v|)(x)) \\geqslant 1$. The fact that $\\psi$ can be assumed to be concave is due to the monotonicity theorem \\cite[(1.2) p. 43]{van1998tame} \\cite[Lemma 2]{kurdyka1998gradients} by following the argument in \\cite[Theorem 4.1]{attouch2010proximal}. Let $\\rho:= \\inf \\{(v-v')/2: v,v' \\in V, v > v'\\}>0$ and $\\rho' : = \\min \\{ \\rho, \\epsilon\\}$. Observe that $c>0$ where\n\\begin{equation*}\n    c := \\inf \\{ \\|s\\| : \ns \\in \\partial f(x), ~ x\\in X, ~ | f(x) - v | \\geqslant \\rho',~ v\\in V \\}.\n\\end{equation*}\nIndeed, assume that there exist sequences $(s_k)_{k\\in \\mathbb{N}}$ and $(x_k)_{k\\in\\mathbb{N}}$ such that $\\|s_k\\|$ converges to zero and $s_k \\in \\partial f(x_k)$, $x_k\\in X$, and $| f(x_k) - v | \\geqslant \\rho'$ for all $v\\in V$. Since $X$ is bounded, there exist subsequences (again denoted $s_k$ and $x_k$) such that $x_k$ has a limit $x^*$ in $\\overline{X}$. Since $f$ is continuous, $| f(x^*) - v | \\geqslant \\rho'>0$ for all $v \\in V$. Also, since $s_k$ converges to zero, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990}, we have $0\\in \\partial f(x^*)$. This yields a contradiction. \n\nAfter possibly reducing $\\epsilon$, we may assume that $\\lim_{t\\nearrow\\rho'} \\psi'(t)>0$. If $\\lim_{t\\nearrow\\rho'} \\psi'(t) \\geqslant 1/c$, then we may extend $\\psi$ to an affine function on $[\\rho',\\infty)$ with slope equal to $\\lim_{t\\nearrow\\rho'} \\psi'(t)$. Otherwise, we can multiply $\\psi$ by $1/(c\\lim_{t\\nearrow\\rho'} \\psi'(t))$ before taking the affine extension. Now let $x \\in X$ be such that $0 \\notin \\partial \\tilde{f}(x)$. If $f(x)$ is strictly within $\\rho'$ distance of some $v \\in V$, then from $\\rho' \\leqslant \\rho$ it follows that $\\tilde{f}(\\tilde{x}) = d(f(\\tilde{x}),V) = | f(\\tilde{x})-v|$ for all $\\tilde{x}$ in a neighborhood of $x$ in $\\mathbb{R}^n$, and thus $d(0,\\partial ( \\psi \\circ \\tilde{f})(x)) = d(0,\\partial (\\psi \\circ | f-v|)(x)) \\geqslant 1$. The inequality is due to $\\rho' \\leqslant \\epsilon$. Otherwise, by \\cite[2.3.9 Theorem (Chain Rule I) (ii) p. 42]{clarke1990} we have $d(0,\\partial ( \\psi \\circ \\tilde{f})(x)) = \\psi'(\\tilde{f}(x)) d(0,\\partial \\tilde{f} (x)) = \\psi'(\\tilde{f}(x)) d(0,\\partial f(x)) \\geqslant \\psi'(\\tilde{f}(x)) c \\geqslant 1$. \n\\end{proof}\n\n\\begin{proposition}\n\\label{prop:chain_rule}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a definable function and let $x:(0,T)\\rightarrow \\mathbb{R}^n$ with $0 < T \\leqslant \\infty$ be an absolutely continuous function. If $f$ is locally Lipschitz at $x(t)$ for all $t \\in (0,T)$ and $x'(t) \\in -\\partial f(x(t))$ for almost every $t\\in(0,T)$, then\n\\begin{equation*}\n    (f \\circ x)'(t) = -\\|x'(t)\\|^2 = -d(0,\\partial f(x(t)))^2,~~~\\mathrm{for~a.e.}~t \\in (0,T).\n\\end{equation*}\n\\end{proposition}\n\n\\begin{proof}\nSince $f$ is definable and locally Lipschitz at $x(t)$ for all $t\\in(0,T)$, it satisfies the chain rule \\cite[Proposition 2 (iv)]{bolte2020conservative} (see also \\cite[Theorem 5.8]{davis2020stochastic} and \\cite[Lemma 2.10]{drusvyatskiy2015quadratic}), that is to say $(f \\circ x)'(t) = \\langle s , x'(t) \\rangle$ for all $s\\in \\partial f(x(t))$ and almost every $t\\in (0,T)$. In particular, we may take $s := - x'(t) \\in \\partial f(x(t))$.\nFinally, by \\cite[Lemma 5.2]{davis2020stochastic} (see also \\cite[Proposition 4.10]{drusvyatskiy2015curves}) it holds that $\\|x'(t)\\| = d(0,\\partial f(x(t)))$ for almost every $t\\in (0,T)$. \n\\end{proof}\n\n\\begin{proposition}\n\\label{prop:length_continuous}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz definable function and let $X$ be a bounded subset of $\\mathbb{R}^n$. Assume that $f$ has at most $m\\in \\mathbb{N}^*$ critical values in $\\overline{X}$. Let $\\psi$ be a desingularizing function of $f$ on $X$. If $x:[0,T]\\rightarrow X$ with $T \\geqslant 0$ is an absolutely continuous function such that $x'(t) \\in -\\partial f(x(t))$ for almost every $t\\in(0,T)$, then\n\\begin{equation}\n\\label{eq:length_continuous}\n    \\frac{1}{2m}\\int_0^T\\|x'(t)\\|dt ~\\leqslant~ \\psi\\left(\\frac{f(x(0))-f(x(T))}{2m}\\right).\n\\end{equation}\n\\end{proposition}\n\n\\begin{proof}\nWe first consider the special case where $0 \\notin \\partial \\tilde{f}(x(t))$ for all $t\\in (0,T)$ where $\\tilde{f}$ is defined as in Proposition \\ref{prop:KL}. Consider the change of variables $\\bar{x} = x \\circ \\varphi$ where $\\bar{t}=\\varphi^{-1}(t) = \\int_0^{t} ds /(\\psi' \\circ \\tilde{f} \\circ x)(s)$ and $t=\\varphi(\\bar{t}) = \\int_0^{\\bar{t}} (\\psi' \\circ \\tilde{f} \\circ \\bar{x})(\\bar{s}) d\\bar{s}$ by the inverse function theorem \\cite[Theorem 17.7.2]{tao2006analysis}. We have $\\bar{x}'(\\bar{t}) = \\varphi'(\\bar{t})x'(t) \\in -(\\psi' \\circ \\tilde{f} \\circ \\bar{x})(\\bar{t}) \\partial f(\\bar{x}(\\bar{t})) = \\pm \\partial (\\psi \\circ \\tilde{f})(\\bar{x}(\\bar{t}))$ for almost every $\\bar{t}\\in (0,\\overline{T})$, where $\\overline{T} = \\varphi^{-1}(T)$ and the sign is constant. By Proposition \\ref{prop:chain_rule} and the Cauchy-Schwarz inequality, we thus have\n\\begin{subequations}\n\\begin{align}\n    \\overline{T} \\leqslant & \\int_0^{\\overline{T}} d(0,\\partial (\\psi \\circ \\tilde{f})(\\bar{x}(\\bar{t})))d\\bar{t} \\label{eq:god_a} \\\\\n    = & \\int_0^{\\overline{T}} \\|\\bar{x}'(\\bar{t})\\|d\\bar{t} \\label{eq:god_b} \\\\\n    = & \\int_0^{T} \\|x'(t)\\|dt \\\\\n    \\leqslant & \\sqrt{\\overline{T}} \\sqrt{\\int_0^{\\overline{T}} \\|\\bar{x}'(\\bar{t})\\|^2d\\bar{t}} \\\\\n    = & \\sqrt{\\overline{T}}\\sqrt{|(\\psi\\circ \\tilde{f}\\circ\\bar{x})(0)-(\\psi\\circ \\tilde{f}\\circ\\bar{x})(\\overline{T})|}.\n\\end{align}\n\\end{subequations}\nHence $\\overline{T} \\leqslant |(\\psi\\circ \\tilde{f}\\circ\\bar{x})(0)-(\\psi\\circ \\tilde{f}\\circ\\bar{x})(\\overline{T})|$ and, since $\\psi$ is concave, we have\n\\begin{align*}\n     \\int_0^T \\|x'(t)\\|dt & \\leqslant |(\\psi\\circ \\tilde{f}\\circ\\bar{x})(0)-(\\psi\\circ \\tilde{f}\\circ\\bar{x})(\\overline{T})| \\\\\n     & \\leqslant \\psi (|( \\tilde{f}\\circ\\bar{x})(0)-(\\tilde{f}\\circ\\bar{x})(\\overline{T})|) \\\\[2mm]\n     & = \\psi (f(x(0))-f(x(T))).\n\\end{align*}\nWe next consider the general case where there may exist $t\\in (0,T)$ such that $0 \\in \\partial \\tilde{f}(x(t))$, namely $0 < t_1 < \\cdots < t_{k} < T$, and potentially $(t_k,T)$. For notational convenience, let $t_0 := 0$ and $t_{k+1}:=T$ where $k$ is possibly equal to zero. Since $k \\leqslant 2m-1$ and $\\psi$ is concave, we have\n\\begin{align*}\n    \\int_0^T \\|x'(t)\\|dt & = \\sum_{i=0}^k \\int_{t_i}^{t_{i+1}} \\|x'(t)\\|dt \\\\\n    & \\leqslant \\sum_{i=0}^k \\psi (f(x(t_i))-f(x(t_{i+1}))) \\\\[-.5mm]\n    & \\leqslant (k+1) ~ \\psi \\left( \\frac{1}{k+1} \\sum_{i=0}^k f(x(t_i))-f(x(t_{i+1}))\\right) \\\\\n    & = (k+1) ~ \\psi \\left( \\frac{f(x(0)) - f(x(T))}{k+1} \\right) \\\\\n    & \\leqslant 2m ~ \\psi \\left( \\frac{f(x(0)) - f(x(T))}{2m} \\right). \\qedhere\n\\end{align*}\n\\end{proof}\n\n\\begin{proposition}\n\\label{prop:length_discrete}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a definable differentiable function with a locally Lipschitz gradient and let $X$ be a bounded open subset of $\\mathbb{R}^n$. Assume that $f$ has at most $m\\in \\mathbb{N}^*$ critical values in $\\overline{X}$ and that $L>0$ is a Lipschitz constant of $f$ on the convex hull of $X$. Let $\\psi$ be a desingularizing function of $f$ on $X$ and $\\epsilon>0$. There exists $\\bar{\\alpha}>0$ such that, for all $K \\in \\mathbb{N}$, $\\alpha_0,\\hdots,\\alpha_K \\in (0,\\bar{\\alpha}]$, and $(x_0,\\hdots,x_{K+1}) \\in X \\times \\cdots \\times X \\times \\mathbb{R}^n$ such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for $k = 0,\\hdots,K$, we have\n\\begin{equation}\n\\label{eq:length_discrete}\n    \\frac{1}{(2+\\epsilon)m}\\sum_{k=0}^K\\|x_{k+1}-x_k\\| ~\\leqslant~ \\psi\\left(\\frac{f(x_0)-f(x_K)}{2m}\\right) + \\frac{2L}{2+\\epsilon}\\max_{k=0,\\hdots,K} \\alpha_k\n\\end{equation}\nand $f(x_0)\\geqslant \\cdots \\geqslant f(x_{K+1})$.\n\\end{proposition}\n\n\\begin{proof}\nLet $M>0$ be a Lipschitz constant of $\\nabla f$ with respect to $\\|\\cdot\\|$ on the convex hull of $B(X,1):=X+B(0,1)$ and let $\\bar{\\alpha} := \\min\\{L^{-1},2\\epsilon(6+\\epsilon)^{-1}M^{-1}\\}$. Let $K \\in \\mathbb{N}$, $\\alpha_0,\\hdots,\\alpha_K \\in (0,\\bar{\\alpha}]$, and $(x_0,\\hdots,x_{K+1}) \\in X \\times \\cdots \\times X \\times \\mathbb{R}^n$ be such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for $k=0,\\hdots,K$. It holds that $\\|x_{K+1}-x_K\\| = \\alpha_K\\|\\nabla f(x_K)\\| \\leqslant \\bar{\\alpha}L \\leqslant 1$ and thus $x_0,\\hdots,x_{K+1} \\in B(X,1)$. A bound on the Taylor expansion of $f$ yields\n\\begin{align*}\n    f(x_{k+1})-f(x_k) & \\leqslant \\langle \\nabla f(x_k), x_{k+1}-x_k\\rangle + \\frac{M}{2}\\|x_{k+1}-x_k\\|^2 \\\\\n    & = \\left(\\frac{M\\alpha_k}{2}-1\\right) \\|x_{k+1}-x_k\\| \\|\\nabla f(x_k)\\|\n\\end{align*}\nfor $k=0,\\hdots,K$, and thus\n\\begin{equation}\n\\label{eq:descent_lemma_1}\n    \\|x_{k+1}-x_k\\|\\|\\nabla f(x_k)\\| \\leqslant \\frac{2}{2- M\\alpha_k}(f(x_k) - f(x_{k+1})).\n\\end{equation}\nWe also have\n    \\begin{align*}\n        \\|\\nabla f(x_{k+1})\\| & \\leqslant \\| \\nabla f(x_{k+1}) - \\nabla f(x_k) \\| + \\|\\nabla f(x_k)\\| \\\\\n        & \\leqslant M \\| x_{k+1} - x_k \\| + \\|\\nabla f(x_k)\\| \\\\\n        & \\leqslant (M\\alpha_k+1) \\|\\nabla f(x_k)\\|\n    \\end{align*}\nand thus \n\\begin{equation}\n\\label{eq:descent_lemma_2}\n    \\|x_{k+1}-x_k\\|\\|\\nabla f(x_{k+1})\\| \\leqslant \\frac{2+2M\\alpha_k}{2- M\\alpha_k}(f(x_k) - f(x_{k+1})).\n\\end{equation}\nNote that \n\\begin{equation*}\n    1 \\leqslant \\frac{2}{2- M\\alpha_k} \\leqslant \\frac{2+2M\\alpha_k}{2- M\\alpha_k} \\leqslant 1+\\frac{\\epsilon}{2}.\n\\end{equation*}\nLet $\\tilde{f}$ and $V$ be defined as in Proposition \\ref{prop:KL}.\n\nAssume that $[f(x_K),f(x_0)]$ excludes the elements of $V$ and the averages of any two consecutive elements of $V$. Then $0\\notin \\partial \\tilde{f}(x_k)$ and $1 \\leqslant \\| \\nabla(\\psi \\circ \\tilde{f})(x_k)\\| = \\psi'(\\tilde{f}(x_k)) \\|\\nabla \\tilde{f}(x_k)\\|$ for $k=0,\\hdots,K$. Let $k \\in \\{0,\\hdots,K-1\\}$. If $\\tilde{f}(x_k) \\geqslant \\tilde{f}(x_{k+1})$, then multiplying \\eqref{eq:descent_lemma_1} by $\\psi'(\\tilde{f}(x_k))$ and using the concavity of $\\psi$, we find that \n\\begin{align*}\n    \\|x_{k+1}-x_k\\| & \\leqslant (1+\\epsilon/2)\\psi'(\\tilde{f}(x_k))(f(x_k) - f(x_{k+1})) \\\\\n    & = (1+\\epsilon/2)\\psi'(\\tilde{f}(x_k))(\\tilde{f}(x_k) - \\tilde{f}(x_{k+1})) \\\\\n    & \\leqslant (1+\\epsilon/2)(\\psi (\\tilde{f}(x_k)) - \\psi (\\tilde{f}(x_{k+1}))).\n\\end{align*}\nIf $\\tilde{f}(x_k) \\leqslant \\tilde{f}(x_{k+1})$, then multiplying \\eqref{eq:descent_lemma_2} by $\\psi'(\\tilde{f}(x_{k+1}))$ and using the concavity of $\\psi$, we find that \n\\begin{align*}\n    \\|x_{k+1}-x_k\\| & \\leqslant (1+\\epsilon/2)\\psi'(\\tilde{f}(x_{k+1}))(f(x_k) - f(x_{k+1})) \\\\\n    & = (1+\\epsilon/2)\\psi'(\\tilde{f}(x_{k+1}))(\\tilde{f}(x_{k+1}) - \\tilde{f}(x_k)) \\\\\n    & \\leqslant (1+\\epsilon/2)(\\psi (\\tilde{f}(x_{k+1})) - \\psi (\\tilde{f}(x_k))).\n\\end{align*}\nAs a result,\n\\begin{equation*}\n    \\|x_{k+1}-x_k\\| \\leqslant (1+\\epsilon/2)|\\psi (\\tilde{f}(x_k)) - \\psi (\\tilde{f}(x_{k+1}))|, ~~~ k = 0,\\hdots,K-1.\n\\end{equation*}\nWe obtain the telescoping sum \n\\begin{subequations}\n    \\label{eq:length_finite} \n    \\begin{align}\n    \\sum_{k=0}^K \\|x_{k+1}-x_k\\| & \\leqslant (1+\\epsilon/2) |\\psi(\\tilde{f}(x_0)) - \\psi(\\tilde{f}(x_K))| + \\|x_{K+1} - x_K\\| \\\\[-4mm]\n    & \\leqslant (1+\\epsilon/2) \\psi ( | \\tilde{f}(x_0) - \\tilde{f}(x_K) |) + \\alpha_K\\|\\nabla f(x_K)\\| \\\\\n    & \\leqslant (1+\\epsilon/2) \\psi( f(x_0) - f(x_K)) + L\\max_{k=0,\\hdots,K} \\alpha_k.\n    \\end{align}\n\\end{subequations}\n\nAssume that $[f(x_K),f(x_0))$ excludes the elements of $V$ and the averages of any two consecutive elements of $V$. Since we are excluding a finite number of elements, there exists $\\epsilon>0$ such that $[f(x_K)-\\epsilon,f(x_0))$ satisfies the same property. If $x_0$ is a local minimum of $f$ on the open set $X$, then $\\nabla f(x_0) = 0$ by Fermat's rule. The length of $x_0,\\hdots,x_{K+1}$ is then equal to zero and the formula obtained in \\eqref{eq:length_finite} trivially holds. Otherwise, there exists a sequence $(x_0^l)_{l\\in\\mathbb{N}}$ in $\\mathbb{R}^n$ such that $f(x_0^l) < f(x_0)$ and $x_0^l$ converges to $x_0$. For every $l \\in \\mathbb{N}$, consider the sequence $(x^l_k)_{k\\in \\mathbb{N}}$ such that $x^l_{k+1} = x^l_k - \\alpha_k \\nabla f(x_k^l)$ for all $k \\in \\mathbb{N}$. Since $\\nabla f$ is continuous, for any fixed $k \\in \\{0,\\hdots,K\\}$, $x_k^l$ converges to $x_k \\in X$ and eventually belongs to the open set $X$. Thus $[f(x^l_K),f(x^l_0)] \\subset [f(x_K)-\\epsilon,f(x_0))$ eventually excludes the elements of $V$ and the averages of any two consecutive elements of $V$. We can apply the formula obtained in \\eqref{eq:length_finite}, namely\n\\begin{equation*}\n    \\sum_{k=0}^K \\|x^l_{k+1}-x^l_k\\| \\leqslant (1+\\epsilon/2) \\psi( f(x^l_0) - f(x^l_K)) + L\\max_{k=0,\\hdots,K} \\alpha_k.\n\\end{equation*}\nPassing to the limit yields\n\\begin{equation*}\n    \\sum_{k=0}^K \\|x_{k+1}-x_k\\| \\leqslant (1+\\epsilon/2) \\psi( f(x_0) - f(x_K)) + L\\max_{k=0,\\hdots,K} \\alpha_k.\n\\end{equation*}\n\nWe next consider the general case where\n\\begin{equation*}\n    [f(x_K),f(x_{K_p+1})) \\cup \\cdots \\cup [f(x_{K_2}),f(x_{K_1+1})) \\cup [f(x_{K_1}),f(x_0))\n\\end{equation*}\nexcludes the elements of $V$ and the averages of any two consecutive elements of $V$. For notational convenience, let $K_0 := -1$ and $K_{p+1} := K$. Since $p \\leqslant 2m-1$, we have\n\\begin{align*}\n    \\sum_{k=0}^K \\|x_{k+1}-x_k\\| = & \\sum_{i=0}^p \\sum_{k=K_i+1}^{K_{i+1}} \\|x_{k+1}-x_k\\| \\\\\n    \\leqslant & \\sum_{i=0}^p \\left((1+\\epsilon/2)\\psi( f(x_{{K_i}+1}) - f(x_{K_{i+1}})) + L\\max_{k=0,\\hdots,K} \\alpha_k\\right) \\\\[-.5mm]\n    \\leqslant & (1+\\epsilon/2)(p+1) ~ \\psi \\left( \\frac{1}{p+1} \\sum_{i=0}^p  f(x_{{K_i}+1}) - f(x_{K_{i+1}}) \\right) + \\\\ \n    & (p+1)L\\max_{k=0,\\hdots,K} \\alpha_k \\\\\n    \\leqslant & (1+\\epsilon/2)(p+1) \\psi \\left( \\frac{f(x_0)-f(x_K)}{p+1} \\right) + (p+1)L\\max_{k=0,\\hdots,K} \\alpha_k \\\\\n    \\leqslant & (2+\\epsilon)m ~ \\psi \\left( \\frac{f(x_0)-f(x_K)}{2m} \\right) + 2m L\\max_{k=0,\\hdots,K} \\alpha_k.\\qedhere\n\\end{align*}\n\\end{proof}\n\n\\section{Proof of Lemma \\ref{lemma:uniform_length}}\n\\label{sec:Proof of Lemma lemma:uniform_length}\n \nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz definable function with bounded continuous subgradient trajectories. Let $X_0$ be a bounded subset of $\\mathbb{R}^n$. We will show that $\\sigma(X_0) < \\infty$ where \n\\begin{subequations}\n\\label{eq:sup_trajectory}\n\\begin{align}\n    \\sigma(X_0) := & \\sup\\limits_{x \\in \\mathcal{A}(\\mathbb{R}_+,\\mathbb{R}^n)} ~~ \\int_0^{\\infty} \\|x'(t)\\|dt \\\\\n  & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx'(t) \\in - \\partial f(x(t)), ~\\mathrm{for~a.e.}~ t > 0,\\\\[3mm] x(0) \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\nWithout loss of generality, we may assume that $X_0\\neq \\emptyset$. By Propositions \\ref{prop:existence_trajectory} and \\ref{prop:bounded_global}, the feasible set of \\eqref{eq:sup_trajectory} is thus non-empty and $\\sigma(X_0)>-\\infty$.\n\nWe first consider the case with finite time horizon $T\\geqslant 0$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Notice that $\\sigma_T(X_0) \\leqslant \\sqrt{T(\\sup_{X_0} f - m(f))}$ where $m(f)$ is the smallest critical value of $f$ and \n\\begin{subequations}\n\\label{eq:sup_trajectory_T}\n\\begin{align}\n    \\sigma_T(X_0) := & \\sup\\limits_{x \\in \\mathcal{A}(\\mathbb{R}_+,\\mathbb{R}^n)} ~~ \\int_0^T \\|x'(t)\\|dt \\\\\n  & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx'(t) \\in - \\partial f(x(t)), ~\\mathrm{for~a.e.}~ t > 0,\\\\[3mm] x(0) \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\nIndeed, by the Cauchy-Schwarz inequality and Propositions \\ref{prop:chain_rule} and \\ref{prop:length_continuous}, for any feasible point $x(\\cdot)$ of \\eqref{eq:sup_trajectory_T} we have\n\\begin{subequations}\n\\begin{align}\n    \\int_0^T \\|x'(t)\\|dt & \\leqslant \\sqrt{T} \\sqrt{ \\int_0^T \\|x'(t)\\|^2 dt} \\\\\n    & \\leqslant \\sqrt{T} \\sqrt{ \\int_0^{\\infty} \\|x'(t)\\|^2 dt} \\\\\n    & = \\sqrt{T} \\sqrt{f(x(0)) - f\\left(\\lim_{t\\rightarrow\\infty} x(t)\\right)} \\\\\n    & \\leqslant \\sqrt{T\\left(\\sup_{X_0} f - m(f)\\right)}. \\label{eq:finite_time}\n\\end{align}\n\\end{subequations}\n\nWe next treat the case with infinite time horizon. Consider a sequence of feasible points $x_0(\\cdot), x_1(\\cdot), x_2(\\cdot), \\ldots$ of \\eqref{eq:sup_trajectory}  such that $\\int_0^{\\infty} \\|x_k'(t)\\|dt$ converges to $\\sigma(X_0)$. We proceed to show that the sequence is equicontinuous. Let $\\epsilon>0$ and $t\\geqslant 0$. Consider problem \\eqref{eq:sup_trajectory_T} with finite time horizon $T:=t+\\epsilon$. Since $x_0(\\cdot),x_1(\\cdot),x_2(\\cdot),\\ldots$ are feasible points of \\eqref{eq:sup_trajectory_T}, for all $s \\in [0,T]$ and $k=0,1,2,\\ldots$ we have\n\\begin{subequations}\n    \\begin{align}\n        \\|x_k(s)\\| & \\leqslant \\|x_k(s) - x_k(0) \\| + \\|x_k(0)\\| \\label{upper_0} \\\\ & \\leqslant \\int_0^s \\|x_k'(\\tau)\\|d\\tau + \\|x_k(0)\\| \\label{upper_1} \\\\\n        & \\leqslant \\sigma_T(X_0) + \\sup_{x_0 \\in X_0} \\|x_0\\|. \\label{upper_2}\n    \\end{align}\n\\end{subequations}\nAll the trajectories $x_0(\\cdot),x_1(\\cdot),x_2(\\cdot),\\ldots$ hence belong to a common ball centered at the origin up to time $T$. Let $L>0$ be a Lipschitz constant of $f$ on that ball. As a result, for all $s \\in [t-\\delta,t+\\delta]$ with $\\delta:=\\epsilon/(2L)$ and for all $k\\in \\mathbb{N}$, we have\n\\begin{equation*}\n    \\|x_k(t)-x_k(s)\\| = \\left\\| \\int_{s}^t x'_k(\\tau)d\\tau \\right\\| \\leqslant \\int_{t-\\delta}^{t+\\delta} \\|x'_k(\\tau)\\| d\\tau \\leqslant 2\\delta L = \\epsilon,\n\\end{equation*}\nIt follows that $x_0(\\cdot),x_1(\\cdot),x_2(\\cdot),\\ldots$ is equicontinuous on $\\mathbb{R}_+$. In addition, \\eqref{upper_0}-\\eqref{upper_2} imply that, for all $t\\geqslant 0$, the sequence $x_0(t),x_1(t),x_2(t),\\ldots$ is bounded. The Arzel\\`a-Ascoli theorem \\cite[Theorem 1 p. 13]{aubin1984differential} implies that there exists a subsequence (again denoted $x_k(\\cdot)$) converging uniformly over compact intervals to a continuous function $x:\\mathbb{R}_+\\rightarrow\\mathbb{R}^n$. \n\nWe next show that $x(\\cdot)$ is a continuous subgradient trajectory of $f$. Let $T\\geqslant 0$. By virtue of \\eqref{upper_0}-\\eqref{upper_2} and local Lipschitz continuity of $f$, the restrictions of $x'_0(\\cdot),x'_1(\\cdot),x'_2(\\cdot),\\ldots$ to $[0,T]$ lie in a ball of the dual space of $L^1([0,T],\\mathbb{R}^n)$, namely $L^{\\infty}([0,T],\\mathbb{R}^n)$. By the Banach-Alaoglu theorem \\cite[Theorem 3 p. 13]{aubin1984differential}, there exists a subsequence (again denoted $x_k'(\\cdot)$) that converges weakly\\textsuperscript{$\\ast$} to a function $y(\\cdot)$ in $L^{\\infty}([0,T],\\mathbb{R}^n)$. Together with $L^{\\infty}([0,T],\\mathbb{R}^n)\\subset L^1([0,T],\\mathbb{R}^n)$, we find that $x_k'(\\cdot)$ converges weakly to $y(\\cdot)$ in $L^1([0,T],\\mathbb{R}^n)$. Since $x_k(t)-x_k(s) = \\int_s^t x_k'(\\tau)d\\tau$ for all $0\\leqslant s \\leqslant t \\leqslant T$, we have $x(t)-x(s) = \\int_s^t y(\\tau)d\\tau$. As a result, $x'(t) = y(t)$ for almost every $t\\in (0,T)$. According to \\cite[Convergence Theorem p. 60]{aubin1984differential}, it follows that $x'(t) \\in -\\partial f(x(t))$ for almost every $t\\in(0,T)$. As $T \\geqslant 0$ was arbitrary, we have $x'(t) \\in -\\partial f(x(t))$ for almost every $t>0$.\n\n% https://www.math.ucdavis.edu/~hunter/pdes/ch6A.pdf\n\nSince $f$ is definable and has bounded continuous subgradient trajectories, by Proposition \\ref{prop:length_continuous} $x(\\cdot)$ has finite length and converges to a critical point $x^*$ of $f$. Let $\\epsilon>0$ and let $m \\in \\mathbb{N}^*$ be the number of critical values of $f$ in $B(\\overline{x(\\mathbb{R}_+)},\\epsilon)$. Let $\\psi$ be a desingularizing function of $f$ on $B(\\overline{x(\\mathbb{R}_+)},\\epsilon)$. Since $f$ is continuous, there exists $\\delta \\in (0,\\epsilon/2)$ such that \n\\begin{equation}\n\\label{eq:f_bound_continuous}\n    f(x) - f(x^*) \\leqslant m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right),~~~ \\forall x\\in B(x^*,\\delta).\n\\end{equation}\n\nLet $t^*\\geqslant 0$ be such that $\\|x(t)-x^*\\| \\leqslant \\delta/2$ for all $t\\geqslant t^*$. By the uniform convergence of $x_k(\\cdot)$, we have that $\\|x_k(t)-x(t)\\|\\leqslant \\delta/2$ for all $t\\in [0,t^*]$ for all $k$ large enough. Hence $\\|x_k(t^*) - x^*\\|\\leqslant \\|x_k(t^*)-x(t^*)\\|+\\|x(t^*)-x^*\\| \\leqslant \\delta/2 + \\delta/2 = \\delta$. If $T_k := \\inf\\{ t \\geqslant t^* : x_k(t) \\notin \\mathring{B}(x^*,\\epsilon)\\}<\\infty$, then\n\\begin{subequations}\n    \\begin{align}\n        2m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right) & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{\\epsilon-\\delta}{2m}\\right) \\label{eq:decrease_continuous_a} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{\\|x_k(T_k)-x^*\\| - \\|x_k(t^*)-x^*\\|}{2m}\\right) \\label{eq:decrease_continuous_b} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{\\|x_k(T_k)-x_k(t^*)\\|}{2m}\\right) \\label{eq:decrease_continuous_c} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{2m}\\int_{t^*}^{T_k}\\|x_k'(t)\\|dt \\right) \\label{eq:decrease_continuous_d} \\\\\n        & \\leqslant f(x_k(t^*))-f(x_k(T_k)) \\label{eq:decrease_continuous_e} \\\\[3mm]\n        & \\leqslant m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right) + f(x^*) - f(x_k(T_k)). \\label{eq:decrease_continuous_f}\n    \\end{align}\n\\end{subequations}\nAbove, \\eqref{eq:decrease_continuous_a} through \\eqref{eq:decrease_continuous_d} rely on the fact that $\\psi^{-1}$ is increasing. \\eqref{eq:decrease_continuous_a} is due to $\\delta < \\epsilon/2$. \\eqref{eq:decrease_continuous_b} holds because $\\|x_k(T_k)-x^*\\|\\geqslant \\epsilon$ by continuity of $x_k(\\cdot)$ and $x_k(t^*) \\in B(x^*,\\delta)$. \\eqref{eq:decrease_continuous_c} is a consequence of the triangular inequality. \\eqref{eq:decrease_continuous_d} holds because $x_k(\\cdot)$ is absolutely continuous. \\eqref{eq:decrease_continuous_e} is due to Proposition \\ref{prop:length_continuous} and the fact that $x_k(t) \\in B(x^*,\\epsilon)$ for all $t \\in [t^*,T_k]$ by continuity of $x_k(\\cdot)$. Finally, \\eqref{eq:decrease_continuous_f} is due to $x_k(t^*)\\in B(x^*,\\delta)$ and \\eqref{eq:f_bound_continuous}. Since $x_k(t) \\in B(\\overline{x(\\mathbb{R}_+)},\\epsilon)$ for all $t\\in [0,T_k]$ and $x_k(T_k)$ belongs to\n\\begin{equation*}\n     X_1 := B(x^*,\\epsilon) \\bigcap \\left\\{ x \\in \\mathbb{R}^n : f(x) \\leqslant f(x^*) - m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right) \\right\\},\n\\end{equation*}\nby Proposition \\ref{prop:length_continuous} and the definition of $\\sigma(\\cdot)$ in \\eqref{eq:sup_trajectory} we have \n    \\begin{align*}\n            \\int_0^{\\infty} \\|x_k'(t)\\|dt & = \\int_{0}^{T_k} \\|x_k'(t)\\|dt + \\int_{T_k}^{\\infty} \\|x_k'(t)\\|dt \\\\\n            & \\leqslant 2m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(x^*,\\epsilon)} f\\right) \\right) + \\max\\{0,\\sigma(X_1)\\}.\n    \\end{align*}\nNote that the inequality still holds if $T_k = \\infty$. By taking the limit, we get\n\\begin{equation*}\n    \\sigma(X_0) ~\\leqslant~ 2m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(x^*,\\epsilon)} f\\right) \\right) + \\max\\{0,\\sigma(X_1)\\}.\n\\end{equation*}\n\nIt now suffices to replace $X_0$ by $X_1$ and repeat the proof starting below \\eqref{eq:finite_time}. A maximizing sequence $\\bar{x}_k(\\cdot)$ corresponding to $\\sigma(X_1)$ converges to a continuous subgradient trajectory $\\bar{x}(\\cdot)$ whose initial point lies in the compact set $X_1$. If $X_1 \\neq \\emptyset$, then the critical value $f(\\lim_{t\\rightarrow\\infty} \\bar{x}(t))$ is less than or equal to $f(x^*) - m\\psi^{-1}(\\epsilon/(4m)) < f(x^*)$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Thus, it is eventually the case that one of the sets $X_0,X_1,\\hdots$ is empty. We conclude that $\\sigma(X_0)<\\infty$ by the above recursive formula. \n\n\\section{Proof of Theorem \\ref{thm:convergence}}\n\\label{sec:Proof of Theorem thm:convergence}\n\n($1\\Longrightarrow 2$) We prove the contrapositive. Assume that there exists $0< T \\leqslant \\infty$ and a differentiable function $x:[0,T)\\rightarrow\\mathbb{R}^n$ such that\n\\begin{equation*}\n    x'(t) = -\\nabla f(x(t)), ~~~ \\forall t \\in (0,T),\n\\end{equation*}\nfor which, for all $c>0$, there exists $t\\in [0,T)$ such that $\\|x(t)\\| > c$. Let $x_0 := x(0)$. Let $\\bar{\\alpha}$ and $c$ be some positive constants. Let $\\tilde{t} \\in (0,T)$ be such that $\\|x(\\tilde{t})\\| \\geqslant c+2$. By Proposition \\ref{prop:tracking} and Remark \\ref{rem:tracking}, there exists $\\tilde{\\alpha}\\in (0,\\bar{\\alpha}]$ such that for all $\\alpha \\in (0,\\tilde{\\alpha}]$, the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by $x_{k+1} = x_k -\\alpha \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$ satisfies $\\|x_k-x(k\\alpha)\\| \\leqslant 1/2$ for all $k\\in \\{0,\\hdots, \\lfloor \\tilde{t}/\\alpha \\rfloor \\}$. We use the notation $\\lfloor t \\rfloor$ to denote the floor of a real number $t$ which is the unique integer such that $\\lfloor t \\rfloor \\leqslant t < \\lfloor t \\rfloor + 1$. If $\\tilde{k} = \\lfloor \\tilde{t}/\\alpha \\rfloor$, then $\\|x_{\\tilde{k}} - x(\\tilde{t})\\| \\leqslant \\|x_{\\tilde{k}}-x(\\tilde{k}\\alpha)\\|+\\|x(\\tilde{k}\\alpha)-x(\\tilde{t})\\| \\leqslant 1/2 + \\tilde{L}| \\tilde{k}\\alpha-\\tilde{t}| \\leqslant 1/2 + \\tilde{L}\\alpha$ where $\\tilde{L}>0$ is a Lipschitz constant of $f$ on the convex hull of $x([\\tilde{t}-\\tilde{\\alpha},\\tilde{t}])$. Let $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$ be the constant sequence equal to $\\min \\{\\tilde{\\alpha}, 1/(2\\tilde{L})\\}$ and consider the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by $x_{k+1} = x_k -\\alpha_k \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$. By the triangular inequality, we have $\\|x_{\\tilde{k}}\\| \\geqslant \\|x(\\tilde{t})\\| - \\|x_{\\tilde{k}}-x(\\tilde{t})\\| \\geqslant c+2 - 1/2 - 1/2>c$ where $\\tilde{k} := \\lfloor \\tilde{t}/\\alpha \\rfloor$.\n\n($2 \\Longrightarrow 3$) Let $X_0$ be a bounded subset of $\\mathbb{R}^n$. We will show that there exists $\\bar{\\alpha}>0$ such that $\\sigma(X_0,\\bar{\\alpha})<\\infty$ where\n\\begin{subequations}\n\\label{eq:sup_discrete_X_0}\n\\begin{align}\n    \\sigma(X_0,\\bar{\\alpha}) := & \\sup\\limits_{\\tiny \\begin{array}{c}x \\in (\\mathbb{R}^n)^{\\mathbb{N}}\\\\ \\alpha \\in (0,\\bar{\\alpha}]^\\mathbb{N} \\end{array}} ~~ \\sum\\limits_{k=0}^{\\infty} \\|x_{k+1}-x_k\\| \\\\\n   & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx_{k+1} = x_k - \\alpha_k \\nabla f(x_k), ~ \\forall k\\in \\mathbb{N},\\\\[2mm] x_0 \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\n\nLet $\\Phi:\\mathbb{R}_+ \\times \\mathbb{R}^n \\rightarrow \\mathbb{R}^n$ be the gradient flow of $f$ defined for all $(t,x_0) \\in \\mathbb{R}_+ \\times \\mathbb{R}^n$ by $\\Phi(t,x_0) := x(t)$ where $x(\\cdot)$ is the unique continuous gradient trajectory of $f$ initialized at $x_0$. Uniqueness follows from the Picard–Lindel{\\\"o}f theorem \\cite[Theorem 3.1 p. 12]{coddington1955theory}. Let $C$ be the set of critical points of $f$ in $\\overline{\\Phi(\\mathbb{R}_+,X_0)}$. If $X_0 \\subset C$, then we have $\\sigma(X_0,\\bar{\\alpha}) \\leqslant 0$ for all $\\bar{\\alpha}>0$ and there is nothing left to prove. Otherwise, since $C$ is closed by \\cite[2.1.5 Proposition p. 29]{clarke1990}, there exists $\\epsilon>0$ be such that $X_0\\setminus \\mathring{B}(C,\\epsilon/6) \\neq \\emptyset$ where $\\mathring{B}(C,\\epsilon/6) := C + \\mathring{B}(0,\\epsilon/6)$. It is safe to assume this from now on.\n\nLet $\\psi$ be a desingularizing function of $f$ on $\\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$, where we use the shorthand $\\Phi_0 := \\Phi(\\mathbb{R}_+,X_0)$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Let $m \\in \\mathbb{N}$ be the number of critical values of $f$ in $B(\\overline{\\Phi}_0,\\epsilon)$. Note that $m \\geqslant 1$ due to Proposition \\ref{prop:length_continuous} and the fact that $X_0 \\neq \\emptyset$. Since $f$ is continuous and $C$ is compact by Lemma \\ref{lemma:uniform_length}, there exists $\\delta \\in (0,\\epsilon/2)$ such that  \n\\begin{equation}\n    \\label{eq:f_bound_discrete}\n    f(x) - \\max_{C} f \\leqslant m~ \\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right),~~~ \\forall x\\in B(C,\\delta).\n\\end{equation}\nNote that $\\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$ is bounded due to Lemma \\ref{lemma:uniform_length}. Let $L>0$ be a Lipschitz constant of $f$ on the convex hull of $\\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$. By Proposition \\ref{prop:length_discrete}, there exists $\\alpha_\\delta \\in (0,\\delta/(5mL)]$ such that, for all $K \\in \\mathbb{N}^*$, $\\alpha_0,\\hdots,\\alpha_{K-1} \\in (0,\\alpha_\\delta]$, and $(x_0,\\hdots,x_K) \\in \\mathring{B}(\\overline{\\Phi}_0,\\epsilon) \\times \\cdots \\times \\mathring{B}(\\overline{\\Phi}_0,\\epsilon) \\times \\mathbb{R}^n$ such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for $k = 0,\\hdots,K-1$, we have\n\\begin{equation}\n\\label{eq:length_discrete_K}\n    \\frac{1}{4m}\\sum_{k=0}^{K-1}\\|x_{k+1}-x_k\\| ~\\leqslant~ \\psi\\left(\\frac{f(x_0)-f(x_{K-1})}{2m}\\right) + \\frac{L}{2}\\max_{k=0,\\hdots,K-1} \\alpha_k.\n\\end{equation}\n\nSince $\\nabla f$ is continuous, its norm attains its infimum $\\nu$ on the nonempty compact set $\\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3)$. \nIt is nonempty because $\\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3) \\supset X_0\\setminus \\mathring{B}(C,\\epsilon/6) \\neq \\emptyset$ and $\\delta < \\epsilon/2$.\nIf $\\nu=0$, then there exists $x^* \\in \\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3)$ such that $\\|\\nabla f(x^*)\\| = 0$. Then $x^*\\in C \\setminus \\mathring{B}(C,\\delta/3)$, which is a contradiction. We thus have $\\nu>0$. Hence we may define $T := 2\\sigma(X_0)/\\nu$ where $\\sigma(X_0)$ is defined in \\eqref{eq:sup_trajectory} and is finite by Lemma \\ref{lemma:uniform_length}. Note that $\\sigma(X_0)>0$ and thus $T>0$ because $X_0\\not\\subset C$. By Proposition \\ref{prop:tracking}, there exists $\\bar{\\alpha} \\in (0,\\alpha_\\delta]$ such that for any feasible point $((x_k)_{k\\in \\mathbb{N}} , (\\alpha_k)_{k\\in\\mathbb{N}})$ of \\eqref{eq:sup_discrete_X_0},\nthe continuous gradient trajectory $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ of $f$ initialized at $x_0$ satisfies\n\\begin{equation}\n\\label{eq:tracking_T}\n   \\forall k\\in \\mathbb{N}^*,~~~ \\alpha_0+\\cdots+\\alpha_{k-1} \\leqslant T ~~~\\Longrightarrow~~~ \\|x_k - x(\\alpha_0+\\cdots+\\alpha_{k-1})\\| \\leqslant \\frac{\\delta}{3}.\n\\end{equation}\nNow suppose that $\\|x'(t)\\| \\geqslant 2\\sigma(X_0)/T$ for all $t\\in (0,T)$. Then we obtain the following contradiction\n\\begin{equation*}\n     \\sigma(X_0) < T \\frac{2\\sigma(X_0)}{T} \\leqslant \\int_0^{T} \\|x'(t)\\|dt \\leqslant \\int_0^{\\infty} \\|x'(t)\\|dt \\leqslant \\sigma(X_0).\n\\end{equation*}\nHence, there exists $t^* \\in (0,T)$ such that $\\|x'(t^*)\\| = \\|\\nabla f(x(t^*))\\| < 2\\sigma(X_0)/T = 2\\sigma(X_0)/(2\\sigma(X_0)/\\nu) = \\nu$. Since  $x(t^*) \\in \\Phi(\\mathbb{R}_+,X_0) \\subset \\overline{\\Phi}_0$ and the infimum of the norm of $\\nabla f$ on $\\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3)$ is equal to $\\nu$, it must be that $x(t^*) \\in \\mathring{B}(C,\\delta/3)$. Hence there exists $x^* \\in C$ such that $\\|x(t^*)-x^*\\|\\leqslant \\delta/3$. \n\nIf $\\sum_{k=0}^{\\infty} \\alpha_k \\leqslant T$, then by \\eqref{eq:tracking_T} we have $x_k \\in B(\\overline{\\Phi}_0,\\delta/3)$ for all $k \\in \\mathbb{N}$. Otherwise, since $\\alpha_k \\leqslant \\bar{\\alpha} \\leqslant \\alpha_\\delta \\leqslant \\delta/(5mL) \\leqslant \\delta/(3L)$ for all $k \\in \\mathbb{N}$, there exists $k^* \\in \\mathbb{N}$ such that $t_{k^*} := \\sum_{k=0}^{k^*-1} \\alpha_k \\in [ t^* - \\delta/(3L) , t^*]$, where $\\sum_{k=0}^{k^*-1} \\alpha_k = 0$ if $k^* = 0$. Thus $\\|x_{k^*}-x^*\\| \\leqslant \\|x_{k^*}-x(t_{k^*})\\| + \\|x(t_{k^*}) - x(t^*)\\| + \\|x(t^*)-x^*\\| \\leqslant \\delta/3 + L | t_{k^*}-t^* | + \\delta/3 \\leqslant \\delta/3+\\delta/3+\\delta/3 = \\delta$.\nIf $K := \\inf\\{ k \\geqslant k^* : x_k \\notin \\mathring{B}(C,\\epsilon)\\}<\\infty$, then\n\\begin{subequations}\n    \\begin{align}\n        2m~\\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right) & = 2m~ \\psi^{-1}\\left(\\frac{1}{8m} \\epsilon - \\frac{L}{2}\\frac{\\epsilon}{5mL} \\right) \\label{eq:decrease_discrete_a} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m}(\\epsilon-\\delta) - \\frac{L\\alpha_\\delta}{2} \\right) \\label{eq:decrease_discrete_b} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m} \\left(\\|x_K-x^*\\|-\\|x_{k^*}-x^*\\|\\right) - \\frac{L\\alpha_\\delta}{2} \\right) \\label{eq:decrease_discrete_c} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m} \\|x_K-x_{k^*}\\| - \\frac{L\\bar{\\alpha}}{2} \\right) \\label{eq:decrease_discrete_d} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m}\\sum_{k=k^*}^{K-1} \\|x_{k+1}-x_k\\| - \\frac{L}{2}\\max_{k=0,\\hdots,K-1} \\alpha_k \\right) \\label{eq:decrease_discrete_e} \\\\\n        & \\leqslant f(x_{k^*}) - f(x_{K-1}) \\label{eq:decrease_discrete_f} \\\\\n        & \\leqslant m~\\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right) + \\max_C f - f(x_{K-1}). \\label{eq:decrease_discrete_g}\n    \\end{align}\n\\end{subequations}\nAbove, the arguments of $\\psi^{-1}$ in \\eqref{eq:decrease_discrete_a} are equal. \\eqref{eq:decrease_discrete_b} through \\eqref{eq:decrease_discrete_e} rely on the fact that $\\psi^{-1}$ is an increasing function. \\eqref{eq:decrease_discrete_b} is due to $\\delta<\\epsilon/2$. \\eqref{eq:decrease_discrete_c} holds because $x_K \\notin \\mathring{B}(C,\\epsilon)$, $x^* \\in C$, and $x_{k^*} \\in B(x^*,\\delta)$. \\eqref{eq:decrease_discrete_d} and \\eqref{eq:decrease_discrete_e} are consequences of the triangular inequality. \\eqref{eq:decrease_discrete_f} is due to the length formula \\eqref{eq:length_discrete_K} and the fact that $x_{k^*},\\hdots,x_{K-1} \\in \\mathring{B}(C,\\epsilon) \\subset \\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$. Finally, \\eqref{eq:decrease_discrete_g} is due to $x_{k^*}\\in B(C,\\delta)$ and \\eqref{eq:f_bound_discrete}. We remark that $K\\geqslant k^*+2$ since $\\|x_{k^*+1}-x^*\\| \\leqslant \\|x_{k^*+1}-x_{k^*}\\| +\\|x_{k^*}-x^*\\| \\leqslant \\alpha_{k^*}\\|\\nabla f(x_{k^*})\\|+\\delta \\leqslant (\\delta/(5mL))L+\\delta < \\epsilon$. Since $x_0,\\hdots,x_{K-2} \\in \\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$ and $x_{K-1}$ belongs to \n\\begin{equation*}\n    X_1 := B(C,\\epsilon) \\bigcap \\left\\{ x \\in \\mathbb{R}^n : f(x) \\leqslant \\max_{C} f - m~\\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right) \\right\\},    \n\\end{equation*}\nby the length formula \\eqref{eq:length_discrete_K} and the definition of $\\sigma(\\cdot,\\cdot)$ in \\eqref{eq:sup_discrete_X_0} we have\n    \\begin{align*}\n            \\sum_{k=0}^\\infty \\|x_{k+1}-x_k\\| & = \\sum_{k=0}^{K-2} \\|x_{k+1}-x_k\\| + \\sum_{k=K-1}^\\infty \\|x_{k+1}-x_k\\| \\\\\n            & \\leqslant 4m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(\\overline{\\Phi}_0,\\epsilon)} f\\right) \\right) + 2mL\\bar{\\alpha} + \\max\\{0,\\sigma(X_1,\\bar{\\alpha})\\}.\n    \\end{align*}\nNote that the inequality still holds if $K = \\infty$ or $\\sum_{k=0}^{\\infty} \\alpha_k \\leqslant T$. Hence\n\\begin{equation*}\n    \\sigma(X_0,\\bar{\\alpha}) ~\\leqslant~ 4m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(\\overline{\\Phi}_0,\\epsilon)} f\\right) \\right) + 2mL\\bar{\\alpha} + \\max\\{0,\\sigma(X_1,\\bar{\\alpha})\\}.\n\\end{equation*}\n\nIt now suffices to replace $X_0$ by $X_1$ and repeat the proof starting below \\eqref{eq:sup_discrete_X_0}. Since $f(\\Phi(t,x_1)) \\leqslant f(\\Phi(0,x_1)) \\leqslant \\max_{C} f - m\\psi^{-1}(\\epsilon/(40m))< \\max_{C} f$ for all $t\\geqslant 0$ and $x_1\\in X_1$, the maximal critical value of $f$ in $\\overline{\\Phi(\\mathbb{R}_+,X_1)}$ is less than the maximal critical value of $f$ in $\\overline{\\Phi(\\mathbb{R}_+,X_0)}$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Thus, it is eventually the case that one of the sets $X_0,X_1,\\hdots$ is empty. In order to conclude, one simply needs to choose an upper bound on the step sizes $\\hat{\\alpha}$ corresponding to $X_1$ that is less than or equal to the upper bound $\\bar{\\alpha}$ used for $X_0$. $\\sigma(X_0,\\cdot)$ is finite when evaluated at the last upper bound thus obtained. Indeed, the above recursive formula still holds if we replace $\\bar{\\alpha}$ by any $\\alpha \\in (0,\\bar{\\alpha}]$. In particular, we may take $\\alpha := \\hat{\\alpha}$.\n\n($3 \\Longrightarrow 1$) Obvious.\\\\\n\n","units":[{"raw_tex":"\\begin{lemma}\n\\label{lemma:uniform_length}\nIf a locally Lipschitz definable function has bounded continuous subgradient trajectories, then continuous subgradient trajectories initialized in a bounded set have uniformly bounded lengths.\n\\end{lemma}","start":12812,"end":13059},{"raw_tex":"\\begin{proposition}\n\\label{prop:existence_trajectory}\nLocally Lipschitz functions have at least one continuous subgradient trajectory for every initial point.\n\\end{proposition}","start":31369,"end":31545},{"raw_tex":"\\begin{proof}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz function and let $x_0\\in\\mathbb{R}^n$. Since $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d) p. 29]{clarke1990} with nonempty, compact and convex values \\cite[2.1.2 Proposition (a) p. 27]{clarke1990}. By virtue of \\cite[Theorem 3 p. 98]{aubin1984differential}, there exist $\\epsilon>0$ and an absolutely continuous function $x:[0,\\epsilon)\\rightarrow\\mathbb{R}^n$ such that  \n\\begin{equation*}\n    x'(t) \\in -\\partial f(x(t)), ~\\mathrm{for~a.e.}~ t\\in (0,\\epsilon),~~~ x(0) = x_0.\n\\end{equation*}\nLet $S_{x_0}$ be the set of functions in $S$ from Definition \\ref{def:trajectory} such that $x(0) = x_0$. Then $S_{x_0} \\neq \\emptyset$. Let $P$ be a nonempty totally ordered subset of $S_{x_0}$. Let $0<T \\leqslant\\infty$ be the supremum of the endpoints of the domains of all functions in $P$. Consider the function $x:[0,T)\\rightarrow\\mathbb{R}^n$ such that for all $\\bar{x}:[0,\\overline{T})\\rightarrow \\mathbb{R}^n$ in $P$ we have $x(t) := \\bar{x}(t)$ for all $t\\in [0,\\overline{T})$. It is well defined and it is an upper bound of $P$ in $S_{x_0}$. By virtue of Zorn's lemma \\cite[Corollary 2.5 p. 884]{lang2012algebra}, $S_{x_0}$ contains a maximal element. \\end{proof}","start":31546,"end":32866},{"raw_tex":"\\begin{proposition}\n\\label{prop:bounded_global}\nBounded continuous subgradient trajectories are globally defined.\n\\end{proposition}","start":32868,"end":32999},{"raw_tex":"\\begin{proof}\nLet $x:[0,T)\\rightarrow\\mathbb{R}^n$ be a bounded continuous subgradient trajectory of $f$ where $0<T \\leqslant \\infty$. We reason by contradiction and assume that $T<\\infty$. By Definition \\ref{def:bounded_trajectory}, there exists $c>0$ such that $\\|x(t)\\| \\leqslant c$ for all $t\\in [0,T)$. Consider a sequence $t_0,t_1,t_2,\\hdots \\in [0,T)$ converging to $T$. For all $k\\in \\mathbb{N}$, let $x_k:[0,T]\\rightarrow\\mathbb{R}^n$ be defined by $x_k(t) := x(t)$ for all $t\\in [0,t_k]$ and $x_k(t) = 0$ for all $t\\in (t_k,T]$. By construction, $x'_k$ converges pointwise almost everywhere to $x'$ on $(0,T)$. Let $L>0$ be a Lipschitz constant of $f$ on $B(0,c)$. By \\cite[2.1.2 Proposition (a) p. 27]{clarke1990}, for all $x\\in B(0,c)$ and $s\\in \\partial f(x)$, we have $\\|s\\| \\leqslant L$. For almost every $t\\in (0,T)$, we thus have $\\|x_k'(t)\\| \\leqslant \\|x'(t)\\| \\leqslant L$. By Lebesgue's dominated convergence theorem, $\\int_0^{t_k} x'(t)dt = \\int_0^T x'_k(t)dt \\rightarrow \\int_0^T x'(t)dt$. Since $x(\\cdot)$ is absolutely continuous on $[0,t_k]$, we have \n$x(t_k) - x(0) = \\int_0^{t_k} x'(t)dt$. Thus $x(\\cdot)$ can be extended to an absolutely continuous function on $[0,T]$ where \n$x(T) := x(0) + \\int_0^T x'(t)dt$. According to Proposition \\ref{prop:existence_trajectory}, $x(\\cdot)$ can further be extended to an absolutely continuous function on $[0,T+\\epsilon)$ for some $\\epsilon>0$ while satisfying the differential inclusion. This contradicts the maximality of $x(\\cdot)$. \n\\end{proof}","start":33000,"end":34516},{"raw_tex":"\\begin{proposition}\n\\label{prop:tracking}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a lower bounded differentiable function with a locally Lipschitz gradient. Let $X_0$ be a bounded subset of $\\mathbb{R}^n$ and let $T>0$. For all $\\epsilon>0$, there exists $\\bar{\\alpha}>0$ such that for all $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$ and for all sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ such that $x_0 \\in X_0$ and $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$, the continuous gradient trajectory $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ of $f$ initialized at $x_0$ satisfies\n\\begin{equation}\n\\label{eq:tracking}\n   \\forall k\\in \\mathbb{N}^*,~~~ \\alpha_0+\\cdots+\\alpha_{k-1} \\leqslant T ~~~\\Longrightarrow~~~ \\|x_k - x(\\alpha_0+\\cdots+\\alpha_{k-1})\\| \\leqslant \\epsilon.\n\\end{equation}\n\\end{proposition}","start":37589,"end":38440},{"raw_tex":"\\begin{proof}\nWithout loss of generality, we may assume that $X_0 \\neq \\emptyset$. Let $L>0$ and $M>0$ respectively denote Lipschitz constants of $f$ and $\\nabla f$ with respect to $\\|\\cdot\\|$ on the convex hull of $B(X_0,\\sigma_T(X_0)+1):= X_0 + B(0,\\sigma_T(X_0)+1)$ where \n\\begin{subequations}\n\\label{eq:sup_trajectory_T_smooth}\n\\begin{align}\n    \\sigma_T(X_0) := & \\sup\\limits_{x \\in \\mathcal{C}^1(\\mathbb{R}_+,\\mathbb{R}^n)} ~~ \\int_0^T \\|x'(t)\\|dt \\\\\n  & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx'(t) = - \\nabla f(x(t)), ~\\forall t > 0,\\\\[3mm] x(0) \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\nAbove, $\\mathcal{C}^1(\\mathbb{R}_+,\\mathbb{R}^n)$ is the set of continuous functions on $[0,\\infty )$ that are continuously differentiable on $(0,\\infty)$. Note that $\\sigma_T(X_0)>-\\infty$ since $f$ is lower bounded and differentiable with a locally Lipschitz gradient. For every $x_0\\in X_0$, there hence exists a unique globally defined continuous gradient trajectory of $f$ initialized at $x_0$ \\cite[Theorem 17.1.1]{attouch2014variational}. Also, $\\sigma_T(X_0)<\\infty$ since for any feasible point $x(\\cdot)$ of \\eqref{eq:sup_trajectory_T_smooth}, we have \n\\begin{subequations}\n\\begin{align}\n    \\int_0^T \\|x'(t)\\|dt & \\leqslant \\sqrt{T} \\sqrt{ \\int_0^T \\|x'(t)\\|^2 dt} \\label{eq:length_finite_smooth_a} \\\\\n    & = \\sqrt{T} \\sqrt{ -\\int_0^T \\langle \\nabla f(x(t)), x'(t) \\rangle dt} \\label{eq:length_finite_smooth_b} \\\\\n    & = \\sqrt{T} \\sqrt{ -\\int_0^T (f \\circ x)'(t) dt} \\label{eq:length_finite_smooth_c} \\\\\n    & = \\sqrt{T} \\sqrt{ f(x(0)) - f(x(T))} \\label{eq:length_finite_smooth_d} \\\\\n    & \\leqslant \\sqrt{T\\left(\\sup_{X_0} f - \\inf_{\\mathbb{R}^n} f\\right)} < \\infty. \\label{eq:length_finite_smooth_e}\n\\end{align}\n\\end{subequations}\nIndeed, \\eqref{eq:length_finite_smooth_a} is due to the Cauchy-Schwarz inequality. \\eqref{eq:length_finite_smooth_e} holds because $X_0$ is bounded and $f$ is lower bounded.\n\nIt is easy to check that $L$ and $ML$ are respectively Lipschitz and gradient Lipschitz constants on $[0,T]$ of all continuous gradient trajectories of $f$ initialized in $X_0$. Indeed, let $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ be a continuous gradient trajectory initialized in $X_0$. Since $x(t) \\in B(X_0,\\sigma_T(X_0))$ for all $t\\in [0,T]$, we have $\\|x'(t)\\| = \\|\\nabla f(x(t))\\| \\leqslant L$. By the mean value theorem \\cite[5.19 Theorem]{rudin1964principles}, for all $s,t \\in [0,T]$ we have $\\|x'(t)-x'(s)\\| = \\|\\nabla f(x(t)) - \\nabla f(x(s))\\| \\leqslant M \\|x(t)-x(s)\\| \\leqslant M L |t-s|$. As a byproduct, we get the Taylor bound\n\\begin{subequations}\n    \\begin{align}\n    \\|x(t)-x(s) - x'(s)(t-s)\\| & = \\left\\| \\int_s^t (x'(\\tau) - x'(s)) d\\tau \\right\\| \\\\\n    & \\leqslant \\left| \\int_s^t \\| x'(\\tau) - x'(s) \\| d\\tau \\right| \\\\\n    & \\leqslant \\left|\\int_s^t ML | \\tau - s | d\\tau \\right| \\\\\n    & = \\frac{ML}{2}(t-s)^2. \\label{eq:local_error}\n    \\end{align}\n\\end{subequations}\n\nLet $\\epsilon \\in (0,1)$ and let $\\bar{\\alpha} := 2\\epsilon e^{-MT}/(LMT)$. Consider $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$ and a sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$ and $x_0 \\in X_0$. Let $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ be the unique continuous gradient trajectory of $f$ initialized at $x_0$. Let $t_0:=0$ and $t_k := \\alpha_0+\\cdots+\\alpha_{k-1}$ when $k\\in \\mathbb{N}^*$ so that $\\|x_0 - x(t_0)\\| = 0 \\leqslant \\epsilon$. Assume that \\eqref{eq:tracking} holds up to some index $K$. For $k=0,\\hdots,K$, we have\n\\begin{subequations}\n    \\begin{align}\n        \\|x_{k+1} - x(t_{k+1})\\| = &  ~\\| x_k - \\alpha_k \\nabla f(x_k) - x(t_{k+1}) \\| \\label{local_a} \\\\[2mm]\n        \\leqslant  & ~ \\| x_k - \\alpha_k \\nabla f(x_k) - [x(t_k) - \\alpha_k \\nabla f(x(t_k))] \\|~+ \\label{local_b} \\\\[2mm] & ~ \\| x(t_k) - \\alpha_k \\nabla f(x(t_k)) - x(t_{k+1}) \\| \\label{local_c} \\\\[2mm]\n        \\leqslant & ~ \\|x_k - x(t_k)\\| + \\alpha_k \\|\\nabla f(x_k) - \\nabla f(x(t_k))\\| ~+ \\label{local_d} \\\\[2mm]\n        & ~ ML\\alpha_k^2/2 \\label{local_e} \\\\[2mm]\n        \\leqslant & ~ (1+\\alpha_k M)\\|x_k - x(t_k)\\| + ML\\alpha_k^2/2. \\label{local_f}\n    \\end{align}\n\\end{subequations}\nThe term in \\eqref{local_c} is equal to the local truncation error $\\|x(t_{k+1})  - x(t_k) - x'(t_k)(t_{k+1}-t_k) \\|$, and can hence be bounded above using \\eqref{eq:local_error}. In the second term of \\eqref{local_d}, we invoke a Lipschitz constant of $\\nabla f$ on the convex hull of $B(X_0,\\sigma_T(X_0)+1)$. Indeed, by the induction hypothesis and $\\epsilon < 1$, $x_k$ and $x(t_k)$ belong to $B(X_0,\\sigma_T(X_0)+\\epsilon)$. Hence \n\\begin{subequations}\n    \\begin{align}\n        \\|x_{K+1} - x(t_{K+1})\\| \\leqslant &~ \\prod_{k=0}^K (1+\\alpha_kM)\\|x_0-x(t_0)\\|~+ \\label{eq:global_a}\\\\\n        &~ \\sum_{k=0}^K \\frac{ML\\alpha_k^2}{2} \\prod_{l=k+1}^{K} (1+\\alpha_lM)\\label{eq:global_b}\\\\\n        \\leqslant &~ \\prod_{l=0}^{K} (1+\\alpha_lM) \\sum_{k=0}^K \\frac{ML\\alpha_k^2}{2}\\label{eq:global_c}\\\\\n        \\leqslant &~ \\left(1+\\frac{M}{K+1}\\sum_{k=0}^K \\alpha_k\\right)^{K+1} \\sum_{k=0}^K \\frac{ML\\bar{\\alpha}\\alpha_k}{2}\\label{eq:global_d}\\\\\n        \\leqslant &~ \\left(1+\\frac{MT}{K+1}\\right)^{K+1} \\frac{ML\\bar{\\alpha}T}{2}\\label{eq:global_e}\\\\\n        \\leqslant &~~ \\bar{\\alpha} LMTe^{MT}/2\\label{eq:global_f}\\\\\n        = &~~ \\epsilon.\\label{eq:global_g}\n    \\end{align}\n\\end{subequations}\nThe term in \\eqref{eq:global_a} is equal to zero because $x_0 = x(t_0)$. By convention, we set the product in \\eqref{eq:global_b} to be equal to one if the index set for $l$ is empty. We bound the partial products ranging from $k+1$ to $K$ inside the summation of \\eqref{eq:global_b} by the full product ranging from $0$ to $K$ in order to obtain \\eqref{eq:global_c}. We then use the inequality of arithmetic and geometric means and $\\alpha_k\\leqslant\\bar{\\alpha}$ in order to obtain \\eqref{eq:global_d}. In order to prove the induction step, we assume that $\\alpha_0+\\cdots+\\alpha_K \\leqslant T$, hence \\eqref{eq:global_e}. \\eqref{eq:global_f} follows from $\\ln(1+x) \\leqslant x$ for all $x>0$. Finally, \\eqref{eq:global_g} holds because $\\bar{\\alpha} = 2\\epsilon e^{-MT}/(LMT)$.\n\\end{proof}","start":38442,"end":44644},{"raw_tex":"\\begin{proposition}\n\\label{prop:KL}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz definable function and let $X$ be a bounded subset of $\\mathbb{R}^n$. Let $V$ be the set of critical values of $f$ in $\\overline{X}$ if it is nonempty, otherwise let $V:=\\{0\\}$. There exists a concave definable diffeomorphism $\\psi:[0,\\infty)\\rightarrow[0,\\infty)$ such that\n\\begin{equation*}\n    \\forall x \\in X \\setminus (\\partial \\tilde{f})^{-1}(0), ~~~ d(0,\\partial (\\psi \\circ \\tilde{f})(x)) \\geqslant 1\n\\end{equation*}\nwhere $\\tilde{f}(x) := d(f(x),V)$ for all $x\\in \\mathbb{R}^n$.\n\\end{proposition}","start":49370,"end":49978},{"raw_tex":"\\begin{proof}\nBy the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, the set $V$ is finite. In addition, by the Kurdyka-\\L{}ojasiewicz inequality \\cite[Corollary 15]{bolte2007clarke}, there exist $\\epsilon>0$ and a strictly increasing concave continuous definable function $\\psi:[0,\\epsilon)\\rightarrow \\mathbb{R}$ that is continuously differentiable on $(0,\\epsilon)$ with $\\psi(0) = 0$ such that, for all $x \\in X$ and $v \\in V$ satisfying $0 < | f(x)-v | < \\epsilon$, we have $d(0,\\partial (\\psi \\circ | f-v|)(x)) \\geqslant 1$. The fact that $\\psi$ can be assumed to be concave is due to the monotonicity theorem \\cite[(1.2) p. 43]{van1998tame} \\cite[Lemma 2]{kurdyka1998gradients} by following the argument in \\cite[Theorem 4.1]{attouch2010proximal}. Let $\\rho:= \\inf \\{(v-v')/2: v,v' \\in V, v > v'\\}>0$ and $\\rho' : = \\min \\{ \\rho, \\epsilon\\}$. Observe that $c>0$ where\n\\begin{equation*}\n    c := \\inf \\{ \\|s\\| : \ns \\in \\partial f(x), ~ x\\in X, ~ | f(x) - v | \\geqslant \\rho',~ v\\in V \\}.\n\\end{equation*}\nIndeed, assume that there exist sequences $(s_k)_{k\\in \\mathbb{N}}$ and $(x_k)_{k\\in\\mathbb{N}}$ such that $\\|s_k\\|$ converges to zero and $s_k \\in \\partial f(x_k)$, $x_k\\in X$, and $| f(x_k) - v | \\geqslant \\rho'$ for all $v\\in V$. Since $X$ is bounded, there exist subsequences (again denoted $s_k$ and $x_k$) such that $x_k$ has a limit $x^*$ in $\\overline{X}$. Since $f$ is continuous, $| f(x^*) - v | \\geqslant \\rho'>0$ for all $v \\in V$. Also, since $s_k$ converges to zero, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990}, we have $0\\in \\partial f(x^*)$. This yields a contradiction. \n\nAfter possibly reducing $\\epsilon$, we may assume that $\\lim_{t\\nearrow\\rho'} \\psi'(t)>0$. If $\\lim_{t\\nearrow\\rho'} \\psi'(t) \\geqslant 1/c$, then we may extend $\\psi$ to an affine function on $[\\rho',\\infty)$ with slope equal to $\\lim_{t\\nearrow\\rho'} \\psi'(t)$. Otherwise, we can multiply $\\psi$ by $1/(c\\lim_{t\\nearrow\\rho'} \\psi'(t))$ before taking the affine extension. Now let $x \\in X$ be such that $0 \\notin \\partial \\tilde{f}(x)$. If $f(x)$ is strictly within $\\rho'$ distance of some $v \\in V$, then from $\\rho' \\leqslant \\rho$ it follows that $\\tilde{f}(\\tilde{x}) = d(f(\\tilde{x}),V) = | f(\\tilde{x})-v|$ for all $\\tilde{x}$ in a neighborhood of $x$ in $\\mathbb{R}^n$, and thus $d(0,\\partial ( \\psi \\circ \\tilde{f})(x)) = d(0,\\partial (\\psi \\circ | f-v|)(x)) \\geqslant 1$. The inequality is due to $\\rho' \\leqslant \\epsilon$. Otherwise, by \\cite[2.3.9 Theorem (Chain Rule I) (ii) p. 42]{clarke1990} we have $d(0,\\partial ( \\psi \\circ \\tilde{f})(x)) = \\psi'(\\tilde{f}(x)) d(0,\\partial \\tilde{f} (x)) = \\psi'(\\tilde{f}(x)) d(0,\\partial f(x)) \\geqslant \\psi'(\\tilde{f}(x)) c \\geqslant 1$. \n\\end{proof}","start":49979,"end":52712},{"raw_tex":"\\begin{proposition}\n\\label{prop:chain_rule}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a definable function and let $x:(0,T)\\rightarrow \\mathbb{R}^n$ with $0 < T \\leqslant \\infty$ be an absolutely continuous function. If $f$ is locally Lipschitz at $x(t)$ for all $t \\in (0,T)$ and $x'(t) \\in -\\partial f(x(t))$ for almost every $t\\in(0,T)$, then\n\\begin{equation*}\n    (f \\circ x)'(t) = -\\|x'(t)\\|^2 = -d(0,\\partial f(x(t)))^2,~~~\\mathrm{for~a.e.}~t \\in (0,T).\n\\end{equation*}\n\\end{proposition}","start":52900,"end":53398},{"raw_tex":"\\begin{proof}\nSince $f$ is definable and locally Lipschitz at $x(t)$ for all $t\\in(0,T)$, it satisfies the chain rule \\cite[Proposition 2 (iv)]{bolte2020conservative} (see also \\cite[Theorem 5.8]{davis2020stochastic} and \\cite[Lemma 2.10]{drusvyatskiy2015quadratic}), that is to say $(f \\circ x)'(t) = \\langle s , x'(t) \\rangle$ for all $s\\in \\partial f(x(t))$ and almost every $t\\in (0,T)$. In particular, we may take $s := - x'(t) \\in \\partial f(x(t))$.\nFinally, by \\cite[Lemma 5.2]{davis2020stochastic} (see also \\cite[Proposition 4.10]{drusvyatskiy2015curves}) it holds that $\\|x'(t)\\| = d(0,\\partial f(x(t)))$ for almost every $t\\in (0,T)$. \n\\end{proof}","start":53399,"end":54057},{"raw_tex":"\\begin{proposition}\n\\label{prop:length_continuous}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz definable function and let $X$ be a bounded subset of $\\mathbb{R}^n$. Assume that $f$ has at most $m\\in \\mathbb{N}^*$ critical values in $\\overline{X}$. Let $\\psi$ be a desingularizing function of $f$ on $X$. If $x:[0,T]\\rightarrow X$ with $T \\geqslant 0$ is an absolutely continuous function such that $x'(t) \\in -\\partial f(x(t))$ for almost every $t\\in(0,T)$, then\n\\begin{equation}\n\\label{eq:length_continuous}\n    \\frac{1}{2m}\\int_0^T\\|x'(t)\\|dt ~\\leqslant~ \\psi\\left(\\frac{f(x(0))-f(x(T))}{2m}\\right).\n\\end{equation}\n\\end{proposition}","start":54179,"end":54836},{"raw_tex":"\\begin{proof}\nWe first consider the special case where $0 \\notin \\partial \\tilde{f}(x(t))$ for all $t\\in (0,T)$ where $\\tilde{f}$ is defined as in Proposition \\ref{prop:KL}. Consider the change of variables $\\bar{x} = x \\circ \\varphi$ where $\\bar{t}=\\varphi^{-1}(t) = \\int_0^{t} ds /(\\psi' \\circ \\tilde{f} \\circ x)(s)$ and $t=\\varphi(\\bar{t}) = \\int_0^{\\bar{t}} (\\psi' \\circ \\tilde{f} \\circ \\bar{x})(\\bar{s}) d\\bar{s}$ by the inverse function theorem \\cite[Theorem 17.7.2]{tao2006analysis}. We have $\\bar{x}'(\\bar{t}) = \\varphi'(\\bar{t})x'(t) \\in -(\\psi' \\circ \\tilde{f} \\circ \\bar{x})(\\bar{t}) \\partial f(\\bar{x}(\\bar{t})) = \\pm \\partial (\\psi \\circ \\tilde{f})(\\bar{x}(\\bar{t}))$ for almost every $\\bar{t}\\in (0,\\overline{T})$, where $\\overline{T} = \\varphi^{-1}(T)$ and the sign is constant. By Proposition \\ref{prop:chain_rule} and the Cauchy-Schwarz inequality, we thus have\n\\begin{subequations}\n\\begin{align}\n    \\overline{T} \\leqslant & \\int_0^{\\overline{T}} d(0,\\partial (\\psi \\circ \\tilde{f})(\\bar{x}(\\bar{t})))d\\bar{t} \\label{eq:god_a} \\\\\n    = & \\int_0^{\\overline{T}} \\|\\bar{x}'(\\bar{t})\\|d\\bar{t} \\label{eq:god_b} \\\\\n    = & \\int_0^{T} \\|x'(t)\\|dt \\\\\n    \\leqslant & \\sqrt{\\overline{T}} \\sqrt{\\int_0^{\\overline{T}} \\|\\bar{x}'(\\bar{t})\\|^2d\\bar{t}} \\\\\n    = & \\sqrt{\\overline{T}}\\sqrt{|(\\psi\\circ \\tilde{f}\\circ\\bar{x})(0)-(\\psi\\circ \\tilde{f}\\circ\\bar{x})(\\overline{T})|}.\n\\end{align}\n\\end{subequations}\nHence $\\overline{T} \\leqslant |(\\psi\\circ \\tilde{f}\\circ\\bar{x})(0)-(\\psi\\circ \\tilde{f}\\circ\\bar{x})(\\overline{T})|$ and, since $\\psi$ is concave, we have\n\\begin{align*}\n     \\int_0^T \\|x'(t)\\|dt & \\leqslant |(\\psi\\circ \\tilde{f}\\circ\\bar{x})(0)-(\\psi\\circ \\tilde{f}\\circ\\bar{x})(\\overline{T})| \\\\\n     & \\leqslant \\psi (|( \\tilde{f}\\circ\\bar{x})(0)-(\\tilde{f}\\circ\\bar{x})(\\overline{T})|) \\\\[2mm]\n     & = \\psi (f(x(0))-f(x(T))).\n\\end{align*}\nWe next consider the general case where there may exist $t\\in (0,T)$ such that $0 \\in \\partial \\tilde{f}(x(t))$, namely $0 < t_1 < \\cdots < t_{k} < T$, and potentially $(t_k,T)$. For notational convenience, let $t_0 := 0$ and $t_{k+1}:=T$ where $k$ is possibly equal to zero. Since $k \\leqslant 2m-1$ and $\\psi$ is concave, we have\n\\begin{align*}\n    \\int_0^T \\|x'(t)\\|dt & = \\sum_{i=0}^k \\int_{t_i}^{t_{i+1}} \\|x'(t)\\|dt \\\\\n    & \\leqslant \\sum_{i=0}^k \\psi (f(x(t_i))-f(x(t_{i+1}))) \\\\[-.5mm]\n    & \\leqslant (k+1) ~ \\psi \\left( \\frac{1}{k+1} \\sum_{i=0}^k f(x(t_i))-f(x(t_{i+1}))\\right) \\\\\n    & = (k+1) ~ \\psi \\left( \\frac{f(x(0)) - f(x(T))}{k+1} \\right) \\\\\n    & \\leqslant 2m ~ \\psi \\left( \\frac{f(x(0)) - f(x(T))}{2m} \\right). \\qedhere\n\\end{align*}\n\\end{proof}","start":54837,"end":57462},{"raw_tex":"\\begin{proposition}\n\\label{prop:length_discrete}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a definable differentiable function with a locally Lipschitz gradient and let $X$ be a bounded open subset of $\\mathbb{R}^n$. Assume that $f$ has at most $m\\in \\mathbb{N}^*$ critical values in $\\overline{X}$ and that $L>0$ is a Lipschitz constant of $f$ on the convex hull of $X$. Let $\\psi$ be a desingularizing function of $f$ on $X$ and $\\epsilon>0$. There exists $\\bar{\\alpha}>0$ such that, for all $K \\in \\mathbb{N}$, $\\alpha_0,\\hdots,\\alpha_K \\in (0,\\bar{\\alpha}]$, and $(x_0,\\hdots,x_{K+1}) \\in X \\times \\cdots \\times X \\times \\mathbb{R}^n$ such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for $k = 0,\\hdots,K$, we have\n\\begin{equation}\n\\label{eq:length_discrete}\n    \\frac{1}{(2+\\epsilon)m}\\sum_{k=0}^K\\|x_{k+1}-x_k\\| ~\\leqslant~ \\psi\\left(\\frac{f(x_0)-f(x_K)}{2m}\\right) + \\frac{2L}{2+\\epsilon}\\max_{k=0,\\hdots,K} \\alpha_k\n\\end{equation}\nand $f(x_0)\\geqslant \\cdots \\geqslant f(x_{K+1})$.\n\\end{proposition}","start":57545,"end":58558},{"raw_tex":"\\begin{proof}\nLet $M>0$ be a Lipschitz constant of $\\nabla f$ with respect to $\\|\\cdot\\|$ on the convex hull of $B(X,1):=X+B(0,1)$ and let $\\bar{\\alpha} := \\min\\{L^{-1},2\\epsilon(6+\\epsilon)^{-1}M^{-1}\\}$. Let $K \\in \\mathbb{N}$, $\\alpha_0,\\hdots,\\alpha_K \\in (0,\\bar{\\alpha}]$, and $(x_0,\\hdots,x_{K+1}) \\in X \\times \\cdots \\times X \\times \\mathbb{R}^n$ be such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for $k=0,\\hdots,K$. It holds that $\\|x_{K+1}-x_K\\| = \\alpha_K\\|\\nabla f(x_K)\\| \\leqslant \\bar{\\alpha}L \\leqslant 1$ and thus $x_0,\\hdots,x_{K+1} \\in B(X,1)$. A bound on the Taylor expansion of $f$ yields\n\\begin{align*}\n    f(x_{k+1})-f(x_k) & \\leqslant \\langle \\nabla f(x_k), x_{k+1}-x_k\\rangle + \\frac{M}{2}\\|x_{k+1}-x_k\\|^2 \\\\\n    & = \\left(\\frac{M\\alpha_k}{2}-1\\right) \\|x_{k+1}-x_k\\| \\|\\nabla f(x_k)\\|\n\\end{align*}\nfor $k=0,\\hdots,K$, and thus\n\\begin{equation}\n\\label{eq:descent_lemma_1}\n    \\|x_{k+1}-x_k\\|\\|\\nabla f(x_k)\\| \\leqslant \\frac{2}{2- M\\alpha_k}(f(x_k) - f(x_{k+1})).\n\\end{equation}\nWe also have\n    \\begin{align*}\n        \\|\\nabla f(x_{k+1})\\| & \\leqslant \\| \\nabla f(x_{k+1}) - \\nabla f(x_k) \\| + \\|\\nabla f(x_k)\\| \\\\\n        & \\leqslant M \\| x_{k+1} - x_k \\| + \\|\\nabla f(x_k)\\| \\\\\n        & \\leqslant (M\\alpha_k+1) \\|\\nabla f(x_k)\\|\n    \\end{align*}\nand thus \n\\begin{equation}\n\\label{eq:descent_lemma_2}\n    \\|x_{k+1}-x_k\\|\\|\\nabla f(x_{k+1})\\| \\leqslant \\frac{2+2M\\alpha_k}{2- M\\alpha_k}(f(x_k) - f(x_{k+1})).\n\\end{equation}\nNote that \n\\begin{equation*}\n    1 \\leqslant \\frac{2}{2- M\\alpha_k} \\leqslant \\frac{2+2M\\alpha_k}{2- M\\alpha_k} \\leqslant 1+\\frac{\\epsilon}{2}.\n\\end{equation*}\nLet $\\tilde{f}$ and $V$ be defined as in Proposition \\ref{prop:KL}.\n\nAssume that $[f(x_K),f(x_0)]$ excludes the elements of $V$ and the averages of any two consecutive elements of $V$. Then $0\\notin \\partial \\tilde{f}(x_k)$ and $1 \\leqslant \\| \\nabla(\\psi \\circ \\tilde{f})(x_k)\\| = \\psi'(\\tilde{f}(x_k)) \\|\\nabla \\tilde{f}(x_k)\\|$ for $k=0,\\hdots,K$. Let $k \\in \\{0,\\hdots,K-1\\}$. If $\\tilde{f}(x_k) \\geqslant \\tilde{f}(x_{k+1})$, then multiplying \\eqref{eq:descent_lemma_1} by $\\psi'(\\tilde{f}(x_k))$ and using the concavity of $\\psi$, we find that \n\\begin{align*}\n    \\|x_{k+1}-x_k\\| & \\leqslant (1+\\epsilon/2)\\psi'(\\tilde{f}(x_k))(f(x_k) - f(x_{k+1})) \\\\\n    & = (1+\\epsilon/2)\\psi'(\\tilde{f}(x_k))(\\tilde{f}(x_k) - \\tilde{f}(x_{k+1})) \\\\\n    & \\leqslant (1+\\epsilon/2)(\\psi (\\tilde{f}(x_k)) - \\psi (\\tilde{f}(x_{k+1}))).\n\\end{align*}\nIf $\\tilde{f}(x_k) \\leqslant \\tilde{f}(x_{k+1})$, then multiplying \\eqref{eq:descent_lemma_2} by $\\psi'(\\tilde{f}(x_{k+1}))$ and using the concavity of $\\psi$, we find that \n\\begin{align*}\n    \\|x_{k+1}-x_k\\| & \\leqslant (1+\\epsilon/2)\\psi'(\\tilde{f}(x_{k+1}))(f(x_k) - f(x_{k+1})) \\\\\n    & = (1+\\epsilon/2)\\psi'(\\tilde{f}(x_{k+1}))(\\tilde{f}(x_{k+1}) - \\tilde{f}(x_k)) \\\\\n    & \\leqslant (1+\\epsilon/2)(\\psi (\\tilde{f}(x_{k+1})) - \\psi (\\tilde{f}(x_k))).\n\\end{align*}\nAs a result,\n\\begin{equation*}\n    \\|x_{k+1}-x_k\\| \\leqslant (1+\\epsilon/2)|\\psi (\\tilde{f}(x_k)) - \\psi (\\tilde{f}(x_{k+1}))|, ~~~ k = 0,\\hdots,K-1.\n\\end{equation*}\nWe obtain the telescoping sum \n\\begin{subequations}\n    \\label{eq:length_finite} \n    \\begin{align}\n    \\sum_{k=0}^K \\|x_{k+1}-x_k\\| & \\leqslant (1+\\epsilon/2) |\\psi(\\tilde{f}(x_0)) - \\psi(\\tilde{f}(x_K))| + \\|x_{K+1} - x_K\\| \\\\[-4mm]\n    & \\leqslant (1+\\epsilon/2) \\psi ( | \\tilde{f}(x_0) - \\tilde{f}(x_K) |) + \\alpha_K\\|\\nabla f(x_K)\\| \\\\\n    & \\leqslant (1+\\epsilon/2) \\psi( f(x_0) - f(x_K)) + L\\max_{k=0,\\hdots,K} \\alpha_k.\n    \\end{align}\n\\end{subequations}\n\nAssume that $[f(x_K),f(x_0))$ excludes the elements of $V$ and the averages of any two consecutive elements of $V$. Since we are excluding a finite number of elements, there exists $\\epsilon>0$ such that $[f(x_K)-\\epsilon,f(x_0))$ satisfies the same property. If $x_0$ is a local minimum of $f$ on the open set $X$, then $\\nabla f(x_0) = 0$ by Fermat's rule. The length of $x_0,\\hdots,x_{K+1}$ is then equal to zero and the formula obtained in \\eqref{eq:length_finite} trivially holds. Otherwise, there exists a sequence $(x_0^l)_{l\\in\\mathbb{N}}$ in $\\mathbb{R}^n$ such that $f(x_0^l) < f(x_0)$ and $x_0^l$ converges to $x_0$. For every $l \\in \\mathbb{N}$, consider the sequence $(x^l_k)_{k\\in \\mathbb{N}}$ such that $x^l_{k+1} = x^l_k - \\alpha_k \\nabla f(x_k^l)$ for all $k \\in \\mathbb{N}$. Since $\\nabla f$ is continuous, for any fixed $k \\in \\{0,\\hdots,K\\}$, $x_k^l$ converges to $x_k \\in X$ and eventually belongs to the open set $X$. Thus $[f(x^l_K),f(x^l_0)] \\subset [f(x_K)-\\epsilon,f(x_0))$ eventually excludes the elements of $V$ and the averages of any two consecutive elements of $V$. We can apply the formula obtained in \\eqref{eq:length_finite}, namely\n\\begin{equation*}\n    \\sum_{k=0}^K \\|x^l_{k+1}-x^l_k\\| \\leqslant (1+\\epsilon/2) \\psi( f(x^l_0) - f(x^l_K)) + L\\max_{k=0,\\hdots,K} \\alpha_k.\n\\end{equation*}\nPassing to the limit yields\n\\begin{equation*}\n    \\sum_{k=0}^K \\|x_{k+1}-x_k\\| \\leqslant (1+\\epsilon/2) \\psi( f(x_0) - f(x_K)) + L\\max_{k=0,\\hdots,K} \\alpha_k.\n\\end{equation*}\n\nWe next consider the general case where\n\\begin{equation*}\n    [f(x_K),f(x_{K_p+1})) \\cup \\cdots \\cup [f(x_{K_2}),f(x_{K_1+1})) \\cup [f(x_{K_1}),f(x_0))\n\\end{equation*}\nexcludes the elements of $V$ and the averages of any two consecutive elements of $V$. For notational convenience, let $K_0 := -1$ and $K_{p+1} := K$. Since $p \\leqslant 2m-1$, we have\n\\begin{align*}\n    \\sum_{k=0}^K \\|x_{k+1}-x_k\\| = & \\sum_{i=0}^p \\sum_{k=K_i+1}^{K_{i+1}} \\|x_{k+1}-x_k\\| \\\\\n    \\leqslant & \\sum_{i=0}^p \\left((1+\\epsilon/2)\\psi( f(x_{{K_i}+1}) - f(x_{K_{i+1}})) + L\\max_{k=0,\\hdots,K} \\alpha_k\\right) \\\\[-.5mm]\n    \\leqslant & (1+\\epsilon/2)(p+1) ~ \\psi \\left( \\frac{1}{p+1} \\sum_{i=0}^p  f(x_{{K_i}+1}) - f(x_{K_{i+1}}) \\right) + \\\\ \n    & (p+1)L\\max_{k=0,\\hdots,K} \\alpha_k \\\\\n    \\leqslant & (1+\\epsilon/2)(p+1) \\psi \\left( \\frac{f(x_0)-f(x_K)}{p+1} \\right) + (p+1)L\\max_{k=0,\\hdots,K} \\alpha_k \\\\\n    \\leqslant & (2+\\epsilon)m ~ \\psi \\left( \\frac{f(x_0)-f(x_K)}{2m} \\right) + 2m L\\max_{k=0,\\hdots,K} \\alpha_k.\\qedhere\n\\end{align*}\n\\end{proof}","start":58559,"end":64644},{"raw_tex":"\\section{Proof of Lemma \\ref{lemma:uniform_length}}\n\\label{sec:Proof of Lemma lemma:uniform_length}\n \nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz definable function with bounded continuous subgradient trajectories. Let $X_0$ be a bounded subset of $\\mathbb{R}^n$. We will show that $\\sigma(X_0) < \\infty$ where \n\\begin{subequations}\n\\label{eq:sup_trajectory}\n\\begin{align}\n    \\sigma(X_0) := & \\sup\\limits_{x \\in \\mathcal{A}(\\mathbb{R}_+,\\mathbb{R}^n)} ~~ \\int_0^{\\infty} \\|x'(t)\\|dt \\\\\n  & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx'(t) \\in - \\partial f(x(t)), ~\\mathrm{for~a.e.}~ t > 0,\\\\[3mm] x(0) \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\nWithout loss of generality, we may assume that $X_0\\neq \\emptyset$. By Propositions \\ref{prop:existence_trajectory} and \\ref{prop:bounded_global}, the feasible set of \\eqref{eq:sup_trajectory} is thus non-empty and $\\sigma(X_0)>-\\infty$.\n\nWe first consider the case with finite time horizon $T\\geqslant 0$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Notice that $\\sigma_T(X_0) \\leqslant \\sqrt{T(\\sup_{X_0} f - m(f))}$ where $m(f)$ is the smallest critical value of $f$ and \n\\begin{subequations}\n\\label{eq:sup_trajectory_T}\n\\begin{align}\n    \\sigma_T(X_0) := & \\sup\\limits_{x \\in \\mathcal{A}(\\mathbb{R}_+,\\mathbb{R}^n)} ~~ \\int_0^T \\|x'(t)\\|dt \\\\\n  & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx'(t) \\in - \\partial f(x(t)), ~\\mathrm{for~a.e.}~ t > 0,\\\\[3mm] x(0) \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\nIndeed, by the Cauchy-Schwarz inequality and Propositions \\ref{prop:chain_rule} and \\ref{prop:length_continuous}, for any feasible point $x(\\cdot)$ of \\eqref{eq:sup_trajectory_T} we have\n\\begin{subequations}\n\\begin{align}\n    \\int_0^T \\|x'(t)\\|dt & \\leqslant \\sqrt{T} \\sqrt{ \\int_0^T \\|x'(t)\\|^2 dt} \\\\\n    & \\leqslant \\sqrt{T} \\sqrt{ \\int_0^{\\infty} \\|x'(t)\\|^2 dt} \\\\\n    & = \\sqrt{T} \\sqrt{f(x(0)) - f\\left(\\lim_{t\\rightarrow\\infty} x(t)\\right)} \\\\\n    & \\leqslant \\sqrt{T\\left(\\sup_{X_0} f - m(f)\\right)}. \\label{eq:finite_time}\n\\end{align}\n\\end{subequations}\n\nWe next treat the case with infinite time horizon. Consider a sequence of feasible points $x_0(\\cdot), x_1(\\cdot), x_2(\\cdot), \\ldots$ of \\eqref{eq:sup_trajectory}  such that $\\int_0^{\\infty} \\|x_k'(t)\\|dt$ converges to $\\sigma(X_0)$. We proceed to show that the sequence is equicontinuous. Let $\\epsilon>0$ and $t\\geqslant 0$. Consider problem \\eqref{eq:sup_trajectory_T} with finite time horizon $T:=t+\\epsilon$. Since $x_0(\\cdot),x_1(\\cdot),x_2(\\cdot),\\ldots$ are feasible points of \\eqref{eq:sup_trajectory_T}, for all $s \\in [0,T]$ and $k=0,1,2,\\ldots$ we have\n\\begin{subequations}\n    \\begin{align}\n        \\|x_k(s)\\| & \\leqslant \\|x_k(s) - x_k(0) \\| + \\|x_k(0)\\| \\label{upper_0} \\\\ & \\leqslant \\int_0^s \\|x_k'(\\tau)\\|d\\tau + \\|x_k(0)\\| \\label{upper_1} \\\\\n        & \\leqslant \\sigma_T(X_0) + \\sup_{x_0 \\in X_0} \\|x_0\\|. \\label{upper_2}\n    \\end{align}\n\\end{subequations}\nAll the trajectories $x_0(\\cdot),x_1(\\cdot),x_2(\\cdot),\\ldots$ hence belong to a common ball centered at the origin up to time $T$. Let $L>0$ be a Lipschitz constant of $f$ on that ball. As a result, for all $s \\in [t-\\delta,t+\\delta]$ with $\\delta:=\\epsilon/(2L)$ and for all $k\\in \\mathbb{N}$, we have\n\\begin{equation*}\n    \\|x_k(t)-x_k(s)\\| = \\left\\| \\int_{s}^t x'_k(\\tau)d\\tau \\right\\| \\leqslant \\int_{t-\\delta}^{t+\\delta} \\|x'_k(\\tau)\\| d\\tau \\leqslant 2\\delta L = \\epsilon,\n\\end{equation*}\nIt follows that $x_0(\\cdot),x_1(\\cdot),x_2(\\cdot),\\ldots$ is equicontinuous on $\\mathbb{R}_+$. In addition, \\eqref{upper_0}-\\eqref{upper_2} imply that, for all $t\\geqslant 0$, the sequence $x_0(t),x_1(t),x_2(t),\\ldots$ is bounded. The Arzel\\`a-Ascoli theorem \\cite[Theorem 1 p. 13]{aubin1984differential} implies that there exists a subsequence (again denoted $x_k(\\cdot)$) converging uniformly over compact intervals to a continuous function $x:\\mathbb{R}_+\\rightarrow\\mathbb{R}^n$. \n\nWe next show that $x(\\cdot)$ is a continuous subgradient trajectory of $f$. Let $T\\geqslant 0$. By virtue of \\eqref{upper_0}-\\eqref{upper_2} and local Lipschitz continuity of $f$, the restrictions of $x'_0(\\cdot),x'_1(\\cdot),x'_2(\\cdot),\\ldots$ to $[0,T]$ lie in a ball of the dual space of $L^1([0,T],\\mathbb{R}^n)$, namely $L^{\\infty}([0,T],\\mathbb{R}^n)$. By the Banach-Alaoglu theorem \\cite[Theorem 3 p. 13]{aubin1984differential}, there exists a subsequence (again denoted $x_k'(\\cdot)$) that converges weakly\\textsuperscript{$\\ast$} to a function $y(\\cdot)$ in $L^{\\infty}([0,T],\\mathbb{R}^n)$. Together with $L^{\\infty}([0,T],\\mathbb{R}^n)\\subset L^1([0,T],\\mathbb{R}^n)$, we find that $x_k'(\\cdot)$ converges weakly to $y(\\cdot)$ in $L^1([0,T],\\mathbb{R}^n)$. Since $x_k(t)-x_k(s) = \\int_s^t x_k'(\\tau)d\\tau$ for all $0\\leqslant s \\leqslant t \\leqslant T$, we have $x(t)-x(s) = \\int_s^t y(\\tau)d\\tau$. As a result, $x'(t) = y(t)$ for almost every $t\\in (0,T)$. According to \\cite[Convergence Theorem p. 60]{aubin1984differential}, it follows that $x'(t) \\in -\\partial f(x(t))$ for almost every $t\\in(0,T)$. As $T \\geqslant 0$ was arbitrary, we have $x'(t) \\in -\\partial f(x(t))$ for almost every $t>0$.\n\n% https://www.math.ucdavis.edu/~hunter/pdes/ch6A.pdf\n\nSince $f$ is definable and has bounded continuous subgradient trajectories, by Proposition \\ref{prop:length_continuous} $x(\\cdot)$ has finite length and converges to a critical point $x^*$ of $f$. Let $\\epsilon>0$ and let $m \\in \\mathbb{N}^*$ be the number of critical values of $f$ in $B(\\overline{x(\\mathbb{R}_+)},\\epsilon)$. Let $\\psi$ be a desingularizing function of $f$ on $B(\\overline{x(\\mathbb{R}_+)},\\epsilon)$. Since $f$ is continuous, there exists $\\delta \\in (0,\\epsilon/2)$ such that \n\\begin{equation}\n\\label{eq:f_bound_continuous}\n    f(x) - f(x^*) \\leqslant m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right),~~~ \\forall x\\in B(x^*,\\delta).\n\\end{equation}\n\nLet $t^*\\geqslant 0$ be such that $\\|x(t)-x^*\\| \\leqslant \\delta/2$ for all $t\\geqslant t^*$. By the uniform convergence of $x_k(\\cdot)$, we have that $\\|x_k(t)-x(t)\\|\\leqslant \\delta/2$ for all $t\\in [0,t^*]$ for all $k$ large enough. Hence $\\|x_k(t^*) - x^*\\|\\leqslant \\|x_k(t^*)-x(t^*)\\|+\\|x(t^*)-x^*\\| \\leqslant \\delta/2 + \\delta/2 = \\delta$. If $T_k := \\inf\\{ t \\geqslant t^* : x_k(t) \\notin \\mathring{B}(x^*,\\epsilon)\\}<\\infty$, then\n\\begin{subequations}\n    \\begin{align}\n        2m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right) & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{\\epsilon-\\delta}{2m}\\right) \\label{eq:decrease_continuous_a} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{\\|x_k(T_k)-x^*\\| - \\|x_k(t^*)-x^*\\|}{2m}\\right) \\label{eq:decrease_continuous_b} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{\\|x_k(T_k)-x_k(t^*)\\|}{2m}\\right) \\label{eq:decrease_continuous_c} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{2m}\\int_{t^*}^{T_k}\\|x_k'(t)\\|dt \\right) \\label{eq:decrease_continuous_d} \\\\\n        & \\leqslant f(x_k(t^*))-f(x_k(T_k)) \\label{eq:decrease_continuous_e} \\\\[3mm]\n        & \\leqslant m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right) + f(x^*) - f(x_k(T_k)). \\label{eq:decrease_continuous_f}\n    \\end{align}\n\\end{subequations}\nAbove, \\eqref{eq:decrease_continuous_a} through \\eqref{eq:decrease_continuous_d} rely on the fact that $\\psi^{-1}$ is increasing. \\eqref{eq:decrease_continuous_a} is due to $\\delta < \\epsilon/2$. \\eqref{eq:decrease_continuous_b} holds because $\\|x_k(T_k)-x^*\\|\\geqslant \\epsilon$ by continuity of $x_k(\\cdot)$ and $x_k(t^*) \\in B(x^*,\\delta)$. \\eqref{eq:decrease_continuous_c} is a consequence of the triangular inequality. \\eqref{eq:decrease_continuous_d} holds because $x_k(\\cdot)$ is absolutely continuous. \\eqref{eq:decrease_continuous_e} is due to Proposition \\ref{prop:length_continuous} and the fact that $x_k(t) \\in B(x^*,\\epsilon)$ for all $t \\in [t^*,T_k]$ by continuity of $x_k(\\cdot)$. Finally, \\eqref{eq:decrease_continuous_f} is due to $x_k(t^*)\\in B(x^*,\\delta)$ and \\eqref{eq:f_bound_continuous}. Since $x_k(t) \\in B(\\overline{x(\\mathbb{R}_+)},\\epsilon)$ for all $t\\in [0,T_k]$ and $x_k(T_k)$ belongs to\n\\begin{equation*}\n     X_1 := B(x^*,\\epsilon) \\bigcap \\left\\{ x \\in \\mathbb{R}^n : f(x) \\leqslant f(x^*) - m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right) \\right\\},\n\\end{equation*}\nby Proposition \\ref{prop:length_continuous} and the definition of $\\sigma(\\cdot)$ in \\eqref{eq:sup_trajectory} we have \n    \\begin{align*}\n            \\int_0^{\\infty} \\|x_k'(t)\\|dt & = \\int_{0}^{T_k} \\|x_k'(t)\\|dt + \\int_{T_k}^{\\infty} \\|x_k'(t)\\|dt \\\\\n            & \\leqslant 2m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(x^*,\\epsilon)} f\\right) \\right) + \\max\\{0,\\sigma(X_1)\\}.\n    \\end{align*}\nNote that the inequality still holds if $T_k = \\infty$. By taking the limit, we get\n\\begin{equation*}\n    \\sigma(X_0) ~\\leqslant~ 2m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(x^*,\\epsilon)} f\\right) \\right) + \\max\\{0,\\sigma(X_1)\\}.\n\\end{equation*}\n\nIt now suffices to replace $X_0$ by $X_1$ and repeat the proof starting below \\eqref{eq:finite_time}. A maximizing sequence $\\bar{x}_k(\\cdot)$ corresponding to $\\sigma(X_1)$ converges to a continuous subgradient trajectory $\\bar{x}(\\cdot)$ whose initial point lies in the compact set $X_1$. If $X_1 \\neq \\emptyset$, then the critical value $f(\\lim_{t\\rightarrow\\infty} \\bar{x}(t))$ is less than or equal to $f(x^*) - m\\psi^{-1}(\\epsilon/(4m)) < f(x^*)$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Thus, it is eventually the case that one of the sets $X_0,X_1,\\hdots$ is empty. We conclude that $\\sigma(X_0)<\\infty$ by the above recursive formula. \n\n\\section{Proof of Theorem \\ref{thm:convergence}}\n\\label{sec:Proof of Theorem thm:convergence}\n\n($1\\Longrightarrow 2$) We prove the contrapositive. Assume that there exists $0< T \\leqslant \\infty$ and a differentiable function $x:[0,T)\\rightarrow\\mathbb{R}^n$ such that\n\\begin{equation*}\n    x'(t) = -\\nabla f(x(t)), ~~~ \\forall t \\in (0,T),\n\\end{equation*}\nfor which, for all $c>0$, there exists $t\\in [0,T)$ such that $\\|x(t)\\| > c$. Let $x_0 := x(0)$. Let $\\bar{\\alpha}$ and $c$ be some positive constants. Let $\\tilde{t} \\in (0,T)$ be such that $\\|x(\\tilde{t})\\| \\geqslant c+2$. By Proposition \\ref{prop:tracking} and Remark \\ref{rem:tracking}, there exists $\\tilde{\\alpha}\\in (0,\\bar{\\alpha}]$ such that for all $\\alpha \\in (0,\\tilde{\\alpha}]$, the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by $x_{k+1} = x_k -\\alpha \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$ satisfies $\\|x_k-x(k\\alpha)\\| \\leqslant 1/2$ for all $k\\in \\{0,\\hdots, \\lfloor \\tilde{t}/\\alpha \\rfloor \\}$. We use the notation $\\lfloor t \\rfloor$ to denote the floor of a real number $t$ which is the unique integer such that $\\lfloor t \\rfloor \\leqslant t < \\lfloor t \\rfloor + 1$. If $\\tilde{k} = \\lfloor \\tilde{t}/\\alpha \\rfloor$, then $\\|x_{\\tilde{k}} - x(\\tilde{t})\\| \\leqslant \\|x_{\\tilde{k}}-x(\\tilde{k}\\alpha)\\|+\\|x(\\tilde{k}\\alpha)-x(\\tilde{t})\\| \\leqslant 1/2 + \\tilde{L}| \\tilde{k}\\alpha-\\tilde{t}| \\leqslant 1/2 + \\tilde{L}\\alpha$ where $\\tilde{L}>0$ is a Lipschitz constant of $f$ on the convex hull of $x([\\tilde{t}-\\tilde{\\alpha},\\tilde{t}])$. Let $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$ be the constant sequence equal to $\\min \\{\\tilde{\\alpha}, 1/(2\\tilde{L})\\}$ and consider the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by $x_{k+1} = x_k -\\alpha_k \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$. By the triangular inequality, we have $\\|x_{\\tilde{k}}\\| \\geqslant \\|x(\\tilde{t})\\| - \\|x_{\\tilde{k}}-x(\\tilde{t})\\| \\geqslant c+2 - 1/2 - 1/2>c$ where $\\tilde{k} := \\lfloor \\tilde{t}/\\alpha \\rfloor$.\n\n($2 \\Longrightarrow 3$) Let $X_0$ be a bounded subset of $\\mathbb{R}^n$. We will show that there exists $\\bar{\\alpha}>0$ such that $\\sigma(X_0,\\bar{\\alpha})<\\infty$ where\n\\begin{subequations}\n\\label{eq:sup_discrete_X_0}\n\\begin{align}\n    \\sigma(X_0,\\bar{\\alpha}) := & \\sup\\limits_{\\tiny \\begin{array}{c}x \\in (\\mathbb{R}^n)^{\\mathbb{N}}\\\\ \\alpha \\in (0,\\bar{\\alpha}]^\\mathbb{N} \\end{array}} ~~ \\sum\\limits_{k=0}^{\\infty} \\|x_{k+1}-x_k\\| \\\\\n   & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx_{k+1} = x_k - \\alpha_k \\nabla f(x_k), ~ \\forall k\\in \\mathbb{N},\\\\[2mm] x_0 \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\n\nLet $\\Phi:\\mathbb{R}_+ \\times \\mathbb{R}^n \\rightarrow \\mathbb{R}^n$ be the gradient flow of $f$ defined for all $(t,x_0) \\in \\mathbb{R}_+ \\times \\mathbb{R}^n$ by $\\Phi(t,x_0) := x(t)$ where $x(\\cdot)$ is the unique continuous gradient trajectory of $f$ initialized at $x_0$. Uniqueness follows from the Picard–Lindel{\\\"o}f theorem \\cite[Theorem 3.1 p. 12]{coddington1955theory}. Let $C$ be the set of critical points of $f$ in $\\overline{\\Phi(\\mathbb{R}_+,X_0)}$. If $X_0 \\subset C$, then we have $\\sigma(X_0,\\bar{\\alpha}) \\leqslant 0$ for all $\\bar{\\alpha}>0$ and there is nothing left to prove. Otherwise, since $C$ is closed by \\cite[2.1.5 Proposition p. 29]{clarke1990}, there exists $\\epsilon>0$ be such that $X_0\\setminus \\mathring{B}(C,\\epsilon/6) \\neq \\emptyset$ where $\\mathring{B}(C,\\epsilon/6) := C + \\mathring{B}(0,\\epsilon/6)$. It is safe to assume this from now on.\n\nLet $\\psi$ be a desingularizing function of $f$ on $\\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$, where we use the shorthand $\\Phi_0 := \\Phi(\\mathbb{R}_+,X_0)$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Let $m \\in \\mathbb{N}$ be the number of critical values of $f$ in $B(\\overline{\\Phi}_0,\\epsilon)$. Note that $m \\geqslant 1$ due to Proposition \\ref{prop:length_continuous} and the fact that $X_0 \\neq \\emptyset$. Since $f$ is continuous and $C$ is compact by Lemma \\ref{lemma:uniform_length}, there exists $\\delta \\in (0,\\epsilon/2)$ such that  \n\\begin{equation}\n    \\label{eq:f_bound_discrete}\n    f(x) - \\max_{C} f \\leqslant m~ \\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right),~~~ \\forall x\\in B(C,\\delta).\n\\end{equation}\nNote that $\\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$ is bounded due to Lemma \\ref{lemma:uniform_length}. Let $L>0$ be a Lipschitz constant of $f$ on the convex hull of $\\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$. By Proposition \\ref{prop:length_discrete}, there exists $\\alpha_\\delta \\in (0,\\delta/(5mL)]$ such that, for all $K \\in \\mathbb{N}^*$, $\\alpha_0,\\hdots,\\alpha_{K-1} \\in (0,\\alpha_\\delta]$, and $(x_0,\\hdots,x_K) \\in \\mathring{B}(\\overline{\\Phi}_0,\\epsilon) \\times \\cdots \\times \\mathring{B}(\\overline{\\Phi}_0,\\epsilon) \\times \\mathbb{R}^n$ such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for $k = 0,\\hdots,K-1$, we have\n\\begin{equation}\n\\label{eq:length_discrete_K}\n    \\frac{1}{4m}\\sum_{k=0}^{K-1}\\|x_{k+1}-x_k\\| ~\\leqslant~ \\psi\\left(\\frac{f(x_0)-f(x_{K-1})}{2m}\\right) + \\frac{L}{2}\\max_{k=0,\\hdots,K-1} \\alpha_k.\n\\end{equation}\n\nSince $\\nabla f$ is continuous, its norm attains its infimum $\\nu$ on the nonempty compact set $\\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3)$. \nIt is nonempty because $\\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3) \\supset X_0\\setminus \\mathring{B}(C,\\epsilon/6) \\neq \\emptyset$ and $\\delta < \\epsilon/2$.\nIf $\\nu=0$, then there exists $x^* \\in \\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3)$ such that $\\|\\nabla f(x^*)\\| = 0$. Then $x^*\\in C \\setminus \\mathring{B}(C,\\delta/3)$, which is a contradiction. We thus have $\\nu>0$. Hence we may define $T := 2\\sigma(X_0)/\\nu$ where $\\sigma(X_0)$ is defined in \\eqref{eq:sup_trajectory} and is finite by Lemma \\ref{lemma:uniform_length}. Note that $\\sigma(X_0)>0$ and thus $T>0$ because $X_0\\not\\subset C$. By Proposition \\ref{prop:tracking}, there exists $\\bar{\\alpha} \\in (0,\\alpha_\\delta]$ such that for any feasible point $((x_k)_{k\\in \\mathbb{N}} , (\\alpha_k)_{k\\in\\mathbb{N}})$ of \\eqref{eq:sup_discrete_X_0},\nthe continuous gradient trajectory $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ of $f$ initialized at $x_0$ satisfies\n\\begin{equation}\n\\label{eq:tracking_T}\n   \\forall k\\in \\mathbb{N}^*,~~~ \\alpha_0+\\cdots+\\alpha_{k-1} \\leqslant T ~~~\\Longrightarrow~~~ \\|x_k - x(\\alpha_0+\\cdots+\\alpha_{k-1})\\| \\leqslant \\frac{\\delta}{3}.\n\\end{equation}\nNow suppose that $\\|x'(t)\\| \\geqslant 2\\sigma(X_0)/T$ for all $t\\in (0,T)$. Then we obtain the following contradiction\n\\begin{equation*}\n     \\sigma(X_0) < T \\frac{2\\sigma(X_0)}{T} \\leqslant \\int_0^{T} \\|x'(t)\\|dt \\leqslant \\int_0^{\\infty} \\|x'(t)\\|dt \\leqslant \\sigma(X_0).\n\\end{equation*}\nHence, there exists $t^* \\in (0,T)$ such that $\\|x'(t^*)\\| = \\|\\nabla f(x(t^*))\\| < 2\\sigma(X_0)/T = 2\\sigma(X_0)/(2\\sigma(X_0)/\\nu) = \\nu$. Since  $x(t^*) \\in \\Phi(\\mathbb{R}_+,X_0) \\subset \\overline{\\Phi}_0$ and the infimum of the norm of $\\nabla f$ on $\\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3)$ is equal to $\\nu$, it must be that $x(t^*) \\in \\mathring{B}(C,\\delta/3)$. Hence there exists $x^* \\in C$ such that $\\|x(t^*)-x^*\\|\\leqslant \\delta/3$. \n\nIf $\\sum_{k=0}^{\\infty} \\alpha_k \\leqslant T$, then by \\eqref{eq:tracking_T} we have $x_k \\in B(\\overline{\\Phi}_0,\\delta/3)$ for all $k \\in \\mathbb{N}$. Otherwise, since $\\alpha_k \\leqslant \\bar{\\alpha} \\leqslant \\alpha_\\delta \\leqslant \\delta/(5mL) \\leqslant \\delta/(3L)$ for all $k \\in \\mathbb{N}$, there exists $k^* \\in \\mathbb{N}$ such that $t_{k^*} := \\sum_{k=0}^{k^*-1} \\alpha_k \\in [ t^* - \\delta/(3L) , t^*]$, where $\\sum_{k=0}^{k^*-1} \\alpha_k = 0$ if $k^* = 0$. Thus $\\|x_{k^*}-x^*\\| \\leqslant \\|x_{k^*}-x(t_{k^*})\\| + \\|x(t_{k^*}) - x(t^*)\\| + \\|x(t^*)-x^*\\| \\leqslant \\delta/3 + L | t_{k^*}-t^* | + \\delta/3 \\leqslant \\delta/3+\\delta/3+\\delta/3 = \\delta$.\nIf $K := \\inf\\{ k \\geqslant k^* : x_k \\notin \\mathring{B}(C,\\epsilon)\\}<\\infty$, then\n\\begin{subequations}\n    \\begin{align}\n        2m~\\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right) & = 2m~ \\psi^{-1}\\left(\\frac{1}{8m} \\epsilon - \\frac{L}{2}\\frac{\\epsilon}{5mL} \\right) \\label{eq:decrease_discrete_a} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m}(\\epsilon-\\delta) - \\frac{L\\alpha_\\delta}{2} \\right) \\label{eq:decrease_discrete_b} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m} \\left(\\|x_K-x^*\\|-\\|x_{k^*}-x^*\\|\\right) - \\frac{L\\alpha_\\delta}{2} \\right) \\label{eq:decrease_discrete_c} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m} \\|x_K-x_{k^*}\\| - \\frac{L\\bar{\\alpha}}{2} \\right) \\label{eq:decrease_discrete_d} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m}\\sum_{k=k^*}^{K-1} \\|x_{k+1}-x_k\\| - \\frac{L}{2}\\max_{k=0,\\hdots,K-1} \\alpha_k \\right) \\label{eq:decrease_discrete_e} \\\\\n        & \\leqslant f(x_{k^*}) - f(x_{K-1}) \\label{eq:decrease_discrete_f} \\\\\n        & \\leqslant m~\\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right) + \\max_C f - f(x_{K-1}). \\label{eq:decrease_discrete_g}\n    \\end{align}\n\\end{subequations}\nAbove, the arguments of $\\psi^{-1}$ in \\eqref{eq:decrease_discrete_a} are equal. \\eqref{eq:decrease_discrete_b} through \\eqref{eq:decrease_discrete_e} rely on the fact that $\\psi^{-1}$ is an increasing function. \\eqref{eq:decrease_discrete_b} is due to $\\delta<\\epsilon/2$. \\eqref{eq:decrease_discrete_c} holds because $x_K \\notin \\mathring{B}(C,\\epsilon)$, $x^* \\in C$, and $x_{k^*} \\in B(x^*,\\delta)$. \\eqref{eq:decrease_discrete_d} and \\eqref{eq:decrease_discrete_e} are consequences of the triangular inequality. \\eqref{eq:decrease_discrete_f} is due to the length formula \\eqref{eq:length_discrete_K} and the fact that $x_{k^*},\\hdots,x_{K-1} \\in \\mathring{B}(C,\\epsilon) \\subset \\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$. Finally, \\eqref{eq:decrease_discrete_g} is due to $x_{k^*}\\in B(C,\\delta)$ and \\eqref{eq:f_bound_discrete}. We remark that $K\\geqslant k^*+2$ since $\\|x_{k^*+1}-x^*\\| \\leqslant \\|x_{k^*+1}-x_{k^*}\\| +\\|x_{k^*}-x^*\\| \\leqslant \\alpha_{k^*}\\|\\nabla f(x_{k^*})\\|+\\delta \\leqslant (\\delta/(5mL))L+\\delta < \\epsilon$. Since $x_0,\\hdots,x_{K-2} \\in \\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$ and $x_{K-1}$ belongs to \n\\begin{equation*}\n    X_1 := B(C,\\epsilon) \\bigcap \\left\\{ x \\in \\mathbb{R}^n : f(x) \\leqslant \\max_{C} f - m~\\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right) \\right\\},    \n\\end{equation*}\nby the length formula \\eqref{eq:length_discrete_K} and the definition of $\\sigma(\\cdot,\\cdot)$ in \\eqref{eq:sup_discrete_X_0} we have\n    \\begin{align*}\n            \\sum_{k=0}^\\infty \\|x_{k+1}-x_k\\| & = \\sum_{k=0}^{K-2} \\|x_{k+1}-x_k\\| + \\sum_{k=K-1}^\\infty \\|x_{k+1}-x_k\\| \\\\\n            & \\leqslant 4m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(\\overline{\\Phi}_0,\\epsilon)} f\\right) \\right) + 2mL\\bar{\\alpha} + \\max\\{0,\\sigma(X_1,\\bar{\\alpha})\\}.\n    \\end{align*}\nNote that the inequality still holds if $K = \\infty$ or $\\sum_{k=0}^{\\infty} \\alpha_k \\leqslant T$. Hence\n\\begin{equation*}\n    \\sigma(X_0,\\bar{\\alpha}) ~\\leqslant~ 4m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(\\overline{\\Phi}_0,\\epsilon)} f\\right) \\right) + 2mL\\bar{\\alpha} + \\max\\{0,\\sigma(X_1,\\bar{\\alpha})\\}.\n\\end{equation*}\n\nIt now suffices to replace $X_0$ by $X_1$ and repeat the proof starting below \\eqref{eq:sup_discrete_X_0}. Since $f(\\Phi(t,x_1)) \\leqslant f(\\Phi(0,x_1)) \\leqslant \\max_{C} f - m\\psi^{-1}(\\epsilon/(40m))< \\max_{C} f$ for all $t\\geqslant 0$ and $x_1\\in X_1$, the maximal critical value of $f$ in $\\overline{\\Phi(\\mathbb{R}_+,X_1)}$ is less than the maximal critical value of $f$ in $\\overline{\\Phi(\\mathbb{R}_+,X_0)}$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Thus, it is eventually the case that one of the sets $X_0,X_1,\\hdots$ is empty. In order to conclude, one simply needs to choose an upper bound on the step sizes $\\hat{\\alpha}$ corresponding to $X_1$ that is less than or equal to the upper bound $\\bar{\\alpha}$ used for $X_0$. $\\sigma(X_0,\\cdot)$ is finite when evaluated at the last upper bound thus obtained. Indeed, the above recursive formula still holds if we replace $\\bar{\\alpha}$ by any $\\alpha \\in (0,\\bar{\\alpha}]$. In particular, we may take $\\alpha := \\hat{\\alpha}$.\n\n($3 \\Longrightarrow 1$) Obvious.\\\\\n\n","start":72234,"end":94199}]},"nodes":[{"id":"S001","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"指标、范数与固定结构","source_units":[{"raw_tex":"Let $\\mathbb{N} := \\{0,1,2,\\hdots\\}$ and let $\\|\\cdot\\|$ be the induced norm of an inner product $\\langle \\cdot, \\cdot\\rangle$ on $\\mathbb{R}^n$. Throughout this paper, we fix an arbitrary o-minimal structure on the real field (see Definition \\ref{def:o-minimal}), and say that sets or functions are definable if they are definable in this structure. For readers not familiar with o-minimal structures, note that seemingly all objective functions used in practice are definable in some o-minimal structure. Low-rank matrix recovery (for e.g., matrix factorization, matrix completion, and matrix sensing) and deep neural networks are no exception \\cite[Section 6.2]{bolte2020conservative}.\n","start":4045,"end":4734}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"闭球与球面","source_units":[{"raw_tex":"We begin by stating some standard definitions. Let $B(a,r)$ and $S(a,r)$ respectively denote the closed ball and the sphere of center $a\\in \\mathbb{R}^n$ and radius $r \\geqslant 0$.\n\n","start":26428,"end":26611}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":["def:lipschitz"],"declared_external":false,"source_only":false,"title":"局部利普希茨连续","source_units":[{"raw_tex":"\\begin{definition}\n\\label{def:lipschitz}\nA function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}^m$ is locally Lipschitz if for all $a \\in \\mathbb{R}^n$, there exist $r>0$ and $L>0$ such that \n\\begin{equation*}\n        \\forall x,y \\in B(a,r), ~~~ \\|f(x)-f(y)\\| \\leqslant L \\|x-y\\|.\n    \\end{equation*}\n\\end{definition}\n\n","start":26611,"end":26923}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["def:Clarke"],"declared_external":false,"source_only":false,"title":"克拉克次微分","source_units":[{"raw_tex":"\\begin{definition}\n    \\label{def:Clarke}\n    \\cite[Chapter 2]{clarke1990}\n    Let $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz function. The Clarke subdifferential is the set-valued mapping $\\partial f:\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^n$ defined for all $x \\in \\mathbb{R}^n$ by\n    $\\partial f(x) := \\{ s \\in \\mathbb{R}^n : f^\\circ(x,h) \\geqslant \\langle s , h \\rangle, ~ \\forall h\\in \\mathbb{R}^n \\}$ where\n\\begin{equation*}\n    f^\\circ(x,h) := \\limsup_{\\tiny\\begin{array}{c} y\\rightarrow x \\\\\n    t \\searrow 0\n    \\end{array}\n    } \\frac{f(y+th)-f(y)}{t}.\n\\end{equation*}\n\\end{definition}\n\n","start":26923,"end":27541}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"临界点与光滑特化","source_units":[{"raw_tex":"We say that $x \\in \\mathbb{R}^n$ is critical if $0 \\in \\partial f(x)$. If $f$ is continuously differentiable in a neighborhood of $x \\in \\mathbb{R}^n$, then $\\partial f(x) = \\{\\nabla f(x)\\}$ \\cite[2.2.4 Proposition p. 33]{clarke1990}.\n","start":27541,"end":27776}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["def:absolute_continuity"],"declared_external":false,"source_only":false,"title":"绝对连续性","source_units":[{"raw_tex":"\\begin{definition}\n    \\label{def:absolute_continuity} \n    \\cite[Definition 1 p. 12]{aubin1984differential}\n    Given some real numbers $a\\leqslant b$, a function $x:[a,b]\\rightarrow\\mathbb{R}^n$ is absolutely continuous if for all $\\epsilon>0$, there exists $\\delta>0$ such that, for any finite collection of disjoint subintervals $[a_1,b_1],\\hdots,[a_m,b_m]$ of $[a,b]$ such that $\\sum_{i=1}^m b_i-a_i \\leqslant \\delta$, we have $\\sum_{i=1}^m \\|x(b_i) - x(a_i)\\| \\leqslant \\epsilon$.\n\\end{definition}\n\n","start":28557,"end":29062}]},{"id":"S007","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原文第229行引用 nielsen1997introduction, Theorem 20.8；亦属声明的绝对连续性标准分析前提。"],"source_only":false,"title":"绝对连续函数的积分表示","source_units":[{"raw_tex":"By virtue of \\cite[Theorem 20.8]{nielsen1997introduction}, $x:[a,b]\\rightarrow\\mathbb{R}^n$ is absolutely continuous if and only if it is differentiable almost everywhere on $(a,b)$, its derivative $x'(\\cdot)$ is Lebesgue integrable, and $x(t) - x(a) = \\int_a^t x'(\\tau)d\\tau$ for all $t\\in [a,b]$. Given a noncompact interval $I$ of $\\mathbb{R}$, $x:I\\rightarrow \\mathbb{R}^n$ is absolutely continuous if it is absolutely continuous on all compact subintervals of $I$. Given an interval $I$ of $\\mathbb{R}$, let $\\mathcal{A}(I,\\mathbb{R}^n)$ be the set of absolutely continuous functions defined from $I$ to $\\mathbb{R}^n$.\n\n","start":29062,"end":29688}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":["def:trajectory"],"declared_external":false,"source_only":false,"title":"极大连续次梯度轨迹","source_units":[{"raw_tex":"\\begin{definition}\n\\label{def:trajectory}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz function. Let $(S,\\leqslant)$ be the set of absolutely continuous functions $x:[0,T)\\rightarrow\\mathbb{R}^n$ where $0<T\\leqslant \\infty$ such that  \n\\begin{equation*}\n    x'(t) \\in -\\partial f(x(t)), ~~~\\mathrm{for~a.e.}~t\\in (0,T),\n\\end{equation*}\nequipped with the partial order $\\leqslant$ defined by $x_1:[0,T_1)\\rightarrow\\mathbb{R}^n$ is less than or equal to $x_2:[0,T_2)\\rightarrow\\mathbb{R}^n$ if and only if $T_1\\leqslant T_2$ and $x_1(t) = x_2(t)$ for all $t\\in [0,T_1)$. \nWe refer to maximal elements of $(S,\\leqslant)$ as continuous subgradient trajectories.\n\\end{definition}\n\n","start":29688,"end":30387}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"全局轨迹与离散轨迹","source_units":[{"raw_tex":"We say that a continuous subgradient trajectory is globally defined if it is of the form $x:[0,T)\\rightarrow\\mathbb{R}^n$ where $T=\\infty$. We refer to discrete subgradient trajectories as sequences $(x_k)_{k\\in \\mathbb{N}}$ in $\\mathbb{R}^n$ such that $x_{k+1} \\in x_k -\\alpha_k \\partial f(x_k)$ for all $k\\in \\mathbb{N}$ for some $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots>0$. When $f$ is continuously differentiable, we refer to subgradient trajectories as  gradient trajectories.\n\n","start":30387,"end":30864}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":["def:bounded_trajectory"],"declared_external":false,"source_only":false,"title":"连续轨迹有界性","source_units":[{"raw_tex":"\\begin{definition}\n\\label{def:bounded_trajectory}\nA continuous subgradient trajectory $x: [0,T) \\rightarrow \\mathbb{R}^n$ where $0< T\\leqslant \\infty$ is bounded if there exists $c>0$ such that $\\|x(t)\\| \\leqslant c$ for all $t\\in [0,T)$.\n\\end{definition}\n","start":30864,"end":31120}]},{"id":"S011","semantic_type":"proposition","kind":"proposition","labels":["rem:tracking"],"declared_external":false,"source_only":false,"title":"单初值局部跟踪扩展","source_units":[{"raw_tex":"\\begin{remark}\n\\label{rem:tracking}\nIf $X_0$ is a singleton $\\{x_0\\}$ and $x:[0,\\hat{T})\\rightarrow \\mathbb{R}^n$ is a gradient trajectory initialized at $x_0$, then one may remove the assumption that $f$ is lower bounded in Proposition \\ref{prop:tracking} and instead require that $T \\in (0,\\hat{T})$. In that case, $\\sigma_T(X_0)=\\int_0^T \\|x'(t)\\|dt < \\infty$ since $\\nabla f$ is Lipschitz continuous on the bounded set $x([0,T])$.\n\\end{remark}","start":44646,"end":45093}]},{"id":"S012","semantic_type":"proof","kind":"proof","labels":["rem:tracking"],"declared_external":false,"source_only":false,"title":"局部跟踪扩展的作者论证","source_units":[{"raw_tex":"If $X_0$ is a singleton $\\{x_0\\}$ and $x:[0,\\hat{T})\\rightarrow \\mathbb{R}^n$ is a gradient trajectory initialized at $x_0$, then one may remove the assumption that $f$ is lower bounded in Proposition \\ref{prop:tracking} and instead require that $T \\in (0,\\hat{T})$. In that case, $\\sigma_T(X_0)=\\int_0^T \\|x'(t)\\|dt < \\infty$ since $\\nabla f$ is Lipschitz continuous on the bounded set $x([0,T])$.\n","start":44682,"end":45081}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["def:o-minimal"],"declared_external":false,"source_only":false,"title":"极小序结构公理","source_units":[{"raw_tex":"\\begin{definition}\n\\label{def:o-minimal}\n\\cite[Definition p. 503-506]{van1996geometric}\nAn o-minimal structure on the real field is a sequence $S = (S_k)_{k \\in \\mathbb{N}}$ such that for all $k \\in \\mathbb{N}$:\n\\begin{enumerate}\n    \\item $S_k$ is a boolean algebra of subsets of $\\mathbb{R}^k$, with $\\mathbb{R}^k \\in S_k$;\n    \\item $S_k$ contains the diagonal $\\{(x_1,\\hdots,x_k) \\in \\mathbb{R}^k : x_i = x_j\\}$ for $1\\leqslant i<j \\leqslant k$;\n    \\item If $A\\in S_k$, then $A\\times \\mathbb{R}$ and $\\mathbb{R}\\times A$ belong to $S_{k+1}$;\n    \\item If $A \\in S_{k+1}$ and $\\pi:\\mathbb{R}^{k+1}\\rightarrow\\mathbb{R}^k$ is the projection onto the first $k$ coordinates, then $\\pi(A) \\in S_k$;\n    \\item $S_3$ contains the graphs of addition and multiplication;\n    \\item $S_1$ consists exactly of the finite unions of open intervals and singletons. \n\\end{enumerate}\n\\end{definition}\n\n","start":46181,"end":47071}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"可定义集合与函数","source_units":[{"raw_tex":"A subset $A$ of $\\mathbb{R}^n$ is definable in an o-minimal structure $(S_k)_{k\\in\\mathbb{N}}$ if $A \\in S_k$. A function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ is definable in an o-minimal structure if its graph, that is to say $\\{ (x,t) \\in \\mathbb{R}^{n+1} : f(x)=t \\}$, is definable in that structure.\n","start":47071,"end":47376}]},{"id":"S015","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"距离、临界值与逆映射","source_units":[{"raw_tex":"We next state the uniform Kurdyka-\\L{}ojasiewicz inequality. Given a subset of $S$ of $\\mathbb{R}^n$, let $\\mathring{S}$ and $\\overline{S}$ denote the interior and closure of $S$ in $\\mathbb{R}^n$ respectively. A function $\\psi:S \\rightarrow S$ is a homeomorphism if it is a continuous bijection and the inverse function $\\psi^{-1}$ is continuous. $\\psi:S \\rightarrow S$ is a diffeomorphism if $\\mathring{S} \\neq \\emptyset$, $\\psi$ is a homeomorphism, and both $\\psi$ and $\\psi^{-1}$ are continuously differentiable on $\\mathring{S}$. Given $x\\in \\mathbb{R}^n$, consider the distance of $x$ to $S$ defined by $d(x,S) := \\inf \\{ \\|x-y\\| : y \\in S \\}$. Given a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$, a real number $v$ is critical value of $f$ in $S$ if there exists $x \\in S$ such that $v = f(x)$ and $0\\in \\partial f(x)$. Given a set-valued mapping $F:\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^m$ and $y \\in \\mathbb{R}^m$, let $F^{-1}(y) := \\{ x \\in \\mathbb{R}^n : F(x) \\ni y \\}$.\n","start":48364,"end":49369}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"去奇异化函数","source_units":[{"raw_tex":"We say that $\\psi$ in Proposition \\ref{prop:KL} is a desingularizing function of $f$ on $X$. In order to prove Proposition \\ref{prop:length_continuous}, we recall the following result.\n","start":52714,"end":52899}]},{"id":"S017","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"有界轨迹的有限长度","source_units":[{"raw_tex":"An immediate consequence of Propositions \\ref{prop:length_continuous} and \\ref{prop:length_discrete} is that bounded continuous subgradient trajectories and bounded discrete gradient trajectories have finite length, as is well known. An easy fact that we will not prove here is that continuous subgradient trajectories of finite length converge to critical points of the objective function under the sole assumption that it is locally Lipschitz. The same holds for discrete subgradient trajectories if the step sizes are not summable.\n","start":64646,"end":65181}]},{"id":"S018","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明的允许前提：原文第558行明确不予证明的有限长度轨迹临界收敛事实。"],"source_only":false,"title":"有限长度轨迹的临界收敛","source_units":[{"raw_tex":"An immediate consequence of Propositions \\ref{prop:length_continuous} and \\ref{prop:length_discrete} is that bounded continuous subgradient trajectories and bounded discrete gradient trajectories have finite length, as is well known. An easy fact that we will not prove here is that continuous subgradient trajectories of finite length converge to critical points of the objective function under the sole assumption that it is locally Lipschitz. The same holds for discrete subgradient trajectories if the step sizes are not summable.\n","start":64646,"end":65181}]},{"id":"S019","semantic_type":"theorem","kind":"theorem","labels":["eq:gradient_method","thm:convergence"],"declared_external":false,"source_only":false,"title":"连续有界与离散统一长度等价","source_units":[{"raw_tex":"\\begin{theorem}\n\\label{thm:convergence}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a definable differentiable function whose gradient is locally Lipschitz. The following statements are equivalent:\n\\begin{enumerate}\n    \\item For all $x_0 \\in \\mathbb{R}^n$, there exist $\\bar{\\alpha}>0$ and $c>0$ such that for all $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$, the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by \n    \\begin{equation}\n    \\label{eq:gradient_method}\n        x_{k+1} = x_k - \\alpha_k \\nabla f(x_k), ~~~\\forall k \\in \\mathbb{N},\n    \\end{equation}\n    satisfies $\\|x_k\\|\\leqslant c$ for all $k \\in \\mathbb{N}$.\n    \\item For all $0< T \\leqslant \\infty$ and any differentiable function $x:[0,T)\\rightarrow\\mathbb{R}^n$ such that\n    \\begin{equation*}\n        x'(t) = -\\nabla f(x(t)), ~~~ \\forall t \\in (0,T),\n    \\end{equation*}\n    there exists $c>0$ such that $\\|x(t)\\| \\leqslant c$ for all $t\\in [0,T)$.\n    \\item For any bounded subset $X_0$ of $\\mathbb{R}^n$, there exist $\\bar{\\alpha}>0$ and $c>0$ such that, for all $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$ and $x_0 \\in X_0$, the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by \\eqref{eq:gradient_method} obeys \n    \\begin{equation*}\n        \\sum_{k=0}^{\\infty} \\|x_{k+1}-x_k\\| \\leqslant c.\n    \\end{equation*}\n\\end{enumerate}\n\\end{theorem}","start":4735,"end":6096}]},{"id":"S020","semantic_type":"assumption","kind":"assumption","labels":["thm:convergence"],"declared_external":false,"source_only":false,"title":"主定理的目标函数假设","source_units":[{"raw_tex":"Let $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a definable differentiable function whose gradient is locally Lipschitz. The following statements are equivalent:\n","start":4775,"end":4936}]},{"id":"S021","semantic_type":"algorithm","kind":"algorithm","labels":["eq:gradient_method","thm:convergence"],"declared_external":false,"source_only":false,"title":"变步长梯度迭代","source_units":[{"raw_tex":"    \\item For all $x_0 \\in \\mathbb{R}^n$, there exist $\\bar{\\alpha}>0$ and $c>0$ such that for all $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$, the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by \n    \\begin{equation}\n    \\label{eq:gradient_method}\n        x_{k+1} = x_k - \\alpha_k \\nabla f(x_k), ~~~\\forall k \\in \\mathbb{N},\n    \\end{equation}\n    satisfies $\\|x_k\\|\\leqslant c$ for all $k \\in \\mathbb{N}$.\n","start":4954,"end":5385}]},{"id":"S022","semantic_type":"lemma","kind":"lemma","labels":["lemma:uniform_length"],"declared_external":false,"source_only":false,"title":"连续轨迹的统一长度引理","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lemma:uniform_length}\nIf a locally Lipschitz definable function has bounded continuous subgradient trajectories, then continuous subgradient trajectories initialized in a bounded set have uniformly bounded lengths.\n\\end{lemma}","start":12812,"end":13059}]},{"id":"S023","semantic_type":"proposition","kind":"proposition","labels":["prop:existence_trajectory"],"declared_external":false,"source_only":false,"title":"极大次梯度轨迹存在","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:existence_trajectory}\nLocally Lipschitz functions have at least one continuous subgradient trajectory for every initial point.\n\\end{proposition}","start":31369,"end":31545}]},{"id":"S024","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"轨迹存在性的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz function and let $x_0\\in\\mathbb{R}^n$. Since $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d) p. 29]{clarke1990} with nonempty, compact and convex values \\cite[2.1.2 Proposition (a) p. 27]{clarke1990}. By virtue of \\cite[Theorem 3 p. 98]{aubin1984differential}, there exist $\\epsilon>0$ and an absolutely continuous function $x:[0,\\epsilon)\\rightarrow\\mathbb{R}^n$ such that  \n\\begin{equation*}\n    x'(t) \\in -\\partial f(x(t)), ~\\mathrm{for~a.e.}~ t\\in (0,\\epsilon),~~~ x(0) = x_0.\n\\end{equation*}\nLet $S_{x_0}$ be the set of functions in $S$ from Definition \\ref{def:trajectory} such that $x(0) = x_0$. Then $S_{x_0} \\neq \\emptyset$. Let $P$ be a nonempty totally ordered subset of $S_{x_0}$. Let $0<T \\leqslant\\infty$ be the supremum of the endpoints of the domains of all functions in $P$. Consider the function $x:[0,T)\\rightarrow\\mathbb{R}^n$ such that for all $\\bar{x}:[0,\\overline{T})\\rightarrow \\mathbb{R}^n$ in $P$ we have $x(t) := \\bar{x}(t)$ for all $t\\in [0,\\overline{T})$. It is well defined and it is an upper bound of $P$ in $S_{x_0}$. By virtue of Zorn's lemma \\cite[Corollary 2.5 p. 884]{lang2012algebra}, $S_{x_0}$ contains a maximal element. \\end{proof}","start":31546,"end":32866}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"由次微分性质获得局部解","source_units":[{"raw_tex":"\\begin{proof}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz function and let $x_0\\in\\mathbb{R}^n$. Since $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d) p. 29]{clarke1990} with nonempty, compact and convex values \\cite[2.1.2 Proposition (a) p. 27]{clarke1990}. By virtue of \\cite[Theorem 3 p. 98]{aubin1984differential}, there exist $\\epsilon>0$ and an absolutely continuous function $x:[0,\\epsilon)\\rightarrow\\mathbb{R}^n$ such that  \n\\begin{equation*}\n    x'(t) \\in -\\partial f(x(t)), ~\\mathrm{for~a.e.}~ t\\in (0,\\epsilon),~~~ x(0) = x_0.\n\\end{equation*}\n","start":31546,"end":32192}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"按延拓偏序取极大元","source_units":[{"raw_tex":"Let $S_{x_0}$ be the set of functions in $S$ from Definition \\ref{def:trajectory} such that $x(0) = x_0$. Then $S_{x_0} \\neq \\emptyset$. Let $P$ be a nonempty totally ordered subset of $S_{x_0}$. Let $0<T \\leqslant\\infty$ be the supremum of the endpoints of the domains of all functions in $P$. Consider the function $x:[0,T)\\rightarrow\\mathbb{R}^n$ such that for all $\\bar{x}:[0,\\overline{T})\\rightarrow \\mathbb{R}^n$ in $P$ we have $x(t) := \\bar{x}(t)$ for all $t\\in [0,\\overline{T})$. It is well defined and it is an upper bound of $P$ in $S_{x_0}$. By virtue of Zorn's lemma \\cite[Corollary 2.5 p. 884]{lang2012algebra}, $S_{x_0}$ contains a maximal element. \\end{proof}","start":32192,"end":32866}]},{"id":"S027","semantic_type":"proposition","kind":"proposition","labels":["prop:bounded_global"],"declared_external":false,"source_only":false,"title":"有界极大轨迹全局存在","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:bounded_global}\nBounded continuous subgradient trajectories are globally defined.\n\\end{proposition}","start":32868,"end":32999}]},{"id":"S028","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"有界轨迹全局性的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $x:[0,T)\\rightarrow\\mathbb{R}^n$ be a bounded continuous subgradient trajectory of $f$ where $0<T \\leqslant \\infty$. We reason by contradiction and assume that $T<\\infty$. By Definition \\ref{def:bounded_trajectory}, there exists $c>0$ such that $\\|x(t)\\| \\leqslant c$ for all $t\\in [0,T)$. Consider a sequence $t_0,t_1,t_2,\\hdots \\in [0,T)$ converging to $T$. For all $k\\in \\mathbb{N}$, let $x_k:[0,T]\\rightarrow\\mathbb{R}^n$ be defined by $x_k(t) := x(t)$ for all $t\\in [0,t_k]$ and $x_k(t) = 0$ for all $t\\in (t_k,T]$. By construction, $x'_k$ converges pointwise almost everywhere to $x'$ on $(0,T)$. Let $L>0$ be a Lipschitz constant of $f$ on $B(0,c)$. By \\cite[2.1.2 Proposition (a) p. 27]{clarke1990}, for all $x\\in B(0,c)$ and $s\\in \\partial f(x)$, we have $\\|s\\| \\leqslant L$. For almost every $t\\in (0,T)$, we thus have $\\|x_k'(t)\\| \\leqslant \\|x'(t)\\| \\leqslant L$. By Lebesgue's dominated convergence theorem, $\\int_0^{t_k} x'(t)dt = \\int_0^T x'_k(t)dt \\rightarrow \\int_0^T x'(t)dt$. Since $x(\\cdot)$ is absolutely continuous on $[0,t_k]$, we have \n$x(t_k) - x(0) = \\int_0^{t_k} x'(t)dt$. Thus $x(\\cdot)$ can be extended to an absolutely continuous function on $[0,T]$ where \n$x(T) := x(0) + \\int_0^T x'(t)dt$. According to Proposition \\ref{prop:existence_trajectory}, $x(\\cdot)$ can further be extended to an absolutely continuous function on $[0,T+\\epsilon)$ for some $\\epsilon>0$ while satisfying the differential inclusion. This contradicts the maximality of $x(\\cdot)$. \n\\end{proof}","start":33000,"end":34516}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"有限终点处的积分极限","source_units":[{"raw_tex":"\\begin{proof}\nLet $x:[0,T)\\rightarrow\\mathbb{R}^n$ be a bounded continuous subgradient trajectory of $f$ where $0<T \\leqslant \\infty$. We reason by contradiction and assume that $T<\\infty$. By Definition \\ref{def:bounded_trajectory}, there exists $c>0$ such that $\\|x(t)\\| \\leqslant c$ for all $t\\in [0,T)$. Consider a sequence $t_0,t_1,t_2,\\hdots \\in [0,T)$ converging to $T$. For all $k\\in \\mathbb{N}$, let $x_k:[0,T]\\rightarrow\\mathbb{R}^n$ be defined by $x_k(t) := x(t)$ for all $t\\in [0,t_k]$ and $x_k(t) = 0$ for all $t\\in (t_k,T]$. By construction, $x'_k$ converges pointwise almost everywhere to $x'$ on $(0,T)$. Let $L>0$ be a Lipschitz constant of $f$ on $B(0,c)$. By \\cite[2.1.2 Proposition (a) p. 27]{clarke1990}, for all $x\\in B(0,c)$ and $s\\in \\partial f(x)$, we have $\\|s\\| \\leqslant L$. For almost every $t\\in (0,T)$, we thus have $\\|x_k'(t)\\| \\leqslant \\|x'(t)\\| \\leqslant L$. By Lebesgue's dominated convergence theorem, $\\int_0^{t_k} x'(t)dt = \\int_0^T x'_k(t)dt \\rightarrow \\int_0^T x'(t)dt$. Since $x(\\cdot)$ is absolutely continuous on $[0,t_k]$, we have \n$x(t_k) - x(0) = \\int_0^{t_k} x'(t)dt$. Thus $x(\\cdot)$ can be extended to an absolutely continuous function on $[0,T]$ where \n","start":33000,"end":34205}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"从终点继续解而矛盾","source_units":[{"raw_tex":"$x(T) := x(0) + \\int_0^T x'(t)dt$. According to Proposition \\ref{prop:existence_trajectory}, $x(\\cdot)$ can further be extended to an absolutely continuous function on $[0,T+\\epsilon)$ for some $\\epsilon>0$ while satisfying the differential inclusion. This contradicts the maximality of $x(\\cdot)$. \n\\end{proof}","start":34205,"end":34516}]},{"id":"S031","semantic_type":"proposition","kind":"proposition","labels":["eq:tracking","prop:tracking"],"declared_external":false,"source_only":false,"title":"有界初值集的有限时跟踪","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:tracking}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a lower bounded differentiable function with a locally Lipschitz gradient. Let $X_0$ be a bounded subset of $\\mathbb{R}^n$ and let $T>0$. For all $\\epsilon>0$, there exists $\\bar{\\alpha}>0$ such that for all $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$ and for all sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ such that $x_0 \\in X_0$ and $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$, the continuous gradient trajectory $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ of $f$ initialized at $x_0$ satisfies\n\\begin{equation}\n\\label{eq:tracking}\n   \\forall k\\in \\mathbb{N}^*,~~~ \\alpha_0+\\cdots+\\alpha_{k-1} \\leqslant T ~~~\\Longrightarrow~~~ \\|x_k - x(\\alpha_0+\\cdots+\\alpha_{k-1})\\| \\leqslant \\epsilon.\n\\end{equation}\n\\end{proposition}","start":37589,"end":38440}]},{"id":"S032","semantic_type":"proof","kind":"proof","labels":["eq:global_a","eq:global_b","eq:global_c","eq:global_d","eq:global_e","eq:global_f","eq:global_g","eq:length_finite_smooth_a","eq:length_finite_smooth_b","eq:length_finite_smooth_c","eq:length_finite_smooth_d","eq:length_finite_smooth_e","eq:local_error","eq:sup_trajectory_T_smooth","local_a","local_b","local_c","local_d","local_e","local_f"],"declared_external":false,"source_only":false,"title":"有限时跟踪的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nWithout loss of generality, we may assume that $X_0 \\neq \\emptyset$. Let $L>0$ and $M>0$ respectively denote Lipschitz constants of $f$ and $\\nabla f$ with respect to $\\|\\cdot\\|$ on the convex hull of $B(X_0,\\sigma_T(X_0)+1):= X_0 + B(0,\\sigma_T(X_0)+1)$ where \n\\begin{subequations}\n\\label{eq:sup_trajectory_T_smooth}\n\\begin{align}\n    \\sigma_T(X_0) := & \\sup\\limits_{x \\in \\mathcal{C}^1(\\mathbb{R}_+,\\mathbb{R}^n)} ~~ \\int_0^T \\|x'(t)\\|dt \\\\\n  & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx'(t) = - \\nabla f(x(t)), ~\\forall t > 0,\\\\[3mm] x(0) \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\nAbove, $\\mathcal{C}^1(\\mathbb{R}_+,\\mathbb{R}^n)$ is the set of continuous functions on $[0,\\infty )$ that are continuously differentiable on $(0,\\infty)$. Note that $\\sigma_T(X_0)>-\\infty$ since $f$ is lower bounded and differentiable with a locally Lipschitz gradient. For every $x_0\\in X_0$, there hence exists a unique globally defined continuous gradient trajectory of $f$ initialized at $x_0$ \\cite[Theorem 17.1.1]{attouch2014variational}. Also, $\\sigma_T(X_0)<\\infty$ since for any feasible point $x(\\cdot)$ of \\eqref{eq:sup_trajectory_T_smooth}, we have \n\\begin{subequations}\n\\begin{align}\n    \\int_0^T \\|x'(t)\\|dt & \\leqslant \\sqrt{T} \\sqrt{ \\int_0^T \\|x'(t)\\|^2 dt} \\label{eq:length_finite_smooth_a} \\\\\n    & = \\sqrt{T} \\sqrt{ -\\int_0^T \\langle \\nabla f(x(t)), x'(t) \\rangle dt} \\label{eq:length_finite_smooth_b} \\\\\n    & = \\sqrt{T} \\sqrt{ -\\int_0^T (f \\circ x)'(t) dt} \\label{eq:length_finite_smooth_c} \\\\\n    & = \\sqrt{T} \\sqrt{ f(x(0)) - f(x(T))} \\label{eq:length_finite_smooth_d} \\\\\n    & \\leqslant \\sqrt{T\\left(\\sup_{X_0} f - \\inf_{\\mathbb{R}^n} f\\right)} < \\infty. \\label{eq:length_finite_smooth_e}\n\\end{align}\n\\end{subequations}\nIndeed, \\eqref{eq:length_finite_smooth_a} is due to the Cauchy-Schwarz inequality. \\eqref{eq:length_finite_smooth_e} holds because $X_0$ is bounded and $f$ is lower bounded.\n\nIt is easy to check that $L$ and $ML$ are respectively Lipschitz and gradient Lipschitz constants on $[0,T]$ of all continuous gradient trajectories of $f$ initialized in $X_0$. Indeed, let $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ be a continuous gradient trajectory initialized in $X_0$. Since $x(t) \\in B(X_0,\\sigma_T(X_0))$ for all $t\\in [0,T]$, we have $\\|x'(t)\\| = \\|\\nabla f(x(t))\\| \\leqslant L$. By the mean value theorem \\cite[5.19 Theorem]{rudin1964principles}, for all $s,t \\in [0,T]$ we have $\\|x'(t)-x'(s)\\| = \\|\\nabla f(x(t)) - \\nabla f(x(s))\\| \\leqslant M \\|x(t)-x(s)\\| \\leqslant M L |t-s|$. As a byproduct, we get the Taylor bound\n\\begin{subequations}\n    \\begin{align}\n    \\|x(t)-x(s) - x'(s)(t-s)\\| & = \\left\\| \\int_s^t (x'(\\tau) - x'(s)) d\\tau \\right\\| \\\\\n    & \\leqslant \\left| \\int_s^t \\| x'(\\tau) - x'(s) \\| d\\tau \\right| \\\\\n    & \\leqslant \\left|\\int_s^t ML | \\tau - s | d\\tau \\right| \\\\\n    & = \\frac{ML}{2}(t-s)^2. \\label{eq:local_error}\n    \\end{align}\n\\end{subequations}\n\nLet $\\epsilon \\in (0,1)$ and let $\\bar{\\alpha} := 2\\epsilon e^{-MT}/(LMT)$. Consider $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$ and a sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$ and $x_0 \\in X_0$. Let $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ be the unique continuous gradient trajectory of $f$ initialized at $x_0$. Let $t_0:=0$ and $t_k := \\alpha_0+\\cdots+\\alpha_{k-1}$ when $k\\in \\mathbb{N}^*$ so that $\\|x_0 - x(t_0)\\| = 0 \\leqslant \\epsilon$. Assume that \\eqref{eq:tracking} holds up to some index $K$. For $k=0,\\hdots,K$, we have\n\\begin{subequations}\n    \\begin{align}\n        \\|x_{k+1} - x(t_{k+1})\\| = &  ~\\| x_k - \\alpha_k \\nabla f(x_k) - x(t_{k+1}) \\| \\label{local_a} \\\\[2mm]\n        \\leqslant  & ~ \\| x_k - \\alpha_k \\nabla f(x_k) - [x(t_k) - \\alpha_k \\nabla f(x(t_k))] \\|~+ \\label{local_b} \\\\[2mm] & ~ \\| x(t_k) - \\alpha_k \\nabla f(x(t_k)) - x(t_{k+1}) \\| \\label{local_c} \\\\[2mm]\n        \\leqslant & ~ \\|x_k - x(t_k)\\| + \\alpha_k \\|\\nabla f(x_k) - \\nabla f(x(t_k))\\| ~+ \\label{local_d} \\\\[2mm]\n        & ~ ML\\alpha_k^2/2 \\label{local_e} \\\\[2mm]\n        \\leqslant & ~ (1+\\alpha_k M)\\|x_k - x(t_k)\\| + ML\\alpha_k^2/2. \\label{local_f}\n    \\end{align}\n\\end{subequations}\nThe term in \\eqref{local_c} is equal to the local truncation error $\\|x(t_{k+1})  - x(t_k) - x'(t_k)(t_{k+1}-t_k) \\|$, and can hence be bounded above using \\eqref{eq:local_error}. In the second term of \\eqref{local_d}, we invoke a Lipschitz constant of $\\nabla f$ on the convex hull of $B(X_0,\\sigma_T(X_0)+1)$. Indeed, by the induction hypothesis and $\\epsilon < 1$, $x_k$ and $x(t_k)$ belong to $B(X_0,\\sigma_T(X_0)+\\epsilon)$. Hence \n\\begin{subequations}\n    \\begin{align}\n        \\|x_{K+1} - x(t_{K+1})\\| \\leqslant &~ \\prod_{k=0}^K (1+\\alpha_kM)\\|x_0-x(t_0)\\|~+ \\label{eq:global_a}\\\\\n        &~ \\sum_{k=0}^K \\frac{ML\\alpha_k^2}{2} \\prod_{l=k+1}^{K} (1+\\alpha_lM)\\label{eq:global_b}\\\\\n        \\leqslant &~ \\prod_{l=0}^{K} (1+\\alpha_lM) \\sum_{k=0}^K \\frac{ML\\alpha_k^2}{2}\\label{eq:global_c}\\\\\n        \\leqslant &~ \\left(1+\\frac{M}{K+1}\\sum_{k=0}^K \\alpha_k\\right)^{K+1} \\sum_{k=0}^K \\frac{ML\\bar{\\alpha}\\alpha_k}{2}\\label{eq:global_d}\\\\\n        \\leqslant &~ \\left(1+\\frac{MT}{K+1}\\right)^{K+1} \\frac{ML\\bar{\\alpha}T}{2}\\label{eq:global_e}\\\\\n        \\leqslant &~~ \\bar{\\alpha} LMTe^{MT}/2\\label{eq:global_f}\\\\\n        = &~~ \\epsilon.\\label{eq:global_g}\n    \\end{align}\n\\end{subequations}\nThe term in \\eqref{eq:global_a} is equal to zero because $x_0 = x(t_0)$. By convention, we set the product in \\eqref{eq:global_b} to be equal to one if the index set for $l$ is empty. We bound the partial products ranging from $k+1$ to $K$ inside the summation of \\eqref{eq:global_b} by the full product ranging from $0$ to $K$ in order to obtain \\eqref{eq:global_c}. We then use the inequality of arithmetic and geometric means and $\\alpha_k\\leqslant\\bar{\\alpha}$ in order to obtain \\eqref{eq:global_d}. In order to prove the induction step, we assume that $\\alpha_0+\\cdots+\\alpha_K \\leqslant T$, hence \\eqref{eq:global_e}. \\eqref{eq:global_f} follows from $\\ln(1+x) \\leqslant x$ for all $x>0$. Finally, \\eqref{eq:global_g} holds because $\\bar{\\alpha} = 2\\epsilon e^{-MT}/(LMT)$.\n\\end{proof}","start":38442,"end":44644}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":["eq:length_finite_smooth_a","eq:length_finite_smooth_b","eq:length_finite_smooth_c","eq:length_finite_smooth_d","eq:length_finite_smooth_e","eq:sup_trajectory_T_smooth"],"declared_external":false,"source_only":false,"title":"有限时连续长度与公共区域","source_units":[{"raw_tex":"\\begin{proof}\nWithout loss of generality, we may assume that $X_0 \\neq \\emptyset$. Let $L>0$ and $M>0$ respectively denote Lipschitz constants of $f$ and $\\nabla f$ with respect to $\\|\\cdot\\|$ on the convex hull of $B(X_0,\\sigma_T(X_0)+1):= X_0 + B(0,\\sigma_T(X_0)+1)$ where \n\\begin{subequations}\n\\label{eq:sup_trajectory_T_smooth}\n\\begin{align}\n    \\sigma_T(X_0) := & \\sup\\limits_{x \\in \\mathcal{C}^1(\\mathbb{R}_+,\\mathbb{R}^n)} ~~ \\int_0^T \\|x'(t)\\|dt \\\\\n  & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx'(t) = - \\nabla f(x(t)), ~\\forall t > 0,\\\\[3mm] x(0) \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\nAbove, $\\mathcal{C}^1(\\mathbb{R}_+,\\mathbb{R}^n)$ is the set of continuous functions on $[0,\\infty )$ that are continuously differentiable on $(0,\\infty)$. Note that $\\sigma_T(X_0)>-\\infty$ since $f$ is lower bounded and differentiable with a locally Lipschitz gradient. For every $x_0\\in X_0$, there hence exists a unique globally defined continuous gradient trajectory of $f$ initialized at $x_0$ \\cite[Theorem 17.1.1]{attouch2014variational}. Also, $\\sigma_T(X_0)<\\infty$ since for any feasible point $x(\\cdot)$ of \\eqref{eq:sup_trajectory_T_smooth}, we have \n\\begin{subequations}\n\\begin{align}\n    \\int_0^T \\|x'(t)\\|dt & \\leqslant \\sqrt{T} \\sqrt{ \\int_0^T \\|x'(t)\\|^2 dt} \\label{eq:length_finite_smooth_a} \\\\\n    & = \\sqrt{T} \\sqrt{ -\\int_0^T \\langle \\nabla f(x(t)), x'(t) \\rangle dt} \\label{eq:length_finite_smooth_b} \\\\\n    & = \\sqrt{T} \\sqrt{ -\\int_0^T (f \\circ x)'(t) dt} \\label{eq:length_finite_smooth_c} \\\\\n    & = \\sqrt{T} \\sqrt{ f(x(0)) - f(x(T))} \\label{eq:length_finite_smooth_d} \\\\\n    & \\leqslant \\sqrt{T\\left(\\sup_{X_0} f - \\inf_{\\mathbb{R}^n} f\\right)} < \\infty. \\label{eq:length_finite_smooth_e}\n\\end{align}\n\\end{subequations}\nIndeed, \\eqref{eq:length_finite_smooth_a} is due to the Cauchy-Schwarz inequality. \\eqref{eq:length_finite_smooth_e} holds because $X_0$ is bounded and $f$ is lower bounded.\n","start":38442,"end":40391}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":["eq:local_error","eq:sup_trajectory_T_smooth"],"declared_external":false,"source_only":false,"title":"连续轨迹的二阶余项界","source_units":[{"raw_tex":"It is easy to check that $L$ and $ML$ are respectively Lipschitz and gradient Lipschitz constants on $[0,T]$ of all continuous gradient trajectories of $f$ initialized in $X_0$. Indeed, let $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ be a continuous gradient trajectory initialized in $X_0$. Since $x(t) \\in B(X_0,\\sigma_T(X_0))$ for all $t\\in [0,T]$, we have $\\|x'(t)\\| = \\|\\nabla f(x(t))\\| \\leqslant L$. By the mean value theorem \\cite[5.19 Theorem]{rudin1964principles}, for all $s,t \\in [0,T]$ we have $\\|x'(t)-x'(s)\\| = \\|\\nabla f(x(t)) - \\nabla f(x(s))\\| \\leqslant M \\|x(t)-x(s)\\| \\leqslant M L |t-s|$. As a byproduct, we get the Taylor bound\n\\begin{subequations}\n    \\begin{align}\n    \\|x(t)-x(s) - x'(s)(t-s)\\| & = \\left\\| \\int_s^t (x'(\\tau) - x'(s)) d\\tau \\right\\| \\\\\n    & \\leqslant \\left| \\int_s^t \\| x'(\\tau) - x'(s) \\| d\\tau \\right| \\\\\n    & \\leqslant \\left|\\int_s^t ML | \\tau - s | d\\tau \\right| \\\\\n    & = \\frac{ML}{2}(t-s)^2. \\label{eq:local_error}\n    \\end{align}\n\\end{subequations}\n","start":40392,"end":41389}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_trajectory_T_smooth","local_a","local_b","local_c","local_d","local_e","local_f"],"declared_external":false,"source_only":false,"title":"建立离散误差递推","source_units":[{"raw_tex":"Let $\\epsilon \\in (0,1)$ and let $\\bar{\\alpha} := 2\\epsilon e^{-MT}/(LMT)$. Consider $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$ and a sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$ and $x_0 \\in X_0$. Let $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ be the unique continuous gradient trajectory of $f$ initialized at $x_0$. Let $t_0:=0$ and $t_k := \\alpha_0+\\cdots+\\alpha_{k-1}$ when $k\\in \\mathbb{N}^*$ so that $\\|x_0 - x(t_0)\\| = 0 \\leqslant \\epsilon$. Assume that \\eqref{eq:tracking} holds up to some index $K$. For $k=0,\\hdots,K$, we have\n\\begin{subequations}\n    \\begin{align}\n        \\|x_{k+1} - x(t_{k+1})\\| = &  ~\\| x_k - \\alpha_k \\nabla f(x_k) - x(t_{k+1}) \\| \\label{local_a} \\\\[2mm]\n        \\leqslant  & ~ \\| x_k - \\alpha_k \\nabla f(x_k) - [x(t_k) - \\alpha_k \\nabla f(x(t_k))] \\|~+ \\label{local_b} \\\\[2mm] & ~ \\| x(t_k) - \\alpha_k \\nabla f(x(t_k)) - x(t_{k+1}) \\| \\label{local_c} \\\\[2mm]\n        \\leqslant & ~ \\|x_k - x(t_k)\\| + \\alpha_k \\|\\nabla f(x_k) - \\nabla f(x(t_k))\\| ~+ \\label{local_d} \\\\[2mm]\n        & ~ ML\\alpha_k^2/2 \\label{local_e} \\\\[2mm]\n        \\leqslant & ~ (1+\\alpha_k M)\\|x_k - x(t_k)\\| + ML\\alpha_k^2/2. \\label{local_f}\n    \\end{align}\n\\end{subequations}\nThe term in \\eqref{local_c} is equal to the local truncation error $\\|x(t_{k+1})  - x(t_k) - x'(t_k)(t_{k+1}-t_k) \\|$, and can hence be bounded above using \\eqref{eq:local_error}. In the second term of \\eqref{local_d}, we invoke a Lipschitz constant of $\\nabla f$ on the convex hull of $B(X_0,\\sigma_T(X_0)+1)$. Indeed, by the induction hypothesis and $\\epsilon < 1$, $x_k$ and $x(t_k)$ belong to $B(X_0,\\sigma_T(X_0)+\\epsilon)$. Hence \n","start":41390,"end":43099}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":["eq:global_a","eq:global_b","eq:global_c","eq:global_d","eq:global_e","eq:global_f","eq:global_g","eq:sup_trajectory_T_smooth"],"declared_external":false,"source_only":false,"title":"累积误差与步长选择","source_units":[{"raw_tex":"\\begin{subequations}\n    \\begin{align}\n        \\|x_{K+1} - x(t_{K+1})\\| \\leqslant &~ \\prod_{k=0}^K (1+\\alpha_kM)\\|x_0-x(t_0)\\|~+ \\label{eq:global_a}\\\\\n        &~ \\sum_{k=0}^K \\frac{ML\\alpha_k^2}{2} \\prod_{l=k+1}^{K} (1+\\alpha_lM)\\label{eq:global_b}\\\\\n        \\leqslant &~ \\prod_{l=0}^{K} (1+\\alpha_lM) \\sum_{k=0}^K \\frac{ML\\alpha_k^2}{2}\\label{eq:global_c}\\\\\n        \\leqslant &~ \\left(1+\\frac{M}{K+1}\\sum_{k=0}^K \\alpha_k\\right)^{K+1} \\sum_{k=0}^K \\frac{ML\\bar{\\alpha}\\alpha_k}{2}\\label{eq:global_d}\\\\\n        \\leqslant &~ \\left(1+\\frac{MT}{K+1}\\right)^{K+1} \\frac{ML\\bar{\\alpha}T}{2}\\label{eq:global_e}\\\\\n        \\leqslant &~~ \\bar{\\alpha} LMTe^{MT}/2\\label{eq:global_f}\\\\\n        = &~~ \\epsilon.\\label{eq:global_g}\n    \\end{align}\n\\end{subequations}\nThe term in \\eqref{eq:global_a} is equal to zero because $x_0 = x(t_0)$. By convention, we set the product in \\eqref{eq:global_b} to be equal to one if the index set for $l$ is empty. We bound the partial products ranging from $k+1$ to $K$ inside the summation of \\eqref{eq:global_b} by the full product ranging from $0$ to $K$ in order to obtain \\eqref{eq:global_c}. We then use the inequality of arithmetic and geometric means and $\\alpha_k\\leqslant\\bar{\\alpha}$ in order to obtain \\eqref{eq:global_d}. In order to prove the induction step, we assume that $\\alpha_0+\\cdots+\\alpha_K \\leqslant T$, hence \\eqref{eq:global_e}. \\eqref{eq:global_f} follows from $\\ln(1+x) \\leqslant x$ for all $x>0$. Finally, \\eqref{eq:global_g} holds because $\\bar{\\alpha} = 2\\epsilon e^{-MT}/(LMT)$.\n\\end{proof}","start":43099,"end":44644}]},{"id":"S037","semantic_type":"proposition","kind":"proposition","labels":["prop:KL"],"declared_external":false,"source_only":false,"title":"跨临界值的统一 KL 不等式","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:KL}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz definable function and let $X$ be a bounded subset of $\\mathbb{R}^n$. Let $V$ be the set of critical values of $f$ in $\\overline{X}$ if it is nonempty, otherwise let $V:=\\{0\\}$. There exists a concave definable diffeomorphism $\\psi:[0,\\infty)\\rightarrow[0,\\infty)$ such that\n\\begin{equation*}\n    \\forall x \\in X \\setminus (\\partial \\tilde{f})^{-1}(0), ~~~ d(0,\\partial (\\psi \\circ \\tilde{f})(x)) \\geqslant 1\n\\end{equation*}\nwhere $\\tilde{f}(x) := d(f(x),V)$ for all $x\\in \\mathbb{R}^n$.\n\\end{proposition}","start":49370,"end":49978}]},{"id":"S038","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"统一 KL 不等式的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nBy the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, the set $V$ is finite. In addition, by the Kurdyka-\\L{}ojasiewicz inequality \\cite[Corollary 15]{bolte2007clarke}, there exist $\\epsilon>0$ and a strictly increasing concave continuous definable function $\\psi:[0,\\epsilon)\\rightarrow \\mathbb{R}$ that is continuously differentiable on $(0,\\epsilon)$ with $\\psi(0) = 0$ such that, for all $x \\in X$ and $v \\in V$ satisfying $0 < | f(x)-v | < \\epsilon$, we have $d(0,\\partial (\\psi \\circ | f-v|)(x)) \\geqslant 1$. The fact that $\\psi$ can be assumed to be concave is due to the monotonicity theorem \\cite[(1.2) p. 43]{van1998tame} \\cite[Lemma 2]{kurdyka1998gradients} by following the argument in \\cite[Theorem 4.1]{attouch2010proximal}. Let $\\rho:= \\inf \\{(v-v')/2: v,v' \\in V, v > v'\\}>0$ and $\\rho' : = \\min \\{ \\rho, \\epsilon\\}$. Observe that $c>0$ where\n\\begin{equation*}\n    c := \\inf \\{ \\|s\\| : \ns \\in \\partial f(x), ~ x\\in X, ~ | f(x) - v | \\geqslant \\rho',~ v\\in V \\}.\n\\end{equation*}\nIndeed, assume that there exist sequences $(s_k)_{k\\in \\mathbb{N}}$ and $(x_k)_{k\\in\\mathbb{N}}$ such that $\\|s_k\\|$ converges to zero and $s_k \\in \\partial f(x_k)$, $x_k\\in X$, and $| f(x_k) - v | \\geqslant \\rho'$ for all $v\\in V$. Since $X$ is bounded, there exist subsequences (again denoted $s_k$ and $x_k$) such that $x_k$ has a limit $x^*$ in $\\overline{X}$. Since $f$ is continuous, $| f(x^*) - v | \\geqslant \\rho'>0$ for all $v \\in V$. Also, since $s_k$ converges to zero, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990}, we have $0\\in \\partial f(x^*)$. This yields a contradiction. \n\nAfter possibly reducing $\\epsilon$, we may assume that $\\lim_{t\\nearrow\\rho'} \\psi'(t)>0$. If $\\lim_{t\\nearrow\\rho'} \\psi'(t) \\geqslant 1/c$, then we may extend $\\psi$ to an affine function on $[\\rho',\\infty)$ with slope equal to $\\lim_{t\\nearrow\\rho'} \\psi'(t)$. Otherwise, we can multiply $\\psi$ by $1/(c\\lim_{t\\nearrow\\rho'} \\psi'(t))$ before taking the affine extension. Now let $x \\in X$ be such that $0 \\notin \\partial \\tilde{f}(x)$. If $f(x)$ is strictly within $\\rho'$ distance of some $v \\in V$, then from $\\rho' \\leqslant \\rho$ it follows that $\\tilde{f}(\\tilde{x}) = d(f(\\tilde{x}),V) = | f(\\tilde{x})-v|$ for all $\\tilde{x}$ in a neighborhood of $x$ in $\\mathbb{R}^n$, and thus $d(0,\\partial ( \\psi \\circ \\tilde{f})(x)) = d(0,\\partial (\\psi \\circ | f-v|)(x)) \\geqslant 1$. The inequality is due to $\\rho' \\leqslant \\epsilon$. Otherwise, by \\cite[2.3.9 Theorem (Chain Rule I) (ii) p. 42]{clarke1990} we have $d(0,\\partial ( \\psi \\circ \\tilde{f})(x)) = \\psi'(\\tilde{f}(x)) d(0,\\partial \\tilde{f} (x)) = \\psi'(\\tilde{f}(x)) d(0,\\partial f(x)) \\geqslant \\psi'(\\tilde{f}(x)) c \\geqslant 1$. \n\\end{proof}","start":49979,"end":52712}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"有限临界值与局部凹 KL 函数","source_units":[{"raw_tex":"\\begin{proof}\nBy the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, the set $V$ is finite. In addition, by the Kurdyka-\\L{}ojasiewicz inequality \\cite[Corollary 15]{bolte2007clarke}, there exist $\\epsilon>0$ and a strictly increasing concave continuous definable function $\\psi:[0,\\epsilon)\\rightarrow \\mathbb{R}$ that is continuously differentiable on $(0,\\epsilon)$ with $\\psi(0) = 0$ such that, for all $x \\in X$ and $v \\in V$ satisfying $0 < | f(x)-v | < \\epsilon$, we have $d(0,\\partial (\\psi \\circ | f-v|)(x)) \\geqslant 1$. The fact that $\\psi$ can be assumed to be concave is due to the monotonicity theorem \\cite[(1.2) p. 43]{van1998tame} \\cite[Lemma 2]{kurdyka1998gradients} by following the argument in \\cite[Theorem 4.1]{attouch2010proximal}. Let $\\rho:= \\inf \\{(v-v')/2: v,v' \\in V, v > v'\\}>0$ and $\\rho' : = \\min \\{ \\rho, \\epsilon\\}$. Observe that $c>0$ where\n","start":49979,"end":50872}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"远离全部临界值的次梯度下界","source_units":[{"raw_tex":"\\begin{equation*}\n    c := \\inf \\{ \\|s\\| : \ns \\in \\partial f(x), ~ x\\in X, ~ | f(x) - v | \\geqslant \\rho',~ v\\in V \\}.\n\\end{equation*}\nIndeed, assume that there exist sequences $(s_k)_{k\\in \\mathbb{N}}$ and $(x_k)_{k\\in\\mathbb{N}}$ such that $\\|s_k\\|$ converges to zero and $s_k \\in \\partial f(x_k)$, $x_k\\in X$, and $| f(x_k) - v | \\geqslant \\rho'$ for all $v\\in V$. Since $X$ is bounded, there exist subsequences (again denoted $s_k$ and $x_k$) such that $x_k$ has a limit $x^*$ in $\\overline{X}$. Since $f$ is continuous, $| f(x^*) - v | \\geqslant \\rho'>0$ for all $v \\in V$. Also, since $s_k$ converges to zero, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990}, we have $0\\in \\partial f(x^*)$. This yields a contradiction. \n","start":50872,"end":51601}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"缩放延拓并验证统一不等式","source_units":[{"raw_tex":"\nAfter possibly reducing $\\epsilon$, we may assume that $\\lim_{t\\nearrow\\rho'} \\psi'(t)>0$. If $\\lim_{t\\nearrow\\rho'} \\psi'(t) \\geqslant 1/c$, then we may extend $\\psi$ to an affine function on $[\\rho',\\infty)$ with slope equal to $\\lim_{t\\nearrow\\rho'} \\psi'(t)$. Otherwise, we can multiply $\\psi$ by $1/(c\\lim_{t\\nearrow\\rho'} \\psi'(t))$ before taking the affine extension. Now let $x \\in X$ be such that $0 \\notin \\partial \\tilde{f}(x)$. If $f(x)$ is strictly within $\\rho'$ distance of some $v \\in V$, then from $\\rho' \\leqslant \\rho$ it follows that $\\tilde{f}(\\tilde{x}) = d(f(\\tilde{x}),V) = | f(\\tilde{x})-v|$ for all $\\tilde{x}$ in a neighborhood of $x$ in $\\mathbb{R}^n$, and thus $d(0,\\partial ( \\psi \\circ \\tilde{f})(x)) = d(0,\\partial (\\psi \\circ | f-v|)(x)) \\geqslant 1$. The inequality is due to $\\rho' \\leqslant \\epsilon$. Otherwise, by \\cite[2.3.9 Theorem (Chain Rule I) (ii) p. 42]{clarke1990} we have $d(0,\\partial ( \\psi \\circ \\tilde{f})(x)) = \\psi'(\\tilde{f}(x)) d(0,\\partial \\tilde{f} (x)) = \\psi'(\\tilde{f}(x)) d(0,\\partial f(x)) \\geqslant \\psi'(\\tilde{f}(x)) c \\geqslant 1$. \n\\end{proof}","start":51601,"end":52712}]},{"id":"S042","semantic_type":"proposition","kind":"proposition","labels":["prop:chain_rule"],"declared_external":false,"source_only":false,"title":"可定义次梯度流的能量恒等式","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:chain_rule}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a definable function and let $x:(0,T)\\rightarrow \\mathbb{R}^n$ with $0 < T \\leqslant \\infty$ be an absolutely continuous function. If $f$ is locally Lipschitz at $x(t)$ for all $t \\in (0,T)$ and $x'(t) \\in -\\partial f(x(t))$ for almost every $t\\in(0,T)$, then\n\\begin{equation*}\n    (f \\circ x)'(t) = -\\|x'(t)\\|^2 = -d(0,\\partial f(x(t)))^2,~~~\\mathrm{for~a.e.}~t \\in (0,T).\n\\end{equation*}\n\\end{proposition}","start":52900,"end":53398}]},{"id":"S043","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"能量恒等式的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nSince $f$ is definable and locally Lipschitz at $x(t)$ for all $t\\in(0,T)$, it satisfies the chain rule \\cite[Proposition 2 (iv)]{bolte2020conservative} (see also \\cite[Theorem 5.8]{davis2020stochastic} and \\cite[Lemma 2.10]{drusvyatskiy2015quadratic}), that is to say $(f \\circ x)'(t) = \\langle s , x'(t) \\rangle$ for all $s\\in \\partial f(x(t))$ and almost every $t\\in (0,T)$. In particular, we may take $s := - x'(t) \\in \\partial f(x(t))$.\nFinally, by \\cite[Lemma 5.2]{davis2020stochastic} (see also \\cite[Proposition 4.10]{drusvyatskiy2015curves}) it holds that $\\|x'(t)\\| = d(0,\\partial f(x(t)))$ for almost every $t\\in (0,T)$. \n\\end{proof}","start":53399,"end":54057}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"沿曲线链式法则与能量下降","source_units":[{"raw_tex":"\\begin{proof}\nSince $f$ is definable and locally Lipschitz at $x(t)$ for all $t\\in(0,T)$, it satisfies the chain rule \\cite[Proposition 2 (iv)]{bolte2020conservative} (see also \\cite[Theorem 5.8]{davis2020stochastic} and \\cite[Lemma 2.10]{drusvyatskiy2015quadratic}), that is to say $(f \\circ x)'(t) = \\langle s , x'(t) \\rangle$ for all $s\\in \\partial f(x(t))$ and almost every $t\\in (0,T)$. In particular, we may take $s := - x'(t) \\in \\partial f(x(t))$.\n","start":53399,"end":53855}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"速度等于最小次梯度范数","source_units":[{"raw_tex":"Finally, by \\cite[Lemma 5.2]{davis2020stochastic} (see also \\cite[Proposition 4.10]{drusvyatskiy2015curves}) it holds that $\\|x'(t)\\| = d(0,\\partial f(x(t)))$ for almost every $t\\in (0,T)$. \n\\end{proof}","start":53855,"end":54057}]},{"id":"S046","semantic_type":"proposition","kind":"proposition","labels":["eq:length_continuous","prop:length_continuous"],"declared_external":false,"source_only":false,"title":"连续轨迹的长度下降公式","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:length_continuous}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz definable function and let $X$ be a bounded subset of $\\mathbb{R}^n$. Assume that $f$ has at most $m\\in \\mathbb{N}^*$ critical values in $\\overline{X}$. Let $\\psi$ be a desingularizing function of $f$ on $X$. If $x:[0,T]\\rightarrow X$ with $T \\geqslant 0$ is an absolutely continuous function such that $x'(t) \\in -\\partial f(x(t))$ for almost every $t\\in(0,T)$, then\n\\begin{equation}\n\\label{eq:length_continuous}\n    \\frac{1}{2m}\\int_0^T\\|x'(t)\\|dt ~\\leqslant~ \\psi\\left(\\frac{f(x(0))-f(x(T))}{2m}\\right).\n\\end{equation}\n\\end{proposition}","start":54179,"end":54836}]},{"id":"S047","semantic_type":"proof","kind":"proof","labels":["eq:god_a","eq:god_b"],"declared_external":false,"source_only":false,"title":"连续长度公式的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nWe first consider the special case where $0 \\notin \\partial \\tilde{f}(x(t))$ for all $t\\in (0,T)$ where $\\tilde{f}$ is defined as in Proposition \\ref{prop:KL}. Consider the change of variables $\\bar{x} = x \\circ \\varphi$ where $\\bar{t}=\\varphi^{-1}(t) = \\int_0^{t} ds /(\\psi' \\circ \\tilde{f} \\circ x)(s)$ and $t=\\varphi(\\bar{t}) = \\int_0^{\\bar{t}} (\\psi' \\circ \\tilde{f} \\circ \\bar{x})(\\bar{s}) d\\bar{s}$ by the inverse function theorem \\cite[Theorem 17.7.2]{tao2006analysis}. We have $\\bar{x}'(\\bar{t}) = \\varphi'(\\bar{t})x'(t) \\in -(\\psi' \\circ \\tilde{f} \\circ \\bar{x})(\\bar{t}) \\partial f(\\bar{x}(\\bar{t})) = \\pm \\partial (\\psi \\circ \\tilde{f})(\\bar{x}(\\bar{t}))$ for almost every $\\bar{t}\\in (0,\\overline{T})$, where $\\overline{T} = \\varphi^{-1}(T)$ and the sign is constant. By Proposition \\ref{prop:chain_rule} and the Cauchy-Schwarz inequality, we thus have\n\\begin{subequations}\n\\begin{align}\n    \\overline{T} \\leqslant & \\int_0^{\\overline{T}} d(0,\\partial (\\psi \\circ \\tilde{f})(\\bar{x}(\\bar{t})))d\\bar{t} \\label{eq:god_a} \\\\\n    = & \\int_0^{\\overline{T}} \\|\\bar{x}'(\\bar{t})\\|d\\bar{t} \\label{eq:god_b} \\\\\n    = & \\int_0^{T} \\|x'(t)\\|dt \\\\\n    \\leqslant & \\sqrt{\\overline{T}} \\sqrt{\\int_0^{\\overline{T}} \\|\\bar{x}'(\\bar{t})\\|^2d\\bar{t}} \\\\\n    = & \\sqrt{\\overline{T}}\\sqrt{|(\\psi\\circ \\tilde{f}\\circ\\bar{x})(0)-(\\psi\\circ \\tilde{f}\\circ\\bar{x})(\\overline{T})|}.\n\\end{align}\n\\end{subequations}\nHence $\\overline{T} \\leqslant |(\\psi\\circ \\tilde{f}\\circ\\bar{x})(0)-(\\psi\\circ \\tilde{f}\\circ\\bar{x})(\\overline{T})|$ and, since $\\psi$ is concave, we have\n\\begin{align*}\n     \\int_0^T \\|x'(t)\\|dt & \\leqslant |(\\psi\\circ \\tilde{f}\\circ\\bar{x})(0)-(\\psi\\circ \\tilde{f}\\circ\\bar{x})(\\overline{T})| \\\\\n     & \\leqslant \\psi (|( \\tilde{f}\\circ\\bar{x})(0)-(\\tilde{f}\\circ\\bar{x})(\\overline{T})|) \\\\[2mm]\n     & = \\psi (f(x(0))-f(x(T))).\n\\end{align*}\nWe next consider the general case where there may exist $t\\in (0,T)$ such that $0 \\in \\partial \\tilde{f}(x(t))$, namely $0 < t_1 < \\cdots < t_{k} < T$, and potentially $(t_k,T)$. For notational convenience, let $t_0 := 0$ and $t_{k+1}:=T$ where $k$ is possibly equal to zero. Since $k \\leqslant 2m-1$ and $\\psi$ is concave, we have\n\\begin{align*}\n    \\int_0^T \\|x'(t)\\|dt & = \\sum_{i=0}^k \\int_{t_i}^{t_{i+1}} \\|x'(t)\\|dt \\\\\n    & \\leqslant \\sum_{i=0}^k \\psi (f(x(t_i))-f(x(t_{i+1}))) \\\\[-.5mm]\n    & \\leqslant (k+1) ~ \\psi \\left( \\frac{1}{k+1} \\sum_{i=0}^k f(x(t_i))-f(x(t_{i+1}))\\right) \\\\\n    & = (k+1) ~ \\psi \\left( \\frac{f(x(0)) - f(x(T))}{k+1} \\right) \\\\\n    & \\leqslant 2m ~ \\psi \\left( \\frac{f(x(0)) - f(x(T))}{2m} \\right). \\qedhere\n\\end{align*}\n\\end{proof}","start":54837,"end":57462}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"无奇异区间上的重参数化","source_units":[{"raw_tex":"\\begin{proof}\nWe first consider the special case where $0 \\notin \\partial \\tilde{f}(x(t))$ for all $t\\in (0,T)$ where $\\tilde{f}$ is defined as in Proposition \\ref{prop:KL}. Consider the change of variables $\\bar{x} = x \\circ \\varphi$ where $\\bar{t}=\\varphi^{-1}(t) = \\int_0^{t} ds /(\\psi' \\circ \\tilde{f} \\circ x)(s)$ and $t=\\varphi(\\bar{t}) = \\int_0^{\\bar{t}} (\\psi' \\circ \\tilde{f} \\circ \\bar{x})(\\bar{s}) d\\bar{s}$ by the inverse function theorem \\cite[Theorem 17.7.2]{tao2006analysis}. We have $\\bar{x}'(\\bar{t}) = \\varphi'(\\bar{t})x'(t) \\in -(\\psi' \\circ \\tilde{f} \\circ \\bar{x})(\\bar{t}) \\partial f(\\bar{x}(\\bar{t})) = \\pm \\partial (\\psi \\circ \\tilde{f})(\\bar{x}(\\bar{t}))$ for almost every $\\bar{t}\\in (0,\\overline{T})$, where $\\overline{T} = \\varphi^{-1}(T)$ and the sign is constant. By Proposition \\ref{prop:chain_rule} and the Cauchy-Schwarz inequality, we thus have\n","start":54837,"end":55716}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["eq:god_a","eq:god_b"],"declared_external":false,"source_only":false,"title":"积分 KL 得单段长度界","source_units":[{"raw_tex":"\\begin{subequations}\n\\begin{align}\n    \\overline{T} \\leqslant & \\int_0^{\\overline{T}} d(0,\\partial (\\psi \\circ \\tilde{f})(\\bar{x}(\\bar{t})))d\\bar{t} \\label{eq:god_a} \\\\\n    = & \\int_0^{\\overline{T}} \\|\\bar{x}'(\\bar{t})\\|d\\bar{t} \\label{eq:god_b} \\\\\n    = & \\int_0^{T} \\|x'(t)\\|dt \\\\\n    \\leqslant & \\sqrt{\\overline{T}} \\sqrt{\\int_0^{\\overline{T}} \\|\\bar{x}'(\\bar{t})\\|^2d\\bar{t}} \\\\\n    = & \\sqrt{\\overline{T}}\\sqrt{|(\\psi\\circ \\tilde{f}\\circ\\bar{x})(0)-(\\psi\\circ \\tilde{f}\\circ\\bar{x})(\\overline{T})|}.\n\\end{align}\n\\end{subequations}\nHence $\\overline{T} \\leqslant |(\\psi\\circ \\tilde{f}\\circ\\bar{x})(0)-(\\psi\\circ \\tilde{f}\\circ\\bar{x})(\\overline{T})|$ and, since $\\psi$ is concave, we have\n\\begin{align*}\n     \\int_0^T \\|x'(t)\\|dt & \\leqslant |(\\psi\\circ \\tilde{f}\\circ\\bar{x})(0)-(\\psi\\circ \\tilde{f}\\circ\\bar{x})(\\overline{T})| \\\\\n     & \\leqslant \\psi (|( \\tilde{f}\\circ\\bar{x})(0)-(\\tilde{f}\\circ\\bar{x})(\\overline{T})|) \\\\[2mm]\n     & = \\psi (f(x(0))-f(x(T))).\n\\end{align*}\n","start":55716,"end":56697}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"跨临界值分段并合并","source_units":[{"raw_tex":"We next consider the general case where there may exist $t\\in (0,T)$ such that $0 \\in \\partial \\tilde{f}(x(t))$, namely $0 < t_1 < \\cdots < t_{k} < T$, and potentially $(t_k,T)$. For notational convenience, let $t_0 := 0$ and $t_{k+1}:=T$ where $k$ is possibly equal to zero. Since $k \\leqslant 2m-1$ and $\\psi$ is concave, we have\n\\begin{align*}\n    \\int_0^T \\|x'(t)\\|dt & = \\sum_{i=0}^k \\int_{t_i}^{t_{i+1}} \\|x'(t)\\|dt \\\\\n    & \\leqslant \\sum_{i=0}^k \\psi (f(x(t_i))-f(x(t_{i+1}))) \\\\[-.5mm]\n    & \\leqslant (k+1) ~ \\psi \\left( \\frac{1}{k+1} \\sum_{i=0}^k f(x(t_i))-f(x(t_{i+1}))\\right) \\\\\n    & = (k+1) ~ \\psi \\left( \\frac{f(x(0)) - f(x(T))}{k+1} \\right) \\\\\n    & \\leqslant 2m ~ \\psi \\left( \\frac{f(x(0)) - f(x(T))}{2m} \\right). \\qedhere\n\\end{align*}\n\\end{proof}","start":56697,"end":57462}]},{"id":"S051","semantic_type":"proposition","kind":"proposition","labels":["eq:length_discrete","prop:length_discrete"],"declared_external":false,"source_only":false,"title":"离散轨迹的长度下降公式","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:length_discrete}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a definable differentiable function with a locally Lipschitz gradient and let $X$ be a bounded open subset of $\\mathbb{R}^n$. Assume that $f$ has at most $m\\in \\mathbb{N}^*$ critical values in $\\overline{X}$ and that $L>0$ is a Lipschitz constant of $f$ on the convex hull of $X$. Let $\\psi$ be a desingularizing function of $f$ on $X$ and $\\epsilon>0$. There exists $\\bar{\\alpha}>0$ such that, for all $K \\in \\mathbb{N}$, $\\alpha_0,\\hdots,\\alpha_K \\in (0,\\bar{\\alpha}]$, and $(x_0,\\hdots,x_{K+1}) \\in X \\times \\cdots \\times X \\times \\mathbb{R}^n$ such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for $k = 0,\\hdots,K$, we have\n\\begin{equation}\n\\label{eq:length_discrete}\n    \\frac{1}{(2+\\epsilon)m}\\sum_{k=0}^K\\|x_{k+1}-x_k\\| ~\\leqslant~ \\psi\\left(\\frac{f(x_0)-f(x_K)}{2m}\\right) + \\frac{2L}{2+\\epsilon}\\max_{k=0,\\hdots,K} \\alpha_k\n\\end{equation}\nand $f(x_0)\\geqslant \\cdots \\geqslant f(x_{K+1})$.\n\\end{proposition}","start":57545,"end":58558}]},{"id":"S052","semantic_type":"proof","kind":"proof","labels":["eq:descent_lemma_1","eq:descent_lemma_2","eq:length_finite"],"declared_external":false,"source_only":false,"title":"离散长度公式的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $M>0$ be a Lipschitz constant of $\\nabla f$ with respect to $\\|\\cdot\\|$ on the convex hull of $B(X,1):=X+B(0,1)$ and let $\\bar{\\alpha} := \\min\\{L^{-1},2\\epsilon(6+\\epsilon)^{-1}M^{-1}\\}$. Let $K \\in \\mathbb{N}$, $\\alpha_0,\\hdots,\\alpha_K \\in (0,\\bar{\\alpha}]$, and $(x_0,\\hdots,x_{K+1}) \\in X \\times \\cdots \\times X \\times \\mathbb{R}^n$ be such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for $k=0,\\hdots,K$. It holds that $\\|x_{K+1}-x_K\\| = \\alpha_K\\|\\nabla f(x_K)\\| \\leqslant \\bar{\\alpha}L \\leqslant 1$ and thus $x_0,\\hdots,x_{K+1} \\in B(X,1)$. A bound on the Taylor expansion of $f$ yields\n\\begin{align*}\n    f(x_{k+1})-f(x_k) & \\leqslant \\langle \\nabla f(x_k), x_{k+1}-x_k\\rangle + \\frac{M}{2}\\|x_{k+1}-x_k\\|^2 \\\\\n    & = \\left(\\frac{M\\alpha_k}{2}-1\\right) \\|x_{k+1}-x_k\\| \\|\\nabla f(x_k)\\|\n\\end{align*}\nfor $k=0,\\hdots,K$, and thus\n\\begin{equation}\n\\label{eq:descent_lemma_1}\n    \\|x_{k+1}-x_k\\|\\|\\nabla f(x_k)\\| \\leqslant \\frac{2}{2- M\\alpha_k}(f(x_k) - f(x_{k+1})).\n\\end{equation}\nWe also have\n    \\begin{align*}\n        \\|\\nabla f(x_{k+1})\\| & \\leqslant \\| \\nabla f(x_{k+1}) - \\nabla f(x_k) \\| + \\|\\nabla f(x_k)\\| \\\\\n        & \\leqslant M \\| x_{k+1} - x_k \\| + \\|\\nabla f(x_k)\\| \\\\\n        & \\leqslant (M\\alpha_k+1) \\|\\nabla f(x_k)\\|\n    \\end{align*}\nand thus \n\\begin{equation}\n\\label{eq:descent_lemma_2}\n    \\|x_{k+1}-x_k\\|\\|\\nabla f(x_{k+1})\\| \\leqslant \\frac{2+2M\\alpha_k}{2- M\\alpha_k}(f(x_k) - f(x_{k+1})).\n\\end{equation}\nNote that \n\\begin{equation*}\n    1 \\leqslant \\frac{2}{2- M\\alpha_k} \\leqslant \\frac{2+2M\\alpha_k}{2- M\\alpha_k} \\leqslant 1+\\frac{\\epsilon}{2}.\n\\end{equation*}\nLet $\\tilde{f}$ and $V$ be defined as in Proposition \\ref{prop:KL}.\n\nAssume that $[f(x_K),f(x_0)]$ excludes the elements of $V$ and the averages of any two consecutive elements of $V$. Then $0\\notin \\partial \\tilde{f}(x_k)$ and $1 \\leqslant \\| \\nabla(\\psi \\circ \\tilde{f})(x_k)\\| = \\psi'(\\tilde{f}(x_k)) \\|\\nabla \\tilde{f}(x_k)\\|$ for $k=0,\\hdots,K$. Let $k \\in \\{0,\\hdots,K-1\\}$. If $\\tilde{f}(x_k) \\geqslant \\tilde{f}(x_{k+1})$, then multiplying \\eqref{eq:descent_lemma_1} by $\\psi'(\\tilde{f}(x_k))$ and using the concavity of $\\psi$, we find that \n\\begin{align*}\n    \\|x_{k+1}-x_k\\| & \\leqslant (1+\\epsilon/2)\\psi'(\\tilde{f}(x_k))(f(x_k) - f(x_{k+1})) \\\\\n    & = (1+\\epsilon/2)\\psi'(\\tilde{f}(x_k))(\\tilde{f}(x_k) - \\tilde{f}(x_{k+1})) \\\\\n    & \\leqslant (1+\\epsilon/2)(\\psi (\\tilde{f}(x_k)) - \\psi (\\tilde{f}(x_{k+1}))).\n\\end{align*}\nIf $\\tilde{f}(x_k) \\leqslant \\tilde{f}(x_{k+1})$, then multiplying \\eqref{eq:descent_lemma_2} by $\\psi'(\\tilde{f}(x_{k+1}))$ and using the concavity of $\\psi$, we find that \n\\begin{align*}\n    \\|x_{k+1}-x_k\\| & \\leqslant (1+\\epsilon/2)\\psi'(\\tilde{f}(x_{k+1}))(f(x_k) - f(x_{k+1})) \\\\\n    & = (1+\\epsilon/2)\\psi'(\\tilde{f}(x_{k+1}))(\\tilde{f}(x_{k+1}) - \\tilde{f}(x_k)) \\\\\n    & \\leqslant (1+\\epsilon/2)(\\psi (\\tilde{f}(x_{k+1})) - \\psi (\\tilde{f}(x_k))).\n\\end{align*}\nAs a result,\n\\begin{equation*}\n    \\|x_{k+1}-x_k\\| \\leqslant (1+\\epsilon/2)|\\psi (\\tilde{f}(x_k)) - \\psi (\\tilde{f}(x_{k+1}))|, ~~~ k = 0,\\hdots,K-1.\n\\end{equation*}\nWe obtain the telescoping sum \n\\begin{subequations}\n    \\label{eq:length_finite} \n    \\begin{align}\n    \\sum_{k=0}^K \\|x_{k+1}-x_k\\| & \\leqslant (1+\\epsilon/2) |\\psi(\\tilde{f}(x_0)) - \\psi(\\tilde{f}(x_K))| + \\|x_{K+1} - x_K\\| \\\\[-4mm]\n    & \\leqslant (1+\\epsilon/2) \\psi ( | \\tilde{f}(x_0) - \\tilde{f}(x_K) |) + \\alpha_K\\|\\nabla f(x_K)\\| \\\\\n    & \\leqslant (1+\\epsilon/2) \\psi( f(x_0) - f(x_K)) + L\\max_{k=0,\\hdots,K} \\alpha_k.\n    \\end{align}\n\\end{subequations}\n\nAssume that $[f(x_K),f(x_0))$ excludes the elements of $V$ and the averages of any two consecutive elements of $V$. Since we are excluding a finite number of elements, there exists $\\epsilon>0$ such that $[f(x_K)-\\epsilon,f(x_0))$ satisfies the same property. If $x_0$ is a local minimum of $f$ on the open set $X$, then $\\nabla f(x_0) = 0$ by Fermat's rule. The length of $x_0,\\hdots,x_{K+1}$ is then equal to zero and the formula obtained in \\eqref{eq:length_finite} trivially holds. Otherwise, there exists a sequence $(x_0^l)_{l\\in\\mathbb{N}}$ in $\\mathbb{R}^n$ such that $f(x_0^l) < f(x_0)$ and $x_0^l$ converges to $x_0$. For every $l \\in \\mathbb{N}$, consider the sequence $(x^l_k)_{k\\in \\mathbb{N}}$ such that $x^l_{k+1} = x^l_k - \\alpha_k \\nabla f(x_k^l)$ for all $k \\in \\mathbb{N}$. Since $\\nabla f$ is continuous, for any fixed $k \\in \\{0,\\hdots,K\\}$, $x_k^l$ converges to $x_k \\in X$ and eventually belongs to the open set $X$. Thus $[f(x^l_K),f(x^l_0)] \\subset [f(x_K)-\\epsilon,f(x_0))$ eventually excludes the elements of $V$ and the averages of any two consecutive elements of $V$. We can apply the formula obtained in \\eqref{eq:length_finite}, namely\n\\begin{equation*}\n    \\sum_{k=0}^K \\|x^l_{k+1}-x^l_k\\| \\leqslant (1+\\epsilon/2) \\psi( f(x^l_0) - f(x^l_K)) + L\\max_{k=0,\\hdots,K} \\alpha_k.\n\\end{equation*}\nPassing to the limit yields\n\\begin{equation*}\n    \\sum_{k=0}^K \\|x_{k+1}-x_k\\| \\leqslant (1+\\epsilon/2) \\psi( f(x_0) - f(x_K)) + L\\max_{k=0,\\hdots,K} \\alpha_k.\n\\end{equation*}\n\nWe next consider the general case where\n\\begin{equation*}\n    [f(x_K),f(x_{K_p+1})) \\cup \\cdots \\cup [f(x_{K_2}),f(x_{K_1+1})) \\cup [f(x_{K_1}),f(x_0))\n\\end{equation*}\nexcludes the elements of $V$ and the averages of any two consecutive elements of $V$. For notational convenience, let $K_0 := -1$ and $K_{p+1} := K$. Since $p \\leqslant 2m-1$, we have\n\\begin{align*}\n    \\sum_{k=0}^K \\|x_{k+1}-x_k\\| = & \\sum_{i=0}^p \\sum_{k=K_i+1}^{K_{i+1}} \\|x_{k+1}-x_k\\| \\\\\n    \\leqslant & \\sum_{i=0}^p \\left((1+\\epsilon/2)\\psi( f(x_{{K_i}+1}) - f(x_{K_{i+1}})) + L\\max_{k=0,\\hdots,K} \\alpha_k\\right) \\\\[-.5mm]\n    \\leqslant & (1+\\epsilon/2)(p+1) ~ \\psi \\left( \\frac{1}{p+1} \\sum_{i=0}^p  f(x_{{K_i}+1}) - f(x_{K_{i+1}}) \\right) + \\\\ \n    & (p+1)L\\max_{k=0,\\hdots,K} \\alpha_k \\\\\n    \\leqslant & (1+\\epsilon/2)(p+1) \\psi \\left( \\frac{f(x_0)-f(x_K)}{p+1} \\right) + (p+1)L\\max_{k=0,\\hdots,K} \\alpha_k \\\\\n    \\leqslant & (2+\\epsilon)m ~ \\psi \\left( \\frac{f(x_0)-f(x_K)}{2m} \\right) + 2m L\\max_{k=0,\\hdots,K} \\alpha_k.\\qedhere\n\\end{align*}\n\\end{proof}","start":58559,"end":64644}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":["eq:descent_lemma_1","eq:length_finite"],"declared_external":false,"source_only":false,"title":"选择步长并获得下降界","source_units":[{"raw_tex":"\\begin{proof}\nLet $M>0$ be a Lipschitz constant of $\\nabla f$ with respect to $\\|\\cdot\\|$ on the convex hull of $B(X,1):=X+B(0,1)$ and let $\\bar{\\alpha} := \\min\\{L^{-1},2\\epsilon(6+\\epsilon)^{-1}M^{-1}\\}$. Let $K \\in \\mathbb{N}$, $\\alpha_0,\\hdots,\\alpha_K \\in (0,\\bar{\\alpha}]$, and $(x_0,\\hdots,x_{K+1}) \\in X \\times \\cdots \\times X \\times \\mathbb{R}^n$ be such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for $k=0,\\hdots,K$. It holds that $\\|x_{K+1}-x_K\\| = \\alpha_K\\|\\nabla f(x_K)\\| \\leqslant \\bar{\\alpha}L \\leqslant 1$ and thus $x_0,\\hdots,x_{K+1} \\in B(X,1)$. A bound on the Taylor expansion of $f$ yields\n\\begin{align*}\n    f(x_{k+1})-f(x_k) & \\leqslant \\langle \\nabla f(x_k), x_{k+1}-x_k\\rangle + \\frac{M}{2}\\|x_{k+1}-x_k\\|^2 \\\\\n    & = \\left(\\frac{M\\alpha_k}{2}-1\\right) \\|x_{k+1}-x_k\\| \\|\\nabla f(x_k)\\|\n\\end{align*}\nfor $k=0,\\hdots,K$, and thus\n\\begin{equation}\n\\label{eq:descent_lemma_1}\n    \\|x_{k+1}-x_k\\|\\|\\nabla f(x_k)\\| \\leqslant \\frac{2}{2- M\\alpha_k}(f(x_k) - f(x_{k+1})).\n\\end{equation}\n","start":58559,"end":59567}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":["eq:descent_lemma_2","eq:length_finite"],"declared_external":false,"source_only":false,"title":"相邻梯度估计与统一系数","source_units":[{"raw_tex":"We also have\n    \\begin{align*}\n        \\|\\nabla f(x_{k+1})\\| & \\leqslant \\| \\nabla f(x_{k+1}) - \\nabla f(x_k) \\| + \\|\\nabla f(x_k)\\| \\\\\n        & \\leqslant M \\| x_{k+1} - x_k \\| + \\|\\nabla f(x_k)\\| \\\\\n        & \\leqslant (M\\alpha_k+1) \\|\\nabla f(x_k)\\|\n    \\end{align*}\nand thus \n\\begin{equation}\n\\label{eq:descent_lemma_2}\n    \\|x_{k+1}-x_k\\|\\|\\nabla f(x_{k+1})\\| \\leqslant \\frac{2+2M\\alpha_k}{2- M\\alpha_k}(f(x_k) - f(x_{k+1})).\n\\end{equation}\nNote that \n\\begin{equation*}\n    1 \\leqslant \\frac{2}{2- M\\alpha_k} \\leqslant \\frac{2+2M\\alpha_k}{2- M\\alpha_k} \\leqslant 1+\\frac{\\epsilon}{2}.\n\\end{equation*}\n","start":59567,"end":60174}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["eq:length_finite"],"declared_external":false,"source_only":false,"title":"单分支上的两种单步长度界","source_units":[{"raw_tex":"Let $\\tilde{f}$ and $V$ be defined as in Proposition \\ref{prop:KL}.\n\nAssume that $[f(x_K),f(x_0)]$ excludes the elements of $V$ and the averages of any two consecutive elements of $V$. Then $0\\notin \\partial \\tilde{f}(x_k)$ and $1 \\leqslant \\| \\nabla(\\psi \\circ \\tilde{f})(x_k)\\| = \\psi'(\\tilde{f}(x_k)) \\|\\nabla \\tilde{f}(x_k)\\|$ for $k=0,\\hdots,K$. Let $k \\in \\{0,\\hdots,K-1\\}$. If $\\tilde{f}(x_k) \\geqslant \\tilde{f}(x_{k+1})$, then multiplying \\eqref{eq:descent_lemma_1} by $\\psi'(\\tilde{f}(x_k))$ and using the concavity of $\\psi$, we find that \n\\begin{align*}\n    \\|x_{k+1}-x_k\\| & \\leqslant (1+\\epsilon/2)\\psi'(\\tilde{f}(x_k))(f(x_k) - f(x_{k+1})) \\\\\n    & = (1+\\epsilon/2)\\psi'(\\tilde{f}(x_k))(\\tilde{f}(x_k) - \\tilde{f}(x_{k+1})) \\\\\n    & \\leqslant (1+\\epsilon/2)(\\psi (\\tilde{f}(x_k)) - \\psi (\\tilde{f}(x_{k+1}))).\n\\end{align*}\nIf $\\tilde{f}(x_k) \\leqslant \\tilde{f}(x_{k+1})$, then multiplying \\eqref{eq:descent_lemma_2} by $\\psi'(\\tilde{f}(x_{k+1}))$ and using the concavity of $\\psi$, we find that \n\\begin{align*}\n    \\|x_{k+1}-x_k\\| & \\leqslant (1+\\epsilon/2)\\psi'(\\tilde{f}(x_{k+1}))(f(x_k) - f(x_{k+1})) \\\\\n    & = (1+\\epsilon/2)\\psi'(\\tilde{f}(x_{k+1}))(\\tilde{f}(x_{k+1}) - \\tilde{f}(x_k)) \\\\\n    & \\leqslant (1+\\epsilon/2)(\\psi (\\tilde{f}(x_{k+1})) - \\psi (\\tilde{f}(x_k))).\n\\end{align*}\nAs a result,\n\\begin{equation*}\n    \\|x_{k+1}-x_k\\| \\leqslant (1+\\epsilon/2)|\\psi (\\tilde{f}(x_k)) - \\psi (\\tilde{f}(x_{k+1}))|, ~~~ k = 0,\\hdots,K-1.\n\\end{equation*}\n","start":60174,"end":61647}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":["eq:length_finite"],"declared_external":false,"source_only":false,"title":"望远镜求和并保留末步误差","source_units":[{"raw_tex":"We obtain the telescoping sum \n\\begin{subequations}\n    \\label{eq:length_finite} \n    \\begin{align}\n    \\sum_{k=0}^K \\|x_{k+1}-x_k\\| & \\leqslant (1+\\epsilon/2) |\\psi(\\tilde{f}(x_0)) - \\psi(\\tilde{f}(x_K))| + \\|x_{K+1} - x_K\\| \\\\[-4mm]\n    & \\leqslant (1+\\epsilon/2) \\psi ( | \\tilde{f}(x_0) - \\tilde{f}(x_K) |) + \\alpha_K\\|\\nabla f(x_K)\\| \\\\\n    & \\leqslant (1+\\epsilon/2) \\psi( f(x_0) - f(x_K)) + L\\max_{k=0,\\hdots,K} \\alpha_k.\n    \\end{align}\n\\end{subequations}\n","start":61647,"end":62110}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":["eq:length_finite"],"declared_external":false,"source_only":false,"title":"半开端点的扰动极限","source_units":[{"raw_tex":"\nAssume that $[f(x_K),f(x_0))$ excludes the elements of $V$ and the averages of any two consecutive elements of $V$. Since we are excluding a finite number of elements, there exists $\\epsilon>0$ such that $[f(x_K)-\\epsilon,f(x_0))$ satisfies the same property. If $x_0$ is a local minimum of $f$ on the open set $X$, then $\\nabla f(x_0) = 0$ by Fermat's rule. The length of $x_0,\\hdots,x_{K+1}$ is then equal to zero and the formula obtained in \\eqref{eq:length_finite} trivially holds. Otherwise, there exists a sequence $(x_0^l)_{l\\in\\mathbb{N}}$ in $\\mathbb{R}^n$ such that $f(x_0^l) < f(x_0)$ and $x_0^l$ converges to $x_0$. For every $l \\in \\mathbb{N}$, consider the sequence $(x^l_k)_{k\\in \\mathbb{N}}$ such that $x^l_{k+1} = x^l_k - \\alpha_k \\nabla f(x_k^l)$ for all $k \\in \\mathbb{N}$. Since $\\nabla f$ is continuous, for any fixed $k \\in \\{0,\\hdots,K\\}$, $x_k^l$ converges to $x_k \\in X$ and eventually belongs to the open set $X$. Thus $[f(x^l_K),f(x^l_0)] \\subset [f(x_K)-\\epsilon,f(x_0))$ eventually excludes the elements of $V$ and the averages of any two consecutive elements of $V$. We can apply the formula obtained in \\eqref{eq:length_finite}, namely\n\\begin{equation*}\n    \\sum_{k=0}^K \\|x^l_{k+1}-x^l_k\\| \\leqslant (1+\\epsilon/2) \\psi( f(x^l_0) - f(x^l_K)) + L\\max_{k=0,\\hdots,K} \\alpha_k.\n\\end{equation*}\nPassing to the limit yields\n\\begin{equation*}\n    \\sum_{k=0}^K \\|x_{k+1}-x_k\\| \\leqslant (1+\\epsilon/2) \\psi( f(x_0) - f(x_K)) + L\\max_{k=0,\\hdots,K} \\alpha_k.\n\\end{equation*}\n","start":62110,"end":63610}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":["eq:length_finite"],"declared_external":false,"source_only":false,"title":"离散分段与凹性合并","source_units":[{"raw_tex":"\nWe next consider the general case where\n\\begin{equation*}\n    [f(x_K),f(x_{K_p+1})) \\cup \\cdots \\cup [f(x_{K_2}),f(x_{K_1+1})) \\cup [f(x_{K_1}),f(x_0))\n\\end{equation*}\nexcludes the elements of $V$ and the averages of any two consecutive elements of $V$. For notational convenience, let $K_0 := -1$ and $K_{p+1} := K$. Since $p \\leqslant 2m-1$, we have\n\\begin{align*}\n    \\sum_{k=0}^K \\|x_{k+1}-x_k\\| = & \\sum_{i=0}^p \\sum_{k=K_i+1}^{K_{i+1}} \\|x_{k+1}-x_k\\| \\\\\n    \\leqslant & \\sum_{i=0}^p \\left((1+\\epsilon/2)\\psi( f(x_{{K_i}+1}) - f(x_{K_{i+1}})) + L\\max_{k=0,\\hdots,K} \\alpha_k\\right) \\\\[-.5mm]\n    \\leqslant & (1+\\epsilon/2)(p+1) ~ \\psi \\left( \\frac{1}{p+1} \\sum_{i=0}^p  f(x_{{K_i}+1}) - f(x_{K_{i+1}}) \\right) + \\\\ \n    & (p+1)L\\max_{k=0,\\hdots,K} \\alpha_k \\\\\n    \\leqslant & (1+\\epsilon/2)(p+1) \\psi \\left( \\frac{f(x_0)-f(x_K)}{p+1} \\right) + (p+1)L\\max_{k=0,\\hdots,K} \\alpha_k \\\\\n    \\leqslant & (2+\\epsilon)m ~ \\psi \\left( \\frac{f(x_0)-f(x_K)}{2m} \\right) + 2m L\\max_{k=0,\\hdots,K} \\alpha_k.\\qedhere\n\\end{align*}\n\\end{proof}","start":63610,"end":64644}]},{"id":"S059","semantic_type":"proof","kind":"proof","labels":["eq:decrease_continuous_a","eq:decrease_continuous_b","eq:decrease_continuous_c","eq:decrease_continuous_d","eq:decrease_continuous_e","eq:decrease_continuous_f","eq:f_bound_continuous","eq:finite_time","eq:sup_trajectory","eq:sup_trajectory_T","sec:Proof of Lemma lemma:uniform_length","upper_0","upper_1","upper_2"],"declared_external":false,"source_only":false,"title":"统一连续长度引理的完整证明","source_units":[{"raw_tex":"\\section{Proof of Lemma \\ref{lemma:uniform_length}}\n\\label{sec:Proof of Lemma lemma:uniform_length}\n \nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz definable function with bounded continuous subgradient trajectories. Let $X_0$ be a bounded subset of $\\mathbb{R}^n$. We will show that $\\sigma(X_0) < \\infty$ where \n\\begin{subequations}\n\\label{eq:sup_trajectory}\n\\begin{align}\n    \\sigma(X_0) := & \\sup\\limits_{x \\in \\mathcal{A}(\\mathbb{R}_+,\\mathbb{R}^n)} ~~ \\int_0^{\\infty} \\|x'(t)\\|dt \\\\\n  & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx'(t) \\in - \\partial f(x(t)), ~\\mathrm{for~a.e.}~ t > 0,\\\\[3mm] x(0) \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\nWithout loss of generality, we may assume that $X_0\\neq \\emptyset$. By Propositions \\ref{prop:existence_trajectory} and \\ref{prop:bounded_global}, the feasible set of \\eqref{eq:sup_trajectory} is thus non-empty and $\\sigma(X_0)>-\\infty$.\n\nWe first consider the case with finite time horizon $T\\geqslant 0$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Notice that $\\sigma_T(X_0) \\leqslant \\sqrt{T(\\sup_{X_0} f - m(f))}$ where $m(f)$ is the smallest critical value of $f$ and \n\\begin{subequations}\n\\label{eq:sup_trajectory_T}\n\\begin{align}\n    \\sigma_T(X_0) := & \\sup\\limits_{x \\in \\mathcal{A}(\\mathbb{R}_+,\\mathbb{R}^n)} ~~ \\int_0^T \\|x'(t)\\|dt \\\\\n  & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx'(t) \\in - \\partial f(x(t)), ~\\mathrm{for~a.e.}~ t > 0,\\\\[3mm] x(0) \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\nIndeed, by the Cauchy-Schwarz inequality and Propositions \\ref{prop:chain_rule} and \\ref{prop:length_continuous}, for any feasible point $x(\\cdot)$ of \\eqref{eq:sup_trajectory_T} we have\n\\begin{subequations}\n\\begin{align}\n    \\int_0^T \\|x'(t)\\|dt & \\leqslant \\sqrt{T} \\sqrt{ \\int_0^T \\|x'(t)\\|^2 dt} \\\\\n    & \\leqslant \\sqrt{T} \\sqrt{ \\int_0^{\\infty} \\|x'(t)\\|^2 dt} \\\\\n    & = \\sqrt{T} \\sqrt{f(x(0)) - f\\left(\\lim_{t\\rightarrow\\infty} x(t)\\right)} \\\\\n    & \\leqslant \\sqrt{T\\left(\\sup_{X_0} f - m(f)\\right)}. \\label{eq:finite_time}\n\\end{align}\n\\end{subequations}\n\nWe next treat the case with infinite time horizon. Consider a sequence of feasible points $x_0(\\cdot), x_1(\\cdot), x_2(\\cdot), \\ldots$ of \\eqref{eq:sup_trajectory}  such that $\\int_0^{\\infty} \\|x_k'(t)\\|dt$ converges to $\\sigma(X_0)$. We proceed to show that the sequence is equicontinuous. Let $\\epsilon>0$ and $t\\geqslant 0$. Consider problem \\eqref{eq:sup_trajectory_T} with finite time horizon $T:=t+\\epsilon$. Since $x_0(\\cdot),x_1(\\cdot),x_2(\\cdot),\\ldots$ are feasible points of \\eqref{eq:sup_trajectory_T}, for all $s \\in [0,T]$ and $k=0,1,2,\\ldots$ we have\n\\begin{subequations}\n    \\begin{align}\n        \\|x_k(s)\\| & \\leqslant \\|x_k(s) - x_k(0) \\| + \\|x_k(0)\\| \\label{upper_0} \\\\ & \\leqslant \\int_0^s \\|x_k'(\\tau)\\|d\\tau + \\|x_k(0)\\| \\label{upper_1} \\\\\n        & \\leqslant \\sigma_T(X_0) + \\sup_{x_0 \\in X_0} \\|x_0\\|. \\label{upper_2}\n    \\end{align}\n\\end{subequations}\nAll the trajectories $x_0(\\cdot),x_1(\\cdot),x_2(\\cdot),\\ldots$ hence belong to a common ball centered at the origin up to time $T$. Let $L>0$ be a Lipschitz constant of $f$ on that ball. As a result, for all $s \\in [t-\\delta,t+\\delta]$ with $\\delta:=\\epsilon/(2L)$ and for all $k\\in \\mathbb{N}$, we have\n\\begin{equation*}\n    \\|x_k(t)-x_k(s)\\| = \\left\\| \\int_{s}^t x'_k(\\tau)d\\tau \\right\\| \\leqslant \\int_{t-\\delta}^{t+\\delta} \\|x'_k(\\tau)\\| d\\tau \\leqslant 2\\delta L = \\epsilon,\n\\end{equation*}\nIt follows that $x_0(\\cdot),x_1(\\cdot),x_2(\\cdot),\\ldots$ is equicontinuous on $\\mathbb{R}_+$. In addition, \\eqref{upper_0}-\\eqref{upper_2} imply that, for all $t\\geqslant 0$, the sequence $x_0(t),x_1(t),x_2(t),\\ldots$ is bounded. The Arzel\\`a-Ascoli theorem \\cite[Theorem 1 p. 13]{aubin1984differential} implies that there exists a subsequence (again denoted $x_k(\\cdot)$) converging uniformly over compact intervals to a continuous function $x:\\mathbb{R}_+\\rightarrow\\mathbb{R}^n$. \n\nWe next show that $x(\\cdot)$ is a continuous subgradient trajectory of $f$. Let $T\\geqslant 0$. By virtue of \\eqref{upper_0}-\\eqref{upper_2} and local Lipschitz continuity of $f$, the restrictions of $x'_0(\\cdot),x'_1(\\cdot),x'_2(\\cdot),\\ldots$ to $[0,T]$ lie in a ball of the dual space of $L^1([0,T],\\mathbb{R}^n)$, namely $L^{\\infty}([0,T],\\mathbb{R}^n)$. By the Banach-Alaoglu theorem \\cite[Theorem 3 p. 13]{aubin1984differential}, there exists a subsequence (again denoted $x_k'(\\cdot)$) that converges weakly\\textsuperscript{$\\ast$} to a function $y(\\cdot)$ in $L^{\\infty}([0,T],\\mathbb{R}^n)$. Together with $L^{\\infty}([0,T],\\mathbb{R}^n)\\subset L^1([0,T],\\mathbb{R}^n)$, we find that $x_k'(\\cdot)$ converges weakly to $y(\\cdot)$ in $L^1([0,T],\\mathbb{R}^n)$. Since $x_k(t)-x_k(s) = \\int_s^t x_k'(\\tau)d\\tau$ for all $0\\leqslant s \\leqslant t \\leqslant T$, we have $x(t)-x(s) = \\int_s^t y(\\tau)d\\tau$. As a result, $x'(t) = y(t)$ for almost every $t\\in (0,T)$. According to \\cite[Convergence Theorem p. 60]{aubin1984differential}, it follows that $x'(t) \\in -\\partial f(x(t))$ for almost every $t\\in(0,T)$. As $T \\geqslant 0$ was arbitrary, we have $x'(t) \\in -\\partial f(x(t))$ for almost every $t>0$.\n\n% https://www.math.ucdavis.edu/~hunter/pdes/ch6A.pdf\n\nSince $f$ is definable and has bounded continuous subgradient trajectories, by Proposition \\ref{prop:length_continuous} $x(\\cdot)$ has finite length and converges to a critical point $x^*$ of $f$. Let $\\epsilon>0$ and let $m \\in \\mathbb{N}^*$ be the number of critical values of $f$ in $B(\\overline{x(\\mathbb{R}_+)},\\epsilon)$. Let $\\psi$ be a desingularizing function of $f$ on $B(\\overline{x(\\mathbb{R}_+)},\\epsilon)$. Since $f$ is continuous, there exists $\\delta \\in (0,\\epsilon/2)$ such that \n\\begin{equation}\n\\label{eq:f_bound_continuous}\n    f(x) - f(x^*) \\leqslant m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right),~~~ \\forall x\\in B(x^*,\\delta).\n\\end{equation}\n\nLet $t^*\\geqslant 0$ be such that $\\|x(t)-x^*\\| \\leqslant \\delta/2$ for all $t\\geqslant t^*$. By the uniform convergence of $x_k(\\cdot)$, we have that $\\|x_k(t)-x(t)\\|\\leqslant \\delta/2$ for all $t\\in [0,t^*]$ for all $k$ large enough. Hence $\\|x_k(t^*) - x^*\\|\\leqslant \\|x_k(t^*)-x(t^*)\\|+\\|x(t^*)-x^*\\| \\leqslant \\delta/2 + \\delta/2 = \\delta$. If $T_k := \\inf\\{ t \\geqslant t^* : x_k(t) \\notin \\mathring{B}(x^*,\\epsilon)\\}<\\infty$, then\n\\begin{subequations}\n    \\begin{align}\n        2m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right) & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{\\epsilon-\\delta}{2m}\\right) \\label{eq:decrease_continuous_a} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{\\|x_k(T_k)-x^*\\| - \\|x_k(t^*)-x^*\\|}{2m}\\right) \\label{eq:decrease_continuous_b} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{\\|x_k(T_k)-x_k(t^*)\\|}{2m}\\right) \\label{eq:decrease_continuous_c} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{2m}\\int_{t^*}^{T_k}\\|x_k'(t)\\|dt \\right) \\label{eq:decrease_continuous_d} \\\\\n        & \\leqslant f(x_k(t^*))-f(x_k(T_k)) \\label{eq:decrease_continuous_e} \\\\[3mm]\n        & \\leqslant m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right) + f(x^*) - f(x_k(T_k)). \\label{eq:decrease_continuous_f}\n    \\end{align}\n\\end{subequations}\nAbove, \\eqref{eq:decrease_continuous_a} through \\eqref{eq:decrease_continuous_d} rely on the fact that $\\psi^{-1}$ is increasing. \\eqref{eq:decrease_continuous_a} is due to $\\delta < \\epsilon/2$. \\eqref{eq:decrease_continuous_b} holds because $\\|x_k(T_k)-x^*\\|\\geqslant \\epsilon$ by continuity of $x_k(\\cdot)$ and $x_k(t^*) \\in B(x^*,\\delta)$. \\eqref{eq:decrease_continuous_c} is a consequence of the triangular inequality. \\eqref{eq:decrease_continuous_d} holds because $x_k(\\cdot)$ is absolutely continuous. \\eqref{eq:decrease_continuous_e} is due to Proposition \\ref{prop:length_continuous} and the fact that $x_k(t) \\in B(x^*,\\epsilon)$ for all $t \\in [t^*,T_k]$ by continuity of $x_k(\\cdot)$. Finally, \\eqref{eq:decrease_continuous_f} is due to $x_k(t^*)\\in B(x^*,\\delta)$ and \\eqref{eq:f_bound_continuous}. Since $x_k(t) \\in B(\\overline{x(\\mathbb{R}_+)},\\epsilon)$ for all $t\\in [0,T_k]$ and $x_k(T_k)$ belongs to\n\\begin{equation*}\n     X_1 := B(x^*,\\epsilon) \\bigcap \\left\\{ x \\in \\mathbb{R}^n : f(x) \\leqslant f(x^*) - m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right) \\right\\},\n\\end{equation*}\nby Proposition \\ref{prop:length_continuous} and the definition of $\\sigma(\\cdot)$ in \\eqref{eq:sup_trajectory} we have \n    \\begin{align*}\n            \\int_0^{\\infty} \\|x_k'(t)\\|dt & = \\int_{0}^{T_k} \\|x_k'(t)\\|dt + \\int_{T_k}^{\\infty} \\|x_k'(t)\\|dt \\\\\n            & \\leqslant 2m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(x^*,\\epsilon)} f\\right) \\right) + \\max\\{0,\\sigma(X_1)\\}.\n    \\end{align*}\nNote that the inequality still holds if $T_k = \\infty$. By taking the limit, we get\n\\begin{equation*}\n    \\sigma(X_0) ~\\leqslant~ 2m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(x^*,\\epsilon)} f\\right) \\right) + \\max\\{0,\\sigma(X_1)\\}.\n\\end{equation*}\n\nIt now suffices to replace $X_0$ by $X_1$ and repeat the proof starting below \\eqref{eq:finite_time}. A maximizing sequence $\\bar{x}_k(\\cdot)$ corresponding to $\\sigma(X_1)$ converges to a continuous subgradient trajectory $\\bar{x}(\\cdot)$ whose initial point lies in the compact set $X_1$. If $X_1 \\neq \\emptyset$, then the critical value $f(\\lim_{t\\rightarrow\\infty} \\bar{x}(t))$ is less than or equal to $f(x^*) - m\\psi^{-1}(\\epsilon/(4m)) < f(x^*)$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Thus, it is eventually the case that one of the sets $X_0,X_1,\\hdots$ is empty. We conclude that $\\sigma(X_0)<\\infty$ by the above recursive formula. \n\n","start":72234,"end":81913}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_trajectory","eq:sup_trajectory_T","sec:Proof of Lemma lemma:uniform_length"],"declared_external":false,"source_only":false,"title":"连续长度上确界与可行性","source_units":[{"raw_tex":"\\section{Proof of Lemma \\ref{lemma:uniform_length}}\n\\label{sec:Proof of Lemma lemma:uniform_length}\n \nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz definable function with bounded continuous subgradient trajectories. Let $X_0$ be a bounded subset of $\\mathbb{R}^n$. We will show that $\\sigma(X_0) < \\infty$ where \n\\begin{subequations}\n\\label{eq:sup_trajectory}\n\\begin{align}\n    \\sigma(X_0) := & \\sup\\limits_{x \\in \\mathcal{A}(\\mathbb{R}_+,\\mathbb{R}^n)} ~~ \\int_0^{\\infty} \\|x'(t)\\|dt \\\\\n  & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx'(t) \\in - \\partial f(x(t)), ~\\mathrm{for~a.e.}~ t > 0,\\\\[3mm] x(0) \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\nWithout loss of generality, we may assume that $X_0\\neq \\emptyset$. By Propositions \\ref{prop:existence_trajectory} and \\ref{prop:bounded_global}, the feasible set of \\eqref{eq:sup_trajectory} is thus non-empty and $\\sigma(X_0)>-\\infty$.\n","start":72234,"end":73169}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":["eq:finite_time","eq:sup_trajectory","eq:sup_trajectory_T","sec:Proof of Lemma lemma:uniform_length"],"declared_external":false,"source_only":false,"title":"由最小临界值控制有限时长度","source_units":[{"raw_tex":"We first consider the case with finite time horizon $T\\geqslant 0$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Notice that $\\sigma_T(X_0) \\leqslant \\sqrt{T(\\sup_{X_0} f - m(f))}$ where $m(f)$ is the smallest critical value of $f$ and \n\\begin{subequations}\n\\label{eq:sup_trajectory_T}\n\\begin{align}\n    \\sigma_T(X_0) := & \\sup\\limits_{x \\in \\mathcal{A}(\\mathbb{R}_+,\\mathbb{R}^n)} ~~ \\int_0^T \\|x'(t)\\|dt \\\\\n  & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx'(t) \\in - \\partial f(x(t)), ~\\mathrm{for~a.e.}~ t > 0,\\\\[3mm] x(0) \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\nIndeed, by the Cauchy-Schwarz inequality and Propositions \\ref{prop:chain_rule} and \\ref{prop:length_continuous}, for any feasible point $x(\\cdot)$ of \\eqref{eq:sup_trajectory_T} we have\n\\begin{subequations}\n\\begin{align}\n    \\int_0^T \\|x'(t)\\|dt & \\leqslant \\sqrt{T} \\sqrt{ \\int_0^T \\|x'(t)\\|^2 dt} \\\\\n    & \\leqslant \\sqrt{T} \\sqrt{ \\int_0^{\\infty} \\|x'(t)\\|^2 dt} \\\\\n    & = \\sqrt{T} \\sqrt{f(x(0)) - f\\left(\\lim_{t\\rightarrow\\infty} x(t)\\right)} \\\\\n    & \\leqslant \\sqrt{T\\left(\\sup_{X_0} f - m(f)\\right)}. \\label{eq:finite_time}\n\\end{align}\n\\end{subequations}\n","start":73170,"end":74397}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_trajectory","eq:sup_trajectory_T","sec:Proof of Lemma lemma:uniform_length","upper_0","upper_1","upper_2"],"declared_external":false,"source_only":false,"title":"极大化序列的紧时间紧性","source_units":[{"raw_tex":"We next treat the case with infinite time horizon. Consider a sequence of feasible points $x_0(\\cdot), x_1(\\cdot), x_2(\\cdot), \\ldots$ of \\eqref{eq:sup_trajectory}  such that $\\int_0^{\\infty} \\|x_k'(t)\\|dt$ converges to $\\sigma(X_0)$. We proceed to show that the sequence is equicontinuous. Let $\\epsilon>0$ and $t\\geqslant 0$. Consider problem \\eqref{eq:sup_trajectory_T} with finite time horizon $T:=t+\\epsilon$. Since $x_0(\\cdot),x_1(\\cdot),x_2(\\cdot),\\ldots$ are feasible points of \\eqref{eq:sup_trajectory_T}, for all $s \\in [0,T]$ and $k=0,1,2,\\ldots$ we have\n\\begin{subequations}\n    \\begin{align}\n        \\|x_k(s)\\| & \\leqslant \\|x_k(s) - x_k(0) \\| + \\|x_k(0)\\| \\label{upper_0} \\\\ & \\leqslant \\int_0^s \\|x_k'(\\tau)\\|d\\tau + \\|x_k(0)\\| \\label{upper_1} \\\\\n        & \\leqslant \\sigma_T(X_0) + \\sup_{x_0 \\in X_0} \\|x_0\\|. \\label{upper_2}\n    \\end{align}\n\\end{subequations}\nAll the trajectories $x_0(\\cdot),x_1(\\cdot),x_2(\\cdot),\\ldots$ hence belong to a common ball centered at the origin up to time $T$. Let $L>0$ be a Lipschitz constant of $f$ on that ball. As a result, for all $s \\in [t-\\delta,t+\\delta]$ with $\\delta:=\\epsilon/(2L)$ and for all $k\\in \\mathbb{N}$, we have\n\\begin{equation*}\n    \\|x_k(t)-x_k(s)\\| = \\left\\| \\int_{s}^t x'_k(\\tau)d\\tau \\right\\| \\leqslant \\int_{t-\\delta}^{t+\\delta} \\|x'_k(\\tau)\\| d\\tau \\leqslant 2\\delta L = \\epsilon,\n\\end{equation*}\nIt follows that $x_0(\\cdot),x_1(\\cdot),x_2(\\cdot),\\ldots$ is equicontinuous on $\\mathbb{R}_+$. In addition, \\eqref{upper_0}-\\eqref{upper_2} imply that, for all $t\\geqslant 0$, the sequence $x_0(t),x_1(t),x_2(t),\\ldots$ is bounded. The Arzel\\`a-Ascoli theorem \\cite[Theorem 1 p. 13]{aubin1984differential} implies that there exists a subsequence (again denoted $x_k(\\cdot)$) converging uniformly over compact intervals to a continuous function $x:\\mathbb{R}_+\\rightarrow\\mathbb{R}^n$. \n","start":74398,"end":76256}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_trajectory","eq:sup_trajectory_T","sec:Proof of Lemma lemma:uniform_length"],"declared_external":false,"source_only":false,"title":"紧性极限仍满足微分包含","source_units":[{"raw_tex":"\nWe next show that $x(\\cdot)$ is a continuous subgradient trajectory of $f$. Let $T\\geqslant 0$. By virtue of \\eqref{upper_0}-\\eqref{upper_2} and local Lipschitz continuity of $f$, the restrictions of $x'_0(\\cdot),x'_1(\\cdot),x'_2(\\cdot),\\ldots$ to $[0,T]$ lie in a ball of the dual space of $L^1([0,T],\\mathbb{R}^n)$, namely $L^{\\infty}([0,T],\\mathbb{R}^n)$. By the Banach-Alaoglu theorem \\cite[Theorem 3 p. 13]{aubin1984differential}, there exists a subsequence (again denoted $x_k'(\\cdot)$) that converges weakly\\textsuperscript{$\\ast$} to a function $y(\\cdot)$ in $L^{\\infty}([0,T],\\mathbb{R}^n)$. Together with $L^{\\infty}([0,T],\\mathbb{R}^n)\\subset L^1([0,T],\\mathbb{R}^n)$, we find that $x_k'(\\cdot)$ converges weakly to $y(\\cdot)$ in $L^1([0,T],\\mathbb{R}^n)$. Since $x_k(t)-x_k(s) = \\int_s^t x_k'(\\tau)d\\tau$ for all $0\\leqslant s \\leqslant t \\leqslant T$, we have $x(t)-x(s) = \\int_s^t y(\\tau)d\\tau$. As a result, $x'(t) = y(t)$ for almost every $t\\in (0,T)$. According to \\cite[Convergence Theorem p. 60]{aubin1984differential}, it follows that $x'(t) \\in -\\partial f(x(t))$ for almost every $t\\in(0,T)$. As $T \\geqslant 0$ was arbitrary, we have $x'(t) \\in -\\partial f(x(t))$ for almost every $t>0$.\n\n% https://www.math.ucdavis.edu/~hunter/pdes/ch6A.pdf\n\n","start":76256,"end":77523}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":["eq:f_bound_continuous","eq:sup_trajectory","eq:sup_trajectory_T","sec:Proof of Lemma lemma:uniform_length"],"declared_external":false,"source_only":false,"title":"极限临界点附近的函数值界","source_units":[{"raw_tex":"Since $f$ is definable and has bounded continuous subgradient trajectories, by Proposition \\ref{prop:length_continuous} $x(\\cdot)$ has finite length and converges to a critical point $x^*$ of $f$. Let $\\epsilon>0$ and let $m \\in \\mathbb{N}^*$ be the number of critical values of $f$ in $B(\\overline{x(\\mathbb{R}_+)},\\epsilon)$. Let $\\psi$ be a desingularizing function of $f$ on $B(\\overline{x(\\mathbb{R}_+)},\\epsilon)$. Since $f$ is continuous, there exists $\\delta \\in (0,\\epsilon/2)$ such that \n\\begin{equation}\n\\label{eq:f_bound_continuous}\n    f(x) - f(x^*) \\leqslant m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right),~~~ \\forall x\\in B(x^*,\\delta).\n\\end{equation}\n","start":77523,"end":78187}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":["eq:decrease_continuous_a","eq:decrease_continuous_b","eq:decrease_continuous_c","eq:decrease_continuous_d","eq:decrease_continuous_e","eq:decrease_continuous_f","eq:sup_trajectory","eq:sup_trajectory_T","sec:Proof of Lemma lemma:uniform_length"],"declared_external":false,"source_only":false,"title":"退出临界点邻域必然降值","source_units":[{"raw_tex":"Let $t^*\\geqslant 0$ be such that $\\|x(t)-x^*\\| \\leqslant \\delta/2$ for all $t\\geqslant t^*$. By the uniform convergence of $x_k(\\cdot)$, we have that $\\|x_k(t)-x(t)\\|\\leqslant \\delta/2$ for all $t\\in [0,t^*]$ for all $k$ large enough. Hence $\\|x_k(t^*) - x^*\\|\\leqslant \\|x_k(t^*)-x(t^*)\\|+\\|x(t^*)-x^*\\| \\leqslant \\delta/2 + \\delta/2 = \\delta$. If $T_k := \\inf\\{ t \\geqslant t^* : x_k(t) \\notin \\mathring{B}(x^*,\\epsilon)\\}<\\infty$, then\n\\begin{subequations}\n    \\begin{align}\n        2m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right) & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{\\epsilon-\\delta}{2m}\\right) \\label{eq:decrease_continuous_a} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{\\|x_k(T_k)-x^*\\| - \\|x_k(t^*)-x^*\\|}{2m}\\right) \\label{eq:decrease_continuous_b} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{\\|x_k(T_k)-x_k(t^*)\\|}{2m}\\right) \\label{eq:decrease_continuous_c} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{2m}\\int_{t^*}^{T_k}\\|x_k'(t)\\|dt \\right) \\label{eq:decrease_continuous_d} \\\\\n        & \\leqslant f(x_k(t^*))-f(x_k(T_k)) \\label{eq:decrease_continuous_e} \\\\[3mm]\n        & \\leqslant m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right) + f(x^*) - f(x_k(T_k)). \\label{eq:decrease_continuous_f}\n    \\end{align}\n\\end{subequations}\nAbove, \\eqref{eq:decrease_continuous_a} through \\eqref{eq:decrease_continuous_d} rely on the fact that $\\psi^{-1}$ is increasing. \\eqref{eq:decrease_continuous_a} is due to $\\delta < \\epsilon/2$. \\eqref{eq:decrease_continuous_b} holds because $\\|x_k(T_k)-x^*\\|\\geqslant \\epsilon$ by continuity of $x_k(\\cdot)$ and $x_k(t^*) \\in B(x^*,\\delta)$. \\eqref{eq:decrease_continuous_c} is a consequence of the triangular inequality. \\eqref{eq:decrease_continuous_d} holds because $x_k(\\cdot)$ is absolutely continuous. \\eqref{eq:decrease_continuous_e} is due to Proposition \\ref{prop:length_continuous} and the fact that $x_k(t) \\in B(x^*,\\epsilon)$ for all $t \\in [t^*,T_k]$ by continuity of $x_k(\\cdot)$. Finally, \\eqref{eq:decrease_continuous_f} is due to $x_k(t^*)\\in B(x^*,\\delta)$ and \\eqref{eq:f_bound_continuous}. Since $x_k(t) \\in B(\\overline{x(\\mathbb{R}_+)},\\epsilon)$ for all $t\\in [0,T_k]$ and $x_k(T_k)$ belongs to\n\\begin{equation*}\n     X_1 := B(x^*,\\epsilon) \\bigcap \\left\\{ x \\in \\mathbb{R}^n : f(x) \\leqslant f(x^*) - m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right) \\right\\},\n\\end{equation*}\n","start":78188,"end":80528}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_trajectory","eq:sup_trajectory_T","sec:Proof of Lemma lemma:uniform_length"],"declared_external":false,"source_only":false,"title":"分割长度得到连续递归","source_units":[{"raw_tex":"by Proposition \\ref{prop:length_continuous} and the definition of $\\sigma(\\cdot)$ in \\eqref{eq:sup_trajectory} we have \n    \\begin{align*}\n            \\int_0^{\\infty} \\|x_k'(t)\\|dt & = \\int_{0}^{T_k} \\|x_k'(t)\\|dt + \\int_{T_k}^{\\infty} \\|x_k'(t)\\|dt \\\\\n            & \\leqslant 2m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(x^*,\\epsilon)} f\\right) \\right) + \\max\\{0,\\sigma(X_1)\\}.\n    \\end{align*}\nNote that the inequality still holds if $T_k = \\infty$. By taking the limit, we get\n\\begin{equation*}\n    \\sigma(X_0) ~\\leqslant~ 2m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(x^*,\\epsilon)} f\\right) \\right) + \\max\\{0,\\sigma(X_1)\\}.\n\\end{equation*}\n","start":80528,"end":81194}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_trajectory","eq:sup_trajectory_T","sec:Proof of Lemma lemma:uniform_length"],"declared_external":false,"source_only":false,"title":"有限临界值终止连续递归","source_units":[{"raw_tex":"\nIt now suffices to replace $X_0$ by $X_1$ and repeat the proof starting below \\eqref{eq:finite_time}. A maximizing sequence $\\bar{x}_k(\\cdot)$ corresponding to $\\sigma(X_1)$ converges to a continuous subgradient trajectory $\\bar{x}(\\cdot)$ whose initial point lies in the compact set $X_1$. If $X_1 \\neq \\emptyset$, then the critical value $f(\\lim_{t\\rightarrow\\infty} \\bar{x}(t))$ is less than or equal to $f(x^*) - m\\psi^{-1}(\\epsilon/(4m)) < f(x^*)$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Thus, it is eventually the case that one of the sets $X_0,X_1,\\hdots$ is empty. We conclude that $\\sigma(X_0)<\\infty$ by the above recursive formula. \n\n","start":81194,"end":81913}]},{"id":"S068","semantic_type":"proof","kind":"proof","labels":["eq:decrease_discrete_a","eq:decrease_discrete_b","eq:decrease_discrete_c","eq:decrease_discrete_d","eq:decrease_discrete_e","eq:decrease_discrete_f","eq:decrease_discrete_g","eq:f_bound_discrete","eq:length_discrete_K","eq:sup_discrete_X_0","eq:tracking_T","sec:Proof of Theorem thm:convergence"],"declared_external":false,"source_only":false,"title":"三项等价的完整主证明","source_units":[{"raw_tex":"\\section{Proof of Theorem \\ref{thm:convergence}}\n\\label{sec:Proof of Theorem thm:convergence}\n\n($1\\Longrightarrow 2$) We prove the contrapositive. Assume that there exists $0< T \\leqslant \\infty$ and a differentiable function $x:[0,T)\\rightarrow\\mathbb{R}^n$ such that\n\\begin{equation*}\n    x'(t) = -\\nabla f(x(t)), ~~~ \\forall t \\in (0,T),\n\\end{equation*}\nfor which, for all $c>0$, there exists $t\\in [0,T)$ such that $\\|x(t)\\| > c$. Let $x_0 := x(0)$. Let $\\bar{\\alpha}$ and $c$ be some positive constants. Let $\\tilde{t} \\in (0,T)$ be such that $\\|x(\\tilde{t})\\| \\geqslant c+2$. By Proposition \\ref{prop:tracking} and Remark \\ref{rem:tracking}, there exists $\\tilde{\\alpha}\\in (0,\\bar{\\alpha}]$ such that for all $\\alpha \\in (0,\\tilde{\\alpha}]$, the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by $x_{k+1} = x_k -\\alpha \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$ satisfies $\\|x_k-x(k\\alpha)\\| \\leqslant 1/2$ for all $k\\in \\{0,\\hdots, \\lfloor \\tilde{t}/\\alpha \\rfloor \\}$. We use the notation $\\lfloor t \\rfloor$ to denote the floor of a real number $t$ which is the unique integer such that $\\lfloor t \\rfloor \\leqslant t < \\lfloor t \\rfloor + 1$. If $\\tilde{k} = \\lfloor \\tilde{t}/\\alpha \\rfloor$, then $\\|x_{\\tilde{k}} - x(\\tilde{t})\\| \\leqslant \\|x_{\\tilde{k}}-x(\\tilde{k}\\alpha)\\|+\\|x(\\tilde{k}\\alpha)-x(\\tilde{t})\\| \\leqslant 1/2 + \\tilde{L}| \\tilde{k}\\alpha-\\tilde{t}| \\leqslant 1/2 + \\tilde{L}\\alpha$ where $\\tilde{L}>0$ is a Lipschitz constant of $f$ on the convex hull of $x([\\tilde{t}-\\tilde{\\alpha},\\tilde{t}])$. Let $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$ be the constant sequence equal to $\\min \\{\\tilde{\\alpha}, 1/(2\\tilde{L})\\}$ and consider the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by $x_{k+1} = x_k -\\alpha_k \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$. By the triangular inequality, we have $\\|x_{\\tilde{k}}\\| \\geqslant \\|x(\\tilde{t})\\| - \\|x_{\\tilde{k}}-x(\\tilde{t})\\| \\geqslant c+2 - 1/2 - 1/2>c$ where $\\tilde{k} := \\lfloor \\tilde{t}/\\alpha \\rfloor$.\n\n($2 \\Longrightarrow 3$) Let $X_0$ be a bounded subset of $\\mathbb{R}^n$. We will show that there exists $\\bar{\\alpha}>0$ such that $\\sigma(X_0,\\bar{\\alpha})<\\infty$ where\n\\begin{subequations}\n\\label{eq:sup_discrete_X_0}\n\\begin{align}\n    \\sigma(X_0,\\bar{\\alpha}) := & \\sup\\limits_{\\tiny \\begin{array}{c}x \\in (\\mathbb{R}^n)^{\\mathbb{N}}\\\\ \\alpha \\in (0,\\bar{\\alpha}]^\\mathbb{N} \\end{array}} ~~ \\sum\\limits_{k=0}^{\\infty} \\|x_{k+1}-x_k\\| \\\\\n   & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx_{k+1} = x_k - \\alpha_k \\nabla f(x_k), ~ \\forall k\\in \\mathbb{N},\\\\[2mm] x_0 \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\n\nLet $\\Phi:\\mathbb{R}_+ \\times \\mathbb{R}^n \\rightarrow \\mathbb{R}^n$ be the gradient flow of $f$ defined for all $(t,x_0) \\in \\mathbb{R}_+ \\times \\mathbb{R}^n$ by $\\Phi(t,x_0) := x(t)$ where $x(\\cdot)$ is the unique continuous gradient trajectory of $f$ initialized at $x_0$. Uniqueness follows from the Picard–Lindel{\\\"o}f theorem \\cite[Theorem 3.1 p. 12]{coddington1955theory}. Let $C$ be the set of critical points of $f$ in $\\overline{\\Phi(\\mathbb{R}_+,X_0)}$. If $X_0 \\subset C$, then we have $\\sigma(X_0,\\bar{\\alpha}) \\leqslant 0$ for all $\\bar{\\alpha}>0$ and there is nothing left to prove. Otherwise, since $C$ is closed by \\cite[2.1.5 Proposition p. 29]{clarke1990}, there exists $\\epsilon>0$ be such that $X_0\\setminus \\mathring{B}(C,\\epsilon/6) \\neq \\emptyset$ where $\\mathring{B}(C,\\epsilon/6) := C + \\mathring{B}(0,\\epsilon/6)$. It is safe to assume this from now on.\n\nLet $\\psi$ be a desingularizing function of $f$ on $\\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$, where we use the shorthand $\\Phi_0 := \\Phi(\\mathbb{R}_+,X_0)$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Let $m \\in \\mathbb{N}$ be the number of critical values of $f$ in $B(\\overline{\\Phi}_0,\\epsilon)$. Note that $m \\geqslant 1$ due to Proposition \\ref{prop:length_continuous} and the fact that $X_0 \\neq \\emptyset$. Since $f$ is continuous and $C$ is compact by Lemma \\ref{lemma:uniform_length}, there exists $\\delta \\in (0,\\epsilon/2)$ such that  \n\\begin{equation}\n    \\label{eq:f_bound_discrete}\n    f(x) - \\max_{C} f \\leqslant m~ \\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right),~~~ \\forall x\\in B(C,\\delta).\n\\end{equation}\nNote that $\\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$ is bounded due to Lemma \\ref{lemma:uniform_length}. Let $L>0$ be a Lipschitz constant of $f$ on the convex hull of $\\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$. By Proposition \\ref{prop:length_discrete}, there exists $\\alpha_\\delta \\in (0,\\delta/(5mL)]$ such that, for all $K \\in \\mathbb{N}^*$, $\\alpha_0,\\hdots,\\alpha_{K-1} \\in (0,\\alpha_\\delta]$, and $(x_0,\\hdots,x_K) \\in \\mathring{B}(\\overline{\\Phi}_0,\\epsilon) \\times \\cdots \\times \\mathring{B}(\\overline{\\Phi}_0,\\epsilon) \\times \\mathbb{R}^n$ such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for $k = 0,\\hdots,K-1$, we have\n\\begin{equation}\n\\label{eq:length_discrete_K}\n    \\frac{1}{4m}\\sum_{k=0}^{K-1}\\|x_{k+1}-x_k\\| ~\\leqslant~ \\psi\\left(\\frac{f(x_0)-f(x_{K-1})}{2m}\\right) + \\frac{L}{2}\\max_{k=0,\\hdots,K-1} \\alpha_k.\n\\end{equation}\n\nSince $\\nabla f$ is continuous, its norm attains its infimum $\\nu$ on the nonempty compact set $\\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3)$. \nIt is nonempty because $\\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3) \\supset X_0\\setminus \\mathring{B}(C,\\epsilon/6) \\neq \\emptyset$ and $\\delta < \\epsilon/2$.\nIf $\\nu=0$, then there exists $x^* \\in \\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3)$ such that $\\|\\nabla f(x^*)\\| = 0$. Then $x^*\\in C \\setminus \\mathring{B}(C,\\delta/3)$, which is a contradiction. We thus have $\\nu>0$. Hence we may define $T := 2\\sigma(X_0)/\\nu$ where $\\sigma(X_0)$ is defined in \\eqref{eq:sup_trajectory} and is finite by Lemma \\ref{lemma:uniform_length}. Note that $\\sigma(X_0)>0$ and thus $T>0$ because $X_0\\not\\subset C$. By Proposition \\ref{prop:tracking}, there exists $\\bar{\\alpha} \\in (0,\\alpha_\\delta]$ such that for any feasible point $((x_k)_{k\\in \\mathbb{N}} , (\\alpha_k)_{k\\in\\mathbb{N}})$ of \\eqref{eq:sup_discrete_X_0},\nthe continuous gradient trajectory $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ of $f$ initialized at $x_0$ satisfies\n\\begin{equation}\n\\label{eq:tracking_T}\n   \\forall k\\in \\mathbb{N}^*,~~~ \\alpha_0+\\cdots+\\alpha_{k-1} \\leqslant T ~~~\\Longrightarrow~~~ \\|x_k - x(\\alpha_0+\\cdots+\\alpha_{k-1})\\| \\leqslant \\frac{\\delta}{3}.\n\\end{equation}\nNow suppose that $\\|x'(t)\\| \\geqslant 2\\sigma(X_0)/T$ for all $t\\in (0,T)$. Then we obtain the following contradiction\n\\begin{equation*}\n     \\sigma(X_0) < T \\frac{2\\sigma(X_0)}{T} \\leqslant \\int_0^{T} \\|x'(t)\\|dt \\leqslant \\int_0^{\\infty} \\|x'(t)\\|dt \\leqslant \\sigma(X_0).\n\\end{equation*}\nHence, there exists $t^* \\in (0,T)$ such that $\\|x'(t^*)\\| = \\|\\nabla f(x(t^*))\\| < 2\\sigma(X_0)/T = 2\\sigma(X_0)/(2\\sigma(X_0)/\\nu) = \\nu$. Since  $x(t^*) \\in \\Phi(\\mathbb{R}_+,X_0) \\subset \\overline{\\Phi}_0$ and the infimum of the norm of $\\nabla f$ on $\\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3)$ is equal to $\\nu$, it must be that $x(t^*) \\in \\mathring{B}(C,\\delta/3)$. Hence there exists $x^* \\in C$ such that $\\|x(t^*)-x^*\\|\\leqslant \\delta/3$. \n\nIf $\\sum_{k=0}^{\\infty} \\alpha_k \\leqslant T$, then by \\eqref{eq:tracking_T} we have $x_k \\in B(\\overline{\\Phi}_0,\\delta/3)$ for all $k \\in \\mathbb{N}$. Otherwise, since $\\alpha_k \\leqslant \\bar{\\alpha} \\leqslant \\alpha_\\delta \\leqslant \\delta/(5mL) \\leqslant \\delta/(3L)$ for all $k \\in \\mathbb{N}$, there exists $k^* \\in \\mathbb{N}$ such that $t_{k^*} := \\sum_{k=0}^{k^*-1} \\alpha_k \\in [ t^* - \\delta/(3L) , t^*]$, where $\\sum_{k=0}^{k^*-1} \\alpha_k = 0$ if $k^* = 0$. Thus $\\|x_{k^*}-x^*\\| \\leqslant \\|x_{k^*}-x(t_{k^*})\\| + \\|x(t_{k^*}) - x(t^*)\\| + \\|x(t^*)-x^*\\| \\leqslant \\delta/3 + L | t_{k^*}-t^* | + \\delta/3 \\leqslant \\delta/3+\\delta/3+\\delta/3 = \\delta$.\nIf $K := \\inf\\{ k \\geqslant k^* : x_k \\notin \\mathring{B}(C,\\epsilon)\\}<\\infty$, then\n\\begin{subequations}\n    \\begin{align}\n        2m~\\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right) & = 2m~ \\psi^{-1}\\left(\\frac{1}{8m} \\epsilon - \\frac{L}{2}\\frac{\\epsilon}{5mL} \\right) \\label{eq:decrease_discrete_a} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m}(\\epsilon-\\delta) - \\frac{L\\alpha_\\delta}{2} \\right) \\label{eq:decrease_discrete_b} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m} \\left(\\|x_K-x^*\\|-\\|x_{k^*}-x^*\\|\\right) - \\frac{L\\alpha_\\delta}{2} \\right) \\label{eq:decrease_discrete_c} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m} \\|x_K-x_{k^*}\\| - \\frac{L\\bar{\\alpha}}{2} \\right) \\label{eq:decrease_discrete_d} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m}\\sum_{k=k^*}^{K-1} \\|x_{k+1}-x_k\\| - \\frac{L}{2}\\max_{k=0,\\hdots,K-1} \\alpha_k \\right) \\label{eq:decrease_discrete_e} \\\\\n        & \\leqslant f(x_{k^*}) - f(x_{K-1}) \\label{eq:decrease_discrete_f} \\\\\n        & \\leqslant m~\\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right) + \\max_C f - f(x_{K-1}). \\label{eq:decrease_discrete_g}\n    \\end{align}\n\\end{subequations}\nAbove, the arguments of $\\psi^{-1}$ in \\eqref{eq:decrease_discrete_a} are equal. \\eqref{eq:decrease_discrete_b} through \\eqref{eq:decrease_discrete_e} rely on the fact that $\\psi^{-1}$ is an increasing function. \\eqref{eq:decrease_discrete_b} is due to $\\delta<\\epsilon/2$. \\eqref{eq:decrease_discrete_c} holds because $x_K \\notin \\mathring{B}(C,\\epsilon)$, $x^* \\in C$, and $x_{k^*} \\in B(x^*,\\delta)$. \\eqref{eq:decrease_discrete_d} and \\eqref{eq:decrease_discrete_e} are consequences of the triangular inequality. \\eqref{eq:decrease_discrete_f} is due to the length formula \\eqref{eq:length_discrete_K} and the fact that $x_{k^*},\\hdots,x_{K-1} \\in \\mathring{B}(C,\\epsilon) \\subset \\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$. Finally, \\eqref{eq:decrease_discrete_g} is due to $x_{k^*}\\in B(C,\\delta)$ and \\eqref{eq:f_bound_discrete}. We remark that $K\\geqslant k^*+2$ since $\\|x_{k^*+1}-x^*\\| \\leqslant \\|x_{k^*+1}-x_{k^*}\\| +\\|x_{k^*}-x^*\\| \\leqslant \\alpha_{k^*}\\|\\nabla f(x_{k^*})\\|+\\delta \\leqslant (\\delta/(5mL))L+\\delta < \\epsilon$. Since $x_0,\\hdots,x_{K-2} \\in \\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$ and $x_{K-1}$ belongs to \n\\begin{equation*}\n    X_1 := B(C,\\epsilon) \\bigcap \\left\\{ x \\in \\mathbb{R}^n : f(x) \\leqslant \\max_{C} f - m~\\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right) \\right\\},    \n\\end{equation*}\nby the length formula \\eqref{eq:length_discrete_K} and the definition of $\\sigma(\\cdot,\\cdot)$ in \\eqref{eq:sup_discrete_X_0} we have\n    \\begin{align*}\n            \\sum_{k=0}^\\infty \\|x_{k+1}-x_k\\| & = \\sum_{k=0}^{K-2} \\|x_{k+1}-x_k\\| + \\sum_{k=K-1}^\\infty \\|x_{k+1}-x_k\\| \\\\\n            & \\leqslant 4m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(\\overline{\\Phi}_0,\\epsilon)} f\\right) \\right) + 2mL\\bar{\\alpha} + \\max\\{0,\\sigma(X_1,\\bar{\\alpha})\\}.\n    \\end{align*}\nNote that the inequality still holds if $K = \\infty$ or $\\sum_{k=0}^{\\infty} \\alpha_k \\leqslant T$. Hence\n\\begin{equation*}\n    \\sigma(X_0,\\bar{\\alpha}) ~\\leqslant~ 4m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(\\overline{\\Phi}_0,\\epsilon)} f\\right) \\right) + 2mL\\bar{\\alpha} + \\max\\{0,\\sigma(X_1,\\bar{\\alpha})\\}.\n\\end{equation*}\n\nIt now suffices to replace $X_0$ by $X_1$ and repeat the proof starting below \\eqref{eq:sup_discrete_X_0}. Since $f(\\Phi(t,x_1)) \\leqslant f(\\Phi(0,x_1)) \\leqslant \\max_{C} f - m\\psi^{-1}(\\epsilon/(40m))< \\max_{C} f$ for all $t\\geqslant 0$ and $x_1\\in X_1$, the maximal critical value of $f$ in $\\overline{\\Phi(\\mathbb{R}_+,X_1)}$ is less than the maximal critical value of $f$ in $\\overline{\\Phi(\\mathbb{R}_+,X_0)}$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Thus, it is eventually the case that one of the sets $X_0,X_1,\\hdots$ is empty. In order to conclude, one simply needs to choose an upper bound on the step sizes $\\hat{\\alpha}$ corresponding to $X_1$ that is less than or equal to the upper bound $\\bar{\\alpha}$ used for $X_0$. $\\sigma(X_0,\\cdot)$ is finite when evaluated at the last upper bound thus obtained. Indeed, the above recursive formula still holds if we replace $\\bar{\\alpha}$ by any $\\alpha \\in (0,\\bar{\\alpha}]$. In particular, we may take $\\alpha := \\hat{\\alpha}$.\n\n($3 \\Longrightarrow 1$) Obvious.\\\\\n\n","start":81913,"end":94199}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_discrete_X_0","sec:Proof of Theorem thm:convergence"],"declared_external":false,"source_only":false,"title":"用有限时跟踪证明逆否命题","source_units":[{"raw_tex":"($1\\Longrightarrow 2$) We prove the contrapositive. Assume that there exists $0< T \\leqslant \\infty$ and a differentiable function $x:[0,T)\\rightarrow\\mathbb{R}^n$ such that\n\\begin{equation*}\n    x'(t) = -\\nabla f(x(t)), ~~~ \\forall t \\in (0,T),\n\\end{equation*}\nfor which, for all $c>0$, there exists $t\\in [0,T)$ such that $\\|x(t)\\| > c$. Let $x_0 := x(0)$. Let $\\bar{\\alpha}$ and $c$ be some positive constants. Let $\\tilde{t} \\in (0,T)$ be such that $\\|x(\\tilde{t})\\| \\geqslant c+2$. By Proposition \\ref{prop:tracking} and Remark \\ref{rem:tracking}, there exists $\\tilde{\\alpha}\\in (0,\\bar{\\alpha}]$ such that for all $\\alpha \\in (0,\\tilde{\\alpha}]$, the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by $x_{k+1} = x_k -\\alpha \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$ satisfies $\\|x_k-x(k\\alpha)\\| \\leqslant 1/2$ for all $k\\in \\{0,\\hdots, \\lfloor \\tilde{t}/\\alpha \\rfloor \\}$. We use the notation $\\lfloor t \\rfloor$ to denote the floor of a real number $t$ which is the unique integer such that $\\lfloor t \\rfloor \\leqslant t < \\lfloor t \\rfloor + 1$. If $\\tilde{k} = \\lfloor \\tilde{t}/\\alpha \\rfloor$, then $\\|x_{\\tilde{k}} - x(\\tilde{t})\\| \\leqslant \\|x_{\\tilde{k}}-x(\\tilde{k}\\alpha)\\|+\\|x(\\tilde{k}\\alpha)-x(\\tilde{t})\\| \\leqslant 1/2 + \\tilde{L}| \\tilde{k}\\alpha-\\tilde{t}| \\leqslant 1/2 + \\tilde{L}\\alpha$ where $\\tilde{L}>0$ is a Lipschitz constant of $f$ on the convex hull of $x([\\tilde{t}-\\tilde{\\alpha},\\tilde{t}])$. Let $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$ be the constant sequence equal to $\\min \\{\\tilde{\\alpha}, 1/(2\\tilde{L})\\}$ and consider the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by $x_{k+1} = x_k -\\alpha_k \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$. By the triangular inequality, we have $\\|x_{\\tilde{k}}\\| \\geqslant \\|x(\\tilde{t})\\| - \\|x_{\\tilde{k}}-x(\\tilde{t})\\| \\geqslant c+2 - 1/2 - 1/2>c$ where $\\tilde{k} := \\lfloor \\tilde{t}/\\alpha \\rfloor$.\n","start":82008,"end":83931}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_discrete_X_0","sec:Proof of Theorem thm:convergence"],"declared_external":false,"source_only":false,"title":"离散长度上确界与梯度流","source_units":[{"raw_tex":"($2 \\Longrightarrow 3$) Let $X_0$ be a bounded subset of $\\mathbb{R}^n$. We will show that there exists $\\bar{\\alpha}>0$ such that $\\sigma(X_0,\\bar{\\alpha})<\\infty$ where\n\\begin{subequations}\n\\label{eq:sup_discrete_X_0}\n\\begin{align}\n    \\sigma(X_0,\\bar{\\alpha}) := & \\sup\\limits_{\\tiny \\begin{array}{c}x \\in (\\mathbb{R}^n)^{\\mathbb{N}}\\\\ \\alpha \\in (0,\\bar{\\alpha}]^\\mathbb{N} \\end{array}} ~~ \\sum\\limits_{k=0}^{\\infty} \\|x_{k+1}-x_k\\| \\\\\n   & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx_{k+1} = x_k - \\alpha_k \\nabla f(x_k), ~ \\forall k\\in \\mathbb{N},\\\\[2mm] x_0 \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\n\nLet $\\Phi:\\mathbb{R}_+ \\times \\mathbb{R}^n \\rightarrow \\mathbb{R}^n$ be the gradient flow of $f$ defined for all $(t,x_0) \\in \\mathbb{R}_+ \\times \\mathbb{R}^n$ by $\\Phi(t,x_0) := x(t)$ where $x(\\cdot)$ is the unique continuous gradient trajectory of $f$ initialized at $x_0$. Uniqueness follows from the Picard–Lindel{\\\"o}f theorem \\cite[Theorem 3.1 p. 12]{coddington1955theory}. Let $C$ be the set of critical points of $f$ in $\\overline{\\Phi(\\mathbb{R}_+,X_0)}$. If $X_0 \\subset C$, then we have $\\sigma(X_0,\\bar{\\alpha}) \\leqslant 0$ for all $\\bar{\\alpha}>0$ and there is nothing left to prove. Otherwise, since $C$ is closed by \\cite[2.1.5 Proposition p. 29]{clarke1990}, there exists $\\epsilon>0$ be such that $X_0\\setminus \\mathring{B}(C,\\epsilon/6) \\neq \\emptyset$ where $\\mathring{B}(C,\\epsilon/6) := C + \\mathring{B}(0,\\epsilon/6)$. It is safe to assume this from now on.\n","start":83932,"end":85451}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":["eq:f_bound_discrete","eq:length_discrete_K","eq:sup_discrete_X_0","sec:Proof of Theorem thm:convergence"],"declared_external":false,"source_only":false,"title":"公共流邻域与离散局部长度界","source_units":[{"raw_tex":"Let $\\psi$ be a desingularizing function of $f$ on $\\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$, where we use the shorthand $\\Phi_0 := \\Phi(\\mathbb{R}_+,X_0)$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Let $m \\in \\mathbb{N}$ be the number of critical values of $f$ in $B(\\overline{\\Phi}_0,\\epsilon)$. Note that $m \\geqslant 1$ due to Proposition \\ref{prop:length_continuous} and the fact that $X_0 \\neq \\emptyset$. Since $f$ is continuous and $C$ is compact by Lemma \\ref{lemma:uniform_length}, there exists $\\delta \\in (0,\\epsilon/2)$ such that  \n\\begin{equation}\n    \\label{eq:f_bound_discrete}\n    f(x) - \\max_{C} f \\leqslant m~ \\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right),~~~ \\forall x\\in B(C,\\delta).\n\\end{equation}\nNote that $\\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$ is bounded due to Lemma \\ref{lemma:uniform_length}. Let $L>0$ be a Lipschitz constant of $f$ on the convex hull of $\\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$. By Proposition \\ref{prop:length_discrete}, there exists $\\alpha_\\delta \\in (0,\\delta/(5mL)]$ such that, for all $K \\in \\mathbb{N}^*$, $\\alpha_0,\\hdots,\\alpha_{K-1} \\in (0,\\alpha_\\delta]$, and $(x_0,\\hdots,x_K) \\in \\mathring{B}(\\overline{\\Phi}_0,\\epsilon) \\times \\cdots \\times \\mathring{B}(\\overline{\\Phi}_0,\\epsilon) \\times \\mathbb{R}^n$ such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for $k = 0,\\hdots,K-1$, we have\n\\begin{equation}\n\\label{eq:length_discrete_K}\n    \\frac{1}{4m}\\sum_{k=0}^{K-1}\\|x_{k+1}-x_k\\| ~\\leqslant~ \\psi\\left(\\frac{f(x_0)-f(x_{K-1})}{2m}\\right) + \\frac{L}{2}\\max_{k=0,\\hdots,K-1} \\alpha_k.\n\\end{equation}\n","start":85452,"end":87085}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_discrete_X_0","eq:tracking_T","sec:Proof of Theorem thm:convergence"],"declared_external":false,"source_only":false,"title":"正梯度下界与统一跟踪时间","source_units":[{"raw_tex":"Since $\\nabla f$ is continuous, its norm attains its infimum $\\nu$ on the nonempty compact set $\\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3)$. \nIt is nonempty because $\\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3) \\supset X_0\\setminus \\mathring{B}(C,\\epsilon/6) \\neq \\emptyset$ and $\\delta < \\epsilon/2$.\nIf $\\nu=0$, then there exists $x^* \\in \\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3)$ such that $\\|\\nabla f(x^*)\\| = 0$. Then $x^*\\in C \\setminus \\mathring{B}(C,\\delta/3)$, which is a contradiction. We thus have $\\nu>0$. Hence we may define $T := 2\\sigma(X_0)/\\nu$ where $\\sigma(X_0)$ is defined in \\eqref{eq:sup_trajectory} and is finite by Lemma \\ref{lemma:uniform_length}. Note that $\\sigma(X_0)>0$ and thus $T>0$ because $X_0\\not\\subset C$. By Proposition \\ref{prop:tracking}, there exists $\\bar{\\alpha} \\in (0,\\alpha_\\delta]$ such that for any feasible point $((x_k)_{k\\in \\mathbb{N}} , (\\alpha_k)_{k\\in\\mathbb{N}})$ of \\eqref{eq:sup_discrete_X_0},\nthe continuous gradient trajectory $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ of $f$ initialized at $x_0$ satisfies\n\\begin{equation}\n\\label{eq:tracking_T}\n   \\forall k\\in \\mathbb{N}^*,~~~ \\alpha_0+\\cdots+\\alpha_{k-1} \\leqslant T ~~~\\Longrightarrow~~~ \\|x_k - x(\\alpha_0+\\cdots+\\alpha_{k-1})\\| \\leqslant \\frac{\\delta}{3}.\n\\end{equation}\n","start":87086,"end":88401}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_discrete_X_0","sec:Proof of Theorem thm:convergence"],"declared_external":false,"source_only":false,"title":"有限时间内接近临界集","source_units":[{"raw_tex":"Now suppose that $\\|x'(t)\\| \\geqslant 2\\sigma(X_0)/T$ for all $t\\in (0,T)$. Then we obtain the following contradiction\n\\begin{equation*}\n     \\sigma(X_0) < T \\frac{2\\sigma(X_0)}{T} \\leqslant \\int_0^{T} \\|x'(t)\\|dt \\leqslant \\int_0^{\\infty} \\|x'(t)\\|dt \\leqslant \\sigma(X_0).\n\\end{equation*}\nHence, there exists $t^* \\in (0,T)$ such that $\\|x'(t^*)\\| = \\|\\nabla f(x(t^*))\\| < 2\\sigma(X_0)/T = 2\\sigma(X_0)/(2\\sigma(X_0)/\\nu) = \\nu$. Since  $x(t^*) \\in \\Phi(\\mathbb{R}_+,X_0) \\subset \\overline{\\Phi}_0$ and the infimum of the norm of $\\nabla f$ on $\\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3)$ is equal to $\\nu$, it must be that $x(t^*) \\in \\mathring{B}(C,\\delta/3)$. Hence there exists $x^* \\in C$ such that $\\|x(t^*)-x^*\\|\\leqslant \\delta/3$. \n","start":88401,"end":89154}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_discrete_X_0","sec:Proof of Theorem thm:convergence"],"declared_external":false,"source_only":false,"title":"小步长使离散点进入临界邻域","source_units":[{"raw_tex":"If $\\sum_{k=0}^{\\infty} \\alpha_k \\leqslant T$, then by \\eqref{eq:tracking_T} we have $x_k \\in B(\\overline{\\Phi}_0,\\delta/3)$ for all $k \\in \\mathbb{N}$. Otherwise, since $\\alpha_k \\leqslant \\bar{\\alpha} \\leqslant \\alpha_\\delta \\leqslant \\delta/(5mL) \\leqslant \\delta/(3L)$ for all $k \\in \\mathbb{N}$, there exists $k^* \\in \\mathbb{N}$ such that $t_{k^*} := \\sum_{k=0}^{k^*-1} \\alpha_k \\in [ t^* - \\delta/(3L) , t^*]$, where $\\sum_{k=0}^{k^*-1} \\alpha_k = 0$ if $k^* = 0$. Thus $\\|x_{k^*}-x^*\\| \\leqslant \\|x_{k^*}-x(t_{k^*})\\| + \\|x(t_{k^*}) - x(t^*)\\| + \\|x(t^*)-x^*\\| \\leqslant \\delta/3 + L | t_{k^*}-t^* | + \\delta/3 \\leqslant \\delta/3+\\delta/3+\\delta/3 = \\delta$.\nIf $K := \\inf\\{ k \\geqslant k^* : x_k \\notin \\mathring{B}(C,\\epsilon)\\}<\\infty$, then\n","start":89155,"end":89909}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":["eq:decrease_discrete_a","eq:decrease_discrete_b","eq:decrease_discrete_c","eq:decrease_discrete_d","eq:decrease_discrete_e","eq:decrease_discrete_f","eq:decrease_discrete_g","eq:sup_discrete_X_0","sec:Proof of Theorem thm:convergence"],"declared_external":false,"source_only":false,"title":"退出临界邻域迫使严格下降","source_units":[{"raw_tex":"\\begin{subequations}\n    \\begin{align}\n        2m~\\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right) & = 2m~ \\psi^{-1}\\left(\\frac{1}{8m} \\epsilon - \\frac{L}{2}\\frac{\\epsilon}{5mL} \\right) \\label{eq:decrease_discrete_a} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m}(\\epsilon-\\delta) - \\frac{L\\alpha_\\delta}{2} \\right) \\label{eq:decrease_discrete_b} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m} \\left(\\|x_K-x^*\\|-\\|x_{k^*}-x^*\\|\\right) - \\frac{L\\alpha_\\delta}{2} \\right) \\label{eq:decrease_discrete_c} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m} \\|x_K-x_{k^*}\\| - \\frac{L\\bar{\\alpha}}{2} \\right) \\label{eq:decrease_discrete_d} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m}\\sum_{k=k^*}^{K-1} \\|x_{k+1}-x_k\\| - \\frac{L}{2}\\max_{k=0,\\hdots,K-1} \\alpha_k \\right) \\label{eq:decrease_discrete_e} \\\\\n        & \\leqslant f(x_{k^*}) - f(x_{K-1}) \\label{eq:decrease_discrete_f} \\\\\n        & \\leqslant m~\\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right) + \\max_C f - f(x_{K-1}). \\label{eq:decrease_discrete_g}\n    \\end{align}\n\\end{subequations}\nAbove, the arguments of $\\psi^{-1}$ in \\eqref{eq:decrease_discrete_a} are equal. \\eqref{eq:decrease_discrete_b} through \\eqref{eq:decrease_discrete_e} rely on the fact that $\\psi^{-1}$ is an increasing function. \\eqref{eq:decrease_discrete_b} is due to $\\delta<\\epsilon/2$. \\eqref{eq:decrease_discrete_c} holds because $x_K \\notin \\mathring{B}(C,\\epsilon)$, $x^* \\in C$, and $x_{k^*} \\in B(x^*,\\delta)$. \\eqref{eq:decrease_discrete_d} and \\eqref{eq:decrease_discrete_e} are consequences of the triangular inequality. \\eqref{eq:decrease_discrete_f} is due to the length formula \\eqref{eq:length_discrete_K} and the fact that $x_{k^*},\\hdots,x_{K-1} \\in \\mathring{B}(C,\\epsilon) \\subset \\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$. Finally, \\eqref{eq:decrease_discrete_g} is due to $x_{k^*}\\in B(C,\\delta)$ and \\eqref{eq:f_bound_discrete}. We remark that $K\\geqslant k^*+2$ since $\\|x_{k^*+1}-x^*\\| \\leqslant \\|x_{k^*+1}-x_{k^*}\\| +\\|x_{k^*}-x^*\\| \\leqslant \\alpha_{k^*}\\|\\nabla f(x_{k^*})\\|+\\delta \\leqslant (\\delta/(5mL))L+\\delta < \\epsilon$. Since $x_0,\\hdots,x_{K-2} \\in \\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$ and $x_{K-1}$ belongs to \n\\begin{equation*}\n    X_1 := B(C,\\epsilon) \\bigcap \\left\\{ x \\in \\mathbb{R}^n : f(x) \\leqslant \\max_{C} f - m~\\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right) \\right\\},    \n\\end{equation*}\n","start":89909,"end":92287}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_discrete_X_0","sec:Proof of Theorem thm:convergence"],"declared_external":false,"source_only":false,"title":"离散长度上确界的递归界","source_units":[{"raw_tex":"by the length formula \\eqref{eq:length_discrete_K} and the definition of $\\sigma(\\cdot,\\cdot)$ in \\eqref{eq:sup_discrete_X_0} we have\n    \\begin{align*}\n            \\sum_{k=0}^\\infty \\|x_{k+1}-x_k\\| & = \\sum_{k=0}^{K-2} \\|x_{k+1}-x_k\\| + \\sum_{k=K-1}^\\infty \\|x_{k+1}-x_k\\| \\\\\n            & \\leqslant 4m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(\\overline{\\Phi}_0,\\epsilon)} f\\right) \\right) + 2mL\\bar{\\alpha} + \\max\\{0,\\sigma(X_1,\\bar{\\alpha})\\}.\n    \\end{align*}\nNote that the inequality still holds if $K = \\infty$ or $\\sum_{k=0}^{\\infty} \\alpha_k \\leqslant T$. Hence\n\\begin{equation*}\n    \\sigma(X_0,\\bar{\\alpha}) ~\\leqslant~ 4m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(\\overline{\\Phi}_0,\\epsilon)} f\\right) \\right) + 2mL\\bar{\\alpha} + \\max\\{0,\\sigma(X_1,\\bar{\\alpha})\\}.\n\\end{equation*}\n","start":92287,"end":93102}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_discrete_X_0","sec:Proof of Theorem thm:convergence"],"declared_external":false,"source_only":false,"title":"临界值下降与共同步长收尾","source_units":[{"raw_tex":"It now suffices to replace $X_0$ by $X_1$ and repeat the proof starting below \\eqref{eq:sup_discrete_X_0}. Since $f(\\Phi(t,x_1)) \\leqslant f(\\Phi(0,x_1)) \\leqslant \\max_{C} f - m\\psi^{-1}(\\epsilon/(40m))< \\max_{C} f$ for all $t\\geqslant 0$ and $x_1\\in X_1$, the maximal critical value of $f$ in $\\overline{\\Phi(\\mathbb{R}_+,X_1)}$ is less than the maximal critical value of $f$ in $\\overline{\\Phi(\\mathbb{R}_+,X_0)}$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Thus, it is eventually the case that one of the sets $X_0,X_1,\\hdots$ is empty. In order to conclude, one simply needs to choose an upper bound on the step sizes $\\hat{\\alpha}$ corresponding to $X_1$ that is less than or equal to the upper bound $\\bar{\\alpha}$ used for $X_0$. $\\sigma(X_0,\\cdot)$ is finite when evaluated at the last upper bound thus obtained. Indeed, the above recursive formula still holds if we replace $\\bar{\\alpha}$ by any $\\alpha \\in (0,\\bar{\\alpha}]$. In particular, we may take $\\alpha := \\hat{\\alpha}$.\n","start":93103,"end":94162}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_discrete_X_0","sec:Proof of Theorem thm:convergence"],"declared_external":false,"source_only":false,"title":"统一长度蕴含逐初值有界","source_units":[{"raw_tex":"($3 \\Longrightarrow 1$) Obvious.\\\\\n\n","start":94163,"end":94199}]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明前提 clarke1990, Chapter 2；原文使用 2.1.2(a), p.27；2.1.5(b),(d), p.29；2.2.4, p.33；2.3.9 Chain Rule I(ii), p.42。"],"source_only":false,"title":"克拉克次微分的基本性质","source_units":[]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明的微分包含存在前提：aubin1984differential, Theorem 3, p.98；极大延拓用 Zorn 引理，原文引 lang2012algebra, Corollary 2.5, p.884。"],"source_only":false,"title":"微分包含局部存在与佐恩引理","source_units":[]},{"id":"S081","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明前提 attouch2014variational, Theorem 17.1.1；原文第319行引用。"],"source_only":false,"title":"下有界目标的全局梯度流","source_units":[]},{"id":"S082","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明前提 coddington1955theory, Theorem 3.1, p.12；原文第733行引用。"],"source_only":false,"title":"局部利普希茨常微分方程唯一性","source_units":[]},{"id":"S083","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明前提 bolte2007clarke, Corollary 9；原文第408、637、708、735、787行引用。"],"source_only":false,"title":"可定义莫尔斯萨德定理","source_units":[]},{"id":"S084","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明的 KL、单调性及凹化前提：bolte2007clarke, Corollary 15；van1998tame, (1.2), p.43；kurdyka1998gradients, Lemma 2；attouch2010proximal, Theorem 4.1；均见第408行。"],"source_only":false,"title":"局部 KL 与凹去奇异化背景","source_units":[]},{"id":"S085","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明前提 bolte2020conservative, Proposition 2(iv)，及 davis2020stochastic, Lemma 5.2；原文第428–429行另列替代参考。"],"source_only":false,"title":"可定义链式法则与最小速度","source_units":[]},{"id":"S086","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明前提 aubin1984differential, Theorems 1 and 3, p.13，及 Convergence Theorem, p.60；原文第671、673行引用。"],"source_only":false,"title":"轨迹紧性与微分包含收敛","source_units":[]},{"id":"S087","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明的标准有限维分析前提；未逐一引用处是 implicit standard fact made explicit by annotator。原文第331行引用 rudin1964principles, 5.19；第443行引用 tao2006analysis, Theorem 17.7.2。"],"source_only":false,"title":"标准有限维分析工具","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S019","edges":[{"source_id":"S001","target_id":"S020","kind":"supports"},{"source_id":"S003","target_id":"S020","kind":"supports"},{"source_id":"S013","target_id":"S020","kind":"supports"},{"source_id":"S014","target_id":"S020","kind":"supports"},{"source_id":"S004","target_id":"S023","kind":"supports"},{"source_id":"S008","target_id":"S023","kind":"supports"},{"source_id":"S079","target_id":"S025","kind":"supports"},{"source_id":"S080","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S008","target_id":"S026","kind":"supports"},{"source_id":"S080","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S010","target_id":"S029","kind":"supports"},{"source_id":"S007","target_id":"S029","kind":"supports"},{"source_id":"S079","target_id":"S029","kind":"supports"},{"source_id":"S087","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S023","target_id":"S030","kind":"supports"},{"source_id":"S008","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S009","target_id":"S031","kind":"supports"},{"source_id":"S003","target_id":"S031","kind":"supports"},{"source_id":"S081","target_id":"S033","kind":"supports"},{"source_id":"S007","target_id":"S033","kind":"supports"},{"source_id":"S087","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S007","target_id":"S034","kind":"supports"},{"source_id":"S087","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S087","target_id":"S036","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S032","kind":"supports"},{"source_id":"S035","target_id":"S032","kind":"supports"},{"source_id":"S036","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S034","target_id":"S012","kind":"supports"},{"source_id":"S035","target_id":"S012","kind":"supports"},{"source_id":"S036","target_id":"S012","kind":"supports"},{"source_id":"S087","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S011","kind":"supports"},{"source_id":"S004","target_id":"S037","kind":"supports"},{"source_id":"S014","target_id":"S037","kind":"supports"},{"source_id":"S015","target_id":"S037","kind":"supports"},{"source_id":"S083","target_id":"S039","kind":"supports"},{"source_id":"S084","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S079","target_id":"S040","kind":"supports"},{"source_id":"S087","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S079","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S041","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S016","kind":"supports"},{"source_id":"S085","target_id":"S044","kind":"supports"},{"source_id":"S085","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S016","target_id":"S046","kind":"supports"},{"source_id":"S007","target_id":"S046","kind":"supports"},{"source_id":"S015","target_id":"S046","kind":"supports"},{"source_id":"S037","target_id":"S048","kind":"supports"},{"source_id":"S087","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S037","target_id":"S049","kind":"supports"},{"source_id":"S042","target_id":"S049","kind":"supports"},{"source_id":"S087","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S042","target_id":"S050","kind":"supports"},{"source_id":"S087","target_id":"S050","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S049","target_id":"S047","kind":"supports"},{"source_id":"S050","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S016","target_id":"S051","kind":"supports"},{"source_id":"S009","target_id":"S051","kind":"supports"},{"source_id":"S015","target_id":"S051","kind":"supports"},{"source_id":"S087","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S087","target_id":"S054","kind":"supports"},{"source_id":"S037","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S087","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S087","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S087","target_id":"S057","kind":"supports"},{"source_id":"S053","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S087","target_id":"S058","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S055","target_id":"S052","kind":"supports"},{"source_id":"S056","target_id":"S052","kind":"supports"},{"source_id":"S057","target_id":"S052","kind":"supports"},{"source_id":"S058","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S046","target_id":"S017","kind":"supports"},{"source_id":"S051","target_id":"S017","kind":"supports"},{"source_id":"S023","target_id":"S060","kind":"supports"},{"source_id":"S027","target_id":"S060","kind":"supports"},{"source_id":"S007","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S046","target_id":"S061","kind":"supports"},{"source_id":"S042","target_id":"S061","kind":"supports"},{"source_id":"S018","target_id":"S061","kind":"supports"},{"source_id":"S083","target_id":"S061","kind":"supports"},{"source_id":"S087","target_id":"S061","kind":"supports"},{"source_id":"S060","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S007","target_id":"S062","kind":"supports"},{"source_id":"S079","target_id":"S062","kind":"supports"},{"source_id":"S086","target_id":"S062","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S007","target_id":"S063","kind":"supports"},{"source_id":"S079","target_id":"S063","kind":"supports"},{"source_id":"S086","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S046","target_id":"S064","kind":"supports"},{"source_id":"S018","target_id":"S064","kind":"supports"},{"source_id":"S037","target_id":"S064","kind":"supports"},{"source_id":"S016","target_id":"S064","kind":"supports"},{"source_id":"S083","target_id":"S064","kind":"supports"},{"source_id":"S087","target_id":"S064","kind":"supports"},{"source_id":"S062","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S046","target_id":"S065","kind":"supports"},{"source_id":"S007","target_id":"S065","kind":"supports"},{"source_id":"S087","target_id":"S065","kind":"supports"},{"source_id":"S060","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S046","target_id":"S066","kind":"supports"},{"source_id":"S061","target_id":"S067","kind":"supports"},{"source_id":"S062","target_id":"S067","kind":"supports"},{"source_id":"S063","target_id":"S067","kind":"supports"},{"source_id":"S064","target_id":"S067","kind":"supports"},{"source_id":"S065","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S083","target_id":"S067","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S061","target_id":"S059","kind":"supports"},{"source_id":"S062","target_id":"S059","kind":"supports"},{"source_id":"S063","target_id":"S059","kind":"supports"},{"source_id":"S064","target_id":"S059","kind":"supports"},{"source_id":"S065","target_id":"S059","kind":"supports"},{"source_id":"S066","target_id":"S059","kind":"supports"},{"source_id":"S067","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S022","kind":"supports"},{"source_id":"S011","target_id":"S069","kind":"supports"},{"source_id":"S020","target_id":"S069","kind":"supports"},{"source_id":"S021","target_id":"S069","kind":"supports"},{"source_id":"S087","target_id":"S069","kind":"supports"},{"source_id":"S020","target_id":"S070","kind":"supports"},{"source_id":"S005","target_id":"S070","kind":"supports"},{"source_id":"S023","target_id":"S070","kind":"supports"},{"source_id":"S027","target_id":"S070","kind":"supports"},{"source_id":"S082","target_id":"S070","kind":"supports"},{"source_id":"S021","target_id":"S070","kind":"supports"},{"source_id":"S079","target_id":"S070","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S022","target_id":"S071","kind":"supports"},{"source_id":"S046","target_id":"S071","kind":"supports"},{"source_id":"S018","target_id":"S071","kind":"supports"},{"source_id":"S083","target_id":"S071","kind":"supports"},{"source_id":"S087","target_id":"S071","kind":"supports"},{"source_id":"S037","target_id":"S071","kind":"supports"},{"source_id":"S016","target_id":"S071","kind":"supports"},{"source_id":"S051","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S022","target_id":"S072","kind":"supports"},{"source_id":"S087","target_id":"S072","kind":"supports"},{"source_id":"S031","target_id":"S072","kind":"supports"},{"source_id":"S060","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S087","target_id":"S073","kind":"supports"},{"source_id":"S071","target_id":"S074","kind":"supports"},{"source_id":"S072","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S087","target_id":"S074","kind":"supports"},{"source_id":"S071","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S087","target_id":"S075","kind":"supports"},{"source_id":"S070","target_id":"S076","kind":"supports"},{"source_id":"S071","target_id":"S076","kind":"supports"},{"source_id":"S074","target_id":"S076","kind":"supports"},{"source_id":"S075","target_id":"S076","kind":"supports"},{"source_id":"S070","target_id":"S077","kind":"supports"},{"source_id":"S071","target_id":"S077","kind":"supports"},{"source_id":"S072","target_id":"S077","kind":"supports"},{"source_id":"S073","target_id":"S077","kind":"supports"},{"source_id":"S074","target_id":"S077","kind":"supports"},{"source_id":"S075","target_id":"S077","kind":"supports"},{"source_id":"S076","target_id":"S077","kind":"supports"},{"source_id":"S042","target_id":"S077","kind":"supports"},{"source_id":"S083","target_id":"S077","kind":"supports"},{"source_id":"S087","target_id":"S078","kind":"supports"},{"source_id":"S069","target_id":"S068","kind":"supports"},{"source_id":"S070","target_id":"S068","kind":"supports"},{"source_id":"S071","target_id":"S068","kind":"supports"},{"source_id":"S072","target_id":"S068","kind":"supports"},{"source_id":"S073","target_id":"S068","kind":"supports"},{"source_id":"S074","target_id":"S068","kind":"supports"},{"source_id":"S075","target_id":"S068","kind":"supports"},{"source_id":"S076","target_id":"S068","kind":"supports"},{"source_id":"S077","target_id":"S068","kind":"supports"},{"source_id":"S078","target_id":"S068","kind":"supports"},{"source_id":"S068","target_id":"S019","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S019","edges":[{"source_id":"S001","target_id":"S020","kind":"supports"},{"source_id":"S003","target_id":"S020","kind":"supports"},{"source_id":"S013","target_id":"S020","kind":"supports"},{"source_id":"S014","target_id":"S020","kind":"supports"},{"source_id":"S004","target_id":"S023","kind":"supports"},{"source_id":"S008","target_id":"S023","kind":"supports"},{"source_id":"S079","target_id":"S025","kind":"supports"},{"source_id":"S080","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S008","target_id":"S026","kind":"supports"},{"source_id":"S080","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S010","target_id":"S029","kind":"supports"},{"source_id":"S007","target_id":"S029","kind":"supports"},{"source_id":"S079","target_id":"S029","kind":"supports"},{"source_id":"S087","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S023","target_id":"S030","kind":"supports"},{"source_id":"S008","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S009","target_id":"S031","kind":"supports"},{"source_id":"S003","target_id":"S031","kind":"supports"},{"source_id":"S081","target_id":"S033","kind":"supports"},{"source_id":"S007","target_id":"S033","kind":"supports"},{"source_id":"S087","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S007","target_id":"S034","kind":"supports"},{"source_id":"S087","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S087","target_id":"S036","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S032","kind":"supports"},{"source_id":"S035","target_id":"S032","kind":"supports"},{"source_id":"S036","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S034","target_id":"S012","kind":"supports"},{"source_id":"S035","target_id":"S012","kind":"supports"},{"source_id":"S036","target_id":"S012","kind":"supports"},{"source_id":"S087","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S011","kind":"supports"},{"source_id":"S004","target_id":"S037","kind":"supports"},{"source_id":"S014","target_id":"S037","kind":"supports"},{"source_id":"S015","target_id":"S037","kind":"supports"},{"source_id":"S083","target_id":"S039","kind":"supports"},{"source_id":"S084","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S079","target_id":"S040","kind":"supports"},{"source_id":"S087","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S079","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S041","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S016","kind":"supports"},{"source_id":"S085","target_id":"S044","kind":"supports"},{"source_id":"S085","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S016","target_id":"S046","kind":"supports"},{"source_id":"S007","target_id":"S046","kind":"supports"},{"source_id":"S015","target_id":"S046","kind":"supports"},{"source_id":"S037","target_id":"S048","kind":"supports"},{"source_id":"S087","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S037","target_id":"S049","kind":"supports"},{"source_id":"S042","target_id":"S049","kind":"supports"},{"source_id":"S087","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S042","target_id":"S050","kind":"supports"},{"source_id":"S087","target_id":"S050","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S049","target_id":"S047","kind":"supports"},{"source_id":"S050","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S016","target_id":"S051","kind":"supports"},{"source_id":"S009","target_id":"S051","kind":"supports"},{"source_id":"S015","target_id":"S051","kind":"supports"},{"source_id":"S087","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S087","target_id":"S054","kind":"supports"},{"source_id":"S037","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S087","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S087","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S087","target_id":"S057","kind":"supports"},{"source_id":"S053","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S087","target_id":"S058","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S055","target_id":"S052","kind":"supports"},{"source_id":"S056","target_id":"S052","kind":"supports"},{"source_id":"S057","target_id":"S052","kind":"supports"},{"source_id":"S058","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S046","target_id":"S017","kind":"supports"},{"source_id":"S051","target_id":"S017","kind":"supports"},{"source_id":"S023","target_id":"S060","kind":"supports"},{"source_id":"S027","target_id":"S060","kind":"supports"},{"source_id":"S007","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S046","target_id":"S061","kind":"supports"},{"source_id":"S042","target_id":"S061","kind":"supports"},{"source_id":"S018","target_id":"S061","kind":"supports"},{"source_id":"S083","target_id":"S061","kind":"supports"},{"source_id":"S087","target_id":"S061","kind":"supports"},{"source_id":"S060","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S007","target_id":"S062","kind":"supports"},{"source_id":"S079","target_id":"S062","kind":"supports"},{"source_id":"S086","target_id":"S062","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S007","target_id":"S063","kind":"supports"},{"source_id":"S079","target_id":"S063","kind":"supports"},{"source_id":"S086","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S046","target_id":"S064","kind":"supports"},{"source_id":"S018","target_id":"S064","kind":"supports"},{"source_id":"S037","target_id":"S064","kind":"supports"},{"source_id":"S016","target_id":"S064","kind":"supports"},{"source_id":"S083","target_id":"S064","kind":"supports"},{"source_id":"S087","target_id":"S064","kind":"supports"},{"source_id":"S062","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S046","target_id":"S065","kind":"supports"},{"source_id":"S007","target_id":"S065","kind":"supports"},{"source_id":"S087","target_id":"S065","kind":"supports"},{"source_id":"S060","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S046","target_id":"S066","kind":"supports"},{"source_id":"S061","target_id":"S067","kind":"supports"},{"source_id":"S062","target_id":"S067","kind":"supports"},{"source_id":"S063","target_id":"S067","kind":"supports"},{"source_id":"S064","target_id":"S067","kind":"supports"},{"source_id":"S065","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S083","target_id":"S067","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S061","target_id":"S059","kind":"supports"},{"source_id":"S062","target_id":"S059","kind":"supports"},{"source_id":"S063","target_id":"S059","kind":"supports"},{"source_id":"S064","target_id":"S059","kind":"supports"},{"source_id":"S065","target_id":"S059","kind":"supports"},{"source_id":"S066","target_id":"S059","kind":"supports"},{"source_id":"S067","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S022","kind":"supports"},{"source_id":"S011","target_id":"S069","kind":"supports"},{"source_id":"S020","target_id":"S069","kind":"supports"},{"source_id":"S021","target_id":"S069","kind":"supports"},{"source_id":"S087","target_id":"S069","kind":"supports"},{"source_id":"S020","target_id":"S070","kind":"supports"},{"source_id":"S005","target_id":"S070","kind":"supports"},{"source_id":"S023","target_id":"S070","kind":"supports"},{"source_id":"S027","target_id":"S070","kind":"supports"},{"source_id":"S082","target_id":"S070","kind":"supports"},{"source_id":"S021","target_id":"S070","kind":"supports"},{"source_id":"S079","target_id":"S070","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S022","target_id":"S071","kind":"supports"},{"source_id":"S046","target_id":"S071","kind":"supports"},{"source_id":"S018","target_id":"S071","kind":"supports"},{"source_id":"S083","target_id":"S071","kind":"supports"},{"source_id":"S087","target_id":"S071","kind":"supports"},{"source_id":"S037","target_id":"S071","kind":"supports"},{"source_id":"S016","target_id":"S071","kind":"supports"},{"source_id":"S051","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S022","target_id":"S072","kind":"supports"},{"source_id":"S087","target_id":"S072","kind":"supports"},{"source_id":"S031","target_id":"S072","kind":"supports"},{"source_id":"S060","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S087","target_id":"S073","kind":"supports"},{"source_id":"S071","target_id":"S074","kind":"supports"},{"source_id":"S072","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S087","target_id":"S074","kind":"supports"},{"source_id":"S071","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S087","target_id":"S075","kind":"supports"},{"source_id":"S070","target_id":"S076","kind":"supports"},{"source_id":"S071","target_id":"S076","kind":"supports"},{"source_id":"S074","target_id":"S076","kind":"supports"},{"source_id":"S075","target_id":"S076","kind":"supports"},{"source_id":"S070","target_id":"S077","kind":"supports"},{"source_id":"S071","target_id":"S077","kind":"supports"},{"source_id":"S072","target_id":"S077","kind":"supports"},{"source_id":"S073","target_id":"S077","kind":"supports"},{"source_id":"S074","target_id":"S077","kind":"supports"},{"source_id":"S075","target_id":"S077","kind":"supports"},{"source_id":"S076","target_id":"S077","kind":"supports"},{"source_id":"S042","target_id":"S077","kind":"supports"},{"source_id":"S083","target_id":"S077","kind":"supports"},{"source_id":"S087","target_id":"S078","kind":"supports"},{"source_id":"S069","target_id":"S068","kind":"supports"},{"source_id":"S070","target_id":"S068","kind":"supports"},{"source_id":"S071","target_id":"S068","kind":"supports"},{"source_id":"S072","target_id":"S068","kind":"supports"},{"source_id":"S073","target_id":"S068","kind":"supports"},{"source_id":"S074","target_id":"S068","kind":"supports"},{"source_id":"S075","target_id":"S068","kind":"supports"},{"source_id":"S076","target_id":"S068","kind":"supports"},{"source_id":"S077","target_id":"S068","kind":"supports"},{"source_id":"S078","target_id":"S068","kind":"supports"},{"source_id":"S068","target_id":"S019","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"指标、范数与固定结构","source_units":[{"raw_tex":"Let $\\mathbb{N} := \\{0,1,2,\\hdots\\}$ and let $\\|\\cdot\\|$ be the induced norm of an inner product $\\langle \\cdot, \\cdot\\rangle$ on $\\mathbb{R}^n$. Throughout this paper, we fix an arbitrary o-minimal structure on the real field (see Definition \\ref{def:o-minimal}), and say that sets or functions are definable if they are definable in this structure. For readers not familiar with o-minimal structures, note that seemingly all objective functions used in practice are definable in some o-minimal structure. Low-rank matrix recovery (for e.g., matrix factorization, matrix completion, and matrix sensing) and deep neural networks are no exception \\cite[Section 6.2]{bolte2020conservative}.\n","start":4045,"end":4734}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"闭球与球面","source_units":[{"raw_tex":"We begin by stating some standard definitions. Let $B(a,r)$ and $S(a,r)$ respectively denote the closed ball and the sphere of center $a\\in \\mathbb{R}^n$ and radius $r \\geqslant 0$.\n\n","start":26428,"end":26611}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":["def:lipschitz"],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"局部利普希茨连续","source_units":[{"raw_tex":"\\begin{definition}\n\\label{def:lipschitz}\nA function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}^m$ is locally Lipschitz if for all $a \\in \\mathbb{R}^n$, there exist $r>0$ and $L>0$ such that \n\\begin{equation*}\n        \\forall x,y \\in B(a,r), ~~~ \\|f(x)-f(y)\\| \\leqslant L \\|x-y\\|.\n    \\end{equation*}\n\\end{definition}\n\n","start":26611,"end":26923}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["def:Clarke"],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"克拉克次微分","source_units":[{"raw_tex":"\\begin{definition}\n    \\label{def:Clarke}\n    \\cite[Chapter 2]{clarke1990}\n    Let $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz function. The Clarke subdifferential is the set-valued mapping $\\partial f:\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^n$ defined for all $x \\in \\mathbb{R}^n$ by\n    $\\partial f(x) := \\{ s \\in \\mathbb{R}^n : f^\\circ(x,h) \\geqslant \\langle s , h \\rangle, ~ \\forall h\\in \\mathbb{R}^n \\}$ where\n\\begin{equation*}\n    f^\\circ(x,h) := \\limsup_{\\tiny\\begin{array}{c} y\\rightarrow x \\\\\n    t \\searrow 0\n    \\end{array}\n    } \\frac{f(y+th)-f(y)}{t}.\n\\end{equation*}\n\\end{definition}\n\n","start":26923,"end":27541}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"临界点与光滑特化","source_units":[{"raw_tex":"We say that $x \\in \\mathbb{R}^n$ is critical if $0 \\in \\partial f(x)$. If $f$ is continuously differentiable in a neighborhood of $x \\in \\mathbb{R}^n$, then $\\partial f(x) = \\{\\nabla f(x)\\}$ \\cite[2.2.4 Proposition p. 33]{clarke1990}.\n","start":27541,"end":27776}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["def:absolute_continuity"],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"绝对连续性","source_units":[{"raw_tex":"\\begin{definition}\n    \\label{def:absolute_continuity} \n    \\cite[Definition 1 p. 12]{aubin1984differential}\n    Given some real numbers $a\\leqslant b$, a function $x:[a,b]\\rightarrow\\mathbb{R}^n$ is absolutely continuous if for all $\\epsilon>0$, there exists $\\delta>0$ such that, for any finite collection of disjoint subintervals $[a_1,b_1],\\hdots,[a_m,b_m]$ of $[a,b]$ such that $\\sum_{i=1}^m b_i-a_i \\leqslant \\delta$, we have $\\sum_{i=1}^m \\|x(b_i) - x(a_i)\\| \\leqslant \\epsilon$.\n\\end{definition}\n\n","start":28557,"end":29062}]},{"id":"S007","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S007"],"declared_external":true,"declaration_text":["原文第229行引用 nielsen1997introduction, Theorem 20.8；亦属声明的绝对连续性标准分析前提。"],"source_only":false,"title":"绝对连续函数的积分表示","source_units":[{"raw_tex":"By virtue of \\cite[Theorem 20.8]{nielsen1997introduction}, $x:[a,b]\\rightarrow\\mathbb{R}^n$ is absolutely continuous if and only if it is differentiable almost everywhere on $(a,b)$, its derivative $x'(\\cdot)$ is Lebesgue integrable, and $x(t) - x(a) = \\int_a^t x'(\\tau)d\\tau$ for all $t\\in [a,b]$. Given a noncompact interval $I$ of $\\mathbb{R}$, $x:I\\rightarrow \\mathbb{R}^n$ is absolutely continuous if it is absolutely continuous on all compact subintervals of $I$. Given an interval $I$ of $\\mathbb{R}$, let $\\mathcal{A}(I,\\mathbb{R}^n)$ be the set of absolutely continuous functions defined from $I$ to $\\mathbb{R}^n$.\n\n","start":29062,"end":29688}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":["def:trajectory"],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"极大连续次梯度轨迹","source_units":[{"raw_tex":"\\begin{definition}\n\\label{def:trajectory}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz function. Let $(S,\\leqslant)$ be the set of absolutely continuous functions $x:[0,T)\\rightarrow\\mathbb{R}^n$ where $0<T\\leqslant \\infty$ such that  \n\\begin{equation*}\n    x'(t) \\in -\\partial f(x(t)), ~~~\\mathrm{for~a.e.}~t\\in (0,T),\n\\end{equation*}\nequipped with the partial order $\\leqslant$ defined by $x_1:[0,T_1)\\rightarrow\\mathbb{R}^n$ is less than or equal to $x_2:[0,T_2)\\rightarrow\\mathbb{R}^n$ if and only if $T_1\\leqslant T_2$ and $x_1(t) = x_2(t)$ for all $t\\in [0,T_1)$. \nWe refer to maximal elements of $(S,\\leqslant)$ as continuous subgradient trajectories.\n\\end{definition}\n\n","start":29688,"end":30387}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"全局轨迹与离散轨迹","source_units":[{"raw_tex":"We say that a continuous subgradient trajectory is globally defined if it is of the form $x:[0,T)\\rightarrow\\mathbb{R}^n$ where $T=\\infty$. We refer to discrete subgradient trajectories as sequences $(x_k)_{k\\in \\mathbb{N}}$ in $\\mathbb{R}^n$ such that $x_{k+1} \\in x_k -\\alpha_k \\partial f(x_k)$ for all $k\\in \\mathbb{N}$ for some $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots>0$. When $f$ is continuously differentiable, we refer to subgradient trajectories as  gradient trajectories.\n\n","start":30387,"end":30864}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":["def:bounded_trajectory"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"连续轨迹有界性","source_units":[{"raw_tex":"\\begin{definition}\n\\label{def:bounded_trajectory}\nA continuous subgradient trajectory $x: [0,T) \\rightarrow \\mathbb{R}^n$ where $0< T\\leqslant \\infty$ is bounded if there exists $c>0$ such that $\\|x(t)\\| \\leqslant c$ for all $t\\in [0,T)$.\n\\end{definition}\n","start":30864,"end":31120}]},{"id":"S011","semantic_type":"proposition","kind":"proposition","labels":["rem:tracking"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"单初值局部跟踪扩展","source_units":[{"raw_tex":"\\begin{remark}\n\\label{rem:tracking}\nIf $X_0$ is a singleton $\\{x_0\\}$ and $x:[0,\\hat{T})\\rightarrow \\mathbb{R}^n$ is a gradient trajectory initialized at $x_0$, then one may remove the assumption that $f$ is lower bounded in Proposition \\ref{prop:tracking} and instead require that $T \\in (0,\\hat{T})$. In that case, $\\sigma_T(X_0)=\\int_0^T \\|x'(t)\\|dt < \\infty$ since $\\nabla f$ is Lipschitz continuous on the bounded set $x([0,T])$.\n\\end{remark}","start":44646,"end":45093}]},{"id":"S012","semantic_type":"proof","kind":"proof","labels":["rem:tracking"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"局部跟踪扩展的作者论证","source_units":[{"raw_tex":"If $X_0$ is a singleton $\\{x_0\\}$ and $x:[0,\\hat{T})\\rightarrow \\mathbb{R}^n$ is a gradient trajectory initialized at $x_0$, then one may remove the assumption that $f$ is lower bounded in Proposition \\ref{prop:tracking} and instead require that $T \\in (0,\\hat{T})$. In that case, $\\sigma_T(X_0)=\\int_0^T \\|x'(t)\\|dt < \\infty$ since $\\nabla f$ is Lipschitz continuous on the bounded set $x([0,T])$.\n","start":44682,"end":45081}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["def:o-minimal"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"极小序结构公理","source_units":[{"raw_tex":"\\begin{definition}\n\\label{def:o-minimal}\n\\cite[Definition p. 503-506]{van1996geometric}\nAn o-minimal structure on the real field is a sequence $S = (S_k)_{k \\in \\mathbb{N}}$ such that for all $k \\in \\mathbb{N}$:\n\\begin{enumerate}\n    \\item $S_k$ is a boolean algebra of subsets of $\\mathbb{R}^k$, with $\\mathbb{R}^k \\in S_k$;\n    \\item $S_k$ contains the diagonal $\\{(x_1,\\hdots,x_k) \\in \\mathbb{R}^k : x_i = x_j\\}$ for $1\\leqslant i<j \\leqslant k$;\n    \\item If $A\\in S_k$, then $A\\times \\mathbb{R}$ and $\\mathbb{R}\\times A$ belong to $S_{k+1}$;\n    \\item If $A \\in S_{k+1}$ and $\\pi:\\mathbb{R}^{k+1}\\rightarrow\\mathbb{R}^k$ is the projection onto the first $k$ coordinates, then $\\pi(A) \\in S_k$;\n    \\item $S_3$ contains the graphs of addition and multiplication;\n    \\item $S_1$ consists exactly of the finite unions of open intervals and singletons. \n\\end{enumerate}\n\\end{definition}\n\n","start":46181,"end":47071}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"可定义集合与函数","source_units":[{"raw_tex":"A subset $A$ of $\\mathbb{R}^n$ is definable in an o-minimal structure $(S_k)_{k\\in\\mathbb{N}}$ if $A \\in S_k$. A function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ is definable in an o-minimal structure if its graph, that is to say $\\{ (x,t) \\in \\mathbb{R}^{n+1} : f(x)=t \\}$, is definable in that structure.\n","start":47071,"end":47376}]},{"id":"S015","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"距离、临界值与逆映射","source_units":[{"raw_tex":"We next state the uniform Kurdyka-\\L{}ojasiewicz inequality. Given a subset of $S$ of $\\mathbb{R}^n$, let $\\mathring{S}$ and $\\overline{S}$ denote the interior and closure of $S$ in $\\mathbb{R}^n$ respectively. A function $\\psi:S \\rightarrow S$ is a homeomorphism if it is a continuous bijection and the inverse function $\\psi^{-1}$ is continuous. $\\psi:S \\rightarrow S$ is a diffeomorphism if $\\mathring{S} \\neq \\emptyset$, $\\psi$ is a homeomorphism, and both $\\psi$ and $\\psi^{-1}$ are continuously differentiable on $\\mathring{S}$. Given $x\\in \\mathbb{R}^n$, consider the distance of $x$ to $S$ defined by $d(x,S) := \\inf \\{ \\|x-y\\| : y \\in S \\}$. Given a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$, a real number $v$ is critical value of $f$ in $S$ if there exists $x \\in S$ such that $v = f(x)$ and $0\\in \\partial f(x)$. Given a set-valued mapping $F:\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^m$ and $y \\in \\mathbb{R}^m$, let $F^{-1}(y) := \\{ x \\in \\mathbb{R}^n : F(x) \\ni y \\}$.\n","start":48364,"end":49369}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"去奇异化函数","source_units":[{"raw_tex":"We say that $\\psi$ in Proposition \\ref{prop:KL} is a desingularizing function of $f$ on $X$. In order to prove Proposition \\ref{prop:length_continuous}, we recall the following result.\n","start":52714,"end":52899}]},{"id":"S017","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"有界轨迹的有限长度","source_units":[{"raw_tex":"An immediate consequence of Propositions \\ref{prop:length_continuous} and \\ref{prop:length_discrete} is that bounded continuous subgradient trajectories and bounded discrete gradient trajectories have finite length, as is well known. An easy fact that we will not prove here is that continuous subgradient trajectories of finite length converge to critical points of the objective function under the sole assumption that it is locally Lipschitz. The same holds for discrete subgradient trajectories if the step sizes are not summable.\n","start":64646,"end":65181}]},{"id":"S018","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S018"],"declared_external":true,"declaration_text":["声明的允许前提：原文第558行明确不予证明的有限长度轨迹临界收敛事实。"],"source_only":false,"title":"有限长度轨迹的临界收敛","source_units":[{"raw_tex":"An immediate consequence of Propositions \\ref{prop:length_continuous} and \\ref{prop:length_discrete} is that bounded continuous subgradient trajectories and bounded discrete gradient trajectories have finite length, as is well known. An easy fact that we will not prove here is that continuous subgradient trajectories of finite length converge to critical points of the objective function under the sole assumption that it is locally Lipschitz. The same holds for discrete subgradient trajectories if the step sizes are not summable.\n","start":64646,"end":65181}]},{"id":"S019","semantic_type":"theorem","kind":"theorem","labels":["eq:gradient_method","thm:convergence"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"连续有界与离散统一长度等价","source_units":[{"raw_tex":"\\begin{theorem}\n\\label{thm:convergence}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a definable differentiable function whose gradient is locally Lipschitz. The following statements are equivalent:\n\\begin{enumerate}\n    \\item For all $x_0 \\in \\mathbb{R}^n$, there exist $\\bar{\\alpha}>0$ and $c>0$ such that for all $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$, the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by \n    \\begin{equation}\n    \\label{eq:gradient_method}\n        x_{k+1} = x_k - \\alpha_k \\nabla f(x_k), ~~~\\forall k \\in \\mathbb{N},\n    \\end{equation}\n    satisfies $\\|x_k\\|\\leqslant c$ for all $k \\in \\mathbb{N}$.\n    \\item For all $0< T \\leqslant \\infty$ and any differentiable function $x:[0,T)\\rightarrow\\mathbb{R}^n$ such that\n    \\begin{equation*}\n        x'(t) = -\\nabla f(x(t)), ~~~ \\forall t \\in (0,T),\n    \\end{equation*}\n    there exists $c>0$ such that $\\|x(t)\\| \\leqslant c$ for all $t\\in [0,T)$.\n    \\item For any bounded subset $X_0$ of $\\mathbb{R}^n$, there exist $\\bar{\\alpha}>0$ and $c>0$ such that, for all $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$ and $x_0 \\in X_0$, the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by \\eqref{eq:gradient_method} obeys \n    \\begin{equation*}\n        \\sum_{k=0}^{\\infty} \\|x_{k+1}-x_k\\| \\leqslant c.\n    \\end{equation*}\n\\end{enumerate}\n\\end{theorem}","start":4735,"end":6096}]},{"id":"S020","semantic_type":"assumption","kind":"assumption","labels":["thm:convergence"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"主定理的目标函数假设","source_units":[{"raw_tex":"Let $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a definable differentiable function whose gradient is locally Lipschitz. The following statements are equivalent:\n","start":4775,"end":4936}]},{"id":"S021","semantic_type":"algorithm","kind":"algorithm","labels":["eq:gradient_method","thm:convergence"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"变步长梯度迭代","source_units":[{"raw_tex":"    \\item For all $x_0 \\in \\mathbb{R}^n$, there exist $\\bar{\\alpha}>0$ and $c>0$ such that for all $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$, the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by \n    \\begin{equation}\n    \\label{eq:gradient_method}\n        x_{k+1} = x_k - \\alpha_k \\nabla f(x_k), ~~~\\forall k \\in \\mathbb{N},\n    \\end{equation}\n    satisfies $\\|x_k\\|\\leqslant c$ for all $k \\in \\mathbb{N}$.\n","start":4954,"end":5385}]},{"id":"S022","semantic_type":"lemma","kind":"lemma","labels":["lemma:uniform_length"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"连续轨迹的统一长度引理","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lemma:uniform_length}\nIf a locally Lipschitz definable function has bounded continuous subgradient trajectories, then continuous subgradient trajectories initialized in a bounded set have uniformly bounded lengths.\n\\end{lemma}","start":12812,"end":13059}]},{"id":"S023","semantic_type":"proposition","kind":"proposition","labels":["prop:existence_trajectory"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"极大次梯度轨迹存在","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:existence_trajectory}\nLocally Lipschitz functions have at least one continuous subgradient trajectory for every initial point.\n\\end{proposition}","start":31369,"end":31545}]},{"id":"S024","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"轨迹存在性的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz function and let $x_0\\in\\mathbb{R}^n$. Since $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d) p. 29]{clarke1990} with nonempty, compact and convex values \\cite[2.1.2 Proposition (a) p. 27]{clarke1990}. By virtue of \\cite[Theorem 3 p. 98]{aubin1984differential}, there exist $\\epsilon>0$ and an absolutely continuous function $x:[0,\\epsilon)\\rightarrow\\mathbb{R}^n$ such that  \n\\begin{equation*}\n    x'(t) \\in -\\partial f(x(t)), ~\\mathrm{for~a.e.}~ t\\in (0,\\epsilon),~~~ x(0) = x_0.\n\\end{equation*}\nLet $S_{x_0}$ be the set of functions in $S$ from Definition \\ref{def:trajectory} such that $x(0) = x_0$. Then $S_{x_0} \\neq \\emptyset$. Let $P$ be a nonempty totally ordered subset of $S_{x_0}$. Let $0<T \\leqslant\\infty$ be the supremum of the endpoints of the domains of all functions in $P$. Consider the function $x:[0,T)\\rightarrow\\mathbb{R}^n$ such that for all $\\bar{x}:[0,\\overline{T})\\rightarrow \\mathbb{R}^n$ in $P$ we have $x(t) := \\bar{x}(t)$ for all $t\\in [0,\\overline{T})$. It is well defined and it is an upper bound of $P$ in $S_{x_0}$. By virtue of Zorn's lemma \\cite[Corollary 2.5 p. 884]{lang2012algebra}, $S_{x_0}$ contains a maximal element. \\end{proof}","start":31546,"end":32866}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"由次微分性质获得局部解","source_units":[{"raw_tex":"\\begin{proof}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz function and let $x_0\\in\\mathbb{R}^n$. Since $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d) p. 29]{clarke1990} with nonempty, compact and convex values \\cite[2.1.2 Proposition (a) p. 27]{clarke1990}. By virtue of \\cite[Theorem 3 p. 98]{aubin1984differential}, there exist $\\epsilon>0$ and an absolutely continuous function $x:[0,\\epsilon)\\rightarrow\\mathbb{R}^n$ such that  \n\\begin{equation*}\n    x'(t) \\in -\\partial f(x(t)), ~\\mathrm{for~a.e.}~ t\\in (0,\\epsilon),~~~ x(0) = x_0.\n\\end{equation*}\n","start":31546,"end":32192}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"按延拓偏序取极大元","source_units":[{"raw_tex":"Let $S_{x_0}$ be the set of functions in $S$ from Definition \\ref{def:trajectory} such that $x(0) = x_0$. Then $S_{x_0} \\neq \\emptyset$. Let $P$ be a nonempty totally ordered subset of $S_{x_0}$. Let $0<T \\leqslant\\infty$ be the supremum of the endpoints of the domains of all functions in $P$. Consider the function $x:[0,T)\\rightarrow\\mathbb{R}^n$ such that for all $\\bar{x}:[0,\\overline{T})\\rightarrow \\mathbb{R}^n$ in $P$ we have $x(t) := \\bar{x}(t)$ for all $t\\in [0,\\overline{T})$. It is well defined and it is an upper bound of $P$ in $S_{x_0}$. By virtue of Zorn's lemma \\cite[Corollary 2.5 p. 884]{lang2012algebra}, $S_{x_0}$ contains a maximal element. \\end{proof}","start":32192,"end":32866}]},{"id":"S027","semantic_type":"proposition","kind":"proposition","labels":["prop:bounded_global"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"有界极大轨迹全局存在","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:bounded_global}\nBounded continuous subgradient trajectories are globally defined.\n\\end{proposition}","start":32868,"end":32999}]},{"id":"S028","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"有界轨迹全局性的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $x:[0,T)\\rightarrow\\mathbb{R}^n$ be a bounded continuous subgradient trajectory of $f$ where $0<T \\leqslant \\infty$. We reason by contradiction and assume that $T<\\infty$. By Definition \\ref{def:bounded_trajectory}, there exists $c>0$ such that $\\|x(t)\\| \\leqslant c$ for all $t\\in [0,T)$. Consider a sequence $t_0,t_1,t_2,\\hdots \\in [0,T)$ converging to $T$. For all $k\\in \\mathbb{N}$, let $x_k:[0,T]\\rightarrow\\mathbb{R}^n$ be defined by $x_k(t) := x(t)$ for all $t\\in [0,t_k]$ and $x_k(t) = 0$ for all $t\\in (t_k,T]$. By construction, $x'_k$ converges pointwise almost everywhere to $x'$ on $(0,T)$. Let $L>0$ be a Lipschitz constant of $f$ on $B(0,c)$. By \\cite[2.1.2 Proposition (a) p. 27]{clarke1990}, for all $x\\in B(0,c)$ and $s\\in \\partial f(x)$, we have $\\|s\\| \\leqslant L$. For almost every $t\\in (0,T)$, we thus have $\\|x_k'(t)\\| \\leqslant \\|x'(t)\\| \\leqslant L$. By Lebesgue's dominated convergence theorem, $\\int_0^{t_k} x'(t)dt = \\int_0^T x'_k(t)dt \\rightarrow \\int_0^T x'(t)dt$. Since $x(\\cdot)$ is absolutely continuous on $[0,t_k]$, we have \n$x(t_k) - x(0) = \\int_0^{t_k} x'(t)dt$. Thus $x(\\cdot)$ can be extended to an absolutely continuous function on $[0,T]$ where \n$x(T) := x(0) + \\int_0^T x'(t)dt$. According to Proposition \\ref{prop:existence_trajectory}, $x(\\cdot)$ can further be extended to an absolutely continuous function on $[0,T+\\epsilon)$ for some $\\epsilon>0$ while satisfying the differential inclusion. This contradicts the maximality of $x(\\cdot)$. \n\\end{proof}","start":33000,"end":34516}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"有限终点处的积分极限","source_units":[{"raw_tex":"\\begin{proof}\nLet $x:[0,T)\\rightarrow\\mathbb{R}^n$ be a bounded continuous subgradient trajectory of $f$ where $0<T \\leqslant \\infty$. We reason by contradiction and assume that $T<\\infty$. By Definition \\ref{def:bounded_trajectory}, there exists $c>0$ such that $\\|x(t)\\| \\leqslant c$ for all $t\\in [0,T)$. Consider a sequence $t_0,t_1,t_2,\\hdots \\in [0,T)$ converging to $T$. For all $k\\in \\mathbb{N}$, let $x_k:[0,T]\\rightarrow\\mathbb{R}^n$ be defined by $x_k(t) := x(t)$ for all $t\\in [0,t_k]$ and $x_k(t) = 0$ for all $t\\in (t_k,T]$. By construction, $x'_k$ converges pointwise almost everywhere to $x'$ on $(0,T)$. Let $L>0$ be a Lipschitz constant of $f$ on $B(0,c)$. By \\cite[2.1.2 Proposition (a) p. 27]{clarke1990}, for all $x\\in B(0,c)$ and $s\\in \\partial f(x)$, we have $\\|s\\| \\leqslant L$. For almost every $t\\in (0,T)$, we thus have $\\|x_k'(t)\\| \\leqslant \\|x'(t)\\| \\leqslant L$. By Lebesgue's dominated convergence theorem, $\\int_0^{t_k} x'(t)dt = \\int_0^T x'_k(t)dt \\rightarrow \\int_0^T x'(t)dt$. Since $x(\\cdot)$ is absolutely continuous on $[0,t_k]$, we have \n$x(t_k) - x(0) = \\int_0^{t_k} x'(t)dt$. Thus $x(\\cdot)$ can be extended to an absolutely continuous function on $[0,T]$ where \n","start":33000,"end":34205}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"从终点继续解而矛盾","source_units":[{"raw_tex":"$x(T) := x(0) + \\int_0^T x'(t)dt$. According to Proposition \\ref{prop:existence_trajectory}, $x(\\cdot)$ can further be extended to an absolutely continuous function on $[0,T+\\epsilon)$ for some $\\epsilon>0$ while satisfying the differential inclusion. This contradicts the maximality of $x(\\cdot)$. \n\\end{proof}","start":34205,"end":34516}]},{"id":"S031","semantic_type":"proposition","kind":"proposition","labels":["eq:tracking","prop:tracking"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"有界初值集的有限时跟踪","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:tracking}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a lower bounded differentiable function with a locally Lipschitz gradient. Let $X_0$ be a bounded subset of $\\mathbb{R}^n$ and let $T>0$. For all $\\epsilon>0$, there exists $\\bar{\\alpha}>0$ such that for all $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$ and for all sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ such that $x_0 \\in X_0$ and $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$, the continuous gradient trajectory $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ of $f$ initialized at $x_0$ satisfies\n\\begin{equation}\n\\label{eq:tracking}\n   \\forall k\\in \\mathbb{N}^*,~~~ \\alpha_0+\\cdots+\\alpha_{k-1} \\leqslant T ~~~\\Longrightarrow~~~ \\|x_k - x(\\alpha_0+\\cdots+\\alpha_{k-1})\\| \\leqslant \\epsilon.\n\\end{equation}\n\\end{proposition}","start":37589,"end":38440}]},{"id":"S032","semantic_type":"proof","kind":"proof","labels":["eq:global_a","eq:global_b","eq:global_c","eq:global_d","eq:global_e","eq:global_f","eq:global_g","eq:length_finite_smooth_a","eq:length_finite_smooth_b","eq:length_finite_smooth_c","eq:length_finite_smooth_d","eq:length_finite_smooth_e","eq:local_error","eq:sup_trajectory_T_smooth","local_a","local_b","local_c","local_d","local_e","local_f"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"有限时跟踪的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nWithout loss of generality, we may assume that $X_0 \\neq \\emptyset$. Let $L>0$ and $M>0$ respectively denote Lipschitz constants of $f$ and $\\nabla f$ with respect to $\\|\\cdot\\|$ on the convex hull of $B(X_0,\\sigma_T(X_0)+1):= X_0 + B(0,\\sigma_T(X_0)+1)$ where \n\\begin{subequations}\n\\label{eq:sup_trajectory_T_smooth}\n\\begin{align}\n    \\sigma_T(X_0) := & \\sup\\limits_{x \\in \\mathcal{C}^1(\\mathbb{R}_+,\\mathbb{R}^n)} ~~ \\int_0^T \\|x'(t)\\|dt \\\\\n  & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx'(t) = - \\nabla f(x(t)), ~\\forall t > 0,\\\\[3mm] x(0) \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\nAbove, $\\mathcal{C}^1(\\mathbb{R}_+,\\mathbb{R}^n)$ is the set of continuous functions on $[0,\\infty )$ that are continuously differentiable on $(0,\\infty)$. Note that $\\sigma_T(X_0)>-\\infty$ since $f$ is lower bounded and differentiable with a locally Lipschitz gradient. For every $x_0\\in X_0$, there hence exists a unique globally defined continuous gradient trajectory of $f$ initialized at $x_0$ \\cite[Theorem 17.1.1]{attouch2014variational}. Also, $\\sigma_T(X_0)<\\infty$ since for any feasible point $x(\\cdot)$ of \\eqref{eq:sup_trajectory_T_smooth}, we have \n\\begin{subequations}\n\\begin{align}\n    \\int_0^T \\|x'(t)\\|dt & \\leqslant \\sqrt{T} \\sqrt{ \\int_0^T \\|x'(t)\\|^2 dt} \\label{eq:length_finite_smooth_a} \\\\\n    & = \\sqrt{T} \\sqrt{ -\\int_0^T \\langle \\nabla f(x(t)), x'(t) \\rangle dt} \\label{eq:length_finite_smooth_b} \\\\\n    & = \\sqrt{T} \\sqrt{ -\\int_0^T (f \\circ x)'(t) dt} \\label{eq:length_finite_smooth_c} \\\\\n    & = \\sqrt{T} \\sqrt{ f(x(0)) - f(x(T))} \\label{eq:length_finite_smooth_d} \\\\\n    & \\leqslant \\sqrt{T\\left(\\sup_{X_0} f - \\inf_{\\mathbb{R}^n} f\\right)} < \\infty. \\label{eq:length_finite_smooth_e}\n\\end{align}\n\\end{subequations}\nIndeed, \\eqref{eq:length_finite_smooth_a} is due to the Cauchy-Schwarz inequality. \\eqref{eq:length_finite_smooth_e} holds because $X_0$ is bounded and $f$ is lower bounded.\n\nIt is easy to check that $L$ and $ML$ are respectively Lipschitz and gradient Lipschitz constants on $[0,T]$ of all continuous gradient trajectories of $f$ initialized in $X_0$. Indeed, let $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ be a continuous gradient trajectory initialized in $X_0$. Since $x(t) \\in B(X_0,\\sigma_T(X_0))$ for all $t\\in [0,T]$, we have $\\|x'(t)\\| = \\|\\nabla f(x(t))\\| \\leqslant L$. By the mean value theorem \\cite[5.19 Theorem]{rudin1964principles}, for all $s,t \\in [0,T]$ we have $\\|x'(t)-x'(s)\\| = \\|\\nabla f(x(t)) - \\nabla f(x(s))\\| \\leqslant M \\|x(t)-x(s)\\| \\leqslant M L |t-s|$. As a byproduct, we get the Taylor bound\n\\begin{subequations}\n    \\begin{align}\n    \\|x(t)-x(s) - x'(s)(t-s)\\| & = \\left\\| \\int_s^t (x'(\\tau) - x'(s)) d\\tau \\right\\| \\\\\n    & \\leqslant \\left| \\int_s^t \\| x'(\\tau) - x'(s) \\| d\\tau \\right| \\\\\n    & \\leqslant \\left|\\int_s^t ML | \\tau - s | d\\tau \\right| \\\\\n    & = \\frac{ML}{2}(t-s)^2. \\label{eq:local_error}\n    \\end{align}\n\\end{subequations}\n\nLet $\\epsilon \\in (0,1)$ and let $\\bar{\\alpha} := 2\\epsilon e^{-MT}/(LMT)$. Consider $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$ and a sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$ and $x_0 \\in X_0$. Let $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ be the unique continuous gradient trajectory of $f$ initialized at $x_0$. Let $t_0:=0$ and $t_k := \\alpha_0+\\cdots+\\alpha_{k-1}$ when $k\\in \\mathbb{N}^*$ so that $\\|x_0 - x(t_0)\\| = 0 \\leqslant \\epsilon$. Assume that \\eqref{eq:tracking} holds up to some index $K$. For $k=0,\\hdots,K$, we have\n\\begin{subequations}\n    \\begin{align}\n        \\|x_{k+1} - x(t_{k+1})\\| = &  ~\\| x_k - \\alpha_k \\nabla f(x_k) - x(t_{k+1}) \\| \\label{local_a} \\\\[2mm]\n        \\leqslant  & ~ \\| x_k - \\alpha_k \\nabla f(x_k) - [x(t_k) - \\alpha_k \\nabla f(x(t_k))] \\|~+ \\label{local_b} \\\\[2mm] & ~ \\| x(t_k) - \\alpha_k \\nabla f(x(t_k)) - x(t_{k+1}) \\| \\label{local_c} \\\\[2mm]\n        \\leqslant & ~ \\|x_k - x(t_k)\\| + \\alpha_k \\|\\nabla f(x_k) - \\nabla f(x(t_k))\\| ~+ \\label{local_d} \\\\[2mm]\n        & ~ ML\\alpha_k^2/2 \\label{local_e} \\\\[2mm]\n        \\leqslant & ~ (1+\\alpha_k M)\\|x_k - x(t_k)\\| + ML\\alpha_k^2/2. \\label{local_f}\n    \\end{align}\n\\end{subequations}\nThe term in \\eqref{local_c} is equal to the local truncation error $\\|x(t_{k+1})  - x(t_k) - x'(t_k)(t_{k+1}-t_k) \\|$, and can hence be bounded above using \\eqref{eq:local_error}. In the second term of \\eqref{local_d}, we invoke a Lipschitz constant of $\\nabla f$ on the convex hull of $B(X_0,\\sigma_T(X_0)+1)$. Indeed, by the induction hypothesis and $\\epsilon < 1$, $x_k$ and $x(t_k)$ belong to $B(X_0,\\sigma_T(X_0)+\\epsilon)$. Hence \n\\begin{subequations}\n    \\begin{align}\n        \\|x_{K+1} - x(t_{K+1})\\| \\leqslant &~ \\prod_{k=0}^K (1+\\alpha_kM)\\|x_0-x(t_0)\\|~+ \\label{eq:global_a}\\\\\n        &~ \\sum_{k=0}^K \\frac{ML\\alpha_k^2}{2} \\prod_{l=k+1}^{K} (1+\\alpha_lM)\\label{eq:global_b}\\\\\n        \\leqslant &~ \\prod_{l=0}^{K} (1+\\alpha_lM) \\sum_{k=0}^K \\frac{ML\\alpha_k^2}{2}\\label{eq:global_c}\\\\\n        \\leqslant &~ \\left(1+\\frac{M}{K+1}\\sum_{k=0}^K \\alpha_k\\right)^{K+1} \\sum_{k=0}^K \\frac{ML\\bar{\\alpha}\\alpha_k}{2}\\label{eq:global_d}\\\\\n        \\leqslant &~ \\left(1+\\frac{MT}{K+1}\\right)^{K+1} \\frac{ML\\bar{\\alpha}T}{2}\\label{eq:global_e}\\\\\n        \\leqslant &~~ \\bar{\\alpha} LMTe^{MT}/2\\label{eq:global_f}\\\\\n        = &~~ \\epsilon.\\label{eq:global_g}\n    \\end{align}\n\\end{subequations}\nThe term in \\eqref{eq:global_a} is equal to zero because $x_0 = x(t_0)$. By convention, we set the product in \\eqref{eq:global_b} to be equal to one if the index set for $l$ is empty. We bound the partial products ranging from $k+1$ to $K$ inside the summation of \\eqref{eq:global_b} by the full product ranging from $0$ to $K$ in order to obtain \\eqref{eq:global_c}. We then use the inequality of arithmetic and geometric means and $\\alpha_k\\leqslant\\bar{\\alpha}$ in order to obtain \\eqref{eq:global_d}. In order to prove the induction step, we assume that $\\alpha_0+\\cdots+\\alpha_K \\leqslant T$, hence \\eqref{eq:global_e}. \\eqref{eq:global_f} follows from $\\ln(1+x) \\leqslant x$ for all $x>0$. Finally, \\eqref{eq:global_g} holds because $\\bar{\\alpha} = 2\\epsilon e^{-MT}/(LMT)$.\n\\end{proof}","start":38442,"end":44644}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":["eq:length_finite_smooth_a","eq:length_finite_smooth_b","eq:length_finite_smooth_c","eq:length_finite_smooth_d","eq:length_finite_smooth_e","eq:sup_trajectory_T_smooth"],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"有限时连续长度与公共区域","source_units":[{"raw_tex":"\\begin{proof}\nWithout loss of generality, we may assume that $X_0 \\neq \\emptyset$. Let $L>0$ and $M>0$ respectively denote Lipschitz constants of $f$ and $\\nabla f$ with respect to $\\|\\cdot\\|$ on the convex hull of $B(X_0,\\sigma_T(X_0)+1):= X_0 + B(0,\\sigma_T(X_0)+1)$ where \n\\begin{subequations}\n\\label{eq:sup_trajectory_T_smooth}\n\\begin{align}\n    \\sigma_T(X_0) := & \\sup\\limits_{x \\in \\mathcal{C}^1(\\mathbb{R}_+,\\mathbb{R}^n)} ~~ \\int_0^T \\|x'(t)\\|dt \\\\\n  & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx'(t) = - \\nabla f(x(t)), ~\\forall t > 0,\\\\[3mm] x(0) \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\nAbove, $\\mathcal{C}^1(\\mathbb{R}_+,\\mathbb{R}^n)$ is the set of continuous functions on $[0,\\infty )$ that are continuously differentiable on $(0,\\infty)$. Note that $\\sigma_T(X_0)>-\\infty$ since $f$ is lower bounded and differentiable with a locally Lipschitz gradient. For every $x_0\\in X_0$, there hence exists a unique globally defined continuous gradient trajectory of $f$ initialized at $x_0$ \\cite[Theorem 17.1.1]{attouch2014variational}. Also, $\\sigma_T(X_0)<\\infty$ since for any feasible point $x(\\cdot)$ of \\eqref{eq:sup_trajectory_T_smooth}, we have \n\\begin{subequations}\n\\begin{align}\n    \\int_0^T \\|x'(t)\\|dt & \\leqslant \\sqrt{T} \\sqrt{ \\int_0^T \\|x'(t)\\|^2 dt} \\label{eq:length_finite_smooth_a} \\\\\n    & = \\sqrt{T} \\sqrt{ -\\int_0^T \\langle \\nabla f(x(t)), x'(t) \\rangle dt} \\label{eq:length_finite_smooth_b} \\\\\n    & = \\sqrt{T} \\sqrt{ -\\int_0^T (f \\circ x)'(t) dt} \\label{eq:length_finite_smooth_c} \\\\\n    & = \\sqrt{T} \\sqrt{ f(x(0)) - f(x(T))} \\label{eq:length_finite_smooth_d} \\\\\n    & \\leqslant \\sqrt{T\\left(\\sup_{X_0} f - \\inf_{\\mathbb{R}^n} f\\right)} < \\infty. \\label{eq:length_finite_smooth_e}\n\\end{align}\n\\end{subequations}\nIndeed, \\eqref{eq:length_finite_smooth_a} is due to the Cauchy-Schwarz inequality. \\eqref{eq:length_finite_smooth_e} holds because $X_0$ is bounded and $f$ is lower bounded.\n","start":38442,"end":40391}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":["eq:local_error","eq:sup_trajectory_T_smooth"],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"连续轨迹的二阶余项界","source_units":[{"raw_tex":"It is easy to check that $L$ and $ML$ are respectively Lipschitz and gradient Lipschitz constants on $[0,T]$ of all continuous gradient trajectories of $f$ initialized in $X_0$. Indeed, let $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ be a continuous gradient trajectory initialized in $X_0$. Since $x(t) \\in B(X_0,\\sigma_T(X_0))$ for all $t\\in [0,T]$, we have $\\|x'(t)\\| = \\|\\nabla f(x(t))\\| \\leqslant L$. By the mean value theorem \\cite[5.19 Theorem]{rudin1964principles}, for all $s,t \\in [0,T]$ we have $\\|x'(t)-x'(s)\\| = \\|\\nabla f(x(t)) - \\nabla f(x(s))\\| \\leqslant M \\|x(t)-x(s)\\| \\leqslant M L |t-s|$. As a byproduct, we get the Taylor bound\n\\begin{subequations}\n    \\begin{align}\n    \\|x(t)-x(s) - x'(s)(t-s)\\| & = \\left\\| \\int_s^t (x'(\\tau) - x'(s)) d\\tau \\right\\| \\\\\n    & \\leqslant \\left| \\int_s^t \\| x'(\\tau) - x'(s) \\| d\\tau \\right| \\\\\n    & \\leqslant \\left|\\int_s^t ML | \\tau - s | d\\tau \\right| \\\\\n    & = \\frac{ML}{2}(t-s)^2. \\label{eq:local_error}\n    \\end{align}\n\\end{subequations}\n","start":40392,"end":41389}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_trajectory_T_smooth","local_a","local_b","local_c","local_d","local_e","local_f"],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"建立离散误差递推","source_units":[{"raw_tex":"Let $\\epsilon \\in (0,1)$ and let $\\bar{\\alpha} := 2\\epsilon e^{-MT}/(LMT)$. Consider $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$ and a sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$ and $x_0 \\in X_0$. Let $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ be the unique continuous gradient trajectory of $f$ initialized at $x_0$. Let $t_0:=0$ and $t_k := \\alpha_0+\\cdots+\\alpha_{k-1}$ when $k\\in \\mathbb{N}^*$ so that $\\|x_0 - x(t_0)\\| = 0 \\leqslant \\epsilon$. Assume that \\eqref{eq:tracking} holds up to some index $K$. For $k=0,\\hdots,K$, we have\n\\begin{subequations}\n    \\begin{align}\n        \\|x_{k+1} - x(t_{k+1})\\| = &  ~\\| x_k - \\alpha_k \\nabla f(x_k) - x(t_{k+1}) \\| \\label{local_a} \\\\[2mm]\n        \\leqslant  & ~ \\| x_k - \\alpha_k \\nabla f(x_k) - [x(t_k) - \\alpha_k \\nabla f(x(t_k))] \\|~+ \\label{local_b} \\\\[2mm] & ~ \\| x(t_k) - \\alpha_k \\nabla f(x(t_k)) - x(t_{k+1}) \\| \\label{local_c} \\\\[2mm]\n        \\leqslant & ~ \\|x_k - x(t_k)\\| + \\alpha_k \\|\\nabla f(x_k) - \\nabla f(x(t_k))\\| ~+ \\label{local_d} \\\\[2mm]\n        & ~ ML\\alpha_k^2/2 \\label{local_e} \\\\[2mm]\n        \\leqslant & ~ (1+\\alpha_k M)\\|x_k - x(t_k)\\| + ML\\alpha_k^2/2. \\label{local_f}\n    \\end{align}\n\\end{subequations}\nThe term in \\eqref{local_c} is equal to the local truncation error $\\|x(t_{k+1})  - x(t_k) - x'(t_k)(t_{k+1}-t_k) \\|$, and can hence be bounded above using \\eqref{eq:local_error}. In the second term of \\eqref{local_d}, we invoke a Lipschitz constant of $\\nabla f$ on the convex hull of $B(X_0,\\sigma_T(X_0)+1)$. Indeed, by the induction hypothesis and $\\epsilon < 1$, $x_k$ and $x(t_k)$ belong to $B(X_0,\\sigma_T(X_0)+\\epsilon)$. Hence \n","start":41390,"end":43099}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":["eq:global_a","eq:global_b","eq:global_c","eq:global_d","eq:global_e","eq:global_f","eq:global_g","eq:sup_trajectory_T_smooth"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"累积误差与步长选择","source_units":[{"raw_tex":"\\begin{subequations}\n    \\begin{align}\n        \\|x_{K+1} - x(t_{K+1})\\| \\leqslant &~ \\prod_{k=0}^K (1+\\alpha_kM)\\|x_0-x(t_0)\\|~+ \\label{eq:global_a}\\\\\n        &~ \\sum_{k=0}^K \\frac{ML\\alpha_k^2}{2} \\prod_{l=k+1}^{K} (1+\\alpha_lM)\\label{eq:global_b}\\\\\n        \\leqslant &~ \\prod_{l=0}^{K} (1+\\alpha_lM) \\sum_{k=0}^K \\frac{ML\\alpha_k^2}{2}\\label{eq:global_c}\\\\\n        \\leqslant &~ \\left(1+\\frac{M}{K+1}\\sum_{k=0}^K \\alpha_k\\right)^{K+1} \\sum_{k=0}^K \\frac{ML\\bar{\\alpha}\\alpha_k}{2}\\label{eq:global_d}\\\\\n        \\leqslant &~ \\left(1+\\frac{MT}{K+1}\\right)^{K+1} \\frac{ML\\bar{\\alpha}T}{2}\\label{eq:global_e}\\\\\n        \\leqslant &~~ \\bar{\\alpha} LMTe^{MT}/2\\label{eq:global_f}\\\\\n        = &~~ \\epsilon.\\label{eq:global_g}\n    \\end{align}\n\\end{subequations}\nThe term in \\eqref{eq:global_a} is equal to zero because $x_0 = x(t_0)$. By convention, we set the product in \\eqref{eq:global_b} to be equal to one if the index set for $l$ is empty. We bound the partial products ranging from $k+1$ to $K$ inside the summation of \\eqref{eq:global_b} by the full product ranging from $0$ to $K$ in order to obtain \\eqref{eq:global_c}. We then use the inequality of arithmetic and geometric means and $\\alpha_k\\leqslant\\bar{\\alpha}$ in order to obtain \\eqref{eq:global_d}. In order to prove the induction step, we assume that $\\alpha_0+\\cdots+\\alpha_K \\leqslant T$, hence \\eqref{eq:global_e}. \\eqref{eq:global_f} follows from $\\ln(1+x) \\leqslant x$ for all $x>0$. Finally, \\eqref{eq:global_g} holds because $\\bar{\\alpha} = 2\\epsilon e^{-MT}/(LMT)$.\n\\end{proof}","start":43099,"end":44644}]},{"id":"S037","semantic_type":"proposition","kind":"proposition","labels":["prop:KL"],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"跨临界值的统一 KL 不等式","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:KL}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz definable function and let $X$ be a bounded subset of $\\mathbb{R}^n$. Let $V$ be the set of critical values of $f$ in $\\overline{X}$ if it is nonempty, otherwise let $V:=\\{0\\}$. There exists a concave definable diffeomorphism $\\psi:[0,\\infty)\\rightarrow[0,\\infty)$ such that\n\\begin{equation*}\n    \\forall x \\in X \\setminus (\\partial \\tilde{f})^{-1}(0), ~~~ d(0,\\partial (\\psi \\circ \\tilde{f})(x)) \\geqslant 1\n\\end{equation*}\nwhere $\\tilde{f}(x) := d(f(x),V)$ for all $x\\in \\mathbb{R}^n$.\n\\end{proposition}","start":49370,"end":49978}]},{"id":"S038","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"统一 KL 不等式的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nBy the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, the set $V$ is finite. In addition, by the Kurdyka-\\L{}ojasiewicz inequality \\cite[Corollary 15]{bolte2007clarke}, there exist $\\epsilon>0$ and a strictly increasing concave continuous definable function $\\psi:[0,\\epsilon)\\rightarrow \\mathbb{R}$ that is continuously differentiable on $(0,\\epsilon)$ with $\\psi(0) = 0$ such that, for all $x \\in X$ and $v \\in V$ satisfying $0 < | f(x)-v | < \\epsilon$, we have $d(0,\\partial (\\psi \\circ | f-v|)(x)) \\geqslant 1$. The fact that $\\psi$ can be assumed to be concave is due to the monotonicity theorem \\cite[(1.2) p. 43]{van1998tame} \\cite[Lemma 2]{kurdyka1998gradients} by following the argument in \\cite[Theorem 4.1]{attouch2010proximal}. Let $\\rho:= \\inf \\{(v-v')/2: v,v' \\in V, v > v'\\}>0$ and $\\rho' : = \\min \\{ \\rho, \\epsilon\\}$. Observe that $c>0$ where\n\\begin{equation*}\n    c := \\inf \\{ \\|s\\| : \ns \\in \\partial f(x), ~ x\\in X, ~ | f(x) - v | \\geqslant \\rho',~ v\\in V \\}.\n\\end{equation*}\nIndeed, assume that there exist sequences $(s_k)_{k\\in \\mathbb{N}}$ and $(x_k)_{k\\in\\mathbb{N}}$ such that $\\|s_k\\|$ converges to zero and $s_k \\in \\partial f(x_k)$, $x_k\\in X$, and $| f(x_k) - v | \\geqslant \\rho'$ for all $v\\in V$. Since $X$ is bounded, there exist subsequences (again denoted $s_k$ and $x_k$) such that $x_k$ has a limit $x^*$ in $\\overline{X}$. Since $f$ is continuous, $| f(x^*) - v | \\geqslant \\rho'>0$ for all $v \\in V$. Also, since $s_k$ converges to zero, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990}, we have $0\\in \\partial f(x^*)$. This yields a contradiction. \n\nAfter possibly reducing $\\epsilon$, we may assume that $\\lim_{t\\nearrow\\rho'} \\psi'(t)>0$. If $\\lim_{t\\nearrow\\rho'} \\psi'(t) \\geqslant 1/c$, then we may extend $\\psi$ to an affine function on $[\\rho',\\infty)$ with slope equal to $\\lim_{t\\nearrow\\rho'} \\psi'(t)$. Otherwise, we can multiply $\\psi$ by $1/(c\\lim_{t\\nearrow\\rho'} \\psi'(t))$ before taking the affine extension. Now let $x \\in X$ be such that $0 \\notin \\partial \\tilde{f}(x)$. If $f(x)$ is strictly within $\\rho'$ distance of some $v \\in V$, then from $\\rho' \\leqslant \\rho$ it follows that $\\tilde{f}(\\tilde{x}) = d(f(\\tilde{x}),V) = | f(\\tilde{x})-v|$ for all $\\tilde{x}$ in a neighborhood of $x$ in $\\mathbb{R}^n$, and thus $d(0,\\partial ( \\psi \\circ \\tilde{f})(x)) = d(0,\\partial (\\psi \\circ | f-v|)(x)) \\geqslant 1$. The inequality is due to $\\rho' \\leqslant \\epsilon$. Otherwise, by \\cite[2.3.9 Theorem (Chain Rule I) (ii) p. 42]{clarke1990} we have $d(0,\\partial ( \\psi \\circ \\tilde{f})(x)) = \\psi'(\\tilde{f}(x)) d(0,\\partial \\tilde{f} (x)) = \\psi'(\\tilde{f}(x)) d(0,\\partial f(x)) \\geqslant \\psi'(\\tilde{f}(x)) c \\geqslant 1$. \n\\end{proof}","start":49979,"end":52712}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"有限临界值与局部凹 KL 函数","source_units":[{"raw_tex":"\\begin{proof}\nBy the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, the set $V$ is finite. In addition, by the Kurdyka-\\L{}ojasiewicz inequality \\cite[Corollary 15]{bolte2007clarke}, there exist $\\epsilon>0$ and a strictly increasing concave continuous definable function $\\psi:[0,\\epsilon)\\rightarrow \\mathbb{R}$ that is continuously differentiable on $(0,\\epsilon)$ with $\\psi(0) = 0$ such that, for all $x \\in X$ and $v \\in V$ satisfying $0 < | f(x)-v | < \\epsilon$, we have $d(0,\\partial (\\psi \\circ | f-v|)(x)) \\geqslant 1$. The fact that $\\psi$ can be assumed to be concave is due to the monotonicity theorem \\cite[(1.2) p. 43]{van1998tame} \\cite[Lemma 2]{kurdyka1998gradients} by following the argument in \\cite[Theorem 4.1]{attouch2010proximal}. Let $\\rho:= \\inf \\{(v-v')/2: v,v' \\in V, v > v'\\}>0$ and $\\rho' : = \\min \\{ \\rho, \\epsilon\\}$. Observe that $c>0$ where\n","start":49979,"end":50872}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"远离全部临界值的次梯度下界","source_units":[{"raw_tex":"\\begin{equation*}\n    c := \\inf \\{ \\|s\\| : \ns \\in \\partial f(x), ~ x\\in X, ~ | f(x) - v | \\geqslant \\rho',~ v\\in V \\}.\n\\end{equation*}\nIndeed, assume that there exist sequences $(s_k)_{k\\in \\mathbb{N}}$ and $(x_k)_{k\\in\\mathbb{N}}$ such that $\\|s_k\\|$ converges to zero and $s_k \\in \\partial f(x_k)$, $x_k\\in X$, and $| f(x_k) - v | \\geqslant \\rho'$ for all $v\\in V$. Since $X$ is bounded, there exist subsequences (again denoted $s_k$ and $x_k$) such that $x_k$ has a limit $x^*$ in $\\overline{X}$. Since $f$ is continuous, $| f(x^*) - v | \\geqslant \\rho'>0$ for all $v \\in V$. Also, since $s_k$ converges to zero, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990}, we have $0\\in \\partial f(x^*)$. This yields a contradiction. \n","start":50872,"end":51601}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"缩放延拓并验证统一不等式","source_units":[{"raw_tex":"\nAfter possibly reducing $\\epsilon$, we may assume that $\\lim_{t\\nearrow\\rho'} \\psi'(t)>0$. If $\\lim_{t\\nearrow\\rho'} \\psi'(t) \\geqslant 1/c$, then we may extend $\\psi$ to an affine function on $[\\rho',\\infty)$ with slope equal to $\\lim_{t\\nearrow\\rho'} \\psi'(t)$. Otherwise, we can multiply $\\psi$ by $1/(c\\lim_{t\\nearrow\\rho'} \\psi'(t))$ before taking the affine extension. Now let $x \\in X$ be such that $0 \\notin \\partial \\tilde{f}(x)$. If $f(x)$ is strictly within $\\rho'$ distance of some $v \\in V$, then from $\\rho' \\leqslant \\rho$ it follows that $\\tilde{f}(\\tilde{x}) = d(f(\\tilde{x}),V) = | f(\\tilde{x})-v|$ for all $\\tilde{x}$ in a neighborhood of $x$ in $\\mathbb{R}^n$, and thus $d(0,\\partial ( \\psi \\circ \\tilde{f})(x)) = d(0,\\partial (\\psi \\circ | f-v|)(x)) \\geqslant 1$. The inequality is due to $\\rho' \\leqslant \\epsilon$. Otherwise, by \\cite[2.3.9 Theorem (Chain Rule I) (ii) p. 42]{clarke1990} we have $d(0,\\partial ( \\psi \\circ \\tilde{f})(x)) = \\psi'(\\tilde{f}(x)) d(0,\\partial \\tilde{f} (x)) = \\psi'(\\tilde{f}(x)) d(0,\\partial f(x)) \\geqslant \\psi'(\\tilde{f}(x)) c \\geqslant 1$. \n\\end{proof}","start":51601,"end":52712}]},{"id":"S042","semantic_type":"proposition","kind":"proposition","labels":["prop:chain_rule"],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"可定义次梯度流的能量恒等式","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:chain_rule}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a definable function and let $x:(0,T)\\rightarrow \\mathbb{R}^n$ with $0 < T \\leqslant \\infty$ be an absolutely continuous function. If $f$ is locally Lipschitz at $x(t)$ for all $t \\in (0,T)$ and $x'(t) \\in -\\partial f(x(t))$ for almost every $t\\in(0,T)$, then\n\\begin{equation*}\n    (f \\circ x)'(t) = -\\|x'(t)\\|^2 = -d(0,\\partial f(x(t)))^2,~~~\\mathrm{for~a.e.}~t \\in (0,T).\n\\end{equation*}\n\\end{proposition}","start":52900,"end":53398}]},{"id":"S043","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"能量恒等式的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nSince $f$ is definable and locally Lipschitz at $x(t)$ for all $t\\in(0,T)$, it satisfies the chain rule \\cite[Proposition 2 (iv)]{bolte2020conservative} (see also \\cite[Theorem 5.8]{davis2020stochastic} and \\cite[Lemma 2.10]{drusvyatskiy2015quadratic}), that is to say $(f \\circ x)'(t) = \\langle s , x'(t) \\rangle$ for all $s\\in \\partial f(x(t))$ and almost every $t\\in (0,T)$. In particular, we may take $s := - x'(t) \\in \\partial f(x(t))$.\nFinally, by \\cite[Lemma 5.2]{davis2020stochastic} (see also \\cite[Proposition 4.10]{drusvyatskiy2015curves}) it holds that $\\|x'(t)\\| = d(0,\\partial f(x(t)))$ for almost every $t\\in (0,T)$. \n\\end{proof}","start":53399,"end":54057}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"沿曲线链式法则与能量下降","source_units":[{"raw_tex":"\\begin{proof}\nSince $f$ is definable and locally Lipschitz at $x(t)$ for all $t\\in(0,T)$, it satisfies the chain rule \\cite[Proposition 2 (iv)]{bolte2020conservative} (see also \\cite[Theorem 5.8]{davis2020stochastic} and \\cite[Lemma 2.10]{drusvyatskiy2015quadratic}), that is to say $(f \\circ x)'(t) = \\langle s , x'(t) \\rangle$ for all $s\\in \\partial f(x(t))$ and almost every $t\\in (0,T)$. In particular, we may take $s := - x'(t) \\in \\partial f(x(t))$.\n","start":53399,"end":53855}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"速度等于最小次梯度范数","source_units":[{"raw_tex":"Finally, by \\cite[Lemma 5.2]{davis2020stochastic} (see also \\cite[Proposition 4.10]{drusvyatskiy2015curves}) it holds that $\\|x'(t)\\| = d(0,\\partial f(x(t)))$ for almost every $t\\in (0,T)$. \n\\end{proof}","start":53855,"end":54057}]},{"id":"S046","semantic_type":"proposition","kind":"proposition","labels":["eq:length_continuous","prop:length_continuous"],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"连续轨迹的长度下降公式","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:length_continuous}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz definable function and let $X$ be a bounded subset of $\\mathbb{R}^n$. Assume that $f$ has at most $m\\in \\mathbb{N}^*$ critical values in $\\overline{X}$. Let $\\psi$ be a desingularizing function of $f$ on $X$. If $x:[0,T]\\rightarrow X$ with $T \\geqslant 0$ is an absolutely continuous function such that $x'(t) \\in -\\partial f(x(t))$ for almost every $t\\in(0,T)$, then\n\\begin{equation}\n\\label{eq:length_continuous}\n    \\frac{1}{2m}\\int_0^T\\|x'(t)\\|dt ~\\leqslant~ \\psi\\left(\\frac{f(x(0))-f(x(T))}{2m}\\right).\n\\end{equation}\n\\end{proposition}","start":54179,"end":54836}]},{"id":"S047","semantic_type":"proof","kind":"proof","labels":["eq:god_a","eq:god_b"],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"连续长度公式的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nWe first consider the special case where $0 \\notin \\partial \\tilde{f}(x(t))$ for all $t\\in (0,T)$ where $\\tilde{f}$ is defined as in Proposition \\ref{prop:KL}. Consider the change of variables $\\bar{x} = x \\circ \\varphi$ where $\\bar{t}=\\varphi^{-1}(t) = \\int_0^{t} ds /(\\psi' \\circ \\tilde{f} \\circ x)(s)$ and $t=\\varphi(\\bar{t}) = \\int_0^{\\bar{t}} (\\psi' \\circ \\tilde{f} \\circ \\bar{x})(\\bar{s}) d\\bar{s}$ by the inverse function theorem \\cite[Theorem 17.7.2]{tao2006analysis}. We have $\\bar{x}'(\\bar{t}) = \\varphi'(\\bar{t})x'(t) \\in -(\\psi' \\circ \\tilde{f} \\circ \\bar{x})(\\bar{t}) \\partial f(\\bar{x}(\\bar{t})) = \\pm \\partial (\\psi \\circ \\tilde{f})(\\bar{x}(\\bar{t}))$ for almost every $\\bar{t}\\in (0,\\overline{T})$, where $\\overline{T} = \\varphi^{-1}(T)$ and the sign is constant. By Proposition \\ref{prop:chain_rule} and the Cauchy-Schwarz inequality, we thus have\n\\begin{subequations}\n\\begin{align}\n    \\overline{T} \\leqslant & \\int_0^{\\overline{T}} d(0,\\partial (\\psi \\circ \\tilde{f})(\\bar{x}(\\bar{t})))d\\bar{t} \\label{eq:god_a} \\\\\n    = & \\int_0^{\\overline{T}} \\|\\bar{x}'(\\bar{t})\\|d\\bar{t} \\label{eq:god_b} \\\\\n    = & \\int_0^{T} \\|x'(t)\\|dt \\\\\n    \\leqslant & \\sqrt{\\overline{T}} \\sqrt{\\int_0^{\\overline{T}} \\|\\bar{x}'(\\bar{t})\\|^2d\\bar{t}} \\\\\n    = & \\sqrt{\\overline{T}}\\sqrt{|(\\psi\\circ \\tilde{f}\\circ\\bar{x})(0)-(\\psi\\circ \\tilde{f}\\circ\\bar{x})(\\overline{T})|}.\n\\end{align}\n\\end{subequations}\nHence $\\overline{T} \\leqslant |(\\psi\\circ \\tilde{f}\\circ\\bar{x})(0)-(\\psi\\circ \\tilde{f}\\circ\\bar{x})(\\overline{T})|$ and, since $\\psi$ is concave, we have\n\\begin{align*}\n     \\int_0^T \\|x'(t)\\|dt & \\leqslant |(\\psi\\circ \\tilde{f}\\circ\\bar{x})(0)-(\\psi\\circ \\tilde{f}\\circ\\bar{x})(\\overline{T})| \\\\\n     & \\leqslant \\psi (|( \\tilde{f}\\circ\\bar{x})(0)-(\\tilde{f}\\circ\\bar{x})(\\overline{T})|) \\\\[2mm]\n     & = \\psi (f(x(0))-f(x(T))).\n\\end{align*}\nWe next consider the general case where there may exist $t\\in (0,T)$ such that $0 \\in \\partial \\tilde{f}(x(t))$, namely $0 < t_1 < \\cdots < t_{k} < T$, and potentially $(t_k,T)$. For notational convenience, let $t_0 := 0$ and $t_{k+1}:=T$ where $k$ is possibly equal to zero. Since $k \\leqslant 2m-1$ and $\\psi$ is concave, we have\n\\begin{align*}\n    \\int_0^T \\|x'(t)\\|dt & = \\sum_{i=0}^k \\int_{t_i}^{t_{i+1}} \\|x'(t)\\|dt \\\\\n    & \\leqslant \\sum_{i=0}^k \\psi (f(x(t_i))-f(x(t_{i+1}))) \\\\[-.5mm]\n    & \\leqslant (k+1) ~ \\psi \\left( \\frac{1}{k+1} \\sum_{i=0}^k f(x(t_i))-f(x(t_{i+1}))\\right) \\\\\n    & = (k+1) ~ \\psi \\left( \\frac{f(x(0)) - f(x(T))}{k+1} \\right) \\\\\n    & \\leqslant 2m ~ \\psi \\left( \\frac{f(x(0)) - f(x(T))}{2m} \\right). \\qedhere\n\\end{align*}\n\\end{proof}","start":54837,"end":57462}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"无奇异区间上的重参数化","source_units":[{"raw_tex":"\\begin{proof}\nWe first consider the special case where $0 \\notin \\partial \\tilde{f}(x(t))$ for all $t\\in (0,T)$ where $\\tilde{f}$ is defined as in Proposition \\ref{prop:KL}. Consider the change of variables $\\bar{x} = x \\circ \\varphi$ where $\\bar{t}=\\varphi^{-1}(t) = \\int_0^{t} ds /(\\psi' \\circ \\tilde{f} \\circ x)(s)$ and $t=\\varphi(\\bar{t}) = \\int_0^{\\bar{t}} (\\psi' \\circ \\tilde{f} \\circ \\bar{x})(\\bar{s}) d\\bar{s}$ by the inverse function theorem \\cite[Theorem 17.7.2]{tao2006analysis}. We have $\\bar{x}'(\\bar{t}) = \\varphi'(\\bar{t})x'(t) \\in -(\\psi' \\circ \\tilde{f} \\circ \\bar{x})(\\bar{t}) \\partial f(\\bar{x}(\\bar{t})) = \\pm \\partial (\\psi \\circ \\tilde{f})(\\bar{x}(\\bar{t}))$ for almost every $\\bar{t}\\in (0,\\overline{T})$, where $\\overline{T} = \\varphi^{-1}(T)$ and the sign is constant. By Proposition \\ref{prop:chain_rule} and the Cauchy-Schwarz inequality, we thus have\n","start":54837,"end":55716}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["eq:god_a","eq:god_b"],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"积分 KL 得单段长度界","source_units":[{"raw_tex":"\\begin{subequations}\n\\begin{align}\n    \\overline{T} \\leqslant & \\int_0^{\\overline{T}} d(0,\\partial (\\psi \\circ \\tilde{f})(\\bar{x}(\\bar{t})))d\\bar{t} \\label{eq:god_a} \\\\\n    = & \\int_0^{\\overline{T}} \\|\\bar{x}'(\\bar{t})\\|d\\bar{t} \\label{eq:god_b} \\\\\n    = & \\int_0^{T} \\|x'(t)\\|dt \\\\\n    \\leqslant & \\sqrt{\\overline{T}} \\sqrt{\\int_0^{\\overline{T}} \\|\\bar{x}'(\\bar{t})\\|^2d\\bar{t}} \\\\\n    = & \\sqrt{\\overline{T}}\\sqrt{|(\\psi\\circ \\tilde{f}\\circ\\bar{x})(0)-(\\psi\\circ \\tilde{f}\\circ\\bar{x})(\\overline{T})|}.\n\\end{align}\n\\end{subequations}\nHence $\\overline{T} \\leqslant |(\\psi\\circ \\tilde{f}\\circ\\bar{x})(0)-(\\psi\\circ \\tilde{f}\\circ\\bar{x})(\\overline{T})|$ and, since $\\psi$ is concave, we have\n\\begin{align*}\n     \\int_0^T \\|x'(t)\\|dt & \\leqslant |(\\psi\\circ \\tilde{f}\\circ\\bar{x})(0)-(\\psi\\circ \\tilde{f}\\circ\\bar{x})(\\overline{T})| \\\\\n     & \\leqslant \\psi (|( \\tilde{f}\\circ\\bar{x})(0)-(\\tilde{f}\\circ\\bar{x})(\\overline{T})|) \\\\[2mm]\n     & = \\psi (f(x(0))-f(x(T))).\n\\end{align*}\n","start":55716,"end":56697}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"跨临界值分段并合并","source_units":[{"raw_tex":"We next consider the general case where there may exist $t\\in (0,T)$ such that $0 \\in \\partial \\tilde{f}(x(t))$, namely $0 < t_1 < \\cdots < t_{k} < T$, and potentially $(t_k,T)$. For notational convenience, let $t_0 := 0$ and $t_{k+1}:=T$ where $k$ is possibly equal to zero. Since $k \\leqslant 2m-1$ and $\\psi$ is concave, we have\n\\begin{align*}\n    \\int_0^T \\|x'(t)\\|dt & = \\sum_{i=0}^k \\int_{t_i}^{t_{i+1}} \\|x'(t)\\|dt \\\\\n    & \\leqslant \\sum_{i=0}^k \\psi (f(x(t_i))-f(x(t_{i+1}))) \\\\[-.5mm]\n    & \\leqslant (k+1) ~ \\psi \\left( \\frac{1}{k+1} \\sum_{i=0}^k f(x(t_i))-f(x(t_{i+1}))\\right) \\\\\n    & = (k+1) ~ \\psi \\left( \\frac{f(x(0)) - f(x(T))}{k+1} \\right) \\\\\n    & \\leqslant 2m ~ \\psi \\left( \\frac{f(x(0)) - f(x(T))}{2m} \\right). \\qedhere\n\\end{align*}\n\\end{proof}","start":56697,"end":57462}]},{"id":"S051","semantic_type":"proposition","kind":"proposition","labels":["eq:length_discrete","prop:length_discrete"],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"离散轨迹的长度下降公式","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:length_discrete}\nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a definable differentiable function with a locally Lipschitz gradient and let $X$ be a bounded open subset of $\\mathbb{R}^n$. Assume that $f$ has at most $m\\in \\mathbb{N}^*$ critical values in $\\overline{X}$ and that $L>0$ is a Lipschitz constant of $f$ on the convex hull of $X$. Let $\\psi$ be a desingularizing function of $f$ on $X$ and $\\epsilon>0$. There exists $\\bar{\\alpha}>0$ such that, for all $K \\in \\mathbb{N}$, $\\alpha_0,\\hdots,\\alpha_K \\in (0,\\bar{\\alpha}]$, and $(x_0,\\hdots,x_{K+1}) \\in X \\times \\cdots \\times X \\times \\mathbb{R}^n$ such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for $k = 0,\\hdots,K$, we have\n\\begin{equation}\n\\label{eq:length_discrete}\n    \\frac{1}{(2+\\epsilon)m}\\sum_{k=0}^K\\|x_{k+1}-x_k\\| ~\\leqslant~ \\psi\\left(\\frac{f(x_0)-f(x_K)}{2m}\\right) + \\frac{2L}{2+\\epsilon}\\max_{k=0,\\hdots,K} \\alpha_k\n\\end{equation}\nand $f(x_0)\\geqslant \\cdots \\geqslant f(x_{K+1})$.\n\\end{proposition}","start":57545,"end":58558}]},{"id":"S052","semantic_type":"proof","kind":"proof","labels":["eq:descent_lemma_1","eq:descent_lemma_2","eq:length_finite"],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"离散长度公式的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $M>0$ be a Lipschitz constant of $\\nabla f$ with respect to $\\|\\cdot\\|$ on the convex hull of $B(X,1):=X+B(0,1)$ and let $\\bar{\\alpha} := \\min\\{L^{-1},2\\epsilon(6+\\epsilon)^{-1}M^{-1}\\}$. Let $K \\in \\mathbb{N}$, $\\alpha_0,\\hdots,\\alpha_K \\in (0,\\bar{\\alpha}]$, and $(x_0,\\hdots,x_{K+1}) \\in X \\times \\cdots \\times X \\times \\mathbb{R}^n$ be such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for $k=0,\\hdots,K$. It holds that $\\|x_{K+1}-x_K\\| = \\alpha_K\\|\\nabla f(x_K)\\| \\leqslant \\bar{\\alpha}L \\leqslant 1$ and thus $x_0,\\hdots,x_{K+1} \\in B(X,1)$. A bound on the Taylor expansion of $f$ yields\n\\begin{align*}\n    f(x_{k+1})-f(x_k) & \\leqslant \\langle \\nabla f(x_k), x_{k+1}-x_k\\rangle + \\frac{M}{2}\\|x_{k+1}-x_k\\|^2 \\\\\n    & = \\left(\\frac{M\\alpha_k}{2}-1\\right) \\|x_{k+1}-x_k\\| \\|\\nabla f(x_k)\\|\n\\end{align*}\nfor $k=0,\\hdots,K$, and thus\n\\begin{equation}\n\\label{eq:descent_lemma_1}\n    \\|x_{k+1}-x_k\\|\\|\\nabla f(x_k)\\| \\leqslant \\frac{2}{2- M\\alpha_k}(f(x_k) - f(x_{k+1})).\n\\end{equation}\nWe also have\n    \\begin{align*}\n        \\|\\nabla f(x_{k+1})\\| & \\leqslant \\| \\nabla f(x_{k+1}) - \\nabla f(x_k) \\| + \\|\\nabla f(x_k)\\| \\\\\n        & \\leqslant M \\| x_{k+1} - x_k \\| + \\|\\nabla f(x_k)\\| \\\\\n        & \\leqslant (M\\alpha_k+1) \\|\\nabla f(x_k)\\|\n    \\end{align*}\nand thus \n\\begin{equation}\n\\label{eq:descent_lemma_2}\n    \\|x_{k+1}-x_k\\|\\|\\nabla f(x_{k+1})\\| \\leqslant \\frac{2+2M\\alpha_k}{2- M\\alpha_k}(f(x_k) - f(x_{k+1})).\n\\end{equation}\nNote that \n\\begin{equation*}\n    1 \\leqslant \\frac{2}{2- M\\alpha_k} \\leqslant \\frac{2+2M\\alpha_k}{2- M\\alpha_k} \\leqslant 1+\\frac{\\epsilon}{2}.\n\\end{equation*}\nLet $\\tilde{f}$ and $V$ be defined as in Proposition \\ref{prop:KL}.\n\nAssume that $[f(x_K),f(x_0)]$ excludes the elements of $V$ and the averages of any two consecutive elements of $V$. Then $0\\notin \\partial \\tilde{f}(x_k)$ and $1 \\leqslant \\| \\nabla(\\psi \\circ \\tilde{f})(x_k)\\| = \\psi'(\\tilde{f}(x_k)) \\|\\nabla \\tilde{f}(x_k)\\|$ for $k=0,\\hdots,K$. Let $k \\in \\{0,\\hdots,K-1\\}$. If $\\tilde{f}(x_k) \\geqslant \\tilde{f}(x_{k+1})$, then multiplying \\eqref{eq:descent_lemma_1} by $\\psi'(\\tilde{f}(x_k))$ and using the concavity of $\\psi$, we find that \n\\begin{align*}\n    \\|x_{k+1}-x_k\\| & \\leqslant (1+\\epsilon/2)\\psi'(\\tilde{f}(x_k))(f(x_k) - f(x_{k+1})) \\\\\n    & = (1+\\epsilon/2)\\psi'(\\tilde{f}(x_k))(\\tilde{f}(x_k) - \\tilde{f}(x_{k+1})) \\\\\n    & \\leqslant (1+\\epsilon/2)(\\psi (\\tilde{f}(x_k)) - \\psi (\\tilde{f}(x_{k+1}))).\n\\end{align*}\nIf $\\tilde{f}(x_k) \\leqslant \\tilde{f}(x_{k+1})$, then multiplying \\eqref{eq:descent_lemma_2} by $\\psi'(\\tilde{f}(x_{k+1}))$ and using the concavity of $\\psi$, we find that \n\\begin{align*}\n    \\|x_{k+1}-x_k\\| & \\leqslant (1+\\epsilon/2)\\psi'(\\tilde{f}(x_{k+1}))(f(x_k) - f(x_{k+1})) \\\\\n    & = (1+\\epsilon/2)\\psi'(\\tilde{f}(x_{k+1}))(\\tilde{f}(x_{k+1}) - \\tilde{f}(x_k)) \\\\\n    & \\leqslant (1+\\epsilon/2)(\\psi (\\tilde{f}(x_{k+1})) - \\psi (\\tilde{f}(x_k))).\n\\end{align*}\nAs a result,\n\\begin{equation*}\n    \\|x_{k+1}-x_k\\| \\leqslant (1+\\epsilon/2)|\\psi (\\tilde{f}(x_k)) - \\psi (\\tilde{f}(x_{k+1}))|, ~~~ k = 0,\\hdots,K-1.\n\\end{equation*}\nWe obtain the telescoping sum \n\\begin{subequations}\n    \\label{eq:length_finite} \n    \\begin{align}\n    \\sum_{k=0}^K \\|x_{k+1}-x_k\\| & \\leqslant (1+\\epsilon/2) |\\psi(\\tilde{f}(x_0)) - \\psi(\\tilde{f}(x_K))| + \\|x_{K+1} - x_K\\| \\\\[-4mm]\n    & \\leqslant (1+\\epsilon/2) \\psi ( | \\tilde{f}(x_0) - \\tilde{f}(x_K) |) + \\alpha_K\\|\\nabla f(x_K)\\| \\\\\n    & \\leqslant (1+\\epsilon/2) \\psi( f(x_0) - f(x_K)) + L\\max_{k=0,\\hdots,K} \\alpha_k.\n    \\end{align}\n\\end{subequations}\n\nAssume that $[f(x_K),f(x_0))$ excludes the elements of $V$ and the averages of any two consecutive elements of $V$. Since we are excluding a finite number of elements, there exists $\\epsilon>0$ such that $[f(x_K)-\\epsilon,f(x_0))$ satisfies the same property. If $x_0$ is a local minimum of $f$ on the open set $X$, then $\\nabla f(x_0) = 0$ by Fermat's rule. The length of $x_0,\\hdots,x_{K+1}$ is then equal to zero and the formula obtained in \\eqref{eq:length_finite} trivially holds. Otherwise, there exists a sequence $(x_0^l)_{l\\in\\mathbb{N}}$ in $\\mathbb{R}^n$ such that $f(x_0^l) < f(x_0)$ and $x_0^l$ converges to $x_0$. For every $l \\in \\mathbb{N}$, consider the sequence $(x^l_k)_{k\\in \\mathbb{N}}$ such that $x^l_{k+1} = x^l_k - \\alpha_k \\nabla f(x_k^l)$ for all $k \\in \\mathbb{N}$. Since $\\nabla f$ is continuous, for any fixed $k \\in \\{0,\\hdots,K\\}$, $x_k^l$ converges to $x_k \\in X$ and eventually belongs to the open set $X$. Thus $[f(x^l_K),f(x^l_0)] \\subset [f(x_K)-\\epsilon,f(x_0))$ eventually excludes the elements of $V$ and the averages of any two consecutive elements of $V$. We can apply the formula obtained in \\eqref{eq:length_finite}, namely\n\\begin{equation*}\n    \\sum_{k=0}^K \\|x^l_{k+1}-x^l_k\\| \\leqslant (1+\\epsilon/2) \\psi( f(x^l_0) - f(x^l_K)) + L\\max_{k=0,\\hdots,K} \\alpha_k.\n\\end{equation*}\nPassing to the limit yields\n\\begin{equation*}\n    \\sum_{k=0}^K \\|x_{k+1}-x_k\\| \\leqslant (1+\\epsilon/2) \\psi( f(x_0) - f(x_K)) + L\\max_{k=0,\\hdots,K} \\alpha_k.\n\\end{equation*}\n\nWe next consider the general case where\n\\begin{equation*}\n    [f(x_K),f(x_{K_p+1})) \\cup \\cdots \\cup [f(x_{K_2}),f(x_{K_1+1})) \\cup [f(x_{K_1}),f(x_0))\n\\end{equation*}\nexcludes the elements of $V$ and the averages of any two consecutive elements of $V$. For notational convenience, let $K_0 := -1$ and $K_{p+1} := K$. Since $p \\leqslant 2m-1$, we have\n\\begin{align*}\n    \\sum_{k=0}^K \\|x_{k+1}-x_k\\| = & \\sum_{i=0}^p \\sum_{k=K_i+1}^{K_{i+1}} \\|x_{k+1}-x_k\\| \\\\\n    \\leqslant & \\sum_{i=0}^p \\left((1+\\epsilon/2)\\psi( f(x_{{K_i}+1}) - f(x_{K_{i+1}})) + L\\max_{k=0,\\hdots,K} \\alpha_k\\right) \\\\[-.5mm]\n    \\leqslant & (1+\\epsilon/2)(p+1) ~ \\psi \\left( \\frac{1}{p+1} \\sum_{i=0}^p  f(x_{{K_i}+1}) - f(x_{K_{i+1}}) \\right) + \\\\ \n    & (p+1)L\\max_{k=0,\\hdots,K} \\alpha_k \\\\\n    \\leqslant & (1+\\epsilon/2)(p+1) \\psi \\left( \\frac{f(x_0)-f(x_K)}{p+1} \\right) + (p+1)L\\max_{k=0,\\hdots,K} \\alpha_k \\\\\n    \\leqslant & (2+\\epsilon)m ~ \\psi \\left( \\frac{f(x_0)-f(x_K)}{2m} \\right) + 2m L\\max_{k=0,\\hdots,K} \\alpha_k.\\qedhere\n\\end{align*}\n\\end{proof}","start":58559,"end":64644}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":["eq:descent_lemma_1","eq:length_finite"],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"选择步长并获得下降界","source_units":[{"raw_tex":"\\begin{proof}\nLet $M>0$ be a Lipschitz constant of $\\nabla f$ with respect to $\\|\\cdot\\|$ on the convex hull of $B(X,1):=X+B(0,1)$ and let $\\bar{\\alpha} := \\min\\{L^{-1},2\\epsilon(6+\\epsilon)^{-1}M^{-1}\\}$. Let $K \\in \\mathbb{N}$, $\\alpha_0,\\hdots,\\alpha_K \\in (0,\\bar{\\alpha}]$, and $(x_0,\\hdots,x_{K+1}) \\in X \\times \\cdots \\times X \\times \\mathbb{R}^n$ be such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for $k=0,\\hdots,K$. It holds that $\\|x_{K+1}-x_K\\| = \\alpha_K\\|\\nabla f(x_K)\\| \\leqslant \\bar{\\alpha}L \\leqslant 1$ and thus $x_0,\\hdots,x_{K+1} \\in B(X,1)$. A bound on the Taylor expansion of $f$ yields\n\\begin{align*}\n    f(x_{k+1})-f(x_k) & \\leqslant \\langle \\nabla f(x_k), x_{k+1}-x_k\\rangle + \\frac{M}{2}\\|x_{k+1}-x_k\\|^2 \\\\\n    & = \\left(\\frac{M\\alpha_k}{2}-1\\right) \\|x_{k+1}-x_k\\| \\|\\nabla f(x_k)\\|\n\\end{align*}\nfor $k=0,\\hdots,K$, and thus\n\\begin{equation}\n\\label{eq:descent_lemma_1}\n    \\|x_{k+1}-x_k\\|\\|\\nabla f(x_k)\\| \\leqslant \\frac{2}{2- M\\alpha_k}(f(x_k) - f(x_{k+1})).\n\\end{equation}\n","start":58559,"end":59567}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":["eq:descent_lemma_2","eq:length_finite"],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"相邻梯度估计与统一系数","source_units":[{"raw_tex":"We also have\n    \\begin{align*}\n        \\|\\nabla f(x_{k+1})\\| & \\leqslant \\| \\nabla f(x_{k+1}) - \\nabla f(x_k) \\| + \\|\\nabla f(x_k)\\| \\\\\n        & \\leqslant M \\| x_{k+1} - x_k \\| + \\|\\nabla f(x_k)\\| \\\\\n        & \\leqslant (M\\alpha_k+1) \\|\\nabla f(x_k)\\|\n    \\end{align*}\nand thus \n\\begin{equation}\n\\label{eq:descent_lemma_2}\n    \\|x_{k+1}-x_k\\|\\|\\nabla f(x_{k+1})\\| \\leqslant \\frac{2+2M\\alpha_k}{2- M\\alpha_k}(f(x_k) - f(x_{k+1})).\n\\end{equation}\nNote that \n\\begin{equation*}\n    1 \\leqslant \\frac{2}{2- M\\alpha_k} \\leqslant \\frac{2+2M\\alpha_k}{2- M\\alpha_k} \\leqslant 1+\\frac{\\epsilon}{2}.\n\\end{equation*}\n","start":59567,"end":60174}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["eq:length_finite"],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"单分支上的两种单步长度界","source_units":[{"raw_tex":"Let $\\tilde{f}$ and $V$ be defined as in Proposition \\ref{prop:KL}.\n\nAssume that $[f(x_K),f(x_0)]$ excludes the elements of $V$ and the averages of any two consecutive elements of $V$. Then $0\\notin \\partial \\tilde{f}(x_k)$ and $1 \\leqslant \\| \\nabla(\\psi \\circ \\tilde{f})(x_k)\\| = \\psi'(\\tilde{f}(x_k)) \\|\\nabla \\tilde{f}(x_k)\\|$ for $k=0,\\hdots,K$. Let $k \\in \\{0,\\hdots,K-1\\}$. If $\\tilde{f}(x_k) \\geqslant \\tilde{f}(x_{k+1})$, then multiplying \\eqref{eq:descent_lemma_1} by $\\psi'(\\tilde{f}(x_k))$ and using the concavity of $\\psi$, we find that \n\\begin{align*}\n    \\|x_{k+1}-x_k\\| & \\leqslant (1+\\epsilon/2)\\psi'(\\tilde{f}(x_k))(f(x_k) - f(x_{k+1})) \\\\\n    & = (1+\\epsilon/2)\\psi'(\\tilde{f}(x_k))(\\tilde{f}(x_k) - \\tilde{f}(x_{k+1})) \\\\\n    & \\leqslant (1+\\epsilon/2)(\\psi (\\tilde{f}(x_k)) - \\psi (\\tilde{f}(x_{k+1}))).\n\\end{align*}\nIf $\\tilde{f}(x_k) \\leqslant \\tilde{f}(x_{k+1})$, then multiplying \\eqref{eq:descent_lemma_2} by $\\psi'(\\tilde{f}(x_{k+1}))$ and using the concavity of $\\psi$, we find that \n\\begin{align*}\n    \\|x_{k+1}-x_k\\| & \\leqslant (1+\\epsilon/2)\\psi'(\\tilde{f}(x_{k+1}))(f(x_k) - f(x_{k+1})) \\\\\n    & = (1+\\epsilon/2)\\psi'(\\tilde{f}(x_{k+1}))(\\tilde{f}(x_{k+1}) - \\tilde{f}(x_k)) \\\\\n    & \\leqslant (1+\\epsilon/2)(\\psi (\\tilde{f}(x_{k+1})) - \\psi (\\tilde{f}(x_k))).\n\\end{align*}\nAs a result,\n\\begin{equation*}\n    \\|x_{k+1}-x_k\\| \\leqslant (1+\\epsilon/2)|\\psi (\\tilde{f}(x_k)) - \\psi (\\tilde{f}(x_{k+1}))|, ~~~ k = 0,\\hdots,K-1.\n\\end{equation*}\n","start":60174,"end":61647}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":["eq:length_finite"],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"望远镜求和并保留末步误差","source_units":[{"raw_tex":"We obtain the telescoping sum \n\\begin{subequations}\n    \\label{eq:length_finite} \n    \\begin{align}\n    \\sum_{k=0}^K \\|x_{k+1}-x_k\\| & \\leqslant (1+\\epsilon/2) |\\psi(\\tilde{f}(x_0)) - \\psi(\\tilde{f}(x_K))| + \\|x_{K+1} - x_K\\| \\\\[-4mm]\n    & \\leqslant (1+\\epsilon/2) \\psi ( | \\tilde{f}(x_0) - \\tilde{f}(x_K) |) + \\alpha_K\\|\\nabla f(x_K)\\| \\\\\n    & \\leqslant (1+\\epsilon/2) \\psi( f(x_0) - f(x_K)) + L\\max_{k=0,\\hdots,K} \\alpha_k.\n    \\end{align}\n\\end{subequations}\n","start":61647,"end":62110}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":["eq:length_finite"],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"半开端点的扰动极限","source_units":[{"raw_tex":"\nAssume that $[f(x_K),f(x_0))$ excludes the elements of $V$ and the averages of any two consecutive elements of $V$. Since we are excluding a finite number of elements, there exists $\\epsilon>0$ such that $[f(x_K)-\\epsilon,f(x_0))$ satisfies the same property. If $x_0$ is a local minimum of $f$ on the open set $X$, then $\\nabla f(x_0) = 0$ by Fermat's rule. The length of $x_0,\\hdots,x_{K+1}$ is then equal to zero and the formula obtained in \\eqref{eq:length_finite} trivially holds. Otherwise, there exists a sequence $(x_0^l)_{l\\in\\mathbb{N}}$ in $\\mathbb{R}^n$ such that $f(x_0^l) < f(x_0)$ and $x_0^l$ converges to $x_0$. For every $l \\in \\mathbb{N}$, consider the sequence $(x^l_k)_{k\\in \\mathbb{N}}$ such that $x^l_{k+1} = x^l_k - \\alpha_k \\nabla f(x_k^l)$ for all $k \\in \\mathbb{N}$. Since $\\nabla f$ is continuous, for any fixed $k \\in \\{0,\\hdots,K\\}$, $x_k^l$ converges to $x_k \\in X$ and eventually belongs to the open set $X$. Thus $[f(x^l_K),f(x^l_0)] \\subset [f(x_K)-\\epsilon,f(x_0))$ eventually excludes the elements of $V$ and the averages of any two consecutive elements of $V$. We can apply the formula obtained in \\eqref{eq:length_finite}, namely\n\\begin{equation*}\n    \\sum_{k=0}^K \\|x^l_{k+1}-x^l_k\\| \\leqslant (1+\\epsilon/2) \\psi( f(x^l_0) - f(x^l_K)) + L\\max_{k=0,\\hdots,K} \\alpha_k.\n\\end{equation*}\nPassing to the limit yields\n\\begin{equation*}\n    \\sum_{k=0}^K \\|x_{k+1}-x_k\\| \\leqslant (1+\\epsilon/2) \\psi( f(x_0) - f(x_K)) + L\\max_{k=0,\\hdots,K} \\alpha_k.\n\\end{equation*}\n","start":62110,"end":63610}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":["eq:length_finite"],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"离散分段与凹性合并","source_units":[{"raw_tex":"\nWe next consider the general case where\n\\begin{equation*}\n    [f(x_K),f(x_{K_p+1})) \\cup \\cdots \\cup [f(x_{K_2}),f(x_{K_1+1})) \\cup [f(x_{K_1}),f(x_0))\n\\end{equation*}\nexcludes the elements of $V$ and the averages of any two consecutive elements of $V$. For notational convenience, let $K_0 := -1$ and $K_{p+1} := K$. Since $p \\leqslant 2m-1$, we have\n\\begin{align*}\n    \\sum_{k=0}^K \\|x_{k+1}-x_k\\| = & \\sum_{i=0}^p \\sum_{k=K_i+1}^{K_{i+1}} \\|x_{k+1}-x_k\\| \\\\\n    \\leqslant & \\sum_{i=0}^p \\left((1+\\epsilon/2)\\psi( f(x_{{K_i}+1}) - f(x_{K_{i+1}})) + L\\max_{k=0,\\hdots,K} \\alpha_k\\right) \\\\[-.5mm]\n    \\leqslant & (1+\\epsilon/2)(p+1) ~ \\psi \\left( \\frac{1}{p+1} \\sum_{i=0}^p  f(x_{{K_i}+1}) - f(x_{K_{i+1}}) \\right) + \\\\ \n    & (p+1)L\\max_{k=0,\\hdots,K} \\alpha_k \\\\\n    \\leqslant & (1+\\epsilon/2)(p+1) \\psi \\left( \\frac{f(x_0)-f(x_K)}{p+1} \\right) + (p+1)L\\max_{k=0,\\hdots,K} \\alpha_k \\\\\n    \\leqslant & (2+\\epsilon)m ~ \\psi \\left( \\frac{f(x_0)-f(x_K)}{2m} \\right) + 2m L\\max_{k=0,\\hdots,K} \\alpha_k.\\qedhere\n\\end{align*}\n\\end{proof}","start":63610,"end":64644}]},{"id":"S059","semantic_type":"proof","kind":"proof","labels":["eq:decrease_continuous_a","eq:decrease_continuous_b","eq:decrease_continuous_c","eq:decrease_continuous_d","eq:decrease_continuous_e","eq:decrease_continuous_f","eq:f_bound_continuous","eq:finite_time","eq:sup_trajectory","eq:sup_trajectory_T","sec:Proof of Lemma lemma:uniform_length","upper_0","upper_1","upper_2"],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"统一连续长度引理的完整证明","source_units":[{"raw_tex":"\\section{Proof of Lemma \\ref{lemma:uniform_length}}\n\\label{sec:Proof of Lemma lemma:uniform_length}\n \nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz definable function with bounded continuous subgradient trajectories. Let $X_0$ be a bounded subset of $\\mathbb{R}^n$. We will show that $\\sigma(X_0) < \\infty$ where \n\\begin{subequations}\n\\label{eq:sup_trajectory}\n\\begin{align}\n    \\sigma(X_0) := & \\sup\\limits_{x \\in \\mathcal{A}(\\mathbb{R}_+,\\mathbb{R}^n)} ~~ \\int_0^{\\infty} \\|x'(t)\\|dt \\\\\n  & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx'(t) \\in - \\partial f(x(t)), ~\\mathrm{for~a.e.}~ t > 0,\\\\[3mm] x(0) \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\nWithout loss of generality, we may assume that $X_0\\neq \\emptyset$. By Propositions \\ref{prop:existence_trajectory} and \\ref{prop:bounded_global}, the feasible set of \\eqref{eq:sup_trajectory} is thus non-empty and $\\sigma(X_0)>-\\infty$.\n\nWe first consider the case with finite time horizon $T\\geqslant 0$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Notice that $\\sigma_T(X_0) \\leqslant \\sqrt{T(\\sup_{X_0} f - m(f))}$ where $m(f)$ is the smallest critical value of $f$ and \n\\begin{subequations}\n\\label{eq:sup_trajectory_T}\n\\begin{align}\n    \\sigma_T(X_0) := & \\sup\\limits_{x \\in \\mathcal{A}(\\mathbb{R}_+,\\mathbb{R}^n)} ~~ \\int_0^T \\|x'(t)\\|dt \\\\\n  & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx'(t) \\in - \\partial f(x(t)), ~\\mathrm{for~a.e.}~ t > 0,\\\\[3mm] x(0) \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\nIndeed, by the Cauchy-Schwarz inequality and Propositions \\ref{prop:chain_rule} and \\ref{prop:length_continuous}, for any feasible point $x(\\cdot)$ of \\eqref{eq:sup_trajectory_T} we have\n\\begin{subequations}\n\\begin{align}\n    \\int_0^T \\|x'(t)\\|dt & \\leqslant \\sqrt{T} \\sqrt{ \\int_0^T \\|x'(t)\\|^2 dt} \\\\\n    & \\leqslant \\sqrt{T} \\sqrt{ \\int_0^{\\infty} \\|x'(t)\\|^2 dt} \\\\\n    & = \\sqrt{T} \\sqrt{f(x(0)) - f\\left(\\lim_{t\\rightarrow\\infty} x(t)\\right)} \\\\\n    & \\leqslant \\sqrt{T\\left(\\sup_{X_0} f - m(f)\\right)}. \\label{eq:finite_time}\n\\end{align}\n\\end{subequations}\n\nWe next treat the case with infinite time horizon. Consider a sequence of feasible points $x_0(\\cdot), x_1(\\cdot), x_2(\\cdot), \\ldots$ of \\eqref{eq:sup_trajectory}  such that $\\int_0^{\\infty} \\|x_k'(t)\\|dt$ converges to $\\sigma(X_0)$. We proceed to show that the sequence is equicontinuous. Let $\\epsilon>0$ and $t\\geqslant 0$. Consider problem \\eqref{eq:sup_trajectory_T} with finite time horizon $T:=t+\\epsilon$. Since $x_0(\\cdot),x_1(\\cdot),x_2(\\cdot),\\ldots$ are feasible points of \\eqref{eq:sup_trajectory_T}, for all $s \\in [0,T]$ and $k=0,1,2,\\ldots$ we have\n\\begin{subequations}\n    \\begin{align}\n        \\|x_k(s)\\| & \\leqslant \\|x_k(s) - x_k(0) \\| + \\|x_k(0)\\| \\label{upper_0} \\\\ & \\leqslant \\int_0^s \\|x_k'(\\tau)\\|d\\tau + \\|x_k(0)\\| \\label{upper_1} \\\\\n        & \\leqslant \\sigma_T(X_0) + \\sup_{x_0 \\in X_0} \\|x_0\\|. \\label{upper_2}\n    \\end{align}\n\\end{subequations}\nAll the trajectories $x_0(\\cdot),x_1(\\cdot),x_2(\\cdot),\\ldots$ hence belong to a common ball centered at the origin up to time $T$. Let $L>0$ be a Lipschitz constant of $f$ on that ball. As a result, for all $s \\in [t-\\delta,t+\\delta]$ with $\\delta:=\\epsilon/(2L)$ and for all $k\\in \\mathbb{N}$, we have\n\\begin{equation*}\n    \\|x_k(t)-x_k(s)\\| = \\left\\| \\int_{s}^t x'_k(\\tau)d\\tau \\right\\| \\leqslant \\int_{t-\\delta}^{t+\\delta} \\|x'_k(\\tau)\\| d\\tau \\leqslant 2\\delta L = \\epsilon,\n\\end{equation*}\nIt follows that $x_0(\\cdot),x_1(\\cdot),x_2(\\cdot),\\ldots$ is equicontinuous on $\\mathbb{R}_+$. In addition, \\eqref{upper_0}-\\eqref{upper_2} imply that, for all $t\\geqslant 0$, the sequence $x_0(t),x_1(t),x_2(t),\\ldots$ is bounded. The Arzel\\`a-Ascoli theorem \\cite[Theorem 1 p. 13]{aubin1984differential} implies that there exists a subsequence (again denoted $x_k(\\cdot)$) converging uniformly over compact intervals to a continuous function $x:\\mathbb{R}_+\\rightarrow\\mathbb{R}^n$. \n\nWe next show that $x(\\cdot)$ is a continuous subgradient trajectory of $f$. Let $T\\geqslant 0$. By virtue of \\eqref{upper_0}-\\eqref{upper_2} and local Lipschitz continuity of $f$, the restrictions of $x'_0(\\cdot),x'_1(\\cdot),x'_2(\\cdot),\\ldots$ to $[0,T]$ lie in a ball of the dual space of $L^1([0,T],\\mathbb{R}^n)$, namely $L^{\\infty}([0,T],\\mathbb{R}^n)$. By the Banach-Alaoglu theorem \\cite[Theorem 3 p. 13]{aubin1984differential}, there exists a subsequence (again denoted $x_k'(\\cdot)$) that converges weakly\\textsuperscript{$\\ast$} to a function $y(\\cdot)$ in $L^{\\infty}([0,T],\\mathbb{R}^n)$. Together with $L^{\\infty}([0,T],\\mathbb{R}^n)\\subset L^1([0,T],\\mathbb{R}^n)$, we find that $x_k'(\\cdot)$ converges weakly to $y(\\cdot)$ in $L^1([0,T],\\mathbb{R}^n)$. Since $x_k(t)-x_k(s) = \\int_s^t x_k'(\\tau)d\\tau$ for all $0\\leqslant s \\leqslant t \\leqslant T$, we have $x(t)-x(s) = \\int_s^t y(\\tau)d\\tau$. As a result, $x'(t) = y(t)$ for almost every $t\\in (0,T)$. According to \\cite[Convergence Theorem p. 60]{aubin1984differential}, it follows that $x'(t) \\in -\\partial f(x(t))$ for almost every $t\\in(0,T)$. As $T \\geqslant 0$ was arbitrary, we have $x'(t) \\in -\\partial f(x(t))$ for almost every $t>0$.\n\n% https://www.math.ucdavis.edu/~hunter/pdes/ch6A.pdf\n\nSince $f$ is definable and has bounded continuous subgradient trajectories, by Proposition \\ref{prop:length_continuous} $x(\\cdot)$ has finite length and converges to a critical point $x^*$ of $f$. Let $\\epsilon>0$ and let $m \\in \\mathbb{N}^*$ be the number of critical values of $f$ in $B(\\overline{x(\\mathbb{R}_+)},\\epsilon)$. Let $\\psi$ be a desingularizing function of $f$ on $B(\\overline{x(\\mathbb{R}_+)},\\epsilon)$. Since $f$ is continuous, there exists $\\delta \\in (0,\\epsilon/2)$ such that \n\\begin{equation}\n\\label{eq:f_bound_continuous}\n    f(x) - f(x^*) \\leqslant m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right),~~~ \\forall x\\in B(x^*,\\delta).\n\\end{equation}\n\nLet $t^*\\geqslant 0$ be such that $\\|x(t)-x^*\\| \\leqslant \\delta/2$ for all $t\\geqslant t^*$. By the uniform convergence of $x_k(\\cdot)$, we have that $\\|x_k(t)-x(t)\\|\\leqslant \\delta/2$ for all $t\\in [0,t^*]$ for all $k$ large enough. Hence $\\|x_k(t^*) - x^*\\|\\leqslant \\|x_k(t^*)-x(t^*)\\|+\\|x(t^*)-x^*\\| \\leqslant \\delta/2 + \\delta/2 = \\delta$. If $T_k := \\inf\\{ t \\geqslant t^* : x_k(t) \\notin \\mathring{B}(x^*,\\epsilon)\\}<\\infty$, then\n\\begin{subequations}\n    \\begin{align}\n        2m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right) & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{\\epsilon-\\delta}{2m}\\right) \\label{eq:decrease_continuous_a} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{\\|x_k(T_k)-x^*\\| - \\|x_k(t^*)-x^*\\|}{2m}\\right) \\label{eq:decrease_continuous_b} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{\\|x_k(T_k)-x_k(t^*)\\|}{2m}\\right) \\label{eq:decrease_continuous_c} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{2m}\\int_{t^*}^{T_k}\\|x_k'(t)\\|dt \\right) \\label{eq:decrease_continuous_d} \\\\\n        & \\leqslant f(x_k(t^*))-f(x_k(T_k)) \\label{eq:decrease_continuous_e} \\\\[3mm]\n        & \\leqslant m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right) + f(x^*) - f(x_k(T_k)). \\label{eq:decrease_continuous_f}\n    \\end{align}\n\\end{subequations}\nAbove, \\eqref{eq:decrease_continuous_a} through \\eqref{eq:decrease_continuous_d} rely on the fact that $\\psi^{-1}$ is increasing. \\eqref{eq:decrease_continuous_a} is due to $\\delta < \\epsilon/2$. \\eqref{eq:decrease_continuous_b} holds because $\\|x_k(T_k)-x^*\\|\\geqslant \\epsilon$ by continuity of $x_k(\\cdot)$ and $x_k(t^*) \\in B(x^*,\\delta)$. \\eqref{eq:decrease_continuous_c} is a consequence of the triangular inequality. \\eqref{eq:decrease_continuous_d} holds because $x_k(\\cdot)$ is absolutely continuous. \\eqref{eq:decrease_continuous_e} is due to Proposition \\ref{prop:length_continuous} and the fact that $x_k(t) \\in B(x^*,\\epsilon)$ for all $t \\in [t^*,T_k]$ by continuity of $x_k(\\cdot)$. Finally, \\eqref{eq:decrease_continuous_f} is due to $x_k(t^*)\\in B(x^*,\\delta)$ and \\eqref{eq:f_bound_continuous}. Since $x_k(t) \\in B(\\overline{x(\\mathbb{R}_+)},\\epsilon)$ for all $t\\in [0,T_k]$ and $x_k(T_k)$ belongs to\n\\begin{equation*}\n     X_1 := B(x^*,\\epsilon) \\bigcap \\left\\{ x \\in \\mathbb{R}^n : f(x) \\leqslant f(x^*) - m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right) \\right\\},\n\\end{equation*}\nby Proposition \\ref{prop:length_continuous} and the definition of $\\sigma(\\cdot)$ in \\eqref{eq:sup_trajectory} we have \n    \\begin{align*}\n            \\int_0^{\\infty} \\|x_k'(t)\\|dt & = \\int_{0}^{T_k} \\|x_k'(t)\\|dt + \\int_{T_k}^{\\infty} \\|x_k'(t)\\|dt \\\\\n            & \\leqslant 2m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(x^*,\\epsilon)} f\\right) \\right) + \\max\\{0,\\sigma(X_1)\\}.\n    \\end{align*}\nNote that the inequality still holds if $T_k = \\infty$. By taking the limit, we get\n\\begin{equation*}\n    \\sigma(X_0) ~\\leqslant~ 2m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(x^*,\\epsilon)} f\\right) \\right) + \\max\\{0,\\sigma(X_1)\\}.\n\\end{equation*}\n\nIt now suffices to replace $X_0$ by $X_1$ and repeat the proof starting below \\eqref{eq:finite_time}. A maximizing sequence $\\bar{x}_k(\\cdot)$ corresponding to $\\sigma(X_1)$ converges to a continuous subgradient trajectory $\\bar{x}(\\cdot)$ whose initial point lies in the compact set $X_1$. If $X_1 \\neq \\emptyset$, then the critical value $f(\\lim_{t\\rightarrow\\infty} \\bar{x}(t))$ is less than or equal to $f(x^*) - m\\psi^{-1}(\\epsilon/(4m)) < f(x^*)$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Thus, it is eventually the case that one of the sets $X_0,X_1,\\hdots$ is empty. We conclude that $\\sigma(X_0)<\\infty$ by the above recursive formula. \n\n","start":72234,"end":81913}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_trajectory","eq:sup_trajectory_T","sec:Proof of Lemma lemma:uniform_length"],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"连续长度上确界与可行性","source_units":[{"raw_tex":"\\section{Proof of Lemma \\ref{lemma:uniform_length}}\n\\label{sec:Proof of Lemma lemma:uniform_length}\n \nLet $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz definable function with bounded continuous subgradient trajectories. Let $X_0$ be a bounded subset of $\\mathbb{R}^n$. We will show that $\\sigma(X_0) < \\infty$ where \n\\begin{subequations}\n\\label{eq:sup_trajectory}\n\\begin{align}\n    \\sigma(X_0) := & \\sup\\limits_{x \\in \\mathcal{A}(\\mathbb{R}_+,\\mathbb{R}^n)} ~~ \\int_0^{\\infty} \\|x'(t)\\|dt \\\\\n  & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx'(t) \\in - \\partial f(x(t)), ~\\mathrm{for~a.e.}~ t > 0,\\\\[3mm] x(0) \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\nWithout loss of generality, we may assume that $X_0\\neq \\emptyset$. By Propositions \\ref{prop:existence_trajectory} and \\ref{prop:bounded_global}, the feasible set of \\eqref{eq:sup_trajectory} is thus non-empty and $\\sigma(X_0)>-\\infty$.\n","start":72234,"end":73169}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":["eq:finite_time","eq:sup_trajectory","eq:sup_trajectory_T","sec:Proof of Lemma lemma:uniform_length"],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"由最小临界值控制有限时长度","source_units":[{"raw_tex":"We first consider the case with finite time horizon $T\\geqslant 0$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Notice that $\\sigma_T(X_0) \\leqslant \\sqrt{T(\\sup_{X_0} f - m(f))}$ where $m(f)$ is the smallest critical value of $f$ and \n\\begin{subequations}\n\\label{eq:sup_trajectory_T}\n\\begin{align}\n    \\sigma_T(X_0) := & \\sup\\limits_{x \\in \\mathcal{A}(\\mathbb{R}_+,\\mathbb{R}^n)} ~~ \\int_0^T \\|x'(t)\\|dt \\\\\n  & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx'(t) \\in - \\partial f(x(t)), ~\\mathrm{for~a.e.}~ t > 0,\\\\[3mm] x(0) \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\nIndeed, by the Cauchy-Schwarz inequality and Propositions \\ref{prop:chain_rule} and \\ref{prop:length_continuous}, for any feasible point $x(\\cdot)$ of \\eqref{eq:sup_trajectory_T} we have\n\\begin{subequations}\n\\begin{align}\n    \\int_0^T \\|x'(t)\\|dt & \\leqslant \\sqrt{T} \\sqrt{ \\int_0^T \\|x'(t)\\|^2 dt} \\\\\n    & \\leqslant \\sqrt{T} \\sqrt{ \\int_0^{\\infty} \\|x'(t)\\|^2 dt} \\\\\n    & = \\sqrt{T} \\sqrt{f(x(0)) - f\\left(\\lim_{t\\rightarrow\\infty} x(t)\\right)} \\\\\n    & \\leqslant \\sqrt{T\\left(\\sup_{X_0} f - m(f)\\right)}. \\label{eq:finite_time}\n\\end{align}\n\\end{subequations}\n","start":73170,"end":74397}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_trajectory","eq:sup_trajectory_T","sec:Proof of Lemma lemma:uniform_length","upper_0","upper_1","upper_2"],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"极大化序列的紧时间紧性","source_units":[{"raw_tex":"We next treat the case with infinite time horizon. Consider a sequence of feasible points $x_0(\\cdot), x_1(\\cdot), x_2(\\cdot), \\ldots$ of \\eqref{eq:sup_trajectory}  such that $\\int_0^{\\infty} \\|x_k'(t)\\|dt$ converges to $\\sigma(X_0)$. We proceed to show that the sequence is equicontinuous. Let $\\epsilon>0$ and $t\\geqslant 0$. Consider problem \\eqref{eq:sup_trajectory_T} with finite time horizon $T:=t+\\epsilon$. Since $x_0(\\cdot),x_1(\\cdot),x_2(\\cdot),\\ldots$ are feasible points of \\eqref{eq:sup_trajectory_T}, for all $s \\in [0,T]$ and $k=0,1,2,\\ldots$ we have\n\\begin{subequations}\n    \\begin{align}\n        \\|x_k(s)\\| & \\leqslant \\|x_k(s) - x_k(0) \\| + \\|x_k(0)\\| \\label{upper_0} \\\\ & \\leqslant \\int_0^s \\|x_k'(\\tau)\\|d\\tau + \\|x_k(0)\\| \\label{upper_1} \\\\\n        & \\leqslant \\sigma_T(X_0) + \\sup_{x_0 \\in X_0} \\|x_0\\|. \\label{upper_2}\n    \\end{align}\n\\end{subequations}\nAll the trajectories $x_0(\\cdot),x_1(\\cdot),x_2(\\cdot),\\ldots$ hence belong to a common ball centered at the origin up to time $T$. Let $L>0$ be a Lipschitz constant of $f$ on that ball. As a result, for all $s \\in [t-\\delta,t+\\delta]$ with $\\delta:=\\epsilon/(2L)$ and for all $k\\in \\mathbb{N}$, we have\n\\begin{equation*}\n    \\|x_k(t)-x_k(s)\\| = \\left\\| \\int_{s}^t x'_k(\\tau)d\\tau \\right\\| \\leqslant \\int_{t-\\delta}^{t+\\delta} \\|x'_k(\\tau)\\| d\\tau \\leqslant 2\\delta L = \\epsilon,\n\\end{equation*}\nIt follows that $x_0(\\cdot),x_1(\\cdot),x_2(\\cdot),\\ldots$ is equicontinuous on $\\mathbb{R}_+$. In addition, \\eqref{upper_0}-\\eqref{upper_2} imply that, for all $t\\geqslant 0$, the sequence $x_0(t),x_1(t),x_2(t),\\ldots$ is bounded. The Arzel\\`a-Ascoli theorem \\cite[Theorem 1 p. 13]{aubin1984differential} implies that there exists a subsequence (again denoted $x_k(\\cdot)$) converging uniformly over compact intervals to a continuous function $x:\\mathbb{R}_+\\rightarrow\\mathbb{R}^n$. \n","start":74398,"end":76256}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_trajectory","eq:sup_trajectory_T","sec:Proof of Lemma lemma:uniform_length"],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"紧性极限仍满足微分包含","source_units":[{"raw_tex":"\nWe next show that $x(\\cdot)$ is a continuous subgradient trajectory of $f$. Let $T\\geqslant 0$. By virtue of \\eqref{upper_0}-\\eqref{upper_2} and local Lipschitz continuity of $f$, the restrictions of $x'_0(\\cdot),x'_1(\\cdot),x'_2(\\cdot),\\ldots$ to $[0,T]$ lie in a ball of the dual space of $L^1([0,T],\\mathbb{R}^n)$, namely $L^{\\infty}([0,T],\\mathbb{R}^n)$. By the Banach-Alaoglu theorem \\cite[Theorem 3 p. 13]{aubin1984differential}, there exists a subsequence (again denoted $x_k'(\\cdot)$) that converges weakly\\textsuperscript{$\\ast$} to a function $y(\\cdot)$ in $L^{\\infty}([0,T],\\mathbb{R}^n)$. Together with $L^{\\infty}([0,T],\\mathbb{R}^n)\\subset L^1([0,T],\\mathbb{R}^n)$, we find that $x_k'(\\cdot)$ converges weakly to $y(\\cdot)$ in $L^1([0,T],\\mathbb{R}^n)$. Since $x_k(t)-x_k(s) = \\int_s^t x_k'(\\tau)d\\tau$ for all $0\\leqslant s \\leqslant t \\leqslant T$, we have $x(t)-x(s) = \\int_s^t y(\\tau)d\\tau$. As a result, $x'(t) = y(t)$ for almost every $t\\in (0,T)$. According to \\cite[Convergence Theorem p. 60]{aubin1984differential}, it follows that $x'(t) \\in -\\partial f(x(t))$ for almost every $t\\in(0,T)$. As $T \\geqslant 0$ was arbitrary, we have $x'(t) \\in -\\partial f(x(t))$ for almost every $t>0$.\n\n% https://www.math.ucdavis.edu/~hunter/pdes/ch6A.pdf\n\n","start":76256,"end":77523}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":["eq:f_bound_continuous","eq:sup_trajectory","eq:sup_trajectory_T","sec:Proof of Lemma lemma:uniform_length"],"member_ids":["S064"],"declared_external":false,"source_only":false,"title":"极限临界点附近的函数值界","source_units":[{"raw_tex":"Since $f$ is definable and has bounded continuous subgradient trajectories, by Proposition \\ref{prop:length_continuous} $x(\\cdot)$ has finite length and converges to a critical point $x^*$ of $f$. Let $\\epsilon>0$ and let $m \\in \\mathbb{N}^*$ be the number of critical values of $f$ in $B(\\overline{x(\\mathbb{R}_+)},\\epsilon)$. Let $\\psi$ be a desingularizing function of $f$ on $B(\\overline{x(\\mathbb{R}_+)},\\epsilon)$. Since $f$ is continuous, there exists $\\delta \\in (0,\\epsilon/2)$ such that \n\\begin{equation}\n\\label{eq:f_bound_continuous}\n    f(x) - f(x^*) \\leqslant m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right),~~~ \\forall x\\in B(x^*,\\delta).\n\\end{equation}\n","start":77523,"end":78187}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":["eq:decrease_continuous_a","eq:decrease_continuous_b","eq:decrease_continuous_c","eq:decrease_continuous_d","eq:decrease_continuous_e","eq:decrease_continuous_f","eq:sup_trajectory","eq:sup_trajectory_T","sec:Proof of Lemma lemma:uniform_length"],"member_ids":["S065"],"declared_external":false,"source_only":false,"title":"退出临界点邻域必然降值","source_units":[{"raw_tex":"Let $t^*\\geqslant 0$ be such that $\\|x(t)-x^*\\| \\leqslant \\delta/2$ for all $t\\geqslant t^*$. By the uniform convergence of $x_k(\\cdot)$, we have that $\\|x_k(t)-x(t)\\|\\leqslant \\delta/2$ for all $t\\in [0,t^*]$ for all $k$ large enough. Hence $\\|x_k(t^*) - x^*\\|\\leqslant \\|x_k(t^*)-x(t^*)\\|+\\|x(t^*)-x^*\\| \\leqslant \\delta/2 + \\delta/2 = \\delta$. If $T_k := \\inf\\{ t \\geqslant t^* : x_k(t) \\notin \\mathring{B}(x^*,\\epsilon)\\}<\\infty$, then\n\\begin{subequations}\n    \\begin{align}\n        2m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right) & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{\\epsilon-\\delta}{2m}\\right) \\label{eq:decrease_continuous_a} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{\\|x_k(T_k)-x^*\\| - \\|x_k(t^*)-x^*\\|}{2m}\\right) \\label{eq:decrease_continuous_b} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{\\|x_k(T_k)-x_k(t^*)\\|}{2m}\\right) \\label{eq:decrease_continuous_c} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{2m}\\int_{t^*}^{T_k}\\|x_k'(t)\\|dt \\right) \\label{eq:decrease_continuous_d} \\\\\n        & \\leqslant f(x_k(t^*))-f(x_k(T_k)) \\label{eq:decrease_continuous_e} \\\\[3mm]\n        & \\leqslant m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right) + f(x^*) - f(x_k(T_k)). \\label{eq:decrease_continuous_f}\n    \\end{align}\n\\end{subequations}\nAbove, \\eqref{eq:decrease_continuous_a} through \\eqref{eq:decrease_continuous_d} rely on the fact that $\\psi^{-1}$ is increasing. \\eqref{eq:decrease_continuous_a} is due to $\\delta < \\epsilon/2$. \\eqref{eq:decrease_continuous_b} holds because $\\|x_k(T_k)-x^*\\|\\geqslant \\epsilon$ by continuity of $x_k(\\cdot)$ and $x_k(t^*) \\in B(x^*,\\delta)$. \\eqref{eq:decrease_continuous_c} is a consequence of the triangular inequality. \\eqref{eq:decrease_continuous_d} holds because $x_k(\\cdot)$ is absolutely continuous. \\eqref{eq:decrease_continuous_e} is due to Proposition \\ref{prop:length_continuous} and the fact that $x_k(t) \\in B(x^*,\\epsilon)$ for all $t \\in [t^*,T_k]$ by continuity of $x_k(\\cdot)$. Finally, \\eqref{eq:decrease_continuous_f} is due to $x_k(t^*)\\in B(x^*,\\delta)$ and \\eqref{eq:f_bound_continuous}. Since $x_k(t) \\in B(\\overline{x(\\mathbb{R}_+)},\\epsilon)$ for all $t\\in [0,T_k]$ and $x_k(T_k)$ belongs to\n\\begin{equation*}\n     X_1 := B(x^*,\\epsilon) \\bigcap \\left\\{ x \\in \\mathbb{R}^n : f(x) \\leqslant f(x^*) - m~\\psi^{-1}\\left(\\frac{\\epsilon}{4m}\\right) \\right\\},\n\\end{equation*}\n","start":78188,"end":80528}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_trajectory","eq:sup_trajectory_T","sec:Proof of Lemma lemma:uniform_length"],"member_ids":["S066"],"declared_external":false,"source_only":false,"title":"分割长度得到连续递归","source_units":[{"raw_tex":"by Proposition \\ref{prop:length_continuous} and the definition of $\\sigma(\\cdot)$ in \\eqref{eq:sup_trajectory} we have \n    \\begin{align*}\n            \\int_0^{\\infty} \\|x_k'(t)\\|dt & = \\int_{0}^{T_k} \\|x_k'(t)\\|dt + \\int_{T_k}^{\\infty} \\|x_k'(t)\\|dt \\\\\n            & \\leqslant 2m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(x^*,\\epsilon)} f\\right) \\right) + \\max\\{0,\\sigma(X_1)\\}.\n    \\end{align*}\nNote that the inequality still holds if $T_k = \\infty$. By taking the limit, we get\n\\begin{equation*}\n    \\sigma(X_0) ~\\leqslant~ 2m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(x^*,\\epsilon)} f\\right) \\right) + \\max\\{0,\\sigma(X_1)\\}.\n\\end{equation*}\n","start":80528,"end":81194}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_trajectory","eq:sup_trajectory_T","sec:Proof of Lemma lemma:uniform_length"],"member_ids":["S067"],"declared_external":false,"source_only":false,"title":"有限临界值终止连续递归","source_units":[{"raw_tex":"\nIt now suffices to replace $X_0$ by $X_1$ and repeat the proof starting below \\eqref{eq:finite_time}. A maximizing sequence $\\bar{x}_k(\\cdot)$ corresponding to $\\sigma(X_1)$ converges to a continuous subgradient trajectory $\\bar{x}(\\cdot)$ whose initial point lies in the compact set $X_1$. If $X_1 \\neq \\emptyset$, then the critical value $f(\\lim_{t\\rightarrow\\infty} \\bar{x}(t))$ is less than or equal to $f(x^*) - m\\psi^{-1}(\\epsilon/(4m)) < f(x^*)$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Thus, it is eventually the case that one of the sets $X_0,X_1,\\hdots$ is empty. We conclude that $\\sigma(X_0)<\\infty$ by the above recursive formula. \n\n","start":81194,"end":81913}]},{"id":"S068","semantic_type":"proof","kind":"proof","labels":["eq:decrease_discrete_a","eq:decrease_discrete_b","eq:decrease_discrete_c","eq:decrease_discrete_d","eq:decrease_discrete_e","eq:decrease_discrete_f","eq:decrease_discrete_g","eq:f_bound_discrete","eq:length_discrete_K","eq:sup_discrete_X_0","eq:tracking_T","sec:Proof of Theorem thm:convergence"],"member_ids":["S068"],"declared_external":false,"source_only":false,"title":"三项等价的完整主证明","source_units":[{"raw_tex":"\\section{Proof of Theorem \\ref{thm:convergence}}\n\\label{sec:Proof of Theorem thm:convergence}\n\n($1\\Longrightarrow 2$) We prove the contrapositive. Assume that there exists $0< T \\leqslant \\infty$ and a differentiable function $x:[0,T)\\rightarrow\\mathbb{R}^n$ such that\n\\begin{equation*}\n    x'(t) = -\\nabla f(x(t)), ~~~ \\forall t \\in (0,T),\n\\end{equation*}\nfor which, for all $c>0$, there exists $t\\in [0,T)$ such that $\\|x(t)\\| > c$. Let $x_0 := x(0)$. Let $\\bar{\\alpha}$ and $c$ be some positive constants. Let $\\tilde{t} \\in (0,T)$ be such that $\\|x(\\tilde{t})\\| \\geqslant c+2$. By Proposition \\ref{prop:tracking} and Remark \\ref{rem:tracking}, there exists $\\tilde{\\alpha}\\in (0,\\bar{\\alpha}]$ such that for all $\\alpha \\in (0,\\tilde{\\alpha}]$, the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by $x_{k+1} = x_k -\\alpha \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$ satisfies $\\|x_k-x(k\\alpha)\\| \\leqslant 1/2$ for all $k\\in \\{0,\\hdots, \\lfloor \\tilde{t}/\\alpha \\rfloor \\}$. We use the notation $\\lfloor t \\rfloor$ to denote the floor of a real number $t$ which is the unique integer such that $\\lfloor t \\rfloor \\leqslant t < \\lfloor t \\rfloor + 1$. If $\\tilde{k} = \\lfloor \\tilde{t}/\\alpha \\rfloor$, then $\\|x_{\\tilde{k}} - x(\\tilde{t})\\| \\leqslant \\|x_{\\tilde{k}}-x(\\tilde{k}\\alpha)\\|+\\|x(\\tilde{k}\\alpha)-x(\\tilde{t})\\| \\leqslant 1/2 + \\tilde{L}| \\tilde{k}\\alpha-\\tilde{t}| \\leqslant 1/2 + \\tilde{L}\\alpha$ where $\\tilde{L}>0$ is a Lipschitz constant of $f$ on the convex hull of $x([\\tilde{t}-\\tilde{\\alpha},\\tilde{t}])$. Let $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$ be the constant sequence equal to $\\min \\{\\tilde{\\alpha}, 1/(2\\tilde{L})\\}$ and consider the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by $x_{k+1} = x_k -\\alpha_k \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$. By the triangular inequality, we have $\\|x_{\\tilde{k}}\\| \\geqslant \\|x(\\tilde{t})\\| - \\|x_{\\tilde{k}}-x(\\tilde{t})\\| \\geqslant c+2 - 1/2 - 1/2>c$ where $\\tilde{k} := \\lfloor \\tilde{t}/\\alpha \\rfloor$.\n\n($2 \\Longrightarrow 3$) Let $X_0$ be a bounded subset of $\\mathbb{R}^n$. We will show that there exists $\\bar{\\alpha}>0$ such that $\\sigma(X_0,\\bar{\\alpha})<\\infty$ where\n\\begin{subequations}\n\\label{eq:sup_discrete_X_0}\n\\begin{align}\n    \\sigma(X_0,\\bar{\\alpha}) := & \\sup\\limits_{\\tiny \\begin{array}{c}x \\in (\\mathbb{R}^n)^{\\mathbb{N}}\\\\ \\alpha \\in (0,\\bar{\\alpha}]^\\mathbb{N} \\end{array}} ~~ \\sum\\limits_{k=0}^{\\infty} \\|x_{k+1}-x_k\\| \\\\\n   & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx_{k+1} = x_k - \\alpha_k \\nabla f(x_k), ~ \\forall k\\in \\mathbb{N},\\\\[2mm] x_0 \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\n\nLet $\\Phi:\\mathbb{R}_+ \\times \\mathbb{R}^n \\rightarrow \\mathbb{R}^n$ be the gradient flow of $f$ defined for all $(t,x_0) \\in \\mathbb{R}_+ \\times \\mathbb{R}^n$ by $\\Phi(t,x_0) := x(t)$ where $x(\\cdot)$ is the unique continuous gradient trajectory of $f$ initialized at $x_0$. Uniqueness follows from the Picard–Lindel{\\\"o}f theorem \\cite[Theorem 3.1 p. 12]{coddington1955theory}. Let $C$ be the set of critical points of $f$ in $\\overline{\\Phi(\\mathbb{R}_+,X_0)}$. If $X_0 \\subset C$, then we have $\\sigma(X_0,\\bar{\\alpha}) \\leqslant 0$ for all $\\bar{\\alpha}>0$ and there is nothing left to prove. Otherwise, since $C$ is closed by \\cite[2.1.5 Proposition p. 29]{clarke1990}, there exists $\\epsilon>0$ be such that $X_0\\setminus \\mathring{B}(C,\\epsilon/6) \\neq \\emptyset$ where $\\mathring{B}(C,\\epsilon/6) := C + \\mathring{B}(0,\\epsilon/6)$. It is safe to assume this from now on.\n\nLet $\\psi$ be a desingularizing function of $f$ on $\\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$, where we use the shorthand $\\Phi_0 := \\Phi(\\mathbb{R}_+,X_0)$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Let $m \\in \\mathbb{N}$ be the number of critical values of $f$ in $B(\\overline{\\Phi}_0,\\epsilon)$. Note that $m \\geqslant 1$ due to Proposition \\ref{prop:length_continuous} and the fact that $X_0 \\neq \\emptyset$. Since $f$ is continuous and $C$ is compact by Lemma \\ref{lemma:uniform_length}, there exists $\\delta \\in (0,\\epsilon/2)$ such that  \n\\begin{equation}\n    \\label{eq:f_bound_discrete}\n    f(x) - \\max_{C} f \\leqslant m~ \\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right),~~~ \\forall x\\in B(C,\\delta).\n\\end{equation}\nNote that $\\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$ is bounded due to Lemma \\ref{lemma:uniform_length}. Let $L>0$ be a Lipschitz constant of $f$ on the convex hull of $\\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$. By Proposition \\ref{prop:length_discrete}, there exists $\\alpha_\\delta \\in (0,\\delta/(5mL)]$ such that, for all $K \\in \\mathbb{N}^*$, $\\alpha_0,\\hdots,\\alpha_{K-1} \\in (0,\\alpha_\\delta]$, and $(x_0,\\hdots,x_K) \\in \\mathring{B}(\\overline{\\Phi}_0,\\epsilon) \\times \\cdots \\times \\mathring{B}(\\overline{\\Phi}_0,\\epsilon) \\times \\mathbb{R}^n$ such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for $k = 0,\\hdots,K-1$, we have\n\\begin{equation}\n\\label{eq:length_discrete_K}\n    \\frac{1}{4m}\\sum_{k=0}^{K-1}\\|x_{k+1}-x_k\\| ~\\leqslant~ \\psi\\left(\\frac{f(x_0)-f(x_{K-1})}{2m}\\right) + \\frac{L}{2}\\max_{k=0,\\hdots,K-1} \\alpha_k.\n\\end{equation}\n\nSince $\\nabla f$ is continuous, its norm attains its infimum $\\nu$ on the nonempty compact set $\\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3)$. \nIt is nonempty because $\\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3) \\supset X_0\\setminus \\mathring{B}(C,\\epsilon/6) \\neq \\emptyset$ and $\\delta < \\epsilon/2$.\nIf $\\nu=0$, then there exists $x^* \\in \\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3)$ such that $\\|\\nabla f(x^*)\\| = 0$. Then $x^*\\in C \\setminus \\mathring{B}(C,\\delta/3)$, which is a contradiction. We thus have $\\nu>0$. Hence we may define $T := 2\\sigma(X_0)/\\nu$ where $\\sigma(X_0)$ is defined in \\eqref{eq:sup_trajectory} and is finite by Lemma \\ref{lemma:uniform_length}. Note that $\\sigma(X_0)>0$ and thus $T>0$ because $X_0\\not\\subset C$. By Proposition \\ref{prop:tracking}, there exists $\\bar{\\alpha} \\in (0,\\alpha_\\delta]$ such that for any feasible point $((x_k)_{k\\in \\mathbb{N}} , (\\alpha_k)_{k\\in\\mathbb{N}})$ of \\eqref{eq:sup_discrete_X_0},\nthe continuous gradient trajectory $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ of $f$ initialized at $x_0$ satisfies\n\\begin{equation}\n\\label{eq:tracking_T}\n   \\forall k\\in \\mathbb{N}^*,~~~ \\alpha_0+\\cdots+\\alpha_{k-1} \\leqslant T ~~~\\Longrightarrow~~~ \\|x_k - x(\\alpha_0+\\cdots+\\alpha_{k-1})\\| \\leqslant \\frac{\\delta}{3}.\n\\end{equation}\nNow suppose that $\\|x'(t)\\| \\geqslant 2\\sigma(X_0)/T$ for all $t\\in (0,T)$. Then we obtain the following contradiction\n\\begin{equation*}\n     \\sigma(X_0) < T \\frac{2\\sigma(X_0)}{T} \\leqslant \\int_0^{T} \\|x'(t)\\|dt \\leqslant \\int_0^{\\infty} \\|x'(t)\\|dt \\leqslant \\sigma(X_0).\n\\end{equation*}\nHence, there exists $t^* \\in (0,T)$ such that $\\|x'(t^*)\\| = \\|\\nabla f(x(t^*))\\| < 2\\sigma(X_0)/T = 2\\sigma(X_0)/(2\\sigma(X_0)/\\nu) = \\nu$. Since  $x(t^*) \\in \\Phi(\\mathbb{R}_+,X_0) \\subset \\overline{\\Phi}_0$ and the infimum of the norm of $\\nabla f$ on $\\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3)$ is equal to $\\nu$, it must be that $x(t^*) \\in \\mathring{B}(C,\\delta/3)$. Hence there exists $x^* \\in C$ such that $\\|x(t^*)-x^*\\|\\leqslant \\delta/3$. \n\nIf $\\sum_{k=0}^{\\infty} \\alpha_k \\leqslant T$, then by \\eqref{eq:tracking_T} we have $x_k \\in B(\\overline{\\Phi}_0,\\delta/3)$ for all $k \\in \\mathbb{N}$. Otherwise, since $\\alpha_k \\leqslant \\bar{\\alpha} \\leqslant \\alpha_\\delta \\leqslant \\delta/(5mL) \\leqslant \\delta/(3L)$ for all $k \\in \\mathbb{N}$, there exists $k^* \\in \\mathbb{N}$ such that $t_{k^*} := \\sum_{k=0}^{k^*-1} \\alpha_k \\in [ t^* - \\delta/(3L) , t^*]$, where $\\sum_{k=0}^{k^*-1} \\alpha_k = 0$ if $k^* = 0$. Thus $\\|x_{k^*}-x^*\\| \\leqslant \\|x_{k^*}-x(t_{k^*})\\| + \\|x(t_{k^*}) - x(t^*)\\| + \\|x(t^*)-x^*\\| \\leqslant \\delta/3 + L | t_{k^*}-t^* | + \\delta/3 \\leqslant \\delta/3+\\delta/3+\\delta/3 = \\delta$.\nIf $K := \\inf\\{ k \\geqslant k^* : x_k \\notin \\mathring{B}(C,\\epsilon)\\}<\\infty$, then\n\\begin{subequations}\n    \\begin{align}\n        2m~\\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right) & = 2m~ \\psi^{-1}\\left(\\frac{1}{8m} \\epsilon - \\frac{L}{2}\\frac{\\epsilon}{5mL} \\right) \\label{eq:decrease_discrete_a} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m}(\\epsilon-\\delta) - \\frac{L\\alpha_\\delta}{2} \\right) \\label{eq:decrease_discrete_b} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m} \\left(\\|x_K-x^*\\|-\\|x_{k^*}-x^*\\|\\right) - \\frac{L\\alpha_\\delta}{2} \\right) \\label{eq:decrease_discrete_c} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m} \\|x_K-x_{k^*}\\| - \\frac{L\\bar{\\alpha}}{2} \\right) \\label{eq:decrease_discrete_d} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m}\\sum_{k=k^*}^{K-1} \\|x_{k+1}-x_k\\| - \\frac{L}{2}\\max_{k=0,\\hdots,K-1} \\alpha_k \\right) \\label{eq:decrease_discrete_e} \\\\\n        & \\leqslant f(x_{k^*}) - f(x_{K-1}) \\label{eq:decrease_discrete_f} \\\\\n        & \\leqslant m~\\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right) + \\max_C f - f(x_{K-1}). \\label{eq:decrease_discrete_g}\n    \\end{align}\n\\end{subequations}\nAbove, the arguments of $\\psi^{-1}$ in \\eqref{eq:decrease_discrete_a} are equal. \\eqref{eq:decrease_discrete_b} through \\eqref{eq:decrease_discrete_e} rely on the fact that $\\psi^{-1}$ is an increasing function. \\eqref{eq:decrease_discrete_b} is due to $\\delta<\\epsilon/2$. \\eqref{eq:decrease_discrete_c} holds because $x_K \\notin \\mathring{B}(C,\\epsilon)$, $x^* \\in C$, and $x_{k^*} \\in B(x^*,\\delta)$. \\eqref{eq:decrease_discrete_d} and \\eqref{eq:decrease_discrete_e} are consequences of the triangular inequality. \\eqref{eq:decrease_discrete_f} is due to the length formula \\eqref{eq:length_discrete_K} and the fact that $x_{k^*},\\hdots,x_{K-1} \\in \\mathring{B}(C,\\epsilon) \\subset \\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$. Finally, \\eqref{eq:decrease_discrete_g} is due to $x_{k^*}\\in B(C,\\delta)$ and \\eqref{eq:f_bound_discrete}. We remark that $K\\geqslant k^*+2$ since $\\|x_{k^*+1}-x^*\\| \\leqslant \\|x_{k^*+1}-x_{k^*}\\| +\\|x_{k^*}-x^*\\| \\leqslant \\alpha_{k^*}\\|\\nabla f(x_{k^*})\\|+\\delta \\leqslant (\\delta/(5mL))L+\\delta < \\epsilon$. Since $x_0,\\hdots,x_{K-2} \\in \\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$ and $x_{K-1}$ belongs to \n\\begin{equation*}\n    X_1 := B(C,\\epsilon) \\bigcap \\left\\{ x \\in \\mathbb{R}^n : f(x) \\leqslant \\max_{C} f - m~\\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right) \\right\\},    \n\\end{equation*}\nby the length formula \\eqref{eq:length_discrete_K} and the definition of $\\sigma(\\cdot,\\cdot)$ in \\eqref{eq:sup_discrete_X_0} we have\n    \\begin{align*}\n            \\sum_{k=0}^\\infty \\|x_{k+1}-x_k\\| & = \\sum_{k=0}^{K-2} \\|x_{k+1}-x_k\\| + \\sum_{k=K-1}^\\infty \\|x_{k+1}-x_k\\| \\\\\n            & \\leqslant 4m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(\\overline{\\Phi}_0,\\epsilon)} f\\right) \\right) + 2mL\\bar{\\alpha} + \\max\\{0,\\sigma(X_1,\\bar{\\alpha})\\}.\n    \\end{align*}\nNote that the inequality still holds if $K = \\infty$ or $\\sum_{k=0}^{\\infty} \\alpha_k \\leqslant T$. Hence\n\\begin{equation*}\n    \\sigma(X_0,\\bar{\\alpha}) ~\\leqslant~ 4m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(\\overline{\\Phi}_0,\\epsilon)} f\\right) \\right) + 2mL\\bar{\\alpha} + \\max\\{0,\\sigma(X_1,\\bar{\\alpha})\\}.\n\\end{equation*}\n\nIt now suffices to replace $X_0$ by $X_1$ and repeat the proof starting below \\eqref{eq:sup_discrete_X_0}. Since $f(\\Phi(t,x_1)) \\leqslant f(\\Phi(0,x_1)) \\leqslant \\max_{C} f - m\\psi^{-1}(\\epsilon/(40m))< \\max_{C} f$ for all $t\\geqslant 0$ and $x_1\\in X_1$, the maximal critical value of $f$ in $\\overline{\\Phi(\\mathbb{R}_+,X_1)}$ is less than the maximal critical value of $f$ in $\\overline{\\Phi(\\mathbb{R}_+,X_0)}$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Thus, it is eventually the case that one of the sets $X_0,X_1,\\hdots$ is empty. In order to conclude, one simply needs to choose an upper bound on the step sizes $\\hat{\\alpha}$ corresponding to $X_1$ that is less than or equal to the upper bound $\\bar{\\alpha}$ used for $X_0$. $\\sigma(X_0,\\cdot)$ is finite when evaluated at the last upper bound thus obtained. Indeed, the above recursive formula still holds if we replace $\\bar{\\alpha}$ by any $\\alpha \\in (0,\\bar{\\alpha}]$. In particular, we may take $\\alpha := \\hat{\\alpha}$.\n\n($3 \\Longrightarrow 1$) Obvious.\\\\\n\n","start":81913,"end":94199}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_discrete_X_0","sec:Proof of Theorem thm:convergence"],"member_ids":["S069"],"declared_external":false,"source_only":false,"title":"用有限时跟踪证明逆否命题","source_units":[{"raw_tex":"($1\\Longrightarrow 2$) We prove the contrapositive. Assume that there exists $0< T \\leqslant \\infty$ and a differentiable function $x:[0,T)\\rightarrow\\mathbb{R}^n$ such that\n\\begin{equation*}\n    x'(t) = -\\nabla f(x(t)), ~~~ \\forall t \\in (0,T),\n\\end{equation*}\nfor which, for all $c>0$, there exists $t\\in [0,T)$ such that $\\|x(t)\\| > c$. Let $x_0 := x(0)$. Let $\\bar{\\alpha}$ and $c$ be some positive constants. Let $\\tilde{t} \\in (0,T)$ be such that $\\|x(\\tilde{t})\\| \\geqslant c+2$. By Proposition \\ref{prop:tracking} and Remark \\ref{rem:tracking}, there exists $\\tilde{\\alpha}\\in (0,\\bar{\\alpha}]$ such that for all $\\alpha \\in (0,\\tilde{\\alpha}]$, the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by $x_{k+1} = x_k -\\alpha \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$ satisfies $\\|x_k-x(k\\alpha)\\| \\leqslant 1/2$ for all $k\\in \\{0,\\hdots, \\lfloor \\tilde{t}/\\alpha \\rfloor \\}$. We use the notation $\\lfloor t \\rfloor$ to denote the floor of a real number $t$ which is the unique integer such that $\\lfloor t \\rfloor \\leqslant t < \\lfloor t \\rfloor + 1$. If $\\tilde{k} = \\lfloor \\tilde{t}/\\alpha \\rfloor$, then $\\|x_{\\tilde{k}} - x(\\tilde{t})\\| \\leqslant \\|x_{\\tilde{k}}-x(\\tilde{k}\\alpha)\\|+\\|x(\\tilde{k}\\alpha)-x(\\tilde{t})\\| \\leqslant 1/2 + \\tilde{L}| \\tilde{k}\\alpha-\\tilde{t}| \\leqslant 1/2 + \\tilde{L}\\alpha$ where $\\tilde{L}>0$ is a Lipschitz constant of $f$ on the convex hull of $x([\\tilde{t}-\\tilde{\\alpha},\\tilde{t}])$. Let $\\alpha_0,\\alpha_1,\\alpha_2,\\hdots \\in (0,\\bar{\\alpha}]$ be the constant sequence equal to $\\min \\{\\tilde{\\alpha}, 1/(2\\tilde{L})\\}$ and consider the sequence $x_0,x_1,x_2,\\hdots \\in \\mathbb{R}^n$ defined by $x_{k+1} = x_k -\\alpha_k \\nabla f(x_k)$ for all $k \\in \\mathbb{N}$. By the triangular inequality, we have $\\|x_{\\tilde{k}}\\| \\geqslant \\|x(\\tilde{t})\\| - \\|x_{\\tilde{k}}-x(\\tilde{t})\\| \\geqslant c+2 - 1/2 - 1/2>c$ where $\\tilde{k} := \\lfloor \\tilde{t}/\\alpha \\rfloor$.\n","start":82008,"end":83931}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_discrete_X_0","sec:Proof of Theorem thm:convergence"],"member_ids":["S070"],"declared_external":false,"source_only":false,"title":"离散长度上确界与梯度流","source_units":[{"raw_tex":"($2 \\Longrightarrow 3$) Let $X_0$ be a bounded subset of $\\mathbb{R}^n$. We will show that there exists $\\bar{\\alpha}>0$ such that $\\sigma(X_0,\\bar{\\alpha})<\\infty$ where\n\\begin{subequations}\n\\label{eq:sup_discrete_X_0}\n\\begin{align}\n    \\sigma(X_0,\\bar{\\alpha}) := & \\sup\\limits_{\\tiny \\begin{array}{c}x \\in (\\mathbb{R}^n)^{\\mathbb{N}}\\\\ \\alpha \\in (0,\\bar{\\alpha}]^\\mathbb{N} \\end{array}} ~~ \\sum\\limits_{k=0}^{\\infty} \\|x_{k+1}-x_k\\| \\\\\n   & ~~ \\mathrm{subject~to} ~~~ \n\\left\\{ \n\\begin{array}{l}\nx_{k+1} = x_k - \\alpha_k \\nabla f(x_k), ~ \\forall k\\in \\mathbb{N},\\\\[2mm] x_0 \\in X_0.\n\\end{array}\n\\right.\n\\end{align}\n\\end{subequations}\n\nLet $\\Phi:\\mathbb{R}_+ \\times \\mathbb{R}^n \\rightarrow \\mathbb{R}^n$ be the gradient flow of $f$ defined for all $(t,x_0) \\in \\mathbb{R}_+ \\times \\mathbb{R}^n$ by $\\Phi(t,x_0) := x(t)$ where $x(\\cdot)$ is the unique continuous gradient trajectory of $f$ initialized at $x_0$. Uniqueness follows from the Picard–Lindel{\\\"o}f theorem \\cite[Theorem 3.1 p. 12]{coddington1955theory}. Let $C$ be the set of critical points of $f$ in $\\overline{\\Phi(\\mathbb{R}_+,X_0)}$. If $X_0 \\subset C$, then we have $\\sigma(X_0,\\bar{\\alpha}) \\leqslant 0$ for all $\\bar{\\alpha}>0$ and there is nothing left to prove. Otherwise, since $C$ is closed by \\cite[2.1.5 Proposition p. 29]{clarke1990}, there exists $\\epsilon>0$ be such that $X_0\\setminus \\mathring{B}(C,\\epsilon/6) \\neq \\emptyset$ where $\\mathring{B}(C,\\epsilon/6) := C + \\mathring{B}(0,\\epsilon/6)$. It is safe to assume this from now on.\n","start":83932,"end":85451}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":["eq:f_bound_discrete","eq:length_discrete_K","eq:sup_discrete_X_0","sec:Proof of Theorem thm:convergence"],"member_ids":["S071"],"declared_external":false,"source_only":false,"title":"公共流邻域与离散局部长度界","source_units":[{"raw_tex":"Let $\\psi$ be a desingularizing function of $f$ on $\\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$, where we use the shorthand $\\Phi_0 := \\Phi(\\mathbb{R}_+,X_0)$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Let $m \\in \\mathbb{N}$ be the number of critical values of $f$ in $B(\\overline{\\Phi}_0,\\epsilon)$. Note that $m \\geqslant 1$ due to Proposition \\ref{prop:length_continuous} and the fact that $X_0 \\neq \\emptyset$. Since $f$ is continuous and $C$ is compact by Lemma \\ref{lemma:uniform_length}, there exists $\\delta \\in (0,\\epsilon/2)$ such that  \n\\begin{equation}\n    \\label{eq:f_bound_discrete}\n    f(x) - \\max_{C} f \\leqslant m~ \\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right),~~~ \\forall x\\in B(C,\\delta).\n\\end{equation}\nNote that $\\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$ is bounded due to Lemma \\ref{lemma:uniform_length}. Let $L>0$ be a Lipschitz constant of $f$ on the convex hull of $\\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$. By Proposition \\ref{prop:length_discrete}, there exists $\\alpha_\\delta \\in (0,\\delta/(5mL)]$ such that, for all $K \\in \\mathbb{N}^*$, $\\alpha_0,\\hdots,\\alpha_{K-1} \\in (0,\\alpha_\\delta]$, and $(x_0,\\hdots,x_K) \\in \\mathring{B}(\\overline{\\Phi}_0,\\epsilon) \\times \\cdots \\times \\mathring{B}(\\overline{\\Phi}_0,\\epsilon) \\times \\mathbb{R}^n$ such that $x_{k+1} = x_k - \\alpha_k \\nabla f(x_k)$ for $k = 0,\\hdots,K-1$, we have\n\\begin{equation}\n\\label{eq:length_discrete_K}\n    \\frac{1}{4m}\\sum_{k=0}^{K-1}\\|x_{k+1}-x_k\\| ~\\leqslant~ \\psi\\left(\\frac{f(x_0)-f(x_{K-1})}{2m}\\right) + \\frac{L}{2}\\max_{k=0,\\hdots,K-1} \\alpha_k.\n\\end{equation}\n","start":85452,"end":87085}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_discrete_X_0","eq:tracking_T","sec:Proof of Theorem thm:convergence"],"member_ids":["S072"],"declared_external":false,"source_only":false,"title":"正梯度下界与统一跟踪时间","source_units":[{"raw_tex":"Since $\\nabla f$ is continuous, its norm attains its infimum $\\nu$ on the nonempty compact set $\\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3)$. \nIt is nonempty because $\\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3) \\supset X_0\\setminus \\mathring{B}(C,\\epsilon/6) \\neq \\emptyset$ and $\\delta < \\epsilon/2$.\nIf $\\nu=0$, then there exists $x^* \\in \\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3)$ such that $\\|\\nabla f(x^*)\\| = 0$. Then $x^*\\in C \\setminus \\mathring{B}(C,\\delta/3)$, which is a contradiction. We thus have $\\nu>0$. Hence we may define $T := 2\\sigma(X_0)/\\nu$ where $\\sigma(X_0)$ is defined in \\eqref{eq:sup_trajectory} and is finite by Lemma \\ref{lemma:uniform_length}. Note that $\\sigma(X_0)>0$ and thus $T>0$ because $X_0\\not\\subset C$. By Proposition \\ref{prop:tracking}, there exists $\\bar{\\alpha} \\in (0,\\alpha_\\delta]$ such that for any feasible point $((x_k)_{k\\in \\mathbb{N}} , (\\alpha_k)_{k\\in\\mathbb{N}})$ of \\eqref{eq:sup_discrete_X_0},\nthe continuous gradient trajectory $x:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ of $f$ initialized at $x_0$ satisfies\n\\begin{equation}\n\\label{eq:tracking_T}\n   \\forall k\\in \\mathbb{N}^*,~~~ \\alpha_0+\\cdots+\\alpha_{k-1} \\leqslant T ~~~\\Longrightarrow~~~ \\|x_k - x(\\alpha_0+\\cdots+\\alpha_{k-1})\\| \\leqslant \\frac{\\delta}{3}.\n\\end{equation}\n","start":87086,"end":88401}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_discrete_X_0","sec:Proof of Theorem thm:convergence"],"member_ids":["S073"],"declared_external":false,"source_only":false,"title":"有限时间内接近临界集","source_units":[{"raw_tex":"Now suppose that $\\|x'(t)\\| \\geqslant 2\\sigma(X_0)/T$ for all $t\\in (0,T)$. Then we obtain the following contradiction\n\\begin{equation*}\n     \\sigma(X_0) < T \\frac{2\\sigma(X_0)}{T} \\leqslant \\int_0^{T} \\|x'(t)\\|dt \\leqslant \\int_0^{\\infty} \\|x'(t)\\|dt \\leqslant \\sigma(X_0).\n\\end{equation*}\nHence, there exists $t^* \\in (0,T)$ such that $\\|x'(t^*)\\| = \\|\\nabla f(x(t^*))\\| < 2\\sigma(X_0)/T = 2\\sigma(X_0)/(2\\sigma(X_0)/\\nu) = \\nu$. Since  $x(t^*) \\in \\Phi(\\mathbb{R}_+,X_0) \\subset \\overline{\\Phi}_0$ and the infimum of the norm of $\\nabla f$ on $\\overline{\\Phi}_0 \\setminus \\mathring{B}(C,\\delta/3)$ is equal to $\\nu$, it must be that $x(t^*) \\in \\mathring{B}(C,\\delta/3)$. Hence there exists $x^* \\in C$ such that $\\|x(t^*)-x^*\\|\\leqslant \\delta/3$. \n","start":88401,"end":89154}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_discrete_X_0","sec:Proof of Theorem thm:convergence"],"member_ids":["S074"],"declared_external":false,"source_only":false,"title":"小步长使离散点进入临界邻域","source_units":[{"raw_tex":"If $\\sum_{k=0}^{\\infty} \\alpha_k \\leqslant T$, then by \\eqref{eq:tracking_T} we have $x_k \\in B(\\overline{\\Phi}_0,\\delta/3)$ for all $k \\in \\mathbb{N}$. Otherwise, since $\\alpha_k \\leqslant \\bar{\\alpha} \\leqslant \\alpha_\\delta \\leqslant \\delta/(5mL) \\leqslant \\delta/(3L)$ for all $k \\in \\mathbb{N}$, there exists $k^* \\in \\mathbb{N}$ such that $t_{k^*} := \\sum_{k=0}^{k^*-1} \\alpha_k \\in [ t^* - \\delta/(3L) , t^*]$, where $\\sum_{k=0}^{k^*-1} \\alpha_k = 0$ if $k^* = 0$. Thus $\\|x_{k^*}-x^*\\| \\leqslant \\|x_{k^*}-x(t_{k^*})\\| + \\|x(t_{k^*}) - x(t^*)\\| + \\|x(t^*)-x^*\\| \\leqslant \\delta/3 + L | t_{k^*}-t^* | + \\delta/3 \\leqslant \\delta/3+\\delta/3+\\delta/3 = \\delta$.\nIf $K := \\inf\\{ k \\geqslant k^* : x_k \\notin \\mathring{B}(C,\\epsilon)\\}<\\infty$, then\n","start":89155,"end":89909}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":["eq:decrease_discrete_a","eq:decrease_discrete_b","eq:decrease_discrete_c","eq:decrease_discrete_d","eq:decrease_discrete_e","eq:decrease_discrete_f","eq:decrease_discrete_g","eq:sup_discrete_X_0","sec:Proof of Theorem thm:convergence"],"member_ids":["S075"],"declared_external":false,"source_only":false,"title":"退出临界邻域迫使严格下降","source_units":[{"raw_tex":"\\begin{subequations}\n    \\begin{align}\n        2m~\\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right) & = 2m~ \\psi^{-1}\\left(\\frac{1}{8m} \\epsilon - \\frac{L}{2}\\frac{\\epsilon}{5mL} \\right) \\label{eq:decrease_discrete_a} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m}(\\epsilon-\\delta) - \\frac{L\\alpha_\\delta}{2} \\right) \\label{eq:decrease_discrete_b} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m} \\left(\\|x_K-x^*\\|-\\|x_{k^*}-x^*\\|\\right) - \\frac{L\\alpha_\\delta}{2} \\right) \\label{eq:decrease_discrete_c} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m} \\|x_K-x_{k^*}\\| - \\frac{L\\bar{\\alpha}}{2} \\right) \\label{eq:decrease_discrete_d} \\\\\n        & \\leqslant 2m~ \\psi^{-1}\\left(\\frac{1}{4m}\\sum_{k=k^*}^{K-1} \\|x_{k+1}-x_k\\| - \\frac{L}{2}\\max_{k=0,\\hdots,K-1} \\alpha_k \\right) \\label{eq:decrease_discrete_e} \\\\\n        & \\leqslant f(x_{k^*}) - f(x_{K-1}) \\label{eq:decrease_discrete_f} \\\\\n        & \\leqslant m~\\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right) + \\max_C f - f(x_{K-1}). \\label{eq:decrease_discrete_g}\n    \\end{align}\n\\end{subequations}\nAbove, the arguments of $\\psi^{-1}$ in \\eqref{eq:decrease_discrete_a} are equal. \\eqref{eq:decrease_discrete_b} through \\eqref{eq:decrease_discrete_e} rely on the fact that $\\psi^{-1}$ is an increasing function. \\eqref{eq:decrease_discrete_b} is due to $\\delta<\\epsilon/2$. \\eqref{eq:decrease_discrete_c} holds because $x_K \\notin \\mathring{B}(C,\\epsilon)$, $x^* \\in C$, and $x_{k^*} \\in B(x^*,\\delta)$. \\eqref{eq:decrease_discrete_d} and \\eqref{eq:decrease_discrete_e} are consequences of the triangular inequality. \\eqref{eq:decrease_discrete_f} is due to the length formula \\eqref{eq:length_discrete_K} and the fact that $x_{k^*},\\hdots,x_{K-1} \\in \\mathring{B}(C,\\epsilon) \\subset \\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$. Finally, \\eqref{eq:decrease_discrete_g} is due to $x_{k^*}\\in B(C,\\delta)$ and \\eqref{eq:f_bound_discrete}. We remark that $K\\geqslant k^*+2$ since $\\|x_{k^*+1}-x^*\\| \\leqslant \\|x_{k^*+1}-x_{k^*}\\| +\\|x_{k^*}-x^*\\| \\leqslant \\alpha_{k^*}\\|\\nabla f(x_{k^*})\\|+\\delta \\leqslant (\\delta/(5mL))L+\\delta < \\epsilon$. Since $x_0,\\hdots,x_{K-2} \\in \\mathring{B}(\\overline{\\Phi}_0,\\epsilon)$ and $x_{K-1}$ belongs to \n\\begin{equation*}\n    X_1 := B(C,\\epsilon) \\bigcap \\left\\{ x \\in \\mathbb{R}^n : f(x) \\leqslant \\max_{C} f - m~\\psi^{-1}\\left(\\frac{\\epsilon}{40m}\\right) \\right\\},    \n\\end{equation*}\n","start":89909,"end":92287}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_discrete_X_0","sec:Proof of Theorem thm:convergence"],"member_ids":["S076"],"declared_external":false,"source_only":false,"title":"离散长度上确界的递归界","source_units":[{"raw_tex":"by the length formula \\eqref{eq:length_discrete_K} and the definition of $\\sigma(\\cdot,\\cdot)$ in \\eqref{eq:sup_discrete_X_0} we have\n    \\begin{align*}\n            \\sum_{k=0}^\\infty \\|x_{k+1}-x_k\\| & = \\sum_{k=0}^{K-2} \\|x_{k+1}-x_k\\| + \\sum_{k=K-1}^\\infty \\|x_{k+1}-x_k\\| \\\\\n            & \\leqslant 4m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(\\overline{\\Phi}_0,\\epsilon)} f\\right) \\right) + 2mL\\bar{\\alpha} + \\max\\{0,\\sigma(X_1,\\bar{\\alpha})\\}.\n    \\end{align*}\nNote that the inequality still holds if $K = \\infty$ or $\\sum_{k=0}^{\\infty} \\alpha_k \\leqslant T$. Hence\n\\begin{equation*}\n    \\sigma(X_0,\\bar{\\alpha}) ~\\leqslant~ 4m~\\psi\\left( \\frac{1}{2m} \\left(\\sup_{X_0} f - \\min_{B(\\overline{\\Phi}_0,\\epsilon)} f\\right) \\right) + 2mL\\bar{\\alpha} + \\max\\{0,\\sigma(X_1,\\bar{\\alpha})\\}.\n\\end{equation*}\n","start":92287,"end":93102}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_discrete_X_0","sec:Proof of Theorem thm:convergence"],"member_ids":["S077"],"declared_external":false,"source_only":false,"title":"临界值下降与共同步长收尾","source_units":[{"raw_tex":"It now suffices to replace $X_0$ by $X_1$ and repeat the proof starting below \\eqref{eq:sup_discrete_X_0}. Since $f(\\Phi(t,x_1)) \\leqslant f(\\Phi(0,x_1)) \\leqslant \\max_{C} f - m\\psi^{-1}(\\epsilon/(40m))< \\max_{C} f$ for all $t\\geqslant 0$ and $x_1\\in X_1$, the maximal critical value of $f$ in $\\overline{\\Phi(\\mathbb{R}_+,X_1)}$ is less than the maximal critical value of $f$ in $\\overline{\\Phi(\\mathbb{R}_+,X_0)}$. By the definable Morse-Sard theorem \\cite[Corollary 9]{bolte2007clarke}, $f$ has finitely many critical values. Thus, it is eventually the case that one of the sets $X_0,X_1,\\hdots$ is empty. In order to conclude, one simply needs to choose an upper bound on the step sizes $\\hat{\\alpha}$ corresponding to $X_1$ that is less than or equal to the upper bound $\\bar{\\alpha}$ used for $X_0$. $\\sigma(X_0,\\cdot)$ is finite when evaluated at the last upper bound thus obtained. Indeed, the above recursive formula still holds if we replace $\\bar{\\alpha}$ by any $\\alpha \\in (0,\\bar{\\alpha}]$. In particular, we may take $\\alpha := \\hat{\\alpha}$.\n","start":93103,"end":94162}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sup_discrete_X_0","sec:Proof of Theorem thm:convergence"],"member_ids":["S078"],"declared_external":false,"source_only":false,"title":"统一长度蕴含逐初值有界","source_units":[{"raw_tex":"($3 \\Longrightarrow 1$) Obvious.\\\\\n\n","start":94163,"end":94199}]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S079"],"declared_external":true,"declaration_text":["声明前提 clarke1990, Chapter 2；原文使用 2.1.2(a), p.27；2.1.5(b),(d), p.29；2.2.4, p.33；2.3.9 Chain Rule I(ii), p.42。"],"source_only":false,"title":"克拉克次微分的基本性质","source_units":[]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S080"],"declared_external":true,"declaration_text":["声明的微分包含存在前提：aubin1984differential, Theorem 3, p.98；极大延拓用 Zorn 引理，原文引 lang2012algebra, Corollary 2.5, p.884。"],"source_only":false,"title":"微分包含局部存在与佐恩引理","source_units":[]},{"id":"S081","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S081"],"declared_external":true,"declaration_text":["声明前提 attouch2014variational, Theorem 17.1.1；原文第319行引用。"],"source_only":false,"title":"下有界目标的全局梯度流","source_units":[]},{"id":"S082","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S082"],"declared_external":true,"declaration_text":["声明前提 coddington1955theory, Theorem 3.1, p.12；原文第733行引用。"],"source_only":false,"title":"局部利普希茨常微分方程唯一性","source_units":[]},{"id":"S083","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S083"],"declared_external":true,"declaration_text":["声明前提 bolte2007clarke, Corollary 9；原文第408、637、708、735、787行引用。"],"source_only":false,"title":"可定义莫尔斯萨德定理","source_units":[]},{"id":"S084","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S084"],"declared_external":true,"declaration_text":["声明的 KL、单调性及凹化前提：bolte2007clarke, Corollary 15；van1998tame, (1.2), p.43；kurdyka1998gradients, Lemma 2；attouch2010proximal, Theorem 4.1；均见第408行。"],"source_only":false,"title":"局部 KL 与凹去奇异化背景","source_units":[]},{"id":"S085","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S085"],"declared_external":true,"declaration_text":["声明前提 bolte2020conservative, Proposition 2(iv)，及 davis2020stochastic, Lemma 5.2；原文第428–429行另列替代参考。"],"source_only":false,"title":"可定义链式法则与最小速度","source_units":[]},{"id":"S086","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S086"],"declared_external":true,"declaration_text":["声明前提 aubin1984differential, Theorems 1 and 3, p.13，及 Convergence Theorem, p.60；原文第671、673行引用。"],"source_only":false,"title":"轨迹紧性与微分包含收敛","source_units":[]},{"id":"S087","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S087"],"declared_external":true,"declaration_text":["声明的标准有限维分析前提；未逐一引用处是 implicit standard fact made explicit by annotator。原文第331行引用 rudin1964principles, 5.19；第443行引用 tao2006analysis, Theorem 17.7.2。"],"source_only":false,"title":"标准有限维分析工具","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076","S077":"S077","S078":"S078","S079":"S079","S080":"S080","S081":"S081","S082":"S082","S083":"S083","S084":"S084","S085":"S085","S086":"S086","S087":"S087"}}}}
{"record_id":"1710.03475v4","paper":{"title":"Sparse Semidefinite Programs with Guaranteed Near-Linear Time Complexity via Dualized Clique Tree Conversion","authors":["Zhang, Richard Y.","Lavaei, Javad"],"arxiv_id":"1710.03475","version":"v4","journal":"Mathematical Programming","abstract_url":"https://arxiv.org/abs/1710.03475v4","pdf_url":"https://arxiv.org/pdf/1710.03475v4","doi":"10.1007/s10107-020-01516-y"},"context":{"raw_tex":"Given $n\\times n$ real symmetric matrices $C,A_{1},\\ldots,A_{m}$\r\nand real scalars $b_{1},\\ldots,b_{m}$, we consider the standard form\r\nsemidefinite program \r\n\\begin{alignat}{2}\r\n & \\text{minimize } & C\\bullet X & \\tag{SDP}\\label{eq:SDP}\\\\\r\n & \\text{subject to } & A_{i}\\bullet X & =b_{i}\\quad\\text{for all }i\\in\\{1,\\ldots,m\\}\\nonumber \\\\\r\n &  & X & \\succeq0,\\nonumber \r\n\\end{alignat}\r\nover the $n\\times n$ real symmetric matrix variable $X$. Here, $A_{i}\\bullet X=\\tr A_{i}X$\r\nrefers to the usual matrix inner product, and $X\\succeq0$ restricts\r\nto be symmetric positive semidefinite. Instances of (\\ref{eq:SDP})\r\narise as some of the best convex relaxations to nonconvex problems\r\nlike graph optimization~\\cite{lovasz1979shannon,goemans1995improved},\r\ninteger programming~\\cite{sherali1990hierarchy,lovasz1991cones,lasserre2001explicit,laurent2003comparison},\r\nand polynomial optimization~\\cite{lasserre2001global,parrilo2000structured}.\r\n\n\nTo guarantee an exact reformulation, clique tree conversion chooses\r\nthe index sets $J_{1},\\ldots,J_{\\ell}$ in (\\ref{eq:relax}) as the\r\n\\emph{bags} of a \\emph{tree decomposition} for the \\emph{sparsity\r\ngraph} of the data matrices $C,A_{1},\\ldots,A_{m}$. Accordingly,\r\nthe parameter $\\omega$ in (\\ref{eq:omega_def-1}) can only be small\r\nif the sparsity graph has a small \\emph{treewidth}. Below, we define\r\na graph $G$ by its vertex set $V(G)\\subseteq\\{1,2,\\ldots,n\\}$ and\r\nits edge set $E(G)\\subseteq V(G)\\times V(G)$.\r\n\\begin{definition}[Sparsity graph]\r\n\\label{def:sparsitygraph}The $n\\times n$ matrix $M$ (resp. the\r\nset of $n\\times n$ matrices $\\{M_{1},\\ldots,M_{m}\\}$) is said to\r\nhave \\emph{sparsity graph} $G$ if $G$ is an undirected simple graph\r\non $n$ vertices $V(G)=\\{1,\\ldots,n\\}$ and that $(i,j)\\in E(G)$\r\nif $M[i,j]\\ne0$ (resp. if there exists $M\\in\\{M_{1},\\ldots,M_{m}\\}$\r\nsuch that $M[i,j]\\ne0$). \r\n\\end{definition}\r\n%\r\n\\begin{definition}[Tree decomposition]\r\n\\label{def:treedecomp}A \\emph{tree decomposition} $\\T$ of a graph\r\n$G$ is a pair $(\\J,T)$, where each \\emph{bag} of vertices $J_{j}\\in\\J$\r\nis a subset of $V(G)$, and $T$ is a tree on $|\\J|\\le n$ vertices,\r\nsuch that:\r\n\\begin{enumerate}\r\n\\item (Vertex cover) For every $v\\in V(G)$, there exists $J_{k}\\in\\J$\r\nsuch that $v\\in J_{k}$;\r\n\\item (Edge cover) For every $(u,v)\\in E(G)$, there exists $J_{k}\\in\\J$\r\nsuch that $u\\in J_{k}$ and $v\\in J_{k}$; and\r\n\\item (Running intersection) If $v\\in J_{i}$ and $v\\in J_{j}$, then we\r\nalso have $v\\in J_{k}$ for every $k$ that lies on the path from\r\n$i$ to $j$ in the tree $T$.\r\n\\end{enumerate}\r\nThe \\emph{width} $\\wid(\\T)$ of the tree decomposition $\\T=(\\J,T)$\r\nis the size of its largest bag minus one, as in $\\max\\{|J_{k}|:J_{k}\\in\\J\\}-1.$\r\nThe \\emph{treewidth} $\\tw(G)$ of the graph $G$ is the minimum width\r\namongst all tree decompositions $\\T$.\r\n\\end{definition}\r\nThroughout this paper, we make the implicit assumption that a tree\r\ndecomposition with small width is known a priori for the sparsity\r\ngraph. In practice, such a tree decomposition can usually be found\r\nusing fill-reducing heuristics for sparse linear algebra; see Section~\\ref{sec:Preliminaries}.\r\n\r\nWe also make two explicit assumptions, which are standard in the literature\r\non interior-point methods.\r\n\r\n\\begin{assume}[Linear independence]\\label{ass:lin}We have $\\sum_{i=1}^{m}y_{i}A_{i}=0$\r\nif and only if $y=0$.\r\n\r\n\\end{assume}\r\n\r\nThe assumption is without loss of generality, because it can either\r\nbe enforced by eliminating $A_{i}\\bullet X=b_{i}$ for select $i$,\r\nor else these constraints are not consistent for all $i$. Under Assumption~\\ref{ass:lin},\r\nthe total number of constraints is bounded $m\\le\\omega n$ (due to\r\nthe fact that $|E(G)|\\le n\\cdot\\tw(G)$~\\cite{arnborg1987complexity}).\r\n\r\n\\begin{assume}[Primal-dual Slater's condition]\\label{ass:slater}There\r\nexist $X\\succ0,$ $y,$ and $S\\succ0$ satisfying $A_{i}\\bullet X=b_{i}$\r\nfor all $i\\in\\{1,\\ldots,m\\}$ and $\\sum_{i=1}^{m}y_{i}A_{i}+S=C$. \r\n\r\n\\end{assume}\r\n\n\nThe sets $\\R^{n}$ and $\\R^{m\\times n}$ are the length-$n$ real\r\nvectors and $m\\times n$ real matrices. We use ``MATLAB notation''\r\nin concatenating vectors and matrices:\r\n\r\n\\[\r\n[a,b]=\\begin{bmatrix}a & b\\end{bmatrix},\\qquad[a;b]=\\begin{bmatrix}a\\\\\r\nb\r\n\\end{bmatrix},\\qquad\\diag(a,b)=\\begin{bmatrix}a & 0\\\\\r\n0 & b\r\n\\end{bmatrix},\r\n\\]\r\nand the following short-hand to construct them:\r\n\\[\r\n[x_{i}]_{i=1}^{n}=\\begin{bmatrix}x_{1}\\\\\r\n\\vdots\\\\\r\nx_{n}\r\n\\end{bmatrix},\\qquad[x_{i,j}]_{i,j=1}^{m,n}=\\begin{bmatrix}x_{1,1} & \\cdots & x_{1,n}\\\\\r\n\\vdots & \\ddots & \\vdots\\\\\r\nx_{m,1} & \\cdots & x_{m,n}\r\n\\end{bmatrix}.\r\n\\]\r\nThe notation $X[i,j]$ refers to the element of $X$ in the $i$-th\r\nrow and $j$-th column, and $X[I,J]$ refers to the submatrix of $X$\r\nformed from the rows in $I\\subseteq\\{1,\\ldots,m\\}$ and columns in\r\n$J\\subseteq\\{1,\\ldots,n\\}$. The Frobenius inner product is $X\\bullet Y=\\tr(X^{T}Y)$,\r\nand the Frobenius norm is $\\|X\\|_{F}=\\sqrt{X\\bullet X}$. We use $\\nnz(X)$\r\nto denote the number of nonzero elements in $X$. \r\n\r\nThe sets $\\S^{n}\\subseteq\\R^{n\\times n},$ $\\S_{+}^{n}\\subset\\S^{n},$\r\nand $\\S_{++}^{n}\\subset\\S_{+}^{n}$ are the $n\\times n$ real symmetric\r\nmatrices, positive semidefinite matrices, and positive definite matrices,\r\nrespective. We write $X\\succeq Y$ to mean $X-Y\\in\\S_{+}^{n}$ and\r\n$X\\succ Y$ to mean $X-Y\\in\\S_{++}^{n}$. The (symmetric) vectorization\r\n\\[\r\n\\vector(X)=[X[1,1];\\sqrt{2}X[2,1];\\ldots;\\sqrt{2}X[m,1];X[2,2],\\ldots]\r\n\\]\r\noutputs the lower-triangular part of a symmetric matrix as a vector,\r\nwith factors of $\\sqrt{2}$ added so that $\\vector(X)^{T}\\vector(Y)=X\\bullet Y$. \r\n\r\nA graph $G$ is defined by its vertex set $V(G)\\subseteq\\{1,2,3,\\ldots\\}$\r\nand its edge set $E(G)\\subseteq V(G)\\times V(G)$. The graph $T$\r\nis a \\emph{tree} if it is connected and does not contain any cycles;\r\nwe refer to its vertices $V(T)$ as its \\emph{nodes}. Designating\r\na special node $r\\in V(T)$ as the \\emph{root }of the tree allows\r\nus to define the \\emph{parent} $p(v)$ of each node $v\\ne r$ as the\r\nfirst node encountered on the path from $v$ to $r$, and $p(r)=r$\r\nfor consistency. The set of children is defined $\\ch(v)=\\{u\\in V(T)\\backslash v:p(u)=v\\}$.\r\nNote that the edges $E(T)$ are fully determined by the parent pointer\r\n$p$ as $\\{v,p(v)\\}$ for all $v\\ne r$.\r\n\r\nThe set $\\S_{G}^{n}\\subseteq\\S^{n}$ is the set of $n\\times n$ real\r\nsymmetric matrices with sparsity graph $G$. We denote $P_{G}(X)=\\min_{Y\\in\\S_{G}^{n}}\\|X-Y\\|_{F}$\r\nas the Euclidean projection of $X\\in\\S^{n}$ onto $\\S_{G}^{n}$.\r\n\n\nLet $\\T=(\\{J_{1},\\ldots,J_{\\ell}\\},T)$ be a tree decomposition with\r\nsmall width for the sparsity graph $G$ of the data matrices $C,A_{1},\\ldots,A_{m}$.\r\nWe define the graph $F\\supseteq G$ by taking each index set $J_{j}$\r\nof $\\T$ and interconnecting all pairs of vertices $u,v\\in J_{j}$,\r\nas in \r\n\\begin{align}\r\nV(F) & =V(G), & E(F) & =\\bigcup_{j=1}^{\\ell}\\{(u,v):u,v\\in J_{j}\\}.\\label{eq:chordal_compl}\r\n\\end{align}\r\nThe following fundamental result was first established by Grone et\r\nal.~\\cite{grone1984positive}. Constructive proofs allow us to recover\r\nall elements in $X\\succeq0$ from only the elements in $P_{F}(X)$\r\nusing a closed-form formula.\r\n\\begin{theorem}[Grone et al.~\\cite{grone1984positive}]\r\n\\label{thm:prim_sep}Given $Z\\in\\S_{F}^{n}$, there exists an $X\\succeq0$\r\nsatisfying $P_{F}(X)=Z$ if and only if $Z[J_{j},J_{j}]\\succeq0$\r\nfor all $j\\in\\{1,2,\\ldots,\\ell\\}$.\r\n\\end{theorem}\r\nWe can use Theorem~\\ref{thm:prim_sep} to reformulate (\\ref{eq:SDP})\r\ninto a reduced-complexity form. The key is to view (\\ref{eq:SDP})\r\nas an optimization over $P_{F}(X)$, since \r\n\\[\r\nC\\bullet X=\\sum_{i,j=1}^{n}C_{i,j}X_{i,j}=\\sum_{(i,j)\\in F}C_{i,j}X_{i,j}=C\\bullet P_{F}(X),\r\n\\]\r\nand similarly $A_{i}\\bullet X=A_{i}\\bullet P_{F}(X)$. Theorem~\\ref{thm:prim_sep}\r\nallows us to account for $X\\succeq0$ implicitly, by optimizing over\r\n$Z=P_{F}(X)$ in the following\r\n\\begin{alignat}{3}\r\n & \\text{minimize }\\quad & C\\bullet Z\\label{eq:CSDP}\\\\\r\n & \\text{subject to } & A_{i}\\bullet Z & =b_{i} & \\qquad\\text{for all } & i\\in\\{1,\\ldots,m\\},\\nonumber \\\\\r\n &  & Z[J_{j},J_{j}] & \\succeq0 & \\text{for all } & j\\in\\{1,\\ldots,\\ell\\}.\\nonumber \r\n\\end{alignat}\r\nNext, we split the principal submatrices into distinct matrix variables,\r\ncoupled by the need for their overlapping elements to agree. Define\r\nthe \\emph{overlap} operator $\\NN_{i,j}(\\cdot)$ to output the overlapping\r\nelements of two principal submatrices given the latter as input:\r\n\\[\r\n\\NN_{i,j}(X[J_{j},J_{j}])=X[J_{i}\\cap J_{j},\\;J_{i}\\cap J_{j}]=\\NN_{j,i}(X[J_{i},J_{i}]).\r\n\\]\r\nThe running intersection property of the tree decomposition allows\r\nus to enforce this agreement using $\\ell-1$ pairwise block comparisons.\r\n\\begin{theorem}[Fukuda et al.~\\cite{fukuda2001exploiting}]\r\n\\label{thm:ctree}Given $X_{1},X_{2},\\ldots,X_{\\ell}$ for $X_{j}\\in\\S^{|J_{j}|}$,\r\nthere exists $Z$ satisfying $Z[J_{j},J_{j}]=X_{j}$ for all $j\\in\\{1,2,\\ldots,\\ell\\}$\r\nif and only if $\\NN_{i,j}(X_{j})=\\NN_{j,i}(X_{i})$ for all $(i,j)\\in E(T)$. \r\n\\end{theorem}\r\nSplitting the objective $C$ and constraint matrices $A_{1},\\ldots,A_{m}$\r\ninto $C_{1},\\ldots,C_{\\ell}$ and $A_{1,1},\\ldots,A_{m,\\ell}$ to\r\nsatisfy \r\n\\begin{align}\r\nC_{1}\\bullet X[J_{1},J_{1}]+C_{2}\\bullet X[J_{2},J_{2}]+\\cdots+C_{\\ell}\\bullet X[J_{\\ell},J_{\\ell}] & =C\\bullet X,\\label{eq:split_constr}\\\\\r\nA_{i,1}\\bullet X[J_{1},J_{1}]+A_{i,2}\\bullet X[J_{2},J_{2}]+\\cdots+A_{i,\\ell}\\bullet X[J_{\\ell},J_{\\ell}] & =A_{i}\\bullet X,\\nonumber \r\n\\end{align}\r\nand applying Theorem~\\ref{thm:ctree} yields the following\r\n\\begin{alignat}{3}\r\n & \\text{minimize }\\quad & \\sum_{j=1}^{\\ell}C_{j}\\bullet X_{j} &  &  & \\tag{CTC}\\label{eq:CTC}\\\\\r\n & \\text{subject to } & \\sum_{j=1}^{\\ell}A_{i,j}\\bullet X_{j} & =b_{i} & \\text{for all } & i\\in\\{1,\\ldots,m\\},\\nonumber \\\\\r\n &  & \\NN_{i,j}(X_{j}) & =\\NN_{j,i}(X_{i}) & \\qquad\\text{for all } & (i,j)\\in E(T),\\nonumber \\\\\r\n &  & X_{j} & \\succeq0 & \\text{for all } & j\\in\\{1,\\ldots,\\ell\\},\\nonumber \r\n\\end{alignat}\r\nwhich vectorizes into a linear conic program in standard form\r\n\\begin{alignat}{4}\r\n & \\text{minimize } & c^{T}x, &  & \\qquad\\qquad & \\text{maximize } & \\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}^{T}y,\\label{eq:vecCTC}\\\\\r\n & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}x & =\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}, &  & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}y+s & =c,\\nonumber \\\\\r\n &  & x & \\in\\K, &  &  & s & \\in\\K_{*}\\nonumber \r\n\\end{alignat}\r\nover the Cartesian product of $\\ell\\le n$ smaller semidefinite cones\r\n\\begin{equation}\r\n\\K=\\K_{*}=\\S_{+}^{|J_{1}|}\\times\\S_{+}^{|J_{2}|}\\times\\cdots\\times\\S_{+}^{|J_{\\ell}|}.\r\n\\end{equation}\r\nHere, $\\A=[\\vector(A_{i,j})^{T}]_{i,j=1}^{m,\\ell}$ and $c=[\\vector(C_{j})]_{j=1}^{\\ell}$\r\ncorrespond to (\\ref{eq:split_constr}), and the \\emph{overlap constraints}\r\nmatrix $\\N=[\\N_{i,j}]_{i,j=1}^{\\ell,\\ell}$ is implicitly defined\r\nby the relation\r\n\\begin{equation}\r\n\\N_{i,j}\\vector(X_{j})=\\begin{cases}\r\n+\\vector(\\NN_{p(i),i}(X_{i})) & j=i,\\\\\r\n-\\vector(\\NN_{i,p(i)}(X_{p(i)})) & j=p(i),\\\\\r\n0 & \\text{otherwise},\r\n\\end{cases}\\label{eq:Nmat}\r\n\\end{equation}\r\nfor every non-root node $i$ on $T$. (To avoid all-zero rows in $\\N$,\r\nwe define $\\N_{i,j}\\,\\vector(X_{j})$ as the empty length-zero vector\r\n$\\R^{0}$ if $i$ is the root node.)\r\n\n\nwhere the \\emph{scaling matrix} $\\D_{s}$ is block-diagonal with fully-dense\r\nblocks \r\n\\begin{equation}\r\n\\D_{s}=\\diag(\\D_{s,1},\\ldots,\\D_{s,\\ell}),\\qquad\\D_{s,j}\\succ0\\quad\\text{for all }j\\in\\{1,\\ldots,\\ell\\}.\\label{eq:scal}\r\n\\end{equation}\r\n\n\n\\begin{definition}[Topological ordering]\r\n\\label{def:topological}An ordering\\emph{ }$\\pi:\\{1,2,\\ldots,n\\}\\to V(T)$\r\non the tree graph $T$ with $n$ nodes is said to be \\emph{topological}~\\cite[p.~10]{vandenberghe2015chordal}\r\nif, by designating $\\pi(n)$ as the root of $T$, each node is indexed\r\nbefore its parent:\r\n\\[\r\n\\pi^{-1}(v)<\\pi^{-1}(p(v))\\qquad\\text{ for all }v\\ne r,\r\n\\]\r\nwhere $\\pi^{-1}(v)$ denotes the index associated with the node $v$.\r\n\\end{definition}\r\n\\begin{lemma}[No block fill]\r\n\\label{lem:blocktree}Let $J_{1},\\ldots,J_{n}$ satisfy $\\bigcup_{j=1}^{n}J_{j}=\\{1,\\ldots,d\\}$\r\nand $J_{i}\\cap J_{j}=\\emptyset$ for all $i\\ne j$, and let $H\\succ0$\r\nbe a $d\\times d$ matrix satisfying\r\n\\[\r\nH[J_{i},J_{j}]\\ne0\\qquad\\implies\\qquad(i,j)\\in E(T)\r\n\\]\r\nfor a tree graph $T$ on $n$ nodes. If $\\pi$ is a topological ordering\r\non $T$ and $\\Pi$ is a permutation matrix satisfying \r\n\\[\r\n(\\Pi H\\Pi)[J_{i},J_{j}]=H[J_{\\pi(i)},J_{\\pi(j)}]\\qquad\\text{for all }i,j\\in\\{1,\\ldots,n\\},\r\n\\]\r\nthen $\\Pi H\\Pi^{T}$ factors into $LL^{T}$ where the Cholesky factor\r\n$L$ satisfies\r\n\\[\r\nL[J_{i},J_{j}]\\ne0\\qquad\\implies\\qquad(\\Pi H\\Pi)[J_{i},J_{j}]\\ne0\\qquad\\text{ for all }i>j.\r\n\\]\r\nTherefore, sparse Cholesky factorization solves $Hx=b$ for $x$ by:\r\n(i) factoring $\\Pi H\\Pi^{T}$ into $LL^{T}$ in $O(\\beta^{3}n)$ operations\r\nand $O(\\beta^{2}n)$ memory where $\\beta=\\max_{j}|J_{j}|$, and (ii)\r\nsolving $Ly=\\Pi b$ and $L^{T}z=y$ and $x=\\Pi^{T}z$ in $O(\\beta^{2}n)$\r\noperations and memory. \r\n\\end{lemma}\r\nThis is a simple block-wise extension of the tree elimination result\r\noriginally due to Parter~\\cite{parter1961use}; see also George and\r\nLiu~\\cite[Lemma 6.3.1]{george1981computer}. In practice, a topological\r\nordering can be found by assigning indices $n,n-1,n-2,\\ldots$ in\r\ndecreasing ordering during a depth-first search traversal of the tree.\r\nIn fact, the minimum degree heuristic is guaranteed to generate a\r\ntopological ordering~\\cite{george1989evolution}.\r\n\n\nThe dualization technique of Löfberg~\\cite{lofberg2009dualize} swaps\r\nthe roles played by the primal and the dual problems in a linear conic\r\nprogram, by rewriting a primal standard form problem into dual standard\r\nform, and vice versa. Applying dualization to (\\ref{eq:vecCTC}) yields\r\nthe following\r\n\\begin{alignat}{4}\r\n & \\text{minimize } & \\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}^{T}x_{1} &  & \\qquad & \\text{maximize } & -c^{T}y\\label{eq:dualCTC}\\\\\r\n & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}x_{1}-x_{2} & =-c, &  & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}y+s_{1} & =\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix},\\nonumber \\\\\r\n &  & x_{1}\\in\\R^{f},\\,x_{2} & \\in\\K. &  &  & -y+s_{2} & =0,\\nonumber \\\\\r\n &  &  &  &  &  & s_{1}\\in\\{0\\}^{f},\\,s_{2} & \\in\\K.\\nonumber \r\n\\end{alignat}\r\nwhere we use $f$ to denote the number of equality constraints in\r\n(\\ref{eq:CTC}). Observe that the dual problem in (\\ref{eq:dualCTC})\r\nis identical to the primal problem in (\\ref{eq:vecCTC}), so that\r\na dual solution $y^{\\star}$ to (\\ref{eq:dualCTC}) immediately serves\r\nas a primal solution to (\\ref{eq:vecCTC}), and hence also (\\ref{eq:CTC}).\r\n\r\nModern interior-point methods solve (\\ref{eq:dualCTC}) by embeding\r\nthe free variable $x_{1}\\in\\R^{f}$ and fixed variable $s_{1}\\in\\{0\\}^{f}$\r\ninto a second-order cone (see Sturm~\\cite{sturm1999using} and Andersen~\\cite{andersen2002handling}):\r\n\\begin{alignat}{4}\r\n & \\text{minimize } & \\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}^{T}x_{1} &  & \\qquad & \\text{maximize } & -c^{T}y\\label{eq:dualCTC-2}\\\\\r\n & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}x_{1}-x_{2} & =-c, &  & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}y+s_{1} & =\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix},\\nonumber \\\\\r\n &  & \\|x_{1}\\|\\le x_{0},\\,x_{2} & \\in\\K. &  &  & -y+s_{2} & =0,\\nonumber \\\\\r\n &  &  &  &  &  & s_{0} & =0,\\nonumber \\\\\r\n &  &  &  &  &  & \\|s_{1}\\|\\le s_{0},\\,s_{2} & \\in\\K.\\nonumber \r\n\\end{alignat}\r\nWhen (\\ref{eq:dualCTC-2}) is solved using an interior-point method,\r\nthe normal equation solved at each iteration takes the form\r\n\\begin{equation}\r\n\\left(\\D_{s}+\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}\\D_{f}\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}\\right)\\Delta y=r\\label{eq:Hxr_dual}\r\n\\end{equation}\r\nwhere $\\D_{s}$ is comparable as before in (\\ref{eq:scal}), and \r\n\\begin{equation}\r\n\\D_{f}=\\sigma I+ww^{T},\\qquad\\sigma>0\\label{eq:scal_f}\r\n\\end{equation}\r\nis the rank-1 perturbation of a scaled identity matrix. The standard\r\nprocedure, as implemented in SeDuMi~\\cite{sturm1999using,sturm2002implementation}\r\nand MOSEK~\\cite{andersen2003implementing}, is to form the sparse\r\nmatrix $\\H$ and dense vector $\\q$, defined\r\n\\begin{align}\r\n\\H & =\\D_{s}+\\sigma\\A^{T}\\A+\\sigma\\N^{T}\\N, & \\q & =\\begin{bmatrix}\\A^{T} & \\N^{T}\\end{bmatrix}w.\\label{eq:Hqdef}\r\n\\end{align}\r\nand then solve (\\ref{eq:Hxr_dual}) using a rank-1 update\\footnote{To keep our derivations simple, we perform the rank-1 update using\r\nthe Sherman\\textendash Morrison\\textemdash Woodbury (SMW) formula.\r\nIn practice, the product-form Cholesky Factor (PFCF) formula of Goldfarb\r\nand Scheinberg~\\cite{goldfarb2005product} is more numerically stable\r\nand more widely used~\\cite{sturm1999using,sturm2002implementation}.\r\nOur complexity results remain valid in either cases because the PFCF\r\nis a constant factor of approximately two times slower than the SWM~\\cite{goldfarb2005product}.}\r\n\\begin{equation}\r\n\\Delta y=(\\H+\\q\\q^{T})^{-1}r=\\left(I-\\frac{(\\H^{-1}\\q)\\q^{T}}{1+\\q^{T}(\\H^{-1}\\q)}\\right)\\H^{-1}r,\\label{eq:update}\r\n\\end{equation}\r\nat a cost comparable to the solution of $\\H u=r$ for two right-hand\r\nsides. (In Appendix~\\ref{sec:inequality}, we repeat these derivations\r\nfor the version of (\\ref{eq:vecCTC}) in which $\\A x=b$ is replaced\r\nby the inequality $\\A x\\le b$.)\r\n\n\nDefine the $m\\times\\frac{1}{2}n(n+1)$ matrix $\\M$ as the vectorization\r\nof the linear constraints in (\\ref{eq:SDP}), as in\r\n\\[\r\n\\M\\,\\vector(X)=\\begin{bmatrix}\\vector(A_{1})^{T}\\\\\r\n\\vdots\\\\\r\n\\vector(A_{m})^{T}\r\n\\end{bmatrix}\\vector(X)=\\begin{bmatrix}\\vector(A_{1})^{T}\\vector(X)\\\\\r\n\\vdots\\\\\r\n\\vector(A_{m})^{T}\\vector(X)\r\n\\end{bmatrix}=\\begin{bmatrix}A_{1}\\bullet X\\\\\r\n\\vdots\\\\\r\nA_{m}\\bullet X\r\n\\end{bmatrix}.\r\n\\]\r\nIn reformulating (\\ref{eq:SDP}) into (\\ref{eq:CTC}), the splitting\r\nconditions (\\ref{eq:split_constr}) can be rewritten as the following\r\n\\begin{align}\r\nc^{T}\\P & =\\vector(C)^{T}, & \\A\\P & =\\M,\\label{eq:split_constr2}\r\n\\end{align}\r\nwhere $c=[\\vector(C_{j})]_{j=1}^{\\ell}$ and $\\A=[\\vector(A_{i,j})^{T}]_{i,j=1}^{m,\\ell}$\r\nare the data for the vectorized verison of (\\ref{eq:CTC}).\r\n\n\nWe solve the dualized problem (\\ref{eq:dualCTC-2}) by solving its\r\nextended homogeneous self-dual embedding\\begin{subequations}\\label{eq:hsd}\r\n\\begin{alignat}{2}\r\n & \\text{min.}\\quad & (\\nu+1)\\theta\\\\\r\n & \\text{s.t.} & \\begin{bmatrix}0 & +\\M^{T} & -\\c & -r_{d}\\\\\r\n-\\M & 0 & +\\b & -r_{p}\\\\\r\n+\\c^{T} & -\\b^{T} & 0 & -r_{c}\\\\\r\nr_{d}^{T} & r_{p}^{T} & r_{c} & 0\r\n\\end{bmatrix}\\begin{bmatrix}x\\\\\r\ny\\\\\r\n\\tau\\\\\r\n\\theta\r\n\\end{bmatrix}+\\begin{bmatrix}s\\\\\r\n0\\\\\r\n\\kappa\\\\\r\n0\r\n\\end{bmatrix} & =\\begin{bmatrix}0\\\\\r\n0\\\\\r\n0\\\\\r\n\\nu+1\r\n\\end{bmatrix}\\label{eq:hsdfeas}\\\\\r\n &  & x,s\\in\\C\\quad\\kappa,\\tau & \\ge0,\\label{eq:hsdcone}\r\n\\end{alignat}\r\nwhere the data is given in standard form\r\n\\begin{gather}\r\n\\M=\\begin{bmatrix}0 & -\\begin{bmatrix}\\A^{T} & \\N^{T}\\end{bmatrix} & +I\\end{bmatrix},\\quad\\c^{T}=\\begin{bmatrix}0 & \\begin{bmatrix}b^{T} & 0\\end{bmatrix} & 0\\end{bmatrix},\\quad\\b=-c,\\\\\r\n\\C=\\{(x_{0},x_{1}\\}\\in\\R^{p+1}:\\|x_{1}\\|\\le x_{0}\\}\\times\\K,\r\n\\end{gather}\r\nand the residual vectors are defined\r\n\\begin{equation}\r\nr_{d}=\\one_{\\C}-\\c,\\quad r_{p}=-\\M\\one_{\\C}+\\b,\\quad r_{c}=1+\\c^{T}\\one_{\\C}.\r\n\\end{equation}\r\n\\end{subequations}Here, $\\nu$ is the order of the cone $\\C$, and\r\n$\\one_{\\C}$ is its identity element\r\n\\begin{align}\r\n\\nu & =1+|J_{1}|+\\cdots+|J_{\\ell}|, & \\one_{\\C} & =[1;0;\\ldots;0;\\vector(I_{|J_{1}|});\\ldots;\\vector(I_{|J_{\\ell}|})].\r\n\\end{align}\r\nProblem (\\ref{eq:hsd}) has optimal value $\\theta^{\\star}=0$. Under\r\nthe primal-dual Slater's conditions (Assumption~\\ref{ass:slater}),\r\nan interior-point method is guaranteed to converge to an $\\epsilon$-accurate\r\nsolution with $\\tau>0$, and this yields an $\\epsilon$-feasible and\r\n$\\epsilon$-accurate solution to the dualized problem (\\ref{eq:dualCTC-2})\r\nby rescaling $x/\\tau$ and $y=y/\\tau$ and $s=s/\\tau$. The following\r\nresult is adapted from~\\cite[Lemma~5.7.2]{de2000self} and \\cite[Theorem~22.7]{vanderbei2015linear}.\r\n\n\nWe solve the homogeneous self-dual embedding (\\ref{eq:hsd}) using\r\nthe short-step method of Nesterov and Todd~\\cite[Algorithm~6.1]{nesterov1998primal}\r\n(and also Sturm and S.~Zhang~\\cite[Section~5.1]{sturm1999symmetric}),\r\nnoting that SeDuMi reduces to it in the worst case; see~\\cite{sturm2002implementation}\r\nand~\\cite{sturm1997wide}. Beginning at the following strictly feasible,\r\nperfectly centered point\r\n\\begin{gather}\r\n\\theta^{(0)}=\\tau^{(0)}=\\kappa^{(0)}=1,\\quad y^{(0)}=0,\\quad x^{(0)}=s^{(0)}=\\one_{\\C},\\label{eq:hsdinit}\r\n\\end{gather}\r\nwith barrier parameter $\\mu=1$, we take the following steps\\begin{subequations}\\label{eq:hsd_ipm}\r\n\\begin{align}\r\n\\mu^{+} & =\\left(1-\\frac{1}{15\\sqrt{\\nu+1}}\\right)\\cdot\\frac{x^{T}s+\\tau\\kappa}{\\nu+1},\\\\\r\n(x^{+},y^{+},s^{+},\\tau^{+},\\theta^{+},\\kappa^{+}) & =(x,y,s,\\tau,\\theta,\\kappa)+(\\Delta x,\\Delta y,\\Delta s,\\Delta\\tau,\\Delta\\theta,\\Delta\\kappa).\\nonumber \r\n\\end{align}\r\n\\end{subequations}along the search direction defined by the linear\r\nsystem~\\cite[Eqn. 9]{todd1998nesterov} \\begin{subequations}\\label{eq:hsd_nt}\r\n\\begin{align}\r\n\\begin{bmatrix}0 & +\\M^{T} & -\\c & -r_{p}\\\\\r\n-\\M & 0 & +\\b & -r_{d}\\\\\r\n+\\c^{T} & -\\b^{T} & 0 & -r_{c}\\\\\r\n+r_{p}^{T} & +r_{d}^{T} & +r_{c} & 0\r\n\\end{bmatrix}\\begin{bmatrix}\\Delta x\\\\\r\n\\Delta y\\\\\r\n\\Delta\\tau\\\\\r\n\\Delta\\theta\r\n\\end{bmatrix}+\\begin{bmatrix}\\Delta s\\\\\r\n0\\\\\r\n\\Delta\\kappa\\\\\r\n0\r\n\\end{bmatrix} & =0,\\\\\r\ns+\\Delta s+\\nabla^{2}F(w)\\Delta x+\\mu^{+}\\nabla F(x) & =0,\\\\\r\n\\kappa+\\Delta\\kappa+(\\kappa/\\tau)\\Delta\\tau-\\mu^{+}\\tau^{-1} & =0.\r\n\\end{align}\r\n\\end{subequations}Here, $F$ is the usual self-concordant barrier\r\nfunction on $\\C$\r\n\\begin{align}\r\nF([x_{0};x_{1};\\vector(X_{1});\\ldots;\\vector(X_{\\ell})])= & -\\log\\left(\\frac{1}{2}x_{0}^{2}-\\frac{1}{2}x_{1}^{T}x_{1}\\right)-\\sum_{j=1}^{\\ell}\\log\\det(X_{j})\\label{eq:Fdef}\r\n\\end{align}\r\nand $w\\in\\Int(\\C)$ is the unique \\emph{scaling point} satisfying\r\n$\\nabla^{2}F(w)x=s$, which can be computed from $x$ and $s$ in\r\nclosed-form. The following iteration bound is an immediate consequence\r\nof~\\cite[Theorem~6.4]{nesterov1998primal}; see also~\\cite[Theorem~5.1]{sturm1999symmetric}.\r\n\\begin{lemma}[Short-Step Method]\r\n\\label{lem:short-step}The sequence in (\\ref{eq:hsd_ipm}) arrives\r\nat an iterate $(x,y,s,\\tau,\\theta,\\kappa)$ satisfying the conditions\r\nof Lemma~\\ref{lem:eps} with $\\gamma=9/10$ in at most $O(\\sqrt{\\nu}\\log(1/\\epsilon))$\r\niterations. \r\n\\end{lemma}\r\nThe cost of each interior-point iteration is dominated by the cost\r\nof computing the search direction in (\\ref{eq:hsd_nt}). Using elementary\r\nbut tedious linear algebra, we can show that if\\begin{subequations}\\label{eq:newt}\r\n\\begin{equation}\r\n(\\M\\D^{-1}\\M^{T})\\begin{bmatrix}v_{1} & v_{2} & v_{3}\\end{bmatrix}=\\begin{bmatrix}0 & -\\b & r_{p}\\end{bmatrix}-\\M\\D^{-1}\\begin{bmatrix}d & \\c & r_{d}\\end{bmatrix}\\label{eq:hsd_nrm}\r\n\\end{equation}\r\nwhere $\\D=\\nabla^{2}F(w)$ and $d=-s-\\mu^{+}\\nabla F(x)$, and \r\n\\begin{equation}\r\n\\begin{bmatrix}u_{1} & u_{2} & u_{3}\\end{bmatrix}=\\D^{-1}(\\begin{bmatrix}d & c & r_{d}\\end{bmatrix}+\\M^{T}\\begin{bmatrix}v_{1} & v_{2} & v_{3}\\end{bmatrix}),\\label{eq:hsd_back1}\r\n\\end{equation}\r\nthen\r\n\\begin{align}\r\n\\left(\\begin{bmatrix}-\\D_{0} & -r_{c}\\\\\r\nr_{c} & 0\r\n\\end{bmatrix}-\\begin{bmatrix}\\c & r_{d}\\\\\r\n-\\b & r_{p}\r\n\\end{bmatrix}^{T}\\begin{bmatrix}u_{2} & u_{3}\\\\\r\nv_{2} & v_{3}\r\n\\end{bmatrix}\\right)\\begin{bmatrix}\\Delta\\tau\\\\\r\n\\Delta\\theta\r\n\\end{bmatrix} & =\\begin{bmatrix}-d_{0}\\\\\r\n0\r\n\\end{bmatrix}-\\begin{bmatrix}\\c & r_{d}\\\\\r\n-\\b & r_{p}\r\n\\end{bmatrix}^{T}\\begin{bmatrix}u_{1}\\\\\r\nv_{1}\r\n\\end{bmatrix},\\\\\r\n\\begin{bmatrix}\\Delta x\\\\\r\n\\Delta y\r\n\\end{bmatrix} & =\\begin{bmatrix}u_{1}\\\\\r\nv_{1}\r\n\\end{bmatrix}-\\begin{bmatrix}u_{1} & u_{2}\\\\\r\nv_{1} & v_{2}\r\n\\end{bmatrix}\\begin{bmatrix}\\Delta\\tau\\\\\r\n\\Delta\\theta\r\n\\end{bmatrix},\\\\\r\n\\Delta s & =d-\\D\\Delta x,\\\\\r\n\\Delta\\kappa & =d_{0}-\\D_{0}\\Delta\\tau,\\label{eq:hsd_back2}\r\n\\end{align}\r\n\\end{subequations}where $\\D_{0}=\\kappa/\\tau$ and $d_{0}=-\\kappa+\\mu^{+}\\tau^{-1}$.\r\nHence, the cost of computing the search direction is dominated by\r\nthe cost of solving the normal equation for three different right-hand\r\nsides. Here, the normal matrix is written\r\n\r\n\\[\r\n\\M\\D^{-1}\\M^{T}=\\diag(W_{1}\\otimes_{s}W_{1},\\ldots,W_{\\ell}\\otimes_{s}W_{\\ell})+\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}(w_{1}w_{1}^{T}+\\sigma I)\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix},\r\n\\]\r\nwhere $\\sigma=\\frac{1}{2}(w_{0}^{2}-w_{1}^{T}w_{1})>0$ and $\\otimes_{s}$\r\ndenotes the symmetric Kronecker product~\\cite{alizadeh1998primal}\r\nimplicitly defined to satisfy\r\n\\[\r\n(A\\otimes_{s}B)\\vector(X)=\\frac{1}{2}\\vector(AXB^{T}+BXA^{T})\\qquad\\text{for all }X=X^{T}.\r\n\\]\r\nUnder the hypothesis on $\\A$ stated in Theorem~\\ref{thm:lintime},\r\nthe normal matrix satisfies the assumptions of Lemma~\\ref{lem:normal},\r\nand can therefore be solved in linear $O(n)$ time and memory. \r\n","units":[{"raw_tex":"Given $n\\times n$ real symmetric matrices $C,A_{1},\\ldots,A_{m}$\r\nand real scalars $b_{1},\\ldots,b_{m}$, we consider the standard form\r\nsemidefinite program \r\n\\begin{alignat}{2}\r\n & \\text{minimize } & C\\bullet X & \\tag{SDP}\\label{eq:SDP}\\\\\r\n & \\text{subject to } & A_{i}\\bullet X & =b_{i}\\quad\\text{for all }i\\in\\{1,\\ldots,m\\}\\nonumber \\\\\r\n &  & X & \\succeq0,\\nonumber \r\n\\end{alignat}\r\nover the $n\\times n$ real symmetric matrix variable $X$. Here, $A_{i}\\bullet X=\\tr A_{i}X$\r\nrefers to the usual matrix inner product, and $X\\succeq0$ restricts\r\nto be symmetric positive semidefinite. Instances of (\\ref{eq:SDP})\r\narise as some of the best convex relaxations to nonconvex problems\r\nlike graph optimization~\\cite{lovasz1979shannon,goemans1995improved},\r\ninteger programming~\\cite{sherali1990hierarchy,lovasz1991cones,lasserre2001explicit,laurent2003comparison},\r\nand polynomial optimization~\\cite{lasserre2001global,parrilo2000structured}.\r\n","start":4930,"end":5871},{"raw_tex":"To guarantee an exact reformulation, clique tree conversion chooses\r\nthe index sets $J_{1},\\ldots,J_{\\ell}$ in (\\ref{eq:relax}) as the\r\n\\emph{bags} of a \\emph{tree decomposition} for the \\emph{sparsity\r\ngraph} of the data matrices $C,A_{1},\\ldots,A_{m}$. Accordingly,\r\nthe parameter $\\omega$ in (\\ref{eq:omega_def-1}) can only be small\r\nif the sparsity graph has a small \\emph{treewidth}. Below, we define\r\na graph $G$ by its vertex set $V(G)\\subseteq\\{1,2,\\ldots,n\\}$ and\r\nits edge set $E(G)\\subseteq V(G)\\times V(G)$.\r\n\\begin{definition}[Sparsity graph]\r\n\\label{def:sparsitygraph}The $n\\times n$ matrix $M$ (resp. the\r\nset of $n\\times n$ matrices $\\{M_{1},\\ldots,M_{m}\\}$) is said to\r\nhave \\emph{sparsity graph} $G$ if $G$ is an undirected simple graph\r\non $n$ vertices $V(G)=\\{1,\\ldots,n\\}$ and that $(i,j)\\in E(G)$\r\nif $M[i,j]\\ne0$ (resp. if there exists $M\\in\\{M_{1},\\ldots,M_{m}\\}$\r\nsuch that $M[i,j]\\ne0$). \r\n\\end{definition}\r\n%\r\n\\begin{definition}[Tree decomposition]\r\n\\label{def:treedecomp}A \\emph{tree decomposition} $\\T$ of a graph\r\n$G$ is a pair $(\\J,T)$, where each \\emph{bag} of vertices $J_{j}\\in\\J$\r\nis a subset of $V(G)$, and $T$ is a tree on $|\\J|\\le n$ vertices,\r\nsuch that:\r\n\\begin{enumerate}\r\n\\item (Vertex cover) For every $v\\in V(G)$, there exists $J_{k}\\in\\J$\r\nsuch that $v\\in J_{k}$;\r\n\\item (Edge cover) For every $(u,v)\\in E(G)$, there exists $J_{k}\\in\\J$\r\nsuch that $u\\in J_{k}$ and $v\\in J_{k}$; and\r\n\\item (Running intersection) If $v\\in J_{i}$ and $v\\in J_{j}$, then we\r\nalso have $v\\in J_{k}$ for every $k$ that lies on the path from\r\n$i$ to $j$ in the tree $T$.\r\n\\end{enumerate}\r\nThe \\emph{width} $\\wid(\\T)$ of the tree decomposition $\\T=(\\J,T)$\r\nis the size of its largest bag minus one, as in $\\max\\{|J_{k}|:J_{k}\\in\\J\\}-1.$\r\nThe \\emph{treewidth} $\\tw(G)$ of the graph $G$ is the minimum width\r\namongst all tree decompositions $\\T$.\r\n\\end{definition}\r\nThroughout this paper, we make the implicit assumption that a tree\r\ndecomposition with small width is known a priori for the sparsity\r\ngraph. In practice, such a tree decomposition can usually be found\r\nusing fill-reducing heuristics for sparse linear algebra; see Section~\\ref{sec:Preliminaries}.\r\n\r\nWe also make two explicit assumptions, which are standard in the literature\r\non interior-point methods.\r\n\r\n\\begin{assume}[Linear independence]\\label{ass:lin}We have $\\sum_{i=1}^{m}y_{i}A_{i}=0$\r\nif and only if $y=0$.\r\n\r\n\\end{assume}\r\n\r\nThe assumption is without loss of generality, because it can either\r\nbe enforced by eliminating $A_{i}\\bullet X=b_{i}$ for select $i$,\r\nor else these constraints are not consistent for all $i$. Under Assumption~\\ref{ass:lin},\r\nthe total number of constraints is bounded $m\\le\\omega n$ (due to\r\nthe fact that $|E(G)|\\le n\\cdot\\tw(G)$~\\cite{arnborg1987complexity}).\r\n\r\n\\begin{assume}[Primal-dual Slater's condition]\\label{ass:slater}There\r\nexist $X\\succ0,$ $y,$ and $S\\succ0$ satisfying $A_{i}\\bullet X=b_{i}$\r\nfor all $i\\in\\{1,\\ldots,m\\}$ and $\\sum_{i=1}^{m}y_{i}A_{i}+S=C$. \r\n\r\n\\end{assume}\r\n","start":17649,"end":20664},{"raw_tex":"The sets $\\R^{n}$ and $\\R^{m\\times n}$ are the length-$n$ real\r\nvectors and $m\\times n$ real matrices. We use ``MATLAB notation''\r\nin concatenating vectors and matrices:\r\n\r\n\\[\r\n[a,b]=\\begin{bmatrix}a & b\\end{bmatrix},\\qquad[a;b]=\\begin{bmatrix}a\\\\\r\nb\r\n\\end{bmatrix},\\qquad\\diag(a,b)=\\begin{bmatrix}a & 0\\\\\r\n0 & b\r\n\\end{bmatrix},\r\n\\]\r\nand the following short-hand to construct them:\r\n\\[\r\n[x_{i}]_{i=1}^{n}=\\begin{bmatrix}x_{1}\\\\\r\n\\vdots\\\\\r\nx_{n}\r\n\\end{bmatrix},\\qquad[x_{i,j}]_{i,j=1}^{m,n}=\\begin{bmatrix}x_{1,1} & \\cdots & x_{1,n}\\\\\r\n\\vdots & \\ddots & \\vdots\\\\\r\nx_{m,1} & \\cdots & x_{m,n}\r\n\\end{bmatrix}.\r\n\\]\r\nThe notation $X[i,j]$ refers to the element of $X$ in the $i$-th\r\nrow and $j$-th column, and $X[I,J]$ refers to the submatrix of $X$\r\nformed from the rows in $I\\subseteq\\{1,\\ldots,m\\}$ and columns in\r\n$J\\subseteq\\{1,\\ldots,n\\}$. The Frobenius inner product is $X\\bullet Y=\\tr(X^{T}Y)$,\r\nand the Frobenius norm is $\\|X\\|_{F}=\\sqrt{X\\bullet X}$. We use $\\nnz(X)$\r\nto denote the number of nonzero elements in $X$. \r\n\r\nThe sets $\\S^{n}\\subseteq\\R^{n\\times n},$ $\\S_{+}^{n}\\subset\\S^{n},$\r\nand $\\S_{++}^{n}\\subset\\S_{+}^{n}$ are the $n\\times n$ real symmetric\r\nmatrices, positive semidefinite matrices, and positive definite matrices,\r\nrespective. We write $X\\succeq Y$ to mean $X-Y\\in\\S_{+}^{n}$ and\r\n$X\\succ Y$ to mean $X-Y\\in\\S_{++}^{n}$. The (symmetric) vectorization\r\n\\[\r\n\\vector(X)=[X[1,1];\\sqrt{2}X[2,1];\\ldots;\\sqrt{2}X[m,1];X[2,2],\\ldots]\r\n\\]\r\noutputs the lower-triangular part of a symmetric matrix as a vector,\r\nwith factors of $\\sqrt{2}$ added so that $\\vector(X)^{T}\\vector(Y)=X\\bullet Y$. \r\n\r\nA graph $G$ is defined by its vertex set $V(G)\\subseteq\\{1,2,3,\\ldots\\}$\r\nand its edge set $E(G)\\subseteq V(G)\\times V(G)$. The graph $T$\r\nis a \\emph{tree} if it is connected and does not contain any cycles;\r\nwe refer to its vertices $V(T)$ as its \\emph{nodes}. Designating\r\na special node $r\\in V(T)$ as the \\emph{root }of the tree allows\r\nus to define the \\emph{parent} $p(v)$ of each node $v\\ne r$ as the\r\nfirst node encountered on the path from $v$ to $r$, and $p(r)=r$\r\nfor consistency. The set of children is defined $\\ch(v)=\\{u\\in V(T)\\backslash v:p(u)=v\\}$.\r\nNote that the edges $E(T)$ are fully determined by the parent pointer\r\n$p$ as $\\{v,p(v)\\}$ for all $v\\ne r$.\r\n\r\nThe set $\\S_{G}^{n}\\subseteq\\S^{n}$ is the set of $n\\times n$ real\r\nsymmetric matrices with sparsity graph $G$. We denote $P_{G}(X)=\\min_{Y\\in\\S_{G}^{n}}\\|X-Y\\|_{F}$\r\nas the Euclidean projection of $X\\in\\S^{n}$ onto $\\S_{G}^{n}$.\r\n","start":30594,"end":33117},{"raw_tex":"Let $\\T=(\\{J_{1},\\ldots,J_{\\ell}\\},T)$ be a tree decomposition with\r\nsmall width for the sparsity graph $G$ of the data matrices $C,A_{1},\\ldots,A_{m}$.\r\nWe define the graph $F\\supseteq G$ by taking each index set $J_{j}$\r\nof $\\T$ and interconnecting all pairs of vertices $u,v\\in J_{j}$,\r\nas in \r\n\\begin{align}\r\nV(F) & =V(G), & E(F) & =\\bigcup_{j=1}^{\\ell}\\{(u,v):u,v\\in J_{j}\\}.\\label{eq:chordal_compl}\r\n\\end{align}\r\nThe following fundamental result was first established by Grone et\r\nal.~\\cite{grone1984positive}. Constructive proofs allow us to recover\r\nall elements in $X\\succeq0$ from only the elements in $P_{F}(X)$\r\nusing a closed-form formula.\r\n\\begin{theorem}[Grone et al.~\\cite{grone1984positive}]\r\n\\label{thm:prim_sep}Given $Z\\in\\S_{F}^{n}$, there exists an $X\\succeq0$\r\nsatisfying $P_{F}(X)=Z$ if and only if $Z[J_{j},J_{j}]\\succeq0$\r\nfor all $j\\in\\{1,2,\\ldots,\\ell\\}$.\r\n\\end{theorem}\r\nWe can use Theorem~\\ref{thm:prim_sep} to reformulate (\\ref{eq:SDP})\r\ninto a reduced-complexity form. The key is to view (\\ref{eq:SDP})\r\nas an optimization over $P_{F}(X)$, since \r\n\\[\r\nC\\bullet X=\\sum_{i,j=1}^{n}C_{i,j}X_{i,j}=\\sum_{(i,j)\\in F}C_{i,j}X_{i,j}=C\\bullet P_{F}(X),\r\n\\]\r\nand similarly $A_{i}\\bullet X=A_{i}\\bullet P_{F}(X)$. Theorem~\\ref{thm:prim_sep}\r\nallows us to account for $X\\succeq0$ implicitly, by optimizing over\r\n$Z=P_{F}(X)$ in the following\r\n\\begin{alignat}{3}\r\n & \\text{minimize }\\quad & C\\bullet Z\\label{eq:CSDP}\\\\\r\n & \\text{subject to } & A_{i}\\bullet Z & =b_{i} & \\qquad\\text{for all } & i\\in\\{1,\\ldots,m\\},\\nonumber \\\\\r\n &  & Z[J_{j},J_{j}] & \\succeq0 & \\text{for all } & j\\in\\{1,\\ldots,\\ell\\}.\\nonumber \r\n\\end{alignat}\r\nNext, we split the principal submatrices into distinct matrix variables,\r\ncoupled by the need for their overlapping elements to agree. Define\r\nthe \\emph{overlap} operator $\\NN_{i,j}(\\cdot)$ to output the overlapping\r\nelements of two principal submatrices given the latter as input:\r\n\\[\r\n\\NN_{i,j}(X[J_{j},J_{j}])=X[J_{i}\\cap J_{j},\\;J_{i}\\cap J_{j}]=\\NN_{j,i}(X[J_{i},J_{i}]).\r\n\\]\r\nThe running intersection property of the tree decomposition allows\r\nus to enforce this agreement using $\\ell-1$ pairwise block comparisons.\r\n\\begin{theorem}[Fukuda et al.~\\cite{fukuda2001exploiting}]\r\n\\label{thm:ctree}Given $X_{1},X_{2},\\ldots,X_{\\ell}$ for $X_{j}\\in\\S^{|J_{j}|}$,\r\nthere exists $Z$ satisfying $Z[J_{j},J_{j}]=X_{j}$ for all $j\\in\\{1,2,\\ldots,\\ell\\}$\r\nif and only if $\\NN_{i,j}(X_{j})=\\NN_{j,i}(X_{i})$ for all $(i,j)\\in E(T)$. \r\n\\end{theorem}\r\nSplitting the objective $C$ and constraint matrices $A_{1},\\ldots,A_{m}$\r\ninto $C_{1},\\ldots,C_{\\ell}$ and $A_{1,1},\\ldots,A_{m,\\ell}$ to\r\nsatisfy \r\n\\begin{align}\r\nC_{1}\\bullet X[J_{1},J_{1}]+C_{2}\\bullet X[J_{2},J_{2}]+\\cdots+C_{\\ell}\\bullet X[J_{\\ell},J_{\\ell}] & =C\\bullet X,\\label{eq:split_constr}\\\\\r\nA_{i,1}\\bullet X[J_{1},J_{1}]+A_{i,2}\\bullet X[J_{2},J_{2}]+\\cdots+A_{i,\\ell}\\bullet X[J_{\\ell},J_{\\ell}] & =A_{i}\\bullet X,\\nonumber \r\n\\end{align}\r\nand applying Theorem~\\ref{thm:ctree} yields the following\r\n\\begin{alignat}{3}\r\n & \\text{minimize }\\quad & \\sum_{j=1}^{\\ell}C_{j}\\bullet X_{j} &  &  & \\tag{CTC}\\label{eq:CTC}\\\\\r\n & \\text{subject to } & \\sum_{j=1}^{\\ell}A_{i,j}\\bullet X_{j} & =b_{i} & \\text{for all } & i\\in\\{1,\\ldots,m\\},\\nonumber \\\\\r\n &  & \\NN_{i,j}(X_{j}) & =\\NN_{j,i}(X_{i}) & \\qquad\\text{for all } & (i,j)\\in E(T),\\nonumber \\\\\r\n &  & X_{j} & \\succeq0 & \\text{for all } & j\\in\\{1,\\ldots,\\ell\\},\\nonumber \r\n\\end{alignat}\r\nwhich vectorizes into a linear conic program in standard form\r\n\\begin{alignat}{4}\r\n & \\text{minimize } & c^{T}x, &  & \\qquad\\qquad & \\text{maximize } & \\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}^{T}y,\\label{eq:vecCTC}\\\\\r\n & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}x & =\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}, &  & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}y+s & =c,\\nonumber \\\\\r\n &  & x & \\in\\K, &  &  & s & \\in\\K_{*}\\nonumber \r\n\\end{alignat}\r\nover the Cartesian product of $\\ell\\le n$ smaller semidefinite cones\r\n\\begin{equation}\r\n\\K=\\K_{*}=\\S_{+}^{|J_{1}|}\\times\\S_{+}^{|J_{2}|}\\times\\cdots\\times\\S_{+}^{|J_{\\ell}|}.\r\n\\end{equation}\r\nHere, $\\A=[\\vector(A_{i,j})^{T}]_{i,j=1}^{m,\\ell}$ and $c=[\\vector(C_{j})]_{j=1}^{\\ell}$\r\ncorrespond to (\\ref{eq:split_constr}), and the \\emph{overlap constraints}\r\nmatrix $\\N=[\\N_{i,j}]_{i,j=1}^{\\ell,\\ell}$ is implicitly defined\r\nby the relation\r\n\\begin{equation}\r\n\\N_{i,j}\\vector(X_{j})=\\begin{cases}\r\n+\\vector(\\NN_{p(i),i}(X_{i})) & j=i,\\\\\r\n-\\vector(\\NN_{i,p(i)}(X_{p(i)})) & j=p(i),\\\\\r\n0 & \\text{otherwise},\r\n\\end{cases}\\label{eq:Nmat}\r\n\\end{equation}\r\nfor every non-root node $i$ on $T$. (To avoid all-zero rows in $\\N$,\r\nwe define $\\N_{i,j}\\,\\vector(X_{j})$ as the empty length-zero vector\r\n$\\R^{0}$ if $i$ is the root node.)\r\n","start":35830,"end":40566},{"raw_tex":"where the \\emph{scaling matrix} $\\D_{s}$ is block-diagonal with fully-dense\r\nblocks \r\n\\begin{equation}\r\n\\D_{s}=\\diag(\\D_{s,1},\\ldots,\\D_{s,\\ell}),\\qquad\\D_{s,j}\\succ0\\quad\\text{for all }j\\in\\{1,\\ldots,\\ell\\}.\\label{eq:scal}\r\n\\end{equation}\r\n","start":43294,"end":43535},{"raw_tex":"\\begin{definition}[Topological ordering]\r\n\\label{def:topological}An ordering\\emph{ }$\\pi:\\{1,2,\\ldots,n\\}\\to V(T)$\r\non the tree graph $T$ with $n$ nodes is said to be \\emph{topological}~\\cite[p.~10]{vandenberghe2015chordal}\r\nif, by designating $\\pi(n)$ as the root of $T$, each node is indexed\r\nbefore its parent:\r\n\\[\r\n\\pi^{-1}(v)<\\pi^{-1}(p(v))\\qquad\\text{ for all }v\\ne r,\r\n\\]\r\nwhere $\\pi^{-1}(v)$ denotes the index associated with the node $v$.\r\n\\end{definition}\r\n\\begin{lemma}[No block fill]\r\n\\label{lem:blocktree}Let $J_{1},\\ldots,J_{n}$ satisfy $\\bigcup_{j=1}^{n}J_{j}=\\{1,\\ldots,d\\}$\r\nand $J_{i}\\cap J_{j}=\\emptyset$ for all $i\\ne j$, and let $H\\succ0$\r\nbe a $d\\times d$ matrix satisfying\r\n\\[\r\nH[J_{i},J_{j}]\\ne0\\qquad\\implies\\qquad(i,j)\\in E(T)\r\n\\]\r\nfor a tree graph $T$ on $n$ nodes. If $\\pi$ is a topological ordering\r\non $T$ and $\\Pi$ is a permutation matrix satisfying \r\n\\[\r\n(\\Pi H\\Pi)[J_{i},J_{j}]=H[J_{\\pi(i)},J_{\\pi(j)}]\\qquad\\text{for all }i,j\\in\\{1,\\ldots,n\\},\r\n\\]\r\nthen $\\Pi H\\Pi^{T}$ factors into $LL^{T}$ where the Cholesky factor\r\n$L$ satisfies\r\n\\[\r\nL[J_{i},J_{j}]\\ne0\\qquad\\implies\\qquad(\\Pi H\\Pi)[J_{i},J_{j}]\\ne0\\qquad\\text{ for all }i>j.\r\n\\]\r\nTherefore, sparse Cholesky factorization solves $Hx=b$ for $x$ by:\r\n(i) factoring $\\Pi H\\Pi^{T}$ into $LL^{T}$ in $O(\\beta^{3}n)$ operations\r\nand $O(\\beta^{2}n)$ memory where $\\beta=\\max_{j}|J_{j}|$, and (ii)\r\nsolving $Ly=\\Pi b$ and $L^{T}z=y$ and $x=\\Pi^{T}z$ in $O(\\beta^{2}n)$\r\noperations and memory. \r\n\\end{lemma}\r\nThis is a simple block-wise extension of the tree elimination result\r\noriginally due to Parter~\\cite{parter1961use}; see also George and\r\nLiu~\\cite[Lemma 6.3.1]{george1981computer}. In practice, a topological\r\nordering can be found by assigning indices $n,n-1,n-2,\\ldots$ in\r\ndecreasing ordering during a depth-first search traversal of the tree.\r\nIn fact, the minimum degree heuristic is guaranteed to generate a\r\ntopological ordering~\\cite{george1989evolution}.\r\n","start":46596,"end":48548},{"raw_tex":"The dualization technique of Löfberg~\\cite{lofberg2009dualize} swaps\r\nthe roles played by the primal and the dual problems in a linear conic\r\nprogram, by rewriting a primal standard form problem into dual standard\r\nform, and vice versa. Applying dualization to (\\ref{eq:vecCTC}) yields\r\nthe following\r\n\\begin{alignat}{4}\r\n & \\text{minimize } & \\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}^{T}x_{1} &  & \\qquad & \\text{maximize } & -c^{T}y\\label{eq:dualCTC}\\\\\r\n & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}x_{1}-x_{2} & =-c, &  & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}y+s_{1} & =\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix},\\nonumber \\\\\r\n &  & x_{1}\\in\\R^{f},\\,x_{2} & \\in\\K. &  &  & -y+s_{2} & =0,\\nonumber \\\\\r\n &  &  &  &  &  & s_{1}\\in\\{0\\}^{f},\\,s_{2} & \\in\\K.\\nonumber \r\n\\end{alignat}\r\nwhere we use $f$ to denote the number of equality constraints in\r\n(\\ref{eq:CTC}). Observe that the dual problem in (\\ref{eq:dualCTC})\r\nis identical to the primal problem in (\\ref{eq:vecCTC}), so that\r\na dual solution $y^{\\star}$ to (\\ref{eq:dualCTC}) immediately serves\r\nas a primal solution to (\\ref{eq:vecCTC}), and hence also (\\ref{eq:CTC}).\r\n\r\nModern interior-point methods solve (\\ref{eq:dualCTC}) by embeding\r\nthe free variable $x_{1}\\in\\R^{f}$ and fixed variable $s_{1}\\in\\{0\\}^{f}$\r\ninto a second-order cone (see Sturm~\\cite{sturm1999using} and Andersen~\\cite{andersen2002handling}):\r\n\\begin{alignat}{4}\r\n & \\text{minimize } & \\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}^{T}x_{1} &  & \\qquad & \\text{maximize } & -c^{T}y\\label{eq:dualCTC-2}\\\\\r\n & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}x_{1}-x_{2} & =-c, &  & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}y+s_{1} & =\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix},\\nonumber \\\\\r\n &  & \\|x_{1}\\|\\le x_{0},\\,x_{2} & \\in\\K. &  &  & -y+s_{2} & =0,\\nonumber \\\\\r\n &  &  &  &  &  & s_{0} & =0,\\nonumber \\\\\r\n &  &  &  &  &  & \\|s_{1}\\|\\le s_{0},\\,s_{2} & \\in\\K.\\nonumber \r\n\\end{alignat}\r\nWhen (\\ref{eq:dualCTC-2}) is solved using an interior-point method,\r\nthe normal equation solved at each iteration takes the form\r\n\\begin{equation}\r\n\\left(\\D_{s}+\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}\\D_{f}\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}\\right)\\Delta y=r\\label{eq:Hxr_dual}\r\n\\end{equation}\r\nwhere $\\D_{s}$ is comparable as before in (\\ref{eq:scal}), and \r\n\\begin{equation}\r\n\\D_{f}=\\sigma I+ww^{T},\\qquad\\sigma>0\\label{eq:scal_f}\r\n\\end{equation}\r\nis the rank-1 perturbation of a scaled identity matrix. The standard\r\nprocedure, as implemented in SeDuMi~\\cite{sturm1999using,sturm2002implementation}\r\nand MOSEK~\\cite{andersen2003implementing}, is to form the sparse\r\nmatrix $\\H$ and dense vector $\\q$, defined\r\n\\begin{align}\r\n\\H & =\\D_{s}+\\sigma\\A^{T}\\A+\\sigma\\N^{T}\\N, & \\q & =\\begin{bmatrix}\\A^{T} & \\N^{T}\\end{bmatrix}w.\\label{eq:Hqdef}\r\n\\end{align}\r\nand then solve (\\ref{eq:Hxr_dual}) using a rank-1 update\\footnote{To keep our derivations simple, we perform the rank-1 update using\r\nthe Sherman\\textendash Morrison\\textemdash Woodbury (SMW) formula.\r\nIn practice, the product-form Cholesky Factor (PFCF) formula of Goldfarb\r\nand Scheinberg~\\cite{goldfarb2005product} is more numerically stable\r\nand more widely used~\\cite{sturm1999using,sturm2002implementation}.\r\nOur complexity results remain valid in either cases because the PFCF\r\nis a constant factor of approximately two times slower than the SWM~\\cite{goldfarb2005product}.}\r\n\\begin{equation}\r\n\\Delta y=(\\H+\\q\\q^{T})^{-1}r=\\left(I-\\frac{(\\H^{-1}\\q)\\q^{T}}{1+\\q^{T}(\\H^{-1}\\q)}\\right)\\H^{-1}r,\\label{eq:update}\r\n\\end{equation}\r\nat a cost comparable to the solution of $\\H u=r$ for two right-hand\r\nsides. (In Appendix~\\ref{sec:inequality}, we repeat these derivations\r\nfor the version of (\\ref{eq:vecCTC}) in which $\\A x=b$ is replaced\r\nby the inequality $\\A x\\le b$.)\r\n","start":50272,"end":54082},{"raw_tex":"Define the $m\\times\\frac{1}{2}n(n+1)$ matrix $\\M$ as the vectorization\r\nof the linear constraints in (\\ref{eq:SDP}), as in\r\n\\[\r\n\\M\\,\\vector(X)=\\begin{bmatrix}\\vector(A_{1})^{T}\\\\\r\n\\vdots\\\\\r\n\\vector(A_{m})^{T}\r\n\\end{bmatrix}\\vector(X)=\\begin{bmatrix}\\vector(A_{1})^{T}\\vector(X)\\\\\r\n\\vdots\\\\\r\n\\vector(A_{m})^{T}\\vector(X)\r\n\\end{bmatrix}=\\begin{bmatrix}A_{1}\\bullet X\\\\\r\n\\vdots\\\\\r\nA_{m}\\bullet X\r\n\\end{bmatrix}.\r\n\\]\r\nIn reformulating (\\ref{eq:SDP}) into (\\ref{eq:CTC}), the splitting\r\nconditions (\\ref{eq:split_constr}) can be rewritten as the following\r\n\\begin{align}\r\nc^{T}\\P & =\\vector(C)^{T}, & \\A\\P & =\\M,\\label{eq:split_constr2}\r\n\\end{align}\r\nwhere $c=[\\vector(C_{j})]_{j=1}^{\\ell}$ and $\\A=[\\vector(A_{i,j})^{T}]_{i,j=1}^{m,\\ell}$\r\nare the data for the vectorized verison of (\\ref{eq:CTC}).\r\n","start":109481,"end":110277},{"raw_tex":"We solve the dualized problem (\\ref{eq:dualCTC-2}) by solving its\r\nextended homogeneous self-dual embedding\\begin{subequations}\\label{eq:hsd}\r\n\\begin{alignat}{2}\r\n & \\text{min.}\\quad & (\\nu+1)\\theta\\\\\r\n & \\text{s.t.} & \\begin{bmatrix}0 & +\\M^{T} & -\\c & -r_{d}\\\\\r\n-\\M & 0 & +\\b & -r_{p}\\\\\r\n+\\c^{T} & -\\b^{T} & 0 & -r_{c}\\\\\r\nr_{d}^{T} & r_{p}^{T} & r_{c} & 0\r\n\\end{bmatrix}\\begin{bmatrix}x\\\\\r\ny\\\\\r\n\\tau\\\\\r\n\\theta\r\n\\end{bmatrix}+\\begin{bmatrix}s\\\\\r\n0\\\\\r\n\\kappa\\\\\r\n0\r\n\\end{bmatrix} & =\\begin{bmatrix}0\\\\\r\n0\\\\\r\n0\\\\\r\n\\nu+1\r\n\\end{bmatrix}\\label{eq:hsdfeas}\\\\\r\n &  & x,s\\in\\C\\quad\\kappa,\\tau & \\ge0,\\label{eq:hsdcone}\r\n\\end{alignat}\r\nwhere the data is given in standard form\r\n\\begin{gather}\r\n\\M=\\begin{bmatrix}0 & -\\begin{bmatrix}\\A^{T} & \\N^{T}\\end{bmatrix} & +I\\end{bmatrix},\\quad\\c^{T}=\\begin{bmatrix}0 & \\begin{bmatrix}b^{T} & 0\\end{bmatrix} & 0\\end{bmatrix},\\quad\\b=-c,\\\\\r\n\\C=\\{(x_{0},x_{1}\\}\\in\\R^{p+1}:\\|x_{1}\\|\\le x_{0}\\}\\times\\K,\r\n\\end{gather}\r\nand the residual vectors are defined\r\n\\begin{equation}\r\nr_{d}=\\one_{\\C}-\\c,\\quad r_{p}=-\\M\\one_{\\C}+\\b,\\quad r_{c}=1+\\c^{T}\\one_{\\C}.\r\n\\end{equation}\r\n\\end{subequations}Here, $\\nu$ is the order of the cone $\\C$, and\r\n$\\one_{\\C}$ is its identity element\r\n\\begin{align}\r\n\\nu & =1+|J_{1}|+\\cdots+|J_{\\ell}|, & \\one_{\\C} & =[1;0;\\ldots;0;\\vector(I_{|J_{1}|});\\ldots;\\vector(I_{|J_{\\ell}|})].\r\n\\end{align}\r\nProblem (\\ref{eq:hsd}) has optimal value $\\theta^{\\star}=0$. Under\r\nthe primal-dual Slater's conditions (Assumption~\\ref{ass:slater}),\r\nan interior-point method is guaranteed to converge to an $\\epsilon$-accurate\r\nsolution with $\\tau>0$, and this yields an $\\epsilon$-feasible and\r\n$\\epsilon$-accurate solution to the dualized problem (\\ref{eq:dualCTC-2})\r\nby rescaling $x/\\tau$ and $y=y/\\tau$ and $s=s/\\tau$. The following\r\nresult is adapted from~\\cite[Lemma~5.7.2]{de2000self} and \\cite[Theorem~22.7]{vanderbei2015linear}.\r\n","start":115490,"end":117365},{"raw_tex":"We solve the homogeneous self-dual embedding (\\ref{eq:hsd}) using\r\nthe short-step method of Nesterov and Todd~\\cite[Algorithm~6.1]{nesterov1998primal}\r\n(and also Sturm and S.~Zhang~\\cite[Section~5.1]{sturm1999symmetric}),\r\nnoting that SeDuMi reduces to it in the worst case; see~\\cite{sturm2002implementation}\r\nand~\\cite{sturm1997wide}. Beginning at the following strictly feasible,\r\nperfectly centered point\r\n\\begin{gather}\r\n\\theta^{(0)}=\\tau^{(0)}=\\kappa^{(0)}=1,\\quad y^{(0)}=0,\\quad x^{(0)}=s^{(0)}=\\one_{\\C},\\label{eq:hsdinit}\r\n\\end{gather}\r\nwith barrier parameter $\\mu=1$, we take the following steps\\begin{subequations}\\label{eq:hsd_ipm}\r\n\\begin{align}\r\n\\mu^{+} & =\\left(1-\\frac{1}{15\\sqrt{\\nu+1}}\\right)\\cdot\\frac{x^{T}s+\\tau\\kappa}{\\nu+1},\\\\\r\n(x^{+},y^{+},s^{+},\\tau^{+},\\theta^{+},\\kappa^{+}) & =(x,y,s,\\tau,\\theta,\\kappa)+(\\Delta x,\\Delta y,\\Delta s,\\Delta\\tau,\\Delta\\theta,\\Delta\\kappa).\\nonumber \r\n\\end{align}\r\n\\end{subequations}along the search direction defined by the linear\r\nsystem~\\cite[Eqn. 9]{todd1998nesterov} \\begin{subequations}\\label{eq:hsd_nt}\r\n\\begin{align}\r\n\\begin{bmatrix}0 & +\\M^{T} & -\\c & -r_{p}\\\\\r\n-\\M & 0 & +\\b & -r_{d}\\\\\r\n+\\c^{T} & -\\b^{T} & 0 & -r_{c}\\\\\r\n+r_{p}^{T} & +r_{d}^{T} & +r_{c} & 0\r\n\\end{bmatrix}\\begin{bmatrix}\\Delta x\\\\\r\n\\Delta y\\\\\r\n\\Delta\\tau\\\\\r\n\\Delta\\theta\r\n\\end{bmatrix}+\\begin{bmatrix}\\Delta s\\\\\r\n0\\\\\r\n\\Delta\\kappa\\\\\r\n0\r\n\\end{bmatrix} & =0,\\\\\r\ns+\\Delta s+\\nabla^{2}F(w)\\Delta x+\\mu^{+}\\nabla F(x) & =0,\\\\\r\n\\kappa+\\Delta\\kappa+(\\kappa/\\tau)\\Delta\\tau-\\mu^{+}\\tau^{-1} & =0.\r\n\\end{align}\r\n\\end{subequations}Here, $F$ is the usual self-concordant barrier\r\nfunction on $\\C$\r\n\\begin{align}\r\nF([x_{0};x_{1};\\vector(X_{1});\\ldots;\\vector(X_{\\ell})])= & -\\log\\left(\\frac{1}{2}x_{0}^{2}-\\frac{1}{2}x_{1}^{T}x_{1}\\right)-\\sum_{j=1}^{\\ell}\\log\\det(X_{j})\\label{eq:Fdef}\r\n\\end{align}\r\nand $w\\in\\Int(\\C)$ is the unique \\emph{scaling point} satisfying\r\n$\\nabla^{2}F(w)x=s$, which can be computed from $x$ and $s$ in\r\nclosed-form. The following iteration bound is an immediate consequence\r\nof~\\cite[Theorem~6.4]{nesterov1998primal}; see also~\\cite[Theorem~5.1]{sturm1999symmetric}.\r\n\\begin{lemma}[Short-Step Method]\r\n\\label{lem:short-step}The sequence in (\\ref{eq:hsd_ipm}) arrives\r\nat an iterate $(x,y,s,\\tau,\\theta,\\kappa)$ satisfying the conditions\r\nof Lemma~\\ref{lem:eps} with $\\gamma=9/10$ in at most $O(\\sqrt{\\nu}\\log(1/\\epsilon))$\r\niterations. \r\n\\end{lemma}\r\nThe cost of each interior-point iteration is dominated by the cost\r\nof computing the search direction in (\\ref{eq:hsd_nt}). Using elementary\r\nbut tedious linear algebra, we can show that if\\begin{subequations}\\label{eq:newt}\r\n\\begin{equation}\r\n(\\M\\D^{-1}\\M^{T})\\begin{bmatrix}v_{1} & v_{2} & v_{3}\\end{bmatrix}=\\begin{bmatrix}0 & -\\b & r_{p}\\end{bmatrix}-\\M\\D^{-1}\\begin{bmatrix}d & \\c & r_{d}\\end{bmatrix}\\label{eq:hsd_nrm}\r\n\\end{equation}\r\nwhere $\\D=\\nabla^{2}F(w)$ and $d=-s-\\mu^{+}\\nabla F(x)$, and \r\n\\begin{equation}\r\n\\begin{bmatrix}u_{1} & u_{2} & u_{3}\\end{bmatrix}=\\D^{-1}(\\begin{bmatrix}d & c & r_{d}\\end{bmatrix}+\\M^{T}\\begin{bmatrix}v_{1} & v_{2} & v_{3}\\end{bmatrix}),\\label{eq:hsd_back1}\r\n\\end{equation}\r\nthen\r\n\\begin{align}\r\n\\left(\\begin{bmatrix}-\\D_{0} & -r_{c}\\\\\r\nr_{c} & 0\r\n\\end{bmatrix}-\\begin{bmatrix}\\c & r_{d}\\\\\r\n-\\b & r_{p}\r\n\\end{bmatrix}^{T}\\begin{bmatrix}u_{2} & u_{3}\\\\\r\nv_{2} & v_{3}\r\n\\end{bmatrix}\\right)\\begin{bmatrix}\\Delta\\tau\\\\\r\n\\Delta\\theta\r\n\\end{bmatrix} & =\\begin{bmatrix}-d_{0}\\\\\r\n0\r\n\\end{bmatrix}-\\begin{bmatrix}\\c & r_{d}\\\\\r\n-\\b & r_{p}\r\n\\end{bmatrix}^{T}\\begin{bmatrix}u_{1}\\\\\r\nv_{1}\r\n\\end{bmatrix},\\\\\r\n\\begin{bmatrix}\\Delta x\\\\\r\n\\Delta y\r\n\\end{bmatrix} & =\\begin{bmatrix}u_{1}\\\\\r\nv_{1}\r\n\\end{bmatrix}-\\begin{bmatrix}u_{1} & u_{2}\\\\\r\nv_{1} & v_{2}\r\n\\end{bmatrix}\\begin{bmatrix}\\Delta\\tau\\\\\r\n\\Delta\\theta\r\n\\end{bmatrix},\\\\\r\n\\Delta s & =d-\\D\\Delta x,\\\\\r\n\\Delta\\kappa & =d_{0}-\\D_{0}\\Delta\\tau,\\label{eq:hsd_back2}\r\n\\end{align}\r\n\\end{subequations}where $\\D_{0}=\\kappa/\\tau$ and $d_{0}=-\\kappa+\\mu^{+}\\tau^{-1}$.\r\nHence, the cost of computing the search direction is dominated by\r\nthe cost of solving the normal equation for three different right-hand\r\nsides. Here, the normal matrix is written\r\n\r\n\\[\r\n\\M\\D^{-1}\\M^{T}=\\diag(W_{1}\\otimes_{s}W_{1},\\ldots,W_{\\ell}\\otimes_{s}W_{\\ell})+\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}(w_{1}w_{1}^{T}+\\sigma I)\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix},\r\n\\]\r\nwhere $\\sigma=\\frac{1}{2}(w_{0}^{2}-w_{1}^{T}w_{1})>0$ and $\\otimes_{s}$\r\ndenotes the symmetric Kronecker product~\\cite{alizadeh1998primal}\r\nimplicitly defined to satisfy\r\n\\[\r\n(A\\otimes_{s}B)\\vector(X)=\\frac{1}{2}\\vector(AXB^{T}+BXA^{T})\\qquad\\text{for all }X=X^{T}.\r\n\\]\r\nUnder the hypothesis on $\\A$ stated in Theorem~\\ref{thm:lintime},\r\nthe normal matrix satisfies the assumptions of Lemma~\\ref{lem:normal},\r\nand can therefore be solved in linear $O(n)$ time and memory. \r\n","start":120060,"end":124866}]},"statement":{"raw_tex":"\\begin{theorem}[Near-linear time]\r\n\\label{thm:lintime}Let $\\T=(\\{J_{1},\\ldots,J_{\\ell}\\},T)$ be a\r\ntree decomposition for the sparsity graph of $C,A_{1},\\ldots,A_{m}\\in\\S^{n}$.\r\nIn the corresponding instance of (\\ref{eq:CTC}), let each constraint\r\nbe written\r\n\\begin{equation}\r\n\\sum_{j=1}^{\\ell}A_{i,j}\\bullet X_{j}=A_{i,j}\\bullet X_{j}+A_{i,k}\\bullet X_{k}=b_{i}\\qquad(j,k)\\in E(T).\\label{eq:structA}\r\n\\end{equation}\r\nUnder Assumptions~\\ref{ass:lin} \\&~\\ref{ass:slater}, there exists\r\nan algorithm that computes an iterate $(x,y,s)\\in\\K\\times\\R^{p}\\times\\K_{*}$\r\nsatisfying\r\n\\begin{align}\r\n\\left\\Vert \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}x-\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}\\right\\Vert  & \\le\\epsilon, & \\left\\Vert \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}y-s+c\\right\\Vert  & \\le\\epsilon, & \\frac{x^{T}s}{\\sum_{j=1}^{\\ell}|J_{j}|} & \\le\\epsilon\\label{eq:eps_accurate}\r\n\\end{align}\r\nin $O(\\omega^{6.5}n^{1.5}\\log(1/\\epsilon))$ time and $O(\\omega^{4}n)$\r\nspace, where $\\omega=\\max_{j}|J_{j}|=1+\\wid(\\T)$.\r\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}[Near-linear time]\r\n\\label{thm:lintime}Let $\\T=(\\{J_{1},\\ldots,J_{\\ell}\\},T)$ be a\r\ntree decomposition for the sparsity graph of $C,A_{1},\\ldots,A_{m}\\in\\S^{n}$.\r\nIn the corresponding instance of (\\ref{eq:CTC}), let each constraint\r\nbe written\r\n\\begin{equation}\r\n\\sum_{j=1}^{\\ell}A_{i,j}\\bullet X_{j}=A_{i,j}\\bullet X_{j}+A_{i,k}\\bullet X_{k}=b_{i}\\qquad(j,k)\\in E(T).\\label{eq:structA}\r\n\\end{equation}\r\nUnder Assumptions~\\ref{ass:lin} \\&~\\ref{ass:slater}, there exists\r\nan algorithm that computes an iterate $(x,y,s)\\in\\K\\times\\R^{p}\\times\\K_{*}$\r\nsatisfying\r\n\\begin{align}\r\n\\left\\Vert \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}x-\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}\\right\\Vert  & \\le\\epsilon, & \\left\\Vert \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}y-s+c\\right\\Vert  & \\le\\epsilon, & \\frac{x^{T}s}{\\sum_{j=1}^{\\ell}|J_{j}|} & \\le\\epsilon\\label{eq:eps_accurate}\r\n\\end{align}\r\nin $O(\\omega^{6.5}n^{1.5}\\log(1/\\epsilon))$ time and $O(\\omega^{4}n)$\r\nspace, where $\\omega=\\max_{j}|J_{j}|=1+\\wid(\\T)$.\r\n\\end{theorem}","start":59053,"end":60075}]},"proof":{"raw_tex":"\\begin{lemma}[Linear independence]\r\n\\label{lem:lin_indep}There exists $[u;v]\\ne0$ such that $\\A^{T}u+\\N^{T}v=0$\r\nif and only if there exists $y\\ne0$ such that $\\sum_{i}y_{i}A_{i}=0$.\r\n\\end{lemma}\n\n\\begin{lemma}[Primal Slater]\r\n\\label{lem:slaterp}There exists $x\\in\\Int(\\K)$ satisfying $\\A x=b$\r\nand $\\N x=0$ if and only if there exists an $X\\succ0$ satisfying\r\n$A_{i}\\bullet X=b_{i}$ for all $i\\in\\{1,\\ldots,m\\}$.\r\n\\end{lemma}\n\n\\begin{lemma}[Dual Slater]\r\n\\label{lem:slaterd}There exists $u,v$ satisfying $c-\\A^{T}u-\\N^{T}v\\in\\Int(\\K_{*})$\r\nif and only if there exists $y$ satisfying $C-\\sum_{i}y_{i}A_{i}\\succ0$.\r\n\\end{lemma}\n\n\\begin{lemma}[Linear-time normal equation]\r\n\\label{lem:normal}Let there exist $v_{i}\\in V(T)$ for each $i\\in\\{1,\\ldots,m\\}$\r\nsuch that\r\n\\begin{equation}\r\nA_{i,j}\\ne0\\qquad\\implies\\qquad j=v_{i}\\text{ or }j=p(v_{i}).\\label{eq:Asparsity}\r\n\\end{equation}\r\nDefine $\\H$ and $\\q$ according to (\\ref{eq:Hqdef}). Then, under\r\nAssumption~\\ref{ass:lin}:\r\n\\begin{enumerate}\r\n\\item (Forming) It costs $O(\\omega^{6}n)$ time and $O(\\omega^{4}n)$ space\r\nto form $\\H$ and $\\q$, where $\\omega=\\max_{j}|J_{j}|=1+\\wid(\\T)$.\r\n\\item (Factoring) Let $\\pi$ be a topological ordering on $T$, and define\r\nthe associated block topological permutation $\\Pi$ as in Lemma~\\ref{lem:blocktree}.\r\nThen, it costs $O(\\omega^{6}n)$ time and $O(\\omega^{4}n)$ space\r\nto factor $\\Pi\\H\\Pi^{T}$ into $\\L\\L^{T}$. \r\n\\item (Solving) Given $r$, $\\q$, $\\Pi$, and the Cholesky factor $\\L$\r\nsatisfying $\\L\\L^{T}=\\Pi\\H\\Pi^{T}$, it costs $O(\\omega^{4}n)$ time\r\nand space to solve $(\\H+\\q\\q^{T})u=r$ for $u$.\r\n\\end{enumerate}\r\n\\end{lemma}\n\n\\begin{proof}\r\nFor an instance of (\\ref{eq:CTC}), denote $\\ell=|\\mathcal{J}|\\le n$\r\nas its number of conic constraints, and $d=\\frac{1}{2}\\sum_{j=1}^{\\ell}|J_{j}|(|J_{j}|+1)\\le\\omega^{2}\\ell$\r\nas its total number of variables. Under linear independence (Assumption~\\ref{ass:lin}),\r\nthe constraint matrix $[\\A;\\N]$ associated with (\\ref{eq:CTC}) has\r\n$d$ columns and at most $d$ rows (Lemma~\\ref{lem:lin_indep}).\r\nWrite $\\xi_{i}^{T}$ as the $i$-th row of $[\\A;\\N]$, and assume\r\nwithout loss of generality that $[\\A;\\N]$ has exactly $d$ rows.\r\nObserve that $\\nnz(\\xi_{i})\\le\\omega(\\omega+1)$ by the definition\r\nof $\\N$ (\\ref{eq:Nmat}) and the hypothesis on $\\A$ via (\\ref{eq:Asparsity}),\r\nso $\\nnz([\\A;\\N])\\le2\\omega^{4}\\ell$. \r\n\r\n\\textbf{(i)} We form $\\H$ by setting $\\H\\gets\\D_{s}$ and then adding\r\n$\\H\\gets\\H+\\sigma\\xi_{i}\\xi_{i}^{T}$ one at a time, for $i\\in\\{1,2,\\ldots,d\\}$.\r\nThe first step forms $\\D_{s}=\\diag(\\D_{s}^{(1)},\\D_{s}^{(2)},\\ldots,\\D_{s}^{(\\ell)})$\r\nwhere $\\D_{s}^{(j)}=W_{j}\\otimes W_{j}$. Each $\\nnz(\\D_{s}^{(j)})=\\nnz(W_{j})^{2}=|J_{j}|^{2}(|J_{j}|+1)^{2}/4\\le\\omega^{4}$\r\nfor $j\\in\\{1,2,\\ldots,\\ell\\}$, so the total cost is $O(\\omega^{4}n)$\r\ntime and space. The second step adds $\\nnz(\\xi_{i}\\xi_{i}^{T})^{2}\\le\\omega^{2}(\\omega+1)^{2}$\r\nnonzeros per constraint over $d$ total constraints, for a total cost\r\nof $O(\\omega^{6}n)$ time and apparently $O(\\omega^{6}n)$ space.\r\nHowever, by the definition of $\\N$ (\\ref{eq:Nmat}) and the hypothesis\r\non $\\A$ via (\\ref{eq:Asparsity}), the $(j,k)$-th off-diagonal block\r\nof $\\xi_{i}\\xi_{i}^{T}$ is nonzero only if $(j,k)$ is an edge of\r\nthe tree $T$, as in\r\n\\[\r\n\\xi_{i}\\xi_{i}^{T}[J_{j},J_{k}]\\ne0\\qquad\\implies\\qquad(j,k)\\in E(T)\\qquad\\text{for all }j\\ne k.\r\n\\]\r\nHence, adding $\\mathbf{H}\\gets\\mathbf{H}+\\sigma\\xi_{i}\\xi_{i}^{T}$\r\none at a time results in at most $|V(T)|+|E(T)|$ dense blocks of\r\nat most $\\frac{1}{2}\\omega(\\omega+1)\\times\\frac{1}{2}\\omega(\\omega+1)$,\r\nfor a total memory cost of $O(\\omega^{4}n)$. \r\n\r\n\\textbf{(ii)} We form $\\q=[\\A^{T},\\N^{T}]w_{1}$ using a sparse matrix-vector\r\nproduct. Given that $\\nnz(w_{1})\\le d$ and $\\nnz([\\A;\\N])\\le2\\omega^{4}\\ell$,\r\nthis step costs $O(\\omega^{4}n)$ time and space.\r\n\r\n\\textbf{(iii)} We partition $\\H$ into $[\\H_{i,j}]_{i,j=1}^{\\ell}$\r\nto reveal a block sparsity pattern that coincides with the adjacency\r\nmatrix of $T$:\r\n\\begin{align*}\r\n\\H_{i,j} & =\\begin{cases}\r\n\\D_{s,i}+\\sigma\\sum_{k=1}^{\\ell}\\N_{k,i}^{T}\\N_{k,i}+\\sigma\\sum_{q=1}^{m}a_{q,i}a_{q,i}^{T} & i=j\\\\\r\n\\sigma\\sum_{k=1}^{\\ell}\\N_{k,i}^{T}\\N_{k,j}+\\sigma\\sum_{q=1}^{m}a_{q,i}a_{q,j}^{T} & (i,j)\\in E(T)\\\\\r\n0 & \\text{otherwise}\r\n\\end{cases}\r\n\\end{align*}\r\nwhere $a_{q,i}=\\vector(A_{q,i})$. According to Lemma~\\ref{lem:blocktree},\r\nthe permuted matrix $\\Pi\\H\\Pi^{T}$ factors into $\\L\\L^{T}$ with\r\nno block fill in $O(\\omega^{6}n)$ time and $O(\\omega^{4}n)$ space,\r\nbecause each block $\\H_{i,j}$ for $i,j\\in\\{1,2,\\ldots,\\ell\\}$ is\r\nat most order $\\frac{1}{2}\\omega(\\omega+1)$.\r\n\r\n\\textbf{(iv)} Using the rank-1 update formula (\\ref{eq:update}),\r\nthe cost of solving $(\\H+\\q\\q^{T})u=r$ is the same as the cost of\r\nsolving $\\H u=r$ for two right-hand sides, plus algebraic manipulations\r\nin $O(d)=O(\\omega^{2}n)$ time. Applying Lemma~\\ref{lem:blocktree}\r\nshows that the cost of solving $\\H u=r$ for each right-hand side\r\nis $O(\\omega^{4}n)$ time and space. \\qed\r\n\\end{proof}\n\n\\begin{lemma}\r\n\\label{lem:row-rank}The matrix $\\N$ in (\\ref{eq:Nmat}) has full\r\nrow rank, that is $\\det(\\N\\N^{T})\\ne0$.\r\n\\end{lemma}\n\n\\begin{proof}\r\nWe make $\\N=[\\N_{i,j}]_{i,j=1}^{\\ell}$ upper-triangular by ordering\r\nits blocks topologically on $T$: each nonempty block row $\\N_{j}$\r\ncontains a nonzero block at $\\N_{j,j}$ and a nonzero block at $\\N_{j,p(j)}$\r\nwhere the parent node $p(j)>j$ is ordered after $j$. Then, the claim\r\nfollows because each diagonal block $\\N_{j,j}$ implements a surjection\r\nand must therefore have full row-rank. \\qed\r\n\\end{proof}\n\n\\begin{lemma}[Orthogonal complement]\r\n\\label{lem:orth}Let $d=\\frac{1}{2}\\sum_{j=1}^{\\ell}|J_{j}|(|J_{j}|+1)$.\r\nImplicitly define the $d\\times\\frac{1}{2}n(n+1)$ matrix $\\P$ to\r\nsatisfy\r\n\\begin{align*}\r\n\\P\\,\\vector(X) & =[\\vector(X[J_{1},J_{1}]);\\ldots;\\vector(X[J_{\\ell},J_{\\ell}])]\\qquad\\forall X\\in\\S^{n}.\r\n\\end{align*}\r\nThen, (i) $\\N\\P=0$; (ii) every $x\\in\\R^{d}$ can be decomposed as\r\n$x=\\N^{T}u+\\P v$. \r\n\\end{lemma}\n\n\\begin{proof}\r\nFor each $x=[\\vector(X_{j})]_{j=1}^{\\ell}\\in\\R^{d}$, Theorem~\\ref{thm:ctree}\r\nsays that there exists a $Z$ satisfying $\\P\\,\\vector(Z)=x$ if and\r\nonly if $\\N x=0$. Equivalently, $x\\in\\mathrm{span}(\\P)$ if and only\r\nif $x\\perp\\mathrm{span}(\\N^{T})$. The ``only if'' part implies\r\n(i), while the ``if'' part implies (ii).\r\n\\end{proof}\n\n\\begin{proof}[Lemma~\\ref{lem:lin_indep}]We will prove that\r\n\\[\r\n\\text{exists }[u;v]\\ne0,\\quad\\A^{T}u+\\N^{T}v=0\\qquad\\iff\\qquad\\text{exists }y\\ne0,\\quad\\sum_{i=1}^{m}y_{i}A_{i}=0.\r\n\\]\r\n\\textbf{($\\implies$)} We must have $u\\ne0$, because $\\N$ has full\r\nrow rank by Lemma~\\ref{lem:row-rank}, and so $\\A^{T}0+\\N^{T}v=0$\r\nif and only if $v=0$. Multiplying by $\\P$ yields $\\P^{T}(\\A^{T}u+\\N^{T}v)=\\M^{T}u+0=0$\r\nand so setting $y=u\\ne0$ yields $\\M^{T}y=0$.\\textbf{ ($\\impliedby$)\r\n}We use Lemma~\\ref{lem:orth} to decompose $\\A^{T}y=\\P z+\\N^{T}v$.\r\nIf $\\vector(\\sum_{i}y_{i}A_{i})=\\M^{T}y=\\P^{T}\\A^{T}y=0,$ then $\\P^{T}\\P z=\\P^{T}\\A^{T}y-\\P^{T}\\N^{T}v=0$\r\nand so $\\P z=0$. Setting $u=-y\\ne0$ yields $\\A^{T}u+\\N^{T}v=0$.\\qed \\end{proof}\n\n\\begin{proof}[Lemma~\\ref{lem:slaterp}]We will prove that \r\n\\[\r\n\\text{exists }x\\in\\Int(\\K),\\quad\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}x=\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}\\qquad\\iff\\qquad\\text{exists }X\\succeq0,\\quad\\M\\,\\vector(X)=b.\r\n\\]\r\nDefine the chordal completion $F$ as in (\\ref{eq:chordal_compl}).\r\nObserve that $\\M\\,\\vector(X)=\\M\\,\\vector(Z)$ holds for all pairs\r\nof $P_{F}(X)=Z$, because each $A_{i}\\in\\S_{F}^{n}$ satisfies $A_{i}\\bullet X=A_{i}\\bullet P_{F}(X)$.\r\nAdditionally, the positive definite version of Theorem~\\ref{thm:prim_sep}\r\nis written\r\n\\begin{gather}\r\n\\text{exists }X\\succ0,\\quad P_{F}(X)=Z\\qquad\\iff\\qquad\\P\\,\\vector(Z)\\in\\Int(\\K).\\label{eq:prim_sep}\r\n\\end{gather}\r\nThis result was first established by Grone et al.~\\cite{grone1984positive};\r\na succinct proof can be found in~\\cite[Theorem~10.1]{vandenberghe2015chordal}.\\textbf{\r\n($\\implies$)} For every $x$ satisfying $\\N x=0$, there exists $Z$\r\nsuch that $\\P\\,\\vector(Z)=x$ due to Lemma~\\ref{lem:orth}. If additionally\r\n$x\\in\\Int(\\K)$, then there exists $X\\succ0$ satisfying $Z=P_{F}(X)$\r\ndue to (\\ref{eq:prim_sep}). We can verify that $\\M\\,\\vector(X)=\\M\\,\\vector(Z)=\\A\\P\\,\\vector(Z)=\\A x=b$.\\textbf{\r\n($\\impliedby$)} For every $X\\succ0,$ there exists $Z$ satisfying\r\n$Z=P_{F}(X)$ and $\\P\\,\\vector(Z)\\in\\Int(\\K)$ due to (\\ref{eq:prim_sep}).\r\nSet $u=\\P\\,\\vector(Z)$ and observe that $u\\in\\Int(\\K)$ and $\\N u=\\N\\P\\,\\vector(Z)=0$.\r\nIf additionally $\\M\\,\\vector(X)=b$, then $\\A u=\\A\\P\\,\\vector(Z)=\\M\\,\\vector(Z)=b$.\\qed \\end{proof}\n\n\\begin{proof}[Lemma~\\ref{lem:slaterp}]We will prove that\r\n\\[\r\n\\text{exists }u,v,\\quad c-\\A^{T}u-\\N^{T}v\\in\\Int(\\K_{*})\\qquad\\iff\\qquad\\text{exists }y,\\quad C-\\sum_{i}y_{i}A_{i}\\succ0.\r\n\\]\r\nDefine the chordal completion $F$ as in (\\ref{eq:chordal_compl}).\r\nTheorem~\\ref{thm:prim_sep} in (\\ref{eq:prim_sep}) has a dual theorem\r\n\\begin{gather}\r\n\\text{exists }S\\succ0,\\quad S\\in\\S_{F}^{n}\\qquad\\iff\\qquad\\text{exists }h\\in\\Int(\\K_{*}),\\quad\\vector(S)=\\P^{T}h.\\label{eq:dual_sep}\r\n\\end{gather}\r\nThis result readily follows from the positive semidefinite version\r\nproved by Alger et al.~\\cite{agler1988positive}; see also~\\cite[Theorem~9.2]{vandenberghe2015chordal}.\\textbf{\r\n($\\implies$)} For each $h=c-\\A^{T}u-\\N^{T}v$, define $S=C-\\sum_{i}u_{i}A_{i}$\r\nand observe that \r\n\\[\r\n\\P^{T}h=\\P^{T}(c-\\A^{T}u-\\N^{T}v)=\\vector(C)-\\M^{T}u-0=\\vector(S).\r\n\\]\r\nIf additionally $h\\in\\K_{*}$, then $S\\succ0$ due to (\\ref{eq:dual_sep}).\\textbf{\r\n($\\impliedby$)} For each $S=C-\\sum_{i}y_{i}A_{i}\\succ0$, there exists\r\nan $h\\in\\Int(\\K_{*})$ satisfying $\\vector(S)=\\P^{T}h$ due to (\\ref{eq:dual_sep}).\r\nWe use Lemma~\\ref{lem:orth} to decompose $h=\\P u+\\N^{T}v$. Given\r\nthat $\\vector(S)=\\P^{T}h=\\P^{T}\\P u+0$, we must actually have $\\P u=c-\\A^{T}y$\r\nsince $\\P^{T}(c-\\A^{T}y)=\\vector(C)-\\M^{T}y=\\vector(S)$. Hence $h=c-\\A^{T}y+\\N^{T}v$\r\nand $h\\in\\Int(\\K_{*})$.\\qed \\end{proof}\n\n\\begin{lemma}[$\\epsilon$-accurate and $\\epsilon$-feasible]\r\n\\label{lem:eps}If $(x,y,s,\\tau,\\theta,\\kappa)$ satisfies (\\ref{eq:hsdfeas})\r\nand (\\ref{eq:hsdcone}) and \r\n\\begin{align*}\r\n\\mu=\\frac{x^{T}s+\\tau\\kappa}{\\nu+1} & \\le\\epsilon, & \\tau\\kappa\\ge & \\gamma\\mu\r\n\\end{align*}\r\nfor constants $\\epsilon,\\gamma>0$, then the rescaled point $(x/\\tau,y/\\tau,s/\\tau)$\r\nsatisfies\r\n\\begin{align*}\r\n\\|\\M(x/\\tau)-\\b\\| & \\le K\\epsilon, & \\|\\M^{T}(y/\\tau)+(s/\\tau)-\\c\\| & \\le K\\epsilon, & \\frac{(x/\\tau)^{T}(s/\\tau)}{\\nu+1}\\le K\\epsilon\r\n\\end{align*}\r\nwhere $K$ is a constant.\r\n\\end{lemma}\n\n\\begin{proof}\r\nNote that (\\ref{eq:hsdfeas}) implies $\\mu=\\theta$ and\r\n\\begin{align*}\r\n\\|\\M(x/\\tau)-\\b\\| & =\\frac{\\|r_{p}\\|\\mu}{\\tau}, & \\|\\M^{T}(y/\\tau)+(s/\\tau)-\\c\\| & =\\frac{\\|r_{d}\\|\\mu}{\\tau}, & \\frac{(x/\\tau)^{T}(s/\\tau)}{\\nu+1}=\\frac{\\mu}{\\tau^{2}}.\r\n\\end{align*}\r\nHence, we obtain our desired result by upper-bounding $1/\\tau$. Let\r\n$(x^{\\star},y^{\\star},s^{\\star},\\tau^{\\star},\\theta^{\\star},\\kappa^{\\star})$\r\nbe a solution of (\\ref{eq:hsd}), and note that for every $(x,y,s,\\tau,\\theta,\\kappa)$\r\nsatisfying (\\ref{eq:hsdfeas}) and (\\ref{eq:hsdcone}), we have the\r\nfollowing via the skew-symmetry of (\\ref{eq:hsdfeas})\r\n\\[\r\n(x-x^{\\star})^{T}(s-s^{\\star})+(\\tau-\\tau^{\\star})(\\kappa-\\kappa^{\\star})=0.\r\n\\]\r\nRearranging yields\r\n\\[\r\n(\\nu+1)\\mu=x^{T}s+\\tau\\kappa=(x^{\\star})^{T}s+x^{T}(s^{\\star})+\\tau\\kappa^{\\star}+\\tau^{\\star}\\kappa\\ge\\tau^{\\star}\\kappa\r\n\\]\r\nand hence\r\n\\[\r\n\\kappa\\tau^{\\star}\\le\\mu(\\nu+1),\\quad\\tau\\kappa\\ge\\gamma\\mu\\qquad\\implies\\qquad\\tau\\ge\\frac{\\gamma}{\\nu+1}\\tau^{\\star}.\r\n\\]\r\n\r\nIf (\\ref{eq:SDP}) satisfies the primal-dual Slater's condition, then\r\n(\\ref{eq:CTC}) also satisfies the primal-dual Slater's condition\r\n(Lemmas~\\ref{lem:slaterp} \\&~\\ref{lem:slaterd}). Therefore, the\r\nvectorized version (\\ref{eq:vecCTC}) of (\\ref{eq:CTC}) attains a\r\nsolution $(\\hat{x},\\hat{y},\\hat{s})$ with $\\hat{x}^{T}\\hat{s}=0$,\r\nand the following\r\n\\[\r\nx^{\\star}=\\tau^{\\star}\\begin{bmatrix}\\|\\hat{y}\\|\\\\\r\n\\hat{y}\\\\\r\n\\hat{s}\r\n\\end{bmatrix},\\quad y^{\\star}=\\tau^{\\star}\\hat{x},\\quad s^{\\star}=\\tau^{\\star}\\begin{bmatrix}0\\\\\r\n0\\\\\r\n\\hat{x}\r\n\\end{bmatrix},\\quad\\tau^{\\star}=\\frac{\\nu+1}{\\|\\hat{y}\\|+\\one_{\\K}^{T}\\hat{s}+\\one_{\\K}^{T}\\hat{x}+1},\r\n\\]\r\nwith $\\theta^{\\star}=\\kappa^{\\star}=0$ is a solution to (\\ref{eq:hsd}).\r\nThis proves the following upper-bound \r\n\\[\r\n\\frac{1}{\\tau}\\le K_{\\tau}=\\frac{1}{\\gamma}\\cdot\\min_{\\hat{x},\\hat{y},\\hat{s}}\\{\\|\\hat{y}\\|+\\one_{\\K}^{T}\\hat{s}+\\one_{\\K}^{T}\\hat{x}+1:(\\hat{x},\\hat{y},\\hat{s})\\text{ solves (\\ref{eq:vecCTC}) with }\\hat{x}^{T}\\hat{s}=0\\}.\r\n\\]\r\nSetting $K=\\max\\{\\|r_{p}\\|K_{\\tau},\\|r_{d}\\|K_{\\tau},K_{\\tau}^{2}\\}$\r\nyields our desired result. \\qed\r\n\\end{proof}\n\n\\begin{proof}[Theorem~\\ref{thm:lintime}]Combining Lemma~\\ref{lem:eps}\r\nand Lemma~\\ref{lem:short-step} shows that the desired $\\epsilon$-accurate,\r\n$\\epsilon$-feasible iterate is obtained after $O(\\sqrt{\\nu}\\log(1/\\epsilon))$\r\ninterior-point iterations. At each iteration we perform the following\r\nsteps: 1) compute the scaling point $w$; 2) solve the normal equation\r\n(\\ref{eq:hsd_nrm}) for three right-hand sides; 3) back-substitute\r\n(\\ref{eq:hsd_back1})-(\\ref{eq:hsd_back2}) for the search direction\r\nand take the step in (\\ref{eq:hsd_ipm}). Note from the proof of Lemma~\\ref{lem:normal}\r\nthat the matrix $[\\A;\\N]$ has at most $O(\\omega^{2}n)$ rows under\r\nAssumption~\\ref{ass:lin}, and therefore $\\nnz(\\M)=O(\\omega^{4}n)$\r\nunder the hypothesis of Theorem~\\ref{thm:lintime}. Below, we show\r\nthat the cost of each step is bounded by $O(\\omega^{6}n)$ time and\r\n$O(\\omega^{4}n)$ memory.\r\n\r\n\\textbf{Scaling point.} We partition $x=[x_{0};x_{1};\\vector(X_{1});\\ldots;\\vector(X_{\\ell})]$\r\nand similarly for $s$. Then, the scaling point $w$ is given in closed-form~\\cite[Section~5]{sturm2002implementation}\r\n\\begin{gather*}\r\n\\begin{bmatrix}u_{0}\\\\\r\nu_{1}\r\n\\end{bmatrix}=2^{-3/4}\\begin{bmatrix}1 & 1\\\\\r\n-s_{1}/\\|s_{1}\\| & s_{1}/\\|s_{1}\\|\r\n\\end{bmatrix}\\begin{bmatrix}(s_{0}-\\|s_{1}\\|)^{1/2}\\\\\r\n(s_{0}+\\|s_{1}\\|)^{1/2}\r\n\\end{bmatrix},\\quad\\begin{bmatrix}v_{0}\\\\\r\nv_{1}\r\n\\end{bmatrix}=\\begin{bmatrix}u_{0} & u_{1}^{T}\\\\\r\nu_{1} & \\frac{1}{2}(u_{0}^{2}-u_{1}^{T}u_{1})I\r\n\\end{bmatrix}\\begin{bmatrix}x_{0}\\\\\r\nx_{1}\r\n\\end{bmatrix},\\\\\r\n\\begin{bmatrix}w_{0}\\\\\r\nw_{1}\r\n\\end{bmatrix}=2^{-1/4}\\begin{bmatrix}u_{0} & u_{1}^{T}\\\\\r\nu_{1} & \\frac{1}{2}(u_{0}^{2}-u_{1}^{T}u_{1})I\r\n\\end{bmatrix}\\begin{bmatrix}1 & 1\\\\\r\n-v_{1}/\\|v_{1}\\| & v_{1}/\\|v_{1}\\|\r\n\\end{bmatrix}\\begin{bmatrix}(v_{0}-\\|v_{1}\\|)^{-1/2}\\\\\r\n(v_{0}+\\|v_{1}\\|)^{-1/2}\r\n\\end{bmatrix},\\\\\r\nW_{j}=S_{j}^{1/2}(S_{j}^{1/2}X_{j}S_{j}^{1/2})^{-1/2}S_{j}^{1/2}\\qquad\\text{for all }j\\in\\{1,\\ldots,\\ell\\}.\r\n\\end{gather*}\r\nNoting that $\\nnz(w_{1})\\le O(\\omega^{2}n)$, $\\ell\\le n$ and each\r\n$W_{j}$ is at most $\\omega\\times\\omega$, the cost of forming $w=[w_{0};w_{1};\\vector(W_{1});\\ldots;\\vector(W_{\\ell})]$\r\nis at most $O(\\omega^{3}n)$ time and $O(\\omega^{2}n)$ memory. Also,\r\nsince\r\n\\[\r\n\\D=\\nabla^{2}F(w)=\\diag\\left(\\begin{bmatrix}w_{0} & w_{1}^{T}\\\\\r\nw_{1} & \\frac{1}{2}(w_{0}^{2}-w_{1}^{T}w_{1})I\r\n\\end{bmatrix},W_{1}\\otimes_{s}W_{1},\\ldots,W_{\\ell}\\otimes_{s}W_{\\ell}\\right)^{-1},\r\n\\]\r\nthe cost of each matrix-vector product with $\\D$ and $\\D^{-1}$ is\r\nalso $O(\\omega^{3}n)$ time and $O(\\omega^{2}n)$ memory.\r\n\r\n\\textbf{Normal equation. }The cost of matrix-vector products with\r\n$\\M$ and $\\M^{T}$ is $\\nnz(\\M)=O(\\omega^{4}n)$ time and memory.\r\nUsing Lemma~\\ref{lem:normal}, we form the right-hand sides and solve\r\nthe three normal equations in (\\ref{eq:hsd_nrm}) in $O(\\omega^{6}n)$\r\ntime and $O(\\omega^{4}n)$ memory.\r\n\r\n\\textbf{Back-substitution.} The cost of back substituting (\\ref{eq:hsd_back1})-(\\ref{eq:hsd_back2})\r\nand making the step (\\ref{eq:hsd_ipm}) is dominated by matrix-vector\r\nproducts with $\\D$, $\\D^{-1}$, $\\M$, and $\\M^{T}$ at $O(\\omega^{4}n)$\r\ntime and memory.\\qed \\end{proof}","units":[{"raw_tex":"\\begin{lemma}[Linear independence]\r\n\\label{lem:lin_indep}There exists $[u;v]\\ne0$ such that $\\A^{T}u+\\N^{T}v=0$\r\nif and only if there exists $y\\ne0$ such that $\\sum_{i}y_{i}A_{i}=0$.\r\n\\end{lemma}","start":40848,"end":41043},{"raw_tex":"\\begin{lemma}[Primal Slater]\r\n\\label{lem:slaterp}There exists $x\\in\\Int(\\K)$ satisfying $\\A x=b$\r\nand $\\N x=0$ if and only if there exists an $X\\succ0$ satisfying\r\n$A_{i}\\bullet X=b_{i}$ for all $i\\in\\{1,\\ldots,m\\}$.\r\n\\end{lemma}","start":41048,"end":41277},{"raw_tex":"\\begin{lemma}[Dual Slater]\r\n\\label{lem:slaterd}There exists $u,v$ satisfying $c-\\A^{T}u-\\N^{T}v\\in\\Int(\\K_{*})$\r\nif and only if there exists $y$ satisfying $C-\\sum_{i}y_{i}A_{i}\\succ0$.\r\n\\end{lemma}","start":41282,"end":41480},{"raw_tex":"\\begin{lemma}[Linear-time normal equation]\r\n\\label{lem:normal}Let there exist $v_{i}\\in V(T)$ for each $i\\in\\{1,\\ldots,m\\}$\r\nsuch that\r\n\\begin{equation}\r\nA_{i,j}\\ne0\\qquad\\implies\\qquad j=v_{i}\\text{ or }j=p(v_{i}).\\label{eq:Asparsity}\r\n\\end{equation}\r\nDefine $\\H$ and $\\q$ according to (\\ref{eq:Hqdef}). Then, under\r\nAssumption~\\ref{ass:lin}:\r\n\\begin{enumerate}\r\n\\item (Forming) It costs $O(\\omega^{6}n)$ time and $O(\\omega^{4}n)$ space\r\nto form $\\H$ and $\\q$, where $\\omega=\\max_{j}|J_{j}|=1+\\wid(\\T)$.\r\n\\item (Factoring) Let $\\pi$ be a topological ordering on $T$, and define\r\nthe associated block topological permutation $\\Pi$ as in Lemma~\\ref{lem:blocktree}.\r\nThen, it costs $O(\\omega^{6}n)$ time and $O(\\omega^{4}n)$ space\r\nto factor $\\Pi\\H\\Pi^{T}$ into $\\L\\L^{T}$. \r\n\\item (Solving) Given $r$, $\\q$, $\\Pi$, and the Cholesky factor $\\L$\r\nsatisfying $\\L\\L^{T}=\\Pi\\H\\Pi^{T}$, it costs $O(\\omega^{4}n)$ time\r\nand space to solve $(\\H+\\q\\q^{T})u=r$ for $u$.\r\n\\end{enumerate}\r\n\\end{lemma}","start":54512,"end":55500},{"raw_tex":"\\begin{proof}\r\nFor an instance of (\\ref{eq:CTC}), denote $\\ell=|\\mathcal{J}|\\le n$\r\nas its number of conic constraints, and $d=\\frac{1}{2}\\sum_{j=1}^{\\ell}|J_{j}|(|J_{j}|+1)\\le\\omega^{2}\\ell$\r\nas its total number of variables. Under linear independence (Assumption~\\ref{ass:lin}),\r\nthe constraint matrix $[\\A;\\N]$ associated with (\\ref{eq:CTC}) has\r\n$d$ columns and at most $d$ rows (Lemma~\\ref{lem:lin_indep}).\r\nWrite $\\xi_{i}^{T}$ as the $i$-th row of $[\\A;\\N]$, and assume\r\nwithout loss of generality that $[\\A;\\N]$ has exactly $d$ rows.\r\nObserve that $\\nnz(\\xi_{i})\\le\\omega(\\omega+1)$ by the definition\r\nof $\\N$ (\\ref{eq:Nmat}) and the hypothesis on $\\A$ via (\\ref{eq:Asparsity}),\r\nso $\\nnz([\\A;\\N])\\le2\\omega^{4}\\ell$. \r\n\r\n\\textbf{(i)} We form $\\H$ by setting $\\H\\gets\\D_{s}$ and then adding\r\n$\\H\\gets\\H+\\sigma\\xi_{i}\\xi_{i}^{T}$ one at a time, for $i\\in\\{1,2,\\ldots,d\\}$.\r\nThe first step forms $\\D_{s}=\\diag(\\D_{s}^{(1)},\\D_{s}^{(2)},\\ldots,\\D_{s}^{(\\ell)})$\r\nwhere $\\D_{s}^{(j)}=W_{j}\\otimes W_{j}$. Each $\\nnz(\\D_{s}^{(j)})=\\nnz(W_{j})^{2}=|J_{j}|^{2}(|J_{j}|+1)^{2}/4\\le\\omega^{4}$\r\nfor $j\\in\\{1,2,\\ldots,\\ell\\}$, so the total cost is $O(\\omega^{4}n)$\r\ntime and space. The second step adds $\\nnz(\\xi_{i}\\xi_{i}^{T})^{2}\\le\\omega^{2}(\\omega+1)^{2}$\r\nnonzeros per constraint over $d$ total constraints, for a total cost\r\nof $O(\\omega^{6}n)$ time and apparently $O(\\omega^{6}n)$ space.\r\nHowever, by the definition of $\\N$ (\\ref{eq:Nmat}) and the hypothesis\r\non $\\A$ via (\\ref{eq:Asparsity}), the $(j,k)$-th off-diagonal block\r\nof $\\xi_{i}\\xi_{i}^{T}$ is nonzero only if $(j,k)$ is an edge of\r\nthe tree $T$, as in\r\n\\[\r\n\\xi_{i}\\xi_{i}^{T}[J_{j},J_{k}]\\ne0\\qquad\\implies\\qquad(j,k)\\in E(T)\\qquad\\text{for all }j\\ne k.\r\n\\]\r\nHence, adding $\\mathbf{H}\\gets\\mathbf{H}+\\sigma\\xi_{i}\\xi_{i}^{T}$\r\none at a time results in at most $|V(T)|+|E(T)|$ dense blocks of\r\nat most $\\frac{1}{2}\\omega(\\omega+1)\\times\\frac{1}{2}\\omega(\\omega+1)$,\r\nfor a total memory cost of $O(\\omega^{4}n)$. \r\n\r\n\\textbf{(ii)} We form $\\q=[\\A^{T},\\N^{T}]w_{1}$ using a sparse matrix-vector\r\nproduct. Given that $\\nnz(w_{1})\\le d$ and $\\nnz([\\A;\\N])\\le2\\omega^{4}\\ell$,\r\nthis step costs $O(\\omega^{4}n)$ time and space.\r\n\r\n\\textbf{(iii)} We partition $\\H$ into $[\\H_{i,j}]_{i,j=1}^{\\ell}$\r\nto reveal a block sparsity pattern that coincides with the adjacency\r\nmatrix of $T$:\r\n\\begin{align*}\r\n\\H_{i,j} & =\\begin{cases}\r\n\\D_{s,i}+\\sigma\\sum_{k=1}^{\\ell}\\N_{k,i}^{T}\\N_{k,i}+\\sigma\\sum_{q=1}^{m}a_{q,i}a_{q,i}^{T} & i=j\\\\\r\n\\sigma\\sum_{k=1}^{\\ell}\\N_{k,i}^{T}\\N_{k,j}+\\sigma\\sum_{q=1}^{m}a_{q,i}a_{q,j}^{T} & (i,j)\\in E(T)\\\\\r\n0 & \\text{otherwise}\r\n\\end{cases}\r\n\\end{align*}\r\nwhere $a_{q,i}=\\vector(A_{q,i})$. According to Lemma~\\ref{lem:blocktree},\r\nthe permuted matrix $\\Pi\\H\\Pi^{T}$ factors into $\\L\\L^{T}$ with\r\nno block fill in $O(\\omega^{6}n)$ time and $O(\\omega^{4}n)$ space,\r\nbecause each block $\\H_{i,j}$ for $i,j\\in\\{1,2,\\ldots,\\ell\\}$ is\r\nat most order $\\frac{1}{2}\\omega(\\omega+1)$.\r\n\r\n\\textbf{(iv)} Using the rank-1 update formula (\\ref{eq:update}),\r\nthe cost of solving $(\\H+\\q\\q^{T})u=r$ is the same as the cost of\r\nsolving $\\H u=r$ for two right-hand sides, plus algebraic manipulations\r\nin $O(d)=O(\\omega^{2}n)$ time. Applying Lemma~\\ref{lem:blocktree}\r\nshows that the cost of solving $\\H u=r$ for each right-hand side\r\nis $O(\\omega^{4}n)$ time and space. \\qed\r\n\\end{proof}","start":55502,"end":58856},{"raw_tex":"\\begin{lemma}\r\n\\label{lem:row-rank}The matrix $\\N$ in (\\ref{eq:Nmat}) has full\r\nrow rank, that is $\\det(\\N\\N^{T})\\ne0$.\r\n\\end{lemma}","start":108146,"end":108278},{"raw_tex":"\\begin{proof}\r\nWe make $\\N=[\\N_{i,j}]_{i,j=1}^{\\ell}$ upper-triangular by ordering\r\nits blocks topologically on $T$: each nonempty block row $\\N_{j}$\r\ncontains a nonzero block at $\\N_{j,j}$ and a nonzero block at $\\N_{j,p(j)}$\r\nwhere the parent node $p(j)>j$ is ordered after $j$. Then, the claim\r\nfollows because each diagonal block $\\N_{j,j}$ implements a surjection\r\nand must therefore have full row-rank. \\qed\r\n\\end{proof}","start":108280,"end":108706},{"raw_tex":"\\begin{lemma}[Orthogonal complement]\r\n\\label{lem:orth}Let $d=\\frac{1}{2}\\sum_{j=1}^{\\ell}|J_{j}|(|J_{j}|+1)$.\r\nImplicitly define the $d\\times\\frac{1}{2}n(n+1)$ matrix $\\P$ to\r\nsatisfy\r\n\\begin{align*}\r\n\\P\\,\\vector(X) & =[\\vector(X[J_{1},J_{1}]);\\ldots;\\vector(X[J_{\\ell},J_{\\ell}])]\\qquad\\forall X\\in\\S^{n}.\r\n\\end{align*}\r\nThen, (i) $\\N\\P=0$; (ii) every $x\\in\\R^{d}$ can be decomposed as\r\n$x=\\N^{T}u+\\P v$. \r\n\\end{lemma}","start":108710,"end":109129},{"raw_tex":"\\begin{proof}\r\nFor each $x=[\\vector(X_{j})]_{j=1}^{\\ell}\\in\\R^{d}$, Theorem~\\ref{thm:ctree}\r\nsays that there exists a $Z$ satisfying $\\P\\,\\vector(Z)=x$ if and\r\nonly if $\\N x=0$. Equivalently, $x\\in\\mathrm{span}(\\P)$ if and only\r\nif $x\\perp\\mathrm{span}(\\N^{T})$. The ``only if'' part implies\r\n(i), while the ``if'' part implies (ii).\r\n\\end{proof}","start":109131,"end":109477},{"raw_tex":"\\begin{proof}[Lemma~\\ref{lem:lin_indep}]We will prove that\r\n\\[\r\n\\text{exists }[u;v]\\ne0,\\quad\\A^{T}u+\\N^{T}v=0\\qquad\\iff\\qquad\\text{exists }y\\ne0,\\quad\\sum_{i=1}^{m}y_{i}A_{i}=0.\r\n\\]\r\n\\textbf{($\\implies$)} We must have $u\\ne0$, because $\\N$ has full\r\nrow rank by Lemma~\\ref{lem:row-rank}, and so $\\A^{T}0+\\N^{T}v=0$\r\nif and only if $v=0$. Multiplying by $\\P$ yields $\\P^{T}(\\A^{T}u+\\N^{T}v)=\\M^{T}u+0=0$\r\nand so setting $y=u\\ne0$ yields $\\M^{T}y=0$.\\textbf{ ($\\impliedby$)\r\n}We use Lemma~\\ref{lem:orth} to decompose $\\A^{T}y=\\P z+\\N^{T}v$.\r\nIf $\\vector(\\sum_{i}y_{i}A_{i})=\\M^{T}y=\\P^{T}\\A^{T}y=0,$ then $\\P^{T}\\P z=\\P^{T}\\A^{T}y-\\P^{T}\\N^{T}v=0$\r\nand so $\\P z=0$. Setting $u=-y\\ne0$ yields $\\A^{T}u+\\N^{T}v=0$.\\qed \\end{proof}","start":110279,"end":111006},{"raw_tex":"\\begin{proof}[Lemma~\\ref{lem:slaterp}]We will prove that \r\n\\[\r\n\\text{exists }x\\in\\Int(\\K),\\quad\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}x=\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}\\qquad\\iff\\qquad\\text{exists }X\\succeq0,\\quad\\M\\,\\vector(X)=b.\r\n\\]\r\nDefine the chordal completion $F$ as in (\\ref{eq:chordal_compl}).\r\nObserve that $\\M\\,\\vector(X)=\\M\\,\\vector(Z)$ holds for all pairs\r\nof $P_{F}(X)=Z$, because each $A_{i}\\in\\S_{F}^{n}$ satisfies $A_{i}\\bullet X=A_{i}\\bullet P_{F}(X)$.\r\nAdditionally, the positive definite version of Theorem~\\ref{thm:prim_sep}\r\nis written\r\n\\begin{gather}\r\n\\text{exists }X\\succ0,\\quad P_{F}(X)=Z\\qquad\\iff\\qquad\\P\\,\\vector(Z)\\in\\Int(\\K).\\label{eq:prim_sep}\r\n\\end{gather}\r\nThis result was first established by Grone et al.~\\cite{grone1984positive};\r\na succinct proof can be found in~\\cite[Theorem~10.1]{vandenberghe2015chordal}.\\textbf{\r\n($\\implies$)} For every $x$ satisfying $\\N x=0$, there exists $Z$\r\nsuch that $\\P\\,\\vector(Z)=x$ due to Lemma~\\ref{lem:orth}. If additionally\r\n$x\\in\\Int(\\K)$, then there exists $X\\succ0$ satisfying $Z=P_{F}(X)$\r\ndue to (\\ref{eq:prim_sep}). We can verify that $\\M\\,\\vector(X)=\\M\\,\\vector(Z)=\\A\\P\\,\\vector(Z)=\\A x=b$.\\textbf{\r\n($\\impliedby$)} For every $X\\succ0,$ there exists $Z$ satisfying\r\n$Z=P_{F}(X)$ and $\\P\\,\\vector(Z)\\in\\Int(\\K)$ due to (\\ref{eq:prim_sep}).\r\nSet $u=\\P\\,\\vector(Z)$ and observe that $u\\in\\Int(\\K)$ and $\\N u=\\N\\P\\,\\vector(Z)=0$.\r\nIf additionally $\\M\\,\\vector(X)=b$, then $\\A u=\\A\\P\\,\\vector(Z)=\\M\\,\\vector(Z)=b$.\\qed \\end{proof}","start":111010,"end":112516},{"raw_tex":"\\begin{proof}[Lemma~\\ref{lem:slaterp}]We will prove that\r\n\\[\r\n\\text{exists }u,v,\\quad c-\\A^{T}u-\\N^{T}v\\in\\Int(\\K_{*})\\qquad\\iff\\qquad\\text{exists }y,\\quad C-\\sum_{i}y_{i}A_{i}\\succ0.\r\n\\]\r\nDefine the chordal completion $F$ as in (\\ref{eq:chordal_compl}).\r\nTheorem~\\ref{thm:prim_sep} in (\\ref{eq:prim_sep}) has a dual theorem\r\n\\begin{gather}\r\n\\text{exists }S\\succ0,\\quad S\\in\\S_{F}^{n}\\qquad\\iff\\qquad\\text{exists }h\\in\\Int(\\K_{*}),\\quad\\vector(S)=\\P^{T}h.\\label{eq:dual_sep}\r\n\\end{gather}\r\nThis result readily follows from the positive semidefinite version\r\nproved by Alger et al.~\\cite{agler1988positive}; see also~\\cite[Theorem~9.2]{vandenberghe2015chordal}.\\textbf{\r\n($\\implies$)} For each $h=c-\\A^{T}u-\\N^{T}v$, define $S=C-\\sum_{i}u_{i}A_{i}$\r\nand observe that \r\n\\[\r\n\\P^{T}h=\\P^{T}(c-\\A^{T}u-\\N^{T}v)=\\vector(C)-\\M^{T}u-0=\\vector(S).\r\n\\]\r\nIf additionally $h\\in\\K_{*}$, then $S\\succ0$ due to (\\ref{eq:dual_sep}).\\textbf{\r\n($\\impliedby$)} For each $S=C-\\sum_{i}y_{i}A_{i}\\succ0$, there exists\r\nan $h\\in\\Int(\\K_{*})$ satisfying $\\vector(S)=\\P^{T}h$ due to (\\ref{eq:dual_sep}).\r\nWe use Lemma~\\ref{lem:orth} to decompose $h=\\P u+\\N^{T}v$. Given\r\nthat $\\vector(S)=\\P^{T}h=\\P^{T}\\P u+0$, we must actually have $\\P u=c-\\A^{T}y$\r\nsince $\\P^{T}(c-\\A^{T}y)=\\vector(C)-\\M^{T}y=\\vector(S)$. Hence $h=c-\\A^{T}y+\\N^{T}v$\r\nand $h\\in\\Int(\\K_{*})$.\\qed \\end{proof}","start":112520,"end":113871},{"raw_tex":"\\begin{lemma}[$\\epsilon$-accurate and $\\epsilon$-feasible]\r\n\\label{lem:eps}If $(x,y,s,\\tau,\\theta,\\kappa)$ satisfies (\\ref{eq:hsdfeas})\r\nand (\\ref{eq:hsdcone}) and \r\n\\begin{align*}\r\n\\mu=\\frac{x^{T}s+\\tau\\kappa}{\\nu+1} & \\le\\epsilon, & \\tau\\kappa\\ge & \\gamma\\mu\r\n\\end{align*}\r\nfor constants $\\epsilon,\\gamma>0$, then the rescaled point $(x/\\tau,y/\\tau,s/\\tau)$\r\nsatisfies\r\n\\begin{align*}\r\n\\|\\M(x/\\tau)-\\b\\| & \\le K\\epsilon, & \\|\\M^{T}(y/\\tau)+(s/\\tau)-\\c\\| & \\le K\\epsilon, & \\frac{(x/\\tau)^{T}(s/\\tau)}{\\nu+1}\\le K\\epsilon\r\n\\end{align*}\r\nwhere $K$ is a constant.\r\n\\end{lemma}","start":117365,"end":117940},{"raw_tex":"\\begin{proof}\r\nNote that (\\ref{eq:hsdfeas}) implies $\\mu=\\theta$ and\r\n\\begin{align*}\r\n\\|\\M(x/\\tau)-\\b\\| & =\\frac{\\|r_{p}\\|\\mu}{\\tau}, & \\|\\M^{T}(y/\\tau)+(s/\\tau)-\\c\\| & =\\frac{\\|r_{d}\\|\\mu}{\\tau}, & \\frac{(x/\\tau)^{T}(s/\\tau)}{\\nu+1}=\\frac{\\mu}{\\tau^{2}}.\r\n\\end{align*}\r\nHence, we obtain our desired result by upper-bounding $1/\\tau$. Let\r\n$(x^{\\star},y^{\\star},s^{\\star},\\tau^{\\star},\\theta^{\\star},\\kappa^{\\star})$\r\nbe a solution of (\\ref{eq:hsd}), and note that for every $(x,y,s,\\tau,\\theta,\\kappa)$\r\nsatisfying (\\ref{eq:hsdfeas}) and (\\ref{eq:hsdcone}), we have the\r\nfollowing via the skew-symmetry of (\\ref{eq:hsdfeas})\r\n\\[\r\n(x-x^{\\star})^{T}(s-s^{\\star})+(\\tau-\\tau^{\\star})(\\kappa-\\kappa^{\\star})=0.\r\n\\]\r\nRearranging yields\r\n\\[\r\n(\\nu+1)\\mu=x^{T}s+\\tau\\kappa=(x^{\\star})^{T}s+x^{T}(s^{\\star})+\\tau\\kappa^{\\star}+\\tau^{\\star}\\kappa\\ge\\tau^{\\star}\\kappa\r\n\\]\r\nand hence\r\n\\[\r\n\\kappa\\tau^{\\star}\\le\\mu(\\nu+1),\\quad\\tau\\kappa\\ge\\gamma\\mu\\qquad\\implies\\qquad\\tau\\ge\\frac{\\gamma}{\\nu+1}\\tau^{\\star}.\r\n\\]\r\n\r\nIf (\\ref{eq:SDP}) satisfies the primal-dual Slater's condition, then\r\n(\\ref{eq:CTC}) also satisfies the primal-dual Slater's condition\r\n(Lemmas~\\ref{lem:slaterp} \\&~\\ref{lem:slaterd}). Therefore, the\r\nvectorized version (\\ref{eq:vecCTC}) of (\\ref{eq:CTC}) attains a\r\nsolution $(\\hat{x},\\hat{y},\\hat{s})$ with $\\hat{x}^{T}\\hat{s}=0$,\r\nand the following\r\n\\[\r\nx^{\\star}=\\tau^{\\star}\\begin{bmatrix}\\|\\hat{y}\\|\\\\\r\n\\hat{y}\\\\\r\n\\hat{s}\r\n\\end{bmatrix},\\quad y^{\\star}=\\tau^{\\star}\\hat{x},\\quad s^{\\star}=\\tau^{\\star}\\begin{bmatrix}0\\\\\r\n0\\\\\r\n\\hat{x}\r\n\\end{bmatrix},\\quad\\tau^{\\star}=\\frac{\\nu+1}{\\|\\hat{y}\\|+\\one_{\\K}^{T}\\hat{s}+\\one_{\\K}^{T}\\hat{x}+1},\r\n\\]\r\nwith $\\theta^{\\star}=\\kappa^{\\star}=0$ is a solution to (\\ref{eq:hsd}).\r\nThis proves the following upper-bound \r\n\\[\r\n\\frac{1}{\\tau}\\le K_{\\tau}=\\frac{1}{\\gamma}\\cdot\\min_{\\hat{x},\\hat{y},\\hat{s}}\\{\\|\\hat{y}\\|+\\one_{\\K}^{T}\\hat{s}+\\one_{\\K}^{T}\\hat{x}+1:(\\hat{x},\\hat{y},\\hat{s})\\text{ solves (\\ref{eq:vecCTC}) with }\\hat{x}^{T}\\hat{s}=0\\}.\r\n\\]\r\nSetting $K=\\max\\{\\|r_{p}\\|K_{\\tau},\\|r_{d}\\|K_{\\tau},K_{\\tau}^{2}\\}$\r\nyields our desired result. \\qed\r\n\\end{proof}","start":117942,"end":120056},{"raw_tex":"\\begin{proof}[Theorem~\\ref{thm:lintime}]Combining Lemma~\\ref{lem:eps}\r\nand Lemma~\\ref{lem:short-step} shows that the desired $\\epsilon$-accurate,\r\n$\\epsilon$-feasible iterate is obtained after $O(\\sqrt{\\nu}\\log(1/\\epsilon))$\r\ninterior-point iterations. At each iteration we perform the following\r\nsteps: 1) compute the scaling point $w$; 2) solve the normal equation\r\n(\\ref{eq:hsd_nrm}) for three right-hand sides; 3) back-substitute\r\n(\\ref{eq:hsd_back1})-(\\ref{eq:hsd_back2}) for the search direction\r\nand take the step in (\\ref{eq:hsd_ipm}). Note from the proof of Lemma~\\ref{lem:normal}\r\nthat the matrix $[\\A;\\N]$ has at most $O(\\omega^{2}n)$ rows under\r\nAssumption~\\ref{ass:lin}, and therefore $\\nnz(\\M)=O(\\omega^{4}n)$\r\nunder the hypothesis of Theorem~\\ref{thm:lintime}. Below, we show\r\nthat the cost of each step is bounded by $O(\\omega^{6}n)$ time and\r\n$O(\\omega^{4}n)$ memory.\r\n\r\n\\textbf{Scaling point.} We partition $x=[x_{0};x_{1};\\vector(X_{1});\\ldots;\\vector(X_{\\ell})]$\r\nand similarly for $s$. Then, the scaling point $w$ is given in closed-form~\\cite[Section~5]{sturm2002implementation}\r\n\\begin{gather*}\r\n\\begin{bmatrix}u_{0}\\\\\r\nu_{1}\r\n\\end{bmatrix}=2^{-3/4}\\begin{bmatrix}1 & 1\\\\\r\n-s_{1}/\\|s_{1}\\| & s_{1}/\\|s_{1}\\|\r\n\\end{bmatrix}\\begin{bmatrix}(s_{0}-\\|s_{1}\\|)^{1/2}\\\\\r\n(s_{0}+\\|s_{1}\\|)^{1/2}\r\n\\end{bmatrix},\\quad\\begin{bmatrix}v_{0}\\\\\r\nv_{1}\r\n\\end{bmatrix}=\\begin{bmatrix}u_{0} & u_{1}^{T}\\\\\r\nu_{1} & \\frac{1}{2}(u_{0}^{2}-u_{1}^{T}u_{1})I\r\n\\end{bmatrix}\\begin{bmatrix}x_{0}\\\\\r\nx_{1}\r\n\\end{bmatrix},\\\\\r\n\\begin{bmatrix}w_{0}\\\\\r\nw_{1}\r\n\\end{bmatrix}=2^{-1/4}\\begin{bmatrix}u_{0} & u_{1}^{T}\\\\\r\nu_{1} & \\frac{1}{2}(u_{0}^{2}-u_{1}^{T}u_{1})I\r\n\\end{bmatrix}\\begin{bmatrix}1 & 1\\\\\r\n-v_{1}/\\|v_{1}\\| & v_{1}/\\|v_{1}\\|\r\n\\end{bmatrix}\\begin{bmatrix}(v_{0}-\\|v_{1}\\|)^{-1/2}\\\\\r\n(v_{0}+\\|v_{1}\\|)^{-1/2}\r\n\\end{bmatrix},\\\\\r\nW_{j}=S_{j}^{1/2}(S_{j}^{1/2}X_{j}S_{j}^{1/2})^{-1/2}S_{j}^{1/2}\\qquad\\text{for all }j\\in\\{1,\\ldots,\\ell\\}.\r\n\\end{gather*}\r\nNoting that $\\nnz(w_{1})\\le O(\\omega^{2}n)$, $\\ell\\le n$ and each\r\n$W_{j}$ is at most $\\omega\\times\\omega$, the cost of forming $w=[w_{0};w_{1};\\vector(W_{1});\\ldots;\\vector(W_{\\ell})]$\r\nis at most $O(\\omega^{3}n)$ time and $O(\\omega^{2}n)$ memory. Also,\r\nsince\r\n\\[\r\n\\D=\\nabla^{2}F(w)=\\diag\\left(\\begin{bmatrix}w_{0} & w_{1}^{T}\\\\\r\nw_{1} & \\frac{1}{2}(w_{0}^{2}-w_{1}^{T}w_{1})I\r\n\\end{bmatrix},W_{1}\\otimes_{s}W_{1},\\ldots,W_{\\ell}\\otimes_{s}W_{\\ell}\\right)^{-1},\r\n\\]\r\nthe cost of each matrix-vector product with $\\D$ and $\\D^{-1}$ is\r\nalso $O(\\omega^{3}n)$ time and $O(\\omega^{2}n)$ memory.\r\n\r\n\\textbf{Normal equation. }The cost of matrix-vector products with\r\n$\\M$ and $\\M^{T}$ is $\\nnz(\\M)=O(\\omega^{4}n)$ time and memory.\r\nUsing Lemma~\\ref{lem:normal}, we form the right-hand sides and solve\r\nthe three normal equations in (\\ref{eq:hsd_nrm}) in $O(\\omega^{6}n)$\r\ntime and $O(\\omega^{4}n)$ memory.\r\n\r\n\\textbf{Back-substitution.} The cost of back substituting (\\ref{eq:hsd_back1})-(\\ref{eq:hsd_back2})\r\nand making the step (\\ref{eq:hsd_ipm}) is dominated by matrix-vector\r\nproducts with $\\D$, $\\D^{-1}$, $\\M$, and $\\M^{T}$ at $O(\\omega^{4}n)$\r\ntime and memory.\\qed \\end{proof}","start":124868,"end":128002}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:SDP"],"declared_external":false,"source_only":false,"title":"原始半定规划","source_units":[{"raw_tex":"Given $n\\times n$ real symmetric matrices $C,A_{1},\\ldots,A_{m}$\r\nand real scalars $b_{1},\\ldots,b_{m}$, we consider the standard form\r\nsemidefinite program \r\n\\begin{alignat}{2}\r\n & \\text{minimize } & C\\bullet X & \\tag{SDP}\\label{eq:SDP}\\\\\r\n & \\text{subject to } & A_{i}\\bullet X & =b_{i}\\quad\\text{for all }i\\in\\{1,\\ldots,m\\}\\nonumber \\\\\r\n &  & X & \\succeq0,\\nonumber \r\n\\end{alignat}\r\nover the $n\\times n$ real symmetric matrix variable $X$. Here, $A_{i}\\bullet X=\\tr A_{i}X$\r\nrefers to the usual matrix inner product, and $X\\succeq0$ restricts\r\nto be symmetric positive semidefinite. Instances of (\\ref{eq:SDP})\r\n","start":4930,"end":5545}]},{"id":"S002","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"凸松弛应用背景","source_units":[{"raw_tex":"arise as some of the best convex relaxations to nonconvex problems\r\nlike graph optimization~\\cite{lovasz1979shannon,goemans1995improved},\r\ninteger programming~\\cite{sherali1990hierarchy,lovasz1991cones,lasserre2001explicit,laurent2003comparison},\r\nand polynomial optimization~\\cite{lasserre2001global,parrilo2000structured}.\r\n","start":5545,"end":5871}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":["def:sparsitygraph"],"declared_external":false,"source_only":false,"title":"数据稀疏图","source_units":[{"raw_tex":"To guarantee an exact reformulation, clique tree conversion chooses\r\nthe index sets $J_{1},\\ldots,J_{\\ell}$ in (\\ref{eq:relax}) as the\r\n\\emph{bags} of a \\emph{tree decomposition} for the \\emph{sparsity\r\ngraph} of the data matrices $C,A_{1},\\ldots,A_{m}$. Accordingly,\r\nthe parameter $\\omega$ in (\\ref{eq:omega_def-1}) can only be small\r\nif the sparsity graph has a small \\emph{treewidth}. Below, we define\r\na graph $G$ by its vertex set $V(G)\\subseteq\\{1,2,\\ldots,n\\}$ and\r\nits edge set $E(G)\\subseteq V(G)\\times V(G)$.\r\n\\begin{definition}[Sparsity graph]\r\n\\label{def:sparsitygraph}The $n\\times n$ matrix $M$ (resp. the\r\nset of $n\\times n$ matrices $\\{M_{1},\\ldots,M_{m}\\}$) is said to\r\nhave \\emph{sparsity graph} $G$ if $G$ is an undirected simple graph\r\non $n$ vertices $V(G)=\\{1,\\ldots,n\\}$ and that $(i,j)\\in E(G)$\r\nif $M[i,j]\\ne0$ (resp. if there exists $M\\in\\{M_{1},\\ldots,M_{m}\\}$\r\nsuch that $M[i,j]\\ne0$). \r\n\\end{definition}\r\n%\r\n","start":17649,"end":18586}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["def:treedecomp"],"declared_external":false,"source_only":false,"title":"树分解与宽度","source_units":[{"raw_tex":"\\begin{definition}[Tree decomposition]\r\n\\label{def:treedecomp}A \\emph{tree decomposition} $\\T$ of a graph\r\n$G$ is a pair $(\\J,T)$, where each \\emph{bag} of vertices $J_{j}\\in\\J$\r\nis a subset of $V(G)$, and $T$ is a tree on $|\\J|\\le n$ vertices,\r\nsuch that:\r\n\\begin{enumerate}\r\n\\item (Vertex cover) For every $v\\in V(G)$, there exists $J_{k}\\in\\J$\r\nsuch that $v\\in J_{k}$;\r\n\\item (Edge cover) For every $(u,v)\\in E(G)$, there exists $J_{k}\\in\\J$\r\nsuch that $u\\in J_{k}$ and $v\\in J_{k}$; and\r\n\\item (Running intersection) If $v\\in J_{i}$ and $v\\in J_{j}$, then we\r\nalso have $v\\in J_{k}$ for every $k$ that lies on the path from\r\n$i$ to $j$ in the tree $T$.\r\n\\end{enumerate}\r\nThe \\emph{width} $\\wid(\\T)$ of the tree decomposition $\\T=(\\J,T)$\r\nis the size of its largest bag minus one, as in $\\max\\{|J_{k}|:J_{k}\\in\\J\\}-1.$\r\nThe \\emph{treewidth} $\\tw(G)$ of the graph $G$ is the minimum width\r\namongst all tree decompositions $\\T$.\r\n\\end{definition}\r\n","start":18586,"end":19535}]},{"id":"S005","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"预先给定树分解","source_units":[{"raw_tex":"Throughout this paper, we make the implicit assumption that a tree\r\ndecomposition with small width is known a priori for the sparsity\r\ngraph. In practice, such a tree decomposition can usually be found\r\nusing fill-reducing heuristics for sparse linear algebra; see Section~\\ref{sec:Preliminaries}.\r\n\r\nWe also make two explicit assumptions, which are standard in the literature\r\non interior-point methods.\r\n\r\n","start":19535,"end":19943}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","labels":["ass:lin"],"declared_external":false,"source_only":false,"title":"约束线性无关","source_units":[{"raw_tex":"\\begin{assume}[Linear independence]\\label{ass:lin}We have $\\sum_{i=1}^{m}y_{i}A_{i}=0$\r\nif and only if $y=0$.\r\n\r\n\\end{assume}\r\n","start":19943,"end":20070}]},{"id":"S007","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"原始约束数上界","source_units":[{"raw_tex":"\r\nThe assumption is without loss of generality, because it can either\r\nbe enforced by eliminating $A_{i}\\bullet X=b_{i}$ for select $i$,\r\nor else these constraints are not consistent for all $i$. Under Assumption~\\ref{ass:lin},\r\nthe total number of constraints is bounded $m\\le\\omega n$ (due to\r\nthe fact that $|E(G)|\\le n\\cdot\\tw(G)$~\\cite{arnborg1987complexity}).\r\n\r\n","start":20070,"end":20439}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","labels":["ass:slater"],"declared_external":false,"source_only":false,"title":"原始双侧严格可行","source_units":[{"raw_tex":"\\begin{assume}[Primal-dual Slater's condition]\\label{ass:slater}There\r\nexist $X\\succ0,$ $y,$ and $S\\succ0$ satisfying $A_{i}\\bullet X=b_{i}$\r\nfor all $i\\in\\{1,\\ldots,m\\}$ and $\\sum_{i=1}^{m}y_{i}A_{i}+S=C$. \r\n\r\n\\end{assume}\r\n","start":20439,"end":20664}]},{"id":"S009","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"矩阵与范数记号","source_units":[{"raw_tex":"The sets $\\R^{n}$ and $\\R^{m\\times n}$ are the length-$n$ real\r\nvectors and $m\\times n$ real matrices. We use ``MATLAB notation''\r\nin concatenating vectors and matrices:\r\n\r\n\\[\r\n[a,b]=\\begin{bmatrix}a & b\\end{bmatrix},\\qquad[a;b]=\\begin{bmatrix}a\\\\\r\nb\r\n\\end{bmatrix},\\qquad\\diag(a,b)=\\begin{bmatrix}a & 0\\\\\r\n0 & b\r\n\\end{bmatrix},\r\n\\]\r\nand the following short-hand to construct them:\r\n\\[\r\n[x_{i}]_{i=1}^{n}=\\begin{bmatrix}x_{1}\\\\\r\n\\vdots\\\\\r\nx_{n}\r\n\\end{bmatrix},\\qquad[x_{i,j}]_{i,j=1}^{m,n}=\\begin{bmatrix}x_{1,1} & \\cdots & x_{1,n}\\\\\r\n\\vdots & \\ddots & \\vdots\\\\\r\nx_{m,1} & \\cdots & x_{m,n}\r\n\\end{bmatrix}.\r\n\\]\r\nThe notation $X[i,j]$ refers to the element of $X$ in the $i$-th\r\nrow and $j$-th column, and $X[I,J]$ refers to the submatrix of $X$\r\nformed from the rows in $I\\subseteq\\{1,\\ldots,m\\}$ and columns in\r\n$J\\subseteq\\{1,\\ldots,n\\}$. The Frobenius inner product is $X\\bullet Y=\\tr(X^{T}Y)$,\r\nand the Frobenius norm is $\\|X\\|_{F}=\\sqrt{X\\bullet X}$. We use $\\nnz(X)$\r\nto denote the number of nonzero elements in $X$. \r\n\r\n","start":30594,"end":31620}]},{"id":"S010","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"对称锥与向量化","source_units":[{"raw_tex":"The sets $\\S^{n}\\subseteq\\R^{n\\times n},$ $\\S_{+}^{n}\\subset\\S^{n},$\r\nand $\\S_{++}^{n}\\subset\\S_{+}^{n}$ are the $n\\times n$ real symmetric\r\nmatrices, positive semidefinite matrices, and positive definite matrices,\r\nrespective. We write $X\\succeq Y$ to mean $X-Y\\in\\S_{+}^{n}$ and\r\n$X\\succ Y$ to mean $X-Y\\in\\S_{++}^{n}$. The (symmetric) vectorization\r\n\\[\r\n\\vector(X)=[X[1,1];\\sqrt{2}X[2,1];\\ldots;\\sqrt{2}X[m,1];X[2,2],\\ldots]\r\n\\]\r\noutputs the lower-triangular part of a symmetric matrix as a vector,\r\nwith factors of $\\sqrt{2}$ added so that $\\vector(X)^{T}\\vector(Y)=X\\bullet Y$. \r\n\r\n","start":31620,"end":32207}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"根树与父指针","source_units":[{"raw_tex":"A graph $G$ is defined by its vertex set $V(G)\\subseteq\\{1,2,3,\\ldots\\}$\r\nand its edge set $E(G)\\subseteq V(G)\\times V(G)$. The graph $T$\r\nis a \\emph{tree} if it is connected and does not contain any cycles;\r\nwe refer to its vertices $V(T)$ as its \\emph{nodes}. Designating\r\na special node $r\\in V(T)$ as the \\emph{root }of the tree allows\r\nus to define the \\emph{parent} $p(v)$ of each node $v\\ne r$ as the\r\nfirst node encountered on the path from $v$ to $r$, and $p(r)=r$\r\nfor consistency. The set of children is defined $\\ch(v)=\\{u\\in V(T)\\backslash v:p(u)=v\\}$.\r\nNote that the edges $E(T)$ are fully determined by the parent pointer\r\n$p$ as $\\{v,p(v)\\}$ for all $v\\ne r$.\r\n\r\n","start":32207,"end":32886}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"稀疏矩阵投影","source_units":[{"raw_tex":"The set $\\S_{G}^{n}\\subseteq\\S^{n}$ is the set of $n\\times n$ real\r\nsymmetric matrices with sparsity graph $G$. We denote $P_{G}(X)=\\min_{Y\\in\\S_{G}^{n}}\\|X-Y\\|_{F}$\r\nas the Euclidean projection of $X\\in\\S^{n}$ onto $\\S_{G}^{n}$.\r\n","start":32886,"end":33117}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["eq:chordal_compl"],"declared_external":false,"source_only":false,"title":"由袋构造弦图","source_units":[{"raw_tex":"Let $\\T=(\\{J_{1},\\ldots,J_{\\ell}\\},T)$ be a tree decomposition with\r\nsmall width for the sparsity graph $G$ of the data matrices $C,A_{1},\\ldots,A_{m}$.\r\nWe define the graph $F\\supseteq G$ by taking each index set $J_{j}$\r\nof $\\T$ and interconnecting all pairs of vertices $u,v\\in J_{j}$,\r\nas in \r\n\\begin{align}\r\nV(F) & =V(G), & E(F) & =\\bigcup_{j=1}^{\\ell}\\{(u,v):u,v\\in J_{j}\\}.\\label{eq:chordal_compl}\r\n\\end{align}\r\nThe following fundamental result was first established by Grone et\r\nal.~\\cite{grone1984positive}. Constructive proofs allow us to recover\r\nall elements in $X\\succeq0$ from only the elements in $P_{F}(X)$\r\nusing a closed-form formula.\r\n","start":35830,"end":36484}]},{"id":"S014","semantic_type":"external_fact","kind":"external_fact","labels":["thm:prim_sep"],"declared_external":true,"declaration_text":["原始前提列表允许的 B0045；Grone et al., grone1984positive。"],"source_only":false,"title":"半正定补全定理","source_units":[{"raw_tex":"\\begin{theorem}[Grone et al.~\\cite{grone1984positive}]\r\n\\label{thm:prim_sep}Given $Z\\in\\S_{F}^{n}$, there exists an $X\\succeq0$\r\nsatisfying $P_{F}(X)=Z$ if and only if $Z[J_{j},J_{j}]\\succeq0$\r\nfor all $j\\in\\{1,2,\\ldots,\\ell\\}$.\r\n\\end{theorem}\r\n","start":36484,"end":36729}]},{"id":"S015","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:CSDP"],"declared_external":false,"source_only":false,"title":"投影后的等价问题","source_units":[{"raw_tex":"We can use Theorem~\\ref{thm:prim_sep} to reformulate (\\ref{eq:SDP})\r\ninto a reduced-complexity form. The key is to view (\\ref{eq:SDP})\r\nas an optimization over $P_{F}(X)$, since \r\n\\[\r\nC\\bullet X=\\sum_{i,j=1}^{n}C_{i,j}X_{i,j}=\\sum_{(i,j)\\in F}C_{i,j}X_{i,j}=C\\bullet P_{F}(X),\r\n\\]\r\nand similarly $A_{i}\\bullet X=A_{i}\\bullet P_{F}(X)$. Theorem~\\ref{thm:prim_sep}\r\nallows us to account for $X\\succeq0$ implicitly, by optimizing over\r\n$Z=P_{F}(X)$ in the following\r\n\\begin{alignat}{3}\r\n & \\text{minimize }\\quad & C\\bullet Z\\label{eq:CSDP}\\\\\r\n & \\text{subject to } & A_{i}\\bullet Z & =b_{i} & \\qquad\\text{for all } & i\\in\\{1,\\ldots,m\\},\\nonumber \\\\\r\n &  & Z[J_{j},J_{j}] & \\succeq0 & \\text{for all } & j\\in\\{1,\\ldots,\\ell\\}.\\nonumber \r\n\\end{alignat}\r\n","start":36729,"end":37477}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"重叠子矩阵算子","source_units":[{"raw_tex":"Next, we split the principal submatrices into distinct matrix variables,\r\ncoupled by the need for their overlapping elements to agree. Define\r\nthe \\emph{overlap} operator $\\NN_{i,j}(\\cdot)$ to output the overlapping\r\nelements of two principal submatrices given the latter as input:\r\n\\[\r\n\\NN_{i,j}(X[J_{j},J_{j}])=X[J_{i}\\cap J_{j},\\;J_{i}\\cap J_{j}]=\\NN_{j,i}(X[J_{i},J_{i}]).\r\n\\]\r\nThe running intersection property of the tree decomposition allows\r\nus to enforce this agreement using $\\ell-1$ pairwise block comparisons.\r\n","start":37477,"end":38000}]},{"id":"S017","semantic_type":"external_fact","kind":"external_fact","labels":["thm:ctree"],"declared_external":true,"declaration_text":["原始前提列表允许的 B0049；Fukuda et al., fukuda2001exploiting。"],"source_only":false,"title":"树边重叠一致性","source_units":[{"raw_tex":"\\begin{theorem}[Fukuda et al.~\\cite{fukuda2001exploiting}]\r\n\\label{thm:ctree}Given $X_{1},X_{2},\\ldots,X_{\\ell}$ for $X_{j}\\in\\S^{|J_{j}|}$,\r\nthere exists $Z$ satisfying $Z[J_{j},J_{j}]=X_{j}$ for all $j\\in\\{1,2,\\ldots,\\ell\\}$\r\nif and only if $\\NN_{i,j}(X_{j})=\\NN_{j,i}(X_{i})$ for all $(i,j)\\in E(T)$. \r\n\\end{theorem}\r\n","start":38000,"end":38321}]},{"id":"S018","semantic_type":"assumption","kind":"assumption","labels":["eq:split_constr"],"declared_external":false,"source_only":false,"title":"数据拆分恒等式","source_units":[{"raw_tex":"Splitting the objective $C$ and constraint matrices $A_{1},\\ldots,A_{m}$\r\ninto $C_{1},\\ldots,C_{\\ell}$ and $A_{1,1},\\ldots,A_{m,\\ell}$ to\r\nsatisfy \r\n\\begin{align}\r\nC_{1}\\bullet X[J_{1},J_{1}]+C_{2}\\bullet X[J_{2},J_{2}]+\\cdots+C_{\\ell}\\bullet X[J_{\\ell},J_{\\ell}] & =C\\bullet X,\\label{eq:split_constr}\\\\\r\nA_{i,1}\\bullet X[J_{1},J_{1}]+A_{i,2}\\bullet X[J_{2},J_{2}]+\\cdots+A_{i,\\ell}\\bullet X[J_{\\ell},J_{\\ell}] & =A_{i}\\bullet X,\\nonumber \r\n\\end{align}\r\n","start":38321,"end":38775}]},{"id":"S019","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:CTC"],"declared_external":false,"source_only":false,"title":"团树转换问题","source_units":[{"raw_tex":"and applying Theorem~\\ref{thm:ctree} yields the following\r\n\\begin{alignat}{3}\r\n & \\text{minimize }\\quad & \\sum_{j=1}^{\\ell}C_{j}\\bullet X_{j} &  &  & \\tag{CTC}\\label{eq:CTC}\\\\\r\n & \\text{subject to } & \\sum_{j=1}^{\\ell}A_{i,j}\\bullet X_{j} & =b_{i} & \\text{for all } & i\\in\\{1,\\ldots,m\\},\\nonumber \\\\\r\n &  & \\NN_{i,j}(X_{j}) & =\\NN_{j,i}(X_{i}) & \\qquad\\text{for all } & (i,j)\\in E(T),\\nonumber \\\\\r\n &  & X_{j} & \\succeq0 & \\text{for all } & j\\in\\{1,\\ldots,\\ell\\},\\nonumber \r\n\\end{alignat}\r\n","start":38775,"end":39265}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":["eq:vecCTC"],"declared_external":false,"source_only":false,"title":"向量化原对偶与乘积锥","source_units":[{"raw_tex":"which vectorizes into a linear conic program in standard form\r\n\\begin{alignat}{4}\r\n & \\text{minimize } & c^{T}x, &  & \\qquad\\qquad & \\text{maximize } & \\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}^{T}y,\\label{eq:vecCTC}\\\\\r\n & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}x & =\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}, &  & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}y+s & =c,\\nonumber \\\\\r\n &  & x & \\in\\K, &  &  & s & \\in\\K_{*}\\nonumber \r\n\\end{alignat}\r\nover the Cartesian product of $\\ell\\le n$ smaller semidefinite cones\r\n\\begin{equation}\r\n\\K=\\K_{*}=\\S_{+}^{|J_{1}|}\\times\\S_{+}^{|J_{2}|}\\times\\cdots\\times\\S_{+}^{|J_{\\ell}|}.\r\n\\end{equation}\r\n","start":39265,"end":39933}]},{"id":"S021","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"拆分数据坐标","source_units":[{"raw_tex":"Here, $\\A=[\\vector(A_{i,j})^{T}]_{i,j=1}^{m,\\ell}$ and $c=[\\vector(C_{j})]_{j=1}^{\\ell}$\r\ncorrespond to (\\ref{eq:split_constr}), and the \\emph{overlap constraints}\r\nmatrix $\\N=[\\N_{i,j}]_{i,j=1}^{\\ell,\\ell}$ is implicitly defined\r\nby the relation\r\n","start":39933,"end":40181}]},{"id":"S022","semantic_type":"definition","kind":"definition","labels":["eq:Nmat"],"declared_external":false,"source_only":false,"title":"重叠矩阵的树边结构","source_units":[{"raw_tex":"\\begin{equation}\r\n\\N_{i,j}\\vector(X_{j})=\\begin{cases}\r\n+\\vector(\\NN_{p(i),i}(X_{i})) & j=i,\\\\\r\n-\\vector(\\NN_{i,p(i)}(X_{p(i)})) & j=p(i),\\\\\r\n0 & \\text{otherwise},\r\n\\end{cases}\\label{eq:Nmat}\r\n\\end{equation}\r\nfor every non-root node $i$ on $T$. (To avoid all-zero rows in $\\N$,\r\nwe define $\\N_{i,j}\\,\\vector(X_{j})$ as the empty length-zero vector\r\n$\\R^{0}$ if $i$ is the root node.)\r\n","start":40181,"end":40566}]},{"id":"S023","semantic_type":"assumption","kind":"assumption","labels":["eq:scal"],"declared_external":false,"source_only":false,"title":"块对角正定缩放","source_units":[{"raw_tex":"where the \\emph{scaling matrix} $\\D_{s}$ is block-diagonal with fully-dense\r\nblocks \r\n\\begin{equation}\r\n\\D_{s}=\\diag(\\D_{s,1},\\ldots,\\D_{s,\\ell}),\\qquad\\D_{s,j}\\succ0\\quad\\text{for all }j\\in\\{1,\\ldots,\\ell\\}.\\label{eq:scal}\r\n\\end{equation}\r\n","start":43294,"end":43535}]},{"id":"S024","semantic_type":"definition","kind":"definition","labels":["def:topological"],"declared_external":false,"source_only":false,"title":"子先父后的拓扑序","source_units":[{"raw_tex":"\\begin{definition}[Topological ordering]\r\n\\label{def:topological}An ordering\\emph{ }$\\pi:\\{1,2,\\ldots,n\\}\\to V(T)$\r\non the tree graph $T$ with $n$ nodes is said to be \\emph{topological}~\\cite[p.~10]{vandenberghe2015chordal}\r\nif, by designating $\\pi(n)$ as the root of $T$, each node is indexed\r\nbefore its parent:\r\n\\[\r\n\\pi^{-1}(v)<\\pi^{-1}(p(v))\\qquad\\text{ for all }v\\ne r,\r\n\\]\r\nwhere $\\pi^{-1}(v)$ denotes the index associated with the node $v$.\r\n\\end{definition}\r\n","start":46596,"end":47063}]},{"id":"S025","semantic_type":"external_fact","kind":"external_fact","labels":["lem:blocktree"],"declared_external":true,"declaration_text":["原始前提列表允许的 B0068；Parter, parter1961use；George–Liu, george1981computer, Lemma 6.3.1。"],"source_only":false,"title":"树块消元无填充","source_units":[{"raw_tex":"\\begin{lemma}[No block fill]\r\n\\label{lem:blocktree}Let $J_{1},\\ldots,J_{n}$ satisfy $\\bigcup_{j=1}^{n}J_{j}=\\{1,\\ldots,d\\}$\r\nand $J_{i}\\cap J_{j}=\\emptyset$ for all $i\\ne j$, and let $H\\succ0$\r\nbe a $d\\times d$ matrix satisfying\r\n\\[\r\nH[J_{i},J_{j}]\\ne0\\qquad\\implies\\qquad(i,j)\\in E(T)\r\n\\]\r\nfor a tree graph $T$ on $n$ nodes. If $\\pi$ is a topological ordering\r\non $T$ and $\\Pi$ is a permutation matrix satisfying \r\n\\[\r\n(\\Pi H\\Pi)[J_{i},J_{j}]=H[J_{\\pi(i)},J_{\\pi(j)}]\\qquad\\text{for all }i,j\\in\\{1,\\ldots,n\\},\r\n\\]\r\nthen $\\Pi H\\Pi^{T}$ factors into $LL^{T}$ where the Cholesky factor\r\n$L$ satisfies\r\n\\[\r\nL[J_{i},J_{j}]\\ne0\\qquad\\implies\\qquad(\\Pi H\\Pi)[J_{i},J_{j}]\\ne0\\qquad\\text{ for all }i>j.\r\n\\]\r\nTherefore, sparse Cholesky factorization solves $Hx=b$ for $x$ by:\r\n(i) factoring $\\Pi H\\Pi^{T}$ into $LL^{T}$ in $O(\\beta^{3}n)$ operations\r\nand $O(\\beta^{2}n)$ memory where $\\beta=\\max_{j}|J_{j}|$, and (ii)\r\nsolving $Ly=\\Pi b$ and $L^{T}z=y$ and $x=\\Pi^{T}z$ in $O(\\beta^{2}n)$\r\noperations and memory. \r\n\\end{lemma}\r\nThis is a simple block-wise extension of the tree elimination result\r\noriginally due to Parter~\\cite{parter1961use}; see also George and\r\nLiu~\\cite[Lemma 6.3.1]{george1981computer}. In practice, a topological\r\n","start":47063,"end":48293}]},{"id":"S026","semantic_type":"algorithm","kind":"algorithm","labels":[],"declared_external":false,"source_only":false,"title":"拓扑序的实际构造","source_units":[{"raw_tex":"ordering can be found by assigning indices $n,n-1,n-2,\\ldots$ in\r\ndecreasing ordering during a depth-first search traversal of the tree.\r\nIn fact, the minimum degree heuristic is guaranteed to generate a\r\ntopological ordering~\\cite{george1989evolution}.\r\n","start":48293,"end":48548}]},{"id":"S027","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:dualCTC"],"declared_external":false,"source_only":false,"title":"交换原对偶角色","source_units":[{"raw_tex":"The dualization technique of Löfberg~\\cite{lofberg2009dualize} swaps\r\nthe roles played by the primal and the dual problems in a linear conic\r\nprogram, by rewriting a primal standard form problem into dual standard\r\nform, and vice versa. Applying dualization to (\\ref{eq:vecCTC}) yields\r\nthe following\r\n\\begin{alignat}{4}\r\n & \\text{minimize } & \\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}^{T}x_{1} &  & \\qquad & \\text{maximize } & -c^{T}y\\label{eq:dualCTC}\\\\\r\n & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}x_{1}-x_{2} & =-c, &  & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}y+s_{1} & =\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix},\\nonumber \\\\\r\n &  & x_{1}\\in\\R^{f},\\,x_{2} & \\in\\K. &  &  & -y+s_{2} & =0,\\nonumber \\\\\r\n &  &  &  &  &  & s_{1}\\in\\{0\\}^{f},\\,s_{2} & \\in\\K.\\nonumber \r\n\\end{alignat}\r\nwhere we use $f$ to denote the number of equality constraints in\r\n(\\ref{eq:CTC}). Observe that the dual problem in (\\ref{eq:dualCTC})\r\nis identical to the primal problem in (\\ref{eq:vecCTC}), so that\r\na dual solution $y^{\\star}$ to (\\ref{eq:dualCTC}) immediately serves\r\nas a primal solution to (\\ref{eq:vecCTC}), and hence also (\\ref{eq:CTC}).\r\n\r\n","start":50272,"end":51436}]},{"id":"S028","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:dualCTC-2"],"declared_external":false,"source_only":false,"title":"自由变量的二阶锥嵌入","source_units":[{"raw_tex":"Modern interior-point methods solve (\\ref{eq:dualCTC}) by embeding\r\nthe free variable $x_{1}\\in\\R^{f}$ and fixed variable $s_{1}\\in\\{0\\}^{f}$\r\ninto a second-order cone (see Sturm~\\cite{sturm1999using} and Andersen~\\cite{andersen2002handling}):\r\n\\begin{alignat}{4}\r\n & \\text{minimize } & \\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}^{T}x_{1} &  & \\qquad & \\text{maximize } & -c^{T}y\\label{eq:dualCTC-2}\\\\\r\n & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}x_{1}-x_{2} & =-c, &  & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}y+s_{1} & =\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix},\\nonumber \\\\\r\n &  & \\|x_{1}\\|\\le x_{0},\\,x_{2} & \\in\\K. &  &  & -y+s_{2} & =0,\\nonumber \\\\\r\n &  &  &  &  &  & s_{0} & =0,\\nonumber \\\\\r\n &  &  &  &  &  & \\|s_{1}\\|\\le s_{0},\\,s_{2} & \\in\\K.\\nonumber \r\n\\end{alignat}\r\n","start":51436,"end":52245}]},{"id":"S029","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:Hxr_dual","eq:scal_f"],"declared_external":false,"source_only":false,"title":"对偶正常方程结构","source_units":[{"raw_tex":"When (\\ref{eq:dualCTC-2}) is solved using an interior-point method,\r\nthe normal equation solved at each iteration takes the form\r\n\\begin{equation}\r\n\\left(\\D_{s}+\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}\\D_{f}\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}\\right)\\Delta y=r\\label{eq:Hxr_dual}\r\n\\end{equation}\r\nwhere $\\D_{s}$ is comparable as before in (\\ref{eq:scal}), and \r\n\\begin{equation}\r\n\\D_{f}=\\sigma I+ww^{T},\\qquad\\sigma>0\\label{eq:scal_f}\r\n\\end{equation}\r\n","start":52245,"end":52701}]},{"id":"S030","semantic_type":"definition","kind":"definition","labels":["eq:Hqdef"],"declared_external":false,"source_only":false,"title":"稀疏基矩阵与秩一向量","source_units":[{"raw_tex":"is the rank-1 perturbation of a scaled identity matrix. The standard\r\nprocedure, as implemented in SeDuMi~\\cite{sturm1999using,sturm2002implementation}\r\nand MOSEK~\\cite{andersen2003implementing}, is to form the sparse\r\nmatrix $\\H$ and dense vector $\\q$, defined\r\n\\begin{align}\r\n\\H & =\\D_{s}+\\sigma\\A^{T}\\A+\\sigma\\N^{T}\\N, & \\q & =\\begin{bmatrix}\\A^{T} & \\N^{T}\\end{bmatrix}w.\\label{eq:Hqdef}\r\n\\end{align}\r\n","start":52701,"end":53107}]},{"id":"S031","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"秩一更新的实现比较","source_units":[{"raw_tex":"and then solve (\\ref{eq:Hxr_dual}) using a rank-1 update\\footnote{To keep our derivations simple, we perform the rank-1 update using\r\nthe Sherman\\textendash Morrison\\textemdash Woodbury (SMW) formula.\r\nIn practice, the product-form Cholesky Factor (PFCF) formula of Goldfarb\r\nand Scheinberg~\\cite{goldfarb2005product} is more numerically stable\r\nand more widely used~\\cite{sturm1999using,sturm2002implementation}.\r\nOur complexity results remain valid in either cases because the PFCF\r\nis a constant factor of approximately two times slower than the SWM~\\cite{goldfarb2005product}.}\r\n","start":53107,"end":53690}]},{"id":"S032","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:update"],"declared_external":false,"source_only":false,"title":"两次求解完成秩一更新","source_units":[{"raw_tex":"\\begin{equation}\r\n\\Delta y=(\\H+\\q\\q^{T})^{-1}r=\\left(I-\\frac{(\\H^{-1}\\q)\\q^{T}}{1+\\q^{T}(\\H^{-1}\\q)}\\right)\\H^{-1}r,\\label{eq:update}\r\n\\end{equation}\r\nat a cost comparable to the solution of $\\H u=r$ for two right-hand\r\nsides. (In Appendix~\\ref{sec:inequality}, we repeat these derivations\r\nfor the version of (\\ref{eq:vecCTC}) in which $\\A x=b$ is replaced\r\nby the inequality $\\A x\\le b$.)\r\n","start":53690,"end":54082}]},{"id":"S033","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"原始约束矩阵化","source_units":[{"raw_tex":"Define the $m\\times\\frac{1}{2}n(n+1)$ matrix $\\M$ as the vectorization\r\nof the linear constraints in (\\ref{eq:SDP}), as in\r\n\\[\r\n\\M\\,\\vector(X)=\\begin{bmatrix}\\vector(A_{1})^{T}\\\\\r\n\\vdots\\\\\r\n\\vector(A_{m})^{T}\r\n\\end{bmatrix}\\vector(X)=\\begin{bmatrix}\\vector(A_{1})^{T}\\vector(X)\\\\\r\n\\vdots\\\\\r\n\\vector(A_{m})^{T}\\vector(X)\r\n\\end{bmatrix}=\\begin{bmatrix}A_{1}\\bullet X\\\\\r\n\\vdots\\\\\r\nA_{m}\\bullet X\r\n\\end{bmatrix}.\r\n\\]\r\n","start":109481,"end":109895}]},{"id":"S034","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:split_constr2"],"declared_external":false,"source_only":false,"title":"拆分的算子恒等式","source_units":[{"raw_tex":"In reformulating (\\ref{eq:SDP}) into (\\ref{eq:CTC}), the splitting\r\nconditions (\\ref{eq:split_constr}) can be rewritten as the following\r\n\\begin{align}\r\nc^{T}\\P & =\\vector(C)^{T}, & \\A\\P & =\\M,\\label{eq:split_constr2}\r\n\\end{align}\r\nwhere $c=[\\vector(C_{j})]_{j=1}^{\\ell}$ and $\\A=[\\vector(A_{i,j})^{T}]_{i,j=1}^{m,\\ell}$\r\nare the data for the vectorized verison of (\\ref{eq:CTC}).\r\n","start":109895,"end":110277}]},{"id":"S035","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:hsdcone","eq:hsdfeas"],"declared_external":false,"source_only":false,"title":"扩展齐次自对偶嵌入","source_units":[{"raw_tex":"We solve the dualized problem (\\ref{eq:dualCTC-2}) by solving its\r\nextended homogeneous self-dual embedding\\begin{subequations}\\label{eq:hsd}\r\n\\begin{alignat}{2}\r\n & \\text{min.}\\quad & (\\nu+1)\\theta\\\\\r\n & \\text{s.t.} & \\begin{bmatrix}0 & +\\M^{T} & -\\c & -r_{d}\\\\\r\n-\\M & 0 & +\\b & -r_{p}\\\\\r\n+\\c^{T} & -\\b^{T} & 0 & -r_{c}\\\\\r\nr_{d}^{T} & r_{p}^{T} & r_{c} & 0\r\n\\end{bmatrix}\\begin{bmatrix}x\\\\\r\ny\\\\\r\n\\tau\\\\\r\n\\theta\r\n\\end{bmatrix}+\\begin{bmatrix}s\\\\\r\n0\\\\\r\n\\kappa\\\\\r\n0\r\n\\end{bmatrix} & =\\begin{bmatrix}0\\\\\r\n0\\\\\r\n0\\\\\r\n\\nu+1\r\n\\end{bmatrix}\\label{eq:hsdfeas}\\\\\r\n &  & x,s\\in\\C\\quad\\kappa,\\tau & \\ge0,\\label{eq:hsdcone}\r\n\\end{alignat}\r\nwhere the data is given in standard form\r\n\\begin{gather}\r\n\\M=\\begin{bmatrix}0 & -\\begin{bmatrix}\\A^{T} & \\N^{T}\\end{bmatrix} & +I\\end{bmatrix},\\quad\\c^{T}=\\begin{bmatrix}0 & \\begin{bmatrix}b^{T} & 0\\end{bmatrix} & 0\\end{bmatrix},\\quad\\b=-c,\\\\\r\n\\C=\\{(x_{0},x_{1}\\}\\in\\R^{p+1}:\\|x_{1}\\|\\le x_{0}\\}\\times\\K,\r\n\\end{gather}\r\n","start":115490,"end":116437}]},{"id":"S036","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"嵌入残差与锥阶","source_units":[{"raw_tex":"and the residual vectors are defined\r\n\\begin{equation}\r\nr_{d}=\\one_{\\C}-\\c,\\quad r_{p}=-\\M\\one_{\\C}+\\b,\\quad r_{c}=1+\\c^{T}\\one_{\\C}.\r\n\\end{equation}\r\n\\end{subequations}Here, $\\nu$ is the order of the cone $\\C$, and\r\n$\\one_{\\C}$ is its identity element\r\n\\begin{align}\r\n\\nu & =1+|J_{1}|+\\cdots+|J_{\\ell}|, & \\one_{\\C} & =[1;0;\\ldots;0;\\vector(I_{|J_{1}|});\\ldots;\\vector(I_{|J_{\\ell}|})].\r\n\\end{align}\r\n","start":116437,"end":116839}]},{"id":"S037","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"嵌入解的重缩放解释","source_units":[{"raw_tex":"Problem (\\ref{eq:hsd}) has optimal value $\\theta^{\\star}=0$. Under\r\nthe primal-dual Slater's conditions (Assumption~\\ref{ass:slater}),\r\nan interior-point method is guaranteed to converge to an $\\epsilon$-accurate\r\nsolution with $\\tau>0$, and this yields an $\\epsilon$-feasible and\r\n$\\epsilon$-accurate solution to the dualized problem (\\ref{eq:dualCTC-2})\r\nby rescaling $x/\\tau$ and $y=y/\\tau$ and $s=s/\\tau$. The following\r\nresult is adapted from~\\cite[Lemma~5.7.2]{de2000self} and \\cite[Theorem~22.7]{vanderbei2015linear}.\r\n","start":116839,"end":117365}]},{"id":"S038","semantic_type":"algorithm","kind":"algorithm","labels":["eq:hsdinit"],"declared_external":false,"source_only":false,"title":"短步初始化与迭代","source_units":[{"raw_tex":"We solve the homogeneous self-dual embedding (\\ref{eq:hsd}) using\r\nthe short-step method of Nesterov and Todd~\\cite[Algorithm~6.1]{nesterov1998primal}\r\n(and also Sturm and S.~Zhang~\\cite[Section~5.1]{sturm1999symmetric}),\r\nnoting that SeDuMi reduces to it in the worst case; see~\\cite{sturm2002implementation}\r\nand~\\cite{sturm1997wide}. Beginning at the following strictly feasible,\r\nperfectly centered point\r\n\\begin{gather}\r\n\\theta^{(0)}=\\tau^{(0)}=\\kappa^{(0)}=1,\\quad y^{(0)}=0,\\quad x^{(0)}=s^{(0)}=\\one_{\\C},\\label{eq:hsdinit}\r\n\\end{gather}\r\nwith barrier parameter $\\mu=1$, we take the following steps\\begin{subequations}\\label{eq:hsd_ipm}\r\n\\begin{align}\r\n\\mu^{+} & =\\left(1-\\frac{1}{15\\sqrt{\\nu+1}}\\right)\\cdot\\frac{x^{T}s+\\tau\\kappa}{\\nu+1},\\\\\r\n(x^{+},y^{+},s^{+},\\tau^{+},\\theta^{+},\\kappa^{+}) & =(x,y,s,\\tau,\\theta,\\kappa)+(\\Delta x,\\Delta y,\\Delta s,\\Delta\\tau,\\Delta\\theta,\\Delta\\kappa).\\nonumber \r\n\\end{align}\r\n\\end{subequations}along the search direction defined by the linear\r\n","start":120060,"end":121052}]},{"id":"S039","semantic_type":"algorithm","kind":"algorithm","labels":[],"declared_external":false,"source_only":false,"title":"缩放 Newton 搜索方向","source_units":[{"raw_tex":"system~\\cite[Eqn. 9]{todd1998nesterov} \\begin{subequations}\\label{eq:hsd_nt}\r\n\\begin{align}\r\n\\begin{bmatrix}0 & +\\M^{T} & -\\c & -r_{p}\\\\\r\n-\\M & 0 & +\\b & -r_{d}\\\\\r\n+\\c^{T} & -\\b^{T} & 0 & -r_{c}\\\\\r\n+r_{p}^{T} & +r_{d}^{T} & +r_{c} & 0\r\n\\end{bmatrix}\\begin{bmatrix}\\Delta x\\\\\r\n\\Delta y\\\\\r\n\\Delta\\tau\\\\\r\n\\Delta\\theta\r\n\\end{bmatrix}+\\begin{bmatrix}\\Delta s\\\\\r\n0\\\\\r\n\\Delta\\kappa\\\\\r\n0\r\n\\end{bmatrix} & =0,\\\\\r\ns+\\Delta s+\\nabla^{2}F(w)\\Delta x+\\mu^{+}\\nabla F(x) & =0,\\\\\r\n\\kappa+\\Delta\\kappa+(\\kappa/\\tau)\\Delta\\tau-\\mu^{+}\\tau^{-1} & =0.\r\n\\end{align}\r\n","start":121052,"end":121599}]},{"id":"S040","semantic_type":"definition","kind":"definition","labels":["eq:Fdef"],"declared_external":false,"source_only":false,"title":"障碍函数与缩放点","source_units":[{"raw_tex":"\\end{subequations}Here, $F$ is the usual self-concordant barrier\r\nfunction on $\\C$\r\n\\begin{align}\r\nF([x_{0};x_{1};\\vector(X_{1});\\ldots;\\vector(X_{\\ell})])= & -\\log\\left(\\frac{1}{2}x_{0}^{2}-\\frac{1}{2}x_{1}^{T}x_{1}\\right)-\\sum_{j=1}^{\\ell}\\log\\det(X_{j})\\label{eq:Fdef}\r\n\\end{align}\r\nand $w\\in\\Int(\\C)$ is the unique \\emph{scaling point} satisfying\r\n$\\nabla^{2}F(w)x=s$, which can be computed from $x$ and $s$ in\r\nclosed-form. The following iteration bound is an immediate consequence\r\nof~\\cite[Theorem~6.4]{nesterov1998primal}; see also~\\cite[Theorem~5.1]{sturm1999symmetric}.\r\n","start":121599,"end":122180}]},{"id":"S041","semantic_type":"external_fact","kind":"external_fact","labels":["lem:short-step"],"declared_external":true,"declaration_text":["原始前提列表允许的 B0177；作者在 2283–2284 行归于 Nesterov–Todd, nesterov1998primal, Theorem 6.4，及 Sturm–Zhang, sturm1999symmetric, Theorem 5.1。"],"source_only":false,"title":"短步迭代次数界","source_units":[{"raw_tex":"\\begin{lemma}[Short-Step Method]\r\n\\label{lem:short-step}The sequence in (\\ref{eq:hsd_ipm}) arrives\r\nat an iterate $(x,y,s,\\tau,\\theta,\\kappa)$ satisfying the conditions\r\nof Lemma~\\ref{lem:eps} with $\\gamma=9/10$ in at most $O(\\sqrt{\\nu}\\log(1/\\epsilon))$\r\niterations. \r\n\\end{lemma}\r\n","start":122180,"end":122463}]},{"id":"S042","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:hsd_nrm"],"declared_external":false,"source_only":false,"title":"三个右端的正常方程","source_units":[{"raw_tex":"The cost of each interior-point iteration is dominated by the cost\r\nof computing the search direction in (\\ref{eq:hsd_nt}). Using elementary\r\nbut tedious linear algebra, we can show that if\\begin{subequations}\\label{eq:newt}\r\n\\begin{equation}\r\n(\\M\\D^{-1}\\M^{T})\\begin{bmatrix}v_{1} & v_{2} & v_{3}\\end{bmatrix}=\\begin{bmatrix}0 & -\\b & r_{p}\\end{bmatrix}-\\M\\D^{-1}\\begin{bmatrix}d & \\c & r_{d}\\end{bmatrix}\\label{eq:hsd_nrm}\r\n\\end{equation}\r\nwhere $\\D=\\nabla^{2}F(w)$ and $d=-s-\\mu^{+}\\nabla F(x)$, and \r\n","start":122463,"end":122968}]},{"id":"S043","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:hsd_back1","eq:hsd_back2"],"declared_external":false,"source_only":false,"title":"搜索方向的回代公式","source_units":[{"raw_tex":"\\begin{equation}\r\n\\begin{bmatrix}u_{1} & u_{2} & u_{3}\\end{bmatrix}=\\D^{-1}(\\begin{bmatrix}d & c & r_{d}\\end{bmatrix}+\\M^{T}\\begin{bmatrix}v_{1} & v_{2} & v_{3}\\end{bmatrix}),\\label{eq:hsd_back1}\r\n\\end{equation}\r\nthen\r\n\\begin{align}\r\n\\left(\\begin{bmatrix}-\\D_{0} & -r_{c}\\\\\r\nr_{c} & 0\r\n\\end{bmatrix}-\\begin{bmatrix}\\c & r_{d}\\\\\r\n-\\b & r_{p}\r\n\\end{bmatrix}^{T}\\begin{bmatrix}u_{2} & u_{3}\\\\\r\nv_{2} & v_{3}\r\n\\end{bmatrix}\\right)\\begin{bmatrix}\\Delta\\tau\\\\\r\n\\Delta\\theta\r\n\\end{bmatrix} & =\\begin{bmatrix}-d_{0}\\\\\r\n0\r\n\\end{bmatrix}-\\begin{bmatrix}\\c & r_{d}\\\\\r\n-\\b & r_{p}\r\n\\end{bmatrix}^{T}\\begin{bmatrix}u_{1}\\\\\r\nv_{1}\r\n\\end{bmatrix},\\\\\r\n\\begin{bmatrix}\\Delta x\\\\\r\n\\Delta y\r\n\\end{bmatrix} & =\\begin{bmatrix}u_{1}\\\\\r\nv_{1}\r\n\\end{bmatrix}-\\begin{bmatrix}u_{1} & u_{2}\\\\\r\nv_{1} & v_{2}\r\n\\end{bmatrix}\\begin{bmatrix}\\Delta\\tau\\\\\r\n\\Delta\\theta\r\n\\end{bmatrix},\\\\\r\n\\Delta s & =d-\\D\\Delta x,\\\\\r\n\\Delta\\kappa & =d_{0}-\\D_{0}\\Delta\\tau,\\label{eq:hsd_back2}\r\n\\end{align}\r\n\\end{subequations}where $\\D_{0}=\\kappa/\\tau$ and $d_{0}=-\\kappa+\\mu^{+}\\tau^{-1}$.\r\n","start":122968,"end":124011}]},{"id":"S044","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"嵌入正常矩阵的稀疏分解","source_units":[{"raw_tex":"Hence, the cost of computing the search direction is dominated by\r\nthe cost of solving the normal equation for three different right-hand\r\nsides. Here, the normal matrix is written\r\n\r\n\\[\r\n\\M\\D^{-1}\\M^{T}=\\diag(W_{1}\\otimes_{s}W_{1},\\ldots,W_{\\ell}\\otimes_{s}W_{\\ell})+\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}(w_{1}w_{1}^{T}+\\sigma I)\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix},\r\n\\]\r\nwhere $\\sigma=\\frac{1}{2}(w_{0}^{2}-w_{1}^{T}w_{1})>0$ and $\\otimes_{s}$\r\ndenotes the symmetric Kronecker product~\\cite{alizadeh1998primal}\r\n","start":124011,"end":124532}]},{"id":"S045","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"对称 Kronecker 乘积","source_units":[{"raw_tex":"implicitly defined to satisfy\r\n\\[\r\n(A\\otimes_{s}B)\\vector(X)=\\frac{1}{2}\\vector(AXB^{T}+BXA^{T})\\qquad\\text{for all }X=X^{T}.\r\n\\]\r\n","start":124532,"end":124663}]},{"id":"S046","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"正常方程引理的实例化","source_units":[{"raw_tex":"Under the hypothesis on $\\A$ stated in Theorem~\\ref{thm:lintime},\r\nthe normal matrix satisfies the assumptions of Lemma~\\ref{lem:normal},\r\nand can therefore be solved in linear $O(n)$ time and memory. \r\n","start":124663,"end":124866}]},{"id":"S047","semantic_type":"theorem","kind":"theorem","labels":["eq:eps_accurate","eq:structA","thm:lintime"],"declared_external":false,"source_only":false,"title":"近线性时间主定理","source_units":[{"raw_tex":"\\begin{theorem}[Near-linear time]\r\n\\label{thm:lintime}Let $\\T=(\\{J_{1},\\ldots,J_{\\ell}\\},T)$ be a\r\ntree decomposition for the sparsity graph of $C,A_{1},\\ldots,A_{m}\\in\\S^{n}$.\r\nIn the corresponding instance of (\\ref{eq:CTC}), let each constraint\r\nbe written\r\n\\begin{equation}\r\n\\sum_{j=1}^{\\ell}A_{i,j}\\bullet X_{j}=A_{i,j}\\bullet X_{j}+A_{i,k}\\bullet X_{k}=b_{i}\\qquad(j,k)\\in E(T).\\label{eq:structA}\r\n\\end{equation}\r\nUnder Assumptions~\\ref{ass:lin} \\&~\\ref{ass:slater}, there exists\r\nan algorithm that computes an iterate $(x,y,s)\\in\\K\\times\\R^{p}\\times\\K_{*}$\r\nsatisfying\r\n\\begin{align}\r\n\\left\\Vert \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}x-\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}\\right\\Vert  & \\le\\epsilon, & \\left\\Vert \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}y-s+c\\right\\Vert  & \\le\\epsilon, & \\frac{x^{T}s}{\\sum_{j=1}^{\\ell}|J_{j}|} & \\le\\epsilon\\label{eq:eps_accurate}\r\n\\end{align}\r\nin $O(\\omega^{6.5}n^{1.5}\\log(1/\\epsilon))$ time and $O(\\omega^{4}n)$\r\nspace, where $\\omega=\\max_{j}|J_{j}|=1+\\wid(\\T)$.\r\n\\end{theorem}","start":59053,"end":60075}]},{"id":"S048","semantic_type":"assumption","kind":"assumption","labels":["eq:structA","thm:lintime"],"declared_external":false,"source_only":false,"title":"主定理的局部结构条件","source_units":[{"raw_tex":"\\label{thm:lintime}Let $\\T=(\\{J_{1},\\ldots,J_{\\ell}\\},T)$ be a\r\ntree decomposition for the sparsity graph of $C,A_{1},\\ldots,A_{m}\\in\\S^{n}$.\r\nIn the corresponding instance of (\\ref{eq:CTC}), let each constraint\r\nbe written\r\n\\begin{equation}\r\n\\sum_{j=1}^{\\ell}A_{i,j}\\bullet X_{j}=A_{i,j}\\bullet X_{j}+A_{i,k}\\bullet X_{k}=b_{i}\\qquad(j,k)\\in E(T).\\label{eq:structA}\r\n\\end{equation}\r\nUnder Assumptions~\\ref{ass:lin} \\&~\\ref{ass:slater}, there exists\r\n","start":59088,"end":59539}]},{"id":"S049","semantic_type":"lemma","kind":"lemma","labels":["lem:lin_indep"],"declared_external":false,"source_only":false,"title":"线性无关的传递","source_units":[{"raw_tex":"\\begin{lemma}[Linear independence]\r\n\\label{lem:lin_indep}There exists $[u;v]\\ne0$ such that $\\A^{T}u+\\N^{T}v=0$\r\nif and only if there exists $y\\ne0$ such that $\\sum_{i}y_{i}A_{i}=0$.\r\n\\end{lemma}","start":40848,"end":41043}]},{"id":"S050","semantic_type":"lemma","kind":"lemma","labels":["lem:slaterp"],"declared_external":false,"source_only":false,"title":"原始严格可行的传递","source_units":[{"raw_tex":"\\begin{lemma}[Primal Slater]\r\n\\label{lem:slaterp}There exists $x\\in\\Int(\\K)$ satisfying $\\A x=b$\r\nand $\\N x=0$ if and only if there exists an $X\\succ0$ satisfying\r\n$A_{i}\\bullet X=b_{i}$ for all $i\\in\\{1,\\ldots,m\\}$.\r\n\\end{lemma}","start":41048,"end":41277}]},{"id":"S051","semantic_type":"lemma","kind":"lemma","labels":["lem:slaterd"],"declared_external":false,"source_only":false,"title":"对偶严格可行的传递","source_units":[{"raw_tex":"\\begin{lemma}[Dual Slater]\r\n\\label{lem:slaterd}There exists $u,v$ satisfying $c-\\A^{T}u-\\N^{T}v\\in\\Int(\\K_{*})$\r\nif and only if there exists $y$ satisfying $C-\\sum_{i}y_{i}A_{i}\\succ0$.\r\n\\end{lemma}","start":41282,"end":41480}]},{"id":"S052","semantic_type":"lemma","kind":"lemma","labels":["eq:Asparsity","lem:normal"],"declared_external":false,"source_only":false,"title":"线性时间正常方程","source_units":[{"raw_tex":"\\begin{lemma}[Linear-time normal equation]\r\n\\label{lem:normal}Let there exist $v_{i}\\in V(T)$ for each $i\\in\\{1,\\ldots,m\\}$\r\nsuch that\r\n\\begin{equation}\r\nA_{i,j}\\ne0\\qquad\\implies\\qquad j=v_{i}\\text{ or }j=p(v_{i}).\\label{eq:Asparsity}\r\n\\end{equation}\r\nDefine $\\H$ and $\\q$ according to (\\ref{eq:Hqdef}). Then, under\r\nAssumption~\\ref{ass:lin}:\r\n\\begin{enumerate}\r\n\\item (Forming) It costs $O(\\omega^{6}n)$ time and $O(\\omega^{4}n)$ space\r\nto form $\\H$ and $\\q$, where $\\omega=\\max_{j}|J_{j}|=1+\\wid(\\T)$.\r\n\\item (Factoring) Let $\\pi$ be a topological ordering on $T$, and define\r\nthe associated block topological permutation $\\Pi$ as in Lemma~\\ref{lem:blocktree}.\r\nThen, it costs $O(\\omega^{6}n)$ time and $O(\\omega^{4}n)$ space\r\nto factor $\\Pi\\H\\Pi^{T}$ into $\\L\\L^{T}$. \r\n\\item (Solving) Given $r$, $\\q$, $\\Pi$, and the Cholesky factor $\\L$\r\nsatisfying $\\L\\L^{T}=\\Pi\\H\\Pi^{T}$, it costs $O(\\omega^{4}n)$ time\r\nand space to solve $(\\H+\\q\\q^{T})u=r$ for $u$.\r\n\\end{enumerate}\r\n\\end{lemma}","start":54512,"end":55500}]},{"id":"S053","semantic_type":"assumption","kind":"assumption","labels":["eq:Asparsity","lem:normal"],"declared_external":false,"source_only":false,"title":"父子袋支撑假设","source_units":[{"raw_tex":"\\label{lem:normal}Let there exist $v_{i}\\in V(T)$ for each $i\\in\\{1,\\ldots,m\\}$\r\nsuch that\r\n\\begin{equation}\r\nA_{i,j}\\ne0\\qquad\\implies\\qquad j=v_{i}\\text{ or }j=p(v_{i}).\\label{eq:Asparsity}\r\n\\end{equation}\r\n","start":54556,"end":54765}]},{"id":"S054","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"正常方程引理完整证明","source_units":[{"raw_tex":"\\begin{proof}\r\nFor an instance of (\\ref{eq:CTC}), denote $\\ell=|\\mathcal{J}|\\le n$\r\nas its number of conic constraints, and $d=\\frac{1}{2}\\sum_{j=1}^{\\ell}|J_{j}|(|J_{j}|+1)\\le\\omega^{2}\\ell$\r\nas its total number of variables. Under linear independence (Assumption~\\ref{ass:lin}),\r\nthe constraint matrix $[\\A;\\N]$ associated with (\\ref{eq:CTC}) has\r\n$d$ columns and at most $d$ rows (Lemma~\\ref{lem:lin_indep}).\r\nWrite $\\xi_{i}^{T}$ as the $i$-th row of $[\\A;\\N]$, and assume\r\nwithout loss of generality that $[\\A;\\N]$ has exactly $d$ rows.\r\nObserve that $\\nnz(\\xi_{i})\\le\\omega(\\omega+1)$ by the definition\r\nof $\\N$ (\\ref{eq:Nmat}) and the hypothesis on $\\A$ via (\\ref{eq:Asparsity}),\r\nso $\\nnz([\\A;\\N])\\le2\\omega^{4}\\ell$. \r\n\r\n\\textbf{(i)} We form $\\H$ by setting $\\H\\gets\\D_{s}$ and then adding\r\n$\\H\\gets\\H+\\sigma\\xi_{i}\\xi_{i}^{T}$ one at a time, for $i\\in\\{1,2,\\ldots,d\\}$.\r\nThe first step forms $\\D_{s}=\\diag(\\D_{s}^{(1)},\\D_{s}^{(2)},\\ldots,\\D_{s}^{(\\ell)})$\r\nwhere $\\D_{s}^{(j)}=W_{j}\\otimes W_{j}$. Each $\\nnz(\\D_{s}^{(j)})=\\nnz(W_{j})^{2}=|J_{j}|^{2}(|J_{j}|+1)^{2}/4\\le\\omega^{4}$\r\nfor $j\\in\\{1,2,\\ldots,\\ell\\}$, so the total cost is $O(\\omega^{4}n)$\r\ntime and space. The second step adds $\\nnz(\\xi_{i}\\xi_{i}^{T})^{2}\\le\\omega^{2}(\\omega+1)^{2}$\r\nnonzeros per constraint over $d$ total constraints, for a total cost\r\nof $O(\\omega^{6}n)$ time and apparently $O(\\omega^{6}n)$ space.\r\nHowever, by the definition of $\\N$ (\\ref{eq:Nmat}) and the hypothesis\r\non $\\A$ via (\\ref{eq:Asparsity}), the $(j,k)$-th off-diagonal block\r\nof $\\xi_{i}\\xi_{i}^{T}$ is nonzero only if $(j,k)$ is an edge of\r\nthe tree $T$, as in\r\n\\[\r\n\\xi_{i}\\xi_{i}^{T}[J_{j},J_{k}]\\ne0\\qquad\\implies\\qquad(j,k)\\in E(T)\\qquad\\text{for all }j\\ne k.\r\n\\]\r\nHence, adding $\\mathbf{H}\\gets\\mathbf{H}+\\sigma\\xi_{i}\\xi_{i}^{T}$\r\none at a time results in at most $|V(T)|+|E(T)|$ dense blocks of\r\nat most $\\frac{1}{2}\\omega(\\omega+1)\\times\\frac{1}{2}\\omega(\\omega+1)$,\r\nfor a total memory cost of $O(\\omega^{4}n)$. \r\n\r\n\\textbf{(ii)} We form $\\q=[\\A^{T},\\N^{T}]w_{1}$ using a sparse matrix-vector\r\nproduct. Given that $\\nnz(w_{1})\\le d$ and $\\nnz([\\A;\\N])\\le2\\omega^{4}\\ell$,\r\nthis step costs $O(\\omega^{4}n)$ time and space.\r\n\r\n\\textbf{(iii)} We partition $\\H$ into $[\\H_{i,j}]_{i,j=1}^{\\ell}$\r\nto reveal a block sparsity pattern that coincides with the adjacency\r\nmatrix of $T$:\r\n\\begin{align*}\r\n\\H_{i,j} & =\\begin{cases}\r\n\\D_{s,i}+\\sigma\\sum_{k=1}^{\\ell}\\N_{k,i}^{T}\\N_{k,i}+\\sigma\\sum_{q=1}^{m}a_{q,i}a_{q,i}^{T} & i=j\\\\\r\n\\sigma\\sum_{k=1}^{\\ell}\\N_{k,i}^{T}\\N_{k,j}+\\sigma\\sum_{q=1}^{m}a_{q,i}a_{q,j}^{T} & (i,j)\\in E(T)\\\\\r\n0 & \\text{otherwise}\r\n\\end{cases}\r\n\\end{align*}\r\nwhere $a_{q,i}=\\vector(A_{q,i})$. According to Lemma~\\ref{lem:blocktree},\r\nthe permuted matrix $\\Pi\\H\\Pi^{T}$ factors into $\\L\\L^{T}$ with\r\nno block fill in $O(\\omega^{6}n)$ time and $O(\\omega^{4}n)$ space,\r\nbecause each block $\\H_{i,j}$ for $i,j\\in\\{1,2,\\ldots,\\ell\\}$ is\r\nat most order $\\frac{1}{2}\\omega(\\omega+1)$.\r\n\r\n\\textbf{(iv)} Using the rank-1 update formula (\\ref{eq:update}),\r\nthe cost of solving $(\\H+\\q\\q^{T})u=r$ is the same as the cost of\r\nsolving $\\H u=r$ for two right-hand sides, plus algebraic manipulations\r\nin $O(d)=O(\\omega^{2}n)$ time. Applying Lemma~\\ref{lem:blocktree}\r\nshows that the cost of solving $\\H u=r$ for each right-hand side\r\nis $O(\\omega^{4}n)$ time and space. \\qed\r\n\\end{proof}","start":55502,"end":58856}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"维数与行稀疏度","source_units":[{"raw_tex":"\\begin{proof}\r\nFor an instance of (\\ref{eq:CTC}), denote $\\ell=|\\mathcal{J}|\\le n$\r\nas its number of conic constraints, and $d=\\frac{1}{2}\\sum_{j=1}^{\\ell}|J_{j}|(|J_{j}|+1)\\le\\omega^{2}\\ell$\r\nas its total number of variables. Under linear independence (Assumption~\\ref{ass:lin}),\r\nthe constraint matrix $[\\A;\\N]$ associated with (\\ref{eq:CTC}) has\r\n$d$ columns and at most $d$ rows (Lemma~\\ref{lem:lin_indep}).\r\nWrite $\\xi_{i}^{T}$ as the $i$-th row of $[\\A;\\N]$, and assume\r\nwithout loss of generality that $[\\A;\\N]$ has exactly $d$ rows.\r\nObserve that $\\nnz(\\xi_{i})\\le\\omega(\\omega+1)$ by the definition\r\nof $\\N$ (\\ref{eq:Nmat}) and the hypothesis on $\\A$ via (\\ref{eq:Asparsity}),\r\nso $\\nnz([\\A;\\N])\\le2\\omega^{4}\\ell$. \r\n","start":55502,"end":56229}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"形成矩阵并控制存储","source_units":[{"raw_tex":"\\textbf{(i)} We form $\\H$ by setting $\\H\\gets\\D_{s}$ and then adding\r\n$\\H\\gets\\H+\\sigma\\xi_{i}\\xi_{i}^{T}$ one at a time, for $i\\in\\{1,2,\\ldots,d\\}$.\r\nThe first step forms $\\D_{s}=\\diag(\\D_{s}^{(1)},\\D_{s}^{(2)},\\ldots,\\D_{s}^{(\\ell)})$\r\nwhere $\\D_{s}^{(j)}=W_{j}\\otimes W_{j}$. Each $\\nnz(\\D_{s}^{(j)})=\\nnz(W_{j})^{2}=|J_{j}|^{2}(|J_{j}|+1)^{2}/4\\le\\omega^{4}$\r\nfor $j\\in\\{1,2,\\ldots,\\ell\\}$, so the total cost is $O(\\omega^{4}n)$\r\ntime and space. The second step adds $\\nnz(\\xi_{i}\\xi_{i}^{T})^{2}\\le\\omega^{2}(\\omega+1)^{2}$\r\nnonzeros per constraint over $d$ total constraints, for a total cost\r\nof $O(\\omega^{6}n)$ time and apparently $O(\\omega^{6}n)$ space.\r\nHowever, by the definition of $\\N$ (\\ref{eq:Nmat}) and the hypothesis\r\non $\\A$ via (\\ref{eq:Asparsity}), the $(j,k)$-th off-diagonal block\r\nof $\\xi_{i}\\xi_{i}^{T}$ is nonzero only if $(j,k)$ is an edge of\r\nthe tree $T$, as in\r\n\\[\r\n\\xi_{i}\\xi_{i}^{T}[J_{j},J_{k}]\\ne0\\qquad\\implies\\qquad(j,k)\\in E(T)\\qquad\\text{for all }j\\ne k.\r\n\\]\r\nHence, adding $\\mathbf{H}\\gets\\mathbf{H}+\\sigma\\xi_{i}\\xi_{i}^{T}$\r\none at a time results in at most $|V(T)|+|E(T)|$ dense blocks of\r\nat most $\\frac{1}{2}\\omega(\\omega+1)\\times\\frac{1}{2}\\omega(\\omega+1)$,\r\nfor a total memory cost of $O(\\omega^{4}n)$. \r\n","start":56231,"end":57483}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"稀疏乘法形成向量","source_units":[{"raw_tex":"\\textbf{(ii)} We form $\\q=[\\A^{T},\\N^{T}]w_{1}$ using a sparse matrix-vector\r\nproduct. Given that $\\nnz(w_{1})\\le d$ and $\\nnz([\\A;\\N])\\le2\\omega^{4}\\ell$,\r\nthis step costs $O(\\omega^{4}n)$ time and space.\r\n","start":57485,"end":57692}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"拓扑块 Cholesky 分解","source_units":[{"raw_tex":"\\textbf{(iii)} We partition $\\H$ into $[\\H_{i,j}]_{i,j=1}^{\\ell}$\r\nto reveal a block sparsity pattern that coincides with the adjacency\r\nmatrix of $T$:\r\n\\begin{align*}\r\n\\H_{i,j} & =\\begin{cases}\r\n\\D_{s,i}+\\sigma\\sum_{k=1}^{\\ell}\\N_{k,i}^{T}\\N_{k,i}+\\sigma\\sum_{q=1}^{m}a_{q,i}a_{q,i}^{T} & i=j\\\\\r\n\\sigma\\sum_{k=1}^{\\ell}\\N_{k,i}^{T}\\N_{k,j}+\\sigma\\sum_{q=1}^{m}a_{q,i}a_{q,j}^{T} & (i,j)\\in E(T)\\\\\r\n0 & \\text{otherwise}\r\n\\end{cases}\r\n\\end{align*}\r\nwhere $a_{q,i}=\\vector(A_{q,i})$. According to Lemma~\\ref{lem:blocktree},\r\nthe permuted matrix $\\Pi\\H\\Pi^{T}$ factors into $\\L\\L^{T}$ with\r\nno block fill in $O(\\omega^{6}n)$ time and $O(\\omega^{4}n)$ space,\r\nbecause each block $\\H_{i,j}$ for $i,j\\in\\{1,2,\\ldots,\\ell\\}$ is\r\nat most order $\\frac{1}{2}\\omega(\\omega+1)$.\r\n","start":57694,"end":58462}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"回代与秩一更新","source_units":[{"raw_tex":"\\textbf{(iv)} Using the rank-1 update formula (\\ref{eq:update}),\r\nthe cost of solving $(\\H+\\q\\q^{T})u=r$ is the same as the cost of\r\nsolving $\\H u=r$ for two right-hand sides, plus algebraic manipulations\r\nin $O(d)=O(\\omega^{2}n)$ time. Applying Lemma~\\ref{lem:blocktree}\r\nshows that the cost of solving $\\H u=r$ for each right-hand side\r\nis $O(\\omega^{4}n)$ time and space. \\qed\r\n\\end{proof}","start":58464,"end":58856}]},{"id":"S060","semantic_type":"lemma","kind":"lemma","labels":["lem:row-rank"],"declared_external":false,"source_only":false,"title":"重叠矩阵满行秩","source_units":[{"raw_tex":"\\begin{lemma}\r\n\\label{lem:row-rank}The matrix $\\N$ in (\\ref{eq:Nmat}) has full\r\nrow rank, that is $\\det(\\N\\N^{T})\\ne0$.\r\n\\end{lemma}","start":108146,"end":108278}]},{"id":"S061","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"满行秩的完整证明","source_units":[{"raw_tex":"\\begin{proof}\r\nWe make $\\N=[\\N_{i,j}]_{i,j=1}^{\\ell}$ upper-triangular by ordering\r\nits blocks topologically on $T$: each nonempty block row $\\N_{j}$\r\ncontains a nonzero block at $\\N_{j,j}$ and a nonzero block at $\\N_{j,p(j)}$\r\nwhere the parent node $p(j)>j$ is ordered after $j$. Then, the claim\r\nfollows because each diagonal block $\\N_{j,j}$ implements a surjection\r\nand must therefore have full row-rank. \\qed\r\n\\end{proof}","start":108280,"end":108706}]},{"id":"S062","semantic_type":"lemma","kind":"lemma","labels":["lem:orth"],"declared_external":false,"source_only":false,"title":"袋抽取与正交补","source_units":[{"raw_tex":"\\begin{lemma}[Orthogonal complement]\r\n\\label{lem:orth}Let $d=\\frac{1}{2}\\sum_{j=1}^{\\ell}|J_{j}|(|J_{j}|+1)$.\r\nImplicitly define the $d\\times\\frac{1}{2}n(n+1)$ matrix $\\P$ to\r\nsatisfy\r\n\\begin{align*}\r\n\\P\\,\\vector(X) & =[\\vector(X[J_{1},J_{1}]);\\ldots;\\vector(X[J_{\\ell},J_{\\ell}])]\\qquad\\forall X\\in\\S^{n}.\r\n\\end{align*}\r\nThen, (i) $\\N\\P=0$; (ii) every $x\\in\\R^{d}$ can be decomposed as\r\n$x=\\N^{T}u+\\P v$. \r\n\\end{lemma}","start":108710,"end":109129}]},{"id":"S063","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"正交补的完整证明","source_units":[{"raw_tex":"\\begin{proof}\r\nFor each $x=[\\vector(X_{j})]_{j=1}^{\\ell}\\in\\R^{d}$, Theorem~\\ref{thm:ctree}\r\nsays that there exists a $Z$ satisfying $\\P\\,\\vector(Z)=x$ if and\r\nonly if $\\N x=0$. Equivalently, $x\\in\\mathrm{span}(\\P)$ if and only\r\nif $x\\perp\\mathrm{span}(\\N^{T})$. The ``only if'' part implies\r\n(i), while the ``if'' part implies (ii).\r\n\\end{proof}","start":109131,"end":109477}]},{"id":"S064","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"线性无关传递完整证明","source_units":[{"raw_tex":"\\begin{proof}[Lemma~\\ref{lem:lin_indep}]We will prove that\r\n\\[\r\n\\text{exists }[u;v]\\ne0,\\quad\\A^{T}u+\\N^{T}v=0\\qquad\\iff\\qquad\\text{exists }y\\ne0,\\quad\\sum_{i=1}^{m}y_{i}A_{i}=0.\r\n\\]\r\n\\textbf{($\\implies$)} We must have $u\\ne0$, because $\\N$ has full\r\nrow rank by Lemma~\\ref{lem:row-rank}, and so $\\A^{T}0+\\N^{T}v=0$\r\nif and only if $v=0$. Multiplying by $\\P$ yields $\\P^{T}(\\A^{T}u+\\N^{T}v)=\\M^{T}u+0=0$\r\nand so setting $y=u\\ne0$ yields $\\M^{T}y=0$.\\textbf{ ($\\impliedby$)\r\n}We use Lemma~\\ref{lem:orth} to decompose $\\A^{T}y=\\P z+\\N^{T}v$.\r\nIf $\\vector(\\sum_{i}y_{i}A_{i})=\\M^{T}y=\\P^{T}\\A^{T}y=0,$ then $\\P^{T}\\P z=\\P^{T}\\A^{T}y-\\P^{T}\\N^{T}v=0$\r\nand so $\\P z=0$. Setting $u=-y\\ne0$ yields $\\A^{T}u+\\N^{T}v=0$.\\qed \\end{proof}","start":110279,"end":111006}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"转换相关推出原始相关","source_units":[{"raw_tex":"\\textbf{($\\implies$)} We must have $u\\ne0$, because $\\N$ has full\r\nrow rank by Lemma~\\ref{lem:row-rank}, and so $\\A^{T}0+\\N^{T}v=0$\r\nif and only if $v=0$. Multiplying by $\\P$ yields $\\P^{T}(\\A^{T}u+\\N^{T}v)=\\M^{T}u+0=0$\r\nand so setting $y=u\\ne0$ yields $\\M^{T}y=0$.\\textbf{ ($\\impliedby$)\r\n","start":110463,"end":110753}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"原始相关推出转换相关","source_units":[{"raw_tex":"and so setting $y=u\\ne0$ yields $\\M^{T}y=0$.\\textbf{ ($\\impliedby$)\r\n}We use Lemma~\\ref{lem:orth} to decompose $\\A^{T}y=\\P z+\\N^{T}v$.\r\nIf $\\vector(\\sum_{i}y_{i}A_{i})=\\M^{T}y=\\P^{T}\\A^{T}y=0,$ then $\\P^{T}\\P z=\\P^{T}\\A^{T}y-\\P^{T}\\N^{T}v=0$\r\nand so $\\P z=0$. Setting $u=-y\\ne0$ yields $\\A^{T}u+\\N^{T}v=0$.\\qed \\end{proof}","start":110684,"end":111006}]},{"id":"S067","semantic_type":"proof","kind":"proof","labels":["eq:prim_sep"],"declared_external":false,"source_only":false,"title":"原始 Slater 完整证明","source_units":[{"raw_tex":"\\begin{proof}[Lemma~\\ref{lem:slaterp}]We will prove that \r\n\\[\r\n\\text{exists }x\\in\\Int(\\K),\\quad\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}x=\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}\\qquad\\iff\\qquad\\text{exists }X\\succeq0,\\quad\\M\\,\\vector(X)=b.\r\n\\]\r\nDefine the chordal completion $F$ as in (\\ref{eq:chordal_compl}).\r\nObserve that $\\M\\,\\vector(X)=\\M\\,\\vector(Z)$ holds for all pairs\r\nof $P_{F}(X)=Z$, because each $A_{i}\\in\\S_{F}^{n}$ satisfies $A_{i}\\bullet X=A_{i}\\bullet P_{F}(X)$.\r\nAdditionally, the positive definite version of Theorem~\\ref{thm:prim_sep}\r\nis written\r\n\\begin{gather}\r\n\\text{exists }X\\succ0,\\quad P_{F}(X)=Z\\qquad\\iff\\qquad\\P\\,\\vector(Z)\\in\\Int(\\K).\\label{eq:prim_sep}\r\n\\end{gather}\r\nThis result was first established by Grone et al.~\\cite{grone1984positive};\r\na succinct proof can be found in~\\cite[Theorem~10.1]{vandenberghe2015chordal}.\\textbf{\r\n($\\implies$)} For every $x$ satisfying $\\N x=0$, there exists $Z$\r\nsuch that $\\P\\,\\vector(Z)=x$ due to Lemma~\\ref{lem:orth}. If additionally\r\n$x\\in\\Int(\\K)$, then there exists $X\\succ0$ satisfying $Z=P_{F}(X)$\r\ndue to (\\ref{eq:prim_sep}). We can verify that $\\M\\,\\vector(X)=\\M\\,\\vector(Z)=\\A\\P\\,\\vector(Z)=\\A x=b$.\\textbf{\r\n($\\impliedby$)} For every $X\\succ0,$ there exists $Z$ satisfying\r\n$Z=P_{F}(X)$ and $\\P\\,\\vector(Z)\\in\\Int(\\K)$ due to (\\ref{eq:prim_sep}).\r\nSet $u=\\P\\,\\vector(Z)$ and observe that $u\\in\\Int(\\K)$ and $\\N u=\\N\\P\\,\\vector(Z)=0$.\r\nIf additionally $\\M\\,\\vector(X)=b$, then $\\A u=\\A\\P\\,\\vector(Z)=\\M\\,\\vector(Z)=b$.\\qed \\end{proof}","start":111010,"end":112516}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"严格可行证明的投影准备","source_units":[{"raw_tex":"\\begin{proof}[Lemma~\\ref{lem:slaterp}]We will prove that \r\n\\[\r\n\\text{exists }x\\in\\Int(\\K),\\quad\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}x=\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}\\qquad\\iff\\qquad\\text{exists }X\\succeq0,\\quad\\M\\,\\vector(X)=b.\r\n\\]\r\nDefine the chordal completion $F$ as in (\\ref{eq:chordal_compl}).\r\nObserve that $\\M\\,\\vector(X)=\\M\\,\\vector(Z)$ holds for all pairs\r\nof $P_{F}(X)=Z$, because each $A_{i}\\in\\S_{F}^{n}$ satisfies $A_{i}\\bullet X=A_{i}\\bullet P_{F}(X)$.\r\n","start":111010,"end":111484}]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","labels":["eq:prim_sep"],"declared_external":true,"declaration_text":["原始前提列表明确包含 B0045 的正定版本；Grone et al., grone1984positive；Vandenberghe–Andersen, vandenberghe2015chordal, Theorem 10.1。"],"source_only":false,"title":"正定补全版本","source_units":[{"raw_tex":"Additionally, the positive definite version of Theorem~\\ref{thm:prim_sep}\r\nis written\r\n\\begin{gather}\r\n\\text{exists }X\\succ0,\\quad P_{F}(X)=Z\\qquad\\iff\\qquad\\P\\,\\vector(Z)\\in\\Int(\\K).\\label{eq:prim_sep}\r\n\\end{gather}\r\nThis result was first established by Grone et al.~\\cite{grone1984positive};\r\na succinct proof can be found in~\\cite[Theorem~10.1]{vandenberghe2015chordal}.\\textbf{\r\n","start":111484,"end":111867}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"由严格可行袋恢复全局矩阵","source_units":[{"raw_tex":"a succinct proof can be found in~\\cite[Theorem~10.1]{vandenberghe2015chordal}.\\textbf{\r\n($\\implies$)} For every $x$ satisfying $\\N x=0$, there exists $Z$\r\nsuch that $\\P\\,\\vector(Z)=x$ due to Lemma~\\ref{lem:orth}. If additionally\r\n$x\\in\\Int(\\K)$, then there exists $X\\succ0$ satisfying $Z=P_{F}(X)$\r\ndue to (\\ref{eq:prim_sep}). We can verify that $\\M\\,\\vector(X)=\\M\\,\\vector(Z)=\\A\\P\\,\\vector(Z)=\\A x=b$.\\textbf{\r\n","start":111779,"end":112191}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"由全局严格可行构造袋向量","source_units":[{"raw_tex":"($\\impliedby$)} For every $X\\succ0,$ there exists $Z$ satisfying\r\n$Z=P_{F}(X)$ and $\\P\\,\\vector(Z)\\in\\Int(\\K)$ due to (\\ref{eq:prim_sep}).\r\nSet $u=\\P\\,\\vector(Z)$ and observe that $u\\in\\Int(\\K)$ and $\\N u=\\N\\P\\,\\vector(Z)=0$.\r\nIf additionally $\\M\\,\\vector(X)=b$, then $\\A u=\\A\\P\\,\\vector(Z)=\\M\\,\\vector(Z)=b$.\\qed \\end{proof}","start":112191,"end":112516}]},{"id":"S072","semantic_type":"proof","kind":"proof","labels":["eq:dual_sep"],"declared_external":false,"source_only":false,"title":"对偶 Slater 完整作者证明","source_units":[{"raw_tex":"\\begin{proof}[Lemma~\\ref{lem:slaterp}]We will prove that\r\n\\[\r\n\\text{exists }u,v,\\quad c-\\A^{T}u-\\N^{T}v\\in\\Int(\\K_{*})\\qquad\\iff\\qquad\\text{exists }y,\\quad C-\\sum_{i}y_{i}A_{i}\\succ0.\r\n\\]\r\nDefine the chordal completion $F$ as in (\\ref{eq:chordal_compl}).\r\nTheorem~\\ref{thm:prim_sep} in (\\ref{eq:prim_sep}) has a dual theorem\r\n\\begin{gather}\r\n\\text{exists }S\\succ0,\\quad S\\in\\S_{F}^{n}\\qquad\\iff\\qquad\\text{exists }h\\in\\Int(\\K_{*}),\\quad\\vector(S)=\\P^{T}h.\\label{eq:dual_sep}\r\n\\end{gather}\r\nThis result readily follows from the positive semidefinite version\r\nproved by Alger et al.~\\cite{agler1988positive}; see also~\\cite[Theorem~9.2]{vandenberghe2015chordal}.\\textbf{\r\n($\\implies$)} For each $h=c-\\A^{T}u-\\N^{T}v$, define $S=C-\\sum_{i}u_{i}A_{i}$\r\nand observe that \r\n\\[\r\n\\P^{T}h=\\P^{T}(c-\\A^{T}u-\\N^{T}v)=\\vector(C)-\\M^{T}u-0=\\vector(S).\r\n\\]\r\nIf additionally $h\\in\\K_{*}$, then $S\\succ0$ due to (\\ref{eq:dual_sep}).\\textbf{\r\n($\\impliedby$)} For each $S=C-\\sum_{i}y_{i}A_{i}\\succ0$, there exists\r\nan $h\\in\\Int(\\K_{*})$ satisfying $\\vector(S)=\\P^{T}h$ due to (\\ref{eq:dual_sep}).\r\nWe use Lemma~\\ref{lem:orth} to decompose $h=\\P u+\\N^{T}v$. Given\r\nthat $\\vector(S)=\\P^{T}h=\\P^{T}\\P u+0$, we must actually have $\\P u=c-\\A^{T}y$\r\nsince $\\P^{T}(c-\\A^{T}y)=\\vector(C)-\\M^{T}y=\\vector(S)$. Hence $h=c-\\A^{T}y+\\N^{T}v$\r\nand $h\\in\\Int(\\K_{*})$.\\qed \\end{proof}","start":112520,"end":113871}]},{"id":"S073","semantic_type":"external_fact","kind":"external_fact","labels":["eq:dual_sep"],"declared_external":true,"declaration_text":["原始前提列表明确列入 Agler et al., agler1988positive，及 Vandenberghe–Andersen, vandenberghe2015chordal, Theorem 9.2，包含作者所用严格版本。"],"source_only":false,"title":"稀疏正定矩阵的严格团分解","source_units":[{"raw_tex":"Define the chordal completion $F$ as in (\\ref{eq:chordal_compl}).\r\nTheorem~\\ref{thm:prim_sep} in (\\ref{eq:prim_sep}) has a dual theorem\r\n\\begin{gather}\r\n\\text{exists }S\\succ0,\\quad S\\in\\S_{F}^{n}\\qquad\\iff\\qquad\\text{exists }h\\in\\Int(\\K_{*}),\\quad\\vector(S)=\\P^{T}h.\\label{eq:dual_sep}\r\n\\end{gather}\r\nThis result readily follows from the positive semidefinite version\r\nproved by Alger et al.~\\cite{agler1988positive}; see also~\\cite[Theorem~9.2]{vandenberghe2015chordal}.\\textbf{\r\n","start":112709,"end":113190}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"从袋对偶松弛到全局松弛","source_units":[{"raw_tex":"proved by Alger et al.~\\cite{agler1988positive}; see also~\\cite[Theorem~9.2]{vandenberghe2015chordal}.\\textbf{\r\n($\\implies$)} For each $h=c-\\A^{T}u-\\N^{T}v$, define $S=C-\\sum_{i}u_{i}A_{i}$\r\nand observe that \r\n\\[\r\n\\P^{T}h=\\P^{T}(c-\\A^{T}u-\\N^{T}v)=\\vector(C)-\\M^{T}u-0=\\vector(S).\r\n\\]\r\nIf additionally $h\\in\\K_{*}$, then $S\\succ0$ due to (\\ref{eq:dual_sep}).\\textbf{\r\n","start":113078,"end":113446}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"从全局松弛构造袋对偶松弛","source_units":[{"raw_tex":"($\\impliedby$)} For each $S=C-\\sum_{i}y_{i}A_{i}\\succ0$, there exists\r\nan $h\\in\\Int(\\K_{*})$ satisfying $\\vector(S)=\\P^{T}h$ due to (\\ref{eq:dual_sep}).\r\nWe use Lemma~\\ref{lem:orth} to decompose $h=\\P u+\\N^{T}v$. Given\r\nthat $\\vector(S)=\\P^{T}h=\\P^{T}\\P u+0$, we must actually have $\\P u=c-\\A^{T}y$\r\nsince $\\P^{T}(c-\\A^{T}y)=\\vector(C)-\\M^{T}y=\\vector(S)$. Hence $h=c-\\A^{T}y+\\N^{T}v$\r\nand $h\\in\\Int(\\K_{*})$.\\qed \\end{proof}","start":113446,"end":113871}]},{"id":"S076","semantic_type":"lemma","kind":"lemma","labels":["lem:eps"],"declared_external":false,"source_only":false,"title":"互补精度转为重缩放精度","source_units":[{"raw_tex":"\\begin{lemma}[$\\epsilon$-accurate and $\\epsilon$-feasible]\r\n\\label{lem:eps}If $(x,y,s,\\tau,\\theta,\\kappa)$ satisfies (\\ref{eq:hsdfeas})\r\nand (\\ref{eq:hsdcone}) and \r\n\\begin{align*}\r\n\\mu=\\frac{x^{T}s+\\tau\\kappa}{\\nu+1} & \\le\\epsilon, & \\tau\\kappa\\ge & \\gamma\\mu\r\n\\end{align*}\r\nfor constants $\\epsilon,\\gamma>0$, then the rescaled point $(x/\\tau,y/\\tau,s/\\tau)$\r\nsatisfies\r\n\\begin{align*}\r\n\\|\\M(x/\\tau)-\\b\\| & \\le K\\epsilon, & \\|\\M^{T}(y/\\tau)+(s/\\tau)-\\c\\| & \\le K\\epsilon, & \\frac{(x/\\tau)^{T}(s/\\tau)}{\\nu+1}\\le K\\epsilon\r\n\\end{align*}\r\nwhere $K$ is a constant.\r\n\\end{lemma}","start":117365,"end":117940}]},{"id":"S077","semantic_type":"assumption","kind":"assumption","labels":["lem:eps"],"declared_external":false,"source_only":false,"title":"重缩放引理的点条件","source_units":[{"raw_tex":"\\label{lem:eps}If $(x,y,s,\\tau,\\theta,\\kappa)$ satisfies (\\ref{eq:hsdfeas})\r\nand (\\ref{eq:hsdcone}) and \r\n\\begin{align*}\r\n\\mu=\\frac{x^{T}s+\\tau\\kappa}{\\nu+1} & \\le\\epsilon, & \\tau\\kappa\\ge & \\gamma\\mu\r\n\\end{align*}\r\nfor constants $\\epsilon,\\gamma>0$, then the rescaled point $(x/\\tau,y/\\tau,s/\\tau)$\r\n","start":117425,"end":117726}]},{"id":"S078","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"重缩放精度完整证明","source_units":[{"raw_tex":"\\begin{proof}\r\nNote that (\\ref{eq:hsdfeas}) implies $\\mu=\\theta$ and\r\n\\begin{align*}\r\n\\|\\M(x/\\tau)-\\b\\| & =\\frac{\\|r_{p}\\|\\mu}{\\tau}, & \\|\\M^{T}(y/\\tau)+(s/\\tau)-\\c\\| & =\\frac{\\|r_{d}\\|\\mu}{\\tau}, & \\frac{(x/\\tau)^{T}(s/\\tau)}{\\nu+1}=\\frac{\\mu}{\\tau^{2}}.\r\n\\end{align*}\r\nHence, we obtain our desired result by upper-bounding $1/\\tau$. Let\r\n$(x^{\\star},y^{\\star},s^{\\star},\\tau^{\\star},\\theta^{\\star},\\kappa^{\\star})$\r\nbe a solution of (\\ref{eq:hsd}), and note that for every $(x,y,s,\\tau,\\theta,\\kappa)$\r\nsatisfying (\\ref{eq:hsdfeas}) and (\\ref{eq:hsdcone}), we have the\r\nfollowing via the skew-symmetry of (\\ref{eq:hsdfeas})\r\n\\[\r\n(x-x^{\\star})^{T}(s-s^{\\star})+(\\tau-\\tau^{\\star})(\\kappa-\\kappa^{\\star})=0.\r\n\\]\r\nRearranging yields\r\n\\[\r\n(\\nu+1)\\mu=x^{T}s+\\tau\\kappa=(x^{\\star})^{T}s+x^{T}(s^{\\star})+\\tau\\kappa^{\\star}+\\tau^{\\star}\\kappa\\ge\\tau^{\\star}\\kappa\r\n\\]\r\nand hence\r\n\\[\r\n\\kappa\\tau^{\\star}\\le\\mu(\\nu+1),\\quad\\tau\\kappa\\ge\\gamma\\mu\\qquad\\implies\\qquad\\tau\\ge\\frac{\\gamma}{\\nu+1}\\tau^{\\star}.\r\n\\]\r\n\r\nIf (\\ref{eq:SDP}) satisfies the primal-dual Slater's condition, then\r\n(\\ref{eq:CTC}) also satisfies the primal-dual Slater's condition\r\n(Lemmas~\\ref{lem:slaterp} \\&~\\ref{lem:slaterd}). Therefore, the\r\nvectorized version (\\ref{eq:vecCTC}) of (\\ref{eq:CTC}) attains a\r\nsolution $(\\hat{x},\\hat{y},\\hat{s})$ with $\\hat{x}^{T}\\hat{s}=0$,\r\nand the following\r\n\\[\r\nx^{\\star}=\\tau^{\\star}\\begin{bmatrix}\\|\\hat{y}\\|\\\\\r\n\\hat{y}\\\\\r\n\\hat{s}\r\n\\end{bmatrix},\\quad y^{\\star}=\\tau^{\\star}\\hat{x},\\quad s^{\\star}=\\tau^{\\star}\\begin{bmatrix}0\\\\\r\n0\\\\\r\n\\hat{x}\r\n\\end{bmatrix},\\quad\\tau^{\\star}=\\frac{\\nu+1}{\\|\\hat{y}\\|+\\one_{\\K}^{T}\\hat{s}+\\one_{\\K}^{T}\\hat{x}+1},\r\n\\]\r\nwith $\\theta^{\\star}=\\kappa^{\\star}=0$ is a solution to (\\ref{eq:hsd}).\r\nThis proves the following upper-bound \r\n\\[\r\n\\frac{1}{\\tau}\\le K_{\\tau}=\\frac{1}{\\gamma}\\cdot\\min_{\\hat{x},\\hat{y},\\hat{s}}\\{\\|\\hat{y}\\|+\\one_{\\K}^{T}\\hat{s}+\\one_{\\K}^{T}\\hat{x}+1:(\\hat{x},\\hat{y},\\hat{s})\\text{ solves (\\ref{eq:vecCTC}) with }\\hat{x}^{T}\\hat{s}=0\\}.\r\n\\]\r\nSetting $K=\\max\\{\\|r_{p}\\|K_{\\tau},\\|r_{d}\\|K_{\\tau},K_{\\tau}^{2}\\}$\r\nyields our desired result. \\qed\r\n\\end{proof}","start":117942,"end":120056}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"嵌入误差的显式表达","source_units":[{"raw_tex":"\\begin{proof}\r\nNote that (\\ref{eq:hsdfeas}) implies $\\mu=\\theta$ and\r\n\\begin{align*}\r\n\\|\\M(x/\\tau)-\\b\\| & =\\frac{\\|r_{p}\\|\\mu}{\\tau}, & \\|\\M^{T}(y/\\tau)+(s/\\tau)-\\c\\| & =\\frac{\\|r_{d}\\|\\mu}{\\tau}, & \\frac{(x/\\tau)^{T}(s/\\tau)}{\\nu+1}=\\frac{\\mu}{\\tau^{2}}.\r\n\\end{align*}\r\n","start":117942,"end":118213}]},{"id":"S080","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"由斜对称性控制 τ","source_units":[{"raw_tex":"Hence, we obtain our desired result by upper-bounding $1/\\tau$. Let\r\n$(x^{\\star},y^{\\star},s^{\\star},\\tau^{\\star},\\theta^{\\star},\\kappa^{\\star})$\r\nbe a solution of (\\ref{eq:hsd}), and note that for every $(x,y,s,\\tau,\\theta,\\kappa)$\r\nsatisfying (\\ref{eq:hsdfeas}) and (\\ref{eq:hsdcone}), we have the\r\nfollowing via the skew-symmetry of (\\ref{eq:hsdfeas})\r\n\\[\r\n(x-x^{\\star})^{T}(s-s^{\\star})+(\\tau-\\tau^{\\star})(\\kappa-\\kappa^{\\star})=0.\r\n\\]\r\nRearranging yields\r\n\\[\r\n(\\nu+1)\\mu=x^{T}s+\\tau\\kappa=(x^{\\star})^{T}s+x^{T}(s^{\\star})+\\tau\\kappa^{\\star}+\\tau^{\\star}\\kappa\\ge\\tau^{\\star}\\kappa\r\n\\]\r\nand hence\r\n\\[\r\n\\kappa\\tau^{\\star}\\le\\mu(\\nu+1),\\quad\\tau\\kappa\\ge\\gamma\\mu\\qquad\\implies\\qquad\\tau\\ge\\frac{\\gamma}{\\nu+1}\\tau^{\\star}.\r\n\\]\r\n","start":118213,"end":118946}]},{"id":"S081","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"构造正 τ 的最优嵌入点","source_units":[{"raw_tex":"If (\\ref{eq:SDP}) satisfies the primal-dual Slater's condition, then\r\n(\\ref{eq:CTC}) also satisfies the primal-dual Slater's condition\r\n(Lemmas~\\ref{lem:slaterp} \\&~\\ref{lem:slaterd}). Therefore, the\r\nvectorized version (\\ref{eq:vecCTC}) of (\\ref{eq:CTC}) attains a\r\nsolution $(\\hat{x},\\hat{y},\\hat{s})$ with $\\hat{x}^{T}\\hat{s}=0$,\r\nand the following\r\n\\[\r\nx^{\\star}=\\tau^{\\star}\\begin{bmatrix}\\|\\hat{y}\\|\\\\\r\n\\hat{y}\\\\\r\n\\hat{s}\r\n\\end{bmatrix},\\quad y^{\\star}=\\tau^{\\star}\\hat{x},\\quad s^{\\star}=\\tau^{\\star}\\begin{bmatrix}0\\\\\r\n0\\\\\r\n\\hat{x}\r\n\\end{bmatrix},\\quad\\tau^{\\star}=\\frac{\\nu+1}{\\|\\hat{y}\\|+\\one_{\\K}^{T}\\hat{s}+\\one_{\\K}^{T}\\hat{x}+1},\r\n\\]\r\nwith $\\theta^{\\star}=\\kappa^{\\star}=0$ is a solution to (\\ref{eq:hsd}).\r\n","start":118948,"end":119670}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"精度常数的选择","source_units":[{"raw_tex":"This proves the following upper-bound \r\n\\[\r\n\\frac{1}{\\tau}\\le K_{\\tau}=\\frac{1}{\\gamma}\\cdot\\min_{\\hat{x},\\hat{y},\\hat{s}}\\{\\|\\hat{y}\\|+\\one_{\\K}^{T}\\hat{s}+\\one_{\\K}^{T}\\hat{x}+1:(\\hat{x},\\hat{y},\\hat{s})\\text{ solves (\\ref{eq:vecCTC}) with }\\hat{x}^{T}\\hat{s}=0\\}.\r\n\\]\r\nSetting $K=\\max\\{\\|r_{p}\\|K_{\\tau},\\|r_{d}\\|K_{\\tau},K_{\\tau}^{2}\\}$\r\nyields our desired result. \\qed\r\n\\end{proof}","start":119670,"end":120056}]},{"id":"S083","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"近线性主定理完整证明","source_units":[{"raw_tex":"\\begin{proof}[Theorem~\\ref{thm:lintime}]Combining Lemma~\\ref{lem:eps}\r\nand Lemma~\\ref{lem:short-step} shows that the desired $\\epsilon$-accurate,\r\n$\\epsilon$-feasible iterate is obtained after $O(\\sqrt{\\nu}\\log(1/\\epsilon))$\r\ninterior-point iterations. At each iteration we perform the following\r\nsteps: 1) compute the scaling point $w$; 2) solve the normal equation\r\n(\\ref{eq:hsd_nrm}) for three right-hand sides; 3) back-substitute\r\n(\\ref{eq:hsd_back1})-(\\ref{eq:hsd_back2}) for the search direction\r\nand take the step in (\\ref{eq:hsd_ipm}). Note from the proof of Lemma~\\ref{lem:normal}\r\nthat the matrix $[\\A;\\N]$ has at most $O(\\omega^{2}n)$ rows under\r\nAssumption~\\ref{ass:lin}, and therefore $\\nnz(\\M)=O(\\omega^{4}n)$\r\nunder the hypothesis of Theorem~\\ref{thm:lintime}. Below, we show\r\nthat the cost of each step is bounded by $O(\\omega^{6}n)$ time and\r\n$O(\\omega^{4}n)$ memory.\r\n\r\n\\textbf{Scaling point.} We partition $x=[x_{0};x_{1};\\vector(X_{1});\\ldots;\\vector(X_{\\ell})]$\r\nand similarly for $s$. Then, the scaling point $w$ is given in closed-form~\\cite[Section~5]{sturm2002implementation}\r\n\\begin{gather*}\r\n\\begin{bmatrix}u_{0}\\\\\r\nu_{1}\r\n\\end{bmatrix}=2^{-3/4}\\begin{bmatrix}1 & 1\\\\\r\n-s_{1}/\\|s_{1}\\| & s_{1}/\\|s_{1}\\|\r\n\\end{bmatrix}\\begin{bmatrix}(s_{0}-\\|s_{1}\\|)^{1/2}\\\\\r\n(s_{0}+\\|s_{1}\\|)^{1/2}\r\n\\end{bmatrix},\\quad\\begin{bmatrix}v_{0}\\\\\r\nv_{1}\r\n\\end{bmatrix}=\\begin{bmatrix}u_{0} & u_{1}^{T}\\\\\r\nu_{1} & \\frac{1}{2}(u_{0}^{2}-u_{1}^{T}u_{1})I\r\n\\end{bmatrix}\\begin{bmatrix}x_{0}\\\\\r\nx_{1}\r\n\\end{bmatrix},\\\\\r\n\\begin{bmatrix}w_{0}\\\\\r\nw_{1}\r\n\\end{bmatrix}=2^{-1/4}\\begin{bmatrix}u_{0} & u_{1}^{T}\\\\\r\nu_{1} & \\frac{1}{2}(u_{0}^{2}-u_{1}^{T}u_{1})I\r\n\\end{bmatrix}\\begin{bmatrix}1 & 1\\\\\r\n-v_{1}/\\|v_{1}\\| & v_{1}/\\|v_{1}\\|\r\n\\end{bmatrix}\\begin{bmatrix}(v_{0}-\\|v_{1}\\|)^{-1/2}\\\\\r\n(v_{0}+\\|v_{1}\\|)^{-1/2}\r\n\\end{bmatrix},\\\\\r\nW_{j}=S_{j}^{1/2}(S_{j}^{1/2}X_{j}S_{j}^{1/2})^{-1/2}S_{j}^{1/2}\\qquad\\text{for all }j\\in\\{1,\\ldots,\\ell\\}.\r\n\\end{gather*}\r\nNoting that $\\nnz(w_{1})\\le O(\\omega^{2}n)$, $\\ell\\le n$ and each\r\n$W_{j}$ is at most $\\omega\\times\\omega$, the cost of forming $w=[w_{0};w_{1};\\vector(W_{1});\\ldots;\\vector(W_{\\ell})]$\r\nis at most $O(\\omega^{3}n)$ time and $O(\\omega^{2}n)$ memory. Also,\r\nsince\r\n\\[\r\n\\D=\\nabla^{2}F(w)=\\diag\\left(\\begin{bmatrix}w_{0} & w_{1}^{T}\\\\\r\nw_{1} & \\frac{1}{2}(w_{0}^{2}-w_{1}^{T}w_{1})I\r\n\\end{bmatrix},W_{1}\\otimes_{s}W_{1},\\ldots,W_{\\ell}\\otimes_{s}W_{\\ell}\\right)^{-1},\r\n\\]\r\nthe cost of each matrix-vector product with $\\D$ and $\\D^{-1}$ is\r\nalso $O(\\omega^{3}n)$ time and $O(\\omega^{2}n)$ memory.\r\n\r\n\\textbf{Normal equation. }The cost of matrix-vector products with\r\n$\\M$ and $\\M^{T}$ is $\\nnz(\\M)=O(\\omega^{4}n)$ time and memory.\r\nUsing Lemma~\\ref{lem:normal}, we form the right-hand sides and solve\r\nthe three normal equations in (\\ref{eq:hsd_nrm}) in $O(\\omega^{6}n)$\r\ntime and $O(\\omega^{4}n)$ memory.\r\n\r\n\\textbf{Back-substitution.} The cost of back substituting (\\ref{eq:hsd_back1})-(\\ref{eq:hsd_back2})\r\nand making the step (\\ref{eq:hsd_ipm}) is dominated by matrix-vector\r\nproducts with $\\D$, $\\D^{-1}$, $\\M$, and $\\M^{T}$ at $O(\\omega^{4}n)$\r\ntime and memory.\\qed \\end{proof}","start":124868,"end":128002}]},{"id":"S084","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"组合精度与迭代次数","source_units":[{"raw_tex":"\\begin{proof}[Theorem~\\ref{thm:lintime}]Combining Lemma~\\ref{lem:eps}\r\nand Lemma~\\ref{lem:short-step} shows that the desired $\\epsilon$-accurate,\r\n$\\epsilon$-feasible iterate is obtained after $O(\\sqrt{\\nu}\\log(1/\\epsilon))$\r\ninterior-point iterations. At each iteration we perform the following\r\n","start":124868,"end":125165}]},{"id":"S085","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"每次迭代的任务与数据规模","source_units":[{"raw_tex":"interior-point iterations. At each iteration we perform the following\r\nsteps: 1) compute the scaling point $w$; 2) solve the normal equation\r\n(\\ref{eq:hsd_nrm}) for three right-hand sides; 3) back-substitute\r\n(\\ref{eq:hsd_back1})-(\\ref{eq:hsd_back2}) for the search direction\r\nand take the step in (\\ref{eq:hsd_ipm}). Note from the proof of Lemma~\\ref{lem:normal}\r\nthat the matrix $[\\A;\\N]$ has at most $O(\\omega^{2}n)$ rows under\r\nAssumption~\\ref{ass:lin}, and therefore $\\nnz(\\M)=O(\\omega^{4}n)$\r\nunder the hypothesis of Theorem~\\ref{thm:lintime}. Below, we show\r\nthat the cost of each step is bounded by $O(\\omega^{6}n)$ time and\r\n$O(\\omega^{4}n)$ memory.\r\n","start":125094,"end":125754}]},{"id":"S086","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"缩放点与 Hessian 乘法成本","source_units":[{"raw_tex":"\\textbf{Scaling point.} We partition $x=[x_{0};x_{1};\\vector(X_{1});\\ldots;\\vector(X_{\\ell})]$\r\nand similarly for $s$. Then, the scaling point $w$ is given in closed-form~\\cite[Section~5]{sturm2002implementation}\r\n\\begin{gather*}\r\n\\begin{bmatrix}u_{0}\\\\\r\nu_{1}\r\n\\end{bmatrix}=2^{-3/4}\\begin{bmatrix}1 & 1\\\\\r\n-s_{1}/\\|s_{1}\\| & s_{1}/\\|s_{1}\\|\r\n\\end{bmatrix}\\begin{bmatrix}(s_{0}-\\|s_{1}\\|)^{1/2}\\\\\r\n(s_{0}+\\|s_{1}\\|)^{1/2}\r\n\\end{bmatrix},\\quad\\begin{bmatrix}v_{0}\\\\\r\nv_{1}\r\n\\end{bmatrix}=\\begin{bmatrix}u_{0} & u_{1}^{T}\\\\\r\nu_{1} & \\frac{1}{2}(u_{0}^{2}-u_{1}^{T}u_{1})I\r\n\\end{bmatrix}\\begin{bmatrix}x_{0}\\\\\r\nx_{1}\r\n\\end{bmatrix},\\\\\r\n\\begin{bmatrix}w_{0}\\\\\r\nw_{1}\r\n\\end{bmatrix}=2^{-1/4}\\begin{bmatrix}u_{0} & u_{1}^{T}\\\\\r\nu_{1} & \\frac{1}{2}(u_{0}^{2}-u_{1}^{T}u_{1})I\r\n\\end{bmatrix}\\begin{bmatrix}1 & 1\\\\\r\n-v_{1}/\\|v_{1}\\| & v_{1}/\\|v_{1}\\|\r\n\\end{bmatrix}\\begin{bmatrix}(v_{0}-\\|v_{1}\\|)^{-1/2}\\\\\r\n(v_{0}+\\|v_{1}\\|)^{-1/2}\r\n\\end{bmatrix},\\\\\r\nW_{j}=S_{j}^{1/2}(S_{j}^{1/2}X_{j}S_{j}^{1/2})^{-1/2}S_{j}^{1/2}\\qquad\\text{for all }j\\in\\{1,\\ldots,\\ell\\}.\r\n\\end{gather*}\r\nNoting that $\\nnz(w_{1})\\le O(\\omega^{2}n)$, $\\ell\\le n$ and each\r\n$W_{j}$ is at most $\\omega\\times\\omega$, the cost of forming $w=[w_{0};w_{1};\\vector(W_{1});\\ldots;\\vector(W_{\\ell})]$\r\nis at most $O(\\omega^{3}n)$ time and $O(\\omega^{2}n)$ memory. Also,\r\nsince\r\n\\[\r\n\\D=\\nabla^{2}F(w)=\\diag\\left(\\begin{bmatrix}w_{0} & w_{1}^{T}\\\\\r\nw_{1} & \\frac{1}{2}(w_{0}^{2}-w_{1}^{T}w_{1})I\r\n\\end{bmatrix},W_{1}\\otimes_{s}W_{1},\\ldots,W_{\\ell}\\otimes_{s}W_{\\ell}\\right)^{-1},\r\n\\]\r\nthe cost of each matrix-vector product with $\\D$ and $\\D^{-1}$ is\r\nalso $O(\\omega^{3}n)$ time and $O(\\omega^{2}n)$ memory.\r\n","start":125756,"end":127417}]},{"id":"S087","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"形成并求解三个正常方程","source_units":[{"raw_tex":"\\textbf{Normal equation. }The cost of matrix-vector products with\r\n$\\M$ and $\\M^{T}$ is $\\nnz(\\M)=O(\\omega^{4}n)$ time and memory.\r\nUsing Lemma~\\ref{lem:normal}, we form the right-hand sides and solve\r\nthe three normal equations in (\\ref{eq:hsd_nrm}) in $O(\\omega^{6}n)$\r\ntime and $O(\\omega^{4}n)$ memory.\r\n","start":127419,"end":127726}]},{"id":"S088","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"回代和更新的成本","source_units":[{"raw_tex":"\\textbf{Back-substitution.} The cost of back substituting (\\ref{eq:hsd_back1})-(\\ref{eq:hsd_back2})\r\nand making the step (\\ref{eq:hsd_ipm}) is dominated by matrix-vector\r\nproducts with $\\D$, $\\D^{-1}$, $\\M$, and $\\M^{T}$ at $O(\\omega^{4}n)$\r\ntime and memory.\\qed \\end{proof}","start":127728,"end":128002}]},{"id":"S089","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"合成总时间与峰值空间","source_units":[{"raw_tex":"\\begin{proof}[Theorem~\\ref{thm:lintime}]Combining Lemma~\\ref{lem:eps}\r\nand Lemma~\\ref{lem:short-step} shows that the desired $\\epsilon$-accurate,\r\n$\\epsilon$-feasible iterate is obtained after $O(\\sqrt{\\nu}\\log(1/\\epsilon))$\r\ninterior-point iterations. At each iteration we perform the following\r\nsteps: 1) compute the scaling point $w$; 2) solve the normal equation\r\n(\\ref{eq:hsd_nrm}) for three right-hand sides; 3) back-substitute\r\n(\\ref{eq:hsd_back1})-(\\ref{eq:hsd_back2}) for the search direction\r\nand take the step in (\\ref{eq:hsd_ipm}). Note from the proof of Lemma~\\ref{lem:normal}\r\nthat the matrix $[\\A;\\N]$ has at most $O(\\omega^{2}n)$ rows under\r\nAssumption~\\ref{ass:lin}, and therefore $\\nnz(\\M)=O(\\omega^{4}n)$\r\nunder the hypothesis of Theorem~\\ref{thm:lintime}. Below, we show\r\nthat the cost of each step is bounded by $O(\\omega^{6}n)$ time and\r\n$O(\\omega^{4}n)$ memory.\r\n","start":124868,"end":125754},{"raw_tex":"\\textbf{Normal equation. }The cost of matrix-vector products with\r\n$\\M$ and $\\M^{T}$ is $\\nnz(\\M)=O(\\omega^{4}n)$ time and memory.\r\nUsing Lemma~\\ref{lem:normal}, we form the right-hand sides and solve\r\nthe three normal equations in (\\ref{eq:hsd_nrm}) in $O(\\omega^{6}n)$\r\ntime and $O(\\omega^{4}n)$ memory.\r\n\r\n\\textbf{Back-substitution.} The cost of back substituting (\\ref{eq:hsd_back1})-(\\ref{eq:hsd_back2})\r\nand making the step (\\ref{eq:hsd_ipm}) is dominated by matrix-vector\r\nproducts with $\\D$, $\\D^{-1}$, $\\M$, and $\\M^{T}$ at $O(\\omega^{4}n)$\r\ntime and memory.\\qed \\end{proof}","start":127419,"end":128002}]},{"id":"S090","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始外部前提列表中的 Standard conic strong duality and attainment under primal-dual Slater conditions。"],"source_only":false,"title":"锥强对偶与最优解可达","source_units":[]},{"id":"S091","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表中的 Standard sparse Cholesky, Sherman–Morrison–Woodbury, and Nesterov–Todd scaling formulas；作者在 1037–1038 行点名 SMW。"],"source_only":false,"title":"Sherman–Morrison–Woodbury 公式","source_units":[]},{"id":"S092","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表中的标准 Nesterov–Todd scaling formulas；作者于 2367 行明确引用 Sturm, sturm2002implementation, Section 5。"],"source_only":false,"title":"Nesterov–Todd 闭式缩放","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S047","edges":[{"source_id":"S003","target_id":"S013","kind":"supports"},{"source_id":"S004","target_id":"S013","kind":"supports"},{"source_id":"S005","target_id":"S013","kind":"supports"},{"source_id":"S001","target_id":"S015","kind":"supports"},{"source_id":"S012","target_id":"S015","kind":"supports"},{"source_id":"S013","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S004","target_id":"S016","kind":"supports"},{"source_id":"S009","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S019","kind":"supports"},{"source_id":"S016","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S017","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S021","kind":"supports"},{"source_id":"S018","target_id":"S021","kind":"supports"},{"source_id":"S010","target_id":"S022","kind":"supports"},{"source_id":"S011","target_id":"S022","kind":"supports"},{"source_id":"S016","target_id":"S022","kind":"supports"},{"source_id":"S010","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S023","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S023","target_id":"S032","kind":"supports"},{"source_id":"S091","target_id":"S032","kind":"supports"},{"source_id":"S024","target_id":"S061","kind":"supports"},{"source_id":"S022","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S017","target_id":"S063","kind":"supports"},{"source_id":"S022","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S062","kind":"supports"},{"source_id":"S018","target_id":"S034","kind":"supports"},{"source_id":"S021","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S062","target_id":"S034","kind":"supports"},{"source_id":"S060","target_id":"S065","kind":"supports"},{"source_id":"S062","target_id":"S065","kind":"supports"},{"source_id":"S034","target_id":"S065","kind":"supports"},{"source_id":"S062","target_id":"S066","kind":"supports"},{"source_id":"S034","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S049","kind":"supports"},{"source_id":"S013","target_id":"S068","kind":"supports"},{"source_id":"S012","target_id":"S068","kind":"supports"},{"source_id":"S033","target_id":"S068","kind":"supports"},{"source_id":"S068","target_id":"S070","kind":"supports"},{"source_id":"S062","target_id":"S070","kind":"supports"},{"source_id":"S069","target_id":"S070","kind":"supports"},{"source_id":"S034","target_id":"S070","kind":"supports"},{"source_id":"S068","target_id":"S071","kind":"supports"},{"source_id":"S062","target_id":"S071","kind":"supports"},{"source_id":"S069","target_id":"S071","kind":"supports"},{"source_id":"S034","target_id":"S071","kind":"supports"},{"source_id":"S068","target_id":"S067","kind":"supports"},{"source_id":"S070","target_id":"S067","kind":"supports"},{"source_id":"S071","target_id":"S067","kind":"supports"},{"source_id":"S067","target_id":"S050","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S034","target_id":"S074","kind":"supports"},{"source_id":"S062","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S075","kind":"supports"},{"source_id":"S062","target_id":"S075","kind":"supports"},{"source_id":"S034","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S072","kind":"supports"},{"source_id":"S075","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S051","kind":"supports"},{"source_id":"S006","target_id":"S055","kind":"supports"},{"source_id":"S049","target_id":"S055","kind":"supports"},{"source_id":"S020","target_id":"S055","kind":"supports"},{"source_id":"S022","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S023","target_id":"S056","kind":"supports"},{"source_id":"S022","target_id":"S056","kind":"supports"},{"source_id":"S053","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S057","kind":"supports"},{"source_id":"S030","target_id":"S057","kind":"supports"},{"source_id":"S056","target_id":"S058","kind":"supports"},{"source_id":"S023","target_id":"S058","kind":"supports"},{"source_id":"S024","target_id":"S058","kind":"supports"},{"source_id":"S025","target_id":"S058","kind":"supports"},{"source_id":"S055","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S032","target_id":"S059","kind":"supports"},{"source_id":"S025","target_id":"S059","kind":"supports"},{"source_id":"S055","target_id":"S054","kind":"supports"},{"source_id":"S056","target_id":"S054","kind":"supports"},{"source_id":"S057","target_id":"S054","kind":"supports"},{"source_id":"S058","target_id":"S054","kind":"supports"},{"source_id":"S059","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S003","target_id":"S048","kind":"supports"},{"source_id":"S004","target_id":"S048","kind":"supports"},{"source_id":"S006","target_id":"S048","kind":"supports"},{"source_id":"S008","target_id":"S048","kind":"supports"},{"source_id":"S019","target_id":"S048","kind":"supports"},{"source_id":"S009","target_id":"S047","kind":"supports"},{"source_id":"S010","target_id":"S047","kind":"supports"},{"source_id":"S020","target_id":"S047","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S020","target_id":"S035","kind":"supports"},{"source_id":"S028","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S010","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S077","kind":"supports"},{"source_id":"S036","target_id":"S077","kind":"supports"},{"source_id":"S035","target_id":"S079","kind":"supports"},{"source_id":"S036","target_id":"S079","kind":"supports"},{"source_id":"S077","target_id":"S079","kind":"supports"},{"source_id":"S008","target_id":"S081","kind":"supports"},{"source_id":"S050","target_id":"S081","kind":"supports"},{"source_id":"S051","target_id":"S081","kind":"supports"},{"source_id":"S090","target_id":"S081","kind":"supports"},{"source_id":"S020","target_id":"S081","kind":"supports"},{"source_id":"S035","target_id":"S081","kind":"supports"},{"source_id":"S036","target_id":"S081","kind":"supports"},{"source_id":"S035","target_id":"S080","kind":"supports"},{"source_id":"S036","target_id":"S080","kind":"supports"},{"source_id":"S077","target_id":"S080","kind":"supports"},{"source_id":"S081","target_id":"S080","kind":"supports"},{"source_id":"S079","target_id":"S082","kind":"supports"},{"source_id":"S080","target_id":"S082","kind":"supports"},{"source_id":"S081","target_id":"S082","kind":"supports"},{"source_id":"S079","target_id":"S078","kind":"supports"},{"source_id":"S080","target_id":"S078","kind":"supports"},{"source_id":"S081","target_id":"S078","kind":"supports"},{"source_id":"S082","target_id":"S078","kind":"supports"},{"source_id":"S078","target_id":"S076","kind":"supports"},{"source_id":"S035","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S035","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S077","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S042","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S040","target_id":"S043","kind":"supports"},{"source_id":"S035","target_id":"S044","kind":"supports"},{"source_id":"S040","target_id":"S044","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S044","target_id":"S046","kind":"supports"},{"source_id":"S052","target_id":"S046","kind":"supports"},{"source_id":"S076","target_id":"S084","kind":"supports"},{"source_id":"S041","target_id":"S084","kind":"supports"},{"source_id":"S055","target_id":"S085","kind":"supports"},{"source_id":"S048","target_id":"S085","kind":"supports"},{"source_id":"S035","target_id":"S085","kind":"supports"},{"source_id":"S092","target_id":"S086","kind":"supports"},{"source_id":"S040","target_id":"S086","kind":"supports"},{"source_id":"S020","target_id":"S086","kind":"supports"},{"source_id":"S085","target_id":"S086","kind":"supports"},{"source_id":"S045","target_id":"S086","kind":"supports"},{"source_id":"S046","target_id":"S087","kind":"supports"},{"source_id":"S052","target_id":"S087","kind":"supports"},{"source_id":"S042","target_id":"S087","kind":"supports"},{"source_id":"S085","target_id":"S087","kind":"supports"},{"source_id":"S086","target_id":"S087","kind":"supports"},{"source_id":"S043","target_id":"S088","kind":"supports"},{"source_id":"S038","target_id":"S088","kind":"supports"},{"source_id":"S085","target_id":"S088","kind":"supports"},{"source_id":"S086","target_id":"S088","kind":"supports"},{"source_id":"S084","target_id":"S089","kind":"supports"},{"source_id":"S086","target_id":"S089","kind":"supports"},{"source_id":"S087","target_id":"S089","kind":"supports"},{"source_id":"S088","target_id":"S089","kind":"supports"},{"source_id":"S036","target_id":"S089","kind":"supports"},{"source_id":"S084","target_id":"S083","kind":"supports"},{"source_id":"S085","target_id":"S083","kind":"supports"},{"source_id":"S086","target_id":"S083","kind":"supports"},{"source_id":"S087","target_id":"S083","kind":"supports"},{"source_id":"S088","target_id":"S083","kind":"supports"},{"source_id":"S089","target_id":"S083","kind":"supports"},{"source_id":"S083","target_id":"S047","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S047","edges":[{"source_id":"S003","target_id":"S013","kind":"supports"},{"source_id":"S004","target_id":"S013","kind":"supports"},{"source_id":"S005","target_id":"S013","kind":"supports"},{"source_id":"S001","target_id":"S015","kind":"supports"},{"source_id":"S012","target_id":"S015","kind":"supports"},{"source_id":"S013","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S004","target_id":"S016","kind":"supports"},{"source_id":"S009","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S019","kind":"supports"},{"source_id":"S016","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S017","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S021","kind":"supports"},{"source_id":"S018","target_id":"S021","kind":"supports"},{"source_id":"S010","target_id":"S022","kind":"supports"},{"source_id":"S011","target_id":"S022","kind":"supports"},{"source_id":"S016","target_id":"S022","kind":"supports"},{"source_id":"S010","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S023","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S023","target_id":"S032","kind":"supports"},{"source_id":"S091","target_id":"S032","kind":"supports"},{"source_id":"S024","target_id":"S060","kind":"supports"},{"source_id":"S022","target_id":"S060","kind":"supports"},{"source_id":"S017","target_id":"S062","kind":"supports"},{"source_id":"S022","target_id":"S062","kind":"supports"},{"source_id":"S018","target_id":"S034","kind":"supports"},{"source_id":"S021","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S062","target_id":"S034","kind":"supports"},{"source_id":"S060","target_id":"S049","kind":"supports"},{"source_id":"S062","target_id":"S049","kind":"supports"},{"source_id":"S034","target_id":"S049","kind":"supports"},{"source_id":"S013","target_id":"S050","kind":"supports"},{"source_id":"S012","target_id":"S050","kind":"supports"},{"source_id":"S033","target_id":"S050","kind":"supports"},{"source_id":"S062","target_id":"S050","kind":"supports"},{"source_id":"S069","target_id":"S050","kind":"supports"},{"source_id":"S034","target_id":"S050","kind":"supports"},{"source_id":"S073","target_id":"S051","kind":"supports"},{"source_id":"S034","target_id":"S051","kind":"supports"},{"source_id":"S062","target_id":"S051","kind":"supports"},{"source_id":"S006","target_id":"S055","kind":"supports"},{"source_id":"S049","target_id":"S055","kind":"supports"},{"source_id":"S020","target_id":"S055","kind":"supports"},{"source_id":"S022","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S052","kind":"supports"},{"source_id":"S023","target_id":"S052","kind":"supports"},{"source_id":"S022","target_id":"S052","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S030","target_id":"S052","kind":"supports"},{"source_id":"S024","target_id":"S052","kind":"supports"},{"source_id":"S025","target_id":"S052","kind":"supports"},{"source_id":"S032","target_id":"S052","kind":"supports"},{"source_id":"S003","target_id":"S048","kind":"supports"},{"source_id":"S004","target_id":"S048","kind":"supports"},{"source_id":"S006","target_id":"S048","kind":"supports"},{"source_id":"S008","target_id":"S048","kind":"supports"},{"source_id":"S019","target_id":"S048","kind":"supports"},{"source_id":"S009","target_id":"S047","kind":"supports"},{"source_id":"S010","target_id":"S047","kind":"supports"},{"source_id":"S020","target_id":"S047","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S020","target_id":"S035","kind":"supports"},{"source_id":"S028","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S010","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S077","kind":"supports"},{"source_id":"S036","target_id":"S077","kind":"supports"},{"source_id":"S035","target_id":"S076","kind":"supports"},{"source_id":"S036","target_id":"S076","kind":"supports"},{"source_id":"S077","target_id":"S076","kind":"supports"},{"source_id":"S008","target_id":"S076","kind":"supports"},{"source_id":"S050","target_id":"S076","kind":"supports"},{"source_id":"S051","target_id":"S076","kind":"supports"},{"source_id":"S090","target_id":"S076","kind":"supports"},{"source_id":"S020","target_id":"S076","kind":"supports"},{"source_id":"S035","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S035","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S077","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S042","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S040","target_id":"S043","kind":"supports"},{"source_id":"S035","target_id":"S044","kind":"supports"},{"source_id":"S040","target_id":"S044","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S044","target_id":"S046","kind":"supports"},{"source_id":"S052","target_id":"S046","kind":"supports"},{"source_id":"S076","target_id":"S084","kind":"supports"},{"source_id":"S041","target_id":"S084","kind":"supports"},{"source_id":"S055","target_id":"S085","kind":"supports"},{"source_id":"S048","target_id":"S085","kind":"supports"},{"source_id":"S035","target_id":"S085","kind":"supports"},{"source_id":"S092","target_id":"S086","kind":"supports"},{"source_id":"S040","target_id":"S086","kind":"supports"},{"source_id":"S020","target_id":"S086","kind":"supports"},{"source_id":"S085","target_id":"S086","kind":"supports"},{"source_id":"S045","target_id":"S086","kind":"supports"},{"source_id":"S046","target_id":"S087","kind":"supports"},{"source_id":"S052","target_id":"S087","kind":"supports"},{"source_id":"S042","target_id":"S087","kind":"supports"},{"source_id":"S085","target_id":"S087","kind":"supports"},{"source_id":"S086","target_id":"S087","kind":"supports"},{"source_id":"S043","target_id":"S088","kind":"supports"},{"source_id":"S038","target_id":"S088","kind":"supports"},{"source_id":"S085","target_id":"S088","kind":"supports"},{"source_id":"S086","target_id":"S088","kind":"supports"},{"source_id":"S084","target_id":"S089","kind":"supports"},{"source_id":"S086","target_id":"S089","kind":"supports"},{"source_id":"S087","target_id":"S089","kind":"supports"},{"source_id":"S088","target_id":"S089","kind":"supports"},{"source_id":"S036","target_id":"S089","kind":"supports"},{"source_id":"S084","target_id":"S083","kind":"supports"},{"source_id":"S085","target_id":"S083","kind":"supports"},{"source_id":"S086","target_id":"S083","kind":"supports"},{"source_id":"S087","target_id":"S083","kind":"supports"},{"source_id":"S088","target_id":"S083","kind":"supports"},{"source_id":"S089","target_id":"S083","kind":"supports"},{"source_id":"S083","target_id":"S047","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:SDP"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"原始半定规划","source_units":[{"raw_tex":"Given $n\\times n$ real symmetric matrices $C,A_{1},\\ldots,A_{m}$\r\nand real scalars $b_{1},\\ldots,b_{m}$, we consider the standard form\r\nsemidefinite program \r\n\\begin{alignat}{2}\r\n & \\text{minimize } & C\\bullet X & \\tag{SDP}\\label{eq:SDP}\\\\\r\n & \\text{subject to } & A_{i}\\bullet X & =b_{i}\\quad\\text{for all }i\\in\\{1,\\ldots,m\\}\\nonumber \\\\\r\n &  & X & \\succeq0,\\nonumber \r\n\\end{alignat}\r\nover the $n\\times n$ real symmetric matrix variable $X$. Here, $A_{i}\\bullet X=\\tr A_{i}X$\r\nrefers to the usual matrix inner product, and $X\\succeq0$ restricts\r\nto be symmetric positive semidefinite. Instances of (\\ref{eq:SDP})\r\n","start":4930,"end":5545}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":["def:sparsitygraph"],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"数据稀疏图","source_units":[{"raw_tex":"To guarantee an exact reformulation, clique tree conversion chooses\r\nthe index sets $J_{1},\\ldots,J_{\\ell}$ in (\\ref{eq:relax}) as the\r\n\\emph{bags} of a \\emph{tree decomposition} for the \\emph{sparsity\r\ngraph} of the data matrices $C,A_{1},\\ldots,A_{m}$. Accordingly,\r\nthe parameter $\\omega$ in (\\ref{eq:omega_def-1}) can only be small\r\nif the sparsity graph has a small \\emph{treewidth}. Below, we define\r\na graph $G$ by its vertex set $V(G)\\subseteq\\{1,2,\\ldots,n\\}$ and\r\nits edge set $E(G)\\subseteq V(G)\\times V(G)$.\r\n\\begin{definition}[Sparsity graph]\r\n\\label{def:sparsitygraph}The $n\\times n$ matrix $M$ (resp. the\r\nset of $n\\times n$ matrices $\\{M_{1},\\ldots,M_{m}\\}$) is said to\r\nhave \\emph{sparsity graph} $G$ if $G$ is an undirected simple graph\r\non $n$ vertices $V(G)=\\{1,\\ldots,n\\}$ and that $(i,j)\\in E(G)$\r\nif $M[i,j]\\ne0$ (resp. if there exists $M\\in\\{M_{1},\\ldots,M_{m}\\}$\r\nsuch that $M[i,j]\\ne0$). \r\n\\end{definition}\r\n%\r\n","start":17649,"end":18586}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["def:treedecomp"],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"树分解与宽度","source_units":[{"raw_tex":"\\begin{definition}[Tree decomposition]\r\n\\label{def:treedecomp}A \\emph{tree decomposition} $\\T$ of a graph\r\n$G$ is a pair $(\\J,T)$, where each \\emph{bag} of vertices $J_{j}\\in\\J$\r\nis a subset of $V(G)$, and $T$ is a tree on $|\\J|\\le n$ vertices,\r\nsuch that:\r\n\\begin{enumerate}\r\n\\item (Vertex cover) For every $v\\in V(G)$, there exists $J_{k}\\in\\J$\r\nsuch that $v\\in J_{k}$;\r\n\\item (Edge cover) For every $(u,v)\\in E(G)$, there exists $J_{k}\\in\\J$\r\nsuch that $u\\in J_{k}$ and $v\\in J_{k}$; and\r\n\\item (Running intersection) If $v\\in J_{i}$ and $v\\in J_{j}$, then we\r\nalso have $v\\in J_{k}$ for every $k$ that lies on the path from\r\n$i$ to $j$ in the tree $T$.\r\n\\end{enumerate}\r\nThe \\emph{width} $\\wid(\\T)$ of the tree decomposition $\\T=(\\J,T)$\r\nis the size of its largest bag minus one, as in $\\max\\{|J_{k}|:J_{k}\\in\\J\\}-1.$\r\nThe \\emph{treewidth} $\\tw(G)$ of the graph $G$ is the minimum width\r\namongst all tree decompositions $\\T$.\r\n\\end{definition}\r\n","start":18586,"end":19535}]},{"id":"S005","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"预先给定树分解","source_units":[{"raw_tex":"Throughout this paper, we make the implicit assumption that a tree\r\ndecomposition with small width is known a priori for the sparsity\r\ngraph. In practice, such a tree decomposition can usually be found\r\nusing fill-reducing heuristics for sparse linear algebra; see Section~\\ref{sec:Preliminaries}.\r\n\r\nWe also make two explicit assumptions, which are standard in the literature\r\non interior-point methods.\r\n\r\n","start":19535,"end":19943}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","labels":["ass:lin"],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"约束线性无关","source_units":[{"raw_tex":"\\begin{assume}[Linear independence]\\label{ass:lin}We have $\\sum_{i=1}^{m}y_{i}A_{i}=0$\r\nif and only if $y=0$.\r\n\r\n\\end{assume}\r\n","start":19943,"end":20070}]},{"id":"S007","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"原始约束数上界","source_units":[{"raw_tex":"\r\nThe assumption is without loss of generality, because it can either\r\nbe enforced by eliminating $A_{i}\\bullet X=b_{i}$ for select $i$,\r\nor else these constraints are not consistent for all $i$. Under Assumption~\\ref{ass:lin},\r\nthe total number of constraints is bounded $m\\le\\omega n$ (due to\r\nthe fact that $|E(G)|\\le n\\cdot\\tw(G)$~\\cite{arnborg1987complexity}).\r\n\r\n","start":20070,"end":20439}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","labels":["ass:slater"],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"原始双侧严格可行","source_units":[{"raw_tex":"\\begin{assume}[Primal-dual Slater's condition]\\label{ass:slater}There\r\nexist $X\\succ0,$ $y,$ and $S\\succ0$ satisfying $A_{i}\\bullet X=b_{i}$\r\nfor all $i\\in\\{1,\\ldots,m\\}$ and $\\sum_{i=1}^{m}y_{i}A_{i}+S=C$. \r\n\r\n\\end{assume}\r\n","start":20439,"end":20664}]},{"id":"S009","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"矩阵与范数记号","source_units":[{"raw_tex":"The sets $\\R^{n}$ and $\\R^{m\\times n}$ are the length-$n$ real\r\nvectors and $m\\times n$ real matrices. We use ``MATLAB notation''\r\nin concatenating vectors and matrices:\r\n\r\n\\[\r\n[a,b]=\\begin{bmatrix}a & b\\end{bmatrix},\\qquad[a;b]=\\begin{bmatrix}a\\\\\r\nb\r\n\\end{bmatrix},\\qquad\\diag(a,b)=\\begin{bmatrix}a & 0\\\\\r\n0 & b\r\n\\end{bmatrix},\r\n\\]\r\nand the following short-hand to construct them:\r\n\\[\r\n[x_{i}]_{i=1}^{n}=\\begin{bmatrix}x_{1}\\\\\r\n\\vdots\\\\\r\nx_{n}\r\n\\end{bmatrix},\\qquad[x_{i,j}]_{i,j=1}^{m,n}=\\begin{bmatrix}x_{1,1} & \\cdots & x_{1,n}\\\\\r\n\\vdots & \\ddots & \\vdots\\\\\r\nx_{m,1} & \\cdots & x_{m,n}\r\n\\end{bmatrix}.\r\n\\]\r\nThe notation $X[i,j]$ refers to the element of $X$ in the $i$-th\r\nrow and $j$-th column, and $X[I,J]$ refers to the submatrix of $X$\r\nformed from the rows in $I\\subseteq\\{1,\\ldots,m\\}$ and columns in\r\n$J\\subseteq\\{1,\\ldots,n\\}$. The Frobenius inner product is $X\\bullet Y=\\tr(X^{T}Y)$,\r\nand the Frobenius norm is $\\|X\\|_{F}=\\sqrt{X\\bullet X}$. We use $\\nnz(X)$\r\nto denote the number of nonzero elements in $X$. \r\n\r\n","start":30594,"end":31620}]},{"id":"S010","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"对称锥与向量化","source_units":[{"raw_tex":"The sets $\\S^{n}\\subseteq\\R^{n\\times n},$ $\\S_{+}^{n}\\subset\\S^{n},$\r\nand $\\S_{++}^{n}\\subset\\S_{+}^{n}$ are the $n\\times n$ real symmetric\r\nmatrices, positive semidefinite matrices, and positive definite matrices,\r\nrespective. We write $X\\succeq Y$ to mean $X-Y\\in\\S_{+}^{n}$ and\r\n$X\\succ Y$ to mean $X-Y\\in\\S_{++}^{n}$. The (symmetric) vectorization\r\n\\[\r\n\\vector(X)=[X[1,1];\\sqrt{2}X[2,1];\\ldots;\\sqrt{2}X[m,1];X[2,2],\\ldots]\r\n\\]\r\noutputs the lower-triangular part of a symmetric matrix as a vector,\r\nwith factors of $\\sqrt{2}$ added so that $\\vector(X)^{T}\\vector(Y)=X\\bullet Y$. \r\n\r\n","start":31620,"end":32207}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"根树与父指针","source_units":[{"raw_tex":"A graph $G$ is defined by its vertex set $V(G)\\subseteq\\{1,2,3,\\ldots\\}$\r\nand its edge set $E(G)\\subseteq V(G)\\times V(G)$. The graph $T$\r\nis a \\emph{tree} if it is connected and does not contain any cycles;\r\nwe refer to its vertices $V(T)$ as its \\emph{nodes}. Designating\r\na special node $r\\in V(T)$ as the \\emph{root }of the tree allows\r\nus to define the \\emph{parent} $p(v)$ of each node $v\\ne r$ as the\r\nfirst node encountered on the path from $v$ to $r$, and $p(r)=r$\r\nfor consistency. The set of children is defined $\\ch(v)=\\{u\\in V(T)\\backslash v:p(u)=v\\}$.\r\nNote that the edges $E(T)$ are fully determined by the parent pointer\r\n$p$ as $\\{v,p(v)\\}$ for all $v\\ne r$.\r\n\r\n","start":32207,"end":32886}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"稀疏矩阵投影","source_units":[{"raw_tex":"The set $\\S_{G}^{n}\\subseteq\\S^{n}$ is the set of $n\\times n$ real\r\nsymmetric matrices with sparsity graph $G$. We denote $P_{G}(X)=\\min_{Y\\in\\S_{G}^{n}}\\|X-Y\\|_{F}$\r\nas the Euclidean projection of $X\\in\\S^{n}$ onto $\\S_{G}^{n}$.\r\n","start":32886,"end":33117}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["eq:chordal_compl"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"由袋构造弦图","source_units":[{"raw_tex":"Let $\\T=(\\{J_{1},\\ldots,J_{\\ell}\\},T)$ be a tree decomposition with\r\nsmall width for the sparsity graph $G$ of the data matrices $C,A_{1},\\ldots,A_{m}$.\r\nWe define the graph $F\\supseteq G$ by taking each index set $J_{j}$\r\nof $\\T$ and interconnecting all pairs of vertices $u,v\\in J_{j}$,\r\nas in \r\n\\begin{align}\r\nV(F) & =V(G), & E(F) & =\\bigcup_{j=1}^{\\ell}\\{(u,v):u,v\\in J_{j}\\}.\\label{eq:chordal_compl}\r\n\\end{align}\r\nThe following fundamental result was first established by Grone et\r\nal.~\\cite{grone1984positive}. Constructive proofs allow us to recover\r\nall elements in $X\\succeq0$ from only the elements in $P_{F}(X)$\r\nusing a closed-form formula.\r\n","start":35830,"end":36484}]},{"id":"S014","semantic_type":"external_fact","kind":"external_fact","labels":["thm:prim_sep"],"member_ids":["S014"],"declared_external":true,"declaration_text":["原始前提列表允许的 B0045；Grone et al., grone1984positive。"],"source_only":false,"title":"半正定补全定理","source_units":[{"raw_tex":"\\begin{theorem}[Grone et al.~\\cite{grone1984positive}]\r\n\\label{thm:prim_sep}Given $Z\\in\\S_{F}^{n}$, there exists an $X\\succeq0$\r\nsatisfying $P_{F}(X)=Z$ if and only if $Z[J_{j},J_{j}]\\succeq0$\r\nfor all $j\\in\\{1,2,\\ldots,\\ell\\}$.\r\n\\end{theorem}\r\n","start":36484,"end":36729}]},{"id":"S015","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:CSDP"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"投影后的等价问题","source_units":[{"raw_tex":"We can use Theorem~\\ref{thm:prim_sep} to reformulate (\\ref{eq:SDP})\r\ninto a reduced-complexity form. The key is to view (\\ref{eq:SDP})\r\nas an optimization over $P_{F}(X)$, since \r\n\\[\r\nC\\bullet X=\\sum_{i,j=1}^{n}C_{i,j}X_{i,j}=\\sum_{(i,j)\\in F}C_{i,j}X_{i,j}=C\\bullet P_{F}(X),\r\n\\]\r\nand similarly $A_{i}\\bullet X=A_{i}\\bullet P_{F}(X)$. Theorem~\\ref{thm:prim_sep}\r\nallows us to account for $X\\succeq0$ implicitly, by optimizing over\r\n$Z=P_{F}(X)$ in the following\r\n\\begin{alignat}{3}\r\n & \\text{minimize }\\quad & C\\bullet Z\\label{eq:CSDP}\\\\\r\n & \\text{subject to } & A_{i}\\bullet Z & =b_{i} & \\qquad\\text{for all } & i\\in\\{1,\\ldots,m\\},\\nonumber \\\\\r\n &  & Z[J_{j},J_{j}] & \\succeq0 & \\text{for all } & j\\in\\{1,\\ldots,\\ell\\}.\\nonumber \r\n\\end{alignat}\r\n","start":36729,"end":37477}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"重叠子矩阵算子","source_units":[{"raw_tex":"Next, we split the principal submatrices into distinct matrix variables,\r\ncoupled by the need for their overlapping elements to agree. Define\r\nthe \\emph{overlap} operator $\\NN_{i,j}(\\cdot)$ to output the overlapping\r\nelements of two principal submatrices given the latter as input:\r\n\\[\r\n\\NN_{i,j}(X[J_{j},J_{j}])=X[J_{i}\\cap J_{j},\\;J_{i}\\cap J_{j}]=\\NN_{j,i}(X[J_{i},J_{i}]).\r\n\\]\r\nThe running intersection property of the tree decomposition allows\r\nus to enforce this agreement using $\\ell-1$ pairwise block comparisons.\r\n","start":37477,"end":38000}]},{"id":"S017","semantic_type":"external_fact","kind":"external_fact","labels":["thm:ctree"],"member_ids":["S017"],"declared_external":true,"declaration_text":["原始前提列表允许的 B0049；Fukuda et al., fukuda2001exploiting。"],"source_only":false,"title":"树边重叠一致性","source_units":[{"raw_tex":"\\begin{theorem}[Fukuda et al.~\\cite{fukuda2001exploiting}]\r\n\\label{thm:ctree}Given $X_{1},X_{2},\\ldots,X_{\\ell}$ for $X_{j}\\in\\S^{|J_{j}|}$,\r\nthere exists $Z$ satisfying $Z[J_{j},J_{j}]=X_{j}$ for all $j\\in\\{1,2,\\ldots,\\ell\\}$\r\nif and only if $\\NN_{i,j}(X_{j})=\\NN_{j,i}(X_{i})$ for all $(i,j)\\in E(T)$. \r\n\\end{theorem}\r\n","start":38000,"end":38321}]},{"id":"S018","semantic_type":"assumption","kind":"assumption","labels":["eq:split_constr"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"数据拆分恒等式","source_units":[{"raw_tex":"Splitting the objective $C$ and constraint matrices $A_{1},\\ldots,A_{m}$\r\ninto $C_{1},\\ldots,C_{\\ell}$ and $A_{1,1},\\ldots,A_{m,\\ell}$ to\r\nsatisfy \r\n\\begin{align}\r\nC_{1}\\bullet X[J_{1},J_{1}]+C_{2}\\bullet X[J_{2},J_{2}]+\\cdots+C_{\\ell}\\bullet X[J_{\\ell},J_{\\ell}] & =C\\bullet X,\\label{eq:split_constr}\\\\\r\nA_{i,1}\\bullet X[J_{1},J_{1}]+A_{i,2}\\bullet X[J_{2},J_{2}]+\\cdots+A_{i,\\ell}\\bullet X[J_{\\ell},J_{\\ell}] & =A_{i}\\bullet X,\\nonumber \r\n\\end{align}\r\n","start":38321,"end":38775}]},{"id":"S019","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:CTC"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"团树转换问题","source_units":[{"raw_tex":"and applying Theorem~\\ref{thm:ctree} yields the following\r\n\\begin{alignat}{3}\r\n & \\text{minimize }\\quad & \\sum_{j=1}^{\\ell}C_{j}\\bullet X_{j} &  &  & \\tag{CTC}\\label{eq:CTC}\\\\\r\n & \\text{subject to } & \\sum_{j=1}^{\\ell}A_{i,j}\\bullet X_{j} & =b_{i} & \\text{for all } & i\\in\\{1,\\ldots,m\\},\\nonumber \\\\\r\n &  & \\NN_{i,j}(X_{j}) & =\\NN_{j,i}(X_{i}) & \\qquad\\text{for all } & (i,j)\\in E(T),\\nonumber \\\\\r\n &  & X_{j} & \\succeq0 & \\text{for all } & j\\in\\{1,\\ldots,\\ell\\},\\nonumber \r\n\\end{alignat}\r\n","start":38775,"end":39265}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":["eq:vecCTC"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"向量化原对偶与乘积锥","source_units":[{"raw_tex":"which vectorizes into a linear conic program in standard form\r\n\\begin{alignat}{4}\r\n & \\text{minimize } & c^{T}x, &  & \\qquad\\qquad & \\text{maximize } & \\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}^{T}y,\\label{eq:vecCTC}\\\\\r\n & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}x & =\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}, &  & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}y+s & =c,\\nonumber \\\\\r\n &  & x & \\in\\K, &  &  & s & \\in\\K_{*}\\nonumber \r\n\\end{alignat}\r\nover the Cartesian product of $\\ell\\le n$ smaller semidefinite cones\r\n\\begin{equation}\r\n\\K=\\K_{*}=\\S_{+}^{|J_{1}|}\\times\\S_{+}^{|J_{2}|}\\times\\cdots\\times\\S_{+}^{|J_{\\ell}|}.\r\n\\end{equation}\r\n","start":39265,"end":39933}]},{"id":"S021","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"拆分数据坐标","source_units":[{"raw_tex":"Here, $\\A=[\\vector(A_{i,j})^{T}]_{i,j=1}^{m,\\ell}$ and $c=[\\vector(C_{j})]_{j=1}^{\\ell}$\r\ncorrespond to (\\ref{eq:split_constr}), and the \\emph{overlap constraints}\r\nmatrix $\\N=[\\N_{i,j}]_{i,j=1}^{\\ell,\\ell}$ is implicitly defined\r\nby the relation\r\n","start":39933,"end":40181}]},{"id":"S022","semantic_type":"definition","kind":"definition","labels":["eq:Nmat"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"重叠矩阵的树边结构","source_units":[{"raw_tex":"\\begin{equation}\r\n\\N_{i,j}\\vector(X_{j})=\\begin{cases}\r\n+\\vector(\\NN_{p(i),i}(X_{i})) & j=i,\\\\\r\n-\\vector(\\NN_{i,p(i)}(X_{p(i)})) & j=p(i),\\\\\r\n0 & \\text{otherwise},\r\n\\end{cases}\\label{eq:Nmat}\r\n\\end{equation}\r\nfor every non-root node $i$ on $T$. (To avoid all-zero rows in $\\N$,\r\nwe define $\\N_{i,j}\\,\\vector(X_{j})$ as the empty length-zero vector\r\n$\\R^{0}$ if $i$ is the root node.)\r\n","start":40181,"end":40566}]},{"id":"S023","semantic_type":"assumption","kind":"assumption","labels":["eq:scal"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"块对角正定缩放","source_units":[{"raw_tex":"where the \\emph{scaling matrix} $\\D_{s}$ is block-diagonal with fully-dense\r\nblocks \r\n\\begin{equation}\r\n\\D_{s}=\\diag(\\D_{s,1},\\ldots,\\D_{s,\\ell}),\\qquad\\D_{s,j}\\succ0\\quad\\text{for all }j\\in\\{1,\\ldots,\\ell\\}.\\label{eq:scal}\r\n\\end{equation}\r\n","start":43294,"end":43535}]},{"id":"S024","semantic_type":"definition","kind":"definition","labels":["def:topological"],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"子先父后的拓扑序","source_units":[{"raw_tex":"\\begin{definition}[Topological ordering]\r\n\\label{def:topological}An ordering\\emph{ }$\\pi:\\{1,2,\\ldots,n\\}\\to V(T)$\r\non the tree graph $T$ with $n$ nodes is said to be \\emph{topological}~\\cite[p.~10]{vandenberghe2015chordal}\r\nif, by designating $\\pi(n)$ as the root of $T$, each node is indexed\r\nbefore its parent:\r\n\\[\r\n\\pi^{-1}(v)<\\pi^{-1}(p(v))\\qquad\\text{ for all }v\\ne r,\r\n\\]\r\nwhere $\\pi^{-1}(v)$ denotes the index associated with the node $v$.\r\n\\end{definition}\r\n","start":46596,"end":47063}]},{"id":"S025","semantic_type":"external_fact","kind":"external_fact","labels":["lem:blocktree"],"member_ids":["S025"],"declared_external":true,"declaration_text":["原始前提列表允许的 B0068；Parter, parter1961use；George–Liu, george1981computer, Lemma 6.3.1。"],"source_only":false,"title":"树块消元无填充","source_units":[{"raw_tex":"\\begin{lemma}[No block fill]\r\n\\label{lem:blocktree}Let $J_{1},\\ldots,J_{n}$ satisfy $\\bigcup_{j=1}^{n}J_{j}=\\{1,\\ldots,d\\}$\r\nand $J_{i}\\cap J_{j}=\\emptyset$ for all $i\\ne j$, and let $H\\succ0$\r\nbe a $d\\times d$ matrix satisfying\r\n\\[\r\nH[J_{i},J_{j}]\\ne0\\qquad\\implies\\qquad(i,j)\\in E(T)\r\n\\]\r\nfor a tree graph $T$ on $n$ nodes. If $\\pi$ is a topological ordering\r\non $T$ and $\\Pi$ is a permutation matrix satisfying \r\n\\[\r\n(\\Pi H\\Pi)[J_{i},J_{j}]=H[J_{\\pi(i)},J_{\\pi(j)}]\\qquad\\text{for all }i,j\\in\\{1,\\ldots,n\\},\r\n\\]\r\nthen $\\Pi H\\Pi^{T}$ factors into $LL^{T}$ where the Cholesky factor\r\n$L$ satisfies\r\n\\[\r\nL[J_{i},J_{j}]\\ne0\\qquad\\implies\\qquad(\\Pi H\\Pi)[J_{i},J_{j}]\\ne0\\qquad\\text{ for all }i>j.\r\n\\]\r\nTherefore, sparse Cholesky factorization solves $Hx=b$ for $x$ by:\r\n(i) factoring $\\Pi H\\Pi^{T}$ into $LL^{T}$ in $O(\\beta^{3}n)$ operations\r\nand $O(\\beta^{2}n)$ memory where $\\beta=\\max_{j}|J_{j}|$, and (ii)\r\nsolving $Ly=\\Pi b$ and $L^{T}z=y$ and $x=\\Pi^{T}z$ in $O(\\beta^{2}n)$\r\noperations and memory. \r\n\\end{lemma}\r\nThis is a simple block-wise extension of the tree elimination result\r\noriginally due to Parter~\\cite{parter1961use}; see also George and\r\nLiu~\\cite[Lemma 6.3.1]{george1981computer}. In practice, a topological\r\n","start":47063,"end":48293}]},{"id":"S026","semantic_type":"algorithm","kind":"algorithm","labels":[],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"拓扑序的实际构造","source_units":[{"raw_tex":"ordering can be found by assigning indices $n,n-1,n-2,\\ldots$ in\r\ndecreasing ordering during a depth-first search traversal of the tree.\r\nIn fact, the minimum degree heuristic is guaranteed to generate a\r\ntopological ordering~\\cite{george1989evolution}.\r\n","start":48293,"end":48548}]},{"id":"S027","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:dualCTC"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"交换原对偶角色","source_units":[{"raw_tex":"The dualization technique of Löfberg~\\cite{lofberg2009dualize} swaps\r\nthe roles played by the primal and the dual problems in a linear conic\r\nprogram, by rewriting a primal standard form problem into dual standard\r\nform, and vice versa. Applying dualization to (\\ref{eq:vecCTC}) yields\r\nthe following\r\n\\begin{alignat}{4}\r\n & \\text{minimize } & \\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}^{T}x_{1} &  & \\qquad & \\text{maximize } & -c^{T}y\\label{eq:dualCTC}\\\\\r\n & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}x_{1}-x_{2} & =-c, &  & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}y+s_{1} & =\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix},\\nonumber \\\\\r\n &  & x_{1}\\in\\R^{f},\\,x_{2} & \\in\\K. &  &  & -y+s_{2} & =0,\\nonumber \\\\\r\n &  &  &  &  &  & s_{1}\\in\\{0\\}^{f},\\,s_{2} & \\in\\K.\\nonumber \r\n\\end{alignat}\r\nwhere we use $f$ to denote the number of equality constraints in\r\n(\\ref{eq:CTC}). Observe that the dual problem in (\\ref{eq:dualCTC})\r\nis identical to the primal problem in (\\ref{eq:vecCTC}), so that\r\na dual solution $y^{\\star}$ to (\\ref{eq:dualCTC}) immediately serves\r\nas a primal solution to (\\ref{eq:vecCTC}), and hence also (\\ref{eq:CTC}).\r\n\r\n","start":50272,"end":51436}]},{"id":"S028","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:dualCTC-2"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"自由变量的二阶锥嵌入","source_units":[{"raw_tex":"Modern interior-point methods solve (\\ref{eq:dualCTC}) by embeding\r\nthe free variable $x_{1}\\in\\R^{f}$ and fixed variable $s_{1}\\in\\{0\\}^{f}$\r\ninto a second-order cone (see Sturm~\\cite{sturm1999using} and Andersen~\\cite{andersen2002handling}):\r\n\\begin{alignat}{4}\r\n & \\text{minimize } & \\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}^{T}x_{1} &  & \\qquad & \\text{maximize } & -c^{T}y\\label{eq:dualCTC-2}\\\\\r\n & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}x_{1}-x_{2} & =-c, &  & \\text{subject to } & \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}y+s_{1} & =\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix},\\nonumber \\\\\r\n &  & \\|x_{1}\\|\\le x_{0},\\,x_{2} & \\in\\K. &  &  & -y+s_{2} & =0,\\nonumber \\\\\r\n &  &  &  &  &  & s_{0} & =0,\\nonumber \\\\\r\n &  &  &  &  &  & \\|s_{1}\\|\\le s_{0},\\,s_{2} & \\in\\K.\\nonumber \r\n\\end{alignat}\r\n","start":51436,"end":52245}]},{"id":"S029","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:Hxr_dual","eq:scal_f"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"对偶正常方程结构","source_units":[{"raw_tex":"When (\\ref{eq:dualCTC-2}) is solved using an interior-point method,\r\nthe normal equation solved at each iteration takes the form\r\n\\begin{equation}\r\n\\left(\\D_{s}+\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}\\D_{f}\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}\\right)\\Delta y=r\\label{eq:Hxr_dual}\r\n\\end{equation}\r\nwhere $\\D_{s}$ is comparable as before in (\\ref{eq:scal}), and \r\n\\begin{equation}\r\n\\D_{f}=\\sigma I+ww^{T},\\qquad\\sigma>0\\label{eq:scal_f}\r\n\\end{equation}\r\n","start":52245,"end":52701}]},{"id":"S030","semantic_type":"definition","kind":"definition","labels":["eq:Hqdef"],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"稀疏基矩阵与秩一向量","source_units":[{"raw_tex":"is the rank-1 perturbation of a scaled identity matrix. The standard\r\nprocedure, as implemented in SeDuMi~\\cite{sturm1999using,sturm2002implementation}\r\nand MOSEK~\\cite{andersen2003implementing}, is to form the sparse\r\nmatrix $\\H$ and dense vector $\\q$, defined\r\n\\begin{align}\r\n\\H & =\\D_{s}+\\sigma\\A^{T}\\A+\\sigma\\N^{T}\\N, & \\q & =\\begin{bmatrix}\\A^{T} & \\N^{T}\\end{bmatrix}w.\\label{eq:Hqdef}\r\n\\end{align}\r\n","start":52701,"end":53107}]},{"id":"S032","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:update"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"两次求解完成秩一更新","source_units":[{"raw_tex":"\\begin{equation}\r\n\\Delta y=(\\H+\\q\\q^{T})^{-1}r=\\left(I-\\frac{(\\H^{-1}\\q)\\q^{T}}{1+\\q^{T}(\\H^{-1}\\q)}\\right)\\H^{-1}r,\\label{eq:update}\r\n\\end{equation}\r\nat a cost comparable to the solution of $\\H u=r$ for two right-hand\r\nsides. (In Appendix~\\ref{sec:inequality}, we repeat these derivations\r\nfor the version of (\\ref{eq:vecCTC}) in which $\\A x=b$ is replaced\r\nby the inequality $\\A x\\le b$.)\r\n","start":53690,"end":54082}]},{"id":"S033","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"原始约束矩阵化","source_units":[{"raw_tex":"Define the $m\\times\\frac{1}{2}n(n+1)$ matrix $\\M$ as the vectorization\r\nof the linear constraints in (\\ref{eq:SDP}), as in\r\n\\[\r\n\\M\\,\\vector(X)=\\begin{bmatrix}\\vector(A_{1})^{T}\\\\\r\n\\vdots\\\\\r\n\\vector(A_{m})^{T}\r\n\\end{bmatrix}\\vector(X)=\\begin{bmatrix}\\vector(A_{1})^{T}\\vector(X)\\\\\r\n\\vdots\\\\\r\n\\vector(A_{m})^{T}\\vector(X)\r\n\\end{bmatrix}=\\begin{bmatrix}A_{1}\\bullet X\\\\\r\n\\vdots\\\\\r\nA_{m}\\bullet X\r\n\\end{bmatrix}.\r\n\\]\r\n","start":109481,"end":109895}]},{"id":"S034","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:split_constr2"],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"拆分的算子恒等式","source_units":[{"raw_tex":"In reformulating (\\ref{eq:SDP}) into (\\ref{eq:CTC}), the splitting\r\nconditions (\\ref{eq:split_constr}) can be rewritten as the following\r\n\\begin{align}\r\nc^{T}\\P & =\\vector(C)^{T}, & \\A\\P & =\\M,\\label{eq:split_constr2}\r\n\\end{align}\r\nwhere $c=[\\vector(C_{j})]_{j=1}^{\\ell}$ and $\\A=[\\vector(A_{i,j})^{T}]_{i,j=1}^{m,\\ell}$\r\nare the data for the vectorized verison of (\\ref{eq:CTC}).\r\n","start":109895,"end":110277}]},{"id":"S035","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:hsdcone","eq:hsdfeas"],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"扩展齐次自对偶嵌入","source_units":[{"raw_tex":"We solve the dualized problem (\\ref{eq:dualCTC-2}) by solving its\r\nextended homogeneous self-dual embedding\\begin{subequations}\\label{eq:hsd}\r\n\\begin{alignat}{2}\r\n & \\text{min.}\\quad & (\\nu+1)\\theta\\\\\r\n & \\text{s.t.} & \\begin{bmatrix}0 & +\\M^{T} & -\\c & -r_{d}\\\\\r\n-\\M & 0 & +\\b & -r_{p}\\\\\r\n+\\c^{T} & -\\b^{T} & 0 & -r_{c}\\\\\r\nr_{d}^{T} & r_{p}^{T} & r_{c} & 0\r\n\\end{bmatrix}\\begin{bmatrix}x\\\\\r\ny\\\\\r\n\\tau\\\\\r\n\\theta\r\n\\end{bmatrix}+\\begin{bmatrix}s\\\\\r\n0\\\\\r\n\\kappa\\\\\r\n0\r\n\\end{bmatrix} & =\\begin{bmatrix}0\\\\\r\n0\\\\\r\n0\\\\\r\n\\nu+1\r\n\\end{bmatrix}\\label{eq:hsdfeas}\\\\\r\n &  & x,s\\in\\C\\quad\\kappa,\\tau & \\ge0,\\label{eq:hsdcone}\r\n\\end{alignat}\r\nwhere the data is given in standard form\r\n\\begin{gather}\r\n\\M=\\begin{bmatrix}0 & -\\begin{bmatrix}\\A^{T} & \\N^{T}\\end{bmatrix} & +I\\end{bmatrix},\\quad\\c^{T}=\\begin{bmatrix}0 & \\begin{bmatrix}b^{T} & 0\\end{bmatrix} & 0\\end{bmatrix},\\quad\\b=-c,\\\\\r\n\\C=\\{(x_{0},x_{1}\\}\\in\\R^{p+1}:\\|x_{1}\\|\\le x_{0}\\}\\times\\K,\r\n\\end{gather}\r\n","start":115490,"end":116437}]},{"id":"S036","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"嵌入残差与锥阶","source_units":[{"raw_tex":"and the residual vectors are defined\r\n\\begin{equation}\r\nr_{d}=\\one_{\\C}-\\c,\\quad r_{p}=-\\M\\one_{\\C}+\\b,\\quad r_{c}=1+\\c^{T}\\one_{\\C}.\r\n\\end{equation}\r\n\\end{subequations}Here, $\\nu$ is the order of the cone $\\C$, and\r\n$\\one_{\\C}$ is its identity element\r\n\\begin{align}\r\n\\nu & =1+|J_{1}|+\\cdots+|J_{\\ell}|, & \\one_{\\C} & =[1;0;\\ldots;0;\\vector(I_{|J_{1}|});\\ldots;\\vector(I_{|J_{\\ell}|})].\r\n\\end{align}\r\n","start":116437,"end":116839}]},{"id":"S038","semantic_type":"algorithm","kind":"algorithm","labels":["eq:hsdinit"],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"短步初始化与迭代","source_units":[{"raw_tex":"We solve the homogeneous self-dual embedding (\\ref{eq:hsd}) using\r\nthe short-step method of Nesterov and Todd~\\cite[Algorithm~6.1]{nesterov1998primal}\r\n(and also Sturm and S.~Zhang~\\cite[Section~5.1]{sturm1999symmetric}),\r\nnoting that SeDuMi reduces to it in the worst case; see~\\cite{sturm2002implementation}\r\nand~\\cite{sturm1997wide}. Beginning at the following strictly feasible,\r\nperfectly centered point\r\n\\begin{gather}\r\n\\theta^{(0)}=\\tau^{(0)}=\\kappa^{(0)}=1,\\quad y^{(0)}=0,\\quad x^{(0)}=s^{(0)}=\\one_{\\C},\\label{eq:hsdinit}\r\n\\end{gather}\r\nwith barrier parameter $\\mu=1$, we take the following steps\\begin{subequations}\\label{eq:hsd_ipm}\r\n\\begin{align}\r\n\\mu^{+} & =\\left(1-\\frac{1}{15\\sqrt{\\nu+1}}\\right)\\cdot\\frac{x^{T}s+\\tau\\kappa}{\\nu+1},\\\\\r\n(x^{+},y^{+},s^{+},\\tau^{+},\\theta^{+},\\kappa^{+}) & =(x,y,s,\\tau,\\theta,\\kappa)+(\\Delta x,\\Delta y,\\Delta s,\\Delta\\tau,\\Delta\\theta,\\Delta\\kappa).\\nonumber \r\n\\end{align}\r\n\\end{subequations}along the search direction defined by the linear\r\n","start":120060,"end":121052}]},{"id":"S039","semantic_type":"algorithm","kind":"algorithm","labels":[],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"缩放 Newton 搜索方向","source_units":[{"raw_tex":"system~\\cite[Eqn. 9]{todd1998nesterov} \\begin{subequations}\\label{eq:hsd_nt}\r\n\\begin{align}\r\n\\begin{bmatrix}0 & +\\M^{T} & -\\c & -r_{p}\\\\\r\n-\\M & 0 & +\\b & -r_{d}\\\\\r\n+\\c^{T} & -\\b^{T} & 0 & -r_{c}\\\\\r\n+r_{p}^{T} & +r_{d}^{T} & +r_{c} & 0\r\n\\end{bmatrix}\\begin{bmatrix}\\Delta x\\\\\r\n\\Delta y\\\\\r\n\\Delta\\tau\\\\\r\n\\Delta\\theta\r\n\\end{bmatrix}+\\begin{bmatrix}\\Delta s\\\\\r\n0\\\\\r\n\\Delta\\kappa\\\\\r\n0\r\n\\end{bmatrix} & =0,\\\\\r\ns+\\Delta s+\\nabla^{2}F(w)\\Delta x+\\mu^{+}\\nabla F(x) & =0,\\\\\r\n\\kappa+\\Delta\\kappa+(\\kappa/\\tau)\\Delta\\tau-\\mu^{+}\\tau^{-1} & =0.\r\n\\end{align}\r\n","start":121052,"end":121599}]},{"id":"S040","semantic_type":"definition","kind":"definition","labels":["eq:Fdef"],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"障碍函数与缩放点","source_units":[{"raw_tex":"\\end{subequations}Here, $F$ is the usual self-concordant barrier\r\nfunction on $\\C$\r\n\\begin{align}\r\nF([x_{0};x_{1};\\vector(X_{1});\\ldots;\\vector(X_{\\ell})])= & -\\log\\left(\\frac{1}{2}x_{0}^{2}-\\frac{1}{2}x_{1}^{T}x_{1}\\right)-\\sum_{j=1}^{\\ell}\\log\\det(X_{j})\\label{eq:Fdef}\r\n\\end{align}\r\nand $w\\in\\Int(\\C)$ is the unique \\emph{scaling point} satisfying\r\n$\\nabla^{2}F(w)x=s$, which can be computed from $x$ and $s$ in\r\nclosed-form. The following iteration bound is an immediate consequence\r\nof~\\cite[Theorem~6.4]{nesterov1998primal}; see also~\\cite[Theorem~5.1]{sturm1999symmetric}.\r\n","start":121599,"end":122180}]},{"id":"S041","semantic_type":"external_fact","kind":"external_fact","labels":["lem:short-step"],"member_ids":["S041"],"declared_external":true,"declaration_text":["原始前提列表允许的 B0177；作者在 2283–2284 行归于 Nesterov–Todd, nesterov1998primal, Theorem 6.4，及 Sturm–Zhang, sturm1999symmetric, Theorem 5.1。"],"source_only":false,"title":"短步迭代次数界","source_units":[{"raw_tex":"\\begin{lemma}[Short-Step Method]\r\n\\label{lem:short-step}The sequence in (\\ref{eq:hsd_ipm}) arrives\r\nat an iterate $(x,y,s,\\tau,\\theta,\\kappa)$ satisfying the conditions\r\nof Lemma~\\ref{lem:eps} with $\\gamma=9/10$ in at most $O(\\sqrt{\\nu}\\log(1/\\epsilon))$\r\niterations. \r\n\\end{lemma}\r\n","start":122180,"end":122463}]},{"id":"S042","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:hsd_nrm"],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"三个右端的正常方程","source_units":[{"raw_tex":"The cost of each interior-point iteration is dominated by the cost\r\nof computing the search direction in (\\ref{eq:hsd_nt}). Using elementary\r\nbut tedious linear algebra, we can show that if\\begin{subequations}\\label{eq:newt}\r\n\\begin{equation}\r\n(\\M\\D^{-1}\\M^{T})\\begin{bmatrix}v_{1} & v_{2} & v_{3}\\end{bmatrix}=\\begin{bmatrix}0 & -\\b & r_{p}\\end{bmatrix}-\\M\\D^{-1}\\begin{bmatrix}d & \\c & r_{d}\\end{bmatrix}\\label{eq:hsd_nrm}\r\n\\end{equation}\r\nwhere $\\D=\\nabla^{2}F(w)$ and $d=-s-\\mu^{+}\\nabla F(x)$, and \r\n","start":122463,"end":122968}]},{"id":"S043","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:hsd_back1","eq:hsd_back2"],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"搜索方向的回代公式","source_units":[{"raw_tex":"\\begin{equation}\r\n\\begin{bmatrix}u_{1} & u_{2} & u_{3}\\end{bmatrix}=\\D^{-1}(\\begin{bmatrix}d & c & r_{d}\\end{bmatrix}+\\M^{T}\\begin{bmatrix}v_{1} & v_{2} & v_{3}\\end{bmatrix}),\\label{eq:hsd_back1}\r\n\\end{equation}\r\nthen\r\n\\begin{align}\r\n\\left(\\begin{bmatrix}-\\D_{0} & -r_{c}\\\\\r\nr_{c} & 0\r\n\\end{bmatrix}-\\begin{bmatrix}\\c & r_{d}\\\\\r\n-\\b & r_{p}\r\n\\end{bmatrix}^{T}\\begin{bmatrix}u_{2} & u_{3}\\\\\r\nv_{2} & v_{3}\r\n\\end{bmatrix}\\right)\\begin{bmatrix}\\Delta\\tau\\\\\r\n\\Delta\\theta\r\n\\end{bmatrix} & =\\begin{bmatrix}-d_{0}\\\\\r\n0\r\n\\end{bmatrix}-\\begin{bmatrix}\\c & r_{d}\\\\\r\n-\\b & r_{p}\r\n\\end{bmatrix}^{T}\\begin{bmatrix}u_{1}\\\\\r\nv_{1}\r\n\\end{bmatrix},\\\\\r\n\\begin{bmatrix}\\Delta x\\\\\r\n\\Delta y\r\n\\end{bmatrix} & =\\begin{bmatrix}u_{1}\\\\\r\nv_{1}\r\n\\end{bmatrix}-\\begin{bmatrix}u_{1} & u_{2}\\\\\r\nv_{1} & v_{2}\r\n\\end{bmatrix}\\begin{bmatrix}\\Delta\\tau\\\\\r\n\\Delta\\theta\r\n\\end{bmatrix},\\\\\r\n\\Delta s & =d-\\D\\Delta x,\\\\\r\n\\Delta\\kappa & =d_{0}-\\D_{0}\\Delta\\tau,\\label{eq:hsd_back2}\r\n\\end{align}\r\n\\end{subequations}where $\\D_{0}=\\kappa/\\tau$ and $d_{0}=-\\kappa+\\mu^{+}\\tau^{-1}$.\r\n","start":122968,"end":124011}]},{"id":"S044","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"嵌入正常矩阵的稀疏分解","source_units":[{"raw_tex":"Hence, the cost of computing the search direction is dominated by\r\nthe cost of solving the normal equation for three different right-hand\r\nsides. Here, the normal matrix is written\r\n\r\n\\[\r\n\\M\\D^{-1}\\M^{T}=\\diag(W_{1}\\otimes_{s}W_{1},\\ldots,W_{\\ell}\\otimes_{s}W_{\\ell})+\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}(w_{1}w_{1}^{T}+\\sigma I)\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix},\r\n\\]\r\nwhere $\\sigma=\\frac{1}{2}(w_{0}^{2}-w_{1}^{T}w_{1})>0$ and $\\otimes_{s}$\r\ndenotes the symmetric Kronecker product~\\cite{alizadeh1998primal}\r\n","start":124011,"end":124532}]},{"id":"S045","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"对称 Kronecker 乘积","source_units":[{"raw_tex":"implicitly defined to satisfy\r\n\\[\r\n(A\\otimes_{s}B)\\vector(X)=\\frac{1}{2}\\vector(AXB^{T}+BXA^{T})\\qquad\\text{for all }X=X^{T}.\r\n\\]\r\n","start":124532,"end":124663}]},{"id":"S046","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"正常方程引理的实例化","source_units":[{"raw_tex":"Under the hypothesis on $\\A$ stated in Theorem~\\ref{thm:lintime},\r\nthe normal matrix satisfies the assumptions of Lemma~\\ref{lem:normal},\r\nand can therefore be solved in linear $O(n)$ time and memory. \r\n","start":124663,"end":124866}]},{"id":"S047","semantic_type":"theorem","kind":"theorem","labels":["eq:eps_accurate","eq:structA","thm:lintime"],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"近线性时间主定理","source_units":[{"raw_tex":"\\begin{theorem}[Near-linear time]\r\n\\label{thm:lintime}Let $\\T=(\\{J_{1},\\ldots,J_{\\ell}\\},T)$ be a\r\ntree decomposition for the sparsity graph of $C,A_{1},\\ldots,A_{m}\\in\\S^{n}$.\r\nIn the corresponding instance of (\\ref{eq:CTC}), let each constraint\r\nbe written\r\n\\begin{equation}\r\n\\sum_{j=1}^{\\ell}A_{i,j}\\bullet X_{j}=A_{i,j}\\bullet X_{j}+A_{i,k}\\bullet X_{k}=b_{i}\\qquad(j,k)\\in E(T).\\label{eq:structA}\r\n\\end{equation}\r\nUnder Assumptions~\\ref{ass:lin} \\&~\\ref{ass:slater}, there exists\r\nan algorithm that computes an iterate $(x,y,s)\\in\\K\\times\\R^{p}\\times\\K_{*}$\r\nsatisfying\r\n\\begin{align}\r\n\\left\\Vert \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}x-\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}\\right\\Vert  & \\le\\epsilon, & \\left\\Vert \\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}^{T}y-s+c\\right\\Vert  & \\le\\epsilon, & \\frac{x^{T}s}{\\sum_{j=1}^{\\ell}|J_{j}|} & \\le\\epsilon\\label{eq:eps_accurate}\r\n\\end{align}\r\nin $O(\\omega^{6.5}n^{1.5}\\log(1/\\epsilon))$ time and $O(\\omega^{4}n)$\r\nspace, where $\\omega=\\max_{j}|J_{j}|=1+\\wid(\\T)$.\r\n\\end{theorem}","start":59053,"end":60075}]},{"id":"S048","semantic_type":"assumption","kind":"assumption","labels":["eq:structA","thm:lintime"],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"主定理的局部结构条件","source_units":[{"raw_tex":"\\label{thm:lintime}Let $\\T=(\\{J_{1},\\ldots,J_{\\ell}\\},T)$ be a\r\ntree decomposition for the sparsity graph of $C,A_{1},\\ldots,A_{m}\\in\\S^{n}$.\r\nIn the corresponding instance of (\\ref{eq:CTC}), let each constraint\r\nbe written\r\n\\begin{equation}\r\n\\sum_{j=1}^{\\ell}A_{i,j}\\bullet X_{j}=A_{i,j}\\bullet X_{j}+A_{i,k}\\bullet X_{k}=b_{i}\\qquad(j,k)\\in E(T).\\label{eq:structA}\r\n\\end{equation}\r\nUnder Assumptions~\\ref{ass:lin} \\&~\\ref{ass:slater}, there exists\r\n","start":59088,"end":59539}]},{"id":"S049","semantic_type":"lemma","kind":"lemma","labels":["lem:lin_indep"],"member_ids":["S049","S064","S065","S066"],"declared_external":false,"source_only":false,"title":"线性无关的传递","source_units":[{"raw_tex":"\\begin{lemma}[Linear independence]\r\n\\label{lem:lin_indep}There exists $[u;v]\\ne0$ such that $\\A^{T}u+\\N^{T}v=0$\r\nif and only if there exists $y\\ne0$ such that $\\sum_{i}y_{i}A_{i}=0$.\r\n\\end{lemma}","start":40848,"end":41043},{"raw_tex":"\\begin{proof}[Lemma~\\ref{lem:lin_indep}]We will prove that\r\n\\[\r\n\\text{exists }[u;v]\\ne0,\\quad\\A^{T}u+\\N^{T}v=0\\qquad\\iff\\qquad\\text{exists }y\\ne0,\\quad\\sum_{i=1}^{m}y_{i}A_{i}=0.\r\n\\]\r\n\\textbf{($\\implies$)} We must have $u\\ne0$, because $\\N$ has full\r\nrow rank by Lemma~\\ref{lem:row-rank}, and so $\\A^{T}0+\\N^{T}v=0$\r\nif and only if $v=0$. Multiplying by $\\P$ yields $\\P^{T}(\\A^{T}u+\\N^{T}v)=\\M^{T}u+0=0$\r\nand so setting $y=u\\ne0$ yields $\\M^{T}y=0$.\\textbf{ ($\\impliedby$)\r\n}We use Lemma~\\ref{lem:orth} to decompose $\\A^{T}y=\\P z+\\N^{T}v$.\r\nIf $\\vector(\\sum_{i}y_{i}A_{i})=\\M^{T}y=\\P^{T}\\A^{T}y=0,$ then $\\P^{T}\\P z=\\P^{T}\\A^{T}y-\\P^{T}\\N^{T}v=0$\r\nand so $\\P z=0$. Setting $u=-y\\ne0$ yields $\\A^{T}u+\\N^{T}v=0$.\\qed \\end{proof}","start":110279,"end":111006},{"raw_tex":"\\textbf{($\\implies$)} We must have $u\\ne0$, because $\\N$ has full\r\nrow rank by Lemma~\\ref{lem:row-rank}, and so $\\A^{T}0+\\N^{T}v=0$\r\nif and only if $v=0$. Multiplying by $\\P$ yields $\\P^{T}(\\A^{T}u+\\N^{T}v)=\\M^{T}u+0=0$\r\nand so setting $y=u\\ne0$ yields $\\M^{T}y=0$.\\textbf{ ($\\impliedby$)\r\n","start":110463,"end":110753},{"raw_tex":"and so setting $y=u\\ne0$ yields $\\M^{T}y=0$.\\textbf{ ($\\impliedby$)\r\n}We use Lemma~\\ref{lem:orth} to decompose $\\A^{T}y=\\P z+\\N^{T}v$.\r\nIf $\\vector(\\sum_{i}y_{i}A_{i})=\\M^{T}y=\\P^{T}\\A^{T}y=0,$ then $\\P^{T}\\P z=\\P^{T}\\A^{T}y-\\P^{T}\\N^{T}v=0$\r\nand so $\\P z=0$. Setting $u=-y\\ne0$ yields $\\A^{T}u+\\N^{T}v=0$.\\qed \\end{proof}","start":110684,"end":111006}]},{"id":"S050","semantic_type":"lemma","kind":"lemma","labels":["lem:slaterp"],"member_ids":["S050","S067","S068","S070","S071"],"declared_external":false,"source_only":false,"title":"原始严格可行的传递","source_units":[{"raw_tex":"\\begin{lemma}[Primal Slater]\r\n\\label{lem:slaterp}There exists $x\\in\\Int(\\K)$ satisfying $\\A x=b$\r\nand $\\N x=0$ if and only if there exists an $X\\succ0$ satisfying\r\n$A_{i}\\bullet X=b_{i}$ for all $i\\in\\{1,\\ldots,m\\}$.\r\n\\end{lemma}","start":41048,"end":41277},{"raw_tex":"\\begin{proof}[Lemma~\\ref{lem:slaterp}]We will prove that \r\n\\[\r\n\\text{exists }x\\in\\Int(\\K),\\quad\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}x=\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}\\qquad\\iff\\qquad\\text{exists }X\\succeq0,\\quad\\M\\,\\vector(X)=b.\r\n\\]\r\nDefine the chordal completion $F$ as in (\\ref{eq:chordal_compl}).\r\nObserve that $\\M\\,\\vector(X)=\\M\\,\\vector(Z)$ holds for all pairs\r\nof $P_{F}(X)=Z$, because each $A_{i}\\in\\S_{F}^{n}$ satisfies $A_{i}\\bullet X=A_{i}\\bullet P_{F}(X)$.\r\n","start":111010,"end":111484},{"raw_tex":"\\begin{proof}[Lemma~\\ref{lem:slaterp}]We will prove that \r\n\\[\r\n\\text{exists }x\\in\\Int(\\K),\\quad\\begin{bmatrix}\\A\\\\\r\n\\N\r\n\\end{bmatrix}x=\\begin{bmatrix}b\\\\\r\n0\r\n\\end{bmatrix}\\qquad\\iff\\qquad\\text{exists }X\\succeq0,\\quad\\M\\,\\vector(X)=b.\r\n\\]\r\nDefine the chordal completion $F$ as in (\\ref{eq:chordal_compl}).\r\nObserve that $\\M\\,\\vector(X)=\\M\\,\\vector(Z)$ holds for all pairs\r\nof $P_{F}(X)=Z$, because each $A_{i}\\in\\S_{F}^{n}$ satisfies $A_{i}\\bullet X=A_{i}\\bullet P_{F}(X)$.\r\nAdditionally, the positive definite version of Theorem~\\ref{thm:prim_sep}\r\nis written\r\n\\begin{gather}\r\n\\text{exists }X\\succ0,\\quad P_{F}(X)=Z\\qquad\\iff\\qquad\\P\\,\\vector(Z)\\in\\Int(\\K).\\label{eq:prim_sep}\r\n\\end{gather}\r\nThis result was first established by Grone et al.~\\cite{grone1984positive};\r\na succinct proof can be found in~\\cite[Theorem~10.1]{vandenberghe2015chordal}.\\textbf{\r\n($\\implies$)} For every $x$ satisfying $\\N x=0$, there exists $Z$\r\nsuch that $\\P\\,\\vector(Z)=x$ due to Lemma~\\ref{lem:orth}. If additionally\r\n$x\\in\\Int(\\K)$, then there exists $X\\succ0$ satisfying $Z=P_{F}(X)$\r\ndue to (\\ref{eq:prim_sep}). We can verify that $\\M\\,\\vector(X)=\\M\\,\\vector(Z)=\\A\\P\\,\\vector(Z)=\\A x=b$.\\textbf{\r\n($\\impliedby$)} For every $X\\succ0,$ there exists $Z$ satisfying\r\n$Z=P_{F}(X)$ and $\\P\\,\\vector(Z)\\in\\Int(\\K)$ due to (\\ref{eq:prim_sep}).\r\nSet $u=\\P\\,\\vector(Z)$ and observe that $u\\in\\Int(\\K)$ and $\\N u=\\N\\P\\,\\vector(Z)=0$.\r\nIf additionally $\\M\\,\\vector(X)=b$, then $\\A u=\\A\\P\\,\\vector(Z)=\\M\\,\\vector(Z)=b$.\\qed \\end{proof}","start":111010,"end":112516},{"raw_tex":"a succinct proof can be found in~\\cite[Theorem~10.1]{vandenberghe2015chordal}.\\textbf{\r\n($\\implies$)} For every $x$ satisfying $\\N x=0$, there exists $Z$\r\nsuch that $\\P\\,\\vector(Z)=x$ due to Lemma~\\ref{lem:orth}. If additionally\r\n$x\\in\\Int(\\K)$, then there exists $X\\succ0$ satisfying $Z=P_{F}(X)$\r\ndue to (\\ref{eq:prim_sep}). We can verify that $\\M\\,\\vector(X)=\\M\\,\\vector(Z)=\\A\\P\\,\\vector(Z)=\\A x=b$.\\textbf{\r\n","start":111779,"end":112191},{"raw_tex":"($\\impliedby$)} For every $X\\succ0,$ there exists $Z$ satisfying\r\n$Z=P_{F}(X)$ and $\\P\\,\\vector(Z)\\in\\Int(\\K)$ due to (\\ref{eq:prim_sep}).\r\nSet $u=\\P\\,\\vector(Z)$ and observe that $u\\in\\Int(\\K)$ and $\\N u=\\N\\P\\,\\vector(Z)=0$.\r\nIf additionally $\\M\\,\\vector(X)=b$, then $\\A u=\\A\\P\\,\\vector(Z)=\\M\\,\\vector(Z)=b$.\\qed \\end{proof}","start":112191,"end":112516}]},{"id":"S051","semantic_type":"lemma","kind":"lemma","labels":["lem:slaterd"],"member_ids":["S051","S072","S074","S075"],"declared_external":false,"source_only":false,"title":"对偶严格可行的传递","source_units":[{"raw_tex":"\\begin{lemma}[Dual Slater]\r\n\\label{lem:slaterd}There exists $u,v$ satisfying $c-\\A^{T}u-\\N^{T}v\\in\\Int(\\K_{*})$\r\nif and only if there exists $y$ satisfying $C-\\sum_{i}y_{i}A_{i}\\succ0$.\r\n\\end{lemma}","start":41282,"end":41480},{"raw_tex":"\\begin{proof}[Lemma~\\ref{lem:slaterp}]We will prove that\r\n\\[\r\n\\text{exists }u,v,\\quad c-\\A^{T}u-\\N^{T}v\\in\\Int(\\K_{*})\\qquad\\iff\\qquad\\text{exists }y,\\quad C-\\sum_{i}y_{i}A_{i}\\succ0.\r\n\\]\r\nDefine the chordal completion $F$ as in (\\ref{eq:chordal_compl}).\r\nTheorem~\\ref{thm:prim_sep} in (\\ref{eq:prim_sep}) has a dual theorem\r\n\\begin{gather}\r\n\\text{exists }S\\succ0,\\quad S\\in\\S_{F}^{n}\\qquad\\iff\\qquad\\text{exists }h\\in\\Int(\\K_{*}),\\quad\\vector(S)=\\P^{T}h.\\label{eq:dual_sep}\r\n\\end{gather}\r\nThis result readily follows from the positive semidefinite version\r\nproved by Alger et al.~\\cite{agler1988positive}; see also~\\cite[Theorem~9.2]{vandenberghe2015chordal}.\\textbf{\r\n($\\implies$)} For each $h=c-\\A^{T}u-\\N^{T}v$, define $S=C-\\sum_{i}u_{i}A_{i}$\r\nand observe that \r\n\\[\r\n\\P^{T}h=\\P^{T}(c-\\A^{T}u-\\N^{T}v)=\\vector(C)-\\M^{T}u-0=\\vector(S).\r\n\\]\r\nIf additionally $h\\in\\K_{*}$, then $S\\succ0$ due to (\\ref{eq:dual_sep}).\\textbf{\r\n($\\impliedby$)} For each $S=C-\\sum_{i}y_{i}A_{i}\\succ0$, there exists\r\nan $h\\in\\Int(\\K_{*})$ satisfying $\\vector(S)=\\P^{T}h$ due to (\\ref{eq:dual_sep}).\r\nWe use Lemma~\\ref{lem:orth} to decompose $h=\\P u+\\N^{T}v$. Given\r\nthat $\\vector(S)=\\P^{T}h=\\P^{T}\\P u+0$, we must actually have $\\P u=c-\\A^{T}y$\r\nsince $\\P^{T}(c-\\A^{T}y)=\\vector(C)-\\M^{T}y=\\vector(S)$. Hence $h=c-\\A^{T}y+\\N^{T}v$\r\nand $h\\in\\Int(\\K_{*})$.\\qed \\end{proof}","start":112520,"end":113871},{"raw_tex":"proved by Alger et al.~\\cite{agler1988positive}; see also~\\cite[Theorem~9.2]{vandenberghe2015chordal}.\\textbf{\r\n($\\implies$)} For each $h=c-\\A^{T}u-\\N^{T}v$, define $S=C-\\sum_{i}u_{i}A_{i}$\r\nand observe that \r\n\\[\r\n\\P^{T}h=\\P^{T}(c-\\A^{T}u-\\N^{T}v)=\\vector(C)-\\M^{T}u-0=\\vector(S).\r\n\\]\r\nIf additionally $h\\in\\K_{*}$, then $S\\succ0$ due to (\\ref{eq:dual_sep}).\\textbf{\r\n","start":113078,"end":113446},{"raw_tex":"($\\impliedby$)} For each $S=C-\\sum_{i}y_{i}A_{i}\\succ0$, there exists\r\nan $h\\in\\Int(\\K_{*})$ satisfying $\\vector(S)=\\P^{T}h$ due to (\\ref{eq:dual_sep}).\r\nWe use Lemma~\\ref{lem:orth} to decompose $h=\\P u+\\N^{T}v$. Given\r\nthat $\\vector(S)=\\P^{T}h=\\P^{T}\\P u+0$, we must actually have $\\P u=c-\\A^{T}y$\r\nsince $\\P^{T}(c-\\A^{T}y)=\\vector(C)-\\M^{T}y=\\vector(S)$. Hence $h=c-\\A^{T}y+\\N^{T}v$\r\nand $h\\in\\Int(\\K_{*})$.\\qed \\end{proof}","start":113446,"end":113871}]},{"id":"S052","semantic_type":"lemma","kind":"lemma","labels":["eq:Asparsity","lem:normal"],"member_ids":["S052","S054","S056","S057","S058","S059"],"declared_external":false,"source_only":false,"title":"线性时间正常方程","source_units":[{"raw_tex":"\\begin{lemma}[Linear-time normal equation]\r\n\\label{lem:normal}Let there exist $v_{i}\\in V(T)$ for each $i\\in\\{1,\\ldots,m\\}$\r\nsuch that\r\n\\begin{equation}\r\nA_{i,j}\\ne0\\qquad\\implies\\qquad j=v_{i}\\text{ or }j=p(v_{i}).\\label{eq:Asparsity}\r\n\\end{equation}\r\nDefine $\\H$ and $\\q$ according to (\\ref{eq:Hqdef}). Then, under\r\nAssumption~\\ref{ass:lin}:\r\n\\begin{enumerate}\r\n\\item (Forming) It costs $O(\\omega^{6}n)$ time and $O(\\omega^{4}n)$ space\r\nto form $\\H$ and $\\q$, where $\\omega=\\max_{j}|J_{j}|=1+\\wid(\\T)$.\r\n\\item (Factoring) Let $\\pi$ be a topological ordering on $T$, and define\r\nthe associated block topological permutation $\\Pi$ as in Lemma~\\ref{lem:blocktree}.\r\nThen, it costs $O(\\omega^{6}n)$ time and $O(\\omega^{4}n)$ space\r\nto factor $\\Pi\\H\\Pi^{T}$ into $\\L\\L^{T}$. \r\n\\item (Solving) Given $r$, $\\q$, $\\Pi$, and the Cholesky factor $\\L$\r\nsatisfying $\\L\\L^{T}=\\Pi\\H\\Pi^{T}$, it costs $O(\\omega^{4}n)$ time\r\nand space to solve $(\\H+\\q\\q^{T})u=r$ for $u$.\r\n\\end{enumerate}\r\n\\end{lemma}","start":54512,"end":55500},{"raw_tex":"\\begin{proof}\r\nFor an instance of (\\ref{eq:CTC}), denote $\\ell=|\\mathcal{J}|\\le n$\r\nas its number of conic constraints, and $d=\\frac{1}{2}\\sum_{j=1}^{\\ell}|J_{j}|(|J_{j}|+1)\\le\\omega^{2}\\ell$\r\nas its total number of variables. Under linear independence (Assumption~\\ref{ass:lin}),\r\nthe constraint matrix $[\\A;\\N]$ associated with (\\ref{eq:CTC}) has\r\n$d$ columns and at most $d$ rows (Lemma~\\ref{lem:lin_indep}).\r\nWrite $\\xi_{i}^{T}$ as the $i$-th row of $[\\A;\\N]$, and assume\r\nwithout loss of generality that $[\\A;\\N]$ has exactly $d$ rows.\r\nObserve that $\\nnz(\\xi_{i})\\le\\omega(\\omega+1)$ by the definition\r\nof $\\N$ (\\ref{eq:Nmat}) and the hypothesis on $\\A$ via (\\ref{eq:Asparsity}),\r\nso $\\nnz([\\A;\\N])\\le2\\omega^{4}\\ell$. \r\n\r\n\\textbf{(i)} We form $\\H$ by setting $\\H\\gets\\D_{s}$ and then adding\r\n$\\H\\gets\\H+\\sigma\\xi_{i}\\xi_{i}^{T}$ one at a time, for $i\\in\\{1,2,\\ldots,d\\}$.\r\nThe first step forms $\\D_{s}=\\diag(\\D_{s}^{(1)},\\D_{s}^{(2)},\\ldots,\\D_{s}^{(\\ell)})$\r\nwhere $\\D_{s}^{(j)}=W_{j}\\otimes W_{j}$. Each $\\nnz(\\D_{s}^{(j)})=\\nnz(W_{j})^{2}=|J_{j}|^{2}(|J_{j}|+1)^{2}/4\\le\\omega^{4}$\r\nfor $j\\in\\{1,2,\\ldots,\\ell\\}$, so the total cost is $O(\\omega^{4}n)$\r\ntime and space. The second step adds $\\nnz(\\xi_{i}\\xi_{i}^{T})^{2}\\le\\omega^{2}(\\omega+1)^{2}$\r\nnonzeros per constraint over $d$ total constraints, for a total cost\r\nof $O(\\omega^{6}n)$ time and apparently $O(\\omega^{6}n)$ space.\r\nHowever, by the definition of $\\N$ (\\ref{eq:Nmat}) and the hypothesis\r\non $\\A$ via (\\ref{eq:Asparsity}), the $(j,k)$-th off-diagonal block\r\nof $\\xi_{i}\\xi_{i}^{T}$ is nonzero only if $(j,k)$ is an edge of\r\nthe tree $T$, as in\r\n\\[\r\n\\xi_{i}\\xi_{i}^{T}[J_{j},J_{k}]\\ne0\\qquad\\implies\\qquad(j,k)\\in E(T)\\qquad\\text{for all }j\\ne k.\r\n\\]\r\nHence, adding $\\mathbf{H}\\gets\\mathbf{H}+\\sigma\\xi_{i}\\xi_{i}^{T}$\r\none at a time results in at most $|V(T)|+|E(T)|$ dense blocks of\r\nat most $\\frac{1}{2}\\omega(\\omega+1)\\times\\frac{1}{2}\\omega(\\omega+1)$,\r\nfor a total memory cost of $O(\\omega^{4}n)$. \r\n\r\n\\textbf{(ii)} We form $\\q=[\\A^{T},\\N^{T}]w_{1}$ using a sparse matrix-vector\r\nproduct. Given that $\\nnz(w_{1})\\le d$ and $\\nnz([\\A;\\N])\\le2\\omega^{4}\\ell$,\r\nthis step costs $O(\\omega^{4}n)$ time and space.\r\n\r\n\\textbf{(iii)} We partition $\\H$ into $[\\H_{i,j}]_{i,j=1}^{\\ell}$\r\nto reveal a block sparsity pattern that coincides with the adjacency\r\nmatrix of $T$:\r\n\\begin{align*}\r\n\\H_{i,j} & =\\begin{cases}\r\n\\D_{s,i}+\\sigma\\sum_{k=1}^{\\ell}\\N_{k,i}^{T}\\N_{k,i}+\\sigma\\sum_{q=1}^{m}a_{q,i}a_{q,i}^{T} & i=j\\\\\r\n\\sigma\\sum_{k=1}^{\\ell}\\N_{k,i}^{T}\\N_{k,j}+\\sigma\\sum_{q=1}^{m}a_{q,i}a_{q,j}^{T} & (i,j)\\in E(T)\\\\\r\n0 & \\text{otherwise}\r\n\\end{cases}\r\n\\end{align*}\r\nwhere $a_{q,i}=\\vector(A_{q,i})$. According to Lemma~\\ref{lem:blocktree},\r\nthe permuted matrix $\\Pi\\H\\Pi^{T}$ factors into $\\L\\L^{T}$ with\r\nno block fill in $O(\\omega^{6}n)$ time and $O(\\omega^{4}n)$ space,\r\nbecause each block $\\H_{i,j}$ for $i,j\\in\\{1,2,\\ldots,\\ell\\}$ is\r\nat most order $\\frac{1}{2}\\omega(\\omega+1)$.\r\n\r\n\\textbf{(iv)} Using the rank-1 update formula (\\ref{eq:update}),\r\nthe cost of solving $(\\H+\\q\\q^{T})u=r$ is the same as the cost of\r\nsolving $\\H u=r$ for two right-hand sides, plus algebraic manipulations\r\nin $O(d)=O(\\omega^{2}n)$ time. Applying Lemma~\\ref{lem:blocktree}\r\nshows that the cost of solving $\\H u=r$ for each right-hand side\r\nis $O(\\omega^{4}n)$ time and space. \\qed\r\n\\end{proof}","start":55502,"end":58856},{"raw_tex":"\\textbf{(i)} We form $\\H$ by setting $\\H\\gets\\D_{s}$ and then adding\r\n$\\H\\gets\\H+\\sigma\\xi_{i}\\xi_{i}^{T}$ one at a time, for $i\\in\\{1,2,\\ldots,d\\}$.\r\nThe first step forms $\\D_{s}=\\diag(\\D_{s}^{(1)},\\D_{s}^{(2)},\\ldots,\\D_{s}^{(\\ell)})$\r\nwhere $\\D_{s}^{(j)}=W_{j}\\otimes W_{j}$. Each $\\nnz(\\D_{s}^{(j)})=\\nnz(W_{j})^{2}=|J_{j}|^{2}(|J_{j}|+1)^{2}/4\\le\\omega^{4}$\r\nfor $j\\in\\{1,2,\\ldots,\\ell\\}$, so the total cost is $O(\\omega^{4}n)$\r\ntime and space. The second step adds $\\nnz(\\xi_{i}\\xi_{i}^{T})^{2}\\le\\omega^{2}(\\omega+1)^{2}$\r\nnonzeros per constraint over $d$ total constraints, for a total cost\r\nof $O(\\omega^{6}n)$ time and apparently $O(\\omega^{6}n)$ space.\r\nHowever, by the definition of $\\N$ (\\ref{eq:Nmat}) and the hypothesis\r\non $\\A$ via (\\ref{eq:Asparsity}), the $(j,k)$-th off-diagonal block\r\nof $\\xi_{i}\\xi_{i}^{T}$ is nonzero only if $(j,k)$ is an edge of\r\nthe tree $T$, as in\r\n\\[\r\n\\xi_{i}\\xi_{i}^{T}[J_{j},J_{k}]\\ne0\\qquad\\implies\\qquad(j,k)\\in E(T)\\qquad\\text{for all }j\\ne k.\r\n\\]\r\nHence, adding $\\mathbf{H}\\gets\\mathbf{H}+\\sigma\\xi_{i}\\xi_{i}^{T}$\r\none at a time results in at most $|V(T)|+|E(T)|$ dense blocks of\r\nat most $\\frac{1}{2}\\omega(\\omega+1)\\times\\frac{1}{2}\\omega(\\omega+1)$,\r\nfor a total memory cost of $O(\\omega^{4}n)$. \r\n","start":56231,"end":57483},{"raw_tex":"\\textbf{(ii)} We form $\\q=[\\A^{T},\\N^{T}]w_{1}$ using a sparse matrix-vector\r\nproduct. Given that $\\nnz(w_{1})\\le d$ and $\\nnz([\\A;\\N])\\le2\\omega^{4}\\ell$,\r\nthis step costs $O(\\omega^{4}n)$ time and space.\r\n","start":57485,"end":57692},{"raw_tex":"\\textbf{(iii)} We partition $\\H$ into $[\\H_{i,j}]_{i,j=1}^{\\ell}$\r\nto reveal a block sparsity pattern that coincides with the adjacency\r\nmatrix of $T$:\r\n\\begin{align*}\r\n\\H_{i,j} & =\\begin{cases}\r\n\\D_{s,i}+\\sigma\\sum_{k=1}^{\\ell}\\N_{k,i}^{T}\\N_{k,i}+\\sigma\\sum_{q=1}^{m}a_{q,i}a_{q,i}^{T} & i=j\\\\\r\n\\sigma\\sum_{k=1}^{\\ell}\\N_{k,i}^{T}\\N_{k,j}+\\sigma\\sum_{q=1}^{m}a_{q,i}a_{q,j}^{T} & (i,j)\\in E(T)\\\\\r\n0 & \\text{otherwise}\r\n\\end{cases}\r\n\\end{align*}\r\nwhere $a_{q,i}=\\vector(A_{q,i})$. According to Lemma~\\ref{lem:blocktree},\r\nthe permuted matrix $\\Pi\\H\\Pi^{T}$ factors into $\\L\\L^{T}$ with\r\nno block fill in $O(\\omega^{6}n)$ time and $O(\\omega^{4}n)$ space,\r\nbecause each block $\\H_{i,j}$ for $i,j\\in\\{1,2,\\ldots,\\ell\\}$ is\r\nat most order $\\frac{1}{2}\\omega(\\omega+1)$.\r\n","start":57694,"end":58462},{"raw_tex":"\\textbf{(iv)} Using the rank-1 update formula (\\ref{eq:update}),\r\nthe cost of solving $(\\H+\\q\\q^{T})u=r$ is the same as the cost of\r\nsolving $\\H u=r$ for two right-hand sides, plus algebraic manipulations\r\nin $O(d)=O(\\omega^{2}n)$ time. Applying Lemma~\\ref{lem:blocktree}\r\nshows that the cost of solving $\\H u=r$ for each right-hand side\r\nis $O(\\omega^{4}n)$ time and space. \\qed\r\n\\end{proof}","start":58464,"end":58856}]},{"id":"S053","semantic_type":"assumption","kind":"assumption","labels":["eq:Asparsity","lem:normal"],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"父子袋支撑假设","source_units":[{"raw_tex":"\\label{lem:normal}Let there exist $v_{i}\\in V(T)$ for each $i\\in\\{1,\\ldots,m\\}$\r\nsuch that\r\n\\begin{equation}\r\nA_{i,j}\\ne0\\qquad\\implies\\qquad j=v_{i}\\text{ or }j=p(v_{i}).\\label{eq:Asparsity}\r\n\\end{equation}\r\n","start":54556,"end":54765}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"维数与行稀疏度","source_units":[{"raw_tex":"\\begin{proof}\r\nFor an instance of (\\ref{eq:CTC}), denote $\\ell=|\\mathcal{J}|\\le n$\r\nas its number of conic constraints, and $d=\\frac{1}{2}\\sum_{j=1}^{\\ell}|J_{j}|(|J_{j}|+1)\\le\\omega^{2}\\ell$\r\nas its total number of variables. Under linear independence (Assumption~\\ref{ass:lin}),\r\nthe constraint matrix $[\\A;\\N]$ associated with (\\ref{eq:CTC}) has\r\n$d$ columns and at most $d$ rows (Lemma~\\ref{lem:lin_indep}).\r\nWrite $\\xi_{i}^{T}$ as the $i$-th row of $[\\A;\\N]$, and assume\r\nwithout loss of generality that $[\\A;\\N]$ has exactly $d$ rows.\r\nObserve that $\\nnz(\\xi_{i})\\le\\omega(\\omega+1)$ by the definition\r\nof $\\N$ (\\ref{eq:Nmat}) and the hypothesis on $\\A$ via (\\ref{eq:Asparsity}),\r\nso $\\nnz([\\A;\\N])\\le2\\omega^{4}\\ell$. \r\n","start":55502,"end":56229}]},{"id":"S060","semantic_type":"lemma","kind":"lemma","labels":["lem:row-rank"],"member_ids":["S060","S061"],"declared_external":false,"source_only":false,"title":"重叠矩阵满行秩","source_units":[{"raw_tex":"\\begin{lemma}\r\n\\label{lem:row-rank}The matrix $\\N$ in (\\ref{eq:Nmat}) has full\r\nrow rank, that is $\\det(\\N\\N^{T})\\ne0$.\r\n\\end{lemma}","start":108146,"end":108278},{"raw_tex":"\\begin{proof}\r\nWe make $\\N=[\\N_{i,j}]_{i,j=1}^{\\ell}$ upper-triangular by ordering\r\nits blocks topologically on $T$: each nonempty block row $\\N_{j}$\r\ncontains a nonzero block at $\\N_{j,j}$ and a nonzero block at $\\N_{j,p(j)}$\r\nwhere the parent node $p(j)>j$ is ordered after $j$. Then, the claim\r\nfollows because each diagonal block $\\N_{j,j}$ implements a surjection\r\nand must therefore have full row-rank. \\qed\r\n\\end{proof}","start":108280,"end":108706}]},{"id":"S062","semantic_type":"lemma","kind":"lemma","labels":["lem:orth"],"member_ids":["S062","S063"],"declared_external":false,"source_only":false,"title":"袋抽取与正交补","source_units":[{"raw_tex":"\\begin{lemma}[Orthogonal complement]\r\n\\label{lem:orth}Let $d=\\frac{1}{2}\\sum_{j=1}^{\\ell}|J_{j}|(|J_{j}|+1)$.\r\nImplicitly define the $d\\times\\frac{1}{2}n(n+1)$ matrix $\\P$ to\r\nsatisfy\r\n\\begin{align*}\r\n\\P\\,\\vector(X) & =[\\vector(X[J_{1},J_{1}]);\\ldots;\\vector(X[J_{\\ell},J_{\\ell}])]\\qquad\\forall X\\in\\S^{n}.\r\n\\end{align*}\r\nThen, (i) $\\N\\P=0$; (ii) every $x\\in\\R^{d}$ can be decomposed as\r\n$x=\\N^{T}u+\\P v$. \r\n\\end{lemma}","start":108710,"end":109129},{"raw_tex":"\\begin{proof}\r\nFor each $x=[\\vector(X_{j})]_{j=1}^{\\ell}\\in\\R^{d}$, Theorem~\\ref{thm:ctree}\r\nsays that there exists a $Z$ satisfying $\\P\\,\\vector(Z)=x$ if and\r\nonly if $\\N x=0$. Equivalently, $x\\in\\mathrm{span}(\\P)$ if and only\r\nif $x\\perp\\mathrm{span}(\\N^{T})$. The ``only if'' part implies\r\n(i), while the ``if'' part implies (ii).\r\n\\end{proof}","start":109131,"end":109477}]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","labels":["eq:prim_sep"],"member_ids":["S069"],"declared_external":true,"declaration_text":["原始前提列表明确包含 B0045 的正定版本；Grone et al., grone1984positive；Vandenberghe–Andersen, vandenberghe2015chordal, Theorem 10.1。"],"source_only":false,"title":"正定补全版本","source_units":[{"raw_tex":"Additionally, the positive definite version of Theorem~\\ref{thm:prim_sep}\r\nis written\r\n\\begin{gather}\r\n\\text{exists }X\\succ0,\\quad P_{F}(X)=Z\\qquad\\iff\\qquad\\P\\,\\vector(Z)\\in\\Int(\\K).\\label{eq:prim_sep}\r\n\\end{gather}\r\nThis result was first established by Grone et al.~\\cite{grone1984positive};\r\na succinct proof can be found in~\\cite[Theorem~10.1]{vandenberghe2015chordal}.\\textbf{\r\n","start":111484,"end":111867}]},{"id":"S073","semantic_type":"external_fact","kind":"external_fact","labels":["eq:dual_sep"],"member_ids":["S073"],"declared_external":true,"declaration_text":["原始前提列表明确列入 Agler et al., agler1988positive，及 Vandenberghe–Andersen, vandenberghe2015chordal, Theorem 9.2，包含作者所用严格版本。"],"source_only":false,"title":"稀疏正定矩阵的严格团分解","source_units":[{"raw_tex":"Define the chordal completion $F$ as in (\\ref{eq:chordal_compl}).\r\nTheorem~\\ref{thm:prim_sep} in (\\ref{eq:prim_sep}) has a dual theorem\r\n\\begin{gather}\r\n\\text{exists }S\\succ0,\\quad S\\in\\S_{F}^{n}\\qquad\\iff\\qquad\\text{exists }h\\in\\Int(\\K_{*}),\\quad\\vector(S)=\\P^{T}h.\\label{eq:dual_sep}\r\n\\end{gather}\r\nThis result readily follows from the positive semidefinite version\r\nproved by Alger et al.~\\cite{agler1988positive}; see also~\\cite[Theorem~9.2]{vandenberghe2015chordal}.\\textbf{\r\n","start":112709,"end":113190}]},{"id":"S076","semantic_type":"lemma","kind":"lemma","labels":["lem:eps"],"member_ids":["S076","S078","S079","S080","S081","S082"],"declared_external":false,"source_only":false,"title":"互补精度转为重缩放精度","source_units":[{"raw_tex":"\\begin{lemma}[$\\epsilon$-accurate and $\\epsilon$-feasible]\r\n\\label{lem:eps}If $(x,y,s,\\tau,\\theta,\\kappa)$ satisfies (\\ref{eq:hsdfeas})\r\nand (\\ref{eq:hsdcone}) and \r\n\\begin{align*}\r\n\\mu=\\frac{x^{T}s+\\tau\\kappa}{\\nu+1} & \\le\\epsilon, & \\tau\\kappa\\ge & \\gamma\\mu\r\n\\end{align*}\r\nfor constants $\\epsilon,\\gamma>0$, then the rescaled point $(x/\\tau,y/\\tau,s/\\tau)$\r\nsatisfies\r\n\\begin{align*}\r\n\\|\\M(x/\\tau)-\\b\\| & \\le K\\epsilon, & \\|\\M^{T}(y/\\tau)+(s/\\tau)-\\c\\| & \\le K\\epsilon, & \\frac{(x/\\tau)^{T}(s/\\tau)}{\\nu+1}\\le K\\epsilon\r\n\\end{align*}\r\nwhere $K$ is a constant.\r\n\\end{lemma}","start":117365,"end":117940},{"raw_tex":"\\begin{proof}\r\nNote that (\\ref{eq:hsdfeas}) implies $\\mu=\\theta$ and\r\n\\begin{align*}\r\n\\|\\M(x/\\tau)-\\b\\| & =\\frac{\\|r_{p}\\|\\mu}{\\tau}, & \\|\\M^{T}(y/\\tau)+(s/\\tau)-\\c\\| & =\\frac{\\|r_{d}\\|\\mu}{\\tau}, & \\frac{(x/\\tau)^{T}(s/\\tau)}{\\nu+1}=\\frac{\\mu}{\\tau^{2}}.\r\n\\end{align*}\r\n","start":117942,"end":118213},{"raw_tex":"\\begin{proof}\r\nNote that (\\ref{eq:hsdfeas}) implies $\\mu=\\theta$ and\r\n\\begin{align*}\r\n\\|\\M(x/\\tau)-\\b\\| & =\\frac{\\|r_{p}\\|\\mu}{\\tau}, & \\|\\M^{T}(y/\\tau)+(s/\\tau)-\\c\\| & =\\frac{\\|r_{d}\\|\\mu}{\\tau}, & \\frac{(x/\\tau)^{T}(s/\\tau)}{\\nu+1}=\\frac{\\mu}{\\tau^{2}}.\r\n\\end{align*}\r\nHence, we obtain our desired result by upper-bounding $1/\\tau$. Let\r\n$(x^{\\star},y^{\\star},s^{\\star},\\tau^{\\star},\\theta^{\\star},\\kappa^{\\star})$\r\nbe a solution of (\\ref{eq:hsd}), and note that for every $(x,y,s,\\tau,\\theta,\\kappa)$\r\nsatisfying (\\ref{eq:hsdfeas}) and (\\ref{eq:hsdcone}), we have the\r\nfollowing via the skew-symmetry of (\\ref{eq:hsdfeas})\r\n\\[\r\n(x-x^{\\star})^{T}(s-s^{\\star})+(\\tau-\\tau^{\\star})(\\kappa-\\kappa^{\\star})=0.\r\n\\]\r\nRearranging yields\r\n\\[\r\n(\\nu+1)\\mu=x^{T}s+\\tau\\kappa=(x^{\\star})^{T}s+x^{T}(s^{\\star})+\\tau\\kappa^{\\star}+\\tau^{\\star}\\kappa\\ge\\tau^{\\star}\\kappa\r\n\\]\r\nand hence\r\n\\[\r\n\\kappa\\tau^{\\star}\\le\\mu(\\nu+1),\\quad\\tau\\kappa\\ge\\gamma\\mu\\qquad\\implies\\qquad\\tau\\ge\\frac{\\gamma}{\\nu+1}\\tau^{\\star}.\r\n\\]\r\n\r\nIf (\\ref{eq:SDP}) satisfies the primal-dual Slater's condition, then\r\n(\\ref{eq:CTC}) also satisfies the primal-dual Slater's condition\r\n(Lemmas~\\ref{lem:slaterp} \\&~\\ref{lem:slaterd}). Therefore, the\r\nvectorized version (\\ref{eq:vecCTC}) of (\\ref{eq:CTC}) attains a\r\nsolution $(\\hat{x},\\hat{y},\\hat{s})$ with $\\hat{x}^{T}\\hat{s}=0$,\r\nand the following\r\n\\[\r\nx^{\\star}=\\tau^{\\star}\\begin{bmatrix}\\|\\hat{y}\\|\\\\\r\n\\hat{y}\\\\\r\n\\hat{s}\r\n\\end{bmatrix},\\quad y^{\\star}=\\tau^{\\star}\\hat{x},\\quad s^{\\star}=\\tau^{\\star}\\begin{bmatrix}0\\\\\r\n0\\\\\r\n\\hat{x}\r\n\\end{bmatrix},\\quad\\tau^{\\star}=\\frac{\\nu+1}{\\|\\hat{y}\\|+\\one_{\\K}^{T}\\hat{s}+\\one_{\\K}^{T}\\hat{x}+1},\r\n\\]\r\nwith $\\theta^{\\star}=\\kappa^{\\star}=0$ is a solution to (\\ref{eq:hsd}).\r\nThis proves the following upper-bound \r\n\\[\r\n\\frac{1}{\\tau}\\le K_{\\tau}=\\frac{1}{\\gamma}\\cdot\\min_{\\hat{x},\\hat{y},\\hat{s}}\\{\\|\\hat{y}\\|+\\one_{\\K}^{T}\\hat{s}+\\one_{\\K}^{T}\\hat{x}+1:(\\hat{x},\\hat{y},\\hat{s})\\text{ solves (\\ref{eq:vecCTC}) with }\\hat{x}^{T}\\hat{s}=0\\}.\r\n\\]\r\nSetting $K=\\max\\{\\|r_{p}\\|K_{\\tau},\\|r_{d}\\|K_{\\tau},K_{\\tau}^{2}\\}$\r\nyields our desired result. \\qed\r\n\\end{proof}","start":117942,"end":120056},{"raw_tex":"Hence, we obtain our desired result by upper-bounding $1/\\tau$. Let\r\n$(x^{\\star},y^{\\star},s^{\\star},\\tau^{\\star},\\theta^{\\star},\\kappa^{\\star})$\r\nbe a solution of (\\ref{eq:hsd}), and note that for every $(x,y,s,\\tau,\\theta,\\kappa)$\r\nsatisfying (\\ref{eq:hsdfeas}) and (\\ref{eq:hsdcone}), we have the\r\nfollowing via the skew-symmetry of (\\ref{eq:hsdfeas})\r\n\\[\r\n(x-x^{\\star})^{T}(s-s^{\\star})+(\\tau-\\tau^{\\star})(\\kappa-\\kappa^{\\star})=0.\r\n\\]\r\nRearranging yields\r\n\\[\r\n(\\nu+1)\\mu=x^{T}s+\\tau\\kappa=(x^{\\star})^{T}s+x^{T}(s^{\\star})+\\tau\\kappa^{\\star}+\\tau^{\\star}\\kappa\\ge\\tau^{\\star}\\kappa\r\n\\]\r\nand hence\r\n\\[\r\n\\kappa\\tau^{\\star}\\le\\mu(\\nu+1),\\quad\\tau\\kappa\\ge\\gamma\\mu\\qquad\\implies\\qquad\\tau\\ge\\frac{\\gamma}{\\nu+1}\\tau^{\\star}.\r\n\\]\r\n","start":118213,"end":118946},{"raw_tex":"If (\\ref{eq:SDP}) satisfies the primal-dual Slater's condition, then\r\n(\\ref{eq:CTC}) also satisfies the primal-dual Slater's condition\r\n(Lemmas~\\ref{lem:slaterp} \\&~\\ref{lem:slaterd}). Therefore, the\r\nvectorized version (\\ref{eq:vecCTC}) of (\\ref{eq:CTC}) attains a\r\nsolution $(\\hat{x},\\hat{y},\\hat{s})$ with $\\hat{x}^{T}\\hat{s}=0$,\r\nand the following\r\n\\[\r\nx^{\\star}=\\tau^{\\star}\\begin{bmatrix}\\|\\hat{y}\\|\\\\\r\n\\hat{y}\\\\\r\n\\hat{s}\r\n\\end{bmatrix},\\quad y^{\\star}=\\tau^{\\star}\\hat{x},\\quad s^{\\star}=\\tau^{\\star}\\begin{bmatrix}0\\\\\r\n0\\\\\r\n\\hat{x}\r\n\\end{bmatrix},\\quad\\tau^{\\star}=\\frac{\\nu+1}{\\|\\hat{y}\\|+\\one_{\\K}^{T}\\hat{s}+\\one_{\\K}^{T}\\hat{x}+1},\r\n\\]\r\nwith $\\theta^{\\star}=\\kappa^{\\star}=0$ is a solution to (\\ref{eq:hsd}).\r\n","start":118948,"end":119670},{"raw_tex":"This proves the following upper-bound \r\n\\[\r\n\\frac{1}{\\tau}\\le K_{\\tau}=\\frac{1}{\\gamma}\\cdot\\min_{\\hat{x},\\hat{y},\\hat{s}}\\{\\|\\hat{y}\\|+\\one_{\\K}^{T}\\hat{s}+\\one_{\\K}^{T}\\hat{x}+1:(\\hat{x},\\hat{y},\\hat{s})\\text{ solves (\\ref{eq:vecCTC}) with }\\hat{x}^{T}\\hat{s}=0\\}.\r\n\\]\r\nSetting $K=\\max\\{\\|r_{p}\\|K_{\\tau},\\|r_{d}\\|K_{\\tau},K_{\\tau}^{2}\\}$\r\nyields our desired result. \\qed\r\n\\end{proof}","start":119670,"end":120056}]},{"id":"S077","semantic_type":"assumption","kind":"assumption","labels":["lem:eps"],"member_ids":["S077"],"declared_external":false,"source_only":false,"title":"重缩放引理的点条件","source_units":[{"raw_tex":"\\label{lem:eps}If $(x,y,s,\\tau,\\theta,\\kappa)$ satisfies (\\ref{eq:hsdfeas})\r\nand (\\ref{eq:hsdcone}) and \r\n\\begin{align*}\r\n\\mu=\\frac{x^{T}s+\\tau\\kappa}{\\nu+1} & \\le\\epsilon, & \\tau\\kappa\\ge & \\gamma\\mu\r\n\\end{align*}\r\nfor constants $\\epsilon,\\gamma>0$, then the rescaled point $(x/\\tau,y/\\tau,s/\\tau)$\r\n","start":117425,"end":117726}]},{"id":"S083","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S083"],"declared_external":false,"source_only":false,"title":"近线性主定理完整证明","source_units":[{"raw_tex":"\\begin{proof}[Theorem~\\ref{thm:lintime}]Combining Lemma~\\ref{lem:eps}\r\nand Lemma~\\ref{lem:short-step} shows that the desired $\\epsilon$-accurate,\r\n$\\epsilon$-feasible iterate is obtained after $O(\\sqrt{\\nu}\\log(1/\\epsilon))$\r\ninterior-point iterations. At each iteration we perform the following\r\nsteps: 1) compute the scaling point $w$; 2) solve the normal equation\r\n(\\ref{eq:hsd_nrm}) for three right-hand sides; 3) back-substitute\r\n(\\ref{eq:hsd_back1})-(\\ref{eq:hsd_back2}) for the search direction\r\nand take the step in (\\ref{eq:hsd_ipm}). Note from the proof of Lemma~\\ref{lem:normal}\r\nthat the matrix $[\\A;\\N]$ has at most $O(\\omega^{2}n)$ rows under\r\nAssumption~\\ref{ass:lin}, and therefore $\\nnz(\\M)=O(\\omega^{4}n)$\r\nunder the hypothesis of Theorem~\\ref{thm:lintime}. Below, we show\r\nthat the cost of each step is bounded by $O(\\omega^{6}n)$ time and\r\n$O(\\omega^{4}n)$ memory.\r\n\r\n\\textbf{Scaling point.} We partition $x=[x_{0};x_{1};\\vector(X_{1});\\ldots;\\vector(X_{\\ell})]$\r\nand similarly for $s$. Then, the scaling point $w$ is given in closed-form~\\cite[Section~5]{sturm2002implementation}\r\n\\begin{gather*}\r\n\\begin{bmatrix}u_{0}\\\\\r\nu_{1}\r\n\\end{bmatrix}=2^{-3/4}\\begin{bmatrix}1 & 1\\\\\r\n-s_{1}/\\|s_{1}\\| & s_{1}/\\|s_{1}\\|\r\n\\end{bmatrix}\\begin{bmatrix}(s_{0}-\\|s_{1}\\|)^{1/2}\\\\\r\n(s_{0}+\\|s_{1}\\|)^{1/2}\r\n\\end{bmatrix},\\quad\\begin{bmatrix}v_{0}\\\\\r\nv_{1}\r\n\\end{bmatrix}=\\begin{bmatrix}u_{0} & u_{1}^{T}\\\\\r\nu_{1} & \\frac{1}{2}(u_{0}^{2}-u_{1}^{T}u_{1})I\r\n\\end{bmatrix}\\begin{bmatrix}x_{0}\\\\\r\nx_{1}\r\n\\end{bmatrix},\\\\\r\n\\begin{bmatrix}w_{0}\\\\\r\nw_{1}\r\n\\end{bmatrix}=2^{-1/4}\\begin{bmatrix}u_{0} & u_{1}^{T}\\\\\r\nu_{1} & \\frac{1}{2}(u_{0}^{2}-u_{1}^{T}u_{1})I\r\n\\end{bmatrix}\\begin{bmatrix}1 & 1\\\\\r\n-v_{1}/\\|v_{1}\\| & v_{1}/\\|v_{1}\\|\r\n\\end{bmatrix}\\begin{bmatrix}(v_{0}-\\|v_{1}\\|)^{-1/2}\\\\\r\n(v_{0}+\\|v_{1}\\|)^{-1/2}\r\n\\end{bmatrix},\\\\\r\nW_{j}=S_{j}^{1/2}(S_{j}^{1/2}X_{j}S_{j}^{1/2})^{-1/2}S_{j}^{1/2}\\qquad\\text{for all }j\\in\\{1,\\ldots,\\ell\\}.\r\n\\end{gather*}\r\nNoting that $\\nnz(w_{1})\\le O(\\omega^{2}n)$, $\\ell\\le n$ and each\r\n$W_{j}$ is at most $\\omega\\times\\omega$, the cost of forming $w=[w_{0};w_{1};\\vector(W_{1});\\ldots;\\vector(W_{\\ell})]$\r\nis at most $O(\\omega^{3}n)$ time and $O(\\omega^{2}n)$ memory. Also,\r\nsince\r\n\\[\r\n\\D=\\nabla^{2}F(w)=\\diag\\left(\\begin{bmatrix}w_{0} & w_{1}^{T}\\\\\r\nw_{1} & \\frac{1}{2}(w_{0}^{2}-w_{1}^{T}w_{1})I\r\n\\end{bmatrix},W_{1}\\otimes_{s}W_{1},\\ldots,W_{\\ell}\\otimes_{s}W_{\\ell}\\right)^{-1},\r\n\\]\r\nthe cost of each matrix-vector product with $\\D$ and $\\D^{-1}$ is\r\nalso $O(\\omega^{3}n)$ time and $O(\\omega^{2}n)$ memory.\r\n\r\n\\textbf{Normal equation. }The cost of matrix-vector products with\r\n$\\M$ and $\\M^{T}$ is $\\nnz(\\M)=O(\\omega^{4}n)$ time and memory.\r\nUsing Lemma~\\ref{lem:normal}, we form the right-hand sides and solve\r\nthe three normal equations in (\\ref{eq:hsd_nrm}) in $O(\\omega^{6}n)$\r\ntime and $O(\\omega^{4}n)$ memory.\r\n\r\n\\textbf{Back-substitution.} The cost of back substituting (\\ref{eq:hsd_back1})-(\\ref{eq:hsd_back2})\r\nand making the step (\\ref{eq:hsd_ipm}) is dominated by matrix-vector\r\nproducts with $\\D$, $\\D^{-1}$, $\\M$, and $\\M^{T}$ at $O(\\omega^{4}n)$\r\ntime and memory.\\qed \\end{proof}","start":124868,"end":128002}]},{"id":"S084","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S084"],"declared_external":false,"source_only":false,"title":"组合精度与迭代次数","source_units":[{"raw_tex":"\\begin{proof}[Theorem~\\ref{thm:lintime}]Combining Lemma~\\ref{lem:eps}\r\nand Lemma~\\ref{lem:short-step} shows that the desired $\\epsilon$-accurate,\r\n$\\epsilon$-feasible iterate is obtained after $O(\\sqrt{\\nu}\\log(1/\\epsilon))$\r\ninterior-point iterations. At each iteration we perform the following\r\n","start":124868,"end":125165}]},{"id":"S085","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S085"],"declared_external":false,"source_only":false,"title":"每次迭代的任务与数据规模","source_units":[{"raw_tex":"interior-point iterations. At each iteration we perform the following\r\nsteps: 1) compute the scaling point $w$; 2) solve the normal equation\r\n(\\ref{eq:hsd_nrm}) for three right-hand sides; 3) back-substitute\r\n(\\ref{eq:hsd_back1})-(\\ref{eq:hsd_back2}) for the search direction\r\nand take the step in (\\ref{eq:hsd_ipm}). Note from the proof of Lemma~\\ref{lem:normal}\r\nthat the matrix $[\\A;\\N]$ has at most $O(\\omega^{2}n)$ rows under\r\nAssumption~\\ref{ass:lin}, and therefore $\\nnz(\\M)=O(\\omega^{4}n)$\r\nunder the hypothesis of Theorem~\\ref{thm:lintime}. Below, we show\r\nthat the cost of each step is bounded by $O(\\omega^{6}n)$ time and\r\n$O(\\omega^{4}n)$ memory.\r\n","start":125094,"end":125754}]},{"id":"S086","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S086"],"declared_external":false,"source_only":false,"title":"缩放点与 Hessian 乘法成本","source_units":[{"raw_tex":"\\textbf{Scaling point.} We partition $x=[x_{0};x_{1};\\vector(X_{1});\\ldots;\\vector(X_{\\ell})]$\r\nand similarly for $s$. Then, the scaling point $w$ is given in closed-form~\\cite[Section~5]{sturm2002implementation}\r\n\\begin{gather*}\r\n\\begin{bmatrix}u_{0}\\\\\r\nu_{1}\r\n\\end{bmatrix}=2^{-3/4}\\begin{bmatrix}1 & 1\\\\\r\n-s_{1}/\\|s_{1}\\| & s_{1}/\\|s_{1}\\|\r\n\\end{bmatrix}\\begin{bmatrix}(s_{0}-\\|s_{1}\\|)^{1/2}\\\\\r\n(s_{0}+\\|s_{1}\\|)^{1/2}\r\n\\end{bmatrix},\\quad\\begin{bmatrix}v_{0}\\\\\r\nv_{1}\r\n\\end{bmatrix}=\\begin{bmatrix}u_{0} & u_{1}^{T}\\\\\r\nu_{1} & \\frac{1}{2}(u_{0}^{2}-u_{1}^{T}u_{1})I\r\n\\end{bmatrix}\\begin{bmatrix}x_{0}\\\\\r\nx_{1}\r\n\\end{bmatrix},\\\\\r\n\\begin{bmatrix}w_{0}\\\\\r\nw_{1}\r\n\\end{bmatrix}=2^{-1/4}\\begin{bmatrix}u_{0} & u_{1}^{T}\\\\\r\nu_{1} & \\frac{1}{2}(u_{0}^{2}-u_{1}^{T}u_{1})I\r\n\\end{bmatrix}\\begin{bmatrix}1 & 1\\\\\r\n-v_{1}/\\|v_{1}\\| & v_{1}/\\|v_{1}\\|\r\n\\end{bmatrix}\\begin{bmatrix}(v_{0}-\\|v_{1}\\|)^{-1/2}\\\\\r\n(v_{0}+\\|v_{1}\\|)^{-1/2}\r\n\\end{bmatrix},\\\\\r\nW_{j}=S_{j}^{1/2}(S_{j}^{1/2}X_{j}S_{j}^{1/2})^{-1/2}S_{j}^{1/2}\\qquad\\text{for all }j\\in\\{1,\\ldots,\\ell\\}.\r\n\\end{gather*}\r\nNoting that $\\nnz(w_{1})\\le O(\\omega^{2}n)$, $\\ell\\le n$ and each\r\n$W_{j}$ is at most $\\omega\\times\\omega$, the cost of forming $w=[w_{0};w_{1};\\vector(W_{1});\\ldots;\\vector(W_{\\ell})]$\r\nis at most $O(\\omega^{3}n)$ time and $O(\\omega^{2}n)$ memory. Also,\r\nsince\r\n\\[\r\n\\D=\\nabla^{2}F(w)=\\diag\\left(\\begin{bmatrix}w_{0} & w_{1}^{T}\\\\\r\nw_{1} & \\frac{1}{2}(w_{0}^{2}-w_{1}^{T}w_{1})I\r\n\\end{bmatrix},W_{1}\\otimes_{s}W_{1},\\ldots,W_{\\ell}\\otimes_{s}W_{\\ell}\\right)^{-1},\r\n\\]\r\nthe cost of each matrix-vector product with $\\D$ and $\\D^{-1}$ is\r\nalso $O(\\omega^{3}n)$ time and $O(\\omega^{2}n)$ memory.\r\n","start":125756,"end":127417}]},{"id":"S087","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S087"],"declared_external":false,"source_only":false,"title":"形成并求解三个正常方程","source_units":[{"raw_tex":"\\textbf{Normal equation. }The cost of matrix-vector products with\r\n$\\M$ and $\\M^{T}$ is $\\nnz(\\M)=O(\\omega^{4}n)$ time and memory.\r\nUsing Lemma~\\ref{lem:normal}, we form the right-hand sides and solve\r\nthe three normal equations in (\\ref{eq:hsd_nrm}) in $O(\\omega^{6}n)$\r\ntime and $O(\\omega^{4}n)$ memory.\r\n","start":127419,"end":127726}]},{"id":"S088","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S088"],"declared_external":false,"source_only":false,"title":"回代和更新的成本","source_units":[{"raw_tex":"\\textbf{Back-substitution.} The cost of back substituting (\\ref{eq:hsd_back1})-(\\ref{eq:hsd_back2})\r\nand making the step (\\ref{eq:hsd_ipm}) is dominated by matrix-vector\r\nproducts with $\\D$, $\\D^{-1}$, $\\M$, and $\\M^{T}$ at $O(\\omega^{4}n)$\r\ntime and memory.\\qed \\end{proof}","start":127728,"end":128002}]},{"id":"S089","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S089"],"declared_external":false,"source_only":false,"title":"合成总时间与峰值空间","source_units":[{"raw_tex":"\\begin{proof}[Theorem~\\ref{thm:lintime}]Combining Lemma~\\ref{lem:eps}\r\nand Lemma~\\ref{lem:short-step} shows that the desired $\\epsilon$-accurate,\r\n$\\epsilon$-feasible iterate is obtained after $O(\\sqrt{\\nu}\\log(1/\\epsilon))$\r\ninterior-point iterations. At each iteration we perform the following\r\nsteps: 1) compute the scaling point $w$; 2) solve the normal equation\r\n(\\ref{eq:hsd_nrm}) for three right-hand sides; 3) back-substitute\r\n(\\ref{eq:hsd_back1})-(\\ref{eq:hsd_back2}) for the search direction\r\nand take the step in (\\ref{eq:hsd_ipm}). Note from the proof of Lemma~\\ref{lem:normal}\r\nthat the matrix $[\\A;\\N]$ has at most $O(\\omega^{2}n)$ rows under\r\nAssumption~\\ref{ass:lin}, and therefore $\\nnz(\\M)=O(\\omega^{4}n)$\r\nunder the hypothesis of Theorem~\\ref{thm:lintime}. Below, we show\r\nthat the cost of each step is bounded by $O(\\omega^{6}n)$ time and\r\n$O(\\omega^{4}n)$ memory.\r\n","start":124868,"end":125754},{"raw_tex":"\\textbf{Normal equation. }The cost of matrix-vector products with\r\n$\\M$ and $\\M^{T}$ is $\\nnz(\\M)=O(\\omega^{4}n)$ time and memory.\r\nUsing Lemma~\\ref{lem:normal}, we form the right-hand sides and solve\r\nthe three normal equations in (\\ref{eq:hsd_nrm}) in $O(\\omega^{6}n)$\r\ntime and $O(\\omega^{4}n)$ memory.\r\n\r\n\\textbf{Back-substitution.} The cost of back substituting (\\ref{eq:hsd_back1})-(\\ref{eq:hsd_back2})\r\nand making the step (\\ref{eq:hsd_ipm}) is dominated by matrix-vector\r\nproducts with $\\D$, $\\D^{-1}$, $\\M$, and $\\M^{T}$ at $O(\\omega^{4}n)$\r\ntime and memory.\\qed \\end{proof}","start":127419,"end":128002}]},{"id":"S090","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S090"],"declared_external":true,"declaration_text":["原始外部前提列表中的 Standard conic strong duality and attainment under primal-dual Slater conditions。"],"source_only":false,"title":"锥强对偶与最优解可达","source_units":[]},{"id":"S091","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S091"],"declared_external":true,"declaration_text":["原始前提列表中的 Standard sparse Cholesky, Sherman–Morrison–Woodbury, and Nesterov–Todd scaling formulas；作者在 1037–1038 行点名 SMW。"],"source_only":false,"title":"Sherman–Morrison–Woodbury 公式","source_units":[]},{"id":"S092","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S092"],"declared_external":true,"declaration_text":["原始前提列表中的标准 Nesterov–Todd scaling formulas；作者于 2367 行明确引用 Sturm, sturm2002implementation, Section 5。"],"source_only":false,"title":"Nesterov–Todd 闭式缩放","source_units":[]}],"collapse_map":{"S001":"S001","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S052","S055":"S055","S056":"S052","S057":"S052","S058":"S052","S059":"S052","S060":"S060","S061":"S060","S062":"S062","S063":"S062","S064":"S049","S065":"S049","S066":"S049","S067":"S050","S068":"S050","S069":"S069","S070":"S050","S071":"S050","S072":"S051","S073":"S073","S074":"S051","S075":"S051","S076":"S076","S077":"S077","S078":"S076","S079":"S076","S080":"S076","S081":"S076","S082":"S076","S083":"S083","S084":"S084","S085":"S085","S086":"S086","S087":"S087","S088":"S088","S089":"S089","S090":"S090","S091":"S091","S092":"S092"}}}}
{"record_id":"2211.14850v2","paper":{"title":"Lyapunov stability of the subgradient method with constant step size","authors":["Josz, Cédric","Lai, Lexiao"],"arxiv_id":"2211.14850","version":"v2","journal":"Mathematical Programming","abstract_url":"https://arxiv.org/abs/2211.14850v2","pdf_url":"https://arxiv.org/pdf/2211.14850v2","doi":"10.1007/s10107-023-01936-6"},"context":{"raw_tex":"The subgradient method with constant step size for minimizing a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ consists in choosing an initial point $x_0 \\in \\mathbb{R}^n$ and generating a sequence of iterates according to the update rule $x_{k+1} \\in x_k - \\alpha \\partial f(x_k), ~ \\forall k \\in \\mathbb{N} := \\{0,1,2,\\hdots\\}$, where $\\alpha>0$ is the step size and $\\partial f$ is the Clarke subdifferential \\cite[Chapter 2]{clarke1990}. While this method is often used in practice to solve nonconvex and nonsmooth problems, there is little theoretical understanding of the behavior of its iterates. To the best of our knowledge, the only known results are in the convex setting, as we next describe.\n\n\nLet $\\|\\cdot\\|$ be the induced norm of an inner product $\\langle \\cdot, \\cdot\\rangle$ on $\\mathbb{R}^n$. Let $B(a,r)$ and $\\mathring{B}(a,r)$ respectively denote the closed ball and the open ball of center $a\\in \\mathbb{R}^n$ and radius $r>0$. We next define the notion of discrete Lyupanov stability.\n\n\\begin{definition}\n\\label{def:discrete_lyapunov}\n\\normalfont\nWe say that $x^*\\in \\mathbb{R}^n$ is a stable point of a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if for all $\\epsilon>0$, there exist $\\delta>0$ and $\\bar{\\alpha}>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$, the subgradient method with constant step size $\\alpha$ initialized in $B(x^*,\\delta)$ has all its iterates in $B(x^*,\\epsilon)$.\n\\end{definition}\n\n\nWe will apply Lemma \\ref{lemma:euler_method} to the case where the set-valued mapping $F$ is the opposite of the Clarke subdifferential of a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. Recall that a point $x^* \\in \\mathbb{R}^n$ is a local minimum (respectively strict local minimum) of a function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if there exists a positive constant $\\epsilon$ such that $f(x^*) \\leqslant f(x)$ for all $x \\in B(x^*,\\epsilon)\\setminus \\{x^*\\}$ (respectively $f(x^*) < f(x)$). Using Lemma \\ref{lemma:euler_method}, we obtain the following necessary condition for stability.\n","units":[{"raw_tex":"The subgradient method with constant step size for minimizing a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ consists in choosing an initial point $x_0 \\in \\mathbb{R}^n$ and generating a sequence of iterates according to the update rule $x_{k+1} \\in x_k - \\alpha \\partial f(x_k), ~ \\forall k \\in \\mathbb{N} := \\{0,1,2,\\hdots\\}$, where $\\alpha>0$ is the step size and $\\partial f$ is the Clarke subdifferential \\cite[Chapter 2]{clarke1990}. While this method is often used in practice to solve nonconvex and nonsmooth problems, there is little theoretical understanding of the behavior of its iterates. To the best of our knowledge, the only known results are in the convex setting, as we next describe.\n","start":2451,"end":3174},{"raw_tex":"Let $\\|\\cdot\\|$ be the induced norm of an inner product $\\langle \\cdot, \\cdot\\rangle$ on $\\mathbb{R}^n$. Let $B(a,r)$ and $\\mathring{B}(a,r)$ respectively denote the closed ball and the open ball of center $a\\in \\mathbb{R}^n$ and radius $r>0$. We next define the notion of discrete Lyupanov stability.\n\n\\begin{definition}\n\\label{def:discrete_lyapunov}\n\\normalfont\nWe say that $x^*\\in \\mathbb{R}^n$ is a stable point of a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if for all $\\epsilon>0$, there exist $\\delta>0$ and $\\bar{\\alpha}>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$, the subgradient method with constant step size $\\alpha$ initialized in $B(x^*,\\delta)$ has all its iterates in $B(x^*,\\epsilon)$.\n\\end{definition}\n","start":7298,"end":8049},{"raw_tex":"We will apply Lemma \\ref{lemma:euler_method} to the case where the set-valued mapping $F$ is the opposite of the Clarke subdifferential of a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. Recall that a point $x^* \\in \\mathbb{R}^n$ is a local minimum (respectively strict local minimum) of a function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if there exists a positive constant $\\epsilon$ such that $f(x^*) \\leqslant f(x)$ for all $x \\in B(x^*,\\epsilon)\\setminus \\{x^*\\}$ (respectively $f(x^*) < f(x)$). Using Lemma \\ref{lemma:euler_method}, we obtain the following necessary condition for stability.\n","start":14005,"end":14621}]},"statement":{"raw_tex":"\\begin{theorem}\n\\label{thm:necessary}\nStable points of locally Lipschitz semi-algebraic functions are local minima.\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\n\\label{thm:necessary}\nStable points of locally Lipschitz semi-algebraic functions are local minima.\n\\end{theorem}","start":14622,"end":14751}]},"proof":{"raw_tex":"\\begin{lemma}\n\\label{lemma:euler_method}\n    Let $F:\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^n$ be an upper semicontinuous mapping with nonempty, compact, and convex values. Let $X_0$ be a compact subset of $\\mathbb{R}^n$ and let $T>0$. Assume that there exist $\\bar{\\alpha},r>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$ and for all sequence $(x_k)_{k\\in\\mathbb{N}}$ such that\n    \\begin{equation}\n    \\label{eq:Euler}\n                x_{k+1} \\in x_k + \\alpha F(x_k), ~~~ \\forall k \\in \\mathbb{N},~~~ x_0 \\in X_0,\n    \\end{equation}\n    we have that $x_0,\\hdots,x_{\\lfloor T/\\alpha \\rfloor+1} \\in B(0,r)$. For all $\\epsilon>0$, there exists $\\hat{\\alpha}\\in (0,\\bar{\\alpha}]$ such that for all $\\alpha \\in (0,\\hat{\\alpha}]$ and for all sequence $(x_k)_{k\\in\\mathbb{N}}$ satisfying \\eqref{eq:Euler}, there exists an absolutely continuous function $x:[0,T]\\rightarrow\\mathbb{R}^n$ such that\n    \\begin{equation}\n    \\label{eq:ivp}\n                x'(t) \\in F(x(t)),~~~ \\text{for a.e.}~ \n                t \\in (0,T), ~~~ x(0) \\in X_0, \n    \\end{equation}\n    for which $\\|\\bar{x}(t) - x(t) \\| \\leqslant \\epsilon$ for all $t \\in [0,T]$, where $\\bar{x}:[0,T]\\rightarrow\\mathbb{R}^n$ is defined by $\\bar{x}(t) := x_k + (t-\\alpha k)/\\alpha  (x_{k+1}-x_k)$ for all $t\\in [\\alpha k , \\min\\{\\alpha(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha\\rfloor\\}$.\n\\end{lemma}\n\n\\begin{proof}\nLet $(\\alpha_m)_{m \\in \\mathbb{N}}$ denote a sequence of positive numbers that converges to zero. Without loss of generality, we may assume that the sequence is bounded above by $\\bar{\\alpha}$. To each term in the sequence, we attribute a sequence $(x_k^{m})_{k\\in\\mathbb{N}}$ generated by the Euler method with step size $\\alpha_m$ and initialized in $X_0$, that is to say, which satisfies \\eqref{eq:Euler} with $\\alpha:=\\alpha_m$. By assumption, $x_0^{m},\\hdots,x_{\\lfloor T/\\alpha_m \\rfloor+1}^{m} \\in B(0,r)$. Consider the linear interpolation of those iterates, that is to say, the function $\\bar{x}^{m}:[0,T]\\rightarrow\\mathbb{R}^n$ defined by $\\bar{x}^{m}(t) := x_k^{m} + (t-\\alpha_m k)(x_{k+1}^{m}-x_k^{m})/\\alpha_m$ for all $t\\in [\\alpha_m k , \\min\\{\\alpha_m(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$. Since $B(0,r)$ is convex, it holds that $\\|\\bar{x}^{m}(t)\\| \\leqslant r $ for all $t\\in [0,T]$. In addition, since $F$ is upper semicontinuous and compact valued, by \\cite[Proposition 3 p. 42]{aubin1984differential} there exists $r'>0$ such that $F(B(0,r)) \\subset B(0,r')$. Observe that $(\\bar{x}^{m})'(t) = (x_{k+1}^{m}-x_k^{m})/\\alpha_m \\in F(x_k^{m})$ for all $t\\in (\\alpha_m k , \\min\\{\\alpha_m(k+1),T\\})$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$. Hence, we have that $\\|(\\bar{x}^{m})'(t)\\| \\leqslant r'$ for almost every $t\\in (0,T)$. By successively applying the Arzel\\`a-Ascoli and the Banach-Alaoglu theorems (see \\cite[Theorem 4 p. 13]{aubin1984differential}), there exists a subsequence (again denoted $(\\alpha_{m})_{m\\in \\mathbb{N}}$) and an absolutely continuous function $x:[0,T]\\rightarrow \\mathbb{R}^n$ such that $\\bar{x}^{m}(\\cdot)$ converges uniformly to $x(\\cdot)$ and $(\\bar{x}^{m})'(\\cdot)$ converges weakly to $x'(\\cdot)$ in $L^1([0,T],\\mathbb{R}^n)$. Furthermore, for all $t\\in (\\alpha_{m} k , \\min\\{\\alpha_{m}(k+1),T\\})$ and $k \\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$, observe that\n\\begin{subequations}\n\\begin{align*}\n    (\\bar{x}^{m}(t),(\\bar{x}^{m})'(t)) \n    & = \\left(x_k^{m} + (t-\\alpha_m k)\\frac{x_{k+1}^{m}-x_k^{m}}{\\alpha_m},\\frac{x_{k+1}^{m}-x_k^{m}}{\\alpha_m}\\right) \\\\\n    & \\in \\left(\\{x_k^{m}\\} + (t-\\alpha_m k)F(x_k^{m})\\right) \\times F(x_k^{m})  \\\\\n    & = \\{x_k^{m}\\} \\times F(x_k^{m}) + (t-\\alpha_m k)F(x_k^{m}) \\times \\{0\\}  \\\\\n    & \\subset \\mathrm{graph}(F) + B(0,r'\\alpha_m)\\times \\{0\\}.\n\\end{align*}\n\\end{subequations}\nAccording to \\cite[Convergence Theorem p. 60]{aubin1984differential}, it follows that $x'(t) \\in F(x(t))$ for almost every $t\\in (0,T)$. The sequence of initial points $(x^{m}(0))_{m\\in\\mathbb{N}}$ lies in the closed set $X_0$, hence its limit $x(0)$ lies in $X_0$ as well. As a result, $x(\\cdot)$ is a solution to the differential inclusion \\eqref{eq:ivp}. \n\nTo sum up, we have shown that for every sequence $(\\alpha_m)_{m \\in \\mathbb{N}}$ of positive numbers converging to zero, there exists a subsequence for which the corresponding linear interpolations uniformly converge towards a solution of the differential inclusion \\eqref{eq:ivp}. The conclusion of the theorem now easily follows. To see why, one can reason by contradiction and assume that there exists $\\epsilon>0$ such that for all $\\hat{\\alpha} \\in (0,\\bar{\\alpha}]$, there exist $\\alpha \\in (0,\\hat{\\alpha}]$ and a sequence $(x_k)_{k \\in \\mathbb{N}}$ generated by the Euler method with step size $\\alpha$ and initialized in $X_0$ such that, for any solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp}, it holds that $\\|\\bar{x}(t) - x(t) \\| > \\epsilon$ for some $t \\in [0,T]$. We can then generate a sequence $(\\alpha_m)_{m \\in \\mathbb{N}}$ of positive numbers converging to zero such that, for any solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp}, it holds that $\\|\\bar{x}^{m}(t) - x(t) \\| > \\epsilon$ for some $t \\in [0,T]$. Since there exists a subsequence $(\\alpha_{\\varphi(m)})_{m \\in \\mathbb{N}}$ such that $\\bar{x}^{\\varphi(m)}$ uniformly converges to a solution to the differential inclusion \\eqref{eq:ivp}, we obtain a contradiction. \n\\end{proof}\n\n\\begin{proof}\nLet $x^* \\in \\mathbb{R}^n$ denote a stable point of a locally Lipschitz semi-algebraic function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. We reason by contradiction and assume that $x^*$ is not a local minimum of $f$. Given $x\\in \\mathbb{R}^n$ and $S \\subset \\mathbb{R}^n$, let $d(x,S) := \\inf \\{\\|x - y\\| : y \\in S\\}$. According to the Kurdyka-\\L{}ojasiewicz inequality \\cite[Theorem 14]{bolte2007clarke} (see also \\cite[Theorem 4.1]{attouch2010proximal}), there exist $r,\\rho>0$ and a strictly increasing concave continuous semi-algebraic function $\\psi:[0,\\rho) \\rightarrow [0,\\infty)$ that is continuously differentiable on $(0,\\rho)$ with $\\psi(0) = 0$ such that $d(0,\\partial f(x)) \\geqslant 1/\\psi'(|f(x) - f(x^*)|)$ for all $x \\in B(x^*,r)$ whenever $0<|f(x)-f(x^*)|< \\rho$. After possibly reducing $r$, the inequality holds for all $x \\in B(x^*,r)$ such that $f(x) \\neq f(x^*)$. \n\nLet $\\epsilon \\in (0,r/2)$. By the definition of stability (Definition \\ref{def:discrete_lyapunov}), there exist $\\delta>0$ and $\\bar{\\alpha}>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$, the subgradient method with constant step size $\\alpha$ initialized in $B(x^*,\\delta)$ has all its iterates in $B(x^*,\\epsilon)$. Since $x^*$ is not a local minimum, we can take an initial iterate $x_0$ in $B(x^*,\\delta)$ such that $f(x_0)<f(x^*)$. Let $L>0$ denote a Lipschitz constant of $f$ on $B(x^*,r)$ and let $T := (2\\epsilon + \\delta)L\\psi'(f(x^*) - f(x_0))^2>0$.\nConsider the differential inclusion\n\\begin{equation}\n\\label{eq:ivp_x_0}\n     x'(t) \\in - \\partial f(x(t)), ~~~ \\text{for a.e.}~ \n     t \\in (0,T),~~~ x(0) = x_0.\n\\end{equation}\nSince $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d)]{clarke1990} with nonempty, compact and convex values \\cite[2.1.2 Proposition (a)]{clarke1990}. By Lemma \\ref{lemma:euler_method}, there exists $\\hat{\\alpha} \\in (0,\\bar{\\alpha}]$ such that, for all $\\alpha \\in (0,\\hat{\\alpha}]$  and for all sequence $(x_k)_{k\\in \\mathbb{N}}$ generated by the subgradient method with constant step size $\\alpha$ and initialized at $x_0$, there exists a solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp_x_0} for which $\\|\\bar{x}(t) - x(t)\\| \\leqslant \\epsilon/2$ for all $t \\in [0,T]$, where $\\bar{x}:[0,T]\\rightarrow\\mathbb{R}^n$ is the piecewise linear function defined by $\\bar{x}(t) := x_k + (t-\\alpha k)/\\alpha  (x_{k+1}-x_k)$ for all $t\\in [\\alpha k , \\min\\{\\alpha(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha\\rfloor\\}$.\n\nLet us fix some $\\alpha \\in (0,\\min\\{\\hat{\\alpha},T/10\\}]$ from now on. Consider a sequence $(x_k)_{k\\in\\mathbb{N}}$ generated by the subgradient method with constant step size $\\alpha$ and initialized at $x_0$. Consider also the linear interpolation $\\bar{x}(\\cdot)$ of those iterates up to iteration $K+1$ where $K := \\lfloor T/\\alpha \\rfloor$, as well as a solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp_x_0} such that $\\|\\bar{x}(t)-x(t)\\| \\leqslant \\epsilon/2$ for all $t\\in [0,T]$. Since $f$ is semi-algebraic, by \\cite[Lemma 5.2]{davis2020stochastic} (see also \\cite{drusvyatskiy2015curves}) it holds that\n\\begin{equation}\n\\label{eq:lyapunov}\n    f(x(t)) - f(x(0)) = - \\int_0^t d(0,\\partial f(x(\\tau)))^2 d\\tau, ~~~ \\forall t \\in [0,T].\n\\end{equation}\nAs a result, $f(x(t)) \\leqslant f(x(0)) = f(x_0)$ for all $t \\in [0,T]$. Also, $\\|x(t) - x^*\\| \\leqslant \\|x(t) - \\bar{x}(t)\\| + \\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon/2 + \\epsilon = 3\\epsilon/2 \\leqslant r$, where the inequality $\\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon$ follows from the convexity of $B(x^*,\\epsilon)$. Hence $0<f(x^*) - f(x_0) \\leqslant f(x^*) - f(x(t))$ and $d(0,\\partial f(x(t))) \\geqslant  1/\\psi'(f(x^*)-f(x(t))) \\geqslant 1/\\psi'(f(x^*)-f(x_0))$ for all $t\\in[0,T]$ by concavity of $\\psi$. Together with \\eqref{eq:lyapunov}, it follows that $f(x(K\\alpha)) - f(x_0) \\leqslant -K\\alpha M^2$ where $K := \\lfloor T/\\alpha \\rfloor$ and $M:= 1/\\psi'(f(x^*)-f(x_0))$. Recall that $L$ is a Lipschitz constant of $f$ on $B(x^*,r)$, so that we have $|f(x(K\\alpha))-f(x_0)| \\leqslant L \\|x(K\\alpha)-x_0\\|$. We thus obtain the lower bound $\\|x(K\\alpha)-x_0\\| \\geqslant K\\alpha M^2/L = \\lfloor T/\\alpha \\rfloor \\alpha M^2/L \\geqslant (T-\\alpha)M^2/L \\geqslant (T-T/10)M^2/L = 9/10( 2\\epsilon + \\delta) $ (recall that $T = (2\\epsilon + \\delta)L/M^2$). Hence $\\|x(K\\alpha)-x^*\\| \\geqslant \\|x(K\\alpha)-x_0\\| - \\|x_0-x^*\\| \\geqslant 9/10(2\\epsilon + \\delta) - \\delta = 9 \\epsilon/5 - \\delta/10 \\geqslant 9 \\epsilon/5 - \\epsilon/10 = 17\\epsilon/10$ and finally $\\|x_K-x^*\\| \\geqslant \\|x(K\\alpha)-x^*\\| -\\|x_K-x(K\\alpha)\\| \\geqslant 17\\epsilon/10 - \\epsilon/2 = 6\\epsilon/5>\\epsilon$. However, since $0<\\alpha \\leqslant \\hat{\\alpha} \\leqslant \\bar{\\alpha}$ and $x_0 \\in B(x^*,\\delta)$, by stability of $x^*$ we have $x_K \\in B(x^*,\\epsilon)$. We have reached a contradiction. \n\\end{proof}","units":[{"raw_tex":"\\begin{lemma}\n\\label{lemma:euler_method}\n    Let $F:\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^n$ be an upper semicontinuous mapping with nonempty, compact, and convex values. Let $X_0$ be a compact subset of $\\mathbb{R}^n$ and let $T>0$. Assume that there exist $\\bar{\\alpha},r>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$ and for all sequence $(x_k)_{k\\in\\mathbb{N}}$ such that\n    \\begin{equation}\n    \\label{eq:Euler}\n                x_{k+1} \\in x_k + \\alpha F(x_k), ~~~ \\forall k \\in \\mathbb{N},~~~ x_0 \\in X_0,\n    \\end{equation}\n    we have that $x_0,\\hdots,x_{\\lfloor T/\\alpha \\rfloor+1} \\in B(0,r)$. For all $\\epsilon>0$, there exists $\\hat{\\alpha}\\in (0,\\bar{\\alpha}]$ such that for all $\\alpha \\in (0,\\hat{\\alpha}]$ and for all sequence $(x_k)_{k\\in\\mathbb{N}}$ satisfying \\eqref{eq:Euler}, there exists an absolutely continuous function $x:[0,T]\\rightarrow\\mathbb{R}^n$ such that\n    \\begin{equation}\n    \\label{eq:ivp}\n                x'(t) \\in F(x(t)),~~~ \\text{for a.e.}~ \n                t \\in (0,T), ~~~ x(0) \\in X_0, \n    \\end{equation}\n    for which $\\|\\bar{x}(t) - x(t) \\| \\leqslant \\epsilon$ for all $t \\in [0,T]$, where $\\bar{x}:[0,T]\\rightarrow\\mathbb{R}^n$ is defined by $\\bar{x}(t) := x_k + (t-\\alpha k)/\\alpha  (x_{k+1}-x_k)$ for all $t\\in [\\alpha k , \\min\\{\\alpha(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha\\rfloor\\}$.\n\\end{lemma}","start":8563,"end":9928},{"raw_tex":"\\begin{proof}\nLet $(\\alpha_m)_{m \\in \\mathbb{N}}$ denote a sequence of positive numbers that converges to zero. Without loss of generality, we may assume that the sequence is bounded above by $\\bar{\\alpha}$. To each term in the sequence, we attribute a sequence $(x_k^{m})_{k\\in\\mathbb{N}}$ generated by the Euler method with step size $\\alpha_m$ and initialized in $X_0$, that is to say, which satisfies \\eqref{eq:Euler} with $\\alpha:=\\alpha_m$. By assumption, $x_0^{m},\\hdots,x_{\\lfloor T/\\alpha_m \\rfloor+1}^{m} \\in B(0,r)$. Consider the linear interpolation of those iterates, that is to say, the function $\\bar{x}^{m}:[0,T]\\rightarrow\\mathbb{R}^n$ defined by $\\bar{x}^{m}(t) := x_k^{m} + (t-\\alpha_m k)(x_{k+1}^{m}-x_k^{m})/\\alpha_m$ for all $t\\in [\\alpha_m k , \\min\\{\\alpha_m(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$. Since $B(0,r)$ is convex, it holds that $\\|\\bar{x}^{m}(t)\\| \\leqslant r $ for all $t\\in [0,T]$. In addition, since $F$ is upper semicontinuous and compact valued, by \\cite[Proposition 3 p. 42]{aubin1984differential} there exists $r'>0$ such that $F(B(0,r)) \\subset B(0,r')$. Observe that $(\\bar{x}^{m})'(t) = (x_{k+1}^{m}-x_k^{m})/\\alpha_m \\in F(x_k^{m})$ for all $t\\in (\\alpha_m k , \\min\\{\\alpha_m(k+1),T\\})$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$. Hence, we have that $\\|(\\bar{x}^{m})'(t)\\| \\leqslant r'$ for almost every $t\\in (0,T)$. By successively applying the Arzel\\`a-Ascoli and the Banach-Alaoglu theorems (see \\cite[Theorem 4 p. 13]{aubin1984differential}), there exists a subsequence (again denoted $(\\alpha_{m})_{m\\in \\mathbb{N}}$) and an absolutely continuous function $x:[0,T]\\rightarrow \\mathbb{R}^n$ such that $\\bar{x}^{m}(\\cdot)$ converges uniformly to $x(\\cdot)$ and $(\\bar{x}^{m})'(\\cdot)$ converges weakly to $x'(\\cdot)$ in $L^1([0,T],\\mathbb{R}^n)$. Furthermore, for all $t\\in (\\alpha_{m} k , \\min\\{\\alpha_{m}(k+1),T\\})$ and $k \\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$, observe that\n\\begin{subequations}\n\\begin{align*}\n    (\\bar{x}^{m}(t),(\\bar{x}^{m})'(t)) \n    & = \\left(x_k^{m} + (t-\\alpha_m k)\\frac{x_{k+1}^{m}-x_k^{m}}{\\alpha_m},\\frac{x_{k+1}^{m}-x_k^{m}}{\\alpha_m}\\right) \\\\\n    & \\in \\left(\\{x_k^{m}\\} + (t-\\alpha_m k)F(x_k^{m})\\right) \\times F(x_k^{m})  \\\\\n    & = \\{x_k^{m}\\} \\times F(x_k^{m}) + (t-\\alpha_m k)F(x_k^{m}) \\times \\{0\\}  \\\\\n    & \\subset \\mathrm{graph}(F) + B(0,r'\\alpha_m)\\times \\{0\\}.\n\\end{align*}\n\\end{subequations}\nAccording to \\cite[Convergence Theorem p. 60]{aubin1984differential}, it follows that $x'(t) \\in F(x(t))$ for almost every $t\\in (0,T)$. The sequence of initial points $(x^{m}(0))_{m\\in\\mathbb{N}}$ lies in the closed set $X_0$, hence its limit $x(0)$ lies in $X_0$ as well. As a result, $x(\\cdot)$ is a solution to the differential inclusion \\eqref{eq:ivp}. \n\nTo sum up, we have shown that for every sequence $(\\alpha_m)_{m \\in \\mathbb{N}}$ of positive numbers converging to zero, there exists a subsequence for which the corresponding linear interpolations uniformly converge towards a solution of the differential inclusion \\eqref{eq:ivp}. The conclusion of the theorem now easily follows. To see why, one can reason by contradiction and assume that there exists $\\epsilon>0$ such that for all $\\hat{\\alpha} \\in (0,\\bar{\\alpha}]$, there exist $\\alpha \\in (0,\\hat{\\alpha}]$ and a sequence $(x_k)_{k \\in \\mathbb{N}}$ generated by the Euler method with step size $\\alpha$ and initialized in $X_0$ such that, for any solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp}, it holds that $\\|\\bar{x}(t) - x(t) \\| > \\epsilon$ for some $t \\in [0,T]$. We can then generate a sequence $(\\alpha_m)_{m \\in \\mathbb{N}}$ of positive numbers converging to zero such that, for any solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp}, it holds that $\\|\\bar{x}^{m}(t) - x(t) \\| > \\epsilon$ for some $t \\in [0,T]$. Since there exists a subsequence $(\\alpha_{\\varphi(m)})_{m \\in \\mathbb{N}}$ such that $\\bar{x}^{\\varphi(m)}$ uniformly converges to a solution to the differential inclusion \\eqref{eq:ivp}, we obtain a contradiction. \n\\end{proof}","start":9930,"end":14003},{"raw_tex":"\\begin{proof}\nLet $x^* \\in \\mathbb{R}^n$ denote a stable point of a locally Lipschitz semi-algebraic function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. We reason by contradiction and assume that $x^*$ is not a local minimum of $f$. Given $x\\in \\mathbb{R}^n$ and $S \\subset \\mathbb{R}^n$, let $d(x,S) := \\inf \\{\\|x - y\\| : y \\in S\\}$. According to the Kurdyka-\\L{}ojasiewicz inequality \\cite[Theorem 14]{bolte2007clarke} (see also \\cite[Theorem 4.1]{attouch2010proximal}), there exist $r,\\rho>0$ and a strictly increasing concave continuous semi-algebraic function $\\psi:[0,\\rho) \\rightarrow [0,\\infty)$ that is continuously differentiable on $(0,\\rho)$ with $\\psi(0) = 0$ such that $d(0,\\partial f(x)) \\geqslant 1/\\psi'(|f(x) - f(x^*)|)$ for all $x \\in B(x^*,r)$ whenever $0<|f(x)-f(x^*)|< \\rho$. After possibly reducing $r$, the inequality holds for all $x \\in B(x^*,r)$ such that $f(x) \\neq f(x^*)$. \n\nLet $\\epsilon \\in (0,r/2)$. By the definition of stability (Definition \\ref{def:discrete_lyapunov}), there exist $\\delta>0$ and $\\bar{\\alpha}>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$, the subgradient method with constant step size $\\alpha$ initialized in $B(x^*,\\delta)$ has all its iterates in $B(x^*,\\epsilon)$. Since $x^*$ is not a local minimum, we can take an initial iterate $x_0$ in $B(x^*,\\delta)$ such that $f(x_0)<f(x^*)$. Let $L>0$ denote a Lipschitz constant of $f$ on $B(x^*,r)$ and let $T := (2\\epsilon + \\delta)L\\psi'(f(x^*) - f(x_0))^2>0$.\nConsider the differential inclusion\n\\begin{equation}\n\\label{eq:ivp_x_0}\n     x'(t) \\in - \\partial f(x(t)), ~~~ \\text{for a.e.}~ \n     t \\in (0,T),~~~ x(0) = x_0.\n\\end{equation}\nSince $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d)]{clarke1990} with nonempty, compact and convex values \\cite[2.1.2 Proposition (a)]{clarke1990}. By Lemma \\ref{lemma:euler_method}, there exists $\\hat{\\alpha} \\in (0,\\bar{\\alpha}]$ such that, for all $\\alpha \\in (0,\\hat{\\alpha}]$  and for all sequence $(x_k)_{k\\in \\mathbb{N}}$ generated by the subgradient method with constant step size $\\alpha$ and initialized at $x_0$, there exists a solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp_x_0} for which $\\|\\bar{x}(t) - x(t)\\| \\leqslant \\epsilon/2$ for all $t \\in [0,T]$, where $\\bar{x}:[0,T]\\rightarrow\\mathbb{R}^n$ is the piecewise linear function defined by $\\bar{x}(t) := x_k + (t-\\alpha k)/\\alpha  (x_{k+1}-x_k)$ for all $t\\in [\\alpha k , \\min\\{\\alpha(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha\\rfloor\\}$.\n\nLet us fix some $\\alpha \\in (0,\\min\\{\\hat{\\alpha},T/10\\}]$ from now on. Consider a sequence $(x_k)_{k\\in\\mathbb{N}}$ generated by the subgradient method with constant step size $\\alpha$ and initialized at $x_0$. Consider also the linear interpolation $\\bar{x}(\\cdot)$ of those iterates up to iteration $K+1$ where $K := \\lfloor T/\\alpha \\rfloor$, as well as a solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp_x_0} such that $\\|\\bar{x}(t)-x(t)\\| \\leqslant \\epsilon/2$ for all $t\\in [0,T]$. Since $f$ is semi-algebraic, by \\cite[Lemma 5.2]{davis2020stochastic} (see also \\cite{drusvyatskiy2015curves}) it holds that\n\\begin{equation}\n\\label{eq:lyapunov}\n    f(x(t)) - f(x(0)) = - \\int_0^t d(0,\\partial f(x(\\tau)))^2 d\\tau, ~~~ \\forall t \\in [0,T].\n\\end{equation}\nAs a result, $f(x(t)) \\leqslant f(x(0)) = f(x_0)$ for all $t \\in [0,T]$. Also, $\\|x(t) - x^*\\| \\leqslant \\|x(t) - \\bar{x}(t)\\| + \\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon/2 + \\epsilon = 3\\epsilon/2 \\leqslant r$, where the inequality $\\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon$ follows from the convexity of $B(x^*,\\epsilon)$. Hence $0<f(x^*) - f(x_0) \\leqslant f(x^*) - f(x(t))$ and $d(0,\\partial f(x(t))) \\geqslant  1/\\psi'(f(x^*)-f(x(t))) \\geqslant 1/\\psi'(f(x^*)-f(x_0))$ for all $t\\in[0,T]$ by concavity of $\\psi$. Together with \\eqref{eq:lyapunov}, it follows that $f(x(K\\alpha)) - f(x_0) \\leqslant -K\\alpha M^2$ where $K := \\lfloor T/\\alpha \\rfloor$ and $M:= 1/\\psi'(f(x^*)-f(x_0))$. Recall that $L$ is a Lipschitz constant of $f$ on $B(x^*,r)$, so that we have $|f(x(K\\alpha))-f(x_0)| \\leqslant L \\|x(K\\alpha)-x_0\\|$. We thus obtain the lower bound $\\|x(K\\alpha)-x_0\\| \\geqslant K\\alpha M^2/L = \\lfloor T/\\alpha \\rfloor \\alpha M^2/L \\geqslant (T-\\alpha)M^2/L \\geqslant (T-T/10)M^2/L = 9/10( 2\\epsilon + \\delta) $ (recall that $T = (2\\epsilon + \\delta)L/M^2$). Hence $\\|x(K\\alpha)-x^*\\| \\geqslant \\|x(K\\alpha)-x_0\\| - \\|x_0-x^*\\| \\geqslant 9/10(2\\epsilon + \\delta) - \\delta = 9 \\epsilon/5 - \\delta/10 \\geqslant 9 \\epsilon/5 - \\epsilon/10 = 17\\epsilon/10$ and finally $\\|x_K-x^*\\| \\geqslant \\|x(K\\alpha)-x^*\\| -\\|x_K-x(K\\alpha)\\| \\geqslant 17\\epsilon/10 - \\epsilon/2 = 6\\epsilon/5>\\epsilon$. However, since $0<\\alpha \\leqslant \\hat{\\alpha} \\leqslant \\bar{\\alpha}$ and $x_0 \\in B(x^*,\\delta)$, by stability of $x^*$ we have $x_K \\in B(x^*,\\epsilon)$. We have reached a contradiction. \n\\end{proof}","start":14753,"end":19670}]},"nodes":[{"id":"S001","semantic_type":"algorithm","kind":"algorithm","labels":[],"declared_external":false,"source_only":false,"title":"常步长次梯度法","source_units":[{"raw_tex":"The subgradient method with constant step size for minimizing a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ consists in choosing an initial point $x_0 \\in \\mathbb{R}^n$ and generating a sequence of iterates according to the update rule $x_{k+1} \\in x_k - \\alpha \\partial f(x_k), ~ \\forall k \\in \\mathbb{N} := \\{0,1,2,\\hdots\\}$, where $\\alpha>0$ is the step size and $\\partial f$ is the Clarke subdifferential \\cite[Chapter 2]{clarke1990}. While this method is often used in practice to solve nonconvex and nonsmooth problems, there is little theoretical understanding of the behavior of its iterates. To the best of our knowledge, the only known results are in the convex setting, as we next describe.\n","start":2451,"end":3174}]},{"id":"S002","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"非凸分析背景","source_units":[{"raw_tex":"The subgradient method with constant step size for minimizing a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ consists in choosing an initial point $x_0 \\in \\mathbb{R}^n$ and generating a sequence of iterates according to the update rule $x_{k+1} \\in x_k - \\alpha \\partial f(x_k), ~ \\forall k \\in \\mathbb{N} := \\{0,1,2,\\hdots\\}$, where $\\alpha>0$ is the step size and $\\partial f$ is the Clarke subdifferential \\cite[Chapter 2]{clarke1990}. While this method is often used in practice to solve nonconvex and nonsmooth problems, there is little theoretical understanding of the behavior of its iterates. To the best of our knowledge, the only known results are in the convex setting, as we next describe.\n","start":2451,"end":3174}]},{"id":"S003","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"内积范数与球","source_units":[{"raw_tex":"Let $\\|\\cdot\\|$ be the induced norm of an inner product $\\langle \\cdot, \\cdot\\rangle$ on $\\mathbb{R}^n$. Let $B(a,r)$ and $\\mathring{B}(a,r)$ respectively denote the closed ball and the open ball of center $a\\in \\mathbb{R}^n$ and radius $r>0$. We next define the notion of discrete Lyupanov stability.\n\n","start":7298,"end":7601}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["def:discrete_lyapunov"],"declared_external":false,"source_only":false,"title":"离散稳定性","source_units":[{"raw_tex":"\\begin{definition}\n\\label{def:discrete_lyapunov}\n\\normalfont\nWe say that $x^*\\in \\mathbb{R}^n$ is a stable point of a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if for all $\\epsilon>0$, there exist $\\delta>0$ and $\\bar{\\alpha}>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$, the subgradient method with constant step size $\\alpha$ initialized in $B(x^*,\\delta)$ has all its iterates in $B(x^*,\\epsilon)$.\n\\end{definition}\n","start":7601,"end":8049}]},{"id":"S005","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"负次微分特化","source_units":[{"raw_tex":"We will apply Lemma \\ref{lemma:euler_method} to the case where the set-valued mapping $F$ is the opposite of the Clarke subdifferential of a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. Recall that a point $x^* \\in \\mathbb{R}^n$ is a local minimum (respectively strict local minimum) of a function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if there exists a positive constant $\\epsilon$ such that $f(x^*) \\leqslant f(x)$ for all $x \\in B(x^*,\\epsilon)\\setminus \\{x^*\\}$ (respectively $f(x^*) < f(x)$). Using Lemma \\ref{lemma:euler_method}, we obtain the following necessary condition for stability.\n","start":14005,"end":14621}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"局部极小定义","source_units":[{"raw_tex":"We will apply Lemma \\ref{lemma:euler_method} to the case where the set-valued mapping $F$ is the opposite of the Clarke subdifferential of a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. Recall that a point $x^* \\in \\mathbb{R}^n$ is a local minimum (respectively strict local minimum) of a function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if there exists a positive constant $\\epsilon$ such that $f(x^*) \\leqslant f(x)$ for all $x \\in B(x^*,\\epsilon)\\setminus \\{x^*\\}$ (respectively $f(x^*) < f(x)$). Using Lemma \\ref{lemma:euler_method}, we obtain the following necessary condition for stability.\n","start":14005,"end":14621}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"严格极小定义","source_units":[{"raw_tex":"We will apply Lemma \\ref{lemma:euler_method} to the case where the set-valued mapping $F$ is the opposite of the Clarke subdifferential of a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. Recall that a point $x^* \\in \\mathbb{R}^n$ is a local minimum (respectively strict local minimum) of a function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if there exists a positive constant $\\epsilon$ such that $f(x^*) \\leqslant f(x)$ for all $x \\in B(x^*,\\epsilon)\\setminus \\{x^*\\}$ (respectively $f(x^*) < f(x)$). Using Lemma \\ref{lemma:euler_method}, we obtain the following necessary condition for stability.\n","start":14005,"end":14621}]},{"id":"S008","semantic_type":"lemma","kind":"lemma","labels":["eq:Euler","eq:ivp","lemma:euler_method"],"declared_external":false,"source_only":false,"title":"有界 Euler 轨道逼近","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lemma:euler_method}\n    Let $F:\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^n$ be an upper semicontinuous mapping with nonempty, compact, and convex values. Let $X_0$ be a compact subset of $\\mathbb{R}^n$ and let $T>0$. Assume that there exist $\\bar{\\alpha},r>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$ and for all sequence $(x_k)_{k\\in\\mathbb{N}}$ such that\n    \\begin{equation}\n    \\label{eq:Euler}\n                x_{k+1} \\in x_k + \\alpha F(x_k), ~~~ \\forall k \\in \\mathbb{N},~~~ x_0 \\in X_0,\n    \\end{equation}\n    we have that $x_0,\\hdots,x_{\\lfloor T/\\alpha \\rfloor+1} \\in B(0,r)$. For all $\\epsilon>0$, there exists $\\hat{\\alpha}\\in (0,\\bar{\\alpha}]$ such that for all $\\alpha \\in (0,\\hat{\\alpha}]$ and for all sequence $(x_k)_{k\\in\\mathbb{N}}$ satisfying \\eqref{eq:Euler}, there exists an absolutely continuous function $x:[0,T]\\rightarrow\\mathbb{R}^n$ such that\n    \\begin{equation}\n    \\label{eq:ivp}\n                x'(t) \\in F(x(t)),~~~ \\text{for a.e.}~ \n                t \\in (0,T), ~~~ x(0) \\in X_0, \n    \\end{equation}\n    for which $\\|\\bar{x}(t) - x(t) \\| \\leqslant \\epsilon$ for all $t \\in [0,T]$, where $\\bar{x}:[0,T]\\rightarrow\\mathbb{R}^n$ is defined by $\\bar{x}(t) := x_k + (t-\\alpha k)/\\alpha  (x_{k+1}-x_k)$ for all $t\\in [\\alpha k , \\min\\{\\alpha(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha\\rfloor\\}$.\n\\end{lemma}","start":8563,"end":9928}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","labels":["lemma:euler_method"],"declared_external":false,"source_only":false,"title":"集合值映射条件","source_units":[{"raw_tex":"    Let $F:\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^n$ be an upper semicontinuous mapping with nonempty, compact, and convex values. Let $X_0$ be a compact subset of $\\mathbb{R}^n$ and let $T>0$. Assume that there exist $\\bar{\\alpha},r>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$ and for all sequence $(x_k)_{k\\in\\mathbb{N}}$ such that\n","start":8604,"end":8946}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","labels":["lemma:euler_method"],"declared_external":false,"source_only":false,"title":"紧初值集与时域","source_units":[{"raw_tex":"    Let $F:\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^n$ be an upper semicontinuous mapping with nonempty, compact, and convex values. Let $X_0$ be a compact subset of $\\mathbb{R}^n$ and let $T>0$. Assume that there exist $\\bar{\\alpha},r>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$ and for all sequence $(x_k)_{k\\in\\mathbb{N}}$ such that\n","start":8604,"end":8946}]},{"id":"S011","semantic_type":"algorithm","kind":"algorithm","labels":["eq:Euler","lemma:euler_method"],"declared_external":false,"source_only":false,"title":"集合值 Euler 迭代","source_units":[{"raw_tex":"    \\begin{equation}\n    \\label{eq:Euler}\n                x_{k+1} \\in x_k + \\alpha F(x_k), ~~~ \\forall k \\in \\mathbb{N},~~~ x_0 \\in X_0,\n    \\end{equation}\n","start":8946,"end":9102}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","labels":["lemma:euler_method"],"declared_external":false,"source_only":false,"title":"离散轨道一致有界","source_units":[{"raw_tex":"    Let $F:\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^n$ be an upper semicontinuous mapping with nonempty, compact, and convex values. Let $X_0$ be a compact subset of $\\mathbb{R}^n$ and let $T>0$. Assume that there exist $\\bar{\\alpha},r>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$ and for all sequence $(x_k)_{k\\in\\mathbb{N}}$ such that\n","start":8604,"end":8946},{"raw_tex":"    we have that $x_0,\\hdots,x_{\\lfloor T/\\alpha \\rfloor+1} \\in B(0,r)$. For all $\\epsilon>0$, there exists $\\hat{\\alpha}\\in (0,\\bar{\\alpha}]$ such that for all $\\alpha \\in (0,\\hat{\\alpha}]$ and for all sequence $(x_k)_{k\\in\\mathbb{N}}$ satisfying \\eqref{eq:Euler}, there exists an absolutely continuous function $x:[0,T]\\rightarrow\\mathbb{R}^n$ such that\n","start":9102,"end":9458}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["eq:ivp","lemma:euler_method"],"declared_external":false,"source_only":false,"title":"极限微分包含","source_units":[{"raw_tex":"    we have that $x_0,\\hdots,x_{\\lfloor T/\\alpha \\rfloor+1} \\in B(0,r)$. For all $\\epsilon>0$, there exists $\\hat{\\alpha}\\in (0,\\bar{\\alpha}]$ such that for all $\\alpha \\in (0,\\hat{\\alpha}]$ and for all sequence $(x_k)_{k\\in\\mathbb{N}}$ satisfying \\eqref{eq:Euler}, there exists an absolutely continuous function $x:[0,T]\\rightarrow\\mathbb{R}^n$ such that\n    \\begin{equation}\n    \\label{eq:ivp}\n                x'(t) \\in F(x(t)),~~~ \\text{for a.e.}~ \n                t \\in (0,T), ~~~ x(0) \\in X_0, \n    \\end{equation}\n","start":9102,"end":9621}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":["lemma:euler_method"],"declared_external":false,"source_only":false,"title":"分段线性插值","source_units":[{"raw_tex":"    for which $\\|\\bar{x}(t) - x(t) \\| \\leqslant \\epsilon$ for all $t \\in [0,T]$, where $\\bar{x}:[0,T]\\rightarrow\\mathbb{R}^n$ is defined by $\\bar{x}(t) := x_k + (t-\\alpha k)/\\alpha  (x_{k+1}-x_k)$ for all $t\\in [\\alpha k , \\min\\{\\alpha(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha\\rfloor\\}$.\n","start":9621,"end":9917}]},{"id":"S015","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"Euler 引理完整证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $(\\alpha_m)_{m \\in \\mathbb{N}}$ denote a sequence of positive numbers that converges to zero. Without loss of generality, we may assume that the sequence is bounded above by $\\bar{\\alpha}$. To each term in the sequence, we attribute a sequence $(x_k^{m})_{k\\in\\mathbb{N}}$ generated by the Euler method with step size $\\alpha_m$ and initialized in $X_0$, that is to say, which satisfies \\eqref{eq:Euler} with $\\alpha:=\\alpha_m$. By assumption, $x_0^{m},\\hdots,x_{\\lfloor T/\\alpha_m \\rfloor+1}^{m} \\in B(0,r)$. Consider the linear interpolation of those iterates, that is to say, the function $\\bar{x}^{m}:[0,T]\\rightarrow\\mathbb{R}^n$ defined by $\\bar{x}^{m}(t) := x_k^{m} + (t-\\alpha_m k)(x_{k+1}^{m}-x_k^{m})/\\alpha_m$ for all $t\\in [\\alpha_m k , \\min\\{\\alpha_m(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$. Since $B(0,r)$ is convex, it holds that $\\|\\bar{x}^{m}(t)\\| \\leqslant r $ for all $t\\in [0,T]$. In addition, since $F$ is upper semicontinuous and compact valued, by \\cite[Proposition 3 p. 42]{aubin1984differential} there exists $r'>0$ such that $F(B(0,r)) \\subset B(0,r')$. Observe that $(\\bar{x}^{m})'(t) = (x_{k+1}^{m}-x_k^{m})/\\alpha_m \\in F(x_k^{m})$ for all $t\\in (\\alpha_m k , \\min\\{\\alpha_m(k+1),T\\})$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$. Hence, we have that $\\|(\\bar{x}^{m})'(t)\\| \\leqslant r'$ for almost every $t\\in (0,T)$. By successively applying the Arzel\\`a-Ascoli and the Banach-Alaoglu theorems (see \\cite[Theorem 4 p. 13]{aubin1984differential}), there exists a subsequence (again denoted $(\\alpha_{m})_{m\\in \\mathbb{N}}$) and an absolutely continuous function $x:[0,T]\\rightarrow \\mathbb{R}^n$ such that $\\bar{x}^{m}(\\cdot)$ converges uniformly to $x(\\cdot)$ and $(\\bar{x}^{m})'(\\cdot)$ converges weakly to $x'(\\cdot)$ in $L^1([0,T],\\mathbb{R}^n)$. Furthermore, for all $t\\in (\\alpha_{m} k , \\min\\{\\alpha_{m}(k+1),T\\})$ and $k \\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$, observe that\n\\begin{subequations}\n\\begin{align*}\n    (\\bar{x}^{m}(t),(\\bar{x}^{m})'(t)) \n    & = \\left(x_k^{m} + (t-\\alpha_m k)\\frac{x_{k+1}^{m}-x_k^{m}}{\\alpha_m},\\frac{x_{k+1}^{m}-x_k^{m}}{\\alpha_m}\\right) \\\\\n    & \\in \\left(\\{x_k^{m}\\} + (t-\\alpha_m k)F(x_k^{m})\\right) \\times F(x_k^{m})  \\\\\n    & = \\{x_k^{m}\\} \\times F(x_k^{m}) + (t-\\alpha_m k)F(x_k^{m}) \\times \\{0\\}  \\\\\n    & \\subset \\mathrm{graph}(F) + B(0,r'\\alpha_m)\\times \\{0\\}.\n\\end{align*}\n\\end{subequations}\nAccording to \\cite[Convergence Theorem p. 60]{aubin1984differential}, it follows that $x'(t) \\in F(x(t))$ for almost every $t\\in (0,T)$. The sequence of initial points $(x^{m}(0))_{m\\in\\mathbb{N}}$ lies in the closed set $X_0$, hence its limit $x(0)$ lies in $X_0$ as well. As a result, $x(\\cdot)$ is a solution to the differential inclusion \\eqref{eq:ivp}. \n\nTo sum up, we have shown that for every sequence $(\\alpha_m)_{m \\in \\mathbb{N}}$ of positive numbers converging to zero, there exists a subsequence for which the corresponding linear interpolations uniformly converge towards a solution of the differential inclusion \\eqref{eq:ivp}. The conclusion of the theorem now easily follows. To see why, one can reason by contradiction and assume that there exists $\\epsilon>0$ such that for all $\\hat{\\alpha} \\in (0,\\bar{\\alpha}]$, there exist $\\alpha \\in (0,\\hat{\\alpha}]$ and a sequence $(x_k)_{k \\in \\mathbb{N}}$ generated by the Euler method with step size $\\alpha$ and initialized in $X_0$ such that, for any solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp}, it holds that $\\|\\bar{x}(t) - x(t) \\| > \\epsilon$ for some $t \\in [0,T]$. We can then generate a sequence $(\\alpha_m)_{m \\in \\mathbb{N}}$ of positive numbers converging to zero such that, for any solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp}, it holds that $\\|\\bar{x}^{m}(t) - x(t) \\| > \\epsilon$ for some $t \\in [0,T]$. Since there exists a subsequence $(\\alpha_{\\varphi(m)})_{m \\in \\mathbb{N}}$ such that $\\bar{x}^{\\varphi(m)}$ uniformly converges to a solution to the differential inclusion \\eqref{eq:ivp}, we obtain a contradiction. \n\\end{proof}","start":9930,"end":14003}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"趋零步长与插值","source_units":[{"raw_tex":"Let $(\\alpha_m)_{m \\in \\mathbb{N}}$ denote a sequence of positive numbers that converges to zero. Without loss of generality, we may assume that the sequence is bounded above by $\\bar{\\alpha}$. To each term in the sequence, we attribute a sequence $(x_k^{m})_{k\\in\\mathbb{N}}$ generated by the Euler method with step size $\\alpha_m$ and initialized in $X_0$, that is to say, which satisfies \\eqref{eq:Euler} with $\\alpha:=\\alpha_m$. By assumption, $x_0^{m},\\hdots,x_{\\lfloor T/\\alpha_m \\rfloor+1}^{m} \\in B(0,r)$. Consider the linear interpolation of those iterates, that is to say, the function $\\bar{x}^{m}:[0,T]\\rightarrow\\mathbb{R}^n$ defined by $\\bar{x}^{m}(t) := x_k^{m} + (t-\\alpha_m k)(x_{k+1}^{m}-x_k^{m})/\\alpha_m$ for all $t\\in [\\alpha_m k , \\min\\{\\alpha_m(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$. Since $B(0,r)$ is convex, it holds that $\\|\\bar{x}^{m}(t)\\| \\leqslant r $ for all $t\\in [0,T]$. In addition, since $F$ is upper semicontinuous and compact valued, by \\cite[Proposition 3 p. 42]{aubin1984differential} there exists $r'>0$ such that $F(B(0,r)) \\subset B(0,r')$. Observe that $(\\bar{x}^{m})'(t) = (x_{k+1}^{m}-x_k^{m})/\\alpha_m \\in F(x_k^{m})$ for all $t\\in (\\alpha_m k , \\min\\{\\alpha_m(k+1),T\\})$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$. Hence, we have that $\\|(\\bar{x}^{m})'(t)\\| \\leqslant r'$ for almost every $t\\in (0,T)$. By successively applying the Arzel\\`a-Ascoli and the Banach-Alaoglu theorems (see \\cite[Theorem 4 p. 13]{aubin1984differential}), there exists a subsequence (again denoted $(\\alpha_{m})_{m\\in \\mathbb{N}}$) and an absolutely continuous function $x:[0,T]\\rightarrow \\mathbb{R}^n$ such that $\\bar{x}^{m}(\\cdot)$ converges uniformly to $x(\\cdot)$ and $(\\bar{x}^{m})'(\\cdot)$ converges weakly to $x'(\\cdot)$ in $L^1([0,T],\\mathbb{R}^n)$. Furthermore, for all $t\\in (\\alpha_{m} k , \\min\\{\\alpha_{m}(k+1),T\\})$ and $k \\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$, observe that\n","start":9944,"end":11895}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"插值位置有界","source_units":[{"raw_tex":"Let $(\\alpha_m)_{m \\in \\mathbb{N}}$ denote a sequence of positive numbers that converges to zero. Without loss of generality, we may assume that the sequence is bounded above by $\\bar{\\alpha}$. To each term in the sequence, we attribute a sequence $(x_k^{m})_{k\\in\\mathbb{N}}$ generated by the Euler method with step size $\\alpha_m$ and initialized in $X_0$, that is to say, which satisfies \\eqref{eq:Euler} with $\\alpha:=\\alpha_m$. By assumption, $x_0^{m},\\hdots,x_{\\lfloor T/\\alpha_m \\rfloor+1}^{m} \\in B(0,r)$. Consider the linear interpolation of those iterates, that is to say, the function $\\bar{x}^{m}:[0,T]\\rightarrow\\mathbb{R}^n$ defined by $\\bar{x}^{m}(t) := x_k^{m} + (t-\\alpha_m k)(x_{k+1}^{m}-x_k^{m})/\\alpha_m$ for all $t\\in [\\alpha_m k , \\min\\{\\alpha_m(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$. Since $B(0,r)$ is convex, it holds that $\\|\\bar{x}^{m}(t)\\| \\leqslant r $ for all $t\\in [0,T]$. In addition, since $F$ is upper semicontinuous and compact valued, by \\cite[Proposition 3 p. 42]{aubin1984differential} there exists $r'>0$ such that $F(B(0,r)) \\subset B(0,r')$. Observe that $(\\bar{x}^{m})'(t) = (x_{k+1}^{m}-x_k^{m})/\\alpha_m \\in F(x_k^{m})$ for all $t\\in (\\alpha_m k , \\min\\{\\alpha_m(k+1),T\\})$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$. Hence, we have that $\\|(\\bar{x}^{m})'(t)\\| \\leqslant r'$ for almost every $t\\in (0,T)$. By successively applying the Arzel\\`a-Ascoli and the Banach-Alaoglu theorems (see \\cite[Theorem 4 p. 13]{aubin1984differential}), there exists a subsequence (again denoted $(\\alpha_{m})_{m\\in \\mathbb{N}}$) and an absolutely continuous function $x:[0,T]\\rightarrow \\mathbb{R}^n$ such that $\\bar{x}^{m}(\\cdot)$ converges uniformly to $x(\\cdot)$ and $(\\bar{x}^{m})'(\\cdot)$ converges weakly to $x'(\\cdot)$ in $L^1([0,T],\\mathbb{R}^n)$. Furthermore, for all $t\\in (\\alpha_{m} k , \\min\\{\\alpha_{m}(k+1),T\\})$ and $k \\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$, observe that\n","start":9944,"end":11895}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"插值速度一致有界","source_units":[{"raw_tex":"Let $(\\alpha_m)_{m \\in \\mathbb{N}}$ denote a sequence of positive numbers that converges to zero. Without loss of generality, we may assume that the sequence is bounded above by $\\bar{\\alpha}$. To each term in the sequence, we attribute a sequence $(x_k^{m})_{k\\in\\mathbb{N}}$ generated by the Euler method with step size $\\alpha_m$ and initialized in $X_0$, that is to say, which satisfies \\eqref{eq:Euler} with $\\alpha:=\\alpha_m$. By assumption, $x_0^{m},\\hdots,x_{\\lfloor T/\\alpha_m \\rfloor+1}^{m} \\in B(0,r)$. Consider the linear interpolation of those iterates, that is to say, the function $\\bar{x}^{m}:[0,T]\\rightarrow\\mathbb{R}^n$ defined by $\\bar{x}^{m}(t) := x_k^{m} + (t-\\alpha_m k)(x_{k+1}^{m}-x_k^{m})/\\alpha_m$ for all $t\\in [\\alpha_m k , \\min\\{\\alpha_m(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$. Since $B(0,r)$ is convex, it holds that $\\|\\bar{x}^{m}(t)\\| \\leqslant r $ for all $t\\in [0,T]$. In addition, since $F$ is upper semicontinuous and compact valued, by \\cite[Proposition 3 p. 42]{aubin1984differential} there exists $r'>0$ such that $F(B(0,r)) \\subset B(0,r')$. Observe that $(\\bar{x}^{m})'(t) = (x_{k+1}^{m}-x_k^{m})/\\alpha_m \\in F(x_k^{m})$ for all $t\\in (\\alpha_m k , \\min\\{\\alpha_m(k+1),T\\})$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$. Hence, we have that $\\|(\\bar{x}^{m})'(t)\\| \\leqslant r'$ for almost every $t\\in (0,T)$. By successively applying the Arzel\\`a-Ascoli and the Banach-Alaoglu theorems (see \\cite[Theorem 4 p. 13]{aubin1984differential}), there exists a subsequence (again denoted $(\\alpha_{m})_{m\\in \\mathbb{N}}$) and an absolutely continuous function $x:[0,T]\\rightarrow \\mathbb{R}^n$ such that $\\bar{x}^{m}(\\cdot)$ converges uniformly to $x(\\cdot)$ and $(\\bar{x}^{m})'(\\cdot)$ converges weakly to $x'(\\cdot)$ in $L^1([0,T],\\mathbb{R}^n)$. Furthermore, for all $t\\in (\\alpha_{m} k , \\min\\{\\alpha_{m}(k+1),T\\})$ and $k \\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$, observe that\n","start":9944,"end":11895}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"提取函数与导数极限","source_units":[{"raw_tex":"Let $(\\alpha_m)_{m \\in \\mathbb{N}}$ denote a sequence of positive numbers that converges to zero. Without loss of generality, we may assume that the sequence is bounded above by $\\bar{\\alpha}$. To each term in the sequence, we attribute a sequence $(x_k^{m})_{k\\in\\mathbb{N}}$ generated by the Euler method with step size $\\alpha_m$ and initialized in $X_0$, that is to say, which satisfies \\eqref{eq:Euler} with $\\alpha:=\\alpha_m$. By assumption, $x_0^{m},\\hdots,x_{\\lfloor T/\\alpha_m \\rfloor+1}^{m} \\in B(0,r)$. Consider the linear interpolation of those iterates, that is to say, the function $\\bar{x}^{m}:[0,T]\\rightarrow\\mathbb{R}^n$ defined by $\\bar{x}^{m}(t) := x_k^{m} + (t-\\alpha_m k)(x_{k+1}^{m}-x_k^{m})/\\alpha_m$ for all $t\\in [\\alpha_m k , \\min\\{\\alpha_m(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$. Since $B(0,r)$ is convex, it holds that $\\|\\bar{x}^{m}(t)\\| \\leqslant r $ for all $t\\in [0,T]$. In addition, since $F$ is upper semicontinuous and compact valued, by \\cite[Proposition 3 p. 42]{aubin1984differential} there exists $r'>0$ such that $F(B(0,r)) \\subset B(0,r')$. Observe that $(\\bar{x}^{m})'(t) = (x_{k+1}^{m}-x_k^{m})/\\alpha_m \\in F(x_k^{m})$ for all $t\\in (\\alpha_m k , \\min\\{\\alpha_m(k+1),T\\})$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$. Hence, we have that $\\|(\\bar{x}^{m})'(t)\\| \\leqslant r'$ for almost every $t\\in (0,T)$. By successively applying the Arzel\\`a-Ascoli and the Banach-Alaoglu theorems (see \\cite[Theorem 4 p. 13]{aubin1984differential}), there exists a subsequence (again denoted $(\\alpha_{m})_{m\\in \\mathbb{N}}$) and an absolutely continuous function $x:[0,T]\\rightarrow \\mathbb{R}^n$ such that $\\bar{x}^{m}(\\cdot)$ converges uniformly to $x(\\cdot)$ and $(\\bar{x}^{m})'(\\cdot)$ converges weakly to $x'(\\cdot)$ in $L^1([0,T],\\mathbb{R}^n)$. Furthermore, for all $t\\in (\\alpha_{m} k , \\min\\{\\alpha_{m}(k+1),T\\})$ and $k \\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$, observe that\n","start":9944,"end":11895}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"插值对接近映射图","source_units":[{"raw_tex":"\\begin{subequations}\n\\begin{align*}\n    (\\bar{x}^{m}(t),(\\bar{x}^{m})'(t)) \n    & = \\left(x_k^{m} + (t-\\alpha_m k)\\frac{x_{k+1}^{m}-x_k^{m}}{\\alpha_m},\\frac{x_{k+1}^{m}-x_k^{m}}{\\alpha_m}\\right) \\\\\n    & \\in \\left(\\{x_k^{m}\\} + (t-\\alpha_m k)F(x_k^{m})\\right) \\times F(x_k^{m})  \\\\\n    & = \\{x_k^{m}\\} \\times F(x_k^{m}) + (t-\\alpha_m k)F(x_k^{m}) \\times \\{0\\}  \\\\\n    & \\subset \\mathrm{graph}(F) + B(0,r'\\alpha_m)\\times \\{0\\}.\n\\end{align*}\n\\end{subequations}\n","start":11895,"end":12354}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"极限满足包含关系","source_units":[{"raw_tex":"According to \\cite[Convergence Theorem p. 60]{aubin1984differential}, it follows that $x'(t) \\in F(x(t))$ for almost every $t\\in (0,T)$. The sequence of initial points $(x^{m}(0))_{m\\in\\mathbb{N}}$ lies in the closed set $X_0$, hence its limit $x(0)$ lies in $X_0$ as well. As a result, $x(\\cdot)$ is a solution to the differential inclusion \\eqref{eq:ivp}. \n","start":12354,"end":12713}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"初值约束传递","source_units":[{"raw_tex":"According to \\cite[Convergence Theorem p. 60]{aubin1984differential}, it follows that $x'(t) \\in F(x(t))$ for almost every $t\\in (0,T)$. The sequence of initial points $(x^{m}(0))_{m\\in\\mathbb{N}}$ lies in the closed set $X_0$, hence its limit $x(0)$ lies in $X_0$ as well. As a result, $x(\\cdot)$ is a solution to the differential inclusion \\eqref{eq:ivp}. \n","start":12354,"end":12713}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"统一逼近的反证","source_units":[{"raw_tex":"To sum up, we have shown that for every sequence $(\\alpha_m)_{m \\in \\mathbb{N}}$ of positive numbers converging to zero, there exists a subsequence for which the corresponding linear interpolations uniformly converge towards a solution of the differential inclusion \\eqref{eq:ivp}. The conclusion of the theorem now easily follows. To see why, one can reason by contradiction and assume that there exists $\\epsilon>0$ such that for all $\\hat{\\alpha} \\in (0,\\bar{\\alpha}]$, there exist $\\alpha \\in (0,\\hat{\\alpha}]$ and a sequence $(x_k)_{k \\in \\mathbb{N}}$ generated by the Euler method with step size $\\alpha$ and initialized in $X_0$ such that, for any solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp}, it holds that $\\|\\bar{x}(t) - x(t) \\| > \\epsilon$ for some $t \\in [0,T]$. We can then generate a sequence $(\\alpha_m)_{m \\in \\mathbb{N}}$ of positive numbers converging to zero such that, for any solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp}, it holds that $\\|\\bar{x}^{m}(t) - x(t) \\| > \\epsilon$ for some $t \\in [0,T]$. Since there exists a subsequence $(\\alpha_{\\varphi(m)})_{m \\in \\mathbb{N}}$ such that $\\bar{x}^{\\varphi(m)}$ uniformly converges to a solution to the differential inclusion \\eqref{eq:ivp}, we obtain a contradiction. \n","start":12714,"end":13992}]},{"id":"S024","semantic_type":"theorem","kind":"theorem","labels":["thm:necessary"],"declared_external":false,"source_only":false,"title":"稳定点必为局部极小","source_units":[{"raw_tex":"\\begin{theorem}\n\\label{thm:necessary}\nStable points of locally Lipschitz semi-algebraic functions are local minima.\n\\end{theorem}","start":14622,"end":14751}]},{"id":"S025","semantic_type":"proof","kind":"proof","labels":["eq:ivp_x_0","eq:lyapunov"],"declared_external":false,"source_only":false,"title":"必要性定理完整证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $x^* \\in \\mathbb{R}^n$ denote a stable point of a locally Lipschitz semi-algebraic function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. We reason by contradiction and assume that $x^*$ is not a local minimum of $f$. Given $x\\in \\mathbb{R}^n$ and $S \\subset \\mathbb{R}^n$, let $d(x,S) := \\inf \\{\\|x - y\\| : y \\in S\\}$. According to the Kurdyka-\\L{}ojasiewicz inequality \\cite[Theorem 14]{bolte2007clarke} (see also \\cite[Theorem 4.1]{attouch2010proximal}), there exist $r,\\rho>0$ and a strictly increasing concave continuous semi-algebraic function $\\psi:[0,\\rho) \\rightarrow [0,\\infty)$ that is continuously differentiable on $(0,\\rho)$ with $\\psi(0) = 0$ such that $d(0,\\partial f(x)) \\geqslant 1/\\psi'(|f(x) - f(x^*)|)$ for all $x \\in B(x^*,r)$ whenever $0<|f(x)-f(x^*)|< \\rho$. After possibly reducing $r$, the inequality holds for all $x \\in B(x^*,r)$ such that $f(x) \\neq f(x^*)$. \n\nLet $\\epsilon \\in (0,r/2)$. By the definition of stability (Definition \\ref{def:discrete_lyapunov}), there exist $\\delta>0$ and $\\bar{\\alpha}>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$, the subgradient method with constant step size $\\alpha$ initialized in $B(x^*,\\delta)$ has all its iterates in $B(x^*,\\epsilon)$. Since $x^*$ is not a local minimum, we can take an initial iterate $x_0$ in $B(x^*,\\delta)$ such that $f(x_0)<f(x^*)$. Let $L>0$ denote a Lipschitz constant of $f$ on $B(x^*,r)$ and let $T := (2\\epsilon + \\delta)L\\psi'(f(x^*) - f(x_0))^2>0$.\nConsider the differential inclusion\n\\begin{equation}\n\\label{eq:ivp_x_0}\n     x'(t) \\in - \\partial f(x(t)), ~~~ \\text{for a.e.}~ \n     t \\in (0,T),~~~ x(0) = x_0.\n\\end{equation}\nSince $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d)]{clarke1990} with nonempty, compact and convex values \\cite[2.1.2 Proposition (a)]{clarke1990}. By Lemma \\ref{lemma:euler_method}, there exists $\\hat{\\alpha} \\in (0,\\bar{\\alpha}]$ such that, for all $\\alpha \\in (0,\\hat{\\alpha}]$  and for all sequence $(x_k)_{k\\in \\mathbb{N}}$ generated by the subgradient method with constant step size $\\alpha$ and initialized at $x_0$, there exists a solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp_x_0} for which $\\|\\bar{x}(t) - x(t)\\| \\leqslant \\epsilon/2$ for all $t \\in [0,T]$, where $\\bar{x}:[0,T]\\rightarrow\\mathbb{R}^n$ is the piecewise linear function defined by $\\bar{x}(t) := x_k + (t-\\alpha k)/\\alpha  (x_{k+1}-x_k)$ for all $t\\in [\\alpha k , \\min\\{\\alpha(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha\\rfloor\\}$.\n\nLet us fix some $\\alpha \\in (0,\\min\\{\\hat{\\alpha},T/10\\}]$ from now on. Consider a sequence $(x_k)_{k\\in\\mathbb{N}}$ generated by the subgradient method with constant step size $\\alpha$ and initialized at $x_0$. Consider also the linear interpolation $\\bar{x}(\\cdot)$ of those iterates up to iteration $K+1$ where $K := \\lfloor T/\\alpha \\rfloor$, as well as a solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp_x_0} such that $\\|\\bar{x}(t)-x(t)\\| \\leqslant \\epsilon/2$ for all $t\\in [0,T]$. Since $f$ is semi-algebraic, by \\cite[Lemma 5.2]{davis2020stochastic} (see also \\cite{drusvyatskiy2015curves}) it holds that\n\\begin{equation}\n\\label{eq:lyapunov}\n    f(x(t)) - f(x(0)) = - \\int_0^t d(0,\\partial f(x(\\tau)))^2 d\\tau, ~~~ \\forall t \\in [0,T].\n\\end{equation}\nAs a result, $f(x(t)) \\leqslant f(x(0)) = f(x_0)$ for all $t \\in [0,T]$. Also, $\\|x(t) - x^*\\| \\leqslant \\|x(t) - \\bar{x}(t)\\| + \\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon/2 + \\epsilon = 3\\epsilon/2 \\leqslant r$, where the inequality $\\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon$ follows from the convexity of $B(x^*,\\epsilon)$. Hence $0<f(x^*) - f(x_0) \\leqslant f(x^*) - f(x(t))$ and $d(0,\\partial f(x(t))) \\geqslant  1/\\psi'(f(x^*)-f(x(t))) \\geqslant 1/\\psi'(f(x^*)-f(x_0))$ for all $t\\in[0,T]$ by concavity of $\\psi$. Together with \\eqref{eq:lyapunov}, it follows that $f(x(K\\alpha)) - f(x_0) \\leqslant -K\\alpha M^2$ where $K := \\lfloor T/\\alpha \\rfloor$ and $M:= 1/\\psi'(f(x^*)-f(x_0))$. Recall that $L$ is a Lipschitz constant of $f$ on $B(x^*,r)$, so that we have $|f(x(K\\alpha))-f(x_0)| \\leqslant L \\|x(K\\alpha)-x_0\\|$. We thus obtain the lower bound $\\|x(K\\alpha)-x_0\\| \\geqslant K\\alpha M^2/L = \\lfloor T/\\alpha \\rfloor \\alpha M^2/L \\geqslant (T-\\alpha)M^2/L \\geqslant (T-T/10)M^2/L = 9/10( 2\\epsilon + \\delta) $ (recall that $T = (2\\epsilon + \\delta)L/M^2$). Hence $\\|x(K\\alpha)-x^*\\| \\geqslant \\|x(K\\alpha)-x_0\\| - \\|x_0-x^*\\| \\geqslant 9/10(2\\epsilon + \\delta) - \\delta = 9 \\epsilon/5 - \\delta/10 \\geqslant 9 \\epsilon/5 - \\epsilon/10 = 17\\epsilon/10$ and finally $\\|x_K-x^*\\| \\geqslant \\|x(K\\alpha)-x^*\\| -\\|x_K-x(K\\alpha)\\| \\geqslant 17\\epsilon/10 - \\epsilon/2 = 6\\epsilon/5>\\epsilon$. However, since $0<\\alpha \\leqslant \\hat{\\alpha} \\leqslant \\bar{\\alpha}$ and $x_0 \\in B(x^*,\\delta)$, by stability of $x^*$ we have $x_K \\in B(x^*,\\epsilon)$. We have reached a contradiction. \n\\end{proof}","start":14753,"end":19670}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"反设非局部极小","source_units":[{"raw_tex":"Let $x^* \\in \\mathbb{R}^n$ denote a stable point of a locally Lipschitz semi-algebraic function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. We reason by contradiction and assume that $x^*$ is not a local minimum of $f$. Given $x\\in \\mathbb{R}^n$ and $S \\subset \\mathbb{R}^n$, let $d(x,S) := \\inf \\{\\|x - y\\| : y \\in S\\}$. According to the Kurdyka-\\L{}ojasiewicz inequality \\cite[Theorem 14]{bolte2007clarke} (see also \\cite[Theorem 4.1]{attouch2010proximal}), there exist $r,\\rho>0$ and a strictly increasing concave continuous semi-algebraic function $\\psi:[0,\\rho) \\rightarrow [0,\\infty)$ that is continuously differentiable on $(0,\\rho)$ with $\\psi(0) = 0$ such that $d(0,\\partial f(x)) \\geqslant 1/\\psi'(|f(x) - f(x^*)|)$ for all $x \\in B(x^*,r)$ whenever $0<|f(x)-f(x^*)|< \\rho$. After possibly reducing $r$, the inequality holds for all $x \\in B(x^*,r)$ such that $f(x) \\neq f(x^*)$. \n","start":14767,"end":15653}]},{"id":"S027","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"点集距离","source_units":[{"raw_tex":"Let $x^* \\in \\mathbb{R}^n$ denote a stable point of a locally Lipschitz semi-algebraic function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. We reason by contradiction and assume that $x^*$ is not a local minimum of $f$. Given $x\\in \\mathbb{R}^n$ and $S \\subset \\mathbb{R}^n$, let $d(x,S) := \\inf \\{\\|x - y\\| : y \\in S\\}$. According to the Kurdyka-\\L{}ojasiewicz inequality \\cite[Theorem 14]{bolte2007clarke} (see also \\cite[Theorem 4.1]{attouch2010proximal}), there exist $r,\\rho>0$ and a strictly increasing concave continuous semi-algebraic function $\\psi:[0,\\rho) \\rightarrow [0,\\infty)$ that is continuously differentiable on $(0,\\rho)$ with $\\psi(0) = 0$ such that $d(0,\\partial f(x)) \\geqslant 1/\\psi'(|f(x) - f(x^*)|)$ for all $x \\in B(x^*,r)$ whenever $0<|f(x)-f(x^*)|< \\rho$. After possibly reducing $r$, the inequality holds for all $x \\in B(x^*,r)$ such that $f(x) \\neq f(x^*)$. \n","start":14767,"end":15653}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"局部 KL 下界","source_units":[{"raw_tex":"Let $x^* \\in \\mathbb{R}^n$ denote a stable point of a locally Lipschitz semi-algebraic function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. We reason by contradiction and assume that $x^*$ is not a local minimum of $f$. Given $x\\in \\mathbb{R}^n$ and $S \\subset \\mathbb{R}^n$, let $d(x,S) := \\inf \\{\\|x - y\\| : y \\in S\\}$. According to the Kurdyka-\\L{}ojasiewicz inequality \\cite[Theorem 14]{bolte2007clarke} (see also \\cite[Theorem 4.1]{attouch2010proximal}), there exist $r,\\rho>0$ and a strictly increasing concave continuous semi-algebraic function $\\psi:[0,\\rho) \\rightarrow [0,\\infty)$ that is continuously differentiable on $(0,\\rho)$ with $\\psi(0) = 0$ such that $d(0,\\partial f(x)) \\geqslant 1/\\psi'(|f(x) - f(x^*)|)$ for all $x \\in B(x^*,r)$ whenever $0<|f(x)-f(x^*)|< \\rho$. After possibly reducing $r$, the inequality holds for all $x \\in B(x^*,r)$ such that $f(x) \\neq f(x^*)$. \n","start":14767,"end":15653}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"稳定性控制离散轨道","source_units":[{"raw_tex":"Let $\\epsilon \\in (0,r/2)$. By the definition of stability (Definition \\ref{def:discrete_lyapunov}), there exist $\\delta>0$ and $\\bar{\\alpha}>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$, the subgradient method with constant step size $\\alpha$ initialized in $B(x^*,\\delta)$ has all its iterates in $B(x^*,\\epsilon)$. Since $x^*$ is not a local minimum, we can take an initial iterate $x_0$ in $B(x^*,\\delta)$ such that $f(x_0)<f(x^*)$. Let $L>0$ denote a Lipschitz constant of $f$ on $B(x^*,r)$ and let $T := (2\\epsilon + \\delta)L\\psi'(f(x^*) - f(x_0))^2>0$.\n","start":15654,"end":16220}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"选择低值初点","source_units":[{"raw_tex":"Let $\\epsilon \\in (0,r/2)$. By the definition of stability (Definition \\ref{def:discrete_lyapunov}), there exist $\\delta>0$ and $\\bar{\\alpha}>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$, the subgradient method with constant step size $\\alpha$ initialized in $B(x^*,\\delta)$ has all its iterates in $B(x^*,\\epsilon)$. Since $x^*$ is not a local minimum, we can take an initial iterate $x_0$ in $B(x^*,\\delta)$ such that $f(x_0)<f(x^*)$. Let $L>0$ denote a Lipschitz constant of $f$ on $B(x^*,r)$ and let $T := (2\\epsilon + \\delta)L\\psi'(f(x^*) - f(x_0))^2>0$.\n","start":15654,"end":16220}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"设定 Lipschitz 常数与时域","source_units":[{"raw_tex":"Let $\\epsilon \\in (0,r/2)$. By the definition of stability (Definition \\ref{def:discrete_lyapunov}), there exist $\\delta>0$ and $\\bar{\\alpha}>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$, the subgradient method with constant step size $\\alpha$ initialized in $B(x^*,\\delta)$ has all its iterates in $B(x^*,\\epsilon)$. Since $x^*$ is not a local minimum, we can take an initial iterate $x_0$ in $B(x^*,\\delta)$ such that $f(x_0)<f(x^*)$. Let $L>0$ denote a Lipschitz constant of $f$ on $B(x^*,r)$ and let $T := (2\\epsilon + \\delta)L\\psi'(f(x^*) - f(x_0))^2>0$.\n","start":15654,"end":16220}]},{"id":"S032","semantic_type":"definition","kind":"definition","labels":["eq:ivp_x_0"],"declared_external":false,"source_only":false,"title":"指定初值的连续动力学","source_units":[{"raw_tex":"Consider the differential inclusion\n\\begin{equation}\n\\label{eq:ivp_x_0}\n     x'(t) \\in - \\partial f(x(t)), ~~~ \\text{for a.e.}~ \n     t \\in (0,T),~~~ x(0) = x_0.\n\\end{equation}\n","start":16220,"end":16397}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"核对 Clarke 映射条件","source_units":[{"raw_tex":"Since $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d)]{clarke1990} with nonempty, compact and convex values \\cite[2.1.2 Proposition (a)]{clarke1990}. By Lemma \\ref{lemma:euler_method}, there exists $\\hat{\\alpha} \\in (0,\\bar{\\alpha}]$ such that, for all $\\alpha \\in (0,\\hat{\\alpha}]$  and for all sequence $(x_k)_{k\\in \\mathbb{N}}$ generated by the subgradient method with constant step size $\\alpha$ and initialized at $x_0$, there exists a solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp_x_0} for which $\\|\\bar{x}(t) - x(t)\\| \\leqslant \\epsilon/2$ for all $t \\in [0,T]$, where $\\bar{x}:[0,T]\\rightarrow\\mathbb{R}^n$ is the piecewise linear function defined by $\\bar{x}(t) := x_k + (t-\\alpha k)/\\alpha  (x_{k+1}-x_k)$ for all $t\\in [\\alpha k , \\min\\{\\alpha(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha\\rfloor\\}$.\n","start":16397,"end":17298}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"取得连续轨道逼近","source_units":[{"raw_tex":"Since $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d)]{clarke1990} with nonempty, compact and convex values \\cite[2.1.2 Proposition (a)]{clarke1990}. By Lemma \\ref{lemma:euler_method}, there exists $\\hat{\\alpha} \\in (0,\\bar{\\alpha}]$ such that, for all $\\alpha \\in (0,\\hat{\\alpha}]$  and for all sequence $(x_k)_{k\\in \\mathbb{N}}$ generated by the subgradient method with constant step size $\\alpha$ and initialized at $x_0$, there exists a solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp_x_0} for which $\\|\\bar{x}(t) - x(t)\\| \\leqslant \\epsilon/2$ for all $t \\in [0,T]$, where $\\bar{x}:[0,T]\\rightarrow\\mathbb{R}^n$ is the piecewise linear function defined by $\\bar{x}(t) := x_k + (t-\\alpha k)/\\alpha  (x_{k+1}-x_k)$ for all $t\\in [\\alpha k , \\min\\{\\alpha(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha\\rfloor\\}$.\n","start":16397,"end":17298}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"固定网格与逼近解","source_units":[{"raw_tex":"Let us fix some $\\alpha \\in (0,\\min\\{\\hat{\\alpha},T/10\\}]$ from now on. Consider a sequence $(x_k)_{k\\in\\mathbb{N}}$ generated by the subgradient method with constant step size $\\alpha$ and initialized at $x_0$. Consider also the linear interpolation $\\bar{x}(\\cdot)$ of those iterates up to iteration $K+1$ where $K := \\lfloor T/\\alpha \\rfloor$, as well as a solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp_x_0} such that $\\|\\bar{x}(t)-x(t)\\| \\leqslant \\epsilon/2$ for all $t\\in [0,T]$. Since $f$ is semi-algebraic, by \\cite[Lemma 5.2]{davis2020stochastic} (see also \\cite{drusvyatskiy2015curves}) it holds that\n","start":17299,"end":17928}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":["eq:lyapunov"],"declared_external":false,"source_only":false,"title":"连续轨道能量恒等式","source_units":[{"raw_tex":"Let us fix some $\\alpha \\in (0,\\min\\{\\hat{\\alpha},T/10\\}]$ from now on. Consider a sequence $(x_k)_{k\\in\\mathbb{N}}$ generated by the subgradient method with constant step size $\\alpha$ and initialized at $x_0$. Consider also the linear interpolation $\\bar{x}(\\cdot)$ of those iterates up to iteration $K+1$ where $K := \\lfloor T/\\alpha \\rfloor$, as well as a solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp_x_0} such that $\\|\\bar{x}(t)-x(t)\\| \\leqslant \\epsilon/2$ for all $t\\in [0,T]$. Since $f$ is semi-algebraic, by \\cite[Lemma 5.2]{davis2020stochastic} (see also \\cite{drusvyatskiy2015curves}) it holds that\n\\begin{equation}\n\\label{eq:lyapunov}\n    f(x(t)) - f(x(0)) = - \\int_0^t d(0,\\partial f(x(\\tau)))^2 d\\tau, ~~~ \\forall t \\in [0,T].\n\\end{equation}\n","start":17299,"end":18074}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"函数值保持低于临界值","source_units":[{"raw_tex":"As a result, $f(x(t)) \\leqslant f(x(0)) = f(x_0)$ for all $t \\in [0,T]$. Also, $\\|x(t) - x^*\\| \\leqslant \\|x(t) - \\bar{x}(t)\\| + \\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon/2 + \\epsilon = 3\\epsilon/2 \\leqslant r$, where the inequality $\\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon$ follows from the convexity of $B(x^*,\\epsilon)$. Hence $0<f(x^*) - f(x_0) \\leqslant f(x^*) - f(x(t))$ and $d(0,\\partial f(x(t))) \\geqslant  1/\\psi'(f(x^*)-f(x(t))) \\geqslant 1/\\psi'(f(x^*)-f(x_0))$ for all $t\\in[0,T]$ by concavity of $\\psi$. Together with \\eqref{eq:lyapunov}, it follows that $f(x(K\\alpha)) - f(x_0) \\leqslant -K\\alpha M^2$ where $K := \\lfloor T/\\alpha \\rfloor$ and $M:= 1/\\psi'(f(x^*)-f(x_0))$. Recall that $L$ is a Lipschitz constant of $f$ on $B(x^*,r)$, so that we have $|f(x(K\\alpha))-f(x_0)| \\leqslant L \\|x(K\\alpha)-x_0\\|$. We thus obtain the lower bound $\\|x(K\\alpha)-x_0\\| \\geqslant K\\alpha M^2/L = \\lfloor T/\\alpha \\rfloor \\alpha M^2/L \\geqslant (T-\\alpha)M^2/L \\geqslant (T-T/10)M^2/L = 9/10( 2\\epsilon + \\delta) $ (recall that $T = (2\\epsilon + \\delta)L/M^2$). Hence $\\|x(K\\alpha)-x^*\\| \\geqslant \\|x(K\\alpha)-x_0\\| - \\|x_0-x^*\\| \\geqslant 9/10(2\\epsilon + \\delta) - \\delta = 9 \\epsilon/5 - \\delta/10 \\geqslant 9 \\epsilon/5 - \\epsilon/10 = 17\\epsilon/10$ and finally $\\|x_K-x^*\\| \\geqslant \\|x(K\\alpha)-x^*\\| -\\|x_K-x(K\\alpha)\\| \\geqslant 17\\epsilon/10 - \\epsilon/2 = 6\\epsilon/5>\\epsilon$. However, since $0<\\alpha \\leqslant \\hat{\\alpha} \\leqslant \\bar{\\alpha}$ and $x_0 \\in B(x^*,\\delta)$, by stability of $x^*$ we have $x_K \\in B(x^*,\\epsilon)$. We have reached a contradiction. \n","start":18074,"end":19659}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"连续轨道留在 KL 球","source_units":[{"raw_tex":"As a result, $f(x(t)) \\leqslant f(x(0)) = f(x_0)$ for all $t \\in [0,T]$. Also, $\\|x(t) - x^*\\| \\leqslant \\|x(t) - \\bar{x}(t)\\| + \\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon/2 + \\epsilon = 3\\epsilon/2 \\leqslant r$, where the inequality $\\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon$ follows from the convexity of $B(x^*,\\epsilon)$. Hence $0<f(x^*) - f(x_0) \\leqslant f(x^*) - f(x(t))$ and $d(0,\\partial f(x(t))) \\geqslant  1/\\psi'(f(x^*)-f(x(t))) \\geqslant 1/\\psi'(f(x^*)-f(x_0))$ for all $t\\in[0,T]$ by concavity of $\\psi$. Together with \\eqref{eq:lyapunov}, it follows that $f(x(K\\alpha)) - f(x_0) \\leqslant -K\\alpha M^2$ where $K := \\lfloor T/\\alpha \\rfloor$ and $M:= 1/\\psi'(f(x^*)-f(x_0))$. Recall that $L$ is a Lipschitz constant of $f$ on $B(x^*,r)$, so that we have $|f(x(K\\alpha))-f(x_0)| \\leqslant L \\|x(K\\alpha)-x_0\\|$. We thus obtain the lower bound $\\|x(K\\alpha)-x_0\\| \\geqslant K\\alpha M^2/L = \\lfloor T/\\alpha \\rfloor \\alpha M^2/L \\geqslant (T-\\alpha)M^2/L \\geqslant (T-T/10)M^2/L = 9/10( 2\\epsilon + \\delta) $ (recall that $T = (2\\epsilon + \\delta)L/M^2$). Hence $\\|x(K\\alpha)-x^*\\| \\geqslant \\|x(K\\alpha)-x_0\\| - \\|x_0-x^*\\| \\geqslant 9/10(2\\epsilon + \\delta) - \\delta = 9 \\epsilon/5 - \\delta/10 \\geqslant 9 \\epsilon/5 - \\epsilon/10 = 17\\epsilon/10$ and finally $\\|x_K-x^*\\| \\geqslant \\|x(K\\alpha)-x^*\\| -\\|x_K-x(K\\alpha)\\| \\geqslant 17\\epsilon/10 - \\epsilon/2 = 6\\epsilon/5>\\epsilon$. However, since $0<\\alpha \\leqslant \\hat{\\alpha} \\leqslant \\bar{\\alpha}$ and $x_0 \\in B(x^*,\\delta)$, by stability of $x^*$ we have $x_K \\in B(x^*,\\epsilon)$. We have reached a contradiction. \n","start":18074,"end":19659}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"获得统一正斜率","source_units":[{"raw_tex":"As a result, $f(x(t)) \\leqslant f(x(0)) = f(x_0)$ for all $t \\in [0,T]$. Also, $\\|x(t) - x^*\\| \\leqslant \\|x(t) - \\bar{x}(t)\\| + \\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon/2 + \\epsilon = 3\\epsilon/2 \\leqslant r$, where the inequality $\\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon$ follows from the convexity of $B(x^*,\\epsilon)$. Hence $0<f(x^*) - f(x_0) \\leqslant f(x^*) - f(x(t))$ and $d(0,\\partial f(x(t))) \\geqslant  1/\\psi'(f(x^*)-f(x(t))) \\geqslant 1/\\psi'(f(x^*)-f(x_0))$ for all $t\\in[0,T]$ by concavity of $\\psi$. Together with \\eqref{eq:lyapunov}, it follows that $f(x(K\\alpha)) - f(x_0) \\leqslant -K\\alpha M^2$ where $K := \\lfloor T/\\alpha \\rfloor$ and $M:= 1/\\psi'(f(x^*)-f(x_0))$. Recall that $L$ is a Lipschitz constant of $f$ on $B(x^*,r)$, so that we have $|f(x(K\\alpha))-f(x_0)| \\leqslant L \\|x(K\\alpha)-x_0\\|$. We thus obtain the lower bound $\\|x(K\\alpha)-x_0\\| \\geqslant K\\alpha M^2/L = \\lfloor T/\\alpha \\rfloor \\alpha M^2/L \\geqslant (T-\\alpha)M^2/L \\geqslant (T-T/10)M^2/L = 9/10( 2\\epsilon + \\delta) $ (recall that $T = (2\\epsilon + \\delta)L/M^2$). Hence $\\|x(K\\alpha)-x^*\\| \\geqslant \\|x(K\\alpha)-x_0\\| - \\|x_0-x^*\\| \\geqslant 9/10(2\\epsilon + \\delta) - \\delta = 9 \\epsilon/5 - \\delta/10 \\geqslant 9 \\epsilon/5 - \\epsilon/10 = 17\\epsilon/10$ and finally $\\|x_K-x^*\\| \\geqslant \\|x(K\\alpha)-x^*\\| -\\|x_K-x(K\\alpha)\\| \\geqslant 17\\epsilon/10 - \\epsilon/2 = 6\\epsilon/5>\\epsilon$. However, since $0<\\alpha \\leqslant \\hat{\\alpha} \\leqslant \\bar{\\alpha}$ and $x_0 \\in B(x^*,\\delta)$, by stability of $x^*$ we have $x_K \\in B(x^*,\\epsilon)$. We have reached a contradiction. \n","start":18074,"end":19659}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"累计定量能量下降","source_units":[{"raw_tex":"As a result, $f(x(t)) \\leqslant f(x(0)) = f(x_0)$ for all $t \\in [0,T]$. Also, $\\|x(t) - x^*\\| \\leqslant \\|x(t) - \\bar{x}(t)\\| + \\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon/2 + \\epsilon = 3\\epsilon/2 \\leqslant r$, where the inequality $\\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon$ follows from the convexity of $B(x^*,\\epsilon)$. Hence $0<f(x^*) - f(x_0) \\leqslant f(x^*) - f(x(t))$ and $d(0,\\partial f(x(t))) \\geqslant  1/\\psi'(f(x^*)-f(x(t))) \\geqslant 1/\\psi'(f(x^*)-f(x_0))$ for all $t\\in[0,T]$ by concavity of $\\psi$. Together with \\eqref{eq:lyapunov}, it follows that $f(x(K\\alpha)) - f(x_0) \\leqslant -K\\alpha M^2$ where $K := \\lfloor T/\\alpha \\rfloor$ and $M:= 1/\\psi'(f(x^*)-f(x_0))$. Recall that $L$ is a Lipschitz constant of $f$ on $B(x^*,r)$, so that we have $|f(x(K\\alpha))-f(x_0)| \\leqslant L \\|x(K\\alpha)-x_0\\|$. We thus obtain the lower bound $\\|x(K\\alpha)-x_0\\| \\geqslant K\\alpha M^2/L = \\lfloor T/\\alpha \\rfloor \\alpha M^2/L \\geqslant (T-\\alpha)M^2/L \\geqslant (T-T/10)M^2/L = 9/10( 2\\epsilon + \\delta) $ (recall that $T = (2\\epsilon + \\delta)L/M^2$). Hence $\\|x(K\\alpha)-x^*\\| \\geqslant \\|x(K\\alpha)-x_0\\| - \\|x_0-x^*\\| \\geqslant 9/10(2\\epsilon + \\delta) - \\delta = 9 \\epsilon/5 - \\delta/10 \\geqslant 9 \\epsilon/5 - \\epsilon/10 = 17\\epsilon/10$ and finally $\\|x_K-x^*\\| \\geqslant \\|x(K\\alpha)-x^*\\| -\\|x_K-x(K\\alpha)\\| \\geqslant 17\\epsilon/10 - \\epsilon/2 = 6\\epsilon/5>\\epsilon$. However, since $0<\\alpha \\leqslant \\hat{\\alpha} \\leqslant \\bar{\\alpha}$ and $x_0 \\in B(x^*,\\delta)$, by stability of $x^*$ we have $x_K \\in B(x^*,\\epsilon)$. We have reached a contradiction. \n","start":18074,"end":19659}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"能量下降迫使位移","source_units":[{"raw_tex":"As a result, $f(x(t)) \\leqslant f(x(0)) = f(x_0)$ for all $t \\in [0,T]$. Also, $\\|x(t) - x^*\\| \\leqslant \\|x(t) - \\bar{x}(t)\\| + \\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon/2 + \\epsilon = 3\\epsilon/2 \\leqslant r$, where the inequality $\\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon$ follows from the convexity of $B(x^*,\\epsilon)$. Hence $0<f(x^*) - f(x_0) \\leqslant f(x^*) - f(x(t))$ and $d(0,\\partial f(x(t))) \\geqslant  1/\\psi'(f(x^*)-f(x(t))) \\geqslant 1/\\psi'(f(x^*)-f(x_0))$ for all $t\\in[0,T]$ by concavity of $\\psi$. Together with \\eqref{eq:lyapunov}, it follows that $f(x(K\\alpha)) - f(x_0) \\leqslant -K\\alpha M^2$ where $K := \\lfloor T/\\alpha \\rfloor$ and $M:= 1/\\psi'(f(x^*)-f(x_0))$. Recall that $L$ is a Lipschitz constant of $f$ on $B(x^*,r)$, so that we have $|f(x(K\\alpha))-f(x_0)| \\leqslant L \\|x(K\\alpha)-x_0\\|$. We thus obtain the lower bound $\\|x(K\\alpha)-x_0\\| \\geqslant K\\alpha M^2/L = \\lfloor T/\\alpha \\rfloor \\alpha M^2/L \\geqslant (T-\\alpha)M^2/L \\geqslant (T-T/10)M^2/L = 9/10( 2\\epsilon + \\delta) $ (recall that $T = (2\\epsilon + \\delta)L/M^2$). Hence $\\|x(K\\alpha)-x^*\\| \\geqslant \\|x(K\\alpha)-x_0\\| - \\|x_0-x^*\\| \\geqslant 9/10(2\\epsilon + \\delta) - \\delta = 9 \\epsilon/5 - \\delta/10 \\geqslant 9 \\epsilon/5 - \\epsilon/10 = 17\\epsilon/10$ and finally $\\|x_K-x^*\\| \\geqslant \\|x(K\\alpha)-x^*\\| -\\|x_K-x(K\\alpha)\\| \\geqslant 17\\epsilon/10 - \\epsilon/2 = 6\\epsilon/5>\\epsilon$. However, since $0<\\alpha \\leqslant \\hat{\\alpha} \\leqslant \\bar{\\alpha}$ and $x_0 \\in B(x^*,\\delta)$, by stability of $x^*$ we have $x_K \\in B(x^*,\\epsilon)$. We have reached a contradiction. \n","start":18074,"end":19659}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"连续终点远离稳定点","source_units":[{"raw_tex":"As a result, $f(x(t)) \\leqslant f(x(0)) = f(x_0)$ for all $t \\in [0,T]$. Also, $\\|x(t) - x^*\\| \\leqslant \\|x(t) - \\bar{x}(t)\\| + \\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon/2 + \\epsilon = 3\\epsilon/2 \\leqslant r$, where the inequality $\\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon$ follows from the convexity of $B(x^*,\\epsilon)$. Hence $0<f(x^*) - f(x_0) \\leqslant f(x^*) - f(x(t))$ and $d(0,\\partial f(x(t))) \\geqslant  1/\\psi'(f(x^*)-f(x(t))) \\geqslant 1/\\psi'(f(x^*)-f(x_0))$ for all $t\\in[0,T]$ by concavity of $\\psi$. Together with \\eqref{eq:lyapunov}, it follows that $f(x(K\\alpha)) - f(x_0) \\leqslant -K\\alpha M^2$ where $K := \\lfloor T/\\alpha \\rfloor$ and $M:= 1/\\psi'(f(x^*)-f(x_0))$. Recall that $L$ is a Lipschitz constant of $f$ on $B(x^*,r)$, so that we have $|f(x(K\\alpha))-f(x_0)| \\leqslant L \\|x(K\\alpha)-x_0\\|$. We thus obtain the lower bound $\\|x(K\\alpha)-x_0\\| \\geqslant K\\alpha M^2/L = \\lfloor T/\\alpha \\rfloor \\alpha M^2/L \\geqslant (T-\\alpha)M^2/L \\geqslant (T-T/10)M^2/L = 9/10( 2\\epsilon + \\delta) $ (recall that $T = (2\\epsilon + \\delta)L/M^2$). Hence $\\|x(K\\alpha)-x^*\\| \\geqslant \\|x(K\\alpha)-x_0\\| - \\|x_0-x^*\\| \\geqslant 9/10(2\\epsilon + \\delta) - \\delta = 9 \\epsilon/5 - \\delta/10 \\geqslant 9 \\epsilon/5 - \\epsilon/10 = 17\\epsilon/10$ and finally $\\|x_K-x^*\\| \\geqslant \\|x(K\\alpha)-x^*\\| -\\|x_K-x(K\\alpha)\\| \\geqslant 17\\epsilon/10 - \\epsilon/2 = 6\\epsilon/5>\\epsilon$. However, since $0<\\alpha \\leqslant \\hat{\\alpha} \\leqslant \\bar{\\alpha}$ and $x_0 \\in B(x^*,\\delta)$, by stability of $x^*$ we have $x_K \\in B(x^*,\\epsilon)$. We have reached a contradiction. \n","start":18074,"end":19659}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"离散节点越出稳定球","source_units":[{"raw_tex":"As a result, $f(x(t)) \\leqslant f(x(0)) = f(x_0)$ for all $t \\in [0,T]$. Also, $\\|x(t) - x^*\\| \\leqslant \\|x(t) - \\bar{x}(t)\\| + \\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon/2 + \\epsilon = 3\\epsilon/2 \\leqslant r$, where the inequality $\\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon$ follows from the convexity of $B(x^*,\\epsilon)$. Hence $0<f(x^*) - f(x_0) \\leqslant f(x^*) - f(x(t))$ and $d(0,\\partial f(x(t))) \\geqslant  1/\\psi'(f(x^*)-f(x(t))) \\geqslant 1/\\psi'(f(x^*)-f(x_0))$ for all $t\\in[0,T]$ by concavity of $\\psi$. Together with \\eqref{eq:lyapunov}, it follows that $f(x(K\\alpha)) - f(x_0) \\leqslant -K\\alpha M^2$ where $K := \\lfloor T/\\alpha \\rfloor$ and $M:= 1/\\psi'(f(x^*)-f(x_0))$. Recall that $L$ is a Lipschitz constant of $f$ on $B(x^*,r)$, so that we have $|f(x(K\\alpha))-f(x_0)| \\leqslant L \\|x(K\\alpha)-x_0\\|$. We thus obtain the lower bound $\\|x(K\\alpha)-x_0\\| \\geqslant K\\alpha M^2/L = \\lfloor T/\\alpha \\rfloor \\alpha M^2/L \\geqslant (T-\\alpha)M^2/L \\geqslant (T-T/10)M^2/L = 9/10( 2\\epsilon + \\delta) $ (recall that $T = (2\\epsilon + \\delta)L/M^2$). Hence $\\|x(K\\alpha)-x^*\\| \\geqslant \\|x(K\\alpha)-x_0\\| - \\|x_0-x^*\\| \\geqslant 9/10(2\\epsilon + \\delta) - \\delta = 9 \\epsilon/5 - \\delta/10 \\geqslant 9 \\epsilon/5 - \\epsilon/10 = 17\\epsilon/10$ and finally $\\|x_K-x^*\\| \\geqslant \\|x(K\\alpha)-x^*\\| -\\|x_K-x(K\\alpha)\\| \\geqslant 17\\epsilon/10 - \\epsilon/2 = 6\\epsilon/5>\\epsilon$. However, since $0<\\alpha \\leqslant \\hat{\\alpha} \\leqslant \\bar{\\alpha}$ and $x_0 \\in B(x^*,\\delta)$, by stability of $x^*$ we have $x_K \\in B(x^*,\\epsilon)$. We have reached a contradiction. \n","start":18074,"end":19659}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"与稳定性矛盾","source_units":[{"raw_tex":"As a result, $f(x(t)) \\leqslant f(x(0)) = f(x_0)$ for all $t \\in [0,T]$. Also, $\\|x(t) - x^*\\| \\leqslant \\|x(t) - \\bar{x}(t)\\| + \\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon/2 + \\epsilon = 3\\epsilon/2 \\leqslant r$, where the inequality $\\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon$ follows from the convexity of $B(x^*,\\epsilon)$. Hence $0<f(x^*) - f(x_0) \\leqslant f(x^*) - f(x(t))$ and $d(0,\\partial f(x(t))) \\geqslant  1/\\psi'(f(x^*)-f(x(t))) \\geqslant 1/\\psi'(f(x^*)-f(x_0))$ for all $t\\in[0,T]$ by concavity of $\\psi$. Together with \\eqref{eq:lyapunov}, it follows that $f(x(K\\alpha)) - f(x_0) \\leqslant -K\\alpha M^2$ where $K := \\lfloor T/\\alpha \\rfloor$ and $M:= 1/\\psi'(f(x^*)-f(x_0))$. Recall that $L$ is a Lipschitz constant of $f$ on $B(x^*,r)$, so that we have $|f(x(K\\alpha))-f(x_0)| \\leqslant L \\|x(K\\alpha)-x_0\\|$. We thus obtain the lower bound $\\|x(K\\alpha)-x_0\\| \\geqslant K\\alpha M^2/L = \\lfloor T/\\alpha \\rfloor \\alpha M^2/L \\geqslant (T-\\alpha)M^2/L \\geqslant (T-T/10)M^2/L = 9/10( 2\\epsilon + \\delta) $ (recall that $T = (2\\epsilon + \\delta)L/M^2$). Hence $\\|x(K\\alpha)-x^*\\| \\geqslant \\|x(K\\alpha)-x_0\\| - \\|x_0-x^*\\| \\geqslant 9/10(2\\epsilon + \\delta) - \\delta = 9 \\epsilon/5 - \\delta/10 \\geqslant 9 \\epsilon/5 - \\epsilon/10 = 17\\epsilon/10$ and finally $\\|x_K-x^*\\| \\geqslant \\|x(K\\alpha)-x^*\\| -\\|x_K-x(K\\alpha)\\| \\geqslant 17\\epsilon/10 - \\epsilon/2 = 6\\epsilon/5>\\epsilon$. However, since $0<\\alpha \\leqslant \\hat{\\alpha} \\leqslant \\bar{\\alpha}$ and $x_0 \\in B(x^*,\\delta)$, by stability of $x^*$ we have $x_K \\in B(x^*,\\epsilon)$. We have reached a contradiction. \n","start":18074,"end":19659}]},{"id":"S045","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单：bolte2007clarke, Theorem 14；attouch2010proximal, Theorem 4.1。原文 L131 明确引用。"],"source_only":false,"title":"半代数 KL 不等式","source_units":[]},{"id":"S046","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单：clarke1990, Proposition 2.1.5(d)。原文 L140 明确引用。"],"source_only":false,"title":"Clarke 映射上半连续","source_units":[]},{"id":"S047","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单：clarke1990, Proposition 2.1.2(a)。原文 L140 明确引用。"],"source_only":false,"title":"Clarke 次微分值性质","source_units":[]},{"id":"S048","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单：aubin1984differential, Proposition 3, p.42。原文 L108 明确引用。"],"source_only":false,"title":"紧域上的映射有界性","source_units":[]},{"id":"S049","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单：Arzelà–Ascoli 与 Banach–Alaoglu，如 aubin1984differential, Theorem 4, p.13 中所用。原文 L108 明确引用。"],"source_only":false,"title":"函数与导数紧性","source_units":[]},{"id":"S050","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单：aubin1984differential, Convergence Theorem, p.60。原文 L118 明确引用。"],"source_only":false,"title":"微分包含收敛定理","source_units":[]},{"id":"S051","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单：davis2020stochastic, Lemma 5.2；drusvyatskiy2015curves。原文 L142 明确引用。"],"source_only":false,"title":"半代数轨道能量公式","source_units":[]},{"id":"S052","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单中的标准有限维紧性与凸性事实；由标注者显式列出，原文未逐项命名引用。"],"source_only":false,"title":"有限维球与闭集事实","source_units":[]},{"id":"S053","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；属于原始前提清单的标准紧性背景，作者在 L120 自行展开其反证应用。"],"source_only":false,"title":"序列化统一逼近","source_units":[]},{"id":"S054","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；是原始前提清单中 KL 脱奇异函数条件的标准一元凹性背景。"],"source_only":false,"title":"凹函数导数比较","source_units":[]},{"id":"S055","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；属于原始前提清单的标准有限维及绝对连续性背景。"],"source_only":false,"title":"范数与取整估计","source_units":[]},{"id":"S056","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；属于原始前提清单的标准有限维紧性背景，原文 L131、L133 使用而未单独引用。"],"source_only":false,"title":"紧球上的 Lipschitz 常数","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S024","edges":[{"source_id":"S001","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S009","target_id":"S008","kind":"supports"},{"source_id":"S010","target_id":"S008","kind":"supports"},{"source_id":"S011","target_id":"S008","kind":"supports"},{"source_id":"S012","target_id":"S008","kind":"supports"},{"source_id":"S013","target_id":"S008","kind":"supports"},{"source_id":"S014","target_id":"S008","kind":"supports"},{"source_id":"S010","target_id":"S016","kind":"supports"},{"source_id":"S011","target_id":"S016","kind":"supports"},{"source_id":"S012","target_id":"S016","kind":"supports"},{"source_id":"S014","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S052","target_id":"S017","kind":"supports"},{"source_id":"S009","target_id":"S018","kind":"supports"},{"source_id":"S011","target_id":"S018","kind":"supports"},{"source_id":"S016","target_id":"S018","kind":"supports"},{"source_id":"S048","target_id":"S018","kind":"supports"},{"source_id":"S052","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S049","target_id":"S019","kind":"supports"},{"source_id":"S016","target_id":"S020","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S011","target_id":"S020","kind":"supports"},{"source_id":"S009","target_id":"S021","kind":"supports"},{"source_id":"S016","target_id":"S021","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S050","target_id":"S021","kind":"supports"},{"source_id":"S010","target_id":"S022","kind":"supports"},{"source_id":"S016","target_id":"S022","kind":"supports"},{"source_id":"S019","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S052","target_id":"S022","kind":"supports"},{"source_id":"S013","target_id":"S023","kind":"supports"},{"source_id":"S016","target_id":"S023","kind":"supports"},{"source_id":"S019","target_id":"S023","kind":"supports"},{"source_id":"S021","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S053","target_id":"S023","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S015","kind":"supports"},{"source_id":"S018","target_id":"S015","kind":"supports"},{"source_id":"S019","target_id":"S015","kind":"supports"},{"source_id":"S020","target_id":"S015","kind":"supports"},{"source_id":"S021","target_id":"S015","kind":"supports"},{"source_id":"S022","target_id":"S015","kind":"supports"},{"source_id":"S023","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S008","kind":"supports"},{"source_id":"S004","target_id":"S026","kind":"supports"},{"source_id":"S006","target_id":"S026","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S026","target_id":"S028","kind":"supports"},{"source_id":"S045","target_id":"S028","kind":"supports"},{"source_id":"S056","target_id":"S028","kind":"supports"},{"source_id":"S026","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S004","target_id":"S029","kind":"supports"},{"source_id":"S026","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S006","target_id":"S030","kind":"supports"},{"source_id":"S026","target_id":"S031","kind":"supports"},{"source_id":"S028","target_id":"S031","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S054","target_id":"S031","kind":"supports"},{"source_id":"S056","target_id":"S031","kind":"supports"},{"source_id":"S005","target_id":"S032","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S026","target_id":"S033","kind":"supports"},{"source_id":"S046","target_id":"S033","kind":"supports"},{"source_id":"S047","target_id":"S033","kind":"supports"},{"source_id":"S001","target_id":"S034","kind":"supports"},{"source_id":"S008","target_id":"S034","kind":"supports"},{"source_id":"S029","target_id":"S034","kind":"supports"},{"source_id":"S030","target_id":"S034","kind":"supports"},{"source_id":"S031","target_id":"S034","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S052","target_id":"S034","kind":"supports"},{"source_id":"S055","target_id":"S034","kind":"supports"},{"source_id":"S014","target_id":"S034","kind":"supports"},{"source_id":"S031","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S026","target_id":"S036","kind":"supports"},{"source_id":"S032","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S051","target_id":"S036","kind":"supports"},{"source_id":"S030","target_id":"S037","kind":"supports"},{"source_id":"S032","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S055","target_id":"S037","kind":"supports"},{"source_id":"S029","target_id":"S038","kind":"supports"},{"source_id":"S034","target_id":"S038","kind":"supports"},{"source_id":"S035","target_id":"S038","kind":"supports"},{"source_id":"S052","target_id":"S038","kind":"supports"},{"source_id":"S055","target_id":"S038","kind":"supports"},{"source_id":"S028","target_id":"S039","kind":"supports"},{"source_id":"S030","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S054","target_id":"S039","kind":"supports"},{"source_id":"S035","target_id":"S040","kind":"supports"},{"source_id":"S036","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S055","target_id":"S040","kind":"supports"},{"source_id":"S030","target_id":"S041","kind":"supports"},{"source_id":"S031","target_id":"S041","kind":"supports"},{"source_id":"S035","target_id":"S041","kind":"supports"},{"source_id":"S038","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S055","target_id":"S041","kind":"supports"},{"source_id":"S030","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S055","target_id":"S042","kind":"supports"},{"source_id":"S034","target_id":"S043","kind":"supports"},{"source_id":"S035","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S014","target_id":"S043","kind":"supports"},{"source_id":"S055","target_id":"S043","kind":"supports"},{"source_id":"S029","target_id":"S044","kind":"supports"},{"source_id":"S030","target_id":"S044","kind":"supports"},{"source_id":"S034","target_id":"S044","kind":"supports"},{"source_id":"S035","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S028","target_id":"S025","kind":"supports"},{"source_id":"S029","target_id":"S025","kind":"supports"},{"source_id":"S030","target_id":"S025","kind":"supports"},{"source_id":"S031","target_id":"S025","kind":"supports"},{"source_id":"S032","target_id":"S025","kind":"supports"},{"source_id":"S033","target_id":"S025","kind":"supports"},{"source_id":"S034","target_id":"S025","kind":"supports"},{"source_id":"S035","target_id":"S025","kind":"supports"},{"source_id":"S036","target_id":"S025","kind":"supports"},{"source_id":"S037","target_id":"S025","kind":"supports"},{"source_id":"S038","target_id":"S025","kind":"supports"},{"source_id":"S039","target_id":"S025","kind":"supports"},{"source_id":"S040","target_id":"S025","kind":"supports"},{"source_id":"S041","target_id":"S025","kind":"supports"},{"source_id":"S042","target_id":"S025","kind":"supports"},{"source_id":"S043","target_id":"S025","kind":"supports"},{"source_id":"S044","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S024","edges":[{"source_id":"S001","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S009","target_id":"S008","kind":"supports"},{"source_id":"S010","target_id":"S008","kind":"supports"},{"source_id":"S011","target_id":"S008","kind":"supports"},{"source_id":"S012","target_id":"S008","kind":"supports"},{"source_id":"S013","target_id":"S008","kind":"supports"},{"source_id":"S014","target_id":"S008","kind":"supports"},{"source_id":"S052","target_id":"S008","kind":"supports"},{"source_id":"S048","target_id":"S008","kind":"supports"},{"source_id":"S049","target_id":"S008","kind":"supports"},{"source_id":"S050","target_id":"S008","kind":"supports"},{"source_id":"S053","target_id":"S008","kind":"supports"},{"source_id":"S004","target_id":"S026","kind":"supports"},{"source_id":"S006","target_id":"S026","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S026","target_id":"S028","kind":"supports"},{"source_id":"S045","target_id":"S028","kind":"supports"},{"source_id":"S056","target_id":"S028","kind":"supports"},{"source_id":"S026","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S004","target_id":"S029","kind":"supports"},{"source_id":"S026","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S006","target_id":"S030","kind":"supports"},{"source_id":"S026","target_id":"S031","kind":"supports"},{"source_id":"S028","target_id":"S031","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S054","target_id":"S031","kind":"supports"},{"source_id":"S056","target_id":"S031","kind":"supports"},{"source_id":"S005","target_id":"S032","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S026","target_id":"S033","kind":"supports"},{"source_id":"S046","target_id":"S033","kind":"supports"},{"source_id":"S047","target_id":"S033","kind":"supports"},{"source_id":"S001","target_id":"S034","kind":"supports"},{"source_id":"S008","target_id":"S034","kind":"supports"},{"source_id":"S029","target_id":"S034","kind":"supports"},{"source_id":"S030","target_id":"S034","kind":"supports"},{"source_id":"S031","target_id":"S034","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S052","target_id":"S034","kind":"supports"},{"source_id":"S055","target_id":"S034","kind":"supports"},{"source_id":"S014","target_id":"S034","kind":"supports"},{"source_id":"S031","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S026","target_id":"S036","kind":"supports"},{"source_id":"S032","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S051","target_id":"S036","kind":"supports"},{"source_id":"S030","target_id":"S037","kind":"supports"},{"source_id":"S032","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S055","target_id":"S037","kind":"supports"},{"source_id":"S029","target_id":"S038","kind":"supports"},{"source_id":"S034","target_id":"S038","kind":"supports"},{"source_id":"S035","target_id":"S038","kind":"supports"},{"source_id":"S052","target_id":"S038","kind":"supports"},{"source_id":"S055","target_id":"S038","kind":"supports"},{"source_id":"S028","target_id":"S039","kind":"supports"},{"source_id":"S030","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S054","target_id":"S039","kind":"supports"},{"source_id":"S035","target_id":"S040","kind":"supports"},{"source_id":"S036","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S055","target_id":"S040","kind":"supports"},{"source_id":"S030","target_id":"S041","kind":"supports"},{"source_id":"S031","target_id":"S041","kind":"supports"},{"source_id":"S035","target_id":"S041","kind":"supports"},{"source_id":"S038","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S055","target_id":"S041","kind":"supports"},{"source_id":"S030","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S055","target_id":"S042","kind":"supports"},{"source_id":"S034","target_id":"S043","kind":"supports"},{"source_id":"S035","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S014","target_id":"S043","kind":"supports"},{"source_id":"S055","target_id":"S043","kind":"supports"},{"source_id":"S029","target_id":"S044","kind":"supports"},{"source_id":"S030","target_id":"S044","kind":"supports"},{"source_id":"S034","target_id":"S044","kind":"supports"},{"source_id":"S035","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S028","target_id":"S025","kind":"supports"},{"source_id":"S029","target_id":"S025","kind":"supports"},{"source_id":"S030","target_id":"S025","kind":"supports"},{"source_id":"S031","target_id":"S025","kind":"supports"},{"source_id":"S032","target_id":"S025","kind":"supports"},{"source_id":"S033","target_id":"S025","kind":"supports"},{"source_id":"S034","target_id":"S025","kind":"supports"},{"source_id":"S035","target_id":"S025","kind":"supports"},{"source_id":"S036","target_id":"S025","kind":"supports"},{"source_id":"S037","target_id":"S025","kind":"supports"},{"source_id":"S038","target_id":"S025","kind":"supports"},{"source_id":"S039","target_id":"S025","kind":"supports"},{"source_id":"S040","target_id":"S025","kind":"supports"},{"source_id":"S041","target_id":"S025","kind":"supports"},{"source_id":"S042","target_id":"S025","kind":"supports"},{"source_id":"S043","target_id":"S025","kind":"supports"},{"source_id":"S044","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"algorithm","kind":"algorithm","labels":[],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"常步长次梯度法","source_units":[{"raw_tex":"The subgradient method with constant step size for minimizing a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ consists in choosing an initial point $x_0 \\in \\mathbb{R}^n$ and generating a sequence of iterates according to the update rule $x_{k+1} \\in x_k - \\alpha \\partial f(x_k), ~ \\forall k \\in \\mathbb{N} := \\{0,1,2,\\hdots\\}$, where $\\alpha>0$ is the step size and $\\partial f$ is the Clarke subdifferential \\cite[Chapter 2]{clarke1990}. While this method is often used in practice to solve nonconvex and nonsmooth problems, there is little theoretical understanding of the behavior of its iterates. To the best of our knowledge, the only known results are in the convex setting, as we next describe.\n","start":2451,"end":3174}]},{"id":"S002","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"非凸分析背景","source_units":[{"raw_tex":"The subgradient method with constant step size for minimizing a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ consists in choosing an initial point $x_0 \\in \\mathbb{R}^n$ and generating a sequence of iterates according to the update rule $x_{k+1} \\in x_k - \\alpha \\partial f(x_k), ~ \\forall k \\in \\mathbb{N} := \\{0,1,2,\\hdots\\}$, where $\\alpha>0$ is the step size and $\\partial f$ is the Clarke subdifferential \\cite[Chapter 2]{clarke1990}. While this method is often used in practice to solve nonconvex and nonsmooth problems, there is little theoretical understanding of the behavior of its iterates. To the best of our knowledge, the only known results are in the convex setting, as we next describe.\n","start":2451,"end":3174}]},{"id":"S003","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"内积范数与球","source_units":[{"raw_tex":"Let $\\|\\cdot\\|$ be the induced norm of an inner product $\\langle \\cdot, \\cdot\\rangle$ on $\\mathbb{R}^n$. Let $B(a,r)$ and $\\mathring{B}(a,r)$ respectively denote the closed ball and the open ball of center $a\\in \\mathbb{R}^n$ and radius $r>0$. We next define the notion of discrete Lyupanov stability.\n\n","start":7298,"end":7601}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["def:discrete_lyapunov"],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"离散稳定性","source_units":[{"raw_tex":"\\begin{definition}\n\\label{def:discrete_lyapunov}\n\\normalfont\nWe say that $x^*\\in \\mathbb{R}^n$ is a stable point of a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if for all $\\epsilon>0$, there exist $\\delta>0$ and $\\bar{\\alpha}>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$, the subgradient method with constant step size $\\alpha$ initialized in $B(x^*,\\delta)$ has all its iterates in $B(x^*,\\epsilon)$.\n\\end{definition}\n","start":7601,"end":8049}]},{"id":"S005","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"负次微分特化","source_units":[{"raw_tex":"We will apply Lemma \\ref{lemma:euler_method} to the case where the set-valued mapping $F$ is the opposite of the Clarke subdifferential of a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. Recall that a point $x^* \\in \\mathbb{R}^n$ is a local minimum (respectively strict local minimum) of a function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if there exists a positive constant $\\epsilon$ such that $f(x^*) \\leqslant f(x)$ for all $x \\in B(x^*,\\epsilon)\\setminus \\{x^*\\}$ (respectively $f(x^*) < f(x)$). Using Lemma \\ref{lemma:euler_method}, we obtain the following necessary condition for stability.\n","start":14005,"end":14621}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"局部极小定义","source_units":[{"raw_tex":"We will apply Lemma \\ref{lemma:euler_method} to the case where the set-valued mapping $F$ is the opposite of the Clarke subdifferential of a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. Recall that a point $x^* \\in \\mathbb{R}^n$ is a local minimum (respectively strict local minimum) of a function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if there exists a positive constant $\\epsilon$ such that $f(x^*) \\leqslant f(x)$ for all $x \\in B(x^*,\\epsilon)\\setminus \\{x^*\\}$ (respectively $f(x^*) < f(x)$). Using Lemma \\ref{lemma:euler_method}, we obtain the following necessary condition for stability.\n","start":14005,"end":14621}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"严格极小定义","source_units":[{"raw_tex":"We will apply Lemma \\ref{lemma:euler_method} to the case where the set-valued mapping $F$ is the opposite of the Clarke subdifferential of a locally Lipschitz function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. Recall that a point $x^* \\in \\mathbb{R}^n$ is a local minimum (respectively strict local minimum) of a function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if there exists a positive constant $\\epsilon$ such that $f(x^*) \\leqslant f(x)$ for all $x \\in B(x^*,\\epsilon)\\setminus \\{x^*\\}$ (respectively $f(x^*) < f(x)$). Using Lemma \\ref{lemma:euler_method}, we obtain the following necessary condition for stability.\n","start":14005,"end":14621}]},{"id":"S008","semantic_type":"lemma","kind":"lemma","labels":["eq:Euler","eq:ivp","lemma:euler_method"],"member_ids":["S008","S015","S016","S017","S018","S019","S020","S021","S022","S023"],"declared_external":false,"source_only":false,"title":"有界 Euler 轨道逼近","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lemma:euler_method}\n    Let $F:\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^n$ be an upper semicontinuous mapping with nonempty, compact, and convex values. Let $X_0$ be a compact subset of $\\mathbb{R}^n$ and let $T>0$. Assume that there exist $\\bar{\\alpha},r>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$ and for all sequence $(x_k)_{k\\in\\mathbb{N}}$ such that\n    \\begin{equation}\n    \\label{eq:Euler}\n                x_{k+1} \\in x_k + \\alpha F(x_k), ~~~ \\forall k \\in \\mathbb{N},~~~ x_0 \\in X_0,\n    \\end{equation}\n    we have that $x_0,\\hdots,x_{\\lfloor T/\\alpha \\rfloor+1} \\in B(0,r)$. For all $\\epsilon>0$, there exists $\\hat{\\alpha}\\in (0,\\bar{\\alpha}]$ such that for all $\\alpha \\in (0,\\hat{\\alpha}]$ and for all sequence $(x_k)_{k\\in\\mathbb{N}}$ satisfying \\eqref{eq:Euler}, there exists an absolutely continuous function $x:[0,T]\\rightarrow\\mathbb{R}^n$ such that\n    \\begin{equation}\n    \\label{eq:ivp}\n                x'(t) \\in F(x(t)),~~~ \\text{for a.e.}~ \n                t \\in (0,T), ~~~ x(0) \\in X_0, \n    \\end{equation}\n    for which $\\|\\bar{x}(t) - x(t) \\| \\leqslant \\epsilon$ for all $t \\in [0,T]$, where $\\bar{x}:[0,T]\\rightarrow\\mathbb{R}^n$ is defined by $\\bar{x}(t) := x_k + (t-\\alpha k)/\\alpha  (x_{k+1}-x_k)$ for all $t\\in [\\alpha k , \\min\\{\\alpha(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha\\rfloor\\}$.\n\\end{lemma}","start":8563,"end":9928},{"raw_tex":"\\begin{proof}\nLet $(\\alpha_m)_{m \\in \\mathbb{N}}$ denote a sequence of positive numbers that converges to zero. Without loss of generality, we may assume that the sequence is bounded above by $\\bar{\\alpha}$. To each term in the sequence, we attribute a sequence $(x_k^{m})_{k\\in\\mathbb{N}}$ generated by the Euler method with step size $\\alpha_m$ and initialized in $X_0$, that is to say, which satisfies \\eqref{eq:Euler} with $\\alpha:=\\alpha_m$. By assumption, $x_0^{m},\\hdots,x_{\\lfloor T/\\alpha_m \\rfloor+1}^{m} \\in B(0,r)$. Consider the linear interpolation of those iterates, that is to say, the function $\\bar{x}^{m}:[0,T]\\rightarrow\\mathbb{R}^n$ defined by $\\bar{x}^{m}(t) := x_k^{m} + (t-\\alpha_m k)(x_{k+1}^{m}-x_k^{m})/\\alpha_m$ for all $t\\in [\\alpha_m k , \\min\\{\\alpha_m(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$. Since $B(0,r)$ is convex, it holds that $\\|\\bar{x}^{m}(t)\\| \\leqslant r $ for all $t\\in [0,T]$. In addition, since $F$ is upper semicontinuous and compact valued, by \\cite[Proposition 3 p. 42]{aubin1984differential} there exists $r'>0$ such that $F(B(0,r)) \\subset B(0,r')$. Observe that $(\\bar{x}^{m})'(t) = (x_{k+1}^{m}-x_k^{m})/\\alpha_m \\in F(x_k^{m})$ for all $t\\in (\\alpha_m k , \\min\\{\\alpha_m(k+1),T\\})$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$. Hence, we have that $\\|(\\bar{x}^{m})'(t)\\| \\leqslant r'$ for almost every $t\\in (0,T)$. By successively applying the Arzel\\`a-Ascoli and the Banach-Alaoglu theorems (see \\cite[Theorem 4 p. 13]{aubin1984differential}), there exists a subsequence (again denoted $(\\alpha_{m})_{m\\in \\mathbb{N}}$) and an absolutely continuous function $x:[0,T]\\rightarrow \\mathbb{R}^n$ such that $\\bar{x}^{m}(\\cdot)$ converges uniformly to $x(\\cdot)$ and $(\\bar{x}^{m})'(\\cdot)$ converges weakly to $x'(\\cdot)$ in $L^1([0,T],\\mathbb{R}^n)$. Furthermore, for all $t\\in (\\alpha_{m} k , \\min\\{\\alpha_{m}(k+1),T\\})$ and $k \\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$, observe that\n\\begin{subequations}\n\\begin{align*}\n    (\\bar{x}^{m}(t),(\\bar{x}^{m})'(t)) \n    & = \\left(x_k^{m} + (t-\\alpha_m k)\\frac{x_{k+1}^{m}-x_k^{m}}{\\alpha_m},\\frac{x_{k+1}^{m}-x_k^{m}}{\\alpha_m}\\right) \\\\\n    & \\in \\left(\\{x_k^{m}\\} + (t-\\alpha_m k)F(x_k^{m})\\right) \\times F(x_k^{m})  \\\\\n    & = \\{x_k^{m}\\} \\times F(x_k^{m}) + (t-\\alpha_m k)F(x_k^{m}) \\times \\{0\\}  \\\\\n    & \\subset \\mathrm{graph}(F) + B(0,r'\\alpha_m)\\times \\{0\\}.\n\\end{align*}\n\\end{subequations}\nAccording to \\cite[Convergence Theorem p. 60]{aubin1984differential}, it follows that $x'(t) \\in F(x(t))$ for almost every $t\\in (0,T)$. The sequence of initial points $(x^{m}(0))_{m\\in\\mathbb{N}}$ lies in the closed set $X_0$, hence its limit $x(0)$ lies in $X_0$ as well. As a result, $x(\\cdot)$ is a solution to the differential inclusion \\eqref{eq:ivp}. \n\nTo sum up, we have shown that for every sequence $(\\alpha_m)_{m \\in \\mathbb{N}}$ of positive numbers converging to zero, there exists a subsequence for which the corresponding linear interpolations uniformly converge towards a solution of the differential inclusion \\eqref{eq:ivp}. The conclusion of the theorem now easily follows. To see why, one can reason by contradiction and assume that there exists $\\epsilon>0$ such that for all $\\hat{\\alpha} \\in (0,\\bar{\\alpha}]$, there exist $\\alpha \\in (0,\\hat{\\alpha}]$ and a sequence $(x_k)_{k \\in \\mathbb{N}}$ generated by the Euler method with step size $\\alpha$ and initialized in $X_0$ such that, for any solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp}, it holds that $\\|\\bar{x}(t) - x(t) \\| > \\epsilon$ for some $t \\in [0,T]$. We can then generate a sequence $(\\alpha_m)_{m \\in \\mathbb{N}}$ of positive numbers converging to zero such that, for any solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp}, it holds that $\\|\\bar{x}^{m}(t) - x(t) \\| > \\epsilon$ for some $t \\in [0,T]$. Since there exists a subsequence $(\\alpha_{\\varphi(m)})_{m \\in \\mathbb{N}}$ such that $\\bar{x}^{\\varphi(m)}$ uniformly converges to a solution to the differential inclusion \\eqref{eq:ivp}, we obtain a contradiction. \n\\end{proof}","start":9930,"end":14003},{"raw_tex":"Let $(\\alpha_m)_{m \\in \\mathbb{N}}$ denote a sequence of positive numbers that converges to zero. Without loss of generality, we may assume that the sequence is bounded above by $\\bar{\\alpha}$. To each term in the sequence, we attribute a sequence $(x_k^{m})_{k\\in\\mathbb{N}}$ generated by the Euler method with step size $\\alpha_m$ and initialized in $X_0$, that is to say, which satisfies \\eqref{eq:Euler} with $\\alpha:=\\alpha_m$. By assumption, $x_0^{m},\\hdots,x_{\\lfloor T/\\alpha_m \\rfloor+1}^{m} \\in B(0,r)$. Consider the linear interpolation of those iterates, that is to say, the function $\\bar{x}^{m}:[0,T]\\rightarrow\\mathbb{R}^n$ defined by $\\bar{x}^{m}(t) := x_k^{m} + (t-\\alpha_m k)(x_{k+1}^{m}-x_k^{m})/\\alpha_m$ for all $t\\in [\\alpha_m k , \\min\\{\\alpha_m(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$. Since $B(0,r)$ is convex, it holds that $\\|\\bar{x}^{m}(t)\\| \\leqslant r $ for all $t\\in [0,T]$. In addition, since $F$ is upper semicontinuous and compact valued, by \\cite[Proposition 3 p. 42]{aubin1984differential} there exists $r'>0$ such that $F(B(0,r)) \\subset B(0,r')$. Observe that $(\\bar{x}^{m})'(t) = (x_{k+1}^{m}-x_k^{m})/\\alpha_m \\in F(x_k^{m})$ for all $t\\in (\\alpha_m k , \\min\\{\\alpha_m(k+1),T\\})$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$. Hence, we have that $\\|(\\bar{x}^{m})'(t)\\| \\leqslant r'$ for almost every $t\\in (0,T)$. By successively applying the Arzel\\`a-Ascoli and the Banach-Alaoglu theorems (see \\cite[Theorem 4 p. 13]{aubin1984differential}), there exists a subsequence (again denoted $(\\alpha_{m})_{m\\in \\mathbb{N}}$) and an absolutely continuous function $x:[0,T]\\rightarrow \\mathbb{R}^n$ such that $\\bar{x}^{m}(\\cdot)$ converges uniformly to $x(\\cdot)$ and $(\\bar{x}^{m})'(\\cdot)$ converges weakly to $x'(\\cdot)$ in $L^1([0,T],\\mathbb{R}^n)$. Furthermore, for all $t\\in (\\alpha_{m} k , \\min\\{\\alpha_{m}(k+1),T\\})$ and $k \\in \\{0,\\hdots,\\lfloor T/\\alpha_m \\rfloor\\}$, observe that\n","start":9944,"end":11895},{"raw_tex":"\\begin{subequations}\n\\begin{align*}\n    (\\bar{x}^{m}(t),(\\bar{x}^{m})'(t)) \n    & = \\left(x_k^{m} + (t-\\alpha_m k)\\frac{x_{k+1}^{m}-x_k^{m}}{\\alpha_m},\\frac{x_{k+1}^{m}-x_k^{m}}{\\alpha_m}\\right) \\\\\n    & \\in \\left(\\{x_k^{m}\\} + (t-\\alpha_m k)F(x_k^{m})\\right) \\times F(x_k^{m})  \\\\\n    & = \\{x_k^{m}\\} \\times F(x_k^{m}) + (t-\\alpha_m k)F(x_k^{m}) \\times \\{0\\}  \\\\\n    & \\subset \\mathrm{graph}(F) + B(0,r'\\alpha_m)\\times \\{0\\}.\n\\end{align*}\n\\end{subequations}\n","start":11895,"end":12354},{"raw_tex":"According to \\cite[Convergence Theorem p. 60]{aubin1984differential}, it follows that $x'(t) \\in F(x(t))$ for almost every $t\\in (0,T)$. The sequence of initial points $(x^{m}(0))_{m\\in\\mathbb{N}}$ lies in the closed set $X_0$, hence its limit $x(0)$ lies in $X_0$ as well. As a result, $x(\\cdot)$ is a solution to the differential inclusion \\eqref{eq:ivp}. \n","start":12354,"end":12713},{"raw_tex":"To sum up, we have shown that for every sequence $(\\alpha_m)_{m \\in \\mathbb{N}}$ of positive numbers converging to zero, there exists a subsequence for which the corresponding linear interpolations uniformly converge towards a solution of the differential inclusion \\eqref{eq:ivp}. The conclusion of the theorem now easily follows. To see why, one can reason by contradiction and assume that there exists $\\epsilon>0$ such that for all $\\hat{\\alpha} \\in (0,\\bar{\\alpha}]$, there exist $\\alpha \\in (0,\\hat{\\alpha}]$ and a sequence $(x_k)_{k \\in \\mathbb{N}}$ generated by the Euler method with step size $\\alpha$ and initialized in $X_0$ such that, for any solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp}, it holds that $\\|\\bar{x}(t) - x(t) \\| > \\epsilon$ for some $t \\in [0,T]$. We can then generate a sequence $(\\alpha_m)_{m \\in \\mathbb{N}}$ of positive numbers converging to zero such that, for any solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp}, it holds that $\\|\\bar{x}^{m}(t) - x(t) \\| > \\epsilon$ for some $t \\in [0,T]$. Since there exists a subsequence $(\\alpha_{\\varphi(m)})_{m \\in \\mathbb{N}}$ such that $\\bar{x}^{\\varphi(m)}$ uniformly converges to a solution to the differential inclusion \\eqref{eq:ivp}, we obtain a contradiction. \n","start":12714,"end":13992}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","labels":["lemma:euler_method"],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"集合值映射条件","source_units":[{"raw_tex":"    Let $F:\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^n$ be an upper semicontinuous mapping with nonempty, compact, and convex values. Let $X_0$ be a compact subset of $\\mathbb{R}^n$ and let $T>0$. Assume that there exist $\\bar{\\alpha},r>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$ and for all sequence $(x_k)_{k\\in\\mathbb{N}}$ such that\n","start":8604,"end":8946}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","labels":["lemma:euler_method"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"紧初值集与时域","source_units":[{"raw_tex":"    Let $F:\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^n$ be an upper semicontinuous mapping with nonempty, compact, and convex values. Let $X_0$ be a compact subset of $\\mathbb{R}^n$ and let $T>0$. Assume that there exist $\\bar{\\alpha},r>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$ and for all sequence $(x_k)_{k\\in\\mathbb{N}}$ such that\n","start":8604,"end":8946}]},{"id":"S011","semantic_type":"algorithm","kind":"algorithm","labels":["eq:Euler","lemma:euler_method"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"集合值 Euler 迭代","source_units":[{"raw_tex":"    \\begin{equation}\n    \\label{eq:Euler}\n                x_{k+1} \\in x_k + \\alpha F(x_k), ~~~ \\forall k \\in \\mathbb{N},~~~ x_0 \\in X_0,\n    \\end{equation}\n","start":8946,"end":9102}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","labels":["lemma:euler_method"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"离散轨道一致有界","source_units":[{"raw_tex":"    Let $F:\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^n$ be an upper semicontinuous mapping with nonempty, compact, and convex values. Let $X_0$ be a compact subset of $\\mathbb{R}^n$ and let $T>0$. Assume that there exist $\\bar{\\alpha},r>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$ and for all sequence $(x_k)_{k\\in\\mathbb{N}}$ such that\n","start":8604,"end":8946},{"raw_tex":"    we have that $x_0,\\hdots,x_{\\lfloor T/\\alpha \\rfloor+1} \\in B(0,r)$. For all $\\epsilon>0$, there exists $\\hat{\\alpha}\\in (0,\\bar{\\alpha}]$ such that for all $\\alpha \\in (0,\\hat{\\alpha}]$ and for all sequence $(x_k)_{k\\in\\mathbb{N}}$ satisfying \\eqref{eq:Euler}, there exists an absolutely continuous function $x:[0,T]\\rightarrow\\mathbb{R}^n$ such that\n","start":9102,"end":9458}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["eq:ivp","lemma:euler_method"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"极限微分包含","source_units":[{"raw_tex":"    we have that $x_0,\\hdots,x_{\\lfloor T/\\alpha \\rfloor+1} \\in B(0,r)$. For all $\\epsilon>0$, there exists $\\hat{\\alpha}\\in (0,\\bar{\\alpha}]$ such that for all $\\alpha \\in (0,\\hat{\\alpha}]$ and for all sequence $(x_k)_{k\\in\\mathbb{N}}$ satisfying \\eqref{eq:Euler}, there exists an absolutely continuous function $x:[0,T]\\rightarrow\\mathbb{R}^n$ such that\n    \\begin{equation}\n    \\label{eq:ivp}\n                x'(t) \\in F(x(t)),~~~ \\text{for a.e.}~ \n                t \\in (0,T), ~~~ x(0) \\in X_0, \n    \\end{equation}\n","start":9102,"end":9621}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":["lemma:euler_method"],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"分段线性插值","source_units":[{"raw_tex":"    for which $\\|\\bar{x}(t) - x(t) \\| \\leqslant \\epsilon$ for all $t \\in [0,T]$, where $\\bar{x}:[0,T]\\rightarrow\\mathbb{R}^n$ is defined by $\\bar{x}(t) := x_k + (t-\\alpha k)/\\alpha  (x_{k+1}-x_k)$ for all $t\\in [\\alpha k , \\min\\{\\alpha(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha\\rfloor\\}$.\n","start":9621,"end":9917}]},{"id":"S024","semantic_type":"theorem","kind":"theorem","labels":["thm:necessary"],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"稳定点必为局部极小","source_units":[{"raw_tex":"\\begin{theorem}\n\\label{thm:necessary}\nStable points of locally Lipschitz semi-algebraic functions are local minima.\n\\end{theorem}","start":14622,"end":14751}]},{"id":"S025","semantic_type":"proof","kind":"proof","labels":["eq:ivp_x_0","eq:lyapunov"],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"必要性定理完整证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $x^* \\in \\mathbb{R}^n$ denote a stable point of a locally Lipschitz semi-algebraic function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. We reason by contradiction and assume that $x^*$ is not a local minimum of $f$. Given $x\\in \\mathbb{R}^n$ and $S \\subset \\mathbb{R}^n$, let $d(x,S) := \\inf \\{\\|x - y\\| : y \\in S\\}$. According to the Kurdyka-\\L{}ojasiewicz inequality \\cite[Theorem 14]{bolte2007clarke} (see also \\cite[Theorem 4.1]{attouch2010proximal}), there exist $r,\\rho>0$ and a strictly increasing concave continuous semi-algebraic function $\\psi:[0,\\rho) \\rightarrow [0,\\infty)$ that is continuously differentiable on $(0,\\rho)$ with $\\psi(0) = 0$ such that $d(0,\\partial f(x)) \\geqslant 1/\\psi'(|f(x) - f(x^*)|)$ for all $x \\in B(x^*,r)$ whenever $0<|f(x)-f(x^*)|< \\rho$. After possibly reducing $r$, the inequality holds for all $x \\in B(x^*,r)$ such that $f(x) \\neq f(x^*)$. \n\nLet $\\epsilon \\in (0,r/2)$. By the definition of stability (Definition \\ref{def:discrete_lyapunov}), there exist $\\delta>0$ and $\\bar{\\alpha}>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$, the subgradient method with constant step size $\\alpha$ initialized in $B(x^*,\\delta)$ has all its iterates in $B(x^*,\\epsilon)$. Since $x^*$ is not a local minimum, we can take an initial iterate $x_0$ in $B(x^*,\\delta)$ such that $f(x_0)<f(x^*)$. Let $L>0$ denote a Lipschitz constant of $f$ on $B(x^*,r)$ and let $T := (2\\epsilon + \\delta)L\\psi'(f(x^*) - f(x_0))^2>0$.\nConsider the differential inclusion\n\\begin{equation}\n\\label{eq:ivp_x_0}\n     x'(t) \\in - \\partial f(x(t)), ~~~ \\text{for a.e.}~ \n     t \\in (0,T),~~~ x(0) = x_0.\n\\end{equation}\nSince $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d)]{clarke1990} with nonempty, compact and convex values \\cite[2.1.2 Proposition (a)]{clarke1990}. By Lemma \\ref{lemma:euler_method}, there exists $\\hat{\\alpha} \\in (0,\\bar{\\alpha}]$ such that, for all $\\alpha \\in (0,\\hat{\\alpha}]$  and for all sequence $(x_k)_{k\\in \\mathbb{N}}$ generated by the subgradient method with constant step size $\\alpha$ and initialized at $x_0$, there exists a solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp_x_0} for which $\\|\\bar{x}(t) - x(t)\\| \\leqslant \\epsilon/2$ for all $t \\in [0,T]$, where $\\bar{x}:[0,T]\\rightarrow\\mathbb{R}^n$ is the piecewise linear function defined by $\\bar{x}(t) := x_k + (t-\\alpha k)/\\alpha  (x_{k+1}-x_k)$ for all $t\\in [\\alpha k , \\min\\{\\alpha(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha\\rfloor\\}$.\n\nLet us fix some $\\alpha \\in (0,\\min\\{\\hat{\\alpha},T/10\\}]$ from now on. Consider a sequence $(x_k)_{k\\in\\mathbb{N}}$ generated by the subgradient method with constant step size $\\alpha$ and initialized at $x_0$. Consider also the linear interpolation $\\bar{x}(\\cdot)$ of those iterates up to iteration $K+1$ where $K := \\lfloor T/\\alpha \\rfloor$, as well as a solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp_x_0} such that $\\|\\bar{x}(t)-x(t)\\| \\leqslant \\epsilon/2$ for all $t\\in [0,T]$. Since $f$ is semi-algebraic, by \\cite[Lemma 5.2]{davis2020stochastic} (see also \\cite{drusvyatskiy2015curves}) it holds that\n\\begin{equation}\n\\label{eq:lyapunov}\n    f(x(t)) - f(x(0)) = - \\int_0^t d(0,\\partial f(x(\\tau)))^2 d\\tau, ~~~ \\forall t \\in [0,T].\n\\end{equation}\nAs a result, $f(x(t)) \\leqslant f(x(0)) = f(x_0)$ for all $t \\in [0,T]$. Also, $\\|x(t) - x^*\\| \\leqslant \\|x(t) - \\bar{x}(t)\\| + \\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon/2 + \\epsilon = 3\\epsilon/2 \\leqslant r$, where the inequality $\\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon$ follows from the convexity of $B(x^*,\\epsilon)$. Hence $0<f(x^*) - f(x_0) \\leqslant f(x^*) - f(x(t))$ and $d(0,\\partial f(x(t))) \\geqslant  1/\\psi'(f(x^*)-f(x(t))) \\geqslant 1/\\psi'(f(x^*)-f(x_0))$ for all $t\\in[0,T]$ by concavity of $\\psi$. Together with \\eqref{eq:lyapunov}, it follows that $f(x(K\\alpha)) - f(x_0) \\leqslant -K\\alpha M^2$ where $K := \\lfloor T/\\alpha \\rfloor$ and $M:= 1/\\psi'(f(x^*)-f(x_0))$. Recall that $L$ is a Lipschitz constant of $f$ on $B(x^*,r)$, so that we have $|f(x(K\\alpha))-f(x_0)| \\leqslant L \\|x(K\\alpha)-x_0\\|$. We thus obtain the lower bound $\\|x(K\\alpha)-x_0\\| \\geqslant K\\alpha M^2/L = \\lfloor T/\\alpha \\rfloor \\alpha M^2/L \\geqslant (T-\\alpha)M^2/L \\geqslant (T-T/10)M^2/L = 9/10( 2\\epsilon + \\delta) $ (recall that $T = (2\\epsilon + \\delta)L/M^2$). Hence $\\|x(K\\alpha)-x^*\\| \\geqslant \\|x(K\\alpha)-x_0\\| - \\|x_0-x^*\\| \\geqslant 9/10(2\\epsilon + \\delta) - \\delta = 9 \\epsilon/5 - \\delta/10 \\geqslant 9 \\epsilon/5 - \\epsilon/10 = 17\\epsilon/10$ and finally $\\|x_K-x^*\\| \\geqslant \\|x(K\\alpha)-x^*\\| -\\|x_K-x(K\\alpha)\\| \\geqslant 17\\epsilon/10 - \\epsilon/2 = 6\\epsilon/5>\\epsilon$. However, since $0<\\alpha \\leqslant \\hat{\\alpha} \\leqslant \\bar{\\alpha}$ and $x_0 \\in B(x^*,\\delta)$, by stability of $x^*$ we have $x_K \\in B(x^*,\\epsilon)$. We have reached a contradiction. \n\\end{proof}","start":14753,"end":19670}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"反设非局部极小","source_units":[{"raw_tex":"Let $x^* \\in \\mathbb{R}^n$ denote a stable point of a locally Lipschitz semi-algebraic function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. We reason by contradiction and assume that $x^*$ is not a local minimum of $f$. Given $x\\in \\mathbb{R}^n$ and $S \\subset \\mathbb{R}^n$, let $d(x,S) := \\inf \\{\\|x - y\\| : y \\in S\\}$. According to the Kurdyka-\\L{}ojasiewicz inequality \\cite[Theorem 14]{bolte2007clarke} (see also \\cite[Theorem 4.1]{attouch2010proximal}), there exist $r,\\rho>0$ and a strictly increasing concave continuous semi-algebraic function $\\psi:[0,\\rho) \\rightarrow [0,\\infty)$ that is continuously differentiable on $(0,\\rho)$ with $\\psi(0) = 0$ such that $d(0,\\partial f(x)) \\geqslant 1/\\psi'(|f(x) - f(x^*)|)$ for all $x \\in B(x^*,r)$ whenever $0<|f(x)-f(x^*)|< \\rho$. After possibly reducing $r$, the inequality holds for all $x \\in B(x^*,r)$ such that $f(x) \\neq f(x^*)$. \n","start":14767,"end":15653}]},{"id":"S027","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"点集距离","source_units":[{"raw_tex":"Let $x^* \\in \\mathbb{R}^n$ denote a stable point of a locally Lipschitz semi-algebraic function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. We reason by contradiction and assume that $x^*$ is not a local minimum of $f$. Given $x\\in \\mathbb{R}^n$ and $S \\subset \\mathbb{R}^n$, let $d(x,S) := \\inf \\{\\|x - y\\| : y \\in S\\}$. According to the Kurdyka-\\L{}ojasiewicz inequality \\cite[Theorem 14]{bolte2007clarke} (see also \\cite[Theorem 4.1]{attouch2010proximal}), there exist $r,\\rho>0$ and a strictly increasing concave continuous semi-algebraic function $\\psi:[0,\\rho) \\rightarrow [0,\\infty)$ that is continuously differentiable on $(0,\\rho)$ with $\\psi(0) = 0$ such that $d(0,\\partial f(x)) \\geqslant 1/\\psi'(|f(x) - f(x^*)|)$ for all $x \\in B(x^*,r)$ whenever $0<|f(x)-f(x^*)|< \\rho$. After possibly reducing $r$, the inequality holds for all $x \\in B(x^*,r)$ such that $f(x) \\neq f(x^*)$. \n","start":14767,"end":15653}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"局部 KL 下界","source_units":[{"raw_tex":"Let $x^* \\in \\mathbb{R}^n$ denote a stable point of a locally Lipschitz semi-algebraic function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. We reason by contradiction and assume that $x^*$ is not a local minimum of $f$. Given $x\\in \\mathbb{R}^n$ and $S \\subset \\mathbb{R}^n$, let $d(x,S) := \\inf \\{\\|x - y\\| : y \\in S\\}$. According to the Kurdyka-\\L{}ojasiewicz inequality \\cite[Theorem 14]{bolte2007clarke} (see also \\cite[Theorem 4.1]{attouch2010proximal}), there exist $r,\\rho>0$ and a strictly increasing concave continuous semi-algebraic function $\\psi:[0,\\rho) \\rightarrow [0,\\infty)$ that is continuously differentiable on $(0,\\rho)$ with $\\psi(0) = 0$ such that $d(0,\\partial f(x)) \\geqslant 1/\\psi'(|f(x) - f(x^*)|)$ for all $x \\in B(x^*,r)$ whenever $0<|f(x)-f(x^*)|< \\rho$. After possibly reducing $r$, the inequality holds for all $x \\in B(x^*,r)$ such that $f(x) \\neq f(x^*)$. \n","start":14767,"end":15653}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"稳定性控制离散轨道","source_units":[{"raw_tex":"Let $\\epsilon \\in (0,r/2)$. By the definition of stability (Definition \\ref{def:discrete_lyapunov}), there exist $\\delta>0$ and $\\bar{\\alpha}>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$, the subgradient method with constant step size $\\alpha$ initialized in $B(x^*,\\delta)$ has all its iterates in $B(x^*,\\epsilon)$. Since $x^*$ is not a local minimum, we can take an initial iterate $x_0$ in $B(x^*,\\delta)$ such that $f(x_0)<f(x^*)$. Let $L>0$ denote a Lipschitz constant of $f$ on $B(x^*,r)$ and let $T := (2\\epsilon + \\delta)L\\psi'(f(x^*) - f(x_0))^2>0$.\n","start":15654,"end":16220}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"选择低值初点","source_units":[{"raw_tex":"Let $\\epsilon \\in (0,r/2)$. By the definition of stability (Definition \\ref{def:discrete_lyapunov}), there exist $\\delta>0$ and $\\bar{\\alpha}>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$, the subgradient method with constant step size $\\alpha$ initialized in $B(x^*,\\delta)$ has all its iterates in $B(x^*,\\epsilon)$. Since $x^*$ is not a local minimum, we can take an initial iterate $x_0$ in $B(x^*,\\delta)$ such that $f(x_0)<f(x^*)$. Let $L>0$ denote a Lipschitz constant of $f$ on $B(x^*,r)$ and let $T := (2\\epsilon + \\delta)L\\psi'(f(x^*) - f(x_0))^2>0$.\n","start":15654,"end":16220}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"设定 Lipschitz 常数与时域","source_units":[{"raw_tex":"Let $\\epsilon \\in (0,r/2)$. By the definition of stability (Definition \\ref{def:discrete_lyapunov}), there exist $\\delta>0$ and $\\bar{\\alpha}>0$ such that for all $\\alpha \\in (0,\\bar{\\alpha}]$, the subgradient method with constant step size $\\alpha$ initialized in $B(x^*,\\delta)$ has all its iterates in $B(x^*,\\epsilon)$. Since $x^*$ is not a local minimum, we can take an initial iterate $x_0$ in $B(x^*,\\delta)$ such that $f(x_0)<f(x^*)$. Let $L>0$ denote a Lipschitz constant of $f$ on $B(x^*,r)$ and let $T := (2\\epsilon + \\delta)L\\psi'(f(x^*) - f(x_0))^2>0$.\n","start":15654,"end":16220}]},{"id":"S032","semantic_type":"definition","kind":"definition","labels":["eq:ivp_x_0"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"指定初值的连续动力学","source_units":[{"raw_tex":"Consider the differential inclusion\n\\begin{equation}\n\\label{eq:ivp_x_0}\n     x'(t) \\in - \\partial f(x(t)), ~~~ \\text{for a.e.}~ \n     t \\in (0,T),~~~ x(0) = x_0.\n\\end{equation}\n","start":16220,"end":16397}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"核对 Clarke 映射条件","source_units":[{"raw_tex":"Since $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d)]{clarke1990} with nonempty, compact and convex values \\cite[2.1.2 Proposition (a)]{clarke1990}. By Lemma \\ref{lemma:euler_method}, there exists $\\hat{\\alpha} \\in (0,\\bar{\\alpha}]$ such that, for all $\\alpha \\in (0,\\hat{\\alpha}]$  and for all sequence $(x_k)_{k\\in \\mathbb{N}}$ generated by the subgradient method with constant step size $\\alpha$ and initialized at $x_0$, there exists a solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp_x_0} for which $\\|\\bar{x}(t) - x(t)\\| \\leqslant \\epsilon/2$ for all $t \\in [0,T]$, where $\\bar{x}:[0,T]\\rightarrow\\mathbb{R}^n$ is the piecewise linear function defined by $\\bar{x}(t) := x_k + (t-\\alpha k)/\\alpha  (x_{k+1}-x_k)$ for all $t\\in [\\alpha k , \\min\\{\\alpha(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha\\rfloor\\}$.\n","start":16397,"end":17298}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"取得连续轨道逼近","source_units":[{"raw_tex":"Since $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d)]{clarke1990} with nonempty, compact and convex values \\cite[2.1.2 Proposition (a)]{clarke1990}. By Lemma \\ref{lemma:euler_method}, there exists $\\hat{\\alpha} \\in (0,\\bar{\\alpha}]$ such that, for all $\\alpha \\in (0,\\hat{\\alpha}]$  and for all sequence $(x_k)_{k\\in \\mathbb{N}}$ generated by the subgradient method with constant step size $\\alpha$ and initialized at $x_0$, there exists a solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp_x_0} for which $\\|\\bar{x}(t) - x(t)\\| \\leqslant \\epsilon/2$ for all $t \\in [0,T]$, where $\\bar{x}:[0,T]\\rightarrow\\mathbb{R}^n$ is the piecewise linear function defined by $\\bar{x}(t) := x_k + (t-\\alpha k)/\\alpha  (x_{k+1}-x_k)$ for all $t\\in [\\alpha k , \\min\\{\\alpha(k+1),T\\}]$ and $k\\in \\{0,\\hdots,\\lfloor T/\\alpha\\rfloor\\}$.\n","start":16397,"end":17298}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"固定网格与逼近解","source_units":[{"raw_tex":"Let us fix some $\\alpha \\in (0,\\min\\{\\hat{\\alpha},T/10\\}]$ from now on. Consider a sequence $(x_k)_{k\\in\\mathbb{N}}$ generated by the subgradient method with constant step size $\\alpha$ and initialized at $x_0$. Consider also the linear interpolation $\\bar{x}(\\cdot)$ of those iterates up to iteration $K+1$ where $K := \\lfloor T/\\alpha \\rfloor$, as well as a solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp_x_0} such that $\\|\\bar{x}(t)-x(t)\\| \\leqslant \\epsilon/2$ for all $t\\in [0,T]$. Since $f$ is semi-algebraic, by \\cite[Lemma 5.2]{davis2020stochastic} (see also \\cite{drusvyatskiy2015curves}) it holds that\n","start":17299,"end":17928}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":["eq:lyapunov"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"连续轨道能量恒等式","source_units":[{"raw_tex":"Let us fix some $\\alpha \\in (0,\\min\\{\\hat{\\alpha},T/10\\}]$ from now on. Consider a sequence $(x_k)_{k\\in\\mathbb{N}}$ generated by the subgradient method with constant step size $\\alpha$ and initialized at $x_0$. Consider also the linear interpolation $\\bar{x}(\\cdot)$ of those iterates up to iteration $K+1$ where $K := \\lfloor T/\\alpha \\rfloor$, as well as a solution $x(\\cdot)$ to the differential inclusion \\eqref{eq:ivp_x_0} such that $\\|\\bar{x}(t)-x(t)\\| \\leqslant \\epsilon/2$ for all $t\\in [0,T]$. Since $f$ is semi-algebraic, by \\cite[Lemma 5.2]{davis2020stochastic} (see also \\cite{drusvyatskiy2015curves}) it holds that\n\\begin{equation}\n\\label{eq:lyapunov}\n    f(x(t)) - f(x(0)) = - \\int_0^t d(0,\\partial f(x(\\tau)))^2 d\\tau, ~~~ \\forall t \\in [0,T].\n\\end{equation}\n","start":17299,"end":18074}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"函数值保持低于临界值","source_units":[{"raw_tex":"As a result, $f(x(t)) \\leqslant f(x(0)) = f(x_0)$ for all $t \\in [0,T]$. Also, $\\|x(t) - x^*\\| \\leqslant \\|x(t) - \\bar{x}(t)\\| + \\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon/2 + \\epsilon = 3\\epsilon/2 \\leqslant r$, where the inequality $\\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon$ follows from the convexity of $B(x^*,\\epsilon)$. Hence $0<f(x^*) - f(x_0) \\leqslant f(x^*) - f(x(t))$ and $d(0,\\partial f(x(t))) \\geqslant  1/\\psi'(f(x^*)-f(x(t))) \\geqslant 1/\\psi'(f(x^*)-f(x_0))$ for all $t\\in[0,T]$ by concavity of $\\psi$. Together with \\eqref{eq:lyapunov}, it follows that $f(x(K\\alpha)) - f(x_0) \\leqslant -K\\alpha M^2$ where $K := \\lfloor T/\\alpha \\rfloor$ and $M:= 1/\\psi'(f(x^*)-f(x_0))$. Recall that $L$ is a Lipschitz constant of $f$ on $B(x^*,r)$, so that we have $|f(x(K\\alpha))-f(x_0)| \\leqslant L \\|x(K\\alpha)-x_0\\|$. We thus obtain the lower bound $\\|x(K\\alpha)-x_0\\| \\geqslant K\\alpha M^2/L = \\lfloor T/\\alpha \\rfloor \\alpha M^2/L \\geqslant (T-\\alpha)M^2/L \\geqslant (T-T/10)M^2/L = 9/10( 2\\epsilon + \\delta) $ (recall that $T = (2\\epsilon + \\delta)L/M^2$). Hence $\\|x(K\\alpha)-x^*\\| \\geqslant \\|x(K\\alpha)-x_0\\| - \\|x_0-x^*\\| \\geqslant 9/10(2\\epsilon + \\delta) - \\delta = 9 \\epsilon/5 - \\delta/10 \\geqslant 9 \\epsilon/5 - \\epsilon/10 = 17\\epsilon/10$ and finally $\\|x_K-x^*\\| \\geqslant \\|x(K\\alpha)-x^*\\| -\\|x_K-x(K\\alpha)\\| \\geqslant 17\\epsilon/10 - \\epsilon/2 = 6\\epsilon/5>\\epsilon$. However, since $0<\\alpha \\leqslant \\hat{\\alpha} \\leqslant \\bar{\\alpha}$ and $x_0 \\in B(x^*,\\delta)$, by stability of $x^*$ we have $x_K \\in B(x^*,\\epsilon)$. We have reached a contradiction. \n","start":18074,"end":19659}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"连续轨道留在 KL 球","source_units":[{"raw_tex":"As a result, $f(x(t)) \\leqslant f(x(0)) = f(x_0)$ for all $t \\in [0,T]$. Also, $\\|x(t) - x^*\\| \\leqslant \\|x(t) - \\bar{x}(t)\\| + \\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon/2 + \\epsilon = 3\\epsilon/2 \\leqslant r$, where the inequality $\\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon$ follows from the convexity of $B(x^*,\\epsilon)$. Hence $0<f(x^*) - f(x_0) \\leqslant f(x^*) - f(x(t))$ and $d(0,\\partial f(x(t))) \\geqslant  1/\\psi'(f(x^*)-f(x(t))) \\geqslant 1/\\psi'(f(x^*)-f(x_0))$ for all $t\\in[0,T]$ by concavity of $\\psi$. Together with \\eqref{eq:lyapunov}, it follows that $f(x(K\\alpha)) - f(x_0) \\leqslant -K\\alpha M^2$ where $K := \\lfloor T/\\alpha \\rfloor$ and $M:= 1/\\psi'(f(x^*)-f(x_0))$. Recall that $L$ is a Lipschitz constant of $f$ on $B(x^*,r)$, so that we have $|f(x(K\\alpha))-f(x_0)| \\leqslant L \\|x(K\\alpha)-x_0\\|$. We thus obtain the lower bound $\\|x(K\\alpha)-x_0\\| \\geqslant K\\alpha M^2/L = \\lfloor T/\\alpha \\rfloor \\alpha M^2/L \\geqslant (T-\\alpha)M^2/L \\geqslant (T-T/10)M^2/L = 9/10( 2\\epsilon + \\delta) $ (recall that $T = (2\\epsilon + \\delta)L/M^2$). Hence $\\|x(K\\alpha)-x^*\\| \\geqslant \\|x(K\\alpha)-x_0\\| - \\|x_0-x^*\\| \\geqslant 9/10(2\\epsilon + \\delta) - \\delta = 9 \\epsilon/5 - \\delta/10 \\geqslant 9 \\epsilon/5 - \\epsilon/10 = 17\\epsilon/10$ and finally $\\|x_K-x^*\\| \\geqslant \\|x(K\\alpha)-x^*\\| -\\|x_K-x(K\\alpha)\\| \\geqslant 17\\epsilon/10 - \\epsilon/2 = 6\\epsilon/5>\\epsilon$. However, since $0<\\alpha \\leqslant \\hat{\\alpha} \\leqslant \\bar{\\alpha}$ and $x_0 \\in B(x^*,\\delta)$, by stability of $x^*$ we have $x_K \\in B(x^*,\\epsilon)$. We have reached a contradiction. \n","start":18074,"end":19659}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"获得统一正斜率","source_units":[{"raw_tex":"As a result, $f(x(t)) \\leqslant f(x(0)) = f(x_0)$ for all $t \\in [0,T]$. Also, $\\|x(t) - x^*\\| \\leqslant \\|x(t) - \\bar{x}(t)\\| + \\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon/2 + \\epsilon = 3\\epsilon/2 \\leqslant r$, where the inequality $\\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon$ follows from the convexity of $B(x^*,\\epsilon)$. Hence $0<f(x^*) - f(x_0) \\leqslant f(x^*) - f(x(t))$ and $d(0,\\partial f(x(t))) \\geqslant  1/\\psi'(f(x^*)-f(x(t))) \\geqslant 1/\\psi'(f(x^*)-f(x_0))$ for all $t\\in[0,T]$ by concavity of $\\psi$. Together with \\eqref{eq:lyapunov}, it follows that $f(x(K\\alpha)) - f(x_0) \\leqslant -K\\alpha M^2$ where $K := \\lfloor T/\\alpha \\rfloor$ and $M:= 1/\\psi'(f(x^*)-f(x_0))$. Recall that $L$ is a Lipschitz constant of $f$ on $B(x^*,r)$, so that we have $|f(x(K\\alpha))-f(x_0)| \\leqslant L \\|x(K\\alpha)-x_0\\|$. We thus obtain the lower bound $\\|x(K\\alpha)-x_0\\| \\geqslant K\\alpha M^2/L = \\lfloor T/\\alpha \\rfloor \\alpha M^2/L \\geqslant (T-\\alpha)M^2/L \\geqslant (T-T/10)M^2/L = 9/10( 2\\epsilon + \\delta) $ (recall that $T = (2\\epsilon + \\delta)L/M^2$). Hence $\\|x(K\\alpha)-x^*\\| \\geqslant \\|x(K\\alpha)-x_0\\| - \\|x_0-x^*\\| \\geqslant 9/10(2\\epsilon + \\delta) - \\delta = 9 \\epsilon/5 - \\delta/10 \\geqslant 9 \\epsilon/5 - \\epsilon/10 = 17\\epsilon/10$ and finally $\\|x_K-x^*\\| \\geqslant \\|x(K\\alpha)-x^*\\| -\\|x_K-x(K\\alpha)\\| \\geqslant 17\\epsilon/10 - \\epsilon/2 = 6\\epsilon/5>\\epsilon$. However, since $0<\\alpha \\leqslant \\hat{\\alpha} \\leqslant \\bar{\\alpha}$ and $x_0 \\in B(x^*,\\delta)$, by stability of $x^*$ we have $x_K \\in B(x^*,\\epsilon)$. We have reached a contradiction. \n","start":18074,"end":19659}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"累计定量能量下降","source_units":[{"raw_tex":"As a result, $f(x(t)) \\leqslant f(x(0)) = f(x_0)$ for all $t \\in [0,T]$. Also, $\\|x(t) - x^*\\| \\leqslant \\|x(t) - \\bar{x}(t)\\| + \\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon/2 + \\epsilon = 3\\epsilon/2 \\leqslant r$, where the inequality $\\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon$ follows from the convexity of $B(x^*,\\epsilon)$. Hence $0<f(x^*) - f(x_0) \\leqslant f(x^*) - f(x(t))$ and $d(0,\\partial f(x(t))) \\geqslant  1/\\psi'(f(x^*)-f(x(t))) \\geqslant 1/\\psi'(f(x^*)-f(x_0))$ for all $t\\in[0,T]$ by concavity of $\\psi$. Together with \\eqref{eq:lyapunov}, it follows that $f(x(K\\alpha)) - f(x_0) \\leqslant -K\\alpha M^2$ where $K := \\lfloor T/\\alpha \\rfloor$ and $M:= 1/\\psi'(f(x^*)-f(x_0))$. Recall that $L$ is a Lipschitz constant of $f$ on $B(x^*,r)$, so that we have $|f(x(K\\alpha))-f(x_0)| \\leqslant L \\|x(K\\alpha)-x_0\\|$. We thus obtain the lower bound $\\|x(K\\alpha)-x_0\\| \\geqslant K\\alpha M^2/L = \\lfloor T/\\alpha \\rfloor \\alpha M^2/L \\geqslant (T-\\alpha)M^2/L \\geqslant (T-T/10)M^2/L = 9/10( 2\\epsilon + \\delta) $ (recall that $T = (2\\epsilon + \\delta)L/M^2$). Hence $\\|x(K\\alpha)-x^*\\| \\geqslant \\|x(K\\alpha)-x_0\\| - \\|x_0-x^*\\| \\geqslant 9/10(2\\epsilon + \\delta) - \\delta = 9 \\epsilon/5 - \\delta/10 \\geqslant 9 \\epsilon/5 - \\epsilon/10 = 17\\epsilon/10$ and finally $\\|x_K-x^*\\| \\geqslant \\|x(K\\alpha)-x^*\\| -\\|x_K-x(K\\alpha)\\| \\geqslant 17\\epsilon/10 - \\epsilon/2 = 6\\epsilon/5>\\epsilon$. However, since $0<\\alpha \\leqslant \\hat{\\alpha} \\leqslant \\bar{\\alpha}$ and $x_0 \\in B(x^*,\\delta)$, by stability of $x^*$ we have $x_K \\in B(x^*,\\epsilon)$. We have reached a contradiction. \n","start":18074,"end":19659}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"能量下降迫使位移","source_units":[{"raw_tex":"As a result, $f(x(t)) \\leqslant f(x(0)) = f(x_0)$ for all $t \\in [0,T]$. Also, $\\|x(t) - x^*\\| \\leqslant \\|x(t) - \\bar{x}(t)\\| + \\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon/2 + \\epsilon = 3\\epsilon/2 \\leqslant r$, where the inequality $\\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon$ follows from the convexity of $B(x^*,\\epsilon)$. Hence $0<f(x^*) - f(x_0) \\leqslant f(x^*) - f(x(t))$ and $d(0,\\partial f(x(t))) \\geqslant  1/\\psi'(f(x^*)-f(x(t))) \\geqslant 1/\\psi'(f(x^*)-f(x_0))$ for all $t\\in[0,T]$ by concavity of $\\psi$. Together with \\eqref{eq:lyapunov}, it follows that $f(x(K\\alpha)) - f(x_0) \\leqslant -K\\alpha M^2$ where $K := \\lfloor T/\\alpha \\rfloor$ and $M:= 1/\\psi'(f(x^*)-f(x_0))$. Recall that $L$ is a Lipschitz constant of $f$ on $B(x^*,r)$, so that we have $|f(x(K\\alpha))-f(x_0)| \\leqslant L \\|x(K\\alpha)-x_0\\|$. We thus obtain the lower bound $\\|x(K\\alpha)-x_0\\| \\geqslant K\\alpha M^2/L = \\lfloor T/\\alpha \\rfloor \\alpha M^2/L \\geqslant (T-\\alpha)M^2/L \\geqslant (T-T/10)M^2/L = 9/10( 2\\epsilon + \\delta) $ (recall that $T = (2\\epsilon + \\delta)L/M^2$). Hence $\\|x(K\\alpha)-x^*\\| \\geqslant \\|x(K\\alpha)-x_0\\| - \\|x_0-x^*\\| \\geqslant 9/10(2\\epsilon + \\delta) - \\delta = 9 \\epsilon/5 - \\delta/10 \\geqslant 9 \\epsilon/5 - \\epsilon/10 = 17\\epsilon/10$ and finally $\\|x_K-x^*\\| \\geqslant \\|x(K\\alpha)-x^*\\| -\\|x_K-x(K\\alpha)\\| \\geqslant 17\\epsilon/10 - \\epsilon/2 = 6\\epsilon/5>\\epsilon$. However, since $0<\\alpha \\leqslant \\hat{\\alpha} \\leqslant \\bar{\\alpha}$ and $x_0 \\in B(x^*,\\delta)$, by stability of $x^*$ we have $x_K \\in B(x^*,\\epsilon)$. We have reached a contradiction. \n","start":18074,"end":19659}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"连续终点远离稳定点","source_units":[{"raw_tex":"As a result, $f(x(t)) \\leqslant f(x(0)) = f(x_0)$ for all $t \\in [0,T]$. Also, $\\|x(t) - x^*\\| \\leqslant \\|x(t) - \\bar{x}(t)\\| + \\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon/2 + \\epsilon = 3\\epsilon/2 \\leqslant r$, where the inequality $\\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon$ follows from the convexity of $B(x^*,\\epsilon)$. Hence $0<f(x^*) - f(x_0) \\leqslant f(x^*) - f(x(t))$ and $d(0,\\partial f(x(t))) \\geqslant  1/\\psi'(f(x^*)-f(x(t))) \\geqslant 1/\\psi'(f(x^*)-f(x_0))$ for all $t\\in[0,T]$ by concavity of $\\psi$. Together with \\eqref{eq:lyapunov}, it follows that $f(x(K\\alpha)) - f(x_0) \\leqslant -K\\alpha M^2$ where $K := \\lfloor T/\\alpha \\rfloor$ and $M:= 1/\\psi'(f(x^*)-f(x_0))$. Recall that $L$ is a Lipschitz constant of $f$ on $B(x^*,r)$, so that we have $|f(x(K\\alpha))-f(x_0)| \\leqslant L \\|x(K\\alpha)-x_0\\|$. We thus obtain the lower bound $\\|x(K\\alpha)-x_0\\| \\geqslant K\\alpha M^2/L = \\lfloor T/\\alpha \\rfloor \\alpha M^2/L \\geqslant (T-\\alpha)M^2/L \\geqslant (T-T/10)M^2/L = 9/10( 2\\epsilon + \\delta) $ (recall that $T = (2\\epsilon + \\delta)L/M^2$). Hence $\\|x(K\\alpha)-x^*\\| \\geqslant \\|x(K\\alpha)-x_0\\| - \\|x_0-x^*\\| \\geqslant 9/10(2\\epsilon + \\delta) - \\delta = 9 \\epsilon/5 - \\delta/10 \\geqslant 9 \\epsilon/5 - \\epsilon/10 = 17\\epsilon/10$ and finally $\\|x_K-x^*\\| \\geqslant \\|x(K\\alpha)-x^*\\| -\\|x_K-x(K\\alpha)\\| \\geqslant 17\\epsilon/10 - \\epsilon/2 = 6\\epsilon/5>\\epsilon$. However, since $0<\\alpha \\leqslant \\hat{\\alpha} \\leqslant \\bar{\\alpha}$ and $x_0 \\in B(x^*,\\delta)$, by stability of $x^*$ we have $x_K \\in B(x^*,\\epsilon)$. We have reached a contradiction. \n","start":18074,"end":19659}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"离散节点越出稳定球","source_units":[{"raw_tex":"As a result, $f(x(t)) \\leqslant f(x(0)) = f(x_0)$ for all $t \\in [0,T]$. Also, $\\|x(t) - x^*\\| \\leqslant \\|x(t) - \\bar{x}(t)\\| + \\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon/2 + \\epsilon = 3\\epsilon/2 \\leqslant r$, where the inequality $\\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon$ follows from the convexity of $B(x^*,\\epsilon)$. Hence $0<f(x^*) - f(x_0) \\leqslant f(x^*) - f(x(t))$ and $d(0,\\partial f(x(t))) \\geqslant  1/\\psi'(f(x^*)-f(x(t))) \\geqslant 1/\\psi'(f(x^*)-f(x_0))$ for all $t\\in[0,T]$ by concavity of $\\psi$. Together with \\eqref{eq:lyapunov}, it follows that $f(x(K\\alpha)) - f(x_0) \\leqslant -K\\alpha M^2$ where $K := \\lfloor T/\\alpha \\rfloor$ and $M:= 1/\\psi'(f(x^*)-f(x_0))$. Recall that $L$ is a Lipschitz constant of $f$ on $B(x^*,r)$, so that we have $|f(x(K\\alpha))-f(x_0)| \\leqslant L \\|x(K\\alpha)-x_0\\|$. We thus obtain the lower bound $\\|x(K\\alpha)-x_0\\| \\geqslant K\\alpha M^2/L = \\lfloor T/\\alpha \\rfloor \\alpha M^2/L \\geqslant (T-\\alpha)M^2/L \\geqslant (T-T/10)M^2/L = 9/10( 2\\epsilon + \\delta) $ (recall that $T = (2\\epsilon + \\delta)L/M^2$). Hence $\\|x(K\\alpha)-x^*\\| \\geqslant \\|x(K\\alpha)-x_0\\| - \\|x_0-x^*\\| \\geqslant 9/10(2\\epsilon + \\delta) - \\delta = 9 \\epsilon/5 - \\delta/10 \\geqslant 9 \\epsilon/5 - \\epsilon/10 = 17\\epsilon/10$ and finally $\\|x_K-x^*\\| \\geqslant \\|x(K\\alpha)-x^*\\| -\\|x_K-x(K\\alpha)\\| \\geqslant 17\\epsilon/10 - \\epsilon/2 = 6\\epsilon/5>\\epsilon$. However, since $0<\\alpha \\leqslant \\hat{\\alpha} \\leqslant \\bar{\\alpha}$ and $x_0 \\in B(x^*,\\delta)$, by stability of $x^*$ we have $x_K \\in B(x^*,\\epsilon)$. We have reached a contradiction. \n","start":18074,"end":19659}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"与稳定性矛盾","source_units":[{"raw_tex":"As a result, $f(x(t)) \\leqslant f(x(0)) = f(x_0)$ for all $t \\in [0,T]$. Also, $\\|x(t) - x^*\\| \\leqslant \\|x(t) - \\bar{x}(t)\\| + \\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon/2 + \\epsilon = 3\\epsilon/2 \\leqslant r$, where the inequality $\\|\\bar{x}(t) - x^*\\| \\leqslant \\epsilon$ follows from the convexity of $B(x^*,\\epsilon)$. Hence $0<f(x^*) - f(x_0) \\leqslant f(x^*) - f(x(t))$ and $d(0,\\partial f(x(t))) \\geqslant  1/\\psi'(f(x^*)-f(x(t))) \\geqslant 1/\\psi'(f(x^*)-f(x_0))$ for all $t\\in[0,T]$ by concavity of $\\psi$. Together with \\eqref{eq:lyapunov}, it follows that $f(x(K\\alpha)) - f(x_0) \\leqslant -K\\alpha M^2$ where $K := \\lfloor T/\\alpha \\rfloor$ and $M:= 1/\\psi'(f(x^*)-f(x_0))$. Recall that $L$ is a Lipschitz constant of $f$ on $B(x^*,r)$, so that we have $|f(x(K\\alpha))-f(x_0)| \\leqslant L \\|x(K\\alpha)-x_0\\|$. We thus obtain the lower bound $\\|x(K\\alpha)-x_0\\| \\geqslant K\\alpha M^2/L = \\lfloor T/\\alpha \\rfloor \\alpha M^2/L \\geqslant (T-\\alpha)M^2/L \\geqslant (T-T/10)M^2/L = 9/10( 2\\epsilon + \\delta) $ (recall that $T = (2\\epsilon + \\delta)L/M^2$). Hence $\\|x(K\\alpha)-x^*\\| \\geqslant \\|x(K\\alpha)-x_0\\| - \\|x_0-x^*\\| \\geqslant 9/10(2\\epsilon + \\delta) - \\delta = 9 \\epsilon/5 - \\delta/10 \\geqslant 9 \\epsilon/5 - \\epsilon/10 = 17\\epsilon/10$ and finally $\\|x_K-x^*\\| \\geqslant \\|x(K\\alpha)-x^*\\| -\\|x_K-x(K\\alpha)\\| \\geqslant 17\\epsilon/10 - \\epsilon/2 = 6\\epsilon/5>\\epsilon$. However, since $0<\\alpha \\leqslant \\hat{\\alpha} \\leqslant \\bar{\\alpha}$ and $x_0 \\in B(x^*,\\delta)$, by stability of $x^*$ we have $x_K \\in B(x^*,\\epsilon)$. We have reached a contradiction. \n","start":18074,"end":19659}]},{"id":"S045","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S045"],"declared_external":true,"declaration_text":["原始前提清单：bolte2007clarke, Theorem 14；attouch2010proximal, Theorem 4.1。原文 L131 明确引用。"],"source_only":false,"title":"半代数 KL 不等式","source_units":[]},{"id":"S046","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S046"],"declared_external":true,"declaration_text":["原始前提清单：clarke1990, Proposition 2.1.5(d)。原文 L140 明确引用。"],"source_only":false,"title":"Clarke 映射上半连续","source_units":[]},{"id":"S047","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S047"],"declared_external":true,"declaration_text":["原始前提清单：clarke1990, Proposition 2.1.2(a)。原文 L140 明确引用。"],"source_only":false,"title":"Clarke 次微分值性质","source_units":[]},{"id":"S048","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S048"],"declared_external":true,"declaration_text":["原始前提清单：aubin1984differential, Proposition 3, p.42。原文 L108 明确引用。"],"source_only":false,"title":"紧域上的映射有界性","source_units":[]},{"id":"S049","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S049"],"declared_external":true,"declaration_text":["原始前提清单：Arzelà–Ascoli 与 Banach–Alaoglu，如 aubin1984differential, Theorem 4, p.13 中所用。原文 L108 明确引用。"],"source_only":false,"title":"函数与导数紧性","source_units":[]},{"id":"S050","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S050"],"declared_external":true,"declaration_text":["原始前提清单：aubin1984differential, Convergence Theorem, p.60。原文 L118 明确引用。"],"source_only":false,"title":"微分包含收敛定理","source_units":[]},{"id":"S051","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S051"],"declared_external":true,"declaration_text":["原始前提清单：davis2020stochastic, Lemma 5.2；drusvyatskiy2015curves。原文 L142 明确引用。"],"source_only":false,"title":"半代数轨道能量公式","source_units":[]},{"id":"S052","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S052"],"declared_external":true,"declaration_text":["原始前提清单中的标准有限维紧性与凸性事实；由标注者显式列出，原文未逐项命名引用。"],"source_only":false,"title":"有限维球与闭集事实","source_units":[]},{"id":"S053","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S053"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；属于原始前提清单的标准紧性背景，作者在 L120 自行展开其反证应用。"],"source_only":false,"title":"序列化统一逼近","source_units":[]},{"id":"S054","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S054"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；是原始前提清单中 KL 脱奇异函数条件的标准一元凹性背景。"],"source_only":false,"title":"凹函数导数比较","source_units":[]},{"id":"S055","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S055"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；属于原始前提清单的标准有限维及绝对连续性背景。"],"source_only":false,"title":"范数与取整估计","source_units":[]},{"id":"S056","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S056"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；属于原始前提清单的标准有限维紧性背景，原文 L131、L133 使用而未单独引用。"],"source_only":false,"title":"紧球上的 Lipschitz 常数","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S008","S016":"S008","S017":"S008","S018":"S008","S019":"S008","S020":"S008","S021":"S008","S022":"S008","S023":"S008","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056"}}}}
{"record_id":"2002.07321v4","paper":{"title":"Sampling Kaczmarz Motzkin Method for Linear Feasibility Problems: Generalization & Acceleration","authors":["Morshed, Md Sarowar","Islam, Md Saiful","Noor-E-Alam, Md."],"arxiv_id":"2002.07321","version":"v4","journal":"Mathematical Programming","abstract_url":"https://arxiv.org/abs/2002.07321v4","pdf_url":"https://arxiv.org/pdf/2002.07321v4","doi":"10.1007/s10107-021-01649-8"},"context":{"raw_tex":"We consider the following \\textit{Linear Feasibility} (LF) problem:\n\\begin{align}\n\\label{eq:1}\nAx \\leq b, \\ \\ b \\in \\R^m, \\ A \\in \\R^{m\\times n}.\n\\end{align}\n\n\n\\subsection{Notation}\n\nWe follow the standard linear algebra notation in this work. $\\mathbb{R}^n$ denotes the $n$ dimensional real space, $\\mathbb{R}^{m\\times n}$ denotes the set of $m \\times n$ real-valued matrices. For any matrix $A\\in \\mathbb{R}^{m\\times n}$, $A^T$ denotes the transpose matrix $A$ and $a_i^T$ for $i = 1,2,..,m$ denotes the rows of matrix $A$. Furthermore, $P = \\{x \\in \\R^n | \\ Ax \\leq b\\}$ denotes the feasible region of the feasibility problem and $\\mathcal{P}(x)$ denotes the projection of $x \\in \\R^n $ onto the feasible region $P$. The notation $d(x,P)$ denotes the distance between $x \\in \\R^n $ and the feasible region $P$, i.e., $d(x,P) \\ = \\ \\inf_{z \\in P} \\|x-z\\| \\ = \\ \\|x-\\mathcal{P}(x)\\|$. For any matrix $A$, the spectral norm and Frobenius norm are denoted by $\\|A\\|$  and $\\|A\\|_F$, respectively. For any function $f:X\\mapsto Y$, we use $\\nabla f$ to represent the gradient of $f$. Finally, $\\langle x,y\\rangle = x^Ty$ denotes the standard inner product and $\\|x\\| = \\sqrt{\\langle x, x \\rangle}$ as the euclidean ($L_2$) norm. The notation $x^+$ denotes the positive part of any real number (ie., $x^+ = \\max \\{x,0\\}$). For any two arbitrary matrices $M,\\ N$, the notation $M \\succ N$ implies the positive definiteness of the matrix $M-N$. The notation $\\E_\\mathbb{S}[\\cdot]$ is used to denote the expectation with respect to the sampling distribution $\\mathbb{S}$. %Throughout the paper, we will frequently use the  tower property of expectation.\n\n\n\\subsection{Assumptions}\nThroughout the paper, we assume that the system $Ax \\leq b$ is consistent and the matrix $A$ has no zero rows. We also assumed that the rows of matrix $A$ are normalized (i.e., $\\|a_i\\|^2=1$ for all $i$). Note that, normalization simplifies the convergence analysis considerably. The normalization doesn't impact the computational time significantly (we could simply normalize each row for the first time it occurs during the computation). Moreover, normalization simplifies the convergence analysis considerably. In the description of algorithms, we do not enforce the assumption. Furthermore, it can be noted that the proposed algorithms generate the same iterates irrespective of normalization. \n\n\n\\subsection{Expectation}\nFor the convergence analysis of Algorithm \\ref{alg:skm} and its variations (any algorithm that uses that specific type of sampling distribution), we need to discuss a specific expectation calculation. First of all, let us sort the residual vector $(Ax-b)^+$ from smallest to largest for any iterate $x$ and denote $(Ax-b)^+_{\\underline{\\mathbf{i_j}}}$ as the $(\\beta+j)^{th}$ entry on the sorted list \\footnote{We use the notation $(Ax-b)^+_{\\underline{\\mathbf{i_j}}}$ throughout the paper to express the underlying expectation, where the indices $\\underline{\\mathbf{i_j}}$ represent the sampling process of \\ref{eq:sampling}.}, i.e.,\n\\begin{align}\n\\label{eq:sampling}\n  \\underbrace{(Ax-b)^+_{\\underline{\\mathbf{i_0}}}}_{\\beta^{th}} \\ \\leq ... \\leq \\ \\underbrace{(Ax-b)^+_{\\underline{\\mathbf{i_j}}}}_{(\\beta+j)^{th}} \\ \\leq ... \\leq \\ \\underbrace{(Ax-b)^+_{\\underline{\\mathbf{i_{m-\\beta}}}}}_{m^{th}}.\n\\end{align}\n%Let's denote, the set $T(x)$\\footnote{$|T(x)| = |T(x_k)| = m-\\beta +1$. The notation $T(x_k)$ will be used to denote the corresponding set for any random iterate $x_k \\in \\R^n$.} as the collection of all indices from the list given in \\eqref{eq:sampling}.\nNow, consider the list with all of the entries of the residual vector $(Ax-b)^+$, then we need to calculate the probability that particular entry of the residual vector is selected at any given iteration. Note that, the probability that any sample is selected is $\\frac{1}{\\binom{m}{\\beta}}$ and each sample has an equal probability of selection. Another intersecting fact can be noted that the size of the residual list controls the order and frequency that each entry of the residual vector will be expected to be selected. From now on, we will denote this specific choice of sampling distribution as $\\mathbb{S}$ for any point $x \\in \\R^n$ \\footnote{For ease of notation, throughout the paper, we will use $\\mathbb{S}_k$ to denote the sampling distribution corresponding to any random iterate $x_k \\in \\R^n$ }. To calculate the resulting expectation with respect to the above-mentioned sampling distribution, let us first denote, $\\tau \\sim \\mathbb{S}$ as the set of sampled $\\beta$ constraints and $i^{*}$ as \\footnote{ Similarly, we will use $\\tau_k \\sim \\mathbb{S}_k$ to denote the sampled set and $i^* = \\argmax_{i \\in \\tau_k \\sim \\mathbb{S}_k} \\{a_i^Tx_k-b_i, 0\\} \\ = \\ \\argmax_{i \\in \\tau_k \\sim \\mathbb{S}_k} (A_{\\tau_k}x_k-b_{\\tau_k})^+_i$ for any iterate $x_k \\in \\R^n$.}\n\\begin{align}\n\\label{def:i1}\ni^* = \\argmax_{i \\in \\tau \\sim \\mathbb{S}} \\{a_i^Tx-b_i, 0\\} \\ = \\ \\argmax_{i} (A_{\\tau}x-b_{\\tau})^+_i,\n\\end{align}\nwhere, $A_{\\tau}$ denotes the collection of rows of $A$ restricted to the index set $\\tau$ and $(A_{\\tau}x-b_{\\tau})_i$ denotes the $i^{th}$ entry of $A_{\\tau}x-b_{\\tau}$. Using the above discussion with the list provided in equation \\eqref{eq:sampling}, we have the following:\n{\\allowdisplaybreaks\n\\begin{align}\n\\label{def:exp}\n\\E_{\\mathbb{S}} \\left[\\big |(a_{i^*}^Tx-b_{i^*})^{+}\\big |^2\\right] = \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} \\big | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}} \\big |^2, \\end{align}}\nwhere, $\\E_{\\mathbb{S}}$ denotes the required expectation corresponding to the sampling distribution $\\mathbb{S}$. The above expectation calculation was first used by De Loera \\textit{et.al} in their work \\cite{haddock:2017} where they first introduced the SKM method.\n\n\n\\subsection{Function $f(x)$}\nIn this section, we formalize the definition of function $f:\\R^n \\rightarrow \\R$. Throughout section \\ref{sec:conv}, we will use the properties of function $f(x)$ \\footnote{Similar type of functions with uniform sampling have been studied in \\cite{needell:2016} \\cite{razaviyayn:2019} in the context of stochastic gradient descent and alternating projection algorithms respectively.}. First, for any index $i$, let us define the following function\n\\begin{align}\n\\label{def:function_i}\n    f_i(x) = \\frac{1}{2} |(a_{i}^Tx-b_{i})^+|^2, \\quad \\nabla f_i(x) = (a_{i}^Tx-b_{i})^+ a_{i}.\n\\end{align}\nThen to simplify the expectation expression of \\eqref{def:exp} further, we define the function $f$ and the gradient of $f$ as follows:\n\\begin{align}\n\\label{def:function}\n    f(x) =  \\E_{\\mathbb{S}}\\left[f_{i^*}(x)\\right], \\quad \\nabla f(x) = \\E_{\\mathbb{S}} \\left[\\nabla f_{i^*}(x)\\right],\n\\end{align}\nwhere, the index $i^*$ is selected by the rule provided in \\eqref{def:i1}.\n\n\n\\begin{algorithm}\n\\caption{PASKM Algorithm: $x_{k+1} = \\textbf{PASKM}(A,b,x_0,K, \\delta, \\beta)$}\n\\label{alg:paskm}\n\\begin{algorithmic}\n\\STATE{Initialize $v_0 \\leftarrow x_0, \\  k \\leftarrow 0$;}\n\\WHILE{$k \\leq K$}\n\\STATE{Choose $\\gamma, \\omega,  \\alpha$ considering either \\eqref{cond} or \\eqref{eq:askm12} and update\n\\begin{align}\n\\label{eq:yk}\n  y_k = \\alpha v_k + (1-\\alpha)x_k;\n\\end{align}}\n\\STATE{Choose a sample of $\\beta$ constraints, $\\tau_k$, uniformly at random from the rows of matrix $A$. From these $\\beta$ constraints, choose $i^* = \\argmax_{i \\in \\tau_k} \\{a_i^Ty_k-b_i, 0\\}$; \\ Update \n\\begin{align}\nx_{k+1} & = y_k- \\delta \\frac{\\left(a_{i^*}^Ty_k -b_{i^*}\\right)^+}{\\|a_{i^*}\\|^2} a_{i^*}; \\label{eq:askm1} \\\\\nv_{k+1} & = \\omega v_k+ (1-\\omega)y_k-\\gamma \\frac{\\left(a_{i^*}^Ty_k -b_{i^*}\\right)^+}{\\|a_{i^*}\\|^2} a_{i^*}; \\label{eq:askm2}\n\\end{align}\n}\n\\STATE{$k \\leftarrow k+1$;}\n\\ENDWHILE\n\\RETURN $x$\n\\end{algorithmic}\n\\end{algorithm}\n\n\n\\begin{mdframed}[backgroundcolor=gray!15,   topline=false,   bottomline =false,   rightline=false,   leftline=false] \\begin{lemma}\n\\label{lem0}\n(Hoffman \\cite{hoffman}, Theorem 4.4 in \\cite{lewis:2010}) Let $x \\in \\R^n$ and $P$ be the feasible region, then there exists a constant $L > 0$ such that the following identity holds:\n\\begin{align*}\n  d(x,P)^2 \\leq L^2 \\ \\|(Ax-b)^+\\|^2.\n\\end{align*}\n\\end{lemma} \\end{mdframed}\n\n\n\\begin{mdframed}[backgroundcolor=gray!15,   topline=false,   bottomline =false,   rightline=false,   leftline=false] \\begin{lemma}\n\\label{lem:skmseq}\n(Lemma 2.1 in \\cite{haddock:2017})  Let $\\{x_k\\},\\ \\{y_k\\}$ be real non-negative sequences such that $x_{k+1} > x_k > 0$ and $y_{k+1} \\geq y_k \\geq 0$, then\n\\begin{align*}\n  \\sum\\limits_{k=1}^{n} x_k y_k \\ \\geq \\   \\sum\\limits_{k=1}^{n} \\overline{x} y_k, \\quad \\text{where} \\ \\ \\overline{x} = \\frac{1}{n}\\sum\\limits_{k=1}^{n} x_k.\n\\end{align*}\n\\end{lemma} \\end{mdframed}\n\n\n\\begin{mdframed}[backgroundcolor=gray!15,   topline=false,   bottomline =false,   rightline=false,   leftline=false] \\begin{lemma}\n\\label{lem:distance}\nFor any $x \\in \\R^n$ and $\\bar{x} \\in P$, the following identity holds,\n\\begin{align*}\n    d(x,P)^2 \\ = \\ \\|x-\\mathcal{P}(x) \\|^2 \\ \\leq \\ \\|x-\\bar{x} \\|^2.\n\\end{align*}\n\\end{lemma} \\end{mdframed}\n\n\n\\begin{align}\n\\label{eq:askm12}\n\\zeta =\\frac{3.99\\eta \\mu_1}{(1-\\mu_1)^2} , \\ \\omega = 1-\\frac{\\zeta \\mu_1^2+2\\gamma \\mu_1-\\zeta \\mu_1}{1+\\zeta \\mu_1^2} , \\ \\ \\gamma = \\sqrt{ \\zeta \\eta \\mu_1}, \\ \\ \\alpha = \\frac{ \\eta}{ \\eta + \\gamma },\n\\end{align}\n","units":[{"raw_tex":"We consider the following \\textit{Linear Feasibility} (LF) problem:\n\\begin{align}\n\\label{eq:1}\nAx \\leq b, \\ \\ b \\in \\R^m, \\ A \\in \\R^{m\\times n}.\n\\end{align}\n","start":4861,"end":5019},{"raw_tex":"\\subsection{Notation}\n\nWe follow the standard linear algebra notation in this work. $\\mathbb{R}^n$ denotes the $n$ dimensional real space, $\\mathbb{R}^{m\\times n}$ denotes the set of $m \\times n$ real-valued matrices. For any matrix $A\\in \\mathbb{R}^{m\\times n}$, $A^T$ denotes the transpose matrix $A$ and $a_i^T$ for $i = 1,2,..,m$ denotes the rows of matrix $A$. Furthermore, $P = \\{x \\in \\R^n | \\ Ax \\leq b\\}$ denotes the feasible region of the feasibility problem and $\\mathcal{P}(x)$ denotes the projection of $x \\in \\R^n $ onto the feasible region $P$. The notation $d(x,P)$ denotes the distance between $x \\in \\R^n $ and the feasible region $P$, i.e., $d(x,P) \\ = \\ \\inf_{z \\in P} \\|x-z\\| \\ = \\ \\|x-\\mathcal{P}(x)\\|$. For any matrix $A$, the spectral norm and Frobenius norm are denoted by $\\|A\\|$  and $\\|A\\|_F$, respectively. For any function $f:X\\mapsto Y$, we use $\\nabla f$ to represent the gradient of $f$. Finally, $\\langle x,y\\rangle = x^Ty$ denotes the standard inner product and $\\|x\\| = \\sqrt{\\langle x, x \\rangle}$ as the euclidean ($L_2$) norm. The notation $x^+$ denotes the positive part of any real number (ie., $x^+ = \\max \\{x,0\\}$). For any two arbitrary matrices $M,\\ N$, the notation $M \\succ N$ implies the positive definiteness of the matrix $M-N$. The notation $\\E_\\mathbb{S}[\\cdot]$ is used to denote the expectation with respect to the sampling distribution $\\mathbb{S}$. %Throughout the paper, we will frequently use the  tower property of expectation.\n","start":20813,"end":22300},{"raw_tex":"\\subsection{Assumptions}\nThroughout the paper, we assume that the system $Ax \\leq b$ is consistent and the matrix $A$ has no zero rows. We also assumed that the rows of matrix $A$ are normalized (i.e., $\\|a_i\\|^2=1$ for all $i$). Note that, normalization simplifies the convergence analysis considerably. The normalization doesn't impact the computational time significantly (we could simply normalize each row for the first time it occurs during the computation). Moreover, normalization simplifies the convergence analysis considerably. In the description of algorithms, we do not enforce the assumption. Furthermore, it can be noted that the proposed algorithms generate the same iterates irrespective of normalization. \n","start":22549,"end":23273},{"raw_tex":"\\subsection{Expectation}\nFor the convergence analysis of Algorithm \\ref{alg:skm} and its variations (any algorithm that uses that specific type of sampling distribution), we need to discuss a specific expectation calculation. First of all, let us sort the residual vector $(Ax-b)^+$ from smallest to largest for any iterate $x$ and denote $(Ax-b)^+_{\\underline{\\mathbf{i_j}}}$ as the $(\\beta+j)^{th}$ entry on the sorted list \\footnote{We use the notation $(Ax-b)^+_{\\underline{\\mathbf{i_j}}}$ throughout the paper to express the underlying expectation, where the indices $\\underline{\\mathbf{i_j}}$ represent the sampling process of \\ref{eq:sampling}.}, i.e.,\n\\begin{align}\n\\label{eq:sampling}\n  \\underbrace{(Ax-b)^+_{\\underline{\\mathbf{i_0}}}}_{\\beta^{th}} \\ \\leq ... \\leq \\ \\underbrace{(Ax-b)^+_{\\underline{\\mathbf{i_j}}}}_{(\\beta+j)^{th}} \\ \\leq ... \\leq \\ \\underbrace{(Ax-b)^+_{\\underline{\\mathbf{i_{m-\\beta}}}}}_{m^{th}}.\n\\end{align}\n%Let's denote, the set $T(x)$\\footnote{$|T(x)| = |T(x_k)| = m-\\beta +1$. The notation $T(x_k)$ will be used to denote the corresponding set for any random iterate $x_k \\in \\R^n$.} as the collection of all indices from the list given in \\eqref{eq:sampling}.\nNow, consider the list with all of the entries of the residual vector $(Ax-b)^+$, then we need to calculate the probability that particular entry of the residual vector is selected at any given iteration. Note that, the probability that any sample is selected is $\\frac{1}{\\binom{m}{\\beta}}$ and each sample has an equal probability of selection. Another intersecting fact can be noted that the size of the residual list controls the order and frequency that each entry of the residual vector will be expected to be selected. From now on, we will denote this specific choice of sampling distribution as $\\mathbb{S}$ for any point $x \\in \\R^n$ \\footnote{For ease of notation, throughout the paper, we will use $\\mathbb{S}_k$ to denote the sampling distribution corresponding to any random iterate $x_k \\in \\R^n$ }. To calculate the resulting expectation with respect to the above-mentioned sampling distribution, let us first denote, $\\tau \\sim \\mathbb{S}$ as the set of sampled $\\beta$ constraints and $i^{*}$ as \\footnote{ Similarly, we will use $\\tau_k \\sim \\mathbb{S}_k$ to denote the sampled set and $i^* = \\argmax_{i \\in \\tau_k \\sim \\mathbb{S}_k} \\{a_i^Tx_k-b_i, 0\\} \\ = \\ \\argmax_{i \\in \\tau_k \\sim \\mathbb{S}_k} (A_{\\tau_k}x_k-b_{\\tau_k})^+_i$ for any iterate $x_k \\in \\R^n$.}\n\\begin{align}\n\\label{def:i1}\ni^* = \\argmax_{i \\in \\tau \\sim \\mathbb{S}} \\{a_i^Tx-b_i, 0\\} \\ = \\ \\argmax_{i} (A_{\\tau}x-b_{\\tau})^+_i,\n\\end{align}\nwhere, $A_{\\tau}$ denotes the collection of rows of $A$ restricted to the index set $\\tau$ and $(A_{\\tau}x-b_{\\tau})_i$ denotes the $i^{th}$ entry of $A_{\\tau}x-b_{\\tau}$. Using the above discussion with the list provided in equation \\eqref{eq:sampling}, we have the following:\n{\\allowdisplaybreaks\n\\begin{align}\n\\label{def:exp}\n\\E_{\\mathbb{S}} \\left[\\big |(a_{i^*}^Tx-b_{i^*})^{+}\\big |^2\\right] = \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} \\big | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}} \\big |^2, \\end{align}}\nwhere, $\\E_{\\mathbb{S}}$ denotes the required expectation corresponding to the sampling distribution $\\mathbb{S}$. The above expectation calculation was first used by De Loera \\textit{et.al} in their work \\cite{haddock:2017} where they first introduced the SKM method.\n","start":25742,"end":29188},{"raw_tex":"\\subsection{Function $f(x)$}\nIn this section, we formalize the definition of function $f:\\R^n \\rightarrow \\R$. Throughout section \\ref{sec:conv}, we will use the properties of function $f(x)$ \\footnote{Similar type of functions with uniform sampling have been studied in \\cite{needell:2016} \\cite{razaviyayn:2019} in the context of stochastic gradient descent and alternating projection algorithms respectively.}. First, for any index $i$, let us define the following function\n\\begin{align}\n\\label{def:function_i}\n    f_i(x) = \\frac{1}{2} |(a_{i}^Tx-b_{i})^+|^2, \\quad \\nabla f_i(x) = (a_{i}^Tx-b_{i})^+ a_{i}.\n\\end{align}\nThen to simplify the expectation expression of \\eqref{def:exp} further, we define the function $f$ and the gradient of $f$ as follows:\n\\begin{align}\n\\label{def:function}\n    f(x) =  \\E_{\\mathbb{S}}\\left[f_{i^*}(x)\\right], \\quad \\nabla f(x) = \\E_{\\mathbb{S}} \\left[\\nabla f_{i^*}(x)\\right],\n\\end{align}\nwhere, the index $i^*$ is selected by the rule provided in \\eqref{def:i1}.\n","start":29445,"end":30445},{"raw_tex":"\\begin{algorithm}\n\\caption{PASKM Algorithm: $x_{k+1} = \\textbf{PASKM}(A,b,x_0,K, \\delta, \\beta)$}\n\\label{alg:paskm}\n\\begin{algorithmic}\n\\STATE{Initialize $v_0 \\leftarrow x_0, \\  k \\leftarrow 0$;}\n\\WHILE{$k \\leq K$}\n\\STATE{Choose $\\gamma, \\omega,  \\alpha$ considering either \\eqref{cond} or \\eqref{eq:askm12} and update\n\\begin{align}\n\\label{eq:yk}\n  y_k = \\alpha v_k + (1-\\alpha)x_k;\n\\end{align}}\n\\STATE{Choose a sample of $\\beta$ constraints, $\\tau_k$, uniformly at random from the rows of matrix $A$. From these $\\beta$ constraints, choose $i^* = \\argmax_{i \\in \\tau_k} \\{a_i^Ty_k-b_i, 0\\}$; \\ Update \n\\begin{align}\nx_{k+1} & = y_k- \\delta \\frac{\\left(a_{i^*}^Ty_k -b_{i^*}\\right)^+}{\\|a_{i^*}\\|^2} a_{i^*}; \\label{eq:askm1} \\\\\nv_{k+1} & = \\omega v_k+ (1-\\omega)y_k-\\gamma \\frac{\\left(a_{i^*}^Ty_k -b_{i^*}\\right)^+}{\\|a_{i^*}\\|^2} a_{i^*}; \\label{eq:askm2}\n\\end{align}\n}\n\\STATE{$k \\leftarrow k+1$;}\n\\ENDWHILE\n\\RETURN $x$\n\\end{algorithmic}\n\\end{algorithm}\n","start":36846,"end":37803},{"raw_tex":"\\begin{mdframed}[backgroundcolor=gray!15,   topline=false,   bottomline =false,   rightline=false,   leftline=false] \\begin{lemma}\n\\label{lem0}\n(Hoffman \\cite{hoffman}, Theorem 4.4 in \\cite{lewis:2010}) Let $x \\in \\R^n$ and $P$ be the feasible region, then there exists a constant $L > 0$ such that the following identity holds:\n\\begin{align*}\n  d(x,P)^2 \\leq L^2 \\ \\|(Ax-b)^+\\|^2.\n\\end{align*}\n\\end{lemma} \\end{mdframed}\n","start":45149,"end":45571},{"raw_tex":"\\begin{mdframed}[backgroundcolor=gray!15,   topline=false,   bottomline =false,   rightline=false,   leftline=false] \\begin{lemma}\n\\label{lem:skmseq}\n(Lemma 2.1 in \\cite{haddock:2017})  Let $\\{x_k\\},\\ \\{y_k\\}$ be real non-negative sequences such that $x_{k+1} > x_k > 0$ and $y_{k+1} \\geq y_k \\geq 0$, then\n\\begin{align*}\n  \\sum\\limits_{k=1}^{n} x_k y_k \\ \\geq \\   \\sum\\limits_{k=1}^{n} \\overline{x} y_k, \\quad \\text{where} \\ \\ \\overline{x} = \\frac{1}{n}\\sum\\limits_{k=1}^{n} x_k.\n\\end{align*}\n\\end{lemma} \\end{mdframed}\n","start":45880,"end":46401},{"raw_tex":"\\begin{mdframed}[backgroundcolor=gray!15,   topline=false,   bottomline =false,   rightline=false,   leftline=false] \\begin{lemma}\n\\label{lem:distance}\nFor any $x \\in \\R^n$ and $\\bar{x} \\in P$, the following identity holds,\n\\begin{align*}\n    d(x,P)^2 \\ = \\ \\|x-\\mathcal{P}(x) \\|^2 \\ \\leq \\ \\|x-\\bar{x} \\|^2.\n\\end{align*}\n\\end{lemma} \\end{mdframed}\n","start":46948,"end":47297},{"raw_tex":"\\begin{align}\n\\label{eq:askm12}\n\\zeta =\\frac{3.99\\eta \\mu_1}{(1-\\mu_1)^2} , \\ \\omega = 1-\\frac{\\zeta \\mu_1^2+2\\gamma \\mu_1-\\zeta \\mu_1}{1+\\zeta \\mu_1^2} , \\ \\ \\gamma = \\sqrt{ \\zeta \\eta \\mu_1}, \\ \\ \\alpha = \\frac{ \\eta}{ \\eta + \\gamma },\n\\end{align}\n","start":84654,"end":84904}]},"statement":{"raw_tex":"\\begin{theorem}\n\\label{th:10}\nLet, $v_{k+1}$ and $x_{k+1}$ are generated by Algorithm \\ref{alg:paskm}. If we select the parameters $\\omega, \\ \\gamma, \\ \\alpha$ as\n\\begin{align*}\n\\omega = 1-\\frac{\\zeta \\mu_1^2+2\\gamma \\mu_1-\\zeta \\mu_1}{1+\\zeta \\mu_1^2} , \\ \\ \\gamma = \\sqrt{ \\zeta \\eta \\mu_1}, \\ \\ \\alpha = \\frac{ \\eta}{ \\eta + \\gamma },\n\\end{align*}\nwhere, $\\zeta$ is chosen as $ 0 < \\zeta  < \\frac{4\\eta \\mu_1}{(1-\\mu_1)^2} $ if $\\mu_1 < 1$, otherwise choose any $\\zeta > 0$. Then, for any $0 < \\delta < 2$, the sequence of iterates $\\{v_k\\}, \\ \\{x_k\\}$ converges and the following result holds:\n\\begin{align*}\n\\E \\left[ d(v_{k+1},P)^2 + \\zeta \\mu_1 \\  d(x_{k+1},P)^2 \\right] \\ & \\leq \\ \\omega^{k+1} \\ \\E \\left[ d(v_{0},P)^2+ \\zeta \\mu_1 \\  d(x_{0},P)^2 \\right] \\\\\n& = (1+\\zeta \\mu_1) \\ \\omega^{k+1} \\ d(x_{0},P)^2.\n\\end{align*}  \nThis theorem implies that the PASKM algorithm converges linearly with a rate of $\\omega$, which accumulates to a total of $\\mathcal{O}(\\frac{1+\\zeta \\mu_1^2}{\\zeta \\mu_1^2+2\\gamma \\mu_1-\\zeta \\mu_1}\\log{1/\\epsilon})$ iterations to bring the given error below $\\epsilon > 0$.\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\n\\label{th:10}\nLet, $v_{k+1}$ and $x_{k+1}$ are generated by Algorithm \\ref{alg:paskm}. If we select the parameters $\\omega, \\ \\gamma, \\ \\alpha$ as\n\\begin{align*}\n\\omega = 1-\\frac{\\zeta \\mu_1^2+2\\gamma \\mu_1-\\zeta \\mu_1}{1+\\zeta \\mu_1^2} , \\ \\ \\gamma = \\sqrt{ \\zeta \\eta \\mu_1}, \\ \\ \\alpha = \\frac{ \\eta}{ \\eta + \\gamma },\n\\end{align*}\nwhere, $\\zeta$ is chosen as $ 0 < \\zeta  < \\frac{4\\eta \\mu_1}{(1-\\mu_1)^2} $ if $\\mu_1 < 1$, otherwise choose any $\\zeta > 0$. Then, for any $0 < \\delta < 2$, the sequence of iterates $\\{v_k\\}, \\ \\{x_k\\}$ converges and the following result holds:\n\\begin{align*}\n\\E \\left[ d(v_{k+1},P)^2 + \\zeta \\mu_1 \\  d(x_{k+1},P)^2 \\right] \\ & \\leq \\ \\omega^{k+1} \\ \\E \\left[ d(v_{0},P)^2+ \\zeta \\mu_1 \\  d(x_{0},P)^2 \\right] \\\\\n& = (1+\\zeta \\mu_1) \\ \\omega^{k+1} \\ d(x_{0},P)^2.\n\\end{align*}  \nThis theorem implies that the PASKM algorithm converges linearly with a rate of $\\omega$, which accumulates to a total of $\\mathcal{O}(\\frac{1+\\zeta \\mu_1^2}{\\zeta \\mu_1^2+2\\gamma \\mu_1-\\zeta \\mu_1}\\log{1/\\epsilon})$ iterations to bring the given error below $\\epsilon > 0$.\n\\end{theorem}","start":78323,"end":79444}]},"proof":{"raw_tex":"\\begin{lemma}\n\\label{lem1}\nLet $\\lambda_j$ be the $j^{th}$ eigenvalue of the matrix $W = \\E_{\\mathbb{S}}\\left[a_{i^*}a_{i^*}^T\\right]$, then for all $j$, the bound $0 \\leq \\lambda_j \\leq 1$ holds.\n\\end{lemma}\n\n\\begin{proof}\nSince $W$ is positive semi-definite, we can write $\\lambda_j \\geq 0$ for all $j$. Also as the mapping $\\lambda_{\\max}(X)$ is convex, using Jensen's inequality we have,\n\\begin{align*}\n    \\lambda_{\\max} (W) = \\lambda_{\\max} \\left[ \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] \\right] \\leq \\E_{\\mathbb{S}} \\left[\\lambda_{\\max}\\left(a_{i^*}a_{i^*}^T\\right)\\right] \\leq 1.\n\\end{align*}\n\\end{proof}\n\n\\begin{lemma}\n\\label{lem2}\nFor any $1 \\leq \\beta \\leq m$, we have the following:\n\\begin{align*}\n \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] \\preceq \\frac{\\beta}{m} A^TA.\n\\end{align*}\n\\end{lemma}\n\n\\begin{proof}\nSee Appendix 1.\n\\end{proof}\n\n\\begin{lemma}\n\\label{lem3}\nFor any $x \\in \\R^n$ with $\\lambda_{\\max} = \\lambda_{\\max}(A^TA)$, we have the following:\n\\begin{align*}\n    \\frac{\\mu_1}{ 2} \\ d(x,P)^2 \\ \\leq \\ f(x) \\ \\leq \\  \\frac{\\mu_2}{2}\\ d(x,P)^2,\n\\end{align*}\nwith $ 0 < \\mu_1 = \\frac{1}{m L^2} \\leq \\ \\mu_2 = \\min \\left\\{1, \\frac{\\beta}{m} \\lambda_{\\max}\\right\\} \\leq 1$. \n\\end{lemma}\n\n\\begin{proof}\nSee Appendix 1.\n\\end{proof}\n\n\\begin{lemma}\n\\label{lem:grad}\nFor any $x, y \\in \\R^n$, we have the following:\n\\begin{align*}\n \\langle  x-y, \\E_{\\mathbb{S}}  \\left[(a_{i^*}^Ty-b_{i^*})^{+} a_{i^*}\\right]  \\rangle  & =  \\langle  x-y, \\nabla f(y)   \\rangle   \\\\\n & \\leq    f(x) - f(y) \\leq     \\frac{\\mu_2}{2} \\ d(x,P)^2 -  \\frac{\\mu_1}{2} \\ d(y,P)^2.\n\\end{align*}\n\\end{lemma}\n\n\\begin{proof}\nSee Appendix 1.\n\\end{proof}\n\n\\begin{lemma}\n\\label{lem:grad1}\nFor any $y \\in \\R^n$ and $\\bar{y}$ such that $A \\bar{y} \\leq b$, we have the following:\n\\begin{align*}\n\\langle  \\bar{y}-y, \\E_{\\mathbb{S}} \\left[a_{i^*}(a_{i^*}^Ty-b_{i^*})^{+}\\right]  \\rangle = \\ \\langle  \\bar{y}-y, \\nabla f(y)  \\rangle    \\leq  -2f(y)  \\leq \\ - \\mu_1  d(y,P)^2.\n\\end{align*}\n\\end{lemma}\n\n\\begin{proof}\nSee Appendix 1.\n\\end{proof}\n\n\\begin{lemma}\n\\label{lem4}\nFor any $x \\in \\R^n$ and $0 < \\delta < 2$, we have the following:\n\\begin{align*}\n  \\E_{\\mathbb{S}} \\left[ d(z,P)^2\\right] = \\E_{\\mathbb{S}} \\left[ \\Big \\| x- \\mathcal{P} (x) - \\delta \\left(a_{i^*}^Tx-b_{i^*}\\right)^+ a_{i^*}\\Big \\|^2 \\right]  \\leq   h(\\delta) \\ d(x,P)^2,\n\\end{align*}\nwhere, $z = x - \\delta \\left(a_{i^*}^Tx-b_{i^*}\\right)^+ a_{i^*}$, $ \\eta = 2 \\delta -\\delta^2$ and $h(\\delta) = 1- \\eta \\mu_1 < 1$.\n\\end{lemma}\n\n\\begin{proof}\nSee Appendix 1.\n\\end{proof}\n\n\\begin{proof}\nSee Appendix 3.\n\\end{proof}\n\n\\paragraph{Proof of Lemma \\ref{lem2}}\nUsing the expectation expression given in the first section we have,\n\\begin{align*}\n     \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] & = \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} (A^TA)_{\\underline{\\mathbf{i_j}}} \\\\\n     &  \\preceq \\ \\frac{\\binom{m-1}{\\beta-1}}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta}  (A^TA)_{\\underline{\\mathbf{i_j}}} \\ \\preceq \\ \\frac{\\beta}{m} \\sum\\limits_{i = 1}^{m}  a_{i} a_{i}^T  \\ = \\ \\frac{\\beta}{m} A^TA.\n\\end{align*}\nHere, the notation $(A^TA)_{\\underline{\\mathbf{i_j}}}$ denotes the matrix $a_l a_l^T$ where the index $l$ belongs to the list \\eqref{eq:sampling}. Furthermore, the index $l$ corresponds to the $(\\beta+ j)^{th}$ entry on the list \\eqref{eq:sampling}. This proves the Lemma.\n%$(Ax-b)^{+}_{\\fbox{${i_j}$}}$\n%$(Ax-b)^{+}_{\\fbox{\\strut ${i_j}$}}$\n%$(Ax-b)^{+}_{\\fbox{$\\mathstrut$ ${i_j}$}}$\n\n%$(Ax-b)^{+}_{\\underline{\\mathbf{i_j}}}$\n\n% $\\overline{\\mathbf{x}}$  $\\underline{\\mathbf{x}}$ \n\n\n\\paragraph{Proof of Lemma \\ref{lem3}}\nUsing the definition of the expectation from \\eqref{def:exp}, we have,\n\\allowdisplaybreaks{\n\\begin{align*}\n \\E_{\\mathbb{S}} \\left[\\big | (a_{i^*}^Tx-b_{i^*})^+\\big |^2\\right] & \\overset{\\eqref{def:exp}}{=} \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1}  | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}} |^2 \\\\\n    & \\overset{\\text{Lemma} \\ \\ref{lem:skmseq}}{\\geq} \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta}  \\frac{\\sum\\limits_{l = 0}^{m-\\beta} \\binom{\\beta-1+l}{\\beta-1}}{m-\\beta +1} | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}} |^2 \\\\\n    & \\geq \\frac{1}{m-\\beta+1} \\sum\\limits_{j = 0}^{m-\\beta}  \\big |(Ax - b)^+_{\\underline{\\mathbf{i_j}}} \\big |^2  \\\\\n    & \\geq \\frac{1}{m-\\beta+1} \\min\\{\\frac{m-\\beta+1}{m-s}, 1\\} \\ \\|(Ax - b)^+\\|^2 \\\\\n    &  \\overset{\\text{Lemma} \\ \\ref{lem0}}{\\geq} \\ \\frac{1}{m L^2}  \\ d(x,P)^2. \n\\end{align*}}\nHere, $s$ is the number of zero entries in the residual $(Ax - b)^+$, which also corresponds to the number of satisfied constraints for $x$. Since $A \\mathcal{P} (x) \\leq b$, we have the following:\n\\begin{align*}\n    \\E_{\\mathbb{S}} \\left[\\big | (a_{i^*}^Tx-b_{i^*})^+\\big |^2\\right]   & \\overset{\\eqref{def:exp}}{=}  \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}}|^2 \\\\\n    & \\leq  \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} \\ \\big | (Ax-A\\mathcal{P} (x))_{\\underline{\\mathbf{i_j}}}   \\big|^2 \\\\\n    & =  \\frac{1}{\\binom{m}{\\beta}} (x-\\mathcal{P} (x))^T\\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} \\ (A^TA)_{\\underline{\\mathbf{i_j}}} (x-\\mathcal{P} (x) ) \\\\\n    & =  (x-\\mathcal{P} (x) )^T \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] (x-\\mathcal{P} (x) ) \\\\\n    & \\overset{\\text{Lemma} \\ \\ref{lem1} \\ \\& \\ \\ref{lem2} }{\\leq} \\  \\min \\left\\{1, \\frac{\\beta}{m} \\lambda_{\\max}\\right\\} \\big \\| x- \\mathcal{P} (x) \\big \\|^2 \\\\\n    & = \\  \\min \\left\\{1, \\frac{\\beta}{m} \\lambda_{\\max}\\right\\} d(x,P)^2.\n\\end{align*}\nCombining the above identities and using the expression for $f(x)$ from \\eqref{def:function} we get,\n\\begin{align*}\n    \\frac{\\mu_1}{ 2} \\ d(x,P)^2 \\ \\leq \\ f(x) \\ \\leq \\  \\frac{\\mu_2}{2} \\ d(x,P)^2,\n\\end{align*}\nwhich proves the Lemma. \n\n\\paragraph{Proof of Lemma \\ref{lem:grad}}\n\nFrom the definition of $f(x)$, it can be easily checked that $f(x)$ is a convex function. Now, by the convexity property of $f(x)$, for any $x, y \\in \\R^n$, we have the following:\n\\begin{align}\n\\label{eq:grad}\n    \\langle x-y, \\nabla f(y) \\rangle \\leq f(x) - f(y).\n\\end{align}\nTherefore, we have\n\\begin{align*}\n\\big \\langle  x-y, \\E_{\\mathbb{S}}  \\left[(a_{i^*}^Ty-b_{i^*})^{+} a_{i^*}\\right] \\big \\rangle & =  \\langle x-y, \\nabla f(y) \\rangle  \\leq  \\ f(x) - f(y) \\\\\n&  \\overset{\\text{Lemma} \\  \\ref{lem3} }{\\leq}  \\    \\frac{\\mu_2}{2} \\ d(x,P)^2 -  \\frac{\\mu_1}{2} \\ d(y,P)^2.\n\\end{align*}\nThis completes the proof.\n\n\n\\paragraph{Proof of Lemma \\ref{lem:grad1}}\n\n\\iffalse\n\nLet's denote, $ \\mathcal{J} = \\{i_j \\ | \\ c_{i_j}^Ty-d_{i_j} \\geq 0, \\ i_j \\in T(y)\\}$. Then following the construction of \\eqref{eq:f}, the function $f(y)$ can be expressed as $f(y) = \\|C_{\\mathcal{J}} y-d_{\\mathcal{J}} \\|^2$.\nSince, $C_{\\mathcal{J}} y-d_{\\mathcal{J}} \\geq 0$ and $C_{\\mathcal{J}}\\bar{y}-d_{\\mathcal{J}} \\leq 0$, we have,\n\\begin{align*}\n    \\big \\langle C_{\\mathcal{J}}y-d_{\\mathcal{J}} , C_{\\mathcal{J}}\\bar{y}-d_{\\mathcal{J}} \\big \\rangle \\ \\leq \\ 0.\n\\end{align*}\nAdding $\\|C_{\\mathcal{J}}y-d_{\\mathcal{J}}\\|^2$ to both sides and rearranging terms we get,\n\\begin{align}\n\\label{eq:grad1}\n \\|C_{\\mathcal{J}}y-d_{\\mathcal{J}} \\|^2 \\ \\leq \\  \\big \\langle C_{\\mathcal{J}}^T \\left( C_{\\mathcal{J}}y-d_{\\mathcal{J}} \\right), y - \\bar{y} \\big \\rangle.\n\\end{align}\n\n\\fi\n\nSince, $\\bar{y} \\in P$, from the definition we have,\n\\begin{align*}\n   \\langle  \\bar{y}-y, \\E_{\\mathbb{S}}  \\left[a_{i^*}(a_{i^*}^Ty-b_{i^*})^{+}\\right]  \\rangle \\ & = \\E_{\\mathbb{S}} \\left[(a_{i^*}^Ty-b_{i^*})^{+} \\left(a_{i^*}^T\\bar{y}- a_{i^*}^T y\\right)\\right] \\\\\n   & \\leq \\E_{\\mathbb{S}} \\left[(a_{i^*}^Ty-b_{i^*})^{+} \\left(b_{i^*}- a_{i^*}^T y\\right)\\right] \\\\\n   & = - \\E_{\\mathbb{S}} \\left[\\big | (a_{i^*}^Ty-b_{i^*})^{+} \\big |^2\\right] \\\\\n   &= - 2f(y) \\overset{\\text{Lemma} \\ \\ref{lem3} }{\\leq} \\ \\ -\\mu_1 \\ d(y,P)^2.\n\\end{align*}\nHere, we used the identity $xx^+ = |x^+|^2$. This proves the Lemma.\n\n\\paragraph{Proof of Lemma \\ref{lem4}}\n\nSince, $\\mathcal{P}(x) \\in P$, we have\n{\\allowdisplaybreaks\n\\begin{align*}\n    \\E_{\\mathbb{S}}  \\left[ d(z,P)^2\\right] &  \\overset{\\text{Lemma} \\ \\ref{lem:distance}}{\\leq} \\  \\E_{\\mathbb{S}}  \\left[ \\|z- \\mathcal{P}(x)\\|^2\\right] = \\E_{\\mathbb{S}} \\left[ \\big \\| x- \\mathcal{P} (x) - \\delta \\left(a_{i^*}^Tx-b_{i^*}\\right)^+ a_{i^*} \\big \\|^2 \\right] \\\\\n    & \\overset{\\eqref{def:function}}{=} \\ \\|x - \\mathcal{P} (x) \\|^2 + 2\\delta^2 f(x)  + 2 \\delta \\ \\big  \\langle  \\mathcal{P}(x)-x, \\nabla f(x) \\big \\rangle \\\\\n    & \\overset{\\text{Lemma} \\ \\ref{lem3} }{\\leq} \\ \\|x - \\mathcal{P} (x) \\|^2 -2(2\\delta-\\delta^2) f(x) \\\\\n    & \\leq \\ \\|x-  \\mathcal{P} (x) \\|^2 - (2\\delta-\\delta^2) \\ \\mu_1  \\|x-  \\mathcal{P} (x) \\|^2 = h(\\delta) \\ d(x,P)^2.\n\\end{align*}}\nHere, we used the lower bound of the expected value from Lemma \\ref{lem3}.\n\n\n\n\\paragraph{Proof of Theorem \\ref{th:10}}\n\nLet us define, $\\mathcal{V} = \\omega \\mathcal{P}(v_{k}) +(1-\\omega) \\mathcal{P}(y_{k}) $. Since $\\mathcal{V} \\in P$, using the update formula of $v_{k+1}$ from equation \\eqref{eq:askm2}, we have,\n\\allowdisplaybreaks{\n\\begin{align}\n\\label{eq:askm4}\n    d(v_{k+1},P)^2 & = \\|v_{k+1} - \\mathcal{P}(v_{k+1})\\|^2 \\overset{\\text{Lemma} \\ \\ref{lem:distance}}{\\leq}  \\|v_{k+1} - \\mathcal{V}\\|^2  \\nonumber \\\\\n    & \\overset{\\eqref{eq:askm2}}{=}  \\big \\|\\omega v_k + (1-\\omega) y_k- \\mathcal{V} -\\gamma (a_{i^*}^Ty_k- b_{i^*})^+ a_{i^*} \\big \\|^2 \\nonumber \\\\\n    & =   \\underbrace{ \\|\\omega v_k + (1-\\omega) y_k- \\mathcal{V} \\|^2}_{I_1} + \\gamma^2 \\underbrace{\\|(a_{i^*}^Ty_k- b_{i^*})^+a_{i^*}\\|^2}_{I_2} \\nonumber \\\\\n    & - 2 \\gamma  \\underbrace{\\big \\langle \\omega v_k + (1-\\omega) y_k- \\mathcal{V} \\ ,\\  a_{i^*} (a_{i^*}^Ty_k-b_{i^{*}})^{+} \\big \\rangle }_{I_3} \\nonumber \\\\\n    & = I_1 + \\gamma^2 I_2 -2 \\gamma I_3.\n\\end{align}}\nSince $\\|\\cdot\\|^{2}$ is a convex function and $0 < \\omega < 1$, we can bound the expected first term as follows,\n\\begin{align}\n\\label{eq:askm5}\n    \\E_{\\mathbb{S}_k} [I_1]  &= \\E_{\\mathbb{S}_k} \\left[\\|\\omega v_k + (1-\\omega) y_k-\\mathcal{V}\\|^2\\right] \\nonumber \\\\\n    & = \\E_{\\mathbb{S}_k} \\left[\\|\\omega v_k + (1-\\omega) y_k- \\omega \\mathcal{P}(v_{k}) - (1-\\omega) \\mathcal{P}(y_{k})\\|^2\\right] \\nonumber \\\\\n    & \\leq \\omega \\| v_k-\\mathcal{P}(v_{k})\\|^2 + (1-\\omega) \\|y_k-\\mathcal{P}(y_{k})\\|^2\\nonumber \\\\\n    & =  \\omega \\ d(v_k,P)^2 + (1-\\omega) \\ d(y_k,P)^2.\n\\end{align}\nTaking expectation with respect to the sampling distribution in the second term of equation \\eqref{eq:askm4} and using Lemma \\ref{lem4} with the choice $z = x_{k+1}, \\ x = y_k$ and $\\eta = 2\\delta-\\delta^2$, we get,\n\\begin{align}\n\\label{eq:askm6}\n   \\gamma^2 \\E_{\\mathbb{S}_k} \\left[\\|(a_{i^*}^Ty_k- b_{i^*})^+ a_{i^*}\\|^2\\right] & \\ = \\  \\gamma^2 \\E_{\\mathbb{S}_k} \\left[|(a_{i^*}^Ty_k- b_{i^*})^+ |^2\\right] \\nonumber \\\\\n    \\overset{\\text{Lemma} \\ \\ref{lem4}}{\\leq} & \\ \\frac{\\gamma^2 }{\\eta} \\left[d(y_k,P)^2 - \\E \\left[ d(x_{k+1},P)^2 \\right]\\right].\n\\end{align}\n\nNow, taking expectation in the third term of \\eqref{eq:askm4} we get,\n\\begin{align}\n\\label{eq:askm9}\n    - 2  \\gamma  \\E_{\\mathbb{S}_k} & [I_3 ]  = - 2 \\gamma \\big \\langle \\omega v_k + (1-\\omega) y_k-\\mathcal{V}, \\E_{\\mathbb{S}_k} \\left[ a_{i^*} (a_{i^*}^Ty_k-b_{i^*})^{+}\\right]  \\big \\rangle \\nonumber \\\\\n    & \\overset{\\eqref{eq:yk} \\ \\& \\ \\eqref{def:function} }{=} - 2 \\gamma \\big \\langle \\frac{\\omega}{\\alpha} \\left[y_k - (1-\\alpha) x_k\\right] + (1-\\omega) y_k-\\mathcal{V},  \\nabla f(y_k)  \\big \\rangle \\nonumber \\\\\n    & = - 2 \\gamma \\big \\langle \\frac{\\omega(1-\\alpha)}{\\alpha}(y_k-x_k)+ y_k-\\mathcal{V}, \\nabla f(y_k)  \\big \\rangle.\n\\end{align}\nUsing Lemma \\ref{lem:grad} and Lemma \\ref{lem:grad1} we can simplify equation \\eqref{eq:askm9} as follows,\n\\begin{align}\n\\label{eq:askm10}\n  & \\ -  2 \\gamma  \\E_{\\mathbb{S}_k}  [I_3 ]  =   - 2 \\gamma \\big \\langle \\frac{\\omega(1-\\alpha)}{\\alpha}(y_k-x_k)+ y_k-\\mathcal{V}, \\nabla f(y_k)  \\big \\rangle \\nonumber \\\\\n    & \\quad =    2 \\gamma \\frac{\\omega(1-\\alpha)}{\\alpha} \\big \\langle x_k-y_k, \\nabla f(y_k)  \\big \\rangle + 2 \\gamma \\big \\langle \\mathcal{V}-y_k, \\nabla f(y_k)  \\big \\rangle \\nonumber \\\\\n  & \\overset{\\text{Lemma} \\ \\ref{lem:grad} \\ \\& \\ \\ref{lem:grad1}}{\\leq} \\frac{\\gamma \\omega (1-\\alpha)}{\\alpha} \\  d(x_k,P)^2 - \\frac{ \\mu_1 \\gamma  \\omega (1-\\alpha)}{\\alpha} \\ d(y_k,P)^2 - 2 \\mu_1 \\gamma \\ d(y_k,P)^2.\n\\end{align}\nNow, substituting the values of equation \\eqref{eq:askm5}, \\eqref{eq:askm6} \\& \\eqref{eq:askm10} in equation \\eqref{eq:askm4} we get the following:\n\\allowdisplaybreaks{\n\\begin{align*}\n \\E [ & d(v_{k+1} ,P)^2  ]  =  I_1 + \\gamma^2 \\E_{\\mathbb{S}_k}[I_2 ]- 2 \\gamma \\E_{\\mathbb{S}_k}[I_3] \\nonumber \\\\\n & = \\omega \\ d(v_{k},P)^2 + (1-\\omega) \\ d(y_k,P)^2 + \\frac{\\gamma^2 }{\\eta} \\left\\{d(y_k,P)^2 - \\E \\left[ d(x_{k+1},P)^2 \\right]\\right\\} \\nonumber \\\\\n & \\quad + \\frac{\\gamma \\omega (1-\\alpha)}{\\alpha}  d(x_k,P)^2 - \\gamma \\mu_1 \\left(2+\\frac{ \\omega (1-\\alpha)}{\\alpha}\\right)  d(y_k,P)^2.\n\\end{align*}}\nWith further simplification, the above identity can be written as follows:\n\\allowdisplaybreaks{\n\\begin{align}\n\\label{eq:askm11}\n & \\E \\left[ d(v_{k+1},P)^2 + \\frac{\\gamma^2}{\\eta} d(x_{k+1},P)^2 \\right]  = \\omega \\ \\left[d(v_{k},P)^2 + \\frac{\\gamma (1-\\alpha)}{\\alpha} d(x_{k},P)^2 \\right] \\nonumber \\\\\n & \\quad \\quad \\quad \\quad \\quad \\quad \\quad + d(y_k,P)^2 \\ \\left\\{1-\\omega+ \\frac{\\gamma^2 }{\\eta}  - 2\\gamma \\mu_1 - \\frac{\\gamma \\omega \\mu_1 (1-\\alpha)}{\\alpha}  \\right\\}.\n\\end{align}}\nNow, let's choose the parameters as in equation \\eqref{eq:askm12} along with $0 < \\zeta  < \\frac{4\\eta \\mu_1}{(1-\\mu_1)^2} $. We can easily see that $\\frac{\\gamma^2 }{\\eta} = \\frac{ \\gamma(1-\\alpha)}{\\alpha}$ and $ \\alpha \\in (0,1)$. Also note that,\n\\allowdisplaybreaks{\\begin{align}\n 2 \\mu_1 \\gamma \\ = \\ 2 \\mu_1 \\sqrt{\\eta \\zeta \\mu_1} \\ > \\ 2 \\mu_1 \\sqrt{\\frac{\\zeta (1-\\mu_1)^2 \\zeta}{4}} = \\mu_1 \\zeta(1-\\mu_1),\n\\end{align}}\nwhich implies $\\omega < 1$. Similarly, whenever $\\mu_1 < 1$ we have\n\\begin{align}\n    2\\gamma - \\zeta - \\frac{1}{\\mu_1} < 2\\sqrt{\\eta \\mu_1 \\zeta} - \\zeta - 1 \\leq  2 \\sqrt{\\zeta} - \\zeta - 1  = -(\\sqrt{\\zeta}-1)^2 \\leq 0,\n\\end{align}\nwhich implies $\\omega > 0$. Also, using the parameter choice of \\eqref{eq:askm12}, we have, \n\\begin{align}\n\\label{eq:askm13}\n  1-\\omega+ \\frac{\\gamma^2 }{\\eta}  - 2\\gamma \\mu_1 - & \\frac{\\gamma \\omega \\mu_1 (1-\\alpha)}{\\alpha}   = 1-\\omega + \\zeta \\mu_1 - 2\\gamma \\mu_1 - \\omega \\zeta \\mu_1^2 \\nonumber \\\\\n  & = 1- 2\\gamma \\mu_1 + \\zeta \\mu_1 - \\omega (1+\\zeta \\mu_1^2) = 0.\n\\end{align}\nNow, using all of the above relations (equation \\eqref{eq:askm12}, \\eqref{eq:askm13}) in equation \\eqref{eq:askm11}, we get the following:\n\\begin{align}\n\\label{eq:askm14}\n  \\E \\left[ d(v_{k+1},P)^2 + \\frac{\\gamma^2}{\\eta} d(x_{k+1},P)^2 \\right] &  \\ \\leq \\ \\omega \\Big[d(v_{k},P)^2 + \\underbrace{\\frac{\\gamma (1-\\alpha)}{\\alpha}}_{= \\frac{\\gamma^2 }{\\eta}} \\ d(x_{k},P)^2 \\Big] \\nonumber \\\\\n  + & \\Big [\\underbrace{1-\\omega+ \\frac{\\gamma^2 }{\\eta}  - 2\\gamma \\mu_1 - \\frac{\\gamma \\omega \\mu_1 (1-\\alpha)}{\\alpha} }_{= \\ 0} \\Big ] \\ d(y_k,P)^2 \\nonumber \\\\\n   = \\ & \\omega \\ \\left[ d(v_{k},P)^2+ \\frac{\\gamma^2}{\\eta} d(x_{k},P)^2 \\right].\n\\end{align}\nFinally, taking expectation again with tower rule and substituting $\\frac{\\gamma^2 }{\\eta}= \\zeta \\mu_1 $ we have,\n\\begin{align*}\n\\E \\left[ d(v_{k+1},P)^2 + \\zeta \\mu_1 \\  d(x_{k+1},P)^2 \\right] \\ & \\leq \\ \\omega^{k+1} \\ \\E \\left[ d(v_{0},P)^2+ \\zeta \\mu_1 \\  d(x_{0},P)^2 \\right] \\\\\n& = (1+\\zeta \\mu_1) \\ \\omega^{k+1} \\ d(x_{0},P)^2.\n\\end{align*}\nThis proves the Theorem. Furthermore, for faster convergence, we need to choose parameters such that, $\\omega$ becomes as small as possible. In the proof, we assumed $\\mu_1 <1$ holds which is the most probable scenario. Whenever $\\mu_1 = 1$, we must have $\\mu_1 = \\mu_2 =1$ and Lemma 6 holds with both equality, i.e., $f(x) = d(x,P)^2$. Therefore if we choose $\\alpha, \\gamma, \\omega$ as $\\alpha = \\frac{\\eta}{\\eta+ \\gamma}, \\ \\gamma = \\sqrt{\\zeta \\eta}, \\ \\omega = 1- \\frac{2\\gamma}{1+\\zeta}$, we can check that condition (89) holds and $0 < \\omega < 1$ holds for any $\\zeta > 0$.\n\n\n\n\n\n\n\n","units":[{"raw_tex":"\\begin{lemma}\n\\label{lem1}\nLet $\\lambda_j$ be the $j^{th}$ eigenvalue of the matrix $W = \\E_{\\mathbb{S}}\\left[a_{i^*}a_{i^*}^T\\right]$, then for all $j$, the bound $0 \\leq \\lambda_j \\leq 1$ holds.\n\\end{lemma}","start":47416,"end":47624},{"raw_tex":"\\begin{proof}\nSince $W$ is positive semi-definite, we can write $\\lambda_j \\geq 0$ for all $j$. Also as the mapping $\\lambda_{\\max}(X)$ is convex, using Jensen's inequality we have,\n\\begin{align*}\n    \\lambda_{\\max} (W) = \\lambda_{\\max} \\left[ \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] \\right] \\leq \\E_{\\mathbb{S}} \\left[\\lambda_{\\max}\\left(a_{i^*}a_{i^*}^T\\right)\\right] \\leq 1.\n\\end{align*}\n\\end{proof}","start":47641,"end":48049},{"raw_tex":"\\begin{lemma}\n\\label{lem2}\nFor any $1 \\leq \\beta \\leq m$, we have the following:\n\\begin{align*}\n \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] \\preceq \\frac{\\beta}{m} A^TA.\n\\end{align*}\n\\end{lemma}","start":48168,"end":48365},{"raw_tex":"\\begin{proof}\nSee Appendix 1.\n\\end{proof}","start":48382,"end":48423},{"raw_tex":"\\begin{lemma}\n\\label{lem3}\nFor any $x \\in \\R^n$ with $\\lambda_{\\max} = \\lambda_{\\max}(A^TA)$, we have the following:\n\\begin{align*}\n    \\frac{\\mu_1}{ 2} \\ d(x,P)^2 \\ \\leq \\ f(x) \\ \\leq \\  \\frac{\\mu_2}{2}\\ d(x,P)^2,\n\\end{align*}\nwith $ 0 < \\mu_1 = \\frac{1}{m L^2} \\leq \\ \\mu_2 = \\min \\left\\{1, \\frac{\\beta}{m} \\lambda_{\\max}\\right\\} \\leq 1$. \n\\end{lemma}","start":48545,"end":48898},{"raw_tex":"\\begin{proof}\nSee Appendix 1.\n\\end{proof}","start":48915,"end":48956},{"raw_tex":"\\begin{lemma}\n\\label{lem:grad}\nFor any $x, y \\in \\R^n$, we have the following:\n\\begin{align*}\n \\langle  x-y, \\E_{\\mathbb{S}}  \\left[(a_{i^*}^Ty-b_{i^*})^{+} a_{i^*}\\right]  \\rangle  & =  \\langle  x-y, \\nabla f(y)   \\rangle   \\\\\n & \\leq    f(x) - f(y) \\leq     \\frac{\\mu_2}{2} \\ d(x,P)^2 -  \\frac{\\mu_1}{2} \\ d(y,P)^2.\n\\end{align*}\n\\end{lemma}","start":50379,"end":50721},{"raw_tex":"\\begin{proof}\nSee Appendix 1.\n\\end{proof}","start":50738,"end":50779},{"raw_tex":"\\begin{lemma}\n\\label{lem:grad1}\nFor any $y \\in \\R^n$ and $\\bar{y}$ such that $A \\bar{y} \\leq b$, we have the following:\n\\begin{align*}\n\\langle  \\bar{y}-y, \\E_{\\mathbb{S}} \\left[a_{i^*}(a_{i^*}^Ty-b_{i^*})^{+}\\right]  \\rangle = \\ \\langle  \\bar{y}-y, \\nabla f(y)  \\rangle    \\leq  -2f(y)  \\leq \\ - \\mu_1  d(y,P)^2.\n\\end{align*}\n\\end{lemma}","start":52283,"end":52620},{"raw_tex":"\\begin{proof}\nSee Appendix 1.\n\\end{proof}","start":52637,"end":52678},{"raw_tex":"\\begin{lemma}\n\\label{lem4}\nFor any $x \\in \\R^n$ and $0 < \\delta < 2$, we have the following:\n\\begin{align*}\n  \\E_{\\mathbb{S}} \\left[ d(z,P)^2\\right] = \\E_{\\mathbb{S}} \\left[ \\Big \\| x- \\mathcal{P} (x) - \\delta \\left(a_{i^*}^Tx-b_{i^*}\\right)^+ a_{i^*}\\Big \\|^2 \\right]  \\leq   h(\\delta) \\ d(x,P)^2,\n\\end{align*}\nwhere, $z = x - \\delta \\left(a_{i^*}^Tx-b_{i^*}\\right)^+ a_{i^*}$, $ \\eta = 2 \\delta -\\delta^2$ and $h(\\delta) = 1- \\eta \\mu_1 < 1$.\n\\end{lemma}","start":53802,"end":54258},{"raw_tex":"\\begin{proof}\nSee Appendix 1.\n\\end{proof}","start":54275,"end":54316},{"raw_tex":"\\begin{proof}\nSee Appendix 3.\n\\end{proof}","start":79461,"end":79502},{"raw_tex":"\\paragraph{Proof of Lemma \\ref{lem2}}\nUsing the expectation expression given in the first section we have,\n\\begin{align*}\n     \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] & = \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} (A^TA)_{\\underline{\\mathbf{i_j}}} \\\\\n     &  \\preceq \\ \\frac{\\binom{m-1}{\\beta-1}}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta}  (A^TA)_{\\underline{\\mathbf{i_j}}} \\ \\preceq \\ \\frac{\\beta}{m} \\sum\\limits_{i = 1}^{m}  a_{i} a_{i}^T  \\ = \\ \\frac{\\beta}{m} A^TA.\n\\end{align*}\nHere, the notation $(A^TA)_{\\underline{\\mathbf{i_j}}}$ denotes the matrix $a_l a_l^T$ where the index $l$ belongs to the list \\eqref{eq:sampling}. Furthermore, the index $l$ corresponds to the $(\\beta+ j)^{th}$ entry on the list \\eqref{eq:sampling}. This proves the Lemma.\n%$(Ax-b)^{+}_{\\fbox{${i_j}$}}$\n%$(Ax-b)^{+}_{\\fbox{\\strut ${i_j}$}}$\n%$(Ax-b)^{+}_{\\fbox{$\\mathstrut$ ${i_j}$}}$\n\n%$(Ax-b)^{+}_{\\underline{\\mathbf{i_j}}}$\n\n% $\\overline{\\mathbf{x}}$  $\\underline{\\mathbf{x}}$ \n\n\n\\paragraph{Proof of Lemma \\ref{lem3}}\nUsing the definition of the expectation from \\eqref{def:exp}, we have,\n\\allowdisplaybreaks{\n\\begin{align*}\n \\E_{\\mathbb{S}} \\left[\\big | (a_{i^*}^Tx-b_{i^*})^+\\big |^2\\right] & \\overset{\\eqref{def:exp}}{=} \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1}  | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}} |^2 \\\\\n    & \\overset{\\text{Lemma} \\ \\ref{lem:skmseq}}{\\geq} \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta}  \\frac{\\sum\\limits_{l = 0}^{m-\\beta} \\binom{\\beta-1+l}{\\beta-1}}{m-\\beta +1} | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}} |^2 \\\\\n    & \\geq \\frac{1}{m-\\beta+1} \\sum\\limits_{j = 0}^{m-\\beta}  \\big |(Ax - b)^+_{\\underline{\\mathbf{i_j}}} \\big |^2  \\\\\n    & \\geq \\frac{1}{m-\\beta+1} \\min\\{\\frac{m-\\beta+1}{m-s}, 1\\} \\ \\|(Ax - b)^+\\|^2 \\\\\n    &  \\overset{\\text{Lemma} \\ \\ref{lem0}}{\\geq} \\ \\frac{1}{m L^2}  \\ d(x,P)^2. \n\\end{align*}}\nHere, $s$ is the number of zero entries in the residual $(Ax - b)^+$, which also corresponds to the number of satisfied constraints for $x$. Since $A \\mathcal{P} (x) \\leq b$, we have the following:\n\\begin{align*}\n    \\E_{\\mathbb{S}} \\left[\\big | (a_{i^*}^Tx-b_{i^*})^+\\big |^2\\right]   & \\overset{\\eqref{def:exp}}{=}  \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}}|^2 \\\\\n    & \\leq  \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} \\ \\big | (Ax-A\\mathcal{P} (x))_{\\underline{\\mathbf{i_j}}}   \\big|^2 \\\\\n    & =  \\frac{1}{\\binom{m}{\\beta}} (x-\\mathcal{P} (x))^T\\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} \\ (A^TA)_{\\underline{\\mathbf{i_j}}} (x-\\mathcal{P} (x) ) \\\\\n    & =  (x-\\mathcal{P} (x) )^T \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] (x-\\mathcal{P} (x) ) \\\\\n    & \\overset{\\text{Lemma} \\ \\ref{lem1} \\ \\& \\ \\ref{lem2} }{\\leq} \\  \\min \\left\\{1, \\frac{\\beta}{m} \\lambda_{\\max}\\right\\} \\big \\| x- \\mathcal{P} (x) \\big \\|^2 \\\\\n    & = \\  \\min \\left\\{1, \\frac{\\beta}{m} \\lambda_{\\max}\\right\\} d(x,P)^2.\n\\end{align*}\nCombining the above identities and using the expression for $f(x)$ from \\eqref{def:function} we get,\n\\begin{align*}\n    \\frac{\\mu_1}{ 2} \\ d(x,P)^2 \\ \\leq \\ f(x) \\ \\leq \\  \\frac{\\mu_2}{2} \\ d(x,P)^2,\n\\end{align*}\nwhich proves the Lemma. \n\n\\paragraph{Proof of Lemma \\ref{lem:grad}}\n\nFrom the definition of $f(x)$, it can be easily checked that $f(x)$ is a convex function. Now, by the convexity property of $f(x)$, for any $x, y \\in \\R^n$, we have the following:\n\\begin{align}\n\\label{eq:grad}\n    \\langle x-y, \\nabla f(y) \\rangle \\leq f(x) - f(y).\n\\end{align}\nTherefore, we have\n\\begin{align*}\n\\big \\langle  x-y, \\E_{\\mathbb{S}}  \\left[(a_{i^*}^Ty-b_{i^*})^{+} a_{i^*}\\right] \\big \\rangle & =  \\langle x-y, \\nabla f(y) \\rangle  \\leq  \\ f(x) - f(y) \\\\\n&  \\overset{\\text{Lemma} \\  \\ref{lem3} }{\\leq}  \\    \\frac{\\mu_2}{2} \\ d(x,P)^2 -  \\frac{\\mu_1}{2} \\ d(y,P)^2.\n\\end{align*}\nThis completes the proof.\n\n\n\\paragraph{Proof of Lemma \\ref{lem:grad1}}\n\n\\iffalse\n\nLet's denote, $ \\mathcal{J} = \\{i_j \\ | \\ c_{i_j}^Ty-d_{i_j} \\geq 0, \\ i_j \\in T(y)\\}$. Then following the construction of \\eqref{eq:f}, the function $f(y)$ can be expressed as $f(y) = \\|C_{\\mathcal{J}} y-d_{\\mathcal{J}} \\|^2$.\nSince, $C_{\\mathcal{J}} y-d_{\\mathcal{J}} \\geq 0$ and $C_{\\mathcal{J}}\\bar{y}-d_{\\mathcal{J}} \\leq 0$, we have,\n\\begin{align*}\n    \\big \\langle C_{\\mathcal{J}}y-d_{\\mathcal{J}} , C_{\\mathcal{J}}\\bar{y}-d_{\\mathcal{J}} \\big \\rangle \\ \\leq \\ 0.\n\\end{align*}\nAdding $\\|C_{\\mathcal{J}}y-d_{\\mathcal{J}}\\|^2$ to both sides and rearranging terms we get,\n\\begin{align}\n\\label{eq:grad1}\n \\|C_{\\mathcal{J}}y-d_{\\mathcal{J}} \\|^2 \\ \\leq \\  \\big \\langle C_{\\mathcal{J}}^T \\left( C_{\\mathcal{J}}y-d_{\\mathcal{J}} \\right), y - \\bar{y} \\big \\rangle.\n\\end{align}\n\n\\fi\n\nSince, $\\bar{y} \\in P$, from the definition we have,\n\\begin{align*}\n   \\langle  \\bar{y}-y, \\E_{\\mathbb{S}}  \\left[a_{i^*}(a_{i^*}^Ty-b_{i^*})^{+}\\right]  \\rangle \\ & = \\E_{\\mathbb{S}} \\left[(a_{i^*}^Ty-b_{i^*})^{+} \\left(a_{i^*}^T\\bar{y}- a_{i^*}^T y\\right)\\right] \\\\\n   & \\leq \\E_{\\mathbb{S}} \\left[(a_{i^*}^Ty-b_{i^*})^{+} \\left(b_{i^*}- a_{i^*}^T y\\right)\\right] \\\\\n   & = - \\E_{\\mathbb{S}} \\left[\\big | (a_{i^*}^Ty-b_{i^*})^{+} \\big |^2\\right] \\\\\n   &= - 2f(y) \\overset{\\text{Lemma} \\ \\ref{lem3} }{\\leq} \\ \\ -\\mu_1 \\ d(y,P)^2.\n\\end{align*}\nHere, we used the identity $xx^+ = |x^+|^2$. This proves the Lemma.\n\n\\paragraph{Proof of Lemma \\ref{lem4}}\n\nSince, $\\mathcal{P}(x) \\in P$, we have\n{\\allowdisplaybreaks\n\\begin{align*}\n    \\E_{\\mathbb{S}}  \\left[ d(z,P)^2\\right] &  \\overset{\\text{Lemma} \\ \\ref{lem:distance}}{\\leq} \\  \\E_{\\mathbb{S}}  \\left[ \\|z- \\mathcal{P}(x)\\|^2\\right] = \\E_{\\mathbb{S}} \\left[ \\big \\| x- \\mathcal{P} (x) - \\delta \\left(a_{i^*}^Tx-b_{i^*}\\right)^+ a_{i^*} \\big \\|^2 \\right] \\\\\n    & \\overset{\\eqref{def:function}}{=} \\ \\|x - \\mathcal{P} (x) \\|^2 + 2\\delta^2 f(x)  + 2 \\delta \\ \\big  \\langle  \\mathcal{P}(x)-x, \\nabla f(x) \\big \\rangle \\\\\n    & \\overset{\\text{Lemma} \\ \\ref{lem3} }{\\leq} \\ \\|x - \\mathcal{P} (x) \\|^2 -2(2\\delta-\\delta^2) f(x) \\\\\n    & \\leq \\ \\|x-  \\mathcal{P} (x) \\|^2 - (2\\delta-\\delta^2) \\ \\mu_1  \\|x-  \\mathcal{P} (x) \\|^2 = h(\\delta) \\ d(x,P)^2.\n\\end{align*}}\nHere, we used the lower bound of the expected value from Lemma \\ref{lem3}.\n\n","start":115634,"end":121931},{"raw_tex":"\\paragraph{Proof of Theorem \\ref{th:10}}\n\nLet us define, $\\mathcal{V} = \\omega \\mathcal{P}(v_{k}) +(1-\\omega) \\mathcal{P}(y_{k}) $. Since $\\mathcal{V} \\in P$, using the update formula of $v_{k+1}$ from equation \\eqref{eq:askm2}, we have,\n\\allowdisplaybreaks{\n\\begin{align}\n\\label{eq:askm4}\n    d(v_{k+1},P)^2 & = \\|v_{k+1} - \\mathcal{P}(v_{k+1})\\|^2 \\overset{\\text{Lemma} \\ \\ref{lem:distance}}{\\leq}  \\|v_{k+1} - \\mathcal{V}\\|^2  \\nonumber \\\\\n    & \\overset{\\eqref{eq:askm2}}{=}  \\big \\|\\omega v_k + (1-\\omega) y_k- \\mathcal{V} -\\gamma (a_{i^*}^Ty_k- b_{i^*})^+ a_{i^*} \\big \\|^2 \\nonumber \\\\\n    & =   \\underbrace{ \\|\\omega v_k + (1-\\omega) y_k- \\mathcal{V} \\|^2}_{I_1} + \\gamma^2 \\underbrace{\\|(a_{i^*}^Ty_k- b_{i^*})^+a_{i^*}\\|^2}_{I_2} \\nonumber \\\\\n    & - 2 \\gamma  \\underbrace{\\big \\langle \\omega v_k + (1-\\omega) y_k- \\mathcal{V} \\ ,\\  a_{i^*} (a_{i^*}^Ty_k-b_{i^{*}})^{+} \\big \\rangle }_{I_3} \\nonumber \\\\\n    & = I_1 + \\gamma^2 I_2 -2 \\gamma I_3.\n\\end{align}}\nSince $\\|\\cdot\\|^{2}$ is a convex function and $0 < \\omega < 1$, we can bound the expected first term as follows,\n\\begin{align}\n\\label{eq:askm5}\n    \\E_{\\mathbb{S}_k} [I_1]  &= \\E_{\\mathbb{S}_k} \\left[\\|\\omega v_k + (1-\\omega) y_k-\\mathcal{V}\\|^2\\right] \\nonumber \\\\\n    & = \\E_{\\mathbb{S}_k} \\left[\\|\\omega v_k + (1-\\omega) y_k- \\omega \\mathcal{P}(v_{k}) - (1-\\omega) \\mathcal{P}(y_{k})\\|^2\\right] \\nonumber \\\\\n    & \\leq \\omega \\| v_k-\\mathcal{P}(v_{k})\\|^2 + (1-\\omega) \\|y_k-\\mathcal{P}(y_{k})\\|^2\\nonumber \\\\\n    & =  \\omega \\ d(v_k,P)^2 + (1-\\omega) \\ d(y_k,P)^2.\n\\end{align}\nTaking expectation with respect to the sampling distribution in the second term of equation \\eqref{eq:askm4} and using Lemma \\ref{lem4} with the choice $z = x_{k+1}, \\ x = y_k$ and $\\eta = 2\\delta-\\delta^2$, we get,\n\\begin{align}\n\\label{eq:askm6}\n   \\gamma^2 \\E_{\\mathbb{S}_k} \\left[\\|(a_{i^*}^Ty_k- b_{i^*})^+ a_{i^*}\\|^2\\right] & \\ = \\  \\gamma^2 \\E_{\\mathbb{S}_k} \\left[|(a_{i^*}^Ty_k- b_{i^*})^+ |^2\\right] \\nonumber \\\\\n    \\overset{\\text{Lemma} \\ \\ref{lem4}}{\\leq} & \\ \\frac{\\gamma^2 }{\\eta} \\left[d(y_k,P)^2 - \\E \\left[ d(x_{k+1},P)^2 \\right]\\right].\n\\end{align}\n\nNow, taking expectation in the third term of \\eqref{eq:askm4} we get,\n\\begin{align}\n\\label{eq:askm9}\n    - 2  \\gamma  \\E_{\\mathbb{S}_k} & [I_3 ]  = - 2 \\gamma \\big \\langle \\omega v_k + (1-\\omega) y_k-\\mathcal{V}, \\E_{\\mathbb{S}_k} \\left[ a_{i^*} (a_{i^*}^Ty_k-b_{i^*})^{+}\\right]  \\big \\rangle \\nonumber \\\\\n    & \\overset{\\eqref{eq:yk} \\ \\& \\ \\eqref{def:function} }{=} - 2 \\gamma \\big \\langle \\frac{\\omega}{\\alpha} \\left[y_k - (1-\\alpha) x_k\\right] + (1-\\omega) y_k-\\mathcal{V},  \\nabla f(y_k)  \\big \\rangle \\nonumber \\\\\n    & = - 2 \\gamma \\big \\langle \\frac{\\omega(1-\\alpha)}{\\alpha}(y_k-x_k)+ y_k-\\mathcal{V}, \\nabla f(y_k)  \\big \\rangle.\n\\end{align}\nUsing Lemma \\ref{lem:grad} and Lemma \\ref{lem:grad1} we can simplify equation \\eqref{eq:askm9} as follows,\n\\begin{align}\n\\label{eq:askm10}\n  & \\ -  2 \\gamma  \\E_{\\mathbb{S}_k}  [I_3 ]  =   - 2 \\gamma \\big \\langle \\frac{\\omega(1-\\alpha)}{\\alpha}(y_k-x_k)+ y_k-\\mathcal{V}, \\nabla f(y_k)  \\big \\rangle \\nonumber \\\\\n    & \\quad =    2 \\gamma \\frac{\\omega(1-\\alpha)}{\\alpha} \\big \\langle x_k-y_k, \\nabla f(y_k)  \\big \\rangle + 2 \\gamma \\big \\langle \\mathcal{V}-y_k, \\nabla f(y_k)  \\big \\rangle \\nonumber \\\\\n  & \\overset{\\text{Lemma} \\ \\ref{lem:grad} \\ \\& \\ \\ref{lem:grad1}}{\\leq} \\frac{\\gamma \\omega (1-\\alpha)}{\\alpha} \\  d(x_k,P)^2 - \\frac{ \\mu_1 \\gamma  \\omega (1-\\alpha)}{\\alpha} \\ d(y_k,P)^2 - 2 \\mu_1 \\gamma \\ d(y_k,P)^2.\n\\end{align}\nNow, substituting the values of equation \\eqref{eq:askm5}, \\eqref{eq:askm6} \\& \\eqref{eq:askm10} in equation \\eqref{eq:askm4} we get the following:\n\\allowdisplaybreaks{\n\\begin{align*}\n \\E [ & d(v_{k+1} ,P)^2  ]  =  I_1 + \\gamma^2 \\E_{\\mathbb{S}_k}[I_2 ]- 2 \\gamma \\E_{\\mathbb{S}_k}[I_3] \\nonumber \\\\\n & = \\omega \\ d(v_{k},P)^2 + (1-\\omega) \\ d(y_k,P)^2 + \\frac{\\gamma^2 }{\\eta} \\left\\{d(y_k,P)^2 - \\E \\left[ d(x_{k+1},P)^2 \\right]\\right\\} \\nonumber \\\\\n & \\quad + \\frac{\\gamma \\omega (1-\\alpha)}{\\alpha}  d(x_k,P)^2 - \\gamma \\mu_1 \\left(2+\\frac{ \\omega (1-\\alpha)}{\\alpha}\\right)  d(y_k,P)^2.\n\\end{align*}}\nWith further simplification, the above identity can be written as follows:\n\\allowdisplaybreaks{\n\\begin{align}\n\\label{eq:askm11}\n & \\E \\left[ d(v_{k+1},P)^2 + \\frac{\\gamma^2}{\\eta} d(x_{k+1},P)^2 \\right]  = \\omega \\ \\left[d(v_{k},P)^2 + \\frac{\\gamma (1-\\alpha)}{\\alpha} d(x_{k},P)^2 \\right] \\nonumber \\\\\n & \\quad \\quad \\quad \\quad \\quad \\quad \\quad + d(y_k,P)^2 \\ \\left\\{1-\\omega+ \\frac{\\gamma^2 }{\\eta}  - 2\\gamma \\mu_1 - \\frac{\\gamma \\omega \\mu_1 (1-\\alpha)}{\\alpha}  \\right\\}.\n\\end{align}}\nNow, let's choose the parameters as in equation \\eqref{eq:askm12} along with $0 < \\zeta  < \\frac{4\\eta \\mu_1}{(1-\\mu_1)^2} $. We can easily see that $\\frac{\\gamma^2 }{\\eta} = \\frac{ \\gamma(1-\\alpha)}{\\alpha}$ and $ \\alpha \\in (0,1)$. Also note that,\n\\allowdisplaybreaks{\\begin{align}\n 2 \\mu_1 \\gamma \\ = \\ 2 \\mu_1 \\sqrt{\\eta \\zeta \\mu_1} \\ > \\ 2 \\mu_1 \\sqrt{\\frac{\\zeta (1-\\mu_1)^2 \\zeta}{4}} = \\mu_1 \\zeta(1-\\mu_1),\n\\end{align}}\nwhich implies $\\omega < 1$. Similarly, whenever $\\mu_1 < 1$ we have\n\\begin{align}\n    2\\gamma - \\zeta - \\frac{1}{\\mu_1} < 2\\sqrt{\\eta \\mu_1 \\zeta} - \\zeta - 1 \\leq  2 \\sqrt{\\zeta} - \\zeta - 1  = -(\\sqrt{\\zeta}-1)^2 \\leq 0,\n\\end{align}\nwhich implies $\\omega > 0$. Also, using the parameter choice of \\eqref{eq:askm12}, we have, \n\\begin{align}\n\\label{eq:askm13}\n  1-\\omega+ \\frac{\\gamma^2 }{\\eta}  - 2\\gamma \\mu_1 - & \\frac{\\gamma \\omega \\mu_1 (1-\\alpha)}{\\alpha}   = 1-\\omega + \\zeta \\mu_1 - 2\\gamma \\mu_1 - \\omega \\zeta \\mu_1^2 \\nonumber \\\\\n  & = 1- 2\\gamma \\mu_1 + \\zeta \\mu_1 - \\omega (1+\\zeta \\mu_1^2) = 0.\n\\end{align}\nNow, using all of the above relations (equation \\eqref{eq:askm12}, \\eqref{eq:askm13}) in equation \\eqref{eq:askm11}, we get the following:\n\\begin{align}\n\\label{eq:askm14}\n  \\E \\left[ d(v_{k+1},P)^2 + \\frac{\\gamma^2}{\\eta} d(x_{k+1},P)^2 \\right] &  \\ \\leq \\ \\omega \\Big[d(v_{k},P)^2 + \\underbrace{\\frac{\\gamma (1-\\alpha)}{\\alpha}}_{= \\frac{\\gamma^2 }{\\eta}} \\ d(x_{k},P)^2 \\Big] \\nonumber \\\\\n  + & \\Big [\\underbrace{1-\\omega+ \\frac{\\gamma^2 }{\\eta}  - 2\\gamma \\mu_1 - \\frac{\\gamma \\omega \\mu_1 (1-\\alpha)}{\\alpha} }_{= \\ 0} \\Big ] \\ d(y_k,P)^2 \\nonumber \\\\\n   = \\ & \\omega \\ \\left[ d(v_{k},P)^2+ \\frac{\\gamma^2}{\\eta} d(x_{k},P)^2 \\right].\n\\end{align}\nFinally, taking expectation again with tower rule and substituting $\\frac{\\gamma^2 }{\\eta}= \\zeta \\mu_1 $ we have,\n\\begin{align*}\n\\E \\left[ d(v_{k+1},P)^2 + \\zeta \\mu_1 \\  d(x_{k+1},P)^2 \\right] \\ & \\leq \\ \\omega^{k+1} \\ \\E \\left[ d(v_{0},P)^2+ \\zeta \\mu_1 \\  d(x_{0},P)^2 \\right] \\\\\n& = (1+\\zeta \\mu_1) \\ \\omega^{k+1} \\ d(x_{0},P)^2.\n\\end{align*}\nThis proves the Theorem. Furthermore, for faster convergence, we need to choose parameters such that, $\\omega$ becomes as small as possible. In the proof, we assumed $\\mu_1 <1$ holds which is the most probable scenario. Whenever $\\mu_1 = 1$, we must have $\\mu_1 = \\mu_2 =1$ and Lemma 6 holds with both equality, i.e., $f(x) = d(x,P)^2$. Therefore if we choose $\\alpha, \\gamma, \\omega$ as $\\alpha = \\frac{\\eta}{\\eta+ \\gamma}, \\ \\gamma = \\sqrt{\\zeta \\eta}, \\ \\omega = 1- \\frac{2\\gamma}{1+\\zeta}$, we can check that condition (89) holds and $0 < \\omega < 1$ holds for any $\\zeta > 0$.\n\n\n\n\n\n\n\n","start":156521,"end":163768}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:1"],"declared_external":false,"source_only":false,"title":"线性可行性问题","source_units":[{"raw_tex":"We consider the following \\textit{Linear Feasibility} (LF) problem:\n\\begin{align}\n\\label{eq:1}\nAx \\leq b, \\ \\ b \\in \\R^m, \\ A \\in \\R^{m\\times n}.\n\\end{align}\n","start":4861,"end":5019}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"可行域与距离记号","source_units":[{"raw_tex":"\\subsection{Notation}\n\nWe follow the standard linear algebra notation in this work. $\\mathbb{R}^n$ denotes the $n$ dimensional real space, $\\mathbb{R}^{m\\times n}$ denotes the set of $m \\times n$ real-valued matrices. For any matrix $A\\in \\mathbb{R}^{m\\times n}$, $A^T$ denotes the transpose matrix $A$ and $a_i^T$ for $i = 1,2,..,m$ denotes the rows of matrix $A$. Furthermore, $P = \\{x \\in \\R^n | \\ Ax \\leq b\\}$ denotes the feasible region of the feasibility problem and $\\mathcal{P}(x)$ denotes the projection of $x \\in \\R^n $ onto the feasible region $P$. The notation $d(x,P)$ denotes the distance between $x \\in \\R^n $ and the feasible region $P$, i.e., $d(x,P) \\ = \\ \\inf_{z \\in P} \\|x-z\\| \\ = \\ \\|x-\\mathcal{P}(x)\\|$. For any matrix $A$, the spectral norm and Frobenius norm are denoted by $\\|A\\|$  and $\\|A\\|_F$, respectively. For any function $f:X\\mapsto Y$, we use $\\nabla f$ to represent the gradient of $f$. Finally, $\\langle x,y\\rangle = x^Ty$ denotes the standard inner product and $\\|x\\| = \\sqrt{\\langle x, x \\rangle}$ as the euclidean ($L_2$) norm. The notation $x^+$ denotes the positive part of any real number (ie., $x^+ = \\max \\{x,0\\}$). For any two arbitrary matrices $M,\\ N$, the notation $M \\succ N$ implies the positive definiteness of the matrix $M-N$. The notation $\\E_\\mathbb{S}[\\cdot]$ is used to denote the expectation with respect to the sampling distribution $\\mathbb{S}$. %Throughout the paper, we will frequently use the  tower property of expectation.\n","start":20813,"end":22300}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"一致性与行归一化","source_units":[{"raw_tex":"\\subsection{Assumptions}\nThroughout the paper, we assume that the system $Ax \\leq b$ is consistent and the matrix $A$ has no zero rows. We also assumed that the rows of matrix $A$ are normalized (i.e., $\\|a_i\\|^2=1$ for all $i$). Note that, normalization simplifies the convergence analysis considerably. The normalization doesn't impact the computational time significantly (we could simply normalize each row for the first time it occurs during the computation). Moreover, normalization simplifies the convergence analysis considerably. In the description of algorithms, we do not enforce the assumption. Furthermore, it can be noted that the proposed algorithms generate the same iterates irrespective of normalization. \n","start":22549,"end":23273}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["def:i1","eq:sampling"],"declared_external":false,"source_only":false,"title":"采样与残差排序","source_units":[{"raw_tex":"\\subsection{Expectation}\nFor the convergence analysis of Algorithm \\ref{alg:skm} and its variations (any algorithm that uses that specific type of sampling distribution), we need to discuss a specific expectation calculation. First of all, let us sort the residual vector $(Ax-b)^+$ from smallest to largest for any iterate $x$ and denote $(Ax-b)^+_{\\underline{\\mathbf{i_j}}}$ as the $(\\beta+j)^{th}$ entry on the sorted list \\footnote{We use the notation $(Ax-b)^+_{\\underline{\\mathbf{i_j}}}$ throughout the paper to express the underlying expectation, where the indices $\\underline{\\mathbf{i_j}}$ represent the sampling process of \\ref{eq:sampling}.}, i.e.,\n\\begin{align}\n\\label{eq:sampling}\n  \\underbrace{(Ax-b)^+_{\\underline{\\mathbf{i_0}}}}_{\\beta^{th}} \\ \\leq ... \\leq \\ \\underbrace{(Ax-b)^+_{\\underline{\\mathbf{i_j}}}}_{(\\beta+j)^{th}} \\ \\leq ... \\leq \\ \\underbrace{(Ax-b)^+_{\\underline{\\mathbf{i_{m-\\beta}}}}}_{m^{th}}.\n\\end{align}\n%Let's denote, the set $T(x)$\\footnote{$|T(x)| = |T(x_k)| = m-\\beta +1$. The notation $T(x_k)$ will be used to denote the corresponding set for any random iterate $x_k \\in \\R^n$.} as the collection of all indices from the list given in \\eqref{eq:sampling}.\nNow, consider the list with all of the entries of the residual vector $(Ax-b)^+$, then we need to calculate the probability that particular entry of the residual vector is selected at any given iteration. Note that, the probability that any sample is selected is $\\frac{1}{\\binom{m}{\\beta}}$ and each sample has an equal probability of selection. Another intersecting fact can be noted that the size of the residual list controls the order and frequency that each entry of the residual vector will be expected to be selected. From now on, we will denote this specific choice of sampling distribution as $\\mathbb{S}$ for any point $x \\in \\R^n$ \\footnote{For ease of notation, throughout the paper, we will use $\\mathbb{S}_k$ to denote the sampling distribution corresponding to any random iterate $x_k \\in \\R^n$ }. To calculate the resulting expectation with respect to the above-mentioned sampling distribution, let us first denote, $\\tau \\sim \\mathbb{S}$ as the set of sampled $\\beta$ constraints and $i^{*}$ as \\footnote{ Similarly, we will use $\\tau_k \\sim \\mathbb{S}_k$ to denote the sampled set and $i^* = \\argmax_{i \\in \\tau_k \\sim \\mathbb{S}_k} \\{a_i^Tx_k-b_i, 0\\} \\ = \\ \\argmax_{i \\in \\tau_k \\sim \\mathbb{S}_k} (A_{\\tau_k}x_k-b_{\\tau_k})^+_i$ for any iterate $x_k \\in \\R^n$.}\n\\begin{align}\n\\label{def:i1}\ni^* = \\argmax_{i \\in \\tau \\sim \\mathbb{S}} \\{a_i^Tx-b_i, 0\\} \\ = \\ \\argmax_{i} (A_{\\tau}x-b_{\\tau})^+_i,\n\\end{align}\nwhere, $A_{\\tau}$ denotes the collection of rows of $A$ restricted to the index set $\\tau$ and $(A_{\\tau}x-b_{\\tau})_i$ denotes the $i^{th}$ entry of $A_{\\tau}x-b_{\\tau}$. Using the above discussion with the list provided in equation \\eqref{eq:sampling}, we have the following:\n","start":25742,"end":28646}]},{"id":"S005","semantic_type":"derived_fact","kind":"derived_fact","labels":["def:exp"],"declared_external":false,"source_only":false,"title":"选中残差的期望公式","source_units":[{"raw_tex":"{\\allowdisplaybreaks\n\\begin{align}\n\\label{def:exp}\n\\E_{\\mathbb{S}} \\left[\\big |(a_{i^*}^Tx-b_{i^*})^{+}\\big |^2\\right] = \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} \\big | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}} \\big |^2, \\end{align}}\nwhere, $\\E_{\\mathbb{S}}$ denotes the required expectation corresponding to the sampling distribution $\\mathbb{S}$. The above expectation calculation was first used by De Loera \\textit{et.al} in their work \\cite{haddock:2017} where they first introduced the SKM method.\n","start":28646,"end":29188}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["def:function_i"],"declared_external":false,"source_only":false,"title":"单行残差函数","source_units":[{"raw_tex":"\\subsection{Function $f(x)$}\nIn this section, we formalize the definition of function $f:\\R^n \\rightarrow \\R$. Throughout section \\ref{sec:conv}, we will use the properties of function $f(x)$ \\footnote{Similar type of functions with uniform sampling have been studied in \\cite{needell:2016} \\cite{razaviyayn:2019} in the context of stochastic gradient descent and alternating projection algorithms respectively.}. First, for any index $i$, let us define the following function\n\\begin{align}\n\\label{def:function_i}\n    f_i(x) = \\frac{1}{2} |(a_{i}^Tx-b_{i})^+|^2, \\quad \\nabla f_i(x) = (a_{i}^Tx-b_{i})^+ a_{i}.\n\\end{align}\n","start":29445,"end":30068}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":["def:function"],"declared_external":false,"source_only":false,"title":"期望残差函数","source_units":[{"raw_tex":"Then to simplify the expectation expression of \\eqref{def:exp} further, we define the function $f$ and the gradient of $f$ as follows:\n\\begin{align}\n\\label{def:function}\n    f(x) =  \\E_{\\mathbb{S}}\\left[f_{i^*}(x)\\right], \\quad \\nabla f(x) = \\E_{\\mathbb{S}} \\left[\\nabla f_{i^*}(x)\\right],\n\\end{align}\nwhere, the index $i^*$ is selected by the rule provided in \\eqref{def:i1}.\n","start":30068,"end":30445}]},{"id":"S008","semantic_type":"algorithm","kind":"algorithm","labels":["alg:paskm","eq:askm1","eq:askm2","eq:yk"],"declared_external":false,"source_only":false,"title":"PASKM完整算法","source_units":[{"raw_tex":"\\begin{algorithm}\n\\caption{PASKM Algorithm: $x_{k+1} = \\textbf{PASKM}(A,b,x_0,K, \\delta, \\beta)$}\n\\label{alg:paskm}\n\\begin{algorithmic}\n\\STATE{Initialize $v_0 \\leftarrow x_0, \\  k \\leftarrow 0$;}\n\\WHILE{$k \\leq K$}\n\\STATE{Choose $\\gamma, \\omega,  \\alpha$ considering either \\eqref{cond} or \\eqref{eq:askm12} and update\n\\begin{align}\n\\label{eq:yk}\n  y_k = \\alpha v_k + (1-\\alpha)x_k;\n\\end{align}}\n\\STATE{Choose a sample of $\\beta$ constraints, $\\tau_k$, uniformly at random from the rows of matrix $A$. From these $\\beta$ constraints, choose $i^* = \\argmax_{i \\in \\tau_k} \\{a_i^Ty_k-b_i, 0\\}$; \\ Update \n\\begin{align}\nx_{k+1} & = y_k- \\delta \\frac{\\left(a_{i^*}^Ty_k -b_{i^*}\\right)^+}{\\|a_{i^*}\\|^2} a_{i^*}; \\label{eq:askm1} \\\\\nv_{k+1} & = \\omega v_k+ (1-\\omega)y_k-\\gamma \\frac{\\left(a_{i^*}^Ty_k -b_{i^*}\\right)^+}{\\|a_{i^*}\\|^2} a_{i^*}; \\label{eq:askm2}\n\\end{align}\n}\n\\STATE{$k \\leftarrow k+1$;}\n\\ENDWHILE\n\\RETURN $x$\n\\end{algorithmic}\n\\end{algorithm}\n","start":36846,"end":37803}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":["alg:paskm"],"declared_external":false,"source_only":false,"title":"共同初始点","source_units":[{"raw_tex":"\\begin{algorithm}\n\\caption{PASKM Algorithm: $x_{k+1} = \\textbf{PASKM}(A,b,x_0,K, \\delta, \\beta)$}\n\\label{alg:paskm}\n\\begin{algorithmic}\n\\STATE{Initialize $v_0 \\leftarrow x_0, \\  k \\leftarrow 0$;}\n","start":36846,"end":37042}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":["alg:paskm","eq:yk"],"declared_external":false,"source_only":false,"title":"混合点更新","source_units":[{"raw_tex":"\\WHILE{$k \\leq K$}\n\\STATE{Choose $\\gamma, \\omega,  \\alpha$ considering either \\eqref{cond} or \\eqref{eq:askm12} and update\n\\begin{align}\n\\label{eq:yk}\n  y_k = \\alpha v_k + (1-\\alpha)x_k;\n\\end{align}}\n","start":37042,"end":37242}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":["alg:paskm","eq:askm1","eq:askm2"],"declared_external":false,"source_only":false,"title":"随机投影与动量更新","source_units":[{"raw_tex":"\\STATE{Choose a sample of $\\beta$ constraints, $\\tau_k$, uniformly at random from the rows of matrix $A$. From these $\\beta$ constraints, choose $i^* = \\argmax_{i \\in \\tau_k} \\{a_i^Ty_k-b_i, 0\\}$; \\ Update \n\\begin{align}\nx_{k+1} & = y_k- \\delta \\frac{\\left(a_{i^*}^Ty_k -b_{i^*}\\right)^+}{\\|a_{i^*}\\|^2} a_{i^*}; \\label{eq:askm1} \\\\\nv_{k+1} & = \\omega v_k+ (1-\\omega)y_k-\\gamma \\frac{\\left(a_{i^*}^Ty_k -b_{i^*}\\right)^+}{\\|a_{i^*}\\|^2} a_{i^*}; \\label{eq:askm2}\n\\end{align}\n}\n","start":37242,"end":37719}]},{"id":"S012","semantic_type":"external_fact","kind":"external_fact","labels":["lem0"],"declared_external":true,"declaration_text":["原始前提清单：Hoffman's error bound；B0046引用hoffman及lewis:2010的Theorem 4.4。"],"source_only":false,"title":"Hoffman误差界","source_units":[{"raw_tex":"\\begin{mdframed}[backgroundcolor=gray!15,   topline=false,   bottomline =false,   rightline=false,   leftline=false] \\begin{lemma}\n\\label{lem0}\n(Hoffman \\cite{hoffman}, Theorem 4.4 in \\cite{lewis:2010}) Let $x \\in \\R^n$ and $P$ be the feasible region, then there exists a constant $L > 0$ such that the following identity holds:\n\\begin{align*}\n  d(x,P)^2 \\leq L^2 \\ \\|(Ax-b)^+\\|^2.\n\\end{align*}\n\\end{lemma} \\end{mdframed}\n","start":45149,"end":45571}]},{"id":"S013","semantic_type":"external_fact","kind":"external_fact","labels":["lem:skmseq"],"declared_external":true,"declaration_text":["原始前提清单：B0049，归于haddock:2017的Lemma 2.1。"],"source_only":false,"title":"同向序列加权不等式","source_units":[{"raw_tex":"\\begin{mdframed}[backgroundcolor=gray!15,   topline=false,   bottomline =false,   rightline=false,   leftline=false] \\begin{lemma}\n\\label{lem:skmseq}\n(Lemma 2.1 in \\cite{haddock:2017})  Let $\\{x_k\\},\\ \\{y_k\\}$ be real non-negative sequences such that $x_{k+1} > x_k > 0$ and $y_{k+1} \\geq y_k \\geq 0$, then\n\\begin{align*}\n  \\sum\\limits_{k=1}^{n} x_k y_k \\ \\geq \\   \\sum\\limits_{k=1}^{n} \\overline{x} y_k, \\quad \\text{where} \\ \\ \\overline{x} = \\frac{1}{n}\\sum\\limits_{k=1}^{n} x_k.\n\\end{align*}\n\\end{lemma} \\end{mdframed}\n","start":45880,"end":46401}]},{"id":"S014","semantic_type":"external_fact","kind":"external_fact","labels":["lem:distance"],"declared_external":true,"declaration_text":["原始前提清单：标准欧氏投影性质，包括B0051及可行域凸性。"],"source_only":false,"title":"最近点距离比较","source_units":[{"raw_tex":"\\begin{mdframed}[backgroundcolor=gray!15,   topline=false,   bottomline =false,   rightline=false,   leftline=false] \\begin{lemma}\n\\label{lem:distance}\nFor any $x \\in \\R^n$ and $\\bar{x} \\in P$, the following identity holds,\n\\begin{align*}\n    d(x,P)^2 \\ = \\ \\|x-\\mathcal{P}(x) \\|^2 \\ \\leq \\ \\|x-\\bar{x} \\|^2.\n\\end{align*}\n\\end{lemma} \\end{mdframed}\n","start":46948,"end":47297}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":["eq:askm12"],"declared_external":false,"source_only":false,"title":"固定ζ的参数方案","source_units":[{"raw_tex":"\\begin{align}\n\\label{eq:askm12}\n\\zeta =\\frac{3.99\\eta \\mu_1}{(1-\\mu_1)^2} , \\ \\omega = 1-\\frac{\\zeta \\mu_1^2+2\\gamma \\mu_1-\\zeta \\mu_1}{1+\\zeta \\mu_1^2} , \\ \\ \\gamma = \\sqrt{ \\zeta \\eta \\mu_1}, \\ \\ \\alpha = \\frac{ \\eta}{ \\eta + \\gamma },\n\\end{align}\n","start":84654,"end":84904}]},{"id":"S016","semantic_type":"lemma","kind":"lemma","labels":["lem1"],"declared_external":false,"source_only":false,"title":"平均行外积的谱界","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem1}\nLet $\\lambda_j$ be the $j^{th}$ eigenvalue of the matrix $W = \\E_{\\mathbb{S}}\\left[a_{i^*}a_{i^*}^T\\right]$, then for all $j$, the bound $0 \\leq \\lambda_j \\leq 1$ holds.\n\\end{lemma}","start":47416,"end":47624}]},{"id":"S017","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"平均外积谱界的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nSince $W$ is positive semi-definite, we can write $\\lambda_j \\geq 0$ for all $j$. Also as the mapping $\\lambda_{\\max}(X)$ is convex, using Jensen's inequality we have,\n\\begin{align*}\n    \\lambda_{\\max} (W) = \\lambda_{\\max} \\left[ \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] \\right] \\leq \\E_{\\mathbb{S}} \\left[\\lambda_{\\max}\\left(a_{i^*}a_{i^*}^T\\right)\\right] \\leq 1.\n\\end{align*}\n\\end{proof}","start":47641,"end":48049}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"平均外积半正定","source_units":[{"raw_tex":"Since $W$ is positive semi-definite, we can write $\\lambda_j \\geq 0$ for all $j$. Also as the mapping $\\lambda_{\\max}(X)$ is convex, using Jensen's inequality we have,\n","start":47655,"end":47823}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"Jensen控制最大特征值","source_units":[{"raw_tex":"Since $W$ is positive semi-definite, we can write $\\lambda_j \\geq 0$ for all $j$. Also as the mapping $\\lambda_{\\max}(X)$ is convex, using Jensen's inequality we have,\n\\begin{align*}\n    \\lambda_{\\max} (W) = \\lambda_{\\max} \\left[ \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] \\right] \\leq \\E_{\\mathbb{S}} \\left[\\lambda_{\\max}\\left(a_{i^*}a_{i^*}^T\\right)\\right] \\leq 1.\n\\end{align*}\n","start":47655,"end":48038}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","labels":["lem2"],"declared_external":false,"source_only":false,"title":"采样外积的矩阵上界","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem2}\nFor any $1 \\leq \\beta \\leq m$, we have the following:\n\\begin{align*}\n \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] \\preceq \\frac{\\beta}{m} A^TA.\n\\end{align*}\n\\end{lemma}","start":48168,"end":48365}]},{"id":"S021","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"采样矩阵界的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nSee Appendix 1.\n\\end{proof}","start":48382,"end":48423},{"raw_tex":"\\paragraph{Proof of Lemma \\ref{lem2}}\nUsing the expectation expression given in the first section we have,\n\\begin{align*}\n     \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] & = \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} (A^TA)_{\\underline{\\mathbf{i_j}}} \\\\\n     &  \\preceq \\ \\frac{\\binom{m-1}{\\beta-1}}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta}  (A^TA)_{\\underline{\\mathbf{i_j}}} \\ \\preceq \\ \\frac{\\beta}{m} \\sum\\limits_{i = 1}^{m}  a_{i} a_{i}^T  \\ = \\ \\frac{\\beta}{m} A^TA.\n\\end{align*}\nHere, the notation $(A^TA)_{\\underline{\\mathbf{i_j}}}$ denotes the matrix $a_l a_l^T$ where the index $l$ belongs to the list \\eqref{eq:sampling}. Furthermore, the index $l$ corresponds to the $(\\beta+ j)^{th}$ entry on the list \\eqref{eq:sampling}. This proves the Lemma.\n%$(Ax-b)^{+}_{\\fbox{${i_j}$}}$\n%$(Ax-b)^{+}_{\\fbox{\\strut ${i_j}$}}$\n%$(Ax-b)^{+}_{\\fbox{$\\mathstrut$ ${i_j}$}}$\n\n%$(Ax-b)^{+}_{\\underline{\\mathbf{i_j}}}$\n\n% $\\overline{\\mathbf{x}}$  $\\underline{\\mathbf{x}}$ \n\n\n","start":115634,"end":116654}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"组合权重控制矩阵和","source_units":[{"raw_tex":"\\paragraph{Proof of Lemma \\ref{lem2}}\nUsing the expectation expression given in the first section we have,\n\\begin{align*}\n     \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] & = \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} (A^TA)_{\\underline{\\mathbf{i_j}}} \\\\\n     &  \\preceq \\ \\frac{\\binom{m-1}{\\beta-1}}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta}  (A^TA)_{\\underline{\\mathbf{i_j}}} \\ \\preceq \\ \\frac{\\beta}{m} \\sum\\limits_{i = 1}^{m}  a_{i} a_{i}^T  \\ = \\ \\frac{\\beta}{m} A^TA.\n\\end{align*}\nHere, the notation $(A^TA)_{\\underline{\\mathbf{i_j}}}$ denotes the matrix $a_l a_l^T$ where the index $l$ belongs to the list \\eqref{eq:sampling}. Furthermore, the index $l$ corresponds to the $(\\beta+ j)^{th}$ entry on the list \\eqref{eq:sampling}. This proves the Lemma.\n","start":115634,"end":116443}]},{"id":"S023","semantic_type":"lemma","kind":"lemma","labels":["lem3"],"declared_external":false,"source_only":false,"title":"函数值与距离的双边界","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem3}\nFor any $x \\in \\R^n$ with $\\lambda_{\\max} = \\lambda_{\\max}(A^TA)$, we have the following:\n\\begin{align*}\n    \\frac{\\mu_1}{ 2} \\ d(x,P)^2 \\ \\leq \\ f(x) \\ \\leq \\  \\frac{\\mu_2}{2}\\ d(x,P)^2,\n\\end{align*}\nwith $ 0 < \\mu_1 = \\frac{1}{m L^2} \\leq \\ \\mu_2 = \\min \\left\\{1, \\frac{\\beta}{m} \\lambda_{\\max}\\right\\} \\leq 1$. \n\\end{lemma}","start":48545,"end":48898}]},{"id":"S024","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"距离双边界的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nSee Appendix 1.\n\\end{proof}","start":48915,"end":48956},{"raw_tex":"\\paragraph{Proof of Lemma \\ref{lem3}}\nUsing the definition of the expectation from \\eqref{def:exp}, we have,\n\\allowdisplaybreaks{\n\\begin{align*}\n \\E_{\\mathbb{S}} \\left[\\big | (a_{i^*}^Tx-b_{i^*})^+\\big |^2\\right] & \\overset{\\eqref{def:exp}}{=} \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1}  | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}} |^2 \\\\\n    & \\overset{\\text{Lemma} \\ \\ref{lem:skmseq}}{\\geq} \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta}  \\frac{\\sum\\limits_{l = 0}^{m-\\beta} \\binom{\\beta-1+l}{\\beta-1}}{m-\\beta +1} | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}} |^2 \\\\\n    & \\geq \\frac{1}{m-\\beta+1} \\sum\\limits_{j = 0}^{m-\\beta}  \\big |(Ax - b)^+_{\\underline{\\mathbf{i_j}}} \\big |^2  \\\\\n    & \\geq \\frac{1}{m-\\beta+1} \\min\\{\\frac{m-\\beta+1}{m-s}, 1\\} \\ \\|(Ax - b)^+\\|^2 \\\\\n    &  \\overset{\\text{Lemma} \\ \\ref{lem0}}{\\geq} \\ \\frac{1}{m L^2}  \\ d(x,P)^2. \n\\end{align*}}\nHere, $s$ is the number of zero entries in the residual $(Ax - b)^+$, which also corresponds to the number of satisfied constraints for $x$. Since $A \\mathcal{P} (x) \\leq b$, we have the following:\n\\begin{align*}\n    \\E_{\\mathbb{S}} \\left[\\big | (a_{i^*}^Tx-b_{i^*})^+\\big |^2\\right]   & \\overset{\\eqref{def:exp}}{=}  \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}}|^2 \\\\\n    & \\leq  \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} \\ \\big | (Ax-A\\mathcal{P} (x))_{\\underline{\\mathbf{i_j}}}   \\big|^2 \\\\\n    & =  \\frac{1}{\\binom{m}{\\beta}} (x-\\mathcal{P} (x))^T\\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} \\ (A^TA)_{\\underline{\\mathbf{i_j}}} (x-\\mathcal{P} (x) ) \\\\\n    & =  (x-\\mathcal{P} (x) )^T \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] (x-\\mathcal{P} (x) ) \\\\\n    & \\overset{\\text{Lemma} \\ \\ref{lem1} \\ \\& \\ \\ref{lem2} }{\\leq} \\  \\min \\left\\{1, \\frac{\\beta}{m} \\lambda_{\\max}\\right\\} \\big \\| x- \\mathcal{P} (x) \\big \\|^2 \\\\\n    & = \\  \\min \\left\\{1, \\frac{\\beta}{m} \\lambda_{\\max}\\right\\} d(x,P)^2.\n\\end{align*}\nCombining the above identities and using the expression for $f(x)$ from \\eqref{def:function} we get,\n\\begin{align*}\n    \\frac{\\mu_1}{ 2} \\ d(x,P)^2 \\ \\leq \\ f(x) \\ \\leq \\  \\frac{\\mu_2}{2} \\ d(x,P)^2,\n\\end{align*}\nwhich proves the Lemma. \n\n","start":116654,"end":118947}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"残差期望的距离下界","source_units":[{"raw_tex":"\\paragraph{Proof of Lemma \\ref{lem3}}\nUsing the definition of the expectation from \\eqref{def:exp}, we have,\n\\allowdisplaybreaks{\n\\begin{align*}\n \\E_{\\mathbb{S}} \\left[\\big | (a_{i^*}^Tx-b_{i^*})^+\\big |^2\\right] & \\overset{\\eqref{def:exp}}{=} \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1}  | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}} |^2 \\\\\n    & \\overset{\\text{Lemma} \\ \\ref{lem:skmseq}}{\\geq} \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta}  \\frac{\\sum\\limits_{l = 0}^{m-\\beta} \\binom{\\beta-1+l}{\\beta-1}}{m-\\beta +1} | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}} |^2 \\\\\n    & \\geq \\frac{1}{m-\\beta+1} \\sum\\limits_{j = 0}^{m-\\beta}  \\big |(Ax - b)^+_{\\underline{\\mathbf{i_j}}} \\big |^2  \\\\\n    & \\geq \\frac{1}{m-\\beta+1} \\min\\{\\frac{m-\\beta+1}{m-s}, 1\\} \\ \\|(Ax - b)^+\\|^2 \\\\\n    &  \\overset{\\text{Lemma} \\ \\ref{lem0}}{\\geq} \\ \\frac{1}{m L^2}  \\ d(x,P)^2. \n\\end{align*}}\nHere, $s$ is the number of zero entries in the residual $(Ax - b)^+$, which also corresponds to the number of satisfied constraints for $x$. Since $A \\mathcal{P} (x) \\leq b$, we have the following:\n","start":116654,"end":117763}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"可行投影给出谱上界","source_units":[{"raw_tex":"Here, $s$ is the number of zero entries in the residual $(Ax - b)^+$, which also corresponds to the number of satisfied constraints for $x$. Since $A \\mathcal{P} (x) \\leq b$, we have the following:\n\\begin{align*}\n    \\E_{\\mathbb{S}} \\left[\\big | (a_{i^*}^Tx-b_{i^*})^+\\big |^2\\right]   & \\overset{\\eqref{def:exp}}{=}  \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}}|^2 \\\\\n    & \\leq  \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} \\ \\big | (Ax-A\\mathcal{P} (x))_{\\underline{\\mathbf{i_j}}}   \\big|^2 \\\\\n    & =  \\frac{1}{\\binom{m}{\\beta}} (x-\\mathcal{P} (x))^T\\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} \\ (A^TA)_{\\underline{\\mathbf{i_j}}} (x-\\mathcal{P} (x) ) \\\\\n    & =  (x-\\mathcal{P} (x) )^T \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] (x-\\mathcal{P} (x) ) \\\\\n    & \\overset{\\text{Lemma} \\ \\ref{lem1} \\ \\& \\ \\ref{lem2} }{\\leq} \\  \\min \\left\\{1, \\frac{\\beta}{m} \\lambda_{\\max}\\right\\} \\big \\| x- \\mathcal{P} (x) \\big \\|^2 \\\\\n    & = \\  \\min \\left\\{1, \\frac{\\beta}{m} \\lambda_{\\max}\\right\\} d(x,P)^2.\n\\end{align*}\n","start":117565,"end":118708}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"合并函数值双边界","source_units":[{"raw_tex":"Combining the above identities and using the expression for $f(x)$ from \\eqref{def:function} we get,\n\\begin{align*}\n    \\frac{\\mu_1}{ 2} \\ d(x,P)^2 \\ \\leq \\ f(x) \\ \\leq \\  \\frac{\\mu_2}{2} \\ d(x,P)^2,\n\\end{align*}\nwhich proves the Lemma. \n\n","start":118708,"end":118947}]},{"id":"S028","semantic_type":"lemma","kind":"lemma","labels":["lem:grad"],"declared_external":false,"source_only":false,"title":"两点梯度内积上界","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:grad}\nFor any $x, y \\in \\R^n$, we have the following:\n\\begin{align*}\n \\langle  x-y, \\E_{\\mathbb{S}}  \\left[(a_{i^*}^Ty-b_{i^*})^{+} a_{i^*}\\right]  \\rangle  & =  \\langle  x-y, \\nabla f(y)   \\rangle   \\\\\n & \\leq    f(x) - f(y) \\leq     \\frac{\\mu_2}{2} \\ d(x,P)^2 -  \\frac{\\mu_1}{2} \\ d(y,P)^2.\n\\end{align*}\n\\end{lemma}","start":50379,"end":50721}]},{"id":"S029","semantic_type":"proof","kind":"proof","labels":["eq:grad"],"declared_external":false,"source_only":false,"title":"两点梯度界的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nSee Appendix 1.\n\\end{proof}","start":50738,"end":50779},{"raw_tex":"\\paragraph{Proof of Lemma \\ref{lem:grad}}\n\nFrom the definition of $f(x)$, it can be easily checked that $f(x)$ is a convex function. Now, by the convexity property of $f(x)$, for any $x, y \\in \\R^n$, we have the following:\n\\begin{align}\n\\label{eq:grad}\n    \\langle x-y, \\nabla f(y) \\rangle \\leq f(x) - f(y).\n\\end{align}\nTherefore, we have\n\\begin{align*}\n\\big \\langle  x-y, \\E_{\\mathbb{S}}  \\left[(a_{i^*}^Ty-b_{i^*})^{+} a_{i^*}\\right] \\big \\rangle & =  \\langle x-y, \\nabla f(y) \\rangle  \\leq  \\ f(x) - f(y) \\\\\n&  \\overset{\\text{Lemma} \\  \\ref{lem3} }{\\leq}  \\    \\frac{\\mu_2}{2} \\ d(x,P)^2 -  \\frac{\\mu_1}{2} \\ d(y,P)^2.\n\\end{align*}\nThis completes the proof.\n\n\n","start":118947,"end":119610}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":["eq:grad"],"declared_external":false,"source_only":false,"title":"凸性与一阶支持","source_units":[{"raw_tex":"\\paragraph{Proof of Lemma \\ref{lem:grad}}\n\nFrom the definition of $f(x)$, it can be easily checked that $f(x)$ is a convex function. Now, by the convexity property of $f(x)$, for any $x, y \\in \\R^n$, we have the following:\n\\begin{align}\n\\label{eq:grad}\n    \\langle x-y, \\nabla f(y) \\rangle \\leq f(x) - f(y).\n\\end{align}\n","start":118947,"end":119267}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"代入距离双边估计","source_units":[{"raw_tex":"Therefore, we have\n\\begin{align*}\n\\big \\langle  x-y, \\E_{\\mathbb{S}}  \\left[(a_{i^*}^Ty-b_{i^*})^{+} a_{i^*}\\right] \\big \\rangle & =  \\langle x-y, \\nabla f(y) \\rangle  \\leq  \\ f(x) - f(y) \\\\\n&  \\overset{\\text{Lemma} \\  \\ref{lem3} }{\\leq}  \\    \\frac{\\mu_2}{2} \\ d(x,P)^2 -  \\frac{\\mu_1}{2} \\ d(y,P)^2.\n\\end{align*}\nThis completes the proof.\n\n\n","start":119267,"end":119610}]},{"id":"S032","semantic_type":"lemma","kind":"lemma","labels":["lem:grad1"],"declared_external":false,"source_only":false,"title":"可行点方向的梯度界","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:grad1}\nFor any $y \\in \\R^n$ and $\\bar{y}$ such that $A \\bar{y} \\leq b$, we have the following:\n\\begin{align*}\n\\langle  \\bar{y}-y, \\E_{\\mathbb{S}} \\left[a_{i^*}(a_{i^*}^Ty-b_{i^*})^{+}\\right]  \\rangle = \\ \\langle  \\bar{y}-y, \\nabla f(y)  \\rangle    \\leq  -2f(y)  \\leq \\ - \\mu_1  d(y,P)^2.\n\\end{align*}\n\\end{lemma}","start":52283,"end":52620}]},{"id":"S033","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"可行方向界的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nSee Appendix 1.\n\\end{proof}","start":52637,"end":52678},{"raw_tex":"\\paragraph{Proof of Lemma \\ref{lem:grad1}}\n\n\\iffalse\n\nLet's denote, $ \\mathcal{J} = \\{i_j \\ | \\ c_{i_j}^Ty-d_{i_j} \\geq 0, \\ i_j \\in T(y)\\}$. Then following the construction of \\eqref{eq:f}, the function $f(y)$ can be expressed as $f(y) = \\|C_{\\mathcal{J}} y-d_{\\mathcal{J}} \\|^2$.\nSince, $C_{\\mathcal{J}} y-d_{\\mathcal{J}} \\geq 0$ and $C_{\\mathcal{J}}\\bar{y}-d_{\\mathcal{J}} \\leq 0$, we have,\n\\begin{align*}\n    \\big \\langle C_{\\mathcal{J}}y-d_{\\mathcal{J}} , C_{\\mathcal{J}}\\bar{y}-d_{\\mathcal{J}} \\big \\rangle \\ \\leq \\ 0.\n\\end{align*}\nAdding $\\|C_{\\mathcal{J}}y-d_{\\mathcal{J}}\\|^2$ to both sides and rearranging terms we get,\n\\begin{align}\n\\label{eq:grad1}\n \\|C_{\\mathcal{J}}y-d_{\\mathcal{J}} \\|^2 \\ \\leq \\  \\big \\langle C_{\\mathcal{J}}^T \\left( C_{\\mathcal{J}}y-d_{\\mathcal{J}} \\right), y - \\bar{y} \\big \\rangle.\n\\end{align}\n\n\\fi\n\nSince, $\\bar{y} \\in P$, from the definition we have,\n\\begin{align*}\n   \\langle  \\bar{y}-y, \\E_{\\mathbb{S}}  \\left[a_{i^*}(a_{i^*}^Ty-b_{i^*})^{+}\\right]  \\rangle \\ & = \\E_{\\mathbb{S}} \\left[(a_{i^*}^Ty-b_{i^*})^{+} \\left(a_{i^*}^T\\bar{y}- a_{i^*}^T y\\right)\\right] \\\\\n   & \\leq \\E_{\\mathbb{S}} \\left[(a_{i^*}^Ty-b_{i^*})^{+} \\left(b_{i^*}- a_{i^*}^T y\\right)\\right] \\\\\n   & = - \\E_{\\mathbb{S}} \\left[\\big | (a_{i^*}^Ty-b_{i^*})^{+} \\big |^2\\right] \\\\\n   &= - 2f(y) \\overset{\\text{Lemma} \\ \\ref{lem3} }{\\leq} \\ \\ -\\mu_1 \\ d(y,P)^2.\n\\end{align*}\nHere, we used the identity $xx^+ = |x^+|^2$. This proves the Lemma.\n\n","start":119610,"end":121059}]},{"id":"S034","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"禁用的矩阵表示旧稿","source_units":[{"raw_tex":"\\iffalse\n\nLet's denote, $ \\mathcal{J} = \\{i_j \\ | \\ c_{i_j}^Ty-d_{i_j} \\geq 0, \\ i_j \\in T(y)\\}$. Then following the construction of \\eqref{eq:f}, the function $f(y)$ can be expressed as $f(y) = \\|C_{\\mathcal{J}} y-d_{\\mathcal{J}} \\|^2$.\nSince, $C_{\\mathcal{J}} y-d_{\\mathcal{J}} \\geq 0$ and $C_{\\mathcal{J}}\\bar{y}-d_{\\mathcal{J}} \\leq 0$, we have,\n\\begin{align*}\n    \\big \\langle C_{\\mathcal{J}}y-d_{\\mathcal{J}} , C_{\\mathcal{J}}\\bar{y}-d_{\\mathcal{J}} \\big \\rangle \\ \\leq \\ 0.\n\\end{align*}\nAdding $\\|C_{\\mathcal{J}}y-d_{\\mathcal{J}}\\|^2$ to both sides and rearranging terms we get,\n\\begin{align}\n\\label{eq:grad1}\n \\|C_{\\mathcal{J}}y-d_{\\mathcal{J}} \\|^2 \\ \\leq \\  \\big \\langle C_{\\mathcal{J}}^T \\left( C_{\\mathcal{J}}y-d_{\\mathcal{J}} \\right), y - \\bar{y} \\big \\rangle.\n\\end{align}\n\n\\fi\n","start":119654,"end":120445}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"逐行可行性压低内积","source_units":[{"raw_tex":"Since, $\\bar{y} \\in P$, from the definition we have,\n\\begin{align*}\n   \\langle  \\bar{y}-y, \\E_{\\mathbb{S}}  \\left[a_{i^*}(a_{i^*}^Ty-b_{i^*})^{+}\\right]  \\rangle \\ & = \\E_{\\mathbb{S}} \\left[(a_{i^*}^Ty-b_{i^*})^{+} \\left(a_{i^*}^T\\bar{y}- a_{i^*}^T y\\right)\\right] \\\\\n   & \\leq \\E_{\\mathbb{S}} \\left[(a_{i^*}^Ty-b_{i^*})^{+} \\left(b_{i^*}- a_{i^*}^T y\\right)\\right] \\\\\n   & = - \\E_{\\mathbb{S}} \\left[\\big | (a_{i^*}^Ty-b_{i^*})^{+} \\big |^2\\right] \\\\\n   &= - 2f(y) \\overset{\\text{Lemma} \\ \\ref{lem3} }{\\leq} \\ \\ -\\mu_1 \\ d(y,P)^2.\n\\end{align*}\nHere, we used the identity $xx^+ = |x^+|^2$. This proves the Lemma.\n\n","start":120446,"end":121059}]},{"id":"S036","semantic_type":"lemma","kind":"lemma","labels":["lem4"],"declared_external":false,"source_only":false,"title":"单步SKM距离收缩","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem4}\nFor any $x \\in \\R^n$ and $0 < \\delta < 2$, we have the following:\n\\begin{align*}\n  \\E_{\\mathbb{S}} \\left[ d(z,P)^2\\right] = \\E_{\\mathbb{S}} \\left[ \\Big \\| x- \\mathcal{P} (x) - \\delta \\left(a_{i^*}^Tx-b_{i^*}\\right)^+ a_{i^*}\\Big \\|^2 \\right]  \\leq   h(\\delta) \\ d(x,P)^2,\n\\end{align*}\nwhere, $z = x - \\delta \\left(a_{i^*}^Tx-b_{i^*}\\right)^+ a_{i^*}$, $ \\eta = 2 \\delta -\\delta^2$ and $h(\\delta) = 1- \\eta \\mu_1 < 1$.\n\\end{lemma}","start":53802,"end":54258}]},{"id":"S037","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"单步收缩的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nSee Appendix 1.\n\\end{proof}","start":54275,"end":54316},{"raw_tex":"\\paragraph{Proof of Lemma \\ref{lem4}}\n\nSince, $\\mathcal{P}(x) \\in P$, we have\n{\\allowdisplaybreaks\n\\begin{align*}\n    \\E_{\\mathbb{S}}  \\left[ d(z,P)^2\\right] &  \\overset{\\text{Lemma} \\ \\ref{lem:distance}}{\\leq} \\  \\E_{\\mathbb{S}}  \\left[ \\|z- \\mathcal{P}(x)\\|^2\\right] = \\E_{\\mathbb{S}} \\left[ \\big \\| x- \\mathcal{P} (x) - \\delta \\left(a_{i^*}^Tx-b_{i^*}\\right)^+ a_{i^*} \\big \\|^2 \\right] \\\\\n    & \\overset{\\eqref{def:function}}{=} \\ \\|x - \\mathcal{P} (x) \\|^2 + 2\\delta^2 f(x)  + 2 \\delta \\ \\big  \\langle  \\mathcal{P}(x)-x, \\nabla f(x) \\big \\rangle \\\\\n    & \\overset{\\text{Lemma} \\ \\ref{lem3} }{\\leq} \\ \\|x - \\mathcal{P} (x) \\|^2 -2(2\\delta-\\delta^2) f(x) \\\\\n    & \\leq \\ \\|x-  \\mathcal{P} (x) \\|^2 - (2\\delta-\\delta^2) \\ \\mu_1  \\|x-  \\mathcal{P} (x) \\|^2 = h(\\delta) \\ d(x,P)^2.\n\\end{align*}}\nHere, we used the lower bound of the expected value from Lemma \\ref{lem3}.\n\n","start":121059,"end":121931}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"旧投影比较与平方展开","source_units":[{"raw_tex":"\\paragraph{Proof of Lemma \\ref{lem4}}\n\nSince, $\\mathcal{P}(x) \\in P$, we have\n{\\allowdisplaybreaks\n\\begin{align*}\n    \\E_{\\mathbb{S}}  \\left[ d(z,P)^2\\right] &  \\overset{\\text{Lemma} \\ \\ref{lem:distance}}{\\leq} \\  \\E_{\\mathbb{S}}  \\left[ \\|z- \\mathcal{P}(x)\\|^2\\right] = \\E_{\\mathbb{S}} \\left[ \\big \\| x- \\mathcal{P} (x) - \\delta \\left(a_{i^*}^Tx-b_{i^*}\\right)^+ a_{i^*} \\big \\|^2 \\right] \\\\\n    & \\overset{\\eqref{def:function}}{=} \\ \\|x - \\mathcal{P} (x) \\|^2 + 2\\delta^2 f(x)  + 2 \\delta \\ \\big  \\langle  \\mathcal{P}(x)-x, \\nabla f(x) \\big \\rangle \\\\\n","start":121059,"end":121613}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"保留函数值的下降量","source_units":[{"raw_tex":"    & \\overset{\\text{Lemma} \\ \\ref{lem3} }{\\leq} \\ \\|x - \\mathcal{P} (x) \\|^2 -2(2\\delta-\\delta^2) f(x) \\\\\n","start":121613,"end":121720}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"函数下降转为距离收缩","source_units":[{"raw_tex":"    & \\leq \\ \\|x-  \\mathcal{P} (x) \\|^2 - (2\\delta-\\delta^2) \\ \\mu_1  \\|x-  \\mathcal{P} (x) \\|^2 = h(\\delta) \\ d(x,P)^2.\n\\end{align*}}\nHere, we used the lower bound of the expected value from Lemma \\ref{lem3}.\n\n","start":121720,"end":121931}]},{"id":"S041","semantic_type":"theorem","kind":"theorem","labels":["th:10"],"declared_external":false,"source_only":false,"title":"PASKM加权误差线性收敛","source_units":[{"raw_tex":"\\begin{theorem}\n\\label{th:10}\nLet, $v_{k+1}$ and $x_{k+1}$ are generated by Algorithm \\ref{alg:paskm}. If we select the parameters $\\omega, \\ \\gamma, \\ \\alpha$ as\n\\begin{align*}\n\\omega = 1-\\frac{\\zeta \\mu_1^2+2\\gamma \\mu_1-\\zeta \\mu_1}{1+\\zeta \\mu_1^2} , \\ \\ \\gamma = \\sqrt{ \\zeta \\eta \\mu_1}, \\ \\ \\alpha = \\frac{ \\eta}{ \\eta + \\gamma },\n\\end{align*}\nwhere, $\\zeta$ is chosen as $ 0 < \\zeta  < \\frac{4\\eta \\mu_1}{(1-\\mu_1)^2} $ if $\\mu_1 < 1$, otherwise choose any $\\zeta > 0$. Then, for any $0 < \\delta < 2$, the sequence of iterates $\\{v_k\\}, \\ \\{x_k\\}$ converges and the following result holds:\n\\begin{align*}\n\\E \\left[ d(v_{k+1},P)^2 + \\zeta \\mu_1 \\  d(x_{k+1},P)^2 \\right] \\ & \\leq \\ \\omega^{k+1} \\ \\E \\left[ d(v_{0},P)^2+ \\zeta \\mu_1 \\  d(x_{0},P)^2 \\right] \\\\\n& = (1+\\zeta \\mu_1) \\ \\omega^{k+1} \\ d(x_{0},P)^2.\n\\end{align*}  \nThis theorem implies that the PASKM algorithm converges linearly with a rate of $\\omega$, which accumulates to a total of $\\mathcal{O}(\\frac{1+\\zeta \\mu_1^2}{\\zeta \\mu_1^2+2\\gamma \\mu_1-\\zeta \\mu_1}\\log{1/\\epsilon})$ iterations to bring the given error below $\\epsilon > 0$.\n\\end{theorem}","start":78323,"end":79444}]},{"id":"S042","semantic_type":"proof","kind":"proof","labels":["eq:askm10","eq:askm11","eq:askm13","eq:askm14","eq:askm4","eq:askm5","eq:askm6","eq:askm9"],"declared_external":false,"source_only":false,"title":"目标定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nSee Appendix 3.\n\\end{proof}","start":79461,"end":79502},{"raw_tex":"\\paragraph{Proof of Theorem \\ref{th:10}}\n\nLet us define, $\\mathcal{V} = \\omega \\mathcal{P}(v_{k}) +(1-\\omega) \\mathcal{P}(y_{k}) $. Since $\\mathcal{V} \\in P$, using the update formula of $v_{k+1}$ from equation \\eqref{eq:askm2}, we have,\n\\allowdisplaybreaks{\n\\begin{align}\n\\label{eq:askm4}\n    d(v_{k+1},P)^2 & = \\|v_{k+1} - \\mathcal{P}(v_{k+1})\\|^2 \\overset{\\text{Lemma} \\ \\ref{lem:distance}}{\\leq}  \\|v_{k+1} - \\mathcal{V}\\|^2  \\nonumber \\\\\n    & \\overset{\\eqref{eq:askm2}}{=}  \\big \\|\\omega v_k + (1-\\omega) y_k- \\mathcal{V} -\\gamma (a_{i^*}^Ty_k- b_{i^*})^+ a_{i^*} \\big \\|^2 \\nonumber \\\\\n    & =   \\underbrace{ \\|\\omega v_k + (1-\\omega) y_k- \\mathcal{V} \\|^2}_{I_1} + \\gamma^2 \\underbrace{\\|(a_{i^*}^Ty_k- b_{i^*})^+a_{i^*}\\|^2}_{I_2} \\nonumber \\\\\n    & - 2 \\gamma  \\underbrace{\\big \\langle \\omega v_k + (1-\\omega) y_k- \\mathcal{V} \\ ,\\  a_{i^*} (a_{i^*}^Ty_k-b_{i^{*}})^{+} \\big \\rangle }_{I_3} \\nonumber \\\\\n    & = I_1 + \\gamma^2 I_2 -2 \\gamma I_3.\n\\end{align}}\nSince $\\|\\cdot\\|^{2}$ is a convex function and $0 < \\omega < 1$, we can bound the expected first term as follows,\n\\begin{align}\n\\label{eq:askm5}\n    \\E_{\\mathbb{S}_k} [I_1]  &= \\E_{\\mathbb{S}_k} \\left[\\|\\omega v_k + (1-\\omega) y_k-\\mathcal{V}\\|^2\\right] \\nonumber \\\\\n    & = \\E_{\\mathbb{S}_k} \\left[\\|\\omega v_k + (1-\\omega) y_k- \\omega \\mathcal{P}(v_{k}) - (1-\\omega) \\mathcal{P}(y_{k})\\|^2\\right] \\nonumber \\\\\n    & \\leq \\omega \\| v_k-\\mathcal{P}(v_{k})\\|^2 + (1-\\omega) \\|y_k-\\mathcal{P}(y_{k})\\|^2\\nonumber \\\\\n    & =  \\omega \\ d(v_k,P)^2 + (1-\\omega) \\ d(y_k,P)^2.\n\\end{align}\nTaking expectation with respect to the sampling distribution in the second term of equation \\eqref{eq:askm4} and using Lemma \\ref{lem4} with the choice $z = x_{k+1}, \\ x = y_k$ and $\\eta = 2\\delta-\\delta^2$, we get,\n\\begin{align}\n\\label{eq:askm6}\n   \\gamma^2 \\E_{\\mathbb{S}_k} \\left[\\|(a_{i^*}^Ty_k- b_{i^*})^+ a_{i^*}\\|^2\\right] & \\ = \\  \\gamma^2 \\E_{\\mathbb{S}_k} \\left[|(a_{i^*}^Ty_k- b_{i^*})^+ |^2\\right] \\nonumber \\\\\n    \\overset{\\text{Lemma} \\ \\ref{lem4}}{\\leq} & \\ \\frac{\\gamma^2 }{\\eta} \\left[d(y_k,P)^2 - \\E \\left[ d(x_{k+1},P)^2 \\right]\\right].\n\\end{align}\n\nNow, taking expectation in the third term of \\eqref{eq:askm4} we get,\n\\begin{align}\n\\label{eq:askm9}\n    - 2  \\gamma  \\E_{\\mathbb{S}_k} & [I_3 ]  = - 2 \\gamma \\big \\langle \\omega v_k + (1-\\omega) y_k-\\mathcal{V}, \\E_{\\mathbb{S}_k} \\left[ a_{i^*} (a_{i^*}^Ty_k-b_{i^*})^{+}\\right]  \\big \\rangle \\nonumber \\\\\n    & \\overset{\\eqref{eq:yk} \\ \\& \\ \\eqref{def:function} }{=} - 2 \\gamma \\big \\langle \\frac{\\omega}{\\alpha} \\left[y_k - (1-\\alpha) x_k\\right] + (1-\\omega) y_k-\\mathcal{V},  \\nabla f(y_k)  \\big \\rangle \\nonumber \\\\\n    & = - 2 \\gamma \\big \\langle \\frac{\\omega(1-\\alpha)}{\\alpha}(y_k-x_k)+ y_k-\\mathcal{V}, \\nabla f(y_k)  \\big \\rangle.\n\\end{align}\nUsing Lemma \\ref{lem:grad} and Lemma \\ref{lem:grad1} we can simplify equation \\eqref{eq:askm9} as follows,\n\\begin{align}\n\\label{eq:askm10}\n  & \\ -  2 \\gamma  \\E_{\\mathbb{S}_k}  [I_3 ]  =   - 2 \\gamma \\big \\langle \\frac{\\omega(1-\\alpha)}{\\alpha}(y_k-x_k)+ y_k-\\mathcal{V}, \\nabla f(y_k)  \\big \\rangle \\nonumber \\\\\n    & \\quad =    2 \\gamma \\frac{\\omega(1-\\alpha)}{\\alpha} \\big \\langle x_k-y_k, \\nabla f(y_k)  \\big \\rangle + 2 \\gamma \\big \\langle \\mathcal{V}-y_k, \\nabla f(y_k)  \\big \\rangle \\nonumber \\\\\n  & \\overset{\\text{Lemma} \\ \\ref{lem:grad} \\ \\& \\ \\ref{lem:grad1}}{\\leq} \\frac{\\gamma \\omega (1-\\alpha)}{\\alpha} \\  d(x_k,P)^2 - \\frac{ \\mu_1 \\gamma  \\omega (1-\\alpha)}{\\alpha} \\ d(y_k,P)^2 - 2 \\mu_1 \\gamma \\ d(y_k,P)^2.\n\\end{align}\nNow, substituting the values of equation \\eqref{eq:askm5}, \\eqref{eq:askm6} \\& \\eqref{eq:askm10} in equation \\eqref{eq:askm4} we get the following:\n\\allowdisplaybreaks{\n\\begin{align*}\n \\E [ & d(v_{k+1} ,P)^2  ]  =  I_1 + \\gamma^2 \\E_{\\mathbb{S}_k}[I_2 ]- 2 \\gamma \\E_{\\mathbb{S}_k}[I_3] \\nonumber \\\\\n & = \\omega \\ d(v_{k},P)^2 + (1-\\omega) \\ d(y_k,P)^2 + \\frac{\\gamma^2 }{\\eta} \\left\\{d(y_k,P)^2 - \\E \\left[ d(x_{k+1},P)^2 \\right]\\right\\} \\nonumber \\\\\n & \\quad + \\frac{\\gamma \\omega (1-\\alpha)}{\\alpha}  d(x_k,P)^2 - \\gamma \\mu_1 \\left(2+\\frac{ \\omega (1-\\alpha)}{\\alpha}\\right)  d(y_k,P)^2.\n\\end{align*}}\nWith further simplification, the above identity can be written as follows:\n\\allowdisplaybreaks{\n\\begin{align}\n\\label{eq:askm11}\n & \\E \\left[ d(v_{k+1},P)^2 + \\frac{\\gamma^2}{\\eta} d(x_{k+1},P)^2 \\right]  = \\omega \\ \\left[d(v_{k},P)^2 + \\frac{\\gamma (1-\\alpha)}{\\alpha} d(x_{k},P)^2 \\right] \\nonumber \\\\\n & \\quad \\quad \\quad \\quad \\quad \\quad \\quad + d(y_k,P)^2 \\ \\left\\{1-\\omega+ \\frac{\\gamma^2 }{\\eta}  - 2\\gamma \\mu_1 - \\frac{\\gamma \\omega \\mu_1 (1-\\alpha)}{\\alpha}  \\right\\}.\n\\end{align}}\nNow, let's choose the parameters as in equation \\eqref{eq:askm12} along with $0 < \\zeta  < \\frac{4\\eta \\mu_1}{(1-\\mu_1)^2} $. We can easily see that $\\frac{\\gamma^2 }{\\eta} = \\frac{ \\gamma(1-\\alpha)}{\\alpha}$ and $ \\alpha \\in (0,1)$. Also note that,\n\\allowdisplaybreaks{\\begin{align}\n 2 \\mu_1 \\gamma \\ = \\ 2 \\mu_1 \\sqrt{\\eta \\zeta \\mu_1} \\ > \\ 2 \\mu_1 \\sqrt{\\frac{\\zeta (1-\\mu_1)^2 \\zeta}{4}} = \\mu_1 \\zeta(1-\\mu_1),\n\\end{align}}\nwhich implies $\\omega < 1$. Similarly, whenever $\\mu_1 < 1$ we have\n\\begin{align}\n    2\\gamma - \\zeta - \\frac{1}{\\mu_1} < 2\\sqrt{\\eta \\mu_1 \\zeta} - \\zeta - 1 \\leq  2 \\sqrt{\\zeta} - \\zeta - 1  = -(\\sqrt{\\zeta}-1)^2 \\leq 0,\n\\end{align}\nwhich implies $\\omega > 0$. Also, using the parameter choice of \\eqref{eq:askm12}, we have, \n\\begin{align}\n\\label{eq:askm13}\n  1-\\omega+ \\frac{\\gamma^2 }{\\eta}  - 2\\gamma \\mu_1 - & \\frac{\\gamma \\omega \\mu_1 (1-\\alpha)}{\\alpha}   = 1-\\omega + \\zeta \\mu_1 - 2\\gamma \\mu_1 - \\omega \\zeta \\mu_1^2 \\nonumber \\\\\n  & = 1- 2\\gamma \\mu_1 + \\zeta \\mu_1 - \\omega (1+\\zeta \\mu_1^2) = 0.\n\\end{align}\nNow, using all of the above relations (equation \\eqref{eq:askm12}, \\eqref{eq:askm13}) in equation \\eqref{eq:askm11}, we get the following:\n\\begin{align}\n\\label{eq:askm14}\n  \\E \\left[ d(v_{k+1},P)^2 + \\frac{\\gamma^2}{\\eta} d(x_{k+1},P)^2 \\right] &  \\ \\leq \\ \\omega \\Big[d(v_{k},P)^2 + \\underbrace{\\frac{\\gamma (1-\\alpha)}{\\alpha}}_{= \\frac{\\gamma^2 }{\\eta}} \\ d(x_{k},P)^2 \\Big] \\nonumber \\\\\n  + & \\Big [\\underbrace{1-\\omega+ \\frac{\\gamma^2 }{\\eta}  - 2\\gamma \\mu_1 - \\frac{\\gamma \\omega \\mu_1 (1-\\alpha)}{\\alpha} }_{= \\ 0} \\Big ] \\ d(y_k,P)^2 \\nonumber \\\\\n   = \\ & \\omega \\ \\left[ d(v_{k},P)^2+ \\frac{\\gamma^2}{\\eta} d(x_{k},P)^2 \\right].\n\\end{align}\nFinally, taking expectation again with tower rule and substituting $\\frac{\\gamma^2 }{\\eta}= \\zeta \\mu_1 $ we have,\n\\begin{align*}\n\\E \\left[ d(v_{k+1},P)^2 + \\zeta \\mu_1 \\  d(x_{k+1},P)^2 \\right] \\ & \\leq \\ \\omega^{k+1} \\ \\E \\left[ d(v_{0},P)^2+ \\zeta \\mu_1 \\  d(x_{0},P)^2 \\right] \\\\\n& = (1+\\zeta \\mu_1) \\ \\omega^{k+1} \\ d(x_{0},P)^2.\n\\end{align*}\nThis proves the Theorem. Furthermore, for faster convergence, we need to choose parameters such that, $\\omega$ becomes as small as possible. In the proof, we assumed $\\mu_1 <1$ holds which is the most probable scenario. Whenever $\\mu_1 = 1$, we must have $\\mu_1 = \\mu_2 =1$ and Lemma 6 holds with both equality, i.e., $f(x) = d(x,P)^2$. Therefore if we choose $\\alpha, \\gamma, \\omega$ as $\\alpha = \\frac{\\eta}{\\eta+ \\gamma}, \\ \\gamma = \\sqrt{\\zeta \\eta}, \\ \\omega = 1- \\frac{2\\gamma}{1+\\zeta}$, we can check that condition (89) holds and $0 < \\omega < 1$ holds for any $\\zeta > 0$.\n\n\n\n\n\n\n\n","start":156521,"end":163768}]},{"id":"S043","semantic_type":"assumption","kind":"assumption","labels":["th:10"],"declared_external":false,"source_only":false,"title":"目标定理的参数假设","source_units":[{"raw_tex":"Let, $v_{k+1}$ and $x_{k+1}$ are generated by Algorithm \\ref{alg:paskm}. If we select the parameters $\\omega, \\ \\gamma, \\ \\alpha$ as\n\\begin{align*}\n\\omega = 1-\\frac{\\zeta \\mu_1^2+2\\gamma \\mu_1-\\zeta \\mu_1}{1+\\zeta \\mu_1^2} , \\ \\ \\gamma = \\sqrt{ \\zeta \\eta \\mu_1}, \\ \\ \\alpha = \\frac{ \\eta}{ \\eta + \\gamma },\n\\end{align*}\nwhere, $\\zeta$ is chosen as $ 0 < \\zeta  < \\frac{4\\eta \\mu_1}{(1-\\mu_1)^2} $ if $\\mu_1 < 1$, otherwise choose any $\\zeta > 0$. Then, for any $0 < \\delta < 2$, the sequence of iterates $\\{v_k\\}, \\ \\{x_k\\}$ converges and the following result holds:\n","start":78353,"end":78921}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":["eq:askm4"],"declared_external":false,"source_only":false,"title":"可行组合点与三项展开","source_units":[{"raw_tex":"\\paragraph{Proof of Theorem \\ref{th:10}}\n\nLet us define, $\\mathcal{V} = \\omega \\mathcal{P}(v_{k}) +(1-\\omega) \\mathcal{P}(y_{k}) $. Since $\\mathcal{V} \\in P$, using the update formula of $v_{k+1}$ from equation \\eqref{eq:askm2}, we have,\n\\allowdisplaybreaks{\n\\begin{align}\n\\label{eq:askm4}\n    d(v_{k+1},P)^2 & = \\|v_{k+1} - \\mathcal{P}(v_{k+1})\\|^2 \\overset{\\text{Lemma} \\ \\ref{lem:distance}}{\\leq}  \\|v_{k+1} - \\mathcal{V}\\|^2  \\nonumber \\\\\n    & \\overset{\\eqref{eq:askm2}}{=}  \\big \\|\\omega v_k + (1-\\omega) y_k- \\mathcal{V} -\\gamma (a_{i^*}^Ty_k- b_{i^*})^+ a_{i^*} \\big \\|^2 \\nonumber \\\\\n    & =   \\underbrace{ \\|\\omega v_k + (1-\\omega) y_k- \\mathcal{V} \\|^2}_{I_1} + \\gamma^2 \\underbrace{\\|(a_{i^*}^Ty_k- b_{i^*})^+a_{i^*}\\|^2}_{I_2} \\nonumber \\\\\n    & - 2 \\gamma  \\underbrace{\\big \\langle \\omega v_k + (1-\\omega) y_k- \\mathcal{V} \\ ,\\  a_{i^*} (a_{i^*}^Ty_k-b_{i^{*}})^{+} \\big \\rangle }_{I_3} \\nonumber \\\\\n    & = I_1 + \\gamma^2 I_2 -2 \\gamma I_3.\n\\end{align}}\n","start":156521,"end":157490}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["eq:askm5"],"declared_external":false,"source_only":false,"title":"凸性控制混合误差","source_units":[{"raw_tex":"Since $\\|\\cdot\\|^{2}$ is a convex function and $0 < \\omega < 1$, we can bound the expected first term as follows,\n\\begin{align}\n\\label{eq:askm5}\n    \\E_{\\mathbb{S}_k} [I_1]  &= \\E_{\\mathbb{S}_k} \\left[\\|\\omega v_k + (1-\\omega) y_k-\\mathcal{V}\\|^2\\right] \\nonumber \\\\\n    & = \\E_{\\mathbb{S}_k} \\left[\\|\\omega v_k + (1-\\omega) y_k- \\omega \\mathcal{P}(v_{k}) - (1-\\omega) \\mathcal{P}(y_{k})\\|^2\\right] \\nonumber \\\\\n    & \\leq \\omega \\| v_k-\\mathcal{P}(v_{k})\\|^2 + (1-\\omega) \\|y_k-\\mathcal{P}(y_{k})\\|^2\\nonumber \\\\\n    & =  \\omega \\ d(v_k,P)^2 + (1-\\omega) \\ d(y_k,P)^2.\n\\end{align}\n","start":157490,"end":158072}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["eq:askm6"],"declared_external":false,"source_only":false,"title":"残差项换成距离下降","source_units":[{"raw_tex":"Taking expectation with respect to the sampling distribution in the second term of equation \\eqref{eq:askm4} and using Lemma \\ref{lem4} with the choice $z = x_{k+1}, \\ x = y_k$ and $\\eta = 2\\delta-\\delta^2$, we get,\n\\begin{align}\n\\label{eq:askm6}\n   \\gamma^2 \\E_{\\mathbb{S}_k} \\left[\\|(a_{i^*}^Ty_k- b_{i^*})^+ a_{i^*}\\|^2\\right] & \\ = \\  \\gamma^2 \\E_{\\mathbb{S}_k} \\left[|(a_{i^*}^Ty_k- b_{i^*})^+ |^2\\right] \\nonumber \\\\\n    \\overset{\\text{Lemma} \\ \\ref{lem4}}{\\leq} & \\ \\frac{\\gamma^2 }{\\eta} \\left[d(y_k,P)^2 - \\E \\left[ d(x_{k+1},P)^2 \\right]\\right].\n\\end{align}\n","start":158072,"end":158640}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":["eq:askm9"],"declared_external":false,"source_only":false,"title":"交叉项的梯度改写","source_units":[{"raw_tex":"\nNow, taking expectation in the third term of \\eqref{eq:askm4} we get,\n\\begin{align}\n\\label{eq:askm9}\n    - 2  \\gamma  \\E_{\\mathbb{S}_k} & [I_3 ]  = - 2 \\gamma \\big \\langle \\omega v_k + (1-\\omega) y_k-\\mathcal{V}, \\E_{\\mathbb{S}_k} \\left[ a_{i^*} (a_{i^*}^Ty_k-b_{i^*})^{+}\\right]  \\big \\rangle \\nonumber \\\\\n    & \\overset{\\eqref{eq:yk} \\ \\& \\ \\eqref{def:function} }{=} - 2 \\gamma \\big \\langle \\frac{\\omega}{\\alpha} \\left[y_k - (1-\\alpha) x_k\\right] + (1-\\omega) y_k-\\mathcal{V},  \\nabla f(y_k)  \\big \\rangle \\nonumber \\\\\n    & = - 2 \\gamma \\big \\langle \\frac{\\omega(1-\\alpha)}{\\alpha}(y_k-x_k)+ y_k-\\mathcal{V}, \\nabla f(y_k)  \\big \\rangle.\n\\end{align}\n","start":158640,"end":159294}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":["eq:askm10"],"declared_external":false,"source_only":false,"title":"两种内积界控制交叉项","source_units":[{"raw_tex":"Using Lemma \\ref{lem:grad} and Lemma \\ref{lem:grad1} we can simplify equation \\eqref{eq:askm9} as follows,\n\\begin{align}\n\\label{eq:askm10}\n  & \\ -  2 \\gamma  \\E_{\\mathbb{S}_k}  [I_3 ]  =   - 2 \\gamma \\big \\langle \\frac{\\omega(1-\\alpha)}{\\alpha}(y_k-x_k)+ y_k-\\mathcal{V}, \\nabla f(y_k)  \\big \\rangle \\nonumber \\\\\n    & \\quad =    2 \\gamma \\frac{\\omega(1-\\alpha)}{\\alpha} \\big \\langle x_k-y_k, \\nabla f(y_k)  \\big \\rangle + 2 \\gamma \\big \\langle \\mathcal{V}-y_k, \\nabla f(y_k)  \\big \\rangle \\nonumber \\\\\n  & \\overset{\\text{Lemma} \\ \\ref{lem:grad} \\ \\& \\ \\ref{lem:grad1}}{\\leq} \\frac{\\gamma \\omega (1-\\alpha)}{\\alpha} \\  d(x_k,P)^2 - \\frac{ \\mu_1 \\gamma  \\omega (1-\\alpha)}{\\alpha} \\ d(y_k,P)^2 - 2 \\mu_1 \\gamma \\ d(y_k,P)^2.\n\\end{align}\n","start":159294,"end":160030}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"合并三项距离估计","source_units":[{"raw_tex":"Now, substituting the values of equation \\eqref{eq:askm5}, \\eqref{eq:askm6} \\& \\eqref{eq:askm10} in equation \\eqref{eq:askm4} we get the following:\n\\allowdisplaybreaks{\n\\begin{align*}\n \\E [ & d(v_{k+1} ,P)^2  ]  =  I_1 + \\gamma^2 \\E_{\\mathbb{S}_k}[I_2 ]- 2 \\gamma \\E_{\\mathbb{S}_k}[I_3] \\nonumber \\\\\n & = \\omega \\ d(v_{k},P)^2 + (1-\\omega) \\ d(y_k,P)^2 + \\frac{\\gamma^2 }{\\eta} \\left\\{d(y_k,P)^2 - \\E \\left[ d(x_{k+1},P)^2 \\right]\\right\\} \\nonumber \\\\\n & \\quad + \\frac{\\gamma \\omega (1-\\alpha)}{\\alpha}  d(x_k,P)^2 - \\gamma \\mu_1 \\left(2+\\frac{ \\omega (1-\\alpha)}{\\alpha}\\right)  d(y_k,P)^2.\n\\end{align*}}\n","start":160030,"end":160636}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":["eq:askm11"],"declared_external":false,"source_only":false,"title":"加权能量的一步关系","source_units":[{"raw_tex":"With further simplification, the above identity can be written as follows:\n\\allowdisplaybreaks{\n\\begin{align}\n\\label{eq:askm11}\n & \\E \\left[ d(v_{k+1},P)^2 + \\frac{\\gamma^2}{\\eta} d(x_{k+1},P)^2 \\right]  = \\omega \\ \\left[d(v_{k},P)^2 + \\frac{\\gamma (1-\\alpha)}{\\alpha} d(x_{k},P)^2 \\right] \\nonumber \\\\\n & \\quad \\quad \\quad \\quad \\quad \\quad \\quad + d(y_k,P)^2 \\ \\left\\{1-\\omega+ \\frac{\\gamma^2 }{\\eta}  - 2\\gamma \\mu_1 - \\frac{\\gamma \\omega \\mu_1 (1-\\alpha)}{\\alpha}  \\right\\}.\n\\end{align}}\n","start":160636,"end":161128}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"匹配新旧能量权重","source_units":[{"raw_tex":"Now, let's choose the parameters as in equation \\eqref{eq:askm12} along with $0 < \\zeta  < \\frac{4\\eta \\mu_1}{(1-\\mu_1)^2} $. We can easily see that $\\frac{\\gamma^2 }{\\eta} = \\frac{ \\gamma(1-\\alpha)}{\\alpha}$ and $ \\alpha \\in (0,1)$. Also note that,\n","start":161128,"end":161378}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"验证速率小于一","source_units":[{"raw_tex":"\\allowdisplaybreaks{\\begin{align}\n 2 \\mu_1 \\gamma \\ = \\ 2 \\mu_1 \\sqrt{\\eta \\zeta \\mu_1} \\ > \\ 2 \\mu_1 \\sqrt{\\frac{\\zeta (1-\\mu_1)^2 \\zeta}{4}} = \\mu_1 \\zeta(1-\\mu_1),\n\\end{align}}\nwhich implies $\\omega < 1$. Similarly, whenever $\\mu_1 < 1$ we have\n","start":161378,"end":161626}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"验证速率为正","source_units":[{"raw_tex":"which implies $\\omega < 1$. Similarly, whenever $\\mu_1 < 1$ we have\n\\begin{align}\n    2\\gamma - \\zeta - \\frac{1}{\\mu_1} < 2\\sqrt{\\eta \\mu_1 \\zeta} - \\zeta - 1 \\leq  2 \\sqrt{\\zeta} - \\zeta - 1  = -(\\sqrt{\\zeta}-1)^2 \\leq 0,\n\\end{align}\nwhich implies $\\omega > 0$. Also, using the parameter choice of \\eqref{eq:askm12}, we have, \n","start":161558,"end":161886}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":["eq:askm13"],"declared_external":false,"source_only":false,"title":"消去中间点误差系数","source_units":[{"raw_tex":"which implies $\\omega > 0$. Also, using the parameter choice of \\eqref{eq:askm12}, we have, \n\\begin{align}\n\\label{eq:askm13}\n  1-\\omega+ \\frac{\\gamma^2 }{\\eta}  - 2\\gamma \\mu_1 - & \\frac{\\gamma \\omega \\mu_1 (1-\\alpha)}{\\alpha}   = 1-\\omega + \\zeta \\mu_1 - 2\\gamma \\mu_1 - \\omega \\zeta \\mu_1^2 \\nonumber \\\\\n  & = 1- 2\\gamma \\mu_1 + \\zeta \\mu_1 - \\omega (1+\\zeta \\mu_1^2) = 0.\n\\end{align}\n","start":161793,"end":162180}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["eq:askm14"],"declared_external":false,"source_only":false,"title":"得到加权能量收缩","source_units":[{"raw_tex":"Now, using all of the above relations (equation \\eqref{eq:askm12}, \\eqref{eq:askm13}) in equation \\eqref{eq:askm11}, we get the following:\n\\begin{align}\n\\label{eq:askm14}\n  \\E \\left[ d(v_{k+1},P)^2 + \\frac{\\gamma^2}{\\eta} d(x_{k+1},P)^2 \\right] &  \\ \\leq \\ \\omega \\Big[d(v_{k},P)^2 + \\underbrace{\\frac{\\gamma (1-\\alpha)}{\\alpha}}_{= \\frac{\\gamma^2 }{\\eta}} \\ d(x_{k},P)^2 \\Big] \\nonumber \\\\\n  + & \\Big [\\underbrace{1-\\omega+ \\frac{\\gamma^2 }{\\eta}  - 2\\gamma \\mu_1 - \\frac{\\gamma \\omega \\mu_1 (1-\\alpha)}{\\alpha} }_{= \\ 0} \\Big ] \\ d(y_k,P)^2 \\nonumber \\\\\n   = \\ & \\omega \\ \\left[ d(v_{k},P)^2+ \\frac{\\gamma^2}{\\eta} d(x_{k},P)^2 \\right].\n\\end{align}\n","start":162180,"end":162831}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"塔式取期望并迭代","source_units":[{"raw_tex":"Finally, taking expectation again with tower rule and substituting $\\frac{\\gamma^2 }{\\eta}= \\zeta \\mu_1 $ we have,\n\\begin{align*}\n\\E \\left[ d(v_{k+1},P)^2 + \\zeta \\mu_1 \\  d(x_{k+1},P)^2 \\right] \\ & \\leq \\ \\omega^{k+1} \\ \\E \\left[ d(v_{0},P)^2+ \\zeta \\mu_1 \\  d(x_{0},P)^2 \\right] \\\\\n& = (1+\\zeta \\mu_1) \\ \\omega^{k+1} \\ d(x_{0},P)^2.\n\\end{align*}\n","start":162831,"end":163179}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"μ一等于一的边界讨论","source_units":[{"raw_tex":"This proves the Theorem. Furthermore, for faster convergence, we need to choose parameters such that, $\\omega$ becomes as small as possible. In the proof, we assumed $\\mu_1 <1$ holds which is the most probable scenario. Whenever $\\mu_1 = 1$, we must have $\\mu_1 = \\mu_2 =1$ and Lemma 6 holds with both equality, i.e., $f(x) = d(x,P)^2$. Therefore if we choose $\\alpha, \\gamma, \\omega$ as $\\alpha = \\frac{\\eta}{\\eta+ \\gamma}, \\ \\gamma = \\sqrt{\\zeta \\eta}, \\ \\omega = 1- \\frac{2\\gamma}{1+\\zeta}$, we can check that condition (89) holds and $0 < \\omega < 1$ holds for any $\\zeta > 0$.\n\n\n\n\n\n\n\n","start":163179,"end":163768}]},{"id":"S058","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单：Jensen's inequality、convexity of the largest eigenvalue及elementary spectral bounds。"],"source_only":false,"title":"谱凸性与半正定运算","source_units":[]},{"id":"S059","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单：standard Euclidean projection properties、convexity of the feasible region及Jensen's inequality。"],"source_only":false,"title":"凸可行域与平方范数","source_units":[]},{"id":"S060","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单：convex first-order inequalities；次梯度表述仅用于说明该标准事实的适用条件。"],"source_only":false,"title":"凸函数一阶不等式","source_units":[]},{"id":"S061","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单：tower property of expectation；作者在行2250明确使用tower rule。"],"source_only":false,"title":"期望塔式法则","source_units":[]},{"id":"S062","semantic_type":"interpretation","kind":"interpretation","labels":["th:10"],"declared_external":false,"source_only":false,"title":"线性速率与复杂度解释","source_units":[{"raw_tex":"This theorem implies that the PASKM algorithm converges linearly with a rate of $\\omega$, which accumulates to a total of $\\mathcal{O}(\\frac{1+\\zeta \\mu_1^2}{\\zeta \\mu_1^2+2\\gamma \\mu_1-\\zeta \\mu_1}\\log{1/\\epsilon})$ iterations to bring the given error below $\\epsilon > 0$.\n","start":79156,"end":79431}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S041","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S004","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S009","target_id":"S008","kind":"supports"},{"source_id":"S010","target_id":"S008","kind":"supports"},{"source_id":"S011","target_id":"S008","kind":"supports"},{"source_id":"S004","target_id":"S011","kind":"supports"},{"source_id":"S058","target_id":"S018","kind":"supports"},{"source_id":"S058","target_id":"S019","kind":"supports"},{"source_id":"S003","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S004","target_id":"S022","kind":"supports"},{"source_id":"S005","target_id":"S022","kind":"supports"},{"source_id":"S058","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S005","target_id":"S025","kind":"supports"},{"source_id":"S013","target_id":"S025","kind":"supports"},{"source_id":"S012","target_id":"S025","kind":"supports"},{"source_id":"S004","target_id":"S025","kind":"supports"},{"source_id":"S002","target_id":"S025","kind":"supports"},{"source_id":"S005","target_id":"S026","kind":"supports"},{"source_id":"S016","target_id":"S026","kind":"supports"},{"source_id":"S020","target_id":"S026","kind":"supports"},{"source_id":"S002","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S007","target_id":"S027","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S027","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S007","target_id":"S030","kind":"supports"},{"source_id":"S060","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S007","target_id":"S031","kind":"supports"},{"source_id":"S023","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S002","target_id":"S035","kind":"supports"},{"source_id":"S007","target_id":"S035","kind":"supports"},{"source_id":"S023","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S014","target_id":"S038","kind":"supports"},{"source_id":"S002","target_id":"S038","kind":"supports"},{"source_id":"S007","target_id":"S038","kind":"supports"},{"source_id":"S003","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S032","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S023","target_id":"S040","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S037","kind":"supports"},{"source_id":"S040","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S008","target_id":"S043","kind":"supports"},{"source_id":"S023","target_id":"S043","kind":"supports"},{"source_id":"S036","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S051","kind":"supports"},{"source_id":"S015","target_id":"S051","kind":"supports"},{"source_id":"S043","target_id":"S052","kind":"supports"},{"source_id":"S023","target_id":"S052","kind":"supports"},{"source_id":"S043","target_id":"S053","kind":"supports"},{"source_id":"S023","target_id":"S053","kind":"supports"},{"source_id":"S036","target_id":"S053","kind":"supports"},{"source_id":"S011","target_id":"S044","kind":"supports"},{"source_id":"S014","target_id":"S044","kind":"supports"},{"source_id":"S002","target_id":"S044","kind":"supports"},{"source_id":"S059","target_id":"S044","kind":"supports"},{"source_id":"S052","target_id":"S044","kind":"supports"},{"source_id":"S053","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S059","target_id":"S045","kind":"supports"},{"source_id":"S052","target_id":"S045","kind":"supports"},{"source_id":"S053","target_id":"S045","kind":"supports"},{"source_id":"S036","target_id":"S046","kind":"supports"},{"source_id":"S039","target_id":"S046","kind":"supports"},{"source_id":"S011","target_id":"S046","kind":"supports"},{"source_id":"S003","target_id":"S046","kind":"supports"},{"source_id":"S043","target_id":"S046","kind":"supports"},{"source_id":"S010","target_id":"S047","kind":"supports"},{"source_id":"S007","target_id":"S047","kind":"supports"},{"source_id":"S044","target_id":"S047","kind":"supports"},{"source_id":"S051","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S028","target_id":"S048","kind":"supports"},{"source_id":"S032","target_id":"S048","kind":"supports"},{"source_id":"S023","target_id":"S048","kind":"supports"},{"source_id":"S044","target_id":"S048","kind":"supports"},{"source_id":"S051","target_id":"S048","kind":"supports"},{"source_id":"S053","target_id":"S048","kind":"supports"},{"source_id":"S052","target_id":"S048","kind":"supports"},{"source_id":"S044","target_id":"S049","kind":"supports"},{"source_id":"S045","target_id":"S049","kind":"supports"},{"source_id":"S046","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S043","target_id":"S054","kind":"supports"},{"source_id":"S051","target_id":"S054","kind":"supports"},{"source_id":"S015","target_id":"S054","kind":"supports"},{"source_id":"S050","target_id":"S055","kind":"supports"},{"source_id":"S051","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S061","target_id":"S056","kind":"supports"},{"source_id":"S043","target_id":"S056","kind":"supports"},{"source_id":"S009","target_id":"S056","kind":"supports"},{"source_id":"S043","target_id":"S057","kind":"supports"},{"source_id":"S023","target_id":"S057","kind":"supports"},{"source_id":"S044","target_id":"S042","kind":"supports"},{"source_id":"S045","target_id":"S042","kind":"supports"},{"source_id":"S046","target_id":"S042","kind":"supports"},{"source_id":"S047","target_id":"S042","kind":"supports"},{"source_id":"S048","target_id":"S042","kind":"supports"},{"source_id":"S049","target_id":"S042","kind":"supports"},{"source_id":"S050","target_id":"S042","kind":"supports"},{"source_id":"S051","target_id":"S042","kind":"supports"},{"source_id":"S052","target_id":"S042","kind":"supports"},{"source_id":"S053","target_id":"S042","kind":"supports"},{"source_id":"S054","target_id":"S042","kind":"supports"},{"source_id":"S055","target_id":"S042","kind":"supports"},{"source_id":"S056","target_id":"S042","kind":"supports"},{"source_id":"S057","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S062","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S041","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S004","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S009","target_id":"S008","kind":"supports"},{"source_id":"S010","target_id":"S008","kind":"supports"},{"source_id":"S011","target_id":"S008","kind":"supports"},{"source_id":"S004","target_id":"S011","kind":"supports"},{"source_id":"S058","target_id":"S018","kind":"supports"},{"source_id":"S058","target_id":"S019","kind":"supports"},{"source_id":"S003","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S004","target_id":"S022","kind":"supports"},{"source_id":"S005","target_id":"S022","kind":"supports"},{"source_id":"S058","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S005","target_id":"S025","kind":"supports"},{"source_id":"S013","target_id":"S025","kind":"supports"},{"source_id":"S012","target_id":"S025","kind":"supports"},{"source_id":"S004","target_id":"S025","kind":"supports"},{"source_id":"S002","target_id":"S025","kind":"supports"},{"source_id":"S005","target_id":"S026","kind":"supports"},{"source_id":"S016","target_id":"S026","kind":"supports"},{"source_id":"S020","target_id":"S026","kind":"supports"},{"source_id":"S002","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S007","target_id":"S027","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S027","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S007","target_id":"S030","kind":"supports"},{"source_id":"S060","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S007","target_id":"S031","kind":"supports"},{"source_id":"S023","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S002","target_id":"S035","kind":"supports"},{"source_id":"S007","target_id":"S035","kind":"supports"},{"source_id":"S023","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S014","target_id":"S038","kind":"supports"},{"source_id":"S002","target_id":"S038","kind":"supports"},{"source_id":"S007","target_id":"S038","kind":"supports"},{"source_id":"S003","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S032","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S023","target_id":"S040","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S037","kind":"supports"},{"source_id":"S040","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S008","target_id":"S043","kind":"supports"},{"source_id":"S023","target_id":"S043","kind":"supports"},{"source_id":"S036","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S051","kind":"supports"},{"source_id":"S015","target_id":"S051","kind":"supports"},{"source_id":"S043","target_id":"S052","kind":"supports"},{"source_id":"S023","target_id":"S052","kind":"supports"},{"source_id":"S043","target_id":"S053","kind":"supports"},{"source_id":"S023","target_id":"S053","kind":"supports"},{"source_id":"S036","target_id":"S053","kind":"supports"},{"source_id":"S011","target_id":"S044","kind":"supports"},{"source_id":"S014","target_id":"S044","kind":"supports"},{"source_id":"S002","target_id":"S044","kind":"supports"},{"source_id":"S059","target_id":"S044","kind":"supports"},{"source_id":"S052","target_id":"S044","kind":"supports"},{"source_id":"S053","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S059","target_id":"S045","kind":"supports"},{"source_id":"S052","target_id":"S045","kind":"supports"},{"source_id":"S053","target_id":"S045","kind":"supports"},{"source_id":"S036","target_id":"S046","kind":"supports"},{"source_id":"S039","target_id":"S046","kind":"supports"},{"source_id":"S011","target_id":"S046","kind":"supports"},{"source_id":"S003","target_id":"S046","kind":"supports"},{"source_id":"S043","target_id":"S046","kind":"supports"},{"source_id":"S010","target_id":"S047","kind":"supports"},{"source_id":"S007","target_id":"S047","kind":"supports"},{"source_id":"S044","target_id":"S047","kind":"supports"},{"source_id":"S051","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S028","target_id":"S048","kind":"supports"},{"source_id":"S032","target_id":"S048","kind":"supports"},{"source_id":"S023","target_id":"S048","kind":"supports"},{"source_id":"S044","target_id":"S048","kind":"supports"},{"source_id":"S051","target_id":"S048","kind":"supports"},{"source_id":"S053","target_id":"S048","kind":"supports"},{"source_id":"S052","target_id":"S048","kind":"supports"},{"source_id":"S044","target_id":"S049","kind":"supports"},{"source_id":"S045","target_id":"S049","kind":"supports"},{"source_id":"S046","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S043","target_id":"S054","kind":"supports"},{"source_id":"S051","target_id":"S054","kind":"supports"},{"source_id":"S015","target_id":"S054","kind":"supports"},{"source_id":"S050","target_id":"S055","kind":"supports"},{"source_id":"S051","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S061","target_id":"S056","kind":"supports"},{"source_id":"S043","target_id":"S056","kind":"supports"},{"source_id":"S009","target_id":"S056","kind":"supports"},{"source_id":"S043","target_id":"S057","kind":"supports"},{"source_id":"S023","target_id":"S057","kind":"supports"},{"source_id":"S044","target_id":"S042","kind":"supports"},{"source_id":"S045","target_id":"S042","kind":"supports"},{"source_id":"S046","target_id":"S042","kind":"supports"},{"source_id":"S047","target_id":"S042","kind":"supports"},{"source_id":"S048","target_id":"S042","kind":"supports"},{"source_id":"S049","target_id":"S042","kind":"supports"},{"source_id":"S050","target_id":"S042","kind":"supports"},{"source_id":"S051","target_id":"S042","kind":"supports"},{"source_id":"S052","target_id":"S042","kind":"supports"},{"source_id":"S053","target_id":"S042","kind":"supports"},{"source_id":"S054","target_id":"S042","kind":"supports"},{"source_id":"S055","target_id":"S042","kind":"supports"},{"source_id":"S056","target_id":"S042","kind":"supports"},{"source_id":"S057","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:1"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"线性可行性问题","source_units":[{"raw_tex":"We consider the following \\textit{Linear Feasibility} (LF) problem:\n\\begin{align}\n\\label{eq:1}\nAx \\leq b, \\ \\ b \\in \\R^m, \\ A \\in \\R^{m\\times n}.\n\\end{align}\n","start":4861,"end":5019}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"可行域与距离记号","source_units":[{"raw_tex":"\\subsection{Notation}\n\nWe follow the standard linear algebra notation in this work. $\\mathbb{R}^n$ denotes the $n$ dimensional real space, $\\mathbb{R}^{m\\times n}$ denotes the set of $m \\times n$ real-valued matrices. For any matrix $A\\in \\mathbb{R}^{m\\times n}$, $A^T$ denotes the transpose matrix $A$ and $a_i^T$ for $i = 1,2,..,m$ denotes the rows of matrix $A$. Furthermore, $P = \\{x \\in \\R^n | \\ Ax \\leq b\\}$ denotes the feasible region of the feasibility problem and $\\mathcal{P}(x)$ denotes the projection of $x \\in \\R^n $ onto the feasible region $P$. The notation $d(x,P)$ denotes the distance between $x \\in \\R^n $ and the feasible region $P$, i.e., $d(x,P) \\ = \\ \\inf_{z \\in P} \\|x-z\\| \\ = \\ \\|x-\\mathcal{P}(x)\\|$. For any matrix $A$, the spectral norm and Frobenius norm are denoted by $\\|A\\|$  and $\\|A\\|_F$, respectively. For any function $f:X\\mapsto Y$, we use $\\nabla f$ to represent the gradient of $f$. Finally, $\\langle x,y\\rangle = x^Ty$ denotes the standard inner product and $\\|x\\| = \\sqrt{\\langle x, x \\rangle}$ as the euclidean ($L_2$) norm. The notation $x^+$ denotes the positive part of any real number (ie., $x^+ = \\max \\{x,0\\}$). For any two arbitrary matrices $M,\\ N$, the notation $M \\succ N$ implies the positive definiteness of the matrix $M-N$. The notation $\\E_\\mathbb{S}[\\cdot]$ is used to denote the expectation with respect to the sampling distribution $\\mathbb{S}$. %Throughout the paper, we will frequently use the  tower property of expectation.\n","start":20813,"end":22300}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"一致性与行归一化","source_units":[{"raw_tex":"\\subsection{Assumptions}\nThroughout the paper, we assume that the system $Ax \\leq b$ is consistent and the matrix $A$ has no zero rows. We also assumed that the rows of matrix $A$ are normalized (i.e., $\\|a_i\\|^2=1$ for all $i$). Note that, normalization simplifies the convergence analysis considerably. The normalization doesn't impact the computational time significantly (we could simply normalize each row for the first time it occurs during the computation). Moreover, normalization simplifies the convergence analysis considerably. In the description of algorithms, we do not enforce the assumption. Furthermore, it can be noted that the proposed algorithms generate the same iterates irrespective of normalization. \n","start":22549,"end":23273}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["def:i1","eq:sampling"],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"采样与残差排序","source_units":[{"raw_tex":"\\subsection{Expectation}\nFor the convergence analysis of Algorithm \\ref{alg:skm} and its variations (any algorithm that uses that specific type of sampling distribution), we need to discuss a specific expectation calculation. First of all, let us sort the residual vector $(Ax-b)^+$ from smallest to largest for any iterate $x$ and denote $(Ax-b)^+_{\\underline{\\mathbf{i_j}}}$ as the $(\\beta+j)^{th}$ entry on the sorted list \\footnote{We use the notation $(Ax-b)^+_{\\underline{\\mathbf{i_j}}}$ throughout the paper to express the underlying expectation, where the indices $\\underline{\\mathbf{i_j}}$ represent the sampling process of \\ref{eq:sampling}.}, i.e.,\n\\begin{align}\n\\label{eq:sampling}\n  \\underbrace{(Ax-b)^+_{\\underline{\\mathbf{i_0}}}}_{\\beta^{th}} \\ \\leq ... \\leq \\ \\underbrace{(Ax-b)^+_{\\underline{\\mathbf{i_j}}}}_{(\\beta+j)^{th}} \\ \\leq ... \\leq \\ \\underbrace{(Ax-b)^+_{\\underline{\\mathbf{i_{m-\\beta}}}}}_{m^{th}}.\n\\end{align}\n%Let's denote, the set $T(x)$\\footnote{$|T(x)| = |T(x_k)| = m-\\beta +1$. The notation $T(x_k)$ will be used to denote the corresponding set for any random iterate $x_k \\in \\R^n$.} as the collection of all indices from the list given in \\eqref{eq:sampling}.\nNow, consider the list with all of the entries of the residual vector $(Ax-b)^+$, then we need to calculate the probability that particular entry of the residual vector is selected at any given iteration. Note that, the probability that any sample is selected is $\\frac{1}{\\binom{m}{\\beta}}$ and each sample has an equal probability of selection. Another intersecting fact can be noted that the size of the residual list controls the order and frequency that each entry of the residual vector will be expected to be selected. From now on, we will denote this specific choice of sampling distribution as $\\mathbb{S}$ for any point $x \\in \\R^n$ \\footnote{For ease of notation, throughout the paper, we will use $\\mathbb{S}_k$ to denote the sampling distribution corresponding to any random iterate $x_k \\in \\R^n$ }. To calculate the resulting expectation with respect to the above-mentioned sampling distribution, let us first denote, $\\tau \\sim \\mathbb{S}$ as the set of sampled $\\beta$ constraints and $i^{*}$ as \\footnote{ Similarly, we will use $\\tau_k \\sim \\mathbb{S}_k$ to denote the sampled set and $i^* = \\argmax_{i \\in \\tau_k \\sim \\mathbb{S}_k} \\{a_i^Tx_k-b_i, 0\\} \\ = \\ \\argmax_{i \\in \\tau_k \\sim \\mathbb{S}_k} (A_{\\tau_k}x_k-b_{\\tau_k})^+_i$ for any iterate $x_k \\in \\R^n$.}\n\\begin{align}\n\\label{def:i1}\ni^* = \\argmax_{i \\in \\tau \\sim \\mathbb{S}} \\{a_i^Tx-b_i, 0\\} \\ = \\ \\argmax_{i} (A_{\\tau}x-b_{\\tau})^+_i,\n\\end{align}\nwhere, $A_{\\tau}$ denotes the collection of rows of $A$ restricted to the index set $\\tau$ and $(A_{\\tau}x-b_{\\tau})_i$ denotes the $i^{th}$ entry of $A_{\\tau}x-b_{\\tau}$. Using the above discussion with the list provided in equation \\eqref{eq:sampling}, we have the following:\n","start":25742,"end":28646}]},{"id":"S005","semantic_type":"derived_fact","kind":"derived_fact","labels":["def:exp"],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"选中残差的期望公式","source_units":[{"raw_tex":"{\\allowdisplaybreaks\n\\begin{align}\n\\label{def:exp}\n\\E_{\\mathbb{S}} \\left[\\big |(a_{i^*}^Tx-b_{i^*})^{+}\\big |^2\\right] = \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} \\big | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}} \\big |^2, \\end{align}}\nwhere, $\\E_{\\mathbb{S}}$ denotes the required expectation corresponding to the sampling distribution $\\mathbb{S}$. The above expectation calculation was first used by De Loera \\textit{et.al} in their work \\cite{haddock:2017} where they first introduced the SKM method.\n","start":28646,"end":29188}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["def:function_i"],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"单行残差函数","source_units":[{"raw_tex":"\\subsection{Function $f(x)$}\nIn this section, we formalize the definition of function $f:\\R^n \\rightarrow \\R$. Throughout section \\ref{sec:conv}, we will use the properties of function $f(x)$ \\footnote{Similar type of functions with uniform sampling have been studied in \\cite{needell:2016} \\cite{razaviyayn:2019} in the context of stochastic gradient descent and alternating projection algorithms respectively.}. First, for any index $i$, let us define the following function\n\\begin{align}\n\\label{def:function_i}\n    f_i(x) = \\frac{1}{2} |(a_{i}^Tx-b_{i})^+|^2, \\quad \\nabla f_i(x) = (a_{i}^Tx-b_{i})^+ a_{i}.\n\\end{align}\n","start":29445,"end":30068}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":["def:function"],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"期望残差函数","source_units":[{"raw_tex":"Then to simplify the expectation expression of \\eqref{def:exp} further, we define the function $f$ and the gradient of $f$ as follows:\n\\begin{align}\n\\label{def:function}\n    f(x) =  \\E_{\\mathbb{S}}\\left[f_{i^*}(x)\\right], \\quad \\nabla f(x) = \\E_{\\mathbb{S}} \\left[\\nabla f_{i^*}(x)\\right],\n\\end{align}\nwhere, the index $i^*$ is selected by the rule provided in \\eqref{def:i1}.\n","start":30068,"end":30445}]},{"id":"S008","semantic_type":"algorithm","kind":"algorithm","labels":["alg:paskm","eq:askm1","eq:askm2","eq:yk"],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"PASKM完整算法","source_units":[{"raw_tex":"\\begin{algorithm}\n\\caption{PASKM Algorithm: $x_{k+1} = \\textbf{PASKM}(A,b,x_0,K, \\delta, \\beta)$}\n\\label{alg:paskm}\n\\begin{algorithmic}\n\\STATE{Initialize $v_0 \\leftarrow x_0, \\  k \\leftarrow 0$;}\n\\WHILE{$k \\leq K$}\n\\STATE{Choose $\\gamma, \\omega,  \\alpha$ considering either \\eqref{cond} or \\eqref{eq:askm12} and update\n\\begin{align}\n\\label{eq:yk}\n  y_k = \\alpha v_k + (1-\\alpha)x_k;\n\\end{align}}\n\\STATE{Choose a sample of $\\beta$ constraints, $\\tau_k$, uniformly at random from the rows of matrix $A$. From these $\\beta$ constraints, choose $i^* = \\argmax_{i \\in \\tau_k} \\{a_i^Ty_k-b_i, 0\\}$; \\ Update \n\\begin{align}\nx_{k+1} & = y_k- \\delta \\frac{\\left(a_{i^*}^Ty_k -b_{i^*}\\right)^+}{\\|a_{i^*}\\|^2} a_{i^*}; \\label{eq:askm1} \\\\\nv_{k+1} & = \\omega v_k+ (1-\\omega)y_k-\\gamma \\frac{\\left(a_{i^*}^Ty_k -b_{i^*}\\right)^+}{\\|a_{i^*}\\|^2} a_{i^*}; \\label{eq:askm2}\n\\end{align}\n}\n\\STATE{$k \\leftarrow k+1$;}\n\\ENDWHILE\n\\RETURN $x$\n\\end{algorithmic}\n\\end{algorithm}\n","start":36846,"end":37803}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":["alg:paskm"],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"共同初始点","source_units":[{"raw_tex":"\\begin{algorithm}\n\\caption{PASKM Algorithm: $x_{k+1} = \\textbf{PASKM}(A,b,x_0,K, \\delta, \\beta)$}\n\\label{alg:paskm}\n\\begin{algorithmic}\n\\STATE{Initialize $v_0 \\leftarrow x_0, \\  k \\leftarrow 0$;}\n","start":36846,"end":37042}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":["alg:paskm","eq:yk"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"混合点更新","source_units":[{"raw_tex":"\\WHILE{$k \\leq K$}\n\\STATE{Choose $\\gamma, \\omega,  \\alpha$ considering either \\eqref{cond} or \\eqref{eq:askm12} and update\n\\begin{align}\n\\label{eq:yk}\n  y_k = \\alpha v_k + (1-\\alpha)x_k;\n\\end{align}}\n","start":37042,"end":37242}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":["alg:paskm","eq:askm1","eq:askm2"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"随机投影与动量更新","source_units":[{"raw_tex":"\\STATE{Choose a sample of $\\beta$ constraints, $\\tau_k$, uniformly at random from the rows of matrix $A$. From these $\\beta$ constraints, choose $i^* = \\argmax_{i \\in \\tau_k} \\{a_i^Ty_k-b_i, 0\\}$; \\ Update \n\\begin{align}\nx_{k+1} & = y_k- \\delta \\frac{\\left(a_{i^*}^Ty_k -b_{i^*}\\right)^+}{\\|a_{i^*}\\|^2} a_{i^*}; \\label{eq:askm1} \\\\\nv_{k+1} & = \\omega v_k+ (1-\\omega)y_k-\\gamma \\frac{\\left(a_{i^*}^Ty_k -b_{i^*}\\right)^+}{\\|a_{i^*}\\|^2} a_{i^*}; \\label{eq:askm2}\n\\end{align}\n}\n","start":37242,"end":37719}]},{"id":"S012","semantic_type":"external_fact","kind":"external_fact","labels":["lem0"],"member_ids":["S012"],"declared_external":true,"declaration_text":["原始前提清单：Hoffman's error bound；B0046引用hoffman及lewis:2010的Theorem 4.4。"],"source_only":false,"title":"Hoffman误差界","source_units":[{"raw_tex":"\\begin{mdframed}[backgroundcolor=gray!15,   topline=false,   bottomline =false,   rightline=false,   leftline=false] \\begin{lemma}\n\\label{lem0}\n(Hoffman \\cite{hoffman}, Theorem 4.4 in \\cite{lewis:2010}) Let $x \\in \\R^n$ and $P$ be the feasible region, then there exists a constant $L > 0$ such that the following identity holds:\n\\begin{align*}\n  d(x,P)^2 \\leq L^2 \\ \\|(Ax-b)^+\\|^2.\n\\end{align*}\n\\end{lemma} \\end{mdframed}\n","start":45149,"end":45571}]},{"id":"S013","semantic_type":"external_fact","kind":"external_fact","labels":["lem:skmseq"],"member_ids":["S013"],"declared_external":true,"declaration_text":["原始前提清单：B0049，归于haddock:2017的Lemma 2.1。"],"source_only":false,"title":"同向序列加权不等式","source_units":[{"raw_tex":"\\begin{mdframed}[backgroundcolor=gray!15,   topline=false,   bottomline =false,   rightline=false,   leftline=false] \\begin{lemma}\n\\label{lem:skmseq}\n(Lemma 2.1 in \\cite{haddock:2017})  Let $\\{x_k\\},\\ \\{y_k\\}$ be real non-negative sequences such that $x_{k+1} > x_k > 0$ and $y_{k+1} \\geq y_k \\geq 0$, then\n\\begin{align*}\n  \\sum\\limits_{k=1}^{n} x_k y_k \\ \\geq \\   \\sum\\limits_{k=1}^{n} \\overline{x} y_k, \\quad \\text{where} \\ \\ \\overline{x} = \\frac{1}{n}\\sum\\limits_{k=1}^{n} x_k.\n\\end{align*}\n\\end{lemma} \\end{mdframed}\n","start":45880,"end":46401}]},{"id":"S014","semantic_type":"external_fact","kind":"external_fact","labels":["lem:distance"],"member_ids":["S014"],"declared_external":true,"declaration_text":["原始前提清单：标准欧氏投影性质，包括B0051及可行域凸性。"],"source_only":false,"title":"最近点距离比较","source_units":[{"raw_tex":"\\begin{mdframed}[backgroundcolor=gray!15,   topline=false,   bottomline =false,   rightline=false,   leftline=false] \\begin{lemma}\n\\label{lem:distance}\nFor any $x \\in \\R^n$ and $\\bar{x} \\in P$, the following identity holds,\n\\begin{align*}\n    d(x,P)^2 \\ = \\ \\|x-\\mathcal{P}(x) \\|^2 \\ \\leq \\ \\|x-\\bar{x} \\|^2.\n\\end{align*}\n\\end{lemma} \\end{mdframed}\n","start":46948,"end":47297}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":["eq:askm12"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"固定ζ的参数方案","source_units":[{"raw_tex":"\\begin{align}\n\\label{eq:askm12}\n\\zeta =\\frac{3.99\\eta \\mu_1}{(1-\\mu_1)^2} , \\ \\omega = 1-\\frac{\\zeta \\mu_1^2+2\\gamma \\mu_1-\\zeta \\mu_1}{1+\\zeta \\mu_1^2} , \\ \\ \\gamma = \\sqrt{ \\zeta \\eta \\mu_1}, \\ \\ \\alpha = \\frac{ \\eta}{ \\eta + \\gamma },\n\\end{align}\n","start":84654,"end":84904}]},{"id":"S016","semantic_type":"lemma","kind":"lemma","labels":["lem1"],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"平均行外积的谱界","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem1}\nLet $\\lambda_j$ be the $j^{th}$ eigenvalue of the matrix $W = \\E_{\\mathbb{S}}\\left[a_{i^*}a_{i^*}^T\\right]$, then for all $j$, the bound $0 \\leq \\lambda_j \\leq 1$ holds.\n\\end{lemma}","start":47416,"end":47624}]},{"id":"S017","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"平均外积谱界的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nSince $W$ is positive semi-definite, we can write $\\lambda_j \\geq 0$ for all $j$. Also as the mapping $\\lambda_{\\max}(X)$ is convex, using Jensen's inequality we have,\n\\begin{align*}\n    \\lambda_{\\max} (W) = \\lambda_{\\max} \\left[ \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] \\right] \\leq \\E_{\\mathbb{S}} \\left[\\lambda_{\\max}\\left(a_{i^*}a_{i^*}^T\\right)\\right] \\leq 1.\n\\end{align*}\n\\end{proof}","start":47641,"end":48049}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"平均外积半正定","source_units":[{"raw_tex":"Since $W$ is positive semi-definite, we can write $\\lambda_j \\geq 0$ for all $j$. Also as the mapping $\\lambda_{\\max}(X)$ is convex, using Jensen's inequality we have,\n","start":47655,"end":47823}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"Jensen控制最大特征值","source_units":[{"raw_tex":"Since $W$ is positive semi-definite, we can write $\\lambda_j \\geq 0$ for all $j$. Also as the mapping $\\lambda_{\\max}(X)$ is convex, using Jensen's inequality we have,\n\\begin{align*}\n    \\lambda_{\\max} (W) = \\lambda_{\\max} \\left[ \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] \\right] \\leq \\E_{\\mathbb{S}} \\left[\\lambda_{\\max}\\left(a_{i^*}a_{i^*}^T\\right)\\right] \\leq 1.\n\\end{align*}\n","start":47655,"end":48038}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","labels":["lem2"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"采样外积的矩阵上界","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem2}\nFor any $1 \\leq \\beta \\leq m$, we have the following:\n\\begin{align*}\n \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] \\preceq \\frac{\\beta}{m} A^TA.\n\\end{align*}\n\\end{lemma}","start":48168,"end":48365}]},{"id":"S021","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"采样矩阵界的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nSee Appendix 1.\n\\end{proof}","start":48382,"end":48423},{"raw_tex":"\\paragraph{Proof of Lemma \\ref{lem2}}\nUsing the expectation expression given in the first section we have,\n\\begin{align*}\n     \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] & = \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} (A^TA)_{\\underline{\\mathbf{i_j}}} \\\\\n     &  \\preceq \\ \\frac{\\binom{m-1}{\\beta-1}}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta}  (A^TA)_{\\underline{\\mathbf{i_j}}} \\ \\preceq \\ \\frac{\\beta}{m} \\sum\\limits_{i = 1}^{m}  a_{i} a_{i}^T  \\ = \\ \\frac{\\beta}{m} A^TA.\n\\end{align*}\nHere, the notation $(A^TA)_{\\underline{\\mathbf{i_j}}}$ denotes the matrix $a_l a_l^T$ where the index $l$ belongs to the list \\eqref{eq:sampling}. Furthermore, the index $l$ corresponds to the $(\\beta+ j)^{th}$ entry on the list \\eqref{eq:sampling}. This proves the Lemma.\n%$(Ax-b)^{+}_{\\fbox{${i_j}$}}$\n%$(Ax-b)^{+}_{\\fbox{\\strut ${i_j}$}}$\n%$(Ax-b)^{+}_{\\fbox{$\\mathstrut$ ${i_j}$}}$\n\n%$(Ax-b)^{+}_{\\underline{\\mathbf{i_j}}}$\n\n% $\\overline{\\mathbf{x}}$  $\\underline{\\mathbf{x}}$ \n\n\n","start":115634,"end":116654}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"组合权重控制矩阵和","source_units":[{"raw_tex":"\\paragraph{Proof of Lemma \\ref{lem2}}\nUsing the expectation expression given in the first section we have,\n\\begin{align*}\n     \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] & = \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} (A^TA)_{\\underline{\\mathbf{i_j}}} \\\\\n     &  \\preceq \\ \\frac{\\binom{m-1}{\\beta-1}}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta}  (A^TA)_{\\underline{\\mathbf{i_j}}} \\ \\preceq \\ \\frac{\\beta}{m} \\sum\\limits_{i = 1}^{m}  a_{i} a_{i}^T  \\ = \\ \\frac{\\beta}{m} A^TA.\n\\end{align*}\nHere, the notation $(A^TA)_{\\underline{\\mathbf{i_j}}}$ denotes the matrix $a_l a_l^T$ where the index $l$ belongs to the list \\eqref{eq:sampling}. Furthermore, the index $l$ corresponds to the $(\\beta+ j)^{th}$ entry on the list \\eqref{eq:sampling}. This proves the Lemma.\n","start":115634,"end":116443}]},{"id":"S023","semantic_type":"lemma","kind":"lemma","labels":["lem3"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"函数值与距离的双边界","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem3}\nFor any $x \\in \\R^n$ with $\\lambda_{\\max} = \\lambda_{\\max}(A^TA)$, we have the following:\n\\begin{align*}\n    \\frac{\\mu_1}{ 2} \\ d(x,P)^2 \\ \\leq \\ f(x) \\ \\leq \\  \\frac{\\mu_2}{2}\\ d(x,P)^2,\n\\end{align*}\nwith $ 0 < \\mu_1 = \\frac{1}{m L^2} \\leq \\ \\mu_2 = \\min \\left\\{1, \\frac{\\beta}{m} \\lambda_{\\max}\\right\\} \\leq 1$. \n\\end{lemma}","start":48545,"end":48898}]},{"id":"S024","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"距离双边界的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nSee Appendix 1.\n\\end{proof}","start":48915,"end":48956},{"raw_tex":"\\paragraph{Proof of Lemma \\ref{lem3}}\nUsing the definition of the expectation from \\eqref{def:exp}, we have,\n\\allowdisplaybreaks{\n\\begin{align*}\n \\E_{\\mathbb{S}} \\left[\\big | (a_{i^*}^Tx-b_{i^*})^+\\big |^2\\right] & \\overset{\\eqref{def:exp}}{=} \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1}  | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}} |^2 \\\\\n    & \\overset{\\text{Lemma} \\ \\ref{lem:skmseq}}{\\geq} \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta}  \\frac{\\sum\\limits_{l = 0}^{m-\\beta} \\binom{\\beta-1+l}{\\beta-1}}{m-\\beta +1} | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}} |^2 \\\\\n    & \\geq \\frac{1}{m-\\beta+1} \\sum\\limits_{j = 0}^{m-\\beta}  \\big |(Ax - b)^+_{\\underline{\\mathbf{i_j}}} \\big |^2  \\\\\n    & \\geq \\frac{1}{m-\\beta+1} \\min\\{\\frac{m-\\beta+1}{m-s}, 1\\} \\ \\|(Ax - b)^+\\|^2 \\\\\n    &  \\overset{\\text{Lemma} \\ \\ref{lem0}}{\\geq} \\ \\frac{1}{m L^2}  \\ d(x,P)^2. \n\\end{align*}}\nHere, $s$ is the number of zero entries in the residual $(Ax - b)^+$, which also corresponds to the number of satisfied constraints for $x$. Since $A \\mathcal{P} (x) \\leq b$, we have the following:\n\\begin{align*}\n    \\E_{\\mathbb{S}} \\left[\\big | (a_{i^*}^Tx-b_{i^*})^+\\big |^2\\right]   & \\overset{\\eqref{def:exp}}{=}  \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}}|^2 \\\\\n    & \\leq  \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} \\ \\big | (Ax-A\\mathcal{P} (x))_{\\underline{\\mathbf{i_j}}}   \\big|^2 \\\\\n    & =  \\frac{1}{\\binom{m}{\\beta}} (x-\\mathcal{P} (x))^T\\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} \\ (A^TA)_{\\underline{\\mathbf{i_j}}} (x-\\mathcal{P} (x) ) \\\\\n    & =  (x-\\mathcal{P} (x) )^T \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] (x-\\mathcal{P} (x) ) \\\\\n    & \\overset{\\text{Lemma} \\ \\ref{lem1} \\ \\& \\ \\ref{lem2} }{\\leq} \\  \\min \\left\\{1, \\frac{\\beta}{m} \\lambda_{\\max}\\right\\} \\big \\| x- \\mathcal{P} (x) \\big \\|^2 \\\\\n    & = \\  \\min \\left\\{1, \\frac{\\beta}{m} \\lambda_{\\max}\\right\\} d(x,P)^2.\n\\end{align*}\nCombining the above identities and using the expression for $f(x)$ from \\eqref{def:function} we get,\n\\begin{align*}\n    \\frac{\\mu_1}{ 2} \\ d(x,P)^2 \\ \\leq \\ f(x) \\ \\leq \\  \\frac{\\mu_2}{2} \\ d(x,P)^2,\n\\end{align*}\nwhich proves the Lemma. \n\n","start":116654,"end":118947}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"残差期望的距离下界","source_units":[{"raw_tex":"\\paragraph{Proof of Lemma \\ref{lem3}}\nUsing the definition of the expectation from \\eqref{def:exp}, we have,\n\\allowdisplaybreaks{\n\\begin{align*}\n \\E_{\\mathbb{S}} \\left[\\big | (a_{i^*}^Tx-b_{i^*})^+\\big |^2\\right] & \\overset{\\eqref{def:exp}}{=} \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1}  | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}} |^2 \\\\\n    & \\overset{\\text{Lemma} \\ \\ref{lem:skmseq}}{\\geq} \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta}  \\frac{\\sum\\limits_{l = 0}^{m-\\beta} \\binom{\\beta-1+l}{\\beta-1}}{m-\\beta +1} | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}} |^2 \\\\\n    & \\geq \\frac{1}{m-\\beta+1} \\sum\\limits_{j = 0}^{m-\\beta}  \\big |(Ax - b)^+_{\\underline{\\mathbf{i_j}}} \\big |^2  \\\\\n    & \\geq \\frac{1}{m-\\beta+1} \\min\\{\\frac{m-\\beta+1}{m-s}, 1\\} \\ \\|(Ax - b)^+\\|^2 \\\\\n    &  \\overset{\\text{Lemma} \\ \\ref{lem0}}{\\geq} \\ \\frac{1}{m L^2}  \\ d(x,P)^2. \n\\end{align*}}\nHere, $s$ is the number of zero entries in the residual $(Ax - b)^+$, which also corresponds to the number of satisfied constraints for $x$. Since $A \\mathcal{P} (x) \\leq b$, we have the following:\n","start":116654,"end":117763}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"可行投影给出谱上界","source_units":[{"raw_tex":"Here, $s$ is the number of zero entries in the residual $(Ax - b)^+$, which also corresponds to the number of satisfied constraints for $x$. Since $A \\mathcal{P} (x) \\leq b$, we have the following:\n\\begin{align*}\n    \\E_{\\mathbb{S}} \\left[\\big | (a_{i^*}^Tx-b_{i^*})^+\\big |^2\\right]   & \\overset{\\eqref{def:exp}}{=}  \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} | (Ax-b)^{+}_{\\underline{\\mathbf{i_j}}}|^2 \\\\\n    & \\leq  \\frac{1}{\\binom{m}{\\beta}} \\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} \\ \\big | (Ax-A\\mathcal{P} (x))_{\\underline{\\mathbf{i_j}}}   \\big|^2 \\\\\n    & =  \\frac{1}{\\binom{m}{\\beta}} (x-\\mathcal{P} (x))^T\\sum\\limits_{j = 0}^{m-\\beta} \\binom{\\beta-1+j}{\\beta-1} \\ (A^TA)_{\\underline{\\mathbf{i_j}}} (x-\\mathcal{P} (x) ) \\\\\n    & =  (x-\\mathcal{P} (x) )^T \\E_{\\mathbb{S}} \\left[a_{i^*}a_{i^*}^T\\right] (x-\\mathcal{P} (x) ) \\\\\n    & \\overset{\\text{Lemma} \\ \\ref{lem1} \\ \\& \\ \\ref{lem2} }{\\leq} \\  \\min \\left\\{1, \\frac{\\beta}{m} \\lambda_{\\max}\\right\\} \\big \\| x- \\mathcal{P} (x) \\big \\|^2 \\\\\n    & = \\  \\min \\left\\{1, \\frac{\\beta}{m} \\lambda_{\\max}\\right\\} d(x,P)^2.\n\\end{align*}\n","start":117565,"end":118708}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"合并函数值双边界","source_units":[{"raw_tex":"Combining the above identities and using the expression for $f(x)$ from \\eqref{def:function} we get,\n\\begin{align*}\n    \\frac{\\mu_1}{ 2} \\ d(x,P)^2 \\ \\leq \\ f(x) \\ \\leq \\  \\frac{\\mu_2}{2} \\ d(x,P)^2,\n\\end{align*}\nwhich proves the Lemma. \n\n","start":118708,"end":118947}]},{"id":"S028","semantic_type":"lemma","kind":"lemma","labels":["lem:grad"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"两点梯度内积上界","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:grad}\nFor any $x, y \\in \\R^n$, we have the following:\n\\begin{align*}\n \\langle  x-y, \\E_{\\mathbb{S}}  \\left[(a_{i^*}^Ty-b_{i^*})^{+} a_{i^*}\\right]  \\rangle  & =  \\langle  x-y, \\nabla f(y)   \\rangle   \\\\\n & \\leq    f(x) - f(y) \\leq     \\frac{\\mu_2}{2} \\ d(x,P)^2 -  \\frac{\\mu_1}{2} \\ d(y,P)^2.\n\\end{align*}\n\\end{lemma}","start":50379,"end":50721}]},{"id":"S029","semantic_type":"proof","kind":"proof","labels":["eq:grad"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"两点梯度界的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nSee Appendix 1.\n\\end{proof}","start":50738,"end":50779},{"raw_tex":"\\paragraph{Proof of Lemma \\ref{lem:grad}}\n\nFrom the definition of $f(x)$, it can be easily checked that $f(x)$ is a convex function. Now, by the convexity property of $f(x)$, for any $x, y \\in \\R^n$, we have the following:\n\\begin{align}\n\\label{eq:grad}\n    \\langle x-y, \\nabla f(y) \\rangle \\leq f(x) - f(y).\n\\end{align}\nTherefore, we have\n\\begin{align*}\n\\big \\langle  x-y, \\E_{\\mathbb{S}}  \\left[(a_{i^*}^Ty-b_{i^*})^{+} a_{i^*}\\right] \\big \\rangle & =  \\langle x-y, \\nabla f(y) \\rangle  \\leq  \\ f(x) - f(y) \\\\\n&  \\overset{\\text{Lemma} \\  \\ref{lem3} }{\\leq}  \\    \\frac{\\mu_2}{2} \\ d(x,P)^2 -  \\frac{\\mu_1}{2} \\ d(y,P)^2.\n\\end{align*}\nThis completes the proof.\n\n\n","start":118947,"end":119610}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":["eq:grad"],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"凸性与一阶支持","source_units":[{"raw_tex":"\\paragraph{Proof of Lemma \\ref{lem:grad}}\n\nFrom the definition of $f(x)$, it can be easily checked that $f(x)$ is a convex function. Now, by the convexity property of $f(x)$, for any $x, y \\in \\R^n$, we have the following:\n\\begin{align}\n\\label{eq:grad}\n    \\langle x-y, \\nabla f(y) \\rangle \\leq f(x) - f(y).\n\\end{align}\n","start":118947,"end":119267}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"代入距离双边估计","source_units":[{"raw_tex":"Therefore, we have\n\\begin{align*}\n\\big \\langle  x-y, \\E_{\\mathbb{S}}  \\left[(a_{i^*}^Ty-b_{i^*})^{+} a_{i^*}\\right] \\big \\rangle & =  \\langle x-y, \\nabla f(y) \\rangle  \\leq  \\ f(x) - f(y) \\\\\n&  \\overset{\\text{Lemma} \\  \\ref{lem3} }{\\leq}  \\    \\frac{\\mu_2}{2} \\ d(x,P)^2 -  \\frac{\\mu_1}{2} \\ d(y,P)^2.\n\\end{align*}\nThis completes the proof.\n\n\n","start":119267,"end":119610}]},{"id":"S032","semantic_type":"lemma","kind":"lemma","labels":["lem:grad1"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"可行点方向的梯度界","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:grad1}\nFor any $y \\in \\R^n$ and $\\bar{y}$ such that $A \\bar{y} \\leq b$, we have the following:\n\\begin{align*}\n\\langle  \\bar{y}-y, \\E_{\\mathbb{S}} \\left[a_{i^*}(a_{i^*}^Ty-b_{i^*})^{+}\\right]  \\rangle = \\ \\langle  \\bar{y}-y, \\nabla f(y)  \\rangle    \\leq  -2f(y)  \\leq \\ - \\mu_1  d(y,P)^2.\n\\end{align*}\n\\end{lemma}","start":52283,"end":52620}]},{"id":"S033","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"可行方向界的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nSee Appendix 1.\n\\end{proof}","start":52637,"end":52678},{"raw_tex":"\\paragraph{Proof of Lemma \\ref{lem:grad1}}\n\n\\iffalse\n\nLet's denote, $ \\mathcal{J} = \\{i_j \\ | \\ c_{i_j}^Ty-d_{i_j} \\geq 0, \\ i_j \\in T(y)\\}$. Then following the construction of \\eqref{eq:f}, the function $f(y)$ can be expressed as $f(y) = \\|C_{\\mathcal{J}} y-d_{\\mathcal{J}} \\|^2$.\nSince, $C_{\\mathcal{J}} y-d_{\\mathcal{J}} \\geq 0$ and $C_{\\mathcal{J}}\\bar{y}-d_{\\mathcal{J}} \\leq 0$, we have,\n\\begin{align*}\n    \\big \\langle C_{\\mathcal{J}}y-d_{\\mathcal{J}} , C_{\\mathcal{J}}\\bar{y}-d_{\\mathcal{J}} \\big \\rangle \\ \\leq \\ 0.\n\\end{align*}\nAdding $\\|C_{\\mathcal{J}}y-d_{\\mathcal{J}}\\|^2$ to both sides and rearranging terms we get,\n\\begin{align}\n\\label{eq:grad1}\n \\|C_{\\mathcal{J}}y-d_{\\mathcal{J}} \\|^2 \\ \\leq \\  \\big \\langle C_{\\mathcal{J}}^T \\left( C_{\\mathcal{J}}y-d_{\\mathcal{J}} \\right), y - \\bar{y} \\big \\rangle.\n\\end{align}\n\n\\fi\n\nSince, $\\bar{y} \\in P$, from the definition we have,\n\\begin{align*}\n   \\langle  \\bar{y}-y, \\E_{\\mathbb{S}}  \\left[a_{i^*}(a_{i^*}^Ty-b_{i^*})^{+}\\right]  \\rangle \\ & = \\E_{\\mathbb{S}} \\left[(a_{i^*}^Ty-b_{i^*})^{+} \\left(a_{i^*}^T\\bar{y}- a_{i^*}^T y\\right)\\right] \\\\\n   & \\leq \\E_{\\mathbb{S}} \\left[(a_{i^*}^Ty-b_{i^*})^{+} \\left(b_{i^*}- a_{i^*}^T y\\right)\\right] \\\\\n   & = - \\E_{\\mathbb{S}} \\left[\\big | (a_{i^*}^Ty-b_{i^*})^{+} \\big |^2\\right] \\\\\n   &= - 2f(y) \\overset{\\text{Lemma} \\ \\ref{lem3} }{\\leq} \\ \\ -\\mu_1 \\ d(y,P)^2.\n\\end{align*}\nHere, we used the identity $xx^+ = |x^+|^2$. This proves the Lemma.\n\n","start":119610,"end":121059}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"逐行可行性压低内积","source_units":[{"raw_tex":"Since, $\\bar{y} \\in P$, from the definition we have,\n\\begin{align*}\n   \\langle  \\bar{y}-y, \\E_{\\mathbb{S}}  \\left[a_{i^*}(a_{i^*}^Ty-b_{i^*})^{+}\\right]  \\rangle \\ & = \\E_{\\mathbb{S}} \\left[(a_{i^*}^Ty-b_{i^*})^{+} \\left(a_{i^*}^T\\bar{y}- a_{i^*}^T y\\right)\\right] \\\\\n   & \\leq \\E_{\\mathbb{S}} \\left[(a_{i^*}^Ty-b_{i^*})^{+} \\left(b_{i^*}- a_{i^*}^T y\\right)\\right] \\\\\n   & = - \\E_{\\mathbb{S}} \\left[\\big | (a_{i^*}^Ty-b_{i^*})^{+} \\big |^2\\right] \\\\\n   &= - 2f(y) \\overset{\\text{Lemma} \\ \\ref{lem3} }{\\leq} \\ \\ -\\mu_1 \\ d(y,P)^2.\n\\end{align*}\nHere, we used the identity $xx^+ = |x^+|^2$. This proves the Lemma.\n\n","start":120446,"end":121059}]},{"id":"S036","semantic_type":"lemma","kind":"lemma","labels":["lem4"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"单步SKM距离收缩","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem4}\nFor any $x \\in \\R^n$ and $0 < \\delta < 2$, we have the following:\n\\begin{align*}\n  \\E_{\\mathbb{S}} \\left[ d(z,P)^2\\right] = \\E_{\\mathbb{S}} \\left[ \\Big \\| x- \\mathcal{P} (x) - \\delta \\left(a_{i^*}^Tx-b_{i^*}\\right)^+ a_{i^*}\\Big \\|^2 \\right]  \\leq   h(\\delta) \\ d(x,P)^2,\n\\end{align*}\nwhere, $z = x - \\delta \\left(a_{i^*}^Tx-b_{i^*}\\right)^+ a_{i^*}$, $ \\eta = 2 \\delta -\\delta^2$ and $h(\\delta) = 1- \\eta \\mu_1 < 1$.\n\\end{lemma}","start":53802,"end":54258}]},{"id":"S037","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"单步收缩的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nSee Appendix 1.\n\\end{proof}","start":54275,"end":54316},{"raw_tex":"\\paragraph{Proof of Lemma \\ref{lem4}}\n\nSince, $\\mathcal{P}(x) \\in P$, we have\n{\\allowdisplaybreaks\n\\begin{align*}\n    \\E_{\\mathbb{S}}  \\left[ d(z,P)^2\\right] &  \\overset{\\text{Lemma} \\ \\ref{lem:distance}}{\\leq} \\  \\E_{\\mathbb{S}}  \\left[ \\|z- \\mathcal{P}(x)\\|^2\\right] = \\E_{\\mathbb{S}} \\left[ \\big \\| x- \\mathcal{P} (x) - \\delta \\left(a_{i^*}^Tx-b_{i^*}\\right)^+ a_{i^*} \\big \\|^2 \\right] \\\\\n    & \\overset{\\eqref{def:function}}{=} \\ \\|x - \\mathcal{P} (x) \\|^2 + 2\\delta^2 f(x)  + 2 \\delta \\ \\big  \\langle  \\mathcal{P}(x)-x, \\nabla f(x) \\big \\rangle \\\\\n    & \\overset{\\text{Lemma} \\ \\ref{lem3} }{\\leq} \\ \\|x - \\mathcal{P} (x) \\|^2 -2(2\\delta-\\delta^2) f(x) \\\\\n    & \\leq \\ \\|x-  \\mathcal{P} (x) \\|^2 - (2\\delta-\\delta^2) \\ \\mu_1  \\|x-  \\mathcal{P} (x) \\|^2 = h(\\delta) \\ d(x,P)^2.\n\\end{align*}}\nHere, we used the lower bound of the expected value from Lemma \\ref{lem3}.\n\n","start":121059,"end":121931}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"旧投影比较与平方展开","source_units":[{"raw_tex":"\\paragraph{Proof of Lemma \\ref{lem4}}\n\nSince, $\\mathcal{P}(x) \\in P$, we have\n{\\allowdisplaybreaks\n\\begin{align*}\n    \\E_{\\mathbb{S}}  \\left[ d(z,P)^2\\right] &  \\overset{\\text{Lemma} \\ \\ref{lem:distance}}{\\leq} \\  \\E_{\\mathbb{S}}  \\left[ \\|z- \\mathcal{P}(x)\\|^2\\right] = \\E_{\\mathbb{S}} \\left[ \\big \\| x- \\mathcal{P} (x) - \\delta \\left(a_{i^*}^Tx-b_{i^*}\\right)^+ a_{i^*} \\big \\|^2 \\right] \\\\\n    & \\overset{\\eqref{def:function}}{=} \\ \\|x - \\mathcal{P} (x) \\|^2 + 2\\delta^2 f(x)  + 2 \\delta \\ \\big  \\langle  \\mathcal{P}(x)-x, \\nabla f(x) \\big \\rangle \\\\\n","start":121059,"end":121613}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"保留函数值的下降量","source_units":[{"raw_tex":"    & \\overset{\\text{Lemma} \\ \\ref{lem3} }{\\leq} \\ \\|x - \\mathcal{P} (x) \\|^2 -2(2\\delta-\\delta^2) f(x) \\\\\n","start":121613,"end":121720}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"函数下降转为距离收缩","source_units":[{"raw_tex":"    & \\leq \\ \\|x-  \\mathcal{P} (x) \\|^2 - (2\\delta-\\delta^2) \\ \\mu_1  \\|x-  \\mathcal{P} (x) \\|^2 = h(\\delta) \\ d(x,P)^2.\n\\end{align*}}\nHere, we used the lower bound of the expected value from Lemma \\ref{lem3}.\n\n","start":121720,"end":121931}]},{"id":"S041","semantic_type":"theorem","kind":"theorem","labels":["th:10"],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"PASKM加权误差线性收敛","source_units":[{"raw_tex":"\\begin{theorem}\n\\label{th:10}\nLet, $v_{k+1}$ and $x_{k+1}$ are generated by Algorithm \\ref{alg:paskm}. If we select the parameters $\\omega, \\ \\gamma, \\ \\alpha$ as\n\\begin{align*}\n\\omega = 1-\\frac{\\zeta \\mu_1^2+2\\gamma \\mu_1-\\zeta \\mu_1}{1+\\zeta \\mu_1^2} , \\ \\ \\gamma = \\sqrt{ \\zeta \\eta \\mu_1}, \\ \\ \\alpha = \\frac{ \\eta}{ \\eta + \\gamma },\n\\end{align*}\nwhere, $\\zeta$ is chosen as $ 0 < \\zeta  < \\frac{4\\eta \\mu_1}{(1-\\mu_1)^2} $ if $\\mu_1 < 1$, otherwise choose any $\\zeta > 0$. Then, for any $0 < \\delta < 2$, the sequence of iterates $\\{v_k\\}, \\ \\{x_k\\}$ converges and the following result holds:\n\\begin{align*}\n\\E \\left[ d(v_{k+1},P)^2 + \\zeta \\mu_1 \\  d(x_{k+1},P)^2 \\right] \\ & \\leq \\ \\omega^{k+1} \\ \\E \\left[ d(v_{0},P)^2+ \\zeta \\mu_1 \\  d(x_{0},P)^2 \\right] \\\\\n& = (1+\\zeta \\mu_1) \\ \\omega^{k+1} \\ d(x_{0},P)^2.\n\\end{align*}  \nThis theorem implies that the PASKM algorithm converges linearly with a rate of $\\omega$, which accumulates to a total of $\\mathcal{O}(\\frac{1+\\zeta \\mu_1^2}{\\zeta \\mu_1^2+2\\gamma \\mu_1-\\zeta \\mu_1}\\log{1/\\epsilon})$ iterations to bring the given error below $\\epsilon > 0$.\n\\end{theorem}","start":78323,"end":79444}]},{"id":"S042","semantic_type":"proof","kind":"proof","labels":["eq:askm10","eq:askm11","eq:askm13","eq:askm14","eq:askm4","eq:askm5","eq:askm6","eq:askm9"],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"目标定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nSee Appendix 3.\n\\end{proof}","start":79461,"end":79502},{"raw_tex":"\\paragraph{Proof of Theorem \\ref{th:10}}\n\nLet us define, $\\mathcal{V} = \\omega \\mathcal{P}(v_{k}) +(1-\\omega) \\mathcal{P}(y_{k}) $. Since $\\mathcal{V} \\in P$, using the update formula of $v_{k+1}$ from equation \\eqref{eq:askm2}, we have,\n\\allowdisplaybreaks{\n\\begin{align}\n\\label{eq:askm4}\n    d(v_{k+1},P)^2 & = \\|v_{k+1} - \\mathcal{P}(v_{k+1})\\|^2 \\overset{\\text{Lemma} \\ \\ref{lem:distance}}{\\leq}  \\|v_{k+1} - \\mathcal{V}\\|^2  \\nonumber \\\\\n    & \\overset{\\eqref{eq:askm2}}{=}  \\big \\|\\omega v_k + (1-\\omega) y_k- \\mathcal{V} -\\gamma (a_{i^*}^Ty_k- b_{i^*})^+ a_{i^*} \\big \\|^2 \\nonumber \\\\\n    & =   \\underbrace{ \\|\\omega v_k + (1-\\omega) y_k- \\mathcal{V} \\|^2}_{I_1} + \\gamma^2 \\underbrace{\\|(a_{i^*}^Ty_k- b_{i^*})^+a_{i^*}\\|^2}_{I_2} \\nonumber \\\\\n    & - 2 \\gamma  \\underbrace{\\big \\langle \\omega v_k + (1-\\omega) y_k- \\mathcal{V} \\ ,\\  a_{i^*} (a_{i^*}^Ty_k-b_{i^{*}})^{+} \\big \\rangle }_{I_3} \\nonumber \\\\\n    & = I_1 + \\gamma^2 I_2 -2 \\gamma I_3.\n\\end{align}}\nSince $\\|\\cdot\\|^{2}$ is a convex function and $0 < \\omega < 1$, we can bound the expected first term as follows,\n\\begin{align}\n\\label{eq:askm5}\n    \\E_{\\mathbb{S}_k} [I_1]  &= \\E_{\\mathbb{S}_k} \\left[\\|\\omega v_k + (1-\\omega) y_k-\\mathcal{V}\\|^2\\right] \\nonumber \\\\\n    & = \\E_{\\mathbb{S}_k} \\left[\\|\\omega v_k + (1-\\omega) y_k- \\omega \\mathcal{P}(v_{k}) - (1-\\omega) \\mathcal{P}(y_{k})\\|^2\\right] \\nonumber \\\\\n    & \\leq \\omega \\| v_k-\\mathcal{P}(v_{k})\\|^2 + (1-\\omega) \\|y_k-\\mathcal{P}(y_{k})\\|^2\\nonumber \\\\\n    & =  \\omega \\ d(v_k,P)^2 + (1-\\omega) \\ d(y_k,P)^2.\n\\end{align}\nTaking expectation with respect to the sampling distribution in the second term of equation \\eqref{eq:askm4} and using Lemma \\ref{lem4} with the choice $z = x_{k+1}, \\ x = y_k$ and $\\eta = 2\\delta-\\delta^2$, we get,\n\\begin{align}\n\\label{eq:askm6}\n   \\gamma^2 \\E_{\\mathbb{S}_k} \\left[\\|(a_{i^*}^Ty_k- b_{i^*})^+ a_{i^*}\\|^2\\right] & \\ = \\  \\gamma^2 \\E_{\\mathbb{S}_k} \\left[|(a_{i^*}^Ty_k- b_{i^*})^+ |^2\\right] \\nonumber \\\\\n    \\overset{\\text{Lemma} \\ \\ref{lem4}}{\\leq} & \\ \\frac{\\gamma^2 }{\\eta} \\left[d(y_k,P)^2 - \\E \\left[ d(x_{k+1},P)^2 \\right]\\right].\n\\end{align}\n\nNow, taking expectation in the third term of \\eqref{eq:askm4} we get,\n\\begin{align}\n\\label{eq:askm9}\n    - 2  \\gamma  \\E_{\\mathbb{S}_k} & [I_3 ]  = - 2 \\gamma \\big \\langle \\omega v_k + (1-\\omega) y_k-\\mathcal{V}, \\E_{\\mathbb{S}_k} \\left[ a_{i^*} (a_{i^*}^Ty_k-b_{i^*})^{+}\\right]  \\big \\rangle \\nonumber \\\\\n    & \\overset{\\eqref{eq:yk} \\ \\& \\ \\eqref{def:function} }{=} - 2 \\gamma \\big \\langle \\frac{\\omega}{\\alpha} \\left[y_k - (1-\\alpha) x_k\\right] + (1-\\omega) y_k-\\mathcal{V},  \\nabla f(y_k)  \\big \\rangle \\nonumber \\\\\n    & = - 2 \\gamma \\big \\langle \\frac{\\omega(1-\\alpha)}{\\alpha}(y_k-x_k)+ y_k-\\mathcal{V}, \\nabla f(y_k)  \\big \\rangle.\n\\end{align}\nUsing Lemma \\ref{lem:grad} and Lemma \\ref{lem:grad1} we can simplify equation \\eqref{eq:askm9} as follows,\n\\begin{align}\n\\label{eq:askm10}\n  & \\ -  2 \\gamma  \\E_{\\mathbb{S}_k}  [I_3 ]  =   - 2 \\gamma \\big \\langle \\frac{\\omega(1-\\alpha)}{\\alpha}(y_k-x_k)+ y_k-\\mathcal{V}, \\nabla f(y_k)  \\big \\rangle \\nonumber \\\\\n    & \\quad =    2 \\gamma \\frac{\\omega(1-\\alpha)}{\\alpha} \\big \\langle x_k-y_k, \\nabla f(y_k)  \\big \\rangle + 2 \\gamma \\big \\langle \\mathcal{V}-y_k, \\nabla f(y_k)  \\big \\rangle \\nonumber \\\\\n  & \\overset{\\text{Lemma} \\ \\ref{lem:grad} \\ \\& \\ \\ref{lem:grad1}}{\\leq} \\frac{\\gamma \\omega (1-\\alpha)}{\\alpha} \\  d(x_k,P)^2 - \\frac{ \\mu_1 \\gamma  \\omega (1-\\alpha)}{\\alpha} \\ d(y_k,P)^2 - 2 \\mu_1 \\gamma \\ d(y_k,P)^2.\n\\end{align}\nNow, substituting the values of equation \\eqref{eq:askm5}, \\eqref{eq:askm6} \\& \\eqref{eq:askm10} in equation \\eqref{eq:askm4} we get the following:\n\\allowdisplaybreaks{\n\\begin{align*}\n \\E [ & d(v_{k+1} ,P)^2  ]  =  I_1 + \\gamma^2 \\E_{\\mathbb{S}_k}[I_2 ]- 2 \\gamma \\E_{\\mathbb{S}_k}[I_3] \\nonumber \\\\\n & = \\omega \\ d(v_{k},P)^2 + (1-\\omega) \\ d(y_k,P)^2 + \\frac{\\gamma^2 }{\\eta} \\left\\{d(y_k,P)^2 - \\E \\left[ d(x_{k+1},P)^2 \\right]\\right\\} \\nonumber \\\\\n & \\quad + \\frac{\\gamma \\omega (1-\\alpha)}{\\alpha}  d(x_k,P)^2 - \\gamma \\mu_1 \\left(2+\\frac{ \\omega (1-\\alpha)}{\\alpha}\\right)  d(y_k,P)^2.\n\\end{align*}}\nWith further simplification, the above identity can be written as follows:\n\\allowdisplaybreaks{\n\\begin{align}\n\\label{eq:askm11}\n & \\E \\left[ d(v_{k+1},P)^2 + \\frac{\\gamma^2}{\\eta} d(x_{k+1},P)^2 \\right]  = \\omega \\ \\left[d(v_{k},P)^2 + \\frac{\\gamma (1-\\alpha)}{\\alpha} d(x_{k},P)^2 \\right] \\nonumber \\\\\n & \\quad \\quad \\quad \\quad \\quad \\quad \\quad + d(y_k,P)^2 \\ \\left\\{1-\\omega+ \\frac{\\gamma^2 }{\\eta}  - 2\\gamma \\mu_1 - \\frac{\\gamma \\omega \\mu_1 (1-\\alpha)}{\\alpha}  \\right\\}.\n\\end{align}}\nNow, let's choose the parameters as in equation \\eqref{eq:askm12} along with $0 < \\zeta  < \\frac{4\\eta \\mu_1}{(1-\\mu_1)^2} $. We can easily see that $\\frac{\\gamma^2 }{\\eta} = \\frac{ \\gamma(1-\\alpha)}{\\alpha}$ and $ \\alpha \\in (0,1)$. Also note that,\n\\allowdisplaybreaks{\\begin{align}\n 2 \\mu_1 \\gamma \\ = \\ 2 \\mu_1 \\sqrt{\\eta \\zeta \\mu_1} \\ > \\ 2 \\mu_1 \\sqrt{\\frac{\\zeta (1-\\mu_1)^2 \\zeta}{4}} = \\mu_1 \\zeta(1-\\mu_1),\n\\end{align}}\nwhich implies $\\omega < 1$. Similarly, whenever $\\mu_1 < 1$ we have\n\\begin{align}\n    2\\gamma - \\zeta - \\frac{1}{\\mu_1} < 2\\sqrt{\\eta \\mu_1 \\zeta} - \\zeta - 1 \\leq  2 \\sqrt{\\zeta} - \\zeta - 1  = -(\\sqrt{\\zeta}-1)^2 \\leq 0,\n\\end{align}\nwhich implies $\\omega > 0$. Also, using the parameter choice of \\eqref{eq:askm12}, we have, \n\\begin{align}\n\\label{eq:askm13}\n  1-\\omega+ \\frac{\\gamma^2 }{\\eta}  - 2\\gamma \\mu_1 - & \\frac{\\gamma \\omega \\mu_1 (1-\\alpha)}{\\alpha}   = 1-\\omega + \\zeta \\mu_1 - 2\\gamma \\mu_1 - \\omega \\zeta \\mu_1^2 \\nonumber \\\\\n  & = 1- 2\\gamma \\mu_1 + \\zeta \\mu_1 - \\omega (1+\\zeta \\mu_1^2) = 0.\n\\end{align}\nNow, using all of the above relations (equation \\eqref{eq:askm12}, \\eqref{eq:askm13}) in equation \\eqref{eq:askm11}, we get the following:\n\\begin{align}\n\\label{eq:askm14}\n  \\E \\left[ d(v_{k+1},P)^2 + \\frac{\\gamma^2}{\\eta} d(x_{k+1},P)^2 \\right] &  \\ \\leq \\ \\omega \\Big[d(v_{k},P)^2 + \\underbrace{\\frac{\\gamma (1-\\alpha)}{\\alpha}}_{= \\frac{\\gamma^2 }{\\eta}} \\ d(x_{k},P)^2 \\Big] \\nonumber \\\\\n  + & \\Big [\\underbrace{1-\\omega+ \\frac{\\gamma^2 }{\\eta}  - 2\\gamma \\mu_1 - \\frac{\\gamma \\omega \\mu_1 (1-\\alpha)}{\\alpha} }_{= \\ 0} \\Big ] \\ d(y_k,P)^2 \\nonumber \\\\\n   = \\ & \\omega \\ \\left[ d(v_{k},P)^2+ \\frac{\\gamma^2}{\\eta} d(x_{k},P)^2 \\right].\n\\end{align}\nFinally, taking expectation again with tower rule and substituting $\\frac{\\gamma^2 }{\\eta}= \\zeta \\mu_1 $ we have,\n\\begin{align*}\n\\E \\left[ d(v_{k+1},P)^2 + \\zeta \\mu_1 \\  d(x_{k+1},P)^2 \\right] \\ & \\leq \\ \\omega^{k+1} \\ \\E \\left[ d(v_{0},P)^2+ \\zeta \\mu_1 \\  d(x_{0},P)^2 \\right] \\\\\n& = (1+\\zeta \\mu_1) \\ \\omega^{k+1} \\ d(x_{0},P)^2.\n\\end{align*}\nThis proves the Theorem. Furthermore, for faster convergence, we need to choose parameters such that, $\\omega$ becomes as small as possible. In the proof, we assumed $\\mu_1 <1$ holds which is the most probable scenario. Whenever $\\mu_1 = 1$, we must have $\\mu_1 = \\mu_2 =1$ and Lemma 6 holds with both equality, i.e., $f(x) = d(x,P)^2$. Therefore if we choose $\\alpha, \\gamma, \\omega$ as $\\alpha = \\frac{\\eta}{\\eta+ \\gamma}, \\ \\gamma = \\sqrt{\\zeta \\eta}, \\ \\omega = 1- \\frac{2\\gamma}{1+\\zeta}$, we can check that condition (89) holds and $0 < \\omega < 1$ holds for any $\\zeta > 0$.\n\n\n\n\n\n\n\n","start":156521,"end":163768}]},{"id":"S043","semantic_type":"assumption","kind":"assumption","labels":["th:10"],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"目标定理的参数假设","source_units":[{"raw_tex":"Let, $v_{k+1}$ and $x_{k+1}$ are generated by Algorithm \\ref{alg:paskm}. If we select the parameters $\\omega, \\ \\gamma, \\ \\alpha$ as\n\\begin{align*}\n\\omega = 1-\\frac{\\zeta \\mu_1^2+2\\gamma \\mu_1-\\zeta \\mu_1}{1+\\zeta \\mu_1^2} , \\ \\ \\gamma = \\sqrt{ \\zeta \\eta \\mu_1}, \\ \\ \\alpha = \\frac{ \\eta}{ \\eta + \\gamma },\n\\end{align*}\nwhere, $\\zeta$ is chosen as $ 0 < \\zeta  < \\frac{4\\eta \\mu_1}{(1-\\mu_1)^2} $ if $\\mu_1 < 1$, otherwise choose any $\\zeta > 0$. Then, for any $0 < \\delta < 2$, the sequence of iterates $\\{v_k\\}, \\ \\{x_k\\}$ converges and the following result holds:\n","start":78353,"end":78921}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":["eq:askm4"],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"可行组合点与三项展开","source_units":[{"raw_tex":"\\paragraph{Proof of Theorem \\ref{th:10}}\n\nLet us define, $\\mathcal{V} = \\omega \\mathcal{P}(v_{k}) +(1-\\omega) \\mathcal{P}(y_{k}) $. Since $\\mathcal{V} \\in P$, using the update formula of $v_{k+1}$ from equation \\eqref{eq:askm2}, we have,\n\\allowdisplaybreaks{\n\\begin{align}\n\\label{eq:askm4}\n    d(v_{k+1},P)^2 & = \\|v_{k+1} - \\mathcal{P}(v_{k+1})\\|^2 \\overset{\\text{Lemma} \\ \\ref{lem:distance}}{\\leq}  \\|v_{k+1} - \\mathcal{V}\\|^2  \\nonumber \\\\\n    & \\overset{\\eqref{eq:askm2}}{=}  \\big \\|\\omega v_k + (1-\\omega) y_k- \\mathcal{V} -\\gamma (a_{i^*}^Ty_k- b_{i^*})^+ a_{i^*} \\big \\|^2 \\nonumber \\\\\n    & =   \\underbrace{ \\|\\omega v_k + (1-\\omega) y_k- \\mathcal{V} \\|^2}_{I_1} + \\gamma^2 \\underbrace{\\|(a_{i^*}^Ty_k- b_{i^*})^+a_{i^*}\\|^2}_{I_2} \\nonumber \\\\\n    & - 2 \\gamma  \\underbrace{\\big \\langle \\omega v_k + (1-\\omega) y_k- \\mathcal{V} \\ ,\\  a_{i^*} (a_{i^*}^Ty_k-b_{i^{*}})^{+} \\big \\rangle }_{I_3} \\nonumber \\\\\n    & = I_1 + \\gamma^2 I_2 -2 \\gamma I_3.\n\\end{align}}\n","start":156521,"end":157490}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["eq:askm5"],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"凸性控制混合误差","source_units":[{"raw_tex":"Since $\\|\\cdot\\|^{2}$ is a convex function and $0 < \\omega < 1$, we can bound the expected first term as follows,\n\\begin{align}\n\\label{eq:askm5}\n    \\E_{\\mathbb{S}_k} [I_1]  &= \\E_{\\mathbb{S}_k} \\left[\\|\\omega v_k + (1-\\omega) y_k-\\mathcal{V}\\|^2\\right] \\nonumber \\\\\n    & = \\E_{\\mathbb{S}_k} \\left[\\|\\omega v_k + (1-\\omega) y_k- \\omega \\mathcal{P}(v_{k}) - (1-\\omega) \\mathcal{P}(y_{k})\\|^2\\right] \\nonumber \\\\\n    & \\leq \\omega \\| v_k-\\mathcal{P}(v_{k})\\|^2 + (1-\\omega) \\|y_k-\\mathcal{P}(y_{k})\\|^2\\nonumber \\\\\n    & =  \\omega \\ d(v_k,P)^2 + (1-\\omega) \\ d(y_k,P)^2.\n\\end{align}\n","start":157490,"end":158072}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["eq:askm6"],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"残差项换成距离下降","source_units":[{"raw_tex":"Taking expectation with respect to the sampling distribution in the second term of equation \\eqref{eq:askm4} and using Lemma \\ref{lem4} with the choice $z = x_{k+1}, \\ x = y_k$ and $\\eta = 2\\delta-\\delta^2$, we get,\n\\begin{align}\n\\label{eq:askm6}\n   \\gamma^2 \\E_{\\mathbb{S}_k} \\left[\\|(a_{i^*}^Ty_k- b_{i^*})^+ a_{i^*}\\|^2\\right] & \\ = \\  \\gamma^2 \\E_{\\mathbb{S}_k} \\left[|(a_{i^*}^Ty_k- b_{i^*})^+ |^2\\right] \\nonumber \\\\\n    \\overset{\\text{Lemma} \\ \\ref{lem4}}{\\leq} & \\ \\frac{\\gamma^2 }{\\eta} \\left[d(y_k,P)^2 - \\E \\left[ d(x_{k+1},P)^2 \\right]\\right].\n\\end{align}\n","start":158072,"end":158640}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":["eq:askm9"],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"交叉项的梯度改写","source_units":[{"raw_tex":"\nNow, taking expectation in the third term of \\eqref{eq:askm4} we get,\n\\begin{align}\n\\label{eq:askm9}\n    - 2  \\gamma  \\E_{\\mathbb{S}_k} & [I_3 ]  = - 2 \\gamma \\big \\langle \\omega v_k + (1-\\omega) y_k-\\mathcal{V}, \\E_{\\mathbb{S}_k} \\left[ a_{i^*} (a_{i^*}^Ty_k-b_{i^*})^{+}\\right]  \\big \\rangle \\nonumber \\\\\n    & \\overset{\\eqref{eq:yk} \\ \\& \\ \\eqref{def:function} }{=} - 2 \\gamma \\big \\langle \\frac{\\omega}{\\alpha} \\left[y_k - (1-\\alpha) x_k\\right] + (1-\\omega) y_k-\\mathcal{V},  \\nabla f(y_k)  \\big \\rangle \\nonumber \\\\\n    & = - 2 \\gamma \\big \\langle \\frac{\\omega(1-\\alpha)}{\\alpha}(y_k-x_k)+ y_k-\\mathcal{V}, \\nabla f(y_k)  \\big \\rangle.\n\\end{align}\n","start":158640,"end":159294}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":["eq:askm10"],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"两种内积界控制交叉项","source_units":[{"raw_tex":"Using Lemma \\ref{lem:grad} and Lemma \\ref{lem:grad1} we can simplify equation \\eqref{eq:askm9} as follows,\n\\begin{align}\n\\label{eq:askm10}\n  & \\ -  2 \\gamma  \\E_{\\mathbb{S}_k}  [I_3 ]  =   - 2 \\gamma \\big \\langle \\frac{\\omega(1-\\alpha)}{\\alpha}(y_k-x_k)+ y_k-\\mathcal{V}, \\nabla f(y_k)  \\big \\rangle \\nonumber \\\\\n    & \\quad =    2 \\gamma \\frac{\\omega(1-\\alpha)}{\\alpha} \\big \\langle x_k-y_k, \\nabla f(y_k)  \\big \\rangle + 2 \\gamma \\big \\langle \\mathcal{V}-y_k, \\nabla f(y_k)  \\big \\rangle \\nonumber \\\\\n  & \\overset{\\text{Lemma} \\ \\ref{lem:grad} \\ \\& \\ \\ref{lem:grad1}}{\\leq} \\frac{\\gamma \\omega (1-\\alpha)}{\\alpha} \\  d(x_k,P)^2 - \\frac{ \\mu_1 \\gamma  \\omega (1-\\alpha)}{\\alpha} \\ d(y_k,P)^2 - 2 \\mu_1 \\gamma \\ d(y_k,P)^2.\n\\end{align}\n","start":159294,"end":160030}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"合并三项距离估计","source_units":[{"raw_tex":"Now, substituting the values of equation \\eqref{eq:askm5}, \\eqref{eq:askm6} \\& \\eqref{eq:askm10} in equation \\eqref{eq:askm4} we get the following:\n\\allowdisplaybreaks{\n\\begin{align*}\n \\E [ & d(v_{k+1} ,P)^2  ]  =  I_1 + \\gamma^2 \\E_{\\mathbb{S}_k}[I_2 ]- 2 \\gamma \\E_{\\mathbb{S}_k}[I_3] \\nonumber \\\\\n & = \\omega \\ d(v_{k},P)^2 + (1-\\omega) \\ d(y_k,P)^2 + \\frac{\\gamma^2 }{\\eta} \\left\\{d(y_k,P)^2 - \\E \\left[ d(x_{k+1},P)^2 \\right]\\right\\} \\nonumber \\\\\n & \\quad + \\frac{\\gamma \\omega (1-\\alpha)}{\\alpha}  d(x_k,P)^2 - \\gamma \\mu_1 \\left(2+\\frac{ \\omega (1-\\alpha)}{\\alpha}\\right)  d(y_k,P)^2.\n\\end{align*}}\n","start":160030,"end":160636}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":["eq:askm11"],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"加权能量的一步关系","source_units":[{"raw_tex":"With further simplification, the above identity can be written as follows:\n\\allowdisplaybreaks{\n\\begin{align}\n\\label{eq:askm11}\n & \\E \\left[ d(v_{k+1},P)^2 + \\frac{\\gamma^2}{\\eta} d(x_{k+1},P)^2 \\right]  = \\omega \\ \\left[d(v_{k},P)^2 + \\frac{\\gamma (1-\\alpha)}{\\alpha} d(x_{k},P)^2 \\right] \\nonumber \\\\\n & \\quad \\quad \\quad \\quad \\quad \\quad \\quad + d(y_k,P)^2 \\ \\left\\{1-\\omega+ \\frac{\\gamma^2 }{\\eta}  - 2\\gamma \\mu_1 - \\frac{\\gamma \\omega \\mu_1 (1-\\alpha)}{\\alpha}  \\right\\}.\n\\end{align}}\n","start":160636,"end":161128}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"匹配新旧能量权重","source_units":[{"raw_tex":"Now, let's choose the parameters as in equation \\eqref{eq:askm12} along with $0 < \\zeta  < \\frac{4\\eta \\mu_1}{(1-\\mu_1)^2} $. We can easily see that $\\frac{\\gamma^2 }{\\eta} = \\frac{ \\gamma(1-\\alpha)}{\\alpha}$ and $ \\alpha \\in (0,1)$. Also note that,\n","start":161128,"end":161378}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"验证速率小于一","source_units":[{"raw_tex":"\\allowdisplaybreaks{\\begin{align}\n 2 \\mu_1 \\gamma \\ = \\ 2 \\mu_1 \\sqrt{\\eta \\zeta \\mu_1} \\ > \\ 2 \\mu_1 \\sqrt{\\frac{\\zeta (1-\\mu_1)^2 \\zeta}{4}} = \\mu_1 \\zeta(1-\\mu_1),\n\\end{align}}\nwhich implies $\\omega < 1$. Similarly, whenever $\\mu_1 < 1$ we have\n","start":161378,"end":161626}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"验证速率为正","source_units":[{"raw_tex":"which implies $\\omega < 1$. Similarly, whenever $\\mu_1 < 1$ we have\n\\begin{align}\n    2\\gamma - \\zeta - \\frac{1}{\\mu_1} < 2\\sqrt{\\eta \\mu_1 \\zeta} - \\zeta - 1 \\leq  2 \\sqrt{\\zeta} - \\zeta - 1  = -(\\sqrt{\\zeta}-1)^2 \\leq 0,\n\\end{align}\nwhich implies $\\omega > 0$. Also, using the parameter choice of \\eqref{eq:askm12}, we have, \n","start":161558,"end":161886}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":["eq:askm13"],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"消去中间点误差系数","source_units":[{"raw_tex":"which implies $\\omega > 0$. Also, using the parameter choice of \\eqref{eq:askm12}, we have, \n\\begin{align}\n\\label{eq:askm13}\n  1-\\omega+ \\frac{\\gamma^2 }{\\eta}  - 2\\gamma \\mu_1 - & \\frac{\\gamma \\omega \\mu_1 (1-\\alpha)}{\\alpha}   = 1-\\omega + \\zeta \\mu_1 - 2\\gamma \\mu_1 - \\omega \\zeta \\mu_1^2 \\nonumber \\\\\n  & = 1- 2\\gamma \\mu_1 + \\zeta \\mu_1 - \\omega (1+\\zeta \\mu_1^2) = 0.\n\\end{align}\n","start":161793,"end":162180}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["eq:askm14"],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"得到加权能量收缩","source_units":[{"raw_tex":"Now, using all of the above relations (equation \\eqref{eq:askm12}, \\eqref{eq:askm13}) in equation \\eqref{eq:askm11}, we get the following:\n\\begin{align}\n\\label{eq:askm14}\n  \\E \\left[ d(v_{k+1},P)^2 + \\frac{\\gamma^2}{\\eta} d(x_{k+1},P)^2 \\right] &  \\ \\leq \\ \\omega \\Big[d(v_{k},P)^2 + \\underbrace{\\frac{\\gamma (1-\\alpha)}{\\alpha}}_{= \\frac{\\gamma^2 }{\\eta}} \\ d(x_{k},P)^2 \\Big] \\nonumber \\\\\n  + & \\Big [\\underbrace{1-\\omega+ \\frac{\\gamma^2 }{\\eta}  - 2\\gamma \\mu_1 - \\frac{\\gamma \\omega \\mu_1 (1-\\alpha)}{\\alpha} }_{= \\ 0} \\Big ] \\ d(y_k,P)^2 \\nonumber \\\\\n   = \\ & \\omega \\ \\left[ d(v_{k},P)^2+ \\frac{\\gamma^2}{\\eta} d(x_{k},P)^2 \\right].\n\\end{align}\n","start":162180,"end":162831}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"塔式取期望并迭代","source_units":[{"raw_tex":"Finally, taking expectation again with tower rule and substituting $\\frac{\\gamma^2 }{\\eta}= \\zeta \\mu_1 $ we have,\n\\begin{align*}\n\\E \\left[ d(v_{k+1},P)^2 + \\zeta \\mu_1 \\  d(x_{k+1},P)^2 \\right] \\ & \\leq \\ \\omega^{k+1} \\ \\E \\left[ d(v_{0},P)^2+ \\zeta \\mu_1 \\  d(x_{0},P)^2 \\right] \\\\\n& = (1+\\zeta \\mu_1) \\ \\omega^{k+1} \\ d(x_{0},P)^2.\n\\end{align*}\n","start":162831,"end":163179}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"μ一等于一的边界讨论","source_units":[{"raw_tex":"This proves the Theorem. Furthermore, for faster convergence, we need to choose parameters such that, $\\omega$ becomes as small as possible. In the proof, we assumed $\\mu_1 <1$ holds which is the most probable scenario. Whenever $\\mu_1 = 1$, we must have $\\mu_1 = \\mu_2 =1$ and Lemma 6 holds with both equality, i.e., $f(x) = d(x,P)^2$. Therefore if we choose $\\alpha, \\gamma, \\omega$ as $\\alpha = \\frac{\\eta}{\\eta+ \\gamma}, \\ \\gamma = \\sqrt{\\zeta \\eta}, \\ \\omega = 1- \\frac{2\\gamma}{1+\\zeta}$, we can check that condition (89) holds and $0 < \\omega < 1$ holds for any $\\zeta > 0$.\n\n\n\n\n\n\n\n","start":163179,"end":163768}]},{"id":"S058","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S058"],"declared_external":true,"declaration_text":["原始前提清单：Jensen's inequality、convexity of the largest eigenvalue及elementary spectral bounds。"],"source_only":false,"title":"谱凸性与半正定运算","source_units":[]},{"id":"S059","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S059"],"declared_external":true,"declaration_text":["原始前提清单：standard Euclidean projection properties、convexity of the feasible region及Jensen's inequality。"],"source_only":false,"title":"凸可行域与平方范数","source_units":[]},{"id":"S060","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S060"],"declared_external":true,"declaration_text":["原始前提清单：convex first-order inequalities；次梯度表述仅用于说明该标准事实的适用条件。"],"source_only":false,"title":"凸函数一阶不等式","source_units":[]},{"id":"S061","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S061"],"declared_external":true,"declaration_text":["原始前提清单：tower property of expectation；作者在行2250明确使用tower rule。"],"source_only":false,"title":"期望塔式法则","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061"}}}}
{"record_id":"2207.01789v2","paper":{"title":"Improved Global Guarantees for the Nonconvex Burer--Monteiro Factorization via Rank Overparameterization","authors":["Zhang, Richard Y."],"arxiv_id":"2207.01789","version":"v2","journal":"Mathematical Programming","abstract_url":"https://arxiv.org/abs/2207.01789v2","pdf_url":"https://arxiv.org/pdf/2207.01789v2","doi":"10.1007/s10107-024-02160-6"},"context":{"raw_tex":"Consider minimizing a convex objective $\\phi$ over an $n\\times n$\npositive semidefinite matrix $M\\succeq0$, as in\n\\begin{equation}\nM^{\\star}=\\quad\\mathrm{minimize}\\quad\\phi(M)\\quad\\text{over }\\quad M\\succeq0,\\tag{P}\\label{eq:cvx}\n\\end{equation}\nIn practice, the matrix order $n$ is often so large---from tens\nof thousands to hundreds of millions---that even explicitly forming\nthe $n^{2}$ elements of the $n\\times n$ matrix variable $M$ would\nbe intractable. Where $r^{\\star}=\\rank(M^{\\star})$ is known a priori\nto be small, the standard approach is the nonconvex Burer--Monteiro\nfactorization~\\citep{burer2003nonlinear}, which rewrites $M=XX^{T}$\nand then directly optimizes over $X$ using a local optimization algorithm:\n\\begin{equation}\nX^{\\star}=\\quad\\mathrm{minimize}\\quad f(X)\\eqdef\\phi(XX^{T})\\text{ where }X\\text{ is }n\\times r\\text{ and }r\\ge r^{\\star}.\\tag{BM}\\label{eq:ncvx}\n\\end{equation}\nThis way, the semidefinite constraint $M=XX^{T}\\succeq0$ is automatically\nenforced for free. Moreover, by using a small search rank $r\\ll n$,\nthe number of variables is reduced to $O(n)$. However, this improved\nscalability comes at the cost of giving up the convexity of (\\ref{eq:cvx}).\nIn principle, local optimization can fail by getting stuck at a \\emph{spurious}\nlocal minimum of (\\ref{eq:ncvx})---a local minimum that is strictly\nworse than that of the global minimum. In practice, however, there\nseems to be many situations where this failure mode does not occur~\\citep{candes2015phase,rosen2019se,zhang2019spurious}.\n\n\nTo state this precisely, suppose that $\\phi$ is twice-differentiable,\n$L$-smooth, and $\\mu$-strongly convex over the entire set of $n\\times n$\nreal symmetric matrices $\\S^{n}$, as in\n\\[\n\\mu\\|E\\|_{F}^{2}\\le\\inner{\\nabla^{2}\\phi(M)[E]}E\\le L\\|E\\|_{F}^{2}\\qquad\\text{for all }M\\in\\S^{n},\n\\]\nwhere $\\inner EF\\eqdef\\tr(E^{T}F)$ and $\\|E\\|_{F}\\eqdef\\sqrt{\\inner EE}$\ndenote the matrix Euclidean inner product and norm respectively. \\citet{bhojanapalli2016global}\npointed out that if $\\phi$ has condition number $L/\\mu<3/2$, then\nthe usual second-order necessary conditions for \\emph{local} optimality\nin $f$ are also sufficient for \\emph{global} optimality, meaning\nthat\n\\[\n\\nabla f(X)=0,\\quad\\nabla^{2}f(X)\\succeq0\\quad\\iff\\quad f(X)=\\min_{U}f(U).\n\\]\n\n\n\\textbf{Basic linear algebra.} Lower-case letters are vectors and\nupper-case letters are matrices. We use ``MATLAB notation'' in concatenating\nvectors and matrices:\n\n\\[\n[a,b]=\\begin{bmatrix}a & b\\end{bmatrix},\\qquad[a;b]=\\begin{bmatrix}a\\\\\nb\n\\end{bmatrix},\\qquad\\diag(a,b)=\\begin{bmatrix}a & 0\\\\\n0 & b\n\\end{bmatrix},\n\\]\nand the following short-hand to construct them:\n\\[\n[x_{i}]_{i=1}^{n}=\\begin{bmatrix}x_{1}\\\\\n\\vdots\\\\\nx_{n}\n\\end{bmatrix},\\qquad[x_{i,j}]_{i,j=1}^{m,n}=\\begin{bmatrix}x_{1,1} & \\cdots & x_{1,n}\\\\\n\\vdots & \\ddots & \\vdots\\\\\nx_{m,1} & \\cdots & x_{m,n}\n\\end{bmatrix}.\n\\]\nDenote $\\one=[1,1,\\dots,1]^{T}$ as the vector-of-ones and $I=\\diag(\\one)$\nas the identity matrix; we will infer their dimensions from context.\nDenote $\\langle X,Y\\rangle=\\tr(X^{T}Y)$ and $\\|X\\|_{F}^{2}=\\langle X,X\\rangle$\nas the Frobenius inner product and norm. Denote $\\nnz(x)$ as the\nnumber of nonzero elements in $x$. The sets $\\S^{n}\\supset\\S_{+}^{n}$\nare the $n\\times n$ real symmetric matrices, and the corresponding\npositive semidefinite cone.\n\n\\textbf{Positive cones and projections.} The sets $\\R^{n}\\supset\\R_{+}^{n}$\nare the $n$ vectors, and the corresponding positive orthant. Denote\n$x_{+}=\\max\\{0,x\\}=\\arg\\min\\{\\|x-y\\|:y\\in\\R_{+}^{n}\\}$ as the projection\nonto the positive orthant. \n\n\\textbf{Vectorization and Kronecker product.} Denote $\\vect(X)$ as\nthe usual column-stacking vectorizating operator, and $\\mat(x)$ as\nits inverse (the dimensions of the matrix are inferred from context).\nThe Kronecker product $\\otimes$ is defined to satisfy the identity\n$\\vect(AXB^{T})=(B\\otimes A)\\vect(X)$.\n\n\\textbf{Pseudoinverse.} Denote the (Moore--Penrose) pseudoinverse\n$A^{\\dagger}\\eqdef VS^{-1}U^{T}$ of matrix $A=USV^{T}$ where $U^{T}U=V^{T}V=I_{r}$\nand $S\\succ0$. Define $0^{\\dagger}\\eqdef0$.\n\n\\textbf{Error vector $\\e$ and Jacobian matrix $\\J_{X}$}. Given fixed\n$X\\in\\R^{n\\times r}$ and $Z\\in\\R^{n\\times r}$, we denote $\\e=\\vect(XX^{T}-ZZ^{T})$\nand implicitly define $\\J_{X}\\in\\R^{n^{2}\\times nr}$ to satisfy $\\J_{X}\\vect(V)=\\vect(XV^{T}+VX^{T})$.\nNote that the associate adjoint operator satisfies $\\J_{X}^{T}\\vect(M)=\\vect((M+M^{T})X)$. \n\n\nWe will establish ``no spurious local minima'' guarantees by showing\nthat there cannot exist a counterexample that is simultaneously highly\noverparameterized, with a large value of $r/r^{\\star}$, and also\nwell-conditioned, with a small value of $\\kappa$. \n\\begin{definition}\nThe function $\\phi:\\S^{n}\\to\\R$ is said to be a $(\\kappa,r,r^{\\star})$-\\emph{counterexample}\nif it satisfies the following:\n\\begin{itemize}\n\\item The function $\\phi$ is twice-differentiable, $L$-smooth, $\\mu$-strongly\nconvex, and $L/\\mu\\le\\kappa.$\n\\item The minimizer $M^{\\star}=\\arg\\min_{M\\succeq0}\\phi(M)$ satisfies $\\rank(M^{\\star})=r^{\\star}\\le r$.\n\\item There exists an $n\\times r$ spurious point $X$ with $XX^{T}\\ne M^{\\star}$\nthat satisfies $\\nabla f(X)=0$ and $\\nabla^{2}f(X)\\succeq0$ for\n$f(U)\\eqdef\\phi(UU^{T})$. \n\\end{itemize}\n\\end{definition}\nFor a fixed $r$ and $r^{\\star}$, our central claim in \\thmref{main}\nis that the condition number $\\kappa^{\\star}$ of the \\emph{best-conditioned}\ncounterexample is exactly determined as \n\\begin{equation}\n\\kappa^{\\star}=\\inf_{\\phi:\\S^{n}\\to\\R}\\{\\kappa:\\phi\\text{ is }(\\kappa,r,r^{\\star})\\text{-counterexample}\\}=1+2\\sqrt{r/r^{\\star}}.\\label{eq:inf_kappa}\n\\end{equation}\nThe sufficient condition follows because there exists no $(\\kappa,r,r^{\\star})$-counterexample\nwith $\\kappa<1+2\\sqrt{r/r^{\\star}}$, and so for a fixed $\\kappa$,\nwe can always overparameterize $r/r^{\\star}>\\frac{1}{4}(\\kappa-1)^{2}$\nto eliminate all counterexamples. The neccessary condition holds because\na best-conditioned $(\\kappa^{\\star},r,r^{\\star})$-counterexample\ndoes exist.\n\nTo prove that $\\kappa^{\\star}$ is equal to the value stated in (\\ref{eq:inf_kappa}),\nwe proceed by reformulating (\\ref{eq:inf_kappa}) into two-stage optimization\nproblem \n\\begin{align*}\n\\kappa^{\\star} & =\\inf_{X,Z\\in\\R^{n\\times r}}\\{\\kappa(X,Z):\\rank(Z)=r^{\\star},\\;XX^{T}\\ne ZZ^{T}\\}\n\\end{align*}\nin which $\\kappa(X,Z)$ corresponds to the best-conditioned counterexample\nassociated with a \\emph{fixed} minimizer $ZZ^{T}$ and a \\emph{fixed\n}spurious point $X$:\n\\begin{equation}\n\\kappa(X,Z)=\\inf_{\\begin{subarray}{c}\n\\phi:\\S^{n}\\to\\R\\end{subarray}}\\left\\{ \\kappa:\\begin{array}{c}\n\\phi\\text{ is }1\\text{-strongly convex and }\\kappa\\text{-smooth,}\\\\\nZZ^{T}=\\arg\\min_{M\\succeq0}\\phi(M),\\\\\n\\nabla f(X)=0,\\;\\nabla^{2}f(X)\\succeq0\\text{ where }f(U)\\eqdef\\phi(UU^{T}).\n\\end{array}\\right\\} \\label{eq:mu_reform}\n\\end{equation}\nThis two-stage reformulation is inspired by \\citet{zhang2018much,zhang2019sharp},\nwho showed for a fixed $X,Z$ that finding the best-conditioned \\emph{quadratic}\ncounterexample $\\phi$ that satisfies $\\nabla\\phi(ZZ^{T})=0$ can\nbe exactly posed as a standard-form \\emph{semidefinite program} or\nSDP. Our proof extends this SDP formulation to a nonquadratic $\\phi$\nwith a possibly nonzero $\\nabla\\phi(ZZ^{T})\\succeq0$ at optimality\nby introducing a slack variable.   \n\n\n\\begin{remark}\nThe definition of $\\beta$ becomes ambiguous when $Z_{\\perp}=0$.\nThis is without loss of precision, because $Z_{\\perp}=0$ implies\n$\\alpha=0$, and therefore $\\kappa_{\\lb}(X,Z)=1$ regardless of the\nvalue of $\\beta$. \n\\end{remark}\n\n\nOur proof of \\thmref{EckartYoung} follows a trace inequality of Ruhe~\\citep{ruhe1970perturbation}\n(see also \\citep[Chapter~9]{marshall2011inequalities}).\\footnote{This simplified proof via Ruhe's inequality is due to an anonymous\nreviewer.} The inequality states that, for $n\\times n$ positive semidefinite\nmatrices $A,B$, we have\n\\begin{equation}\n\\sum_{i=1}^{n}\\lambda_{i}(A)\\lambda_{n-i+1}(B)\\le\\inner AB\\le\\sum_{i=1}^{n}\\lambda_{i}(A)\\lambda_{i}(B)\\label{eq:Ruhe}\n\\end{equation}\nwhere the eigenvalues are ordered $\\lambda_{1}(A)\\ge\\cdots\\ge\\lambda_{n}(A)$\nand $\\lambda_{1}(B)\\ge\\cdots\\ge\\lambda_{n}(B)$. \n\n\n\\begin{equation}\n\\J_{X}\\vect(V)=\\vect(XV^{T}+VX^{T}),\\qquad\\J_{X}^{T}\\vect(M)=\\vect((M+M^{T})X)\\label{eq:Jxdef}\n\\end{equation}\n\n\\begin{equation}\n\\mat((I-\\J_{X}\\J_{X}^{\\dagger})\\e)=(I-XX^{\\dagger})(XX^{T}-ZZ^{T})(I-XX^{\\dagger})=-Z_{\\perp}Z_{\\perp}^{T}\\label{eq:e2}\n\\end{equation}\n\n\\begin{claim}\nLet $W=\\sum_{j=1}^{m}\\vect(V_{j})\\vect(V_{j})^{T}$ where $V_{1},\\dots,V_{m}\\in\\R^{n\\times r}$.\nPartition $W$ into $r\\times r$ blocks of $n\\times n$, as in $W=[W_{i,j}]_{i,j=1}^{r}$\nfor $W_{i,j}\\in\\R^{n\\times n}$. Then, \n\\[\n\\sum_{i=1}^{r}W_{i,i}=\\sum_{j=1}^{m}V_{j}V_{j}^{T},\\qquad\\inner{\\J_{X}^{T}\\J_{X}}W=\\sum_{j=1}^{m}\\|XV_{j}^{T}+V_{j}X^{T}\\|_{F}^{2}.\n\\]\n\\end{claim}\n\n\\begin{gather}\n\\sum_{i=1}^{r}W_{i,i}=\\gamma_{2}\\cdot\\sum_{j=1}^{r}Z_{\\perp}e_{i}v_{r}^{T}v_{r}e_{i}^{T}Z_{\\perp}^{T}=\\gamma_{2}\\cdot Z_{\\perp}\\left(\\sum_{i=1}^{r}e_{i}e_{i}^{T}\\right)Z_{\\perp}^{T}=\\gamma_{2}\\cdot Z_{\\perp}Z_{\\perp}^{T}.\\label{eq:Wsol1}\n\\end{gather}\n\n\\begin{align}\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle & =\\gamma_{2}\\cdot\\sum_{i=1}^{m}\\|\\J_{X}\\vect(Z_{\\perp}e_{i}v_{r}^{T})\\|^{2}=\\gamma_{2}\\cdot\\sum_{i=1}^{m}\\|Xv_{r}e_{i}^{T}Z_{\\perp}^{T}+Z_{\\perp}e_{i}v_{r}^{T}X^{T}\\|_{F}^{2}\\nonumber \\\\\n & =\\gamma_{2}\\cdot\\sum_{i=1}^{m}2\\|Z_{\\perp}e_{i}v_{r}^{T}X^{T}\\|_{F}^{2}=2\\gamma_{2}\\cdot\\lambda_{\\min}(X^{T}X)\\|Z_{\\perp}\\|_{F}^{2}\\label{eq:Wsol2}\n\\end{align}\n\nFinally, we evoke the following lemma, which was previously used in\nthe proof of \\citet[Theorem~12]{zhang2019sharp} to fully solve the\nrank-1 case in closed-form.\n\\begin{lemma}\n\\label{lem:optim}Given $0\\le\\alpha<1$ and $\\beta\\ge0$, let $\\gamma_{\\alpha,\\beta}:\\R_{+}\\to[0,1]$\nsatisfy $\\gamma_{\\alpha,\\beta}(t)=\\alpha\\cdot(t/\\beta)+\\sqrt{1-\\alpha^{2}}\\sqrt{1-t^{2}/\\beta^{2}}$\nfor $t\\le\\alpha\\beta$. Then, we have\n\\[\n\\max_{0\\le t\\le\\alpha\\beta}\\frac{1+\\gamma_{\\alpha,\\beta}(t)}{2t+1-\\gamma_{\\alpha,\\beta}(t)}=\\begin{cases}\n{\\displaystyle \\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}}} & \\text{if }\\beta\\ge{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\\\\\n{\\displaystyle \\frac{1-\\alpha\\beta}{\\beta(\\alpha-\\beta)}} & \\text{if }\\beta\\le{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}}.\n\\end{cases}\n\\]\n\\end{lemma}\n\n\nFinally, we give a constructive proof for the heuristic upper-bound\nin \\lemref{heuristic}. First, recall from \\lemref{sdp_formu} that\nour SDP upper-bound is defined as the following\n\\begin{equation}\n\\kappa_{\\ub}(X,Z)=\\min_{\\kappa,\\H}\\left\\{ \\kappa:I\\preceq\\H\\preceq\\kappa I,\\:\\J_{X}^{T}\\H\\e=0,\\;-2I_{r}\\otimes\\mat(\\H\\e)\\preceq\\J_{X}^{T}\\H\\J_{X}\\right\\} \\label{eq:kappaub_prob}\n\\end{equation}\nwhere $\\e=\\vect(XX^{T}-ZZ^{T})$ and its Jacobian $\\J_{X}$ is defined\nto satisfy $\\J_{X}\\vect(V)=\\vect(XV^{T}+VX^{T})$ for all $V$. Below,\nwe derive a set of sufficient conditions for a given $(\\kappa,\\H)$\nto be feasible for (\\ref{eq:kappaub_prob}). \n\n\n\\begin{example}[$\\kappa=1+2\\sqrt{r/r^{\\star}}$ counterexample]\n\\label{exa:overparam}Given $r,r^{\\star},n$ satisfying $1\\le r^{\\star}\\le r<n$,\nlet $u_{1},u_{2},\\ldots,u_{n}$ be an orthonormal basis for $\\R^{n}$.\nDefine the quadratic function\n\\[\n\\phi_{0}(M)=\\frac{1}{2}\\sum_{i=1}^{n}\\sum_{j=1}^{n}\\left|\\langle A^{(i,j)},M-M^{\\star}\\rangle\\right|^{2}\\quad\\text{ where }M^{\\star}=\\sum_{k=r+1}^{r+r^{\\star}}u_{k}u_{k}^{T},\n\\]\nand $A^{(i,j)}=\\sqrt{\\kappa}\\cdot u_{i}u_{j}^{T}$ for all $1\\le i,j\\le n$\nexcept the following\n\\begin{gather*}\nA^{(1,1)}=\\sqrt{\\frac{\\kappa}{2r}}\\sum_{j=1}^{r}u_{j}u_{j}^{T}+\\sqrt{\\frac{\\kappa}{2r^{\\star}}}\\sum_{k=r+1}^{r+r^{\\star}}u_{k}u_{k}^{T},\\\\\nA^{(2,2)}=\\frac{1}{\\sqrt{2r}}\\sum_{j=1}^{r}u_{j}u_{j}^{T}-\\frac{1}{\\sqrt{2r^{\\star}}}\\sum_{k=r+1}^{r+r^{\\star}}u_{k}u_{k}^{T},\\\\\nA^{(i,i)}=u_{\\pi(i)}u_{\\pi(i)}^{T}-\\sum_{j=1}^{i-1}\\frac{A^{(j,j)}}{\\|A^{(j,j)}\\|_{F}^{2}}\\inner{A^{(j,j)}}{u_{\\pi(i)}u_{\\pi(i)}^{T}}\\text{ for }i\\in\\{3,\\dots,r+r^{\\star}\\},\n\\end{gather*}\nwith the ordering $\\pi=(1,r+1,2,3,\\dots,r,r+2,r+3,\\dots,r+r^{\\star})$.\nNote that $\\{A^{(i,j)}\\}$ is an orthogonal basis for $\\R^{n\\times n}$\nwith $1\\le\\|A^{(i,j)}\\|_{F}^{2}\\le\\kappa$, so the quadratic function\n$\\phi_{0}$ is twice-differentiable, $1$-strongly convex and $\\kappa$-smooth,\n\\[\n\\|E\\|_{F}^{2}\\le\\inner{\\nabla^{2}\\phi_{0}(M)[E]}E\\le\\kappa\\|E\\|_{F}^{2}\\qquad\\text{for all }M\\in\\R^{n\\times n}.\n\\]\nIts minimizer $M^{\\star}=\\arg\\min_{M}\\phi_{0}(M)$ satisfies $r^{\\star}=\\rank(M^{\\star}),$\nbut its corresponding $f_{0}(U)\\eqdef\\phi_{0}(UU^{T})$ admits the\n$n\\times r$ matrix $X=\\frac{1}{\\sqrt{1+\\sqrt{r/r^{\\star}}}}[u_{1},\\dots,u_{r}]$\nas a second-order critical point \n\\begin{align*}\nf_{0}(X)-\\min_{U}f_{0}(U) & =\\left(1+\\frac{1}{1+\\sqrt{r^{\\star}/r}}\\right)\\frac{\\|M^{\\star}\\|_{F}^{2}}{2}, & \\nabla f_{0}(X) & =0, & \\nabla^{2}f_{0}(X) & \\succeq0.\n\\end{align*}\n \\qed\n\\end{example}","units":[{"raw_tex":"Consider minimizing a convex objective $\\phi$ over an $n\\times n$\npositive semidefinite matrix $M\\succeq0$, as in\n\\begin{equation}\nM^{\\star}=\\quad\\mathrm{minimize}\\quad\\phi(M)\\quad\\text{over }\\quad M\\succeq0,\\tag{P}\\label{eq:cvx}\n\\end{equation}\nIn practice, the matrix order $n$ is often so large---from tens\nof thousands to hundreds of millions---that even explicitly forming\nthe $n^{2}$ elements of the $n\\times n$ matrix variable $M$ would\nbe intractable. Where $r^{\\star}=\\rank(M^{\\star})$ is known a priori\nto be small, the standard approach is the nonconvex Burer--Monteiro\nfactorization~\\citep{burer2003nonlinear}, which rewrites $M=XX^{T}$\nand then directly optimizes over $X$ using a local optimization algorithm:\n\\begin{equation}\nX^{\\star}=\\quad\\mathrm{minimize}\\quad f(X)\\eqdef\\phi(XX^{T})\\text{ where }X\\text{ is }n\\times r\\text{ and }r\\ge r^{\\star}.\\tag{BM}\\label{eq:ncvx}\n\\end{equation}\nThis way, the semidefinite constraint $M=XX^{T}\\succeq0$ is automatically\nenforced for free. Moreover, by using a small search rank $r\\ll n$,\nthe number of variables is reduced to $O(n)$. However, this improved\nscalability comes at the cost of giving up the convexity of (\\ref{eq:cvx}).\nIn principle, local optimization can fail by getting stuck at a \\emph{spurious}\nlocal minimum of (\\ref{eq:ncvx})---a local minimum that is strictly\nworse than that of the global minimum. In practice, however, there\nseems to be many situations where this failure mode does not occur~\\citep{candes2015phase,rosen2019se,zhang2019spurious}.\n","start":5847,"end":7372},{"raw_tex":"To state this precisely, suppose that $\\phi$ is twice-differentiable,\n$L$-smooth, and $\\mu$-strongly convex over the entire set of $n\\times n$\nreal symmetric matrices $\\S^{n}$, as in\n\\[\n\\mu\\|E\\|_{F}^{2}\\le\\inner{\\nabla^{2}\\phi(M)[E]}E\\le L\\|E\\|_{F}^{2}\\qquad\\text{for all }M\\in\\S^{n},\n\\]\nwhere $\\inner EF\\eqdef\\tr(E^{T}F)$ and $\\|E\\|_{F}\\eqdef\\sqrt{\\inner EE}$\ndenote the matrix Euclidean inner product and norm respectively. \\citet{bhojanapalli2016global}\npointed out that if $\\phi$ has condition number $L/\\mu<3/2$, then\nthe usual second-order necessary conditions for \\emph{local} optimality\nin $f$ are also sufficient for \\emph{global} optimality, meaning\nthat\n\\[\n\\nabla f(X)=0,\\quad\\nabla^{2}f(X)\\succeq0\\quad\\iff\\quad f(X)=\\min_{U}f(U).\n\\]\n","start":7706,"end":8452},{"raw_tex":"\\textbf{Basic linear algebra.} Lower-case letters are vectors and\nupper-case letters are matrices. We use ``MATLAB notation'' in concatenating\nvectors and matrices:\n\n\\[\n[a,b]=\\begin{bmatrix}a & b\\end{bmatrix},\\qquad[a;b]=\\begin{bmatrix}a\\\\\nb\n\\end{bmatrix},\\qquad\\diag(a,b)=\\begin{bmatrix}a & 0\\\\\n0 & b\n\\end{bmatrix},\n\\]\nand the following short-hand to construct them:\n\\[\n[x_{i}]_{i=1}^{n}=\\begin{bmatrix}x_{1}\\\\\n\\vdots\\\\\nx_{n}\n\\end{bmatrix},\\qquad[x_{i,j}]_{i,j=1}^{m,n}=\\begin{bmatrix}x_{1,1} & \\cdots & x_{1,n}\\\\\n\\vdots & \\ddots & \\vdots\\\\\nx_{m,1} & \\cdots & x_{m,n}\n\\end{bmatrix}.\n\\]\nDenote $\\one=[1,1,\\dots,1]^{T}$ as the vector-of-ones and $I=\\diag(\\one)$\nas the identity matrix; we will infer their dimensions from context.\nDenote $\\langle X,Y\\rangle=\\tr(X^{T}Y)$ and $\\|X\\|_{F}^{2}=\\langle X,X\\rangle$\nas the Frobenius inner product and norm. Denote $\\nnz(x)$ as the\nnumber of nonzero elements in $x$. The sets $\\S^{n}\\supset\\S_{+}^{n}$\nare the $n\\times n$ real symmetric matrices, and the corresponding\npositive semidefinite cone.\n\n\\textbf{Positive cones and projections.} The sets $\\R^{n}\\supset\\R_{+}^{n}$\nare the $n$ vectors, and the corresponding positive orthant. Denote\n$x_{+}=\\max\\{0,x\\}=\\arg\\min\\{\\|x-y\\|:y\\in\\R_{+}^{n}\\}$ as the projection\nonto the positive orthant. \n\n\\textbf{Vectorization and Kronecker product.} Denote $\\vect(X)$ as\nthe usual column-stacking vectorizating operator, and $\\mat(x)$ as\nits inverse (the dimensions of the matrix are inferred from context).\nThe Kronecker product $\\otimes$ is defined to satisfy the identity\n$\\vect(AXB^{T})=(B\\otimes A)\\vect(X)$.\n\n\\textbf{Pseudoinverse.} Denote the (Moore--Penrose) pseudoinverse\n$A^{\\dagger}\\eqdef VS^{-1}U^{T}$ of matrix $A=USV^{T}$ where $U^{T}U=V^{T}V=I_{r}$\nand $S\\succ0$. Define $0^{\\dagger}\\eqdef0$.\n\n\\textbf{Error vector $\\e$ and Jacobian matrix $\\J_{X}$}. Given fixed\n$X\\in\\R^{n\\times r}$ and $Z\\in\\R^{n\\times r}$, we denote $\\e=\\vect(XX^{T}-ZZ^{T})$\nand implicitly define $\\J_{X}\\in\\R^{n^{2}\\times nr}$ to satisfy $\\J_{X}\\vect(V)=\\vect(XV^{T}+VX^{T})$.\nNote that the associate adjoint operator satisfies $\\J_{X}^{T}\\vect(M)=\\vect((M+M^{T})X)$. \n","start":32917,"end":35055},{"raw_tex":"We will establish ``no spurious local minima'' guarantees by showing\nthat there cannot exist a counterexample that is simultaneously highly\noverparameterized, with a large value of $r/r^{\\star}$, and also\nwell-conditioned, with a small value of $\\kappa$. \n\\begin{definition}\nThe function $\\phi:\\S^{n}\\to\\R$ is said to be a $(\\kappa,r,r^{\\star})$-\\emph{counterexample}\nif it satisfies the following:\n\\begin{itemize}\n\\item The function $\\phi$ is twice-differentiable, $L$-smooth, $\\mu$-strongly\nconvex, and $L/\\mu\\le\\kappa.$\n\\item The minimizer $M^{\\star}=\\arg\\min_{M\\succeq0}\\phi(M)$ satisfies $\\rank(M^{\\star})=r^{\\star}\\le r$.\n\\item There exists an $n\\times r$ spurious point $X$ with $XX^{T}\\ne M^{\\star}$\nthat satisfies $\\nabla f(X)=0$ and $\\nabla^{2}f(X)\\succeq0$ for\n$f(U)\\eqdef\\phi(UU^{T})$. \n\\end{itemize}\n\\end{definition}\nFor a fixed $r$ and $r^{\\star}$, our central claim in \\thmref{main}\nis that the condition number $\\kappa^{\\star}$ of the \\emph{best-conditioned}\ncounterexample is exactly determined as \n\\begin{equation}\n\\kappa^{\\star}=\\inf_{\\phi:\\S^{n}\\to\\R}\\{\\kappa:\\phi\\text{ is }(\\kappa,r,r^{\\star})\\text{-counterexample}\\}=1+2\\sqrt{r/r^{\\star}}.\\label{eq:inf_kappa}\n\\end{equation}\nThe sufficient condition follows because there exists no $(\\kappa,r,r^{\\star})$-counterexample\nwith $\\kappa<1+2\\sqrt{r/r^{\\star}}$, and so for a fixed $\\kappa$,\nwe can always overparameterize $r/r^{\\star}>\\frac{1}{4}(\\kappa-1)^{2}$\nto eliminate all counterexamples. The neccessary condition holds because\na best-conditioned $(\\kappa^{\\star},r,r^{\\star})$-counterexample\ndoes exist.\n\nTo prove that $\\kappa^{\\star}$ is equal to the value stated in (\\ref{eq:inf_kappa}),\nwe proceed by reformulating (\\ref{eq:inf_kappa}) into two-stage optimization\nproblem \n\\begin{align*}\n\\kappa^{\\star} & =\\inf_{X,Z\\in\\R^{n\\times r}}\\{\\kappa(X,Z):\\rank(Z)=r^{\\star},\\;XX^{T}\\ne ZZ^{T}\\}\n\\end{align*}\nin which $\\kappa(X,Z)$ corresponds to the best-conditioned counterexample\nassociated with a \\emph{fixed} minimizer $ZZ^{T}$ and a \\emph{fixed\n}spurious point $X$:\n\\begin{equation}\n\\kappa(X,Z)=\\inf_{\\begin{subarray}{c}\n\\phi:\\S^{n}\\to\\R\\end{subarray}}\\left\\{ \\kappa:\\begin{array}{c}\n\\phi\\text{ is }1\\text{-strongly convex and }\\kappa\\text{-smooth,}\\\\\nZZ^{T}=\\arg\\min_{M\\succeq0}\\phi(M),\\\\\n\\nabla f(X)=0,\\;\\nabla^{2}f(X)\\succeq0\\text{ where }f(U)\\eqdef\\phi(UU^{T}).\n\\end{array}\\right\\} \\label{eq:mu_reform}\n\\end{equation}\nThis two-stage reformulation is inspired by \\citet{zhang2018much,zhang2019sharp},\nwho showed for a fixed $X,Z$ that finding the best-conditioned \\emph{quadratic}\ncounterexample $\\phi$ that satisfies $\\nabla\\phi(ZZ^{T})=0$ can\nbe exactly posed as a standard-form \\emph{semidefinite program} or\nSDP. Our proof extends this SDP formulation to a nonquadratic $\\phi$\nwith a possibly nonzero $\\nabla\\phi(ZZ^{T})\\succeq0$ at optimality\nby introducing a slack variable.   \n","start":35109,"end":37972},{"raw_tex":"\\begin{remark}\nThe definition of $\\beta$ becomes ambiguous when $Z_{\\perp}=0$.\nThis is without loss of precision, because $Z_{\\perp}=0$ implies\n$\\alpha=0$, and therefore $\\kappa_{\\lb}(X,Z)=1$ regardless of the\nvalue of $\\beta$. \n\\end{remark}\n","start":44417,"end":44659},{"raw_tex":"Our proof of \\thmref{EckartYoung} follows a trace inequality of Ruhe~\\citep{ruhe1970perturbation}\n(see also \\citep[Chapter~9]{marshall2011inequalities}).\\footnote{This simplified proof via Ruhe's inequality is due to an anonymous\nreviewer.} The inequality states that, for $n\\times n$ positive semidefinite\nmatrices $A,B$, we have\n\\begin{equation}\n\\sum_{i=1}^{n}\\lambda_{i}(A)\\lambda_{n-i+1}(B)\\le\\inner AB\\le\\sum_{i=1}^{n}\\lambda_{i}(A)\\lambda_{i}(B)\\label{eq:Ruhe}\n\\end{equation}\nwhere the eigenvalues are ordered $\\lambda_{1}(A)\\ge\\cdots\\ge\\lambda_{n}(A)$\nand $\\lambda_{1}(B)\\ge\\cdots\\ge\\lambda_{n}(B)$. \n","start":51731,"end":52339},{"raw_tex":"\\begin{equation}\n\\J_{X}\\vect(V)=\\vect(XV^{T}+VX^{T}),\\qquad\\J_{X}^{T}\\vect(M)=\\vect((M+M^{T})X)\\label{eq:Jxdef}\n\\end{equation}","start":60030,"end":60156},{"raw_tex":"\\begin{equation}\n\\mat((I-\\J_{X}\\J_{X}^{\\dagger})\\e)=(I-XX^{\\dagger})(XX^{T}-ZZ^{T})(I-XX^{\\dagger})=-Z_{\\perp}Z_{\\perp}^{T}\\label{eq:e2}\n\\end{equation}","start":62940,"end":63091},{"raw_tex":"\\begin{claim}\nLet $W=\\sum_{j=1}^{m}\\vect(V_{j})\\vect(V_{j})^{T}$ where $V_{1},\\dots,V_{m}\\in\\R^{n\\times r}$.\nPartition $W$ into $r\\times r$ blocks of $n\\times n$, as in $W=[W_{i,j}]_{i,j=1}^{r}$\nfor $W_{i,j}\\in\\R^{n\\times n}$. Then, \n\\[\n\\sum_{i=1}^{r}W_{i,i}=\\sum_{j=1}^{m}V_{j}V_{j}^{T},\\qquad\\inner{\\J_{X}^{T}\\J_{X}}W=\\sum_{j=1}^{m}\\|XV_{j}^{T}+V_{j}X^{T}\\|_{F}^{2}.\n\\]\n\\end{claim}","start":63317,"end":63700},{"raw_tex":"\\begin{gather}\n\\sum_{i=1}^{r}W_{i,i}=\\gamma_{2}\\cdot\\sum_{j=1}^{r}Z_{\\perp}e_{i}v_{r}^{T}v_{r}e_{i}^{T}Z_{\\perp}^{T}=\\gamma_{2}\\cdot Z_{\\perp}\\left(\\sum_{i=1}^{r}e_{i}e_{i}^{T}\\right)Z_{\\perp}^{T}=\\gamma_{2}\\cdot Z_{\\perp}Z_{\\perp}^{T}.\\label{eq:Wsol1}\n\\end{gather}","start":64405,"end":64670},{"raw_tex":"\\begin{align}\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle & =\\gamma_{2}\\cdot\\sum_{i=1}^{m}\\|\\J_{X}\\vect(Z_{\\perp}e_{i}v_{r}^{T})\\|^{2}=\\gamma_{2}\\cdot\\sum_{i=1}^{m}\\|Xv_{r}e_{i}^{T}Z_{\\perp}^{T}+Z_{\\perp}e_{i}v_{r}^{T}X^{T}\\|_{F}^{2}\\nonumber \\\\\n & =\\gamma_{2}\\cdot\\sum_{i=1}^{m}2\\|Z_{\\perp}e_{i}v_{r}^{T}X^{T}\\|_{F}^{2}=2\\gamma_{2}\\cdot\\lambda_{\\min}(X^{T}X)\\|Z_{\\perp}\\|_{F}^{2}\\label{eq:Wsol2}\n\\end{align}","start":64689,"end":65086},{"raw_tex":"Finally, we evoke the following lemma, which was previously used in\nthe proof of \\citet[Theorem~12]{zhang2019sharp} to fully solve the\nrank-1 case in closed-form.\n\\begin{lemma}\n\\label{lem:optim}Given $0\\le\\alpha<1$ and $\\beta\\ge0$, let $\\gamma_{\\alpha,\\beta}:\\R_{+}\\to[0,1]$\nsatisfy $\\gamma_{\\alpha,\\beta}(t)=\\alpha\\cdot(t/\\beta)+\\sqrt{1-\\alpha^{2}}\\sqrt{1-t^{2}/\\beta^{2}}$\nfor $t\\le\\alpha\\beta$. Then, we have\n\\[\n\\max_{0\\le t\\le\\alpha\\beta}\\frac{1+\\gamma_{\\alpha,\\beta}(t)}{2t+1-\\gamma_{\\alpha,\\beta}(t)}=\\begin{cases}\n{\\displaystyle \\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}}} & \\text{if }\\beta\\ge{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\\\\\n{\\displaystyle \\frac{1-\\alpha\\beta}{\\beta(\\alpha-\\beta)}} & \\text{if }\\beta\\le{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}}.\n\\end{cases}\n\\]\n\\end{lemma}\n","start":68018,"end":68842},{"raw_tex":"Finally, we give a constructive proof for the heuristic upper-bound\nin \\lemref{heuristic}. First, recall from \\lemref{sdp_formu} that\nour SDP upper-bound is defined as the following\n\\begin{equation}\n\\kappa_{\\ub}(X,Z)=\\min_{\\kappa,\\H}\\left\\{ \\kappa:I\\preceq\\H\\preceq\\kappa I,\\:\\J_{X}^{T}\\H\\e=0,\\;-2I_{r}\\otimes\\mat(\\H\\e)\\preceq\\J_{X}^{T}\\H\\J_{X}\\right\\} \\label{eq:kappaub_prob}\n\\end{equation}\nwhere $\\e=\\vect(XX^{T}-ZZ^{T})$ and its Jacobian $\\J_{X}$ is defined\nto satisfy $\\J_{X}\\vect(V)=\\vect(XV^{T}+VX^{T})$ for all $V$. Below,\nwe derive a set of sufficient conditions for a given $(\\kappa,\\H)$\nto be feasible for (\\ref{eq:kappaub_prob}). \n","start":69136,"end":69778},{"raw_tex":"\\begin{example}[$\\kappa=1+2\\sqrt{r/r^{\\star}}$ counterexample]\n\\label{exa:overparam}Given $r,r^{\\star},n$ satisfying $1\\le r^{\\star}\\le r<n$,\nlet $u_{1},u_{2},\\ldots,u_{n}$ be an orthonormal basis for $\\R^{n}$.\nDefine the quadratic function\n\\[\n\\phi_{0}(M)=\\frac{1}{2}\\sum_{i=1}^{n}\\sum_{j=1}^{n}\\left|\\langle A^{(i,j)},M-M^{\\star}\\rangle\\right|^{2}\\quad\\text{ where }M^{\\star}=\\sum_{k=r+1}^{r+r^{\\star}}u_{k}u_{k}^{T},\n\\]\nand $A^{(i,j)}=\\sqrt{\\kappa}\\cdot u_{i}u_{j}^{T}$ for all $1\\le i,j\\le n$\nexcept the following\n\\begin{gather*}\nA^{(1,1)}=\\sqrt{\\frac{\\kappa}{2r}}\\sum_{j=1}^{r}u_{j}u_{j}^{T}+\\sqrt{\\frac{\\kappa}{2r^{\\star}}}\\sum_{k=r+1}^{r+r^{\\star}}u_{k}u_{k}^{T},\\\\\nA^{(2,2)}=\\frac{1}{\\sqrt{2r}}\\sum_{j=1}^{r}u_{j}u_{j}^{T}-\\frac{1}{\\sqrt{2r^{\\star}}}\\sum_{k=r+1}^{r+r^{\\star}}u_{k}u_{k}^{T},\\\\\nA^{(i,i)}=u_{\\pi(i)}u_{\\pi(i)}^{T}-\\sum_{j=1}^{i-1}\\frac{A^{(j,j)}}{\\|A^{(j,j)}\\|_{F}^{2}}\\inner{A^{(j,j)}}{u_{\\pi(i)}u_{\\pi(i)}^{T}}\\text{ for }i\\in\\{3,\\dots,r+r^{\\star}\\},\n\\end{gather*}\nwith the ordering $\\pi=(1,r+1,2,3,\\dots,r,r+2,r+3,\\dots,r+r^{\\star})$.\nNote that $\\{A^{(i,j)}\\}$ is an orthogonal basis for $\\R^{n\\times n}$\nwith $1\\le\\|A^{(i,j)}\\|_{F}^{2}\\le\\kappa$, so the quadratic function\n$\\phi_{0}$ is twice-differentiable, $1$-strongly convex and $\\kappa$-smooth,\n\\[\n\\|E\\|_{F}^{2}\\le\\inner{\\nabla^{2}\\phi_{0}(M)[E]}E\\le\\kappa\\|E\\|_{F}^{2}\\qquad\\text{for all }M\\in\\R^{n\\times n}.\n\\]\nIts minimizer $M^{\\star}=\\arg\\min_{M}\\phi_{0}(M)$ satisfies $r^{\\star}=\\rank(M^{\\star}),$\nbut its corresponding $f_{0}(U)\\eqdef\\phi_{0}(UU^{T})$ admits the\n$n\\times r$ matrix $X=\\frac{1}{\\sqrt{1+\\sqrt{r/r^{\\star}}}}[u_{1},\\dots,u_{r}]$\nas a second-order critical point \n\\begin{align*}\nf_{0}(X)-\\min_{U}f_{0}(U) & =\\left(1+\\frac{1}{1+\\sqrt{r^{\\star}/r}}\\right)\\frac{\\|M^{\\star}\\|_{F}^{2}}{2}, & \\nabla f_{0}(X) & =0, & \\nabla^{2}f_{0}(X) & \\succeq0.\n\\end{align*}\n \\qed\n\\end{example}","start":73342,"end":75217}]},"statement":{"raw_tex":"\\begin{theorem}[Overparameterization]\n\\label{thm:main}Let $\\phi:\\S^{n}\\to\\R$ be twice-differentiable,\n$L$-smooth and $\\mu$-strongly convex, let the minimizer $M^{\\star}=\\arg\\min_{M\\succeq0}\\phi(M)$\nhave true rank $r^{\\star}=\\rank(M^{\\star})$.\n\\begin{itemize}\n\\item (Sufficiency) If $r>\\frac{1}{4}(L/\\mu-1)^{2}\\,r^{\\star}$ and $r^{\\star}\\le r<n$,\nthen the Burer--Monteiro function $f:\\R^{n\\times r}\\to\\R$ defined\nas $f(U)\\eqdef\\phi(UU^{T})$ has no spurious local minima:\n\\[\n\\nabla f(X)=0,\\quad\\nabla^{2}f(X)\\succeq0\\quad\\iff\\quad f(X)=\\min_{U}f(U).\n\\]\n\\item (Necessity) If $r\\le\\frac{1}{4}(L/\\mu-1)^{2}\\,r^{\\star}$ and $r^{\\star}\\le r<n$,\nthen there exists an $L$-smooth and $\\mu$-strongly convex quadratic\ncounterexample $\\phi_{0}$ whose minimizer $M_{0}^{\\star}=\\arg\\min_{M\\succeq0}\\phi_{0}(M)$\nhas $\\rank(M_{0}^{\\star})=r^{\\star}$, but whose $f_{0}(U)\\eqdef\\phi_{0}(UU^{T})$\nadmits an $n\\times r$ spurious second-order point $X_{\\spur}$:\n\\[\n\\nabla f_{0}(X_{\\spur})=0,\\quad\\nabla^{2}f_{0}(X_{\\spur})\\succeq0,\\quad f_{0}(X_{\\spur})-\\min_{U}f_{0}(U)>\\frac{\\mu}{2}\\cdot\\|M_{0}^{\\star}\\|_{F}^{2}.\n\\]\n\\end{itemize}\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}[Overparameterization]\n\\label{thm:main}Let $\\phi:\\S^{n}\\to\\R$ be twice-differentiable,\n$L$-smooth and $\\mu$-strongly convex, let the minimizer $M^{\\star}=\\arg\\min_{M\\succeq0}\\phi(M)$\nhave true rank $r^{\\star}=\\rank(M^{\\star})$.\n\\begin{itemize}\n\\item (Sufficiency) If $r>\\frac{1}{4}(L/\\mu-1)^{2}\\,r^{\\star}$ and $r^{\\star}\\le r<n$,\nthen the Burer--Monteiro function $f:\\R^{n\\times r}\\to\\R$ defined\nas $f(U)\\eqdef\\phi(UU^{T})$ has no spurious local minima:\n\\[\n\\nabla f(X)=0,\\quad\\nabla^{2}f(X)\\succeq0\\quad\\iff\\quad f(X)=\\min_{U}f(U).\n\\]\n\\item (Necessity) If $r\\le\\frac{1}{4}(L/\\mu-1)^{2}\\,r^{\\star}$ and $r^{\\star}\\le r<n$,\nthen there exists an $L$-smooth and $\\mu$-strongly convex quadratic\ncounterexample $\\phi_{0}$ whose minimizer $M_{0}^{\\star}=\\arg\\min_{M\\succeq0}\\phi_{0}(M)$\nhas $\\rank(M_{0}^{\\star})=r^{\\star}$, but whose $f_{0}(U)\\eqdef\\phi_{0}(UU^{T})$\nadmits an $n\\times r$ spurious second-order point $X_{\\spur}$:\n\\[\n\\nabla f_{0}(X_{\\spur})=0,\\quad\\nabla^{2}f_{0}(X_{\\spur})\\succeq0,\\quad f_{0}(X_{\\spur})-\\min_{U}f_{0}(U)>\\frac{\\mu}{2}\\cdot\\|M_{0}^{\\star}\\|_{F}^{2}.\n\\]\n\\end{itemize}\n\\end{theorem}","start":12631,"end":13755}]},"proof":{"raw_tex":"\\begin{lemma}[SDP formulation]\n\\label{lem:sdp_formu}Define $\\kappa(X,Z)$ as in (\\ref{eq:mu_reform}).\nWe have $\\kappa_{\\ub}(X,Z)\\ge\\kappa(X,Z)\\ge\\kappa_{\\lb}(X,Z)$ where\n\\begin{gather*}\n\\kappa_{\\ub}(X,Z)\\eqdef\\min_{\\kappa,\\H}\\left\\{ \\kappa:I\\preceq\\H\\preceq\\kappa I,\\;\\J_{X}^{T}\\H\\e=0,\\;-2I_{r}\\otimes\\mat(\\H\\e)\\preceq\\J_{X}^{T}\\H\\J_{X}\\right\\} \\\\\n\\kappa_{\\lb}(X,Z)\\eqdef\\min_{\\kappa,s,\\H}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H\\preceq\\kappa I,\\quad\\mat(s)\\succeq0,\\quad\\J_{Z}^{T}s=0,\\\\\n\\J_{X}^{T}(\\H\\e+s)=0,\\quad-2I_{r}\\otimes\\mat(\\H\\e+s)\\preceq\\kappa\\J_{X}^{T}\\J_{X}\n\\end{array}\\right\\} \n\\end{gather*}\nand $\\e=\\vect(XX^{T}-ZZ^{T})$ and $\\J_{U}\\vect(V)=\\vect(UV^{T}+VU^{T})$\nfor all $V$.\n\\end{lemma}\n\n\\begin{proof}\nDenote $S_{M}$ as the convex gradient $\\nabla\\phi(M)$ evaluated\nat $M$, and denote $\\H_{M}$ as the matrix representation of the\nconvex Hessian operator $\\nabla^{2}\\phi(M)$ evaluated at $M$. It\nfollows from $\\inner{\\nabla^{2}\\phi(M)[E]}E=\\vect(E)^{T}\\H_{M}\\vect(E)$\nthat: \n\\begin{equation}\n\\phi\\text{ is }1\\text{-strongly convex and }\\kappa\\text{-smooth}\\iff I\\preceq\\H_{M}\\preceq\\kappa I\\quad\\text{for all }M\\in\\S^{n}.\\label{eq:cond}\n\\end{equation}\nAdditionally, the Karush--Kuhn--Tucker conditions at $M^{\\star}=ZZ^{T}$\nread:\n\\begin{equation}\nZZ^{T}=\\arg\\min_{M\\succeq0}\\phi(M)\\quad\\iff\\quad S_{ZZ^{T}}\\succeq0,\\quad S_{ZZ^{T}}Z=0.\\label{eq:kktZZ}\n\\end{equation}\nNote that strong duality holds in the above because $ZZ^{T}+\\epsilon I$\nis a strictly feasible point with a bounded objective due to $\\kappa$-smoothness.\nNext, we observe that the two gradient evaluations $S_{XX^{T}}$ and\n$S_{ZZ^{T}}$ are related via the fundamental theorem of calculus:\n\\begin{gather*}\n\\nabla\\phi(XX^{T})=\\nabla\\phi(ZZ^{T})+\\int_{0}^{1}\\nabla^{2}\\phi((1-t)ZZ^{T}+tXX^{T})[XX^{T}-ZZ^{T}]\\,\\d t\\\\\n\\iff\\quad\\vect(S_{XX^{T}})=\\vect(S_{ZZ^{T}})+\\left[\\int_{0}^{1}\\H_{M(t)}\\,\\d t\\right]\\vect(XX^{T}-ZZ^{T}).\n\\end{gather*}\nDenoting $s=\\vect(S_{ZZ^{T}})$ and $\\H_{\\av}=\\int_{0}^{1}\\H_{M(t)}\\,\\d t$\nand $\\e=\\vect(XX^{T}-ZZ^{T})$ allows us to rewrite the directional\nderivatives of $f(U)\\eqdef\\phi(UU^{T})$ as follows: \n\\begin{align}\n\\inner{\\nabla f(X)}V & =\\vect(V)^{T}\\J_{X}^{T}(\\H_{\\av}\\e+s),\\nonumber \\\\\n\\inner{\\nabla^{2}f(X)[V]}V & =\\vect(V)^{T}[2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)+\\J_{X}^{T}\\H_{XX^{T}}\\J_{X}]\\,\\vect(V).\\label{eq:vecdf}\n\\end{align}\nSubstituting (\\ref{eq:cond}), (\\ref{eq:vecdf}), and (\\ref{eq:kktZZ})\ninto (\\ref{eq:mu_reform}) yields exactly\n\\begin{equation}\n\\kappa(X,Z)=\\inf_{\\kappa,\\phi:\\S^{n}\\to\\R}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H_{M}\\preceq\\kappa I\\quad\\text{for all }M\\in\\S^{n},\\\\\n\\mat(s)\\succeq0,\\quad\\J_{Z}^{T}s=0,\\quad\\J_{X}^{T}(\\H_{\\av}\\e+s)=0,\\\\\n-2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)\\preceq\\J_{X}^{T}\\H_{XX^{T}}\\J_{X}.\n\\end{array}\\right\\} \\label{eq:kappa_sdp1}\n\\end{equation}\nWe obtain the upper-bound $\\kappa_{\\ub}(X,Z)$ from $\\kappa(X,Z)$\nin  (\\ref{eq:kappa_sdp1}) by fixing $\\H_{M}\\equiv\\H$ for all $M$,\nand $S_{ZZ^{T}}\\equiv\\mat(s)=0$. We obtain the lower-bound $\\kappa_{\\lb}(X,Z)$\nfrom $\\kappa(X,Z)$ in (\\ref{eq:kappa_sdp1}) by substituting the\nrelaxation \n\\[\n-2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)\\quad\\preceq\\quad\\J_{X}^{T}\\H_{XX^{T}}\\J_{X}\\quad\\preceq\\quad\\kappa\\cdot\\J_{X}^{T}\\J_{X},\n\\]\nin order to avoid explicitly optimizing over $\\H_{XX^{T}}$:\n\\begin{align}\n\\kappa(X,Z) & \\ge\\inf_{\\kappa,\\phi:\\S^{n}\\to\\R}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H_{M}\\preceq\\kappa I\\quad\\text{for all }M\\in\\S^{n},\\\\\n\\mat(s)\\succeq0,\\quad\\J_{Z}^{T}s=0,\\quad\\J_{X}^{T}(\\H_{\\av}\\e+s)=0,\\\\\n-2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)\\preceq\\kappa\\cdot\\J_{X}^{T}\\J_{X}.\n\\end{array}\\right\\} \\label{eq:kappasdp2}\n\\end{align}\nFinally, we impose $I\\preceq\\H_{M}\\preceq\\kappa I$ over the average\nHessian $\\H_{\\av}=\\int_{0}^{1}\\H_{M(t)}\\,\\d t$ and then relax this\nconstraint over $\\H_{M}$ for all other values of $M$.\\qed\n\\end{proof}\n\n\\begin{lemma}[Heuristic upper-bound]\n\\label{lem:heuristic}Let $[Q_{1},Q_{2}]$ have orthonormal columns\nwith $Q_{1}\\in\\R^{n\\times r}$ and $Q_{2}\\in\\R^{n\\times r^{\\star}}$.\nThen, we have\n\\[\n\\kappa_{\\ub}(X,Z)\\le1+2\\sqrt{r/r^{\\star}}\\quad\\text{where }X=Q_{1},\\quad Z=\\sqrt{1+\\sqrt{r/r^{\\star}}}Q_{2}.\n\\]\n\\end{lemma}\n\n\\begin{lemma}[Closed-form lower-bound]\n\\label{lem:abdef}For $X,Z\\in\\R^{n\\times r}$ such that $XX^{T}\\ne ZZ^{T}$,\nlet $Z_{\\perp}=(I-XX^{\\dagger})Z$. We have\n\\begin{gather*}\n\\kappa_{\\lb}(X,Z)\\ge\\begin{cases}\n{\\displaystyle \\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}}} & \\text{if }\\beta\\ge{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\\\\\n{\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}} & \\text{if }\\beta\\le{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\n\\end{cases}\\\\\n\\text{ where }\\qquad\\alpha=\\frac{\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}{\\|XX^{T}-ZZ^{T}\\|_{F}},\\qquad\\beta=\\frac{\\lambda_{\\min}(X^{T}X)}{\\|XX^{T}-ZZ^{T}\\|_{F}}{\\displaystyle \\frac{\\tr(Z_{\\perp}Z_{\\perp}^{T})}{\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}}.\n\\end{gather*}\n\\end{lemma}\n\n\\begin{lemma}[Valid inequality]\n\\label{lem:ab1}For all $X,Z\\in\\R^{n\\times r}$ with $\\rank(Z)=r^{\\star}$,\nthe parameters $\\alpha$ and $\\beta$ defined in \\Lemref{abdef} satisfy\n$\\alpha^{2}+(r/r^{\\star})\\beta^{2}\\le1+[(\\beta-\\alpha)_{+}]^{2}.$\n\\end{lemma}\n\n\\begin{proof}\n[Theorem~\\ref{thm:main}]For a fixed $r$ and $r^{\\star}$, recall\nthat we have defined $\\kappa^{\\star}$ as the condition number of\nthe \\emph{best-conditioned} counterexample, and formulated it as the\noptimal value for a two-stage optimization problem of the following\nform\n\\begin{align*}\n\\kappa^{\\star} & =\\inf_{\\phi:\\S^{n}\\to\\R}\\{\\kappa:\\phi\\text{ is }(\\kappa,r,r^{\\star})\\text{-counterexample}\\}\\\\\n & =\\inf_{X,Z\\in\\R^{n\\times r}}\\{\\kappa(X,Z):\\rank(Z)=r^{\\star},\\;XX^{T}\\ne ZZ^{T}\\}\n\\end{align*}\nWe claim that the minimum of the following expression over $\\alpha,\\beta\\ge0$\n\\[\n\\gamma(\\alpha,\\beta)\\eqdef\\begin{cases}\n\\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}} & \\text{if }\\beta\\ge{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\\\\\n{\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}} & \\text{if }\\beta\\le{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\n\\end{cases}\n\\]\nsubject to the valid inequality from \\lemref{ab1} is given:\n\\begin{equation}\n1+2\\sqrt{r/r^{\\star}}=\\min_{\\alpha,\\beta\\ge0}\\{\\gamma(\\alpha,\\beta):\\alpha^{2}+(r/r^{\\star})\\min\\{\\alpha^{2},\\beta^{2}\\}\\le1\\}.\\label{eq:minab}\n\\end{equation}\nTaking (\\ref{eq:minab}) to be true, substituting into \\lemref{abdef}\nand evoking \\lemref{heuristic} yields\n\\[\n1+2\\sqrt{r/r^{\\star}}\\le\\inf_{\\begin{subarray}{c}\nXX^{T}\\ne ZZ^{T}\\\\\n\\rank(Z)=r^{\\star}\n\\end{subarray}}\\kappa_{\\lb}(X,Z)\\le\\kappa^{\\star}\\le\\kappa_{\\ub}(X,Z)\\le1+2\\sqrt{r/r^{\\star}}.\n\\]\nThe upper-bound proves necessity: if $r/r^{\\star}\\le\\frac{1}{4}(\\kappa-1)^{2}$,\nthen there exists a $(\\kappa,r,r^{\\star})$-counterexample. The lower-bound\nproves sufficiency: if $r>\\frac{1}{4}(\\kappa-1)^{2}\\,r^{\\star}$ and\n$r^{\\star}\\le r<n$, then there exists no $(\\kappa,r,r^{\\star})$-counterexample\nto refute a ``no spurious local minima'' guarantee.\n\nWe now prove the claim in (\\ref{eq:minab}). We first minimize $\\gamma(\\alpha,\\beta)$\nsubject to $\\beta\\ge\\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}$ and find\nvia monotonicity with respect to $\\alpha$ that\n\\[\n\\min_{0\\le\\alpha\\le1}\\left\\{ \\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}}:\\beta\\ge{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}}\\right\\} ={\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}}.\n\\]\nIndeed, the minimizer $\\alpha$ lies at the boundary $\\beta=\\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}=\\frac{1-\\sqrt{1-\\alpha^{2}}}{\\alpha}$,\nwhere\n\\[\n\\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}}=\\frac{\\sqrt{1-\\alpha^{2}}+(1-\\alpha^{2})}{\\sqrt{1-\\alpha^{2}}-(1-\\alpha^{2})}=\\frac{1+\\sqrt{1-\\alpha^{2}}-\\alpha^{2}}{\\alpha^{2}-(1-\\sqrt{1-\\alpha^{2}})}={\\displaystyle \\frac{\\beta^{-1}-\\alpha}{\\alpha-\\beta}.}\n\\]\nNext, we minimize $\\gamma(\\alpha,\\beta)$ subject to $\\beta\\le\\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}\\le\\alpha$\nand the valid inequality:\n\\begin{align*}\n & \\min_{0\\le\\beta\\le\\alpha}\\left\\{ {\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}}:\\alpha^{2}+\\frac{r}{r^{\\star}}\\beta^{2}\\le1\\right\\} \\\\\n= & \\min_{\\beta\\ge0,\\;\\rho\\ge1}\\left\\{ {\\displaystyle \\frac{1-\\rho\\beta^{2}}{(\\rho-1)\\beta^{2}}}:(\\rho^{2}+\\frac{r}{r^{\\star}})\\beta^{2}\\le1\\right\\} =\\min_{\\rho\\ge1}\\left\\{ \\rho+\\frac{r}{r^{\\star}}\\frac{1}{\\rho-1}\\right\\} =1+2\\sqrt{\\frac{r}{r^{\\star}}}.\n\\end{align*}\nWe obtain the second line from the first by substituting $\\alpha=\\rho\\beta$\nfor $\\rho\\ge1$. The minimizers are $\\beta^{2}=(\\rho^{2}+\\frac{r}{r^{\\star}})^{-1}$\nand $\\rho=1+\\sqrt{r/r^{\\star}}$. \\qed\n\\end{proof}\n\n\\begin{theorem}[Regularized Eckart--Young]\n\\label{thm:EckartYoung}Given $A\\in\\S_{+}^{n}$ and $B\\in\\S_{+}^{r}$\nwith $r\\le n$, let $A=\\sum_{i=1}^{n}s_{i}u_{i}u_{i}^{T}$ and $B=\\sum_{i=1}^{r}d_{i}v_{i}v_{i}^{T}$\ndenote the usual orthonormal eigendecompositions with $s_{1}\\ge\\cdots\\ge s_{n}\\ge0$\nand $0\\le d_{1}\\le\\cdots\\le d_{r}.$ Then, \n\\begin{align}\n\\min_{Y\\in\\R^{n\\times r}}\\left\\{ \\|A-YY^{T}\\|_{F}^{2}+2\\langle B,Y^{T}Y\\rangle\\right\\}  & =\\sum_{i=1}^{n}s_{i}^{2}-\\sum_{i=1}^{r}[(s_{i}-d_{i})_{+}]^{2}\\label{eq:ey-sol}\n\\end{align}\nwith minimizer $Y^{\\star}=\\sum_{i=1}^{r}u_{i}v_{i}^{T}\\sqrt{(s_{i}-d_{i})_{+}}$\nwhere $(\\cdot)_{+}\\eqdef\\max\\{0,\\cdot\\}$. \n\\end{theorem}\n\n\\begin{proof}\n[\\thmref{EckartYoung}]Recall that $A=\\sum_{i=1}^{n}s_{i}u_{i}u_{i}^{T}$\nand $B=\\sum_{i=1}^{r}d_{i}v_{i}v_{i}^{T}$ with $s_{1}\\ge\\cdots\\ge s_{n}\\ge0$\nand $0\\le d_{1}\\le\\cdots\\le d_{r}.$ It follows from (\\ref{eq:Ruhe})\nthat\n\\begin{gather*}\n\\inner A{YY^{T}}\\le\\sum_{i=1}^{n}\\lambda_{i}(YY^{T})\\cdot\\lambda_{i}(A)=\\sum_{i=1}^{r}s_{i}\\sigma_{i}^{2}(Y),\\\\\n\\inner B{Y^{T}Y}\\ge\\sum_{i=1}^{r}\\lambda_{i}(Y^{T}Y)\\cdot\\lambda_{n-i+1}(B)=\\sum_{i=1}^{r}d_{i}\\sigma_{i}^{2}(Y).\n\\end{gather*}\nSubstituting yields the following lower-bound, which is valid for\nall choices of $Y$:\n\\begin{align*}\n\\|A-YY^{T}\\|_{F}^{2}+2\\inner B{Y^{T}Y}= & \\|A\\|_{F}^{2}+\\|YY^{T}\\|_{F}^{2}-2\\inner A{YY^{T}}+2\\inner B{Y^{T}Y}\\\\\n\\ge & \\sum_{i=1}^{n}s_{i}^{2}+\\sum_{i=1}^{r}\\sigma_{i}^{4}(Y)-2\\sum_{i=1}^{r}s_{i}\\sigma_{i}^{2}(Y)+2\\sum_{i=1}^{r}d_{i}\\sigma_{i}^{2}(Y)\\\\\n\\ge & \\sum_{i=1}^{n}s_{i}^{2}-\\sum_{i=1}^{r}[(s_{i}-d_{i})_{+}]^{2}.\n\\end{align*}\nThe final line applied the following result as a lower-bound:\n\\begin{align*}\n\\sigma_{i}^{4}(Y)-2(s_{i}-d_{i})\\sigma_{i}^{2}(Y)\\ge & \\min_{x_{i}\\ge0}x_{i}^{2}-2(s_{i}-d_{i})x_{i}\\\\\n= & \\min_{x_{i}\\ge0}[x_{i}-(s_{i}-d_{i})]^{2}-(s_{i}-d_{i})^{2}\\\\\n= & [(s_{i}-d_{i})_{+}-(s_{i}-d_{i})]^{2}-(s_{i}-d_{i})^{2}=-[(s_{i}-d_{i})_{+}]^{2}.\n\\end{align*}\nFinally, we verify that $Y^{\\star}=\\sum_{i=1}^{n}u_{i}v_{i}\\sqrt{(s_{i}-d_{i})_{+}}$\nattains the lower-bound derived above. \\qed\n\\end{proof}\n\n\\begin{lemma}\n\\label{lem:numcard}If $x\\ge0$ and $\\one^{T}x\\le\\|x\\|^{2}$, then\n$\\one^{T}\\left(I-xx^{T}/\\|x\\|^{2}\\right)\\one\\ge\\|(\\one-x)_{+}\\|^{2}$.\n\\end{lemma}\n\n\\begin{proof}\nDefine $u=\\alpha x,$ where $\\alpha=\\one^{T}x/\\|x\\|^{2}$ is chosen\nso that $\\mathbf{1}^{T}u=\\|u\\|^{2}$. First, verify that $\\mathbf{1}^{T}(I-xx^{T}/\\|x\\|^{2})\\mathbf{1}=\\|\\mathbf{1}-u\\|^{2}\\ge\\|(\\mathbf{1}-u)_{+}\\|^{2}$.\nNext, observe that $\\psi(\\alpha)\\equiv[(1-\\alpha t)_{+}]^{2}$ is\na decreasing function of $\\alpha\\ge0$ when $t\\ge0$, and therefore\n$\\|(\\one-\\alpha x)_{+}\\|^{2}\\ge\\|(\\one-x)_{+}\\|^{2}$ for $x\\ge0$\nand $0\\le\\alpha\\le1$. Finally, the hypotheses $x\\ge0$ and $\\one^{T}x\\le\\|x\\|^{2}$\nensure that $0\\le\\alpha\\le1$. \\qed\n\\end{proof}\n\n\\begin{lemma}\n\\label{lem:keyclaim}Given $s,d\\in\\R_{+}^{n}$, let $s_{i}\\ge s_{\\lb}\\ge0$\nfor all $i$. Then we have\n\\begin{align*}\n\\|s\\|^{2}-\\|(s-d)_{+}\\|^{2}\\ge & \\begin{cases}\ns_{\\lb}^{2}\\frac{(\\one^{T}d)^{2}}{\\|d\\|^{2}} & \\text{if }s_{\\lb}\\one^{T}d\\le\\|d\\|^{2},\\\\\n2s_{\\lb}\\one^{T}d-\\|d\\|^{2} & \\text{if }s_{\\lb}\\one^{T}d\\ge\\|d\\|^{2}.\n\\end{cases}\n\\end{align*}\n\\end{lemma}\n\n\\begin{proof}\nFirst, we observe that $\\psi(t)\\equiv t^{2}-[(t-\\alpha)_{+}]^{2}$\nis an increasing function of $t\\ge0$. It follows from element-wise\nmonotonicity that\n\\[\n\\|s\\|^{2}-\\|(s-d)_{+}\\|^{2}\\ge\\|s_{\\lb}\\one\\|^{2}-\\|(s_{\\lb}\\one-d)_{+}\\|^{2}=s_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-d/s_{\\lb})_{+}\\|^{2}\\right)\n\\]\nIf $s_{\\lb}\\one^{T}d\\le\\|d\\|^{2}$, then write $x=d/s_{\\lb}\\ge0$,\nand observe that $\\one^{T}x\\le\\|x\\|^{2}$ holds. Therefore, applying\n\\lemref{numcard} yields\n\\begin{align*}\ns_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-d/s_{\\lb})_{+}\\|^{2}\\right) & =s_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-x)_{+}\\|^{2}\\right)\\ge s_{\\lb}^{2}\\one^{T}\\left(xx^{T}/\\|x\\|^{2}\\right)\\one\\\\\n & =s_{\\lb}^{2}\\left([\\one^{T}(d/s_{\\lb})]^{2}/\\|d/s_{\\lb}\\|^{2}\\right)=s_{\\lb}^{2}(\\one^{T}d)^{2}/\\|d\\|^{2}.\n\\end{align*}\nIf otherwise $s_{\\lb}\\one^{T}d\\ge\\|d\\|^{2}$, then we have \n\\begin{align*}\ns_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-d/s_{\\lb})_{+}\\|^{2}\\right)\\ge s_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|\\one-d/s_{\\lb}\\|^{2}\\right)=2s_{\\lb}\\one^{T}d-\\|d\\|^{2}.\n\\end{align*}\n\\qed\n\\end{proof}\n\n\\begin{lemma}\n\\label{lem:ab2}For $X,Z\\in\\R^{n\\times r}$, define the parameters\n\\[\n\\alpha=\\frac{\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}{\\|XX^{T}-ZZ^{T}\\|_{F}},\\qquad\\beta=\\frac{\\lambda_{\\min}(X^{T}X)}{\\|XX^{T}-ZZ^{T}\\|_{F}}\\cdot\\frac{\\tr(Z_{\\perp}Z_{\\perp}^{T})}{\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}.\n\\]\nThen, $\\alpha^{2}+(r/p)\\beta^{2}\\le1+[(\\beta-\\alpha)_{+}]^{2}$ holds\nfor all $p$ satisfying $\\rank(Z)\\le p\\le r$.\n\\end{lemma}\n\n\\begin{proof}\nThe two parameters $\\alpha$ and $\\beta$ are unitarily invariant\nwith respect to $X,Z$, so we may assume without loss of generality\nthat $X,Z$ are partitioned as\n\\[\nX=\\begin{bmatrix}X_{1}\\\\\n0\n\\end{bmatrix}\\qquad Z=\\begin{bmatrix}Z_{1}\\\\\nZ_{2}\n\\end{bmatrix}\\qquad\\text{where }X_{1}\\in\\R^{r\\times r},\\;Z_{1}\\in\\R^{r\\times p},\\;Z_{2}\\in\\R^{(n-r)\\times p}.\n\\]\n(Otherwise, take the QR decomposition $X=QR$ and note that $\\alpha,\\beta$\nremain unchanged with $X\\gets Q^{T}X$ and $Z\\gets Q^{T}Z$.) Denote\n$s\\in\\R^{r}$ as the $r$ eigenvalues of $X_{1}X_{1}^{T}$ and $d\\in\\R^{p}$\nas the $p$ eigenvalues of $Z_{2}^{T}Z_{2}$ \n\\begin{align*}\ns_{i}\\eqdef & \\lambda_{i}(X_{1}X_{1}^{T})=\\lambda_{i}(XX^{T}),\\quad s_{1}\\ge s_{2}\\ge\\cdots\\ge s_{r}\\ge0,\\\\\nd_{i}\\eqdef & \\lambda_{i}(Z_{2}^{T}Z_{2})=\\lambda_{i}(Z_{\\perp}Z_{\\perp}^{T}),\\quad0\\le d_{1}\\le d_{2}\\le\\cdots\\le d_{p}.\n\\end{align*}\nWriting $\\e=\\vect(XX^{T}-ZZ^{T})$ yields\n\\begin{align*}\n\\alpha & =\\frac{\\|Z_{2}Z_{2}^{T}\\|_{F}}{\\|XX^{T}-ZZ^{T}\\|_{F}}=\\frac{\\|d\\|_{F}}{\\|\\e\\|_{F}}, & \\beta & =\\frac{\\lambda_{\\min}(X_{1}^{T}X_{1})}{\\|XX^{T}-ZZ^{T}\\|_{F}}\\cdot\\frac{\\tr(Z_{2}Z_{2}^{T})}{\\|Z_{2}Z_{2}^{T}\\|_{F}}=\\frac{s_{r}\\cdot\\one^{T}d}{\\|\\e\\|\\|d\\|}.\n\\end{align*}\nWe expand $\\|XX^{T}-ZZ^{T}\\|_{F}^{2}$ block-wise and evoke our Regularized\nEckart--Young Theorem (\\thmref{EckartYoung}) over the $r\\times p$\nmatrix block $Z_{1}$:\n\\begin{align*}\n\\|XX^{T}-ZZ^{T}\\|_{F}^{2} & =\\|X_{1}X_{1}^{T}-Z_{1}Z_{1}^{T}\\|_{F}^{2}+2\\langle Z_{2}^{T}Z_{2},Z_{1}^{T}Z_{1}\\rangle+\\|Z_{2}Z_{2}^{T}\\|^{2}\\\\\n & \\ge\\min_{Z_{1}\\in\\R^{r\\times p}}\\{\\|X_{1}X_{1}^{T}-Z_{1}Z_{1}^{T}\\|_{F}^{2}+2\\langle Z_{2}^{T}Z_{2},Z_{1}^{T}Z_{1}\\rangle\\}+\\|Z_{2}Z_{2}^{T}\\|^{2}\\\\\n & =\\|s\\|^{2}-\\|(s'-d)_{+}\\|^{2}+\\|d\\|^{2}\n\\end{align*}\nwhere $s'=(s_{1},s_{2},\\dots,s_{p})$. Now, evoking our technical\nlemma (\\lemref{keyclaim}) with $s_{\\lb}\\equiv s_{r}$ yields the\nfollowing two lower-bounds on $\\|XX^{T}-ZZ^{T}\\|_{F}^{2}=\\|\\e\\|^{2}$:\\begin{subequations}\\label{eq:key-claim}\n\\begin{gather}\ns_{r}\\one^{T}d\\le\\|d\\|^{2}\\quad\\implies\\quad\\|\\e\\|^{2}\\ge s_{r}^{2}\\frac{(\\one^{T}d)^{2}}{\\|d\\|^{2}}+(r-p)s_{r}^{2}+\\|d\\|^{2},\\label{eq:key-claim1}\\\\\ns_{r}\\one^{T}d\\ge\\|d\\|^{2}\\quad\\implies\\quad\\|\\e\\|^{2}\\ge2s_{r}\\one^{T}d-\\|d\\|^{2}+(r-p)s_{r}^{2}+\\|d\\|^{2}.\\label{eq:key-claim2}\n\\end{gather}\n\\end{subequations}Substituting $s_{r}=\\beta\\cdot\\|\\e\\|\\|d\\|/(\\one^{T}d)$\nyields\n\\[\n(r-p)s_{r}^{2}=(r-p)\\frac{\\|d\\|^{2}}{(\\one^{T}d)^{2}}\\beta^{2}\\|\\e\\|^{2}\\ge(r-p)\\frac{1}{\\nnz(d)}\\beta^{2}\\|\\e\\|^{2}\\ge\\frac{(r-p)}{p}\\beta^{2}\\|\\e\\|^{2},\n\\]\nand substituting into (\\ref{eq:key-claim}) yields\\begin{subequations}\n\\begin{gather}\n\\beta\\le\\alpha\\quad\\implies\\quad1\\ge\\beta^{2}+\\alpha^{2}+\\frac{(r-p)}{p}\\beta^{2}=\\alpha^{2}+\\frac{r}{p}\\beta^{2},\\label{eq:key-claim3}\\\\\n\\beta\\ge\\alpha\\quad\\implies\\quad1\\ge2\\alpha\\beta+\\frac{(r-p)}{p}\\beta^{2}=\\alpha^{2}+\\frac{r}{p}\\beta^{2}-(\\beta-\\alpha)^{2},\\label{eq:key-claim4}\n\\end{gather}\n\\end{subequations}as desired. \\qed\n\\end{proof}\n\n\\begin{lemma}\n\\label{lem:I-JJ+}For any $X\\in\\R^{n\\times r}$, the matrix $\\J_{X}$\ndefined in (\\ref{eq:Jxdef}) satisfies $I-\\J_{X}\\J_{X}^{\\dagger}=(I-XX^{\\dagger})\\otimes(I-XX^{\\dagger})$\nwhere $\\dagger$ denotes the pseudoinverse. \n\\end{lemma}\n\n\\begin{proof}\nLet $X=QR$ be the usual QR decomposition with orthogonal complement\n$Q_{\\perp}$, so that $XX^{\\dagger}=QQ^{T}$ and $I-XX^{\\dagger}=Q_{\\perp}Q_{\\perp}^{T}$.\nObserve that for any $M\\in\\R^{n\\times n}$:\n\\[\n\\J_{X}^{T}\\vect(M)=0\\iff(M+M^{T})Q=0\\iff M=Q_{\\perp}\\hat{M}Q_{\\perp}^{T}.\n\\]\nVectorizing the final equality as $\\vect(M)=(Q_{\\perp}\\otimes Q_{\\perp})\\vect(\\hat{M})$\nyields \n\\begin{align*}\n(I-\\J_{X}\\J_{X}^{\\dagger})b & =\\arg\\min_{v}\\{\\|v-b\\|:\\J_{X}^{T}v=0\\}\\\\\n & =\\arg\\min_{v}\\{\\|v-b\\|:v=(Q_{\\perp}\\otimes Q_{\\perp})\\hat{v}\\}\\\\\n & =(Q_{\\perp}\\otimes Q_{\\perp})(Q_{\\perp}\\otimes Q_{\\perp})^{T}b=(Q_{\\perp}Q_{\\perp}^{T}\\otimes Q_{\\perp}Q_{\\perp}^{T})b.\\quad\\qed\n\\end{align*}\n\\end{proof}\n\n\\begin{lemma}\n\\label{lem:tradeoff}We have\n\\[\n\\kappa_{\\lb}(X,Z)=\\max_{t\\ge0}\\frac{1+\\cos\\theta(t)}{2t+1-\\cos\\theta(t)}\n\\]\nwhere $\\cos\\theta(t)$ is defined as follows\n\\begin{equation}\n\\cos\\theta(t)=\\max_{y,W_{i,j}}\\left\\{ \\e^{T}\\f:\\begin{array}{c}\n\\f\\equiv\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\quad\\|\\e\\|\\|\\f\\|=1,\\\\\n(I-ZZ^{\\dagger})\\,\\mat(\\f)\\,(I-ZZ^{\\dagger})\\succeq0,\\\\\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle=2t,\\quad W=[W_{i,j}]_{i,j=1}^{r}\\succeq0,\n\\end{array}\\right\\} \\label{eq:cosprob}\n\\end{equation}\nwith respect to $y\\in\\R^{nr}$ and $W_{i,j}\\in\\R^{n\\times n}$ for\n$i,j\\in\\{1,2,\\dots,r\\}$.\n\\end{lemma}\n\n\\begin{lemma}\n\\label{lem:costheta}We have $\\cos\\theta(t)\\ge\\alpha\\cdot(t/\\beta)+\\sqrt{1-\\alpha^{2}}\\sqrt{1-t^{2}/\\beta^{2}}$\nfor $0\\le t\\le\\alpha\\beta$ where $Z_{\\perp}=(I-XX^{\\dagger})Z$ and\n\\begin{align*}\n\\alpha & =\\frac{\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}{\\|XX^{T}-ZZ^{T}\\|_{F}}, & \\beta & =\\frac{\\lambda_{\\min}(X^{T}X)\\cdot\\tr(Z_{\\perp}Z_{\\perp}^{T})}{\\|XX^{T}-ZZ^{T}\\|_{F}\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}.\n\\end{align*}\n\\end{lemma}\n\n\\begin{proof}\nWrite $\\Pi=I-ZZ^{\\dagger}$ and $\\e=\\vect(XX^{T}-ZZ^{T})$, and decompose\n$\\e_{1}=(\\J_{X}\\J_{X}^{\\dagger})\\e$ and $\\e_{2}=(I-\\J_{X}\\J_{X}^{\\dagger})\\e$.\nWe verify that the following heuristic choice\n\\begin{gather*}\ny=\\underbrace{\\frac{\\sqrt{1-(t/\\beta)^{2}}}{\\|\\e\\|\\|\\e_{1}\\|}}_{\\gamma_{1}}\\;\\J_{X}^{\\dagger}\\e,\\quad W=\\underbrace{\\frac{t/\\beta}{\\|\\e\\|\\|\\e_{2}\\|}}_{\\gamma_{2}}\\;\\sum_{i=1}^{r}\\vect(Z_{\\perp}e_{i}v_{r}^{T})\\vect(Z_{\\perp}e_{i}v_{r}^{T})^{T},\n\\end{gather*}\nis feasible for (\\ref{eq:cosprob}). Indeed, $W\\succeq0$ holds, and\nwe have from (\\ref{eq:e2}), (\\ref{eq:Wsol1}), and (\\ref{eq:Wsol2}):\n\\begin{gather*}\n\\f=\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i})=\\gamma_{1}\\cdot\\J_{X}\\J_{X}^{\\dagger}\\e-\\gamma_{2}\\vect(Z_{\\perp}Z_{\\perp}^{T})=\\gamma_{1}\\cdot\\e_{1}+\\gamma_{2}\\e_{2},\\\\\n\\|\\e\\|^{2}\\|\\f\\|^{2}=\\gamma_{1}^{2}\\cdot\\|\\e\\|^{2}\\|\\e_{1}\\|^{2}+\\gamma_{2}^{2}\\cdot\\|\\e\\|^{2}\\|\\e_{2}\\|^{2}=(1-(t/\\beta)^{2})+(t/\\beta)^{2}=1,\\\\\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle=2\\left(\\frac{t/\\beta}{\\|\\e\\|\\|\\e_{2}\\|}\\right)\\cdot\\lambda_{\\min}(X^{T}X)\\|Z_{\\perp}\\|_{F}^{2}=\\frac{2t}{\\beta}\\cdot\\frac{\\sigma_{r}^{2}\\cdot\\tr(Z_{\\perp}Z_{\\perp}^{T})}{\\|\\e\\|\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}=2t.\n\\end{gather*}\nTo verify that $\\Pi\\,\\mat(\\f)\\,\\Pi\\succeq0$, note that $t\\le\\alpha\\beta$\nimplies $t/\\beta\\le\\alpha\\le1$, so it follows from $\\alpha=\\|\\e_{2}\\|/\\|\\e\\|$\nand $\\sqrt{1-\\alpha^{2}}=\\|\\e_{1}\\|/\\|\\e\\|$ that\n\\[\n\\gamma_{1}=\\frac{\\sqrt{1-(t/\\beta)^{2}}}{\\|\\e_{1}\\|\\|\\e\\|}\\ge\\frac{\\sqrt{1-\\alpha^{2}}}{\\|\\e_{1}\\|\\|\\e\\|}=\\frac{1}{\\|\\e\\|^{2}}=\\frac{\\alpha}{\\|\\e_{2}\\|\\|\\e\\|}\\ge\\frac{t/\\beta}{\\|\\e_{2}\\|\\|\\e\\|}=\\gamma_{2}.\n\\]\nIn turn, $\\gamma_{1}\\ge\\gamma_{2}$ implies\n\\begin{align*}\n\\f & =\\gamma_{1}\\cdot\\e_{1}+\\gamma_{2}\\cdot\\e_{2}=\\gamma_{1}\\cdot\\e-(\\gamma_{1}-\\gamma_{2})\\e_{2},\\\\\n\\mat(\\f) & =\\gamma_{1}\\cdot(XX^{T}-ZZ^{T})+(\\gamma_{1}-\\gamma_{2})\\cdot Z_{\\perp}Z_{\\perp}^{T},\\\\\n\\Pi\\,\\mat(\\f)\\,\\Pi & =\\gamma_{1}\\cdot\\Pi(XX^{T})\\Pi+(\\gamma_{1}-\\gamma_{2})\\cdot\\Pi(Z_{\\perp}Z_{\\perp}^{T})\\Pi\\succeq0.\n\\end{align*}\nHaving verified $y$ and $W$ as feasible for (\\ref{eq:cosprob}),\nwe use its objective value to heuristically lower-bound the maximum\n\\[\n\\cos\\theta(t)\\ge\\e^{T}\\f=\\gamma_{1}\\cdot\\|\\e_{1}\\|^{2}+\\gamma_{2}\\cdot\\|\\e_{2}\\|^{2}=\\sqrt{1-(t/\\beta)^{2}}\\sqrt{1-\\alpha^{2}}+(t/\\beta)\\cdot\\alpha.\n\\]\n \\qed\n\\end{proof}\n\n\\begin{lemma}\n\\label{lem:counter}Let $[Q_{1},Q_{2}]$ have orthonormal columns\nwith $Q_{1}\\in\\R^{n\\times r}$ and $Q_{2}\\in\\R^{n\\times r^{\\star}}$.\nIf $\\kappa=1+2\\sqrt{r/r^{\\star}}$, and $\\H$ is any $n^{2}\\times n^{2}$\nmatrix that satisfies $I\\preceq\\H\\preceq\\kappa I$ and the following\n$rr^{\\star}+2$ eigenvalue equations\n\\begin{gather*}\n\\H\\vect\\left(Q_{1}Q_{1}^{T}+\\sqrt{r/r^{\\star}}Q_{2}Q_{2}\\right)=\\kappa\\cdot\\vect\\left(Q_{1}Q_{1}^{T}+\\sqrt{r/r^{\\star}}Q_{2}Q_{2}\\right),\\\\\n\\H\\vect\\left(Q_{1}Q_{1}^{T}-\\sqrt{r/r^{\\star}}Q_{2}Q_{2}\\right)=1\\cdot\\vect\\left(Q_{1}Q_{1}^{T}-\\sqrt{r/r^{\\star}}Q_{2}Q_{2}\\right),\\\\\n\\H\\vect\\left(Q_{1}VQ_{2}^{T}+Q_{2}V^{T}Q_{1}^{T}\\right)=\\kappa\\cdot\\vect\\left(Q_{1}VQ_{2}^{T}+Q_{2}V^{T}Q_{1}^{T}\\right)\\qquad\\text{for all }V\\in\\R^{r\\times r^{\\star}},\n\\end{gather*}\nthen $(\\kappa,\\H)$ is a feasible point for (\\ref{eq:kappaub_prob})\nwith $X=Q_{1}$ and $Z=\\sqrt{1+\\sqrt{r/r^{\\star}}}Q_{2}$. Moveover,\n$\\e^{T}\\H\\e=(1+2\\sqrt{r/r^{\\star}})\\cdot(1+\\sqrt{r/r^{\\star}})\\cdot r^{\\star}$.\n\\end{lemma}\n\n\\begin{proof}\nWrite $\\rho\\equiv\\sqrt{r/r^{\\star}}$, and take $[Q_{1},Q_{2}]=I_{r+r^{\\star}}$\nwithout loss of generality. Decomposing $\\e=\\vect(XX^{T}-ZZ^{T})$\ninto eigenvectors and applying $\\H$ yields\n\\begin{gather*}\n\\mat(\\e)=\\begin{bmatrix}I_{r} & 0\\\\\n0 & -(1+\\rho)I_{r^{\\star}}\n\\end{bmatrix}=-\\frac{1}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & \\rho I_{r^{\\star}}\n\\end{bmatrix}+\\frac{1+2\\rho}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & -\\rho I_{r^{\\star}}\n\\end{bmatrix},\\\\\n\\mat(\\H\\e)=-\\frac{1+2\\rho}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & \\rho I_{r^{\\star}}\n\\end{bmatrix}+\\frac{1+2\\rho}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & -\\rho I_{r^{\\star}}\n\\end{bmatrix}=\\begin{bmatrix}0_{r} & 0\\\\\n0 & -\\kappa I_{r^{\\star}}\n\\end{bmatrix}.\n\\end{gather*}\nClearly, $\\inner{\\mat(\\e)}{\\mat(\\H\\e)}=\\kappa(1+\\rho)r^{\\star}.$\nFor $v=\\vect(V)=\\vect([V_{1};V_{2}])$ with $V_{1}\\in\\R^{r\\times r}$\nand $V_{2}\\in\\R^{r^{\\star}\\times r}$, observe that \n\\begin{gather*}\nv^{T}\\J_{X}\\H\\e=\\left\\langle \\begin{bmatrix}V_{1}+V_{1}^{T} & V_{2}^{T}\\\\\nV_{2} & 0_{r^{\\star}}\n\\end{bmatrix},\\begin{bmatrix}0_{r} & 0\\\\\n0 & -\\kappa I_{r^{\\star}}\n\\end{bmatrix}\\right\\rangle =0,\\\\\nv^{T}[I_{r}\\otimes\\mat(\\H\\e)]v=\\langle\\mat(\\H\\e),VV^{T}\\rangle=\\left\\langle \\begin{bmatrix}0_{r} & 0\\\\\n0 & -\\kappa I_{r^{\\star}}\n\\end{bmatrix},\\begin{bmatrix}V_{1}V_{1}^{T} & V_{1}V_{2}^{T}\\\\\nV_{2}V_{1}^{T} & V_{2}V_{2}^{T}\n\\end{bmatrix}\\right\\rangle =-\\kappa\\|V_{2}\\|^{2},\\\\\nv^{T}\\J_{X}^{T}\\H\\J_{X}v=\\left\\Vert \\H^{1/2}\\vect\\left(\\begin{bmatrix}V_{1}+V_{1}^{T} & V_{2}^{T}\\\\\nV_{2} & 0_{r^{\\star}}\n\\end{bmatrix}\\right)\\right\\Vert ^{2}\\ge\\left\\Vert \\H^{1/2}\\vect\\left(\\begin{bmatrix}0_{r} & V_{2}^{T}\\\\\nV_{2} & 0_{r^{\\star}}\n\\end{bmatrix}\\right)\\right\\Vert ^{2}=2\\kappa\\|V_{2}\\|^{2}.\n\\end{gather*}\nTherefore, $v^{T}\\J_{X}\\H\\e=0$ and $2v^{T}[I_{r}\\otimes\\mat(\\H\\e)]v\\ge-v^{T}\\J_{X}^{T}\\H\\J_{X}v$\nhold for all $v$. \\qed\n\\end{proof}\n\n\\section{\\label{app:tradeoff}Derivation of the dual for the lower-bound problem\n(Proof of \\lemref{tradeoff})}\n\nWe will derive the Lagrangian dual for the following\n\\begin{align*}\n\\kappa_{\\lb}(X,Z) & =\\min_{\\kappa,\\H,s}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H\\preceq\\kappa I,\\qquad\\J_{Z}^{T}s=0,\\qquad s\\in\\vect(\\S_{+}^{n}),\\\\\n\\J_{X}^{T}(\\H\\e+s)=0,\\quad2I_{r}\\otimes\\mat(\\H\\e+s)+\\kappa\\J_{X}^{T}\\J_{X}\\succeq0.\n\\end{array}\\right\\} \n\\end{align*}\nRecall that $\\e=\\vect(XX^{T}-ZZ^{T})$ and $\\J_{X}\\vect(V)=\\vect(XV^{T}+VX^{T})$\nwith respect to fixed $X,Z\\in\\R^{n\\times r}$. Observe that\n\\[\n\\J_{Z}^{T}\\vect(S)=0\\iff(S+S^{T})Z=0\\iff S=Q_{\\perp}S_{\\perp}Q_{\\perp}^{T}\\iff\\vect(S)=\\Q_{\\perp}\\vect(S_{\\perp})\n\\]\nwhere the orthogonal complement $Q_{\\perp}$ of $Z$ is such that\n$Q_{\\perp}Q_{\\perp}^{T}=I_{n}-ZZ^{\\dagger}$, and $\\Q_{\\perp}=Q_{\\perp}\\otimes Q_{\\perp}$.\nSubstituting and taking the Lagrangian dual yields\n\\begin{align}\n\\kappa_{\\lb}(X,Z) & =\\min_{\\kappa,\\H,s_{\\perp}}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H\\preceq\\kappa I,\\quad s_{\\perp}\\in\\vect(\\S_{+}^{n-r^{\\star}}),\\quad\\J_{X}^{T}(\\H\\e+\\Q_{\\perp}s_{\\perp})=0,\\\\\n2I_{r}\\otimes\\mat(\\H\\e+\\Q_{\\perp}s_{\\perp})+\\kappa\\J_{X}^{T}\\J_{X}\\succeq0\n\\end{array}\\right\\} \\nonumber \\\\\n & =\\max_{y,W_{i,j},U,V}\\left\\{ \\tr(U):\\begin{array}{c}\n\\f\\e^{T}+\\e\\f^{T}=U-V,\\quad\\tr(V)+\\inner{\\J_{X}^{T}\\J_{X}}W=1,\\\\\n\\f\\equiv\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\quad\\Q_{\\perp}^{T}\\f\\in\\vect(\\S_{+}^{n-r^{\\star}}),\\\\\nU,V\\succeq0,\\quad W=[W_{i,j}]_{i,j=1}^{r}\\succeq0\n\\end{array}\\right\\} \\label{eq:ub2-1}\n\\end{align}\nin which $y\\in\\R^{nr}$ and $W_{i,j}\\in\\R^{n\\times n}$ for $i,j\\in\\{1,2,\\dots,r\\}$.\nStrong duality holds here because $y=0,$ $W=\\epsilon\\cdot I_{nr},$\n$U=\\frac{1}{n}I_{n},$ and $V=\\frac{1}{n}I_{n}+\\epsilon\\cdot r\\cdot[\\e\\vect(I_{n})^{T}+\\e\\vect(I_{n})^{T}]$\nis strictly feasible for a sufficiently small $\\epsilon>0$. \n\nThe dual problem (\\ref{eq:ub2-1}) has a closed-form solution over\n$U$ and $V$ \\citep[Lemma~13]{zhang2019sharp}\n\\begin{align*}\n\\frac{\\tr([M]_{-})+\\alpha}{\\tr([M]_{+})}= & \\min_{t,U,V}\\left\\{ \\tr(V)+\\alpha\\cdot t:\\begin{array}{c}\ntM=U-V,\\quad t\\ge0,\\\\\n\\tr(U)=1,\\quad U,V\\succeq0\n\\end{array}\\right\\} ,\n\\end{align*}\nand $\\f\\e^{T}+\\e\\f^{T}$ has exactly two nonzero eigenvalues $\\e^{T}\\f\\pm\\|\\e\\|\\|\\f\\|$\n\\citep[Lemma~14]{zhang2019sharp}. Substituting this solution yields\nexactly \n\\begin{align*}\n\\kappa_{\\lb}(X,Z) & =\\max_{y,W_{i,j}}\\left\\{ \\frac{\\|\\e\\|\\|\\f\\|+\\e^{T}\\f}{\\|\\e\\|\\|\\f\\|-\\e^{T}\\f+\\inner{\\J_{X}^{T}\\J_{X}}W}:\\begin{array}{c}\n\\Q_{\\perp}\\Q_{\\perp}^{T}\\f\\in\\vect(\\S_{+}^{n}),\\\\\n\\f=\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\\\\nW=[W_{i,j}]_{i,j=1}^{r}\\succeq0\n\\end{array}\\right\\} \\\\\n & =\\max_{t\\ge0}\\frac{1+\\cos\\theta(t)}{2t+1-\\cos\\theta(t)}\n\\end{align*}\nwhere $\\cos\\theta(t)$ is itself is defined as the following maximization\n\\[\n\\cos\\theta(t)=\\max_{y,W_{i,j}}\\left\\{ \\e^{T}\\f:\\begin{array}{c}\n\\f\\equiv\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\quad\\|\\e\\|\\|\\f\\|=1,\\\\\n(I-ZZ^{\\dagger})\\,\\mat(\\f)\\,(I-ZZ^{\\dagger})\\succeq0,\\\\\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle=2t,\\quad W=[W_{i,j}]_{i,j=1}^{r}\\succeq0.\n\\end{array}\\right\\} \n\\]\n\n","units":[{"raw_tex":"\\begin{lemma}[SDP formulation]\n\\label{lem:sdp_formu}Define $\\kappa(X,Z)$ as in (\\ref{eq:mu_reform}).\nWe have $\\kappa_{\\ub}(X,Z)\\ge\\kappa(X,Z)\\ge\\kappa_{\\lb}(X,Z)$ where\n\\begin{gather*}\n\\kappa_{\\ub}(X,Z)\\eqdef\\min_{\\kappa,\\H}\\left\\{ \\kappa:I\\preceq\\H\\preceq\\kappa I,\\;\\J_{X}^{T}\\H\\e=0,\\;-2I_{r}\\otimes\\mat(\\H\\e)\\preceq\\J_{X}^{T}\\H\\J_{X}\\right\\} \\\\\n\\kappa_{\\lb}(X,Z)\\eqdef\\min_{\\kappa,s,\\H}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H\\preceq\\kappa I,\\quad\\mat(s)\\succeq0,\\quad\\J_{Z}^{T}s=0,\\\\\n\\J_{X}^{T}(\\H\\e+s)=0,\\quad-2I_{r}\\otimes\\mat(\\H\\e+s)\\preceq\\kappa\\J_{X}^{T}\\J_{X}\n\\end{array}\\right\\} \n\\end{gather*}\nand $\\e=\\vect(XX^{T}-ZZ^{T})$ and $\\J_{U}\\vect(V)=\\vect(UV^{T}+VU^{T})$\nfor all $V$.\n\\end{lemma}","start":37973,"end":38675},{"raw_tex":"\\begin{proof}\nDenote $S_{M}$ as the convex gradient $\\nabla\\phi(M)$ evaluated\nat $M$, and denote $\\H_{M}$ as the matrix representation of the\nconvex Hessian operator $\\nabla^{2}\\phi(M)$ evaluated at $M$. It\nfollows from $\\inner{\\nabla^{2}\\phi(M)[E]}E=\\vect(E)^{T}\\H_{M}\\vect(E)$\nthat: \n\\begin{equation}\n\\phi\\text{ is }1\\text{-strongly convex and }\\kappa\\text{-smooth}\\iff I\\preceq\\H_{M}\\preceq\\kappa I\\quad\\text{for all }M\\in\\S^{n}.\\label{eq:cond}\n\\end{equation}\nAdditionally, the Karush--Kuhn--Tucker conditions at $M^{\\star}=ZZ^{T}$\nread:\n\\begin{equation}\nZZ^{T}=\\arg\\min_{M\\succeq0}\\phi(M)\\quad\\iff\\quad S_{ZZ^{T}}\\succeq0,\\quad S_{ZZ^{T}}Z=0.\\label{eq:kktZZ}\n\\end{equation}\nNote that strong duality holds in the above because $ZZ^{T}+\\epsilon I$\nis a strictly feasible point with a bounded objective due to $\\kappa$-smoothness.\nNext, we observe that the two gradient evaluations $S_{XX^{T}}$ and\n$S_{ZZ^{T}}$ are related via the fundamental theorem of calculus:\n\\begin{gather*}\n\\nabla\\phi(XX^{T})=\\nabla\\phi(ZZ^{T})+\\int_{0}^{1}\\nabla^{2}\\phi((1-t)ZZ^{T}+tXX^{T})[XX^{T}-ZZ^{T}]\\,\\d t\\\\\n\\iff\\quad\\vect(S_{XX^{T}})=\\vect(S_{ZZ^{T}})+\\left[\\int_{0}^{1}\\H_{M(t)}\\,\\d t\\right]\\vect(XX^{T}-ZZ^{T}).\n\\end{gather*}\nDenoting $s=\\vect(S_{ZZ^{T}})$ and $\\H_{\\av}=\\int_{0}^{1}\\H_{M(t)}\\,\\d t$\nand $\\e=\\vect(XX^{T}-ZZ^{T})$ allows us to rewrite the directional\nderivatives of $f(U)\\eqdef\\phi(UU^{T})$ as follows: \n\\begin{align}\n\\inner{\\nabla f(X)}V & =\\vect(V)^{T}\\J_{X}^{T}(\\H_{\\av}\\e+s),\\nonumber \\\\\n\\inner{\\nabla^{2}f(X)[V]}V & =\\vect(V)^{T}[2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)+\\J_{X}^{T}\\H_{XX^{T}}\\J_{X}]\\,\\vect(V).\\label{eq:vecdf}\n\\end{align}\nSubstituting (\\ref{eq:cond}), (\\ref{eq:vecdf}), and (\\ref{eq:kktZZ})\ninto (\\ref{eq:mu_reform}) yields exactly\n\\begin{equation}\n\\kappa(X,Z)=\\inf_{\\kappa,\\phi:\\S^{n}\\to\\R}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H_{M}\\preceq\\kappa I\\quad\\text{for all }M\\in\\S^{n},\\\\\n\\mat(s)\\succeq0,\\quad\\J_{Z}^{T}s=0,\\quad\\J_{X}^{T}(\\H_{\\av}\\e+s)=0,\\\\\n-2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)\\preceq\\J_{X}^{T}\\H_{XX^{T}}\\J_{X}.\n\\end{array}\\right\\} \\label{eq:kappa_sdp1}\n\\end{equation}\nWe obtain the upper-bound $\\kappa_{\\ub}(X,Z)$ from $\\kappa(X,Z)$\nin  (\\ref{eq:kappa_sdp1}) by fixing $\\H_{M}\\equiv\\H$ for all $M$,\nand $S_{ZZ^{T}}\\equiv\\mat(s)=0$. We obtain the lower-bound $\\kappa_{\\lb}(X,Z)$\nfrom $\\kappa(X,Z)$ in (\\ref{eq:kappa_sdp1}) by substituting the\nrelaxation \n\\[\n-2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)\\quad\\preceq\\quad\\J_{X}^{T}\\H_{XX^{T}}\\J_{X}\\quad\\preceq\\quad\\kappa\\cdot\\J_{X}^{T}\\J_{X},\n\\]\nin order to avoid explicitly optimizing over $\\H_{XX^{T}}$:\n\\begin{align}\n\\kappa(X,Z) & \\ge\\inf_{\\kappa,\\phi:\\S^{n}\\to\\R}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H_{M}\\preceq\\kappa I\\quad\\text{for all }M\\in\\S^{n},\\\\\n\\mat(s)\\succeq0,\\quad\\J_{Z}^{T}s=0,\\quad\\J_{X}^{T}(\\H_{\\av}\\e+s)=0,\\\\\n-2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)\\preceq\\kappa\\cdot\\J_{X}^{T}\\J_{X}.\n\\end{array}\\right\\} \\label{eq:kappasdp2}\n\\end{align}\nFinally, we impose $I\\preceq\\H_{M}\\preceq\\kappa I$ over the average\nHessian $\\H_{\\av}=\\int_{0}^{1}\\H_{M(t)}\\,\\d t$ and then relax this\nconstraint over $\\H_{M}$ for all other values of $M$.\\qed\n\\end{proof}","start":38676,"end":41797},{"raw_tex":"\\begin{lemma}[Heuristic upper-bound]\n\\label{lem:heuristic}Let $[Q_{1},Q_{2}]$ have orthonormal columns\nwith $Q_{1}\\in\\R^{n\\times r}$ and $Q_{2}\\in\\R^{n\\times r^{\\star}}$.\nThen, we have\n\\[\n\\kappa_{\\ub}(X,Z)\\le1+2\\sqrt{r/r^{\\star}}\\quad\\text{where }X=Q_{1},\\quad Z=\\sqrt{1+\\sqrt{r/r^{\\star}}}Q_{2}.\n\\]\n\\end{lemma}","start":43032,"end":43343},{"raw_tex":"\\begin{lemma}[Closed-form lower-bound]\n\\label{lem:abdef}For $X,Z\\in\\R^{n\\times r}$ such that $XX^{T}\\ne ZZ^{T}$,\nlet $Z_{\\perp}=(I-XX^{\\dagger})Z$. We have\n\\begin{gather*}\n\\kappa_{\\lb}(X,Z)\\ge\\begin{cases}\n{\\displaystyle \\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}}} & \\text{if }\\beta\\ge{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\\\\\n{\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}} & \\text{if }\\beta\\le{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\n\\end{cases}\\\\\n\\text{ where }\\qquad\\alpha=\\frac{\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}{\\|XX^{T}-ZZ^{T}\\|_{F}},\\qquad\\beta=\\frac{\\lambda_{\\min}(X^{T}X)}{\\|XX^{T}-ZZ^{T}\\|_{F}}{\\displaystyle \\frac{\\tr(Z_{\\perp}Z_{\\perp}^{T})}{\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}}.\n\\end{gather*}\n\\end{lemma}","start":43659,"end":44416},{"raw_tex":"\\begin{lemma}[Valid inequality]\n\\label{lem:ab1}For all $X,Z\\in\\R^{n\\times r}$ with $\\rank(Z)=r^{\\star}$,\nthe parameters $\\alpha$ and $\\beta$ defined in \\Lemref{abdef} satisfy\n$\\alpha^{2}+(r/r^{\\star})\\beta^{2}\\le1+[(\\beta-\\alpha)_{+}]^{2}.$\n\\end{lemma}","start":46119,"end":46371},{"raw_tex":"\\begin{proof}\n[Theorem~\\ref{thm:main}]For a fixed $r$ and $r^{\\star}$, recall\nthat we have defined $\\kappa^{\\star}$ as the condition number of\nthe \\emph{best-conditioned} counterexample, and formulated it as the\noptimal value for a two-stage optimization problem of the following\nform\n\\begin{align*}\n\\kappa^{\\star} & =\\inf_{\\phi:\\S^{n}\\to\\R}\\{\\kappa:\\phi\\text{ is }(\\kappa,r,r^{\\star})\\text{-counterexample}\\}\\\\\n & =\\inf_{X,Z\\in\\R^{n\\times r}}\\{\\kappa(X,Z):\\rank(Z)=r^{\\star},\\;XX^{T}\\ne ZZ^{T}\\}\n\\end{align*}\nWe claim that the minimum of the following expression over $\\alpha,\\beta\\ge0$\n\\[\n\\gamma(\\alpha,\\beta)\\eqdef\\begin{cases}\n\\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}} & \\text{if }\\beta\\ge{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\\\\\n{\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}} & \\text{if }\\beta\\le{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\n\\end{cases}\n\\]\nsubject to the valid inequality from \\lemref{ab1} is given:\n\\begin{equation}\n1+2\\sqrt{r/r^{\\star}}=\\min_{\\alpha,\\beta\\ge0}\\{\\gamma(\\alpha,\\beta):\\alpha^{2}+(r/r^{\\star})\\min\\{\\alpha^{2},\\beta^{2}\\}\\le1\\}.\\label{eq:minab}\n\\end{equation}\nTaking (\\ref{eq:minab}) to be true, substituting into \\lemref{abdef}\nand evoking \\lemref{heuristic} yields\n\\[\n1+2\\sqrt{r/r^{\\star}}\\le\\inf_{\\begin{subarray}{c}\nXX^{T}\\ne ZZ^{T}\\\\\n\\rank(Z)=r^{\\star}\n\\end{subarray}}\\kappa_{\\lb}(X,Z)\\le\\kappa^{\\star}\\le\\kappa_{\\ub}(X,Z)\\le1+2\\sqrt{r/r^{\\star}}.\n\\]\nThe upper-bound proves necessity: if $r/r^{\\star}\\le\\frac{1}{4}(\\kappa-1)^{2}$,\nthen there exists a $(\\kappa,r,r^{\\star})$-counterexample. The lower-bound\nproves sufficiency: if $r>\\frac{1}{4}(\\kappa-1)^{2}\\,r^{\\star}$ and\n$r^{\\star}\\le r<n$, then there exists no $(\\kappa,r,r^{\\star})$-counterexample\nto refute a ``no spurious local minima'' guarantee.\n\nWe now prove the claim in (\\ref{eq:minab}). We first minimize $\\gamma(\\alpha,\\beta)$\nsubject to $\\beta\\ge\\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}$ and find\nvia monotonicity with respect to $\\alpha$ that\n\\[\n\\min_{0\\le\\alpha\\le1}\\left\\{ \\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}}:\\beta\\ge{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}}\\right\\} ={\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}}.\n\\]\nIndeed, the minimizer $\\alpha$ lies at the boundary $\\beta=\\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}=\\frac{1-\\sqrt{1-\\alpha^{2}}}{\\alpha}$,\nwhere\n\\[\n\\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}}=\\frac{\\sqrt{1-\\alpha^{2}}+(1-\\alpha^{2})}{\\sqrt{1-\\alpha^{2}}-(1-\\alpha^{2})}=\\frac{1+\\sqrt{1-\\alpha^{2}}-\\alpha^{2}}{\\alpha^{2}-(1-\\sqrt{1-\\alpha^{2}})}={\\displaystyle \\frac{\\beta^{-1}-\\alpha}{\\alpha-\\beta}.}\n\\]\nNext, we minimize $\\gamma(\\alpha,\\beta)$ subject to $\\beta\\le\\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}\\le\\alpha$\nand the valid inequality:\n\\begin{align*}\n & \\min_{0\\le\\beta\\le\\alpha}\\left\\{ {\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}}:\\alpha^{2}+\\frac{r}{r^{\\star}}\\beta^{2}\\le1\\right\\} \\\\\n= & \\min_{\\beta\\ge0,\\;\\rho\\ge1}\\left\\{ {\\displaystyle \\frac{1-\\rho\\beta^{2}}{(\\rho-1)\\beta^{2}}}:(\\rho^{2}+\\frac{r}{r^{\\star}})\\beta^{2}\\le1\\right\\} =\\min_{\\rho\\ge1}\\left\\{ \\rho+\\frac{r}{r^{\\star}}\\frac{1}{\\rho-1}\\right\\} =1+2\\sqrt{\\frac{r}{r^{\\star}}}.\n\\end{align*}\nWe obtain the second line from the first by substituting $\\alpha=\\rho\\beta$\nfor $\\rho\\ge1$. The minimizers are $\\beta^{2}=(\\rho^{2}+\\frac{r}{r^{\\star}})^{-1}$\nand $\\rho=1+\\sqrt{r/r^{\\star}}$. \\qed\n\\end{proof}","start":46684,"end":50074},{"raw_tex":"\\begin{theorem}[Regularized Eckart--Young]\n\\label{thm:EckartYoung}Given $A\\in\\S_{+}^{n}$ and $B\\in\\S_{+}^{r}$\nwith $r\\le n$, let $A=\\sum_{i=1}^{n}s_{i}u_{i}u_{i}^{T}$ and $B=\\sum_{i=1}^{r}d_{i}v_{i}v_{i}^{T}$\ndenote the usual orthonormal eigendecompositions with $s_{1}\\ge\\cdots\\ge s_{n}\\ge0$\nand $0\\le d_{1}\\le\\cdots\\le d_{r}.$ Then, \n\\begin{align}\n\\min_{Y\\in\\R^{n\\times r}}\\left\\{ \\|A-YY^{T}\\|_{F}^{2}+2\\langle B,Y^{T}Y\\rangle\\right\\}  & =\\sum_{i=1}^{n}s_{i}^{2}-\\sum_{i=1}^{r}[(s_{i}-d_{i})_{+}]^{2}\\label{eq:ey-sol}\n\\end{align}\nwith minimizer $Y^{\\star}=\\sum_{i=1}^{r}u_{i}v_{i}^{T}\\sqrt{(s_{i}-d_{i})_{+}}$\nwhere $(\\cdot)_{+}\\eqdef\\max\\{0,\\cdot\\}$. \n\\end{theorem}","start":50630,"end":51298},{"raw_tex":"\\begin{proof}\n[\\thmref{EckartYoung}]Recall that $A=\\sum_{i=1}^{n}s_{i}u_{i}u_{i}^{T}$\nand $B=\\sum_{i=1}^{r}d_{i}v_{i}v_{i}^{T}$ with $s_{1}\\ge\\cdots\\ge s_{n}\\ge0$\nand $0\\le d_{1}\\le\\cdots\\le d_{r}.$ It follows from (\\ref{eq:Ruhe})\nthat\n\\begin{gather*}\n\\inner A{YY^{T}}\\le\\sum_{i=1}^{n}\\lambda_{i}(YY^{T})\\cdot\\lambda_{i}(A)=\\sum_{i=1}^{r}s_{i}\\sigma_{i}^{2}(Y),\\\\\n\\inner B{Y^{T}Y}\\ge\\sum_{i=1}^{r}\\lambda_{i}(Y^{T}Y)\\cdot\\lambda_{n-i+1}(B)=\\sum_{i=1}^{r}d_{i}\\sigma_{i}^{2}(Y).\n\\end{gather*}\nSubstituting yields the following lower-bound, which is valid for\nall choices of $Y$:\n\\begin{align*}\n\\|A-YY^{T}\\|_{F}^{2}+2\\inner B{Y^{T}Y}= & \\|A\\|_{F}^{2}+\\|YY^{T}\\|_{F}^{2}-2\\inner A{YY^{T}}+2\\inner B{Y^{T}Y}\\\\\n\\ge & \\sum_{i=1}^{n}s_{i}^{2}+\\sum_{i=1}^{r}\\sigma_{i}^{4}(Y)-2\\sum_{i=1}^{r}s_{i}\\sigma_{i}^{2}(Y)+2\\sum_{i=1}^{r}d_{i}\\sigma_{i}^{2}(Y)\\\\\n\\ge & \\sum_{i=1}^{n}s_{i}^{2}-\\sum_{i=1}^{r}[(s_{i}-d_{i})_{+}]^{2}.\n\\end{align*}\nThe final line applied the following result as a lower-bound:\n\\begin{align*}\n\\sigma_{i}^{4}(Y)-2(s_{i}-d_{i})\\sigma_{i}^{2}(Y)\\ge & \\min_{x_{i}\\ge0}x_{i}^{2}-2(s_{i}-d_{i})x_{i}\\\\\n= & \\min_{x_{i}\\ge0}[x_{i}-(s_{i}-d_{i})]^{2}-(s_{i}-d_{i})^{2}\\\\\n= & [(s_{i}-d_{i})_{+}-(s_{i}-d_{i})]^{2}-(s_{i}-d_{i})^{2}=-[(s_{i}-d_{i})_{+}]^{2}.\n\\end{align*}\nFinally, we verify that $Y^{\\star}=\\sum_{i=1}^{n}u_{i}v_{i}\\sqrt{(s_{i}-d_{i})_{+}}$\nattains the lower-bound derived above. \\qed\n\\end{proof}","start":52339,"end":53752},{"raw_tex":"\\begin{lemma}\n\\label{lem:numcard}If $x\\ge0$ and $\\one^{T}x\\le\\|x\\|^{2}$, then\n$\\one^{T}\\left(I-xx^{T}/\\|x\\|^{2}\\right)\\one\\ge\\|(\\one-x)_{+}\\|^{2}$.\n\\end{lemma}","start":53893,"end":54052},{"raw_tex":"\\begin{proof}\nDefine $u=\\alpha x,$ where $\\alpha=\\one^{T}x/\\|x\\|^{2}$ is chosen\nso that $\\mathbf{1}^{T}u=\\|u\\|^{2}$. First, verify that $\\mathbf{1}^{T}(I-xx^{T}/\\|x\\|^{2})\\mathbf{1}=\\|\\mathbf{1}-u\\|^{2}\\ge\\|(\\mathbf{1}-u)_{+}\\|^{2}$.\nNext, observe that $\\psi(\\alpha)\\equiv[(1-\\alpha t)_{+}]^{2}$ is\na decreasing function of $\\alpha\\ge0$ when $t\\ge0$, and therefore\n$\\|(\\one-\\alpha x)_{+}\\|^{2}\\ge\\|(\\one-x)_{+}\\|^{2}$ for $x\\ge0$\nand $0\\le\\alpha\\le1$. Finally, the hypotheses $x\\ge0$ and $\\one^{T}x\\le\\|x\\|^{2}$\nensure that $0\\le\\alpha\\le1$. \\qed\n\\end{proof}","start":54053,"end":54611},{"raw_tex":"\\begin{lemma}\n\\label{lem:keyclaim}Given $s,d\\in\\R_{+}^{n}$, let $s_{i}\\ge s_{\\lb}\\ge0$\nfor all $i$. Then we have\n\\begin{align*}\n\\|s\\|^{2}-\\|(s-d)_{+}\\|^{2}\\ge & \\begin{cases}\ns_{\\lb}^{2}\\frac{(\\one^{T}d)^{2}}{\\|d\\|^{2}} & \\text{if }s_{\\lb}\\one^{T}d\\le\\|d\\|^{2},\\\\\n2s_{\\lb}\\one^{T}d-\\|d\\|^{2} & \\text{if }s_{\\lb}\\one^{T}d\\ge\\|d\\|^{2}.\n\\end{cases}\n\\end{align*}\n\\end{lemma}","start":54613,"end":54983},{"raw_tex":"\\begin{proof}\nFirst, we observe that $\\psi(t)\\equiv t^{2}-[(t-\\alpha)_{+}]^{2}$\nis an increasing function of $t\\ge0$. It follows from element-wise\nmonotonicity that\n\\[\n\\|s\\|^{2}-\\|(s-d)_{+}\\|^{2}\\ge\\|s_{\\lb}\\one\\|^{2}-\\|(s_{\\lb}\\one-d)_{+}\\|^{2}=s_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-d/s_{\\lb})_{+}\\|^{2}\\right)\n\\]\nIf $s_{\\lb}\\one^{T}d\\le\\|d\\|^{2}$, then write $x=d/s_{\\lb}\\ge0$,\nand observe that $\\one^{T}x\\le\\|x\\|^{2}$ holds. Therefore, applying\n\\lemref{numcard} yields\n\\begin{align*}\ns_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-d/s_{\\lb})_{+}\\|^{2}\\right) & =s_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-x)_{+}\\|^{2}\\right)\\ge s_{\\lb}^{2}\\one^{T}\\left(xx^{T}/\\|x\\|^{2}\\right)\\one\\\\\n & =s_{\\lb}^{2}\\left([\\one^{T}(d/s_{\\lb})]^{2}/\\|d/s_{\\lb}\\|^{2}\\right)=s_{\\lb}^{2}(\\one^{T}d)^{2}/\\|d\\|^{2}.\n\\end{align*}\nIf otherwise $s_{\\lb}\\one^{T}d\\ge\\|d\\|^{2}$, then we have \n\\begin{align*}\ns_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-d/s_{\\lb})_{+}\\|^{2}\\right)\\ge s_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|\\one-d/s_{\\lb}\\|^{2}\\right)=2s_{\\lb}\\one^{T}d-\\|d\\|^{2}.\n\\end{align*}\n\\qed\n\\end{proof}","start":54984,"end":56040},{"raw_tex":"\\begin{lemma}\n\\label{lem:ab2}For $X,Z\\in\\R^{n\\times r}$, define the parameters\n\\[\n\\alpha=\\frac{\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}{\\|XX^{T}-ZZ^{T}\\|_{F}},\\qquad\\beta=\\frac{\\lambda_{\\min}(X^{T}X)}{\\|XX^{T}-ZZ^{T}\\|_{F}}\\cdot\\frac{\\tr(Z_{\\perp}Z_{\\perp}^{T})}{\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}.\n\\]\nThen, $\\alpha^{2}+(r/p)\\beta^{2}\\le1+[(\\beta-\\alpha)_{+}]^{2}$ holds\nfor all $p$ satisfying $\\rank(Z)\\le p\\le r$.\n\\end{lemma}","start":56125,"end":56540},{"raw_tex":"\\begin{proof}\nThe two parameters $\\alpha$ and $\\beta$ are unitarily invariant\nwith respect to $X,Z$, so we may assume without loss of generality\nthat $X,Z$ are partitioned as\n\\[\nX=\\begin{bmatrix}X_{1}\\\\\n0\n\\end{bmatrix}\\qquad Z=\\begin{bmatrix}Z_{1}\\\\\nZ_{2}\n\\end{bmatrix}\\qquad\\text{where }X_{1}\\in\\R^{r\\times r},\\;Z_{1}\\in\\R^{r\\times p},\\;Z_{2}\\in\\R^{(n-r)\\times p}.\n\\]\n(Otherwise, take the QR decomposition $X=QR$ and note that $\\alpha,\\beta$\nremain unchanged with $X\\gets Q^{T}X$ and $Z\\gets Q^{T}Z$.) Denote\n$s\\in\\R^{r}$ as the $r$ eigenvalues of $X_{1}X_{1}^{T}$ and $d\\in\\R^{p}$\nas the $p$ eigenvalues of $Z_{2}^{T}Z_{2}$ \n\\begin{align*}\ns_{i}\\eqdef & \\lambda_{i}(X_{1}X_{1}^{T})=\\lambda_{i}(XX^{T}),\\quad s_{1}\\ge s_{2}\\ge\\cdots\\ge s_{r}\\ge0,\\\\\nd_{i}\\eqdef & \\lambda_{i}(Z_{2}^{T}Z_{2})=\\lambda_{i}(Z_{\\perp}Z_{\\perp}^{T}),\\quad0\\le d_{1}\\le d_{2}\\le\\cdots\\le d_{p}.\n\\end{align*}\nWriting $\\e=\\vect(XX^{T}-ZZ^{T})$ yields\n\\begin{align*}\n\\alpha & =\\frac{\\|Z_{2}Z_{2}^{T}\\|_{F}}{\\|XX^{T}-ZZ^{T}\\|_{F}}=\\frac{\\|d\\|_{F}}{\\|\\e\\|_{F}}, & \\beta & =\\frac{\\lambda_{\\min}(X_{1}^{T}X_{1})}{\\|XX^{T}-ZZ^{T}\\|_{F}}\\cdot\\frac{\\tr(Z_{2}Z_{2}^{T})}{\\|Z_{2}Z_{2}^{T}\\|_{F}}=\\frac{s_{r}\\cdot\\one^{T}d}{\\|\\e\\|\\|d\\|}.\n\\end{align*}\nWe expand $\\|XX^{T}-ZZ^{T}\\|_{F}^{2}$ block-wise and evoke our Regularized\nEckart--Young Theorem (\\thmref{EckartYoung}) over the $r\\times p$\nmatrix block $Z_{1}$:\n\\begin{align*}\n\\|XX^{T}-ZZ^{T}\\|_{F}^{2} & =\\|X_{1}X_{1}^{T}-Z_{1}Z_{1}^{T}\\|_{F}^{2}+2\\langle Z_{2}^{T}Z_{2},Z_{1}^{T}Z_{1}\\rangle+\\|Z_{2}Z_{2}^{T}\\|^{2}\\\\\n & \\ge\\min_{Z_{1}\\in\\R^{r\\times p}}\\{\\|X_{1}X_{1}^{T}-Z_{1}Z_{1}^{T}\\|_{F}^{2}+2\\langle Z_{2}^{T}Z_{2},Z_{1}^{T}Z_{1}\\rangle\\}+\\|Z_{2}Z_{2}^{T}\\|^{2}\\\\\n & =\\|s\\|^{2}-\\|(s'-d)_{+}\\|^{2}+\\|d\\|^{2}\n\\end{align*}\nwhere $s'=(s_{1},s_{2},\\dots,s_{p})$. Now, evoking our technical\nlemma (\\lemref{keyclaim}) with $s_{\\lb}\\equiv s_{r}$ yields the\nfollowing two lower-bounds on $\\|XX^{T}-ZZ^{T}\\|_{F}^{2}=\\|\\e\\|^{2}$:\\begin{subequations}\\label{eq:key-claim}\n\\begin{gather}\ns_{r}\\one^{T}d\\le\\|d\\|^{2}\\quad\\implies\\quad\\|\\e\\|^{2}\\ge s_{r}^{2}\\frac{(\\one^{T}d)^{2}}{\\|d\\|^{2}}+(r-p)s_{r}^{2}+\\|d\\|^{2},\\label{eq:key-claim1}\\\\\ns_{r}\\one^{T}d\\ge\\|d\\|^{2}\\quad\\implies\\quad\\|\\e\\|^{2}\\ge2s_{r}\\one^{T}d-\\|d\\|^{2}+(r-p)s_{r}^{2}+\\|d\\|^{2}.\\label{eq:key-claim2}\n\\end{gather}\n\\end{subequations}Substituting $s_{r}=\\beta\\cdot\\|\\e\\|\\|d\\|/(\\one^{T}d)$\nyields\n\\[\n(r-p)s_{r}^{2}=(r-p)\\frac{\\|d\\|^{2}}{(\\one^{T}d)^{2}}\\beta^{2}\\|\\e\\|^{2}\\ge(r-p)\\frac{1}{\\nnz(d)}\\beta^{2}\\|\\e\\|^{2}\\ge\\frac{(r-p)}{p}\\beta^{2}\\|\\e\\|^{2},\n\\]\nand substituting into (\\ref{eq:key-claim}) yields\\begin{subequations}\n\\begin{gather}\n\\beta\\le\\alpha\\quad\\implies\\quad1\\ge\\beta^{2}+\\alpha^{2}+\\frac{(r-p)}{p}\\beta^{2}=\\alpha^{2}+\\frac{r}{p}\\beta^{2},\\label{eq:key-claim3}\\\\\n\\beta\\ge\\alpha\\quad\\implies\\quad1\\ge2\\alpha\\beta+\\frac{(r-p)}{p}\\beta^{2}=\\alpha^{2}+\\frac{r}{p}\\beta^{2}-(\\beta-\\alpha)^{2},\\label{eq:key-claim4}\n\\end{gather}\n\\end{subequations}as desired. \\qed\n\\end{proof}","start":56798,"end":59758},{"raw_tex":"\\begin{lemma}\n\\label{lem:I-JJ+}For any $X\\in\\R^{n\\times r}$, the matrix $\\J_{X}$\ndefined in (\\ref{eq:Jxdef}) satisfies $I-\\J_{X}\\J_{X}^{\\dagger}=(I-XX^{\\dagger})\\otimes(I-XX^{\\dagger})$\nwhere $\\dagger$ denotes the pseudoinverse. \n\\end{lemma}","start":60213,"end":60454},{"raw_tex":"\\begin{proof}\nLet $X=QR$ be the usual QR decomposition with orthogonal complement\n$Q_{\\perp}$, so that $XX^{\\dagger}=QQ^{T}$ and $I-XX^{\\dagger}=Q_{\\perp}Q_{\\perp}^{T}$.\nObserve that for any $M\\in\\R^{n\\times n}$:\n\\[\n\\J_{X}^{T}\\vect(M)=0\\iff(M+M^{T})Q=0\\iff M=Q_{\\perp}\\hat{M}Q_{\\perp}^{T}.\n\\]\nVectorizing the final equality as $\\vect(M)=(Q_{\\perp}\\otimes Q_{\\perp})\\vect(\\hat{M})$\nyields \n\\begin{align*}\n(I-\\J_{X}\\J_{X}^{\\dagger})b & =\\arg\\min_{v}\\{\\|v-b\\|:\\J_{X}^{T}v=0\\}\\\\\n & =\\arg\\min_{v}\\{\\|v-b\\|:v=(Q_{\\perp}\\otimes Q_{\\perp})\\hat{v}\\}\\\\\n & =(Q_{\\perp}\\otimes Q_{\\perp})(Q_{\\perp}\\otimes Q_{\\perp})^{T}b=(Q_{\\perp}Q_{\\perp}^{T}\\otimes Q_{\\perp}Q_{\\perp}^{T})b.\\quad\\qed\n\\end{align*}\n\\end{proof}","start":60455,"end":61154},{"raw_tex":"\\begin{lemma}\n\\label{lem:tradeoff}We have\n\\[\n\\kappa_{\\lb}(X,Z)=\\max_{t\\ge0}\\frac{1+\\cos\\theta(t)}{2t+1-\\cos\\theta(t)}\n\\]\nwhere $\\cos\\theta(t)$ is defined as follows\n\\begin{equation}\n\\cos\\theta(t)=\\max_{y,W_{i,j}}\\left\\{ \\e^{T}\\f:\\begin{array}{c}\n\\f\\equiv\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\quad\\|\\e\\|\\|\\f\\|=1,\\\\\n(I-ZZ^{\\dagger})\\,\\mat(\\f)\\,(I-ZZ^{\\dagger})\\succeq0,\\\\\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle=2t,\\quad W=[W_{i,j}]_{i,j=1}^{r}\\succeq0,\n\\end{array}\\right\\} \\label{eq:cosprob}\n\\end{equation}\nwith respect to $y\\in\\R^{nr}$ and $W_{i,j}\\in\\R^{n\\times n}$ for\n$i,j\\in\\{1,2,\\dots,r\\}$.\n\\end{lemma}","start":61392,"end":61992},{"raw_tex":"\\begin{lemma}\n\\label{lem:costheta}We have $\\cos\\theta(t)\\ge\\alpha\\cdot(t/\\beta)+\\sqrt{1-\\alpha^{2}}\\sqrt{1-t^{2}/\\beta^{2}}$\nfor $0\\le t\\le\\alpha\\beta$ where $Z_{\\perp}=(I-XX^{\\dagger})Z$ and\n\\begin{align*}\n\\alpha & =\\frac{\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}{\\|XX^{T}-ZZ^{T}\\|_{F}}, & \\beta & =\\frac{\\lambda_{\\min}(X^{T}X)\\cdot\\tr(Z_{\\perp}Z_{\\perp}^{T})}{\\|XX^{T}-ZZ^{T}\\|_{F}\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}.\n\\end{align*}\n\\end{lemma}","start":65276,"end":65706},{"raw_tex":"\\begin{proof}\nWrite $\\Pi=I-ZZ^{\\dagger}$ and $\\e=\\vect(XX^{T}-ZZ^{T})$, and decompose\n$\\e_{1}=(\\J_{X}\\J_{X}^{\\dagger})\\e$ and $\\e_{2}=(I-\\J_{X}\\J_{X}^{\\dagger})\\e$.\nWe verify that the following heuristic choice\n\\begin{gather*}\ny=\\underbrace{\\frac{\\sqrt{1-(t/\\beta)^{2}}}{\\|\\e\\|\\|\\e_{1}\\|}}_{\\gamma_{1}}\\;\\J_{X}^{\\dagger}\\e,\\quad W=\\underbrace{\\frac{t/\\beta}{\\|\\e\\|\\|\\e_{2}\\|}}_{\\gamma_{2}}\\;\\sum_{i=1}^{r}\\vect(Z_{\\perp}e_{i}v_{r}^{T})\\vect(Z_{\\perp}e_{i}v_{r}^{T})^{T},\n\\end{gather*}\nis feasible for (\\ref{eq:cosprob}). Indeed, $W\\succeq0$ holds, and\nwe have from (\\ref{eq:e2}), (\\ref{eq:Wsol1}), and (\\ref{eq:Wsol2}):\n\\begin{gather*}\n\\f=\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i})=\\gamma_{1}\\cdot\\J_{X}\\J_{X}^{\\dagger}\\e-\\gamma_{2}\\vect(Z_{\\perp}Z_{\\perp}^{T})=\\gamma_{1}\\cdot\\e_{1}+\\gamma_{2}\\e_{2},\\\\\n\\|\\e\\|^{2}\\|\\f\\|^{2}=\\gamma_{1}^{2}\\cdot\\|\\e\\|^{2}\\|\\e_{1}\\|^{2}+\\gamma_{2}^{2}\\cdot\\|\\e\\|^{2}\\|\\e_{2}\\|^{2}=(1-(t/\\beta)^{2})+(t/\\beta)^{2}=1,\\\\\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle=2\\left(\\frac{t/\\beta}{\\|\\e\\|\\|\\e_{2}\\|}\\right)\\cdot\\lambda_{\\min}(X^{T}X)\\|Z_{\\perp}\\|_{F}^{2}=\\frac{2t}{\\beta}\\cdot\\frac{\\sigma_{r}^{2}\\cdot\\tr(Z_{\\perp}Z_{\\perp}^{T})}{\\|\\e\\|\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}=2t.\n\\end{gather*}\nTo verify that $\\Pi\\,\\mat(\\f)\\,\\Pi\\succeq0$, note that $t\\le\\alpha\\beta$\nimplies $t/\\beta\\le\\alpha\\le1$, so it follows from $\\alpha=\\|\\e_{2}\\|/\\|\\e\\|$\nand $\\sqrt{1-\\alpha^{2}}=\\|\\e_{1}\\|/\\|\\e\\|$ that\n\\[\n\\gamma_{1}=\\frac{\\sqrt{1-(t/\\beta)^{2}}}{\\|\\e_{1}\\|\\|\\e\\|}\\ge\\frac{\\sqrt{1-\\alpha^{2}}}{\\|\\e_{1}\\|\\|\\e\\|}=\\frac{1}{\\|\\e\\|^{2}}=\\frac{\\alpha}{\\|\\e_{2}\\|\\|\\e\\|}\\ge\\frac{t/\\beta}{\\|\\e_{2}\\|\\|\\e\\|}=\\gamma_{2}.\n\\]\nIn turn, $\\gamma_{1}\\ge\\gamma_{2}$ implies\n\\begin{align*}\n\\f & =\\gamma_{1}\\cdot\\e_{1}+\\gamma_{2}\\cdot\\e_{2}=\\gamma_{1}\\cdot\\e-(\\gamma_{1}-\\gamma_{2})\\e_{2},\\\\\n\\mat(\\f) & =\\gamma_{1}\\cdot(XX^{T}-ZZ^{T})+(\\gamma_{1}-\\gamma_{2})\\cdot Z_{\\perp}Z_{\\perp}^{T},\\\\\n\\Pi\\,\\mat(\\f)\\,\\Pi & =\\gamma_{1}\\cdot\\Pi(XX^{T})\\Pi+(\\gamma_{1}-\\gamma_{2})\\cdot\\Pi(Z_{\\perp}Z_{\\perp}^{T})\\Pi\\succeq0.\n\\end{align*}\nHaving verified $y$ and $W$ as feasible for (\\ref{eq:cosprob}),\nwe use its objective value to heuristically lower-bound the maximum\n\\[\n\\cos\\theta(t)\\ge\\e^{T}\\f=\\gamma_{1}\\cdot\\|\\e_{1}\\|^{2}+\\gamma_{2}\\cdot\\|\\e_{2}\\|^{2}=\\sqrt{1-(t/\\beta)^{2}}\\sqrt{1-\\alpha^{2}}+(t/\\beta)\\cdot\\alpha.\n\\]\n \\qed\n\\end{proof}","start":65707,"end":68016},{"raw_tex":"\\begin{lemma}\n\\label{lem:counter}Let $[Q_{1},Q_{2}]$ have orthonormal columns\nwith $Q_{1}\\in\\R^{n\\times r}$ and $Q_{2}\\in\\R^{n\\times r^{\\star}}$.\nIf $\\kappa=1+2\\sqrt{r/r^{\\star}}$, and $\\H$ is any $n^{2}\\times n^{2}$\nmatrix that satisfies $I\\preceq\\H\\preceq\\kappa I$ and the following\n$rr^{\\star}+2$ eigenvalue equations\n\\begin{gather*}\n\\H\\vect\\left(Q_{1}Q_{1}^{T}+\\sqrt{r/r^{\\star}}Q_{2}Q_{2}\\right)=\\kappa\\cdot\\vect\\left(Q_{1}Q_{1}^{T}+\\sqrt{r/r^{\\star}}Q_{2}Q_{2}\\right),\\\\\n\\H\\vect\\left(Q_{1}Q_{1}^{T}-\\sqrt{r/r^{\\star}}Q_{2}Q_{2}\\right)=1\\cdot\\vect\\left(Q_{1}Q_{1}^{T}-\\sqrt{r/r^{\\star}}Q_{2}Q_{2}\\right),\\\\\n\\H\\vect\\left(Q_{1}VQ_{2}^{T}+Q_{2}V^{T}Q_{1}^{T}\\right)=\\kappa\\cdot\\vect\\left(Q_{1}VQ_{2}^{T}+Q_{2}V^{T}Q_{1}^{T}\\right)\\qquad\\text{for all }V\\in\\R^{r\\times r^{\\star}},\n\\end{gather*}\nthen $(\\kappa,\\H)$ is a feasible point for (\\ref{eq:kappaub_prob})\nwith $X=Q_{1}$ and $Z=\\sqrt{1+\\sqrt{r/r^{\\star}}}Q_{2}$. Moveover,\n$\\e^{T}\\H\\e=(1+2\\sqrt{r/r^{\\star}})\\cdot(1+\\sqrt{r/r^{\\star}})\\cdot r^{\\star}$.\n\\end{lemma}","start":69778,"end":70798},{"raw_tex":"\\begin{proof}\nWrite $\\rho\\equiv\\sqrt{r/r^{\\star}}$, and take $[Q_{1},Q_{2}]=I_{r+r^{\\star}}$\nwithout loss of generality. Decomposing $\\e=\\vect(XX^{T}-ZZ^{T})$\ninto eigenvectors and applying $\\H$ yields\n\\begin{gather*}\n\\mat(\\e)=\\begin{bmatrix}I_{r} & 0\\\\\n0 & -(1+\\rho)I_{r^{\\star}}\n\\end{bmatrix}=-\\frac{1}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & \\rho I_{r^{\\star}}\n\\end{bmatrix}+\\frac{1+2\\rho}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & -\\rho I_{r^{\\star}}\n\\end{bmatrix},\\\\\n\\mat(\\H\\e)=-\\frac{1+2\\rho}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & \\rho I_{r^{\\star}}\n\\end{bmatrix}+\\frac{1+2\\rho}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & -\\rho I_{r^{\\star}}\n\\end{bmatrix}=\\begin{bmatrix}0_{r} & 0\\\\\n0 & -\\kappa I_{r^{\\star}}\n\\end{bmatrix}.\n\\end{gather*}\nClearly, $\\inner{\\mat(\\e)}{\\mat(\\H\\e)}=\\kappa(1+\\rho)r^{\\star}.$\nFor $v=\\vect(V)=\\vect([V_{1};V_{2}])$ with $V_{1}\\in\\R^{r\\times r}$\nand $V_{2}\\in\\R^{r^{\\star}\\times r}$, observe that \n\\begin{gather*}\nv^{T}\\J_{X}\\H\\e=\\left\\langle \\begin{bmatrix}V_{1}+V_{1}^{T} & V_{2}^{T}\\\\\nV_{2} & 0_{r^{\\star}}\n\\end{bmatrix},\\begin{bmatrix}0_{r} & 0\\\\\n0 & -\\kappa I_{r^{\\star}}\n\\end{bmatrix}\\right\\rangle =0,\\\\\nv^{T}[I_{r}\\otimes\\mat(\\H\\e)]v=\\langle\\mat(\\H\\e),VV^{T}\\rangle=\\left\\langle \\begin{bmatrix}0_{r} & 0\\\\\n0 & -\\kappa I_{r^{\\star}}\n\\end{bmatrix},\\begin{bmatrix}V_{1}V_{1}^{T} & V_{1}V_{2}^{T}\\\\\nV_{2}V_{1}^{T} & V_{2}V_{2}^{T}\n\\end{bmatrix}\\right\\rangle =-\\kappa\\|V_{2}\\|^{2},\\\\\nv^{T}\\J_{X}^{T}\\H\\J_{X}v=\\left\\Vert \\H^{1/2}\\vect\\left(\\begin{bmatrix}V_{1}+V_{1}^{T} & V_{2}^{T}\\\\\nV_{2} & 0_{r^{\\star}}\n\\end{bmatrix}\\right)\\right\\Vert ^{2}\\ge\\left\\Vert \\H^{1/2}\\vect\\left(\\begin{bmatrix}0_{r} & V_{2}^{T}\\\\\nV_{2} & 0_{r^{\\star}}\n\\end{bmatrix}\\right)\\right\\Vert ^{2}=2\\kappa\\|V_{2}\\|^{2}.\n\\end{gather*}\nTherefore, $v^{T}\\J_{X}\\H\\e=0$ and $2v^{T}[I_{r}\\otimes\\mat(\\H\\e)]v\\ge-v^{T}\\J_{X}^{T}\\H\\J_{X}v$\nhold for all $v$. \\qed\n\\end{proof}","start":70799,"end":72653},{"raw_tex":"\\section{\\label{app:tradeoff}Derivation of the dual for the lower-bound problem\n(Proof of \\lemref{tradeoff})}\n\nWe will derive the Lagrangian dual for the following\n\\begin{align*}\n\\kappa_{\\lb}(X,Z) & =\\min_{\\kappa,\\H,s}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H\\preceq\\kappa I,\\qquad\\J_{Z}^{T}s=0,\\qquad s\\in\\vect(\\S_{+}^{n}),\\\\\n\\J_{X}^{T}(\\H\\e+s)=0,\\quad2I_{r}\\otimes\\mat(\\H\\e+s)+\\kappa\\J_{X}^{T}\\J_{X}\\succeq0.\n\\end{array}\\right\\} \n\\end{align*}\nRecall that $\\e=\\vect(XX^{T}-ZZ^{T})$ and $\\J_{X}\\vect(V)=\\vect(XV^{T}+VX^{T})$\nwith respect to fixed $X,Z\\in\\R^{n\\times r}$. Observe that\n\\[\n\\J_{Z}^{T}\\vect(S)=0\\iff(S+S^{T})Z=0\\iff S=Q_{\\perp}S_{\\perp}Q_{\\perp}^{T}\\iff\\vect(S)=\\Q_{\\perp}\\vect(S_{\\perp})\n\\]\nwhere the orthogonal complement $Q_{\\perp}$ of $Z$ is such that\n$Q_{\\perp}Q_{\\perp}^{T}=I_{n}-ZZ^{\\dagger}$, and $\\Q_{\\perp}=Q_{\\perp}\\otimes Q_{\\perp}$.\nSubstituting and taking the Lagrangian dual yields\n\\begin{align}\n\\kappa_{\\lb}(X,Z) & =\\min_{\\kappa,\\H,s_{\\perp}}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H\\preceq\\kappa I,\\quad s_{\\perp}\\in\\vect(\\S_{+}^{n-r^{\\star}}),\\quad\\J_{X}^{T}(\\H\\e+\\Q_{\\perp}s_{\\perp})=0,\\\\\n2I_{r}\\otimes\\mat(\\H\\e+\\Q_{\\perp}s_{\\perp})+\\kappa\\J_{X}^{T}\\J_{X}\\succeq0\n\\end{array}\\right\\} \\nonumber \\\\\n & =\\max_{y,W_{i,j},U,V}\\left\\{ \\tr(U):\\begin{array}{c}\n\\f\\e^{T}+\\e\\f^{T}=U-V,\\quad\\tr(V)+\\inner{\\J_{X}^{T}\\J_{X}}W=1,\\\\\n\\f\\equiv\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\quad\\Q_{\\perp}^{T}\\f\\in\\vect(\\S_{+}^{n-r^{\\star}}),\\\\\nU,V\\succeq0,\\quad W=[W_{i,j}]_{i,j=1}^{r}\\succeq0\n\\end{array}\\right\\} \\label{eq:ub2-1}\n\\end{align}\nin which $y\\in\\R^{nr}$ and $W_{i,j}\\in\\R^{n\\times n}$ for $i,j\\in\\{1,2,\\dots,r\\}$.\nStrong duality holds here because $y=0,$ $W=\\epsilon\\cdot I_{nr},$\n$U=\\frac{1}{n}I_{n},$ and $V=\\frac{1}{n}I_{n}+\\epsilon\\cdot r\\cdot[\\e\\vect(I_{n})^{T}+\\e\\vect(I_{n})^{T}]$\nis strictly feasible for a sufficiently small $\\epsilon>0$. \n\nThe dual problem (\\ref{eq:ub2-1}) has a closed-form solution over\n$U$ and $V$ \\citep[Lemma~13]{zhang2019sharp}\n\\begin{align*}\n\\frac{\\tr([M]_{-})+\\alpha}{\\tr([M]_{+})}= & \\min_{t,U,V}\\left\\{ \\tr(V)+\\alpha\\cdot t:\\begin{array}{c}\ntM=U-V,\\quad t\\ge0,\\\\\n\\tr(U)=1,\\quad U,V\\succeq0\n\\end{array}\\right\\} ,\n\\end{align*}\nand $\\f\\e^{T}+\\e\\f^{T}$ has exactly two nonzero eigenvalues $\\e^{T}\\f\\pm\\|\\e\\|\\|\\f\\|$\n\\citep[Lemma~14]{zhang2019sharp}. Substituting this solution yields\nexactly \n\\begin{align*}\n\\kappa_{\\lb}(X,Z) & =\\max_{y,W_{i,j}}\\left\\{ \\frac{\\|\\e\\|\\|\\f\\|+\\e^{T}\\f}{\\|\\e\\|\\|\\f\\|-\\e^{T}\\f+\\inner{\\J_{X}^{T}\\J_{X}}W}:\\begin{array}{c}\n\\Q_{\\perp}\\Q_{\\perp}^{T}\\f\\in\\vect(\\S_{+}^{n}),\\\\\n\\f=\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\\\\nW=[W_{i,j}]_{i,j=1}^{r}\\succeq0\n\\end{array}\\right\\} \\\\\n & =\\max_{t\\ge0}\\frac{1+\\cos\\theta(t)}{2t+1-\\cos\\theta(t)}\n\\end{align*}\nwhere $\\cos\\theta(t)$ is itself is defined as the following maximization\n\\[\n\\cos\\theta(t)=\\max_{y,W_{i,j}}\\left\\{ \\e^{T}\\f:\\begin{array}{c}\n\\f\\equiv\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\quad\\|\\e\\|\\|\\f\\|=1,\\\\\n(I-ZZ^{\\dagger})\\,\\mat(\\f)\\,(I-ZZ^{\\dagger})\\succeq0,\\\\\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle=2t,\\quad W=[W_{i,j}]_{i,j=1}^{r}\\succeq0.\n\\end{array}\\right\\} \n\\]\n\n","start":84068,"end":87151}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:cvx","eq:ncvx"],"declared_external":false,"source_only":false,"title":"半正定优化与因子化问题","source_units":[{"raw_tex":"Consider minimizing a convex objective $\\phi$ over an $n\\times n$\npositive semidefinite matrix $M\\succeq0$, as in\n\\begin{equation}\nM^{\\star}=\\quad\\mathrm{minimize}\\quad\\phi(M)\\quad\\text{over }\\quad M\\succeq0,\\tag{P}\\label{eq:cvx}\n\\end{equation}\nIn practice, the matrix order $n$ is often so large---from tens\nof thousands to hundreds of millions---that even explicitly forming\nthe $n^{2}$ elements of the $n\\times n$ matrix variable $M$ would\nbe intractable. Where $r^{\\star}=\\rank(M^{\\star})$ is known a priori\nto be small, the standard approach is the nonconvex Burer--Monteiro\nfactorization~\\citep{burer2003nonlinear}, which rewrites $M=XX^{T}$\nand then directly optimizes over $X$ using a local optimization algorithm:\n\\begin{equation}\nX^{\\star}=\\quad\\mathrm{minimize}\\quad f(X)\\eqdef\\phi(XX^{T})\\text{ where }X\\text{ is }n\\times r\\text{ and }r\\ge r^{\\star}.\\tag{BM}\\label{eq:ncvx}\n\\end{equation}\n","start":5847,"end":6748}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"伪局部极小与计算动机","source_units":[{"raw_tex":"This way, the semidefinite constraint $M=XX^{T}\\succeq0$ is automatically\nenforced for free. Moreover, by using a small search rank $r\\ll n$,\nthe number of variables is reduced to $O(n)$. However, this improved\nscalability comes at the cost of giving up the convexity of (\\ref{eq:cvx}).\nIn principle, local optimization can fail by getting stuck at a \\emph{spurious}\nlocal minimum of (\\ref{eq:ncvx})---a local minimum that is strictly\nworse than that of the global minimum. In practice, however, there\nseems to be many situations where this failure mode does not occur~\\citep{candes2015phase,rosen2019se,zhang2019spurious}.\n","start":6748,"end":7372}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"全域光滑与强凸假设","source_units":[{"raw_tex":"To state this precisely, suppose that $\\phi$ is twice-differentiable,\n$L$-smooth, and $\\mu$-strongly convex over the entire set of $n\\times n$\nreal symmetric matrices $\\S^{n}$, as in\n\\[\n\\mu\\|E\\|_{F}^{2}\\le\\inner{\\nabla^{2}\\phi(M)[E]}E\\le L\\|E\\|_{F}^{2}\\qquad\\text{for all }M\\in\\S^{n},\n\\]\nwhere $\\inner EF\\eqdef\\tr(E^{T}F)$ and $\\|E\\|_{F}\\eqdef\\sqrt{\\inner EE}$\ndenote the matrix Euclidean inner product and norm respectively. \\citet{bhojanapalli2016global}\n","start":7706,"end":8163}]},{"id":"S004","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"既有条件数保证","source_units":[{"raw_tex":"pointed out that if $\\phi$ has condition number $L/\\mu<3/2$, then\nthe usual second-order necessary conditions for \\emph{local} optimality\nin $f$ are also sufficient for \\emph{global} optimality, meaning\nthat\n\\[\n\\nabla f(X)=0,\\quad\\nabla^{2}f(X)\\succeq0\\quad\\iff\\quad f(X)=\\min_{U}f(U).\n\\]\n","start":8163,"end":8452}]},{"id":"S005","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"矩阵、锥与正部记号","source_units":[{"raw_tex":"\\textbf{Basic linear algebra.} Lower-case letters are vectors and\nupper-case letters are matrices. We use ``MATLAB notation'' in concatenating\nvectors and matrices:\n\n\\[\n[a,b]=\\begin{bmatrix}a & b\\end{bmatrix},\\qquad[a;b]=\\begin{bmatrix}a\\\\\nb\n\\end{bmatrix},\\qquad\\diag(a,b)=\\begin{bmatrix}a & 0\\\\\n0 & b\n\\end{bmatrix},\n\\]\nand the following short-hand to construct them:\n\\[\n[x_{i}]_{i=1}^{n}=\\begin{bmatrix}x_{1}\\\\\n\\vdots\\\\\nx_{n}\n\\end{bmatrix},\\qquad[x_{i,j}]_{i,j=1}^{m,n}=\\begin{bmatrix}x_{1,1} & \\cdots & x_{1,n}\\\\\n\\vdots & \\ddots & \\vdots\\\\\nx_{m,1} & \\cdots & x_{m,n}\n\\end{bmatrix}.\n\\]\nDenote $\\one=[1,1,\\dots,1]^{T}$ as the vector-of-ones and $I=\\diag(\\one)$\nas the identity matrix; we will infer their dimensions from context.\nDenote $\\langle X,Y\\rangle=\\tr(X^{T}Y)$ and $\\|X\\|_{F}^{2}=\\langle X,X\\rangle$\nas the Frobenius inner product and norm. Denote $\\nnz(x)$ as the\nnumber of nonzero elements in $x$. The sets $\\S^{n}\\supset\\S_{+}^{n}$\nare the $n\\times n$ real symmetric matrices, and the corresponding\npositive semidefinite cone.\n\n\\textbf{Positive cones and projections.} The sets $\\R^{n}\\supset\\R_{+}^{n}$\nare the $n$ vectors, and the corresponding positive orthant. Denote\n$x_{+}=\\max\\{0,x\\}=\\arg\\min\\{\\|x-y\\|:y\\in\\R_{+}^{n}\\}$ as the projection\nonto the positive orthant. \n","start":32917,"end":34202}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"向量化与Kronecker积","source_units":[{"raw_tex":"\n\\textbf{Vectorization and Kronecker product.} Denote $\\vect(X)$ as\nthe usual column-stacking vectorizating operator, and $\\mat(x)$ as\nits inverse (the dimensions of the matrix are inferred from context).\nThe Kronecker product $\\otimes$ is defined to satisfy the identity\n$\\vect(AXB^{T})=(B\\otimes A)\\vect(X)$.\n\n","start":34202,"end":34514}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"Moore–Penrose伪逆","source_units":[{"raw_tex":"\\textbf{Pseudoinverse.} Denote the (Moore--Penrose) pseudoinverse\n$A^{\\dagger}\\eqdef VS^{-1}U^{T}$ of matrix $A=USV^{T}$ where $U^{T}U=V^{T}V=I_{r}$\nand $S\\succ0$. Define $0^{\\dagger}\\eqdef0$.\n\n","start":34514,"end":34708}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"误差向量与因子Jacobian","source_units":[{"raw_tex":"\\textbf{Error vector $\\e$ and Jacobian matrix $\\J_{X}$}. Given fixed\n$X\\in\\R^{n\\times r}$ and $Z\\in\\R^{n\\times r}$, we denote $\\e=\\vect(XX^{T}-ZZ^{T})$\nand implicitly define $\\J_{X}\\in\\R^{n^{2}\\times nr}$ to satisfy $\\J_{X}\\vect(V)=\\vect(XV^{T}+VX^{T})$.\nNote that the associate adjoint operator satisfies $\\J_{X}^{T}\\vect(M)=\\vect((M+M^{T})X)$. \n","start":34708,"end":35055}]},{"id":"S009","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"排除良态过参数化反例","source_units":[{"raw_tex":"We will establish ``no spurious local minima'' guarantees by showing\nthat there cannot exist a counterexample that is simultaneously highly\noverparameterized, with a large value of $r/r^{\\star}$, and also\nwell-conditioned, with a small value of $\\kappa$. \n","start":35109,"end":35365}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"条件数受限反例","source_units":[{"raw_tex":"\\begin{definition}\nThe function $\\phi:\\S^{n}\\to\\R$ is said to be a $(\\kappa,r,r^{\\star})$-\\emph{counterexample}\nif it satisfies the following:\n\\begin{itemize}\n\\item The function $\\phi$ is twice-differentiable, $L$-smooth, $\\mu$-strongly\nconvex, and $L/\\mu\\le\\kappa.$\n\\item The minimizer $M^{\\star}=\\arg\\min_{M\\succeq0}\\phi(M)$ satisfies $\\rank(M^{\\star})=r^{\\star}\\le r$.\n\\item There exists an $n\\times r$ spurious point $X$ with $XX^{T}\\ne M^{\\star}$\nthat satisfies $\\nabla f(X)=0$ and $\\nabla^{2}f(X)\\succeq0$ for\n$f(U)\\eqdef\\phi(UU^{T})$. \n\\end{itemize}\n\\end{definition}\n","start":35365,"end":35939}]},{"id":"S011","semantic_type":"interpretation","kind":"interpretation","labels":["eq:inf_kappa"],"declared_external":false,"source_only":false,"title":"最优反例阈值的证明目标","source_units":[{"raw_tex":"For a fixed $r$ and $r^{\\star}$, our central claim in \\thmref{main}\nis that the condition number $\\kappa^{\\star}$ of the \\emph{best-conditioned}\ncounterexample is exactly determined as \n\\begin{equation}\n\\kappa^{\\star}=\\inf_{\\phi:\\S^{n}\\to\\R}\\{\\kappa:\\phi\\text{ is }(\\kappa,r,r^{\\star})\\text{-counterexample}\\}=1+2\\sqrt{r/r^{\\star}}.\\label{eq:inf_kappa}\n\\end{equation}\nThe sufficient condition follows because there exists no $(\\kappa,r,r^{\\star})$-counterexample\nwith $\\kappa<1+2\\sqrt{r/r^{\\star}}$, and so for a fixed $\\kappa$,\nwe can always overparameterize $r/r^{\\star}>\\frac{1}{4}(\\kappa-1)^{2}$\nto eliminate all counterexamples. The neccessary condition holds because\na best-conditioned $(\\kappa^{\\star},r,r^{\\star})$-counterexample\ndoes exist.\n","start":35939,"end":36689}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["eq:mu_reform"],"declared_external":false,"source_only":false,"title":"两阶段条件数优化","source_units":[{"raw_tex":"\nTo prove that $\\kappa^{\\star}$ is equal to the value stated in (\\ref{eq:inf_kappa}),\nwe proceed by reformulating (\\ref{eq:inf_kappa}) into two-stage optimization\nproblem \n\\begin{align*}\n\\kappa^{\\star} & =\\inf_{X,Z\\in\\R^{n\\times r}}\\{\\kappa(X,Z):\\rank(Z)=r^{\\star},\\;XX^{T}\\ne ZZ^{T}\\}\n\\end{align*}\nin which $\\kappa(X,Z)$ corresponds to the best-conditioned counterexample\nassociated with a \\emph{fixed} minimizer $ZZ^{T}$ and a \\emph{fixed\n}spurious point $X$:\n\\begin{equation}\n\\kappa(X,Z)=\\inf_{\\begin{subarray}{c}\n\\phi:\\S^{n}\\to\\R\\end{subarray}}\\left\\{ \\kappa:\\begin{array}{c}\n\\phi\\text{ is }1\\text{-strongly convex and }\\kappa\\text{-smooth,}\\\\\nZZ^{T}=\\arg\\min_{M\\succeq0}\\phi(M),\\\\\n\\nabla f(X)=0,\\;\\nabla^{2}f(X)\\succeq0\\text{ where }f(U)\\eqdef\\phi(UU^{T}).\n\\end{array}\\right\\} \\label{eq:mu_reform}\n\\end{equation}\n","start":36689,"end":37507}]},{"id":"S013","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"SDP路线的文献背景","source_units":[{"raw_tex":"This two-stage reformulation is inspired by \\citet{zhang2018much,zhang2019sharp},\nwho showed for a fixed $X,Z$ that finding the best-conditioned \\emph{quadratic}\ncounterexample $\\phi$ that satisfies $\\nabla\\phi(ZZ^{T})=0$ can\nbe exactly posed as a standard-form \\emph{semidefinite program} or\nSDP. Our proof extends this SDP formulation to a nonquadratic $\\phi$\nwith a possibly nonzero $\\nabla\\phi(ZZ^{T})\\succeq0$ at optimality\nby introducing a slack variable.   \n","start":37507,"end":37972}]},{"id":"S014","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"垂直分量消失的源文约定","source_units":[{"raw_tex":"\\begin{remark}\nThe definition of $\\beta$ becomes ambiguous when $Z_{\\perp}=0$.\nThis is without loss of precision, because $Z_{\\perp}=0$ implies\n$\\alpha=0$, and therefore $\\kappa_{\\lb}(X,Z)=1$ regardless of the\nvalue of $\\beta$. \n\\end{remark}\n","start":44417,"end":44659}]},{"id":"S015","semantic_type":"external_fact","kind":"external_fact","labels":["eq:Ruhe"],"declared_external":true,"declaration_text":["原始前提列表第3项；源文引用ruhe1970perturbation及marshall2011inequalities第9章。"],"source_only":false,"title":"Ruhe迹不等式","source_units":[{"raw_tex":"Our proof of \\thmref{EckartYoung} follows a trace inequality of Ruhe~\\citep{ruhe1970perturbation}\n(see also \\citep[Chapter~9]{marshall2011inequalities}).\\footnote{This simplified proof via Ruhe's inequality is due to an anonymous\nreviewer.} The inequality states that, for $n\\times n$ positive semidefinite\nmatrices $A,B$, we have\n\\begin{equation}\n\\sum_{i=1}^{n}\\lambda_{i}(A)\\lambda_{n-i+1}(B)\\le\\inner AB\\le\\sum_{i=1}^{n}\\lambda_{i}(A)\\lambda_{i}(B)\\label{eq:Ruhe}\n\\end{equation}\nwhere the eigenvalues are ordered $\\lambda_{1}(A)\\ge\\cdots\\ge\\lambda_{n}(A)$\nand $\\lambda_{1}(B)\\ge\\cdots\\ge\\lambda_{n}(B)$. \n","start":51731,"end":52339}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":["eq:Jxdef"],"declared_external":false,"source_only":false,"title":"Jacobian及伴随的再陈述","source_units":[{"raw_tex":"\\begin{equation}\n\\J_{X}\\vect(V)=\\vect(XV^{T}+VX^{T}),\\qquad\\J_{X}^{T}\\vect(M)=\\vect((M+M^{T})X)\\label{eq:Jxdef}\n\\end{equation}","start":60030,"end":60156}]},{"id":"S017","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:e2"],"declared_external":false,"source_only":false,"title":"误差的垂直投影","source_units":[{"raw_tex":"\\begin{equation}\n\\mat((I-\\J_{X}\\J_{X}^{\\dagger})\\e)=(I-XX^{\\dagger})(XX^{T}-ZZ^{T})(I-XX^{\\dagger})=-Z_{\\perp}Z_{\\perp}^{T}\\label{eq:e2}\n\\end{equation}","start":62940,"end":63091}]},{"id":"S018","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表第6项明确接受B0102的初等分块迹和Jacobian恒等式，无需单独证明。"],"source_only":false,"title":"分块迹与Jacobian恒等式","source_units":[{"raw_tex":"\\begin{claim}\nLet $W=\\sum_{j=1}^{m}\\vect(V_{j})\\vect(V_{j})^{T}$ where $V_{1},\\dots,V_{m}\\in\\R^{n\\times r}$.\nPartition $W$ into $r\\times r$ blocks of $n\\times n$, as in $W=[W_{i,j}]_{i,j=1}^{r}$\nfor $W_{i,j}\\in\\R^{n\\times n}$. Then, \n\\[\n\\sum_{i=1}^{r}W_{i,i}=\\sum_{j=1}^{m}V_{j}V_{j}^{T},\\qquad\\inner{\\J_{X}^{T}\\J_{X}}W=\\sum_{j=1}^{m}\\|XV_{j}^{T}+V_{j}X^{T}\\|_{F}^{2}.\n\\]\n\\end{claim}","start":63317,"end":63700}]},{"id":"S019","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:Wsol1"],"declared_external":false,"source_only":false,"title":"候选W的对角块和","source_units":[{"raw_tex":"\\begin{gather}\n\\sum_{i=1}^{r}W_{i,i}=\\gamma_{2}\\cdot\\sum_{j=1}^{r}Z_{\\perp}e_{i}v_{r}^{T}v_{r}e_{i}^{T}Z_{\\perp}^{T}=\\gamma_{2}\\cdot Z_{\\perp}\\left(\\sum_{i=1}^{r}e_{i}e_{i}^{T}\\right)Z_{\\perp}^{T}=\\gamma_{2}\\cdot Z_{\\perp}Z_{\\perp}^{T}.\\label{eq:Wsol1}\n\\end{gather}","start":64405,"end":64670}]},{"id":"S020","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:Wsol2"],"declared_external":false,"source_only":false,"title":"候选W的Jacobian代价","source_units":[{"raw_tex":"\\begin{align}\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle & =\\gamma_{2}\\cdot\\sum_{i=1}^{m}\\|\\J_{X}\\vect(Z_{\\perp}e_{i}v_{r}^{T})\\|^{2}=\\gamma_{2}\\cdot\\sum_{i=1}^{m}\\|Xv_{r}e_{i}^{T}Z_{\\perp}^{T}+Z_{\\perp}e_{i}v_{r}^{T}X^{T}\\|_{F}^{2}\\nonumber \\\\\n & =\\gamma_{2}\\cdot\\sum_{i=1}^{m}2\\|Z_{\\perp}e_{i}v_{r}^{T}X^{T}\\|_{F}^{2}=2\\gamma_{2}\\cdot\\lambda_{\\min}(X^{T}X)\\|Z_{\\perp}\\|_{F}^{2}\\label{eq:Wsol2}\n\\end{align}","start":64689,"end":65086}]},{"id":"S021","semantic_type":"external_fact","kind":"external_fact","labels":["lem:optim"],"declared_external":true,"declaration_text":["原始前提列表第4项；源文归于zhang2019sharp的Theorem 12证明中使用的标量恒等式。"],"source_only":false,"title":"标量分式最大化恒等式","source_units":[{"raw_tex":"Finally, we evoke the following lemma, which was previously used in\nthe proof of \\citet[Theorem~12]{zhang2019sharp} to fully solve the\nrank-1 case in closed-form.\n\\begin{lemma}\n\\label{lem:optim}Given $0\\le\\alpha<1$ and $\\beta\\ge0$, let $\\gamma_{\\alpha,\\beta}:\\R_{+}\\to[0,1]$\nsatisfy $\\gamma_{\\alpha,\\beta}(t)=\\alpha\\cdot(t/\\beta)+\\sqrt{1-\\alpha^{2}}\\sqrt{1-t^{2}/\\beta^{2}}$\nfor $t\\le\\alpha\\beta$. Then, we have\n\\[\n\\max_{0\\le t\\le\\alpha\\beta}\\frac{1+\\gamma_{\\alpha,\\beta}(t)}{2t+1-\\gamma_{\\alpha,\\beta}(t)}=\\begin{cases}\n{\\displaystyle \\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}}} & \\text{if }\\beta\\ge{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\\\\\n{\\displaystyle \\frac{1-\\alpha\\beta}{\\beta(\\alpha-\\beta)}} & \\text{if }\\beta\\le{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}}.\n\\end{cases}\n\\]\n\\end{lemma}\n","start":68018,"end":68842}]},{"id":"S022","semantic_type":"definition","kind":"definition","labels":["eq:kappaub_prob"],"declared_external":false,"source_only":false,"title":"反例构造所用上界SDP","source_units":[{"raw_tex":"Finally, we give a constructive proof for the heuristic upper-bound\nin \\lemref{heuristic}. First, recall from \\lemref{sdp_formu} that\nour SDP upper-bound is defined as the following\n\\begin{equation}\n\\kappa_{\\ub}(X,Z)=\\min_{\\kappa,\\H}\\left\\{ \\kappa:I\\preceq\\H\\preceq\\kappa I,\\:\\J_{X}^{T}\\H\\e=0,\\;-2I_{r}\\otimes\\mat(\\H\\e)\\preceq\\J_{X}^{T}\\H\\J_{X}\\right\\} \\label{eq:kappaub_prob}\n\\end{equation}\nwhere $\\e=\\vect(XX^{T}-ZZ^{T})$ and its Jacobian $\\J_{X}$ is defined\nto satisfy $\\J_{X}\\vect(V)=\\vect(XV^{T}+VX^{T})$ for all $V$. Below,\nwe derive a set of sufficient conditions for a given $(\\kappa,\\H)$\nto be feasible for (\\ref{eq:kappaub_prob}). \n","start":69136,"end":69778}]},{"id":"S023","semantic_type":"definition","kind":"definition","labels":["exa:overparam"],"declared_external":false,"source_only":false,"title":"显式二次反例的基构造","source_units":[{"raw_tex":"\\begin{example}[$\\kappa=1+2\\sqrt{r/r^{\\star}}$ counterexample]\n\\label{exa:overparam}Given $r,r^{\\star},n$ satisfying $1\\le r^{\\star}\\le r<n$,\nlet $u_{1},u_{2},\\ldots,u_{n}$ be an orthonormal basis for $\\R^{n}$.\nDefine the quadratic function\n\\[\n\\phi_{0}(M)=\\frac{1}{2}\\sum_{i=1}^{n}\\sum_{j=1}^{n}\\left|\\langle A^{(i,j)},M-M^{\\star}\\rangle\\right|^{2}\\quad\\text{ where }M^{\\star}=\\sum_{k=r+1}^{r+r^{\\star}}u_{k}u_{k}^{T},\n\\]\nand $A^{(i,j)}=\\sqrt{\\kappa}\\cdot u_{i}u_{j}^{T}$ for all $1\\le i,j\\le n$\nexcept the following\n\\begin{gather*}\nA^{(1,1)}=\\sqrt{\\frac{\\kappa}{2r}}\\sum_{j=1}^{r}u_{j}u_{j}^{T}+\\sqrt{\\frac{\\kappa}{2r^{\\star}}}\\sum_{k=r+1}^{r+r^{\\star}}u_{k}u_{k}^{T},\\\\\nA^{(2,2)}=\\frac{1}{\\sqrt{2r}}\\sum_{j=1}^{r}u_{j}u_{j}^{T}-\\frac{1}{\\sqrt{2r^{\\star}}}\\sum_{k=r+1}^{r+r^{\\star}}u_{k}u_{k}^{T},\\\\\nA^{(i,i)}=u_{\\pi(i)}u_{\\pi(i)}^{T}-\\sum_{j=1}^{i-1}\\frac{A^{(j,j)}}{\\|A^{(j,j)}\\|_{F}^{2}}\\inner{A^{(j,j)}}{u_{\\pi(i)}u_{\\pi(i)}^{T}}\\text{ for }i\\in\\{3,\\dots,r+r^{\\star}\\},\n\\end{gather*}\nwith the ordering $\\pi=(1,r+1,2,3,\\dots,r,r+2,r+3,\\dots,r+r^{\\star})$.\n","start":73342,"end":74402}]},{"id":"S024","semantic_type":"derived_fact","kind":"derived_fact","labels":["exa:overparam"],"declared_external":false,"source_only":false,"title":"反例的谱界与正则性","source_units":[{"raw_tex":"Note that $\\{A^{(i,j)}\\}$ is an orthogonal basis for $\\R^{n\\times n}$\nwith $1\\le\\|A^{(i,j)}\\|_{F}^{2}\\le\\kappa$, so the quadratic function\n$\\phi_{0}$ is twice-differentiable, $1$-strongly convex and $\\kappa$-smooth,\n\\[\n\\|E\\|_{F}^{2}\\le\\inner{\\nabla^{2}\\phi_{0}(M)[E]}E\\le\\kappa\\|E\\|_{F}^{2}\\qquad\\text{for all }M\\in\\R^{n\\times n}.\n\\]\n","start":74402,"end":74736}]},{"id":"S025","semantic_type":"derived_fact","kind":"derived_fact","labels":["exa:overparam"],"declared_external":false,"source_only":false,"title":"伪二阶点及严格目标间隙","source_units":[{"raw_tex":"Its minimizer $M^{\\star}=\\arg\\min_{M}\\phi_{0}(M)$ satisfies $r^{\\star}=\\rank(M^{\\star}),$\nbut its corresponding $f_{0}(U)\\eqdef\\phi_{0}(UU^{T})$ admits the\n$n\\times r$ matrix $X=\\frac{1}{\\sqrt{1+\\sqrt{r/r^{\\star}}}}[u_{1},\\dots,u_{r}]$\nas a second-order critical point \n\\begin{align*}\nf_{0}(X)-\\min_{U}f_{0}(U) & =\\left(1+\\frac{1}{1+\\sqrt{r^{\\star}/r}}\\right)\\frac{\\|M^{\\star}\\|_{F}^{2}}{2}, & \\nabla f_{0}(X) & =0, & \\nabla^{2}f_{0}(X) & \\succeq0.\n\\end{align*}\n \\qed\n\\end{example}","start":74736,"end":75217}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"用新正交基实现特征条件","source_units":[{"raw_tex":"We now state an explicit counterexample that satisfies the conditions\nin \\lemref{counter}. Without loss of generality, take $u_{1},\\dots,u_{n}$\nto be the standard basis for $\\R^{n}$. The counterexample is constructed\nby taking the rescaled standard basis $A^{(i,j)}=\\sqrt{\\kappa}u_{i}u_{j}^{T}$\nfor $\\R^{n\\times n}$, and then replacing the elements $\\{A^{(i,i)}\\}_{i=1}^{n}$\nwith a new orthogonal basis for $\\diag(\\R^{n})$ whose first two elements\n$A^{(1,1)}$ and $A^{(2,2)}$ are exactly the first two eigenvectors\nin Lemma 6.1. The ordering $\\pi$ for the Gram--Schmidt orthogonalization\nbelow is chosen specifically to ensure that $\\mathrm{span}\\{A^{(i,i)}\\}_{i=1}^{n}=\\diag(\\R^{n})$. \n","start":72655,"end":73342}]},{"id":"S027","semantic_type":"theorem","kind":"theorem","labels":["thm:main"],"declared_external":false,"source_only":false,"title":"秩过参数化的尖锐阈值","source_units":[{"raw_tex":"\\begin{theorem}[Overparameterization]\n\\label{thm:main}Let $\\phi:\\S^{n}\\to\\R$ be twice-differentiable,\n$L$-smooth and $\\mu$-strongly convex, let the minimizer $M^{\\star}=\\arg\\min_{M\\succeq0}\\phi(M)$\nhave true rank $r^{\\star}=\\rank(M^{\\star})$.\n\\begin{itemize}\n\\item (Sufficiency) If $r>\\frac{1}{4}(L/\\mu-1)^{2}\\,r^{\\star}$ and $r^{\\star}\\le r<n$,\nthen the Burer--Monteiro function $f:\\R^{n\\times r}\\to\\R$ defined\nas $f(U)\\eqdef\\phi(UU^{T})$ has no spurious local minima:\n\\[\n\\nabla f(X)=0,\\quad\\nabla^{2}f(X)\\succeq0\\quad\\iff\\quad f(X)=\\min_{U}f(U).\n\\]\n\\item (Necessity) If $r\\le\\frac{1}{4}(L/\\mu-1)^{2}\\,r^{\\star}$ and $r^{\\star}\\le r<n$,\nthen there exists an $L$-smooth and $\\mu$-strongly convex quadratic\ncounterexample $\\phi_{0}$ whose minimizer $M_{0}^{\\star}=\\arg\\min_{M\\succeq0}\\phi_{0}(M)$\nhas $\\rank(M_{0}^{\\star})=r^{\\star}$, but whose $f_{0}(U)\\eqdef\\phi_{0}(UU^{T})$\nadmits an $n\\times r$ spurious second-order point $X_{\\spur}$:\n\\[\n\\nabla f_{0}(X_{\\spur})=0,\\quad\\nabla^{2}f_{0}(X_{\\spur})\\succeq0,\\quad f_{0}(X_{\\spur})-\\min_{U}f_{0}(U)>\\frac{\\mu}{2}\\cdot\\|M_{0}^{\\star}\\|_{F}^{2}.\n\\]\n\\end{itemize}\n\\end{theorem}","start":12631,"end":13755}]},{"id":"S028","semantic_type":"lemma","kind":"lemma","labels":["lem:sdp_formu"],"declared_external":false,"source_only":false,"title":"固定因子的SDP上下界","source_units":[{"raw_tex":"\\begin{lemma}[SDP formulation]\n\\label{lem:sdp_formu}Define $\\kappa(X,Z)$ as in (\\ref{eq:mu_reform}).\nWe have $\\kappa_{\\ub}(X,Z)\\ge\\kappa(X,Z)\\ge\\kappa_{\\lb}(X,Z)$ where\n\\begin{gather*}\n\\kappa_{\\ub}(X,Z)\\eqdef\\min_{\\kappa,\\H}\\left\\{ \\kappa:I\\preceq\\H\\preceq\\kappa I,\\;\\J_{X}^{T}\\H\\e=0,\\;-2I_{r}\\otimes\\mat(\\H\\e)\\preceq\\J_{X}^{T}\\H\\J_{X}\\right\\} \\\\\n\\kappa_{\\lb}(X,Z)\\eqdef\\min_{\\kappa,s,\\H}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H\\preceq\\kappa I,\\quad\\mat(s)\\succeq0,\\quad\\J_{Z}^{T}s=0,\\\\\n\\J_{X}^{T}(\\H\\e+s)=0,\\quad-2I_{r}\\otimes\\mat(\\H\\e+s)\\preceq\\kappa\\J_{X}^{T}\\J_{X}\n\\end{array}\\right\\} \n\\end{gather*}\nand $\\e=\\vect(XX^{T}-ZZ^{T})$ and $\\J_{U}\\vect(V)=\\vect(UV^{T}+VU^{T})$\nfor all $V$.\n\\end{lemma}","start":37973,"end":38675}]},{"id":"S029","semantic_type":"proof","kind":"proof","labels":["eq:cond","eq:kappa_sdp1","eq:kappasdp2","eq:kktZZ","eq:vecdf"],"declared_external":false,"source_only":false,"title":"SDP上下界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nDenote $S_{M}$ as the convex gradient $\\nabla\\phi(M)$ evaluated\nat $M$, and denote $\\H_{M}$ as the matrix representation of the\nconvex Hessian operator $\\nabla^{2}\\phi(M)$ evaluated at $M$. It\nfollows from $\\inner{\\nabla^{2}\\phi(M)[E]}E=\\vect(E)^{T}\\H_{M}\\vect(E)$\nthat: \n\\begin{equation}\n\\phi\\text{ is }1\\text{-strongly convex and }\\kappa\\text{-smooth}\\iff I\\preceq\\H_{M}\\preceq\\kappa I\\quad\\text{for all }M\\in\\S^{n}.\\label{eq:cond}\n\\end{equation}\nAdditionally, the Karush--Kuhn--Tucker conditions at $M^{\\star}=ZZ^{T}$\nread:\n\\begin{equation}\nZZ^{T}=\\arg\\min_{M\\succeq0}\\phi(M)\\quad\\iff\\quad S_{ZZ^{T}}\\succeq0,\\quad S_{ZZ^{T}}Z=0.\\label{eq:kktZZ}\n\\end{equation}\nNote that strong duality holds in the above because $ZZ^{T}+\\epsilon I$\nis a strictly feasible point with a bounded objective due to $\\kappa$-smoothness.\nNext, we observe that the two gradient evaluations $S_{XX^{T}}$ and\n$S_{ZZ^{T}}$ are related via the fundamental theorem of calculus:\n\\begin{gather*}\n\\nabla\\phi(XX^{T})=\\nabla\\phi(ZZ^{T})+\\int_{0}^{1}\\nabla^{2}\\phi((1-t)ZZ^{T}+tXX^{T})[XX^{T}-ZZ^{T}]\\,\\d t\\\\\n\\iff\\quad\\vect(S_{XX^{T}})=\\vect(S_{ZZ^{T}})+\\left[\\int_{0}^{1}\\H_{M(t)}\\,\\d t\\right]\\vect(XX^{T}-ZZ^{T}).\n\\end{gather*}\nDenoting $s=\\vect(S_{ZZ^{T}})$ and $\\H_{\\av}=\\int_{0}^{1}\\H_{M(t)}\\,\\d t$\nand $\\e=\\vect(XX^{T}-ZZ^{T})$ allows us to rewrite the directional\nderivatives of $f(U)\\eqdef\\phi(UU^{T})$ as follows: \n\\begin{align}\n\\inner{\\nabla f(X)}V & =\\vect(V)^{T}\\J_{X}^{T}(\\H_{\\av}\\e+s),\\nonumber \\\\\n\\inner{\\nabla^{2}f(X)[V]}V & =\\vect(V)^{T}[2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)+\\J_{X}^{T}\\H_{XX^{T}}\\J_{X}]\\,\\vect(V).\\label{eq:vecdf}\n\\end{align}\nSubstituting (\\ref{eq:cond}), (\\ref{eq:vecdf}), and (\\ref{eq:kktZZ})\ninto (\\ref{eq:mu_reform}) yields exactly\n\\begin{equation}\n\\kappa(X,Z)=\\inf_{\\kappa,\\phi:\\S^{n}\\to\\R}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H_{M}\\preceq\\kappa I\\quad\\text{for all }M\\in\\S^{n},\\\\\n\\mat(s)\\succeq0,\\quad\\J_{Z}^{T}s=0,\\quad\\J_{X}^{T}(\\H_{\\av}\\e+s)=0,\\\\\n-2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)\\preceq\\J_{X}^{T}\\H_{XX^{T}}\\J_{X}.\n\\end{array}\\right\\} \\label{eq:kappa_sdp1}\n\\end{equation}\nWe obtain the upper-bound $\\kappa_{\\ub}(X,Z)$ from $\\kappa(X,Z)$\nin  (\\ref{eq:kappa_sdp1}) by fixing $\\H_{M}\\equiv\\H$ for all $M$,\nand $S_{ZZ^{T}}\\equiv\\mat(s)=0$. We obtain the lower-bound $\\kappa_{\\lb}(X,Z)$\nfrom $\\kappa(X,Z)$ in (\\ref{eq:kappa_sdp1}) by substituting the\nrelaxation \n\\[\n-2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)\\quad\\preceq\\quad\\J_{X}^{T}\\H_{XX^{T}}\\J_{X}\\quad\\preceq\\quad\\kappa\\cdot\\J_{X}^{T}\\J_{X},\n\\]\nin order to avoid explicitly optimizing over $\\H_{XX^{T}}$:\n\\begin{align}\n\\kappa(X,Z) & \\ge\\inf_{\\kappa,\\phi:\\S^{n}\\to\\R}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H_{M}\\preceq\\kappa I\\quad\\text{for all }M\\in\\S^{n},\\\\\n\\mat(s)\\succeq0,\\quad\\J_{Z}^{T}s=0,\\quad\\J_{X}^{T}(\\H_{\\av}\\e+s)=0,\\\\\n-2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)\\preceq\\kappa\\cdot\\J_{X}^{T}\\J_{X}.\n\\end{array}\\right\\} \\label{eq:kappasdp2}\n\\end{align}\nFinally, we impose $I\\preceq\\H_{M}\\preceq\\kappa I$ over the average\nHessian $\\H_{\\av}=\\int_{0}^{1}\\H_{M(t)}\\,\\d t$ and then relax this\nconstraint over $\\H_{M}$ for all other values of $M$.\\qed\n\\end{proof}","start":38676,"end":41797}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":["eq:cond","eq:kktZZ"],"declared_external":false,"source_only":false,"title":"谱夹界与最优点KKT","source_units":[{"raw_tex":"\\begin{proof}\nDenote $S_{M}$ as the convex gradient $\\nabla\\phi(M)$ evaluated\nat $M$, and denote $\\H_{M}$ as the matrix representation of the\nconvex Hessian operator $\\nabla^{2}\\phi(M)$ evaluated at $M$. It\nfollows from $\\inner{\\nabla^{2}\\phi(M)[E]}E=\\vect(E)^{T}\\H_{M}\\vect(E)$\nthat: \n\\begin{equation}\n\\phi\\text{ is }1\\text{-strongly convex and }\\kappa\\text{-smooth}\\iff I\\preceq\\H_{M}\\preceq\\kappa I\\quad\\text{for all }M\\in\\S^{n}.\\label{eq:cond}\n\\end{equation}\nAdditionally, the Karush--Kuhn--Tucker conditions at $M^{\\star}=ZZ^{T}$\nread:\n\\begin{equation}\nZZ^{T}=\\arg\\min_{M\\succeq0}\\phi(M)\\quad\\iff\\quad S_{ZZ^{T}}\\succeq0,\\quad S_{ZZ^{T}}Z=0.\\label{eq:kktZZ}\n\\end{equation}\nNote that strong duality holds in the above because $ZZ^{T}+\\epsilon I$\nis a strictly feasible point with a bounded objective due to $\\kappa$-smoothness.\n","start":38676,"end":39508}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":["eq:vecdf"],"declared_external":false,"source_only":false,"title":"平均Hessian与因子导数","source_units":[{"raw_tex":"Next, we observe that the two gradient evaluations $S_{XX^{T}}$ and\n$S_{ZZ^{T}}$ are related via the fundamental theorem of calculus:\n\\begin{gather*}\n\\nabla\\phi(XX^{T})=\\nabla\\phi(ZZ^{T})+\\int_{0}^{1}\\nabla^{2}\\phi((1-t)ZZ^{T}+tXX^{T})[XX^{T}-ZZ^{T}]\\,\\d t\\\\\n\\iff\\quad\\vect(S_{XX^{T}})=\\vect(S_{ZZ^{T}})+\\left[\\int_{0}^{1}\\H_{M(t)}\\,\\d t\\right]\\vect(XX^{T}-ZZ^{T}).\n\\end{gather*}\nDenoting $s=\\vect(S_{ZZ^{T}})$ and $\\H_{\\av}=\\int_{0}^{1}\\H_{M(t)}\\,\\d t$\nand $\\e=\\vect(XX^{T}-ZZ^{T})$ allows us to rewrite the directional\nderivatives of $f(U)\\eqdef\\phi(UU^{T})$ as follows: \n\\begin{align}\n\\inner{\\nabla f(X)}V & =\\vect(V)^{T}\\J_{X}^{T}(\\H_{\\av}\\e+s),\\nonumber \\\\\n\\inner{\\nabla^{2}f(X)[V]}V & =\\vect(V)^{T}[2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)+\\J_{X}^{T}\\H_{XX^{T}}\\J_{X}]\\,\\vect(V).\\label{eq:vecdf}\n\\end{align}\n","start":39508,"end":40313}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":["eq:kappa_sdp1"],"declared_external":false,"source_only":false,"title":"反例条件的精确重写","source_units":[{"raw_tex":"Substituting (\\ref{eq:cond}), (\\ref{eq:vecdf}), and (\\ref{eq:kktZZ})\ninto (\\ref{eq:mu_reform}) yields exactly\n\\begin{equation}\n\\kappa(X,Z)=\\inf_{\\kappa,\\phi:\\S^{n}\\to\\R}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H_{M}\\preceq\\kappa I\\quad\\text{for all }M\\in\\S^{n},\\\\\n\\mat(s)\\succeq0,\\quad\\J_{Z}^{T}s=0,\\quad\\J_{X}^{T}(\\H_{\\av}\\e+s)=0,\\\\\n-2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)\\preceq\\J_{X}^{T}\\H_{XX^{T}}\\J_{X}.\n\\end{array}\\right\\} \\label{eq:kappa_sdp1}\n\\end{equation}\n","start":40313,"end":40772}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"限制为零最优梯度二次函数","source_units":[{"raw_tex":"We obtain the upper-bound $\\kappa_{\\ub}(X,Z)$ from $\\kappa(X,Z)$\nin  (\\ref{eq:kappa_sdp1}) by fixing $\\H_{M}\\equiv\\H$ for all $M$,\nand $S_{ZZ^{T}}\\equiv\\mat(s)=0$. We obtain the lower-bound $\\kappa_{\\lb}(X,Z)$\n","start":40772,"end":40982}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":["eq:kappasdp2"],"declared_external":false,"source_only":false,"title":"光滑性放松得到下界SDP","source_units":[{"raw_tex":"and $S_{ZZ^{T}}\\equiv\\mat(s)=0$. We obtain the lower-bound $\\kappa_{\\lb}(X,Z)$\nfrom $\\kappa(X,Z)$ in (\\ref{eq:kappa_sdp1}) by substituting the\nrelaxation \n\\[\n-2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)\\quad\\preceq\\quad\\J_{X}^{T}\\H_{XX^{T}}\\J_{X}\\quad\\preceq\\quad\\kappa\\cdot\\J_{X}^{T}\\J_{X},\n\\]\nin order to avoid explicitly optimizing over $\\H_{XX^{T}}$:\n\\begin{align}\n\\kappa(X,Z) & \\ge\\inf_{\\kappa,\\phi:\\S^{n}\\to\\R}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H_{M}\\preceq\\kappa I\\quad\\text{for all }M\\in\\S^{n},\\\\\n\\mat(s)\\succeq0,\\quad\\J_{Z}^{T}s=0,\\quad\\J_{X}^{T}(\\H_{\\av}\\e+s)=0,\\\\\n-2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)\\preceq\\kappa\\cdot\\J_{X}^{T}\\J_{X}.\n\\end{array}\\right\\} \\label{eq:kappasdp2}\n\\end{align}\nFinally, we impose $I\\preceq\\H_{M}\\preceq\\kappa I$ over the average\nHessian $\\H_{\\av}=\\int_{0}^{1}\\H_{M(t)}\\,\\d t$ and then relax this\nconstraint over $\\H_{M}$ for all other values of $M$.\\qed\n\\end{proof}","start":40903,"end":41797}]},{"id":"S035","semantic_type":"lemma","kind":"lemma","labels":["lem:heuristic"],"declared_external":false,"source_only":false,"title":"正交因子的启发式上界","source_units":[{"raw_tex":"\\begin{lemma}[Heuristic upper-bound]\n\\label{lem:heuristic}Let $[Q_{1},Q_{2}]$ have orthonormal columns\nwith $Q_{1}\\in\\R^{n\\times r}$ and $Q_{2}\\in\\R^{n\\times r^{\\star}}$.\nThen, we have\n\\[\n\\kappa_{\\ub}(X,Z)\\le1+2\\sqrt{r/r^{\\star}}\\quad\\text{where }X=Q_{1},\\quad Z=\\sqrt{1+\\sqrt{r/r^{\\star}}}Q_{2}.\n\\]\n\\end{lemma}","start":43032,"end":43343}]},{"id":"S036","semantic_type":"proof","kind":"proof","labels":["exa:overparam"],"declared_external":false,"source_only":true,"title":"启发式上界的构造性作者证明","source_units":[{"raw_tex":"We now state an explicit counterexample that satisfies the conditions\nin \\lemref{counter}. Without loss of generality, take $u_{1},\\dots,u_{n}$\nto be the standard basis for $\\R^{n}$. The counterexample is constructed\nby taking the rescaled standard basis $A^{(i,j)}=\\sqrt{\\kappa}u_{i}u_{j}^{T}$\nfor $\\R^{n\\times n}$, and then replacing the elements $\\{A^{(i,i)}\\}_{i=1}^{n}$\nwith a new orthogonal basis for $\\diag(\\R^{n})$ whose first two elements\n$A^{(1,1)}$ and $A^{(2,2)}$ are exactly the first two eigenvectors\nin Lemma 6.1. The ordering $\\pi$ for the Gram--Schmidt orthogonalization\nbelow is chosen specifically to ensure that $\\mathrm{span}\\{A^{(i,i)}\\}_{i=1}^{n}=\\diag(\\R^{n})$. \n\\begin{example}[$\\kappa=1+2\\sqrt{r/r^{\\star}}$ counterexample]\n\\label{exa:overparam}Given $r,r^{\\star},n$ satisfying $1\\le r^{\\star}\\le r<n$,\nlet $u_{1},u_{2},\\ldots,u_{n}$ be an orthonormal basis for $\\R^{n}$.\nDefine the quadratic function\n\\[\n\\phi_{0}(M)=\\frac{1}{2}\\sum_{i=1}^{n}\\sum_{j=1}^{n}\\left|\\langle A^{(i,j)},M-M^{\\star}\\rangle\\right|^{2}\\quad\\text{ where }M^{\\star}=\\sum_{k=r+1}^{r+r^{\\star}}u_{k}u_{k}^{T},\n\\]\nand $A^{(i,j)}=\\sqrt{\\kappa}\\cdot u_{i}u_{j}^{T}$ for all $1\\le i,j\\le n$\nexcept the following\n\\begin{gather*}\nA^{(1,1)}=\\sqrt{\\frac{\\kappa}{2r}}\\sum_{j=1}^{r}u_{j}u_{j}^{T}+\\sqrt{\\frac{\\kappa}{2r^{\\star}}}\\sum_{k=r+1}^{r+r^{\\star}}u_{k}u_{k}^{T},\\\\\nA^{(2,2)}=\\frac{1}{\\sqrt{2r}}\\sum_{j=1}^{r}u_{j}u_{j}^{T}-\\frac{1}{\\sqrt{2r^{\\star}}}\\sum_{k=r+1}^{r+r^{\\star}}u_{k}u_{k}^{T},\\\\\nA^{(i,i)}=u_{\\pi(i)}u_{\\pi(i)}^{T}-\\sum_{j=1}^{i-1}\\frac{A^{(j,j)}}{\\|A^{(j,j)}\\|_{F}^{2}}\\inner{A^{(j,j)}}{u_{\\pi(i)}u_{\\pi(i)}^{T}}\\text{ for }i\\in\\{3,\\dots,r+r^{\\star}\\},\n\\end{gather*}\nwith the ordering $\\pi=(1,r+1,2,3,\\dots,r,r+2,r+3,\\dots,r+r^{\\star})$.\nNote that $\\{A^{(i,j)}\\}$ is an orthogonal basis for $\\R^{n\\times n}$\nwith $1\\le\\|A^{(i,j)}\\|_{F}^{2}\\le\\kappa$, so the quadratic function\n$\\phi_{0}$ is twice-differentiable, $1$-strongly convex and $\\kappa$-smooth,\n\\[\n\\|E\\|_{F}^{2}\\le\\inner{\\nabla^{2}\\phi_{0}(M)[E]}E\\le\\kappa\\|E\\|_{F}^{2}\\qquad\\text{for all }M\\in\\R^{n\\times n}.\n\\]\nIts minimizer $M^{\\star}=\\arg\\min_{M}\\phi_{0}(M)$ satisfies $r^{\\star}=\\rank(M^{\\star}),$\nbut its corresponding $f_{0}(U)\\eqdef\\phi_{0}(UU^{T})$ admits the\n$n\\times r$ matrix $X=\\frac{1}{\\sqrt{1+\\sqrt{r/r^{\\star}}}}[u_{1},\\dots,u_{r}]$\nas a second-order critical point \n\\begin{align*}\nf_{0}(X)-\\min_{U}f_{0}(U) & =\\left(1+\\frac{1}{1+\\sqrt{r^{\\star}/r}}\\right)\\frac{\\|M^{\\star}\\|_{F}^{2}}{2}, & \\nabla f_{0}(X) & =0, & \\nabla^{2}f_{0}(X) & \\succeq0.\n\\end{align*}\n \\qed\n\\end{example}\n","start":72655,"end":75218}]},{"id":"S037","semantic_type":"lemma","kind":"lemma","labels":["lem:abdef"],"declared_external":false,"source_only":false,"title":"条件数下界的闭式表达","source_units":[{"raw_tex":"\\begin{lemma}[Closed-form lower-bound]\n\\label{lem:abdef}For $X,Z\\in\\R^{n\\times r}$ such that $XX^{T}\\ne ZZ^{T}$,\nlet $Z_{\\perp}=(I-XX^{\\dagger})Z$. We have\n\\begin{gather*}\n\\kappa_{\\lb}(X,Z)\\ge\\begin{cases}\n{\\displaystyle \\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}}} & \\text{if }\\beta\\ge{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\\\\\n{\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}} & \\text{if }\\beta\\le{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\n\\end{cases}\\\\\n\\text{ where }\\qquad\\alpha=\\frac{\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}{\\|XX^{T}-ZZ^{T}\\|_{F}},\\qquad\\beta=\\frac{\\lambda_{\\min}(X^{T}X)}{\\|XX^{T}-ZZ^{T}\\|_{F}}{\\displaystyle \\frac{\\tr(Z_{\\perp}Z_{\\perp}^{T})}{\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}}.\n\\end{gather*}\n\\end{lemma}","start":43659,"end":44416}]},{"id":"S038","semantic_type":"proof","kind":"proof","labels":["eq:Wsol","eq:Wsol1","eq:Wsol2","eq:e2","eq:ysol"],"declared_external":false,"source_only":true,"title":"闭式下界的完整散文证明","source_units":[{"raw_tex":"Our heuristic choice of $y$ and $W_{i,j}$ is motivated by the optimal\nsolution to the much simpler rank-1 case previously derived by \\citet{zhang2019sharp}.\nThis choice is not necessarily optimal for the rank-$r$ case, but\nit does remain simple enough for the final maximization over $t\\ge0$\nin \\lemref{tradeoff} to be solved in closed-form. \n\nWe begin by recalling the core insights used to derive the rank-1\nsolution. Our goal of maximizing $\\e^{T}(\\J_{X}y-w)$ where $w=\\sum_{i=1}^{r}\\vect(W_{i,i})$\ncan equivalently be understood as minimizing $\\|\\J_{X}y-w-\\e\\|$ over\n$y$ and $w$, subject to $\\mat(w)\\succeq0$. This is a projection\nproblem, so a basic idea is to decompose $\\e$ into orthogonal components:\n\\begin{equation}\n\\J_{X}y=\\gamma_{1}\\cdot(\\J_{X}\\J_{X}^{\\dagger})\\e,\\qquad w=-\\gamma_{2}\\cdot(I-\\J_{X}\\J_{X}^{\\dagger})\\e.\\label{eq:ysol}\n\\end{equation}\nWe verify that $\\mat(w)\\succeq0$ indeed holds for $\\gamma_{2}\\ge0$\nvia \\lemref{I-JJ+}\n\\begin{equation}\n\\mat((I-\\J_{X}\\J_{X}^{\\dagger})\\e)=(I-XX^{\\dagger})(XX^{T}-ZZ^{T})(I-XX^{\\dagger})=-Z_{\\perp}Z_{\\perp}^{T}\\label{eq:e2}\n\\end{equation}\nwhere we have written $Z_{\\perp}=(I-XX^{\\dagger})Z$. \n\nEquation (\\ref{eq:ysol}) specifies a unique choice of $y$, but leaves\nseveral possible valid choices of $W_{i,j}$. We need the following\nclaim to make a concrete choice.\n","start":61993,"end":63317},{"raw_tex":"Following the claim, the obvious first attempt is to pick $W=\\gamma_{2}\\cdot\\vect(Z_{\\perp})\\vect(Z_{\\perp})^{T}$.\nIndeed, this yields $\\sum_{i=1}^{r}W_{i,i}=\\gamma_{2}\\cdot Z_{\\perp}Z_{\\perp}^{T}$\nas specified in (\\ref{eq:ysol}), and is in fact optimal for the rank-1\ncase. In the rank-$r$ case, however, the choice of $W$ can be improved\nby making $\\inner{\\J_{X}^{T}\\J_{X}}W$ as small as possible. A more\nsubtle second attempt is to pick \n\\begin{equation}\nW=\\gamma_{2}\\cdot\\sum_{j=1}^{r}\\vect(Z_{\\perp}e_{j}v_{r}^{T})\\vect(Z_{\\perp}e_{j}v_{r}^{T})^{T}\\label{eq:Wsol}\n\\end{equation}\nwhere $v_{r}$ is the $r$-th unit eigenvector of $X^{T}X$. Indeed,\nwe can verify that (\\ref{eq:ysol}) is also satisfied:\n\\begin{gather}\n\\sum_{i=1}^{r}W_{i,i}=\\gamma_{2}\\cdot\\sum_{j=1}^{r}Z_{\\perp}e_{i}v_{r}^{T}v_{r}e_{i}^{T}Z_{\\perp}^{T}=\\gamma_{2}\\cdot Z_{\\perp}\\left(\\sum_{i=1}^{r}e_{i}e_{i}^{T}\\right)Z_{\\perp}^{T}=\\gamma_{2}\\cdot Z_{\\perp}Z_{\\perp}^{T}.\\label{eq:Wsol1}\n\\end{gather}\nMoreover, we have\n\\begin{align}\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle & =\\gamma_{2}\\cdot\\sum_{i=1}^{m}\\|\\J_{X}\\vect(Z_{\\perp}e_{i}v_{r}^{T})\\|^{2}=\\gamma_{2}\\cdot\\sum_{i=1}^{m}\\|Xv_{r}e_{i}^{T}Z_{\\perp}^{T}+Z_{\\perp}e_{i}v_{r}^{T}X^{T}\\|_{F}^{2}\\nonumber \\\\\n & =\\gamma_{2}\\cdot\\sum_{i=1}^{m}2\\|Z_{\\perp}e_{i}v_{r}^{T}X^{T}\\|_{F}^{2}=2\\gamma_{2}\\cdot\\lambda_{\\min}(X^{T}X)\\|Z_{\\perp}\\|_{F}^{2}\\label{eq:Wsol2}\n\\end{align}\nWe now plug our choice of $y$ and $W_{i,j}$ specified in (\\ref{eq:ysol})\nand (\\ref{eq:Wsol}) into \\lemref{tradeoff}, along with a good choice\nof coefficients $\\gamma_{1}$ and $\\gamma_{2}$.\n","start":63701,"end":65276},{"raw_tex":"The proof of \\lemref{abdef} immediately follows by substituting \\lemref{costheta}\ninto \\lemref{tradeoff} and then solving the resulting optimization\nover $t$ using \\lemref{optim}.\n","start":68842,"end":69022}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":["eq:e2","eq:ysol"],"declared_external":false,"source_only":true,"title":"按Jacobian像空间分解误差","source_units":[{"raw_tex":"Our heuristic choice of $y$ and $W_{i,j}$ is motivated by the optimal\nsolution to the much simpler rank-1 case previously derived by \\citet{zhang2019sharp}.\nThis choice is not necessarily optimal for the rank-$r$ case, but\nit does remain simple enough for the final maximization over $t\\ge0$\nin \\lemref{tradeoff} to be solved in closed-form. \n\nWe begin by recalling the core insights used to derive the rank-1\nsolution. Our goal of maximizing $\\e^{T}(\\J_{X}y-w)$ where $w=\\sum_{i=1}^{r}\\vect(W_{i,i})$\ncan equivalently be understood as minimizing $\\|\\J_{X}y-w-\\e\\|$ over\n$y$ and $w$, subject to $\\mat(w)\\succeq0$. This is a projection\nproblem, so a basic idea is to decompose $\\e$ into orthogonal components:\n\\begin{equation}\n\\J_{X}y=\\gamma_{1}\\cdot(\\J_{X}\\J_{X}^{\\dagger})\\e,\\qquad w=-\\gamma_{2}\\cdot(I-\\J_{X}\\J_{X}^{\\dagger})\\e.\\label{eq:ysol}\n\\end{equation}\nWe verify that $\\mat(w)\\succeq0$ indeed holds for $\\gamma_{2}\\ge0$\nvia \\lemref{I-JJ+}\n\\begin{equation}\n\\mat((I-\\J_{X}\\J_{X}^{\\dagger})\\e)=(I-XX^{\\dagger})(XX^{T}-ZZ^{T})(I-XX^{\\dagger})=-Z_{\\perp}Z_{\\perp}^{T}\\label{eq:e2}\n\\end{equation}\nwhere we have written $Z_{\\perp}=(I-XX^{\\dagger})Z$. \n\nEquation (\\ref{eq:ysol}) specifies a unique choice of $y$, but leaves\nseveral possible valid choices of $W_{i,j}$. We need the following\nclaim to make a concrete choice.\n","start":61993,"end":63317}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":["eq:Wsol"],"declared_external":false,"source_only":true,"title":"以最小特征向量降低证书代价","source_units":[{"raw_tex":"Following the claim, the obvious first attempt is to pick $W=\\gamma_{2}\\cdot\\vect(Z_{\\perp})\\vect(Z_{\\perp})^{T}$.\nIndeed, this yields $\\sum_{i=1}^{r}W_{i,i}=\\gamma_{2}\\cdot Z_{\\perp}Z_{\\perp}^{T}$\nas specified in (\\ref{eq:ysol}), and is in fact optimal for the rank-1\ncase. In the rank-$r$ case, however, the choice of $W$ can be improved\nby making $\\inner{\\J_{X}^{T}\\J_{X}}W$ as small as possible. A more\nsubtle second attempt is to pick \n\\begin{equation}\nW=\\gamma_{2}\\cdot\\sum_{j=1}^{r}\\vect(Z_{\\perp}e_{j}v_{r}^{T})\\vect(Z_{\\perp}e_{j}v_{r}^{T})^{T}\\label{eq:Wsol}\n\\end{equation}\nwhere $v_{r}$ is the $r$-th unit eigenvector of $X^{T}X$. Indeed,\nwe can verify that (\\ref{eq:ysol}) is also satisfied:\n","start":63701,"end":64405}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"代入对偶并完成标量最大化","source_units":[{"raw_tex":"We now plug our choice of $y$ and $W_{i,j}$ specified in (\\ref{eq:ysol})\nand (\\ref{eq:Wsol}) into \\lemref{tradeoff}, along with a good choice\nof coefficients $\\gamma_{1}$ and $\\gamma_{2}$.\n","start":65087,"end":65276},{"raw_tex":"The proof of \\lemref{abdef} immediately follows by substituting \\lemref{costheta}\ninto \\lemref{tradeoff} and then solving the resulting optimization\nover $t$ using \\lemref{optim}.\n","start":68842,"end":69022}]},{"id":"S042","semantic_type":"lemma","kind":"lemma","labels":["lem:ab1"],"declared_external":false,"source_only":false,"title":"捕获真实秩的有效不等式","source_units":[{"raw_tex":"\\begin{lemma}[Valid inequality]\n\\label{lem:ab1}For all $X,Z\\in\\R^{n\\times r}$ with $\\rank(Z)=r^{\\star}$,\nthe parameters $\\alpha$ and $\\beta$ defined in \\Lemref{abdef} satisfy\n$\\alpha^{2}+(r/r^{\\star})\\beta^{2}\\le1+[(\\beta-\\alpha)_{+}]^{2}.$\n\\end{lemma}","start":46119,"end":46371}]},{"id":"S043","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"由秩上界版本恢复有效不等式","source_units":[{"raw_tex":"Instead of proving \\lemref{ab1}, we will prove the following equivalent\nstatement.\n","start":56042,"end":56125},{"raw_tex":"Assuming the above to be true, setting $p=\\rank(Z)$ yields exactly\n\\lemref{ab1}. Conversely, if the above is true for $p=\\rank(Z)$,\nthen it is obviously true for any $p\\ge\\rank(Z)$, given that the\nleft-hand side decreases monotonically with increasing $p$.\n","start":56541,"end":56798}]},{"id":"S044","semantic_type":"proof","kind":"proof","labels":["eq:minab"],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n[Theorem~\\ref{thm:main}]For a fixed $r$ and $r^{\\star}$, recall\nthat we have defined $\\kappa^{\\star}$ as the condition number of\nthe \\emph{best-conditioned} counterexample, and formulated it as the\noptimal value for a two-stage optimization problem of the following\nform\n\\begin{align*}\n\\kappa^{\\star} & =\\inf_{\\phi:\\S^{n}\\to\\R}\\{\\kappa:\\phi\\text{ is }(\\kappa,r,r^{\\star})\\text{-counterexample}\\}\\\\\n & =\\inf_{X,Z\\in\\R^{n\\times r}}\\{\\kappa(X,Z):\\rank(Z)=r^{\\star},\\;XX^{T}\\ne ZZ^{T}\\}\n\\end{align*}\nWe claim that the minimum of the following expression over $\\alpha,\\beta\\ge0$\n\\[\n\\gamma(\\alpha,\\beta)\\eqdef\\begin{cases}\n\\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}} & \\text{if }\\beta\\ge{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\\\\\n{\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}} & \\text{if }\\beta\\le{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\n\\end{cases}\n\\]\nsubject to the valid inequality from \\lemref{ab1} is given:\n\\begin{equation}\n1+2\\sqrt{r/r^{\\star}}=\\min_{\\alpha,\\beta\\ge0}\\{\\gamma(\\alpha,\\beta):\\alpha^{2}+(r/r^{\\star})\\min\\{\\alpha^{2},\\beta^{2}\\}\\le1\\}.\\label{eq:minab}\n\\end{equation}\nTaking (\\ref{eq:minab}) to be true, substituting into \\lemref{abdef}\nand evoking \\lemref{heuristic} yields\n\\[\n1+2\\sqrt{r/r^{\\star}}\\le\\inf_{\\begin{subarray}{c}\nXX^{T}\\ne ZZ^{T}\\\\\n\\rank(Z)=r^{\\star}\n\\end{subarray}}\\kappa_{\\lb}(X,Z)\\le\\kappa^{\\star}\\le\\kappa_{\\ub}(X,Z)\\le1+2\\sqrt{r/r^{\\star}}.\n\\]\nThe upper-bound proves necessity: if $r/r^{\\star}\\le\\frac{1}{4}(\\kappa-1)^{2}$,\nthen there exists a $(\\kappa,r,r^{\\star})$-counterexample. The lower-bound\nproves sufficiency: if $r>\\frac{1}{4}(\\kappa-1)^{2}\\,r^{\\star}$ and\n$r^{\\star}\\le r<n$, then there exists no $(\\kappa,r,r^{\\star})$-counterexample\nto refute a ``no spurious local minima'' guarantee.\n\nWe now prove the claim in (\\ref{eq:minab}). We first minimize $\\gamma(\\alpha,\\beta)$\nsubject to $\\beta\\ge\\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}$ and find\nvia monotonicity with respect to $\\alpha$ that\n\\[\n\\min_{0\\le\\alpha\\le1}\\left\\{ \\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}}:\\beta\\ge{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}}\\right\\} ={\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}}.\n\\]\nIndeed, the minimizer $\\alpha$ lies at the boundary $\\beta=\\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}=\\frac{1-\\sqrt{1-\\alpha^{2}}}{\\alpha}$,\nwhere\n\\[\n\\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}}=\\frac{\\sqrt{1-\\alpha^{2}}+(1-\\alpha^{2})}{\\sqrt{1-\\alpha^{2}}-(1-\\alpha^{2})}=\\frac{1+\\sqrt{1-\\alpha^{2}}-\\alpha^{2}}{\\alpha^{2}-(1-\\sqrt{1-\\alpha^{2}})}={\\displaystyle \\frac{\\beta^{-1}-\\alpha}{\\alpha-\\beta}.}\n\\]\nNext, we minimize $\\gamma(\\alpha,\\beta)$ subject to $\\beta\\le\\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}\\le\\alpha$\nand the valid inequality:\n\\begin{align*}\n & \\min_{0\\le\\beta\\le\\alpha}\\left\\{ {\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}}:\\alpha^{2}+\\frac{r}{r^{\\star}}\\beta^{2}\\le1\\right\\} \\\\\n= & \\min_{\\beta\\ge0,\\;\\rho\\ge1}\\left\\{ {\\displaystyle \\frac{1-\\rho\\beta^{2}}{(\\rho-1)\\beta^{2}}}:(\\rho^{2}+\\frac{r}{r^{\\star}})\\beta^{2}\\le1\\right\\} =\\min_{\\rho\\ge1}\\left\\{ \\rho+\\frac{r}{r^{\\star}}\\frac{1}{\\rho-1}\\right\\} =1+2\\sqrt{\\frac{r}{r^{\\star}}}.\n\\end{align*}\nWe obtain the second line from the first by substituting $\\alpha=\\rho\\beta$\nfor $\\rho\\ge1$. The minimizers are $\\beta^{2}=(\\rho^{2}+\\frac{r}{r^{\\star}})^{-1}$\nand $\\rho=1+\\sqrt{r/r^{\\star}}$. \\qed\n\\end{proof}","start":46684,"end":50074}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"重述最优反例的两阶段形式","source_units":[{"raw_tex":"\\begin{proof}\n[Theorem~\\ref{thm:main}]For a fixed $r$ and $r^{\\star}$, recall\nthat we have defined $\\kappa^{\\star}$ as the condition number of\nthe \\emph{best-conditioned} counterexample, and formulated it as the\noptimal value for a two-stage optimization problem of the following\nform\n\\begin{align*}\n\\kappa^{\\star} & =\\inf_{\\phi:\\S^{n}\\to\\R}\\{\\kappa:\\phi\\text{ is }(\\kappa,r,r^{\\star})\\text{-counterexample}\\}\\\\\n & =\\inf_{X,Z\\in\\R^{n\\times r}}\\{\\kappa(X,Z):\\rank(Z)=r^{\\star},\\;XX^{T}\\ne ZZ^{T}\\}\n\\end{align*}\n","start":46684,"end":47194}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["eq:minab"],"declared_external":false,"source_only":false,"title":"标量松弛问题与最小值主张","source_units":[{"raw_tex":"We claim that the minimum of the following expression over $\\alpha,\\beta\\ge0$\n\\[\n\\gamma(\\alpha,\\beta)\\eqdef\\begin{cases}\n\\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}} & \\text{if }\\beta\\ge{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\\\\\n{\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}} & \\text{if }\\beta\\le{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\n\\end{cases}\n\\]\nsubject to the valid inequality from \\lemref{ab1} is given:\n\\begin{equation}\n1+2\\sqrt{r/r^{\\star}}=\\min_{\\alpha,\\beta\\ge0}\\{\\gamma(\\alpha,\\beta):\\alpha^{2}+(r/r^{\\star})\\min\\{\\alpha^{2},\\beta^{2}\\}\\le1\\}.\\label{eq:minab}\n\\end{equation}\n","start":47194,"end":47826}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"上下界夹逼并解释两方向","source_units":[{"raw_tex":"Taking (\\ref{eq:minab}) to be true, substituting into \\lemref{abdef}\nand evoking \\lemref{heuristic} yields\n\\[\n1+2\\sqrt{r/r^{\\star}}\\le\\inf_{\\begin{subarray}{c}\nXX^{T}\\ne ZZ^{T}\\\\\n\\rank(Z)=r^{\\star}\n\\end{subarray}}\\kappa_{\\lb}(X,Z)\\le\\kappa^{\\star}\\le\\kappa_{\\ub}(X,Z)\\le1+2\\sqrt{r/r^{\\star}}.\n\\]\nThe upper-bound proves necessity: if $r/r^{\\star}\\le\\frac{1}{4}(\\kappa-1)^{2}$,\nthen there exists a $(\\kappa,r,r^{\\star})$-counterexample. The lower-bound\nproves sufficiency: if $r>\\frac{1}{4}(\\kappa-1)^{2}\\,r^{\\star}$ and\n$r^{\\star}\\le r<n$, then there exists no $(\\kappa,r,r^{\\star})$-counterexample\nto refute a ``no spurious local minima'' guarantee.\n","start":47826,"end":48476}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"第一分支移至两段交界","source_units":[{"raw_tex":"\nWe now prove the claim in (\\ref{eq:minab}). We first minimize $\\gamma(\\alpha,\\beta)$\nsubject to $\\beta\\ge\\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}$ and find\nvia monotonicity with respect to $\\alpha$ that\n\\[\n\\min_{0\\le\\alpha\\le1}\\left\\{ \\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}}:\\beta\\ge{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}}\\right\\} ={\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}}.\n\\]\nIndeed, the minimizer $\\alpha$ lies at the boundary $\\beta=\\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}=\\frac{1-\\sqrt{1-\\alpha^{2}}}{\\alpha}$,\nwhere\n\\[\n\\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}}=\\frac{\\sqrt{1-\\alpha^{2}}+(1-\\alpha^{2})}{\\sqrt{1-\\alpha^{2}}-(1-\\alpha^{2})}=\\frac{1+\\sqrt{1-\\alpha^{2}}-\\alpha^{2}}{\\alpha^{2}-(1-\\sqrt{1-\\alpha^{2}})}={\\displaystyle \\frac{\\beta^{-1}-\\alpha}{\\alpha-\\beta}.}\n\\]\n","start":48476,"end":49301}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"第二分支代换并求最小值","source_units":[{"raw_tex":"Next, we minimize $\\gamma(\\alpha,\\beta)$ subject to $\\beta\\le\\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}\\le\\alpha$\nand the valid inequality:\n\\begin{align*}\n & \\min_{0\\le\\beta\\le\\alpha}\\left\\{ {\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}}:\\alpha^{2}+\\frac{r}{r^{\\star}}\\beta^{2}\\le1\\right\\} \\\\\n= & \\min_{\\beta\\ge0,\\;\\rho\\ge1}\\left\\{ {\\displaystyle \\frac{1-\\rho\\beta^{2}}{(\\rho-1)\\beta^{2}}}:(\\rho^{2}+\\frac{r}{r^{\\star}})\\beta^{2}\\le1\\right\\} =\\min_{\\rho\\ge1}\\left\\{ \\rho+\\frac{r}{r^{\\star}}\\frac{1}{\\rho-1}\\right\\} =1+2\\sqrt{\\frac{r}{r^{\\star}}}.\n\\end{align*}\nWe obtain the second line from the first by substituting $\\alpha=\\rho\\beta$\nfor $\\rho\\ge1$. The minimizers are $\\beta^{2}=(\\rho^{2}+\\frac{r}{r^{\\star}})^{-1}$\nand $\\rho=1+\\sqrt{r/r^{\\star}}$. \\qed\n\\end{proof}","start":49301,"end":50074}]},{"id":"S050","semantic_type":"theorem","kind":"theorem","labels":["eq:ey-sol","thm:EckartYoung"],"declared_external":false,"source_only":false,"title":"正则化Eckart–Young定理","source_units":[{"raw_tex":"\\begin{theorem}[Regularized Eckart--Young]\n\\label{thm:EckartYoung}Given $A\\in\\S_{+}^{n}$ and $B\\in\\S_{+}^{r}$\nwith $r\\le n$, let $A=\\sum_{i=1}^{n}s_{i}u_{i}u_{i}^{T}$ and $B=\\sum_{i=1}^{r}d_{i}v_{i}v_{i}^{T}$\ndenote the usual orthonormal eigendecompositions with $s_{1}\\ge\\cdots\\ge s_{n}\\ge0$\nand $0\\le d_{1}\\le\\cdots\\le d_{r}.$ Then, \n\\begin{align}\n\\min_{Y\\in\\R^{n\\times r}}\\left\\{ \\|A-YY^{T}\\|_{F}^{2}+2\\langle B,Y^{T}Y\\rangle\\right\\}  & =\\sum_{i=1}^{n}s_{i}^{2}-\\sum_{i=1}^{r}[(s_{i}-d_{i})_{+}]^{2}\\label{eq:ey-sol}\n\\end{align}\nwith minimizer $Y^{\\star}=\\sum_{i=1}^{r}u_{i}v_{i}^{T}\\sqrt{(s_{i}-d_{i})_{+}}$\nwhere $(\\cdot)_{+}\\eqdef\\max\\{0,\\cdot\\}$. \n\\end{theorem}","start":50630,"end":51298}]},{"id":"S051","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"正则化逼近的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n[\\thmref{EckartYoung}]Recall that $A=\\sum_{i=1}^{n}s_{i}u_{i}u_{i}^{T}$\nand $B=\\sum_{i=1}^{r}d_{i}v_{i}v_{i}^{T}$ with $s_{1}\\ge\\cdots\\ge s_{n}\\ge0$\nand $0\\le d_{1}\\le\\cdots\\le d_{r}.$ It follows from (\\ref{eq:Ruhe})\nthat\n\\begin{gather*}\n\\inner A{YY^{T}}\\le\\sum_{i=1}^{n}\\lambda_{i}(YY^{T})\\cdot\\lambda_{i}(A)=\\sum_{i=1}^{r}s_{i}\\sigma_{i}^{2}(Y),\\\\\n\\inner B{Y^{T}Y}\\ge\\sum_{i=1}^{r}\\lambda_{i}(Y^{T}Y)\\cdot\\lambda_{n-i+1}(B)=\\sum_{i=1}^{r}d_{i}\\sigma_{i}^{2}(Y).\n\\end{gather*}\nSubstituting yields the following lower-bound, which is valid for\nall choices of $Y$:\n\\begin{align*}\n\\|A-YY^{T}\\|_{F}^{2}+2\\inner B{Y^{T}Y}= & \\|A\\|_{F}^{2}+\\|YY^{T}\\|_{F}^{2}-2\\inner A{YY^{T}}+2\\inner B{Y^{T}Y}\\\\\n\\ge & \\sum_{i=1}^{n}s_{i}^{2}+\\sum_{i=1}^{r}\\sigma_{i}^{4}(Y)-2\\sum_{i=1}^{r}s_{i}\\sigma_{i}^{2}(Y)+2\\sum_{i=1}^{r}d_{i}\\sigma_{i}^{2}(Y)\\\\\n\\ge & \\sum_{i=1}^{n}s_{i}^{2}-\\sum_{i=1}^{r}[(s_{i}-d_{i})_{+}]^{2}.\n\\end{align*}\nThe final line applied the following result as a lower-bound:\n\\begin{align*}\n\\sigma_{i}^{4}(Y)-2(s_{i}-d_{i})\\sigma_{i}^{2}(Y)\\ge & \\min_{x_{i}\\ge0}x_{i}^{2}-2(s_{i}-d_{i})x_{i}\\\\\n= & \\min_{x_{i}\\ge0}[x_{i}-(s_{i}-d_{i})]^{2}-(s_{i}-d_{i})^{2}\\\\\n= & [(s_{i}-d_{i})_{+}-(s_{i}-d_{i})]^{2}-(s_{i}-d_{i})^{2}=-[(s_{i}-d_{i})_{+}]^{2}.\n\\end{align*}\nFinally, we verify that $Y^{\\star}=\\sum_{i=1}^{n}u_{i}v_{i}\\sqrt{(s_{i}-d_{i})_{+}}$\nattains the lower-bound derived above. \\qed\n\\end{proof}","start":52339,"end":53752}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"以Ruhe不等式配对谱值","source_units":[{"raw_tex":"\\begin{proof}\n[\\thmref{EckartYoung}]Recall that $A=\\sum_{i=1}^{n}s_{i}u_{i}u_{i}^{T}$\nand $B=\\sum_{i=1}^{r}d_{i}v_{i}v_{i}^{T}$ with $s_{1}\\ge\\cdots\\ge s_{n}\\ge0$\nand $0\\le d_{1}\\le\\cdots\\le d_{r}.$ It follows from (\\ref{eq:Ruhe})\nthat\n\\begin{gather*}\n\\inner A{YY^{T}}\\le\\sum_{i=1}^{n}\\lambda_{i}(YY^{T})\\cdot\\lambda_{i}(A)=\\sum_{i=1}^{r}s_{i}\\sigma_{i}^{2}(Y),\\\\\n\\inner B{Y^{T}Y}\\ge\\sum_{i=1}^{r}\\lambda_{i}(Y^{T}Y)\\cdot\\lambda_{n-i+1}(B)=\\sum_{i=1}^{r}d_{i}\\sigma_{i}^{2}(Y).\n\\end{gather*}\n","start":52339,"end":52831}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"逐项非负二次最小化","source_units":[{"raw_tex":"Substituting yields the following lower-bound, which is valid for\nall choices of $Y$:\n\\begin{align*}\n\\|A-YY^{T}\\|_{F}^{2}+2\\inner B{Y^{T}Y}= & \\|A\\|_{F}^{2}+\\|YY^{T}\\|_{F}^{2}-2\\inner A{YY^{T}}+2\\inner B{Y^{T}Y}\\\\\n\\ge & \\sum_{i=1}^{n}s_{i}^{2}+\\sum_{i=1}^{r}\\sigma_{i}^{4}(Y)-2\\sum_{i=1}^{r}s_{i}\\sigma_{i}^{2}(Y)+2\\sum_{i=1}^{r}d_{i}\\sigma_{i}^{2}(Y)\\\\\n\\ge & \\sum_{i=1}^{n}s_{i}^{2}-\\sum_{i=1}^{r}[(s_{i}-d_{i})_{+}]^{2}.\n\\end{align*}\nThe final line applied the following result as a lower-bound:\n\\begin{align*}\n\\sigma_{i}^{4}(Y)-2(s_{i}-d_{i})\\sigma_{i}^{2}(Y)\\ge & \\min_{x_{i}\\ge0}x_{i}^{2}-2(s_{i}-d_{i})x_{i}\\\\\n= & \\min_{x_{i}\\ge0}[x_{i}-(s_{i}-d_{i})]^{2}-(s_{i}-d_{i})^{2}\\\\\n= & [(s_{i}-d_{i})_{+}-(s_{i}-d_{i})]^{2}-(s_{i}-d_{i})^{2}=-[(s_{i}-d_{i})_{+}]^{2}.\n\\end{align*}\n","start":52831,"end":53612}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"声明候选达到下界","source_units":[{"raw_tex":"Finally, we verify that $Y^{\\star}=\\sum_{i=1}^{n}u_{i}v_{i}\\sqrt{(s_{i}-d_{i})_{+}}$\nattains the lower-bound derived above. \\qed\n\\end{proof}","start":53612,"end":53752}]},{"id":"S055","semantic_type":"lemma","kind":"lemma","labels":["lem:numcard"],"declared_external":false,"source_only":false,"title":"正部残差的投影下界","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:numcard}If $x\\ge0$ and $\\one^{T}x\\le\\|x\\|^{2}$, then\n$\\one^{T}\\left(I-xx^{T}/\\|x\\|^{2}\\right)\\one\\ge\\|(\\one-x)_{+}\\|^{2}$.\n\\end{lemma}","start":53893,"end":54052}]},{"id":"S056","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"投影下界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nDefine $u=\\alpha x,$ where $\\alpha=\\one^{T}x/\\|x\\|^{2}$ is chosen\nso that $\\mathbf{1}^{T}u=\\|u\\|^{2}$. First, verify that $\\mathbf{1}^{T}(I-xx^{T}/\\|x\\|^{2})\\mathbf{1}=\\|\\mathbf{1}-u\\|^{2}\\ge\\|(\\mathbf{1}-u)_{+}\\|^{2}$.\nNext, observe that $\\psi(\\alpha)\\equiv[(1-\\alpha t)_{+}]^{2}$ is\na decreasing function of $\\alpha\\ge0$ when $t\\ge0$, and therefore\n$\\|(\\one-\\alpha x)_{+}\\|^{2}\\ge\\|(\\one-x)_{+}\\|^{2}$ for $x\\ge0$\nand $0\\le\\alpha\\le1$. Finally, the hypotheses $x\\ge0$ and $\\one^{T}x\\le\\|x\\|^{2}$\nensure that $0\\le\\alpha\\le1$. \\qed\n\\end{proof}","start":54053,"end":54611}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"缩放至投影恒等式","source_units":[{"raw_tex":"\\begin{proof}\nDefine $u=\\alpha x,$ where $\\alpha=\\one^{T}x/\\|x\\|^{2}$ is chosen\nso that $\\mathbf{1}^{T}u=\\|u\\|^{2}$. First, verify that $\\mathbf{1}^{T}(I-xx^{T}/\\|x\\|^{2})\\mathbf{1}=\\|\\mathbf{1}-u\\|^{2}\\ge\\|(\\mathbf{1}-u)_{+}\\|^{2}$.\n","start":54053,"end":54287}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"正部残差的缩放单调性","source_units":[{"raw_tex":"Next, observe that $\\psi(\\alpha)\\equiv[(1-\\alpha t)_{+}]^{2}$ is\na decreasing function of $\\alpha\\ge0$ when $t\\ge0$, and therefore\n$\\|(\\one-\\alpha x)_{+}\\|^{2}\\ge\\|(\\one-x)_{+}\\|^{2}$ for $x\\ge0$\nand $0\\le\\alpha\\le1$. Finally, the hypotheses $x\\ge0$ and $\\one^{T}x\\le\\|x\\|^{2}$\nensure that $0\\le\\alpha\\le1$. \\qed\n\\end{proof}","start":54287,"end":54611}]},{"id":"S059","semantic_type":"lemma","kind":"lemma","labels":["lem:keyclaim"],"declared_external":false,"source_only":false,"title":"截断平方差的两段下界","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:keyclaim}Given $s,d\\in\\R_{+}^{n}$, let $s_{i}\\ge s_{\\lb}\\ge0$\nfor all $i$. Then we have\n\\begin{align*}\n\\|s\\|^{2}-\\|(s-d)_{+}\\|^{2}\\ge & \\begin{cases}\ns_{\\lb}^{2}\\frac{(\\one^{T}d)^{2}}{\\|d\\|^{2}} & \\text{if }s_{\\lb}\\one^{T}d\\le\\|d\\|^{2},\\\\\n2s_{\\lb}\\one^{T}d-\\|d\\|^{2} & \\text{if }s_{\\lb}\\one^{T}d\\ge\\|d\\|^{2}.\n\\end{cases}\n\\end{align*}\n\\end{lemma}","start":54613,"end":54983}]},{"id":"S060","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"截断平方差的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nFirst, we observe that $\\psi(t)\\equiv t^{2}-[(t-\\alpha)_{+}]^{2}$\nis an increasing function of $t\\ge0$. It follows from element-wise\nmonotonicity that\n\\[\n\\|s\\|^{2}-\\|(s-d)_{+}\\|^{2}\\ge\\|s_{\\lb}\\one\\|^{2}-\\|(s_{\\lb}\\one-d)_{+}\\|^{2}=s_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-d/s_{\\lb})_{+}\\|^{2}\\right)\n\\]\nIf $s_{\\lb}\\one^{T}d\\le\\|d\\|^{2}$, then write $x=d/s_{\\lb}\\ge0$,\nand observe that $\\one^{T}x\\le\\|x\\|^{2}$ holds. Therefore, applying\n\\lemref{numcard} yields\n\\begin{align*}\ns_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-d/s_{\\lb})_{+}\\|^{2}\\right) & =s_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-x)_{+}\\|^{2}\\right)\\ge s_{\\lb}^{2}\\one^{T}\\left(xx^{T}/\\|x\\|^{2}\\right)\\one\\\\\n & =s_{\\lb}^{2}\\left([\\one^{T}(d/s_{\\lb})]^{2}/\\|d/s_{\\lb}\\|^{2}\\right)=s_{\\lb}^{2}(\\one^{T}d)^{2}/\\|d\\|^{2}.\n\\end{align*}\nIf otherwise $s_{\\lb}\\one^{T}d\\ge\\|d\\|^{2}$, then we have \n\\begin{align*}\ns_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-d/s_{\\lb})_{+}\\|^{2}\\right)\\ge s_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|\\one-d/s_{\\lb}\\|^{2}\\right)=2s_{\\lb}\\one^{T}d-\\|d\\|^{2}.\n\\end{align*}\n\\qed\n\\end{proof}","start":54984,"end":56040}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"以逐项单调性降至常数向量","source_units":[{"raw_tex":"\\begin{proof}\nFirst, we observe that $\\psi(t)\\equiv t^{2}-[(t-\\alpha)_{+}]^{2}$\nis an increasing function of $t\\ge0$. It follows from element-wise\nmonotonicity that\n\\[\n\\|s\\|^{2}-\\|(s-d)_{+}\\|^{2}\\ge\\|s_{\\lb}\\one\\|^{2}-\\|(s_{\\lb}\\one-d)_{+}\\|^{2}=s_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-d/s_{\\lb})_{+}\\|^{2}\\right)\n\\]\n","start":54984,"end":55299}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"第一分支应用投影引理","source_units":[{"raw_tex":"If $s_{\\lb}\\one^{T}d\\le\\|d\\|^{2}$, then write $x=d/s_{\\lb}\\ge0$,\nand observe that $\\one^{T}x\\le\\|x\\|^{2}$ holds. Therefore, applying\n\\lemref{numcard} yields\n\\begin{align*}\ns_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-d/s_{\\lb})_{+}\\|^{2}\\right) & =s_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-x)_{+}\\|^{2}\\right)\\ge s_{\\lb}^{2}\\one^{T}\\left(xx^{T}/\\|x\\|^{2}\\right)\\one\\\\\n & =s_{\\lb}^{2}\\left([\\one^{T}(d/s_{\\lb})]^{2}/\\|d/s_{\\lb}\\|^{2}\\right)=s_{\\lb}^{2}(\\one^{T}d)^{2}/\\|d\\|^{2}.\n\\end{align*}\n","start":55299,"end":55779}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"第二分支去掉正部","source_units":[{"raw_tex":"If otherwise $s_{\\lb}\\one^{T}d\\ge\\|d\\|^{2}$, then we have \n\\begin{align*}\ns_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-d/s_{\\lb})_{+}\\|^{2}\\right)\\ge s_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|\\one-d/s_{\\lb}\\|^{2}\\right)=2s_{\\lb}\\one^{T}d-\\|d\\|^{2}.\n\\end{align*}\n\\qed\n\\end{proof}","start":55779,"end":56040}]},{"id":"S064","semantic_type":"lemma","kind":"lemma","labels":["lem:ab2"],"declared_external":false,"source_only":false,"title":"真实秩上界版本的有效不等式","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:ab2}For $X,Z\\in\\R^{n\\times r}$, define the parameters\n\\[\n\\alpha=\\frac{\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}{\\|XX^{T}-ZZ^{T}\\|_{F}},\\qquad\\beta=\\frac{\\lambda_{\\min}(X^{T}X)}{\\|XX^{T}-ZZ^{T}\\|_{F}}\\cdot\\frac{\\tr(Z_{\\perp}Z_{\\perp}^{T})}{\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}.\n\\]\nThen, $\\alpha^{2}+(r/p)\\beta^{2}\\le1+[(\\beta-\\alpha)_{+}]^{2}$ holds\nfor all $p$ satisfying $\\rank(Z)\\le p\\le r$.\n\\end{lemma}","start":56125,"end":56540}]},{"id":"S065","semantic_type":"proof","kind":"proof","labels":["eq:key-claim","eq:key-claim1","eq:key-claim2","eq:key-claim3","eq:key-claim4"],"declared_external":false,"source_only":false,"title":"有效不等式的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nThe two parameters $\\alpha$ and $\\beta$ are unitarily invariant\nwith respect to $X,Z$, so we may assume without loss of generality\nthat $X,Z$ are partitioned as\n\\[\nX=\\begin{bmatrix}X_{1}\\\\\n0\n\\end{bmatrix}\\qquad Z=\\begin{bmatrix}Z_{1}\\\\\nZ_{2}\n\\end{bmatrix}\\qquad\\text{where }X_{1}\\in\\R^{r\\times r},\\;Z_{1}\\in\\R^{r\\times p},\\;Z_{2}\\in\\R^{(n-r)\\times p}.\n\\]\n(Otherwise, take the QR decomposition $X=QR$ and note that $\\alpha,\\beta$\nremain unchanged with $X\\gets Q^{T}X$ and $Z\\gets Q^{T}Z$.) Denote\n$s\\in\\R^{r}$ as the $r$ eigenvalues of $X_{1}X_{1}^{T}$ and $d\\in\\R^{p}$\nas the $p$ eigenvalues of $Z_{2}^{T}Z_{2}$ \n\\begin{align*}\ns_{i}\\eqdef & \\lambda_{i}(X_{1}X_{1}^{T})=\\lambda_{i}(XX^{T}),\\quad s_{1}\\ge s_{2}\\ge\\cdots\\ge s_{r}\\ge0,\\\\\nd_{i}\\eqdef & \\lambda_{i}(Z_{2}^{T}Z_{2})=\\lambda_{i}(Z_{\\perp}Z_{\\perp}^{T}),\\quad0\\le d_{1}\\le d_{2}\\le\\cdots\\le d_{p}.\n\\end{align*}\nWriting $\\e=\\vect(XX^{T}-ZZ^{T})$ yields\n\\begin{align*}\n\\alpha & =\\frac{\\|Z_{2}Z_{2}^{T}\\|_{F}}{\\|XX^{T}-ZZ^{T}\\|_{F}}=\\frac{\\|d\\|_{F}}{\\|\\e\\|_{F}}, & \\beta & =\\frac{\\lambda_{\\min}(X_{1}^{T}X_{1})}{\\|XX^{T}-ZZ^{T}\\|_{F}}\\cdot\\frac{\\tr(Z_{2}Z_{2}^{T})}{\\|Z_{2}Z_{2}^{T}\\|_{F}}=\\frac{s_{r}\\cdot\\one^{T}d}{\\|\\e\\|\\|d\\|}.\n\\end{align*}\nWe expand $\\|XX^{T}-ZZ^{T}\\|_{F}^{2}$ block-wise and evoke our Regularized\nEckart--Young Theorem (\\thmref{EckartYoung}) over the $r\\times p$\nmatrix block $Z_{1}$:\n\\begin{align*}\n\\|XX^{T}-ZZ^{T}\\|_{F}^{2} & =\\|X_{1}X_{1}^{T}-Z_{1}Z_{1}^{T}\\|_{F}^{2}+2\\langle Z_{2}^{T}Z_{2},Z_{1}^{T}Z_{1}\\rangle+\\|Z_{2}Z_{2}^{T}\\|^{2}\\\\\n & \\ge\\min_{Z_{1}\\in\\R^{r\\times p}}\\{\\|X_{1}X_{1}^{T}-Z_{1}Z_{1}^{T}\\|_{F}^{2}+2\\langle Z_{2}^{T}Z_{2},Z_{1}^{T}Z_{1}\\rangle\\}+\\|Z_{2}Z_{2}^{T}\\|^{2}\\\\\n & =\\|s\\|^{2}-\\|(s'-d)_{+}\\|^{2}+\\|d\\|^{2}\n\\end{align*}\nwhere $s'=(s_{1},s_{2},\\dots,s_{p})$. Now, evoking our technical\nlemma (\\lemref{keyclaim}) with $s_{\\lb}\\equiv s_{r}$ yields the\nfollowing two lower-bounds on $\\|XX^{T}-ZZ^{T}\\|_{F}^{2}=\\|\\e\\|^{2}$:\\begin{subequations}\\label{eq:key-claim}\n\\begin{gather}\ns_{r}\\one^{T}d\\le\\|d\\|^{2}\\quad\\implies\\quad\\|\\e\\|^{2}\\ge s_{r}^{2}\\frac{(\\one^{T}d)^{2}}{\\|d\\|^{2}}+(r-p)s_{r}^{2}+\\|d\\|^{2},\\label{eq:key-claim1}\\\\\ns_{r}\\one^{T}d\\ge\\|d\\|^{2}\\quad\\implies\\quad\\|\\e\\|^{2}\\ge2s_{r}\\one^{T}d-\\|d\\|^{2}+(r-p)s_{r}^{2}+\\|d\\|^{2}.\\label{eq:key-claim2}\n\\end{gather}\n\\end{subequations}Substituting $s_{r}=\\beta\\cdot\\|\\e\\|\\|d\\|/(\\one^{T}d)$\nyields\n\\[\n(r-p)s_{r}^{2}=(r-p)\\frac{\\|d\\|^{2}}{(\\one^{T}d)^{2}}\\beta^{2}\\|\\e\\|^{2}\\ge(r-p)\\frac{1}{\\nnz(d)}\\beta^{2}\\|\\e\\|^{2}\\ge\\frac{(r-p)}{p}\\beta^{2}\\|\\e\\|^{2},\n\\]\nand substituting into (\\ref{eq:key-claim}) yields\\begin{subequations}\n\\begin{gather}\n\\beta\\le\\alpha\\quad\\implies\\quad1\\ge\\beta^{2}+\\alpha^{2}+\\frac{(r-p)}{p}\\beta^{2}=\\alpha^{2}+\\frac{r}{p}\\beta^{2},\\label{eq:key-claim3}\\\\\n\\beta\\ge\\alpha\\quad\\implies\\quad1\\ge2\\alpha\\beta+\\frac{(r-p)}{p}\\beta^{2}=\\alpha^{2}+\\frac{r}{p}\\beta^{2}-(\\beta-\\alpha)^{2},\\label{eq:key-claim4}\n\\end{gather}\n\\end{subequations}as desired. \\qed\n\\end{proof}","start":56798,"end":59758}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":["eq:key-claim"],"declared_external":false,"source_only":false,"title":"酉不变分块与谱参数","source_units":[{"raw_tex":"\\begin{proof}\nThe two parameters $\\alpha$ and $\\beta$ are unitarily invariant\nwith respect to $X,Z$, so we may assume without loss of generality\nthat $X,Z$ are partitioned as\n\\[\nX=\\begin{bmatrix}X_{1}\\\\\n0\n\\end{bmatrix}\\qquad Z=\\begin{bmatrix}Z_{1}\\\\\nZ_{2}\n\\end{bmatrix}\\qquad\\text{where }X_{1}\\in\\R^{r\\times r},\\;Z_{1}\\in\\R^{r\\times p},\\;Z_{2}\\in\\R^{(n-r)\\times p}.\n\\]\n(Otherwise, take the QR decomposition $X=QR$ and note that $\\alpha,\\beta$\nremain unchanged with $X\\gets Q^{T}X$ and $Z\\gets Q^{T}Z$.) Denote\n$s\\in\\R^{r}$ as the $r$ eigenvalues of $X_{1}X_{1}^{T}$ and $d\\in\\R^{p}$\nas the $p$ eigenvalues of $Z_{2}^{T}Z_{2}$ \n\\begin{align*}\ns_{i}\\eqdef & \\lambda_{i}(X_{1}X_{1}^{T})=\\lambda_{i}(XX^{T}),\\quad s_{1}\\ge s_{2}\\ge\\cdots\\ge s_{r}\\ge0,\\\\\nd_{i}\\eqdef & \\lambda_{i}(Z_{2}^{T}Z_{2})=\\lambda_{i}(Z_{\\perp}Z_{\\perp}^{T}),\\quad0\\le d_{1}\\le d_{2}\\le\\cdots\\le d_{p}.\n\\end{align*}\nWriting $\\e=\\vect(XX^{T}-ZZ^{T})$ yields\n\\begin{align*}\n\\alpha & =\\frac{\\|Z_{2}Z_{2}^{T}\\|_{F}}{\\|XX^{T}-ZZ^{T}\\|_{F}}=\\frac{\\|d\\|_{F}}{\\|\\e\\|_{F}}, & \\beta & =\\frac{\\lambda_{\\min}(X_{1}^{T}X_{1})}{\\|XX^{T}-ZZ^{T}\\|_{F}}\\cdot\\frac{\\tr(Z_{2}Z_{2}^{T})}{\\|Z_{2}Z_{2}^{T}\\|_{F}}=\\frac{s_{r}\\cdot\\one^{T}d}{\\|\\e\\|\\|d\\|}.\n\\end{align*}\n","start":56798,"end":58013}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":["eq:key-claim"],"declared_external":false,"source_only":false,"title":"分块误差应用正则化逼近","source_units":[{"raw_tex":"We expand $\\|XX^{T}-ZZ^{T}\\|_{F}^{2}$ block-wise and evoke our Regularized\nEckart--Young Theorem (\\thmref{EckartYoung}) over the $r\\times p$\nmatrix block $Z_{1}$:\n\\begin{align*}\n\\|XX^{T}-ZZ^{T}\\|_{F}^{2} & =\\|X_{1}X_{1}^{T}-Z_{1}Z_{1}^{T}\\|_{F}^{2}+2\\langle Z_{2}^{T}Z_{2},Z_{1}^{T}Z_{1}\\rangle+\\|Z_{2}Z_{2}^{T}\\|^{2}\\\\\n & \\ge\\min_{Z_{1}\\in\\R^{r\\times p}}\\{\\|X_{1}X_{1}^{T}-Z_{1}Z_{1}^{T}\\|_{F}^{2}+2\\langle Z_{2}^{T}Z_{2},Z_{1}^{T}Z_{1}\\rangle\\}+\\|Z_{2}Z_{2}^{T}\\|^{2}\\\\\n & =\\|s\\|^{2}-\\|(s'-d)_{+}\\|^{2}+\\|d\\|^{2}\n\\end{align*}\n","start":58013,"end":58541}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":["eq:key-claim","eq:key-claim1","eq:key-claim2"],"declared_external":false,"source_only":false,"title":"按最小特征值取得两段误差界","source_units":[{"raw_tex":"where $s'=(s_{1},s_{2},\\dots,s_{p})$. Now, evoking our technical\nlemma (\\lemref{keyclaim}) with $s_{\\lb}\\equiv s_{r}$ yields the\nfollowing two lower-bounds on $\\|XX^{T}-ZZ^{T}\\|_{F}^{2}=\\|\\e\\|^{2}$:\\begin{subequations}\\label{eq:key-claim}\n\\begin{gather}\ns_{r}\\one^{T}d\\le\\|d\\|^{2}\\quad\\implies\\quad\\|\\e\\|^{2}\\ge s_{r}^{2}\\frac{(\\one^{T}d)^{2}}{\\|d\\|^{2}}+(r-p)s_{r}^{2}+\\|d\\|^{2},\\label{eq:key-claim1}\\\\\ns_{r}\\one^{T}d\\ge\\|d\\|^{2}\\quad\\implies\\quad\\|\\e\\|^{2}\\ge2s_{r}\\one^{T}d-\\|d\\|^{2}+(r-p)s_{r}^{2}+\\|d\\|^{2}.\\label{eq:key-claim2}\n\\end{gather}\n","start":58541,"end":59088}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":["eq:key-claim","eq:key-claim3","eq:key-claim4"],"declared_external":false,"source_only":false,"title":"秩计数与归一化得到有效域","source_units":[{"raw_tex":"\\end{subequations}Substituting $s_{r}=\\beta\\cdot\\|\\e\\|\\|d\\|/(\\one^{T}d)$\nyields\n\\[\n(r-p)s_{r}^{2}=(r-p)\\frac{\\|d\\|^{2}}{(\\one^{T}d)^{2}}\\beta^{2}\\|\\e\\|^{2}\\ge(r-p)\\frac{1}{\\nnz(d)}\\beta^{2}\\|\\e\\|^{2}\\ge\\frac{(r-p)}{p}\\beta^{2}\\|\\e\\|^{2},\n\\]\nand substituting into (\\ref{eq:key-claim}) yields\\begin{subequations}\n\\begin{gather}\n\\beta\\le\\alpha\\quad\\implies\\quad1\\ge\\beta^{2}+\\alpha^{2}+\\frac{(r-p)}{p}\\beta^{2}=\\alpha^{2}+\\frac{r}{p}\\beta^{2},\\label{eq:key-claim3}\\\\\n\\beta\\ge\\alpha\\quad\\implies\\quad1\\ge2\\alpha\\beta+\\frac{(r-p)}{p}\\beta^{2}=\\alpha^{2}+\\frac{r}{p}\\beta^{2}-(\\beta-\\alpha)^{2},\\label{eq:key-claim4}\n\\end{gather}\n\\end{subequations}as desired. \\qed\n\\end{proof}","start":59088,"end":59758}]},{"id":"S070","semantic_type":"lemma","kind":"lemma","labels":["lem:I-JJ+"],"declared_external":false,"source_only":false,"title":"Jacobian余投影恒等式","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:I-JJ+}For any $X\\in\\R^{n\\times r}$, the matrix $\\J_{X}$\ndefined in (\\ref{eq:Jxdef}) satisfies $I-\\J_{X}\\J_{X}^{\\dagger}=(I-XX^{\\dagger})\\otimes(I-XX^{\\dagger})$\nwhere $\\dagger$ denotes the pseudoinverse. \n\\end{lemma}","start":60213,"end":60454}]},{"id":"S071","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"余投影恒等式的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $X=QR$ be the usual QR decomposition with orthogonal complement\n$Q_{\\perp}$, so that $XX^{\\dagger}=QQ^{T}$ and $I-XX^{\\dagger}=Q_{\\perp}Q_{\\perp}^{T}$.\nObserve that for any $M\\in\\R^{n\\times n}$:\n\\[\n\\J_{X}^{T}\\vect(M)=0\\iff(M+M^{T})Q=0\\iff M=Q_{\\perp}\\hat{M}Q_{\\perp}^{T}.\n\\]\nVectorizing the final equality as $\\vect(M)=(Q_{\\perp}\\otimes Q_{\\perp})\\vect(\\hat{M})$\nyields \n\\begin{align*}\n(I-\\J_{X}\\J_{X}^{\\dagger})b & =\\arg\\min_{v}\\{\\|v-b\\|:\\J_{X}^{T}v=0\\}\\\\\n & =\\arg\\min_{v}\\{\\|v-b\\|:v=(Q_{\\perp}\\otimes Q_{\\perp})\\hat{v}\\}\\\\\n & =(Q_{\\perp}\\otimes Q_{\\perp})(Q_{\\perp}\\otimes Q_{\\perp})^{T}b=(Q_{\\perp}Q_{\\perp}^{T}\\otimes Q_{\\perp}Q_{\\perp}^{T})b.\\quad\\qed\n\\end{align*}\n\\end{proof}","start":60455,"end":61154}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"QR分解识别伴随核","source_units":[{"raw_tex":"\\begin{proof}\nLet $X=QR$ be the usual QR decomposition with orthogonal complement\n$Q_{\\perp}$, so that $XX^{\\dagger}=QQ^{T}$ and $I-XX^{\\dagger}=Q_{\\perp}Q_{\\perp}^{T}$.\nObserve that for any $M\\in\\R^{n\\times n}$:\n\\[\n\\J_{X}^{T}\\vect(M)=0\\iff(M+M^{T})Q=0\\iff M=Q_{\\perp}\\hat{M}Q_{\\perp}^{T}.\n\\]\nVectorizing the final equality as $\\vect(M)=(Q_{\\perp}\\otimes Q_{\\perp})\\vect(\\hat{M})$\n","start":60455,"end":60836}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"向量化核空间的正交投影","source_units":[{"raw_tex":"yields \n\\begin{align*}\n(I-\\J_{X}\\J_{X}^{\\dagger})b & =\\arg\\min_{v}\\{\\|v-b\\|:\\J_{X}^{T}v=0\\}\\\\\n & =\\arg\\min_{v}\\{\\|v-b\\|:v=(Q_{\\perp}\\otimes Q_{\\perp})\\hat{v}\\}\\\\\n & =(Q_{\\perp}\\otimes Q_{\\perp})(Q_{\\perp}\\otimes Q_{\\perp})^{T}b=(Q_{\\perp}Q_{\\perp}^{T}\\otimes Q_{\\perp}Q_{\\perp}^{T})b.\\quad\\qed\n\\end{align*}\n\\end{proof}","start":60836,"end":61154}]},{"id":"S074","semantic_type":"lemma","kind":"lemma","labels":["eq:cosprob","lem:tradeoff"],"declared_external":false,"source_only":false,"title":"下界SDP的角度对偶形式","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:tradeoff}We have\n\\[\n\\kappa_{\\lb}(X,Z)=\\max_{t\\ge0}\\frac{1+\\cos\\theta(t)}{2t+1-\\cos\\theta(t)}\n\\]\nwhere $\\cos\\theta(t)$ is defined as follows\n\\begin{equation}\n\\cos\\theta(t)=\\max_{y,W_{i,j}}\\left\\{ \\e^{T}\\f:\\begin{array}{c}\n\\f\\equiv\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\quad\\|\\e\\|\\|\\f\\|=1,\\\\\n(I-ZZ^{\\dagger})\\,\\mat(\\f)\\,(I-ZZ^{\\dagger})\\succeq0,\\\\\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle=2t,\\quad W=[W_{i,j}]_{i,j=1}^{r}\\succeq0,\n\\end{array}\\right\\} \\label{eq:cosprob}\n\\end{equation}\nwith respect to $y\\in\\R^{nr}$ and $W_{i,j}\\in\\R^{n\\times n}$ for\n$i,j\\in\\{1,2,\\dots,r\\}$.\n\\end{lemma}","start":61392,"end":61992}]},{"id":"S075","semantic_type":"proof","kind":"proof","labels":["app:tradeoff","eq:ub2-1"],"declared_external":false,"source_only":false,"title":"角度对偶形式的完整作者证明","source_units":[{"raw_tex":"\\section{\\label{app:tradeoff}Derivation of the dual for the lower-bound problem\n(Proof of \\lemref{tradeoff})}\n\nWe will derive the Lagrangian dual for the following\n\\begin{align*}\n\\kappa_{\\lb}(X,Z) & =\\min_{\\kappa,\\H,s}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H\\preceq\\kappa I,\\qquad\\J_{Z}^{T}s=0,\\qquad s\\in\\vect(\\S_{+}^{n}),\\\\\n\\J_{X}^{T}(\\H\\e+s)=0,\\quad2I_{r}\\otimes\\mat(\\H\\e+s)+\\kappa\\J_{X}^{T}\\J_{X}\\succeq0.\n\\end{array}\\right\\} \n\\end{align*}\nRecall that $\\e=\\vect(XX^{T}-ZZ^{T})$ and $\\J_{X}\\vect(V)=\\vect(XV^{T}+VX^{T})$\nwith respect to fixed $X,Z\\in\\R^{n\\times r}$. Observe that\n\\[\n\\J_{Z}^{T}\\vect(S)=0\\iff(S+S^{T})Z=0\\iff S=Q_{\\perp}S_{\\perp}Q_{\\perp}^{T}\\iff\\vect(S)=\\Q_{\\perp}\\vect(S_{\\perp})\n\\]\nwhere the orthogonal complement $Q_{\\perp}$ of $Z$ is such that\n$Q_{\\perp}Q_{\\perp}^{T}=I_{n}-ZZ^{\\dagger}$, and $\\Q_{\\perp}=Q_{\\perp}\\otimes Q_{\\perp}$.\nSubstituting and taking the Lagrangian dual yields\n\\begin{align}\n\\kappa_{\\lb}(X,Z) & =\\min_{\\kappa,\\H,s_{\\perp}}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H\\preceq\\kappa I,\\quad s_{\\perp}\\in\\vect(\\S_{+}^{n-r^{\\star}}),\\quad\\J_{X}^{T}(\\H\\e+\\Q_{\\perp}s_{\\perp})=0,\\\\\n2I_{r}\\otimes\\mat(\\H\\e+\\Q_{\\perp}s_{\\perp})+\\kappa\\J_{X}^{T}\\J_{X}\\succeq0\n\\end{array}\\right\\} \\nonumber \\\\\n & =\\max_{y,W_{i,j},U,V}\\left\\{ \\tr(U):\\begin{array}{c}\n\\f\\e^{T}+\\e\\f^{T}=U-V,\\quad\\tr(V)+\\inner{\\J_{X}^{T}\\J_{X}}W=1,\\\\\n\\f\\equiv\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\quad\\Q_{\\perp}^{T}\\f\\in\\vect(\\S_{+}^{n-r^{\\star}}),\\\\\nU,V\\succeq0,\\quad W=[W_{i,j}]_{i,j=1}^{r}\\succeq0\n\\end{array}\\right\\} \\label{eq:ub2-1}\n\\end{align}\nin which $y\\in\\R^{nr}$ and $W_{i,j}\\in\\R^{n\\times n}$ for $i,j\\in\\{1,2,\\dots,r\\}$.\nStrong duality holds here because $y=0,$ $W=\\epsilon\\cdot I_{nr},$\n$U=\\frac{1}{n}I_{n},$ and $V=\\frac{1}{n}I_{n}+\\epsilon\\cdot r\\cdot[\\e\\vect(I_{n})^{T}+\\e\\vect(I_{n})^{T}]$\nis strictly feasible for a sufficiently small $\\epsilon>0$. \n\nThe dual problem (\\ref{eq:ub2-1}) has a closed-form solution over\n$U$ and $V$ \\citep[Lemma~13]{zhang2019sharp}\n\\begin{align*}\n\\frac{\\tr([M]_{-})+\\alpha}{\\tr([M]_{+})}= & \\min_{t,U,V}\\left\\{ \\tr(V)+\\alpha\\cdot t:\\begin{array}{c}\ntM=U-V,\\quad t\\ge0,\\\\\n\\tr(U)=1,\\quad U,V\\succeq0\n\\end{array}\\right\\} ,\n\\end{align*}\nand $\\f\\e^{T}+\\e\\f^{T}$ has exactly two nonzero eigenvalues $\\e^{T}\\f\\pm\\|\\e\\|\\|\\f\\|$\n\\citep[Lemma~14]{zhang2019sharp}. Substituting this solution yields\nexactly \n\\begin{align*}\n\\kappa_{\\lb}(X,Z) & =\\max_{y,W_{i,j}}\\left\\{ \\frac{\\|\\e\\|\\|\\f\\|+\\e^{T}\\f}{\\|\\e\\|\\|\\f\\|-\\e^{T}\\f+\\inner{\\J_{X}^{T}\\J_{X}}W}:\\begin{array}{c}\n\\Q_{\\perp}\\Q_{\\perp}^{T}\\f\\in\\vect(\\S_{+}^{n}),\\\\\n\\f=\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\\\\nW=[W_{i,j}]_{i,j=1}^{r}\\succeq0\n\\end{array}\\right\\} \\\\\n & =\\max_{t\\ge0}\\frac{1+\\cos\\theta(t)}{2t+1-\\cos\\theta(t)}\n\\end{align*}\nwhere $\\cos\\theta(t)$ is itself is defined as the following maximization\n\\[\n\\cos\\theta(t)=\\max_{y,W_{i,j}}\\left\\{ \\e^{T}\\f:\\begin{array}{c}\n\\f\\equiv\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\quad\\|\\e\\|\\|\\f\\|=1,\\\\\n(I-ZZ^{\\dagger})\\,\\mat(\\f)\\,(I-ZZ^{\\dagger})\\succeq0,\\\\\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle=2t,\\quad W=[W_{i,j}]_{i,j=1}^{r}\\succeq0.\n\\end{array}\\right\\} \n\\]\n\n","start":84068,"end":87151}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","labels":["app:tradeoff"],"declared_external":false,"source_only":false,"title":"限制梯度松弛的支持空间","source_units":[{"raw_tex":"\\section{\\label{app:tradeoff}Derivation of the dual for the lower-bound problem\n(Proof of \\lemref{tradeoff})}\n\nWe will derive the Lagrangian dual for the following\n\\begin{align*}\n\\kappa_{\\lb}(X,Z) & =\\min_{\\kappa,\\H,s}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H\\preceq\\kappa I,\\qquad\\J_{Z}^{T}s=0,\\qquad s\\in\\vect(\\S_{+}^{n}),\\\\\n\\J_{X}^{T}(\\H\\e+s)=0,\\quad2I_{r}\\otimes\\mat(\\H\\e+s)+\\kappa\\J_{X}^{T}\\J_{X}\\succeq0.\n\\end{array}\\right\\} \n\\end{align*}\nRecall that $\\e=\\vect(XX^{T}-ZZ^{T})$ and $\\J_{X}\\vect(V)=\\vect(XV^{T}+VX^{T})$\nwith respect to fixed $X,Z\\in\\R^{n\\times r}$. Observe that\n\\[\n\\J_{Z}^{T}\\vect(S)=0\\iff(S+S^{T})Z=0\\iff S=Q_{\\perp}S_{\\perp}Q_{\\perp}^{T}\\iff\\vect(S)=\\Q_{\\perp}\\vect(S_{\\perp})\n\\]\nwhere the orthogonal complement $Q_{\\perp}$ of $Z$ is such that\n$Q_{\\perp}Q_{\\perp}^{T}=I_{n}-ZZ^{\\dagger}$, and $\\Q_{\\perp}=Q_{\\perp}\\otimes Q_{\\perp}$.\n","start":84068,"end":84927}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","labels":["app:tradeoff","eq:ub2-1"],"declared_external":false,"source_only":false,"title":"形成对偶并声称强对偶","source_units":[{"raw_tex":"Substituting and taking the Lagrangian dual yields\n\\begin{align}\n\\kappa_{\\lb}(X,Z) & =\\min_{\\kappa,\\H,s_{\\perp}}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H\\preceq\\kappa I,\\quad s_{\\perp}\\in\\vect(\\S_{+}^{n-r^{\\star}}),\\quad\\J_{X}^{T}(\\H\\e+\\Q_{\\perp}s_{\\perp})=0,\\\\\n2I_{r}\\otimes\\mat(\\H\\e+\\Q_{\\perp}s_{\\perp})+\\kappa\\J_{X}^{T}\\J_{X}\\succeq0\n\\end{array}\\right\\} \\nonumber \\\\\n & =\\max_{y,W_{i,j},U,V}\\left\\{ \\tr(U):\\begin{array}{c}\n\\f\\e^{T}+\\e\\f^{T}=U-V,\\quad\\tr(V)+\\inner{\\J_{X}^{T}\\J_{X}}W=1,\\\\\n\\f\\equiv\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\quad\\Q_{\\perp}^{T}\\f\\in\\vect(\\S_{+}^{n-r^{\\star}}),\\\\\nU,V\\succeq0,\\quad W=[W_{i,j}]_{i,j=1}^{r}\\succeq0\n\\end{array}\\right\\} \\label{eq:ub2-1}\n\\end{align}\nin which $y\\in\\R^{nr}$ and $W_{i,j}\\in\\R^{n\\times n}$ for $i,j\\in\\{1,2,\\dots,r\\}$.\nStrong duality holds here because $y=0,$ $W=\\epsilon\\cdot I_{nr},$\n$U=\\frac{1}{n}I_{n},$ and $V=\\frac{1}{n}I_{n}+\\epsilon\\cdot r\\cdot[\\e\\vect(I_{n})^{T}+\\e\\vect(I_{n})^{T}]$\nis strictly feasible for a sufficiently small $\\epsilon>0$. \n","start":84927,"end":85936}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","labels":["app:tradeoff"],"declared_external":false,"source_only":false,"title":"消去半正定变量并计算秩二谱","source_units":[{"raw_tex":"\nThe dual problem (\\ref{eq:ub2-1}) has a closed-form solution over\n$U$ and $V$ \\citep[Lemma~13]{zhang2019sharp}\n\\begin{align*}\n\\frac{\\tr([M]_{-})+\\alpha}{\\tr([M]_{+})}= & \\min_{t,U,V}\\left\\{ \\tr(V)+\\alpha\\cdot t:\\begin{array}{c}\ntM=U-V,\\quad t\\ge0,\\\\\n\\tr(U)=1,\\quad U,V\\succeq0\n\\end{array}\\right\\} ,\n\\end{align*}\nand $\\f\\e^{T}+\\e\\f^{T}$ has exactly two nonzero eigenvalues $\\e^{T}\\f\\pm\\|\\e\\|\\|\\f\\|$\n\\citep[Lemma~14]{zhang2019sharp}. Substituting this solution yields\n","start":85936,"end":86403}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","labels":["app:tradeoff"],"declared_external":false,"source_only":false,"title":"齐次归一化为角度最大化","source_units":[{"raw_tex":"exactly \n\\begin{align*}\n\\kappa_{\\lb}(X,Z) & =\\max_{y,W_{i,j}}\\left\\{ \\frac{\\|\\e\\|\\|\\f\\|+\\e^{T}\\f}{\\|\\e\\|\\|\\f\\|-\\e^{T}\\f+\\inner{\\J_{X}^{T}\\J_{X}}W}:\\begin{array}{c}\n\\Q_{\\perp}\\Q_{\\perp}^{T}\\f\\in\\vect(\\S_{+}^{n}),\\\\\n\\f=\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\\\\nW=[W_{i,j}]_{i,j=1}^{r}\\succeq0\n\\end{array}\\right\\} \\\\\n & =\\max_{t\\ge0}\\frac{1+\\cos\\theta(t)}{2t+1-\\cos\\theta(t)}\n\\end{align*}\nwhere $\\cos\\theta(t)$ is itself is defined as the following maximization\n\\[\n\\cos\\theta(t)=\\max_{y,W_{i,j}}\\left\\{ \\e^{T}\\f:\\begin{array}{c}\n\\f\\equiv\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\quad\\|\\e\\|\\|\\f\\|=1,\\\\\n(I-ZZ^{\\dagger})\\,\\mat(\\f)\\,(I-ZZ^{\\dagger})\\succeq0,\\\\\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle=2t,\\quad W=[W_{i,j}]_{i,j=1}^{r}\\succeq0.\n\\end{array}\\right\\} \n\\]\n\n","start":86403,"end":87151}]},{"id":"S080","semantic_type":"lemma","kind":"lemma","labels":["lem:costheta"],"declared_external":false,"source_only":false,"title":"角度目标的可行证书下界","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:costheta}We have $\\cos\\theta(t)\\ge\\alpha\\cdot(t/\\beta)+\\sqrt{1-\\alpha^{2}}\\sqrt{1-t^{2}/\\beta^{2}}$\nfor $0\\le t\\le\\alpha\\beta$ where $Z_{\\perp}=(I-XX^{\\dagger})Z$ and\n\\begin{align*}\n\\alpha & =\\frac{\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}{\\|XX^{T}-ZZ^{T}\\|_{F}}, & \\beta & =\\frac{\\lambda_{\\min}(X^{T}X)\\cdot\\tr(Z_{\\perp}Z_{\\perp}^{T})}{\\|XX^{T}-ZZ^{T}\\|_{F}\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}.\n\\end{align*}\n\\end{lemma}","start":65276,"end":65706}]},{"id":"S081","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"角度下界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nWrite $\\Pi=I-ZZ^{\\dagger}$ and $\\e=\\vect(XX^{T}-ZZ^{T})$, and decompose\n$\\e_{1}=(\\J_{X}\\J_{X}^{\\dagger})\\e$ and $\\e_{2}=(I-\\J_{X}\\J_{X}^{\\dagger})\\e$.\nWe verify that the following heuristic choice\n\\begin{gather*}\ny=\\underbrace{\\frac{\\sqrt{1-(t/\\beta)^{2}}}{\\|\\e\\|\\|\\e_{1}\\|}}_{\\gamma_{1}}\\;\\J_{X}^{\\dagger}\\e,\\quad W=\\underbrace{\\frac{t/\\beta}{\\|\\e\\|\\|\\e_{2}\\|}}_{\\gamma_{2}}\\;\\sum_{i=1}^{r}\\vect(Z_{\\perp}e_{i}v_{r}^{T})\\vect(Z_{\\perp}e_{i}v_{r}^{T})^{T},\n\\end{gather*}\nis feasible for (\\ref{eq:cosprob}). Indeed, $W\\succeq0$ holds, and\nwe have from (\\ref{eq:e2}), (\\ref{eq:Wsol1}), and (\\ref{eq:Wsol2}):\n\\begin{gather*}\n\\f=\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i})=\\gamma_{1}\\cdot\\J_{X}\\J_{X}^{\\dagger}\\e-\\gamma_{2}\\vect(Z_{\\perp}Z_{\\perp}^{T})=\\gamma_{1}\\cdot\\e_{1}+\\gamma_{2}\\e_{2},\\\\\n\\|\\e\\|^{2}\\|\\f\\|^{2}=\\gamma_{1}^{2}\\cdot\\|\\e\\|^{2}\\|\\e_{1}\\|^{2}+\\gamma_{2}^{2}\\cdot\\|\\e\\|^{2}\\|\\e_{2}\\|^{2}=(1-(t/\\beta)^{2})+(t/\\beta)^{2}=1,\\\\\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle=2\\left(\\frac{t/\\beta}{\\|\\e\\|\\|\\e_{2}\\|}\\right)\\cdot\\lambda_{\\min}(X^{T}X)\\|Z_{\\perp}\\|_{F}^{2}=\\frac{2t}{\\beta}\\cdot\\frac{\\sigma_{r}^{2}\\cdot\\tr(Z_{\\perp}Z_{\\perp}^{T})}{\\|\\e\\|\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}=2t.\n\\end{gather*}\nTo verify that $\\Pi\\,\\mat(\\f)\\,\\Pi\\succeq0$, note that $t\\le\\alpha\\beta$\nimplies $t/\\beta\\le\\alpha\\le1$, so it follows from $\\alpha=\\|\\e_{2}\\|/\\|\\e\\|$\nand $\\sqrt{1-\\alpha^{2}}=\\|\\e_{1}\\|/\\|\\e\\|$ that\n\\[\n\\gamma_{1}=\\frac{\\sqrt{1-(t/\\beta)^{2}}}{\\|\\e_{1}\\|\\|\\e\\|}\\ge\\frac{\\sqrt{1-\\alpha^{2}}}{\\|\\e_{1}\\|\\|\\e\\|}=\\frac{1}{\\|\\e\\|^{2}}=\\frac{\\alpha}{\\|\\e_{2}\\|\\|\\e\\|}\\ge\\frac{t/\\beta}{\\|\\e_{2}\\|\\|\\e\\|}=\\gamma_{2}.\n\\]\nIn turn, $\\gamma_{1}\\ge\\gamma_{2}$ implies\n\\begin{align*}\n\\f & =\\gamma_{1}\\cdot\\e_{1}+\\gamma_{2}\\cdot\\e_{2}=\\gamma_{1}\\cdot\\e-(\\gamma_{1}-\\gamma_{2})\\e_{2},\\\\\n\\mat(\\f) & =\\gamma_{1}\\cdot(XX^{T}-ZZ^{T})+(\\gamma_{1}-\\gamma_{2})\\cdot Z_{\\perp}Z_{\\perp}^{T},\\\\\n\\Pi\\,\\mat(\\f)\\,\\Pi & =\\gamma_{1}\\cdot\\Pi(XX^{T})\\Pi+(\\gamma_{1}-\\gamma_{2})\\cdot\\Pi(Z_{\\perp}Z_{\\perp}^{T})\\Pi\\succeq0.\n\\end{align*}\nHaving verified $y$ and $W$ as feasible for (\\ref{eq:cosprob}),\nwe use its objective value to heuristically lower-bound the maximum\n\\[\n\\cos\\theta(t)\\ge\\e^{T}\\f=\\gamma_{1}\\cdot\\|\\e_{1}\\|^{2}+\\gamma_{2}\\cdot\\|\\e_{2}\\|^{2}=\\sqrt{1-(t/\\beta)^{2}}\\sqrt{1-\\alpha^{2}}+(t/\\beta)\\cdot\\alpha.\n\\]\n \\qed\n\\end{proof}","start":65707,"end":68016}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"构造证书并验证归一化与代价","source_units":[{"raw_tex":"\\begin{proof}\nWrite $\\Pi=I-ZZ^{\\dagger}$ and $\\e=\\vect(XX^{T}-ZZ^{T})$, and decompose\n$\\e_{1}=(\\J_{X}\\J_{X}^{\\dagger})\\e$ and $\\e_{2}=(I-\\J_{X}\\J_{X}^{\\dagger})\\e$.\nWe verify that the following heuristic choice\n\\begin{gather*}\ny=\\underbrace{\\frac{\\sqrt{1-(t/\\beta)^{2}}}{\\|\\e\\|\\|\\e_{1}\\|}}_{\\gamma_{1}}\\;\\J_{X}^{\\dagger}\\e,\\quad W=\\underbrace{\\frac{t/\\beta}{\\|\\e\\|\\|\\e_{2}\\|}}_{\\gamma_{2}}\\;\\sum_{i=1}^{r}\\vect(Z_{\\perp}e_{i}v_{r}^{T})\\vect(Z_{\\perp}e_{i}v_{r}^{T})^{T},\n\\end{gather*}\nis feasible for (\\ref{eq:cosprob}). Indeed, $W\\succeq0$ holds, and\nwe have from (\\ref{eq:e2}), (\\ref{eq:Wsol1}), and (\\ref{eq:Wsol2}):\n\\begin{gather*}\n\\f=\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i})=\\gamma_{1}\\cdot\\J_{X}\\J_{X}^{\\dagger}\\e-\\gamma_{2}\\vect(Z_{\\perp}Z_{\\perp}^{T})=\\gamma_{1}\\cdot\\e_{1}+\\gamma_{2}\\e_{2},\\\\\n\\|\\e\\|^{2}\\|\\f\\|^{2}=\\gamma_{1}^{2}\\cdot\\|\\e\\|^{2}\\|\\e_{1}\\|^{2}+\\gamma_{2}^{2}\\cdot\\|\\e\\|^{2}\\|\\e_{2}\\|^{2}=(1-(t/\\beta)^{2})+(t/\\beta)^{2}=1,\\\\\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle=2\\left(\\frac{t/\\beta}{\\|\\e\\|\\|\\e_{2}\\|}\\right)\\cdot\\lambda_{\\min}(X^{T}X)\\|Z_{\\perp}\\|_{F}^{2}=\\frac{2t}{\\beta}\\cdot\\frac{\\sigma_{r}^{2}\\cdot\\tr(Z_{\\perp}Z_{\\perp}^{T})}{\\|\\e\\|\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}=2t.\n\\end{gather*}\n","start":65707,"end":66910}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"以系数次序验证正交补半正定","source_units":[{"raw_tex":"To verify that $\\Pi\\,\\mat(\\f)\\,\\Pi\\succeq0$, note that $t\\le\\alpha\\beta$\nimplies $t/\\beta\\le\\alpha\\le1$, so it follows from $\\alpha=\\|\\e_{2}\\|/\\|\\e\\|$\nand $\\sqrt{1-\\alpha^{2}}=\\|\\e_{1}\\|/\\|\\e\\|$ that\n\\[\n\\gamma_{1}=\\frac{\\sqrt{1-(t/\\beta)^{2}}}{\\|\\e_{1}\\|\\|\\e\\|}\\ge\\frac{\\sqrt{1-\\alpha^{2}}}{\\|\\e_{1}\\|\\|\\e\\|}=\\frac{1}{\\|\\e\\|^{2}}=\\frac{\\alpha}{\\|\\e_{2}\\|\\|\\e\\|}\\ge\\frac{t/\\beta}{\\|\\e_{2}\\|\\|\\e\\|}=\\gamma_{2}.\n\\]\nIn turn, $\\gamma_{1}\\ge\\gamma_{2}$ implies\n\\begin{align*}\n\\f & =\\gamma_{1}\\cdot\\e_{1}+\\gamma_{2}\\cdot\\e_{2}=\\gamma_{1}\\cdot\\e-(\\gamma_{1}-\\gamma_{2})\\e_{2},\\\\\n\\mat(\\f) & =\\gamma_{1}\\cdot(XX^{T}-ZZ^{T})+(\\gamma_{1}-\\gamma_{2})\\cdot Z_{\\perp}Z_{\\perp}^{T},\\\\\n\\Pi\\,\\mat(\\f)\\,\\Pi & =\\gamma_{1}\\cdot\\Pi(XX^{T})\\Pi+(\\gamma_{1}-\\gamma_{2})\\cdot\\Pi(Z_{\\perp}Z_{\\perp}^{T})\\Pi\\succeq0.\n\\end{align*}\n","start":66910,"end":67712}]},{"id":"S084","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"计算可行证书的角度值","source_units":[{"raw_tex":"Having verified $y$ and $W$ as feasible for (\\ref{eq:cosprob}),\nwe use its objective value to heuristically lower-bound the maximum\n\\[\n\\cos\\theta(t)\\ge\\e^{T}\\f=\\gamma_{1}\\cdot\\|\\e_{1}\\|^{2}+\\gamma_{2}\\cdot\\|\\e_{2}\\|^{2}=\\sqrt{1-(t/\\beta)^{2}}\\sqrt{1-\\alpha^{2}}+(t/\\beta)\\cdot\\alpha.\n\\]\n \\qed\n\\end{proof}","start":67712,"end":68016}]},{"id":"S085","semantic_type":"lemma","kind":"lemma","labels":["lem:counter"],"declared_external":false,"source_only":false,"title":"上界SDP的特征方向证书","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:counter}Let $[Q_{1},Q_{2}]$ have orthonormal columns\nwith $Q_{1}\\in\\R^{n\\times r}$ and $Q_{2}\\in\\R^{n\\times r^{\\star}}$.\nIf $\\kappa=1+2\\sqrt{r/r^{\\star}}$, and $\\H$ is any $n^{2}\\times n^{2}$\nmatrix that satisfies $I\\preceq\\H\\preceq\\kappa I$ and the following\n$rr^{\\star}+2$ eigenvalue equations\n\\begin{gather*}\n\\H\\vect\\left(Q_{1}Q_{1}^{T}+\\sqrt{r/r^{\\star}}Q_{2}Q_{2}\\right)=\\kappa\\cdot\\vect\\left(Q_{1}Q_{1}^{T}+\\sqrt{r/r^{\\star}}Q_{2}Q_{2}\\right),\\\\\n\\H\\vect\\left(Q_{1}Q_{1}^{T}-\\sqrt{r/r^{\\star}}Q_{2}Q_{2}\\right)=1\\cdot\\vect\\left(Q_{1}Q_{1}^{T}-\\sqrt{r/r^{\\star}}Q_{2}Q_{2}\\right),\\\\\n\\H\\vect\\left(Q_{1}VQ_{2}^{T}+Q_{2}V^{T}Q_{1}^{T}\\right)=\\kappa\\cdot\\vect\\left(Q_{1}VQ_{2}^{T}+Q_{2}V^{T}Q_{1}^{T}\\right)\\qquad\\text{for all }V\\in\\R^{r\\times r^{\\star}},\n\\end{gather*}\nthen $(\\kappa,\\H)$ is a feasible point for (\\ref{eq:kappaub_prob})\nwith $X=Q_{1}$ and $Z=\\sqrt{1+\\sqrt{r/r^{\\star}}}Q_{2}$. Moveover,\n$\\e^{T}\\H\\e=(1+2\\sqrt{r/r^{\\star}})\\cdot(1+\\sqrt{r/r^{\\star}})\\cdot r^{\\star}$.\n\\end{lemma}","start":69778,"end":70798}]},{"id":"S086","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"特征方向证书的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nWrite $\\rho\\equiv\\sqrt{r/r^{\\star}}$, and take $[Q_{1},Q_{2}]=I_{r+r^{\\star}}$\nwithout loss of generality. Decomposing $\\e=\\vect(XX^{T}-ZZ^{T})$\ninto eigenvectors and applying $\\H$ yields\n\\begin{gather*}\n\\mat(\\e)=\\begin{bmatrix}I_{r} & 0\\\\\n0 & -(1+\\rho)I_{r^{\\star}}\n\\end{bmatrix}=-\\frac{1}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & \\rho I_{r^{\\star}}\n\\end{bmatrix}+\\frac{1+2\\rho}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & -\\rho I_{r^{\\star}}\n\\end{bmatrix},\\\\\n\\mat(\\H\\e)=-\\frac{1+2\\rho}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & \\rho I_{r^{\\star}}\n\\end{bmatrix}+\\frac{1+2\\rho}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & -\\rho I_{r^{\\star}}\n\\end{bmatrix}=\\begin{bmatrix}0_{r} & 0\\\\\n0 & -\\kappa I_{r^{\\star}}\n\\end{bmatrix}.\n\\end{gather*}\nClearly, $\\inner{\\mat(\\e)}{\\mat(\\H\\e)}=\\kappa(1+\\rho)r^{\\star}.$\nFor $v=\\vect(V)=\\vect([V_{1};V_{2}])$ with $V_{1}\\in\\R^{r\\times r}$\nand $V_{2}\\in\\R^{r^{\\star}\\times r}$, observe that \n\\begin{gather*}\nv^{T}\\J_{X}\\H\\e=\\left\\langle \\begin{bmatrix}V_{1}+V_{1}^{T} & V_{2}^{T}\\\\\nV_{2} & 0_{r^{\\star}}\n\\end{bmatrix},\\begin{bmatrix}0_{r} & 0\\\\\n0 & -\\kappa I_{r^{\\star}}\n\\end{bmatrix}\\right\\rangle =0,\\\\\nv^{T}[I_{r}\\otimes\\mat(\\H\\e)]v=\\langle\\mat(\\H\\e),VV^{T}\\rangle=\\left\\langle \\begin{bmatrix}0_{r} & 0\\\\\n0 & -\\kappa I_{r^{\\star}}\n\\end{bmatrix},\\begin{bmatrix}V_{1}V_{1}^{T} & V_{1}V_{2}^{T}\\\\\nV_{2}V_{1}^{T} & V_{2}V_{2}^{T}\n\\end{bmatrix}\\right\\rangle =-\\kappa\\|V_{2}\\|^{2},\\\\\nv^{T}\\J_{X}^{T}\\H\\J_{X}v=\\left\\Vert \\H^{1/2}\\vect\\left(\\begin{bmatrix}V_{1}+V_{1}^{T} & V_{2}^{T}\\\\\nV_{2} & 0_{r^{\\star}}\n\\end{bmatrix}\\right)\\right\\Vert ^{2}\\ge\\left\\Vert \\H^{1/2}\\vect\\left(\\begin{bmatrix}0_{r} & V_{2}^{T}\\\\\nV_{2} & 0_{r^{\\star}}\n\\end{bmatrix}\\right)\\right\\Vert ^{2}=2\\kappa\\|V_{2}\\|^{2}.\n\\end{gather*}\nTherefore, $v^{T}\\J_{X}\\H\\e=0$ and $2v^{T}[I_{r}\\otimes\\mat(\\H\\e)]v\\ge-v^{T}\\J_{X}^{T}\\H\\J_{X}v$\nhold for all $v$. \\qed\n\\end{proof}","start":70799,"end":72653}]},{"id":"S087","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"误差特征分解与能量计算","source_units":[{"raw_tex":"\\begin{proof}\nWrite $\\rho\\equiv\\sqrt{r/r^{\\star}}$, and take $[Q_{1},Q_{2}]=I_{r+r^{\\star}}$\nwithout loss of generality. Decomposing $\\e=\\vect(XX^{T}-ZZ^{T})$\ninto eigenvectors and applying $\\H$ yields\n\\begin{gather*}\n\\mat(\\e)=\\begin{bmatrix}I_{r} & 0\\\\\n0 & -(1+\\rho)I_{r^{\\star}}\n\\end{bmatrix}=-\\frac{1}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & \\rho I_{r^{\\star}}\n\\end{bmatrix}+\\frac{1+2\\rho}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & -\\rho I_{r^{\\star}}\n\\end{bmatrix},\\\\\n\\mat(\\H\\e)=-\\frac{1+2\\rho}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & \\rho I_{r^{\\star}}\n\\end{bmatrix}+\\frac{1+2\\rho}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & -\\rho I_{r^{\\star}}\n\\end{bmatrix}=\\begin{bmatrix}0_{r} & 0\\\\\n0 & -\\kappa I_{r^{\\star}}\n\\end{bmatrix}.\n\\end{gather*}\nClearly, $\\inner{\\mat(\\e)}{\\mat(\\H\\e)}=\\kappa(1+\\rho)r^{\\star}.$\n","start":70799,"end":71593}]},{"id":"S088","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"分块扰动验证一阶消失","source_units":[{"raw_tex":"For $v=\\vect(V)=\\vect([V_{1};V_{2}])$ with $V_{1}\\in\\R^{r\\times r}$\nand $V_{2}\\in\\R^{r^{\\star}\\times r}$, observe that \n\\begin{gather*}\nv^{T}\\J_{X}\\H\\e=\\left\\langle \\begin{bmatrix}V_{1}+V_{1}^{T} & V_{2}^{T}\\\\\nV_{2} & 0_{r^{\\star}}\n\\end{bmatrix},\\begin{bmatrix}0_{r} & 0\\\\\n0 & -\\kappa I_{r^{\\star}}\n\\end{bmatrix}\\right\\rangle =0,\\\\\n","start":71593,"end":71925}]},{"id":"S089","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"交叉块特征值验证二阶约束","source_units":[{"raw_tex":"v^{T}[I_{r}\\otimes\\mat(\\H\\e)]v=\\langle\\mat(\\H\\e),VV^{T}\\rangle=\\left\\langle \\begin{bmatrix}0_{r} & 0\\\\\n0 & -\\kappa I_{r^{\\star}}\n\\end{bmatrix},\\begin{bmatrix}V_{1}V_{1}^{T} & V_{1}V_{2}^{T}\\\\\nV_{2}V_{1}^{T} & V_{2}V_{2}^{T}\n\\end{bmatrix}\\right\\rangle =-\\kappa\\|V_{2}\\|^{2},\\\\\nv^{T}\\J_{X}^{T}\\H\\J_{X}v=\\left\\Vert \\H^{1/2}\\vect\\left(\\begin{bmatrix}V_{1}+V_{1}^{T} & V_{2}^{T}\\\\\nV_{2} & 0_{r^{\\star}}\n\\end{bmatrix}\\right)\\right\\Vert ^{2}\\ge\\left\\Vert \\H^{1/2}\\vect\\left(\\begin{bmatrix}0_{r} & V_{2}^{T}\\\\\nV_{2} & 0_{r^{\\star}}\n\\end{bmatrix}\\right)\\right\\Vert ^{2}=2\\kappa\\|V_{2}\\|^{2}.\n\\end{gather*}\nTherefore, $v^{T}\\J_{X}\\H\\e=0$ and $2v^{T}[I_{r}\\otimes\\mat(\\H\\e)]v\\ge-v^{T}\\J_{X}^{T}\\H\\J_{X}v$\nhold for all $v$. \\qed\n\\end{proof}","start":71925,"end":72653}]},{"id":"S090","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始外部前提列表第1项：standard convex KKT conditions over the positive semidefinite cone。"],"source_only":false,"title":"半正定锥上的凸KKT条件","source_units":[]},{"id":"S091","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始外部前提列表第1项：Slater strong duality及标准凸KKT背景。"],"source_only":false,"title":"Lagrange对偶与Slater强对偶","source_units":[]},{"id":"S092","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始外部前提列表第2项：standard matrix calculus及fundamental theorem of calculus。"],"source_only":false,"title":"矩阵链式法则与微积分基本定理","source_units":[]},{"id":"S093","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始外部前提列表第2项：spectral and QR decompositions、Moore–Penrose projection identities。"],"source_only":false,"title":"谱分解、QR与伪逆投影背景","source_units":[]},{"id":"S094","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始外部前提列表第2项明确允许Cauchy–Schwarz。"],"source_only":false,"title":"Cauchy–Schwarz计数界","source_units":[]},{"id":"S095","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始外部前提列表第5项；源文L1564–1571引用zhang2019sharp，Lemma 13。"],"source_only":false,"title":"半正定变量消元恒等式","source_units":[]},{"id":"S096","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始外部前提列表第5项；源文L1572–1573引用zhang2019sharp，Lemma 14。"],"source_only":false,"title":"对称秩二矩阵的谱公式","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S027","edges":[{"source_id":"S001","target_id":"S027","kind":"supports"},{"source_id":"S002","target_id":"S027","kind":"supports"},{"source_id":"S003","target_id":"S027","kind":"supports"},{"source_id":"S001","target_id":"S010","kind":"supports"},{"source_id":"S003","target_id":"S010","kind":"supports"},{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S028","kind":"supports"},{"source_id":"S008","target_id":"S028","kind":"supports"},{"source_id":"S003","target_id":"S030","kind":"supports"},{"source_id":"S090","target_id":"S030","kind":"supports"},{"source_id":"S091","target_id":"S030","kind":"supports"},{"source_id":"S006","target_id":"S031","kind":"supports"},{"source_id":"S008","target_id":"S031","kind":"supports"},{"source_id":"S092","target_id":"S031","kind":"supports"},{"source_id":"S012","target_id":"S032","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S030","target_id":"S034","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S032","target_id":"S029","kind":"supports"},{"source_id":"S033","target_id":"S029","kind":"supports"},{"source_id":"S034","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S085","kind":"supports"},{"source_id":"S093","target_id":"S087","kind":"supports"},{"source_id":"S006","target_id":"S087","kind":"supports"},{"source_id":"S008","target_id":"S087","kind":"supports"},{"source_id":"S087","target_id":"S088","kind":"supports"},{"source_id":"S008","target_id":"S088","kind":"supports"},{"source_id":"S087","target_id":"S089","kind":"supports"},{"source_id":"S006","target_id":"S089","kind":"supports"},{"source_id":"S008","target_id":"S089","kind":"supports"},{"source_id":"S087","target_id":"S086","kind":"supports"},{"source_id":"S088","target_id":"S086","kind":"supports"},{"source_id":"S089","target_id":"S086","kind":"supports"},{"source_id":"S086","target_id":"S085","kind":"supports"},{"source_id":"S085","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S092","target_id":"S024","kind":"supports"},{"source_id":"S093","target_id":"S024","kind":"supports"},{"source_id":"S085","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S026","target_id":"S036","kind":"supports"},{"source_id":"S023","target_id":"S036","kind":"supports"},{"source_id":"S024","target_id":"S036","kind":"supports"},{"source_id":"S025","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S015","target_id":"S052","kind":"supports"},{"source_id":"S093","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S005","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S005","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S058","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S055","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S055","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S063","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S060","kind":"supports"},{"source_id":"S063","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S007","target_id":"S066","kind":"supports"},{"source_id":"S008","target_id":"S066","kind":"supports"},{"source_id":"S093","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S050","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S059","target_id":"S068","kind":"supports"},{"source_id":"S066","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S094","target_id":"S069","kind":"supports"},{"source_id":"S005","target_id":"S069","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"},{"source_id":"S067","target_id":"S065","kind":"supports"},{"source_id":"S068","target_id":"S065","kind":"supports"},{"source_id":"S069","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S016","target_id":"S072","kind":"supports"},{"source_id":"S007","target_id":"S072","kind":"supports"},{"source_id":"S093","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S006","target_id":"S073","kind":"supports"},{"source_id":"S093","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S071","kind":"supports"},{"source_id":"S073","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S070","kind":"supports"},{"source_id":"S028","target_id":"S076","kind":"supports"},{"source_id":"S006","target_id":"S076","kind":"supports"},{"source_id":"S007","target_id":"S076","kind":"supports"},{"source_id":"S008","target_id":"S076","kind":"supports"},{"source_id":"S093","target_id":"S076","kind":"supports"},{"source_id":"S076","target_id":"S077","kind":"supports"},{"source_id":"S091","target_id":"S077","kind":"supports"},{"source_id":"S077","target_id":"S078","kind":"supports"},{"source_id":"S095","target_id":"S078","kind":"supports"},{"source_id":"S096","target_id":"S078","kind":"supports"},{"source_id":"S076","target_id":"S079","kind":"supports"},{"source_id":"S077","target_id":"S079","kind":"supports"},{"source_id":"S078","target_id":"S079","kind":"supports"},{"source_id":"S076","target_id":"S075","kind":"supports"},{"source_id":"S077","target_id":"S075","kind":"supports"},{"source_id":"S078","target_id":"S075","kind":"supports"},{"source_id":"S079","target_id":"S075","kind":"supports"},{"source_id":"S075","target_id":"S074","kind":"supports"},{"source_id":"S074","target_id":"S039","kind":"supports"},{"source_id":"S070","target_id":"S039","kind":"supports"},{"source_id":"S008","target_id":"S039","kind":"supports"},{"source_id":"S070","target_id":"S017","kind":"supports"},{"source_id":"S008","target_id":"S017","kind":"supports"},{"source_id":"S006","target_id":"S017","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S018","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S040","target_id":"S020","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S016","target_id":"S020","kind":"supports"},{"source_id":"S074","target_id":"S082","kind":"supports"},{"source_id":"S017","target_id":"S082","kind":"supports"},{"source_id":"S019","target_id":"S082","kind":"supports"},{"source_id":"S020","target_id":"S082","kind":"supports"},{"source_id":"S007","target_id":"S082","kind":"supports"},{"source_id":"S082","target_id":"S083","kind":"supports"},{"source_id":"S017","target_id":"S083","kind":"supports"},{"source_id":"S074","target_id":"S084","kind":"supports"},{"source_id":"S082","target_id":"S084","kind":"supports"},{"source_id":"S083","target_id":"S084","kind":"supports"},{"source_id":"S082","target_id":"S081","kind":"supports"},{"source_id":"S083","target_id":"S081","kind":"supports"},{"source_id":"S084","target_id":"S081","kind":"supports"},{"source_id":"S081","target_id":"S080","kind":"supports"},{"source_id":"S080","target_id":"S041","kind":"supports"},{"source_id":"S074","target_id":"S041","kind":"supports"},{"source_id":"S021","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S019","target_id":"S038","kind":"supports"},{"source_id":"S020","target_id":"S038","kind":"supports"},{"source_id":"S041","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S042","kind":"supports"},{"source_id":"S010","target_id":"S045","kind":"supports"},{"source_id":"S012","target_id":"S045","kind":"supports"},{"source_id":"S037","target_id":"S046","kind":"supports"},{"source_id":"S042","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S049","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S028","target_id":"S047","kind":"supports"},{"source_id":"S037","target_id":"S047","kind":"supports"},{"source_id":"S035","target_id":"S047","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S044","kind":"supports"},{"source_id":"S049","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S027","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S027","edges":[{"source_id":"S001","target_id":"S027","kind":"supports"},{"source_id":"S002","target_id":"S027","kind":"supports"},{"source_id":"S003","target_id":"S027","kind":"supports"},{"source_id":"S001","target_id":"S010","kind":"supports"},{"source_id":"S003","target_id":"S010","kind":"supports"},{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S028","kind":"supports"},{"source_id":"S008","target_id":"S028","kind":"supports"},{"source_id":"S003","target_id":"S030","kind":"supports"},{"source_id":"S090","target_id":"S030","kind":"supports"},{"source_id":"S091","target_id":"S030","kind":"supports"},{"source_id":"S006","target_id":"S031","kind":"supports"},{"source_id":"S008","target_id":"S031","kind":"supports"},{"source_id":"S092","target_id":"S031","kind":"supports"},{"source_id":"S012","target_id":"S032","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S030","target_id":"S034","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S032","target_id":"S029","kind":"supports"},{"source_id":"S033","target_id":"S029","kind":"supports"},{"source_id":"S034","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S085","kind":"supports"},{"source_id":"S093","target_id":"S087","kind":"supports"},{"source_id":"S006","target_id":"S087","kind":"supports"},{"source_id":"S008","target_id":"S087","kind":"supports"},{"source_id":"S087","target_id":"S088","kind":"supports"},{"source_id":"S008","target_id":"S088","kind":"supports"},{"source_id":"S087","target_id":"S089","kind":"supports"},{"source_id":"S006","target_id":"S089","kind":"supports"},{"source_id":"S008","target_id":"S089","kind":"supports"},{"source_id":"S087","target_id":"S086","kind":"supports"},{"source_id":"S088","target_id":"S086","kind":"supports"},{"source_id":"S089","target_id":"S086","kind":"supports"},{"source_id":"S086","target_id":"S085","kind":"supports"},{"source_id":"S085","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S092","target_id":"S024","kind":"supports"},{"source_id":"S093","target_id":"S024","kind":"supports"},{"source_id":"S085","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S026","target_id":"S036","kind":"supports"},{"source_id":"S023","target_id":"S036","kind":"supports"},{"source_id":"S024","target_id":"S036","kind":"supports"},{"source_id":"S025","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S015","target_id":"S052","kind":"supports"},{"source_id":"S093","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S005","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S005","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S058","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S055","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S055","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S063","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S060","kind":"supports"},{"source_id":"S063","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S007","target_id":"S066","kind":"supports"},{"source_id":"S008","target_id":"S066","kind":"supports"},{"source_id":"S093","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S050","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S059","target_id":"S068","kind":"supports"},{"source_id":"S066","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S094","target_id":"S069","kind":"supports"},{"source_id":"S005","target_id":"S069","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"},{"source_id":"S067","target_id":"S065","kind":"supports"},{"source_id":"S068","target_id":"S065","kind":"supports"},{"source_id":"S069","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S016","target_id":"S072","kind":"supports"},{"source_id":"S007","target_id":"S072","kind":"supports"},{"source_id":"S093","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S006","target_id":"S073","kind":"supports"},{"source_id":"S093","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S071","kind":"supports"},{"source_id":"S073","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S070","kind":"supports"},{"source_id":"S028","target_id":"S076","kind":"supports"},{"source_id":"S006","target_id":"S076","kind":"supports"},{"source_id":"S007","target_id":"S076","kind":"supports"},{"source_id":"S008","target_id":"S076","kind":"supports"},{"source_id":"S093","target_id":"S076","kind":"supports"},{"source_id":"S076","target_id":"S077","kind":"supports"},{"source_id":"S091","target_id":"S077","kind":"supports"},{"source_id":"S077","target_id":"S078","kind":"supports"},{"source_id":"S095","target_id":"S078","kind":"supports"},{"source_id":"S096","target_id":"S078","kind":"supports"},{"source_id":"S076","target_id":"S079","kind":"supports"},{"source_id":"S077","target_id":"S079","kind":"supports"},{"source_id":"S078","target_id":"S079","kind":"supports"},{"source_id":"S076","target_id":"S075","kind":"supports"},{"source_id":"S077","target_id":"S075","kind":"supports"},{"source_id":"S078","target_id":"S075","kind":"supports"},{"source_id":"S079","target_id":"S075","kind":"supports"},{"source_id":"S075","target_id":"S074","kind":"supports"},{"source_id":"S074","target_id":"S039","kind":"supports"},{"source_id":"S070","target_id":"S039","kind":"supports"},{"source_id":"S008","target_id":"S039","kind":"supports"},{"source_id":"S070","target_id":"S017","kind":"supports"},{"source_id":"S008","target_id":"S017","kind":"supports"},{"source_id":"S006","target_id":"S017","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S018","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S040","target_id":"S020","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S016","target_id":"S020","kind":"supports"},{"source_id":"S074","target_id":"S082","kind":"supports"},{"source_id":"S017","target_id":"S082","kind":"supports"},{"source_id":"S019","target_id":"S082","kind":"supports"},{"source_id":"S020","target_id":"S082","kind":"supports"},{"source_id":"S007","target_id":"S082","kind":"supports"},{"source_id":"S082","target_id":"S083","kind":"supports"},{"source_id":"S017","target_id":"S083","kind":"supports"},{"source_id":"S074","target_id":"S084","kind":"supports"},{"source_id":"S082","target_id":"S084","kind":"supports"},{"source_id":"S083","target_id":"S084","kind":"supports"},{"source_id":"S082","target_id":"S081","kind":"supports"},{"source_id":"S083","target_id":"S081","kind":"supports"},{"source_id":"S084","target_id":"S081","kind":"supports"},{"source_id":"S081","target_id":"S080","kind":"supports"},{"source_id":"S080","target_id":"S041","kind":"supports"},{"source_id":"S074","target_id":"S041","kind":"supports"},{"source_id":"S021","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S019","target_id":"S038","kind":"supports"},{"source_id":"S020","target_id":"S038","kind":"supports"},{"source_id":"S041","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S042","kind":"supports"},{"source_id":"S010","target_id":"S045","kind":"supports"},{"source_id":"S012","target_id":"S045","kind":"supports"},{"source_id":"S037","target_id":"S046","kind":"supports"},{"source_id":"S042","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S049","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S028","target_id":"S047","kind":"supports"},{"source_id":"S037","target_id":"S047","kind":"supports"},{"source_id":"S035","target_id":"S047","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S044","kind":"supports"},{"source_id":"S049","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S027","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:cvx","eq:ncvx"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"半正定优化与因子化问题","source_units":[{"raw_tex":"Consider minimizing a convex objective $\\phi$ over an $n\\times n$\npositive semidefinite matrix $M\\succeq0$, as in\n\\begin{equation}\nM^{\\star}=\\quad\\mathrm{minimize}\\quad\\phi(M)\\quad\\text{over }\\quad M\\succeq0,\\tag{P}\\label{eq:cvx}\n\\end{equation}\nIn practice, the matrix order $n$ is often so large---from tens\nof thousands to hundreds of millions---that even explicitly forming\nthe $n^{2}$ elements of the $n\\times n$ matrix variable $M$ would\nbe intractable. Where $r^{\\star}=\\rank(M^{\\star})$ is known a priori\nto be small, the standard approach is the nonconvex Burer--Monteiro\nfactorization~\\citep{burer2003nonlinear}, which rewrites $M=XX^{T}$\nand then directly optimizes over $X$ using a local optimization algorithm:\n\\begin{equation}\nX^{\\star}=\\quad\\mathrm{minimize}\\quad f(X)\\eqdef\\phi(XX^{T})\\text{ where }X\\text{ is }n\\times r\\text{ and }r\\ge r^{\\star}.\\tag{BM}\\label{eq:ncvx}\n\\end{equation}\n","start":5847,"end":6748}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"伪局部极小与计算动机","source_units":[{"raw_tex":"This way, the semidefinite constraint $M=XX^{T}\\succeq0$ is automatically\nenforced for free. Moreover, by using a small search rank $r\\ll n$,\nthe number of variables is reduced to $O(n)$. However, this improved\nscalability comes at the cost of giving up the convexity of (\\ref{eq:cvx}).\nIn principle, local optimization can fail by getting stuck at a \\emph{spurious}\nlocal minimum of (\\ref{eq:ncvx})---a local minimum that is strictly\nworse than that of the global minimum. In practice, however, there\nseems to be many situations where this failure mode does not occur~\\citep{candes2015phase,rosen2019se,zhang2019spurious}.\n","start":6748,"end":7372}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"全域光滑与强凸假设","source_units":[{"raw_tex":"To state this precisely, suppose that $\\phi$ is twice-differentiable,\n$L$-smooth, and $\\mu$-strongly convex over the entire set of $n\\times n$\nreal symmetric matrices $\\S^{n}$, as in\n\\[\n\\mu\\|E\\|_{F}^{2}\\le\\inner{\\nabla^{2}\\phi(M)[E]}E\\le L\\|E\\|_{F}^{2}\\qquad\\text{for all }M\\in\\S^{n},\n\\]\nwhere $\\inner EF\\eqdef\\tr(E^{T}F)$ and $\\|E\\|_{F}\\eqdef\\sqrt{\\inner EE}$\ndenote the matrix Euclidean inner product and norm respectively. \\citet{bhojanapalli2016global}\n","start":7706,"end":8163}]},{"id":"S005","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"矩阵、锥与正部记号","source_units":[{"raw_tex":"\\textbf{Basic linear algebra.} Lower-case letters are vectors and\nupper-case letters are matrices. We use ``MATLAB notation'' in concatenating\nvectors and matrices:\n\n\\[\n[a,b]=\\begin{bmatrix}a & b\\end{bmatrix},\\qquad[a;b]=\\begin{bmatrix}a\\\\\nb\n\\end{bmatrix},\\qquad\\diag(a,b)=\\begin{bmatrix}a & 0\\\\\n0 & b\n\\end{bmatrix},\n\\]\nand the following short-hand to construct them:\n\\[\n[x_{i}]_{i=1}^{n}=\\begin{bmatrix}x_{1}\\\\\n\\vdots\\\\\nx_{n}\n\\end{bmatrix},\\qquad[x_{i,j}]_{i,j=1}^{m,n}=\\begin{bmatrix}x_{1,1} & \\cdots & x_{1,n}\\\\\n\\vdots & \\ddots & \\vdots\\\\\nx_{m,1} & \\cdots & x_{m,n}\n\\end{bmatrix}.\n\\]\nDenote $\\one=[1,1,\\dots,1]^{T}$ as the vector-of-ones and $I=\\diag(\\one)$\nas the identity matrix; we will infer their dimensions from context.\nDenote $\\langle X,Y\\rangle=\\tr(X^{T}Y)$ and $\\|X\\|_{F}^{2}=\\langle X,X\\rangle$\nas the Frobenius inner product and norm. Denote $\\nnz(x)$ as the\nnumber of nonzero elements in $x$. The sets $\\S^{n}\\supset\\S_{+}^{n}$\nare the $n\\times n$ real symmetric matrices, and the corresponding\npositive semidefinite cone.\n\n\\textbf{Positive cones and projections.} The sets $\\R^{n}\\supset\\R_{+}^{n}$\nare the $n$ vectors, and the corresponding positive orthant. Denote\n$x_{+}=\\max\\{0,x\\}=\\arg\\min\\{\\|x-y\\|:y\\in\\R_{+}^{n}\\}$ as the projection\nonto the positive orthant. \n","start":32917,"end":34202}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"向量化与Kronecker积","source_units":[{"raw_tex":"\n\\textbf{Vectorization and Kronecker product.} Denote $\\vect(X)$ as\nthe usual column-stacking vectorizating operator, and $\\mat(x)$ as\nits inverse (the dimensions of the matrix are inferred from context).\nThe Kronecker product $\\otimes$ is defined to satisfy the identity\n$\\vect(AXB^{T})=(B\\otimes A)\\vect(X)$.\n\n","start":34202,"end":34514}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"Moore–Penrose伪逆","source_units":[{"raw_tex":"\\textbf{Pseudoinverse.} Denote the (Moore--Penrose) pseudoinverse\n$A^{\\dagger}\\eqdef VS^{-1}U^{T}$ of matrix $A=USV^{T}$ where $U^{T}U=V^{T}V=I_{r}$\nand $S\\succ0$. Define $0^{\\dagger}\\eqdef0$.\n\n","start":34514,"end":34708}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"误差向量与因子Jacobian","source_units":[{"raw_tex":"\\textbf{Error vector $\\e$ and Jacobian matrix $\\J_{X}$}. Given fixed\n$X\\in\\R^{n\\times r}$ and $Z\\in\\R^{n\\times r}$, we denote $\\e=\\vect(XX^{T}-ZZ^{T})$\nand implicitly define $\\J_{X}\\in\\R^{n^{2}\\times nr}$ to satisfy $\\J_{X}\\vect(V)=\\vect(XV^{T}+VX^{T})$.\nNote that the associate adjoint operator satisfies $\\J_{X}^{T}\\vect(M)=\\vect((M+M^{T})X)$. \n","start":34708,"end":35055}]},{"id":"S009","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"排除良态过参数化反例","source_units":[{"raw_tex":"We will establish ``no spurious local minima'' guarantees by showing\nthat there cannot exist a counterexample that is simultaneously highly\noverparameterized, with a large value of $r/r^{\\star}$, and also\nwell-conditioned, with a small value of $\\kappa$. \n","start":35109,"end":35365}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"条件数受限反例","source_units":[{"raw_tex":"\\begin{definition}\nThe function $\\phi:\\S^{n}\\to\\R$ is said to be a $(\\kappa,r,r^{\\star})$-\\emph{counterexample}\nif it satisfies the following:\n\\begin{itemize}\n\\item The function $\\phi$ is twice-differentiable, $L$-smooth, $\\mu$-strongly\nconvex, and $L/\\mu\\le\\kappa.$\n\\item The minimizer $M^{\\star}=\\arg\\min_{M\\succeq0}\\phi(M)$ satisfies $\\rank(M^{\\star})=r^{\\star}\\le r$.\n\\item There exists an $n\\times r$ spurious point $X$ with $XX^{T}\\ne M^{\\star}$\nthat satisfies $\\nabla f(X)=0$ and $\\nabla^{2}f(X)\\succeq0$ for\n$f(U)\\eqdef\\phi(UU^{T})$. \n\\end{itemize}\n\\end{definition}\n","start":35365,"end":35939}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["eq:mu_reform"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"两阶段条件数优化","source_units":[{"raw_tex":"\nTo prove that $\\kappa^{\\star}$ is equal to the value stated in (\\ref{eq:inf_kappa}),\nwe proceed by reformulating (\\ref{eq:inf_kappa}) into two-stage optimization\nproblem \n\\begin{align*}\n\\kappa^{\\star} & =\\inf_{X,Z\\in\\R^{n\\times r}}\\{\\kappa(X,Z):\\rank(Z)=r^{\\star},\\;XX^{T}\\ne ZZ^{T}\\}\n\\end{align*}\nin which $\\kappa(X,Z)$ corresponds to the best-conditioned counterexample\nassociated with a \\emph{fixed} minimizer $ZZ^{T}$ and a \\emph{fixed\n}spurious point $X$:\n\\begin{equation}\n\\kappa(X,Z)=\\inf_{\\begin{subarray}{c}\n\\phi:\\S^{n}\\to\\R\\end{subarray}}\\left\\{ \\kappa:\\begin{array}{c}\n\\phi\\text{ is }1\\text{-strongly convex and }\\kappa\\text{-smooth,}\\\\\nZZ^{T}=\\arg\\min_{M\\succeq0}\\phi(M),\\\\\n\\nabla f(X)=0,\\;\\nabla^{2}f(X)\\succeq0\\text{ where }f(U)\\eqdef\\phi(UU^{T}).\n\\end{array}\\right\\} \\label{eq:mu_reform}\n\\end{equation}\n","start":36689,"end":37507}]},{"id":"S015","semantic_type":"external_fact","kind":"external_fact","labels":["eq:Ruhe"],"member_ids":["S015"],"declared_external":true,"declaration_text":["原始前提列表第3项；源文引用ruhe1970perturbation及marshall2011inequalities第9章。"],"source_only":false,"title":"Ruhe迹不等式","source_units":[{"raw_tex":"Our proof of \\thmref{EckartYoung} follows a trace inequality of Ruhe~\\citep{ruhe1970perturbation}\n(see also \\citep[Chapter~9]{marshall2011inequalities}).\\footnote{This simplified proof via Ruhe's inequality is due to an anonymous\nreviewer.} The inequality states that, for $n\\times n$ positive semidefinite\nmatrices $A,B$, we have\n\\begin{equation}\n\\sum_{i=1}^{n}\\lambda_{i}(A)\\lambda_{n-i+1}(B)\\le\\inner AB\\le\\sum_{i=1}^{n}\\lambda_{i}(A)\\lambda_{i}(B)\\label{eq:Ruhe}\n\\end{equation}\nwhere the eigenvalues are ordered $\\lambda_{1}(A)\\ge\\cdots\\ge\\lambda_{n}(A)$\nand $\\lambda_{1}(B)\\ge\\cdots\\ge\\lambda_{n}(B)$. \n","start":51731,"end":52339}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":["eq:Jxdef"],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"Jacobian及伴随的再陈述","source_units":[{"raw_tex":"\\begin{equation}\n\\J_{X}\\vect(V)=\\vect(XV^{T}+VX^{T}),\\qquad\\J_{X}^{T}\\vect(M)=\\vect((M+M^{T})X)\\label{eq:Jxdef}\n\\end{equation}","start":60030,"end":60156}]},{"id":"S017","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:e2"],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"误差的垂直投影","source_units":[{"raw_tex":"\\begin{equation}\n\\mat((I-\\J_{X}\\J_{X}^{\\dagger})\\e)=(I-XX^{\\dagger})(XX^{T}-ZZ^{T})(I-XX^{\\dagger})=-Z_{\\perp}Z_{\\perp}^{T}\\label{eq:e2}\n\\end{equation}","start":62940,"end":63091}]},{"id":"S018","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S018"],"declared_external":true,"declaration_text":["原始前提列表第6项明确接受B0102的初等分块迹和Jacobian恒等式，无需单独证明。"],"source_only":false,"title":"分块迹与Jacobian恒等式","source_units":[{"raw_tex":"\\begin{claim}\nLet $W=\\sum_{j=1}^{m}\\vect(V_{j})\\vect(V_{j})^{T}$ where $V_{1},\\dots,V_{m}\\in\\R^{n\\times r}$.\nPartition $W$ into $r\\times r$ blocks of $n\\times n$, as in $W=[W_{i,j}]_{i,j=1}^{r}$\nfor $W_{i,j}\\in\\R^{n\\times n}$. Then, \n\\[\n\\sum_{i=1}^{r}W_{i,i}=\\sum_{j=1}^{m}V_{j}V_{j}^{T},\\qquad\\inner{\\J_{X}^{T}\\J_{X}}W=\\sum_{j=1}^{m}\\|XV_{j}^{T}+V_{j}X^{T}\\|_{F}^{2}.\n\\]\n\\end{claim}","start":63317,"end":63700}]},{"id":"S019","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:Wsol1"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"候选W的对角块和","source_units":[{"raw_tex":"\\begin{gather}\n\\sum_{i=1}^{r}W_{i,i}=\\gamma_{2}\\cdot\\sum_{j=1}^{r}Z_{\\perp}e_{i}v_{r}^{T}v_{r}e_{i}^{T}Z_{\\perp}^{T}=\\gamma_{2}\\cdot Z_{\\perp}\\left(\\sum_{i=1}^{r}e_{i}e_{i}^{T}\\right)Z_{\\perp}^{T}=\\gamma_{2}\\cdot Z_{\\perp}Z_{\\perp}^{T}.\\label{eq:Wsol1}\n\\end{gather}","start":64405,"end":64670}]},{"id":"S020","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:Wsol2"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"候选W的Jacobian代价","source_units":[{"raw_tex":"\\begin{align}\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle & =\\gamma_{2}\\cdot\\sum_{i=1}^{m}\\|\\J_{X}\\vect(Z_{\\perp}e_{i}v_{r}^{T})\\|^{2}=\\gamma_{2}\\cdot\\sum_{i=1}^{m}\\|Xv_{r}e_{i}^{T}Z_{\\perp}^{T}+Z_{\\perp}e_{i}v_{r}^{T}X^{T}\\|_{F}^{2}\\nonumber \\\\\n & =\\gamma_{2}\\cdot\\sum_{i=1}^{m}2\\|Z_{\\perp}e_{i}v_{r}^{T}X^{T}\\|_{F}^{2}=2\\gamma_{2}\\cdot\\lambda_{\\min}(X^{T}X)\\|Z_{\\perp}\\|_{F}^{2}\\label{eq:Wsol2}\n\\end{align}","start":64689,"end":65086}]},{"id":"S021","semantic_type":"external_fact","kind":"external_fact","labels":["lem:optim"],"member_ids":["S021"],"declared_external":true,"declaration_text":["原始前提列表第4项；源文归于zhang2019sharp的Theorem 12证明中使用的标量恒等式。"],"source_only":false,"title":"标量分式最大化恒等式","source_units":[{"raw_tex":"Finally, we evoke the following lemma, which was previously used in\nthe proof of \\citet[Theorem~12]{zhang2019sharp} to fully solve the\nrank-1 case in closed-form.\n\\begin{lemma}\n\\label{lem:optim}Given $0\\le\\alpha<1$ and $\\beta\\ge0$, let $\\gamma_{\\alpha,\\beta}:\\R_{+}\\to[0,1]$\nsatisfy $\\gamma_{\\alpha,\\beta}(t)=\\alpha\\cdot(t/\\beta)+\\sqrt{1-\\alpha^{2}}\\sqrt{1-t^{2}/\\beta^{2}}$\nfor $t\\le\\alpha\\beta$. Then, we have\n\\[\n\\max_{0\\le t\\le\\alpha\\beta}\\frac{1+\\gamma_{\\alpha,\\beta}(t)}{2t+1-\\gamma_{\\alpha,\\beta}(t)}=\\begin{cases}\n{\\displaystyle \\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}}} & \\text{if }\\beta\\ge{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\\\\\n{\\displaystyle \\frac{1-\\alpha\\beta}{\\beta(\\alpha-\\beta)}} & \\text{if }\\beta\\le{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}}.\n\\end{cases}\n\\]\n\\end{lemma}\n","start":68018,"end":68842}]},{"id":"S022","semantic_type":"definition","kind":"definition","labels":["eq:kappaub_prob"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"反例构造所用上界SDP","source_units":[{"raw_tex":"Finally, we give a constructive proof for the heuristic upper-bound\nin \\lemref{heuristic}. First, recall from \\lemref{sdp_formu} that\nour SDP upper-bound is defined as the following\n\\begin{equation}\n\\kappa_{\\ub}(X,Z)=\\min_{\\kappa,\\H}\\left\\{ \\kappa:I\\preceq\\H\\preceq\\kappa I,\\:\\J_{X}^{T}\\H\\e=0,\\;-2I_{r}\\otimes\\mat(\\H\\e)\\preceq\\J_{X}^{T}\\H\\J_{X}\\right\\} \\label{eq:kappaub_prob}\n\\end{equation}\nwhere $\\e=\\vect(XX^{T}-ZZ^{T})$ and its Jacobian $\\J_{X}$ is defined\nto satisfy $\\J_{X}\\vect(V)=\\vect(XV^{T}+VX^{T})$ for all $V$. Below,\nwe derive a set of sufficient conditions for a given $(\\kappa,\\H)$\nto be feasible for (\\ref{eq:kappaub_prob}). \n","start":69136,"end":69778}]},{"id":"S023","semantic_type":"definition","kind":"definition","labels":["exa:overparam"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"显式二次反例的基构造","source_units":[{"raw_tex":"\\begin{example}[$\\kappa=1+2\\sqrt{r/r^{\\star}}$ counterexample]\n\\label{exa:overparam}Given $r,r^{\\star},n$ satisfying $1\\le r^{\\star}\\le r<n$,\nlet $u_{1},u_{2},\\ldots,u_{n}$ be an orthonormal basis for $\\R^{n}$.\nDefine the quadratic function\n\\[\n\\phi_{0}(M)=\\frac{1}{2}\\sum_{i=1}^{n}\\sum_{j=1}^{n}\\left|\\langle A^{(i,j)},M-M^{\\star}\\rangle\\right|^{2}\\quad\\text{ where }M^{\\star}=\\sum_{k=r+1}^{r+r^{\\star}}u_{k}u_{k}^{T},\n\\]\nand $A^{(i,j)}=\\sqrt{\\kappa}\\cdot u_{i}u_{j}^{T}$ for all $1\\le i,j\\le n$\nexcept the following\n\\begin{gather*}\nA^{(1,1)}=\\sqrt{\\frac{\\kappa}{2r}}\\sum_{j=1}^{r}u_{j}u_{j}^{T}+\\sqrt{\\frac{\\kappa}{2r^{\\star}}}\\sum_{k=r+1}^{r+r^{\\star}}u_{k}u_{k}^{T},\\\\\nA^{(2,2)}=\\frac{1}{\\sqrt{2r}}\\sum_{j=1}^{r}u_{j}u_{j}^{T}-\\frac{1}{\\sqrt{2r^{\\star}}}\\sum_{k=r+1}^{r+r^{\\star}}u_{k}u_{k}^{T},\\\\\nA^{(i,i)}=u_{\\pi(i)}u_{\\pi(i)}^{T}-\\sum_{j=1}^{i-1}\\frac{A^{(j,j)}}{\\|A^{(j,j)}\\|_{F}^{2}}\\inner{A^{(j,j)}}{u_{\\pi(i)}u_{\\pi(i)}^{T}}\\text{ for }i\\in\\{3,\\dots,r+r^{\\star}\\},\n\\end{gather*}\nwith the ordering $\\pi=(1,r+1,2,3,\\dots,r,r+2,r+3,\\dots,r+r^{\\star})$.\n","start":73342,"end":74402}]},{"id":"S024","semantic_type":"derived_fact","kind":"derived_fact","labels":["exa:overparam"],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"反例的谱界与正则性","source_units":[{"raw_tex":"Note that $\\{A^{(i,j)}\\}$ is an orthogonal basis for $\\R^{n\\times n}$\nwith $1\\le\\|A^{(i,j)}\\|_{F}^{2}\\le\\kappa$, so the quadratic function\n$\\phi_{0}$ is twice-differentiable, $1$-strongly convex and $\\kappa$-smooth,\n\\[\n\\|E\\|_{F}^{2}\\le\\inner{\\nabla^{2}\\phi_{0}(M)[E]}E\\le\\kappa\\|E\\|_{F}^{2}\\qquad\\text{for all }M\\in\\R^{n\\times n}.\n\\]\n","start":74402,"end":74736}]},{"id":"S025","semantic_type":"derived_fact","kind":"derived_fact","labels":["exa:overparam"],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"伪二阶点及严格目标间隙","source_units":[{"raw_tex":"Its minimizer $M^{\\star}=\\arg\\min_{M}\\phi_{0}(M)$ satisfies $r^{\\star}=\\rank(M^{\\star}),$\nbut its corresponding $f_{0}(U)\\eqdef\\phi_{0}(UU^{T})$ admits the\n$n\\times r$ matrix $X=\\frac{1}{\\sqrt{1+\\sqrt{r/r^{\\star}}}}[u_{1},\\dots,u_{r}]$\nas a second-order critical point \n\\begin{align*}\nf_{0}(X)-\\min_{U}f_{0}(U) & =\\left(1+\\frac{1}{1+\\sqrt{r^{\\star}/r}}\\right)\\frac{\\|M^{\\star}\\|_{F}^{2}}{2}, & \\nabla f_{0}(X) & =0, & \\nabla^{2}f_{0}(X) & \\succeq0.\n\\end{align*}\n \\qed\n\\end{example}","start":74736,"end":75217}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S026"],"declared_external":false,"source_only":true,"title":"用新正交基实现特征条件","source_units":[{"raw_tex":"We now state an explicit counterexample that satisfies the conditions\nin \\lemref{counter}. Without loss of generality, take $u_{1},\\dots,u_{n}$\nto be the standard basis for $\\R^{n}$. The counterexample is constructed\nby taking the rescaled standard basis $A^{(i,j)}=\\sqrt{\\kappa}u_{i}u_{j}^{T}$\nfor $\\R^{n\\times n}$, and then replacing the elements $\\{A^{(i,i)}\\}_{i=1}^{n}$\nwith a new orthogonal basis for $\\diag(\\R^{n})$ whose first two elements\n$A^{(1,1)}$ and $A^{(2,2)}$ are exactly the first two eigenvectors\nin Lemma 6.1. The ordering $\\pi$ for the Gram--Schmidt orthogonalization\nbelow is chosen specifically to ensure that $\\mathrm{span}\\{A^{(i,i)}\\}_{i=1}^{n}=\\diag(\\R^{n})$. \n","start":72655,"end":73342}]},{"id":"S027","semantic_type":"theorem","kind":"theorem","labels":["thm:main"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"秩过参数化的尖锐阈值","source_units":[{"raw_tex":"\\begin{theorem}[Overparameterization]\n\\label{thm:main}Let $\\phi:\\S^{n}\\to\\R$ be twice-differentiable,\n$L$-smooth and $\\mu$-strongly convex, let the minimizer $M^{\\star}=\\arg\\min_{M\\succeq0}\\phi(M)$\nhave true rank $r^{\\star}=\\rank(M^{\\star})$.\n\\begin{itemize}\n\\item (Sufficiency) If $r>\\frac{1}{4}(L/\\mu-1)^{2}\\,r^{\\star}$ and $r^{\\star}\\le r<n$,\nthen the Burer--Monteiro function $f:\\R^{n\\times r}\\to\\R$ defined\nas $f(U)\\eqdef\\phi(UU^{T})$ has no spurious local minima:\n\\[\n\\nabla f(X)=0,\\quad\\nabla^{2}f(X)\\succeq0\\quad\\iff\\quad f(X)=\\min_{U}f(U).\n\\]\n\\item (Necessity) If $r\\le\\frac{1}{4}(L/\\mu-1)^{2}\\,r^{\\star}$ and $r^{\\star}\\le r<n$,\nthen there exists an $L$-smooth and $\\mu$-strongly convex quadratic\ncounterexample $\\phi_{0}$ whose minimizer $M_{0}^{\\star}=\\arg\\min_{M\\succeq0}\\phi_{0}(M)$\nhas $\\rank(M_{0}^{\\star})=r^{\\star}$, but whose $f_{0}(U)\\eqdef\\phi_{0}(UU^{T})$\nadmits an $n\\times r$ spurious second-order point $X_{\\spur}$:\n\\[\n\\nabla f_{0}(X_{\\spur})=0,\\quad\\nabla^{2}f_{0}(X_{\\spur})\\succeq0,\\quad f_{0}(X_{\\spur})-\\min_{U}f_{0}(U)>\\frac{\\mu}{2}\\cdot\\|M_{0}^{\\star}\\|_{F}^{2}.\n\\]\n\\end{itemize}\n\\end{theorem}","start":12631,"end":13755}]},{"id":"S028","semantic_type":"lemma","kind":"lemma","labels":["lem:sdp_formu"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"固定因子的SDP上下界","source_units":[{"raw_tex":"\\begin{lemma}[SDP formulation]\n\\label{lem:sdp_formu}Define $\\kappa(X,Z)$ as in (\\ref{eq:mu_reform}).\nWe have $\\kappa_{\\ub}(X,Z)\\ge\\kappa(X,Z)\\ge\\kappa_{\\lb}(X,Z)$ where\n\\begin{gather*}\n\\kappa_{\\ub}(X,Z)\\eqdef\\min_{\\kappa,\\H}\\left\\{ \\kappa:I\\preceq\\H\\preceq\\kappa I,\\;\\J_{X}^{T}\\H\\e=0,\\;-2I_{r}\\otimes\\mat(\\H\\e)\\preceq\\J_{X}^{T}\\H\\J_{X}\\right\\} \\\\\n\\kappa_{\\lb}(X,Z)\\eqdef\\min_{\\kappa,s,\\H}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H\\preceq\\kappa I,\\quad\\mat(s)\\succeq0,\\quad\\J_{Z}^{T}s=0,\\\\\n\\J_{X}^{T}(\\H\\e+s)=0,\\quad-2I_{r}\\otimes\\mat(\\H\\e+s)\\preceq\\kappa\\J_{X}^{T}\\J_{X}\n\\end{array}\\right\\} \n\\end{gather*}\nand $\\e=\\vect(XX^{T}-ZZ^{T})$ and $\\J_{U}\\vect(V)=\\vect(UV^{T}+VU^{T})$\nfor all $V$.\n\\end{lemma}","start":37973,"end":38675}]},{"id":"S029","semantic_type":"proof","kind":"proof","labels":["eq:cond","eq:kappa_sdp1","eq:kappasdp2","eq:kktZZ","eq:vecdf"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"SDP上下界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nDenote $S_{M}$ as the convex gradient $\\nabla\\phi(M)$ evaluated\nat $M$, and denote $\\H_{M}$ as the matrix representation of the\nconvex Hessian operator $\\nabla^{2}\\phi(M)$ evaluated at $M$. It\nfollows from $\\inner{\\nabla^{2}\\phi(M)[E]}E=\\vect(E)^{T}\\H_{M}\\vect(E)$\nthat: \n\\begin{equation}\n\\phi\\text{ is }1\\text{-strongly convex and }\\kappa\\text{-smooth}\\iff I\\preceq\\H_{M}\\preceq\\kappa I\\quad\\text{for all }M\\in\\S^{n}.\\label{eq:cond}\n\\end{equation}\nAdditionally, the Karush--Kuhn--Tucker conditions at $M^{\\star}=ZZ^{T}$\nread:\n\\begin{equation}\nZZ^{T}=\\arg\\min_{M\\succeq0}\\phi(M)\\quad\\iff\\quad S_{ZZ^{T}}\\succeq0,\\quad S_{ZZ^{T}}Z=0.\\label{eq:kktZZ}\n\\end{equation}\nNote that strong duality holds in the above because $ZZ^{T}+\\epsilon I$\nis a strictly feasible point with a bounded objective due to $\\kappa$-smoothness.\nNext, we observe that the two gradient evaluations $S_{XX^{T}}$ and\n$S_{ZZ^{T}}$ are related via the fundamental theorem of calculus:\n\\begin{gather*}\n\\nabla\\phi(XX^{T})=\\nabla\\phi(ZZ^{T})+\\int_{0}^{1}\\nabla^{2}\\phi((1-t)ZZ^{T}+tXX^{T})[XX^{T}-ZZ^{T}]\\,\\d t\\\\\n\\iff\\quad\\vect(S_{XX^{T}})=\\vect(S_{ZZ^{T}})+\\left[\\int_{0}^{1}\\H_{M(t)}\\,\\d t\\right]\\vect(XX^{T}-ZZ^{T}).\n\\end{gather*}\nDenoting $s=\\vect(S_{ZZ^{T}})$ and $\\H_{\\av}=\\int_{0}^{1}\\H_{M(t)}\\,\\d t$\nand $\\e=\\vect(XX^{T}-ZZ^{T})$ allows us to rewrite the directional\nderivatives of $f(U)\\eqdef\\phi(UU^{T})$ as follows: \n\\begin{align}\n\\inner{\\nabla f(X)}V & =\\vect(V)^{T}\\J_{X}^{T}(\\H_{\\av}\\e+s),\\nonumber \\\\\n\\inner{\\nabla^{2}f(X)[V]}V & =\\vect(V)^{T}[2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)+\\J_{X}^{T}\\H_{XX^{T}}\\J_{X}]\\,\\vect(V).\\label{eq:vecdf}\n\\end{align}\nSubstituting (\\ref{eq:cond}), (\\ref{eq:vecdf}), and (\\ref{eq:kktZZ})\ninto (\\ref{eq:mu_reform}) yields exactly\n\\begin{equation}\n\\kappa(X,Z)=\\inf_{\\kappa,\\phi:\\S^{n}\\to\\R}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H_{M}\\preceq\\kappa I\\quad\\text{for all }M\\in\\S^{n},\\\\\n\\mat(s)\\succeq0,\\quad\\J_{Z}^{T}s=0,\\quad\\J_{X}^{T}(\\H_{\\av}\\e+s)=0,\\\\\n-2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)\\preceq\\J_{X}^{T}\\H_{XX^{T}}\\J_{X}.\n\\end{array}\\right\\} \\label{eq:kappa_sdp1}\n\\end{equation}\nWe obtain the upper-bound $\\kappa_{\\ub}(X,Z)$ from $\\kappa(X,Z)$\nin  (\\ref{eq:kappa_sdp1}) by fixing $\\H_{M}\\equiv\\H$ for all $M$,\nand $S_{ZZ^{T}}\\equiv\\mat(s)=0$. We obtain the lower-bound $\\kappa_{\\lb}(X,Z)$\nfrom $\\kappa(X,Z)$ in (\\ref{eq:kappa_sdp1}) by substituting the\nrelaxation \n\\[\n-2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)\\quad\\preceq\\quad\\J_{X}^{T}\\H_{XX^{T}}\\J_{X}\\quad\\preceq\\quad\\kappa\\cdot\\J_{X}^{T}\\J_{X},\n\\]\nin order to avoid explicitly optimizing over $\\H_{XX^{T}}$:\n\\begin{align}\n\\kappa(X,Z) & \\ge\\inf_{\\kappa,\\phi:\\S^{n}\\to\\R}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H_{M}\\preceq\\kappa I\\quad\\text{for all }M\\in\\S^{n},\\\\\n\\mat(s)\\succeq0,\\quad\\J_{Z}^{T}s=0,\\quad\\J_{X}^{T}(\\H_{\\av}\\e+s)=0,\\\\\n-2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)\\preceq\\kappa\\cdot\\J_{X}^{T}\\J_{X}.\n\\end{array}\\right\\} \\label{eq:kappasdp2}\n\\end{align}\nFinally, we impose $I\\preceq\\H_{M}\\preceq\\kappa I$ over the average\nHessian $\\H_{\\av}=\\int_{0}^{1}\\H_{M(t)}\\,\\d t$ and then relax this\nconstraint over $\\H_{M}$ for all other values of $M$.\\qed\n\\end{proof}","start":38676,"end":41797}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":["eq:cond","eq:kktZZ"],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"谱夹界与最优点KKT","source_units":[{"raw_tex":"\\begin{proof}\nDenote $S_{M}$ as the convex gradient $\\nabla\\phi(M)$ evaluated\nat $M$, and denote $\\H_{M}$ as the matrix representation of the\nconvex Hessian operator $\\nabla^{2}\\phi(M)$ evaluated at $M$. It\nfollows from $\\inner{\\nabla^{2}\\phi(M)[E]}E=\\vect(E)^{T}\\H_{M}\\vect(E)$\nthat: \n\\begin{equation}\n\\phi\\text{ is }1\\text{-strongly convex and }\\kappa\\text{-smooth}\\iff I\\preceq\\H_{M}\\preceq\\kappa I\\quad\\text{for all }M\\in\\S^{n}.\\label{eq:cond}\n\\end{equation}\nAdditionally, the Karush--Kuhn--Tucker conditions at $M^{\\star}=ZZ^{T}$\nread:\n\\begin{equation}\nZZ^{T}=\\arg\\min_{M\\succeq0}\\phi(M)\\quad\\iff\\quad S_{ZZ^{T}}\\succeq0,\\quad S_{ZZ^{T}}Z=0.\\label{eq:kktZZ}\n\\end{equation}\nNote that strong duality holds in the above because $ZZ^{T}+\\epsilon I$\nis a strictly feasible point with a bounded objective due to $\\kappa$-smoothness.\n","start":38676,"end":39508}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":["eq:vecdf"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"平均Hessian与因子导数","source_units":[{"raw_tex":"Next, we observe that the two gradient evaluations $S_{XX^{T}}$ and\n$S_{ZZ^{T}}$ are related via the fundamental theorem of calculus:\n\\begin{gather*}\n\\nabla\\phi(XX^{T})=\\nabla\\phi(ZZ^{T})+\\int_{0}^{1}\\nabla^{2}\\phi((1-t)ZZ^{T}+tXX^{T})[XX^{T}-ZZ^{T}]\\,\\d t\\\\\n\\iff\\quad\\vect(S_{XX^{T}})=\\vect(S_{ZZ^{T}})+\\left[\\int_{0}^{1}\\H_{M(t)}\\,\\d t\\right]\\vect(XX^{T}-ZZ^{T}).\n\\end{gather*}\nDenoting $s=\\vect(S_{ZZ^{T}})$ and $\\H_{\\av}=\\int_{0}^{1}\\H_{M(t)}\\,\\d t$\nand $\\e=\\vect(XX^{T}-ZZ^{T})$ allows us to rewrite the directional\nderivatives of $f(U)\\eqdef\\phi(UU^{T})$ as follows: \n\\begin{align}\n\\inner{\\nabla f(X)}V & =\\vect(V)^{T}\\J_{X}^{T}(\\H_{\\av}\\e+s),\\nonumber \\\\\n\\inner{\\nabla^{2}f(X)[V]}V & =\\vect(V)^{T}[2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)+\\J_{X}^{T}\\H_{XX^{T}}\\J_{X}]\\,\\vect(V).\\label{eq:vecdf}\n\\end{align}\n","start":39508,"end":40313}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":["eq:kappa_sdp1"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"反例条件的精确重写","source_units":[{"raw_tex":"Substituting (\\ref{eq:cond}), (\\ref{eq:vecdf}), and (\\ref{eq:kktZZ})\ninto (\\ref{eq:mu_reform}) yields exactly\n\\begin{equation}\n\\kappa(X,Z)=\\inf_{\\kappa,\\phi:\\S^{n}\\to\\R}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H_{M}\\preceq\\kappa I\\quad\\text{for all }M\\in\\S^{n},\\\\\n\\mat(s)\\succeq0,\\quad\\J_{Z}^{T}s=0,\\quad\\J_{X}^{T}(\\H_{\\av}\\e+s)=0,\\\\\n-2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)\\preceq\\J_{X}^{T}\\H_{XX^{T}}\\J_{X}.\n\\end{array}\\right\\} \\label{eq:kappa_sdp1}\n\\end{equation}\n","start":40313,"end":40772}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"限制为零最优梯度二次函数","source_units":[{"raw_tex":"We obtain the upper-bound $\\kappa_{\\ub}(X,Z)$ from $\\kappa(X,Z)$\nin  (\\ref{eq:kappa_sdp1}) by fixing $\\H_{M}\\equiv\\H$ for all $M$,\nand $S_{ZZ^{T}}\\equiv\\mat(s)=0$. We obtain the lower-bound $\\kappa_{\\lb}(X,Z)$\n","start":40772,"end":40982}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":["eq:kappasdp2"],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"光滑性放松得到下界SDP","source_units":[{"raw_tex":"and $S_{ZZ^{T}}\\equiv\\mat(s)=0$. We obtain the lower-bound $\\kappa_{\\lb}(X,Z)$\nfrom $\\kappa(X,Z)$ in (\\ref{eq:kappa_sdp1}) by substituting the\nrelaxation \n\\[\n-2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)\\quad\\preceq\\quad\\J_{X}^{T}\\H_{XX^{T}}\\J_{X}\\quad\\preceq\\quad\\kappa\\cdot\\J_{X}^{T}\\J_{X},\n\\]\nin order to avoid explicitly optimizing over $\\H_{XX^{T}}$:\n\\begin{align}\n\\kappa(X,Z) & \\ge\\inf_{\\kappa,\\phi:\\S^{n}\\to\\R}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H_{M}\\preceq\\kappa I\\quad\\text{for all }M\\in\\S^{n},\\\\\n\\mat(s)\\succeq0,\\quad\\J_{Z}^{T}s=0,\\quad\\J_{X}^{T}(\\H_{\\av}\\e+s)=0,\\\\\n-2I_{r}\\otimes\\mat(\\H_{\\av}\\e+s)\\preceq\\kappa\\cdot\\J_{X}^{T}\\J_{X}.\n\\end{array}\\right\\} \\label{eq:kappasdp2}\n\\end{align}\nFinally, we impose $I\\preceq\\H_{M}\\preceq\\kappa I$ over the average\nHessian $\\H_{\\av}=\\int_{0}^{1}\\H_{M(t)}\\,\\d t$ and then relax this\nconstraint over $\\H_{M}$ for all other values of $M$.\\qed\n\\end{proof}","start":40903,"end":41797}]},{"id":"S035","semantic_type":"lemma","kind":"lemma","labels":["lem:heuristic"],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"正交因子的启发式上界","source_units":[{"raw_tex":"\\begin{lemma}[Heuristic upper-bound]\n\\label{lem:heuristic}Let $[Q_{1},Q_{2}]$ have orthonormal columns\nwith $Q_{1}\\in\\R^{n\\times r}$ and $Q_{2}\\in\\R^{n\\times r^{\\star}}$.\nThen, we have\n\\[\n\\kappa_{\\ub}(X,Z)\\le1+2\\sqrt{r/r^{\\star}}\\quad\\text{where }X=Q_{1},\\quad Z=\\sqrt{1+\\sqrt{r/r^{\\star}}}Q_{2}.\n\\]\n\\end{lemma}","start":43032,"end":43343}]},{"id":"S036","semantic_type":"proof","kind":"proof","labels":["exa:overparam"],"member_ids":["S036"],"declared_external":false,"source_only":true,"title":"启发式上界的构造性作者证明","source_units":[{"raw_tex":"We now state an explicit counterexample that satisfies the conditions\nin \\lemref{counter}. Without loss of generality, take $u_{1},\\dots,u_{n}$\nto be the standard basis for $\\R^{n}$. The counterexample is constructed\nby taking the rescaled standard basis $A^{(i,j)}=\\sqrt{\\kappa}u_{i}u_{j}^{T}$\nfor $\\R^{n\\times n}$, and then replacing the elements $\\{A^{(i,i)}\\}_{i=1}^{n}$\nwith a new orthogonal basis for $\\diag(\\R^{n})$ whose first two elements\n$A^{(1,1)}$ and $A^{(2,2)}$ are exactly the first two eigenvectors\nin Lemma 6.1. The ordering $\\pi$ for the Gram--Schmidt orthogonalization\nbelow is chosen specifically to ensure that $\\mathrm{span}\\{A^{(i,i)}\\}_{i=1}^{n}=\\diag(\\R^{n})$. \n\\begin{example}[$\\kappa=1+2\\sqrt{r/r^{\\star}}$ counterexample]\n\\label{exa:overparam}Given $r,r^{\\star},n$ satisfying $1\\le r^{\\star}\\le r<n$,\nlet $u_{1},u_{2},\\ldots,u_{n}$ be an orthonormal basis for $\\R^{n}$.\nDefine the quadratic function\n\\[\n\\phi_{0}(M)=\\frac{1}{2}\\sum_{i=1}^{n}\\sum_{j=1}^{n}\\left|\\langle A^{(i,j)},M-M^{\\star}\\rangle\\right|^{2}\\quad\\text{ where }M^{\\star}=\\sum_{k=r+1}^{r+r^{\\star}}u_{k}u_{k}^{T},\n\\]\nand $A^{(i,j)}=\\sqrt{\\kappa}\\cdot u_{i}u_{j}^{T}$ for all $1\\le i,j\\le n$\nexcept the following\n\\begin{gather*}\nA^{(1,1)}=\\sqrt{\\frac{\\kappa}{2r}}\\sum_{j=1}^{r}u_{j}u_{j}^{T}+\\sqrt{\\frac{\\kappa}{2r^{\\star}}}\\sum_{k=r+1}^{r+r^{\\star}}u_{k}u_{k}^{T},\\\\\nA^{(2,2)}=\\frac{1}{\\sqrt{2r}}\\sum_{j=1}^{r}u_{j}u_{j}^{T}-\\frac{1}{\\sqrt{2r^{\\star}}}\\sum_{k=r+1}^{r+r^{\\star}}u_{k}u_{k}^{T},\\\\\nA^{(i,i)}=u_{\\pi(i)}u_{\\pi(i)}^{T}-\\sum_{j=1}^{i-1}\\frac{A^{(j,j)}}{\\|A^{(j,j)}\\|_{F}^{2}}\\inner{A^{(j,j)}}{u_{\\pi(i)}u_{\\pi(i)}^{T}}\\text{ for }i\\in\\{3,\\dots,r+r^{\\star}\\},\n\\end{gather*}\nwith the ordering $\\pi=(1,r+1,2,3,\\dots,r,r+2,r+3,\\dots,r+r^{\\star})$.\nNote that $\\{A^{(i,j)}\\}$ is an orthogonal basis for $\\R^{n\\times n}$\nwith $1\\le\\|A^{(i,j)}\\|_{F}^{2}\\le\\kappa$, so the quadratic function\n$\\phi_{0}$ is twice-differentiable, $1$-strongly convex and $\\kappa$-smooth,\n\\[\n\\|E\\|_{F}^{2}\\le\\inner{\\nabla^{2}\\phi_{0}(M)[E]}E\\le\\kappa\\|E\\|_{F}^{2}\\qquad\\text{for all }M\\in\\R^{n\\times n}.\n\\]\nIts minimizer $M^{\\star}=\\arg\\min_{M}\\phi_{0}(M)$ satisfies $r^{\\star}=\\rank(M^{\\star}),$\nbut its corresponding $f_{0}(U)\\eqdef\\phi_{0}(UU^{T})$ admits the\n$n\\times r$ matrix $X=\\frac{1}{\\sqrt{1+\\sqrt{r/r^{\\star}}}}[u_{1},\\dots,u_{r}]$\nas a second-order critical point \n\\begin{align*}\nf_{0}(X)-\\min_{U}f_{0}(U) & =\\left(1+\\frac{1}{1+\\sqrt{r^{\\star}/r}}\\right)\\frac{\\|M^{\\star}\\|_{F}^{2}}{2}, & \\nabla f_{0}(X) & =0, & \\nabla^{2}f_{0}(X) & \\succeq0.\n\\end{align*}\n \\qed\n\\end{example}\n","start":72655,"end":75218}]},{"id":"S037","semantic_type":"lemma","kind":"lemma","labels":["lem:abdef"],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"条件数下界的闭式表达","source_units":[{"raw_tex":"\\begin{lemma}[Closed-form lower-bound]\n\\label{lem:abdef}For $X,Z\\in\\R^{n\\times r}$ such that $XX^{T}\\ne ZZ^{T}$,\nlet $Z_{\\perp}=(I-XX^{\\dagger})Z$. We have\n\\begin{gather*}\n\\kappa_{\\lb}(X,Z)\\ge\\begin{cases}\n{\\displaystyle \\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}}} & \\text{if }\\beta\\ge{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\\\\\n{\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}} & \\text{if }\\beta\\le{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\n\\end{cases}\\\\\n\\text{ where }\\qquad\\alpha=\\frac{\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}{\\|XX^{T}-ZZ^{T}\\|_{F}},\\qquad\\beta=\\frac{\\lambda_{\\min}(X^{T}X)}{\\|XX^{T}-ZZ^{T}\\|_{F}}{\\displaystyle \\frac{\\tr(Z_{\\perp}Z_{\\perp}^{T})}{\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}}.\n\\end{gather*}\n\\end{lemma}","start":43659,"end":44416}]},{"id":"S038","semantic_type":"proof","kind":"proof","labels":["eq:Wsol","eq:Wsol1","eq:Wsol2","eq:e2","eq:ysol"],"member_ids":["S038"],"declared_external":false,"source_only":true,"title":"闭式下界的完整散文证明","source_units":[{"raw_tex":"Our heuristic choice of $y$ and $W_{i,j}$ is motivated by the optimal\nsolution to the much simpler rank-1 case previously derived by \\citet{zhang2019sharp}.\nThis choice is not necessarily optimal for the rank-$r$ case, but\nit does remain simple enough for the final maximization over $t\\ge0$\nin \\lemref{tradeoff} to be solved in closed-form. \n\nWe begin by recalling the core insights used to derive the rank-1\nsolution. Our goal of maximizing $\\e^{T}(\\J_{X}y-w)$ where $w=\\sum_{i=1}^{r}\\vect(W_{i,i})$\ncan equivalently be understood as minimizing $\\|\\J_{X}y-w-\\e\\|$ over\n$y$ and $w$, subject to $\\mat(w)\\succeq0$. This is a projection\nproblem, so a basic idea is to decompose $\\e$ into orthogonal components:\n\\begin{equation}\n\\J_{X}y=\\gamma_{1}\\cdot(\\J_{X}\\J_{X}^{\\dagger})\\e,\\qquad w=-\\gamma_{2}\\cdot(I-\\J_{X}\\J_{X}^{\\dagger})\\e.\\label{eq:ysol}\n\\end{equation}\nWe verify that $\\mat(w)\\succeq0$ indeed holds for $\\gamma_{2}\\ge0$\nvia \\lemref{I-JJ+}\n\\begin{equation}\n\\mat((I-\\J_{X}\\J_{X}^{\\dagger})\\e)=(I-XX^{\\dagger})(XX^{T}-ZZ^{T})(I-XX^{\\dagger})=-Z_{\\perp}Z_{\\perp}^{T}\\label{eq:e2}\n\\end{equation}\nwhere we have written $Z_{\\perp}=(I-XX^{\\dagger})Z$. \n\nEquation (\\ref{eq:ysol}) specifies a unique choice of $y$, but leaves\nseveral possible valid choices of $W_{i,j}$. We need the following\nclaim to make a concrete choice.\n","start":61993,"end":63317},{"raw_tex":"Following the claim, the obvious first attempt is to pick $W=\\gamma_{2}\\cdot\\vect(Z_{\\perp})\\vect(Z_{\\perp})^{T}$.\nIndeed, this yields $\\sum_{i=1}^{r}W_{i,i}=\\gamma_{2}\\cdot Z_{\\perp}Z_{\\perp}^{T}$\nas specified in (\\ref{eq:ysol}), and is in fact optimal for the rank-1\ncase. In the rank-$r$ case, however, the choice of $W$ can be improved\nby making $\\inner{\\J_{X}^{T}\\J_{X}}W$ as small as possible. A more\nsubtle second attempt is to pick \n\\begin{equation}\nW=\\gamma_{2}\\cdot\\sum_{j=1}^{r}\\vect(Z_{\\perp}e_{j}v_{r}^{T})\\vect(Z_{\\perp}e_{j}v_{r}^{T})^{T}\\label{eq:Wsol}\n\\end{equation}\nwhere $v_{r}$ is the $r$-th unit eigenvector of $X^{T}X$. Indeed,\nwe can verify that (\\ref{eq:ysol}) is also satisfied:\n\\begin{gather}\n\\sum_{i=1}^{r}W_{i,i}=\\gamma_{2}\\cdot\\sum_{j=1}^{r}Z_{\\perp}e_{i}v_{r}^{T}v_{r}e_{i}^{T}Z_{\\perp}^{T}=\\gamma_{2}\\cdot Z_{\\perp}\\left(\\sum_{i=1}^{r}e_{i}e_{i}^{T}\\right)Z_{\\perp}^{T}=\\gamma_{2}\\cdot Z_{\\perp}Z_{\\perp}^{T}.\\label{eq:Wsol1}\n\\end{gather}\nMoreover, we have\n\\begin{align}\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle & =\\gamma_{2}\\cdot\\sum_{i=1}^{m}\\|\\J_{X}\\vect(Z_{\\perp}e_{i}v_{r}^{T})\\|^{2}=\\gamma_{2}\\cdot\\sum_{i=1}^{m}\\|Xv_{r}e_{i}^{T}Z_{\\perp}^{T}+Z_{\\perp}e_{i}v_{r}^{T}X^{T}\\|_{F}^{2}\\nonumber \\\\\n & =\\gamma_{2}\\cdot\\sum_{i=1}^{m}2\\|Z_{\\perp}e_{i}v_{r}^{T}X^{T}\\|_{F}^{2}=2\\gamma_{2}\\cdot\\lambda_{\\min}(X^{T}X)\\|Z_{\\perp}\\|_{F}^{2}\\label{eq:Wsol2}\n\\end{align}\nWe now plug our choice of $y$ and $W_{i,j}$ specified in (\\ref{eq:ysol})\nand (\\ref{eq:Wsol}) into \\lemref{tradeoff}, along with a good choice\nof coefficients $\\gamma_{1}$ and $\\gamma_{2}$.\n","start":63701,"end":65276},{"raw_tex":"The proof of \\lemref{abdef} immediately follows by substituting \\lemref{costheta}\ninto \\lemref{tradeoff} and then solving the resulting optimization\nover $t$ using \\lemref{optim}.\n","start":68842,"end":69022}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":["eq:e2","eq:ysol"],"member_ids":["S039"],"declared_external":false,"source_only":true,"title":"按Jacobian像空间分解误差","source_units":[{"raw_tex":"Our heuristic choice of $y$ and $W_{i,j}$ is motivated by the optimal\nsolution to the much simpler rank-1 case previously derived by \\citet{zhang2019sharp}.\nThis choice is not necessarily optimal for the rank-$r$ case, but\nit does remain simple enough for the final maximization over $t\\ge0$\nin \\lemref{tradeoff} to be solved in closed-form. \n\nWe begin by recalling the core insights used to derive the rank-1\nsolution. Our goal of maximizing $\\e^{T}(\\J_{X}y-w)$ where $w=\\sum_{i=1}^{r}\\vect(W_{i,i})$\ncan equivalently be understood as minimizing $\\|\\J_{X}y-w-\\e\\|$ over\n$y$ and $w$, subject to $\\mat(w)\\succeq0$. This is a projection\nproblem, so a basic idea is to decompose $\\e$ into orthogonal components:\n\\begin{equation}\n\\J_{X}y=\\gamma_{1}\\cdot(\\J_{X}\\J_{X}^{\\dagger})\\e,\\qquad w=-\\gamma_{2}\\cdot(I-\\J_{X}\\J_{X}^{\\dagger})\\e.\\label{eq:ysol}\n\\end{equation}\nWe verify that $\\mat(w)\\succeq0$ indeed holds for $\\gamma_{2}\\ge0$\nvia \\lemref{I-JJ+}\n\\begin{equation}\n\\mat((I-\\J_{X}\\J_{X}^{\\dagger})\\e)=(I-XX^{\\dagger})(XX^{T}-ZZ^{T})(I-XX^{\\dagger})=-Z_{\\perp}Z_{\\perp}^{T}\\label{eq:e2}\n\\end{equation}\nwhere we have written $Z_{\\perp}=(I-XX^{\\dagger})Z$. \n\nEquation (\\ref{eq:ysol}) specifies a unique choice of $y$, but leaves\nseveral possible valid choices of $W_{i,j}$. We need the following\nclaim to make a concrete choice.\n","start":61993,"end":63317}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":["eq:Wsol"],"member_ids":["S040"],"declared_external":false,"source_only":true,"title":"以最小特征向量降低证书代价","source_units":[{"raw_tex":"Following the claim, the obvious first attempt is to pick $W=\\gamma_{2}\\cdot\\vect(Z_{\\perp})\\vect(Z_{\\perp})^{T}$.\nIndeed, this yields $\\sum_{i=1}^{r}W_{i,i}=\\gamma_{2}\\cdot Z_{\\perp}Z_{\\perp}^{T}$\nas specified in (\\ref{eq:ysol}), and is in fact optimal for the rank-1\ncase. In the rank-$r$ case, however, the choice of $W$ can be improved\nby making $\\inner{\\J_{X}^{T}\\J_{X}}W$ as small as possible. A more\nsubtle second attempt is to pick \n\\begin{equation}\nW=\\gamma_{2}\\cdot\\sum_{j=1}^{r}\\vect(Z_{\\perp}e_{j}v_{r}^{T})\\vect(Z_{\\perp}e_{j}v_{r}^{T})^{T}\\label{eq:Wsol}\n\\end{equation}\nwhere $v_{r}$ is the $r$-th unit eigenvector of $X^{T}X$. Indeed,\nwe can verify that (\\ref{eq:ysol}) is also satisfied:\n","start":63701,"end":64405}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S041"],"declared_external":false,"source_only":true,"title":"代入对偶并完成标量最大化","source_units":[{"raw_tex":"We now plug our choice of $y$ and $W_{i,j}$ specified in (\\ref{eq:ysol})\nand (\\ref{eq:Wsol}) into \\lemref{tradeoff}, along with a good choice\nof coefficients $\\gamma_{1}$ and $\\gamma_{2}$.\n","start":65087,"end":65276},{"raw_tex":"The proof of \\lemref{abdef} immediately follows by substituting \\lemref{costheta}\ninto \\lemref{tradeoff} and then solving the resulting optimization\nover $t$ using \\lemref{optim}.\n","start":68842,"end":69022}]},{"id":"S042","semantic_type":"lemma","kind":"lemma","labels":["lem:ab1"],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"捕获真实秩的有效不等式","source_units":[{"raw_tex":"\\begin{lemma}[Valid inequality]\n\\label{lem:ab1}For all $X,Z\\in\\R^{n\\times r}$ with $\\rank(Z)=r^{\\star}$,\nthe parameters $\\alpha$ and $\\beta$ defined in \\Lemref{abdef} satisfy\n$\\alpha^{2}+(r/r^{\\star})\\beta^{2}\\le1+[(\\beta-\\alpha)_{+}]^{2}.$\n\\end{lemma}","start":46119,"end":46371}]},{"id":"S043","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S043"],"declared_external":false,"source_only":true,"title":"由秩上界版本恢复有效不等式","source_units":[{"raw_tex":"Instead of proving \\lemref{ab1}, we will prove the following equivalent\nstatement.\n","start":56042,"end":56125},{"raw_tex":"Assuming the above to be true, setting $p=\\rank(Z)$ yields exactly\n\\lemref{ab1}. Conversely, if the above is true for $p=\\rank(Z)$,\nthen it is obviously true for any $p\\ge\\rank(Z)$, given that the\nleft-hand side decreases monotonically with increasing $p$.\n","start":56541,"end":56798}]},{"id":"S044","semantic_type":"proof","kind":"proof","labels":["eq:minab"],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n[Theorem~\\ref{thm:main}]For a fixed $r$ and $r^{\\star}$, recall\nthat we have defined $\\kappa^{\\star}$ as the condition number of\nthe \\emph{best-conditioned} counterexample, and formulated it as the\noptimal value for a two-stage optimization problem of the following\nform\n\\begin{align*}\n\\kappa^{\\star} & =\\inf_{\\phi:\\S^{n}\\to\\R}\\{\\kappa:\\phi\\text{ is }(\\kappa,r,r^{\\star})\\text{-counterexample}\\}\\\\\n & =\\inf_{X,Z\\in\\R^{n\\times r}}\\{\\kappa(X,Z):\\rank(Z)=r^{\\star},\\;XX^{T}\\ne ZZ^{T}\\}\n\\end{align*}\nWe claim that the minimum of the following expression over $\\alpha,\\beta\\ge0$\n\\[\n\\gamma(\\alpha,\\beta)\\eqdef\\begin{cases}\n\\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}} & \\text{if }\\beta\\ge{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\\\\\n{\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}} & \\text{if }\\beta\\le{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\n\\end{cases}\n\\]\nsubject to the valid inequality from \\lemref{ab1} is given:\n\\begin{equation}\n1+2\\sqrt{r/r^{\\star}}=\\min_{\\alpha,\\beta\\ge0}\\{\\gamma(\\alpha,\\beta):\\alpha^{2}+(r/r^{\\star})\\min\\{\\alpha^{2},\\beta^{2}\\}\\le1\\}.\\label{eq:minab}\n\\end{equation}\nTaking (\\ref{eq:minab}) to be true, substituting into \\lemref{abdef}\nand evoking \\lemref{heuristic} yields\n\\[\n1+2\\sqrt{r/r^{\\star}}\\le\\inf_{\\begin{subarray}{c}\nXX^{T}\\ne ZZ^{T}\\\\\n\\rank(Z)=r^{\\star}\n\\end{subarray}}\\kappa_{\\lb}(X,Z)\\le\\kappa^{\\star}\\le\\kappa_{\\ub}(X,Z)\\le1+2\\sqrt{r/r^{\\star}}.\n\\]\nThe upper-bound proves necessity: if $r/r^{\\star}\\le\\frac{1}{4}(\\kappa-1)^{2}$,\nthen there exists a $(\\kappa,r,r^{\\star})$-counterexample. The lower-bound\nproves sufficiency: if $r>\\frac{1}{4}(\\kappa-1)^{2}\\,r^{\\star}$ and\n$r^{\\star}\\le r<n$, then there exists no $(\\kappa,r,r^{\\star})$-counterexample\nto refute a ``no spurious local minima'' guarantee.\n\nWe now prove the claim in (\\ref{eq:minab}). We first minimize $\\gamma(\\alpha,\\beta)$\nsubject to $\\beta\\ge\\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}$ and find\nvia monotonicity with respect to $\\alpha$ that\n\\[\n\\min_{0\\le\\alpha\\le1}\\left\\{ \\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}}:\\beta\\ge{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}}\\right\\} ={\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}}.\n\\]\nIndeed, the minimizer $\\alpha$ lies at the boundary $\\beta=\\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}=\\frac{1-\\sqrt{1-\\alpha^{2}}}{\\alpha}$,\nwhere\n\\[\n\\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}}=\\frac{\\sqrt{1-\\alpha^{2}}+(1-\\alpha^{2})}{\\sqrt{1-\\alpha^{2}}-(1-\\alpha^{2})}=\\frac{1+\\sqrt{1-\\alpha^{2}}-\\alpha^{2}}{\\alpha^{2}-(1-\\sqrt{1-\\alpha^{2}})}={\\displaystyle \\frac{\\beta^{-1}-\\alpha}{\\alpha-\\beta}.}\n\\]\nNext, we minimize $\\gamma(\\alpha,\\beta)$ subject to $\\beta\\le\\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}\\le\\alpha$\nand the valid inequality:\n\\begin{align*}\n & \\min_{0\\le\\beta\\le\\alpha}\\left\\{ {\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}}:\\alpha^{2}+\\frac{r}{r^{\\star}}\\beta^{2}\\le1\\right\\} \\\\\n= & \\min_{\\beta\\ge0,\\;\\rho\\ge1}\\left\\{ {\\displaystyle \\frac{1-\\rho\\beta^{2}}{(\\rho-1)\\beta^{2}}}:(\\rho^{2}+\\frac{r}{r^{\\star}})\\beta^{2}\\le1\\right\\} =\\min_{\\rho\\ge1}\\left\\{ \\rho+\\frac{r}{r^{\\star}}\\frac{1}{\\rho-1}\\right\\} =1+2\\sqrt{\\frac{r}{r^{\\star}}}.\n\\end{align*}\nWe obtain the second line from the first by substituting $\\alpha=\\rho\\beta$\nfor $\\rho\\ge1$. The minimizers are $\\beta^{2}=(\\rho^{2}+\\frac{r}{r^{\\star}})^{-1}$\nand $\\rho=1+\\sqrt{r/r^{\\star}}$. \\qed\n\\end{proof}","start":46684,"end":50074}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"重述最优反例的两阶段形式","source_units":[{"raw_tex":"\\begin{proof}\n[Theorem~\\ref{thm:main}]For a fixed $r$ and $r^{\\star}$, recall\nthat we have defined $\\kappa^{\\star}$ as the condition number of\nthe \\emph{best-conditioned} counterexample, and formulated it as the\noptimal value for a two-stage optimization problem of the following\nform\n\\begin{align*}\n\\kappa^{\\star} & =\\inf_{\\phi:\\S^{n}\\to\\R}\\{\\kappa:\\phi\\text{ is }(\\kappa,r,r^{\\star})\\text{-counterexample}\\}\\\\\n & =\\inf_{X,Z\\in\\R^{n\\times r}}\\{\\kappa(X,Z):\\rank(Z)=r^{\\star},\\;XX^{T}\\ne ZZ^{T}\\}\n\\end{align*}\n","start":46684,"end":47194}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["eq:minab"],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"标量松弛问题与最小值主张","source_units":[{"raw_tex":"We claim that the minimum of the following expression over $\\alpha,\\beta\\ge0$\n\\[\n\\gamma(\\alpha,\\beta)\\eqdef\\begin{cases}\n\\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}} & \\text{if }\\beta\\ge{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\\\\\n{\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}} & \\text{if }\\beta\\le{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}},\n\\end{cases}\n\\]\nsubject to the valid inequality from \\lemref{ab1} is given:\n\\begin{equation}\n1+2\\sqrt{r/r^{\\star}}=\\min_{\\alpha,\\beta\\ge0}\\{\\gamma(\\alpha,\\beta):\\alpha^{2}+(r/r^{\\star})\\min\\{\\alpha^{2},\\beta^{2}\\}\\le1\\}.\\label{eq:minab}\n\\end{equation}\n","start":47194,"end":47826}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"上下界夹逼并解释两方向","source_units":[{"raw_tex":"Taking (\\ref{eq:minab}) to be true, substituting into \\lemref{abdef}\nand evoking \\lemref{heuristic} yields\n\\[\n1+2\\sqrt{r/r^{\\star}}\\le\\inf_{\\begin{subarray}{c}\nXX^{T}\\ne ZZ^{T}\\\\\n\\rank(Z)=r^{\\star}\n\\end{subarray}}\\kappa_{\\lb}(X,Z)\\le\\kappa^{\\star}\\le\\kappa_{\\ub}(X,Z)\\le1+2\\sqrt{r/r^{\\star}}.\n\\]\nThe upper-bound proves necessity: if $r/r^{\\star}\\le\\frac{1}{4}(\\kappa-1)^{2}$,\nthen there exists a $(\\kappa,r,r^{\\star})$-counterexample. The lower-bound\nproves sufficiency: if $r>\\frac{1}{4}(\\kappa-1)^{2}\\,r^{\\star}$ and\n$r^{\\star}\\le r<n$, then there exists no $(\\kappa,r,r^{\\star})$-counterexample\nto refute a ``no spurious local minima'' guarantee.\n","start":47826,"end":48476}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"第一分支移至两段交界","source_units":[{"raw_tex":"\nWe now prove the claim in (\\ref{eq:minab}). We first minimize $\\gamma(\\alpha,\\beta)$\nsubject to $\\beta\\ge\\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}$ and find\nvia monotonicity with respect to $\\alpha$ that\n\\[\n\\min_{0\\le\\alpha\\le1}\\left\\{ \\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}}:\\beta\\ge{\\displaystyle \\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}}\\right\\} ={\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}}.\n\\]\nIndeed, the minimizer $\\alpha$ lies at the boundary $\\beta=\\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}=\\frac{1-\\sqrt{1-\\alpha^{2}}}{\\alpha}$,\nwhere\n\\[\n\\frac{1+\\sqrt{1-\\alpha^{2}}}{1-\\sqrt{1-\\alpha^{2}}}=\\frac{\\sqrt{1-\\alpha^{2}}+(1-\\alpha^{2})}{\\sqrt{1-\\alpha^{2}}-(1-\\alpha^{2})}=\\frac{1+\\sqrt{1-\\alpha^{2}}-\\alpha^{2}}{\\alpha^{2}-(1-\\sqrt{1-\\alpha^{2}})}={\\displaystyle \\frac{\\beta^{-1}-\\alpha}{\\alpha-\\beta}.}\n\\]\n","start":48476,"end":49301}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"第二分支代换并求最小值","source_units":[{"raw_tex":"Next, we minimize $\\gamma(\\alpha,\\beta)$ subject to $\\beta\\le\\frac{\\alpha}{1+\\sqrt{1-\\alpha^{2}}}\\le\\alpha$\nand the valid inequality:\n\\begin{align*}\n & \\min_{0\\le\\beta\\le\\alpha}\\left\\{ {\\displaystyle \\frac{1-\\alpha\\beta}{(\\alpha-\\beta)\\beta}}:\\alpha^{2}+\\frac{r}{r^{\\star}}\\beta^{2}\\le1\\right\\} \\\\\n= & \\min_{\\beta\\ge0,\\;\\rho\\ge1}\\left\\{ {\\displaystyle \\frac{1-\\rho\\beta^{2}}{(\\rho-1)\\beta^{2}}}:(\\rho^{2}+\\frac{r}{r^{\\star}})\\beta^{2}\\le1\\right\\} =\\min_{\\rho\\ge1}\\left\\{ \\rho+\\frac{r}{r^{\\star}}\\frac{1}{\\rho-1}\\right\\} =1+2\\sqrt{\\frac{r}{r^{\\star}}}.\n\\end{align*}\nWe obtain the second line from the first by substituting $\\alpha=\\rho\\beta$\nfor $\\rho\\ge1$. The minimizers are $\\beta^{2}=(\\rho^{2}+\\frac{r}{r^{\\star}})^{-1}$\nand $\\rho=1+\\sqrt{r/r^{\\star}}$. \\qed\n\\end{proof}","start":49301,"end":50074}]},{"id":"S050","semantic_type":"theorem","kind":"theorem","labels":["eq:ey-sol","thm:EckartYoung"],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"正则化Eckart–Young定理","source_units":[{"raw_tex":"\\begin{theorem}[Regularized Eckart--Young]\n\\label{thm:EckartYoung}Given $A\\in\\S_{+}^{n}$ and $B\\in\\S_{+}^{r}$\nwith $r\\le n$, let $A=\\sum_{i=1}^{n}s_{i}u_{i}u_{i}^{T}$ and $B=\\sum_{i=1}^{r}d_{i}v_{i}v_{i}^{T}$\ndenote the usual orthonormal eigendecompositions with $s_{1}\\ge\\cdots\\ge s_{n}\\ge0$\nand $0\\le d_{1}\\le\\cdots\\le d_{r}.$ Then, \n\\begin{align}\n\\min_{Y\\in\\R^{n\\times r}}\\left\\{ \\|A-YY^{T}\\|_{F}^{2}+2\\langle B,Y^{T}Y\\rangle\\right\\}  & =\\sum_{i=1}^{n}s_{i}^{2}-\\sum_{i=1}^{r}[(s_{i}-d_{i})_{+}]^{2}\\label{eq:ey-sol}\n\\end{align}\nwith minimizer $Y^{\\star}=\\sum_{i=1}^{r}u_{i}v_{i}^{T}\\sqrt{(s_{i}-d_{i})_{+}}$\nwhere $(\\cdot)_{+}\\eqdef\\max\\{0,\\cdot\\}$. \n\\end{theorem}","start":50630,"end":51298}]},{"id":"S051","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"正则化逼近的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n[\\thmref{EckartYoung}]Recall that $A=\\sum_{i=1}^{n}s_{i}u_{i}u_{i}^{T}$\nand $B=\\sum_{i=1}^{r}d_{i}v_{i}v_{i}^{T}$ with $s_{1}\\ge\\cdots\\ge s_{n}\\ge0$\nand $0\\le d_{1}\\le\\cdots\\le d_{r}.$ It follows from (\\ref{eq:Ruhe})\nthat\n\\begin{gather*}\n\\inner A{YY^{T}}\\le\\sum_{i=1}^{n}\\lambda_{i}(YY^{T})\\cdot\\lambda_{i}(A)=\\sum_{i=1}^{r}s_{i}\\sigma_{i}^{2}(Y),\\\\\n\\inner B{Y^{T}Y}\\ge\\sum_{i=1}^{r}\\lambda_{i}(Y^{T}Y)\\cdot\\lambda_{n-i+1}(B)=\\sum_{i=1}^{r}d_{i}\\sigma_{i}^{2}(Y).\n\\end{gather*}\nSubstituting yields the following lower-bound, which is valid for\nall choices of $Y$:\n\\begin{align*}\n\\|A-YY^{T}\\|_{F}^{2}+2\\inner B{Y^{T}Y}= & \\|A\\|_{F}^{2}+\\|YY^{T}\\|_{F}^{2}-2\\inner A{YY^{T}}+2\\inner B{Y^{T}Y}\\\\\n\\ge & \\sum_{i=1}^{n}s_{i}^{2}+\\sum_{i=1}^{r}\\sigma_{i}^{4}(Y)-2\\sum_{i=1}^{r}s_{i}\\sigma_{i}^{2}(Y)+2\\sum_{i=1}^{r}d_{i}\\sigma_{i}^{2}(Y)\\\\\n\\ge & \\sum_{i=1}^{n}s_{i}^{2}-\\sum_{i=1}^{r}[(s_{i}-d_{i})_{+}]^{2}.\n\\end{align*}\nThe final line applied the following result as a lower-bound:\n\\begin{align*}\n\\sigma_{i}^{4}(Y)-2(s_{i}-d_{i})\\sigma_{i}^{2}(Y)\\ge & \\min_{x_{i}\\ge0}x_{i}^{2}-2(s_{i}-d_{i})x_{i}\\\\\n= & \\min_{x_{i}\\ge0}[x_{i}-(s_{i}-d_{i})]^{2}-(s_{i}-d_{i})^{2}\\\\\n= & [(s_{i}-d_{i})_{+}-(s_{i}-d_{i})]^{2}-(s_{i}-d_{i})^{2}=-[(s_{i}-d_{i})_{+}]^{2}.\n\\end{align*}\nFinally, we verify that $Y^{\\star}=\\sum_{i=1}^{n}u_{i}v_{i}\\sqrt{(s_{i}-d_{i})_{+}}$\nattains the lower-bound derived above. \\qed\n\\end{proof}","start":52339,"end":53752}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"以Ruhe不等式配对谱值","source_units":[{"raw_tex":"\\begin{proof}\n[\\thmref{EckartYoung}]Recall that $A=\\sum_{i=1}^{n}s_{i}u_{i}u_{i}^{T}$\nand $B=\\sum_{i=1}^{r}d_{i}v_{i}v_{i}^{T}$ with $s_{1}\\ge\\cdots\\ge s_{n}\\ge0$\nand $0\\le d_{1}\\le\\cdots\\le d_{r}.$ It follows from (\\ref{eq:Ruhe})\nthat\n\\begin{gather*}\n\\inner A{YY^{T}}\\le\\sum_{i=1}^{n}\\lambda_{i}(YY^{T})\\cdot\\lambda_{i}(A)=\\sum_{i=1}^{r}s_{i}\\sigma_{i}^{2}(Y),\\\\\n\\inner B{Y^{T}Y}\\ge\\sum_{i=1}^{r}\\lambda_{i}(Y^{T}Y)\\cdot\\lambda_{n-i+1}(B)=\\sum_{i=1}^{r}d_{i}\\sigma_{i}^{2}(Y).\n\\end{gather*}\n","start":52339,"end":52831}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"逐项非负二次最小化","source_units":[{"raw_tex":"Substituting yields the following lower-bound, which is valid for\nall choices of $Y$:\n\\begin{align*}\n\\|A-YY^{T}\\|_{F}^{2}+2\\inner B{Y^{T}Y}= & \\|A\\|_{F}^{2}+\\|YY^{T}\\|_{F}^{2}-2\\inner A{YY^{T}}+2\\inner B{Y^{T}Y}\\\\\n\\ge & \\sum_{i=1}^{n}s_{i}^{2}+\\sum_{i=1}^{r}\\sigma_{i}^{4}(Y)-2\\sum_{i=1}^{r}s_{i}\\sigma_{i}^{2}(Y)+2\\sum_{i=1}^{r}d_{i}\\sigma_{i}^{2}(Y)\\\\\n\\ge & \\sum_{i=1}^{n}s_{i}^{2}-\\sum_{i=1}^{r}[(s_{i}-d_{i})_{+}]^{2}.\n\\end{align*}\nThe final line applied the following result as a lower-bound:\n\\begin{align*}\n\\sigma_{i}^{4}(Y)-2(s_{i}-d_{i})\\sigma_{i}^{2}(Y)\\ge & \\min_{x_{i}\\ge0}x_{i}^{2}-2(s_{i}-d_{i})x_{i}\\\\\n= & \\min_{x_{i}\\ge0}[x_{i}-(s_{i}-d_{i})]^{2}-(s_{i}-d_{i})^{2}\\\\\n= & [(s_{i}-d_{i})_{+}-(s_{i}-d_{i})]^{2}-(s_{i}-d_{i})^{2}=-[(s_{i}-d_{i})_{+}]^{2}.\n\\end{align*}\n","start":52831,"end":53612}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"声明候选达到下界","source_units":[{"raw_tex":"Finally, we verify that $Y^{\\star}=\\sum_{i=1}^{n}u_{i}v_{i}\\sqrt{(s_{i}-d_{i})_{+}}$\nattains the lower-bound derived above. \\qed\n\\end{proof}","start":53612,"end":53752}]},{"id":"S055","semantic_type":"lemma","kind":"lemma","labels":["lem:numcard"],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"正部残差的投影下界","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:numcard}If $x\\ge0$ and $\\one^{T}x\\le\\|x\\|^{2}$, then\n$\\one^{T}\\left(I-xx^{T}/\\|x\\|^{2}\\right)\\one\\ge\\|(\\one-x)_{+}\\|^{2}$.\n\\end{lemma}","start":53893,"end":54052}]},{"id":"S056","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"投影下界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nDefine $u=\\alpha x,$ where $\\alpha=\\one^{T}x/\\|x\\|^{2}$ is chosen\nso that $\\mathbf{1}^{T}u=\\|u\\|^{2}$. First, verify that $\\mathbf{1}^{T}(I-xx^{T}/\\|x\\|^{2})\\mathbf{1}=\\|\\mathbf{1}-u\\|^{2}\\ge\\|(\\mathbf{1}-u)_{+}\\|^{2}$.\nNext, observe that $\\psi(\\alpha)\\equiv[(1-\\alpha t)_{+}]^{2}$ is\na decreasing function of $\\alpha\\ge0$ when $t\\ge0$, and therefore\n$\\|(\\one-\\alpha x)_{+}\\|^{2}\\ge\\|(\\one-x)_{+}\\|^{2}$ for $x\\ge0$\nand $0\\le\\alpha\\le1$. Finally, the hypotheses $x\\ge0$ and $\\one^{T}x\\le\\|x\\|^{2}$\nensure that $0\\le\\alpha\\le1$. \\qed\n\\end{proof}","start":54053,"end":54611}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"缩放至投影恒等式","source_units":[{"raw_tex":"\\begin{proof}\nDefine $u=\\alpha x,$ where $\\alpha=\\one^{T}x/\\|x\\|^{2}$ is chosen\nso that $\\mathbf{1}^{T}u=\\|u\\|^{2}$. First, verify that $\\mathbf{1}^{T}(I-xx^{T}/\\|x\\|^{2})\\mathbf{1}=\\|\\mathbf{1}-u\\|^{2}\\ge\\|(\\mathbf{1}-u)_{+}\\|^{2}$.\n","start":54053,"end":54287}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"正部残差的缩放单调性","source_units":[{"raw_tex":"Next, observe that $\\psi(\\alpha)\\equiv[(1-\\alpha t)_{+}]^{2}$ is\na decreasing function of $\\alpha\\ge0$ when $t\\ge0$, and therefore\n$\\|(\\one-\\alpha x)_{+}\\|^{2}\\ge\\|(\\one-x)_{+}\\|^{2}$ for $x\\ge0$\nand $0\\le\\alpha\\le1$. Finally, the hypotheses $x\\ge0$ and $\\one^{T}x\\le\\|x\\|^{2}$\nensure that $0\\le\\alpha\\le1$. \\qed\n\\end{proof}","start":54287,"end":54611}]},{"id":"S059","semantic_type":"lemma","kind":"lemma","labels":["lem:keyclaim"],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"截断平方差的两段下界","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:keyclaim}Given $s,d\\in\\R_{+}^{n}$, let $s_{i}\\ge s_{\\lb}\\ge0$\nfor all $i$. Then we have\n\\begin{align*}\n\\|s\\|^{2}-\\|(s-d)_{+}\\|^{2}\\ge & \\begin{cases}\ns_{\\lb}^{2}\\frac{(\\one^{T}d)^{2}}{\\|d\\|^{2}} & \\text{if }s_{\\lb}\\one^{T}d\\le\\|d\\|^{2},\\\\\n2s_{\\lb}\\one^{T}d-\\|d\\|^{2} & \\text{if }s_{\\lb}\\one^{T}d\\ge\\|d\\|^{2}.\n\\end{cases}\n\\end{align*}\n\\end{lemma}","start":54613,"end":54983}]},{"id":"S060","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"截断平方差的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nFirst, we observe that $\\psi(t)\\equiv t^{2}-[(t-\\alpha)_{+}]^{2}$\nis an increasing function of $t\\ge0$. It follows from element-wise\nmonotonicity that\n\\[\n\\|s\\|^{2}-\\|(s-d)_{+}\\|^{2}\\ge\\|s_{\\lb}\\one\\|^{2}-\\|(s_{\\lb}\\one-d)_{+}\\|^{2}=s_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-d/s_{\\lb})_{+}\\|^{2}\\right)\n\\]\nIf $s_{\\lb}\\one^{T}d\\le\\|d\\|^{2}$, then write $x=d/s_{\\lb}\\ge0$,\nand observe that $\\one^{T}x\\le\\|x\\|^{2}$ holds. Therefore, applying\n\\lemref{numcard} yields\n\\begin{align*}\ns_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-d/s_{\\lb})_{+}\\|^{2}\\right) & =s_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-x)_{+}\\|^{2}\\right)\\ge s_{\\lb}^{2}\\one^{T}\\left(xx^{T}/\\|x\\|^{2}\\right)\\one\\\\\n & =s_{\\lb}^{2}\\left([\\one^{T}(d/s_{\\lb})]^{2}/\\|d/s_{\\lb}\\|^{2}\\right)=s_{\\lb}^{2}(\\one^{T}d)^{2}/\\|d\\|^{2}.\n\\end{align*}\nIf otherwise $s_{\\lb}\\one^{T}d\\ge\\|d\\|^{2}$, then we have \n\\begin{align*}\ns_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-d/s_{\\lb})_{+}\\|^{2}\\right)\\ge s_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|\\one-d/s_{\\lb}\\|^{2}\\right)=2s_{\\lb}\\one^{T}d-\\|d\\|^{2}.\n\\end{align*}\n\\qed\n\\end{proof}","start":54984,"end":56040}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"以逐项单调性降至常数向量","source_units":[{"raw_tex":"\\begin{proof}\nFirst, we observe that $\\psi(t)\\equiv t^{2}-[(t-\\alpha)_{+}]^{2}$\nis an increasing function of $t\\ge0$. It follows from element-wise\nmonotonicity that\n\\[\n\\|s\\|^{2}-\\|(s-d)_{+}\\|^{2}\\ge\\|s_{\\lb}\\one\\|^{2}-\\|(s_{\\lb}\\one-d)_{+}\\|^{2}=s_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-d/s_{\\lb})_{+}\\|^{2}\\right)\n\\]\n","start":54984,"end":55299}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"第一分支应用投影引理","source_units":[{"raw_tex":"If $s_{\\lb}\\one^{T}d\\le\\|d\\|^{2}$, then write $x=d/s_{\\lb}\\ge0$,\nand observe that $\\one^{T}x\\le\\|x\\|^{2}$ holds. Therefore, applying\n\\lemref{numcard} yields\n\\begin{align*}\ns_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-d/s_{\\lb})_{+}\\|^{2}\\right) & =s_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-x)_{+}\\|^{2}\\right)\\ge s_{\\lb}^{2}\\one^{T}\\left(xx^{T}/\\|x\\|^{2}\\right)\\one\\\\\n & =s_{\\lb}^{2}\\left([\\one^{T}(d/s_{\\lb})]^{2}/\\|d/s_{\\lb}\\|^{2}\\right)=s_{\\lb}^{2}(\\one^{T}d)^{2}/\\|d\\|^{2}.\n\\end{align*}\n","start":55299,"end":55779}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"第二分支去掉正部","source_units":[{"raw_tex":"If otherwise $s_{\\lb}\\one^{T}d\\ge\\|d\\|^{2}$, then we have \n\\begin{align*}\ns_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|(\\one-d/s_{\\lb})_{+}\\|^{2}\\right)\\ge s_{\\lb}^{2}\\left(\\|\\one\\|^{2}-\\|\\one-d/s_{\\lb}\\|^{2}\\right)=2s_{\\lb}\\one^{T}d-\\|d\\|^{2}.\n\\end{align*}\n\\qed\n\\end{proof}","start":55779,"end":56040}]},{"id":"S064","semantic_type":"lemma","kind":"lemma","labels":["lem:ab2"],"member_ids":["S064"],"declared_external":false,"source_only":false,"title":"真实秩上界版本的有效不等式","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:ab2}For $X,Z\\in\\R^{n\\times r}$, define the parameters\n\\[\n\\alpha=\\frac{\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}{\\|XX^{T}-ZZ^{T}\\|_{F}},\\qquad\\beta=\\frac{\\lambda_{\\min}(X^{T}X)}{\\|XX^{T}-ZZ^{T}\\|_{F}}\\cdot\\frac{\\tr(Z_{\\perp}Z_{\\perp}^{T})}{\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}.\n\\]\nThen, $\\alpha^{2}+(r/p)\\beta^{2}\\le1+[(\\beta-\\alpha)_{+}]^{2}$ holds\nfor all $p$ satisfying $\\rank(Z)\\le p\\le r$.\n\\end{lemma}","start":56125,"end":56540}]},{"id":"S065","semantic_type":"proof","kind":"proof","labels":["eq:key-claim","eq:key-claim1","eq:key-claim2","eq:key-claim3","eq:key-claim4"],"member_ids":["S065"],"declared_external":false,"source_only":false,"title":"有效不等式的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nThe two parameters $\\alpha$ and $\\beta$ are unitarily invariant\nwith respect to $X,Z$, so we may assume without loss of generality\nthat $X,Z$ are partitioned as\n\\[\nX=\\begin{bmatrix}X_{1}\\\\\n0\n\\end{bmatrix}\\qquad Z=\\begin{bmatrix}Z_{1}\\\\\nZ_{2}\n\\end{bmatrix}\\qquad\\text{where }X_{1}\\in\\R^{r\\times r},\\;Z_{1}\\in\\R^{r\\times p},\\;Z_{2}\\in\\R^{(n-r)\\times p}.\n\\]\n(Otherwise, take the QR decomposition $X=QR$ and note that $\\alpha,\\beta$\nremain unchanged with $X\\gets Q^{T}X$ and $Z\\gets Q^{T}Z$.) Denote\n$s\\in\\R^{r}$ as the $r$ eigenvalues of $X_{1}X_{1}^{T}$ and $d\\in\\R^{p}$\nas the $p$ eigenvalues of $Z_{2}^{T}Z_{2}$ \n\\begin{align*}\ns_{i}\\eqdef & \\lambda_{i}(X_{1}X_{1}^{T})=\\lambda_{i}(XX^{T}),\\quad s_{1}\\ge s_{2}\\ge\\cdots\\ge s_{r}\\ge0,\\\\\nd_{i}\\eqdef & \\lambda_{i}(Z_{2}^{T}Z_{2})=\\lambda_{i}(Z_{\\perp}Z_{\\perp}^{T}),\\quad0\\le d_{1}\\le d_{2}\\le\\cdots\\le d_{p}.\n\\end{align*}\nWriting $\\e=\\vect(XX^{T}-ZZ^{T})$ yields\n\\begin{align*}\n\\alpha & =\\frac{\\|Z_{2}Z_{2}^{T}\\|_{F}}{\\|XX^{T}-ZZ^{T}\\|_{F}}=\\frac{\\|d\\|_{F}}{\\|\\e\\|_{F}}, & \\beta & =\\frac{\\lambda_{\\min}(X_{1}^{T}X_{1})}{\\|XX^{T}-ZZ^{T}\\|_{F}}\\cdot\\frac{\\tr(Z_{2}Z_{2}^{T})}{\\|Z_{2}Z_{2}^{T}\\|_{F}}=\\frac{s_{r}\\cdot\\one^{T}d}{\\|\\e\\|\\|d\\|}.\n\\end{align*}\nWe expand $\\|XX^{T}-ZZ^{T}\\|_{F}^{2}$ block-wise and evoke our Regularized\nEckart--Young Theorem (\\thmref{EckartYoung}) over the $r\\times p$\nmatrix block $Z_{1}$:\n\\begin{align*}\n\\|XX^{T}-ZZ^{T}\\|_{F}^{2} & =\\|X_{1}X_{1}^{T}-Z_{1}Z_{1}^{T}\\|_{F}^{2}+2\\langle Z_{2}^{T}Z_{2},Z_{1}^{T}Z_{1}\\rangle+\\|Z_{2}Z_{2}^{T}\\|^{2}\\\\\n & \\ge\\min_{Z_{1}\\in\\R^{r\\times p}}\\{\\|X_{1}X_{1}^{T}-Z_{1}Z_{1}^{T}\\|_{F}^{2}+2\\langle Z_{2}^{T}Z_{2},Z_{1}^{T}Z_{1}\\rangle\\}+\\|Z_{2}Z_{2}^{T}\\|^{2}\\\\\n & =\\|s\\|^{2}-\\|(s'-d)_{+}\\|^{2}+\\|d\\|^{2}\n\\end{align*}\nwhere $s'=(s_{1},s_{2},\\dots,s_{p})$. Now, evoking our technical\nlemma (\\lemref{keyclaim}) with $s_{\\lb}\\equiv s_{r}$ yields the\nfollowing two lower-bounds on $\\|XX^{T}-ZZ^{T}\\|_{F}^{2}=\\|\\e\\|^{2}$:\\begin{subequations}\\label{eq:key-claim}\n\\begin{gather}\ns_{r}\\one^{T}d\\le\\|d\\|^{2}\\quad\\implies\\quad\\|\\e\\|^{2}\\ge s_{r}^{2}\\frac{(\\one^{T}d)^{2}}{\\|d\\|^{2}}+(r-p)s_{r}^{2}+\\|d\\|^{2},\\label{eq:key-claim1}\\\\\ns_{r}\\one^{T}d\\ge\\|d\\|^{2}\\quad\\implies\\quad\\|\\e\\|^{2}\\ge2s_{r}\\one^{T}d-\\|d\\|^{2}+(r-p)s_{r}^{2}+\\|d\\|^{2}.\\label{eq:key-claim2}\n\\end{gather}\n\\end{subequations}Substituting $s_{r}=\\beta\\cdot\\|\\e\\|\\|d\\|/(\\one^{T}d)$\nyields\n\\[\n(r-p)s_{r}^{2}=(r-p)\\frac{\\|d\\|^{2}}{(\\one^{T}d)^{2}}\\beta^{2}\\|\\e\\|^{2}\\ge(r-p)\\frac{1}{\\nnz(d)}\\beta^{2}\\|\\e\\|^{2}\\ge\\frac{(r-p)}{p}\\beta^{2}\\|\\e\\|^{2},\n\\]\nand substituting into (\\ref{eq:key-claim}) yields\\begin{subequations}\n\\begin{gather}\n\\beta\\le\\alpha\\quad\\implies\\quad1\\ge\\beta^{2}+\\alpha^{2}+\\frac{(r-p)}{p}\\beta^{2}=\\alpha^{2}+\\frac{r}{p}\\beta^{2},\\label{eq:key-claim3}\\\\\n\\beta\\ge\\alpha\\quad\\implies\\quad1\\ge2\\alpha\\beta+\\frac{(r-p)}{p}\\beta^{2}=\\alpha^{2}+\\frac{r}{p}\\beta^{2}-(\\beta-\\alpha)^{2},\\label{eq:key-claim4}\n\\end{gather}\n\\end{subequations}as desired. \\qed\n\\end{proof}","start":56798,"end":59758}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":["eq:key-claim"],"member_ids":["S066"],"declared_external":false,"source_only":false,"title":"酉不变分块与谱参数","source_units":[{"raw_tex":"\\begin{proof}\nThe two parameters $\\alpha$ and $\\beta$ are unitarily invariant\nwith respect to $X,Z$, so we may assume without loss of generality\nthat $X,Z$ are partitioned as\n\\[\nX=\\begin{bmatrix}X_{1}\\\\\n0\n\\end{bmatrix}\\qquad Z=\\begin{bmatrix}Z_{1}\\\\\nZ_{2}\n\\end{bmatrix}\\qquad\\text{where }X_{1}\\in\\R^{r\\times r},\\;Z_{1}\\in\\R^{r\\times p},\\;Z_{2}\\in\\R^{(n-r)\\times p}.\n\\]\n(Otherwise, take the QR decomposition $X=QR$ and note that $\\alpha,\\beta$\nremain unchanged with $X\\gets Q^{T}X$ and $Z\\gets Q^{T}Z$.) Denote\n$s\\in\\R^{r}$ as the $r$ eigenvalues of $X_{1}X_{1}^{T}$ and $d\\in\\R^{p}$\nas the $p$ eigenvalues of $Z_{2}^{T}Z_{2}$ \n\\begin{align*}\ns_{i}\\eqdef & \\lambda_{i}(X_{1}X_{1}^{T})=\\lambda_{i}(XX^{T}),\\quad s_{1}\\ge s_{2}\\ge\\cdots\\ge s_{r}\\ge0,\\\\\nd_{i}\\eqdef & \\lambda_{i}(Z_{2}^{T}Z_{2})=\\lambda_{i}(Z_{\\perp}Z_{\\perp}^{T}),\\quad0\\le d_{1}\\le d_{2}\\le\\cdots\\le d_{p}.\n\\end{align*}\nWriting $\\e=\\vect(XX^{T}-ZZ^{T})$ yields\n\\begin{align*}\n\\alpha & =\\frac{\\|Z_{2}Z_{2}^{T}\\|_{F}}{\\|XX^{T}-ZZ^{T}\\|_{F}}=\\frac{\\|d\\|_{F}}{\\|\\e\\|_{F}}, & \\beta & =\\frac{\\lambda_{\\min}(X_{1}^{T}X_{1})}{\\|XX^{T}-ZZ^{T}\\|_{F}}\\cdot\\frac{\\tr(Z_{2}Z_{2}^{T})}{\\|Z_{2}Z_{2}^{T}\\|_{F}}=\\frac{s_{r}\\cdot\\one^{T}d}{\\|\\e\\|\\|d\\|}.\n\\end{align*}\n","start":56798,"end":58013}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":["eq:key-claim"],"member_ids":["S067"],"declared_external":false,"source_only":false,"title":"分块误差应用正则化逼近","source_units":[{"raw_tex":"We expand $\\|XX^{T}-ZZ^{T}\\|_{F}^{2}$ block-wise and evoke our Regularized\nEckart--Young Theorem (\\thmref{EckartYoung}) over the $r\\times p$\nmatrix block $Z_{1}$:\n\\begin{align*}\n\\|XX^{T}-ZZ^{T}\\|_{F}^{2} & =\\|X_{1}X_{1}^{T}-Z_{1}Z_{1}^{T}\\|_{F}^{2}+2\\langle Z_{2}^{T}Z_{2},Z_{1}^{T}Z_{1}\\rangle+\\|Z_{2}Z_{2}^{T}\\|^{2}\\\\\n & \\ge\\min_{Z_{1}\\in\\R^{r\\times p}}\\{\\|X_{1}X_{1}^{T}-Z_{1}Z_{1}^{T}\\|_{F}^{2}+2\\langle Z_{2}^{T}Z_{2},Z_{1}^{T}Z_{1}\\rangle\\}+\\|Z_{2}Z_{2}^{T}\\|^{2}\\\\\n & =\\|s\\|^{2}-\\|(s'-d)_{+}\\|^{2}+\\|d\\|^{2}\n\\end{align*}\n","start":58013,"end":58541}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":["eq:key-claim","eq:key-claim1","eq:key-claim2"],"member_ids":["S068"],"declared_external":false,"source_only":false,"title":"按最小特征值取得两段误差界","source_units":[{"raw_tex":"where $s'=(s_{1},s_{2},\\dots,s_{p})$. Now, evoking our technical\nlemma (\\lemref{keyclaim}) with $s_{\\lb}\\equiv s_{r}$ yields the\nfollowing two lower-bounds on $\\|XX^{T}-ZZ^{T}\\|_{F}^{2}=\\|\\e\\|^{2}$:\\begin{subequations}\\label{eq:key-claim}\n\\begin{gather}\ns_{r}\\one^{T}d\\le\\|d\\|^{2}\\quad\\implies\\quad\\|\\e\\|^{2}\\ge s_{r}^{2}\\frac{(\\one^{T}d)^{2}}{\\|d\\|^{2}}+(r-p)s_{r}^{2}+\\|d\\|^{2},\\label{eq:key-claim1}\\\\\ns_{r}\\one^{T}d\\ge\\|d\\|^{2}\\quad\\implies\\quad\\|\\e\\|^{2}\\ge2s_{r}\\one^{T}d-\\|d\\|^{2}+(r-p)s_{r}^{2}+\\|d\\|^{2}.\\label{eq:key-claim2}\n\\end{gather}\n","start":58541,"end":59088}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":["eq:key-claim","eq:key-claim3","eq:key-claim4"],"member_ids":["S069"],"declared_external":false,"source_only":false,"title":"秩计数与归一化得到有效域","source_units":[{"raw_tex":"\\end{subequations}Substituting $s_{r}=\\beta\\cdot\\|\\e\\|\\|d\\|/(\\one^{T}d)$\nyields\n\\[\n(r-p)s_{r}^{2}=(r-p)\\frac{\\|d\\|^{2}}{(\\one^{T}d)^{2}}\\beta^{2}\\|\\e\\|^{2}\\ge(r-p)\\frac{1}{\\nnz(d)}\\beta^{2}\\|\\e\\|^{2}\\ge\\frac{(r-p)}{p}\\beta^{2}\\|\\e\\|^{2},\n\\]\nand substituting into (\\ref{eq:key-claim}) yields\\begin{subequations}\n\\begin{gather}\n\\beta\\le\\alpha\\quad\\implies\\quad1\\ge\\beta^{2}+\\alpha^{2}+\\frac{(r-p)}{p}\\beta^{2}=\\alpha^{2}+\\frac{r}{p}\\beta^{2},\\label{eq:key-claim3}\\\\\n\\beta\\ge\\alpha\\quad\\implies\\quad1\\ge2\\alpha\\beta+\\frac{(r-p)}{p}\\beta^{2}=\\alpha^{2}+\\frac{r}{p}\\beta^{2}-(\\beta-\\alpha)^{2},\\label{eq:key-claim4}\n\\end{gather}\n\\end{subequations}as desired. \\qed\n\\end{proof}","start":59088,"end":59758}]},{"id":"S070","semantic_type":"lemma","kind":"lemma","labels":["lem:I-JJ+"],"member_ids":["S070"],"declared_external":false,"source_only":false,"title":"Jacobian余投影恒等式","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:I-JJ+}For any $X\\in\\R^{n\\times r}$, the matrix $\\J_{X}$\ndefined in (\\ref{eq:Jxdef}) satisfies $I-\\J_{X}\\J_{X}^{\\dagger}=(I-XX^{\\dagger})\\otimes(I-XX^{\\dagger})$\nwhere $\\dagger$ denotes the pseudoinverse. \n\\end{lemma}","start":60213,"end":60454}]},{"id":"S071","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S071"],"declared_external":false,"source_only":false,"title":"余投影恒等式的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $X=QR$ be the usual QR decomposition with orthogonal complement\n$Q_{\\perp}$, so that $XX^{\\dagger}=QQ^{T}$ and $I-XX^{\\dagger}=Q_{\\perp}Q_{\\perp}^{T}$.\nObserve that for any $M\\in\\R^{n\\times n}$:\n\\[\n\\J_{X}^{T}\\vect(M)=0\\iff(M+M^{T})Q=0\\iff M=Q_{\\perp}\\hat{M}Q_{\\perp}^{T}.\n\\]\nVectorizing the final equality as $\\vect(M)=(Q_{\\perp}\\otimes Q_{\\perp})\\vect(\\hat{M})$\nyields \n\\begin{align*}\n(I-\\J_{X}\\J_{X}^{\\dagger})b & =\\arg\\min_{v}\\{\\|v-b\\|:\\J_{X}^{T}v=0\\}\\\\\n & =\\arg\\min_{v}\\{\\|v-b\\|:v=(Q_{\\perp}\\otimes Q_{\\perp})\\hat{v}\\}\\\\\n & =(Q_{\\perp}\\otimes Q_{\\perp})(Q_{\\perp}\\otimes Q_{\\perp})^{T}b=(Q_{\\perp}Q_{\\perp}^{T}\\otimes Q_{\\perp}Q_{\\perp}^{T})b.\\quad\\qed\n\\end{align*}\n\\end{proof}","start":60455,"end":61154}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S072"],"declared_external":false,"source_only":false,"title":"QR分解识别伴随核","source_units":[{"raw_tex":"\\begin{proof}\nLet $X=QR$ be the usual QR decomposition with orthogonal complement\n$Q_{\\perp}$, so that $XX^{\\dagger}=QQ^{T}$ and $I-XX^{\\dagger}=Q_{\\perp}Q_{\\perp}^{T}$.\nObserve that for any $M\\in\\R^{n\\times n}$:\n\\[\n\\J_{X}^{T}\\vect(M)=0\\iff(M+M^{T})Q=0\\iff M=Q_{\\perp}\\hat{M}Q_{\\perp}^{T}.\n\\]\nVectorizing the final equality as $\\vect(M)=(Q_{\\perp}\\otimes Q_{\\perp})\\vect(\\hat{M})$\n","start":60455,"end":60836}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S073"],"declared_external":false,"source_only":false,"title":"向量化核空间的正交投影","source_units":[{"raw_tex":"yields \n\\begin{align*}\n(I-\\J_{X}\\J_{X}^{\\dagger})b & =\\arg\\min_{v}\\{\\|v-b\\|:\\J_{X}^{T}v=0\\}\\\\\n & =\\arg\\min_{v}\\{\\|v-b\\|:v=(Q_{\\perp}\\otimes Q_{\\perp})\\hat{v}\\}\\\\\n & =(Q_{\\perp}\\otimes Q_{\\perp})(Q_{\\perp}\\otimes Q_{\\perp})^{T}b=(Q_{\\perp}Q_{\\perp}^{T}\\otimes Q_{\\perp}Q_{\\perp}^{T})b.\\quad\\qed\n\\end{align*}\n\\end{proof}","start":60836,"end":61154}]},{"id":"S074","semantic_type":"lemma","kind":"lemma","labels":["eq:cosprob","lem:tradeoff"],"member_ids":["S074"],"declared_external":false,"source_only":false,"title":"下界SDP的角度对偶形式","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:tradeoff}We have\n\\[\n\\kappa_{\\lb}(X,Z)=\\max_{t\\ge0}\\frac{1+\\cos\\theta(t)}{2t+1-\\cos\\theta(t)}\n\\]\nwhere $\\cos\\theta(t)$ is defined as follows\n\\begin{equation}\n\\cos\\theta(t)=\\max_{y,W_{i,j}}\\left\\{ \\e^{T}\\f:\\begin{array}{c}\n\\f\\equiv\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\quad\\|\\e\\|\\|\\f\\|=1,\\\\\n(I-ZZ^{\\dagger})\\,\\mat(\\f)\\,(I-ZZ^{\\dagger})\\succeq0,\\\\\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle=2t,\\quad W=[W_{i,j}]_{i,j=1}^{r}\\succeq0,\n\\end{array}\\right\\} \\label{eq:cosprob}\n\\end{equation}\nwith respect to $y\\in\\R^{nr}$ and $W_{i,j}\\in\\R^{n\\times n}$ for\n$i,j\\in\\{1,2,\\dots,r\\}$.\n\\end{lemma}","start":61392,"end":61992}]},{"id":"S075","semantic_type":"proof","kind":"proof","labels":["app:tradeoff","eq:ub2-1"],"member_ids":["S075"],"declared_external":false,"source_only":false,"title":"角度对偶形式的完整作者证明","source_units":[{"raw_tex":"\\section{\\label{app:tradeoff}Derivation of the dual for the lower-bound problem\n(Proof of \\lemref{tradeoff})}\n\nWe will derive the Lagrangian dual for the following\n\\begin{align*}\n\\kappa_{\\lb}(X,Z) & =\\min_{\\kappa,\\H,s}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H\\preceq\\kappa I,\\qquad\\J_{Z}^{T}s=0,\\qquad s\\in\\vect(\\S_{+}^{n}),\\\\\n\\J_{X}^{T}(\\H\\e+s)=0,\\quad2I_{r}\\otimes\\mat(\\H\\e+s)+\\kappa\\J_{X}^{T}\\J_{X}\\succeq0.\n\\end{array}\\right\\} \n\\end{align*}\nRecall that $\\e=\\vect(XX^{T}-ZZ^{T})$ and $\\J_{X}\\vect(V)=\\vect(XV^{T}+VX^{T})$\nwith respect to fixed $X,Z\\in\\R^{n\\times r}$. Observe that\n\\[\n\\J_{Z}^{T}\\vect(S)=0\\iff(S+S^{T})Z=0\\iff S=Q_{\\perp}S_{\\perp}Q_{\\perp}^{T}\\iff\\vect(S)=\\Q_{\\perp}\\vect(S_{\\perp})\n\\]\nwhere the orthogonal complement $Q_{\\perp}$ of $Z$ is such that\n$Q_{\\perp}Q_{\\perp}^{T}=I_{n}-ZZ^{\\dagger}$, and $\\Q_{\\perp}=Q_{\\perp}\\otimes Q_{\\perp}$.\nSubstituting and taking the Lagrangian dual yields\n\\begin{align}\n\\kappa_{\\lb}(X,Z) & =\\min_{\\kappa,\\H,s_{\\perp}}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H\\preceq\\kappa I,\\quad s_{\\perp}\\in\\vect(\\S_{+}^{n-r^{\\star}}),\\quad\\J_{X}^{T}(\\H\\e+\\Q_{\\perp}s_{\\perp})=0,\\\\\n2I_{r}\\otimes\\mat(\\H\\e+\\Q_{\\perp}s_{\\perp})+\\kappa\\J_{X}^{T}\\J_{X}\\succeq0\n\\end{array}\\right\\} \\nonumber \\\\\n & =\\max_{y,W_{i,j},U,V}\\left\\{ \\tr(U):\\begin{array}{c}\n\\f\\e^{T}+\\e\\f^{T}=U-V,\\quad\\tr(V)+\\inner{\\J_{X}^{T}\\J_{X}}W=1,\\\\\n\\f\\equiv\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\quad\\Q_{\\perp}^{T}\\f\\in\\vect(\\S_{+}^{n-r^{\\star}}),\\\\\nU,V\\succeq0,\\quad W=[W_{i,j}]_{i,j=1}^{r}\\succeq0\n\\end{array}\\right\\} \\label{eq:ub2-1}\n\\end{align}\nin which $y\\in\\R^{nr}$ and $W_{i,j}\\in\\R^{n\\times n}$ for $i,j\\in\\{1,2,\\dots,r\\}$.\nStrong duality holds here because $y=0,$ $W=\\epsilon\\cdot I_{nr},$\n$U=\\frac{1}{n}I_{n},$ and $V=\\frac{1}{n}I_{n}+\\epsilon\\cdot r\\cdot[\\e\\vect(I_{n})^{T}+\\e\\vect(I_{n})^{T}]$\nis strictly feasible for a sufficiently small $\\epsilon>0$. \n\nThe dual problem (\\ref{eq:ub2-1}) has a closed-form solution over\n$U$ and $V$ \\citep[Lemma~13]{zhang2019sharp}\n\\begin{align*}\n\\frac{\\tr([M]_{-})+\\alpha}{\\tr([M]_{+})}= & \\min_{t,U,V}\\left\\{ \\tr(V)+\\alpha\\cdot t:\\begin{array}{c}\ntM=U-V,\\quad t\\ge0,\\\\\n\\tr(U)=1,\\quad U,V\\succeq0\n\\end{array}\\right\\} ,\n\\end{align*}\nand $\\f\\e^{T}+\\e\\f^{T}$ has exactly two nonzero eigenvalues $\\e^{T}\\f\\pm\\|\\e\\|\\|\\f\\|$\n\\citep[Lemma~14]{zhang2019sharp}. Substituting this solution yields\nexactly \n\\begin{align*}\n\\kappa_{\\lb}(X,Z) & =\\max_{y,W_{i,j}}\\left\\{ \\frac{\\|\\e\\|\\|\\f\\|+\\e^{T}\\f}{\\|\\e\\|\\|\\f\\|-\\e^{T}\\f+\\inner{\\J_{X}^{T}\\J_{X}}W}:\\begin{array}{c}\n\\Q_{\\perp}\\Q_{\\perp}^{T}\\f\\in\\vect(\\S_{+}^{n}),\\\\\n\\f=\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\\\\nW=[W_{i,j}]_{i,j=1}^{r}\\succeq0\n\\end{array}\\right\\} \\\\\n & =\\max_{t\\ge0}\\frac{1+\\cos\\theta(t)}{2t+1-\\cos\\theta(t)}\n\\end{align*}\nwhere $\\cos\\theta(t)$ is itself is defined as the following maximization\n\\[\n\\cos\\theta(t)=\\max_{y,W_{i,j}}\\left\\{ \\e^{T}\\f:\\begin{array}{c}\n\\f\\equiv\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\quad\\|\\e\\|\\|\\f\\|=1,\\\\\n(I-ZZ^{\\dagger})\\,\\mat(\\f)\\,(I-ZZ^{\\dagger})\\succeq0,\\\\\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle=2t,\\quad W=[W_{i,j}]_{i,j=1}^{r}\\succeq0.\n\\end{array}\\right\\} \n\\]\n\n","start":84068,"end":87151}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","labels":["app:tradeoff"],"member_ids":["S076"],"declared_external":false,"source_only":false,"title":"限制梯度松弛的支持空间","source_units":[{"raw_tex":"\\section{\\label{app:tradeoff}Derivation of the dual for the lower-bound problem\n(Proof of \\lemref{tradeoff})}\n\nWe will derive the Lagrangian dual for the following\n\\begin{align*}\n\\kappa_{\\lb}(X,Z) & =\\min_{\\kappa,\\H,s}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H\\preceq\\kappa I,\\qquad\\J_{Z}^{T}s=0,\\qquad s\\in\\vect(\\S_{+}^{n}),\\\\\n\\J_{X}^{T}(\\H\\e+s)=0,\\quad2I_{r}\\otimes\\mat(\\H\\e+s)+\\kappa\\J_{X}^{T}\\J_{X}\\succeq0.\n\\end{array}\\right\\} \n\\end{align*}\nRecall that $\\e=\\vect(XX^{T}-ZZ^{T})$ and $\\J_{X}\\vect(V)=\\vect(XV^{T}+VX^{T})$\nwith respect to fixed $X,Z\\in\\R^{n\\times r}$. Observe that\n\\[\n\\J_{Z}^{T}\\vect(S)=0\\iff(S+S^{T})Z=0\\iff S=Q_{\\perp}S_{\\perp}Q_{\\perp}^{T}\\iff\\vect(S)=\\Q_{\\perp}\\vect(S_{\\perp})\n\\]\nwhere the orthogonal complement $Q_{\\perp}$ of $Z$ is such that\n$Q_{\\perp}Q_{\\perp}^{T}=I_{n}-ZZ^{\\dagger}$, and $\\Q_{\\perp}=Q_{\\perp}\\otimes Q_{\\perp}$.\n","start":84068,"end":84927}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","labels":["app:tradeoff","eq:ub2-1"],"member_ids":["S077"],"declared_external":false,"source_only":false,"title":"形成对偶并声称强对偶","source_units":[{"raw_tex":"Substituting and taking the Lagrangian dual yields\n\\begin{align}\n\\kappa_{\\lb}(X,Z) & =\\min_{\\kappa,\\H,s_{\\perp}}\\left\\{ \\kappa:\\begin{array}{c}\nI\\preceq\\H\\preceq\\kappa I,\\quad s_{\\perp}\\in\\vect(\\S_{+}^{n-r^{\\star}}),\\quad\\J_{X}^{T}(\\H\\e+\\Q_{\\perp}s_{\\perp})=0,\\\\\n2I_{r}\\otimes\\mat(\\H\\e+\\Q_{\\perp}s_{\\perp})+\\kappa\\J_{X}^{T}\\J_{X}\\succeq0\n\\end{array}\\right\\} \\nonumber \\\\\n & =\\max_{y,W_{i,j},U,V}\\left\\{ \\tr(U):\\begin{array}{c}\n\\f\\e^{T}+\\e\\f^{T}=U-V,\\quad\\tr(V)+\\inner{\\J_{X}^{T}\\J_{X}}W=1,\\\\\n\\f\\equiv\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\quad\\Q_{\\perp}^{T}\\f\\in\\vect(\\S_{+}^{n-r^{\\star}}),\\\\\nU,V\\succeq0,\\quad W=[W_{i,j}]_{i,j=1}^{r}\\succeq0\n\\end{array}\\right\\} \\label{eq:ub2-1}\n\\end{align}\nin which $y\\in\\R^{nr}$ and $W_{i,j}\\in\\R^{n\\times n}$ for $i,j\\in\\{1,2,\\dots,r\\}$.\nStrong duality holds here because $y=0,$ $W=\\epsilon\\cdot I_{nr},$\n$U=\\frac{1}{n}I_{n},$ and $V=\\frac{1}{n}I_{n}+\\epsilon\\cdot r\\cdot[\\e\\vect(I_{n})^{T}+\\e\\vect(I_{n})^{T}]$\nis strictly feasible for a sufficiently small $\\epsilon>0$. \n","start":84927,"end":85936}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","labels":["app:tradeoff"],"member_ids":["S078"],"declared_external":false,"source_only":false,"title":"消去半正定变量并计算秩二谱","source_units":[{"raw_tex":"\nThe dual problem (\\ref{eq:ub2-1}) has a closed-form solution over\n$U$ and $V$ \\citep[Lemma~13]{zhang2019sharp}\n\\begin{align*}\n\\frac{\\tr([M]_{-})+\\alpha}{\\tr([M]_{+})}= & \\min_{t,U,V}\\left\\{ \\tr(V)+\\alpha\\cdot t:\\begin{array}{c}\ntM=U-V,\\quad t\\ge0,\\\\\n\\tr(U)=1,\\quad U,V\\succeq0\n\\end{array}\\right\\} ,\n\\end{align*}\nand $\\f\\e^{T}+\\e\\f^{T}$ has exactly two nonzero eigenvalues $\\e^{T}\\f\\pm\\|\\e\\|\\|\\f\\|$\n\\citep[Lemma~14]{zhang2019sharp}. Substituting this solution yields\n","start":85936,"end":86403}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","labels":["app:tradeoff"],"member_ids":["S079"],"declared_external":false,"source_only":false,"title":"齐次归一化为角度最大化","source_units":[{"raw_tex":"exactly \n\\begin{align*}\n\\kappa_{\\lb}(X,Z) & =\\max_{y,W_{i,j}}\\left\\{ \\frac{\\|\\e\\|\\|\\f\\|+\\e^{T}\\f}{\\|\\e\\|\\|\\f\\|-\\e^{T}\\f+\\inner{\\J_{X}^{T}\\J_{X}}W}:\\begin{array}{c}\n\\Q_{\\perp}\\Q_{\\perp}^{T}\\f\\in\\vect(\\S_{+}^{n}),\\\\\n\\f=\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\\\\nW=[W_{i,j}]_{i,j=1}^{r}\\succeq0\n\\end{array}\\right\\} \\\\\n & =\\max_{t\\ge0}\\frac{1+\\cos\\theta(t)}{2t+1-\\cos\\theta(t)}\n\\end{align*}\nwhere $\\cos\\theta(t)$ is itself is defined as the following maximization\n\\[\n\\cos\\theta(t)=\\max_{y,W_{i,j}}\\left\\{ \\e^{T}\\f:\\begin{array}{c}\n\\f\\equiv\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i}),\\quad\\|\\e\\|\\|\\f\\|=1,\\\\\n(I-ZZ^{\\dagger})\\,\\mat(\\f)\\,(I-ZZ^{\\dagger})\\succeq0,\\\\\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle=2t,\\quad W=[W_{i,j}]_{i,j=1}^{r}\\succeq0.\n\\end{array}\\right\\} \n\\]\n\n","start":86403,"end":87151}]},{"id":"S080","semantic_type":"lemma","kind":"lemma","labels":["lem:costheta"],"member_ids":["S080"],"declared_external":false,"source_only":false,"title":"角度目标的可行证书下界","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:costheta}We have $\\cos\\theta(t)\\ge\\alpha\\cdot(t/\\beta)+\\sqrt{1-\\alpha^{2}}\\sqrt{1-t^{2}/\\beta^{2}}$\nfor $0\\le t\\le\\alpha\\beta$ where $Z_{\\perp}=(I-XX^{\\dagger})Z$ and\n\\begin{align*}\n\\alpha & =\\frac{\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}{\\|XX^{T}-ZZ^{T}\\|_{F}}, & \\beta & =\\frac{\\lambda_{\\min}(X^{T}X)\\cdot\\tr(Z_{\\perp}Z_{\\perp}^{T})}{\\|XX^{T}-ZZ^{T}\\|_{F}\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}.\n\\end{align*}\n\\end{lemma}","start":65276,"end":65706}]},{"id":"S081","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S081"],"declared_external":false,"source_only":false,"title":"角度下界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nWrite $\\Pi=I-ZZ^{\\dagger}$ and $\\e=\\vect(XX^{T}-ZZ^{T})$, and decompose\n$\\e_{1}=(\\J_{X}\\J_{X}^{\\dagger})\\e$ and $\\e_{2}=(I-\\J_{X}\\J_{X}^{\\dagger})\\e$.\nWe verify that the following heuristic choice\n\\begin{gather*}\ny=\\underbrace{\\frac{\\sqrt{1-(t/\\beta)^{2}}}{\\|\\e\\|\\|\\e_{1}\\|}}_{\\gamma_{1}}\\;\\J_{X}^{\\dagger}\\e,\\quad W=\\underbrace{\\frac{t/\\beta}{\\|\\e\\|\\|\\e_{2}\\|}}_{\\gamma_{2}}\\;\\sum_{i=1}^{r}\\vect(Z_{\\perp}e_{i}v_{r}^{T})\\vect(Z_{\\perp}e_{i}v_{r}^{T})^{T},\n\\end{gather*}\nis feasible for (\\ref{eq:cosprob}). Indeed, $W\\succeq0$ holds, and\nwe have from (\\ref{eq:e2}), (\\ref{eq:Wsol1}), and (\\ref{eq:Wsol2}):\n\\begin{gather*}\n\\f=\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i})=\\gamma_{1}\\cdot\\J_{X}\\J_{X}^{\\dagger}\\e-\\gamma_{2}\\vect(Z_{\\perp}Z_{\\perp}^{T})=\\gamma_{1}\\cdot\\e_{1}+\\gamma_{2}\\e_{2},\\\\\n\\|\\e\\|^{2}\\|\\f\\|^{2}=\\gamma_{1}^{2}\\cdot\\|\\e\\|^{2}\\|\\e_{1}\\|^{2}+\\gamma_{2}^{2}\\cdot\\|\\e\\|^{2}\\|\\e_{2}\\|^{2}=(1-(t/\\beta)^{2})+(t/\\beta)^{2}=1,\\\\\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle=2\\left(\\frac{t/\\beta}{\\|\\e\\|\\|\\e_{2}\\|}\\right)\\cdot\\lambda_{\\min}(X^{T}X)\\|Z_{\\perp}\\|_{F}^{2}=\\frac{2t}{\\beta}\\cdot\\frac{\\sigma_{r}^{2}\\cdot\\tr(Z_{\\perp}Z_{\\perp}^{T})}{\\|\\e\\|\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}=2t.\n\\end{gather*}\nTo verify that $\\Pi\\,\\mat(\\f)\\,\\Pi\\succeq0$, note that $t\\le\\alpha\\beta$\nimplies $t/\\beta\\le\\alpha\\le1$, so it follows from $\\alpha=\\|\\e_{2}\\|/\\|\\e\\|$\nand $\\sqrt{1-\\alpha^{2}}=\\|\\e_{1}\\|/\\|\\e\\|$ that\n\\[\n\\gamma_{1}=\\frac{\\sqrt{1-(t/\\beta)^{2}}}{\\|\\e_{1}\\|\\|\\e\\|}\\ge\\frac{\\sqrt{1-\\alpha^{2}}}{\\|\\e_{1}\\|\\|\\e\\|}=\\frac{1}{\\|\\e\\|^{2}}=\\frac{\\alpha}{\\|\\e_{2}\\|\\|\\e\\|}\\ge\\frac{t/\\beta}{\\|\\e_{2}\\|\\|\\e\\|}=\\gamma_{2}.\n\\]\nIn turn, $\\gamma_{1}\\ge\\gamma_{2}$ implies\n\\begin{align*}\n\\f & =\\gamma_{1}\\cdot\\e_{1}+\\gamma_{2}\\cdot\\e_{2}=\\gamma_{1}\\cdot\\e-(\\gamma_{1}-\\gamma_{2})\\e_{2},\\\\\n\\mat(\\f) & =\\gamma_{1}\\cdot(XX^{T}-ZZ^{T})+(\\gamma_{1}-\\gamma_{2})\\cdot Z_{\\perp}Z_{\\perp}^{T},\\\\\n\\Pi\\,\\mat(\\f)\\,\\Pi & =\\gamma_{1}\\cdot\\Pi(XX^{T})\\Pi+(\\gamma_{1}-\\gamma_{2})\\cdot\\Pi(Z_{\\perp}Z_{\\perp}^{T})\\Pi\\succeq0.\n\\end{align*}\nHaving verified $y$ and $W$ as feasible for (\\ref{eq:cosprob}),\nwe use its objective value to heuristically lower-bound the maximum\n\\[\n\\cos\\theta(t)\\ge\\e^{T}\\f=\\gamma_{1}\\cdot\\|\\e_{1}\\|^{2}+\\gamma_{2}\\cdot\\|\\e_{2}\\|^{2}=\\sqrt{1-(t/\\beta)^{2}}\\sqrt{1-\\alpha^{2}}+(t/\\beta)\\cdot\\alpha.\n\\]\n \\qed\n\\end{proof}","start":65707,"end":68016}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S082"],"declared_external":false,"source_only":false,"title":"构造证书并验证归一化与代价","source_units":[{"raw_tex":"\\begin{proof}\nWrite $\\Pi=I-ZZ^{\\dagger}$ and $\\e=\\vect(XX^{T}-ZZ^{T})$, and decompose\n$\\e_{1}=(\\J_{X}\\J_{X}^{\\dagger})\\e$ and $\\e_{2}=(I-\\J_{X}\\J_{X}^{\\dagger})\\e$.\nWe verify that the following heuristic choice\n\\begin{gather*}\ny=\\underbrace{\\frac{\\sqrt{1-(t/\\beta)^{2}}}{\\|\\e\\|\\|\\e_{1}\\|}}_{\\gamma_{1}}\\;\\J_{X}^{\\dagger}\\e,\\quad W=\\underbrace{\\frac{t/\\beta}{\\|\\e\\|\\|\\e_{2}\\|}}_{\\gamma_{2}}\\;\\sum_{i=1}^{r}\\vect(Z_{\\perp}e_{i}v_{r}^{T})\\vect(Z_{\\perp}e_{i}v_{r}^{T})^{T},\n\\end{gather*}\nis feasible for (\\ref{eq:cosprob}). Indeed, $W\\succeq0$ holds, and\nwe have from (\\ref{eq:e2}), (\\ref{eq:Wsol1}), and (\\ref{eq:Wsol2}):\n\\begin{gather*}\n\\f=\\J_{X}y-\\sum_{i=1}^{r}\\vect(W_{i,i})=\\gamma_{1}\\cdot\\J_{X}\\J_{X}^{\\dagger}\\e-\\gamma_{2}\\vect(Z_{\\perp}Z_{\\perp}^{T})=\\gamma_{1}\\cdot\\e_{1}+\\gamma_{2}\\e_{2},\\\\\n\\|\\e\\|^{2}\\|\\f\\|^{2}=\\gamma_{1}^{2}\\cdot\\|\\e\\|^{2}\\|\\e_{1}\\|^{2}+\\gamma_{2}^{2}\\cdot\\|\\e\\|^{2}\\|\\e_{2}\\|^{2}=(1-(t/\\beta)^{2})+(t/\\beta)^{2}=1,\\\\\n\\langle\\J_{X}^{T}\\J_{X},W\\rangle=2\\left(\\frac{t/\\beta}{\\|\\e\\|\\|\\e_{2}\\|}\\right)\\cdot\\lambda_{\\min}(X^{T}X)\\|Z_{\\perp}\\|_{F}^{2}=\\frac{2t}{\\beta}\\cdot\\frac{\\sigma_{r}^{2}\\cdot\\tr(Z_{\\perp}Z_{\\perp}^{T})}{\\|\\e\\|\\|Z_{\\perp}Z_{\\perp}^{T}\\|_{F}}=2t.\n\\end{gather*}\n","start":65707,"end":66910}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S083"],"declared_external":false,"source_only":false,"title":"以系数次序验证正交补半正定","source_units":[{"raw_tex":"To verify that $\\Pi\\,\\mat(\\f)\\,\\Pi\\succeq0$, note that $t\\le\\alpha\\beta$\nimplies $t/\\beta\\le\\alpha\\le1$, so it follows from $\\alpha=\\|\\e_{2}\\|/\\|\\e\\|$\nand $\\sqrt{1-\\alpha^{2}}=\\|\\e_{1}\\|/\\|\\e\\|$ that\n\\[\n\\gamma_{1}=\\frac{\\sqrt{1-(t/\\beta)^{2}}}{\\|\\e_{1}\\|\\|\\e\\|}\\ge\\frac{\\sqrt{1-\\alpha^{2}}}{\\|\\e_{1}\\|\\|\\e\\|}=\\frac{1}{\\|\\e\\|^{2}}=\\frac{\\alpha}{\\|\\e_{2}\\|\\|\\e\\|}\\ge\\frac{t/\\beta}{\\|\\e_{2}\\|\\|\\e\\|}=\\gamma_{2}.\n\\]\nIn turn, $\\gamma_{1}\\ge\\gamma_{2}$ implies\n\\begin{align*}\n\\f & =\\gamma_{1}\\cdot\\e_{1}+\\gamma_{2}\\cdot\\e_{2}=\\gamma_{1}\\cdot\\e-(\\gamma_{1}-\\gamma_{2})\\e_{2},\\\\\n\\mat(\\f) & =\\gamma_{1}\\cdot(XX^{T}-ZZ^{T})+(\\gamma_{1}-\\gamma_{2})\\cdot Z_{\\perp}Z_{\\perp}^{T},\\\\\n\\Pi\\,\\mat(\\f)\\,\\Pi & =\\gamma_{1}\\cdot\\Pi(XX^{T})\\Pi+(\\gamma_{1}-\\gamma_{2})\\cdot\\Pi(Z_{\\perp}Z_{\\perp}^{T})\\Pi\\succeq0.\n\\end{align*}\n","start":66910,"end":67712}]},{"id":"S084","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S084"],"declared_external":false,"source_only":false,"title":"计算可行证书的角度值","source_units":[{"raw_tex":"Having verified $y$ and $W$ as feasible for (\\ref{eq:cosprob}),\nwe use its objective value to heuristically lower-bound the maximum\n\\[\n\\cos\\theta(t)\\ge\\e^{T}\\f=\\gamma_{1}\\cdot\\|\\e_{1}\\|^{2}+\\gamma_{2}\\cdot\\|\\e_{2}\\|^{2}=\\sqrt{1-(t/\\beta)^{2}}\\sqrt{1-\\alpha^{2}}+(t/\\beta)\\cdot\\alpha.\n\\]\n \\qed\n\\end{proof}","start":67712,"end":68016}]},{"id":"S085","semantic_type":"lemma","kind":"lemma","labels":["lem:counter"],"member_ids":["S085"],"declared_external":false,"source_only":false,"title":"上界SDP的特征方向证书","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:counter}Let $[Q_{1},Q_{2}]$ have orthonormal columns\nwith $Q_{1}\\in\\R^{n\\times r}$ and $Q_{2}\\in\\R^{n\\times r^{\\star}}$.\nIf $\\kappa=1+2\\sqrt{r/r^{\\star}}$, and $\\H$ is any $n^{2}\\times n^{2}$\nmatrix that satisfies $I\\preceq\\H\\preceq\\kappa I$ and the following\n$rr^{\\star}+2$ eigenvalue equations\n\\begin{gather*}\n\\H\\vect\\left(Q_{1}Q_{1}^{T}+\\sqrt{r/r^{\\star}}Q_{2}Q_{2}\\right)=\\kappa\\cdot\\vect\\left(Q_{1}Q_{1}^{T}+\\sqrt{r/r^{\\star}}Q_{2}Q_{2}\\right),\\\\\n\\H\\vect\\left(Q_{1}Q_{1}^{T}-\\sqrt{r/r^{\\star}}Q_{2}Q_{2}\\right)=1\\cdot\\vect\\left(Q_{1}Q_{1}^{T}-\\sqrt{r/r^{\\star}}Q_{2}Q_{2}\\right),\\\\\n\\H\\vect\\left(Q_{1}VQ_{2}^{T}+Q_{2}V^{T}Q_{1}^{T}\\right)=\\kappa\\cdot\\vect\\left(Q_{1}VQ_{2}^{T}+Q_{2}V^{T}Q_{1}^{T}\\right)\\qquad\\text{for all }V\\in\\R^{r\\times r^{\\star}},\n\\end{gather*}\nthen $(\\kappa,\\H)$ is a feasible point for (\\ref{eq:kappaub_prob})\nwith $X=Q_{1}$ and $Z=\\sqrt{1+\\sqrt{r/r^{\\star}}}Q_{2}$. Moveover,\n$\\e^{T}\\H\\e=(1+2\\sqrt{r/r^{\\star}})\\cdot(1+\\sqrt{r/r^{\\star}})\\cdot r^{\\star}$.\n\\end{lemma}","start":69778,"end":70798}]},{"id":"S086","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S086"],"declared_external":false,"source_only":false,"title":"特征方向证书的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nWrite $\\rho\\equiv\\sqrt{r/r^{\\star}}$, and take $[Q_{1},Q_{2}]=I_{r+r^{\\star}}$\nwithout loss of generality. Decomposing $\\e=\\vect(XX^{T}-ZZ^{T})$\ninto eigenvectors and applying $\\H$ yields\n\\begin{gather*}\n\\mat(\\e)=\\begin{bmatrix}I_{r} & 0\\\\\n0 & -(1+\\rho)I_{r^{\\star}}\n\\end{bmatrix}=-\\frac{1}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & \\rho I_{r^{\\star}}\n\\end{bmatrix}+\\frac{1+2\\rho}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & -\\rho I_{r^{\\star}}\n\\end{bmatrix},\\\\\n\\mat(\\H\\e)=-\\frac{1+2\\rho}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & \\rho I_{r^{\\star}}\n\\end{bmatrix}+\\frac{1+2\\rho}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & -\\rho I_{r^{\\star}}\n\\end{bmatrix}=\\begin{bmatrix}0_{r} & 0\\\\\n0 & -\\kappa I_{r^{\\star}}\n\\end{bmatrix}.\n\\end{gather*}\nClearly, $\\inner{\\mat(\\e)}{\\mat(\\H\\e)}=\\kappa(1+\\rho)r^{\\star}.$\nFor $v=\\vect(V)=\\vect([V_{1};V_{2}])$ with $V_{1}\\in\\R^{r\\times r}$\nand $V_{2}\\in\\R^{r^{\\star}\\times r}$, observe that \n\\begin{gather*}\nv^{T}\\J_{X}\\H\\e=\\left\\langle \\begin{bmatrix}V_{1}+V_{1}^{T} & V_{2}^{T}\\\\\nV_{2} & 0_{r^{\\star}}\n\\end{bmatrix},\\begin{bmatrix}0_{r} & 0\\\\\n0 & -\\kappa I_{r^{\\star}}\n\\end{bmatrix}\\right\\rangle =0,\\\\\nv^{T}[I_{r}\\otimes\\mat(\\H\\e)]v=\\langle\\mat(\\H\\e),VV^{T}\\rangle=\\left\\langle \\begin{bmatrix}0_{r} & 0\\\\\n0 & -\\kappa I_{r^{\\star}}\n\\end{bmatrix},\\begin{bmatrix}V_{1}V_{1}^{T} & V_{1}V_{2}^{T}\\\\\nV_{2}V_{1}^{T} & V_{2}V_{2}^{T}\n\\end{bmatrix}\\right\\rangle =-\\kappa\\|V_{2}\\|^{2},\\\\\nv^{T}\\J_{X}^{T}\\H\\J_{X}v=\\left\\Vert \\H^{1/2}\\vect\\left(\\begin{bmatrix}V_{1}+V_{1}^{T} & V_{2}^{T}\\\\\nV_{2} & 0_{r^{\\star}}\n\\end{bmatrix}\\right)\\right\\Vert ^{2}\\ge\\left\\Vert \\H^{1/2}\\vect\\left(\\begin{bmatrix}0_{r} & V_{2}^{T}\\\\\nV_{2} & 0_{r^{\\star}}\n\\end{bmatrix}\\right)\\right\\Vert ^{2}=2\\kappa\\|V_{2}\\|^{2}.\n\\end{gather*}\nTherefore, $v^{T}\\J_{X}\\H\\e=0$ and $2v^{T}[I_{r}\\otimes\\mat(\\H\\e)]v\\ge-v^{T}\\J_{X}^{T}\\H\\J_{X}v$\nhold for all $v$. \\qed\n\\end{proof}","start":70799,"end":72653}]},{"id":"S087","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S087"],"declared_external":false,"source_only":false,"title":"误差特征分解与能量计算","source_units":[{"raw_tex":"\\begin{proof}\nWrite $\\rho\\equiv\\sqrt{r/r^{\\star}}$, and take $[Q_{1},Q_{2}]=I_{r+r^{\\star}}$\nwithout loss of generality. Decomposing $\\e=\\vect(XX^{T}-ZZ^{T})$\ninto eigenvectors and applying $\\H$ yields\n\\begin{gather*}\n\\mat(\\e)=\\begin{bmatrix}I_{r} & 0\\\\\n0 & -(1+\\rho)I_{r^{\\star}}\n\\end{bmatrix}=-\\frac{1}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & \\rho I_{r^{\\star}}\n\\end{bmatrix}+\\frac{1+2\\rho}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & -\\rho I_{r^{\\star}}\n\\end{bmatrix},\\\\\n\\mat(\\H\\e)=-\\frac{1+2\\rho}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & \\rho I_{r^{\\star}}\n\\end{bmatrix}+\\frac{1+2\\rho}{2\\rho}\\begin{bmatrix}I_{r} & 0\\\\\n0 & -\\rho I_{r^{\\star}}\n\\end{bmatrix}=\\begin{bmatrix}0_{r} & 0\\\\\n0 & -\\kappa I_{r^{\\star}}\n\\end{bmatrix}.\n\\end{gather*}\nClearly, $\\inner{\\mat(\\e)}{\\mat(\\H\\e)}=\\kappa(1+\\rho)r^{\\star}.$\n","start":70799,"end":71593}]},{"id":"S088","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S088"],"declared_external":false,"source_only":false,"title":"分块扰动验证一阶消失","source_units":[{"raw_tex":"For $v=\\vect(V)=\\vect([V_{1};V_{2}])$ with $V_{1}\\in\\R^{r\\times r}$\nand $V_{2}\\in\\R^{r^{\\star}\\times r}$, observe that \n\\begin{gather*}\nv^{T}\\J_{X}\\H\\e=\\left\\langle \\begin{bmatrix}V_{1}+V_{1}^{T} & V_{2}^{T}\\\\\nV_{2} & 0_{r^{\\star}}\n\\end{bmatrix},\\begin{bmatrix}0_{r} & 0\\\\\n0 & -\\kappa I_{r^{\\star}}\n\\end{bmatrix}\\right\\rangle =0,\\\\\n","start":71593,"end":71925}]},{"id":"S089","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S089"],"declared_external":false,"source_only":false,"title":"交叉块特征值验证二阶约束","source_units":[{"raw_tex":"v^{T}[I_{r}\\otimes\\mat(\\H\\e)]v=\\langle\\mat(\\H\\e),VV^{T}\\rangle=\\left\\langle \\begin{bmatrix}0_{r} & 0\\\\\n0 & -\\kappa I_{r^{\\star}}\n\\end{bmatrix},\\begin{bmatrix}V_{1}V_{1}^{T} & V_{1}V_{2}^{T}\\\\\nV_{2}V_{1}^{T} & V_{2}V_{2}^{T}\n\\end{bmatrix}\\right\\rangle =-\\kappa\\|V_{2}\\|^{2},\\\\\nv^{T}\\J_{X}^{T}\\H\\J_{X}v=\\left\\Vert \\H^{1/2}\\vect\\left(\\begin{bmatrix}V_{1}+V_{1}^{T} & V_{2}^{T}\\\\\nV_{2} & 0_{r^{\\star}}\n\\end{bmatrix}\\right)\\right\\Vert ^{2}\\ge\\left\\Vert \\H^{1/2}\\vect\\left(\\begin{bmatrix}0_{r} & V_{2}^{T}\\\\\nV_{2} & 0_{r^{\\star}}\n\\end{bmatrix}\\right)\\right\\Vert ^{2}=2\\kappa\\|V_{2}\\|^{2}.\n\\end{gather*}\nTherefore, $v^{T}\\J_{X}\\H\\e=0$ and $2v^{T}[I_{r}\\otimes\\mat(\\H\\e)]v\\ge-v^{T}\\J_{X}^{T}\\H\\J_{X}v$\nhold for all $v$. \\qed\n\\end{proof}","start":71925,"end":72653}]},{"id":"S090","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S090"],"declared_external":true,"declaration_text":["原始外部前提列表第1项：standard convex KKT conditions over the positive semidefinite cone。"],"source_only":false,"title":"半正定锥上的凸KKT条件","source_units":[]},{"id":"S091","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S091"],"declared_external":true,"declaration_text":["原始外部前提列表第1项：Slater strong duality及标准凸KKT背景。"],"source_only":false,"title":"Lagrange对偶与Slater强对偶","source_units":[]},{"id":"S092","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S092"],"declared_external":true,"declaration_text":["原始外部前提列表第2项：standard matrix calculus及fundamental theorem of calculus。"],"source_only":false,"title":"矩阵链式法则与微积分基本定理","source_units":[]},{"id":"S093","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S093"],"declared_external":true,"declaration_text":["原始外部前提列表第2项：spectral and QR decompositions、Moore–Penrose projection identities。"],"source_only":false,"title":"谱分解、QR与伪逆投影背景","source_units":[]},{"id":"S094","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S094"],"declared_external":true,"declaration_text":["原始外部前提列表第2项明确允许Cauchy–Schwarz。"],"source_only":false,"title":"Cauchy–Schwarz计数界","source_units":[]},{"id":"S095","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S095"],"declared_external":true,"declaration_text":["原始外部前提列表第5项；源文L1564–1571引用zhang2019sharp，Lemma 13。"],"source_only":false,"title":"半正定变量消元恒等式","source_units":[]},{"id":"S096","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S096"],"declared_external":true,"declaration_text":["原始外部前提列表第5项；源文L1572–1573引用zhang2019sharp，Lemma 14。"],"source_only":false,"title":"对称秩二矩阵的谱公式","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S012":"S012","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076","S077":"S077","S078":"S078","S079":"S079","S080":"S080","S081":"S081","S082":"S082","S083":"S083","S084":"S084","S085":"S085","S086":"S086","S087":"S087","S088":"S088","S089":"S089","S090":"S090","S091":"S091","S092":"S092","S093":"S093","S094":"S094","S095":"S095","S096":"S096"}}}}
{"record_id":"2012.02522v5","paper":{"title":"Accelerating Inexact Successive Quadratic Approximation for Regularized Optimization Through Manifold Identification","authors":["Lee, Ching-pei"],"arxiv_id":"2012.02522","version":"v5","journal":"Mathematical Programming","abstract_url":"https://arxiv.org/abs/2012.02522v5","pdf_url":"https://arxiv.org/pdf/2012.02522v5","doi":"10.1007/s10107-022-01916-2"},"context":{"raw_tex":"Consider the following regularized optimization problem:\n\\begin{equation}\n\t\\min_{x \\in \\H}\\quad F(x) \\coloneqq f(x) + \\Psi(x),\n\t\\label{eq:f}\n\\end{equation}\nwhere $\\H$ is a Euclidean space with an inner product\n$\\inprod{\\cdot}{\\cdot}$ and its induced norm $\\norm{\\cdot}$,\nthe regularizer $\\Psi$ is extended-valued, convex, proper, and\nlower-semicontinuous, $f$ is \\modify{continuously differentiable with\nLipschitz-continuous gradients},\nand the solution set $\\Omega$ is non-empty.\nThis type of problems is ubiquitous in applications such as machine\nlearning and signal processing \\modify{(see, for example,\n\\cite{Tib96a,CanR09a,CanRT06a})}.\nOne widely-used method for \\cref{eq:f} is inexact successive quadratic\napproximation (\\ipvm).\nAt the $t$th iteration with iterate $x^t$,\n\\ipvm obtains the update direction $p^t$ by approximately solving\n\\begin{gather}\n\\label{eq:quadratic}\np^t \\approx \\argmin_{p \\in \\H}\\quad Q_{H_t}\\left( p; x^t\n\t\\right),\\\\\n\\label{eq:Qdef}\nQ_{H_t}\\left( p; x^t \\right) \\coloneqq \\inprod{\\nabla f\\left( x^t\n\t\\right)}{p} + \\frac12 \\inprod{p}{ H_t p} + \\Psi\\left( x^t + p\n\t\\right) - \\Psi\\left( x^t \\right),\n\\end{gather}\nwhere $H_t$ is a self-adjoint positive-semidefinite linear\nendomorphism of $\\H$.\nThe iterate is then updated along $p^t$ with a step size $\\alpha_t\n> 0$.\n\n\nA possible remedy when $\\Psi$ is partly smooth \\citep{Lew02a}\nis to switch to smooth optimization after identifying an active\nmanifold $\\M$ that contains a solution \\modify{$\\hat x$} to \\cref{eq:f}\nand makes $\\Psi$ confined to it smooth.\nWe say an algorithm can identify $\\M$ if there is a neighborhood $U$\nof \\modify{$\\hat x$} such that $x^t \\in U$ implies $x^{t+1} \\in \\M$, and call it\npossesses the manifold identification property.\n\n\n\\begin{equation}\nr^t \\coloneqq \\argmin_{r} \\|r\\|, \\quad \\text{ subject to } \\quad r \\in\n\\partial_{p} Q_{H_t}(p^t;x^t),\n\\label{eq:subnorm}\n\\end{equation}\n\n\nWe denote the minimum of \\cref{eq:f} by $F^*$; the domain of $\\Psi$\nby $\\dom(\\Psi)$; and the set of convex, proper, and lower\nsemicontinuous functions by $\\Gamma_0$.\n\\modify{For any set $C$, $\\relint(C)$ denotes its relative interior.}\nWe will frequently use the following notations.\n\\[\n\t\\delta_t \\coloneqq F(x^t) - F^*,\\quad\nP_{\\Omega}(x) \\coloneqq \\argmin_{y \\in \\Omega} \\|x - y\\|,\\quad\n\t\\dist(x,\\Omega) \\coloneqq \\|x - P_\\Omega(x)\\|.\n\\]\nThe level set $\\levbet \\coloneqq \\left\\{ x\\mid  F(x) - F^* \\leq \\xi\n\\right\\}$ for any $\\xi \\geq 0$  is closed but not necessarily\n\\modify{bounded.}\nA function is $L$-smooth if it is \\modify{differentiable with the gradient\n$L$-Lipschitz continuous}.\nWe denote the identity operator by $I$.\nFor self-adjoint linear endomorphisms $A,B$ of $\\H$,\n$A \\succ B$ ($\\succeq$) means $A-B$ is positive definite (positive\nsemidefinite).\nWe abbreviate $A \\succ \\tau I$ to $A \\succ \\tau$ for $\\tau \\in \\R$.\nThe set of $A$ with $A \\succ 0$ is denoted by $\\SS$.\nThe subdifferential $\\partial \\Psi(x)$ of $\\Psi$ at $x$ is\nwell-defined as $\\Psi \\in \\Gamma_0$,\nhence so is the generalized gradient $\\partial F(x) = \\nabla f(x) +\n\\partial \\Psi(x)$.\nFor any $g \\in \\Gamma_0$, $\\tau\\geq 0$, and \\modify{$\\Lambda \\in \\SS$},\nthe proximal mapping\n\\begin{equation}\n\\label{eq:proximal}\n\t\\prox^{\\Lambda}_{\\tau g}(x) \\coloneqq \\argmin_{y \\in \\H}\\, \\frac12\n\t\\inprod{x - y}{\\Lambda (x-y)} + \\tau g\\left( y \\right)\n\\end{equation}\nis continuous and finite in $\\H$ even outside\n$\\dom(g)$.\nWhen $\\Lambda = I$, \\cref{eq:proximal} is shorten to\n$\\prox_{\\tau g}(x)$.\nFor \\cref{eq:quadratic}, we denote its optimal solution by\n$p^{t*}$. When there is no ambiguity, we abbreviate $Q_{H_t}(\\cdot;\nx^t)$ to $Q_t(\\cdot)$, $Q_t(p^t)$ to $\\hat Q_t$, and $Q_t(p^{t*})$ to\n$Q_t^*$.\n\n\nAfter obtaining $p^t$ from \\cref{eq:quadratic}, we need to find a\nstep size $\\alpha_t > 0$ for it  to ensure sufficient objective\ndecrease.\nGiven $\\gamma,\\beta \\in (0,1)$, we take $\\alpha_t$ as the\nlargest value in $\\{\\beta^0,\\beta^1,\\dotsc\\}$ satisfying an\nArmijo-like condition.\n\\begin{equation}\n\tF\\left( x^t + \\alpha_t p^t \\right) \\leq F\\left( x^t \\right) +\n\t\\gamma \\alpha_t Q_t(p^t).\n\t\\label{eq:armijo}\n\\end{equation}\nThis condition is satisfied by all $\\alpha_t$ small enough as long as\n$Q_t(p^t) < 0$ and $Q_t(\\cdot)$ is strongly convex; see\n\\cite[Lemma~3]{LeeW18a}.\n\nFor the choice of $H_t$,\nwe only make the following blanket assumption without further\nspecification to make our analysis more general.\n\\begin{equation}\n\\exists M,m > 0,\\quad \\text{ such that } \\quad M \\succeq H_t  \\succeq\nm, \\;\\forall t \\geq 0.\n\\label{assum:Hbound}\n\\end{equation}\n\n\nFor \\cref{eq:quadratic}, any suitable solver for regularized\noptimization, such as (accelerated) proximal gradient,\n(variance-reduced) stochastic gradient methods, and their variants,\ncan be used, and the following are common for their inner loop\ntermination:\n\\begin{gather}\n\t\\hat Q_t - Q_t^* \\leq \\epsilon_t,\n\t\\label{eq:subprob}\\\\\n\\left\\| r^t \\right\\| \\leq \\epsilon_t,\n\\quad \\text{ or }\n\t\\label{eq:partial}\\\\\nG^\\tau_{H_t} \\left( p^t;x^t \\right)\\coloneqq \\norm{p^t - \\bar\np^t_\\tau}\\leq \\epsilon_t,\n\t\\label{eq:gm}\n\\end{gather}\nfor some given $\\epsilon_t\\geq 0$ and $\\tau > 0$, where\n\\begin{align}\n\\label{eq:p}\n\\bar p^t_\\tau &\\coloneqq \\prox_{\\tau \\Psi}\\left(\\left(x^t + p^t \\right) -\n\t\\tau \\left(\\nabla f(x^t) + H_t p^t\\right)\\right) - x^t.\n\\end{align}\n\\modify{\nThe point $\\bar p^t_\\tau$ in \\cref{eq:p} is computed by taking a\nproximal gradient step of the subproblem \\cref{eq:quadratic} from\n$p^t$, and thus $\\bar p^t - p^t$ is \\modifyy{the proximal gradient}\n(with step size $\\tau$) of $Q_t$ at $p^t$.\nBecause the subproblem is strongly convex, the norm of the proximal\ngradient is zero at $p^t$ if and only if $p^t$ is the unique solution\nto the subproblem.\nWe will see in the next subsection that the norm of this proximal\ngradient squared is also equivalent to the objective distance to the\noptimum of the subproblem.}\n\n\n\\begin{algorithm}[h]\n\\DontPrintSemicolon\n\\SetKwInOut{Input}{input}\\SetKwInOut{Output}{output}\n\\caption{A Framework of \\ipvm for \\cref{eq:f}}\n\\label{alg:vm}\n\\Input{$x^0 \\in \\H$, $\\gamma, \\beta \\in (0,1)$}\n\n\\For{$t=0,1,\\dotsc$}{\n\t$\\alpha_t \\leftarrow 1$,\n\tpick $\\epsilon_t \\geq 0$ and $H_t$, and solve \\cref{eq:quadratic}\n\tfor $p^t$ satisfying \\cref{eq:subprob},\n\t\\cref{eq:partial}, or \\cref{eq:gm}\n\n\n\t\\lWhile{\\cref{eq:armijo} is not satisfied}\n\t{$\\alpha_t \\leftarrow \\beta \\alpha_t$}\n\t$x^{t+1} \\leftarrow x^t + \\alpha_t p^t$\n}\n\\end{algorithm}\n\n\n\\subsection{Basic Properties}\nUnder \\cref{assum:Hbound}, \\cref{eq:Qdef} is $m$-strongly convex with\nrespect to $p$, so the following standard results hold for any $\\tau\n\\in (0, 1/M]$ and any $p^t \\in \\H$ \\citep{Nes18a,DruL16a}.\n\\begin{gather}\n\\|r^t\\|^2 \\geq 2 m \\left( Q_t(p^t) - Q_t^* \\right)\n\\geq \\frac{m}{M} G^\\tau_{H_t}\\left(\n\tp^t;x^t \\right)^2,\n\\label{eq:PG}\\\\\nG^\\tau_{H_t}\\left( p^t;x^t \\right)^2 \\geq \\tau \\left( \\frac{(2m^{-1} +\n\t\\tau)(1 + M \\tau)}{\\tau} - \\frac{1}{2} \\right)^{-1}\n\t\\left(Q_t\\left( \\bar p^t_\\tau\\right) - Q_t^* \\right).\n\\label{eq:EB}\n\\end{gather}\n\n\nA central focus of this work is manifold identification, so we first\nformally define manifolds following \\cite{Vai84a}.\nA set $\\M \\in \\R^m$ is a $p$-dimensional $\\mathcal{C}^k$ manifold\naround $x \\in \\R^m$ if there is a $\\mathcal{C}^k$ function $\\Phi: \\R^m\n\\rightarrow \\R^{m-p}$ whose derivative at $x$ is surjective such that for\n$y$ close enough to $x$, $y \\in \\M$ if and only if $\\Phi(y) = 0$.\nThrough the implicit function theorem, we can also use a\n$\\mathcal{C}^k$ parameterization $\\phi: \\R^p \\rightarrow \\M$, with\n$\\phi(y) = x$ and the derivative injective at $y$, to describe a\nneighborhood of $x$ on $\\M$.\nIn our definition and proof below, we will require the function being\n(subdifferentially) regular and prox-regular at a point $x$.\nThese two conditions are satisfied by convex functions everywhere and\nthus we do not provide their definitions here, and interested readers\ncan refer to \\cite[][Definition~7.25, Definition~13.27]{RocW09a} for details.\nNow we are ready for the definition of partial smoothness\n\\citep{Lew02a} that we assume for the regularizer when discussing\nmanifold identification.\n\\begin{definition}[Partly smooth]\n\t\\label{def:ps}\n\tA function $\\Psi$ is partly\n\tsmooth at a point $x^*$ relative to a set $\\M$ containing $x^*$ if\n\t$\\partial \\Psi(x^*) \\neq \\emptyset$ and:\n\t\\begin{enumerate}\n\t\t\t\\item Around $x^*$, $\\M$ is a $\\mathcal{C}^2$-manifold and\n\t\t\t\t$\\Psi|_{\\M}$ is $\\mathcal{C}^2$.\n\t\t\\item The affine span of $\\partial \\Psi(x)$ is a translate\n\t\t\tof the normal space to $\\M$ at $x^*$.\n\t\t\\item $\\partial \\Psi$ is continuous at $x^*$ relative to $\\M$.\n\t\t\\item \\modify{$\\Psi$ is regular at all points $x \\in \\M$ with\n\t\t\t$\\partial \\Psi(x) \\neq  \\emptyset$\n\t\taround $x^*$.}\n\t\\end{enumerate}\n\\end{definition}\nIntuitively, it means $\\Psi$ is smooth around\n$x^*$ in $\\M$ but changes drastically along directions leaving the\nmanifold.\nWe also call this $\\M$ the active manifold.\n\n\nWe therefore provide another sufficient condition for \\ipvm to\nidentify the active manifold that is satisfied by most of the\nwidely-used solvers for \\cref{eq:quadratic},\nshowing that \\ipvm essentially possesses the manifold identification\nproperty.\nThis result uses the condition \\cref{eq:subprob}, which is weaker than\n\\cref{eq:subnorm}, and we follow \\cite{LeeW18a} to define\n$\\{\\epsilon_t\\}$ using a given $\\eta \\in [0,1)$:\n\\begin{equation}\n\t\\epsilon_t = \\eta \\left( Q_t(0) - Q_t^* \\right) = -\\eta Q_t^*.\n\t\\label{eq:multiplicative}\n\\end{equation}\nAs argued in \\cite{LeeW18a} and practically adopted in various\nimplementations including \\cite{LeeC17a,DunLGBHJ18a,LeeLW19a},\nit is easy to ensure that \\cref{eq:subprob} with\n\\cref{eq:multiplicative} holds for some $\\eta <1$ under\n\t\\cref{assum:Hbound} (although the explicit value might be unknown)\nif we apply a linear-convergent\nsubproblem solver with at least a pre-specified number of iterations\nto \\cref{eq:quadratic}.\n","units":[{"raw_tex":"Consider the following regularized optimization problem:\n\\begin{equation}\n\t\\min_{x \\in \\H}\\quad F(x) \\coloneqq f(x) + \\Psi(x),\n\t\\label{eq:f}\n\\end{equation}\nwhere $\\H$ is a Euclidean space with an inner product\n$\\inprod{\\cdot}{\\cdot}$ and its induced norm $\\norm{\\cdot}$,\nthe regularizer $\\Psi$ is extended-valued, convex, proper, and\nlower-semicontinuous, $f$ is \\modify{continuously differentiable with\nLipschitz-continuous gradients},\nand the solution set $\\Omega$ is non-empty.\nThis type of problems is ubiquitous in applications such as machine\nlearning and signal processing \\modify{(see, for example,\n\\cite{Tib96a,CanR09a,CanRT06a})}.\nOne widely-used method for \\cref{eq:f} is inexact successive quadratic\napproximation (\\ipvm).\nAt the $t$th iteration with iterate $x^t$,\n\\ipvm obtains the update direction $p^t$ by approximately solving\n\\begin{gather}\n\\label{eq:quadratic}\np^t \\approx \\argmin_{p \\in \\H}\\quad Q_{H_t}\\left( p; x^t\n\t\\right),\\\\\n\\label{eq:Qdef}\nQ_{H_t}\\left( p; x^t \\right) \\coloneqq \\inprod{\\nabla f\\left( x^t\n\t\\right)}{p} + \\frac12 \\inprod{p}{ H_t p} + \\Psi\\left( x^t + p\n\t\\right) - \\Psi\\left( x^t \\right),\n\\end{gather}\nwhere $H_t$ is a self-adjoint positive-semidefinite linear\nendomorphism of $\\H$.\nThe iterate is then updated along $p^t$ with a step size $\\alpha_t\n> 0$.\n","start":6048,"end":7344},{"raw_tex":"A possible remedy when $\\Psi$ is partly smooth \\citep{Lew02a}\nis to switch to smooth optimization after identifying an active\nmanifold $\\M$ that contains a solution \\modify{$\\hat x$} to \\cref{eq:f}\nand makes $\\Psi$ confined to it smooth.\nWe say an algorithm can identify $\\M$ if there is a neighborhood $U$\nof \\modify{$\\hat x$} such that $x^t \\in U$ implies $x^{t+1} \\in \\M$, and call it\npossesses the manifold identification property.\n","start":8590,"end":9026},{"raw_tex":"\\begin{equation}\nr^t \\coloneqq \\argmin_{r} \\|r\\|, \\quad \\text{ subject to } \\quad r \\in\n\\partial_{p} Q_{H_t}(p^t;x^t),\n\\label{eq:subnorm}\n\\end{equation}\n","start":10306,"end":10459},{"raw_tex":"We denote the minimum of \\cref{eq:f} by $F^*$; the domain of $\\Psi$\nby $\\dom(\\Psi)$; and the set of convex, proper, and lower\nsemicontinuous functions by $\\Gamma_0$.\n\\modify{For any set $C$, $\\relint(C)$ denotes its relative interior.}\nWe will frequently use the following notations.\n\\[\n\t\\delta_t \\coloneqq F(x^t) - F^*,\\quad\nP_{\\Omega}(x) \\coloneqq \\argmin_{y \\in \\Omega} \\|x - y\\|,\\quad\n\t\\dist(x,\\Omega) \\coloneqq \\|x - P_\\Omega(x)\\|.\n\\]\nThe level set $\\levbet \\coloneqq \\left\\{ x\\mid  F(x) - F^* \\leq \\xi\n\\right\\}$ for any $\\xi \\geq 0$  is closed but not necessarily\n\\modify{bounded.}\nA function is $L$-smooth if it is \\modify{differentiable with the gradient\n$L$-Lipschitz continuous}.\nWe denote the identity operator by $I$.\nFor self-adjoint linear endomorphisms $A,B$ of $\\H$,\n$A \\succ B$ ($\\succeq$) means $A-B$ is positive definite (positive\nsemidefinite).\nWe abbreviate $A \\succ \\tau I$ to $A \\succ \\tau$ for $\\tau \\in \\R$.\nThe set of $A$ with $A \\succ 0$ is denoted by $\\SS$.\nThe subdifferential $\\partial \\Psi(x)$ of $\\Psi$ at $x$ is\nwell-defined as $\\Psi \\in \\Gamma_0$,\nhence so is the generalized gradient $\\partial F(x) = \\nabla f(x) +\n\\partial \\Psi(x)$.\nFor any $g \\in \\Gamma_0$, $\\tau\\geq 0$, and \\modify{$\\Lambda \\in \\SS$},\nthe proximal mapping\n\\begin{equation}\n\\label{eq:proximal}\n\t\\prox^{\\Lambda}_{\\tau g}(x) \\coloneqq \\argmin_{y \\in \\H}\\, \\frac12\n\t\\inprod{x - y}{\\Lambda (x-y)} + \\tau g\\left( y \\right)\n\\end{equation}\nis continuous and finite in $\\H$ even outside\n$\\dom(g)$.\nWhen $\\Lambda = I$, \\cref{eq:proximal} is shorten to\n$\\prox_{\\tau g}(x)$.\nFor \\cref{eq:quadratic}, we denote its optimal solution by\n$p^{t*}$. When there is no ambiguity, we abbreviate $Q_{H_t}(\\cdot;\nx^t)$ to $Q_t(\\cdot)$, $Q_t(p^t)$ to $\\hat Q_t$, and $Q_t(p^{t*})$ to\n$Q_t^*$.\n","start":19364,"end":21139},{"raw_tex":"After obtaining $p^t$ from \\cref{eq:quadratic}, we need to find a\nstep size $\\alpha_t > 0$ for it  to ensure sufficient objective\ndecrease.\nGiven $\\gamma,\\beta \\in (0,1)$, we take $\\alpha_t$ as the\nlargest value in $\\{\\beta^0,\\beta^1,\\dotsc\\}$ satisfying an\nArmijo-like condition.\n\\begin{equation}\n\tF\\left( x^t + \\alpha_t p^t \\right) \\leq F\\left( x^t \\right) +\n\t\\gamma \\alpha_t Q_t(p^t).\n\t\\label{eq:armijo}\n\\end{equation}\nThis condition is satisfied by all $\\alpha_t$ small enough as long as\n$Q_t(p^t) < 0$ and $Q_t(\\cdot)$ is strongly convex; see\n\\cite[Lemma~3]{LeeW18a}.\n\nFor the choice of $H_t$,\nwe only make the following blanket assumption without further\nspecification to make our analysis more general.\n\\begin{equation}\n\\exists M,m > 0,\\quad \\text{ such that } \\quad M \\succeq H_t  \\succeq\nm, \\;\\forall t \\geq 0.\n\\label{assum:Hbound}\n\\end{equation}\n","start":23469,"end":24325},{"raw_tex":"For \\cref{eq:quadratic}, any suitable solver for regularized\noptimization, such as (accelerated) proximal gradient,\n(variance-reduced) stochastic gradient methods, and their variants,\ncan be used, and the following are common for their inner loop\ntermination:\n\\begin{gather}\n\t\\hat Q_t - Q_t^* \\leq \\epsilon_t,\n\t\\label{eq:subprob}\\\\\n\\left\\| r^t \\right\\| \\leq \\epsilon_t,\n\\quad \\text{ or }\n\t\\label{eq:partial}\\\\\nG^\\tau_{H_t} \\left( p^t;x^t \\right)\\coloneqq \\norm{p^t - \\bar\np^t_\\tau}\\leq \\epsilon_t,\n\t\\label{eq:gm}\n\\end{gather}\nfor some given $\\epsilon_t\\geq 0$ and $\\tau > 0$, where\n\\begin{align}\n\\label{eq:p}\n\\bar p^t_\\tau &\\coloneqq \\prox_{\\tau \\Psi}\\left(\\left(x^t + p^t \\right) -\n\t\\tau \\left(\\nabla f(x^t) + H_t p^t\\right)\\right) - x^t.\n\\end{align}\n\\modify{\nThe point $\\bar p^t_\\tau$ in \\cref{eq:p} is computed by taking a\nproximal gradient step of the subproblem \\cref{eq:quadratic} from\n$p^t$, and thus $\\bar p^t - p^t$ is \\modifyy{the proximal gradient}\n(with step size $\\tau$) of $Q_t$ at $p^t$.\nBecause the subproblem is strongly convex, the norm of the proximal\ngradient is zero at $p^t$ if and only if $p^t$ is the unique solution\nto the subproblem.\nWe will see in the next subsection that the norm of this proximal\ngradient squared is also equivalent to the objective distance to the\noptimum of the subproblem.}\n","start":24326,"end":25649},{"raw_tex":"\\begin{algorithm}[h]\n\\DontPrintSemicolon\n\\SetKwInOut{Input}{input}\\SetKwInOut{Output}{output}\n\\caption{A Framework of \\ipvm for \\cref{eq:f}}\n\\label{alg:vm}\n\\Input{$x^0 \\in \\H$, $\\gamma, \\beta \\in (0,1)$}\n\n\\For{$t=0,1,\\dotsc$}{\n\t$\\alpha_t \\leftarrow 1$,\n\tpick $\\epsilon_t \\geq 0$ and $H_t$, and solve \\cref{eq:quadratic}\n\tfor $p^t$ satisfying \\cref{eq:subprob},\n\t\\cref{eq:partial}, or \\cref{eq:gm}\n\n\n\t\\lWhile{\\cref{eq:armijo} is not satisfied}\n\t{$\\alpha_t \\leftarrow \\beta \\alpha_t$}\n\t$x^{t+1} \\leftarrow x^t + \\alpha_t p^t$\n}\n\\end{algorithm}\n","start":25696,"end":26238},{"raw_tex":"\\subsection{Basic Properties}\nUnder \\cref{assum:Hbound}, \\cref{eq:Qdef} is $m$-strongly convex with\nrespect to $p$, so the following standard results hold for any $\\tau\n\\in (0, 1/M]$ and any $p^t \\in \\H$ \\citep{Nes18a,DruL16a}.\n\\begin{gather}\n\\|r^t\\|^2 \\geq 2 m \\left( Q_t(p^t) - Q_t^* \\right)\n\\geq \\frac{m}{M} G^\\tau_{H_t}\\left(\n\tp^t;x^t \\right)^2,\n\\label{eq:PG}\\\\\nG^\\tau_{H_t}\\left( p^t;x^t \\right)^2 \\geq \\tau \\left( \\frac{(2m^{-1} +\n\t\\tau)(1 + M \\tau)}{\\tau} - \\frac{1}{2} \\right)^{-1}\n\t\\left(Q_t\\left( \\bar p^t_\\tau\\right) - Q_t^* \\right).\n\\label{eq:EB}\n\\end{gather}\n","start":26240,"end":26812},{"raw_tex":"A central focus of this work is manifold identification, so we first\nformally define manifolds following \\cite{Vai84a}.\nA set $\\M \\in \\R^m$ is a $p$-dimensional $\\mathcal{C}^k$ manifold\naround $x \\in \\R^m$ if there is a $\\mathcal{C}^k$ function $\\Phi: \\R^m\n\\rightarrow \\R^{m-p}$ whose derivative at $x$ is surjective such that for\n$y$ close enough to $x$, $y \\in \\M$ if and only if $\\Phi(y) = 0$.\nThrough the implicit function theorem, we can also use a\n$\\mathcal{C}^k$ parameterization $\\phi: \\R^p \\rightarrow \\M$, with\n$\\phi(y) = x$ and the derivative injective at $y$, to describe a\nneighborhood of $x$ on $\\M$.\nIn our definition and proof below, we will require the function being\n(subdifferentially) regular and prox-regular at a point $x$.\nThese two conditions are satisfied by convex functions everywhere and\nthus we do not provide their definitions here, and interested readers\ncan refer to \\cite[][Definition~7.25, Definition~13.27]{RocW09a} for details.\nNow we are ready for the definition of partial smoothness\n\\citep{Lew02a} that we assume for the regularizer when discussing\nmanifold identification.\n\\begin{definition}[Partly smooth]\n\t\\label{def:ps}\n\tA function $\\Psi$ is partly\n\tsmooth at a point $x^*$ relative to a set $\\M$ containing $x^*$ if\n\t$\\partial \\Psi(x^*) \\neq \\emptyset$ and:\n\t\\begin{enumerate}\n\t\t\t\\item Around $x^*$, $\\M$ is a $\\mathcal{C}^2$-manifold and\n\t\t\t\t$\\Psi|_{\\M}$ is $\\mathcal{C}^2$.\n\t\t\\item The affine span of $\\partial \\Psi(x)$ is a translate\n\t\t\tof the normal space to $\\M$ at $x^*$.\n\t\t\\item $\\partial \\Psi$ is continuous at $x^*$ relative to $\\M$.\n\t\t\\item \\modify{$\\Psi$ is regular at all points $x \\in \\M$ with\n\t\t\t$\\partial \\Psi(x) \\neq  \\emptyset$\n\t\taround $x^*$.}\n\t\\end{enumerate}\n\\end{definition}\nIntuitively, it means $\\Psi$ is smooth around\n$x^*$ in $\\M$ but changes drastically along directions leaving the\nmanifold.\nWe also call this $\\M$ the active manifold.\n","start":27228,"end":29135},{"raw_tex":"We therefore provide another sufficient condition for \\ipvm to\nidentify the active manifold that is satisfied by most of the\nwidely-used solvers for \\cref{eq:quadratic},\nshowing that \\ipvm essentially possesses the manifold identification\nproperty.\nThis result uses the condition \\cref{eq:subprob}, which is weaker than\n\\cref{eq:subnorm}, and we follow \\cite{LeeW18a} to define\n$\\{\\epsilon_t\\}$ using a given $\\eta \\in [0,1)$:\n\\begin{equation}\n\t\\epsilon_t = \\eta \\left( Q_t(0) - Q_t^* \\right) = -\\eta Q_t^*.\n\t\\label{eq:multiplicative}\n\\end{equation}\nAs argued in \\cite{LeeW18a} and practically adopted in various\nimplementations including \\cite{LeeC17a,DunLGBHJ18a,LeeLW19a},\nit is easy to ensure that \\cref{eq:subprob} with\n\\cref{eq:multiplicative} holds for some $\\eta <1$ under\n\t\\cref{assum:Hbound} (although the explicit value might be unknown)\nif we apply a linear-convergent\nsubproblem solver with at least a pre-specified number of iterations\nto \\cref{eq:quadratic}.\n","start":36322,"end":37296}]},"statement":{"raw_tex":"\\begin{theorem}\nConsider the setting of \\cref{thm:identify1}.\nIf \\cref{alg:vm} is run with \\cref{eq:subprob} and\n\\cref{eq:multiplicative} for some $\\eta \\in [0,1)$,\n\t\\cref{assum:Hbound} holds,\nand the update direction $p^{t}$ satisfies\n\\begin{equation}\n\t\\label{eq:prox}\n\tx^t + p^{t} = \\prox^{\\Lambda_{t}}_{\\Psi}\\left(y^{t} -\n\t\\Lambda_{t}^{-1} \\left( \\nabla f\\left( x^t \\right) + H_t \\left( y^{t} -\n\tx^t \\right) + s^{t} \\right) \\right),\n\\end{equation}\nwhere $s^{t}$ satisfies $\\|s^{t}\\| \\leq R\\left( \\norm{y^{t}\n- (x^t + p^{t*})} \\right)$ for some \\modify{$R:[0,\\infty) \\rightarrow\n[0,\\infty)$ continuous in its domain} with $R(0) = 0$,\n$\\Lambda_{t} \\in \\SS$ with $M_1 \\succeq \\Lambda_{t}$ for $M_1 >\n0$, and $y^{t}$ satisfies\n\\begin{align}\n\\label{eq:bdd0}\n\\norm{\\left(y^{t} - x^t\\right) - p^{t*}} &\\leq \\eta_1 \\left(Q_t(0) -\nQ_t^*\\right)^\\nu\n\\end{align}\nfor some $\\nu > 0$ and $\\eta_1 \\geq 0$,\nthen there exists $\\epsilon, \\delta > 0$ such that $\\|x^t - x^*\\| \\leq\n\\delta, |Q_t^*| \\leq \\epsilon$, and $\\alpha_t = 1$ imply $x^{t+1} \\in \\M$.\n\\label{thm:identify2}\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\nConsider the setting of \\cref{thm:identify1}.\nIf \\cref{alg:vm} is run with \\cref{eq:subprob} and\n\\cref{eq:multiplicative} for some $\\eta \\in [0,1)$,\n\t\\cref{assum:Hbound} holds,\nand the update direction $p^{t}$ satisfies\n\\begin{equation}\n\t\\label{eq:prox}\n\tx^t + p^{t} = \\prox^{\\Lambda_{t}}_{\\Psi}\\left(y^{t} -\n\t\\Lambda_{t}^{-1} \\left( \\nabla f\\left( x^t \\right) + H_t \\left( y^{t} -\n\tx^t \\right) + s^{t} \\right) \\right),\n\\end{equation}\nwhere $s^{t}$ satisfies $\\|s^{t}\\| \\leq R\\left( \\norm{y^{t}\n- (x^t + p^{t*})} \\right)$ for some \\modify{$R:[0,\\infty) \\rightarrow\n[0,\\infty)$ continuous in its domain} with $R(0) = 0$,\n$\\Lambda_{t} \\in \\SS$ with $M_1 \\succeq \\Lambda_{t}$ for $M_1 >\n0$, and $y^{t}$ satisfies\n\\begin{align}\n\\label{eq:bdd0}\n\\norm{\\left(y^{t} - x^t\\right) - p^{t*}} &\\leq \\eta_1 \\left(Q_t(0) -\nQ_t^*\\right)^\\nu\n\\end{align}\nfor some $\\nu > 0$ and $\\eta_1 \\geq 0$,\nthen there exists $\\epsilon, \\delta > 0$ such that $\\|x^t - x^*\\| \\leq\n\\delta, |Q_t^*| \\leq \\epsilon$, and $\\alpha_t = 1$ imply $x^{t+1} \\in \\M$.\n\\label{thm:identify2}\n\\end{theorem}","start":38004,"end":39079}]},"proof":{"raw_tex":"\\begin{lemma}\nConsider \\cref{eq:f} with $f\\in \\mathcal{C}^1$ and $\\Psi$\npartly\nsmooth at a point $x^*$ relative to a $\\mathcal{C}^2$-manifold\n$\\M$.\nIf $\\Psi$ is prox-regular at $x^*$,\nthe nondegenerate condition\n\\begin{equation}\n0 \\in \\relint \\left(\\partial F(x^*)\\right) = \\nabla f(x^*) + \\relint\n\\left(\\partial \\Psi(x^*)\\right)\n\\label{eq:nod}\n\\end{equation}\nholds, and there is a sequence $\\{x^t\\}$ converging to $x^*$ with\n$F(x^t) \\rightarrow F(x^*)$,\nthen\n\\[\n\t\\dist\\left( 0, \\partial F(x^t) \\right) \\rightarrow 0\n\t\\quad \\Leftrightarrow \\quad x^t \\in \\M \\text{ for all t large}.\n\\]\n\\label{lemma:identify0}\n\\end{lemma}\n\n\\begin{proof}\nWe first observe that as $f \\in \\mathcal{C}^1$, $x^t \\rightarrow x^*$\nimplies $f(x^t) \\rightarrow f(x^*)$,\nwhose combination with $F(x^t) \\rightarrow F(x^*)$ further implies\n$\\Psi(x^t) \\rightarrow \\Psi(x^*)$.\nTherefore, the premises of \\cite[Theorem~4.10]{LewZ13a} on $\\Psi$ are\nsatisfied.\nWe then note that\n\\begin{equation}\n\t\\label{eq:convdist}\n\t\\dist\\left(0, \\partial F(x^t)\\right) \\rightarrow 0 \\quad\n\t\\Leftrightarrow \\quad\n\t\\dist\\left(-\\nabla f(x^t), \\partial \\Psi(x^t)\\right) \\rightarrow 0.\n\\end{equation}\nAgain from that $f \\in \\mathcal{C}^1$, $x^k \\rightarrow x^*$ implies\n$\\nabla f(x^k) \\rightarrow \\nabla f(x^*)$,\nso by \\cref{eq:nod}, \\cref{eq:convdist} is\nfurther equivalent to\n\\[\n\t\\dist\\left(-\\nabla f(x^*), \\partial \\Psi(x^t)\\right) \\rightarrow\n\t0,\n\\]\nwhich is the necessary and sufficient condition for $x^t \\in \\M$ for\nall $t$ large in \\cite[Theorem~4.10]{LewZ13a} because \\cref{eq:nod}\nindicates that $- \\nabla f(x^*) \\in \\relint(\\partial \\Psi(x^*))$.\nWe then apply that theorem to obtain the desired result.\nHere we note that for the requirements\n\\qed\n\\end{proof}\n\n\\begin{lemma}\n\tConsider \\cref{eq:f} with \\modify{$f \\in \\mathcal{C}^1$} and $\\Psi \\in\n\\Gamma_0$.\nIf $\\Psi$ is partly smooth relative to a manifold $\\M$ at a point\n\\modify{$x^*$ satisfying \\cref{eq:nod}}, and there is a sequence $\\{x^t\\}$\nconverging to $x^*$, then we have\n\\[\n\t\\dist\\left( 0, \\partial F(x^t) \\right) \\rightarrow 0\n\t\\quad \\Rightarrow \\quad x^t \\in \\M \\text{ for all t large}.\n\\]\n\\label{lemma:identify}\n\\end{lemma}\n\n\\begin{proof}\nAs $\\Psi \\in \\Gamma_0$, it is subdifferentially continuous at\n$x^* \\in \\dom(\\Psi)$ by \\cite[Example 13.30]{RocW09a}.\nThus, $\\dist(0, \\partial F(x^t)) \\rightarrow 0 \\in \\partial F(x^*)$\nand $x^t \\rightarrow x^*$ imply that $F(x^t) \\rightarrow F(x^*)$.\nThe desired result is then obtained by applying\n\\modify{\\cref{lemma:identify0}}.\n\\qed\n\\end{proof}\n\n\\begin{theorem}\nConsider a point $x^*$ satisfying \\cref{eq:nod} with $\\Psi \\in \\Gamma_0$\npartly smooth at $x^*$ relative to some manifold $\\M$.\nAssume $f$ is locally $L$-smooth\nfor $L > 0$ around $x^*$.\nIf \\cref{alg:vm} is run with the condition \\cref{eq:partial} and\n\\cref{assum:Hbound} holds,\nthen\nthere exist $\\epsilon, \\delta > 0$ such that\n$\\|\\modify{x^t} - x^* \\| \\leq \\delta, \\epsilon_t \\leq \\epsilon$, and $\\alpha_t = 1$\nimply $x^{t+1} \\in \\M$.\n\\label{thm:identify1}\n\\end{theorem}\n\n\\begin{proof}\nSince each iteration of \\cref{alg:vm} is independent of the previous\nones, we abuse the notation to let $x^t$ be the input of\n\\cref{alg:vm} at the $t$th iteration and $p^t$ the corresponding\ninexact solution to \\cref{eq:quadratic}, but $x^{t+1}$ is\nirrelevant to $p^t$ or $\\alpha_t$.\nAssume for contradiction the statement is false.\nThen there exist a sequence $\\{x^t\\}\\subset \\H$ converging to $x^*$, a\nnonnegative sequence $\\{\\epsilon_t\\}$ converging to $0$, a sequence\n$\\{H_t\\} \\subset \\SS$ satisfying \\cref{assum:Hbound}, and a sequence\n$\\{p^t\\}\\subset \\H$ such that $Q_{H_t}(p^t;x^t)$ in \\cref{eq:Qdef}\nsatisfies\n\\cref{eq:partial} for all $t$, yet $x^t + p^t \\notin \\M$ for all $t$.\nFrom \\cref{eq:subnorm},\n\\begin{equation}\n\tr^t - \\nabla f\\left( x^t \\right) - H_t p^t\n\t\\in \\partial \\Psi\\left( x^{t} + p^t \\right), \\quad \\forall t \\ge 0.\n\\label{eq:r}\n\\end{equation}\nTherefore, we get from \\cref{eq:r} that\n\\begin{align}\n\t\\nonumber\n\t\\dist\\left(0, \\partial F\\left(x^t + p^t \\right) \\right) &=\n\t\\dist\\left(-\\nabla f\\left( x^t + p^t \\right), \\partial \\Psi\\left(\n\tx^t + p^t \\right)\\right)\\\\\n\t\\nonumber\n\t&\\leq \\norm{-\\nabla f\\left( x^t + p^t\\right) - \\left(r^{t} - \\nabla\n\tf\\left( x^{t} \\right) - H_{t} p^t \\right)}\\\\\n\t\\nonumber\n\t&\\leq \\norm{\\nabla f\\left( x^t + p^t\\right) - \\nabla f\\left( x^{t}\n\t\\right)} + \\norm{r^{t}} +\n\t\\norm{H_{t}} \\norm{p^t}\\\\\n\t&\\leq \\left(L + M \\right) \\norm{p^t} + \\norm{r^{t}}.\n\t\\label{eq:dist}\n\\end{align}\nFrom \\cref{assum:Hbound} and the convexity of $\\Psi$,\n$Q_{H_t}(\\cdot;x^t)$ is $m$-strongly convex, which implies\n\\begin{equation}\n\tQ_{H_t}\\left(p;x^t \\right) - Q_{H_t}\\left( p^{t*};x^t \\right) \\geq\n\\frac{m}{2} \\norm{p - p^{t*}}^2, \\quad  \\forall p \\in \\H.\n\\label{eq:qg}\n\\end{equation}\nCombining \\cref{eq:qg} and \\cref{eq:PG} shows that\n\\begin{equation}\n\t\\label{eq:pbound}\n\tm^{-1} \\|r^t\\| + \\|p^{t*}\\| \\geq \\|p^t\\|.\n\\end{equation}\nSince $\\{x^t\\}$ converges to $x^*$, by the argument in\n\\cite[Lemma~3.2]{Wri12a}, we get\n\\begin{equation}\np^{t*} = O(\\|x^t - x^*\\|)\n\\label{eq:pt}\n\\end{equation}\nwhenever $x^t$ is close enough to $x^*$.\nThus \\cref{eq:pbound}, \\cref{eq:pt}, \\cref{eq:partial}, and that\n$\\epsilon_t \\downarrow 0$ imply\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty} \\|p^t\\| = 0,\n\\label{eq:pt2}\n\\end{equation}\nSubstituting \\cref{eq:partial} and \\cref{eq:pt2} into \\cref{eq:dist}\ngives $\\dist(0, \\partial F(x^t)) \\rightarrow 0$, and by \\cref{eq:pt2}\nwe also get $ x^t + p^t \\rightarrow x^* + 0 = x^*$.  Therefore\n\\cref{lemma:identify} implies that\n$x^{t} + p^{t} \\in \\M$ for all $t$ large enough, proving the\ndesired contradiction.\n\\qed\n\\end{proof}\n\n\\begin{proof}\nSuppose the statement is not true for contradiction.\nThen there exist a continuous \\modify{function $R: [0, \\infty)\n\\rightarrow [0, \\infty)$ with}\n$R(0) = 0$, $\\eta_1 \\geq 0$, $M_1 > 0$, a sequence $\\{x^t\\} \\subset\n\\H$ converging to $x^*$,\na sequence $\\{H_t\\} \\subset \\SS$ satisfying \\cref{assum:Hbound} and\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty}\\, \\min_{p}\\quad Q_{H_t}\\left( p;x^t\n\\right) = 0,\n\\label{eq:Qconv}\n\\end{equation}\nthree sequences $\\{p^t\\}, \\{y^t\\}, \\{s^t\\} \\subset \\H$\nand a sequence $\\{\\Lambda_t\\}\\subset \\SS$ with $M_1 \\succeq \\Lambda_t$\nsuch that \\cref{eq:subprob} with \\cref{eq:multiplicative} and\n\\cref{eq:prox}-\\cref{eq:bdd0} hold, yet $x^t + p^t \\notin \\M$\nfor all $t$.\nWe abuse the notation to let $p^{t*}$ and $Q^*_t$\nrespectively denote the optimal solution and objective value for\n$\\min_p Q_{H_t}(p;x^t)$, but $x^{t+1}$ is irrelevant to $p^t$ or\n$\\alpha_t$.\n\nThe optimality condition of \\cref{eq:proximal} applied to\n\\cref{eq:prox} indicates that\n\\begin{equation}\n\t-\\Lambda_{t}\\left(x^{t} - y^{t}\\right) - \\left( \\nabla\n\tf\\left( x^t \\right) + H_t \\left( y^{t} - x^t \\right) + s^{t}\n\t\\right) \\in \\partial \\Psi\\left( x^{t} + p^t \\right).\n\t\\label{eq:opt}\n\\end{equation}\nThus, we have\n\\begin{align}\n\\nonumber\n&~\\dist\\left( 0, \\partial F\\left( x^t + p^t \\right) \\right)\\\\\n\\nonumber\n\\leq &~\\left\\|\\nabla f\\left( x^t + p^t\n\t\\right) -\\Lambda_{t}\\left(x^t + p^t -\n\ty^{t}\\right) - \\nabla f\\left( x^{t} \\right) \\right.\n\t\\left. -\n\tH_{t} \\left( y^{t} - x^{t} \\right) -\n\ts^{t} \\right\\|\\\\\n\\nonumber\n\\leq &~\\norm{\\nabla f\\left( x^t + p^t \\right)- \\nabla\n\tf\\left( x^{t} \\right)} + M_1 \\left( \\norm{x^t - y^{t}} +\n\t\\norm{p^t}\\right)\n\t+ M\n\t\\norm{y^{t} - x^{t}} + \\norm{s^{t}}\\\\\n\\leq &~(L + M_1) \\norm{p^t} +\n\t(M + M_1) \\norm{y^{t} - x^{t}} +\n\t\\norm{s^{t}}.\n\\label{eq:tobdd}\n\\end{align}\nFor the first two terms in \\cref{eq:tobdd}, the\ntriangle inequality and \\cref{eq:qg} imply\n\\begin{gather}\n\t\\begin{aligned}\n\\norm{p^{t}} \n\\leq \\norm{p^{t} - p^{t*}} + \\norm{p^{t*}}\n\\leq\n&~\\sqrt{2m^{-1}}\\sqrt{ Q_{t}\\left( p^{t} \\right) -\nQ^*_{t} } + \\sqrt{2m^{-1}}\\sqrt{ 0 - Q^*_{t}}\\\\\n\\stackrel{\\cref{eq:multiplicative}}{\\leq}&~ \\left( \\sqrt{\\eta} + 1\n\\right)\\sqrt{2m^{-1}}\\sqrt{ - Q^*_{t}}, \\text{ and }\n\\end{aligned}\n\\label{eq:bdd1}\\\\\n\\norm{y^{t}- x^{t}}\n\\leq \\norm{\\left(y^{t}- x^{t}\\right) - p^{t*}} +\n\\norm{p^{t*}}\n\\stackrel{\\cref{eq:bdd0}}{\\leq}\n\\eta_1 \\left( - Q^*_{t}\\right)^\\nu + \\sqrt{2m^{-1}}\\sqrt{ - Q^*_{t}}.\n\\label{eq:bdd3}\n\\end{gather}\nFor the last term in \\cref{eq:tobdd}, we have from \\modify{our definition of\n$s^t$ that\n\\begin{align}\n\\norm{s^{t}} &\\leq\nR \\left(\\norm{y^{t} - \\left(x^{t} + p^{t*}\\right)}\\right).\n\\label{eq:bdd4}\n\\end{align}\n}\nBy substituting \\cref{eq:bdd1}-\\cref{eq:bdd4} back into\n\\cref{eq:tobdd}, clearly there are $C_1, C_2 > 0$ such that\n\\begin{equation}\n\\dist\\left( 0, \\partial F\\left( x^t + p^t \\right) \\right)\n\\leq\nC_1 \\sqrt{ - Q^*_t} + C_2 \\left( -Q^*_t \\right)^\\nu \\modify{+\n\tR \\left(\\norm{y^{t} - \\left(x^{t} + p^{t*}\\right)}\\right).}\n\\label{eq:tobdd2}\n\\end{equation}\nNote that $Q_{H_t}(0;x^t) \\equiv 0$, so $-Q^*_t \\geq 0$ from its\noptimality and the right-hand side of  \\cref{eq:tobdd2} is\nwell-defined.\n\\modify{Next, we see from \\cref{eq:Qconv}, \\cref{eq:bdd0}, and the\ncontinuity of $R$ that\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty} \\norm{\\left(y^{t} - x^t\\right) - p^{t*}} =\n0, \\quad \\Rightarrow \\quad\n\\lim_{t \\rightarrow \\infty} R\\left( \\norm{\\left(y^{t} - x^t\\right) -\np^{t*}}\\right) =\n0.\n\\label{eq:sconv}\n\\end{equation}\nApplying \\cref{eq:Qconv} and \\cref{eq:sconv}} to \\cref{eq:tobdd2} and\nletting $t$ approach infinity then yield\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty} \\dist\\left( 0, \\partial F\\left(\nx^t + p^t \\right) \\right) = 0.\n\\label{eq:partialconv}\n\\end{equation}\n\nNext, from \\cref{eq:bdd1} and \\cref{eq:Qconv}, it is also clear that\n$\\norm{p^t} \\rightarrow 0$,\nso from the convergence of $x^t$ to $x^*$ we have\n\\begin{equation}\nx^t + p^t \\rightarrow x^* + 0 = x^*.\n\\label{eq:convxp}\n\\end{equation}\nNow \\cref{eq:convxp} and \\cref{eq:partialconv} allow us to apply\n\\cref{lemma:identify}\nso $x^t + p^t \\in \\M$ for all $t$ large enough,\nleading to the desired contradiction.\n\\qed\n\\end{proof}","units":[{"raw_tex":"\\begin{lemma}\nConsider \\cref{eq:f} with $f\\in \\mathcal{C}^1$ and $\\Psi$\npartly\nsmooth at a point $x^*$ relative to a $\\mathcal{C}^2$-manifold\n$\\M$.\nIf $\\Psi$ is prox-regular at $x^*$,\nthe nondegenerate condition\n\\begin{equation}\n0 \\in \\relint \\left(\\partial F(x^*)\\right) = \\nabla f(x^*) + \\relint\n\\left(\\partial \\Psi(x^*)\\right)\n\\label{eq:nod}\n\\end{equation}\nholds, and there is a sequence $\\{x^t\\}$ converging to $x^*$ with\n$F(x^t) \\rightarrow F(x^*)$,\nthen\n\\[\n\t\\dist\\left( 0, \\partial F(x^t) \\right) \\rightarrow 0\n\t\\quad \\Leftrightarrow \\quad x^t \\in \\M \\text{ for all t large}.\n\\]\n\\label{lemma:identify0}\n\\end{lemma}","start":29434,"end":30054},{"raw_tex":"\\begin{proof}\nWe first observe that as $f \\in \\mathcal{C}^1$, $x^t \\rightarrow x^*$\nimplies $f(x^t) \\rightarrow f(x^*)$,\nwhose combination with $F(x^t) \\rightarrow F(x^*)$ further implies\n$\\Psi(x^t) \\rightarrow \\Psi(x^*)$.\nTherefore, the premises of \\cite[Theorem~4.10]{LewZ13a} on $\\Psi$ are\nsatisfied.\nWe then note that\n\\begin{equation}\n\t\\label{eq:convdist}\n\t\\dist\\left(0, \\partial F(x^t)\\right) \\rightarrow 0 \\quad\n\t\\Leftrightarrow \\quad\n\t\\dist\\left(-\\nabla f(x^t), \\partial \\Psi(x^t)\\right) \\rightarrow 0.\n\\end{equation}\nAgain from that $f \\in \\mathcal{C}^1$, $x^k \\rightarrow x^*$ implies\n$\\nabla f(x^k) \\rightarrow \\nabla f(x^*)$,\nso by \\cref{eq:nod}, \\cref{eq:convdist} is\nfurther equivalent to\n\\[\n\t\\dist\\left(-\\nabla f(x^*), \\partial \\Psi(x^t)\\right) \\rightarrow\n\t0,\n\\]\nwhich is the necessary and sufficient condition for $x^t \\in \\M$ for\nall $t$ large in \\cite[Theorem~4.10]{LewZ13a} because \\cref{eq:nod}\nindicates that $- \\nabla f(x^*) \\in \\relint(\\partial \\Psi(x^*))$.\nWe then apply that theorem to obtain the desired result.\nHere we note that for the requirements\n\\qed\n\\end{proof}","start":30066,"end":31159},{"raw_tex":"\\begin{lemma}\n\tConsider \\cref{eq:f} with \\modify{$f \\in \\mathcal{C}^1$} and $\\Psi \\in\n\\Gamma_0$.\nIf $\\Psi$ is partly smooth relative to a manifold $\\M$ at a point\n\\modify{$x^*$ satisfying \\cref{eq:nod}}, and there is a sequence $\\{x^t\\}$\nconverging to $x^*$, then we have\n\\[\n\t\\dist\\left( 0, \\partial F(x^t) \\right) \\rightarrow 0\n\t\\quad \\Rightarrow \\quad x^t \\in \\M \\text{ for all t large}.\n\\]\n\\label{lemma:identify}\n\\end{lemma}","start":31407,"end":31834},{"raw_tex":"\\begin{proof}\nAs $\\Psi \\in \\Gamma_0$, it is subdifferentially continuous at\n$x^* \\in \\dom(\\Psi)$ by \\cite[Example 13.30]{RocW09a}.\nThus, $\\dist(0, \\partial F(x^t)) \\rightarrow 0 \\in \\partial F(x^*)$\nand $x^t \\rightarrow x^*$ imply that $F(x^t) \\rightarrow F(x^*)$.\nThe desired result is then obtained by applying\n\\modify{\\cref{lemma:identify0}}.\n\\qed\n\\end{proof}","start":31835,"end":32197},{"raw_tex":"\\begin{theorem}\nConsider a point $x^*$ satisfying \\cref{eq:nod} with $\\Psi \\in \\Gamma_0$\npartly smooth at $x^*$ relative to some manifold $\\M$.\nAssume $f$ is locally $L$-smooth\nfor $L > 0$ around $x^*$.\nIf \\cref{alg:vm} is run with the condition \\cref{eq:partial} and\n\\cref{assum:Hbound} holds,\nthen\nthere exist $\\epsilon, \\delta > 0$ such that\n$\\|\\modify{x^t} - x^* \\| \\leq \\delta, \\epsilon_t \\leq \\epsilon$, and $\\alpha_t = 1$\nimply $x^{t+1} \\in \\M$.\n\\label{thm:identify1}\n\\end{theorem}","start":32727,"end":33215},{"raw_tex":"\\begin{proof}\nSince each iteration of \\cref{alg:vm} is independent of the previous\nones, we abuse the notation to let $x^t$ be the input of\n\\cref{alg:vm} at the $t$th iteration and $p^t$ the corresponding\ninexact solution to \\cref{eq:quadratic}, but $x^{t+1}$ is\nirrelevant to $p^t$ or $\\alpha_t$.\nAssume for contradiction the statement is false.\nThen there exist a sequence $\\{x^t\\}\\subset \\H$ converging to $x^*$, a\nnonnegative sequence $\\{\\epsilon_t\\}$ converging to $0$, a sequence\n$\\{H_t\\} \\subset \\SS$ satisfying \\cref{assum:Hbound}, and a sequence\n$\\{p^t\\}\\subset \\H$ such that $Q_{H_t}(p^t;x^t)$ in \\cref{eq:Qdef}\nsatisfies\n\\cref{eq:partial} for all $t$, yet $x^t + p^t \\notin \\M$ for all $t$.\nFrom \\cref{eq:subnorm},\n\\begin{equation}\n\tr^t - \\nabla f\\left( x^t \\right) - H_t p^t\n\t\\in \\partial \\Psi\\left( x^{t} + p^t \\right), \\quad \\forall t \\ge 0.\n\\label{eq:r}\n\\end{equation}\nTherefore, we get from \\cref{eq:r} that\n\\begin{align}\n\t\\nonumber\n\t\\dist\\left(0, \\partial F\\left(x^t + p^t \\right) \\right) &=\n\t\\dist\\left(-\\nabla f\\left( x^t + p^t \\right), \\partial \\Psi\\left(\n\tx^t + p^t \\right)\\right)\\\\\n\t\\nonumber\n\t&\\leq \\norm{-\\nabla f\\left( x^t + p^t\\right) - \\left(r^{t} - \\nabla\n\tf\\left( x^{t} \\right) - H_{t} p^t \\right)}\\\\\n\t\\nonumber\n\t&\\leq \\norm{\\nabla f\\left( x^t + p^t\\right) - \\nabla f\\left( x^{t}\n\t\\right)} + \\norm{r^{t}} +\n\t\\norm{H_{t}} \\norm{p^t}\\\\\n\t&\\leq \\left(L + M \\right) \\norm{p^t} + \\norm{r^{t}}.\n\t\\label{eq:dist}\n\\end{align}\nFrom \\cref{assum:Hbound} and the convexity of $\\Psi$,\n$Q_{H_t}(\\cdot;x^t)$ is $m$-strongly convex, which implies\n\\begin{equation}\n\tQ_{H_t}\\left(p;x^t \\right) - Q_{H_t}\\left( p^{t*};x^t \\right) \\geq\n\\frac{m}{2} \\norm{p - p^{t*}}^2, \\quad  \\forall p \\in \\H.\n\\label{eq:qg}\n\\end{equation}\nCombining \\cref{eq:qg} and \\cref{eq:PG} shows that\n\\begin{equation}\n\t\\label{eq:pbound}\n\tm^{-1} \\|r^t\\| + \\|p^{t*}\\| \\geq \\|p^t\\|.\n\\end{equation}\nSince $\\{x^t\\}$ converges to $x^*$, by the argument in\n\\cite[Lemma~3.2]{Wri12a}, we get\n\\begin{equation}\np^{t*} = O(\\|x^t - x^*\\|)\n\\label{eq:pt}\n\\end{equation}\nwhenever $x^t$ is close enough to $x^*$.\nThus \\cref{eq:pbound}, \\cref{eq:pt}, \\cref{eq:partial}, and that\n$\\epsilon_t \\downarrow 0$ imply\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty} \\|p^t\\| = 0,\n\\label{eq:pt2}\n\\end{equation}\nSubstituting \\cref{eq:partial} and \\cref{eq:pt2} into \\cref{eq:dist}\ngives $\\dist(0, \\partial F(x^t)) \\rightarrow 0$, and by \\cref{eq:pt2}\nwe also get $ x^t + p^t \\rightarrow x^* + 0 = x^*$.  Therefore\n\\cref{lemma:identify} implies that\n$x^{t} + p^{t} \\in \\M$ for all $t$ large enough, proving the\ndesired contradiction.\n\\qed\n\\end{proof}","start":33216,"end":35815},{"raw_tex":"\\begin{proof}\nSuppose the statement is not true for contradiction.\nThen there exist a continuous \\modify{function $R: [0, \\infty)\n\\rightarrow [0, \\infty)$ with}\n$R(0) = 0$, $\\eta_1 \\geq 0$, $M_1 > 0$, a sequence $\\{x^t\\} \\subset\n\\H$ converging to $x^*$,\na sequence $\\{H_t\\} \\subset \\SS$ satisfying \\cref{assum:Hbound} and\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty}\\, \\min_{p}\\quad Q_{H_t}\\left( p;x^t\n\\right) = 0,\n\\label{eq:Qconv}\n\\end{equation}\nthree sequences $\\{p^t\\}, \\{y^t\\}, \\{s^t\\} \\subset \\H$\nand a sequence $\\{\\Lambda_t\\}\\subset \\SS$ with $M_1 \\succeq \\Lambda_t$\nsuch that \\cref{eq:subprob} with \\cref{eq:multiplicative} and\n\\cref{eq:prox}-\\cref{eq:bdd0} hold, yet $x^t + p^t \\notin \\M$\nfor all $t$.\nWe abuse the notation to let $p^{t*}$ and $Q^*_t$\nrespectively denote the optimal solution and objective value for\n$\\min_p Q_{H_t}(p;x^t)$, but $x^{t+1}$ is irrelevant to $p^t$ or\n$\\alpha_t$.\n\nThe optimality condition of \\cref{eq:proximal} applied to\n\\cref{eq:prox} indicates that\n\\begin{equation}\n\t-\\Lambda_{t}\\left(x^{t} - y^{t}\\right) - \\left( \\nabla\n\tf\\left( x^t \\right) + H_t \\left( y^{t} - x^t \\right) + s^{t}\n\t\\right) \\in \\partial \\Psi\\left( x^{t} + p^t \\right).\n\t\\label{eq:opt}\n\\end{equation}\nThus, we have\n\\begin{align}\n\\nonumber\n&~\\dist\\left( 0, \\partial F\\left( x^t + p^t \\right) \\right)\\\\\n\\nonumber\n\\leq &~\\left\\|\\nabla f\\left( x^t + p^t\n\t\\right) -\\Lambda_{t}\\left(x^t + p^t -\n\ty^{t}\\right) - \\nabla f\\left( x^{t} \\right) \\right.\n\t\\left. -\n\tH_{t} \\left( y^{t} - x^{t} \\right) -\n\ts^{t} \\right\\|\\\\\n\\nonumber\n\\leq &~\\norm{\\nabla f\\left( x^t + p^t \\right)- \\nabla\n\tf\\left( x^{t} \\right)} + M_1 \\left( \\norm{x^t - y^{t}} +\n\t\\norm{p^t}\\right)\n\t+ M\n\t\\norm{y^{t} - x^{t}} + \\norm{s^{t}}\\\\\n\\leq &~(L + M_1) \\norm{p^t} +\n\t(M + M_1) \\norm{y^{t} - x^{t}} +\n\t\\norm{s^{t}}.\n\\label{eq:tobdd}\n\\end{align}\nFor the first two terms in \\cref{eq:tobdd}, the\ntriangle inequality and \\cref{eq:qg} imply\n\\begin{gather}\n\t\\begin{aligned}\n\\norm{p^{t}} \n\\leq \\norm{p^{t} - p^{t*}} + \\norm{p^{t*}}\n\\leq\n&~\\sqrt{2m^{-1}}\\sqrt{ Q_{t}\\left( p^{t} \\right) -\nQ^*_{t} } + \\sqrt{2m^{-1}}\\sqrt{ 0 - Q^*_{t}}\\\\\n\\stackrel{\\cref{eq:multiplicative}}{\\leq}&~ \\left( \\sqrt{\\eta} + 1\n\\right)\\sqrt{2m^{-1}}\\sqrt{ - Q^*_{t}}, \\text{ and }\n\\end{aligned}\n\\label{eq:bdd1}\\\\\n\\norm{y^{t}- x^{t}}\n\\leq \\norm{\\left(y^{t}- x^{t}\\right) - p^{t*}} +\n\\norm{p^{t*}}\n\\stackrel{\\cref{eq:bdd0}}{\\leq}\n\\eta_1 \\left( - Q^*_{t}\\right)^\\nu + \\sqrt{2m^{-1}}\\sqrt{ - Q^*_{t}}.\n\\label{eq:bdd3}\n\\end{gather}\nFor the last term in \\cref{eq:tobdd}, we have from \\modify{our definition of\n$s^t$ that\n\\begin{align}\n\\norm{s^{t}} &\\leq\nR \\left(\\norm{y^{t} - \\left(x^{t} + p^{t*}\\right)}\\right).\n\\label{eq:bdd4}\n\\end{align}\n}\nBy substituting \\cref{eq:bdd1}-\\cref{eq:bdd4} back into\n\\cref{eq:tobdd}, clearly there are $C_1, C_2 > 0$ such that\n\\begin{equation}\n\\dist\\left( 0, \\partial F\\left( x^t + p^t \\right) \\right)\n\\leq\nC_1 \\sqrt{ - Q^*_t} + C_2 \\left( -Q^*_t \\right)^\\nu \\modify{+\n\tR \\left(\\norm{y^{t} - \\left(x^{t} + p^{t*}\\right)}\\right).}\n\\label{eq:tobdd2}\n\\end{equation}\nNote that $Q_{H_t}(0;x^t) \\equiv 0$, so $-Q^*_t \\geq 0$ from its\noptimality and the right-hand side of  \\cref{eq:tobdd2} is\nwell-defined.\n\\modify{Next, we see from \\cref{eq:Qconv}, \\cref{eq:bdd0}, and the\ncontinuity of $R$ that\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty} \\norm{\\left(y^{t} - x^t\\right) - p^{t*}} =\n0, \\quad \\Rightarrow \\quad\n\\lim_{t \\rightarrow \\infty} R\\left( \\norm{\\left(y^{t} - x^t\\right) -\np^{t*}}\\right) =\n0.\n\\label{eq:sconv}\n\\end{equation}\nApplying \\cref{eq:Qconv} and \\cref{eq:sconv}} to \\cref{eq:tobdd2} and\nletting $t$ approach infinity then yield\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty} \\dist\\left( 0, \\partial F\\left(\nx^t + p^t \\right) \\right) = 0.\n\\label{eq:partialconv}\n\\end{equation}\n\nNext, from \\cref{eq:bdd1} and \\cref{eq:Qconv}, it is also clear that\n$\\norm{p^t} \\rightarrow 0$,\nso from the convergence of $x^t$ to $x^*$ we have\n\\begin{equation}\nx^t + p^t \\rightarrow x^* + 0 = x^*.\n\\label{eq:convxp}\n\\end{equation}\nNow \\cref{eq:convxp} and \\cref{eq:partialconv} allow us to apply\n\\cref{lemma:identify}\nso $x^t + p^t \\in \\M$ for all $t$ large enough,\nleading to the desired contradiction.\n\\qed\n\\end{proof}","start":39081,"end":43250}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:f"],"declared_external":false,"source_only":false,"title":"正则化优化问题","source_units":[{"raw_tex":"Consider the following regularized optimization problem:\n\\begin{equation}\n\t\\min_{x \\in \\H}\\quad F(x) \\coloneqq f(x) + \\Psi(x),\n\t\\label{eq:f}\n\\end{equation}\nwhere $\\H$ is a Euclidean space with an inner product\n$\\inprod{\\cdot}{\\cdot}$ and its induced norm $\\norm{\\cdot}$,\nthe regularizer $\\Psi$ is extended-valued, convex, proper, and\nlower-semicontinuous, $f$ is \\modify{continuously differentiable with\nLipschitz-continuous gradients},\nand the solution set $\\Omega$ is non-empty.\nThis type of problems is ubiquitous in applications such as machine\nlearning and signal processing \\modify{(see, for example,\n\\cite{Tib96a,CanR09a,CanRT06a})}.\n","start":6048,"end":6689}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":["eq:Qdef","eq:quadratic"],"declared_external":false,"source_only":false,"title":"二次模型与更新方向","source_units":[{"raw_tex":"One widely-used method for \\cref{eq:f} is inexact successive quadratic\napproximation (\\ipvm).\nAt the $t$th iteration with iterate $x^t$,\n\\ipvm obtains the update direction $p^t$ by approximately solving\n\\begin{gather}\n\\label{eq:quadratic}\np^t \\approx \\argmin_{p \\in \\H}\\quad Q_{H_t}\\left( p; x^t\n\t\\right),\\\\\n\\label{eq:Qdef}\nQ_{H_t}\\left( p; x^t \\right) \\coloneqq \\inprod{\\nabla f\\left( x^t\n\t\\right)}{p} + \\frac12 \\inprod{p}{ H_t p} + \\Psi\\left( x^t + p\n\t\\right) - \\Psi\\left( x^t \\right),\n\\end{gather}\nwhere $H_t$ is a self-adjoint positive-semidefinite linear\nendomorphism of $\\H$.\nThe iterate is then updated along $p^t$ with a step size $\\alpha_t\n> 0$.\n","start":6689,"end":7344}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"流形识别的含义","source_units":[{"raw_tex":"A possible remedy when $\\Psi$ is partly smooth \\citep{Lew02a}\nis to switch to smooth optimization after identifying an active\nmanifold $\\M$ that contains a solution \\modify{$\\hat x$} to \\cref{eq:f}\nand makes $\\Psi$ confined to it smooth.\nWe say an algorithm can identify $\\M$ if there is a neighborhood $U$\nof \\modify{$\\hat x$} such that $x^t \\in U$ implies $x^{t+1} \\in \\M$, and call it\npossesses the manifold identification property.\n","start":8590,"end":9026}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["eq:subnorm"],"declared_external":false,"source_only":false,"title":"最小范数模型残差","source_units":[{"raw_tex":"\\begin{equation}\nr^t \\coloneqq \\argmin_{r} \\|r\\|, \\quad \\text{ subject to } \\quad r \\in\n\\partial_{p} Q_{H_t}(p^t;x^t),\n\\label{eq:subnorm}\n\\end{equation}\n","start":10306,"end":10459}]},{"id":"S005","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"最优值与集合记号","source_units":[{"raw_tex":"We denote the minimum of \\cref{eq:f} by $F^*$; the domain of $\\Psi$\nby $\\dom(\\Psi)$; and the set of convex, proper, and lower\nsemicontinuous functions by $\\Gamma_0$.\n\\modify{For any set $C$, $\\relint(C)$ denotes its relative interior.}\nWe will frequently use the following notations.\n\\[\n\t\\delta_t \\coloneqq F(x^t) - F^*,\\quad\nP_{\\Omega}(x) \\coloneqq \\argmin_{y \\in \\Omega} \\|x - y\\|,\\quad\n\t\\dist(x,\\Omega) \\coloneqq \\|x - P_\\Omega(x)\\|.\n\\]\nThe level set $\\levbet \\coloneqq \\left\\{ x\\mid  F(x) - F^* \\leq \\xi\n\\right\\}$ for any $\\xi \\geq 0$  is closed but not necessarily\n\\modify{bounded.}\n","start":19364,"end":19952}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"光滑性与算子序","source_units":[{"raw_tex":"A function is $L$-smooth if it is \\modify{differentiable with the gradient\n$L$-Lipschitz continuous}.\nWe denote the identity operator by $I$.\nFor self-adjoint linear endomorphisms $A,B$ of $\\H$,\n$A \\succ B$ ($\\succeq$) means $A-B$ is positive definite (positive\nsemidefinite).\nWe abbreviate $A \\succ \\tau I$ to $A \\succ \\tau$ for $\\tau \\in \\R$.\nThe set of $A$ with $A \\succ 0$ is denoted by $\\SS$.\n","start":19952,"end":20350}]},{"id":"S007","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"复合目标的次微分","source_units":[{"raw_tex":"The subdifferential $\\partial \\Psi(x)$ of $\\Psi$ at $x$ is\nwell-defined as $\\Psi \\in \\Gamma_0$,\nhence so is the generalized gradient $\\partial F(x) = \\nabla f(x) +\n\\partial \\Psi(x)$.\n","start":20350,"end":20533}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":["eq:proximal"],"declared_external":false,"source_only":false,"title":"广义近端映射","source_units":[{"raw_tex":"For any $g \\in \\Gamma_0$, $\\tau\\geq 0$, and \\modify{$\\Lambda \\in \\SS$},\nthe proximal mapping\n\\begin{equation}\n\\label{eq:proximal}\n\t\\prox^{\\Lambda}_{\\tau g}(x) \\coloneqq \\argmin_{y \\in \\H}\\, \\frac12\n\t\\inprod{x - y}{\\Lambda (x-y)} + \\tau g\\left( y \\right)\n\\end{equation}\nis continuous and finite in $\\H$ even outside\n$\\dom(g)$.\nWhen $\\Lambda = I$, \\cref{eq:proximal} is shorten to\n$\\prox_{\\tau g}(x)$.\n","start":20533,"end":20933}]},{"id":"S009","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"模型最优解记号","source_units":[{"raw_tex":"For \\cref{eq:quadratic}, we denote its optimal solution by\n$p^{t*}$. When there is no ambiguity, we abbreviate $Q_{H_t}(\\cdot;\nx^t)$ to $Q_t(\\cdot)$, $Q_t(p^t)$ to $\\hat Q_t$, and $Q_t(p^{t*})$ to\n$Q_t^*$.\n","start":20933,"end":21139}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","labels":["eq:armijo"],"declared_external":false,"source_only":false,"title":"Armijo 回溯规则","source_units":[{"raw_tex":"After obtaining $p^t$ from \\cref{eq:quadratic}, we need to find a\nstep size $\\alpha_t > 0$ for it  to ensure sufficient objective\ndecrease.\nGiven $\\gamma,\\beta \\in (0,1)$, we take $\\alpha_t$ as the\nlargest value in $\\{\\beta^0,\\beta^1,\\dotsc\\}$ satisfying an\nArmijo-like condition.\n\\begin{equation}\n\tF\\left( x^t + \\alpha_t p^t \\right) \\leq F\\left( x^t \\right) +\n\t\\gamma \\alpha_t Q_t(p^t).\n\t\\label{eq:armijo}\n\\end{equation}\n","start":23469,"end":23891}]},{"id":"S011","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":true,"declaration_text":["原文第559–561行引用 LeeW18a, Lemma 3；仅保留为算法背景，不作为选定识别证明的输入前提。"],"source_only":false,"title":"回溯可行性的背景","source_units":[{"raw_tex":"This condition is satisfied by all $\\alpha_t$ small enough as long as\n$Q_t(p^t) < 0$ and $Q_t(\\cdot)$ is strongly convex; see\n\\cite[Lemma~3]{LeeW18a}.\n\n","start":23891,"end":24043}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","labels":["assum:Hbound"],"declared_external":false,"source_only":false,"title":"模型算子的统一界","source_units":[{"raw_tex":"For the choice of $H_t$,\nwe only make the following blanket assumption without further\nspecification to make our analysis more general.\n\\begin{equation}\n\\exists M,m > 0,\\quad \\text{ such that } \\quad M \\succeq H_t  \\succeq\nm, \\;\\forall t \\geq 0.\n\\label{assum:Hbound}\n\\end{equation}\n","start":24043,"end":24325}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"模型目标误差准则","source_units":[{"raw_tex":"For \\cref{eq:quadratic}, any suitable solver for regularized\noptimization, such as (accelerated) proximal gradient,\n(variance-reduced) stochastic gradient methods, and their variants,\ncan be used, and the following are common for their inner loop\ntermination:\n\\begin{gather}\n\t\\hat Q_t - Q_t^* \\leq \\epsilon_t,\n\t\\label{eq:subprob}\\\\\n","start":24326,"end":24658}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":["eq:gm","eq:partial","eq:subprob"],"declared_external":false,"source_only":false,"title":"模型残差停止准则","source_units":[{"raw_tex":"\\left\\| r^t \\right\\| \\leq \\epsilon_t,\n\\quad \\text{ or }\n\t\\label{eq:partial}\\\\\n","start":24658,"end":24736}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":["eq:p"],"declared_external":false,"source_only":false,"title":"近端梯度残差","source_units":[{"raw_tex":"G^\\tau_{H_t} \\left( p^t;x^t \\right)\\coloneqq \\norm{p^t - \\bar\np^t_\\tau}\\leq \\epsilon_t,\n\t\\label{eq:gm}\n\\end{gather}\nfor some given $\\epsilon_t\\geq 0$ and $\\tau > 0$, where\n\\begin{align}\n\\label{eq:p}\n\\bar p^t_\\tau &\\coloneqq \\prox_{\\tau \\Psi}\\left(\\left(x^t + p^t \\right) -\n\t\\tau \\left(\\nabla f(x^t) + H_t p^t\\right)\\right) - x^t.\n\\end{align}\n","start":24736,"end":25078}]},{"id":"S016","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"近端梯度零点解释","source_units":[{"raw_tex":"\\modify{\nThe point $\\bar p^t_\\tau$ in \\cref{eq:p} is computed by taking a\nproximal gradient step of the subproblem \\cref{eq:quadratic} from\n$p^t$, and thus $\\bar p^t - p^t$ is \\modifyy{the proximal gradient}\n(with step size $\\tau$) of $Q_t$ at $p^t$.\nBecause the subproblem is strongly convex, the norm of the proximal\ngradient is zero at $p^t$ if and only if $p^t$ is the unique solution\nto the subproblem.\nWe will see in the next subsection that the norm of this proximal\ngradient squared is also equivalent to the objective distance to the\noptimum of the subproblem.}\n","start":25078,"end":25649}]},{"id":"S017","semantic_type":"algorithm","kind":"algorithm","labels":["alg:vm"],"declared_external":false,"source_only":false,"title":"ISQA 算法框架","source_units":[{"raw_tex":"\\begin{algorithm}[h]\n\\DontPrintSemicolon\n\\SetKwInOut{Input}{input}\\SetKwInOut{Output}{output}\n\\caption{A Framework of \\ipvm for \\cref{eq:f}}\n\\label{alg:vm}\n\\Input{$x^0 \\in \\H$, $\\gamma, \\beta \\in (0,1)$}\n\n\\For{$t=0,1,\\dotsc$}{\n\t$\\alpha_t \\leftarrow 1$,\n\tpick $\\epsilon_t \\geq 0$ and $H_t$, and solve \\cref{eq:quadratic}\n\tfor $p^t$ satisfying \\cref{eq:subprob},\n\t\\cref{eq:partial}, or \\cref{eq:gm}\n\n\n\t\\lWhile{\\cref{eq:armijo} is not satisfied}\n\t{$\\alpha_t \\leftarrow \\beta \\alpha_t$}\n\t$x^{t+1} \\leftarrow x^t + \\alpha_t p^t$\n}\n\\end{algorithm}\n","start":25696,"end":26238}]},{"id":"S018","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"模型强凸性","source_units":[{"raw_tex":"\\subsection{Basic Properties}\nUnder \\cref{assum:Hbound}, \\cref{eq:Qdef} is $m$-strongly convex with\nrespect to $p$, so the following standard results hold for any $\\tau\n\\in (0, 1/M]$ and any $p^t \\in \\H$ \\citep{Nes18a,DruL16a}.\n","start":26240,"end":26468}]},{"id":"S019","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:EB","eq:PG"],"declared_external":true,"declaration_text":["声明的先决结果：第627–640行引用 Nes18a、DruL16a 的强凸残差及误差界；此处为 eq:PG。"],"source_only":false,"title":"残差控制模型误差","source_units":[{"raw_tex":"\\begin{gather}\n\\|r^t\\|^2 \\geq 2 m \\left( Q_t(p^t) - Q_t^* \\right)\n\\geq \\frac{m}{M} G^\\tau_{H_t}\\left(\n\tp^t;x^t \\right)^2,\n\\label{eq:PG}\\\\\n","start":26468,"end":26606}]},{"id":"S020","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":true,"declaration_text":["声明的先决结果：第627–640行引用 Nes18a、DruL16a 的残差界；此处为 eq:EB。"],"source_only":false,"title":"近端步目标误差界","source_units":[{"raw_tex":"G^\\tau_{H_t}\\left( p^t;x^t \\right)^2 \\geq \\tau \\left( \\frac{(2m^{-1} +\n\t\\tau)(1 + M \\tau)}{\\tau} - \\frac{1}{2} \\right)^{-1}\n\t\\left(Q_t\\left( \\bar p^t_\\tau\\right) - Q_t^* \\right).\n\\label{eq:EB}\n\\end{gather}\n","start":26606,"end":26812}]},{"id":"S021","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"局部流形与参数化","source_units":[{"raw_tex":"A central focus of this work is manifold identification, so we first\nformally define manifolds following \\cite{Vai84a}.\nA set $\\M \\in \\R^m$ is a $p$-dimensional $\\mathcal{C}^k$ manifold\naround $x \\in \\R^m$ if there is a $\\mathcal{C}^k$ function $\\Phi: \\R^m\n\\rightarrow \\R^{m-p}$ whose derivative at $x$ is surjective such that for\n$y$ close enough to $x$, $y \\in \\M$ if and only if $\\Phi(y) = 0$.\nThrough the implicit function theorem, we can also use a\n$\\mathcal{C}^k$ parameterization $\\phi: \\R^p \\rightarrow \\M$, with\n$\\phi(y) = x$ and the derivative injective at $y$, to describe a\nneighborhood of $x$ on $\\M$.\n","start":27228,"end":27843}]},{"id":"S022","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明的 RocW09a 凸正则性及 prox-regularity 先决事实；原文第659–663行明确说明并引用 Definition 7.25、Definition 13.27。"],"source_only":false,"title":"凸函数的正则性质","source_units":[{"raw_tex":"In our definition and proof below, we will require the function being\n(subdifferentially) regular and prox-regular at a point $x$.\nThese two conditions are satisfied by convex functions everywhere and\nthus we do not provide their definitions here, and interested readers\ncan refer to \\cite[][Definition~7.25, Definition~13.27]{RocW09a} for details.\nNow we are ready for the definition of partial smoothness\n\\citep{Lew02a} that we assume for the regularizer when discussing\nmanifold identification.\n","start":27843,"end":28341}]},{"id":"S023","semantic_type":"definition","kind":"definition","labels":["def:ps"],"declared_external":false,"source_only":false,"title":"部分光滑性","source_units":[{"raw_tex":"\\begin{definition}[Partly smooth]\n\t\\label{def:ps}\n\tA function $\\Psi$ is partly\n\tsmooth at a point $x^*$ relative to a set $\\M$ containing $x^*$ if\n\t$\\partial \\Psi(x^*) \\neq \\emptyset$ and:\n\t\\begin{enumerate}\n\t\t\t\\item Around $x^*$, $\\M$ is a $\\mathcal{C}^2$-manifold and\n\t\t\t\t$\\Psi|_{\\M}$ is $\\mathcal{C}^2$.\n\t\t\\item The affine span of $\\partial \\Psi(x)$ is a translate\n\t\t\tof the normal space to $\\M$ at $x^*$.\n\t\t\\item $\\partial \\Psi$ is continuous at $x^*$ relative to $\\M$.\n\t\t\\item \\modify{$\\Psi$ is regular at all points $x \\in \\M$ with\n\t\t\t$\\partial \\Psi(x) \\neq  \\emptyset$\n\t\taround $x^*$.}\n\t\\end{enumerate}\n\\end{definition}\n","start":28341,"end":28968}]},{"id":"S024","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"活跃流形的直观含义","source_units":[{"raw_tex":"Intuitively, it means $\\Psi$ is smooth around\n$x^*$ in $\\M$ but changes drastically along directions leaving the\nmanifold.\nWe also call this $\\M$ the active manifold.\n","start":28968,"end":29135}]},{"id":"S025","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"固定精度条件的动机","source_units":[{"raw_tex":"We therefore provide another sufficient condition for \\ipvm to\nidentify the active manifold that is satisfied by most of the\nwidely-used solvers for \\cref{eq:quadratic},\nshowing that \\ipvm essentially possesses the manifold identification\nproperty.\nThis result uses the condition \\cref{eq:subprob}, which is weaker than\n\\cref{eq:subnorm}, and we follow \\cite{LeeW18a} to define\n$\\{\\epsilon_t\\}$ using a given $\\eta \\in [0,1)$:\n","start":36322,"end":36749}]},{"id":"S026","semantic_type":"definition","kind":"definition","labels":["eq:multiplicative"],"declared_external":false,"source_only":false,"title":"固定相对模型容差","source_units":[{"raw_tex":"\\begin{equation}\n\t\\epsilon_t = \\eta \\left( Q_t(0) - Q_t^* \\right) = -\\eta Q_t^*.\n\t\\label{eq:multiplicative}\n\\end{equation}\n","start":36749,"end":36872}]},{"id":"S027","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"固定内迭代数的解释","source_units":[{"raw_tex":"As argued in \\cite{LeeW18a} and practically adopted in various\nimplementations including \\cite{LeeC17a,DunLGBHJ18a,LeeLW19a},\nit is easy to ensure that \\cref{eq:subprob} with\n\\cref{eq:multiplicative} holds for some $\\eta <1$ under\n\t\\cref{assum:Hbound} (although the explicit value might be unknown)\nif we apply a linear-convergent\nsubproblem solver with at least a pre-specified number of iterations\nto \\cref{eq:quadratic}.\n","start":36872,"end":37296}]},{"id":"S028","semantic_type":"assumption","kind":"assumption","labels":["eq:nod","lemma:identify0"],"declared_external":false,"source_only":false,"title":"一般识别引理的前提","source_units":[{"raw_tex":"\\begin{lemma}\nConsider \\cref{eq:f} with $f\\in \\mathcal{C}^1$ and $\\Psi$\npartly\nsmooth at a point $x^*$ relative to a $\\mathcal{C}^2$-manifold\n$\\M$.\nIf $\\Psi$ is prox-regular at $x^*$,\nthe nondegenerate condition\n\\begin{equation}\n0 \\in \\relint \\left(\\partial F(x^*)\\right) = \\nabla f(x^*) + \\relint\n\\left(\\partial \\Psi(x^*)\\right)\n\\label{eq:nod}\n\\end{equation}\nholds, and there is a sequence $\\{x^t\\}$ converging to $x^*$ with\n$F(x^t) \\rightarrow F(x^*)$,\n","start":29434,"end":29889}]},{"id":"S029","semantic_type":"assumption","kind":"assumption","labels":["eq:nod","lemma:identify0"],"declared_external":false,"source_only":false,"title":"非退化驻点条件","source_units":[{"raw_tex":"\\begin{equation}\n0 \\in \\relint \\left(\\partial F(x^*)\\right) = \\nabla f(x^*) + \\relint\n\\left(\\partial \\Psi(x^*)\\right)\n\\label{eq:nod}\n\\end{equation}\n","start":29646,"end":29794}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","labels":["eq:nod","lemma:identify0"],"declared_external":false,"source_only":false,"title":"一般部分光滑识别引理","source_units":[{"raw_tex":"\\begin{lemma}\nConsider \\cref{eq:f} with $f\\in \\mathcal{C}^1$ and $\\Psi$\npartly\nsmooth at a point $x^*$ relative to a $\\mathcal{C}^2$-manifold\n$\\M$.\nIf $\\Psi$ is prox-regular at $x^*$,\nthe nondegenerate condition\n\\begin{equation}\n0 \\in \\relint \\left(\\partial F(x^*)\\right) = \\nabla f(x^*) + \\relint\n\\left(\\partial \\Psi(x^*)\\right)\n\\label{eq:nod}\n\\end{equation}\nholds, and there is a sequence $\\{x^t\\}$ converging to $x^*$ with\n$F(x^t) \\rightarrow F(x^*)$,\nthen\n\\[\n\t\\dist\\left( 0, \\partial F(x^t) \\right) \\rightarrow 0\n\t\\quad \\Leftrightarrow \\quad x^t \\in \\M \\text{ for all t large}.\n\\]\n\\label{lemma:identify0}\n\\end{lemma}","start":29434,"end":30054}]},{"id":"S031","semantic_type":"proof","kind":"proof","labels":["eq:convdist"],"declared_external":false,"source_only":false,"title":"一般识别引理的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nWe first observe that as $f \\in \\mathcal{C}^1$, $x^t \\rightarrow x^*$\nimplies $f(x^t) \\rightarrow f(x^*)$,\nwhose combination with $F(x^t) \\rightarrow F(x^*)$ further implies\n$\\Psi(x^t) \\rightarrow \\Psi(x^*)$.\nTherefore, the premises of \\cite[Theorem~4.10]{LewZ13a} on $\\Psi$ are\nsatisfied.\nWe then note that\n\\begin{equation}\n\t\\label{eq:convdist}\n\t\\dist\\left(0, \\partial F(x^t)\\right) \\rightarrow 0 \\quad\n\t\\Leftrightarrow \\quad\n\t\\dist\\left(-\\nabla f(x^t), \\partial \\Psi(x^t)\\right) \\rightarrow 0.\n\\end{equation}\nAgain from that $f \\in \\mathcal{C}^1$, $x^k \\rightarrow x^*$ implies\n$\\nabla f(x^k) \\rightarrow \\nabla f(x^*)$,\nso by \\cref{eq:nod}, \\cref{eq:convdist} is\nfurther equivalent to\n\\[\n\t\\dist\\left(-\\nabla f(x^*), \\partial \\Psi(x^t)\\right) \\rightarrow\n\t0,\n\\]\nwhich is the necessary and sufficient condition for $x^t \\in \\M$ for\nall $t$ large in \\cite[Theorem~4.10]{LewZ13a} because \\cref{eq:nod}\nindicates that $- \\nabla f(x^*) \\in \\relint(\\partial \\Psi(x^*))$.\nWe then apply that theorem to obtain the desired result.\nHere we note that for the requirements\n\\qed\n\\end{proof}","start":30066,"end":31159}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"推出正则项值收敛","source_units":[{"raw_tex":"\\begin{proof}\nWe first observe that as $f \\in \\mathcal{C}^1$, $x^t \\rightarrow x^*$\nimplies $f(x^t) \\rightarrow f(x^*)$,\nwhose combination with $F(x^t) \\rightarrow F(x^*)$ further implies\n$\\Psi(x^t) \\rightarrow \\Psi(x^*)$.\nTherefore, the premises of \\cite[Theorem~4.10]{LewZ13a} on $\\Psi$ are\nsatisfied.\n","start":30066,"end":30370}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":["eq:convdist"],"declared_external":false,"source_only":false,"title":"平移次微分残差","source_units":[{"raw_tex":"We then note that\n\\begin{equation}\n\t\\label{eq:convdist}\n\t\\dist\\left(0, \\partial F(x^t)\\right) \\rightarrow 0 \\quad\n\t\\Leftrightarrow \\quad\n\t\\dist\\left(-\\nabla f(x^t), \\partial \\Psi(x^t)\\right) \\rightarrow 0.\n\\end{equation}\n","start":30370,"end":30591}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"固定极限梯度向量","source_units":[{"raw_tex":"Again from that $f \\in \\mathcal{C}^1$, $x^k \\rightarrow x^*$ implies\n$\\nabla f(x^k) \\rightarrow \\nabla f(x^*)$,\nso by \\cref{eq:nod}, \\cref{eq:convdist} is\nfurther equivalent to\n\\[\n\t\\dist\\left(-\\nabla f(x^*), \\partial \\Psi(x^t)\\right) \\rightarrow\n\t0,\n\\]\n","start":30591,"end":30844}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"应用外部识别等价性","source_units":[{"raw_tex":"which is the necessary and sufficient condition for $x^t \\in \\M$ for\nall $t$ large in \\cite[Theorem~4.10]{LewZ13a} because \\cref{eq:nod}\nindicates that $- \\nabla f(x^*) \\in \\relint(\\partial \\Psi(x^*))$.\nWe then apply that theorem to obtain the desired result.\nHere we note that for the requirements\n\\qed\n\\end{proof}","start":30844,"end":31159}]},{"id":"S036","semantic_type":"assumption","kind":"assumption","labels":["lemma:identify"],"declared_external":false,"source_only":false,"title":"凸识别引理的前提","source_units":[{"raw_tex":"\\begin{lemma}\n\tConsider \\cref{eq:f} with \\modify{$f \\in \\mathcal{C}^1$} and $\\Psi \\in\n\\Gamma_0$.\nIf $\\Psi$ is partly smooth relative to a manifold $\\M$ at a point\n\\modify{$x^*$ satisfying \\cref{eq:nod}}, and there is a sequence $\\{x^t\\}$\nconverging to $x^*$, then we have\n","start":31407,"end":31679}]},{"id":"S037","semantic_type":"lemma","kind":"lemma","labels":["lemma:identify"],"declared_external":false,"source_only":false,"title":"凸正则项的识别引理","source_units":[{"raw_tex":"\\begin{lemma}\n\tConsider \\cref{eq:f} with \\modify{$f \\in \\mathcal{C}^1$} and $\\Psi \\in\n\\Gamma_0$.\nIf $\\Psi$ is partly smooth relative to a manifold $\\M$ at a point\n\\modify{$x^*$ satisfying \\cref{eq:nod}}, and there is a sequence $\\{x^t\\}$\nconverging to $x^*$, then we have\n\\[\n\t\\dist\\left( 0, \\partial F(x^t) \\right) \\rightarrow 0\n\t\\quad \\Rightarrow \\quad x^t \\in \\M \\text{ for all t large}.\n\\]\n\\label{lemma:identify}\n\\end{lemma}","start":31407,"end":31834}]},{"id":"S038","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"凸识别引理的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nAs $\\Psi \\in \\Gamma_0$, it is subdifferentially continuous at\n$x^* \\in \\dom(\\Psi)$ by \\cite[Example 13.30]{RocW09a}.\nThus, $\\dist(0, \\partial F(x^t)) \\rightarrow 0 \\in \\partial F(x^*)$\nand $x^t \\rightarrow x^*$ imply that $F(x^t) \\rightarrow F(x^*)$.\nThe desired result is then obtained by applying\n\\modify{\\cref{lemma:identify0}}.\n\\qed\n\\end{proof}","start":31835,"end":32197}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"次微分连续性补足值收敛","source_units":[{"raw_tex":"\\begin{proof}\nAs $\\Psi \\in \\Gamma_0$, it is subdifferentially continuous at\n$x^* \\in \\dom(\\Psi)$ by \\cite[Example 13.30]{RocW09a}.\nThus, $\\dist(0, \\partial F(x^t)) \\rightarrow 0 \\in \\partial F(x^*)$\nand $x^t \\rightarrow x^*$ imply that $F(x^t) \\rightarrow F(x^*)$.\n","start":31835,"end":32100}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"归约到一般识别引理","source_units":[{"raw_tex":"The desired result is then obtained by applying\n\\modify{\\cref{lemma:identify0}}.\n\\qed\n\\end{proof}","start":32100,"end":32197}]},{"id":"S041","semantic_type":"assumption","kind":"assumption","labels":["thm:identify1"],"declared_external":false,"source_only":false,"title":"识别定理的几何设置","source_units":[{"raw_tex":"\\begin{theorem}\nConsider a point $x^*$ satisfying \\cref{eq:nod} with $\\Psi \\in \\Gamma_0$\npartly smooth at $x^*$ relative to some manifold $\\M$.\nAssume $f$ is locally $L$-smooth\nfor $L > 0$ around $x^*$.\n","start":32727,"end":32930}]},{"id":"S042","semantic_type":"theorem","kind":"theorem","labels":["thm:identify1"],"declared_external":false,"source_only":false,"title":"小模型残差下的识别","source_units":[{"raw_tex":"\\begin{theorem}\nConsider a point $x^*$ satisfying \\cref{eq:nod} with $\\Psi \\in \\Gamma_0$\npartly smooth at $x^*$ relative to some manifold $\\M$.\nAssume $f$ is locally $L$-smooth\nfor $L > 0$ around $x^*$.\nIf \\cref{alg:vm} is run with the condition \\cref{eq:partial} and\n\\cref{assum:Hbound} holds,\nthen\nthere exist $\\epsilon, \\delta > 0$ such that\n$\\|\\modify{x^t} - x^* \\| \\leq \\delta, \\epsilon_t \\leq \\epsilon$, and $\\alpha_t = 1$\nimply $x^{t+1} \\in \\M$.\n\\label{thm:identify1}\n\\end{theorem}","start":32727,"end":33215}]},{"id":"S043","semantic_type":"proof","kind":"proof","labels":["eq:dist","eq:pbound","eq:pt","eq:pt2","eq:qg","eq:r"],"declared_external":false,"source_only":false,"title":"残差识别定理的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nSince each iteration of \\cref{alg:vm} is independent of the previous\nones, we abuse the notation to let $x^t$ be the input of\n\\cref{alg:vm} at the $t$th iteration and $p^t$ the corresponding\ninexact solution to \\cref{eq:quadratic}, but $x^{t+1}$ is\nirrelevant to $p^t$ or $\\alpha_t$.\nAssume for contradiction the statement is false.\nThen there exist a sequence $\\{x^t\\}\\subset \\H$ converging to $x^*$, a\nnonnegative sequence $\\{\\epsilon_t\\}$ converging to $0$, a sequence\n$\\{H_t\\} \\subset \\SS$ satisfying \\cref{assum:Hbound}, and a sequence\n$\\{p^t\\}\\subset \\H$ such that $Q_{H_t}(p^t;x^t)$ in \\cref{eq:Qdef}\nsatisfies\n\\cref{eq:partial} for all $t$, yet $x^t + p^t \\notin \\M$ for all $t$.\nFrom \\cref{eq:subnorm},\n\\begin{equation}\n\tr^t - \\nabla f\\left( x^t \\right) - H_t p^t\n\t\\in \\partial \\Psi\\left( x^{t} + p^t \\right), \\quad \\forall t \\ge 0.\n\\label{eq:r}\n\\end{equation}\nTherefore, we get from \\cref{eq:r} that\n\\begin{align}\n\t\\nonumber\n\t\\dist\\left(0, \\partial F\\left(x^t + p^t \\right) \\right) &=\n\t\\dist\\left(-\\nabla f\\left( x^t + p^t \\right), \\partial \\Psi\\left(\n\tx^t + p^t \\right)\\right)\\\\\n\t\\nonumber\n\t&\\leq \\norm{-\\nabla f\\left( x^t + p^t\\right) - \\left(r^{t} - \\nabla\n\tf\\left( x^{t} \\right) - H_{t} p^t \\right)}\\\\\n\t\\nonumber\n\t&\\leq \\norm{\\nabla f\\left( x^t + p^t\\right) - \\nabla f\\left( x^{t}\n\t\\right)} + \\norm{r^{t}} +\n\t\\norm{H_{t}} \\norm{p^t}\\\\\n\t&\\leq \\left(L + M \\right) \\norm{p^t} + \\norm{r^{t}}.\n\t\\label{eq:dist}\n\\end{align}\nFrom \\cref{assum:Hbound} and the convexity of $\\Psi$,\n$Q_{H_t}(\\cdot;x^t)$ is $m$-strongly convex, which implies\n\\begin{equation}\n\tQ_{H_t}\\left(p;x^t \\right) - Q_{H_t}\\left( p^{t*};x^t \\right) \\geq\n\\frac{m}{2} \\norm{p - p^{t*}}^2, \\quad  \\forall p \\in \\H.\n\\label{eq:qg}\n\\end{equation}\nCombining \\cref{eq:qg} and \\cref{eq:PG} shows that\n\\begin{equation}\n\t\\label{eq:pbound}\n\tm^{-1} \\|r^t\\| + \\|p^{t*}\\| \\geq \\|p^t\\|.\n\\end{equation}\nSince $\\{x^t\\}$ converges to $x^*$, by the argument in\n\\cite[Lemma~3.2]{Wri12a}, we get\n\\begin{equation}\np^{t*} = O(\\|x^t - x^*\\|)\n\\label{eq:pt}\n\\end{equation}\nwhenever $x^t$ is close enough to $x^*$.\nThus \\cref{eq:pbound}, \\cref{eq:pt}, \\cref{eq:partial}, and that\n$\\epsilon_t \\downarrow 0$ imply\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty} \\|p^t\\| = 0,\n\\label{eq:pt2}\n\\end{equation}\nSubstituting \\cref{eq:partial} and \\cref{eq:pt2} into \\cref{eq:dist}\ngives $\\dist(0, \\partial F(x^t)) \\rightarrow 0$, and by \\cref{eq:pt2}\nwe also get $ x^t + p^t \\rightarrow x^* + 0 = x^*$.  Therefore\n\\cref{lemma:identify} implies that\n$x^{t} + p^{t} \\in \\M$ for all $t$ large enough, proving the\ndesired contradiction.\n\\qed\n\\end{proof}","start":33216,"end":35815}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"构造残差准则下的坏序列","source_units":[{"raw_tex":"\\begin{proof}\nSince each iteration of \\cref{alg:vm} is independent of the previous\nones, we abuse the notation to let $x^t$ be the input of\n\\cref{alg:vm} at the $t$th iteration and $p^t$ the corresponding\ninexact solution to \\cref{eq:quadratic}, but $x^{t+1}$ is\nirrelevant to $p^t$ or $\\alpha_t$.\nAssume for contradiction the statement is false.\nThen there exist a sequence $\\{x^t\\}\\subset \\H$ converging to $x^*$, a\nnonnegative sequence $\\{\\epsilon_t\\}$ converging to $0$, a sequence\n$\\{H_t\\} \\subset \\SS$ satisfying \\cref{assum:Hbound}, and a sequence\n$\\{p^t\\}\\subset \\H$ such that $Q_{H_t}(p^t;x^t)$ in \\cref{eq:Qdef}\nsatisfies\n\\cref{eq:partial} for all $t$, yet $x^t + p^t \\notin \\M$ for all $t$.\n","start":33216,"end":33918}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["eq:r"],"declared_external":false,"source_only":false,"title":"从模型残差得到次梯度","source_units":[{"raw_tex":"From \\cref{eq:subnorm},\n\\begin{equation}\n\tr^t - \\nabla f\\left( x^t \\right) - H_t p^t\n\t\\in \\partial \\Psi\\left( x^{t} + p^t \\right), \\quad \\forall t \\ge 0.\n\\label{eq:r}\n\\end{equation}\n","start":33918,"end":34100}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["eq:dist"],"declared_external":false,"source_only":false,"title":"辅助证明的试探点残差界","source_units":[{"raw_tex":"Therefore, we get from \\cref{eq:r} that\n\\begin{align}\n\t\\nonumber\n\t\\dist\\left(0, \\partial F\\left(x^t + p^t \\right) \\right) &=\n\t\\dist\\left(-\\nabla f\\left( x^t + p^t \\right), \\partial \\Psi\\left(\n\tx^t + p^t \\right)\\right)\\\\\n\t\\nonumber\n\t&\\leq \\norm{-\\nabla f\\left( x^t + p^t\\right) - \\left(r^{t} - \\nabla\n\tf\\left( x^{t} \\right) - H_{t} p^t \\right)}\\\\\n\t\\nonumber\n\t&\\leq \\norm{\\nabla f\\left( x^t + p^t\\right) - \\nabla f\\left( x^{t}\n\t\\right)} + \\norm{r^{t}} +\n\t\\norm{H_{t}} \\norm{p^t}\\\\\n\t&\\leq \\left(L + M \\right) \\norm{p^t} + \\norm{r^{t}}.\n\t\\label{eq:dist}\n\\end{align}\n","start":34100,"end":34662}]},{"id":"S047","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:qg"],"declared_external":false,"source_only":false,"title":"模型的二次增长估计","source_units":[{"raw_tex":"From \\cref{assum:Hbound} and the convexity of $\\Psi$,\n$Q_{H_t}(\\cdot;x^t)$ is $m$-strongly convex, which implies\n\\begin{equation}\n\tQ_{H_t}\\left(p;x^t \\right) - Q_{H_t}\\left( p^{t*};x^t \\right) \\geq\n\\frac{m}{2} \\norm{p - p^{t*}}^2, \\quad  \\forall p \\in \\H.\n\\label{eq:qg}\n\\end{equation}\n","start":34662,"end":34947}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pbound"],"declared_external":false,"source_only":false,"title":"残差控制近似步长","source_units":[{"raw_tex":"Combining \\cref{eq:qg} and \\cref{eq:PG} shows that\n\\begin{equation}\n\t\\label{eq:pbound}\n\tm^{-1} \\|r^t\\| + \\|p^{t*}\\| \\geq \\|p^t\\|.\n\\end{equation}\n","start":34947,"end":35092}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pt"],"declared_external":false,"source_only":false,"title":"精确模型步的局部界","source_units":[{"raw_tex":"Since $\\{x^t\\}$ converges to $x^*$, by the argument in\n\\cite[Lemma~3.2]{Wri12a}, we get\n\\begin{equation}\np^{t*} = O(\\|x^t - x^*\\|)\n\\label{eq:pt}\n\\end{equation}\nwhenever $x^t$ is close enough to $x^*$.\n","start":35092,"end":35293}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pt2"],"declared_external":false,"source_only":false,"title":"辅助证明的近似步消失","source_units":[{"raw_tex":"Thus \\cref{eq:pbound}, \\cref{eq:pt}, \\cref{eq:partial}, and that\n$\\epsilon_t \\downarrow 0$ imply\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty} \\|p^t\\| = 0,\n\\label{eq:pt2}\n\\end{equation}\n","start":35293,"end":35478}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"辅助证明的识别矛盾","source_units":[{"raw_tex":"Substituting \\cref{eq:partial} and \\cref{eq:pt2} into \\cref{eq:dist}\ngives $\\dist(0, \\partial F(x^t)) \\rightarrow 0$, and by \\cref{eq:pt2}\nwe also get $ x^t + p^t \\rightarrow x^* + 0 = x^*$.  Therefore\n\\cref{lemma:identify} implies that\n$x^{t} + p^{t} \\in \\M$ for all $t$ large enough, proving the\ndesired contradiction.\n\\qed\n\\end{proof}","start":35478,"end":35815}]},{"id":"S052","semantic_type":"assumption","kind":"assumption","labels":["thm:identify2"],"declared_external":false,"source_only":false,"title":"主定理的算法前提","source_units":[{"raw_tex":"Consider the setting of \\cref{thm:identify1}.\nIf \\cref{alg:vm} is run with \\cref{eq:subprob} and\n\\cref{eq:multiplicative} for some $\\eta \\in [0,1)$,\n\t\\cref{assum:Hbound} holds,\nand the update direction $p^{t}$ satisfies\n","start":38020,"end":38240}]},{"id":"S053","semantic_type":"assumption","kind":"assumption","labels":["eq:prox","thm:identify2"],"declared_external":false,"source_only":false,"title":"方向的结构化近端形式","source_units":[{"raw_tex":"\\begin{equation}\n\t\\label{eq:prox}\n\tx^t + p^{t} = \\prox^{\\Lambda_{t}}_{\\Psi}\\left(y^{t} -\n\t\\Lambda_{t}^{-1} \\left( \\nabla f\\left( x^t \\right) + H_t \\left( y^{t} -\n\tx^t \\right) + s^{t} \\right) \\right),\n\\end{equation}\n","start":38240,"end":38455}]},{"id":"S054","semantic_type":"assumption","kind":"assumption","labels":["thm:identify2"],"declared_external":false,"source_only":false,"title":"连续扰动控制函数","source_units":[{"raw_tex":"where $s^{t}$ satisfies $\\|s^{t}\\| \\leq R\\left( \\norm{y^{t}\n- (x^t + p^{t*})} \\right)$ for some \\modify{$R:[0,\\infty) \\rightarrow\n[0,\\infty)$ continuous in its domain} with $R(0) = 0$,\n","start":38455,"end":38640}]},{"id":"S055","semantic_type":"assumption","kind":"assumption","labels":["thm:identify2"],"declared_external":false,"source_only":false,"title":"近端度量的上界","source_units":[{"raw_tex":"$\\Lambda_{t} \\in \\SS$ with $M_1 \\succeq \\Lambda_{t}$ for $M_1 >\n0$, and $y^{t}$ satisfies\n","start":38640,"end":38730}]},{"id":"S056","semantic_type":"assumption","kind":"assumption","labels":["eq:bdd0","thm:identify2"],"declared_external":false,"source_only":false,"title":"近端输入点的精度","source_units":[{"raw_tex":"\\begin{align}\n\\label{eq:bdd0}\n\\norm{\\left(y^{t} - x^t\\right) - p^{t*}} &\\leq \\eta_1 \\left(Q_t(0) -\nQ_t^*\\right)^\\nu\n\\end{align}\nfor some $\\nu > 0$ and $\\eta_1 \\geq 0$,\n","start":38730,"end":38898}]},{"id":"S057","semantic_type":"theorem","kind":"theorem","labels":["eq:bdd0","eq:prox","thm:identify2"],"declared_external":false,"source_only":false,"title":"结构化非精确方向的流形识别","source_units":[{"raw_tex":"\\begin{theorem}\nConsider the setting of \\cref{thm:identify1}.\nIf \\cref{alg:vm} is run with \\cref{eq:subprob} and\n\\cref{eq:multiplicative} for some $\\eta \\in [0,1)$,\n\t\\cref{assum:Hbound} holds,\nand the update direction $p^{t}$ satisfies\n\\begin{equation}\n\t\\label{eq:prox}\n\tx^t + p^{t} = \\prox^{\\Lambda_{t}}_{\\Psi}\\left(y^{t} -\n\t\\Lambda_{t}^{-1} \\left( \\nabla f\\left( x^t \\right) + H_t \\left( y^{t} -\n\tx^t \\right) + s^{t} \\right) \\right),\n\\end{equation}\nwhere $s^{t}$ satisfies $\\|s^{t}\\| \\leq R\\left( \\norm{y^{t}\n- (x^t + p^{t*})} \\right)$ for some \\modify{$R:[0,\\infty) \\rightarrow\n[0,\\infty)$ continuous in its domain} with $R(0) = 0$,\n$\\Lambda_{t} \\in \\SS$ with $M_1 \\succeq \\Lambda_{t}$ for $M_1 >\n0$, and $y^{t}$ satisfies\n\\begin{align}\n\\label{eq:bdd0}\n\\norm{\\left(y^{t} - x^t\\right) - p^{t*}} &\\leq \\eta_1 \\left(Q_t(0) -\nQ_t^*\\right)^\\nu\n\\end{align}\nfor some $\\nu > 0$ and $\\eta_1 \\geq 0$,\nthen there exists $\\epsilon, \\delta > 0$ such that $\\|x^t - x^*\\| \\leq\n\\delta, |Q_t^*| \\leq \\epsilon$, and $\\alpha_t = 1$ imply $x^{t+1} \\in \\M$.\n\\label{thm:identify2}\n\\end{theorem}","start":38004,"end":39079}]},{"id":"S058","semantic_type":"proof","kind":"proof","labels":["eq:Qconv","eq:bdd1","eq:bdd3","eq:bdd4","eq:convxp","eq:opt","eq:partialconv","eq:sconv","eq:tobdd","eq:tobdd2"],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nSuppose the statement is not true for contradiction.\nThen there exist a continuous \\modify{function $R: [0, \\infty)\n\\rightarrow [0, \\infty)$ with}\n$R(0) = 0$, $\\eta_1 \\geq 0$, $M_1 > 0$, a sequence $\\{x^t\\} \\subset\n\\H$ converging to $x^*$,\na sequence $\\{H_t\\} \\subset \\SS$ satisfying \\cref{assum:Hbound} and\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty}\\, \\min_{p}\\quad Q_{H_t}\\left( p;x^t\n\\right) = 0,\n\\label{eq:Qconv}\n\\end{equation}\nthree sequences $\\{p^t\\}, \\{y^t\\}, \\{s^t\\} \\subset \\H$\nand a sequence $\\{\\Lambda_t\\}\\subset \\SS$ with $M_1 \\succeq \\Lambda_t$\nsuch that \\cref{eq:subprob} with \\cref{eq:multiplicative} and\n\\cref{eq:prox}-\\cref{eq:bdd0} hold, yet $x^t + p^t \\notin \\M$\nfor all $t$.\nWe abuse the notation to let $p^{t*}$ and $Q^*_t$\nrespectively denote the optimal solution and objective value for\n$\\min_p Q_{H_t}(p;x^t)$, but $x^{t+1}$ is irrelevant to $p^t$ or\n$\\alpha_t$.\n\nThe optimality condition of \\cref{eq:proximal} applied to\n\\cref{eq:prox} indicates that\n\\begin{equation}\n\t-\\Lambda_{t}\\left(x^{t} - y^{t}\\right) - \\left( \\nabla\n\tf\\left( x^t \\right) + H_t \\left( y^{t} - x^t \\right) + s^{t}\n\t\\right) \\in \\partial \\Psi\\left( x^{t} + p^t \\right).\n\t\\label{eq:opt}\n\\end{equation}\nThus, we have\n\\begin{align}\n\\nonumber\n&~\\dist\\left( 0, \\partial F\\left( x^t + p^t \\right) \\right)\\\\\n\\nonumber\n\\leq &~\\left\\|\\nabla f\\left( x^t + p^t\n\t\\right) -\\Lambda_{t}\\left(x^t + p^t -\n\ty^{t}\\right) - \\nabla f\\left( x^{t} \\right) \\right.\n\t\\left. -\n\tH_{t} \\left( y^{t} - x^{t} \\right) -\n\ts^{t} \\right\\|\\\\\n\\nonumber\n\\leq &~\\norm{\\nabla f\\left( x^t + p^t \\right)- \\nabla\n\tf\\left( x^{t} \\right)} + M_1 \\left( \\norm{x^t - y^{t}} +\n\t\\norm{p^t}\\right)\n\t+ M\n\t\\norm{y^{t} - x^{t}} + \\norm{s^{t}}\\\\\n\\leq &~(L + M_1) \\norm{p^t} +\n\t(M + M_1) \\norm{y^{t} - x^{t}} +\n\t\\norm{s^{t}}.\n\\label{eq:tobdd}\n\\end{align}\nFor the first two terms in \\cref{eq:tobdd}, the\ntriangle inequality and \\cref{eq:qg} imply\n\\begin{gather}\n\t\\begin{aligned}\n\\norm{p^{t}} \n\\leq \\norm{p^{t} - p^{t*}} + \\norm{p^{t*}}\n\\leq\n&~\\sqrt{2m^{-1}}\\sqrt{ Q_{t}\\left( p^{t} \\right) -\nQ^*_{t} } + \\sqrt{2m^{-1}}\\sqrt{ 0 - Q^*_{t}}\\\\\n\\stackrel{\\cref{eq:multiplicative}}{\\leq}&~ \\left( \\sqrt{\\eta} + 1\n\\right)\\sqrt{2m^{-1}}\\sqrt{ - Q^*_{t}}, \\text{ and }\n\\end{aligned}\n\\label{eq:bdd1}\\\\\n\\norm{y^{t}- x^{t}}\n\\leq \\norm{\\left(y^{t}- x^{t}\\right) - p^{t*}} +\n\\norm{p^{t*}}\n\\stackrel{\\cref{eq:bdd0}}{\\leq}\n\\eta_1 \\left( - Q^*_{t}\\right)^\\nu + \\sqrt{2m^{-1}}\\sqrt{ - Q^*_{t}}.\n\\label{eq:bdd3}\n\\end{gather}\nFor the last term in \\cref{eq:tobdd}, we have from \\modify{our definition of\n$s^t$ that\n\\begin{align}\n\\norm{s^{t}} &\\leq\nR \\left(\\norm{y^{t} - \\left(x^{t} + p^{t*}\\right)}\\right).\n\\label{eq:bdd4}\n\\end{align}\n}\nBy substituting \\cref{eq:bdd1}-\\cref{eq:bdd4} back into\n\\cref{eq:tobdd}, clearly there are $C_1, C_2 > 0$ such that\n\\begin{equation}\n\\dist\\left( 0, \\partial F\\left( x^t + p^t \\right) \\right)\n\\leq\nC_1 \\sqrt{ - Q^*_t} + C_2 \\left( -Q^*_t \\right)^\\nu \\modify{+\n\tR \\left(\\norm{y^{t} - \\left(x^{t} + p^{t*}\\right)}\\right).}\n\\label{eq:tobdd2}\n\\end{equation}\nNote that $Q_{H_t}(0;x^t) \\equiv 0$, so $-Q^*_t \\geq 0$ from its\noptimality and the right-hand side of  \\cref{eq:tobdd2} is\nwell-defined.\n\\modify{Next, we see from \\cref{eq:Qconv}, \\cref{eq:bdd0}, and the\ncontinuity of $R$ that\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty} \\norm{\\left(y^{t} - x^t\\right) - p^{t*}} =\n0, \\quad \\Rightarrow \\quad\n\\lim_{t \\rightarrow \\infty} R\\left( \\norm{\\left(y^{t} - x^t\\right) -\np^{t*}}\\right) =\n0.\n\\label{eq:sconv}\n\\end{equation}\nApplying \\cref{eq:Qconv} and \\cref{eq:sconv}} to \\cref{eq:tobdd2} and\nletting $t$ approach infinity then yield\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty} \\dist\\left( 0, \\partial F\\left(\nx^t + p^t \\right) \\right) = 0.\n\\label{eq:partialconv}\n\\end{equation}\n\nNext, from \\cref{eq:bdd1} and \\cref{eq:Qconv}, it is also clear that\n$\\norm{p^t} \\rightarrow 0$,\nso from the convergence of $x^t$ to $x^*$ we have\n\\begin{equation}\nx^t + p^t \\rightarrow x^* + 0 = x^*.\n\\label{eq:convxp}\n\\end{equation}\nNow \\cref{eq:convxp} and \\cref{eq:partialconv} allow us to apply\n\\cref{lemma:identify}\nso $x^t + p^t \\in \\M$ for all $t$ large enough,\nleading to the desired contradiction.\n\\qed\n\\end{proof}","start":39081,"end":43250}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":["eq:Qconv"],"declared_external":false,"source_only":false,"title":"构造结构化坏序列","source_units":[{"raw_tex":"\\begin{proof}\nSuppose the statement is not true for contradiction.\nThen there exist a continuous \\modify{function $R: [0, \\infty)\n\\rightarrow [0, \\infty)$ with}\n$R(0) = 0$, $\\eta_1 \\geq 0$, $M_1 > 0$, a sequence $\\{x^t\\} \\subset\n\\H$ converging to $x^*$,\na sequence $\\{H_t\\} \\subset \\SS$ satisfying \\cref{assum:Hbound} and\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty}\\, \\min_{p}\\quad Q_{H_t}\\left( p;x^t\n\\right) = 0,\n\\label{eq:Qconv}\n\\end{equation}\nthree sequences $\\{p^t\\}, \\{y^t\\}, \\{s^t\\} \\subset \\H$\nand a sequence $\\{\\Lambda_t\\}\\subset \\SS$ with $M_1 \\succeq \\Lambda_t$\nsuch that \\cref{eq:subprob} with \\cref{eq:multiplicative} and\n\\cref{eq:prox}-\\cref{eq:bdd0} hold, yet $x^t + p^t \\notin \\M$\nfor all $t$.\nWe abuse the notation to let $p^{t*}$ and $Q^*_t$\nrespectively denote the optimal solution and objective value for\n$\\min_p Q_{H_t}(p;x^t)$, but $x^{t+1}$ is irrelevant to $p^t$ or\n$\\alpha_t$.\n","start":39081,"end":39984}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":["eq:opt"],"declared_external":false,"source_only":false,"title":"作者写出的近端最优性包含","source_units":[{"raw_tex":"The optimality condition of \\cref{eq:proximal} applied to\n\\cref{eq:prox} indicates that\n\\begin{equation}\n\t-\\Lambda_{t}\\left(x^{t} - y^{t}\\right) - \\left( \\nabla\n\tf\\left( x^t \\right) + H_t \\left( y^{t} - x^t \\right) + s^{t}\n\t\\right) \\in \\partial \\Psi\\left( x^{t} + p^t \\right).\n\t\\label{eq:opt}\n\\end{equation}\n","start":39985,"end":40293}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":["eq:tobdd"],"declared_external":false,"source_only":false,"title":"主证明的试探点残差界","source_units":[{"raw_tex":"Thus, we have\n\\begin{align}\n\\nonumber\n&~\\dist\\left( 0, \\partial F\\left( x^t + p^t \\right) \\right)\\\\\n\\nonumber\n\\leq &~\\left\\|\\nabla f\\left( x^t + p^t\n\t\\right) -\\Lambda_{t}\\left(x^t + p^t -\n\ty^{t}\\right) - \\nabla f\\left( x^{t} \\right) \\right.\n\t\\left. -\n\tH_{t} \\left( y^{t} - x^{t} \\right) -\n\ts^{t} \\right\\|\\\\\n\\nonumber\n\\leq &~\\norm{\\nabla f\\left( x^t + p^t \\right)- \\nabla\n\tf\\left( x^{t} \\right)} + M_1 \\left( \\norm{x^t - y^{t}} +\n\t\\norm{p^t}\\right)\n\t+ M\n\t\\norm{y^{t} - x^{t}} + \\norm{s^{t}}\\\\\n\\leq &~(L + M_1) \\norm{p^t} +\n\t(M + M_1) \\norm{y^{t} - x^{t}} +\n\t\\norm{s^{t}}.\n\\label{eq:tobdd}\n\\end{align}\n","start":40293,"end":40893}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"模型误差控制方向大小","source_units":[{"raw_tex":"For the first two terms in \\cref{eq:tobdd}, the\ntriangle inequality and \\cref{eq:qg} imply\n\\begin{gather}\n\t\\begin{aligned}\n\\norm{p^{t}} \n\\leq \\norm{p^{t} - p^{t*}} + \\norm{p^{t*}}\n\\leq\n&~\\sqrt{2m^{-1}}\\sqrt{ Q_{t}\\left( p^{t} \\right) -\nQ^*_{t} } + \\sqrt{2m^{-1}}\\sqrt{ 0 - Q^*_{t}}\\\\\n\\stackrel{\\cref{eq:multiplicative}}{\\leq}&~ \\left( \\sqrt{\\eta} + 1\n\\right)\\sqrt{2m^{-1}}\\sqrt{ - Q^*_{t}}, \\text{ and }\n\\end{aligned}\n\\label{eq:bdd1}\\\\\n","start":40893,"end":41329}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"近端输入点的位移界","source_units":[{"raw_tex":"\\norm{y^{t}- x^{t}}\n\\leq \\norm{\\left(y^{t}- x^{t}\\right) - p^{t*}} +\n\\norm{p^{t*}}\n\\stackrel{\\cref{eq:bdd0}}{\\leq}\n\\eta_1 \\left( - Q^*_{t}\\right)^\\nu + \\sqrt{2m^{-1}}\\sqrt{ - Q^*_{t}}.\n\\label{eq:bdd3}\n\\end{gather}\n","start":41329,"end":41543}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":["eq:bdd4"],"declared_external":false,"source_only":false,"title":"保留扰动函数控制","source_units":[{"raw_tex":"For the last term in \\cref{eq:tobdd}, we have from \\modify{our definition of\n$s^t$ that\n\\begin{align}\n\\norm{s^{t}} &\\leq\nR \\left(\\norm{y^{t} - \\left(x^{t} + p^{t*}\\right)}\\right).\n\\label{eq:bdd4}\n\\end{align}\n}\n","start":41543,"end":41753}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":["eq:tobdd2"],"declared_external":false,"source_only":false,"title":"合并为模型最优值控制","source_units":[{"raw_tex":"By substituting \\cref{eq:bdd1}-\\cref{eq:bdd4} back into\n\\cref{eq:tobdd}, clearly there are $C_1, C_2 > 0$ such that\n\\begin{equation}\n\\dist\\left( 0, \\partial F\\left( x^t + p^t \\right) \\right)\n\\leq\nC_1 \\sqrt{ - Q^*_t} + C_2 \\left( -Q^*_t \\right)^\\nu \\modify{+\n\tR \\left(\\norm{y^{t} - \\left(x^{t} + p^{t*}\\right)}\\right).}\n\\label{eq:tobdd2}\n\\end{equation}\n","start":41753,"end":42105}]},{"id":"S066","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"模型最优值的符号","source_units":[{"raw_tex":"Note that $Q_{H_t}(0;x^t) \\equiv 0$, so $-Q^*_t \\geq 0$ from its\noptimality and the right-hand side of  \\cref{eq:tobdd2} is\nwell-defined.\n","start":42105,"end":42243}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sconv"],"declared_external":false,"source_only":false,"title":"输入点误差与扰动消失","source_units":[{"raw_tex":"\\modify{Next, we see from \\cref{eq:Qconv}, \\cref{eq:bdd0}, and the\ncontinuity of $R$ that\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty} \\norm{\\left(y^{t} - x^t\\right) - p^{t*}} =\n0, \\quad \\Rightarrow \\quad\n\\lim_{t \\rightarrow \\infty} R\\left( \\norm{\\left(y^{t} - x^t\\right) -\np^{t*}}\\right) =\n0.\n\\label{eq:sconv}\n\\end{equation}\n","start":42243,"end":42569}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":["eq:partialconv"],"declared_external":false,"source_only":false,"title":"试探点驻点残差消失","source_units":[{"raw_tex":"Applying \\cref{eq:Qconv} and \\cref{eq:sconv}} to \\cref{eq:tobdd2} and\nletting $t$ approach infinity then yield\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty} \\dist\\left( 0, \\partial F\\left(\nx^t + p^t \\right) \\right) = 0.\n\\label{eq:partialconv}\n\\end{equation}\n","start":42569,"end":42826}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":["eq:convxp"],"declared_external":false,"source_only":false,"title":"试探点收敛到非退化点","source_units":[{"raw_tex":"\nNext, from \\cref{eq:bdd1} and \\cref{eq:Qconv}, it is also clear that\n$\\norm{p^t} \\rightarrow 0$,\nso from the convergence of $x^t$ to $x^*$ we have\n\\begin{equation}\nx^t + p^t \\rightarrow x^* + 0 = x^*.\n\\label{eq:convxp}\n\\end{equation}\n","start":42826,"end":43061}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"应用凸识别引理并得矛盾","source_units":[{"raw_tex":"Now \\cref{eq:convxp} and \\cref{eq:partialconv} allow us to apply\n\\cref{lemma:identify}\nso $x^t + p^t \\in \\M$ for all $t$ large enough,\nleading to the desired contradiction.\n\\qed\n\\end{proof}","start":43061,"end":43250}]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["任务明确声明的先决结果 LewZ13a, Theorem 4.10；原文第722行和第739–742行调用。"],"source_only":false,"title":"外部部分光滑识别定理","source_units":[]},{"id":"S072","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["任务明确声明的先决结果 RocW09a, Example 13.30；原文第766–769行调用。"],"source_only":false,"title":"凸函数的次微分连续性","source_units":[]},{"id":"S073","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["任务明确声明的先决结果 Wri12a, Lemma 3.2；原文第848–854行援引其论证。"],"source_only":false,"title":"精确模型步的外部局部估计","source_units":[]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["任务声明允许的 standard convex subdifferential calculus and proximal optimality；一般识别引理中的平滑项平移为 annotator 显式标出的 implicit standard fact。"],"source_only":false,"title":"次微分与近端最优性规则","source_units":[]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["任务声明允许的 strong-convexity quadratic growth；原文第836–842行以“which implies”使用，annotator 将标准事实显式化。"],"source_only":false,"title":"强凸函数的二次增长","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S057","edges":[{"source_id":"S002","target_id":"S009","kind":"supports"},{"source_id":"S001","target_id":"S007","kind":"supports"},{"source_id":"S074","target_id":"S007","kind":"supports"},{"source_id":"S008","target_id":"S016","kind":"supports"},{"source_id":"S002","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S016","kind":"supports"},{"source_id":"S074","target_id":"S016","kind":"supports"},{"source_id":"S012","target_id":"S018","kind":"supports"},{"source_id":"S002","target_id":"S018","kind":"supports"},{"source_id":"S001","target_id":"S018","kind":"supports"},{"source_id":"S021","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S002","target_id":"S017","kind":"supports"},{"source_id":"S013","target_id":"S017","kind":"supports"},{"source_id":"S014","target_id":"S017","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S010","target_id":"S017","kind":"supports"},{"source_id":"S023","target_id":"S028","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S005","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S032","kind":"supports"},{"source_id":"S074","target_id":"S033","kind":"supports"},{"source_id":"S028","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S029","target_id":"S034","kind":"supports"},{"source_id":"S028","target_id":"S035","kind":"supports"},{"source_id":"S032","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S029","target_id":"S035","kind":"supports"},{"source_id":"S071","target_id":"S035","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S034","target_id":"S031","kind":"supports"},{"source_id":"S035","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S036","target_id":"S039","kind":"supports"},{"source_id":"S029","target_id":"S039","kind":"supports"},{"source_id":"S072","target_id":"S039","kind":"supports"},{"source_id":"S074","target_id":"S039","kind":"supports"},{"source_id":"S036","target_id":"S040","kind":"supports"},{"source_id":"S022","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S030","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S029","target_id":"S041","kind":"supports"},{"source_id":"S023","target_id":"S041","kind":"supports"},{"source_id":"S006","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S044","kind":"supports"},{"source_id":"S014","target_id":"S044","kind":"supports"},{"source_id":"S012","target_id":"S044","kind":"supports"},{"source_id":"S017","target_id":"S044","kind":"supports"},{"source_id":"S004","target_id":"S045","kind":"supports"},{"source_id":"S002","target_id":"S045","kind":"supports"},{"source_id":"S074","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S007","target_id":"S046","kind":"supports"},{"source_id":"S012","target_id":"S046","kind":"supports"},{"source_id":"S041","target_id":"S046","kind":"supports"},{"source_id":"S012","target_id":"S047","kind":"supports"},{"source_id":"S002","target_id":"S047","kind":"supports"},{"source_id":"S041","target_id":"S047","kind":"supports"},{"source_id":"S075","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S019","target_id":"S048","kind":"supports"},{"source_id":"S041","target_id":"S049","kind":"supports"},{"source_id":"S012","target_id":"S049","kind":"supports"},{"source_id":"S044","target_id":"S049","kind":"supports"},{"source_id":"S073","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S014","target_id":"S050","kind":"supports"},{"source_id":"S044","target_id":"S050","kind":"supports"},{"source_id":"S046","target_id":"S051","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S014","target_id":"S051","kind":"supports"},{"source_id":"S044","target_id":"S051","kind":"supports"},{"source_id":"S037","target_id":"S051","kind":"supports"},{"source_id":"S041","target_id":"S051","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"},{"source_id":"S046","target_id":"S043","kind":"supports"},{"source_id":"S047","target_id":"S043","kind":"supports"},{"source_id":"S048","target_id":"S043","kind":"supports"},{"source_id":"S049","target_id":"S043","kind":"supports"},{"source_id":"S050","target_id":"S043","kind":"supports"},{"source_id":"S051","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S052","kind":"supports"},{"source_id":"S017","target_id":"S052","kind":"supports"},{"source_id":"S013","target_id":"S052","kind":"supports"},{"source_id":"S026","target_id":"S052","kind":"supports"},{"source_id":"S012","target_id":"S052","kind":"supports"},{"source_id":"S008","target_id":"S053","kind":"supports"},{"source_id":"S009","target_id":"S053","kind":"supports"},{"source_id":"S006","target_id":"S055","kind":"supports"},{"source_id":"S052","target_id":"S059","kind":"supports"},{"source_id":"S053","target_id":"S059","kind":"supports"},{"source_id":"S054","target_id":"S059","kind":"supports"},{"source_id":"S055","target_id":"S059","kind":"supports"},{"source_id":"S056","target_id":"S059","kind":"supports"},{"source_id":"S053","target_id":"S060","kind":"supports"},{"source_id":"S008","target_id":"S060","kind":"supports"},{"source_id":"S074","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S007","target_id":"S061","kind":"supports"},{"source_id":"S012","target_id":"S061","kind":"supports"},{"source_id":"S055","target_id":"S061","kind":"supports"},{"source_id":"S041","target_id":"S061","kind":"supports"},{"source_id":"S002","target_id":"S066","kind":"supports"},{"source_id":"S009","target_id":"S066","kind":"supports"},{"source_id":"S047","target_id":"S062","kind":"supports"},{"source_id":"S013","target_id":"S062","kind":"supports"},{"source_id":"S026","target_id":"S062","kind":"supports"},{"source_id":"S066","target_id":"S062","kind":"supports"},{"source_id":"S056","target_id":"S063","kind":"supports"},{"source_id":"S047","target_id":"S063","kind":"supports"},{"source_id":"S066","target_id":"S063","kind":"supports"},{"source_id":"S054","target_id":"S064","kind":"supports"},{"source_id":"S061","target_id":"S065","kind":"supports"},{"source_id":"S062","target_id":"S065","kind":"supports"},{"source_id":"S063","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S059","target_id":"S067","kind":"supports"},{"source_id":"S056","target_id":"S067","kind":"supports"},{"source_id":"S054","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S065","target_id":"S068","kind":"supports"},{"source_id":"S059","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S056","target_id":"S068","kind":"supports"},{"source_id":"S062","target_id":"S069","kind":"supports"},{"source_id":"S059","target_id":"S069","kind":"supports"},{"source_id":"S069","target_id":"S070","kind":"supports"},{"source_id":"S068","target_id":"S070","kind":"supports"},{"source_id":"S037","target_id":"S070","kind":"supports"},{"source_id":"S041","target_id":"S070","kind":"supports"},{"source_id":"S059","target_id":"S070","kind":"supports"},{"source_id":"S059","target_id":"S058","kind":"supports"},{"source_id":"S060","target_id":"S058","kind":"supports"},{"source_id":"S061","target_id":"S058","kind":"supports"},{"source_id":"S062","target_id":"S058","kind":"supports"},{"source_id":"S063","target_id":"S058","kind":"supports"},{"source_id":"S064","target_id":"S058","kind":"supports"},{"source_id":"S065","target_id":"S058","kind":"supports"},{"source_id":"S066","target_id":"S058","kind":"supports"},{"source_id":"S067","target_id":"S058","kind":"supports"},{"source_id":"S068","target_id":"S058","kind":"supports"},{"source_id":"S069","target_id":"S058","kind":"supports"},{"source_id":"S070","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S003","target_id":"S057","kind":"supports"},{"source_id":"S023","target_id":"S057","kind":"supports"},{"source_id":"S017","target_id":"S057","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S057","edges":[{"source_id":"S002","target_id":"S009","kind":"supports"},{"source_id":"S001","target_id":"S007","kind":"supports"},{"source_id":"S074","target_id":"S007","kind":"supports"},{"source_id":"S008","target_id":"S016","kind":"supports"},{"source_id":"S002","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S016","kind":"supports"},{"source_id":"S074","target_id":"S016","kind":"supports"},{"source_id":"S012","target_id":"S018","kind":"supports"},{"source_id":"S002","target_id":"S018","kind":"supports"},{"source_id":"S001","target_id":"S018","kind":"supports"},{"source_id":"S021","target_id":"S023","kind":"supports"},{"source_id":"S002","target_id":"S017","kind":"supports"},{"source_id":"S013","target_id":"S017","kind":"supports"},{"source_id":"S014","target_id":"S017","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S010","target_id":"S017","kind":"supports"},{"source_id":"S023","target_id":"S028","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S005","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S030","kind":"supports"},{"source_id":"S074","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S071","target_id":"S030","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S029","target_id":"S037","kind":"supports"},{"source_id":"S072","target_id":"S037","kind":"supports"},{"source_id":"S074","target_id":"S037","kind":"supports"},{"source_id":"S022","target_id":"S037","kind":"supports"},{"source_id":"S030","target_id":"S037","kind":"supports"},{"source_id":"S029","target_id":"S041","kind":"supports"},{"source_id":"S023","target_id":"S041","kind":"supports"},{"source_id":"S006","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S014","target_id":"S042","kind":"supports"},{"source_id":"S012","target_id":"S042","kind":"supports"},{"source_id":"S017","target_id":"S042","kind":"supports"},{"source_id":"S004","target_id":"S042","kind":"supports"},{"source_id":"S002","target_id":"S042","kind":"supports"},{"source_id":"S074","target_id":"S042","kind":"supports"},{"source_id":"S007","target_id":"S042","kind":"supports"},{"source_id":"S012","target_id":"S047","kind":"supports"},{"source_id":"S002","target_id":"S047","kind":"supports"},{"source_id":"S041","target_id":"S047","kind":"supports"},{"source_id":"S075","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S042","kind":"supports"},{"source_id":"S019","target_id":"S042","kind":"supports"},{"source_id":"S073","target_id":"S042","kind":"supports"},{"source_id":"S037","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S052","kind":"supports"},{"source_id":"S017","target_id":"S052","kind":"supports"},{"source_id":"S013","target_id":"S052","kind":"supports"},{"source_id":"S026","target_id":"S052","kind":"supports"},{"source_id":"S012","target_id":"S052","kind":"supports"},{"source_id":"S008","target_id":"S053","kind":"supports"},{"source_id":"S009","target_id":"S053","kind":"supports"},{"source_id":"S006","target_id":"S055","kind":"supports"},{"source_id":"S052","target_id":"S059","kind":"supports"},{"source_id":"S053","target_id":"S059","kind":"supports"},{"source_id":"S054","target_id":"S059","kind":"supports"},{"source_id":"S055","target_id":"S059","kind":"supports"},{"source_id":"S056","target_id":"S059","kind":"supports"},{"source_id":"S053","target_id":"S060","kind":"supports"},{"source_id":"S008","target_id":"S060","kind":"supports"},{"source_id":"S074","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S007","target_id":"S061","kind":"supports"},{"source_id":"S012","target_id":"S061","kind":"supports"},{"source_id":"S055","target_id":"S061","kind":"supports"},{"source_id":"S041","target_id":"S061","kind":"supports"},{"source_id":"S002","target_id":"S066","kind":"supports"},{"source_id":"S009","target_id":"S066","kind":"supports"},{"source_id":"S047","target_id":"S062","kind":"supports"},{"source_id":"S013","target_id":"S062","kind":"supports"},{"source_id":"S026","target_id":"S062","kind":"supports"},{"source_id":"S066","target_id":"S062","kind":"supports"},{"source_id":"S056","target_id":"S063","kind":"supports"},{"source_id":"S047","target_id":"S063","kind":"supports"},{"source_id":"S066","target_id":"S063","kind":"supports"},{"source_id":"S054","target_id":"S064","kind":"supports"},{"source_id":"S061","target_id":"S065","kind":"supports"},{"source_id":"S062","target_id":"S065","kind":"supports"},{"source_id":"S063","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S059","target_id":"S067","kind":"supports"},{"source_id":"S056","target_id":"S067","kind":"supports"},{"source_id":"S054","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S065","target_id":"S068","kind":"supports"},{"source_id":"S059","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S056","target_id":"S068","kind":"supports"},{"source_id":"S062","target_id":"S069","kind":"supports"},{"source_id":"S059","target_id":"S069","kind":"supports"},{"source_id":"S069","target_id":"S070","kind":"supports"},{"source_id":"S068","target_id":"S070","kind":"supports"},{"source_id":"S037","target_id":"S070","kind":"supports"},{"source_id":"S041","target_id":"S070","kind":"supports"},{"source_id":"S059","target_id":"S070","kind":"supports"},{"source_id":"S059","target_id":"S058","kind":"supports"},{"source_id":"S060","target_id":"S058","kind":"supports"},{"source_id":"S061","target_id":"S058","kind":"supports"},{"source_id":"S062","target_id":"S058","kind":"supports"},{"source_id":"S063","target_id":"S058","kind":"supports"},{"source_id":"S064","target_id":"S058","kind":"supports"},{"source_id":"S065","target_id":"S058","kind":"supports"},{"source_id":"S066","target_id":"S058","kind":"supports"},{"source_id":"S067","target_id":"S058","kind":"supports"},{"source_id":"S068","target_id":"S058","kind":"supports"},{"source_id":"S069","target_id":"S058","kind":"supports"},{"source_id":"S070","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S003","target_id":"S057","kind":"supports"},{"source_id":"S023","target_id":"S057","kind":"supports"},{"source_id":"S017","target_id":"S057","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:f"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"正则化优化问题","source_units":[{"raw_tex":"Consider the following regularized optimization problem:\n\\begin{equation}\n\t\\min_{x \\in \\H}\\quad F(x) \\coloneqq f(x) + \\Psi(x),\n\t\\label{eq:f}\n\\end{equation}\nwhere $\\H$ is a Euclidean space with an inner product\n$\\inprod{\\cdot}{\\cdot}$ and its induced norm $\\norm{\\cdot}$,\nthe regularizer $\\Psi$ is extended-valued, convex, proper, and\nlower-semicontinuous, $f$ is \\modify{continuously differentiable with\nLipschitz-continuous gradients},\nand the solution set $\\Omega$ is non-empty.\nThis type of problems is ubiquitous in applications such as machine\nlearning and signal processing \\modify{(see, for example,\n\\cite{Tib96a,CanR09a,CanRT06a})}.\n","start":6048,"end":6689}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":["eq:Qdef","eq:quadratic"],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"二次模型与更新方向","source_units":[{"raw_tex":"One widely-used method for \\cref{eq:f} is inexact successive quadratic\napproximation (\\ipvm).\nAt the $t$th iteration with iterate $x^t$,\n\\ipvm obtains the update direction $p^t$ by approximately solving\n\\begin{gather}\n\\label{eq:quadratic}\np^t \\approx \\argmin_{p \\in \\H}\\quad Q_{H_t}\\left( p; x^t\n\t\\right),\\\\\n\\label{eq:Qdef}\nQ_{H_t}\\left( p; x^t \\right) \\coloneqq \\inprod{\\nabla f\\left( x^t\n\t\\right)}{p} + \\frac12 \\inprod{p}{ H_t p} + \\Psi\\left( x^t + p\n\t\\right) - \\Psi\\left( x^t \\right),\n\\end{gather}\nwhere $H_t$ is a self-adjoint positive-semidefinite linear\nendomorphism of $\\H$.\nThe iterate is then updated along $p^t$ with a step size $\\alpha_t\n> 0$.\n","start":6689,"end":7344}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"流形识别的含义","source_units":[{"raw_tex":"A possible remedy when $\\Psi$ is partly smooth \\citep{Lew02a}\nis to switch to smooth optimization after identifying an active\nmanifold $\\M$ that contains a solution \\modify{$\\hat x$} to \\cref{eq:f}\nand makes $\\Psi$ confined to it smooth.\nWe say an algorithm can identify $\\M$ if there is a neighborhood $U$\nof \\modify{$\\hat x$} such that $x^t \\in U$ implies $x^{t+1} \\in \\M$, and call it\npossesses the manifold identification property.\n","start":8590,"end":9026}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["eq:subnorm"],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"最小范数模型残差","source_units":[{"raw_tex":"\\begin{equation}\nr^t \\coloneqq \\argmin_{r} \\|r\\|, \\quad \\text{ subject to } \\quad r \\in\n\\partial_{p} Q_{H_t}(p^t;x^t),\n\\label{eq:subnorm}\n\\end{equation}\n","start":10306,"end":10459}]},{"id":"S005","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"最优值与集合记号","source_units":[{"raw_tex":"We denote the minimum of \\cref{eq:f} by $F^*$; the domain of $\\Psi$\nby $\\dom(\\Psi)$; and the set of convex, proper, and lower\nsemicontinuous functions by $\\Gamma_0$.\n\\modify{For any set $C$, $\\relint(C)$ denotes its relative interior.}\nWe will frequently use the following notations.\n\\[\n\t\\delta_t \\coloneqq F(x^t) - F^*,\\quad\nP_{\\Omega}(x) \\coloneqq \\argmin_{y \\in \\Omega} \\|x - y\\|,\\quad\n\t\\dist(x,\\Omega) \\coloneqq \\|x - P_\\Omega(x)\\|.\n\\]\nThe level set $\\levbet \\coloneqq \\left\\{ x\\mid  F(x) - F^* \\leq \\xi\n\\right\\}$ for any $\\xi \\geq 0$  is closed but not necessarily\n\\modify{bounded.}\n","start":19364,"end":19952}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"光滑性与算子序","source_units":[{"raw_tex":"A function is $L$-smooth if it is \\modify{differentiable with the gradient\n$L$-Lipschitz continuous}.\nWe denote the identity operator by $I$.\nFor self-adjoint linear endomorphisms $A,B$ of $\\H$,\n$A \\succ B$ ($\\succeq$) means $A-B$ is positive definite (positive\nsemidefinite).\nWe abbreviate $A \\succ \\tau I$ to $A \\succ \\tau$ for $\\tau \\in \\R$.\nThe set of $A$ with $A \\succ 0$ is denoted by $\\SS$.\n","start":19952,"end":20350}]},{"id":"S007","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"复合目标的次微分","source_units":[{"raw_tex":"The subdifferential $\\partial \\Psi(x)$ of $\\Psi$ at $x$ is\nwell-defined as $\\Psi \\in \\Gamma_0$,\nhence so is the generalized gradient $\\partial F(x) = \\nabla f(x) +\n\\partial \\Psi(x)$.\n","start":20350,"end":20533}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":["eq:proximal"],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"广义近端映射","source_units":[{"raw_tex":"For any $g \\in \\Gamma_0$, $\\tau\\geq 0$, and \\modify{$\\Lambda \\in \\SS$},\nthe proximal mapping\n\\begin{equation}\n\\label{eq:proximal}\n\t\\prox^{\\Lambda}_{\\tau g}(x) \\coloneqq \\argmin_{y \\in \\H}\\, \\frac12\n\t\\inprod{x - y}{\\Lambda (x-y)} + \\tau g\\left( y \\right)\n\\end{equation}\nis continuous and finite in $\\H$ even outside\n$\\dom(g)$.\nWhen $\\Lambda = I$, \\cref{eq:proximal} is shorten to\n$\\prox_{\\tau g}(x)$.\n","start":20533,"end":20933}]},{"id":"S009","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"模型最优解记号","source_units":[{"raw_tex":"For \\cref{eq:quadratic}, we denote its optimal solution by\n$p^{t*}$. When there is no ambiguity, we abbreviate $Q_{H_t}(\\cdot;\nx^t)$ to $Q_t(\\cdot)$, $Q_t(p^t)$ to $\\hat Q_t$, and $Q_t(p^{t*})$ to\n$Q_t^*$.\n","start":20933,"end":21139}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","labels":["eq:armijo"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"Armijo 回溯规则","source_units":[{"raw_tex":"After obtaining $p^t$ from \\cref{eq:quadratic}, we need to find a\nstep size $\\alpha_t > 0$ for it  to ensure sufficient objective\ndecrease.\nGiven $\\gamma,\\beta \\in (0,1)$, we take $\\alpha_t$ as the\nlargest value in $\\{\\beta^0,\\beta^1,\\dotsc\\}$ satisfying an\nArmijo-like condition.\n\\begin{equation}\n\tF\\left( x^t + \\alpha_t p^t \\right) \\leq F\\left( x^t \\right) +\n\t\\gamma \\alpha_t Q_t(p^t).\n\t\\label{eq:armijo}\n\\end{equation}\n","start":23469,"end":23891}]},{"id":"S011","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S011"],"declared_external":true,"declaration_text":["原文第559–561行引用 LeeW18a, Lemma 3；仅保留为算法背景，不作为选定识别证明的输入前提。"],"source_only":false,"title":"回溯可行性的背景","source_units":[{"raw_tex":"This condition is satisfied by all $\\alpha_t$ small enough as long as\n$Q_t(p^t) < 0$ and $Q_t(\\cdot)$ is strongly convex; see\n\\cite[Lemma~3]{LeeW18a}.\n\n","start":23891,"end":24043}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","labels":["assum:Hbound"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"模型算子的统一界","source_units":[{"raw_tex":"For the choice of $H_t$,\nwe only make the following blanket assumption without further\nspecification to make our analysis more general.\n\\begin{equation}\n\\exists M,m > 0,\\quad \\text{ such that } \\quad M \\succeq H_t  \\succeq\nm, \\;\\forall t \\geq 0.\n\\label{assum:Hbound}\n\\end{equation}\n","start":24043,"end":24325}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"模型目标误差准则","source_units":[{"raw_tex":"For \\cref{eq:quadratic}, any suitable solver for regularized\noptimization, such as (accelerated) proximal gradient,\n(variance-reduced) stochastic gradient methods, and their variants,\ncan be used, and the following are common for their inner loop\ntermination:\n\\begin{gather}\n\t\\hat Q_t - Q_t^* \\leq \\epsilon_t,\n\t\\label{eq:subprob}\\\\\n","start":24326,"end":24658}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":["eq:gm","eq:partial","eq:subprob"],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"模型残差停止准则","source_units":[{"raw_tex":"\\left\\| r^t \\right\\| \\leq \\epsilon_t,\n\\quad \\text{ or }\n\t\\label{eq:partial}\\\\\n","start":24658,"end":24736}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":["eq:p"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"近端梯度残差","source_units":[{"raw_tex":"G^\\tau_{H_t} \\left( p^t;x^t \\right)\\coloneqq \\norm{p^t - \\bar\np^t_\\tau}\\leq \\epsilon_t,\n\t\\label{eq:gm}\n\\end{gather}\nfor some given $\\epsilon_t\\geq 0$ and $\\tau > 0$, where\n\\begin{align}\n\\label{eq:p}\n\\bar p^t_\\tau &\\coloneqq \\prox_{\\tau \\Psi}\\left(\\left(x^t + p^t \\right) -\n\t\\tau \\left(\\nabla f(x^t) + H_t p^t\\right)\\right) - x^t.\n\\end{align}\n","start":24736,"end":25078}]},{"id":"S016","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"近端梯度零点解释","source_units":[{"raw_tex":"\\modify{\nThe point $\\bar p^t_\\tau$ in \\cref{eq:p} is computed by taking a\nproximal gradient step of the subproblem \\cref{eq:quadratic} from\n$p^t$, and thus $\\bar p^t - p^t$ is \\modifyy{the proximal gradient}\n(with step size $\\tau$) of $Q_t$ at $p^t$.\nBecause the subproblem is strongly convex, the norm of the proximal\ngradient is zero at $p^t$ if and only if $p^t$ is the unique solution\nto the subproblem.\nWe will see in the next subsection that the norm of this proximal\ngradient squared is also equivalent to the objective distance to the\noptimum of the subproblem.}\n","start":25078,"end":25649}]},{"id":"S017","semantic_type":"algorithm","kind":"algorithm","labels":["alg:vm"],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"ISQA 算法框架","source_units":[{"raw_tex":"\\begin{algorithm}[h]\n\\DontPrintSemicolon\n\\SetKwInOut{Input}{input}\\SetKwInOut{Output}{output}\n\\caption{A Framework of \\ipvm for \\cref{eq:f}}\n\\label{alg:vm}\n\\Input{$x^0 \\in \\H$, $\\gamma, \\beta \\in (0,1)$}\n\n\\For{$t=0,1,\\dotsc$}{\n\t$\\alpha_t \\leftarrow 1$,\n\tpick $\\epsilon_t \\geq 0$ and $H_t$, and solve \\cref{eq:quadratic}\n\tfor $p^t$ satisfying \\cref{eq:subprob},\n\t\\cref{eq:partial}, or \\cref{eq:gm}\n\n\n\t\\lWhile{\\cref{eq:armijo} is not satisfied}\n\t{$\\alpha_t \\leftarrow \\beta \\alpha_t$}\n\t$x^{t+1} \\leftarrow x^t + \\alpha_t p^t$\n}\n\\end{algorithm}\n","start":25696,"end":26238}]},{"id":"S018","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"模型强凸性","source_units":[{"raw_tex":"\\subsection{Basic Properties}\nUnder \\cref{assum:Hbound}, \\cref{eq:Qdef} is $m$-strongly convex with\nrespect to $p$, so the following standard results hold for any $\\tau\n\\in (0, 1/M]$ and any $p^t \\in \\H$ \\citep{Nes18a,DruL16a}.\n","start":26240,"end":26468}]},{"id":"S019","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:EB","eq:PG"],"member_ids":["S019"],"declared_external":true,"declaration_text":["声明的先决结果：第627–640行引用 Nes18a、DruL16a 的强凸残差及误差界；此处为 eq:PG。"],"source_only":false,"title":"残差控制模型误差","source_units":[{"raw_tex":"\\begin{gather}\n\\|r^t\\|^2 \\geq 2 m \\left( Q_t(p^t) - Q_t^* \\right)\n\\geq \\frac{m}{M} G^\\tau_{H_t}\\left(\n\tp^t;x^t \\right)^2,\n\\label{eq:PG}\\\\\n","start":26468,"end":26606}]},{"id":"S020","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S020"],"declared_external":true,"declaration_text":["声明的先决结果：第627–640行引用 Nes18a、DruL16a 的残差界；此处为 eq:EB。"],"source_only":false,"title":"近端步目标误差界","source_units":[{"raw_tex":"G^\\tau_{H_t}\\left( p^t;x^t \\right)^2 \\geq \\tau \\left( \\frac{(2m^{-1} +\n\t\\tau)(1 + M \\tau)}{\\tau} - \\frac{1}{2} \\right)^{-1}\n\t\\left(Q_t\\left( \\bar p^t_\\tau\\right) - Q_t^* \\right).\n\\label{eq:EB}\n\\end{gather}\n","start":26606,"end":26812}]},{"id":"S021","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"局部流形与参数化","source_units":[{"raw_tex":"A central focus of this work is manifold identification, so we first\nformally define manifolds following \\cite{Vai84a}.\nA set $\\M \\in \\R^m$ is a $p$-dimensional $\\mathcal{C}^k$ manifold\naround $x \\in \\R^m$ if there is a $\\mathcal{C}^k$ function $\\Phi: \\R^m\n\\rightarrow \\R^{m-p}$ whose derivative at $x$ is surjective such that for\n$y$ close enough to $x$, $y \\in \\M$ if and only if $\\Phi(y) = 0$.\nThrough the implicit function theorem, we can also use a\n$\\mathcal{C}^k$ parameterization $\\phi: \\R^p \\rightarrow \\M$, with\n$\\phi(y) = x$ and the derivative injective at $y$, to describe a\nneighborhood of $x$ on $\\M$.\n","start":27228,"end":27843}]},{"id":"S022","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S022"],"declared_external":true,"declaration_text":["声明的 RocW09a 凸正则性及 prox-regularity 先决事实；原文第659–663行明确说明并引用 Definition 7.25、Definition 13.27。"],"source_only":false,"title":"凸函数的正则性质","source_units":[{"raw_tex":"In our definition and proof below, we will require the function being\n(subdifferentially) regular and prox-regular at a point $x$.\nThese two conditions are satisfied by convex functions everywhere and\nthus we do not provide their definitions here, and interested readers\ncan refer to \\cite[][Definition~7.25, Definition~13.27]{RocW09a} for details.\nNow we are ready for the definition of partial smoothness\n\\citep{Lew02a} that we assume for the regularizer when discussing\nmanifold identification.\n","start":27843,"end":28341}]},{"id":"S023","semantic_type":"definition","kind":"definition","labels":["def:ps"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"部分光滑性","source_units":[{"raw_tex":"\\begin{definition}[Partly smooth]\n\t\\label{def:ps}\n\tA function $\\Psi$ is partly\n\tsmooth at a point $x^*$ relative to a set $\\M$ containing $x^*$ if\n\t$\\partial \\Psi(x^*) \\neq \\emptyset$ and:\n\t\\begin{enumerate}\n\t\t\t\\item Around $x^*$, $\\M$ is a $\\mathcal{C}^2$-manifold and\n\t\t\t\t$\\Psi|_{\\M}$ is $\\mathcal{C}^2$.\n\t\t\\item The affine span of $\\partial \\Psi(x)$ is a translate\n\t\t\tof the normal space to $\\M$ at $x^*$.\n\t\t\\item $\\partial \\Psi$ is continuous at $x^*$ relative to $\\M$.\n\t\t\\item \\modify{$\\Psi$ is regular at all points $x \\in \\M$ with\n\t\t\t$\\partial \\Psi(x) \\neq  \\emptyset$\n\t\taround $x^*$.}\n\t\\end{enumerate}\n\\end{definition}\n","start":28341,"end":28968}]},{"id":"S026","semantic_type":"definition","kind":"definition","labels":["eq:multiplicative"],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"固定相对模型容差","source_units":[{"raw_tex":"\\begin{equation}\n\t\\epsilon_t = \\eta \\left( Q_t(0) - Q_t^* \\right) = -\\eta Q_t^*.\n\t\\label{eq:multiplicative}\n\\end{equation}\n","start":36749,"end":36872}]},{"id":"S028","semantic_type":"assumption","kind":"assumption","labels":["eq:nod","lemma:identify0"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"一般识别引理的前提","source_units":[{"raw_tex":"\\begin{lemma}\nConsider \\cref{eq:f} with $f\\in \\mathcal{C}^1$ and $\\Psi$\npartly\nsmooth at a point $x^*$ relative to a $\\mathcal{C}^2$-manifold\n$\\M$.\nIf $\\Psi$ is prox-regular at $x^*$,\nthe nondegenerate condition\n\\begin{equation}\n0 \\in \\relint \\left(\\partial F(x^*)\\right) = \\nabla f(x^*) + \\relint\n\\left(\\partial \\Psi(x^*)\\right)\n\\label{eq:nod}\n\\end{equation}\nholds, and there is a sequence $\\{x^t\\}$ converging to $x^*$ with\n$F(x^t) \\rightarrow F(x^*)$,\n","start":29434,"end":29889}]},{"id":"S029","semantic_type":"assumption","kind":"assumption","labels":["eq:nod","lemma:identify0"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"非退化驻点条件","source_units":[{"raw_tex":"\\begin{equation}\n0 \\in \\relint \\left(\\partial F(x^*)\\right) = \\nabla f(x^*) + \\relint\n\\left(\\partial \\Psi(x^*)\\right)\n\\label{eq:nod}\n\\end{equation}\n","start":29646,"end":29794}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","labels":["eq:nod","lemma:identify0"],"member_ids":["S030","S031","S032","S033","S034","S035"],"declared_external":false,"source_only":false,"title":"一般部分光滑识别引理","source_units":[{"raw_tex":"\\begin{lemma}\nConsider \\cref{eq:f} with $f\\in \\mathcal{C}^1$ and $\\Psi$\npartly\nsmooth at a point $x^*$ relative to a $\\mathcal{C}^2$-manifold\n$\\M$.\nIf $\\Psi$ is prox-regular at $x^*$,\nthe nondegenerate condition\n\\begin{equation}\n0 \\in \\relint \\left(\\partial F(x^*)\\right) = \\nabla f(x^*) + \\relint\n\\left(\\partial \\Psi(x^*)\\right)\n\\label{eq:nod}\n\\end{equation}\nholds, and there is a sequence $\\{x^t\\}$ converging to $x^*$ with\n$F(x^t) \\rightarrow F(x^*)$,\nthen\n\\[\n\t\\dist\\left( 0, \\partial F(x^t) \\right) \\rightarrow 0\n\t\\quad \\Leftrightarrow \\quad x^t \\in \\M \\text{ for all t large}.\n\\]\n\\label{lemma:identify0}\n\\end{lemma}","start":29434,"end":30054},{"raw_tex":"\\begin{proof}\nWe first observe that as $f \\in \\mathcal{C}^1$, $x^t \\rightarrow x^*$\nimplies $f(x^t) \\rightarrow f(x^*)$,\nwhose combination with $F(x^t) \\rightarrow F(x^*)$ further implies\n$\\Psi(x^t) \\rightarrow \\Psi(x^*)$.\nTherefore, the premises of \\cite[Theorem~4.10]{LewZ13a} on $\\Psi$ are\nsatisfied.\n","start":30066,"end":30370},{"raw_tex":"\\begin{proof}\nWe first observe that as $f \\in \\mathcal{C}^1$, $x^t \\rightarrow x^*$\nimplies $f(x^t) \\rightarrow f(x^*)$,\nwhose combination with $F(x^t) \\rightarrow F(x^*)$ further implies\n$\\Psi(x^t) \\rightarrow \\Psi(x^*)$.\nTherefore, the premises of \\cite[Theorem~4.10]{LewZ13a} on $\\Psi$ are\nsatisfied.\nWe then note that\n\\begin{equation}\n\t\\label{eq:convdist}\n\t\\dist\\left(0, \\partial F(x^t)\\right) \\rightarrow 0 \\quad\n\t\\Leftrightarrow \\quad\n\t\\dist\\left(-\\nabla f(x^t), \\partial \\Psi(x^t)\\right) \\rightarrow 0.\n\\end{equation}\nAgain from that $f \\in \\mathcal{C}^1$, $x^k \\rightarrow x^*$ implies\n$\\nabla f(x^k) \\rightarrow \\nabla f(x^*)$,\nso by \\cref{eq:nod}, \\cref{eq:convdist} is\nfurther equivalent to\n\\[\n\t\\dist\\left(-\\nabla f(x^*), \\partial \\Psi(x^t)\\right) \\rightarrow\n\t0,\n\\]\nwhich is the necessary and sufficient condition for $x^t \\in \\M$ for\nall $t$ large in \\cite[Theorem~4.10]{LewZ13a} because \\cref{eq:nod}\nindicates that $- \\nabla f(x^*) \\in \\relint(\\partial \\Psi(x^*))$.\nWe then apply that theorem to obtain the desired result.\nHere we note that for the requirements\n\\qed\n\\end{proof}","start":30066,"end":31159},{"raw_tex":"We then note that\n\\begin{equation}\n\t\\label{eq:convdist}\n\t\\dist\\left(0, \\partial F(x^t)\\right) \\rightarrow 0 \\quad\n\t\\Leftrightarrow \\quad\n\t\\dist\\left(-\\nabla f(x^t), \\partial \\Psi(x^t)\\right) \\rightarrow 0.\n\\end{equation}\n","start":30370,"end":30591},{"raw_tex":"Again from that $f \\in \\mathcal{C}^1$, $x^k \\rightarrow x^*$ implies\n$\\nabla f(x^k) \\rightarrow \\nabla f(x^*)$,\nso by \\cref{eq:nod}, \\cref{eq:convdist} is\nfurther equivalent to\n\\[\n\t\\dist\\left(-\\nabla f(x^*), \\partial \\Psi(x^t)\\right) \\rightarrow\n\t0,\n\\]\n","start":30591,"end":30844},{"raw_tex":"which is the necessary and sufficient condition for $x^t \\in \\M$ for\nall $t$ large in \\cite[Theorem~4.10]{LewZ13a} because \\cref{eq:nod}\nindicates that $- \\nabla f(x^*) \\in \\relint(\\partial \\Psi(x^*))$.\nWe then apply that theorem to obtain the desired result.\nHere we note that for the requirements\n\\qed\n\\end{proof}","start":30844,"end":31159}]},{"id":"S036","semantic_type":"assumption","kind":"assumption","labels":["lemma:identify"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"凸识别引理的前提","source_units":[{"raw_tex":"\\begin{lemma}\n\tConsider \\cref{eq:f} with \\modify{$f \\in \\mathcal{C}^1$} and $\\Psi \\in\n\\Gamma_0$.\nIf $\\Psi$ is partly smooth relative to a manifold $\\M$ at a point\n\\modify{$x^*$ satisfying \\cref{eq:nod}}, and there is a sequence $\\{x^t\\}$\nconverging to $x^*$, then we have\n","start":31407,"end":31679}]},{"id":"S037","semantic_type":"lemma","kind":"lemma","labels":["lemma:identify"],"member_ids":["S037","S038","S039","S040"],"declared_external":false,"source_only":false,"title":"凸正则项的识别引理","source_units":[{"raw_tex":"\\begin{lemma}\n\tConsider \\cref{eq:f} with \\modify{$f \\in \\mathcal{C}^1$} and $\\Psi \\in\n\\Gamma_0$.\nIf $\\Psi$ is partly smooth relative to a manifold $\\M$ at a point\n\\modify{$x^*$ satisfying \\cref{eq:nod}}, and there is a sequence $\\{x^t\\}$\nconverging to $x^*$, then we have\n\\[\n\t\\dist\\left( 0, \\partial F(x^t) \\right) \\rightarrow 0\n\t\\quad \\Rightarrow \\quad x^t \\in \\M \\text{ for all t large}.\n\\]\n\\label{lemma:identify}\n\\end{lemma}","start":31407,"end":31834},{"raw_tex":"\\begin{proof}\nAs $\\Psi \\in \\Gamma_0$, it is subdifferentially continuous at\n$x^* \\in \\dom(\\Psi)$ by \\cite[Example 13.30]{RocW09a}.\nThus, $\\dist(0, \\partial F(x^t)) \\rightarrow 0 \\in \\partial F(x^*)$\nand $x^t \\rightarrow x^*$ imply that $F(x^t) \\rightarrow F(x^*)$.\n","start":31835,"end":32100},{"raw_tex":"\\begin{proof}\nAs $\\Psi \\in \\Gamma_0$, it is subdifferentially continuous at\n$x^* \\in \\dom(\\Psi)$ by \\cite[Example 13.30]{RocW09a}.\nThus, $\\dist(0, \\partial F(x^t)) \\rightarrow 0 \\in \\partial F(x^*)$\nand $x^t \\rightarrow x^*$ imply that $F(x^t) \\rightarrow F(x^*)$.\nThe desired result is then obtained by applying\n\\modify{\\cref{lemma:identify0}}.\n\\qed\n\\end{proof}","start":31835,"end":32197},{"raw_tex":"The desired result is then obtained by applying\n\\modify{\\cref{lemma:identify0}}.\n\\qed\n\\end{proof}","start":32100,"end":32197}]},{"id":"S041","semantic_type":"assumption","kind":"assumption","labels":["thm:identify1"],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"识别定理的几何设置","source_units":[{"raw_tex":"\\begin{theorem}\nConsider a point $x^*$ satisfying \\cref{eq:nod} with $\\Psi \\in \\Gamma_0$\npartly smooth at $x^*$ relative to some manifold $\\M$.\nAssume $f$ is locally $L$-smooth\nfor $L > 0$ around $x^*$.\n","start":32727,"end":32930}]},{"id":"S042","semantic_type":"theorem","kind":"theorem","labels":["thm:identify1"],"member_ids":["S042","S043","S044","S045","S046","S048","S049","S050","S051"],"declared_external":false,"source_only":false,"title":"小模型残差下的识别","source_units":[{"raw_tex":"\\begin{theorem}\nConsider a point $x^*$ satisfying \\cref{eq:nod} with $\\Psi \\in \\Gamma_0$\npartly smooth at $x^*$ relative to some manifold $\\M$.\nAssume $f$ is locally $L$-smooth\nfor $L > 0$ around $x^*$.\nIf \\cref{alg:vm} is run with the condition \\cref{eq:partial} and\n\\cref{assum:Hbound} holds,\nthen\nthere exist $\\epsilon, \\delta > 0$ such that\n$\\|\\modify{x^t} - x^* \\| \\leq \\delta, \\epsilon_t \\leq \\epsilon$, and $\\alpha_t = 1$\nimply $x^{t+1} \\in \\M$.\n\\label{thm:identify1}\n\\end{theorem}","start":32727,"end":33215},{"raw_tex":"\\begin{proof}\nSince each iteration of \\cref{alg:vm} is independent of the previous\nones, we abuse the notation to let $x^t$ be the input of\n\\cref{alg:vm} at the $t$th iteration and $p^t$ the corresponding\ninexact solution to \\cref{eq:quadratic}, but $x^{t+1}$ is\nirrelevant to $p^t$ or $\\alpha_t$.\nAssume for contradiction the statement is false.\nThen there exist a sequence $\\{x^t\\}\\subset \\H$ converging to $x^*$, a\nnonnegative sequence $\\{\\epsilon_t\\}$ converging to $0$, a sequence\n$\\{H_t\\} \\subset \\SS$ satisfying \\cref{assum:Hbound}, and a sequence\n$\\{p^t\\}\\subset \\H$ such that $Q_{H_t}(p^t;x^t)$ in \\cref{eq:Qdef}\nsatisfies\n\\cref{eq:partial} for all $t$, yet $x^t + p^t \\notin \\M$ for all $t$.\n","start":33216,"end":33918},{"raw_tex":"\\begin{proof}\nSince each iteration of \\cref{alg:vm} is independent of the previous\nones, we abuse the notation to let $x^t$ be the input of\n\\cref{alg:vm} at the $t$th iteration and $p^t$ the corresponding\ninexact solution to \\cref{eq:quadratic}, but $x^{t+1}$ is\nirrelevant to $p^t$ or $\\alpha_t$.\nAssume for contradiction the statement is false.\nThen there exist a sequence $\\{x^t\\}\\subset \\H$ converging to $x^*$, a\nnonnegative sequence $\\{\\epsilon_t\\}$ converging to $0$, a sequence\n$\\{H_t\\} \\subset \\SS$ satisfying \\cref{assum:Hbound}, and a sequence\n$\\{p^t\\}\\subset \\H$ such that $Q_{H_t}(p^t;x^t)$ in \\cref{eq:Qdef}\nsatisfies\n\\cref{eq:partial} for all $t$, yet $x^t + p^t \\notin \\M$ for all $t$.\nFrom \\cref{eq:subnorm},\n\\begin{equation}\n\tr^t - \\nabla f\\left( x^t \\right) - H_t p^t\n\t\\in \\partial \\Psi\\left( x^{t} + p^t \\right), \\quad \\forall t \\ge 0.\n\\label{eq:r}\n\\end{equation}\nTherefore, we get from \\cref{eq:r} that\n\\begin{align}\n\t\\nonumber\n\t\\dist\\left(0, \\partial F\\left(x^t + p^t \\right) \\right) &=\n\t\\dist\\left(-\\nabla f\\left( x^t + p^t \\right), \\partial \\Psi\\left(\n\tx^t + p^t \\right)\\right)\\\\\n\t\\nonumber\n\t&\\leq \\norm{-\\nabla f\\left( x^t + p^t\\right) - \\left(r^{t} - \\nabla\n\tf\\left( x^{t} \\right) - H_{t} p^t \\right)}\\\\\n\t\\nonumber\n\t&\\leq \\norm{\\nabla f\\left( x^t + p^t\\right) - \\nabla f\\left( x^{t}\n\t\\right)} + \\norm{r^{t}} +\n\t\\norm{H_{t}} \\norm{p^t}\\\\\n\t&\\leq \\left(L + M \\right) \\norm{p^t} + \\norm{r^{t}}.\n\t\\label{eq:dist}\n\\end{align}\nFrom \\cref{assum:Hbound} and the convexity of $\\Psi$,\n$Q_{H_t}(\\cdot;x^t)$ is $m$-strongly convex, which implies\n\\begin{equation}\n\tQ_{H_t}\\left(p;x^t \\right) - Q_{H_t}\\left( p^{t*};x^t \\right) \\geq\n\\frac{m}{2} \\norm{p - p^{t*}}^2, \\quad  \\forall p \\in \\H.\n\\label{eq:qg}\n\\end{equation}\nCombining \\cref{eq:qg} and \\cref{eq:PG} shows that\n\\begin{equation}\n\t\\label{eq:pbound}\n\tm^{-1} \\|r^t\\| + \\|p^{t*}\\| \\geq \\|p^t\\|.\n\\end{equation}\nSince $\\{x^t\\}$ converges to $x^*$, by the argument in\n\\cite[Lemma~3.2]{Wri12a}, we get\n\\begin{equation}\np^{t*} = O(\\|x^t - x^*\\|)\n\\label{eq:pt}\n\\end{equation}\nwhenever $x^t$ is close enough to $x^*$.\nThus \\cref{eq:pbound}, \\cref{eq:pt}, \\cref{eq:partial}, and that\n$\\epsilon_t \\downarrow 0$ imply\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty} \\|p^t\\| = 0,\n\\label{eq:pt2}\n\\end{equation}\nSubstituting \\cref{eq:partial} and \\cref{eq:pt2} into \\cref{eq:dist}\ngives $\\dist(0, \\partial F(x^t)) \\rightarrow 0$, and by \\cref{eq:pt2}\nwe also get $ x^t + p^t \\rightarrow x^* + 0 = x^*$.  Therefore\n\\cref{lemma:identify} implies that\n$x^{t} + p^{t} \\in \\M$ for all $t$ large enough, proving the\ndesired contradiction.\n\\qed\n\\end{proof}","start":33216,"end":35815},{"raw_tex":"From \\cref{eq:subnorm},\n\\begin{equation}\n\tr^t - \\nabla f\\left( x^t \\right) - H_t p^t\n\t\\in \\partial \\Psi\\left( x^{t} + p^t \\right), \\quad \\forall t \\ge 0.\n\\label{eq:r}\n\\end{equation}\n","start":33918,"end":34100},{"raw_tex":"Therefore, we get from \\cref{eq:r} that\n\\begin{align}\n\t\\nonumber\n\t\\dist\\left(0, \\partial F\\left(x^t + p^t \\right) \\right) &=\n\t\\dist\\left(-\\nabla f\\left( x^t + p^t \\right), \\partial \\Psi\\left(\n\tx^t + p^t \\right)\\right)\\\\\n\t\\nonumber\n\t&\\leq \\norm{-\\nabla f\\left( x^t + p^t\\right) - \\left(r^{t} - \\nabla\n\tf\\left( x^{t} \\right) - H_{t} p^t \\right)}\\\\\n\t\\nonumber\n\t&\\leq \\norm{\\nabla f\\left( x^t + p^t\\right) - \\nabla f\\left( x^{t}\n\t\\right)} + \\norm{r^{t}} +\n\t\\norm{H_{t}} \\norm{p^t}\\\\\n\t&\\leq \\left(L + M \\right) \\norm{p^t} + \\norm{r^{t}}.\n\t\\label{eq:dist}\n\\end{align}\n","start":34100,"end":34662},{"raw_tex":"Combining \\cref{eq:qg} and \\cref{eq:PG} shows that\n\\begin{equation}\n\t\\label{eq:pbound}\n\tm^{-1} \\|r^t\\| + \\|p^{t*}\\| \\geq \\|p^t\\|.\n\\end{equation}\n","start":34947,"end":35092},{"raw_tex":"Since $\\{x^t\\}$ converges to $x^*$, by the argument in\n\\cite[Lemma~3.2]{Wri12a}, we get\n\\begin{equation}\np^{t*} = O(\\|x^t - x^*\\|)\n\\label{eq:pt}\n\\end{equation}\nwhenever $x^t$ is close enough to $x^*$.\n","start":35092,"end":35293},{"raw_tex":"Thus \\cref{eq:pbound}, \\cref{eq:pt}, \\cref{eq:partial}, and that\n$\\epsilon_t \\downarrow 0$ imply\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty} \\|p^t\\| = 0,\n\\label{eq:pt2}\n\\end{equation}\n","start":35293,"end":35478},{"raw_tex":"Substituting \\cref{eq:partial} and \\cref{eq:pt2} into \\cref{eq:dist}\ngives $\\dist(0, \\partial F(x^t)) \\rightarrow 0$, and by \\cref{eq:pt2}\nwe also get $ x^t + p^t \\rightarrow x^* + 0 = x^*$.  Therefore\n\\cref{lemma:identify} implies that\n$x^{t} + p^{t} \\in \\M$ for all $t$ large enough, proving the\ndesired contradiction.\n\\qed\n\\end{proof}","start":35478,"end":35815}]},{"id":"S047","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:qg"],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"模型的二次增长估计","source_units":[{"raw_tex":"From \\cref{assum:Hbound} and the convexity of $\\Psi$,\n$Q_{H_t}(\\cdot;x^t)$ is $m$-strongly convex, which implies\n\\begin{equation}\n\tQ_{H_t}\\left(p;x^t \\right) - Q_{H_t}\\left( p^{t*};x^t \\right) \\geq\n\\frac{m}{2} \\norm{p - p^{t*}}^2, \\quad  \\forall p \\in \\H.\n\\label{eq:qg}\n\\end{equation}\n","start":34662,"end":34947}]},{"id":"S052","semantic_type":"assumption","kind":"assumption","labels":["thm:identify2"],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"主定理的算法前提","source_units":[{"raw_tex":"Consider the setting of \\cref{thm:identify1}.\nIf \\cref{alg:vm} is run with \\cref{eq:subprob} and\n\\cref{eq:multiplicative} for some $\\eta \\in [0,1)$,\n\t\\cref{assum:Hbound} holds,\nand the update direction $p^{t}$ satisfies\n","start":38020,"end":38240}]},{"id":"S053","semantic_type":"assumption","kind":"assumption","labels":["eq:prox","thm:identify2"],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"方向的结构化近端形式","source_units":[{"raw_tex":"\\begin{equation}\n\t\\label{eq:prox}\n\tx^t + p^{t} = \\prox^{\\Lambda_{t}}_{\\Psi}\\left(y^{t} -\n\t\\Lambda_{t}^{-1} \\left( \\nabla f\\left( x^t \\right) + H_t \\left( y^{t} -\n\tx^t \\right) + s^{t} \\right) \\right),\n\\end{equation}\n","start":38240,"end":38455}]},{"id":"S054","semantic_type":"assumption","kind":"assumption","labels":["thm:identify2"],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"连续扰动控制函数","source_units":[{"raw_tex":"where $s^{t}$ satisfies $\\|s^{t}\\| \\leq R\\left( \\norm{y^{t}\n- (x^t + p^{t*})} \\right)$ for some \\modify{$R:[0,\\infty) \\rightarrow\n[0,\\infty)$ continuous in its domain} with $R(0) = 0$,\n","start":38455,"end":38640}]},{"id":"S055","semantic_type":"assumption","kind":"assumption","labels":["thm:identify2"],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"近端度量的上界","source_units":[{"raw_tex":"$\\Lambda_{t} \\in \\SS$ with $M_1 \\succeq \\Lambda_{t}$ for $M_1 >\n0$, and $y^{t}$ satisfies\n","start":38640,"end":38730}]},{"id":"S056","semantic_type":"assumption","kind":"assumption","labels":["eq:bdd0","thm:identify2"],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"近端输入点的精度","source_units":[{"raw_tex":"\\begin{align}\n\\label{eq:bdd0}\n\\norm{\\left(y^{t} - x^t\\right) - p^{t*}} &\\leq \\eta_1 \\left(Q_t(0) -\nQ_t^*\\right)^\\nu\n\\end{align}\nfor some $\\nu > 0$ and $\\eta_1 \\geq 0$,\n","start":38730,"end":38898}]},{"id":"S057","semantic_type":"theorem","kind":"theorem","labels":["eq:bdd0","eq:prox","thm:identify2"],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"结构化非精确方向的流形识别","source_units":[{"raw_tex":"\\begin{theorem}\nConsider the setting of \\cref{thm:identify1}.\nIf \\cref{alg:vm} is run with \\cref{eq:subprob} and\n\\cref{eq:multiplicative} for some $\\eta \\in [0,1)$,\n\t\\cref{assum:Hbound} holds,\nand the update direction $p^{t}$ satisfies\n\\begin{equation}\n\t\\label{eq:prox}\n\tx^t + p^{t} = \\prox^{\\Lambda_{t}}_{\\Psi}\\left(y^{t} -\n\t\\Lambda_{t}^{-1} \\left( \\nabla f\\left( x^t \\right) + H_t \\left( y^{t} -\n\tx^t \\right) + s^{t} \\right) \\right),\n\\end{equation}\nwhere $s^{t}$ satisfies $\\|s^{t}\\| \\leq R\\left( \\norm{y^{t}\n- (x^t + p^{t*})} \\right)$ for some \\modify{$R:[0,\\infty) \\rightarrow\n[0,\\infty)$ continuous in its domain} with $R(0) = 0$,\n$\\Lambda_{t} \\in \\SS$ with $M_1 \\succeq \\Lambda_{t}$ for $M_1 >\n0$, and $y^{t}$ satisfies\n\\begin{align}\n\\label{eq:bdd0}\n\\norm{\\left(y^{t} - x^t\\right) - p^{t*}} &\\leq \\eta_1 \\left(Q_t(0) -\nQ_t^*\\right)^\\nu\n\\end{align}\nfor some $\\nu > 0$ and $\\eta_1 \\geq 0$,\nthen there exists $\\epsilon, \\delta > 0$ such that $\\|x^t - x^*\\| \\leq\n\\delta, |Q_t^*| \\leq \\epsilon$, and $\\alpha_t = 1$ imply $x^{t+1} \\in \\M$.\n\\label{thm:identify2}\n\\end{theorem}","start":38004,"end":39079}]},{"id":"S058","semantic_type":"proof","kind":"proof","labels":["eq:Qconv","eq:bdd1","eq:bdd3","eq:bdd4","eq:convxp","eq:opt","eq:partialconv","eq:sconv","eq:tobdd","eq:tobdd2"],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nSuppose the statement is not true for contradiction.\nThen there exist a continuous \\modify{function $R: [0, \\infty)\n\\rightarrow [0, \\infty)$ with}\n$R(0) = 0$, $\\eta_1 \\geq 0$, $M_1 > 0$, a sequence $\\{x^t\\} \\subset\n\\H$ converging to $x^*$,\na sequence $\\{H_t\\} \\subset \\SS$ satisfying \\cref{assum:Hbound} and\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty}\\, \\min_{p}\\quad Q_{H_t}\\left( p;x^t\n\\right) = 0,\n\\label{eq:Qconv}\n\\end{equation}\nthree sequences $\\{p^t\\}, \\{y^t\\}, \\{s^t\\} \\subset \\H$\nand a sequence $\\{\\Lambda_t\\}\\subset \\SS$ with $M_1 \\succeq \\Lambda_t$\nsuch that \\cref{eq:subprob} with \\cref{eq:multiplicative} and\n\\cref{eq:prox}-\\cref{eq:bdd0} hold, yet $x^t + p^t \\notin \\M$\nfor all $t$.\nWe abuse the notation to let $p^{t*}$ and $Q^*_t$\nrespectively denote the optimal solution and objective value for\n$\\min_p Q_{H_t}(p;x^t)$, but $x^{t+1}$ is irrelevant to $p^t$ or\n$\\alpha_t$.\n\nThe optimality condition of \\cref{eq:proximal} applied to\n\\cref{eq:prox} indicates that\n\\begin{equation}\n\t-\\Lambda_{t}\\left(x^{t} - y^{t}\\right) - \\left( \\nabla\n\tf\\left( x^t \\right) + H_t \\left( y^{t} - x^t \\right) + s^{t}\n\t\\right) \\in \\partial \\Psi\\left( x^{t} + p^t \\right).\n\t\\label{eq:opt}\n\\end{equation}\nThus, we have\n\\begin{align}\n\\nonumber\n&~\\dist\\left( 0, \\partial F\\left( x^t + p^t \\right) \\right)\\\\\n\\nonumber\n\\leq &~\\left\\|\\nabla f\\left( x^t + p^t\n\t\\right) -\\Lambda_{t}\\left(x^t + p^t -\n\ty^{t}\\right) - \\nabla f\\left( x^{t} \\right) \\right.\n\t\\left. -\n\tH_{t} \\left( y^{t} - x^{t} \\right) -\n\ts^{t} \\right\\|\\\\\n\\nonumber\n\\leq &~\\norm{\\nabla f\\left( x^t + p^t \\right)- \\nabla\n\tf\\left( x^{t} \\right)} + M_1 \\left( \\norm{x^t - y^{t}} +\n\t\\norm{p^t}\\right)\n\t+ M\n\t\\norm{y^{t} - x^{t}} + \\norm{s^{t}}\\\\\n\\leq &~(L + M_1) \\norm{p^t} +\n\t(M + M_1) \\norm{y^{t} - x^{t}} +\n\t\\norm{s^{t}}.\n\\label{eq:tobdd}\n\\end{align}\nFor the first two terms in \\cref{eq:tobdd}, the\ntriangle inequality and \\cref{eq:qg} imply\n\\begin{gather}\n\t\\begin{aligned}\n\\norm{p^{t}} \n\\leq \\norm{p^{t} - p^{t*}} + \\norm{p^{t*}}\n\\leq\n&~\\sqrt{2m^{-1}}\\sqrt{ Q_{t}\\left( p^{t} \\right) -\nQ^*_{t} } + \\sqrt{2m^{-1}}\\sqrt{ 0 - Q^*_{t}}\\\\\n\\stackrel{\\cref{eq:multiplicative}}{\\leq}&~ \\left( \\sqrt{\\eta} + 1\n\\right)\\sqrt{2m^{-1}}\\sqrt{ - Q^*_{t}}, \\text{ and }\n\\end{aligned}\n\\label{eq:bdd1}\\\\\n\\norm{y^{t}- x^{t}}\n\\leq \\norm{\\left(y^{t}- x^{t}\\right) - p^{t*}} +\n\\norm{p^{t*}}\n\\stackrel{\\cref{eq:bdd0}}{\\leq}\n\\eta_1 \\left( - Q^*_{t}\\right)^\\nu + \\sqrt{2m^{-1}}\\sqrt{ - Q^*_{t}}.\n\\label{eq:bdd3}\n\\end{gather}\nFor the last term in \\cref{eq:tobdd}, we have from \\modify{our definition of\n$s^t$ that\n\\begin{align}\n\\norm{s^{t}} &\\leq\nR \\left(\\norm{y^{t} - \\left(x^{t} + p^{t*}\\right)}\\right).\n\\label{eq:bdd4}\n\\end{align}\n}\nBy substituting \\cref{eq:bdd1}-\\cref{eq:bdd4} back into\n\\cref{eq:tobdd}, clearly there are $C_1, C_2 > 0$ such that\n\\begin{equation}\n\\dist\\left( 0, \\partial F\\left( x^t + p^t \\right) \\right)\n\\leq\nC_1 \\sqrt{ - Q^*_t} + C_2 \\left( -Q^*_t \\right)^\\nu \\modify{+\n\tR \\left(\\norm{y^{t} - \\left(x^{t} + p^{t*}\\right)}\\right).}\n\\label{eq:tobdd2}\n\\end{equation}\nNote that $Q_{H_t}(0;x^t) \\equiv 0$, so $-Q^*_t \\geq 0$ from its\noptimality and the right-hand side of  \\cref{eq:tobdd2} is\nwell-defined.\n\\modify{Next, we see from \\cref{eq:Qconv}, \\cref{eq:bdd0}, and the\ncontinuity of $R$ that\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty} \\norm{\\left(y^{t} - x^t\\right) - p^{t*}} =\n0, \\quad \\Rightarrow \\quad\n\\lim_{t \\rightarrow \\infty} R\\left( \\norm{\\left(y^{t} - x^t\\right) -\np^{t*}}\\right) =\n0.\n\\label{eq:sconv}\n\\end{equation}\nApplying \\cref{eq:Qconv} and \\cref{eq:sconv}} to \\cref{eq:tobdd2} and\nletting $t$ approach infinity then yield\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty} \\dist\\left( 0, \\partial F\\left(\nx^t + p^t \\right) \\right) = 0.\n\\label{eq:partialconv}\n\\end{equation}\n\nNext, from \\cref{eq:bdd1} and \\cref{eq:Qconv}, it is also clear that\n$\\norm{p^t} \\rightarrow 0$,\nso from the convergence of $x^t$ to $x^*$ we have\n\\begin{equation}\nx^t + p^t \\rightarrow x^* + 0 = x^*.\n\\label{eq:convxp}\n\\end{equation}\nNow \\cref{eq:convxp} and \\cref{eq:partialconv} allow us to apply\n\\cref{lemma:identify}\nso $x^t + p^t \\in \\M$ for all $t$ large enough,\nleading to the desired contradiction.\n\\qed\n\\end{proof}","start":39081,"end":43250}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":["eq:Qconv"],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"构造结构化坏序列","source_units":[{"raw_tex":"\\begin{proof}\nSuppose the statement is not true for contradiction.\nThen there exist a continuous \\modify{function $R: [0, \\infty)\n\\rightarrow [0, \\infty)$ with}\n$R(0) = 0$, $\\eta_1 \\geq 0$, $M_1 > 0$, a sequence $\\{x^t\\} \\subset\n\\H$ converging to $x^*$,\na sequence $\\{H_t\\} \\subset \\SS$ satisfying \\cref{assum:Hbound} and\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty}\\, \\min_{p}\\quad Q_{H_t}\\left( p;x^t\n\\right) = 0,\n\\label{eq:Qconv}\n\\end{equation}\nthree sequences $\\{p^t\\}, \\{y^t\\}, \\{s^t\\} \\subset \\H$\nand a sequence $\\{\\Lambda_t\\}\\subset \\SS$ with $M_1 \\succeq \\Lambda_t$\nsuch that \\cref{eq:subprob} with \\cref{eq:multiplicative} and\n\\cref{eq:prox}-\\cref{eq:bdd0} hold, yet $x^t + p^t \\notin \\M$\nfor all $t$.\nWe abuse the notation to let $p^{t*}$ and $Q^*_t$\nrespectively denote the optimal solution and objective value for\n$\\min_p Q_{H_t}(p;x^t)$, but $x^{t+1}$ is irrelevant to $p^t$ or\n$\\alpha_t$.\n","start":39081,"end":39984}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":["eq:opt"],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"作者写出的近端最优性包含","source_units":[{"raw_tex":"The optimality condition of \\cref{eq:proximal} applied to\n\\cref{eq:prox} indicates that\n\\begin{equation}\n\t-\\Lambda_{t}\\left(x^{t} - y^{t}\\right) - \\left( \\nabla\n\tf\\left( x^t \\right) + H_t \\left( y^{t} - x^t \\right) + s^{t}\n\t\\right) \\in \\partial \\Psi\\left( x^{t} + p^t \\right).\n\t\\label{eq:opt}\n\\end{equation}\n","start":39985,"end":40293}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":["eq:tobdd"],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"主证明的试探点残差界","source_units":[{"raw_tex":"Thus, we have\n\\begin{align}\n\\nonumber\n&~\\dist\\left( 0, \\partial F\\left( x^t + p^t \\right) \\right)\\\\\n\\nonumber\n\\leq &~\\left\\|\\nabla f\\left( x^t + p^t\n\t\\right) -\\Lambda_{t}\\left(x^t + p^t -\n\ty^{t}\\right) - \\nabla f\\left( x^{t} \\right) \\right.\n\t\\left. -\n\tH_{t} \\left( y^{t} - x^{t} \\right) -\n\ts^{t} \\right\\|\\\\\n\\nonumber\n\\leq &~\\norm{\\nabla f\\left( x^t + p^t \\right)- \\nabla\n\tf\\left( x^{t} \\right)} + M_1 \\left( \\norm{x^t - y^{t}} +\n\t\\norm{p^t}\\right)\n\t+ M\n\t\\norm{y^{t} - x^{t}} + \\norm{s^{t}}\\\\\n\\leq &~(L + M_1) \\norm{p^t} +\n\t(M + M_1) \\norm{y^{t} - x^{t}} +\n\t\\norm{s^{t}}.\n\\label{eq:tobdd}\n\\end{align}\n","start":40293,"end":40893}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"模型误差控制方向大小","source_units":[{"raw_tex":"For the first two terms in \\cref{eq:tobdd}, the\ntriangle inequality and \\cref{eq:qg} imply\n\\begin{gather}\n\t\\begin{aligned}\n\\norm{p^{t}} \n\\leq \\norm{p^{t} - p^{t*}} + \\norm{p^{t*}}\n\\leq\n&~\\sqrt{2m^{-1}}\\sqrt{ Q_{t}\\left( p^{t} \\right) -\nQ^*_{t} } + \\sqrt{2m^{-1}}\\sqrt{ 0 - Q^*_{t}}\\\\\n\\stackrel{\\cref{eq:multiplicative}}{\\leq}&~ \\left( \\sqrt{\\eta} + 1\n\\right)\\sqrt{2m^{-1}}\\sqrt{ - Q^*_{t}}, \\text{ and }\n\\end{aligned}\n\\label{eq:bdd1}\\\\\n","start":40893,"end":41329}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"近端输入点的位移界","source_units":[{"raw_tex":"\\norm{y^{t}- x^{t}}\n\\leq \\norm{\\left(y^{t}- x^{t}\\right) - p^{t*}} +\n\\norm{p^{t*}}\n\\stackrel{\\cref{eq:bdd0}}{\\leq}\n\\eta_1 \\left( - Q^*_{t}\\right)^\\nu + \\sqrt{2m^{-1}}\\sqrt{ - Q^*_{t}}.\n\\label{eq:bdd3}\n\\end{gather}\n","start":41329,"end":41543}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":["eq:bdd4"],"member_ids":["S064"],"declared_external":false,"source_only":false,"title":"保留扰动函数控制","source_units":[{"raw_tex":"For the last term in \\cref{eq:tobdd}, we have from \\modify{our definition of\n$s^t$ that\n\\begin{align}\n\\norm{s^{t}} &\\leq\nR \\left(\\norm{y^{t} - \\left(x^{t} + p^{t*}\\right)}\\right).\n\\label{eq:bdd4}\n\\end{align}\n}\n","start":41543,"end":41753}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":["eq:tobdd2"],"member_ids":["S065"],"declared_external":false,"source_only":false,"title":"合并为模型最优值控制","source_units":[{"raw_tex":"By substituting \\cref{eq:bdd1}-\\cref{eq:bdd4} back into\n\\cref{eq:tobdd}, clearly there are $C_1, C_2 > 0$ such that\n\\begin{equation}\n\\dist\\left( 0, \\partial F\\left( x^t + p^t \\right) \\right)\n\\leq\nC_1 \\sqrt{ - Q^*_t} + C_2 \\left( -Q^*_t \\right)^\\nu \\modify{+\n\tR \\left(\\norm{y^{t} - \\left(x^{t} + p^{t*}\\right)}\\right).}\n\\label{eq:tobdd2}\n\\end{equation}\n","start":41753,"end":42105}]},{"id":"S066","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S066"],"declared_external":false,"source_only":false,"title":"模型最优值的符号","source_units":[{"raw_tex":"Note that $Q_{H_t}(0;x^t) \\equiv 0$, so $-Q^*_t \\geq 0$ from its\noptimality and the right-hand side of  \\cref{eq:tobdd2} is\nwell-defined.\n","start":42105,"end":42243}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sconv"],"member_ids":["S067"],"declared_external":false,"source_only":false,"title":"输入点误差与扰动消失","source_units":[{"raw_tex":"\\modify{Next, we see from \\cref{eq:Qconv}, \\cref{eq:bdd0}, and the\ncontinuity of $R$ that\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty} \\norm{\\left(y^{t} - x^t\\right) - p^{t*}} =\n0, \\quad \\Rightarrow \\quad\n\\lim_{t \\rightarrow \\infty} R\\left( \\norm{\\left(y^{t} - x^t\\right) -\np^{t*}}\\right) =\n0.\n\\label{eq:sconv}\n\\end{equation}\n","start":42243,"end":42569}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":["eq:partialconv"],"member_ids":["S068"],"declared_external":false,"source_only":false,"title":"试探点驻点残差消失","source_units":[{"raw_tex":"Applying \\cref{eq:Qconv} and \\cref{eq:sconv}} to \\cref{eq:tobdd2} and\nletting $t$ approach infinity then yield\n\\begin{equation}\n\\lim_{t \\rightarrow \\infty} \\dist\\left( 0, \\partial F\\left(\nx^t + p^t \\right) \\right) = 0.\n\\label{eq:partialconv}\n\\end{equation}\n","start":42569,"end":42826}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":["eq:convxp"],"member_ids":["S069"],"declared_external":false,"source_only":false,"title":"试探点收敛到非退化点","source_units":[{"raw_tex":"\nNext, from \\cref{eq:bdd1} and \\cref{eq:Qconv}, it is also clear that\n$\\norm{p^t} \\rightarrow 0$,\nso from the convergence of $x^t$ to $x^*$ we have\n\\begin{equation}\nx^t + p^t \\rightarrow x^* + 0 = x^*.\n\\label{eq:convxp}\n\\end{equation}\n","start":42826,"end":43061}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S070"],"declared_external":false,"source_only":false,"title":"应用凸识别引理并得矛盾","source_units":[{"raw_tex":"Now \\cref{eq:convxp} and \\cref{eq:partialconv} allow us to apply\n\\cref{lemma:identify}\nso $x^t + p^t \\in \\M$ for all $t$ large enough,\nleading to the desired contradiction.\n\\qed\n\\end{proof}","start":43061,"end":43250}]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S071"],"declared_external":true,"declaration_text":["任务明确声明的先决结果 LewZ13a, Theorem 4.10；原文第722行和第739–742行调用。"],"source_only":false,"title":"外部部分光滑识别定理","source_units":[]},{"id":"S072","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S072"],"declared_external":true,"declaration_text":["任务明确声明的先决结果 RocW09a, Example 13.30；原文第766–769行调用。"],"source_only":false,"title":"凸函数的次微分连续性","source_units":[]},{"id":"S073","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S073"],"declared_external":true,"declaration_text":["任务明确声明的先决结果 Wri12a, Lemma 3.2；原文第848–854行援引其论证。"],"source_only":false,"title":"精确模型步的外部局部估计","source_units":[]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S074"],"declared_external":true,"declaration_text":["任务声明允许的 standard convex subdifferential calculus and proximal optimality；一般识别引理中的平滑项平移为 annotator 显式标出的 implicit standard fact。"],"source_only":false,"title":"次微分与近端最优性规则","source_units":[]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S075"],"declared_external":true,"declaration_text":["任务声明允许的 strong-convexity quadratic growth；原文第836–842行以“which implies”使用，annotator 将标准事实显式化。"],"source_only":false,"title":"强凸函数的二次增长","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S026":"S026","S028":"S028","S029":"S029","S030":"S030","S031":"S030","S032":"S030","S033":"S030","S034":"S030","S035":"S030","S036":"S036","S037":"S037","S038":"S037","S039":"S037","S040":"S037","S041":"S041","S042":"S042","S043":"S042","S044":"S042","S045":"S042","S046":"S042","S047":"S047","S048":"S042","S049":"S042","S050":"S042","S051":"S042","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075"}}}}
{"record_id":"1408.4685v3","paper":{"title":"Partial facial reduction: simplified, equivalent SDPs via approximations of the PSD cone","authors":["Permenter, Frank","Parrilo, Pablo"],"arxiv_id":"1408.4685","version":"v3","journal":"Mathematical Programming","abstract_url":"https://arxiv.org/abs/1408.4685v3","pdf_url":"https://arxiv.org/pdf/1408.4685v3","doi":"10.1007/s10107-017-1169-9"},"context":{"raw_tex":"Let $\\mathcal{E}$ denote a finite-dimensional vector space over $\\mathbb{R}$ with inner product $\\langle \\cdot, \\cdot \\rangle$.  For a subset $\\mathcal{S}$ of $\\mathcal{E}$, let $\\lin \\mathcal{S} \\subseteq \\mathcal{E}$ denote \nthe linear span of elements in $\\mathcal{S}$ and let $\\mathcal{S}^{\\perp} \\subseteq \\mathcal{E}$ denote the orthogonal complement of $\\lin \\mathcal{S}$.  For $y \\in \\mathcal{E}$, let $\\lin y$ denote $\\lin \\{ y \\}$ and let $y^{\\perp}$  denote $\\{ y \\}^{\\perp}$.\nA  convex cone $\\coneName$ is a  convex subset of $\\mathcal{E}$ (not necessarily full dimensional) that satisfies\n\\[\n x  \\in \\coneName \\Rightarrow  \\alpha x \\in \\coneName \\;\\;\\; \\forall \\alpha \\ge 0.\n\\]\nThe dual cone of $\\coneName$, denoted  $\\coneName^*$, is the set of linear functionals non-negative on $\\coneName$:\n\\[\n  \\coneName^* := \\left\\{ y : \\langle y,x\\rangle \\ge 0 \\;\\;\\; \\forall x \\in \\coneName\\right\\}.\n\\]\n\n\nFor discussions specific to semidefinite programming, we let $\\mathbb{S}^n$ denote the vector space of $n\\times n$ symmetric matrices and $\\mathbb{S}^n_{+} \\subseteq \\mathbb{S}^n$\ndenote the convex cone of matrices that are positive semidefinite. We will use capital letters to denote elements of\n$\\mathbb{S}^n$ to emphasize that they are matrices.  For $A,B \\in \\mathbb{S}^n$, we let $A \\cdot B$  denote the trace inner product $\\trace AB$. \nFinally, we let  $A \\succeq 0$ (resp. $A \\succ 0$)  denote the condition that $A$ is positive semidefinite (resp.  positive definite).\n\n\n\\subsection{Approximations of $\\mathbb{S}^d_{+}$} \\label{sec:approx}\nIn this section, we explore an  outer approximation $\\outerName$ of  $\\mathbb{S}^d_{+}$  parametrized by a set $\\setName$\nof $d \\times k$ rectangular matrices. The parametrization is chosen such that the dual cone $\\approxName$ equals the \\emph{Minkowski sum} of faces $W_i \\mathbb{S}^{k}_{+} W_i^T$ of $\\mathbb{S}^d_{+}$ \nfor $W_i \\in \\setName$.  It is defined below:\n","units":[{"raw_tex":"Let $\\mathcal{E}$ denote a finite-dimensional vector space over $\\mathbb{R}$ with inner product $\\langle \\cdot, \\cdot \\rangle$.  For a subset $\\mathcal{S}$ of $\\mathcal{E}$, let $\\lin \\mathcal{S} \\subseteq \\mathcal{E}$ denote \nthe linear span of elements in $\\mathcal{S}$ and let $\\mathcal{S}^{\\perp} \\subseteq \\mathcal{E}$ denote the orthogonal complement of $\\lin \\mathcal{S}$.  For $y \\in \\mathcal{E}$, let $\\lin y$ denote $\\lin \\{ y \\}$ and let $y^{\\perp}$  denote $\\{ y \\}^{\\perp}$.\nA  convex cone $\\coneName$ is a  convex subset of $\\mathcal{E}$ (not necessarily full dimensional) that satisfies\n\\[\n x  \\in \\coneName \\Rightarrow  \\alpha x \\in \\coneName \\;\\;\\; \\forall \\alpha \\ge 0.\n\\]\nThe dual cone of $\\coneName$, denoted  $\\coneName^*$, is the set of linear functionals non-negative on $\\coneName$:\n\\[\n  \\coneName^* := \\left\\{ y : \\langle y,x\\rangle \\ge 0 \\;\\;\\; \\forall x \\in \\coneName\\right\\}.\n\\]\n","start":14718,"end":15625},{"raw_tex":"For discussions specific to semidefinite programming, we let $\\mathbb{S}^n$ denote the vector space of $n\\times n$ symmetric matrices and $\\mathbb{S}^n_{+} \\subseteq \\mathbb{S}^n$\ndenote the convex cone of matrices that are positive semidefinite. We will use capital letters to denote elements of\n$\\mathbb{S}^n$ to emphasize that they are matrices.  For $A,B \\in \\mathbb{S}^n$, we let $A \\cdot B$  denote the trace inner product $\\trace AB$. \nFinally, we let  $A \\succeq 0$ (resp. $A \\succ 0$)  denote the condition that $A$ is positive semidefinite (resp.  positive definite).\n","start":16411,"end":16989},{"raw_tex":"\\subsection{Approximations of $\\mathbb{S}^d_{+}$} \\label{sec:approx}\nIn this section, we explore an  outer approximation $\\outerName$ of  $\\mathbb{S}^d_{+}$  parametrized by a set $\\setName$\nof $d \\times k$ rectangular matrices. The parametrization is chosen such that the dual cone $\\approxName$ equals the \\emph{Minkowski sum} of faces $W_i \\mathbb{S}^{k}_{+} W_i^T$ of $\\mathbb{S}^d_{+}$ \nfor $W_i \\in \\setName$.  It is defined below:\n","start":41239,"end":41677}]},"statement":{"raw_tex":"\\begin{cor} \\label{cor:maxRank}\n A matrix $X \\in \\perpName \\cap \\approxName$ of maximum rank is given by any optimal solution\n $(X,X_i,T_i)$   to the SDP \\eqref{sdp:sumOfConesMaxRank}. \n\\end{cor}","units":[{"raw_tex":"\\begin{cor} \\label{cor:maxRank}\n A matrix $X \\in \\perpName \\cap \\approxName$ of maximum rank is given by any optimal solution\n $(X,X_i,T_i)$   to the SDP \\eqref{sdp:sumOfConesMaxRank}. \n\\end{cor}","start":62385,"end":62580}]},"proof":{"raw_tex":"\\begin{lem}\nFor a set $\\setName := \\left\\{ W_1, W_2,\\ldots, W_{|\\setName|} \\right\\}$ of $d \\times k$ matrices, let $\\outerName$ denote the following convex cone:\n\\begin{align*}\n\\outerName := \\left\\{  X \\in  \\mathbb{S}^d : W_i^T X W_i \\in \\mathbb{S}^k_{+} \\;\\;\\;i = 1, \\ldots, |\\setName| \\right\\}.\n\\end{align*}\nThe dual cone $\\approxName$ satisfies\n\\begin{align} \\label{def:coneProdInner}\n \\approxName = \\left\\{ \\sum^{|\\setName|}_{i=1}  W_i X_i W_i^T : X_i \\in \\mathbb{S}_{+}^k  \\right\\},\n\\end{align}\nand the following inclusions hold:\n\\[\n\\approxName \\subseteq \\mathbb{S}^d_{+} \\subseteq \\outerName.\n\\]\n\\begin{proof}\nThe inclusions are obvious from the definitions of $\\approxName$ and $\\outerName$ (as is the fact that  $\\outerName$ is a convex cone).  It remains to show correctness of  \\eqref{def:coneProdInner}. To show this,\nlet $\\mathcal{T}$ denote the set on the right-hand side of \\eqref{def:coneProdInner}.  It is easy to check that  $\\mathcal{T}^*= \\outerName$, which implies $\\mathcal{T}^{**} = \\approxName$.\nSince $\\mathcal{T}$ is a convex cone (as is easily checked), $\\mathcal{T}^{**}$ equals the closure of $\\mathcal{T}$. The result therefore follows by showing $\\mathcal{T}$ is closed. To see this, note that $\\mathcal{T}$ equals the Minkowski sum of closed cones $W_i \\mathbb{S}^k_{+} W_i^T$.\nFor  matrices $Z_i \\in W_i \\mathbb{S}^k_{+} W_i^T$, we have that $\\sum^{|\\setName|}_{i=1} Z_i=0$\nonly if $Z_i= 0$ for each $i$.  This shows that $\\sum^{|\\setName|}_{i =1}Z_i = 0$ only if $Z_i$ is in the lineality space of $W_i \\mathbb{S}^k_{+} W_i^T$.\n Direct application of the closedness criteria Corollary 9.1.3 of Rockafellar \\cite{rockafellar1997convex} shows $\\mathcal{T}$ is closed.\n\\end{proof}\n\n\\end{lem}\n\n\\begin{lem} \\label{lem:rankX_i} \nLet $\\perpName$ be a subspace of $\\mathbb{S}^d$. If $X^\\star:=\\sum^{|\\setName|}_{i=1} W_i X^\\star_i W^T_i$  maximizes   $\\sum^{|\\setName|}_{i=1} \\rank X_i$ over $\\perpName \\cap \\approxName$, then    $X^\\star$  maximizes $\\rank X$ over $\\perpName \\cap \\approxName$.\n\\begin{proof}\n\nWe will  argue the kernel of $X^\\star$ is contained in the kernel of any  $X \\in \\perpName \\cap \\approxName$, which immediately implies $\\rank X^\\star \\ge \\rank X$. \n\nTo begin, we first argue for any $X = \\sum^{|\\setName|}_{i=1} W_i X_i W^T_i \\in  \\perpName \\cap \\approxName$ that $\\nullspace X^\\star_i \\subseteq \\nullspace X_i$ for all $i \\in \\{1,\\ldots,|\\setName|\\}$. To see this, first  note  that for any $X  \\in  \\perpName \\cap \\approxName$\nthe matrix\n\\[\n X^\\star + X =  \\sum^{|\\setName|}_{i=1} W_i ( X^\\star_i + X_i )W^T_i \n\\]\nis also in  $\\perpName \\cap \\approxName$ and  satisfies $\\rank (X^\\star_i + X_i) \\ge \\rank X^\\star_i$.  Now suppose for some $d \\in \\{1,\\ldots,|\\setName|\\}$ that  $\\nullspace X^\\star_d \\not\\subseteq \\nullspace X_d$. This implies that  $\\nullspace (X^\\star_d + X_d) =  \\nullspace X^\\star_d \\cap \\nullspace X_d  \\subset \\nullspace X^\\star_d$ which in turn implies $\\rank (X^\\star_d + X_d ) > \\rank X^\\star_d$.\nBut this contradicts our assumption that  $X^\\star$ maximizes $\\sum_i \\rank X_i$.  Hence, $\\nullspace X^\\star_i \\subseteq \\nullspace X_i$ for all $i \\in \\{1,\\ldots,|\\setName|\\}$.\n\nNow suppose an  $X \\in \\perpName \\cap \\approxName$ exists for which  $X^\\star w = 0$ but $Xw \\ne 0$ for some $w$.  Since $X w = 0$ if and only if $X_i W_i^T w = 0$ for all $i$, we must have for some $d$ that $W_d^Tw$ is in the kernel of $X^\\star_d$ but not in the kernel of $X_d$. But we have already established that $\\nullspace X^\\star_d \\subseteq \\nullspace X_d$.  Hence, $w$ cannot exist. We therefore have that $\\nullspace X^\\star \\subseteq \\nullspace X$ for any $X\\in \\perpName \\cap \\approxName$, which completes the proof. \n\\end{proof}\n\\end{lem}\n\n\\begin{lem}  \\label{lem:sumOfConesMaxRank}\nLet $\\perpName$ be a subspace of $\\mathbb{S}^d$.  A matrix $X$ maximizing $\\sum^{|\\setName|}_{i=1} \\rank X_i$  over $\\perpName \\cap \\approxName$  is given by any optimal solution $(X,X_i,T_i)$  to the following SDP:\n\\begin{align} \\label{sdp:sumOfConesMaxRank}\n\\begin{array}{lll}\n \\mbox{\\rm maximize}  &  \\sum^{|\\setName|}_{i=1} \\trace{T_i} \\  \\\\\n \\mbox{\\rm subject to } & X \\in \\perpName,\\\\\n&X = \\sum^{|\\setName|}_{i=1} W_i X_i W^T_i   & \\mbox{ \\rm  i.e. } X \\in \\approxName \\\\\n& X_i \\succeq T_i  & \\forall i\\in\\{1, \\ldots, |\\setName|\\}\\\\\n&  I \\succeq  T_i \\succeq 0 &  \\forall i\\in\\{ 1, \\ldots, |\\setName|\\}.\n\\end{array} \n\\end{align}\n\n\n\\begin{proof}\nLet $r_{\\max}$ equal the maximum of  $\\sum^{|\\setName|}_{i=1} \\rank X_i$ over the  set of feasible $X_i$.  We will\nshow at optimality $\\sum^{|\\setName|}_{i=1} \\rank X_i = r_{\\max}$. \n\nTo begin, the constraint $I \\succeq T_i \\succeq 0$ implies the eigenvalues of $T_i$ are  less than one. Hence,  $\\rank T_i  \\ge \\trace T_i $.  \nSince  $X_i \\succeq T_i$ , we also have $\\rank{X_i} \\ge \\rank{T_i}$.  Thus,  any feasible $(X_i,T_i)$ pair satisfies\n\\begin{align} \\label{ineq:maxrank}\nr_{\\max} \\ge \\sum^{|\\setName|}_{i=1} \\rank{X_i}  \\ge   \\sum^{|\\setName|}_{i=1} \\rank{T_i}  \\ge  \\sum^{|\\setName|}_{i=1} \\trace T_i.\n\\end{align}\n\nNow note for any feasible $(X,X_i)$ we can pick $\\alpha > 0$  and construct a feasible point $(\\alpha X, \\alpha X_i, \\hat{T}_i)$  that satisfies $\\sum^{|\\setName|}_{i=1} \\trace{\\hat{T_i}} =  \\sum^{|\\setName|}_{i=1}\\rank{X_i}$; if $X_i$ has  eigen-decomposition $\\sum_j \\lambda_j u_j u_j^T$ for $\\lambda_j > 0$, simply take  $\\hat{T_i} = \\sum_j u_j u_j^T$ and $\\alpha$ equal to\n\\[\n\\max \\bigcup_i \\left\\{ \\frac{1}{\\lambda} : \\lambda \\mbox{ is a  positive eigenvalue of $X_i$ } \\right\\}.\n\\]  \nHence, some feasible point $(\\hat{X},\\hat{X_i},\\hat{T_i})$ satisfies  $\\sum^{|\\setName|}_{i=1} \\trace{\\hat{T_i}} = r_{\\max}$. Therefore, the optimal $(X,X_i,T_i)$ satisfies \n\\[\n\\sum^{|\\setName|}_{i=1} \\trace T_i  \\ge  r_{\\max}.\n\\]\nCombining this inequality with \\eqref{ineq:maxrank} yields that at optimality\n\\[\n\\sum^{|\\setName|}_{i=1} \\trace T_i   = \\sum^{|\\setName|}_{i=1}\\rank{X_i}  = r_{\\max},\n\\]\nwhich completes the proof.\n\\end{proof}\n\\end{lem}","units":[{"raw_tex":"\\begin{lem}\nFor a set $\\setName := \\left\\{ W_1, W_2,\\ldots, W_{|\\setName|} \\right\\}$ of $d \\times k$ matrices, let $\\outerName$ denote the following convex cone:\n\\begin{align*}\n\\outerName := \\left\\{  X \\in  \\mathbb{S}^d : W_i^T X W_i \\in \\mathbb{S}^k_{+} \\;\\;\\;i = 1, \\ldots, |\\setName| \\right\\}.\n\\end{align*}\nThe dual cone $\\approxName$ satisfies\n\\begin{align} \\label{def:coneProdInner}\n \\approxName = \\left\\{ \\sum^{|\\setName|}_{i=1}  W_i X_i W_i^T : X_i \\in \\mathbb{S}_{+}^k  \\right\\},\n\\end{align}\nand the following inclusions hold:\n\\[\n\\approxName \\subseteq \\mathbb{S}^d_{+} \\subseteq \\outerName.\n\\]\n\\begin{proof}\nThe inclusions are obvious from the definitions of $\\approxName$ and $\\outerName$ (as is the fact that  $\\outerName$ is a convex cone).  It remains to show correctness of  \\eqref{def:coneProdInner}. To show this,\nlet $\\mathcal{T}$ denote the set on the right-hand side of \\eqref{def:coneProdInner}.  It is easy to check that  $\\mathcal{T}^*= \\outerName$, which implies $\\mathcal{T}^{**} = \\approxName$.\nSince $\\mathcal{T}$ is a convex cone (as is easily checked), $\\mathcal{T}^{**}$ equals the closure of $\\mathcal{T}$. The result therefore follows by showing $\\mathcal{T}$ is closed. To see this, note that $\\mathcal{T}$ equals the Minkowski sum of closed cones $W_i \\mathbb{S}^k_{+} W_i^T$.\nFor  matrices $Z_i \\in W_i \\mathbb{S}^k_{+} W_i^T$, we have that $\\sum^{|\\setName|}_{i=1} Z_i=0$\nonly if $Z_i= 0$ for each $i$.  This shows that $\\sum^{|\\setName|}_{i =1}Z_i = 0$ only if $Z_i$ is in the lineality space of $W_i \\mathbb{S}^k_{+} W_i^T$.\n Direct application of the closedness criteria Corollary 9.1.3 of Rockafellar \\cite{rockafellar1997convex} shows $\\mathcal{T}$ is closed.\n\\end{proof}\n\n\\end{lem}","start":41677,"end":43398},{"raw_tex":"\\begin{lem} \\label{lem:rankX_i} \nLet $\\perpName$ be a subspace of $\\mathbb{S}^d$. If $X^\\star:=\\sum^{|\\setName|}_{i=1} W_i X^\\star_i W^T_i$  maximizes   $\\sum^{|\\setName|}_{i=1} \\rank X_i$ over $\\perpName \\cap \\approxName$, then    $X^\\star$  maximizes $\\rank X$ over $\\perpName \\cap \\approxName$.\n\\begin{proof}\n\nWe will  argue the kernel of $X^\\star$ is contained in the kernel of any  $X \\in \\perpName \\cap \\approxName$, which immediately implies $\\rank X^\\star \\ge \\rank X$. \n\nTo begin, we first argue for any $X = \\sum^{|\\setName|}_{i=1} W_i X_i W^T_i \\in  \\perpName \\cap \\approxName$ that $\\nullspace X^\\star_i \\subseteq \\nullspace X_i$ for all $i \\in \\{1,\\ldots,|\\setName|\\}$. To see this, first  note  that for any $X  \\in  \\perpName \\cap \\approxName$\nthe matrix\n\\[\n X^\\star + X =  \\sum^{|\\setName|}_{i=1} W_i ( X^\\star_i + X_i )W^T_i \n\\]\nis also in  $\\perpName \\cap \\approxName$ and  satisfies $\\rank (X^\\star_i + X_i) \\ge \\rank X^\\star_i$.  Now suppose for some $d \\in \\{1,\\ldots,|\\setName|\\}$ that  $\\nullspace X^\\star_d \\not\\subseteq \\nullspace X_d$. This implies that  $\\nullspace (X^\\star_d + X_d) =  \\nullspace X^\\star_d \\cap \\nullspace X_d  \\subset \\nullspace X^\\star_d$ which in turn implies $\\rank (X^\\star_d + X_d ) > \\rank X^\\star_d$.\nBut this contradicts our assumption that  $X^\\star$ maximizes $\\sum_i \\rank X_i$.  Hence, $\\nullspace X^\\star_i \\subseteq \\nullspace X_i$ for all $i \\in \\{1,\\ldots,|\\setName|\\}$.\n\nNow suppose an  $X \\in \\perpName \\cap \\approxName$ exists for which  $X^\\star w = 0$ but $Xw \\ne 0$ for some $w$.  Since $X w = 0$ if and only if $X_i W_i^T w = 0$ for all $i$, we must have for some $d$ that $W_d^Tw$ is in the kernel of $X^\\star_d$ but not in the kernel of $X_d$. But we have already established that $\\nullspace X^\\star_d \\subseteq \\nullspace X_d$.  Hence, $w$ cannot exist. We therefore have that $\\nullspace X^\\star \\subseteq \\nullspace X$ for any $X\\in \\perpName \\cap \\approxName$, which completes the proof. \n\\end{proof}\n\\end{lem}","start":57705,"end":59691},{"raw_tex":"\\begin{lem}  \\label{lem:sumOfConesMaxRank}\nLet $\\perpName$ be a subspace of $\\mathbb{S}^d$.  A matrix $X$ maximizing $\\sum^{|\\setName|}_{i=1} \\rank X_i$  over $\\perpName \\cap \\approxName$  is given by any optimal solution $(X,X_i,T_i)$  to the following SDP:\n\\begin{align} \\label{sdp:sumOfConesMaxRank}\n\\begin{array}{lll}\n \\mbox{\\rm maximize}  &  \\sum^{|\\setName|}_{i=1} \\trace{T_i} \\  \\\\\n \\mbox{\\rm subject to } & X \\in \\perpName,\\\\\n&X = \\sum^{|\\setName|}_{i=1} W_i X_i W^T_i   & \\mbox{ \\rm  i.e. } X \\in \\approxName \\\\\n& X_i \\succeq T_i  & \\forall i\\in\\{1, \\ldots, |\\setName|\\}\\\\\n&  I \\succeq  T_i \\succeq 0 &  \\forall i\\in\\{ 1, \\ldots, |\\setName|\\}.\n\\end{array} \n\\end{align}\n\n\n\\begin{proof}\nLet $r_{\\max}$ equal the maximum of  $\\sum^{|\\setName|}_{i=1} \\rank X_i$ over the  set of feasible $X_i$.  We will\nshow at optimality $\\sum^{|\\setName|}_{i=1} \\rank X_i = r_{\\max}$. \n\nTo begin, the constraint $I \\succeq T_i \\succeq 0$ implies the eigenvalues of $T_i$ are  less than one. Hence,  $\\rank T_i  \\ge \\trace T_i $.  \nSince  $X_i \\succeq T_i$ , we also have $\\rank{X_i} \\ge \\rank{T_i}$.  Thus,  any feasible $(X_i,T_i)$ pair satisfies\n\\begin{align} \\label{ineq:maxrank}\nr_{\\max} \\ge \\sum^{|\\setName|}_{i=1} \\rank{X_i}  \\ge   \\sum^{|\\setName|}_{i=1} \\rank{T_i}  \\ge  \\sum^{|\\setName|}_{i=1} \\trace T_i.\n\\end{align}\n\nNow note for any feasible $(X,X_i)$ we can pick $\\alpha > 0$  and construct a feasible point $(\\alpha X, \\alpha X_i, \\hat{T}_i)$  that satisfies $\\sum^{|\\setName|}_{i=1} \\trace{\\hat{T_i}} =  \\sum^{|\\setName|}_{i=1}\\rank{X_i}$; if $X_i$ has  eigen-decomposition $\\sum_j \\lambda_j u_j u_j^T$ for $\\lambda_j > 0$, simply take  $\\hat{T_i} = \\sum_j u_j u_j^T$ and $\\alpha$ equal to\n\\[\n\\max \\bigcup_i \\left\\{ \\frac{1}{\\lambda} : \\lambda \\mbox{ is a  positive eigenvalue of $X_i$ } \\right\\}.\n\\]  \nHence, some feasible point $(\\hat{X},\\hat{X_i},\\hat{T_i})$ satisfies  $\\sum^{|\\setName|}_{i=1} \\trace{\\hat{T_i}} = r_{\\max}$. Therefore, the optimal $(X,X_i,T_i)$ satisfies \n\\[\n\\sum^{|\\setName|}_{i=1} \\trace T_i  \\ge  r_{\\max}.\n\\]\nCombining this inequality with \\eqref{ineq:maxrank} yields that at optimality\n\\[\n\\sum^{|\\setName|}_{i=1} \\trace T_i   = \\sum^{|\\setName|}_{i=1}\\rank{X_i}  = r_{\\max},\n\\]\nwhich completes the proof.\n\\end{proof}\n\\end{lem}","start":60020,"end":62278}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"有限维实内积空间","source_units":[{"raw_tex":"Let $\\mathcal{E}$ denote a finite-dimensional vector space over $\\mathbb{R}$ with inner product $\\langle \\cdot, \\cdot \\rangle$.  For a subset $\\mathcal{S}$ of $\\mathcal{E}$, let $\\lin \\mathcal{S} \\subseteq \\mathcal{E}$ denote \nthe linear span of elements in $\\mathcal{S}$ and let $\\mathcal{S}^{\\perp} \\subseteq \\mathcal{E}$ denote the orthogonal complement of $\\lin \\mathcal{S}$.  For $y \\in \\mathcal{E}$, let $\\lin y$ denote $\\lin \\{ y \\}$ and let $y^{\\perp}$  denote $\\{ y \\}^{\\perp}$.\n","start":14718,"end":15206}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"线性张成与正交补","source_units":[{"raw_tex":"Let $\\mathcal{E}$ denote a finite-dimensional vector space over $\\mathbb{R}$ with inner product $\\langle \\cdot, \\cdot \\rangle$.  For a subset $\\mathcal{S}$ of $\\mathcal{E}$, let $\\lin \\mathcal{S} \\subseteq \\mathcal{E}$ denote \nthe linear span of elements in $\\mathcal{S}$ and let $\\mathcal{S}^{\\perp} \\subseteq \\mathcal{E}$ denote the orthogonal complement of $\\lin \\mathcal{S}$.  For $y \\in \\mathcal{E}$, let $\\lin y$ denote $\\lin \\{ y \\}$ and let $y^{\\perp}$  denote $\\{ y \\}^{\\perp}$.\n","start":14718,"end":15206}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"凸锥定义","source_units":[{"raw_tex":"A  convex cone $\\coneName$ is a  convex subset of $\\mathcal{E}$ (not necessarily full dimensional) that satisfies\n\\[\n x  \\in \\coneName \\Rightarrow  \\alpha x \\in \\coneName \\;\\;\\; \\forall \\alpha \\ge 0.\n\\]\n","start":15206,"end":15409}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"对偶锥定义","source_units":[{"raw_tex":"The dual cone of $\\coneName$, denoted  $\\coneName^*$, is the set of linear functionals non-negative on $\\coneName$:\n\\[\n  \\coneName^* := \\left\\{ y : \\langle y,x\\rangle \\ge 0 \\;\\;\\; \\forall x \\in \\coneName\\right\\}.\n\\]\n","start":15409,"end":15625}]},{"id":"S005","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"对称矩阵与半正定锥","source_units":[{"raw_tex":"For discussions specific to semidefinite programming, we let $\\mathbb{S}^n$ denote the vector space of $n\\times n$ symmetric matrices and $\\mathbb{S}^n_{+} \\subseteq \\mathbb{S}^n$\ndenote the convex cone of matrices that are positive semidefinite. We will use capital letters to denote elements of\n","start":16411,"end":16708}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"迹内积","source_units":[{"raw_tex":"$\\mathbb{S}^n$ to emphasize that they are matrices.  For $A,B \\in \\mathbb{S}^n$, we let $A \\cdot B$  denote the trace inner product $\\trace AB$. \n","start":16708,"end":16854}]},{"id":"S007","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"半正定序记号","source_units":[{"raw_tex":"Finally, we let  $A \\succeq 0$ (resp. $A \\succ 0$)  denote the condition that $A$ is positive semidefinite (resp.  positive definite).\n","start":16854,"end":16989}]},{"id":"S008","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"参数化近似锥族","source_units":[{"raw_tex":"\\subsection{Approximations of $\\mathbb{S}^d_{+}$} \\label{sec:approx}\nIn this section, we explore an  outer approximation $\\outerName$ of  $\\mathbb{S}^d_{+}$  parametrized by a set $\\setName$\nof $d \\times k$ rectangular matrices. The parametrization is chosen such that the dual cone $\\approxName$ equals the \\emph{Minkowski sum} of faces $W_i \\mathbb{S}^{k}_{+} W_i^T$ of $\\mathbb{S}^d_{+}$ \nfor $W_i \\in \\setName$.  It is defined below:\n","start":41239,"end":41677}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"有限矩阵族参数","source_units":[{"raw_tex":"For a set $\\setName := \\left\\{ W_1, W_2,\\ldots, W_{|\\setName|} \\right\\}$ of $d \\times k$ matrices, let $\\outerName$ denote the following convex cone:\n","start":41689,"end":41839}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"压缩约束外近似","source_units":[{"raw_tex":"For a set $\\setName := \\left\\{ W_1, W_2,\\ldots, W_{|\\setName|} \\right\\}$ of $d \\times k$ matrices, let $\\outerName$ denote the following convex cone:\n\\begin{align*}\n\\outerName := \\left\\{  X \\in  \\mathbb{S}^d : W_i^T X W_i \\in \\mathbb{S}^k_{+} \\;\\;\\;i = 1, \\ldots, |\\setName| \\right\\}.\n\\end{align*}\n","start":41689,"end":41987}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":["def:coneProdInner"],"declared_external":false,"source_only":false,"title":"半正定块和集合","source_units":[{"raw_tex":"\\begin{align} \\label{def:coneProdInner}\n \\approxName = \\left\\{ \\sum^{|\\setName|}_{i=1}  W_i X_i W_i^T : X_i \\in \\mathbb{S}_{+}^k  \\right\\},\n\\end{align}\n","start":42025,"end":42177}]},{"id":"S012","semantic_type":"lemma","kind":"lemma","labels":["def:coneProdInner"],"declared_external":false,"source_only":false,"title":"近似锥的对偶表示","source_units":[{"raw_tex":"\\begin{lem}\nFor a set $\\setName := \\left\\{ W_1, W_2,\\ldots, W_{|\\setName|} \\right\\}$ of $d \\times k$ matrices, let $\\outerName$ denote the following convex cone:\n\\begin{align*}\n\\outerName := \\left\\{  X \\in  \\mathbb{S}^d : W_i^T X W_i \\in \\mathbb{S}^k_{+} \\;\\;\\;i = 1, \\ldots, |\\setName| \\right\\}.\n\\end{align*}\nThe dual cone $\\approxName$ satisfies\n\\begin{align} \\label{def:coneProdInner}\n \\approxName = \\left\\{ \\sum^{|\\setName|}_{i=1}  W_i X_i W_i^T : X_i \\in \\mathbb{S}_{+}^k  \\right\\},\n\\end{align}\nand the following inclusions hold:\n\\[\n\\approxName \\subseteq \\mathbb{S}^d_{+} \\subseteq \\outerName.\n\\]\n","start":41677,"end":42279},{"raw_tex":"\n\\end{lem}","start":43388,"end":43398}]},{"id":"S013","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"内外近似包含关系","source_units":[{"raw_tex":"and the following inclusions hold:\n\\[\n\\approxName \\subseteq \\mathbb{S}^d_{+} \\subseteq \\outerName.\n\\]\n","start":42177,"end":42279},{"raw_tex":"The inclusions are obvious from the definitions of $\\approxName$ and $\\outerName$ (as is the fact that  $\\outerName$ is a convex cone).  It remains to show correctness of  \\eqref{def:coneProdInner}. To show this,\n","start":42293,"end":42506}]},{"id":"S014","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"对偶表示的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nThe inclusions are obvious from the definitions of $\\approxName$ and $\\outerName$ (as is the fact that  $\\outerName$ is a convex cone).  It remains to show correctness of  \\eqref{def:coneProdInner}. To show this,\nlet $\\mathcal{T}$ denote the set on the right-hand side of \\eqref{def:coneProdInner}.  It is easy to check that  $\\mathcal{T}^*= \\outerName$, which implies $\\mathcal{T}^{**} = \\approxName$.\nSince $\\mathcal{T}$ is a convex cone (as is easily checked), $\\mathcal{T}^{**}$ equals the closure of $\\mathcal{T}$. The result therefore follows by showing $\\mathcal{T}$ is closed. To see this, note that $\\mathcal{T}$ equals the Minkowski sum of closed cones $W_i \\mathbb{S}^k_{+} W_i^T$.\nFor  matrices $Z_i \\in W_i \\mathbb{S}^k_{+} W_i^T$, we have that $\\sum^{|\\setName|}_{i=1} Z_i=0$\nonly if $Z_i= 0$ for each $i$.  This shows that $\\sum^{|\\setName|}_{i =1}Z_i = 0$ only if $Z_i$ is in the lineality space of $W_i \\mathbb{S}^k_{+} W_i^T$.\n Direct application of the closedness criteria Corollary 9.1.3 of Rockafellar \\cite{rockafellar1997convex} shows $\\mathcal{T}$ is closed.\n\\end{proof}\n","start":42279,"end":43388}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"计算块和锥的对偶","source_units":[{"raw_tex":"let $\\mathcal{T}$ denote the set on the right-hand side of \\eqref{def:coneProdInner}.  It is easy to check that  $\\mathcal{T}^*= \\outerName$, which implies $\\mathcal{T}^{**} = \\approxName$.\n","start":42506,"end":42696}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"双极归结为闭性","source_units":[{"raw_tex":"Since $\\mathcal{T}$ is a convex cone (as is easily checked), $\\mathcal{T}^{**}$ equals the closure of $\\mathcal{T}$. The result therefore follows by showing $\\mathcal{T}$ is closed. To see this, note that $\\mathcal{T}$ equals the Minkowski sum of closed cones $W_i \\mathbb{S}^k_{+} W_i^T$.\n","start":42696,"end":42986}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"半正定项不能相消","source_units":[{"raw_tex":"For  matrices $Z_i \\in W_i \\mathbb{S}^k_{+} W_i^T$, we have that $\\sum^{|\\setName|}_{i=1} Z_i=0$\nonly if $Z_i= 0$ for each $i$.  This shows that $\\sum^{|\\setName|}_{i =1}Z_i = 0$ only if $Z_i$ is in the lineality space of $W_i \\mathbb{S}^k_{+} W_i^T$.\n","start":42986,"end":43238}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"应用锥和闭性判据","source_units":[{"raw_tex":"Since $\\mathcal{T}$ is a convex cone (as is easily checked), $\\mathcal{T}^{**}$ equals the closure of $\\mathcal{T}$. The result therefore follows by showing $\\mathcal{T}$ is closed. To see this, note that $\\mathcal{T}$ equals the Minkowski sum of closed cones $W_i \\mathbb{S}^k_{+} W_i^T$.\n","start":42696,"end":42986},{"raw_tex":" Direct application of the closedness criteria Corollary 9.1.3 of Rockafellar \\cite{rockafellar1997convex} shows $\\mathcal{T}$ is closed.\n","start":43238,"end":43376}]},{"id":"S019","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisites：Finite-dimensional bipolar theorem for convex cones."],"source_only":false,"title":"有限维凸锥双极定理","source_units":[]},{"id":"S020","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisites：Rockafellar, Convex Analysis, Corollary 9.1.3；原文 L628 引用 rockafellar1997convex。"],"source_only":false,"title":"闭凸锥有限和的判据","source_units":[]},{"id":"S021","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisites：Standard spectral decomposition, PSD kernel identities, and rank-nullity；本节点提取其中谱分解与 PSD 核性质。"],"source_only":false,"title":"谱分解与半正定核性质","source_units":[]},{"id":"S022","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；原文 L623–624 的省略背景，未在原始 prerequisite 列表中单列。"],"source_only":false,"title":"半正定锥自对偶性","source_units":[]},{"id":"S023","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；显式化原文 L625 直接采用的同余像锥闭性。"],"source_only":false,"title":"半正定同余像锥闭性","source_units":[]},{"id":"S024","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisites：Standard spectral decomposition, PSD kernel identities, and rank-nullity；本节点提取 rank-nullity。"],"source_only":false,"title":"秩与核维数关系","source_units":[]},{"id":"S025","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":true,"title":"线性子空间内最大秩问题","source_units":[{"raw_tex":"To ease notation, we drop the subscript $i$ and also consider a more general question: how does one find maximum rank matrices in the set $\\perpName \\cap \\approxName$, when $\\perpName$ is an arbitrary linear subspace? \n(In the above, $\\perpName$ is the subspace $\\left\\{  U_i^T S_i U_i : C \\cdot S_i = 0, A_j \\cdot S_i = 0 \\right\\}$.)  An answer to this question  arises  from the next two lemmas.\n","start":57305,"end":57703}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","labels":["lem:rankX_i"],"declared_external":false,"source_only":false,"title":"块秩和最大推出总矩阵秩最大","source_units":[{"raw_tex":"\\begin{lem} \\label{lem:rankX_i} \nLet $\\perpName$ be a subspace of $\\mathbb{S}^d$. If $X^\\star:=\\sum^{|\\setName|}_{i=1} W_i X^\\star_i W^T_i$  maximizes   $\\sum^{|\\setName|}_{i=1} \\rank X_i$ over $\\perpName \\cap \\approxName$, then    $X^\\star$  maximizes $\\rank X$ over $\\perpName \\cap \\approxName$.\n","start":57705,"end":58003},{"raw_tex":"\\end{lem}","start":59682,"end":59691}]},{"id":"S027","semantic_type":"proof","kind":"proof","labels":["lem:rankX_i"],"declared_external":false,"source_only":false,"title":"块秩转移的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\nWe will  argue the kernel of $X^\\star$ is contained in the kernel of any  $X \\in \\perpName \\cap \\approxName$, which immediately implies $\\rank X^\\star \\ge \\rank X$. \n\nTo begin, we first argue for any $X = \\sum^{|\\setName|}_{i=1} W_i X_i W^T_i \\in  \\perpName \\cap \\approxName$ that $\\nullspace X^\\star_i \\subseteq \\nullspace X_i$ for all $i \\in \\{1,\\ldots,|\\setName|\\}$. To see this, first  note  that for any $X  \\in  \\perpName \\cap \\approxName$\nthe matrix\n\\[\n X^\\star + X =  \\sum^{|\\setName|}_{i=1} W_i ( X^\\star_i + X_i )W^T_i \n\\]\nis also in  $\\perpName \\cap \\approxName$ and  satisfies $\\rank (X^\\star_i + X_i) \\ge \\rank X^\\star_i$.  Now suppose for some $d \\in \\{1,\\ldots,|\\setName|\\}$ that  $\\nullspace X^\\star_d \\not\\subseteq \\nullspace X_d$. This implies that  $\\nullspace (X^\\star_d + X_d) =  \\nullspace X^\\star_d \\cap \\nullspace X_d  \\subset \\nullspace X^\\star_d$ which in turn implies $\\rank (X^\\star_d + X_d ) > \\rank X^\\star_d$.\nBut this contradicts our assumption that  $X^\\star$ maximizes $\\sum_i \\rank X_i$.  Hence, $\\nullspace X^\\star_i \\subseteq \\nullspace X_i$ for all $i \\in \\{1,\\ldots,|\\setName|\\}$.\n\nNow suppose an  $X \\in \\perpName \\cap \\approxName$ exists for which  $X^\\star w = 0$ but $Xw \\ne 0$ for some $w$.  Since $X w = 0$ if and only if $X_i W_i^T w = 0$ for all $i$, we must have for some $d$ that $W_d^Tw$ is in the kernel of $X^\\star_d$ but not in the kernel of $X_d$. But we have already established that $\\nullspace X^\\star_d \\subseteq \\nullspace X_d$.  Hence, $w$ cannot exist. We therefore have that $\\nullspace X^\\star \\subseteq \\nullspace X$ for any $X\\in \\perpName \\cap \\approxName$, which completes the proof. \n\\end{proof}\n","start":58003,"end":59682}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":["lem:rankX_i"],"declared_external":false,"source_only":false,"title":"把秩比较归结为核包含","source_units":[{"raw_tex":"We will  argue the kernel of $X^\\star$ is contained in the kernel of any  $X \\in \\perpName \\cap \\approxName$, which immediately implies $\\rank X^\\star \\ge \\rank X$. \n","start":58018,"end":58184}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":["lem:rankX_i"],"declared_external":false,"source_only":false,"title":"相加保持可行并不减块秩","source_units":[{"raw_tex":"To begin, we first argue for any $X = \\sum^{|\\setName|}_{i=1} W_i X_i W^T_i \\in  \\perpName \\cap \\approxName$ that $\\nullspace X^\\star_i \\subseteq \\nullspace X_i$ for all $i \\in \\{1,\\ldots,|\\setName|\\}$. To see this, first  note  that for any $X  \\in  \\perpName \\cap \\approxName$\nthe matrix\n\\[\n X^\\star + X =  \\sum^{|\\setName|}_{i=1} W_i ( X^\\star_i + X_i )W^T_i \n\\]\nis also in  $\\perpName \\cap \\approxName$ and  satisfies $\\rank (X^\\star_i + X_i) \\ge \\rank X^\\star_i$.  Now suppose for some $d \\in \\{1,\\ldots,|\\setName|\\}$ that  $\\nullspace X^\\star_d \\not\\subseteq \\nullspace X_d$. This implies that  $\\nullspace (X^\\star_d + X_d) =  \\nullspace X^\\star_d \\cap \\nullspace X_d  \\subset \\nullspace X^\\star_d$ which in turn implies $\\rank (X^\\star_d + X_d ) > \\rank X^\\star_d$.\n","start":58185,"end":58959}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":["lem:rankX_i"],"declared_external":false,"source_only":false,"title":"最优块核包含的反证","source_units":[{"raw_tex":"is also in  $\\perpName \\cap \\approxName$ and  satisfies $\\rank (X^\\star_i + X_i) \\ge \\rank X^\\star_i$.  Now suppose for some $d \\in \\{1,\\ldots,|\\setName|\\}$ that  $\\nullspace X^\\star_d \\not\\subseteq \\nullspace X_d$. This implies that  $\\nullspace (X^\\star_d + X_d) =  \\nullspace X^\\star_d \\cap \\nullspace X_d  \\subset \\nullspace X^\\star_d$ which in turn implies $\\rank (X^\\star_d + X_d ) > \\rank X^\\star_d$.\nBut this contradicts our assumption that  $X^\\star$ maximizes $\\sum_i \\rank X_i$.  Hence, $\\nullspace X^\\star_i \\subseteq \\nullspace X_i$ for all $i \\in \\{1,\\ldots,|\\setName|\\}$.\n","start":58551,"end":59138}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":["lem:rankX_i"],"declared_external":false,"source_only":false,"title":"从块核传递到总矩阵核","source_units":[{"raw_tex":"Now suppose an  $X \\in \\perpName \\cap \\approxName$ exists for which  $X^\\star w = 0$ but $Xw \\ne 0$ for some $w$.  Since $X w = 0$ if and only if $X_i W_i^T w = 0$ for all $i$, we must have for some $d$ that $W_d^Tw$ is in the kernel of $X^\\star_d$ but not in the kernel of $X_d$. But we have already established that $\\nullspace X^\\star_d \\subseteq \\nullspace X_d$.  Hence, $w$ cannot exist. We therefore have that $\\nullspace X^\\star \\subseteq \\nullspace X$ for any $X\\in \\perpName \\cap \\approxName$, which completes the proof. \n","start":59139,"end":59670}]},{"id":"S032","semantic_type":"lemma","kind":"lemma","labels":["lem:sumOfConesMaxRank","sdp:sumOfConesMaxRank"],"declared_external":false,"source_only":false,"title":"迹目标精确实现块秩和最大化","source_units":[{"raw_tex":"\\begin{lem}  \\label{lem:sumOfConesMaxRank}\nLet $\\perpName$ be a subspace of $\\mathbb{S}^d$.  A matrix $X$ maximizing $\\sum^{|\\setName|}_{i=1} \\rank X_i$  over $\\perpName \\cap \\approxName$  is given by any optimal solution $(X,X_i,T_i)$  to the following SDP:\n\\begin{align} \\label{sdp:sumOfConesMaxRank}\n\\begin{array}{lll}\n \\mbox{\\rm maximize}  &  \\sum^{|\\setName|}_{i=1} \\trace{T_i} \\  \\\\\n \\mbox{\\rm subject to } & X \\in \\perpName,\\\\\n&X = \\sum^{|\\setName|}_{i=1} W_i X_i W^T_i   & \\mbox{ \\rm  i.e. } X \\in \\approxName \\\\\n& X_i \\succeq T_i  & \\forall i\\in\\{1, \\ldots, |\\setName|\\}\\\\\n&  I \\succeq  T_i \\succeq 0 &  \\forall i\\in\\{ 1, \\ldots, |\\setName|\\}.\n\\end{array} \n\\end{align}\n\n\n","start":60020,"end":60700},{"raw_tex":"\\end{lem}","start":62269,"end":62278}]},{"id":"S033","semantic_type":"algorithm","kind":"algorithm","labels":["lem:sumOfConesMaxRank","sdp:sumOfConesMaxRank"],"declared_external":false,"source_only":false,"title":"最大块秩和半正定规划","source_units":[{"raw_tex":"\\begin{align} \\label{sdp:sumOfConesMaxRank}\n\\begin{array}{lll}\n \\mbox{\\rm maximize}  &  \\sum^{|\\setName|}_{i=1} \\trace{T_i} \\  \\\\\n \\mbox{\\rm subject to } & X \\in \\perpName,\\\\\n&X = \\sum^{|\\setName|}_{i=1} W_i X_i W^T_i   & \\mbox{ \\rm  i.e. } X \\in \\approxName \\\\\n& X_i \\succeq T_i  & \\forall i\\in\\{1, \\ldots, |\\setName|\\}\\\\\n&  I \\succeq  T_i \\succeq 0 &  \\forall i\\in\\{ 1, \\ldots, |\\setName|\\}.\n\\end{array} \n\\end{align}\n","start":60279,"end":60698}]},{"id":"S034","semantic_type":"proof","kind":"proof","labels":["ineq:maxrank","lem:sumOfConesMaxRank"],"declared_external":false,"source_only":false,"title":"迹秩等价的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $r_{\\max}$ equal the maximum of  $\\sum^{|\\setName|}_{i=1} \\rank X_i$ over the  set of feasible $X_i$.  We will\nshow at optimality $\\sum^{|\\setName|}_{i=1} \\rank X_i = r_{\\max}$. \n\nTo begin, the constraint $I \\succeq T_i \\succeq 0$ implies the eigenvalues of $T_i$ are  less than one. Hence,  $\\rank T_i  \\ge \\trace T_i $.  \nSince  $X_i \\succeq T_i$ , we also have $\\rank{X_i} \\ge \\rank{T_i}$.  Thus,  any feasible $(X_i,T_i)$ pair satisfies\n\\begin{align} \\label{ineq:maxrank}\nr_{\\max} \\ge \\sum^{|\\setName|}_{i=1} \\rank{X_i}  \\ge   \\sum^{|\\setName|}_{i=1} \\rank{T_i}  \\ge  \\sum^{|\\setName|}_{i=1} \\trace T_i.\n\\end{align}\n\nNow note for any feasible $(X,X_i)$ we can pick $\\alpha > 0$  and construct a feasible point $(\\alpha X, \\alpha X_i, \\hat{T}_i)$  that satisfies $\\sum^{|\\setName|}_{i=1} \\trace{\\hat{T_i}} =  \\sum^{|\\setName|}_{i=1}\\rank{X_i}$; if $X_i$ has  eigen-decomposition $\\sum_j \\lambda_j u_j u_j^T$ for $\\lambda_j > 0$, simply take  $\\hat{T_i} = \\sum_j u_j u_j^T$ and $\\alpha$ equal to\n\\[\n\\max \\bigcup_i \\left\\{ \\frac{1}{\\lambda} : \\lambda \\mbox{ is a  positive eigenvalue of $X_i$ } \\right\\}.\n\\]  \nHence, some feasible point $(\\hat{X},\\hat{X_i},\\hat{T_i})$ satisfies  $\\sum^{|\\setName|}_{i=1} \\trace{\\hat{T_i}} = r_{\\max}$. Therefore, the optimal $(X,X_i,T_i)$ satisfies \n\\[\n\\sum^{|\\setName|}_{i=1} \\trace T_i  \\ge  r_{\\max}.\n\\]\nCombining this inequality with \\eqref{ineq:maxrank} yields that at optimality\n\\[\n\\sum^{|\\setName|}_{i=1} \\trace T_i   = \\sum^{|\\setName|}_{i=1}\\rank{X_i}  = r_{\\max},\n\\]\nwhich completes the proof.\n\\end{proof}\n","start":60700,"end":62269}]},{"id":"S035","semantic_type":"notation","kind":"notation","labels":["lem:sumOfConesMaxRank"],"declared_external":false,"source_only":false,"title":"最大块秩和上界","source_units":[{"raw_tex":"Let $r_{\\max}$ equal the maximum of  $\\sum^{|\\setName|}_{i=1} \\rank X_i$ over the  set of feasible $X_i$.  We will\nshow at optimality $\\sum^{|\\setName|}_{i=1} \\rank X_i = r_{\\max}$. \n","start":60714,"end":60897}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":["lem:sumOfConesMaxRank"],"declared_external":false,"source_only":false,"title":"投影上界约束控制迹","source_units":[{"raw_tex":"To begin, the constraint $I \\succeq T_i \\succeq 0$ implies the eigenvalues of $T_i$ are  less than one. Hence,  $\\rank T_i  \\ge \\trace T_i $.  \n","start":60898,"end":61042}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":["lem:sumOfConesMaxRank"],"declared_external":false,"source_only":false,"title":"半正定序控制块秩","source_units":[{"raw_tex":"Since  $X_i \\succeq T_i$ , we also have $\\rank{X_i} \\ge \\rank{T_i}$.  Thus,  any feasible $(X_i,T_i)$ pair satisfies\n","start":61042,"end":61159}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":["ineq:maxrank","lem:sumOfConesMaxRank"],"declared_external":false,"source_only":false,"title":"迹目标的全局上界链","source_units":[{"raw_tex":"\\begin{align} \\label{ineq:maxrank}\nr_{\\max} \\ge \\sum^{|\\setName|}_{i=1} \\rank{X_i}  \\ge   \\sum^{|\\setName|}_{i=1} \\rank{T_i}  \\ge  \\sum^{|\\setName|}_{i=1} \\trace T_i.\n\\end{align}\n","start":61159,"end":61338}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":["lem:sumOfConesMaxRank"],"declared_external":false,"source_only":false,"title":"统一缩放并取支撑投影","source_units":[{"raw_tex":"Now note for any feasible $(X,X_i)$ we can pick $\\alpha > 0$  and construct a feasible point $(\\alpha X, \\alpha X_i, \\hat{T}_i)$  that satisfies $\\sum^{|\\setName|}_{i=1} \\trace{\\hat{T_i}} =  \\sum^{|\\setName|}_{i=1}\\rank{X_i}$; if $X_i$ has  eigen-decomposition $\\sum_j \\lambda_j u_j u_j^T$ for $\\lambda_j > 0$, simply take  $\\hat{T_i} = \\sum_j u_j u_j^T$ and $\\alpha$ equal to\n\\[\n\\max \\bigcup_i \\left\\{ \\frac{1}{\\lambda} : \\lambda \\mbox{ is a  positive eigenvalue of $X_i$ } \\right\\}.\n\\]  \n","start":61339,"end":61829}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":["lem:sumOfConesMaxRank"],"declared_external":false,"source_only":false,"title":"迹目标达到最大块秩和","source_units":[{"raw_tex":"Hence, some feasible point $(\\hat{X},\\hat{X_i},\\hat{T_i})$ satisfies  $\\sum^{|\\setName|}_{i=1} \\trace{\\hat{T_i}} = r_{\\max}$. Therefore, the optimal $(X,X_i,T_i)$ satisfies \n\\[\n\\sum^{|\\setName|}_{i=1} \\trace T_i  \\ge  r_{\\max}.\n\\]\n","start":61829,"end":62060}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":["lem:sumOfConesMaxRank"],"declared_external":false,"source_only":false,"title":"夹逼得到最优解的秩和等式","source_units":[{"raw_tex":"Combining this inequality with \\eqref{ineq:maxrank} yields that at optimality\n\\[\n\\sum^{|\\setName|}_{i=1} \\trace T_i   = \\sum^{|\\setName|}_{i=1}\\rank{X_i}  = r_{\\max},\n\\]\nwhich completes the proof.\n","start":62060,"end":62257}]},{"id":"S042","semantic_type":"corollary","kind":"corollary","labels":["cor:maxRank"],"declared_external":false,"source_only":false,"title":"半正定规划产生最大秩矩阵","source_units":[{"raw_tex":"\\begin{cor} \\label{cor:maxRank}\n A matrix $X \\in \\perpName \\cap \\approxName$ of maximum rank is given by any optimal solution\n $(X,X_i,T_i)$   to the SDP \\eqref{sdp:sumOfConesMaxRank}. \n\\end{cor}","start":62385,"end":62580}]},{"id":"S043","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"主推论的作者组合证明","source_units":[{"raw_tex":"\\noindent Combining the previous two lemmas shows how to maximize rank over $\\perpName \\cap \\approxName$:\n","start":62279,"end":62385}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"依次应用两个最大秩引理","source_units":[{"raw_tex":"\\noindent Combining the previous two lemmas shows how to maximize rank over $\\perpName \\cap \\approxName$:\n","start":62279,"end":62385}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S042","edges":[{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S005","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S011","kind":"supports"},{"source_id":"S005","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S013","kind":"supports"},{"source_id":"S004","target_id":"S013","kind":"supports"},{"source_id":"S021","target_id":"S013","kind":"supports"},{"source_id":"S022","target_id":"S013","kind":"supports"},{"source_id":"S011","target_id":"S015","kind":"supports"},{"source_id":"S010","target_id":"S015","kind":"supports"},{"source_id":"S004","target_id":"S015","kind":"supports"},{"source_id":"S006","target_id":"S015","kind":"supports"},{"source_id":"S022","target_id":"S015","kind":"supports"},{"source_id":"S001","target_id":"S016","kind":"supports"},{"source_id":"S003","target_id":"S016","kind":"supports"},{"source_id":"S011","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S019","target_id":"S016","kind":"supports"},{"source_id":"S011","target_id":"S017","kind":"supports"},{"source_id":"S021","target_id":"S017","kind":"supports"},{"source_id":"S023","target_id":"S018","kind":"supports"},{"source_id":"S009","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S020","target_id":"S018","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S016","target_id":"S014","kind":"supports"},{"source_id":"S017","target_id":"S014","kind":"supports"},{"source_id":"S018","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S012","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S012","target_id":"S026","kind":"supports"},{"source_id":"S024","target_id":"S028","kind":"supports"},{"source_id":"S025","target_id":"S029","kind":"supports"},{"source_id":"S012","target_id":"S029","kind":"supports"},{"source_id":"S021","target_id":"S029","kind":"supports"},{"source_id":"S024","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S021","target_id":"S030","kind":"supports"},{"source_id":"S024","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S012","target_id":"S031","kind":"supports"},{"source_id":"S021","target_id":"S031","kind":"supports"},{"source_id":"S024","target_id":"S031","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S027","kind":"supports"},{"source_id":"S031","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S033","kind":"supports"},{"source_id":"S009","target_id":"S033","kind":"supports"},{"source_id":"S007","target_id":"S033","kind":"supports"},{"source_id":"S006","target_id":"S033","kind":"supports"},{"source_id":"S011","target_id":"S033","kind":"supports"},{"source_id":"S012","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S033","target_id":"S036","kind":"supports"},{"source_id":"S021","target_id":"S036","kind":"supports"},{"source_id":"S033","target_id":"S037","kind":"supports"},{"source_id":"S021","target_id":"S037","kind":"supports"},{"source_id":"S024","target_id":"S037","kind":"supports"},{"source_id":"S035","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S033","target_id":"S039","kind":"supports"},{"source_id":"S025","target_id":"S039","kind":"supports"},{"source_id":"S021","target_id":"S039","kind":"supports"},{"source_id":"S035","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S038","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S034","kind":"supports"},{"source_id":"S037","target_id":"S034","kind":"supports"},{"source_id":"S038","target_id":"S034","kind":"supports"},{"source_id":"S039","target_id":"S034","kind":"supports"},{"source_id":"S040","target_id":"S034","kind":"supports"},{"source_id":"S041","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S044","kind":"supports"},{"source_id":"S026","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S033","target_id":"S042","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S042","edges":[{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S005","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S011","kind":"supports"},{"source_id":"S005","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S013","kind":"supports"},{"source_id":"S004","target_id":"S013","kind":"supports"},{"source_id":"S021","target_id":"S013","kind":"supports"},{"source_id":"S022","target_id":"S013","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S004","target_id":"S012","kind":"supports"},{"source_id":"S006","target_id":"S012","kind":"supports"},{"source_id":"S022","target_id":"S012","kind":"supports"},{"source_id":"S001","target_id":"S012","kind":"supports"},{"source_id":"S003","target_id":"S012","kind":"supports"},{"source_id":"S019","target_id":"S012","kind":"supports"},{"source_id":"S021","target_id":"S012","kind":"supports"},{"source_id":"S023","target_id":"S012","kind":"supports"},{"source_id":"S009","target_id":"S012","kind":"supports"},{"source_id":"S020","target_id":"S012","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S012","target_id":"S026","kind":"supports"},{"source_id":"S024","target_id":"S026","kind":"supports"},{"source_id":"S021","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S033","kind":"supports"},{"source_id":"S009","target_id":"S033","kind":"supports"},{"source_id":"S007","target_id":"S033","kind":"supports"},{"source_id":"S006","target_id":"S033","kind":"supports"},{"source_id":"S011","target_id":"S033","kind":"supports"},{"source_id":"S012","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S021","target_id":"S032","kind":"supports"},{"source_id":"S024","target_id":"S032","kind":"supports"},{"source_id":"S035","target_id":"S032","kind":"supports"},{"source_id":"S025","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S044","kind":"supports"},{"source_id":"S026","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S033","target_id":"S042","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"有限维实内积空间","source_units":[{"raw_tex":"Let $\\mathcal{E}$ denote a finite-dimensional vector space over $\\mathbb{R}$ with inner product $\\langle \\cdot, \\cdot \\rangle$.  For a subset $\\mathcal{S}$ of $\\mathcal{E}$, let $\\lin \\mathcal{S} \\subseteq \\mathcal{E}$ denote \nthe linear span of elements in $\\mathcal{S}$ and let $\\mathcal{S}^{\\perp} \\subseteq \\mathcal{E}$ denote the orthogonal complement of $\\lin \\mathcal{S}$.  For $y \\in \\mathcal{E}$, let $\\lin y$ denote $\\lin \\{ y \\}$ and let $y^{\\perp}$  denote $\\{ y \\}^{\\perp}$.\n","start":14718,"end":15206}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"线性张成与正交补","source_units":[{"raw_tex":"Let $\\mathcal{E}$ denote a finite-dimensional vector space over $\\mathbb{R}$ with inner product $\\langle \\cdot, \\cdot \\rangle$.  For a subset $\\mathcal{S}$ of $\\mathcal{E}$, let $\\lin \\mathcal{S} \\subseteq \\mathcal{E}$ denote \nthe linear span of elements in $\\mathcal{S}$ and let $\\mathcal{S}^{\\perp} \\subseteq \\mathcal{E}$ denote the orthogonal complement of $\\lin \\mathcal{S}$.  For $y \\in \\mathcal{E}$, let $\\lin y$ denote $\\lin \\{ y \\}$ and let $y^{\\perp}$  denote $\\{ y \\}^{\\perp}$.\n","start":14718,"end":15206}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"凸锥定义","source_units":[{"raw_tex":"A  convex cone $\\coneName$ is a  convex subset of $\\mathcal{E}$ (not necessarily full dimensional) that satisfies\n\\[\n x  \\in \\coneName \\Rightarrow  \\alpha x \\in \\coneName \\;\\;\\; \\forall \\alpha \\ge 0.\n\\]\n","start":15206,"end":15409}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"对偶锥定义","source_units":[{"raw_tex":"The dual cone of $\\coneName$, denoted  $\\coneName^*$, is the set of linear functionals non-negative on $\\coneName$:\n\\[\n  \\coneName^* := \\left\\{ y : \\langle y,x\\rangle \\ge 0 \\;\\;\\; \\forall x \\in \\coneName\\right\\}.\n\\]\n","start":15409,"end":15625}]},{"id":"S005","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"对称矩阵与半正定锥","source_units":[{"raw_tex":"For discussions specific to semidefinite programming, we let $\\mathbb{S}^n$ denote the vector space of $n\\times n$ symmetric matrices and $\\mathbb{S}^n_{+} \\subseteq \\mathbb{S}^n$\ndenote the convex cone of matrices that are positive semidefinite. We will use capital letters to denote elements of\n","start":16411,"end":16708}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"迹内积","source_units":[{"raw_tex":"$\\mathbb{S}^n$ to emphasize that they are matrices.  For $A,B \\in \\mathbb{S}^n$, we let $A \\cdot B$  denote the trace inner product $\\trace AB$. \n","start":16708,"end":16854}]},{"id":"S007","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"半正定序记号","source_units":[{"raw_tex":"Finally, we let  $A \\succeq 0$ (resp. $A \\succ 0$)  denote the condition that $A$ is positive semidefinite (resp.  positive definite).\n","start":16854,"end":16989}]},{"id":"S008","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"参数化近似锥族","source_units":[{"raw_tex":"\\subsection{Approximations of $\\mathbb{S}^d_{+}$} \\label{sec:approx}\nIn this section, we explore an  outer approximation $\\outerName$ of  $\\mathbb{S}^d_{+}$  parametrized by a set $\\setName$\nof $d \\times k$ rectangular matrices. The parametrization is chosen such that the dual cone $\\approxName$ equals the \\emph{Minkowski sum} of faces $W_i \\mathbb{S}^{k}_{+} W_i^T$ of $\\mathbb{S}^d_{+}$ \nfor $W_i \\in \\setName$.  It is defined below:\n","start":41239,"end":41677}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"有限矩阵族参数","source_units":[{"raw_tex":"For a set $\\setName := \\left\\{ W_1, W_2,\\ldots, W_{|\\setName|} \\right\\}$ of $d \\times k$ matrices, let $\\outerName$ denote the following convex cone:\n","start":41689,"end":41839}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"压缩约束外近似","source_units":[{"raw_tex":"For a set $\\setName := \\left\\{ W_1, W_2,\\ldots, W_{|\\setName|} \\right\\}$ of $d \\times k$ matrices, let $\\outerName$ denote the following convex cone:\n\\begin{align*}\n\\outerName := \\left\\{  X \\in  \\mathbb{S}^d : W_i^T X W_i \\in \\mathbb{S}^k_{+} \\;\\;\\;i = 1, \\ldots, |\\setName| \\right\\}.\n\\end{align*}\n","start":41689,"end":41987}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":["def:coneProdInner"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"半正定块和集合","source_units":[{"raw_tex":"\\begin{align} \\label{def:coneProdInner}\n \\approxName = \\left\\{ \\sum^{|\\setName|}_{i=1}  W_i X_i W_i^T : X_i \\in \\mathbb{S}_{+}^k  \\right\\},\n\\end{align}\n","start":42025,"end":42177}]},{"id":"S012","semantic_type":"lemma","kind":"lemma","labels":["def:coneProdInner"],"member_ids":["S012","S014","S015","S016","S017","S018"],"declared_external":false,"source_only":false,"title":"近似锥的对偶表示","source_units":[{"raw_tex":"\\begin{lem}\nFor a set $\\setName := \\left\\{ W_1, W_2,\\ldots, W_{|\\setName|} \\right\\}$ of $d \\times k$ matrices, let $\\outerName$ denote the following convex cone:\n\\begin{align*}\n\\outerName := \\left\\{  X \\in  \\mathbb{S}^d : W_i^T X W_i \\in \\mathbb{S}^k_{+} \\;\\;\\;i = 1, \\ldots, |\\setName| \\right\\}.\n\\end{align*}\nThe dual cone $\\approxName$ satisfies\n\\begin{align} \\label{def:coneProdInner}\n \\approxName = \\left\\{ \\sum^{|\\setName|}_{i=1}  W_i X_i W_i^T : X_i \\in \\mathbb{S}_{+}^k  \\right\\},\n\\end{align}\nand the following inclusions hold:\n\\[\n\\approxName \\subseteq \\mathbb{S}^d_{+} \\subseteq \\outerName.\n\\]\n","start":41677,"end":42279},{"raw_tex":"\\begin{proof}\nThe inclusions are obvious from the definitions of $\\approxName$ and $\\outerName$ (as is the fact that  $\\outerName$ is a convex cone).  It remains to show correctness of  \\eqref{def:coneProdInner}. To show this,\nlet $\\mathcal{T}$ denote the set on the right-hand side of \\eqref{def:coneProdInner}.  It is easy to check that  $\\mathcal{T}^*= \\outerName$, which implies $\\mathcal{T}^{**} = \\approxName$.\nSince $\\mathcal{T}$ is a convex cone (as is easily checked), $\\mathcal{T}^{**}$ equals the closure of $\\mathcal{T}$. The result therefore follows by showing $\\mathcal{T}$ is closed. To see this, note that $\\mathcal{T}$ equals the Minkowski sum of closed cones $W_i \\mathbb{S}^k_{+} W_i^T$.\nFor  matrices $Z_i \\in W_i \\mathbb{S}^k_{+} W_i^T$, we have that $\\sum^{|\\setName|}_{i=1} Z_i=0$\nonly if $Z_i= 0$ for each $i$.  This shows that $\\sum^{|\\setName|}_{i =1}Z_i = 0$ only if $Z_i$ is in the lineality space of $W_i \\mathbb{S}^k_{+} W_i^T$.\n Direct application of the closedness criteria Corollary 9.1.3 of Rockafellar \\cite{rockafellar1997convex} shows $\\mathcal{T}$ is closed.\n\\end{proof}\n","start":42279,"end":43388},{"raw_tex":"let $\\mathcal{T}$ denote the set on the right-hand side of \\eqref{def:coneProdInner}.  It is easy to check that  $\\mathcal{T}^*= \\outerName$, which implies $\\mathcal{T}^{**} = \\approxName$.\n","start":42506,"end":42696},{"raw_tex":"Since $\\mathcal{T}$ is a convex cone (as is easily checked), $\\mathcal{T}^{**}$ equals the closure of $\\mathcal{T}$. The result therefore follows by showing $\\mathcal{T}$ is closed. To see this, note that $\\mathcal{T}$ equals the Minkowski sum of closed cones $W_i \\mathbb{S}^k_{+} W_i^T$.\n","start":42696,"end":42986},{"raw_tex":"For  matrices $Z_i \\in W_i \\mathbb{S}^k_{+} W_i^T$, we have that $\\sum^{|\\setName|}_{i=1} Z_i=0$\nonly if $Z_i= 0$ for each $i$.  This shows that $\\sum^{|\\setName|}_{i =1}Z_i = 0$ only if $Z_i$ is in the lineality space of $W_i \\mathbb{S}^k_{+} W_i^T$.\n","start":42986,"end":43238},{"raw_tex":" Direct application of the closedness criteria Corollary 9.1.3 of Rockafellar \\cite{rockafellar1997convex} shows $\\mathcal{T}$ is closed.\n","start":43238,"end":43376},{"raw_tex":"\n\\end{lem}","start":43388,"end":43398}]},{"id":"S013","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"内外近似包含关系","source_units":[{"raw_tex":"and the following inclusions hold:\n\\[\n\\approxName \\subseteq \\mathbb{S}^d_{+} \\subseteq \\outerName.\n\\]\n","start":42177,"end":42279},{"raw_tex":"The inclusions are obvious from the definitions of $\\approxName$ and $\\outerName$ (as is the fact that  $\\outerName$ is a convex cone).  It remains to show correctness of  \\eqref{def:coneProdInner}. To show this,\n","start":42293,"end":42506}]},{"id":"S019","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S019"],"declared_external":true,"declaration_text":["原始 prerequisites：Finite-dimensional bipolar theorem for convex cones."],"source_only":false,"title":"有限维凸锥双极定理","source_units":[]},{"id":"S020","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S020"],"declared_external":true,"declaration_text":["原始 prerequisites：Rockafellar, Convex Analysis, Corollary 9.1.3；原文 L628 引用 rockafellar1997convex。"],"source_only":false,"title":"闭凸锥有限和的判据","source_units":[]},{"id":"S021","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S021"],"declared_external":true,"declaration_text":["原始 prerequisites：Standard spectral decomposition, PSD kernel identities, and rank-nullity；本节点提取其中谱分解与 PSD 核性质。"],"source_only":false,"title":"谱分解与半正定核性质","source_units":[]},{"id":"S022","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S022"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；原文 L623–624 的省略背景，未在原始 prerequisite 列表中单列。"],"source_only":false,"title":"半正定锥自对偶性","source_units":[]},{"id":"S023","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S023"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；显式化原文 L625 直接采用的同余像锥闭性。"],"source_only":false,"title":"半正定同余像锥闭性","source_units":[]},{"id":"S024","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S024"],"declared_external":true,"declaration_text":["原始 prerequisites：Standard spectral decomposition, PSD kernel identities, and rank-nullity；本节点提取 rank-nullity。"],"source_only":false,"title":"秩与核维数关系","source_units":[]},{"id":"S025","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S025"],"declared_external":false,"source_only":true,"title":"线性子空间内最大秩问题","source_units":[{"raw_tex":"To ease notation, we drop the subscript $i$ and also consider a more general question: how does one find maximum rank matrices in the set $\\perpName \\cap \\approxName$, when $\\perpName$ is an arbitrary linear subspace? \n(In the above, $\\perpName$ is the subspace $\\left\\{  U_i^T S_i U_i : C \\cdot S_i = 0, A_j \\cdot S_i = 0 \\right\\}$.)  An answer to this question  arises  from the next two lemmas.\n","start":57305,"end":57703}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","labels":["lem:rankX_i"],"member_ids":["S026","S027","S028","S029","S030","S031"],"declared_external":false,"source_only":false,"title":"块秩和最大推出总矩阵秩最大","source_units":[{"raw_tex":"\\begin{lem} \\label{lem:rankX_i} \nLet $\\perpName$ be a subspace of $\\mathbb{S}^d$. If $X^\\star:=\\sum^{|\\setName|}_{i=1} W_i X^\\star_i W^T_i$  maximizes   $\\sum^{|\\setName|}_{i=1} \\rank X_i$ over $\\perpName \\cap \\approxName$, then    $X^\\star$  maximizes $\\rank X$ over $\\perpName \\cap \\approxName$.\n","start":57705,"end":58003},{"raw_tex":"\\begin{proof}\n\nWe will  argue the kernel of $X^\\star$ is contained in the kernel of any  $X \\in \\perpName \\cap \\approxName$, which immediately implies $\\rank X^\\star \\ge \\rank X$. \n\nTo begin, we first argue for any $X = \\sum^{|\\setName|}_{i=1} W_i X_i W^T_i \\in  \\perpName \\cap \\approxName$ that $\\nullspace X^\\star_i \\subseteq \\nullspace X_i$ for all $i \\in \\{1,\\ldots,|\\setName|\\}$. To see this, first  note  that for any $X  \\in  \\perpName \\cap \\approxName$\nthe matrix\n\\[\n X^\\star + X =  \\sum^{|\\setName|}_{i=1} W_i ( X^\\star_i + X_i )W^T_i \n\\]\nis also in  $\\perpName \\cap \\approxName$ and  satisfies $\\rank (X^\\star_i + X_i) \\ge \\rank X^\\star_i$.  Now suppose for some $d \\in \\{1,\\ldots,|\\setName|\\}$ that  $\\nullspace X^\\star_d \\not\\subseteq \\nullspace X_d$. This implies that  $\\nullspace (X^\\star_d + X_d) =  \\nullspace X^\\star_d \\cap \\nullspace X_d  \\subset \\nullspace X^\\star_d$ which in turn implies $\\rank (X^\\star_d + X_d ) > \\rank X^\\star_d$.\nBut this contradicts our assumption that  $X^\\star$ maximizes $\\sum_i \\rank X_i$.  Hence, $\\nullspace X^\\star_i \\subseteq \\nullspace X_i$ for all $i \\in \\{1,\\ldots,|\\setName|\\}$.\n\nNow suppose an  $X \\in \\perpName \\cap \\approxName$ exists for which  $X^\\star w = 0$ but $Xw \\ne 0$ for some $w$.  Since $X w = 0$ if and only if $X_i W_i^T w = 0$ for all $i$, we must have for some $d$ that $W_d^Tw$ is in the kernel of $X^\\star_d$ but not in the kernel of $X_d$. But we have already established that $\\nullspace X^\\star_d \\subseteq \\nullspace X_d$.  Hence, $w$ cannot exist. We therefore have that $\\nullspace X^\\star \\subseteq \\nullspace X$ for any $X\\in \\perpName \\cap \\approxName$, which completes the proof. \n\\end{proof}\n","start":58003,"end":59682},{"raw_tex":"We will  argue the kernel of $X^\\star$ is contained in the kernel of any  $X \\in \\perpName \\cap \\approxName$, which immediately implies $\\rank X^\\star \\ge \\rank X$. \n","start":58018,"end":58184},{"raw_tex":"To begin, we first argue for any $X = \\sum^{|\\setName|}_{i=1} W_i X_i W^T_i \\in  \\perpName \\cap \\approxName$ that $\\nullspace X^\\star_i \\subseteq \\nullspace X_i$ for all $i \\in \\{1,\\ldots,|\\setName|\\}$. To see this, first  note  that for any $X  \\in  \\perpName \\cap \\approxName$\nthe matrix\n\\[\n X^\\star + X =  \\sum^{|\\setName|}_{i=1} W_i ( X^\\star_i + X_i )W^T_i \n\\]\nis also in  $\\perpName \\cap \\approxName$ and  satisfies $\\rank (X^\\star_i + X_i) \\ge \\rank X^\\star_i$.  Now suppose for some $d \\in \\{1,\\ldots,|\\setName|\\}$ that  $\\nullspace X^\\star_d \\not\\subseteq \\nullspace X_d$. This implies that  $\\nullspace (X^\\star_d + X_d) =  \\nullspace X^\\star_d \\cap \\nullspace X_d  \\subset \\nullspace X^\\star_d$ which in turn implies $\\rank (X^\\star_d + X_d ) > \\rank X^\\star_d$.\n","start":58185,"end":58959},{"raw_tex":"is also in  $\\perpName \\cap \\approxName$ and  satisfies $\\rank (X^\\star_i + X_i) \\ge \\rank X^\\star_i$.  Now suppose for some $d \\in \\{1,\\ldots,|\\setName|\\}$ that  $\\nullspace X^\\star_d \\not\\subseteq \\nullspace X_d$. This implies that  $\\nullspace (X^\\star_d + X_d) =  \\nullspace X^\\star_d \\cap \\nullspace X_d  \\subset \\nullspace X^\\star_d$ which in turn implies $\\rank (X^\\star_d + X_d ) > \\rank X^\\star_d$.\nBut this contradicts our assumption that  $X^\\star$ maximizes $\\sum_i \\rank X_i$.  Hence, $\\nullspace X^\\star_i \\subseteq \\nullspace X_i$ for all $i \\in \\{1,\\ldots,|\\setName|\\}$.\n","start":58551,"end":59138},{"raw_tex":"Now suppose an  $X \\in \\perpName \\cap \\approxName$ exists for which  $X^\\star w = 0$ but $Xw \\ne 0$ for some $w$.  Since $X w = 0$ if and only if $X_i W_i^T w = 0$ for all $i$, we must have for some $d$ that $W_d^Tw$ is in the kernel of $X^\\star_d$ but not in the kernel of $X_d$. But we have already established that $\\nullspace X^\\star_d \\subseteq \\nullspace X_d$.  Hence, $w$ cannot exist. We therefore have that $\\nullspace X^\\star \\subseteq \\nullspace X$ for any $X\\in \\perpName \\cap \\approxName$, which completes the proof. \n","start":59139,"end":59670},{"raw_tex":"\\end{lem}","start":59682,"end":59691}]},{"id":"S032","semantic_type":"lemma","kind":"lemma","labels":["lem:sumOfConesMaxRank","sdp:sumOfConesMaxRank"],"member_ids":["S032","S034","S036","S037","S038","S039","S040","S041"],"declared_external":false,"source_only":false,"title":"迹目标精确实现块秩和最大化","source_units":[{"raw_tex":"\\begin{lem}  \\label{lem:sumOfConesMaxRank}\nLet $\\perpName$ be a subspace of $\\mathbb{S}^d$.  A matrix $X$ maximizing $\\sum^{|\\setName|}_{i=1} \\rank X_i$  over $\\perpName \\cap \\approxName$  is given by any optimal solution $(X,X_i,T_i)$  to the following SDP:\n\\begin{align} \\label{sdp:sumOfConesMaxRank}\n\\begin{array}{lll}\n \\mbox{\\rm maximize}  &  \\sum^{|\\setName|}_{i=1} \\trace{T_i} \\  \\\\\n \\mbox{\\rm subject to } & X \\in \\perpName,\\\\\n&X = \\sum^{|\\setName|}_{i=1} W_i X_i W^T_i   & \\mbox{ \\rm  i.e. } X \\in \\approxName \\\\\n& X_i \\succeq T_i  & \\forall i\\in\\{1, \\ldots, |\\setName|\\}\\\\\n&  I \\succeq  T_i \\succeq 0 &  \\forall i\\in\\{ 1, \\ldots, |\\setName|\\}.\n\\end{array} \n\\end{align}\n\n\n","start":60020,"end":60700},{"raw_tex":"\\begin{proof}\nLet $r_{\\max}$ equal the maximum of  $\\sum^{|\\setName|}_{i=1} \\rank X_i$ over the  set of feasible $X_i$.  We will\nshow at optimality $\\sum^{|\\setName|}_{i=1} \\rank X_i = r_{\\max}$. \n\nTo begin, the constraint $I \\succeq T_i \\succeq 0$ implies the eigenvalues of $T_i$ are  less than one. Hence,  $\\rank T_i  \\ge \\trace T_i $.  \nSince  $X_i \\succeq T_i$ , we also have $\\rank{X_i} \\ge \\rank{T_i}$.  Thus,  any feasible $(X_i,T_i)$ pair satisfies\n\\begin{align} \\label{ineq:maxrank}\nr_{\\max} \\ge \\sum^{|\\setName|}_{i=1} \\rank{X_i}  \\ge   \\sum^{|\\setName|}_{i=1} \\rank{T_i}  \\ge  \\sum^{|\\setName|}_{i=1} \\trace T_i.\n\\end{align}\n\nNow note for any feasible $(X,X_i)$ we can pick $\\alpha > 0$  and construct a feasible point $(\\alpha X, \\alpha X_i, \\hat{T}_i)$  that satisfies $\\sum^{|\\setName|}_{i=1} \\trace{\\hat{T_i}} =  \\sum^{|\\setName|}_{i=1}\\rank{X_i}$; if $X_i$ has  eigen-decomposition $\\sum_j \\lambda_j u_j u_j^T$ for $\\lambda_j > 0$, simply take  $\\hat{T_i} = \\sum_j u_j u_j^T$ and $\\alpha$ equal to\n\\[\n\\max \\bigcup_i \\left\\{ \\frac{1}{\\lambda} : \\lambda \\mbox{ is a  positive eigenvalue of $X_i$ } \\right\\}.\n\\]  \nHence, some feasible point $(\\hat{X},\\hat{X_i},\\hat{T_i})$ satisfies  $\\sum^{|\\setName|}_{i=1} \\trace{\\hat{T_i}} = r_{\\max}$. Therefore, the optimal $(X,X_i,T_i)$ satisfies \n\\[\n\\sum^{|\\setName|}_{i=1} \\trace T_i  \\ge  r_{\\max}.\n\\]\nCombining this inequality with \\eqref{ineq:maxrank} yields that at optimality\n\\[\n\\sum^{|\\setName|}_{i=1} \\trace T_i   = \\sum^{|\\setName|}_{i=1}\\rank{X_i}  = r_{\\max},\n\\]\nwhich completes the proof.\n\\end{proof}\n","start":60700,"end":62269},{"raw_tex":"To begin, the constraint $I \\succeq T_i \\succeq 0$ implies the eigenvalues of $T_i$ are  less than one. Hence,  $\\rank T_i  \\ge \\trace T_i $.  \n","start":60898,"end":61042},{"raw_tex":"Since  $X_i \\succeq T_i$ , we also have $\\rank{X_i} \\ge \\rank{T_i}$.  Thus,  any feasible $(X_i,T_i)$ pair satisfies\n","start":61042,"end":61159},{"raw_tex":"\\begin{align} \\label{ineq:maxrank}\nr_{\\max} \\ge \\sum^{|\\setName|}_{i=1} \\rank{X_i}  \\ge   \\sum^{|\\setName|}_{i=1} \\rank{T_i}  \\ge  \\sum^{|\\setName|}_{i=1} \\trace T_i.\n\\end{align}\n","start":61159,"end":61338},{"raw_tex":"Now note for any feasible $(X,X_i)$ we can pick $\\alpha > 0$  and construct a feasible point $(\\alpha X, \\alpha X_i, \\hat{T}_i)$  that satisfies $\\sum^{|\\setName|}_{i=1} \\trace{\\hat{T_i}} =  \\sum^{|\\setName|}_{i=1}\\rank{X_i}$; if $X_i$ has  eigen-decomposition $\\sum_j \\lambda_j u_j u_j^T$ for $\\lambda_j > 0$, simply take  $\\hat{T_i} = \\sum_j u_j u_j^T$ and $\\alpha$ equal to\n\\[\n\\max \\bigcup_i \\left\\{ \\frac{1}{\\lambda} : \\lambda \\mbox{ is a  positive eigenvalue of $X_i$ } \\right\\}.\n\\]  \n","start":61339,"end":61829},{"raw_tex":"Hence, some feasible point $(\\hat{X},\\hat{X_i},\\hat{T_i})$ satisfies  $\\sum^{|\\setName|}_{i=1} \\trace{\\hat{T_i}} = r_{\\max}$. Therefore, the optimal $(X,X_i,T_i)$ satisfies \n\\[\n\\sum^{|\\setName|}_{i=1} \\trace T_i  \\ge  r_{\\max}.\n\\]\n","start":61829,"end":62060},{"raw_tex":"Combining this inequality with \\eqref{ineq:maxrank} yields that at optimality\n\\[\n\\sum^{|\\setName|}_{i=1} \\trace T_i   = \\sum^{|\\setName|}_{i=1}\\rank{X_i}  = r_{\\max},\n\\]\nwhich completes the proof.\n","start":62060,"end":62257},{"raw_tex":"\\end{lem}","start":62269,"end":62278}]},{"id":"S033","semantic_type":"algorithm","kind":"algorithm","labels":["lem:sumOfConesMaxRank","sdp:sumOfConesMaxRank"],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"最大块秩和半正定规划","source_units":[{"raw_tex":"\\begin{align} \\label{sdp:sumOfConesMaxRank}\n\\begin{array}{lll}\n \\mbox{\\rm maximize}  &  \\sum^{|\\setName|}_{i=1} \\trace{T_i} \\  \\\\\n \\mbox{\\rm subject to } & X \\in \\perpName,\\\\\n&X = \\sum^{|\\setName|}_{i=1} W_i X_i W^T_i   & \\mbox{ \\rm  i.e. } X \\in \\approxName \\\\\n& X_i \\succeq T_i  & \\forall i\\in\\{1, \\ldots, |\\setName|\\}\\\\\n&  I \\succeq  T_i \\succeq 0 &  \\forall i\\in\\{ 1, \\ldots, |\\setName|\\}.\n\\end{array} \n\\end{align}\n","start":60279,"end":60698}]},{"id":"S035","semantic_type":"notation","kind":"notation","labels":["lem:sumOfConesMaxRank"],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"最大块秩和上界","source_units":[{"raw_tex":"Let $r_{\\max}$ equal the maximum of  $\\sum^{|\\setName|}_{i=1} \\rank X_i$ over the  set of feasible $X_i$.  We will\nshow at optimality $\\sum^{|\\setName|}_{i=1} \\rank X_i = r_{\\max}$. \n","start":60714,"end":60897}]},{"id":"S042","semantic_type":"corollary","kind":"corollary","labels":["cor:maxRank"],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"半正定规划产生最大秩矩阵","source_units":[{"raw_tex":"\\begin{cor} \\label{cor:maxRank}\n A matrix $X \\in \\perpName \\cap \\approxName$ of maximum rank is given by any optimal solution\n $(X,X_i,T_i)$   to the SDP \\eqref{sdp:sumOfConesMaxRank}. \n\\end{cor}","start":62385,"end":62580}]},{"id":"S043","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S043"],"declared_external":false,"source_only":true,"title":"主推论的作者组合证明","source_units":[{"raw_tex":"\\noindent Combining the previous two lemmas shows how to maximize rank over $\\perpName \\cap \\approxName$:\n","start":62279,"end":62385}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S044"],"declared_external":false,"source_only":true,"title":"依次应用两个最大秩引理","source_units":[{"raw_tex":"\\noindent Combining the previous two lemmas shows how to maximize rank over $\\perpName \\cap \\approxName$:\n","start":62279,"end":62385}]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S012","S015":"S012","S016":"S012","S017":"S012","S018":"S012","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S026","S028":"S026","S029":"S026","S030":"S026","S031":"S026","S032":"S032","S033":"S033","S034":"S032","S035":"S035","S036":"S032","S037":"S032","S038":"S032","S039":"S032","S040":"S032","S041":"S032","S042":"S042","S043":"S043","S044":"S044"}}}}
{"record_id":"1810.12197v3","paper":{"title":"Semidefinite programming hierarchies for constrained bilinear optimization","authors":["Berta, Mario","Borderi, Francesco","Fawzi, Omar","Scholz, Volkher"],"arxiv_id":"1810.12197","version":"v3","journal":"Mathematical Programming","abstract_url":"https://arxiv.org/abs/1810.12197v3","pdf_url":"https://arxiv.org/pdf/1810.12197v3","doi":"10.1007/s10107-021-01650-1"},"context":{"raw_tex":"To discuss our approach, we first rewrite~\\eqref{eq:form} by defining $G_{AB}= (\\Pi_{A \\to D}^{\\dagger} \\otimes \\Pi_{B \\to E}^{\\dagger})(H)$, where $\\Pi^{\\dagger}$ denotes the adjoint map of $\\Pi$ in the Hilbert-Schmidt inner product. This leads to the form\n\\begin{align}\\label{original_program}\nQ=\\max &\\quad \\mathrm{Tr}\\big[G_{AB} (W_{A} \\otimes W_{B})\\big]\\\\\ns.t. &\\quad W_{A} \\succeq 0,\\quad W_{B} \\succeq 0,\\quad\\mathrm{Tr}[W_A] = \\mathrm{Tr}[W_B] = 1\\\\\n&\\quad\\Lambda_{A\\to C_A}\\left(W_{A}\\right)=X_{C_A},\\quad\\Gamma_{B\\to C_B}\\left(W_{B}\\right)=Y_{C_B}, \n\\end{align}\nwhere $G_{AB}\\in\\mathcal{S}_A\\otimes\\mathcal{S}_B$, $\\Lambda_{A\\to C_A}:\\mathcal{S}_A \\to \\mathcal{S}_{C_A}$ and $\\Gamma_{B\\to C_B}:\\mathcal{S}_B \\to \\mathcal{S}_{C_B}$ denote linear maps, and $X_{C_A}\\in\\mathcal{S}_{C_A}$ and $Y_{C_B}\\in\\mathcal{S}_{C_B}$ are the matrices defining $\\mathcal{A}_{A}$ and $\\mathcal{A}_B$ as the affine subspaces associated with the kernels of the linear maps $\\Lambda_{A\\to C_A}$ and $\\Gamma_{B\\to C_B}$, respectively.\nNow, by the linearity of the objective function we can equivalently optimise over the convex hull of feasible points\n\\begin{align}\\label{eq:transformed_program}\nQ=\\max &\\quad \\mathrm{Tr}\\left[G_{AB}\\left(\\sum_{i\\in I}p_iW_A^i \\otimes W_B^i\\right)\\right]\\\\\ns.t. &\\quad p_i\\geq0,\\quad W_A^i \\succeq 0,\\quad W_B^i \\succeq 0,\\quad\\mathrm{Tr}\\left[W_A^i\\right] = \\mathrm{Tr}\\left[W_B^i\\right] = 1\\quad\\forall i\\in I,\\quad\\sum_{i\\in I}p_i=1\\\\\n&\\quad\\Lambda_{A\\to C_A}\\left(W_A^i\\right)=X_{C_A},\\quad\\Gamma_{B\\to C_B}\\left(W_B^i\\right)=Y_{C_B}\\quad\\forall i\\in I.\n\\end{align}\nThat is, in the language of quantum information theory we are maximizing over a subset of the so-called {\\it separable quantum states}\\,---\\,where the latter is defined on $A\\otimes B$ as\n\\begin{align}\n\\text{Sep}(A:B)=\\left\\{\\sum_{i\\in I}p_iW_A^i \\otimes W_B^i: W_A^i \\succeq 0, W_B^i \\succeq 0, \\mathrm{Tr}\\left[W_A^i\\right]=\\mathrm{Tr}\\left[W_B^i\\right] = 1,p_i\\geq0,\\sum_{i\\in I}p_i=1\\right\\}.\n\\end{align}\nRecall that matrices $W_A\\in\\mathcal{S}^+_A$ are called quantum states on system $A$\\,---\\,and similarly for bipartite states on $A\\otimes B$.\n\n\nIn the following, we introduce some notation that is standard in quantum information theory. A $d_A$-dimensional quantum system (or in short {\\it system}) is given by an inner product space $\\mathbb{C}^{d_A}$ and denoted by $A$. {\\it Quantum states} (or in short {\\it states}) on $A$ are matrices\\footnote{Here and henceforth we use the symbol $:=$ as {\\it equal by definition}.}\n\\begin{align}\n\\text{$W_A\\in\\mathcal{S}_A:=\\mathbb{C}^{d_A\\times d_A}$ with $\\mathrm{Tr}[W_A]=1$ and $W_A\\succeq0$,}\n\\end{align}\nwhere $\\succeq$ denotes the operator (Loewner) order. Quantum states of rank one are called {\\it pure} and can be written as $W_A=\\ket{\\psi}\\bra{\\psi}_A$, where $\\ket{\\psi}_A\\in\\mathbb{C}^{d_A}$ and $\\ket{\\psi}\\bra{\\psi}_A\\in\\mathcal{S}_A$ denotes the rank-one projector on the vector $\\ket{\\psi}_A$.\n\nA {\\it bipartite system} $AB:=A\\otimes B$ is given by an inner product space $\\mathbb{C}^{d_A}\\otimes\\mathbb{C}^{d_B}$, where $\\otimes$ denotes the Kronecker tensor product. Correspondingly, states on $AB$ are matrices $W_{AB}\\in\\mathcal{S}_A\\otimes\\mathcal{S}_B$ with $\\mathrm{Tr}[W_{AB}]=1$ and $W_{AB}\\succeq0$. {\\it Separable states} are states on $AB$ that are in the convex hull of product states $W_A\\otimes W_B$, with $W_A$ and $W_B$ states on $A$ and $B$, respectively. The {\\it maximally entangled state} $\\Phi_{AB}:=\\ket{\\Phi}\\bra{\\Phi}_{AB}$ on $AB$ for $d:=d_A=d_B$ is not separable and defined via\n\\begin{align}\n\\text{$\\ket{\\Phi}_{AB}:=\\frac{1}{\\sqrt{d}}\\sum_{x=1}^d\\ket{x}_A\\otimes\\ket{x}_B$ for some orthonormal basis $\\{\\ket{x}\\}_{x=1}^d$ of $\\mathbb{C}^d$.}\n\\end{align}\nThe {\\it swap operator} $F_{AB}$ on $A\\otimes B$ exchanges the two quantum systems, i.e., $F_{AB}(W_A\\otimes W_B)=W_B\\otimes W_A$ for every state $W_A$ and $W_B$ on $A$ and $B$, respectively. {\\it Classical-quantum states} are bipartite states that can be written in the form\n\\begin{align}\nW_{XB}=\\sum_{x=1}^{d_X}p_x\\ket{x}\\bra{x}_X\\otimes W_B^x\n\\end{align}\nfor a probability distribution $\\{p_x\\}_{x=1}^{d_A}$, an orthonormal basis $\\{\\ket{x}_X\\}_{x=1}^{d_X}$ of $\\mathbb{C}^{d_X}$, and states $W_B^x$ on $B$ for $x=1,\\dots,d_X$. We refer to $X$ as the classical part of the bipartite classical-quantum system $XB$.\n\n{\\it Quantum channels} (or in short {\\it channels}) are linear maps $\\mathcal{N}_{A\\to B}:\\mathcal{S}_A\\to\\mathcal{S}_B$ that are trace preserving and completely positive\\footnote{A linear map $\\mathcal{N}_{A\\to B}:\\mathcal{S}_A\\to\\mathcal{S}_B$ is said to be completely positive if $\\mathcal{N}_{A\\to B}\\otimes \\mathcal{I}_C$ is a positive map for every quantum system $C$, where $\\mathcal{I}_C$ denotes the identity map on $\\mathcal{S}_C$} (cp). In particular, they map states from the input system $A$ to states on the output system $B$. We often abbreviate bipartite channels $\\mathcal{I}_A\\otimes\\mathcal{N}_B$ that act trivially on the $A$-system as $\\mathcal{N}_B$, where $\\mathcal{I}_A$ denotes the identity channel on $\\mathcal{S}_A$. The partial trace $\\mathrm{Tr}_B[\\cdot]$ is a channel from $AB$ to $A$ defined via\n\\begin{align}\n\\mathrm{Tr}_B[\\cdot]:=\\sum_{x=1}^{d_B}\\big(1_A\\otimes\\bra{x}_B\\big)(\\cdot)\\big(1_A\\otimes\\ket{x}_B\\big),\n\\end{align}\nwhere $1_A$ denotes the identity matrix on $A$, and $\\{\\ket{x}\\}_{x=1}^{d_B}$ an orthonormal basis of $\\mathbb{C}^{d_B}$. For bipartite states $W_{AB}$, we write $W_A=\\mathrm{Tr}_B[W_{AB}]$ for the reduced state on $A$. {\\it Quantum measurements} (or in short {\\it measurements}) are a special case of channels that can be written in the form\n\\begin{align}\n\\mathcal{M}_{A\\to I}(\\cdot):=\\sum_{i\\in I}\\mathrm{Tr}\\left[M_A^i(\\cdot)\\right]\\ket{i}\\bra{i}_I\n\\end{align}\nwith an orthonormal basis $\\{\\ket{i}_I\\}_{i\\in I}$ and $M_A^i\\succeq0$ $\\forall i\\in I$ with $\\sum_{i\\in I}M_A^i=1_A$.\n\n\nThe distance between states is quantified by the metric induced from the Schatten one-norm $\\|X\\|_1:=\\mathrm{Tr}\\left[|X|\\right]$. The distance between channels is quantified by the metric induced from the Diamond norm\n\\begin{align}\n\\|\\mathcal{N}_{A\\to B}\\|_{\\Diamond}:=\\sup_{\\|X\\|_1\\leq1}\\|(\\mathcal{N}_{A\\to B}\\otimes\\mathcal{I}_A)(X_{AA})\\|_1.\n\\end{align}\n\nA multipartite state $W_{AB_1^n}$ on $AB_1^n\\equiv AB_1\\cdots B_n$ is called {\\it symmetric with respect to $A$} if\n\\begin{align}\n(\\mathcal{I}_A\\otimes\\mathcal{U}_{B_1^n}^\\pi)(W_{AB_1^n})=W_{AB_1^n}\\quad\\forall\\pi\\in\\mathfrak{S}_n,\n\\end{align}\nwhere $\\mathfrak{S}_n$ denotes the symmetric group of $n$ elements and\n\\begin{align}\n\\mathcal{U}_{B_1^n}^\\pi(W_{B_1}\\otimes\\cdots\\otimes W_{B_n}):=W_{B_{\\pi^{-1}(1)}}\\otimes\\cdots\\otimes W_{B_{\\pi^{-1}(n)}}.\n\\end{align}\nA bipartite state $W_{AB}$ is called {\\it $n$-extendable} if there exists a multipartite extension $W_{AB_1^n}$, i.e., $\\mathrm{Tr}_{B_2^n}(W_{AB_1^n})=W_{AB}$, that is symmetric with respect to $A$.\n\n\nThe starting point for our proof technique is the use of the chain rule of the conditional mutual information, first used in this context in~\\cite{brandao11} and further exploited in~\\cite{brandao13b}. More precisely, we will use the quantum relative entropy defined as\n\\begin{align}\nD(\\rho\\|\\sigma):=\n\\begin{cases} \\tr(\\rho \\log \\rho) - \\tr(\\rho \\log \\sigma) & \\mbox{if } support(\\rho)\\subseteq support(\\sigma) \\\\ \\infty & \\mbox{otherwise} \\end{cases},\n\\end{align}\nwhere $\\rho$ and $\\sigma$ are quantum states and the logarithm is taken with respect to the basis two. Recall that the support of an operator is defined as the orthogonal complement of its kernel. The following lemma, which can be found in~\\cite{brandao13b}, says that if some classical systems $Z_1^n$ are symmetric with respect to $A$, then conditioning on $Z_1^m$ for some value of $m$ breaks the correlations between $A$ and $Z_{m+1}$. Before stating the lemma, we introduce notation that will be used throughout the section. For a state $\\rho_{A Z}$ with a classical $Z$-system, we write\n\\begin{align}\n\\rho_{A|z}:= \\frac{\\mathrm{Tr}_{Z}\\Big[\\rho_{AZ} \\left(1_{A} \\otimes \\ket{z}\\bra{z}\\right)\\Big]}{\\mathrm{Tr}\\Big[\\rho_{AZ} \\left(1_{A} \\otimes \\ket{z}\\bra{z}\\right)\\Big]}.\n\\end{align}\nWe simply write $\\exc{z_{1}^{m}}{\\cdot}$ for the expectation over the choices of $z_1^m$ and the distribution will be clear from the context.\n\n\nTo prove the de Finetti theorem, we will crucially make use of so-called informationally complete measurements for which the loss in distinguishability, or {\\it distortion}, can be bounded.\n\n\\begin{lemma}{\\cite[Lemma 14]{brandao13b}}\\label{lem:product-ic-meas}\nThere exist a product measurement $\\mathcal{M}_A\\otimes\\mathcal{M}_B$ with finitely many outcomes such that for any Hermitian and traceless matrix $\\xi_{AB}$ on $A\\otimes B$, we have \n\\begin{align}\n\\|(\\mathcal{M}_{A} \\otimes \\mathcal{M}_{B})(\\xi_{AB})\\|_1 \\geq \\frac{1}{18\\sqrt{d_A d_B}}\\| \\xi_{AB}\\|_{1}.\n\\end{align}\n\\end{lemma}\n\nThis \\cite[Lemma 14]{brandao13b} follows from the methods in \\cite{lancien13}. More generally, we define the {\\it minimal distortion} for the bipartite system $A \\otimes B$ as\n\\begin{align}\\label{eq:def-f1}\nf(A, B):=\\inf_{\\cM_{A}, \\cM_{B}}\\max_{\\substack{\\xi_{AB}^{\\dagger} = \\xi_{AB}\\\\ \\xi_A = 0, \\xi_B = 0}} \\frac{\\| \\xi_{AB} \\|_1}{\\| (\\cM_{A} \\otimes \\cM_B)(\\xi_{AB})\\|_1},\n\\end{align}\nwhere the infimum is over all product measurements on $AB$. In this notation, Lemma~\\ref{lem:product-ic-meas} shows that\n\\begin{align}\nf(A, B) \\leq 18 \\sqrt{d_A d_B}.\n\\end{align}\nNote that in the definition of $f(A,B)$ we restricted the maximization to matrices satisfying $\\xi_{A} = 0$ and $\\xi_{B} = 0$ because this is sufficient for us.\n\nA drawback of Lemma~\\ref{lem:product-ic-meas} is that the distortion depends on the dimension $d_A$. More generally, we define the {\\it minimal distortion with side information} for a system $B$ as\n\\begin{align}\\label{eq:def-f2}\nf(B|\\cdot):=\\inf_{\\cM_B} \\sup_{\\substack{\\xi_{AB}^\\dagger=\\xi_{AB}\\\\ \\xi_{A} = 0, \\xi_B = 0}} \\frac{\\| \\xi_{AB} \\|_1}{\\| (\\cI_{A} \\otimes \\cM_B)(\\xi_{AB})\\|_1},\n\\end{align}\nwhere the infimum is over all measurements on $B$ and the supremum is over all finite-dimensional systems $A$. In Lemma~\\ref{lem:ic-meas-side-info} we give an elementary proof that\n\\begin{align}\nf(B|\\cdot) \\leq d_B^2 (d_B+1)\n\\end{align}\nusing state two-designs and properties of weighted non-commutative $L_p$-spaces. In fact, after completion of our work we realised that methods from operator space theory even give the stronger bound\n\\begin{align}\nf(B|\\cdot) \\leq \\sqrt{18d_B^3},\n\\end{align}\nwhich is discussed in \\cite[Equation 66]{brandao13c}. We leave it as an open question to determine the exact dimensional dependence of the minimal distortion with side information.\n\n\nWe note that the existence of such two-designs is known for any dimension, see e.g., \\cite[Corollary 5.3]{scott08} for unitary two-designs and applying these unitaries to any fixed state leads to a state two-design.\n","units":[{"raw_tex":"To discuss our approach, we first rewrite~\\eqref{eq:form} by defining $G_{AB}= (\\Pi_{A \\to D}^{\\dagger} \\otimes \\Pi_{B \\to E}^{\\dagger})(H)$, where $\\Pi^{\\dagger}$ denotes the adjoint map of $\\Pi$ in the Hilbert-Schmidt inner product. This leads to the form\n\\begin{align}\\label{original_program}\nQ=\\max &\\quad \\mathrm{Tr}\\big[G_{AB} (W_{A} \\otimes W_{B})\\big]\\\\\ns.t. &\\quad W_{A} \\succeq 0,\\quad W_{B} \\succeq 0,\\quad\\mathrm{Tr}[W_A] = \\mathrm{Tr}[W_B] = 1\\\\\n&\\quad\\Lambda_{A\\to C_A}\\left(W_{A}\\right)=X_{C_A},\\quad\\Gamma_{B\\to C_B}\\left(W_{B}\\right)=Y_{C_B}, \n\\end{align}\nwhere $G_{AB}\\in\\mathcal{S}_A\\otimes\\mathcal{S}_B$, $\\Lambda_{A\\to C_A}:\\mathcal{S}_A \\to \\mathcal{S}_{C_A}$ and $\\Gamma_{B\\to C_B}:\\mathcal{S}_B \\to \\mathcal{S}_{C_B}$ denote linear maps, and $X_{C_A}\\in\\mathcal{S}_{C_A}$ and $Y_{C_B}\\in\\mathcal{S}_{C_B}$ are the matrices defining $\\mathcal{A}_{A}$ and $\\mathcal{A}_B$ as the affine subspaces associated with the kernels of the linear maps $\\Lambda_{A\\to C_A}$ and $\\Gamma_{B\\to C_B}$, respectively.\nNow, by the linearity of the objective function we can equivalently optimise over the convex hull of feasible points\n\\begin{align}\\label{eq:transformed_program}\nQ=\\max &\\quad \\mathrm{Tr}\\left[G_{AB}\\left(\\sum_{i\\in I}p_iW_A^i \\otimes W_B^i\\right)\\right]\\\\\ns.t. &\\quad p_i\\geq0,\\quad W_A^i \\succeq 0,\\quad W_B^i \\succeq 0,\\quad\\mathrm{Tr}\\left[W_A^i\\right] = \\mathrm{Tr}\\left[W_B^i\\right] = 1\\quad\\forall i\\in I,\\quad\\sum_{i\\in I}p_i=1\\\\\n&\\quad\\Lambda_{A\\to C_A}\\left(W_A^i\\right)=X_{C_A},\\quad\\Gamma_{B\\to C_B}\\left(W_B^i\\right)=Y_{C_B}\\quad\\forall i\\in I.\n\\end{align}\nThat is, in the language of quantum information theory we are maximizing over a subset of the so-called {\\it separable quantum states}\\,---\\,where the latter is defined on $A\\otimes B$ as\n\\begin{align}\n\\text{Sep}(A:B)=\\left\\{\\sum_{i\\in I}p_iW_A^i \\otimes W_B^i: W_A^i \\succeq 0, W_B^i \\succeq 0, \\mathrm{Tr}\\left[W_A^i\\right]=\\mathrm{Tr}\\left[W_B^i\\right] = 1,p_i\\geq0,\\sum_{i\\in I}p_i=1\\right\\}.\n\\end{align}\nRecall that matrices $W_A\\in\\mathcal{S}^+_A$ are called quantum states on system $A$\\,---\\,and similarly for bipartite states on $A\\otimes B$.\n","start":7954,"end":10100},{"raw_tex":"In the following, we introduce some notation that is standard in quantum information theory. A $d_A$-dimensional quantum system (or in short {\\it system}) is given by an inner product space $\\mathbb{C}^{d_A}$ and denoted by $A$. {\\it Quantum states} (or in short {\\it states}) on $A$ are matrices\\footnote{Here and henceforth we use the symbol $:=$ as {\\it equal by definition}.}\n\\begin{align}\n\\text{$W_A\\in\\mathcal{S}_A:=\\mathbb{C}^{d_A\\times d_A}$ with $\\mathrm{Tr}[W_A]=1$ and $W_A\\succeq0$,}\n\\end{align}\nwhere $\\succeq$ denotes the operator (Loewner) order. Quantum states of rank one are called {\\it pure} and can be written as $W_A=\\ket{\\psi}\\bra{\\psi}_A$, where $\\ket{\\psi}_A\\in\\mathbb{C}^{d_A}$ and $\\ket{\\psi}\\bra{\\psi}_A\\in\\mathcal{S}_A$ denotes the rank-one projector on the vector $\\ket{\\psi}_A$.\n\nA {\\it bipartite system} $AB:=A\\otimes B$ is given by an inner product space $\\mathbb{C}^{d_A}\\otimes\\mathbb{C}^{d_B}$, where $\\otimes$ denotes the Kronecker tensor product. Correspondingly, states on $AB$ are matrices $W_{AB}\\in\\mathcal{S}_A\\otimes\\mathcal{S}_B$ with $\\mathrm{Tr}[W_{AB}]=1$ and $W_{AB}\\succeq0$. {\\it Separable states} are states on $AB$ that are in the convex hull of product states $W_A\\otimes W_B$, with $W_A$ and $W_B$ states on $A$ and $B$, respectively. The {\\it maximally entangled state} $\\Phi_{AB}:=\\ket{\\Phi}\\bra{\\Phi}_{AB}$ on $AB$ for $d:=d_A=d_B$ is not separable and defined via\n\\begin{align}\n\\text{$\\ket{\\Phi}_{AB}:=\\frac{1}{\\sqrt{d}}\\sum_{x=1}^d\\ket{x}_A\\otimes\\ket{x}_B$ for some orthonormal basis $\\{\\ket{x}\\}_{x=1}^d$ of $\\mathbb{C}^d$.}\n\\end{align}\nThe {\\it swap operator} $F_{AB}$ on $A\\otimes B$ exchanges the two quantum systems, i.e., $F_{AB}(W_A\\otimes W_B)=W_B\\otimes W_A$ for every state $W_A$ and $W_B$ on $A$ and $B$, respectively. {\\it Classical-quantum states} are bipartite states that can be written in the form\n\\begin{align}\nW_{XB}=\\sum_{x=1}^{d_X}p_x\\ket{x}\\bra{x}_X\\otimes W_B^x\n\\end{align}\nfor a probability distribution $\\{p_x\\}_{x=1}^{d_A}$, an orthonormal basis $\\{\\ket{x}_X\\}_{x=1}^{d_X}$ of $\\mathbb{C}^{d_X}$, and states $W_B^x$ on $B$ for $x=1,\\dots,d_X$. We refer to $X$ as the classical part of the bipartite classical-quantum system $XB$.\n\n{\\it Quantum channels} (or in short {\\it channels}) are linear maps $\\mathcal{N}_{A\\to B}:\\mathcal{S}_A\\to\\mathcal{S}_B$ that are trace preserving and completely positive\\footnote{A linear map $\\mathcal{N}_{A\\to B}:\\mathcal{S}_A\\to\\mathcal{S}_B$ is said to be completely positive if $\\mathcal{N}_{A\\to B}\\otimes \\mathcal{I}_C$ is a positive map for every quantum system $C$, where $\\mathcal{I}_C$ denotes the identity map on $\\mathcal{S}_C$} (cp). In particular, they map states from the input system $A$ to states on the output system $B$. We often abbreviate bipartite channels $\\mathcal{I}_A\\otimes\\mathcal{N}_B$ that act trivially on the $A$-system as $\\mathcal{N}_B$, where $\\mathcal{I}_A$ denotes the identity channel on $\\mathcal{S}_A$. The partial trace $\\mathrm{Tr}_B[\\cdot]$ is a channel from $AB$ to $A$ defined via\n\\begin{align}\n\\mathrm{Tr}_B[\\cdot]:=\\sum_{x=1}^{d_B}\\big(1_A\\otimes\\bra{x}_B\\big)(\\cdot)\\big(1_A\\otimes\\ket{x}_B\\big),\n\\end{align}\nwhere $1_A$ denotes the identity matrix on $A$, and $\\{\\ket{x}\\}_{x=1}^{d_B}$ an orthonormal basis of $\\mathbb{C}^{d_B}$. For bipartite states $W_{AB}$, we write $W_A=\\mathrm{Tr}_B[W_{AB}]$ for the reduced state on $A$. {\\it Quantum measurements} (or in short {\\it measurements}) are a special case of channels that can be written in the form\n\\begin{align}\n\\mathcal{M}_{A\\to I}(\\cdot):=\\sum_{i\\in I}\\mathrm{Tr}\\left[M_A^i(\\cdot)\\right]\\ket{i}\\bra{i}_I\n\\end{align}\nwith an orthonormal basis $\\{\\ket{i}_I\\}_{i\\in I}$ and $M_A^i\\succeq0$ $\\forall i\\in I$ with $\\sum_{i\\in I}M_A^i=1_A$.\n","start":17076,"end":20833},{"raw_tex":"The distance between states is quantified by the metric induced from the Schatten one-norm $\\|X\\|_1:=\\mathrm{Tr}\\left[|X|\\right]$. The distance between channels is quantified by the metric induced from the Diamond norm\n\\begin{align}\n\\|\\mathcal{N}_{A\\to B}\\|_{\\Diamond}:=\\sup_{\\|X\\|_1\\leq1}\\|(\\mathcal{N}_{A\\to B}\\otimes\\mathcal{I}_A)(X_{AA})\\|_1.\n\\end{align}\n\nA multipartite state $W_{AB_1^n}$ on $AB_1^n\\equiv AB_1\\cdots B_n$ is called {\\it symmetric with respect to $A$} if\n\\begin{align}\n(\\mathcal{I}_A\\otimes\\mathcal{U}_{B_1^n}^\\pi)(W_{AB_1^n})=W_{AB_1^n}\\quad\\forall\\pi\\in\\mathfrak{S}_n,\n\\end{align}\nwhere $\\mathfrak{S}_n$ denotes the symmetric group of $n$ elements and\n\\begin{align}\n\\mathcal{U}_{B_1^n}^\\pi(W_{B_1}\\otimes\\cdots\\otimes W_{B_n}):=W_{B_{\\pi^{-1}(1)}}\\otimes\\cdots\\otimes W_{B_{\\pi^{-1}(n)}}.\n\\end{align}\nA bipartite state $W_{AB}$ is called {\\it $n$-extendable} if there exists a multipartite extension $W_{AB_1^n}$, i.e., $\\mathrm{Tr}_{B_2^n}(W_{AB_1^n})=W_{AB}$, that is symmetric with respect to $A$.\n","start":21568,"end":22592},{"raw_tex":"The starting point for our proof technique is the use of the chain rule of the conditional mutual information, first used in this context in~\\cite{brandao11} and further exploited in~\\cite{brandao13b}. More precisely, we will use the quantum relative entropy defined as\n\\begin{align}\nD(\\rho\\|\\sigma):=\n\\begin{cases} \\tr(\\rho \\log \\rho) - \\tr(\\rho \\log \\sigma) & \\mbox{if } support(\\rho)\\subseteq support(\\sigma) \\\\ \\infty & \\mbox{otherwise} \\end{cases},\n\\end{align}\nwhere $\\rho$ and $\\sigma$ are quantum states and the logarithm is taken with respect to the basis two. Recall that the support of an operator is defined as the orthogonal complement of its kernel. The following lemma, which can be found in~\\cite{brandao13b}, says that if some classical systems $Z_1^n$ are symmetric with respect to $A$, then conditioning on $Z_1^m$ for some value of $m$ breaks the correlations between $A$ and $Z_{m+1}$. Before stating the lemma, we introduce notation that will be used throughout the section. For a state $\\rho_{A Z}$ with a classical $Z$-system, we write\n\\begin{align}\n\\rho_{A|z}:= \\frac{\\mathrm{Tr}_{Z}\\Big[\\rho_{AZ} \\left(1_{A} \\otimes \\ket{z}\\bra{z}\\right)\\Big]}{\\mathrm{Tr}\\Big[\\rho_{AZ} \\left(1_{A} \\otimes \\ket{z}\\bra{z}\\right)\\Big]}.\n\\end{align}\nWe simply write $\\exc{z_{1}^{m}}{\\cdot}$ for the expectation over the choices of $z_1^m$ and the distribution will be clear from the context.\n","start":26930,"end":28329},{"raw_tex":"To prove the de Finetti theorem, we will crucially make use of so-called informationally complete measurements for which the loss in distinguishability, or {\\it distortion}, can be bounded.\n\n\\begin{lemma}{\\cite[Lemma 14]{brandao13b}}\\label{lem:product-ic-meas}\nThere exist a product measurement $\\mathcal{M}_A\\otimes\\mathcal{M}_B$ with finitely many outcomes such that for any Hermitian and traceless matrix $\\xi_{AB}$ on $A\\otimes B$, we have \n\\begin{align}\n\\|(\\mathcal{M}_{A} \\otimes \\mathcal{M}_{B})(\\xi_{AB})\\|_1 \\geq \\frac{1}{18\\sqrt{d_A d_B}}\\| \\xi_{AB}\\|_{1}.\n\\end{align}\n\\end{lemma}\n\nThis \\cite[Lemma 14]{brandao13b} follows from the methods in \\cite{lancien13}. More generally, we define the {\\it minimal distortion} for the bipartite system $A \\otimes B$ as\n\\begin{align}\\label{eq:def-f1}\nf(A, B):=\\inf_{\\cM_{A}, \\cM_{B}}\\max_{\\substack{\\xi_{AB}^{\\dagger} = \\xi_{AB}\\\\ \\xi_A = 0, \\xi_B = 0}} \\frac{\\| \\xi_{AB} \\|_1}{\\| (\\cM_{A} \\otimes \\cM_B)(\\xi_{AB})\\|_1},\n\\end{align}\nwhere the infimum is over all product measurements on $AB$. In this notation, Lemma~\\ref{lem:product-ic-meas} shows that\n\\begin{align}\nf(A, B) \\leq 18 \\sqrt{d_A d_B}.\n\\end{align}\nNote that in the definition of $f(A,B)$ we restricted the maximization to matrices satisfying $\\xi_{A} = 0$ and $\\xi_{B} = 0$ because this is sufficient for us.\n\nA drawback of Lemma~\\ref{lem:product-ic-meas} is that the distortion depends on the dimension $d_A$. More generally, we define the {\\it minimal distortion with side information} for a system $B$ as\n\\begin{align}\\label{eq:def-f2}\nf(B|\\cdot):=\\inf_{\\cM_B} \\sup_{\\substack{\\xi_{AB}^\\dagger=\\xi_{AB}\\\\ \\xi_{A} = 0, \\xi_B = 0}} \\frac{\\| \\xi_{AB} \\|_1}{\\| (\\cI_{A} \\otimes \\cM_B)(\\xi_{AB})\\|_1},\n\\end{align}\nwhere the infimum is over all measurements on $B$ and the supremum is over all finite-dimensional systems $A$. In Lemma~\\ref{lem:ic-meas-side-info} we give an elementary proof that\n\\begin{align}\nf(B|\\cdot) \\leq d_B^2 (d_B+1)\n\\end{align}\nusing state two-designs and properties of weighted non-commutative $L_p$-spaces. In fact, after completion of our work we realised that methods from operator space theory even give the stronger bound\n\\begin{align}\nf(B|\\cdot) \\leq \\sqrt{18d_B^3},\n\\end{align}\nwhich is discussed in \\cite[Equation 66]{brandao13c}. We leave it as an open question to determine the exact dimensional dependence of the minimal distortion with side information.\n","start":29993,"end":32393},{"raw_tex":"We note that the existence of such two-designs is known for any dimension, see e.g., \\cite[Corollary 5.3]{scott08} for unitary two-designs and applying these unitaries to any fixed state leads to a state two-design.\n","start":124771,"end":124987}]},"statement":{"raw_tex":"\\begin{theorem}\nFor the SDPs\n\\begin{align}\n\\mathrm{SDP}_n :=\\max &\\quad \\mathrm{Tr}\\big[G_{AB} W_{AB_1}\\big]\\\\\ns.t. \n&\\quad W_{AB_1^n}\\succeq0, \\mathrm{Tr}(W_{AB_1^n}) = 1, \\;W_{AB_1^n}=\\mathcal{U}_{B_1^n}^\\pi\\left(W_{AB_1^n}\\right)\\;\\forall\\pi\\in\\mathfrak{S}_n\\\\\n& \\quad \\Lambda_{A\\to C_A}\\left(W_{AB_1^n}\\right)=X_{C_A}\\otimes W_{B_1^n},\\;\\Gamma_{B_n\\to C_B}\\left(W_{B_1^n}\\right)=W_{B_1^{n-1}}\\otimes Y_{C_B}\n\\end{align}\nand $Q$ defined as above, we have for $d:=\\max\\{d_A,d_B\\}$ that\n\\begin{align}\n0 \\leq \\mathrm{SDP}_n - Q \\leq \\frac{\\mathrm{poly}(d)}{\\sqrt{n}}\\quad\\text{implying}\\quad Q=\\lim_{n\\to\\infty}\\mathrm{SDP}_n.\n\\end{align}\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\nFor the SDPs\n\\begin{align}\n\\mathrm{SDP}_n :=\\max &\\quad \\mathrm{Tr}\\big[G_{AB} W_{AB_1}\\big]\\\\\ns.t. \n&\\quad W_{AB_1^n}\\succeq0, \\mathrm{Tr}(W_{AB_1^n}) = 1, \\;W_{AB_1^n}=\\mathcal{U}_{B_1^n}^\\pi\\left(W_{AB_1^n}\\right)\\;\\forall\\pi\\in\\mathfrak{S}_n\\\\\n& \\quad \\Lambda_{A\\to C_A}\\left(W_{AB_1^n}\\right)=X_{C_A}\\otimes W_{B_1^n},\\;\\Gamma_{B_n\\to C_B}\\left(W_{B_1^n}\\right)=W_{B_1^{n-1}}\\otimes Y_{C_B}\n\\end{align}\nand $Q$ defined as above, we have for $d:=\\max\\{d_A,d_B\\}$ that\n\\begin{align}\n0 \\leq \\mathrm{SDP}_n - Q \\leq \\frac{\\mathrm{poly}(d)}{\\sqrt{n}}\\quad\\text{implying}\\quad Q=\\lim_{n\\to\\infty}\\mathrm{SDP}_n.\n\\end{align}\n\\end{theorem}","start":48380,"end":49032}]},"proof":{"raw_tex":"\\begin{lemma}\\cite{brandao13b}\\label{lem:conditioning-breaks-correlations}\nLet $\\rho_{A Z_1^n}$ be a classical-quantum state with the $Z_1^n$-systems classical and $\\mathcal{U}^\\pi_{Z_1^n}(\\rho_{A Z_1^n})=\\rho_{A Z_1^n}$ for all $\\pi\\in\\mathfrak{S}_n$. Then, there exists $0\\leq m < n$ such that\n\\begin{align}\n\\exc{z_{1}^{m}}{D(\\rho_{AZ_{m+1}|z_1^m} \\| \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m})} &\\leq \\frac{\\log d_A}{n}\n\\end{align}\nas well as\n\\begin{align}\n\\exc{z_{1}^{m}}{\\| \\rho_{AZ_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m} \\|_1^2} &\\leq \\frac{(2 \\ln 2) \\log d_A}{n},\n\\end{align}\nwhere $\\ln(\\cdot)$ denotes the natural logarithm.\n\\end{lemma}\n\n\\begin{proof}\nFor the quantum mutual information we have $I\\left(A:Z_1^n\\right)_{\\rho}:=D(\\rho_{AZ_1^n}\\|\\rho_A\\otimes \\rho_{Z_1^n}) \\leq \\log d_A$ as well as (see, e.g., \\cite[Chapter 11]{nielsen00})\n\\begin{align}\nI\\left(A:Z_1^n\\right)_{\\rho}=\\sum_{m=0}^{n-1} I(A : Z_{m+1} | Z_1^m)_{\\rho}\n\\end{align}\nfor the quantum conditional mutual information $I(A : Z_{m+1} | Z_1^m)_\\rho:=I(A:Z_1^{m+1})_\\rho-I(A:Z_1^m)_\\rho$. As a result, there exists an $m \\in \\{0, \\cdots, n-1\\}$ such that $I(A : Z_{m+1} | Z_1^m)_{\\rho} \\leq \\frac{\\log d_A}{n}$, which implies\n\\begin{align}\n\\exc{z_1^m}{I(A:Z_{m+1})_{\\rho_{AZ_{m+1}|z_1^m}}} \\leq \\frac{\\log d_A}{n},\n\\end{align}\nwhere we used $I(A : Z_{m+1} | Z_1^m)_{\\rho}=\\exc{z_1^m}{I(A:Z_{m+1})_{\\rho_{AZ_{m+1}|z_1^m}}}$, which holds since the conditioning systems are classical. The second statement then follows directly from Pinsker's inequality $D(\\rho\\|\\sigma)\\geq\\frac{1}{2\\ln 2}\\left\\|\\rho-\\sigma\\right\\|_1^2$ \\cite[Theorem 5.38]{watrous-book}.\n\\end{proof}\n\n\\begin{theorem}\\label{thm:main-deFinetti}\nLet $\\rho_{AB_1^n}$ be a quantum state, $\\Lambda_{A\\to C_A},\\Gamma_{B\\to C_B}$ linear maps, and $X_{C_A},Y_{C_B}$ matrices such that\n\\begin{align}\n\\mathcal{U}_{B_1^n}^\\pi(\\rho_{AB_1^n})&=\\rho_{AB_1^n}\\;\\forall\\pi\\in\\mathfrak{S}_n\\qquad\\text{symmetric with respect to $A$}\\\\\n\\Lambda_{A\\to C_A}(\\rho_{AB_1^n})&=X_{C_A}\\otimes\\rho_{B_1^n}\\qquad\\quad\\;\\;\\text{linear constraint on $A$}\\\\\n\\Gamma_{B_n\\to C_B}(\\rho_{B_1^n})&=\\rho_{B_1^{n-1}}\\otimes Y_{C_B}\\qquad\\quad\\text{linear constraint on $B$}.\n\\end{align}\nThen, we have that\n\\begin{align}\n\\left\\|\\rho_{AB}-\\sum_{i\\in I}p_i\\sigma^i_{A}\\otimes\\omega^i_B\\right\\|_1\\leq&\\;\\min\\big\\{ f(A,B) , f(B|\\cdot) \\big\\}\\sqrt{ \\frac{(2 \\ln 2) \\log \\left(d_A\\right)}{n}}\n\\end{align}\nwith $\\{p_i\\}_{i\\in I}$ a probability distribution, $\\rho_{AB}=\\tr_{B_2^n}\\left[\\rho_{AB_1^n}\\right]$, and quantum states $\\sigma^i_A,\\omega_B^i$ such that for $i\\in I$:\n\\begin{align}\n\\Lambda_{A\\to C_A}\\left(\\sigma^i_A\\right)=X_{C_A}\\quad\\text{and}\\quad\\Gamma_{B\\to C_B}\\left(\\omega^i_B\\right)=Y_{C_B}.\n\\end{align}\nAs stated in Section~\\ref{sec:proof-tools}, we can, e.g., take $f(A,B)\\leq18\\sqrt{d_A d_B}$ or $f(B|\\cdot)\\leq\\sqrt{18d_B^3}$.\n\\end{theorem}\n\n\\begin{proof}\nLet $\\cM_{B}$ be a measurement of the $B$ system and call the outcome system $Z$. Consider the state $\\rho_{AZ_1^n}$ obtained by measuring all the $B$ systems with $\\cM_{B}$. This distribution is symmetric with respect to $A$ and so we can apply Lemma~\\ref{lem:conditioning-breaks-correlations}. We find that there exists an $m \\in \\{0, \\cdots, n-1\\}$ such that\n\\begin{align}\n\\exc{z_1^m}{\\| \\rho_{AZ_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m} \\|_1^2} \\leq \\frac{(2 \\ln 2) \\log d_{A}}{n}.\n\\end{align}\nNote that we have for any $z_1^m$, $\\rho_{AZ_{m+1}|z_1^m} = (\\cI_{A} \\otimes \\cM_{B})(\\rho_{AB_{m+1}|z_1^m})$ and correspondingly $\\rho_{Z_{m+1}|z_1^m} = \\cM_{B}(\\rho_{B_{m+1}|z_1^m})$. Now, we choose the measurement $\\cM_{B}$ to be as in Lemma~\\ref{lem:ic-meas-side-info} and achieving $f(B|\\cdot)$ in \\eqref{eq:def-f2}, we get that $\\| \\xi_{AB} \\|_1^2 \\leq f(B|\\cdot)^2 \\| (\\cI_{A} \\otimes \\cM_{B})(\\xi_{AB}) \\|_1^2$, where $\\xi_{AB} = \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}$. As a result, we have\n\\begin{align}\n\\exc{z_1^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1^2} \\leq f(B|\\cdot)^2\\frac{(2 \\ln 2) \\log d_{A}}{n}.\n\\end{align}\nBut note we can also choose measurements $\\cM_{A}$ and $\\cM_{B}$ achieving $f(A,B)$ in \\eqref{eq:def-f1}. In this case,\n\\begin{align} \n\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1^2 &\\leq f(A,B)^2\\| (\\cM_A \\otimes \\cM_{B})(\\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}) \\|_1^2 \\\\\n&\\leq f(A,B)^2 \\| (\\cI_A \\otimes \\cM_{B})(\\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}) \\|_1^2 \\\\\n&= f(A,B)^2 \\| \\rho_{AZ_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m} \\|_1^2 ,\n\\end{align}\nwhere we used the fact that the trace norm cannot increase when applying the quantum channel $\\cM_{A}$ \\cite[Theorem 3.39]{watrous-book}. As a result, we get\n\\begin{align}\n\\exc{z_1^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1^2} \\leq f(A,B)^2 \\frac{(2 \\ln 2) \\log d_{A}}{n}.\n\\end{align}\nNow, using the convexity of the square function, we get\n\\begin{align}\n\\exc{z_{1}^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1}&\\leq \\sqrt{\\exc{z_{1}^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|^2_1}} \\\\\n&\\leq  \\min\\big\\{f(A, B), f(B|\\cdot)\\big\\}\\sqrt{\\frac{(2 \\ln 2) \\log d_{A}}{n}}.\n\\end{align}\nThen, using the convexity of the norm and the fact that $\\exc{z_1^m}{\\rho_{AB_{m+1}|z_1^m}} = \\rho_{AB_{m+1}}$, we obtain\n\\begin{align}\n\\left\\| \\rho_{AB_{m+1}} - \\exc{z_1^m}{\\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}} \\right\\|_1 &\\leq \\min\\big\\{f(A,B), f(B|\\cdot)\\big\\}\\sqrt{\\frac{(2 \\ln 2) \\log d_{A}}{n}}.\n\\end{align}\nThe state $\\exc{z_1^m}{\\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}}$ corresponds to our candidate mixture of product states. It now remains to show that all the states in the mixture satisfy the linear constraints. Indeed we have for any $z_1^m$, writing $M_{z}$ for matrices of the measurement $\\cM_{B}$,\n\\begin{align}\n\\Lambda_{A \\to C_{A}}(\\rho_{A|z_1^m})\n&= \\frac{\\tr_{B_1^m}\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m}) \\Lambda_{A \\to C_{A}}(\\rho_{A B_1^m})\\Big]}{\\tr\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m})\\rho_{A B_1^m}\\Big]} \\\\\n&= \\frac{\\tr_{B_1^m}\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m}) (X_{C_A} \\otimes \\rho_{B_1^m}) \\Big]}{\\tr\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m})\\rho_{A B_1^m}\\Big]} \\\\\n&= X_{C_A},\n\\end{align}\nand similarly\n\\begin{align}\n\\Gamma_{B \\to C_{B}}(\\rho_{B_{m+1}|z_1^m})\n&= \\frac{\\tr_{B_1^m}\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{C_{B}} ) \\Gamma_{B_{m+1} \\to C_{B}}(\\rho_{B_1^{m+1}})\\Big]}{\\tr\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{B_{m+1}} )\\rho_{B_1^{m+1}}\\Big]} \\\\\n&= \\frac{\\tr_{B_1^m}\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{B_{m+1}} ) (\\rho_{B_1 \\cdots B_{m}} \\otimes Y_{C_B})\\Big]}{\\tr\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{B_{m+1}} )\\rho_{B_1^{m+1}}\\Big]} \\\\\n&= Y_{C_B}.\n\\end{align}\n\\end{proof}\n\n\\begin{proof}\nWe have by construction $0 \\leq \\mathrm{SDP}_n - Q $ and the remaining inequality arises from\n\\begin{align}\n\\mathrm{Tr}\\left[G_{AB} W_{AB_1}\\right]&=\\mathrm{Tr}\\left[G_{AB} (W_{A} \\otimes W_{B})\\right]+\\mathrm{Tr}\\left[G_{AB}\\left(W_{AB_1} - W_{A} \\otimes W_{B}\\right)\\right] \\\\\n& \\leq \\mathrm{Tr}\\left[G_{AB} (W_{A} \\otimes W_{B})\\right]+\\| G_{AB} \\|_\\infty \\cdot \\|W_{AB_1} - W_{A} \\otimes W_{B} \\|_1 \\\\\n&\\leq\\mathrm{Tr}\\left[G_{AB} (W_{A} \\otimes W_{B})\\right]+\\frac{\\mathrm{poly}(d)}{\\sqrt{n}},\n\\end{align}\nwhere we used H\\\"older's inequality and the de Finetti argument as in Theorem \\ref{thm:main-deFinetti}.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:ic-meas-side-info}\nConsider a state two-design on $B$, i.e., a set of rank-one projectors $\\{P_z\\}_{z \\in \\{1, \\dots, t\\}}$ such that $\\frac{1}{t} \\sum_{z=1}^t P_z \\otimes P_z = \\frac{2 P^{\\mathrm{sym}}}{d_B (d_B+1)}$, where $P^{\\mathrm{sym}}$ denotes the projector onto the symmetric subspace of $B \\otimes B$. Let $\\mathcal{M}_B$ be the measurement defined as\n\\begin{align}\n\\mathcal{M}_B(X) = \\sum_{z} \\frac{d_B}{t} \\cdot\\mathrm{Tr}\\big[P_z X\\big] |z\\rangle\\langle z|,\n\\end{align}\nand $\\xi_{AB}$ be a Hermitian matrix on $A\\otimes B$. Then, we have that\n\\begin{align}\n\\| (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\|_1\\geq \\frac{1}{d_B^2 (d_B+1)} \\| \\xi_{AB} \\|_{1}.\n\\end{align}\n\\end{lemma}\n\n\\begin{proof}\nFor any full rank quantum state $\\sigma_{A}$, we have by a H\\\"older type inequality for $\\sigma$-weighted Schatten norms that (see, e.g., \\cite{OLKIEWICZ1999246} or \\cite{beigi13})\n\\begin{align}\n\\| (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\|_1 \\geq  \\frac{\\left\\| \\sigma_{A}^{-1/4} (\\mathcal{I}_{A} \\otimes \\mathcal{M})(\\xi_{AB}) \\sigma_{A}^{-1/4} \\right\\|^2_{2}}{\\left\\| \\sigma_{A}^{-1/2} (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\sigma_{A}^{-1/2} \\right\\|_{\\infty}}.\n\\end{align}\nFor example, the above inequality can be obtained using \\cite[Corollary 3]{beigi13} with the operator $\\sigma_{A}^{-1/2} (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\sigma_{A}^{-1/2}$, weight $\\sigma$, and $p_\\theta = 2$, $\\theta = 1/2$, $p_0 = 1$, $p_1 = \\infty$. We note that this particular H\\\"older type inequality for $\\sigma$-weighted norms is elementary and follows easily from the usual H\\\"older inequality, but one way of potentially improving the dimension dependence in Lemma~\\ref{lem:ic-meas-side-info} might be to use another H\\\"older inequality, in particular the $(1,4)$ inequality.\n\nHenceforth, we abbreviate $d\\equiv d_B$. To further bound the numerator, letting $\\tilde{\\xi}_{AB}:=\\sigma_{A}^{-1/4} \\xi_{AB} \\sigma_{A}^{-1/4}$ we get\n\\begin{align}\n\\| (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\tilde{\\xi}_{AB}) \\|^2_{2}\n&= \\left\\| \\sum_{z} \\frac{d}{t} |z\\rangle \\langle z| \\otimes  \\mathrm{Tr}_{B}\\left[(1_{A} \\otimes P_z) \\tilde{\\xi}_{AB}\\right] \\right\\|_2^2 \\\\\n&= \\sum_{z}  \\frac{d^2}{t^2} \\mathrm{Tr}\\left[\\mathrm{Tr}_{B}\\left[(1_{A} \\otimes P_z) \\tilde{\\xi}_{AB}\\right] \\otimes \\mathrm{Tr}_{\\bar{B}}\\left[(1_{\\bar{A}} \\otimes P_z) \\tilde{\\xi}_{\\bar{A}\\bar{B}}\\right]^{\\dagger} F_{A\\bar{A}} \\right] \\\\\n&= \\sum_{z} \\frac{d^2}{t^2} \\mathrm{Tr}\\left[\\left(\\left((1_{A} \\otimes P_z) \\tilde{\\xi}_{AB}\\right) \\otimes \\left((1_{\\bar{A}} \\otimes P_z) \\tilde{\\xi}_{\\bar{A}\\bar{B}}\\right)^{\\dagger}\\right)\\left(F_{A\\bar{A}} \\otimes1_{B\\bar{B}}\\right)\\right] \\\\\n&= \\frac{d^2}{t^2} \\mathrm{Tr}\\left[\\left(\\tilde{\\xi}_{AB} \\otimes \\tilde{\\xi}_{\\bar{A}\\bar{B}}^{\\dagger}\\right)\\left(\\sum_{z} (1_{A\\bar{A}} \\otimes P_z \\otimes P_{z})\\right)\\left(F_{A\\bar{A}} \\otimes1_{B \\bar{B}}\\right)\\right] \\\\\n&= \\frac{1}{t} \\frac{d^2}{d(d+1)} \\mathrm{Tr}\\left[\\left(\\tilde{\\xi}_{AB} \\otimes \\tilde{\\xi}_{\\bar{A}\\bar{B}}^{\\dagger}\\right) (1_{A\\bar{A}} \\otimes (1_{B\\bar{B}} + F_{B\\bar{B}}))\\left(F_{A\\bar{A}} \\otimes1_{B \\bar{B}}\\right)\\right] \\\\\n&= \\frac{1}{t} \\frac{d^2}{d(d+1)} \\Big(\\underbrace{\\mathrm{Tr}\\left[\\tilde{\\xi}_{A} \\tilde{\\xi}_{A}^{\\dagger}\\right]}_{\\geq 0} + \\underbrace{\\mathrm{Tr}\\left[\\tilde{\\xi}_{AB} \\tilde{\\xi}_{AB}^{\\dagger}\\right]}_{=\\left\\|\\tilde{\\xi}_{AB}\\right\\|_2^2} \\Big) \\\\\n&\\geq \\frac{1}{t}\\frac{d^2}{d^2 (d+1)} \\| \\xi_{AB} \\|^2_{1} ,\n\\end{align}\nwhere $F$ denotes the swap operator (as defined in Section \\ref{sec:notation}) and in the last step used the H\\\"older inequality (see, e.g., \\cite{bhatia97})\n\\begin{align}\n\\left\\|\\xi_{AB}\\right\\|_1=\\left\\|\\sigma^{1/4}\\sigma^{-1/4}\\xi_{AB}\\sigma^{-1/4}\\sigma^{1/4}\\right\\|_1\\leq\\left\\|\\sigma^{1/4}\\otimes1_B\\right\\|_4\\left\\|\\sigma_A^{-1/4}\\xi_{AB}\\sigma_A^{-1/4}\\right\\|_2\\left\\|\\sigma^{1/4}\\otimes1_B\\right\\|_4\\leq\\sqrt{d}\\left\\|\\tilde{\\xi}_{AB}\\right\\|_2.\n\\end{align}\nFor further bounding the denominator we write\n\\begin{align}\n\\left\\| \\sigma_{A}^{-1/2} (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\sigma_{A}^{-1/2} \\right\\|_\\infty\n&= \\max_{z} \\frac{d}{t} \\left\\| \\mathrm{Tr}_{B}\\left[(1_{A} \\otimes P_z) \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2}\\right]\\right\\|_\\infty \\\\\n&\\leq \\frac{d}{t} \\max_{\\ket{\\phi}_{A}, \\ket{\\psi}_{B}} \\bra{\\phi}_{A} \\otimes \\bra{\\psi}_{B} \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2} \\ket{\\phi}_{A} \\otimes \\ket{\\psi}_{B} \\\\\n&\\leq \\frac{d}{t} \\left\\| \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2} \\right\\|_\\infty,\n\\end{align}\nwhere we used the fact that $P_z$ is a rank 1 projector. Now, observe that for any $\\xi_{AB}$, there exists a $\\sigma_A$ of unit trace such that\n\\begin{align}\n\\frac{\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}}}{\\| \\xi_{AB} \\|_{1}} \\preceq d\\cdot\\sigma_{A} \\otimes1_{B}.\n\\end{align}\nThis just follows from, e.g., \\cite[Lemma B.6]{bertachristandl11}, where it is shown that we can in fact choose\\footnote{By a continuity argument $\\sigma_A$ can be assumed to have full rank.}\n\\begin{align}\n\\sigma_{A} = \\| \\xi_{AB} \\|_{1}^{-1}\\cdot\\mathrm{Tr}_{B}\\left[\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}}\\right].\n\\end{align}\nAs a result, we have\n\\begin{align}\n\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}} \\preceq d\\| \\xi_{AB} \\|_{1}\\cdot\\sigma_{A} \\otimes1_{B}.\n\\end{align}\nAs $\\xi_{AB}$ is Hermitian, we can decompose it into the positive and negative part $\\xi_{AB} = P - Q$ with $P$ and $Q$ positive semidefinite and $PQ = 0$, then $\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}} = P + Q$ and so $-\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}} \\preceq \\xi_{AB} \\preceq \\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}}$. Thus, we get\n\\begin{align}\n- d_{B} \\| \\xi_{AB} \\|_1\\cdot\\sigma_{A} \\otimes1_{B} \\preceq \\xi_{AB} \\preceq d_{B} \\| \\xi_{AB} \\|_1 \\cdot\\sigma_{A} \\otimes1_{B},\n\\end{align}\nand we find $\\| \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2} \\|_{\\infty} \\leq d\\| \\xi_{AB} \\|_1$. This concludes the proof.\n\\end{proof}","units":[{"raw_tex":"\\begin{lemma}\\cite{brandao13b}\\label{lem:conditioning-breaks-correlations}\nLet $\\rho_{A Z_1^n}$ be a classical-quantum state with the $Z_1^n$-systems classical and $\\mathcal{U}^\\pi_{Z_1^n}(\\rho_{A Z_1^n})=\\rho_{A Z_1^n}$ for all $\\pi\\in\\mathfrak{S}_n$. Then, there exists $0\\leq m < n$ such that\n\\begin{align}\n\\exc{z_{1}^{m}}{D(\\rho_{AZ_{m+1}|z_1^m} \\| \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m})} &\\leq \\frac{\\log d_A}{n}\n\\end{align}\nas well as\n\\begin{align}\n\\exc{z_{1}^{m}}{\\| \\rho_{AZ_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m} \\|_1^2} &\\leq \\frac{(2 \\ln 2) \\log d_A}{n},\n\\end{align}\nwhere $\\ln(\\cdot)$ denotes the natural logarithm.\n\\end{lemma}","start":28330,"end":28995},{"raw_tex":"\\begin{proof}\nFor the quantum mutual information we have $I\\left(A:Z_1^n\\right)_{\\rho}:=D(\\rho_{AZ_1^n}\\|\\rho_A\\otimes \\rho_{Z_1^n}) \\leq \\log d_A$ as well as (see, e.g., \\cite[Chapter 11]{nielsen00})\n\\begin{align}\nI\\left(A:Z_1^n\\right)_{\\rho}=\\sum_{m=0}^{n-1} I(A : Z_{m+1} | Z_1^m)_{\\rho}\n\\end{align}\nfor the quantum conditional mutual information $I(A : Z_{m+1} | Z_1^m)_\\rho:=I(A:Z_1^{m+1})_\\rho-I(A:Z_1^m)_\\rho$. As a result, there exists an $m \\in \\{0, \\cdots, n-1\\}$ such that $I(A : Z_{m+1} | Z_1^m)_{\\rho} \\leq \\frac{\\log d_A}{n}$, which implies\n\\begin{align}\n\\exc{z_1^m}{I(A:Z_{m+1})_{\\rho_{AZ_{m+1}|z_1^m}}} \\leq \\frac{\\log d_A}{n},\n\\end{align}\nwhere we used $I(A : Z_{m+1} | Z_1^m)_{\\rho}=\\exc{z_1^m}{I(A:Z_{m+1})_{\\rho_{AZ_{m+1}|z_1^m}}}$, which holds since the conditioning systems are classical. The second statement then follows directly from Pinsker's inequality $D(\\rho\\|\\sigma)\\geq\\frac{1}{2\\ln 2}\\left\\|\\rho-\\sigma\\right\\|_1^2$ \\cite[Theorem 5.38]{watrous-book}.\n\\end{proof}","start":28997,"end":29991},{"raw_tex":"\\begin{theorem}\\label{thm:main-deFinetti}\nLet $\\rho_{AB_1^n}$ be a quantum state, $\\Lambda_{A\\to C_A},\\Gamma_{B\\to C_B}$ linear maps, and $X_{C_A},Y_{C_B}$ matrices such that\n\\begin{align}\n\\mathcal{U}_{B_1^n}^\\pi(\\rho_{AB_1^n})&=\\rho_{AB_1^n}\\;\\forall\\pi\\in\\mathfrak{S}_n\\qquad\\text{symmetric with respect to $A$}\\\\\n\\Lambda_{A\\to C_A}(\\rho_{AB_1^n})&=X_{C_A}\\otimes\\rho_{B_1^n}\\qquad\\quad\\;\\;\\text{linear constraint on $A$}\\\\\n\\Gamma_{B_n\\to C_B}(\\rho_{B_1^n})&=\\rho_{B_1^{n-1}}\\otimes Y_{C_B}\\qquad\\quad\\text{linear constraint on $B$}.\n\\end{align}\nThen, we have that\n\\begin{align}\n\\left\\|\\rho_{AB}-\\sum_{i\\in I}p_i\\sigma^i_{A}\\otimes\\omega^i_B\\right\\|_1\\leq&\\;\\min\\big\\{ f(A,B) , f(B|\\cdot) \\big\\}\\sqrt{ \\frac{(2 \\ln 2) \\log \\left(d_A\\right)}{n}}\n\\end{align}\nwith $\\{p_i\\}_{i\\in I}$ a probability distribution, $\\rho_{AB}=\\tr_{B_2^n}\\left[\\rho_{AB_1^n}\\right]$, and quantum states $\\sigma^i_A,\\omega_B^i$ such that for $i\\in I$:\n\\begin{align}\n\\Lambda_{A\\to C_A}\\left(\\sigma^i_A\\right)=X_{C_A}\\quad\\text{and}\\quad\\Gamma_{B\\to C_B}\\left(\\omega^i_B\\right)=Y_{C_B}.\n\\end{align}\nAs stated in Section~\\ref{sec:proof-tools}, we can, e.g., take $f(A,B)\\leq18\\sqrt{d_A d_B}$ or $f(B|\\cdot)\\leq\\sqrt{18d_B^3}$.\n\\end{theorem}","start":32631,"end":33845},{"raw_tex":"\\begin{proof}\nLet $\\cM_{B}$ be a measurement of the $B$ system and call the outcome system $Z$. Consider the state $\\rho_{AZ_1^n}$ obtained by measuring all the $B$ systems with $\\cM_{B}$. This distribution is symmetric with respect to $A$ and so we can apply Lemma~\\ref{lem:conditioning-breaks-correlations}. We find that there exists an $m \\in \\{0, \\cdots, n-1\\}$ such that\n\\begin{align}\n\\exc{z_1^m}{\\| \\rho_{AZ_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m} \\|_1^2} \\leq \\frac{(2 \\ln 2) \\log d_{A}}{n}.\n\\end{align}\nNote that we have for any $z_1^m$, $\\rho_{AZ_{m+1}|z_1^m} = (\\cI_{A} \\otimes \\cM_{B})(\\rho_{AB_{m+1}|z_1^m})$ and correspondingly $\\rho_{Z_{m+1}|z_1^m} = \\cM_{B}(\\rho_{B_{m+1}|z_1^m})$. Now, we choose the measurement $\\cM_{B}$ to be as in Lemma~\\ref{lem:ic-meas-side-info} and achieving $f(B|\\cdot)$ in \\eqref{eq:def-f2}, we get that $\\| \\xi_{AB} \\|_1^2 \\leq f(B|\\cdot)^2 \\| (\\cI_{A} \\otimes \\cM_{B})(\\xi_{AB}) \\|_1^2$, where $\\xi_{AB} = \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}$. As a result, we have\n\\begin{align}\n\\exc{z_1^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1^2} \\leq f(B|\\cdot)^2\\frac{(2 \\ln 2) \\log d_{A}}{n}.\n\\end{align}\nBut note we can also choose measurements $\\cM_{A}$ and $\\cM_{B}$ achieving $f(A,B)$ in \\eqref{eq:def-f1}. In this case,\n\\begin{align} \n\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1^2 &\\leq f(A,B)^2\\| (\\cM_A \\otimes \\cM_{B})(\\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}) \\|_1^2 \\\\\n&\\leq f(A,B)^2 \\| (\\cI_A \\otimes \\cM_{B})(\\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}) \\|_1^2 \\\\\n&= f(A,B)^2 \\| \\rho_{AZ_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m} \\|_1^2 ,\n\\end{align}\nwhere we used the fact that the trace norm cannot increase when applying the quantum channel $\\cM_{A}$ \\cite[Theorem 3.39]{watrous-book}. As a result, we get\n\\begin{align}\n\\exc{z_1^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1^2} \\leq f(A,B)^2 \\frac{(2 \\ln 2) \\log d_{A}}{n}.\n\\end{align}\nNow, using the convexity of the square function, we get\n\\begin{align}\n\\exc{z_{1}^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1}&\\leq \\sqrt{\\exc{z_{1}^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|^2_1}} \\\\\n&\\leq  \\min\\big\\{f(A, B), f(B|\\cdot)\\big\\}\\sqrt{\\frac{(2 \\ln 2) \\log d_{A}}{n}}.\n\\end{align}\nThen, using the convexity of the norm and the fact that $\\exc{z_1^m}{\\rho_{AB_{m+1}|z_1^m}} = \\rho_{AB_{m+1}}$, we obtain\n\\begin{align}\n\\left\\| \\rho_{AB_{m+1}} - \\exc{z_1^m}{\\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}} \\right\\|_1 &\\leq \\min\\big\\{f(A,B), f(B|\\cdot)\\big\\}\\sqrt{\\frac{(2 \\ln 2) \\log d_{A}}{n}}.\n\\end{align}\nThe state $\\exc{z_1^m}{\\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}}$ corresponds to our candidate mixture of product states. It now remains to show that all the states in the mixture satisfy the linear constraints. Indeed we have for any $z_1^m$, writing $M_{z}$ for matrices of the measurement $\\cM_{B}$,\n\\begin{align}\n\\Lambda_{A \\to C_{A}}(\\rho_{A|z_1^m})\n&= \\frac{\\tr_{B_1^m}\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m}) \\Lambda_{A \\to C_{A}}(\\rho_{A B_1^m})\\Big]}{\\tr\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m})\\rho_{A B_1^m}\\Big]} \\\\\n&= \\frac{\\tr_{B_1^m}\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m}) (X_{C_A} \\otimes \\rho_{B_1^m}) \\Big]}{\\tr\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m})\\rho_{A B_1^m}\\Big]} \\\\\n&= X_{C_A},\n\\end{align}\nand similarly\n\\begin{align}\n\\Gamma_{B \\to C_{B}}(\\rho_{B_{m+1}|z_1^m})\n&= \\frac{\\tr_{B_1^m}\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{C_{B}} ) \\Gamma_{B_{m+1} \\to C_{B}}(\\rho_{B_1^{m+1}})\\Big]}{\\tr\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{B_{m+1}} )\\rho_{B_1^{m+1}}\\Big]} \\\\\n&= \\frac{\\tr_{B_1^m}\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{B_{m+1}} ) (\\rho_{B_1 \\cdots B_{m}} \\otimes Y_{C_B})\\Big]}{\\tr\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{B_{m+1}} )\\rho_{B_1^{m+1}}\\Big]} \\\\\n&= Y_{C_B}.\n\\end{align}\n\\end{proof}","start":33847,"end":37993},{"raw_tex":"\\begin{proof}\nWe have by construction $0 \\leq \\mathrm{SDP}_n - Q $ and the remaining inequality arises from\n\\begin{align}\n\\mathrm{Tr}\\left[G_{AB} W_{AB_1}\\right]&=\\mathrm{Tr}\\left[G_{AB} (W_{A} \\otimes W_{B})\\right]+\\mathrm{Tr}\\left[G_{AB}\\left(W_{AB_1} - W_{A} \\otimes W_{B}\\right)\\right] \\\\\n& \\leq \\mathrm{Tr}\\left[G_{AB} (W_{A} \\otimes W_{B})\\right]+\\| G_{AB} \\|_\\infty \\cdot \\|W_{AB_1} - W_{A} \\otimes W_{B} \\|_1 \\\\\n&\\leq\\mathrm{Tr}\\left[G_{AB} (W_{A} \\otimes W_{B})\\right]+\\frac{\\mathrm{poly}(d)}{\\sqrt{n}},\n\\end{align}\nwhere we used H\\\"older's inequality and the de Finetti argument as in Theorem \\ref{thm:main-deFinetti}.\n\\end{proof}","start":49034,"end":49674},{"raw_tex":"\\begin{lemma}\\label{lem:ic-meas-side-info}\nConsider a state two-design on $B$, i.e., a set of rank-one projectors $\\{P_z\\}_{z \\in \\{1, \\dots, t\\}}$ such that $\\frac{1}{t} \\sum_{z=1}^t P_z \\otimes P_z = \\frac{2 P^{\\mathrm{sym}}}{d_B (d_B+1)}$, where $P^{\\mathrm{sym}}$ denotes the projector onto the symmetric subspace of $B \\otimes B$. Let $\\mathcal{M}_B$ be the measurement defined as\n\\begin{align}\n\\mathcal{M}_B(X) = \\sum_{z} \\frac{d_B}{t} \\cdot\\mathrm{Tr}\\big[P_z X\\big] |z\\rangle\\langle z|,\n\\end{align}\nand $\\xi_{AB}$ be a Hermitian matrix on $A\\otimes B$. Then, we have that\n\\begin{align}\n\\| (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\|_1\\geq \\frac{1}{d_B^2 (d_B+1)} \\| \\xi_{AB} \\|_{1}.\n\\end{align}\n\\end{lemma}","start":124046,"end":124769},{"raw_tex":"\\begin{proof}\nFor any full rank quantum state $\\sigma_{A}$, we have by a H\\\"older type inequality for $\\sigma$-weighted Schatten norms that (see, e.g., \\cite{OLKIEWICZ1999246} or \\cite{beigi13})\n\\begin{align}\n\\| (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\|_1 \\geq  \\frac{\\left\\| \\sigma_{A}^{-1/4} (\\mathcal{I}_{A} \\otimes \\mathcal{M})(\\xi_{AB}) \\sigma_{A}^{-1/4} \\right\\|^2_{2}}{\\left\\| \\sigma_{A}^{-1/2} (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\sigma_{A}^{-1/2} \\right\\|_{\\infty}}.\n\\end{align}\nFor example, the above inequality can be obtained using \\cite[Corollary 3]{beigi13} with the operator $\\sigma_{A}^{-1/2} (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\sigma_{A}^{-1/2}$, weight $\\sigma$, and $p_\\theta = 2$, $\\theta = 1/2$, $p_0 = 1$, $p_1 = \\infty$. We note that this particular H\\\"older type inequality for $\\sigma$-weighted norms is elementary and follows easily from the usual H\\\"older inequality, but one way of potentially improving the dimension dependence in Lemma~\\ref{lem:ic-meas-side-info} might be to use another H\\\"older inequality, in particular the $(1,4)$ inequality.\n\nHenceforth, we abbreviate $d\\equiv d_B$. To further bound the numerator, letting $\\tilde{\\xi}_{AB}:=\\sigma_{A}^{-1/4} \\xi_{AB} \\sigma_{A}^{-1/4}$ we get\n\\begin{align}\n\\| (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\tilde{\\xi}_{AB}) \\|^2_{2}\n&= \\left\\| \\sum_{z} \\frac{d}{t} |z\\rangle \\langle z| \\otimes  \\mathrm{Tr}_{B}\\left[(1_{A} \\otimes P_z) \\tilde{\\xi}_{AB}\\right] \\right\\|_2^2 \\\\\n&= \\sum_{z}  \\frac{d^2}{t^2} \\mathrm{Tr}\\left[\\mathrm{Tr}_{B}\\left[(1_{A} \\otimes P_z) \\tilde{\\xi}_{AB}\\right] \\otimes \\mathrm{Tr}_{\\bar{B}}\\left[(1_{\\bar{A}} \\otimes P_z) \\tilde{\\xi}_{\\bar{A}\\bar{B}}\\right]^{\\dagger} F_{A\\bar{A}} \\right] \\\\\n&= \\sum_{z} \\frac{d^2}{t^2} \\mathrm{Tr}\\left[\\left(\\left((1_{A} \\otimes P_z) \\tilde{\\xi}_{AB}\\right) \\otimes \\left((1_{\\bar{A}} \\otimes P_z) \\tilde{\\xi}_{\\bar{A}\\bar{B}}\\right)^{\\dagger}\\right)\\left(F_{A\\bar{A}} \\otimes1_{B\\bar{B}}\\right)\\right] \\\\\n&= \\frac{d^2}{t^2} \\mathrm{Tr}\\left[\\left(\\tilde{\\xi}_{AB} \\otimes \\tilde{\\xi}_{\\bar{A}\\bar{B}}^{\\dagger}\\right)\\left(\\sum_{z} (1_{A\\bar{A}} \\otimes P_z \\otimes P_{z})\\right)\\left(F_{A\\bar{A}} \\otimes1_{B \\bar{B}}\\right)\\right] \\\\\n&= \\frac{1}{t} \\frac{d^2}{d(d+1)} \\mathrm{Tr}\\left[\\left(\\tilde{\\xi}_{AB} \\otimes \\tilde{\\xi}_{\\bar{A}\\bar{B}}^{\\dagger}\\right) (1_{A\\bar{A}} \\otimes (1_{B\\bar{B}} + F_{B\\bar{B}}))\\left(F_{A\\bar{A}} \\otimes1_{B \\bar{B}}\\right)\\right] \\\\\n&= \\frac{1}{t} \\frac{d^2}{d(d+1)} \\Big(\\underbrace{\\mathrm{Tr}\\left[\\tilde{\\xi}_{A} \\tilde{\\xi}_{A}^{\\dagger}\\right]}_{\\geq 0} + \\underbrace{\\mathrm{Tr}\\left[\\tilde{\\xi}_{AB} \\tilde{\\xi}_{AB}^{\\dagger}\\right]}_{=\\left\\|\\tilde{\\xi}_{AB}\\right\\|_2^2} \\Big) \\\\\n&\\geq \\frac{1}{t}\\frac{d^2}{d^2 (d+1)} \\| \\xi_{AB} \\|^2_{1} ,\n\\end{align}\nwhere $F$ denotes the swap operator (as defined in Section \\ref{sec:notation}) and in the last step used the H\\\"older inequality (see, e.g., \\cite{bhatia97})\n\\begin{align}\n\\left\\|\\xi_{AB}\\right\\|_1=\\left\\|\\sigma^{1/4}\\sigma^{-1/4}\\xi_{AB}\\sigma^{-1/4}\\sigma^{1/4}\\right\\|_1\\leq\\left\\|\\sigma^{1/4}\\otimes1_B\\right\\|_4\\left\\|\\sigma_A^{-1/4}\\xi_{AB}\\sigma_A^{-1/4}\\right\\|_2\\left\\|\\sigma^{1/4}\\otimes1_B\\right\\|_4\\leq\\sqrt{d}\\left\\|\\tilde{\\xi}_{AB}\\right\\|_2.\n\\end{align}\nFor further bounding the denominator we write\n\\begin{align}\n\\left\\| \\sigma_{A}^{-1/2} (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\sigma_{A}^{-1/2} \\right\\|_\\infty\n&= \\max_{z} \\frac{d}{t} \\left\\| \\mathrm{Tr}_{B}\\left[(1_{A} \\otimes P_z) \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2}\\right]\\right\\|_\\infty \\\\\n&\\leq \\frac{d}{t} \\max_{\\ket{\\phi}_{A}, \\ket{\\psi}_{B}} \\bra{\\phi}_{A} \\otimes \\bra{\\psi}_{B} \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2} \\ket{\\phi}_{A} \\otimes \\ket{\\psi}_{B} \\\\\n&\\leq \\frac{d}{t} \\left\\| \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2} \\right\\|_\\infty,\n\\end{align}\nwhere we used the fact that $P_z$ is a rank 1 projector. Now, observe that for any $\\xi_{AB}$, there exists a $\\sigma_A$ of unit trace such that\n\\begin{align}\n\\frac{\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}}}{\\| \\xi_{AB} \\|_{1}} \\preceq d\\cdot\\sigma_{A} \\otimes1_{B}.\n\\end{align}\nThis just follows from, e.g., \\cite[Lemma B.6]{bertachristandl11}, where it is shown that we can in fact choose\\footnote{By a continuity argument $\\sigma_A$ can be assumed to have full rank.}\n\\begin{align}\n\\sigma_{A} = \\| \\xi_{AB} \\|_{1}^{-1}\\cdot\\mathrm{Tr}_{B}\\left[\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}}\\right].\n\\end{align}\nAs a result, we have\n\\begin{align}\n\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}} \\preceq d\\| \\xi_{AB} \\|_{1}\\cdot\\sigma_{A} \\otimes1_{B}.\n\\end{align}\nAs $\\xi_{AB}$ is Hermitian, we can decompose it into the positive and negative part $\\xi_{AB} = P - Q$ with $P$ and $Q$ positive semidefinite and $PQ = 0$, then $\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}} = P + Q$ and so $-\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}} \\preceq \\xi_{AB} \\preceq \\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}}$. Thus, we get\n\\begin{align}\n- d_{B} \\| \\xi_{AB} \\|_1\\cdot\\sigma_{A} \\otimes1_{B} \\preceq \\xi_{AB} \\preceq d_{B} \\| \\xi_{AB} \\|_1 \\cdot\\sigma_{A} \\otimes1_{B},\n\\end{align}\nand we find $\\| \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2} \\|_{\\infty} \\leq d\\| \\xi_{AB} \\|_1$. This concludes the proof.\n\\end{proof}","start":124988,"end":130190}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["original_program"],"declared_external":false,"source_only":false,"title":"受约束双线性优化","source_units":[{"raw_tex":"To discuss our approach, we first rewrite~\\eqref{eq:form} by defining $G_{AB}= (\\Pi_{A \\to D}^{\\dagger} \\otimes \\Pi_{B \\to E}^{\\dagger})(H)$, where $\\Pi^{\\dagger}$ denotes the adjoint map of $\\Pi$ in the Hilbert-Schmidt inner product. This leads to the form\n\\begin{align}\\label{original_program}\nQ=\\max &\\quad \\mathrm{Tr}\\big[G_{AB} (W_{A} \\otimes W_{B})\\big]\\\\\ns.t. &\\quad W_{A} \\succeq 0,\\quad W_{B} \\succeq 0,\\quad\\mathrm{Tr}[W_A] = \\mathrm{Tr}[W_B] = 1\\\\\n&\\quad\\Lambda_{A\\to C_A}\\left(W_{A}\\right)=X_{C_A},\\quad\\Gamma_{B\\to C_B}\\left(W_{B}\\right)=Y_{C_B}, \n\\end{align}\nwhere $G_{AB}\\in\\mathcal{S}_A\\otimes\\mathcal{S}_B$, $\\Lambda_{A\\to C_A}:\\mathcal{S}_A \\to \\mathcal{S}_{C_A}$ and $\\Gamma_{B\\to C_B}:\\mathcal{S}_B \\to \\mathcal{S}_{C_B}$ denote linear maps, and $X_{C_A}\\in\\mathcal{S}_{C_A}$ and $Y_{C_B}\\in\\mathcal{S}_{C_B}$ are the matrices defining $\\mathcal{A}_{A}$ and $\\mathcal{A}_B$ as the affine subspaces associated with the kernels of the linear maps $\\Lambda_{A\\to C_A}$ and $\\Gamma_{B\\to C_B}$, respectively.\n","start":7954,"end":8979}]},{"id":"S002","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:transformed_program"],"declared_external":false,"source_only":false,"title":"凸包优化等价性","source_units":[{"raw_tex":"Now, by the linearity of the objective function we can equivalently optimise over the convex hull of feasible points\n\\begin{align}\\label{eq:transformed_program}\nQ=\\max &\\quad \\mathrm{Tr}\\left[G_{AB}\\left(\\sum_{i\\in I}p_iW_A^i \\otimes W_B^i\\right)\\right]\\\\\ns.t. &\\quad p_i\\geq0,\\quad W_A^i \\succeq 0,\\quad W_B^i \\succeq 0,\\quad\\mathrm{Tr}\\left[W_A^i\\right] = \\mathrm{Tr}\\left[W_B^i\\right] = 1\\quad\\forall i\\in I,\\quad\\sum_{i\\in I}p_i=1\\\\\n&\\quad\\Lambda_{A\\to C_A}\\left(W_A^i\\right)=X_{C_A},\\quad\\Gamma_{B\\to C_B}\\left(W_B^i\\right)=Y_{C_B}\\quad\\forall i\\in I.\n\\end{align}\n","start":8979,"end":9548}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"可分态集合","source_units":[{"raw_tex":"That is, in the language of quantum information theory we are maximizing over a subset of the so-called {\\it separable quantum states}\\,---\\,where the latter is defined on $A\\otimes B$ as\n\\begin{align}\n\\text{Sep}(A:B)=\\left\\{\\sum_{i\\in I}p_iW_A^i \\otimes W_B^i: W_A^i \\succeq 0, W_B^i \\succeq 0, \\mathrm{Tr}\\left[W_A^i\\right]=\\mathrm{Tr}\\left[W_B^i\\right] = 1,p_i\\geq0,\\sum_{i\\in I}p_i=1\\right\\}.\n\\end{align}\nRecall that matrices $W_A\\in\\mathcal{S}^+_A$ are called quantum states on system $A$\\,---\\,and similarly for bipartite states on $A\\otimes B$.\n","start":9548,"end":10100}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"系统与量子态","source_units":[{"raw_tex":"In the following, we introduce some notation that is standard in quantum information theory. A $d_A$-dimensional quantum system (or in short {\\it system}) is given by an inner product space $\\mathbb{C}^{d_A}$ and denoted by $A$. {\\it Quantum states} (or in short {\\it states}) on $A$ are matrices\\footnote{Here and henceforth we use the symbol $:=$ as {\\it equal by definition}.}\n\\begin{align}\n\\text{$W_A\\in\\mathcal{S}_A:=\\mathbb{C}^{d_A\\times d_A}$ with $\\mathrm{Tr}[W_A]=1$ and $W_A\\succeq0$,}\n\\end{align}\nwhere $\\succeq$ denotes the operator (Loewner) order. Quantum states of rank one are called {\\it pure} and can be written as $W_A=\\ket{\\psi}\\bra{\\psi}_A$, where $\\ket{\\psi}_A\\in\\mathbb{C}^{d_A}$ and $\\ket{\\psi}\\bra{\\psi}_A\\in\\mathcal{S}_A$ denotes the rank-one projector on the vector $\\ket{\\psi}_A$.\n\n","start":17076,"end":17886}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"复合系统与积态","source_units":[{"raw_tex":"A {\\it bipartite system} $AB:=A\\otimes B$ is given by an inner product space $\\mathbb{C}^{d_A}\\otimes\\mathbb{C}^{d_B}$, where $\\otimes$ denotes the Kronecker tensor product. Correspondingly, states on $AB$ are matrices $W_{AB}\\in\\mathcal{S}_A\\otimes\\mathcal{S}_B$ with $\\mathrm{Tr}[W_{AB}]=1$ and $W_{AB}\\succeq0$. {\\it Separable states} are states on $AB$ that are in the convex hull of product states $W_A\\otimes W_B$, with $W_A$ and $W_B$ states on $A$ and $B$, respectively. The {\\it maximally entangled state} $\\Phi_{AB}:=\\ket{\\Phi}\\bra{\\Phi}_{AB}$ on $AB$ for $d:=d_A=d_B$ is not separable and defined via\n","start":17886,"end":18498}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"最大纠缠态背景","source_units":[{"raw_tex":"A {\\it bipartite system} $AB:=A\\otimes B$ is given by an inner product space $\\mathbb{C}^{d_A}\\otimes\\mathbb{C}^{d_B}$, where $\\otimes$ denotes the Kronecker tensor product. Correspondingly, states on $AB$ are matrices $W_{AB}\\in\\mathcal{S}_A\\otimes\\mathcal{S}_B$ with $\\mathrm{Tr}[W_{AB}]=1$ and $W_{AB}\\succeq0$. {\\it Separable states} are states on $AB$ that are in the convex hull of product states $W_A\\otimes W_B$, with $W_A$ and $W_B$ states on $A$ and $B$, respectively. The {\\it maximally entangled state} $\\Phi_{AB}:=\\ket{\\Phi}\\bra{\\Phi}_{AB}$ on $AB$ for $d:=d_A=d_B$ is not separable and defined via\n\\begin{align}\n\\text{$\\ket{\\Phi}_{AB}:=\\frac{1}{\\sqrt{d}}\\sum_{x=1}^d\\ket{x}_A\\otimes\\ket{x}_B$ for some orthonormal basis $\\{\\ket{x}\\}_{x=1}^d$ of $\\mathbb{C}^d$.}\n\\end{align}\n","start":17886,"end":18674}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"交换算子","source_units":[{"raw_tex":"The {\\it swap operator} $F_{AB}$ on $A\\otimes B$ exchanges the two quantum systems, i.e., $F_{AB}(W_A\\otimes W_B)=W_B\\otimes W_A$ for every state $W_A$ and $W_B$ on $A$ and $B$, respectively. {\\it Classical-quantum states} are bipartite states that can be written in the form\n","start":18674,"end":18950}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"经典量子态","source_units":[{"raw_tex":"The {\\it swap operator} $F_{AB}$ on $A\\otimes B$ exchanges the two quantum systems, i.e., $F_{AB}(W_A\\otimes W_B)=W_B\\otimes W_A$ for every state $W_A$ and $W_B$ on $A$ and $B$, respectively. {\\it Classical-quantum states} are bipartite states that can be written in the form\n\\begin{align}\nW_{XB}=\\sum_{x=1}^{d_X}p_x\\ket{x}\\bra{x}_X\\otimes W_B^x\n\\end{align}\nfor a probability distribution $\\{p_x\\}_{x=1}^{d_A}$, an orthonormal basis $\\{\\ket{x}_X\\}_{x=1}^{d_X}$ of $\\mathbb{C}^{d_X}$, and states $W_B^x$ on $B$ for $x=1,\\dots,d_X$. We refer to $X$ as the classical part of the bipartite classical-quantum system $XB$.\n\n","start":18674,"end":19292}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"量子信道","source_units":[{"raw_tex":"{\\it Quantum channels} (or in short {\\it channels}) are linear maps $\\mathcal{N}_{A\\to B}:\\mathcal{S}_A\\to\\mathcal{S}_B$ that are trace preserving and completely positive\\footnote{A linear map $\\mathcal{N}_{A\\to B}:\\mathcal{S}_A\\to\\mathcal{S}_B$ is said to be completely positive if $\\mathcal{N}_{A\\to B}\\otimes \\mathcal{I}_C$ is a positive map for every quantum system $C$, where $\\mathcal{I}_C$ denotes the identity map on $\\mathcal{S}_C$} (cp). In particular, they map states from the input system $A$ to states on the output system $B$. We often abbreviate bipartite channels $\\mathcal{I}_A\\otimes\\mathcal{N}_B$ that act trivially on the $A$-system as $\\mathcal{N}_B$, where $\\mathcal{I}_A$ denotes the identity channel on $\\mathcal{S}_A$. The partial trace $\\mathrm{Tr}_B[\\cdot]$ is a channel from $AB$ to $A$ defined via\n","start":19292,"end":20119}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"偏迹与约化态","source_units":[{"raw_tex":"{\\it Quantum channels} (or in short {\\it channels}) are linear maps $\\mathcal{N}_{A\\to B}:\\mathcal{S}_A\\to\\mathcal{S}_B$ that are trace preserving and completely positive\\footnote{A linear map $\\mathcal{N}_{A\\to B}:\\mathcal{S}_A\\to\\mathcal{S}_B$ is said to be completely positive if $\\mathcal{N}_{A\\to B}\\otimes \\mathcal{I}_C$ is a positive map for every quantum system $C$, where $\\mathcal{I}_C$ denotes the identity map on $\\mathcal{S}_C$} (cp). In particular, they map states from the input system $A$ to states on the output system $B$. We often abbreviate bipartite channels $\\mathcal{I}_A\\otimes\\mathcal{N}_B$ that act trivially on the $A$-system as $\\mathcal{N}_B$, where $\\mathcal{I}_A$ denotes the identity channel on $\\mathcal{S}_A$. The partial trace $\\mathrm{Tr}_B[\\cdot]$ is a channel from $AB$ to $A$ defined via\n\\begin{align}\n\\mathrm{Tr}_B[\\cdot]:=\\sum_{x=1}^{d_B}\\big(1_A\\otimes\\bra{x}_B\\big)(\\cdot)\\big(1_A\\otimes\\ket{x}_B\\big),\n\\end{align}\nwhere $1_A$ denotes the identity matrix on $A$, and $\\{\\ket{x}\\}_{x=1}^{d_B}$ an orthonormal basis of $\\mathbb{C}^{d_B}$. For bipartite states $W_{AB}$, we write $W_A=\\mathrm{Tr}_B[W_{AB}]$ for the reduced state on $A$. {\\it Quantum measurements} (or in short {\\it measurements}) are a special case of channels that can be written in the form\n","start":19292,"end":20593}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"测量信道","source_units":[{"raw_tex":"where $1_A$ denotes the identity matrix on $A$, and $\\{\\ket{x}\\}_{x=1}^{d_B}$ an orthonormal basis of $\\mathbb{C}^{d_B}$. For bipartite states $W_{AB}$, we write $W_A=\\mathrm{Tr}_B[W_{AB}]$ for the reduced state on $A$. {\\it Quantum measurements} (or in short {\\it measurements}) are a special case of channels that can be written in the form\n\\begin{align}\n\\mathcal{M}_{A\\to I}(\\cdot):=\\sum_{i\\in I}\\mathrm{Tr}\\left[M_A^i(\\cdot)\\right]\\ket{i}\\bra{i}_I\n\\end{align}\nwith an orthonormal basis $\\{\\ket{i}_I\\}_{i\\in I}$ and $M_A^i\\succeq0$ $\\forall i\\in I$ with $\\sum_{i\\in I}M_A^i=1_A$.\n","start":20250,"end":20833}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"迹范数距离","source_units":[{"raw_tex":"The distance between states is quantified by the metric induced from the Schatten one-norm $\\|X\\|_1:=\\mathrm{Tr}\\left[|X|\\right]$. The distance between channels is quantified by the metric induced from the Diamond norm\n","start":21568,"end":21787}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"钻石范数背景","source_units":[{"raw_tex":"The distance between states is quantified by the metric induced from the Schatten one-norm $\\|X\\|_1:=\\mathrm{Tr}\\left[|X|\\right]$. The distance between channels is quantified by the metric induced from the Diamond norm\n\\begin{align}\n\\|\\mathcal{N}_{A\\to B}\\|_{\\Diamond}:=\\sup_{\\|X\\|_1\\leq1}\\|(\\mathcal{N}_{A\\to B}\\otimes\\mathcal{I}_A)(X_{AA})\\|_1.\n\\end{align}\n\n","start":21568,"end":21928}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"多体置换对称性","source_units":[{"raw_tex":"A multipartite state $W_{AB_1^n}$ on $AB_1^n\\equiv AB_1\\cdots B_n$ is called {\\it symmetric with respect to $A$} if\n\\begin{align}\n(\\mathcal{I}_A\\otimes\\mathcal{U}_{B_1^n}^\\pi)(W_{AB_1^n})=W_{AB_1^n}\\quad\\forall\\pi\\in\\mathfrak{S}_n,\n\\end{align}\nwhere $\\mathfrak{S}_n$ denotes the symmetric group of $n$ elements and\n\\begin{align}\n\\mathcal{U}_{B_1^n}^\\pi(W_{B_1}\\otimes\\cdots\\otimes W_{B_n}):=W_{B_{\\pi^{-1}(1)}}\\otimes\\cdots\\otimes W_{B_{\\pi^{-1}(n)}}.\n\\end{align}\n","start":21928,"end":22392}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"对称可扩张性","source_units":[{"raw_tex":"A bipartite state $W_{AB}$ is called {\\it $n$-extendable} if there exists a multipartite extension $W_{AB_1^n}$, i.e., $\\mathrm{Tr}_{B_2^n}(W_{AB_1^n})=W_{AB}$, that is symmetric with respect to $A$.\n","start":22392,"end":22592}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"量子相对熵","source_units":[{"raw_tex":"The starting point for our proof technique is the use of the chain rule of the conditional mutual information, first used in this context in~\\cite{brandao11} and further exploited in~\\cite{brandao13b}. More precisely, we will use the quantum relative entropy defined as\n\\begin{align}\nD(\\rho\\|\\sigma):=\n\\begin{cases} \\tr(\\rho \\log \\rho) - \\tr(\\rho \\log \\sigma) & \\mbox{if } support(\\rho)\\subseteq support(\\sigma) \\\\ \\infty & \\mbox{otherwise} \\end{cases},\n\\end{align}\nwhere $\\rho$ and $\\sigma$ are quantum states and the logarithm is taken with respect to the basis two. Recall that the support of an operator is defined as the orthogonal complement of its kernel. The following lemma, which can be found in~\\cite{brandao13b}, says that if some classical systems $Z_1^n$ are symmetric with respect to $A$, then conditioning on $Z_1^m$ for some value of $m$ breaks the correlations between $A$ and $Z_{m+1}$. Before stating the lemma, we introduce notation that will be used throughout the section. For a state $\\rho_{A Z}$ with a classical $Z$-system, we write\n","start":26930,"end":27989}]},{"id":"S017","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"条件态与期望","source_units":[{"raw_tex":"where $\\rho$ and $\\sigma$ are quantum states and the logarithm is taken with respect to the basis two. Recall that the support of an operator is defined as the orthogonal complement of its kernel. The following lemma, which can be found in~\\cite{brandao13b}, says that if some classical systems $Z_1^n$ are symmetric with respect to $A$, then conditioning on $Z_1^m$ for some value of $m$ breaks the correlations between $A$ and $Z_{m+1}$. Before stating the lemma, we introduce notation that will be used throughout the section. For a state $\\rho_{A Z}$ with a classical $Z$-system, we write\n\\begin{align}\n\\rho_{A|z}:= \\frac{\\mathrm{Tr}_{Z}\\Big[\\rho_{AZ} \\left(1_{A} \\otimes \\ket{z}\\bra{z}\\right)\\Big]}{\\mathrm{Tr}\\Big[\\rho_{AZ} \\left(1_{A} \\otimes \\ket{z}\\bra{z}\\right)\\Big]}.\n\\end{align}\nWe simply write $\\exc{z_{1}^{m}}{\\cdot}$ for the expectation over the choices of $z_1^m$ and the distribution will be clear from the context.\n","start":27396,"end":28329}]},{"id":"S018","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"测量失真动机","source_units":[{"raw_tex":"To prove the de Finetti theorem, we will crucially make use of so-called informationally complete measurements for which the loss in distinguishability, or {\\it distortion}, can be bounded.\n\n","start":29993,"end":30184}]},{"id":"S019","semantic_type":"lemma","kind":"lemma","labels":["lem:product-ic-meas"],"declared_external":true,"declaration_text":["原始先决条件明确许可 B0039，出处为 brandao13b, Lemma 14；本文没有提供其证明。"],"source_only":false,"title":"乘积测量失真界","source_units":[{"raw_tex":"\\begin{lemma}{\\cite[Lemma 14]{brandao13b}}\\label{lem:product-ic-meas}\nThere exist a product measurement $\\mathcal{M}_A\\otimes\\mathcal{M}_B$ with finitely many outcomes such that for any Hermitian and traceless matrix $\\xi_{AB}$ on $A\\otimes B$, we have \n\\begin{align}\n\\|(\\mathcal{M}_{A} \\otimes \\mathcal{M}_{B})(\\xi_{AB})\\|_1 \\geq \\frac{1}{18\\sqrt{d_A d_B}}\\| \\xi_{AB}\\|_{1}.\n\\end{align}\n\\end{lemma}\n","start":30184,"end":30584}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":["eq:def-f1"],"declared_external":false,"source_only":false,"title":"乘积最小失真","source_units":[{"raw_tex":"\nThis \\cite[Lemma 14]{brandao13b} follows from the methods in \\cite{lancien13}. More generally, we define the {\\it minimal distortion} for the bipartite system $A \\otimes B$ as\n\\begin{align}\\label{eq:def-f1}\nf(A, B):=\\inf_{\\cM_{A}, \\cM_{B}}\\max_{\\substack{\\xi_{AB}^{\\dagger} = \\xi_{AB}\\\\ \\xi_A = 0, \\xi_B = 0}} \\frac{\\| \\xi_{AB} \\|_1}{\\| (\\cM_{A} \\otimes \\cM_B)(\\xi_{AB})\\|_1},\n\\end{align}\nwhere the infimum is over all product measurements on $AB$. In this notation, Lemma~\\ref{lem:product-ic-meas} shows that\n","start":30584,"end":31095},{"raw_tex":"Note that in the definition of $f(A,B)$ we restricted the maximization to matrices satisfying $\\xi_{A} = 0$ and $\\xi_{B} = 0$ because this is sufficient for us.\n\n","start":31153,"end":31315}]},{"id":"S021","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"乘积失真的维数界","source_units":[{"raw_tex":"where the infimum is over all product measurements on $AB$. In this notation, Lemma~\\ref{lem:product-ic-meas} shows that\n\\begin{align}\nf(A, B) \\leq 18 \\sqrt{d_A d_B}.\n\\end{align}\nNote that in the definition of $f(A,B)$ we restricted the maximization to matrices satisfying $\\xi_{A} = 0$ and $\\xi_{B} = 0$ because this is sufficient for us.\n","start":30974,"end":31314}]},{"id":"S022","semantic_type":"definition","kind":"definition","labels":["eq:def-f2"],"declared_external":false,"source_only":false,"title":"带旁信息最小失真","source_units":[{"raw_tex":"A drawback of Lemma~\\ref{lem:product-ic-meas} is that the distortion depends on the dimension $d_A$. More generally, we define the {\\it minimal distortion with side information} for a system $B$ as\n\\begin{align}\\label{eq:def-f2}\nf(B|\\cdot):=\\inf_{\\cM_B} \\sup_{\\substack{\\xi_{AB}^\\dagger=\\xi_{AB}\\\\ \\xi_{A} = 0, \\xi_B = 0}} \\frac{\\| \\xi_{AB} \\|_1}{\\| (\\cI_{A} \\otimes \\cM_B)(\\xi_{AB})\\|_1},\n\\end{align}\nwhere the infimum is over all measurements on $B$ and the supremum is over all finite-dimensional systems $A$. In Lemma~\\ref{lem:ic-meas-side-info} we give an elementary proof that\n","start":31315,"end":31898}]},{"id":"S023","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"旁信息失真的初等界","source_units":[{"raw_tex":"where the infimum is over all measurements on $B$ and the supremum is over all finite-dimensional systems $A$. In Lemma~\\ref{lem:ic-meas-side-info} we give an elementary proof that\n\\begin{align}\nf(B|\\cdot) \\leq d_B^2 (d_B+1)\n\\end{align}\nusing state two-designs and properties of weighted non-commutative $L_p$-spaces. In fact, after completion of our work we realised that methods from operator space theory even give the stronger bound\n","start":31717,"end":32154}]},{"id":"S024","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件列出的 brandao13c, Equation 66；本段没有包含该外部论证。"],"source_only":false,"title":"旁信息失真的较强界","source_units":[{"raw_tex":"using state two-designs and properties of weighted non-commutative $L_p$-spaces. In fact, after completion of our work we realised that methods from operator space theory even give the stronger bound\n\\begin{align}\nf(B|\\cdot) \\leq \\sqrt{18d_B^3},\n\\end{align}\nwhich is discussed in \\cite[Equation 66]{brandao13c}. We leave it as an open question to determine the exact dimensional dependence of the minimal distortion with side information.\n","start":31954,"end":32393}]},{"id":"S025","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件中的 scott08, Corollary 5.3，以及原文说明的酉两设计到态两设计的构造。"],"source_only":false,"title":"有限两设计存在性","source_units":[{"raw_tex":"We note that the existence of such two-designs is known for any dimension, see e.g., \\cite[Corollary 5.3]{scott08} for unitary two-designs and applying these unitaries to any fixed state leads to a state two-design.\n","start":124771,"end":124987}]},{"id":"S026","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":true,"title":"积态混合的对称扩张","source_units":[{"raw_tex":"Now, to approximate the set of separable states within the set of bipartite states is a ubiquitous but hard problem in quantum information theory (see, e.g., \\cite{Barak12}). Nevertheless, as realized in~\\cite{Doherty04} the set of separable states can be approximated by the sum-of-squares hierarchies of Lasserre~\\cite{Lasserre00} and Parrilo~\\cite{Parrilo2003}. This lead to the {\\it semidefinite programming hierarchy} of Doherty-Parrilo-Spedalieri (DPS), which is extensively employed to characterize quantum correlations in quantum information theory~\\cite{Doherty02}. The underlying idea of the DPS hierarchy is that separable states $W_{AB}=\\sum_i p_i W_A^i \\otimes W_B^i$ on $A\\otimes B$, where $\\{p_i\\}_{i\\in I}$ is a probability distribution, are {\\it $n$-extendible} to $W_{AB_1^n}=\\sum_i p_iW_A^i \\otimes (W_B^i)^{\\otimes n}$ on $A\\otimes B^{\\otimes n}$ for any $n$, such that we have for any permutation $\\pi$ that\\footnote{Here and henceforth we use the notation $B_i^j$ to denote the systems $B_i\\otimes\\cdots\\otimes B_j$, which should be interpreted as empty if $i>j$.}\n\\begin{align}\nW_{AB_1^n}=\\left(\\mathcal{I}_A\\otimes\\mathcal{U}_{B_1^n}^\\pi\\right)(W_{AB_1^n})\n\\end{align}\n","start":10101,"end":11294}]},{"id":"S027","semantic_type":"lemma","kind":"lemma","labels":["lem:conditioning-breaks-correlations"],"declared_external":false,"source_only":false,"title":"条件化消除相关性","source_units":[{"raw_tex":"\\begin{lemma}\\cite{brandao13b}\\label{lem:conditioning-breaks-correlations}\nLet $\\rho_{A Z_1^n}$ be a classical-quantum state with the $Z_1^n$-systems classical and $\\mathcal{U}^\\pi_{Z_1^n}(\\rho_{A Z_1^n})=\\rho_{A Z_1^n}$ for all $\\pi\\in\\mathfrak{S}_n$. Then, there exists $0\\leq m < n$ such that\n\\begin{align}\n\\exc{z_{1}^{m}}{D(\\rho_{AZ_{m+1}|z_1^m} \\| \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m})} &\\leq \\frac{\\log d_A}{n}\n\\end{align}\nas well as\n\\begin{align}\n\\exc{z_{1}^{m}}{\\| \\rho_{AZ_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m} \\|_1^2} &\\leq \\frac{(2 \\ln 2) \\log d_A}{n},\n\\end{align}\nwhere $\\ln(\\cdot)$ denotes the natural logarithm.\n\\end{lemma}","start":28330,"end":28995}]},{"id":"S028","semantic_type":"derived_fact","kind":"derived_fact","labels":["lem:conditioning-breaks-correlations"],"declared_external":false,"source_only":false,"title":"条件相对熵界","source_units":[{"raw_tex":"\\begin{lemma}\\cite{brandao13b}\\label{lem:conditioning-breaks-correlations}\nLet $\\rho_{A Z_1^n}$ be a classical-quantum state with the $Z_1^n$-systems classical and $\\mathcal{U}^\\pi_{Z_1^n}(\\rho_{A Z_1^n})=\\rho_{A Z_1^n}$ for all $\\pi\\in\\mathfrak{S}_n$. Then, there exists $0\\leq m < n$ such that\n\\begin{align}\n\\exc{z_{1}^{m}}{D(\\rho_{AZ_{m+1}|z_1^m} \\| \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m})} &\\leq \\frac{\\log d_A}{n}\n\\end{align}\n","start":28330,"end":28766}]},{"id":"S029","semantic_type":"derived_fact","kind":"derived_fact","labels":["lem:conditioning-breaks-correlations"],"declared_external":false,"source_only":false,"title":"条件迹范数平方界","source_units":[{"raw_tex":"\\begin{lemma}\\cite{brandao13b}\\label{lem:conditioning-breaks-correlations}\nLet $\\rho_{A Z_1^n}$ be a classical-quantum state with the $Z_1^n$-systems classical and $\\mathcal{U}^\\pi_{Z_1^n}(\\rho_{A Z_1^n})=\\rho_{A Z_1^n}$ for all $\\pi\\in\\mathfrak{S}_n$. Then, there exists $0\\leq m < n$ such that\n","start":28330,"end":28626},{"raw_tex":"as well as\n\\begin{align}\n\\exc{z_{1}^{m}}{\\| \\rho_{AZ_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m} \\|_1^2} &\\leq \\frac{(2 \\ln 2) \\log d_A}{n},\n\\end{align}\nwhere $\\ln(\\cdot)$ denotes the natural logarithm.\n","start":28766,"end":28984}]},{"id":"S030","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"消相关引理完整证明","source_units":[{"raw_tex":"\\begin{proof}\nFor the quantum mutual information we have $I\\left(A:Z_1^n\\right)_{\\rho}:=D(\\rho_{AZ_1^n}\\|\\rho_A\\otimes \\rho_{Z_1^n}) \\leq \\log d_A$ as well as (see, e.g., \\cite[Chapter 11]{nielsen00})\n\\begin{align}\nI\\left(A:Z_1^n\\right)_{\\rho}=\\sum_{m=0}^{n-1} I(A : Z_{m+1} | Z_1^m)_{\\rho}\n\\end{align}\nfor the quantum conditional mutual information $I(A : Z_{m+1} | Z_1^m)_\\rho:=I(A:Z_1^{m+1})_\\rho-I(A:Z_1^m)_\\rho$. As a result, there exists an $m \\in \\{0, \\cdots, n-1\\}$ such that $I(A : Z_{m+1} | Z_1^m)_{\\rho} \\leq \\frac{\\log d_A}{n}$, which implies\n\\begin{align}\n\\exc{z_1^m}{I(A:Z_{m+1})_{\\rho_{AZ_{m+1}|z_1^m}}} \\leq \\frac{\\log d_A}{n},\n\\end{align}\nwhere we used $I(A : Z_{m+1} | Z_1^m)_{\\rho}=\\exc{z_1^m}{I(A:Z_{m+1})_{\\rho_{AZ_{m+1}|z_1^m}}}$, which holds since the conditioning systems are classical. The second statement then follows directly from Pinsker's inequality $D(\\rho\\|\\sigma)\\geq\\frac{1}{2\\ln 2}\\left\\|\\rho-\\sigma\\right\\|_1^2$ \\cite[Theorem 5.38]{watrous-book}.\n\\end{proof}","start":28997,"end":29991}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"链式展开选择条件长度","source_units":[{"raw_tex":"\\begin{proof}\nFor the quantum mutual information we have $I\\left(A:Z_1^n\\right)_{\\rho}:=D(\\rho_{AZ_1^n}\\|\\rho_A\\otimes \\rho_{Z_1^n}) \\leq \\log d_A$ as well as (see, e.g., \\cite[Chapter 11]{nielsen00})\n\\begin{align}\nI\\left(A:Z_1^n\\right)_{\\rho}=\\sum_{m=0}^{n-1} I(A : Z_{m+1} | Z_1^m)_{\\rho}\n\\end{align}\nfor the quantum conditional mutual information $I(A : Z_{m+1} | Z_1^m)_\\rho:=I(A:Z_1^{m+1})_\\rho-I(A:Z_1^m)_\\rho$. As a result, there exists an $m \\in \\{0, \\cdots, n-1\\}$ such that $I(A : Z_{m+1} | Z_1^m)_{\\rho} \\leq \\frac{\\log d_A}{n}$, which implies\n","start":28997,"end":29552}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"经典条件化转为期望","source_units":[{"raw_tex":"for the quantum conditional mutual information $I(A : Z_{m+1} | Z_1^m)_\\rho:=I(A:Z_1^{m+1})_\\rho-I(A:Z_1^m)_\\rho$. As a result, there exists an $m \\in \\{0, \\cdots, n-1\\}$ such that $I(A : Z_{m+1} | Z_1^m)_{\\rho} \\leq \\frac{\\log d_A}{n}$, which implies\n\\begin{align}\n\\exc{z_1^m}{I(A:Z_{m+1})_{\\rho_{AZ_{m+1}|z_1^m}}} \\leq \\frac{\\log d_A}{n},\n\\end{align}\nwhere we used $I(A : Z_{m+1} | Z_1^m)_{\\rho}=\\exc{z_1^m}{I(A:Z_{m+1})_{\\rho_{AZ_{m+1}|z_1^m}}}$, which holds since the conditioning systems are classical. The second statement then follows directly from Pinsker's inequality $D(\\rho\\|\\sigma)\\geq\\frac{1}{2\\ln 2}\\left\\|\\rho-\\sigma\\right\\|_1^2$ \\cite[Theorem 5.38]{watrous-book}.\n","start":29300,"end":29980}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"Pinsker 转为迹范数","source_units":[{"raw_tex":"where we used $I(A : Z_{m+1} | Z_1^m)_{\\rho}=\\exc{z_1^m}{I(A:Z_{m+1})_{\\rho_{AZ_{m+1}|z_1^m}}}$, which holds since the conditioning systems are classical. The second statement then follows directly from Pinsker's inequality $D(\\rho\\|\\sigma)\\geq\\frac{1}{2\\ln 2}\\left\\|\\rho-\\sigma\\right\\|_1^2$ \\cite[Theorem 5.38]{watrous-book}.\n\\end{proof}","start":29653,"end":29991}]},{"id":"S034","semantic_type":"theorem","kind":"theorem","labels":["thm:main-deFinetti"],"declared_external":false,"source_only":false,"title":"带线性约束的 de Finetti 定理","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm:main-deFinetti}\nLet $\\rho_{AB_1^n}$ be a quantum state, $\\Lambda_{A\\to C_A},\\Gamma_{B\\to C_B}$ linear maps, and $X_{C_A},Y_{C_B}$ matrices such that\n\\begin{align}\n\\mathcal{U}_{B_1^n}^\\pi(\\rho_{AB_1^n})&=\\rho_{AB_1^n}\\;\\forall\\pi\\in\\mathfrak{S}_n\\qquad\\text{symmetric with respect to $A$}\\\\\n\\Lambda_{A\\to C_A}(\\rho_{AB_1^n})&=X_{C_A}\\otimes\\rho_{B_1^n}\\qquad\\quad\\;\\;\\text{linear constraint on $A$}\\\\\n\\Gamma_{B_n\\to C_B}(\\rho_{B_1^n})&=\\rho_{B_1^{n-1}}\\otimes Y_{C_B}\\qquad\\quad\\text{linear constraint on $B$}.\n\\end{align}\nThen, we have that\n\\begin{align}\n\\left\\|\\rho_{AB}-\\sum_{i\\in I}p_i\\sigma^i_{A}\\otimes\\omega^i_B\\right\\|_1\\leq&\\;\\min\\big\\{ f(A,B) , f(B|\\cdot) \\big\\}\\sqrt{ \\frac{(2 \\ln 2) \\log \\left(d_A\\right)}{n}}\n\\end{align}\nwith $\\{p_i\\}_{i\\in I}$ a probability distribution, $\\rho_{AB}=\\tr_{B_2^n}\\left[\\rho_{AB_1^n}\\right]$, and quantum states $\\sigma^i_A,\\omega_B^i$ such that for $i\\in I$:\n\\begin{align}\n\\Lambda_{A\\to C_A}\\left(\\sigma^i_A\\right)=X_{C_A}\\quad\\text{and}\\quad\\Gamma_{B\\to C_B}\\left(\\omega^i_B\\right)=Y_{C_B}.\n\\end{align}\nAs stated in Section~\\ref{sec:proof-tools}, we can, e.g., take $f(A,B)\\leq18\\sqrt{d_A d_B}$ or $f(B|\\cdot)\\leq\\sqrt{18d_B^3}$.\n\\end{theorem}","start":32631,"end":33845}]},{"id":"S035","semantic_type":"assumption","kind":"assumption","labels":["thm:main-deFinetti"],"declared_external":false,"source_only":false,"title":"受约束扩张的假设","source_units":[{"raw_tex":"Let $\\rho_{AB_1^n}$ be a quantum state, $\\Lambda_{A\\to C_A},\\Gamma_{B\\to C_B}$ linear maps, and $X_{C_A},Y_{C_B}$ matrices such that\n\\begin{align}\n\\mathcal{U}_{B_1^n}^\\pi(\\rho_{AB_1^n})&=\\rho_{AB_1^n}\\;\\forall\\pi\\in\\mathfrak{S}_n\\qquad\\text{symmetric with respect to $A$}\\\\\n\\Lambda_{A\\to C_A}(\\rho_{AB_1^n})&=X_{C_A}\\otimes\\rho_{B_1^n}\\qquad\\quad\\;\\;\\text{linear constraint on $A$}\\\\\n\\Gamma_{B_n\\to C_B}(\\rho_{B_1^n})&=\\rho_{B_1^{n-1}}\\otimes Y_{C_B}\\qquad\\quad\\text{linear constraint on $B$}.\n\\end{align}\n","start":32673,"end":33179}]},{"id":"S036","semantic_type":"derived_fact","kind":"derived_fact","labels":["thm:main-deFinetti"],"declared_external":false,"source_only":false,"title":"定理中的示例常数","source_units":[{"raw_tex":"As stated in Section~\\ref{sec:proof-tools}, we can, e.g., take $f(A,B)\\leq18\\sqrt{d_A d_B}$ or $f(B|\\cdot)\\leq\\sqrt{18d_B^3}$.\n\\end{theorem}","start":33705,"end":33845}]},{"id":"S037","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"受约束 de Finetti 完整证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $\\cM_{B}$ be a measurement of the $B$ system and call the outcome system $Z$. Consider the state $\\rho_{AZ_1^n}$ obtained by measuring all the $B$ systems with $\\cM_{B}$. This distribution is symmetric with respect to $A$ and so we can apply Lemma~\\ref{lem:conditioning-breaks-correlations}. We find that there exists an $m \\in \\{0, \\cdots, n-1\\}$ such that\n\\begin{align}\n\\exc{z_1^m}{\\| \\rho_{AZ_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m} \\|_1^2} \\leq \\frac{(2 \\ln 2) \\log d_{A}}{n}.\n\\end{align}\nNote that we have for any $z_1^m$, $\\rho_{AZ_{m+1}|z_1^m} = (\\cI_{A} \\otimes \\cM_{B})(\\rho_{AB_{m+1}|z_1^m})$ and correspondingly $\\rho_{Z_{m+1}|z_1^m} = \\cM_{B}(\\rho_{B_{m+1}|z_1^m})$. Now, we choose the measurement $\\cM_{B}$ to be as in Lemma~\\ref{lem:ic-meas-side-info} and achieving $f(B|\\cdot)$ in \\eqref{eq:def-f2}, we get that $\\| \\xi_{AB} \\|_1^2 \\leq f(B|\\cdot)^2 \\| (\\cI_{A} \\otimes \\cM_{B})(\\xi_{AB}) \\|_1^2$, where $\\xi_{AB} = \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}$. As a result, we have\n\\begin{align}\n\\exc{z_1^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1^2} \\leq f(B|\\cdot)^2\\frac{(2 \\ln 2) \\log d_{A}}{n}.\n\\end{align}\nBut note we can also choose measurements $\\cM_{A}$ and $\\cM_{B}$ achieving $f(A,B)$ in \\eqref{eq:def-f1}. In this case,\n\\begin{align} \n\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1^2 &\\leq f(A,B)^2\\| (\\cM_A \\otimes \\cM_{B})(\\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}) \\|_1^2 \\\\\n&\\leq f(A,B)^2 \\| (\\cI_A \\otimes \\cM_{B})(\\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}) \\|_1^2 \\\\\n&= f(A,B)^2 \\| \\rho_{AZ_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m} \\|_1^2 ,\n\\end{align}\nwhere we used the fact that the trace norm cannot increase when applying the quantum channel $\\cM_{A}$ \\cite[Theorem 3.39]{watrous-book}. As a result, we get\n\\begin{align}\n\\exc{z_1^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1^2} \\leq f(A,B)^2 \\frac{(2 \\ln 2) \\log d_{A}}{n}.\n\\end{align}\nNow, using the convexity of the square function, we get\n\\begin{align}\n\\exc{z_{1}^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1}&\\leq \\sqrt{\\exc{z_{1}^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|^2_1}} \\\\\n&\\leq  \\min\\big\\{f(A, B), f(B|\\cdot)\\big\\}\\sqrt{\\frac{(2 \\ln 2) \\log d_{A}}{n}}.\n\\end{align}\nThen, using the convexity of the norm and the fact that $\\exc{z_1^m}{\\rho_{AB_{m+1}|z_1^m}} = \\rho_{AB_{m+1}}$, we obtain\n\\begin{align}\n\\left\\| \\rho_{AB_{m+1}} - \\exc{z_1^m}{\\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}} \\right\\|_1 &\\leq \\min\\big\\{f(A,B), f(B|\\cdot)\\big\\}\\sqrt{\\frac{(2 \\ln 2) \\log d_{A}}{n}}.\n\\end{align}\nThe state $\\exc{z_1^m}{\\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}}$ corresponds to our candidate mixture of product states. It now remains to show that all the states in the mixture satisfy the linear constraints. Indeed we have for any $z_1^m$, writing $M_{z}$ for matrices of the measurement $\\cM_{B}$,\n\\begin{align}\n\\Lambda_{A \\to C_{A}}(\\rho_{A|z_1^m})\n&= \\frac{\\tr_{B_1^m}\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m}) \\Lambda_{A \\to C_{A}}(\\rho_{A B_1^m})\\Big]}{\\tr\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m})\\rho_{A B_1^m}\\Big]} \\\\\n&= \\frac{\\tr_{B_1^m}\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m}) (X_{C_A} \\otimes \\rho_{B_1^m}) \\Big]}{\\tr\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m})\\rho_{A B_1^m}\\Big]} \\\\\n&= X_{C_A},\n\\end{align}\nand similarly\n\\begin{align}\n\\Gamma_{B \\to C_{B}}(\\rho_{B_{m+1}|z_1^m})\n&= \\frac{\\tr_{B_1^m}\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{C_{B}} ) \\Gamma_{B_{m+1} \\to C_{B}}(\\rho_{B_1^{m+1}})\\Big]}{\\tr\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{B_{m+1}} )\\rho_{B_1^{m+1}}\\Big]} \\\\\n&= \\frac{\\tr_{B_1^m}\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{B_{m+1}} ) (\\rho_{B_1 \\cdots B_{m}} \\otimes Y_{C_B})\\Big]}{\\tr\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{B_{m+1}} )\\rho_{B_1^{m+1}}\\Big]} \\\\\n&= Y_{C_B}.\n\\end{align}\n\\end{proof}","start":33847,"end":37993}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"测量后应用消相关引理","source_units":[{"raw_tex":"\\begin{proof}\nLet $\\cM_{B}$ be a measurement of the $B$ system and call the outcome system $Z$. Consider the state $\\rho_{AZ_1^n}$ obtained by measuring all the $B$ systems with $\\cM_{B}$. This distribution is symmetric with respect to $A$ and so we can apply Lemma~\\ref{lem:conditioning-breaks-correlations}. We find that there exists an $m \\in \\{0, \\cdots, n-1\\}$ such that\n\\begin{align}\n\\exc{z_1^m}{\\| \\rho_{AZ_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m} \\|_1^2} \\leq \\frac{(2 \\ln 2) \\log d_{A}}{n}.\n\\end{align}\n","start":33847,"end":34377}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"旁信息失真恢复量子距离","source_units":[{"raw_tex":"Note that we have for any $z_1^m$, $\\rho_{AZ_{m+1}|z_1^m} = (\\cI_{A} \\otimes \\cM_{B})(\\rho_{AB_{m+1}|z_1^m})$ and correspondingly $\\rho_{Z_{m+1}|z_1^m} = \\cM_{B}(\\rho_{B_{m+1}|z_1^m})$. Now, we choose the measurement $\\cM_{B}$ to be as in Lemma~\\ref{lem:ic-meas-side-info} and achieving $f(B|\\cdot)$ in \\eqref{eq:def-f2}, we get that $\\| \\xi_{AB} \\|_1^2 \\leq f(B|\\cdot)^2 \\| (\\cI_{A} \\otimes \\cM_{B})(\\xi_{AB}) \\|_1^2$, where $\\xi_{AB} = \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}$. As a result, we have\n\\begin{align}\n\\exc{z_1^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1^2} \\leq f(B|\\cdot)^2\\frac{(2 \\ln 2) \\log d_{A}}{n}.\n\\end{align}\n","start":34377,"end":35072}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"乘积失真与信道收缩","source_units":[{"raw_tex":"But note we can also choose measurements $\\cM_{A}$ and $\\cM_{B}$ achieving $f(A,B)$ in \\eqref{eq:def-f1}. In this case,\n\\begin{align} \n\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1^2 &\\leq f(A,B)^2\\| (\\cM_A \\otimes \\cM_{B})(\\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}) \\|_1^2 \\\\\n&\\leq f(A,B)^2 \\| (\\cI_A \\otimes \\cM_{B})(\\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}) \\|_1^2 \\\\\n&= f(A,B)^2 \\| \\rho_{AZ_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m} \\|_1^2 ,\n\\end{align}\nwhere we used the fact that the trace norm cannot increase when applying the quantum channel $\\cM_{A}$ \\cite[Theorem 3.39]{watrous-book}. As a result, we get\n\\begin{align}\n\\exc{z_1^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1^2} \\leq f(A,B)^2 \\frac{(2 \\ln 2) \\log d_{A}}{n}.\n\\end{align}\n","start":35072,"end":35951}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"平方误差转平均距离","source_units":[{"raw_tex":"Now, using the convexity of the square function, we get\n\\begin{align}\n\\exc{z_{1}^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1}&\\leq \\sqrt{\\exc{z_{1}^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|^2_1}} \\\\\n&\\leq  \\min\\big\\{f(A, B), f(B|\\cdot)\\big\\}\\sqrt{\\frac{(2 \\ln 2) \\log d_{A}}{n}}.\n\\end{align}\n","start":35951,"end":36313}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"形成候选积态混合","source_units":[{"raw_tex":"Then, using the convexity of the norm and the fact that $\\exc{z_1^m}{\\rho_{AB_{m+1}|z_1^m}} = \\rho_{AB_{m+1}}$, we obtain\n\\begin{align}\n\\left\\| \\rho_{AB_{m+1}} - \\exc{z_1^m}{\\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}} \\right\\|_1 &\\leq \\min\\big\\{f(A,B), f(B|\\cdot)\\big\\}\\sqrt{\\frac{(2 \\ln 2) \\log d_{A}}{n}}.\n\\end{align}\nThe state $\\exc{z_1^m}{\\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}}$ corresponds to our candidate mixture of product states. It now remains to show that all the states in the mixture satisfy the linear constraints. Indeed we have for any $z_1^m$, writing $M_{z}$ for matrices of the measurement $\\cM_{B}$,\n","start":36313,"end":36940}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"验证每项的 A 约束","source_units":[{"raw_tex":"The state $\\exc{z_1^m}{\\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}}$ corresponds to our candidate mixture of product states. It now remains to show that all the states in the mixture satisfy the linear constraints. Indeed we have for any $z_1^m$, writing $M_{z}$ for matrices of the measurement $\\cM_{B}$,\n\\begin{align}\n\\Lambda_{A \\to C_{A}}(\\rho_{A|z_1^m})\n&= \\frac{\\tr_{B_1^m}\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m}) \\Lambda_{A \\to C_{A}}(\\rho_{A B_1^m})\\Big]}{\\tr\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m})\\rho_{A B_1^m}\\Big]} \\\\\n&= \\frac{\\tr_{B_1^m}\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m}) (X_{C_A} \\otimes \\rho_{B_1^m}) \\Big]}{\\tr\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m})\\rho_{A B_1^m}\\Big]} \\\\\n&= X_{C_A},\n\\end{align}\n","start":36634,"end":37426}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"验证每项的 B 约束","source_units":[{"raw_tex":"and similarly\n\\begin{align}\n\\Gamma_{B \\to C_{B}}(\\rho_{B_{m+1}|z_1^m})\n&= \\frac{\\tr_{B_1^m}\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{C_{B}} ) \\Gamma_{B_{m+1} \\to C_{B}}(\\rho_{B_1^{m+1}})\\Big]}{\\tr\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{B_{m+1}} )\\rho_{B_1^{m+1}}\\Big]} \\\\\n&= \\frac{\\tr_{B_1^m}\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{B_{m+1}} ) (\\rho_{B_1 \\cdots B_{m}} \\otimes Y_{C_B})\\Big]}{\\tr\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{B_{m+1}} )\\rho_{B_1^{m+1}}\\Big]} \\\\\n&= Y_{C_B}.\n\\end{align}\n\\end{proof}","start":37426,"end":37993}]},{"id":"S045","semantic_type":"lemma","kind":"lemma","labels":["lem:ic-meas-side-info"],"declared_external":false,"source_only":false,"title":"两设计的旁信息失真界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:ic-meas-side-info}\nConsider a state two-design on $B$, i.e., a set of rank-one projectors $\\{P_z\\}_{z \\in \\{1, \\dots, t\\}}$ such that $\\frac{1}{t} \\sum_{z=1}^t P_z \\otimes P_z = \\frac{2 P^{\\mathrm{sym}}}{d_B (d_B+1)}$, where $P^{\\mathrm{sym}}$ denotes the projector onto the symmetric subspace of $B \\otimes B$. Let $\\mathcal{M}_B$ be the measurement defined as\n\\begin{align}\n\\mathcal{M}_B(X) = \\sum_{z} \\frac{d_B}{t} \\cdot\\mathrm{Tr}\\big[P_z X\\big] |z\\rangle\\langle z|,\n\\end{align}\nand $\\xi_{AB}$ be a Hermitian matrix on $A\\otimes B$. Then, we have that\n\\begin{align}\n\\| (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\|_1\\geq \\frac{1}{d_B^2 (d_B+1)} \\| \\xi_{AB} \\|_{1}.\n\\end{align}\n\\end{lemma}","start":124046,"end":124769}]},{"id":"S046","semantic_type":"definition","kind":"definition","labels":["lem:ic-meas-side-info"],"declared_external":false,"source_only":false,"title":"态两设计测量","source_units":[{"raw_tex":"Consider a state two-design on $B$, i.e., a set of rank-one projectors $\\{P_z\\}_{z \\in \\{1, \\dots, t\\}}$ such that $\\frac{1}{t} \\sum_{z=1}^t P_z \\otimes P_z = \\frac{2 P^{\\mathrm{sym}}}{d_B (d_B+1)}$, where $P^{\\mathrm{sym}}$ denotes the projector onto the symmetric subspace of $B \\otimes B$. Let $\\mathcal{M}_B$ be the measurement defined as\n\\begin{align}\n\\mathcal{M}_B(X) = \\sum_{z} \\frac{d_B}{t} \\cdot\\mathrm{Tr}\\big[P_z X\\big] |z\\rangle\\langle z|,\n\\end{align}\n","start":124089,"end":124553}]},{"id":"S047","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"旁信息失真完整证明","source_units":[{"raw_tex":"\\begin{proof}\nFor any full rank quantum state $\\sigma_{A}$, we have by a H\\\"older type inequality for $\\sigma$-weighted Schatten norms that (see, e.g., \\cite{OLKIEWICZ1999246} or \\cite{beigi13})\n\\begin{align}\n\\| (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\|_1 \\geq  \\frac{\\left\\| \\sigma_{A}^{-1/4} (\\mathcal{I}_{A} \\otimes \\mathcal{M})(\\xi_{AB}) \\sigma_{A}^{-1/4} \\right\\|^2_{2}}{\\left\\| \\sigma_{A}^{-1/2} (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\sigma_{A}^{-1/2} \\right\\|_{\\infty}}.\n\\end{align}\nFor example, the above inequality can be obtained using \\cite[Corollary 3]{beigi13} with the operator $\\sigma_{A}^{-1/2} (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\sigma_{A}^{-1/2}$, weight $\\sigma$, and $p_\\theta = 2$, $\\theta = 1/2$, $p_0 = 1$, $p_1 = \\infty$. We note that this particular H\\\"older type inequality for $\\sigma$-weighted norms is elementary and follows easily from the usual H\\\"older inequality, but one way of potentially improving the dimension dependence in Lemma~\\ref{lem:ic-meas-side-info} might be to use another H\\\"older inequality, in particular the $(1,4)$ inequality.\n\nHenceforth, we abbreviate $d\\equiv d_B$. To further bound the numerator, letting $\\tilde{\\xi}_{AB}:=\\sigma_{A}^{-1/4} \\xi_{AB} \\sigma_{A}^{-1/4}$ we get\n\\begin{align}\n\\| (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\tilde{\\xi}_{AB}) \\|^2_{2}\n&= \\left\\| \\sum_{z} \\frac{d}{t} |z\\rangle \\langle z| \\otimes  \\mathrm{Tr}_{B}\\left[(1_{A} \\otimes P_z) \\tilde{\\xi}_{AB}\\right] \\right\\|_2^2 \\\\\n&= \\sum_{z}  \\frac{d^2}{t^2} \\mathrm{Tr}\\left[\\mathrm{Tr}_{B}\\left[(1_{A} \\otimes P_z) \\tilde{\\xi}_{AB}\\right] \\otimes \\mathrm{Tr}_{\\bar{B}}\\left[(1_{\\bar{A}} \\otimes P_z) \\tilde{\\xi}_{\\bar{A}\\bar{B}}\\right]^{\\dagger} F_{A\\bar{A}} \\right] \\\\\n&= \\sum_{z} \\frac{d^2}{t^2} \\mathrm{Tr}\\left[\\left(\\left((1_{A} \\otimes P_z) \\tilde{\\xi}_{AB}\\right) \\otimes \\left((1_{\\bar{A}} \\otimes P_z) \\tilde{\\xi}_{\\bar{A}\\bar{B}}\\right)^{\\dagger}\\right)\\left(F_{A\\bar{A}} \\otimes1_{B\\bar{B}}\\right)\\right] \\\\\n&= \\frac{d^2}{t^2} \\mathrm{Tr}\\left[\\left(\\tilde{\\xi}_{AB} \\otimes \\tilde{\\xi}_{\\bar{A}\\bar{B}}^{\\dagger}\\right)\\left(\\sum_{z} (1_{A\\bar{A}} \\otimes P_z \\otimes P_{z})\\right)\\left(F_{A\\bar{A}} \\otimes1_{B \\bar{B}}\\right)\\right] \\\\\n&= \\frac{1}{t} \\frac{d^2}{d(d+1)} \\mathrm{Tr}\\left[\\left(\\tilde{\\xi}_{AB} \\otimes \\tilde{\\xi}_{\\bar{A}\\bar{B}}^{\\dagger}\\right) (1_{A\\bar{A}} \\otimes (1_{B\\bar{B}} + F_{B\\bar{B}}))\\left(F_{A\\bar{A}} \\otimes1_{B \\bar{B}}\\right)\\right] \\\\\n&= \\frac{1}{t} \\frac{d^2}{d(d+1)} \\Big(\\underbrace{\\mathrm{Tr}\\left[\\tilde{\\xi}_{A} \\tilde{\\xi}_{A}^{\\dagger}\\right]}_{\\geq 0} + \\underbrace{\\mathrm{Tr}\\left[\\tilde{\\xi}_{AB} \\tilde{\\xi}_{AB}^{\\dagger}\\right]}_{=\\left\\|\\tilde{\\xi}_{AB}\\right\\|_2^2} \\Big) \\\\\n&\\geq \\frac{1}{t}\\frac{d^2}{d^2 (d+1)} \\| \\xi_{AB} \\|^2_{1} ,\n\\end{align}\nwhere $F$ denotes the swap operator (as defined in Section \\ref{sec:notation}) and in the last step used the H\\\"older inequality (see, e.g., \\cite{bhatia97})\n\\begin{align}\n\\left\\|\\xi_{AB}\\right\\|_1=\\left\\|\\sigma^{1/4}\\sigma^{-1/4}\\xi_{AB}\\sigma^{-1/4}\\sigma^{1/4}\\right\\|_1\\leq\\left\\|\\sigma^{1/4}\\otimes1_B\\right\\|_4\\left\\|\\sigma_A^{-1/4}\\xi_{AB}\\sigma_A^{-1/4}\\right\\|_2\\left\\|\\sigma^{1/4}\\otimes1_B\\right\\|_4\\leq\\sqrt{d}\\left\\|\\tilde{\\xi}_{AB}\\right\\|_2.\n\\end{align}\nFor further bounding the denominator we write\n\\begin{align}\n\\left\\| \\sigma_{A}^{-1/2} (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\sigma_{A}^{-1/2} \\right\\|_\\infty\n&= \\max_{z} \\frac{d}{t} \\left\\| \\mathrm{Tr}_{B}\\left[(1_{A} \\otimes P_z) \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2}\\right]\\right\\|_\\infty \\\\\n&\\leq \\frac{d}{t} \\max_{\\ket{\\phi}_{A}, \\ket{\\psi}_{B}} \\bra{\\phi}_{A} \\otimes \\bra{\\psi}_{B} \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2} \\ket{\\phi}_{A} \\otimes \\ket{\\psi}_{B} \\\\\n&\\leq \\frac{d}{t} \\left\\| \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2} \\right\\|_\\infty,\n\\end{align}\nwhere we used the fact that $P_z$ is a rank 1 projector. Now, observe that for any $\\xi_{AB}$, there exists a $\\sigma_A$ of unit trace such that\n\\begin{align}\n\\frac{\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}}}{\\| \\xi_{AB} \\|_{1}} \\preceq d\\cdot\\sigma_{A} \\otimes1_{B}.\n\\end{align}\nThis just follows from, e.g., \\cite[Lemma B.6]{bertachristandl11}, where it is shown that we can in fact choose\\footnote{By a continuity argument $\\sigma_A$ can be assumed to have full rank.}\n\\begin{align}\n\\sigma_{A} = \\| \\xi_{AB} \\|_{1}^{-1}\\cdot\\mathrm{Tr}_{B}\\left[\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}}\\right].\n\\end{align}\nAs a result, we have\n\\begin{align}\n\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}} \\preceq d\\| \\xi_{AB} \\|_{1}\\cdot\\sigma_{A} \\otimes1_{B}.\n\\end{align}\nAs $\\xi_{AB}$ is Hermitian, we can decompose it into the positive and negative part $\\xi_{AB} = P - Q$ with $P$ and $Q$ positive semidefinite and $PQ = 0$, then $\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}} = P + Q$ and so $-\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}} \\preceq \\xi_{AB} \\preceq \\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}}$. Thus, we get\n\\begin{align}\n- d_{B} \\| \\xi_{AB} \\|_1\\cdot\\sigma_{A} \\otimes1_{B} \\preceq \\xi_{AB} \\preceq d_{B} \\| \\xi_{AB} \\|_1 \\cdot\\sigma_{A} \\otimes1_{B},\n\\end{align}\nand we find $\\| \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2} \\|_{\\infty} \\leq d\\| \\xi_{AB} \\|_1$. This concludes the proof.\n\\end{proof}","start":124988,"end":130190}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"加权 Schatten 插值","source_units":[{"raw_tex":"\\begin{proof}\nFor any full rank quantum state $\\sigma_{A}$, we have by a H\\\"older type inequality for $\\sigma$-weighted Schatten norms that (see, e.g., \\cite{OLKIEWICZ1999246} or \\cite{beigi13})\n\\begin{align}\n\\| (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\|_1 \\geq  \\frac{\\left\\| \\sigma_{A}^{-1/4} (\\mathcal{I}_{A} \\otimes \\mathcal{M})(\\xi_{AB}) \\sigma_{A}^{-1/4} \\right\\|^2_{2}}{\\left\\| \\sigma_{A}^{-1/2} (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\sigma_{A}^{-1/2} \\right\\|_{\\infty}}.\n\\end{align}\nFor example, the above inequality can be obtained using \\cite[Corollary 3]{beigi13} with the operator $\\sigma_{A}^{-1/2} (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\sigma_{A}^{-1/2}$, weight $\\sigma$, and $p_\\theta = 2$, $\\theta = 1/2$, $p_0 = 1$, $p_1 = \\infty$. We note that this particular H\\\"older type inequality for $\\sigma$-weighted norms is elementary and follows easily from the usual H\\\"older inequality, but one way of potentially improving the dimension dependence in Lemma~\\ref{lem:ic-meas-side-info} might be to use another H\\\"older inequality, in particular the $(1,4)$ inequality.\n","start":124988,"end":126105}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"两设计计算二阶分子","source_units":[{"raw_tex":"Henceforth, we abbreviate $d\\equiv d_B$. To further bound the numerator, letting $\\tilde{\\xi}_{AB}:=\\sigma_{A}^{-1/4} \\xi_{AB} \\sigma_{A}^{-1/4}$ we get\n\\begin{align}\n\\| (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\tilde{\\xi}_{AB}) \\|^2_{2}\n&= \\left\\| \\sum_{z} \\frac{d}{t} |z\\rangle \\langle z| \\otimes  \\mathrm{Tr}_{B}\\left[(1_{A} \\otimes P_z) \\tilde{\\xi}_{AB}\\right] \\right\\|_2^2 \\\\\n&= \\sum_{z}  \\frac{d^2}{t^2} \\mathrm{Tr}\\left[\\mathrm{Tr}_{B}\\left[(1_{A} \\otimes P_z) \\tilde{\\xi}_{AB}\\right] \\otimes \\mathrm{Tr}_{\\bar{B}}\\left[(1_{\\bar{A}} \\otimes P_z) \\tilde{\\xi}_{\\bar{A}\\bar{B}}\\right]^{\\dagger} F_{A\\bar{A}} \\right] \\\\\n&= \\sum_{z} \\frac{d^2}{t^2} \\mathrm{Tr}\\left[\\left(\\left((1_{A} \\otimes P_z) \\tilde{\\xi}_{AB}\\right) \\otimes \\left((1_{\\bar{A}} \\otimes P_z) \\tilde{\\xi}_{\\bar{A}\\bar{B}}\\right)^{\\dagger}\\right)\\left(F_{A\\bar{A}} \\otimes1_{B\\bar{B}}\\right)\\right] \\\\\n&= \\frac{d^2}{t^2} \\mathrm{Tr}\\left[\\left(\\tilde{\\xi}_{AB} \\otimes \\tilde{\\xi}_{\\bar{A}\\bar{B}}^{\\dagger}\\right)\\left(\\sum_{z} (1_{A\\bar{A}} \\otimes P_z \\otimes P_{z})\\right)\\left(F_{A\\bar{A}} \\otimes1_{B \\bar{B}}\\right)\\right] \\\\\n&= \\frac{1}{t} \\frac{d^2}{d(d+1)} \\mathrm{Tr}\\left[\\left(\\tilde{\\xi}_{AB} \\otimes \\tilde{\\xi}_{\\bar{A}\\bar{B}}^{\\dagger}\\right) (1_{A\\bar{A}} \\otimes (1_{B\\bar{B}} + F_{B\\bar{B}}))\\left(F_{A\\bar{A}} \\otimes1_{B \\bar{B}}\\right)\\right] \\\\\n&= \\frac{1}{t} \\frac{d^2}{d(d+1)} \\Big(\\underbrace{\\mathrm{Tr}\\left[\\tilde{\\xi}_{A} \\tilde{\\xi}_{A}^{\\dagger}\\right]}_{\\geq 0} + \\underbrace{\\mathrm{Tr}\\left[\\tilde{\\xi}_{AB} \\tilde{\\xi}_{AB}^{\\dagger}\\right]}_{=\\left\\|\\tilde{\\xi}_{AB}\\right\\|_2^2} \\Big) \\\\\n&\\geq \\frac{1}{t}\\frac{d^2}{d^2 (d+1)} \\| \\xi_{AB} \\|^2_{1} ,\n\\end{align}\n","start":126106,"end":127777}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"一范数与加权二范数","source_units":[{"raw_tex":"where $F$ denotes the swap operator (as defined in Section \\ref{sec:notation}) and in the last step used the H\\\"older inequality (see, e.g., \\cite{bhatia97})\n\\begin{align}\n\\left\\|\\xi_{AB}\\right\\|_1=\\left\\|\\sigma^{1/4}\\sigma^{-1/4}\\xi_{AB}\\sigma^{-1/4}\\sigma^{1/4}\\right\\|_1\\leq\\left\\|\\sigma^{1/4}\\otimes1_B\\right\\|_4\\left\\|\\sigma_A^{-1/4}\\xi_{AB}\\sigma_A^{-1/4}\\right\\|_2\\left\\|\\sigma^{1/4}\\otimes1_B\\right\\|_4\\leq\\sqrt{d}\\left\\|\\tilde{\\xi}_{AB}\\right\\|_2.\n\\end{align}\n","start":127777,"end":128246}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"测后算子范数分母界","source_units":[{"raw_tex":"For further bounding the denominator we write\n\\begin{align}\n\\left\\| \\sigma_{A}^{-1/2} (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\sigma_{A}^{-1/2} \\right\\|_\\infty\n&= \\max_{z} \\frac{d}{t} \\left\\| \\mathrm{Tr}_{B}\\left[(1_{A} \\otimes P_z) \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2}\\right]\\right\\|_\\infty \\\\\n&\\leq \\frac{d}{t} \\max_{\\ket{\\phi}_{A}, \\ket{\\psi}_{B}} \\bra{\\phi}_{A} \\otimes \\bra{\\psi}_{B} \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2} \\ket{\\phi}_{A} \\otimes \\ket{\\psi}_{B} \\\\\n&\\leq \\frac{d}{t} \\left\\| \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2} \\right\\|_\\infty,\n\\end{align}\nwhere we used the fact that $P_z$ is a rank 1 projector. Now, observe that for any $\\xi_{AB}$, there exists a $\\sigma_A$ of unit trace such that\n","start":128246,"end":128984}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"按绝对值边缘选权重态","source_units":[{"raw_tex":"where we used the fact that $P_z$ is a rank 1 projector. Now, observe that for any $\\xi_{AB}$, there exists a $\\sigma_A$ of unit trace such that\n\\begin{align}\n\\frac{\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}}}{\\| \\xi_{AB} \\|_{1}} \\preceq d\\cdot\\sigma_{A} \\otimes1_{B}.\n\\end{align}\nThis just follows from, e.g., \\cite[Lemma B.6]{bertachristandl11}, where it is shown that we can in fact choose\\footnote{By a continuity argument $\\sigma_A$ can be assumed to have full rank.}\n\\begin{align}\n\\sigma_{A} = \\| \\xi_{AB} \\|_{1}^{-1}\\cdot\\mathrm{Tr}_{B}\\left[\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}}\\right].\n\\end{align}\n","start":128839,"end":129434}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"绝对值的未归一化支配","source_units":[{"raw_tex":"As a result, we have\n\\begin{align}\n\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}} \\preceq d\\| \\xi_{AB} \\|_{1}\\cdot\\sigma_{A} \\otimes1_{B}.\n\\end{align}\n","start":129434,"end":129573}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"Hermitian 正负部分夹逼","source_units":[{"raw_tex":"As $\\xi_{AB}$ is Hermitian, we can decompose it into the positive and negative part $\\xi_{AB} = P - Q$ with $P$ and $Q$ positive semidefinite and $PQ = 0$, then $\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}} = P + Q$ and so $-\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}} \\preceq \\xi_{AB} \\preceq \\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}}$. Thus, we get\n\\begin{align}\n- d_{B} \\| \\xi_{AB} \\|_1\\cdot\\sigma_{A} \\otimes1_{B} \\preceq \\xi_{AB} \\preceq d_{B} \\| \\xi_{AB} \\|_1 \\cdot\\sigma_{A} \\otimes1_{B},\n\\end{align}\n","start":129573,"end":130055}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"控制加权范数并收尾","source_units":[{"raw_tex":"and we find $\\| \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2} \\|_{\\infty} \\leq d\\| \\xi_{AB} \\|_1$. This concludes the proof.\n\\end{proof}","start":130055,"end":130190}]},{"id":"S056","semantic_type":"theorem","kind":"theorem","labels":[],"declared_external":false,"source_only":false,"title":"受约束 SDP 层级收敛","source_units":[{"raw_tex":"\\begin{theorem}\nFor the SDPs\n\\begin{align}\n\\mathrm{SDP}_n :=\\max &\\quad \\mathrm{Tr}\\big[G_{AB} W_{AB_1}\\big]\\\\\ns.t. \n&\\quad W_{AB_1^n}\\succeq0, \\mathrm{Tr}(W_{AB_1^n}) = 1, \\;W_{AB_1^n}=\\mathcal{U}_{B_1^n}^\\pi\\left(W_{AB_1^n}\\right)\\;\\forall\\pi\\in\\mathfrak{S}_n\\\\\n& \\quad \\Lambda_{A\\to C_A}\\left(W_{AB_1^n}\\right)=X_{C_A}\\otimes W_{B_1^n},\\;\\Gamma_{B_n\\to C_B}\\left(W_{B_1^n}\\right)=W_{B_1^{n-1}}\\otimes Y_{C_B}\n\\end{align}\nand $Q$ defined as above, we have for $d:=\\max\\{d_A,d_B\\}$ that\n\\begin{align}\n0 \\leq \\mathrm{SDP}_n - Q \\leq \\frac{\\mathrm{poly}(d)}{\\sqrt{n}}\\quad\\text{implying}\\quad Q=\\lim_{n\\to\\infty}\\mathrm{SDP}_n.\n\\end{align}\n\\end{theorem}","start":48380,"end":49032}]},{"id":"S057","semantic_type":"algorithm","kind":"algorithm","labels":[],"declared_external":false,"source_only":false,"title":"对称扩张 SDP 层级","source_units":[{"raw_tex":"For the SDPs\n\\begin{align}\n\\mathrm{SDP}_n :=\\max &\\quad \\mathrm{Tr}\\big[G_{AB} W_{AB_1}\\big]\\\\\ns.t. \n&\\quad W_{AB_1^n}\\succeq0, \\mathrm{Tr}(W_{AB_1^n}) = 1, \\;W_{AB_1^n}=\\mathcal{U}_{B_1^n}^\\pi\\left(W_{AB_1^n}\\right)\\;\\forall\\pi\\in\\mathfrak{S}_n\\\\\n& \\quad \\Lambda_{A\\to C_A}\\left(W_{AB_1^n}\\right)=X_{C_A}\\otimes W_{B_1^n},\\;\\Gamma_{B_n\\to C_B}\\left(W_{B_1^n}\\right)=W_{B_1^{n-1}}\\otimes Y_{C_B}\n\\end{align}\n","start":48396,"end":48804}]},{"id":"S058","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"层级收敛完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nWe have by construction $0 \\leq \\mathrm{SDP}_n - Q $ and the remaining inequality arises from\n\\begin{align}\n\\mathrm{Tr}\\left[G_{AB} W_{AB_1}\\right]&=\\mathrm{Tr}\\left[G_{AB} (W_{A} \\otimes W_{B})\\right]+\\mathrm{Tr}\\left[G_{AB}\\left(W_{AB_1} - W_{A} \\otimes W_{B}\\right)\\right] \\\\\n& \\leq \\mathrm{Tr}\\left[G_{AB} (W_{A} \\otimes W_{B})\\right]+\\| G_{AB} \\|_\\infty \\cdot \\|W_{AB_1} - W_{A} \\otimes W_{B} \\|_1 \\\\\n&\\leq\\mathrm{Tr}\\left[G_{AB} (W_{A} \\otimes W_{B})\\right]+\\frac{\\mathrm{poly}(d)}{\\sqrt{n}},\n\\end{align}\nwhere we used H\\\"older's inequality and the de Finetti argument as in Theorem \\ref{thm:main-deFinetti}.\n\\end{proof}","start":49034,"end":49674}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"构造给出松弛外界","source_units":[{"raw_tex":"\\begin{proof}\nWe have by construction $0 \\leq \\mathrm{SDP}_n - Q $ and the remaining inequality arises from\n","start":49034,"end":49142}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"分解目标为比较值与误差","source_units":[{"raw_tex":"\\begin{align}\n\\mathrm{Tr}\\left[G_{AB} W_{AB_1}\\right]&=\\mathrm{Tr}\\left[G_{AB} (W_{A} \\otimes W_{B})\\right]+\\mathrm{Tr}\\left[G_{AB}\\left(W_{AB_1} - W_{A} \\otimes W_{B}\\right)\\right] \\\\\n","start":49142,"end":49327}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"Hölder 控制目标误差","source_units":[{"raw_tex":"& \\leq \\mathrm{Tr}\\left[G_{AB} (W_{A} \\otimes W_{B})\\right]+\\| G_{AB} \\|_\\infty \\cdot \\|W_{AB_1} - W_{A} \\otimes W_{B} \\|_1 \\\\\n","start":49327,"end":49454},{"raw_tex":"where we used H\\\"older's inequality and the de Finetti argument as in Theorem \\ref{thm:main-deFinetti}.\n","start":49559,"end":49663}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"援引 de Finetti 宣告误差率","source_units":[{"raw_tex":"&\\leq\\mathrm{Tr}\\left[G_{AB} (W_{A} \\otimes W_{B})\\right]+\\frac{\\mathrm{poly}(d)}{\\sqrt{n}},\n\\end{align}\nwhere we used H\\\"older's inequality and the de Finetti argument as in Theorem \\ref{thm:main-deFinetti}.\n\\end{proof}","start":49454,"end":49674}]},{"id":"S063","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件明确列出的 nielsen00, Chapter 11；本文在第354行引用并在第356—362行使用。"],"source_only":false,"title":"经典量子互信息恒等式","source_units":[]},{"id":"S064","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件中的 watrous-book, Theorem 5.38；原文第362行明确引用。"],"source_only":false,"title":"量子 Pinsker 不等式","source_units":[]},{"id":"S065","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件中的 watrous-book, Theorem 3.39；原文第437行明确引用。"],"source_only":false,"title":"信道的迹范数收缩","source_units":[]},{"id":"S066","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件中的 OLKIEWICZ1999246 与 beigi13, Corollary 3；第1322、1326行引用。"],"source_only":false,"title":"加权 Schatten 插值事实","source_units":[]},{"id":"S067","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件中的 bertachristandl11, Lemma B.6；原文第1354行引用。"],"source_only":false,"title":"双体正算子支配界","source_units":[]},{"id":"S068","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件列出的标准有限维矩阵分析；附录第1339行另引用 bhatia97，主证明第592行仅具名使用 Hölder 不等式。"],"source_only":false,"title":"普通 Schatten Hölder 事实","source_units":[]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件中的 Jensen 不等式与范数凸性；原文第441、446行明确具名使用。"],"source_only":false,"title":"Jensen 与范数凸性","source_units":[]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件中的标准有限维迹恒等式和 Schatten 范数；implicit standard fact made explicit by annotator。"],"source_only":false,"title":"分块与交换迹恒等式","source_units":[]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件中的标准谱分解和连续性；implicit standard fact made explicit by annotator。"],"source_only":false,"title":"谱分解与连续性背景","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S056","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S003","kind":"supports"},{"source_id":"S005","target_id":"S003","kind":"supports"},{"source_id":"S009","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S017","kind":"supports"},{"source_id":"S008","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S045","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S021","target_id":"S036","kind":"supports"},{"source_id":"S024","target_id":"S036","kind":"supports"},{"source_id":"S008","target_id":"S027","kind":"supports"},{"source_id":"S014","target_id":"S027","kind":"supports"},{"source_id":"S016","target_id":"S027","kind":"supports"},{"source_id":"S017","target_id":"S027","kind":"supports"},{"source_id":"S063","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S063","target_id":"S032","kind":"supports"},{"source_id":"S016","target_id":"S032","kind":"supports"},{"source_id":"S017","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S064","target_id":"S033","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S030","kind":"supports"},{"source_id":"S033","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S004","target_id":"S034","kind":"supports"},{"source_id":"S010","target_id":"S034","kind":"supports"},{"source_id":"S014","target_id":"S034","kind":"supports"},{"source_id":"S020","target_id":"S034","kind":"supports"},{"source_id":"S022","target_id":"S034","kind":"supports"},{"source_id":"S035","target_id":"S038","kind":"supports"},{"source_id":"S011","target_id":"S038","kind":"supports"},{"source_id":"S029","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S022","target_id":"S039","kind":"supports"},{"source_id":"S045","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S020","target_id":"S040","kind":"supports"},{"source_id":"S065","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S069","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S017","target_id":"S042","kind":"supports"},{"source_id":"S069","target_id":"S042","kind":"supports"},{"source_id":"S035","target_id":"S043","kind":"supports"},{"source_id":"S017","target_id":"S043","kind":"supports"},{"source_id":"S011","target_id":"S043","kind":"supports"},{"source_id":"S010","target_id":"S043","kind":"supports"},{"source_id":"S035","target_id":"S044","kind":"supports"},{"source_id":"S014","target_id":"S044","kind":"supports"},{"source_id":"S017","target_id":"S044","kind":"supports"},{"source_id":"S011","target_id":"S044","kind":"supports"},{"source_id":"S010","target_id":"S044","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S037","kind":"supports"},{"source_id":"S040","target_id":"S037","kind":"supports"},{"source_id":"S041","target_id":"S037","kind":"supports"},{"source_id":"S042","target_id":"S037","kind":"supports"},{"source_id":"S043","target_id":"S037","kind":"supports"},{"source_id":"S044","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S034","kind":"supports"},{"source_id":"S011","target_id":"S045","kind":"supports"},{"source_id":"S012","target_id":"S045","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S066","target_id":"S048","kind":"supports"},{"source_id":"S068","target_id":"S050","kind":"supports"},{"source_id":"S046","target_id":"S049","kind":"supports"},{"source_id":"S007","target_id":"S049","kind":"supports"},{"source_id":"S010","target_id":"S049","kind":"supports"},{"source_id":"S070","target_id":"S049","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S046","target_id":"S051","kind":"supports"},{"source_id":"S070","target_id":"S051","kind":"supports"},{"source_id":"S067","target_id":"S052","kind":"supports"},{"source_id":"S071","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S071","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S071","target_id":"S055","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S049","target_id":"S047","kind":"supports"},{"source_id":"S051","target_id":"S047","kind":"supports"},{"source_id":"S055","target_id":"S047","kind":"supports"},{"source_id":"S050","target_id":"S047","kind":"supports"},{"source_id":"S052","target_id":"S047","kind":"supports"},{"source_id":"S053","target_id":"S047","kind":"supports"},{"source_id":"S054","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S001","target_id":"S057","kind":"supports"},{"source_id":"S010","target_id":"S057","kind":"supports"},{"source_id":"S014","target_id":"S057","kind":"supports"},{"source_id":"S001","target_id":"S059","kind":"supports"},{"source_id":"S057","target_id":"S059","kind":"supports"},{"source_id":"S026","target_id":"S059","kind":"supports"},{"source_id":"S001","target_id":"S060","kind":"supports"},{"source_id":"S057","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S068","target_id":"S061","kind":"supports"},{"source_id":"S034","target_id":"S062","kind":"supports"},{"source_id":"S021","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S059","target_id":"S058","kind":"supports"},{"source_id":"S060","target_id":"S058","kind":"supports"},{"source_id":"S061","target_id":"S058","kind":"supports"},{"source_id":"S062","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S056","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S056","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S003","kind":"supports"},{"source_id":"S005","target_id":"S003","kind":"supports"},{"source_id":"S009","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S017","kind":"supports"},{"source_id":"S008","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S045","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S021","target_id":"S036","kind":"supports"},{"source_id":"S024","target_id":"S036","kind":"supports"},{"source_id":"S008","target_id":"S027","kind":"supports"},{"source_id":"S014","target_id":"S027","kind":"supports"},{"source_id":"S016","target_id":"S027","kind":"supports"},{"source_id":"S017","target_id":"S027","kind":"supports"},{"source_id":"S063","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S063","target_id":"S032","kind":"supports"},{"source_id":"S016","target_id":"S032","kind":"supports"},{"source_id":"S017","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S064","target_id":"S033","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S030","kind":"supports"},{"source_id":"S033","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S004","target_id":"S034","kind":"supports"},{"source_id":"S010","target_id":"S034","kind":"supports"},{"source_id":"S014","target_id":"S034","kind":"supports"},{"source_id":"S020","target_id":"S034","kind":"supports"},{"source_id":"S022","target_id":"S034","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S011","target_id":"S034","kind":"supports"},{"source_id":"S029","target_id":"S034","kind":"supports"},{"source_id":"S045","target_id":"S034","kind":"supports"},{"source_id":"S065","target_id":"S034","kind":"supports"},{"source_id":"S069","target_id":"S034","kind":"supports"},{"source_id":"S017","target_id":"S034","kind":"supports"},{"source_id":"S011","target_id":"S045","kind":"supports"},{"source_id":"S012","target_id":"S045","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S066","target_id":"S045","kind":"supports"},{"source_id":"S068","target_id":"S045","kind":"supports"},{"source_id":"S007","target_id":"S045","kind":"supports"},{"source_id":"S010","target_id":"S045","kind":"supports"},{"source_id":"S070","target_id":"S045","kind":"supports"},{"source_id":"S067","target_id":"S045","kind":"supports"},{"source_id":"S071","target_id":"S045","kind":"supports"},{"source_id":"S001","target_id":"S057","kind":"supports"},{"source_id":"S010","target_id":"S057","kind":"supports"},{"source_id":"S014","target_id":"S057","kind":"supports"},{"source_id":"S001","target_id":"S059","kind":"supports"},{"source_id":"S057","target_id":"S059","kind":"supports"},{"source_id":"S026","target_id":"S059","kind":"supports"},{"source_id":"S001","target_id":"S060","kind":"supports"},{"source_id":"S057","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S068","target_id":"S061","kind":"supports"},{"source_id":"S034","target_id":"S062","kind":"supports"},{"source_id":"S021","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S059","target_id":"S058","kind":"supports"},{"source_id":"S060","target_id":"S058","kind":"supports"},{"source_id":"S061","target_id":"S058","kind":"supports"},{"source_id":"S062","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S056","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["original_program"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"受约束双线性优化","source_units":[{"raw_tex":"To discuss our approach, we first rewrite~\\eqref{eq:form} by defining $G_{AB}= (\\Pi_{A \\to D}^{\\dagger} \\otimes \\Pi_{B \\to E}^{\\dagger})(H)$, where $\\Pi^{\\dagger}$ denotes the adjoint map of $\\Pi$ in the Hilbert-Schmidt inner product. This leads to the form\n\\begin{align}\\label{original_program}\nQ=\\max &\\quad \\mathrm{Tr}\\big[G_{AB} (W_{A} \\otimes W_{B})\\big]\\\\\ns.t. &\\quad W_{A} \\succeq 0,\\quad W_{B} \\succeq 0,\\quad\\mathrm{Tr}[W_A] = \\mathrm{Tr}[W_B] = 1\\\\\n&\\quad\\Lambda_{A\\to C_A}\\left(W_{A}\\right)=X_{C_A},\\quad\\Gamma_{B\\to C_B}\\left(W_{B}\\right)=Y_{C_B}, \n\\end{align}\nwhere $G_{AB}\\in\\mathcal{S}_A\\otimes\\mathcal{S}_B$, $\\Lambda_{A\\to C_A}:\\mathcal{S}_A \\to \\mathcal{S}_{C_A}$ and $\\Gamma_{B\\to C_B}:\\mathcal{S}_B \\to \\mathcal{S}_{C_B}$ denote linear maps, and $X_{C_A}\\in\\mathcal{S}_{C_A}$ and $Y_{C_B}\\in\\mathcal{S}_{C_B}$ are the matrices defining $\\mathcal{A}_{A}$ and $\\mathcal{A}_B$ as the affine subspaces associated with the kernels of the linear maps $\\Lambda_{A\\to C_A}$ and $\\Gamma_{B\\to C_B}$, respectively.\n","start":7954,"end":8979}]},{"id":"S002","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:transformed_program"],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"凸包优化等价性","source_units":[{"raw_tex":"Now, by the linearity of the objective function we can equivalently optimise over the convex hull of feasible points\n\\begin{align}\\label{eq:transformed_program}\nQ=\\max &\\quad \\mathrm{Tr}\\left[G_{AB}\\left(\\sum_{i\\in I}p_iW_A^i \\otimes W_B^i\\right)\\right]\\\\\ns.t. &\\quad p_i\\geq0,\\quad W_A^i \\succeq 0,\\quad W_B^i \\succeq 0,\\quad\\mathrm{Tr}\\left[W_A^i\\right] = \\mathrm{Tr}\\left[W_B^i\\right] = 1\\quad\\forall i\\in I,\\quad\\sum_{i\\in I}p_i=1\\\\\n&\\quad\\Lambda_{A\\to C_A}\\left(W_A^i\\right)=X_{C_A},\\quad\\Gamma_{B\\to C_B}\\left(W_B^i\\right)=Y_{C_B}\\quad\\forall i\\in I.\n\\end{align}\n","start":8979,"end":9548}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"可分态集合","source_units":[{"raw_tex":"That is, in the language of quantum information theory we are maximizing over a subset of the so-called {\\it separable quantum states}\\,---\\,where the latter is defined on $A\\otimes B$ as\n\\begin{align}\n\\text{Sep}(A:B)=\\left\\{\\sum_{i\\in I}p_iW_A^i \\otimes W_B^i: W_A^i \\succeq 0, W_B^i \\succeq 0, \\mathrm{Tr}\\left[W_A^i\\right]=\\mathrm{Tr}\\left[W_B^i\\right] = 1,p_i\\geq0,\\sum_{i\\in I}p_i=1\\right\\}.\n\\end{align}\nRecall that matrices $W_A\\in\\mathcal{S}^+_A$ are called quantum states on system $A$\\,---\\,and similarly for bipartite states on $A\\otimes B$.\n","start":9548,"end":10100}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"系统与量子态","source_units":[{"raw_tex":"In the following, we introduce some notation that is standard in quantum information theory. A $d_A$-dimensional quantum system (or in short {\\it system}) is given by an inner product space $\\mathbb{C}^{d_A}$ and denoted by $A$. {\\it Quantum states} (or in short {\\it states}) on $A$ are matrices\\footnote{Here and henceforth we use the symbol $:=$ as {\\it equal by definition}.}\n\\begin{align}\n\\text{$W_A\\in\\mathcal{S}_A:=\\mathbb{C}^{d_A\\times d_A}$ with $\\mathrm{Tr}[W_A]=1$ and $W_A\\succeq0$,}\n\\end{align}\nwhere $\\succeq$ denotes the operator (Loewner) order. Quantum states of rank one are called {\\it pure} and can be written as $W_A=\\ket{\\psi}\\bra{\\psi}_A$, where $\\ket{\\psi}_A\\in\\mathbb{C}^{d_A}$ and $\\ket{\\psi}\\bra{\\psi}_A\\in\\mathcal{S}_A$ denotes the rank-one projector on the vector $\\ket{\\psi}_A$.\n\n","start":17076,"end":17886}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"复合系统与积态","source_units":[{"raw_tex":"A {\\it bipartite system} $AB:=A\\otimes B$ is given by an inner product space $\\mathbb{C}^{d_A}\\otimes\\mathbb{C}^{d_B}$, where $\\otimes$ denotes the Kronecker tensor product. Correspondingly, states on $AB$ are matrices $W_{AB}\\in\\mathcal{S}_A\\otimes\\mathcal{S}_B$ with $\\mathrm{Tr}[W_{AB}]=1$ and $W_{AB}\\succeq0$. {\\it Separable states} are states on $AB$ that are in the convex hull of product states $W_A\\otimes W_B$, with $W_A$ and $W_B$ states on $A$ and $B$, respectively. The {\\it maximally entangled state} $\\Phi_{AB}:=\\ket{\\Phi}\\bra{\\Phi}_{AB}$ on $AB$ for $d:=d_A=d_B$ is not separable and defined via\n","start":17886,"end":18498}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"最大纠缠态背景","source_units":[{"raw_tex":"A {\\it bipartite system} $AB:=A\\otimes B$ is given by an inner product space $\\mathbb{C}^{d_A}\\otimes\\mathbb{C}^{d_B}$, where $\\otimes$ denotes the Kronecker tensor product. Correspondingly, states on $AB$ are matrices $W_{AB}\\in\\mathcal{S}_A\\otimes\\mathcal{S}_B$ with $\\mathrm{Tr}[W_{AB}]=1$ and $W_{AB}\\succeq0$. {\\it Separable states} are states on $AB$ that are in the convex hull of product states $W_A\\otimes W_B$, with $W_A$ and $W_B$ states on $A$ and $B$, respectively. The {\\it maximally entangled state} $\\Phi_{AB}:=\\ket{\\Phi}\\bra{\\Phi}_{AB}$ on $AB$ for $d:=d_A=d_B$ is not separable and defined via\n\\begin{align}\n\\text{$\\ket{\\Phi}_{AB}:=\\frac{1}{\\sqrt{d}}\\sum_{x=1}^d\\ket{x}_A\\otimes\\ket{x}_B$ for some orthonormal basis $\\{\\ket{x}\\}_{x=1}^d$ of $\\mathbb{C}^d$.}\n\\end{align}\n","start":17886,"end":18674}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"交换算子","source_units":[{"raw_tex":"The {\\it swap operator} $F_{AB}$ on $A\\otimes B$ exchanges the two quantum systems, i.e., $F_{AB}(W_A\\otimes W_B)=W_B\\otimes W_A$ for every state $W_A$ and $W_B$ on $A$ and $B$, respectively. {\\it Classical-quantum states} are bipartite states that can be written in the form\n","start":18674,"end":18950}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"经典量子态","source_units":[{"raw_tex":"The {\\it swap operator} $F_{AB}$ on $A\\otimes B$ exchanges the two quantum systems, i.e., $F_{AB}(W_A\\otimes W_B)=W_B\\otimes W_A$ for every state $W_A$ and $W_B$ on $A$ and $B$, respectively. {\\it Classical-quantum states} are bipartite states that can be written in the form\n\\begin{align}\nW_{XB}=\\sum_{x=1}^{d_X}p_x\\ket{x}\\bra{x}_X\\otimes W_B^x\n\\end{align}\nfor a probability distribution $\\{p_x\\}_{x=1}^{d_A}$, an orthonormal basis $\\{\\ket{x}_X\\}_{x=1}^{d_X}$ of $\\mathbb{C}^{d_X}$, and states $W_B^x$ on $B$ for $x=1,\\dots,d_X$. We refer to $X$ as the classical part of the bipartite classical-quantum system $XB$.\n\n","start":18674,"end":19292}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"量子信道","source_units":[{"raw_tex":"{\\it Quantum channels} (or in short {\\it channels}) are linear maps $\\mathcal{N}_{A\\to B}:\\mathcal{S}_A\\to\\mathcal{S}_B$ that are trace preserving and completely positive\\footnote{A linear map $\\mathcal{N}_{A\\to B}:\\mathcal{S}_A\\to\\mathcal{S}_B$ is said to be completely positive if $\\mathcal{N}_{A\\to B}\\otimes \\mathcal{I}_C$ is a positive map for every quantum system $C$, where $\\mathcal{I}_C$ denotes the identity map on $\\mathcal{S}_C$} (cp). In particular, they map states from the input system $A$ to states on the output system $B$. We often abbreviate bipartite channels $\\mathcal{I}_A\\otimes\\mathcal{N}_B$ that act trivially on the $A$-system as $\\mathcal{N}_B$, where $\\mathcal{I}_A$ denotes the identity channel on $\\mathcal{S}_A$. The partial trace $\\mathrm{Tr}_B[\\cdot]$ is a channel from $AB$ to $A$ defined via\n","start":19292,"end":20119}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"偏迹与约化态","source_units":[{"raw_tex":"{\\it Quantum channels} (or in short {\\it channels}) are linear maps $\\mathcal{N}_{A\\to B}:\\mathcal{S}_A\\to\\mathcal{S}_B$ that are trace preserving and completely positive\\footnote{A linear map $\\mathcal{N}_{A\\to B}:\\mathcal{S}_A\\to\\mathcal{S}_B$ is said to be completely positive if $\\mathcal{N}_{A\\to B}\\otimes \\mathcal{I}_C$ is a positive map for every quantum system $C$, where $\\mathcal{I}_C$ denotes the identity map on $\\mathcal{S}_C$} (cp). In particular, they map states from the input system $A$ to states on the output system $B$. We often abbreviate bipartite channels $\\mathcal{I}_A\\otimes\\mathcal{N}_B$ that act trivially on the $A$-system as $\\mathcal{N}_B$, where $\\mathcal{I}_A$ denotes the identity channel on $\\mathcal{S}_A$. The partial trace $\\mathrm{Tr}_B[\\cdot]$ is a channel from $AB$ to $A$ defined via\n\\begin{align}\n\\mathrm{Tr}_B[\\cdot]:=\\sum_{x=1}^{d_B}\\big(1_A\\otimes\\bra{x}_B\\big)(\\cdot)\\big(1_A\\otimes\\ket{x}_B\\big),\n\\end{align}\nwhere $1_A$ denotes the identity matrix on $A$, and $\\{\\ket{x}\\}_{x=1}^{d_B}$ an orthonormal basis of $\\mathbb{C}^{d_B}$. For bipartite states $W_{AB}$, we write $W_A=\\mathrm{Tr}_B[W_{AB}]$ for the reduced state on $A$. {\\it Quantum measurements} (or in short {\\it measurements}) are a special case of channels that can be written in the form\n","start":19292,"end":20593}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"测量信道","source_units":[{"raw_tex":"where $1_A$ denotes the identity matrix on $A$, and $\\{\\ket{x}\\}_{x=1}^{d_B}$ an orthonormal basis of $\\mathbb{C}^{d_B}$. For bipartite states $W_{AB}$, we write $W_A=\\mathrm{Tr}_B[W_{AB}]$ for the reduced state on $A$. {\\it Quantum measurements} (or in short {\\it measurements}) are a special case of channels that can be written in the form\n\\begin{align}\n\\mathcal{M}_{A\\to I}(\\cdot):=\\sum_{i\\in I}\\mathrm{Tr}\\left[M_A^i(\\cdot)\\right]\\ket{i}\\bra{i}_I\n\\end{align}\nwith an orthonormal basis $\\{\\ket{i}_I\\}_{i\\in I}$ and $M_A^i\\succeq0$ $\\forall i\\in I$ with $\\sum_{i\\in I}M_A^i=1_A$.\n","start":20250,"end":20833}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"迹范数距离","source_units":[{"raw_tex":"The distance between states is quantified by the metric induced from the Schatten one-norm $\\|X\\|_1:=\\mathrm{Tr}\\left[|X|\\right]$. The distance between channels is quantified by the metric induced from the Diamond norm\n","start":21568,"end":21787}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"钻石范数背景","source_units":[{"raw_tex":"The distance between states is quantified by the metric induced from the Schatten one-norm $\\|X\\|_1:=\\mathrm{Tr}\\left[|X|\\right]$. The distance between channels is quantified by the metric induced from the Diamond norm\n\\begin{align}\n\\|\\mathcal{N}_{A\\to B}\\|_{\\Diamond}:=\\sup_{\\|X\\|_1\\leq1}\\|(\\mathcal{N}_{A\\to B}\\otimes\\mathcal{I}_A)(X_{AA})\\|_1.\n\\end{align}\n\n","start":21568,"end":21928}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"多体置换对称性","source_units":[{"raw_tex":"A multipartite state $W_{AB_1^n}$ on $AB_1^n\\equiv AB_1\\cdots B_n$ is called {\\it symmetric with respect to $A$} if\n\\begin{align}\n(\\mathcal{I}_A\\otimes\\mathcal{U}_{B_1^n}^\\pi)(W_{AB_1^n})=W_{AB_1^n}\\quad\\forall\\pi\\in\\mathfrak{S}_n,\n\\end{align}\nwhere $\\mathfrak{S}_n$ denotes the symmetric group of $n$ elements and\n\\begin{align}\n\\mathcal{U}_{B_1^n}^\\pi(W_{B_1}\\otimes\\cdots\\otimes W_{B_n}):=W_{B_{\\pi^{-1}(1)}}\\otimes\\cdots\\otimes W_{B_{\\pi^{-1}(n)}}.\n\\end{align}\n","start":21928,"end":22392}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"对称可扩张性","source_units":[{"raw_tex":"A bipartite state $W_{AB}$ is called {\\it $n$-extendable} if there exists a multipartite extension $W_{AB_1^n}$, i.e., $\\mathrm{Tr}_{B_2^n}(W_{AB_1^n})=W_{AB}$, that is symmetric with respect to $A$.\n","start":22392,"end":22592}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"量子相对熵","source_units":[{"raw_tex":"The starting point for our proof technique is the use of the chain rule of the conditional mutual information, first used in this context in~\\cite{brandao11} and further exploited in~\\cite{brandao13b}. More precisely, we will use the quantum relative entropy defined as\n\\begin{align}\nD(\\rho\\|\\sigma):=\n\\begin{cases} \\tr(\\rho \\log \\rho) - \\tr(\\rho \\log \\sigma) & \\mbox{if } support(\\rho)\\subseteq support(\\sigma) \\\\ \\infty & \\mbox{otherwise} \\end{cases},\n\\end{align}\nwhere $\\rho$ and $\\sigma$ are quantum states and the logarithm is taken with respect to the basis two. Recall that the support of an operator is defined as the orthogonal complement of its kernel. The following lemma, which can be found in~\\cite{brandao13b}, says that if some classical systems $Z_1^n$ are symmetric with respect to $A$, then conditioning on $Z_1^m$ for some value of $m$ breaks the correlations between $A$ and $Z_{m+1}$. Before stating the lemma, we introduce notation that will be used throughout the section. For a state $\\rho_{A Z}$ with a classical $Z$-system, we write\n","start":26930,"end":27989}]},{"id":"S017","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"条件态与期望","source_units":[{"raw_tex":"where $\\rho$ and $\\sigma$ are quantum states and the logarithm is taken with respect to the basis two. Recall that the support of an operator is defined as the orthogonal complement of its kernel. The following lemma, which can be found in~\\cite{brandao13b}, says that if some classical systems $Z_1^n$ are symmetric with respect to $A$, then conditioning on $Z_1^m$ for some value of $m$ breaks the correlations between $A$ and $Z_{m+1}$. Before stating the lemma, we introduce notation that will be used throughout the section. For a state $\\rho_{A Z}$ with a classical $Z$-system, we write\n\\begin{align}\n\\rho_{A|z}:= \\frac{\\mathrm{Tr}_{Z}\\Big[\\rho_{AZ} \\left(1_{A} \\otimes \\ket{z}\\bra{z}\\right)\\Big]}{\\mathrm{Tr}\\Big[\\rho_{AZ} \\left(1_{A} \\otimes \\ket{z}\\bra{z}\\right)\\Big]}.\n\\end{align}\nWe simply write $\\exc{z_{1}^{m}}{\\cdot}$ for the expectation over the choices of $z_1^m$ and the distribution will be clear from the context.\n","start":27396,"end":28329}]},{"id":"S018","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"测量失真动机","source_units":[{"raw_tex":"To prove the de Finetti theorem, we will crucially make use of so-called informationally complete measurements for which the loss in distinguishability, or {\\it distortion}, can be bounded.\n\n","start":29993,"end":30184}]},{"id":"S019","semantic_type":"lemma","kind":"lemma","labels":["lem:product-ic-meas"],"member_ids":["S019"],"declared_external":true,"declaration_text":["原始先决条件明确许可 B0039，出处为 brandao13b, Lemma 14；本文没有提供其证明。"],"source_only":false,"title":"乘积测量失真界","source_units":[{"raw_tex":"\\begin{lemma}{\\cite[Lemma 14]{brandao13b}}\\label{lem:product-ic-meas}\nThere exist a product measurement $\\mathcal{M}_A\\otimes\\mathcal{M}_B$ with finitely many outcomes such that for any Hermitian and traceless matrix $\\xi_{AB}$ on $A\\otimes B$, we have \n\\begin{align}\n\\|(\\mathcal{M}_{A} \\otimes \\mathcal{M}_{B})(\\xi_{AB})\\|_1 \\geq \\frac{1}{18\\sqrt{d_A d_B}}\\| \\xi_{AB}\\|_{1}.\n\\end{align}\n\\end{lemma}\n","start":30184,"end":30584}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":["eq:def-f1"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"乘积最小失真","source_units":[{"raw_tex":"\nThis \\cite[Lemma 14]{brandao13b} follows from the methods in \\cite{lancien13}. More generally, we define the {\\it minimal distortion} for the bipartite system $A \\otimes B$ as\n\\begin{align}\\label{eq:def-f1}\nf(A, B):=\\inf_{\\cM_{A}, \\cM_{B}}\\max_{\\substack{\\xi_{AB}^{\\dagger} = \\xi_{AB}\\\\ \\xi_A = 0, \\xi_B = 0}} \\frac{\\| \\xi_{AB} \\|_1}{\\| (\\cM_{A} \\otimes \\cM_B)(\\xi_{AB})\\|_1},\n\\end{align}\nwhere the infimum is over all product measurements on $AB$. In this notation, Lemma~\\ref{lem:product-ic-meas} shows that\n","start":30584,"end":31095},{"raw_tex":"Note that in the definition of $f(A,B)$ we restricted the maximization to matrices satisfying $\\xi_{A} = 0$ and $\\xi_{B} = 0$ because this is sufficient for us.\n\n","start":31153,"end":31315}]},{"id":"S021","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"乘积失真的维数界","source_units":[{"raw_tex":"where the infimum is over all product measurements on $AB$. In this notation, Lemma~\\ref{lem:product-ic-meas} shows that\n\\begin{align}\nf(A, B) \\leq 18 \\sqrt{d_A d_B}.\n\\end{align}\nNote that in the definition of $f(A,B)$ we restricted the maximization to matrices satisfying $\\xi_{A} = 0$ and $\\xi_{B} = 0$ because this is sufficient for us.\n","start":30974,"end":31314}]},{"id":"S022","semantic_type":"definition","kind":"definition","labels":["eq:def-f2"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"带旁信息最小失真","source_units":[{"raw_tex":"A drawback of Lemma~\\ref{lem:product-ic-meas} is that the distortion depends on the dimension $d_A$. More generally, we define the {\\it minimal distortion with side information} for a system $B$ as\n\\begin{align}\\label{eq:def-f2}\nf(B|\\cdot):=\\inf_{\\cM_B} \\sup_{\\substack{\\xi_{AB}^\\dagger=\\xi_{AB}\\\\ \\xi_{A} = 0, \\xi_B = 0}} \\frac{\\| \\xi_{AB} \\|_1}{\\| (\\cI_{A} \\otimes \\cM_B)(\\xi_{AB})\\|_1},\n\\end{align}\nwhere the infimum is over all measurements on $B$ and the supremum is over all finite-dimensional systems $A$. In Lemma~\\ref{lem:ic-meas-side-info} we give an elementary proof that\n","start":31315,"end":31898}]},{"id":"S023","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"旁信息失真的初等界","source_units":[{"raw_tex":"where the infimum is over all measurements on $B$ and the supremum is over all finite-dimensional systems $A$. In Lemma~\\ref{lem:ic-meas-side-info} we give an elementary proof that\n\\begin{align}\nf(B|\\cdot) \\leq d_B^2 (d_B+1)\n\\end{align}\nusing state two-designs and properties of weighted non-commutative $L_p$-spaces. In fact, after completion of our work we realised that methods from operator space theory even give the stronger bound\n","start":31717,"end":32154}]},{"id":"S024","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S024"],"declared_external":true,"declaration_text":["原始先决条件列出的 brandao13c, Equation 66；本段没有包含该外部论证。"],"source_only":false,"title":"旁信息失真的较强界","source_units":[{"raw_tex":"using state two-designs and properties of weighted non-commutative $L_p$-spaces. In fact, after completion of our work we realised that methods from operator space theory even give the stronger bound\n\\begin{align}\nf(B|\\cdot) \\leq \\sqrt{18d_B^3},\n\\end{align}\nwhich is discussed in \\cite[Equation 66]{brandao13c}. We leave it as an open question to determine the exact dimensional dependence of the minimal distortion with side information.\n","start":31954,"end":32393}]},{"id":"S025","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S025"],"declared_external":true,"declaration_text":["原始先决条件中的 scott08, Corollary 5.3，以及原文说明的酉两设计到态两设计的构造。"],"source_only":false,"title":"有限两设计存在性","source_units":[{"raw_tex":"We note that the existence of such two-designs is known for any dimension, see e.g., \\cite[Corollary 5.3]{scott08} for unitary two-designs and applying these unitaries to any fixed state leads to a state two-design.\n","start":124771,"end":124987}]},{"id":"S026","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S026"],"declared_external":false,"source_only":true,"title":"积态混合的对称扩张","source_units":[{"raw_tex":"Now, to approximate the set of separable states within the set of bipartite states is a ubiquitous but hard problem in quantum information theory (see, e.g., \\cite{Barak12}). Nevertheless, as realized in~\\cite{Doherty04} the set of separable states can be approximated by the sum-of-squares hierarchies of Lasserre~\\cite{Lasserre00} and Parrilo~\\cite{Parrilo2003}. This lead to the {\\it semidefinite programming hierarchy} of Doherty-Parrilo-Spedalieri (DPS), which is extensively employed to characterize quantum correlations in quantum information theory~\\cite{Doherty02}. The underlying idea of the DPS hierarchy is that separable states $W_{AB}=\\sum_i p_i W_A^i \\otimes W_B^i$ on $A\\otimes B$, where $\\{p_i\\}_{i\\in I}$ is a probability distribution, are {\\it $n$-extendible} to $W_{AB_1^n}=\\sum_i p_iW_A^i \\otimes (W_B^i)^{\\otimes n}$ on $A\\otimes B^{\\otimes n}$ for any $n$, such that we have for any permutation $\\pi$ that\\footnote{Here and henceforth we use the notation $B_i^j$ to denote the systems $B_i\\otimes\\cdots\\otimes B_j$, which should be interpreted as empty if $i>j$.}\n\\begin{align}\nW_{AB_1^n}=\\left(\\mathcal{I}_A\\otimes\\mathcal{U}_{B_1^n}^\\pi\\right)(W_{AB_1^n})\n\\end{align}\n","start":10101,"end":11294}]},{"id":"S027","semantic_type":"lemma","kind":"lemma","labels":["lem:conditioning-breaks-correlations"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"条件化消除相关性","source_units":[{"raw_tex":"\\begin{lemma}\\cite{brandao13b}\\label{lem:conditioning-breaks-correlations}\nLet $\\rho_{A Z_1^n}$ be a classical-quantum state with the $Z_1^n$-systems classical and $\\mathcal{U}^\\pi_{Z_1^n}(\\rho_{A Z_1^n})=\\rho_{A Z_1^n}$ for all $\\pi\\in\\mathfrak{S}_n$. Then, there exists $0\\leq m < n$ such that\n\\begin{align}\n\\exc{z_{1}^{m}}{D(\\rho_{AZ_{m+1}|z_1^m} \\| \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m})} &\\leq \\frac{\\log d_A}{n}\n\\end{align}\nas well as\n\\begin{align}\n\\exc{z_{1}^{m}}{\\| \\rho_{AZ_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m} \\|_1^2} &\\leq \\frac{(2 \\ln 2) \\log d_A}{n},\n\\end{align}\nwhere $\\ln(\\cdot)$ denotes the natural logarithm.\n\\end{lemma}","start":28330,"end":28995}]},{"id":"S028","semantic_type":"derived_fact","kind":"derived_fact","labels":["lem:conditioning-breaks-correlations"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"条件相对熵界","source_units":[{"raw_tex":"\\begin{lemma}\\cite{brandao13b}\\label{lem:conditioning-breaks-correlations}\nLet $\\rho_{A Z_1^n}$ be a classical-quantum state with the $Z_1^n$-systems classical and $\\mathcal{U}^\\pi_{Z_1^n}(\\rho_{A Z_1^n})=\\rho_{A Z_1^n}$ for all $\\pi\\in\\mathfrak{S}_n$. Then, there exists $0\\leq m < n$ such that\n\\begin{align}\n\\exc{z_{1}^{m}}{D(\\rho_{AZ_{m+1}|z_1^m} \\| \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m})} &\\leq \\frac{\\log d_A}{n}\n\\end{align}\n","start":28330,"end":28766}]},{"id":"S029","semantic_type":"derived_fact","kind":"derived_fact","labels":["lem:conditioning-breaks-correlations"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"条件迹范数平方界","source_units":[{"raw_tex":"\\begin{lemma}\\cite{brandao13b}\\label{lem:conditioning-breaks-correlations}\nLet $\\rho_{A Z_1^n}$ be a classical-quantum state with the $Z_1^n$-systems classical and $\\mathcal{U}^\\pi_{Z_1^n}(\\rho_{A Z_1^n})=\\rho_{A Z_1^n}$ for all $\\pi\\in\\mathfrak{S}_n$. Then, there exists $0\\leq m < n$ such that\n","start":28330,"end":28626},{"raw_tex":"as well as\n\\begin{align}\n\\exc{z_{1}^{m}}{\\| \\rho_{AZ_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m} \\|_1^2} &\\leq \\frac{(2 \\ln 2) \\log d_A}{n},\n\\end{align}\nwhere $\\ln(\\cdot)$ denotes the natural logarithm.\n","start":28766,"end":28984}]},{"id":"S030","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"消相关引理完整证明","source_units":[{"raw_tex":"\\begin{proof}\nFor the quantum mutual information we have $I\\left(A:Z_1^n\\right)_{\\rho}:=D(\\rho_{AZ_1^n}\\|\\rho_A\\otimes \\rho_{Z_1^n}) \\leq \\log d_A$ as well as (see, e.g., \\cite[Chapter 11]{nielsen00})\n\\begin{align}\nI\\left(A:Z_1^n\\right)_{\\rho}=\\sum_{m=0}^{n-1} I(A : Z_{m+1} | Z_1^m)_{\\rho}\n\\end{align}\nfor the quantum conditional mutual information $I(A : Z_{m+1} | Z_1^m)_\\rho:=I(A:Z_1^{m+1})_\\rho-I(A:Z_1^m)_\\rho$. As a result, there exists an $m \\in \\{0, \\cdots, n-1\\}$ such that $I(A : Z_{m+1} | Z_1^m)_{\\rho} \\leq \\frac{\\log d_A}{n}$, which implies\n\\begin{align}\n\\exc{z_1^m}{I(A:Z_{m+1})_{\\rho_{AZ_{m+1}|z_1^m}}} \\leq \\frac{\\log d_A}{n},\n\\end{align}\nwhere we used $I(A : Z_{m+1} | Z_1^m)_{\\rho}=\\exc{z_1^m}{I(A:Z_{m+1})_{\\rho_{AZ_{m+1}|z_1^m}}}$, which holds since the conditioning systems are classical. The second statement then follows directly from Pinsker's inequality $D(\\rho\\|\\sigma)\\geq\\frac{1}{2\\ln 2}\\left\\|\\rho-\\sigma\\right\\|_1^2$ \\cite[Theorem 5.38]{watrous-book}.\n\\end{proof}","start":28997,"end":29991}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"链式展开选择条件长度","source_units":[{"raw_tex":"\\begin{proof}\nFor the quantum mutual information we have $I\\left(A:Z_1^n\\right)_{\\rho}:=D(\\rho_{AZ_1^n}\\|\\rho_A\\otimes \\rho_{Z_1^n}) \\leq \\log d_A$ as well as (see, e.g., \\cite[Chapter 11]{nielsen00})\n\\begin{align}\nI\\left(A:Z_1^n\\right)_{\\rho}=\\sum_{m=0}^{n-1} I(A : Z_{m+1} | Z_1^m)_{\\rho}\n\\end{align}\nfor the quantum conditional mutual information $I(A : Z_{m+1} | Z_1^m)_\\rho:=I(A:Z_1^{m+1})_\\rho-I(A:Z_1^m)_\\rho$. As a result, there exists an $m \\in \\{0, \\cdots, n-1\\}$ such that $I(A : Z_{m+1} | Z_1^m)_{\\rho} \\leq \\frac{\\log d_A}{n}$, which implies\n","start":28997,"end":29552}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"经典条件化转为期望","source_units":[{"raw_tex":"for the quantum conditional mutual information $I(A : Z_{m+1} | Z_1^m)_\\rho:=I(A:Z_1^{m+1})_\\rho-I(A:Z_1^m)_\\rho$. As a result, there exists an $m \\in \\{0, \\cdots, n-1\\}$ such that $I(A : Z_{m+1} | Z_1^m)_{\\rho} \\leq \\frac{\\log d_A}{n}$, which implies\n\\begin{align}\n\\exc{z_1^m}{I(A:Z_{m+1})_{\\rho_{AZ_{m+1}|z_1^m}}} \\leq \\frac{\\log d_A}{n},\n\\end{align}\nwhere we used $I(A : Z_{m+1} | Z_1^m)_{\\rho}=\\exc{z_1^m}{I(A:Z_{m+1})_{\\rho_{AZ_{m+1}|z_1^m}}}$, which holds since the conditioning systems are classical. The second statement then follows directly from Pinsker's inequality $D(\\rho\\|\\sigma)\\geq\\frac{1}{2\\ln 2}\\left\\|\\rho-\\sigma\\right\\|_1^2$ \\cite[Theorem 5.38]{watrous-book}.\n","start":29300,"end":29980}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"Pinsker 转为迹范数","source_units":[{"raw_tex":"where we used $I(A : Z_{m+1} | Z_1^m)_{\\rho}=\\exc{z_1^m}{I(A:Z_{m+1})_{\\rho_{AZ_{m+1}|z_1^m}}}$, which holds since the conditioning systems are classical. The second statement then follows directly from Pinsker's inequality $D(\\rho\\|\\sigma)\\geq\\frac{1}{2\\ln 2}\\left\\|\\rho-\\sigma\\right\\|_1^2$ \\cite[Theorem 5.38]{watrous-book}.\n\\end{proof}","start":29653,"end":29991}]},{"id":"S034","semantic_type":"theorem","kind":"theorem","labels":["thm:main-deFinetti"],"member_ids":["S034","S037","S038","S039","S040","S041","S042","S043","S044"],"declared_external":false,"source_only":false,"title":"带线性约束的 de Finetti 定理","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm:main-deFinetti}\nLet $\\rho_{AB_1^n}$ be a quantum state, $\\Lambda_{A\\to C_A},\\Gamma_{B\\to C_B}$ linear maps, and $X_{C_A},Y_{C_B}$ matrices such that\n\\begin{align}\n\\mathcal{U}_{B_1^n}^\\pi(\\rho_{AB_1^n})&=\\rho_{AB_1^n}\\;\\forall\\pi\\in\\mathfrak{S}_n\\qquad\\text{symmetric with respect to $A$}\\\\\n\\Lambda_{A\\to C_A}(\\rho_{AB_1^n})&=X_{C_A}\\otimes\\rho_{B_1^n}\\qquad\\quad\\;\\;\\text{linear constraint on $A$}\\\\\n\\Gamma_{B_n\\to C_B}(\\rho_{B_1^n})&=\\rho_{B_1^{n-1}}\\otimes Y_{C_B}\\qquad\\quad\\text{linear constraint on $B$}.\n\\end{align}\nThen, we have that\n\\begin{align}\n\\left\\|\\rho_{AB}-\\sum_{i\\in I}p_i\\sigma^i_{A}\\otimes\\omega^i_B\\right\\|_1\\leq&\\;\\min\\big\\{ f(A,B) , f(B|\\cdot) \\big\\}\\sqrt{ \\frac{(2 \\ln 2) \\log \\left(d_A\\right)}{n}}\n\\end{align}\nwith $\\{p_i\\}_{i\\in I}$ a probability distribution, $\\rho_{AB}=\\tr_{B_2^n}\\left[\\rho_{AB_1^n}\\right]$, and quantum states $\\sigma^i_A,\\omega_B^i$ such that for $i\\in I$:\n\\begin{align}\n\\Lambda_{A\\to C_A}\\left(\\sigma^i_A\\right)=X_{C_A}\\quad\\text{and}\\quad\\Gamma_{B\\to C_B}\\left(\\omega^i_B\\right)=Y_{C_B}.\n\\end{align}\nAs stated in Section~\\ref{sec:proof-tools}, we can, e.g., take $f(A,B)\\leq18\\sqrt{d_A d_B}$ or $f(B|\\cdot)\\leq\\sqrt{18d_B^3}$.\n\\end{theorem}","start":32631,"end":33845},{"raw_tex":"\\begin{proof}\nLet $\\cM_{B}$ be a measurement of the $B$ system and call the outcome system $Z$. Consider the state $\\rho_{AZ_1^n}$ obtained by measuring all the $B$ systems with $\\cM_{B}$. This distribution is symmetric with respect to $A$ and so we can apply Lemma~\\ref{lem:conditioning-breaks-correlations}. We find that there exists an $m \\in \\{0, \\cdots, n-1\\}$ such that\n\\begin{align}\n\\exc{z_1^m}{\\| \\rho_{AZ_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m} \\|_1^2} \\leq \\frac{(2 \\ln 2) \\log d_{A}}{n}.\n\\end{align}\n","start":33847,"end":34377},{"raw_tex":"\\begin{proof}\nLet $\\cM_{B}$ be a measurement of the $B$ system and call the outcome system $Z$. Consider the state $\\rho_{AZ_1^n}$ obtained by measuring all the $B$ systems with $\\cM_{B}$. This distribution is symmetric with respect to $A$ and so we can apply Lemma~\\ref{lem:conditioning-breaks-correlations}. We find that there exists an $m \\in \\{0, \\cdots, n-1\\}$ such that\n\\begin{align}\n\\exc{z_1^m}{\\| \\rho_{AZ_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m} \\|_1^2} \\leq \\frac{(2 \\ln 2) \\log d_{A}}{n}.\n\\end{align}\nNote that we have for any $z_1^m$, $\\rho_{AZ_{m+1}|z_1^m} = (\\cI_{A} \\otimes \\cM_{B})(\\rho_{AB_{m+1}|z_1^m})$ and correspondingly $\\rho_{Z_{m+1}|z_1^m} = \\cM_{B}(\\rho_{B_{m+1}|z_1^m})$. Now, we choose the measurement $\\cM_{B}$ to be as in Lemma~\\ref{lem:ic-meas-side-info} and achieving $f(B|\\cdot)$ in \\eqref{eq:def-f2}, we get that $\\| \\xi_{AB} \\|_1^2 \\leq f(B|\\cdot)^2 \\| (\\cI_{A} \\otimes \\cM_{B})(\\xi_{AB}) \\|_1^2$, where $\\xi_{AB} = \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}$. As a result, we have\n\\begin{align}\n\\exc{z_1^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1^2} \\leq f(B|\\cdot)^2\\frac{(2 \\ln 2) \\log d_{A}}{n}.\n\\end{align}\nBut note we can also choose measurements $\\cM_{A}$ and $\\cM_{B}$ achieving $f(A,B)$ in \\eqref{eq:def-f1}. In this case,\n\\begin{align} \n\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1^2 &\\leq f(A,B)^2\\| (\\cM_A \\otimes \\cM_{B})(\\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}) \\|_1^2 \\\\\n&\\leq f(A,B)^2 \\| (\\cI_A \\otimes \\cM_{B})(\\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}) \\|_1^2 \\\\\n&= f(A,B)^2 \\| \\rho_{AZ_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m} \\|_1^2 ,\n\\end{align}\nwhere we used the fact that the trace norm cannot increase when applying the quantum channel $\\cM_{A}$ \\cite[Theorem 3.39]{watrous-book}. As a result, we get\n\\begin{align}\n\\exc{z_1^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1^2} \\leq f(A,B)^2 \\frac{(2 \\ln 2) \\log d_{A}}{n}.\n\\end{align}\nNow, using the convexity of the square function, we get\n\\begin{align}\n\\exc{z_{1}^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1}&\\leq \\sqrt{\\exc{z_{1}^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|^2_1}} \\\\\n&\\leq  \\min\\big\\{f(A, B), f(B|\\cdot)\\big\\}\\sqrt{\\frac{(2 \\ln 2) \\log d_{A}}{n}}.\n\\end{align}\nThen, using the convexity of the norm and the fact that $\\exc{z_1^m}{\\rho_{AB_{m+1}|z_1^m}} = \\rho_{AB_{m+1}}$, we obtain\n\\begin{align}\n\\left\\| \\rho_{AB_{m+1}} - \\exc{z_1^m}{\\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}} \\right\\|_1 &\\leq \\min\\big\\{f(A,B), f(B|\\cdot)\\big\\}\\sqrt{\\frac{(2 \\ln 2) \\log d_{A}}{n}}.\n\\end{align}\nThe state $\\exc{z_1^m}{\\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}}$ corresponds to our candidate mixture of product states. It now remains to show that all the states in the mixture satisfy the linear constraints. Indeed we have for any $z_1^m$, writing $M_{z}$ for matrices of the measurement $\\cM_{B}$,\n\\begin{align}\n\\Lambda_{A \\to C_{A}}(\\rho_{A|z_1^m})\n&= \\frac{\\tr_{B_1^m}\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m}) \\Lambda_{A \\to C_{A}}(\\rho_{A B_1^m})\\Big]}{\\tr\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m})\\rho_{A B_1^m}\\Big]} \\\\\n&= \\frac{\\tr_{B_1^m}\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m}) (X_{C_A} \\otimes \\rho_{B_1^m}) \\Big]}{\\tr\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m})\\rho_{A B_1^m}\\Big]} \\\\\n&= X_{C_A},\n\\end{align}\nand similarly\n\\begin{align}\n\\Gamma_{B \\to C_{B}}(\\rho_{B_{m+1}|z_1^m})\n&= \\frac{\\tr_{B_1^m}\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{C_{B}} ) \\Gamma_{B_{m+1} \\to C_{B}}(\\rho_{B_1^{m+1}})\\Big]}{\\tr\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{B_{m+1}} )\\rho_{B_1^{m+1}}\\Big]} \\\\\n&= \\frac{\\tr_{B_1^m}\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{B_{m+1}} ) (\\rho_{B_1 \\cdots B_{m}} \\otimes Y_{C_B})\\Big]}{\\tr\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{B_{m+1}} )\\rho_{B_1^{m+1}}\\Big]} \\\\\n&= Y_{C_B}.\n\\end{align}\n\\end{proof}","start":33847,"end":37993},{"raw_tex":"Note that we have for any $z_1^m$, $\\rho_{AZ_{m+1}|z_1^m} = (\\cI_{A} \\otimes \\cM_{B})(\\rho_{AB_{m+1}|z_1^m})$ and correspondingly $\\rho_{Z_{m+1}|z_1^m} = \\cM_{B}(\\rho_{B_{m+1}|z_1^m})$. Now, we choose the measurement $\\cM_{B}$ to be as in Lemma~\\ref{lem:ic-meas-side-info} and achieving $f(B|\\cdot)$ in \\eqref{eq:def-f2}, we get that $\\| \\xi_{AB} \\|_1^2 \\leq f(B|\\cdot)^2 \\| (\\cI_{A} \\otimes \\cM_{B})(\\xi_{AB}) \\|_1^2$, where $\\xi_{AB} = \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}$. As a result, we have\n\\begin{align}\n\\exc{z_1^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1^2} \\leq f(B|\\cdot)^2\\frac{(2 \\ln 2) \\log d_{A}}{n}.\n\\end{align}\n","start":34377,"end":35072},{"raw_tex":"But note we can also choose measurements $\\cM_{A}$ and $\\cM_{B}$ achieving $f(A,B)$ in \\eqref{eq:def-f1}. In this case,\n\\begin{align} \n\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1^2 &\\leq f(A,B)^2\\| (\\cM_A \\otimes \\cM_{B})(\\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}) \\|_1^2 \\\\\n&\\leq f(A,B)^2 \\| (\\cI_A \\otimes \\cM_{B})(\\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}) \\|_1^2 \\\\\n&= f(A,B)^2 \\| \\rho_{AZ_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{Z_{m+1}|z_1^m} \\|_1^2 ,\n\\end{align}\nwhere we used the fact that the trace norm cannot increase when applying the quantum channel $\\cM_{A}$ \\cite[Theorem 3.39]{watrous-book}. As a result, we get\n\\begin{align}\n\\exc{z_1^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1^2} \\leq f(A,B)^2 \\frac{(2 \\ln 2) \\log d_{A}}{n}.\n\\end{align}\n","start":35072,"end":35951},{"raw_tex":"Now, using the convexity of the square function, we get\n\\begin{align}\n\\exc{z_{1}^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|_1}&\\leq \\sqrt{\\exc{z_{1}^m}{\\| \\rho_{AB_{m+1}|z_1^m} - \\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m} \\|^2_1}} \\\\\n&\\leq  \\min\\big\\{f(A, B), f(B|\\cdot)\\big\\}\\sqrt{\\frac{(2 \\ln 2) \\log d_{A}}{n}}.\n\\end{align}\n","start":35951,"end":36313},{"raw_tex":"Then, using the convexity of the norm and the fact that $\\exc{z_1^m}{\\rho_{AB_{m+1}|z_1^m}} = \\rho_{AB_{m+1}}$, we obtain\n\\begin{align}\n\\left\\| \\rho_{AB_{m+1}} - \\exc{z_1^m}{\\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}} \\right\\|_1 &\\leq \\min\\big\\{f(A,B), f(B|\\cdot)\\big\\}\\sqrt{\\frac{(2 \\ln 2) \\log d_{A}}{n}}.\n\\end{align}\nThe state $\\exc{z_1^m}{\\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}}$ corresponds to our candidate mixture of product states. It now remains to show that all the states in the mixture satisfy the linear constraints. Indeed we have for any $z_1^m$, writing $M_{z}$ for matrices of the measurement $\\cM_{B}$,\n","start":36313,"end":36940},{"raw_tex":"The state $\\exc{z_1^m}{\\rho_{A|z_1^m} \\otimes \\rho_{B_{m+1}|z_1^m}}$ corresponds to our candidate mixture of product states. It now remains to show that all the states in the mixture satisfy the linear constraints. Indeed we have for any $z_1^m$, writing $M_{z}$ for matrices of the measurement $\\cM_{B}$,\n\\begin{align}\n\\Lambda_{A \\to C_{A}}(\\rho_{A|z_1^m})\n&= \\frac{\\tr_{B_1^m}\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m}) \\Lambda_{A \\to C_{A}}(\\rho_{A B_1^m})\\Big]}{\\tr\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m})\\rho_{A B_1^m}\\Big]} \\\\\n&= \\frac{\\tr_{B_1^m}\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m}) (X_{C_A} \\otimes \\rho_{B_1^m}) \\Big]}{\\tr\\Big[(1_{A}\\otimes M_{z_1} \\otimes \\cdots \\otimes M_{z_m})\\rho_{A B_1^m}\\Big]} \\\\\n&= X_{C_A},\n\\end{align}\n","start":36634,"end":37426},{"raw_tex":"and similarly\n\\begin{align}\n\\Gamma_{B \\to C_{B}}(\\rho_{B_{m+1}|z_1^m})\n&= \\frac{\\tr_{B_1^m}\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{C_{B}} ) \\Gamma_{B_{m+1} \\to C_{B}}(\\rho_{B_1^{m+1}})\\Big]}{\\tr\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{B_{m+1}} )\\rho_{B_1^{m+1}}\\Big]} \\\\\n&= \\frac{\\tr_{B_1^m}\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{B_{m+1}} ) (\\rho_{B_1 \\cdots B_{m}} \\otimes Y_{C_B})\\Big]}{\\tr\\Big[(M_{z_1} \\otimes \\cdots \\otimes M_{z_m} \\otimes 1_{B_{m+1}} )\\rho_{B_1^{m+1}}\\Big]} \\\\\n&= Y_{C_B}.\n\\end{align}\n\\end{proof}","start":37426,"end":37993}]},{"id":"S035","semantic_type":"assumption","kind":"assumption","labels":["thm:main-deFinetti"],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"受约束扩张的假设","source_units":[{"raw_tex":"Let $\\rho_{AB_1^n}$ be a quantum state, $\\Lambda_{A\\to C_A},\\Gamma_{B\\to C_B}$ linear maps, and $X_{C_A},Y_{C_B}$ matrices such that\n\\begin{align}\n\\mathcal{U}_{B_1^n}^\\pi(\\rho_{AB_1^n})&=\\rho_{AB_1^n}\\;\\forall\\pi\\in\\mathfrak{S}_n\\qquad\\text{symmetric with respect to $A$}\\\\\n\\Lambda_{A\\to C_A}(\\rho_{AB_1^n})&=X_{C_A}\\otimes\\rho_{B_1^n}\\qquad\\quad\\;\\;\\text{linear constraint on $A$}\\\\\n\\Gamma_{B_n\\to C_B}(\\rho_{B_1^n})&=\\rho_{B_1^{n-1}}\\otimes Y_{C_B}\\qquad\\quad\\text{linear constraint on $B$}.\n\\end{align}\n","start":32673,"end":33179}]},{"id":"S036","semantic_type":"derived_fact","kind":"derived_fact","labels":["thm:main-deFinetti"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"定理中的示例常数","source_units":[{"raw_tex":"As stated in Section~\\ref{sec:proof-tools}, we can, e.g., take $f(A,B)\\leq18\\sqrt{d_A d_B}$ or $f(B|\\cdot)\\leq\\sqrt{18d_B^3}$.\n\\end{theorem}","start":33705,"end":33845}]},{"id":"S045","semantic_type":"lemma","kind":"lemma","labels":["lem:ic-meas-side-info"],"member_ids":["S045","S047","S048","S049","S050","S051","S052","S053","S054","S055"],"declared_external":false,"source_only":false,"title":"两设计的旁信息失真界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:ic-meas-side-info}\nConsider a state two-design on $B$, i.e., a set of rank-one projectors $\\{P_z\\}_{z \\in \\{1, \\dots, t\\}}$ such that $\\frac{1}{t} \\sum_{z=1}^t P_z \\otimes P_z = \\frac{2 P^{\\mathrm{sym}}}{d_B (d_B+1)}$, where $P^{\\mathrm{sym}}$ denotes the projector onto the symmetric subspace of $B \\otimes B$. Let $\\mathcal{M}_B$ be the measurement defined as\n\\begin{align}\n\\mathcal{M}_B(X) = \\sum_{z} \\frac{d_B}{t} \\cdot\\mathrm{Tr}\\big[P_z X\\big] |z\\rangle\\langle z|,\n\\end{align}\nand $\\xi_{AB}$ be a Hermitian matrix on $A\\otimes B$. Then, we have that\n\\begin{align}\n\\| (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\|_1\\geq \\frac{1}{d_B^2 (d_B+1)} \\| \\xi_{AB} \\|_{1}.\n\\end{align}\n\\end{lemma}","start":124046,"end":124769},{"raw_tex":"\\begin{proof}\nFor any full rank quantum state $\\sigma_{A}$, we have by a H\\\"older type inequality for $\\sigma$-weighted Schatten norms that (see, e.g., \\cite{OLKIEWICZ1999246} or \\cite{beigi13})\n\\begin{align}\n\\| (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\|_1 \\geq  \\frac{\\left\\| \\sigma_{A}^{-1/4} (\\mathcal{I}_{A} \\otimes \\mathcal{M})(\\xi_{AB}) \\sigma_{A}^{-1/4} \\right\\|^2_{2}}{\\left\\| \\sigma_{A}^{-1/2} (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\sigma_{A}^{-1/2} \\right\\|_{\\infty}}.\n\\end{align}\nFor example, the above inequality can be obtained using \\cite[Corollary 3]{beigi13} with the operator $\\sigma_{A}^{-1/2} (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\sigma_{A}^{-1/2}$, weight $\\sigma$, and $p_\\theta = 2$, $\\theta = 1/2$, $p_0 = 1$, $p_1 = \\infty$. We note that this particular H\\\"older type inequality for $\\sigma$-weighted norms is elementary and follows easily from the usual H\\\"older inequality, but one way of potentially improving the dimension dependence in Lemma~\\ref{lem:ic-meas-side-info} might be to use another H\\\"older inequality, in particular the $(1,4)$ inequality.\n","start":124988,"end":126105},{"raw_tex":"\\begin{proof}\nFor any full rank quantum state $\\sigma_{A}$, we have by a H\\\"older type inequality for $\\sigma$-weighted Schatten norms that (see, e.g., \\cite{OLKIEWICZ1999246} or \\cite{beigi13})\n\\begin{align}\n\\| (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\|_1 \\geq  \\frac{\\left\\| \\sigma_{A}^{-1/4} (\\mathcal{I}_{A} \\otimes \\mathcal{M})(\\xi_{AB}) \\sigma_{A}^{-1/4} \\right\\|^2_{2}}{\\left\\| \\sigma_{A}^{-1/2} (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\sigma_{A}^{-1/2} \\right\\|_{\\infty}}.\n\\end{align}\nFor example, the above inequality can be obtained using \\cite[Corollary 3]{beigi13} with the operator $\\sigma_{A}^{-1/2} (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\sigma_{A}^{-1/2}$, weight $\\sigma$, and $p_\\theta = 2$, $\\theta = 1/2$, $p_0 = 1$, $p_1 = \\infty$. We note that this particular H\\\"older type inequality for $\\sigma$-weighted norms is elementary and follows easily from the usual H\\\"older inequality, but one way of potentially improving the dimension dependence in Lemma~\\ref{lem:ic-meas-side-info} might be to use another H\\\"older inequality, in particular the $(1,4)$ inequality.\n\nHenceforth, we abbreviate $d\\equiv d_B$. To further bound the numerator, letting $\\tilde{\\xi}_{AB}:=\\sigma_{A}^{-1/4} \\xi_{AB} \\sigma_{A}^{-1/4}$ we get\n\\begin{align}\n\\| (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\tilde{\\xi}_{AB}) \\|^2_{2}\n&= \\left\\| \\sum_{z} \\frac{d}{t} |z\\rangle \\langle z| \\otimes  \\mathrm{Tr}_{B}\\left[(1_{A} \\otimes P_z) \\tilde{\\xi}_{AB}\\right] \\right\\|_2^2 \\\\\n&= \\sum_{z}  \\frac{d^2}{t^2} \\mathrm{Tr}\\left[\\mathrm{Tr}_{B}\\left[(1_{A} \\otimes P_z) \\tilde{\\xi}_{AB}\\right] \\otimes \\mathrm{Tr}_{\\bar{B}}\\left[(1_{\\bar{A}} \\otimes P_z) \\tilde{\\xi}_{\\bar{A}\\bar{B}}\\right]^{\\dagger} F_{A\\bar{A}} \\right] \\\\\n&= \\sum_{z} \\frac{d^2}{t^2} \\mathrm{Tr}\\left[\\left(\\left((1_{A} \\otimes P_z) \\tilde{\\xi}_{AB}\\right) \\otimes \\left((1_{\\bar{A}} \\otimes P_z) \\tilde{\\xi}_{\\bar{A}\\bar{B}}\\right)^{\\dagger}\\right)\\left(F_{A\\bar{A}} \\otimes1_{B\\bar{B}}\\right)\\right] \\\\\n&= \\frac{d^2}{t^2} \\mathrm{Tr}\\left[\\left(\\tilde{\\xi}_{AB} \\otimes \\tilde{\\xi}_{\\bar{A}\\bar{B}}^{\\dagger}\\right)\\left(\\sum_{z} (1_{A\\bar{A}} \\otimes P_z \\otimes P_{z})\\right)\\left(F_{A\\bar{A}} \\otimes1_{B \\bar{B}}\\right)\\right] \\\\\n&= \\frac{1}{t} \\frac{d^2}{d(d+1)} \\mathrm{Tr}\\left[\\left(\\tilde{\\xi}_{AB} \\otimes \\tilde{\\xi}_{\\bar{A}\\bar{B}}^{\\dagger}\\right) (1_{A\\bar{A}} \\otimes (1_{B\\bar{B}} + F_{B\\bar{B}}))\\left(F_{A\\bar{A}} \\otimes1_{B \\bar{B}}\\right)\\right] \\\\\n&= \\frac{1}{t} \\frac{d^2}{d(d+1)} \\Big(\\underbrace{\\mathrm{Tr}\\left[\\tilde{\\xi}_{A} \\tilde{\\xi}_{A}^{\\dagger}\\right]}_{\\geq 0} + \\underbrace{\\mathrm{Tr}\\left[\\tilde{\\xi}_{AB} \\tilde{\\xi}_{AB}^{\\dagger}\\right]}_{=\\left\\|\\tilde{\\xi}_{AB}\\right\\|_2^2} \\Big) \\\\\n&\\geq \\frac{1}{t}\\frac{d^2}{d^2 (d+1)} \\| \\xi_{AB} \\|^2_{1} ,\n\\end{align}\nwhere $F$ denotes the swap operator (as defined in Section \\ref{sec:notation}) and in the last step used the H\\\"older inequality (see, e.g., \\cite{bhatia97})\n\\begin{align}\n\\left\\|\\xi_{AB}\\right\\|_1=\\left\\|\\sigma^{1/4}\\sigma^{-1/4}\\xi_{AB}\\sigma^{-1/4}\\sigma^{1/4}\\right\\|_1\\leq\\left\\|\\sigma^{1/4}\\otimes1_B\\right\\|_4\\left\\|\\sigma_A^{-1/4}\\xi_{AB}\\sigma_A^{-1/4}\\right\\|_2\\left\\|\\sigma^{1/4}\\otimes1_B\\right\\|_4\\leq\\sqrt{d}\\left\\|\\tilde{\\xi}_{AB}\\right\\|_2.\n\\end{align}\nFor further bounding the denominator we write\n\\begin{align}\n\\left\\| \\sigma_{A}^{-1/2} (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\sigma_{A}^{-1/2} \\right\\|_\\infty\n&= \\max_{z} \\frac{d}{t} \\left\\| \\mathrm{Tr}_{B}\\left[(1_{A} \\otimes P_z) \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2}\\right]\\right\\|_\\infty \\\\\n&\\leq \\frac{d}{t} \\max_{\\ket{\\phi}_{A}, \\ket{\\psi}_{B}} \\bra{\\phi}_{A} \\otimes \\bra{\\psi}_{B} \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2} \\ket{\\phi}_{A} \\otimes \\ket{\\psi}_{B} \\\\\n&\\leq \\frac{d}{t} \\left\\| \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2} \\right\\|_\\infty,\n\\end{align}\nwhere we used the fact that $P_z$ is a rank 1 projector. Now, observe that for any $\\xi_{AB}$, there exists a $\\sigma_A$ of unit trace such that\n\\begin{align}\n\\frac{\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}}}{\\| \\xi_{AB} \\|_{1}} \\preceq d\\cdot\\sigma_{A} \\otimes1_{B}.\n\\end{align}\nThis just follows from, e.g., \\cite[Lemma B.6]{bertachristandl11}, where it is shown that we can in fact choose\\footnote{By a continuity argument $\\sigma_A$ can be assumed to have full rank.}\n\\begin{align}\n\\sigma_{A} = \\| \\xi_{AB} \\|_{1}^{-1}\\cdot\\mathrm{Tr}_{B}\\left[\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}}\\right].\n\\end{align}\nAs a result, we have\n\\begin{align}\n\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}} \\preceq d\\| \\xi_{AB} \\|_{1}\\cdot\\sigma_{A} \\otimes1_{B}.\n\\end{align}\nAs $\\xi_{AB}$ is Hermitian, we can decompose it into the positive and negative part $\\xi_{AB} = P - Q$ with $P$ and $Q$ positive semidefinite and $PQ = 0$, then $\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}} = P + Q$ and so $-\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}} \\preceq \\xi_{AB} \\preceq \\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}}$. Thus, we get\n\\begin{align}\n- d_{B} \\| \\xi_{AB} \\|_1\\cdot\\sigma_{A} \\otimes1_{B} \\preceq \\xi_{AB} \\preceq d_{B} \\| \\xi_{AB} \\|_1 \\cdot\\sigma_{A} \\otimes1_{B},\n\\end{align}\nand we find $\\| \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2} \\|_{\\infty} \\leq d\\| \\xi_{AB} \\|_1$. This concludes the proof.\n\\end{proof}","start":124988,"end":130190},{"raw_tex":"Henceforth, we abbreviate $d\\equiv d_B$. To further bound the numerator, letting $\\tilde{\\xi}_{AB}:=\\sigma_{A}^{-1/4} \\xi_{AB} \\sigma_{A}^{-1/4}$ we get\n\\begin{align}\n\\| (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\tilde{\\xi}_{AB}) \\|^2_{2}\n&= \\left\\| \\sum_{z} \\frac{d}{t} |z\\rangle \\langle z| \\otimes  \\mathrm{Tr}_{B}\\left[(1_{A} \\otimes P_z) \\tilde{\\xi}_{AB}\\right] \\right\\|_2^2 \\\\\n&= \\sum_{z}  \\frac{d^2}{t^2} \\mathrm{Tr}\\left[\\mathrm{Tr}_{B}\\left[(1_{A} \\otimes P_z) \\tilde{\\xi}_{AB}\\right] \\otimes \\mathrm{Tr}_{\\bar{B}}\\left[(1_{\\bar{A}} \\otimes P_z) \\tilde{\\xi}_{\\bar{A}\\bar{B}}\\right]^{\\dagger} F_{A\\bar{A}} \\right] \\\\\n&= \\sum_{z} \\frac{d^2}{t^2} \\mathrm{Tr}\\left[\\left(\\left((1_{A} \\otimes P_z) \\tilde{\\xi}_{AB}\\right) \\otimes \\left((1_{\\bar{A}} \\otimes P_z) \\tilde{\\xi}_{\\bar{A}\\bar{B}}\\right)^{\\dagger}\\right)\\left(F_{A\\bar{A}} \\otimes1_{B\\bar{B}}\\right)\\right] \\\\\n&= \\frac{d^2}{t^2} \\mathrm{Tr}\\left[\\left(\\tilde{\\xi}_{AB} \\otimes \\tilde{\\xi}_{\\bar{A}\\bar{B}}^{\\dagger}\\right)\\left(\\sum_{z} (1_{A\\bar{A}} \\otimes P_z \\otimes P_{z})\\right)\\left(F_{A\\bar{A}} \\otimes1_{B \\bar{B}}\\right)\\right] \\\\\n&= \\frac{1}{t} \\frac{d^2}{d(d+1)} \\mathrm{Tr}\\left[\\left(\\tilde{\\xi}_{AB} \\otimes \\tilde{\\xi}_{\\bar{A}\\bar{B}}^{\\dagger}\\right) (1_{A\\bar{A}} \\otimes (1_{B\\bar{B}} + F_{B\\bar{B}}))\\left(F_{A\\bar{A}} \\otimes1_{B \\bar{B}}\\right)\\right] \\\\\n&= \\frac{1}{t} \\frac{d^2}{d(d+1)} \\Big(\\underbrace{\\mathrm{Tr}\\left[\\tilde{\\xi}_{A} \\tilde{\\xi}_{A}^{\\dagger}\\right]}_{\\geq 0} + \\underbrace{\\mathrm{Tr}\\left[\\tilde{\\xi}_{AB} \\tilde{\\xi}_{AB}^{\\dagger}\\right]}_{=\\left\\|\\tilde{\\xi}_{AB}\\right\\|_2^2} \\Big) \\\\\n&\\geq \\frac{1}{t}\\frac{d^2}{d^2 (d+1)} \\| \\xi_{AB} \\|^2_{1} ,\n\\end{align}\n","start":126106,"end":127777},{"raw_tex":"where $F$ denotes the swap operator (as defined in Section \\ref{sec:notation}) and in the last step used the H\\\"older inequality (see, e.g., \\cite{bhatia97})\n\\begin{align}\n\\left\\|\\xi_{AB}\\right\\|_1=\\left\\|\\sigma^{1/4}\\sigma^{-1/4}\\xi_{AB}\\sigma^{-1/4}\\sigma^{1/4}\\right\\|_1\\leq\\left\\|\\sigma^{1/4}\\otimes1_B\\right\\|_4\\left\\|\\sigma_A^{-1/4}\\xi_{AB}\\sigma_A^{-1/4}\\right\\|_2\\left\\|\\sigma^{1/4}\\otimes1_B\\right\\|_4\\leq\\sqrt{d}\\left\\|\\tilde{\\xi}_{AB}\\right\\|_2.\n\\end{align}\n","start":127777,"end":128246},{"raw_tex":"For further bounding the denominator we write\n\\begin{align}\n\\left\\| \\sigma_{A}^{-1/2} (\\mathcal{I}_{A} \\otimes \\mathcal{M}_B)(\\xi_{AB}) \\sigma_{A}^{-1/2} \\right\\|_\\infty\n&= \\max_{z} \\frac{d}{t} \\left\\| \\mathrm{Tr}_{B}\\left[(1_{A} \\otimes P_z) \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2}\\right]\\right\\|_\\infty \\\\\n&\\leq \\frac{d}{t} \\max_{\\ket{\\phi}_{A}, \\ket{\\psi}_{B}} \\bra{\\phi}_{A} \\otimes \\bra{\\psi}_{B} \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2} \\ket{\\phi}_{A} \\otimes \\ket{\\psi}_{B} \\\\\n&\\leq \\frac{d}{t} \\left\\| \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2} \\right\\|_\\infty,\n\\end{align}\nwhere we used the fact that $P_z$ is a rank 1 projector. Now, observe that for any $\\xi_{AB}$, there exists a $\\sigma_A$ of unit trace such that\n","start":128246,"end":128984},{"raw_tex":"where we used the fact that $P_z$ is a rank 1 projector. Now, observe that for any $\\xi_{AB}$, there exists a $\\sigma_A$ of unit trace such that\n\\begin{align}\n\\frac{\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}}}{\\| \\xi_{AB} \\|_{1}} \\preceq d\\cdot\\sigma_{A} \\otimes1_{B}.\n\\end{align}\nThis just follows from, e.g., \\cite[Lemma B.6]{bertachristandl11}, where it is shown that we can in fact choose\\footnote{By a continuity argument $\\sigma_A$ can be assumed to have full rank.}\n\\begin{align}\n\\sigma_{A} = \\| \\xi_{AB} \\|_{1}^{-1}\\cdot\\mathrm{Tr}_{B}\\left[\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}}\\right].\n\\end{align}\n","start":128839,"end":129434},{"raw_tex":"As a result, we have\n\\begin{align}\n\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}} \\preceq d\\| \\xi_{AB} \\|_{1}\\cdot\\sigma_{A} \\otimes1_{B}.\n\\end{align}\n","start":129434,"end":129573},{"raw_tex":"As $\\xi_{AB}$ is Hermitian, we can decompose it into the positive and negative part $\\xi_{AB} = P - Q$ with $P$ and $Q$ positive semidefinite and $PQ = 0$, then $\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}} = P + Q$ and so $-\\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}} \\preceq \\xi_{AB} \\preceq \\sqrt{\\xi_{AB} \\xi_{AB}^{\\dagger}}$. Thus, we get\n\\begin{align}\n- d_{B} \\| \\xi_{AB} \\|_1\\cdot\\sigma_{A} \\otimes1_{B} \\preceq \\xi_{AB} \\preceq d_{B} \\| \\xi_{AB} \\|_1 \\cdot\\sigma_{A} \\otimes1_{B},\n\\end{align}\n","start":129573,"end":130055},{"raw_tex":"and we find $\\| \\sigma_{A}^{-1/2} \\xi_{AB} \\sigma_{A}^{-1/2} \\|_{\\infty} \\leq d\\| \\xi_{AB} \\|_1$. This concludes the proof.\n\\end{proof}","start":130055,"end":130190}]},{"id":"S046","semantic_type":"definition","kind":"definition","labels":["lem:ic-meas-side-info"],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"态两设计测量","source_units":[{"raw_tex":"Consider a state two-design on $B$, i.e., a set of rank-one projectors $\\{P_z\\}_{z \\in \\{1, \\dots, t\\}}$ such that $\\frac{1}{t} \\sum_{z=1}^t P_z \\otimes P_z = \\frac{2 P^{\\mathrm{sym}}}{d_B (d_B+1)}$, where $P^{\\mathrm{sym}}$ denotes the projector onto the symmetric subspace of $B \\otimes B$. Let $\\mathcal{M}_B$ be the measurement defined as\n\\begin{align}\n\\mathcal{M}_B(X) = \\sum_{z} \\frac{d_B}{t} \\cdot\\mathrm{Tr}\\big[P_z X\\big] |z\\rangle\\langle z|,\n\\end{align}\n","start":124089,"end":124553}]},{"id":"S056","semantic_type":"theorem","kind":"theorem","labels":[],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"受约束 SDP 层级收敛","source_units":[{"raw_tex":"\\begin{theorem}\nFor the SDPs\n\\begin{align}\n\\mathrm{SDP}_n :=\\max &\\quad \\mathrm{Tr}\\big[G_{AB} W_{AB_1}\\big]\\\\\ns.t. \n&\\quad W_{AB_1^n}\\succeq0, \\mathrm{Tr}(W_{AB_1^n}) = 1, \\;W_{AB_1^n}=\\mathcal{U}_{B_1^n}^\\pi\\left(W_{AB_1^n}\\right)\\;\\forall\\pi\\in\\mathfrak{S}_n\\\\\n& \\quad \\Lambda_{A\\to C_A}\\left(W_{AB_1^n}\\right)=X_{C_A}\\otimes W_{B_1^n},\\;\\Gamma_{B_n\\to C_B}\\left(W_{B_1^n}\\right)=W_{B_1^{n-1}}\\otimes Y_{C_B}\n\\end{align}\nand $Q$ defined as above, we have for $d:=\\max\\{d_A,d_B\\}$ that\n\\begin{align}\n0 \\leq \\mathrm{SDP}_n - Q \\leq \\frac{\\mathrm{poly}(d)}{\\sqrt{n}}\\quad\\text{implying}\\quad Q=\\lim_{n\\to\\infty}\\mathrm{SDP}_n.\n\\end{align}\n\\end{theorem}","start":48380,"end":49032}]},{"id":"S057","semantic_type":"algorithm","kind":"algorithm","labels":[],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"对称扩张 SDP 层级","source_units":[{"raw_tex":"For the SDPs\n\\begin{align}\n\\mathrm{SDP}_n :=\\max &\\quad \\mathrm{Tr}\\big[G_{AB} W_{AB_1}\\big]\\\\\ns.t. \n&\\quad W_{AB_1^n}\\succeq0, \\mathrm{Tr}(W_{AB_1^n}) = 1, \\;W_{AB_1^n}=\\mathcal{U}_{B_1^n}^\\pi\\left(W_{AB_1^n}\\right)\\;\\forall\\pi\\in\\mathfrak{S}_n\\\\\n& \\quad \\Lambda_{A\\to C_A}\\left(W_{AB_1^n}\\right)=X_{C_A}\\otimes W_{B_1^n},\\;\\Gamma_{B_n\\to C_B}\\left(W_{B_1^n}\\right)=W_{B_1^{n-1}}\\otimes Y_{C_B}\n\\end{align}\n","start":48396,"end":48804}]},{"id":"S058","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"层级收敛完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nWe have by construction $0 \\leq \\mathrm{SDP}_n - Q $ and the remaining inequality arises from\n\\begin{align}\n\\mathrm{Tr}\\left[G_{AB} W_{AB_1}\\right]&=\\mathrm{Tr}\\left[G_{AB} (W_{A} \\otimes W_{B})\\right]+\\mathrm{Tr}\\left[G_{AB}\\left(W_{AB_1} - W_{A} \\otimes W_{B}\\right)\\right] \\\\\n& \\leq \\mathrm{Tr}\\left[G_{AB} (W_{A} \\otimes W_{B})\\right]+\\| G_{AB} \\|_\\infty \\cdot \\|W_{AB_1} - W_{A} \\otimes W_{B} \\|_1 \\\\\n&\\leq\\mathrm{Tr}\\left[G_{AB} (W_{A} \\otimes W_{B})\\right]+\\frac{\\mathrm{poly}(d)}{\\sqrt{n}},\n\\end{align}\nwhere we used H\\\"older's inequality and the de Finetti argument as in Theorem \\ref{thm:main-deFinetti}.\n\\end{proof}","start":49034,"end":49674}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"构造给出松弛外界","source_units":[{"raw_tex":"\\begin{proof}\nWe have by construction $0 \\leq \\mathrm{SDP}_n - Q $ and the remaining inequality arises from\n","start":49034,"end":49142}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"分解目标为比较值与误差","source_units":[{"raw_tex":"\\begin{align}\n\\mathrm{Tr}\\left[G_{AB} W_{AB_1}\\right]&=\\mathrm{Tr}\\left[G_{AB} (W_{A} \\otimes W_{B})\\right]+\\mathrm{Tr}\\left[G_{AB}\\left(W_{AB_1} - W_{A} \\otimes W_{B}\\right)\\right] \\\\\n","start":49142,"end":49327}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"Hölder 控制目标误差","source_units":[{"raw_tex":"& \\leq \\mathrm{Tr}\\left[G_{AB} (W_{A} \\otimes W_{B})\\right]+\\| G_{AB} \\|_\\infty \\cdot \\|W_{AB_1} - W_{A} \\otimes W_{B} \\|_1 \\\\\n","start":49327,"end":49454},{"raw_tex":"where we used H\\\"older's inequality and the de Finetti argument as in Theorem \\ref{thm:main-deFinetti}.\n","start":49559,"end":49663}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"援引 de Finetti 宣告误差率","source_units":[{"raw_tex":"&\\leq\\mathrm{Tr}\\left[G_{AB} (W_{A} \\otimes W_{B})\\right]+\\frac{\\mathrm{poly}(d)}{\\sqrt{n}},\n\\end{align}\nwhere we used H\\\"older's inequality and the de Finetti argument as in Theorem \\ref{thm:main-deFinetti}.\n\\end{proof}","start":49454,"end":49674}]},{"id":"S063","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S063"],"declared_external":true,"declaration_text":["原始先决条件明确列出的 nielsen00, Chapter 11；本文在第354行引用并在第356—362行使用。"],"source_only":false,"title":"经典量子互信息恒等式","source_units":[]},{"id":"S064","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S064"],"declared_external":true,"declaration_text":["原始先决条件中的 watrous-book, Theorem 5.38；原文第362行明确引用。"],"source_only":false,"title":"量子 Pinsker 不等式","source_units":[]},{"id":"S065","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S065"],"declared_external":true,"declaration_text":["原始先决条件中的 watrous-book, Theorem 3.39；原文第437行明确引用。"],"source_only":false,"title":"信道的迹范数收缩","source_units":[]},{"id":"S066","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S066"],"declared_external":true,"declaration_text":["原始先决条件中的 OLKIEWICZ1999246 与 beigi13, Corollary 3；第1322、1326行引用。"],"source_only":false,"title":"加权 Schatten 插值事实","source_units":[]},{"id":"S067","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S067"],"declared_external":true,"declaration_text":["原始先决条件中的 bertachristandl11, Lemma B.6；原文第1354行引用。"],"source_only":false,"title":"双体正算子支配界","source_units":[]},{"id":"S068","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S068"],"declared_external":true,"declaration_text":["原始先决条件列出的标准有限维矩阵分析；附录第1339行另引用 bhatia97，主证明第592行仅具名使用 Hölder 不等式。"],"source_only":false,"title":"普通 Schatten Hölder 事实","source_units":[]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S069"],"declared_external":true,"declaration_text":["原始先决条件中的 Jensen 不等式与范数凸性；原文第441、446行明确具名使用。"],"source_only":false,"title":"Jensen 与范数凸性","source_units":[]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S070"],"declared_external":true,"declaration_text":["原始先决条件中的标准有限维迹恒等式和 Schatten 范数；implicit standard fact made explicit by annotator。"],"source_only":false,"title":"分块与交换迹恒等式","source_units":[]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S071"],"declared_external":true,"declaration_text":["原始先决条件中的标准谱分解和连续性；implicit standard fact made explicit by annotator。"],"source_only":false,"title":"谱分解与连续性背景","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S034","S038":"S034","S039":"S034","S040":"S034","S041":"S034","S042":"S034","S043":"S034","S044":"S034","S045":"S045","S046":"S046","S047":"S045","S048":"S045","S049":"S045","S050":"S045","S051":"S045","S052":"S045","S053":"S045","S054":"S045","S055":"S045","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071"}}}}
{"record_id":"2110.09854v4","paper":{"title":"A New Extension of Chubanov's Method to Symmetric Cones","authors":["Kanoh, Shin-ichi","Yoshise, Akiko"],"arxiv_id":"2110.09854","version":"v4","journal":"Mathematical Programming","abstract_url":"https://arxiv.org/abs/2110.09854v4","pdf_url":"https://arxiv.org/pdf/2110.09854v4","doi":"10.1007/s10107-023-01995-9"},"context":{"raw_tex":"Let $\\mathbb{E}$ be a real-valued vector space equipped with an inner product $\\langle \\cdot, \\cdot \\rangle$ and a bilinear operation $\\circ$ : $\\mathbb{E} \\times \\mathbb{E} \\rightarrow \\mathbb{E}$,\r\nand $e$ be the identity element, i.e.,$x \\circ e = e \\circ x = x$ holds for any $ x \\in \\mathbb{E}$.\r\n$(\\mathbb{E}, \\circ)$ is called a Euclidean Jordan algebra if it satisfies\r\n\\begin{align*}\r\nx \\circ y = y  \\circ x,  \\ \\\r\nx \\circ (x^2 \\circ y) = x^2 \\circ (x \\circ y), \\ \\\r\n\\langle x \\circ y , z \\rangle = \\langle y , x \\circ z \\rangle\r\n\\end{align*}\r\nfor all $x,y,z \\in \\mathbb{E}$ and $x^2 := x \\circ x$.\r\nWe denote $y \\in \\mathbb{E}$ as $x^{-1}$ if  $y$ satisfies $x \\circ y = e$.\r\n$c \\in \\mathbb{E}$ is called an {\\em idempotent} if it satisfies $c \\circ c = c$, and an idempotent $c$ is called {\\em primitive} if it can not be written as a sum of two or more nonzero idempotents. \r\nA set of primitive idempotents $c_1, c_2, \\ldots c_k$ is called a {\\em Jordan frame} if $c_1, \\ldots c_k$ satisfy\r\n\\begin{equation}\r\n\\notag\r\nc_i \\circ c_j = 0 \\ (i \\neq j), \\ \\\r\nc_i \\circ c_i = c_i \\ (i= 1 , \\dots , k), \\ \\\r\n\\sum_{i=1}^k c_i = e.\r\n\\end{equation}\r\n\n\n\\begin{proposition}[Spectral theorem (cf. Theorem III.1.2 of \\cite{Faraut1994})]\r\n\\label{prop:spectral}\r\nLet $(\\mathbb{E}, \\circ)$ be a Euclidean Jordan algebra having rank $r$. \r\nFor any $x \\in \\mathbb{E} $, there exist real numbers $\\lambda_1 , \\dots , \\lambda_r$ and a Jordan frame $c_1 , \\dots , c_r $ for which the following holds:\r\n\\begin{equation}\r\nx = \\sum_{i=1}^r \\lambda_i c_i \\notag.\r\n\\end{equation}\r\nThe numbers $\\lambda_1 , \\dots , \\lambda_r$ are uniquely determined {\\em eigenvalues} of $x$ (with their multiplicities).\r\nFurthermore, ${\\rm trace}(x) := \\sum_{i=1}^r \\lambda_i$, $\\det(x) := \\prod_{i=1}^r \\lambda_i$.\r\n\\end{proposition}\r\n\n\nA proper cone is symmetric if it is self-dual and homogeneous. \r\nIt is known that the set of squares $\\mathcal{K} = \\{ x^2 : x \\in \\mathbb{E} \\}$ is the symmetric cone of  $\\mathbb{E}$  (cf. Theorems III.2.1 and III.3.1 of \\cite{Faraut1994}).\r\nThe following properties can be derived from the results in \\cite{Faraut1994}, as in  Corollary 2.3 of \\cite{Yoshise2007}:\r\n\\begin{proposition}\r\n\\label{prop:lambda-Jordan}\r\nLet $x \\in \\mathbb{E}$ and let $\\sum_{j=1}^r \\lambda_j c_j$ be a decomposition of $x$ given by Propositoin \\ref{prop:spectral}. Then\r\n\\begin{description}\r\n\\item[(i)]\r\n$x \\in \\mathcal{K}$ if and only if  $\\lambda_j \\geq 0 \\ (j=1,2,\\ldots,r)$, \r\n\\item[(ii)]\r\n$x \\in {\\rm int} \\hspace{0.75mm} \\mathcal{K}$ if and only if $\\lambda_j > 0 \\ (j=1,2,\\ldots,r)$.\r\n\\end{description}\r\n\\end{proposition}\r\n\r\nFrom Propositions \\ref{prop:spectral} and \\ref{prop:lambda-Jordan} for any $x \\in \\mathbb{E}$, its projection $P_\\mathcal{K} (x)$ onto $\\mathcal{K}$ can be written as an operation to round all negative eigenvalues of $x$ to $0$, i.e., \r\n$P_\\mathcal{K} (x) = \\sum_{i=1}^r [\\lambda_i]^+ c_i$, where $[\\cdot]^+$ denotes the projection onto the nonnegative orthant. \r\n\n\nA Euclidean Jordan algebra $(\\mathbb{E} , \\circ)$ is called {\\em simple} if it cannot be written as any Cartesian product of non-zero Euclidean Jordan algebras.\r\nIf the Euclidean Jordan algebra $(\\mathbb{E} , \\circ)$ associated with a symmetric cone $\\mathcal{K}$ is simple, then we say that $\\mathcal{K}$ is {\\em simple}.\r\nIn this paper, we will consider that $\\mathcal{K}$ is given by a Cartesian product of $p$ simple symmetric cones $\\mathcal{K}_\\ell$,\r\n$\\mathcal{K} := \\mathcal{K}_1 \\times \\dots \\times \\mathcal{K}_p$, whose rank and identity element are $r_\\ell$ and $e_\\ell$ $(\\ell=1, \\ldots, p)$.\r\nThe rank $r$ and the identity element of  $\\mathcal{K}$ are given by\r\n\\begin{equation}\r\n\\label{eq:r and e}\r\nr = \\sum_{\\ell=1}^p r_\\ell, \\ \\ e = (e_1 , \\dots , e_p).\r\n\\end{equation}\r\nIn what follows, $x_\\ell$ stands for the $\\ell$-th block element of  $x \\in \\mathcal{K}$, i.e., $x = (x_1, \\dots, x_p) \\in \\mathcal{K}_1 \\times \\dots \\times \\mathcal{K}_p$.\r\nFor each $\\ell=1, \\cdots, p$, we define $\\lambda_{\\min}(x_\\ell) := \\min\\{ \\lambda_1, \\cdots, \\lambda_{r_\\ell} \\}$ where $\\lambda_1, \\cdots,  \\lambda_{r_\\ell}$ are eigenvalues of $x_\\ell$.\r\nThe minimum eigenvalue $\\lambda_{\\min}(x)$ of $x \\in \\mathcal{K}$ is given by $\\lambda_{\\min}(x) = \\mbox{min} \\{ \\lambda_{\\min}(x_1), \\cdots, \\lambda_{\\min}(x_p) \\}$.\r\n\n\nThis subsection summarizes the \\modifyFirst{notations} used in this paper.\r\nFor any $x,y \\in \\mathbb{E}$, we define the inner product  $\\langle \\cdot,\\cdot \\rangle$ and the norm $\\|\\cdot\\|_{J}$ as $\\langle x , y \\rangle := {\\rm trace}(x \\circ y)$ and $\\| x \\|_J := \\sqrt{ \\langle x , x \\rangle }$, respectively.\r\nFor any $x \\in \\mathbb{E}$ having decomposition  $x = \\sum_{i=1}^r \\lambda_i c_i$ as in  Proposition \\ref{prop:spectral}, we also define $\\| x \\|_1 := |\\lambda_1| + \\dots + |\\lambda_r|$, $\\|x\\|_\\infty := \\max \\{|\\lambda_1| , \\dots , |\\lambda_r| \\}$.\r\nFor $x \\in \\mathcal{K}$, we obtain the following equivalent representations: $\\|x\\|_1 = \\langle e , x \\rangle$, $\\|x\\|_\\infty = \\lambda_{\\max} (x)$.\r\nThe following is a list of other definitions and frequently used symbols in the paper.\r\n\\begin{itemize}\r\n\\item $d$: the dimension of the Euclidean space $\\mathbb{E}$ corresponding to $\\mathcal{K}$,\r\n\\item $F_{{\\rm P}_{S_{\\infty}}(\\mathcal{A})}$: the feasible region of ${\\rm P}_{S_{\\infty}}(\\mathcal{A})$,\r\n\\item $P_{\\mathcal{A}} (\\cdot) $: the projection map onto $\\mbox{ker} \\hspace{0.3mm} \\mathcal{A}$,\r\n%\\item $Q_{\\mathcal{A}} (\\cdot) $: the projection map onto $\\mbox{range} \\hspace{0.3mm} \\mathcal{A}^*$, \r\n\\item $\\mathcal{P}_\\mathcal{K} (\\cdot) $: the projection map onto $\\mathcal{K}$,\r\n\\item $\\lambda(x) \\in \\mathbb{R}^r$: an $r$-dimensional vector composed of the eigenvalues of $ x \\in \\mathcal{K}$,\r\n\\item $\\lambda(x_\\ell) \\in \\mathbb{R}^{r_\\ell}$: an $r_\\ell$-dimensional vector composed of the eigenvalues of $x_\\ell \\in \\mathcal{K}_\\ell$ ($\\ell =1,\\ldots,p$),\r\n\\item $c(x_\\ell)_i \\in \\mathcal{K}_\\ell$: the $i$-th primitive idempotent of $x_\\ell \\in \\mathbb{E}_\\ell$. When $\\mathcal{K}$ is simple, it is abbreviated as $c_i$.\r\n\\item $\\left[ \\cdot \\right]^+$: the projection map onto the nonnegative orthant, and\r\n\\item $\\mathcal{A}^*(\\cdot)$: the adjoint operator of the linear operator $\\mathcal{A}(\\cdot)$, i.e., $\\langle \\mathcal{A}(x) , y \\rangle = \\left \\langle x , \\mathcal{A}^* (y) \\right\\rangle$ for all $x \\in \\mathcal{K}$ and $y \\in \\mathbb{R}^m$.\r\n\\end{itemize}\r\n\n\nWe focus on the feasibility of the following problem ${\\rm P}_{S_{\\infty}}(\\mathcal{A})$, which is equivalent to  ${\\rm P}(\\mathcal{A})$: \r\n\\begin{equation}\r\n\\begin{array}{lllllllll}\r\n{\\rm P}_{S_{\\infty}}(\\mathcal{A})&\t&\\mbox{find}   &x \\in \\mbox{int} \\hspace{0.75mm} \\mathcal{K}\t&\t&\\mbox{s.t.}\t&\\mathcal{A} (x) = \\bm{0},\t&\\|x\\|_\\infty \\leq 1.\r\n\\end{array}\r\n\\notag\r\n\\end{equation}\r\nThe alternative problem ${\\rm D}(\\mathcal{A})$ of ${\\rm P}(\\mathcal{A})$ is \r\n\\begin{equation}\r\n\\notag\r\n\\begin{array}{llllllllll}\r\n{\\rm D}(\\mathcal{A})\t&\t&\\mbox{find}   &y \\in \\mathcal{K}\t&\t&\\mbox{s.t.}\t&y \\in \\mbox{range} \\hspace{0.3mm} \\mathcal{A}^*,\t&y \\neq \\bm{0},\r\n\\end{array}\r\n\\end{equation}\r\nwhere  $\\mbox{range} \\hspace{0.3mm} \\mathcal{A}^*$ is the orthogonal complement of $\\mbox{ker} \\hspace{0.3mm} \\mathcal{A}$.\r\nAs we mentioned in section \\ref{subsec:symmetric cone}, \r\nwe assume that $\\mathcal{K}$ is given by a Cartesian product of $p$ simple symmetric cones $\\mathcal{K}_\\ell (\\ell=1, \\ldots, p)$, i.e., $\\mathcal{K} = \\mathcal{K}_1 \\times \\dots \\times \\mathcal{K}_p $.\r\n","units":[{"raw_tex":"Let $\\mathbb{E}$ be a real-valued vector space equipped with an inner product $\\langle \\cdot, \\cdot \\rangle$ and a bilinear operation $\\circ$ : $\\mathbb{E} \\times \\mathbb{E} \\rightarrow \\mathbb{E}$,\r\nand $e$ be the identity element, i.e.,$x \\circ e = e \\circ x = x$ holds for any $ x \\in \\mathbb{E}$.\r\n$(\\mathbb{E}, \\circ)$ is called a Euclidean Jordan algebra if it satisfies\r\n\\begin{align*}\r\nx \\circ y = y  \\circ x,  \\ \\\r\nx \\circ (x^2 \\circ y) = x^2 \\circ (x \\circ y), \\ \\\r\n\\langle x \\circ y , z \\rangle = \\langle y , x \\circ z \\rangle\r\n\\end{align*}\r\nfor all $x,y,z \\in \\mathbb{E}$ and $x^2 := x \\circ x$.\r\nWe denote $y \\in \\mathbb{E}$ as $x^{-1}$ if  $y$ satisfies $x \\circ y = e$.\r\n$c \\in \\mathbb{E}$ is called an {\\em idempotent} if it satisfies $c \\circ c = c$, and an idempotent $c$ is called {\\em primitive} if it can not be written as a sum of two or more nonzero idempotents. \r\nA set of primitive idempotents $c_1, c_2, \\ldots c_k$ is called a {\\em Jordan frame} if $c_1, \\ldots c_k$ satisfy\r\n\\begin{equation}\r\n\\notag\r\nc_i \\circ c_j = 0 \\ (i \\neq j), \\ \\\r\nc_i \\circ c_i = c_i \\ (i= 1 , \\dots , k), \\ \\\r\n\\sum_{i=1}^k c_i = e.\r\n\\end{equation}\r\n","start":19433,"end":20585},{"raw_tex":"\\begin{proposition}[Spectral theorem (cf. Theorem III.1.2 of \\cite{Faraut1994})]\r\n\\label{prop:spectral}\r\nLet $(\\mathbb{E}, \\circ)$ be a Euclidean Jordan algebra having rank $r$. \r\nFor any $x \\in \\mathbb{E} $, there exist real numbers $\\lambda_1 , \\dots , \\lambda_r$ and a Jordan frame $c_1 , \\dots , c_r $ for which the following holds:\r\n\\begin{equation}\r\nx = \\sum_{i=1}^r \\lambda_i c_i \\notag.\r\n\\end{equation}\r\nThe numbers $\\lambda_1 , \\dots , \\lambda_r$ are uniquely determined {\\em eigenvalues} of $x$ (with their multiplicities).\r\nFurthermore, ${\\rm trace}(x) := \\sum_{i=1}^r \\lambda_i$, $\\det(x) := \\prod_{i=1}^r \\lambda_i$.\r\n\\end{proposition}\r\n","start":20879,"end":21529},{"raw_tex":"A proper cone is symmetric if it is self-dual and homogeneous. \r\nIt is known that the set of squares $\\mathcal{K} = \\{ x^2 : x \\in \\mathbb{E} \\}$ is the symmetric cone of  $\\mathbb{E}$  (cf. Theorems III.2.1 and III.3.1 of \\cite{Faraut1994}).\r\nThe following properties can be derived from the results in \\cite{Faraut1994}, as in  Corollary 2.3 of \\cite{Yoshise2007}:\r\n\\begin{proposition}\r\n\\label{prop:lambda-Jordan}\r\nLet $x \\in \\mathbb{E}$ and let $\\sum_{j=1}^r \\lambda_j c_j$ be a decomposition of $x$ given by Propositoin \\ref{prop:spectral}. Then\r\n\\begin{description}\r\n\\item[(i)]\r\n$x \\in \\mathcal{K}$ if and only if  $\\lambda_j \\geq 0 \\ (j=1,2,\\ldots,r)$, \r\n\\item[(ii)]\r\n$x \\in {\\rm int} \\hspace{0.75mm} \\mathcal{K}$ if and only if $\\lambda_j > 0 \\ (j=1,2,\\ldots,r)$.\r\n\\end{description}\r\n\\end{proposition}\r\n\r\nFrom Propositions \\ref{prop:spectral} and \\ref{prop:lambda-Jordan} for any $x \\in \\mathbb{E}$, its projection $P_\\mathcal{K} (x)$ onto $\\mathcal{K}$ can be written as an operation to round all negative eigenvalues of $x$ to $0$, i.e., \r\n$P_\\mathcal{K} (x) = \\sum_{i=1}^r [\\lambda_i]^+ c_i$, where $[\\cdot]^+$ denotes the projection onto the nonnegative orthant. \r\n","start":22073,"end":23249},{"raw_tex":"A Euclidean Jordan algebra $(\\mathbb{E} , \\circ)$ is called {\\em simple} if it cannot be written as any Cartesian product of non-zero Euclidean Jordan algebras.\r\nIf the Euclidean Jordan algebra $(\\mathbb{E} , \\circ)$ associated with a symmetric cone $\\mathcal{K}$ is simple, then we say that $\\mathcal{K}$ is {\\em simple}.\r\nIn this paper, we will consider that $\\mathcal{K}$ is given by a Cartesian product of $p$ simple symmetric cones $\\mathcal{K}_\\ell$,\r\n$\\mathcal{K} := \\mathcal{K}_1 \\times \\dots \\times \\mathcal{K}_p$, whose rank and identity element are $r_\\ell$ and $e_\\ell$ $(\\ell=1, \\ldots, p)$.\r\nThe rank $r$ and the identity element of  $\\mathcal{K}$ are given by\r\n\\begin{equation}\r\n\\label{eq:r and e}\r\nr = \\sum_{\\ell=1}^p r_\\ell, \\ \\ e = (e_1 , \\dots , e_p).\r\n\\end{equation}\r\nIn what follows, $x_\\ell$ stands for the $\\ell$-th block element of  $x \\in \\mathcal{K}$, i.e., $x = (x_1, \\dots, x_p) \\in \\mathcal{K}_1 \\times \\dots \\times \\mathcal{K}_p$.\r\nFor each $\\ell=1, \\cdots, p$, we define $\\lambda_{\\min}(x_\\ell) := \\min\\{ \\lambda_1, \\cdots, \\lambda_{r_\\ell} \\}$ where $\\lambda_1, \\cdots,  \\lambda_{r_\\ell}$ are eigenvalues of $x_\\ell$.\r\nThe minimum eigenvalue $\\lambda_{\\min}(x)$ of $x \\in \\mathcal{K}$ is given by $\\lambda_{\\min}(x) = \\mbox{min} \\{ \\lambda_{\\min}(x_1), \\cdots, \\lambda_{\\min}(x_p) \\}$.\r\n","start":24057,"end":25376},{"raw_tex":"This subsection summarizes the \\modifyFirst{notations} used in this paper.\r\nFor any $x,y \\in \\mathbb{E}$, we define the inner product  $\\langle \\cdot,\\cdot \\rangle$ and the norm $\\|\\cdot\\|_{J}$ as $\\langle x , y \\rangle := {\\rm trace}(x \\circ y)$ and $\\| x \\|_J := \\sqrt{ \\langle x , x \\rangle }$, respectively.\r\nFor any $x \\in \\mathbb{E}$ having decomposition  $x = \\sum_{i=1}^r \\lambda_i c_i$ as in  Proposition \\ref{prop:spectral}, we also define $\\| x \\|_1 := |\\lambda_1| + \\dots + |\\lambda_r|$, $\\|x\\|_\\infty := \\max \\{|\\lambda_1| , \\dots , |\\lambda_r| \\}$.\r\nFor $x \\in \\mathcal{K}$, we obtain the following equivalent representations: $\\|x\\|_1 = \\langle e , x \\rangle$, $\\|x\\|_\\infty = \\lambda_{\\max} (x)$.\r\nThe following is a list of other definitions and frequently used symbols in the paper.\r\n\\begin{itemize}\r\n\\item $d$: the dimension of the Euclidean space $\\mathbb{E}$ corresponding to $\\mathcal{K}$,\r\n\\item $F_{{\\rm P}_{S_{\\infty}}(\\mathcal{A})}$: the feasible region of ${\\rm P}_{S_{\\infty}}(\\mathcal{A})$,\r\n\\item $P_{\\mathcal{A}} (\\cdot) $: the projection map onto $\\mbox{ker} \\hspace{0.3mm} \\mathcal{A}$,\r\n%\\item $Q_{\\mathcal{A}} (\\cdot) $: the projection map onto $\\mbox{range} \\hspace{0.3mm} \\mathcal{A}^*$, \r\n\\item $\\mathcal{P}_\\mathcal{K} (\\cdot) $: the projection map onto $\\mathcal{K}$,\r\n\\item $\\lambda(x) \\in \\mathbb{R}^r$: an $r$-dimensional vector composed of the eigenvalues of $ x \\in \\mathcal{K}$,\r\n\\item $\\lambda(x_\\ell) \\in \\mathbb{R}^{r_\\ell}$: an $r_\\ell$-dimensional vector composed of the eigenvalues of $x_\\ell \\in \\mathcal{K}_\\ell$ ($\\ell =1,\\ldots,p$),\r\n\\item $c(x_\\ell)_i \\in \\mathcal{K}_\\ell$: the $i$-th primitive idempotent of $x_\\ell \\in \\mathbb{E}_\\ell$. When $\\mathcal{K}$ is simple, it is abbreviated as $c_i$.\r\n\\item $\\left[ \\cdot \\right]^+$: the projection map onto the nonnegative orthant, and\r\n\\item $\\mathcal{A}^*(\\cdot)$: the adjoint operator of the linear operator $\\mathcal{A}(\\cdot)$, i.e., $\\langle \\mathcal{A}(x) , y \\rangle = \\left \\langle x , \\mathcal{A}^* (y) \\right\\rangle$ for all $x \\in \\mathcal{K}$ and $y \\in \\mathbb{R}^m$.\r\n\\end{itemize}\r\n","start":31084,"end":33187},{"raw_tex":"We focus on the feasibility of the following problem ${\\rm P}_{S_{\\infty}}(\\mathcal{A})$, which is equivalent to  ${\\rm P}(\\mathcal{A})$: \r\n\\begin{equation}\r\n\\begin{array}{lllllllll}\r\n{\\rm P}_{S_{\\infty}}(\\mathcal{A})&\t&\\mbox{find}   &x \\in \\mbox{int} \\hspace{0.75mm} \\mathcal{K}\t&\t&\\mbox{s.t.}\t&\\mathcal{A} (x) = \\bm{0},\t&\\|x\\|_\\infty \\leq 1.\r\n\\end{array}\r\n\\notag\r\n\\end{equation}\r\nThe alternative problem ${\\rm D}(\\mathcal{A})$ of ${\\rm P}(\\mathcal{A})$ is \r\n\\begin{equation}\r\n\\notag\r\n\\begin{array}{llllllllll}\r\n{\\rm D}(\\mathcal{A})\t&\t&\\mbox{find}   &y \\in \\mathcal{K}\t&\t&\\mbox{s.t.}\t&y \\in \\mbox{range} \\hspace{0.3mm} \\mathcal{A}^*,\t&y \\neq \\bm{0},\r\n\\end{array}\r\n\\end{equation}\r\nwhere  $\\mbox{range} \\hspace{0.3mm} \\mathcal{A}^*$ is the orthogonal complement of $\\mbox{ker} \\hspace{0.3mm} \\mathcal{A}$.\r\nAs we mentioned in section \\ref{subsec:symmetric cone}, \r\nwe assume that $\\mathcal{K}$ is given by a Cartesian product of $p$ simple symmetric cones $\\mathcal{K}_\\ell (\\ell=1, \\ldots, p)$, i.e., $\\mathcal{K} = \\mathcal{K}_1 \\times \\dots \\times \\mathcal{K}_p $.\r\n","start":33772,"end":34840}]},"statement":{"raw_tex":"\\begin{proposition}\r\n\\label{prop:upper-bound-direct}\r\nSuppose that, for any $v \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*$, the $\\ell$-th block element $v_\\ell$ of $v \\in \\mathbb{E}$ is decomposed into $v_\\ell = \\sum_{i=1}^{r_\\ell} {\\lambda(v_\\ell)}_i {c(v_\\ell)}_i$ as in Proposition \\ref{prop:spectral}.\r\nFor each $\\ell \\in \\{1,\\ldots,p \\}$ and $i \\in \\{1,\\ldots,r_p\\}$, define\r\n\\begin{equation}\r\n\\label{eq:q_ell,i(alpha)}\r\nq_{\\ell,i}(\\alpha) := \\left[ 1 - \\alpha {\\lambda(v_\\ell)}_i \\right]^+ + \\sum_{k \\neq i}^{r_\\ell} \\left[ -\\alpha {\\lambda(v_\\ell)}_k \\right]^+ + \\sum_{j \\neq \\ell}^p \\sum_{k=1}^{r_j} \\left[  -\\alpha {\\lambda(v_j)}_k \\right]^+ .\r\n\\end{equation}\r\nThen, the following relations hold for any feasible solution $x$ of ${\\rm P}_{S_{\\infty}}(\\mathcal{A})$, $\\ell \\in \\{1,\\ldots,p \\}$ and $i \\in \\{1,\\ldots,r_p\\}$.\r\n\\begin{equation}\r\n\\label{eq:upper-bound-direct1}\r\n\\langle {c(v_\\ell)}_i , x_\\ell \\rangle \\leq  \\min_{\\alpha \\in \\mathbb{R} } q_{\\ell,i}(\\alpha) =\r\n\\begin{cases}\r\n\\min \\left\\{  1 ,  \\left \\langle  e , \\mathcal{P}_{\\mathcal{K}} \\left( -  \\frac{1}{{\\lambda(v_\\ell)}_i} v  \\right) \\right \\rangle \\right\\} & \\mbox{if ${\\lambda(v_\\ell)}_i \\neq 0$}, \\\\\r\n1 & \\mbox{if ${\\lambda(v_\\ell)}_i = 0$} \r\n\\end{cases}\r\n.\r\n\\end{equation}\r\n\\end{proposition}","units":[{"raw_tex":"\\begin{proposition}\r\n\\label{prop:upper-bound-direct}\r\nSuppose that, for any $v \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*$, the $\\ell$-th block element $v_\\ell$ of $v \\in \\mathbb{E}$ is decomposed into $v_\\ell = \\sum_{i=1}^{r_\\ell} {\\lambda(v_\\ell)}_i {c(v_\\ell)}_i$ as in Proposition \\ref{prop:spectral}.\r\nFor each $\\ell \\in \\{1,\\ldots,p \\}$ and $i \\in \\{1,\\ldots,r_p\\}$, define\r\n\\begin{equation}\r\n\\label{eq:q_ell,i(alpha)}\r\nq_{\\ell,i}(\\alpha) := \\left[ 1 - \\alpha {\\lambda(v_\\ell)}_i \\right]^+ + \\sum_{k \\neq i}^{r_\\ell} \\left[ -\\alpha {\\lambda(v_\\ell)}_k \\right]^+ + \\sum_{j \\neq \\ell}^p \\sum_{k=1}^{r_j} \\left[  -\\alpha {\\lambda(v_j)}_k \\right]^+ .\r\n\\end{equation}\r\nThen, the following relations hold for any feasible solution $x$ of ${\\rm P}_{S_{\\infty}}(\\mathcal{A})$, $\\ell \\in \\{1,\\ldots,p \\}$ and $i \\in \\{1,\\ldots,r_p\\}$.\r\n\\begin{equation}\r\n\\label{eq:upper-bound-direct1}\r\n\\langle {c(v_\\ell)}_i , x_\\ell \\rangle \\leq  \\min_{\\alpha \\in \\mathbb{R} } q_{\\ell,i}(\\alpha) =\r\n\\begin{cases}\r\n\\min \\left\\{  1 ,  \\left \\langle  e , \\mathcal{P}_{\\mathcal{K}} \\left( -  \\frac{1}{{\\lambda(v_\\ell)}_i} v  \\right) \\right \\rangle \\right\\} & \\mbox{if ${\\lambda(v_\\ell)}_i \\neq 0$}, \\\\\r\n1 & \\mbox{if ${\\lambda(v_\\ell)}_i = 0$} \r\n\\end{cases}\r\n.\r\n\\end{equation}\r\n\\end{proposition}","start":55306,"end":56584}]},"proof":{"raw_tex":"\\begin{lemma}\r\n\\label{lemma:0}\r\nLet $(\\mathbb{E}, \\circ)$ be a Euclidean Jordan algebra with the \\modifySecond{associated} symmetric cone $\\mathcal{K}$.\r\nFor any $y \\in \\mathbb{E}$, the following equation holds:\r\n\\begin{equation}\r\n\\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\langle y ,x \\rangle  =  \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( y \\right)  ,e \\right \\rangle . \\notag\r\n\\end{equation}\r\n\\end{lemma}\n\n\\begin{proof}\r\nUsing the decomposition $y = \\sum_{i=1}^r \\lambda_i c_i$ obtained by Proposition \\ref{prop:spectral}, we see that \r\n\\begin{equation}\r\n\\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\langle y ,x \\rangle  = \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\left \\langle \\sum_{i=1}^r \\lambda_i c_i , x \\right \\rangle = \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}}  \\sum_{i=1}^r \\lambda_i  \\left \\langle c_i , x \\right \\rangle . \\label{lemma:0-eq1}\r\n\\end{equation}\r\nNoting that $x \\in \\mathcal{K}, e-x \\in \\mathcal{K}$ from $\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}$, since $c_i \\in \\mathcal{K}$ is primitive idempotent, we find that \r\n$\\langle c_i,x \\rangle \\geq 0$ and $\\langle c_i, e-x \\rangle \\geq 0$, which implies that $0 \\leq \\langle c_i,x \\rangle \\leq 1$.\r\nThus, letting $I_1$ be the set of indices for which $\\lambda_i \\leq 0$ and $I_2$ be the set of indices for which $\\lambda_i > 0$, if there exists an $x$ satisfying \r\n\\begin{equation}\r\n\\langle c_i,x \\rangle =\r\n\\begin{cases}\r\n0 & i \\in I_1 \\\\\r\n1 & i \\in I_2\r\n\\end{cases}, \r\n\\label{lemma:0-eq2}\r\n\\end{equation}\r\nthen such an $x$ is an optimal solution of (\\ref{lemma:0-eq1}). In fact, if we define $x^* = \\sum_{i \\in I_2} c_i$, then by the dedfinition of the Jordan frame, $x^*$ satisfies (\\ref{lemma:0-eq2}) and $\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}$ and becomes an optimal solution of (\\ref{lemma:0-eq1}). In this case, the optimal value of (\\ref{lemma:0-eq1}) turns out to be\r\n\\begin{equation}\r\n\\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}}  \\sum_{i=1}^r \\lambda_i  \\left \\langle c_i , x \\right \\rangle = \\sum_{i=1}^r \\lambda_i  \\left \\langle c_i , x^* \\right \\rangle \r\n= \\sum_{i \\in I_2} \\lambda_i \r\n= \\sum_{i=1}^r [\\lambda_i]^+\r\n= \\langle \\mathcal{P}_\\mathcal{K} (y) , e \\rangle. \r\n\\notag\r\n\\end{equation}\r\n\\end{proof}\n\n\\begin{proposition}\r\n\\label{prop:p-d-relation} \r\nLet $(\\mathbb{E}, \\circ)$ be a Euclidean Jordan Algebra with the corresponding symmetric cone  $\\mathcal{K}$.\r\nFor a given $c \\in \\mathbb{E}$, consider the problem\r\n\\begin{equation}\r\n\\notag\r\n\\begin{array}{llllll}\r\n\\mbox{\\rm max}\t&\\langle c , x \\rangle \t&\t&\\mbox{\\rm s.t}\t&\\mathcal{A}(x) = \\bm{0},\t&\\bm{0} \\leq \\lambda (x) \\leq \\bm{1}  \\notag.\r\n\\end{array}\r\n\\end{equation}\r\nThe dual problem of the above is \r\n\\begin{equation}\r\n\\notag\r\n\\begin{array}{lllll}\r\n\\mbox{\\rm min}\t&\\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c - u  \\right) , e \\right \\rangle\t&\t&\\mbox{\\rm s.t}\t&u \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*. \\notag\r\n\\end{array}\r\n\\end{equation}\r\n\\end{proposition}\n\n\\begin{proof}\r\nDefine the Lagrangian function $L(x,w)$ as $L(x,w) :=   \\langle c , x \\rangle -  w^\\top \\mathcal{A}(x)$ where $w \\in \\mathbb{R}^m$ is the Lagrange multiplier. \r\n\\modifyThird{\r\nSupoose that $x^*$ is an optimal sotution of the primal problem. \r\nThen, for any $w \\in \\mathbb{R}^m$,  we have $\\langle c,x^* \\rangle = L(w,x^*) \\leq \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} L(w,x)$, and hence,}\r\n\\begin{align*}\r\n\\modifyThird{\\langle c, x^* \\rangle} \\leq \\min_w \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} L(x,w)\r\n&=  \\min_w \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\{  \\langle c , x \\rangle - \\langle \\mathcal{A}^*(w) ,x \\rangle \\}  \\\\\r\n&=  \\min_w \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\{  \\langle c -  \\mathcal{A}^*(w) ,x \\rangle \\}  \\\\\r\n&=  \\min_w   \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c -  \\mathcal{A}^*(w) \\right)  ,e \\right \\rangle \\ \\ \\ \\mbox{(by  lemma \\ref{lemma:0})}  \\\\\r\n&= \\min_{u \\in \\mbox{\\footnotesize range} \\mathcal{A}^*}  \\langle \\mathcal{P}_\\mathcal{K} \\left( c - u \\right) , e \\rangle.\r\n\\end{align*}\r\n%and the dual problem is \r\n%\\begin{equation}\r\n%\\notag\r\n%\\begin{array}{lllll}\r\n%\\mbox{\\rm min}\t&\\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c - u  \\right) , e \\right \\rangle\t&\t&\\mbox{\\rm s.t}\t&u \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*. \\notag\r\n%\\end{array}\r\n%\\end{equation}\r\n\\end{proof}\n\n\\begin{proposition}\r\n\\label{prop:upper-bound}\r\n\r\nSuppose that $v \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*$ is given by $v = \\sum_{i=1}^r \\lambda_i c_i$ as in Proposition \\ref{prop:spectral}. \r\nFor each $i \\in \\{ 1, \\dots , r \\}$ and $\\alpha \\in \\mathbb{R}$, define $q_i(\\alpha) := \\left[    1-\\alpha \\lambda_i  \\right]^+ + \\sum_{j \\neq i }^r \\left[ -  \\alpha \\lambda_j  \\right]^+$.\r\nThen, the following relations hold for any $x \\in F_{{\\rm P}_{S_{\\infty}}(\\mathcal{A})}$ and $i \\in \\{ 1, \\dots , r \\}$\\modifySecond{:}\r\n\\begin{equation}\r\n\\label{eq:bound_q}\r\n\\langle c_i , x \\rangle \\leq  \r\n\\min_{\\alpha \\in \\mathbb{R}} q_i(\\alpha) = \r\n\\begin{cases}\r\n\\min \\left\\{  1 ,  \\left \\langle  e , \\mathcal{P}_{\\mathcal{K}} \\left( -  \\frac{1}{\\lambda_i} v  \\right) \\right \\rangle \\right\\} & \\mbox{if $\\lambda_i \\neq 0$}, \\\\\r\n1 & \\mbox{if $\\lambda_i = 0$}\\modifySecond{.}\r\n\\end{cases}\r\n\\end{equation}\r\n\\end{proposition}\n\n\\begin{proof}\r\n\r\nFor each $i \\in \\{ 1, 2, \\dots , r \\}$, we have \r\n\\begin{equation}\r\n\\mathcal{P}_\\mathcal{K} \\left( c_i -  \\alpha v \\right) = \\mathcal{P}_\\mathcal{K} \\left( c_i - \\alpha \\sum_{j=1}^r \\lambda_j c_j \\right) = \\mathcal{P}_\\mathcal{K} \\left( (1-\\alpha \\lambda_i ) c_i -   \\sum_{j \\neq i }^r \\alpha \\lambda_j c_j  \\right), \\notag\r\n\\end{equation}\r\nand hence, \r\n\\begin{equation}\r\n\\label{eq:q_i(alpha)}\r\n\\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c_i -  \\alpha v \\right) , e \\right \\rangle\r\n=\r\n\\left \\langle \\mathcal{P}_\\mathcal{K} \\left(    (1-\\alpha \\lambda_i ) c_i -   \\sum_{j \\neq i }^r \\alpha \\lambda_j c_j   \\right) , \\sum_{i=1}^r c_i \\right \\rangle \r\n= \\left[    1-\\alpha \\lambda_i  \\right]^+ + \\sum_{j \\neq i }^r \\left[ -  \\alpha \\lambda_j  \\right]^+  = q_i(\\alpha) .\r\n\\end{equation}\r\n\r\nNote that, since $q_i(\\alpha)$ is a piece-wise linear convex function, if $\\lambda_i = 0$, it attains the minimum at $\\alpha = 0$ with $q_i(0) = 1$, and if $\\lambda_i \\neq 0$, it attains the minimum at $\\alpha = 0$ with $q_i(0) = 1$ or at $\\alpha =  \\frac{1}{\\lambda_i}$ with\r\n\\begin{equation}\r\n\\notag\r\nq \\left( \\frac{1}{\\lambda_i} \\right) = \\sum_{j \\neq i}^r \\left[ - \\frac{\\lambda_j}{\\lambda_i}  \\right]^+ \r\n= \\sum_{j=1}^r \\left[ - \\frac{\\lambda_j}{\\lambda_i}  \\right]^+ \r\n= \\left \\langle e , \\mathcal{P}_{\\mathcal{K}} \\left( -  \\frac{1}{\\lambda_i} v  \\right) \\right \\rangle.\r\n\\end{equation}\r\nThus, we obtain equivalence in (\\ref{eq:bound_q}).\r\nSince $\\alpha v \\in \\mbox{range} \\hspace{0.3mm} \\mathcal{A}^*$ for all $\\alpha \\in \\mathbb{R}$, \r\nfor each $i \\in \\{ 1, \\dots , r \\}$, Proposition \\ref{prop:p-d-relation} and  (\\ref{eq:q_i(alpha)})  ensure that \r\n$\\langle c_i , x \\rangle \\leq  \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c_i -  \\alpha v \\right) , e \\right \\rangle = q_i(\\alpha)$\r\nfor all $\\alpha \\in \\mathbb{R}$, which implies the inequality in (\\ref{eq:bound_q}).\r\n\\end{proof}\n\n\\begin{proof}\r\nLet $c \\in \\mathbb {E}$ be an element whose $\\ell$-th block element is $c_\\ell = c (v_\\ell)_i$ and other block elements take $0$.\r\nFor any real number $\\alpha \\in \\mathbb{R}$, Proposition \\ref {prop:p-d-relation} ensures that\r\n\\begin{align}\r\n\\label{eq:upper-bound-direct2}\r\n\\langle {c(v_\\ell)}_i , x_\\ell \\rangle = \\langle c,x \\rangle &\\leq \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c -  \\alpha v \\right) , e \\right \\rangle \\notag \\\\\r\n&= \\left \\langle \\mathcal{P}_{\\mathcal{K}_\\ell} \\left( {c(v_\\ell)}_i -  \\alpha v_\\ell \\right) , e_\\ell \\right \\rangle + \\sum_{j \\neq \\ell}^p \\left \\langle \\mathcal{P}_{\\mathcal{K}_j} \\left( -  \\alpha v_j \\right) , e_j \\right \\rangle \\notag \\\\\r\n&= \\left[ 1 - \\alpha {\\lambda(v_\\ell)}_i \\right]^+ + \\sum_{k \\neq i}^{r_\\ell} \\left[ -\\alpha {\\lambda(v_\\ell)}_k \\right]^+ + \\sum_{j \\neq \\ell}^p \\sum_{k=1}^{r_j} \\left[  -\\alpha {\\lambda(v_j)}_k \\right]^+  = q_{\\ell,i}(\\alpha). \r\n\\end{align}\r\nWe obtain (\\ref{eq:upper-bound-direct1}) by following a similar argument to the one used in the proof of Proposition \\ref{prop:upper-bound}.\r\n\\end{proof}","units":[{"raw_tex":"\\begin{lemma}\r\n\\label{lemma:0}\r\nLet $(\\mathbb{E}, \\circ)$ be a Euclidean Jordan algebra with the \\modifySecond{associated} symmetric cone $\\mathcal{K}$.\r\nFor any $y \\in \\mathbb{E}$, the following equation holds:\r\n\\begin{equation}\r\n\\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\langle y ,x \\rangle  =  \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( y \\right)  ,e \\right \\rangle . \\notag\r\n\\end{equation}\r\n\\end{lemma}","start":38796,"end":39205},{"raw_tex":"\\begin{proof}\r\nUsing the decomposition $y = \\sum_{i=1}^r \\lambda_i c_i$ obtained by Proposition \\ref{prop:spectral}, we see that \r\n\\begin{equation}\r\n\\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\langle y ,x \\rangle  = \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\left \\langle \\sum_{i=1}^r \\lambda_i c_i , x \\right \\rangle = \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}}  \\sum_{i=1}^r \\lambda_i  \\left \\langle c_i , x \\right \\rangle . \\label{lemma:0-eq1}\r\n\\end{equation}\r\nNoting that $x \\in \\mathcal{K}, e-x \\in \\mathcal{K}$ from $\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}$, since $c_i \\in \\mathcal{K}$ is primitive idempotent, we find that \r\n$\\langle c_i,x \\rangle \\geq 0$ and $\\langle c_i, e-x \\rangle \\geq 0$, which implies that $0 \\leq \\langle c_i,x \\rangle \\leq 1$.\r\nThus, letting $I_1$ be the set of indices for which $\\lambda_i \\leq 0$ and $I_2$ be the set of indices for which $\\lambda_i > 0$, if there exists an $x$ satisfying \r\n\\begin{equation}\r\n\\langle c_i,x \\rangle =\r\n\\begin{cases}\r\n0 & i \\in I_1 \\\\\r\n1 & i \\in I_2\r\n\\end{cases}, \r\n\\label{lemma:0-eq2}\r\n\\end{equation}\r\nthen such an $x$ is an optimal solution of (\\ref{lemma:0-eq1}). In fact, if we define $x^* = \\sum_{i \\in I_2} c_i$, then by the dedfinition of the Jordan frame, $x^*$ satisfies (\\ref{lemma:0-eq2}) and $\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}$ and becomes an optimal solution of (\\ref{lemma:0-eq1}). In this case, the optimal value of (\\ref{lemma:0-eq1}) turns out to be\r\n\\begin{equation}\r\n\\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}}  \\sum_{i=1}^r \\lambda_i  \\left \\langle c_i , x \\right \\rangle = \\sum_{i=1}^r \\lambda_i  \\left \\langle c_i , x^* \\right \\rangle \r\n= \\sum_{i \\in I_2} \\lambda_i \r\n= \\sum_{i=1}^r [\\lambda_i]^+\r\n= \\langle \\mathcal{P}_\\mathcal{K} (y) , e \\rangle. \r\n\\notag\r\n\\end{equation}\r\n\\end{proof}","start":39207,"end":40975},{"raw_tex":"\\begin{proposition}\r\n\\label{prop:p-d-relation} \r\nLet $(\\mathbb{E}, \\circ)$ be a Euclidean Jordan Algebra with the corresponding symmetric cone  $\\mathcal{K}$.\r\nFor a given $c \\in \\mathbb{E}$, consider the problem\r\n\\begin{equation}\r\n\\notag\r\n\\begin{array}{llllll}\r\n\\mbox{\\rm max}\t&\\langle c , x \\rangle \t&\t&\\mbox{\\rm s.t}\t&\\mathcal{A}(x) = \\bm{0},\t&\\bm{0} \\leq \\lambda (x) \\leq \\bm{1}  \\notag.\r\n\\end{array}\r\n\\end{equation}\r\nThe dual problem of the above is \r\n\\begin{equation}\r\n\\notag\r\n\\begin{array}{lllll}\r\n\\mbox{\\rm min}\t&\\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c - u  \\right) , e \\right \\rangle\t&\t&\\mbox{\\rm s.t}\t&u \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*. \\notag\r\n\\end{array}\r\n\\end{equation}\r\n\\end{proposition}","start":40979,"end":41705},{"raw_tex":"\\begin{proof}\r\nDefine the Lagrangian function $L(x,w)$ as $L(x,w) :=   \\langle c , x \\rangle -  w^\\top \\mathcal{A}(x)$ where $w \\in \\mathbb{R}^m$ is the Lagrange multiplier. \r\n\\modifyThird{\r\nSupoose that $x^*$ is an optimal sotution of the primal problem. \r\nThen, for any $w \\in \\mathbb{R}^m$,  we have $\\langle c,x^* \\rangle = L(w,x^*) \\leq \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} L(w,x)$, and hence,}\r\n\\begin{align*}\r\n\\modifyThird{\\langle c, x^* \\rangle} \\leq \\min_w \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} L(x,w)\r\n&=  \\min_w \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\{  \\langle c , x \\rangle - \\langle \\mathcal{A}^*(w) ,x \\rangle \\}  \\\\\r\n&=  \\min_w \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\{  \\langle c -  \\mathcal{A}^*(w) ,x \\rangle \\}  \\\\\r\n&=  \\min_w   \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c -  \\mathcal{A}^*(w) \\right)  ,e \\right \\rangle \\ \\ \\ \\mbox{(by  lemma \\ref{lemma:0})}  \\\\\r\n&= \\min_{u \\in \\mbox{\\footnotesize range} \\mathcal{A}^*}  \\langle \\mathcal{P}_\\mathcal{K} \\left( c - u \\right) , e \\rangle.\r\n\\end{align*}\r\n%and the dual problem is \r\n%\\begin{equation}\r\n%\\notag\r\n%\\begin{array}{lllll}\r\n%\\mbox{\\rm min}\t&\\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c - u  \\right) , e \\right \\rangle\t&\t&\\mbox{\\rm s.t}\t&u \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*. \\notag\r\n%\\end{array}\r\n%\\end{equation}\r\n\\end{proof}","start":41707,"end":43045},{"raw_tex":"\\begin{proposition}\r\n\\label{prop:upper-bound}\r\n\r\nSuppose that $v \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*$ is given by $v = \\sum_{i=1}^r \\lambda_i c_i$ as in Proposition \\ref{prop:spectral}. \r\nFor each $i \\in \\{ 1, \\dots , r \\}$ and $\\alpha \\in \\mathbb{R}$, define $q_i(\\alpha) := \\left[    1-\\alpha \\lambda_i  \\right]^+ + \\sum_{j \\neq i }^r \\left[ -  \\alpha \\lambda_j  \\right]^+$.\r\nThen, the following relations hold for any $x \\in F_{{\\rm P}_{S_{\\infty}}(\\mathcal{A})}$ and $i \\in \\{ 1, \\dots , r \\}$\\modifySecond{:}\r\n\\begin{equation}\r\n\\label{eq:bound_q}\r\n\\langle c_i , x \\rangle \\leq  \r\n\\min_{\\alpha \\in \\mathbb{R}} q_i(\\alpha) = \r\n\\begin{cases}\r\n\\min \\left\\{  1 ,  \\left \\langle  e , \\mathcal{P}_{\\mathcal{K}} \\left( -  \\frac{1}{\\lambda_i} v  \\right) \\right \\rangle \\right\\} & \\mbox{if $\\lambda_i \\neq 0$}, \\\\\r\n1 & \\mbox{if $\\lambda_i = 0$}\\modifySecond{.}\r\n\\end{cases}\r\n\\end{equation}\r\n\\end{proposition}","start":43141,"end":44058},{"raw_tex":"\\begin{proof}\r\n\r\nFor each $i \\in \\{ 1, 2, \\dots , r \\}$, we have \r\n\\begin{equation}\r\n\\mathcal{P}_\\mathcal{K} \\left( c_i -  \\alpha v \\right) = \\mathcal{P}_\\mathcal{K} \\left( c_i - \\alpha \\sum_{j=1}^r \\lambda_j c_j \\right) = \\mathcal{P}_\\mathcal{K} \\left( (1-\\alpha \\lambda_i ) c_i -   \\sum_{j \\neq i }^r \\alpha \\lambda_j c_j  \\right), \\notag\r\n\\end{equation}\r\nand hence, \r\n\\begin{equation}\r\n\\label{eq:q_i(alpha)}\r\n\\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c_i -  \\alpha v \\right) , e \\right \\rangle\r\n=\r\n\\left \\langle \\mathcal{P}_\\mathcal{K} \\left(    (1-\\alpha \\lambda_i ) c_i -   \\sum_{j \\neq i }^r \\alpha \\lambda_j c_j   \\right) , \\sum_{i=1}^r c_i \\right \\rangle \r\n= \\left[    1-\\alpha \\lambda_i  \\right]^+ + \\sum_{j \\neq i }^r \\left[ -  \\alpha \\lambda_j  \\right]^+  = q_i(\\alpha) .\r\n\\end{equation}\r\n\r\nNote that, since $q_i(\\alpha)$ is a piece-wise linear convex function, if $\\lambda_i = 0$, it attains the minimum at $\\alpha = 0$ with $q_i(0) = 1$, and if $\\lambda_i \\neq 0$, it attains the minimum at $\\alpha = 0$ with $q_i(0) = 1$ or at $\\alpha =  \\frac{1}{\\lambda_i}$ with\r\n\\begin{equation}\r\n\\notag\r\nq \\left( \\frac{1}{\\lambda_i} \\right) = \\sum_{j \\neq i}^r \\left[ - \\frac{\\lambda_j}{\\lambda_i}  \\right]^+ \r\n= \\sum_{j=1}^r \\left[ - \\frac{\\lambda_j}{\\lambda_i}  \\right]^+ \r\n= \\left \\langle e , \\mathcal{P}_{\\mathcal{K}} \\left( -  \\frac{1}{\\lambda_i} v  \\right) \\right \\rangle.\r\n\\end{equation}\r\nThus, we obtain equivalence in (\\ref{eq:bound_q}).\r\nSince $\\alpha v \\in \\mbox{range} \\hspace{0.3mm} \\mathcal{A}^*$ for all $\\alpha \\in \\mathbb{R}$, \r\nfor each $i \\in \\{ 1, \\dots , r \\}$, Proposition \\ref{prop:p-d-relation} and  (\\ref{eq:q_i(alpha)})  ensure that \r\n$\\langle c_i , x \\rangle \\leq  \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c_i -  \\alpha v \\right) , e \\right \\rangle = q_i(\\alpha)$\r\nfor all $\\alpha \\in \\mathbb{R}$, which implies the inequality in (\\ref{eq:bound_q}).\r\n\\end{proof}","start":44064,"end":45961},{"raw_tex":"\\begin{proof}\r\nLet $c \\in \\mathbb {E}$ be an element whose $\\ell$-th block element is $c_\\ell = c (v_\\ell)_i$ and other block elements take $0$.\r\nFor any real number $\\alpha \\in \\mathbb{R}$, Proposition \\ref {prop:p-d-relation} ensures that\r\n\\begin{align}\r\n\\label{eq:upper-bound-direct2}\r\n\\langle {c(v_\\ell)}_i , x_\\ell \\rangle = \\langle c,x \\rangle &\\leq \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c -  \\alpha v \\right) , e \\right \\rangle \\notag \\\\\r\n&= \\left \\langle \\mathcal{P}_{\\mathcal{K}_\\ell} \\left( {c(v_\\ell)}_i -  \\alpha v_\\ell \\right) , e_\\ell \\right \\rangle + \\sum_{j \\neq \\ell}^p \\left \\langle \\mathcal{P}_{\\mathcal{K}_j} \\left( -  \\alpha v_j \\right) , e_j \\right \\rangle \\notag \\\\\r\n&= \\left[ 1 - \\alpha {\\lambda(v_\\ell)}_i \\right]^+ + \\sum_{k \\neq i}^{r_\\ell} \\left[ -\\alpha {\\lambda(v_\\ell)}_k \\right]^+ + \\sum_{j \\neq \\ell}^p \\sum_{k=1}^{r_j} \\left[  -\\alpha {\\lambda(v_j)}_k \\right]^+  = q_{\\ell,i}(\\alpha). \r\n\\end{align}\r\nWe obtain (\\ref{eq:upper-bound-direct1}) by following a similar argument to the one used in the proof of Proposition \\ref{prop:upper-bound}.\r\n\\end{proof}","start":56588,"end":57682}]},"nodes":[{"id":"S001","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"欧几里得若尔当代数","source_units":[{"raw_tex":"Let $\\mathbb{E}$ be a real-valued vector space equipped with an inner product $\\langle \\cdot, \\cdot \\rangle$ and a bilinear operation $\\circ$ : $\\mathbb{E} \\times \\mathbb{E} \\rightarrow \\mathbb{E}$,\r\nand $e$ be the identity element, i.e.,$x \\circ e = e \\circ x = x$ holds for any $ x \\in \\mathbb{E}$.\r\n$(\\mathbb{E}, \\circ)$ is called a Euclidean Jordan algebra if it satisfies\r\n\\begin{align*}\r\nx \\circ y = y  \\circ x,  \\ \\\r\nx \\circ (x^2 \\circ y) = x^2 \\circ (x \\circ y), \\ \\\r\n\\langle x \\circ y , z \\rangle = \\langle y , x \\circ z \\rangle\r\n\\end{align*}\r\nfor all $x,y,z \\in \\mathbb{E}$ and $x^2 := x \\circ x$.\r\n","start":19433,"end":20042}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"逆元记号","source_units":[{"raw_tex":"We denote $y \\in \\mathbb{E}$ as $x^{-1}$ if  $y$ satisfies $x \\circ y = e$.\r\n","start":20042,"end":20119}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"本原幂等元与若尔当标架","source_units":[{"raw_tex":"$c \\in \\mathbb{E}$ is called an {\\em idempotent} if it satisfies $c \\circ c = c$, and an idempotent $c$ is called {\\em primitive} if it can not be written as a sum of two or more nonzero idempotents. \r\nA set of primitive idempotents $c_1, c_2, \\ldots c_k$ is called a {\\em Jordan frame} if $c_1, \\ldots c_k$ satisfy\r\n\\begin{equation}\r\n\\notag\r\nc_i \\circ c_j = 0 \\ (i \\neq j), \\ \\\r\nc_i \\circ c_i = c_i \\ (i= 1 , \\dots , k), \\ \\\r\n\\sum_{i=1}^k c_i = e.\r\n\\end{equation}\r\n","start":20119,"end":20585}]},{"id":"S004","semantic_type":"proposition","kind":"proposition","labels":["prop:spectral"],"declared_external":true,"declaration_text":["原始前提清单许可 B0014；原文引用 Faraut and Korányi (1994), Theorem III.1.2。"],"source_only":false,"title":"若尔当谱定理","source_units":[{"raw_tex":"\\begin{proposition}[Spectral theorem (cf. Theorem III.1.2 of \\cite{Faraut1994})]\r\n\\label{prop:spectral}\r\nLet $(\\mathbb{E}, \\circ)$ be a Euclidean Jordan algebra having rank $r$. \r\nFor any $x \\in \\mathbb{E} $, there exist real numbers $\\lambda_1 , \\dots , \\lambda_r$ and a Jordan frame $c_1 , \\dots , c_r $ for which the following holds:\r\n\\begin{equation}\r\nx = \\sum_{i=1}^r \\lambda_i c_i \\notag.\r\n\\end{equation}\r\nThe numbers $\\lambda_1 , \\dots , \\lambda_r$ are uniquely determined {\\em eigenvalues} of $x$ (with their multiplicities).\r\n","start":20879,"end":21414},{"raw_tex":"\\end{proposition}\r\n","start":21510,"end":21529}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":["prop:spectral"],"declared_external":false,"source_only":false,"title":"谱迹","source_units":[{"raw_tex":"Furthermore, ${\\rm trace}(x) := \\sum_{i=1}^r \\lambda_i$, $\\det(x) := \\prod_{i=1}^r \\lambda_i$.\r\n","start":21414,"end":21510}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["prop:spectral"],"declared_external":false,"source_only":false,"title":"谱行列式","source_units":[{"raw_tex":"Furthermore, ${\\rm trace}(x) := \\sum_{i=1}^r \\lambda_i$, $\\det(x) := \\prod_{i=1}^r \\lambda_i$.\r\n","start":21414,"end":21510}]},{"id":"S007","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单中的标准自对偶性；原文同时引用 Faraut and Korányi (1994), Theorems III.2.1、III.3.1，并交代后续谱性质的文献来源。"],"source_only":false,"title":"对称锥与自对偶性","source_units":[{"raw_tex":"A proper cone is symmetric if it is self-dual and homogeneous. \r\nIt is known that the set of squares $\\mathcal{K} = \\{ x^2 : x \\in \\mathbb{E} \\}$ is the symmetric cone of  $\\mathbb{E}$  (cf. Theorems III.2.1 and III.3.1 of \\cite{Faraut1994}).\r\nThe following properties can be derived from the results in \\cite{Faraut1994}, as in  Corollary 2.3 of \\cite{Yoshise2007}:\r\n","start":22073,"end":22441}]},{"id":"S008","semantic_type":"proposition","kind":"proposition","labels":["prop:lambda-Jordan"],"declared_external":true,"declaration_text":["原始前提清单许可 B0017；原文在第284行说明可由 Faraut1994 的结果推出，并参照 Yoshise2007, Corollary 2.3。"],"source_only":false,"title":"锥归属的谱刻画","source_units":[{"raw_tex":"\\begin{proposition}\r\n\\label{prop:lambda-Jordan}\r\nLet $x \\in \\mathbb{E}$ and let $\\sum_{j=1}^r \\lambda_j c_j$ be a decomposition of $x$ given by Propositoin \\ref{prop:spectral}. Then\r\n\\begin{description}\r\n\\item[(i)]\r\n$x \\in \\mathcal{K}$ if and only if  $\\lambda_j \\geq 0 \\ (j=1,2,\\ldots,r)$, \r\n\\item[(ii)]\r\n$x \\in {\\rm int} \\hspace{0.75mm} \\mathcal{K}$ if and only if $\\lambda_j > 0 \\ (j=1,2,\\ldots,r)$.\r\n\\end{description}\r\n\\end{proposition}\r\n","start":22441,"end":22883}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"锥投影的谱截断","source_units":[{"raw_tex":"\r\nFrom Propositions \\ref{prop:spectral} and \\ref{prop:lambda-Jordan} for any $x \\in \\mathbb{E}$, its projection $P_\\mathcal{K} (x)$ onto $\\mathcal{K}$ can be written as an operation to round all negative eigenvalues of $x$ to $0$, i.e., \r\n$P_\\mathcal{K} (x) = \\sum_{i=1}^r [\\lambda_i]^+ c_i$, where $[\\cdot]^+$ denotes the projection onto the nonnegative orthant. \r\n","start":22883,"end":23249}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"简单代数与简单锥","source_units":[{"raw_tex":"A Euclidean Jordan algebra $(\\mathbb{E} , \\circ)$ is called {\\em simple} if it cannot be written as any Cartesian product of non-zero Euclidean Jordan algebras.\r\nIf the Euclidean Jordan algebra $(\\mathbb{E} , \\circ)$ associated with a symmetric cone $\\mathcal{K}$ is simple, then we say that $\\mathcal{K}$ is {\\em simple}.\r\n","start":24057,"end":24381}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","labels":["eq:r and e"],"declared_external":false,"source_only":false,"title":"乘积锥与块秩","source_units":[{"raw_tex":"In this paper, we will consider that $\\mathcal{K}$ is given by a Cartesian product of $p$ simple symmetric cones $\\mathcal{K}_\\ell$,\r\n$\\mathcal{K} := \\mathcal{K}_1 \\times \\dots \\times \\mathcal{K}_p$, whose rank and identity element are $r_\\ell$ and $e_\\ell$ $(\\ell=1, \\ldots, p)$.\r\nThe rank $r$ and the identity element of  $\\mathcal{K}$ are given by\r\n\\begin{equation}\r\n\\label{eq:r and e}\r\nr = \\sum_{\\ell=1}^p r_\\ell, \\ \\ e = (e_1 , \\dots , e_p).\r\n\\end{equation}\r\n","start":24381,"end":24845}]},{"id":"S012","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"块元素记号","source_units":[{"raw_tex":"In what follows, $x_\\ell$ stands for the $\\ell$-th block element of  $x \\in \\mathcal{K}$, i.e., $x = (x_1, \\dots, x_p) \\in \\mathcal{K}_1 \\times \\dots \\times \\mathcal{K}_p$.\r\n","start":24845,"end":25019}]},{"id":"S013","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"最小特征值记号","source_units":[{"raw_tex":"For each $\\ell=1, \\cdots, p$, we define $\\lambda_{\\min}(x_\\ell) := \\min\\{ \\lambda_1, \\cdots, \\lambda_{r_\\ell} \\}$ where $\\lambda_1, \\cdots,  \\lambda_{r_\\ell}$ are eigenvalues of $x_\\ell$.\r\nThe minimum eigenvalue $\\lambda_{\\min}(x)$ of $x \\in \\mathcal{K}$ is given by $\\lambda_{\\min}(x) = \\mbox{min} \\{ \\lambda_{\\min}(x_1), \\cdots, \\lambda_{\\min}(x_p) \\}$.\r\n","start":25019,"end":25376}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"迹内积","source_units":[{"raw_tex":"This subsection summarizes the \\modifyFirst{notations} used in this paper.\r\nFor any $x,y \\in \\mathbb{E}$, we define the inner product  $\\langle \\cdot,\\cdot \\rangle$ and the norm $\\|\\cdot\\|_{J}$ as $\\langle x , y \\rangle := {\\rm trace}(x \\circ y)$ and $\\| x \\|_J := \\sqrt{ \\langle x , x \\rangle }$, respectively.\r\n","start":31084,"end":31397}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"若尔当范数","source_units":[{"raw_tex":"For any $x,y \\in \\mathbb{E}$, we define the inner product  $\\langle \\cdot,\\cdot \\rangle$ and the norm $\\|\\cdot\\|_{J}$ as $\\langle x , y \\rangle := {\\rm trace}(x \\circ y)$ and $\\| x \\|_J := \\sqrt{ \\langle x , x \\rangle }$, respectively.\r\n","start":31160,"end":31397}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"谱一范数","source_units":[{"raw_tex":"For any $x \\in \\mathbb{E}$ having decomposition  $x = \\sum_{i=1}^r \\lambda_i c_i$ as in  Proposition \\ref{prop:spectral}, we also define $\\| x \\|_1 := |\\lambda_1| + \\dots + |\\lambda_r|$, $\\|x\\|_\\infty := \\max \\{|\\lambda_1| , \\dots , |\\lambda_r| \\}$.\r\n","start":31397,"end":31648}]},{"id":"S017","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"谱无穷范数","source_units":[{"raw_tex":"For any $x \\in \\mathbb{E}$ having decomposition  $x = \\sum_{i=1}^r \\lambda_i c_i$ as in  Proposition \\ref{prop:spectral}, we also define $\\| x \\|_1 := |\\lambda_1| + \\dots + |\\lambda_r|$, $\\|x\\|_\\infty := \\max \\{|\\lambda_1| , \\dots , |\\lambda_r| \\}$.\r\n","start":31397,"end":31648}]},{"id":"S018","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"锥内范数的等价表示","source_units":[{"raw_tex":"For $x \\in \\mathcal{K}$, we obtain the following equivalent representations: $\\|x\\|_1 = \\langle e , x \\rangle$, $\\|x\\|_\\infty = \\lambda_{\\max} (x)$.\r\n","start":31648,"end":31798}]},{"id":"S019","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"空间维数","source_units":[{"raw_tex":"The following is a list of other definitions and frequently used symbols in the paper.\r\n\\begin{itemize}\r\n\\item $d$: the dimension of the Euclidean space $\\mathbb{E}$ corresponding to $\\mathcal{K}$,\r\n","start":31798,"end":31997}]},{"id":"S020","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"归一化可行域记号","source_units":[{"raw_tex":"\\item $F_{{\\rm P}_{S_{\\infty}}(\\mathcal{A})}$: the feasible region of ${\\rm P}_{S_{\\infty}}(\\mathcal{A})$,\r\n","start":31997,"end":32105}]},{"id":"S021","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"核空间投影记号","source_units":[{"raw_tex":"\\item $P_{\\mathcal{A}} (\\cdot) $: the projection map onto $\\mbox{ker} \\hspace{0.3mm} \\mathcal{A}$,\r\n%\\item $Q_{\\mathcal{A}} (\\cdot) $: the projection map onto $\\mbox{range} \\hspace{0.3mm} \\mathcal{A}^*$, \r\n","start":32105,"end":32311}]},{"id":"S022","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"投影与分块谱记号","source_units":[{"raw_tex":"\\item $\\mathcal{P}_\\mathcal{K} (\\cdot) $: the projection map onto $\\mathcal{K}$,\r\n\\item $\\lambda(x) \\in \\mathbb{R}^r$: an $r$-dimensional vector composed of the eigenvalues of $ x \\in \\mathcal{K}$,\r\n\\item $\\lambda(x_\\ell) \\in \\mathbb{R}^{r_\\ell}$: an $r_\\ell$-dimensional vector composed of the eigenvalues of $x_\\ell \\in \\mathcal{K}_\\ell$ ($\\ell =1,\\ldots,p$),\r\n\\item $c(x_\\ell)_i \\in \\mathcal{K}_\\ell$: the $i$-th primitive idempotent of $x_\\ell \\in \\mathbb{E}_\\ell$. When $\\mathcal{K}$ is simple, it is abbreviated as $c_i$.\r\n\\item $\\left[ \\cdot \\right]^+$: the projection map onto the nonnegative orthant, and\r\n","start":32311,"end":32926}]},{"id":"S023","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"线性算子的伴随","source_units":[{"raw_tex":"\\item $\\mathcal{A}^*(\\cdot)$: the adjoint operator of the linear operator $\\mathcal{A}(\\cdot)$, i.e., $\\langle \\mathcal{A}(x) , y \\rangle = \\left \\langle x , \\mathcal{A}^* (y) \\right\\rangle$ for all $x \\in \\mathcal{K}$ and $y \\in \\mathbb{R}^m$.\r\n\\end{itemize}\r\n","start":32926,"end":33187}]},{"id":"S024","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"归一化锥可行问题","source_units":[{"raw_tex":"We focus on the feasibility of the following problem ${\\rm P}_{S_{\\infty}}(\\mathcal{A})$, which is equivalent to  ${\\rm P}(\\mathcal{A})$: \r\n\\begin{equation}\r\n\\begin{array}{lllllllll}\r\n{\\rm P}_{S_{\\infty}}(\\mathcal{A})&\t&\\mbox{find}   &x \\in \\mbox{int} \\hspace{0.75mm} \\mathcal{K}\t&\t&\\mbox{s.t.}\t&\\mathcal{A} (x) = \\bm{0},\t&\\|x\\|_\\infty \\leq 1.\r\n\\end{array}\r\n\\notag\r\n\\end{equation}\r\n","start":33772,"end":34154}]},{"id":"S025","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"替代可行问题","source_units":[{"raw_tex":"The alternative problem ${\\rm D}(\\mathcal{A})$ of ${\\rm P}(\\mathcal{A})$ is \r\n\\begin{equation}\r\n\\notag\r\n\\begin{array}{llllllllll}\r\n{\\rm D}(\\mathcal{A})\t&\t&\\mbox{find}   &y \\in \\mathcal{K}\t&\t&\\mbox{s.t.}\t&y \\in \\mbox{range} \\hspace{0.3mm} \\mathcal{A}^*,\t&y \\neq \\bm{0},\r\n\\end{array}\r\n\\end{equation}\r\n","start":34154,"end":34453}]},{"id":"S026","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"伴随像与核的正交关系","source_units":[{"raw_tex":"where  $\\mbox{range} \\hspace{0.3mm} \\mathcal{A}^*$ is the orthogonal complement of $\\mbox{ker} \\hspace{0.3mm} \\mathcal{A}$.\r\n","start":34453,"end":34578}]},{"id":"S027","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"重申乘积锥假设","source_units":[{"raw_tex":"As we mentioned in section \\ref{subsec:symmetric cone}, \r\nwe assume that $\\mathcal{K}$ is given by a Cartesian product of $p$ simple symmetric cones $\\mathcal{K}_\\ell (\\ell=1, \\ldots, p)$, i.e., $\\mathcal{K} = \\mathcal{K}_1 \\times \\dots \\times \\mathcal{K}_p $.\r\n","start":34578,"end":34840}]},{"id":"S028","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单中的 Standard self-duality and trace-orthonormality of Jordan frames, including their blockwise product-cone versions。"],"source_only":false,"title":"标架的迹正交归一性","source_units":[]},{"id":"S029","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单中的 blockwise product-cone versions；逐块正交投影是 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"乘积空间的分块运算","source_units":[]},{"id":"S030","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单中的 Standard Lagrangian weak duality。"],"source_only":false,"title":"拉格朗日弱对偶","source_units":[]},{"id":"S031","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单中的 elementary minimization of convex piecewise-linear functions。"],"source_only":false,"title":"分段线性函数的最小化","source_units":[]},{"id":"S032","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":true,"title":"简单锥的初始迹上界","source_units":[{"raw_tex":"Let us consider the case where $\\mathcal {K}$ is simple. It is obvious that, for any feasible solution $x$ of ${\\rm P}_{S_{\\infty}}(\\mathcal{A})$, the constraint $\\|x\\|_\\infty \\leq 1$ implies that $\\langle e , x\\rangle \\leq r$, since $x \\in \\mathcal{K}$. In Proposition \\ref{prop:upper-bound}, we show that this bound may be improved as $\\langle e , x\\rangle <r$ by using a point $v \\in \\mbox{range} \\hspace{0.3mm} \\mathcal{A}^*\\setminus \\{0\\}$. To prove Proposition \\ref{prop:upper-bound}, we need the following Lemma \\ref{lemma:0} and Proposition \\ref{prop:p-d-relation}.\r\n","start":38221,"end":38796}]},{"id":"S033","semantic_type":"lemma","kind":"lemma","labels":["lemma:0"],"declared_external":false,"source_only":false,"title":"谱盒的支撑函数","source_units":[{"raw_tex":"\\begin{lemma}\r\n\\label{lemma:0}\r\nLet $(\\mathbb{E}, \\circ)$ be a Euclidean Jordan algebra with the \\modifySecond{associated} symmetric cone $\\mathcal{K}$.\r\nFor any $y \\in \\mathbb{E}$, the following equation holds:\r\n\\begin{equation}\r\n\\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\langle y ,x \\rangle  =  \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( y \\right)  ,e \\right \\rangle . \\notag\r\n\\end{equation}\r\n\\end{lemma}","start":38796,"end":39205}]},{"id":"S034","semantic_type":"proof","kind":"proof","labels":["lemma:0-eq1","lemma:0-eq2"],"declared_external":false,"source_only":false,"title":"谱盒支撑函数的完整证明","source_units":[{"raw_tex":"\\begin{proof}\r\nUsing the decomposition $y = \\sum_{i=1}^r \\lambda_i c_i$ obtained by Proposition \\ref{prop:spectral}, we see that \r\n\\begin{equation}\r\n\\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\langle y ,x \\rangle  = \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\left \\langle \\sum_{i=1}^r \\lambda_i c_i , x \\right \\rangle = \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}}  \\sum_{i=1}^r \\lambda_i  \\left \\langle c_i , x \\right \\rangle . \\label{lemma:0-eq1}\r\n\\end{equation}\r\nNoting that $x \\in \\mathcal{K}, e-x \\in \\mathcal{K}$ from $\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}$, since $c_i \\in \\mathcal{K}$ is primitive idempotent, we find that \r\n$\\langle c_i,x \\rangle \\geq 0$ and $\\langle c_i, e-x \\rangle \\geq 0$, which implies that $0 \\leq \\langle c_i,x \\rangle \\leq 1$.\r\nThus, letting $I_1$ be the set of indices for which $\\lambda_i \\leq 0$ and $I_2$ be the set of indices for which $\\lambda_i > 0$, if there exists an $x$ satisfying \r\n\\begin{equation}\r\n\\langle c_i,x \\rangle =\r\n\\begin{cases}\r\n0 & i \\in I_1 \\\\\r\n1 & i \\in I_2\r\n\\end{cases}, \r\n\\label{lemma:0-eq2}\r\n\\end{equation}\r\nthen such an $x$ is an optimal solution of (\\ref{lemma:0-eq1}). In fact, if we define $x^* = \\sum_{i \\in I_2} c_i$, then by the dedfinition of the Jordan frame, $x^*$ satisfies (\\ref{lemma:0-eq2}) and $\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}$ and becomes an optimal solution of (\\ref{lemma:0-eq1}). In this case, the optimal value of (\\ref{lemma:0-eq1}) turns out to be\r\n\\begin{equation}\r\n\\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}}  \\sum_{i=1}^r \\lambda_i  \\left \\langle c_i , x \\right \\rangle = \\sum_{i=1}^r \\lambda_i  \\left \\langle c_i , x^* \\right \\rangle \r\n= \\sum_{i \\in I_2} \\lambda_i \r\n= \\sum_{i=1}^r [\\lambda_i]^+\r\n= \\langle \\mathcal{P}_\\mathcal{K} (y) , e \\rangle. \r\n\\notag\r\n\\end{equation}\r\n\\end{proof}","start":39207,"end":40975}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":["lemma:0-eq1"],"declared_external":false,"source_only":false,"title":"沿谱标架展开目标","source_units":[{"raw_tex":"Using the decomposition $y = \\sum_{i=1}^r \\lambda_i c_i$ obtained by Proposition \\ref{prop:spectral}, we see that \r\n\\begin{equation}\r\n\\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\langle y ,x \\rangle  = \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\left \\langle \\sum_{i=1}^r \\lambda_i c_i , x \\right \\rangle = \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}}  \\sum_{i=1}^r \\lambda_i  \\left \\langle c_i , x \\right \\rangle . \\label{lemma:0-eq1}\r\n\\end{equation}\r\n","start":39222,"end":39669}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"标架坐标落入单位区间","source_units":[{"raw_tex":"Noting that $x \\in \\mathcal{K}, e-x \\in \\mathcal{K}$ from $\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}$, since $c_i \\in \\mathcal{K}$ is primitive idempotent, we find that \r\n$\\langle c_i,x \\rangle \\geq 0$ and $\\langle c_i, e-x \\rangle \\geq 0$, which implies that $0 \\leq \\langle c_i,x \\rangle \\leq 1$.\r\n","start":39669,"end":39962}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":["lemma:0-eq2"],"declared_external":false,"source_only":false,"title":"构造达到上界的谱点","source_units":[{"raw_tex":"Thus, letting $I_1$ be the set of indices for which $\\lambda_i \\leq 0$ and $I_2$ be the set of indices for which $\\lambda_i > 0$, if there exists an $x$ satisfying \r\n\\begin{equation}\r\n\\langle c_i,x \\rangle =\r\n\\begin{cases}\r\n0 & i \\in I_1 \\\\\r\n1 & i \\in I_2\r\n\\end{cases}, \r\n\\label{lemma:0-eq2}\r\n\\end{equation}\r\nthen such an $x$ is an optimal solution of (\\ref{lemma:0-eq1}). In fact, if we define $x^* = \\sum_{i \\in I_2} c_i$, then by the dedfinition of the Jordan frame, $x^*$ satisfies (\\ref{lemma:0-eq2}) and $\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}$ and becomes an optimal solution of (\\ref{lemma:0-eq1}). In this case, the optimal value of (\\ref{lemma:0-eq1}) turns out to be\r\n","start":39962,"end":40637}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"识别最优值为投影迹","source_units":[{"raw_tex":"\\begin{equation}\r\n\\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}}  \\sum_{i=1}^r \\lambda_i  \\left \\langle c_i , x \\right \\rangle = \\sum_{i=1}^r \\lambda_i  \\left \\langle c_i , x^* \\right \\rangle \r\n= \\sum_{i \\in I_2} \\lambda_i \r\n= \\sum_{i=1}^r [\\lambda_i]^+\r\n= \\langle \\mathcal{P}_\\mathcal{K} (y) , e \\rangle. \r\n\\notag\r\n\\end{equation}\r\n","start":40637,"end":40964}]},{"id":"S039","semantic_type":"proposition","kind":"proposition","labels":["prop:p-d-relation"],"declared_external":false,"source_only":false,"title":"谱盒约束的拉格朗日对偶","source_units":[{"raw_tex":"\\begin{proposition}\r\n\\label{prop:p-d-relation} \r\nLet $(\\mathbb{E}, \\circ)$ be a Euclidean Jordan Algebra with the corresponding symmetric cone  $\\mathcal{K}$.\r\nFor a given $c \\in \\mathbb{E}$, consider the problem\r\n\\begin{equation}\r\n\\notag\r\n\\begin{array}{llllll}\r\n\\mbox{\\rm max}\t&\\langle c , x \\rangle \t&\t&\\mbox{\\rm s.t}\t&\\mathcal{A}(x) = \\bm{0},\t&\\bm{0} \\leq \\lambda (x) \\leq \\bm{1}  \\notag.\r\n\\end{array}\r\n\\end{equation}\r\nThe dual problem of the above is \r\n\\begin{equation}\r\n\\notag\r\n\\begin{array}{lllll}\r\n\\mbox{\\rm min}\t&\\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c - u  \\right) , e \\right \\rangle\t&\t&\\mbox{\\rm s.t}\t&u \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*. \\notag\r\n\\end{array}\r\n\\end{equation}\r\n\\end{proposition}","start":40979,"end":41705}]},{"id":"S040","semantic_type":"problem_setting","kind":"problem_setting","labels":["prop:p-d-relation"],"declared_external":false,"source_only":false,"title":"闭谱盒上的线性优化","source_units":[{"raw_tex":"Let $(\\mathbb{E}, \\circ)$ be a Euclidean Jordan Algebra with the corresponding symmetric cone  $\\mathcal{K}$.\r\nFor a given $c \\in \\mathbb{E}$, consider the problem\r\n\\begin{equation}\r\n\\notag\r\n\\begin{array}{llllll}\r\n\\mbox{\\rm max}\t&\\langle c , x \\rangle \t&\t&\\mbox{\\rm s.t}\t&\\mathcal{A}(x) = \\bm{0},\t&\\bm{0} \\leq \\lambda (x) \\leq \\bm{1}  \\notag.\r\n\\end{array}\r\n\\end{equation}\r\n","start":41028,"end":41401}]},{"id":"S041","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"对偶表达式的完整证明","source_units":[{"raw_tex":"\\begin{proof}\r\nDefine the Lagrangian function $L(x,w)$ as $L(x,w) :=   \\langle c , x \\rangle -  w^\\top \\mathcal{A}(x)$ where $w \\in \\mathbb{R}^m$ is the Lagrange multiplier. \r\n\\modifyThird{\r\nSupoose that $x^*$ is an optimal sotution of the primal problem. \r\nThen, for any $w \\in \\mathbb{R}^m$,  we have $\\langle c,x^* \\rangle = L(w,x^*) \\leq \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} L(w,x)$, and hence,}\r\n\\begin{align*}\r\n\\modifyThird{\\langle c, x^* \\rangle} \\leq \\min_w \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} L(x,w)\r\n&=  \\min_w \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\{  \\langle c , x \\rangle - \\langle \\mathcal{A}^*(w) ,x \\rangle \\}  \\\\\r\n&=  \\min_w \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\{  \\langle c -  \\mathcal{A}^*(w) ,x \\rangle \\}  \\\\\r\n&=  \\min_w   \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c -  \\mathcal{A}^*(w) \\right)  ,e \\right \\rangle \\ \\ \\ \\mbox{(by  lemma \\ref{lemma:0})}  \\\\\r\n&= \\min_{u \\in \\mbox{\\footnotesize range} \\mathcal{A}^*}  \\langle \\mathcal{P}_\\mathcal{K} \\left( c - u \\right) , e \\rangle.\r\n\\end{align*}\r\n%and the dual problem is \r\n%\\begin{equation}\r\n%\\notag\r\n%\\begin{array}{lllll}\r\n%\\mbox{\\rm min}\t&\\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c - u  \\right) , e \\right \\rangle\t&\t&\\mbox{\\rm s.t}\t&u \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*. \\notag\r\n%\\end{array}\r\n%\\end{equation}\r\n\\end{proof}","start":41707,"end":43045}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"定义拉格朗日函数","source_units":[{"raw_tex":"Define the Lagrangian function $L(x,w)$ as $L(x,w) :=   \\langle c , x \\rangle -  w^\\top \\mathcal{A}(x)$ where $w \\in \\mathbb{R}^m$ is the Lagrange multiplier. \r\n\\modifyThird{\r\n","start":41722,"end":41898}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"建立弱对偶上界","source_units":[{"raw_tex":"Supoose that $x^*$ is an optimal sotution of the primal problem. \r\nThen, for any $w \\in \\mathbb{R}^m$,  we have $\\langle c,x^* \\rangle = L(w,x^*) \\leq \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} L(w,x)$, and hence,}\r\n\\begin{align*}\r\n\\modifyThird{\\langle c, x^* \\rangle} \\leq \\min_w \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} L(x,w)\r\n&=  \\min_w \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\{  \\langle c , x \\rangle - \\langle \\mathcal{A}^*(w) ,x \\rangle \\}  \\\\\r\n&=  \\min_w \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\{  \\langle c -  \\mathcal{A}^*(w) ,x \\rangle \\}  \\\\\r\n","start":41898,"end":42459}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"计算并改写对偶目标","source_units":[{"raw_tex":"&=  \\min_w   \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c -  \\mathcal{A}^*(w) \\right)  ,e \\right \\rangle \\ \\ \\ \\mbox{(by  lemma \\ref{lemma:0})}  \\\\\r\n&= \\min_{u \\in \\mbox{\\footnotesize range} \\mathcal{A}^*}  \\langle \\mathcal{P}_\\mathcal{K} \\left( c - u \\right) , e \\rangle.\r\n\\end{align*}\r\n","start":42459,"end":42749}]},{"id":"S045","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"已注释的对偶重述","source_units":[{"raw_tex":"%and the dual problem is \r\n%\\begin{equation}\r\n%\\notag\r\n%\\begin{array}{lllll}\r\n%\\mbox{\\rm min}\t&\\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c - u  \\right) , e \\right \\rangle\t&\t&\\mbox{\\rm s.t}\t&u \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*. \\notag\r\n%\\end{array}\r\n%\\end{equation}\r\n","start":42749,"end":43034}]},{"id":"S046","semantic_type":"proposition","kind":"proposition","labels":["eq:bound_q","prop:upper-bound"],"declared_external":false,"source_only":false,"title":"简单锥的谱坐标割界","source_units":[{"raw_tex":"\\begin{proposition}\r\n\\label{prop:upper-bound}\r\n\r\nSuppose that $v \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*$ is given by $v = \\sum_{i=1}^r \\lambda_i c_i$ as in Proposition \\ref{prop:spectral}. \r\nFor each $i \\in \\{ 1, \\dots , r \\}$ and $\\alpha \\in \\mathbb{R}$, define $q_i(\\alpha) := \\left[    1-\\alpha \\lambda_i  \\right]^+ + \\sum_{j \\neq i }^r \\left[ -  \\alpha \\lambda_j  \\right]^+$.\r\nThen, the following relations hold for any $x \\in F_{{\\rm P}_{S_{\\infty}}(\\mathcal{A})}$ and $i \\in \\{ 1, \\dots , r \\}$\\modifySecond{:}\r\n\\begin{equation}\r\n\\label{eq:bound_q}\r\n\\langle c_i , x \\rangle \\leq  \r\n\\min_{\\alpha \\in \\mathbb{R}} q_i(\\alpha) = \r\n\\begin{cases}\r\n\\min \\left\\{  1 ,  \\left \\langle  e , \\mathcal{P}_{\\mathcal{K}} \\left( -  \\frac{1}{\\lambda_i} v  \\right) \\right \\rangle \\right\\} & \\mbox{if $\\lambda_i \\neq 0$}, \\\\\r\n1 & \\mbox{if $\\lambda_i = 0$}\\modifySecond{.}\r\n\\end{cases}\r\n\\end{equation}\r\n\\end{proposition}","start":43141,"end":44058}]},{"id":"S047","semantic_type":"assumption","kind":"assumption","labels":["prop:upper-bound"],"declared_external":false,"source_only":false,"title":"辅助割向量的谱表示","source_units":[{"raw_tex":"Suppose that $v \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*$ is given by $v = \\sum_{i=1}^r \\lambda_i c_i$ as in Proposition \\ref{prop:spectral}. \r\n","start":43190,"end":43343}]},{"id":"S048","semantic_type":"definition","kind":"definition","labels":["prop:upper-bound"],"declared_external":false,"source_only":false,"title":"辅助标量上界函数","source_units":[{"raw_tex":"For each $i \\in \\{ 1, \\dots , r \\}$ and $\\alpha \\in \\mathbb{R}$, define $q_i(\\alpha) := \\left[    1-\\alpha \\lambda_i  \\right]^+ + \\sum_{j \\neq i }^r \\left[ -  \\alpha \\lambda_j  \\right]^+$.\r\n","start":43343,"end":43533}]},{"id":"S049","semantic_type":"proof","kind":"proof","labels":["eq:q_i(alpha)"],"declared_external":false,"source_only":false,"title":"简单锥割界的完整证明","source_units":[{"raw_tex":"\\begin{proof}\r\n\r\nFor each $i \\in \\{ 1, 2, \\dots , r \\}$, we have \r\n\\begin{equation}\r\n\\mathcal{P}_\\mathcal{K} \\left( c_i -  \\alpha v \\right) = \\mathcal{P}_\\mathcal{K} \\left( c_i - \\alpha \\sum_{j=1}^r \\lambda_j c_j \\right) = \\mathcal{P}_\\mathcal{K} \\left( (1-\\alpha \\lambda_i ) c_i -   \\sum_{j \\neq i }^r \\alpha \\lambda_j c_j  \\right), \\notag\r\n\\end{equation}\r\nand hence, \r\n\\begin{equation}\r\n\\label{eq:q_i(alpha)}\r\n\\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c_i -  \\alpha v \\right) , e \\right \\rangle\r\n=\r\n\\left \\langle \\mathcal{P}_\\mathcal{K} \\left(    (1-\\alpha \\lambda_i ) c_i -   \\sum_{j \\neq i }^r \\alpha \\lambda_j c_j   \\right) , \\sum_{i=1}^r c_i \\right \\rangle \r\n= \\left[    1-\\alpha \\lambda_i  \\right]^+ + \\sum_{j \\neq i }^r \\left[ -  \\alpha \\lambda_j  \\right]^+  = q_i(\\alpha) .\r\n\\end{equation}\r\n\r\nNote that, since $q_i(\\alpha)$ is a piece-wise linear convex function, if $\\lambda_i = 0$, it attains the minimum at $\\alpha = 0$ with $q_i(0) = 1$, and if $\\lambda_i \\neq 0$, it attains the minimum at $\\alpha = 0$ with $q_i(0) = 1$ or at $\\alpha =  \\frac{1}{\\lambda_i}$ with\r\n\\begin{equation}\r\n\\notag\r\nq \\left( \\frac{1}{\\lambda_i} \\right) = \\sum_{j \\neq i}^r \\left[ - \\frac{\\lambda_j}{\\lambda_i}  \\right]^+ \r\n= \\sum_{j=1}^r \\left[ - \\frac{\\lambda_j}{\\lambda_i}  \\right]^+ \r\n= \\left \\langle e , \\mathcal{P}_{\\mathcal{K}} \\left( -  \\frac{1}{\\lambda_i} v  \\right) \\right \\rangle.\r\n\\end{equation}\r\nThus, we obtain equivalence in (\\ref{eq:bound_q}).\r\nSince $\\alpha v \\in \\mbox{range} \\hspace{0.3mm} \\mathcal{A}^*$ for all $\\alpha \\in \\mathbb{R}$, \r\nfor each $i \\in \\{ 1, \\dots , r \\}$, Proposition \\ref{prop:p-d-relation} and  (\\ref{eq:q_i(alpha)})  ensure that \r\n$\\langle c_i , x \\rangle \\leq  \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c_i -  \\alpha v \\right) , e \\right \\rangle = q_i(\\alpha)$\r\nfor all $\\alpha \\in \\mathbb{R}$, which implies the inequality in (\\ref{eq:bound_q}).\r\n\\end{proof}","start":44064,"end":45961}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":["eq:q_i(alpha)"],"declared_external":false,"source_only":false,"title":"投影迹等于标量函数","source_units":[{"raw_tex":"For each $i \\in \\{ 1, 2, \\dots , r \\}$, we have \r\n\\begin{equation}\r\n\\mathcal{P}_\\mathcal{K} \\left( c_i -  \\alpha v \\right) = \\mathcal{P}_\\mathcal{K} \\left( c_i - \\alpha \\sum_{j=1}^r \\lambda_j c_j \\right) = \\mathcal{P}_\\mathcal{K} \\left( (1-\\alpha \\lambda_i ) c_i -   \\sum_{j \\neq i }^r \\alpha \\lambda_j c_j  \\right), \\notag\r\n\\end{equation}\r\nand hence, \r\n\\begin{equation}\r\n\\label{eq:q_i(alpha)}\r\n\\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c_i -  \\alpha v \\right) , e \\right \\rangle\r\n=\r\n\\left \\langle \\mathcal{P}_\\mathcal{K} \\left(    (1-\\alpha \\lambda_i ) c_i -   \\sum_{j \\neq i }^r \\alpha \\lambda_j c_j   \\right) , \\sum_{i=1}^r c_i \\right \\rangle \r\n= \\left[    1-\\alpha \\lambda_i  \\right]^+ + \\sum_{j \\neq i }^r \\left[ -  \\alpha \\lambda_j  \\right]^+  = q_i(\\alpha) .\r\n\\end{equation}\r\n","start":44081,"end":44867}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"确定标量最小候选点","source_units":[{"raw_tex":"Note that, since $q_i(\\alpha)$ is a piece-wise linear convex function, if $\\lambda_i = 0$, it attains the minimum at $\\alpha = 0$ with $q_i(0) = 1$, and if $\\lambda_i \\neq 0$, it attains the minimum at $\\alpha = 0$ with $q_i(0) = 1$ or at $\\alpha =  \\frac{1}{\\lambda_i}$ with\r\n","start":44869,"end":45146}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"计算非零候选点的值","source_units":[{"raw_tex":"\\begin{equation}\r\n\\notag\r\nq \\left( \\frac{1}{\\lambda_i} \\right) = \\sum_{j \\neq i}^r \\left[ - \\frac{\\lambda_j}{\\lambda_i}  \\right]^+ \r\n= \\sum_{j=1}^r \\left[ - \\frac{\\lambda_j}{\\lambda_i}  \\right]^+ \r\n= \\left \\langle e , \\mathcal{P}_{\\mathcal{K}} \\left( -  \\frac{1}{\\lambda_i} v  \\right) \\right \\rangle.\r\n\\end{equation}\r\nThus, we obtain equivalence in (\\ref{eq:bound_q}).\r\n","start":45146,"end":45516}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"沿像空间直线取得割界","source_units":[{"raw_tex":"Since $\\alpha v \\in \\mbox{range} \\hspace{0.3mm} \\mathcal{A}^*$ for all $\\alpha \\in \\mathbb{R}$, \r\nfor each $i \\in \\{ 1, \\dots , r \\}$, Proposition \\ref{prop:p-d-relation} and  (\\ref{eq:q_i(alpha)})  ensure that \r\n$\\langle c_i , x \\rangle \\leq  \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c_i -  \\alpha v \\right) , e \\right \\rangle = q_i(\\alpha)$\r\nfor all $\\alpha \\in \\mathbb{R}$, which implies the inequality in (\\ref{eq:bound_q}).\r\n","start":45516,"end":45950}]},{"id":"S054","semantic_type":"proposition","kind":"proposition","labels":["eq:q_ell,i(alpha)","eq:upper-bound-direct1","prop:upper-bound-direct"],"declared_external":false,"source_only":false,"title":"乘积对称锥的谱割界","source_units":[{"raw_tex":"\\begin{proposition}\r\n\\label{prop:upper-bound-direct}\r\nSuppose that, for any $v \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*$, the $\\ell$-th block element $v_\\ell$ of $v \\in \\mathbb{E}$ is decomposed into $v_\\ell = \\sum_{i=1}^{r_\\ell} {\\lambda(v_\\ell)}_i {c(v_\\ell)}_i$ as in Proposition \\ref{prop:spectral}.\r\nFor each $\\ell \\in \\{1,\\ldots,p \\}$ and $i \\in \\{1,\\ldots,r_p\\}$, define\r\n\\begin{equation}\r\n\\label{eq:q_ell,i(alpha)}\r\nq_{\\ell,i}(\\alpha) := \\left[ 1 - \\alpha {\\lambda(v_\\ell)}_i \\right]^+ + \\sum_{k \\neq i}^{r_\\ell} \\left[ -\\alpha {\\lambda(v_\\ell)}_k \\right]^+ + \\sum_{j \\neq \\ell}^p \\sum_{k=1}^{r_j} \\left[  -\\alpha {\\lambda(v_j)}_k \\right]^+ .\r\n\\end{equation}\r\nThen, the following relations hold for any feasible solution $x$ of ${\\rm P}_{S_{\\infty}}(\\mathcal{A})$, $\\ell \\in \\{1,\\ldots,p \\}$ and $i \\in \\{1,\\ldots,r_p\\}$.\r\n\\begin{equation}\r\n\\label{eq:upper-bound-direct1}\r\n\\langle {c(v_\\ell)}_i , x_\\ell \\rangle \\leq  \\min_{\\alpha \\in \\mathbb{R} } q_{\\ell,i}(\\alpha) =\r\n\\begin{cases}\r\n\\min \\left\\{  1 ,  \\left \\langle  e , \\mathcal{P}_{\\mathcal{K}} \\left( -  \\frac{1}{{\\lambda(v_\\ell)}_i} v  \\right) \\right \\rangle \\right\\} & \\mbox{if ${\\lambda(v_\\ell)}_i \\neq 0$}, \\\\\r\n1 & \\mbox{if ${\\lambda(v_\\ell)}_i = 0$} \r\n\\end{cases}\r\n.\r\n\\end{equation}\r\n\\end{proposition}","start":55306,"end":56584}]},{"id":"S055","semantic_type":"assumption","kind":"assumption","labels":["prop:upper-bound-direct"],"declared_external":false,"source_only":false,"title":"主命题的分块割向量","source_units":[{"raw_tex":"Suppose that, for any $v \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*$, the $\\ell$-th block element $v_\\ell$ of $v \\in \\mathbb{E}$ is decomposed into $v_\\ell = \\sum_{i=1}^{r_\\ell} {\\lambda(v_\\ell)}_i {c(v_\\ell)}_i$ as in Proposition \\ref{prop:spectral}.\r\n","start":55360,"end":55620}]},{"id":"S056","semantic_type":"definition","kind":"definition","labels":["eq:q_ell,i(alpha)","prop:upper-bound-direct"],"declared_external":false,"source_only":false,"title":"分块标量割函数","source_units":[{"raw_tex":"For each $\\ell \\in \\{1,\\ldots,p \\}$ and $i \\in \\{1,\\ldots,r_p\\}$, define\r\n\\begin{equation}\r\n\\label{eq:q_ell,i(alpha)}\r\nq_{\\ell,i}(\\alpha) := \\left[ 1 - \\alpha {\\lambda(v_\\ell)}_i \\right]^+ + \\sum_{k \\neq i}^{r_\\ell} \\left[ -\\alpha {\\lambda(v_\\ell)}_k \\right]^+ + \\sum_{j \\neq \\ell}^p \\sum_{k=1}^{r_j} \\left[  -\\alpha {\\lambda(v_j)}_k \\right]^+ .\r\n\\end{equation}\r\n","start":55620,"end":55983}]},{"id":"S057","semantic_type":"proof","kind":"proof","labels":["eq:upper-bound-direct2"],"declared_external":false,"source_only":false,"title":"乘积锥割界的完整证明","source_units":[{"raw_tex":"\\begin{proof}\r\nLet $c \\in \\mathbb {E}$ be an element whose $\\ell$-th block element is $c_\\ell = c (v_\\ell)_i$ and other block elements take $0$.\r\nFor any real number $\\alpha \\in \\mathbb{R}$, Proposition \\ref {prop:p-d-relation} ensures that\r\n\\begin{align}\r\n\\label{eq:upper-bound-direct2}\r\n\\langle {c(v_\\ell)}_i , x_\\ell \\rangle = \\langle c,x \\rangle &\\leq \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c -  \\alpha v \\right) , e \\right \\rangle \\notag \\\\\r\n&= \\left \\langle \\mathcal{P}_{\\mathcal{K}_\\ell} \\left( {c(v_\\ell)}_i -  \\alpha v_\\ell \\right) , e_\\ell \\right \\rangle + \\sum_{j \\neq \\ell}^p \\left \\langle \\mathcal{P}_{\\mathcal{K}_j} \\left( -  \\alpha v_j \\right) , e_j \\right \\rangle \\notag \\\\\r\n&= \\left[ 1 - \\alpha {\\lambda(v_\\ell)}_i \\right]^+ + \\sum_{k \\neq i}^{r_\\ell} \\left[ -\\alpha {\\lambda(v_\\ell)}_k \\right]^+ + \\sum_{j \\neq \\ell}^p \\sum_{k=1}^{r_j} \\left[  -\\alpha {\\lambda(v_j)}_k \\right]^+  = q_{\\ell,i}(\\alpha). \r\n\\end{align}\r\nWe obtain (\\ref{eq:upper-bound-direct1}) by following a similar argument to the one used in the proof of Proposition \\ref{prop:upper-bound}.\r\n\\end{proof}","start":56588,"end":57682}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"零填充所选块幂等元","source_units":[{"raw_tex":"Let $c \\in \\mathbb {E}$ be an element whose $\\ell$-th block element is $c_\\ell = c (v_\\ell)_i$ and other block elements take $0$.\r\n","start":56603,"end":56734}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"取得全空间弱对偶上界","source_units":[{"raw_tex":"For any real number $\\alpha \\in \\mathbb{R}$, Proposition \\ref {prop:p-d-relation} ensures that\r\n\\begin{align}\r\n\\label{eq:upper-bound-direct2}\r\n\\langle {c(v_\\ell)}_i , x_\\ell \\rangle = \\langle c,x \\rangle &\\leq \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c -  \\alpha v \\right) , e \\right \\rangle \\notag \\\\\r\n","start":56734,"end":57041}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":["eq:upper-bound-direct2"],"declared_external":false,"source_only":false,"title":"把投影迹分解到各块","source_units":[{"raw_tex":"&= \\left \\langle \\mathcal{P}_{\\mathcal{K}_\\ell} \\left( {c(v_\\ell)}_i -  \\alpha v_\\ell \\right) , e_\\ell \\right \\rangle + \\sum_{j \\neq \\ell}^p \\left \\langle \\mathcal{P}_{\\mathcal{K}_j} \\left( -  \\alpha v_j \\right) , e_j \\right \\rangle \\notag \\\\\r\n","start":57041,"end":57285}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"识别分块正部和","source_units":[{"raw_tex":"&= \\left[ 1 - \\alpha {\\lambda(v_\\ell)}_i \\right]^+ + \\sum_{k \\neq i}^{r_\\ell} \\left[ -\\alpha {\\lambda(v_\\ell)}_k \\right]^+ + \\sum_{j \\neq \\ell}^p \\sum_{k=1}^{r_j} \\left[  -\\alpha {\\lambda(v_j)}_k \\right]^+  = q_{\\ell,i}(\\alpha). \r\n\\end{align}\r\n","start":57285,"end":57529}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"复用标量最小化论证","source_units":[{"raw_tex":"We obtain (\\ref{eq:upper-bound-direct1}) by following a similar argument to the one used in the proof of Proposition \\ref{prop:upper-bound}.\r\n","start":57529,"end":57671}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S054","edges":[{"source_id":"S001","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S008","kind":"supports"},{"source_id":"S004","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S004","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S008","target_id":"S018","kind":"supports"},{"source_id":"S005","target_id":"S018","kind":"supports"},{"source_id":"S014","target_id":"S018","kind":"supports"},{"source_id":"S028","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S020","kind":"supports"},{"source_id":"S024","target_id":"S032","kind":"supports"},{"source_id":"S017","target_id":"S032","kind":"supports"},{"source_id":"S008","target_id":"S032","kind":"supports"},{"source_id":"S005","target_id":"S032","kind":"supports"},{"source_id":"S014","target_id":"S032","kind":"supports"},{"source_id":"S028","target_id":"S032","kind":"supports"},{"source_id":"S004","target_id":"S035","kind":"supports"},{"source_id":"S008","target_id":"S036","kind":"supports"},{"source_id":"S003","target_id":"S036","kind":"supports"},{"source_id":"S007","target_id":"S036","kind":"supports"},{"source_id":"S028","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S003","target_id":"S037","kind":"supports"},{"source_id":"S028","target_id":"S037","kind":"supports"},{"source_id":"S008","target_id":"S037","kind":"supports"},{"source_id":"S035","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S009","target_id":"S038","kind":"supports"},{"source_id":"S005","target_id":"S038","kind":"supports"},{"source_id":"S028","target_id":"S038","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S034","kind":"supports"},{"source_id":"S037","target_id":"S034","kind":"supports"},{"source_id":"S038","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S040","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S023","target_id":"S043","kind":"supports"},{"source_id":"S030","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S033","target_id":"S044","kind":"supports"},{"source_id":"S023","target_id":"S044","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S043","target_id":"S041","kind":"supports"},{"source_id":"S044","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S039","kind":"supports"},{"source_id":"S004","target_id":"S047","kind":"supports"},{"source_id":"S022","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S022","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S050","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S003","target_id":"S050","kind":"supports"},{"source_id":"S009","target_id":"S050","kind":"supports"},{"source_id":"S005","target_id":"S050","kind":"supports"},{"source_id":"S028","target_id":"S050","kind":"supports"},{"source_id":"S048","target_id":"S051","kind":"supports"},{"source_id":"S031","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S048","target_id":"S052","kind":"supports"},{"source_id":"S047","target_id":"S052","kind":"supports"},{"source_id":"S009","target_id":"S052","kind":"supports"},{"source_id":"S005","target_id":"S052","kind":"supports"},{"source_id":"S028","target_id":"S052","kind":"supports"},{"source_id":"S047","target_id":"S053","kind":"supports"},{"source_id":"S039","target_id":"S053","kind":"supports"},{"source_id":"S050","target_id":"S053","kind":"supports"},{"source_id":"S024","target_id":"S053","kind":"supports"},{"source_id":"S017","target_id":"S053","kind":"supports"},{"source_id":"S008","target_id":"S053","kind":"supports"},{"source_id":"S020","target_id":"S046","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S051","target_id":"S049","kind":"supports"},{"source_id":"S052","target_id":"S049","kind":"supports"},{"source_id":"S053","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S046","kind":"supports"},{"source_id":"S004","target_id":"S055","kind":"supports"},{"source_id":"S011","target_id":"S055","kind":"supports"},{"source_id":"S012","target_id":"S055","kind":"supports"},{"source_id":"S022","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S011","target_id":"S056","kind":"supports"},{"source_id":"S022","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S058","kind":"supports"},{"source_id":"S011","target_id":"S058","kind":"supports"},{"source_id":"S012","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S055","target_id":"S059","kind":"supports"},{"source_id":"S039","target_id":"S059","kind":"supports"},{"source_id":"S024","target_id":"S059","kind":"supports"},{"source_id":"S017","target_id":"S059","kind":"supports"},{"source_id":"S008","target_id":"S059","kind":"supports"},{"source_id":"S029","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S060","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S011","target_id":"S060","kind":"supports"},{"source_id":"S014","target_id":"S060","kind":"supports"},{"source_id":"S029","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S055","target_id":"S061","kind":"supports"},{"source_id":"S056","target_id":"S061","kind":"supports"},{"source_id":"S009","target_id":"S061","kind":"supports"},{"source_id":"S005","target_id":"S061","kind":"supports"},{"source_id":"S028","target_id":"S061","kind":"supports"},{"source_id":"S059","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S056","target_id":"S062","kind":"supports"},{"source_id":"S051","target_id":"S062","kind":"supports"},{"source_id":"S052","target_id":"S062","kind":"supports"},{"source_id":"S055","target_id":"S054","kind":"supports"},{"source_id":"S056","target_id":"S054","kind":"supports"},{"source_id":"S024","target_id":"S054","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S059","target_id":"S057","kind":"supports"},{"source_id":"S060","target_id":"S057","kind":"supports"},{"source_id":"S061","target_id":"S057","kind":"supports"},{"source_id":"S062","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S054","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S054","edges":[{"source_id":"S001","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S008","kind":"supports"},{"source_id":"S004","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S004","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S008","target_id":"S018","kind":"supports"},{"source_id":"S005","target_id":"S018","kind":"supports"},{"source_id":"S014","target_id":"S018","kind":"supports"},{"source_id":"S028","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S020","kind":"supports"},{"source_id":"S024","target_id":"S032","kind":"supports"},{"source_id":"S017","target_id":"S032","kind":"supports"},{"source_id":"S008","target_id":"S032","kind":"supports"},{"source_id":"S005","target_id":"S032","kind":"supports"},{"source_id":"S014","target_id":"S032","kind":"supports"},{"source_id":"S028","target_id":"S032","kind":"supports"},{"source_id":"S004","target_id":"S033","kind":"supports"},{"source_id":"S008","target_id":"S033","kind":"supports"},{"source_id":"S003","target_id":"S033","kind":"supports"},{"source_id":"S007","target_id":"S033","kind":"supports"},{"source_id":"S028","target_id":"S033","kind":"supports"},{"source_id":"S009","target_id":"S033","kind":"supports"},{"source_id":"S005","target_id":"S033","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S023","target_id":"S039","kind":"supports"},{"source_id":"S030","target_id":"S039","kind":"supports"},{"source_id":"S033","target_id":"S039","kind":"supports"},{"source_id":"S004","target_id":"S047","kind":"supports"},{"source_id":"S022","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S022","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S003","target_id":"S046","kind":"supports"},{"source_id":"S009","target_id":"S046","kind":"supports"},{"source_id":"S005","target_id":"S046","kind":"supports"},{"source_id":"S028","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S051","kind":"supports"},{"source_id":"S031","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S048","target_id":"S052","kind":"supports"},{"source_id":"S047","target_id":"S052","kind":"supports"},{"source_id":"S009","target_id":"S052","kind":"supports"},{"source_id":"S005","target_id":"S052","kind":"supports"},{"source_id":"S028","target_id":"S052","kind":"supports"},{"source_id":"S039","target_id":"S046","kind":"supports"},{"source_id":"S024","target_id":"S046","kind":"supports"},{"source_id":"S017","target_id":"S046","kind":"supports"},{"source_id":"S008","target_id":"S046","kind":"supports"},{"source_id":"S020","target_id":"S046","kind":"supports"},{"source_id":"S051","target_id":"S046","kind":"supports"},{"source_id":"S052","target_id":"S046","kind":"supports"},{"source_id":"S004","target_id":"S055","kind":"supports"},{"source_id":"S011","target_id":"S055","kind":"supports"},{"source_id":"S012","target_id":"S055","kind":"supports"},{"source_id":"S022","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S011","target_id":"S056","kind":"supports"},{"source_id":"S022","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S058","kind":"supports"},{"source_id":"S011","target_id":"S058","kind":"supports"},{"source_id":"S012","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S055","target_id":"S059","kind":"supports"},{"source_id":"S039","target_id":"S059","kind":"supports"},{"source_id":"S024","target_id":"S059","kind":"supports"},{"source_id":"S017","target_id":"S059","kind":"supports"},{"source_id":"S008","target_id":"S059","kind":"supports"},{"source_id":"S029","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S060","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S011","target_id":"S060","kind":"supports"},{"source_id":"S014","target_id":"S060","kind":"supports"},{"source_id":"S029","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S055","target_id":"S061","kind":"supports"},{"source_id":"S056","target_id":"S061","kind":"supports"},{"source_id":"S009","target_id":"S061","kind":"supports"},{"source_id":"S005","target_id":"S061","kind":"supports"},{"source_id":"S028","target_id":"S061","kind":"supports"},{"source_id":"S059","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S056","target_id":"S062","kind":"supports"},{"source_id":"S051","target_id":"S062","kind":"supports"},{"source_id":"S052","target_id":"S062","kind":"supports"},{"source_id":"S055","target_id":"S054","kind":"supports"},{"source_id":"S056","target_id":"S054","kind":"supports"},{"source_id":"S024","target_id":"S054","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S059","target_id":"S057","kind":"supports"},{"source_id":"S060","target_id":"S057","kind":"supports"},{"source_id":"S061","target_id":"S057","kind":"supports"},{"source_id":"S062","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S054","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"欧几里得若尔当代数","source_units":[{"raw_tex":"Let $\\mathbb{E}$ be a real-valued vector space equipped with an inner product $\\langle \\cdot, \\cdot \\rangle$ and a bilinear operation $\\circ$ : $\\mathbb{E} \\times \\mathbb{E} \\rightarrow \\mathbb{E}$,\r\nand $e$ be the identity element, i.e.,$x \\circ e = e \\circ x = x$ holds for any $ x \\in \\mathbb{E}$.\r\n$(\\mathbb{E}, \\circ)$ is called a Euclidean Jordan algebra if it satisfies\r\n\\begin{align*}\r\nx \\circ y = y  \\circ x,  \\ \\\r\nx \\circ (x^2 \\circ y) = x^2 \\circ (x \\circ y), \\ \\\r\n\\langle x \\circ y , z \\rangle = \\langle y , x \\circ z \\rangle\r\n\\end{align*}\r\nfor all $x,y,z \\in \\mathbb{E}$ and $x^2 := x \\circ x$.\r\n","start":19433,"end":20042}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"逆元记号","source_units":[{"raw_tex":"We denote $y \\in \\mathbb{E}$ as $x^{-1}$ if  $y$ satisfies $x \\circ y = e$.\r\n","start":20042,"end":20119}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"本原幂等元与若尔当标架","source_units":[{"raw_tex":"$c \\in \\mathbb{E}$ is called an {\\em idempotent} if it satisfies $c \\circ c = c$, and an idempotent $c$ is called {\\em primitive} if it can not be written as a sum of two or more nonzero idempotents. \r\nA set of primitive idempotents $c_1, c_2, \\ldots c_k$ is called a {\\em Jordan frame} if $c_1, \\ldots c_k$ satisfy\r\n\\begin{equation}\r\n\\notag\r\nc_i \\circ c_j = 0 \\ (i \\neq j), \\ \\\r\nc_i \\circ c_i = c_i \\ (i= 1 , \\dots , k), \\ \\\r\n\\sum_{i=1}^k c_i = e.\r\n\\end{equation}\r\n","start":20119,"end":20585}]},{"id":"S004","semantic_type":"proposition","kind":"proposition","labels":["prop:spectral"],"member_ids":["S004"],"declared_external":true,"declaration_text":["原始前提清单许可 B0014；原文引用 Faraut and Korányi (1994), Theorem III.1.2。"],"source_only":false,"title":"若尔当谱定理","source_units":[{"raw_tex":"\\begin{proposition}[Spectral theorem (cf. Theorem III.1.2 of \\cite{Faraut1994})]\r\n\\label{prop:spectral}\r\nLet $(\\mathbb{E}, \\circ)$ be a Euclidean Jordan algebra having rank $r$. \r\nFor any $x \\in \\mathbb{E} $, there exist real numbers $\\lambda_1 , \\dots , \\lambda_r$ and a Jordan frame $c_1 , \\dots , c_r $ for which the following holds:\r\n\\begin{equation}\r\nx = \\sum_{i=1}^r \\lambda_i c_i \\notag.\r\n\\end{equation}\r\nThe numbers $\\lambda_1 , \\dots , \\lambda_r$ are uniquely determined {\\em eigenvalues} of $x$ (with their multiplicities).\r\n","start":20879,"end":21414},{"raw_tex":"\\end{proposition}\r\n","start":21510,"end":21529}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":["prop:spectral"],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"谱迹","source_units":[{"raw_tex":"Furthermore, ${\\rm trace}(x) := \\sum_{i=1}^r \\lambda_i$, $\\det(x) := \\prod_{i=1}^r \\lambda_i$.\r\n","start":21414,"end":21510}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["prop:spectral"],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"谱行列式","source_units":[{"raw_tex":"Furthermore, ${\\rm trace}(x) := \\sum_{i=1}^r \\lambda_i$, $\\det(x) := \\prod_{i=1}^r \\lambda_i$.\r\n","start":21414,"end":21510}]},{"id":"S007","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S007"],"declared_external":true,"declaration_text":["原始前提清单中的标准自对偶性；原文同时引用 Faraut and Korányi (1994), Theorems III.2.1、III.3.1，并交代后续谱性质的文献来源。"],"source_only":false,"title":"对称锥与自对偶性","source_units":[{"raw_tex":"A proper cone is symmetric if it is self-dual and homogeneous. \r\nIt is known that the set of squares $\\mathcal{K} = \\{ x^2 : x \\in \\mathbb{E} \\}$ is the symmetric cone of  $\\mathbb{E}$  (cf. Theorems III.2.1 and III.3.1 of \\cite{Faraut1994}).\r\nThe following properties can be derived from the results in \\cite{Faraut1994}, as in  Corollary 2.3 of \\cite{Yoshise2007}:\r\n","start":22073,"end":22441}]},{"id":"S008","semantic_type":"proposition","kind":"proposition","labels":["prop:lambda-Jordan"],"member_ids":["S008"],"declared_external":true,"declaration_text":["原始前提清单许可 B0017；原文在第284行说明可由 Faraut1994 的结果推出，并参照 Yoshise2007, Corollary 2.3。"],"source_only":false,"title":"锥归属的谱刻画","source_units":[{"raw_tex":"\\begin{proposition}\r\n\\label{prop:lambda-Jordan}\r\nLet $x \\in \\mathbb{E}$ and let $\\sum_{j=1}^r \\lambda_j c_j$ be a decomposition of $x$ given by Propositoin \\ref{prop:spectral}. Then\r\n\\begin{description}\r\n\\item[(i)]\r\n$x \\in \\mathcal{K}$ if and only if  $\\lambda_j \\geq 0 \\ (j=1,2,\\ldots,r)$, \r\n\\item[(ii)]\r\n$x \\in {\\rm int} \\hspace{0.75mm} \\mathcal{K}$ if and only if $\\lambda_j > 0 \\ (j=1,2,\\ldots,r)$.\r\n\\end{description}\r\n\\end{proposition}\r\n","start":22441,"end":22883}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"锥投影的谱截断","source_units":[{"raw_tex":"\r\nFrom Propositions \\ref{prop:spectral} and \\ref{prop:lambda-Jordan} for any $x \\in \\mathbb{E}$, its projection $P_\\mathcal{K} (x)$ onto $\\mathcal{K}$ can be written as an operation to round all negative eigenvalues of $x$ to $0$, i.e., \r\n$P_\\mathcal{K} (x) = \\sum_{i=1}^r [\\lambda_i]^+ c_i$, where $[\\cdot]^+$ denotes the projection onto the nonnegative orthant. \r\n","start":22883,"end":23249}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"简单代数与简单锥","source_units":[{"raw_tex":"A Euclidean Jordan algebra $(\\mathbb{E} , \\circ)$ is called {\\em simple} if it cannot be written as any Cartesian product of non-zero Euclidean Jordan algebras.\r\nIf the Euclidean Jordan algebra $(\\mathbb{E} , \\circ)$ associated with a symmetric cone $\\mathcal{K}$ is simple, then we say that $\\mathcal{K}$ is {\\em simple}.\r\n","start":24057,"end":24381}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","labels":["eq:r and e"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"乘积锥与块秩","source_units":[{"raw_tex":"In this paper, we will consider that $\\mathcal{K}$ is given by a Cartesian product of $p$ simple symmetric cones $\\mathcal{K}_\\ell$,\r\n$\\mathcal{K} := \\mathcal{K}_1 \\times \\dots \\times \\mathcal{K}_p$, whose rank and identity element are $r_\\ell$ and $e_\\ell$ $(\\ell=1, \\ldots, p)$.\r\nThe rank $r$ and the identity element of  $\\mathcal{K}$ are given by\r\n\\begin{equation}\r\n\\label{eq:r and e}\r\nr = \\sum_{\\ell=1}^p r_\\ell, \\ \\ e = (e_1 , \\dots , e_p).\r\n\\end{equation}\r\n","start":24381,"end":24845}]},{"id":"S012","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"块元素记号","source_units":[{"raw_tex":"In what follows, $x_\\ell$ stands for the $\\ell$-th block element of  $x \\in \\mathcal{K}$, i.e., $x = (x_1, \\dots, x_p) \\in \\mathcal{K}_1 \\times \\dots \\times \\mathcal{K}_p$.\r\n","start":24845,"end":25019}]},{"id":"S013","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"最小特征值记号","source_units":[{"raw_tex":"For each $\\ell=1, \\cdots, p$, we define $\\lambda_{\\min}(x_\\ell) := \\min\\{ \\lambda_1, \\cdots, \\lambda_{r_\\ell} \\}$ where $\\lambda_1, \\cdots,  \\lambda_{r_\\ell}$ are eigenvalues of $x_\\ell$.\r\nThe minimum eigenvalue $\\lambda_{\\min}(x)$ of $x \\in \\mathcal{K}$ is given by $\\lambda_{\\min}(x) = \\mbox{min} \\{ \\lambda_{\\min}(x_1), \\cdots, \\lambda_{\\min}(x_p) \\}$.\r\n","start":25019,"end":25376}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"迹内积","source_units":[{"raw_tex":"This subsection summarizes the \\modifyFirst{notations} used in this paper.\r\nFor any $x,y \\in \\mathbb{E}$, we define the inner product  $\\langle \\cdot,\\cdot \\rangle$ and the norm $\\|\\cdot\\|_{J}$ as $\\langle x , y \\rangle := {\\rm trace}(x \\circ y)$ and $\\| x \\|_J := \\sqrt{ \\langle x , x \\rangle }$, respectively.\r\n","start":31084,"end":31397}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"若尔当范数","source_units":[{"raw_tex":"For any $x,y \\in \\mathbb{E}$, we define the inner product  $\\langle \\cdot,\\cdot \\rangle$ and the norm $\\|\\cdot\\|_{J}$ as $\\langle x , y \\rangle := {\\rm trace}(x \\circ y)$ and $\\| x \\|_J := \\sqrt{ \\langle x , x \\rangle }$, respectively.\r\n","start":31160,"end":31397}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"谱一范数","source_units":[{"raw_tex":"For any $x \\in \\mathbb{E}$ having decomposition  $x = \\sum_{i=1}^r \\lambda_i c_i$ as in  Proposition \\ref{prop:spectral}, we also define $\\| x \\|_1 := |\\lambda_1| + \\dots + |\\lambda_r|$, $\\|x\\|_\\infty := \\max \\{|\\lambda_1| , \\dots , |\\lambda_r| \\}$.\r\n","start":31397,"end":31648}]},{"id":"S017","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"谱无穷范数","source_units":[{"raw_tex":"For any $x \\in \\mathbb{E}$ having decomposition  $x = \\sum_{i=1}^r \\lambda_i c_i$ as in  Proposition \\ref{prop:spectral}, we also define $\\| x \\|_1 := |\\lambda_1| + \\dots + |\\lambda_r|$, $\\|x\\|_\\infty := \\max \\{|\\lambda_1| , \\dots , |\\lambda_r| \\}$.\r\n","start":31397,"end":31648}]},{"id":"S018","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"锥内范数的等价表示","source_units":[{"raw_tex":"For $x \\in \\mathcal{K}$, we obtain the following equivalent representations: $\\|x\\|_1 = \\langle e , x \\rangle$, $\\|x\\|_\\infty = \\lambda_{\\max} (x)$.\r\n","start":31648,"end":31798}]},{"id":"S019","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"空间维数","source_units":[{"raw_tex":"The following is a list of other definitions and frequently used symbols in the paper.\r\n\\begin{itemize}\r\n\\item $d$: the dimension of the Euclidean space $\\mathbb{E}$ corresponding to $\\mathcal{K}$,\r\n","start":31798,"end":31997}]},{"id":"S020","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"归一化可行域记号","source_units":[{"raw_tex":"\\item $F_{{\\rm P}_{S_{\\infty}}(\\mathcal{A})}$: the feasible region of ${\\rm P}_{S_{\\infty}}(\\mathcal{A})$,\r\n","start":31997,"end":32105}]},{"id":"S021","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"核空间投影记号","source_units":[{"raw_tex":"\\item $P_{\\mathcal{A}} (\\cdot) $: the projection map onto $\\mbox{ker} \\hspace{0.3mm} \\mathcal{A}$,\r\n%\\item $Q_{\\mathcal{A}} (\\cdot) $: the projection map onto $\\mbox{range} \\hspace{0.3mm} \\mathcal{A}^*$, \r\n","start":32105,"end":32311}]},{"id":"S022","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"投影与分块谱记号","source_units":[{"raw_tex":"\\item $\\mathcal{P}_\\mathcal{K} (\\cdot) $: the projection map onto $\\mathcal{K}$,\r\n\\item $\\lambda(x) \\in \\mathbb{R}^r$: an $r$-dimensional vector composed of the eigenvalues of $ x \\in \\mathcal{K}$,\r\n\\item $\\lambda(x_\\ell) \\in \\mathbb{R}^{r_\\ell}$: an $r_\\ell$-dimensional vector composed of the eigenvalues of $x_\\ell \\in \\mathcal{K}_\\ell$ ($\\ell =1,\\ldots,p$),\r\n\\item $c(x_\\ell)_i \\in \\mathcal{K}_\\ell$: the $i$-th primitive idempotent of $x_\\ell \\in \\mathbb{E}_\\ell$. When $\\mathcal{K}$ is simple, it is abbreviated as $c_i$.\r\n\\item $\\left[ \\cdot \\right]^+$: the projection map onto the nonnegative orthant, and\r\n","start":32311,"end":32926}]},{"id":"S023","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"线性算子的伴随","source_units":[{"raw_tex":"\\item $\\mathcal{A}^*(\\cdot)$: the adjoint operator of the linear operator $\\mathcal{A}(\\cdot)$, i.e., $\\langle \\mathcal{A}(x) , y \\rangle = \\left \\langle x , \\mathcal{A}^* (y) \\right\\rangle$ for all $x \\in \\mathcal{K}$ and $y \\in \\mathbb{R}^m$.\r\n\\end{itemize}\r\n","start":32926,"end":33187}]},{"id":"S024","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"归一化锥可行问题","source_units":[{"raw_tex":"We focus on the feasibility of the following problem ${\\rm P}_{S_{\\infty}}(\\mathcal{A})$, which is equivalent to  ${\\rm P}(\\mathcal{A})$: \r\n\\begin{equation}\r\n\\begin{array}{lllllllll}\r\n{\\rm P}_{S_{\\infty}}(\\mathcal{A})&\t&\\mbox{find}   &x \\in \\mbox{int} \\hspace{0.75mm} \\mathcal{K}\t&\t&\\mbox{s.t.}\t&\\mathcal{A} (x) = \\bm{0},\t&\\|x\\|_\\infty \\leq 1.\r\n\\end{array}\r\n\\notag\r\n\\end{equation}\r\n","start":33772,"end":34154}]},{"id":"S025","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"替代可行问题","source_units":[{"raw_tex":"The alternative problem ${\\rm D}(\\mathcal{A})$ of ${\\rm P}(\\mathcal{A})$ is \r\n\\begin{equation}\r\n\\notag\r\n\\begin{array}{llllllllll}\r\n{\\rm D}(\\mathcal{A})\t&\t&\\mbox{find}   &y \\in \\mathcal{K}\t&\t&\\mbox{s.t.}\t&y \\in \\mbox{range} \\hspace{0.3mm} \\mathcal{A}^*,\t&y \\neq \\bm{0},\r\n\\end{array}\r\n\\end{equation}\r\n","start":34154,"end":34453}]},{"id":"S026","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"伴随像与核的正交关系","source_units":[{"raw_tex":"where  $\\mbox{range} \\hspace{0.3mm} \\mathcal{A}^*$ is the orthogonal complement of $\\mbox{ker} \\hspace{0.3mm} \\mathcal{A}$.\r\n","start":34453,"end":34578}]},{"id":"S027","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"重申乘积锥假设","source_units":[{"raw_tex":"As we mentioned in section \\ref{subsec:symmetric cone}, \r\nwe assume that $\\mathcal{K}$ is given by a Cartesian product of $p$ simple symmetric cones $\\mathcal{K}_\\ell (\\ell=1, \\ldots, p)$, i.e., $\\mathcal{K} = \\mathcal{K}_1 \\times \\dots \\times \\mathcal{K}_p $.\r\n","start":34578,"end":34840}]},{"id":"S028","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S028"],"declared_external":true,"declaration_text":["原始前提清单中的 Standard self-duality and trace-orthonormality of Jordan frames, including their blockwise product-cone versions。"],"source_only":false,"title":"标架的迹正交归一性","source_units":[]},{"id":"S029","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S029"],"declared_external":true,"declaration_text":["原始前提清单中的 blockwise product-cone versions；逐块正交投影是 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"乘积空间的分块运算","source_units":[]},{"id":"S030","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S030"],"declared_external":true,"declaration_text":["原始前提清单中的 Standard Lagrangian weak duality。"],"source_only":false,"title":"拉格朗日弱对偶","source_units":[]},{"id":"S031","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S031"],"declared_external":true,"declaration_text":["原始前提清单中的 elementary minimization of convex piecewise-linear functions。"],"source_only":false,"title":"分段线性函数的最小化","source_units":[]},{"id":"S032","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S032"],"declared_external":false,"source_only":true,"title":"简单锥的初始迹上界","source_units":[{"raw_tex":"Let us consider the case where $\\mathcal {K}$ is simple. It is obvious that, for any feasible solution $x$ of ${\\rm P}_{S_{\\infty}}(\\mathcal{A})$, the constraint $\\|x\\|_\\infty \\leq 1$ implies that $\\langle e , x\\rangle \\leq r$, since $x \\in \\mathcal{K}$. In Proposition \\ref{prop:upper-bound}, we show that this bound may be improved as $\\langle e , x\\rangle <r$ by using a point $v \\in \\mbox{range} \\hspace{0.3mm} \\mathcal{A}^*\\setminus \\{0\\}$. To prove Proposition \\ref{prop:upper-bound}, we need the following Lemma \\ref{lemma:0} and Proposition \\ref{prop:p-d-relation}.\r\n","start":38221,"end":38796}]},{"id":"S033","semantic_type":"lemma","kind":"lemma","labels":["lemma:0"],"member_ids":["S033","S034","S035","S036","S037","S038"],"declared_external":false,"source_only":false,"title":"谱盒的支撑函数","source_units":[{"raw_tex":"\\begin{lemma}\r\n\\label{lemma:0}\r\nLet $(\\mathbb{E}, \\circ)$ be a Euclidean Jordan algebra with the \\modifySecond{associated} symmetric cone $\\mathcal{K}$.\r\nFor any $y \\in \\mathbb{E}$, the following equation holds:\r\n\\begin{equation}\r\n\\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\langle y ,x \\rangle  =  \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( y \\right)  ,e \\right \\rangle . \\notag\r\n\\end{equation}\r\n\\end{lemma}","start":38796,"end":39205},{"raw_tex":"\\begin{proof}\r\nUsing the decomposition $y = \\sum_{i=1}^r \\lambda_i c_i$ obtained by Proposition \\ref{prop:spectral}, we see that \r\n\\begin{equation}\r\n\\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\langle y ,x \\rangle  = \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\left \\langle \\sum_{i=1}^r \\lambda_i c_i , x \\right \\rangle = \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}}  \\sum_{i=1}^r \\lambda_i  \\left \\langle c_i , x \\right \\rangle . \\label{lemma:0-eq1}\r\n\\end{equation}\r\nNoting that $x \\in \\mathcal{K}, e-x \\in \\mathcal{K}$ from $\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}$, since $c_i \\in \\mathcal{K}$ is primitive idempotent, we find that \r\n$\\langle c_i,x \\rangle \\geq 0$ and $\\langle c_i, e-x \\rangle \\geq 0$, which implies that $0 \\leq \\langle c_i,x \\rangle \\leq 1$.\r\nThus, letting $I_1$ be the set of indices for which $\\lambda_i \\leq 0$ and $I_2$ be the set of indices for which $\\lambda_i > 0$, if there exists an $x$ satisfying \r\n\\begin{equation}\r\n\\langle c_i,x \\rangle =\r\n\\begin{cases}\r\n0 & i \\in I_1 \\\\\r\n1 & i \\in I_2\r\n\\end{cases}, \r\n\\label{lemma:0-eq2}\r\n\\end{equation}\r\nthen such an $x$ is an optimal solution of (\\ref{lemma:0-eq1}). In fact, if we define $x^* = \\sum_{i \\in I_2} c_i$, then by the dedfinition of the Jordan frame, $x^*$ satisfies (\\ref{lemma:0-eq2}) and $\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}$ and becomes an optimal solution of (\\ref{lemma:0-eq1}). In this case, the optimal value of (\\ref{lemma:0-eq1}) turns out to be\r\n\\begin{equation}\r\n\\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}}  \\sum_{i=1}^r \\lambda_i  \\left \\langle c_i , x \\right \\rangle = \\sum_{i=1}^r \\lambda_i  \\left \\langle c_i , x^* \\right \\rangle \r\n= \\sum_{i \\in I_2} \\lambda_i \r\n= \\sum_{i=1}^r [\\lambda_i]^+\r\n= \\langle \\mathcal{P}_\\mathcal{K} (y) , e \\rangle. \r\n\\notag\r\n\\end{equation}\r\n\\end{proof}","start":39207,"end":40975},{"raw_tex":"Using the decomposition $y = \\sum_{i=1}^r \\lambda_i c_i$ obtained by Proposition \\ref{prop:spectral}, we see that \r\n\\begin{equation}\r\n\\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\langle y ,x \\rangle  = \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\left \\langle \\sum_{i=1}^r \\lambda_i c_i , x \\right \\rangle = \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}}  \\sum_{i=1}^r \\lambda_i  \\left \\langle c_i , x \\right \\rangle . \\label{lemma:0-eq1}\r\n\\end{equation}\r\n","start":39222,"end":39669},{"raw_tex":"Noting that $x \\in \\mathcal{K}, e-x \\in \\mathcal{K}$ from $\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}$, since $c_i \\in \\mathcal{K}$ is primitive idempotent, we find that \r\n$\\langle c_i,x \\rangle \\geq 0$ and $\\langle c_i, e-x \\rangle \\geq 0$, which implies that $0 \\leq \\langle c_i,x \\rangle \\leq 1$.\r\n","start":39669,"end":39962},{"raw_tex":"Thus, letting $I_1$ be the set of indices for which $\\lambda_i \\leq 0$ and $I_2$ be the set of indices for which $\\lambda_i > 0$, if there exists an $x$ satisfying \r\n\\begin{equation}\r\n\\langle c_i,x \\rangle =\r\n\\begin{cases}\r\n0 & i \\in I_1 \\\\\r\n1 & i \\in I_2\r\n\\end{cases}, \r\n\\label{lemma:0-eq2}\r\n\\end{equation}\r\nthen such an $x$ is an optimal solution of (\\ref{lemma:0-eq1}). In fact, if we define $x^* = \\sum_{i \\in I_2} c_i$, then by the dedfinition of the Jordan frame, $x^*$ satisfies (\\ref{lemma:0-eq2}) and $\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}$ and becomes an optimal solution of (\\ref{lemma:0-eq1}). In this case, the optimal value of (\\ref{lemma:0-eq1}) turns out to be\r\n","start":39962,"end":40637},{"raw_tex":"\\begin{equation}\r\n\\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}}  \\sum_{i=1}^r \\lambda_i  \\left \\langle c_i , x \\right \\rangle = \\sum_{i=1}^r \\lambda_i  \\left \\langle c_i , x^* \\right \\rangle \r\n= \\sum_{i \\in I_2} \\lambda_i \r\n= \\sum_{i=1}^r [\\lambda_i]^+\r\n= \\langle \\mathcal{P}_\\mathcal{K} (y) , e \\rangle. \r\n\\notag\r\n\\end{equation}\r\n","start":40637,"end":40964}]},{"id":"S039","semantic_type":"proposition","kind":"proposition","labels":["prop:p-d-relation"],"member_ids":["S039","S041","S042","S043","S044"],"declared_external":false,"source_only":false,"title":"谱盒约束的拉格朗日对偶","source_units":[{"raw_tex":"\\begin{proposition}\r\n\\label{prop:p-d-relation} \r\nLet $(\\mathbb{E}, \\circ)$ be a Euclidean Jordan Algebra with the corresponding symmetric cone  $\\mathcal{K}$.\r\nFor a given $c \\in \\mathbb{E}$, consider the problem\r\n\\begin{equation}\r\n\\notag\r\n\\begin{array}{llllll}\r\n\\mbox{\\rm max}\t&\\langle c , x \\rangle \t&\t&\\mbox{\\rm s.t}\t&\\mathcal{A}(x) = \\bm{0},\t&\\bm{0} \\leq \\lambda (x) \\leq \\bm{1}  \\notag.\r\n\\end{array}\r\n\\end{equation}\r\nThe dual problem of the above is \r\n\\begin{equation}\r\n\\notag\r\n\\begin{array}{lllll}\r\n\\mbox{\\rm min}\t&\\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c - u  \\right) , e \\right \\rangle\t&\t&\\mbox{\\rm s.t}\t&u \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*. \\notag\r\n\\end{array}\r\n\\end{equation}\r\n\\end{proposition}","start":40979,"end":41705},{"raw_tex":"\\begin{proof}\r\nDefine the Lagrangian function $L(x,w)$ as $L(x,w) :=   \\langle c , x \\rangle -  w^\\top \\mathcal{A}(x)$ where $w \\in \\mathbb{R}^m$ is the Lagrange multiplier. \r\n\\modifyThird{\r\nSupoose that $x^*$ is an optimal sotution of the primal problem. \r\nThen, for any $w \\in \\mathbb{R}^m$,  we have $\\langle c,x^* \\rangle = L(w,x^*) \\leq \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} L(w,x)$, and hence,}\r\n\\begin{align*}\r\n\\modifyThird{\\langle c, x^* \\rangle} \\leq \\min_w \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} L(x,w)\r\n&=  \\min_w \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\{  \\langle c , x \\rangle - \\langle \\mathcal{A}^*(w) ,x \\rangle \\}  \\\\\r\n&=  \\min_w \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\{  \\langle c -  \\mathcal{A}^*(w) ,x \\rangle \\}  \\\\\r\n&=  \\min_w   \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c -  \\mathcal{A}^*(w) \\right)  ,e \\right \\rangle \\ \\ \\ \\mbox{(by  lemma \\ref{lemma:0})}  \\\\\r\n&= \\min_{u \\in \\mbox{\\footnotesize range} \\mathcal{A}^*}  \\langle \\mathcal{P}_\\mathcal{K} \\left( c - u \\right) , e \\rangle.\r\n\\end{align*}\r\n%and the dual problem is \r\n%\\begin{equation}\r\n%\\notag\r\n%\\begin{array}{lllll}\r\n%\\mbox{\\rm min}\t&\\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c - u  \\right) , e \\right \\rangle\t&\t&\\mbox{\\rm s.t}\t&u \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*. \\notag\r\n%\\end{array}\r\n%\\end{equation}\r\n\\end{proof}","start":41707,"end":43045},{"raw_tex":"Define the Lagrangian function $L(x,w)$ as $L(x,w) :=   \\langle c , x \\rangle -  w^\\top \\mathcal{A}(x)$ where $w \\in \\mathbb{R}^m$ is the Lagrange multiplier. \r\n\\modifyThird{\r\n","start":41722,"end":41898},{"raw_tex":"Supoose that $x^*$ is an optimal sotution of the primal problem. \r\nThen, for any $w \\in \\mathbb{R}^m$,  we have $\\langle c,x^* \\rangle = L(w,x^*) \\leq \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} L(w,x)$, and hence,}\r\n\\begin{align*}\r\n\\modifyThird{\\langle c, x^* \\rangle} \\leq \\min_w \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} L(x,w)\r\n&=  \\min_w \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\{  \\langle c , x \\rangle - \\langle \\mathcal{A}^*(w) ,x \\rangle \\}  \\\\\r\n&=  \\min_w \\max_{\\bm{0} \\leq \\lambda(x) \\leq \\bm{1}} \\{  \\langle c -  \\mathcal{A}^*(w) ,x \\rangle \\}  \\\\\r\n","start":41898,"end":42459},{"raw_tex":"&=  \\min_w   \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c -  \\mathcal{A}^*(w) \\right)  ,e \\right \\rangle \\ \\ \\ \\mbox{(by  lemma \\ref{lemma:0})}  \\\\\r\n&= \\min_{u \\in \\mbox{\\footnotesize range} \\mathcal{A}^*}  \\langle \\mathcal{P}_\\mathcal{K} \\left( c - u \\right) , e \\rangle.\r\n\\end{align*}\r\n","start":42459,"end":42749}]},{"id":"S040","semantic_type":"problem_setting","kind":"problem_setting","labels":["prop:p-d-relation"],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"闭谱盒上的线性优化","source_units":[{"raw_tex":"Let $(\\mathbb{E}, \\circ)$ be a Euclidean Jordan Algebra with the corresponding symmetric cone  $\\mathcal{K}$.\r\nFor a given $c \\in \\mathbb{E}$, consider the problem\r\n\\begin{equation}\r\n\\notag\r\n\\begin{array}{llllll}\r\n\\mbox{\\rm max}\t&\\langle c , x \\rangle \t&\t&\\mbox{\\rm s.t}\t&\\mathcal{A}(x) = \\bm{0},\t&\\bm{0} \\leq \\lambda (x) \\leq \\bm{1}  \\notag.\r\n\\end{array}\r\n\\end{equation}\r\n","start":41028,"end":41401}]},{"id":"S045","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"已注释的对偶重述","source_units":[{"raw_tex":"%and the dual problem is \r\n%\\begin{equation}\r\n%\\notag\r\n%\\begin{array}{lllll}\r\n%\\mbox{\\rm min}\t&\\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c - u  \\right) , e \\right \\rangle\t&\t&\\mbox{\\rm s.t}\t&u \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*. \\notag\r\n%\\end{array}\r\n%\\end{equation}\r\n","start":42749,"end":43034}]},{"id":"S046","semantic_type":"proposition","kind":"proposition","labels":["eq:bound_q","prop:upper-bound"],"member_ids":["S046","S049","S050","S053"],"declared_external":false,"source_only":false,"title":"简单锥的谱坐标割界","source_units":[{"raw_tex":"\\begin{proposition}\r\n\\label{prop:upper-bound}\r\n\r\nSuppose that $v \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*$ is given by $v = \\sum_{i=1}^r \\lambda_i c_i$ as in Proposition \\ref{prop:spectral}. \r\nFor each $i \\in \\{ 1, \\dots , r \\}$ and $\\alpha \\in \\mathbb{R}$, define $q_i(\\alpha) := \\left[    1-\\alpha \\lambda_i  \\right]^+ + \\sum_{j \\neq i }^r \\left[ -  \\alpha \\lambda_j  \\right]^+$.\r\nThen, the following relations hold for any $x \\in F_{{\\rm P}_{S_{\\infty}}(\\mathcal{A})}$ and $i \\in \\{ 1, \\dots , r \\}$\\modifySecond{:}\r\n\\begin{equation}\r\n\\label{eq:bound_q}\r\n\\langle c_i , x \\rangle \\leq  \r\n\\min_{\\alpha \\in \\mathbb{R}} q_i(\\alpha) = \r\n\\begin{cases}\r\n\\min \\left\\{  1 ,  \\left \\langle  e , \\mathcal{P}_{\\mathcal{K}} \\left( -  \\frac{1}{\\lambda_i} v  \\right) \\right \\rangle \\right\\} & \\mbox{if $\\lambda_i \\neq 0$}, \\\\\r\n1 & \\mbox{if $\\lambda_i = 0$}\\modifySecond{.}\r\n\\end{cases}\r\n\\end{equation}\r\n\\end{proposition}","start":43141,"end":44058},{"raw_tex":"\\begin{proof}\r\n\r\nFor each $i \\in \\{ 1, 2, \\dots , r \\}$, we have \r\n\\begin{equation}\r\n\\mathcal{P}_\\mathcal{K} \\left( c_i -  \\alpha v \\right) = \\mathcal{P}_\\mathcal{K} \\left( c_i - \\alpha \\sum_{j=1}^r \\lambda_j c_j \\right) = \\mathcal{P}_\\mathcal{K} \\left( (1-\\alpha \\lambda_i ) c_i -   \\sum_{j \\neq i }^r \\alpha \\lambda_j c_j  \\right), \\notag\r\n\\end{equation}\r\nand hence, \r\n\\begin{equation}\r\n\\label{eq:q_i(alpha)}\r\n\\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c_i -  \\alpha v \\right) , e \\right \\rangle\r\n=\r\n\\left \\langle \\mathcal{P}_\\mathcal{K} \\left(    (1-\\alpha \\lambda_i ) c_i -   \\sum_{j \\neq i }^r \\alpha \\lambda_j c_j   \\right) , \\sum_{i=1}^r c_i \\right \\rangle \r\n= \\left[    1-\\alpha \\lambda_i  \\right]^+ + \\sum_{j \\neq i }^r \\left[ -  \\alpha \\lambda_j  \\right]^+  = q_i(\\alpha) .\r\n\\end{equation}\r\n\r\nNote that, since $q_i(\\alpha)$ is a piece-wise linear convex function, if $\\lambda_i = 0$, it attains the minimum at $\\alpha = 0$ with $q_i(0) = 1$, and if $\\lambda_i \\neq 0$, it attains the minimum at $\\alpha = 0$ with $q_i(0) = 1$ or at $\\alpha =  \\frac{1}{\\lambda_i}$ with\r\n\\begin{equation}\r\n\\notag\r\nq \\left( \\frac{1}{\\lambda_i} \\right) = \\sum_{j \\neq i}^r \\left[ - \\frac{\\lambda_j}{\\lambda_i}  \\right]^+ \r\n= \\sum_{j=1}^r \\left[ - \\frac{\\lambda_j}{\\lambda_i}  \\right]^+ \r\n= \\left \\langle e , \\mathcal{P}_{\\mathcal{K}} \\left( -  \\frac{1}{\\lambda_i} v  \\right) \\right \\rangle.\r\n\\end{equation}\r\nThus, we obtain equivalence in (\\ref{eq:bound_q}).\r\nSince $\\alpha v \\in \\mbox{range} \\hspace{0.3mm} \\mathcal{A}^*$ for all $\\alpha \\in \\mathbb{R}$, \r\nfor each $i \\in \\{ 1, \\dots , r \\}$, Proposition \\ref{prop:p-d-relation} and  (\\ref{eq:q_i(alpha)})  ensure that \r\n$\\langle c_i , x \\rangle \\leq  \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c_i -  \\alpha v \\right) , e \\right \\rangle = q_i(\\alpha)$\r\nfor all $\\alpha \\in \\mathbb{R}$, which implies the inequality in (\\ref{eq:bound_q}).\r\n\\end{proof}","start":44064,"end":45961},{"raw_tex":"For each $i \\in \\{ 1, 2, \\dots , r \\}$, we have \r\n\\begin{equation}\r\n\\mathcal{P}_\\mathcal{K} \\left( c_i -  \\alpha v \\right) = \\mathcal{P}_\\mathcal{K} \\left( c_i - \\alpha \\sum_{j=1}^r \\lambda_j c_j \\right) = \\mathcal{P}_\\mathcal{K} \\left( (1-\\alpha \\lambda_i ) c_i -   \\sum_{j \\neq i }^r \\alpha \\lambda_j c_j  \\right), \\notag\r\n\\end{equation}\r\nand hence, \r\n\\begin{equation}\r\n\\label{eq:q_i(alpha)}\r\n\\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c_i -  \\alpha v \\right) , e \\right \\rangle\r\n=\r\n\\left \\langle \\mathcal{P}_\\mathcal{K} \\left(    (1-\\alpha \\lambda_i ) c_i -   \\sum_{j \\neq i }^r \\alpha \\lambda_j c_j   \\right) , \\sum_{i=1}^r c_i \\right \\rangle \r\n= \\left[    1-\\alpha \\lambda_i  \\right]^+ + \\sum_{j \\neq i }^r \\left[ -  \\alpha \\lambda_j  \\right]^+  = q_i(\\alpha) .\r\n\\end{equation}\r\n","start":44081,"end":44867},{"raw_tex":"Since $\\alpha v \\in \\mbox{range} \\hspace{0.3mm} \\mathcal{A}^*$ for all $\\alpha \\in \\mathbb{R}$, \r\nfor each $i \\in \\{ 1, \\dots , r \\}$, Proposition \\ref{prop:p-d-relation} and  (\\ref{eq:q_i(alpha)})  ensure that \r\n$\\langle c_i , x \\rangle \\leq  \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c_i -  \\alpha v \\right) , e \\right \\rangle = q_i(\\alpha)$\r\nfor all $\\alpha \\in \\mathbb{R}$, which implies the inequality in (\\ref{eq:bound_q}).\r\n","start":45516,"end":45950}]},{"id":"S047","semantic_type":"assumption","kind":"assumption","labels":["prop:upper-bound"],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"辅助割向量的谱表示","source_units":[{"raw_tex":"Suppose that $v \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*$ is given by $v = \\sum_{i=1}^r \\lambda_i c_i$ as in Proposition \\ref{prop:spectral}. \r\n","start":43190,"end":43343}]},{"id":"S048","semantic_type":"definition","kind":"definition","labels":["prop:upper-bound"],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"辅助标量上界函数","source_units":[{"raw_tex":"For each $i \\in \\{ 1, \\dots , r \\}$ and $\\alpha \\in \\mathbb{R}$, define $q_i(\\alpha) := \\left[    1-\\alpha \\lambda_i  \\right]^+ + \\sum_{j \\neq i }^r \\left[ -  \\alpha \\lambda_j  \\right]^+$.\r\n","start":43343,"end":43533}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"确定标量最小候选点","source_units":[{"raw_tex":"Note that, since $q_i(\\alpha)$ is a piece-wise linear convex function, if $\\lambda_i = 0$, it attains the minimum at $\\alpha = 0$ with $q_i(0) = 1$, and if $\\lambda_i \\neq 0$, it attains the minimum at $\\alpha = 0$ with $q_i(0) = 1$ or at $\\alpha =  \\frac{1}{\\lambda_i}$ with\r\n","start":44869,"end":45146}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"计算非零候选点的值","source_units":[{"raw_tex":"\\begin{equation}\r\n\\notag\r\nq \\left( \\frac{1}{\\lambda_i} \\right) = \\sum_{j \\neq i}^r \\left[ - \\frac{\\lambda_j}{\\lambda_i}  \\right]^+ \r\n= \\sum_{j=1}^r \\left[ - \\frac{\\lambda_j}{\\lambda_i}  \\right]^+ \r\n= \\left \\langle e , \\mathcal{P}_{\\mathcal{K}} \\left( -  \\frac{1}{\\lambda_i} v  \\right) \\right \\rangle.\r\n\\end{equation}\r\nThus, we obtain equivalence in (\\ref{eq:bound_q}).\r\n","start":45146,"end":45516}]},{"id":"S054","semantic_type":"proposition","kind":"proposition","labels":["eq:q_ell,i(alpha)","eq:upper-bound-direct1","prop:upper-bound-direct"],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"乘积对称锥的谱割界","source_units":[{"raw_tex":"\\begin{proposition}\r\n\\label{prop:upper-bound-direct}\r\nSuppose that, for any $v \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*$, the $\\ell$-th block element $v_\\ell$ of $v \\in \\mathbb{E}$ is decomposed into $v_\\ell = \\sum_{i=1}^{r_\\ell} {\\lambda(v_\\ell)}_i {c(v_\\ell)}_i$ as in Proposition \\ref{prop:spectral}.\r\nFor each $\\ell \\in \\{1,\\ldots,p \\}$ and $i \\in \\{1,\\ldots,r_p\\}$, define\r\n\\begin{equation}\r\n\\label{eq:q_ell,i(alpha)}\r\nq_{\\ell,i}(\\alpha) := \\left[ 1 - \\alpha {\\lambda(v_\\ell)}_i \\right]^+ + \\sum_{k \\neq i}^{r_\\ell} \\left[ -\\alpha {\\lambda(v_\\ell)}_k \\right]^+ + \\sum_{j \\neq \\ell}^p \\sum_{k=1}^{r_j} \\left[  -\\alpha {\\lambda(v_j)}_k \\right]^+ .\r\n\\end{equation}\r\nThen, the following relations hold for any feasible solution $x$ of ${\\rm P}_{S_{\\infty}}(\\mathcal{A})$, $\\ell \\in \\{1,\\ldots,p \\}$ and $i \\in \\{1,\\ldots,r_p\\}$.\r\n\\begin{equation}\r\n\\label{eq:upper-bound-direct1}\r\n\\langle {c(v_\\ell)}_i , x_\\ell \\rangle \\leq  \\min_{\\alpha \\in \\mathbb{R} } q_{\\ell,i}(\\alpha) =\r\n\\begin{cases}\r\n\\min \\left\\{  1 ,  \\left \\langle  e , \\mathcal{P}_{\\mathcal{K}} \\left( -  \\frac{1}{{\\lambda(v_\\ell)}_i} v  \\right) \\right \\rangle \\right\\} & \\mbox{if ${\\lambda(v_\\ell)}_i \\neq 0$}, \\\\\r\n1 & \\mbox{if ${\\lambda(v_\\ell)}_i = 0$} \r\n\\end{cases}\r\n.\r\n\\end{equation}\r\n\\end{proposition}","start":55306,"end":56584}]},{"id":"S055","semantic_type":"assumption","kind":"assumption","labels":["prop:upper-bound-direct"],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"主命题的分块割向量","source_units":[{"raw_tex":"Suppose that, for any $v \\in \\mbox{\\rm range} \\hspace{0.3mm} \\mathcal{A}^*$, the $\\ell$-th block element $v_\\ell$ of $v \\in \\mathbb{E}$ is decomposed into $v_\\ell = \\sum_{i=1}^{r_\\ell} {\\lambda(v_\\ell)}_i {c(v_\\ell)}_i$ as in Proposition \\ref{prop:spectral}.\r\n","start":55360,"end":55620}]},{"id":"S056","semantic_type":"definition","kind":"definition","labels":["eq:q_ell,i(alpha)","prop:upper-bound-direct"],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"分块标量割函数","source_units":[{"raw_tex":"For each $\\ell \\in \\{1,\\ldots,p \\}$ and $i \\in \\{1,\\ldots,r_p\\}$, define\r\n\\begin{equation}\r\n\\label{eq:q_ell,i(alpha)}\r\nq_{\\ell,i}(\\alpha) := \\left[ 1 - \\alpha {\\lambda(v_\\ell)}_i \\right]^+ + \\sum_{k \\neq i}^{r_\\ell} \\left[ -\\alpha {\\lambda(v_\\ell)}_k \\right]^+ + \\sum_{j \\neq \\ell}^p \\sum_{k=1}^{r_j} \\left[  -\\alpha {\\lambda(v_j)}_k \\right]^+ .\r\n\\end{equation}\r\n","start":55620,"end":55983}]},{"id":"S057","semantic_type":"proof","kind":"proof","labels":["eq:upper-bound-direct2"],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"乘积锥割界的完整证明","source_units":[{"raw_tex":"\\begin{proof}\r\nLet $c \\in \\mathbb {E}$ be an element whose $\\ell$-th block element is $c_\\ell = c (v_\\ell)_i$ and other block elements take $0$.\r\nFor any real number $\\alpha \\in \\mathbb{R}$, Proposition \\ref {prop:p-d-relation} ensures that\r\n\\begin{align}\r\n\\label{eq:upper-bound-direct2}\r\n\\langle {c(v_\\ell)}_i , x_\\ell \\rangle = \\langle c,x \\rangle &\\leq \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c -  \\alpha v \\right) , e \\right \\rangle \\notag \\\\\r\n&= \\left \\langle \\mathcal{P}_{\\mathcal{K}_\\ell} \\left( {c(v_\\ell)}_i -  \\alpha v_\\ell \\right) , e_\\ell \\right \\rangle + \\sum_{j \\neq \\ell}^p \\left \\langle \\mathcal{P}_{\\mathcal{K}_j} \\left( -  \\alpha v_j \\right) , e_j \\right \\rangle \\notag \\\\\r\n&= \\left[ 1 - \\alpha {\\lambda(v_\\ell)}_i \\right]^+ + \\sum_{k \\neq i}^{r_\\ell} \\left[ -\\alpha {\\lambda(v_\\ell)}_k \\right]^+ + \\sum_{j \\neq \\ell}^p \\sum_{k=1}^{r_j} \\left[  -\\alpha {\\lambda(v_j)}_k \\right]^+  = q_{\\ell,i}(\\alpha). \r\n\\end{align}\r\nWe obtain (\\ref{eq:upper-bound-direct1}) by following a similar argument to the one used in the proof of Proposition \\ref{prop:upper-bound}.\r\n\\end{proof}","start":56588,"end":57682}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"零填充所选块幂等元","source_units":[{"raw_tex":"Let $c \\in \\mathbb {E}$ be an element whose $\\ell$-th block element is $c_\\ell = c (v_\\ell)_i$ and other block elements take $0$.\r\n","start":56603,"end":56734}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"取得全空间弱对偶上界","source_units":[{"raw_tex":"For any real number $\\alpha \\in \\mathbb{R}$, Proposition \\ref {prop:p-d-relation} ensures that\r\n\\begin{align}\r\n\\label{eq:upper-bound-direct2}\r\n\\langle {c(v_\\ell)}_i , x_\\ell \\rangle = \\langle c,x \\rangle &\\leq \\left \\langle \\mathcal{P}_\\mathcal{K} \\left( c -  \\alpha v \\right) , e \\right \\rangle \\notag \\\\\r\n","start":56734,"end":57041}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":["eq:upper-bound-direct2"],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"把投影迹分解到各块","source_units":[{"raw_tex":"&= \\left \\langle \\mathcal{P}_{\\mathcal{K}_\\ell} \\left( {c(v_\\ell)}_i -  \\alpha v_\\ell \\right) , e_\\ell \\right \\rangle + \\sum_{j \\neq \\ell}^p \\left \\langle \\mathcal{P}_{\\mathcal{K}_j} \\left( -  \\alpha v_j \\right) , e_j \\right \\rangle \\notag \\\\\r\n","start":57041,"end":57285}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"识别分块正部和","source_units":[{"raw_tex":"&= \\left[ 1 - \\alpha {\\lambda(v_\\ell)}_i \\right]^+ + \\sum_{k \\neq i}^{r_\\ell} \\left[ -\\alpha {\\lambda(v_\\ell)}_k \\right]^+ + \\sum_{j \\neq \\ell}^p \\sum_{k=1}^{r_j} \\left[  -\\alpha {\\lambda(v_j)}_k \\right]^+  = q_{\\ell,i}(\\alpha). \r\n\\end{align}\r\n","start":57285,"end":57529}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"复用标量最小化论证","source_units":[{"raw_tex":"We obtain (\\ref{eq:upper-bound-direct1}) by following a similar argument to the one used in the proof of Proposition \\ref{prop:upper-bound}.\r\n","start":57529,"end":57671}]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S033","S035":"S033","S036":"S033","S037":"S033","S038":"S033","S039":"S039","S040":"S040","S041":"S039","S042":"S039","S043":"S039","S044":"S039","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S046","S050":"S046","S051":"S051","S052":"S052","S053":"S046","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062"}}}}
{"record_id":"2306.15288v2","paper":{"title":"Complexity of Chordal Conversion for Sparse Semidefinite Programs with Small Treewidth","authors":["Zhang, Richard Y."],"arxiv_id":"2306.15288","version":"v2","journal":"Mathematical Programming","abstract_url":"https://arxiv.org/abs/2306.15288v2","pdf_url":"https://arxiv.org/pdf/2306.15288v2","doi":"10.1007/s10107-024-02137-5"},"context":{"raw_tex":"Consider directly applying a general-purpose interior-point method\nsolver, like SeDuMi~\\citep{sturm1999using}, SDPT3~\\citep{toh1999sdpt3},\nand MOSEK~\\citep{mosek2019}, to solve the standard-form semidefinite\nprogram to high accuracy:\n\\begin{equation}\n\\min_{X\\in\\S^{n}}\\;\\inner CX\\quad\\text{s.t.}\\;X\\succeq0,\\quad\\inner{A_{i}}X\\le b_{i}\\;\\text{for all }i\\in\\{1,2,\\dots,m\\}.\\tag{SDP}\\label{eq:sdp}\n\\end{equation}\nHere, $\\S^{n}$ denotes the set of $n\\times n$ real symmetric matrices\nwith inner product $\\inner{A_{i}}X=\\tr(A_{i}X)$, and $X\\succeq0$\nmeans that $X$ is positive semidefinite. At each iteration, the $n\\times n$\nmatrix variable $X$ is generally fully dense, even when problem data\n$C,A_{1},\\dots,A_{m}\\in\\S^{n}$ and $b_{1},\\dots,b_{m}\\in\\R$ are\nsparse. The per-iteration cost of the solver is usually at least $\\Omega(n^{3})$\ntime, which in practice limits the value of $n$ to no more than a\nfew thousand.\n\n\nIn order for chordal conversion to be fast, a well-known \\emph{necessary}\ncondition is for the underlying aggregate sparsity graph, which is\ndefined $G=(V,E)$ with\n\\begin{gather}\nV=\\{1,2,\\dots,n\\},\\quad E=\\spar(C)\\cup\\spar(A_{1})\\cup\\cdots\\cup\\spar(A_{m}),\\label{eq:Edef}\\\\\n\\text{where }\\spar(C)\\equiv\\{(i,j):C[i,j]\\ne0\\text{ for }i>j\\},\\nonumber \n\\end{gather}\nto be ``tree-like'' with a small\\emph{ treewidth} $\\tw(G)\\ll n$.\nWe defer a formal definition of the treewidth to \\defref{tw}, and\nonly note that the value of $\\omega$ is lower-bounded as $\\omega\\ge1+\\tw(G)$.\nIn other words, while $\\omega$ can be decreased by symmetrically\nreordering the columns and rows of the data matrices, as in $C\\gets\\Pi C\\Pi^{T}$\nand $A_{i}\\gets\\Pi A_{i}\\Pi^{T}$ for some permutation matrix $\\Pi$,\nactually achieving $\\omega\\ll n$ is possible only if $\\tw(G)\\ll n$.\nHowever, as illustrated by \\exaref{diag}, even on a graph with $\\tw(G)=0$,\nand even when using the optimal $\\omega=1$, chordal conversion may\nstill not be fast.\n\n\nIn this paper, we show that a \\emph{sufficient} condition for chordal\nconversion to be fast is for a supergraph $\\o G\\supseteq G$, that\nadditionally captures the correlation between constraints, to also\\footnote{Note that $\\tw(\\o G)\\ge\\tw(G)$ holds by virtue of $\\o G\\supseteq G$.}\nbe ``tree-like'' with a small treewidth $\\tw(\\o G)\\ll n$. Concretely,\nwe construct the \\emph{extended} aggregate sparsity graph $\\o G=(V,\\overline{E})$\nby forcing each constraint matrix $A_{i}$ to be its own clique in\n$G$:\n\\begin{gather}\nV=\\{1,2,\\dots,n\\},\\quad\\overline{E}=\\spar(C)\\cup\\clique(A_{1})\\cup\\cdots\\cup\\clique(A_{m})\\label{eq:Ebardef}\\\\\n\\text{where }\\clique(A)=\\{(i,j):A[i,k]\\ne0\\text{ or }A[k,j]\\ne0\\text{ for some }k\\}.\\nonumber \n\\end{gather}\nThis is the union between $G$ and the constraint intersection graph~\\citep{fulkerson1965incidence}\n(or the dual graph~\\citep{dong2021nearly} or the correlative sparsity~\\citep{waki2006sums,kobayashi2008correlative})\nfor the rank-1 instance of (\\ref{eq:sdp}). In other words, we add\na new edge $(i,j)$ to $E$ whenever $x_{i}$ and $x_{j}$ appear\ntogether in a common constraint $x^{T}A_{k}x\\le b_{k}$ for some $k$.\nThe fact that this contributes $\\clique(A_{k})\\subseteq\\o E$ reflects\nthe reality that each $x^{T}A_{k}x\\le b_{k}$ densely couples all\naffected elements of $x$ together, forcing them to be optimized simultaneously.\nIn contrast, the cost $x^{T}Cx$ can be optimized sequentially over\nthe elements of $x$, which is why $\\clique(C)$ is absent.\n\n\nWrite $\\R^{m\\times n}$ as the set of $m\\times n$ matrices with real\ncoefficients, with associated matrix inner product $\\inner AB=\\tr A^{T}B$\nand norm $\\|A\\|_{F}=\\sqrt{\\inner AA}$. Write $\\S^{n}\\subseteq\\R^{n\\times n}$\nas the set of $n\\times n$ real symmetric matrices, meaning that $X=X^{T}$\nholds for all $X\\in\\S^{n}$, and write $\\S_{+}^{n}\\subseteq\\S^{n}$\nas the set of symmetric positive semidefinite matrices. Write $\\R_{+}^{n}\\subseteq\\R^{n}$\nas the usual positive orthant.\n\nThe set of $n\\times n$ symmetric matrices with sparsity pattern $E$\ncan be defined as\n\\[\n\\S_{E}^{n}\\equiv\\left\\{ X\\in\\S^{n}:X[i,j]=X[j,i]=0\\quad\\text{for all }i\\ne j,\\quad(i,j)\\notin E\\right\\} .\n\\]\nConversely, the minimum sparsity pattern of a \\emph{symmetric} matrix\n$X\\in\\S^{n}$ is denoted\n\\[\n\\spar(X)\\equiv\\{(i,j):X[i,j]\\ne0,\\quad i\\ge j\\}.\n\\]\nWe also write $\\spar(M)\\equiv\\spar(M+M^{T})$ for a nonsymmetric matrix\n$M$ where there is no confusion. We write $\\proj_{E}(M)\\equiv\\arg\\min_{X\\in\\S_{E}^{n}}\\|M-X\\|_{F}$\nas the projection of $M\\in\\S^{n}$ onto the sparsity pattern $E$. \n\nWe define the dense sparsity pattern induced by $J\\subseteq\\{1,2,\\dots,n\\}$\nas follows\n\\[\n\\clique(J)=\\{(i,j):i,j\\in J,\\quad i\\ge j\\}.\n\\]\nWe also define the vertex support of a possibly nonsymmetric matrix\n$M$ as the following \n\\[\n\\supp(M)=\\{i:M[i,j]\\ne0\\quad\\text{for some }j\\}.\n\\]\nWe write $\\clique(A)\\equiv\\clique(\\supp(A))$ where there is no confusion.\nThis notation is motivated by the fact that $\\spar(A)\\subseteq\\clique(A)$\nfor $A\\in\\S^{n}$, and $\\spar(PDP^{T})\\subseteq\\clique(P)$ for $P\\in\\R^{n\\times d}$\nand dense $D\\in\\S^{d}$. \n\nLet $F$ be a sparsity pattern of order $n$ that \\emph{contains all\nof its diagonal elements}, as in $F\\supseteq\\{(i,i):i\\in\\{1,2,\\dots,n\\}\\}$.\nIn this case, $\\dim(\\S_{F}^{n})=|F|$ holds exactly, so we define\na \\emph{symmetric vectorization} operator $\\svec_{F}:\\S_{F}^{n}\\to\\R^{|F|}$\nto implement an isometry with the usual Euclidean space, as in \n\\[\n\\inner{\\svec_{F}(X)}{\\svec_{F}(Y)}=\\inner XY\\text{ for all }X,Y\\in{\\S_{F}^{n}}.\n\\]\nWe will explicitly require $\\svec_{F}(\\cdot)$ to be defined according\nto a column-stacking construction\n\\[\n\\svec_{F}(X)=(x_{j})_{j=1}^{n}\\quad\\text{where }x_{j}=\\left(X[i,i],\\quad\\sqrt{2}\\cdot(X[i,j]:i>j,\\quad(i,j)\\in F)\\right).\n\\]\nWe also define a companion indexing operator $\\idx_{F}(\\cdot,\\cdot)$\nto index elements of the vectorization $x=\\svec_{F}(X)$:\n\\begin{gather*}\nx[\\idx_{F}(i,i)]=X[i,i]\\quad\\text{for all }i\\in\\{1,2,\\dots,n\\},\\\\\nx[\\idx_{F}(i,j)]=\\sqrt{2}X[i,j]\\quad\\text{for all }i>j,\\quad(i,j)\\in F.\n\\end{gather*}\nAs we will see later, the correctness of our proof crucially relies\non the fact that $\\idx_{F}(\\cdot,\\cdot)$ implements a \\emph{raster\nordering} over the elements of $F$. \n\n\nTo solve $Sx=b$ with $S\\succ0$ via Cholesky factorization, we first\ncompute the lower-triangular Cholesky factor $L=\\chol(S)$ according\nto the following recursive rule \n\\[\n\\chol\\left(\\begin{bmatrix}\\alpha & b^{T}\\\\\nb & D\n\\end{bmatrix}\\right)=\\begin{bmatrix}\\sqrt{\\alpha} & 0\\\\\n\\frac{1}{\\sqrt{\\alpha}}b & \\chol\\left(D-\\frac{1}{\\alpha}bb^{T}\\right)\n\\end{bmatrix},\\quad\\chol(\\alpha)=\\sqrt{\\alpha},\n\\]\nand then solve two triangular systems $Ly=b$ and $L^{T}x=y$ via\nback-substitution. If $S$ is sparse, then $L=\\chol(S)$ may also\nbe sparse. The sparsity pattern of $L$ can be directly computed from\nthe sparsity pattern of $S$, and without needing to examine the numerical\nvalues of its nonzeros.\n\\begin{definition}[Symbolic Cholesky]\n\\label{def:symbchol}The \\emph{symbolic Cholesky factor} $\\chol(E)$\nof a sparsity pattern $E$ of order $n$ is defined as $\\chol(E)\\equiv E_{n+1}$\nwhere $E_{1}=E$ and\n\\[\nE_{k+1}=E_{k}\\cup(k,k)\\cup\\{(i,j):(i,k)\\in E_{k},\\quad(j,k)\\in E_{k}\\text{ for }i>j>k\\}.\n\\]\n\\end{definition}\nOne can verify that $\\chol(\\spar(S))=\\spar(\\chol(S))$. Note that\n$\\chol(E)$ can be computed from $E$ in $O(|\\chol(E)|)$ time and\nmemory~\\citep[Theorem 5.4.4]{george1981computer}. The efficiency\nof a sparsity-exploiting algorithm for factorizing $L=\\chol(S)$ and\nsolving $Ly=b$ and $L^{T}x=y$ is determined by the \\emph{frontsize}\nof the sparse matrix $S$. \n\\begin{definition}[Frontsize]\n\\label{def:frontsize}The \\emph{frontsize} $\\omega(E)$ of a sparsity\npattern $E$ is defined $\\max_{j}|\\col_{F}(j)|$ where $F=\\chol(E)$\nand $\\col_{F}(j)\\equiv\\{j\\}\\cup\\{i>j:(i,j)\\in F\\}$. The frontsize\n$\\omega(S)\\equiv\\omega(\\spar(S))$ of a symmetric matrix $S$ is the\nfrontsize of its minimum sparsity pattern. \n\\end{definition}\nIntuitively, the frontsize $\\omega(S)$ is the maximum number of nonzero\nelements in a single column of the Cholesky factor $L=\\chol(S)$.\nThe following is well-known~\\citep{george1981computer}.\n\n\nThe cost of solving $Sx=b$ with sparse $S\\succ0$ can usually be\nreduced by first permuting the rows and columns of the matrix symmetrically,\nand then solving $(\\Pi S\\Pi^{T})\\Pi x=\\Pi y$ for some permutation\nmatrix $\\Pi$. For $E=\\spar(S)$, we write $E_{\\Pi}\\equiv\\spar(\\Pi S\\Pi^{T})$\nto denote its permuted sparsity pattern. It is a fundamental result\nin graph theory and linear algebra that the problem of minimizing\nthe frontsize $\\omega(E_{\\Pi})$ over the set of $n\\times n$ permutation\nmatrices $\\Pi\\in\\Perm(n)$ is the same problem as computing the treewidth\nof the graph $G=(V,E)$.\n\n\\begin{definition}[Treewidth]\n\\label{def:tw}A tree decomposition of a graph $G=(V,E)$ is a pair\n$(\\{J_{j}\\},T)$ in which each \\emph{bag} $J_{j}\\subseteq V$ is a\nsubset of vertices and $T$ is a tree such that:\n\\begin{itemize}\n\\item (Vertex cover) $\\bigcup_{j}J_{j}=V$;\n\\item (Edge cover) $\\bigcup_{j}(J_{j}\\times J_{j})\\supseteq E$;\n\\item (Running intersection) $J_{i}\\cap J_{j}\\subseteq J_{k}$ for every\n$k$ on the path of $i$ to $j$ on $T$.\n\\end{itemize}\nThe \\emph{width} of the tree decomposition is $\\max_{j}|J_{j}|-1$.\nThe \\emph{treewidth} of $G$, denoted $\\tw(G)$, is the minimum width\nover all valid tree decompositions on $G$.\n\\end{definition}\nThe connection is an immediate corollary of the following result,\nwhich establishes an equivalence between tree decompositions and the\nsparsity pattern of Cholesky factors. \n\\begin{proposition}[Perfect elimination ordering]\nGiven a sparsity pattern $E$ of order $n$, let $G=(V,E)$ where\n$V=\\{1,2,\\dots,n\\}$. For every tree decomposition of $G$ with width\n$\\tau$, there exists a perfect elimination ordering $\\Pi\\in\\Perm(n)$\nsuch that $\\omega(E_{\\Pi})=1+\\tau$. \n\\end{proposition}\nWe defer a proof to the texts~\\citep{george1981computer,vandenberghe2015chordal},\nand only note that, given a tree decomposition of width $\\tau$, the\ncorresponding perfect elimination ordering $\\Pi$ can be found in\n$O((1+\\tau)\\cdot n)$ time. \n\n\nThe basic approach for solving an SDP using a general-purpose solver\nis to reformulate the problem into the primal or the dual of the standard-form\nlinear conic program \n\\[\n\\min_{x\\in{\\R^{q}}}\\{\\c^{T}x:\\A x={\\b},x\\in\\K\\}\\ge\\max_{y\\in{\\R^{p}}}\\{\\b^{T}y:\\c-\\A^{T}y\\in{\\K_{*}}\\}\n\\]\n where the data are the matrix $\\A\\in\\R^{p\\times q}$, vectors $\\b\\in\\R^{p}$\nand $\\c\\in\\R^{q}$, and the problem closed convex cone $\\K\\subseteq\\R^{q}$,\nand the notation $\\K_{*}$ means the dual cone of $\\K$. We specify\nthe following basic assumptions on this problem to ensure that it\ncan be solved in polynomial time using a self-dual embedding~\\citep{de2000self}.\nBelow, we denote $\\one_{\\K}$ as the identity element on the cone\n$\\K$, and recall that every semidefinite cone is self-dual $\\K=\\K_{*}$. \n\\begin{definition}[Standard-form SDP]\n\\label{def:clp}We say that the problem data $\\A\\in\\R^{p\\times q},$\n$\\b\\in\\R^{p},$ $\\c\\in\\R^{q},$ and $\\K\\subseteq\\R^{q}$ describe\nan SDP in $(n,\\omega)$-\\emph{standard form} if:\n\\begin{enumerate}\n\\item (Dimensions) The cone $\\K=\\svec(\\S_{+}^{\\omega_{1}})\\times\\cdots\\times\\svec(\\S_{+}^{\\omega_{\\ell}})$\nis the Cartesian product of semidefinite cones whose orders $\\omega_{1},\\omega_{2},\\dots,\\omega_{\\ell}$\nsatisfy \n\\[\n\\omega=\\max_{i}\\omega_{i},\\qquad n=\\sum_{i=1}^{\\ell}\\omega_{i},\\qquad{q}=\\frac{1}{2}\\sum_{i=1}^{\\ell}\\omega_{i}(\\omega_{i}+1).\n\\]\n\\item (Linear independence) $\\A^{T}y=0$ holds if and only if $y=0$. \n\\item (Strong duality is attained) There exist a choice of $x^{\\star},y^{\\star}$\nthat satisfy\n\\begin{gather*}\n\\A x^{\\star}=\\b,\\quad x^{\\star}\\in\\K,\\quad\\c-\\A^{T}y^{\\star}\\in\\K,\\quad\\c^{T}x^{\\star}=\\b^{T}y^{\\star}.\n\\end{gather*}\n\\end{enumerate}\n\\end{definition}\n%\n\\begin{definition}[General-purpose solver]\n\\label{def:ipm}We say that $\\IPM$ implements a \\emph{general-purpose\nsolver} if it satisfies the following conditions\n\\begin{enumerate}\n\\item (Iteration count) Given data $(\\A,\\b,\\c,\\K)$ in $(n,\\omega)$-standard-form,\ncalling $(x,y)=\\IPM(\\epsilon,\\A,\\b,\\c,\\K)$ yields iterates $(x,y)\\in\\K\\times{\\R^{p}}$\nthat satisfy the following, in at most $O(\\sqrt{n}\\log(1/\\epsilon))$\niterations\n\\[\n\\|\\A x-\\b\\|\\le\\epsilon,\\quad{\\c-\\A^{T}y+\\epsilon\\cdot\\one_{\\K}\\in\\K,}\\quad\\c^{T}x-\\b^{T}y\\le\\epsilon\\cdot n.\n\\]\n\\item (Per-iteration costs) Each iteration costs an overhead of $O(\\omega^{2}n)$\ntime and $O(\\omega n)$ memory, plus the cost of solving $O(1)$ instances\nof the \\emph{Schur complement equation} \n\\[\n\\A\\nabla^{2}f(w)\\A^{T}\\Delta y=r,\\qquad f(w)=-\\log\\det(w)\n\\]\nby forming $\\Hess=\\A\\nabla^{2}f(w)\\A^{T}$, factoring $\\L=\\chol(\\Pi\\Hess\\Pi^{T})$\nand then solving $\\L z={\\Pi}r$ and $\\L^{T}(\\Pi\\Delta y)=z$. Here,\nthe fill-reducing permutation $\\Pi$ is required to be no worse than\nthe natural ordering, as in $\\omega(\\Pi\\Hess\\Pi^{T})\\le\\omega(\\Hess)$.\n\\end{enumerate}\n\\end{definition}\nNote that \\defref{ipm} is rigorously satisfied by SeDuMi~\\citep{sturm1999using,sturm2002implementation},\nSDPT3~\\citep{toh1999sdpt3,tutuncu2003solving}, and MOSEK~\\citep{andersen2000mosek,andersen2003implementing,mosek2019}.\nGiven that the correctness of our overall claims crucially depends\non the characterization in \\defref{ipm}, we state a concrete interior-point\nmethod in \\appref{ipm} that implements these specifications. \n\n\n\\begin{algorithm}[t]\n\\caption{\\label{alg:myalg}Chordal conversion}\n\n\\textbf{Input.} Accuracy parameter $\\epsilon>0$, problem data $C,A_{1},A_{2},\\dots,A_{m}\\in\\S^{n}$,\n$b\\in\\R^{m}$, fill-reducing permutation $\\Pi$.\n\n\\textbf{Output.} Approximate solutions $U\\in\\R^{n\\times\\omega}$ and\n$v\\in\\R^{m}$ to the following primal-dual pair:\n\\[\n\\min_{U}\\left\\{ \\inner C{UU^{T}}:\\inner{A_{i}}{UU^{T}}\\le b_{i}\\text{ for all }i\\right\\} \\ge\\max_{v\\le0}\\left\\{ \\inner bv:{\\textstyle \\sum_{i}}v_{i}A_{i}\\preceq C\\right\\} .\n\\]\n\n\\textbf{Algorithm.}\n\\begin{enumerate}\n\\item (Symbolic factorization) Pre-order all data matrices $\\tilde{A}_{i}=\\Pi A_{i}\\Pi^{T}$\nand $\\tilde{C}=\\Pi C\\Pi^{T}$. Compute the permuted aggregate sparsity\npattern $E=\\spar(\\tilde{C})\\cup\\bigcup_{i}\\spar(\\tilde{A}_{i})$,\nits lower-triangular symbolic Cholesky factor $F=\\chol(E)$, and define\nthe following\n\\[\nJ_{j}=\\col_{F}(j)\\equiv\\{j\\}\\cup\\{i>j:(i,j)\\in F\\},\\quad\\omega_{j}\\equiv|J_{j}|,\\quad\\omega\\equiv\\max_{j}\\omega_{j}.\n\\]\n\\item (Numerical solution) Call $(x,y)=\\IPM(\\epsilon,\\A,\\b,\\c,\\K)$ where\n$\\IPM$ is a general-purpose solver (\\defref{ipm}), and the problem\ndata $\\A,\\b,\\c,\\K$ implement the following\n\\[\nY^{\\star}=\\arg\\max_{Y\\in\\S_{F}^{n}}\\inner{-\\tilde{C}}Y\\text{ s.t. }\\begin{array}{cc}\nb_{i}-\\inner{\\tilde{A}_{i}}Y\\ge0 & \\quad\\text{for all }i\\in\\{1,2,\\dots,m\\}\\\\\nY[J_{j},J_{j}]\\succeq0 & \\quad\\text{for all }j\\in\\{1,2,\\dots,n\\}\n\\end{array}\n\\]\nas an instance of $y^{\\star}=\\arg\\max_{y}\\{\\b^{T}y:\\c-\\A^{T}y\\in\\K\\}$\nwith $y^{\\star}=\\svec_{F}(Y^{\\star})$.\n\\item (Back substitution) Recover $Y\\in\\S_{F}^{n}$ from $y=\\svec_{F}(Y)$,\nand compute $\\delta=-\\min_{j}\\{0,\\lambda_{\\min}(Y[J_{j},J_{j}])\\}$.\nSolve the positive semidefinite matrix completion\n\\[\n\\text{find }\\tilde{U}\\in\\R^{n\\times\\omega}\\text{ such that }(\\tilde{U}\\tilde{U}^{T})[J_{j},J_{j}]=Y[J_{j},J_{j}]+\\delta I\\text{ for all }j\n\\]\nusing \\citep[Algorithm~2]{sun2015decomposition}. Output $v=-(x_{i})_{i=1}^{m}$\nand $U=\\Pi^{T}\\tilde{U}$.\n\\end{enumerate}\n\\end{algorithm}\n \\algref{myalg} summarizes the standard implementation of chordal\nconversion, which is known as the ``d-space conversion method using\nbasis representation'' in \\citet{kim2011exploiting}. Our only modification\nis to recover $X=UU^{T}$ from $Y=\\proj_{F}(X)$ in Step 3 using the\n\\emph{low-rank} chordal completion~\\citep[Theorem~1.5]{dancis1992positive},\ninstead of the \\emph{maximum determinant} chordal completion~\\citep[Theorem~2]{grone1984positive}\nas originally proposed by \\citet{fukuda2001exploiting}. We note that\nboth recovery procedures have the same complexity of $O(\\omega^{3}n)$\ntime and $O(\\omega^{2}n)$ memory, but the former puts $X$ in a more\nconvenient form~\\citep{sun2015decomposition}.\n\n\nTo state the Schur complement sparsity $E^{(2)}$ explicitly, note\nthat the problem data $(\\A,\\b,\\c,\\K)$ in Step 2 of \\algref{myalg}\nare \\begin{subequations}\\label{eq:AKdef}\n\\begin{equation}\n\\begin{array}{c}\n\\A=[\\svec_{F}(A_{1}),\\dots,\\svec_{F}(A_{m}),-\\P_{1},\\dots,-\\P_{n}],\\\\\n\\b=-\\svec_{F}(C),\\qquad\\c=(b,0),\\\\\n\\K=\\R_{+}^{m}\\times\\svec(\\S_{+}^{\\omega_{1}})\\times\\svec(\\S_{+}^{\\omega_{2}})\\times\\cdots\\times\\svec(\\S_{+}^{\\omega_{n}})\n\\end{array}\\label{eq:AKdef-1}\n\\end{equation}\nwhere $\\omega_{j}\\equiv|\\col_{F}(j)|$ and each $\\P_{j}$ is implicitly\ndefined to satisfy \n\\begin{equation}\n\\P_{j}^{T}\\svec_{F}(Y)=\\svec(Y[\\col_{F}(j),\\col_{F}(j)])\\quad\\text{for all }Y\\in\\S_{F}^{n}.\\label{eq:AKdef-2}\n\\end{equation}\n\\end{subequations}The resulting Schur complement matrix reads\n\\begin{gather}\n\\A\\nabla^{2}f(w)\\A^{T}=\\sum_{i=1}^{m}d_{i}\\svec_{F}(A_{i})\\svec_{F}(A_{i})^{T}+\\sum_{j=1}^{n}\\P_{j}\\D_{j}\\P_{j}^{T}\\label{eq:Hessdef}\\\\\n\\text{where }d_{i}=w_{i}^{-2},\\qquad\\D_{j}\\svec(X_{j})=\\svec(W_{j}^{-1}X_{j}W_{j}^{-1})\\nonumber \n\\end{gather}\nand $w=(w_{1},\\dots,w_{m},\\svec(W_{1}),\\dots,\\svec(W_{n}))\\in\\interior(\\K)$\nis a scaling point. The associated sparsity pattern, aggregated over\nall possible choices of scaling $w$, is as follows\n\\begin{align}\nE^{(2)} & =\\{(i,j):(\\A\\nabla^{2}f(w)\\A^{T})[i,j]\\ne0\\text{ for some }w\\in\\interior(\\K)\\}\\nonumber \\\\\n & =\\left(\\bigcup_{i=1}^{m}\\spar(a_{i}a_{i}^{T})\\right)\\cup\\left(\\bigcup_{j=1}^{n}\\spar(\\P_{j}\\D_{j}\\P_{j}^{T})\\right)\\nonumber \\\\\n & =\\left(\\bigcup_{i=1}^{m}\\clique(\\supp(a_{i}))\\right)\\cup\\left(\\bigcup_{j=1}^{n}\\clique(\\supp(\\P_{j}))\\right)\\label{eq:E2def}\n\\end{align}\nwhere we have written $a_{i}=\\svec_{F}(A_{i})$. The result below\nsays that if $\\omega(\\o E)=O(1)$, then $\\omega(\\o E^{(2)})=O(1)$. \n\n\nLet us now give an end-to-end complexity guarantee for \\algref{myalg}.\nWe will need the following assumption to ensure that the data $(\\A,\\b,\\c,\\K)$\npreviously defined in (\\ref{eq:AKdef}) specifies an SDP in $(N,\\omega)$-standard\nform, where $N=m+\\omega n$ and $\\omega\\equiv\\omega(E)$. \n\n\\begin{assume}[Strong duality is attained]\\label{asm:strict}There\nexists a primal-dual pair $X^{\\star}\\succeq0$ and $v^{\\star}\\le0$\nthat are feasible $\\inner{A_{i}}{X^{\\star}}\\le b_{i}$ for all $i$\nand $\\sum_{i}v_{i}^{\\star}A_{i}\\preceq C$ and coincide in their objectives\n$\\inner C{X^{\\star}}=\\inner b{v^{\\star}}$.\n\n\\end{assume}\n\n\n$E$ of order $n$ can be viewed as the edge set of an undirected\ngraph $G=(V,E)$ on vertices $V=\\{1,2,\\dots,n\\}$. The underlying\nprinciple behind our proof is the fact that the frontsize is monotone\nunder the subgraph relation: if $G'=(V',E')$ is a subgraph of $G=(V,E)$,\nthen $\\omega(E)\\ge\\omega(E')$. \n\nTo state this formally, we denote the sparsity pattern induced by\na subset of vertices $U=\\{u_{1},u_{2},\\dots,u_{p}\\}\\subseteq V$ as\nfollows\n\\[\nE[U]\\equiv\\{(i,j):(u_{i},u_{j})\\in E\\}\\text{ where }u_{1}<u_{2}<\\cdots<u_{p}.\n\\]\nNote that we always sort the elements of $U$. Our definition is made\nso that if $E=\\spar(X)$, then $E[U]=\\spar(X[U,U])$, without any\nreordering of the rows and columns. \n\n\nOur proof of \\lemref{quadlift} is based on a connection between\nzero-fill sparsity patterns, for which sequential Gaussian elimination\nresults in no additionally fill-in, and a ``sorted'' extension of\nthe running intersection property. \n\\begin{definition}[ZF]\nThe sparsity pattern $F$ is said to be\\emph{ zero-fill} if $F=\\chol(F)$.\n\\end{definition}\nEquivalently, if $F$ is zero-fill, then $(i,k)\\in F$ and $(j,k)\\in F$\nimplies $(i,j)\\in F$ for $i>j>k$ via the definition of the symbolic\nCholesky factor (\\defref{symbchol}). \n\n\\begin{definition}[RIP]\n\\label{def:rip}The sequence of subsets $J_{1},J_{2},\\dots,J_{\\ell}$\nwith $J_{j}\\subset\\N$ satisfies the \\emph{sorted running intersection\nproperty} if there exists a parent pointer $\\p:\\{1,2,\\dots,\\ell-1\\}\\to\\{2,3,\\dots,\\ell\\}$\nsuch that the following holds for all $1\\le j<\\ell$:\n\\begin{gather*}\n\\p(j)>j,\\quad J_{\\p(j)}\\supseteq J_{j}\\cap(J_{j+1}\\cup J_{j+2}\\cup\\cdots\\cup J_{\\ell}),\\quad\\min\\{J_{\\p(j)}\\}>\\max\\{J_{j}\\backslash J_{\\p(j)}\\}.\n\\end{gather*}\n\\end{definition}\nThe symbolic Cholesky factor $F=\\chol(E)$ for a sparsity pattern\n$E$ is the canonical example of a zero-fill sparsity pattern. In\nturn, the corresponding column sets $J_{j}=\\col_{F}(j)$ are the canonical\nexample of sequence of subsets that satisfy the sorted version of\nthe running intersection property.\n\n\nOur proof is based on the fact that the ``lifted'' sparsity pattern\n$F^{(2)}$ can be constructed as $F^{(2)}=\\bigcup_{k=1}^{n}\\clique(J_{k}^{(2)})$\nwith respect to the following ``lifted'' index sets\n\\begin{gather}\nJ_{k}^{(2)}\\equiv\\idx_{F}(\\clique(J_{k}))=\\{\\idx_{F}(i,j):i,j\\in J_{k},\\quad i\\ge j\\}.\\label{eq:Jk2_def}\n\\end{gather}\nWe need to show that, if the original index sets $J_{1},J_{2},\\dots,J_{k}$\nsatisfy the running intersection property, then the lifted index sets\n$J_{1}^{(2)},J_{2}^{(2)},\\dots,J_{k}^{(2)}$ will inherit the running\nintersection property. Our key insight is that the index operator\n$\\idx_{F}$ implements a \\emph{raster ordering}.\n\\begin{lemma}[Raster ordering]\nThe ordering $\\idx_{F}:F\\to\\N$ satisfies the following, for all\n$(i,j)\\in F$ with $i\\ge j$ and $(i',j')\\in F$ with $i'\\ge j'$:\n\\begin{itemize}\n\\item If $j>j'$, then $\\idx(i,j)>\\idx(i',j')$ holds. \n\\item If $j=j'$ and $i>i'$, then $\\idx(i,j)>\\idx(i',j')$ holds. \n\\end{itemize}\n\\end{lemma}\n\n\nWe will need some additional notation. For $V=\\{1,2,\\dots,n\\}$ and\n$J\\subseteq V$, we denote the subset of $J$ induced by the elements\nin $U=\\{u_{1},u_{2},\\dots,u_{p}\\}\\subseteq V$ as follows\n\\[\nJ[U]\\equiv\\{i:u_{i}\\in J\\}\\text{ where }u_{1}<u_{2}<\\cdots<u_{p}.\n\\]\nOur definition is made so that if $E=\\clique(J)$, then $E[U]=\\clique(J[U])$.\n\n\nGiven data $C,A_{1},\\dots,A_{m}\\in\\S^{n}$ and $b\\in\\R^{m}$, define\n$(\\A,\\b,\\c,\\K)$ as in (\\ref{eq:AKdef}). In this section, we verify\nthat $(\\A,\\b,\\c,\\K)$ specifies an SDP that satisfies the regularity\nassumptions in \\defref{clp}. \n\n\nWe prove \\propref{general} by using the short-step method of Nesterov\nand Todd~\\citep[Algorithm~6.1]{nesterov1998primal} to solve the\nextended homogeneous self-dual embedding\\begin{subequations}\\label{eq:hsd}\n\\begin{alignat}{2}\n & \\min_{x,y,s,\\kappa,\\tau,\\theta}\\quad & (n+1)\\theta\\\\\n & \\text{s.t.} & \\begin{bmatrix}0 & +\\A^{T} & -\\c & -r_{d}\\\\\n-\\A & 0 & +\\b & -r_{p}\\\\\n+\\c^{T} & -\\b^{T} & 0 & -r_{c}\\\\\nr_{d}^{T} & r_{p}^{T} & r_{c} & 0\n\\end{bmatrix}\\begin{bmatrix}x\\\\\ny\\\\\n\\tau\\\\\n\\theta\n\\end{bmatrix}+\\begin{bmatrix}s\\\\\n0\\\\\n\\kappa\\\\\n0\n\\end{bmatrix} & =\\begin{bmatrix}0\\\\\n0\\\\\n0\\\\\nn+1\n\\end{bmatrix}\\label{eq:hsdfeas}\\\\\n &  & x,s\\in\\K,\\quad\\kappa,\\tau & \\ge0,\\label{eq:hsdcone}\n\\end{alignat}\n\\end{subequations}where $r_{d}=\\one_{\\K}-\\c$ and $r_{p}=-\\A\\one_{\\K}+\\b$\nand $r_{c}=1+\\c^{T}\\one_{\\K}$. Beginning at the strictly feasible,\nperfectly centered point in (\\ref{eq:hsdinit}) for $\\mu=1$:\n\\begin{gather}\n\\theta^{(0)}=\\tau^{(0)}=\\kappa^{(0)}=1,\\quad y^{(0)}=0,\\quad x^{(0)}=s^{(0)}=\\one_{\\K}.\\label{eq:hsdinit}\n\\end{gather}\n we take the following steps\\begin{subequations}\\label{eq:hsd_ipm}\n\\begin{align}\n\\mu^{+} & =\\left(1-\\frac{1}{15\\sqrt{n+1}}\\right)\\cdot\\frac{x^{T}s+\\tau\\kappa}{n+1},\\\\\n(x^{+},y^{+},s^{+},\\tau^{+},\\theta^{+},\\kappa^{+}) & =(x,y,s,\\tau,\\theta,\\kappa)+(\\Delta x,\\Delta y,\\Delta s,\\Delta\\tau,\\Delta\\theta,\\Delta\\kappa).\\nonumber \n\\end{align}\n\\end{subequations}along the search direction defined by the linear\nsystem~\\citep[Eqn. 9]{todd1998nesterov} \\begin{subequations}\\label{eq:hsd_nt}\n\\begin{align}\n\\begin{bmatrix}0 & +\\A^{T} & -\\c & -r_{p}\\\\\n-\\A & 0 & +\\b & -r_{d}\\\\\n+\\c^{T} & -\\b^{T} & 0 & -r_{c}\\\\\n+r_{p}^{T} & +r_{d}^{T} & +r_{c} & 0\n\\end{bmatrix}\\begin{bmatrix}\\Delta x\\\\\n\\Delta y\\\\\n\\Delta\\tau\\\\\n\\Delta\\theta\n\\end{bmatrix}+\\begin{bmatrix}\\Delta s\\\\\n0\\\\\n\\Delta\\kappa\\\\\n0\n\\end{bmatrix} & =0,\\\\\ns+\\Delta s+\\nabla^{2}f(w)\\Delta x+\\mu^{+}\\nabla f(x) & =0,\\\\\n\\kappa+\\Delta\\kappa+(\\kappa/\\tau)\\Delta\\tau-\\mu^{+}\\tau^{-1} & =0.\n\\end{align}\n\\end{subequations}where $f(w)=-\\log\\det w$ is the usual self-concordant\nbarrier function, and $w\\in\\interior(\\K)$ is the unique point that\nsatisfies $\\nabla^{2}f(w)x=s$. The following is an immediate consequence\nof~\\citep[Theorem~6.4]{nesterov1998primal}.\n\\begin{lemma}[Short-Step Method]\n\\label{lem:short-step}The sequence in (\\ref{eq:hsd_ipm}) arrives\nat an iterate $(x,y,s,\\tau,\\theta,\\kappa)$ satisfying\n\\begin{equation}\n\\frac{x^{T}s+\\tau\\kappa}{n+1}\\le\\epsilon,\\qquad\\tau\\kappa\\ge\\gamma\\epsilon\\label{eq:short-step}\n\\end{equation}\nwith $\\gamma=\\frac{9}{10}$ in at most $O(\\sqrt{n}\\log(1/\\epsilon))$\niterations. \n\\end{lemma}\nFinally, the following result (adapted from~\\citep[Lemma~5.7.2]{de2000self})\nassures us that a feasible point satisfying the optimality condition\n(\\ref{eq:short-step}) will recover a solution to the original problem.\n","units":[{"raw_tex":"Consider directly applying a general-purpose interior-point method\nsolver, like SeDuMi~\\citep{sturm1999using}, SDPT3~\\citep{toh1999sdpt3},\nand MOSEK~\\citep{mosek2019}, to solve the standard-form semidefinite\nprogram to high accuracy:\n\\begin{equation}\n\\min_{X\\in\\S^{n}}\\;\\inner CX\\quad\\text{s.t.}\\;X\\succeq0,\\quad\\inner{A_{i}}X\\le b_{i}\\;\\text{for all }i\\in\\{1,2,\\dots,m\\}.\\tag{SDP}\\label{eq:sdp}\n\\end{equation}\nHere, $\\S^{n}$ denotes the set of $n\\times n$ real symmetric matrices\nwith inner product $\\inner{A_{i}}X=\\tr(A_{i}X)$, and $X\\succeq0$\nmeans that $X$ is positive semidefinite. At each iteration, the $n\\times n$\nmatrix variable $X$ is generally fully dense, even when problem data\n$C,A_{1},\\dots,A_{m}\\in\\S^{n}$ and $b_{1},\\dots,b_{m}\\in\\R$ are\nsparse. The per-iteration cost of the solver is usually at least $\\Omega(n^{3})$\ntime, which in practice limits the value of $n$ to no more than a\nfew thousand.\n","start":7013,"end":7929},{"raw_tex":"In order for chordal conversion to be fast, a well-known \\emph{necessary}\ncondition is for the underlying aggregate sparsity graph, which is\ndefined $G=(V,E)$ with\n\\begin{gather}\nV=\\{1,2,\\dots,n\\},\\quad E=\\spar(C)\\cup\\spar(A_{1})\\cup\\cdots\\cup\\spar(A_{m}),\\label{eq:Edef}\\\\\n\\text{where }\\spar(C)\\equiv\\{(i,j):C[i,j]\\ne0\\text{ for }i>j\\},\\nonumber \n\\end{gather}\nto be ``tree-like'' with a small\\emph{ treewidth} $\\tw(G)\\ll n$.\nWe defer a formal definition of the treewidth to \\defref{tw}, and\nonly note that the value of $\\omega$ is lower-bounded as $\\omega\\ge1+\\tw(G)$.\nIn other words, while $\\omega$ can be decreased by symmetrically\nreordering the columns and rows of the data matrices, as in $C\\gets\\Pi C\\Pi^{T}$\nand $A_{i}\\gets\\Pi A_{i}\\Pi^{T}$ for some permutation matrix $\\Pi$,\nactually achieving $\\omega\\ll n$ is possible only if $\\tw(G)\\ll n$.\nHowever, as illustrated by \\exaref{diag}, even on a graph with $\\tw(G)=0$,\nand even when using the optimal $\\omega=1$, chordal conversion may\nstill not be fast.\n","start":11820,"end":12833},{"raw_tex":"In this paper, we show that a \\emph{sufficient} condition for chordal\nconversion to be fast is for a supergraph $\\o G\\supseteq G$, that\nadditionally captures the correlation between constraints, to also\\footnote{Note that $\\tw(\\o G)\\ge\\tw(G)$ holds by virtue of $\\o G\\supseteq G$.}\nbe ``tree-like'' with a small treewidth $\\tw(\\o G)\\ll n$. Concretely,\nwe construct the \\emph{extended} aggregate sparsity graph $\\o G=(V,\\overline{E})$\nby forcing each constraint matrix $A_{i}$ to be its own clique in\n$G$:\n\\begin{gather}\nV=\\{1,2,\\dots,n\\},\\quad\\overline{E}=\\spar(C)\\cup\\clique(A_{1})\\cup\\cdots\\cup\\clique(A_{m})\\label{eq:Ebardef}\\\\\n\\text{where }\\clique(A)=\\{(i,j):A[i,k]\\ne0\\text{ or }A[k,j]\\ne0\\text{ for some }k\\}.\\nonumber \n\\end{gather}\nThis is the union between $G$ and the constraint intersection graph~\\citep{fulkerson1965incidence}\n(or the dual graph~\\citep{dong2021nearly} or the correlative sparsity~\\citep{waki2006sums,kobayashi2008correlative})\nfor the rank-1 instance of (\\ref{eq:sdp}). In other words, we add\na new edge $(i,j)$ to $E$ whenever $x_{i}$ and $x_{j}$ appear\ntogether in a common constraint $x^{T}A_{k}x\\le b_{k}$ for some $k$.\nThe fact that this contributes $\\clique(A_{k})\\subseteq\\o E$ reflects\nthe reality that each $x^{T}A_{k}x\\le b_{k}$ densely couples all\naffected elements of $x$ together, forcing them to be optimized simultaneously.\nIn contrast, the cost $x^{T}Cx$ can be optimized sequentially over\nthe elements of $x$, which is why $\\clique(C)$ is absent.\n","start":12834,"end":14326},{"raw_tex":"Write $\\R^{m\\times n}$ as the set of $m\\times n$ matrices with real\ncoefficients, with associated matrix inner product $\\inner AB=\\tr A^{T}B$\nand norm $\\|A\\|_{F}=\\sqrt{\\inner AA}$. Write $\\S^{n}\\subseteq\\R^{n\\times n}$\nas the set of $n\\times n$ real symmetric matrices, meaning that $X=X^{T}$\nholds for all $X\\in\\S^{n}$, and write $\\S_{+}^{n}\\subseteq\\S^{n}$\nas the set of symmetric positive semidefinite matrices. Write $\\R_{+}^{n}\\subseteq\\R^{n}$\nas the usual positive orthant.\n\nThe set of $n\\times n$ symmetric matrices with sparsity pattern $E$\ncan be defined as\n\\[\n\\S_{E}^{n}\\equiv\\left\\{ X\\in\\S^{n}:X[i,j]=X[j,i]=0\\quad\\text{for all }i\\ne j,\\quad(i,j)\\notin E\\right\\} .\n\\]\nConversely, the minimum sparsity pattern of a \\emph{symmetric} matrix\n$X\\in\\S^{n}$ is denoted\n\\[\n\\spar(X)\\equiv\\{(i,j):X[i,j]\\ne0,\\quad i\\ge j\\}.\n\\]\nWe also write $\\spar(M)\\equiv\\spar(M+M^{T})$ for a nonsymmetric matrix\n$M$ where there is no confusion. We write $\\proj_{E}(M)\\equiv\\arg\\min_{X\\in\\S_{E}^{n}}\\|M-X\\|_{F}$\nas the projection of $M\\in\\S^{n}$ onto the sparsity pattern $E$. \n\nWe define the dense sparsity pattern induced by $J\\subseteq\\{1,2,\\dots,n\\}$\nas follows\n\\[\n\\clique(J)=\\{(i,j):i,j\\in J,\\quad i\\ge j\\}.\n\\]\nWe also define the vertex support of a possibly nonsymmetric matrix\n$M$ as the following \n\\[\n\\supp(M)=\\{i:M[i,j]\\ne0\\quad\\text{for some }j\\}.\n\\]\nWe write $\\clique(A)\\equiv\\clique(\\supp(A))$ where there is no confusion.\nThis notation is motivated by the fact that $\\spar(A)\\subseteq\\clique(A)$\nfor $A\\in\\S^{n}$, and $\\spar(PDP^{T})\\subseteq\\clique(P)$ for $P\\in\\R^{n\\times d}$\nand dense $D\\in\\S^{d}$. \n\nLet $F$ be a sparsity pattern of order $n$ that \\emph{contains all\nof its diagonal elements}, as in $F\\supseteq\\{(i,i):i\\in\\{1,2,\\dots,n\\}\\}$.\nIn this case, $\\dim(\\S_{F}^{n})=|F|$ holds exactly, so we define\na \\emph{symmetric vectorization} operator $\\svec_{F}:\\S_{F}^{n}\\to\\R^{|F|}$\nto implement an isometry with the usual Euclidean space, as in \n\\[\n\\inner{\\svec_{F}(X)}{\\svec_{F}(Y)}=\\inner XY\\text{ for all }X,Y\\in{\\S_{F}^{n}}.\n\\]\nWe will explicitly require $\\svec_{F}(\\cdot)$ to be defined according\nto a column-stacking construction\n\\[\n\\svec_{F}(X)=(x_{j})_{j=1}^{n}\\quad\\text{where }x_{j}=\\left(X[i,i],\\quad\\sqrt{2}\\cdot(X[i,j]:i>j,\\quad(i,j)\\in F)\\right).\n\\]\nWe also define a companion indexing operator $\\idx_{F}(\\cdot,\\cdot)$\nto index elements of the vectorization $x=\\svec_{F}(X)$:\n\\begin{gather*}\nx[\\idx_{F}(i,i)]=X[i,i]\\quad\\text{for all }i\\in\\{1,2,\\dots,n\\},\\\\\nx[\\idx_{F}(i,j)]=\\sqrt{2}X[i,j]\\quad\\text{for all }i>j,\\quad(i,j)\\in F.\n\\end{gather*}\nAs we will see later, the correctness of our proof crucially relies\non the fact that $\\idx_{F}(\\cdot,\\cdot)$ implements a \\emph{raster\nordering} over the elements of $F$. \n","start":29321,"end":32057},{"raw_tex":"To solve $Sx=b$ with $S\\succ0$ via Cholesky factorization, we first\ncompute the lower-triangular Cholesky factor $L=\\chol(S)$ according\nto the following recursive rule \n\\[\n\\chol\\left(\\begin{bmatrix}\\alpha & b^{T}\\\\\nb & D\n\\end{bmatrix}\\right)=\\begin{bmatrix}\\sqrt{\\alpha} & 0\\\\\n\\frac{1}{\\sqrt{\\alpha}}b & \\chol\\left(D-\\frac{1}{\\alpha}bb^{T}\\right)\n\\end{bmatrix},\\quad\\chol(\\alpha)=\\sqrt{\\alpha},\n\\]\nand then solve two triangular systems $Ly=b$ and $L^{T}x=y$ via\nback-substitution. If $S$ is sparse, then $L=\\chol(S)$ may also\nbe sparse. The sparsity pattern of $L$ can be directly computed from\nthe sparsity pattern of $S$, and without needing to examine the numerical\nvalues of its nonzeros.\n\\begin{definition}[Symbolic Cholesky]\n\\label{def:symbchol}The \\emph{symbolic Cholesky factor} $\\chol(E)$\nof a sparsity pattern $E$ of order $n$ is defined as $\\chol(E)\\equiv E_{n+1}$\nwhere $E_{1}=E$ and\n\\[\nE_{k+1}=E_{k}\\cup(k,k)\\cup\\{(i,j):(i,k)\\in E_{k},\\quad(j,k)\\in E_{k}\\text{ for }i>j>k\\}.\n\\]\n\\end{definition}\nOne can verify that $\\chol(\\spar(S))=\\spar(\\chol(S))$. Note that\n$\\chol(E)$ can be computed from $E$ in $O(|\\chol(E)|)$ time and\nmemory~\\citep[Theorem 5.4.4]{george1981computer}. The efficiency\nof a sparsity-exploiting algorithm for factorizing $L=\\chol(S)$ and\nsolving $Ly=b$ and $L^{T}x=y$ is determined by the \\emph{frontsize}\nof the sparse matrix $S$. \n\\begin{definition}[Frontsize]\n\\label{def:frontsize}The \\emph{frontsize} $\\omega(E)$ of a sparsity\npattern $E$ is defined $\\max_{j}|\\col_{F}(j)|$ where $F=\\chol(E)$\nand $\\col_{F}(j)\\equiv\\{j\\}\\cup\\{i>j:(i,j)\\in F\\}$. The frontsize\n$\\omega(S)\\equiv\\omega(\\spar(S))$ of a symmetric matrix $S$ is the\nfrontsize of its minimum sparsity pattern. \n\\end{definition}\nIntuitively, the frontsize $\\omega(S)$ is the maximum number of nonzero\nelements in a single column of the Cholesky factor $L=\\chol(S)$.\nThe following is well-known~\\citep{george1981computer}.\n","start":32102,"end":34018},{"raw_tex":"The cost of solving $Sx=b$ with sparse $S\\succ0$ can usually be\nreduced by first permuting the rows and columns of the matrix symmetrically,\nand then solving $(\\Pi S\\Pi^{T})\\Pi x=\\Pi y$ for some permutation\nmatrix $\\Pi$. For $E=\\spar(S)$, we write $E_{\\Pi}\\equiv\\spar(\\Pi S\\Pi^{T})$\nto denote its permuted sparsity pattern. It is a fundamental result\nin graph theory and linear algebra that the problem of minimizing\nthe frontsize $\\omega(E_{\\Pi})$ over the set of $n\\times n$ permutation\nmatrices $\\Pi\\in\\Perm(n)$ is the same problem as computing the treewidth\nof the graph $G=(V,E)$.\n\n\\begin{definition}[Treewidth]\n\\label{def:tw}A tree decomposition of a graph $G=(V,E)$ is a pair\n$(\\{J_{j}\\},T)$ in which each \\emph{bag} $J_{j}\\subseteq V$ is a\nsubset of vertices and $T$ is a tree such that:\n\\begin{itemize}\n\\item (Vertex cover) $\\bigcup_{j}J_{j}=V$;\n\\item (Edge cover) $\\bigcup_{j}(J_{j}\\times J_{j})\\supseteq E$;\n\\item (Running intersection) $J_{i}\\cap J_{j}\\subseteq J_{k}$ for every\n$k$ on the path of $i$ to $j$ on $T$.\n\\end{itemize}\nThe \\emph{width} of the tree decomposition is $\\max_{j}|J_{j}|-1$.\nThe \\emph{treewidth} of $G$, denoted $\\tw(G)$, is the minimum width\nover all valid tree decompositions on $G$.\n\\end{definition}\nThe connection is an immediate corollary of the following result,\nwhich establishes an equivalence between tree decompositions and the\nsparsity pattern of Cholesky factors. \n\\begin{proposition}[Perfect elimination ordering]\nGiven a sparsity pattern $E$ of order $n$, let $G=(V,E)$ where\n$V=\\{1,2,\\dots,n\\}$. For every tree decomposition of $G$ with width\n$\\tau$, there exists a perfect elimination ordering $\\Pi\\in\\Perm(n)$\nsuch that $\\omega(E_{\\Pi})=1+\\tau$. \n\\end{proposition}\nWe defer a proof to the texts~\\citep{george1981computer,vandenberghe2015chordal},\nand only note that, given a tree decomposition of width $\\tau$, the\ncorresponding perfect elimination ordering $\\Pi$ can be found in\n$O((1+\\tau)\\cdot n)$ time. \n","start":35337,"end":37297},{"raw_tex":"The basic approach for solving an SDP using a general-purpose solver\nis to reformulate the problem into the primal or the dual of the standard-form\nlinear conic program \n\\[\n\\min_{x\\in{\\R^{q}}}\\{\\c^{T}x:\\A x={\\b},x\\in\\K\\}\\ge\\max_{y\\in{\\R^{p}}}\\{\\b^{T}y:\\c-\\A^{T}y\\in{\\K_{*}}\\}\n\\]\n where the data are the matrix $\\A\\in\\R^{p\\times q}$, vectors $\\b\\in\\R^{p}$\nand $\\c\\in\\R^{q}$, and the problem closed convex cone $\\K\\subseteq\\R^{q}$,\nand the notation $\\K_{*}$ means the dual cone of $\\K$. We specify\nthe following basic assumptions on this problem to ensure that it\ncan be solved in polynomial time using a self-dual embedding~\\citep{de2000self}.\nBelow, we denote $\\one_{\\K}$ as the identity element on the cone\n$\\K$, and recall that every semidefinite cone is self-dual $\\K=\\K_{*}$. \n\\begin{definition}[Standard-form SDP]\n\\label{def:clp}We say that the problem data $\\A\\in\\R^{p\\times q},$\n$\\b\\in\\R^{p},$ $\\c\\in\\R^{q},$ and $\\K\\subseteq\\R^{q}$ describe\nan SDP in $(n,\\omega)$-\\emph{standard form} if:\n\\begin{enumerate}\n\\item (Dimensions) The cone $\\K=\\svec(\\S_{+}^{\\omega_{1}})\\times\\cdots\\times\\svec(\\S_{+}^{\\omega_{\\ell}})$\nis the Cartesian product of semidefinite cones whose orders $\\omega_{1},\\omega_{2},\\dots,\\omega_{\\ell}$\nsatisfy \n\\[\n\\omega=\\max_{i}\\omega_{i},\\qquad n=\\sum_{i=1}^{\\ell}\\omega_{i},\\qquad{q}=\\frac{1}{2}\\sum_{i=1}^{\\ell}\\omega_{i}(\\omega_{i}+1).\n\\]\n\\item (Linear independence) $\\A^{T}y=0$ holds if and only if $y=0$. \n\\item (Strong duality is attained) There exist a choice of $x^{\\star},y^{\\star}$\nthat satisfy\n\\begin{gather*}\n\\A x^{\\star}=\\b,\\quad x^{\\star}\\in\\K,\\quad\\c-\\A^{T}y^{\\star}\\in\\K,\\quad\\c^{T}x^{\\star}=\\b^{T}y^{\\star}.\n\\end{gather*}\n\\end{enumerate}\n\\end{definition}\n%\n\\begin{definition}[General-purpose solver]\n\\label{def:ipm}We say that $\\IPM$ implements a \\emph{general-purpose\nsolver} if it satisfies the following conditions\n\\begin{enumerate}\n\\item (Iteration count) Given data $(\\A,\\b,\\c,\\K)$ in $(n,\\omega)$-standard-form,\ncalling $(x,y)=\\IPM(\\epsilon,\\A,\\b,\\c,\\K)$ yields iterates $(x,y)\\in\\K\\times{\\R^{p}}$\nthat satisfy the following, in at most $O(\\sqrt{n}\\log(1/\\epsilon))$\niterations\n\\[\n\\|\\A x-\\b\\|\\le\\epsilon,\\quad{\\c-\\A^{T}y+\\epsilon\\cdot\\one_{\\K}\\in\\K,}\\quad\\c^{T}x-\\b^{T}y\\le\\epsilon\\cdot n.\n\\]\n\\item (Per-iteration costs) Each iteration costs an overhead of $O(\\omega^{2}n)$\ntime and $O(\\omega n)$ memory, plus the cost of solving $O(1)$ instances\nof the \\emph{Schur complement equation} \n\\[\n\\A\\nabla^{2}f(w)\\A^{T}\\Delta y=r,\\qquad f(w)=-\\log\\det(w)\n\\]\nby forming $\\Hess=\\A\\nabla^{2}f(w)\\A^{T}$, factoring $\\L=\\chol(\\Pi\\Hess\\Pi^{T})$\nand then solving $\\L z={\\Pi}r$ and $\\L^{T}(\\Pi\\Delta y)=z$. Here,\nthe fill-reducing permutation $\\Pi$ is required to be no worse than\nthe natural ordering, as in $\\omega(\\Pi\\Hess\\Pi^{T})\\le\\omega(\\Hess)$.\n\\end{enumerate}\n\\end{definition}\nNote that \\defref{ipm} is rigorously satisfied by SeDuMi~\\citep{sturm1999using,sturm2002implementation},\nSDPT3~\\citep{toh1999sdpt3,tutuncu2003solving}, and MOSEK~\\citep{andersen2000mosek,andersen2003implementing,mosek2019}.\nGiven that the correctness of our overall claims crucially depends\non the characterization in \\defref{ipm}, we state a concrete interior-point\nmethod in \\appref{ipm} that implements these specifications. \n","start":37976,"end":41225},{"raw_tex":"\\begin{algorithm}[t]\n\\caption{\\label{alg:myalg}Chordal conversion}\n\n\\textbf{Input.} Accuracy parameter $\\epsilon>0$, problem data $C,A_{1},A_{2},\\dots,A_{m}\\in\\S^{n}$,\n$b\\in\\R^{m}$, fill-reducing permutation $\\Pi$.\n\n\\textbf{Output.} Approximate solutions $U\\in\\R^{n\\times\\omega}$ and\n$v\\in\\R^{m}$ to the following primal-dual pair:\n\\[\n\\min_{U}\\left\\{ \\inner C{UU^{T}}:\\inner{A_{i}}{UU^{T}}\\le b_{i}\\text{ for all }i\\right\\} \\ge\\max_{v\\le0}\\left\\{ \\inner bv:{\\textstyle \\sum_{i}}v_{i}A_{i}\\preceq C\\right\\} .\n\\]\n\n\\textbf{Algorithm.}\n\\begin{enumerate}\n\\item (Symbolic factorization) Pre-order all data matrices $\\tilde{A}_{i}=\\Pi A_{i}\\Pi^{T}$\nand $\\tilde{C}=\\Pi C\\Pi^{T}$. Compute the permuted aggregate sparsity\npattern $E=\\spar(\\tilde{C})\\cup\\bigcup_{i}\\spar(\\tilde{A}_{i})$,\nits lower-triangular symbolic Cholesky factor $F=\\chol(E)$, and define\nthe following\n\\[\nJ_{j}=\\col_{F}(j)\\equiv\\{j\\}\\cup\\{i>j:(i,j)\\in F\\},\\quad\\omega_{j}\\equiv|J_{j}|,\\quad\\omega\\equiv\\max_{j}\\omega_{j}.\n\\]\n\\item (Numerical solution) Call $(x,y)=\\IPM(\\epsilon,\\A,\\b,\\c,\\K)$ where\n$\\IPM$ is a general-purpose solver (\\defref{ipm}), and the problem\ndata $\\A,\\b,\\c,\\K$ implement the following\n\\[\nY^{\\star}=\\arg\\max_{Y\\in\\S_{F}^{n}}\\inner{-\\tilde{C}}Y\\text{ s.t. }\\begin{array}{cc}\nb_{i}-\\inner{\\tilde{A}_{i}}Y\\ge0 & \\quad\\text{for all }i\\in\\{1,2,\\dots,m\\}\\\\\nY[J_{j},J_{j}]\\succeq0 & \\quad\\text{for all }j\\in\\{1,2,\\dots,n\\}\n\\end{array}\n\\]\nas an instance of $y^{\\star}=\\arg\\max_{y}\\{\\b^{T}y:\\c-\\A^{T}y\\in\\K\\}$\nwith $y^{\\star}=\\svec_{F}(Y^{\\star})$.\n\\item (Back substitution) Recover $Y\\in\\S_{F}^{n}$ from $y=\\svec_{F}(Y)$,\nand compute $\\delta=-\\min_{j}\\{0,\\lambda_{\\min}(Y[J_{j},J_{j}])\\}$.\nSolve the positive semidefinite matrix completion\n\\[\n\\text{find }\\tilde{U}\\in\\R^{n\\times\\omega}\\text{ such that }(\\tilde{U}\\tilde{U}^{T})[J_{j},J_{j}]=Y[J_{j},J_{j}]+\\delta I\\text{ for all }j\n\\]\nusing \\citep[Algorithm~2]{sun2015decomposition}. Output $v=-(x_{i})_{i=1}^{m}$\nand $U=\\Pi^{T}\\tilde{U}$.\n\\end{enumerate}\n\\end{algorithm}\n \\algref{myalg} summarizes the standard implementation of chordal\nconversion, which is known as the ``d-space conversion method using\nbasis representation'' in \\citet{kim2011exploiting}. Our only modification\nis to recover $X=UU^{T}$ from $Y=\\proj_{F}(X)$ in Step 3 using the\n\\emph{low-rank} chordal completion~\\citep[Theorem~1.5]{dancis1992positive},\ninstead of the \\emph{maximum determinant} chordal completion~\\citep[Theorem~2]{grone1984positive}\nas originally proposed by \\citet{fukuda2001exploiting}. We note that\nboth recovery procedures have the same complexity of $O(\\omega^{3}n)$\ntime and $O(\\omega^{2}n)$ memory, but the former puts $X$ in a more\nconvenient form~\\citep{sun2015decomposition}.\n","start":41250,"end":43949},{"raw_tex":"To state the Schur complement sparsity $E^{(2)}$ explicitly, note\nthat the problem data $(\\A,\\b,\\c,\\K)$ in Step 2 of \\algref{myalg}\nare \\begin{subequations}\\label{eq:AKdef}\n\\begin{equation}\n\\begin{array}{c}\n\\A=[\\svec_{F}(A_{1}),\\dots,\\svec_{F}(A_{m}),-\\P_{1},\\dots,-\\P_{n}],\\\\\n\\b=-\\svec_{F}(C),\\qquad\\c=(b,0),\\\\\n\\K=\\R_{+}^{m}\\times\\svec(\\S_{+}^{\\omega_{1}})\\times\\svec(\\S_{+}^{\\omega_{2}})\\times\\cdots\\times\\svec(\\S_{+}^{\\omega_{n}})\n\\end{array}\\label{eq:AKdef-1}\n\\end{equation}\nwhere $\\omega_{j}\\equiv|\\col_{F}(j)|$ and each $\\P_{j}$ is implicitly\ndefined to satisfy \n\\begin{equation}\n\\P_{j}^{T}\\svec_{F}(Y)=\\svec(Y[\\col_{F}(j),\\col_{F}(j)])\\quad\\text{for all }Y\\in\\S_{F}^{n}.\\label{eq:AKdef-2}\n\\end{equation}\n\\end{subequations}The resulting Schur complement matrix reads\n\\begin{gather}\n\\A\\nabla^{2}f(w)\\A^{T}=\\sum_{i=1}^{m}d_{i}\\svec_{F}(A_{i})\\svec_{F}(A_{i})^{T}+\\sum_{j=1}^{n}\\P_{j}\\D_{j}\\P_{j}^{T}\\label{eq:Hessdef}\\\\\n\\text{where }d_{i}=w_{i}^{-2},\\qquad\\D_{j}\\svec(X_{j})=\\svec(W_{j}^{-1}X_{j}W_{j}^{-1})\\nonumber \n\\end{gather}\nand $w=(w_{1},\\dots,w_{m},\\svec(W_{1}),\\dots,\\svec(W_{n}))\\in\\interior(\\K)$\nis a scaling point. The associated sparsity pattern, aggregated over\nall possible choices of scaling $w$, is as follows\n\\begin{align}\nE^{(2)} & =\\{(i,j):(\\A\\nabla^{2}f(w)\\A^{T})[i,j]\\ne0\\text{ for some }w\\in\\interior(\\K)\\}\\nonumber \\\\\n & =\\left(\\bigcup_{i=1}^{m}\\spar(a_{i}a_{i}^{T})\\right)\\cup\\left(\\bigcup_{j=1}^{n}\\spar(\\P_{j}\\D_{j}\\P_{j}^{T})\\right)\\nonumber \\\\\n & =\\left(\\bigcup_{i=1}^{m}\\clique(\\supp(a_{i}))\\right)\\cup\\left(\\bigcup_{j=1}^{n}\\clique(\\supp(\\P_{j}))\\right)\\label{eq:E2def}\n\\end{align}\nwhere we have written $a_{i}=\\svec_{F}(A_{i})$. The result below\nsays that if $\\omega(\\o E)=O(1)$, then $\\omega(\\o E^{(2)})=O(1)$. \n","start":44543,"end":46292},{"raw_tex":"Let us now give an end-to-end complexity guarantee for \\algref{myalg}.\nWe will need the following assumption to ensure that the data $(\\A,\\b,\\c,\\K)$\npreviously defined in (\\ref{eq:AKdef}) specifies an SDP in $(N,\\omega)$-standard\nform, where $N=m+\\omega n$ and $\\omega\\equiv\\omega(E)$. \n\n\\begin{assume}[Strong duality is attained]\\label{asm:strict}There\nexists a primal-dual pair $X^{\\star}\\succeq0$ and $v^{\\star}\\le0$\nthat are feasible $\\inner{A_{i}}{X^{\\star}}\\le b_{i}$ for all $i$\nand $\\sum_{i}v_{i}^{\\star}A_{i}\\preceq C$ and coincide in their objectives\n$\\inner C{X^{\\star}}=\\inner b{v^{\\star}}$.\n\n\\end{assume}\n","start":50696,"end":51314},{"raw_tex":"$E$ of order $n$ can be viewed as the edge set of an undirected\ngraph $G=(V,E)$ on vertices $V=\\{1,2,\\dots,n\\}$. The underlying\nprinciple behind our proof is the fact that the frontsize is monotone\nunder the subgraph relation: if $G'=(V',E')$ is a subgraph of $G=(V,E)$,\nthen $\\omega(E)\\ge\\omega(E')$. \n\nTo state this formally, we denote the sparsity pattern induced by\na subset of vertices $U=\\{u_{1},u_{2},\\dots,u_{p}\\}\\subseteq V$ as\nfollows\n\\[\nE[U]\\equiv\\{(i,j):(u_{i},u_{j})\\in E\\}\\text{ where }u_{1}<u_{2}<\\cdots<u_{p}.\n\\]\nNote that we always sort the elements of $U$. Our definition is made\nso that if $E=\\spar(X)$, then $E[U]=\\spar(X[U,U])$, without any\nreordering of the rows and columns. \n","start":57148,"end":57847},{"raw_tex":"Our proof of \\lemref{quadlift} is based on a connection between\nzero-fill sparsity patterns, for which sequential Gaussian elimination\nresults in no additionally fill-in, and a ``sorted'' extension of\nthe running intersection property. \n\\begin{definition}[ZF]\nThe sparsity pattern $F$ is said to be\\emph{ zero-fill} if $F=\\chol(F)$.\n\\end{definition}\nEquivalently, if $F$ is zero-fill, then $(i,k)\\in F$ and $(j,k)\\in F$\nimplies $(i,j)\\in F$ for $i>j>k$ via the definition of the symbolic\nCholesky factor (\\defref{symbchol}). \n\n\\begin{definition}[RIP]\n\\label{def:rip}The sequence of subsets $J_{1},J_{2},\\dots,J_{\\ell}$\nwith $J_{j}\\subset\\N$ satisfies the \\emph{sorted running intersection\nproperty} if there exists a parent pointer $\\p:\\{1,2,\\dots,\\ell-1\\}\\to\\{2,3,\\dots,\\ell\\}$\nsuch that the following holds for all $1\\le j<\\ell$:\n\\begin{gather*}\n\\p(j)>j,\\quad J_{\\p(j)}\\supseteq J_{j}\\cap(J_{j+1}\\cup J_{j+2}\\cup\\cdots\\cup J_{\\ell}),\\quad\\min\\{J_{\\p(j)}\\}>\\max\\{J_{j}\\backslash J_{\\p(j)}\\}.\n\\end{gather*}\n\\end{definition}\nThe symbolic Cholesky factor $F=\\chol(E)$ for a sparsity pattern\n$E$ is the canonical example of a zero-fill sparsity pattern. In\nturn, the corresponding column sets $J_{j}=\\col_{F}(j)$ are the canonical\nexample of sequence of subsets that satisfy the sorted version of\nthe running intersection property.\n","start":61164,"end":62493},{"raw_tex":"Our proof is based on the fact that the ``lifted'' sparsity pattern\n$F^{(2)}$ can be constructed as $F^{(2)}=\\bigcup_{k=1}^{n}\\clique(J_{k}^{(2)})$\nwith respect to the following ``lifted'' index sets\n\\begin{gather}\nJ_{k}^{(2)}\\equiv\\idx_{F}(\\clique(J_{k}))=\\{\\idx_{F}(i,j):i,j\\in J_{k},\\quad i\\ge j\\}.\\label{eq:Jk2_def}\n\\end{gather}\nWe need to show that, if the original index sets $J_{1},J_{2},\\dots,J_{k}$\nsatisfy the running intersection property, then the lifted index sets\n$J_{1}^{(2)},J_{2}^{(2)},\\dots,J_{k}^{(2)}$ will inherit the running\nintersection property. Our key insight is that the index operator\n$\\idx_{F}$ implements a \\emph{raster ordering}.\n\\begin{lemma}[Raster ordering]\nThe ordering $\\idx_{F}:F\\to\\N$ satisfies the following, for all\n$(i,j)\\in F$ with $i\\ge j$ and $(i',j')\\in F$ with $i'\\ge j'$:\n\\begin{itemize}\n\\item If $j>j'$, then $\\idx(i,j)>\\idx(i',j')$ holds. \n\\item If $j=j'$ and $i>i'$, then $\\idx(i,j)>\\idx(i',j')$ holds. \n\\end{itemize}\n\\end{lemma}\n","start":63638,"end":64618},{"raw_tex":"We will need some additional notation. For $V=\\{1,2,\\dots,n\\}$ and\n$J\\subseteq V$, we denote the subset of $J$ induced by the elements\nin $U=\\{u_{1},u_{2},\\dots,u_{p}\\}\\subseteq V$ as follows\n\\[\nJ[U]\\equiv\\{i:u_{i}\\in J\\}\\text{ where }u_{1}<u_{2}<\\cdots<u_{p}.\n\\]\nOur definition is made so that if $E=\\clique(J)$, then $E[U]=\\clique(J[U])$.\n","start":72588,"end":72929},{"raw_tex":"Given data $C,A_{1},\\dots,A_{m}\\in\\S^{n}$ and $b\\in\\R^{m}$, define\n$(\\A,\\b,\\c,\\K)$ as in (\\ref{eq:AKdef}). In this section, we verify\nthat $(\\A,\\b,\\c,\\K)$ specifies an SDP that satisfies the regularity\nassumptions in \\defref{clp}. \n","start":99088,"end":99320},{"raw_tex":"We prove \\propref{general} by using the short-step method of Nesterov\nand Todd~\\citep[Algorithm~6.1]{nesterov1998primal} to solve the\nextended homogeneous self-dual embedding\\begin{subequations}\\label{eq:hsd}\n\\begin{alignat}{2}\n & \\min_{x,y,s,\\kappa,\\tau,\\theta}\\quad & (n+1)\\theta\\\\\n & \\text{s.t.} & \\begin{bmatrix}0 & +\\A^{T} & -\\c & -r_{d}\\\\\n-\\A & 0 & +\\b & -r_{p}\\\\\n+\\c^{T} & -\\b^{T} & 0 & -r_{c}\\\\\nr_{d}^{T} & r_{p}^{T} & r_{c} & 0\n\\end{bmatrix}\\begin{bmatrix}x\\\\\ny\\\\\n\\tau\\\\\n\\theta\n\\end{bmatrix}+\\begin{bmatrix}s\\\\\n0\\\\\n\\kappa\\\\\n0\n\\end{bmatrix} & =\\begin{bmatrix}0\\\\\n0\\\\\n0\\\\\nn+1\n\\end{bmatrix}\\label{eq:hsdfeas}\\\\\n &  & x,s\\in\\K,\\quad\\kappa,\\tau & \\ge0,\\label{eq:hsdcone}\n\\end{alignat}\n\\end{subequations}where $r_{d}=\\one_{\\K}-\\c$ and $r_{p}=-\\A\\one_{\\K}+\\b$\nand $r_{c}=1+\\c^{T}\\one_{\\K}$. Beginning at the strictly feasible,\nperfectly centered point in (\\ref{eq:hsdinit}) for $\\mu=1$:\n\\begin{gather}\n\\theta^{(0)}=\\tau^{(0)}=\\kappa^{(0)}=1,\\quad y^{(0)}=0,\\quad x^{(0)}=s^{(0)}=\\one_{\\K}.\\label{eq:hsdinit}\n\\end{gather}\n we take the following steps\\begin{subequations}\\label{eq:hsd_ipm}\n\\begin{align}\n\\mu^{+} & =\\left(1-\\frac{1}{15\\sqrt{n+1}}\\right)\\cdot\\frac{x^{T}s+\\tau\\kappa}{n+1},\\\\\n(x^{+},y^{+},s^{+},\\tau^{+},\\theta^{+},\\kappa^{+}) & =(x,y,s,\\tau,\\theta,\\kappa)+(\\Delta x,\\Delta y,\\Delta s,\\Delta\\tau,\\Delta\\theta,\\Delta\\kappa).\\nonumber \n\\end{align}\n\\end{subequations}along the search direction defined by the linear\nsystem~\\citep[Eqn. 9]{todd1998nesterov} \\begin{subequations}\\label{eq:hsd_nt}\n\\begin{align}\n\\begin{bmatrix}0 & +\\A^{T} & -\\c & -r_{p}\\\\\n-\\A & 0 & +\\b & -r_{d}\\\\\n+\\c^{T} & -\\b^{T} & 0 & -r_{c}\\\\\n+r_{p}^{T} & +r_{d}^{T} & +r_{c} & 0\n\\end{bmatrix}\\begin{bmatrix}\\Delta x\\\\\n\\Delta y\\\\\n\\Delta\\tau\\\\\n\\Delta\\theta\n\\end{bmatrix}+\\begin{bmatrix}\\Delta s\\\\\n0\\\\\n\\Delta\\kappa\\\\\n0\n\\end{bmatrix} & =0,\\\\\ns+\\Delta s+\\nabla^{2}f(w)\\Delta x+\\mu^{+}\\nabla f(x) & =0,\\\\\n\\kappa+\\Delta\\kappa+(\\kappa/\\tau)\\Delta\\tau-\\mu^{+}\\tau^{-1} & =0.\n\\end{align}\n\\end{subequations}where $f(w)=-\\log\\det w$ is the usual self-concordant\nbarrier function, and $w\\in\\interior(\\K)$ is the unique point that\nsatisfies $\\nabla^{2}f(w)x=s$. The following is an immediate consequence\nof~\\citep[Theorem~6.4]{nesterov1998primal}.\n\\begin{lemma}[Short-Step Method]\n\\label{lem:short-step}The sequence in (\\ref{eq:hsd_ipm}) arrives\nat an iterate $(x,y,s,\\tau,\\theta,\\kappa)$ satisfying\n\\begin{equation}\n\\frac{x^{T}s+\\tau\\kappa}{n+1}\\le\\epsilon,\\qquad\\tau\\kappa\\ge\\gamma\\epsilon\\label{eq:short-step}\n\\end{equation}\nwith $\\gamma=\\frac{9}{10}$ in at most $O(\\sqrt{n}\\log(1/\\epsilon))$\niterations. \n\\end{lemma}\nFinally, the following result (adapted from~\\citep[Lemma~5.7.2]{de2000self})\nassures us that a feasible point satisfying the optimality condition\n(\\ref{eq:short-step}) will recover a solution to the original problem.\n","start":102960,"end":105763}]},"statement":{"raw_tex":"\\begin{theorem}[Upper complexity]\n\\label{thm:total}Let the data $C,A_{1},\\dots,A_{m}\\in\\S^{n}$ and\n$b\\in\\R^{m}$ satisfy \\asmref{strict}. Given a tree decomposition\nof width $\\o{\\tau}$ for the extended aggregate sparsity graph $\\o G=(V,\\o E)$,\nwhere\n\\[\nV=\\{1,2,\\dots,n\\},\\qquad\\o E=\\spar(C)\\cup\\clique(A_{1})\\cup\\cdots\\cup\\clique(A_{m}),\n\\]\nset $\\Pi$ as the associated perfect elimination ordering. Then, \\algref{myalg}\noutputs $U\\in\\R^{n\\times\\o{\\omega}}$ and $v\\in\\R^{m}$ with $v\\le0$\nsuch that\n\\begin{align*}\n\\inner{A_{i}}{UU^{T}}-b_{i} & \\le\\epsilon\\text{ for all }i, & \\sum_{i=1}^{m}v_{i}A_{i}-C & \\preceq\\epsilon\\cdot I, & \\inner C{UU^{T}}-\\inner bv & \\le\\epsilon\\cdot N,\n\\end{align*}\nin $O(\\sqrt{N}\\log(1/\\epsilon))$ iterations, with per-iteration costs\nof $O(\\o{\\omega}^{4}\\cdot N)$ time and $O(\\o{\\omega}^{2}\\cdot N)$\nmemory, where $N=m+\\o{\\omega}\\cdot n$ and $\\o{\\omega}=1+\\o{\\tau}$.\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}[Upper complexity]\n\\label{thm:total}Let the data $C,A_{1},\\dots,A_{m}\\in\\S^{n}$ and\n$b\\in\\R^{m}$ satisfy \\asmref{strict}. Given a tree decomposition\nof width $\\o{\\tau}$ for the extended aggregate sparsity graph $\\o G=(V,\\o E)$,\nwhere\n\\[\nV=\\{1,2,\\dots,n\\},\\qquad\\o E=\\spar(C)\\cup\\clique(A_{1})\\cup\\cdots\\cup\\clique(A_{m}),\n\\]\nset $\\Pi$ as the associated perfect elimination ordering. Then, \\algref{myalg}\noutputs $U\\in\\R^{n\\times\\o{\\omega}}$ and $v\\in\\R^{m}$ with $v\\le0$\nsuch that\n\\begin{align*}\n\\inner{A_{i}}{UU^{T}}-b_{i} & \\le\\epsilon\\text{ for all }i, & \\sum_{i=1}^{m}v_{i}A_{i}-C & \\preceq\\epsilon\\cdot I, & \\inner C{UU^{T}}-\\inner bv & \\le\\epsilon\\cdot N,\n\\end{align*}\nin $O(\\sqrt{N}\\log(1/\\epsilon))$ iterations, with per-iteration costs\nof $O(\\o{\\omega}^{4}\\cdot N)$ time and $O(\\o{\\omega}^{2}\\cdot N)$\nmemory, where $N=m+\\o{\\omega}\\cdot n$ and $\\o{\\omega}=1+\\o{\\tau}$.\n\\end{theorem}","start":51314,"end":52220}]},"proof":{"raw_tex":"\\begin{proposition}[Sparse Cholesky factorization]\n\\label{prop:chol}Given $S\\in\\S^{n},$ $S\\succ0$, let $\\omega\\equiv\\omega(S)$.\nSparse Cholesky factorization factors $L=\\chol(S)$ in $\\Tim$ arithmetic\noperations and $M$ units of memory, where \n\\begin{align*}\n\\frac{1}{6}(\\omega-1)^{3}+n & \\le\\Tim\\le\\omega^{2}\\cdot n, & \\frac{1}{2}(\\omega-1)^{2}+n & \\le\\Mem\\le\\omega\\cdot n.\n\\end{align*}\n\\end{proposition}\n\n\\begin{proof}\nLet $\\omega_{j}\\equiv|\\col_{F}(j)|$. By inspection, $\\Tim=\\sum_{j=1}^{n}\\frac{1}{2}\\omega_{j}(\\omega_{j}+1)$\nand $\\Mem=\\sum_{j=1}^{n}\\omega_{j}$. The bounds follow by substituting\n$\\omega\\ge\\omega_{j+1}\\ge\\omega_{j}-1$. Indeed,\n\\[\n\\Mem=\\sum_{j=1}^{n}\\omega_{j}\\ge\\underbrace{\\omega+(\\omega-1)+\\cdots+1}_{\\omega\\text{ terms}}+\\underbrace{1+\\cdots+1}_{n-\\omega\\text{ terms}}=\\frac{1}{2}\\omega(\\omega+1)+(n-\\omega)\n\\]\nand similarly $\\Tim=\\sum_{j=1}^{n}\\frac{1}{2}\\omega_{j}(\\omega_{j}+1)\\ge\\frac{1}{6}\\omega(\\omega+1)(\\omega+2)+(n-\\omega).$\n\\qed\n\\end{proof}\n\n\\begin{theorem}[Frontsize of Schur complement sparsity]\n\\label{thm:front}Given $C,A_{1},A_{2},\\dots,A_{m}\\in\\S^{n}$, define\n$\\A,\\K$ as in (\\ref{eq:AKdef}), and define $E^{(2)}$ as in (\\ref{eq:E2def}).\nWe have \n\\[\n\\frac{1}{2}\\omega(\\omega+1)\\le\\omega(E^{(2)})\\le\\frac{1}{2}\\o{\\omega}(\\o{\\omega}+1)\n\\]\nwhere $\\omega\\equiv\\omega(E)$ and $\\o{\\omega}=\\omega(\\o E)$ and\n$E,\\o E$ are defined in (\\ref{eq:Edef}) and (\\ref{eq:Ebardef}).\nMoreover, if $E=\\o E$, then we also have $\\chol(E^{(2)})=E^{(2)}$. \n\\end{theorem}\n\n\\begin{corollary}[Cost of Schur complement equation]\n\\label{cor:normal}Given the data matrix $\\A$, scaling point $w\\in\\interior(\\K)$,\nand right-hand side $g$, define $\\Hess=\\A\\nabla^{2}f(w)\\A^{T}$ as\nin (\\ref{eq:E2def}). Suppose that all columns of $\\A$ are nonzero,\nand all scaling matrices $\\D_{j}$ are fully dense. Then, it takes\n$\\Tim$ arithmetic operations and $\\Mem$ units of storage to form\n$\\Hess$ and solve $\\Hess\\Delta y=g$, where\n\\begin{gather*}\n\\frac{1}{48}(\\omega-1)^{6}+m+n\\le\\Tim\\le4\\o{\\omega}^{4}\\cdot(m+\\omega n),\\\\\n\\frac{1}{8}(\\omega-1)^{4}+m+n\\le\\Mem\\le2\\o{\\omega}^{2}\\cdot(m+\\omega n),\n\\end{gather*}\nin which $\\omega\\equiv\\omega(E)$ and $\\o{\\omega}\\equiv\\omega(\\o E)$\nsatisfy $1\\le\\omega\\le\\o{\\omega}$.\n\\end{corollary}\n\n\\begin{proof}\nLet $a_{i}\\equiv\\svec_{F}(A_{i})$ and $\\omega_{j}\\equiv|\\col_{F}(j)|$.\nWe break the solution of $\\Hess\\Delta y=g$ into five steps:\n\\begin{enumerate}\n\\item (Input) It takes $\\Mem_{\\mathrm{data}}$ memory to state the problem\ndata, where $m+n\\le\\Mem_{\\mathrm{data}}\\le2\\o{\\omega}^{2}m+3\\omega^{2}n$.\nIndeed, $m+n\\le\\nnz(\\A)\\le\\o{\\omega}^{2}m+\\omega^{2}n$ because $1\\le\\nnz(a_{i})\\le\\o{\\omega}^{2}$\n(see Step 2 below) and $\\nnz(\\P_{j})=\\frac{1}{2}\\omega_{j}(\\omega_{j}+1)$.\nAlso, $m+n\\le\\nnz(w)\\le m+\\omega^{2}n$, and $n\\le\\nnz(g)\\le\\omega n$.\n\\item (Build LP part) It takes $\\Tim_{\\mathrm{LP}}$ time to build $\\sum_{i=1}^{m}d_{i}a_{i}a_{i}^{T}$,\nwhere $m\\le\\Tim_{\\mathrm{LP}}\\le2\\o{\\omega}^{4}m$ time and memory.\nThis follows from $1\\le\\nnz(a_{i})\\le\\o{\\omega}^{2}$, where the upper-bound\nis because $\\spar(a_{i}a_{i}^{T})\\subseteq E^{(2)}\\subseteq\\chol(E^{(2)})$,\nand that $\\omega(E^{(2)})$ is, by definition, the maximum number\nof nonzero elements in a single column of $\\chol(E^{(2)})$.\n\\item (Build SDP part) It takes $\\Tim_{\\mathrm{SDP}}$ time to build $\\sum_{j=1}^{n}\\P_{j}\\D_{j}\\P_{j}^{T}$,\nwhere $n\\le\\Tim_{\\mathrm{SDP}}\\le\\omega^{4}n$ time and memory. This\nfollows from $\\nnz(\\P_{j})=\\frac{1}{2}\\omega_{j}(\\omega_{j}+1)$,\nwhich implies $\\nnz(\\P_{j}\\D_{j}\\P_{j}^{T})=\\frac{1}{4}\\omega_{j}^{2}(\\omega_{j}+1)^{2}$\nfor a fully-dense $\\D_{j}\\succ0$. \n\\item (Factorization) It takes $\\Tim_{\\mathrm{fact}}$ time and $\\Mem_{\\mathrm{fact}}$\nmemory to factor $\\L=\\chol(\\Hess)$, where $\\frac{1}{48}(\\omega-1)^{6}+n\\le\\Tim_{\\mathrm{fact}}\\le\\o{\\omega}^{5}n$\nand $\\frac{1}{8}(\\omega-1)^{4}+n\\le\\Mem_{\\mathrm{fact}}\\le\\o{\\omega}^{3}n$.\nThe matrix $\\Hess$ has $|F|$ columns and rows, and frontsize $\\omega(E^{(2)})$.\nThe desired figures follow by substituting $\\frac{1}{2}\\omega^{2}\\le\\omega(E^{(2)})\\le\\o{\\omega}^{2}$\nand $n\\le|F|\\le\\omega n$ into \\propref{chol}.\n\\item (Back-substitution) It takes $\\Mem_{\\mathrm{fact}}$ time and memory\nto solve each of $\\L z=g$ and $\\L^{T}\\Delta y=z$ via triangular\nback-substitution. \n\\end{enumerate}\nThe overall runtime is cumulative, so $\\Tim=\\Tim_{\\mathrm{LP}}+\\Tim_{\\mathrm{SDP}}+\\Tim_{\\mathrm{fact}}+2\\Mem_{\\mathrm{fact}}$.\nThe overall memory use is $\\Mem=\\Mem_{\\mathrm{data}}+\\Mem_{\\mathrm{fact}}$,\nbecause the matrix $\\Hess$ can be constructed and then factored in-place.\n\\qed\n\\end{proof}\n\n\\begin{proof}One can verify that $(\\A,\\b,\\c,\\K)$ defined in (\\ref{eq:AKdef})\nspecifies an SDP in $(N,\\omega)$-standard form (see \\appref{regularity}\nfor the regularity conditions). Moreover, it follows from the monotonicity\nof the frontsize (\\propref{del}) that $\\omega\\equiv\\omega(E_{\\Pi})\\le\\omega(\\o E_{\\Pi})\\equiv\\o{\\omega}$.\nWe will track the cost of \\algref{myalg} step-by-step:\n\\begin{enumerate}\n\\item (Front-reducing permutation) Preordering $A_{i}\\gets\\Pi A_{i}\\Pi^{T}$\nand $C\\gets\\Pi C\\Pi^{T}$ the matrices cost $O(\\nnz(C)+\\sum_{i=1}^{m}\\nnz(A_{i}))=O(\\o{\\omega}^{2}N)$\ntime and memory. This follows from $\\nnz(C)\\le|F|\\le N$ and $\\nnz(A_{i})\\le\\o{\\omega}^{2}$. \n\\item (Conversion) Computing $F=\\chol(E)$ costs $O(|F|)=O(N)$ time and\nspace, where we note that $|F|\\le\\omega n$. \n\\item (Solution) Let $K=2\\cdot\\max\\{1,|\\tr(C)|,|\\tr(A_{1})|,\\dots,|\\tr(A_{m})|\\}$.\nAfter $O(\\sqrt{N}\\log(K/\\epsilon))$ iterations, we arrive at a primal\n$v\\le0$ and $V_{j}\\succeq0$ and dual point $Y\\in\\S_{F}^{n}$ satisfying\n\\begin{gather*}\n\\inner{A_{i}}Y-b_{i}\\le\\epsilon/K,\\quad Y[J_{j},J_{j}]\\succeq-(\\epsilon/K)I,\\\\\n{\\textstyle \\left\\Vert \\sum_{i}v_{i}A_{i}+\\sum_{j}P_{j}V_{j}P_{j}^{T}-C\\right\\Vert \\le\\epsilon/K,}\\\\\n\\inner CY-\\inner bv\\le N\\cdot(\\epsilon/K).\n\\end{gather*}\nEach iteration costs $O(\\o{\\omega}^{4}N)$ time and $O(\\o{\\omega}^{2}N)$\nmemory. This cost is fully determined by the cost of solving $O(1)$\ninstances of the Schur complement equation, which dominates the overhead\nof $O(\\omega^{3}n+\\nnz(\\A))=O(\\o{\\omega}^{2}N)$ time and memory.\n\\item (Recovery) Using the previously recovered $Y$, we recover $U$ such\nthat $\\Pi_{F}(UU^{T})=Y+\\delta I$ where $\\delta=-\\min_{j}\\{0,\\lambda_{\\min}(Y[J_{j},J_{j}])\\}\\le\\epsilon/K$.\nThis takes $O(\\omega^{3}n)=O(\\o{\\omega}^{2}N)$ time and $O(\\omega^{2}n)=O(\\o{\\omega}N)$\nmemory.\n\\item (Output) We output $U$ and $v$, and check for accuracy. It follows\nfrom $K\\ge2|\\tr(C)|$ that\n\\[\n\\inner C{UU^{T}}-\\inner CY=\\delta\\cdot\\tr(C)\\le\\epsilon\\cdot\\frac{\\tr(C)}{K}\\le\\epsilon\\cdot\\frac{|\\tr(C)|}{2|\\tr(C)|}\\le\\frac{1}{2}\\epsilon,\n\\]\nand from $K\\ge2$ and $N\\ge1$ that \n\\[\n\\inner C{UU^{T}}-\\inner bv\\le\\frac{N}{K}\\epsilon+\\frac{1}{2}\\epsilon=N\\epsilon\\cdot\\left(\\frac{1}{2}+\\frac{1}{2N}\\right)\\le N\\cdot\\epsilon.\n\\]\nSimilarly, it follows from $K\\ge2|\\tr(A_{i})|$ and $K\\ge2$ that\n$\\inner{A_{i}}{UU^{T}}-b_{i}\\le\\epsilon$. Finally, it follows from\n$v\\ge0$ and $V_{j}\\succeq0$ that\n\\[\n\\sum_{i}v_{i}A_{i}+\\sum_{j}P_{j}V_{j}P_{j}^{T}-C\\preceq\\epsilon\\cdot I\\implies\\sum_{i}v_{i}A_{i}-C\\preceq\\epsilon\\cdot I.\n\\]\n\\end{enumerate}\n\\qed \\end{proof}\n\n\\begin{proposition}[Subgraph monotonicity]\n\\label{prop:del}Let $E$ be a sparsity pattern of order $n$, and\nlet $U\\subseteq\\{1,2,\\dots,n\\}$. Then, for any sparsity pattern $D$\nof order $|U|$ that satisfies $E[U]\\supseteq D$, we have $\\chol(E)[U]\\supseteq\\chol(D)$,\nand therefore $\\omega(E)\\ge\\omega(D)$.\n\\end{proposition}\n\n\\begin{proof}\nIt is known that $(i,j)\\in\\chol(E)$ for $i>j$ holds if and only\nif there exists a path $(i,p_{1},p_{2},\\dots,p_{\\ell},j)$ whose edges\nare in $E$, and whose internal nodes are ordered $p_{1},p_{2},\\dots,p_{\\ell}<j<i$;\nsee e.g.~\\citep[Theorem~6.1]{vandenberghe2015chordal}. It immediately\nfollows this characterization that $\\chol(\\cdot)$ is monotone with\nrespect to the deletion of edges and isolated vertices: (1) if $D\\subseteq E$,\nthen $\\chol(D)\\subseteq\\chol(E)$; (2) we have $\\chol(E[U])=\\chol(E)[U]$\nfor $U=\\{1,2,\\dots,n\\}\\backslash v$ with isolated vertex $v$. Therefore,\n$\\chol(\\cdot)$ must also be monotone under general vertex and edge\ndeletions, because we can always delete edges to isolate a vertex\nbefore deleting it. \\qed\n\\end{proof}\n\n\\begin{lemma}[Quadratic lift]\n\\label{lem:quadlift}Let $E$ be an arbitrary sparsity pattern of\norder $n$. Define $F=\\chol(E)$ and the lifted sparsity pattern\n$F^{(2)}=\\bigcup_{k=1}^{n}\\clique(\\P_{k})$ in which each $\\P_{k}$\nis implicitly defined to satisfy $\\P_{k}^{T}\\svec_{F}(Y)=\\svec(Y[\\col_{F}(k),\\col_{F}(k)])$\nfor all $Y\\in\\S_{F}^{n}$. Then, we have $\\chol(F^{(2)})=F^{(2)}$\nand $\\omega(F^{(2)})=\\frac{1}{2}\\omega(E)[\\omega(E)+1]$.\n\\end{lemma}\n\n\\begin{lemma}[Sparsity overestimate]\n\\label{lem:ub}Let $\\o E$ and $E^{(2)}$ be the sparsity patterns\ndefined in (\\ref{eq:Ebardef}) and (\\ref{eq:E2def}). Define $\\o F=\\chol(\\o E)$\nand the lifted sparsity pattern $\\o F^{(2)}=\\bigcup_{k=1}^{n}\\clique(\\o{\\P}_{k})$\nin which each $\\o{\\P}_{k}$ is implicitly defined to satisfy $\\o{\\P}_{k}^{T}\\svec_{\\o F}(Y)=\\svec(Y[\\col_{\\o F}(k),\\col_{\\o F}(k)])$\nfor all $Y\\in\\S_{\\o F}^{n}$. Then, $E^{(2)}\\subseteq\\o F^{(2)}[V^{(2)}]$\nholds for $V^{(2)}=\\{\\idx_{\\overline{F}}(i,j):i,j\\in F\\}$.\n\\end{lemma}\n\n\\begin{proposition}[ZF$\\implies$RIP]\n\\label{prop:zf->rip}Let $F$ be a zero-fill sparsity pattern of\norder $n$. Then, the sequence of subsets $J_{1},J_{2},\\dots,J_{n}$\nwith $J_{j}=\\col_{F}(j)\\equiv\\{j\\}\\cup\\{i>j:(i,j)\\in F\\}$ satisfies\nthe sorted running intersection property. \n\\end{proposition}\n\n\\begin{proof}\nDefine $\\p(j)=\\min\\{i>j:i\\in J_{j}\\}$ if $|J_{j}|>1$ and $\\p(j)=n$\nif $|J_{j}|=1$. Clearly, $\\p(j)>j$ holds for all $1\\le j<n$. To\nprove $J_{\\p(j)}\\supseteq J_{j}\\cap\\bigcup_{w=j+1}^{\\ell}J_{w}$,\nlet $i\\in J_{j}\\cap J_{w}$ for some $w>j$. We prove that $i\\in J_{\\p(j)}$\nvia the following steps: \n\\begin{itemize}\n\\item We have $i>j$, because $i\\in J_{w}$ implies that $i\\ge w>j$. \n\\item If $i=\\p(j)$, then $i\\in J_{\\p(j)}$ by definition. \n\\item If $i>\\p(j)$, then $(i,j)\\in F$ and $(\\p(j),j)\\in F$ for $i>\\p(j)>j$\nimplies $(i,\\p(j))\\in F$, and hence $i\\in J_{\\p(j)}$. \n\\end{itemize}\nFinally, we prove $\\min\\{J_{\\p(j)}\\}>\\max\\{J_{j}\\backslash J_{\\p(j)}\\}$\nby noting that $\\max\\{J_{j}\\backslash J_{\\p(j)}\\}=j$ with our construction,\nand that $i=\\min J_{\\p(j)}$ must satisfy $i\\in J_{\\p(j)}$ and therefore\n$i\\ge\\p(j)>j$. \\qed\n\\end{proof}\n\n\\begin{lemma}\n\\label{lem:lift}Let the sequence of subsets $J_{1},J_{2},\\dots,J_{\\ell}$\nwith $J_{j}\\subset\\N$ satisfy the sorted running intersection property.\nThen, $J_{1}^{(2)},J_{2}^{(2)},\\dots,J_{\\ell}^{(2)}$ also satisfy\nthe sorted running intersection property. \n\\end{lemma}\n\n\\begin{proof}\nLet $\\p(\\cdot)$ denote the parent pointer that verifies the sorted\nrunning intersection property in $J_{1},J_{2},\\dots,J_{\\ell}$. We\nwill verify that $\\p(\\cdot)$ also proves the same property in $J_{1}^{(2)},J_{2}^{(2)},\\dots,J_{\\ell}^{(2)}$. \n\nFirst, to prove $J_{u}^{(2)}\\cap\\bigcup_{v=u+1}^{\\ell}J_{v}^{(2)}\\subseteq J_{\\p(u)}^{(2)},$\nlet $k\\in J_{u}^{(2)}\\cap J_{v}^{(2)}$ for $v>u$. The fact that\n$k\\in J_{u}^{(2)}=\\idx(\\clique(J_{u}))$ implies $k=\\idx(i,j)$ for\nsome $i,j$ such that $i,j\\in J_{u}$. Similarly, $k\\in J_{v}^{(2)}=\\idx(\\clique(J_{v}))$\nand the bijectivity of $\\idx$ on $F$ imply that the same $i,j$\nalso satisfy $i,j\\in J_{v}$, where we recall that $v>u$. We conclude\n$i,j\\in J_{u}\\cap J_{v}\\subseteq J_{\\p(u)}$ and therefore $k=\\idx(i,j)\\in J_{\\p(u)}^{(2)}$.\n\nNext, we prove $\\min\\{J_{\\p(u)}^{(2)}\\}>\\max\\{J_{u}^{(2)}\\backslash J_{\\p(u)}^{(2)}\\}$\nby establishing two claims:\n\\begin{itemize}\n\\item $\\min\\{J_{p(u)}^{(2)}\\}=\\idx(\\alpha,\\alpha)$ where $\\alpha=\\min J_{\\p(u)}$.\nFor any $\\idx(i,j)\\in J_{\\p(u)}^{(2)}$ where $i\\ge j$, we must have\n$i,j\\in J_{\\p(u)}$. It follows from the raster property that $\\idx(i,j)$\nis minimized with $i=j=\\min J_{\\p(u)}$.\n\\item $\\max\\{J_{u}^{(2)}\\backslash J_{\\p(u)}^{(2)}\\}=\\idx(\\beta,\\gamma)$\nwhere $\\beta=\\max\\{J_{u}\\}$ and $\\gamma=\\max\\{J_{u}\\backslash J_{\\p(u)}\\}$.\nWe partition $J_{u}$ into $N_{u}=J_{u}\\backslash J_{\\p(u)}$ and\n$A_{u}=J_{u}\\cap J_{\\p(u)}$. \n\\begin{itemize}\n\\item For any $\\idx(i,j)\\in J_{u}^{(2)}\\backslash J_{p(u)}^{(2)}$ where\n$i\\ge j$, we can have one of the following three cases: 1) $i\\in N_{u}$\nand $j\\in A_{u}$; 2) $i\\in A_{u}$ and $j\\in N_{u}$; or 3) $i\\in N_{u}$\nand $j\\in N_{u}$. \n\\item We observe that the first case $i\\in N_{u}$ and $j\\in A_{u}$ is\nimpossible. Indeed, applying $j\\ge\\min\\{J_{\\p(u)}\\}>\\max\\{J_{u}\\backslash J_{\\p(u)}\\}\\ge i$\nwould yield a contradiction with $i\\ge j$. \n\\item Taking the union of the two remaining cases yields $i\\in J_{u}=A_{u}\\cup N_{u}$\nand $j\\in N_{u}$. It follows from the raster property that $\\idx(i,j)$\nis maximized with $i=\\max J_{u}$ and $j=\\max N_{u}$.\n\\end{itemize}\n\\end{itemize}\nWith the two claims established, the hypothesis that $\\min\\{J_{\\p(u)}\\}>\\max\\{J_{u}\\backslash J_{\\p(u)}\\}$\nimplies that $\\alpha>\\gamma$, and therefore $\\idx(\\alpha,\\alpha)>\\idx(\\beta,\\gamma)$\nas desired. \\qed\n\\end{proof}\n\n\\begin{proposition}[RIP$\\implies$ZF]\n\\label{prop:rip->zf}Let $J_{1},J_{2},\\dots,J_{\\ell}$ with $\\bigcup_{j=1}^{\\ell}J_{j}=\\{1,2,\\dots,n\\}$\nsatisfy the sorted running intersection property. Then, $F=\\bigcup_{j=1}^{\\ell}\\clique(J_{j})$\nis zero-fill, and we have $\\omega(F)=\\max_{j}|J_{j}|$.\n\\end{proposition}\n\n\\begin{lemma}\n\\label{lem:colrip}Let $J_{1},J_{2},\\dots,J_{\\ell}$ with $\\bigcup_{j=1}^{\\ell}J_{j}=\\{1,2,\\dots,n\\}$\nsatisfy the sorted running intersection property. For every $j$-th\ncolumn in $F=\\bigcup_{j=1}^{\\ell}\\clique(J_{j})$, there exists some\n$J_{w}$ such that $\\col_{F}(j)\\subseteq J_{w}.$\n\\end{lemma}\n\n\\begin{proof}\nLet $V=\\{1,2,\\dots,n\\}$. For arbitrary $j\\in V$, denote $J_{w}$\nas the last subset in the sequence $J_{1},J_{2},\\dots,J_{\\ell}$ for\nwhich $j\\in J_{w}$. This choice must exist, because $V=\\bigcup_{k=1}^{\\ell}J_{k}$.\nFor every arbitrary $i\\in\\col_{F}(j)$, we will prove that $i\\in J_{w}$\nalso holds:\n\\begin{itemize}\n\\item There exists $u\\le w$ for which $i,j\\in J_{u}$. Indeed, $(i,j)\\in F$\nand $F=\\bigcup_{k=1}^{\\ell}\\clique(J_{k})$ imply $(i,j)\\in\\clique(J_{u})$\nfor some $J_{u}$, or equivalently $i,j\\in J_{u}$. Given that $j\\in J_{u}$\nand $w=\\max\\{k:j\\in J_{k}\\}$ by definition, it follows that $u\\le w$.\n\\item If $u=w$, then $i\\in J_{w}$ holds because $i,j\\in J_{u}$. Otherwise,\nif $u<w$, we use the sorted running intersection property to assert\nthe following\n\\begin{gather}\nu<w,\\qquad i,j\\in J_{u}\\quad\\implies\\quad i,j\\in J_{\\p(u)}.\\label{eq:par1}\n\\end{gather}\nThe fact that $j\\in J_{\\p(u)}$ follows directly the running intersection\nproperty $j\\in J_{u}\\cap J_{w}\\subseteq J_{\\p(u)}$ for $w>u$. By\ncontradiction, suppose that $i\\notin J_{\\p(u)}$. Then, given that\n$i\\in J_{u}$, it follows from the sorted property that $j\\ge\\min\\{J_{\\p(u)}\\}>\\max\\{J_{u}\\backslash J_{\\p(u)}\\}\\ge i,$\nwhich contradicts our initial hypothesis that $i>j$. \n\\item Inductively reapplying (\\ref{eq:par1}), as in $i,j\\in J_{\\p(\\p(u))}$\nand $i,j\\in J_{\\p(\\p(\\p(u)))}$, we arrive at some $v$ such that\n$i,j\\in J_{\\p(v)}$ and $p(v)=w$. The induction must terminate with\n$p(v)\\ge w$ because each $\\p(u)>u$ by construction. It is impossible\nfor $\\p(v)>w$ to occur, because $j\\in J_{\\p(v)}$ and $w=\\max\\{k:j\\in J_{k}\\}$\nby definition. We conclude that $i\\in J_{w}$, as desired.\n\\end{itemize}\n\\qed\n\\end{proof}\n\n\\begin{proof}[Proof of \\propref{rip->zf}]To prove that $F$ is zero-fill,\nwe observe, for arbitrary $(i,k)\\in F$ and $(j,k)\\in F$ with $i>j>k$\nthat $i,j\\in\\col_{F}(k)$. Therefore, it follows from \\lemref{colrip}\nthat there exists $J_{w}$ such that $i,j,k\\in J_{w}$. We conclude\nthat $(i,j)\\in F,$ because $F=\\bigcup_{j=1}^{\\ell}\\clique(J_{j})$\nand $i,j\\in J_{w}$ for some $1\\le w\\le\\ell$. \n\nTo prove that $\\omega(F)=\\max_{j}|J_{j}|$, we choose $k=\\arg\\max_{j}|\\col_{F}(j)|$.\nIt follows from Lemma~\\ref{lem:colrip} that there exists $J_{w}$\nsuch that $\\col_{F}(k)\\subseteq J_{w}$, and therefore $\\omega(F)=|\\col_{F}(k)|\\le|J_{w}|$.\nFinally, given that $\\clique(J_{w})\\subseteq F$ it follows that $(i,j)\\in F$\nholds for all $i\\in J_{w}$ where $j=\\min J_{w}$. Therefore, we conclude\nthat $J_{w}\\subseteq\\col_{F}(j)$, and therefore $|J_{w}|\\le|\\col_{F}(j)|\\le|\\col_{F}(k)|=\\omega(F)$.\n\\qed \\end{proof}\n\n\\begin{proof}[Proof of \\lemref{quadlift}]For an arbitrary sparsity\npattern $E$ of order $n$, let $F=\\chol(E)$ and $J_{k}\\equiv\\col_{F}(k)$.\nFirst, it follows from \\propref{zf->rip} that $J_{1},J_{2},\\dots,J_{n}$\nsatisfy the ordered running intersection property. Therefore, it follows\nfrom \\lemref{lift}, $J_{1}^{(2)},J_{2}^{(2)},\\dots,J_{n}^{(2)}$\ndefined in (\\ref{eq:Jk2_def}) satisfy the same property. Finally,\nwe verify that \n\\begin{align*}\n\\supp(\\P_{k})\\equiv & \\left\\{ \\alpha:\\P_{k}[\\alpha,\\beta]\\ne0\\right\\} =\\left\\{ \\alpha:\\P_{k}^{T}e_{\\alpha}\\ne0\\right\\} \\\\\n\\overset{\\text{(a)}}{=} & \\left\\{ \\idx_{F}(i,j):\\P_{k}^{T}\\svec_{F}(e_{i}e_{j}^{T})\\ne0,\\;(i,j)\\in F\\right\\} \\\\\n\\overset{\\text{(b)}}{=} & \\left\\{ \\idx_{F}(i,j):(e_{i}e_{j}^{T})[J_{k},J_{k}]\\ne0,\\;i\\ge j\\right\\} =J_{k}^{(2)}.\n\\end{align*}\nEquality (a) is obtained by substituting $e_{\\alpha}=\\svec_{F}(e_{i}e_{j}^{T})$\nfor $(i,j)\\in F$. Equality (b) follows the identity $\\P_{k}^{T}\\svec_{F}(Y)=\\svec(Y[J_{k},J_{k}])$,\nwhich was used to define $\\P_{k}$. Therefore, $F^{(2)}=\\bigcup_{k=1}^{n}J_{k}^{(2)}$\nand we conclude via \\propref{rip->zf} that $\\omega(F^{(2)})=\\max_{k}|J_{k}^{(2)}|=\\max_{k}\\frac{1}{2}|J_{k}|(|J_{k}|+1)$,\nand we recall that $\\max_{k}|J_{k}|=\\omega(F)=\\omega(E)$ by definition.\n\\qed \\end{proof}\n\n\\begin{lemma}\nFor every $i\\in\\{1,2,\\dots,m\\}$, there exists $k\\in\\{1,2,\\dots,n\\}$\nsuch that $\\supp(a_{i})\\subseteq\\supp(\\o{\\P}_{k})[V^{(2)}]$ where\n$V^{(2)}=\\{\\idx_{\\overline{F}}(i,j):i,j\\in F\\}$. \n\\end{lemma}\n\n\\begin{proof}\nIt follows by repeating the proof of \\lemref{quadlift} that\n\\begin{gather*}\n\\supp(\\o{\\P}_{k})=\\{\\idx_{\\o F}(i,j):i,j\\in\\col_{\\o F}(k),i\\ge j\\},\\\\\n\\supp(\\o{\\P}_{k})[V^{(2)}]=\\{\\idx_{F}(i,j):i,j\\in\\col_{\\o F}(k),i\\ge j\\}.\n\\end{gather*}\nOur desired claim follows via the following sequence of inclusions\n\\begin{align*}\n\\supp(a_{i}) & =\\{v:a_{i}[v]\\ne0\\}=\\{\\idx_{F}(u,v):A_{i}[u,v]\\ne0,\\quad(u,v)\\in F\\}\\\\\n & \\overset{\\text{(a)}}{\\subseteq}\\{\\idx_{F}(u,v):u,v\\in\\supp(A_{i}),\\quad(u,v)\\in F\\}\\\\\n & \\overset{\\text{(b)}}{\\subseteq}\\{\\idx_{F}(u,v):u,v\\in\\col_{\\o F}(k),\\quad(u,v)\\in F\\}=\\supp(\\o{\\P}_{k})[V^{(2)}].\n\\end{align*}\nInclusion (a) is because $\\spar(A_{i})\\subseteq\\clique(\\supp(A_{i}))$.\nInclusion (b) is true via the following: If $J=\\supp(A_{i})$ satisfies\n$\\clique(J)\\subseteq\\o F$, then $J\\subseteq\\col_{\\o F}(k)$ where\n$k=\\min J$. Indeed, we have $k\\in\\col_{\\o F}(k)$ by definition.\nFor any arbitrary $j\\in J$ with $j>k$, we must have an edge $(j,k)\\in\\clique(J)\\subseteq\\o F$,\nand therefore $j\\in\\col_{\\o F}(k)$. \\qed\n\\end{proof}\n\n\\begin{lemma}\nFor every $k\\in\\{1,2,\\dots,n\\}$, we have $\\supp(\\P_{k})\\subseteq\\supp(\\o{\\P}_{k})[V^{(2)}]$\nwhere $V^{(2)}=\\{\\idx_{\\overline{F}}(i,j):i,j\\in F\\}$. \n\\end{lemma}\n\n\\begin{proof}\nIt again follows by repeating the proof of \\lemref{quadlift} that\n\\begin{align*}\n\\supp(\\P_{k}) & =\\{\\idx_{F}(i,j):i,j\\in\\col_{F}(k),i\\ge j\\}\\\\\n & \\overset{\\text{(a)}}{\\subseteq}\\{\\idx_{F}(i,j):i,j\\in\\col_{\\o F}(k),i\\ge j\\}=\\supp(\\o{\\P}_{k})[V^{(2)}].\n\\end{align*}\nInclusion (a) is true because $E\\subseteq\\o E$ implies $F=\\chol(E)\\subseteq\\chol(\\o E)=\\o F$\nvia \\propref{del}, and therefore $\\col_{F}(k)\\subseteq\\col_{\\o F}(k)$.\n\\qed\n\\end{proof}\n\n\\begin{lemma}[Linear independence]\nWe have $\\A^{T}y=0$ if and only $y=0$. \n\\end{lemma}\n\n\\begin{proof}\nWe will prove $\\sum_{k=1}^{n}\\P_{k}\\P_{k}^{T}\\succeq I$, which implies\n$\\A\\A^{T}\\succeq I$ and hence $\\|\\A^{T}y\\|\\ge\\|y\\|$. For arbitrary\n$Y\\in\\S_{F}^{n}$ with $y=\\svec_{F}(Y)$, we observe that \n\\[\n\\|\\P_{k}^{T}y\\|^{2}=\\|\\svec(Y[J_{k},J_{k}])\\|^{2}=\\sum_{i,j\\in J_{k}}(Y[i,j])^{2}=\\sum_{(i,j)\\in\\clique(J_{k})}\\gamma_{ij}(Y[i,j])^{2}\n\\]\nwhere $\\gamma_{ij}=1$ if $i=j$ and $\\gamma_{ij}=2$ if $i\\ne j$.\nTherefore, we have\n\\[\n\\sum_{k=1}^{n}\\|\\P_{k}^{T}y\\|^{2}=\\sum_{k=1}^{n}\\sum_{(i,j)\\in\\clique(J_{k})}\\gamma_{ij}(Y[i,j])^{2}\\ge\\sum_{(i,j)\\in F}\\gamma_{ij}(Y[i,j])^{2}=\\|y\\|^{2}.\n\\]\nThe inequality is because $F=\\bigcup_{k=1}^{n}\\clique(J_{k})$. \\qed\n\\end{proof}\n\n\\begin{lemma}[Strong duality is attained]\nUnder \\asmref{strict}, there exists $x^{\\star},s^{\\star}\\in\\K$\nsatisfying $\\A x^{\\star}=\\b,$ $\\A^{T}y^{\\star}+s^{\\star}=\\c,$ $(x^{\\star})^{T}s^{\\star}=0$. \n\\end{lemma}\n\n\\begin{proof}\nDefine $P_{j}$ implicitly to satisfy $P_{j}^{T}x=x[\\col_{F}(j)]$\nfor all $x\\in\\R^{m}$. \\asmref{strict} says that there exists $\\hat{X}\\succeq0$\nand $\\hat{y}\\le0$ and that satisfy $\\AA(\\hat{X})\\le b$ and $\\AA^{T}(\\hat{y})+\\hat{S}=C$\nand $\\inner C{\\hat{X}}=\\inner b{\\hat{y}}$. It follows from \\citep[Theorem~9.2]{vandenberghe2015chordal}\nthat $\\hat{S}=C-\\AA^{T}(\\hat{y})\\in\\S_{F}^{n}\\cap\\S_{+}^{n}$ if and\nonly if there exists $\\hat{S}_{j}\\succeq0$ such that $\\hat{S}=\\sum_{j=1}^{n}P_{j}\\hat{S}_{j}P_{j}^{T}$.\nNow, we turn to the primal-dual pair defined by the data $(\\A,\\b,\\c,\\K)$\nin (\\ref{eq:AKdef}), which is written \n\\begin{align*}\n & \\min_{v\\in\\R^{m},V_{j}\\in\\S^{\\omega_{j}}}\\left\\{ \\inner bv:\\begin{array}{c}\n\\proj_{F}\\left[\\AA^{T}(-v)+\\sum_{j=1}^{n}P_{j}V_{j}P_{j}^{T}-C\\right]=0,\\\\\nv\\ge0,\\quad V_{1},\\dots,V_{n}\\succeq0\n\\end{array}\\right\\} \\\\\n= & \\max_{Y\\in\\S_{F}^{n}}\\left\\{ \\inner{-C}Y:\\AA(Y)\\le b,\\;-P_{j}^{T}YP_{j}\\preceq0\\text{ for all }j\\in\\{1,2,\\dots,n\\}\\right\\} \n\\end{align*}\nWe can mechanically verify that $v^{\\star}=-\\hat{y}$ and $V_{j}^{\\star}=\\hat{S}_{j}$\nis feasible for the primal problem, and $Y^{\\star}=\\Pi_{F}(\\hat{X})$\nis feasible for the dual problem, and that the two objectives coincide\n$\\inner b{v^{\\star}}=\\inner{-C}{Y^{\\star}}$. Therefore, we conclude\nthat $x^{\\star}=(v^{\\star},\\svec(V_{1}^{\\star}),\\dots,\\svec(V_{n}^{\\star}))$\nand $y^{\\star}=\\svec_{F}(Y^{\\star})$ is a complementary solution\nsatisfying $\\A^{T}x^{\\star}=\\b,$ $\\A^{T}y^{\\star}+s^{\\star}=\\c,$\nand $\\inner{x^{\\star}}{s^{\\star}}=0$. \\qed \n\\end{proof}\n\n\\begin{proposition}\n\\label{prop:general}Let $(\\A,\\b,\\c,\\K)$ describe an SDP in $(n,\\omega)$-standard\nform, and let $\\opt$ denote its optimal value. Let $\\Tim$ and $\\Mem$\ndenote the time and memory needed, given data $\\A\\in\\R^{M\\times N}$,\n$g\\in\\R^{M}$, and a choice of $w\\in\\interior(\\K)$, to form and solve\n$\\A\\D\\A^{T}\\Delta y=g$ for $\\Delta y\\in\\R^{M}$, where $\\D^{-1}=\\nabla^{2}f(w)$\nand $f=-\\log\\det(w)$. Then, a general-purpose interior-point method\ncomputes $(x,y)$ that satisfy the following in $O(\\sqrt{n}\\log(1/\\epsilon))$\niterations\n\\begin{gather*}\n\\c^{T}x\\le\\mathrm{opt}+n\\cdot\\epsilon,\\quad\\|\\A x-\\b\\|\\le\\epsilon,\\quad x\\in\\K,\\\\\n\\b^{T}y\\ge\\mathrm{opt}-n\\cdot\\epsilon,\\quad\\c-\\A^{T}y+\\epsilon\\one_{\\K}\\in\\K,\n\\end{gather*}\nwith per-iteration costs of $O(\\omega^{2}n+\\nnz(\\A)+\\Tim)$ time and\n$O(\\omega n+\\nnz(\\A)+\\Mem)$ memory.\n\\end{proposition}\n\n\\begin{lemma}[$\\epsilon$-accurate and $\\epsilon$-feasible]\n\\label{lem:eps}Let there exist $(x^{\\star},y^{\\star})$ such that\n$\\A x^{\\star}=\\b$ with $x^{\\star}\\in\\K$ and $\\c-\\A^{T}y^{\\star}\\equiv s^{\\star}\\in\\K$\nthat satisfies strong duality $\\b^{T}y^{\\star}=\\c^{T}x^{\\star}$.\nThen, a point $(x,y,s,\\tau,\\theta,\\kappa)$ that is feasible for (\\ref{eq:hsd})\nand satisfies the optimality condition (\\ref{eq:short-step}) also\nsatisfies the following, where $\\rho=1+\\one_{\\K}^{T}(x^{\\star}+s^{\\star})$:\n\\begin{align*}\n\\|\\A(x/\\tau)-\\b\\| & \\le\\frac{\\rho\\|r_{p}\\|}{\\gamma}\\cdot\\epsilon, & \\|\\A^{T}(y/\\tau)+(s/\\tau)-\\c\\| & \\le\\frac{\\rho\\|r_{d}\\|}{\\gamma}\\cdot\\epsilon, & \\frac{(x/\\tau)^{T}(s/\\tau)}{n+1}\\le\\frac{\\rho^{2}}{\\gamma^{2}}\\cdot\\epsilon.\n\\end{align*}\n\\end{lemma}\n\n\\begin{proof}\nFirst, observe that $(\\hat{x},\\hat{y},\\hat{s},\\hat{\\tau},\\hat{\\theta},\\hat{\\kappa})=(\\hat{\\tau}x^{\\star},\\hat{\\tau}y^{\\star},\\hat{\\tau}s^{\\star},\\hat{\\tau},0,0)$\nwith $\\hat{\\tau}=(n+1)/\\rho$ is a solution to (\\ref{eq:hsd}), because\n$r_{d}^{T}x^{\\star}+r_{p}^{T}y^{\\star}+r_{c}=\\one_{\\K}^{T}(x^{\\star}+s^{\\star})+1=\\rho$.\nNext, we prove that if $(x,y,s,\\tau,\\theta,\\kappa)$ is feasible for\n(\\ref{eq:hsd}) and satisfies (\\ref{eq:short-step}), then $\\tau\\ge\\frac{\\gamma}{\\rho}$.\nIndeed, the skew-symmetry of (\\ref{eq:hsdfeas}) yields $\\theta=\\frac{x^{T}s+\\tau\\kappa}{n+1}$\nand $(x-\\hat{x})^{T}(s-\\hat{s})+(\\tau-\\hat{\\tau})(\\kappa-\\hat{\\kappa})=0.$\nRearranging yields $(n+1)\\theta=x^{T}s+\\tau\\kappa=\\hat{x}^{T}s+x^{T}\\hat{s}+\\tau\\hat{\\kappa}+\\hat{\\tau}\\kappa$\nand hence $\\kappa\\le(n+1)\\theta/\\hat{\\tau}$. Under (\\ref{eq:short-step}),\nwe have $\\theta=\\frac{x^{T}s+\\tau\\kappa}{n+1}\\le\\epsilon$ and $\\tau\\ge\\frac{\\gamma\\epsilon}{\\kappa}\\ge\\frac{\\gamma\\theta}{\\kappa}\\ge\\frac{\\gamma\\theta}{(n+1)\\theta/\\hat{\\tau}}=\\frac{\\gamma}{n+1}\\hat{\\tau}=\\frac{\\gamma}{\\rho}$.\nFinally, divide (\\ref{eq:hsdfeas}) through by $\\tau$ and substitute\n$1/\\tau\\le\\rho/\\gamma$. \\qed\n\\end{proof}\n\n\\begin{proof}[Proof of \\propref{general}]Recall that $(\\A,\\b,\\c,\\K)$\ndescribe an SDP in $(n,\\omega)$-standard form, and hence there exist\n$(x^{\\star},y^{\\star})$ such that $\\A x^{\\star}=\\b$ with $x^{\\star}\\in\\K$\nand $\\c-\\A^{T}y^{\\star}\\in\\K$ that satisfies strong duality $\\b^{T}y^{\\star}=\\c^{T}x^{\\star}$.\nCombining \\lemref{eps} and \\lemref{short-step} shows that iterations\nin (\\ref{eq:hsd_ipm}) converges to the desired $\\epsilon$-accurate,\n$\\epsilon$-feasible iterate after $O(\\sqrt{n}\\log(1/\\epsilon))$\niterations. The cost of each iteration is essentially equal to the\ncost of computing the search direction in (\\ref{eq:hsd_nt}). We account\nfor this cost via the following two steps:\n\\begin{enumerate}\n\\item (Scaling point) We partition $x=[\\svec(X_{j})]_{j=1}^{\\ell}$ and\n$s=[\\svec(S_{j})]_{j=1}^{\\ell}$. Then, the scaling point $w=[\\svec(W_{j})]_{j=1}^{\\ell}$\nis given in closed-form as $W_{j}=S_{j}^{1/2}(S_{j}^{1/2}X_{j}S_{j}^{1/2})^{-1/2}S_{j}^{1/2}$~\\citep[Section~5]{sturm2002implementation}.\nNoting that each $W_{j}$ is at most $\\omega\\times\\omega$, the cost\nof forming $w$ is at most of order \n\\[\n\\sum_{j=1}^{\\ell}\\omega_{j}^{3}\\le\\omega^{2}\\sum_{j=1}^{\\ell}\\omega_{j}=\\omega^{2}n\\text{ time},\\quad\\sum_{j=1}^{\\ell}\\omega_{j}^{2}\\le\\omega\\sum_{j=1}^{\\ell}\\omega_{j}=\\omega n\\text{ memory.}\n\\]\nBy this same calculation, it follows that the Hessian matrix-vector\nproducts $\\nabla^{2}f(w)x=[\\svec(W_{j}^{-1}X_{j}W_{j}^{-1})]_{j=1}^{\\ell}$\nand $\\nabla^{2}f(w)^{-1}x=[\\svec(W_{j}X_{j}W_{j})]_{j=1}^{\\ell}$\nalso cost $O(\\omega^{2}n)$ time and $O(\\omega n)$ memory. \n\\item (Search direction) Using elementary but tedious linear algebra, we\ncan show that if\\begin{subequations}\\label{eq:newt}\n\\begin{equation}\n(\\A\\D\\A^{T})\\begin{bmatrix}v_{1} & v_{2} & v_{3}\\end{bmatrix}=\\begin{bmatrix}0 & -\\b & r_{p}\\end{bmatrix}-\\A\\D\\begin{bmatrix}d & \\c & r_{d}\\end{bmatrix}\\label{eq:hsd_nrm}\n\\end{equation}\nwhere $\\D=[\\nabla^{2}f(w)]^{-1}$ and $d=-s-\\mu^{+}\\nabla f(x)$,\nand \n\\begin{equation}\n\\begin{bmatrix}u_{1} & u_{2} & u_{3}\\end{bmatrix}=\\D(\\begin{bmatrix}d & c & r_{d}\\end{bmatrix}+\\A^{T}\\begin{bmatrix}v_{1} & v_{2} & v_{3}\\end{bmatrix}),\\label{eq:hsd_back1}\n\\end{equation}\nthen\n\\begin{align}\n\\left(\\begin{bmatrix}-\\D_{0}^{-1} & -r_{c}\\\\\nr_{c} & 0\n\\end{bmatrix}-\\begin{bmatrix}\\c & r_{d}\\\\\n-\\b & r_{p}\n\\end{bmatrix}^{T}\\begin{bmatrix}u_{2} & u_{3}\\\\\nv_{2} & v_{3}\n\\end{bmatrix}\\right)\\begin{bmatrix}\\Delta\\tau\\\\\n\\Delta\\theta\n\\end{bmatrix} & =\\begin{bmatrix}-d_{0}\\\\\n0\n\\end{bmatrix}-\\begin{bmatrix}\\c & r_{d}\\\\\n-\\b & r_{p}\n\\end{bmatrix}^{T}\\begin{bmatrix}u_{1}\\\\\nv_{1}\n\\end{bmatrix},\\\\\n\\begin{bmatrix}\\Delta x\\\\\n\\Delta y\n\\end{bmatrix} & =\\begin{bmatrix}u_{1}\\\\\nv_{1}\n\\end{bmatrix}-\\begin{bmatrix}u_{1} & u_{2}\\\\\nv_{1} & v_{2}\n\\end{bmatrix}\\begin{bmatrix}\\Delta\\tau\\\\\n\\Delta\\theta\n\\end{bmatrix},\\\\\n\\Delta s & =d-\\D^{-1}\\Delta x,\\\\\n\\Delta\\kappa & =d_{0}-\\D_{0}^{-1}\\Delta\\tau,\\label{eq:hsd_back2}\n\\end{align}\n\\end{subequations}where $\\D_{0}=\\tau/\\kappa$ and $d_{0}=-\\kappa+\\mu^{+}\\tau^{-1}$.\nTherefore, we conclude that the cost of computing the search direction\nin (\\ref{eq:hsd_nt}) is equal to the cost of solving $O(1)$ instances\nof the Schur complement equation $\\A\\D\\A^{T}\\Delta y=g$, plus $O(1)$\nmatrix-vector products with $\\A,\\A^{T},\\D,\\D^{-1}$, for a total cost\nof $O(\\Tim+\\nnz(\\A)+\\omega^{2}n)$ time and $O(\\Mem+\\nnz(\\A)+\\omega n)$\nmemory respectively. Note that $\\A\\D\\A^{T}\\succ0$ because $\\A\\A^{T}\\succ0$\nby the linear independence assumption, and $\\D^{-1}=\\nabla^{2}f(w)\\succ0$\nfor all $w\\in\\interior(\\K)$.\n\\end{enumerate}\n\\qed \\end{proof}","units":[{"raw_tex":"\\begin{proposition}[Sparse Cholesky factorization]\n\\label{prop:chol}Given $S\\in\\S^{n},$ $S\\succ0$, let $\\omega\\equiv\\omega(S)$.\nSparse Cholesky factorization factors $L=\\chol(S)$ in $\\Tim$ arithmetic\noperations and $M$ units of memory, where \n\\begin{align*}\n\\frac{1}{6}(\\omega-1)^{3}+n & \\le\\Tim\\le\\omega^{2}\\cdot n, & \\frac{1}{2}(\\omega-1)^{2}+n & \\le\\Mem\\le\\omega\\cdot n.\n\\end{align*}\n\\end{proposition}","start":34018,"end":34422},{"raw_tex":"\\begin{proof}\nLet $\\omega_{j}\\equiv|\\col_{F}(j)|$. By inspection, $\\Tim=\\sum_{j=1}^{n}\\frac{1}{2}\\omega_{j}(\\omega_{j}+1)$\nand $\\Mem=\\sum_{j=1}^{n}\\omega_{j}$. The bounds follow by substituting\n$\\omega\\ge\\omega_{j+1}\\ge\\omega_{j}-1$. Indeed,\n\\[\n\\Mem=\\sum_{j=1}^{n}\\omega_{j}\\ge\\underbrace{\\omega+(\\omega-1)+\\cdots+1}_{\\omega\\text{ terms}}+\\underbrace{1+\\cdots+1}_{n-\\omega\\text{ terms}}=\\frac{1}{2}\\omega(\\omega+1)+(n-\\omega)\n\\]\nand similarly $\\Tim=\\sum_{j=1}^{n}\\frac{1}{2}\\omega_{j}(\\omega_{j}+1)\\ge\\frac{1}{6}\\omega(\\omega+1)(\\omega+2)+(n-\\omega).$\n\\qed\n\\end{proof}","start":34423,"end":34991},{"raw_tex":"\\begin{theorem}[Frontsize of Schur complement sparsity]\n\\label{thm:front}Given $C,A_{1},A_{2},\\dots,A_{m}\\in\\S^{n}$, define\n$\\A,\\K$ as in (\\ref{eq:AKdef}), and define $E^{(2)}$ as in (\\ref{eq:E2def}).\nWe have \n\\[\n\\frac{1}{2}\\omega(\\omega+1)\\le\\omega(E^{(2)})\\le\\frac{1}{2}\\o{\\omega}(\\o{\\omega}+1)\n\\]\nwhere $\\omega\\equiv\\omega(E)$ and $\\o{\\omega}=\\omega(\\o E)$ and\n$E,\\o E$ are defined in (\\ref{eq:Edef}) and (\\ref{eq:Ebardef}).\nMoreover, if $E=\\o E$, then we also have $\\chol(E^{(2)})=E^{(2)}$. \n\\end{theorem}","start":46293,"end":46802},{"raw_tex":"\\begin{corollary}[Cost of Schur complement equation]\n\\label{cor:normal}Given the data matrix $\\A$, scaling point $w\\in\\interior(\\K)$,\nand right-hand side $g$, define $\\Hess=\\A\\nabla^{2}f(w)\\A^{T}$ as\nin (\\ref{eq:E2def}). Suppose that all columns of $\\A$ are nonzero,\nand all scaling matrices $\\D_{j}$ are fully dense. Then, it takes\n$\\Tim$ arithmetic operations and $\\Mem$ units of storage to form\n$\\Hess$ and solve $\\Hess\\Delta y=g$, where\n\\begin{gather*}\n\\frac{1}{48}(\\omega-1)^{6}+m+n\\le\\Tim\\le4\\o{\\omega}^{4}\\cdot(m+\\omega n),\\\\\n\\frac{1}{8}(\\omega-1)^{4}+m+n\\le\\Mem\\le2\\o{\\omega}^{2}\\cdot(m+\\omega n),\n\\end{gather*}\nin which $\\omega\\equiv\\omega(E)$ and $\\o{\\omega}\\equiv\\omega(\\o E)$\nsatisfy $1\\le\\omega\\le\\o{\\omega}$.\n\\end{corollary}","start":47617,"end":48355},{"raw_tex":"\\begin{proof}\nLet $a_{i}\\equiv\\svec_{F}(A_{i})$ and $\\omega_{j}\\equiv|\\col_{F}(j)|$.\nWe break the solution of $\\Hess\\Delta y=g$ into five steps:\n\\begin{enumerate}\n\\item (Input) It takes $\\Mem_{\\mathrm{data}}$ memory to state the problem\ndata, where $m+n\\le\\Mem_{\\mathrm{data}}\\le2\\o{\\omega}^{2}m+3\\omega^{2}n$.\nIndeed, $m+n\\le\\nnz(\\A)\\le\\o{\\omega}^{2}m+\\omega^{2}n$ because $1\\le\\nnz(a_{i})\\le\\o{\\omega}^{2}$\n(see Step 2 below) and $\\nnz(\\P_{j})=\\frac{1}{2}\\omega_{j}(\\omega_{j}+1)$.\nAlso, $m+n\\le\\nnz(w)\\le m+\\omega^{2}n$, and $n\\le\\nnz(g)\\le\\omega n$.\n\\item (Build LP part) It takes $\\Tim_{\\mathrm{LP}}$ time to build $\\sum_{i=1}^{m}d_{i}a_{i}a_{i}^{T}$,\nwhere $m\\le\\Tim_{\\mathrm{LP}}\\le2\\o{\\omega}^{4}m$ time and memory.\nThis follows from $1\\le\\nnz(a_{i})\\le\\o{\\omega}^{2}$, where the upper-bound\nis because $\\spar(a_{i}a_{i}^{T})\\subseteq E^{(2)}\\subseteq\\chol(E^{(2)})$,\nand that $\\omega(E^{(2)})$ is, by definition, the maximum number\nof nonzero elements in a single column of $\\chol(E^{(2)})$.\n\\item (Build SDP part) It takes $\\Tim_{\\mathrm{SDP}}$ time to build $\\sum_{j=1}^{n}\\P_{j}\\D_{j}\\P_{j}^{T}$,\nwhere $n\\le\\Tim_{\\mathrm{SDP}}\\le\\omega^{4}n$ time and memory. This\nfollows from $\\nnz(\\P_{j})=\\frac{1}{2}\\omega_{j}(\\omega_{j}+1)$,\nwhich implies $\\nnz(\\P_{j}\\D_{j}\\P_{j}^{T})=\\frac{1}{4}\\omega_{j}^{2}(\\omega_{j}+1)^{2}$\nfor a fully-dense $\\D_{j}\\succ0$. \n\\item (Factorization) It takes $\\Tim_{\\mathrm{fact}}$ time and $\\Mem_{\\mathrm{fact}}$\nmemory to factor $\\L=\\chol(\\Hess)$, where $\\frac{1}{48}(\\omega-1)^{6}+n\\le\\Tim_{\\mathrm{fact}}\\le\\o{\\omega}^{5}n$\nand $\\frac{1}{8}(\\omega-1)^{4}+n\\le\\Mem_{\\mathrm{fact}}\\le\\o{\\omega}^{3}n$.\nThe matrix $\\Hess$ has $|F|$ columns and rows, and frontsize $\\omega(E^{(2)})$.\nThe desired figures follow by substituting $\\frac{1}{2}\\omega^{2}\\le\\omega(E^{(2)})\\le\\o{\\omega}^{2}$\nand $n\\le|F|\\le\\omega n$ into \\propref{chol}.\n\\item (Back-substitution) It takes $\\Mem_{\\mathrm{fact}}$ time and memory\nto solve each of $\\L z=g$ and $\\L^{T}\\Delta y=z$ via triangular\nback-substitution. \n\\end{enumerate}\nThe overall runtime is cumulative, so $\\Tim=\\Tim_{\\mathrm{LP}}+\\Tim_{\\mathrm{SDP}}+\\Tim_{\\mathrm{fact}}+2\\Mem_{\\mathrm{fact}}$.\nThe overall memory use is $\\Mem=\\Mem_{\\mathrm{data}}+\\Mem_{\\mathrm{fact}}$,\nbecause the matrix $\\Hess$ can be constructed and then factored in-place.\n\\qed\n\\end{proof}","start":48356,"end":50694},{"raw_tex":"\\begin{proof}One can verify that $(\\A,\\b,\\c,\\K)$ defined in (\\ref{eq:AKdef})\nspecifies an SDP in $(N,\\omega)$-standard form (see \\appref{regularity}\nfor the regularity conditions). Moreover, it follows from the monotonicity\nof the frontsize (\\propref{del}) that $\\omega\\equiv\\omega(E_{\\Pi})\\le\\omega(\\o E_{\\Pi})\\equiv\\o{\\omega}$.\nWe will track the cost of \\algref{myalg} step-by-step:\n\\begin{enumerate}\n\\item (Front-reducing permutation) Preordering $A_{i}\\gets\\Pi A_{i}\\Pi^{T}$\nand $C\\gets\\Pi C\\Pi^{T}$ the matrices cost $O(\\nnz(C)+\\sum_{i=1}^{m}\\nnz(A_{i}))=O(\\o{\\omega}^{2}N)$\ntime and memory. This follows from $\\nnz(C)\\le|F|\\le N$ and $\\nnz(A_{i})\\le\\o{\\omega}^{2}$. \n\\item (Conversion) Computing $F=\\chol(E)$ costs $O(|F|)=O(N)$ time and\nspace, where we note that $|F|\\le\\omega n$. \n\\item (Solution) Let $K=2\\cdot\\max\\{1,|\\tr(C)|,|\\tr(A_{1})|,\\dots,|\\tr(A_{m})|\\}$.\nAfter $O(\\sqrt{N}\\log(K/\\epsilon))$ iterations, we arrive at a primal\n$v\\le0$ and $V_{j}\\succeq0$ and dual point $Y\\in\\S_{F}^{n}$ satisfying\n\\begin{gather*}\n\\inner{A_{i}}Y-b_{i}\\le\\epsilon/K,\\quad Y[J_{j},J_{j}]\\succeq-(\\epsilon/K)I,\\\\\n{\\textstyle \\left\\Vert \\sum_{i}v_{i}A_{i}+\\sum_{j}P_{j}V_{j}P_{j}^{T}-C\\right\\Vert \\le\\epsilon/K,}\\\\\n\\inner CY-\\inner bv\\le N\\cdot(\\epsilon/K).\n\\end{gather*}\nEach iteration costs $O(\\o{\\omega}^{4}N)$ time and $O(\\o{\\omega}^{2}N)$\nmemory. This cost is fully determined by the cost of solving $O(1)$\ninstances of the Schur complement equation, which dominates the overhead\nof $O(\\omega^{3}n+\\nnz(\\A))=O(\\o{\\omega}^{2}N)$ time and memory.\n\\item (Recovery) Using the previously recovered $Y$, we recover $U$ such\nthat $\\Pi_{F}(UU^{T})=Y+\\delta I$ where $\\delta=-\\min_{j}\\{0,\\lambda_{\\min}(Y[J_{j},J_{j}])\\}\\le\\epsilon/K$.\nThis takes $O(\\omega^{3}n)=O(\\o{\\omega}^{2}N)$ time and $O(\\omega^{2}n)=O(\\o{\\omega}N)$\nmemory.\n\\item (Output) We output $U$ and $v$, and check for accuracy. It follows\nfrom $K\\ge2|\\tr(C)|$ that\n\\[\n\\inner C{UU^{T}}-\\inner CY=\\delta\\cdot\\tr(C)\\le\\epsilon\\cdot\\frac{\\tr(C)}{K}\\le\\epsilon\\cdot\\frac{|\\tr(C)|}{2|\\tr(C)|}\\le\\frac{1}{2}\\epsilon,\n\\]\nand from $K\\ge2$ and $N\\ge1$ that \n\\[\n\\inner C{UU^{T}}-\\inner bv\\le\\frac{N}{K}\\epsilon+\\frac{1}{2}\\epsilon=N\\epsilon\\cdot\\left(\\frac{1}{2}+\\frac{1}{2N}\\right)\\le N\\cdot\\epsilon.\n\\]\nSimilarly, it follows from $K\\ge2|\\tr(A_{i})|$ and $K\\ge2$ that\n$\\inner{A_{i}}{UU^{T}}-b_{i}\\le\\epsilon$. Finally, it follows from\n$v\\ge0$ and $V_{j}\\succeq0$ that\n\\[\n\\sum_{i}v_{i}A_{i}+\\sum_{j}P_{j}V_{j}P_{j}^{T}-C\\preceq\\epsilon\\cdot I\\implies\\sum_{i}v_{i}A_{i}-C\\preceq\\epsilon\\cdot I.\n\\]\n\\end{enumerate}\n\\qed \\end{proof}","start":52221,"end":54795},{"raw_tex":"\\begin{proposition}[Subgraph monotonicity]\n\\label{prop:del}Let $E$ be a sparsity pattern of order $n$, and\nlet $U\\subseteq\\{1,2,\\dots,n\\}$. Then, for any sparsity pattern $D$\nof order $|U|$ that satisfies $E[U]\\supseteq D$, we have $\\chol(E)[U]\\supseteq\\chol(D)$,\nand therefore $\\omega(E)\\ge\\omega(D)$.\n\\end{proposition}","start":57847,"end":58167},{"raw_tex":"\\begin{proof}\nIt is known that $(i,j)\\in\\chol(E)$ for $i>j$ holds if and only\nif there exists a path $(i,p_{1},p_{2},\\dots,p_{\\ell},j)$ whose edges\nare in $E$, and whose internal nodes are ordered $p_{1},p_{2},\\dots,p_{\\ell}<j<i$;\nsee e.g.~\\citep[Theorem~6.1]{vandenberghe2015chordal}. It immediately\nfollows this characterization that $\\chol(\\cdot)$ is monotone with\nrespect to the deletion of edges and isolated vertices: (1) if $D\\subseteq E$,\nthen $\\chol(D)\\subseteq\\chol(E)$; (2) we have $\\chol(E[U])=\\chol(E)[U]$\nfor $U=\\{1,2,\\dots,n\\}\\backslash v$ with isolated vertex $v$. Therefore,\n$\\chol(\\cdot)$ must also be monotone under general vertex and edge\ndeletions, because we can always delete edges to isolate a vertex\nbefore deleting it. \\qed\n\\end{proof}","start":58168,"end":58929},{"raw_tex":"\\begin{lemma}[Quadratic lift]\n\\label{lem:quadlift}Let $E$ be an arbitrary sparsity pattern of\norder $n$. Define $F=\\chol(E)$ and the lifted sparsity pattern\n$F^{(2)}=\\bigcup_{k=1}^{n}\\clique(\\P_{k})$ in which each $\\P_{k}$\nis implicitly defined to satisfy $\\P_{k}^{T}\\svec_{F}(Y)=\\svec(Y[\\col_{F}(k),\\col_{F}(k)])$\nfor all $Y\\in\\S_{F}^{n}$. Then, we have $\\chol(F^{(2)})=F^{(2)}$\nand $\\omega(F^{(2)})=\\frac{1}{2}\\omega(E)[\\omega(E)+1]$.\n\\end{lemma}","start":59082,"end":59530},{"raw_tex":"\\begin{lemma}[Sparsity overestimate]\n\\label{lem:ub}Let $\\o E$ and $E^{(2)}$ be the sparsity patterns\ndefined in (\\ref{eq:Ebardef}) and (\\ref{eq:E2def}). Define $\\o F=\\chol(\\o E)$\nand the lifted sparsity pattern $\\o F^{(2)}=\\bigcup_{k=1}^{n}\\clique(\\o{\\P}_{k})$\nin which each $\\o{\\P}_{k}$ is implicitly defined to satisfy $\\o{\\P}_{k}^{T}\\svec_{\\o F}(Y)=\\svec(Y[\\col_{\\o F}(k),\\col_{\\o F}(k)])$\nfor all $Y\\in\\S_{\\o F}^{n}$. Then, $E^{(2)}\\subseteq\\o F^{(2)}[V^{(2)}]$\nholds for $V^{(2)}=\\{\\idx_{\\overline{F}}(i,j):i,j\\in F\\}$.\n\\end{lemma}","start":60078,"end":60614},{"raw_tex":"\\begin{proposition}[ZF$\\implies$RIP]\n\\label{prop:zf->rip}Let $F$ be a zero-fill sparsity pattern of\norder $n$. Then, the sequence of subsets $J_{1},J_{2},\\dots,J_{n}$\nwith $J_{j}=\\col_{F}(j)\\equiv\\{j\\}\\cup\\{i>j:(i,j)\\in F\\}$ satisfies\nthe sorted running intersection property. \n\\end{proposition}","start":62493,"end":62788},{"raw_tex":"\\begin{proof}\nDefine $\\p(j)=\\min\\{i>j:i\\in J_{j}\\}$ if $|J_{j}|>1$ and $\\p(j)=n$\nif $|J_{j}|=1$. Clearly, $\\p(j)>j$ holds for all $1\\le j<n$. To\nprove $J_{\\p(j)}\\supseteq J_{j}\\cap\\bigcup_{w=j+1}^{\\ell}J_{w}$,\nlet $i\\in J_{j}\\cap J_{w}$ for some $w>j$. We prove that $i\\in J_{\\p(j)}$\nvia the following steps: \n\\begin{itemize}\n\\item We have $i>j$, because $i\\in J_{w}$ implies that $i\\ge w>j$. \n\\item If $i=\\p(j)$, then $i\\in J_{\\p(j)}$ by definition. \n\\item If $i>\\p(j)$, then $(i,j)\\in F$ and $(\\p(j),j)\\in F$ for $i>\\p(j)>j$\nimplies $(i,\\p(j))\\in F$, and hence $i\\in J_{\\p(j)}$. \n\\end{itemize}\nFinally, we prove $\\min\\{J_{\\p(j)}\\}>\\max\\{J_{j}\\backslash J_{\\p(j)}\\}$\nby noting that $\\max\\{J_{j}\\backslash J_{\\p(j)}\\}=j$ with our construction,\nand that $i=\\min J_{\\p(j)}$ must satisfy $i\\in J_{\\p(j)}$ and therefore\n$i\\ge\\p(j)>j$. \\qed\n\\end{proof}","start":62789,"end":63636},{"raw_tex":"\\begin{lemma}\n\\label{lem:lift}Let the sequence of subsets $J_{1},J_{2},\\dots,J_{\\ell}$\nwith $J_{j}\\subset\\N$ satisfy the sorted running intersection property.\nThen, $J_{1}^{(2)},J_{2}^{(2)},\\dots,J_{\\ell}^{(2)}$ also satisfy\nthe sorted running intersection property. \n\\end{lemma}","start":64620,"end":64899},{"raw_tex":"\\begin{proof}\nLet $\\p(\\cdot)$ denote the parent pointer that verifies the sorted\nrunning intersection property in $J_{1},J_{2},\\dots,J_{\\ell}$. We\nwill verify that $\\p(\\cdot)$ also proves the same property in $J_{1}^{(2)},J_{2}^{(2)},\\dots,J_{\\ell}^{(2)}$. \n\nFirst, to prove $J_{u}^{(2)}\\cap\\bigcup_{v=u+1}^{\\ell}J_{v}^{(2)}\\subseteq J_{\\p(u)}^{(2)},$\nlet $k\\in J_{u}^{(2)}\\cap J_{v}^{(2)}$ for $v>u$. The fact that\n$k\\in J_{u}^{(2)}=\\idx(\\clique(J_{u}))$ implies $k=\\idx(i,j)$ for\nsome $i,j$ such that $i,j\\in J_{u}$. Similarly, $k\\in J_{v}^{(2)}=\\idx(\\clique(J_{v}))$\nand the bijectivity of $\\idx$ on $F$ imply that the same $i,j$\nalso satisfy $i,j\\in J_{v}$, where we recall that $v>u$. We conclude\n$i,j\\in J_{u}\\cap J_{v}\\subseteq J_{\\p(u)}$ and therefore $k=\\idx(i,j)\\in J_{\\p(u)}^{(2)}$.\n\nNext, we prove $\\min\\{J_{\\p(u)}^{(2)}\\}>\\max\\{J_{u}^{(2)}\\backslash J_{\\p(u)}^{(2)}\\}$\nby establishing two claims:\n\\begin{itemize}\n\\item $\\min\\{J_{p(u)}^{(2)}\\}=\\idx(\\alpha,\\alpha)$ where $\\alpha=\\min J_{\\p(u)}$.\nFor any $\\idx(i,j)\\in J_{\\p(u)}^{(2)}$ where $i\\ge j$, we must have\n$i,j\\in J_{\\p(u)}$. It follows from the raster property that $\\idx(i,j)$\nis minimized with $i=j=\\min J_{\\p(u)}$.\n\\item $\\max\\{J_{u}^{(2)}\\backslash J_{\\p(u)}^{(2)}\\}=\\idx(\\beta,\\gamma)$\nwhere $\\beta=\\max\\{J_{u}\\}$ and $\\gamma=\\max\\{J_{u}\\backslash J_{\\p(u)}\\}$.\nWe partition $J_{u}$ into $N_{u}=J_{u}\\backslash J_{\\p(u)}$ and\n$A_{u}=J_{u}\\cap J_{\\p(u)}$. \n\\begin{itemize}\n\\item For any $\\idx(i,j)\\in J_{u}^{(2)}\\backslash J_{p(u)}^{(2)}$ where\n$i\\ge j$, we can have one of the following three cases: 1) $i\\in N_{u}$\nand $j\\in A_{u}$; 2) $i\\in A_{u}$ and $j\\in N_{u}$; or 3) $i\\in N_{u}$\nand $j\\in N_{u}$. \n\\item We observe that the first case $i\\in N_{u}$ and $j\\in A_{u}$ is\nimpossible. Indeed, applying $j\\ge\\min\\{J_{\\p(u)}\\}>\\max\\{J_{u}\\backslash J_{\\p(u)}\\}\\ge i$\nwould yield a contradiction with $i\\ge j$. \n\\item Taking the union of the two remaining cases yields $i\\in J_{u}=A_{u}\\cup N_{u}$\nand $j\\in N_{u}$. It follows from the raster property that $\\idx(i,j)$\nis maximized with $i=\\max J_{u}$ and $j=\\max N_{u}$.\n\\end{itemize}\n\\end{itemize}\nWith the two claims established, the hypothesis that $\\min\\{J_{\\p(u)}\\}>\\max\\{J_{u}\\backslash J_{\\p(u)}\\}$\nimplies that $\\alpha>\\gamma$, and therefore $\\idx(\\alpha,\\alpha)>\\idx(\\beta,\\gamma)$\nas desired. \\qed\n\\end{proof}","start":64900,"end":67246},{"raw_tex":"\\begin{proposition}[RIP$\\implies$ZF]\n\\label{prop:rip->zf}Let $J_{1},J_{2},\\dots,J_{\\ell}$ with $\\bigcup_{j=1}^{\\ell}J_{j}=\\{1,2,\\dots,n\\}$\nsatisfy the sorted running intersection property. Then, $F=\\bigcup_{j=1}^{\\ell}\\clique(J_{j})$\nis zero-fill, and we have $\\omega(F)=\\max_{j}|J_{j}|$.\n\\end{proposition}","start":67535,"end":67841},{"raw_tex":"\\begin{lemma}\n\\label{lem:colrip}Let $J_{1},J_{2},\\dots,J_{\\ell}$ with $\\bigcup_{j=1}^{\\ell}J_{j}=\\{1,2,\\dots,n\\}$\nsatisfy the sorted running intersection property. For every $j$-th\ncolumn in $F=\\bigcup_{j=1}^{\\ell}\\clique(J_{j})$, there exists some\n$J_{w}$ such that $\\col_{F}(j)\\subseteq J_{w}.$\n\\end{lemma}","start":67974,"end":68282},{"raw_tex":"\\begin{proof}\nLet $V=\\{1,2,\\dots,n\\}$. For arbitrary $j\\in V$, denote $J_{w}$\nas the last subset in the sequence $J_{1},J_{2},\\dots,J_{\\ell}$ for\nwhich $j\\in J_{w}$. This choice must exist, because $V=\\bigcup_{k=1}^{\\ell}J_{k}$.\nFor every arbitrary $i\\in\\col_{F}(j)$, we will prove that $i\\in J_{w}$\nalso holds:\n\\begin{itemize}\n\\item There exists $u\\le w$ for which $i,j\\in J_{u}$. Indeed, $(i,j)\\in F$\nand $F=\\bigcup_{k=1}^{\\ell}\\clique(J_{k})$ imply $(i,j)\\in\\clique(J_{u})$\nfor some $J_{u}$, or equivalently $i,j\\in J_{u}$. Given that $j\\in J_{u}$\nand $w=\\max\\{k:j\\in J_{k}\\}$ by definition, it follows that $u\\le w$.\n\\item If $u=w$, then $i\\in J_{w}$ holds because $i,j\\in J_{u}$. Otherwise,\nif $u<w$, we use the sorted running intersection property to assert\nthe following\n\\begin{gather}\nu<w,\\qquad i,j\\in J_{u}\\quad\\implies\\quad i,j\\in J_{\\p(u)}.\\label{eq:par1}\n\\end{gather}\nThe fact that $j\\in J_{\\p(u)}$ follows directly the running intersection\nproperty $j\\in J_{u}\\cap J_{w}\\subseteq J_{\\p(u)}$ for $w>u$. By\ncontradiction, suppose that $i\\notin J_{\\p(u)}$. Then, given that\n$i\\in J_{u}$, it follows from the sorted property that $j\\ge\\min\\{J_{\\p(u)}\\}>\\max\\{J_{u}\\backslash J_{\\p(u)}\\}\\ge i,$\nwhich contradicts our initial hypothesis that $i>j$. \n\\item Inductively reapplying (\\ref{eq:par1}), as in $i,j\\in J_{\\p(\\p(u))}$\nand $i,j\\in J_{\\p(\\p(\\p(u)))}$, we arrive at some $v$ such that\n$i,j\\in J_{\\p(v)}$ and $p(v)=w$. The induction must terminate with\n$p(v)\\ge w$ because each $\\p(u)>u$ by construction. It is impossible\nfor $\\p(v)>w$ to occur, because $j\\in J_{\\p(v)}$ and $w=\\max\\{k:j\\in J_{k}\\}$\nby definition. We conclude that $i\\in J_{w}$, as desired.\n\\end{itemize}\n\\qed\n\\end{proof}","start":68283,"end":69982},{"raw_tex":"\\begin{proof}[Proof of \\propref{rip->zf}]To prove that $F$ is zero-fill,\nwe observe, for arbitrary $(i,k)\\in F$ and $(j,k)\\in F$ with $i>j>k$\nthat $i,j\\in\\col_{F}(k)$. Therefore, it follows from \\lemref{colrip}\nthat there exists $J_{w}$ such that $i,j,k\\in J_{w}$. We conclude\nthat $(i,j)\\in F,$ because $F=\\bigcup_{j=1}^{\\ell}\\clique(J_{j})$\nand $i,j\\in J_{w}$ for some $1\\le w\\le\\ell$. \n\nTo prove that $\\omega(F)=\\max_{j}|J_{j}|$, we choose $k=\\arg\\max_{j}|\\col_{F}(j)|$.\nIt follows from Lemma~\\ref{lem:colrip} that there exists $J_{w}$\nsuch that $\\col_{F}(k)\\subseteq J_{w}$, and therefore $\\omega(F)=|\\col_{F}(k)|\\le|J_{w}|$.\nFinally, given that $\\clique(J_{w})\\subseteq F$ it follows that $(i,j)\\in F$\nholds for all $i\\in J_{w}$ where $j=\\min J_{w}$. Therefore, we conclude\nthat $J_{w}\\subseteq\\col_{F}(j)$, and therefore $|J_{w}|\\le|\\col_{F}(j)|\\le|\\col_{F}(k)|=\\omega(F)$.\n\\qed \\end{proof}","start":70128,"end":71024},{"raw_tex":"\\begin{proof}[Proof of \\lemref{quadlift}]For an arbitrary sparsity\npattern $E$ of order $n$, let $F=\\chol(E)$ and $J_{k}\\equiv\\col_{F}(k)$.\nFirst, it follows from \\propref{zf->rip} that $J_{1},J_{2},\\dots,J_{n}$\nsatisfy the ordered running intersection property. Therefore, it follows\nfrom \\lemref{lift}, $J_{1}^{(2)},J_{2}^{(2)},\\dots,J_{n}^{(2)}$\ndefined in (\\ref{eq:Jk2_def}) satisfy the same property. Finally,\nwe verify that \n\\begin{align*}\n\\supp(\\P_{k})\\equiv & \\left\\{ \\alpha:\\P_{k}[\\alpha,\\beta]\\ne0\\right\\} =\\left\\{ \\alpha:\\P_{k}^{T}e_{\\alpha}\\ne0\\right\\} \\\\\n\\overset{\\text{(a)}}{=} & \\left\\{ \\idx_{F}(i,j):\\P_{k}^{T}\\svec_{F}(e_{i}e_{j}^{T})\\ne0,\\;(i,j)\\in F\\right\\} \\\\\n\\overset{\\text{(b)}}{=} & \\left\\{ \\idx_{F}(i,j):(e_{i}e_{j}^{T})[J_{k},J_{k}]\\ne0,\\;i\\ge j\\right\\} =J_{k}^{(2)}.\n\\end{align*}\nEquality (a) is obtained by substituting $e_{\\alpha}=\\svec_{F}(e_{i}e_{j}^{T})$\nfor $(i,j)\\in F$. Equality (b) follows the identity $\\P_{k}^{T}\\svec_{F}(Y)=\\svec(Y[J_{k},J_{k}])$,\nwhich was used to define $\\P_{k}$. Therefore, $F^{(2)}=\\bigcup_{k=1}^{n}J_{k}^{(2)}$\nand we conclude via \\propref{rip->zf} that $\\omega(F^{(2)})=\\max_{k}|J_{k}^{(2)}|=\\max_{k}\\frac{1}{2}|J_{k}|(|J_{k}|+1)$,\nand we recall that $\\max_{k}|J_{k}|=\\omega(F)=\\omega(E)$ by definition.\n\\qed \\end{proof}","start":71246,"end":72527},{"raw_tex":"\\begin{lemma}\nFor every $i\\in\\{1,2,\\dots,m\\}$, there exists $k\\in\\{1,2,\\dots,n\\}$\nsuch that $\\supp(a_{i})\\subseteq\\supp(\\o{\\P}_{k})[V^{(2)}]$ where\n$V^{(2)}=\\{\\idx_{\\overline{F}}(i,j):i,j\\in F\\}$. \n\\end{lemma}","start":73318,"end":73527},{"raw_tex":"\\begin{proof}\nIt follows by repeating the proof of \\lemref{quadlift} that\n\\begin{gather*}\n\\supp(\\o{\\P}_{k})=\\{\\idx_{\\o F}(i,j):i,j\\in\\col_{\\o F}(k),i\\ge j\\},\\\\\n\\supp(\\o{\\P}_{k})[V^{(2)}]=\\{\\idx_{F}(i,j):i,j\\in\\col_{\\o F}(k),i\\ge j\\}.\n\\end{gather*}\nOur desired claim follows via the following sequence of inclusions\n\\begin{align*}\n\\supp(a_{i}) & =\\{v:a_{i}[v]\\ne0\\}=\\{\\idx_{F}(u,v):A_{i}[u,v]\\ne0,\\quad(u,v)\\in F\\}\\\\\n & \\overset{\\text{(a)}}{\\subseteq}\\{\\idx_{F}(u,v):u,v\\in\\supp(A_{i}),\\quad(u,v)\\in F\\}\\\\\n & \\overset{\\text{(b)}}{\\subseteq}\\{\\idx_{F}(u,v):u,v\\in\\col_{\\o F}(k),\\quad(u,v)\\in F\\}=\\supp(\\o{\\P}_{k})[V^{(2)}].\n\\end{align*}\nInclusion (a) is because $\\spar(A_{i})\\subseteq\\clique(\\supp(A_{i}))$.\nInclusion (b) is true via the following: If $J=\\supp(A_{i})$ satisfies\n$\\clique(J)\\subseteq\\o F$, then $J\\subseteq\\col_{\\o F}(k)$ where\n$k=\\min J$. Indeed, we have $k\\in\\col_{\\o F}(k)$ by definition.\nFor any arbitrary $j\\in J$ with $j>k$, we must have an edge $(j,k)\\in\\clique(J)\\subseteq\\o F$,\nand therefore $j\\in\\col_{\\o F}(k)$. \\qed\n\\end{proof}","start":73528,"end":74581},{"raw_tex":"\\begin{lemma}\nFor every $k\\in\\{1,2,\\dots,n\\}$, we have $\\supp(\\P_{k})\\subseteq\\supp(\\o{\\P}_{k})[V^{(2)}]$\nwhere $V^{(2)}=\\{\\idx_{\\overline{F}}(i,j):i,j\\in F\\}$. \n\\end{lemma}","start":74583,"end":74756},{"raw_tex":"\\begin{proof}\nIt again follows by repeating the proof of \\lemref{quadlift} that\n\\begin{align*}\n\\supp(\\P_{k}) & =\\{\\idx_{F}(i,j):i,j\\in\\col_{F}(k),i\\ge j\\}\\\\\n & \\overset{\\text{(a)}}{\\subseteq}\\{\\idx_{F}(i,j):i,j\\in\\col_{\\o F}(k),i\\ge j\\}=\\supp(\\o{\\P}_{k})[V^{(2)}].\n\\end{align*}\nInclusion (a) is true because $E\\subseteq\\o E$ implies $F=\\chol(E)\\subseteq\\chol(\\o E)=\\o F$\nvia \\propref{del}, and therefore $\\col_{F}(k)\\subseteq\\col_{\\o F}(k)$.\n\\qed\n\\end{proof}","start":74757,"end":75215},{"raw_tex":"\\begin{lemma}[Linear independence]\nWe have $\\A^{T}y=0$ if and only $y=0$. \n\\end{lemma}","start":99320,"end":99406},{"raw_tex":"\\begin{proof}\nWe will prove $\\sum_{k=1}^{n}\\P_{k}\\P_{k}^{T}\\succeq I$, which implies\n$\\A\\A^{T}\\succeq I$ and hence $\\|\\A^{T}y\\|\\ge\\|y\\|$. For arbitrary\n$Y\\in\\S_{F}^{n}$ with $y=\\svec_{F}(Y)$, we observe that \n\\[\n\\|\\P_{k}^{T}y\\|^{2}=\\|\\svec(Y[J_{k},J_{k}])\\|^{2}=\\sum_{i,j\\in J_{k}}(Y[i,j])^{2}=\\sum_{(i,j)\\in\\clique(J_{k})}\\gamma_{ij}(Y[i,j])^{2}\n\\]\nwhere $\\gamma_{ij}=1$ if $i=j$ and $\\gamma_{ij}=2$ if $i\\ne j$.\nTherefore, we have\n\\[\n\\sum_{k=1}^{n}\\|\\P_{k}^{T}y\\|^{2}=\\sum_{k=1}^{n}\\sum_{(i,j)\\in\\clique(J_{k})}\\gamma_{ij}(Y[i,j])^{2}\\ge\\sum_{(i,j)\\in F}\\gamma_{ij}(Y[i,j])^{2}=\\|y\\|^{2}.\n\\]\nThe inequality is because $F=\\bigcup_{k=1}^{n}\\clique(J_{k})$. \\qed\n\\end{proof}","start":99407,"end":100080},{"raw_tex":"\\begin{lemma}[Strong duality is attained]\nUnder \\asmref{strict}, there exists $x^{\\star},s^{\\star}\\in\\K$\nsatisfying $\\A x^{\\star}=\\b,$ $\\A^{T}y^{\\star}+s^{\\star}=\\c,$ $(x^{\\star})^{T}s^{\\star}=0$. \n\\end{lemma}","start":100082,"end":100291},{"raw_tex":"\\begin{proof}\nDefine $P_{j}$ implicitly to satisfy $P_{j}^{T}x=x[\\col_{F}(j)]$\nfor all $x\\in\\R^{m}$. \\asmref{strict} says that there exists $\\hat{X}\\succeq0$\nand $\\hat{y}\\le0$ and that satisfy $\\AA(\\hat{X})\\le b$ and $\\AA^{T}(\\hat{y})+\\hat{S}=C$\nand $\\inner C{\\hat{X}}=\\inner b{\\hat{y}}$. It follows from \\citep[Theorem~9.2]{vandenberghe2015chordal}\nthat $\\hat{S}=C-\\AA^{T}(\\hat{y})\\in\\S_{F}^{n}\\cap\\S_{+}^{n}$ if and\nonly if there exists $\\hat{S}_{j}\\succeq0$ such that $\\hat{S}=\\sum_{j=1}^{n}P_{j}\\hat{S}_{j}P_{j}^{T}$.\nNow, we turn to the primal-dual pair defined by the data $(\\A,\\b,\\c,\\K)$\nin (\\ref{eq:AKdef}), which is written \n\\begin{align*}\n & \\min_{v\\in\\R^{m},V_{j}\\in\\S^{\\omega_{j}}}\\left\\{ \\inner bv:\\begin{array}{c}\n\\proj_{F}\\left[\\AA^{T}(-v)+\\sum_{j=1}^{n}P_{j}V_{j}P_{j}^{T}-C\\right]=0,\\\\\nv\\ge0,\\quad V_{1},\\dots,V_{n}\\succeq0\n\\end{array}\\right\\} \\\\\n= & \\max_{Y\\in\\S_{F}^{n}}\\left\\{ \\inner{-C}Y:\\AA(Y)\\le b,\\;-P_{j}^{T}YP_{j}\\preceq0\\text{ for all }j\\in\\{1,2,\\dots,n\\}\\right\\} \n\\end{align*}\nWe can mechanically verify that $v^{\\star}=-\\hat{y}$ and $V_{j}^{\\star}=\\hat{S}_{j}$\nis feasible for the primal problem, and $Y^{\\star}=\\Pi_{F}(\\hat{X})$\nis feasible for the dual problem, and that the two objectives coincide\n$\\inner b{v^{\\star}}=\\inner{-C}{Y^{\\star}}$. Therefore, we conclude\nthat $x^{\\star}=(v^{\\star},\\svec(V_{1}^{\\star}),\\dots,\\svec(V_{n}^{\\star}))$\nand $y^{\\star}=\\svec_{F}(Y^{\\star})$ is a complementary solution\nsatisfying $\\A^{T}x^{\\star}=\\b,$ $\\A^{T}y^{\\star}+s^{\\star}=\\c,$\nand $\\inner{x^{\\star}}{s^{\\star}}=0$. \\qed \n\\end{proof}","start":100292,"end":101852},{"raw_tex":"\\begin{proposition}\n\\label{prop:general}Let $(\\A,\\b,\\c,\\K)$ describe an SDP in $(n,\\omega)$-standard\nform, and let $\\opt$ denote its optimal value. Let $\\Tim$ and $\\Mem$\ndenote the time and memory needed, given data $\\A\\in\\R^{M\\times N}$,\n$g\\in\\R^{M}$, and a choice of $w\\in\\interior(\\K)$, to form and solve\n$\\A\\D\\A^{T}\\Delta y=g$ for $\\Delta y\\in\\R^{M}$, where $\\D^{-1}=\\nabla^{2}f(w)$\nand $f=-\\log\\det(w)$. Then, a general-purpose interior-point method\ncomputes $(x,y)$ that satisfy the following in $O(\\sqrt{n}\\log(1/\\epsilon))$\niterations\n\\begin{gather*}\n\\c^{T}x\\le\\mathrm{opt}+n\\cdot\\epsilon,\\quad\\|\\A x-\\b\\|\\le\\epsilon,\\quad x\\in\\K,\\\\\n\\b^{T}y\\ge\\mathrm{opt}-n\\cdot\\epsilon,\\quad\\c-\\A^{T}y+\\epsilon\\one_{\\K}\\in\\K,\n\\end{gather*}\nwith per-iteration costs of $O(\\omega^{2}n+\\nnz(\\A)+\\Tim)$ time and\n$O(\\omega n+\\nnz(\\A)+\\Mem)$ memory.\n\\end{proposition}","start":102105,"end":102959},{"raw_tex":"\\begin{lemma}[$\\epsilon$-accurate and $\\epsilon$-feasible]\n\\label{lem:eps}Let there exist $(x^{\\star},y^{\\star})$ such that\n$\\A x^{\\star}=\\b$ with $x^{\\star}\\in\\K$ and $\\c-\\A^{T}y^{\\star}\\equiv s^{\\star}\\in\\K$\nthat satisfies strong duality $\\b^{T}y^{\\star}=\\c^{T}x^{\\star}$.\nThen, a point $(x,y,s,\\tau,\\theta,\\kappa)$ that is feasible for (\\ref{eq:hsd})\nand satisfies the optimality condition (\\ref{eq:short-step}) also\nsatisfies the following, where $\\rho=1+\\one_{\\K}^{T}(x^{\\star}+s^{\\star})$:\n\\begin{align*}\n\\|\\A(x/\\tau)-\\b\\| & \\le\\frac{\\rho\\|r_{p}\\|}{\\gamma}\\cdot\\epsilon, & \\|\\A^{T}(y/\\tau)+(s/\\tau)-\\c\\| & \\le\\frac{\\rho\\|r_{d}\\|}{\\gamma}\\cdot\\epsilon, & \\frac{(x/\\tau)^{T}(s/\\tau)}{n+1}\\le\\frac{\\rho^{2}}{\\gamma^{2}}\\cdot\\epsilon.\n\\end{align*}\n\\end{lemma}","start":105764,"end":106525},{"raw_tex":"\\begin{proof}\nFirst, observe that $(\\hat{x},\\hat{y},\\hat{s},\\hat{\\tau},\\hat{\\theta},\\hat{\\kappa})=(\\hat{\\tau}x^{\\star},\\hat{\\tau}y^{\\star},\\hat{\\tau}s^{\\star},\\hat{\\tau},0,0)$\nwith $\\hat{\\tau}=(n+1)/\\rho$ is a solution to (\\ref{eq:hsd}), because\n$r_{d}^{T}x^{\\star}+r_{p}^{T}y^{\\star}+r_{c}=\\one_{\\K}^{T}(x^{\\star}+s^{\\star})+1=\\rho$.\nNext, we prove that if $(x,y,s,\\tau,\\theta,\\kappa)$ is feasible for\n(\\ref{eq:hsd}) and satisfies (\\ref{eq:short-step}), then $\\tau\\ge\\frac{\\gamma}{\\rho}$.\nIndeed, the skew-symmetry of (\\ref{eq:hsdfeas}) yields $\\theta=\\frac{x^{T}s+\\tau\\kappa}{n+1}$\nand $(x-\\hat{x})^{T}(s-\\hat{s})+(\\tau-\\hat{\\tau})(\\kappa-\\hat{\\kappa})=0.$\nRearranging yields $(n+1)\\theta=x^{T}s+\\tau\\kappa=\\hat{x}^{T}s+x^{T}\\hat{s}+\\tau\\hat{\\kappa}+\\hat{\\tau}\\kappa$\nand hence $\\kappa\\le(n+1)\\theta/\\hat{\\tau}$. Under (\\ref{eq:short-step}),\nwe have $\\theta=\\frac{x^{T}s+\\tau\\kappa}{n+1}\\le\\epsilon$ and $\\tau\\ge\\frac{\\gamma\\epsilon}{\\kappa}\\ge\\frac{\\gamma\\theta}{\\kappa}\\ge\\frac{\\gamma\\theta}{(n+1)\\theta/\\hat{\\tau}}=\\frac{\\gamma}{n+1}\\hat{\\tau}=\\frac{\\gamma}{\\rho}$.\nFinally, divide (\\ref{eq:hsdfeas}) through by $\\tau$ and substitute\n$1/\\tau\\le\\rho/\\gamma$. \\qed\n\\end{proof}","start":106526,"end":107705},{"raw_tex":"\\begin{proof}[Proof of \\propref{general}]Recall that $(\\A,\\b,\\c,\\K)$\ndescribe an SDP in $(n,\\omega)$-standard form, and hence there exist\n$(x^{\\star},y^{\\star})$ such that $\\A x^{\\star}=\\b$ with $x^{\\star}\\in\\K$\nand $\\c-\\A^{T}y^{\\star}\\in\\K$ that satisfies strong duality $\\b^{T}y^{\\star}=\\c^{T}x^{\\star}$.\nCombining \\lemref{eps} and \\lemref{short-step} shows that iterations\nin (\\ref{eq:hsd_ipm}) converges to the desired $\\epsilon$-accurate,\n$\\epsilon$-feasible iterate after $O(\\sqrt{n}\\log(1/\\epsilon))$\niterations. The cost of each iteration is essentially equal to the\ncost of computing the search direction in (\\ref{eq:hsd_nt}). We account\nfor this cost via the following two steps:\n\\begin{enumerate}\n\\item (Scaling point) We partition $x=[\\svec(X_{j})]_{j=1}^{\\ell}$ and\n$s=[\\svec(S_{j})]_{j=1}^{\\ell}$. Then, the scaling point $w=[\\svec(W_{j})]_{j=1}^{\\ell}$\nis given in closed-form as $W_{j}=S_{j}^{1/2}(S_{j}^{1/2}X_{j}S_{j}^{1/2})^{-1/2}S_{j}^{1/2}$~\\citep[Section~5]{sturm2002implementation}.\nNoting that each $W_{j}$ is at most $\\omega\\times\\omega$, the cost\nof forming $w$ is at most of order \n\\[\n\\sum_{j=1}^{\\ell}\\omega_{j}^{3}\\le\\omega^{2}\\sum_{j=1}^{\\ell}\\omega_{j}=\\omega^{2}n\\text{ time},\\quad\\sum_{j=1}^{\\ell}\\omega_{j}^{2}\\le\\omega\\sum_{j=1}^{\\ell}\\omega_{j}=\\omega n\\text{ memory.}\n\\]\nBy this same calculation, it follows that the Hessian matrix-vector\nproducts $\\nabla^{2}f(w)x=[\\svec(W_{j}^{-1}X_{j}W_{j}^{-1})]_{j=1}^{\\ell}$\nand $\\nabla^{2}f(w)^{-1}x=[\\svec(W_{j}X_{j}W_{j})]_{j=1}^{\\ell}$\nalso cost $O(\\omega^{2}n)$ time and $O(\\omega n)$ memory. \n\\item (Search direction) Using elementary but tedious linear algebra, we\ncan show that if\\begin{subequations}\\label{eq:newt}\n\\begin{equation}\n(\\A\\D\\A^{T})\\begin{bmatrix}v_{1} & v_{2} & v_{3}\\end{bmatrix}=\\begin{bmatrix}0 & -\\b & r_{p}\\end{bmatrix}-\\A\\D\\begin{bmatrix}d & \\c & r_{d}\\end{bmatrix}\\label{eq:hsd_nrm}\n\\end{equation}\nwhere $\\D=[\\nabla^{2}f(w)]^{-1}$ and $d=-s-\\mu^{+}\\nabla f(x)$,\nand \n\\begin{equation}\n\\begin{bmatrix}u_{1} & u_{2} & u_{3}\\end{bmatrix}=\\D(\\begin{bmatrix}d & c & r_{d}\\end{bmatrix}+\\A^{T}\\begin{bmatrix}v_{1} & v_{2} & v_{3}\\end{bmatrix}),\\label{eq:hsd_back1}\n\\end{equation}\nthen\n\\begin{align}\n\\left(\\begin{bmatrix}-\\D_{0}^{-1} & -r_{c}\\\\\nr_{c} & 0\n\\end{bmatrix}-\\begin{bmatrix}\\c & r_{d}\\\\\n-\\b & r_{p}\n\\end{bmatrix}^{T}\\begin{bmatrix}u_{2} & u_{3}\\\\\nv_{2} & v_{3}\n\\end{bmatrix}\\right)\\begin{bmatrix}\\Delta\\tau\\\\\n\\Delta\\theta\n\\end{bmatrix} & =\\begin{bmatrix}-d_{0}\\\\\n0\n\\end{bmatrix}-\\begin{bmatrix}\\c & r_{d}\\\\\n-\\b & r_{p}\n\\end{bmatrix}^{T}\\begin{bmatrix}u_{1}\\\\\nv_{1}\n\\end{bmatrix},\\\\\n\\begin{bmatrix}\\Delta x\\\\\n\\Delta y\n\\end{bmatrix} & =\\begin{bmatrix}u_{1}\\\\\nv_{1}\n\\end{bmatrix}-\\begin{bmatrix}u_{1} & u_{2}\\\\\nv_{1} & v_{2}\n\\end{bmatrix}\\begin{bmatrix}\\Delta\\tau\\\\\n\\Delta\\theta\n\\end{bmatrix},\\\\\n\\Delta s & =d-\\D^{-1}\\Delta x,\\\\\n\\Delta\\kappa & =d_{0}-\\D_{0}^{-1}\\Delta\\tau,\\label{eq:hsd_back2}\n\\end{align}\n\\end{subequations}where $\\D_{0}=\\tau/\\kappa$ and $d_{0}=-\\kappa+\\mu^{+}\\tau^{-1}$.\nTherefore, we conclude that the cost of computing the search direction\nin (\\ref{eq:hsd_nt}) is equal to the cost of solving $O(1)$ instances\nof the Schur complement equation $\\A\\D\\A^{T}\\Delta y=g$, plus $O(1)$\nmatrix-vector products with $\\A,\\A^{T},\\D,\\D^{-1}$, for a total cost\nof $O(\\Tim+\\nnz(\\A)+\\omega^{2}n)$ time and $O(\\Mem+\\nnz(\\A)+\\omega n)$\nmemory respectively. Note that $\\A\\D\\A^{T}\\succ0$ because $\\A\\A^{T}\\succ0$\nby the linear independence assumption, and $\\D^{-1}=\\nabla^{2}f(w)\\succ0$\nfor all $w\\in\\interior(\\K)$.\n\\end{enumerate}\n\\qed \\end{proof}","start":107707,"end":111259}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:sdp"],"declared_external":false,"source_only":false,"title":"原始半定规划","source_units":[{"raw_tex":"Consider directly applying a general-purpose interior-point method\nsolver, like SeDuMi~\\citep{sturm1999using}, SDPT3~\\citep{toh1999sdpt3},\nand MOSEK~\\citep{mosek2019}, to solve the standard-form semidefinite\nprogram to high accuracy:\n\\begin{equation}\n\\min_{X\\in\\S^{n}}\\;\\inner CX\\quad\\text{s.t.}\\;X\\succeq0,\\quad\\inner{A_{i}}X\\le b_{i}\\;\\text{for all }i\\in\\{1,2,\\dots,m\\}.\\tag{SDP}\\label{eq:sdp}\n\\end{equation}\nHere, $\\S^{n}$ denotes the set of $n\\times n$ real symmetric matrices\nwith inner product $\\inner{A_{i}}X=\\tr(A_{i}X)$, and $X\\succeq0$\nmeans that $X$ is positive semidefinite. At each iteration, the $n\\times n$\nmatrix variable $X$ is generally fully dense, even when problem data\n$C,A_{1},\\dots,A_{m}\\in\\S^{n}$ and $b_{1},\\dots,b_{m}\\in\\R$ are\nsparse. The per-iteration cost of the solver is usually at least $\\Omega(n^{3})$\ntime, which in practice limits the value of $n$ to no more than a\nfew thousand.\n","start":7013,"end":7929}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":["eq:Edef"],"declared_external":false,"source_only":false,"title":"聚合图与对称重排","source_units":[{"raw_tex":"In order for chordal conversion to be fast, a well-known \\emph{necessary}\ncondition is for the underlying aggregate sparsity graph, which is\ndefined $G=(V,E)$ with\n\\begin{gather}\nV=\\{1,2,\\dots,n\\},\\quad E=\\spar(C)\\cup\\spar(A_{1})\\cup\\cdots\\cup\\spar(A_{m}),\\label{eq:Edef}\\\\\n\\text{where }\\spar(C)\\equiv\\{(i,j):C[i,j]\\ne0\\text{ for }i>j\\},\\nonumber \n\\end{gather}\nto be ``tree-like'' with a small\\emph{ treewidth} $\\tw(G)\\ll n$.\nWe defer a formal definition of the treewidth to \\defref{tw}, and\nonly note that the value of $\\omega$ is lower-bounded as $\\omega\\ge1+\\tw(G)$.\nIn other words, while $\\omega$ can be decreased by symmetrically\nreordering the columns and rows of the data matrices, as in $C\\gets\\Pi C\\Pi^{T}$\nand $A_{i}\\gets\\Pi A_{i}\\Pi^{T}$ for some permutation matrix $\\Pi$,\nactually achieving $\\omega\\ll n$ is possible only if $\\tw(G)\\ll n$.\n","start":11820,"end":12672}]},{"id":"S003","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"小聚合树宽仍不足","source_units":[{"raw_tex":"However, as illustrated by \\exaref{diag}, even on a graph with $\\tw(G)=0$,\nand even when using the optimal $\\omega=1$, chordal conversion may\nstill not be fast.\n","start":12672,"end":12833}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["eq:Ebardef"],"declared_external":false,"source_only":false,"title":"扩展聚合稀疏图","source_units":[{"raw_tex":"In this paper, we show that a \\emph{sufficient} condition for chordal\nconversion to be fast is for a supergraph $\\o G\\supseteq G$, that\nadditionally captures the correlation between constraints, to also\\footnote{Note that $\\tw(\\o G)\\ge\\tw(G)$ holds by virtue of $\\o G\\supseteq G$.}\nbe ``tree-like'' with a small treewidth $\\tw(\\o G)\\ll n$. Concretely,\nwe construct the \\emph{extended} aggregate sparsity graph $\\o G=(V,\\overline{E})$\nby forcing each constraint matrix $A_{i}$ to be its own clique in\n$G$:\n\\begin{gather}\nV=\\{1,2,\\dots,n\\},\\quad\\overline{E}=\\spar(C)\\cup\\clique(A_{1})\\cup\\cdots\\cup\\clique(A_{m})\\label{eq:Ebardef}\\\\\n\\text{where }\\clique(A)=\\{(i,j):A[i,k]\\ne0\\text{ or }A[k,j]\\ne0\\text{ for some }k\\}.\\nonumber \n\\end{gather}\n","start":12834,"end":13573}]},{"id":"S005","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"约束耦合的图解释","source_units":[{"raw_tex":"This is the union between $G$ and the constraint intersection graph~\\citep{fulkerson1965incidence}\n(or the dual graph~\\citep{dong2021nearly} or the correlative sparsity~\\citep{waki2006sums,kobayashi2008correlative})\nfor the rank-1 instance of (\\ref{eq:sdp}). In other words, we add\na new edge $(i,j)$ to $E$ whenever $x_{i}$ and $x_{j}$ appear\ntogether in a common constraint $x^{T}A_{k}x\\le b_{k}$ for some $k$.\nThe fact that this contributes $\\clique(A_{k})\\subseteq\\o E$ reflects\nthe reality that each $x^{T}A_{k}x\\le b_{k}$ densely couples all\naffected elements of $x$ together, forcing them to be optimized simultaneously.\nIn contrast, the cost $x^{T}Cx$ can be optimized sequentially over\nthe elements of $x$, which is why $\\clique(C)$ is absent.\n","start":13573,"end":14326}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"矩阵空间与稀疏投影","source_units":[{"raw_tex":"Write $\\R^{m\\times n}$ as the set of $m\\times n$ matrices with real\ncoefficients, with associated matrix inner product $\\inner AB=\\tr A^{T}B$\nand norm $\\|A\\|_{F}=\\sqrt{\\inner AA}$. Write $\\S^{n}\\subseteq\\R^{n\\times n}$\nas the set of $n\\times n$ real symmetric matrices, meaning that $X=X^{T}$\nholds for all $X\\in\\S^{n}$, and write $\\S_{+}^{n}\\subseteq\\S^{n}$\nas the set of symmetric positive semidefinite matrices. Write $\\R_{+}^{n}\\subseteq\\R^{n}$\nas the usual positive orthant.\n\nThe set of $n\\times n$ symmetric matrices with sparsity pattern $E$\ncan be defined as\n\\[\n\\S_{E}^{n}\\equiv\\left\\{ X\\in\\S^{n}:X[i,j]=X[j,i]=0\\quad\\text{for all }i\\ne j,\\quad(i,j)\\notin E\\right\\} .\n\\]\nConversely, the minimum sparsity pattern of a \\emph{symmetric} matrix\n$X\\in\\S^{n}$ is denoted\n\\[\n\\spar(X)\\equiv\\{(i,j):X[i,j]\\ne0,\\quad i\\ge j\\}.\n\\]\nWe also write $\\spar(M)\\equiv\\spar(M+M^{T})$ for a nonsymmetric matrix\n$M$ where there is no confusion. We write $\\proj_{E}(M)\\equiv\\arg\\min_{X\\in\\S_{E}^{n}}\\|M-X\\|_{F}$\nas the projection of $M\\in\\S^{n}$ onto the sparsity pattern $E$. \n","start":29321,"end":30385}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"团模式与顶点支持","source_units":[{"raw_tex":"\nWe define the dense sparsity pattern induced by $J\\subseteq\\{1,2,\\dots,n\\}$\nas follows\n\\[\n\\clique(J)=\\{(i,j):i,j\\in J,\\quad i\\ge j\\}.\n\\]\nWe also define the vertex support of a possibly nonsymmetric matrix\n$M$ as the following \n\\[\n\\supp(M)=\\{i:M[i,j]\\ne0\\quad\\text{for some }j\\}.\n\\]\nWe write $\\clique(A)\\equiv\\clique(\\supp(A))$ where there is no confusion.\n","start":30385,"end":30742}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"支持诱导的稀疏包含","source_units":[{"raw_tex":"This notation is motivated by the fact that $\\spar(A)\\subseteq\\clique(A)$\nfor $A\\in\\S^{n}$, and $\\spar(PDP^{T})\\subseteq\\clique(P)$ for $P\\in\\R^{n\\times d}$\nand dense $D\\in\\S^{d}$. \n\n","start":30742,"end":30925}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"等距向量化与栅格索引","source_units":[{"raw_tex":"Let $F$ be a sparsity pattern of order $n$ that \\emph{contains all\nof its diagonal elements}, as in $F\\supseteq\\{(i,i):i\\in\\{1,2,\\dots,n\\}\\}$.\nIn this case, $\\dim(\\S_{F}^{n})=|F|$ holds exactly, so we define\na \\emph{symmetric vectorization} operator $\\svec_{F}:\\S_{F}^{n}\\to\\R^{|F|}$\nto implement an isometry with the usual Euclidean space, as in \n\\[\n\\inner{\\svec_{F}(X)}{\\svec_{F}(Y)}=\\inner XY\\text{ for all }X,Y\\in{\\S_{F}^{n}}.\n\\]\nWe will explicitly require $\\svec_{F}(\\cdot)$ to be defined according\nto a column-stacking construction\n\\[\n\\svec_{F}(X)=(x_{j})_{j=1}^{n}\\quad\\text{where }x_{j}=\\left(X[i,i],\\quad\\sqrt{2}\\cdot(X[i,j]:i>j,\\quad(i,j)\\in F)\\right).\n\\]\nWe also define a companion indexing operator $\\idx_{F}(\\cdot,\\cdot)$\nto index elements of the vectorization $x=\\svec_{F}(X)$:\n\\begin{gather*}\nx[\\idx_{F}(i,i)]=X[i,i]\\quad\\text{for all }i\\in\\{1,2,\\dots,n\\},\\\\\nx[\\idx_{F}(i,j)]=\\sqrt{2}X[i,j]\\quad\\text{for all }i>j,\\quad(i,j)\\in F.\n\\end{gather*}\nAs we will see later, the correctness of our proof crucially relies\non the fact that $\\idx_{F}(\\cdot,\\cdot)$ implements a \\emph{raster\nordering} over the elements of $F$. \n","start":30925,"end":32057}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","labels":[],"declared_external":false,"source_only":false,"title":"Cholesky 分解与回代","source_units":[{"raw_tex":"To solve $Sx=b$ with $S\\succ0$ via Cholesky factorization, we first\ncompute the lower-triangular Cholesky factor $L=\\chol(S)$ according\nto the following recursive rule \n\\[\n\\chol\\left(\\begin{bmatrix}\\alpha & b^{T}\\\\\nb & D\n\\end{bmatrix}\\right)=\\begin{bmatrix}\\sqrt{\\alpha} & 0\\\\\n\\frac{1}{\\sqrt{\\alpha}}b & \\chol\\left(D-\\frac{1}{\\alpha}bb^{T}\\right)\n\\end{bmatrix},\\quad\\chol(\\alpha)=\\sqrt{\\alpha},\n\\]\nand then solve two triangular systems $Ly=b$ and $L^{T}x=y$ via\nback-substitution. If $S$ is sparse, then $L=\\chol(S)$ may also\nbe sparse. The sparsity pattern of $L$ can be directly computed from\nthe sparsity pattern of $S$, and without needing to examine the numerical\nvalues of its nonzeros.\n","start":32102,"end":32795}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":["def:symbchol"],"declared_external":false,"source_only":false,"title":"符号 Cholesky 闭包","source_units":[{"raw_tex":"\\begin{definition}[Symbolic Cholesky]\n\\label{def:symbchol}The \\emph{symbolic Cholesky factor} $\\chol(E)$\nof a sparsity pattern $E$ of order $n$ is defined as $\\chol(E)\\equiv E_{n+1}$\nwhere $E_{1}=E$ and\n\\[\nE_{k+1}=E_{k}\\cup(k,k)\\cup\\{(i,j):(i,k)\\in E_{k},\\quad(j,k)\\in E_{k}\\text{ for }i>j>k\\}.\n\\]\n\\end{definition}\n","start":32795,"end":33110}]},{"id":"S012","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件：George and Liu, Theorem 5.4.4；L680–682 引用。"],"source_only":false,"title":"符号分解构造开销","source_units":[{"raw_tex":"One can verify that $\\chol(\\spar(S))=\\spar(\\chol(S))$. Note that\n$\\chol(E)$ can be computed from $E$ in $O(|\\chol(E)|)$ time and\nmemory~\\citep[Theorem 5.4.4]{george1981computer}. The efficiency\nof a sparsity-exploiting algorithm for factorizing $L=\\chol(S)$ and\nsolving $Ly=b$ and $L^{T}x=y$ is determined by the \\emph{frontsize}\nof the sparse matrix $S$. \n","start":33110,"end":33467}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["def:frontsize"],"declared_external":false,"source_only":false,"title":"列集合与前沿尺寸","source_units":[{"raw_tex":"\\begin{definition}[Frontsize]\n\\label{def:frontsize}The \\emph{frontsize} $\\omega(E)$ of a sparsity\npattern $E$ is defined $\\max_{j}|\\col_{F}(j)|$ where $F=\\chol(E)$\nand $\\col_{F}(j)\\equiv\\{j\\}\\cup\\{i>j:(i,j)\\in F\\}$. The frontsize\n$\\omega(S)\\equiv\\omega(\\spar(S))$ of a symmetric matrix $S$ is the\nfrontsize of its minimum sparsity pattern. \n\\end{definition}\nIntuitively, the frontsize $\\omega(S)$ is the maximum number of nonzero\nelements in a single column of the Cholesky factor $L=\\chol(S)$.\nThe following is well-known~\\citep{george1981computer}.\n","start":33467,"end":34018}]},{"id":"S014","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"置换后的稀疏模式","source_units":[{"raw_tex":"The cost of solving $Sx=b$ with sparse $S\\succ0$ can usually be\nreduced by first permuting the rows and columns of the matrix symmetrically,\nand then solving $(\\Pi S\\Pi^{T})\\Pi x=\\Pi y$ for some permutation\nmatrix $\\Pi$. For $E=\\spar(S)$, we write $E_{\\Pi}\\equiv\\spar(\\Pi S\\Pi^{T})$\nto denote its permuted sparsity pattern. It is a fundamental result\nin graph theory and linear algebra that the problem of minimizing\nthe frontsize $\\omega(E_{\\Pi})$ over the set of $n\\times n$ permutation\nmatrices $\\Pi\\in\\Perm(n)$ is the same problem as computing the treewidth\nof the graph $G=(V,E)$.\n\n","start":35337,"end":35924}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":["def:tw"],"declared_external":false,"source_only":false,"title":"树分解及其宽度","source_units":[{"raw_tex":"\\begin{definition}[Treewidth]\n\\label{def:tw}A tree decomposition of a graph $G=(V,E)$ is a pair\n$(\\{J_{j}\\},T)$ in which each \\emph{bag} $J_{j}\\subseteq V$ is a\nsubset of vertices and $T$ is a tree such that:\n\\begin{itemize}\n\\item (Vertex cover) $\\bigcup_{j}J_{j}=V$;\n\\item (Edge cover) $\\bigcup_{j}(J_{j}\\times J_{j})\\supseteq E$;\n\\item (Running intersection) $J_{i}\\cap J_{j}\\subseteq J_{k}$ for every\n$k$ on the path of $i$ to $j$ on $T$.\n\\end{itemize}\nThe \\emph{width} of the tree decomposition is $\\max_{j}|J_{j}|-1$.\nThe \\emph{treewidth} of $G$, denoted $\\tw(G)$, is the minimum width\nover all valid tree decompositions on $G$.\n\\end{definition}\nThe connection is an immediate corollary of the following result,\nwhich establishes an equivalence between tree decompositions and the\nsparsity pattern of Cholesky factors. \n","start":35924,"end":36749}]},{"id":"S016","semantic_type":"proposition","kind":"proposition","labels":[],"declared_external":true,"declaration_text":["明确允许的 B0047；证明于 L756 延后至 George and Liu 及 Vandenberghe and Andersen。"],"source_only":false,"title":"给定树分解的消元序","source_units":[{"raw_tex":"\\begin{proposition}[Perfect elimination ordering]\nGiven a sparsity pattern $E$ of order $n$, let $G=(V,E)$ where\n$V=\\{1,2,\\dots,n\\}$. For every tree decomposition of $G$ with width\n$\\tau$, there exists a perfect elimination ordering $\\Pi\\in\\Perm(n)$\nsuch that $\\omega(E_{\\Pi})=1+\\tau$. \n\\end{proposition}\nWe defer a proof to the texts~\\citep{george1981computer,vandenberghe2015chordal},\nand only note that, given a tree decomposition of width $\\tau$, the\ncorresponding perfect elimination ordering $\\Pi$ can be found in\n$O((1+\\tau)\\cdot n)$ time. \n","start":36749,"end":37297}]},{"id":"S017","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"线性锥规划原对偶形式","source_units":[{"raw_tex":"The basic approach for solving an SDP using a general-purpose solver\nis to reformulate the problem into the primal or the dual of the standard-form\nlinear conic program \n\\[\n\\min_{x\\in{\\R^{q}}}\\{\\c^{T}x:\\A x={\\b},x\\in\\K\\}\\ge\\max_{y\\in{\\R^{p}}}\\{\\b^{T}y:\\c-\\A^{T}y\\in{\\K_{*}}\\}\n\\]\n where the data are the matrix $\\A\\in\\R^{p\\times q}$, vectors $\\b\\in\\R^{p}$\nand $\\c\\in\\R^{q}$, and the problem closed convex cone $\\K\\subseteq\\R^{q}$,\nand the notation $\\K_{*}$ means the dual cone of $\\K$. We specify\nthe following basic assumptions on this problem to ensure that it\ncan be solved in polynomial time using a self-dual embedding~\\citep{de2000self}.\nBelow, we denote $\\one_{\\K}$ as the identity element on the cone\n$\\K$, and recall that every semidefinite cone is self-dual $\\K=\\K_{*}$. \n","start":37976,"end":38757}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":["def:clp"],"declared_external":false,"source_only":false,"title":"标准形式的三项条件","source_units":[{"raw_tex":"\\begin{definition}[Standard-form SDP]\n\\label{def:clp}We say that the problem data $\\A\\in\\R^{p\\times q},$\n$\\b\\in\\R^{p},$ $\\c\\in\\R^{q},$ and $\\K\\subseteq\\R^{q}$ describe\nan SDP in $(n,\\omega)$-\\emph{standard form} if:\n\\begin{enumerate}\n\\item (Dimensions) The cone $\\K=\\svec(\\S_{+}^{\\omega_{1}})\\times\\cdots\\times\\svec(\\S_{+}^{\\omega_{\\ell}})$\nis the Cartesian product of semidefinite cones whose orders $\\omega_{1},\\omega_{2},\\dots,\\omega_{\\ell}$\nsatisfy \n\\[\n\\omega=\\max_{i}\\omega_{i},\\qquad n=\\sum_{i=1}^{\\ell}\\omega_{i},\\qquad{q}=\\frac{1}{2}\\sum_{i=1}^{\\ell}\\omega_{i}(\\omega_{i}+1).\n\\]\n\\item (Linear independence) $\\A^{T}y=0$ holds if and only if $y=0$. \n\\item (Strong duality is attained) There exist a choice of $x^{\\star},y^{\\star}$\nthat satisfy\n\\begin{gather*}\n\\A x^{\\star}=\\b,\\quad x^{\\star}\\in\\K,\\quad\\c-\\A^{T}y^{\\star}\\in\\K,\\quad\\c^{T}x^{\\star}=\\b^{T}y^{\\star}.\n\\end{gather*}\n\\end{enumerate}\n\\end{definition}\n%\n","start":38757,"end":39676}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":["def:ipm"],"declared_external":false,"source_only":false,"title":"通用内点求解器契约","source_units":[{"raw_tex":"\\begin{definition}[General-purpose solver]\n\\label{def:ipm}We say that $\\IPM$ implements a \\emph{general-purpose\nsolver} if it satisfies the following conditions\n\\begin{enumerate}\n\\item (Iteration count) Given data $(\\A,\\b,\\c,\\K)$ in $(n,\\omega)$-standard-form,\ncalling $(x,y)=\\IPM(\\epsilon,\\A,\\b,\\c,\\K)$ yields iterates $(x,y)\\in\\K\\times{\\R^{p}}$\nthat satisfy the following, in at most $O(\\sqrt{n}\\log(1/\\epsilon))$\niterations\n\\[\n\\|\\A x-\\b\\|\\le\\epsilon,\\quad{\\c-\\A^{T}y+\\epsilon\\cdot\\one_{\\K}\\in\\K,}\\quad\\c^{T}x-\\b^{T}y\\le\\epsilon\\cdot n.\n\\]\n\\item (Per-iteration costs) Each iteration costs an overhead of $O(\\omega^{2}n)$\ntime and $O(\\omega n)$ memory, plus the cost of solving $O(1)$ instances\nof the \\emph{Schur complement equation} \n\\[\n\\A\\nabla^{2}f(w)\\A^{T}\\Delta y=r,\\qquad f(w)=-\\log\\det(w)\n\\]\nby forming $\\Hess=\\A\\nabla^{2}f(w)\\A^{T}$, factoring $\\L=\\chol(\\Pi\\Hess\\Pi^{T})$\nand then solving $\\L z={\\Pi}r$ and $\\L^{T}(\\Pi\\Delta y)=z$. Here,\nthe fill-reducing permutation $\\Pi$ is required to be no worse than\nthe natural ordering, as in $\\omega(\\Pi\\Hess\\Pi^{T})\\le\\omega(\\Hess)$.\n\\end{enumerate}\n\\end{definition}\n","start":39676,"end":40796}]},{"id":"S020","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"求解器契约的实现说明","source_units":[{"raw_tex":"Note that \\defref{ipm} is rigorously satisfied by SeDuMi~\\citep{sturm1999using,sturm2002implementation},\nSDPT3~\\citep{toh1999sdpt3,tutuncu2003solving}, and MOSEK~\\citep{andersen2000mosek,andersen2003implementing,mosek2019}.\nGiven that the correctness of our overall claims crucially depends\non the characterization in \\defref{ipm}, we state a concrete interior-point\nmethod in \\appref{ipm} that implements these specifications. \n","start":40796,"end":41225}]},{"id":"S021","semantic_type":"algorithm","kind":"algorithm","labels":["alg:myalg"],"declared_external":false,"source_only":false,"title":"弦转换完整算法","source_units":[{"raw_tex":"\\begin{algorithm}[t]\n\\caption{\\label{alg:myalg}Chordal conversion}\n\n\\textbf{Input.} Accuracy parameter $\\epsilon>0$, problem data $C,A_{1},A_{2},\\dots,A_{m}\\in\\S^{n}$,\n$b\\in\\R^{m}$, fill-reducing permutation $\\Pi$.\n\n\\textbf{Output.} Approximate solutions $U\\in\\R^{n\\times\\omega}$ and\n$v\\in\\R^{m}$ to the following primal-dual pair:\n\\[\n\\min_{U}\\left\\{ \\inner C{UU^{T}}:\\inner{A_{i}}{UU^{T}}\\le b_{i}\\text{ for all }i\\right\\} \\ge\\max_{v\\le0}\\left\\{ \\inner bv:{\\textstyle \\sum_{i}}v_{i}A_{i}\\preceq C\\right\\} .\n\\]\n\n\\textbf{Algorithm.}\n\\begin{enumerate}\n\\item (Symbolic factorization) Pre-order all data matrices $\\tilde{A}_{i}=\\Pi A_{i}\\Pi^{T}$\nand $\\tilde{C}=\\Pi C\\Pi^{T}$. Compute the permuted aggregate sparsity\npattern $E=\\spar(\\tilde{C})\\cup\\bigcup_{i}\\spar(\\tilde{A}_{i})$,\nits lower-triangular symbolic Cholesky factor $F=\\chol(E)$, and define\nthe following\n\\[\nJ_{j}=\\col_{F}(j)\\equiv\\{j\\}\\cup\\{i>j:(i,j)\\in F\\},\\quad\\omega_{j}\\equiv|J_{j}|,\\quad\\omega\\equiv\\max_{j}\\omega_{j}.\n\\]\n\\item (Numerical solution) Call $(x,y)=\\IPM(\\epsilon,\\A,\\b,\\c,\\K)$ where\n$\\IPM$ is a general-purpose solver (\\defref{ipm}), and the problem\ndata $\\A,\\b,\\c,\\K$ implement the following\n\\[\nY^{\\star}=\\arg\\max_{Y\\in\\S_{F}^{n}}\\inner{-\\tilde{C}}Y\\text{ s.t. }\\begin{array}{cc}\nb_{i}-\\inner{\\tilde{A}_{i}}Y\\ge0 & \\quad\\text{for all }i\\in\\{1,2,\\dots,m\\}\\\\\nY[J_{j},J_{j}]\\succeq0 & \\quad\\text{for all }j\\in\\{1,2,\\dots,n\\}\n\\end{array}\n\\]\nas an instance of $y^{\\star}=\\arg\\max_{y}\\{\\b^{T}y:\\c-\\A^{T}y\\in\\K\\}$\nwith $y^{\\star}=\\svec_{F}(Y^{\\star})$.\n\\item (Back substitution) Recover $Y\\in\\S_{F}^{n}$ from $y=\\svec_{F}(Y)$,\nand compute $\\delta=-\\min_{j}\\{0,\\lambda_{\\min}(Y[J_{j},J_{j}])\\}$.\nSolve the positive semidefinite matrix completion\n\\[\n\\text{find }\\tilde{U}\\in\\R^{n\\times\\omega}\\text{ such that }(\\tilde{U}\\tilde{U}^{T})[J_{j},J_{j}]=Y[J_{j},J_{j}]+\\delta I\\text{ for all }j\n\\]\nusing \\citep[Algorithm~2]{sun2015decomposition}. Output $v=-(x_{i})_{i=1}^{m}$\nand $U=\\Pi^{T}\\tilde{U}$.\n\\end{enumerate}\n\\end{algorithm}\n","start":41250,"end":43246}]},{"id":"S022","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件：Dancis, Theorem 1.5；Sun, Algorithm 2。算法引用位于 L873–877，成本说明位于 L888–890。"],"source_only":false,"title":"低秩弦补全及恢复成本","source_units":[{"raw_tex":" \\algref{myalg} summarizes the standard implementation of chordal\nconversion, which is known as the ``d-space conversion method using\nbasis representation'' in \\citet{kim2011exploiting}. Our only modification\nis to recover $X=UU^{T}$ from $Y=\\proj_{F}(X)$ in Step 3 using the\n\\emph{low-rank} chordal completion~\\citep[Theorem~1.5]{dancis1992positive},\ninstead of the \\emph{maximum determinant} chordal completion~\\citep[Theorem~2]{grone1984positive}\nas originally proposed by \\citet{fukuda2001exploiting}. We note that\nboth recovery procedures have the same complexity of $O(\\omega^{3}n)$\ntime and $O(\\omega^{2}n)$ memory, but the former puts $X$ in a more\nconvenient form~\\citep{sun2015decomposition}.\n","start":43246,"end":43949}]},{"id":"S023","semantic_type":"definition","kind":"definition","labels":["eq:AKdef-1","eq:AKdef-2"],"declared_external":false,"source_only":false,"title":"转换后的锥数据与提取矩阵","source_units":[{"raw_tex":"To state the Schur complement sparsity $E^{(2)}$ explicitly, note\nthat the problem data $(\\A,\\b,\\c,\\K)$ in Step 2 of \\algref{myalg}\nare \\begin{subequations}\\label{eq:AKdef}\n\\begin{equation}\n\\begin{array}{c}\n\\A=[\\svec_{F}(A_{1}),\\dots,\\svec_{F}(A_{m}),-\\P_{1},\\dots,-\\P_{n}],\\\\\n\\b=-\\svec_{F}(C),\\qquad\\c=(b,0),\\\\\n\\K=\\R_{+}^{m}\\times\\svec(\\S_{+}^{\\omega_{1}})\\times\\svec(\\S_{+}^{\\omega_{2}})\\times\\cdots\\times\\svec(\\S_{+}^{\\omega_{n}})\n\\end{array}\\label{eq:AKdef-1}\n\\end{equation}\nwhere $\\omega_{j}\\equiv|\\col_{F}(j)|$ and each $\\P_{j}$ is implicitly\ndefined to satisfy \n\\begin{equation}\n\\P_{j}^{T}\\svec_{F}(Y)=\\svec(Y[\\col_{F}(j),\\col_{F}(j)])\\quad\\text{for all }Y\\in\\S_{F}^{n}.\\label{eq:AKdef-2}\n\\end{equation}\n\\end{subequations}The resulting Schur complement matrix reads\n","start":44543,"end":45316}]},{"id":"S024","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:E2def","eq:Hessdef"],"declared_external":false,"source_only":false,"title":"Schur 矩阵的结构稀疏式","source_units":[{"raw_tex":"\\begin{gather}\n\\A\\nabla^{2}f(w)\\A^{T}=\\sum_{i=1}^{m}d_{i}\\svec_{F}(A_{i})\\svec_{F}(A_{i})^{T}+\\sum_{j=1}^{n}\\P_{j}\\D_{j}\\P_{j}^{T}\\label{eq:Hessdef}\\\\\n\\text{where }d_{i}=w_{i}^{-2},\\qquad\\D_{j}\\svec(X_{j})=\\svec(W_{j}^{-1}X_{j}W_{j}^{-1})\\nonumber \n\\end{gather}\nand $w=(w_{1},\\dots,w_{m},\\svec(W_{1}),\\dots,\\svec(W_{n}))\\in\\interior(\\K)$\nis a scaling point. The associated sparsity pattern, aggregated over\nall possible choices of scaling $w$, is as follows\n\\begin{align}\nE^{(2)} & =\\{(i,j):(\\A\\nabla^{2}f(w)\\A^{T})[i,j]\\ne0\\text{ for some }w\\in\\interior(\\K)\\}\\nonumber \\\\\n & =\\left(\\bigcup_{i=1}^{m}\\spar(a_{i}a_{i}^{T})\\right)\\cup\\left(\\bigcup_{j=1}^{n}\\spar(\\P_{j}\\D_{j}\\P_{j}^{T})\\right)\\nonumber \\\\\n & =\\left(\\bigcup_{i=1}^{m}\\clique(\\supp(a_{i}))\\right)\\cup\\left(\\bigcup_{j=1}^{n}\\clique(\\supp(\\P_{j}))\\right)\\label{eq:E2def}\n\\end{align}\nwhere we have written $a_{i}=\\svec_{F}(A_{i})$. The result below\nsays that if $\\omega(\\o E)=O(1)$, then $\\omega(\\o E^{(2)})=O(1)$. \n","start":45316,"end":46292}]},{"id":"S025","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"转换规模的记号说明","source_units":[{"raw_tex":"Let us now give an end-to-end complexity guarantee for \\algref{myalg}.\nWe will need the following assumption to ensure that the data $(\\A,\\b,\\c,\\K)$\npreviously defined in (\\ref{eq:AKdef}) specifies an SDP in $(N,\\omega)$-standard\nform, where $N=m+\\omega n$ and $\\omega\\equiv\\omega(E)$. \n\n","start":50696,"end":50984}]},{"id":"S026","semantic_type":"assumption","kind":"assumption","labels":["asm:strict"],"declared_external":false,"source_only":false,"title":"原问题强对偶达到","source_units":[{"raw_tex":"\\begin{assume}[Strong duality is attained]\\label{asm:strict}There\nexists a primal-dual pair $X^{\\star}\\succeq0$ and $v^{\\star}\\le0$\nthat are feasible $\\inner{A_{i}}{X^{\\star}}\\le b_{i}$ for all $i$\nand $\\sum_{i}v_{i}^{\\star}A_{i}\\preceq C$ and coincide in their objectives\n$\\inner C{X^{\\star}}=\\inner b{v^{\\star}}$.\n\n\\end{assume}\n","start":50984,"end":51314}]},{"id":"S027","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"保序诱导子模式","source_units":[{"raw_tex":"$E$ of order $n$ can be viewed as the edge set of an undirected\ngraph $G=(V,E)$ on vertices $V=\\{1,2,\\dots,n\\}$. The underlying\nprinciple behind our proof is the fact that the frontsize is monotone\nunder the subgraph relation: if $G'=(V',E')$ is a subgraph of $G=(V,E)$,\nthen $\\omega(E)\\ge\\omega(E')$. \n\nTo state this formally, we denote the sparsity pattern induced by\na subset of vertices $U=\\{u_{1},u_{2},\\dots,u_{p}\\}\\subseteq V$ as\nfollows\n\\[\nE[U]\\equiv\\{(i,j):(u_{i},u_{j})\\in E\\}\\text{ where }u_{1}<u_{2}<\\cdots<u_{p}.\n\\]\nNote that we always sort the elements of $U$. Our definition is made\nso that if $E=\\spar(X)$, then $E[U]=\\spar(X[U,U])$, without any\nreordering of the rows and columns. \n","start":57148,"end":57847}]},{"id":"S028","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"零填充模式","source_units":[{"raw_tex":"Our proof of \\lemref{quadlift} is based on a connection between\nzero-fill sparsity patterns, for which sequential Gaussian elimination\nresults in no additionally fill-in, and a ``sorted'' extension of\nthe running intersection property. \n\\begin{definition}[ZF]\nThe sparsity pattern $F$ is said to be\\emph{ zero-fill} if $F=\\chol(F)$.\n\\end{definition}\nEquivalently, if $F$ is zero-fill, then $(i,k)\\in F$ and $(j,k)\\in F$\nimplies $(i,j)\\in F$ for $i>j>k$ via the definition of the symbolic\nCholesky factor (\\defref{symbchol}). \n\n","start":61164,"end":61691}]},{"id":"S029","semantic_type":"definition","kind":"definition","labels":["def:rip"],"declared_external":false,"source_only":false,"title":"有序运行交性质","source_units":[{"raw_tex":"\\begin{definition}[RIP]\n\\label{def:rip}The sequence of subsets $J_{1},J_{2},\\dots,J_{\\ell}$\nwith $J_{j}\\subset\\N$ satisfies the \\emph{sorted running intersection\nproperty} if there exists a parent pointer $\\p:\\{1,2,\\dots,\\ell-1\\}\\to\\{2,3,\\dots,\\ell\\}$\nsuch that the following holds for all $1\\le j<\\ell$:\n\\begin{gather*}\n\\p(j)>j,\\quad J_{\\p(j)}\\supseteq J_{j}\\cap(J_{j+1}\\cup J_{j+2}\\cup\\cdots\\cup J_{\\ell}),\\quad\\min\\{J_{\\p(j)}\\}>\\max\\{J_{j}\\backslash J_{\\p(j)}\\}.\n\\end{gather*}\n\\end{definition}\nThe symbolic Cholesky factor $F=\\chol(E)$ for a sparsity pattern\n$E$ is the canonical example of a zero-fill sparsity pattern. In\nturn, the corresponding column sets $J_{j}=\\col_{F}(j)$ are the canonical\nexample of sequence of subsets that satisfy the sorted version of\nthe running intersection property.\n","start":61691,"end":62493}]},{"id":"S030","semantic_type":"definition","kind":"definition","labels":["eq:Jk2_def"],"declared_external":false,"source_only":false,"title":"二次提升索引集合","source_units":[{"raw_tex":"Our proof is based on the fact that the ``lifted'' sparsity pattern\n$F^{(2)}$ can be constructed as $F^{(2)}=\\bigcup_{k=1}^{n}\\clique(J_{k}^{(2)})$\nwith respect to the following ``lifted'' index sets\n\\begin{gather}\nJ_{k}^{(2)}\\equiv\\idx_{F}(\\clique(J_{k}))=\\{\\idx_{F}(i,j):i,j\\in J_{k},\\quad i\\ge j\\}.\\label{eq:Jk2_def}\n\\end{gather}\nWe need to show that, if the original index sets $J_{1},J_{2},\\dots,J_{k}$\nsatisfy the running intersection property, then the lifted index sets\n$J_{1}^{(2)},J_{2}^{(2)},\\dots,J_{k}^{(2)}$ will inherit the running\nintersection property. Our key insight is that the index operator\n$\\idx_{F}$ implements a \\emph{raster ordering}.\n","start":63638,"end":64299}]},{"id":"S031","semantic_type":"lemma","kind":"lemma","labels":[],"declared_external":false,"source_only":false,"title":"列优先栅格次序","source_units":[{"raw_tex":"\\begin{lemma}[Raster ordering]\nThe ordering $\\idx_{F}:F\\to\\N$ satisfies the following, for all\n$(i,j)\\in F$ with $i\\ge j$ and $(i',j')\\in F$ with $i'\\ge j'$:\n\\begin{itemize}\n\\item If $j>j'$, then $\\idx(i,j)>\\idx(i',j')$ holds. \n\\item If $j=j'$ and $i>i'$, then $\\idx(i,j)>\\idx(i',j')$ holds. \n\\end{itemize}\n\\end{lemma}\n","start":64299,"end":64618}]},{"id":"S032","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"集合的保序限制","source_units":[{"raw_tex":"We will need some additional notation. For $V=\\{1,2,\\dots,n\\}$ and\n$J\\subseteq V$, we denote the subset of $J$ induced by the elements\nin $U=\\{u_{1},u_{2},\\dots,u_{p}\\}\\subseteq V$ as follows\n\\[\nJ[U]\\equiv\\{i:u_{i}\\in J\\}\\text{ where }u_{1}<u_{2}<\\cdots<u_{p}.\n\\]\nOur definition is made so that if $E=\\clique(J)$, then $E[U]=\\clique(J[U])$.\n","start":72588,"end":72929}]},{"id":"S033","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"附录正则性验证范围","source_units":[{"raw_tex":"Given data $C,A_{1},\\dots,A_{m}\\in\\S^{n}$ and $b\\in\\R^{m}$, define\n$(\\A,\\b,\\c,\\K)$ as in (\\ref{eq:AKdef}). In this section, we verify\nthat $(\\A,\\b,\\c,\\K)$ specifies an SDP that satisfies the regularity\nassumptions in \\defref{clp}. \n","start":99088,"end":99320}]},{"id":"S034","semantic_type":"algorithm","kind":"algorithm","labels":["eq:hsdcone","eq:hsdfeas","eq:hsdinit"],"declared_external":false,"source_only":false,"title":"自对偶嵌入及中心初始化","source_units":[{"raw_tex":"We prove \\propref{general} by using the short-step method of Nesterov\nand Todd~\\citep[Algorithm~6.1]{nesterov1998primal} to solve the\nextended homogeneous self-dual embedding\\begin{subequations}\\label{eq:hsd}\n\\begin{alignat}{2}\n & \\min_{x,y,s,\\kappa,\\tau,\\theta}\\quad & (n+1)\\theta\\\\\n & \\text{s.t.} & \\begin{bmatrix}0 & +\\A^{T} & -\\c & -r_{d}\\\\\n-\\A & 0 & +\\b & -r_{p}\\\\\n+\\c^{T} & -\\b^{T} & 0 & -r_{c}\\\\\nr_{d}^{T} & r_{p}^{T} & r_{c} & 0\n\\end{bmatrix}\\begin{bmatrix}x\\\\\ny\\\\\n\\tau\\\\\n\\theta\n\\end{bmatrix}+\\begin{bmatrix}s\\\\\n0\\\\\n\\kappa\\\\\n0\n\\end{bmatrix} & =\\begin{bmatrix}0\\\\\n0\\\\\n0\\\\\nn+1\n\\end{bmatrix}\\label{eq:hsdfeas}\\\\\n &  & x,s\\in\\K,\\quad\\kappa,\\tau & \\ge0,\\label{eq:hsdcone}\n\\end{alignat}\n\\end{subequations}where $r_{d}=\\one_{\\K}-\\c$ and $r_{p}=-\\A\\one_{\\K}+\\b$\nand $r_{c}=1+\\c^{T}\\one_{\\K}$. Beginning at the strictly feasible,\nperfectly centered point in (\\ref{eq:hsdinit}) for $\\mu=1$:\n\\begin{gather}\n\\theta^{(0)}=\\tau^{(0)}=\\kappa^{(0)}=1,\\quad y^{(0)}=0,\\quad x^{(0)}=s^{(0)}=\\one_{\\K}.\\label{eq:hsdinit}\n\\end{gather}\n","start":102960,"end":103983}]},{"id":"S035","semantic_type":"algorithm","kind":"algorithm","labels":[],"declared_external":false,"source_only":false,"title":"短步更新与缩放方向","source_units":[{"raw_tex":" we take the following steps\\begin{subequations}\\label{eq:hsd_ipm}\n\\begin{align}\n\\mu^{+} & =\\left(1-\\frac{1}{15\\sqrt{n+1}}\\right)\\cdot\\frac{x^{T}s+\\tau\\kappa}{n+1},\\\\\n(x^{+},y^{+},s^{+},\\tau^{+},\\theta^{+},\\kappa^{+}) & =(x,y,s,\\tau,\\theta,\\kappa)+(\\Delta x,\\Delta y,\\Delta s,\\Delta\\tau,\\Delta\\theta,\\Delta\\kappa).\\nonumber \n\\end{align}\n\\end{subequations}along the search direction defined by the linear\nsystem~\\citep[Eqn. 9]{todd1998nesterov} \\begin{subequations}\\label{eq:hsd_nt}\n\\begin{align}\n\\begin{bmatrix}0 & +\\A^{T} & -\\c & -r_{p}\\\\\n-\\A & 0 & +\\b & -r_{d}\\\\\n+\\c^{T} & -\\b^{T} & 0 & -r_{c}\\\\\n+r_{p}^{T} & +r_{d}^{T} & +r_{c} & 0\n\\end{bmatrix}\\begin{bmatrix}\\Delta x\\\\\n\\Delta y\\\\\n\\Delta\\tau\\\\\n\\Delta\\theta\n\\end{bmatrix}+\\begin{bmatrix}\\Delta s\\\\\n0\\\\\n\\Delta\\kappa\\\\\n0\n\\end{bmatrix} & =0,\\\\\ns+\\Delta s+\\nabla^{2}f(w)\\Delta x+\\mu^{+}\\nabla f(x) & =0,\\\\\n\\kappa+\\Delta\\kappa+(\\kappa/\\tau)\\Delta\\tau-\\mu^{+}\\tau^{-1} & =0.\n\\end{align}\n\\end{subequations}where $f(w)=-\\log\\det w$ is the usual self-concordant\nbarrier function, and $w\\in\\interior(\\K)$ is the unique point that\nsatisfies $\\nabla^{2}f(w)x=s$. The following is an immediate consequence\nof~\\citep[Theorem~6.4]{nesterov1998primal}.\n","start":103983,"end":105173}]},{"id":"S036","semantic_type":"lemma","kind":"lemma","labels":["eq:short-step","lem:short-step"],"declared_external":true,"declaration_text":["明确允许的 B0148；作者于 L1974–1975 归于 Nesterov and Todd, Theorem 6.4。"],"source_only":false,"title":"短步迭代收敛保证","source_units":[{"raw_tex":"\\begin{lemma}[Short-Step Method]\n\\label{lem:short-step}The sequence in (\\ref{eq:hsd_ipm}) arrives\nat an iterate $(x,y,s,\\tau,\\theta,\\kappa)$ satisfying\n\\begin{equation}\n\\frac{x^{T}s+\\tau\\kappa}{n+1}\\le\\epsilon,\\qquad\\tau\\kappa\\ge\\gamma\\epsilon\\label{eq:short-step}\n\\end{equation}\nwith $\\gamma=\\frac{9}{10}$ in at most $O(\\sqrt{n}\\log(1/\\epsilon))$\niterations. \n\\end{lemma}\n","start":105173,"end":105546}]},{"id":"S037","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"嵌入解恢复的引导","source_units":[{"raw_tex":"Finally, the following result (adapted from~\\citep[Lemma~5.7.2]{de2000self})\nassures us that a feasible point satisfying the optimality condition\n(\\ref{eq:short-step}) will recover a solution to the original problem.\n","start":105546,"end":105763}]},{"id":"S038","semantic_type":"theorem","kind":"theorem","labels":["thm:total"],"declared_external":false,"source_only":false,"title":"弦转换复杂度上界","source_units":[{"raw_tex":"\\begin{theorem}[Upper complexity]\n\\label{thm:total}Let the data $C,A_{1},\\dots,A_{m}\\in\\S^{n}$ and\n$b\\in\\R^{m}$ satisfy \\asmref{strict}. Given a tree decomposition\nof width $\\o{\\tau}$ for the extended aggregate sparsity graph $\\o G=(V,\\o E)$,\nwhere\n\\[\nV=\\{1,2,\\dots,n\\},\\qquad\\o E=\\spar(C)\\cup\\clique(A_{1})\\cup\\cdots\\cup\\clique(A_{m}),\n\\]\nset $\\Pi$ as the associated perfect elimination ordering. Then, \\algref{myalg}\noutputs $U\\in\\R^{n\\times\\o{\\omega}}$ and $v\\in\\R^{m}$ with $v\\le0$\nsuch that\n\\begin{align*}\n\\inner{A_{i}}{UU^{T}}-b_{i} & \\le\\epsilon\\text{ for all }i, & \\sum_{i=1}^{m}v_{i}A_{i}-C & \\preceq\\epsilon\\cdot I, & \\inner C{UU^{T}}-\\inner bv & \\le\\epsilon\\cdot N,\n\\end{align*}\nin $O(\\sqrt{N}\\log(1/\\epsilon))$ iterations, with per-iteration costs\nof $O(\\o{\\omega}^{4}\\cdot N)$ time and $O(\\o{\\omega}^{2}\\cdot N)$\nmemory, where $N=m+\\o{\\omega}\\cdot n$ and $\\o{\\omega}=1+\\o{\\tau}$.\n\\end{theorem}","start":51314,"end":52220}]},{"id":"S039","semantic_type":"proposition","kind":"proposition","labels":["prop:chol"],"declared_external":false,"source_only":false,"title":"稀疏 Cholesky 成本界","source_units":[{"raw_tex":"\\begin{proposition}[Sparse Cholesky factorization]\n\\label{prop:chol}Given $S\\in\\S^{n},$ $S\\succ0$, let $\\omega\\equiv\\omega(S)$.\nSparse Cholesky factorization factors $L=\\chol(S)$ in $\\Tim$ arithmetic\noperations and $M$ units of memory, where \n\\begin{align*}\n\\frac{1}{6}(\\omega-1)^{3}+n & \\le\\Tim\\le\\omega^{2}\\cdot n, & \\frac{1}{2}(\\omega-1)^{2}+n & \\le\\Mem\\le\\omega\\cdot n.\n\\end{align*}\n\\end{proposition}","start":34018,"end":34422}]},{"id":"S040","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"Cholesky 成本界原证","source_units":[{"raw_tex":"\\begin{proof}\nLet $\\omega_{j}\\equiv|\\col_{F}(j)|$. By inspection, $\\Tim=\\sum_{j=1}^{n}\\frac{1}{2}\\omega_{j}(\\omega_{j}+1)$\nand $\\Mem=\\sum_{j=1}^{n}\\omega_{j}$. The bounds follow by substituting\n$\\omega\\ge\\omega_{j+1}\\ge\\omega_{j}-1$. Indeed,\n\\[\n\\Mem=\\sum_{j=1}^{n}\\omega_{j}\\ge\\underbrace{\\omega+(\\omega-1)+\\cdots+1}_{\\omega\\text{ terms}}+\\underbrace{1+\\cdots+1}_{n-\\omega\\text{ terms}}=\\frac{1}{2}\\omega(\\omega+1)+(n-\\omega)\n\\]\nand similarly $\\Tim=\\sum_{j=1}^{n}\\frac{1}{2}\\omega_{j}(\\omega_{j}+1)\\ge\\frac{1}{6}\\omega(\\omega+1)(\\omega+2)+(n-\\omega).$\n\\qed\n\\end{proof}","start":34423,"end":34991}]},{"id":"S041","semantic_type":"theorem","kind":"theorem","labels":["thm:front"],"declared_external":false,"source_only":false,"title":"Schur 稀疏前沿界","source_units":[{"raw_tex":"\\begin{theorem}[Frontsize of Schur complement sparsity]\n\\label{thm:front}Given $C,A_{1},A_{2},\\dots,A_{m}\\in\\S^{n}$, define\n$\\A,\\K$ as in (\\ref{eq:AKdef}), and define $E^{(2)}$ as in (\\ref{eq:E2def}).\nWe have \n\\[\n\\frac{1}{2}\\omega(\\omega+1)\\le\\omega(E^{(2)})\\le\\frac{1}{2}\\o{\\omega}(\\o{\\omega}+1)\n\\]\nwhere $\\omega\\equiv\\omega(E)$ and $\\o{\\omega}=\\omega(\\o E)$ and\n$E,\\o E$ are defined in (\\ref{eq:Edef}) and (\\ref{eq:Ebardef}).\nMoreover, if $E=\\o E$, then we also have $\\chol(E^{(2)})=E^{(2)}$. \n\\end{theorem}","start":46293,"end":46802}]},{"id":"S042","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"Schur 前沿定理原证","source_units":[{"raw_tex":"Observe that $E^{(2)}\\supseteq F^{(2)}=\\bigcup_{k=1}^{n}\\clique(\\P_{k})$\nvia (\\ref{eq:E2def}), so it follows immediately from \\propref{del}\n\\[\n\\omega(E^{(2)})\\ge\\omega(F^{(2)})=\\frac{1}{2}\\omega(E)[\\omega(E)+1],\n\\]\nwhich is precisely the lower-bound in \\thmref{front}. For the upper-bound,\nwe will use $\\o F=\\chol(\\o E)$, the symbolic Cholesky factor of the\nextended aggregate sparsity pattern $\\o E$, to construct a similarly\nlifted $\\o F^{(2)}$. Our key insight is that $E^{(2)}$ can be obtained\nfrom $\\o F^{(2)}$ via vertex and edge deletions.\n","start":59531,"end":60078},{"raw_tex":"Substituting $E^{(2)}\\subseteq\\o F^{(2)}[V^{(2)}]$ with the exact\nfrontsize of $\\omega(\\o F^{(2)})$ from \\lemref{quadlift} yields\n\\[\n\\omega(E^{(2)})\\le\\omega(\\o F^{(2)})=\\frac{1}{2}\\omega(\\o E)[\\omega(\\o E)+1],\n\\]\nwhich is precisely the upper-bound in \\thmref{front}. Finally, if\n$E=\\o E$, then $F^{(2)}=\\o F^{(2)}=E^{(2)}$, so $\\chol(E^{(2)})=E^{(2)}$\nvia \\lemref{quadlift}.\n","start":60615,"end":60991}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"提升子模式给前沿下界","source_units":[{"raw_tex":"Observe that $E^{(2)}\\supseteq F^{(2)}=\\bigcup_{k=1}^{n}\\clique(\\P_{k})$\nvia (\\ref{eq:E2def}), so it follows immediately from \\propref{del}\n\\[\n\\omega(E^{(2)})\\ge\\omega(F^{(2)})=\\frac{1}{2}\\omega(E)[\\omega(E)+1],\n\\]\nwhich is precisely the lower-bound in \\thmref{front}. For the upper-bound,\nwe will use $\\o F=\\chol(\\o E)$, the symbolic Cholesky factor of the\nextended aggregate sparsity pattern $\\o E$, to construct a similarly\nlifted $\\o F^{(2)}$. Our key insight is that $E^{(2)}$ can be obtained\nfrom $\\o F^{(2)}$ via vertex and edge deletions.\n","start":59531,"end":60078}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"扩展提升给上界及零填充","source_units":[{"raw_tex":"Substituting $E^{(2)}\\subseteq\\o F^{(2)}[V^{(2)}]$ with the exact\nfrontsize of $\\omega(\\o F^{(2)})$ from \\lemref{quadlift} yields\n\\[\n\\omega(E^{(2)})\\le\\omega(\\o F^{(2)})=\\frac{1}{2}\\omega(\\o E)[\\omega(\\o E)+1],\n\\]\nwhich is precisely the upper-bound in \\thmref{front}. Finally, if\n$E=\\o E$, then $F^{(2)}=\\o F^{(2)}=E^{(2)}$, so $\\chol(E^{(2)})=E^{(2)}$\nvia \\lemref{quadlift}.\n","start":60615,"end":60991}]},{"id":"S045","semantic_type":"corollary","kind":"corollary","labels":["cor:normal"],"declared_external":false,"source_only":false,"title":"Schur 方程求解成本","source_units":[{"raw_tex":"\\begin{corollary}[Cost of Schur complement equation]\n\\label{cor:normal}Given the data matrix $\\A$, scaling point $w\\in\\interior(\\K)$,\nand right-hand side $g$, define $\\Hess=\\A\\nabla^{2}f(w)\\A^{T}$ as\nin (\\ref{eq:E2def}). Suppose that all columns of $\\A$ are nonzero,\nand all scaling matrices $\\D_{j}$ are fully dense. Then, it takes\n$\\Tim$ arithmetic operations and $\\Mem$ units of storage to form\n$\\Hess$ and solve $\\Hess\\Delta y=g$, where\n\\begin{gather*}\n\\frac{1}{48}(\\omega-1)^{6}+m+n\\le\\Tim\\le4\\o{\\omega}^{4}\\cdot(m+\\omega n),\\\\\n\\frac{1}{8}(\\omega-1)^{4}+m+n\\le\\Mem\\le2\\o{\\omega}^{2}\\cdot(m+\\omega n),\n\\end{gather*}\nin which $\\omega\\equiv\\omega(E)$ and $\\o{\\omega}\\equiv\\omega(\\o E)$\nsatisfy $1\\le\\omega\\le\\o{\\omega}$.\n\\end{corollary}","start":47617,"end":48355}]},{"id":"S046","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"Schur 成本推论原证","source_units":[{"raw_tex":"\\begin{proof}\nLet $a_{i}\\equiv\\svec_{F}(A_{i})$ and $\\omega_{j}\\equiv|\\col_{F}(j)|$.\nWe break the solution of $\\Hess\\Delta y=g$ into five steps:\n\\begin{enumerate}\n\\item (Input) It takes $\\Mem_{\\mathrm{data}}$ memory to state the problem\ndata, where $m+n\\le\\Mem_{\\mathrm{data}}\\le2\\o{\\omega}^{2}m+3\\omega^{2}n$.\nIndeed, $m+n\\le\\nnz(\\A)\\le\\o{\\omega}^{2}m+\\omega^{2}n$ because $1\\le\\nnz(a_{i})\\le\\o{\\omega}^{2}$\n(see Step 2 below) and $\\nnz(\\P_{j})=\\frac{1}{2}\\omega_{j}(\\omega_{j}+1)$.\nAlso, $m+n\\le\\nnz(w)\\le m+\\omega^{2}n$, and $n\\le\\nnz(g)\\le\\omega n$.\n\\item (Build LP part) It takes $\\Tim_{\\mathrm{LP}}$ time to build $\\sum_{i=1}^{m}d_{i}a_{i}a_{i}^{T}$,\nwhere $m\\le\\Tim_{\\mathrm{LP}}\\le2\\o{\\omega}^{4}m$ time and memory.\nThis follows from $1\\le\\nnz(a_{i})\\le\\o{\\omega}^{2}$, where the upper-bound\nis because $\\spar(a_{i}a_{i}^{T})\\subseteq E^{(2)}\\subseteq\\chol(E^{(2)})$,\nand that $\\omega(E^{(2)})$ is, by definition, the maximum number\nof nonzero elements in a single column of $\\chol(E^{(2)})$.\n\\item (Build SDP part) It takes $\\Tim_{\\mathrm{SDP}}$ time to build $\\sum_{j=1}^{n}\\P_{j}\\D_{j}\\P_{j}^{T}$,\nwhere $n\\le\\Tim_{\\mathrm{SDP}}\\le\\omega^{4}n$ time and memory. This\nfollows from $\\nnz(\\P_{j})=\\frac{1}{2}\\omega_{j}(\\omega_{j}+1)$,\nwhich implies $\\nnz(\\P_{j}\\D_{j}\\P_{j}^{T})=\\frac{1}{4}\\omega_{j}^{2}(\\omega_{j}+1)^{2}$\nfor a fully-dense $\\D_{j}\\succ0$. \n\\item (Factorization) It takes $\\Tim_{\\mathrm{fact}}$ time and $\\Mem_{\\mathrm{fact}}$\nmemory to factor $\\L=\\chol(\\Hess)$, where $\\frac{1}{48}(\\omega-1)^{6}+n\\le\\Tim_{\\mathrm{fact}}\\le\\o{\\omega}^{5}n$\nand $\\frac{1}{8}(\\omega-1)^{4}+n\\le\\Mem_{\\mathrm{fact}}\\le\\o{\\omega}^{3}n$.\nThe matrix $\\Hess$ has $|F|$ columns and rows, and frontsize $\\omega(E^{(2)})$.\nThe desired figures follow by substituting $\\frac{1}{2}\\omega^{2}\\le\\omega(E^{(2)})\\le\\o{\\omega}^{2}$\nand $n\\le|F|\\le\\omega n$ into \\propref{chol}.\n\\item (Back-substitution) It takes $\\Mem_{\\mathrm{fact}}$ time and memory\nto solve each of $\\L z=g$ and $\\L^{T}\\Delta y=z$ via triangular\nback-substitution. \n\\end{enumerate}\nThe overall runtime is cumulative, so $\\Tim=\\Tim_{\\mathrm{LP}}+\\Tim_{\\mathrm{SDP}}+\\Tim_{\\mathrm{fact}}+2\\Mem_{\\mathrm{fact}}$.\nThe overall memory use is $\\Mem=\\Mem_{\\mathrm{data}}+\\Mem_{\\mathrm{fact}}$,\nbecause the matrix $\\Hess$ can be constructed and then factored in-place.\n\\qed\n\\end{proof}","start":48356,"end":50694}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"输入与约束秩一项成本","source_units":[{"raw_tex":"\\begin{proof}\nLet $a_{i}\\equiv\\svec_{F}(A_{i})$ and $\\omega_{j}\\equiv|\\col_{F}(j)|$.\nWe break the solution of $\\Hess\\Delta y=g$ into five steps:\n\\begin{enumerate}\n\\item (Input) It takes $\\Mem_{\\mathrm{data}}$ memory to state the problem\ndata, where $m+n\\le\\Mem_{\\mathrm{data}}\\le2\\o{\\omega}^{2}m+3\\omega^{2}n$.\nIndeed, $m+n\\le\\nnz(\\A)\\le\\o{\\omega}^{2}m+\\omega^{2}n$ because $1\\le\\nnz(a_{i})\\le\\o{\\omega}^{2}$\n(see Step 2 below) and $\\nnz(\\P_{j})=\\frac{1}{2}\\omega_{j}(\\omega_{j}+1)$.\nAlso, $m+n\\le\\nnz(w)\\le m+\\omega^{2}n$, and $n\\le\\nnz(g)\\le\\omega n$.\n\\item (Build LP part) It takes $\\Tim_{\\mathrm{LP}}$ time to build $\\sum_{i=1}^{m}d_{i}a_{i}a_{i}^{T}$,\nwhere $m\\le\\Tim_{\\mathrm{LP}}\\le2\\o{\\omega}^{4}m$ time and memory.\nThis follows from $1\\le\\nnz(a_{i})\\le\\o{\\omega}^{2}$, where the upper-bound\nis because $\\spar(a_{i}a_{i}^{T})\\subseteq E^{(2)}\\subseteq\\chol(E^{(2)})$,\nand that $\\omega(E^{(2)})$ is, by definition, the maximum number\nof nonzero elements in a single column of $\\chol(E^{(2)})$.\n","start":48356,"end":49357}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"局部半正定项形成成本","source_units":[{"raw_tex":"\\item (Build SDP part) It takes $\\Tim_{\\mathrm{SDP}}$ time to build $\\sum_{j=1}^{n}\\P_{j}\\D_{j}\\P_{j}^{T}$,\nwhere $n\\le\\Tim_{\\mathrm{SDP}}\\le\\omega^{4}n$ time and memory. This\nfollows from $\\nnz(\\P_{j})=\\frac{1}{2}\\omega_{j}(\\omega_{j}+1)$,\nwhich implies $\\nnz(\\P_{j}\\D_{j}\\P_{j}^{T})=\\frac{1}{4}\\omega_{j}^{2}(\\omega_{j}+1)^{2}$\nfor a fully-dense $\\D_{j}\\succ0$. \n","start":49357,"end":49722}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"前沿界代入分解成本","source_units":[{"raw_tex":"\\item (Factorization) It takes $\\Tim_{\\mathrm{fact}}$ time and $\\Mem_{\\mathrm{fact}}$\nmemory to factor $\\L=\\chol(\\Hess)$, where $\\frac{1}{48}(\\omega-1)^{6}+n\\le\\Tim_{\\mathrm{fact}}\\le\\o{\\omega}^{5}n$\nand $\\frac{1}{8}(\\omega-1)^{4}+n\\le\\Mem_{\\mathrm{fact}}\\le\\o{\\omega}^{3}n$.\nThe matrix $\\Hess$ has $|F|$ columns and rows, and frontsize $\\omega(E^{(2)})$.\nThe desired figures follow by substituting $\\frac{1}{2}\\omega^{2}\\le\\omega(E^{(2)})\\le\\o{\\omega}^{2}$\nand $n\\le|F|\\le\\omega n$ into \\propref{chol}.\n","start":49722,"end":50226}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"两个三角系统回代","source_units":[{"raw_tex":"\\item (Back-substitution) It takes $\\Mem_{\\mathrm{fact}}$ time and memory\nto solve each of $\\L z=g$ and $\\L^{T}\\Delta y=z$ via triangular\nback-substitution. \n","start":50226,"end":50384}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"累计时间与原位存储","source_units":[{"raw_tex":"\\end{enumerate}\nThe overall runtime is cumulative, so $\\Tim=\\Tim_{\\mathrm{LP}}+\\Tim_{\\mathrm{SDP}}+\\Tim_{\\mathrm{fact}}+2\\Mem_{\\mathrm{fact}}$.\nThe overall memory use is $\\Mem=\\Mem_{\\mathrm{data}}+\\Mem_{\\mathrm{fact}}$,\nbecause the matrix $\\Hess$ can be constructed and then factored in-place.\n\\qed\n\\end{proof}","start":50384,"end":50694}]},{"id":"S052","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"复杂度主定理完整原证","source_units":[{"raw_tex":"\\begin{proof}One can verify that $(\\A,\\b,\\c,\\K)$ defined in (\\ref{eq:AKdef})\nspecifies an SDP in $(N,\\omega)$-standard form (see \\appref{regularity}\nfor the regularity conditions). Moreover, it follows from the monotonicity\nof the frontsize (\\propref{del}) that $\\omega\\equiv\\omega(E_{\\Pi})\\le\\omega(\\o E_{\\Pi})\\equiv\\o{\\omega}$.\nWe will track the cost of \\algref{myalg} step-by-step:\n\\begin{enumerate}\n\\item (Front-reducing permutation) Preordering $A_{i}\\gets\\Pi A_{i}\\Pi^{T}$\nand $C\\gets\\Pi C\\Pi^{T}$ the matrices cost $O(\\nnz(C)+\\sum_{i=1}^{m}\\nnz(A_{i}))=O(\\o{\\omega}^{2}N)$\ntime and memory. This follows from $\\nnz(C)\\le|F|\\le N$ and $\\nnz(A_{i})\\le\\o{\\omega}^{2}$. \n\\item (Conversion) Computing $F=\\chol(E)$ costs $O(|F|)=O(N)$ time and\nspace, where we note that $|F|\\le\\omega n$. \n\\item (Solution) Let $K=2\\cdot\\max\\{1,|\\tr(C)|,|\\tr(A_{1})|,\\dots,|\\tr(A_{m})|\\}$.\nAfter $O(\\sqrt{N}\\log(K/\\epsilon))$ iterations, we arrive at a primal\n$v\\le0$ and $V_{j}\\succeq0$ and dual point $Y\\in\\S_{F}^{n}$ satisfying\n\\begin{gather*}\n\\inner{A_{i}}Y-b_{i}\\le\\epsilon/K,\\quad Y[J_{j},J_{j}]\\succeq-(\\epsilon/K)I,\\\\\n{\\textstyle \\left\\Vert \\sum_{i}v_{i}A_{i}+\\sum_{j}P_{j}V_{j}P_{j}^{T}-C\\right\\Vert \\le\\epsilon/K,}\\\\\n\\inner CY-\\inner bv\\le N\\cdot(\\epsilon/K).\n\\end{gather*}\nEach iteration costs $O(\\o{\\omega}^{4}N)$ time and $O(\\o{\\omega}^{2}N)$\nmemory. This cost is fully determined by the cost of solving $O(1)$\ninstances of the Schur complement equation, which dominates the overhead\nof $O(\\omega^{3}n+\\nnz(\\A))=O(\\o{\\omega}^{2}N)$ time and memory.\n\\item (Recovery) Using the previously recovered $Y$, we recover $U$ such\nthat $\\Pi_{F}(UU^{T})=Y+\\delta I$ where $\\delta=-\\min_{j}\\{0,\\lambda_{\\min}(Y[J_{j},J_{j}])\\}\\le\\epsilon/K$.\nThis takes $O(\\omega^{3}n)=O(\\o{\\omega}^{2}N)$ time and $O(\\omega^{2}n)=O(\\o{\\omega}N)$\nmemory.\n\\item (Output) We output $U$ and $v$, and check for accuracy. It follows\nfrom $K\\ge2|\\tr(C)|$ that\n\\[\n\\inner C{UU^{T}}-\\inner CY=\\delta\\cdot\\tr(C)\\le\\epsilon\\cdot\\frac{\\tr(C)}{K}\\le\\epsilon\\cdot\\frac{|\\tr(C)|}{2|\\tr(C)|}\\le\\frac{1}{2}\\epsilon,\n\\]\nand from $K\\ge2$ and $N\\ge1$ that \n\\[\n\\inner C{UU^{T}}-\\inner bv\\le\\frac{N}{K}\\epsilon+\\frac{1}{2}\\epsilon=N\\epsilon\\cdot\\left(\\frac{1}{2}+\\frac{1}{2N}\\right)\\le N\\cdot\\epsilon.\n\\]\nSimilarly, it follows from $K\\ge2|\\tr(A_{i})|$ and $K\\ge2$ that\n$\\inner{A_{i}}{UU^{T}}-b_{i}\\le\\epsilon$. Finally, it follows from\n$v\\ge0$ and $V_{j}\\succeq0$ that\n\\[\n\\sum_{i}v_{i}A_{i}+\\sum_{j}P_{j}V_{j}P_{j}^{T}-C\\preceq\\epsilon\\cdot I\\implies\\sum_{i}v_{i}A_{i}-C\\preceq\\epsilon\\cdot I.\n\\]\n\\end{enumerate}\n\\qed \\end{proof}","start":52221,"end":54795}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"标准形式与前沿控制","source_units":[{"raw_tex":"\\begin{proof}One can verify that $(\\A,\\b,\\c,\\K)$ defined in (\\ref{eq:AKdef})\nspecifies an SDP in $(N,\\omega)$-standard form (see \\appref{regularity}\nfor the regularity conditions). Moreover, it follows from the monotonicity\nof the frontsize (\\propref{del}) that $\\omega\\equiv\\omega(E_{\\Pi})\\le\\omega(\\o E_{\\Pi})\\equiv\\o{\\omega}$.\nWe will track the cost of \\algref{myalg} step-by-step:\n","start":52221,"end":52606}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"重排与符号转换开销","source_units":[{"raw_tex":"\\begin{enumerate}\n\\item (Front-reducing permutation) Preordering $A_{i}\\gets\\Pi A_{i}\\Pi^{T}$\nand $C\\gets\\Pi C\\Pi^{T}$ the matrices cost $O(\\nnz(C)+\\sum_{i=1}^{m}\\nnz(A_{i}))=O(\\o{\\omega}^{2}N)$\ntime and memory. This follows from $\\nnz(C)\\le|F|\\le N$ and $\\nnz(A_{i})\\le\\o{\\omega}^{2}$. \n\\item (Conversion) Computing $F=\\chol(E)$ costs $O(|F|)=O(N)$ time and\nspace, where we note that $|F|\\le\\omega n$. \n","start":52606,"end":53010}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"按迹规模重标定精度","source_units":[{"raw_tex":"\\item (Solution) Let $K=2\\cdot\\max\\{1,|\\tr(C)|,|\\tr(A_{1})|,\\dots,|\\tr(A_{m})|\\}$.\nAfter $O(\\sqrt{N}\\log(K/\\epsilon))$ iterations, we arrive at a primal\n$v\\le0$ and $V_{j}\\succeq0$ and dual point $Y\\in\\S_{F}^{n}$ satisfying\n\\begin{gather*}\n\\inner{A_{i}}Y-b_{i}\\le\\epsilon/K,\\quad Y[J_{j},J_{j}]\\succeq-(\\epsilon/K)I,\\\\\n{\\textstyle \\left\\Vert \\sum_{i}v_{i}A_{i}+\\sum_{j}P_{j}V_{j}P_{j}^{T}-C\\right\\Vert \\le\\epsilon/K,}\\\\\n\\inner CY-\\inner bv\\le N\\cdot(\\epsilon/K).\n\\end{gather*}\n","start":53010,"end":53487}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"每步成本由 Schur 求解支配","source_units":[{"raw_tex":"Each iteration costs $O(\\o{\\omega}^{4}N)$ time and $O(\\o{\\omega}^{2}N)$\nmemory. This cost is fully determined by the cost of solving $O(1)$\ninstances of the Schur complement equation, which dominates the overhead\nof $O(\\omega^{3}n+\\nnz(\\A))=O(\\o{\\omega}^{2}N)$ time and memory.\n","start":53487,"end":53765}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"对角修正及低秩恢复","source_units":[{"raw_tex":"\\item (Recovery) Using the previously recovered $Y$, we recover $U$ such\nthat $\\Pi_{F}(UU^{T})=Y+\\delta I$ where $\\delta=-\\min_{j}\\{0,\\lambda_{\\min}(Y[J_{j},J_{j}])\\}\\le\\epsilon/K$.\nThis takes $O(\\omega^{3}n)=O(\\o{\\omega}^{2}N)$ time and $O(\\omega^{2}n)=O(\\o{\\omega}N)$\nmemory.\n","start":53765,"end":54043}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"恢复后的目标与约束误差","source_units":[{"raw_tex":"\\item (Output) We output $U$ and $v$, and check for accuracy. It follows\nfrom $K\\ge2|\\tr(C)|$ that\n\\[\n\\inner C{UU^{T}}-\\inner CY=\\delta\\cdot\\tr(C)\\le\\epsilon\\cdot\\frac{\\tr(C)}{K}\\le\\epsilon\\cdot\\frac{|\\tr(C)|}{2|\\tr(C)|}\\le\\frac{1}{2}\\epsilon,\n\\]\nand from $K\\ge2$ and $N\\ge1$ that \n\\[\n\\inner C{UU^{T}}-\\inner bv\\le\\frac{N}{K}\\epsilon+\\frac{1}{2}\\epsilon=N\\epsilon\\cdot\\left(\\frac{1}{2}+\\frac{1}{2N}\\right)\\le N\\cdot\\epsilon.\n\\]\nSimilarly, it follows from $K\\ge2|\\tr(A_{i})|$ and $K\\ge2$ that\n$\\inner{A_{i}}{UU^{T}}-b_{i}\\le\\epsilon$. Finally, it follows from\n","start":54043,"end":54602}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"移除半正定项的对偶误差","source_units":[{"raw_tex":"$v\\ge0$ and $V_{j}\\succeq0$ that\n\\[\n\\sum_{i}v_{i}A_{i}+\\sum_{j}P_{j}V_{j}P_{j}^{T}-C\\preceq\\epsilon\\cdot I\\implies\\sum_{i}v_{i}A_{i}-C\\preceq\\epsilon\\cdot I.\n\\]\n\\end{enumerate}\n","start":54602,"end":54779}]},{"id":"S060","semantic_type":"proposition","kind":"proposition","labels":["prop:del"],"declared_external":false,"source_only":false,"title":"保序子图的前沿单调性","source_units":[{"raw_tex":"\\begin{proposition}[Subgraph monotonicity]\n\\label{prop:del}Let $E$ be a sparsity pattern of order $n$, and\nlet $U\\subseteq\\{1,2,\\dots,n\\}$. Then, for any sparsity pattern $D$\nof order $|U|$ that satisfies $E[U]\\supseteq D$, we have $\\chol(E)[U]\\supseteq\\chol(D)$,\nand therefore $\\omega(E)\\ge\\omega(D)$.\n\\end{proposition}","start":57847,"end":58167}]},{"id":"S061","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"子图单调性完整原证","source_units":[{"raw_tex":"\\begin{proof}\nIt is known that $(i,j)\\in\\chol(E)$ for $i>j$ holds if and only\nif there exists a path $(i,p_{1},p_{2},\\dots,p_{\\ell},j)$ whose edges\nare in $E$, and whose internal nodes are ordered $p_{1},p_{2},\\dots,p_{\\ell}<j<i$;\nsee e.g.~\\citep[Theorem~6.1]{vandenberghe2015chordal}. It immediately\nfollows this characterization that $\\chol(\\cdot)$ is monotone with\nrespect to the deletion of edges and isolated vertices: (1) if $D\\subseteq E$,\nthen $\\chol(D)\\subseteq\\chol(E)$; (2) we have $\\chol(E[U])=\\chol(E)[U]$\nfor $U=\\{1,2,\\dots,n\\}\\backslash v$ with isolated vertex $v$. Therefore,\n$\\chol(\\cdot)$ must also be monotone under general vertex and edge\ndeletions, because we can always delete edges to isolate a vertex\nbefore deleting it. \\qed\n\\end{proof}","start":58168,"end":58929}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"调用填充路径刻画","source_units":[{"raw_tex":"\\begin{proof}\nIt is known that $(i,j)\\in\\chol(E)$ for $i>j$ holds if and only\nif there exists a path $(i,p_{1},p_{2},\\dots,p_{\\ell},j)$ whose edges\nare in $E$, and whose internal nodes are ordered $p_{1},p_{2},\\dots,p_{\\ell}<j<i$;\nsee e.g.~\\citep[Theorem~6.1]{vandenberghe2015chordal}. It immediately\n","start":58168,"end":58469}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"由删边删点得到包含","source_units":[{"raw_tex":"follows this characterization that $\\chol(\\cdot)$ is monotone with\nrespect to the deletion of edges and isolated vertices: (1) if $D\\subseteq E$,\nthen $\\chol(D)\\subseteq\\chol(E)$; (2) we have $\\chol(E[U])=\\chol(E)[U]$\nfor $U=\\{1,2,\\dots,n\\}\\backslash v$ with isolated vertex $v$. Therefore,\n$\\chol(\\cdot)$ must also be monotone under general vertex and edge\ndeletions, because we can always delete edges to isolate a vertex\nbefore deleting it. \\qed\n\\end{proof}","start":58469,"end":58929}]},{"id":"S064","semantic_type":"lemma","kind":"lemma","labels":["lem:quadlift"],"declared_external":false,"source_only":false,"title":"二次提升的精确前沿","source_units":[{"raw_tex":"\\begin{lemma}[Quadratic lift]\n\\label{lem:quadlift}Let $E$ be an arbitrary sparsity pattern of\norder $n$. Define $F=\\chol(E)$ and the lifted sparsity pattern\n$F^{(2)}=\\bigcup_{k=1}^{n}\\clique(\\P_{k})$ in which each $\\P_{k}$\nis implicitly defined to satisfy $\\P_{k}^{T}\\svec_{F}(Y)=\\svec(Y[\\col_{F}(k),\\col_{F}(k)])$\nfor all $Y\\in\\S_{F}^{n}$. Then, we have $\\chol(F^{(2)})=F^{(2)}$\nand $\\omega(F^{(2)})=\\frac{1}{2}\\omega(E)[\\omega(E)+1]$.\n\\end{lemma}","start":59082,"end":59530}]},{"id":"S065","semantic_type":"lemma","kind":"lemma","labels":["lem:ub"],"declared_external":false,"source_only":false,"title":"扩展提升模式的稀疏高估","source_units":[{"raw_tex":"\\begin{lemma}[Sparsity overestimate]\n\\label{lem:ub}Let $\\o E$ and $E^{(2)}$ be the sparsity patterns\ndefined in (\\ref{eq:Ebardef}) and (\\ref{eq:E2def}). Define $\\o F=\\chol(\\o E)$\nand the lifted sparsity pattern $\\o F^{(2)}=\\bigcup_{k=1}^{n}\\clique(\\o{\\P}_{k})$\nin which each $\\o{\\P}_{k}$ is implicitly defined to satisfy $\\o{\\P}_{k}^{T}\\svec_{\\o F}(Y)=\\svec(Y[\\col_{\\o F}(k),\\col_{\\o F}(k)])$\nfor all $Y\\in\\S_{\\o F}^{n}$. Then, $E^{(2)}\\subseteq\\o F^{(2)}[V^{(2)}]$\nholds for $V^{(2)}=\\{\\idx_{\\overline{F}}(i,j):i,j\\in F\\}$.\n\\end{lemma}","start":60078,"end":60614}]},{"id":"S066","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"稀疏高估引理原证","source_units":[{"raw_tex":"Our desired claim, namely that \n\\begin{align*}\nE^{(2)} & =\\left(\\bigcup_{i=1}^{m}\\clique(\\supp(a_{i}))\\right)\\cup\\left(\\bigcup_{j=1}^{n}\\clique(\\supp(\\P_{j}))\\right)\\\\\n & \\subseteq\\left(\\bigcup_{k=1}^{n}\\clique(\\supp(\\o{\\P}_{k})[V^{(2)}])\\right)=\\o F^{(2)}[V^{(2)}]\n\\end{align*}\nwhere $V^{(2)}=\\{\\idx_{\\overline{F}}(i,j):i,j\\in F\\}$, now follows\nimmediately from the following two lemmas.\n","start":72929,"end":73318}]},{"id":"S067","semantic_type":"proposition","kind":"proposition","labels":["prop:zf->rip"],"declared_external":false,"source_only":false,"title":"零填充推出有序运行交","source_units":[{"raw_tex":"\\begin{proposition}[ZF$\\implies$RIP]\n\\label{prop:zf->rip}Let $F$ be a zero-fill sparsity pattern of\norder $n$. Then, the sequence of subsets $J_{1},J_{2},\\dots,J_{n}$\nwith $J_{j}=\\col_{F}(j)\\equiv\\{j\\}\\cup\\{i>j:(i,j)\\in F\\}$ satisfies\nthe sorted running intersection property. \n\\end{proposition}","start":62493,"end":62788}]},{"id":"S068","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"零填充到运行交完整原证","source_units":[{"raw_tex":"\\begin{proof}\nDefine $\\p(j)=\\min\\{i>j:i\\in J_{j}\\}$ if $|J_{j}|>1$ and $\\p(j)=n$\nif $|J_{j}|=1$. Clearly, $\\p(j)>j$ holds for all $1\\le j<n$. To\nprove $J_{\\p(j)}\\supseteq J_{j}\\cap\\bigcup_{w=j+1}^{\\ell}J_{w}$,\nlet $i\\in J_{j}\\cap J_{w}$ for some $w>j$. We prove that $i\\in J_{\\p(j)}$\nvia the following steps: \n\\begin{itemize}\n\\item We have $i>j$, because $i\\in J_{w}$ implies that $i\\ge w>j$. \n\\item If $i=\\p(j)$, then $i\\in J_{\\p(j)}$ by definition. \n\\item If $i>\\p(j)$, then $(i,j)\\in F$ and $(\\p(j),j)\\in F$ for $i>\\p(j)>j$\nimplies $(i,\\p(j))\\in F$, and hence $i\\in J_{\\p(j)}$. \n\\end{itemize}\nFinally, we prove $\\min\\{J_{\\p(j)}\\}>\\max\\{J_{j}\\backslash J_{\\p(j)}\\}$\nby noting that $\\max\\{J_{j}\\backslash J_{\\p(j)}\\}=j$ with our construction,\nand that $i=\\min J_{\\p(j)}$ must satisfy $i\\in J_{\\p(j)}$ and therefore\n$i\\ge\\p(j)>j$. \\qed\n\\end{proof}","start":62789,"end":63636}]},{"id":"S069","semantic_type":"lemma","kind":"lemma","labels":["lem:lift"],"declared_external":false,"source_only":false,"title":"提升保持有序运行交","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:lift}Let the sequence of subsets $J_{1},J_{2},\\dots,J_{\\ell}$\nwith $J_{j}\\subset\\N$ satisfy the sorted running intersection property.\nThen, $J_{1}^{(2)},J_{2}^{(2)},\\dots,J_{\\ell}^{(2)}$ also satisfy\nthe sorted running intersection property. \n\\end{lemma}","start":64620,"end":64899}]},{"id":"S070","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"提升运行交完整原证","source_units":[{"raw_tex":"\\begin{proof}\nLet $\\p(\\cdot)$ denote the parent pointer that verifies the sorted\nrunning intersection property in $J_{1},J_{2},\\dots,J_{\\ell}$. We\nwill verify that $\\p(\\cdot)$ also proves the same property in $J_{1}^{(2)},J_{2}^{(2)},\\dots,J_{\\ell}^{(2)}$. \n\nFirst, to prove $J_{u}^{(2)}\\cap\\bigcup_{v=u+1}^{\\ell}J_{v}^{(2)}\\subseteq J_{\\p(u)}^{(2)},$\nlet $k\\in J_{u}^{(2)}\\cap J_{v}^{(2)}$ for $v>u$. The fact that\n$k\\in J_{u}^{(2)}=\\idx(\\clique(J_{u}))$ implies $k=\\idx(i,j)$ for\nsome $i,j$ such that $i,j\\in J_{u}$. Similarly, $k\\in J_{v}^{(2)}=\\idx(\\clique(J_{v}))$\nand the bijectivity of $\\idx$ on $F$ imply that the same $i,j$\nalso satisfy $i,j\\in J_{v}$, where we recall that $v>u$. We conclude\n$i,j\\in J_{u}\\cap J_{v}\\subseteq J_{\\p(u)}$ and therefore $k=\\idx(i,j)\\in J_{\\p(u)}^{(2)}$.\n\nNext, we prove $\\min\\{J_{\\p(u)}^{(2)}\\}>\\max\\{J_{u}^{(2)}\\backslash J_{\\p(u)}^{(2)}\\}$\nby establishing two claims:\n\\begin{itemize}\n\\item $\\min\\{J_{p(u)}^{(2)}\\}=\\idx(\\alpha,\\alpha)$ where $\\alpha=\\min J_{\\p(u)}$.\nFor any $\\idx(i,j)\\in J_{\\p(u)}^{(2)}$ where $i\\ge j$, we must have\n$i,j\\in J_{\\p(u)}$. It follows from the raster property that $\\idx(i,j)$\nis minimized with $i=j=\\min J_{\\p(u)}$.\n\\item $\\max\\{J_{u}^{(2)}\\backslash J_{\\p(u)}^{(2)}\\}=\\idx(\\beta,\\gamma)$\nwhere $\\beta=\\max\\{J_{u}\\}$ and $\\gamma=\\max\\{J_{u}\\backslash J_{\\p(u)}\\}$.\nWe partition $J_{u}$ into $N_{u}=J_{u}\\backslash J_{\\p(u)}$ and\n$A_{u}=J_{u}\\cap J_{\\p(u)}$. \n\\begin{itemize}\n\\item For any $\\idx(i,j)\\in J_{u}^{(2)}\\backslash J_{p(u)}^{(2)}$ where\n$i\\ge j$, we can have one of the following three cases: 1) $i\\in N_{u}$\nand $j\\in A_{u}$; 2) $i\\in A_{u}$ and $j\\in N_{u}$; or 3) $i\\in N_{u}$\nand $j\\in N_{u}$. \n\\item We observe that the first case $i\\in N_{u}$ and $j\\in A_{u}$ is\nimpossible. Indeed, applying $j\\ge\\min\\{J_{\\p(u)}\\}>\\max\\{J_{u}\\backslash J_{\\p(u)}\\}\\ge i$\nwould yield a contradiction with $i\\ge j$. \n\\item Taking the union of the two remaining cases yields $i\\in J_{u}=A_{u}\\cup N_{u}$\nand $j\\in N_{u}$. It follows from the raster property that $\\idx(i,j)$\nis maximized with $i=\\max J_{u}$ and $j=\\max N_{u}$.\n\\end{itemize}\n\\end{itemize}\nWith the two claims established, the hypothesis that $\\min\\{J_{\\p(u)}\\}>\\max\\{J_{u}\\backslash J_{\\p(u)}\\}$\nimplies that $\\alpha>\\gamma$, and therefore $\\idx(\\alpha,\\alpha)>\\idx(\\beta,\\gamma)$\nas desired. \\qed\n\\end{proof}","start":64900,"end":67246}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"提升交集沿同一父指针包含","source_units":[{"raw_tex":"\\begin{proof}\nLet $\\p(\\cdot)$ denote the parent pointer that verifies the sorted\nrunning intersection property in $J_{1},J_{2},\\dots,J_{\\ell}$. We\nwill verify that $\\p(\\cdot)$ also proves the same property in $J_{1}^{(2)},J_{2}^{(2)},\\dots,J_{\\ell}^{(2)}$. \n\nFirst, to prove $J_{u}^{(2)}\\cap\\bigcup_{v=u+1}^{\\ell}J_{v}^{(2)}\\subseteq J_{\\p(u)}^{(2)},$\nlet $k\\in J_{u}^{(2)}\\cap J_{v}^{(2)}$ for $v>u$. The fact that\n$k\\in J_{u}^{(2)}=\\idx(\\clique(J_{u}))$ implies $k=\\idx(i,j)$ for\nsome $i,j$ such that $i,j\\in J_{u}$. Similarly, $k\\in J_{v}^{(2)}=\\idx(\\clique(J_{v}))$\nand the bijectivity of $\\idx$ on $F$ imply that the same $i,j$\nalso satisfy $i,j\\in J_{v}$, where we recall that $v>u$. We conclude\n$i,j\\in J_{u}\\cap J_{v}\\subseteq J_{\\p(u)}$ and therefore $k=\\idx(i,j)\\in J_{\\p(u)}^{(2)}$.\n\n","start":64900,"end":65695}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"栅格次序保持排序不等式","source_units":[{"raw_tex":"Next, we prove $\\min\\{J_{\\p(u)}^{(2)}\\}>\\max\\{J_{u}^{(2)}\\backslash J_{\\p(u)}^{(2)}\\}$\nby establishing two claims:\n\\begin{itemize}\n\\item $\\min\\{J_{p(u)}^{(2)}\\}=\\idx(\\alpha,\\alpha)$ where $\\alpha=\\min J_{\\p(u)}$.\nFor any $\\idx(i,j)\\in J_{\\p(u)}^{(2)}$ where $i\\ge j$, we must have\n$i,j\\in J_{\\p(u)}$. It follows from the raster property that $\\idx(i,j)$\nis minimized with $i=j=\\min J_{\\p(u)}$.\n\\item $\\max\\{J_{u}^{(2)}\\backslash J_{\\p(u)}^{(2)}\\}=\\idx(\\beta,\\gamma)$\nwhere $\\beta=\\max\\{J_{u}\\}$ and $\\gamma=\\max\\{J_{u}\\backslash J_{\\p(u)}\\}$.\nWe partition $J_{u}$ into $N_{u}=J_{u}\\backslash J_{\\p(u)}$ and\n$A_{u}=J_{u}\\cap J_{\\p(u)}$. \n\\begin{itemize}\n\\item For any $\\idx(i,j)\\in J_{u}^{(2)}\\backslash J_{p(u)}^{(2)}$ where\n$i\\ge j$, we can have one of the following three cases: 1) $i\\in N_{u}$\nand $j\\in A_{u}$; 2) $i\\in A_{u}$ and $j\\in N_{u}$; or 3) $i\\in N_{u}$\nand $j\\in N_{u}$. \n\\item We observe that the first case $i\\in N_{u}$ and $j\\in A_{u}$ is\nimpossible. Indeed, applying $j\\ge\\min\\{J_{\\p(u)}\\}>\\max\\{J_{u}\\backslash J_{\\p(u)}\\}\\ge i$\nwould yield a contradiction with $i\\ge j$. \n\\item Taking the union of the two remaining cases yields $i\\in J_{u}=A_{u}\\cup N_{u}$\nand $j\\in N_{u}$. It follows from the raster property that $\\idx(i,j)$\nis maximized with $i=\\max J_{u}$ and $j=\\max N_{u}$.\n\\end{itemize}\n\\end{itemize}\nWith the two claims established, the hypothesis that $\\min\\{J_{\\p(u)}\\}>\\max\\{J_{u}\\backslash J_{\\p(u)}\\}$\nimplies that $\\alpha>\\gamma$, and therefore $\\idx(\\alpha,\\alpha)>\\idx(\\beta,\\gamma)$\nas desired. \\qed\n\\end{proof}","start":65695,"end":67246}]},{"id":"S073","semantic_type":"proposition","kind":"proposition","labels":["prop:rip->zf"],"declared_external":false,"source_only":false,"title":"有序运行交推出零填充","source_units":[{"raw_tex":"\\begin{proposition}[RIP$\\implies$ZF]\n\\label{prop:rip->zf}Let $J_{1},J_{2},\\dots,J_{\\ell}$ with $\\bigcup_{j=1}^{\\ell}J_{j}=\\{1,2,\\dots,n\\}$\nsatisfy the sorted running intersection property. Then, $F=\\bigcup_{j=1}^{\\ell}\\clique(J_{j})$\nis zero-fill, and we have $\\omega(F)=\\max_{j}|J_{j}|$.\n\\end{proposition}","start":67535,"end":67841}]},{"id":"S074","semantic_type":"lemma","kind":"lemma","labels":["lem:colrip"],"declared_external":false,"source_only":false,"title":"每个列集落在一个袋中","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:colrip}Let $J_{1},J_{2},\\dots,J_{\\ell}$ with $\\bigcup_{j=1}^{\\ell}J_{j}=\\{1,2,\\dots,n\\}$\nsatisfy the sorted running intersection property. For every $j$-th\ncolumn in $F=\\bigcup_{j=1}^{\\ell}\\clique(J_{j})$, there exists some\n$J_{w}$ such that $\\col_{F}(j)\\subseteq J_{w}.$\n\\end{lemma}","start":67974,"end":68282}]},{"id":"S075","semantic_type":"proof","kind":"proof","labels":["eq:par1"],"declared_external":false,"source_only":false,"title":"列集包含引理完整原证","source_units":[{"raw_tex":"\\begin{proof}\nLet $V=\\{1,2,\\dots,n\\}$. For arbitrary $j\\in V$, denote $J_{w}$\nas the last subset in the sequence $J_{1},J_{2},\\dots,J_{\\ell}$ for\nwhich $j\\in J_{w}$. This choice must exist, because $V=\\bigcup_{k=1}^{\\ell}J_{k}$.\nFor every arbitrary $i\\in\\col_{F}(j)$, we will prove that $i\\in J_{w}$\nalso holds:\n\\begin{itemize}\n\\item There exists $u\\le w$ for which $i,j\\in J_{u}$. Indeed, $(i,j)\\in F$\nand $F=\\bigcup_{k=1}^{\\ell}\\clique(J_{k})$ imply $(i,j)\\in\\clique(J_{u})$\nfor some $J_{u}$, or equivalently $i,j\\in J_{u}$. Given that $j\\in J_{u}$\nand $w=\\max\\{k:j\\in J_{k}\\}$ by definition, it follows that $u\\le w$.\n\\item If $u=w$, then $i\\in J_{w}$ holds because $i,j\\in J_{u}$. Otherwise,\nif $u<w$, we use the sorted running intersection property to assert\nthe following\n\\begin{gather}\nu<w,\\qquad i,j\\in J_{u}\\quad\\implies\\quad i,j\\in J_{\\p(u)}.\\label{eq:par1}\n\\end{gather}\nThe fact that $j\\in J_{\\p(u)}$ follows directly the running intersection\nproperty $j\\in J_{u}\\cap J_{w}\\subseteq J_{\\p(u)}$ for $w>u$. By\ncontradiction, suppose that $i\\notin J_{\\p(u)}$. Then, given that\n$i\\in J_{u}$, it follows from the sorted property that $j\\ge\\min\\{J_{\\p(u)}\\}>\\max\\{J_{u}\\backslash J_{\\p(u)}\\}\\ge i,$\nwhich contradicts our initial hypothesis that $i>j$. \n\\item Inductively reapplying (\\ref{eq:par1}), as in $i,j\\in J_{\\p(\\p(u))}$\nand $i,j\\in J_{\\p(\\p(\\p(u)))}$, we arrive at some $v$ such that\n$i,j\\in J_{\\p(v)}$ and $p(v)=w$. The induction must terminate with\n$p(v)\\ge w$ because each $\\p(u)>u$ by construction. It is impossible\nfor $\\p(v)>w$ to occur, because $j\\in J_{\\p(v)}$ and $w=\\max\\{k:j\\in J_{k}\\}$\nby definition. We conclude that $i\\in J_{w}$, as desired.\n\\end{itemize}\n\\qed\n\\end{proof}","start":68283,"end":69982}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","labels":["eq:par1"],"declared_external":false,"source_only":false,"title":"排序性质迫使成对传播","source_units":[{"raw_tex":"\\begin{proof}\nLet $V=\\{1,2,\\dots,n\\}$. For arbitrary $j\\in V$, denote $J_{w}$\nas the last subset in the sequence $J_{1},J_{2},\\dots,J_{\\ell}$ for\nwhich $j\\in J_{w}$. This choice must exist, because $V=\\bigcup_{k=1}^{\\ell}J_{k}$.\nFor every arbitrary $i\\in\\col_{F}(j)$, we will prove that $i\\in J_{w}$\nalso holds:\n\\begin{itemize}\n\\item There exists $u\\le w$ for which $i,j\\in J_{u}$. Indeed, $(i,j)\\in F$\nand $F=\\bigcup_{k=1}^{\\ell}\\clique(J_{k})$ imply $(i,j)\\in\\clique(J_{u})$\nfor some $J_{u}$, or equivalently $i,j\\in J_{u}$. Given that $j\\in J_{u}$\nand $w=\\max\\{k:j\\in J_{k}\\}$ by definition, it follows that $u\\le w$.\n\\item If $u=w$, then $i\\in J_{w}$ holds because $i,j\\in J_{u}$. Otherwise,\nif $u<w$, we use the sorted running intersection property to assert\nthe following\n\\begin{gather}\nu<w,\\qquad i,j\\in J_{u}\\quad\\implies\\quad i,j\\in J_{\\p(u)}.\\label{eq:par1}\n\\end{gather}\nThe fact that $j\\in J_{\\p(u)}$ follows directly the running intersection\nproperty $j\\in J_{u}\\cap J_{w}\\subseteq J_{\\p(u)}$ for $w>u$. By\ncontradiction, suppose that $i\\notin J_{\\p(u)}$. Then, given that\n$i\\in J_{u}$, it follows from the sorted property that $j\\ge\\min\\{J_{\\p(u)}\\}>\\max\\{J_{u}\\backslash J_{\\p(u)}\\}\\ge i,$\nwhich contradicts our initial hypothesis that $i>j$. \n","start":68283,"end":69541}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"递增父链终止于最后袋","source_units":[{"raw_tex":"\\item Inductively reapplying (\\ref{eq:par1}), as in $i,j\\in J_{\\p(\\p(u))}$\nand $i,j\\in J_{\\p(\\p(\\p(u)))}$, we arrive at some $v$ such that\n$i,j\\in J_{\\p(v)}$ and $p(v)=w$. The induction must terminate with\n$p(v)\\ge w$ because each $\\p(u)>u$ by construction. It is impossible\nfor $\\p(v)>w$ to occur, because $j\\in J_{\\p(v)}$ and $w=\\max\\{k:j\\in J_{k}\\}$\nby definition. We conclude that $i\\in J_{w}$, as desired.\n\\end{itemize}\n\\qed\n\\end{proof}","start":69541,"end":69982}]},{"id":"S078","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"运行交到零填充完整原证","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\propref{rip->zf}]To prove that $F$ is zero-fill,\nwe observe, for arbitrary $(i,k)\\in F$ and $(j,k)\\in F$ with $i>j>k$\nthat $i,j\\in\\col_{F}(k)$. Therefore, it follows from \\lemref{colrip}\nthat there exists $J_{w}$ such that $i,j,k\\in J_{w}$. We conclude\nthat $(i,j)\\in F,$ because $F=\\bigcup_{j=1}^{\\ell}\\clique(J_{j})$\nand $i,j\\in J_{w}$ for some $1\\le w\\le\\ell$. \n\nTo prove that $\\omega(F)=\\max_{j}|J_{j}|$, we choose $k=\\arg\\max_{j}|\\col_{F}(j)|$.\nIt follows from Lemma~\\ref{lem:colrip} that there exists $J_{w}$\nsuch that $\\col_{F}(k)\\subseteq J_{w}$, and therefore $\\omega(F)=|\\col_{F}(k)|\\le|J_{w}|$.\nFinally, given that $\\clique(J_{w})\\subseteq F$ it follows that $(i,j)\\in F$\nholds for all $i\\in J_{w}$ where $j=\\min J_{w}$. Therefore, we conclude\nthat $J_{w}\\subseteq\\col_{F}(j)$, and therefore $|J_{w}|\\le|\\col_{F}(j)|\\le|\\col_{F}(k)|=\\omega(F)$.\n\\qed \\end{proof}","start":70128,"end":71024}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"同列邻居落袋得到零填充","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\propref{rip->zf}]To prove that $F$ is zero-fill,\nwe observe, for arbitrary $(i,k)\\in F$ and $(j,k)\\in F$ with $i>j>k$\nthat $i,j\\in\\col_{F}(k)$. Therefore, it follows from \\lemref{colrip}\nthat there exists $J_{w}$ such that $i,j,k\\in J_{w}$. We conclude\nthat $(i,j)\\in F,$ because $F=\\bigcup_{j=1}^{\\ell}\\clique(J_{j})$\nand $i,j\\in J_{w}$ for some $1\\le w\\le\\ell$. \n","start":70128,"end":70517}]},{"id":"S080","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"袋大小与最大列大小相等","source_units":[{"raw_tex":"\nTo prove that $\\omega(F)=\\max_{j}|J_{j}|$, we choose $k=\\arg\\max_{j}|\\col_{F}(j)|$.\nIt follows from Lemma~\\ref{lem:colrip} that there exists $J_{w}$\nsuch that $\\col_{F}(k)\\subseteq J_{w}$, and therefore $\\omega(F)=|\\col_{F}(k)|\\le|J_{w}|$.\nFinally, given that $\\clique(J_{w})\\subseteq F$ it follows that $(i,j)\\in F$\nholds for all $i\\in J_{w}$ where $j=\\min J_{w}$. Therefore, we conclude\nthat $J_{w}\\subseteq\\col_{F}(j)$, and therefore $|J_{w}|\\le|\\col_{F}(j)|\\le|\\col_{F}(k)|=\\omega(F)$.\n\\qed \\end{proof}","start":70517,"end":71024}]},{"id":"S081","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"二次提升引理完整原证","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\lemref{quadlift}]For an arbitrary sparsity\npattern $E$ of order $n$, let $F=\\chol(E)$ and $J_{k}\\equiv\\col_{F}(k)$.\nFirst, it follows from \\propref{zf->rip} that $J_{1},J_{2},\\dots,J_{n}$\nsatisfy the ordered running intersection property. Therefore, it follows\nfrom \\lemref{lift}, $J_{1}^{(2)},J_{2}^{(2)},\\dots,J_{n}^{(2)}$\ndefined in (\\ref{eq:Jk2_def}) satisfy the same property. Finally,\nwe verify that \n\\begin{align*}\n\\supp(\\P_{k})\\equiv & \\left\\{ \\alpha:\\P_{k}[\\alpha,\\beta]\\ne0\\right\\} =\\left\\{ \\alpha:\\P_{k}^{T}e_{\\alpha}\\ne0\\right\\} \\\\\n\\overset{\\text{(a)}}{=} & \\left\\{ \\idx_{F}(i,j):\\P_{k}^{T}\\svec_{F}(e_{i}e_{j}^{T})\\ne0,\\;(i,j)\\in F\\right\\} \\\\\n\\overset{\\text{(b)}}{=} & \\left\\{ \\idx_{F}(i,j):(e_{i}e_{j}^{T})[J_{k},J_{k}]\\ne0,\\;i\\ge j\\right\\} =J_{k}^{(2)}.\n\\end{align*}\nEquality (a) is obtained by substituting $e_{\\alpha}=\\svec_{F}(e_{i}e_{j}^{T})$\nfor $(i,j)\\in F$. Equality (b) follows the identity $\\P_{k}^{T}\\svec_{F}(Y)=\\svec(Y[J_{k},J_{k}])$,\nwhich was used to define $\\P_{k}$. Therefore, $F^{(2)}=\\bigcup_{k=1}^{n}J_{k}^{(2)}$\nand we conclude via \\propref{rip->zf} that $\\omega(F^{(2)})=\\max_{k}|J_{k}^{(2)}|=\\max_{k}\\frac{1}{2}|J_{k}|(|J_{k}|+1)$,\nand we recall that $\\max_{k}|J_{k}|=\\omega(F)=\\omega(E)$ by definition.\n\\qed \\end{proof}","start":71246,"end":72527}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"列袋及提升袋继承运行交","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\lemref{quadlift}]For an arbitrary sparsity\npattern $E$ of order $n$, let $F=\\chol(E)$ and $J_{k}\\equiv\\col_{F}(k)$.\nFirst, it follows from \\propref{zf->rip} that $J_{1},J_{2},\\dots,J_{n}$\nsatisfy the ordered running intersection property. Therefore, it follows\nfrom \\lemref{lift}, $J_{1}^{(2)},J_{2}^{(2)},\\dots,J_{n}^{(2)}$\ndefined in (\\ref{eq:Jk2_def}) satisfy the same property. Finally,\n","start":71246,"end":71661}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"提取算子支持等于提升袋","source_units":[{"raw_tex":"we verify that \n\\begin{align*}\n\\supp(\\P_{k})\\equiv & \\left\\{ \\alpha:\\P_{k}[\\alpha,\\beta]\\ne0\\right\\} =\\left\\{ \\alpha:\\P_{k}^{T}e_{\\alpha}\\ne0\\right\\} \\\\\n\\overset{\\text{(a)}}{=} & \\left\\{ \\idx_{F}(i,j):\\P_{k}^{T}\\svec_{F}(e_{i}e_{j}^{T})\\ne0,\\;(i,j)\\in F\\right\\} \\\\\n\\overset{\\text{(b)}}{=} & \\left\\{ \\idx_{F}(i,j):(e_{i}e_{j}^{T})[J_{k},J_{k}]\\ne0,\\;i\\ge j\\right\\} =J_{k}^{(2)}.\n\\end{align*}\nEquality (a) is obtained by substituting $e_{\\alpha}=\\svec_{F}(e_{i}e_{j}^{T})$\nfor $(i,j)\\in F$. Equality (b) follows the identity $\\P_{k}^{T}\\svec_{F}(Y)=\\svec(Y[J_{k},J_{k}])$,\nwhich was used to define $\\P_{k}$. Therefore, $F^{(2)}=\\bigcup_{k=1}^{n}J_{k}^{(2)}$\n","start":71661,"end":72317}]},{"id":"S084","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"运行交给出精确二次前沿","source_units":[{"raw_tex":"and we conclude via \\propref{rip->zf} that $\\omega(F^{(2)})=\\max_{k}|J_{k}^{(2)}|=\\max_{k}\\frac{1}{2}|J_{k}|(|J_{k}|+1)$,\nand we recall that $\\max_{k}|J_{k}|=\\omega(F)=\\omega(E)$ by definition.\n\\qed \\end{proof}","start":72317,"end":72527}]},{"id":"S085","semantic_type":"lemma","kind":"lemma","labels":[],"declared_external":false,"source_only":false,"title":"约束支持落入扩展提升袋","source_units":[{"raw_tex":"\\begin{lemma}\nFor every $i\\in\\{1,2,\\dots,m\\}$, there exists $k\\in\\{1,2,\\dots,n\\}$\nsuch that $\\supp(a_{i})\\subseteq\\supp(\\o{\\P}_{k})[V^{(2)}]$ where\n$V^{(2)}=\\{\\idx_{\\overline{F}}(i,j):i,j\\in F\\}$. \n\\end{lemma}","start":73318,"end":73527}]},{"id":"S086","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"约束支持包含完整原证","source_units":[{"raw_tex":"\\begin{proof}\nIt follows by repeating the proof of \\lemref{quadlift} that\n\\begin{gather*}\n\\supp(\\o{\\P}_{k})=\\{\\idx_{\\o F}(i,j):i,j\\in\\col_{\\o F}(k),i\\ge j\\},\\\\\n\\supp(\\o{\\P}_{k})[V^{(2)}]=\\{\\idx_{F}(i,j):i,j\\in\\col_{\\o F}(k),i\\ge j\\}.\n\\end{gather*}\nOur desired claim follows via the following sequence of inclusions\n\\begin{align*}\n\\supp(a_{i}) & =\\{v:a_{i}[v]\\ne0\\}=\\{\\idx_{F}(u,v):A_{i}[u,v]\\ne0,\\quad(u,v)\\in F\\}\\\\\n & \\overset{\\text{(a)}}{\\subseteq}\\{\\idx_{F}(u,v):u,v\\in\\supp(A_{i}),\\quad(u,v)\\in F\\}\\\\\n & \\overset{\\text{(b)}}{\\subseteq}\\{\\idx_{F}(u,v):u,v\\in\\col_{\\o F}(k),\\quad(u,v)\\in F\\}=\\supp(\\o{\\P}_{k})[V^{(2)}].\n\\end{align*}\nInclusion (a) is because $\\spar(A_{i})\\subseteq\\clique(\\supp(A_{i}))$.\nInclusion (b) is true via the following: If $J=\\supp(A_{i})$ satisfies\n$\\clique(J)\\subseteq\\o F$, then $J\\subseteq\\col_{\\o F}(k)$ where\n$k=\\min J$. Indeed, we have $k\\in\\col_{\\o F}(k)$ by definition.\nFor any arbitrary $j\\in J$ with $j>k$, we must have an edge $(j,k)\\in\\clique(J)\\subseteq\\o F$,\nand therefore $j\\in\\col_{\\o F}(k)$. \\qed\n\\end{proof}","start":73528,"end":74581}]},{"id":"S087","semantic_type":"lemma","kind":"lemma","labels":[],"declared_external":false,"source_only":false,"title":"原提取支持嵌入扩展支持","source_units":[{"raw_tex":"\\begin{lemma}\nFor every $k\\in\\{1,2,\\dots,n\\}$, we have $\\supp(\\P_{k})\\subseteq\\supp(\\o{\\P}_{k})[V^{(2)}]$\nwhere $V^{(2)}=\\{\\idx_{\\overline{F}}(i,j):i,j\\in F\\}$. \n\\end{lemma}","start":74583,"end":74756}]},{"id":"S088","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"提取支持嵌入完整原证","source_units":[{"raw_tex":"\\begin{proof}\nIt again follows by repeating the proof of \\lemref{quadlift} that\n\\begin{align*}\n\\supp(\\P_{k}) & =\\{\\idx_{F}(i,j):i,j\\in\\col_{F}(k),i\\ge j\\}\\\\\n & \\overset{\\text{(a)}}{\\subseteq}\\{\\idx_{F}(i,j):i,j\\in\\col_{\\o F}(k),i\\ge j\\}=\\supp(\\o{\\P}_{k})[V^{(2)}].\n\\end{align*}\nInclusion (a) is true because $E\\subseteq\\o E$ implies $F=\\chol(E)\\subseteq\\chol(\\o E)=\\o F$\nvia \\propref{del}, and therefore $\\col_{F}(k)\\subseteq\\col_{\\o F}(k)$.\n\\qed\n\\end{proof}","start":74757,"end":75215}]},{"id":"S089","semantic_type":"lemma","kind":"lemma","labels":[],"declared_external":false,"source_only":false,"title":"转换数据满行秩","source_units":[{"raw_tex":"\\begin{lemma}[Linear independence]\nWe have $\\A^{T}y=0$ if and only $y=0$. \n\\end{lemma}","start":99320,"end":99406}]},{"id":"S090","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"线性独立性完整原证","source_units":[{"raw_tex":"\\begin{proof}\nWe will prove $\\sum_{k=1}^{n}\\P_{k}\\P_{k}^{T}\\succeq I$, which implies\n$\\A\\A^{T}\\succeq I$ and hence $\\|\\A^{T}y\\|\\ge\\|y\\|$. For arbitrary\n$Y\\in\\S_{F}^{n}$ with $y=\\svec_{F}(Y)$, we observe that \n\\[\n\\|\\P_{k}^{T}y\\|^{2}=\\|\\svec(Y[J_{k},J_{k}])\\|^{2}=\\sum_{i,j\\in J_{k}}(Y[i,j])^{2}=\\sum_{(i,j)\\in\\clique(J_{k})}\\gamma_{ij}(Y[i,j])^{2}\n\\]\nwhere $\\gamma_{ij}=1$ if $i=j$ and $\\gamma_{ij}=2$ if $i\\ne j$.\nTherefore, we have\n\\[\n\\sum_{k=1}^{n}\\|\\P_{k}^{T}y\\|^{2}=\\sum_{k=1}^{n}\\sum_{(i,j)\\in\\clique(J_{k})}\\gamma_{ij}(Y[i,j])^{2}\\ge\\sum_{(i,j)\\in F}\\gamma_{ij}(Y[i,j])^{2}=\\|y\\|^{2}.\n\\]\nThe inequality is because $F=\\bigcup_{k=1}^{n}\\clique(J_{k})$. \\qed\n\\end{proof}","start":99407,"end":100080}]},{"id":"S091","semantic_type":"lemma","kind":"lemma","labels":[],"declared_external":false,"source_only":false,"title":"转换问题强对偶达到","source_units":[{"raw_tex":"\\begin{lemma}[Strong duality is attained]\nUnder \\asmref{strict}, there exists $x^{\\star},s^{\\star}\\in\\K$\nsatisfying $\\A x^{\\star}=\\b,$ $\\A^{T}y^{\\star}+s^{\\star}=\\c,$ $(x^{\\star})^{T}s^{\\star}=0$. \n\\end{lemma}","start":100082,"end":100291}]},{"id":"S092","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"强对偶转移完整原证","source_units":[{"raw_tex":"\\begin{proof}\nDefine $P_{j}$ implicitly to satisfy $P_{j}^{T}x=x[\\col_{F}(j)]$\nfor all $x\\in\\R^{m}$. \\asmref{strict} says that there exists $\\hat{X}\\succeq0$\nand $\\hat{y}\\le0$ and that satisfy $\\AA(\\hat{X})\\le b$ and $\\AA^{T}(\\hat{y})+\\hat{S}=C$\nand $\\inner C{\\hat{X}}=\\inner b{\\hat{y}}$. It follows from \\citep[Theorem~9.2]{vandenberghe2015chordal}\nthat $\\hat{S}=C-\\AA^{T}(\\hat{y})\\in\\S_{F}^{n}\\cap\\S_{+}^{n}$ if and\nonly if there exists $\\hat{S}_{j}\\succeq0$ such that $\\hat{S}=\\sum_{j=1}^{n}P_{j}\\hat{S}_{j}P_{j}^{T}$.\nNow, we turn to the primal-dual pair defined by the data $(\\A,\\b,\\c,\\K)$\nin (\\ref{eq:AKdef}), which is written \n\\begin{align*}\n & \\min_{v\\in\\R^{m},V_{j}\\in\\S^{\\omega_{j}}}\\left\\{ \\inner bv:\\begin{array}{c}\n\\proj_{F}\\left[\\AA^{T}(-v)+\\sum_{j=1}^{n}P_{j}V_{j}P_{j}^{T}-C\\right]=0,\\\\\nv\\ge0,\\quad V_{1},\\dots,V_{n}\\succeq0\n\\end{array}\\right\\} \\\\\n= & \\max_{Y\\in\\S_{F}^{n}}\\left\\{ \\inner{-C}Y:\\AA(Y)\\le b,\\;-P_{j}^{T}YP_{j}\\preceq0\\text{ for all }j\\in\\{1,2,\\dots,n\\}\\right\\} \n\\end{align*}\nWe can mechanically verify that $v^{\\star}=-\\hat{y}$ and $V_{j}^{\\star}=\\hat{S}_{j}$\nis feasible for the primal problem, and $Y^{\\star}=\\Pi_{F}(\\hat{X})$\nis feasible for the dual problem, and that the two objectives coincide\n$\\inner b{v^{\\star}}=\\inner{-C}{Y^{\\star}}$. Therefore, we conclude\nthat $x^{\\star}=(v^{\\star},\\svec(V_{1}^{\\star}),\\dots,\\svec(V_{n}^{\\star}))$\nand $y^{\\star}=\\svec_{F}(Y^{\\star})$ is a complementary solution\nsatisfying $\\A^{T}x^{\\star}=\\b,$ $\\A^{T}y^{\\star}+s^{\\star}=\\c,$\nand $\\inner{x^{\\star}}{s^{\\star}}=0$. \\qed \n\\end{proof}","start":100292,"end":101852}]},{"id":"S093","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"稀疏对偶松弛的局部分解","source_units":[{"raw_tex":"\\begin{proof}\nDefine $P_{j}$ implicitly to satisfy $P_{j}^{T}x=x[\\col_{F}(j)]$\nfor all $x\\in\\R^{m}$. \\asmref{strict} says that there exists $\\hat{X}\\succeq0$\nand $\\hat{y}\\le0$ and that satisfy $\\AA(\\hat{X})\\le b$ and $\\AA^{T}(\\hat{y})+\\hat{S}=C$\nand $\\inner C{\\hat{X}}=\\inner b{\\hat{y}}$. It follows from \\citep[Theorem~9.2]{vandenberghe2015chordal}\nthat $\\hat{S}=C-\\AA^{T}(\\hat{y})\\in\\S_{F}^{n}\\cap\\S_{+}^{n}$ if and\nonly if there exists $\\hat{S}_{j}\\succeq0$ such that $\\hat{S}=\\sum_{j=1}^{n}P_{j}\\hat{S}_{j}P_{j}^{T}$.\n","start":100292,"end":100814}]},{"id":"S094","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"构造转换后的互补最优解","source_units":[{"raw_tex":"Now, we turn to the primal-dual pair defined by the data $(\\A,\\b,\\c,\\K)$\nin (\\ref{eq:AKdef}), which is written \n\\begin{align*}\n & \\min_{v\\in\\R^{m},V_{j}\\in\\S^{\\omega_{j}}}\\left\\{ \\inner bv:\\begin{array}{c}\n\\proj_{F}\\left[\\AA^{T}(-v)+\\sum_{j=1}^{n}P_{j}V_{j}P_{j}^{T}-C\\right]=0,\\\\\nv\\ge0,\\quad V_{1},\\dots,V_{n}\\succeq0\n\\end{array}\\right\\} \\\\\n= & \\max_{Y\\in\\S_{F}^{n}}\\left\\{ \\inner{-C}Y:\\AA(Y)\\le b,\\;-P_{j}^{T}YP_{j}\\preceq0\\text{ for all }j\\in\\{1,2,\\dots,n\\}\\right\\} \n\\end{align*}\nWe can mechanically verify that $v^{\\star}=-\\hat{y}$ and $V_{j}^{\\star}=\\hat{S}_{j}$\nis feasible for the primal problem, and $Y^{\\star}=\\Pi_{F}(\\hat{X})$\nis feasible for the dual problem, and that the two objectives coincide\n$\\inner b{v^{\\star}}=\\inner{-C}{Y^{\\star}}$. Therefore, we conclude\nthat $x^{\\star}=(v^{\\star},\\svec(V_{1}^{\\star}),\\dots,\\svec(V_{n}^{\\star}))$\nand $y^{\\star}=\\svec_{F}(Y^{\\star})$ is a complementary solution\nsatisfying $\\A^{T}x^{\\star}=\\b,$ $\\A^{T}y^{\\star}+s^{\\star}=\\c,$\nand $\\inner{x^{\\star}}{s^{\\star}}=0$. \\qed \n\\end{proof}","start":100814,"end":101852}]},{"id":"S095","semantic_type":"proposition","kind":"proposition","labels":["prop:general"],"declared_external":false,"source_only":false,"title":"标准形式内点法复杂度","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:general}Let $(\\A,\\b,\\c,\\K)$ describe an SDP in $(n,\\omega)$-standard\nform, and let $\\opt$ denote its optimal value. Let $\\Tim$ and $\\Mem$\ndenote the time and memory needed, given data $\\A\\in\\R^{M\\times N}$,\n$g\\in\\R^{M}$, and a choice of $w\\in\\interior(\\K)$, to form and solve\n$\\A\\D\\A^{T}\\Delta y=g$ for $\\Delta y\\in\\R^{M}$, where $\\D^{-1}=\\nabla^{2}f(w)$\nand $f=-\\log\\det(w)$. Then, a general-purpose interior-point method\ncomputes $(x,y)$ that satisfy the following in $O(\\sqrt{n}\\log(1/\\epsilon))$\niterations\n\\begin{gather*}\n\\c^{T}x\\le\\mathrm{opt}+n\\cdot\\epsilon,\\quad\\|\\A x-\\b\\|\\le\\epsilon,\\quad x\\in\\K,\\\\\n\\b^{T}y\\ge\\mathrm{opt}-n\\cdot\\epsilon,\\quad\\c-\\A^{T}y+\\epsilon\\one_{\\K}\\in\\K,\n\\end{gather*}\nwith per-iteration costs of $O(\\omega^{2}n+\\nnz(\\A)+\\Tim)$ time and\n$O(\\omega n+\\nnz(\\A)+\\Mem)$ memory.\n\\end{proposition}","start":102105,"end":102959}]},{"id":"S096","semantic_type":"lemma","kind":"lemma","labels":["lem:eps"],"declared_external":false,"source_only":false,"title":"嵌入解的精度恢复","source_units":[{"raw_tex":"\\begin{lemma}[$\\epsilon$-accurate and $\\epsilon$-feasible]\n\\label{lem:eps}Let there exist $(x^{\\star},y^{\\star})$ such that\n$\\A x^{\\star}=\\b$ with $x^{\\star}\\in\\K$ and $\\c-\\A^{T}y^{\\star}\\equiv s^{\\star}\\in\\K$\nthat satisfies strong duality $\\b^{T}y^{\\star}=\\c^{T}x^{\\star}$.\nThen, a point $(x,y,s,\\tau,\\theta,\\kappa)$ that is feasible for (\\ref{eq:hsd})\nand satisfies the optimality condition (\\ref{eq:short-step}) also\nsatisfies the following, where $\\rho=1+\\one_{\\K}^{T}(x^{\\star}+s^{\\star})$:\n\\begin{align*}\n\\|\\A(x/\\tau)-\\b\\| & \\le\\frac{\\rho\\|r_{p}\\|}{\\gamma}\\cdot\\epsilon, & \\|\\A^{T}(y/\\tau)+(s/\\tau)-\\c\\| & \\le\\frac{\\rho\\|r_{d}\\|}{\\gamma}\\cdot\\epsilon, & \\frac{(x/\\tau)^{T}(s/\\tau)}{n+1}\\le\\frac{\\rho^{2}}{\\gamma^{2}}\\cdot\\epsilon.\n\\end{align*}\n\\end{lemma}","start":105764,"end":106525}]},{"id":"S097","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"嵌入精度恢复完整原证","source_units":[{"raw_tex":"\\begin{proof}\nFirst, observe that $(\\hat{x},\\hat{y},\\hat{s},\\hat{\\tau},\\hat{\\theta},\\hat{\\kappa})=(\\hat{\\tau}x^{\\star},\\hat{\\tau}y^{\\star},\\hat{\\tau}s^{\\star},\\hat{\\tau},0,0)$\nwith $\\hat{\\tau}=(n+1)/\\rho$ is a solution to (\\ref{eq:hsd}), because\n$r_{d}^{T}x^{\\star}+r_{p}^{T}y^{\\star}+r_{c}=\\one_{\\K}^{T}(x^{\\star}+s^{\\star})+1=\\rho$.\nNext, we prove that if $(x,y,s,\\tau,\\theta,\\kappa)$ is feasible for\n(\\ref{eq:hsd}) and satisfies (\\ref{eq:short-step}), then $\\tau\\ge\\frac{\\gamma}{\\rho}$.\nIndeed, the skew-symmetry of (\\ref{eq:hsdfeas}) yields $\\theta=\\frac{x^{T}s+\\tau\\kappa}{n+1}$\nand $(x-\\hat{x})^{T}(s-\\hat{s})+(\\tau-\\hat{\\tau})(\\kappa-\\hat{\\kappa})=0.$\nRearranging yields $(n+1)\\theta=x^{T}s+\\tau\\kappa=\\hat{x}^{T}s+x^{T}\\hat{s}+\\tau\\hat{\\kappa}+\\hat{\\tau}\\kappa$\nand hence $\\kappa\\le(n+1)\\theta/\\hat{\\tau}$. Under (\\ref{eq:short-step}),\nwe have $\\theta=\\frac{x^{T}s+\\tau\\kappa}{n+1}\\le\\epsilon$ and $\\tau\\ge\\frac{\\gamma\\epsilon}{\\kappa}\\ge\\frac{\\gamma\\theta}{\\kappa}\\ge\\frac{\\gamma\\theta}{(n+1)\\theta/\\hat{\\tau}}=\\frac{\\gamma}{n+1}\\hat{\\tau}=\\frac{\\gamma}{\\rho}$.\nFinally, divide (\\ref{eq:hsdfeas}) through by $\\tau$ and substitute\n$1/\\tau\\le\\rho/\\gamma$. \\qed\n\\end{proof}","start":106526,"end":107705}]},{"id":"S098","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"最优解生成嵌入可行点","source_units":[{"raw_tex":"\\begin{proof}\nFirst, observe that $(\\hat{x},\\hat{y},\\hat{s},\\hat{\\tau},\\hat{\\theta},\\hat{\\kappa})=(\\hat{\\tau}x^{\\star},\\hat{\\tau}y^{\\star},\\hat{\\tau}s^{\\star},\\hat{\\tau},0,0)$\nwith $\\hat{\\tau}=(n+1)/\\rho$ is a solution to (\\ref{eq:hsd}), because\n$r_{d}^{T}x^{\\star}+r_{p}^{T}y^{\\star}+r_{c}=\\one_{\\K}^{T}(x^{\\star}+s^{\\star})+1=\\rho$.\n","start":106526,"end":106861}]},{"id":"S099","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"下界控制归一化分母","source_units":[{"raw_tex":"Next, we prove that if $(x,y,s,\\tau,\\theta,\\kappa)$ is feasible for\n(\\ref{eq:hsd}) and satisfies (\\ref{eq:short-step}), then $\\tau\\ge\\frac{\\gamma}{\\rho}$.\nIndeed, the skew-symmetry of (\\ref{eq:hsdfeas}) yields $\\theta=\\frac{x^{T}s+\\tau\\kappa}{n+1}$\nand $(x-\\hat{x})^{T}(s-\\hat{s})+(\\tau-\\hat{\\tau})(\\kappa-\\hat{\\kappa})=0.$\nRearranging yields $(n+1)\\theta=x^{T}s+\\tau\\kappa=\\hat{x}^{T}s+x^{T}\\hat{s}+\\tau\\hat{\\kappa}+\\hat{\\tau}\\kappa$\nand hence $\\kappa\\le(n+1)\\theta/\\hat{\\tau}$. Under (\\ref{eq:short-step}),\nwe have $\\theta=\\frac{x^{T}s+\\tau\\kappa}{n+1}\\le\\epsilon$ and $\\tau\\ge\\frac{\\gamma\\epsilon}{\\kappa}\\ge\\frac{\\gamma\\theta}{\\kappa}\\ge\\frac{\\gamma\\theta}{(n+1)\\theta/\\hat{\\tau}}=\\frac{\\gamma}{n+1}\\hat{\\tau}=\\frac{\\gamma}{\\rho}$.\nFinally, divide (\\ref{eq:hsdfeas}) through by $\\tau$ and substitute\n$1/\\tau\\le\\rho/\\gamma$. \\qed\n\\end{proof}","start":106861,"end":107705}]},{"id":"S100","semantic_type":"proof","kind":"proof","labels":["eq:hsd_back1","eq:hsd_back2","eq:hsd_nrm","eq:newt"],"declared_external":false,"source_only":false,"title":"通用内点法命题完整原证","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\propref{general}]Recall that $(\\A,\\b,\\c,\\K)$\ndescribe an SDP in $(n,\\omega)$-standard form, and hence there exist\n$(x^{\\star},y^{\\star})$ such that $\\A x^{\\star}=\\b$ with $x^{\\star}\\in\\K$\nand $\\c-\\A^{T}y^{\\star}\\in\\K$ that satisfies strong duality $\\b^{T}y^{\\star}=\\c^{T}x^{\\star}$.\nCombining \\lemref{eps} and \\lemref{short-step} shows that iterations\nin (\\ref{eq:hsd_ipm}) converges to the desired $\\epsilon$-accurate,\n$\\epsilon$-feasible iterate after $O(\\sqrt{n}\\log(1/\\epsilon))$\niterations. The cost of each iteration is essentially equal to the\ncost of computing the search direction in (\\ref{eq:hsd_nt}). We account\nfor this cost via the following two steps:\n\\begin{enumerate}\n\\item (Scaling point) We partition $x=[\\svec(X_{j})]_{j=1}^{\\ell}$ and\n$s=[\\svec(S_{j})]_{j=1}^{\\ell}$. Then, the scaling point $w=[\\svec(W_{j})]_{j=1}^{\\ell}$\nis given in closed-form as $W_{j}=S_{j}^{1/2}(S_{j}^{1/2}X_{j}S_{j}^{1/2})^{-1/2}S_{j}^{1/2}$~\\citep[Section~5]{sturm2002implementation}.\nNoting that each $W_{j}$ is at most $\\omega\\times\\omega$, the cost\nof forming $w$ is at most of order \n\\[\n\\sum_{j=1}^{\\ell}\\omega_{j}^{3}\\le\\omega^{2}\\sum_{j=1}^{\\ell}\\omega_{j}=\\omega^{2}n\\text{ time},\\quad\\sum_{j=1}^{\\ell}\\omega_{j}^{2}\\le\\omega\\sum_{j=1}^{\\ell}\\omega_{j}=\\omega n\\text{ memory.}\n\\]\nBy this same calculation, it follows that the Hessian matrix-vector\nproducts $\\nabla^{2}f(w)x=[\\svec(W_{j}^{-1}X_{j}W_{j}^{-1})]_{j=1}^{\\ell}$\nand $\\nabla^{2}f(w)^{-1}x=[\\svec(W_{j}X_{j}W_{j})]_{j=1}^{\\ell}$\nalso cost $O(\\omega^{2}n)$ time and $O(\\omega n)$ memory. \n\\item (Search direction) Using elementary but tedious linear algebra, we\ncan show that if\\begin{subequations}\\label{eq:newt}\n\\begin{equation}\n(\\A\\D\\A^{T})\\begin{bmatrix}v_{1} & v_{2} & v_{3}\\end{bmatrix}=\\begin{bmatrix}0 & -\\b & r_{p}\\end{bmatrix}-\\A\\D\\begin{bmatrix}d & \\c & r_{d}\\end{bmatrix}\\label{eq:hsd_nrm}\n\\end{equation}\nwhere $\\D=[\\nabla^{2}f(w)]^{-1}$ and $d=-s-\\mu^{+}\\nabla f(x)$,\nand \n\\begin{equation}\n\\begin{bmatrix}u_{1} & u_{2} & u_{3}\\end{bmatrix}=\\D(\\begin{bmatrix}d & c & r_{d}\\end{bmatrix}+\\A^{T}\\begin{bmatrix}v_{1} & v_{2} & v_{3}\\end{bmatrix}),\\label{eq:hsd_back1}\n\\end{equation}\nthen\n\\begin{align}\n\\left(\\begin{bmatrix}-\\D_{0}^{-1} & -r_{c}\\\\\nr_{c} & 0\n\\end{bmatrix}-\\begin{bmatrix}\\c & r_{d}\\\\\n-\\b & r_{p}\n\\end{bmatrix}^{T}\\begin{bmatrix}u_{2} & u_{3}\\\\\nv_{2} & v_{3}\n\\end{bmatrix}\\right)\\begin{bmatrix}\\Delta\\tau\\\\\n\\Delta\\theta\n\\end{bmatrix} & =\\begin{bmatrix}-d_{0}\\\\\n0\n\\end{bmatrix}-\\begin{bmatrix}\\c & r_{d}\\\\\n-\\b & r_{p}\n\\end{bmatrix}^{T}\\begin{bmatrix}u_{1}\\\\\nv_{1}\n\\end{bmatrix},\\\\\n\\begin{bmatrix}\\Delta x\\\\\n\\Delta y\n\\end{bmatrix} & =\\begin{bmatrix}u_{1}\\\\\nv_{1}\n\\end{bmatrix}-\\begin{bmatrix}u_{1} & u_{2}\\\\\nv_{1} & v_{2}\n\\end{bmatrix}\\begin{bmatrix}\\Delta\\tau\\\\\n\\Delta\\theta\n\\end{bmatrix},\\\\\n\\Delta s & =d-\\D^{-1}\\Delta x,\\\\\n\\Delta\\kappa & =d_{0}-\\D_{0}^{-1}\\Delta\\tau,\\label{eq:hsd_back2}\n\\end{align}\n\\end{subequations}where $\\D_{0}=\\tau/\\kappa$ and $d_{0}=-\\kappa+\\mu^{+}\\tau^{-1}$.\nTherefore, we conclude that the cost of computing the search direction\nin (\\ref{eq:hsd_nt}) is equal to the cost of solving $O(1)$ instances\nof the Schur complement equation $\\A\\D\\A^{T}\\Delta y=g$, plus $O(1)$\nmatrix-vector products with $\\A,\\A^{T},\\D,\\D^{-1}$, for a total cost\nof $O(\\Tim+\\nnz(\\A)+\\omega^{2}n)$ time and $O(\\Mem+\\nnz(\\A)+\\omega n)$\nmemory respectively. Note that $\\A\\D\\A^{T}\\succ0$ because $\\A\\A^{T}\\succ0$\nby the linear independence assumption, and $\\D^{-1}=\\nabla^{2}f(w)\\succ0$\nfor all $w\\in\\interior(\\K)$.\n\\end{enumerate}\n\\qed \\end{proof}","start":107707,"end":111259}]},{"id":"S101","semantic_type":"proof_step","kind":"proof_step","labels":["eq:newt"],"declared_external":false,"source_only":false,"title":"短步与误差恢复合并","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\propref{general}]Recall that $(\\A,\\b,\\c,\\K)$\ndescribe an SDP in $(n,\\omega)$-standard form, and hence there exist\n$(x^{\\star},y^{\\star})$ such that $\\A x^{\\star}=\\b$ with $x^{\\star}\\in\\K$\nand $\\c-\\A^{T}y^{\\star}\\in\\K$ that satisfies strong duality $\\b^{T}y^{\\star}=\\c^{T}x^{\\star}$.\nCombining \\lemref{eps} and \\lemref{short-step} shows that iterations\nin (\\ref{eq:hsd_ipm}) converges to the desired $\\epsilon$-accurate,\n$\\epsilon$-feasible iterate after $O(\\sqrt{n}\\log(1/\\epsilon))$\niterations. The cost of each iteration is essentially equal to the\ncost of computing the search direction in (\\ref{eq:hsd_nt}). We account\nfor this cost via the following two steps:\n","start":107707,"end":108397}]},{"id":"S102","semantic_type":"proof_step","kind":"proof_step","labels":["eq:newt"],"declared_external":false,"source_only":false,"title":"分块缩放与 Hessian 乘法成本","source_units":[{"raw_tex":"\\begin{enumerate}\n\\item (Scaling point) We partition $x=[\\svec(X_{j})]_{j=1}^{\\ell}$ and\n$s=[\\svec(S_{j})]_{j=1}^{\\ell}$. Then, the scaling point $w=[\\svec(W_{j})]_{j=1}^{\\ell}$\nis given in closed-form as $W_{j}=S_{j}^{1/2}(S_{j}^{1/2}X_{j}S_{j}^{1/2})^{-1/2}S_{j}^{1/2}$~\\citep[Section~5]{sturm2002implementation}.\nNoting that each $W_{j}$ is at most $\\omega\\times\\omega$, the cost\nof forming $w$ is at most of order \n\\[\n\\sum_{j=1}^{\\ell}\\omega_{j}^{3}\\le\\omega^{2}\\sum_{j=1}^{\\ell}\\omega_{j}=\\omega^{2}n\\text{ time},\\quad\\sum_{j=1}^{\\ell}\\omega_{j}^{2}\\le\\omega\\sum_{j=1}^{\\ell}\\omega_{j}=\\omega n\\text{ memory.}\n\\]\nBy this same calculation, it follows that the Hessian matrix-vector\nproducts $\\nabla^{2}f(w)x=[\\svec(W_{j}^{-1}X_{j}W_{j}^{-1})]_{j=1}^{\\ell}$\nand $\\nabla^{2}f(w)^{-1}x=[\\svec(W_{j}X_{j}W_{j})]_{j=1}^{\\ell}$\nalso cost $O(\\omega^{2}n)$ time and $O(\\omega n)$ memory. \n","start":108397,"end":109282}]},{"id":"S103","semantic_type":"proof_step","kind":"proof_step","labels":["eq:hsd_back1","eq:hsd_back2","eq:hsd_nrm","eq:newt"],"declared_external":false,"source_only":false,"title":"搜索方向的 Schur 消元实现","source_units":[{"raw_tex":"\\item (Search direction) Using elementary but tedious linear algebra, we\ncan show that if\\begin{subequations}\\label{eq:newt}\n\\begin{equation}\n(\\A\\D\\A^{T})\\begin{bmatrix}v_{1} & v_{2} & v_{3}\\end{bmatrix}=\\begin{bmatrix}0 & -\\b & r_{p}\\end{bmatrix}-\\A\\D\\begin{bmatrix}d & \\c & r_{d}\\end{bmatrix}\\label{eq:hsd_nrm}\n\\end{equation}\nwhere $\\D=[\\nabla^{2}f(w)]^{-1}$ and $d=-s-\\mu^{+}\\nabla f(x)$,\nand \n\\begin{equation}\n\\begin{bmatrix}u_{1} & u_{2} & u_{3}\\end{bmatrix}=\\D(\\begin{bmatrix}d & c & r_{d}\\end{bmatrix}+\\A^{T}\\begin{bmatrix}v_{1} & v_{2} & v_{3}\\end{bmatrix}),\\label{eq:hsd_back1}\n\\end{equation}\nthen\n\\begin{align}\n\\left(\\begin{bmatrix}-\\D_{0}^{-1} & -r_{c}\\\\\nr_{c} & 0\n\\end{bmatrix}-\\begin{bmatrix}\\c & r_{d}\\\\\n-\\b & r_{p}\n\\end{bmatrix}^{T}\\begin{bmatrix}u_{2} & u_{3}\\\\\nv_{2} & v_{3}\n\\end{bmatrix}\\right)\\begin{bmatrix}\\Delta\\tau\\\\\n\\Delta\\theta\n\\end{bmatrix} & =\\begin{bmatrix}-d_{0}\\\\\n0\n\\end{bmatrix}-\\begin{bmatrix}\\c & r_{d}\\\\\n-\\b & r_{p}\n\\end{bmatrix}^{T}\\begin{bmatrix}u_{1}\\\\\nv_{1}\n\\end{bmatrix},\\\\\n\\begin{bmatrix}\\Delta x\\\\\n\\Delta y\n\\end{bmatrix} & =\\begin{bmatrix}u_{1}\\\\\nv_{1}\n\\end{bmatrix}-\\begin{bmatrix}u_{1} & u_{2}\\\\\nv_{1} & v_{2}\n\\end{bmatrix}\\begin{bmatrix}\\Delta\\tau\\\\\n\\Delta\\theta\n\\end{bmatrix},\\\\\n\\Delta s & =d-\\D^{-1}\\Delta x,\\\\\n\\Delta\\kappa & =d_{0}-\\D_{0}^{-1}\\Delta\\tau,\\label{eq:hsd_back2}\n\\end{align}\n\\end{subequations}where $\\D_{0}=\\tau/\\kappa$ and $d_{0}=-\\kappa+\\mu^{+}\\tau^{-1}$.\nTherefore, we conclude that the cost of computing the search direction\nin (\\ref{eq:hsd_nt}) is equal to the cost of solving $O(1)$ instances\nof the Schur complement equation $\\A\\D\\A^{T}\\Delta y=g$, plus $O(1)$\nmatrix-vector products with $\\A,\\A^{T},\\D,\\D^{-1}$, for a total cost\nof $O(\\Tim+\\nnz(\\A)+\\omega^{2}n)$ time and $O(\\Mem+\\nnz(\\A)+\\omega n)$\nmemory respectively. Note that $\\A\\D\\A^{T}\\succ0$ because $\\A\\A^{T}\\succ0$\nby the linear independence assumption, and $\\D^{-1}=\\nabla^{2}f(w)\\succ0$\nfor all $w\\in\\interior(\\K)$.\n\\end{enumerate}\n\\qed \\end{proof}","start":109282,"end":111259}]},{"id":"S104","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件：Vandenberghe and Andersen, Theorem 6.1；原文 L1152–1155 引用。"],"source_only":false,"title":"Cholesky 填充路径定理","source_units":[]},{"id":"S105","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件：Vandenberghe and Andersen, Theorem 9.2；原文 L1874–1876 引用。"],"source_only":false,"title":"弦稀疏半正定分解","source_units":[]},{"id":"S106","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件：Standard positive-definite Cholesky factorization, triangular solves, and matrix-operation costs；正定合同结论为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"基础矩阵运算成本与正定性","source_units":[]},{"id":"S107","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件：Todd and collaborators, equation 9，L1954；Sturm, Section 5，L2028。"],"source_only":false,"title":"Nesterov–Todd 方向与缩放公式","source_units":[]},{"id":"S108","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；用于 L1076–1078 和 L2006–2012 的省略背景。"],"source_only":false,"title":"锥序与残差的标准转换","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S038","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S006","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S004","kind":"supports"},{"source_id":"S007","target_id":"S004","kind":"supports"},{"source_id":"S006","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S009","target_id":"S023","kind":"supports"},{"source_id":"S013","target_id":"S023","kind":"supports"},{"source_id":"S021","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S007","target_id":"S024","kind":"supports"},{"source_id":"S008","target_id":"S024","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S019","kind":"supports"},{"source_id":"S011","target_id":"S028","kind":"supports"},{"source_id":"S009","target_id":"S030","kind":"supports"},{"source_id":"S007","target_id":"S030","kind":"supports"},{"source_id":"S013","target_id":"S030","kind":"supports"},{"source_id":"S009","target_id":"S031","kind":"supports"},{"source_id":"S027","target_id":"S032","kind":"supports"},{"source_id":"S007","target_id":"S032","kind":"supports"},{"source_id":"S026","target_id":"S038","kind":"supports"},{"source_id":"S004","target_id":"S038","kind":"supports"},{"source_id":"S015","target_id":"S038","kind":"supports"},{"source_id":"S021","target_id":"S038","kind":"supports"},{"source_id":"S052","target_id":"S038","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S055","target_id":"S052","kind":"supports"},{"source_id":"S056","target_id":"S052","kind":"supports"},{"source_id":"S057","target_id":"S052","kind":"supports"},{"source_id":"S058","target_id":"S052","kind":"supports"},{"source_id":"S059","target_id":"S052","kind":"supports"},{"source_id":"S018","target_id":"S053","kind":"supports"},{"source_id":"S023","target_id":"S053","kind":"supports"},{"source_id":"S089","target_id":"S053","kind":"supports"},{"source_id":"S091","target_id":"S053","kind":"supports"},{"source_id":"S016","target_id":"S053","kind":"supports"},{"source_id":"S060","target_id":"S053","kind":"supports"},{"source_id":"S002","target_id":"S053","kind":"supports"},{"source_id":"S004","target_id":"S053","kind":"supports"},{"source_id":"S014","target_id":"S053","kind":"supports"},{"source_id":"S012","target_id":"S054","kind":"supports"},{"source_id":"S013","target_id":"S054","kind":"supports"},{"source_id":"S004","target_id":"S054","kind":"supports"},{"source_id":"S060","target_id":"S054","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S019","target_id":"S055","kind":"supports"},{"source_id":"S023","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S095","target_id":"S055","kind":"supports"},{"source_id":"S045","target_id":"S056","kind":"supports"},{"source_id":"S019","target_id":"S056","kind":"supports"},{"source_id":"S053","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S057","kind":"supports"},{"source_id":"S021","target_id":"S057","kind":"supports"},{"source_id":"S022","target_id":"S057","kind":"supports"},{"source_id":"S055","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S006","target_id":"S058","kind":"supports"},{"source_id":"S055","target_id":"S059","kind":"supports"},{"source_id":"S108","target_id":"S059","kind":"supports"},{"source_id":"S010","target_id":"S040","kind":"supports"},{"source_id":"S013","target_id":"S040","kind":"supports"},{"source_id":"S106","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S024","target_id":"S043","kind":"supports"},{"source_id":"S060","target_id":"S043","kind":"supports"},{"source_id":"S064","target_id":"S043","kind":"supports"},{"source_id":"S065","target_id":"S044","kind":"supports"},{"source_id":"S060","target_id":"S044","kind":"supports"},{"source_id":"S064","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S044","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S023","target_id":"S047","kind":"supports"},{"source_id":"S024","target_id":"S047","kind":"supports"},{"source_id":"S041","target_id":"S047","kind":"supports"},{"source_id":"S023","target_id":"S048","kind":"supports"},{"source_id":"S024","target_id":"S048","kind":"supports"},{"source_id":"S013","target_id":"S048","kind":"supports"},{"source_id":"S039","target_id":"S049","kind":"supports"},{"source_id":"S041","target_id":"S049","kind":"supports"},{"source_id":"S013","target_id":"S049","kind":"supports"},{"source_id":"S089","target_id":"S049","kind":"supports"},{"source_id":"S023","target_id":"S049","kind":"supports"},{"source_id":"S106","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S106","target_id":"S050","kind":"supports"},{"source_id":"S047","target_id":"S051","kind":"supports"},{"source_id":"S048","target_id":"S051","kind":"supports"},{"source_id":"S049","target_id":"S051","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S049","target_id":"S046","kind":"supports"},{"source_id":"S050","target_id":"S046","kind":"supports"},{"source_id":"S051","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S104","target_id":"S062","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S027","target_id":"S063","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S063","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S028","target_id":"S068","kind":"supports"},{"source_id":"S029","target_id":"S068","kind":"supports"},{"source_id":"S013","target_id":"S068","kind":"supports"},{"source_id":"S068","target_id":"S067","kind":"supports"},{"source_id":"S029","target_id":"S071","kind":"supports"},{"source_id":"S030","target_id":"S071","kind":"supports"},{"source_id":"S009","target_id":"S071","kind":"supports"},{"source_id":"S031","target_id":"S072","kind":"supports"},{"source_id":"S029","target_id":"S072","kind":"supports"},{"source_id":"S030","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S070","kind":"supports"},{"source_id":"S072","target_id":"S070","kind":"supports"},{"source_id":"S070","target_id":"S069","kind":"supports"},{"source_id":"S029","target_id":"S076","kind":"supports"},{"source_id":"S013","target_id":"S076","kind":"supports"},{"source_id":"S007","target_id":"S076","kind":"supports"},{"source_id":"S076","target_id":"S077","kind":"supports"},{"source_id":"S029","target_id":"S077","kind":"supports"},{"source_id":"S076","target_id":"S075","kind":"supports"},{"source_id":"S077","target_id":"S075","kind":"supports"},{"source_id":"S075","target_id":"S074","kind":"supports"},{"source_id":"S074","target_id":"S079","kind":"supports"},{"source_id":"S028","target_id":"S079","kind":"supports"},{"source_id":"S007","target_id":"S079","kind":"supports"},{"source_id":"S074","target_id":"S080","kind":"supports"},{"source_id":"S079","target_id":"S080","kind":"supports"},{"source_id":"S013","target_id":"S080","kind":"supports"},{"source_id":"S007","target_id":"S080","kind":"supports"},{"source_id":"S079","target_id":"S078","kind":"supports"},{"source_id":"S080","target_id":"S078","kind":"supports"},{"source_id":"S078","target_id":"S073","kind":"supports"},{"source_id":"S067","target_id":"S082","kind":"supports"},{"source_id":"S069","target_id":"S082","kind":"supports"},{"source_id":"S011","target_id":"S082","kind":"supports"},{"source_id":"S028","target_id":"S082","kind":"supports"},{"source_id":"S030","target_id":"S082","kind":"supports"},{"source_id":"S023","target_id":"S083","kind":"supports"},{"source_id":"S009","target_id":"S083","kind":"supports"},{"source_id":"S030","target_id":"S083","kind":"supports"},{"source_id":"S082","target_id":"S084","kind":"supports"},{"source_id":"S083","target_id":"S084","kind":"supports"},{"source_id":"S073","target_id":"S084","kind":"supports"},{"source_id":"S013","target_id":"S084","kind":"supports"},{"source_id":"S082","target_id":"S081","kind":"supports"},{"source_id":"S083","target_id":"S081","kind":"supports"},{"source_id":"S084","target_id":"S081","kind":"supports"},{"source_id":"S081","target_id":"S064","kind":"supports"},{"source_id":"S083","target_id":"S086","kind":"supports"},{"source_id":"S032","target_id":"S086","kind":"supports"},{"source_id":"S004","target_id":"S086","kind":"supports"},{"source_id":"S007","target_id":"S086","kind":"supports"},{"source_id":"S009","target_id":"S086","kind":"supports"},{"source_id":"S086","target_id":"S085","kind":"supports"},{"source_id":"S083","target_id":"S088","kind":"supports"},{"source_id":"S060","target_id":"S088","kind":"supports"},{"source_id":"S002","target_id":"S088","kind":"supports"},{"source_id":"S004","target_id":"S088","kind":"supports"},{"source_id":"S032","target_id":"S088","kind":"supports"},{"source_id":"S088","target_id":"S087","kind":"supports"},{"source_id":"S085","target_id":"S066","kind":"supports"},{"source_id":"S087","target_id":"S066","kind":"supports"},{"source_id":"S024","target_id":"S066","kind":"supports"},{"source_id":"S032","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"},{"source_id":"S023","target_id":"S090","kind":"supports"},{"source_id":"S009","target_id":"S090","kind":"supports"},{"source_id":"S011","target_id":"S090","kind":"supports"},{"source_id":"S013","target_id":"S090","kind":"supports"},{"source_id":"S028","target_id":"S090","kind":"supports"},{"source_id":"S090","target_id":"S089","kind":"supports"},{"source_id":"S026","target_id":"S093","kind":"supports"},{"source_id":"S105","target_id":"S093","kind":"supports"},{"source_id":"S023","target_id":"S093","kind":"supports"},{"source_id":"S011","target_id":"S093","kind":"supports"},{"source_id":"S093","target_id":"S094","kind":"supports"},{"source_id":"S023","target_id":"S094","kind":"supports"},{"source_id":"S009","target_id":"S094","kind":"supports"},{"source_id":"S026","target_id":"S094","kind":"supports"},{"source_id":"S093","target_id":"S092","kind":"supports"},{"source_id":"S094","target_id":"S092","kind":"supports"},{"source_id":"S092","target_id":"S091","kind":"supports"},{"source_id":"S018","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S107","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S018","target_id":"S098","kind":"supports"},{"source_id":"S034","target_id":"S098","kind":"supports"},{"source_id":"S098","target_id":"S099","kind":"supports"},{"source_id":"S034","target_id":"S099","kind":"supports"},{"source_id":"S036","target_id":"S099","kind":"supports"},{"source_id":"S108","target_id":"S099","kind":"supports"},{"source_id":"S098","target_id":"S097","kind":"supports"},{"source_id":"S099","target_id":"S097","kind":"supports"},{"source_id":"S097","target_id":"S096","kind":"supports"},{"source_id":"S018","target_id":"S101","kind":"supports"},{"source_id":"S035","target_id":"S101","kind":"supports"},{"source_id":"S096","target_id":"S101","kind":"supports"},{"source_id":"S036","target_id":"S101","kind":"supports"},{"source_id":"S107","target_id":"S102","kind":"supports"},{"source_id":"S018","target_id":"S102","kind":"supports"},{"source_id":"S035","target_id":"S102","kind":"supports"},{"source_id":"S106","target_id":"S102","kind":"supports"},{"source_id":"S035","target_id":"S103","kind":"supports"},{"source_id":"S102","target_id":"S103","kind":"supports"},{"source_id":"S018","target_id":"S103","kind":"supports"},{"source_id":"S106","target_id":"S103","kind":"supports"},{"source_id":"S101","target_id":"S100","kind":"supports"},{"source_id":"S102","target_id":"S100","kind":"supports"},{"source_id":"S103","target_id":"S100","kind":"supports"},{"source_id":"S100","target_id":"S095","kind":"supports"},{"source_id":"S095","target_id":"S020","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S038","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S006","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S004","kind":"supports"},{"source_id":"S007","target_id":"S004","kind":"supports"},{"source_id":"S006","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S009","target_id":"S023","kind":"supports"},{"source_id":"S013","target_id":"S023","kind":"supports"},{"source_id":"S021","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S007","target_id":"S024","kind":"supports"},{"source_id":"S008","target_id":"S024","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S019","kind":"supports"},{"source_id":"S011","target_id":"S028","kind":"supports"},{"source_id":"S009","target_id":"S030","kind":"supports"},{"source_id":"S007","target_id":"S030","kind":"supports"},{"source_id":"S013","target_id":"S030","kind":"supports"},{"source_id":"S009","target_id":"S031","kind":"supports"},{"source_id":"S027","target_id":"S032","kind":"supports"},{"source_id":"S007","target_id":"S032","kind":"supports"},{"source_id":"S026","target_id":"S038","kind":"supports"},{"source_id":"S004","target_id":"S038","kind":"supports"},{"source_id":"S015","target_id":"S038","kind":"supports"},{"source_id":"S021","target_id":"S038","kind":"supports"},{"source_id":"S052","target_id":"S038","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S055","target_id":"S052","kind":"supports"},{"source_id":"S056","target_id":"S052","kind":"supports"},{"source_id":"S057","target_id":"S052","kind":"supports"},{"source_id":"S058","target_id":"S052","kind":"supports"},{"source_id":"S059","target_id":"S052","kind":"supports"},{"source_id":"S018","target_id":"S053","kind":"supports"},{"source_id":"S023","target_id":"S053","kind":"supports"},{"source_id":"S089","target_id":"S053","kind":"supports"},{"source_id":"S091","target_id":"S053","kind":"supports"},{"source_id":"S016","target_id":"S053","kind":"supports"},{"source_id":"S060","target_id":"S053","kind":"supports"},{"source_id":"S002","target_id":"S053","kind":"supports"},{"source_id":"S004","target_id":"S053","kind":"supports"},{"source_id":"S014","target_id":"S053","kind":"supports"},{"source_id":"S012","target_id":"S054","kind":"supports"},{"source_id":"S013","target_id":"S054","kind":"supports"},{"source_id":"S004","target_id":"S054","kind":"supports"},{"source_id":"S060","target_id":"S054","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S019","target_id":"S055","kind":"supports"},{"source_id":"S023","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S095","target_id":"S055","kind":"supports"},{"source_id":"S045","target_id":"S056","kind":"supports"},{"source_id":"S019","target_id":"S056","kind":"supports"},{"source_id":"S053","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S057","kind":"supports"},{"source_id":"S021","target_id":"S057","kind":"supports"},{"source_id":"S022","target_id":"S057","kind":"supports"},{"source_id":"S055","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S006","target_id":"S058","kind":"supports"},{"source_id":"S055","target_id":"S059","kind":"supports"},{"source_id":"S108","target_id":"S059","kind":"supports"},{"source_id":"S010","target_id":"S039","kind":"supports"},{"source_id":"S013","target_id":"S039","kind":"supports"},{"source_id":"S106","target_id":"S039","kind":"supports"},{"source_id":"S024","target_id":"S041","kind":"supports"},{"source_id":"S060","target_id":"S041","kind":"supports"},{"source_id":"S064","target_id":"S041","kind":"supports"},{"source_id":"S065","target_id":"S041","kind":"supports"},{"source_id":"S023","target_id":"S045","kind":"supports"},{"source_id":"S024","target_id":"S045","kind":"supports"},{"source_id":"S041","target_id":"S045","kind":"supports"},{"source_id":"S013","target_id":"S045","kind":"supports"},{"source_id":"S039","target_id":"S045","kind":"supports"},{"source_id":"S089","target_id":"S045","kind":"supports"},{"source_id":"S106","target_id":"S045","kind":"supports"},{"source_id":"S104","target_id":"S060","kind":"supports"},{"source_id":"S027","target_id":"S060","kind":"supports"},{"source_id":"S028","target_id":"S067","kind":"supports"},{"source_id":"S029","target_id":"S067","kind":"supports"},{"source_id":"S013","target_id":"S067","kind":"supports"},{"source_id":"S029","target_id":"S069","kind":"supports"},{"source_id":"S030","target_id":"S069","kind":"supports"},{"source_id":"S009","target_id":"S069","kind":"supports"},{"source_id":"S031","target_id":"S069","kind":"supports"},{"source_id":"S029","target_id":"S074","kind":"supports"},{"source_id":"S013","target_id":"S074","kind":"supports"},{"source_id":"S007","target_id":"S074","kind":"supports"},{"source_id":"S074","target_id":"S073","kind":"supports"},{"source_id":"S028","target_id":"S073","kind":"supports"},{"source_id":"S007","target_id":"S073","kind":"supports"},{"source_id":"S013","target_id":"S073","kind":"supports"},{"source_id":"S067","target_id":"S064","kind":"supports"},{"source_id":"S069","target_id":"S064","kind":"supports"},{"source_id":"S011","target_id":"S064","kind":"supports"},{"source_id":"S028","target_id":"S064","kind":"supports"},{"source_id":"S030","target_id":"S064","kind":"supports"},{"source_id":"S023","target_id":"S083","kind":"supports"},{"source_id":"S009","target_id":"S083","kind":"supports"},{"source_id":"S030","target_id":"S083","kind":"supports"},{"source_id":"S083","target_id":"S064","kind":"supports"},{"source_id":"S073","target_id":"S064","kind":"supports"},{"source_id":"S013","target_id":"S064","kind":"supports"},{"source_id":"S083","target_id":"S085","kind":"supports"},{"source_id":"S032","target_id":"S085","kind":"supports"},{"source_id":"S004","target_id":"S085","kind":"supports"},{"source_id":"S007","target_id":"S085","kind":"supports"},{"source_id":"S009","target_id":"S085","kind":"supports"},{"source_id":"S083","target_id":"S087","kind":"supports"},{"source_id":"S060","target_id":"S087","kind":"supports"},{"source_id":"S002","target_id":"S087","kind":"supports"},{"source_id":"S004","target_id":"S087","kind":"supports"},{"source_id":"S032","target_id":"S087","kind":"supports"},{"source_id":"S085","target_id":"S065","kind":"supports"},{"source_id":"S087","target_id":"S065","kind":"supports"},{"source_id":"S024","target_id":"S065","kind":"supports"},{"source_id":"S032","target_id":"S065","kind":"supports"},{"source_id":"S023","target_id":"S089","kind":"supports"},{"source_id":"S009","target_id":"S089","kind":"supports"},{"source_id":"S011","target_id":"S089","kind":"supports"},{"source_id":"S013","target_id":"S089","kind":"supports"},{"source_id":"S028","target_id":"S089","kind":"supports"},{"source_id":"S026","target_id":"S091","kind":"supports"},{"source_id":"S105","target_id":"S091","kind":"supports"},{"source_id":"S023","target_id":"S091","kind":"supports"},{"source_id":"S011","target_id":"S091","kind":"supports"},{"source_id":"S009","target_id":"S091","kind":"supports"},{"source_id":"S018","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S107","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S018","target_id":"S096","kind":"supports"},{"source_id":"S034","target_id":"S096","kind":"supports"},{"source_id":"S036","target_id":"S096","kind":"supports"},{"source_id":"S108","target_id":"S096","kind":"supports"},{"source_id":"S018","target_id":"S095","kind":"supports"},{"source_id":"S035","target_id":"S095","kind":"supports"},{"source_id":"S096","target_id":"S095","kind":"supports"},{"source_id":"S036","target_id":"S095","kind":"supports"},{"source_id":"S107","target_id":"S095","kind":"supports"},{"source_id":"S106","target_id":"S095","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:sdp"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"原始半定规划","source_units":[{"raw_tex":"Consider directly applying a general-purpose interior-point method\nsolver, like SeDuMi~\\citep{sturm1999using}, SDPT3~\\citep{toh1999sdpt3},\nand MOSEK~\\citep{mosek2019}, to solve the standard-form semidefinite\nprogram to high accuracy:\n\\begin{equation}\n\\min_{X\\in\\S^{n}}\\;\\inner CX\\quad\\text{s.t.}\\;X\\succeq0,\\quad\\inner{A_{i}}X\\le b_{i}\\;\\text{for all }i\\in\\{1,2,\\dots,m\\}.\\tag{SDP}\\label{eq:sdp}\n\\end{equation}\nHere, $\\S^{n}$ denotes the set of $n\\times n$ real symmetric matrices\nwith inner product $\\inner{A_{i}}X=\\tr(A_{i}X)$, and $X\\succeq0$\nmeans that $X$ is positive semidefinite. At each iteration, the $n\\times n$\nmatrix variable $X$ is generally fully dense, even when problem data\n$C,A_{1},\\dots,A_{m}\\in\\S^{n}$ and $b_{1},\\dots,b_{m}\\in\\R$ are\nsparse. The per-iteration cost of the solver is usually at least $\\Omega(n^{3})$\ntime, which in practice limits the value of $n$ to no more than a\nfew thousand.\n","start":7013,"end":7929}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":["eq:Edef"],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"聚合图与对称重排","source_units":[{"raw_tex":"In order for chordal conversion to be fast, a well-known \\emph{necessary}\ncondition is for the underlying aggregate sparsity graph, which is\ndefined $G=(V,E)$ with\n\\begin{gather}\nV=\\{1,2,\\dots,n\\},\\quad E=\\spar(C)\\cup\\spar(A_{1})\\cup\\cdots\\cup\\spar(A_{m}),\\label{eq:Edef}\\\\\n\\text{where }\\spar(C)\\equiv\\{(i,j):C[i,j]\\ne0\\text{ for }i>j\\},\\nonumber \n\\end{gather}\nto be ``tree-like'' with a small\\emph{ treewidth} $\\tw(G)\\ll n$.\nWe defer a formal definition of the treewidth to \\defref{tw}, and\nonly note that the value of $\\omega$ is lower-bounded as $\\omega\\ge1+\\tw(G)$.\nIn other words, while $\\omega$ can be decreased by symmetrically\nreordering the columns and rows of the data matrices, as in $C\\gets\\Pi C\\Pi^{T}$\nand $A_{i}\\gets\\Pi A_{i}\\Pi^{T}$ for some permutation matrix $\\Pi$,\nactually achieving $\\omega\\ll n$ is possible only if $\\tw(G)\\ll n$.\n","start":11820,"end":12672}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["eq:Ebardef"],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"扩展聚合稀疏图","source_units":[{"raw_tex":"In this paper, we show that a \\emph{sufficient} condition for chordal\nconversion to be fast is for a supergraph $\\o G\\supseteq G$, that\nadditionally captures the correlation between constraints, to also\\footnote{Note that $\\tw(\\o G)\\ge\\tw(G)$ holds by virtue of $\\o G\\supseteq G$.}\nbe ``tree-like'' with a small treewidth $\\tw(\\o G)\\ll n$. Concretely,\nwe construct the \\emph{extended} aggregate sparsity graph $\\o G=(V,\\overline{E})$\nby forcing each constraint matrix $A_{i}$ to be its own clique in\n$G$:\n\\begin{gather}\nV=\\{1,2,\\dots,n\\},\\quad\\overline{E}=\\spar(C)\\cup\\clique(A_{1})\\cup\\cdots\\cup\\clique(A_{m})\\label{eq:Ebardef}\\\\\n\\text{where }\\clique(A)=\\{(i,j):A[i,k]\\ne0\\text{ or }A[k,j]\\ne0\\text{ for some }k\\}.\\nonumber \n\\end{gather}\n","start":12834,"end":13573}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"矩阵空间与稀疏投影","source_units":[{"raw_tex":"Write $\\R^{m\\times n}$ as the set of $m\\times n$ matrices with real\ncoefficients, with associated matrix inner product $\\inner AB=\\tr A^{T}B$\nand norm $\\|A\\|_{F}=\\sqrt{\\inner AA}$. Write $\\S^{n}\\subseteq\\R^{n\\times n}$\nas the set of $n\\times n$ real symmetric matrices, meaning that $X=X^{T}$\nholds for all $X\\in\\S^{n}$, and write $\\S_{+}^{n}\\subseteq\\S^{n}$\nas the set of symmetric positive semidefinite matrices. Write $\\R_{+}^{n}\\subseteq\\R^{n}$\nas the usual positive orthant.\n\nThe set of $n\\times n$ symmetric matrices with sparsity pattern $E$\ncan be defined as\n\\[\n\\S_{E}^{n}\\equiv\\left\\{ X\\in\\S^{n}:X[i,j]=X[j,i]=0\\quad\\text{for all }i\\ne j,\\quad(i,j)\\notin E\\right\\} .\n\\]\nConversely, the minimum sparsity pattern of a \\emph{symmetric} matrix\n$X\\in\\S^{n}$ is denoted\n\\[\n\\spar(X)\\equiv\\{(i,j):X[i,j]\\ne0,\\quad i\\ge j\\}.\n\\]\nWe also write $\\spar(M)\\equiv\\spar(M+M^{T})$ for a nonsymmetric matrix\n$M$ where there is no confusion. We write $\\proj_{E}(M)\\equiv\\arg\\min_{X\\in\\S_{E}^{n}}\\|M-X\\|_{F}$\nas the projection of $M\\in\\S^{n}$ onto the sparsity pattern $E$. \n","start":29321,"end":30385}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"团模式与顶点支持","source_units":[{"raw_tex":"\nWe define the dense sparsity pattern induced by $J\\subseteq\\{1,2,\\dots,n\\}$\nas follows\n\\[\n\\clique(J)=\\{(i,j):i,j\\in J,\\quad i\\ge j\\}.\n\\]\nWe also define the vertex support of a possibly nonsymmetric matrix\n$M$ as the following \n\\[\n\\supp(M)=\\{i:M[i,j]\\ne0\\quad\\text{for some }j\\}.\n\\]\nWe write $\\clique(A)\\equiv\\clique(\\supp(A))$ where there is no confusion.\n","start":30385,"end":30742}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"支持诱导的稀疏包含","source_units":[{"raw_tex":"This notation is motivated by the fact that $\\spar(A)\\subseteq\\clique(A)$\nfor $A\\in\\S^{n}$, and $\\spar(PDP^{T})\\subseteq\\clique(P)$ for $P\\in\\R^{n\\times d}$\nand dense $D\\in\\S^{d}$. \n\n","start":30742,"end":30925}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"等距向量化与栅格索引","source_units":[{"raw_tex":"Let $F$ be a sparsity pattern of order $n$ that \\emph{contains all\nof its diagonal elements}, as in $F\\supseteq\\{(i,i):i\\in\\{1,2,\\dots,n\\}\\}$.\nIn this case, $\\dim(\\S_{F}^{n})=|F|$ holds exactly, so we define\na \\emph{symmetric vectorization} operator $\\svec_{F}:\\S_{F}^{n}\\to\\R^{|F|}$\nto implement an isometry with the usual Euclidean space, as in \n\\[\n\\inner{\\svec_{F}(X)}{\\svec_{F}(Y)}=\\inner XY\\text{ for all }X,Y\\in{\\S_{F}^{n}}.\n\\]\nWe will explicitly require $\\svec_{F}(\\cdot)$ to be defined according\nto a column-stacking construction\n\\[\n\\svec_{F}(X)=(x_{j})_{j=1}^{n}\\quad\\text{where }x_{j}=\\left(X[i,i],\\quad\\sqrt{2}\\cdot(X[i,j]:i>j,\\quad(i,j)\\in F)\\right).\n\\]\nWe also define a companion indexing operator $\\idx_{F}(\\cdot,\\cdot)$\nto index elements of the vectorization $x=\\svec_{F}(X)$:\n\\begin{gather*}\nx[\\idx_{F}(i,i)]=X[i,i]\\quad\\text{for all }i\\in\\{1,2,\\dots,n\\},\\\\\nx[\\idx_{F}(i,j)]=\\sqrt{2}X[i,j]\\quad\\text{for all }i>j,\\quad(i,j)\\in F.\n\\end{gather*}\nAs we will see later, the correctness of our proof crucially relies\non the fact that $\\idx_{F}(\\cdot,\\cdot)$ implements a \\emph{raster\nordering} over the elements of $F$. \n","start":30925,"end":32057}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"Cholesky 分解与回代","source_units":[{"raw_tex":"To solve $Sx=b$ with $S\\succ0$ via Cholesky factorization, we first\ncompute the lower-triangular Cholesky factor $L=\\chol(S)$ according\nto the following recursive rule \n\\[\n\\chol\\left(\\begin{bmatrix}\\alpha & b^{T}\\\\\nb & D\n\\end{bmatrix}\\right)=\\begin{bmatrix}\\sqrt{\\alpha} & 0\\\\\n\\frac{1}{\\sqrt{\\alpha}}b & \\chol\\left(D-\\frac{1}{\\alpha}bb^{T}\\right)\n\\end{bmatrix},\\quad\\chol(\\alpha)=\\sqrt{\\alpha},\n\\]\nand then solve two triangular systems $Ly=b$ and $L^{T}x=y$ via\nback-substitution. If $S$ is sparse, then $L=\\chol(S)$ may also\nbe sparse. The sparsity pattern of $L$ can be directly computed from\nthe sparsity pattern of $S$, and without needing to examine the numerical\nvalues of its nonzeros.\n","start":32102,"end":32795}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":["def:symbchol"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"符号 Cholesky 闭包","source_units":[{"raw_tex":"\\begin{definition}[Symbolic Cholesky]\n\\label{def:symbchol}The \\emph{symbolic Cholesky factor} $\\chol(E)$\nof a sparsity pattern $E$ of order $n$ is defined as $\\chol(E)\\equiv E_{n+1}$\nwhere $E_{1}=E$ and\n\\[\nE_{k+1}=E_{k}\\cup(k,k)\\cup\\{(i,j):(i,k)\\in E_{k},\\quad(j,k)\\in E_{k}\\text{ for }i>j>k\\}.\n\\]\n\\end{definition}\n","start":32795,"end":33110}]},{"id":"S012","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S012"],"declared_external":true,"declaration_text":["原始先决条件：George and Liu, Theorem 5.4.4；L680–682 引用。"],"source_only":false,"title":"符号分解构造开销","source_units":[{"raw_tex":"One can verify that $\\chol(\\spar(S))=\\spar(\\chol(S))$. Note that\n$\\chol(E)$ can be computed from $E$ in $O(|\\chol(E)|)$ time and\nmemory~\\citep[Theorem 5.4.4]{george1981computer}. The efficiency\nof a sparsity-exploiting algorithm for factorizing $L=\\chol(S)$ and\nsolving $Ly=b$ and $L^{T}x=y$ is determined by the \\emph{frontsize}\nof the sparse matrix $S$. \n","start":33110,"end":33467}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["def:frontsize"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"列集合与前沿尺寸","source_units":[{"raw_tex":"\\begin{definition}[Frontsize]\n\\label{def:frontsize}The \\emph{frontsize} $\\omega(E)$ of a sparsity\npattern $E$ is defined $\\max_{j}|\\col_{F}(j)|$ where $F=\\chol(E)$\nand $\\col_{F}(j)\\equiv\\{j\\}\\cup\\{i>j:(i,j)\\in F\\}$. The frontsize\n$\\omega(S)\\equiv\\omega(\\spar(S))$ of a symmetric matrix $S$ is the\nfrontsize of its minimum sparsity pattern. \n\\end{definition}\nIntuitively, the frontsize $\\omega(S)$ is the maximum number of nonzero\nelements in a single column of the Cholesky factor $L=\\chol(S)$.\nThe following is well-known~\\citep{george1981computer}.\n","start":33467,"end":34018}]},{"id":"S014","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"置换后的稀疏模式","source_units":[{"raw_tex":"The cost of solving $Sx=b$ with sparse $S\\succ0$ can usually be\nreduced by first permuting the rows and columns of the matrix symmetrically,\nand then solving $(\\Pi S\\Pi^{T})\\Pi x=\\Pi y$ for some permutation\nmatrix $\\Pi$. For $E=\\spar(S)$, we write $E_{\\Pi}\\equiv\\spar(\\Pi S\\Pi^{T})$\nto denote its permuted sparsity pattern. It is a fundamental result\nin graph theory and linear algebra that the problem of minimizing\nthe frontsize $\\omega(E_{\\Pi})$ over the set of $n\\times n$ permutation\nmatrices $\\Pi\\in\\Perm(n)$ is the same problem as computing the treewidth\nof the graph $G=(V,E)$.\n\n","start":35337,"end":35924}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":["def:tw"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"树分解及其宽度","source_units":[{"raw_tex":"\\begin{definition}[Treewidth]\n\\label{def:tw}A tree decomposition of a graph $G=(V,E)$ is a pair\n$(\\{J_{j}\\},T)$ in which each \\emph{bag} $J_{j}\\subseteq V$ is a\nsubset of vertices and $T$ is a tree such that:\n\\begin{itemize}\n\\item (Vertex cover) $\\bigcup_{j}J_{j}=V$;\n\\item (Edge cover) $\\bigcup_{j}(J_{j}\\times J_{j})\\supseteq E$;\n\\item (Running intersection) $J_{i}\\cap J_{j}\\subseteq J_{k}$ for every\n$k$ on the path of $i$ to $j$ on $T$.\n\\end{itemize}\nThe \\emph{width} of the tree decomposition is $\\max_{j}|J_{j}|-1$.\nThe \\emph{treewidth} of $G$, denoted $\\tw(G)$, is the minimum width\nover all valid tree decompositions on $G$.\n\\end{definition}\nThe connection is an immediate corollary of the following result,\nwhich establishes an equivalence between tree decompositions and the\nsparsity pattern of Cholesky factors. \n","start":35924,"end":36749}]},{"id":"S016","semantic_type":"proposition","kind":"proposition","labels":[],"member_ids":["S016"],"declared_external":true,"declaration_text":["明确允许的 B0047；证明于 L756 延后至 George and Liu 及 Vandenberghe and Andersen。"],"source_only":false,"title":"给定树分解的消元序","source_units":[{"raw_tex":"\\begin{proposition}[Perfect elimination ordering]\nGiven a sparsity pattern $E$ of order $n$, let $G=(V,E)$ where\n$V=\\{1,2,\\dots,n\\}$. For every tree decomposition of $G$ with width\n$\\tau$, there exists a perfect elimination ordering $\\Pi\\in\\Perm(n)$\nsuch that $\\omega(E_{\\Pi})=1+\\tau$. \n\\end{proposition}\nWe defer a proof to the texts~\\citep{george1981computer,vandenberghe2015chordal},\nand only note that, given a tree decomposition of width $\\tau$, the\ncorresponding perfect elimination ordering $\\Pi$ can be found in\n$O((1+\\tau)\\cdot n)$ time. \n","start":36749,"end":37297}]},{"id":"S017","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"线性锥规划原对偶形式","source_units":[{"raw_tex":"The basic approach for solving an SDP using a general-purpose solver\nis to reformulate the problem into the primal or the dual of the standard-form\nlinear conic program \n\\[\n\\min_{x\\in{\\R^{q}}}\\{\\c^{T}x:\\A x={\\b},x\\in\\K\\}\\ge\\max_{y\\in{\\R^{p}}}\\{\\b^{T}y:\\c-\\A^{T}y\\in{\\K_{*}}\\}\n\\]\n where the data are the matrix $\\A\\in\\R^{p\\times q}$, vectors $\\b\\in\\R^{p}$\nand $\\c\\in\\R^{q}$, and the problem closed convex cone $\\K\\subseteq\\R^{q}$,\nand the notation $\\K_{*}$ means the dual cone of $\\K$. We specify\nthe following basic assumptions on this problem to ensure that it\ncan be solved in polynomial time using a self-dual embedding~\\citep{de2000self}.\nBelow, we denote $\\one_{\\K}$ as the identity element on the cone\n$\\K$, and recall that every semidefinite cone is self-dual $\\K=\\K_{*}$. \n","start":37976,"end":38757}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":["def:clp"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"标准形式的三项条件","source_units":[{"raw_tex":"\\begin{definition}[Standard-form SDP]\n\\label{def:clp}We say that the problem data $\\A\\in\\R^{p\\times q},$\n$\\b\\in\\R^{p},$ $\\c\\in\\R^{q},$ and $\\K\\subseteq\\R^{q}$ describe\nan SDP in $(n,\\omega)$-\\emph{standard form} if:\n\\begin{enumerate}\n\\item (Dimensions) The cone $\\K=\\svec(\\S_{+}^{\\omega_{1}})\\times\\cdots\\times\\svec(\\S_{+}^{\\omega_{\\ell}})$\nis the Cartesian product of semidefinite cones whose orders $\\omega_{1},\\omega_{2},\\dots,\\omega_{\\ell}$\nsatisfy \n\\[\n\\omega=\\max_{i}\\omega_{i},\\qquad n=\\sum_{i=1}^{\\ell}\\omega_{i},\\qquad{q}=\\frac{1}{2}\\sum_{i=1}^{\\ell}\\omega_{i}(\\omega_{i}+1).\n\\]\n\\item (Linear independence) $\\A^{T}y=0$ holds if and only if $y=0$. \n\\item (Strong duality is attained) There exist a choice of $x^{\\star},y^{\\star}$\nthat satisfy\n\\begin{gather*}\n\\A x^{\\star}=\\b,\\quad x^{\\star}\\in\\K,\\quad\\c-\\A^{T}y^{\\star}\\in\\K,\\quad\\c^{T}x^{\\star}=\\b^{T}y^{\\star}.\n\\end{gather*}\n\\end{enumerate}\n\\end{definition}\n%\n","start":38757,"end":39676}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":["def:ipm"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"通用内点求解器契约","source_units":[{"raw_tex":"\\begin{definition}[General-purpose solver]\n\\label{def:ipm}We say that $\\IPM$ implements a \\emph{general-purpose\nsolver} if it satisfies the following conditions\n\\begin{enumerate}\n\\item (Iteration count) Given data $(\\A,\\b,\\c,\\K)$ in $(n,\\omega)$-standard-form,\ncalling $(x,y)=\\IPM(\\epsilon,\\A,\\b,\\c,\\K)$ yields iterates $(x,y)\\in\\K\\times{\\R^{p}}$\nthat satisfy the following, in at most $O(\\sqrt{n}\\log(1/\\epsilon))$\niterations\n\\[\n\\|\\A x-\\b\\|\\le\\epsilon,\\quad{\\c-\\A^{T}y+\\epsilon\\cdot\\one_{\\K}\\in\\K,}\\quad\\c^{T}x-\\b^{T}y\\le\\epsilon\\cdot n.\n\\]\n\\item (Per-iteration costs) Each iteration costs an overhead of $O(\\omega^{2}n)$\ntime and $O(\\omega n)$ memory, plus the cost of solving $O(1)$ instances\nof the \\emph{Schur complement equation} \n\\[\n\\A\\nabla^{2}f(w)\\A^{T}\\Delta y=r,\\qquad f(w)=-\\log\\det(w)\n\\]\nby forming $\\Hess=\\A\\nabla^{2}f(w)\\A^{T}$, factoring $\\L=\\chol(\\Pi\\Hess\\Pi^{T})$\nand then solving $\\L z={\\Pi}r$ and $\\L^{T}(\\Pi\\Delta y)=z$. Here,\nthe fill-reducing permutation $\\Pi$ is required to be no worse than\nthe natural ordering, as in $\\omega(\\Pi\\Hess\\Pi^{T})\\le\\omega(\\Hess)$.\n\\end{enumerate}\n\\end{definition}\n","start":39676,"end":40796}]},{"id":"S021","semantic_type":"algorithm","kind":"algorithm","labels":["alg:myalg"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"弦转换完整算法","source_units":[{"raw_tex":"\\begin{algorithm}[t]\n\\caption{\\label{alg:myalg}Chordal conversion}\n\n\\textbf{Input.} Accuracy parameter $\\epsilon>0$, problem data $C,A_{1},A_{2},\\dots,A_{m}\\in\\S^{n}$,\n$b\\in\\R^{m}$, fill-reducing permutation $\\Pi$.\n\n\\textbf{Output.} Approximate solutions $U\\in\\R^{n\\times\\omega}$ and\n$v\\in\\R^{m}$ to the following primal-dual pair:\n\\[\n\\min_{U}\\left\\{ \\inner C{UU^{T}}:\\inner{A_{i}}{UU^{T}}\\le b_{i}\\text{ for all }i\\right\\} \\ge\\max_{v\\le0}\\left\\{ \\inner bv:{\\textstyle \\sum_{i}}v_{i}A_{i}\\preceq C\\right\\} .\n\\]\n\n\\textbf{Algorithm.}\n\\begin{enumerate}\n\\item (Symbolic factorization) Pre-order all data matrices $\\tilde{A}_{i}=\\Pi A_{i}\\Pi^{T}$\nand $\\tilde{C}=\\Pi C\\Pi^{T}$. Compute the permuted aggregate sparsity\npattern $E=\\spar(\\tilde{C})\\cup\\bigcup_{i}\\spar(\\tilde{A}_{i})$,\nits lower-triangular symbolic Cholesky factor $F=\\chol(E)$, and define\nthe following\n\\[\nJ_{j}=\\col_{F}(j)\\equiv\\{j\\}\\cup\\{i>j:(i,j)\\in F\\},\\quad\\omega_{j}\\equiv|J_{j}|,\\quad\\omega\\equiv\\max_{j}\\omega_{j}.\n\\]\n\\item (Numerical solution) Call $(x,y)=\\IPM(\\epsilon,\\A,\\b,\\c,\\K)$ where\n$\\IPM$ is a general-purpose solver (\\defref{ipm}), and the problem\ndata $\\A,\\b,\\c,\\K$ implement the following\n\\[\nY^{\\star}=\\arg\\max_{Y\\in\\S_{F}^{n}}\\inner{-\\tilde{C}}Y\\text{ s.t. }\\begin{array}{cc}\nb_{i}-\\inner{\\tilde{A}_{i}}Y\\ge0 & \\quad\\text{for all }i\\in\\{1,2,\\dots,m\\}\\\\\nY[J_{j},J_{j}]\\succeq0 & \\quad\\text{for all }j\\in\\{1,2,\\dots,n\\}\n\\end{array}\n\\]\nas an instance of $y^{\\star}=\\arg\\max_{y}\\{\\b^{T}y:\\c-\\A^{T}y\\in\\K\\}$\nwith $y^{\\star}=\\svec_{F}(Y^{\\star})$.\n\\item (Back substitution) Recover $Y\\in\\S_{F}^{n}$ from $y=\\svec_{F}(Y)$,\nand compute $\\delta=-\\min_{j}\\{0,\\lambda_{\\min}(Y[J_{j},J_{j}])\\}$.\nSolve the positive semidefinite matrix completion\n\\[\n\\text{find }\\tilde{U}\\in\\R^{n\\times\\omega}\\text{ such that }(\\tilde{U}\\tilde{U}^{T})[J_{j},J_{j}]=Y[J_{j},J_{j}]+\\delta I\\text{ for all }j\n\\]\nusing \\citep[Algorithm~2]{sun2015decomposition}. Output $v=-(x_{i})_{i=1}^{m}$\nand $U=\\Pi^{T}\\tilde{U}$.\n\\end{enumerate}\n\\end{algorithm}\n","start":41250,"end":43246}]},{"id":"S022","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S022"],"declared_external":true,"declaration_text":["原始先决条件：Dancis, Theorem 1.5；Sun, Algorithm 2。算法引用位于 L873–877，成本说明位于 L888–890。"],"source_only":false,"title":"低秩弦补全及恢复成本","source_units":[{"raw_tex":" \\algref{myalg} summarizes the standard implementation of chordal\nconversion, which is known as the ``d-space conversion method using\nbasis representation'' in \\citet{kim2011exploiting}. Our only modification\nis to recover $X=UU^{T}$ from $Y=\\proj_{F}(X)$ in Step 3 using the\n\\emph{low-rank} chordal completion~\\citep[Theorem~1.5]{dancis1992positive},\ninstead of the \\emph{maximum determinant} chordal completion~\\citep[Theorem~2]{grone1984positive}\nas originally proposed by \\citet{fukuda2001exploiting}. We note that\nboth recovery procedures have the same complexity of $O(\\omega^{3}n)$\ntime and $O(\\omega^{2}n)$ memory, but the former puts $X$ in a more\nconvenient form~\\citep{sun2015decomposition}.\n","start":43246,"end":43949}]},{"id":"S023","semantic_type":"definition","kind":"definition","labels":["eq:AKdef-1","eq:AKdef-2"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"转换后的锥数据与提取矩阵","source_units":[{"raw_tex":"To state the Schur complement sparsity $E^{(2)}$ explicitly, note\nthat the problem data $(\\A,\\b,\\c,\\K)$ in Step 2 of \\algref{myalg}\nare \\begin{subequations}\\label{eq:AKdef}\n\\begin{equation}\n\\begin{array}{c}\n\\A=[\\svec_{F}(A_{1}),\\dots,\\svec_{F}(A_{m}),-\\P_{1},\\dots,-\\P_{n}],\\\\\n\\b=-\\svec_{F}(C),\\qquad\\c=(b,0),\\\\\n\\K=\\R_{+}^{m}\\times\\svec(\\S_{+}^{\\omega_{1}})\\times\\svec(\\S_{+}^{\\omega_{2}})\\times\\cdots\\times\\svec(\\S_{+}^{\\omega_{n}})\n\\end{array}\\label{eq:AKdef-1}\n\\end{equation}\nwhere $\\omega_{j}\\equiv|\\col_{F}(j)|$ and each $\\P_{j}$ is implicitly\ndefined to satisfy \n\\begin{equation}\n\\P_{j}^{T}\\svec_{F}(Y)=\\svec(Y[\\col_{F}(j),\\col_{F}(j)])\\quad\\text{for all }Y\\in\\S_{F}^{n}.\\label{eq:AKdef-2}\n\\end{equation}\n\\end{subequations}The resulting Schur complement matrix reads\n","start":44543,"end":45316}]},{"id":"S024","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:E2def","eq:Hessdef"],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"Schur 矩阵的结构稀疏式","source_units":[{"raw_tex":"\\begin{gather}\n\\A\\nabla^{2}f(w)\\A^{T}=\\sum_{i=1}^{m}d_{i}\\svec_{F}(A_{i})\\svec_{F}(A_{i})^{T}+\\sum_{j=1}^{n}\\P_{j}\\D_{j}\\P_{j}^{T}\\label{eq:Hessdef}\\\\\n\\text{where }d_{i}=w_{i}^{-2},\\qquad\\D_{j}\\svec(X_{j})=\\svec(W_{j}^{-1}X_{j}W_{j}^{-1})\\nonumber \n\\end{gather}\nand $w=(w_{1},\\dots,w_{m},\\svec(W_{1}),\\dots,\\svec(W_{n}))\\in\\interior(\\K)$\nis a scaling point. The associated sparsity pattern, aggregated over\nall possible choices of scaling $w$, is as follows\n\\begin{align}\nE^{(2)} & =\\{(i,j):(\\A\\nabla^{2}f(w)\\A^{T})[i,j]\\ne0\\text{ for some }w\\in\\interior(\\K)\\}\\nonumber \\\\\n & =\\left(\\bigcup_{i=1}^{m}\\spar(a_{i}a_{i}^{T})\\right)\\cup\\left(\\bigcup_{j=1}^{n}\\spar(\\P_{j}\\D_{j}\\P_{j}^{T})\\right)\\nonumber \\\\\n & =\\left(\\bigcup_{i=1}^{m}\\clique(\\supp(a_{i}))\\right)\\cup\\left(\\bigcup_{j=1}^{n}\\clique(\\supp(\\P_{j}))\\right)\\label{eq:E2def}\n\\end{align}\nwhere we have written $a_{i}=\\svec_{F}(A_{i})$. The result below\nsays that if $\\omega(\\o E)=O(1)$, then $\\omega(\\o E^{(2)})=O(1)$. \n","start":45316,"end":46292}]},{"id":"S025","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"转换规模的记号说明","source_units":[{"raw_tex":"Let us now give an end-to-end complexity guarantee for \\algref{myalg}.\nWe will need the following assumption to ensure that the data $(\\A,\\b,\\c,\\K)$\npreviously defined in (\\ref{eq:AKdef}) specifies an SDP in $(N,\\omega)$-standard\nform, where $N=m+\\omega n$ and $\\omega\\equiv\\omega(E)$. \n\n","start":50696,"end":50984}]},{"id":"S026","semantic_type":"assumption","kind":"assumption","labels":["asm:strict"],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"原问题强对偶达到","source_units":[{"raw_tex":"\\begin{assume}[Strong duality is attained]\\label{asm:strict}There\nexists a primal-dual pair $X^{\\star}\\succeq0$ and $v^{\\star}\\le0$\nthat are feasible $\\inner{A_{i}}{X^{\\star}}\\le b_{i}$ for all $i$\nand $\\sum_{i}v_{i}^{\\star}A_{i}\\preceq C$ and coincide in their objectives\n$\\inner C{X^{\\star}}=\\inner b{v^{\\star}}$.\n\n\\end{assume}\n","start":50984,"end":51314}]},{"id":"S027","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"保序诱导子模式","source_units":[{"raw_tex":"$E$ of order $n$ can be viewed as the edge set of an undirected\ngraph $G=(V,E)$ on vertices $V=\\{1,2,\\dots,n\\}$. The underlying\nprinciple behind our proof is the fact that the frontsize is monotone\nunder the subgraph relation: if $G'=(V',E')$ is a subgraph of $G=(V,E)$,\nthen $\\omega(E)\\ge\\omega(E')$. \n\nTo state this formally, we denote the sparsity pattern induced by\na subset of vertices $U=\\{u_{1},u_{2},\\dots,u_{p}\\}\\subseteq V$ as\nfollows\n\\[\nE[U]\\equiv\\{(i,j):(u_{i},u_{j})\\in E\\}\\text{ where }u_{1}<u_{2}<\\cdots<u_{p}.\n\\]\nNote that we always sort the elements of $U$. Our definition is made\nso that if $E=\\spar(X)$, then $E[U]=\\spar(X[U,U])$, without any\nreordering of the rows and columns. \n","start":57148,"end":57847}]},{"id":"S028","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"零填充模式","source_units":[{"raw_tex":"Our proof of \\lemref{quadlift} is based on a connection between\nzero-fill sparsity patterns, for which sequential Gaussian elimination\nresults in no additionally fill-in, and a ``sorted'' extension of\nthe running intersection property. \n\\begin{definition}[ZF]\nThe sparsity pattern $F$ is said to be\\emph{ zero-fill} if $F=\\chol(F)$.\n\\end{definition}\nEquivalently, if $F$ is zero-fill, then $(i,k)\\in F$ and $(j,k)\\in F$\nimplies $(i,j)\\in F$ for $i>j>k$ via the definition of the symbolic\nCholesky factor (\\defref{symbchol}). \n\n","start":61164,"end":61691}]},{"id":"S029","semantic_type":"definition","kind":"definition","labels":["def:rip"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"有序运行交性质","source_units":[{"raw_tex":"\\begin{definition}[RIP]\n\\label{def:rip}The sequence of subsets $J_{1},J_{2},\\dots,J_{\\ell}$\nwith $J_{j}\\subset\\N$ satisfies the \\emph{sorted running intersection\nproperty} if there exists a parent pointer $\\p:\\{1,2,\\dots,\\ell-1\\}\\to\\{2,3,\\dots,\\ell\\}$\nsuch that the following holds for all $1\\le j<\\ell$:\n\\begin{gather*}\n\\p(j)>j,\\quad J_{\\p(j)}\\supseteq J_{j}\\cap(J_{j+1}\\cup J_{j+2}\\cup\\cdots\\cup J_{\\ell}),\\quad\\min\\{J_{\\p(j)}\\}>\\max\\{J_{j}\\backslash J_{\\p(j)}\\}.\n\\end{gather*}\n\\end{definition}\nThe symbolic Cholesky factor $F=\\chol(E)$ for a sparsity pattern\n$E$ is the canonical example of a zero-fill sparsity pattern. In\nturn, the corresponding column sets $J_{j}=\\col_{F}(j)$ are the canonical\nexample of sequence of subsets that satisfy the sorted version of\nthe running intersection property.\n","start":61691,"end":62493}]},{"id":"S030","semantic_type":"definition","kind":"definition","labels":["eq:Jk2_def"],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"二次提升索引集合","source_units":[{"raw_tex":"Our proof is based on the fact that the ``lifted'' sparsity pattern\n$F^{(2)}$ can be constructed as $F^{(2)}=\\bigcup_{k=1}^{n}\\clique(J_{k}^{(2)})$\nwith respect to the following ``lifted'' index sets\n\\begin{gather}\nJ_{k}^{(2)}\\equiv\\idx_{F}(\\clique(J_{k}))=\\{\\idx_{F}(i,j):i,j\\in J_{k},\\quad i\\ge j\\}.\\label{eq:Jk2_def}\n\\end{gather}\nWe need to show that, if the original index sets $J_{1},J_{2},\\dots,J_{k}$\nsatisfy the running intersection property, then the lifted index sets\n$J_{1}^{(2)},J_{2}^{(2)},\\dots,J_{k}^{(2)}$ will inherit the running\nintersection property. Our key insight is that the index operator\n$\\idx_{F}$ implements a \\emph{raster ordering}.\n","start":63638,"end":64299}]},{"id":"S031","semantic_type":"lemma","kind":"lemma","labels":[],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"列优先栅格次序","source_units":[{"raw_tex":"\\begin{lemma}[Raster ordering]\nThe ordering $\\idx_{F}:F\\to\\N$ satisfies the following, for all\n$(i,j)\\in F$ with $i\\ge j$ and $(i',j')\\in F$ with $i'\\ge j'$:\n\\begin{itemize}\n\\item If $j>j'$, then $\\idx(i,j)>\\idx(i',j')$ holds. \n\\item If $j=j'$ and $i>i'$, then $\\idx(i,j)>\\idx(i',j')$ holds. \n\\end{itemize}\n\\end{lemma}\n","start":64299,"end":64618}]},{"id":"S032","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"集合的保序限制","source_units":[{"raw_tex":"We will need some additional notation. For $V=\\{1,2,\\dots,n\\}$ and\n$J\\subseteq V$, we denote the subset of $J$ induced by the elements\nin $U=\\{u_{1},u_{2},\\dots,u_{p}\\}\\subseteq V$ as follows\n\\[\nJ[U]\\equiv\\{i:u_{i}\\in J\\}\\text{ where }u_{1}<u_{2}<\\cdots<u_{p}.\n\\]\nOur definition is made so that if $E=\\clique(J)$, then $E[U]=\\clique(J[U])$.\n","start":72588,"end":72929}]},{"id":"S033","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"附录正则性验证范围","source_units":[{"raw_tex":"Given data $C,A_{1},\\dots,A_{m}\\in\\S^{n}$ and $b\\in\\R^{m}$, define\n$(\\A,\\b,\\c,\\K)$ as in (\\ref{eq:AKdef}). In this section, we verify\nthat $(\\A,\\b,\\c,\\K)$ specifies an SDP that satisfies the regularity\nassumptions in \\defref{clp}. \n","start":99088,"end":99320}]},{"id":"S034","semantic_type":"algorithm","kind":"algorithm","labels":["eq:hsdcone","eq:hsdfeas","eq:hsdinit"],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"自对偶嵌入及中心初始化","source_units":[{"raw_tex":"We prove \\propref{general} by using the short-step method of Nesterov\nand Todd~\\citep[Algorithm~6.1]{nesterov1998primal} to solve the\nextended homogeneous self-dual embedding\\begin{subequations}\\label{eq:hsd}\n\\begin{alignat}{2}\n & \\min_{x,y,s,\\kappa,\\tau,\\theta}\\quad & (n+1)\\theta\\\\\n & \\text{s.t.} & \\begin{bmatrix}0 & +\\A^{T} & -\\c & -r_{d}\\\\\n-\\A & 0 & +\\b & -r_{p}\\\\\n+\\c^{T} & -\\b^{T} & 0 & -r_{c}\\\\\nr_{d}^{T} & r_{p}^{T} & r_{c} & 0\n\\end{bmatrix}\\begin{bmatrix}x\\\\\ny\\\\\n\\tau\\\\\n\\theta\n\\end{bmatrix}+\\begin{bmatrix}s\\\\\n0\\\\\n\\kappa\\\\\n0\n\\end{bmatrix} & =\\begin{bmatrix}0\\\\\n0\\\\\n0\\\\\nn+1\n\\end{bmatrix}\\label{eq:hsdfeas}\\\\\n &  & x,s\\in\\K,\\quad\\kappa,\\tau & \\ge0,\\label{eq:hsdcone}\n\\end{alignat}\n\\end{subequations}where $r_{d}=\\one_{\\K}-\\c$ and $r_{p}=-\\A\\one_{\\K}+\\b$\nand $r_{c}=1+\\c^{T}\\one_{\\K}$. Beginning at the strictly feasible,\nperfectly centered point in (\\ref{eq:hsdinit}) for $\\mu=1$:\n\\begin{gather}\n\\theta^{(0)}=\\tau^{(0)}=\\kappa^{(0)}=1,\\quad y^{(0)}=0,\\quad x^{(0)}=s^{(0)}=\\one_{\\K}.\\label{eq:hsdinit}\n\\end{gather}\n","start":102960,"end":103983}]},{"id":"S035","semantic_type":"algorithm","kind":"algorithm","labels":[],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"短步更新与缩放方向","source_units":[{"raw_tex":" we take the following steps\\begin{subequations}\\label{eq:hsd_ipm}\n\\begin{align}\n\\mu^{+} & =\\left(1-\\frac{1}{15\\sqrt{n+1}}\\right)\\cdot\\frac{x^{T}s+\\tau\\kappa}{n+1},\\\\\n(x^{+},y^{+},s^{+},\\tau^{+},\\theta^{+},\\kappa^{+}) & =(x,y,s,\\tau,\\theta,\\kappa)+(\\Delta x,\\Delta y,\\Delta s,\\Delta\\tau,\\Delta\\theta,\\Delta\\kappa).\\nonumber \n\\end{align}\n\\end{subequations}along the search direction defined by the linear\nsystem~\\citep[Eqn. 9]{todd1998nesterov} \\begin{subequations}\\label{eq:hsd_nt}\n\\begin{align}\n\\begin{bmatrix}0 & +\\A^{T} & -\\c & -r_{p}\\\\\n-\\A & 0 & +\\b & -r_{d}\\\\\n+\\c^{T} & -\\b^{T} & 0 & -r_{c}\\\\\n+r_{p}^{T} & +r_{d}^{T} & +r_{c} & 0\n\\end{bmatrix}\\begin{bmatrix}\\Delta x\\\\\n\\Delta y\\\\\n\\Delta\\tau\\\\\n\\Delta\\theta\n\\end{bmatrix}+\\begin{bmatrix}\\Delta s\\\\\n0\\\\\n\\Delta\\kappa\\\\\n0\n\\end{bmatrix} & =0,\\\\\ns+\\Delta s+\\nabla^{2}f(w)\\Delta x+\\mu^{+}\\nabla f(x) & =0,\\\\\n\\kappa+\\Delta\\kappa+(\\kappa/\\tau)\\Delta\\tau-\\mu^{+}\\tau^{-1} & =0.\n\\end{align}\n\\end{subequations}where $f(w)=-\\log\\det w$ is the usual self-concordant\nbarrier function, and $w\\in\\interior(\\K)$ is the unique point that\nsatisfies $\\nabla^{2}f(w)x=s$. The following is an immediate consequence\nof~\\citep[Theorem~6.4]{nesterov1998primal}.\n","start":103983,"end":105173}]},{"id":"S036","semantic_type":"lemma","kind":"lemma","labels":["eq:short-step","lem:short-step"],"member_ids":["S036"],"declared_external":true,"declaration_text":["明确允许的 B0148；作者于 L1974–1975 归于 Nesterov and Todd, Theorem 6.4。"],"source_only":false,"title":"短步迭代收敛保证","source_units":[{"raw_tex":"\\begin{lemma}[Short-Step Method]\n\\label{lem:short-step}The sequence in (\\ref{eq:hsd_ipm}) arrives\nat an iterate $(x,y,s,\\tau,\\theta,\\kappa)$ satisfying\n\\begin{equation}\n\\frac{x^{T}s+\\tau\\kappa}{n+1}\\le\\epsilon,\\qquad\\tau\\kappa\\ge\\gamma\\epsilon\\label{eq:short-step}\n\\end{equation}\nwith $\\gamma=\\frac{9}{10}$ in at most $O(\\sqrt{n}\\log(1/\\epsilon))$\niterations. \n\\end{lemma}\n","start":105173,"end":105546}]},{"id":"S038","semantic_type":"theorem","kind":"theorem","labels":["thm:total"],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"弦转换复杂度上界","source_units":[{"raw_tex":"\\begin{theorem}[Upper complexity]\n\\label{thm:total}Let the data $C,A_{1},\\dots,A_{m}\\in\\S^{n}$ and\n$b\\in\\R^{m}$ satisfy \\asmref{strict}. Given a tree decomposition\nof width $\\o{\\tau}$ for the extended aggregate sparsity graph $\\o G=(V,\\o E)$,\nwhere\n\\[\nV=\\{1,2,\\dots,n\\},\\qquad\\o E=\\spar(C)\\cup\\clique(A_{1})\\cup\\cdots\\cup\\clique(A_{m}),\n\\]\nset $\\Pi$ as the associated perfect elimination ordering. Then, \\algref{myalg}\noutputs $U\\in\\R^{n\\times\\o{\\omega}}$ and $v\\in\\R^{m}$ with $v\\le0$\nsuch that\n\\begin{align*}\n\\inner{A_{i}}{UU^{T}}-b_{i} & \\le\\epsilon\\text{ for all }i, & \\sum_{i=1}^{m}v_{i}A_{i}-C & \\preceq\\epsilon\\cdot I, & \\inner C{UU^{T}}-\\inner bv & \\le\\epsilon\\cdot N,\n\\end{align*}\nin $O(\\sqrt{N}\\log(1/\\epsilon))$ iterations, with per-iteration costs\nof $O(\\o{\\omega}^{4}\\cdot N)$ time and $O(\\o{\\omega}^{2}\\cdot N)$\nmemory, where $N=m+\\o{\\omega}\\cdot n$ and $\\o{\\omega}=1+\\o{\\tau}$.\n\\end{theorem}","start":51314,"end":52220}]},{"id":"S039","semantic_type":"proposition","kind":"proposition","labels":["prop:chol"],"member_ids":["S039","S040"],"declared_external":false,"source_only":false,"title":"稀疏 Cholesky 成本界","source_units":[{"raw_tex":"\\begin{proposition}[Sparse Cholesky factorization]\n\\label{prop:chol}Given $S\\in\\S^{n},$ $S\\succ0$, let $\\omega\\equiv\\omega(S)$.\nSparse Cholesky factorization factors $L=\\chol(S)$ in $\\Tim$ arithmetic\noperations and $M$ units of memory, where \n\\begin{align*}\n\\frac{1}{6}(\\omega-1)^{3}+n & \\le\\Tim\\le\\omega^{2}\\cdot n, & \\frac{1}{2}(\\omega-1)^{2}+n & \\le\\Mem\\le\\omega\\cdot n.\n\\end{align*}\n\\end{proposition}","start":34018,"end":34422},{"raw_tex":"\\begin{proof}\nLet $\\omega_{j}\\equiv|\\col_{F}(j)|$. By inspection, $\\Tim=\\sum_{j=1}^{n}\\frac{1}{2}\\omega_{j}(\\omega_{j}+1)$\nand $\\Mem=\\sum_{j=1}^{n}\\omega_{j}$. The bounds follow by substituting\n$\\omega\\ge\\omega_{j+1}\\ge\\omega_{j}-1$. Indeed,\n\\[\n\\Mem=\\sum_{j=1}^{n}\\omega_{j}\\ge\\underbrace{\\omega+(\\omega-1)+\\cdots+1}_{\\omega\\text{ terms}}+\\underbrace{1+\\cdots+1}_{n-\\omega\\text{ terms}}=\\frac{1}{2}\\omega(\\omega+1)+(n-\\omega)\n\\]\nand similarly $\\Tim=\\sum_{j=1}^{n}\\frac{1}{2}\\omega_{j}(\\omega_{j}+1)\\ge\\frac{1}{6}\\omega(\\omega+1)(\\omega+2)+(n-\\omega).$\n\\qed\n\\end{proof}","start":34423,"end":34991}]},{"id":"S041","semantic_type":"theorem","kind":"theorem","labels":["thm:front"],"member_ids":["S041","S042","S043","S044"],"declared_external":false,"source_only":false,"title":"Schur 稀疏前沿界","source_units":[{"raw_tex":"\\begin{theorem}[Frontsize of Schur complement sparsity]\n\\label{thm:front}Given $C,A_{1},A_{2},\\dots,A_{m}\\in\\S^{n}$, define\n$\\A,\\K$ as in (\\ref{eq:AKdef}), and define $E^{(2)}$ as in (\\ref{eq:E2def}).\nWe have \n\\[\n\\frac{1}{2}\\omega(\\omega+1)\\le\\omega(E^{(2)})\\le\\frac{1}{2}\\o{\\omega}(\\o{\\omega}+1)\n\\]\nwhere $\\omega\\equiv\\omega(E)$ and $\\o{\\omega}=\\omega(\\o E)$ and\n$E,\\o E$ are defined in (\\ref{eq:Edef}) and (\\ref{eq:Ebardef}).\nMoreover, if $E=\\o E$, then we also have $\\chol(E^{(2)})=E^{(2)}$. \n\\end{theorem}","start":46293,"end":46802},{"raw_tex":"Observe that $E^{(2)}\\supseteq F^{(2)}=\\bigcup_{k=1}^{n}\\clique(\\P_{k})$\nvia (\\ref{eq:E2def}), so it follows immediately from \\propref{del}\n\\[\n\\omega(E^{(2)})\\ge\\omega(F^{(2)})=\\frac{1}{2}\\omega(E)[\\omega(E)+1],\n\\]\nwhich is precisely the lower-bound in \\thmref{front}. For the upper-bound,\nwe will use $\\o F=\\chol(\\o E)$, the symbolic Cholesky factor of the\nextended aggregate sparsity pattern $\\o E$, to construct a similarly\nlifted $\\o F^{(2)}$. Our key insight is that $E^{(2)}$ can be obtained\nfrom $\\o F^{(2)}$ via vertex and edge deletions.\n","start":59531,"end":60078},{"raw_tex":"Substituting $E^{(2)}\\subseteq\\o F^{(2)}[V^{(2)}]$ with the exact\nfrontsize of $\\omega(\\o F^{(2)})$ from \\lemref{quadlift} yields\n\\[\n\\omega(E^{(2)})\\le\\omega(\\o F^{(2)})=\\frac{1}{2}\\omega(\\o E)[\\omega(\\o E)+1],\n\\]\nwhich is precisely the upper-bound in \\thmref{front}. Finally, if\n$E=\\o E$, then $F^{(2)}=\\o F^{(2)}=E^{(2)}$, so $\\chol(E^{(2)})=E^{(2)}$\nvia \\lemref{quadlift}.\n","start":60615,"end":60991}]},{"id":"S045","semantic_type":"corollary","kind":"corollary","labels":["cor:normal"],"member_ids":["S045","S046","S047","S048","S049","S050","S051"],"declared_external":false,"source_only":false,"title":"Schur 方程求解成本","source_units":[{"raw_tex":"\\begin{corollary}[Cost of Schur complement equation]\n\\label{cor:normal}Given the data matrix $\\A$, scaling point $w\\in\\interior(\\K)$,\nand right-hand side $g$, define $\\Hess=\\A\\nabla^{2}f(w)\\A^{T}$ as\nin (\\ref{eq:E2def}). Suppose that all columns of $\\A$ are nonzero,\nand all scaling matrices $\\D_{j}$ are fully dense. Then, it takes\n$\\Tim$ arithmetic operations and $\\Mem$ units of storage to form\n$\\Hess$ and solve $\\Hess\\Delta y=g$, where\n\\begin{gather*}\n\\frac{1}{48}(\\omega-1)^{6}+m+n\\le\\Tim\\le4\\o{\\omega}^{4}\\cdot(m+\\omega n),\\\\\n\\frac{1}{8}(\\omega-1)^{4}+m+n\\le\\Mem\\le2\\o{\\omega}^{2}\\cdot(m+\\omega n),\n\\end{gather*}\nin which $\\omega\\equiv\\omega(E)$ and $\\o{\\omega}\\equiv\\omega(\\o E)$\nsatisfy $1\\le\\omega\\le\\o{\\omega}$.\n\\end{corollary}","start":47617,"end":48355},{"raw_tex":"\\begin{proof}\nLet $a_{i}\\equiv\\svec_{F}(A_{i})$ and $\\omega_{j}\\equiv|\\col_{F}(j)|$.\nWe break the solution of $\\Hess\\Delta y=g$ into five steps:\n\\begin{enumerate}\n\\item (Input) It takes $\\Mem_{\\mathrm{data}}$ memory to state the problem\ndata, where $m+n\\le\\Mem_{\\mathrm{data}}\\le2\\o{\\omega}^{2}m+3\\omega^{2}n$.\nIndeed, $m+n\\le\\nnz(\\A)\\le\\o{\\omega}^{2}m+\\omega^{2}n$ because $1\\le\\nnz(a_{i})\\le\\o{\\omega}^{2}$\n(see Step 2 below) and $\\nnz(\\P_{j})=\\frac{1}{2}\\omega_{j}(\\omega_{j}+1)$.\nAlso, $m+n\\le\\nnz(w)\\le m+\\omega^{2}n$, and $n\\le\\nnz(g)\\le\\omega n$.\n\\item (Build LP part) It takes $\\Tim_{\\mathrm{LP}}$ time to build $\\sum_{i=1}^{m}d_{i}a_{i}a_{i}^{T}$,\nwhere $m\\le\\Tim_{\\mathrm{LP}}\\le2\\o{\\omega}^{4}m$ time and memory.\nThis follows from $1\\le\\nnz(a_{i})\\le\\o{\\omega}^{2}$, where the upper-bound\nis because $\\spar(a_{i}a_{i}^{T})\\subseteq E^{(2)}\\subseteq\\chol(E^{(2)})$,\nand that $\\omega(E^{(2)})$ is, by definition, the maximum number\nof nonzero elements in a single column of $\\chol(E^{(2)})$.\n","start":48356,"end":49357},{"raw_tex":"\\begin{proof}\nLet $a_{i}\\equiv\\svec_{F}(A_{i})$ and $\\omega_{j}\\equiv|\\col_{F}(j)|$.\nWe break the solution of $\\Hess\\Delta y=g$ into five steps:\n\\begin{enumerate}\n\\item (Input) It takes $\\Mem_{\\mathrm{data}}$ memory to state the problem\ndata, where $m+n\\le\\Mem_{\\mathrm{data}}\\le2\\o{\\omega}^{2}m+3\\omega^{2}n$.\nIndeed, $m+n\\le\\nnz(\\A)\\le\\o{\\omega}^{2}m+\\omega^{2}n$ because $1\\le\\nnz(a_{i})\\le\\o{\\omega}^{2}$\n(see Step 2 below) and $\\nnz(\\P_{j})=\\frac{1}{2}\\omega_{j}(\\omega_{j}+1)$.\nAlso, $m+n\\le\\nnz(w)\\le m+\\omega^{2}n$, and $n\\le\\nnz(g)\\le\\omega n$.\n\\item (Build LP part) It takes $\\Tim_{\\mathrm{LP}}$ time to build $\\sum_{i=1}^{m}d_{i}a_{i}a_{i}^{T}$,\nwhere $m\\le\\Tim_{\\mathrm{LP}}\\le2\\o{\\omega}^{4}m$ time and memory.\nThis follows from $1\\le\\nnz(a_{i})\\le\\o{\\omega}^{2}$, where the upper-bound\nis because $\\spar(a_{i}a_{i}^{T})\\subseteq E^{(2)}\\subseteq\\chol(E^{(2)})$,\nand that $\\omega(E^{(2)})$ is, by definition, the maximum number\nof nonzero elements in a single column of $\\chol(E^{(2)})$.\n\\item (Build SDP part) It takes $\\Tim_{\\mathrm{SDP}}$ time to build $\\sum_{j=1}^{n}\\P_{j}\\D_{j}\\P_{j}^{T}$,\nwhere $n\\le\\Tim_{\\mathrm{SDP}}\\le\\omega^{4}n$ time and memory. This\nfollows from $\\nnz(\\P_{j})=\\frac{1}{2}\\omega_{j}(\\omega_{j}+1)$,\nwhich implies $\\nnz(\\P_{j}\\D_{j}\\P_{j}^{T})=\\frac{1}{4}\\omega_{j}^{2}(\\omega_{j}+1)^{2}$\nfor a fully-dense $\\D_{j}\\succ0$. \n\\item (Factorization) It takes $\\Tim_{\\mathrm{fact}}$ time and $\\Mem_{\\mathrm{fact}}$\nmemory to factor $\\L=\\chol(\\Hess)$, where $\\frac{1}{48}(\\omega-1)^{6}+n\\le\\Tim_{\\mathrm{fact}}\\le\\o{\\omega}^{5}n$\nand $\\frac{1}{8}(\\omega-1)^{4}+n\\le\\Mem_{\\mathrm{fact}}\\le\\o{\\omega}^{3}n$.\nThe matrix $\\Hess$ has $|F|$ columns and rows, and frontsize $\\omega(E^{(2)})$.\nThe desired figures follow by substituting $\\frac{1}{2}\\omega^{2}\\le\\omega(E^{(2)})\\le\\o{\\omega}^{2}$\nand $n\\le|F|\\le\\omega n$ into \\propref{chol}.\n\\item (Back-substitution) It takes $\\Mem_{\\mathrm{fact}}$ time and memory\nto solve each of $\\L z=g$ and $\\L^{T}\\Delta y=z$ via triangular\nback-substitution. \n\\end{enumerate}\nThe overall runtime is cumulative, so $\\Tim=\\Tim_{\\mathrm{LP}}+\\Tim_{\\mathrm{SDP}}+\\Tim_{\\mathrm{fact}}+2\\Mem_{\\mathrm{fact}}$.\nThe overall memory use is $\\Mem=\\Mem_{\\mathrm{data}}+\\Mem_{\\mathrm{fact}}$,\nbecause the matrix $\\Hess$ can be constructed and then factored in-place.\n\\qed\n\\end{proof}","start":48356,"end":50694},{"raw_tex":"\\item (Build SDP part) It takes $\\Tim_{\\mathrm{SDP}}$ time to build $\\sum_{j=1}^{n}\\P_{j}\\D_{j}\\P_{j}^{T}$,\nwhere $n\\le\\Tim_{\\mathrm{SDP}}\\le\\omega^{4}n$ time and memory. This\nfollows from $\\nnz(\\P_{j})=\\frac{1}{2}\\omega_{j}(\\omega_{j}+1)$,\nwhich implies $\\nnz(\\P_{j}\\D_{j}\\P_{j}^{T})=\\frac{1}{4}\\omega_{j}^{2}(\\omega_{j}+1)^{2}$\nfor a fully-dense $\\D_{j}\\succ0$. \n","start":49357,"end":49722},{"raw_tex":"\\item (Factorization) It takes $\\Tim_{\\mathrm{fact}}$ time and $\\Mem_{\\mathrm{fact}}$\nmemory to factor $\\L=\\chol(\\Hess)$, where $\\frac{1}{48}(\\omega-1)^{6}+n\\le\\Tim_{\\mathrm{fact}}\\le\\o{\\omega}^{5}n$\nand $\\frac{1}{8}(\\omega-1)^{4}+n\\le\\Mem_{\\mathrm{fact}}\\le\\o{\\omega}^{3}n$.\nThe matrix $\\Hess$ has $|F|$ columns and rows, and frontsize $\\omega(E^{(2)})$.\nThe desired figures follow by substituting $\\frac{1}{2}\\omega^{2}\\le\\omega(E^{(2)})\\le\\o{\\omega}^{2}$\nand $n\\le|F|\\le\\omega n$ into \\propref{chol}.\n","start":49722,"end":50226},{"raw_tex":"\\item (Back-substitution) It takes $\\Mem_{\\mathrm{fact}}$ time and memory\nto solve each of $\\L z=g$ and $\\L^{T}\\Delta y=z$ via triangular\nback-substitution. \n","start":50226,"end":50384},{"raw_tex":"\\end{enumerate}\nThe overall runtime is cumulative, so $\\Tim=\\Tim_{\\mathrm{LP}}+\\Tim_{\\mathrm{SDP}}+\\Tim_{\\mathrm{fact}}+2\\Mem_{\\mathrm{fact}}$.\nThe overall memory use is $\\Mem=\\Mem_{\\mathrm{data}}+\\Mem_{\\mathrm{fact}}$,\nbecause the matrix $\\Hess$ can be constructed and then factored in-place.\n\\qed\n\\end{proof}","start":50384,"end":50694}]},{"id":"S052","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"复杂度主定理完整原证","source_units":[{"raw_tex":"\\begin{proof}One can verify that $(\\A,\\b,\\c,\\K)$ defined in (\\ref{eq:AKdef})\nspecifies an SDP in $(N,\\omega)$-standard form (see \\appref{regularity}\nfor the regularity conditions). Moreover, it follows from the monotonicity\nof the frontsize (\\propref{del}) that $\\omega\\equiv\\omega(E_{\\Pi})\\le\\omega(\\o E_{\\Pi})\\equiv\\o{\\omega}$.\nWe will track the cost of \\algref{myalg} step-by-step:\n\\begin{enumerate}\n\\item (Front-reducing permutation) Preordering $A_{i}\\gets\\Pi A_{i}\\Pi^{T}$\nand $C\\gets\\Pi C\\Pi^{T}$ the matrices cost $O(\\nnz(C)+\\sum_{i=1}^{m}\\nnz(A_{i}))=O(\\o{\\omega}^{2}N)$\ntime and memory. This follows from $\\nnz(C)\\le|F|\\le N$ and $\\nnz(A_{i})\\le\\o{\\omega}^{2}$. \n\\item (Conversion) Computing $F=\\chol(E)$ costs $O(|F|)=O(N)$ time and\nspace, where we note that $|F|\\le\\omega n$. \n\\item (Solution) Let $K=2\\cdot\\max\\{1,|\\tr(C)|,|\\tr(A_{1})|,\\dots,|\\tr(A_{m})|\\}$.\nAfter $O(\\sqrt{N}\\log(K/\\epsilon))$ iterations, we arrive at a primal\n$v\\le0$ and $V_{j}\\succeq0$ and dual point $Y\\in\\S_{F}^{n}$ satisfying\n\\begin{gather*}\n\\inner{A_{i}}Y-b_{i}\\le\\epsilon/K,\\quad Y[J_{j},J_{j}]\\succeq-(\\epsilon/K)I,\\\\\n{\\textstyle \\left\\Vert \\sum_{i}v_{i}A_{i}+\\sum_{j}P_{j}V_{j}P_{j}^{T}-C\\right\\Vert \\le\\epsilon/K,}\\\\\n\\inner CY-\\inner bv\\le N\\cdot(\\epsilon/K).\n\\end{gather*}\nEach iteration costs $O(\\o{\\omega}^{4}N)$ time and $O(\\o{\\omega}^{2}N)$\nmemory. This cost is fully determined by the cost of solving $O(1)$\ninstances of the Schur complement equation, which dominates the overhead\nof $O(\\omega^{3}n+\\nnz(\\A))=O(\\o{\\omega}^{2}N)$ time and memory.\n\\item (Recovery) Using the previously recovered $Y$, we recover $U$ such\nthat $\\Pi_{F}(UU^{T})=Y+\\delta I$ where $\\delta=-\\min_{j}\\{0,\\lambda_{\\min}(Y[J_{j},J_{j}])\\}\\le\\epsilon/K$.\nThis takes $O(\\omega^{3}n)=O(\\o{\\omega}^{2}N)$ time and $O(\\omega^{2}n)=O(\\o{\\omega}N)$\nmemory.\n\\item (Output) We output $U$ and $v$, and check for accuracy. It follows\nfrom $K\\ge2|\\tr(C)|$ that\n\\[\n\\inner C{UU^{T}}-\\inner CY=\\delta\\cdot\\tr(C)\\le\\epsilon\\cdot\\frac{\\tr(C)}{K}\\le\\epsilon\\cdot\\frac{|\\tr(C)|}{2|\\tr(C)|}\\le\\frac{1}{2}\\epsilon,\n\\]\nand from $K\\ge2$ and $N\\ge1$ that \n\\[\n\\inner C{UU^{T}}-\\inner bv\\le\\frac{N}{K}\\epsilon+\\frac{1}{2}\\epsilon=N\\epsilon\\cdot\\left(\\frac{1}{2}+\\frac{1}{2N}\\right)\\le N\\cdot\\epsilon.\n\\]\nSimilarly, it follows from $K\\ge2|\\tr(A_{i})|$ and $K\\ge2$ that\n$\\inner{A_{i}}{UU^{T}}-b_{i}\\le\\epsilon$. Finally, it follows from\n$v\\ge0$ and $V_{j}\\succeq0$ that\n\\[\n\\sum_{i}v_{i}A_{i}+\\sum_{j}P_{j}V_{j}P_{j}^{T}-C\\preceq\\epsilon\\cdot I\\implies\\sum_{i}v_{i}A_{i}-C\\preceq\\epsilon\\cdot I.\n\\]\n\\end{enumerate}\n\\qed \\end{proof}","start":52221,"end":54795}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"标准形式与前沿控制","source_units":[{"raw_tex":"\\begin{proof}One can verify that $(\\A,\\b,\\c,\\K)$ defined in (\\ref{eq:AKdef})\nspecifies an SDP in $(N,\\omega)$-standard form (see \\appref{regularity}\nfor the regularity conditions). Moreover, it follows from the monotonicity\nof the frontsize (\\propref{del}) that $\\omega\\equiv\\omega(E_{\\Pi})\\le\\omega(\\o E_{\\Pi})\\equiv\\o{\\omega}$.\nWe will track the cost of \\algref{myalg} step-by-step:\n","start":52221,"end":52606}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"重排与符号转换开销","source_units":[{"raw_tex":"\\begin{enumerate}\n\\item (Front-reducing permutation) Preordering $A_{i}\\gets\\Pi A_{i}\\Pi^{T}$\nand $C\\gets\\Pi C\\Pi^{T}$ the matrices cost $O(\\nnz(C)+\\sum_{i=1}^{m}\\nnz(A_{i}))=O(\\o{\\omega}^{2}N)$\ntime and memory. This follows from $\\nnz(C)\\le|F|\\le N$ and $\\nnz(A_{i})\\le\\o{\\omega}^{2}$. \n\\item (Conversion) Computing $F=\\chol(E)$ costs $O(|F|)=O(N)$ time and\nspace, where we note that $|F|\\le\\omega n$. \n","start":52606,"end":53010}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"按迹规模重标定精度","source_units":[{"raw_tex":"\\item (Solution) Let $K=2\\cdot\\max\\{1,|\\tr(C)|,|\\tr(A_{1})|,\\dots,|\\tr(A_{m})|\\}$.\nAfter $O(\\sqrt{N}\\log(K/\\epsilon))$ iterations, we arrive at a primal\n$v\\le0$ and $V_{j}\\succeq0$ and dual point $Y\\in\\S_{F}^{n}$ satisfying\n\\begin{gather*}\n\\inner{A_{i}}Y-b_{i}\\le\\epsilon/K,\\quad Y[J_{j},J_{j}]\\succeq-(\\epsilon/K)I,\\\\\n{\\textstyle \\left\\Vert \\sum_{i}v_{i}A_{i}+\\sum_{j}P_{j}V_{j}P_{j}^{T}-C\\right\\Vert \\le\\epsilon/K,}\\\\\n\\inner CY-\\inner bv\\le N\\cdot(\\epsilon/K).\n\\end{gather*}\n","start":53010,"end":53487}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"每步成本由 Schur 求解支配","source_units":[{"raw_tex":"Each iteration costs $O(\\o{\\omega}^{4}N)$ time and $O(\\o{\\omega}^{2}N)$\nmemory. This cost is fully determined by the cost of solving $O(1)$\ninstances of the Schur complement equation, which dominates the overhead\nof $O(\\omega^{3}n+\\nnz(\\A))=O(\\o{\\omega}^{2}N)$ time and memory.\n","start":53487,"end":53765}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"对角修正及低秩恢复","source_units":[{"raw_tex":"\\item (Recovery) Using the previously recovered $Y$, we recover $U$ such\nthat $\\Pi_{F}(UU^{T})=Y+\\delta I$ where $\\delta=-\\min_{j}\\{0,\\lambda_{\\min}(Y[J_{j},J_{j}])\\}\\le\\epsilon/K$.\nThis takes $O(\\omega^{3}n)=O(\\o{\\omega}^{2}N)$ time and $O(\\omega^{2}n)=O(\\o{\\omega}N)$\nmemory.\n","start":53765,"end":54043}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"恢复后的目标与约束误差","source_units":[{"raw_tex":"\\item (Output) We output $U$ and $v$, and check for accuracy. It follows\nfrom $K\\ge2|\\tr(C)|$ that\n\\[\n\\inner C{UU^{T}}-\\inner CY=\\delta\\cdot\\tr(C)\\le\\epsilon\\cdot\\frac{\\tr(C)}{K}\\le\\epsilon\\cdot\\frac{|\\tr(C)|}{2|\\tr(C)|}\\le\\frac{1}{2}\\epsilon,\n\\]\nand from $K\\ge2$ and $N\\ge1$ that \n\\[\n\\inner C{UU^{T}}-\\inner bv\\le\\frac{N}{K}\\epsilon+\\frac{1}{2}\\epsilon=N\\epsilon\\cdot\\left(\\frac{1}{2}+\\frac{1}{2N}\\right)\\le N\\cdot\\epsilon.\n\\]\nSimilarly, it follows from $K\\ge2|\\tr(A_{i})|$ and $K\\ge2$ that\n$\\inner{A_{i}}{UU^{T}}-b_{i}\\le\\epsilon$. Finally, it follows from\n","start":54043,"end":54602}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"移除半正定项的对偶误差","source_units":[{"raw_tex":"$v\\ge0$ and $V_{j}\\succeq0$ that\n\\[\n\\sum_{i}v_{i}A_{i}+\\sum_{j}P_{j}V_{j}P_{j}^{T}-C\\preceq\\epsilon\\cdot I\\implies\\sum_{i}v_{i}A_{i}-C\\preceq\\epsilon\\cdot I.\n\\]\n\\end{enumerate}\n","start":54602,"end":54779}]},{"id":"S060","semantic_type":"proposition","kind":"proposition","labels":["prop:del"],"member_ids":["S060","S061","S062","S063"],"declared_external":false,"source_only":false,"title":"保序子图的前沿单调性","source_units":[{"raw_tex":"\\begin{proposition}[Subgraph monotonicity]\n\\label{prop:del}Let $E$ be a sparsity pattern of order $n$, and\nlet $U\\subseteq\\{1,2,\\dots,n\\}$. Then, for any sparsity pattern $D$\nof order $|U|$ that satisfies $E[U]\\supseteq D$, we have $\\chol(E)[U]\\supseteq\\chol(D)$,\nand therefore $\\omega(E)\\ge\\omega(D)$.\n\\end{proposition}","start":57847,"end":58167},{"raw_tex":"\\begin{proof}\nIt is known that $(i,j)\\in\\chol(E)$ for $i>j$ holds if and only\nif there exists a path $(i,p_{1},p_{2},\\dots,p_{\\ell},j)$ whose edges\nare in $E$, and whose internal nodes are ordered $p_{1},p_{2},\\dots,p_{\\ell}<j<i$;\nsee e.g.~\\citep[Theorem~6.1]{vandenberghe2015chordal}. It immediately\n","start":58168,"end":58469},{"raw_tex":"\\begin{proof}\nIt is known that $(i,j)\\in\\chol(E)$ for $i>j$ holds if and only\nif there exists a path $(i,p_{1},p_{2},\\dots,p_{\\ell},j)$ whose edges\nare in $E$, and whose internal nodes are ordered $p_{1},p_{2},\\dots,p_{\\ell}<j<i$;\nsee e.g.~\\citep[Theorem~6.1]{vandenberghe2015chordal}. It immediately\nfollows this characterization that $\\chol(\\cdot)$ is monotone with\nrespect to the deletion of edges and isolated vertices: (1) if $D\\subseteq E$,\nthen $\\chol(D)\\subseteq\\chol(E)$; (2) we have $\\chol(E[U])=\\chol(E)[U]$\nfor $U=\\{1,2,\\dots,n\\}\\backslash v$ with isolated vertex $v$. Therefore,\n$\\chol(\\cdot)$ must also be monotone under general vertex and edge\ndeletions, because we can always delete edges to isolate a vertex\nbefore deleting it. \\qed\n\\end{proof}","start":58168,"end":58929},{"raw_tex":"follows this characterization that $\\chol(\\cdot)$ is monotone with\nrespect to the deletion of edges and isolated vertices: (1) if $D\\subseteq E$,\nthen $\\chol(D)\\subseteq\\chol(E)$; (2) we have $\\chol(E[U])=\\chol(E)[U]$\nfor $U=\\{1,2,\\dots,n\\}\\backslash v$ with isolated vertex $v$. Therefore,\n$\\chol(\\cdot)$ must also be monotone under general vertex and edge\ndeletions, because we can always delete edges to isolate a vertex\nbefore deleting it. \\qed\n\\end{proof}","start":58469,"end":58929}]},{"id":"S064","semantic_type":"lemma","kind":"lemma","labels":["lem:quadlift"],"member_ids":["S064","S081","S082","S084"],"declared_external":false,"source_only":false,"title":"二次提升的精确前沿","source_units":[{"raw_tex":"\\begin{lemma}[Quadratic lift]\n\\label{lem:quadlift}Let $E$ be an arbitrary sparsity pattern of\norder $n$. Define $F=\\chol(E)$ and the lifted sparsity pattern\n$F^{(2)}=\\bigcup_{k=1}^{n}\\clique(\\P_{k})$ in which each $\\P_{k}$\nis implicitly defined to satisfy $\\P_{k}^{T}\\svec_{F}(Y)=\\svec(Y[\\col_{F}(k),\\col_{F}(k)])$\nfor all $Y\\in\\S_{F}^{n}$. Then, we have $\\chol(F^{(2)})=F^{(2)}$\nand $\\omega(F^{(2)})=\\frac{1}{2}\\omega(E)[\\omega(E)+1]$.\n\\end{lemma}","start":59082,"end":59530},{"raw_tex":"\\begin{proof}[Proof of \\lemref{quadlift}]For an arbitrary sparsity\npattern $E$ of order $n$, let $F=\\chol(E)$ and $J_{k}\\equiv\\col_{F}(k)$.\nFirst, it follows from \\propref{zf->rip} that $J_{1},J_{2},\\dots,J_{n}$\nsatisfy the ordered running intersection property. Therefore, it follows\nfrom \\lemref{lift}, $J_{1}^{(2)},J_{2}^{(2)},\\dots,J_{n}^{(2)}$\ndefined in (\\ref{eq:Jk2_def}) satisfy the same property. Finally,\n","start":71246,"end":71661},{"raw_tex":"\\begin{proof}[Proof of \\lemref{quadlift}]For an arbitrary sparsity\npattern $E$ of order $n$, let $F=\\chol(E)$ and $J_{k}\\equiv\\col_{F}(k)$.\nFirst, it follows from \\propref{zf->rip} that $J_{1},J_{2},\\dots,J_{n}$\nsatisfy the ordered running intersection property. Therefore, it follows\nfrom \\lemref{lift}, $J_{1}^{(2)},J_{2}^{(2)},\\dots,J_{n}^{(2)}$\ndefined in (\\ref{eq:Jk2_def}) satisfy the same property. Finally,\nwe verify that \n\\begin{align*}\n\\supp(\\P_{k})\\equiv & \\left\\{ \\alpha:\\P_{k}[\\alpha,\\beta]\\ne0\\right\\} =\\left\\{ \\alpha:\\P_{k}^{T}e_{\\alpha}\\ne0\\right\\} \\\\\n\\overset{\\text{(a)}}{=} & \\left\\{ \\idx_{F}(i,j):\\P_{k}^{T}\\svec_{F}(e_{i}e_{j}^{T})\\ne0,\\;(i,j)\\in F\\right\\} \\\\\n\\overset{\\text{(b)}}{=} & \\left\\{ \\idx_{F}(i,j):(e_{i}e_{j}^{T})[J_{k},J_{k}]\\ne0,\\;i\\ge j\\right\\} =J_{k}^{(2)}.\n\\end{align*}\nEquality (a) is obtained by substituting $e_{\\alpha}=\\svec_{F}(e_{i}e_{j}^{T})$\nfor $(i,j)\\in F$. Equality (b) follows the identity $\\P_{k}^{T}\\svec_{F}(Y)=\\svec(Y[J_{k},J_{k}])$,\nwhich was used to define $\\P_{k}$. Therefore, $F^{(2)}=\\bigcup_{k=1}^{n}J_{k}^{(2)}$\nand we conclude via \\propref{rip->zf} that $\\omega(F^{(2)})=\\max_{k}|J_{k}^{(2)}|=\\max_{k}\\frac{1}{2}|J_{k}|(|J_{k}|+1)$,\nand we recall that $\\max_{k}|J_{k}|=\\omega(F)=\\omega(E)$ by definition.\n\\qed \\end{proof}","start":71246,"end":72527},{"raw_tex":"and we conclude via \\propref{rip->zf} that $\\omega(F^{(2)})=\\max_{k}|J_{k}^{(2)}|=\\max_{k}\\frac{1}{2}|J_{k}|(|J_{k}|+1)$,\nand we recall that $\\max_{k}|J_{k}|=\\omega(F)=\\omega(E)$ by definition.\n\\qed \\end{proof}","start":72317,"end":72527}]},{"id":"S065","semantic_type":"lemma","kind":"lemma","labels":["lem:ub"],"member_ids":["S065","S066"],"declared_external":false,"source_only":false,"title":"扩展提升模式的稀疏高估","source_units":[{"raw_tex":"\\begin{lemma}[Sparsity overestimate]\n\\label{lem:ub}Let $\\o E$ and $E^{(2)}$ be the sparsity patterns\ndefined in (\\ref{eq:Ebardef}) and (\\ref{eq:E2def}). Define $\\o F=\\chol(\\o E)$\nand the lifted sparsity pattern $\\o F^{(2)}=\\bigcup_{k=1}^{n}\\clique(\\o{\\P}_{k})$\nin which each $\\o{\\P}_{k}$ is implicitly defined to satisfy $\\o{\\P}_{k}^{T}\\svec_{\\o F}(Y)=\\svec(Y[\\col_{\\o F}(k),\\col_{\\o F}(k)])$\nfor all $Y\\in\\S_{\\o F}^{n}$. Then, $E^{(2)}\\subseteq\\o F^{(2)}[V^{(2)}]$\nholds for $V^{(2)}=\\{\\idx_{\\overline{F}}(i,j):i,j\\in F\\}$.\n\\end{lemma}","start":60078,"end":60614},{"raw_tex":"Our desired claim, namely that \n\\begin{align*}\nE^{(2)} & =\\left(\\bigcup_{i=1}^{m}\\clique(\\supp(a_{i}))\\right)\\cup\\left(\\bigcup_{j=1}^{n}\\clique(\\supp(\\P_{j}))\\right)\\\\\n & \\subseteq\\left(\\bigcup_{k=1}^{n}\\clique(\\supp(\\o{\\P}_{k})[V^{(2)}])\\right)=\\o F^{(2)}[V^{(2)}]\n\\end{align*}\nwhere $V^{(2)}=\\{\\idx_{\\overline{F}}(i,j):i,j\\in F\\}$, now follows\nimmediately from the following two lemmas.\n","start":72929,"end":73318}]},{"id":"S067","semantic_type":"proposition","kind":"proposition","labels":["prop:zf->rip"],"member_ids":["S067","S068"],"declared_external":false,"source_only":false,"title":"零填充推出有序运行交","source_units":[{"raw_tex":"\\begin{proposition}[ZF$\\implies$RIP]\n\\label{prop:zf->rip}Let $F$ be a zero-fill sparsity pattern of\norder $n$. Then, the sequence of subsets $J_{1},J_{2},\\dots,J_{n}$\nwith $J_{j}=\\col_{F}(j)\\equiv\\{j\\}\\cup\\{i>j:(i,j)\\in F\\}$ satisfies\nthe sorted running intersection property. \n\\end{proposition}","start":62493,"end":62788},{"raw_tex":"\\begin{proof}\nDefine $\\p(j)=\\min\\{i>j:i\\in J_{j}\\}$ if $|J_{j}|>1$ and $\\p(j)=n$\nif $|J_{j}|=1$. Clearly, $\\p(j)>j$ holds for all $1\\le j<n$. To\nprove $J_{\\p(j)}\\supseteq J_{j}\\cap\\bigcup_{w=j+1}^{\\ell}J_{w}$,\nlet $i\\in J_{j}\\cap J_{w}$ for some $w>j$. We prove that $i\\in J_{\\p(j)}$\nvia the following steps: \n\\begin{itemize}\n\\item We have $i>j$, because $i\\in J_{w}$ implies that $i\\ge w>j$. \n\\item If $i=\\p(j)$, then $i\\in J_{\\p(j)}$ by definition. \n\\item If $i>\\p(j)$, then $(i,j)\\in F$ and $(\\p(j),j)\\in F$ for $i>\\p(j)>j$\nimplies $(i,\\p(j))\\in F$, and hence $i\\in J_{\\p(j)}$. \n\\end{itemize}\nFinally, we prove $\\min\\{J_{\\p(j)}\\}>\\max\\{J_{j}\\backslash J_{\\p(j)}\\}$\nby noting that $\\max\\{J_{j}\\backslash J_{\\p(j)}\\}=j$ with our construction,\nand that $i=\\min J_{\\p(j)}$ must satisfy $i\\in J_{\\p(j)}$ and therefore\n$i\\ge\\p(j)>j$. \\qed\n\\end{proof}","start":62789,"end":63636}]},{"id":"S069","semantic_type":"lemma","kind":"lemma","labels":["lem:lift"],"member_ids":["S069","S070","S071","S072"],"declared_external":false,"source_only":false,"title":"提升保持有序运行交","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:lift}Let the sequence of subsets $J_{1},J_{2},\\dots,J_{\\ell}$\nwith $J_{j}\\subset\\N$ satisfy the sorted running intersection property.\nThen, $J_{1}^{(2)},J_{2}^{(2)},\\dots,J_{\\ell}^{(2)}$ also satisfy\nthe sorted running intersection property. \n\\end{lemma}","start":64620,"end":64899},{"raw_tex":"\\begin{proof}\nLet $\\p(\\cdot)$ denote the parent pointer that verifies the sorted\nrunning intersection property in $J_{1},J_{2},\\dots,J_{\\ell}$. We\nwill verify that $\\p(\\cdot)$ also proves the same property in $J_{1}^{(2)},J_{2}^{(2)},\\dots,J_{\\ell}^{(2)}$. \n\nFirst, to prove $J_{u}^{(2)}\\cap\\bigcup_{v=u+1}^{\\ell}J_{v}^{(2)}\\subseteq J_{\\p(u)}^{(2)},$\nlet $k\\in J_{u}^{(2)}\\cap J_{v}^{(2)}$ for $v>u$. The fact that\n$k\\in J_{u}^{(2)}=\\idx(\\clique(J_{u}))$ implies $k=\\idx(i,j)$ for\nsome $i,j$ such that $i,j\\in J_{u}$. Similarly, $k\\in J_{v}^{(2)}=\\idx(\\clique(J_{v}))$\nand the bijectivity of $\\idx$ on $F$ imply that the same $i,j$\nalso satisfy $i,j\\in J_{v}$, where we recall that $v>u$. We conclude\n$i,j\\in J_{u}\\cap J_{v}\\subseteq J_{\\p(u)}$ and therefore $k=\\idx(i,j)\\in J_{\\p(u)}^{(2)}$.\n\n","start":64900,"end":65695},{"raw_tex":"\\begin{proof}\nLet $\\p(\\cdot)$ denote the parent pointer that verifies the sorted\nrunning intersection property in $J_{1},J_{2},\\dots,J_{\\ell}$. We\nwill verify that $\\p(\\cdot)$ also proves the same property in $J_{1}^{(2)},J_{2}^{(2)},\\dots,J_{\\ell}^{(2)}$. \n\nFirst, to prove $J_{u}^{(2)}\\cap\\bigcup_{v=u+1}^{\\ell}J_{v}^{(2)}\\subseteq J_{\\p(u)}^{(2)},$\nlet $k\\in J_{u}^{(2)}\\cap J_{v}^{(2)}$ for $v>u$. The fact that\n$k\\in J_{u}^{(2)}=\\idx(\\clique(J_{u}))$ implies $k=\\idx(i,j)$ for\nsome $i,j$ such that $i,j\\in J_{u}$. Similarly, $k\\in J_{v}^{(2)}=\\idx(\\clique(J_{v}))$\nand the bijectivity of $\\idx$ on $F$ imply that the same $i,j$\nalso satisfy $i,j\\in J_{v}$, where we recall that $v>u$. We conclude\n$i,j\\in J_{u}\\cap J_{v}\\subseteq J_{\\p(u)}$ and therefore $k=\\idx(i,j)\\in J_{\\p(u)}^{(2)}$.\n\nNext, we prove $\\min\\{J_{\\p(u)}^{(2)}\\}>\\max\\{J_{u}^{(2)}\\backslash J_{\\p(u)}^{(2)}\\}$\nby establishing two claims:\n\\begin{itemize}\n\\item $\\min\\{J_{p(u)}^{(2)}\\}=\\idx(\\alpha,\\alpha)$ where $\\alpha=\\min J_{\\p(u)}$.\nFor any $\\idx(i,j)\\in J_{\\p(u)}^{(2)}$ where $i\\ge j$, we must have\n$i,j\\in J_{\\p(u)}$. It follows from the raster property that $\\idx(i,j)$\nis minimized with $i=j=\\min J_{\\p(u)}$.\n\\item $\\max\\{J_{u}^{(2)}\\backslash J_{\\p(u)}^{(2)}\\}=\\idx(\\beta,\\gamma)$\nwhere $\\beta=\\max\\{J_{u}\\}$ and $\\gamma=\\max\\{J_{u}\\backslash J_{\\p(u)}\\}$.\nWe partition $J_{u}$ into $N_{u}=J_{u}\\backslash J_{\\p(u)}$ and\n$A_{u}=J_{u}\\cap J_{\\p(u)}$. \n\\begin{itemize}\n\\item For any $\\idx(i,j)\\in J_{u}^{(2)}\\backslash J_{p(u)}^{(2)}$ where\n$i\\ge j$, we can have one of the following three cases: 1) $i\\in N_{u}$\nand $j\\in A_{u}$; 2) $i\\in A_{u}$ and $j\\in N_{u}$; or 3) $i\\in N_{u}$\nand $j\\in N_{u}$. \n\\item We observe that the first case $i\\in N_{u}$ and $j\\in A_{u}$ is\nimpossible. Indeed, applying $j\\ge\\min\\{J_{\\p(u)}\\}>\\max\\{J_{u}\\backslash J_{\\p(u)}\\}\\ge i$\nwould yield a contradiction with $i\\ge j$. \n\\item Taking the union of the two remaining cases yields $i\\in J_{u}=A_{u}\\cup N_{u}$\nand $j\\in N_{u}$. It follows from the raster property that $\\idx(i,j)$\nis maximized with $i=\\max J_{u}$ and $j=\\max N_{u}$.\n\\end{itemize}\n\\end{itemize}\nWith the two claims established, the hypothesis that $\\min\\{J_{\\p(u)}\\}>\\max\\{J_{u}\\backslash J_{\\p(u)}\\}$\nimplies that $\\alpha>\\gamma$, and therefore $\\idx(\\alpha,\\alpha)>\\idx(\\beta,\\gamma)$\nas desired. \\qed\n\\end{proof}","start":64900,"end":67246},{"raw_tex":"Next, we prove $\\min\\{J_{\\p(u)}^{(2)}\\}>\\max\\{J_{u}^{(2)}\\backslash J_{\\p(u)}^{(2)}\\}$\nby establishing two claims:\n\\begin{itemize}\n\\item $\\min\\{J_{p(u)}^{(2)}\\}=\\idx(\\alpha,\\alpha)$ where $\\alpha=\\min J_{\\p(u)}$.\nFor any $\\idx(i,j)\\in J_{\\p(u)}^{(2)}$ where $i\\ge j$, we must have\n$i,j\\in J_{\\p(u)}$. It follows from the raster property that $\\idx(i,j)$\nis minimized with $i=j=\\min J_{\\p(u)}$.\n\\item $\\max\\{J_{u}^{(2)}\\backslash J_{\\p(u)}^{(2)}\\}=\\idx(\\beta,\\gamma)$\nwhere $\\beta=\\max\\{J_{u}\\}$ and $\\gamma=\\max\\{J_{u}\\backslash J_{\\p(u)}\\}$.\nWe partition $J_{u}$ into $N_{u}=J_{u}\\backslash J_{\\p(u)}$ and\n$A_{u}=J_{u}\\cap J_{\\p(u)}$. \n\\begin{itemize}\n\\item For any $\\idx(i,j)\\in J_{u}^{(2)}\\backslash J_{p(u)}^{(2)}$ where\n$i\\ge j$, we can have one of the following three cases: 1) $i\\in N_{u}$\nand $j\\in A_{u}$; 2) $i\\in A_{u}$ and $j\\in N_{u}$; or 3) $i\\in N_{u}$\nand $j\\in N_{u}$. \n\\item We observe that the first case $i\\in N_{u}$ and $j\\in A_{u}$ is\nimpossible. Indeed, applying $j\\ge\\min\\{J_{\\p(u)}\\}>\\max\\{J_{u}\\backslash J_{\\p(u)}\\}\\ge i$\nwould yield a contradiction with $i\\ge j$. \n\\item Taking the union of the two remaining cases yields $i\\in J_{u}=A_{u}\\cup N_{u}$\nand $j\\in N_{u}$. It follows from the raster property that $\\idx(i,j)$\nis maximized with $i=\\max J_{u}$ and $j=\\max N_{u}$.\n\\end{itemize}\n\\end{itemize}\nWith the two claims established, the hypothesis that $\\min\\{J_{\\p(u)}\\}>\\max\\{J_{u}\\backslash J_{\\p(u)}\\}$\nimplies that $\\alpha>\\gamma$, and therefore $\\idx(\\alpha,\\alpha)>\\idx(\\beta,\\gamma)$\nas desired. \\qed\n\\end{proof}","start":65695,"end":67246}]},{"id":"S073","semantic_type":"proposition","kind":"proposition","labels":["prop:rip->zf"],"member_ids":["S073","S078","S079","S080"],"declared_external":false,"source_only":false,"title":"有序运行交推出零填充","source_units":[{"raw_tex":"\\begin{proposition}[RIP$\\implies$ZF]\n\\label{prop:rip->zf}Let $J_{1},J_{2},\\dots,J_{\\ell}$ with $\\bigcup_{j=1}^{\\ell}J_{j}=\\{1,2,\\dots,n\\}$\nsatisfy the sorted running intersection property. Then, $F=\\bigcup_{j=1}^{\\ell}\\clique(J_{j})$\nis zero-fill, and we have $\\omega(F)=\\max_{j}|J_{j}|$.\n\\end{proposition}","start":67535,"end":67841},{"raw_tex":"\\begin{proof}[Proof of \\propref{rip->zf}]To prove that $F$ is zero-fill,\nwe observe, for arbitrary $(i,k)\\in F$ and $(j,k)\\in F$ with $i>j>k$\nthat $i,j\\in\\col_{F}(k)$. Therefore, it follows from \\lemref{colrip}\nthat there exists $J_{w}$ such that $i,j,k\\in J_{w}$. We conclude\nthat $(i,j)\\in F,$ because $F=\\bigcup_{j=1}^{\\ell}\\clique(J_{j})$\nand $i,j\\in J_{w}$ for some $1\\le w\\le\\ell$. \n","start":70128,"end":70517},{"raw_tex":"\\begin{proof}[Proof of \\propref{rip->zf}]To prove that $F$ is zero-fill,\nwe observe, for arbitrary $(i,k)\\in F$ and $(j,k)\\in F$ with $i>j>k$\nthat $i,j\\in\\col_{F}(k)$. Therefore, it follows from \\lemref{colrip}\nthat there exists $J_{w}$ such that $i,j,k\\in J_{w}$. We conclude\nthat $(i,j)\\in F,$ because $F=\\bigcup_{j=1}^{\\ell}\\clique(J_{j})$\nand $i,j\\in J_{w}$ for some $1\\le w\\le\\ell$. \n\nTo prove that $\\omega(F)=\\max_{j}|J_{j}|$, we choose $k=\\arg\\max_{j}|\\col_{F}(j)|$.\nIt follows from Lemma~\\ref{lem:colrip} that there exists $J_{w}$\nsuch that $\\col_{F}(k)\\subseteq J_{w}$, and therefore $\\omega(F)=|\\col_{F}(k)|\\le|J_{w}|$.\nFinally, given that $\\clique(J_{w})\\subseteq F$ it follows that $(i,j)\\in F$\nholds for all $i\\in J_{w}$ where $j=\\min J_{w}$. Therefore, we conclude\nthat $J_{w}\\subseteq\\col_{F}(j)$, and therefore $|J_{w}|\\le|\\col_{F}(j)|\\le|\\col_{F}(k)|=\\omega(F)$.\n\\qed \\end{proof}","start":70128,"end":71024},{"raw_tex":"\nTo prove that $\\omega(F)=\\max_{j}|J_{j}|$, we choose $k=\\arg\\max_{j}|\\col_{F}(j)|$.\nIt follows from Lemma~\\ref{lem:colrip} that there exists $J_{w}$\nsuch that $\\col_{F}(k)\\subseteq J_{w}$, and therefore $\\omega(F)=|\\col_{F}(k)|\\le|J_{w}|$.\nFinally, given that $\\clique(J_{w})\\subseteq F$ it follows that $(i,j)\\in F$\nholds for all $i\\in J_{w}$ where $j=\\min J_{w}$. Therefore, we conclude\nthat $J_{w}\\subseteq\\col_{F}(j)$, and therefore $|J_{w}|\\le|\\col_{F}(j)|\\le|\\col_{F}(k)|=\\omega(F)$.\n\\qed \\end{proof}","start":70517,"end":71024}]},{"id":"S074","semantic_type":"lemma","kind":"lemma","labels":["lem:colrip"],"member_ids":["S074","S075","S076","S077"],"declared_external":false,"source_only":false,"title":"每个列集落在一个袋中","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:colrip}Let $J_{1},J_{2},\\dots,J_{\\ell}$ with $\\bigcup_{j=1}^{\\ell}J_{j}=\\{1,2,\\dots,n\\}$\nsatisfy the sorted running intersection property. For every $j$-th\ncolumn in $F=\\bigcup_{j=1}^{\\ell}\\clique(J_{j})$, there exists some\n$J_{w}$ such that $\\col_{F}(j)\\subseteq J_{w}.$\n\\end{lemma}","start":67974,"end":68282},{"raw_tex":"\\begin{proof}\nLet $V=\\{1,2,\\dots,n\\}$. For arbitrary $j\\in V$, denote $J_{w}$\nas the last subset in the sequence $J_{1},J_{2},\\dots,J_{\\ell}$ for\nwhich $j\\in J_{w}$. This choice must exist, because $V=\\bigcup_{k=1}^{\\ell}J_{k}$.\nFor every arbitrary $i\\in\\col_{F}(j)$, we will prove that $i\\in J_{w}$\nalso holds:\n\\begin{itemize}\n\\item There exists $u\\le w$ for which $i,j\\in J_{u}$. Indeed, $(i,j)\\in F$\nand $F=\\bigcup_{k=1}^{\\ell}\\clique(J_{k})$ imply $(i,j)\\in\\clique(J_{u})$\nfor some $J_{u}$, or equivalently $i,j\\in J_{u}$. Given that $j\\in J_{u}$\nand $w=\\max\\{k:j\\in J_{k}\\}$ by definition, it follows that $u\\le w$.\n\\item If $u=w$, then $i\\in J_{w}$ holds because $i,j\\in J_{u}$. Otherwise,\nif $u<w$, we use the sorted running intersection property to assert\nthe following\n\\begin{gather}\nu<w,\\qquad i,j\\in J_{u}\\quad\\implies\\quad i,j\\in J_{\\p(u)}.\\label{eq:par1}\n\\end{gather}\nThe fact that $j\\in J_{\\p(u)}$ follows directly the running intersection\nproperty $j\\in J_{u}\\cap J_{w}\\subseteq J_{\\p(u)}$ for $w>u$. By\ncontradiction, suppose that $i\\notin J_{\\p(u)}$. Then, given that\n$i\\in J_{u}$, it follows from the sorted property that $j\\ge\\min\\{J_{\\p(u)}\\}>\\max\\{J_{u}\\backslash J_{\\p(u)}\\}\\ge i,$\nwhich contradicts our initial hypothesis that $i>j$. \n","start":68283,"end":69541},{"raw_tex":"\\begin{proof}\nLet $V=\\{1,2,\\dots,n\\}$. For arbitrary $j\\in V$, denote $J_{w}$\nas the last subset in the sequence $J_{1},J_{2},\\dots,J_{\\ell}$ for\nwhich $j\\in J_{w}$. This choice must exist, because $V=\\bigcup_{k=1}^{\\ell}J_{k}$.\nFor every arbitrary $i\\in\\col_{F}(j)$, we will prove that $i\\in J_{w}$\nalso holds:\n\\begin{itemize}\n\\item There exists $u\\le w$ for which $i,j\\in J_{u}$. Indeed, $(i,j)\\in F$\nand $F=\\bigcup_{k=1}^{\\ell}\\clique(J_{k})$ imply $(i,j)\\in\\clique(J_{u})$\nfor some $J_{u}$, or equivalently $i,j\\in J_{u}$. Given that $j\\in J_{u}$\nand $w=\\max\\{k:j\\in J_{k}\\}$ by definition, it follows that $u\\le w$.\n\\item If $u=w$, then $i\\in J_{w}$ holds because $i,j\\in J_{u}$. Otherwise,\nif $u<w$, we use the sorted running intersection property to assert\nthe following\n\\begin{gather}\nu<w,\\qquad i,j\\in J_{u}\\quad\\implies\\quad i,j\\in J_{\\p(u)}.\\label{eq:par1}\n\\end{gather}\nThe fact that $j\\in J_{\\p(u)}$ follows directly the running intersection\nproperty $j\\in J_{u}\\cap J_{w}\\subseteq J_{\\p(u)}$ for $w>u$. By\ncontradiction, suppose that $i\\notin J_{\\p(u)}$. Then, given that\n$i\\in J_{u}$, it follows from the sorted property that $j\\ge\\min\\{J_{\\p(u)}\\}>\\max\\{J_{u}\\backslash J_{\\p(u)}\\}\\ge i,$\nwhich contradicts our initial hypothesis that $i>j$. \n\\item Inductively reapplying (\\ref{eq:par1}), as in $i,j\\in J_{\\p(\\p(u))}$\nand $i,j\\in J_{\\p(\\p(\\p(u)))}$, we arrive at some $v$ such that\n$i,j\\in J_{\\p(v)}$ and $p(v)=w$. The induction must terminate with\n$p(v)\\ge w$ because each $\\p(u)>u$ by construction. It is impossible\nfor $\\p(v)>w$ to occur, because $j\\in J_{\\p(v)}$ and $w=\\max\\{k:j\\in J_{k}\\}$\nby definition. We conclude that $i\\in J_{w}$, as desired.\n\\end{itemize}\n\\qed\n\\end{proof}","start":68283,"end":69982},{"raw_tex":"\\item Inductively reapplying (\\ref{eq:par1}), as in $i,j\\in J_{\\p(\\p(u))}$\nand $i,j\\in J_{\\p(\\p(\\p(u)))}$, we arrive at some $v$ such that\n$i,j\\in J_{\\p(v)}$ and $p(v)=w$. The induction must terminate with\n$p(v)\\ge w$ because each $\\p(u)>u$ by construction. It is impossible\nfor $\\p(v)>w$ to occur, because $j\\in J_{\\p(v)}$ and $w=\\max\\{k:j\\in J_{k}\\}$\nby definition. We conclude that $i\\in J_{w}$, as desired.\n\\end{itemize}\n\\qed\n\\end{proof}","start":69541,"end":69982}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S083"],"declared_external":false,"source_only":false,"title":"提取算子支持等于提升袋","source_units":[{"raw_tex":"we verify that \n\\begin{align*}\n\\supp(\\P_{k})\\equiv & \\left\\{ \\alpha:\\P_{k}[\\alpha,\\beta]\\ne0\\right\\} =\\left\\{ \\alpha:\\P_{k}^{T}e_{\\alpha}\\ne0\\right\\} \\\\\n\\overset{\\text{(a)}}{=} & \\left\\{ \\idx_{F}(i,j):\\P_{k}^{T}\\svec_{F}(e_{i}e_{j}^{T})\\ne0,\\;(i,j)\\in F\\right\\} \\\\\n\\overset{\\text{(b)}}{=} & \\left\\{ \\idx_{F}(i,j):(e_{i}e_{j}^{T})[J_{k},J_{k}]\\ne0,\\;i\\ge j\\right\\} =J_{k}^{(2)}.\n\\end{align*}\nEquality (a) is obtained by substituting $e_{\\alpha}=\\svec_{F}(e_{i}e_{j}^{T})$\nfor $(i,j)\\in F$. Equality (b) follows the identity $\\P_{k}^{T}\\svec_{F}(Y)=\\svec(Y[J_{k},J_{k}])$,\nwhich was used to define $\\P_{k}$. Therefore, $F^{(2)}=\\bigcup_{k=1}^{n}J_{k}^{(2)}$\n","start":71661,"end":72317}]},{"id":"S085","semantic_type":"lemma","kind":"lemma","labels":[],"member_ids":["S085","S086"],"declared_external":false,"source_only":false,"title":"约束支持落入扩展提升袋","source_units":[{"raw_tex":"\\begin{lemma}\nFor every $i\\in\\{1,2,\\dots,m\\}$, there exists $k\\in\\{1,2,\\dots,n\\}$\nsuch that $\\supp(a_{i})\\subseteq\\supp(\\o{\\P}_{k})[V^{(2)}]$ where\n$V^{(2)}=\\{\\idx_{\\overline{F}}(i,j):i,j\\in F\\}$. \n\\end{lemma}","start":73318,"end":73527},{"raw_tex":"\\begin{proof}\nIt follows by repeating the proof of \\lemref{quadlift} that\n\\begin{gather*}\n\\supp(\\o{\\P}_{k})=\\{\\idx_{\\o F}(i,j):i,j\\in\\col_{\\o F}(k),i\\ge j\\},\\\\\n\\supp(\\o{\\P}_{k})[V^{(2)}]=\\{\\idx_{F}(i,j):i,j\\in\\col_{\\o F}(k),i\\ge j\\}.\n\\end{gather*}\nOur desired claim follows via the following sequence of inclusions\n\\begin{align*}\n\\supp(a_{i}) & =\\{v:a_{i}[v]\\ne0\\}=\\{\\idx_{F}(u,v):A_{i}[u,v]\\ne0,\\quad(u,v)\\in F\\}\\\\\n & \\overset{\\text{(a)}}{\\subseteq}\\{\\idx_{F}(u,v):u,v\\in\\supp(A_{i}),\\quad(u,v)\\in F\\}\\\\\n & \\overset{\\text{(b)}}{\\subseteq}\\{\\idx_{F}(u,v):u,v\\in\\col_{\\o F}(k),\\quad(u,v)\\in F\\}=\\supp(\\o{\\P}_{k})[V^{(2)}].\n\\end{align*}\nInclusion (a) is because $\\spar(A_{i})\\subseteq\\clique(\\supp(A_{i}))$.\nInclusion (b) is true via the following: If $J=\\supp(A_{i})$ satisfies\n$\\clique(J)\\subseteq\\o F$, then $J\\subseteq\\col_{\\o F}(k)$ where\n$k=\\min J$. Indeed, we have $k\\in\\col_{\\o F}(k)$ by definition.\nFor any arbitrary $j\\in J$ with $j>k$, we must have an edge $(j,k)\\in\\clique(J)\\subseteq\\o F$,\nand therefore $j\\in\\col_{\\o F}(k)$. \\qed\n\\end{proof}","start":73528,"end":74581}]},{"id":"S087","semantic_type":"lemma","kind":"lemma","labels":[],"member_ids":["S087","S088"],"declared_external":false,"source_only":false,"title":"原提取支持嵌入扩展支持","source_units":[{"raw_tex":"\\begin{lemma}\nFor every $k\\in\\{1,2,\\dots,n\\}$, we have $\\supp(\\P_{k})\\subseteq\\supp(\\o{\\P}_{k})[V^{(2)}]$\nwhere $V^{(2)}=\\{\\idx_{\\overline{F}}(i,j):i,j\\in F\\}$. \n\\end{lemma}","start":74583,"end":74756},{"raw_tex":"\\begin{proof}\nIt again follows by repeating the proof of \\lemref{quadlift} that\n\\begin{align*}\n\\supp(\\P_{k}) & =\\{\\idx_{F}(i,j):i,j\\in\\col_{F}(k),i\\ge j\\}\\\\\n & \\overset{\\text{(a)}}{\\subseteq}\\{\\idx_{F}(i,j):i,j\\in\\col_{\\o F}(k),i\\ge j\\}=\\supp(\\o{\\P}_{k})[V^{(2)}].\n\\end{align*}\nInclusion (a) is true because $E\\subseteq\\o E$ implies $F=\\chol(E)\\subseteq\\chol(\\o E)=\\o F$\nvia \\propref{del}, and therefore $\\col_{F}(k)\\subseteq\\col_{\\o F}(k)$.\n\\qed\n\\end{proof}","start":74757,"end":75215}]},{"id":"S089","semantic_type":"lemma","kind":"lemma","labels":[],"member_ids":["S089","S090"],"declared_external":false,"source_only":false,"title":"转换数据满行秩","source_units":[{"raw_tex":"\\begin{lemma}[Linear independence]\nWe have $\\A^{T}y=0$ if and only $y=0$. \n\\end{lemma}","start":99320,"end":99406},{"raw_tex":"\\begin{proof}\nWe will prove $\\sum_{k=1}^{n}\\P_{k}\\P_{k}^{T}\\succeq I$, which implies\n$\\A\\A^{T}\\succeq I$ and hence $\\|\\A^{T}y\\|\\ge\\|y\\|$. For arbitrary\n$Y\\in\\S_{F}^{n}$ with $y=\\svec_{F}(Y)$, we observe that \n\\[\n\\|\\P_{k}^{T}y\\|^{2}=\\|\\svec(Y[J_{k},J_{k}])\\|^{2}=\\sum_{i,j\\in J_{k}}(Y[i,j])^{2}=\\sum_{(i,j)\\in\\clique(J_{k})}\\gamma_{ij}(Y[i,j])^{2}\n\\]\nwhere $\\gamma_{ij}=1$ if $i=j$ and $\\gamma_{ij}=2$ if $i\\ne j$.\nTherefore, we have\n\\[\n\\sum_{k=1}^{n}\\|\\P_{k}^{T}y\\|^{2}=\\sum_{k=1}^{n}\\sum_{(i,j)\\in\\clique(J_{k})}\\gamma_{ij}(Y[i,j])^{2}\\ge\\sum_{(i,j)\\in F}\\gamma_{ij}(Y[i,j])^{2}=\\|y\\|^{2}.\n\\]\nThe inequality is because $F=\\bigcup_{k=1}^{n}\\clique(J_{k})$. \\qed\n\\end{proof}","start":99407,"end":100080}]},{"id":"S091","semantic_type":"lemma","kind":"lemma","labels":[],"member_ids":["S091","S092","S093","S094"],"declared_external":false,"source_only":false,"title":"转换问题强对偶达到","source_units":[{"raw_tex":"\\begin{lemma}[Strong duality is attained]\nUnder \\asmref{strict}, there exists $x^{\\star},s^{\\star}\\in\\K$\nsatisfying $\\A x^{\\star}=\\b,$ $\\A^{T}y^{\\star}+s^{\\star}=\\c,$ $(x^{\\star})^{T}s^{\\star}=0$. \n\\end{lemma}","start":100082,"end":100291},{"raw_tex":"\\begin{proof}\nDefine $P_{j}$ implicitly to satisfy $P_{j}^{T}x=x[\\col_{F}(j)]$\nfor all $x\\in\\R^{m}$. \\asmref{strict} says that there exists $\\hat{X}\\succeq0$\nand $\\hat{y}\\le0$ and that satisfy $\\AA(\\hat{X})\\le b$ and $\\AA^{T}(\\hat{y})+\\hat{S}=C$\nand $\\inner C{\\hat{X}}=\\inner b{\\hat{y}}$. It follows from \\citep[Theorem~9.2]{vandenberghe2015chordal}\nthat $\\hat{S}=C-\\AA^{T}(\\hat{y})\\in\\S_{F}^{n}\\cap\\S_{+}^{n}$ if and\nonly if there exists $\\hat{S}_{j}\\succeq0$ such that $\\hat{S}=\\sum_{j=1}^{n}P_{j}\\hat{S}_{j}P_{j}^{T}$.\n","start":100292,"end":100814},{"raw_tex":"\\begin{proof}\nDefine $P_{j}$ implicitly to satisfy $P_{j}^{T}x=x[\\col_{F}(j)]$\nfor all $x\\in\\R^{m}$. \\asmref{strict} says that there exists $\\hat{X}\\succeq0$\nand $\\hat{y}\\le0$ and that satisfy $\\AA(\\hat{X})\\le b$ and $\\AA^{T}(\\hat{y})+\\hat{S}=C$\nand $\\inner C{\\hat{X}}=\\inner b{\\hat{y}}$. It follows from \\citep[Theorem~9.2]{vandenberghe2015chordal}\nthat $\\hat{S}=C-\\AA^{T}(\\hat{y})\\in\\S_{F}^{n}\\cap\\S_{+}^{n}$ if and\nonly if there exists $\\hat{S}_{j}\\succeq0$ such that $\\hat{S}=\\sum_{j=1}^{n}P_{j}\\hat{S}_{j}P_{j}^{T}$.\nNow, we turn to the primal-dual pair defined by the data $(\\A,\\b,\\c,\\K)$\nin (\\ref{eq:AKdef}), which is written \n\\begin{align*}\n & \\min_{v\\in\\R^{m},V_{j}\\in\\S^{\\omega_{j}}}\\left\\{ \\inner bv:\\begin{array}{c}\n\\proj_{F}\\left[\\AA^{T}(-v)+\\sum_{j=1}^{n}P_{j}V_{j}P_{j}^{T}-C\\right]=0,\\\\\nv\\ge0,\\quad V_{1},\\dots,V_{n}\\succeq0\n\\end{array}\\right\\} \\\\\n= & \\max_{Y\\in\\S_{F}^{n}}\\left\\{ \\inner{-C}Y:\\AA(Y)\\le b,\\;-P_{j}^{T}YP_{j}\\preceq0\\text{ for all }j\\in\\{1,2,\\dots,n\\}\\right\\} \n\\end{align*}\nWe can mechanically verify that $v^{\\star}=-\\hat{y}$ and $V_{j}^{\\star}=\\hat{S}_{j}$\nis feasible for the primal problem, and $Y^{\\star}=\\Pi_{F}(\\hat{X})$\nis feasible for the dual problem, and that the two objectives coincide\n$\\inner b{v^{\\star}}=\\inner{-C}{Y^{\\star}}$. Therefore, we conclude\nthat $x^{\\star}=(v^{\\star},\\svec(V_{1}^{\\star}),\\dots,\\svec(V_{n}^{\\star}))$\nand $y^{\\star}=\\svec_{F}(Y^{\\star})$ is a complementary solution\nsatisfying $\\A^{T}x^{\\star}=\\b,$ $\\A^{T}y^{\\star}+s^{\\star}=\\c,$\nand $\\inner{x^{\\star}}{s^{\\star}}=0$. \\qed \n\\end{proof}","start":100292,"end":101852},{"raw_tex":"Now, we turn to the primal-dual pair defined by the data $(\\A,\\b,\\c,\\K)$\nin (\\ref{eq:AKdef}), which is written \n\\begin{align*}\n & \\min_{v\\in\\R^{m},V_{j}\\in\\S^{\\omega_{j}}}\\left\\{ \\inner bv:\\begin{array}{c}\n\\proj_{F}\\left[\\AA^{T}(-v)+\\sum_{j=1}^{n}P_{j}V_{j}P_{j}^{T}-C\\right]=0,\\\\\nv\\ge0,\\quad V_{1},\\dots,V_{n}\\succeq0\n\\end{array}\\right\\} \\\\\n= & \\max_{Y\\in\\S_{F}^{n}}\\left\\{ \\inner{-C}Y:\\AA(Y)\\le b,\\;-P_{j}^{T}YP_{j}\\preceq0\\text{ for all }j\\in\\{1,2,\\dots,n\\}\\right\\} \n\\end{align*}\nWe can mechanically verify that $v^{\\star}=-\\hat{y}$ and $V_{j}^{\\star}=\\hat{S}_{j}$\nis feasible for the primal problem, and $Y^{\\star}=\\Pi_{F}(\\hat{X})$\nis feasible for the dual problem, and that the two objectives coincide\n$\\inner b{v^{\\star}}=\\inner{-C}{Y^{\\star}}$. Therefore, we conclude\nthat $x^{\\star}=(v^{\\star},\\svec(V_{1}^{\\star}),\\dots,\\svec(V_{n}^{\\star}))$\nand $y^{\\star}=\\svec_{F}(Y^{\\star})$ is a complementary solution\nsatisfying $\\A^{T}x^{\\star}=\\b,$ $\\A^{T}y^{\\star}+s^{\\star}=\\c,$\nand $\\inner{x^{\\star}}{s^{\\star}}=0$. \\qed \n\\end{proof}","start":100814,"end":101852}]},{"id":"S095","semantic_type":"proposition","kind":"proposition","labels":["prop:general"],"member_ids":["S095","S100","S101","S102","S103"],"declared_external":false,"source_only":false,"title":"标准形式内点法复杂度","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:general}Let $(\\A,\\b,\\c,\\K)$ describe an SDP in $(n,\\omega)$-standard\nform, and let $\\opt$ denote its optimal value. Let $\\Tim$ and $\\Mem$\ndenote the time and memory needed, given data $\\A\\in\\R^{M\\times N}$,\n$g\\in\\R^{M}$, and a choice of $w\\in\\interior(\\K)$, to form and solve\n$\\A\\D\\A^{T}\\Delta y=g$ for $\\Delta y\\in\\R^{M}$, where $\\D^{-1}=\\nabla^{2}f(w)$\nand $f=-\\log\\det(w)$. Then, a general-purpose interior-point method\ncomputes $(x,y)$ that satisfy the following in $O(\\sqrt{n}\\log(1/\\epsilon))$\niterations\n\\begin{gather*}\n\\c^{T}x\\le\\mathrm{opt}+n\\cdot\\epsilon,\\quad\\|\\A x-\\b\\|\\le\\epsilon,\\quad x\\in\\K,\\\\\n\\b^{T}y\\ge\\mathrm{opt}-n\\cdot\\epsilon,\\quad\\c-\\A^{T}y+\\epsilon\\one_{\\K}\\in\\K,\n\\end{gather*}\nwith per-iteration costs of $O(\\omega^{2}n+\\nnz(\\A)+\\Tim)$ time and\n$O(\\omega n+\\nnz(\\A)+\\Mem)$ memory.\n\\end{proposition}","start":102105,"end":102959},{"raw_tex":"\\begin{proof}[Proof of \\propref{general}]Recall that $(\\A,\\b,\\c,\\K)$\ndescribe an SDP in $(n,\\omega)$-standard form, and hence there exist\n$(x^{\\star},y^{\\star})$ such that $\\A x^{\\star}=\\b$ with $x^{\\star}\\in\\K$\nand $\\c-\\A^{T}y^{\\star}\\in\\K$ that satisfies strong duality $\\b^{T}y^{\\star}=\\c^{T}x^{\\star}$.\nCombining \\lemref{eps} and \\lemref{short-step} shows that iterations\nin (\\ref{eq:hsd_ipm}) converges to the desired $\\epsilon$-accurate,\n$\\epsilon$-feasible iterate after $O(\\sqrt{n}\\log(1/\\epsilon))$\niterations. The cost of each iteration is essentially equal to the\ncost of computing the search direction in (\\ref{eq:hsd_nt}). We account\nfor this cost via the following two steps:\n","start":107707,"end":108397},{"raw_tex":"\\begin{proof}[Proof of \\propref{general}]Recall that $(\\A,\\b,\\c,\\K)$\ndescribe an SDP in $(n,\\omega)$-standard form, and hence there exist\n$(x^{\\star},y^{\\star})$ such that $\\A x^{\\star}=\\b$ with $x^{\\star}\\in\\K$\nand $\\c-\\A^{T}y^{\\star}\\in\\K$ that satisfies strong duality $\\b^{T}y^{\\star}=\\c^{T}x^{\\star}$.\nCombining \\lemref{eps} and \\lemref{short-step} shows that iterations\nin (\\ref{eq:hsd_ipm}) converges to the desired $\\epsilon$-accurate,\n$\\epsilon$-feasible iterate after $O(\\sqrt{n}\\log(1/\\epsilon))$\niterations. The cost of each iteration is essentially equal to the\ncost of computing the search direction in (\\ref{eq:hsd_nt}). We account\nfor this cost via the following two steps:\n\\begin{enumerate}\n\\item (Scaling point) We partition $x=[\\svec(X_{j})]_{j=1}^{\\ell}$ and\n$s=[\\svec(S_{j})]_{j=1}^{\\ell}$. Then, the scaling point $w=[\\svec(W_{j})]_{j=1}^{\\ell}$\nis given in closed-form as $W_{j}=S_{j}^{1/2}(S_{j}^{1/2}X_{j}S_{j}^{1/2})^{-1/2}S_{j}^{1/2}$~\\citep[Section~5]{sturm2002implementation}.\nNoting that each $W_{j}$ is at most $\\omega\\times\\omega$, the cost\nof forming $w$ is at most of order \n\\[\n\\sum_{j=1}^{\\ell}\\omega_{j}^{3}\\le\\omega^{2}\\sum_{j=1}^{\\ell}\\omega_{j}=\\omega^{2}n\\text{ time},\\quad\\sum_{j=1}^{\\ell}\\omega_{j}^{2}\\le\\omega\\sum_{j=1}^{\\ell}\\omega_{j}=\\omega n\\text{ memory.}\n\\]\nBy this same calculation, it follows that the Hessian matrix-vector\nproducts $\\nabla^{2}f(w)x=[\\svec(W_{j}^{-1}X_{j}W_{j}^{-1})]_{j=1}^{\\ell}$\nand $\\nabla^{2}f(w)^{-1}x=[\\svec(W_{j}X_{j}W_{j})]_{j=1}^{\\ell}$\nalso cost $O(\\omega^{2}n)$ time and $O(\\omega n)$ memory. \n\\item (Search direction) Using elementary but tedious linear algebra, we\ncan show that if\\begin{subequations}\\label{eq:newt}\n\\begin{equation}\n(\\A\\D\\A^{T})\\begin{bmatrix}v_{1} & v_{2} & v_{3}\\end{bmatrix}=\\begin{bmatrix}0 & -\\b & r_{p}\\end{bmatrix}-\\A\\D\\begin{bmatrix}d & \\c & r_{d}\\end{bmatrix}\\label{eq:hsd_nrm}\n\\end{equation}\nwhere $\\D=[\\nabla^{2}f(w)]^{-1}$ and $d=-s-\\mu^{+}\\nabla f(x)$,\nand \n\\begin{equation}\n\\begin{bmatrix}u_{1} & u_{2} & u_{3}\\end{bmatrix}=\\D(\\begin{bmatrix}d & c & r_{d}\\end{bmatrix}+\\A^{T}\\begin{bmatrix}v_{1} & v_{2} & v_{3}\\end{bmatrix}),\\label{eq:hsd_back1}\n\\end{equation}\nthen\n\\begin{align}\n\\left(\\begin{bmatrix}-\\D_{0}^{-1} & -r_{c}\\\\\nr_{c} & 0\n\\end{bmatrix}-\\begin{bmatrix}\\c & r_{d}\\\\\n-\\b & r_{p}\n\\end{bmatrix}^{T}\\begin{bmatrix}u_{2} & u_{3}\\\\\nv_{2} & v_{3}\n\\end{bmatrix}\\right)\\begin{bmatrix}\\Delta\\tau\\\\\n\\Delta\\theta\n\\end{bmatrix} & =\\begin{bmatrix}-d_{0}\\\\\n0\n\\end{bmatrix}-\\begin{bmatrix}\\c & r_{d}\\\\\n-\\b & r_{p}\n\\end{bmatrix}^{T}\\begin{bmatrix}u_{1}\\\\\nv_{1}\n\\end{bmatrix},\\\\\n\\begin{bmatrix}\\Delta x\\\\\n\\Delta y\n\\end{bmatrix} & =\\begin{bmatrix}u_{1}\\\\\nv_{1}\n\\end{bmatrix}-\\begin{bmatrix}u_{1} & u_{2}\\\\\nv_{1} & v_{2}\n\\end{bmatrix}\\begin{bmatrix}\\Delta\\tau\\\\\n\\Delta\\theta\n\\end{bmatrix},\\\\\n\\Delta s & =d-\\D^{-1}\\Delta x,\\\\\n\\Delta\\kappa & =d_{0}-\\D_{0}^{-1}\\Delta\\tau,\\label{eq:hsd_back2}\n\\end{align}\n\\end{subequations}where $\\D_{0}=\\tau/\\kappa$ and $d_{0}=-\\kappa+\\mu^{+}\\tau^{-1}$.\nTherefore, we conclude that the cost of computing the search direction\nin (\\ref{eq:hsd_nt}) is equal to the cost of solving $O(1)$ instances\nof the Schur complement equation $\\A\\D\\A^{T}\\Delta y=g$, plus $O(1)$\nmatrix-vector products with $\\A,\\A^{T},\\D,\\D^{-1}$, for a total cost\nof $O(\\Tim+\\nnz(\\A)+\\omega^{2}n)$ time and $O(\\Mem+\\nnz(\\A)+\\omega n)$\nmemory respectively. Note that $\\A\\D\\A^{T}\\succ0$ because $\\A\\A^{T}\\succ0$\nby the linear independence assumption, and $\\D^{-1}=\\nabla^{2}f(w)\\succ0$\nfor all $w\\in\\interior(\\K)$.\n\\end{enumerate}\n\\qed \\end{proof}","start":107707,"end":111259},{"raw_tex":"\\begin{enumerate}\n\\item (Scaling point) We partition $x=[\\svec(X_{j})]_{j=1}^{\\ell}$ and\n$s=[\\svec(S_{j})]_{j=1}^{\\ell}$. Then, the scaling point $w=[\\svec(W_{j})]_{j=1}^{\\ell}$\nis given in closed-form as $W_{j}=S_{j}^{1/2}(S_{j}^{1/2}X_{j}S_{j}^{1/2})^{-1/2}S_{j}^{1/2}$~\\citep[Section~5]{sturm2002implementation}.\nNoting that each $W_{j}$ is at most $\\omega\\times\\omega$, the cost\nof forming $w$ is at most of order \n\\[\n\\sum_{j=1}^{\\ell}\\omega_{j}^{3}\\le\\omega^{2}\\sum_{j=1}^{\\ell}\\omega_{j}=\\omega^{2}n\\text{ time},\\quad\\sum_{j=1}^{\\ell}\\omega_{j}^{2}\\le\\omega\\sum_{j=1}^{\\ell}\\omega_{j}=\\omega n\\text{ memory.}\n\\]\nBy this same calculation, it follows that the Hessian matrix-vector\nproducts $\\nabla^{2}f(w)x=[\\svec(W_{j}^{-1}X_{j}W_{j}^{-1})]_{j=1}^{\\ell}$\nand $\\nabla^{2}f(w)^{-1}x=[\\svec(W_{j}X_{j}W_{j})]_{j=1}^{\\ell}$\nalso cost $O(\\omega^{2}n)$ time and $O(\\omega n)$ memory. \n","start":108397,"end":109282},{"raw_tex":"\\item (Search direction) Using elementary but tedious linear algebra, we\ncan show that if\\begin{subequations}\\label{eq:newt}\n\\begin{equation}\n(\\A\\D\\A^{T})\\begin{bmatrix}v_{1} & v_{2} & v_{3}\\end{bmatrix}=\\begin{bmatrix}0 & -\\b & r_{p}\\end{bmatrix}-\\A\\D\\begin{bmatrix}d & \\c & r_{d}\\end{bmatrix}\\label{eq:hsd_nrm}\n\\end{equation}\nwhere $\\D=[\\nabla^{2}f(w)]^{-1}$ and $d=-s-\\mu^{+}\\nabla f(x)$,\nand \n\\begin{equation}\n\\begin{bmatrix}u_{1} & u_{2} & u_{3}\\end{bmatrix}=\\D(\\begin{bmatrix}d & c & r_{d}\\end{bmatrix}+\\A^{T}\\begin{bmatrix}v_{1} & v_{2} & v_{3}\\end{bmatrix}),\\label{eq:hsd_back1}\n\\end{equation}\nthen\n\\begin{align}\n\\left(\\begin{bmatrix}-\\D_{0}^{-1} & -r_{c}\\\\\nr_{c} & 0\n\\end{bmatrix}-\\begin{bmatrix}\\c & r_{d}\\\\\n-\\b & r_{p}\n\\end{bmatrix}^{T}\\begin{bmatrix}u_{2} & u_{3}\\\\\nv_{2} & v_{3}\n\\end{bmatrix}\\right)\\begin{bmatrix}\\Delta\\tau\\\\\n\\Delta\\theta\n\\end{bmatrix} & =\\begin{bmatrix}-d_{0}\\\\\n0\n\\end{bmatrix}-\\begin{bmatrix}\\c & r_{d}\\\\\n-\\b & r_{p}\n\\end{bmatrix}^{T}\\begin{bmatrix}u_{1}\\\\\nv_{1}\n\\end{bmatrix},\\\\\n\\begin{bmatrix}\\Delta x\\\\\n\\Delta y\n\\end{bmatrix} & =\\begin{bmatrix}u_{1}\\\\\nv_{1}\n\\end{bmatrix}-\\begin{bmatrix}u_{1} & u_{2}\\\\\nv_{1} & v_{2}\n\\end{bmatrix}\\begin{bmatrix}\\Delta\\tau\\\\\n\\Delta\\theta\n\\end{bmatrix},\\\\\n\\Delta s & =d-\\D^{-1}\\Delta x,\\\\\n\\Delta\\kappa & =d_{0}-\\D_{0}^{-1}\\Delta\\tau,\\label{eq:hsd_back2}\n\\end{align}\n\\end{subequations}where $\\D_{0}=\\tau/\\kappa$ and $d_{0}=-\\kappa+\\mu^{+}\\tau^{-1}$.\nTherefore, we conclude that the cost of computing the search direction\nin (\\ref{eq:hsd_nt}) is equal to the cost of solving $O(1)$ instances\nof the Schur complement equation $\\A\\D\\A^{T}\\Delta y=g$, plus $O(1)$\nmatrix-vector products with $\\A,\\A^{T},\\D,\\D^{-1}$, for a total cost\nof $O(\\Tim+\\nnz(\\A)+\\omega^{2}n)$ time and $O(\\Mem+\\nnz(\\A)+\\omega n)$\nmemory respectively. Note that $\\A\\D\\A^{T}\\succ0$ because $\\A\\A^{T}\\succ0$\nby the linear independence assumption, and $\\D^{-1}=\\nabla^{2}f(w)\\succ0$\nfor all $w\\in\\interior(\\K)$.\n\\end{enumerate}\n\\qed \\end{proof}","start":109282,"end":111259}]},{"id":"S096","semantic_type":"lemma","kind":"lemma","labels":["lem:eps"],"member_ids":["S096","S097","S098","S099"],"declared_external":false,"source_only":false,"title":"嵌入解的精度恢复","source_units":[{"raw_tex":"\\begin{lemma}[$\\epsilon$-accurate and $\\epsilon$-feasible]\n\\label{lem:eps}Let there exist $(x^{\\star},y^{\\star})$ such that\n$\\A x^{\\star}=\\b$ with $x^{\\star}\\in\\K$ and $\\c-\\A^{T}y^{\\star}\\equiv s^{\\star}\\in\\K$\nthat satisfies strong duality $\\b^{T}y^{\\star}=\\c^{T}x^{\\star}$.\nThen, a point $(x,y,s,\\tau,\\theta,\\kappa)$ that is feasible for (\\ref{eq:hsd})\nand satisfies the optimality condition (\\ref{eq:short-step}) also\nsatisfies the following, where $\\rho=1+\\one_{\\K}^{T}(x^{\\star}+s^{\\star})$:\n\\begin{align*}\n\\|\\A(x/\\tau)-\\b\\| & \\le\\frac{\\rho\\|r_{p}\\|}{\\gamma}\\cdot\\epsilon, & \\|\\A^{T}(y/\\tau)+(s/\\tau)-\\c\\| & \\le\\frac{\\rho\\|r_{d}\\|}{\\gamma}\\cdot\\epsilon, & \\frac{(x/\\tau)^{T}(s/\\tau)}{n+1}\\le\\frac{\\rho^{2}}{\\gamma^{2}}\\cdot\\epsilon.\n\\end{align*}\n\\end{lemma}","start":105764,"end":106525},{"raw_tex":"\\begin{proof}\nFirst, observe that $(\\hat{x},\\hat{y},\\hat{s},\\hat{\\tau},\\hat{\\theta},\\hat{\\kappa})=(\\hat{\\tau}x^{\\star},\\hat{\\tau}y^{\\star},\\hat{\\tau}s^{\\star},\\hat{\\tau},0,0)$\nwith $\\hat{\\tau}=(n+1)/\\rho$ is a solution to (\\ref{eq:hsd}), because\n$r_{d}^{T}x^{\\star}+r_{p}^{T}y^{\\star}+r_{c}=\\one_{\\K}^{T}(x^{\\star}+s^{\\star})+1=\\rho$.\n","start":106526,"end":106861},{"raw_tex":"\\begin{proof}\nFirst, observe that $(\\hat{x},\\hat{y},\\hat{s},\\hat{\\tau},\\hat{\\theta},\\hat{\\kappa})=(\\hat{\\tau}x^{\\star},\\hat{\\tau}y^{\\star},\\hat{\\tau}s^{\\star},\\hat{\\tau},0,0)$\nwith $\\hat{\\tau}=(n+1)/\\rho$ is a solution to (\\ref{eq:hsd}), because\n$r_{d}^{T}x^{\\star}+r_{p}^{T}y^{\\star}+r_{c}=\\one_{\\K}^{T}(x^{\\star}+s^{\\star})+1=\\rho$.\nNext, we prove that if $(x,y,s,\\tau,\\theta,\\kappa)$ is feasible for\n(\\ref{eq:hsd}) and satisfies (\\ref{eq:short-step}), then $\\tau\\ge\\frac{\\gamma}{\\rho}$.\nIndeed, the skew-symmetry of (\\ref{eq:hsdfeas}) yields $\\theta=\\frac{x^{T}s+\\tau\\kappa}{n+1}$\nand $(x-\\hat{x})^{T}(s-\\hat{s})+(\\tau-\\hat{\\tau})(\\kappa-\\hat{\\kappa})=0.$\nRearranging yields $(n+1)\\theta=x^{T}s+\\tau\\kappa=\\hat{x}^{T}s+x^{T}\\hat{s}+\\tau\\hat{\\kappa}+\\hat{\\tau}\\kappa$\nand hence $\\kappa\\le(n+1)\\theta/\\hat{\\tau}$. Under (\\ref{eq:short-step}),\nwe have $\\theta=\\frac{x^{T}s+\\tau\\kappa}{n+1}\\le\\epsilon$ and $\\tau\\ge\\frac{\\gamma\\epsilon}{\\kappa}\\ge\\frac{\\gamma\\theta}{\\kappa}\\ge\\frac{\\gamma\\theta}{(n+1)\\theta/\\hat{\\tau}}=\\frac{\\gamma}{n+1}\\hat{\\tau}=\\frac{\\gamma}{\\rho}$.\nFinally, divide (\\ref{eq:hsdfeas}) through by $\\tau$ and substitute\n$1/\\tau\\le\\rho/\\gamma$. \\qed\n\\end{proof}","start":106526,"end":107705},{"raw_tex":"Next, we prove that if $(x,y,s,\\tau,\\theta,\\kappa)$ is feasible for\n(\\ref{eq:hsd}) and satisfies (\\ref{eq:short-step}), then $\\tau\\ge\\frac{\\gamma}{\\rho}$.\nIndeed, the skew-symmetry of (\\ref{eq:hsdfeas}) yields $\\theta=\\frac{x^{T}s+\\tau\\kappa}{n+1}$\nand $(x-\\hat{x})^{T}(s-\\hat{s})+(\\tau-\\hat{\\tau})(\\kappa-\\hat{\\kappa})=0.$\nRearranging yields $(n+1)\\theta=x^{T}s+\\tau\\kappa=\\hat{x}^{T}s+x^{T}\\hat{s}+\\tau\\hat{\\kappa}+\\hat{\\tau}\\kappa$\nand hence $\\kappa\\le(n+1)\\theta/\\hat{\\tau}$. Under (\\ref{eq:short-step}),\nwe have $\\theta=\\frac{x^{T}s+\\tau\\kappa}{n+1}\\le\\epsilon$ and $\\tau\\ge\\frac{\\gamma\\epsilon}{\\kappa}\\ge\\frac{\\gamma\\theta}{\\kappa}\\ge\\frac{\\gamma\\theta}{(n+1)\\theta/\\hat{\\tau}}=\\frac{\\gamma}{n+1}\\hat{\\tau}=\\frac{\\gamma}{\\rho}$.\nFinally, divide (\\ref{eq:hsdfeas}) through by $\\tau$ and substitute\n$1/\\tau\\le\\rho/\\gamma$. \\qed\n\\end{proof}","start":106861,"end":107705}]},{"id":"S104","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S104"],"declared_external":true,"declaration_text":["原始先决条件：Vandenberghe and Andersen, Theorem 6.1；原文 L1152–1155 引用。"],"source_only":false,"title":"Cholesky 填充路径定理","source_units":[]},{"id":"S105","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S105"],"declared_external":true,"declaration_text":["原始先决条件：Vandenberghe and Andersen, Theorem 9.2；原文 L1874–1876 引用。"],"source_only":false,"title":"弦稀疏半正定分解","source_units":[]},{"id":"S106","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S106"],"declared_external":true,"declaration_text":["原始先决条件：Standard positive-definite Cholesky factorization, triangular solves, and matrix-operation costs；正定合同结论为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"基础矩阵运算成本与正定性","source_units":[]},{"id":"S107","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S107"],"declared_external":true,"declaration_text":["原始先决条件：Todd and collaborators, equation 9，L1954；Sturm, Section 5，L2028。"],"source_only":false,"title":"Nesterov–Todd 方向与缩放公式","source_units":[]},{"id":"S108","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S108"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；用于 L1076–1078 和 L2006–2012 的省略背景。"],"source_only":false,"title":"锥序与残差的标准转换","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S004":"S004","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S038":"S038","S039":"S039","S040":"S039","S041":"S041","S042":"S041","S043":"S041","S044":"S041","S045":"S045","S046":"S045","S047":"S045","S048":"S045","S049":"S045","S050":"S045","S051":"S045","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S060","S062":"S060","S063":"S060","S064":"S064","S065":"S065","S066":"S065","S067":"S067","S068":"S067","S069":"S069","S070":"S069","S071":"S069","S072":"S069","S073":"S073","S074":"S074","S075":"S074","S076":"S074","S077":"S074","S078":"S073","S079":"S073","S080":"S073","S081":"S064","S082":"S064","S083":"S083","S084":"S064","S085":"S085","S086":"S085","S087":"S087","S088":"S087","S089":"S089","S090":"S089","S091":"S091","S092":"S091","S093":"S091","S094":"S091","S095":"S095","S096":"S096","S097":"S096","S098":"S096","S099":"S096","S100":"S095","S101":"S095","S102":"S095","S103":"S095","S104":"S104","S105":"S105","S106":"S106","S107":"S107","S108":"S108"}}}}
{"record_id":"2203.09762v4","paper":{"title":"Riemannian Interior Point Methods for Constrained Optimization on Manifolds","authors":["Lai, Zhijian","Yoshise, Akiko"],"arxiv_id":"2203.09762","version":"v4","journal":"Journal of Optimization Theory and Applications","abstract_url":"https://arxiv.org/abs/2203.09762v4","pdf_url":"https://arxiv.org/pdf/2203.09762v4","doi":"10.1007/s10957-024-02403-8"},"context":{"raw_tex":"In this paper, we consider the following Riemannian Constrained Optimization Problem (RCOP):\n\\begin{equation}\\tag{RCOP} \\label{RCOP} \n\t\\begin{array}{ll}\n\t\t\\underset{x \\in \\mathcal{M}}{\\min} & f(x) \\\\\n\t\t\\text { s.t. } & h(x)=0, \\text{ and }  g(x) \\leq 0,\\\\\n\t\\end{array}\n\\end{equation}\nwhere $\\mathcal{M}$ is a connected, complete $d$-dimensional Riemannian manifold and $f \\colon \\mathcal{M}\\to \\mathbb{R}, h\\colon\\mathcal{M}\\to \\mathbb{R}^{l}$, and $g\\colon\\mathcal{M}\\to \\mathbb{R}^{m}$ are smooth functions.\n\n\nLet us briefly review some concepts from Riemannian geometry, following the notation of \\cite{boumal2022intromanifolds}.\n$\\mathcal{M}$ denotes a finite-dimensional smooth manifold.\nLet $p \\in \\mathcal{M}$ and $T_p \\mathcal{M}$ be the tangent space at $p$ with $0_{p}$ being its zero element.\nWe use a canonical identification $ T_{p} \\mathcal{E}\\cong \\mathcal{E}$ for a vector space $ \\mathcal{E} $ and $p \\in \\mathcal{E}$.\nA vector field is a map $V \\colon \\mathcal{M} \\to T \\mathcal{M}$ with $V(p) \\in T_{p} \\mathcal{M}$, where $T \\mathcal{M}:=\\bigcup_{p \\in \\mathcal{M}} T_p \\mathcal{M}$ is the tangent bundle. \n$\\mathfrak{X}(\\mathcal{M})$ denotes the set of all smooth vector fields defined on $\\mathcal{M}$.\nFurthermore, $\\mathcal{M}$ is a Riemannian manifold if it is equipped with a Riemannian metric, that is, a choice of inner product $\\langle\\cdot, \\cdot\\rangle_p \\colon T_p \\mathcal{M} \\times T_p \\mathcal{M} \\to \\mathbb{R} $ for each tangent space at $p$ on $\\mathcal{M}$ such that for all $V, W \\in \\mathfrak{X}(\\mathcal{M})$, the map\n\\begin{equation}\\label{eq:RiemannianMetric}\n\tp \\mapsto\\langle V(p), W(p)\\rangle_p\n\\end{equation}\nis a smooth function from $\\mathcal{M}$ to $\\mathbb{R}$.\nRiemannian metric induces the norm $\\|\\xi\\|_p:=\\sqrt{\\langle\\xi, \\xi\\rangle_p}$ for $\\xi \\in T_p \\mathcal{M}$.\nWe often omit the subscript $p$ if it is clear from the context.\nThroughout this paper, we assume that all the manifolds involved are \\emph{connected} and \\emph{complete}.\nGiven a curve segment on $\\mathcal{M}$, $c:[a, b] \\rightarrow \\mathcal{M}$, the length of $c$ is defined as \n$\nL(c):=\\int_a^b \\| \\dot{c}(t)\\|_{c(t)} \\mathrm{d} t,\n$\nwhere $\\dot{c}(t) \\in T_{c(t)} \\mathcal{M}$ is the velocity vector of $c$ at $t$.\nSince $\\mathcal{M}$ is connected, there exists a curve segment connecting any pair of points $p,q \\in \\mathcal{M}$.\nIndeed, $\\mathcal{M}$ is a metric space under the Riemannian distance $d(p, q):=\\inf_{c} L(c) $ where the infimum is taken over all curve segments joining $p $ to $q $ \\cite[Thm. 2.55]{lee2018}.\nFor two manifolds $\\mathcal{M}_1, \\mathcal{M}_2$ and a smooth map $F \\colon \\mathcal{M}_1 \\to \\mathcal{M}_2$, the differential of $F$ at $p \\in \\mathcal{M}_1$ is a linear operator denoted as $\\mathcal{D} F(p) \\colon T_p \\mathcal{M}_1 \\to T_{F(p)} \\mathcal{M}_2$.\nLet $\\mathfrak{F}(\\mathcal{M})$ be the set of all smooth \\emph{scalar fields} (or say real-valued functions) $f\\colon \\mathcal{M} \\to \\mathbb{R}$.\nThe Riemannian gradient of $f$ at $p$, $\\operatorname{grad} f(p)$, is defined as the unique element of $T_p \\mathcal{M}$ that satisfies $\\langle \\xi, \\operatorname{grad} f(p)\\rangle_p = \\mathcal{D} f(p)[\\xi]$ for all $\\xi \\in T_p \\mathcal{M}$, where $\\mathcal{D} f(p) \\colon T_p \\mathcal{M} \\to T_{f(p)} \\mathbb{R} \\cong \\mathbb{R}$.\nNote that for any $f \\in \\mathfrak{F}(\\mathcal{M})$, the gradient vector field $x \\mapsto \\operatorname{grad} f(x)$ is a smooth vector field on $\\mathcal{M}$, i.e., $\\operatorname{grad} f \\in \\mathfrak{X}(\\mathcal{M})$.\nA retraction $\\mathrm{R} \\colon T \\mathcal{M} \\to \\mathcal{M}$ is a smooth map such that\n$\\mathrm{R}_{p}\\left(0_{p}\\right)=p$ and $\\mathcal{D} \\mathrm{R}_{p}\\left(0_{p}\\right) =\\operatorname{id}_{T_p \\mathcal{M}}$, i.e., the identity map on $T_{p} \\mathcal{M}$, where $\\mathrm{R}_{p}$ is the restriction of $\\mathrm{R}$ to $T_{p} \\mathcal{M}$ and $\\mathcal{D} \\mathrm{R}_{p}\\left(0_{p}\\right) \\colon T_{0_p} (T_p \\mathcal{M}) \\cong T_p \\mathcal{M} \\to T_{p} \\mathcal{M}$ is the differential of $\\mathrm{R}_{p}$ at $0_{p}$.\nOne theoretically perfect type of retraction is the exponential map, denoted as $\\operatorname{Exp}$. Since $\\mathcal{M}$ is complete, the exponential map is well-defined on the whole tangent bundle.\nLet $\\operatorname{Exp}_p\\colon T_p \\mathcal{M} \\rightarrow \\mathcal{M}$ be the exponential map at $p$; then $t \\mapsto \\operatorname{Exp}_p(t \\xi)$ is the unique geodesic that passes through $p$ with velocity $\\xi \\in T_p \\mathcal{M}$ when $t=0$.\n\n\\subsection{Riemannian Newton Method}\\label{subsec:RiemannianNewton}\n\nThe Newton method is a powerful tool for finding the zeros of nonlinear functions in the Euclidean setting.\nThe generalized Newton method has been studied in the Riemannian setting; it aims to find a \\emph{singularity} for the vector field $F \\in \\mathfrak{X}(\\mathcal{M})$, i.e., a point $p \\in \\mathcal{M}$ such that,\n\\begin{equation}\\label{eq:Singularity}\n\tF(p)=0_{p} \\in T_p \\mathcal{M}.\n\\end{equation}\nLet $\\nabla$ be the Levi-Civita connection on $\\mathcal{M}$, i.e., the unique symmetric connection compatible with the Riemannian metric.\nThe covariant derivative $\\nabla F$ assigns each point $p \\in \\mathcal{M}$ a linear operator $\\nabla F(p)\\colon T_{p}\\mathcal{M}\\to T_{p}\\mathcal{M}$. \nIn particular, the Riemannian Hessian of $f \\in \\mathfrak{F}(\\mathcal{M})$ at $p$ is a self-adjoint operator on $T_{p} \\mathcal{M}$, defined as $\\operatorname{Hess} f(p):=\\nabla (\\operatorname{grad}f)(p).$\n\n\nIn this section, we will give a comprehensive interpretation of the Riemannian interior point method.\nFollowing common usage in the interior-point literature, big letters denote the associated diagonal matrix, e.g., $Z =\\operatorname{diag}(z_{1}, \\ldots, z_{m})$ with $z \\in \\mathbb{R}^{m}$. $e$ denotes the all-ones vector, and $0$ stands for zero vector/matrix with proper dimensions.\n\n\\subsection{KKT Vector Field and Its Covariant Derivative}\n\nThe Lagrangian function of (\\ref{RCOP}) is\n\\begin{equation*}\n\t\\mathcal{L}(x, y, z):=f(x)+\\sum_{i=1}^l y_i h_i(x)+\\sum_{i=1}^m z_i g_i(x),\n\\end{equation*}\nwhere $y \\in \\mathbb{R}^l$ and $z \\in \\mathbb{R}^m$ are Lagrange multipliers corresponding to the equality and\ninequality constraints, respectively.\nWith respect to the variable $x$, $\\mathcal{L}(\\cdot, y, z)$ is a real-valued function defined on $\\mathcal{M}$, and its Riemannian gradient is\n\\begin{equation*}\n\\operatorname{grad}_{x} \\mathcal{L}(x, y, z)=\\operatorname{grad} f(x) + \\sum_{i =1 }^{l} y_{i} \\operatorname{grad} h_{i}(x) + \\sum_{i =1 }^{m} z_{i} \\operatorname{grad} g_{i}(x),\n\\end{equation*}\nwhere $\\{\\operatorname{grad} h_{i}(x)\\}_{i=1}^{l}$ and $ \\{\\operatorname{grad} g_{i}(x)\\}_{i=1}^{m}$ are the gradients of the component functions of $h$ and $g$. \nThe Riemannian KKT conditions (e.g., see \\cite[Definition 2.3]{liu2020simple} or \\cite{yang2014optimality}) for (\\ref{RCOP}) are given by\n\\begin{equation}\\label{eq:KKTConditions}\n\t\\operatorname{grad}_{x} \\mathcal{L}(x, y, z) =0_{x}; \\;\n\th(x) =0 ,\n\tg(x) \\leq 0 , \n\tz  \\geq 0; \\;\n\tZ g(x) =0.\n\\end{equation}\nThe above conditions can be written in terms of slack variables $\\mathbb{R}^{m} \\ni s:=-g(x)$, as\n\\begin{equation}\\label{eq:KKTVectorField}\n\tF(w):=\\left(\\begin{array}{l}\n\t\t\\operatorname{grad}_{x} \\mathcal{L}(x, y, z) \\\\\n\t\t h(x) \\\\\n\t\tg(x)+s \\\\\n\t\tZ S e\n\t\\end{array}\\right)=0_{w}\n\t=\\left(\\begin{array}{l}\n\t\t0_{x} \\\\\n\t\t0\\\\\n\t\t0\\\\\n\t\t0\n\t\\end{array}\\right) \\in T_w \\mathcal{N},\n\\end{equation}\nand $(z,s) \\geq 0,$\nwhere $w:=(x, y, z,s) \\in \\mathcal{N}:=\\mathcal{M}\\times \\mathbb{R}^{l} \\times \\mathbb{R}^{m} \\times \\mathbb{R}^{m}$.\nHere, we have generated a vector field $F$ in (\\ref{eq:KKTVectorField}) on the product manifold $\\mathcal{N}$, i.e.,\nfor any $w\\in \\mathcal{N}$, \n$\nF(w) \\in\nT_{w} \\mathcal{N} \\cong T_{x} \\mathcal{M} \\times \\mathbb{R}^{l} \\times \\mathbb{R}^{m} \\times \\mathbb{R}^{m}.\n$\nWe will call $F \\colon \\mathcal{N} \\to T\\mathcal{N} $ in (\\ref{eq:KKTVectorField}) above the \\emph{KKT vector field} for (\\ref{RCOP}).\nNote that for $\\xi=\\left(\\xi_x, \\xi_y, \\xi_z, \\xi_s\\right) \\in T_{w} \\mathcal{N} $,\n\\begin{equation}\\label{eq:barR}\n\t\\bar{\\mathrm{R}}_{w}(\\xi):=(\\mathrm{R}_{x}(\\xi_{x}), y+\\xi_{y}, z+\\xi_{z}, s+\\xi_{s})\n\\end{equation}\nis a well-defined retraction on the product manifold $\\mathcal{N}$ as long as $\\mathrm{R}$ is a retraction on $\\mathcal{M}$.\n\nNow, we aim to find a singularity of the KKT vector field $F$ on $\\mathcal{N}$ under some nonnegative constraints $(z,s) \\geq 0$.\nIf the Riemannian Newton method is to be applied to (\\ref{eq:KKTVectorField}), we must formulate the covariant derivative of $F$ at an arbitrary $w \\in \\mathcal{N}$. Before that, we need some new symbols.\nFixing a point $x \\in \\mathcal{M}$, we can define two linear operators $\\mathcal{H}_{x}\\colon \\mathbb{R}^{l} \\rightarrow T_{x} \\mathcal{M}$ and $\\mathcal{G}_{x}\\colon \\mathbb{R}^{m} \\rightarrow T_{x} \\mathcal{M}$ by\n\\begin{equation}\\label{defn:HxGx}\n\t\\mathcal{H}_x [v]:=\\sum_{i=1}^l v_i \\operatorname{grad} h_i(x), \\quad \\mathcal{G}_x [u]:=\\sum_{i=1}^m u_i \\operatorname{grad} g_i(x),\n\\end{equation}\nrespectively.\nThen, for $\\mathcal{H}_{x}$, its adjoint operator $\\mathcal{H}_x^*\\colon T_x \\mathcal{M} \\rightarrow \\mathbb{R}^l$ is given by\n\\begin{equation*}\n\t\\mathcal{H}_x^* [\\xi]=\\left(\\left\\langle\\operatorname{grad} h_1(x), \\xi\\right\\rangle_x, \\cdots,\\left\\langle\\operatorname{grad} h_l(x), \\xi\\right\\rangle_x\\right)^T.\n\\end{equation*}\nAlso, $\\mathcal{G}_x^*\\colon T_x \\mathcal{M} \\rightarrow \\mathbb{R}^m$ with $\\mathcal{G}_x^* [\\xi]=\\left(\\left\\langle\\operatorname{grad} g_1(x), \\xi\\right\\rangle_x, \\cdots,\\left\\langle\\operatorname{grad} g_m(x), \\xi\\right\\rangle_x\\right)^T$. \n\nNow, using the solutions of Exercises 5.4 and 5.13 on covariant derivatives of vector fields on product manifolds in monograph \\cite{boumal2022intromanifolds}, the following results can be easily derived.\nGiven KKT vector field $F $ in (\\ref{eq:KKTVectorField}) and $w \\in \\mathcal{N}$, the covariant derivative of $F$ at $w$ is the operator $\\nabla F(w)\\colon T_{w} \\mathcal{N} \\to T_{w} \\mathcal{N}$ given by\n\\begin{equation}\\label{eq:CovariantDerivativeKKTVectorField}\n\t\\nabla F(w) [\\Delta w] =\n\t\\left(\\begin{array}{l}\n\t\t\\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t\t+\\mathcal{H}_{x} [\\Delta y]\n\t\t+\\mathcal{G}_{x} [\\Delta z]\\\\\n\t\t\\mathcal{H}_{x}^{*}[\\Delta x]\\\\\n\t\t\\mathcal{G}_{x}^{*}[\\Delta x]+\\Delta s\\\\\n\t\tZ \\Delta s + S \\Delta z\n\t\\end{array}\\right),\n\\end{equation}\nwhere $\\Delta w=(\\Delta x, \\Delta y, \\Delta z, \\Delta s) \\in T_{w} \\mathcal{N}$ and $\\operatorname{Hess}_{x} \\mathcal{L}(w)$ is the Riemannian Hessian of real-valued function $\\mathcal{L}(\\cdot, y, z)$.\nIndeed,\n$\\operatorname{Hess}_{x} \\mathcal{L}(w)\\colon T_{x}\\mathcal{M} \\to T_{x}\\mathcal{M}$ satisfies\n\\begin{equation}\\label{eq:HessianLagrange}\n\t\\operatorname{Hess}_{x} \\mathcal{L}(w) =\n\t\\operatorname{Hess} f(x)\n\t+\\sum_{i=1}^{l} y_{i} \\operatorname{Hess} h_{i}(x)\n\t+\\sum_{i=1}^{m} z_{i} \\operatorname{Hess} g_{i}(x),\n\\end{equation}\nwhere $\\{\\operatorname{Hess} h_{i}(x)\\}_{i=1}^{l}$ and $\\{\\operatorname{Hess} g_{i}(x)\\}_{i=1}^{m}$ are Hessians of the component functions of $h$ and $g$.\n\n\nNext, we will discuss an important result about the covariant derivative of the KKT vector field given in (\\ref{eq:CovariantDerivativeKKTVectorField}).\nLet $\\mathbb{A}(x):=\\left\\{i \\mid g_{i}(x)=0\\right\\}$ denote the active set at $x \\in \\mathcal{M}$.\nA prior study on Riemannian optimality conditions \\cite{yang2014optimality,bergmann2019intrinsic} showed that the following assumptions are meaningful.\nWe call them the \\emph{standard Riemannian assumptions} for (\\ref{RCOP}).\nNote that the $x^{*}$ and $w^*$ in \\ref{B3}-\\ref{B5} all refer to those in \\ref{B1}.\n\n\n\\setlist[assumptions,1]{label=\\textbf{(A\\arabic*)}}\n\\begin{assumptions}[leftmargin=9mm]\n\t\\item {Existence.} There exists $w^*=(x^{*}, y^{*}, z^{*}, s^{*})$ satisfying the Riemannian KKT conditions (\\ref{eq:KKTConditions}). Here, we introduce the slack variables $s^{*}:=-g(x^{*})$.\n\t\\label{B1}\t\n%\t\\item {Smoothness of $ C^{2} $.} The functions $f, g,$ and $h$ are $ C^{2} $ on $\\mathcal{M}$.\n%\t\\label{B2}\t\n\t\\item {Linear Independence Constraint Qualification (LICQ).} \\\\ The set $\\left\\{\\operatorname{grad} h_{i}(x^{*})\\right\\}_{i=1}^{l} \\bigcup \\left\\{\\operatorname{grad} g_{i}(x^{*})\\right\\}_{i \\in \\mathbb{A}(x^{*})}$ is linearly independent in $ T_{x^{*}}\\mathcal{M}$.\n\t\\label{B3}\t\n\t\\item {Strict complementarity.} $(z^{*})_{i}>0$ if $g_{i}(x^{*})=0$ for all $i=1, \\cdots,m$.\n\t\\label{B4}\t\n\t\\item {Second-order sufficiency.}\n\t$\\left \\langle\\operatorname{Hess}_{x} \\mathcal{L}(w^{*})[\\xi], \\xi\\right\\rangle>0$ for all nonzero $\\xi \\in T_{x^{*}}\\mathcal{M}$ \\\\ satisfying $\\left\\langle\\xi, \\operatorname{grad} h_{i}(x^{*})\\right\\rangle=0 $ for $i=1,\\dots,l,$ and\n\t$\\left\\langle\\xi, \\operatorname{grad} g_{i}(x^{*})\\right\\rangle=0$ for $ i \\in \\mathbb{A}(x^{*})$. \n\t\\label{B5}\n\\end{assumptions}\n\n\nApplying the Riemannian Newton method directly to the KKT vector field $F: \\mathcal{N} \\rightarrow T \\mathcal{N}$ results in the following Newton equation (see (\\ref{eq:StandardNewtonEquation}) without iteration count $k$) at each iteration:\n\\begin{equation}\\label{eq:OriginalNewtonEquation}\n\t\\nabla F(w) [\\Delta w]+F(w)=0.\n\\end{equation}\nAs with the usual interior point method in the Euclidean setting, once the iterates reach the boundary of the feasible region, they are forced to stick to it \\cite[P6]{wright1997primal}.\nFor the iterates to maintain a sufficient distance from the boundary, we introduce a perturbed complementary equation with some barrier parameter $\\mu>0$ and define the \\textit{perturbed KKT vector field}:\n\\begin{equation}\\label{eq:perturbedKKTvectorfield}\n\tF_{\\mu}(w):=F(w)-\\mu \\hat{e}, \\text { and } \\hat{e}:=\\hat{e}(w):=\\left(0_x, 0,0, e\\right).\n\\end{equation}\nNotice that the perturbation term $\\hat{e}$, indeed, is a special vector field on $\\mathcal{N}$, not a constant, because $0_x$ is essentially dependent on $w$ and/or $x$.\nIn fact, the covariant derivative of the perturbed KKT vector field is the same as that of the original. From the linearity of the connection $\\nabla$, for any $w \\in \\mathcal{N}$ and any $\\mu>0$, we have \n\\begin{equation}\\label{eq:NabulaperturbedKKTvectorfield}\n\t\\nabla F_{\\mu}(w)=\\nabla F(w)-\\mu \\nabla \\hat{e}(w)=\\nabla F(w),\n\\end{equation}\nwhere the last equity comes from $\\nabla \\hat{e}(w) [\\Delta w]=\\left(0_x, 0,0,0\\right)$ for all $\\Delta w \\in T_w \\mathcal{N}$.\nApplying the Riemannian Newton method to perturbed KKT vector field $F_{\\mu}(w)$ yields the \\emph{perturbed Newton equation}: $\\nabla F_{\\mu}(w) [\\Delta w]+F_{\\mu}(w)=0$. \nFrom (\\ref{eq:perturbedKKTvectorfield}) and (\\ref{eq:NabulaperturbedKKTvectorfield}), this equation is equivalent to \n\\begin{equation}\n\t\\nabla F(w) [\\Delta w]+F(w)=\\mu \\hat{e},\n\\end{equation}\nwhich reduces to the ordinary Newton equation (\\ref{eq:OriginalNewtonEquation}) as $\\mu \\to 0$.\nAt this point, we can describe a prototype of the Riemannian Interior Point Method (RIPM) in Algorithm \\ref{algo:prototype_algo}.\n\n\\begin{algorithm}%[H]\n\t\\caption{Prototype Algorithm of RIPM for (\\ref{RCOP})}\n\t\\label{algo:prototype_algo}\n\t\\SetAlgoLined\n\t\n\t\\KwIn{An initial point $w_0=\\left(x_0, y_0, z_0, s_0\\right) \\in \\mathcal{N}$ with $\\left(z_0, s_0\\right)>0$ and a retraction $\\mathrm{R}$ on $\\mathcal{M}$. $\\hat{\\gamma} \\in(0,1)$, $\\mu_0>0$.}\n\t\\KwOut{Sequence $\\left\\{w_k\\right\\} \\subset \\mathcal{N}$ such that $\\left\\{w_k\\right\\} \\rightarrow w^*$ and $w^*$ satisfies the KKT conditions (\\ref{eq:KKTConditions}).} % \n\t\n\tSet $k \\to 0$\\;\n\t\n\t\\While{Stopping criterion not satisfied}{\n\t\t1. Obtain $\\Delta w_{k}=(\\Delta x_{k}, \\Delta y_{k}, \\Delta z_{k},\\Delta s_{k}) \\in T_{w_k} \\mathcal{N}$ by solving the perturbed Newton equation on $T_{w_k} \\mathcal{N}$:\n\t\t\\begin{equation}\\label{eq:prototype}\n\t\t\t\\nabla F(w_{k}) [\\Delta w_{k}]=-F(w_{k})+\\mu_{k} \\hat{e};\n\t\t\\end{equation}\n\t\n\t\t2. Choose $ \\gamma_{k} \\in [\\hat{\\gamma},1]$ and compute the step size $\\alpha_{k}$ defined by\n\t\t\\begin{equation}\\label{rule:StepSizeA}\n\t\t\t\\min \\left\\{1, \\gamma_{k} \\min _{i}\\left\\{-\\frac{(s_{k})_{i}}{(\\Delta s_{k})_{i}} \\mid(\\Delta s_{k})_{i}<0\\right\\},\\gamma_{k} \\min _{i}\\left\\{-\\frac{(z_{k})_{i}}{(\\Delta z_{k})_{i}} \\mid(\\Delta z_{k})_{i}<0\\right\\}\\right\\};\n\t\t\\end{equation}\n\t\n\t\t3. Compute the next point as\n\t\t$w_{k+1}:=\\bar{\\mathrm{R}}_{w_k}\\left(\\alpha_k \\Delta w_k\\right)$, see (\\ref{eq:barR})\\;\n\t\t\n\t\t4. Choose $0<\\mu_{k+1}<\\mu_k$\\;\n\t\t\n\t\t5. $k \\to k+1$\\;\n\t}\n\\end{algorithm}\n\nIn the step 2 of Algorithm \\ref{algo:prototype_algo}, we just wish to compute a step size $0<\\alpha_k \\leq 1$ to ensure new point $w_{k+1}$ with $\\left(z_{k+1},s_{k+1}\\right)>0$. \nThere are many schemes to achieve this purpose. \nThe scheme in (\\ref{rule:StepSizeA}) is simple but sufficient to guarantee the local convergence, as will be proved in Section \\ref{sec:local}.\n\n\nLet us consider Algorithm \\ref{algo:prototype_algo} and omit the iteration count $k$.\nGiven the current point $w \\in \\mathcal{N}$ with $(z, s) > 0$, for the KKT vector field $F(w)$ in (\\ref{eq:KKTVectorField}), we denote its components by $F_x,F_y,F_z,F_s$ in top-to-bottom order, namely,\n$F_x:=\\operatorname{grad}_x \\mathcal{L}(w),F_y:=h(x),F_z:=g(x)+s,F_s:=Z S e.$\nBy using these symbols and the formulation of $\\nabla F(w)$ in (\\ref{eq:CovariantDerivativeKKTVectorField}), the \\emph{full} (perturbed) Newton equation (\\ref{eq:prototype}) defined on $T_w \\mathcal{N} \\cong T_x \\mathcal{M} \\times \\mathbb{R}^l \\times \\mathbb{R}^m \\times \\mathbb{R}^m$ is expanded as:\n\\begin{equation}\\label{eq:FullNewtonEquation}\n\t\\left(\\begin{array}{l}\n\t\t\\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t\t+\\mathcal{H}_{x} [\\Delta y]\n\t\t+\\mathcal{G}_{x} [\\Delta z]\\\\\n\t\t\\mathcal{H}_{x}^{*}[\\Delta x]\\\\\n\t\t\\mathcal{G}_{x}^{*}[\\Delta x]+\\Delta s\\\\\n\t\tZ \\Delta s + S \\Delta z\n\t\\end{array}\\right)\n\t=\n\t\\left(\\begin{array}{l}\n\t\t-F_x\\\\\n\t\t-F_y\\\\\n\t\t-F_z\\\\\n\t\t-F_s+\\mu e\n\t\\end{array}\\right).\n\\end{equation}\n\n\n\\begin{remark}\n\tFor a retraction $\\mathrm{R}$ on $\\mathcal{M}$ and $x \\in \\mathcal{M}$,\n\tby $\\mathcal{D} \\mathrm{R}_{x}\\left(0_{x}\\right)=\\mathrm{id}_{T_{x} \\mathcal{M}}$ and the inverse function theorem, there exists a neighborhood $V$ of $0_{x}$ in $T_{x} \\mathcal{M}$ such that $ \\mathrm{R}_{x} $ is a diffeomorphism on $V$; thus,\n\t$\\mathrm{R}_{x}^{-1}(y)$ is well defined for all $ y \\in \\mathcal{M} $ sufficiently close to $ x $.\n\tIn this case, $\\mathrm{R}_{x}(V) \\subset \\mathcal{M}$ is called a retractive neighborhood of $x$.\n\tFurthermore, the existence of a \\emph{totally} retractive neighborhood \\cite[Theorem 2]{zhu2020riemannian} shows that for any $\\bar{x} \\in \\mathcal{M} $ there is a neighborhood $W$ of $\\bar{x}$ such that\n\t$\\mathrm{R}_{x}^{-1}(y)$ is well defined for all $x, y \\in W$.\n\tIn what follows, we will suppose that an appropriate neighborhood has been chosen by default for the well-definedness of $\\mathrm{R}_x^{-1}(y)$.\n\\end{remark}\n\n\nDefine the Whitney sum\n$\nT \\mathcal{M} \\oplus T \\mathcal{M}:=\\left\\{(\\eta, \\xi): \\eta, \\xi \\in T_x \\mathcal{M}, x \\in \\mathcal{M} \\right\\}.\n$\nA smooth map\n$\n\\mathrm{T} \\colon T \\mathcal{M} \\oplus T \\mathcal{M} \\to T \\mathcal{M}: (\\eta, \\xi) \\mapsto \\mathrm{T}_{\\eta}(\\xi),\n$\nis called a \\emph{vector transport} on $\\mathcal{M}$ if there exists an associated retraction $\\mathrm{R}$ on $\\mathcal{M}$ such that $\\mathrm{T}$ satisfies the following properties for all $x \\in \\mathcal{M}$:\n\\begin{enumerate}\n\t\\item Associated retraction. $\\mathrm{T}_{\\eta}\\left(\\xi\\right) \\in T_{\\mathrm{R}_{x}\\left(\\eta\\right)} \\mathcal{M}$\n\tfor all $\\eta, \\xi \\in T_x \\mathcal{M}$.\t\n\t\\item Consistency. $\\mathrm{T}_{0_{x}}\\left(\\xi\\right)=\\xi$ for all $\\xi \\in T_{x} \\mathcal{M}$.\t\n\t\\item Linearity.\n\t$\\mathrm{T}_{\\eta}\\left(a \\xi+b \\zeta\\right)=a \\mathrm{T}_{\\eta}\\left(\\xi\\right)+b \\mathrm{T}_{\\eta}\\left(\\zeta\\right)$ for all $a, b \\in \\mathbb{R}$ and $\\eta, \\xi, \\zeta \\in T_{x} \\mathcal{M}$.\n\\end{enumerate}\nThus, fixing any $\\eta \\in T_x \\mathcal{M}$, the map\n$\n\\mathrm{T}_\\eta \\colon T_x \\mathcal{M} \\to T_{\\mathrm{R}_x(\\eta)} \\mathcal{M}: \\xi \\mapsto \\mathrm{T}_\\eta(\\xi),\n$\nis a linear operator.\nAdditionally, $\\mathrm{T}$ is \\emph{isometric} if \n$\n\\langle \\mathrm{T}_{\\eta} (\\xi), \\mathrm{T}_{\\eta} (\\zeta)\\rangle =\\langle\\xi, \\zeta\\rangle\n$\nholds, for all $x \\in \\mathcal{M}$ and all $\\xi, \\zeta, \\eta \\in T_{x} \\mathcal{M}$.\nIn other words, for any $\\eta \\in T_x \\mathcal{M}$, the adjoint and the inverse of $\\mathrm{T}_{\\eta}$ coincide, i.e., $ \\mathrm{T}_{\\eta}^{*} =  \\mathrm{T}_{\\eta}^{-1}$.\nThere are two important classes of vector transport as follows. Let $\\mathrm{R}$ be a retraction on $\\mathcal{M}$. \n\\begin{enumerate}[(1)]\n\t\\item \n\tThe \\emph{differentiated retraction} defined by\n\t\\begin{equation}\\label{eq:DifferentiatedRetraction}\n\t\t\\mathrm{T}_{\\eta}(\\xi):=\\mathcal{D} \\mathrm{R}_{x}(\\eta)[\\xi],  \\; \\eta, \\xi \\in T_{x} \\mathcal{M}, x \\in \\mathcal{M}, \n\t\\end{equation}\n\tis a valid vector transport \\cite[equation (8.6)]{absil2009optimization}.\t\n\t\\item\n\tGiven a smooth curve $\\gamma \\colon [0,1] \\to \\mathcal{M}$ and $t_0, t_1 \\in [0,1]$, the \\emph{parallel transport} from the tangent space at $\\gamma(t_0)$ to the tangent space at $\\gamma(t_1)$ along $\\gamma,$\n\tis a linear operator\n\t$ \\mathrm{P}_\\gamma^{t_1 \\to t_0} \\colon T_{\\gamma\\left(t_0\\right)} \\mathcal{M} \\rightarrow T_{\\gamma\\left(t_1\\right)} \\mathcal{M}$ defined by $\\mathrm{P}_\\gamma^{t_1 \\to t_0}(\\xi)=Z\\left(t_1\\right)$, where $Z$ is the unique parallel vector field such that $Z\\left(t_0\\right)=\\xi$.\n\tThen, for any $x \\in \\mathcal{M}, \\eta \\in T_x \\mathcal{M}$, then\n\t\\begin{equation}\\label{eq:ParallelTransport}\n\t\t\\mathrm{T}_{\\eta}\\left(\\xi\\right):=\\mathrm{P}_\\gamma^{1 \\to 0} (\\xi)\n\t\\end{equation}\n\tis a valid vector transport \\cite[equation (8.2)]{absil2009optimization}, where $\\mathrm{P}_\\gamma$ denotes the parallel transport along the curve $t \\mapsto \\gamma(t):=\\mathrm{R}_x\\left(t \\eta\\right)$. We often omit the superscript $^{1 \\to 0}$ if it is clear from the context.\n\tIn particular, parallel transport is isometric.\n\\end{enumerate}\n\n\nGoing one degree higher, let us now discuss the Lipschitz continuity of Hessian operators.\nThroughout this paper, for a linear operator $\\mathcal{A}: \\mathcal{E} \\rightarrow \\mathcal{E}^{\\prime}$ between two finite-dimensional normed vector spaces $\\mathcal{E}$ and $\\mathcal{E}^{\\prime}$, the (operator) norm of $\\mathcal{A}$ is defined by $\\|\\mathcal{A}\\|:=\\sup \\left\\{\\|\\mathcal{A} v\\|_{\\mathcal{E}^{\\prime}} : v \\in \\mathcal{E},\\|v\\|_{\\mathcal{E}}=1\\right.$, or, $\\left.\\|v\\|_{\\mathcal{E}} \\leq 1\\right\\}.$\nThe inverse of $\\mathrm{T}_{\\eta}$ is needed in the following definitions, so we can assume that vector transport $\\mathrm{T}$ is isometric, e.g., parallel transport in (\\ref{eq:ParallelTransport}). In fact, there are many ways to construct isometric vector transports; see \\cite[Section 2.3]{huang2015riemannian}.\n\\begin{definition}[{\\cite[Assumption 3]{huang2015riemannian}}]\n\t\\label{defn:TwiceLipschitzContinuouslyDifferentiable}\n\tA function $f\\colon \\mathcal{M} \\rightarrow \\mathbb{R}$ is \\emph{twice Lipschitz continuously differentiable with respect to $\\mathrm{T}$ in $\\mathcal{U} \\subset \\mathcal{M}$} if it is twice differentiable and there exists a constant $\\kappa>0$ such that, for all $x, y \\in \\mathcal{U}$, \n\t$\n\t\\|\\operatorname{Hess} f(y)-\\mathrm{T}_{\\eta} \\operatorname{Hess} f(x) \\mathrm{T}_{\\eta}^{-1}\\| \\leq \\kappa d(x, y),\n\t$\n\twhere $\\eta=\\mathrm{R}_{x}^{-1} y$.\n\\end{definition}\n\\begin{lemma}[{\\cite[Lemma 4]{huang2015riemannian}}]\n\t\\label{lem:TwiceLipschitzContinuouslyDifferentiable}\n\tIf $f\\colon \\mathcal{M} \\rightarrow \\mathbb{R}$ is $ C^3 $, then for any $\\bar{x} \\in \\mathcal{M}$ and any isometric vector transport $\\mathrm{T}$, there exists a neighborhood $\\mathcal{U}$ of $\\bar{x}$ such that $f$ is twice Lipschitz continuously differentiable with respect to $\\mathrm{T}$ in $\\mathcal{U}$.\n\\end{lemma}\n\nIf the operator, $\\operatorname{Hess}f(x)$, above is replaced by a general covariant derivative $\\nabla F (x)$, we can get the next results in a similar way.\nLemma \\ref{lem:LipCountNabulaF} can be proven in the same way as Lemma \\ref{lem:TwiceLipschitzContinuouslyDifferentiable}.\n\\begin{definition}\n\tGiven a vector field $F$ on $\\mathcal{M}$.\n\tThe map $x \\mapsto \\nabla F(x)$ is \\emph{Lipschitz continuous with respect to $\\mathrm{T}$ in $\\mathcal{U} \\subset \\mathcal{M}$} if there exists a constant $\\kappa>0$ such that, for all $x, y \\in \\mathcal{U}$, it holds that\n\t$\n\t\\| \\nabla F(y)\n\t-\n\t\\mathrm{T}_{\\eta} \\nabla F(x) \\mathrm{T}_{\\eta}^{-1}\n\t\\| \\leq \\kappa d(x, y),\n\t$\n\twhere $\\eta=\\mathrm{R}_{x}^{-1} y$.\n\\end{definition}\n\\begin{lemma}\\label{lem:LipCountNabulaF}\n\tIf $F$ is a $C^{2}$ vector field, then\n\tfor any $\\bar{x} \\in \\mathcal{M}$ and any isometric vector transport $\\mathrm{T}$, there exists a neighborhood $\\mathcal{U}$ of $\\bar{x}$ such that the map $x \\mapsto \\nabla F(x)$ is Lipschitz continuous with respect to $\\mathrm{T}$ in $\\mathcal{U}$.\n\\end{lemma}\n\nNotice that in the previous subsection on the definitions of Lipschitz continuity, we used $\\|\\eta\\|$ with $\\eta=\\mathrm{R}_x^{-1} y$ or $d(x, y)$ to denote the upper bound on the right-hand side. The next lemma shows that the two are not essentially different. When $\\mathcal{M}=\\mathbb{R}^{n}$, both reduce to $\\| x-y \\|$.\n\n\\begin{lemma}[{\\cite[Lemma 2]{huang2015riemannian}}]\\label{lem:distR1}\n\tLet $\\mathcal{M}$ be a Riemannian manifold with a retraction $\\mathrm{R}$ and let $\\bar{x} \\in \\mathcal{M}$. \n\tThen,\n\t\\begin{enumerate}[(i)]\n\t\t \\item there exist $a_{0}, a_{1}, \\delta_{a_{0}, a_{1}}>0$ such that for all $x$ in a sufficiently small neighborhood of $\\bar{x}$ and all $\\xi, \\eta \\in T_{x} \\mathcal{M}$ with $\\|\\xi\\|, \\|\\eta\\|  \\leq \\delta_{a_{0}, a_{1}}$,\n\t\t one has\n\t\t $\n\t\t a_{0}\\|\\xi-\\eta\\| \\leq d(\\mathrm{R}_{x}(\\eta), \\mathrm{R}_{x}(\\xi)) \\leq a_{1}\\|\\xi-\\eta\\|.\n\t\t $\t\n\t\t In particular,\n\t\t $\n\t\t a_{0}\\|\\xi\\| \\leq d(x, \\mathrm{R}_{x}(\\xi)) \\leq a_{1}\\|\\xi\\|\n\t\t $\n\t\t when $\\eta=0$;\n\t\t \n\t\t \\item\n\t\t there exist $a_{0},a_{1}>0$ such that for all $x$ in a sufficiently small neighborhood of $\\bar{x}$, one has\n\t\t $\n\t\t a_{0}\\|\\xi\\| \\leq d(x,\\bar{x}) \\leq a_{1}\\|\\xi\\| \\text { where } \\xi=\\mathrm{R}_{\\bar{x}}^{-1}(x). \n\t\t $\n\t\\end{enumerate}\n\\end{lemma}\n\nThe next lemma is the fundamental theorem of calculus in the Riemannian case.\n\\begin{lemma}[{\\cite[Lemma 8]{huang2015riemannian}}]\\label{lem:FundamentalTheoremOfCalculusRiemannian}\n\tLet $F$ be a $C^{1}$ vector field and $\\bar{x} \\in \\mathcal{M}$. \n\tThen there exist a neighborhood $\\mathcal{U}$ of $\\bar{x}$ and a constant $c_{1} \\geq 0$ such that for all $x, y \\in \\mathcal{U}$,\n\t\\begin{equation*}\n\t\\left\\| \\mathrm{P}_{\\gamma}^{0 \\to 1} [F(y)]-F(x)-\\int_{0}^{1} \\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} [\\eta] \\mathrm{d} t \\right\\| \n\t\\leq c_{1} \\|\\eta\\|^{2},\n\t\\end{equation*}\n\twhere $\\eta=\\mathrm{R}_{x}^{-1}(y)$ and $\\mathrm{P}_{\\gamma}$ is the parallel transport along the curve $\\gamma(t):=\\mathrm{R}_{x}\\left(t \\eta\\right)$.\n\tMoreover, if $\\mathrm{R}=\\operatorname{Exp}$, then indeed $c_1=0$ above (see \\cite[equation (2.4)]{ferreira2002kantorovich}).\n\\end{lemma}\n\n\n\\begin{lemma}[{\\cite[Lemma 3.2]{fernandes2017superlinear}}]\\label{lem:nonsingular}\n\tGiven a vector field $F$ on $\\mathcal{M}$.\n\tIf the map $p \\mapsto \\nabla F(p)$ is continuous at $p^{*}$ and $\\nabla F(p^{*})$ is nonsingular, \n\tthen there exist a neighborhood $\\mathcal{U}$ of $p^{*}$ and a constant $\\Xi > 0$ such that, for all $p \\in \\mathcal{U}$, $\\nabla F(p)$ is nonsingular and $\\left\\|\\nabla F(p)^{-1}\\right\\| \\leq \\Xi$.\n\\end{lemma}\n\\begin{lemma}[{\\cite[Lemma 14.5]{gallivan2012riemannian}}]\\label{lem:dist_zero}\n\tLet $F$ be a  $C^2$ vector field on $\\mathcal{M}$ and $p^{*} \\in \\mathcal{M}$. If $ F(p^{*})=0 $ and $\\nabla F(p^{*}) $ is nonsingular, then there exist a neighborhood $\\mathcal{U}$ of $p^{*}$ and constants $c_{3}, c_{4} > 0$ such that, for all $p \\in \\mathcal{U}$, $ c_{3} d(p, p^{*}) \\leq \\|F(p)\\| \\leq c_{4} d(p, p^{*}).$\n\\end{lemma}\n\n\nHere, for any two nonnegative sequences $\\left\\{u_{k}\\right\\}$ and $\\left\\{v_{k}\\right\\}$, we write $u_{k}=O(v_{k})$ if there is a constant $M >0$ such that\n$u_{k}\\leq M v_{k}$\nfor all sufficiently large $k$; and we write\n$u_{k}=o(v_{k})$\nif $v_{k}>0$ and the sequence of ratios $\\{u_{k} / v_{k}\\} $ approaches zero.\n\n\n\\begin{algorithm}%[H]\n\t\\caption{Perturbed Damped Riemannian Newton Method for (\\ref{eq:Singularity})}\n\t\\label{algo:PerturbedDampedNewtonMethod}\n\t\\SetAlgoLined\n\t\n\t\\KwIn{A vector field $F \\in \\mathfrak{X}(\\mathcal{M})$, an initial point $p_0 \\in \\mathcal{M}$ and a retraction $\\mathrm{R}$ on $\\mathcal{M}$. Set $\\mu_0>0$.}\n\t\\KwOut{Sequence $\\{p_k\\} \\subset \\mathcal{M}$ such that $\\left\\{p_k\\right\\} \\rightarrow p^*$ and $F(p^{*})=0_{p^{*}}$.}\n\t\n\tSet $k \\to 0$\\;\n\t\n\t\\While{Stopping criterion not satisfied}{\n\t\t1. Obtain $\\xi_k \\in T_{p_k}\\mathcal{M}$ by solving the perturbed Newton equation:\n\t\t\\begin{equation}\n\t\t\t\\nabla F(p_{k}) [\\xi_k] = - F(p_{k}) + \\mu_{k} \\hat{e};\n\t\t\\end{equation}\n\t\n\t\t2. Choose a (damped) step size $ 0<\\alpha_{k} \\leq 1$\\;\n\t\t\n\t\t3. Compute the next point as $p_{k+1}:=\\mathrm{R}_{p_{k}} (\\alpha_{k} \\xi_k)$\\;\n\t\t\n\t\t4. Choose $0<\\mu_{k+1}<\\mu_k$\\;\n\t\t\n\t\t5. $k \\to k + 1$\\;\n\t}\n\\end{algorithm}\n\n\n\\begin{assumptions}[leftmargin=9mm]\n\t\\item There exists $p^{*} \\in \\mathcal{M}$ such that $F(p^{*})=0_{p^{*}}$;\n\t\\label{A1}\n\t\\item The covariant derivative $\\nabla F (p^{*})$ is nonsingular;\n\t\\label{A2}\n%\t\\item The map $p \\mapsto \\nabla F(p) $ is Lipschitz continuous with respect to on a neighborhood of $p^{*}$;\n%\t\\label{A3}\n\t\\item The vector field $F$ is $C^2$.\n\t\\label{A3}\n\\end{assumptions}\n\n\n\\begin{lemma}\\label{lem:gamma_alpha}\n\tConsider the Algorithm \\ref{algo:prototype_algo} for solving the problem (\\ref{RCOP}).  \n\tLet \\ref{B1} and \\ref{B4} hold at some $w^{*}=(x^{*}, y^{*}, z^{*}, s^{*})$ and $ \\alpha_{k} $ be as in (\\ref{rule:StepSizeA}). \n\tDefine a constant,\n\t\\begin{equation*}\n\t\\Pi := 2 \\max \\left\\{\\max _{i}\\left\\{\n\t1/(s^{*})_{i}\n\t\\mid (s^{*})_{i}>0\\right\\}, \\max _{i}\\left\\{\n\t1/(z^{*})_{i}\n\t\\mid (z^{*})_{i}>0\\right\\}\\right\\}.\n\t\\end{equation*}\n\tFor $\\gamma_{k} \\in(0,1)$, if\n\t$\n\t\\Pi \\left\\| \\Delta w_{k}\\right\\| \\leq \\gamma_{k},\n\t$\n\tthen\n\t$\n\t0  \\leq 1-\\alpha_{k} \\leq\n\t(1-\\gamma_{k})+\\Pi \\left\\| \\Delta w_{k}\\right\\|.\n\t$\n\\end{lemma}","units":[{"raw_tex":"In this paper, we consider the following Riemannian Constrained Optimization Problem (RCOP):\n\\begin{equation}\\tag{RCOP} \\label{RCOP} \n\t\\begin{array}{ll}\n\t\t\\underset{x \\in \\mathcal{M}}{\\min} & f(x) \\\\\n\t\t\\text { s.t. } & h(x)=0, \\text{ and }  g(x) \\leq 0,\\\\\n\t\\end{array}\n\\end{equation}\nwhere $\\mathcal{M}$ is a connected, complete $d$-dimensional Riemannian manifold and $f \\colon \\mathcal{M}\\to \\mathbb{R}, h\\colon\\mathcal{M}\\to \\mathbb{R}^{l}$, and $g\\colon\\mathcal{M}\\to \\mathbb{R}^{m}$ are smooth functions.\n","start":3609,"end":4119},{"raw_tex":"Let us briefly review some concepts from Riemannian geometry, following the notation of \\cite{boumal2022intromanifolds}.\n$\\mathcal{M}$ denotes a finite-dimensional smooth manifold.\nLet $p \\in \\mathcal{M}$ and $T_p \\mathcal{M}$ be the tangent space at $p$ with $0_{p}$ being its zero element.\nWe use a canonical identification $ T_{p} \\mathcal{E}\\cong \\mathcal{E}$ for a vector space $ \\mathcal{E} $ and $p \\in \\mathcal{E}$.\nA vector field is a map $V \\colon \\mathcal{M} \\to T \\mathcal{M}$ with $V(p) \\in T_{p} \\mathcal{M}$, where $T \\mathcal{M}:=\\bigcup_{p \\in \\mathcal{M}} T_p \\mathcal{M}$ is the tangent bundle. \n$\\mathfrak{X}(\\mathcal{M})$ denotes the set of all smooth vector fields defined on $\\mathcal{M}$.\nFurthermore, $\\mathcal{M}$ is a Riemannian manifold if it is equipped with a Riemannian metric, that is, a choice of inner product $\\langle\\cdot, \\cdot\\rangle_p \\colon T_p \\mathcal{M} \\times T_p \\mathcal{M} \\to \\mathbb{R} $ for each tangent space at $p$ on $\\mathcal{M}$ such that for all $V, W \\in \\mathfrak{X}(\\mathcal{M})$, the map\n\\begin{equation}\\label{eq:RiemannianMetric}\n\tp \\mapsto\\langle V(p), W(p)\\rangle_p\n\\end{equation}\nis a smooth function from $\\mathcal{M}$ to $\\mathbb{R}$.\nRiemannian metric induces the norm $\\|\\xi\\|_p:=\\sqrt{\\langle\\xi, \\xi\\rangle_p}$ for $\\xi \\in T_p \\mathcal{M}$.\nWe often omit the subscript $p$ if it is clear from the context.\nThroughout this paper, we assume that all the manifolds involved are \\emph{connected} and \\emph{complete}.\nGiven a curve segment on $\\mathcal{M}$, $c:[a, b] \\rightarrow \\mathcal{M}$, the length of $c$ is defined as \n$\nL(c):=\\int_a^b \\| \\dot{c}(t)\\|_{c(t)} \\mathrm{d} t,\n$\nwhere $\\dot{c}(t) \\in T_{c(t)} \\mathcal{M}$ is the velocity vector of $c$ at $t$.\nSince $\\mathcal{M}$ is connected, there exists a curve segment connecting any pair of points $p,q \\in \\mathcal{M}$.\nIndeed, $\\mathcal{M}$ is a metric space under the Riemannian distance $d(p, q):=\\inf_{c} L(c) $ where the infimum is taken over all curve segments joining $p $ to $q $ \\cite[Thm. 2.55]{lee2018}.\nFor two manifolds $\\mathcal{M}_1, \\mathcal{M}_2$ and a smooth map $F \\colon \\mathcal{M}_1 \\to \\mathcal{M}_2$, the differential of $F$ at $p \\in \\mathcal{M}_1$ is a linear operator denoted as $\\mathcal{D} F(p) \\colon T_p \\mathcal{M}_1 \\to T_{F(p)} \\mathcal{M}_2$.\nLet $\\mathfrak{F}(\\mathcal{M})$ be the set of all smooth \\emph{scalar fields} (or say real-valued functions) $f\\colon \\mathcal{M} \\to \\mathbb{R}$.\nThe Riemannian gradient of $f$ at $p$, $\\operatorname{grad} f(p)$, is defined as the unique element of $T_p \\mathcal{M}$ that satisfies $\\langle \\xi, \\operatorname{grad} f(p)\\rangle_p = \\mathcal{D} f(p)[\\xi]$ for all $\\xi \\in T_p \\mathcal{M}$, where $\\mathcal{D} f(p) \\colon T_p \\mathcal{M} \\to T_{f(p)} \\mathbb{R} \\cong \\mathbb{R}$.\nNote that for any $f \\in \\mathfrak{F}(\\mathcal{M})$, the gradient vector field $x \\mapsto \\operatorname{grad} f(x)$ is a smooth vector field on $\\mathcal{M}$, i.e., $\\operatorname{grad} f \\in \\mathfrak{X}(\\mathcal{M})$.\nA retraction $\\mathrm{R} \\colon T \\mathcal{M} \\to \\mathcal{M}$ is a smooth map such that\n$\\mathrm{R}_{p}\\left(0_{p}\\right)=p$ and $\\mathcal{D} \\mathrm{R}_{p}\\left(0_{p}\\right) =\\operatorname{id}_{T_p \\mathcal{M}}$, i.e., the identity map on $T_{p} \\mathcal{M}$, where $\\mathrm{R}_{p}$ is the restriction of $\\mathrm{R}$ to $T_{p} \\mathcal{M}$ and $\\mathcal{D} \\mathrm{R}_{p}\\left(0_{p}\\right) \\colon T_{0_p} (T_p \\mathcal{M}) \\cong T_p \\mathcal{M} \\to T_{p} \\mathcal{M}$ is the differential of $\\mathrm{R}_{p}$ at $0_{p}$.\nOne theoretically perfect type of retraction is the exponential map, denoted as $\\operatorname{Exp}$. Since $\\mathcal{M}$ is complete, the exponential map is well-defined on the whole tangent bundle.\nLet $\\operatorname{Exp}_p\\colon T_p \\mathcal{M} \\rightarrow \\mathcal{M}$ be the exponential map at $p$; then $t \\mapsto \\operatorname{Exp}_p(t \\xi)$ is the unique geodesic that passes through $p$ with velocity $\\xi \\in T_p \\mathcal{M}$ when $t=0$.\n\n\\subsection{Riemannian Newton Method}\\label{subsec:RiemannianNewton}\n\nThe Newton method is a powerful tool for finding the zeros of nonlinear functions in the Euclidean setting.\nThe generalized Newton method has been studied in the Riemannian setting; it aims to find a \\emph{singularity} for the vector field $F \\in \\mathfrak{X}(\\mathcal{M})$, i.e., a point $p \\in \\mathcal{M}$ such that,\n\\begin{equation}\\label{eq:Singularity}\n\tF(p)=0_{p} \\in T_p \\mathcal{M}.\n\\end{equation}\nLet $\\nabla$ be the Levi-Civita connection on $\\mathcal{M}$, i.e., the unique symmetric connection compatible with the Riemannian metric.\nThe covariant derivative $\\nabla F$ assigns each point $p \\in \\mathcal{M}$ a linear operator $\\nabla F(p)\\colon T_{p}\\mathcal{M}\\to T_{p}\\mathcal{M}$. \nIn particular, the Riemannian Hessian of $f \\in \\mathfrak{F}(\\mathcal{M})$ at $p$ is a self-adjoint operator on $T_{p} \\mathcal{M}$, defined as $\\operatorname{Hess} f(p):=\\nabla (\\operatorname{grad}f)(p).$\n","start":9491,"end":14443},{"raw_tex":"In this section, we will give a comprehensive interpretation of the Riemannian interior point method.\nFollowing common usage in the interior-point literature, big letters denote the associated diagonal matrix, e.g., $Z =\\operatorname{diag}(z_{1}, \\ldots, z_{m})$ with $z \\in \\mathbb{R}^{m}$. $e$ denotes the all-ones vector, and $0$ stands for zero vector/matrix with proper dimensions.\n\n\\subsection{KKT Vector Field and Its Covariant Derivative}\n\nThe Lagrangian function of (\\ref{RCOP}) is\n\\begin{equation*}\n\t\\mathcal{L}(x, y, z):=f(x)+\\sum_{i=1}^l y_i h_i(x)+\\sum_{i=1}^m z_i g_i(x),\n\\end{equation*}\nwhere $y \\in \\mathbb{R}^l$ and $z \\in \\mathbb{R}^m$ are Lagrange multipliers corresponding to the equality and\ninequality constraints, respectively.\nWith respect to the variable $x$, $\\mathcal{L}(\\cdot, y, z)$ is a real-valued function defined on $\\mathcal{M}$, and its Riemannian gradient is\n\\begin{equation*}\n\\operatorname{grad}_{x} \\mathcal{L}(x, y, z)=\\operatorname{grad} f(x) + \\sum_{i =1 }^{l} y_{i} \\operatorname{grad} h_{i}(x) + \\sum_{i =1 }^{m} z_{i} \\operatorname{grad} g_{i}(x),\n\\end{equation*}\nwhere $\\{\\operatorname{grad} h_{i}(x)\\}_{i=1}^{l}$ and $ \\{\\operatorname{grad} g_{i}(x)\\}_{i=1}^{m}$ are the gradients of the component functions of $h$ and $g$. \nThe Riemannian KKT conditions (e.g., see \\cite[Definition 2.3]{liu2020simple} or \\cite{yang2014optimality}) for (\\ref{RCOP}) are given by\n\\begin{equation}\\label{eq:KKTConditions}\n\t\\operatorname{grad}_{x} \\mathcal{L}(x, y, z) =0_{x}; \\;\n\th(x) =0 ,\n\tg(x) \\leq 0 , \n\tz  \\geq 0; \\;\n\tZ g(x) =0.\n\\end{equation}\nThe above conditions can be written in terms of slack variables $\\mathbb{R}^{m} \\ni s:=-g(x)$, as\n\\begin{equation}\\label{eq:KKTVectorField}\n\tF(w):=\\left(\\begin{array}{l}\n\t\t\\operatorname{grad}_{x} \\mathcal{L}(x, y, z) \\\\\n\t\t h(x) \\\\\n\t\tg(x)+s \\\\\n\t\tZ S e\n\t\\end{array}\\right)=0_{w}\n\t=\\left(\\begin{array}{l}\n\t\t0_{x} \\\\\n\t\t0\\\\\n\t\t0\\\\\n\t\t0\n\t\\end{array}\\right) \\in T_w \\mathcal{N},\n\\end{equation}\nand $(z,s) \\geq 0,$\nwhere $w:=(x, y, z,s) \\in \\mathcal{N}:=\\mathcal{M}\\times \\mathbb{R}^{l} \\times \\mathbb{R}^{m} \\times \\mathbb{R}^{m}$.\nHere, we have generated a vector field $F$ in (\\ref{eq:KKTVectorField}) on the product manifold $\\mathcal{N}$, i.e.,\nfor any $w\\in \\mathcal{N}$, \n$\nF(w) \\in\nT_{w} \\mathcal{N} \\cong T_{x} \\mathcal{M} \\times \\mathbb{R}^{l} \\times \\mathbb{R}^{m} \\times \\mathbb{R}^{m}.\n$\nWe will call $F \\colon \\mathcal{N} \\to T\\mathcal{N} $ in (\\ref{eq:KKTVectorField}) above the \\emph{KKT vector field} for (\\ref{RCOP}).\nNote that for $\\xi=\\left(\\xi_x, \\xi_y, \\xi_z, \\xi_s\\right) \\in T_{w} \\mathcal{N} $,\n\\begin{equation}\\label{eq:barR}\n\t\\bar{\\mathrm{R}}_{w}(\\xi):=(\\mathrm{R}_{x}(\\xi_{x}), y+\\xi_{y}, z+\\xi_{z}, s+\\xi_{s})\n\\end{equation}\nis a well-defined retraction on the product manifold $\\mathcal{N}$ as long as $\\mathrm{R}$ is a retraction on $\\mathcal{M}$.\n\nNow, we aim to find a singularity of the KKT vector field $F$ on $\\mathcal{N}$ under some nonnegative constraints $(z,s) \\geq 0$.\nIf the Riemannian Newton method is to be applied to (\\ref{eq:KKTVectorField}), we must formulate the covariant derivative of $F$ at an arbitrary $w \\in \\mathcal{N}$. Before that, we need some new symbols.\nFixing a point $x \\in \\mathcal{M}$, we can define two linear operators $\\mathcal{H}_{x}\\colon \\mathbb{R}^{l} \\rightarrow T_{x} \\mathcal{M}$ and $\\mathcal{G}_{x}\\colon \\mathbb{R}^{m} \\rightarrow T_{x} \\mathcal{M}$ by\n\\begin{equation}\\label{defn:HxGx}\n\t\\mathcal{H}_x [v]:=\\sum_{i=1}^l v_i \\operatorname{grad} h_i(x), \\quad \\mathcal{G}_x [u]:=\\sum_{i=1}^m u_i \\operatorname{grad} g_i(x),\n\\end{equation}\nrespectively.\nThen, for $\\mathcal{H}_{x}$, its adjoint operator $\\mathcal{H}_x^*\\colon T_x \\mathcal{M} \\rightarrow \\mathbb{R}^l$ is given by\n\\begin{equation*}\n\t\\mathcal{H}_x^* [\\xi]=\\left(\\left\\langle\\operatorname{grad} h_1(x), \\xi\\right\\rangle_x, \\cdots,\\left\\langle\\operatorname{grad} h_l(x), \\xi\\right\\rangle_x\\right)^T.\n\\end{equation*}\nAlso, $\\mathcal{G}_x^*\\colon T_x \\mathcal{M} \\rightarrow \\mathbb{R}^m$ with $\\mathcal{G}_x^* [\\xi]=\\left(\\left\\langle\\operatorname{grad} g_1(x), \\xi\\right\\rangle_x, \\cdots,\\left\\langle\\operatorname{grad} g_m(x), \\xi\\right\\rangle_x\\right)^T$. \n\nNow, using the solutions of Exercises 5.4 and 5.13 on covariant derivatives of vector fields on product manifolds in monograph \\cite{boumal2022intromanifolds}, the following results can be easily derived.\nGiven KKT vector field $F $ in (\\ref{eq:KKTVectorField}) and $w \\in \\mathcal{N}$, the covariant derivative of $F$ at $w$ is the operator $\\nabla F(w)\\colon T_{w} \\mathcal{N} \\to T_{w} \\mathcal{N}$ given by\n\\begin{equation}\\label{eq:CovariantDerivativeKKTVectorField}\n\t\\nabla F(w) [\\Delta w] =\n\t\\left(\\begin{array}{l}\n\t\t\\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t\t+\\mathcal{H}_{x} [\\Delta y]\n\t\t+\\mathcal{G}_{x} [\\Delta z]\\\\\n\t\t\\mathcal{H}_{x}^{*}[\\Delta x]\\\\\n\t\t\\mathcal{G}_{x}^{*}[\\Delta x]+\\Delta s\\\\\n\t\tZ \\Delta s + S \\Delta z\n\t\\end{array}\\right),\n\\end{equation}\nwhere $\\Delta w=(\\Delta x, \\Delta y, \\Delta z, \\Delta s) \\in T_{w} \\mathcal{N}$ and $\\operatorname{Hess}_{x} \\mathcal{L}(w)$ is the Riemannian Hessian of real-valued function $\\mathcal{L}(\\cdot, y, z)$.\nIndeed,\n$\\operatorname{Hess}_{x} \\mathcal{L}(w)\\colon T_{x}\\mathcal{M} \\to T_{x}\\mathcal{M}$ satisfies\n\\begin{equation}\\label{eq:HessianLagrange}\n\t\\operatorname{Hess}_{x} \\mathcal{L}(w) =\n\t\\operatorname{Hess} f(x)\n\t+\\sum_{i=1}^{l} y_{i} \\operatorname{Hess} h_{i}(x)\n\t+\\sum_{i=1}^{m} z_{i} \\operatorname{Hess} g_{i}(x),\n\\end{equation}\nwhere $\\{\\operatorname{Hess} h_{i}(x)\\}_{i=1}^{l}$ and $\\{\\operatorname{Hess} g_{i}(x)\\}_{i=1}^{m}$ are Hessians of the component functions of $h$ and $g$.\n","start":15475,"end":21108},{"raw_tex":"Next, we will discuss an important result about the covariant derivative of the KKT vector field given in (\\ref{eq:CovariantDerivativeKKTVectorField}).\nLet $\\mathbb{A}(x):=\\left\\{i \\mid g_{i}(x)=0\\right\\}$ denote the active set at $x \\in \\mathcal{M}$.\nA prior study on Riemannian optimality conditions \\cite{yang2014optimality,bergmann2019intrinsic} showed that the following assumptions are meaningful.\nWe call them the \\emph{standard Riemannian assumptions} for (\\ref{RCOP}).\nNote that the $x^{*}$ and $w^*$ in \\ref{B3}-\\ref{B5} all refer to those in \\ref{B1}.\n\n\n\\setlist[assumptions,1]{label=\\textbf{(A\\arabic*)}}\n\\begin{assumptions}[leftmargin=9mm]\n\t\\item {Existence.} There exists $w^*=(x^{*}, y^{*}, z^{*}, s^{*})$ satisfying the Riemannian KKT conditions (\\ref{eq:KKTConditions}). Here, we introduce the slack variables $s^{*}:=-g(x^{*})$.\n\t\\label{B1}\t\n%\t\\item {Smoothness of $ C^{2} $.} The functions $f, g,$ and $h$ are $ C^{2} $ on $\\mathcal{M}$.\n%\t\\label{B2}\t\n\t\\item {Linear Independence Constraint Qualification (LICQ).} \\\\ The set $\\left\\{\\operatorname{grad} h_{i}(x^{*})\\right\\}_{i=1}^{l} \\bigcup \\left\\{\\operatorname{grad} g_{i}(x^{*})\\right\\}_{i \\in \\mathbb{A}(x^{*})}$ is linearly independent in $ T_{x^{*}}\\mathcal{M}$.\n\t\\label{B3}\t\n\t\\item {Strict complementarity.} $(z^{*})_{i}>0$ if $g_{i}(x^{*})=0$ for all $i=1, \\cdots,m$.\n\t\\label{B4}\t\n\t\\item {Second-order sufficiency.}\n\t$\\left \\langle\\operatorname{Hess}_{x} \\mathcal{L}(w^{*})[\\xi], \\xi\\right\\rangle>0$ for all nonzero $\\xi \\in T_{x^{*}}\\mathcal{M}$ \\\\ satisfying $\\left\\langle\\xi, \\operatorname{grad} h_{i}(x^{*})\\right\\rangle=0 $ for $i=1,\\dots,l,$ and\n\t$\\left\\langle\\xi, \\operatorname{grad} g_{i}(x^{*})\\right\\rangle=0$ for $ i \\in \\mathbb{A}(x^{*})$. \n\t\\label{B5}\n\\end{assumptions}\n","start":21159,"end":22919},{"raw_tex":"Applying the Riemannian Newton method directly to the KKT vector field $F: \\mathcal{N} \\rightarrow T \\mathcal{N}$ results in the following Newton equation (see (\\ref{eq:StandardNewtonEquation}) without iteration count $k$) at each iteration:\n\\begin{equation}\\label{eq:OriginalNewtonEquation}\n\t\\nabla F(w) [\\Delta w]+F(w)=0.\n\\end{equation}\nAs with the usual interior point method in the Euclidean setting, once the iterates reach the boundary of the feasible region, they are forced to stick to it \\cite[P6]{wright1997primal}.\nFor the iterates to maintain a sufficient distance from the boundary, we introduce a perturbed complementary equation with some barrier parameter $\\mu>0$ and define the \\textit{perturbed KKT vector field}:\n\\begin{equation}\\label{eq:perturbedKKTvectorfield}\n\tF_{\\mu}(w):=F(w)-\\mu \\hat{e}, \\text { and } \\hat{e}:=\\hat{e}(w):=\\left(0_x, 0,0, e\\right).\n\\end{equation}\nNotice that the perturbation term $\\hat{e}$, indeed, is a special vector field on $\\mathcal{N}$, not a constant, because $0_x$ is essentially dependent on $w$ and/or $x$.\nIn fact, the covariant derivative of the perturbed KKT vector field is the same as that of the original. From the linearity of the connection $\\nabla$, for any $w \\in \\mathcal{N}$ and any $\\mu>0$, we have \n\\begin{equation}\\label{eq:NabulaperturbedKKTvectorfield}\n\t\\nabla F_{\\mu}(w)=\\nabla F(w)-\\mu \\nabla \\hat{e}(w)=\\nabla F(w),\n\\end{equation}\nwhere the last equity comes from $\\nabla \\hat{e}(w) [\\Delta w]=\\left(0_x, 0,0,0\\right)$ for all $\\Delta w \\in T_w \\mathcal{N}$.\nApplying the Riemannian Newton method to perturbed KKT vector field $F_{\\mu}(w)$ yields the \\emph{perturbed Newton equation}: $\\nabla F_{\\mu}(w) [\\Delta w]+F_{\\mu}(w)=0$. \nFrom (\\ref{eq:perturbedKKTvectorfield}) and (\\ref{eq:NabulaperturbedKKTvectorfield}), this equation is equivalent to \n\\begin{equation}\n\t\\nabla F(w) [\\Delta w]+F(w)=\\mu \\hat{e},\n\\end{equation}\nwhich reduces to the ordinary Newton equation (\\ref{eq:OriginalNewtonEquation}) as $\\mu \\to 0$.\nAt this point, we can describe a prototype of the Riemannian Interior Point Method (RIPM) in Algorithm \\ref{algo:prototype_algo}.\n\n\\begin{algorithm}%[H]\n\t\\caption{Prototype Algorithm of RIPM for (\\ref{RCOP})}\n\t\\label{algo:prototype_algo}\n\t\\SetAlgoLined\n\t\n\t\\KwIn{An initial point $w_0=\\left(x_0, y_0, z_0, s_0\\right) \\in \\mathcal{N}$ with $\\left(z_0, s_0\\right)>0$ and a retraction $\\mathrm{R}$ on $\\mathcal{M}$. $\\hat{\\gamma} \\in(0,1)$, $\\mu_0>0$.}\n\t\\KwOut{Sequence $\\left\\{w_k\\right\\} \\subset \\mathcal{N}$ such that $\\left\\{w_k\\right\\} \\rightarrow w^*$ and $w^*$ satisfies the KKT conditions (\\ref{eq:KKTConditions}).} % \n\t\n\tSet $k \\to 0$\\;\n\t\n\t\\While{Stopping criterion not satisfied}{\n\t\t1. Obtain $\\Delta w_{k}=(\\Delta x_{k}, \\Delta y_{k}, \\Delta z_{k},\\Delta s_{k}) \\in T_{w_k} \\mathcal{N}$ by solving the perturbed Newton equation on $T_{w_k} \\mathcal{N}$:\n\t\t\\begin{equation}\\label{eq:prototype}\n\t\t\t\\nabla F(w_{k}) [\\Delta w_{k}]=-F(w_{k})+\\mu_{k} \\hat{e};\n\t\t\\end{equation}\n\t\n\t\t2. Choose $ \\gamma_{k} \\in [\\hat{\\gamma},1]$ and compute the step size $\\alpha_{k}$ defined by\n\t\t\\begin{equation}\\label{rule:StepSizeA}\n\t\t\t\\min \\left\\{1, \\gamma_{k} \\min _{i}\\left\\{-\\frac{(s_{k})_{i}}{(\\Delta s_{k})_{i}} \\mid(\\Delta s_{k})_{i}<0\\right\\},\\gamma_{k} \\min _{i}\\left\\{-\\frac{(z_{k})_{i}}{(\\Delta z_{k})_{i}} \\mid(\\Delta z_{k})_{i}<0\\right\\}\\right\\};\n\t\t\\end{equation}\n\t\n\t\t3. Compute the next point as\n\t\t$w_{k+1}:=\\bar{\\mathrm{R}}_{w_k}\\left(\\alpha_k \\Delta w_k\\right)$, see (\\ref{eq:barR})\\;\n\t\t\n\t\t4. Choose $0<\\mu_{k+1}<\\mu_k$\\;\n\t\t\n\t\t5. $k \\to k+1$\\;\n\t}\n\\end{algorithm}\n\nIn the step 2 of Algorithm \\ref{algo:prototype_algo}, we just wish to compute a step size $0<\\alpha_k \\leq 1$ to ensure new point $w_{k+1}$ with $\\left(z_{k+1},s_{k+1}\\right)>0$. \nThere are many schemes to achieve this purpose. \nThe scheme in (\\ref{rule:StepSizeA}) is simple but sufficient to guarantee the local convergence, as will be proved in Section \\ref{sec:local}.\n","start":27451,"end":31382},{"raw_tex":"Let us consider Algorithm \\ref{algo:prototype_algo} and omit the iteration count $k$.\nGiven the current point $w \\in \\mathcal{N}$ with $(z, s) > 0$, for the KKT vector field $F(w)$ in (\\ref{eq:KKTVectorField}), we denote its components by $F_x,F_y,F_z,F_s$ in top-to-bottom order, namely,\n$F_x:=\\operatorname{grad}_x \\mathcal{L}(w),F_y:=h(x),F_z:=g(x)+s,F_s:=Z S e.$\nBy using these symbols and the formulation of $\\nabla F(w)$ in (\\ref{eq:CovariantDerivativeKKTVectorField}), the \\emph{full} (perturbed) Newton equation (\\ref{eq:prototype}) defined on $T_w \\mathcal{N} \\cong T_x \\mathcal{M} \\times \\mathbb{R}^l \\times \\mathbb{R}^m \\times \\mathbb{R}^m$ is expanded as:\n\\begin{equation}\\label{eq:FullNewtonEquation}\n\t\\left(\\begin{array}{l}\n\t\t\\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t\t+\\mathcal{H}_{x} [\\Delta y]\n\t\t+\\mathcal{G}_{x} [\\Delta z]\\\\\n\t\t\\mathcal{H}_{x}^{*}[\\Delta x]\\\\\n\t\t\\mathcal{G}_{x}^{*}[\\Delta x]+\\Delta s\\\\\n\t\tZ \\Delta s + S \\Delta z\n\t\\end{array}\\right)\n\t=\n\t\\left(\\begin{array}{l}\n\t\t-F_x\\\\\n\t\t-F_y\\\\\n\t\t-F_z\\\\\n\t\t-F_s+\\mu e\n\t\\end{array}\\right).\n\\end{equation}\n","start":32122,"end":33200},{"raw_tex":"\\begin{remark}\n\tFor a retraction $\\mathrm{R}$ on $\\mathcal{M}$ and $x \\in \\mathcal{M}$,\n\tby $\\mathcal{D} \\mathrm{R}_{x}\\left(0_{x}\\right)=\\mathrm{id}_{T_{x} \\mathcal{M}}$ and the inverse function theorem, there exists a neighborhood $V$ of $0_{x}$ in $T_{x} \\mathcal{M}$ such that $ \\mathrm{R}_{x} $ is a diffeomorphism on $V$; thus,\n\t$\\mathrm{R}_{x}^{-1}(y)$ is well defined for all $ y \\in \\mathcal{M} $ sufficiently close to $ x $.\n\tIn this case, $\\mathrm{R}_{x}(V) \\subset \\mathcal{M}$ is called a retractive neighborhood of $x$.\n\tFurthermore, the existence of a \\emph{totally} retractive neighborhood \\cite[Theorem 2]{zhu2020riemannian} shows that for any $\\bar{x} \\in \\mathcal{M} $ there is a neighborhood $W$ of $\\bar{x}$ such that\n\t$\\mathrm{R}_{x}^{-1}(y)$ is well defined for all $x, y \\in W$.\n\tIn what follows, we will suppose that an appropriate neighborhood has been chosen by default for the well-definedness of $\\mathrm{R}_x^{-1}(y)$.\n\\end{remark}\n","start":41737,"end":42699},{"raw_tex":"Define the Whitney sum\n$\nT \\mathcal{M} \\oplus T \\mathcal{M}:=\\left\\{(\\eta, \\xi): \\eta, \\xi \\in T_x \\mathcal{M}, x \\in \\mathcal{M} \\right\\}.\n$\nA smooth map\n$\n\\mathrm{T} \\colon T \\mathcal{M} \\oplus T \\mathcal{M} \\to T \\mathcal{M}: (\\eta, \\xi) \\mapsto \\mathrm{T}_{\\eta}(\\xi),\n$\nis called a \\emph{vector transport} on $\\mathcal{M}$ if there exists an associated retraction $\\mathrm{R}$ on $\\mathcal{M}$ such that $\\mathrm{T}$ satisfies the following properties for all $x \\in \\mathcal{M}$:\n\\begin{enumerate}\n\t\\item Associated retraction. $\\mathrm{T}_{\\eta}\\left(\\xi\\right) \\in T_{\\mathrm{R}_{x}\\left(\\eta\\right)} \\mathcal{M}$\n\tfor all $\\eta, \\xi \\in T_x \\mathcal{M}$.\t\n\t\\item Consistency. $\\mathrm{T}_{0_{x}}\\left(\\xi\\right)=\\xi$ for all $\\xi \\in T_{x} \\mathcal{M}$.\t\n\t\\item Linearity.\n\t$\\mathrm{T}_{\\eta}\\left(a \\xi+b \\zeta\\right)=a \\mathrm{T}_{\\eta}\\left(\\xi\\right)+b \\mathrm{T}_{\\eta}\\left(\\zeta\\right)$ for all $a, b \\in \\mathbb{R}$ and $\\eta, \\xi, \\zeta \\in T_{x} \\mathcal{M}$.\n\\end{enumerate}\nThus, fixing any $\\eta \\in T_x \\mathcal{M}$, the map\n$\n\\mathrm{T}_\\eta \\colon T_x \\mathcal{M} \\to T_{\\mathrm{R}_x(\\eta)} \\mathcal{M}: \\xi \\mapsto \\mathrm{T}_\\eta(\\xi),\n$\nis a linear operator.\nAdditionally, $\\mathrm{T}$ is \\emph{isometric} if \n$\n\\langle \\mathrm{T}_{\\eta} (\\xi), \\mathrm{T}_{\\eta} (\\zeta)\\rangle =\\langle\\xi, \\zeta\\rangle\n$\nholds, for all $x \\in \\mathcal{M}$ and all $\\xi, \\zeta, \\eta \\in T_{x} \\mathcal{M}$.\nIn other words, for any $\\eta \\in T_x \\mathcal{M}$, the adjoint and the inverse of $\\mathrm{T}_{\\eta}$ coincide, i.e., $ \\mathrm{T}_{\\eta}^{*} =  \\mathrm{T}_{\\eta}^{-1}$.\nThere are two important classes of vector transport as follows. Let $\\mathrm{R}$ be a retraction on $\\mathcal{M}$. \n\\begin{enumerate}[(1)]\n\t\\item \n\tThe \\emph{differentiated retraction} defined by\n\t\\begin{equation}\\label{eq:DifferentiatedRetraction}\n\t\t\\mathrm{T}_{\\eta}(\\xi):=\\mathcal{D} \\mathrm{R}_{x}(\\eta)[\\xi],  \\; \\eta, \\xi \\in T_{x} \\mathcal{M}, x \\in \\mathcal{M}, \n\t\\end{equation}\n\tis a valid vector transport \\cite[equation (8.6)]{absil2009optimization}.\t\n\t\\item\n\tGiven a smooth curve $\\gamma \\colon [0,1] \\to \\mathcal{M}$ and $t_0, t_1 \\in [0,1]$, the \\emph{parallel transport} from the tangent space at $\\gamma(t_0)$ to the tangent space at $\\gamma(t_1)$ along $\\gamma,$\n\tis a linear operator\n\t$ \\mathrm{P}_\\gamma^{t_1 \\to t_0} \\colon T_{\\gamma\\left(t_0\\right)} \\mathcal{M} \\rightarrow T_{\\gamma\\left(t_1\\right)} \\mathcal{M}$ defined by $\\mathrm{P}_\\gamma^{t_1 \\to t_0}(\\xi)=Z\\left(t_1\\right)$, where $Z$ is the unique parallel vector field such that $Z\\left(t_0\\right)=\\xi$.\n\tThen, for any $x \\in \\mathcal{M}, \\eta \\in T_x \\mathcal{M}$, then\n\t\\begin{equation}\\label{eq:ParallelTransport}\n\t\t\\mathrm{T}_{\\eta}\\left(\\xi\\right):=\\mathrm{P}_\\gamma^{1 \\to 0} (\\xi)\n\t\\end{equation}\n\tis a valid vector transport \\cite[equation (8.2)]{absil2009optimization}, where $\\mathrm{P}_\\gamma$ denotes the parallel transport along the curve $t \\mapsto \\gamma(t):=\\mathrm{R}_x\\left(t \\eta\\right)$. We often omit the superscript $^{1 \\to 0}$ if it is clear from the context.\n\tIn particular, parallel transport is isometric.\n\\end{enumerate}\n","start":42754,"end":45874},{"raw_tex":"Going one degree higher, let us now discuss the Lipschitz continuity of Hessian operators.\nThroughout this paper, for a linear operator $\\mathcal{A}: \\mathcal{E} \\rightarrow \\mathcal{E}^{\\prime}$ between two finite-dimensional normed vector spaces $\\mathcal{E}$ and $\\mathcal{E}^{\\prime}$, the (operator) norm of $\\mathcal{A}$ is defined by $\\|\\mathcal{A}\\|:=\\sup \\left\\{\\|\\mathcal{A} v\\|_{\\mathcal{E}^{\\prime}} : v \\in \\mathcal{E},\\|v\\|_{\\mathcal{E}}=1\\right.$, or, $\\left.\\|v\\|_{\\mathcal{E}} \\leq 1\\right\\}.$\nThe inverse of $\\mathrm{T}_{\\eta}$ is needed in the following definitions, so we can assume that vector transport $\\mathrm{T}$ is isometric, e.g., parallel transport in (\\ref{eq:ParallelTransport}). In fact, there are many ways to construct isometric vector transports; see \\cite[Section 2.3]{huang2015riemannian}.\n\\begin{definition}[{\\cite[Assumption 3]{huang2015riemannian}}]\n\t\\label{defn:TwiceLipschitzContinuouslyDifferentiable}\n\tA function $f\\colon \\mathcal{M} \\rightarrow \\mathbb{R}$ is \\emph{twice Lipschitz continuously differentiable with respect to $\\mathrm{T}$ in $\\mathcal{U} \\subset \\mathcal{M}$} if it is twice differentiable and there exists a constant $\\kappa>0$ such that, for all $x, y \\in \\mathcal{U}$, \n\t$\n\t\\|\\operatorname{Hess} f(y)-\\mathrm{T}_{\\eta} \\operatorname{Hess} f(x) \\mathrm{T}_{\\eta}^{-1}\\| \\leq \\kappa d(x, y),\n\t$\n\twhere $\\eta=\\mathrm{R}_{x}^{-1} y$.\n\\end{definition}\n\\begin{lemma}[{\\cite[Lemma 4]{huang2015riemannian}}]\n\t\\label{lem:TwiceLipschitzContinuouslyDifferentiable}\n\tIf $f\\colon \\mathcal{M} \\rightarrow \\mathbb{R}$ is $ C^3 $, then for any $\\bar{x} \\in \\mathcal{M}$ and any isometric vector transport $\\mathrm{T}$, there exists a neighborhood $\\mathcal{U}$ of $\\bar{x}$ such that $f$ is twice Lipschitz continuously differentiable with respect to $\\mathrm{T}$ in $\\mathcal{U}$.\n\\end{lemma}\n\nIf the operator, $\\operatorname{Hess}f(x)$, above is replaced by a general covariant derivative $\\nabla F (x)$, we can get the next results in a similar way.\nLemma \\ref{lem:LipCountNabulaF} can be proven in the same way as Lemma \\ref{lem:TwiceLipschitzContinuouslyDifferentiable}.\n\\begin{definition}\n\tGiven a vector field $F$ on $\\mathcal{M}$.\n\tThe map $x \\mapsto \\nabla F(x)$ is \\emph{Lipschitz continuous with respect to $\\mathrm{T}$ in $\\mathcal{U} \\subset \\mathcal{M}$} if there exists a constant $\\kappa>0$ such that, for all $x, y \\in \\mathcal{U}$, it holds that\n\t$\n\t\\| \\nabla F(y)\n\t-\n\t\\mathrm{T}_{\\eta} \\nabla F(x) \\mathrm{T}_{\\eta}^{-1}\n\t\\| \\leq \\kappa d(x, y),\n\t$\n\twhere $\\eta=\\mathrm{R}_{x}^{-1} y$.\n\\end{definition}\n\\begin{lemma}\\label{lem:LipCountNabulaF}\n\tIf $F$ is a $C^{2}$ vector field, then\n\tfor any $\\bar{x} \\in \\mathcal{M}$ and any isometric vector transport $\\mathrm{T}$, there exists a neighborhood $\\mathcal{U}$ of $\\bar{x}$ such that the map $x \\mapsto \\nabla F(x)$ is Lipschitz continuous with respect to $\\mathrm{T}$ in $\\mathcal{U}$.\n\\end{lemma}","start":48102,"end":51016},{"raw_tex":"Notice that in the previous subsection on the definitions of Lipschitz continuity, we used $\\|\\eta\\|$ with $\\eta=\\mathrm{R}_x^{-1} y$ or $d(x, y)$ to denote the upper bound on the right-hand side. The next lemma shows that the two are not essentially different. When $\\mathcal{M}=\\mathbb{R}^{n}$, both reduce to $\\| x-y \\|$.\n\n\\begin{lemma}[{\\cite[Lemma 2]{huang2015riemannian}}]\\label{lem:distR1}\n\tLet $\\mathcal{M}$ be a Riemannian manifold with a retraction $\\mathrm{R}$ and let $\\bar{x} \\in \\mathcal{M}$. \n\tThen,\n\t\\begin{enumerate}[(i)]\n\t\t \\item there exist $a_{0}, a_{1}, \\delta_{a_{0}, a_{1}}>0$ such that for all $x$ in a sufficiently small neighborhood of $\\bar{x}$ and all $\\xi, \\eta \\in T_{x} \\mathcal{M}$ with $\\|\\xi\\|, \\|\\eta\\|  \\leq \\delta_{a_{0}, a_{1}}$,\n\t\t one has\n\t\t $\n\t\t a_{0}\\|\\xi-\\eta\\| \\leq d(\\mathrm{R}_{x}(\\eta), \\mathrm{R}_{x}(\\xi)) \\leq a_{1}\\|\\xi-\\eta\\|.\n\t\t $\t\n\t\t In particular,\n\t\t $\n\t\t a_{0}\\|\\xi\\| \\leq d(x, \\mathrm{R}_{x}(\\xi)) \\leq a_{1}\\|\\xi\\|\n\t\t $\n\t\t when $\\eta=0$;\n\t\t \n\t\t \\item\n\t\t there exist $a_{0},a_{1}>0$ such that for all $x$ in a sufficiently small neighborhood of $\\bar{x}$, one has\n\t\t $\n\t\t a_{0}\\|\\xi\\| \\leq d(x,\\bar{x}) \\leq a_{1}\\|\\xi\\| \\text { where } \\xi=\\mathrm{R}_{\\bar{x}}^{-1}(x). \n\t\t $\n\t\\end{enumerate}\n\\end{lemma}\n\nThe next lemma is the fundamental theorem of calculus in the Riemannian case.\n\\begin{lemma}[{\\cite[Lemma 8]{huang2015riemannian}}]\\label{lem:FundamentalTheoremOfCalculusRiemannian}\n\tLet $F$ be a $C^{1}$ vector field and $\\bar{x} \\in \\mathcal{M}$. \n\tThen there exist a neighborhood $\\mathcal{U}$ of $\\bar{x}$ and a constant $c_{1} \\geq 0$ such that for all $x, y \\in \\mathcal{U}$,\n\t\\begin{equation*}\n\t\\left\\| \\mathrm{P}_{\\gamma}^{0 \\to 1} [F(y)]-F(x)-\\int_{0}^{1} \\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} [\\eta] \\mathrm{d} t \\right\\| \n\t\\leq c_{1} \\|\\eta\\|^{2},\n\t\\end{equation*}\n\twhere $\\eta=\\mathrm{R}_{x}^{-1}(y)$ and $\\mathrm{P}_{\\gamma}$ is the parallel transport along the curve $\\gamma(t):=\\mathrm{R}_{x}\\left(t \\eta\\right)$.\n\tMoreover, if $\\mathrm{R}=\\operatorname{Exp}$, then indeed $c_1=0$ above (see \\cite[equation (2.4)]{ferreira2002kantorovich}).\n\\end{lemma}\n","start":51049,"end":53221},{"raw_tex":"\\begin{lemma}[{\\cite[Lemma 3.2]{fernandes2017superlinear}}]\\label{lem:nonsingular}\n\tGiven a vector field $F$ on $\\mathcal{M}$.\n\tIf the map $p \\mapsto \\nabla F(p)$ is continuous at $p^{*}$ and $\\nabla F(p^{*})$ is nonsingular, \n\tthen there exist a neighborhood $\\mathcal{U}$ of $p^{*}$ and a constant $\\Xi > 0$ such that, for all $p \\in \\mathcal{U}$, $\\nabla F(p)$ is nonsingular and $\\left\\|\\nabla F(p)^{-1}\\right\\| \\leq \\Xi$.\n\\end{lemma}\n\\begin{lemma}[{\\cite[Lemma 14.5]{gallivan2012riemannian}}]\\label{lem:dist_zero}\n\tLet $F$ be a  $C^2$ vector field on $\\mathcal{M}$ and $p^{*} \\in \\mathcal{M}$. If $ F(p^{*})=0 $ and $\\nabla F(p^{*}) $ is nonsingular, then there exist a neighborhood $\\mathcal{U}$ of $p^{*}$ and constants $c_{3}, c_{4} > 0$ such that, for all $p \\in \\mathcal{U}$, $ c_{3} d(p, p^{*}) \\leq \\|F(p)\\| \\leq c_{4} d(p, p^{*}).$\n\\end{lemma}\n","start":55894,"end":56751},{"raw_tex":"Here, for any two nonnegative sequences $\\left\\{u_{k}\\right\\}$ and $\\left\\{v_{k}\\right\\}$, we write $u_{k}=O(v_{k})$ if there is a constant $M >0$ such that\n$u_{k}\\leq M v_{k}$\nfor all sufficiently large $k$; and we write\n$u_{k}=o(v_{k})$\nif $v_{k}>0$ and the sequence of ratios $\\{u_{k} / v_{k}\\} $ approaches zero.\n","start":57284,"end":57601},{"raw_tex":"\\begin{algorithm}%[H]\n\t\\caption{Perturbed Damped Riemannian Newton Method for (\\ref{eq:Singularity})}\n\t\\label{algo:PerturbedDampedNewtonMethod}\n\t\\SetAlgoLined\n\t\n\t\\KwIn{A vector field $F \\in \\mathfrak{X}(\\mathcal{M})$, an initial point $p_0 \\in \\mathcal{M}$ and a retraction $\\mathrm{R}$ on $\\mathcal{M}$. Set $\\mu_0>0$.}\n\t\\KwOut{Sequence $\\{p_k\\} \\subset \\mathcal{M}$ such that $\\left\\{p_k\\right\\} \\rightarrow p^*$ and $F(p^{*})=0_{p^{*}}$.}\n\t\n\tSet $k \\to 0$\\;\n\t\n\t\\While{Stopping criterion not satisfied}{\n\t\t1. Obtain $\\xi_k \\in T_{p_k}\\mathcal{M}$ by solving the perturbed Newton equation:\n\t\t\\begin{equation}\n\t\t\t\\nabla F(p_{k}) [\\xi_k] = - F(p_{k}) + \\mu_{k} \\hat{e};\n\t\t\\end{equation}\n\t\n\t\t2. Choose a (damped) step size $ 0<\\alpha_{k} \\leq 1$\\;\n\t\t\n\t\t3. Compute the next point as $p_{k+1}:=\\mathrm{R}_{p_{k}} (\\alpha_{k} \\xi_k)$\\;\n\t\t\n\t\t4. Choose $0<\\mu_{k+1}<\\mu_k$\\;\n\t\t\n\t\t5. $k \\to k + 1$\\;\n\t}\n\\end{algorithm}\n","start":58004,"end":58915},{"raw_tex":"\\begin{assumptions}[leftmargin=9mm]\n\t\\item There exists $p^{*} \\in \\mathcal{M}$ such that $F(p^{*})=0_{p^{*}}$;\n\t\\label{A1}\n\t\\item The covariant derivative $\\nabla F (p^{*})$ is nonsingular;\n\t\\label{A2}\n%\t\\item The map $p \\mapsto \\nabla F(p) $ is Lipschitz continuous with respect to on a neighborhood of $p^{*}$;\n%\t\\label{A3}\n\t\\item The vector field $F$ is $C^2$.\n\t\\label{A3}\n\\end{assumptions}\n","start":59463,"end":59858},{"raw_tex":"\\begin{lemma}\\label{lem:gamma_alpha}\n\tConsider the Algorithm \\ref{algo:prototype_algo} for solving the problem (\\ref{RCOP}).  \n\tLet \\ref{B1} and \\ref{B4} hold at some $w^{*}=(x^{*}, y^{*}, z^{*}, s^{*})$ and $ \\alpha_{k} $ be as in (\\ref{rule:StepSizeA}). \n\tDefine a constant,\n\t\\begin{equation*}\n\t\\Pi := 2 \\max \\left\\{\\max _{i}\\left\\{\n\t1/(s^{*})_{i}\n\t\\mid (s^{*})_{i}>0\\right\\}, \\max _{i}\\left\\{\n\t1/(z^{*})_{i}\n\t\\mid (z^{*})_{i}>0\\right\\}\\right\\}.\n\t\\end{equation*}\n\tFor $\\gamma_{k} \\in(0,1)$, if\n\t$\n\t\\Pi \\left\\| \\Delta w_{k}\\right\\| \\leq \\gamma_{k},\n\t$\n\tthen\n\t$\n\t0  \\leq 1-\\alpha_{k} \\leq\n\t(1-\\gamma_{k})+\\Pi \\left\\| \\Delta w_{k}\\right\\|.\n\t$\n\\end{lemma}","start":65558,"end":66211}]},"statement":{"raw_tex":"\\begin{theorem}[Local convergence of prototype Algorithm \\ref{algo:prototype_algo}]\\label{thm:LocalConvergence}\n\tConsider the Algorithm \\ref{algo:prototype_algo} for solving the problem (\\ref{RCOP}).  \n\tLet \\ref{B1}-\\ref{B5} hold at some $w^{*}$.\n\tChoose parameters $\\mu_k, \\gamma_k$ as follows;\n\tthen there exists a constant $\\delta>0$ such that, for all\n\t$\n\tw_{0} \\in\\mathcal{N}\n\t$\n\twith $d(w_{0} , w^{*})<\\delta,$\n\tthe sequence $\\left\\{w_{k}\\right\\}$ is well defined. Furthermore,\n\t\\begin{enumerate}[(1)]\n\t\t\\item if we choose $\\mu_{k}=o(\\left\\|F(w_{k})\\right\\|)$ and $\\gamma_{k} \\to 1$, then $w_k \\to w^*$ superlinearly;\t\n\t\t\\item if we choose $\\mu_{k}=O(\\left\\|F(w_{k})\\right\\|^{2})$\n\t\tand $1-\\gamma_{k}=O(\\left\\|F(w_{k})\\right\\|)$, then $w_k \\to w^*$ quadratically.\n\t\\end{enumerate}\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}[Local convergence of prototype Algorithm \\ref{algo:prototype_algo}]\\label{thm:LocalConvergence}\n\tConsider the Algorithm \\ref{algo:prototype_algo} for solving the problem (\\ref{RCOP}).  \n\tLet \\ref{B1}-\\ref{B5} hold at some $w^{*}$.\n\tChoose parameters $\\mu_k, \\gamma_k$ as follows;\n\tthen there exists a constant $\\delta>0$ such that, for all\n\t$\n\tw_{0} \\in\\mathcal{N}\n\t$\n\twith $d(w_{0} , w^{*})<\\delta,$\n\tthe sequence $\\left\\{w_{k}\\right\\}$ is well defined. Furthermore,\n\t\\begin{enumerate}[(1)]\n\t\t\\item if we choose $\\mu_{k}=o(\\left\\|F(w_{k})\\right\\|)$ and $\\gamma_{k} \\to 1$, then $w_k \\to w^*$ superlinearly;\t\n\t\t\\item if we choose $\\mu_{k}=O(\\left\\|F(w_{k})\\right\\|^{2})$\n\t\tand $1-\\gamma_{k}=O(\\left\\|F(w_{k})\\right\\|)$, then $w_k \\to w^*$ quadratically.\n\t\\end{enumerate}\n\\end{theorem}","start":66819,"end":67619}]},"proof":{"raw_tex":"\\begin{theorem}\\label{prop:non}\n\tIf the standard Riemannian assumptions \\ref{B1}-\\ref{B5} hold at some point $w^{*}$,\n\tthen the operator $\\nabla F(w^{*})$ in (\\ref{eq:CovariantDerivativeKKTVectorField}) is nonsingular.\n\\end{theorem}\n\n\\begin{proof}\nThis proof omits all the asterisks of the variables. Define $ \\mathbb{E}:=\\{1,\\dots,l\\}$ and $\\mathbb{I}:=\\{1,\\dots,m\\} $. Take some $w=(x, y, z, s) \\in \\mathcal{N} $ satisfying \\ref{B1}-\\ref{B5}, then we have $ s_{i} = -g_{i}(x)$ and $z_{i} s_{i} = 0$ for all $ i \\in \\mathbb{I}$. For short, let $\\mathbb{A}:=\\mathbb{A}(x) \\subset \\mathbb{I} $. \nSuppose that $\\nabla {F}(w) [\\Delta w] =0$ for some $\\Delta w = (\\Delta x, \\Delta y, \\Delta z, \\Delta s) \\in T_{w} \\mathcal{N} \\cong T_x \\mathcal{M} \\times \\mathbb{R}^l \\times \\mathbb{R}^m \\times \\mathbb{R}^m$.\n$\\Delta y_i$ denotes the components of the vector $\\Delta y$, as do $\\Delta z_i$, $\\Delta s_i$.\nTo prove its nonsingularity, we will show that $ \\Delta w=0$.\nExpanding the equation $\\nabla {F}(w) [\\Delta w] =0$ gives\n\\begin{equation}\\label{13}\n\t\\left\\{\n\t\\begin{aligned}\n\t\t0&= \\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n\t\t+\\sum_{i\\in \\mathbb{I}} \\Delta z_{i} \\operatorname{grad} g_{i}(x),\\\\\n\t\t0&=\\left\\langle\\operatorname{grad}h_{i}(x), \\Delta x\\right\\rangle, \\text { for all } i\\in \\mathbb{E},\\\\\n\t\t0&=\\left\\langle\\operatorname{grad} g_{i}(x), \\Delta x\\right\\rangle + \\Delta s_{i}, \\text { for all } i\\in \\mathbb{I}, \\\\\n\t\t0&=z_{i} \\Delta s_{i} + s_{i} \\Delta z_{i}, \\text { for all } i \\in \\mathbb{I}. \n\t\\end{aligned}\n\t\\right.\n\\end{equation}\nStrict complementarity \\ref{B4} and the last equalities above imply that $ \\Delta s_{i}=0 $\nfor all $ i \\in \\mathbb{A}$ and $ \\Delta z_{i}=0 $\nfor $ i \\in  \\mathbb{I} \\setminus \\mathbb{A}$.\nSubstituting those values into the system (\\ref{13}) reduces it to\n\\begin{equation}\\label{sing1}\n\t\\left\\{\n\t\\begin{aligned}\n\t\t0&= \\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n\t\t+\\sum_{i\\in \\mathbb{A}} \\Delta z_{i} \\operatorname{grad} g_{i}(x), \\\\\n\t\t0&=\\left\\langle\\operatorname{grad}h_{i}(x), \\Delta x\\right\\rangle, \\text { for all } i\\in \\mathbb{E},\\\\\n\t\t0&=\\left\\langle\\operatorname{grad}g_{i}(x), \\Delta x\\right\\rangle, \\text { for all } i\\in \\mathbb{A},\n\t\\end{aligned}\n\t\\right.\n\\end{equation}\nand\n$\n\\Delta s_{i}=-\\left\\langle\\operatorname{grad} g_{i}(x), \\Delta x\\right\\rangle \\text { for all } i\\in \\mathbb{I} \\setminus \\mathbb{A}.\n$\nIt follows from system (\\ref{sing1}) that\n\\begin{align*}\n\t0\n\t&=  \\langle \\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n\t+\\sum_{i\\in  \\mathbb{A}} \\Delta z_{i} \\operatorname{grad} g_{i}(x) , \\Delta x \\rangle \\\\\n\t&= \\left\\langle\\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x], \\Delta x\\right\\rangle\n\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\left\\langle\\operatorname{grad}h_{i}(x), \\Delta x\\right\\rangle\n\t+\\sum_{i\\in  \\mathbb{A}} \\Delta z_{i} \\left\\langle\\operatorname{grad}g_{i}(x), \\Delta x\\right\\rangle \\\\\n\t&= \\left\\langle\\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x], \\Delta x\\right\\rangle.\n\\end{align*}\nThus, from second-order sufficiency \\ref{B5}, $ \\Delta x$ must be zero elements.\nAnd then $\\Delta s_{i}=0 \\text { for all } i\\in \\mathbb{I} \\setminus  \\mathbb{A}.$\nNext, substituting $ \\Delta x=0$ into the first equation in (\\ref{sing1}) yields\n$\n0=\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n+\\sum_{i\\in  \\mathbb{A}} \\Delta z_{i} \\operatorname{grad} g_{i}(x).\n$\nThe LICQ \\ref{B3} implies that the coefficients $\\Delta y_{i}$ for $i\\in \\mathbb{E}$ and $ \\Delta z_{i} $ for $ i\\in  \\mathbb{A} $ must be zero. \nThis completes the proof.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:R1est}\n\tLet $F$ be a $ C^2 $ vector field and $\\bar{x}\\in\\mathcal{M}$. Then there exist a neighborhood $\\mathcal{U}$ of $\\bar{x}$ and a constant $c_{2}>0$ such that for all $x \\in \\mathcal{U}$,\n\t\\begin{equation*}\n\t\\left\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} [F(x)]-F(\\bar{x})-\\nabla F(\\bar{x}) [\\eta] \n\t\\right\\|\n\t \\leq c_{2} d^{2}(\\bar{x} , x),\n\t\\end{equation*}\n\twhere $\\eta=\\mathrm{R} _{\\bar{x}}^{-1} x$ and $\\mathrm{P}_{\\gamma}$ is the parallel transport along the curve $\\gamma(t):=\\mathrm{R}_{\\bar{x}}(t \\eta)$.\n\\end{lemma}\n\n\\begin{proof} \nLet $LHS:=\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} [F(x)]-F(\\bar{x})-\\nabla F(\\bar{x}) [\\eta] \\|.$ It follows that\n\\begin{equation*}\n\t\t\\begin{aligned}\n\t\t\tLHS\n\t\t\t& \\leq\n\t\t\t\\left\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} [F(x)]-F(\\bar{x})\n\t\t\t-\\int_{0}^{1} \\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} [\\eta] \\mathrm{d} t  \\right\\| \\\\\n\t\t\t& \\quad +\n\t\t\t\\left\\|\\int_{0}^{1} \\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} [\\eta] \\mathrm{d} t -\\nabla F(\\bar{x}) [\\eta] \\right\\| \\\\\n\t\t\t&\\leq \n\t\t\tc_{1} \\|\\eta\\|^{2} +\n\t\t\t\\Big\\|\n\t\t\t\\int_{0}^{1}\\left(\\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} -\\nabla F(\\bar{x})\\right)\n\t\t\t[\\eta] \\mathrm{d} t\\Big\\|.\n\t\t\t\\text{ (by Lemma \\ref{lem:FundamentalTheoremOfCalculusRiemannian})}\n\t\t\\end{aligned}\n\\end{equation*}\nLet\n$\\theta := \\left\\| \\int_{0}^{1}\\left(\\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} -\\nabla F(\\bar{x})\\right)\n[\\eta] \\mathrm{d} t \\right\\|$.\nNote that\n\\begin{align*}\n\t\\theta \n\t&\\leq \n\t\\int_{0}^{1} \n\t\\left\\|\n\t\\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} -\\nabla F(\\bar{x})\n\t\\right\\|\n\t\\left\\| \\eta \\right\\|\n\t\\mathrm{d} t \\\\\n\t&\\leq  \n\t\\int_{0}^{1} \n\tc_{0} d(\\bar{x}, \\mathrm{R}_{\\bar{x}}(t \\eta))\n\t\\left\\| \\eta \\right\\|\n\t\\mathrm{d} t\t\\text{ (by Lemma \\ref{lem:LipCountNabulaF})} \\\\\n\t&\\leq   \n\t\\int_{0}^{1} \n\tc_{0} a_{1}t \\|\\eta\\|\n\t\\left\\| \\eta \\right\\|\n\t\\mathrm{d} t \t\n\t= \n\t\\frac{1}{2} \n\tc_{0} a_{1} \\|\\eta\\|^{2}. \t\n\t\\text{ (by (i) of Lemma \\ref{lem:distR1})}\n\\end{align*}\nCombining the above results yields\n$$\nLHS\n\\leq (c_{1}+ \\frac{1}{2} \nc_{0} a_{1})\n\\|\\eta\\|^{2}\n\\leq  \n(c_{1}+ \\frac{1}{2} \nc_{0} a_{1})  / a_{0}^{2}\nd^{2}(\\bar{x} , x),\n$$\nwhere the last inequality comes from (ii) of Lemma \\ref{lem:distR1}. Letting $c_{2}:= (c_{1}+ \\frac{1}{2} \nc_{0} a_{1})  / a_{0}^{2} $ completes the proof.\n\\end{proof}\n\n\\begin{proposition}[Local convergence of Algorithm \\ref{algo:PerturbedDampedNewtonMethod}]\\label{prop:PD_Newton}\n\tConsider the perturbed damped Riemannian Newton method\n\tdescribed in Algorithm \\ref{algo:PerturbedDampedNewtonMethod} for the singularity problem (\\ref{eq:Singularity}). \n\tLet \\ref{A1}-\\ref{A3} hold.\n\tChoose parameters $ \\mu_{k}, \\alpha_{k}$ as follows; then there exists a constant $\\delta>0$ such that for all $p_{0}\\in \\mathcal{M}$ with\n\t$\n\td(p_{0} , p^{*})<\\delta,\n\t$\n\tthe sequence $\\left\\{p_{k}\\right\\}$ is well defined.\n\tFurthermore,\n\t\\begin{enumerate}[(1)]\n\t\t\\item if we choose $\\mu_{k}=o(\\|F(p_{k})\\|)$ and $\\alpha_{k} \\to 1$, then $p_{k} \\to p^{*}$ superlinearly;\t\t\n\t\t\\item if we choose $\\mu_{k}=O(\\|F(p_{k})\\|^{2})$ and $1-\\alpha_{k}=O(\\|F(p_{k})\\|)$, then $p_{k} \\to p^{*}$ quadratically.\n\t\\end{enumerate}\n\\end{proposition}\n\n\\begin{proof}\nBy \\ref{A2}-\\ref{A3}, Lemma \\ref{lem:LipCountNabulaF} and Lemma \\ref{lem:nonsingular}, we can let $p_{k} $ be sufficiently close to $p^{*}$ such that $\\nabla F(p_{k})$ is nonsingular, and\n$\\left\\|\\nabla F(p_{k})^{-1}\\right\\| \\leq \\Xi$ for some constant $\\Xi$.\nThen, the next iterate point,\n\\begin{equation*}\np_{k+1}:=\\mathrm{R}_{p_{k}}[\\alpha_{k} \\nabla F(p_{k})^{-1}(-F(p_{k})+\\mu_{k} \\hat{e})],\n\\end{equation*}\nis well defined in Algorithm \\ref{algo:PerturbedDampedNewtonMethod}, and it follows from $ p^{*}=\\mathrm{R}_{p_{k}}(\\eta) $ with $ \\eta := \\mathrm{R}_{p_{k}}^{-1} p^{*} $ and (i) of Lemma \\ref{lem:distR1} that\n\\begin{align}\n\td(p_{k+1} , p^{*})\n\t&\\leq\n\ta_{1}\n\t\\| \\eta - \\alpha_{k} \\nabla F(p_{k})^{-1}(-F(p_{k})+\\mu_{k} \\hat{e}) \\| \\notag \\\\\n\t&=\n\ta_{1}  \n\t\\|\\eta + \\alpha_{k} \\nabla F(p_{k})^{-1}(F(p_{k})-\\mu_{k} \\hat{e})\\|. \\label{eq:1114}\n\\end{align}\nLet \n$r_{k} := \\eta + \\alpha_{k} \\nabla F(p_{k})^{-1}(F(p_{k})-\\mu_{k} \\hat{e})$.\nAlgebraic manipulations\nshow that\n\\begin{equation*}\nr_{k}=\n(1-\\alpha_{k}) \\eta  +\n\\alpha_{k} \\nabla F(p_{k})^{-1} \n[\\nabla F(p_{k}) \\eta + F(p_{k})-\\mathrm{P}_{\\gamma}^{0 \\to 1} F(p^{*})-\\mu_{k} \\hat{e}],\n\\end{equation*}\nwhere $\\mathrm{P}_{\\gamma}$ is the parallel transport along the curve $\\gamma(t)=\\mathrm{R}_{p_{k}}(t \\eta)$ and $F(p^{*})=0.$\nThus, using $\\left\\|\\eta\\right\\| \\leq d(p_{k}, p^{*})/a_{0}$ from (ii) of Lemma \\ref{lem:distR1} and Lemma \\ref{lem:R1est}, we have\n\\begin{align*}\n\t\\| r_{k}\\|&\\leq\n\t(1-\\alpha_{k}) \\| \\eta\\| \\notag \n\t+ \n\t\\alpha_{k} \\|\\nabla F(p_{k})^{-1} \\|\n\t\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} F(p^{*}) - F(p_{k}) -\n\t\\nabla F(p_{k}) \\eta \\| \\\\\n\t& \\quad + \\alpha_{k} \\|\\nabla F(p_{k})^{-1} \\| \\|\\hat{e}\\| \\mu_{k} \\\\ \n\t&\\leq (1-\\alpha_{k})d(p_{k} , p^{*}) /a_{0} +\n\t\\alpha_{k} \\|\\nabla F(p_{k})^{-1} \\| \n\tc_{2} d^{2}(p_{k} , p^{*})\\\\\n\t& \\quad\n\t+ \\alpha_{k}\\|\\nabla F(p_{k})^{-1} \\|  \\|\\hat{e}\\| \\mu_{k}\\\\\n\t&\\leq (1-\\alpha_{k})d(p_{k} , p^{*}) /a_{0} +\n\t\\Xi \n\tc_{2} d^{2}(p_{k} , p^{*})\n\t+ \\Xi  \\|\\hat{e}\\| \\mu_{k}.\n\\end{align*}\nCombining the above with (\\ref{eq:1114}), we conclude that\n\\begin{equation}\\label{77}\n\td(p_{k+1} , p^{*}) \\leq \\kappa_{1}(1-\\alpha_{k}) d(p_{k} , p^{*}) \n\t+ \\kappa_{2} d^{2}(p_{k} , p^{*})\n\t+ \\kappa_{3} \\mu_{k}\n\\end{equation}\nfor some positive constants $ \\kappa_{1},\\kappa_{2},\\kappa_{3}$.\nOn the other hand, by Lemma \\ref{lem:dist_zero}, we have\n\\begin{equation}\\label{eq:LipFbigoh}\n\t\\left\\|F(p_{k})\\right\\|=O(d(p_{k}, p^{*})).\n\\end{equation}\nIn what follows, we prove assertions (1) and (2).\n\n\\textbf{(1)} \nSuppose that $\\alpha_{k} \\to 1$ and $\\mu_{k}=o(\\|F(p_{k})\\|)$, which together with (\\ref{eq:LipFbigoh}) imply $\\mu_{k}=o(d(p_{k}, p^{*}))$.\nBy (\\ref{77}), we have\n\\begin{equation}\\label{eq:superlinear}\n\t\\frac{d(p_{k+1} , p^{*})}{d(p_{k} , p^{*})}\\leq\n\t\\kappa_{1} (1-\\alpha_{k})\n\t+\\kappa_{2} d(p_{k} , p^{*})\n\t+\\kappa_{3}\\frac{\\mu_{k}}{d(p_{k} , p^{*})}.\n\\end{equation}\nWe can take $\\delta$ sufficiently small and $ k $ sufficiently large, if necessary, to conclude that\n$\nd(p_{k+1} , p^{*})<\\frac{1}{2}d(p_{k} , p^{*}) <\\delta.\n$\nThus, $p_{k+1} \\in B_{\\delta}(p^{*})$, the open ball of radius $\\delta$ centered at $p^{*}$ on $\\mathcal{M}$. \nBy induction, it is easy to show that the sequence $\\left\\{p_{k}\\right\\}$ is well defined and converges to $p^{*}$. \nTaking the limit of both sides of (\\ref{eq:superlinear}) proves superlinear convergence.\n\n\n\\textbf{(2)} \nAgain, we start from (\\ref{77}) and rewrite it as:\n\\begin{align}\\label{eq:bigoh}\n\td(p_{k+1} , p^{*}) = (1-\\alpha_{k}) O(d(p_{k} , p^{*}) )\n\t+O(d^{2}(p_{k} , p^{*}) )\n\t+O(\\mu_{k}).\n\\end{align}\nSuppose that $1-\\alpha_{k}=O(\\left\\|F(p_{k})\\right\\|)$\nand $\\mu_{k}=O(\\left\\|F(p_{k})\\right\\|^{2})$.\nUsing (\\ref{eq:LipFbigoh}), the above reduces to\n$\nd(p_{k+1} , p^{*}) = O(d^{2}(p_{k} , p^{*})).\n$\nThis implies that there exists a constant $\\nu$ such that $d(p_{k+1} , p^{*}) \\leq \\nu d^{2}(p_{k} , p^{*})$, and hence,\n$\nd(p_{k+1} , p^{*}) \\leq \\nu d^{2}(p_{k} , p^{*}) \\leq \\nu \\delta^{2}<\\delta,\n$\nif $\\delta$ is sufficiently small.\nAgain, by induction, $\\left\\{p_{k}\\right\\}$ converges to $p^{*}$ quadratically.\n\\end{proof}\n\n\\begin{proof}\nWe only prove (2) because (1) can be proven in the same way.\nLet $w_{k} $ be such that  $d(w_{k} , w^{*})<\\delta$ for sufficiently small $\\delta$.\nFrom Proposition \\ref{prop:non}, \\ref{B1}-\\ref{B5} shows that the KKT vector field $F$ satisfies \\ref{A1}-\\ref{A3}; thus, the discussion in the proof of Proposition \\ref{prop:PD_Newton} applies to KKT vector field $F$ as well, simply by replacing the symbol $p$ with $w$.\nSince we choose $\\mu_{k}=O(\\|F(w_{k})\\|^{2})$, and $\\|F(w_{k})\\|=O(d(w_{k}, w^{*}))$ by (\\ref{eq:LipFbigoh}), we obtain\n$\n\\mu_{k}=O(d^{2}(w_{k}, w^{*})).\n$\nThus,\n\\begin{align*}\n\t \\left\\|\\Delta w_{k}\\right\\|  & =\n\t\\left\\|\n\t\\nabla F(w_{k})^{-1} (-F(w_{k})+\\mu_{k} \\hat{e})\n\t\\right\\| \\text{ (by (\\ref{eq:prototype}))}\\\\\n\t& \\leq \n\t\\Xi\n\t(\\left\\|F(w_{k})\\right\\|+\\mu_{k}\\|\\hat{e}\\|) \\text{ (by Lemma \\ref{lem:nonsingular})}\\\\\n\t& \\leq  O(\\|F(w_k)\\|)+O(\\mu_{k}) \\\\\n\t& =O(d(w_{k} , w^{*}))+O(d^{2}(w_{k} , w^{*}) )\n\t=O(d(w_{k} , w^{*})).\n\\end{align*}\nSince $\\delta$ is sufficiently small, from the above inequalities, the condition of Lemma \\ref{lem:gamma_alpha} is satisfied, i.e.,\n$\n\\left\\|\\Delta w_{k}\\right\\| \\leq \\hat{\\gamma} / \\Pi \\leq \\gamma_k / \\Pi\n$\nfor a constant $\\hat{\\gamma} \\in (0,1).$\nHence, by $1-\\gamma_k=O(\\left\\|F\\left(w_k\\right)\\right\\|)$, one has\n$\n1-\\alpha_{k} \\leq\n(1-\\gamma_{k})+\\Pi\\left\\|\\Delta w_{k}\\right\\| \n= (1-\\gamma_{k})+O(d(w_{k} , w^{*}))\n= O(d(w_{k} , w^{*})).\n$\nFinally, from (\\ref{eq:bigoh}),\nwe have\n$\nd(w_{k+1} , w^{*}) = (1-\\alpha_{k}) O(d(w_{k} , w^{*}) )\n+O(d^{2}(w_{k} , w^{*}) )\n+O(\\mu_{k}) \n=O(d^{2}(w_{k} , w^{*})).\n$\nThis completes the proof.\n\\end{proof}","units":[{"raw_tex":"\\begin{theorem}\\label{prop:non}\n\tIf the standard Riemannian assumptions \\ref{B1}-\\ref{B5} hold at some point $w^{*}$,\n\tthen the operator $\\nabla F(w^{*})$ in (\\ref{eq:CovariantDerivativeKKTVectorField}) is nonsingular.\n\\end{theorem}","start":23641,"end":23873},{"raw_tex":"\\begin{proof}\nThis proof omits all the asterisks of the variables. Define $ \\mathbb{E}:=\\{1,\\dots,l\\}$ and $\\mathbb{I}:=\\{1,\\dots,m\\} $. Take some $w=(x, y, z, s) \\in \\mathcal{N} $ satisfying \\ref{B1}-\\ref{B5}, then we have $ s_{i} = -g_{i}(x)$ and $z_{i} s_{i} = 0$ for all $ i \\in \\mathbb{I}$. For short, let $\\mathbb{A}:=\\mathbb{A}(x) \\subset \\mathbb{I} $. \nSuppose that $\\nabla {F}(w) [\\Delta w] =0$ for some $\\Delta w = (\\Delta x, \\Delta y, \\Delta z, \\Delta s) \\in T_{w} \\mathcal{N} \\cong T_x \\mathcal{M} \\times \\mathbb{R}^l \\times \\mathbb{R}^m \\times \\mathbb{R}^m$.\n$\\Delta y_i$ denotes the components of the vector $\\Delta y$, as do $\\Delta z_i$, $\\Delta s_i$.\nTo prove its nonsingularity, we will show that $ \\Delta w=0$.\nExpanding the equation $\\nabla {F}(w) [\\Delta w] =0$ gives\n\\begin{equation}\\label{13}\n\t\\left\\{\n\t\\begin{aligned}\n\t\t0&= \\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n\t\t+\\sum_{i\\in \\mathbb{I}} \\Delta z_{i} \\operatorname{grad} g_{i}(x),\\\\\n\t\t0&=\\left\\langle\\operatorname{grad}h_{i}(x), \\Delta x\\right\\rangle, \\text { for all } i\\in \\mathbb{E},\\\\\n\t\t0&=\\left\\langle\\operatorname{grad} g_{i}(x), \\Delta x\\right\\rangle + \\Delta s_{i}, \\text { for all } i\\in \\mathbb{I}, \\\\\n\t\t0&=z_{i} \\Delta s_{i} + s_{i} \\Delta z_{i}, \\text { for all } i \\in \\mathbb{I}. \n\t\\end{aligned}\n\t\\right.\n\\end{equation}\nStrict complementarity \\ref{B4} and the last equalities above imply that $ \\Delta s_{i}=0 $\nfor all $ i \\in \\mathbb{A}$ and $ \\Delta z_{i}=0 $\nfor $ i \\in  \\mathbb{I} \\setminus \\mathbb{A}$.\nSubstituting those values into the system (\\ref{13}) reduces it to\n\\begin{equation}\\label{sing1}\n\t\\left\\{\n\t\\begin{aligned}\n\t\t0&= \\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n\t\t+\\sum_{i\\in \\mathbb{A}} \\Delta z_{i} \\operatorname{grad} g_{i}(x), \\\\\n\t\t0&=\\left\\langle\\operatorname{grad}h_{i}(x), \\Delta x\\right\\rangle, \\text { for all } i\\in \\mathbb{E},\\\\\n\t\t0&=\\left\\langle\\operatorname{grad}g_{i}(x), \\Delta x\\right\\rangle, \\text { for all } i\\in \\mathbb{A},\n\t\\end{aligned}\n\t\\right.\n\\end{equation}\nand\n$\n\\Delta s_{i}=-\\left\\langle\\operatorname{grad} g_{i}(x), \\Delta x\\right\\rangle \\text { for all } i\\in \\mathbb{I} \\setminus \\mathbb{A}.\n$\nIt follows from system (\\ref{sing1}) that\n\\begin{align*}\n\t0\n\t&=  \\langle \\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n\t+\\sum_{i\\in  \\mathbb{A}} \\Delta z_{i} \\operatorname{grad} g_{i}(x) , \\Delta x \\rangle \\\\\n\t&= \\left\\langle\\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x], \\Delta x\\right\\rangle\n\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\left\\langle\\operatorname{grad}h_{i}(x), \\Delta x\\right\\rangle\n\t+\\sum_{i\\in  \\mathbb{A}} \\Delta z_{i} \\left\\langle\\operatorname{grad}g_{i}(x), \\Delta x\\right\\rangle \\\\\n\t&= \\left\\langle\\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x], \\Delta x\\right\\rangle.\n\\end{align*}\nThus, from second-order sufficiency \\ref{B5}, $ \\Delta x$ must be zero elements.\nAnd then $\\Delta s_{i}=0 \\text { for all } i\\in \\mathbb{I} \\setminus  \\mathbb{A}.$\nNext, substituting $ \\Delta x=0$ into the first equation in (\\ref{sing1}) yields\n$\n0=\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n+\\sum_{i\\in  \\mathbb{A}} \\Delta z_{i} \\operatorname{grad} g_{i}(x).\n$\nThe LICQ \\ref{B3} implies that the coefficients $\\Delta y_{i}$ for $i\\in \\mathbb{E}$ and $ \\Delta z_{i} $ for $ i\\in  \\mathbb{A} $ must be zero. \nThis completes the proof.\n\\end{proof}","start":23874,"end":27407},{"raw_tex":"\\begin{lemma}\\label{lem:R1est}\n\tLet $F$ be a $ C^2 $ vector field and $\\bar{x}\\in\\mathcal{M}$. Then there exist a neighborhood $\\mathcal{U}$ of $\\bar{x}$ and a constant $c_{2}>0$ such that for all $x \\in \\mathcal{U}$,\n\t\\begin{equation*}\n\t\\left\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} [F(x)]-F(\\bar{x})-\\nabla F(\\bar{x}) [\\eta] \n\t\\right\\|\n\t \\leq c_{2} d^{2}(\\bar{x} , x),\n\t\\end{equation*}\n\twhere $\\eta=\\mathrm{R} _{\\bar{x}}^{-1} x$ and $\\mathrm{P}_{\\gamma}$ is the parallel transport along the curve $\\gamma(t):=\\mathrm{R}_{\\bar{x}}(t \\eta)$.\n\\end{lemma}","start":53373,"end":53917},{"raw_tex":"\\begin{proof} \nLet $LHS:=\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} [F(x)]-F(\\bar{x})-\\nabla F(\\bar{x}) [\\eta] \\|.$ It follows that\n\\begin{equation*}\n\t\t\\begin{aligned}\n\t\t\tLHS\n\t\t\t& \\leq\n\t\t\t\\left\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} [F(x)]-F(\\bar{x})\n\t\t\t-\\int_{0}^{1} \\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} [\\eta] \\mathrm{d} t  \\right\\| \\\\\n\t\t\t& \\quad +\n\t\t\t\\left\\|\\int_{0}^{1} \\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} [\\eta] \\mathrm{d} t -\\nabla F(\\bar{x}) [\\eta] \\right\\| \\\\\n\t\t\t&\\leq \n\t\t\tc_{1} \\|\\eta\\|^{2} +\n\t\t\t\\Big\\|\n\t\t\t\\int_{0}^{1}\\left(\\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} -\\nabla F(\\bar{x})\\right)\n\t\t\t[\\eta] \\mathrm{d} t\\Big\\|.\n\t\t\t\\text{ (by Lemma \\ref{lem:FundamentalTheoremOfCalculusRiemannian})}\n\t\t\\end{aligned}\n\\end{equation*}\nLet\n$\\theta := \\left\\| \\int_{0}^{1}\\left(\\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} -\\nabla F(\\bar{x})\\right)\n[\\eta] \\mathrm{d} t \\right\\|$.\nNote that\n\\begin{align*}\n\t\\theta \n\t&\\leq \n\t\\int_{0}^{1} \n\t\\left\\|\n\t\\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} -\\nabla F(\\bar{x})\n\t\\right\\|\n\t\\left\\| \\eta \\right\\|\n\t\\mathrm{d} t \\\\\n\t&\\leq  \n\t\\int_{0}^{1} \n\tc_{0} d(\\bar{x}, \\mathrm{R}_{\\bar{x}}(t \\eta))\n\t\\left\\| \\eta \\right\\|\n\t\\mathrm{d} t\t\\text{ (by Lemma \\ref{lem:LipCountNabulaF})} \\\\\n\t&\\leq   \n\t\\int_{0}^{1} \n\tc_{0} a_{1}t \\|\\eta\\|\n\t\\left\\| \\eta \\right\\|\n\t\\mathrm{d} t \t\n\t= \n\t\\frac{1}{2} \n\tc_{0} a_{1} \\|\\eta\\|^{2}. \t\n\t\\text{ (by (i) of Lemma \\ref{lem:distR1})}\n\\end{align*}\nCombining the above results yields\n$$\nLHS\n\\leq (c_{1}+ \\frac{1}{2} \nc_{0} a_{1})\n\\|\\eta\\|^{2}\n\\leq  \n(c_{1}+ \\frac{1}{2} \nc_{0} a_{1})  / a_{0}^{2}\nd^{2}(\\bar{x} , x),\n$$\nwhere the last inequality comes from (ii) of Lemma \\ref{lem:distR1}. Letting $c_{2}:= (c_{1}+ \\frac{1}{2} \nc_{0} a_{1})  / a_{0}^{2} $ completes the proof.\n\\end{proof}","start":53918,"end":55838},{"raw_tex":"\\begin{proposition}[Local convergence of Algorithm \\ref{algo:PerturbedDampedNewtonMethod}]\\label{prop:PD_Newton}\n\tConsider the perturbed damped Riemannian Newton method\n\tdescribed in Algorithm \\ref{algo:PerturbedDampedNewtonMethod} for the singularity problem (\\ref{eq:Singularity}). \n\tLet \\ref{A1}-\\ref{A3} hold.\n\tChoose parameters $ \\mu_{k}, \\alpha_{k}$ as follows; then there exists a constant $\\delta>0$ such that for all $p_{0}\\in \\mathcal{M}$ with\n\t$\n\td(p_{0} , p^{*})<\\delta,\n\t$\n\tthe sequence $\\left\\{p_{k}\\right\\}$ is well defined.\n\tFurthermore,\n\t\\begin{enumerate}[(1)]\n\t\t\\item if we choose $\\mu_{k}=o(\\|F(p_{k})\\|)$ and $\\alpha_{k} \\to 1$, then $p_{k} \\to p^{*}$ superlinearly;\t\t\n\t\t\\item if we choose $\\mu_{k}=O(\\|F(p_{k})\\|^{2})$ and $1-\\alpha_{k}=O(\\|F(p_{k})\\|)$, then $p_{k} \\to p^{*}$ quadratically.\n\t\\end{enumerate}\n\\end{proposition}","start":60387,"end":61235},{"raw_tex":"\\begin{proof}\nBy \\ref{A2}-\\ref{A3}, Lemma \\ref{lem:LipCountNabulaF} and Lemma \\ref{lem:nonsingular}, we can let $p_{k} $ be sufficiently close to $p^{*}$ such that $\\nabla F(p_{k})$ is nonsingular, and\n$\\left\\|\\nabla F(p_{k})^{-1}\\right\\| \\leq \\Xi$ for some constant $\\Xi$.\nThen, the next iterate point,\n\\begin{equation*}\np_{k+1}:=\\mathrm{R}_{p_{k}}[\\alpha_{k} \\nabla F(p_{k})^{-1}(-F(p_{k})+\\mu_{k} \\hat{e})],\n\\end{equation*}\nis well defined in Algorithm \\ref{algo:PerturbedDampedNewtonMethod}, and it follows from $ p^{*}=\\mathrm{R}_{p_{k}}(\\eta) $ with $ \\eta := \\mathrm{R}_{p_{k}}^{-1} p^{*} $ and (i) of Lemma \\ref{lem:distR1} that\n\\begin{align}\n\td(p_{k+1} , p^{*})\n\t&\\leq\n\ta_{1}\n\t\\| \\eta - \\alpha_{k} \\nabla F(p_{k})^{-1}(-F(p_{k})+\\mu_{k} \\hat{e}) \\| \\notag \\\\\n\t&=\n\ta_{1}  \n\t\\|\\eta + \\alpha_{k} \\nabla F(p_{k})^{-1}(F(p_{k})-\\mu_{k} \\hat{e})\\|. \\label{eq:1114}\n\\end{align}\nLet \n$r_{k} := \\eta + \\alpha_{k} \\nabla F(p_{k})^{-1}(F(p_{k})-\\mu_{k} \\hat{e})$.\nAlgebraic manipulations\nshow that\n\\begin{equation*}\nr_{k}=\n(1-\\alpha_{k}) \\eta  +\n\\alpha_{k} \\nabla F(p_{k})^{-1} \n[\\nabla F(p_{k}) \\eta + F(p_{k})-\\mathrm{P}_{\\gamma}^{0 \\to 1} F(p^{*})-\\mu_{k} \\hat{e}],\n\\end{equation*}\nwhere $\\mathrm{P}_{\\gamma}$ is the parallel transport along the curve $\\gamma(t)=\\mathrm{R}_{p_{k}}(t \\eta)$ and $F(p^{*})=0.$\nThus, using $\\left\\|\\eta\\right\\| \\leq d(p_{k}, p^{*})/a_{0}$ from (ii) of Lemma \\ref{lem:distR1} and Lemma \\ref{lem:R1est}, we have\n\\begin{align*}\n\t\\| r_{k}\\|&\\leq\n\t(1-\\alpha_{k}) \\| \\eta\\| \\notag \n\t+ \n\t\\alpha_{k} \\|\\nabla F(p_{k})^{-1} \\|\n\t\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} F(p^{*}) - F(p_{k}) -\n\t\\nabla F(p_{k}) \\eta \\| \\\\\n\t& \\quad + \\alpha_{k} \\|\\nabla F(p_{k})^{-1} \\| \\|\\hat{e}\\| \\mu_{k} \\\\ \n\t&\\leq (1-\\alpha_{k})d(p_{k} , p^{*}) /a_{0} +\n\t\\alpha_{k} \\|\\nabla F(p_{k})^{-1} \\| \n\tc_{2} d^{2}(p_{k} , p^{*})\\\\\n\t& \\quad\n\t+ \\alpha_{k}\\|\\nabla F(p_{k})^{-1} \\|  \\|\\hat{e}\\| \\mu_{k}\\\\\n\t&\\leq (1-\\alpha_{k})d(p_{k} , p^{*}) /a_{0} +\n\t\\Xi \n\tc_{2} d^{2}(p_{k} , p^{*})\n\t+ \\Xi  \\|\\hat{e}\\| \\mu_{k}.\n\\end{align*}\nCombining the above with (\\ref{eq:1114}), we conclude that\n\\begin{equation}\\label{77}\n\td(p_{k+1} , p^{*}) \\leq \\kappa_{1}(1-\\alpha_{k}) d(p_{k} , p^{*}) \n\t+ \\kappa_{2} d^{2}(p_{k} , p^{*})\n\t+ \\kappa_{3} \\mu_{k}\n\\end{equation}\nfor some positive constants $ \\kappa_{1},\\kappa_{2},\\kappa_{3}$.\nOn the other hand, by Lemma \\ref{lem:dist_zero}, we have\n\\begin{equation}\\label{eq:LipFbigoh}\n\t\\left\\|F(p_{k})\\right\\|=O(d(p_{k}, p^{*})).\n\\end{equation}\nIn what follows, we prove assertions (1) and (2).\n\n\\textbf{(1)} \nSuppose that $\\alpha_{k} \\to 1$ and $\\mu_{k}=o(\\|F(p_{k})\\|)$, which together with (\\ref{eq:LipFbigoh}) imply $\\mu_{k}=o(d(p_{k}, p^{*}))$.\nBy (\\ref{77}), we have\n\\begin{equation}\\label{eq:superlinear}\n\t\\frac{d(p_{k+1} , p^{*})}{d(p_{k} , p^{*})}\\leq\n\t\\kappa_{1} (1-\\alpha_{k})\n\t+\\kappa_{2} d(p_{k} , p^{*})\n\t+\\kappa_{3}\\frac{\\mu_{k}}{d(p_{k} , p^{*})}.\n\\end{equation}\nWe can take $\\delta$ sufficiently small and $ k $ sufficiently large, if necessary, to conclude that\n$\nd(p_{k+1} , p^{*})<\\frac{1}{2}d(p_{k} , p^{*}) <\\delta.\n$\nThus, $p_{k+1} \\in B_{\\delta}(p^{*})$, the open ball of radius $\\delta$ centered at $p^{*}$ on $\\mathcal{M}$. \nBy induction, it is easy to show that the sequence $\\left\\{p_{k}\\right\\}$ is well defined and converges to $p^{*}$. \nTaking the limit of both sides of (\\ref{eq:superlinear}) proves superlinear convergence.\n\n\n\\textbf{(2)} \nAgain, we start from (\\ref{77}) and rewrite it as:\n\\begin{align}\\label{eq:bigoh}\n\td(p_{k+1} , p^{*}) = (1-\\alpha_{k}) O(d(p_{k} , p^{*}) )\n\t+O(d^{2}(p_{k} , p^{*}) )\n\t+O(\\mu_{k}).\n\\end{align}\nSuppose that $1-\\alpha_{k}=O(\\left\\|F(p_{k})\\right\\|)$\nand $\\mu_{k}=O(\\left\\|F(p_{k})\\right\\|^{2})$.\nUsing (\\ref{eq:LipFbigoh}), the above reduces to\n$\nd(p_{k+1} , p^{*}) = O(d^{2}(p_{k} , p^{*})).\n$\nThis implies that there exists a constant $\\nu$ such that $d(p_{k+1} , p^{*}) \\leq \\nu d^{2}(p_{k} , p^{*})$, and hence,\n$\nd(p_{k+1} , p^{*}) \\leq \\nu d^{2}(p_{k} , p^{*}) \\leq \\nu \\delta^{2}<\\delta,\n$\nif $\\delta$ is sufficiently small.\nAgain, by induction, $\\left\\{p_{k}\\right\\}$ converges to $p^{*}$ quadratically.\n\\end{proof}","start":61236,"end":65344},{"raw_tex":"\\begin{proof}\nWe only prove (2) because (1) can be proven in the same way.\nLet $w_{k} $ be such that  $d(w_{k} , w^{*})<\\delta$ for sufficiently small $\\delta$.\nFrom Proposition \\ref{prop:non}, \\ref{B1}-\\ref{B5} shows that the KKT vector field $F$ satisfies \\ref{A1}-\\ref{A3}; thus, the discussion in the proof of Proposition \\ref{prop:PD_Newton} applies to KKT vector field $F$ as well, simply by replacing the symbol $p$ with $w$.\nSince we choose $\\mu_{k}=O(\\|F(w_{k})\\|^{2})$, and $\\|F(w_{k})\\|=O(d(w_{k}, w^{*}))$ by (\\ref{eq:LipFbigoh}), we obtain\n$\n\\mu_{k}=O(d^{2}(w_{k}, w^{*})).\n$\nThus,\n\\begin{align*}\n\t \\left\\|\\Delta w_{k}\\right\\|  & =\n\t\\left\\|\n\t\\nabla F(w_{k})^{-1} (-F(w_{k})+\\mu_{k} \\hat{e})\n\t\\right\\| \\text{ (by (\\ref{eq:prototype}))}\\\\\n\t& \\leq \n\t\\Xi\n\t(\\left\\|F(w_{k})\\right\\|+\\mu_{k}\\|\\hat{e}\\|) \\text{ (by Lemma \\ref{lem:nonsingular})}\\\\\n\t& \\leq  O(\\|F(w_k)\\|)+O(\\mu_{k}) \\\\\n\t& =O(d(w_{k} , w^{*}))+O(d^{2}(w_{k} , w^{*}) )\n\t=O(d(w_{k} , w^{*})).\n\\end{align*}\nSince $\\delta$ is sufficiently small, from the above inequalities, the condition of Lemma \\ref{lem:gamma_alpha} is satisfied, i.e.,\n$\n\\left\\|\\Delta w_{k}\\right\\| \\leq \\hat{\\gamma} / \\Pi \\leq \\gamma_k / \\Pi\n$\nfor a constant $\\hat{\\gamma} \\in (0,1).$\nHence, by $1-\\gamma_k=O(\\left\\|F\\left(w_k\\right)\\right\\|)$, one has\n$\n1-\\alpha_{k} \\leq\n(1-\\gamma_{k})+\\Pi\\left\\|\\Delta w_{k}\\right\\| \n= (1-\\gamma_{k})+O(d(w_{k} , w^{*}))\n= O(d(w_{k} , w^{*})).\n$\nFinally, from (\\ref{eq:bigoh}),\nwe have\n$\nd(w_{k+1} , w^{*}) = (1-\\alpha_{k}) O(d(w_{k} , w^{*}) )\n+O(d^{2}(w_{k} , w^{*}) )\n+O(\\mu_{k}) \n=O(d^{2}(w_{k} , w^{*})).\n$\nThis completes the proof.\n\\end{proof}","start":67620,"end":69244}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["RCOP"],"declared_external":false,"source_only":false,"title":"流形约束优化问题","source_units":[{"raw_tex":"In this paper, we consider the following Riemannian Constrained Optimization Problem (RCOP):\n\\begin{equation}\\tag{RCOP} \\label{RCOP} \n\t\\begin{array}{ll}\n\t\t\\underset{x \\in \\mathcal{M}}{\\min} & f(x) \\\\\n\t\t\\text { s.t. } & h(x)=0, \\text{ and }  g(x) \\leq 0,\\\\\n\t\\end{array}\n\\end{equation}\nwhere $\\mathcal{M}$ is a connected, complete $d$-dimensional Riemannian manifold and $f \\colon \\mathcal{M}\\to \\mathbb{R}, h\\colon\\mathcal{M}\\to \\mathbb{R}^{l}$, and $g\\colon\\mathcal{M}\\to \\mathbb{R}^{m}$ are smooth functions.\n","start":3609,"end":4119}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":["eq:RiemannianMetric"],"declared_external":false,"source_only":false,"title":"切空间与黎曼度量","source_units":[{"raw_tex":"Let us briefly review some concepts from Riemannian geometry, following the notation of \\cite{boumal2022intromanifolds}.\n$\\mathcal{M}$ denotes a finite-dimensional smooth manifold.\nLet $p \\in \\mathcal{M}$ and $T_p \\mathcal{M}$ be the tangent space at $p$ with $0_{p}$ being its zero element.\nWe use a canonical identification $ T_{p} \\mathcal{E}\\cong \\mathcal{E}$ for a vector space $ \\mathcal{E} $ and $p \\in \\mathcal{E}$.\nA vector field is a map $V \\colon \\mathcal{M} \\to T \\mathcal{M}$ with $V(p) \\in T_{p} \\mathcal{M}$, where $T \\mathcal{M}:=\\bigcup_{p \\in \\mathcal{M}} T_p \\mathcal{M}$ is the tangent bundle. \n$\\mathfrak{X}(\\mathcal{M})$ denotes the set of all smooth vector fields defined on $\\mathcal{M}$.\nFurthermore, $\\mathcal{M}$ is a Riemannian manifold if it is equipped with a Riemannian metric, that is, a choice of inner product $\\langle\\cdot, \\cdot\\rangle_p \\colon T_p \\mathcal{M} \\times T_p \\mathcal{M} \\to \\mathbb{R} $ for each tangent space at $p$ on $\\mathcal{M}$ such that for all $V, W \\in \\mathfrak{X}(\\mathcal{M})$, the map\n\\begin{equation}\\label{eq:RiemannianMetric}\n\tp \\mapsto\\langle V(p), W(p)\\rangle_p\n\\end{equation}\nis a smooth function from $\\mathcal{M}$ to $\\mathbb{R}$.\nRiemannian metric induces the norm $\\|\\xi\\|_p:=\\sqrt{\\langle\\xi, \\xi\\rangle_p}$ for $\\xi \\in T_p \\mathcal{M}$.\nWe often omit the subscript $p$ if it is clear from the context.\nThroughout this paper, we assume that all the manifolds involved are \\emph{connected} and \\emph{complete}.\nGiven a curve segment on $\\mathcal{M}$, $c:[a, b] \\rightarrow \\mathcal{M}$, the length of $c$ is defined as \n$\nL(c):=\\int_a^b \\| \\dot{c}(t)\\|_{c(t)} \\mathrm{d} t,\n$\nwhere $\\dot{c}(t) \\in T_{c(t)} \\mathcal{M}$ is the velocity vector of $c$ at $t$.\nSince $\\mathcal{M}$ is connected, there exists a curve segment connecting any pair of points $p,q \\in \\mathcal{M}$.\nIndeed, $\\mathcal{M}$ is a metric space under the Riemannian distance $d(p, q):=\\inf_{c} L(c) $ where the infimum is taken over all curve segments joining $p $ to $q $ \\cite[Thm. 2.55]{lee2018}.\nFor two manifolds $\\mathcal{M}_1, \\mathcal{M}_2$ and a smooth map $F \\colon \\mathcal{M}_1 \\to \\mathcal{M}_2$, the differential of $F$ at $p \\in \\mathcal{M}_1$ is a linear operator denoted as $\\mathcal{D} F(p) \\colon T_p \\mathcal{M}_1 \\to T_{F(p)} \\mathcal{M}_2$.\nLet $\\mathfrak{F}(\\mathcal{M})$ be the set of all smooth \\emph{scalar fields} (or say real-valued functions) $f\\colon \\mathcal{M} \\to \\mathbb{R}$.\nThe Riemannian gradient of $f$ at $p$, $\\operatorname{grad} f(p)$, is defined as the unique element of $T_p \\mathcal{M}$ that satisfies $\\langle \\xi, \\operatorname{grad} f(p)\\rangle_p = \\mathcal{D} f(p)[\\xi]$ for all $\\xi \\in T_p \\mathcal{M}$, where $\\mathcal{D} f(p) \\colon T_p \\mathcal{M} \\to T_{f(p)} \\mathbb{R} \\cong \\mathbb{R}$.\nNote that for any $f \\in \\mathfrak{F}(\\mathcal{M})$, the gradient vector field $x \\mapsto \\operatorname{grad} f(x)$ is a smooth vector field on $\\mathcal{M}$, i.e., $\\operatorname{grad} f \\in \\mathfrak{X}(\\mathcal{M})$.\n","start":9491,"end":12498}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"回缩与指数映射","source_units":[{"raw_tex":"A retraction $\\mathrm{R} \\colon T \\mathcal{M} \\to \\mathcal{M}$ is a smooth map such that\n$\\mathrm{R}_{p}\\left(0_{p}\\right)=p$ and $\\mathcal{D} \\mathrm{R}_{p}\\left(0_{p}\\right) =\\operatorname{id}_{T_p \\mathcal{M}}$, i.e., the identity map on $T_{p} \\mathcal{M}$, where $\\mathrm{R}_{p}$ is the restriction of $\\mathrm{R}$ to $T_{p} \\mathcal{M}$ and $\\mathcal{D} \\mathrm{R}_{p}\\left(0_{p}\\right) \\colon T_{0_p} (T_p \\mathcal{M}) \\cong T_p \\mathcal{M} \\to T_{p} \\mathcal{M}$ is the differential of $\\mathrm{R}_{p}$ at $0_{p}$.\nOne theoretically perfect type of retraction is the exponential map, denoted as $\\operatorname{Exp}$. Since $\\mathcal{M}$ is complete, the exponential map is well-defined on the whole tangent bundle.\nLet $\\operatorname{Exp}_p\\colon T_p \\mathcal{M} \\rightarrow \\mathcal{M}$ be the exponential map at $p$; then $t \\mapsto \\operatorname{Exp}_p(t \\xi)$ is the unique geodesic that passes through $p$ with velocity $\\xi \\in T_p \\mathcal{M}$ when $t=0$.\n\n","start":12498,"end":13470}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["eq:Singularity"],"declared_external":false,"source_only":false,"title":"向量场零点与协变导数","source_units":[{"raw_tex":"\\subsection{Riemannian Newton Method}\\label{subsec:RiemannianNewton}\n\nThe Newton method is a powerful tool for finding the zeros of nonlinear functions in the Euclidean setting.\nThe generalized Newton method has been studied in the Riemannian setting; it aims to find a \\emph{singularity} for the vector field $F \\in \\mathfrak{X}(\\mathcal{M})$, i.e., a point $p \\in \\mathcal{M}$ such that,\n\\begin{equation}\\label{eq:Singularity}\n\tF(p)=0_{p} \\in T_p \\mathcal{M}.\n\\end{equation}\nLet $\\nabla$ be the Levi-Civita connection on $\\mathcal{M}$, i.e., the unique symmetric connection compatible with the Riemannian metric.\nThe covariant derivative $\\nabla F$ assigns each point $p \\in \\mathcal{M}$ a linear operator $\\nabla F(p)\\colon T_{p}\\mathcal{M}\\to T_{p}\\mathcal{M}$. \nIn particular, the Riemannian Hessian of $f \\in \\mathfrak{F}(\\mathcal{M})$ at $p$ is a self-adjoint operator on $T_{p} \\mathcal{M}$, defined as $\\operatorname{Hess} f(p):=\\nabla (\\operatorname{grad}f)(p).$\n","start":13470,"end":14443}]},{"id":"S005","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"对角矩阵与单位向量","source_units":[{"raw_tex":"In this section, we will give a comprehensive interpretation of the Riemannian interior point method.\nFollowing common usage in the interior-point literature, big letters denote the associated diagonal matrix, e.g., $Z =\\operatorname{diag}(z_{1}, \\ldots, z_{m})$ with $z \\in \\mathbb{R}^{m}$. $e$ denotes the all-ones vector, and $0$ stands for zero vector/matrix with proper dimensions.\n\n\\subsection{KKT Vector Field and Its Covariant Derivative}\n\n","start":15475,"end":15923}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"拉格朗日函数及梯度","source_units":[{"raw_tex":"The Lagrangian function of (\\ref{RCOP}) is\n\\begin{equation*}\n\t\\mathcal{L}(x, y, z):=f(x)+\\sum_{i=1}^l y_i h_i(x)+\\sum_{i=1}^m z_i g_i(x),\n\\end{equation*}\nwhere $y \\in \\mathbb{R}^l$ and $z \\in \\mathbb{R}^m$ are Lagrange multipliers corresponding to the equality and\ninequality constraints, respectively.\nWith respect to the variable $x$, $\\mathcal{L}(\\cdot, y, z)$ is a real-valued function defined on $\\mathcal{M}$, and its Riemannian gradient is\n\\begin{equation*}\n\\operatorname{grad}_{x} \\mathcal{L}(x, y, z)=\\operatorname{grad} f(x) + \\sum_{i =1 }^{l} y_{i} \\operatorname{grad} h_{i}(x) + \\sum_{i =1 }^{m} z_{i} \\operatorname{grad} g_{i}(x),\n\\end{equation*}\nwhere $\\{\\operatorname{grad} h_{i}(x)\\}_{i=1}^{l}$ and $ \\{\\operatorname{grad} g_{i}(x)\\}_{i=1}^{m}$ are the gradients of the component functions of $h$ and $g$. \n","start":15923,"end":16746}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":["eq:KKTConditions"],"declared_external":false,"source_only":false,"title":"黎曼 KKT 条件","source_units":[{"raw_tex":"The Riemannian KKT conditions (e.g., see \\cite[Definition 2.3]{liu2020simple} or \\cite{yang2014optimality}) for (\\ref{RCOP}) are given by\n\\begin{equation}\\label{eq:KKTConditions}\n\t\\operatorname{grad}_{x} \\mathcal{L}(x, y, z) =0_{x}; \\;\n\th(x) =0 ,\n\tg(x) \\leq 0 , \n\tz  \\geq 0; \\;\n\tZ g(x) =0.\n\\end{equation}\n","start":16746,"end":17051}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":["eq:KKTVectorField"],"declared_external":false,"source_only":false,"title":"乘积流形上的 KKT 场","source_units":[{"raw_tex":"The above conditions can be written in terms of slack variables $\\mathbb{R}^{m} \\ni s:=-g(x)$, as\n\\begin{equation}\\label{eq:KKTVectorField}\n\tF(w):=\\left(\\begin{array}{l}\n\t\t\\operatorname{grad}_{x} \\mathcal{L}(x, y, z) \\\\\n\t\t h(x) \\\\\n\t\tg(x)+s \\\\\n\t\tZ S e\n\t\\end{array}\\right)=0_{w}\n\t=\\left(\\begin{array}{l}\n\t\t0_{x} \\\\\n\t\t0\\\\\n\t\t0\\\\\n\t\t0\n\t\\end{array}\\right) \\in T_w \\mathcal{N},\n\\end{equation}\nand $(z,s) \\geq 0,$\nwhere $w:=(x, y, z,s) \\in \\mathcal{N}:=\\mathcal{M}\\times \\mathbb{R}^{l} \\times \\mathbb{R}^{m} \\times \\mathbb{R}^{m}$.\nHere, we have generated a vector field $F$ in (\\ref{eq:KKTVectorField}) on the product manifold $\\mathcal{N}$, i.e.,\nfor any $w\\in \\mathcal{N}$, \n$\nF(w) \\in\nT_{w} \\mathcal{N} \\cong T_{x} \\mathcal{M} \\times \\mathbb{R}^{l} \\times \\mathbb{R}^{m} \\times \\mathbb{R}^{m}.\n$\nWe will call $F \\colon \\mathcal{N} \\to T\\mathcal{N} $ in (\\ref{eq:KKTVectorField}) above the \\emph{KKT vector field} for (\\ref{RCOP}).\n","start":17051,"end":17977}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":["eq:barR"],"declared_external":false,"source_only":false,"title":"乘积回缩","source_units":[{"raw_tex":"Note that for $\\xi=\\left(\\xi_x, \\xi_y, \\xi_z, \\xi_s\\right) \\in T_{w} \\mathcal{N} $,\n\\begin{equation}\\label{eq:barR}\n\t\\bar{\\mathrm{R}}_{w}(\\xi):=(\\mathrm{R}_{x}(\\xi_{x}), y+\\xi_{y}, z+\\xi_{z}, s+\\xi_{s})\n\\end{equation}\nis a well-defined retraction on the product manifold $\\mathcal{N}$ as long as $\\mathrm{R}$ is a retraction on $\\mathcal{M}$.\n","start":17977,"end":18320}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":["defn:HxGx"],"declared_external":false,"source_only":false,"title":"约束梯度算子","source_units":[{"raw_tex":"\nNow, we aim to find a singularity of the KKT vector field $F$ on $\\mathcal{N}$ under some nonnegative constraints $(z,s) \\geq 0$.\nIf the Riemannian Newton method is to be applied to (\\ref{eq:KKTVectorField}), we must formulate the covariant derivative of $F$ at an arbitrary $w \\in \\mathcal{N}$. Before that, we need some new symbols.\nFixing a point $x \\in \\mathcal{M}$, we can define two linear operators $\\mathcal{H}_{x}\\colon \\mathbb{R}^{l} \\rightarrow T_{x} \\mathcal{M}$ and $\\mathcal{G}_{x}\\colon \\mathbb{R}^{m} \\rightarrow T_{x} \\mathcal{M}$ by\n\\begin{equation}\\label{defn:HxGx}\n\t\\mathcal{H}_x [v]:=\\sum_{i=1}^l v_i \\operatorname{grad} h_i(x), \\quad \\mathcal{G}_x [u]:=\\sum_{i=1}^m u_i \\operatorname{grad} g_i(x),\n\\end{equation}\nrespectively.\nThen, for $\\mathcal{H}_{x}$, its adjoint operator $\\mathcal{H}_x^*\\colon T_x \\mathcal{M} \\rightarrow \\mathbb{R}^l$ is given by\n\\begin{equation*}\n\t\\mathcal{H}_x^* [\\xi]=\\left(\\left\\langle\\operatorname{grad} h_1(x), \\xi\\right\\rangle_x, \\cdots,\\left\\langle\\operatorname{grad} h_l(x), \\xi\\right\\rangle_x\\right)^T.\n\\end{equation*}\nAlso, $\\mathcal{G}_x^*\\colon T_x \\mathcal{M} \\rightarrow \\mathbb{R}^m$ with $\\mathcal{G}_x^* [\\xi]=\\left(\\left\\langle\\operatorname{grad} g_1(x), \\xi\\right\\rangle_x, \\cdots,\\left\\langle\\operatorname{grad} g_m(x), \\xi\\right\\rangle_x\\right)^T$. \n","start":18320,"end":19639}]},{"id":"S011","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:CovariantDerivativeKKTVectorField"],"declared_external":false,"source_only":false,"title":"KKT 场的协变导数","source_units":[{"raw_tex":"\nNow, using the solutions of Exercises 5.4 and 5.13 on covariant derivatives of vector fields on product manifolds in monograph \\cite{boumal2022intromanifolds}, the following results can be easily derived.\nGiven KKT vector field $F $ in (\\ref{eq:KKTVectorField}) and $w \\in \\mathcal{N}$, the covariant derivative of $F$ at $w$ is the operator $\\nabla F(w)\\colon T_{w} \\mathcal{N} \\to T_{w} \\mathcal{N}$ given by\n\\begin{equation}\\label{eq:CovariantDerivativeKKTVectorField}\n\t\\nabla F(w) [\\Delta w] =\n\t\\left(\\begin{array}{l}\n\t\t\\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t\t+\\mathcal{H}_{x} [\\Delta y]\n\t\t+\\mathcal{G}_{x} [\\Delta z]\\\\\n\t\t\\mathcal{H}_{x}^{*}[\\Delta x]\\\\\n\t\t\\mathcal{G}_{x}^{*}[\\Delta x]+\\Delta s\\\\\n\t\tZ \\Delta s + S \\Delta z\n\t\\end{array}\\right),\n\\end{equation}\nwhere $\\Delta w=(\\Delta x, \\Delta y, \\Delta z, \\Delta s) \\in T_{w} \\mathcal{N}$ and $\\operatorname{Hess}_{x} \\mathcal{L}(w)$ is the Riemannian Hessian of real-valued function $\\mathcal{L}(\\cdot, y, z)$.\n","start":19639,"end":20618}]},{"id":"S012","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:HessianLagrange"],"declared_external":false,"source_only":false,"title":"拉格朗日 Hessian 展开","source_units":[{"raw_tex":"Indeed,\n$\\operatorname{Hess}_{x} \\mathcal{L}(w)\\colon T_{x}\\mathcal{M} \\to T_{x}\\mathcal{M}$ satisfies\n\\begin{equation}\\label{eq:HessianLagrange}\n\t\\operatorname{Hess}_{x} \\mathcal{L}(w) =\n\t\\operatorname{Hess} f(x)\n\t+\\sum_{i=1}^{l} y_{i} \\operatorname{Hess} h_{i}(x)\n\t+\\sum_{i=1}^{m} z_{i} \\operatorname{Hess} g_{i}(x),\n\\end{equation}\nwhere $\\{\\operatorname{Hess} h_{i}(x)\\}_{i=1}^{l}$ and $\\{\\operatorname{Hess} g_{i}(x)\\}_{i=1}^{m}$ are Hessians of the component functions of $h$ and $g$.\n","start":20618,"end":21108}]},{"id":"S013","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"活动集与假设约定","source_units":[{"raw_tex":"Next, we will discuss an important result about the covariant derivative of the KKT vector field given in (\\ref{eq:CovariantDerivativeKKTVectorField}).\nLet $\\mathbb{A}(x):=\\left\\{i \\mid g_{i}(x)=0\\right\\}$ denote the active set at $x \\in \\mathcal{M}$.\nA prior study on Riemannian optimality conditions \\cite{yang2014optimality,bergmann2019intrinsic} showed that the following assumptions are meaningful.\nWe call them the \\emph{standard Riemannian assumptions} for (\\ref{RCOP}).\nNote that the $x^{*}$ and $w^*$ in \\ref{B3}-\\ref{B5} all refer to those in \\ref{B1}.\n\n\n\\setlist[assumptions,1]{label=\\textbf{(A\\arabic*)}}\n\\begin{assumptions}[leftmargin=9mm]\n","start":21159,"end":21812}]},{"id":"S014","semantic_type":"assumption","kind":"assumption","labels":["B1","B3","B4","B5"],"declared_external":false,"source_only":false,"title":"KKT 解存在","source_units":[{"raw_tex":"\t\\item {Existence.} There exists $w^*=(x^{*}, y^{*}, z^{*}, s^{*})$ satisfying the Riemannian KKT conditions (\\ref{eq:KKTConditions}). Here, we introduce the slack variables $s^{*}:=-g(x^{*})$.\n\t\\label{B1}\t\n%\t\\item {Smoothness of $ C^{2} $.} The functions $f, g,$ and $h$ are $ C^{2} $ on $\\mathcal{M}$.\n%\t\\label{B2}\t\n","start":21812,"end":22130}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","labels":["B1","B3","B4","B5"],"declared_external":false,"source_only":false,"title":"线性无关约束资格","source_units":[{"raw_tex":"\t\\item {Linear Independence Constraint Qualification (LICQ).} \\\\ The set $\\left\\{\\operatorname{grad} h_{i}(x^{*})\\right\\}_{i=1}^{l} \\bigcup \\left\\{\\operatorname{grad} g_{i}(x^{*})\\right\\}_{i \\in \\mathbb{A}(x^{*})}$ is linearly independent in $ T_{x^{*}}\\mathcal{M}$.\n\t\\label{B3}\t\n","start":22130,"end":22410}]},{"id":"S016","semantic_type":"assumption","kind":"assumption","labels":["B1","B3","B4","B5"],"declared_external":false,"source_only":false,"title":"严格互补","source_units":[{"raw_tex":"\t\\item {Strict complementarity.} $(z^{*})_{i}>0$ if $g_{i}(x^{*})=0$ for all $i=1, \\cdots,m$.\n\t\\label{B4}\t\n","start":22410,"end":22517}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"二阶充分条件","source_units":[{"raw_tex":"\t\\item {Second-order sufficiency.}\n\t$\\left \\langle\\operatorname{Hess}_{x} \\mathcal{L}(w^{*})[\\xi], \\xi\\right\\rangle>0$ for all nonzero $\\xi \\in T_{x^{*}}\\mathcal{M}$ \\\\ satisfying $\\left\\langle\\xi, \\operatorname{grad} h_{i}(x^{*})\\right\\rangle=0 $ for $i=1,\\dots,l,$ and\n\t$\\left\\langle\\xi, \\operatorname{grad} g_{i}(x^{*})\\right\\rangle=0$ for $ i \\in \\mathbb{A}(x^{*})$. \n\t\\label{B5}\n\\end{assumptions}\n","start":22517,"end":22919}]},{"id":"S018","semantic_type":"interpretation","kind":"interpretation","labels":["eq:OriginalNewtonEquation"],"declared_external":false,"source_only":false,"title":"未扰动 Newton 方程背景","source_units":[{"raw_tex":"Applying the Riemannian Newton method directly to the KKT vector field $F: \\mathcal{N} \\rightarrow T \\mathcal{N}$ results in the following Newton equation (see (\\ref{eq:StandardNewtonEquation}) without iteration count $k$) at each iteration:\n\\begin{equation}\\label{eq:OriginalNewtonEquation}\n\t\\nabla F(w) [\\Delta w]+F(w)=0.\n\\end{equation}\nAs with the usual interior point method in the Euclidean setting, once the iterates reach the boundary of the feasible region, they are forced to stick to it \\cite[P6]{wright1997primal}.\nFor the iterates to maintain a sufficient distance from the boundary, we introduce a perturbed complementary equation with some barrier parameter $\\mu>0$ and define the \\textit{perturbed KKT vector field}:\n","start":27451,"end":28183}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":["eq:perturbedKKTvectorfield"],"declared_external":false,"source_only":false,"title":"扰动 KKT 场","source_units":[{"raw_tex":"\\begin{equation}\\label{eq:perturbedKKTvectorfield}\n\tF_{\\mu}(w):=F(w)-\\mu \\hat{e}, \\text { and } \\hat{e}:=\\hat{e}(w):=\\left(0_x, 0,0, e\\right).\n\\end{equation}\nNotice that the perturbation term $\\hat{e}$, indeed, is a special vector field on $\\mathcal{N}$, not a constant, because $0_x$ is essentially dependent on $w$ and/or $x$.\n","start":28183,"end":28512}]},{"id":"S020","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:NabulaperturbedKKTvectorfield"],"declared_external":false,"source_only":false,"title":"扰动不改变协变导数","source_units":[{"raw_tex":"In fact, the covariant derivative of the perturbed KKT vector field is the same as that of the original. From the linearity of the connection $\\nabla$, for any $w \\in \\mathcal{N}$ and any $\\mu>0$, we have \n\\begin{equation}\\label{eq:NabulaperturbedKKTvectorfield}\n\t\\nabla F_{\\mu}(w)=\\nabla F(w)-\\mu \\nabla \\hat{e}(w)=\\nabla F(w),\n\\end{equation}\nwhere the last equity comes from $\\nabla \\hat{e}(w) [\\Delta w]=\\left(0_x, 0,0,0\\right)$ for all $\\Delta w \\in T_w \\mathcal{N}$.\nApplying the Riemannian Newton method to perturbed KKT vector field $F_{\\mu}(w)$ yields the \\emph{perturbed Newton equation}: $\\nabla F_{\\mu}(w) [\\Delta w]+F_{\\mu}(w)=0$. \nFrom (\\ref{eq:perturbedKKTvectorfield}) and (\\ref{eq:NabulaperturbedKKTvectorfield}), this equation is equivalent to \n\\begin{equation}\n\t\\nabla F(w) [\\Delta w]+F(w)=\\mu \\hat{e},\n\\end{equation}\nwhich reduces to the ordinary Newton equation (\\ref{eq:OriginalNewtonEquation}) as $\\mu \\to 0$.\nAt this point, we can describe a prototype of the Riemannian Interior Point Method (RIPM) in Algorithm \\ref{algo:prototype_algo}.\n","start":28512,"end":29574}]},{"id":"S021","semantic_type":"algorithm","kind":"algorithm","labels":["algo:prototype_algo","eq:prototype","rule:StepSizeA"],"declared_external":false,"source_only":false,"title":"RIPM 原型算法","source_units":[{"raw_tex":"\n\\begin{algorithm}%[H]\n\t\\caption{Prototype Algorithm of RIPM for (\\ref{RCOP})}\n\t\\label{algo:prototype_algo}\n\t\\SetAlgoLined\n\t\n\t\\KwIn{An initial point $w_0=\\left(x_0, y_0, z_0, s_0\\right) \\in \\mathcal{N}$ with $\\left(z_0, s_0\\right)>0$ and a retraction $\\mathrm{R}$ on $\\mathcal{M}$. $\\hat{\\gamma} \\in(0,1)$, $\\mu_0>0$.}\n\t\\KwOut{Sequence $\\left\\{w_k\\right\\} \\subset \\mathcal{N}$ such that $\\left\\{w_k\\right\\} \\rightarrow w^*$ and $w^*$ satisfies the KKT conditions (\\ref{eq:KKTConditions}).} % \n\t\n\tSet $k \\to 0$\\;\n\t\n\t\\While{Stopping criterion not satisfied}{\n\t\t1. Obtain $\\Delta w_{k}=(\\Delta x_{k}, \\Delta y_{k}, \\Delta z_{k},\\Delta s_{k}) \\in T_{w_k} \\mathcal{N}$ by solving the perturbed Newton equation on $T_{w_k} \\mathcal{N}$:\n\t\t\\begin{equation}\\label{eq:prototype}\n\t\t\t\\nabla F(w_{k}) [\\Delta w_{k}]=-F(w_{k})+\\mu_{k} \\hat{e};\n\t\t\\end{equation}\n\t\n\t\t2. Choose $ \\gamma_{k} \\in [\\hat{\\gamma},1]$ and compute the step size $\\alpha_{k}$ defined by\n\t\t\\begin{equation}\\label{rule:StepSizeA}\n\t\t\t\\min \\left\\{1, \\gamma_{k} \\min _{i}\\left\\{-\\frac{(s_{k})_{i}}{(\\Delta s_{k})_{i}} \\mid(\\Delta s_{k})_{i}<0\\right\\},\\gamma_{k} \\min _{i}\\left\\{-\\frac{(z_{k})_{i}}{(\\Delta z_{k})_{i}} \\mid(\\Delta z_{k})_{i}<0\\right\\}\\right\\};\n\t\t\\end{equation}\n\t\n\t\t3. Compute the next point as\n\t\t$w_{k+1}:=\\bar{\\mathrm{R}}_{w_k}\\left(\\alpha_k \\Delta w_k\\right)$, see (\\ref{eq:barR})\\;\n\t\t\n\t\t4. Choose $0<\\mu_{k+1}<\\mu_k$\\;\n\t\t\n\t\t5. $k \\to k+1$\\;\n\t}\n\\end{algorithm}\n","start":29574,"end":31008}]},{"id":"S022","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"步长规则的设计目的","source_units":[{"raw_tex":"\nIn the step 2 of Algorithm \\ref{algo:prototype_algo}, we just wish to compute a step size $0<\\alpha_k \\leq 1$ to ensure new point $w_{k+1}$ with $\\left(z_{k+1},s_{k+1}\\right)>0$. \nThere are many schemes to achieve this purpose. \nThe scheme in (\\ref{rule:StepSizeA}) is simple but sufficient to guarantee the local convergence, as will be proved in Section \\ref{sec:local}.\n","start":31008,"end":31382}]},{"id":"S023","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:FullNewtonEquation"],"declared_external":false,"source_only":false,"title":"完整扰动 Newton 系统","source_units":[{"raw_tex":"Let us consider Algorithm \\ref{algo:prototype_algo} and omit the iteration count $k$.\nGiven the current point $w \\in \\mathcal{N}$ with $(z, s) > 0$, for the KKT vector field $F(w)$ in (\\ref{eq:KKTVectorField}), we denote its components by $F_x,F_y,F_z,F_s$ in top-to-bottom order, namely,\n$F_x:=\\operatorname{grad}_x \\mathcal{L}(w),F_y:=h(x),F_z:=g(x)+s,F_s:=Z S e.$\nBy using these symbols and the formulation of $\\nabla F(w)$ in (\\ref{eq:CovariantDerivativeKKTVectorField}), the \\emph{full} (perturbed) Newton equation (\\ref{eq:prototype}) defined on $T_w \\mathcal{N} \\cong T_x \\mathcal{M} \\times \\mathbb{R}^l \\times \\mathbb{R}^m \\times \\mathbb{R}^m$ is expanded as:\n\\begin{equation}\\label{eq:FullNewtonEquation}\n\t\\left(\\begin{array}{l}\n\t\t\\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t\t+\\mathcal{H}_{x} [\\Delta y]\n\t\t+\\mathcal{G}_{x} [\\Delta z]\\\\\n\t\t\\mathcal{H}_{x}^{*}[\\Delta x]\\\\\n\t\t\\mathcal{G}_{x}^{*}[\\Delta x]+\\Delta s\\\\\n\t\tZ \\Delta s + S \\Delta z\n\t\\end{array}\\right)\n\t=\n\t\\left(\\begin{array}{l}\n\t\t-F_x\\\\\n\t\t-F_y\\\\\n\t\t-F_z\\\\\n\t\t-F_s+\\mu e\n\t\\end{array}\\right).\n\\end{equation}\n","start":32122,"end":33200}]},{"id":"S024","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"局部逆回缩的存在","source_units":[{"raw_tex":"\\begin{remark}\n\tFor a retraction $\\mathrm{R}$ on $\\mathcal{M}$ and $x \\in \\mathcal{M}$,\n\tby $\\mathcal{D} \\mathrm{R}_{x}\\left(0_{x}\\right)=\\mathrm{id}_{T_{x} \\mathcal{M}}$ and the inverse function theorem, there exists a neighborhood $V$ of $0_{x}$ in $T_{x} \\mathcal{M}$ such that $ \\mathrm{R}_{x} $ is a diffeomorphism on $V$; thus,\n\t$\\mathrm{R}_{x}^{-1}(y)$ is well defined for all $ y \\in \\mathcal{M} $ sufficiently close to $ x $.\n\tIn this case, $\\mathrm{R}_{x}(V) \\subset \\mathcal{M}$ is called a retractive neighborhood of $x$.\n\tFurthermore, the existence of a \\emph{totally} retractive neighborhood \\cite[Theorem 2]{zhu2020riemannian} shows that for any $\\bar{x} \\in \\mathcal{M} $ there is a neighborhood $W$ of $\\bar{x}$ such that\n\t$\\mathrm{R}_{x}^{-1}(y)$ is well defined for all $x, y \\in W$.\n\tIn what follows, we will suppose that an appropriate neighborhood has been chosen by default for the well-definedness of $\\mathrm{R}_x^{-1}(y)$.\n\\end{remark}\n","start":41737,"end":42699}]},{"id":"S025","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"向量输运与等距性","source_units":[{"raw_tex":"Define the Whitney sum\n$\nT \\mathcal{M} \\oplus T \\mathcal{M}:=\\left\\{(\\eta, \\xi): \\eta, \\xi \\in T_x \\mathcal{M}, x \\in \\mathcal{M} \\right\\}.\n$\nA smooth map\n$\n\\mathrm{T} \\colon T \\mathcal{M} \\oplus T \\mathcal{M} \\to T \\mathcal{M}: (\\eta, \\xi) \\mapsto \\mathrm{T}_{\\eta}(\\xi),\n$\nis called a \\emph{vector transport} on $\\mathcal{M}$ if there exists an associated retraction $\\mathrm{R}$ on $\\mathcal{M}$ such that $\\mathrm{T}$ satisfies the following properties for all $x \\in \\mathcal{M}$:\n\\begin{enumerate}\n\t\\item Associated retraction. $\\mathrm{T}_{\\eta}\\left(\\xi\\right) \\in T_{\\mathrm{R}_{x}\\left(\\eta\\right)} \\mathcal{M}$\n\tfor all $\\eta, \\xi \\in T_x \\mathcal{M}$.\t\n\t\\item Consistency. $\\mathrm{T}_{0_{x}}\\left(\\xi\\right)=\\xi$ for all $\\xi \\in T_{x} \\mathcal{M}$.\t\n\t\\item Linearity.\n\t$\\mathrm{T}_{\\eta}\\left(a \\xi+b \\zeta\\right)=a \\mathrm{T}_{\\eta}\\left(\\xi\\right)+b \\mathrm{T}_{\\eta}\\left(\\zeta\\right)$ for all $a, b \\in \\mathbb{R}$ and $\\eta, \\xi, \\zeta \\in T_{x} \\mathcal{M}$.\n\\end{enumerate}\nThus, fixing any $\\eta \\in T_x \\mathcal{M}$, the map\n$\n\\mathrm{T}_\\eta \\colon T_x \\mathcal{M} \\to T_{\\mathrm{R}_x(\\eta)} \\mathcal{M}: \\xi \\mapsto \\mathrm{T}_\\eta(\\xi),\n$\nis a linear operator.\nAdditionally, $\\mathrm{T}$ is \\emph{isometric} if \n$\n\\langle \\mathrm{T}_{\\eta} (\\xi), \\mathrm{T}_{\\eta} (\\zeta)\\rangle =\\langle\\xi, \\zeta\\rangle\n$\nholds, for all $x \\in \\mathcal{M}$ and all $\\xi, \\zeta, \\eta \\in T_{x} \\mathcal{M}$.\nIn other words, for any $\\eta \\in T_x \\mathcal{M}$, the adjoint and the inverse of $\\mathrm{T}_{\\eta}$ coincide, i.e., $ \\mathrm{T}_{\\eta}^{*} =  \\mathrm{T}_{\\eta}^{-1}$.\n","start":42754,"end":44344}]},{"id":"S026","semantic_type":"definition","kind":"definition","labels":["eq:DifferentiatedRetraction"],"declared_external":false,"source_only":false,"title":"微分回缩输运","source_units":[{"raw_tex":"There are two important classes of vector transport as follows. Let $\\mathrm{R}$ be a retraction on $\\mathcal{M}$. \n\\begin{enumerate}[(1)]\n\t\\item \n\tThe \\emph{differentiated retraction} defined by\n\t\\begin{equation}\\label{eq:DifferentiatedRetraction}\n\t\t\\mathrm{T}_{\\eta}(\\xi):=\\mathcal{D} \\mathrm{R}_{x}(\\eta)[\\xi],  \\; \\eta, \\xi \\in T_{x} \\mathcal{M}, x \\in \\mathcal{M}, \n\t\\end{equation}\n\tis a valid vector transport \\cite[equation (8.6)]{absil2009optimization}.\t\n","start":44344,"end":44807}]},{"id":"S027","semantic_type":"definition","kind":"definition","labels":["eq:ParallelTransport"],"declared_external":false,"source_only":false,"title":"沿回缩曲线的平行输运","source_units":[{"raw_tex":"\t\\item\n\tGiven a smooth curve $\\gamma \\colon [0,1] \\to \\mathcal{M}$ and $t_0, t_1 \\in [0,1]$, the \\emph{parallel transport} from the tangent space at $\\gamma(t_0)$ to the tangent space at $\\gamma(t_1)$ along $\\gamma,$\n\tis a linear operator\n\t$ \\mathrm{P}_\\gamma^{t_1 \\to t_0} \\colon T_{\\gamma\\left(t_0\\right)} \\mathcal{M} \\rightarrow T_{\\gamma\\left(t_1\\right)} \\mathcal{M}$ defined by $\\mathrm{P}_\\gamma^{t_1 \\to t_0}(\\xi)=Z\\left(t_1\\right)$, where $Z$ is the unique parallel vector field such that $Z\\left(t_0\\right)=\\xi$.\n\tThen, for any $x \\in \\mathcal{M}, \\eta \\in T_x \\mathcal{M}$, then\n\t\\begin{equation}\\label{eq:ParallelTransport}\n\t\t\\mathrm{T}_{\\eta}\\left(\\xi\\right):=\\mathrm{P}_\\gamma^{1 \\to 0} (\\xi)\n\t\\end{equation}\n\tis a valid vector transport \\cite[equation (8.2)]{absil2009optimization}, where $\\mathrm{P}_\\gamma$ denotes the parallel transport along the curve $t \\mapsto \\gamma(t):=\\mathrm{R}_x\\left(t \\eta\\right)$. We often omit the superscript $^{1 \\to 0}$ if it is clear from the context.\n\tIn particular, parallel transport is isometric.\n\\end{enumerate}\n","start":44807,"end":45874}]},{"id":"S028","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"算子范数与输运可逆性","source_units":[{"raw_tex":"Going one degree higher, let us now discuss the Lipschitz continuity of Hessian operators.\nThroughout this paper, for a linear operator $\\mathcal{A}: \\mathcal{E} \\rightarrow \\mathcal{E}^{\\prime}$ between two finite-dimensional normed vector spaces $\\mathcal{E}$ and $\\mathcal{E}^{\\prime}$, the (operator) norm of $\\mathcal{A}$ is defined by $\\|\\mathcal{A}\\|:=\\sup \\left\\{\\|\\mathcal{A} v\\|_{\\mathcal{E}^{\\prime}} : v \\in \\mathcal{E},\\|v\\|_{\\mathcal{E}}=1\\right.$, or, $\\left.\\|v\\|_{\\mathcal{E}} \\leq 1\\right\\}.$\nThe inverse of $\\mathrm{T}_{\\eta}$ is needed in the following definitions, so we can assume that vector transport $\\mathrm{T}$ is isometric, e.g., parallel transport in (\\ref{eq:ParallelTransport}). In fact, there are many ways to construct isometric vector transports; see \\cite[Section 2.3]{huang2015riemannian}.\n","start":48102,"end":48928}]},{"id":"S029","semantic_type":"definition","kind":"definition","labels":["defn:TwiceLipschitzContinuouslyDifferentiable"],"declared_external":false,"source_only":false,"title":"Hessian 的输运 Lipschitz 性","source_units":[{"raw_tex":"\\begin{definition}[{\\cite[Assumption 3]{huang2015riemannian}}]\n\t\\label{defn:TwiceLipschitzContinuouslyDifferentiable}\n\tA function $f\\colon \\mathcal{M} \\rightarrow \\mathbb{R}$ is \\emph{twice Lipschitz continuously differentiable with respect to $\\mathrm{T}$ in $\\mathcal{U} \\subset \\mathcal{M}$} if it is twice differentiable and there exists a constant $\\kappa>0$ such that, for all $x, y \\in \\mathcal{U}$, \n\t$\n\t\\|\\operatorname{Hess} f(y)-\\mathrm{T}_{\\eta} \\operatorname{Hess} f(x) \\mathrm{T}_{\\eta}^{-1}\\| \\leq \\kappa d(x, y),\n\t$\n\twhere $\\eta=\\mathrm{R}_{x}^{-1} y$.\n\\end{definition}\n","start":48928,"end":49513}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","labels":["lem:TwiceLipschitzContinuouslyDifferentiable"],"declared_external":true,"declaration_text":["原先决条件清单：Huang, Absil, and Gallivan (2015), Lemma 4；原文 B0052 重述。"],"source_only":false,"title":"三阶光滑函数的局部正则性","source_units":[{"raw_tex":"\\begin{lemma}[{\\cite[Lemma 4]{huang2015riemannian}}]\n\t\\label{lem:TwiceLipschitzContinuouslyDifferentiable}\n\tIf $f\\colon \\mathcal{M} \\rightarrow \\mathbb{R}$ is $ C^3 $, then for any $\\bar{x} \\in \\mathcal{M}$ and any isometric vector transport $\\mathrm{T}$, there exists a neighborhood $\\mathcal{U}$ of $\\bar{x}$ such that $f$ is twice Lipschitz continuously differentiable with respect to $\\mathrm{T}$ in $\\mathcal{U}$.\n\\end{lemma}\n","start":49513,"end":49944}]},{"id":"S031","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"协变导数引理的证明类比","source_units":[{"raw_tex":"\nIf the operator, $\\operatorname{Hess}f(x)$, above is replaced by a general covariant derivative $\\nabla F (x)$, we can get the next results in a similar way.\nLemma \\ref{lem:LipCountNabulaF} can be proven in the same way as Lemma \\ref{lem:TwiceLipschitzContinuouslyDifferentiable}.\n","start":49944,"end":50226}]},{"id":"S032","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"协变导数的输运 Lipschitz 性","source_units":[{"raw_tex":"\\begin{definition}\n\tGiven a vector field $F$ on $\\mathcal{M}$.\n\tThe map $x \\mapsto \\nabla F(x)$ is \\emph{Lipschitz continuous with respect to $\\mathrm{T}$ in $\\mathcal{U} \\subset \\mathcal{M}$} if there exists a constant $\\kappa>0$ such that, for all $x, y \\in \\mathcal{U}$, it holds that\n\t$\n\t\\| \\nabla F(y)\n\t-\n\t\\mathrm{T}_{\\eta} \\nabla F(x) \\mathrm{T}_{\\eta}^{-1}\n\t\\| \\leq \\kappa d(x, y),\n\t$\n\twhere $\\eta=\\mathrm{R}_{x}^{-1} y$.\n\\end{definition}\n","start":50226,"end":50672}]},{"id":"S033","semantic_type":"lemma","kind":"lemma","labels":["lem:LipCountNabulaF"],"declared_external":false,"source_only":false,"title":"二阶向量场的局部正则性","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:LipCountNabulaF}\n\tIf $F$ is a $C^{2}$ vector field, then\n\tfor any $\\bar{x} \\in \\mathcal{M}$ and any isometric vector transport $\\mathrm{T}$, there exists a neighborhood $\\mathcal{U}$ of $\\bar{x}$ such that the map $x \\mapsto \\nabla F(x)$ is Lipschitz continuous with respect to $\\mathrm{T}$ in $\\mathcal{U}$.\n\\end{lemma}","start":50672,"end":51016}]},{"id":"S034","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"距离与逆回缩范数的比较动机","source_units":[{"raw_tex":"Notice that in the previous subsection on the definitions of Lipschitz continuity, we used $\\|\\eta\\|$ with $\\eta=\\mathrm{R}_x^{-1} y$ or $d(x, y)$ to denote the upper bound on the right-hand side. The next lemma shows that the two are not essentially different. When $\\mathcal{M}=\\mathbb{R}^{n}$, both reduce to $\\| x-y \\|$.\n\n","start":51049,"end":51375}]},{"id":"S035","semantic_type":"lemma","kind":"lemma","labels":["lem:distR1"],"declared_external":true,"declaration_text":["原先决条件清单：Huang, Absil, and Gallivan (2015), Lemma 2；原文 B0056 重述。"],"source_only":false,"title":"回缩与距离的局部比较","source_units":[{"raw_tex":"\\begin{lemma}[{\\cite[Lemma 2]{huang2015riemannian}}]\\label{lem:distR1}\n\tLet $\\mathcal{M}$ be a Riemannian manifold with a retraction $\\mathrm{R}$ and let $\\bar{x} \\in \\mathcal{M}$. \n\tThen,\n\t\\begin{enumerate}[(i)]\n\t\t \\item there exist $a_{0}, a_{1}, \\delta_{a_{0}, a_{1}}>0$ such that for all $x$ in a sufficiently small neighborhood of $\\bar{x}$ and all $\\xi, \\eta \\in T_{x} \\mathcal{M}$ with $\\|\\xi\\|, \\|\\eta\\|  \\leq \\delta_{a_{0}, a_{1}}$,\n\t\t one has\n\t\t $\n\t\t a_{0}\\|\\xi-\\eta\\| \\leq d(\\mathrm{R}_{x}(\\eta), \\mathrm{R}_{x}(\\xi)) \\leq a_{1}\\|\\xi-\\eta\\|.\n\t\t $\t\n\t\t In particular,\n\t\t $\n\t\t a_{0}\\|\\xi\\| \\leq d(x, \\mathrm{R}_{x}(\\xi)) \\leq a_{1}\\|\\xi\\|\n\t\t $\n\t\t when $\\eta=0$;\n\t\t \n\t\t \\item\n\t\t there exist $a_{0},a_{1}>0$ such that for all $x$ in a sufficiently small neighborhood of $\\bar{x}$, one has\n\t\t $\n\t\t a_{0}\\|\\xi\\| \\leq d(x,\\bar{x}) \\leq a_{1}\\|\\xi\\| \\text { where } \\xi=\\mathrm{R}_{\\bar{x}}^{-1}(x). \n\t\t $\n\t\\end{enumerate}\n\\end{lemma}\n","start":51375,"end":52312}]},{"id":"S036","semantic_type":"lemma","kind":"lemma","labels":["lem:FundamentalTheoremOfCalculusRiemannian"],"declared_external":true,"declaration_text":["原先决条件清单：Huang, Absil, and Gallivan (2015), Lemma 8；原文 B0057 重述，指数映射情形附引 Ferreira and Svaiter (2002), equation (2.4)。"],"source_only":false,"title":"黎曼微积分基本估计","source_units":[{"raw_tex":"\nThe next lemma is the fundamental theorem of calculus in the Riemannian case.\n\\begin{lemma}[{\\cite[Lemma 8]{huang2015riemannian}}]\\label{lem:FundamentalTheoremOfCalculusRiemannian}\n\tLet $F$ be a $C^{1}$ vector field and $\\bar{x} \\in \\mathcal{M}$. \n\tThen there exist a neighborhood $\\mathcal{U}$ of $\\bar{x}$ and a constant $c_{1} \\geq 0$ such that for all $x, y \\in \\mathcal{U}$,\n\t\\begin{equation*}\n\t\\left\\| \\mathrm{P}_{\\gamma}^{0 \\to 1} [F(y)]-F(x)-\\int_{0}^{1} \\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} [\\eta] \\mathrm{d} t \\right\\| \n\t\\leq c_{1} \\|\\eta\\|^{2},\n\t\\end{equation*}\n\twhere $\\eta=\\mathrm{R}_{x}^{-1}(y)$ and $\\mathrm{P}_{\\gamma}$ is the parallel transport along the curve $\\gamma(t):=\\mathrm{R}_{x}\\left(t \\eta\\right)$.\n\tMoreover, if $\\mathrm{R}=\\operatorname{Exp}$, then indeed $c_1=0$ above (see \\cite[equation (2.4)]{ferreira2002kantorovich}).\n\\end{lemma}\n","start":52312,"end":53221}]},{"id":"S037","semantic_type":"lemma","kind":"lemma","labels":["lem:nonsingular"],"declared_external":true,"declaration_text":["原先决条件清单：Fernandes, Ferreira, and Yuan (2017), Lemma 3.2；原文 B0065 重述。"],"source_only":false,"title":"局部逆算子一致有界","source_units":[{"raw_tex":"\\begin{lemma}[{\\cite[Lemma 3.2]{fernandes2017superlinear}}]\\label{lem:nonsingular}\n\tGiven a vector field $F$ on $\\mathcal{M}$.\n\tIf the map $p \\mapsto \\nabla F(p)$ is continuous at $p^{*}$ and $\\nabla F(p^{*})$ is nonsingular, \n\tthen there exist a neighborhood $\\mathcal{U}$ of $p^{*}$ and a constant $\\Xi > 0$ such that, for all $p \\in \\mathcal{U}$, $\\nabla F(p)$ is nonsingular and $\\left\\|\\nabla F(p)^{-1}\\right\\| \\leq \\Xi$.\n\\end{lemma}\n","start":55894,"end":56333}]},{"id":"S038","semantic_type":"lemma","kind":"lemma","labels":["lem:dist_zero"],"declared_external":true,"declaration_text":["原先决条件清单：Gallivan, Qi, and Absil (2012), Lemma 14.5；原文 B0066 重述。"],"source_only":false,"title":"残差与解距离等价","source_units":[{"raw_tex":"\\begin{lemma}[{\\cite[Lemma 14.5]{gallivan2012riemannian}}]\\label{lem:dist_zero}\n\tLet $F$ be a  $C^2$ vector field on $\\mathcal{M}$ and $p^{*} \\in \\mathcal{M}$. If $ F(p^{*})=0 $ and $\\nabla F(p^{*}) $ is nonsingular, then there exist a neighborhood $\\mathcal{U}$ of $p^{*}$ and constants $c_{3}, c_{4} > 0$ such that, for all $p \\in \\mathcal{U}$, $ c_{3} d(p, p^{*}) \\leq \\|F(p)\\| \\leq c_{4} d(p, p^{*}).$\n\\end{lemma}\n","start":56333,"end":56751}]},{"id":"S039","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"渐近阶记号","source_units":[{"raw_tex":"Here, for any two nonnegative sequences $\\left\\{u_{k}\\right\\}$ and $\\left\\{v_{k}\\right\\}$, we write $u_{k}=O(v_{k})$ if there is a constant $M >0$ such that\n$u_{k}\\leq M v_{k}$\nfor all sufficiently large $k$; and we write\n$u_{k}=o(v_{k})$\nif $v_{k}>0$ and the sequence of ratios $\\{u_{k} / v_{k}\\} $ approaches zero.\n","start":57284,"end":57601}]},{"id":"S040","semantic_type":"algorithm","kind":"algorithm","labels":["algo:PerturbedDampedNewtonMethod"],"declared_external":false,"source_only":false,"title":"扰动阻尼黎曼 Newton 法","source_units":[{"raw_tex":"\\begin{algorithm}%[H]\n\t\\caption{Perturbed Damped Riemannian Newton Method for (\\ref{eq:Singularity})}\n\t\\label{algo:PerturbedDampedNewtonMethod}\n\t\\SetAlgoLined\n\t\n\t\\KwIn{A vector field $F \\in \\mathfrak{X}(\\mathcal{M})$, an initial point $p_0 \\in \\mathcal{M}$ and a retraction $\\mathrm{R}$ on $\\mathcal{M}$. Set $\\mu_0>0$.}\n\t\\KwOut{Sequence $\\{p_k\\} \\subset \\mathcal{M}$ such that $\\left\\{p_k\\right\\} \\rightarrow p^*$ and $F(p^{*})=0_{p^{*}}$.}\n\t\n\tSet $k \\to 0$\\;\n\t\n\t\\While{Stopping criterion not satisfied}{\n\t\t1. Obtain $\\xi_k \\in T_{p_k}\\mathcal{M}$ by solving the perturbed Newton equation:\n\t\t\\begin{equation}\n\t\t\t\\nabla F(p_{k}) [\\xi_k] = - F(p_{k}) + \\mu_{k} \\hat{e};\n\t\t\\end{equation}\n\t\n\t\t2. Choose a (damped) step size $ 0<\\alpha_{k} \\leq 1$\\;\n\t\t\n\t\t3. Compute the next point as $p_{k+1}:=\\mathrm{R}_{p_{k}} (\\alpha_{k} \\xi_k)$\\;\n\t\t\n\t\t4. Choose $0<\\mu_{k+1}<\\mu_k$\\;\n\t\t\n\t\t5. $k \\to k + 1$\\;\n\t}\n\\end{algorithm}\n","start":58004,"end":58915}]},{"id":"S041","semantic_type":"assumption","kind":"assumption","labels":["A1","A2","A3"],"declared_external":false,"source_only":false,"title":"Newton 局部假设","source_units":[{"raw_tex":"\\begin{assumptions}[leftmargin=9mm]\n\t\\item There exists $p^{*} \\in \\mathcal{M}$ such that $F(p^{*})=0_{p^{*}}$;\n\t\\label{A1}\n\t\\item The covariant derivative $\\nabla F (p^{*})$ is nonsingular;\n\t\\label{A2}\n%\t\\item The map $p \\mapsto \\nabla F(p) $ is Lipschitz continuous with respect to on a neighborhood of $p^{*}$;\n%\t\\label{A3}\n\t\\item The vector field $F$ is $C^2$.\n\t\\label{A3}\n\\end{assumptions}\n","start":59463,"end":59858}]},{"id":"S042","semantic_type":"lemma","kind":"lemma","labels":["lem:gamma_alpha"],"declared_external":false,"source_only":false,"title":"中心参数控制阻尼缺口","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:gamma_alpha}\n\tConsider the Algorithm \\ref{algo:prototype_algo} for solving the problem (\\ref{RCOP}).  \n\tLet \\ref{B1} and \\ref{B4} hold at some $w^{*}=(x^{*}, y^{*}, z^{*}, s^{*})$ and $ \\alpha_{k} $ be as in (\\ref{rule:StepSizeA}). \n\tDefine a constant,\n\t\\begin{equation*}\n\t\\Pi := 2 \\max \\left\\{\\max _{i}\\left\\{\n\t1/(s^{*})_{i}\n\t\\mid (s^{*})_{i}>0\\right\\}, \\max _{i}\\left\\{\n\t1/(z^{*})_{i}\n\t\\mid (z^{*})_{i}>0\\right\\}\\right\\}.\n\t\\end{equation*}\n\tFor $\\gamma_{k} \\in(0,1)$, if\n\t$\n\t\\Pi \\left\\| \\Delta w_{k}\\right\\| \\leq \\gamma_{k},\n\t$\n\tthen\n\t$\n\t0  \\leq 1-\\alpha_{k} \\leq\n\t(1-\\gamma_{k})+\\Pi \\left\\| \\Delta w_{k}\\right\\|.\n\t$\n\\end{lemma}","start":65558,"end":66211}]},{"id":"S043","semantic_type":"theorem","kind":"theorem","labels":["prop:non"],"declared_external":false,"source_only":false,"title":"解处 KKT 导数非奇异","source_units":[{"raw_tex":"\\begin{theorem}\\label{prop:non}\n\tIf the standard Riemannian assumptions \\ref{B1}-\\ref{B5} hold at some point $w^{*}$,\n\tthen the operator $\\nabla F(w^{*})$ in (\\ref{eq:CovariantDerivativeKKTVectorField}) is nonsingular.\n\\end{theorem}","start":23641,"end":23873}]},{"id":"S044","semantic_type":"proof","kind":"proof","labels":["13","sing1"],"declared_external":false,"source_only":false,"title":"KKT 非奇异性的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nThis proof omits all the asterisks of the variables. Define $ \\mathbb{E}:=\\{1,\\dots,l\\}$ and $\\mathbb{I}:=\\{1,\\dots,m\\} $. Take some $w=(x, y, z, s) \\in \\mathcal{N} $ satisfying \\ref{B1}-\\ref{B5}, then we have $ s_{i} = -g_{i}(x)$ and $z_{i} s_{i} = 0$ for all $ i \\in \\mathbb{I}$. For short, let $\\mathbb{A}:=\\mathbb{A}(x) \\subset \\mathbb{I} $. \nSuppose that $\\nabla {F}(w) [\\Delta w] =0$ for some $\\Delta w = (\\Delta x, \\Delta y, \\Delta z, \\Delta s) \\in T_{w} \\mathcal{N} \\cong T_x \\mathcal{M} \\times \\mathbb{R}^l \\times \\mathbb{R}^m \\times \\mathbb{R}^m$.\n$\\Delta y_i$ denotes the components of the vector $\\Delta y$, as do $\\Delta z_i$, $\\Delta s_i$.\nTo prove its nonsingularity, we will show that $ \\Delta w=0$.\nExpanding the equation $\\nabla {F}(w) [\\Delta w] =0$ gives\n\\begin{equation}\\label{13}\n\t\\left\\{\n\t\\begin{aligned}\n\t\t0&= \\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n\t\t+\\sum_{i\\in \\mathbb{I}} \\Delta z_{i} \\operatorname{grad} g_{i}(x),\\\\\n\t\t0&=\\left\\langle\\operatorname{grad}h_{i}(x), \\Delta x\\right\\rangle, \\text { for all } i\\in \\mathbb{E},\\\\\n\t\t0&=\\left\\langle\\operatorname{grad} g_{i}(x), \\Delta x\\right\\rangle + \\Delta s_{i}, \\text { for all } i\\in \\mathbb{I}, \\\\\n\t\t0&=z_{i} \\Delta s_{i} + s_{i} \\Delta z_{i}, \\text { for all } i \\in \\mathbb{I}. \n\t\\end{aligned}\n\t\\right.\n\\end{equation}\nStrict complementarity \\ref{B4} and the last equalities above imply that $ \\Delta s_{i}=0 $\nfor all $ i \\in \\mathbb{A}$ and $ \\Delta z_{i}=0 $\nfor $ i \\in  \\mathbb{I} \\setminus \\mathbb{A}$.\nSubstituting those values into the system (\\ref{13}) reduces it to\n\\begin{equation}\\label{sing1}\n\t\\left\\{\n\t\\begin{aligned}\n\t\t0&= \\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n\t\t+\\sum_{i\\in \\mathbb{A}} \\Delta z_{i} \\operatorname{grad} g_{i}(x), \\\\\n\t\t0&=\\left\\langle\\operatorname{grad}h_{i}(x), \\Delta x\\right\\rangle, \\text { for all } i\\in \\mathbb{E},\\\\\n\t\t0&=\\left\\langle\\operatorname{grad}g_{i}(x), \\Delta x\\right\\rangle, \\text { for all } i\\in \\mathbb{A},\n\t\\end{aligned}\n\t\\right.\n\\end{equation}\nand\n$\n\\Delta s_{i}=-\\left\\langle\\operatorname{grad} g_{i}(x), \\Delta x\\right\\rangle \\text { for all } i\\in \\mathbb{I} \\setminus \\mathbb{A}.\n$\nIt follows from system (\\ref{sing1}) that\n\\begin{align*}\n\t0\n\t&=  \\langle \\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n\t+\\sum_{i\\in  \\mathbb{A}} \\Delta z_{i} \\operatorname{grad} g_{i}(x) , \\Delta x \\rangle \\\\\n\t&= \\left\\langle\\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x], \\Delta x\\right\\rangle\n\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\left\\langle\\operatorname{grad}h_{i}(x), \\Delta x\\right\\rangle\n\t+\\sum_{i\\in  \\mathbb{A}} \\Delta z_{i} \\left\\langle\\operatorname{grad}g_{i}(x), \\Delta x\\right\\rangle \\\\\n\t&= \\left\\langle\\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x], \\Delta x\\right\\rangle.\n\\end{align*}\nThus, from second-order sufficiency \\ref{B5}, $ \\Delta x$ must be zero elements.\nAnd then $\\Delta s_{i}=0 \\text { for all } i\\in \\mathbb{I} \\setminus  \\mathbb{A}.$\nNext, substituting $ \\Delta x=0$ into the first equation in (\\ref{sing1}) yields\n$\n0=\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n+\\sum_{i\\in  \\mathbb{A}} \\Delta z_{i} \\operatorname{grad} g_{i}(x).\n$\nThe LICQ \\ref{B3} implies that the coefficients $\\Delta y_{i}$ for $i\\in \\mathbb{E}$ and $ \\Delta z_{i} $ for $ i\\in  \\mathbb{A} $ must be zero. \nThis completes the proof.\n\\end{proof}","start":23874,"end":27407}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["13"],"declared_external":false,"source_only":false,"title":"展开核方程","source_units":[{"raw_tex":"\\begin{proof}\nThis proof omits all the asterisks of the variables. Define $ \\mathbb{E}:=\\{1,\\dots,l\\}$ and $\\mathbb{I}:=\\{1,\\dots,m\\} $. Take some $w=(x, y, z, s) \\in \\mathcal{N} $ satisfying \\ref{B1}-\\ref{B5}, then we have $ s_{i} = -g_{i}(x)$ and $z_{i} s_{i} = 0$ for all $ i \\in \\mathbb{I}$. For short, let $\\mathbb{A}:=\\mathbb{A}(x) \\subset \\mathbb{I} $. \nSuppose that $\\nabla {F}(w) [\\Delta w] =0$ for some $\\Delta w = (\\Delta x, \\Delta y, \\Delta z, \\Delta s) \\in T_{w} \\mathcal{N} \\cong T_x \\mathcal{M} \\times \\mathbb{R}^l \\times \\mathbb{R}^m \\times \\mathbb{R}^m$.\n$\\Delta y_i$ denotes the components of the vector $\\Delta y$, as do $\\Delta z_i$, $\\Delta s_i$.\nTo prove its nonsingularity, we will show that $ \\Delta w=0$.\nExpanding the equation $\\nabla {F}(w) [\\Delta w] =0$ gives\n\\begin{equation}\\label{13}\n\t\\left\\{\n\t\\begin{aligned}\n\t\t0&= \\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n\t\t+\\sum_{i\\in \\mathbb{I}} \\Delta z_{i} \\operatorname{grad} g_{i}(x),\\\\\n\t\t0&=\\left\\langle\\operatorname{grad}h_{i}(x), \\Delta x\\right\\rangle, \\text { for all } i\\in \\mathbb{E},\\\\\n\t\t0&=\\left\\langle\\operatorname{grad} g_{i}(x), \\Delta x\\right\\rangle + \\Delta s_{i}, \\text { for all } i\\in \\mathbb{I}, \\\\\n\t\t0&=z_{i} \\Delta s_{i} + s_{i} \\Delta z_{i}, \\text { for all } i \\in \\mathbb{I}. \n\t\\end{aligned}\n\t\\right.\n\\end{equation}\n","start":23874,"end":25262}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["sing1"],"declared_external":false,"source_only":false,"title":"严格互补约化核系统","source_units":[{"raw_tex":"Strict complementarity \\ref{B4} and the last equalities above imply that $ \\Delta s_{i}=0 $\nfor all $ i \\in \\mathbb{A}$ and $ \\Delta z_{i}=0 $\nfor $ i \\in  \\mathbb{I} \\setminus \\mathbb{A}$.\nSubstituting those values into the system (\\ref{13}) reduces it to\n\\begin{equation}\\label{sing1}\n\t\\left\\{\n\t\\begin{aligned}\n\t\t0&= \\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n\t\t+\\sum_{i\\in \\mathbb{A}} \\Delta z_{i} \\operatorname{grad} g_{i}(x), \\\\\n\t\t0&=\\left\\langle\\operatorname{grad}h_{i}(x), \\Delta x\\right\\rangle, \\text { for all } i\\in \\mathbb{E},\\\\\n\t\t0&=\\left\\langle\\operatorname{grad}g_{i}(x), \\Delta x\\right\\rangle, \\text { for all } i\\in \\mathbb{A},\n\t\\end{aligned}\n\t\\right.\n\\end{equation}\nand\n$\n\\Delta s_{i}=-\\left\\langle\\operatorname{grad} g_{i}(x), \\Delta x\\right\\rangle \\text { for all } i\\in \\mathbb{I} \\setminus \\mathbb{A}.\n$\n","start":25262,"end":26162}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"二阶充分性消去主变量","source_units":[{"raw_tex":"It follows from system (\\ref{sing1}) that\n\\begin{align*}\n\t0\n\t&=  \\langle \\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n\t+\\sum_{i\\in  \\mathbb{A}} \\Delta z_{i} \\operatorname{grad} g_{i}(x) , \\Delta x \\rangle \\\\\n\t&= \\left\\langle\\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x], \\Delta x\\right\\rangle\n\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\left\\langle\\operatorname{grad}h_{i}(x), \\Delta x\\right\\rangle\n\t+\\sum_{i\\in  \\mathbb{A}} \\Delta z_{i} \\left\\langle\\operatorname{grad}g_{i}(x), \\Delta x\\right\\rangle \\\\\n\t&= \\left\\langle\\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x], \\Delta x\\right\\rangle.\n\\end{align*}\nThus, from second-order sufficiency \\ref{B5}, $ \\Delta x$ must be zero elements.\nAnd then $\\Delta s_{i}=0 \\text { for all } i\\in \\mathbb{I} \\setminus  \\mathbb{A}.$\n","start":26162,"end":27004}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"LICQ 消去剩余乘子","source_units":[{"raw_tex":"Next, substituting $ \\Delta x=0$ into the first equation in (\\ref{sing1}) yields\n$\n0=\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n+\\sum_{i\\in  \\mathbb{A}} \\Delta z_{i} \\operatorname{grad} g_{i}(x).\n$\nThe LICQ \\ref{B3} implies that the coefficients $\\Delta y_{i}$ for $i\\in \\mathbb{E}$ and $ \\Delta z_{i} $ for $ i\\in  \\mathbb{A} $ must be zero. \nThis completes the proof.\n\\end{proof}","start":27004,"end":27407}]},{"id":"S049","semantic_type":"lemma","kind":"lemma","labels":["lem:R1est"],"declared_external":false,"source_only":false,"title":"向量场的一阶展开余项","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:R1est}\n\tLet $F$ be a $ C^2 $ vector field and $\\bar{x}\\in\\mathcal{M}$. Then there exist a neighborhood $\\mathcal{U}$ of $\\bar{x}$ and a constant $c_{2}>0$ such that for all $x \\in \\mathcal{U}$,\n\t\\begin{equation*}\n\t\\left\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} [F(x)]-F(\\bar{x})-\\nabla F(\\bar{x}) [\\eta] \n\t\\right\\|\n\t \\leq c_{2} d^{2}(\\bar{x} , x),\n\t\\end{equation*}\n\twhere $\\eta=\\mathrm{R} _{\\bar{x}}^{-1} x$ and $\\mathrm{P}_{\\gamma}$ is the parallel transport along the curve $\\gamma(t):=\\mathrm{R}_{\\bar{x}}(t \\eta)$.\n\\end{lemma}","start":53373,"end":53917}]},{"id":"S050","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"二阶余项界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof} \nLet $LHS:=\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} [F(x)]-F(\\bar{x})-\\nabla F(\\bar{x}) [\\eta] \\|.$ It follows that\n\\begin{equation*}\n\t\t\\begin{aligned}\n\t\t\tLHS\n\t\t\t& \\leq\n\t\t\t\\left\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} [F(x)]-F(\\bar{x})\n\t\t\t-\\int_{0}^{1} \\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} [\\eta] \\mathrm{d} t  \\right\\| \\\\\n\t\t\t& \\quad +\n\t\t\t\\left\\|\\int_{0}^{1} \\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} [\\eta] \\mathrm{d} t -\\nabla F(\\bar{x}) [\\eta] \\right\\| \\\\\n\t\t\t&\\leq \n\t\t\tc_{1} \\|\\eta\\|^{2} +\n\t\t\t\\Big\\|\n\t\t\t\\int_{0}^{1}\\left(\\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} -\\nabla F(\\bar{x})\\right)\n\t\t\t[\\eta] \\mathrm{d} t\\Big\\|.\n\t\t\t\\text{ (by Lemma \\ref{lem:FundamentalTheoremOfCalculusRiemannian})}\n\t\t\\end{aligned}\n\\end{equation*}\nLet\n$\\theta := \\left\\| \\int_{0}^{1}\\left(\\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} -\\nabla F(\\bar{x})\\right)\n[\\eta] \\mathrm{d} t \\right\\|$.\nNote that\n\\begin{align*}\n\t\\theta \n\t&\\leq \n\t\\int_{0}^{1} \n\t\\left\\|\n\t\\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} -\\nabla F(\\bar{x})\n\t\\right\\|\n\t\\left\\| \\eta \\right\\|\n\t\\mathrm{d} t \\\\\n\t&\\leq  \n\t\\int_{0}^{1} \n\tc_{0} d(\\bar{x}, \\mathrm{R}_{\\bar{x}}(t \\eta))\n\t\\left\\| \\eta \\right\\|\n\t\\mathrm{d} t\t\\text{ (by Lemma \\ref{lem:LipCountNabulaF})} \\\\\n\t&\\leq   \n\t\\int_{0}^{1} \n\tc_{0} a_{1}t \\|\\eta\\|\n\t\\left\\| \\eta \\right\\|\n\t\\mathrm{d} t \t\n\t= \n\t\\frac{1}{2} \n\tc_{0} a_{1} \\|\\eta\\|^{2}. \t\n\t\\text{ (by (i) of Lemma \\ref{lem:distR1})}\n\\end{align*}\nCombining the above results yields\n$$\nLHS\n\\leq (c_{1}+ \\frac{1}{2} \nc_{0} a_{1})\n\\|\\eta\\|^{2}\n\\leq  \n(c_{1}+ \\frac{1}{2} \nc_{0} a_{1})  / a_{0}^{2}\nd^{2}(\\bar{x} , x),\n$$\nwhere the last inequality comes from (ii) of Lemma \\ref{lem:distR1}. Letting $c_{2}:= (c_{1}+ \\frac{1}{2} \nc_{0} a_{1})  / a_{0}^{2} $ completes the proof.\n\\end{proof}","start":53918,"end":55838}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"基本估计分离余项","source_units":[{"raw_tex":"\\begin{proof} \nLet $LHS:=\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} [F(x)]-F(\\bar{x})-\\nabla F(\\bar{x}) [\\eta] \\|.$ It follows that\n\\begin{equation*}\n\t\t\\begin{aligned}\n\t\t\tLHS\n\t\t\t& \\leq\n\t\t\t\\left\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} [F(x)]-F(\\bar{x})\n\t\t\t-\\int_{0}^{1} \\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} [\\eta] \\mathrm{d} t  \\right\\| \\\\\n\t\t\t& \\quad +\n\t\t\t\\left\\|\\int_{0}^{1} \\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} [\\eta] \\mathrm{d} t -\\nabla F(\\bar{x}) [\\eta] \\right\\| \\\\\n\t\t\t&\\leq \n\t\t\tc_{1} \\|\\eta\\|^{2} +\n\t\t\t\\Big\\|\n\t\t\t\\int_{0}^{1}\\left(\\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} -\\nabla F(\\bar{x})\\right)\n\t\t\t[\\eta] \\mathrm{d} t\\Big\\|.\n\t\t\t\\text{ (by Lemma \\ref{lem:FundamentalTheoremOfCalculusRiemannian})}\n\t\t\\end{aligned}\n\\end{equation*}\n","start":53918,"end":54758}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"积分项的二次估计","source_units":[{"raw_tex":"Let\n$\\theta := \\left\\| \\int_{0}^{1}\\left(\\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} -\\nabla F(\\bar{x})\\right)\n[\\eta] \\mathrm{d} t \\right\\|$.\nNote that\n\\begin{align*}\n\t\\theta \n\t&\\leq \n\t\\int_{0}^{1} \n\t\\left\\|\n\t\\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} -\\nabla F(\\bar{x})\n\t\\right\\|\n\t\\left\\| \\eta \\right\\|\n\t\\mathrm{d} t \\\\\n\t&\\leq  \n\t\\int_{0}^{1} \n\tc_{0} d(\\bar{x}, \\mathrm{R}_{\\bar{x}}(t \\eta))\n\t\\left\\| \\eta \\right\\|\n\t\\mathrm{d} t\t\\text{ (by Lemma \\ref{lem:LipCountNabulaF})} \\\\\n\t&\\leq   \n\t\\int_{0}^{1} \n\tc_{0} a_{1}t \\|\\eta\\|\n\t\\left\\| \\eta \\right\\|\n\t\\mathrm{d} t \t\n\t= \n\t\\frac{1}{2} \n\tc_{0} a_{1} \\|\\eta\\|^{2}. \t\n\t\\text{ (by (i) of Lemma \\ref{lem:distR1})}\n\\end{align*}\n","start":54758,"end":55500}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"转为距离平方余项","source_units":[{"raw_tex":"Combining the above results yields\n$$\nLHS\n\\leq (c_{1}+ \\frac{1}{2} \nc_{0} a_{1})\n\\|\\eta\\|^{2}\n\\leq  \n(c_{1}+ \\frac{1}{2} \nc_{0} a_{1})  / a_{0}^{2}\nd^{2}(\\bar{x} , x),\n$$\nwhere the last inequality comes from (ii) of Lemma \\ref{lem:distR1}. Letting $c_{2}:= (c_{1}+ \\frac{1}{2} \nc_{0} a_{1})  / a_{0}^{2} $ completes the proof.\n\\end{proof}","start":55500,"end":55838}]},{"id":"S054","semantic_type":"proposition","kind":"proposition","labels":["prop:PD_Newton"],"declared_external":false,"source_only":false,"title":"扰动阻尼 Newton 局部收敛","source_units":[{"raw_tex":"\\begin{proposition}[Local convergence of Algorithm \\ref{algo:PerturbedDampedNewtonMethod}]\\label{prop:PD_Newton}\n\tConsider the perturbed damped Riemannian Newton method\n\tdescribed in Algorithm \\ref{algo:PerturbedDampedNewtonMethod} for the singularity problem (\\ref{eq:Singularity}). \n\tLet \\ref{A1}-\\ref{A3} hold.\n\tChoose parameters $ \\mu_{k}, \\alpha_{k}$ as follows; then there exists a constant $\\delta>0$ such that for all $p_{0}\\in \\mathcal{M}$ with\n\t$\n\td(p_{0} , p^{*})<\\delta,\n\t$\n\tthe sequence $\\left\\{p_{k}\\right\\}$ is well defined.\n\tFurthermore,\n\t\\begin{enumerate}[(1)]\n\t\t\\item if we choose $\\mu_{k}=o(\\|F(p_{k})\\|)$ and $\\alpha_{k} \\to 1$, then $p_{k} \\to p^{*}$ superlinearly;\t\t\n\t\t\\item if we choose $\\mu_{k}=O(\\|F(p_{k})\\|^{2})$ and $1-\\alpha_{k}=O(\\|F(p_{k})\\|)$, then $p_{k} \\to p^{*}$ quadratically.\n\t\\end{enumerate}\n\\end{proposition}","start":60387,"end":61235}]},{"id":"S055","semantic_type":"proof","kind":"proof","labels":["77","eq:1114","eq:LipFbigoh","eq:bigoh","eq:superlinear"],"declared_external":false,"source_only":false,"title":"扰动阻尼 Newton 的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nBy \\ref{A2}-\\ref{A3}, Lemma \\ref{lem:LipCountNabulaF} and Lemma \\ref{lem:nonsingular}, we can let $p_{k} $ be sufficiently close to $p^{*}$ such that $\\nabla F(p_{k})$ is nonsingular, and\n$\\left\\|\\nabla F(p_{k})^{-1}\\right\\| \\leq \\Xi$ for some constant $\\Xi$.\nThen, the next iterate point,\n\\begin{equation*}\np_{k+1}:=\\mathrm{R}_{p_{k}}[\\alpha_{k} \\nabla F(p_{k})^{-1}(-F(p_{k})+\\mu_{k} \\hat{e})],\n\\end{equation*}\nis well defined in Algorithm \\ref{algo:PerturbedDampedNewtonMethod}, and it follows from $ p^{*}=\\mathrm{R}_{p_{k}}(\\eta) $ with $ \\eta := \\mathrm{R}_{p_{k}}^{-1} p^{*} $ and (i) of Lemma \\ref{lem:distR1} that\n\\begin{align}\n\td(p_{k+1} , p^{*})\n\t&\\leq\n\ta_{1}\n\t\\| \\eta - \\alpha_{k} \\nabla F(p_{k})^{-1}(-F(p_{k})+\\mu_{k} \\hat{e}) \\| \\notag \\\\\n\t&=\n\ta_{1}  \n\t\\|\\eta + \\alpha_{k} \\nabla F(p_{k})^{-1}(F(p_{k})-\\mu_{k} \\hat{e})\\|. \\label{eq:1114}\n\\end{align}\nLet \n$r_{k} := \\eta + \\alpha_{k} \\nabla F(p_{k})^{-1}(F(p_{k})-\\mu_{k} \\hat{e})$.\nAlgebraic manipulations\nshow that\n\\begin{equation*}\nr_{k}=\n(1-\\alpha_{k}) \\eta  +\n\\alpha_{k} \\nabla F(p_{k})^{-1} \n[\\nabla F(p_{k}) \\eta + F(p_{k})-\\mathrm{P}_{\\gamma}^{0 \\to 1} F(p^{*})-\\mu_{k} \\hat{e}],\n\\end{equation*}\nwhere $\\mathrm{P}_{\\gamma}$ is the parallel transport along the curve $\\gamma(t)=\\mathrm{R}_{p_{k}}(t \\eta)$ and $F(p^{*})=0.$\nThus, using $\\left\\|\\eta\\right\\| \\leq d(p_{k}, p^{*})/a_{0}$ from (ii) of Lemma \\ref{lem:distR1} and Lemma \\ref{lem:R1est}, we have\n\\begin{align*}\n\t\\| r_{k}\\|&\\leq\n\t(1-\\alpha_{k}) \\| \\eta\\| \\notag \n\t+ \n\t\\alpha_{k} \\|\\nabla F(p_{k})^{-1} \\|\n\t\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} F(p^{*}) - F(p_{k}) -\n\t\\nabla F(p_{k}) \\eta \\| \\\\\n\t& \\quad + \\alpha_{k} \\|\\nabla F(p_{k})^{-1} \\| \\|\\hat{e}\\| \\mu_{k} \\\\ \n\t&\\leq (1-\\alpha_{k})d(p_{k} , p^{*}) /a_{0} +\n\t\\alpha_{k} \\|\\nabla F(p_{k})^{-1} \\| \n\tc_{2} d^{2}(p_{k} , p^{*})\\\\\n\t& \\quad\n\t+ \\alpha_{k}\\|\\nabla F(p_{k})^{-1} \\|  \\|\\hat{e}\\| \\mu_{k}\\\\\n\t&\\leq (1-\\alpha_{k})d(p_{k} , p^{*}) /a_{0} +\n\t\\Xi \n\tc_{2} d^{2}(p_{k} , p^{*})\n\t+ \\Xi  \\|\\hat{e}\\| \\mu_{k}.\n\\end{align*}\nCombining the above with (\\ref{eq:1114}), we conclude that\n\\begin{equation}\\label{77}\n\td(p_{k+1} , p^{*}) \\leq \\kappa_{1}(1-\\alpha_{k}) d(p_{k} , p^{*}) \n\t+ \\kappa_{2} d^{2}(p_{k} , p^{*})\n\t+ \\kappa_{3} \\mu_{k}\n\\end{equation}\nfor some positive constants $ \\kappa_{1},\\kappa_{2},\\kappa_{3}$.\nOn the other hand, by Lemma \\ref{lem:dist_zero}, we have\n\\begin{equation}\\label{eq:LipFbigoh}\n\t\\left\\|F(p_{k})\\right\\|=O(d(p_{k}, p^{*})).\n\\end{equation}\nIn what follows, we prove assertions (1) and (2).\n\n\\textbf{(1)} \nSuppose that $\\alpha_{k} \\to 1$ and $\\mu_{k}=o(\\|F(p_{k})\\|)$, which together with (\\ref{eq:LipFbigoh}) imply $\\mu_{k}=o(d(p_{k}, p^{*}))$.\nBy (\\ref{77}), we have\n\\begin{equation}\\label{eq:superlinear}\n\t\\frac{d(p_{k+1} , p^{*})}{d(p_{k} , p^{*})}\\leq\n\t\\kappa_{1} (1-\\alpha_{k})\n\t+\\kappa_{2} d(p_{k} , p^{*})\n\t+\\kappa_{3}\\frac{\\mu_{k}}{d(p_{k} , p^{*})}.\n\\end{equation}\nWe can take $\\delta$ sufficiently small and $ k $ sufficiently large, if necessary, to conclude that\n$\nd(p_{k+1} , p^{*})<\\frac{1}{2}d(p_{k} , p^{*}) <\\delta.\n$\nThus, $p_{k+1} \\in B_{\\delta}(p^{*})$, the open ball of radius $\\delta$ centered at $p^{*}$ on $\\mathcal{M}$. \nBy induction, it is easy to show that the sequence $\\left\\{p_{k}\\right\\}$ is well defined and converges to $p^{*}$. \nTaking the limit of both sides of (\\ref{eq:superlinear}) proves superlinear convergence.\n\n\n\\textbf{(2)} \nAgain, we start from (\\ref{77}) and rewrite it as:\n\\begin{align}\\label{eq:bigoh}\n\td(p_{k+1} , p^{*}) = (1-\\alpha_{k}) O(d(p_{k} , p^{*}) )\n\t+O(d^{2}(p_{k} , p^{*}) )\n\t+O(\\mu_{k}).\n\\end{align}\nSuppose that $1-\\alpha_{k}=O(\\left\\|F(p_{k})\\right\\|)$\nand $\\mu_{k}=O(\\left\\|F(p_{k})\\right\\|^{2})$.\nUsing (\\ref{eq:LipFbigoh}), the above reduces to\n$\nd(p_{k+1} , p^{*}) = O(d^{2}(p_{k} , p^{*})).\n$\nThis implies that there exists a constant $\\nu$ such that $d(p_{k+1} , p^{*}) \\leq \\nu d^{2}(p_{k} , p^{*})$, and hence,\n$\nd(p_{k+1} , p^{*}) \\leq \\nu d^{2}(p_{k} , p^{*}) \\leq \\nu \\delta^{2}<\\delta,\n$\nif $\\delta$ is sufficiently small.\nAgain, by induction, $\\left\\{p_{k}\\right\\}$ converges to $p^{*}$ quadratically.\n\\end{proof}","start":61236,"end":65344}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"局部可逆性与更新定义","source_units":[{"raw_tex":"\\begin{proof}\nBy \\ref{A2}-\\ref{A3}, Lemma \\ref{lem:LipCountNabulaF} and Lemma \\ref{lem:nonsingular}, we can let $p_{k} $ be sufficiently close to $p^{*}$ such that $\\nabla F(p_{k})$ is nonsingular, and\n$\\left\\|\\nabla F(p_{k})^{-1}\\right\\| \\leq \\Xi$ for some constant $\\Xi$.\nThen, the next iterate point,\n\\begin{equation*}\np_{k+1}:=\\mathrm{R}_{p_{k}}[\\alpha_{k} \\nabla F(p_{k})^{-1}(-F(p_{k})+\\mu_{k} \\hat{e})],\n\\end{equation*}\n","start":61236,"end":61663}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":["eq:1114"],"declared_external":false,"source_only":false,"title":"回缩距离误差界","source_units":[{"raw_tex":"is well defined in Algorithm \\ref{algo:PerturbedDampedNewtonMethod}, and it follows from $ p^{*}=\\mathrm{R}_{p_{k}}(\\eta) $ with $ \\eta := \\mathrm{R}_{p_{k}}^{-1} p^{*} $ and (i) of Lemma \\ref{lem:distR1} that\n\\begin{align}\n\td(p_{k+1} , p^{*})\n\t&\\leq\n\ta_{1}\n\t\\| \\eta - \\alpha_{k} \\nabla F(p_{k})^{-1}(-F(p_{k})+\\mu_{k} \\hat{e}) \\| \\notag \\\\\n\t&=\n\ta_{1}  \n\t\\|\\eta + \\alpha_{k} \\nabla F(p_{k})^{-1}(F(p_{k})-\\mu_{k} \\hat{e})\\|. \\label{eq:1114}\n\\end{align}\n","start":61663,"end":62116}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"分离阻尼与线性化余项","source_units":[{"raw_tex":"Let \n$r_{k} := \\eta + \\alpha_{k} \\nabla F(p_{k})^{-1}(F(p_{k})-\\mu_{k} \\hat{e})$.\nAlgebraic manipulations\nshow that\n\\begin{equation*}\nr_{k}=\n(1-\\alpha_{k}) \\eta  +\n\\alpha_{k} \\nabla F(p_{k})^{-1} \n[\\nabla F(p_{k}) \\eta + F(p_{k})-\\mathrm{P}_{\\gamma}^{0 \\to 1} F(p^{*})-\\mu_{k} \\hat{e}],\n\\end{equation*}\nwhere $\\mathrm{P}_{\\gamma}$ is the parallel transport along the curve $\\gamma(t)=\\mathrm{R}_{p_{k}}(t \\eta)$ and $F(p^{*})=0.$\n","start":62116,"end":62546}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"切空间误差的三项界","source_units":[{"raw_tex":"Thus, using $\\left\\|\\eta\\right\\| \\leq d(p_{k}, p^{*})/a_{0}$ from (ii) of Lemma \\ref{lem:distR1} and Lemma \\ref{lem:R1est}, we have\n\\begin{align*}\n\t\\| r_{k}\\|&\\leq\n\t(1-\\alpha_{k}) \\| \\eta\\| \\notag \n\t+ \n\t\\alpha_{k} \\|\\nabla F(p_{k})^{-1} \\|\n\t\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} F(p^{*}) - F(p_{k}) -\n\t\\nabla F(p_{k}) \\eta \\| \\\\\n\t& \\quad + \\alpha_{k} \\|\\nabla F(p_{k})^{-1} \\| \\|\\hat{e}\\| \\mu_{k} \\\\ \n\t&\\leq (1-\\alpha_{k})d(p_{k} , p^{*}) /a_{0} +\n\t\\alpha_{k} \\|\\nabla F(p_{k})^{-1} \\| \n\tc_{2} d^{2}(p_{k} , p^{*})\\\\\n\t& \\quad\n\t+ \\alpha_{k}\\|\\nabla F(p_{k})^{-1} \\|  \\|\\hat{e}\\| \\mu_{k}\\\\\n\t&\\leq (1-\\alpha_{k})d(p_{k} , p^{*}) /a_{0} +\n\t\\Xi \n\tc_{2} d^{2}(p_{k} , p^{*})\n\t+ \\Xi  \\|\\hat{e}\\| \\mu_{k}.\n\\end{align*}\n","start":62546,"end":63251}]},{"id":"S060","semantic_type":"derived_fact","kind":"derived_fact","labels":["77"],"declared_external":false,"source_only":false,"title":"Newton 统一距离递推","source_units":[{"raw_tex":"Combining the above with (\\ref{eq:1114}), we conclude that\n\\begin{equation}\\label{77}\n\td(p_{k+1} , p^{*}) \\leq \\kappa_{1}(1-\\alpha_{k}) d(p_{k} , p^{*}) \n\t+ \\kappa_{2} d^{2}(p_{k} , p^{*})\n\t+ \\kappa_{3} \\mu_{k}\n\\end{equation}\nfor some positive constants $ \\kappa_{1},\\kappa_{2},\\kappa_{3}$.\n","start":63251,"end":63542}]},{"id":"S061","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:LipFbigoh"],"declared_external":false,"source_only":false,"title":"残差的距离阶上界","source_units":[{"raw_tex":"On the other hand, by Lemma \\ref{lem:dist_zero}, we have\n\\begin{equation}\\label{eq:LipFbigoh}\n\t\\left\\|F(p_{k})\\right\\|=O(d(p_{k}, p^{*})).\n\\end{equation}\n","start":63542,"end":63696}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":["eq:superlinear"],"declared_external":false,"source_only":false,"title":"超线性情形的比值估计","source_units":[{"raw_tex":"In what follows, we prove assertions (1) and (2).\n\n\\textbf{(1)} \nSuppose that $\\alpha_{k} \\to 1$ and $\\mu_{k}=o(\\|F(p_{k})\\|)$, which together with (\\ref{eq:LipFbigoh}) imply $\\mu_{k}=o(d(p_{k}, p^{*}))$.\nBy (\\ref{77}), we have\n\\begin{equation}\\label{eq:superlinear}\n\t\\frac{d(p_{k+1} , p^{*})}{d(p_{k} , p^{*})}\\leq\n\t\\kappa_{1} (1-\\alpha_{k})\n\t+\\kappa_{2} d(p_{k} , p^{*})\n\t+\\kappa_{3}\\frac{\\mu_{k}}{d(p_{k} , p^{*})}.\n\\end{equation}\nWe can take $\\delta$ sufficiently small and $ k $ sufficiently large, if necessary, to conclude that\n$\nd(p_{k+1} , p^{*})<\\frac{1}{2}d(p_{k} , p^{*}) <\\delta.\n$\nThus, $p_{k+1} \\in B_{\\delta}(p^{*})$, the open ball of radius $\\delta$ centered at $p^{*}$ on $\\mathcal{M}$. \nBy induction, it is easy to show that the sequence $\\left\\{p_{k}\\right\\}$ is well defined and converges to $p^{*}$. \nTaking the limit of both sides of (\\ref{eq:superlinear}) proves superlinear convergence.\n","start":63696,"end":64608}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":["eq:bigoh"],"declared_external":false,"source_only":false,"title":"二次情形与邻域归纳","source_units":[{"raw_tex":"\n\n\\textbf{(2)} \nAgain, we start from (\\ref{77}) and rewrite it as:\n\\begin{align}\\label{eq:bigoh}\n\td(p_{k+1} , p^{*}) = (1-\\alpha_{k}) O(d(p_{k} , p^{*}) )\n\t+O(d^{2}(p_{k} , p^{*}) )\n\t+O(\\mu_{k}).\n\\end{align}\nSuppose that $1-\\alpha_{k}=O(\\left\\|F(p_{k})\\right\\|)$\nand $\\mu_{k}=O(\\left\\|F(p_{k})\\right\\|^{2})$.\nUsing (\\ref{eq:LipFbigoh}), the above reduces to\n$\nd(p_{k+1} , p^{*}) = O(d^{2}(p_{k} , p^{*})).\n$\nThis implies that there exists a constant $\\nu$ such that $d(p_{k+1} , p^{*}) \\leq \\nu d^{2}(p_{k} , p^{*})$, and hence,\n$\nd(p_{k+1} , p^{*}) \\leq \\nu d^{2}(p_{k} , p^{*}) \\leq \\nu \\delta^{2}<\\delta,\n$\nif $\\delta$ is sufficiently small.\nAgain, by induction, $\\left\\{p_{k}\\right\\}$ converges to $p^{*}$ quadratically.\n\\end{proof}","start":64608,"end":65344}]},{"id":"S064","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"步长引理的外部委托证明","source_units":[{"raw_tex":"\\begin{proof} \nNotice that the fourth line of (\\ref{eq:FullNewtonEquation}) yields\n$$\nS_{k}^{-1} \\Delta s_{k}+Z_{k}^{-1} \\Delta z_{k}=\\mu_{k}(S_{k} Z_{k})^{-1} e-e,\n$$\nwhich is exactly the same as in the usual interior point method in the Euclidean setting. \nThus, the proof entails directly applying \\cite[Lemma 3 and 4]{yamashita1996superlinear} for the Euclidean case to the Riemannian case.\n\\end{proof}\n","start":66212,"end":66619}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"互补方程的相对增量形式","source_units":[{"raw_tex":"\\begin{proof} \nNotice that the fourth line of (\\ref{eq:FullNewtonEquation}) yields\n$$\nS_{k}^{-1} \\Delta s_{k}+Z_{k}^{-1} \\Delta z_{k}=\\mu_{k}(S_{k} Z_{k})^{-1} e-e,\n$$\nwhich is exactly the same as in the usual interior point method in the Euclidean setting. \n","start":66212,"end":66471}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"直接援引欧氏步长引理","source_units":[{"raw_tex":"Thus, the proof entails directly applying \\cite[Lemma 3 and 4]{yamashita1996superlinear} for the Euclidean case to the Riemannian case.\n\\end{proof}\n","start":66471,"end":66619}]},{"id":"S067","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件清单：Yamashita and Yabe (1996), Lemmas 3 and 4；原文 1063 行直接引用。"],"source_only":false,"title":"欧氏内点法步长引理","source_units":[]},{"id":"S068","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件清单：Inverse function theorem；Zhu and Sato (2020), Theorem 2。"],"source_only":false,"title":"逆函数与全回缩邻域定理","source_units":[]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件清单中的 standard Riemannian product-connection properties；原文 300 行另引 Boumal 的 Exercises 5.4 and 5.13。hat e 范数恒定这一省略背景由标注者显式化。"],"source_only":false,"title":"黎曼乘积结构的标准性质","source_units":[]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件清单中的 standard finite-dimensional linear algebra；具体平凡核判据为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"有限维平凡核判据","source_units":[]},{"id":"S071","semantic_type":"theorem","kind":"theorem","labels":["thm:LocalConvergence"],"declared_external":false,"source_only":false,"title":"RIPM 原型的局部收敛","source_units":[{"raw_tex":"\\begin{theorem}[Local convergence of prototype Algorithm \\ref{algo:prototype_algo}]\\label{thm:LocalConvergence}\n\tConsider the Algorithm \\ref{algo:prototype_algo} for solving the problem (\\ref{RCOP}).  \n\tLet \\ref{B1}-\\ref{B5} hold at some $w^{*}$.\n\tChoose parameters $\\mu_k, \\gamma_k$ as follows;\n\tthen there exists a constant $\\delta>0$ such that, for all\n\t$\n\tw_{0} \\in\\mathcal{N}\n\t$\n\twith $d(w_{0} , w^{*})<\\delta,$\n\tthe sequence $\\left\\{w_{k}\\right\\}$ is well defined. Furthermore,\n\t\\begin{enumerate}[(1)]\n\t\t\\item if we choose $\\mu_{k}=o(\\left\\|F(w_{k})\\right\\|)$ and $\\gamma_{k} \\to 1$, then $w_k \\to w^*$ superlinearly;\t\n\t\t\\item if we choose $\\mu_{k}=O(\\left\\|F(w_{k})\\right\\|^{2})$\n\t\tand $1-\\gamma_{k}=O(\\left\\|F(w_{k})\\right\\|)$, then $w_k \\to w^*$ quadratically.\n\t\\end{enumerate}\n\\end{theorem}","start":66819,"end":67619}]},{"id":"S072","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"局部收敛定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nWe only prove (2) because (1) can be proven in the same way.\nLet $w_{k} $ be such that  $d(w_{k} , w^{*})<\\delta$ for sufficiently small $\\delta$.\nFrom Proposition \\ref{prop:non}, \\ref{B1}-\\ref{B5} shows that the KKT vector field $F$ satisfies \\ref{A1}-\\ref{A3}; thus, the discussion in the proof of Proposition \\ref{prop:PD_Newton} applies to KKT vector field $F$ as well, simply by replacing the symbol $p$ with $w$.\nSince we choose $\\mu_{k}=O(\\|F(w_{k})\\|^{2})$, and $\\|F(w_{k})\\|=O(d(w_{k}, w^{*}))$ by (\\ref{eq:LipFbigoh}), we obtain\n$\n\\mu_{k}=O(d^{2}(w_{k}, w^{*})).\n$\nThus,\n\\begin{align*}\n\t \\left\\|\\Delta w_{k}\\right\\|  & =\n\t\\left\\|\n\t\\nabla F(w_{k})^{-1} (-F(w_{k})+\\mu_{k} \\hat{e})\n\t\\right\\| \\text{ (by (\\ref{eq:prototype}))}\\\\\n\t& \\leq \n\t\\Xi\n\t(\\left\\|F(w_{k})\\right\\|+\\mu_{k}\\|\\hat{e}\\|) \\text{ (by Lemma \\ref{lem:nonsingular})}\\\\\n\t& \\leq  O(\\|F(w_k)\\|)+O(\\mu_{k}) \\\\\n\t& =O(d(w_{k} , w^{*}))+O(d^{2}(w_{k} , w^{*}) )\n\t=O(d(w_{k} , w^{*})).\n\\end{align*}\nSince $\\delta$ is sufficiently small, from the above inequalities, the condition of Lemma \\ref{lem:gamma_alpha} is satisfied, i.e.,\n$\n\\left\\|\\Delta w_{k}\\right\\| \\leq \\hat{\\gamma} / \\Pi \\leq \\gamma_k / \\Pi\n$\nfor a constant $\\hat{\\gamma} \\in (0,1).$\nHence, by $1-\\gamma_k=O(\\left\\|F\\left(w_k\\right)\\right\\|)$, one has\n$\n1-\\alpha_{k} \\leq\n(1-\\gamma_{k})+\\Pi\\left\\|\\Delta w_{k}\\right\\| \n= (1-\\gamma_{k})+O(d(w_{k} , w^{*}))\n= O(d(w_{k} , w^{*})).\n$\nFinally, from (\\ref{eq:bigoh}),\nwe have\n$\nd(w_{k+1} , w^{*}) = (1-\\alpha_{k}) O(d(w_{k} , w^{*}) )\n+O(d^{2}(w_{k} , w^{*}) )\n+O(\\mu_{k}) \n=O(d^{2}(w_{k} , w^{*})).\n$\nThis completes the proof.\n\\end{proof}","start":67620,"end":69244}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"声明省略超线性情形","source_units":[{"raw_tex":"\\begin{proof}\nWe only prove (2) because (1) can be proven in the same way.\n","start":67620,"end":67695}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"在 KKT 场上实例化 Newton 假设","source_units":[{"raw_tex":"Let $w_{k} $ be such that  $d(w_{k} , w^{*})<\\delta$ for sufficiently small $\\delta$.\nFrom Proposition \\ref{prop:non}, \\ref{B1}-\\ref{B5} shows that the KKT vector field $F$ satisfies \\ref{A1}-\\ref{A3}; thus, the discussion in the proof of Proposition \\ref{prop:PD_Newton} applies to KKT vector field $F$ as well, simply by replacing the symbol $p$ with $w$.\n","start":67695,"end":68053}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"障碍参数具有距离平方阶","source_units":[{"raw_tex":"Since we choose $\\mu_{k}=O(\\|F(w_{k})\\|^{2})$, and $\\|F(w_{k})\\|=O(d(w_{k}, w^{*}))$ by (\\ref{eq:LipFbigoh}), we obtain\n$\n\\mu_{k}=O(d^{2}(w_{k}, w^{*})).\n$\n","start":68053,"end":68209}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"Newton 方向具有距离一阶界","source_units":[{"raw_tex":"Thus,\n\\begin{align*}\n\t \\left\\|\\Delta w_{k}\\right\\|  & =\n\t\\left\\|\n\t\\nabla F(w_{k})^{-1} (-F(w_{k})+\\mu_{k} \\hat{e})\n\t\\right\\| \\text{ (by (\\ref{eq:prototype}))}\\\\\n\t& \\leq \n\t\\Xi\n\t(\\left\\|F(w_{k})\\right\\|+\\mu_{k}\\|\\hat{e}\\|) \\text{ (by Lemma \\ref{lem:nonsingular})}\\\\\n\t& \\leq  O(\\|F(w_k)\\|)+O(\\mu_{k}) \\\\\n\t& =O(d(w_{k} , w^{*}))+O(d^{2}(w_{k} , w^{*}) )\n\t=O(d(w_{k} , w^{*})).\n\\end{align*}\n","start":68209,"end":68595}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"验证步长引理的小方向条件","source_units":[{"raw_tex":"Since $\\delta$ is sufficiently small, from the above inequalities, the condition of Lemma \\ref{lem:gamma_alpha} is satisfied, i.e.,\n$\n\\left\\|\\Delta w_{k}\\right\\| \\leq \\hat{\\gamma} / \\Pi \\leq \\gamma_k / \\Pi\n$\nfor a constant $\\hat{\\gamma} \\in (0,1).$\n","start":68595,"end":68844}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"中心参数转化为阻尼误差","source_units":[{"raw_tex":"Hence, by $1-\\gamma_k=O(\\left\\|F\\left(w_k\\right)\\right\\|)$, one has\n$\n1-\\alpha_{k} \\leq\n(1-\\gamma_{k})+\\Pi\\left\\|\\Delta w_{k}\\right\\| \n= (1-\\gamma_{k})+O(d(w_{k} , w^{*}))\n= O(d(w_{k} , w^{*})).\n$\n","start":68844,"end":69041}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"代入通用递推得到二次误差","source_units":[{"raw_tex":"Finally, from (\\ref{eq:bigoh}),\nwe have\n$\nd(w_{k+1} , w^{*}) = (1-\\alpha_{k}) O(d(w_{k} , w^{*}) )\n+O(d^{2}(w_{k} , w^{*}) )\n+O(\\mu_{k}) \n=O(d^{2}(w_{k} , w^{*})).\n$\nThis completes the proof.\n\\end{proof}","start":69041,"end":69244}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S071","edges":[{"source_id":"S001","target_id":"S006","kind":"supports"},{"source_id":"S002","target_id":"S006","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S005","target_id":"S008","kind":"supports"},{"source_id":"S003","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S004","target_id":"S011","kind":"supports"},{"source_id":"S069","target_id":"S011","kind":"supports"},{"source_id":"S006","target_id":"S012","kind":"supports"},{"source_id":"S004","target_id":"S012","kind":"supports"},{"source_id":"S007","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S015","kind":"supports"},{"source_id":"S010","target_id":"S015","kind":"supports"},{"source_id":"S013","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S017","kind":"supports"},{"source_id":"S014","target_id":"S017","kind":"supports"},{"source_id":"S008","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S069","target_id":"S020","kind":"supports"},{"source_id":"S008","target_id":"S021","kind":"supports"},{"source_id":"S009","target_id":"S021","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S005","target_id":"S021","kind":"supports"},{"source_id":"S011","target_id":"S023","kind":"supports"},{"source_id":"S008","target_id":"S023","kind":"supports"},{"source_id":"S021","target_id":"S023","kind":"supports"},{"source_id":"S003","target_id":"S024","kind":"supports"},{"source_id":"S068","target_id":"S024","kind":"supports"},{"source_id":"S003","target_id":"S025","kind":"supports"},{"source_id":"S002","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S027","kind":"supports"},{"source_id":"S003","target_id":"S027","kind":"supports"},{"source_id":"S004","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S025","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S028","target_id":"S032","kind":"supports"},{"source_id":"S025","target_id":"S032","kind":"supports"},{"source_id":"S024","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S024","target_id":"S036","kind":"supports"},{"source_id":"S027","target_id":"S036","kind":"supports"},{"source_id":"S004","target_id":"S036","kind":"supports"},{"source_id":"S014","target_id":"S043","kind":"supports"},{"source_id":"S015","target_id":"S043","kind":"supports"},{"source_id":"S016","target_id":"S043","kind":"supports"},{"source_id":"S017","target_id":"S043","kind":"supports"},{"source_id":"S011","target_id":"S043","kind":"supports"},{"source_id":"S014","target_id":"S045","kind":"supports"},{"source_id":"S011","target_id":"S045","kind":"supports"},{"source_id":"S013","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S016","target_id":"S046","kind":"supports"},{"source_id":"S014","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S017","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S015","target_id":"S048","kind":"supports"},{"source_id":"S070","target_id":"S048","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S024","target_id":"S049","kind":"supports"},{"source_id":"S027","target_id":"S049","kind":"supports"},{"source_id":"S036","target_id":"S051","kind":"supports"},{"source_id":"S033","target_id":"S052","kind":"supports"},{"source_id":"S035","target_id":"S052","kind":"supports"},{"source_id":"S027","target_id":"S052","kind":"supports"},{"source_id":"S028","target_id":"S052","kind":"supports"},{"source_id":"S051","target_id":"S053","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S035","target_id":"S053","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S004","target_id":"S040","kind":"supports"},{"source_id":"S003","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S054","kind":"supports"},{"source_id":"S041","target_id":"S054","kind":"supports"},{"source_id":"S039","target_id":"S054","kind":"supports"},{"source_id":"S041","target_id":"S056","kind":"supports"},{"source_id":"S033","target_id":"S056","kind":"supports"},{"source_id":"S037","target_id":"S056","kind":"supports"},{"source_id":"S040","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S035","target_id":"S057","kind":"supports"},{"source_id":"S024","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S041","target_id":"S058","kind":"supports"},{"source_id":"S027","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S056","target_id":"S059","kind":"supports"},{"source_id":"S035","target_id":"S059","kind":"supports"},{"source_id":"S049","target_id":"S059","kind":"supports"},{"source_id":"S040","target_id":"S059","kind":"supports"},{"source_id":"S057","target_id":"S060","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S041","target_id":"S061","kind":"supports"},{"source_id":"S038","target_id":"S061","kind":"supports"},{"source_id":"S060","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S039","target_id":"S062","kind":"supports"},{"source_id":"S060","target_id":"S063","kind":"supports"},{"source_id":"S061","target_id":"S063","kind":"supports"},{"source_id":"S039","target_id":"S063","kind":"supports"},{"source_id":"S056","target_id":"S055","kind":"supports"},{"source_id":"S057","target_id":"S055","kind":"supports"},{"source_id":"S058","target_id":"S055","kind":"supports"},{"source_id":"S059","target_id":"S055","kind":"supports"},{"source_id":"S060","target_id":"S055","kind":"supports"},{"source_id":"S061","target_id":"S055","kind":"supports"},{"source_id":"S062","target_id":"S055","kind":"supports"},{"source_id":"S063","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S054","kind":"supports"},{"source_id":"S014","target_id":"S042","kind":"supports"},{"source_id":"S016","target_id":"S042","kind":"supports"},{"source_id":"S021","target_id":"S042","kind":"supports"},{"source_id":"S023","target_id":"S065","kind":"supports"},{"source_id":"S021","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S067","target_id":"S066","kind":"supports"},{"source_id":"S021","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S042","kind":"supports"},{"source_id":"S001","target_id":"S071","kind":"supports"},{"source_id":"S021","target_id":"S071","kind":"supports"},{"source_id":"S014","target_id":"S071","kind":"supports"},{"source_id":"S015","target_id":"S071","kind":"supports"},{"source_id":"S016","target_id":"S071","kind":"supports"},{"source_id":"S017","target_id":"S071","kind":"supports"},{"source_id":"S039","target_id":"S071","kind":"supports"},{"source_id":"S001","target_id":"S074","kind":"supports"},{"source_id":"S008","target_id":"S074","kind":"supports"},{"source_id":"S014","target_id":"S074","kind":"supports"},{"source_id":"S015","target_id":"S074","kind":"supports"},{"source_id":"S016","target_id":"S074","kind":"supports"},{"source_id":"S017","target_id":"S074","kind":"supports"},{"source_id":"S043","target_id":"S074","kind":"supports"},{"source_id":"S041","target_id":"S074","kind":"supports"},{"source_id":"S040","target_id":"S074","kind":"supports"},{"source_id":"S009","target_id":"S074","kind":"supports"},{"source_id":"S021","target_id":"S074","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S061","target_id":"S075","kind":"supports"},{"source_id":"S039","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S076","kind":"supports"},{"source_id":"S037","target_id":"S076","kind":"supports"},{"source_id":"S021","target_id":"S076","kind":"supports"},{"source_id":"S061","target_id":"S076","kind":"supports"},{"source_id":"S075","target_id":"S076","kind":"supports"},{"source_id":"S019","target_id":"S076","kind":"supports"},{"source_id":"S069","target_id":"S076","kind":"supports"},{"source_id":"S076","target_id":"S077","kind":"supports"},{"source_id":"S021","target_id":"S077","kind":"supports"},{"source_id":"S042","target_id":"S077","kind":"supports"},{"source_id":"S077","target_id":"S078","kind":"supports"},{"source_id":"S042","target_id":"S078","kind":"supports"},{"source_id":"S076","target_id":"S078","kind":"supports"},{"source_id":"S061","target_id":"S078","kind":"supports"},{"source_id":"S074","target_id":"S079","kind":"supports"},{"source_id":"S063","target_id":"S079","kind":"supports"},{"source_id":"S075","target_id":"S079","kind":"supports"},{"source_id":"S078","target_id":"S079","kind":"supports"},{"source_id":"S073","target_id":"S072","kind":"supports"},{"source_id":"S074","target_id":"S072","kind":"supports"},{"source_id":"S075","target_id":"S072","kind":"supports"},{"source_id":"S076","target_id":"S072","kind":"supports"},{"source_id":"S077","target_id":"S072","kind":"supports"},{"source_id":"S078","target_id":"S072","kind":"supports"},{"source_id":"S079","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S071","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S071","edges":[{"source_id":"S001","target_id":"S006","kind":"supports"},{"source_id":"S002","target_id":"S006","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S005","target_id":"S008","kind":"supports"},{"source_id":"S003","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S004","target_id":"S011","kind":"supports"},{"source_id":"S069","target_id":"S011","kind":"supports"},{"source_id":"S006","target_id":"S012","kind":"supports"},{"source_id":"S004","target_id":"S012","kind":"supports"},{"source_id":"S007","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S015","kind":"supports"},{"source_id":"S010","target_id":"S015","kind":"supports"},{"source_id":"S013","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S017","kind":"supports"},{"source_id":"S014","target_id":"S017","kind":"supports"},{"source_id":"S008","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S069","target_id":"S020","kind":"supports"},{"source_id":"S008","target_id":"S021","kind":"supports"},{"source_id":"S009","target_id":"S021","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S005","target_id":"S021","kind":"supports"},{"source_id":"S011","target_id":"S023","kind":"supports"},{"source_id":"S008","target_id":"S023","kind":"supports"},{"source_id":"S021","target_id":"S023","kind":"supports"},{"source_id":"S003","target_id":"S024","kind":"supports"},{"source_id":"S068","target_id":"S024","kind":"supports"},{"source_id":"S003","target_id":"S025","kind":"supports"},{"source_id":"S002","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S027","kind":"supports"},{"source_id":"S003","target_id":"S027","kind":"supports"},{"source_id":"S004","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S025","target_id":"S030","kind":"supports"},{"source_id":"S028","target_id":"S032","kind":"supports"},{"source_id":"S025","target_id":"S032","kind":"supports"},{"source_id":"S024","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S024","target_id":"S036","kind":"supports"},{"source_id":"S027","target_id":"S036","kind":"supports"},{"source_id":"S004","target_id":"S036","kind":"supports"},{"source_id":"S014","target_id":"S043","kind":"supports"},{"source_id":"S015","target_id":"S043","kind":"supports"},{"source_id":"S016","target_id":"S043","kind":"supports"},{"source_id":"S017","target_id":"S043","kind":"supports"},{"source_id":"S011","target_id":"S043","kind":"supports"},{"source_id":"S014","target_id":"S045","kind":"supports"},{"source_id":"S011","target_id":"S045","kind":"supports"},{"source_id":"S013","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S016","target_id":"S046","kind":"supports"},{"source_id":"S014","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S017","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S015","target_id":"S048","kind":"supports"},{"source_id":"S070","target_id":"S048","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S024","target_id":"S049","kind":"supports"},{"source_id":"S027","target_id":"S049","kind":"supports"},{"source_id":"S036","target_id":"S051","kind":"supports"},{"source_id":"S033","target_id":"S052","kind":"supports"},{"source_id":"S035","target_id":"S052","kind":"supports"},{"source_id":"S027","target_id":"S052","kind":"supports"},{"source_id":"S028","target_id":"S052","kind":"supports"},{"source_id":"S051","target_id":"S053","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S035","target_id":"S053","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S004","target_id":"S040","kind":"supports"},{"source_id":"S003","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S054","kind":"supports"},{"source_id":"S041","target_id":"S054","kind":"supports"},{"source_id":"S039","target_id":"S054","kind":"supports"},{"source_id":"S041","target_id":"S056","kind":"supports"},{"source_id":"S033","target_id":"S056","kind":"supports"},{"source_id":"S037","target_id":"S056","kind":"supports"},{"source_id":"S040","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S035","target_id":"S057","kind":"supports"},{"source_id":"S024","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S041","target_id":"S058","kind":"supports"},{"source_id":"S027","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S056","target_id":"S059","kind":"supports"},{"source_id":"S035","target_id":"S059","kind":"supports"},{"source_id":"S049","target_id":"S059","kind":"supports"},{"source_id":"S040","target_id":"S059","kind":"supports"},{"source_id":"S057","target_id":"S060","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S041","target_id":"S061","kind":"supports"},{"source_id":"S038","target_id":"S061","kind":"supports"},{"source_id":"S060","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S039","target_id":"S062","kind":"supports"},{"source_id":"S060","target_id":"S063","kind":"supports"},{"source_id":"S061","target_id":"S063","kind":"supports"},{"source_id":"S039","target_id":"S063","kind":"supports"},{"source_id":"S056","target_id":"S055","kind":"supports"},{"source_id":"S057","target_id":"S055","kind":"supports"},{"source_id":"S058","target_id":"S055","kind":"supports"},{"source_id":"S059","target_id":"S055","kind":"supports"},{"source_id":"S060","target_id":"S055","kind":"supports"},{"source_id":"S061","target_id":"S055","kind":"supports"},{"source_id":"S062","target_id":"S055","kind":"supports"},{"source_id":"S063","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S054","kind":"supports"},{"source_id":"S014","target_id":"S042","kind":"supports"},{"source_id":"S016","target_id":"S042","kind":"supports"},{"source_id":"S021","target_id":"S042","kind":"supports"},{"source_id":"S023","target_id":"S065","kind":"supports"},{"source_id":"S021","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S067","target_id":"S066","kind":"supports"},{"source_id":"S021","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S042","kind":"supports"},{"source_id":"S001","target_id":"S071","kind":"supports"},{"source_id":"S021","target_id":"S071","kind":"supports"},{"source_id":"S014","target_id":"S071","kind":"supports"},{"source_id":"S015","target_id":"S071","kind":"supports"},{"source_id":"S016","target_id":"S071","kind":"supports"},{"source_id":"S017","target_id":"S071","kind":"supports"},{"source_id":"S039","target_id":"S071","kind":"supports"},{"source_id":"S001","target_id":"S074","kind":"supports"},{"source_id":"S008","target_id":"S074","kind":"supports"},{"source_id":"S014","target_id":"S074","kind":"supports"},{"source_id":"S015","target_id":"S074","kind":"supports"},{"source_id":"S016","target_id":"S074","kind":"supports"},{"source_id":"S017","target_id":"S074","kind":"supports"},{"source_id":"S043","target_id":"S074","kind":"supports"},{"source_id":"S041","target_id":"S074","kind":"supports"},{"source_id":"S040","target_id":"S074","kind":"supports"},{"source_id":"S009","target_id":"S074","kind":"supports"},{"source_id":"S021","target_id":"S074","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S061","target_id":"S075","kind":"supports"},{"source_id":"S039","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S076","kind":"supports"},{"source_id":"S037","target_id":"S076","kind":"supports"},{"source_id":"S021","target_id":"S076","kind":"supports"},{"source_id":"S061","target_id":"S076","kind":"supports"},{"source_id":"S075","target_id":"S076","kind":"supports"},{"source_id":"S019","target_id":"S076","kind":"supports"},{"source_id":"S069","target_id":"S076","kind":"supports"},{"source_id":"S076","target_id":"S077","kind":"supports"},{"source_id":"S021","target_id":"S077","kind":"supports"},{"source_id":"S042","target_id":"S077","kind":"supports"},{"source_id":"S077","target_id":"S078","kind":"supports"},{"source_id":"S042","target_id":"S078","kind":"supports"},{"source_id":"S076","target_id":"S078","kind":"supports"},{"source_id":"S061","target_id":"S078","kind":"supports"},{"source_id":"S074","target_id":"S079","kind":"supports"},{"source_id":"S063","target_id":"S079","kind":"supports"},{"source_id":"S075","target_id":"S079","kind":"supports"},{"source_id":"S078","target_id":"S079","kind":"supports"},{"source_id":"S073","target_id":"S072","kind":"supports"},{"source_id":"S074","target_id":"S072","kind":"supports"},{"source_id":"S075","target_id":"S072","kind":"supports"},{"source_id":"S076","target_id":"S072","kind":"supports"},{"source_id":"S077","target_id":"S072","kind":"supports"},{"source_id":"S078","target_id":"S072","kind":"supports"},{"source_id":"S079","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S071","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["RCOP"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"流形约束优化问题","source_units":[{"raw_tex":"In this paper, we consider the following Riemannian Constrained Optimization Problem (RCOP):\n\\begin{equation}\\tag{RCOP} \\label{RCOP} \n\t\\begin{array}{ll}\n\t\t\\underset{x \\in \\mathcal{M}}{\\min} & f(x) \\\\\n\t\t\\text { s.t. } & h(x)=0, \\text{ and }  g(x) \\leq 0,\\\\\n\t\\end{array}\n\\end{equation}\nwhere $\\mathcal{M}$ is a connected, complete $d$-dimensional Riemannian manifold and $f \\colon \\mathcal{M}\\to \\mathbb{R}, h\\colon\\mathcal{M}\\to \\mathbb{R}^{l}$, and $g\\colon\\mathcal{M}\\to \\mathbb{R}^{m}$ are smooth functions.\n","start":3609,"end":4119}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":["eq:RiemannianMetric"],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"切空间与黎曼度量","source_units":[{"raw_tex":"Let us briefly review some concepts from Riemannian geometry, following the notation of \\cite{boumal2022intromanifolds}.\n$\\mathcal{M}$ denotes a finite-dimensional smooth manifold.\nLet $p \\in \\mathcal{M}$ and $T_p \\mathcal{M}$ be the tangent space at $p$ with $0_{p}$ being its zero element.\nWe use a canonical identification $ T_{p} \\mathcal{E}\\cong \\mathcal{E}$ for a vector space $ \\mathcal{E} $ and $p \\in \\mathcal{E}$.\nA vector field is a map $V \\colon \\mathcal{M} \\to T \\mathcal{M}$ with $V(p) \\in T_{p} \\mathcal{M}$, where $T \\mathcal{M}:=\\bigcup_{p \\in \\mathcal{M}} T_p \\mathcal{M}$ is the tangent bundle. \n$\\mathfrak{X}(\\mathcal{M})$ denotes the set of all smooth vector fields defined on $\\mathcal{M}$.\nFurthermore, $\\mathcal{M}$ is a Riemannian manifold if it is equipped with a Riemannian metric, that is, a choice of inner product $\\langle\\cdot, \\cdot\\rangle_p \\colon T_p \\mathcal{M} \\times T_p \\mathcal{M} \\to \\mathbb{R} $ for each tangent space at $p$ on $\\mathcal{M}$ such that for all $V, W \\in \\mathfrak{X}(\\mathcal{M})$, the map\n\\begin{equation}\\label{eq:RiemannianMetric}\n\tp \\mapsto\\langle V(p), W(p)\\rangle_p\n\\end{equation}\nis a smooth function from $\\mathcal{M}$ to $\\mathbb{R}$.\nRiemannian metric induces the norm $\\|\\xi\\|_p:=\\sqrt{\\langle\\xi, \\xi\\rangle_p}$ for $\\xi \\in T_p \\mathcal{M}$.\nWe often omit the subscript $p$ if it is clear from the context.\nThroughout this paper, we assume that all the manifolds involved are \\emph{connected} and \\emph{complete}.\nGiven a curve segment on $\\mathcal{M}$, $c:[a, b] \\rightarrow \\mathcal{M}$, the length of $c$ is defined as \n$\nL(c):=\\int_a^b \\| \\dot{c}(t)\\|_{c(t)} \\mathrm{d} t,\n$\nwhere $\\dot{c}(t) \\in T_{c(t)} \\mathcal{M}$ is the velocity vector of $c$ at $t$.\nSince $\\mathcal{M}$ is connected, there exists a curve segment connecting any pair of points $p,q \\in \\mathcal{M}$.\nIndeed, $\\mathcal{M}$ is a metric space under the Riemannian distance $d(p, q):=\\inf_{c} L(c) $ where the infimum is taken over all curve segments joining $p $ to $q $ \\cite[Thm. 2.55]{lee2018}.\nFor two manifolds $\\mathcal{M}_1, \\mathcal{M}_2$ and a smooth map $F \\colon \\mathcal{M}_1 \\to \\mathcal{M}_2$, the differential of $F$ at $p \\in \\mathcal{M}_1$ is a linear operator denoted as $\\mathcal{D} F(p) \\colon T_p \\mathcal{M}_1 \\to T_{F(p)} \\mathcal{M}_2$.\nLet $\\mathfrak{F}(\\mathcal{M})$ be the set of all smooth \\emph{scalar fields} (or say real-valued functions) $f\\colon \\mathcal{M} \\to \\mathbb{R}$.\nThe Riemannian gradient of $f$ at $p$, $\\operatorname{grad} f(p)$, is defined as the unique element of $T_p \\mathcal{M}$ that satisfies $\\langle \\xi, \\operatorname{grad} f(p)\\rangle_p = \\mathcal{D} f(p)[\\xi]$ for all $\\xi \\in T_p \\mathcal{M}$, where $\\mathcal{D} f(p) \\colon T_p \\mathcal{M} \\to T_{f(p)} \\mathbb{R} \\cong \\mathbb{R}$.\nNote that for any $f \\in \\mathfrak{F}(\\mathcal{M})$, the gradient vector field $x \\mapsto \\operatorname{grad} f(x)$ is a smooth vector field on $\\mathcal{M}$, i.e., $\\operatorname{grad} f \\in \\mathfrak{X}(\\mathcal{M})$.\n","start":9491,"end":12498}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"回缩与指数映射","source_units":[{"raw_tex":"A retraction $\\mathrm{R} \\colon T \\mathcal{M} \\to \\mathcal{M}$ is a smooth map such that\n$\\mathrm{R}_{p}\\left(0_{p}\\right)=p$ and $\\mathcal{D} \\mathrm{R}_{p}\\left(0_{p}\\right) =\\operatorname{id}_{T_p \\mathcal{M}}$, i.e., the identity map on $T_{p} \\mathcal{M}$, where $\\mathrm{R}_{p}$ is the restriction of $\\mathrm{R}$ to $T_{p} \\mathcal{M}$ and $\\mathcal{D} \\mathrm{R}_{p}\\left(0_{p}\\right) \\colon T_{0_p} (T_p \\mathcal{M}) \\cong T_p \\mathcal{M} \\to T_{p} \\mathcal{M}$ is the differential of $\\mathrm{R}_{p}$ at $0_{p}$.\nOne theoretically perfect type of retraction is the exponential map, denoted as $\\operatorname{Exp}$. Since $\\mathcal{M}$ is complete, the exponential map is well-defined on the whole tangent bundle.\nLet $\\operatorname{Exp}_p\\colon T_p \\mathcal{M} \\rightarrow \\mathcal{M}$ be the exponential map at $p$; then $t \\mapsto \\operatorname{Exp}_p(t \\xi)$ is the unique geodesic that passes through $p$ with velocity $\\xi \\in T_p \\mathcal{M}$ when $t=0$.\n\n","start":12498,"end":13470}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["eq:Singularity"],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"向量场零点与协变导数","source_units":[{"raw_tex":"\\subsection{Riemannian Newton Method}\\label{subsec:RiemannianNewton}\n\nThe Newton method is a powerful tool for finding the zeros of nonlinear functions in the Euclidean setting.\nThe generalized Newton method has been studied in the Riemannian setting; it aims to find a \\emph{singularity} for the vector field $F \\in \\mathfrak{X}(\\mathcal{M})$, i.e., a point $p \\in \\mathcal{M}$ such that,\n\\begin{equation}\\label{eq:Singularity}\n\tF(p)=0_{p} \\in T_p \\mathcal{M}.\n\\end{equation}\nLet $\\nabla$ be the Levi-Civita connection on $\\mathcal{M}$, i.e., the unique symmetric connection compatible with the Riemannian metric.\nThe covariant derivative $\\nabla F$ assigns each point $p \\in \\mathcal{M}$ a linear operator $\\nabla F(p)\\colon T_{p}\\mathcal{M}\\to T_{p}\\mathcal{M}$. \nIn particular, the Riemannian Hessian of $f \\in \\mathfrak{F}(\\mathcal{M})$ at $p$ is a self-adjoint operator on $T_{p} \\mathcal{M}$, defined as $\\operatorname{Hess} f(p):=\\nabla (\\operatorname{grad}f)(p).$\n","start":13470,"end":14443}]},{"id":"S005","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"对角矩阵与单位向量","source_units":[{"raw_tex":"In this section, we will give a comprehensive interpretation of the Riemannian interior point method.\nFollowing common usage in the interior-point literature, big letters denote the associated diagonal matrix, e.g., $Z =\\operatorname{diag}(z_{1}, \\ldots, z_{m})$ with $z \\in \\mathbb{R}^{m}$. $e$ denotes the all-ones vector, and $0$ stands for zero vector/matrix with proper dimensions.\n\n\\subsection{KKT Vector Field and Its Covariant Derivative}\n\n","start":15475,"end":15923}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"拉格朗日函数及梯度","source_units":[{"raw_tex":"The Lagrangian function of (\\ref{RCOP}) is\n\\begin{equation*}\n\t\\mathcal{L}(x, y, z):=f(x)+\\sum_{i=1}^l y_i h_i(x)+\\sum_{i=1}^m z_i g_i(x),\n\\end{equation*}\nwhere $y \\in \\mathbb{R}^l$ and $z \\in \\mathbb{R}^m$ are Lagrange multipliers corresponding to the equality and\ninequality constraints, respectively.\nWith respect to the variable $x$, $\\mathcal{L}(\\cdot, y, z)$ is a real-valued function defined on $\\mathcal{M}$, and its Riemannian gradient is\n\\begin{equation*}\n\\operatorname{grad}_{x} \\mathcal{L}(x, y, z)=\\operatorname{grad} f(x) + \\sum_{i =1 }^{l} y_{i} \\operatorname{grad} h_{i}(x) + \\sum_{i =1 }^{m} z_{i} \\operatorname{grad} g_{i}(x),\n\\end{equation*}\nwhere $\\{\\operatorname{grad} h_{i}(x)\\}_{i=1}^{l}$ and $ \\{\\operatorname{grad} g_{i}(x)\\}_{i=1}^{m}$ are the gradients of the component functions of $h$ and $g$. \n","start":15923,"end":16746}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":["eq:KKTConditions"],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"黎曼 KKT 条件","source_units":[{"raw_tex":"The Riemannian KKT conditions (e.g., see \\cite[Definition 2.3]{liu2020simple} or \\cite{yang2014optimality}) for (\\ref{RCOP}) are given by\n\\begin{equation}\\label{eq:KKTConditions}\n\t\\operatorname{grad}_{x} \\mathcal{L}(x, y, z) =0_{x}; \\;\n\th(x) =0 ,\n\tg(x) \\leq 0 , \n\tz  \\geq 0; \\;\n\tZ g(x) =0.\n\\end{equation}\n","start":16746,"end":17051}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":["eq:KKTVectorField"],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"乘积流形上的 KKT 场","source_units":[{"raw_tex":"The above conditions can be written in terms of slack variables $\\mathbb{R}^{m} \\ni s:=-g(x)$, as\n\\begin{equation}\\label{eq:KKTVectorField}\n\tF(w):=\\left(\\begin{array}{l}\n\t\t\\operatorname{grad}_{x} \\mathcal{L}(x, y, z) \\\\\n\t\t h(x) \\\\\n\t\tg(x)+s \\\\\n\t\tZ S e\n\t\\end{array}\\right)=0_{w}\n\t=\\left(\\begin{array}{l}\n\t\t0_{x} \\\\\n\t\t0\\\\\n\t\t0\\\\\n\t\t0\n\t\\end{array}\\right) \\in T_w \\mathcal{N},\n\\end{equation}\nand $(z,s) \\geq 0,$\nwhere $w:=(x, y, z,s) \\in \\mathcal{N}:=\\mathcal{M}\\times \\mathbb{R}^{l} \\times \\mathbb{R}^{m} \\times \\mathbb{R}^{m}$.\nHere, we have generated a vector field $F$ in (\\ref{eq:KKTVectorField}) on the product manifold $\\mathcal{N}$, i.e.,\nfor any $w\\in \\mathcal{N}$, \n$\nF(w) \\in\nT_{w} \\mathcal{N} \\cong T_{x} \\mathcal{M} \\times \\mathbb{R}^{l} \\times \\mathbb{R}^{m} \\times \\mathbb{R}^{m}.\n$\nWe will call $F \\colon \\mathcal{N} \\to T\\mathcal{N} $ in (\\ref{eq:KKTVectorField}) above the \\emph{KKT vector field} for (\\ref{RCOP}).\n","start":17051,"end":17977}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":["eq:barR"],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"乘积回缩","source_units":[{"raw_tex":"Note that for $\\xi=\\left(\\xi_x, \\xi_y, \\xi_z, \\xi_s\\right) \\in T_{w} \\mathcal{N} $,\n\\begin{equation}\\label{eq:barR}\n\t\\bar{\\mathrm{R}}_{w}(\\xi):=(\\mathrm{R}_{x}(\\xi_{x}), y+\\xi_{y}, z+\\xi_{z}, s+\\xi_{s})\n\\end{equation}\nis a well-defined retraction on the product manifold $\\mathcal{N}$ as long as $\\mathrm{R}$ is a retraction on $\\mathcal{M}$.\n","start":17977,"end":18320}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":["defn:HxGx"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"约束梯度算子","source_units":[{"raw_tex":"\nNow, we aim to find a singularity of the KKT vector field $F$ on $\\mathcal{N}$ under some nonnegative constraints $(z,s) \\geq 0$.\nIf the Riemannian Newton method is to be applied to (\\ref{eq:KKTVectorField}), we must formulate the covariant derivative of $F$ at an arbitrary $w \\in \\mathcal{N}$. Before that, we need some new symbols.\nFixing a point $x \\in \\mathcal{M}$, we can define two linear operators $\\mathcal{H}_{x}\\colon \\mathbb{R}^{l} \\rightarrow T_{x} \\mathcal{M}$ and $\\mathcal{G}_{x}\\colon \\mathbb{R}^{m} \\rightarrow T_{x} \\mathcal{M}$ by\n\\begin{equation}\\label{defn:HxGx}\n\t\\mathcal{H}_x [v]:=\\sum_{i=1}^l v_i \\operatorname{grad} h_i(x), \\quad \\mathcal{G}_x [u]:=\\sum_{i=1}^m u_i \\operatorname{grad} g_i(x),\n\\end{equation}\nrespectively.\nThen, for $\\mathcal{H}_{x}$, its adjoint operator $\\mathcal{H}_x^*\\colon T_x \\mathcal{M} \\rightarrow \\mathbb{R}^l$ is given by\n\\begin{equation*}\n\t\\mathcal{H}_x^* [\\xi]=\\left(\\left\\langle\\operatorname{grad} h_1(x), \\xi\\right\\rangle_x, \\cdots,\\left\\langle\\operatorname{grad} h_l(x), \\xi\\right\\rangle_x\\right)^T.\n\\end{equation*}\nAlso, $\\mathcal{G}_x^*\\colon T_x \\mathcal{M} \\rightarrow \\mathbb{R}^m$ with $\\mathcal{G}_x^* [\\xi]=\\left(\\left\\langle\\operatorname{grad} g_1(x), \\xi\\right\\rangle_x, \\cdots,\\left\\langle\\operatorname{grad} g_m(x), \\xi\\right\\rangle_x\\right)^T$. \n","start":18320,"end":19639}]},{"id":"S011","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:CovariantDerivativeKKTVectorField"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"KKT 场的协变导数","source_units":[{"raw_tex":"\nNow, using the solutions of Exercises 5.4 and 5.13 on covariant derivatives of vector fields on product manifolds in monograph \\cite{boumal2022intromanifolds}, the following results can be easily derived.\nGiven KKT vector field $F $ in (\\ref{eq:KKTVectorField}) and $w \\in \\mathcal{N}$, the covariant derivative of $F$ at $w$ is the operator $\\nabla F(w)\\colon T_{w} \\mathcal{N} \\to T_{w} \\mathcal{N}$ given by\n\\begin{equation}\\label{eq:CovariantDerivativeKKTVectorField}\n\t\\nabla F(w) [\\Delta w] =\n\t\\left(\\begin{array}{l}\n\t\t\\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t\t+\\mathcal{H}_{x} [\\Delta y]\n\t\t+\\mathcal{G}_{x} [\\Delta z]\\\\\n\t\t\\mathcal{H}_{x}^{*}[\\Delta x]\\\\\n\t\t\\mathcal{G}_{x}^{*}[\\Delta x]+\\Delta s\\\\\n\t\tZ \\Delta s + S \\Delta z\n\t\\end{array}\\right),\n\\end{equation}\nwhere $\\Delta w=(\\Delta x, \\Delta y, \\Delta z, \\Delta s) \\in T_{w} \\mathcal{N}$ and $\\operatorname{Hess}_{x} \\mathcal{L}(w)$ is the Riemannian Hessian of real-valued function $\\mathcal{L}(\\cdot, y, z)$.\n","start":19639,"end":20618}]},{"id":"S012","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:HessianLagrange"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"拉格朗日 Hessian 展开","source_units":[{"raw_tex":"Indeed,\n$\\operatorname{Hess}_{x} \\mathcal{L}(w)\\colon T_{x}\\mathcal{M} \\to T_{x}\\mathcal{M}$ satisfies\n\\begin{equation}\\label{eq:HessianLagrange}\n\t\\operatorname{Hess}_{x} \\mathcal{L}(w) =\n\t\\operatorname{Hess} f(x)\n\t+\\sum_{i=1}^{l} y_{i} \\operatorname{Hess} h_{i}(x)\n\t+\\sum_{i=1}^{m} z_{i} \\operatorname{Hess} g_{i}(x),\n\\end{equation}\nwhere $\\{\\operatorname{Hess} h_{i}(x)\\}_{i=1}^{l}$ and $\\{\\operatorname{Hess} g_{i}(x)\\}_{i=1}^{m}$ are Hessians of the component functions of $h$ and $g$.\n","start":20618,"end":21108}]},{"id":"S013","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"活动集与假设约定","source_units":[{"raw_tex":"Next, we will discuss an important result about the covariant derivative of the KKT vector field given in (\\ref{eq:CovariantDerivativeKKTVectorField}).\nLet $\\mathbb{A}(x):=\\left\\{i \\mid g_{i}(x)=0\\right\\}$ denote the active set at $x \\in \\mathcal{M}$.\nA prior study on Riemannian optimality conditions \\cite{yang2014optimality,bergmann2019intrinsic} showed that the following assumptions are meaningful.\nWe call them the \\emph{standard Riemannian assumptions} for (\\ref{RCOP}).\nNote that the $x^{*}$ and $w^*$ in \\ref{B3}-\\ref{B5} all refer to those in \\ref{B1}.\n\n\n\\setlist[assumptions,1]{label=\\textbf{(A\\arabic*)}}\n\\begin{assumptions}[leftmargin=9mm]\n","start":21159,"end":21812}]},{"id":"S014","semantic_type":"assumption","kind":"assumption","labels":["B1","B3","B4","B5"],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"KKT 解存在","source_units":[{"raw_tex":"\t\\item {Existence.} There exists $w^*=(x^{*}, y^{*}, z^{*}, s^{*})$ satisfying the Riemannian KKT conditions (\\ref{eq:KKTConditions}). Here, we introduce the slack variables $s^{*}:=-g(x^{*})$.\n\t\\label{B1}\t\n%\t\\item {Smoothness of $ C^{2} $.} The functions $f, g,$ and $h$ are $ C^{2} $ on $\\mathcal{M}$.\n%\t\\label{B2}\t\n","start":21812,"end":22130}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","labels":["B1","B3","B4","B5"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"线性无关约束资格","source_units":[{"raw_tex":"\t\\item {Linear Independence Constraint Qualification (LICQ).} \\\\ The set $\\left\\{\\operatorname{grad} h_{i}(x^{*})\\right\\}_{i=1}^{l} \\bigcup \\left\\{\\operatorname{grad} g_{i}(x^{*})\\right\\}_{i \\in \\mathbb{A}(x^{*})}$ is linearly independent in $ T_{x^{*}}\\mathcal{M}$.\n\t\\label{B3}\t\n","start":22130,"end":22410}]},{"id":"S016","semantic_type":"assumption","kind":"assumption","labels":["B1","B3","B4","B5"],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"严格互补","source_units":[{"raw_tex":"\t\\item {Strict complementarity.} $(z^{*})_{i}>0$ if $g_{i}(x^{*})=0$ for all $i=1, \\cdots,m$.\n\t\\label{B4}\t\n","start":22410,"end":22517}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"二阶充分条件","source_units":[{"raw_tex":"\t\\item {Second-order sufficiency.}\n\t$\\left \\langle\\operatorname{Hess}_{x} \\mathcal{L}(w^{*})[\\xi], \\xi\\right\\rangle>0$ for all nonzero $\\xi \\in T_{x^{*}}\\mathcal{M}$ \\\\ satisfying $\\left\\langle\\xi, \\operatorname{grad} h_{i}(x^{*})\\right\\rangle=0 $ for $i=1,\\dots,l,$ and\n\t$\\left\\langle\\xi, \\operatorname{grad} g_{i}(x^{*})\\right\\rangle=0$ for $ i \\in \\mathbb{A}(x^{*})$. \n\t\\label{B5}\n\\end{assumptions}\n","start":22517,"end":22919}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":["eq:perturbedKKTvectorfield"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"扰动 KKT 场","source_units":[{"raw_tex":"\\begin{equation}\\label{eq:perturbedKKTvectorfield}\n\tF_{\\mu}(w):=F(w)-\\mu \\hat{e}, \\text { and } \\hat{e}:=\\hat{e}(w):=\\left(0_x, 0,0, e\\right).\n\\end{equation}\nNotice that the perturbation term $\\hat{e}$, indeed, is a special vector field on $\\mathcal{N}$, not a constant, because $0_x$ is essentially dependent on $w$ and/or $x$.\n","start":28183,"end":28512}]},{"id":"S020","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:NabulaperturbedKKTvectorfield"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"扰动不改变协变导数","source_units":[{"raw_tex":"In fact, the covariant derivative of the perturbed KKT vector field is the same as that of the original. From the linearity of the connection $\\nabla$, for any $w \\in \\mathcal{N}$ and any $\\mu>0$, we have \n\\begin{equation}\\label{eq:NabulaperturbedKKTvectorfield}\n\t\\nabla F_{\\mu}(w)=\\nabla F(w)-\\mu \\nabla \\hat{e}(w)=\\nabla F(w),\n\\end{equation}\nwhere the last equity comes from $\\nabla \\hat{e}(w) [\\Delta w]=\\left(0_x, 0,0,0\\right)$ for all $\\Delta w \\in T_w \\mathcal{N}$.\nApplying the Riemannian Newton method to perturbed KKT vector field $F_{\\mu}(w)$ yields the \\emph{perturbed Newton equation}: $\\nabla F_{\\mu}(w) [\\Delta w]+F_{\\mu}(w)=0$. \nFrom (\\ref{eq:perturbedKKTvectorfield}) and (\\ref{eq:NabulaperturbedKKTvectorfield}), this equation is equivalent to \n\\begin{equation}\n\t\\nabla F(w) [\\Delta w]+F(w)=\\mu \\hat{e},\n\\end{equation}\nwhich reduces to the ordinary Newton equation (\\ref{eq:OriginalNewtonEquation}) as $\\mu \\to 0$.\nAt this point, we can describe a prototype of the Riemannian Interior Point Method (RIPM) in Algorithm \\ref{algo:prototype_algo}.\n","start":28512,"end":29574}]},{"id":"S021","semantic_type":"algorithm","kind":"algorithm","labels":["algo:prototype_algo","eq:prototype","rule:StepSizeA"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"RIPM 原型算法","source_units":[{"raw_tex":"\n\\begin{algorithm}%[H]\n\t\\caption{Prototype Algorithm of RIPM for (\\ref{RCOP})}\n\t\\label{algo:prototype_algo}\n\t\\SetAlgoLined\n\t\n\t\\KwIn{An initial point $w_0=\\left(x_0, y_0, z_0, s_0\\right) \\in \\mathcal{N}$ with $\\left(z_0, s_0\\right)>0$ and a retraction $\\mathrm{R}$ on $\\mathcal{M}$. $\\hat{\\gamma} \\in(0,1)$, $\\mu_0>0$.}\n\t\\KwOut{Sequence $\\left\\{w_k\\right\\} \\subset \\mathcal{N}$ such that $\\left\\{w_k\\right\\} \\rightarrow w^*$ and $w^*$ satisfies the KKT conditions (\\ref{eq:KKTConditions}).} % \n\t\n\tSet $k \\to 0$\\;\n\t\n\t\\While{Stopping criterion not satisfied}{\n\t\t1. Obtain $\\Delta w_{k}=(\\Delta x_{k}, \\Delta y_{k}, \\Delta z_{k},\\Delta s_{k}) \\in T_{w_k} \\mathcal{N}$ by solving the perturbed Newton equation on $T_{w_k} \\mathcal{N}$:\n\t\t\\begin{equation}\\label{eq:prototype}\n\t\t\t\\nabla F(w_{k}) [\\Delta w_{k}]=-F(w_{k})+\\mu_{k} \\hat{e};\n\t\t\\end{equation}\n\t\n\t\t2. Choose $ \\gamma_{k} \\in [\\hat{\\gamma},1]$ and compute the step size $\\alpha_{k}$ defined by\n\t\t\\begin{equation}\\label{rule:StepSizeA}\n\t\t\t\\min \\left\\{1, \\gamma_{k} \\min _{i}\\left\\{-\\frac{(s_{k})_{i}}{(\\Delta s_{k})_{i}} \\mid(\\Delta s_{k})_{i}<0\\right\\},\\gamma_{k} \\min _{i}\\left\\{-\\frac{(z_{k})_{i}}{(\\Delta z_{k})_{i}} \\mid(\\Delta z_{k})_{i}<0\\right\\}\\right\\};\n\t\t\\end{equation}\n\t\n\t\t3. Compute the next point as\n\t\t$w_{k+1}:=\\bar{\\mathrm{R}}_{w_k}\\left(\\alpha_k \\Delta w_k\\right)$, see (\\ref{eq:barR})\\;\n\t\t\n\t\t4. Choose $0<\\mu_{k+1}<\\mu_k$\\;\n\t\t\n\t\t5. $k \\to k+1$\\;\n\t}\n\\end{algorithm}\n","start":29574,"end":31008}]},{"id":"S023","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:FullNewtonEquation"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"完整扰动 Newton 系统","source_units":[{"raw_tex":"Let us consider Algorithm \\ref{algo:prototype_algo} and omit the iteration count $k$.\nGiven the current point $w \\in \\mathcal{N}$ with $(z, s) > 0$, for the KKT vector field $F(w)$ in (\\ref{eq:KKTVectorField}), we denote its components by $F_x,F_y,F_z,F_s$ in top-to-bottom order, namely,\n$F_x:=\\operatorname{grad}_x \\mathcal{L}(w),F_y:=h(x),F_z:=g(x)+s,F_s:=Z S e.$\nBy using these symbols and the formulation of $\\nabla F(w)$ in (\\ref{eq:CovariantDerivativeKKTVectorField}), the \\emph{full} (perturbed) Newton equation (\\ref{eq:prototype}) defined on $T_w \\mathcal{N} \\cong T_x \\mathcal{M} \\times \\mathbb{R}^l \\times \\mathbb{R}^m \\times \\mathbb{R}^m$ is expanded as:\n\\begin{equation}\\label{eq:FullNewtonEquation}\n\t\\left(\\begin{array}{l}\n\t\t\\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t\t+\\mathcal{H}_{x} [\\Delta y]\n\t\t+\\mathcal{G}_{x} [\\Delta z]\\\\\n\t\t\\mathcal{H}_{x}^{*}[\\Delta x]\\\\\n\t\t\\mathcal{G}_{x}^{*}[\\Delta x]+\\Delta s\\\\\n\t\tZ \\Delta s + S \\Delta z\n\t\\end{array}\\right)\n\t=\n\t\\left(\\begin{array}{l}\n\t\t-F_x\\\\\n\t\t-F_y\\\\\n\t\t-F_z\\\\\n\t\t-F_s+\\mu e\n\t\\end{array}\\right).\n\\end{equation}\n","start":32122,"end":33200}]},{"id":"S024","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"局部逆回缩的存在","source_units":[{"raw_tex":"\\begin{remark}\n\tFor a retraction $\\mathrm{R}$ on $\\mathcal{M}$ and $x \\in \\mathcal{M}$,\n\tby $\\mathcal{D} \\mathrm{R}_{x}\\left(0_{x}\\right)=\\mathrm{id}_{T_{x} \\mathcal{M}}$ and the inverse function theorem, there exists a neighborhood $V$ of $0_{x}$ in $T_{x} \\mathcal{M}$ such that $ \\mathrm{R}_{x} $ is a diffeomorphism on $V$; thus,\n\t$\\mathrm{R}_{x}^{-1}(y)$ is well defined for all $ y \\in \\mathcal{M} $ sufficiently close to $ x $.\n\tIn this case, $\\mathrm{R}_{x}(V) \\subset \\mathcal{M}$ is called a retractive neighborhood of $x$.\n\tFurthermore, the existence of a \\emph{totally} retractive neighborhood \\cite[Theorem 2]{zhu2020riemannian} shows that for any $\\bar{x} \\in \\mathcal{M} $ there is a neighborhood $W$ of $\\bar{x}$ such that\n\t$\\mathrm{R}_{x}^{-1}(y)$ is well defined for all $x, y \\in W$.\n\tIn what follows, we will suppose that an appropriate neighborhood has been chosen by default for the well-definedness of $\\mathrm{R}_x^{-1}(y)$.\n\\end{remark}\n","start":41737,"end":42699}]},{"id":"S025","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"向量输运与等距性","source_units":[{"raw_tex":"Define the Whitney sum\n$\nT \\mathcal{M} \\oplus T \\mathcal{M}:=\\left\\{(\\eta, \\xi): \\eta, \\xi \\in T_x \\mathcal{M}, x \\in \\mathcal{M} \\right\\}.\n$\nA smooth map\n$\n\\mathrm{T} \\colon T \\mathcal{M} \\oplus T \\mathcal{M} \\to T \\mathcal{M}: (\\eta, \\xi) \\mapsto \\mathrm{T}_{\\eta}(\\xi),\n$\nis called a \\emph{vector transport} on $\\mathcal{M}$ if there exists an associated retraction $\\mathrm{R}$ on $\\mathcal{M}$ such that $\\mathrm{T}$ satisfies the following properties for all $x \\in \\mathcal{M}$:\n\\begin{enumerate}\n\t\\item Associated retraction. $\\mathrm{T}_{\\eta}\\left(\\xi\\right) \\in T_{\\mathrm{R}_{x}\\left(\\eta\\right)} \\mathcal{M}$\n\tfor all $\\eta, \\xi \\in T_x \\mathcal{M}$.\t\n\t\\item Consistency. $\\mathrm{T}_{0_{x}}\\left(\\xi\\right)=\\xi$ for all $\\xi \\in T_{x} \\mathcal{M}$.\t\n\t\\item Linearity.\n\t$\\mathrm{T}_{\\eta}\\left(a \\xi+b \\zeta\\right)=a \\mathrm{T}_{\\eta}\\left(\\xi\\right)+b \\mathrm{T}_{\\eta}\\left(\\zeta\\right)$ for all $a, b \\in \\mathbb{R}$ and $\\eta, \\xi, \\zeta \\in T_{x} \\mathcal{M}$.\n\\end{enumerate}\nThus, fixing any $\\eta \\in T_x \\mathcal{M}$, the map\n$\n\\mathrm{T}_\\eta \\colon T_x \\mathcal{M} \\to T_{\\mathrm{R}_x(\\eta)} \\mathcal{M}: \\xi \\mapsto \\mathrm{T}_\\eta(\\xi),\n$\nis a linear operator.\nAdditionally, $\\mathrm{T}$ is \\emph{isometric} if \n$\n\\langle \\mathrm{T}_{\\eta} (\\xi), \\mathrm{T}_{\\eta} (\\zeta)\\rangle =\\langle\\xi, \\zeta\\rangle\n$\nholds, for all $x \\in \\mathcal{M}$ and all $\\xi, \\zeta, \\eta \\in T_{x} \\mathcal{M}$.\nIn other words, for any $\\eta \\in T_x \\mathcal{M}$, the adjoint and the inverse of $\\mathrm{T}_{\\eta}$ coincide, i.e., $ \\mathrm{T}_{\\eta}^{*} =  \\mathrm{T}_{\\eta}^{-1}$.\n","start":42754,"end":44344}]},{"id":"S026","semantic_type":"definition","kind":"definition","labels":["eq:DifferentiatedRetraction"],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"微分回缩输运","source_units":[{"raw_tex":"There are two important classes of vector transport as follows. Let $\\mathrm{R}$ be a retraction on $\\mathcal{M}$. \n\\begin{enumerate}[(1)]\n\t\\item \n\tThe \\emph{differentiated retraction} defined by\n\t\\begin{equation}\\label{eq:DifferentiatedRetraction}\n\t\t\\mathrm{T}_{\\eta}(\\xi):=\\mathcal{D} \\mathrm{R}_{x}(\\eta)[\\xi],  \\; \\eta, \\xi \\in T_{x} \\mathcal{M}, x \\in \\mathcal{M}, \n\t\\end{equation}\n\tis a valid vector transport \\cite[equation (8.6)]{absil2009optimization}.\t\n","start":44344,"end":44807}]},{"id":"S027","semantic_type":"definition","kind":"definition","labels":["eq:ParallelTransport"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"沿回缩曲线的平行输运","source_units":[{"raw_tex":"\t\\item\n\tGiven a smooth curve $\\gamma \\colon [0,1] \\to \\mathcal{M}$ and $t_0, t_1 \\in [0,1]$, the \\emph{parallel transport} from the tangent space at $\\gamma(t_0)$ to the tangent space at $\\gamma(t_1)$ along $\\gamma,$\n\tis a linear operator\n\t$ \\mathrm{P}_\\gamma^{t_1 \\to t_0} \\colon T_{\\gamma\\left(t_0\\right)} \\mathcal{M} \\rightarrow T_{\\gamma\\left(t_1\\right)} \\mathcal{M}$ defined by $\\mathrm{P}_\\gamma^{t_1 \\to t_0}(\\xi)=Z\\left(t_1\\right)$, where $Z$ is the unique parallel vector field such that $Z\\left(t_0\\right)=\\xi$.\n\tThen, for any $x \\in \\mathcal{M}, \\eta \\in T_x \\mathcal{M}$, then\n\t\\begin{equation}\\label{eq:ParallelTransport}\n\t\t\\mathrm{T}_{\\eta}\\left(\\xi\\right):=\\mathrm{P}_\\gamma^{1 \\to 0} (\\xi)\n\t\\end{equation}\n\tis a valid vector transport \\cite[equation (8.2)]{absil2009optimization}, where $\\mathrm{P}_\\gamma$ denotes the parallel transport along the curve $t \\mapsto \\gamma(t):=\\mathrm{R}_x\\left(t \\eta\\right)$. We often omit the superscript $^{1 \\to 0}$ if it is clear from the context.\n\tIn particular, parallel transport is isometric.\n\\end{enumerate}\n","start":44807,"end":45874}]},{"id":"S028","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"算子范数与输运可逆性","source_units":[{"raw_tex":"Going one degree higher, let us now discuss the Lipschitz continuity of Hessian operators.\nThroughout this paper, for a linear operator $\\mathcal{A}: \\mathcal{E} \\rightarrow \\mathcal{E}^{\\prime}$ between two finite-dimensional normed vector spaces $\\mathcal{E}$ and $\\mathcal{E}^{\\prime}$, the (operator) norm of $\\mathcal{A}$ is defined by $\\|\\mathcal{A}\\|:=\\sup \\left\\{\\|\\mathcal{A} v\\|_{\\mathcal{E}^{\\prime}} : v \\in \\mathcal{E},\\|v\\|_{\\mathcal{E}}=1\\right.$, or, $\\left.\\|v\\|_{\\mathcal{E}} \\leq 1\\right\\}.$\nThe inverse of $\\mathrm{T}_{\\eta}$ is needed in the following definitions, so we can assume that vector transport $\\mathrm{T}$ is isometric, e.g., parallel transport in (\\ref{eq:ParallelTransport}). In fact, there are many ways to construct isometric vector transports; see \\cite[Section 2.3]{huang2015riemannian}.\n","start":48102,"end":48928}]},{"id":"S029","semantic_type":"definition","kind":"definition","labels":["defn:TwiceLipschitzContinuouslyDifferentiable"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"Hessian 的输运 Lipschitz 性","source_units":[{"raw_tex":"\\begin{definition}[{\\cite[Assumption 3]{huang2015riemannian}}]\n\t\\label{defn:TwiceLipschitzContinuouslyDifferentiable}\n\tA function $f\\colon \\mathcal{M} \\rightarrow \\mathbb{R}$ is \\emph{twice Lipschitz continuously differentiable with respect to $\\mathrm{T}$ in $\\mathcal{U} \\subset \\mathcal{M}$} if it is twice differentiable and there exists a constant $\\kappa>0$ such that, for all $x, y \\in \\mathcal{U}$, \n\t$\n\t\\|\\operatorname{Hess} f(y)-\\mathrm{T}_{\\eta} \\operatorname{Hess} f(x) \\mathrm{T}_{\\eta}^{-1}\\| \\leq \\kappa d(x, y),\n\t$\n\twhere $\\eta=\\mathrm{R}_{x}^{-1} y$.\n\\end{definition}\n","start":48928,"end":49513}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","labels":["lem:TwiceLipschitzContinuouslyDifferentiable"],"member_ids":["S030"],"declared_external":true,"declaration_text":["原先决条件清单：Huang, Absil, and Gallivan (2015), Lemma 4；原文 B0052 重述。"],"source_only":false,"title":"三阶光滑函数的局部正则性","source_units":[{"raw_tex":"\\begin{lemma}[{\\cite[Lemma 4]{huang2015riemannian}}]\n\t\\label{lem:TwiceLipschitzContinuouslyDifferentiable}\n\tIf $f\\colon \\mathcal{M} \\rightarrow \\mathbb{R}$ is $ C^3 $, then for any $\\bar{x} \\in \\mathcal{M}$ and any isometric vector transport $\\mathrm{T}$, there exists a neighborhood $\\mathcal{U}$ of $\\bar{x}$ such that $f$ is twice Lipschitz continuously differentiable with respect to $\\mathrm{T}$ in $\\mathcal{U}$.\n\\end{lemma}\n","start":49513,"end":49944}]},{"id":"S032","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"协变导数的输运 Lipschitz 性","source_units":[{"raw_tex":"\\begin{definition}\n\tGiven a vector field $F$ on $\\mathcal{M}$.\n\tThe map $x \\mapsto \\nabla F(x)$ is \\emph{Lipschitz continuous with respect to $\\mathrm{T}$ in $\\mathcal{U} \\subset \\mathcal{M}$} if there exists a constant $\\kappa>0$ such that, for all $x, y \\in \\mathcal{U}$, it holds that\n\t$\n\t\\| \\nabla F(y)\n\t-\n\t\\mathrm{T}_{\\eta} \\nabla F(x) \\mathrm{T}_{\\eta}^{-1}\n\t\\| \\leq \\kappa d(x, y),\n\t$\n\twhere $\\eta=\\mathrm{R}_{x}^{-1} y$.\n\\end{definition}\n","start":50226,"end":50672}]},{"id":"S033","semantic_type":"lemma","kind":"lemma","labels":["lem:LipCountNabulaF"],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"二阶向量场的局部正则性","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:LipCountNabulaF}\n\tIf $F$ is a $C^{2}$ vector field, then\n\tfor any $\\bar{x} \\in \\mathcal{M}$ and any isometric vector transport $\\mathrm{T}$, there exists a neighborhood $\\mathcal{U}$ of $\\bar{x}$ such that the map $x \\mapsto \\nabla F(x)$ is Lipschitz continuous with respect to $\\mathrm{T}$ in $\\mathcal{U}$.\n\\end{lemma}","start":50672,"end":51016}]},{"id":"S035","semantic_type":"lemma","kind":"lemma","labels":["lem:distR1"],"member_ids":["S035"],"declared_external":true,"declaration_text":["原先决条件清单：Huang, Absil, and Gallivan (2015), Lemma 2；原文 B0056 重述。"],"source_only":false,"title":"回缩与距离的局部比较","source_units":[{"raw_tex":"\\begin{lemma}[{\\cite[Lemma 2]{huang2015riemannian}}]\\label{lem:distR1}\n\tLet $\\mathcal{M}$ be a Riemannian manifold with a retraction $\\mathrm{R}$ and let $\\bar{x} \\in \\mathcal{M}$. \n\tThen,\n\t\\begin{enumerate}[(i)]\n\t\t \\item there exist $a_{0}, a_{1}, \\delta_{a_{0}, a_{1}}>0$ such that for all $x$ in a sufficiently small neighborhood of $\\bar{x}$ and all $\\xi, \\eta \\in T_{x} \\mathcal{M}$ with $\\|\\xi\\|, \\|\\eta\\|  \\leq \\delta_{a_{0}, a_{1}}$,\n\t\t one has\n\t\t $\n\t\t a_{0}\\|\\xi-\\eta\\| \\leq d(\\mathrm{R}_{x}(\\eta), \\mathrm{R}_{x}(\\xi)) \\leq a_{1}\\|\\xi-\\eta\\|.\n\t\t $\t\n\t\t In particular,\n\t\t $\n\t\t a_{0}\\|\\xi\\| \\leq d(x, \\mathrm{R}_{x}(\\xi)) \\leq a_{1}\\|\\xi\\|\n\t\t $\n\t\t when $\\eta=0$;\n\t\t \n\t\t \\item\n\t\t there exist $a_{0},a_{1}>0$ such that for all $x$ in a sufficiently small neighborhood of $\\bar{x}$, one has\n\t\t $\n\t\t a_{0}\\|\\xi\\| \\leq d(x,\\bar{x}) \\leq a_{1}\\|\\xi\\| \\text { where } \\xi=\\mathrm{R}_{\\bar{x}}^{-1}(x). \n\t\t $\n\t\\end{enumerate}\n\\end{lemma}\n","start":51375,"end":52312}]},{"id":"S036","semantic_type":"lemma","kind":"lemma","labels":["lem:FundamentalTheoremOfCalculusRiemannian"],"member_ids":["S036"],"declared_external":true,"declaration_text":["原先决条件清单：Huang, Absil, and Gallivan (2015), Lemma 8；原文 B0057 重述，指数映射情形附引 Ferreira and Svaiter (2002), equation (2.4)。"],"source_only":false,"title":"黎曼微积分基本估计","source_units":[{"raw_tex":"\nThe next lemma is the fundamental theorem of calculus in the Riemannian case.\n\\begin{lemma}[{\\cite[Lemma 8]{huang2015riemannian}}]\\label{lem:FundamentalTheoremOfCalculusRiemannian}\n\tLet $F$ be a $C^{1}$ vector field and $\\bar{x} \\in \\mathcal{M}$. \n\tThen there exist a neighborhood $\\mathcal{U}$ of $\\bar{x}$ and a constant $c_{1} \\geq 0$ such that for all $x, y \\in \\mathcal{U}$,\n\t\\begin{equation*}\n\t\\left\\| \\mathrm{P}_{\\gamma}^{0 \\to 1} [F(y)]-F(x)-\\int_{0}^{1} \\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} [\\eta] \\mathrm{d} t \\right\\| \n\t\\leq c_{1} \\|\\eta\\|^{2},\n\t\\end{equation*}\n\twhere $\\eta=\\mathrm{R}_{x}^{-1}(y)$ and $\\mathrm{P}_{\\gamma}$ is the parallel transport along the curve $\\gamma(t):=\\mathrm{R}_{x}\\left(t \\eta\\right)$.\n\tMoreover, if $\\mathrm{R}=\\operatorname{Exp}$, then indeed $c_1=0$ above (see \\cite[equation (2.4)]{ferreira2002kantorovich}).\n\\end{lemma}\n","start":52312,"end":53221}]},{"id":"S037","semantic_type":"lemma","kind":"lemma","labels":["lem:nonsingular"],"member_ids":["S037"],"declared_external":true,"declaration_text":["原先决条件清单：Fernandes, Ferreira, and Yuan (2017), Lemma 3.2；原文 B0065 重述。"],"source_only":false,"title":"局部逆算子一致有界","source_units":[{"raw_tex":"\\begin{lemma}[{\\cite[Lemma 3.2]{fernandes2017superlinear}}]\\label{lem:nonsingular}\n\tGiven a vector field $F$ on $\\mathcal{M}$.\n\tIf the map $p \\mapsto \\nabla F(p)$ is continuous at $p^{*}$ and $\\nabla F(p^{*})$ is nonsingular, \n\tthen there exist a neighborhood $\\mathcal{U}$ of $p^{*}$ and a constant $\\Xi > 0$ such that, for all $p \\in \\mathcal{U}$, $\\nabla F(p)$ is nonsingular and $\\left\\|\\nabla F(p)^{-1}\\right\\| \\leq \\Xi$.\n\\end{lemma}\n","start":55894,"end":56333}]},{"id":"S038","semantic_type":"lemma","kind":"lemma","labels":["lem:dist_zero"],"member_ids":["S038"],"declared_external":true,"declaration_text":["原先决条件清单：Gallivan, Qi, and Absil (2012), Lemma 14.5；原文 B0066 重述。"],"source_only":false,"title":"残差与解距离等价","source_units":[{"raw_tex":"\\begin{lemma}[{\\cite[Lemma 14.5]{gallivan2012riemannian}}]\\label{lem:dist_zero}\n\tLet $F$ be a  $C^2$ vector field on $\\mathcal{M}$ and $p^{*} \\in \\mathcal{M}$. If $ F(p^{*})=0 $ and $\\nabla F(p^{*}) $ is nonsingular, then there exist a neighborhood $\\mathcal{U}$ of $p^{*}$ and constants $c_{3}, c_{4} > 0$ such that, for all $p \\in \\mathcal{U}$, $ c_{3} d(p, p^{*}) \\leq \\|F(p)\\| \\leq c_{4} d(p, p^{*}).$\n\\end{lemma}\n","start":56333,"end":56751}]},{"id":"S039","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"渐近阶记号","source_units":[{"raw_tex":"Here, for any two nonnegative sequences $\\left\\{u_{k}\\right\\}$ and $\\left\\{v_{k}\\right\\}$, we write $u_{k}=O(v_{k})$ if there is a constant $M >0$ such that\n$u_{k}\\leq M v_{k}$\nfor all sufficiently large $k$; and we write\n$u_{k}=o(v_{k})$\nif $v_{k}>0$ and the sequence of ratios $\\{u_{k} / v_{k}\\} $ approaches zero.\n","start":57284,"end":57601}]},{"id":"S040","semantic_type":"algorithm","kind":"algorithm","labels":["algo:PerturbedDampedNewtonMethod"],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"扰动阻尼黎曼 Newton 法","source_units":[{"raw_tex":"\\begin{algorithm}%[H]\n\t\\caption{Perturbed Damped Riemannian Newton Method for (\\ref{eq:Singularity})}\n\t\\label{algo:PerturbedDampedNewtonMethod}\n\t\\SetAlgoLined\n\t\n\t\\KwIn{A vector field $F \\in \\mathfrak{X}(\\mathcal{M})$, an initial point $p_0 \\in \\mathcal{M}$ and a retraction $\\mathrm{R}$ on $\\mathcal{M}$. Set $\\mu_0>0$.}\n\t\\KwOut{Sequence $\\{p_k\\} \\subset \\mathcal{M}$ such that $\\left\\{p_k\\right\\} \\rightarrow p^*$ and $F(p^{*})=0_{p^{*}}$.}\n\t\n\tSet $k \\to 0$\\;\n\t\n\t\\While{Stopping criterion not satisfied}{\n\t\t1. Obtain $\\xi_k \\in T_{p_k}\\mathcal{M}$ by solving the perturbed Newton equation:\n\t\t\\begin{equation}\n\t\t\t\\nabla F(p_{k}) [\\xi_k] = - F(p_{k}) + \\mu_{k} \\hat{e};\n\t\t\\end{equation}\n\t\n\t\t2. Choose a (damped) step size $ 0<\\alpha_{k} \\leq 1$\\;\n\t\t\n\t\t3. Compute the next point as $p_{k+1}:=\\mathrm{R}_{p_{k}} (\\alpha_{k} \\xi_k)$\\;\n\t\t\n\t\t4. Choose $0<\\mu_{k+1}<\\mu_k$\\;\n\t\t\n\t\t5. $k \\to k + 1$\\;\n\t}\n\\end{algorithm}\n","start":58004,"end":58915}]},{"id":"S041","semantic_type":"assumption","kind":"assumption","labels":["A1","A2","A3"],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"Newton 局部假设","source_units":[{"raw_tex":"\\begin{assumptions}[leftmargin=9mm]\n\t\\item There exists $p^{*} \\in \\mathcal{M}$ such that $F(p^{*})=0_{p^{*}}$;\n\t\\label{A1}\n\t\\item The covariant derivative $\\nabla F (p^{*})$ is nonsingular;\n\t\\label{A2}\n%\t\\item The map $p \\mapsto \\nabla F(p) $ is Lipschitz continuous with respect to on a neighborhood of $p^{*}$;\n%\t\\label{A3}\n\t\\item The vector field $F$ is $C^2$.\n\t\\label{A3}\n\\end{assumptions}\n","start":59463,"end":59858}]},{"id":"S042","semantic_type":"lemma","kind":"lemma","labels":["lem:gamma_alpha"],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"中心参数控制阻尼缺口","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:gamma_alpha}\n\tConsider the Algorithm \\ref{algo:prototype_algo} for solving the problem (\\ref{RCOP}).  \n\tLet \\ref{B1} and \\ref{B4} hold at some $w^{*}=(x^{*}, y^{*}, z^{*}, s^{*})$ and $ \\alpha_{k} $ be as in (\\ref{rule:StepSizeA}). \n\tDefine a constant,\n\t\\begin{equation*}\n\t\\Pi := 2 \\max \\left\\{\\max _{i}\\left\\{\n\t1/(s^{*})_{i}\n\t\\mid (s^{*})_{i}>0\\right\\}, \\max _{i}\\left\\{\n\t1/(z^{*})_{i}\n\t\\mid (z^{*})_{i}>0\\right\\}\\right\\}.\n\t\\end{equation*}\n\tFor $\\gamma_{k} \\in(0,1)$, if\n\t$\n\t\\Pi \\left\\| \\Delta w_{k}\\right\\| \\leq \\gamma_{k},\n\t$\n\tthen\n\t$\n\t0  \\leq 1-\\alpha_{k} \\leq\n\t(1-\\gamma_{k})+\\Pi \\left\\| \\Delta w_{k}\\right\\|.\n\t$\n\\end{lemma}","start":65558,"end":66211}]},{"id":"S043","semantic_type":"theorem","kind":"theorem","labels":["prop:non"],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"解处 KKT 导数非奇异","source_units":[{"raw_tex":"\\begin{theorem}\\label{prop:non}\n\tIf the standard Riemannian assumptions \\ref{B1}-\\ref{B5} hold at some point $w^{*}$,\n\tthen the operator $\\nabla F(w^{*})$ in (\\ref{eq:CovariantDerivativeKKTVectorField}) is nonsingular.\n\\end{theorem}","start":23641,"end":23873}]},{"id":"S044","semantic_type":"proof","kind":"proof","labels":["13","sing1"],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"KKT 非奇异性的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nThis proof omits all the asterisks of the variables. Define $ \\mathbb{E}:=\\{1,\\dots,l\\}$ and $\\mathbb{I}:=\\{1,\\dots,m\\} $. Take some $w=(x, y, z, s) \\in \\mathcal{N} $ satisfying \\ref{B1}-\\ref{B5}, then we have $ s_{i} = -g_{i}(x)$ and $z_{i} s_{i} = 0$ for all $ i \\in \\mathbb{I}$. For short, let $\\mathbb{A}:=\\mathbb{A}(x) \\subset \\mathbb{I} $. \nSuppose that $\\nabla {F}(w) [\\Delta w] =0$ for some $\\Delta w = (\\Delta x, \\Delta y, \\Delta z, \\Delta s) \\in T_{w} \\mathcal{N} \\cong T_x \\mathcal{M} \\times \\mathbb{R}^l \\times \\mathbb{R}^m \\times \\mathbb{R}^m$.\n$\\Delta y_i$ denotes the components of the vector $\\Delta y$, as do $\\Delta z_i$, $\\Delta s_i$.\nTo prove its nonsingularity, we will show that $ \\Delta w=0$.\nExpanding the equation $\\nabla {F}(w) [\\Delta w] =0$ gives\n\\begin{equation}\\label{13}\n\t\\left\\{\n\t\\begin{aligned}\n\t\t0&= \\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n\t\t+\\sum_{i\\in \\mathbb{I}} \\Delta z_{i} \\operatorname{grad} g_{i}(x),\\\\\n\t\t0&=\\left\\langle\\operatorname{grad}h_{i}(x), \\Delta x\\right\\rangle, \\text { for all } i\\in \\mathbb{E},\\\\\n\t\t0&=\\left\\langle\\operatorname{grad} g_{i}(x), \\Delta x\\right\\rangle + \\Delta s_{i}, \\text { for all } i\\in \\mathbb{I}, \\\\\n\t\t0&=z_{i} \\Delta s_{i} + s_{i} \\Delta z_{i}, \\text { for all } i \\in \\mathbb{I}. \n\t\\end{aligned}\n\t\\right.\n\\end{equation}\nStrict complementarity \\ref{B4} and the last equalities above imply that $ \\Delta s_{i}=0 $\nfor all $ i \\in \\mathbb{A}$ and $ \\Delta z_{i}=0 $\nfor $ i \\in  \\mathbb{I} \\setminus \\mathbb{A}$.\nSubstituting those values into the system (\\ref{13}) reduces it to\n\\begin{equation}\\label{sing1}\n\t\\left\\{\n\t\\begin{aligned}\n\t\t0&= \\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n\t\t+\\sum_{i\\in \\mathbb{A}} \\Delta z_{i} \\operatorname{grad} g_{i}(x), \\\\\n\t\t0&=\\left\\langle\\operatorname{grad}h_{i}(x), \\Delta x\\right\\rangle, \\text { for all } i\\in \\mathbb{E},\\\\\n\t\t0&=\\left\\langle\\operatorname{grad}g_{i}(x), \\Delta x\\right\\rangle, \\text { for all } i\\in \\mathbb{A},\n\t\\end{aligned}\n\t\\right.\n\\end{equation}\nand\n$\n\\Delta s_{i}=-\\left\\langle\\operatorname{grad} g_{i}(x), \\Delta x\\right\\rangle \\text { for all } i\\in \\mathbb{I} \\setminus \\mathbb{A}.\n$\nIt follows from system (\\ref{sing1}) that\n\\begin{align*}\n\t0\n\t&=  \\langle \\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n\t+\\sum_{i\\in  \\mathbb{A}} \\Delta z_{i} \\operatorname{grad} g_{i}(x) , \\Delta x \\rangle \\\\\n\t&= \\left\\langle\\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x], \\Delta x\\right\\rangle\n\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\left\\langle\\operatorname{grad}h_{i}(x), \\Delta x\\right\\rangle\n\t+\\sum_{i\\in  \\mathbb{A}} \\Delta z_{i} \\left\\langle\\operatorname{grad}g_{i}(x), \\Delta x\\right\\rangle \\\\\n\t&= \\left\\langle\\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x], \\Delta x\\right\\rangle.\n\\end{align*}\nThus, from second-order sufficiency \\ref{B5}, $ \\Delta x$ must be zero elements.\nAnd then $\\Delta s_{i}=0 \\text { for all } i\\in \\mathbb{I} \\setminus  \\mathbb{A}.$\nNext, substituting $ \\Delta x=0$ into the first equation in (\\ref{sing1}) yields\n$\n0=\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n+\\sum_{i\\in  \\mathbb{A}} \\Delta z_{i} \\operatorname{grad} g_{i}(x).\n$\nThe LICQ \\ref{B3} implies that the coefficients $\\Delta y_{i}$ for $i\\in \\mathbb{E}$ and $ \\Delta z_{i} $ for $ i\\in  \\mathbb{A} $ must be zero. \nThis completes the proof.\n\\end{proof}","start":23874,"end":27407}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["13"],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"展开核方程","source_units":[{"raw_tex":"\\begin{proof}\nThis proof omits all the asterisks of the variables. Define $ \\mathbb{E}:=\\{1,\\dots,l\\}$ and $\\mathbb{I}:=\\{1,\\dots,m\\} $. Take some $w=(x, y, z, s) \\in \\mathcal{N} $ satisfying \\ref{B1}-\\ref{B5}, then we have $ s_{i} = -g_{i}(x)$ and $z_{i} s_{i} = 0$ for all $ i \\in \\mathbb{I}$. For short, let $\\mathbb{A}:=\\mathbb{A}(x) \\subset \\mathbb{I} $. \nSuppose that $\\nabla {F}(w) [\\Delta w] =0$ for some $\\Delta w = (\\Delta x, \\Delta y, \\Delta z, \\Delta s) \\in T_{w} \\mathcal{N} \\cong T_x \\mathcal{M} \\times \\mathbb{R}^l \\times \\mathbb{R}^m \\times \\mathbb{R}^m$.\n$\\Delta y_i$ denotes the components of the vector $\\Delta y$, as do $\\Delta z_i$, $\\Delta s_i$.\nTo prove its nonsingularity, we will show that $ \\Delta w=0$.\nExpanding the equation $\\nabla {F}(w) [\\Delta w] =0$ gives\n\\begin{equation}\\label{13}\n\t\\left\\{\n\t\\begin{aligned}\n\t\t0&= \\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n\t\t+\\sum_{i\\in \\mathbb{I}} \\Delta z_{i} \\operatorname{grad} g_{i}(x),\\\\\n\t\t0&=\\left\\langle\\operatorname{grad}h_{i}(x), \\Delta x\\right\\rangle, \\text { for all } i\\in \\mathbb{E},\\\\\n\t\t0&=\\left\\langle\\operatorname{grad} g_{i}(x), \\Delta x\\right\\rangle + \\Delta s_{i}, \\text { for all } i\\in \\mathbb{I}, \\\\\n\t\t0&=z_{i} \\Delta s_{i} + s_{i} \\Delta z_{i}, \\text { for all } i \\in \\mathbb{I}. \n\t\\end{aligned}\n\t\\right.\n\\end{equation}\n","start":23874,"end":25262}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["sing1"],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"严格互补约化核系统","source_units":[{"raw_tex":"Strict complementarity \\ref{B4} and the last equalities above imply that $ \\Delta s_{i}=0 $\nfor all $ i \\in \\mathbb{A}$ and $ \\Delta z_{i}=0 $\nfor $ i \\in  \\mathbb{I} \\setminus \\mathbb{A}$.\nSubstituting those values into the system (\\ref{13}) reduces it to\n\\begin{equation}\\label{sing1}\n\t\\left\\{\n\t\\begin{aligned}\n\t\t0&= \\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n\t\t+\\sum_{i\\in \\mathbb{A}} \\Delta z_{i} \\operatorname{grad} g_{i}(x), \\\\\n\t\t0&=\\left\\langle\\operatorname{grad}h_{i}(x), \\Delta x\\right\\rangle, \\text { for all } i\\in \\mathbb{E},\\\\\n\t\t0&=\\left\\langle\\operatorname{grad}g_{i}(x), \\Delta x\\right\\rangle, \\text { for all } i\\in \\mathbb{A},\n\t\\end{aligned}\n\t\\right.\n\\end{equation}\nand\n$\n\\Delta s_{i}=-\\left\\langle\\operatorname{grad} g_{i}(x), \\Delta x\\right\\rangle \\text { for all } i\\in \\mathbb{I} \\setminus \\mathbb{A}.\n$\n","start":25262,"end":26162}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"二阶充分性消去主变量","source_units":[{"raw_tex":"It follows from system (\\ref{sing1}) that\n\\begin{align*}\n\t0\n\t&=  \\langle \\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x]\n\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n\t+\\sum_{i\\in  \\mathbb{A}} \\Delta z_{i} \\operatorname{grad} g_{i}(x) , \\Delta x \\rangle \\\\\n\t&= \\left\\langle\\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x], \\Delta x\\right\\rangle\n\t+\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\left\\langle\\operatorname{grad}h_{i}(x), \\Delta x\\right\\rangle\n\t+\\sum_{i\\in  \\mathbb{A}} \\Delta z_{i} \\left\\langle\\operatorname{grad}g_{i}(x), \\Delta x\\right\\rangle \\\\\n\t&= \\left\\langle\\operatorname{Hess}_{x} \\mathcal{L}(w) [\\Delta x], \\Delta x\\right\\rangle.\n\\end{align*}\nThus, from second-order sufficiency \\ref{B5}, $ \\Delta x$ must be zero elements.\nAnd then $\\Delta s_{i}=0 \\text { for all } i\\in \\mathbb{I} \\setminus  \\mathbb{A}.$\n","start":26162,"end":27004}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"LICQ 消去剩余乘子","source_units":[{"raw_tex":"Next, substituting $ \\Delta x=0$ into the first equation in (\\ref{sing1}) yields\n$\n0=\\sum_{i\\in \\mathbb{E}} \\Delta y_{i} \\operatorname{grad} h_{i}(x)\n+\\sum_{i\\in  \\mathbb{A}} \\Delta z_{i} \\operatorname{grad} g_{i}(x).\n$\nThe LICQ \\ref{B3} implies that the coefficients $\\Delta y_{i}$ for $i\\in \\mathbb{E}$ and $ \\Delta z_{i} $ for $ i\\in  \\mathbb{A} $ must be zero. \nThis completes the proof.\n\\end{proof}","start":27004,"end":27407}]},{"id":"S049","semantic_type":"lemma","kind":"lemma","labels":["lem:R1est"],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"向量场的一阶展开余项","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:R1est}\n\tLet $F$ be a $ C^2 $ vector field and $\\bar{x}\\in\\mathcal{M}$. Then there exist a neighborhood $\\mathcal{U}$ of $\\bar{x}$ and a constant $c_{2}>0$ such that for all $x \\in \\mathcal{U}$,\n\t\\begin{equation*}\n\t\\left\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} [F(x)]-F(\\bar{x})-\\nabla F(\\bar{x}) [\\eta] \n\t\\right\\|\n\t \\leq c_{2} d^{2}(\\bar{x} , x),\n\t\\end{equation*}\n\twhere $\\eta=\\mathrm{R} _{\\bar{x}}^{-1} x$ and $\\mathrm{P}_{\\gamma}$ is the parallel transport along the curve $\\gamma(t):=\\mathrm{R}_{\\bar{x}}(t \\eta)$.\n\\end{lemma}","start":53373,"end":53917}]},{"id":"S050","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"二阶余项界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof} \nLet $LHS:=\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} [F(x)]-F(\\bar{x})-\\nabla F(\\bar{x}) [\\eta] \\|.$ It follows that\n\\begin{equation*}\n\t\t\\begin{aligned}\n\t\t\tLHS\n\t\t\t& \\leq\n\t\t\t\\left\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} [F(x)]-F(\\bar{x})\n\t\t\t-\\int_{0}^{1} \\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} [\\eta] \\mathrm{d} t  \\right\\| \\\\\n\t\t\t& \\quad +\n\t\t\t\\left\\|\\int_{0}^{1} \\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} [\\eta] \\mathrm{d} t -\\nabla F(\\bar{x}) [\\eta] \\right\\| \\\\\n\t\t\t&\\leq \n\t\t\tc_{1} \\|\\eta\\|^{2} +\n\t\t\t\\Big\\|\n\t\t\t\\int_{0}^{1}\\left(\\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} -\\nabla F(\\bar{x})\\right)\n\t\t\t[\\eta] \\mathrm{d} t\\Big\\|.\n\t\t\t\\text{ (by Lemma \\ref{lem:FundamentalTheoremOfCalculusRiemannian})}\n\t\t\\end{aligned}\n\\end{equation*}\nLet\n$\\theta := \\left\\| \\int_{0}^{1}\\left(\\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} -\\nabla F(\\bar{x})\\right)\n[\\eta] \\mathrm{d} t \\right\\|$.\nNote that\n\\begin{align*}\n\t\\theta \n\t&\\leq \n\t\\int_{0}^{1} \n\t\\left\\|\n\t\\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} -\\nabla F(\\bar{x})\n\t\\right\\|\n\t\\left\\| \\eta \\right\\|\n\t\\mathrm{d} t \\\\\n\t&\\leq  \n\t\\int_{0}^{1} \n\tc_{0} d(\\bar{x}, \\mathrm{R}_{\\bar{x}}(t \\eta))\n\t\\left\\| \\eta \\right\\|\n\t\\mathrm{d} t\t\\text{ (by Lemma \\ref{lem:LipCountNabulaF})} \\\\\n\t&\\leq   \n\t\\int_{0}^{1} \n\tc_{0} a_{1}t \\|\\eta\\|\n\t\\left\\| \\eta \\right\\|\n\t\\mathrm{d} t \t\n\t= \n\t\\frac{1}{2} \n\tc_{0} a_{1} \\|\\eta\\|^{2}. \t\n\t\\text{ (by (i) of Lemma \\ref{lem:distR1})}\n\\end{align*}\nCombining the above results yields\n$$\nLHS\n\\leq (c_{1}+ \\frac{1}{2} \nc_{0} a_{1})\n\\|\\eta\\|^{2}\n\\leq  \n(c_{1}+ \\frac{1}{2} \nc_{0} a_{1})  / a_{0}^{2}\nd^{2}(\\bar{x} , x),\n$$\nwhere the last inequality comes from (ii) of Lemma \\ref{lem:distR1}. Letting $c_{2}:= (c_{1}+ \\frac{1}{2} \nc_{0} a_{1})  / a_{0}^{2} $ completes the proof.\n\\end{proof}","start":53918,"end":55838}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"基本估计分离余项","source_units":[{"raw_tex":"\\begin{proof} \nLet $LHS:=\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} [F(x)]-F(\\bar{x})-\\nabla F(\\bar{x}) [\\eta] \\|.$ It follows that\n\\begin{equation*}\n\t\t\\begin{aligned}\n\t\t\tLHS\n\t\t\t& \\leq\n\t\t\t\\left\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} [F(x)]-F(\\bar{x})\n\t\t\t-\\int_{0}^{1} \\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} [\\eta] \\mathrm{d} t  \\right\\| \\\\\n\t\t\t& \\quad +\n\t\t\t\\left\\|\\int_{0}^{1} \\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} [\\eta] \\mathrm{d} t -\\nabla F(\\bar{x}) [\\eta] \\right\\| \\\\\n\t\t\t&\\leq \n\t\t\tc_{1} \\|\\eta\\|^{2} +\n\t\t\t\\Big\\|\n\t\t\t\\int_{0}^{1}\\left(\\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} -\\nabla F(\\bar{x})\\right)\n\t\t\t[\\eta] \\mathrm{d} t\\Big\\|.\n\t\t\t\\text{ (by Lemma \\ref{lem:FundamentalTheoremOfCalculusRiemannian})}\n\t\t\\end{aligned}\n\\end{equation*}\n","start":53918,"end":54758}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"积分项的二次估计","source_units":[{"raw_tex":"Let\n$\\theta := \\left\\| \\int_{0}^{1}\\left(\\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} -\\nabla F(\\bar{x})\\right)\n[\\eta] \\mathrm{d} t \\right\\|$.\nNote that\n\\begin{align*}\n\t\\theta \n\t&\\leq \n\t\\int_{0}^{1} \n\t\\left\\|\n\t\\mathrm{P}_{\\gamma}^{0 \\to t} \\nabla F(\\gamma(t)) \\mathrm{P}_{\\gamma}^{t \\to 0} -\\nabla F(\\bar{x})\n\t\\right\\|\n\t\\left\\| \\eta \\right\\|\n\t\\mathrm{d} t \\\\\n\t&\\leq  \n\t\\int_{0}^{1} \n\tc_{0} d(\\bar{x}, \\mathrm{R}_{\\bar{x}}(t \\eta))\n\t\\left\\| \\eta \\right\\|\n\t\\mathrm{d} t\t\\text{ (by Lemma \\ref{lem:LipCountNabulaF})} \\\\\n\t&\\leq   \n\t\\int_{0}^{1} \n\tc_{0} a_{1}t \\|\\eta\\|\n\t\\left\\| \\eta \\right\\|\n\t\\mathrm{d} t \t\n\t= \n\t\\frac{1}{2} \n\tc_{0} a_{1} \\|\\eta\\|^{2}. \t\n\t\\text{ (by (i) of Lemma \\ref{lem:distR1})}\n\\end{align*}\n","start":54758,"end":55500}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"转为距离平方余项","source_units":[{"raw_tex":"Combining the above results yields\n$$\nLHS\n\\leq (c_{1}+ \\frac{1}{2} \nc_{0} a_{1})\n\\|\\eta\\|^{2}\n\\leq  \n(c_{1}+ \\frac{1}{2} \nc_{0} a_{1})  / a_{0}^{2}\nd^{2}(\\bar{x} , x),\n$$\nwhere the last inequality comes from (ii) of Lemma \\ref{lem:distR1}. Letting $c_{2}:= (c_{1}+ \\frac{1}{2} \nc_{0} a_{1})  / a_{0}^{2} $ completes the proof.\n\\end{proof}","start":55500,"end":55838}]},{"id":"S054","semantic_type":"proposition","kind":"proposition","labels":["prop:PD_Newton"],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"扰动阻尼 Newton 局部收敛","source_units":[{"raw_tex":"\\begin{proposition}[Local convergence of Algorithm \\ref{algo:PerturbedDampedNewtonMethod}]\\label{prop:PD_Newton}\n\tConsider the perturbed damped Riemannian Newton method\n\tdescribed in Algorithm \\ref{algo:PerturbedDampedNewtonMethod} for the singularity problem (\\ref{eq:Singularity}). \n\tLet \\ref{A1}-\\ref{A3} hold.\n\tChoose parameters $ \\mu_{k}, \\alpha_{k}$ as follows; then there exists a constant $\\delta>0$ such that for all $p_{0}\\in \\mathcal{M}$ with\n\t$\n\td(p_{0} , p^{*})<\\delta,\n\t$\n\tthe sequence $\\left\\{p_{k}\\right\\}$ is well defined.\n\tFurthermore,\n\t\\begin{enumerate}[(1)]\n\t\t\\item if we choose $\\mu_{k}=o(\\|F(p_{k})\\|)$ and $\\alpha_{k} \\to 1$, then $p_{k} \\to p^{*}$ superlinearly;\t\t\n\t\t\\item if we choose $\\mu_{k}=O(\\|F(p_{k})\\|^{2})$ and $1-\\alpha_{k}=O(\\|F(p_{k})\\|)$, then $p_{k} \\to p^{*}$ quadratically.\n\t\\end{enumerate}\n\\end{proposition}","start":60387,"end":61235}]},{"id":"S055","semantic_type":"proof","kind":"proof","labels":["77","eq:1114","eq:LipFbigoh","eq:bigoh","eq:superlinear"],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"扰动阻尼 Newton 的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nBy \\ref{A2}-\\ref{A3}, Lemma \\ref{lem:LipCountNabulaF} and Lemma \\ref{lem:nonsingular}, we can let $p_{k} $ be sufficiently close to $p^{*}$ such that $\\nabla F(p_{k})$ is nonsingular, and\n$\\left\\|\\nabla F(p_{k})^{-1}\\right\\| \\leq \\Xi$ for some constant $\\Xi$.\nThen, the next iterate point,\n\\begin{equation*}\np_{k+1}:=\\mathrm{R}_{p_{k}}[\\alpha_{k} \\nabla F(p_{k})^{-1}(-F(p_{k})+\\mu_{k} \\hat{e})],\n\\end{equation*}\nis well defined in Algorithm \\ref{algo:PerturbedDampedNewtonMethod}, and it follows from $ p^{*}=\\mathrm{R}_{p_{k}}(\\eta) $ with $ \\eta := \\mathrm{R}_{p_{k}}^{-1} p^{*} $ and (i) of Lemma \\ref{lem:distR1} that\n\\begin{align}\n\td(p_{k+1} , p^{*})\n\t&\\leq\n\ta_{1}\n\t\\| \\eta - \\alpha_{k} \\nabla F(p_{k})^{-1}(-F(p_{k})+\\mu_{k} \\hat{e}) \\| \\notag \\\\\n\t&=\n\ta_{1}  \n\t\\|\\eta + \\alpha_{k} \\nabla F(p_{k})^{-1}(F(p_{k})-\\mu_{k} \\hat{e})\\|. \\label{eq:1114}\n\\end{align}\nLet \n$r_{k} := \\eta + \\alpha_{k} \\nabla F(p_{k})^{-1}(F(p_{k})-\\mu_{k} \\hat{e})$.\nAlgebraic manipulations\nshow that\n\\begin{equation*}\nr_{k}=\n(1-\\alpha_{k}) \\eta  +\n\\alpha_{k} \\nabla F(p_{k})^{-1} \n[\\nabla F(p_{k}) \\eta + F(p_{k})-\\mathrm{P}_{\\gamma}^{0 \\to 1} F(p^{*})-\\mu_{k} \\hat{e}],\n\\end{equation*}\nwhere $\\mathrm{P}_{\\gamma}$ is the parallel transport along the curve $\\gamma(t)=\\mathrm{R}_{p_{k}}(t \\eta)$ and $F(p^{*})=0.$\nThus, using $\\left\\|\\eta\\right\\| \\leq d(p_{k}, p^{*})/a_{0}$ from (ii) of Lemma \\ref{lem:distR1} and Lemma \\ref{lem:R1est}, we have\n\\begin{align*}\n\t\\| r_{k}\\|&\\leq\n\t(1-\\alpha_{k}) \\| \\eta\\| \\notag \n\t+ \n\t\\alpha_{k} \\|\\nabla F(p_{k})^{-1} \\|\n\t\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} F(p^{*}) - F(p_{k}) -\n\t\\nabla F(p_{k}) \\eta \\| \\\\\n\t& \\quad + \\alpha_{k} \\|\\nabla F(p_{k})^{-1} \\| \\|\\hat{e}\\| \\mu_{k} \\\\ \n\t&\\leq (1-\\alpha_{k})d(p_{k} , p^{*}) /a_{0} +\n\t\\alpha_{k} \\|\\nabla F(p_{k})^{-1} \\| \n\tc_{2} d^{2}(p_{k} , p^{*})\\\\\n\t& \\quad\n\t+ \\alpha_{k}\\|\\nabla F(p_{k})^{-1} \\|  \\|\\hat{e}\\| \\mu_{k}\\\\\n\t&\\leq (1-\\alpha_{k})d(p_{k} , p^{*}) /a_{0} +\n\t\\Xi \n\tc_{2} d^{2}(p_{k} , p^{*})\n\t+ \\Xi  \\|\\hat{e}\\| \\mu_{k}.\n\\end{align*}\nCombining the above with (\\ref{eq:1114}), we conclude that\n\\begin{equation}\\label{77}\n\td(p_{k+1} , p^{*}) \\leq \\kappa_{1}(1-\\alpha_{k}) d(p_{k} , p^{*}) \n\t+ \\kappa_{2} d^{2}(p_{k} , p^{*})\n\t+ \\kappa_{3} \\mu_{k}\n\\end{equation}\nfor some positive constants $ \\kappa_{1},\\kappa_{2},\\kappa_{3}$.\nOn the other hand, by Lemma \\ref{lem:dist_zero}, we have\n\\begin{equation}\\label{eq:LipFbigoh}\n\t\\left\\|F(p_{k})\\right\\|=O(d(p_{k}, p^{*})).\n\\end{equation}\nIn what follows, we prove assertions (1) and (2).\n\n\\textbf{(1)} \nSuppose that $\\alpha_{k} \\to 1$ and $\\mu_{k}=o(\\|F(p_{k})\\|)$, which together with (\\ref{eq:LipFbigoh}) imply $\\mu_{k}=o(d(p_{k}, p^{*}))$.\nBy (\\ref{77}), we have\n\\begin{equation}\\label{eq:superlinear}\n\t\\frac{d(p_{k+1} , p^{*})}{d(p_{k} , p^{*})}\\leq\n\t\\kappa_{1} (1-\\alpha_{k})\n\t+\\kappa_{2} d(p_{k} , p^{*})\n\t+\\kappa_{3}\\frac{\\mu_{k}}{d(p_{k} , p^{*})}.\n\\end{equation}\nWe can take $\\delta$ sufficiently small and $ k $ sufficiently large, if necessary, to conclude that\n$\nd(p_{k+1} , p^{*})<\\frac{1}{2}d(p_{k} , p^{*}) <\\delta.\n$\nThus, $p_{k+1} \\in B_{\\delta}(p^{*})$, the open ball of radius $\\delta$ centered at $p^{*}$ on $\\mathcal{M}$. \nBy induction, it is easy to show that the sequence $\\left\\{p_{k}\\right\\}$ is well defined and converges to $p^{*}$. \nTaking the limit of both sides of (\\ref{eq:superlinear}) proves superlinear convergence.\n\n\n\\textbf{(2)} \nAgain, we start from (\\ref{77}) and rewrite it as:\n\\begin{align}\\label{eq:bigoh}\n\td(p_{k+1} , p^{*}) = (1-\\alpha_{k}) O(d(p_{k} , p^{*}) )\n\t+O(d^{2}(p_{k} , p^{*}) )\n\t+O(\\mu_{k}).\n\\end{align}\nSuppose that $1-\\alpha_{k}=O(\\left\\|F(p_{k})\\right\\|)$\nand $\\mu_{k}=O(\\left\\|F(p_{k})\\right\\|^{2})$.\nUsing (\\ref{eq:LipFbigoh}), the above reduces to\n$\nd(p_{k+1} , p^{*}) = O(d^{2}(p_{k} , p^{*})).\n$\nThis implies that there exists a constant $\\nu$ such that $d(p_{k+1} , p^{*}) \\leq \\nu d^{2}(p_{k} , p^{*})$, and hence,\n$\nd(p_{k+1} , p^{*}) \\leq \\nu d^{2}(p_{k} , p^{*}) \\leq \\nu \\delta^{2}<\\delta,\n$\nif $\\delta$ is sufficiently small.\nAgain, by induction, $\\left\\{p_{k}\\right\\}$ converges to $p^{*}$ quadratically.\n\\end{proof}","start":61236,"end":65344}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"局部可逆性与更新定义","source_units":[{"raw_tex":"\\begin{proof}\nBy \\ref{A2}-\\ref{A3}, Lemma \\ref{lem:LipCountNabulaF} and Lemma \\ref{lem:nonsingular}, we can let $p_{k} $ be sufficiently close to $p^{*}$ such that $\\nabla F(p_{k})$ is nonsingular, and\n$\\left\\|\\nabla F(p_{k})^{-1}\\right\\| \\leq \\Xi$ for some constant $\\Xi$.\nThen, the next iterate point,\n\\begin{equation*}\np_{k+1}:=\\mathrm{R}_{p_{k}}[\\alpha_{k} \\nabla F(p_{k})^{-1}(-F(p_{k})+\\mu_{k} \\hat{e})],\n\\end{equation*}\n","start":61236,"end":61663}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":["eq:1114"],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"回缩距离误差界","source_units":[{"raw_tex":"is well defined in Algorithm \\ref{algo:PerturbedDampedNewtonMethod}, and it follows from $ p^{*}=\\mathrm{R}_{p_{k}}(\\eta) $ with $ \\eta := \\mathrm{R}_{p_{k}}^{-1} p^{*} $ and (i) of Lemma \\ref{lem:distR1} that\n\\begin{align}\n\td(p_{k+1} , p^{*})\n\t&\\leq\n\ta_{1}\n\t\\| \\eta - \\alpha_{k} \\nabla F(p_{k})^{-1}(-F(p_{k})+\\mu_{k} \\hat{e}) \\| \\notag \\\\\n\t&=\n\ta_{1}  \n\t\\|\\eta + \\alpha_{k} \\nabla F(p_{k})^{-1}(F(p_{k})-\\mu_{k} \\hat{e})\\|. \\label{eq:1114}\n\\end{align}\n","start":61663,"end":62116}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"分离阻尼与线性化余项","source_units":[{"raw_tex":"Let \n$r_{k} := \\eta + \\alpha_{k} \\nabla F(p_{k})^{-1}(F(p_{k})-\\mu_{k} \\hat{e})$.\nAlgebraic manipulations\nshow that\n\\begin{equation*}\nr_{k}=\n(1-\\alpha_{k}) \\eta  +\n\\alpha_{k} \\nabla F(p_{k})^{-1} \n[\\nabla F(p_{k}) \\eta + F(p_{k})-\\mathrm{P}_{\\gamma}^{0 \\to 1} F(p^{*})-\\mu_{k} \\hat{e}],\n\\end{equation*}\nwhere $\\mathrm{P}_{\\gamma}$ is the parallel transport along the curve $\\gamma(t)=\\mathrm{R}_{p_{k}}(t \\eta)$ and $F(p^{*})=0.$\n","start":62116,"end":62546}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"切空间误差的三项界","source_units":[{"raw_tex":"Thus, using $\\left\\|\\eta\\right\\| \\leq d(p_{k}, p^{*})/a_{0}$ from (ii) of Lemma \\ref{lem:distR1} and Lemma \\ref{lem:R1est}, we have\n\\begin{align*}\n\t\\| r_{k}\\|&\\leq\n\t(1-\\alpha_{k}) \\| \\eta\\| \\notag \n\t+ \n\t\\alpha_{k} \\|\\nabla F(p_{k})^{-1} \\|\n\t\\|\\mathrm{P}_{\\gamma}^{0 \\to 1} F(p^{*}) - F(p_{k}) -\n\t\\nabla F(p_{k}) \\eta \\| \\\\\n\t& \\quad + \\alpha_{k} \\|\\nabla F(p_{k})^{-1} \\| \\|\\hat{e}\\| \\mu_{k} \\\\ \n\t&\\leq (1-\\alpha_{k})d(p_{k} , p^{*}) /a_{0} +\n\t\\alpha_{k} \\|\\nabla F(p_{k})^{-1} \\| \n\tc_{2} d^{2}(p_{k} , p^{*})\\\\\n\t& \\quad\n\t+ \\alpha_{k}\\|\\nabla F(p_{k})^{-1} \\|  \\|\\hat{e}\\| \\mu_{k}\\\\\n\t&\\leq (1-\\alpha_{k})d(p_{k} , p^{*}) /a_{0} +\n\t\\Xi \n\tc_{2} d^{2}(p_{k} , p^{*})\n\t+ \\Xi  \\|\\hat{e}\\| \\mu_{k}.\n\\end{align*}\n","start":62546,"end":63251}]},{"id":"S060","semantic_type":"derived_fact","kind":"derived_fact","labels":["77"],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"Newton 统一距离递推","source_units":[{"raw_tex":"Combining the above with (\\ref{eq:1114}), we conclude that\n\\begin{equation}\\label{77}\n\td(p_{k+1} , p^{*}) \\leq \\kappa_{1}(1-\\alpha_{k}) d(p_{k} , p^{*}) \n\t+ \\kappa_{2} d^{2}(p_{k} , p^{*})\n\t+ \\kappa_{3} \\mu_{k}\n\\end{equation}\nfor some positive constants $ \\kappa_{1},\\kappa_{2},\\kappa_{3}$.\n","start":63251,"end":63542}]},{"id":"S061","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:LipFbigoh"],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"残差的距离阶上界","source_units":[{"raw_tex":"On the other hand, by Lemma \\ref{lem:dist_zero}, we have\n\\begin{equation}\\label{eq:LipFbigoh}\n\t\\left\\|F(p_{k})\\right\\|=O(d(p_{k}, p^{*})).\n\\end{equation}\n","start":63542,"end":63696}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":["eq:superlinear"],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"超线性情形的比值估计","source_units":[{"raw_tex":"In what follows, we prove assertions (1) and (2).\n\n\\textbf{(1)} \nSuppose that $\\alpha_{k} \\to 1$ and $\\mu_{k}=o(\\|F(p_{k})\\|)$, which together with (\\ref{eq:LipFbigoh}) imply $\\mu_{k}=o(d(p_{k}, p^{*}))$.\nBy (\\ref{77}), we have\n\\begin{equation}\\label{eq:superlinear}\n\t\\frac{d(p_{k+1} , p^{*})}{d(p_{k} , p^{*})}\\leq\n\t\\kappa_{1} (1-\\alpha_{k})\n\t+\\kappa_{2} d(p_{k} , p^{*})\n\t+\\kappa_{3}\\frac{\\mu_{k}}{d(p_{k} , p^{*})}.\n\\end{equation}\nWe can take $\\delta$ sufficiently small and $ k $ sufficiently large, if necessary, to conclude that\n$\nd(p_{k+1} , p^{*})<\\frac{1}{2}d(p_{k} , p^{*}) <\\delta.\n$\nThus, $p_{k+1} \\in B_{\\delta}(p^{*})$, the open ball of radius $\\delta$ centered at $p^{*}$ on $\\mathcal{M}$. \nBy induction, it is easy to show that the sequence $\\left\\{p_{k}\\right\\}$ is well defined and converges to $p^{*}$. \nTaking the limit of both sides of (\\ref{eq:superlinear}) proves superlinear convergence.\n","start":63696,"end":64608}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":["eq:bigoh"],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"二次情形与邻域归纳","source_units":[{"raw_tex":"\n\n\\textbf{(2)} \nAgain, we start from (\\ref{77}) and rewrite it as:\n\\begin{align}\\label{eq:bigoh}\n\td(p_{k+1} , p^{*}) = (1-\\alpha_{k}) O(d(p_{k} , p^{*}) )\n\t+O(d^{2}(p_{k} , p^{*}) )\n\t+O(\\mu_{k}).\n\\end{align}\nSuppose that $1-\\alpha_{k}=O(\\left\\|F(p_{k})\\right\\|)$\nand $\\mu_{k}=O(\\left\\|F(p_{k})\\right\\|^{2})$.\nUsing (\\ref{eq:LipFbigoh}), the above reduces to\n$\nd(p_{k+1} , p^{*}) = O(d^{2}(p_{k} , p^{*})).\n$\nThis implies that there exists a constant $\\nu$ such that $d(p_{k+1} , p^{*}) \\leq \\nu d^{2}(p_{k} , p^{*})$, and hence,\n$\nd(p_{k+1} , p^{*}) \\leq \\nu d^{2}(p_{k} , p^{*}) \\leq \\nu \\delta^{2}<\\delta,\n$\nif $\\delta$ is sufficiently small.\nAgain, by induction, $\\left\\{p_{k}\\right\\}$ converges to $p^{*}$ quadratically.\n\\end{proof}","start":64608,"end":65344}]},{"id":"S064","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S064"],"declared_external":false,"source_only":true,"title":"步长引理的外部委托证明","source_units":[{"raw_tex":"\\begin{proof} \nNotice that the fourth line of (\\ref{eq:FullNewtonEquation}) yields\n$$\nS_{k}^{-1} \\Delta s_{k}+Z_{k}^{-1} \\Delta z_{k}=\\mu_{k}(S_{k} Z_{k})^{-1} e-e,\n$$\nwhich is exactly the same as in the usual interior point method in the Euclidean setting. \nThus, the proof entails directly applying \\cite[Lemma 3 and 4]{yamashita1996superlinear} for the Euclidean case to the Riemannian case.\n\\end{proof}\n","start":66212,"end":66619}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S065"],"declared_external":false,"source_only":true,"title":"互补方程的相对增量形式","source_units":[{"raw_tex":"\\begin{proof} \nNotice that the fourth line of (\\ref{eq:FullNewtonEquation}) yields\n$$\nS_{k}^{-1} \\Delta s_{k}+Z_{k}^{-1} \\Delta z_{k}=\\mu_{k}(S_{k} Z_{k})^{-1} e-e,\n$$\nwhich is exactly the same as in the usual interior point method in the Euclidean setting. \n","start":66212,"end":66471}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S066"],"declared_external":false,"source_only":true,"title":"直接援引欧氏步长引理","source_units":[{"raw_tex":"Thus, the proof entails directly applying \\cite[Lemma 3 and 4]{yamashita1996superlinear} for the Euclidean case to the Riemannian case.\n\\end{proof}\n","start":66471,"end":66619}]},{"id":"S067","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S067"],"declared_external":true,"declaration_text":["原先决条件清单：Yamashita and Yabe (1996), Lemmas 3 and 4；原文 1063 行直接引用。"],"source_only":false,"title":"欧氏内点法步长引理","source_units":[]},{"id":"S068","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S068"],"declared_external":true,"declaration_text":["原先决条件清单：Inverse function theorem；Zhu and Sato (2020), Theorem 2。"],"source_only":false,"title":"逆函数与全回缩邻域定理","source_units":[]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S069"],"declared_external":true,"declaration_text":["原先决条件清单中的 standard Riemannian product-connection properties；原文 300 行另引 Boumal 的 Exercises 5.4 and 5.13。hat e 范数恒定这一省略背景由标注者显式化。"],"source_only":false,"title":"黎曼乘积结构的标准性质","source_units":[]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S070"],"declared_external":true,"declaration_text":["原先决条件清单中的 standard finite-dimensional linear algebra；具体平凡核判据为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"有限维平凡核判据","source_units":[]},{"id":"S071","semantic_type":"theorem","kind":"theorem","labels":["thm:LocalConvergence"],"member_ids":["S071"],"declared_external":false,"source_only":false,"title":"RIPM 原型的局部收敛","source_units":[{"raw_tex":"\\begin{theorem}[Local convergence of prototype Algorithm \\ref{algo:prototype_algo}]\\label{thm:LocalConvergence}\n\tConsider the Algorithm \\ref{algo:prototype_algo} for solving the problem (\\ref{RCOP}).  \n\tLet \\ref{B1}-\\ref{B5} hold at some $w^{*}$.\n\tChoose parameters $\\mu_k, \\gamma_k$ as follows;\n\tthen there exists a constant $\\delta>0$ such that, for all\n\t$\n\tw_{0} \\in\\mathcal{N}\n\t$\n\twith $d(w_{0} , w^{*})<\\delta,$\n\tthe sequence $\\left\\{w_{k}\\right\\}$ is well defined. Furthermore,\n\t\\begin{enumerate}[(1)]\n\t\t\\item if we choose $\\mu_{k}=o(\\left\\|F(w_{k})\\right\\|)$ and $\\gamma_{k} \\to 1$, then $w_k \\to w^*$ superlinearly;\t\n\t\t\\item if we choose $\\mu_{k}=O(\\left\\|F(w_{k})\\right\\|^{2})$\n\t\tand $1-\\gamma_{k}=O(\\left\\|F(w_{k})\\right\\|)$, then $w_k \\to w^*$ quadratically.\n\t\\end{enumerate}\n\\end{theorem}","start":66819,"end":67619}]},{"id":"S072","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S072"],"declared_external":false,"source_only":false,"title":"局部收敛定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nWe only prove (2) because (1) can be proven in the same way.\nLet $w_{k} $ be such that  $d(w_{k} , w^{*})<\\delta$ for sufficiently small $\\delta$.\nFrom Proposition \\ref{prop:non}, \\ref{B1}-\\ref{B5} shows that the KKT vector field $F$ satisfies \\ref{A1}-\\ref{A3}; thus, the discussion in the proof of Proposition \\ref{prop:PD_Newton} applies to KKT vector field $F$ as well, simply by replacing the symbol $p$ with $w$.\nSince we choose $\\mu_{k}=O(\\|F(w_{k})\\|^{2})$, and $\\|F(w_{k})\\|=O(d(w_{k}, w^{*}))$ by (\\ref{eq:LipFbigoh}), we obtain\n$\n\\mu_{k}=O(d^{2}(w_{k}, w^{*})).\n$\nThus,\n\\begin{align*}\n\t \\left\\|\\Delta w_{k}\\right\\|  & =\n\t\\left\\|\n\t\\nabla F(w_{k})^{-1} (-F(w_{k})+\\mu_{k} \\hat{e})\n\t\\right\\| \\text{ (by (\\ref{eq:prototype}))}\\\\\n\t& \\leq \n\t\\Xi\n\t(\\left\\|F(w_{k})\\right\\|+\\mu_{k}\\|\\hat{e}\\|) \\text{ (by Lemma \\ref{lem:nonsingular})}\\\\\n\t& \\leq  O(\\|F(w_k)\\|)+O(\\mu_{k}) \\\\\n\t& =O(d(w_{k} , w^{*}))+O(d^{2}(w_{k} , w^{*}) )\n\t=O(d(w_{k} , w^{*})).\n\\end{align*}\nSince $\\delta$ is sufficiently small, from the above inequalities, the condition of Lemma \\ref{lem:gamma_alpha} is satisfied, i.e.,\n$\n\\left\\|\\Delta w_{k}\\right\\| \\leq \\hat{\\gamma} / \\Pi \\leq \\gamma_k / \\Pi\n$\nfor a constant $\\hat{\\gamma} \\in (0,1).$\nHence, by $1-\\gamma_k=O(\\left\\|F\\left(w_k\\right)\\right\\|)$, one has\n$\n1-\\alpha_{k} \\leq\n(1-\\gamma_{k})+\\Pi\\left\\|\\Delta w_{k}\\right\\| \n= (1-\\gamma_{k})+O(d(w_{k} , w^{*}))\n= O(d(w_{k} , w^{*})).\n$\nFinally, from (\\ref{eq:bigoh}),\nwe have\n$\nd(w_{k+1} , w^{*}) = (1-\\alpha_{k}) O(d(w_{k} , w^{*}) )\n+O(d^{2}(w_{k} , w^{*}) )\n+O(\\mu_{k}) \n=O(d^{2}(w_{k} , w^{*})).\n$\nThis completes the proof.\n\\end{proof}","start":67620,"end":69244}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S073"],"declared_external":false,"source_only":false,"title":"声明省略超线性情形","source_units":[{"raw_tex":"\\begin{proof}\nWe only prove (2) because (1) can be proven in the same way.\n","start":67620,"end":67695}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S074"],"declared_external":false,"source_only":false,"title":"在 KKT 场上实例化 Newton 假设","source_units":[{"raw_tex":"Let $w_{k} $ be such that  $d(w_{k} , w^{*})<\\delta$ for sufficiently small $\\delta$.\nFrom Proposition \\ref{prop:non}, \\ref{B1}-\\ref{B5} shows that the KKT vector field $F$ satisfies \\ref{A1}-\\ref{A3}; thus, the discussion in the proof of Proposition \\ref{prop:PD_Newton} applies to KKT vector field $F$ as well, simply by replacing the symbol $p$ with $w$.\n","start":67695,"end":68053}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S075"],"declared_external":false,"source_only":false,"title":"障碍参数具有距离平方阶","source_units":[{"raw_tex":"Since we choose $\\mu_{k}=O(\\|F(w_{k})\\|^{2})$, and $\\|F(w_{k})\\|=O(d(w_{k}, w^{*}))$ by (\\ref{eq:LipFbigoh}), we obtain\n$\n\\mu_{k}=O(d^{2}(w_{k}, w^{*})).\n$\n","start":68053,"end":68209}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S076"],"declared_external":false,"source_only":false,"title":"Newton 方向具有距离一阶界","source_units":[{"raw_tex":"Thus,\n\\begin{align*}\n\t \\left\\|\\Delta w_{k}\\right\\|  & =\n\t\\left\\|\n\t\\nabla F(w_{k})^{-1} (-F(w_{k})+\\mu_{k} \\hat{e})\n\t\\right\\| \\text{ (by (\\ref{eq:prototype}))}\\\\\n\t& \\leq \n\t\\Xi\n\t(\\left\\|F(w_{k})\\right\\|+\\mu_{k}\\|\\hat{e}\\|) \\text{ (by Lemma \\ref{lem:nonsingular})}\\\\\n\t& \\leq  O(\\|F(w_k)\\|)+O(\\mu_{k}) \\\\\n\t& =O(d(w_{k} , w^{*}))+O(d^{2}(w_{k} , w^{*}) )\n\t=O(d(w_{k} , w^{*})).\n\\end{align*}\n","start":68209,"end":68595}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S077"],"declared_external":false,"source_only":false,"title":"验证步长引理的小方向条件","source_units":[{"raw_tex":"Since $\\delta$ is sufficiently small, from the above inequalities, the condition of Lemma \\ref{lem:gamma_alpha} is satisfied, i.e.,\n$\n\\left\\|\\Delta w_{k}\\right\\| \\leq \\hat{\\gamma} / \\Pi \\leq \\gamma_k / \\Pi\n$\nfor a constant $\\hat{\\gamma} \\in (0,1).$\n","start":68595,"end":68844}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S078"],"declared_external":false,"source_only":false,"title":"中心参数转化为阻尼误差","source_units":[{"raw_tex":"Hence, by $1-\\gamma_k=O(\\left\\|F\\left(w_k\\right)\\right\\|)$, one has\n$\n1-\\alpha_{k} \\leq\n(1-\\gamma_{k})+\\Pi\\left\\|\\Delta w_{k}\\right\\| \n= (1-\\gamma_{k})+O(d(w_{k} , w^{*}))\n= O(d(w_{k} , w^{*})).\n$\n","start":68844,"end":69041}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S079"],"declared_external":false,"source_only":false,"title":"代入通用递推得到二次误差","source_units":[{"raw_tex":"Finally, from (\\ref{eq:bigoh}),\nwe have\n$\nd(w_{k+1} , w^{*}) = (1-\\alpha_{k}) O(d(w_{k} , w^{*}) )\n+O(d^{2}(w_{k} , w^{*}) )\n+O(\\mu_{k}) \n=O(d^{2}(w_{k} , w^{*})).\n$\nThis completes the proof.\n\\end{proof}","start":69041,"end":69244}]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S019":"S019","S020":"S020","S021":"S021","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S032":"S032","S033":"S033","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076","S077":"S077","S078":"S078","S079":"S079"}}}}
{"record_id":"2401.02635v3","paper":{"title":"Bregman Proximal Linearized ADMM for Minimizing Separable Sums Coupled by a Difference of Functions","authors":["Pham, Tan Nhat","Dao, Minh N.","Eberhard, Andrew","Sultanova, Nargiz"],"arxiv_id":"2401.02635","version":"v3","journal":"Journal of Optimization Theory and Applications","abstract_url":"https://arxiv.org/abs/2401.02635v3","pdf_url":"https://arxiv.org/pdf/2401.02635v3","doi":"10.1007/s10957-024-02539-7"},"context":{"raw_tex":"Throughout this paper, the set of nonnegative integers is denoted by $\\mathbb{N}$, the set of real numbers by $\\mathbb{R}$, the set of nonnegative real numbers by $\\mathbb{R}_+$, and the set of the positive real numbers by $\\mathbb{R}_{++}$. We use $\\mathbb{R}^d$ to denote the $d$-dimensional Euclidean space with inner product $\\langle \\cdot, \\cdot \\rangle$ and Euclidean norm $\\|\\cdot\\|$. For a given matrix $M$, $\\Ima(M)$ denotes\nits image.\n\nWe consider the linearly constrained composite optimization problem\n\\begin{align}\\label{eq:prob}\n\\min_{x_i \\in \\mathbb{R}^{d_i}, y\\in \\mathbb{R}^q} ~F(\\mathbf{x},y) :=\\sum_{i=1}^m f_i(x_i)+H(y)+P(\\mathbf{x})-G(\\mathbf{x}) \\quad\\text{s.t.~~}  \\sum_{i=1}^{m}A_ix_i + By=b,\n\\end{align}\nwhere $\\mathbf{x}=(x_1,\\dots,x_m)$; for each $i\\in \\{1,\\dots,m\\}$, $f_i: \\mathbb{R}^{d_i} \\to (-\\infty,+\\infty]$ is a proper lower semicontinuous function; $H:\\mathbb{R}^q \\to \\mathbb{R}$ is a differentiable (possibly\nnonconvex) function whose gradient is Lipschitz continuous; $P:\\mathbb{R}^{d_1}\\times \\dots \\times \\mathbb{R}^{d_m} \\to \\mathbb{R}$ is another differentiable (possibly nonconvex) function whose gradient is Lipschitz continuous; $G:\\mathbb{R}^{d_1}\\times \\dots \\times \\mathbb{R}^{d_m} \\to \\mathbb{R}$ is a continuous (possibly nonsmooth) weakly convex function, $A_i \\in \\mathbb{R}^{p\\times d_i}$, $B\\in \\mathbb{R}^{p\\times q}$, and $b \\in \\mathbb{R}^p$. Problem \\eqref{eq:prob} covers two important models in the literature. Firstly, when $G \\equiv 0$, problem \\eqref{eq:prob} reduces to the form\n\\begin{align}\\label{problem:WTY}\n\\min_{x_i \\in \\mathbb{R}^{d_i}, y\\in \\mathbb{R}^q} ~\\sum_{i=1}^m f_i(x_i)+H(y)+P(\\mathbf{x}) \\quad\\text{s.t.~~}  \\sum_{i=1}^{m}A_ix_i + By=b,\n\\end{align}\nwhich is studied in \\cite{Wang2018}. This class of problems, together with its special cases considered in \\cite{Yang2017,Guo2016}, has a broad range of useful applications in image processing, matrix decomposition, statistical learning, and others \\cite{Wang2018,ADMMsurvey}. In \\cite{Wang2018}, the author studied the multi-block alternating direction method of multipliers (ADMM) in the nonconvex nonsmooth setting, and they introduced new assumptions such as restricted prox-regularity and Lipschitz subminimization path to establish the global convergence. A special case of \\eqref{problem:WTY} in which $P\\equiv0$ was considered in \\cite{Yashtini2021}, and proximal terms and dual stepsize were incorporated into the ADMM to solve the problem. An inertial proximal splitting method was also developed in \\cite{XWang2023} to revisit that problem. Other simpler special cases of \\eqref{problem:WTY} can be addressed by variants of the three-block ADMM  \\cite{Yang2017,Lin2017,RPCA}, or two-block ADMM \\cite{Guo2016,2blockBregADMM,Li_2015}, and their convergence are well studied in the literature. Secondly, when $P \\equiv 0$, $m =1$, $x_1 =x \\in \\mathbb{R}^d$, and $A_1 =A \\in \\mathbb{R}^{p\\times d}$, problem \\eqref{eq:prob} reduces to\n\\begin{align}\\label{DCADMM}\n\\min_{x \\in \\mathbb{R}^d,y \\in \\mathbb{R}^q}~ f(x)-G(x)+H(y) \\quad \\text{s.t.~} Ax+By=c.\n\\end{align}\nThis is known as the linearly constrained difference-of-convex problem \\cite{Sun2017,Tu2019}, in which $f$, $H$, and $G$ are all assumed to be convex. Problem \\eqref{DCADMM} arises from the use of the difference-of-convex regularization terms, which has shown very promising performance \\cite{Gasso2009,Yin2015}. One practical engineering problem that can be solved via problem \\eqref{DCADMM} is the \\emph{total variation image restoration problem} \\cite{Osher2005,Sun2017,Tu2019}, which is one of the most fundamental issues in imaging science. The authors of \\cite{Sun2017} developed a variant of the ADMM which incorporates the Bregman distance in updating steps for solving problem \\eqref{DCADMM} in the case when $f$, $G$, and $H$ are convex functions, with $H$ being differentiable with a Lipschitz continuous gradient. A hybrid Bregman ADMM was then proposed in \\cite{Tu2019} to revisit problem \\eqref{DCADMM} with the same assumptions on $f$, $G$, and $H$. This version introduces an additional step to evaluate the convex conjugate of $G$, allowing the flexibility in choosing the subgradient step or proximal step to update the solution by choosing a control parameter to either activate or deactivate the additional step. By this strategy, this algorithm can include the one in \\cite{Sun2017} as a special case. However, further research on more general form of problem \\eqref{DCADMM} is still not available in the literature, to the best of the authors' knowledge. \n\n\nLet $f\\colon \\mathbb{R}^d\\to \\left(-\\infty,+\\infty\\right]$. The \\emph{domain} of $f$ is $\\dom f :=\\{x\\in \\mathbb{R}^d: f(x) <+\\infty\\}$ and the \\emph{epigraph} of $f$ is $\\epi f := \\{(x,\\xi)\\in \\mathbb{R}^d\\times \\mathbb{R}: f(x)\\leq \\xi\\}$. The function $f$ is said to be \\emph{proper} if $\\dom f \\neq \\varnothing$ and it never takes the value $-\\infty$, \\emph{lower semicontinuous} if its epigraph is closed, and \\emph{convex} if its epigraph is convex. The function $f$ is said to be $\\alpha$-\\emph{strongly convex} ($\\alpha\\in \\mathbb{R}_+$) if $f-\\frac{\\alpha}{2}\\|\\cdot\\|^2$ is convex, and $\\beta$-\\emph{weakly convex} ($\\beta\\in \\mathbb{R}_+$) if $f+\\frac{\\beta}{2}\\|\\cdot\\|^2$ is convex. We say that $f$ is \\emph{coercive} if $f(x)\\to +\\infty$ as $\\|x\\|\\to +\\infty$.\n\nLet $x\\in \\mathbb{R}^d$ with $|f(x)| <+\\infty$. The \\emph{regular} (or \\emph{Fr\\'echet}) \\emph{subdifferential} of $f$ at $x$ is defined by\n\\begin{equation*}\n\\widehat{\\partial} f(x) :=\\left\\{x^*\\in \\mathbb{R}^d:\\; \\liminf_{y\\to x}\\frac{f(y)-f(x)-\\langle x^*,y-x\\rangle}{\\|y-x\\|}\\geq 0\\right\\}\n\\end{equation*}\nand the \\emph{limiting} (or \\emph{Mordukhovich}) \\emph{subdifferential} of $f$ at $x$ is defined by\n\\begin{equation*}\n\\partial_L f(x) :=\\left\\{x^*\\in \\mathbb{R}^d:\\; \\exists x_n\\stackrel{f}{\\to}x,\\; x_n^*\\to x^* \\text{~~with~~} x_n^*\\in \\widehat{\\partial} f(x_n)\\right\\},\n\\end{equation*}\nwhere the notation $y\\stackrel{f}{\\to}x$ means $y\\to x$ with $f(y)\\to f(x)$. In the case where $\\lvert f(x) \\rvert =+\\infty$, both regular subdifferential and limiting subdifferential of $f$ at $x$ are defined to be the empty set. The \\emph{domain} of $\\partial_L f$ is given by $\\dom \\partial_L f :=\\{x\\in \\mathbb{R}^d: \\partial_L f(x) \\neq \\varnothing\\}$. \nWe now collect some important properties of the limiting subdifferential.\n\n\nWe recall the concept of the well-known Bregman distance in a real Hilbert space $\\mathcal{H}$, which was introduced in \\cite{Bregman}. Given a differentiable convex function $\\phi: \\mathcal{H} \\to \\mathbb{R}$, the \\emph{Bregman distance} associated with $\\phi$ is defined as \n\\begin{align*}\n\\forall u, v\\in \\mathcal{H},\\quad D_{\\phi}(u,v)=\\phi(u)-\\phi(v)-\\langle \\nabla\\phi (v),u-v\\rangle,\n\\end{align*}\nWhen $\\phi(x)=\\|x\\|^2$, then $D_{\\phi}(u,v)=\\|u-v\\|^2$. When $\\phi(x)=x^\\top M x$, where $M$ is a positive semidefinite matrix, then $D_{\\phi}(u,v) = \\|u-v\\|^2_M :=(u-v)^\\top M (u-v)$. When $\\mathcal{H}=\\mathbb{R}^{p\\times d}$ and $\\phi(X)=\\|X\\|_F^2$, then $D_{\\phi}(U,V) =\\|U-V\\|_F^2$ \\cite{Dhillon2008}. Some useful properties of the Bregman distance are listed in the following proposition.\n\n\nWe first fix some notations which are used throughout the paper from now on. Let $\\mathbf{A}=[A_1|A_2|\\dots|A_m]$ and let $\\mathbf{I}_{q\\times q}$ be the $q\\times q$ identity matrix. Then $\\mathbf{Ax}=\\sum_{i=1}^{m}A_ix_i$. We also denote by $\\partial_L^{x}$ the limiting subdifferential with respect to the $x$-variable, and by $\\nabla_i f$ the $i^{th}$ block of the gradient vector $\\nabla f$. Given a matrix $M$, $\\lambda_{\\min}(M)$ denotes its smallest eigenvalue while $\\lambda_{\\max}(M)$ denotes its largest eigenvalue. The following assumptions are used in our convergence analysis\n\\begin{assumption}[Standing assumptions]\n\\label{a:standing}\n~\n\\begin{enumerate}\n\\item \n$\\nabla H$ is $\\ell_H$-Lipschitz continuous, $\\nabla P$ is $\\ell_P$-Lipschitz continuous, and $G$ is $\\beta$-weakly convex.\n\\item\\label{a:standing_image} \n$\\Ima(\\mathbf{A}) \\bigcup \\{b \\}\\subseteq \\Ima(B)$.\n\\item\\label{a:standing_rank} \n $\\lambda :=\\lambda_{\\min}(B^\\top B) >0$ (equivalently, $B$ has full column rank).\n\\end{enumerate}\n\\end{assumption}\n\n\nRecall that the augmented Lagrangian of problem \\eqref{eq:prob} is given by\n\\begin{align*}\n\\mathcal{L}_\\rho(\\mathbf{x},y,z)=\\sum_{i=1}^m f_i(x_i)+H(y)+P(\\mathbf{x})-G(\\mathbf{x}) +  \\langle z, \\mathbf{Ax}+By-b \\rangle +\\frac{\\rho}{2}\\|\\mathbf{Ax}+By-b\\|^2,\n\\end{align*}\nwhere $z\\in \\mathbb{R}^p$ and $\\rho\\in \\mathbb{R}_{++}$. Now, we present our splitting algorithm with guaranteed global convergence to a stationary point $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ of $\\mathcal{L}_\\rho$, i.e., $0 \\in \\partial_L \\mathcal{L}_\\rho(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$, or equivalently,\n\\begin{equation}\\label{eq:optimality_conditions}\n\\begin{aligned}\n&0 \\in \\partial_L\\left(\\sum_{i=1}^{m}f_i -G\\right)(\\overline{\\mathbf{x}}) +\\nabla P(\\overline{\\mathbf{x}}) +\\mathbf{A}^\\top \\overline{z},\\\\\n&0 =\\nabla H(\\overline{y}) +B^\\top \\overline{z}, \\\\\n&0 =\\mathbf{A\\overline{x}} +B\\overline{y} -b.\n\\end{aligned}\n\\end{equation}\n\n\n\\begin{tcolorbox}[\n\tleft=0pt,right=0pt,top=0pt,bottom=0pt,\n\tcolback=blue!10!white, colframe=blue!50!white,\n  \tboxrule=0.2pt,\n  \tbreakable]\n\\begin{algorithm}[Bregman proximal linearized ADMM (BPL-ADMM)]\n\\label{algo:BPLA}\n\\step{}\\label{step:A2_1}\nSet $n=0$. Choose $\\mathbf{x}_0 =(x_{1,0}, \\dots, x_{m,0}) \\in \\mathbb{R}^{d_1}\\times \\dots \\times \\mathbb{R}^{d_m}$, $y_0 \\in \\mathbb{R}^q$, and $z_0 \\in \\mathbb{R}^p$. Let $\\phi_i: \\mathbb{R}^{d_i} \\to \\mathbb{R}$, $i \\in \\{1,\\dots,m\\}$, be differentiable $\\alpha$-strongly convex functions with $\\alpha \\in (0, +\\infty)$, and let $\\psi: \\mathbb{R}^q \\to \\mathbb{R}$ be a differentiable convex function with $\\ell_{\\psi}$-Lipschitz continuous gradient. Let \n\\begin{align*}\n\\mu \\in \\left(\\frac{\\ell_P+\\beta}{\\alpha}, +\\infty\\right),\\;\\; \\nu \\in \\left[0, +\\infty\\right), \\text{~~and~~} \\rho \\in \\left(\\frac{\\ell_H +\\sqrt{\\ell_H^2 +8(\\ell_H +2\\nu\\ell_{\\psi})^2}}{2\\lambda}, +\\infty\\right).    \n\\end{align*}\n\n\\step{}\\label{step:A2_2}\nCalculate $\\nabla P(\\mathbf{x}_n)=(\\nabla_1 P(\\mathbf{x}_n),\\dots,\\nabla_m P(\\mathbf{x}_n))$ and $g_{n}=(g_{1,n},\\dots,g_{m,n}) \\in \\partial_L G(\\mathbf{x}_{n})$. Let $\\mu_n \\in [\\mu, +\\infty)$ and $\\nu_n \\in [0, \\nu]$. For each $i \\in \\{1,\\dots,m\\}$, define $\\mathbf{u}_{i,n+1}: \\mathbb{R}^{d_i}\\to \\mathbb{R}^{d_1}\\times \\dots \\times \\mathbb{R}^{d_m}$ by $\\mathbf{u}_{i,n+1}(x_i) =(x_{1,n+1}, \\dots, x_{i-1,n+1}, x_i, x_{i+1,n}, \\dots, x_{m,n})$, and find\n\\begin{align*}\nx_{i,n+1} &\\in  \\argmin_{x_i \\in \\mathbb{R}^{d_i}} \\left(f_i(x_i) +\\langle \\nabla_i P(\\mathbf{x}_n) -g_{i,n}, x_i \\rangle + \\langle z_n, A_ix_i\\rangle+\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,n+1}(x_i)+By_n-b\\|^2 +\\mu_n D_{\\phi_i}(x_i,x_{i,n})\\right),\\\\\ny_{n+1} &\\in  \\argmin_{y \\in \\mathbb{R}^q} \\left(H(y) + \\langle z_n, By\\rangle+\\frac{\\rho}{2}\\|\\mathbf{Ax}_{n+1} + By-b \\|^2 +\\nu_n D_{\\psi}(y,y_n) \\right),\\\\\nz_{n+1} &=z_n+\\rho(\\mathbf{Ax}_{n+1}+By_{n+1}-b),\n\\end{align*}\nwhere $\\mathbf{x}_{n+1} =(x_{1,n+1}, \\dots, x_{m,n+1})$.\n\\step{}\nIf a termination criterion is not met, set $n =n+1$ and go to Step~\\ref{step:A2_2}.\n\\end{algorithm}\n\\end{tcolorbox}\n\n\n\\begin{lemma}\n\\label{l:norm}\nLet $L$ be a symmetric matrix in $\\mathbb{R}^{d\\times d}$ and let $M$ be a matrix in $\\mathbb{R}^{p\\times d}$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{l:norm_Mx}\nFor all $x\\in \\mathbb{R}^d$, $\\sqrt{\\lambda_{\\min}(M^\\top M)}\\|x\\| \\leq \\|Mx\\|$.\n\\item\\label{l:norm_M'x}\nFor all $x\\in \\mathbb{R}^d$, $\\gamma(x^\\top Lx) \\leq \\|Lx\\|^2$, where $\\gamma =\\min\\{|\\lambda|: \\lambda \\text{~is a nonzero eigenvalue of~} L\\}$. Consequently, for all $z\\in \\Ima(M)$, $\\sqrt{\\lambda_*(M^\\top M)}\\|z\\| \\leq \\|M^\\top z\\|$, where $\\lambda_*(M^\\top M)$ is the smallest strictly positive eigenvalue of $M^\\top M$. \n\\end{enumerate}\n\\end{lemma}\n\nWe now establish the convergence of the full sequence generated by Algorithm~\\ref{algo:BPLA}. Recall that a proper lower semicontinuous function $\\mathcal{F}\\colon \\mathcal{H}\\to \\left(-\\infty, +\\infty\\right]$ satisfies the \\emph{Kurdyka--Łojasiewicz (KL) property} \\cite{Kurdyka1998,Loja63} at $\\overline{x} \\in \\dom \\partial_L \\mathcal{F}$ if there are $\\eta \\in (0, +\\infty]$, a neighborhood $V$ of $\\overline{x}$, and a continuous concave function $\\varphi: \\left[0, \\eta\\right) \\to \\mathbb{R}_+$ such that $\\varphi$ is continuously differentiable with $\\varphi' > 0$ on $(0, \\eta)$, $\\varphi(0) = 0$, and, for all $x \\in V$ with $\n\\mathcal{F}(\\overline{x}) < \\mathcal{F}(x) < \\mathcal{F}(\\overline{x}) + \\eta$, \n\\begin{equation*}\n\\varphi'(\\mathcal{F}(x) -\\mathcal{F}(\\overline{x})) \\dist(0, \\partial_L \\mathcal{F}(x)) \\geq 1.   \n\\end{equation*}\nWe say that $\\mathcal{F}$ is a \\emph{KL function} if it satisfies the KL property at any point in $\\dom \\partial_L \\mathcal{F}$. If $\\mathcal{F}$ satisfies the KL property at $\\overline{x} \\in \\dom \\partial_L \\mathcal{F}$, in which the corresponding function $\\varphi$ can be chosen as $\\varphi(t) = c t ^{1 - \\lambda}$ for some $c \\in \\mathbb{R}_{++}$ and $\\lambda \\in [0, 1)$, then $\\mathcal{F}$ is said to satisfy the \\emph{KL property at $\\overline{x}$ with exponent $\\lambda$}. The function $\\mathcal{F}$ is called a \\emph{KL function with exponent $\\lambda$} if it is a KL function and has the same exponent $\\lambda$ at any $x \\in \\dom \\partial_L \\mathcal{F}$. Now we will present the proof of the full sequential convergence in the following theorem.\n","units":[{"raw_tex":"Throughout this paper, the set of nonnegative integers is denoted by $\\mathbb{N}$, the set of real numbers by $\\mathbb{R}$, the set of nonnegative real numbers by $\\mathbb{R}_+$, and the set of the positive real numbers by $\\mathbb{R}_{++}$. We use $\\mathbb{R}^d$ to denote the $d$-dimensional Euclidean space with inner product $\\langle \\cdot, \\cdot \\rangle$ and Euclidean norm $\\|\\cdot\\|$. For a given matrix $M$, $\\Ima(M)$ denotes\nits image.\n\nWe consider the linearly constrained composite optimization problem\n\\begin{align}\\label{eq:prob}\n\\min_{x_i \\in \\mathbb{R}^{d_i}, y\\in \\mathbb{R}^q} ~F(\\mathbf{x},y) :=\\sum_{i=1}^m f_i(x_i)+H(y)+P(\\mathbf{x})-G(\\mathbf{x}) \\quad\\text{s.t.~~}  \\sum_{i=1}^{m}A_ix_i + By=b,\n\\end{align}\nwhere $\\mathbf{x}=(x_1,\\dots,x_m)$; for each $i\\in \\{1,\\dots,m\\}$, $f_i: \\mathbb{R}^{d_i} \\to (-\\infty,+\\infty]$ is a proper lower semicontinuous function; $H:\\mathbb{R}^q \\to \\mathbb{R}$ is a differentiable (possibly\nnonconvex) function whose gradient is Lipschitz continuous; $P:\\mathbb{R}^{d_1}\\times \\dots \\times \\mathbb{R}^{d_m} \\to \\mathbb{R}$ is another differentiable (possibly nonconvex) function whose gradient is Lipschitz continuous; $G:\\mathbb{R}^{d_1}\\times \\dots \\times \\mathbb{R}^{d_m} \\to \\mathbb{R}$ is a continuous (possibly nonsmooth) weakly convex function, $A_i \\in \\mathbb{R}^{p\\times d_i}$, $B\\in \\mathbb{R}^{p\\times q}$, and $b \\in \\mathbb{R}^p$. Problem \\eqref{eq:prob} covers two important models in the literature. Firstly, when $G \\equiv 0$, problem \\eqref{eq:prob} reduces to the form\n\\begin{align}\\label{problem:WTY}\n\\min_{x_i \\in \\mathbb{R}^{d_i}, y\\in \\mathbb{R}^q} ~\\sum_{i=1}^m f_i(x_i)+H(y)+P(\\mathbf{x}) \\quad\\text{s.t.~~}  \\sum_{i=1}^{m}A_ix_i + By=b,\n\\end{align}\nwhich is studied in \\cite{Wang2018}. This class of problems, together with its special cases considered in \\cite{Yang2017,Guo2016}, has a broad range of useful applications in image processing, matrix decomposition, statistical learning, and others \\cite{Wang2018,ADMMsurvey}. In \\cite{Wang2018}, the author studied the multi-block alternating direction method of multipliers (ADMM) in the nonconvex nonsmooth setting, and they introduced new assumptions such as restricted prox-regularity and Lipschitz subminimization path to establish the global convergence. A special case of \\eqref{problem:WTY} in which $P\\equiv0$ was considered in \\cite{Yashtini2021}, and proximal terms and dual stepsize were incorporated into the ADMM to solve the problem. An inertial proximal splitting method was also developed in \\cite{XWang2023} to revisit that problem. Other simpler special cases of \\eqref{problem:WTY} can be addressed by variants of the three-block ADMM  \\cite{Yang2017,Lin2017,RPCA}, or two-block ADMM \\cite{Guo2016,2blockBregADMM,Li_2015}, and their convergence are well studied in the literature. Secondly, when $P \\equiv 0$, $m =1$, $x_1 =x \\in \\mathbb{R}^d$, and $A_1 =A \\in \\mathbb{R}^{p\\times d}$, problem \\eqref{eq:prob} reduces to\n\\begin{align}\\label{DCADMM}\n\\min_{x \\in \\mathbb{R}^d,y \\in \\mathbb{R}^q}~ f(x)-G(x)+H(y) \\quad \\text{s.t.~} Ax+By=c.\n\\end{align}\nThis is known as the linearly constrained difference-of-convex problem \\cite{Sun2017,Tu2019}, in which $f$, $H$, and $G$ are all assumed to be convex. Problem \\eqref{DCADMM} arises from the use of the difference-of-convex regularization terms, which has shown very promising performance \\cite{Gasso2009,Yin2015}. One practical engineering problem that can be solved via problem \\eqref{DCADMM} is the \\emph{total variation image restoration problem} \\cite{Osher2005,Sun2017,Tu2019}, which is one of the most fundamental issues in imaging science. The authors of \\cite{Sun2017} developed a variant of the ADMM which incorporates the Bregman distance in updating steps for solving problem \\eqref{DCADMM} in the case when $f$, $G$, and $H$ are convex functions, with $H$ being differentiable with a Lipschitz continuous gradient. A hybrid Bregman ADMM was then proposed in \\cite{Tu2019} to revisit problem \\eqref{DCADMM} with the same assumptions on $f$, $G$, and $H$. This version introduces an additional step to evaluate the convex conjugate of $G$, allowing the flexibility in choosing the subgradient step or proximal step to update the solution by choosing a control parameter to either activate or deactivate the additional step. By this strategy, this algorithm can include the one in \\cite{Sun2017} as a special case. However, further research on more general form of problem \\eqref{DCADMM} is still not available in the literature, to the best of the authors' knowledge. \n","start":5195,"end":9775},{"raw_tex":"Let $f\\colon \\mathbb{R}^d\\to \\left(-\\infty,+\\infty\\right]$. The \\emph{domain} of $f$ is $\\dom f :=\\{x\\in \\mathbb{R}^d: f(x) <+\\infty\\}$ and the \\emph{epigraph} of $f$ is $\\epi f := \\{(x,\\xi)\\in \\mathbb{R}^d\\times \\mathbb{R}: f(x)\\leq \\xi\\}$. The function $f$ is said to be \\emph{proper} if $\\dom f \\neq \\varnothing$ and it never takes the value $-\\infty$, \\emph{lower semicontinuous} if its epigraph is closed, and \\emph{convex} if its epigraph is convex. The function $f$ is said to be $\\alpha$-\\emph{strongly convex} ($\\alpha\\in \\mathbb{R}_+$) if $f-\\frac{\\alpha}{2}\\|\\cdot\\|^2$ is convex, and $\\beta$-\\emph{weakly convex} ($\\beta\\in \\mathbb{R}_+$) if $f+\\frac{\\beta}{2}\\|\\cdot\\|^2$ is convex. We say that $f$ is \\emph{coercive} if $f(x)\\to +\\infty$ as $\\|x\\|\\to +\\infty$.\n\nLet $x\\in \\mathbb{R}^d$ with $|f(x)| <+\\infty$. The \\emph{regular} (or \\emph{Fr\\'echet}) \\emph{subdifferential} of $f$ at $x$ is defined by\n\\begin{equation*}\n\\widehat{\\partial} f(x) :=\\left\\{x^*\\in \\mathbb{R}^d:\\; \\liminf_{y\\to x}\\frac{f(y)-f(x)-\\langle x^*,y-x\\rangle}{\\|y-x\\|}\\geq 0\\right\\}\n\\end{equation*}\nand the \\emph{limiting} (or \\emph{Mordukhovich}) \\emph{subdifferential} of $f$ at $x$ is defined by\n\\begin{equation*}\n\\partial_L f(x) :=\\left\\{x^*\\in \\mathbb{R}^d:\\; \\exists x_n\\stackrel{f}{\\to}x,\\; x_n^*\\to x^* \\text{~~with~~} x_n^*\\in \\widehat{\\partial} f(x_n)\\right\\},\n\\end{equation*}\nwhere the notation $y\\stackrel{f}{\\to}x$ means $y\\to x$ with $f(y)\\to f(x)$. In the case where $\\lvert f(x) \\rvert =+\\infty$, both regular subdifferential and limiting subdifferential of $f$ at $x$ are defined to be the empty set. The \\emph{domain} of $\\partial_L f$ is given by $\\dom \\partial_L f :=\\{x\\in \\mathbb{R}^d: \\partial_L f(x) \\neq \\varnothing\\}$. \nWe now collect some important properties of the limiting subdifferential.\n","start":14840,"end":16646},{"raw_tex":"We recall the concept of the well-known Bregman distance in a real Hilbert space $\\mathcal{H}$, which was introduced in \\cite{Bregman}. Given a differentiable convex function $\\phi: \\mathcal{H} \\to \\mathbb{R}$, the \\emph{Bregman distance} associated with $\\phi$ is defined as \n\\begin{align*}\n\\forall u, v\\in \\mathcal{H},\\quad D_{\\phi}(u,v)=\\phi(u)-\\phi(v)-\\langle \\nabla\\phi (v),u-v\\rangle,\n\\end{align*}\nWhen $\\phi(x)=\\|x\\|^2$, then $D_{\\phi}(u,v)=\\|u-v\\|^2$. When $\\phi(x)=x^\\top M x$, where $M$ is a positive semidefinite matrix, then $D_{\\phi}(u,v) = \\|u-v\\|^2_M :=(u-v)^\\top M (u-v)$. When $\\mathcal{H}=\\mathbb{R}^{p\\times d}$ and $\\phi(X)=\\|X\\|_F^2$, then $D_{\\phi}(U,V) =\\|U-V\\|_F^2$ \\cite{Dhillon2008}. Some useful properties of the Bregman distance are listed in the following proposition.\n","start":18239,"end":19037},{"raw_tex":"We first fix some notations which are used throughout the paper from now on. Let $\\mathbf{A}=[A_1|A_2|\\dots|A_m]$ and let $\\mathbf{I}_{q\\times q}$ be the $q\\times q$ identity matrix. Then $\\mathbf{Ax}=\\sum_{i=1}^{m}A_ix_i$. We also denote by $\\partial_L^{x}$ the limiting subdifferential with respect to the $x$-variable, and by $\\nabla_i f$ the $i^{th}$ block of the gradient vector $\\nabla f$. Given a matrix $M$, $\\lambda_{\\min}(M)$ denotes its smallest eigenvalue while $\\lambda_{\\max}(M)$ denotes its largest eigenvalue. The following assumptions are used in our convergence analysis\n\\begin{assumption}[Standing assumptions]\n\\label{a:standing}\n~\n\\begin{enumerate}\n\\item \n$\\nabla H$ is $\\ell_H$-Lipschitz continuous, $\\nabla P$ is $\\ell_P$-Lipschitz continuous, and $G$ is $\\beta$-weakly convex.\n\\item\\label{a:standing_image} \n$\\Ima(\\mathbf{A}) \\bigcup \\{b \\}\\subseteq \\Ima(B)$.\n\\item\\label{a:standing_rank} \n $\\lambda :=\\lambda_{\\min}(B^\\top B) >0$ (equivalently, $B$ has full column rank).\n\\end{enumerate}\n\\end{assumption}\n","start":21422,"end":22451},{"raw_tex":"Recall that the augmented Lagrangian of problem \\eqref{eq:prob} is given by\n\\begin{align*}\n\\mathcal{L}_\\rho(\\mathbf{x},y,z)=\\sum_{i=1}^m f_i(x_i)+H(y)+P(\\mathbf{x})-G(\\mathbf{x}) +  \\langle z, \\mathbf{Ax}+By-b \\rangle +\\frac{\\rho}{2}\\|\\mathbf{Ax}+By-b\\|^2,\n\\end{align*}\nwhere $z\\in \\mathbb{R}^p$ and $\\rho\\in \\mathbb{R}_{++}$. Now, we present our splitting algorithm with guaranteed global convergence to a stationary point $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ of $\\mathcal{L}_\\rho$, i.e., $0 \\in \\partial_L \\mathcal{L}_\\rho(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$, or equivalently,\n\\begin{equation}\\label{eq:optimality_conditions}\n\\begin{aligned}\n&0 \\in \\partial_L\\left(\\sum_{i=1}^{m}f_i -G\\right)(\\overline{\\mathbf{x}}) +\\nabla P(\\overline{\\mathbf{x}}) +\\mathbf{A}^\\top \\overline{z},\\\\\n&0 =\\nabla H(\\overline{y}) +B^\\top \\overline{z}, \\\\\n&0 =\\mathbf{A\\overline{x}} +B\\overline{y} -b.\n\\end{aligned}\n\\end{equation}\n","start":22747,"end":23687},{"raw_tex":"\\begin{tcolorbox}[\n\tleft=0pt,right=0pt,top=0pt,bottom=0pt,\n\tcolback=blue!10!white, colframe=blue!50!white,\n  \tboxrule=0.2pt,\n  \tbreakable]\n\\begin{algorithm}[Bregman proximal linearized ADMM (BPL-ADMM)]\n\\label{algo:BPLA}\n\\step{}\\label{step:A2_1}\nSet $n=0$. Choose $\\mathbf{x}_0 =(x_{1,0}, \\dots, x_{m,0}) \\in \\mathbb{R}^{d_1}\\times \\dots \\times \\mathbb{R}^{d_m}$, $y_0 \\in \\mathbb{R}^q$, and $z_0 \\in \\mathbb{R}^p$. Let $\\phi_i: \\mathbb{R}^{d_i} \\to \\mathbb{R}$, $i \\in \\{1,\\dots,m\\}$, be differentiable $\\alpha$-strongly convex functions with $\\alpha \\in (0, +\\infty)$, and let $\\psi: \\mathbb{R}^q \\to \\mathbb{R}$ be a differentiable convex function with $\\ell_{\\psi}$-Lipschitz continuous gradient. Let \n\\begin{align*}\n\\mu \\in \\left(\\frac{\\ell_P+\\beta}{\\alpha}, +\\infty\\right),\\;\\; \\nu \\in \\left[0, +\\infty\\right), \\text{~~and~~} \\rho \\in \\left(\\frac{\\ell_H +\\sqrt{\\ell_H^2 +8(\\ell_H +2\\nu\\ell_{\\psi})^2}}{2\\lambda}, +\\infty\\right).    \n\\end{align*}\n\n\\step{}\\label{step:A2_2}\nCalculate $\\nabla P(\\mathbf{x}_n)=(\\nabla_1 P(\\mathbf{x}_n),\\dots,\\nabla_m P(\\mathbf{x}_n))$ and $g_{n}=(g_{1,n},\\dots,g_{m,n}) \\in \\partial_L G(\\mathbf{x}_{n})$. Let $\\mu_n \\in [\\mu, +\\infty)$ and $\\nu_n \\in [0, \\nu]$. For each $i \\in \\{1,\\dots,m\\}$, define $\\mathbf{u}_{i,n+1}: \\mathbb{R}^{d_i}\\to \\mathbb{R}^{d_1}\\times \\dots \\times \\mathbb{R}^{d_m}$ by $\\mathbf{u}_{i,n+1}(x_i) =(x_{1,n+1}, \\dots, x_{i-1,n+1}, x_i, x_{i+1,n}, \\dots, x_{m,n})$, and find\n\\begin{align*}\nx_{i,n+1} &\\in  \\argmin_{x_i \\in \\mathbb{R}^{d_i}} \\left(f_i(x_i) +\\langle \\nabla_i P(\\mathbf{x}_n) -g_{i,n}, x_i \\rangle + \\langle z_n, A_ix_i\\rangle+\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,n+1}(x_i)+By_n-b\\|^2 +\\mu_n D_{\\phi_i}(x_i,x_{i,n})\\right),\\\\\ny_{n+1} &\\in  \\argmin_{y \\in \\mathbb{R}^q} \\left(H(y) + \\langle z_n, By\\rangle+\\frac{\\rho}{2}\\|\\mathbf{Ax}_{n+1} + By-b \\|^2 +\\nu_n D_{\\psi}(y,y_n) \\right),\\\\\nz_{n+1} &=z_n+\\rho(\\mathbf{Ax}_{n+1}+By_{n+1}-b),\n\\end{align*}\nwhere $\\mathbf{x}_{n+1} =(x_{1,n+1}, \\dots, x_{m,n+1})$.\n\\step{}\nIf a termination criterion is not met, set $n =n+1$ and go to Step~\\ref{step:A2_2}.\n\\end{algorithm}\n\\end{tcolorbox}\n","start":23689,"end":25795},{"raw_tex":"\\begin{lemma}\n\\label{l:norm}\nLet $L$ be a symmetric matrix in $\\mathbb{R}^{d\\times d}$ and let $M$ be a matrix in $\\mathbb{R}^{p\\times d}$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{l:norm_Mx}\nFor all $x\\in \\mathbb{R}^d$, $\\sqrt{\\lambda_{\\min}(M^\\top M)}\\|x\\| \\leq \\|Mx\\|$.\n\\item\\label{l:norm_M'x}\nFor all $x\\in \\mathbb{R}^d$, $\\gamma(x^\\top Lx) \\leq \\|Lx\\|^2$, where $\\gamma =\\min\\{|\\lambda|: \\lambda \\text{~is a nonzero eigenvalue of~} L\\}$. Consequently, for all $z\\in \\Ima(M)$, $\\sqrt{\\lambda_*(M^\\top M)}\\|z\\| \\leq \\|M^\\top z\\|$, where $\\lambda_*(M^\\top M)$ is the smallest strictly positive eigenvalue of $M^\\top M$. \n\\end{enumerate}\n\\end{lemma}","start":28813,"end":29477},{"raw_tex":"We now establish the convergence of the full sequence generated by Algorithm~\\ref{algo:BPLA}. Recall that a proper lower semicontinuous function $\\mathcal{F}\\colon \\mathcal{H}\\to \\left(-\\infty, +\\infty\\right]$ satisfies the \\emph{Kurdyka--Łojasiewicz (KL) property} \\cite{Kurdyka1998,Loja63} at $\\overline{x} \\in \\dom \\partial_L \\mathcal{F}$ if there are $\\eta \\in (0, +\\infty]$, a neighborhood $V$ of $\\overline{x}$, and a continuous concave function $\\varphi: \\left[0, \\eta\\right) \\to \\mathbb{R}_+$ such that $\\varphi$ is continuously differentiable with $\\varphi' > 0$ on $(0, \\eta)$, $\\varphi(0) = 0$, and, for all $x \\in V$ with $\n\\mathcal{F}(\\overline{x}) < \\mathcal{F}(x) < \\mathcal{F}(\\overline{x}) + \\eta$, \n\\begin{equation*}\n\\varphi'(\\mathcal{F}(x) -\\mathcal{F}(\\overline{x})) \\dist(0, \\partial_L \\mathcal{F}(x)) \\geq 1.   \n\\end{equation*}\nWe say that $\\mathcal{F}$ is a \\emph{KL function} if it satisfies the KL property at any point in $\\dom \\partial_L \\mathcal{F}$. If $\\mathcal{F}$ satisfies the KL property at $\\overline{x} \\in \\dom \\partial_L \\mathcal{F}$, in which the corresponding function $\\varphi$ can be chosen as $\\varphi(t) = c t ^{1 - \\lambda}$ for some $c \\in \\mathbb{R}_{++}$ and $\\lambda \\in [0, 1)$, then $\\mathcal{F}$ is said to satisfy the \\emph{KL property at $\\overline{x}$ with exponent $\\lambda$}. The function $\\mathcal{F}$ is called a \\emph{KL function with exponent $\\lambda$} if it is a KL function and has the same exponent $\\lambda$ at any $x \\in \\dom \\partial_L \\mathcal{F}$. Now we will present the proof of the full sequential convergence in the following theorem.\n","start":51504,"end":53113}]},"statement":{"raw_tex":"\\begin{theorem}[Full sequential convergence]\n\\label{t:full}\nSuppose that Assumption~\\ref{a:standing} holds, that $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, that $H$ is bounded below, that $G$ is differentiable with $\\ell_G$-Lipschitz continuous gradient, and that, for all $i\\in \\{1, \\dots, m\\}$, $\\nabla{\\phi_i}$ is $\\ell_{\\phi}$-Lipschitz continuous. Let $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ be the sequence generated by Algorithm~\\ref{algo:BPLA} with $z_0\\in \\Ima(B)$ and $\\limsup_{n\\to +\\infty} \\mu_n =\\overline{\\mu} <+\\infty$.\nDefine\n\\begin{align*}\n\\mathcal{F}(\\mathbf{x},y,z,t):=\\mathcal{L}_\\rho(\\mathbf{x},y,z) +c\\|y-t\\|^2, \\text{~~where~} c=\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}.\n\\end{align*}\n\\begin{enumerate}\n\\item\\label{t:full_cvg} \nSuppose that $\\mathcal{F}$ is a KL function. Then the sequence $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ converges to a stationary point $(\\mathbf{x}^{*},y^{*},z^{*})$ of $\\mathcal{L}_\\rho$ and \\begin{align*}\n\\sum_{n=0}^{+\\infty} \\|(\\mathbf{x}_{n+1},y_{n+1},z_{n+1})-(\\mathbf{x}_{n},y_{n},z_{n})\\| < +\\infty.\n\\end{align*}\n\\item\\label{t:full_rate} \nSuppose that $\\mathcal{F}$ is a KL function with exponent $\\kappa\\in [0,1)$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{t:full_rate_finite}\nIf $\\kappa =0$, then $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ converges to $(\\mathbf{x}^{*},y^{*},z^{*})$ in a finite number of steps. \n\\item\\label{t:full_rate_linear}\nIf $\\kappa\\in (0,\\frac{1}{2}]$, then there exist $\\overline{\\gamma}\\in \\mathbb{R}_{++}$ and $\\zeta\\in \\left(0,1\\right)$ such that, for all $n\\in \\mathbb{N}$, \n\\begin{align*}\n&\\|(\\mathbf{x}_{n},y_{n},z_{n})-(\\mathbf{x}^{*},y^{*},z^{*})\\| \\leq \\overline{\\gamma}\\zeta^{\\frac{n}{2}}, \\\\\n&\\|\\mathbf{Ax}_n+By_n-b\\| \\leq \\overline{\\gamma} \\zeta^{\\frac{n}{2}}, \\\\\n&\\lvert \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) -\\mathcal{L}_\\rho(\\mathbf{x}^{*},y^{*},z^{*}) \\rvert \\leq \\overline{\\gamma}\\zeta^n, \\\\\n\\text{and~} &|F(\\mathbf{x}_n,y_n)-F(\\mathbf{x}^{*},y^{*})| \\leq \\overline{\\gamma} \\zeta^{\\frac{n}{2}}.\n\\end{align*}\n\\item\\label{t:full_rate_sublinear}\nIf $\\kappa\\in (\\frac{1}{2},1)$, then there exists $\\overline{\\gamma}\\in \\mathbb{R}_{++}$ such that, for all $n\\in \\mathbb{N}$,\n\\begin{align*}\n&\\|(\\mathbf{x}_{n},y_{n},z_{n})-(\\mathbf{x}^{*},y^{*},z^{*})\\| \\leq \\overline{\\gamma} n^{-\\frac{1-\\kappa}{2\\kappa-1}}, \\\\\n&\\|\\mathbf{Ax}_n+By_n-b\\| \\leq \\overline{\\gamma} n^{-\\frac{1-\\kappa}{2\\kappa-1}}, \\\\\n&\\lvert \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) -\\mathcal{L}_\\rho(\\mathbf{x}^{*},y^{*},z^{*})\\rvert \\leq \\overline{\\gamma} n^{-\\frac{2-2\\kappa}{2\\kappa-1}}, \\\\\n\\text{and~} &|F(\\mathbf{x}_n,y_n)-F(\\mathbf{x}^{*},y^{*})| \\leq \\overline{\\gamma} n^{-\\frac{1-\\kappa}{2\\kappa-1}}.\n\\end{align*}\n\\end{enumerate}\n\\end{enumerate}\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}[Full sequential convergence]\n\\label{t:full}\nSuppose that Assumption~\\ref{a:standing} holds, that $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, that $H$ is bounded below, that $G$ is differentiable with $\\ell_G$-Lipschitz continuous gradient, and that, for all $i\\in \\{1, \\dots, m\\}$, $\\nabla{\\phi_i}$ is $\\ell_{\\phi}$-Lipschitz continuous. Let $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ be the sequence generated by Algorithm~\\ref{algo:BPLA} with $z_0\\in \\Ima(B)$ and $\\limsup_{n\\to +\\infty} \\mu_n =\\overline{\\mu} <+\\infty$.\nDefine\n\\begin{align*}\n\\mathcal{F}(\\mathbf{x},y,z,t):=\\mathcal{L}_\\rho(\\mathbf{x},y,z) +c\\|y-t\\|^2, \\text{~~where~} c=\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}.\n\\end{align*}\n\\begin{enumerate}\n\\item\\label{t:full_cvg} \nSuppose that $\\mathcal{F}$ is a KL function. Then the sequence $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ converges to a stationary point $(\\mathbf{x}^{*},y^{*},z^{*})$ of $\\mathcal{L}_\\rho$ and \\begin{align*}\n\\sum_{n=0}^{+\\infty} \\|(\\mathbf{x}_{n+1},y_{n+1},z_{n+1})-(\\mathbf{x}_{n},y_{n},z_{n})\\| < +\\infty.\n\\end{align*}\n\\item\\label{t:full_rate} \nSuppose that $\\mathcal{F}$ is a KL function with exponent $\\kappa\\in [0,1)$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{t:full_rate_finite}\nIf $\\kappa =0$, then $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ converges to $(\\mathbf{x}^{*},y^{*},z^{*})$ in a finite number of steps. \n\\item\\label{t:full_rate_linear}\nIf $\\kappa\\in (0,\\frac{1}{2}]$, then there exist $\\overline{\\gamma}\\in \\mathbb{R}_{++}$ and $\\zeta\\in \\left(0,1\\right)$ such that, for all $n\\in \\mathbb{N}$, \n\\begin{align*}\n&\\|(\\mathbf{x}_{n},y_{n},z_{n})-(\\mathbf{x}^{*},y^{*},z^{*})\\| \\leq \\overline{\\gamma}\\zeta^{\\frac{n}{2}}, \\\\\n&\\|\\mathbf{Ax}_n+By_n-b\\| \\leq \\overline{\\gamma} \\zeta^{\\frac{n}{2}}, \\\\\n&\\lvert \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) -\\mathcal{L}_\\rho(\\mathbf{x}^{*},y^{*},z^{*}) \\rvert \\leq \\overline{\\gamma}\\zeta^n, \\\\\n\\text{and~} &|F(\\mathbf{x}_n,y_n)-F(\\mathbf{x}^{*},y^{*})| \\leq \\overline{\\gamma} \\zeta^{\\frac{n}{2}}.\n\\end{align*}\n\\item\\label{t:full_rate_sublinear}\nIf $\\kappa\\in (\\frac{1}{2},1)$, then there exists $\\overline{\\gamma}\\in \\mathbb{R}_{++}$ such that, for all $n\\in \\mathbb{N}$,\n\\begin{align*}\n&\\|(\\mathbf{x}_{n},y_{n},z_{n})-(\\mathbf{x}^{*},y^{*},z^{*})\\| \\leq \\overline{\\gamma} n^{-\\frac{1-\\kappa}{2\\kappa-1}}, \\\\\n&\\|\\mathbf{Ax}_n+By_n-b\\| \\leq \\overline{\\gamma} n^{-\\frac{1-\\kappa}{2\\kappa-1}}, \\\\\n&\\lvert \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) -\\mathcal{L}_\\rho(\\mathbf{x}^{*},y^{*},z^{*})\\rvert \\leq \\overline{\\gamma} n^{-\\frac{2-2\\kappa}{2\\kappa-1}}, \\\\\n\\text{and~} &|F(\\mathbf{x}_n,y_n)-F(\\mathbf{x}^{*},y^{*})| \\leq \\overline{\\gamma} n^{-\\frac{1-\\kappa}{2\\kappa-1}}.\n\\end{align*}\n\\end{enumerate}\n\\end{enumerate}\n\\end{theorem}","start":53117,"end":55930}]},"proof":{"raw_tex":"\\begin{lemma}[Calculus rules]\n\\label{l:cal}\nLet $f, g\\colon \\mathbb{R}^d \\to (-\\infty, +\\infty]$ be proper lower semicontinuous functions, and let $x \\in \\mathbb{R}^d$. \n\\begin{enumerate}\n\\item\\label{l:cal_sum} \n(Sum rule) Suppose that $f$ is finite at $x$ and $g$ is locally Lipschitz around $\\overline{x}$. Then $\\partial_L (f + g)(\\overline{x}) \\subseteq \\partial_L f(\\overline{x})+\\partial_L g(\\overline{x})$. Moreover, if $g$ is strictly differentiable at $\\overline{x}$, then $\\partial_L(f + g)(\\overline{x}) = \\partial_Lf(\\overline{x}) + \\nabla g(\\overline{x})$.\n\\item\\label{l:cal_sep} \n(Separable sum rule) If $f(\\mathbf{x})=\\sum_{i=1}^{m}f_i(x_i)$ with $\\mathbf{x}=(x_1,\\dots,x_m)$, then $\\partial_L f(\\mathbf{x}) = \\partial_L f_1(x_1) \\times \\dots \\times \\partial_L f_m(x_m)$. \n\\end{enumerate}\n\\end{lemma}\n\n\\begin{proof}\n\\ref{l:cal_sum} follows from \\cite[Proposition~1.107(ii) and Theorem~3.36]{Mordukhovich2006}, while \\ref{l:cal_sep} follows from \\cite[Proposition~10.5]{Rockafellar1998}.\n\\end{proof}\n\n\\begin{lemma}[Upper semicontinuity of subdifferential]\n\\label{l:upsemicont}\nLet $f\\colon \\mathbb{R}^d \\to [-\\infty, +\\infty]$ be Lipschitz continuous around $x \\in \\mathbb{R}^d$ with $|f(x)| <+\\infty$, and consider sequences $(x_n)_{n\\in \\mathbb{N}}$ and $(x_n^*)_{n\\in \\mathbb{N}}$ in $\\mathbb{R}^d$ such that $x_n \\to x$ and, for all ${n\\in \\mathbb{N}}$, $x_n^* \\in \\partial_L f(x_n)$. \nThen $(x_n^*)_{n\\in \\mathbb{N}}$ is bounded and its cluster points are contained in $\\partial_L f(x)$.\n\\end{lemma}\n\n\\begin{proof}\nThis follows from \\cite[Lemma 2.2]{Tan2023}.\n\\end{proof}\n\n\\begin{proposition}\n\\label{p:Bregman}\nLet $\\phi: \\mathcal{H} \\to \\mathbb{R}$ be a differentiable convex function and $D_{\\phi}$ is the Bregman distance associated with $\\phi$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{p:Bregman_1} \nFor all $u, v \\in \\mathcal{H}$, $D_{\\phi}(u, v)\\geq 0$, and $D_{\\phi}(u, v)=0$ if and only if $u = v$. \n\n\\item\\label{p:Bregman_2} \nFor each $v \\in \\mathcal{H}$, $D_{\\phi}(\\cdot, v)$ is convex. \n\n\\item\\label{p:Bregman_3} \nIf $\\phi$ is $\\alpha$-strongly convex,  then, for all $u, v \\in \\mathcal{H}$, $D_{\\phi}(u, v) \\geq \\frac{\\alpha}{2} \\|u-v\\|^2$. \n\n\\item\\label{p:Bregman_4} \nIf $\\nabla \\phi$ is $\\ell_{\\phi}$-Lipschitz continuous, then, for all $u, v \\in \\mathcal{H}$, $D_{\\phi}(u,v)\\leq \\frac{\\ell_{\\phi}}{2}\\|u-v\\|^2$. \n\\end{enumerate}\n\\end{proposition}\n\n\\begin{proof}\n\\ref{p:Bregman_1} is given in \\cite[Section 1]{Bregman}. \\ref{p:Bregman_2}, \\ref{p:Bregman_3}, and \\ref{p:Bregman_4} can be easily verified by using the definitions. \n\\end{proof}\n\n\\begin{lemma}\n\\label{l:diff}\nLet $h\\colon \\mathbb{R}^d\\to \\mathbb{R}$ be a differentiable function with $\\ell$-Lipschitz continuous gradient. Then the following hold:\n\\begin{enumerate}\n\\item\\label{l:diff_descent}\nFor all $x, y\\in \\mathbb{R}^d$, $|h(y) -h(x) -\\langle \\nabla h(x), y -x\\rangle|\\leq \\frac{\\ell}{2}\\|y -x\\|^2$.\n\\item\\label{l:diff_bounded}\nIf $h$ is bounded below and $\\ell >0$, then\n\\begin{align*}\n\\inf_{x\\in \\mathbb{R}^d} \\left(h(x) -\\frac{1}{2\\ell}\\|\\nabla h(x)\\|^2\\right) >-\\infty.    \n\\end{align*}\n\\end{enumerate}  \n\\end{lemma}\n\n\\begin{proof}\n\\ref{l:diff_descent}: This follows from \\cite[Lemma~1.2.3]{Nesterov2018}.\n\n\\ref{l:diff_bounded}: Let $x\\in \\mathbb{R}^d$. Following the argument of \\cite[Equation~(1.2.19)]{Nesterov2018}, we apply \\ref{l:diff_descent} with $y =x -\\frac{1}{\\ell}\\nabla h(x)$ to obtain that\n\\begin{align*}\nh\\left(x -\\frac{1}{\\ell}\\nabla h(x)\\right) &\\leq h(x) +\\langle \\nabla h(x), y -x\\rangle +\\frac{\\ell}{2}\\|y -x\\|^2 \\\\\n&= h(x) -\\frac{1}{\\ell}\\|\\nabla h(x)\\|^2 +\\frac{1}{2\\ell}\\|\\nabla h(x)\\|^2 \\\\\n&= h(x) -\\frac{1}{2\\ell}\\|\\nabla h(x)\\|^2,\n\\end{align*}\nwhich together with the boundedness of $h$ implies the conclusion.\n\\end{proof}\n\n\\begin{theorem}[Subsequential convergence]\n\\label{t:cvg}\nSuppose that Assumption~\\ref{a:standing} holds. Let $(\\mathbf{x}_n,y_n,z_n)_{n\\in \\mathbb{N}}$ be the sequence generated by Algorithm~\\ref{algo:BPLA} and, for each $n\\in \\mathbb{N}^*$, define\n\\begin{align*}\nL_n =\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) +c\\|y_n-y_{n-1}\\|^2, \\text{~where~} c=\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}.     \n\\end{align*}\nThen the following hold:\n\\begin{enumerate}\n\\item\\label{t:cvg_decrease}\nFor all $n\\in \\mathbb{N}^*$,\n\\begin{align}\\label{eq:decrease}\nL_{n+1} +\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\delta_y\\|y_{n+1}-y_n\\|^2 \\leq L_n,\n\\end{align}\nwhere $\\delta_{\\mathbf{x}} =\\frac{\\mu\\alpha -\\ell_P -\\beta}{2} > 0$ and $\\delta_y =\\frac{\\lambda\\rho}{2} -\\frac{(\\ell_H+2\\nu\\ell_{\\psi})^2}{\\lambda\\rho} -\\frac{\\ell_H}{2} >0$.\n\n\\item\\label{t:cvg_seq}\nSuppose that $z_0\\in \\Ima(B)$, that $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, and that $H$ is bounded below. \nThen the sequence $(\\mathbf{x}_n,{y}_n,{z}_n)_{n\\in \\mathbb{N}}$ is bounded and $\\mathbf{x}_{n+1}-\\mathbf{x}_n\\to 0$, $y_{n+1}-y_n\\to 0$, $z_{n+1}-z_n\\to 0$, and $\\mathbf{Ax}_n +By_n \\to b$ as $n\\to +\\infty$. Furthermore, for every cluster point $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ of $(\\mathbf{x}_n,{y}_n,{z}_n)_{n\\in \\mathbb{N}}$, it holds that\n\\begin{align*}\n\\mathbf{A}\\overline{\\mathbf{x}} +b\\overline{y} =b \\text{~~and~~} \\lim_{n \\to +\\infty} F(\\mathbf{x}_{n },y_{n}) =\\lim_{n \\to +\\infty}\\mathcal{L}_\\rho(\\mathbf{x}_{n },y_{n},z_{n}) =\\mathcal{L}_\\rho(\\overline{\\mathbf{x}},\\overline{y},\\overline{z}) =F(\\overline{\\mathbf{x}},\\overline{y}),    \n\\end{align*}\nand that if $G$ is strictly differentiable and, for each $i \\in \\{1,\\dots,m\\}$, $\\nabla \\phi_i$ is $\\ell_{\\phi}$-Lipschitz continuous, then $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ is a stationary point of $\\mathcal{L}_\\rho$.\n\\end{enumerate}\n\\end{theorem}\n\n\\begin{proof}\n\\ref{t:cvg_decrease}: Let $n\\in \\mathbb{N}$. The update of $z_{n+1}$ yields\n\\begin{align}\\label{eq:z-update}\nz_{n+1} -z_n =\\rho(\\mathbf{Ax}_{n+1}+By_{n+1}-b).   \n\\end{align}\nBy the optimality condition for the update of $y_{n+1}$ in Step~\\ref{step:A2_2} of Algorithm~\\ref{algo:BPLA}, \n\\begin{align*}\n0 &= \\nabla H(y_{n+1}) + B^\\top z_n +\\rho B^\\top(\\mathbf{Ax}_{n+1} +By_{n+1} -b) +\\nu_n(\\nabla \\psi(y_{n+1}) - \\nabla \\psi(y_n)),\n\\end{align*}\nwhich combined with \\eqref{eq:z-update} yields\n\\begin{align}\\label{eq:B'z}\nB^\\top z_{n+1} =-\\nabla H(y_{n+1}) -\\nu_n(\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)).  \n\\end{align}\nFrom the definition of the augmented Lagrangian, we have that\n\\begin{align}\\label{eq:zz}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n}) &= \\langle z_{n+1}-z_n, \\mathbf{Ax}_{n+1}+By_{n+1}-b \\rangle \\notag\\\\\n&= \\frac{1}{\\rho}\\|z_{n+1}-z_n\\|^2 \n\\end{align}\nand that\n\\begin{align*}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_n,z_n) &= H(y_{n+1}) - H(y_n) + \\langle z_n, B(y_{n+1}-y_n) \\rangle \\\\\n&\\qquad + \\frac{\\rho}{2}(\\|\\mathbf{Ax}_{n+1}+By_{n+1}-b\\|^2 - \\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2).  \n\\end{align*}\nWriting $\\mathbf{Ax}_{n+1}+By_{n}-b =(\\mathbf{Ax}_{n+1}+By_{n+1}-b) -(By_{n+1}- By_n)$, we obtain that\n\\begin{align*}\n&\\|\\mathbf{Ax}_{n+1}+By_{n+1}-b\\|^2 - \\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2 \\\\\n&= -\\|B(y_{n+1}-y_n)\\|^2 +2\\langle  \\mathbf{Ax}_{n+1}+By_{n+1}-b, B(y_{n+1}-y_n)\\rangle \\\\\n&= -\\|B(y_{n+1}-y_n)\\|^2 +\\frac{2}{\\rho}\\langle  z_{n+1}-z_{n}, B(y_{n+1}-y_n)\\rangle,\n\\end{align*}\nand so\n\\begin{align*}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_n,z_n) = H(y_{n+1}) - H(y_n) +\\langle z_{n+1}, B(y_{n+1}-y_n) \\rangle -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2.\n\\end{align*}\nIn view of \\eqref{eq:B'z},\n\\begin{align*}\n\\langle z_{n+1}, B(y_{n+1}-y_n) \\rangle &=\\langle B^\\top z_{n+1}, y_{n+1}-y_n \\rangle \\\\\n&=-\\langle \\nabla H(y_{n+1}), y_{n+1}-y_n \\rangle -\\nu_n\\langle \\nabla \\psi(y_{n+1})- \\nabla \\psi(y_n), y_{n+1}-y_n \\rangle \\\\\n&\\leq -H(y_{n+1}) +H(y_n) +\\frac{\\ell_H}{2}\\|y_{n+1}-y_n\\|^2,\n\\end{align*}\nwhere the last inequality is from the $\\ell_H$-Lipschitz continuity of $\\nabla H$ and Lemma~\\ref{l:diff}\\ref{l:diff_descent}, and from the monotonicity of $\\nabla \\psi$. \nWe therefore obtain that\n\\begin{align}\\label{eq:yy}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_n,z_n) \\leq \\frac{\\ell_H}{2}\\|y_{n+1}-y_n\\|^2 -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2.    \n\\end{align}\nNext, by the update of $x_{i,n+1}$ in Step~\\ref{step:A2_2} of Algorithm~\\ref{algo:BPLA}, for all $i \\in \\{1,\\dots,m\\}$ and all $x_i \\in \\mathbb{R}^{d_i}$,\n\\begin{align}\\label{eq:update_step_x}\n&f_i(x_{i,n+1}) +\\langle \\nabla_i P(\\mathbf{x}_n) -g_{i,n}, x_{i,n+1} \\rangle +\\langle z_n, A_{i}x_{i,n+1} \\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,n+1}(x_{i,n+1}) +By_n -b\\|^2 +\\mu_n D_{\\phi_i}(x_{i,n+1},x_{i,n}) \\notag \\\\\n&\\leq  f_i(x_i) +\\langle \\nabla_i P(\\mathbf{x}_n) -g_{i,n}, x_i \\rangle +\\langle z_n, A_{i}x_i\\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,n+1}(x_i) +By_n -b\\|^2 +\\mu_n D_{\\phi_i}(x_i,x_{i,n}).\n\\end{align}\nLetting $x_i =x_{i,n}$, using Proposition~\\ref{p:Bregman}\\ref{p:Bregman_1}\\&\\ref{p:Bregman_3}, and summing up over $i \\in \\{1,\\dots,m\\}$, we derive that\n\\begin{align}\\label{eq:from x-update}\n&\\sum_{i=1}^m f_i(x_{i,n+1}) +\\langle \\nabla P(\\mathbf{x}_n) -g_n, \\mathbf{x}_{n+1} \\rangle +\\langle z_n, \\mathbf{Ax}_{n+1} \\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{x}_{n+1} +By_n -b\\|^2 +\\frac{\\mu\\alpha}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 \\notag\\\\\n&\\leq \\sum_{i=1}^m f_i(x_{i,n}) +\\langle \\nabla P(\\mathbf{x}_n) -g_n, \\mathbf{x}_{n} \\rangle +\\langle z_n, \\mathbf{Ax}_{n} \\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{x}_n +By_n -b\\|^2.\n\\end{align}\nOn the other hand,\n\\begin{align}\\label{eq:xx}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_n,z_n) \\notag\\\\\n&= \\sum_{i=1}^m f_i(x_{i,n+1}) -\\sum_{i=1}^m f_i(x_{i,n}) +P(\\mathbf{x}_{n+1}) -P(\\mathbf{x}_{n}) -G(\\mathbf{x}_{n+1}) +G(\\mathbf{x}_{n}) \\notag\\\\ \n&\\quad +\\langle z_n, \\mathbf{Ax}_{n+1}-\\mathbf{Ax}_{n}\\rangle +\\frac{\\rho}{2}\\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2 -\\frac{\\rho}{2}\\|\\mathbf{Ax}_{n}+By_{n}-b\\|^2 \\notag\\\\\n&\\leq \\sum_{i=1}^m f_i(x_{i,n+1}) -\\sum_{i=1}^m f_i(x_{i,n}) +\\langle \\nabla P(\\mathbf{x}_n), \\mathbf{x}_{n+1}-\\mathbf{x}_n \\rangle + \\frac{\\ell_P}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_n\\|^2 - \\langle g_n, \\mathbf{x}_{n+1} -\\mathbf{x}_{n} \\rangle + \\frac{\\beta}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 \\notag\\\\\n&\\quad +\\langle z_n, \\mathbf{Ax}_{n+1}-\\mathbf{Ax}_{n}\\rangle  + \\frac{\\rho}{2}\\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2 - \\frac{\\rho}{2}\\|\\mathbf{Ax}_{n}+By_{n}-b\\|^2 \\notag\\\\\n&\\leq -\\frac{\\mu\\alpha -\\ell_P -\\beta}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 =-\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2,\n\\end{align}\nwhere the first inequality is obtained by using Lemma~\\ref{l:diff}\\ref{l:diff_descent} on the $\\ell_P$-Lipschitz continuity of $\\nabla P$ and \\cite[Lemma~4.1]{BDL22} on the $\\beta$-weak convexity of $G$, while the last inequality follows from \\eqref{eq:from x-update}. Summing up three relations \\eqref{eq:zz}, \\eqref{eq:yy}, and \\eqref{eq:xx} yields\n\\begin{align}\\label{eq:LL}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) - \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) \\notag\\\\\n&\\leq -\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\frac{\\ell_H}{2}\\|y_{n+1}-y_n\\|^2 -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2 +\\frac{1}{\\rho}\\|z_{n+1}-z_n\\|^2.\n\\end{align}\n\nNow, since $\\Ima(\\mathbf{A}) \\cup \\{b\\} \\subseteq \\Ima(B)$, it follows from \\eqref{eq:z-update} that\n\\begin{align}\\label{eq:zImB}\nz_{n+1}-z_n \\in \\Ima(B)    \n\\end{align}\nwhich, by Lemma~\\ref{l:norm}\\ref{l:norm_M'x}, yields \n\\begin{align*}\n\\|z_{n+1} -z_n\\| \\leq \\frac{1}{\\sqrt{\\lambda}}\\|B^\\top z_{n+1} -B^\\top z_n\\|.\n\\end{align*}\nLet $n\\in \\mathbb{N}^*$. We derive from \\eqref{eq:B'z} and the Lipschitz continuity of $\\nabla H$ and $\\nabla \\psi$ that\n\\begin{align}\\label{eq:BTZy}\n\\|B^\\top z_{n+1} -B^\\top z_n\\| &\\leq \\|\\nabla H(y_{n+1}) -\\nabla H(y_n)\\| +\\nu_n\\|\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)\\| +\\nu_{n-1}\\|\\nabla \\psi(y_n) - \\nabla \\psi(y_{n-1})\\| \\notag \\\\ \n&\\leq (\\ell_H +\\nu\\ell_{\\psi})\\|y_{n+1} -y_n\\| +\\nu\\ell_{\\psi}\\|y_n -y_{n-1}\\|,\n\\end{align}\nand thus, \n\\begin{align}\\label{eq:zy}\n\\|z_{n+1} -z_n\\| \\leq \\frac{1}{\\sqrt{\\lambda}}\\|B^\\top z_{n+1} -B^\\top z_n\\| \\leq \\frac{1}{\\sqrt{\\lambda}}((\\ell_H +\\nu\\ell_{\\psi})\\|y_{n+1} -y_n\\| +\\nu\\ell_{\\psi}\\|y_n -y_{n-1}\\|).\n\\end{align}\nApplying Cauchy--Schwarz inequality to two vectors $u =(\\sqrt{\\ell_H+\\nu\\ell_{\\psi}},\\sqrt{\\nu\\ell_{\\psi}})$ and $v =(\\sqrt{\\ell_H+\\nu\\ell_{\\psi}}\\|y_{n+1} -y_n\\|,\\sqrt{\\nu\\ell_{\\psi}}\\|y_n -y_{n-1}\\|)$, we obtain that\n\\begin{align*}\n\\|z_{n+1} -z_n\\|^2 \\leq \\frac{\\ell_H +2\\nu\\ell_{\\psi}}{\\lambda}((\\ell_H +\\nu\\ell_{\\psi})\\|y_{n+1} -y_n\\|^2 +\\nu\\ell_{\\psi}\\|y_n -y_{n-1}\\|^2).    \n\\end{align*}\nBy combining this with \\eqref{eq:LL} and using Lemma~\\ref{l:norm}\\ref{l:norm_Mx},\n\\begin{align*}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) - \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) \\notag\\\\\n&\\leq -\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\left(\\frac{(\\ell_H+2\\nu\\ell_{\\psi})(\\ell_H+\\nu\\ell_{\\psi})}{\\lambda\\rho} +\\frac{\\ell_H}{2}\\right)\\|y_{n+1}-y_n\\|^2 \\\\ \n&\\qquad -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2 +\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2 \\\\\n&\\leq -\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 -\\left(\\frac{\\lambda\\rho}{2} -\\frac{(\\ell_H+2\\nu\\ell_{\\psi})(\\ell_H+\\nu\\ell_{\\psi})}{\\lambda\\rho} -\\frac{\\ell_H}{2}\\right)\\|y_{n+1}-y_n\\|^2 \\\\\n&\\qquad +\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2,\n\\end{align*}\nor equivalently,\n\\begin{align*}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) +\\frac{(\\ell_H +2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_{n+1}-y_{n}\\|^2 + \\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\delta_y\\|y_{n+1}-y_n\\|^2 \\\\\n&\\leq \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) +\\frac{(\\ell_H +2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2,\n\\end{align*}\nwhich proves \\eqref{eq:decrease}. Here, we note that $\\delta_{\\mathbf{x}} =\\frac{\\mu\\alpha -\\ell_P -\\beta}{2} > 0$ since $\\mu > \\frac{\\ell_P+\\beta}{\\alpha}$ and that $\\delta_y =\\frac{\\lambda\\rho}{2} -\\frac{(\\ell_H+2\\nu\\ell_{\\psi})^2}{\\lambda\\rho} -\\frac{\\ell_H}{2} >0$ since $\\rho > \\frac{\\ell_H +\\sqrt{\\ell_H^2 +8(\\ell_H +2\\nu\\ell_{\\psi})^2}}{2\\lambda}$.\n\n\\ref{t:cvg_seq}: Let $n\\in \\mathbb{N}^*$. As $z_0 \\in \\Ima(B)$, we have from \\eqref{eq:zImB} that $z_n \\in \\Ima (B)$. Using Lemma~\\ref{l:norm}\\ref{l:norm_M'x}, \\eqref{eq:B'z}, and the Lipschitz continuity of $\\psi$ yields\n\\begin{align*}\n\\sqrt{\\lambda}\\|z_n\\|\\leq \\|B^\\top z_n\\| \\leq\\|\\nabla H(y_n)\\|+ \\nu_n\\|\\nabla \\psi(y_{n}) -\\nabla \\psi(y_{n-1})\\| \n\\leq \\|\\nabla H(y_n)\\|+ \\nu \\ell_{\\psi}\\|y_{n}-y_{n-1}\\|.\n\\end{align*}\nFor $\\omega\\in \\mathbb{R}_{++}$ to be chosen later, invoking Cauchy--Schwarz inequality with two vectors $u =(\\sqrt{\\omega},\\sqrt{\\nu\\ell_{\\psi}})$ and $v =(\\frac{1}{\\sqrt{\\omega}}\\|\\nabla H(y_n)\\|,\\sqrt{\\nu\\ell_{\\psi}}\\|y_n-y_{n-1}\\|)$, we have that\n\\begin{align}\\label{eq:uppper_zn}\n\\|z_n\\|^2 &\\leq \\frac{\\omega +\\nu\\ell_{\\psi}}{\\lambda} \\left(\\frac{1}{\\omega}\\|\\nabla H(y_n)\\|^2 +\\nu\\ell_{\\psi}\\|y_n-y_{n-1}\\|^2\\right) \\notag \\\\\n&= \\frac{1}{\\lambda}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right)\\|\\nabla H(y_n)\\|^2 +\\frac{(\\omega +\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda}\\|y_n-y_{n-1}\\|^2.\n\\end{align}\nCombining this with the fact that\n\\begin{align*}\n\\langle z_n, \\mathbf{Ax}_n+By_n-b \\rangle +\\frac{\\rho}{2}\\|\\mathbf{Ax}_n+By_n-b\\|^2 =\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} +\\mathbf{Ax}_n+By_n-b \\right\\|^2 -\\frac{1}{2\\rho}\\|z_n\\|^2,\n\\end{align*}\nwe deduce that\n\\begin{align*}\nL_n &=\\mathcal{L}_\\rho(\\mathbf{x}_n,y_n,z_n) + c\\|y_n-y_{n-1}\\|^2 \\\\\n&\\geq \\sum_{i=1}^m f_i(x_{i,n}) +H(y_n) +P(\\mathbf{x}_n) -G(\\mathbf{x}_n) \\\\ \n&\\quad +\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} + \\mathbf{Ax}_n+By_n-b \\right\\|^2 -\\frac{1}{2\\lambda\\rho}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right)\\|\\nabla H(y_n)\\|^2 +\\left(c -\\frac{(\\omega +\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{2\\lambda\\rho}\\right)\\|y_n-y_{n-1}\\|^2 \\\\\n&= \\left(\\sum_{i=1}^m f_i(x_{i,n}) +P(\\mathbf{x}_n) -G(\\mathbf{x}_n)\\right) +\\left(H(y_n) -\\frac{1}{2\\ell_H}\\|\\nabla H(y_n)\\|^2\\right) +\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} + \\mathbf{Ax}_n+By_n-b \\right\\|^2 \\\\ \n&\\quad +\\left(\\frac{1}{2\\ell_H} -\\frac{1}{2\\lambda\\rho}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right)\\right)\\|\\nabla H(y_n)\\|^2 +\\frac{\\nu\\ell_{\\psi}}{2\\lambda\\rho}(2\\ell_H +3\\nu\\ell_{\\psi} -\\omega)\\|y_n-y_{n-1}\\|^2.\n\\end{align*}\nNow, we note that\n\\begin{align*}\n\\rho > \\frac{\\ell_H +\\sqrt{\\ell_H^2 +8(\\ell_H +2\\nu\\ell_{\\psi})^2}}{2\\lambda} \\geq \\frac{\\ell_H +\\sqrt{\\ell_H^2 +8\\ell_H^2}}{2\\lambda} =\\frac{2\\ell_H}{\\lambda},    \n\\end{align*}\nand so $\\frac{1}{2\\ell_H} >\\frac{1}{\\lambda\\rho}$. Choosing $\\omega =1$ if $\\nu\\ell_{\\psi} =0$, and $\\omega =\\nu\\ell_{\\psi}$ if $\\nu\\ell_{\\psi} >0$, it holds that\n\\begin{align*}\n&\\theta :=\\frac{1}{2\\ell_H} -\\frac{1}{2\\lambda\\rho}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right) \\geq \\frac{1}{2\\ell_H} -\\frac{1}{2\\lambda\\rho}(1 +1) >0 \\text{~~and~~} \\\\\n&\\frac{\\nu\\ell_{\\psi}}{2\\lambda\\rho}(2\\ell_H +3\\nu\\ell_{\\psi} -\\omega) \\geq \\frac{\\nu\\ell_{\\psi}}{2\\lambda\\rho}(3\\nu\\ell_{\\psi} -\\omega) =\\frac{\\nu^2\\ell_{\\psi}^2}{\\lambda\\rho}. \n\\end{align*}\nThis together with \\ref{t:cvg_decrease} yields\n\\begin{align}\\label{eq:L_n_bounded}\nL_1 \\geq L_n &\\geq \\left(\\sum_{i=1}^m f_i(x_{i,n}) +P(\\mathbf{x}_n) -G(\\mathbf{x}_n)\\right) +\\left(H(y_n) -\\frac{1}{2\\ell_H}\\|\\nabla H(y_n)\\|^2\\right) +\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} + \\mathbf{Ax}_n+By_n-b \\right\\|^2 \\notag \\\\ \n&\\quad +\\theta\\|\\nabla H(y_n)\\|^2 +\\frac{\\nu^2\\ell_{\\psi}^2}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2,\n\\end{align}\nin which the first term on the right hand side is bounded below since $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, the second term is bounded below due to {the assumption on $H$ and Lemma~\\ref{l:diff}\\ref{l:diff_bounded}}, while the remaining terms are nonnegative due to their positive coefficients. Therefore, all the terms are bounded. Again, as $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, the boundedness of the first term implies that $(\\mathbf{x}_{n})_{n\\in \\mathbb{N}}$ is bounded. Since the last two terms are bounded, it follows from \\eqref{eq:uppper_zn} that $(z_n)_{n\\in \\mathbb{N}}$ is bounded. Since $z_{n+1}=z_n+\\rho(\\mathbf{Ax}_{n+1}+By_{n+1}-b)$ and $B$ has full column rank, we also obtain that $(y_n)_{n\\in \\mathbb{N}}$ is bounded. Consequently, the sequence $(\\mathbf{x}_n,y_n,z_n)_{n\\in \\mathbb{N}}$ is bounded.\n\nOn the other hand, since all the terms on the right hand side of \\eqref{eq:L_n_bounded} are bounded below, so is the sequence $(L_n)_{n\\in\\mathbb{N}}$. Combining with the nonincreasing property of $(L_n)_{n\\in\\mathbb{N}}$ due to \\ref{t:cvg_decrease}, it follows that $(L_n)_{n\\in\\mathbb{N}}$ is convergent. By rearranging \\eqref{eq:decrease} and performing telescoping, we then have that\n\\begin{align*}\n\\sum_{n=1}^{+\\infty}\\delta_y \\|y_{n+1}-y_n\\|^2 + \\delta_{\\mathbf{x}}\\sum_{n=1}^{+\\infty}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 \\leq \\sum_{n=1}^{+\\infty}(L_n - L_{n+1}) = L_1 - \\lim_{n\\to+\\infty} L_n < +\\infty,\n\\end{align*}\nwhich implies that $\\sum_{n=1}^{+\\infty}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_n\\|^2 < +\\infty$, $\\sum_{n=1}^{+\\infty}\\|y_{n+1}-y_n\\|^2 <+\\infty$, and by \\eqref{eq:zy}, $\\sum_{n=1}^{+\\infty}\\|z_{n+1}-z_n\\|^2 < +\\infty$. As a result, $\\mathbf{x}_{n+1}-\\mathbf{x}_{n} \\to 0$, $y_{n+1}-y_n \\to 0$, $z_{n+1}-z_n \\to 0$, and $\\mathbf{Ax}_n +By_n =(z_n -z_{n-1})/\\rho +b \\to b$ as $n \\to +\\infty$. It then follows from the definition of $L_n$ that the sequence $(\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}))_{n\\in\\mathbb{N}}$ is also convergent with $\\lim_{n\\to+\\infty} \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) =\\lim_{n\\to+\\infty}L_n$.\n\nLet $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ be a cluster point of the sequence  $(\\mathbf{x}_n,y_n,z_n)_{n\\in \\mathbb{N}}$. Then there exists a subsequence $(\\mathbf{x}_{k_n},y_{k_n},z_{k_n})_{n\\in \\mathbb{N}}$ that converges to $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$. In view of \\eqref{eq:update_step_x}, letting $n =k_n$ and using Proposition~\\ref{p:Bregman}\\ref{p:Bregman_3}\\&\\ref{p:Bregman_4}, we obtain that, for all $i \\in \\{1,\\dots,m\\}$ and all $x_i\\in \\mathbb{R}^{d_i}$,\n\\begin{align}\\label{eq:kn+}\n&f_i(x_{i,k_n+1}) +\\langle \\nabla_i P(\\mathbf{x}_{k_n})- g_{i,k_n}, x_{i,k_n+1} -x_i \\rangle + \\langle z_{k_n}, A_i x_{i,k_n+1} - A_ix_i\\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,k_n+1}(x_{i,k_n+1}) +By_{k_n} -b \\|^2 \\notag \\\\\n&\\quad  - \\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,k_n+1}(x_{i}) +By_{k_n} -b \\|^2 +\\frac{\\mu\\alpha}{2}\\|x_{i,k_n+1}-x_{i,k_n}\\|^2 - \\frac{\\nu\\ell_{\\phi}}{2}\\|x_{i}-x_{i,k_n}\\|^2 \\leq f_i(x_{i}).\n\\end{align}\nSince $\\nabla P$ is continuous, $\\nabla P(\\mathbf{x}_{k_n}) \\to \\nabla P(\\overline{\\mathbf{x}})$ as $n\\to +\\infty$. As $G +\\frac{\\beta}{2}\\|\\cdot\\|^2$ is a continuous convex function, it follows from \\cite[Example~9.14]{Rockafellar1998} that $G$ is locally Lipschitz continuous. Since $\\mathbf{x}_{k_n} \\to \\overline{\\mathbf{x}}$ as $n\\to +\\infty$, using Lemma~\\ref{l:upsemicont} and passing to a subsequence if necessary, we can and do assume that $g_{k_n}\\to \\overline{g} \\in \\partial_L G (\\overline{\\mathbf{x}})$ as $n\\to +\\infty$. Now, for each $i \\in \\{1,\\dots,m\\}$, letting $x_i =\\overline{x}_i$ and $n\\to +\\infty$ in \\eqref{eq:kn+} with noting that $x_{i,k_n+1} \\to \\overline{x}_i$, $y_{k_n} \\to \\overline{y}$, and $z_{k_n} \\to \\overline{z}$, we have that $\\limsup_{n \\to +\\infty} f_i(x_{i,k_n+1}) \\leq f_i(\\overline{x}_i)$, which together with the lower semicontinuity of $f_i$ yields\n\\begin{align*}\n\\lim_{n \\to +\\infty} f_i(x_{i,k_n+1}) = f_i(\\overline{x}_i).\n\\end{align*}\nAs $H$, $P$, and $G$ are continuous, it follows that $\\lim_{n \\to +\\infty}\\mathcal{L}_\\rho(\\mathbf{x}_{k_n +1 },y_{k_n+1},z_{k_n +1}) = \\mathcal{L}_\\rho(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$. Since the sequence $(\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}))_{n\\in\\mathbb{N}}$ is convergent, we deduce that\n\\begin{align*}\n\\lim_{n \\to +\\infty}\\mathcal{L}_\\rho(\\mathbf{x}_{n },y_{n},z_{n}) =\\mathcal{L}_\\rho(\\overline{\\mathbf{x}},\\overline{y},\\overline{z}).    \n\\end{align*}\n\nWe next show that $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ is a stationary point of $\\mathcal{L}_\\rho$. In \\eqref{eq:z-update} and \\eqref{eq:B'z} with $n$ replaced by $k_n-1$, letting $n\\to +\\infty$ and using the coninuity of $\\nabla H$ and the Lipschitz continuity of $\\nabla \\psi$, we obtain that $0 =\\mathbf{A\\overline{x}} + B\\overline{y} -b$ and $0 =\\nabla H(\\overline{y}) + B^\\top \\overline{z}$. In turn, letting $n \\to +\\infty$ in \\eqref{eq:kn+}, we have for all $i \\in \\{1,\\dots,m\\}$ and all $x_i\\in \\mathbb{R}^{d_i}$ that\n\\begin{align*}\n&f_i(\\overline{x}_i) +\\langle \\nabla_i P(\\overline{\\mathbf{x}})-\\overline{g}_i, \\overline{x}_i-x_i \\rangle + \\langle \\overline{z}, A_i\\overline{x}_i-A_ix_i \\rangle\\\\\n&\\quad +\\frac{\\rho}{2}\\|A_1\\overline{x}_1 +\\dots +A_{i-1}\\overline{x}_{i-1} +A_i\\overline{x}_i +A_{i+1}\\overline{x}_{i+1}+\\dots+A_m \\overline{x}_{m} +B\\overline{y} -b \\|^2 \\\\\n&\\quad -\\frac{\\rho}{2}\\|A_1\\overline{x}_1 +\\dots +A_{i-1}\\overline{x}_{i-1} +A_ix_i +A_{i+1}\\overline{x}_{i+1}+\\dots+A_m \\overline{x}_{m} +B\\overline{y} -b \\|^2 -\\frac{\\nu\\ell_{ \\phi}}{2}\\|x_i-\\overline{x}_i\\|^2 \\leq f_i(x_i),\n\\end{align*}\nwhich can be written as\n\\begin{align*}\nh_i(\\overline{x}_i) \\leq h_i(x_i),    \n\\end{align*}\nwhere $h_i(x_i) =f_i(x_i) +\\langle \\nabla_i P(\\overline{\\mathbf{x}}) -\\overline{g}_i, x_i \\rangle + \\langle \\overline{z}, A_ix_i \\rangle +\\frac{\\rho}{2}\\|A_1\\overline{x}_1 +\\dots +A_{i-1}\\overline{x}_{i-1} +A_ix_i +A_{i+1}\\overline{x}_{i+1}+\\dots+A_m \\overline{x}_{m} +B\\overline{y} -b \\|^2 +\\frac{\\nu\\ell_{\\phi}}{2}\\|x_i-\\overline{x}_i\\|^2$. It follows that, for all $i \\in \\{1,\\dots,m\\}$,\n\\begin{align*}\n0 &\\in \\partial_L h_i(\\overline{x}_i) =\\partial_L (f_i)(\\overline{x}_i) +\\nabla_i P(\\overline{\\mathbf{x}})- \\overline{g}_i +A_i^\\top \\overline{z} +\\rho A_i^\\top (\\mathbf{A}\\overline{\\mathbf{x}}+B\\overline{y}-b).\n\\end{align*}\nSince $\\mathbf{A}\\overline{\\mathbf{x}}+B\\overline{y}-b =0$, we obtain that\n\\begin{align*}\n0 \\in \\partial_L (f_1)(\\overline{x}_1)\\times \\cdots \\times \\partial_L (f_m)(\\overline{x}_m) +\\nabla P(\\overline{\\mathbf{x}}) -\\overline{g} +\\mathbf{A}^\\top \\overline{z},\n\\end{align*}\nwhich completes the proof due to Lemma~\\ref{l:cal}\\ref{l:cal_sum}\\&\\ref{l:cal_sep} and the strict differentiability of $G$.\n\\end{proof}\n\n\\begin{proof}\nFor each $n\\in \\mathbb{N}$, set $w_n=(\\mathbf{x}_{n+1},y_{n+1},z_{n+1},y_n)$ and $\\Delta_n =\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\| +\\|y_{n+2}-y_{n+1}\\|$. Let $n \\in \\mathbb{N}^*$. According to Theorem~\\ref{t:cvg}, we have that\n\\begin{align}\\label{eq:auxilary_decrease}\n\\mathcal{F}(w_{n+1}) +\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\|^2 + \\delta_y \\|y_{n+2}-y_{n+1}\\|^2 \\leq \\mathcal{F}(w_{n}),\n\\end{align}\nthat $w_{n+1} - w_n \\to 0$ as $n\\to +\\infty$, that $(w_n)_{n\\in\\mathbb{N}}$ is bounded, that any of its cluster point $\\overline{w}=(\\overline{\\mathbf{x}},\\overline{y},\\overline{z},\\overline{y})$ is a stationary point of \\eqref{eq:prob}, and also that $\\mathcal{F}(w_{n}) \\to \\mathcal{F}(\\overline{w}) = \\mathcal{L}_{\\rho}(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ as $n\\to +\\infty$. Notice that\n\\begin{align*}\n\\Delta_n^2 \\leq \\left(\\frac{1}{\\delta_{\\mathbf{x}}} +\\frac{1}{\\delta_y}\\right) \\left(\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\|^2 + \\delta_y \\|y_{n+2}-y_{n+1}\\|^2\\right),    \n\\end{align*}\nwhich together with \\eqref{eq:auxilary_decrease} yields\n\\begin{align*}\n\\mathcal{F}(w_{n+1}) +C_1\\Delta_n^2 \\leq \\mathcal{F}(w_{n}), \\text{~~where~} C_1 =\\frac{\\delta_{\\mathbf{x}}\\delta_y}{\\delta_{\\mathbf{x}}+\\delta_y}.\n\\end{align*}\n\nNext, we have that\n\\begin{align*}\n\\partial_L ^\\mathbf{x} \\mathcal{F}(w_n) = \\partial_L \\left( \\sum_{i=1}^{m} f_i\\right)(\\mathbf{x}_{n+1}) + \\nabla P(\\mathbf{x}_{n+1}) - \\nabla G(\\mathbf{x}_{n+1}) + \\mathbf{A}^\\top z_{n+1} + \\rho \\mathbf{A}^\\top(\\mathbf{Ax}_{n+1} + By_{n+1} -b)    \n\\end{align*}\nand, for all $i \\in \\{1,\\dots,m\\}$,\n\\begin{align*}\n\\partial_L ^{x_i} \\mathcal{F}(w_n) = \\partial_L f_i(x_{i,n+1})+\\nabla_i P(\\mathbf{x}_{n+1}) - \\nabla_i G(\\mathbf{x}_{n+1}) + A_i^\\top z_{n+1} + \\rho A_i^\\top(\\mathbf{Ax}_{n+1} + By_{n+1} -b).    \n\\end{align*}\nBy revoking the optimality condition for the update of $x_{i,n+1}$ in Step~\\ref{step:A2_2} of Algorithm~\\ref{algo:BPLA},\n\\begin{align*}\n0 &\\in \\partial_L f_i(x_{i,n+1}) +\\nabla_i P(\\mathbf{x}_{n}) -  \\nabla_i G(\\mathbf{x}_n) + A_i^\\top z_n   +\\rho A_i^\\top(\\mathbf{A}\\mathbf{u}_{i,n+1}(x_{i,n+1} ) +By_n -b ) \\\\\n&\\quad +\\mu_n(\\nabla \\phi_i (x_{i,n+1})- \\nabla \\phi_i (x_{i,n})),    \n\\end{align*}\nwhich combined with the above equality implies that\n\\begin{align*}\n&(\\nabla_i P(\\mathbf{x}_{n+1}) -\\nabla_i P(\\mathbf{x}_{n})) -(\\nabla_i G(\\mathbf{x}_{n+1}) -\\nabla_i G(\\mathbf{x}_n)) +A_i^\\top(z_{n+1}-z_n) -\\mu_n(\\nabla \\phi_i (x_{i,n+1}) -\\nabla \\phi_i (x_{i,n})) \\\\\n&\\quad +\\rho A_i^\\top (A_{i+1}x_{i+1,n+1} +\\dots +A_mx_{m,n+1} -A_{i+1}x_{i+1,n} -\\dots -A_mx_{m,n} +By_{n+1} -By_n) \\in \\partial_L ^{x_i} \\mathcal{F}(w_n).\n\\end{align*}\nTherefore,\n\\begin{align}\\label{eq:dxi}\n\\dist(0,\\partial_L ^{x_i} \\mathcal{F}(w_n)) \n&\\leq \\|\\nabla_i P(\\mathbf{x}_{n+1}) -\\nabla_i P(\\mathbf{x}_{n})\\| +\\|\\nabla_i G(\\mathbf{x}_{n+1}) -\\nabla_i G(\\mathbf{x}_n)\\| \\notag \\\\\n&\\quad +\\|A_i^\\top(z_{n+1}-z_n)\\| +\\mu_n\\|\\nabla \\phi_i (x_{i,n+1}) -\\nabla \\phi_i (x_{i,n})\\| \\notag \\\\\n&\\quad +\\rho \\|A_i^\\top (A_{i+1}x_{i+1,n+1} +\\dots +A_mx_{m,n+1} -A_{i+1}x_{i+1,n} -\\dots -A_mx_{m,n} +By_{n+1} -By_n\\| \\notag \\\\ \n&\\leq \\ell_P\\|\\mathbf{x}_{n+1} - \\mathbf{x}_{n}\\| +\\ell_G\\|\\mathbf{x}_{n} - \\mathbf{x}_{n+1}\\| +\\|A_i^\\top\\|\\|z_{n+1}-z_n\\| +\\mu_n \\ell_{\\phi}\\|x_{i,n+1}-x_{i,n}\\| \\notag \\\\\n&\\quad +\\rho\\sum_{j=i+1}^{m} \\|A_i^\\top\\| \\|A_j\\| \\|x_{j,n+1}-x_{j,n}\\| +\\rho\\|A_i^\\top\\| \\|B\\| \\|y_{n+1}-y_n\\| \\notag \\\\\n&\\leq  \\left(\\ell_P +\\ell_G +\\mu_n\\ell_{\\phi} +\\rho\\sum_{j=i+1}^{m} \\|A_i^\\top\\| \\|A_j\\|\\right)\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\| +\\|A_i^\\top\\|\\|z_{n+1}-z_n\\| \\notag \\\\\n&\\quad +\\rho\\|A_i^\\top\\| \\|B\\| \\|y_{n+1}-y_n\\|.\n\\end{align}\n\nWe now see that\n\\begin{align*}\n\\partial_L ^y \\mathcal{F}(w_n) =\\nabla H(y_{n+1}) + B^\\top z_{n+1} +\\rho B^\\top(\\mathbf{Ax}_{n+1} + By_{n+1} -b) +2c(y_{n+1}-y_{n})    \n\\end{align*}\nand from the optimality condition for the update of $y_{n+1}$ in Algorithm~\\ref{algo:BPLA} that \n\\begin{align*}\n0 &= \\nabla H(y_{n+1}) + B^\\top z_n +\\rho B^\\top(\\mathbf{Ax}_{n+1} +By_{n+1} -b) +\\nu_n(\\nabla \\psi(y_{n+1}) - \\nabla \\psi(y_n)).\n\\end{align*}\nThis leads to\n\\begin{align*}\n\\partial_L ^y \\mathcal{F}(w_n) &=B^{\\top}(z_{n+1}-z_n) -\\nu_n(\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)) + 2c(y_{n+1}-y_{n}),   \n\\end{align*}\nand so\n\\begin{align}\\label{eq:dy}\n\\dist(0,\\partial_L ^{y} \\mathcal{F}(w_n)) \n&\\leq \\|B^{\\top}(z_{n+1}-z_n)\\| +\\nu_n\\|\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)\\| + 2c\\|y_{n+1}-y_{n})\\| \\notag \\\\\n&\\leq \\|B^{\\top}(z_{n+1}-z_n)\\| +(\\nu\\ell_\\psi +2c)\\|y_{n+1}-y_{n})\\|.\n\\end{align}\n\nSince $\\partial_L ^z \\mathcal{F}(w_n) =\\mathbf{Ax}_{n+1} + By_{n+1} -b =\\frac{1}{\\rho}(z_{n+1} -z_n)$ and\n$\\partial_L ^t \\mathcal{F}(w_n) =-2c(y_{n+1}-y_{n})$, it holds that\n\\begin{align}\\label{eq:dzdt}\n\\dist(0,\\partial_L ^{z} \\mathcal{F}(w_n)) =\\frac{1}{\\rho}\\|z_{n+1}-z_n\\| \\text{~~and~~}\n\\dist(0,\\partial_L ^{t} \\mathcal{F}(w_n)) =2c\\|y_{n+1}-y_n\\|.    \n\\end{align}\nCombining \\eqref{eq:dxi}, \\eqref{eq:dy}, and \\eqref{eq:dzdt}, noting that $\\limsup_{n\\to +\\infty} \\mu_n =\\overline{\\mu} <+\\infty$, and using \\eqref{eq:BTZy} and \\eqref{eq:zy} in the proof of Theorem~\\ref{t:cvg}, we derive that there exists $n_0\\in \\mathbb{N}^*$ and $C_2 \\in \\mathbb{R}_{++}$ such that, for all $n\\geq n_0$, \n\n\\begin{align*}\n\\dist(0, \\partial_L \\mathcal{F}(w_n)) \n\\leq C_2 \\left(\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\| +\\|y_{n+1}-y_n\\| +\\|y_{n}-y_{n-1}\\|\\right) \n\\leq C_2(\\Delta_{n-1} +\\Delta_{n-2}).\n\\end{align*}\n\n\\ref{t:full_cvg}: We see that all the conditions in the \\emph{abstract convergence} framework \\cite[Theorem~5.1]{BDL22} are satisfied with $I =\\{1,2\\}$, $\\lambda_1=\\lambda_2=1/2$, $\\alpha_n \\equiv C_1$, $\\beta_n \\equiv 1/(2C_2)$, and $\\varepsilon_n \\equiv 0 $. By \\cite[Theorem~5.1(i)]{BDL22}, \n\\begin{align*}\n\\sum_{n=0}^{+\\infty} (\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\|+\\|y_{n+2}-y_{n+1}\\|) =\\sum_{n=0}^{+\\infty} \\Delta_n < +\\infty,\n\\end{align*}\nwhich implies that $\\sum_{n=0}^{+\\infty} \\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\| <+\\infty$ and $\\sum_{n=0}^{+\\infty} \\|y_{n+1}-y_{n}\\| < +\\infty$. Together with \\eqref{eq:zy}, we derive that $\\sum_{n=0}^{+\\infty} \\|z_{n+1}-z_n\\|<+\\infty$, and hence \n\\begin{align*}\n\\sum_{n=0}^{+\\infty} \\|(\\mathbf{x}_{n+1},y_{n+1},z_{n+1})-(\\mathbf{x}_{n},y_{n},z_{n})\\| < +\\infty,\n\\end{align*}\nwhich yields the convergence of $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ to $(\\mathbf{x}^{*},y^{*},z^{*})$. In view of Theorem~\\ref{t:cvg}\\ref{t:cvg_seq}, $(\\mathbf{x}^{*},y^{*},z^{*})$ is a stationary point of $\\mathcal{L}_\\rho$.\n\n\\ref{t:full_rate_finite}: This follows from the arguments in the proof of \\cite[Theorem~5.1]{BDL22} and \\cite[Theorem~2(i)]{Attouch2007}. \n\n\\ref{t:full_rate_linear}: According to \\cite[Theorem~5.1(iv)]{BDL22}, there exist $\\gamma_0\\in \\mathbb{R}_{++}$ and $\\zeta\\in \\left(0,1\\right)$ such that, for all $n\\in \\mathbb{N}^*$, \n\\begin{align*}\n\\|(\\mathbf{x}_{n},y_{n})-(\\mathbf{x}^{*},y^{*})\\|\\leq\\gamma_0 \\zeta^{\\frac{n}{2}} \\text{~~and~~}\n|\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*)| \\leq \\gamma_0\\zeta^n.\n\\end{align*}\nWe derive that, for all $n\\in \\mathbb{N}^*$, $\\|\\mathbf{x}_n-\\mathbf{x}^{*}\\|\\leq \\gamma_0\\zeta^{\\frac{n}{2}}$ and $\\|y_n-y^{*}\\|\\leq \\gamma_0\\zeta^{\\frac{n}{2}}$, which yields \n\\begin{align*}\n\\|y_n-y_{n-1}\\| \\leq \\|y_n-y^{*}\\|+\\|y_{n-1}-y^{*}\\| \\leq (1+\\zeta^{-\\frac{1}{2}})\\gamma_0\\zeta^{\\frac{n}{2}}.    \n\\end{align*}\nBy passing to the limit in \\eqref{eq:B'z}, $B^\\top z^{*}=-\\nabla H(y^{*})$. Following the same steps as in \\eqref{eq:BTZy} and \\eqref{eq:zy}, we obtain that, for all $n\\in \\mathbb{N}^*$,\n\\begin{align*}\n\\|z_n-z^{*}\\| &\\leq \\frac{\\ell_H}{\\sqrt{\\lambda}}\\|y_n-y^{*}\\| + \\frac{\\nu \\ell_{\\psi}}{\\sqrt{\\lambda}}\\|y_{n}-y_{n-1}\\|\\leq \\gamma_1\\zeta^{\\frac{n}{2}}, \\text{~~where~} \\gamma_1 :=\\frac{\\ell_H\\gamma_0+(1+\\zeta^{-\\frac{1}{2}})\\nu\\ell_{\\psi}\\gamma_0}{\\sqrt{\\lambda}}.\n\\end{align*}\nConsequently,\n\\begin{align*}\n&\\|(\\mathbf{x}_{n},y_{n},z_{n})-(\\mathbf{x}^{*},y^{*},z^{*})\\| \\leq \\|\\mathbf{x}_{n}-\\mathbf{x}^{*}\\| + \\|y_{n}-y^{*}\\| + \\|z_{n}-z^{*}\\|\\leq (2\\gamma_0 +\\gamma_1)\\zeta^{\\frac{n}{2}}, \\\\\n&\\|\\mathbf{Ax}_n+By_n-b\\| =\\frac{1}{\\rho}\\|z_n-z_{n-1}\\| \\leq \\frac{1}{\\rho}(\\|z_n-z^{*}\\| +\\|z_{n-1}-z^{*}\\|) \\leq \\frac{(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho} \\zeta^{\\frac{n}{2}}, \\\\\n\\text{and~} &\\|z_n\\| \\leq \\|z_n-z^{*}\\| + \\|z^{*}\\| \\leq \\gamma_1 \\zeta^{\\frac{n}{2}} + \\|z^{*}\\|.\n\\end{align*}\n\nWe now deduce from the definition of $\\mathcal{F}$ that\n\\begin{align*}\n|\\mathcal{L}_{\\rho}(\\mathbf{x}_{n},y_{n},z_{n})- \\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})|&= \\left|\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*) - c\\|y_n-y_{n-1}\\|^2\\right|\\\\\n&\\leq |\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*)| + c\\|y_n-y_{n-1}\\|^2\\\\\n&\\leq \\gamma_2\\zeta^n, \\text{~~where~} \\gamma_2 :=\\gamma_0 + c(1+\\zeta^{-\\frac{1}{2}})^2\\gamma_0^2\n\\end{align*}\nand from the definition of $\\mathcal{L}_\\rho$ that \n\\begin{align*}\n|F(\\mathbf{x}_n,y_n)-F(\\mathbf{x}^{*},y^{*})| &= |\\mathcal{L}_{\\rho}(\\mathbf{x}_n,y_n,z_n)-\\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})-\\langle z_n, \\mathbf{Ax}_n + By_n-b\\rangle - \\frac{\\rho}{2}\\|\\mathbf{Ax}_n + By_n-b\\|^2| \\\\\n&\\leq |\\mathcal{L}_{\\rho}(\\mathbf{x}_n,y_n,z_n)-\\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})| + \\|z_n\\|\\|\\mathbf{Ax}_n + By_n-b\\| + \\frac{\\rho}{2}\\|\\mathbf{Ax}_n + By_n-b\\|^2  \\\\ \n&\\leq \\gamma_2\\zeta^n + \\left(\\gamma_1 \\zeta^{\\frac{n}{2}} + \\|z^{*}\\|\\right)\\frac{(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho} \\zeta^{\\frac{n}{2}}+ \\frac{(1+\\zeta^{-\\frac{1}{2}})^2\\gamma_1^2}{2\\rho}\\zeta^n \\\\\n&\\leq \\left(\\gamma_2 +\\frac{(3+\\zeta^{-\\frac{1}{2}})(1+\\zeta^{-\\frac{1}{2}})\\gamma_1^2}{2\\rho} +\\frac{\\|z^{*}\\|(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho}\\right)\\zeta^{\\frac{n}{2}},\n\\end{align*}\nwhere the last inequality follows from the fact that $\\zeta^n \\leq \\zeta^{\\frac{n}{2}}$ since $\\zeta \\in (0,1)$. By letting \n\\begin{align*}\n\\overline{\\gamma} =\\max \\left\\{ 2\\gamma_0+\\gamma_1, \\frac{(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho}, \\gamma_2 +\\frac{(3+\\zeta^{-\\frac{1}{2}})(1+\\zeta^{-\\frac{1}{2}})\\gamma_1^2}{2\\rho} +\\frac{\\|z^{*}\\|(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho} \\right\\} \n\\end{align*}\nand increasing it if necessary, we arrive at the conclusion.\n\n\\ref{t:full_rate_sublinear}: Following the arguments in \\cite[Theorem~5.1(iv)]{BDL22} and \\cite[Theorem~2(iii)]{Attouch2007}, we find $\\gamma_0 \\in \\mathbb{R}_{++}$ such that, for all $n\\in \\mathbb{N}^*$, \n\\begin{align*}\n\\|(\\mathbf{x}_{n},y_{n})-(\\mathbf{x}^{*},y^{*})\\|\\leq \\gamma_0 n^{-\\frac{1-\\kappa}{2\\kappa-1}} \\text{~~and~~}\n|\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*)| \\leq \\gamma_0 n^{-\\frac{2-2\\kappa}{2\\kappa-1}}.\n\\end{align*}\nSimilar to \\ref{t:full_rate_linear}, for all $n\\geq 2$, since $n-1\\geq \\frac{1}{2}n$ and $\\frac{1-\\kappa}{2\\kappa-1} >0$, we derive that \n\\begin{align*}\n&\\|(\\mathbf{x}_{n},y_{n},z_{n})-(\\mathbf{x}^{*},y^{*},z^{*})\\| \\leq (2\\gamma_0+\\gamma_1) n^{-\\frac{1-\\kappa}{2\\kappa-1}},\\\\\n&\\|\\mathbf{Ax}_n+By_n-b\\|\\leq \\frac{(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1}{\\rho} n^{-\\frac{1-\\kappa}{2\\kappa-1}},\\\\\n&|\\mathcal{L}_{\\rho}(\\mathbf{x}_{n},y_{n},z_{n})- \\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})|\\leq \\gamma_2 n^{-\\frac{2-2\\kappa}{2\\kappa-1}},\n\\end{align*}\nand that\n\\begin{align*}\n|F(\\mathbf{x}_n,y_n)-F(\\mathbf{x}^{*},y^{*})| &\\leq \\gamma_2 n^{-\\frac{2-2\\kappa}{2\\kappa-1}}+ \\left(\\gamma_1 n^{-\\frac{1-\\kappa}{2\\kappa-1}} + \\|z^{*}\\|\\right)\\frac{(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1}{\\rho} n^{-\\frac{1-\\kappa}{2\\kappa-1}} + \\frac{(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})^2\\gamma_1^2}{2\\rho} n^{-\\frac{2-2\\kappa}{2\\kappa-1}}\\\\\n&\\leq \\left(\\gamma_2 +\\frac{(3 +2^{\\frac{1-\\kappa}{2\\kappa-1}})(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1^2}{2\\rho} +\\frac{\\|z^{*}\\|(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1}{\\rho} \\right)n^{-\\frac{1-\\kappa}{2\\kappa-1}},\n\\end{align*}\nwhere $\\gamma_1 :=\\frac{\\ell_H\\gamma_0 +(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\nu\\ell_{\\psi}\\gamma_0}{\\sqrt{\\lambda}}$, $\\gamma_2 :=\\gamma_0 +c(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})^2\\gamma_0^2$, and the last inequality follows from the fact that $n^{-\\frac{2-2\\kappa}{2\\kappa-1}}<n^{-\\frac{1-\\kappa}{2\\kappa-1}}$ for $\\kappa\\in(\\frac{1}{2},1)$. By setting $\\overline{\\gamma}$ in the same way as in \\ref{t:full_rate_linear}, we obtain the conclusion.\n\\end{proof}","units":[{"raw_tex":"\\begin{lemma}[Calculus rules]\n\\label{l:cal}\nLet $f, g\\colon \\mathbb{R}^d \\to (-\\infty, +\\infty]$ be proper lower semicontinuous functions, and let $x \\in \\mathbb{R}^d$. \n\\begin{enumerate}\n\\item\\label{l:cal_sum} \n(Sum rule) Suppose that $f$ is finite at $x$ and $g$ is locally Lipschitz around $\\overline{x}$. Then $\\partial_L (f + g)(\\overline{x}) \\subseteq \\partial_L f(\\overline{x})+\\partial_L g(\\overline{x})$. Moreover, if $g$ is strictly differentiable at $\\overline{x}$, then $\\partial_L(f + g)(\\overline{x}) = \\partial_Lf(\\overline{x}) + \\nabla g(\\overline{x})$.\n\\item\\label{l:cal_sep} \n(Separable sum rule) If $f(\\mathbf{x})=\\sum_{i=1}^{m}f_i(x_i)$ with $\\mathbf{x}=(x_1,\\dots,x_m)$, then $\\partial_L f(\\mathbf{x}) = \\partial_L f_1(x_1) \\times \\dots \\times \\partial_L f_m(x_m)$. \n\\end{enumerate}\n\\end{lemma}","start":16647,"end":17462},{"raw_tex":"\\begin{proof}\n\\ref{l:cal_sum} follows from \\cite[Proposition~1.107(ii) and Theorem~3.36]{Mordukhovich2006}, while \\ref{l:cal_sep} follows from \\cite[Proposition~10.5]{Rockafellar1998}.\n\\end{proof}","start":17463,"end":17659},{"raw_tex":"\\begin{lemma}[Upper semicontinuity of subdifferential]\n\\label{l:upsemicont}\nLet $f\\colon \\mathbb{R}^d \\to [-\\infty, +\\infty]$ be Lipschitz continuous around $x \\in \\mathbb{R}^d$ with $|f(x)| <+\\infty$, and consider sequences $(x_n)_{n\\in \\mathbb{N}}$ and $(x_n^*)_{n\\in \\mathbb{N}}$ in $\\mathbb{R}^d$ such that $x_n \\to x$ and, for all ${n\\in \\mathbb{N}}$, $x_n^* \\in \\partial_L f(x_n)$. \nThen $(x_n^*)_{n\\in \\mathbb{N}}$ is bounded and its cluster points are contained in $\\partial_L f(x)$.\n\\end{lemma}","start":17662,"end":18165},{"raw_tex":"\\begin{proof}\nThis follows from \\cite[Lemma 2.2]{Tan2023}.\n\\end{proof}","start":18166,"end":18236},{"raw_tex":"\\begin{proposition}\n\\label{p:Bregman}\nLet $\\phi: \\mathcal{H} \\to \\mathbb{R}$ be a differentiable convex function and $D_{\\phi}$ is the Bregman distance associated with $\\phi$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{p:Bregman_1} \nFor all $u, v \\in \\mathcal{H}$, $D_{\\phi}(u, v)\\geq 0$, and $D_{\\phi}(u, v)=0$ if and only if $u = v$. \n\n\\item\\label{p:Bregman_2} \nFor each $v \\in \\mathcal{H}$, $D_{\\phi}(\\cdot, v)$ is convex. \n\n\\item\\label{p:Bregman_3} \nIf $\\phi$ is $\\alpha$-strongly convex,  then, for all $u, v \\in \\mathcal{H}$, $D_{\\phi}(u, v) \\geq \\frac{\\alpha}{2} \\|u-v\\|^2$. \n\n\\item\\label{p:Bregman_4} \nIf $\\nabla \\phi$ is $\\ell_{\\phi}$-Lipschitz continuous, then, for all $u, v \\in \\mathcal{H}$, $D_{\\phi}(u,v)\\leq \\frac{\\ell_{\\phi}}{2}\\|u-v\\|^2$. \n\\end{enumerate}\n\\end{proposition}","start":19038,"end":19840},{"raw_tex":"\\begin{proof}\n\\ref{p:Bregman_1} is given in \\cite[Section 1]{Bregman}. \\ref{p:Bregman_2}, \\ref{p:Bregman_3}, and \\ref{p:Bregman_4} can be easily verified by using the definitions. \n\\end{proof}","start":19841,"end":20033},{"raw_tex":"\\begin{lemma}\n\\label{l:diff}\nLet $h\\colon \\mathbb{R}^d\\to \\mathbb{R}$ be a differentiable function with $\\ell$-Lipschitz continuous gradient. Then the following hold:\n\\begin{enumerate}\n\\item\\label{l:diff_descent}\nFor all $x, y\\in \\mathbb{R}^d$, $|h(y) -h(x) -\\langle \\nabla h(x), y -x\\rangle|\\leq \\frac{\\ell}{2}\\|y -x\\|^2$.\n\\item\\label{l:diff_bounded}\nIf $h$ is bounded below and $\\ell >0$, then\n\\begin{align*}\n\\inf_{x\\in \\mathbb{R}^d} \\left(h(x) -\\frac{1}{2\\ell}\\|\\nabla h(x)\\|^2\\right) >-\\infty.    \n\\end{align*}\n\\end{enumerate}  \n\\end{lemma}","start":20164,"end":20708},{"raw_tex":"\\begin{proof}\n\\ref{l:diff_descent}: This follows from \\cite[Lemma~1.2.3]{Nesterov2018}.\n\n\\ref{l:diff_bounded}: Let $x\\in \\mathbb{R}^d$. Following the argument of \\cite[Equation~(1.2.19)]{Nesterov2018}, we apply \\ref{l:diff_descent} with $y =x -\\frac{1}{\\ell}\\nabla h(x)$ to obtain that\n\\begin{align*}\nh\\left(x -\\frac{1}{\\ell}\\nabla h(x)\\right) &\\leq h(x) +\\langle \\nabla h(x), y -x\\rangle +\\frac{\\ell}{2}\\|y -x\\|^2 \\\\\n&= h(x) -\\frac{1}{\\ell}\\|\\nabla h(x)\\|^2 +\\frac{1}{2\\ell}\\|\\nabla h(x)\\|^2 \\\\\n&= h(x) -\\frac{1}{2\\ell}\\|\\nabla h(x)\\|^2,\n\\end{align*}\nwhich together with the boundedness of $h$ implies the conclusion.\n\\end{proof}","start":20709,"end":21339},{"raw_tex":"\\begin{theorem}[Subsequential convergence]\n\\label{t:cvg}\nSuppose that Assumption~\\ref{a:standing} holds. Let $(\\mathbf{x}_n,y_n,z_n)_{n\\in \\mathbb{N}}$ be the sequence generated by Algorithm~\\ref{algo:BPLA} and, for each $n\\in \\mathbb{N}^*$, define\n\\begin{align*}\nL_n =\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) +c\\|y_n-y_{n-1}\\|^2, \\text{~where~} c=\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}.     \n\\end{align*}\nThen the following hold:\n\\begin{enumerate}\n\\item\\label{t:cvg_decrease}\nFor all $n\\in \\mathbb{N}^*$,\n\\begin{align}\\label{eq:decrease}\nL_{n+1} +\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\delta_y\\|y_{n+1}-y_n\\|^2 \\leq L_n,\n\\end{align}\nwhere $\\delta_{\\mathbf{x}} =\\frac{\\mu\\alpha -\\ell_P -\\beta}{2} > 0$ and $\\delta_y =\\frac{\\lambda\\rho}{2} -\\frac{(\\ell_H+2\\nu\\ell_{\\psi})^2}{\\lambda\\rho} -\\frac{\\ell_H}{2} >0$.\n\n\\item\\label{t:cvg_seq}\nSuppose that $z_0\\in \\Ima(B)$, that $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, and that $H$ is bounded below. \nThen the sequence $(\\mathbf{x}_n,{y}_n,{z}_n)_{n\\in \\mathbb{N}}$ is bounded and $\\mathbf{x}_{n+1}-\\mathbf{x}_n\\to 0$, $y_{n+1}-y_n\\to 0$, $z_{n+1}-z_n\\to 0$, and $\\mathbf{Ax}_n +By_n \\to b$ as $n\\to +\\infty$. Furthermore, for every cluster point $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ of $(\\mathbf{x}_n,{y}_n,{z}_n)_{n\\in \\mathbb{N}}$, it holds that\n\\begin{align*}\n\\mathbf{A}\\overline{\\mathbf{x}} +b\\overline{y} =b \\text{~~and~~} \\lim_{n \\to +\\infty} F(\\mathbf{x}_{n },y_{n}) =\\lim_{n \\to +\\infty}\\mathcal{L}_\\rho(\\mathbf{x}_{n },y_{n},z_{n}) =\\mathcal{L}_\\rho(\\overline{\\mathbf{x}},\\overline{y},\\overline{z}) =F(\\overline{\\mathbf{x}},\\overline{y}),    \n\\end{align*}\nand that if $G$ is strictly differentiable and, for each $i \\in \\{1,\\dots,m\\}$, $\\nabla \\phi_i$ is $\\ell_{\\phi}$-Lipschitz continuous, then $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ is a stationary point of $\\mathcal{L}_\\rho$.\n\\end{enumerate}\n\\end{theorem}","start":30177,"end":32134},{"raw_tex":"\\begin{proof}\n\\ref{t:cvg_decrease}: Let $n\\in \\mathbb{N}$. The update of $z_{n+1}$ yields\n\\begin{align}\\label{eq:z-update}\nz_{n+1} -z_n =\\rho(\\mathbf{Ax}_{n+1}+By_{n+1}-b).   \n\\end{align}\nBy the optimality condition for the update of $y_{n+1}$ in Step~\\ref{step:A2_2} of Algorithm~\\ref{algo:BPLA}, \n\\begin{align*}\n0 &= \\nabla H(y_{n+1}) + B^\\top z_n +\\rho B^\\top(\\mathbf{Ax}_{n+1} +By_{n+1} -b) +\\nu_n(\\nabla \\psi(y_{n+1}) - \\nabla \\psi(y_n)),\n\\end{align*}\nwhich combined with \\eqref{eq:z-update} yields\n\\begin{align}\\label{eq:B'z}\nB^\\top z_{n+1} =-\\nabla H(y_{n+1}) -\\nu_n(\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)).  \n\\end{align}\nFrom the definition of the augmented Lagrangian, we have that\n\\begin{align}\\label{eq:zz}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n}) &= \\langle z_{n+1}-z_n, \\mathbf{Ax}_{n+1}+By_{n+1}-b \\rangle \\notag\\\\\n&= \\frac{1}{\\rho}\\|z_{n+1}-z_n\\|^2 \n\\end{align}\nand that\n\\begin{align*}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_n,z_n) &= H(y_{n+1}) - H(y_n) + \\langle z_n, B(y_{n+1}-y_n) \\rangle \\\\\n&\\qquad + \\frac{\\rho}{2}(\\|\\mathbf{Ax}_{n+1}+By_{n+1}-b\\|^2 - \\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2).  \n\\end{align*}\nWriting $\\mathbf{Ax}_{n+1}+By_{n}-b =(\\mathbf{Ax}_{n+1}+By_{n+1}-b) -(By_{n+1}- By_n)$, we obtain that\n\\begin{align*}\n&\\|\\mathbf{Ax}_{n+1}+By_{n+1}-b\\|^2 - \\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2 \\\\\n&= -\\|B(y_{n+1}-y_n)\\|^2 +2\\langle  \\mathbf{Ax}_{n+1}+By_{n+1}-b, B(y_{n+1}-y_n)\\rangle \\\\\n&= -\\|B(y_{n+1}-y_n)\\|^2 +\\frac{2}{\\rho}\\langle  z_{n+1}-z_{n}, B(y_{n+1}-y_n)\\rangle,\n\\end{align*}\nand so\n\\begin{align*}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_n,z_n) = H(y_{n+1}) - H(y_n) +\\langle z_{n+1}, B(y_{n+1}-y_n) \\rangle -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2.\n\\end{align*}\nIn view of \\eqref{eq:B'z},\n\\begin{align*}\n\\langle z_{n+1}, B(y_{n+1}-y_n) \\rangle &=\\langle B^\\top z_{n+1}, y_{n+1}-y_n \\rangle \\\\\n&=-\\langle \\nabla H(y_{n+1}), y_{n+1}-y_n \\rangle -\\nu_n\\langle \\nabla \\psi(y_{n+1})- \\nabla \\psi(y_n), y_{n+1}-y_n \\rangle \\\\\n&\\leq -H(y_{n+1}) +H(y_n) +\\frac{\\ell_H}{2}\\|y_{n+1}-y_n\\|^2,\n\\end{align*}\nwhere the last inequality is from the $\\ell_H$-Lipschitz continuity of $\\nabla H$ and Lemma~\\ref{l:diff}\\ref{l:diff_descent}, and from the monotonicity of $\\nabla \\psi$. \nWe therefore obtain that\n\\begin{align}\\label{eq:yy}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_n,z_n) \\leq \\frac{\\ell_H}{2}\\|y_{n+1}-y_n\\|^2 -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2.    \n\\end{align}\nNext, by the update of $x_{i,n+1}$ in Step~\\ref{step:A2_2} of Algorithm~\\ref{algo:BPLA}, for all $i \\in \\{1,\\dots,m\\}$ and all $x_i \\in \\mathbb{R}^{d_i}$,\n\\begin{align}\\label{eq:update_step_x}\n&f_i(x_{i,n+1}) +\\langle \\nabla_i P(\\mathbf{x}_n) -g_{i,n}, x_{i,n+1} \\rangle +\\langle z_n, A_{i}x_{i,n+1} \\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,n+1}(x_{i,n+1}) +By_n -b\\|^2 +\\mu_n D_{\\phi_i}(x_{i,n+1},x_{i,n}) \\notag \\\\\n&\\leq  f_i(x_i) +\\langle \\nabla_i P(\\mathbf{x}_n) -g_{i,n}, x_i \\rangle +\\langle z_n, A_{i}x_i\\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,n+1}(x_i) +By_n -b\\|^2 +\\mu_n D_{\\phi_i}(x_i,x_{i,n}).\n\\end{align}\nLetting $x_i =x_{i,n}$, using Proposition~\\ref{p:Bregman}\\ref{p:Bregman_1}\\&\\ref{p:Bregman_3}, and summing up over $i \\in \\{1,\\dots,m\\}$, we derive that\n\\begin{align}\\label{eq:from x-update}\n&\\sum_{i=1}^m f_i(x_{i,n+1}) +\\langle \\nabla P(\\mathbf{x}_n) -g_n, \\mathbf{x}_{n+1} \\rangle +\\langle z_n, \\mathbf{Ax}_{n+1} \\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{x}_{n+1} +By_n -b\\|^2 +\\frac{\\mu\\alpha}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 \\notag\\\\\n&\\leq \\sum_{i=1}^m f_i(x_{i,n}) +\\langle \\nabla P(\\mathbf{x}_n) -g_n, \\mathbf{x}_{n} \\rangle +\\langle z_n, \\mathbf{Ax}_{n} \\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{x}_n +By_n -b\\|^2.\n\\end{align}\nOn the other hand,\n\\begin{align}\\label{eq:xx}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_n,z_n) \\notag\\\\\n&= \\sum_{i=1}^m f_i(x_{i,n+1}) -\\sum_{i=1}^m f_i(x_{i,n}) +P(\\mathbf{x}_{n+1}) -P(\\mathbf{x}_{n}) -G(\\mathbf{x}_{n+1}) +G(\\mathbf{x}_{n}) \\notag\\\\ \n&\\quad +\\langle z_n, \\mathbf{Ax}_{n+1}-\\mathbf{Ax}_{n}\\rangle +\\frac{\\rho}{2}\\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2 -\\frac{\\rho}{2}\\|\\mathbf{Ax}_{n}+By_{n}-b\\|^2 \\notag\\\\\n&\\leq \\sum_{i=1}^m f_i(x_{i,n+1}) -\\sum_{i=1}^m f_i(x_{i,n}) +\\langle \\nabla P(\\mathbf{x}_n), \\mathbf{x}_{n+1}-\\mathbf{x}_n \\rangle + \\frac{\\ell_P}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_n\\|^2 - \\langle g_n, \\mathbf{x}_{n+1} -\\mathbf{x}_{n} \\rangle + \\frac{\\beta}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 \\notag\\\\\n&\\quad +\\langle z_n, \\mathbf{Ax}_{n+1}-\\mathbf{Ax}_{n}\\rangle  + \\frac{\\rho}{2}\\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2 - \\frac{\\rho}{2}\\|\\mathbf{Ax}_{n}+By_{n}-b\\|^2 \\notag\\\\\n&\\leq -\\frac{\\mu\\alpha -\\ell_P -\\beta}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 =-\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2,\n\\end{align}\nwhere the first inequality is obtained by using Lemma~\\ref{l:diff}\\ref{l:diff_descent} on the $\\ell_P$-Lipschitz continuity of $\\nabla P$ and \\cite[Lemma~4.1]{BDL22} on the $\\beta$-weak convexity of $G$, while the last inequality follows from \\eqref{eq:from x-update}. Summing up three relations \\eqref{eq:zz}, \\eqref{eq:yy}, and \\eqref{eq:xx} yields\n\\begin{align}\\label{eq:LL}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) - \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) \\notag\\\\\n&\\leq -\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\frac{\\ell_H}{2}\\|y_{n+1}-y_n\\|^2 -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2 +\\frac{1}{\\rho}\\|z_{n+1}-z_n\\|^2.\n\\end{align}\n\nNow, since $\\Ima(\\mathbf{A}) \\cup \\{b\\} \\subseteq \\Ima(B)$, it follows from \\eqref{eq:z-update} that\n\\begin{align}\\label{eq:zImB}\nz_{n+1}-z_n \\in \\Ima(B)    \n\\end{align}\nwhich, by Lemma~\\ref{l:norm}\\ref{l:norm_M'x}, yields \n\\begin{align*}\n\\|z_{n+1} -z_n\\| \\leq \\frac{1}{\\sqrt{\\lambda}}\\|B^\\top z_{n+1} -B^\\top z_n\\|.\n\\end{align*}\nLet $n\\in \\mathbb{N}^*$. We derive from \\eqref{eq:B'z} and the Lipschitz continuity of $\\nabla H$ and $\\nabla \\psi$ that\n\\begin{align}\\label{eq:BTZy}\n\\|B^\\top z_{n+1} -B^\\top z_n\\| &\\leq \\|\\nabla H(y_{n+1}) -\\nabla H(y_n)\\| +\\nu_n\\|\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)\\| +\\nu_{n-1}\\|\\nabla \\psi(y_n) - \\nabla \\psi(y_{n-1})\\| \\notag \\\\ \n&\\leq (\\ell_H +\\nu\\ell_{\\psi})\\|y_{n+1} -y_n\\| +\\nu\\ell_{\\psi}\\|y_n -y_{n-1}\\|,\n\\end{align}\nand thus, \n\\begin{align}\\label{eq:zy}\n\\|z_{n+1} -z_n\\| \\leq \\frac{1}{\\sqrt{\\lambda}}\\|B^\\top z_{n+1} -B^\\top z_n\\| \\leq \\frac{1}{\\sqrt{\\lambda}}((\\ell_H +\\nu\\ell_{\\psi})\\|y_{n+1} -y_n\\| +\\nu\\ell_{\\psi}\\|y_n -y_{n-1}\\|).\n\\end{align}\nApplying Cauchy--Schwarz inequality to two vectors $u =(\\sqrt{\\ell_H+\\nu\\ell_{\\psi}},\\sqrt{\\nu\\ell_{\\psi}})$ and $v =(\\sqrt{\\ell_H+\\nu\\ell_{\\psi}}\\|y_{n+1} -y_n\\|,\\sqrt{\\nu\\ell_{\\psi}}\\|y_n -y_{n-1}\\|)$, we obtain that\n\\begin{align*}\n\\|z_{n+1} -z_n\\|^2 \\leq \\frac{\\ell_H +2\\nu\\ell_{\\psi}}{\\lambda}((\\ell_H +\\nu\\ell_{\\psi})\\|y_{n+1} -y_n\\|^2 +\\nu\\ell_{\\psi}\\|y_n -y_{n-1}\\|^2).    \n\\end{align*}\nBy combining this with \\eqref{eq:LL} and using Lemma~\\ref{l:norm}\\ref{l:norm_Mx},\n\\begin{align*}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) - \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) \\notag\\\\\n&\\leq -\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\left(\\frac{(\\ell_H+2\\nu\\ell_{\\psi})(\\ell_H+\\nu\\ell_{\\psi})}{\\lambda\\rho} +\\frac{\\ell_H}{2}\\right)\\|y_{n+1}-y_n\\|^2 \\\\ \n&\\qquad -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2 +\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2 \\\\\n&\\leq -\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 -\\left(\\frac{\\lambda\\rho}{2} -\\frac{(\\ell_H+2\\nu\\ell_{\\psi})(\\ell_H+\\nu\\ell_{\\psi})}{\\lambda\\rho} -\\frac{\\ell_H}{2}\\right)\\|y_{n+1}-y_n\\|^2 \\\\\n&\\qquad +\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2,\n\\end{align*}\nor equivalently,\n\\begin{align*}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) +\\frac{(\\ell_H +2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_{n+1}-y_{n}\\|^2 + \\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\delta_y\\|y_{n+1}-y_n\\|^2 \\\\\n&\\leq \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) +\\frac{(\\ell_H +2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2,\n\\end{align*}\nwhich proves \\eqref{eq:decrease}. Here, we note that $\\delta_{\\mathbf{x}} =\\frac{\\mu\\alpha -\\ell_P -\\beta}{2} > 0$ since $\\mu > \\frac{\\ell_P+\\beta}{\\alpha}$ and that $\\delta_y =\\frac{\\lambda\\rho}{2} -\\frac{(\\ell_H+2\\nu\\ell_{\\psi})^2}{\\lambda\\rho} -\\frac{\\ell_H}{2} >0$ since $\\rho > \\frac{\\ell_H +\\sqrt{\\ell_H^2 +8(\\ell_H +2\\nu\\ell_{\\psi})^2}}{2\\lambda}$.\n\n\\ref{t:cvg_seq}: Let $n\\in \\mathbb{N}^*$. As $z_0 \\in \\Ima(B)$, we have from \\eqref{eq:zImB} that $z_n \\in \\Ima (B)$. Using Lemma~\\ref{l:norm}\\ref{l:norm_M'x}, \\eqref{eq:B'z}, and the Lipschitz continuity of $\\psi$ yields\n\\begin{align*}\n\\sqrt{\\lambda}\\|z_n\\|\\leq \\|B^\\top z_n\\| \\leq\\|\\nabla H(y_n)\\|+ \\nu_n\\|\\nabla \\psi(y_{n}) -\\nabla \\psi(y_{n-1})\\| \n\\leq \\|\\nabla H(y_n)\\|+ \\nu \\ell_{\\psi}\\|y_{n}-y_{n-1}\\|.\n\\end{align*}\nFor $\\omega\\in \\mathbb{R}_{++}$ to be chosen later, invoking Cauchy--Schwarz inequality with two vectors $u =(\\sqrt{\\omega},\\sqrt{\\nu\\ell_{\\psi}})$ and $v =(\\frac{1}{\\sqrt{\\omega}}\\|\\nabla H(y_n)\\|,\\sqrt{\\nu\\ell_{\\psi}}\\|y_n-y_{n-1}\\|)$, we have that\n\\begin{align}\\label{eq:uppper_zn}\n\\|z_n\\|^2 &\\leq \\frac{\\omega +\\nu\\ell_{\\psi}}{\\lambda} \\left(\\frac{1}{\\omega}\\|\\nabla H(y_n)\\|^2 +\\nu\\ell_{\\psi}\\|y_n-y_{n-1}\\|^2\\right) \\notag \\\\\n&= \\frac{1}{\\lambda}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right)\\|\\nabla H(y_n)\\|^2 +\\frac{(\\omega +\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda}\\|y_n-y_{n-1}\\|^2.\n\\end{align}\nCombining this with the fact that\n\\begin{align*}\n\\langle z_n, \\mathbf{Ax}_n+By_n-b \\rangle +\\frac{\\rho}{2}\\|\\mathbf{Ax}_n+By_n-b\\|^2 =\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} +\\mathbf{Ax}_n+By_n-b \\right\\|^2 -\\frac{1}{2\\rho}\\|z_n\\|^2,\n\\end{align*}\nwe deduce that\n\\begin{align*}\nL_n &=\\mathcal{L}_\\rho(\\mathbf{x}_n,y_n,z_n) + c\\|y_n-y_{n-1}\\|^2 \\\\\n&\\geq \\sum_{i=1}^m f_i(x_{i,n}) +H(y_n) +P(\\mathbf{x}_n) -G(\\mathbf{x}_n) \\\\ \n&\\quad +\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} + \\mathbf{Ax}_n+By_n-b \\right\\|^2 -\\frac{1}{2\\lambda\\rho}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right)\\|\\nabla H(y_n)\\|^2 +\\left(c -\\frac{(\\omega +\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{2\\lambda\\rho}\\right)\\|y_n-y_{n-1}\\|^2 \\\\\n&= \\left(\\sum_{i=1}^m f_i(x_{i,n}) +P(\\mathbf{x}_n) -G(\\mathbf{x}_n)\\right) +\\left(H(y_n) -\\frac{1}{2\\ell_H}\\|\\nabla H(y_n)\\|^2\\right) +\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} + \\mathbf{Ax}_n+By_n-b \\right\\|^2 \\\\ \n&\\quad +\\left(\\frac{1}{2\\ell_H} -\\frac{1}{2\\lambda\\rho}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right)\\right)\\|\\nabla H(y_n)\\|^2 +\\frac{\\nu\\ell_{\\psi}}{2\\lambda\\rho}(2\\ell_H +3\\nu\\ell_{\\psi} -\\omega)\\|y_n-y_{n-1}\\|^2.\n\\end{align*}\nNow, we note that\n\\begin{align*}\n\\rho > \\frac{\\ell_H +\\sqrt{\\ell_H^2 +8(\\ell_H +2\\nu\\ell_{\\psi})^2}}{2\\lambda} \\geq \\frac{\\ell_H +\\sqrt{\\ell_H^2 +8\\ell_H^2}}{2\\lambda} =\\frac{2\\ell_H}{\\lambda},    \n\\end{align*}\nand so $\\frac{1}{2\\ell_H} >\\frac{1}{\\lambda\\rho}$. Choosing $\\omega =1$ if $\\nu\\ell_{\\psi} =0$, and $\\omega =\\nu\\ell_{\\psi}$ if $\\nu\\ell_{\\psi} >0$, it holds that\n\\begin{align*}\n&\\theta :=\\frac{1}{2\\ell_H} -\\frac{1}{2\\lambda\\rho}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right) \\geq \\frac{1}{2\\ell_H} -\\frac{1}{2\\lambda\\rho}(1 +1) >0 \\text{~~and~~} \\\\\n&\\frac{\\nu\\ell_{\\psi}}{2\\lambda\\rho}(2\\ell_H +3\\nu\\ell_{\\psi} -\\omega) \\geq \\frac{\\nu\\ell_{\\psi}}{2\\lambda\\rho}(3\\nu\\ell_{\\psi} -\\omega) =\\frac{\\nu^2\\ell_{\\psi}^2}{\\lambda\\rho}. \n\\end{align*}\nThis together with \\ref{t:cvg_decrease} yields\n\\begin{align}\\label{eq:L_n_bounded}\nL_1 \\geq L_n &\\geq \\left(\\sum_{i=1}^m f_i(x_{i,n}) +P(\\mathbf{x}_n) -G(\\mathbf{x}_n)\\right) +\\left(H(y_n) -\\frac{1}{2\\ell_H}\\|\\nabla H(y_n)\\|^2\\right) +\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} + \\mathbf{Ax}_n+By_n-b \\right\\|^2 \\notag \\\\ \n&\\quad +\\theta\\|\\nabla H(y_n)\\|^2 +\\frac{\\nu^2\\ell_{\\psi}^2}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2,\n\\end{align}\nin which the first term on the right hand side is bounded below since $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, the second term is bounded below due to {the assumption on $H$ and Lemma~\\ref{l:diff}\\ref{l:diff_bounded}}, while the remaining terms are nonnegative due to their positive coefficients. Therefore, all the terms are bounded. Again, as $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, the boundedness of the first term implies that $(\\mathbf{x}_{n})_{n\\in \\mathbb{N}}$ is bounded. Since the last two terms are bounded, it follows from \\eqref{eq:uppper_zn} that $(z_n)_{n\\in \\mathbb{N}}$ is bounded. Since $z_{n+1}=z_n+\\rho(\\mathbf{Ax}_{n+1}+By_{n+1}-b)$ and $B$ has full column rank, we also obtain that $(y_n)_{n\\in \\mathbb{N}}$ is bounded. Consequently, the sequence $(\\mathbf{x}_n,y_n,z_n)_{n\\in \\mathbb{N}}$ is bounded.\n\nOn the other hand, since all the terms on the right hand side of \\eqref{eq:L_n_bounded} are bounded below, so is the sequence $(L_n)_{n\\in\\mathbb{N}}$. Combining with the nonincreasing property of $(L_n)_{n\\in\\mathbb{N}}$ due to \\ref{t:cvg_decrease}, it follows that $(L_n)_{n\\in\\mathbb{N}}$ is convergent. By rearranging \\eqref{eq:decrease} and performing telescoping, we then have that\n\\begin{align*}\n\\sum_{n=1}^{+\\infty}\\delta_y \\|y_{n+1}-y_n\\|^2 + \\delta_{\\mathbf{x}}\\sum_{n=1}^{+\\infty}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 \\leq \\sum_{n=1}^{+\\infty}(L_n - L_{n+1}) = L_1 - \\lim_{n\\to+\\infty} L_n < +\\infty,\n\\end{align*}\nwhich implies that $\\sum_{n=1}^{+\\infty}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_n\\|^2 < +\\infty$, $\\sum_{n=1}^{+\\infty}\\|y_{n+1}-y_n\\|^2 <+\\infty$, and by \\eqref{eq:zy}, $\\sum_{n=1}^{+\\infty}\\|z_{n+1}-z_n\\|^2 < +\\infty$. As a result, $\\mathbf{x}_{n+1}-\\mathbf{x}_{n} \\to 0$, $y_{n+1}-y_n \\to 0$, $z_{n+1}-z_n \\to 0$, and $\\mathbf{Ax}_n +By_n =(z_n -z_{n-1})/\\rho +b \\to b$ as $n \\to +\\infty$. It then follows from the definition of $L_n$ that the sequence $(\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}))_{n\\in\\mathbb{N}}$ is also convergent with $\\lim_{n\\to+\\infty} \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) =\\lim_{n\\to+\\infty}L_n$.\n\nLet $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ be a cluster point of the sequence  $(\\mathbf{x}_n,y_n,z_n)_{n\\in \\mathbb{N}}$. Then there exists a subsequence $(\\mathbf{x}_{k_n},y_{k_n},z_{k_n})_{n\\in \\mathbb{N}}$ that converges to $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$. In view of \\eqref{eq:update_step_x}, letting $n =k_n$ and using Proposition~\\ref{p:Bregman}\\ref{p:Bregman_3}\\&\\ref{p:Bregman_4}, we obtain that, for all $i \\in \\{1,\\dots,m\\}$ and all $x_i\\in \\mathbb{R}^{d_i}$,\n\\begin{align}\\label{eq:kn+}\n&f_i(x_{i,k_n+1}) +\\langle \\nabla_i P(\\mathbf{x}_{k_n})- g_{i,k_n}, x_{i,k_n+1} -x_i \\rangle + \\langle z_{k_n}, A_i x_{i,k_n+1} - A_ix_i\\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,k_n+1}(x_{i,k_n+1}) +By_{k_n} -b \\|^2 \\notag \\\\\n&\\quad  - \\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,k_n+1}(x_{i}) +By_{k_n} -b \\|^2 +\\frac{\\mu\\alpha}{2}\\|x_{i,k_n+1}-x_{i,k_n}\\|^2 - \\frac{\\nu\\ell_{\\phi}}{2}\\|x_{i}-x_{i,k_n}\\|^2 \\leq f_i(x_{i}).\n\\end{align}\nSince $\\nabla P$ is continuous, $\\nabla P(\\mathbf{x}_{k_n}) \\to \\nabla P(\\overline{\\mathbf{x}})$ as $n\\to +\\infty$. As $G +\\frac{\\beta}{2}\\|\\cdot\\|^2$ is a continuous convex function, it follows from \\cite[Example~9.14]{Rockafellar1998} that $G$ is locally Lipschitz continuous. Since $\\mathbf{x}_{k_n} \\to \\overline{\\mathbf{x}}$ as $n\\to +\\infty$, using Lemma~\\ref{l:upsemicont} and passing to a subsequence if necessary, we can and do assume that $g_{k_n}\\to \\overline{g} \\in \\partial_L G (\\overline{\\mathbf{x}})$ as $n\\to +\\infty$. Now, for each $i \\in \\{1,\\dots,m\\}$, letting $x_i =\\overline{x}_i$ and $n\\to +\\infty$ in \\eqref{eq:kn+} with noting that $x_{i,k_n+1} \\to \\overline{x}_i$, $y_{k_n} \\to \\overline{y}$, and $z_{k_n} \\to \\overline{z}$, we have that $\\limsup_{n \\to +\\infty} f_i(x_{i,k_n+1}) \\leq f_i(\\overline{x}_i)$, which together with the lower semicontinuity of $f_i$ yields\n\\begin{align*}\n\\lim_{n \\to +\\infty} f_i(x_{i,k_n+1}) = f_i(\\overline{x}_i).\n\\end{align*}\nAs $H$, $P$, and $G$ are continuous, it follows that $\\lim_{n \\to +\\infty}\\mathcal{L}_\\rho(\\mathbf{x}_{k_n +1 },y_{k_n+1},z_{k_n +1}) = \\mathcal{L}_\\rho(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$. Since the sequence $(\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}))_{n\\in\\mathbb{N}}$ is convergent, we deduce that\n\\begin{align*}\n\\lim_{n \\to +\\infty}\\mathcal{L}_\\rho(\\mathbf{x}_{n },y_{n},z_{n}) =\\mathcal{L}_\\rho(\\overline{\\mathbf{x}},\\overline{y},\\overline{z}).    \n\\end{align*}\n\nWe next show that $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ is a stationary point of $\\mathcal{L}_\\rho$. In \\eqref{eq:z-update} and \\eqref{eq:B'z} with $n$ replaced by $k_n-1$, letting $n\\to +\\infty$ and using the coninuity of $\\nabla H$ and the Lipschitz continuity of $\\nabla \\psi$, we obtain that $0 =\\mathbf{A\\overline{x}} + B\\overline{y} -b$ and $0 =\\nabla H(\\overline{y}) + B^\\top \\overline{z}$. In turn, letting $n \\to +\\infty$ in \\eqref{eq:kn+}, we have for all $i \\in \\{1,\\dots,m\\}$ and all $x_i\\in \\mathbb{R}^{d_i}$ that\n\\begin{align*}\n&f_i(\\overline{x}_i) +\\langle \\nabla_i P(\\overline{\\mathbf{x}})-\\overline{g}_i, \\overline{x}_i-x_i \\rangle + \\langle \\overline{z}, A_i\\overline{x}_i-A_ix_i \\rangle\\\\\n&\\quad +\\frac{\\rho}{2}\\|A_1\\overline{x}_1 +\\dots +A_{i-1}\\overline{x}_{i-1} +A_i\\overline{x}_i +A_{i+1}\\overline{x}_{i+1}+\\dots+A_m \\overline{x}_{m} +B\\overline{y} -b \\|^2 \\\\\n&\\quad -\\frac{\\rho}{2}\\|A_1\\overline{x}_1 +\\dots +A_{i-1}\\overline{x}_{i-1} +A_ix_i +A_{i+1}\\overline{x}_{i+1}+\\dots+A_m \\overline{x}_{m} +B\\overline{y} -b \\|^2 -\\frac{\\nu\\ell_{ \\phi}}{2}\\|x_i-\\overline{x}_i\\|^2 \\leq f_i(x_i),\n\\end{align*}\nwhich can be written as\n\\begin{align*}\nh_i(\\overline{x}_i) \\leq h_i(x_i),    \n\\end{align*}\nwhere $h_i(x_i) =f_i(x_i) +\\langle \\nabla_i P(\\overline{\\mathbf{x}}) -\\overline{g}_i, x_i \\rangle + \\langle \\overline{z}, A_ix_i \\rangle +\\frac{\\rho}{2}\\|A_1\\overline{x}_1 +\\dots +A_{i-1}\\overline{x}_{i-1} +A_ix_i +A_{i+1}\\overline{x}_{i+1}+\\dots+A_m \\overline{x}_{m} +B\\overline{y} -b \\|^2 +\\frac{\\nu\\ell_{\\phi}}{2}\\|x_i-\\overline{x}_i\\|^2$. It follows that, for all $i \\in \\{1,\\dots,m\\}$,\n\\begin{align*}\n0 &\\in \\partial_L h_i(\\overline{x}_i) =\\partial_L (f_i)(\\overline{x}_i) +\\nabla_i P(\\overline{\\mathbf{x}})- \\overline{g}_i +A_i^\\top \\overline{z} +\\rho A_i^\\top (\\mathbf{A}\\overline{\\mathbf{x}}+B\\overline{y}-b).\n\\end{align*}\nSince $\\mathbf{A}\\overline{\\mathbf{x}}+B\\overline{y}-b =0$, we obtain that\n\\begin{align*}\n0 \\in \\partial_L (f_1)(\\overline{x}_1)\\times \\cdots \\times \\partial_L (f_m)(\\overline{x}_m) +\\nabla P(\\overline{\\mathbf{x}}) -\\overline{g} +\\mathbf{A}^\\top \\overline{z},\n\\end{align*}\nwhich completes the proof due to Lemma~\\ref{l:cal}\\ref{l:cal_sum}\\&\\ref{l:cal_sep} and the strict differentiability of $G$.\n\\end{proof}","start":32135,"end":50863},{"raw_tex":"\\begin{proof}\nFor each $n\\in \\mathbb{N}$, set $w_n=(\\mathbf{x}_{n+1},y_{n+1},z_{n+1},y_n)$ and $\\Delta_n =\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\| +\\|y_{n+2}-y_{n+1}\\|$. Let $n \\in \\mathbb{N}^*$. According to Theorem~\\ref{t:cvg}, we have that\n\\begin{align}\\label{eq:auxilary_decrease}\n\\mathcal{F}(w_{n+1}) +\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\|^2 + \\delta_y \\|y_{n+2}-y_{n+1}\\|^2 \\leq \\mathcal{F}(w_{n}),\n\\end{align}\nthat $w_{n+1} - w_n \\to 0$ as $n\\to +\\infty$, that $(w_n)_{n\\in\\mathbb{N}}$ is bounded, that any of its cluster point $\\overline{w}=(\\overline{\\mathbf{x}},\\overline{y},\\overline{z},\\overline{y})$ is a stationary point of \\eqref{eq:prob}, and also that $\\mathcal{F}(w_{n}) \\to \\mathcal{F}(\\overline{w}) = \\mathcal{L}_{\\rho}(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ as $n\\to +\\infty$. Notice that\n\\begin{align*}\n\\Delta_n^2 \\leq \\left(\\frac{1}{\\delta_{\\mathbf{x}}} +\\frac{1}{\\delta_y}\\right) \\left(\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\|^2 + \\delta_y \\|y_{n+2}-y_{n+1}\\|^2\\right),    \n\\end{align*}\nwhich together with \\eqref{eq:auxilary_decrease} yields\n\\begin{align*}\n\\mathcal{F}(w_{n+1}) +C_1\\Delta_n^2 \\leq \\mathcal{F}(w_{n}), \\text{~~where~} C_1 =\\frac{\\delta_{\\mathbf{x}}\\delta_y}{\\delta_{\\mathbf{x}}+\\delta_y}.\n\\end{align*}\n\nNext, we have that\n\\begin{align*}\n\\partial_L ^\\mathbf{x} \\mathcal{F}(w_n) = \\partial_L \\left( \\sum_{i=1}^{m} f_i\\right)(\\mathbf{x}_{n+1}) + \\nabla P(\\mathbf{x}_{n+1}) - \\nabla G(\\mathbf{x}_{n+1}) + \\mathbf{A}^\\top z_{n+1} + \\rho \\mathbf{A}^\\top(\\mathbf{Ax}_{n+1} + By_{n+1} -b)    \n\\end{align*}\nand, for all $i \\in \\{1,\\dots,m\\}$,\n\\begin{align*}\n\\partial_L ^{x_i} \\mathcal{F}(w_n) = \\partial_L f_i(x_{i,n+1})+\\nabla_i P(\\mathbf{x}_{n+1}) - \\nabla_i G(\\mathbf{x}_{n+1}) + A_i^\\top z_{n+1} + \\rho A_i^\\top(\\mathbf{Ax}_{n+1} + By_{n+1} -b).    \n\\end{align*}\nBy revoking the optimality condition for the update of $x_{i,n+1}$ in Step~\\ref{step:A2_2} of Algorithm~\\ref{algo:BPLA},\n\\begin{align*}\n0 &\\in \\partial_L f_i(x_{i,n+1}) +\\nabla_i P(\\mathbf{x}_{n}) -  \\nabla_i G(\\mathbf{x}_n) + A_i^\\top z_n   +\\rho A_i^\\top(\\mathbf{A}\\mathbf{u}_{i,n+1}(x_{i,n+1} ) +By_n -b ) \\\\\n&\\quad +\\mu_n(\\nabla \\phi_i (x_{i,n+1})- \\nabla \\phi_i (x_{i,n})),    \n\\end{align*}\nwhich combined with the above equality implies that\n\\begin{align*}\n&(\\nabla_i P(\\mathbf{x}_{n+1}) -\\nabla_i P(\\mathbf{x}_{n})) -(\\nabla_i G(\\mathbf{x}_{n+1}) -\\nabla_i G(\\mathbf{x}_n)) +A_i^\\top(z_{n+1}-z_n) -\\mu_n(\\nabla \\phi_i (x_{i,n+1}) -\\nabla \\phi_i (x_{i,n})) \\\\\n&\\quad +\\rho A_i^\\top (A_{i+1}x_{i+1,n+1} +\\dots +A_mx_{m,n+1} -A_{i+1}x_{i+1,n} -\\dots -A_mx_{m,n} +By_{n+1} -By_n) \\in \\partial_L ^{x_i} \\mathcal{F}(w_n).\n\\end{align*}\nTherefore,\n\\begin{align}\\label{eq:dxi}\n\\dist(0,\\partial_L ^{x_i} \\mathcal{F}(w_n)) \n&\\leq \\|\\nabla_i P(\\mathbf{x}_{n+1}) -\\nabla_i P(\\mathbf{x}_{n})\\| +\\|\\nabla_i G(\\mathbf{x}_{n+1}) -\\nabla_i G(\\mathbf{x}_n)\\| \\notag \\\\\n&\\quad +\\|A_i^\\top(z_{n+1}-z_n)\\| +\\mu_n\\|\\nabla \\phi_i (x_{i,n+1}) -\\nabla \\phi_i (x_{i,n})\\| \\notag \\\\\n&\\quad +\\rho \\|A_i^\\top (A_{i+1}x_{i+1,n+1} +\\dots +A_mx_{m,n+1} -A_{i+1}x_{i+1,n} -\\dots -A_mx_{m,n} +By_{n+1} -By_n\\| \\notag \\\\ \n&\\leq \\ell_P\\|\\mathbf{x}_{n+1} - \\mathbf{x}_{n}\\| +\\ell_G\\|\\mathbf{x}_{n} - \\mathbf{x}_{n+1}\\| +\\|A_i^\\top\\|\\|z_{n+1}-z_n\\| +\\mu_n \\ell_{\\phi}\\|x_{i,n+1}-x_{i,n}\\| \\notag \\\\\n&\\quad +\\rho\\sum_{j=i+1}^{m} \\|A_i^\\top\\| \\|A_j\\| \\|x_{j,n+1}-x_{j,n}\\| +\\rho\\|A_i^\\top\\| \\|B\\| \\|y_{n+1}-y_n\\| \\notag \\\\\n&\\leq  \\left(\\ell_P +\\ell_G +\\mu_n\\ell_{\\phi} +\\rho\\sum_{j=i+1}^{m} \\|A_i^\\top\\| \\|A_j\\|\\right)\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\| +\\|A_i^\\top\\|\\|z_{n+1}-z_n\\| \\notag \\\\\n&\\quad +\\rho\\|A_i^\\top\\| \\|B\\| \\|y_{n+1}-y_n\\|.\n\\end{align}\n\nWe now see that\n\\begin{align*}\n\\partial_L ^y \\mathcal{F}(w_n) =\\nabla H(y_{n+1}) + B^\\top z_{n+1} +\\rho B^\\top(\\mathbf{Ax}_{n+1} + By_{n+1} -b) +2c(y_{n+1}-y_{n})    \n\\end{align*}\nand from the optimality condition for the update of $y_{n+1}$ in Algorithm~\\ref{algo:BPLA} that \n\\begin{align*}\n0 &= \\nabla H(y_{n+1}) + B^\\top z_n +\\rho B^\\top(\\mathbf{Ax}_{n+1} +By_{n+1} -b) +\\nu_n(\\nabla \\psi(y_{n+1}) - \\nabla \\psi(y_n)).\n\\end{align*}\nThis leads to\n\\begin{align*}\n\\partial_L ^y \\mathcal{F}(w_n) &=B^{\\top}(z_{n+1}-z_n) -\\nu_n(\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)) + 2c(y_{n+1}-y_{n}),   \n\\end{align*}\nand so\n\\begin{align}\\label{eq:dy}\n\\dist(0,\\partial_L ^{y} \\mathcal{F}(w_n)) \n&\\leq \\|B^{\\top}(z_{n+1}-z_n)\\| +\\nu_n\\|\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)\\| + 2c\\|y_{n+1}-y_{n})\\| \\notag \\\\\n&\\leq \\|B^{\\top}(z_{n+1}-z_n)\\| +(\\nu\\ell_\\psi +2c)\\|y_{n+1}-y_{n})\\|.\n\\end{align}\n\nSince $\\partial_L ^z \\mathcal{F}(w_n) =\\mathbf{Ax}_{n+1} + By_{n+1} -b =\\frac{1}{\\rho}(z_{n+1} -z_n)$ and\n$\\partial_L ^t \\mathcal{F}(w_n) =-2c(y_{n+1}-y_{n})$, it holds that\n\\begin{align}\\label{eq:dzdt}\n\\dist(0,\\partial_L ^{z} \\mathcal{F}(w_n)) =\\frac{1}{\\rho}\\|z_{n+1}-z_n\\| \\text{~~and~~}\n\\dist(0,\\partial_L ^{t} \\mathcal{F}(w_n)) =2c\\|y_{n+1}-y_n\\|.    \n\\end{align}\nCombining \\eqref{eq:dxi}, \\eqref{eq:dy}, and \\eqref{eq:dzdt}, noting that $\\limsup_{n\\to +\\infty} \\mu_n =\\overline{\\mu} <+\\infty$, and using \\eqref{eq:BTZy} and \\eqref{eq:zy} in the proof of Theorem~\\ref{t:cvg}, we derive that there exists $n_0\\in \\mathbb{N}^*$ and $C_2 \\in \\mathbb{R}_{++}$ such that, for all $n\\geq n_0$, \n\n\\begin{align*}\n\\dist(0, \\partial_L \\mathcal{F}(w_n)) \n\\leq C_2 \\left(\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\| +\\|y_{n+1}-y_n\\| +\\|y_{n}-y_{n-1}\\|\\right) \n\\leq C_2(\\Delta_{n-1} +\\Delta_{n-2}).\n\\end{align*}\n\n\\ref{t:full_cvg}: We see that all the conditions in the \\emph{abstract convergence} framework \\cite[Theorem~5.1]{BDL22} are satisfied with $I =\\{1,2\\}$, $\\lambda_1=\\lambda_2=1/2$, $\\alpha_n \\equiv C_1$, $\\beta_n \\equiv 1/(2C_2)$, and $\\varepsilon_n \\equiv 0 $. By \\cite[Theorem~5.1(i)]{BDL22}, \n\\begin{align*}\n\\sum_{n=0}^{+\\infty} (\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\|+\\|y_{n+2}-y_{n+1}\\|) =\\sum_{n=0}^{+\\infty} \\Delta_n < +\\infty,\n\\end{align*}\nwhich implies that $\\sum_{n=0}^{+\\infty} \\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\| <+\\infty$ and $\\sum_{n=0}^{+\\infty} \\|y_{n+1}-y_{n}\\| < +\\infty$. Together with \\eqref{eq:zy}, we derive that $\\sum_{n=0}^{+\\infty} \\|z_{n+1}-z_n\\|<+\\infty$, and hence \n\\begin{align*}\n\\sum_{n=0}^{+\\infty} \\|(\\mathbf{x}_{n+1},y_{n+1},z_{n+1})-(\\mathbf{x}_{n},y_{n},z_{n})\\| < +\\infty,\n\\end{align*}\nwhich yields the convergence of $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ to $(\\mathbf{x}^{*},y^{*},z^{*})$. In view of Theorem~\\ref{t:cvg}\\ref{t:cvg_seq}, $(\\mathbf{x}^{*},y^{*},z^{*})$ is a stationary point of $\\mathcal{L}_\\rho$.\n\n\\ref{t:full_rate_finite}: This follows from the arguments in the proof of \\cite[Theorem~5.1]{BDL22} and \\cite[Theorem~2(i)]{Attouch2007}. \n\n\\ref{t:full_rate_linear}: According to \\cite[Theorem~5.1(iv)]{BDL22}, there exist $\\gamma_0\\in \\mathbb{R}_{++}$ and $\\zeta\\in \\left(0,1\\right)$ such that, for all $n\\in \\mathbb{N}^*$, \n\\begin{align*}\n\\|(\\mathbf{x}_{n},y_{n})-(\\mathbf{x}^{*},y^{*})\\|\\leq\\gamma_0 \\zeta^{\\frac{n}{2}} \\text{~~and~~}\n|\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*)| \\leq \\gamma_0\\zeta^n.\n\\end{align*}\nWe derive that, for all $n\\in \\mathbb{N}^*$, $\\|\\mathbf{x}_n-\\mathbf{x}^{*}\\|\\leq \\gamma_0\\zeta^{\\frac{n}{2}}$ and $\\|y_n-y^{*}\\|\\leq \\gamma_0\\zeta^{\\frac{n}{2}}$, which yields \n\\begin{align*}\n\\|y_n-y_{n-1}\\| \\leq \\|y_n-y^{*}\\|+\\|y_{n-1}-y^{*}\\| \\leq (1+\\zeta^{-\\frac{1}{2}})\\gamma_0\\zeta^{\\frac{n}{2}}.    \n\\end{align*}\nBy passing to the limit in \\eqref{eq:B'z}, $B^\\top z^{*}=-\\nabla H(y^{*})$. Following the same steps as in \\eqref{eq:BTZy} and \\eqref{eq:zy}, we obtain that, for all $n\\in \\mathbb{N}^*$,\n\\begin{align*}\n\\|z_n-z^{*}\\| &\\leq \\frac{\\ell_H}{\\sqrt{\\lambda}}\\|y_n-y^{*}\\| + \\frac{\\nu \\ell_{\\psi}}{\\sqrt{\\lambda}}\\|y_{n}-y_{n-1}\\|\\leq \\gamma_1\\zeta^{\\frac{n}{2}}, \\text{~~where~} \\gamma_1 :=\\frac{\\ell_H\\gamma_0+(1+\\zeta^{-\\frac{1}{2}})\\nu\\ell_{\\psi}\\gamma_0}{\\sqrt{\\lambda}}.\n\\end{align*}\nConsequently,\n\\begin{align*}\n&\\|(\\mathbf{x}_{n},y_{n},z_{n})-(\\mathbf{x}^{*},y^{*},z^{*})\\| \\leq \\|\\mathbf{x}_{n}-\\mathbf{x}^{*}\\| + \\|y_{n}-y^{*}\\| + \\|z_{n}-z^{*}\\|\\leq (2\\gamma_0 +\\gamma_1)\\zeta^{\\frac{n}{2}}, \\\\\n&\\|\\mathbf{Ax}_n+By_n-b\\| =\\frac{1}{\\rho}\\|z_n-z_{n-1}\\| \\leq \\frac{1}{\\rho}(\\|z_n-z^{*}\\| +\\|z_{n-1}-z^{*}\\|) \\leq \\frac{(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho} \\zeta^{\\frac{n}{2}}, \\\\\n\\text{and~} &\\|z_n\\| \\leq \\|z_n-z^{*}\\| + \\|z^{*}\\| \\leq \\gamma_1 \\zeta^{\\frac{n}{2}} + \\|z^{*}\\|.\n\\end{align*}\n\nWe now deduce from the definition of $\\mathcal{F}$ that\n\\begin{align*}\n|\\mathcal{L}_{\\rho}(\\mathbf{x}_{n},y_{n},z_{n})- \\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})|&= \\left|\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*) - c\\|y_n-y_{n-1}\\|^2\\right|\\\\\n&\\leq |\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*)| + c\\|y_n-y_{n-1}\\|^2\\\\\n&\\leq \\gamma_2\\zeta^n, \\text{~~where~} \\gamma_2 :=\\gamma_0 + c(1+\\zeta^{-\\frac{1}{2}})^2\\gamma_0^2\n\\end{align*}\nand from the definition of $\\mathcal{L}_\\rho$ that \n\\begin{align*}\n|F(\\mathbf{x}_n,y_n)-F(\\mathbf{x}^{*},y^{*})| &= |\\mathcal{L}_{\\rho}(\\mathbf{x}_n,y_n,z_n)-\\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})-\\langle z_n, \\mathbf{Ax}_n + By_n-b\\rangle - \\frac{\\rho}{2}\\|\\mathbf{Ax}_n + By_n-b\\|^2| \\\\\n&\\leq |\\mathcal{L}_{\\rho}(\\mathbf{x}_n,y_n,z_n)-\\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})| + \\|z_n\\|\\|\\mathbf{Ax}_n + By_n-b\\| + \\frac{\\rho}{2}\\|\\mathbf{Ax}_n + By_n-b\\|^2  \\\\ \n&\\leq \\gamma_2\\zeta^n + \\left(\\gamma_1 \\zeta^{\\frac{n}{2}} + \\|z^{*}\\|\\right)\\frac{(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho} \\zeta^{\\frac{n}{2}}+ \\frac{(1+\\zeta^{-\\frac{1}{2}})^2\\gamma_1^2}{2\\rho}\\zeta^n \\\\\n&\\leq \\left(\\gamma_2 +\\frac{(3+\\zeta^{-\\frac{1}{2}})(1+\\zeta^{-\\frac{1}{2}})\\gamma_1^2}{2\\rho} +\\frac{\\|z^{*}\\|(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho}\\right)\\zeta^{\\frac{n}{2}},\n\\end{align*}\nwhere the last inequality follows from the fact that $\\zeta^n \\leq \\zeta^{\\frac{n}{2}}$ since $\\zeta \\in (0,1)$. By letting \n\\begin{align*}\n\\overline{\\gamma} =\\max \\left\\{ 2\\gamma_0+\\gamma_1, \\frac{(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho}, \\gamma_2 +\\frac{(3+\\zeta^{-\\frac{1}{2}})(1+\\zeta^{-\\frac{1}{2}})\\gamma_1^2}{2\\rho} +\\frac{\\|z^{*}\\|(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho} \\right\\} \n\\end{align*}\nand increasing it if necessary, we arrive at the conclusion.\n\n\\ref{t:full_rate_sublinear}: Following the arguments in \\cite[Theorem~5.1(iv)]{BDL22} and \\cite[Theorem~2(iii)]{Attouch2007}, we find $\\gamma_0 \\in \\mathbb{R}_{++}$ such that, for all $n\\in \\mathbb{N}^*$, \n\\begin{align*}\n\\|(\\mathbf{x}_{n},y_{n})-(\\mathbf{x}^{*},y^{*})\\|\\leq \\gamma_0 n^{-\\frac{1-\\kappa}{2\\kappa-1}} \\text{~~and~~}\n|\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*)| \\leq \\gamma_0 n^{-\\frac{2-2\\kappa}{2\\kappa-1}}.\n\\end{align*}\nSimilar to \\ref{t:full_rate_linear}, for all $n\\geq 2$, since $n-1\\geq \\frac{1}{2}n$ and $\\frac{1-\\kappa}{2\\kappa-1} >0$, we derive that \n\\begin{align*}\n&\\|(\\mathbf{x}_{n},y_{n},z_{n})-(\\mathbf{x}^{*},y^{*},z^{*})\\| \\leq (2\\gamma_0+\\gamma_1) n^{-\\frac{1-\\kappa}{2\\kappa-1}},\\\\\n&\\|\\mathbf{Ax}_n+By_n-b\\|\\leq \\frac{(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1}{\\rho} n^{-\\frac{1-\\kappa}{2\\kappa-1}},\\\\\n&|\\mathcal{L}_{\\rho}(\\mathbf{x}_{n},y_{n},z_{n})- \\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})|\\leq \\gamma_2 n^{-\\frac{2-2\\kappa}{2\\kappa-1}},\n\\end{align*}\nand that\n\\begin{align*}\n|F(\\mathbf{x}_n,y_n)-F(\\mathbf{x}^{*},y^{*})| &\\leq \\gamma_2 n^{-\\frac{2-2\\kappa}{2\\kappa-1}}+ \\left(\\gamma_1 n^{-\\frac{1-\\kappa}{2\\kappa-1}} + \\|z^{*}\\|\\right)\\frac{(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1}{\\rho} n^{-\\frac{1-\\kappa}{2\\kappa-1}} + \\frac{(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})^2\\gamma_1^2}{2\\rho} n^{-\\frac{2-2\\kappa}{2\\kappa-1}}\\\\\n&\\leq \\left(\\gamma_2 +\\frac{(3 +2^{\\frac{1-\\kappa}{2\\kappa-1}})(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1^2}{2\\rho} +\\frac{\\|z^{*}\\|(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1}{\\rho} \\right)n^{-\\frac{1-\\kappa}{2\\kappa-1}},\n\\end{align*}\nwhere $\\gamma_1 :=\\frac{\\ell_H\\gamma_0 +(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\nu\\ell_{\\psi}\\gamma_0}{\\sqrt{\\lambda}}$, $\\gamma_2 :=\\gamma_0 +c(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})^2\\gamma_0^2$, and the last inequality follows from the fact that $n^{-\\frac{2-2\\kappa}{2\\kappa-1}}<n^{-\\frac{1-\\kappa}{2\\kappa-1}}$ for $\\kappa\\in(\\frac{1}{2},1)$. By setting $\\overline{\\gamma}$ in the same way as in \\ref{t:full_rate_linear}, we obtain the conclusion.\n\\end{proof}","start":55931,"end":68289}]},"nodes":[{"id":"S001","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"基本空间与指标","source_units":[{"raw_tex":"Throughout this paper, the set of nonnegative integers is denoted by $\\mathbb{N}$, the set of real numbers by $\\mathbb{R}$, the set of nonnegative real numbers by $\\mathbb{R}_+$, and the set of the positive real numbers by $\\mathbb{R}_{++}$. We use $\\mathbb{R}^d$ to denote the $d$-dimensional Euclidean space with inner product $\\langle \\cdot, \\cdot \\rangle$ and Euclidean norm $\\|\\cdot\\|$. For a given matrix $M$, $\\Ima(M)$ denotes\nits image.\n\n","start":5195,"end":5641}]},{"id":"S002","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:prob"],"declared_external":false,"source_only":false,"title":"线性约束复合问题","source_units":[{"raw_tex":"We consider the linearly constrained composite optimization problem\n\\begin{align}\\label{eq:prob}\n\\min_{x_i \\in \\mathbb{R}^{d_i}, y\\in \\mathbb{R}^q} ~F(\\mathbf{x},y) :=\\sum_{i=1}^m f_i(x_i)+H(y)+P(\\mathbf{x})-G(\\mathbf{x}) \\quad\\text{s.t.~~}  \\sum_{i=1}^{m}A_ix_i + By=b,\n\\end{align}\nwhere $\\mathbf{x}=(x_1,\\dots,x_m)$; for each $i\\in \\{1,\\dots,m\\}$, $f_i: \\mathbb{R}^{d_i} \\to (-\\infty,+\\infty]$ is a proper lower semicontinuous function; $H:\\mathbb{R}^q \\to \\mathbb{R}$ is a differentiable (possibly\nnonconvex) function whose gradient is Lipschitz continuous; $P:\\mathbb{R}^{d_1}\\times \\dots \\times \\mathbb{R}^{d_m} \\to \\mathbb{R}$ is another differentiable (possibly nonconvex) function whose gradient is Lipschitz continuous; $G:\\mathbb{R}^{d_1}\\times \\dots \\times \\mathbb{R}^{d_m} \\to \\mathbb{R}$ is a continuous (possibly nonsmooth) weakly convex function, $A_i \\in \\mathbb{R}^{p\\times d_i}$, $B\\in \\mathbb{R}^{p\\times q}$, and $b \\in \\mathbb{R}^p$. Problem \\eqref{eq:prob} covers two important models in the literature. Firstly, when $G \\equiv 0$, problem \\eqref{eq:prob} reduces to the form\n","start":5641,"end":6739}]},{"id":"S003","semantic_type":"interpretation","kind":"interpretation","labels":["DCADMM","problem:WTY"],"declared_external":false,"source_only":false,"title":"特例与既有研究","source_units":[{"raw_tex":"\\begin{align}\\label{problem:WTY}\n\\min_{x_i \\in \\mathbb{R}^{d_i}, y\\in \\mathbb{R}^q} ~\\sum_{i=1}^m f_i(x_i)+H(y)+P(\\mathbf{x}) \\quad\\text{s.t.~~}  \\sum_{i=1}^{m}A_ix_i + By=b,\n\\end{align}\nwhich is studied in \\cite{Wang2018}. This class of problems, together with its special cases considered in \\cite{Yang2017,Guo2016}, has a broad range of useful applications in image processing, matrix decomposition, statistical learning, and others \\cite{Wang2018,ADMMsurvey}. In \\cite{Wang2018}, the author studied the multi-block alternating direction method of multipliers (ADMM) in the nonconvex nonsmooth setting, and they introduced new assumptions such as restricted prox-regularity and Lipschitz subminimization path to establish the global convergence. A special case of \\eqref{problem:WTY} in which $P\\equiv0$ was considered in \\cite{Yashtini2021}, and proximal terms and dual stepsize were incorporated into the ADMM to solve the problem. An inertial proximal splitting method was also developed in \\cite{XWang2023} to revisit that problem. Other simpler special cases of \\eqref{problem:WTY} can be addressed by variants of the three-block ADMM  \\cite{Yang2017,Lin2017,RPCA}, or two-block ADMM \\cite{Guo2016,2blockBregADMM,Li_2015}, and their convergence are well studied in the literature. Secondly, when $P \\equiv 0$, $m =1$, $x_1 =x \\in \\mathbb{R}^d$, and $A_1 =A \\in \\mathbb{R}^{p\\times d}$, problem \\eqref{eq:prob} reduces to\n\\begin{align}\\label{DCADMM}\n\\min_{x \\in \\mathbb{R}^d,y \\in \\mathbb{R}^q}~ f(x)-G(x)+H(y) \\quad \\text{s.t.~} Ax+By=c.\n\\end{align}\nThis is known as the linearly constrained difference-of-convex problem \\cite{Sun2017,Tu2019}, in which $f$, $H$, and $G$ are all assumed to be convex. Problem \\eqref{DCADMM} arises from the use of the difference-of-convex regularization terms, which has shown very promising performance \\cite{Gasso2009,Yin2015}. One practical engineering problem that can be solved via problem \\eqref{DCADMM} is the \\emph{total variation image restoration problem} \\cite{Osher2005,Sun2017,Tu2019}, which is one of the most fundamental issues in imaging science. The authors of \\cite{Sun2017} developed a variant of the ADMM which incorporates the Bregman distance in updating steps for solving problem \\eqref{DCADMM} in the case when $f$, $G$, and $H$ are convex functions, with $H$ being differentiable with a Lipschitz continuous gradient. A hybrid Bregman ADMM was then proposed in \\cite{Tu2019} to revisit problem \\eqref{DCADMM} with the same assumptions on $f$, $G$, and $H$. This version introduces an additional step to evaluate the convex conjugate of $G$, allowing the flexibility in choosing the subgradient step or proximal step to update the solution by choosing a control parameter to either activate or deactivate the additional step. By this strategy, this algorithm can include the one in \\cite{Sun2017} as a special case. However, further research on more general form of problem \\eqref{DCADMM} is still not available in the literature, to the best of the authors' knowledge. \n","start":6739,"end":9775}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"凸性与强制性","source_units":[{"raw_tex":"Let $f\\colon \\mathbb{R}^d\\to \\left(-\\infty,+\\infty\\right]$. The \\emph{domain} of $f$ is $\\dom f :=\\{x\\in \\mathbb{R}^d: f(x) <+\\infty\\}$ and the \\emph{epigraph} of $f$ is $\\epi f := \\{(x,\\xi)\\in \\mathbb{R}^d\\times \\mathbb{R}: f(x)\\leq \\xi\\}$. The function $f$ is said to be \\emph{proper} if $\\dom f \\neq \\varnothing$ and it never takes the value $-\\infty$, \\emph{lower semicontinuous} if its epigraph is closed, and \\emph{convex} if its epigraph is convex. The function $f$ is said to be $\\alpha$-\\emph{strongly convex} ($\\alpha\\in \\mathbb{R}_+$) if $f-\\frac{\\alpha}{2}\\|\\cdot\\|^2$ is convex, and $\\beta$-\\emph{weakly convex} ($\\beta\\in \\mathbb{R}_+$) if $f+\\frac{\\beta}{2}\\|\\cdot\\|^2$ is convex. We say that $f$ is \\emph{coercive} if $f(x)\\to +\\infty$ as $\\|x\\|\\to +\\infty$.\n\n","start":14840,"end":15616}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"极限次微分","source_units":[{"raw_tex":"Let $x\\in \\mathbb{R}^d$ with $|f(x)| <+\\infty$. The \\emph{regular} (or \\emph{Fr\\'echet}) \\emph{subdifferential} of $f$ at $x$ is defined by\n\\begin{equation*}\n\\widehat{\\partial} f(x) :=\\left\\{x^*\\in \\mathbb{R}^d:\\; \\liminf_{y\\to x}\\frac{f(y)-f(x)-\\langle x^*,y-x\\rangle}{\\|y-x\\|}\\geq 0\\right\\}\n\\end{equation*}\nand the \\emph{limiting} (or \\emph{Mordukhovich}) \\emph{subdifferential} of $f$ at $x$ is defined by\n\\begin{equation*}\n\\partial_L f(x) :=\\left\\{x^*\\in \\mathbb{R}^d:\\; \\exists x_n\\stackrel{f}{\\to}x,\\; x_n^*\\to x^* \\text{~~with~~} x_n^*\\in \\widehat{\\partial} f(x_n)\\right\\},\n\\end{equation*}\nwhere the notation $y\\stackrel{f}{\\to}x$ means $y\\to x$ with $f(y)\\to f(x)$. In the case where $\\lvert f(x) \\rvert =+\\infty$, both regular subdifferential and limiting subdifferential of $f$ at $x$ are defined to be the empty set. The \\emph{domain} of $\\partial_L f$ is given by $\\dom \\partial_L f :=\\{x\\in \\mathbb{R}^d: \\partial_L f(x) \\neq \\varnothing\\}$. \nWe now collect some important properties of the limiting subdifferential.\n","start":15616,"end":16646}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"Bregman距离","source_units":[{"raw_tex":"We recall the concept of the well-known Bregman distance in a real Hilbert space $\\mathcal{H}$, which was introduced in \\cite{Bregman}. Given a differentiable convex function $\\phi: \\mathcal{H} \\to \\mathbb{R}$, the \\emph{Bregman distance} associated with $\\phi$ is defined as \n\\begin{align*}\n\\forall u, v\\in \\mathcal{H},\\quad D_{\\phi}(u,v)=\\phi(u)-\\phi(v)-\\langle \\nabla\\phi (v),u-v\\rangle,\n\\end{align*}\nWhen $\\phi(x)=\\|x\\|^2$, then $D_{\\phi}(u,v)=\\|u-v\\|^2$. When $\\phi(x)=x^\\top M x$, where $M$ is a positive semidefinite matrix, then $D_{\\phi}(u,v) = \\|u-v\\|^2_M :=(u-v)^\\top M (u-v)$. When $\\mathcal{H}=\\mathbb{R}^{p\\times d}$ and $\\phi(X)=\\|X\\|_F^2$, then $D_{\\phi}(U,V) =\\|U-V\\|_F^2$ \\cite{Dhillon2008}. Some useful properties of the Bregman distance are listed in the following proposition.\n","start":18239,"end":19037}]},{"id":"S007","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"分块与谱记号","source_units":[{"raw_tex":"We first fix some notations which are used throughout the paper from now on. Let $\\mathbf{A}=[A_1|A_2|\\dots|A_m]$ and let $\\mathbf{I}_{q\\times q}$ be the $q\\times q$ identity matrix. Then $\\mathbf{Ax}=\\sum_{i=1}^{m}A_ix_i$. We also denote by $\\partial_L^{x}$ the limiting subdifferential with respect to the $x$-variable, and by $\\nabla_i f$ the $i^{th}$ block of the gradient vector $\\nabla f$. Given a matrix $M$, $\\lambda_{\\min}(M)$ denotes its smallest eigenvalue while $\\lambda_{\\max}(M)$ denotes its largest eigenvalue. The following assumptions are used in our convergence analysis\n","start":21422,"end":22011}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","labels":["a:standing","a:standing_image","a:standing_rank"],"declared_external":false,"source_only":false,"title":"光滑性与弱凸性","source_units":[{"raw_tex":"\\begin{assumption}[Standing assumptions]\n\\label{a:standing}\n~\n\\begin{enumerate}\n\\item \n$\\nabla H$ is $\\ell_H$-Lipschitz continuous, $\\nabla P$ is $\\ell_P$-Lipschitz continuous, and $G$ is $\\beta$-weakly convex.\n","start":22011,"end":22222}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","labels":["a:standing","a:standing_image","a:standing_rank"],"declared_external":false,"source_only":false,"title":"像空间包含条件","source_units":[{"raw_tex":"\\item\\label{a:standing_image} \n$\\Ima(\\mathbf{A}) \\bigcup \\{b \\}\\subseteq \\Ima(B)$.\n","start":22222,"end":22305}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"矩阵满列秩","source_units":[{"raw_tex":"\\item\\label{a:standing_rank} \n $\\lambda :=\\lambda_{\\min}(B^\\top B) >0$ (equivalently, $B$ has full column rank).\n\\end{enumerate}\n\\end{assumption}\n","start":22305,"end":22451}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"增广拉格朗日函数","source_units":[{"raw_tex":"Recall that the augmented Lagrangian of problem \\eqref{eq:prob} is given by\n\\begin{align*}\n\\mathcal{L}_\\rho(\\mathbf{x},y,z)=\\sum_{i=1}^m f_i(x_i)+H(y)+P(\\mathbf{x})-G(\\mathbf{x}) +  \\langle z, \\mathbf{Ax}+By-b \\rangle +\\frac{\\rho}{2}\\|\\mathbf{Ax}+By-b\\|^2,\n\\end{align*}\n","start":22747,"end":23017}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["eq:optimality_conditions"],"declared_external":false,"source_only":false,"title":"驻点条件","source_units":[{"raw_tex":"where $z\\in \\mathbb{R}^p$ and $\\rho\\in \\mathbb{R}_{++}$. Now, we present our splitting algorithm with guaranteed global convergence to a stationary point $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ of $\\mathcal{L}_\\rho$, i.e., $0 \\in \\partial_L \\mathcal{L}_\\rho(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$, or equivalently,\n\\begin{equation}\\label{eq:optimality_conditions}\n\\begin{aligned}\n&0 \\in \\partial_L\\left(\\sum_{i=1}^{m}f_i -G\\right)(\\overline{\\mathbf{x}}) +\\nabla P(\\overline{\\mathbf{x}}) +\\mathbf{A}^\\top \\overline{z},\\\\\n&0 =\\nabla H(\\overline{y}) +B^\\top \\overline{z}, \\\\\n&0 =\\mathbf{A\\overline{x}} +B\\overline{y} -b.\n\\end{aligned}\n\\end{equation}\n","start":23017,"end":23687}]},{"id":"S013","semantic_type":"algorithm","kind":"algorithm","labels":["algo:BPLA","step:A2_1","step:A2_2"],"declared_external":false,"source_only":false,"title":"BPL-ADMM迭代","source_units":[{"raw_tex":"\\begin{tcolorbox}[\n\tleft=0pt,right=0pt,top=0pt,bottom=0pt,\n\tcolback=blue!10!white, colframe=blue!50!white,\n  \tboxrule=0.2pt,\n  \tbreakable]\n\\begin{algorithm}[Bregman proximal linearized ADMM (BPL-ADMM)]\n\\label{algo:BPLA}\n\\step{}\\label{step:A2_1}\nSet $n=0$. Choose $\\mathbf{x}_0 =(x_{1,0}, \\dots, x_{m,0}) \\in \\mathbb{R}^{d_1}\\times \\dots \\times \\mathbb{R}^{d_m}$, $y_0 \\in \\mathbb{R}^q$, and $z_0 \\in \\mathbb{R}^p$. Let $\\phi_i: \\mathbb{R}^{d_i} \\to \\mathbb{R}$, $i \\in \\{1,\\dots,m\\}$, be differentiable $\\alpha$-strongly convex functions with $\\alpha \\in (0, +\\infty)$, and let $\\psi: \\mathbb{R}^q \\to \\mathbb{R}$ be a differentiable convex function with $\\ell_{\\psi}$-Lipschitz continuous gradient. Let \n\\begin{align*}\n\\mu \\in \\left(\\frac{\\ell_P+\\beta}{\\alpha}, +\\infty\\right),\\;\\; \\nu \\in \\left[0, +\\infty\\right), \\text{~~and~~} \\rho \\in \\left(\\frac{\\ell_H +\\sqrt{\\ell_H^2 +8(\\ell_H +2\\nu\\ell_{\\psi})^2}}{2\\lambda}, +\\infty\\right).    \n\\end{align*}\n\n\\step{}\\label{step:A2_2}\nCalculate $\\nabla P(\\mathbf{x}_n)=(\\nabla_1 P(\\mathbf{x}_n),\\dots,\\nabla_m P(\\mathbf{x}_n))$ and $g_{n}=(g_{1,n},\\dots,g_{m,n}) \\in \\partial_L G(\\mathbf{x}_{n})$. Let $\\mu_n \\in [\\mu, +\\infty)$ and $\\nu_n \\in [0, \\nu]$. For each $i \\in \\{1,\\dots,m\\}$, define $\\mathbf{u}_{i,n+1}: \\mathbb{R}^{d_i}\\to \\mathbb{R}^{d_1}\\times \\dots \\times \\mathbb{R}^{d_m}$ by $\\mathbf{u}_{i,n+1}(x_i) =(x_{1,n+1}, \\dots, x_{i-1,n+1}, x_i, x_{i+1,n}, \\dots, x_{m,n})$, and find\n\\begin{align*}\nx_{i,n+1} &\\in  \\argmin_{x_i \\in \\mathbb{R}^{d_i}} \\left(f_i(x_i) +\\langle \\nabla_i P(\\mathbf{x}_n) -g_{i,n}, x_i \\rangle + \\langle z_n, A_ix_i\\rangle+\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,n+1}(x_i)+By_n-b\\|^2 +\\mu_n D_{\\phi_i}(x_i,x_{i,n})\\right),\\\\\ny_{n+1} &\\in  \\argmin_{y \\in \\mathbb{R}^q} \\left(H(y) + \\langle z_n, By\\rangle+\\frac{\\rho}{2}\\|\\mathbf{Ax}_{n+1} + By-b \\|^2 +\\nu_n D_{\\psi}(y,y_n) \\right),\\\\\nz_{n+1} &=z_n+\\rho(\\mathbf{Ax}_{n+1}+By_{n+1}-b),\n\\end{align*}\nwhere $\\mathbf{x}_{n+1} =(x_{1,n+1}, \\dots, x_{m,n+1})$.\n\\step{}\nIf a termination criterion is not met, set $n =n+1$ and go to Step~\\ref{step:A2_2}.\n\\end{algorithm}\n\\end{tcolorbox}\n","start":23689,"end":25795}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"KL性质与指数","source_units":[{"raw_tex":"We now establish the convergence of the full sequence generated by Algorithm~\\ref{algo:BPLA}. Recall that a proper lower semicontinuous function $\\mathcal{F}\\colon \\mathcal{H}\\to \\left(-\\infty, +\\infty\\right]$ satisfies the \\emph{Kurdyka--Łojasiewicz (KL) property} \\cite{Kurdyka1998,Loja63} at $\\overline{x} \\in \\dom \\partial_L \\mathcal{F}$ if there are $\\eta \\in (0, +\\infty]$, a neighborhood $V$ of $\\overline{x}$, and a continuous concave function $\\varphi: \\left[0, \\eta\\right) \\to \\mathbb{R}_+$ such that $\\varphi$ is continuously differentiable with $\\varphi' > 0$ on $(0, \\eta)$, $\\varphi(0) = 0$, and, for all $x \\in V$ with $\n\\mathcal{F}(\\overline{x}) < \\mathcal{F}(x) < \\mathcal{F}(\\overline{x}) + \\eta$, \n\\begin{equation*}\n\\varphi'(\\mathcal{F}(x) -\\mathcal{F}(\\overline{x})) \\dist(0, \\partial_L \\mathcal{F}(x)) \\geq 1.   \n\\end{equation*}\nWe say that $\\mathcal{F}$ is a \\emph{KL function} if it satisfies the KL property at any point in $\\dom \\partial_L \\mathcal{F}$. If $\\mathcal{F}$ satisfies the KL property at $\\overline{x} \\in \\dom \\partial_L \\mathcal{F}$, in which the corresponding function $\\varphi$ can be chosen as $\\varphi(t) = c t ^{1 - \\lambda}$ for some $c \\in \\mathbb{R}_{++}$ and $\\lambda \\in [0, 1)$, then $\\mathcal{F}$ is said to satisfy the \\emph{KL property at $\\overline{x}$ with exponent $\\lambda$}. The function $\\mathcal{F}$ is called a \\emph{KL function with exponent $\\lambda$} if it is a KL function and has the same exponent $\\lambda$ at any $x \\in \\dom \\partial_L \\mathcal{F}$. Now we will present the proof of the full sequential convergence in the following theorem.\n","start":51504,"end":53113}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","labels":["t:full","t:full_cvg","t:full_rate","t:full_rate_finite","t:full_rate_linear","t:full_rate_sublinear"],"declared_external":false,"source_only":false,"title":"主定理附加条件","source_units":[{"raw_tex":"Suppose that Assumption~\\ref{a:standing} holds, that $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, that $H$ is bounded below, that $G$ is differentiable with $\\ell_G$-Lipschitz continuous gradient, and that, for all $i\\in \\{1, \\dots, m\\}$, $\\nabla{\\phi_i}$ is $\\ell_{\\phi}$-Lipschitz continuous. Let $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ be the sequence generated by Algorithm~\\ref{algo:BPLA} with $z_0\\in \\Ima(B)$ and $\\limsup_{n\\to +\\infty} \\mu_n =\\overline{\\mu} <+\\infty$.\n","start":53177,"end":53682}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":["t:full","t:full_cvg","t:full_rate","t:full_rate_finite","t:full_rate_linear","t:full_rate_sublinear"],"declared_external":false,"source_only":false,"title":"扩展势函数","source_units":[{"raw_tex":"Define\n\\begin{align*}\n\\mathcal{F}(\\mathbf{x},y,z,t):=\\mathcal{L}_\\rho(\\mathbf{x},y,z) +c\\|y-t\\|^2, \\text{~~where~} c=\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}.\n\\end{align*}\n","start":53682,"end":53872}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","labels":["t:full","t:full_cvg","t:full_rate","t:full_rate_finite","t:full_rate_linear","t:full_rate_sublinear"],"declared_external":false,"source_only":false,"title":"KL分支假设","source_units":[{"raw_tex":"Suppose that $\\mathcal{F}$ is a KL function. Then the sequence $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ converges to a stationary point $(\\mathbf{x}^{*},y^{*},z^{*})$ of $\\mathcal{L}_\\rho$ and \\begin{align*}\n","start":53915,"end":54130},{"raw_tex":"Suppose that $\\mathcal{F}$ is a KL function with exponent $\\kappa\\in [0,1)$. Then the following hold:\n","start":54269,"end":54371}]},{"id":"S018","semantic_type":"theorem","kind":"theorem","labels":["t:full","t:full_cvg","t:full_rate","t:full_rate_finite","t:full_rate_linear","t:full_rate_sublinear"],"declared_external":false,"source_only":false,"title":"完整收敛与KL速率","source_units":[{"raw_tex":"\\begin{theorem}[Full sequential convergence]\n\\label{t:full}\nSuppose that Assumption~\\ref{a:standing} holds, that $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, that $H$ is bounded below, that $G$ is differentiable with $\\ell_G$-Lipschitz continuous gradient, and that, for all $i\\in \\{1, \\dots, m\\}$, $\\nabla{\\phi_i}$ is $\\ell_{\\phi}$-Lipschitz continuous. Let $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ be the sequence generated by Algorithm~\\ref{algo:BPLA} with $z_0\\in \\Ima(B)$ and $\\limsup_{n\\to +\\infty} \\mu_n =\\overline{\\mu} <+\\infty$.\nDefine\n\\begin{align*}\n\\mathcal{F}(\\mathbf{x},y,z,t):=\\mathcal{L}_\\rho(\\mathbf{x},y,z) +c\\|y-t\\|^2, \\text{~~where~} c=\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}.\n\\end{align*}\n\\begin{enumerate}\n\\item\\label{t:full_cvg} \nSuppose that $\\mathcal{F}$ is a KL function. Then the sequence $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ converges to a stationary point $(\\mathbf{x}^{*},y^{*},z^{*})$ of $\\mathcal{L}_\\rho$ and \\begin{align*}\n\\sum_{n=0}^{+\\infty} \\|(\\mathbf{x}_{n+1},y_{n+1},z_{n+1})-(\\mathbf{x}_{n},y_{n},z_{n})\\| < +\\infty.\n\\end{align*}\n\\item\\label{t:full_rate} \nSuppose that $\\mathcal{F}$ is a KL function with exponent $\\kappa\\in [0,1)$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{t:full_rate_finite}\nIf $\\kappa =0$, then $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ converges to $(\\mathbf{x}^{*},y^{*},z^{*})$ in a finite number of steps. \n\\item\\label{t:full_rate_linear}\nIf $\\kappa\\in (0,\\frac{1}{2}]$, then there exist $\\overline{\\gamma}\\in \\mathbb{R}_{++}$ and $\\zeta\\in \\left(0,1\\right)$ such that, for all $n\\in \\mathbb{N}$, \n\\begin{align*}\n&\\|(\\mathbf{x}_{n},y_{n},z_{n})-(\\mathbf{x}^{*},y^{*},z^{*})\\| \\leq \\overline{\\gamma}\\zeta^{\\frac{n}{2}}, \\\\\n&\\|\\mathbf{Ax}_n+By_n-b\\| \\leq \\overline{\\gamma} \\zeta^{\\frac{n}{2}}, \\\\\n&\\lvert \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) -\\mathcal{L}_\\rho(\\mathbf{x}^{*},y^{*},z^{*}) \\rvert \\leq \\overline{\\gamma}\\zeta^n, \\\\\n\\text{and~} &|F(\\mathbf{x}_n,y_n)-F(\\mathbf{x}^{*},y^{*})| \\leq \\overline{\\gamma} \\zeta^{\\frac{n}{2}}.\n\\end{align*}\n\\item\\label{t:full_rate_sublinear}\nIf $\\kappa\\in (\\frac{1}{2},1)$, then there exists $\\overline{\\gamma}\\in \\mathbb{R}_{++}$ such that, for all $n\\in \\mathbb{N}$,\n\\begin{align*}\n&\\|(\\mathbf{x}_{n},y_{n},z_{n})-(\\mathbf{x}^{*},y^{*},z^{*})\\| \\leq \\overline{\\gamma} n^{-\\frac{1-\\kappa}{2\\kappa-1}}, \\\\\n&\\|\\mathbf{Ax}_n+By_n-b\\| \\leq \\overline{\\gamma} n^{-\\frac{1-\\kappa}{2\\kappa-1}}, \\\\\n&\\lvert \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) -\\mathcal{L}_\\rho(\\mathbf{x}^{*},y^{*},z^{*})\\rvert \\leq \\overline{\\gamma} n^{-\\frac{2-2\\kappa}{2\\kappa-1}}, \\\\\n\\text{and~} &|F(\\mathbf{x}_n,y_n)-F(\\mathbf{x}^{*},y^{*})| \\leq \\overline{\\gamma} n^{-\\frac{1-\\kappa}{2\\kappa-1}}.\n\\end{align*}\n\\end{enumerate}\n\\end{enumerate}\n\\end{theorem}","start":53117,"end":55930}]},{"id":"S019","semantic_type":"lemma","kind":"lemma","labels":["l:cal","l:cal_sep","l:cal_sum"],"declared_external":false,"source_only":false,"title":"次微分求和法则","source_units":[{"raw_tex":"\\begin{lemma}[Calculus rules]\n\\label{l:cal}\nLet $f, g\\colon \\mathbb{R}^d \\to (-\\infty, +\\infty]$ be proper lower semicontinuous functions, and let $x \\in \\mathbb{R}^d$. \n\\begin{enumerate}\n\\item\\label{l:cal_sum} \n(Sum rule) Suppose that $f$ is finite at $x$ and $g$ is locally Lipschitz around $\\overline{x}$. Then $\\partial_L (f + g)(\\overline{x}) \\subseteq \\partial_L f(\\overline{x})+\\partial_L g(\\overline{x})$. Moreover, if $g$ is strictly differentiable at $\\overline{x}$, then $\\partial_L(f + g)(\\overline{x}) = \\partial_Lf(\\overline{x}) + \\nabla g(\\overline{x})$.\n","start":16647,"end":17217}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","labels":["l:cal","l:cal_sep","l:cal_sum"],"declared_external":false,"source_only":false,"title":"可分和次微分","source_units":[{"raw_tex":"Let $f, g\\colon \\mathbb{R}^d \\to (-\\infty, +\\infty]$ be proper lower semicontinuous functions, and let $x \\in \\mathbb{R}^d$. \n\\begin{enumerate}\n","start":16691,"end":16835},{"raw_tex":"\\item\\label{l:cal_sep} \n(Separable sum rule) If $f(\\mathbf{x})=\\sum_{i=1}^{m}f_i(x_i)$ with $\\mathbf{x}=(x_1,\\dots,x_m)$, then $\\partial_L f(\\mathbf{x}) = \\partial_L f_1(x_1) \\times \\dots \\times \\partial_L f_m(x_m)$. \n\\end{enumerate}\n\\end{lemma}","start":17217,"end":17462}]},{"id":"S021","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"求和法则作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{l:cal_sum} follows from \\cite[Proposition~1.107(ii) and Theorem~3.36]{Mordukhovich2006}, while \\ref{l:cal_sep} follows from \\cite[Proposition~10.5]{Rockafellar1998}.\n\\end{proof}","start":17463,"end":17659}]},{"id":"S022","semantic_type":"lemma","kind":"lemma","labels":["l:upsemicont"],"declared_external":false,"source_only":false,"title":"次梯度聚点封闭性","source_units":[{"raw_tex":"\\begin{lemma}[Upper semicontinuity of subdifferential]\n\\label{l:upsemicont}\nLet $f\\colon \\mathbb{R}^d \\to [-\\infty, +\\infty]$ be Lipschitz continuous around $x \\in \\mathbb{R}^d$ with $|f(x)| <+\\infty$, and consider sequences $(x_n)_{n\\in \\mathbb{N}}$ and $(x_n^*)_{n\\in \\mathbb{N}}$ in $\\mathbb{R}^d$ such that $x_n \\to x$ and, for all ${n\\in \\mathbb{N}}$, $x_n^* \\in \\partial_L f(x_n)$. \nThen $(x_n^*)_{n\\in \\mathbb{N}}$ is bounded and its cluster points are contained in $\\partial_L f(x)$.\n\\end{lemma}","start":17662,"end":18165}]},{"id":"S023","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"次梯度引理作者证明","source_units":[{"raw_tex":"\\begin{proof}\nThis follows from \\cite[Lemma 2.2]{Tan2023}.\n\\end{proof}","start":18166,"end":18236}]},{"id":"S024","semantic_type":"proposition","kind":"proposition","labels":["p:Bregman","p:Bregman_1","p:Bregman_2","p:Bregman_3","p:Bregman_4"],"declared_external":false,"source_only":false,"title":"Bregman非负性","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{p:Bregman}\nLet $\\phi: \\mathcal{H} \\to \\mathbb{R}$ be a differentiable convex function and $D_{\\phi}$ is the Bregman distance associated with $\\phi$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{p:Bregman_1} \nFor all $u, v \\in \\mathcal{H}$, $D_{\\phi}(u, v)\\geq 0$, and $D_{\\phi}(u, v)=0$ if and only if $u = v$. \n","start":19038,"end":19387}]},{"id":"S025","semantic_type":"proposition","kind":"proposition","labels":["p:Bregman","p:Bregman_1","p:Bregman_2","p:Bregman_3","p:Bregman_4"],"declared_external":false,"source_only":false,"title":"第一变量凸性","source_units":[{"raw_tex":"Let $\\phi: \\mathcal{H} \\to \\mathbb{R}$ be a differentiable convex function and $D_{\\phi}$ is the Bregman distance associated with $\\phi$. Then the following hold:\n","start":19076,"end":19239},{"raw_tex":"\n\\item\\label{p:Bregman_2} \nFor each $v \\in \\mathcal{H}$, $D_{\\phi}(\\cdot, v)$ is convex. \n","start":19387,"end":19477}]},{"id":"S026","semantic_type":"proposition","kind":"proposition","labels":["p:Bregman","p:Bregman_1","p:Bregman_2","p:Bregman_3","p:Bregman_4"],"declared_external":false,"source_only":false,"title":"Bregman二次下界","source_units":[{"raw_tex":"Let $\\phi: \\mathcal{H} \\to \\mathbb{R}$ be a differentiable convex function and $D_{\\phi}$ is the Bregman distance associated with $\\phi$. Then the following hold:\n","start":19076,"end":19239},{"raw_tex":"\n\\item\\label{p:Bregman_3} \nIf $\\phi$ is $\\alpha$-strongly convex,  then, for all $u, v \\in \\mathcal{H}$, $D_{\\phi}(u, v) \\geq \\frac{\\alpha}{2} \\|u-v\\|^2$. \n","start":19477,"end":19633}]},{"id":"S027","semantic_type":"proposition","kind":"proposition","labels":["p:Bregman","p:Bregman_1","p:Bregman_2","p:Bregman_3","p:Bregman_4"],"declared_external":false,"source_only":false,"title":"Bregman二次上界","source_units":[{"raw_tex":"Let $\\phi: \\mathcal{H} \\to \\mathbb{R}$ be a differentiable convex function and $D_{\\phi}$ is the Bregman distance associated with $\\phi$. Then the following hold:\n","start":19076,"end":19239},{"raw_tex":"\n\\item\\label{p:Bregman_4} \nIf $\\nabla \\phi$ is $\\ell_{\\phi}$-Lipschitz continuous, then, for all $u, v \\in \\mathcal{H}$, $D_{\\phi}(u,v)\\leq \\frac{\\ell_{\\phi}}{2}\\|u-v\\|^2$. \n\\end{enumerate}\n\\end{proposition}","start":19633,"end":19840}]},{"id":"S028","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"Bregman性质作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{p:Bregman_1} is given in \\cite[Section 1]{Bregman}. \\ref{p:Bregman_2}, \\ref{p:Bregman_3}, and \\ref{p:Bregman_4} can be easily verified by using the definitions. \n\\end{proof}","start":19841,"end":20033}]},{"id":"S029","semantic_type":"lemma","kind":"lemma","labels":["l:diff","l:diff_bounded","l:diff_descent"],"declared_external":false,"source_only":false,"title":"光滑下降估计","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{l:diff}\nLet $h\\colon \\mathbb{R}^d\\to \\mathbb{R}$ be a differentiable function with $\\ell$-Lipschitz continuous gradient. Then the following hold:\n\\begin{enumerate}\n\\item\\label{l:diff_descent}\nFor all $x, y\\in \\mathbb{R}^d$, $|h(y) -h(x) -\\langle \\nabla h(x), y -x\\rangle|\\leq \\frac{\\ell}{2}\\|y -x\\|^2$.\n","start":20164,"end":20488}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","labels":["l:diff","l:diff_bounded","l:diff_descent"],"declared_external":false,"source_only":false,"title":"梯度校正项有下界","source_units":[{"raw_tex":"Let $h\\colon \\mathbb{R}^d\\to \\mathbb{R}$ be a differentiable function with $\\ell$-Lipschitz continuous gradient. Then the following hold:\n","start":20193,"end":20331},{"raw_tex":"\\item\\label{l:diff_bounded}\nIf $h$ is bounded below and $\\ell >0$, then\n\\begin{align*}\n\\inf_{x\\in \\mathbb{R}^d} \\left(h(x) -\\frac{1}{2\\ell}\\|\\nabla h(x)\\|^2\\right) >-\\infty.    \n\\end{align*}\n\\end{enumerate}  \n\\end{lemma}","start":20488,"end":20708}]},{"id":"S031","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"光滑估计完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{l:diff_descent}: This follows from \\cite[Lemma~1.2.3]{Nesterov2018}.\n\n\\ref{l:diff_bounded}: Let $x\\in \\mathbb{R}^d$. Following the argument of \\cite[Equation~(1.2.19)]{Nesterov2018}, we apply \\ref{l:diff_descent} with $y =x -\\frac{1}{\\ell}\\nabla h(x)$ to obtain that\n\\begin{align*}\nh\\left(x -\\frac{1}{\\ell}\\nabla h(x)\\right) &\\leq h(x) +\\langle \\nabla h(x), y -x\\rangle +\\frac{\\ell}{2}\\|y -x\\|^2 \\\\\n&= h(x) -\\frac{1}{\\ell}\\|\\nabla h(x)\\|^2 +\\frac{1}{2\\ell}\\|\\nabla h(x)\\|^2 \\\\\n&= h(x) -\\frac{1}{2\\ell}\\|\\nabla h(x)\\|^2,\n\\end{align*}\nwhich together with the boundedness of $h$ implies the conclusion.\n\\end{proof}","start":20709,"end":21339}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"援引下降引理","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{l:diff_descent}: This follows from \\cite[Lemma~1.2.3]{Nesterov2018}.\n\n","start":20709,"end":20798}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"负梯度一步估计","source_units":[{"raw_tex":"\\ref{l:diff_bounded}: Let $x\\in \\mathbb{R}^d$. Following the argument of \\cite[Equation~(1.2.19)]{Nesterov2018}, we apply \\ref{l:diff_descent} with $y =x -\\frac{1}{\\ell}\\nabla h(x)$ to obtain that\n\\begin{align*}\nh\\left(x -\\frac{1}{\\ell}\\nabla h(x)\\right) &\\leq h(x) +\\langle \\nabla h(x), y -x\\rangle +\\frac{\\ell}{2}\\|y -x\\|^2 \\\\\n&= h(x) -\\frac{1}{\\ell}\\|\\nabla h(x)\\|^2 +\\frac{1}{2\\ell}\\|\\nabla h(x)\\|^2 \\\\\n&= h(x) -\\frac{1}{2\\ell}\\|\\nabla h(x)\\|^2,\n\\end{align*}\nwhich together with the boundedness of $h$ implies the conclusion.\n","start":20798,"end":21328}]},{"id":"S034","semantic_type":"lemma","kind":"lemma","labels":["l:norm","l:norm_M'x","l:norm_Mx"],"declared_external":false,"source_only":false,"title":"矩阵正向谱下界","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{l:norm}\nLet $L$ be a symmetric matrix in $\\mathbb{R}^{d\\times d}$ and let $M$ be a matrix in $\\mathbb{R}^{p\\times d}$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{l:norm_Mx}\nFor all $x\\in \\mathbb{R}^d$, $\\sqrt{\\lambda_{\\min}(M^\\top M)}\\|x\\| \\leq \\|Mx\\|$.\n","start":28813,"end":29100}]},{"id":"S035","semantic_type":"lemma","kind":"lemma","labels":["l:norm","l:norm_M'x","l:norm_Mx"],"declared_external":false,"source_only":false,"title":"对称矩阵谱估计","source_units":[{"raw_tex":"Let $L$ be a symmetric matrix in $\\mathbb{R}^{d\\times d}$ and let $M$ be a matrix in $\\mathbb{R}^{p\\times d}$. Then the following hold:\n","start":28842,"end":28978},{"raw_tex":"\\item\\label{l:norm_M'x}\nFor all $x\\in \\mathbb{R}^d$, $\\gamma(x^\\top Lx) \\leq \\|Lx\\|^2$, where $\\gamma =\\min\\{|\\lambda|: \\lambda \\text{~is a nonzero eigenvalue of~} L\\}$. Consequently, for all $z\\in \\Ima(M)$, $\\sqrt{\\lambda_*(M^\\top M)}\\|z\\| \\leq \\|M^\\top z\\|$, where $\\lambda_*(M^\\top M)$ is the smallest strictly positive eigenvalue of $M^\\top M$. \n","start":29100,"end":29450}]},{"id":"S036","semantic_type":"lemma","kind":"lemma","labels":["l:norm","l:norm_M'x","l:norm_Mx"],"declared_external":false,"source_only":false,"title":"像空间转置谱下界","source_units":[{"raw_tex":"Let $L$ be a symmetric matrix in $\\mathbb{R}^{d\\times d}$ and let $M$ be a matrix in $\\mathbb{R}^{p\\times d}$. Then the following hold:\n","start":28842,"end":28978},{"raw_tex":"\\item\\label{l:norm_M'x}\nFor all $x\\in \\mathbb{R}^d$, $\\gamma(x^\\top Lx) \\leq \\|Lx\\|^2$, where $\\gamma =\\min\\{|\\lambda|: \\lambda \\text{~is a nonzero eigenvalue of~} L\\}$. Consequently, for all $z\\in \\Ima(M)$, $\\sqrt{\\lambda_*(M^\\top M)}\\|z\\| \\leq \\|M^\\top z\\|$, where $\\lambda_*(M^\\top M)$ is the smallest strictly positive eigenvalue of $M^\\top M$. \n\\end{enumerate}\n\\end{lemma}","start":29100,"end":29477}]},{"id":"S037","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"谱引理源文完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{l:norm_Mx}: This follows from the fact that $\\|Mx\\|^2 =x^\\top (M^\\top M)x$.\n\n\\ref{l:norm_M'x}: \nThe first assertion follows from \\cite[Lemma~4.5]{BDL23} and its proof. Now, let $z\\in \\Ima(M)$. Then there exists $x\\in \\mathbb{R}^d$ such that $z =Mx$. Since $M^\\top M$ is a symmetric matrix in $\\mathbb{R}^{d\\times d}$ with all eigenvalues being nonnegative, we have from the first assertion that\n\\begin{align*}\n\\lambda_*(M^\\top M)\\|z\\|^2 =\\lambda_*(M^\\top M)(x^\\top M^\\top Mx) \\leq \\|M^\\top Mx\\|^2 =\\|M^\\top z\\|^2,    \n\\end{align*}\nwhich completes the proof.\n\\end{proof}\n","start":29478,"end":30067}]},{"id":"S038","semantic_type":"theorem","kind":"theorem","labels":["eq:decrease","t:cvg","t:cvg_decrease","t:cvg_seq"],"declared_external":false,"source_only":false,"title":"子序列收敛定理","source_units":[{"raw_tex":"\\begin{theorem}[Subsequential convergence]\n\\label{t:cvg}\nSuppose that Assumption~\\ref{a:standing} holds. Let $(\\mathbf{x}_n,y_n,z_n)_{n\\in \\mathbb{N}}$ be the sequence generated by Algorithm~\\ref{algo:BPLA} and, for each $n\\in \\mathbb{N}^*$, define\n\\begin{align*}\nL_n =\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) +c\\|y_n-y_{n-1}\\|^2, \\text{~where~} c=\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}.     \n\\end{align*}\nThen the following hold:\n\\begin{enumerate}\n\\item\\label{t:cvg_decrease}\nFor all $n\\in \\mathbb{N}^*$,\n\\begin{align}\\label{eq:decrease}\nL_{n+1} +\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\delta_y\\|y_{n+1}-y_n\\|^2 \\leq L_n,\n\\end{align}\nwhere $\\delta_{\\mathbf{x}} =\\frac{\\mu\\alpha -\\ell_P -\\beta}{2} > 0$ and $\\delta_y =\\frac{\\lambda\\rho}{2} -\\frac{(\\ell_H+2\\nu\\ell_{\\psi})^2}{\\lambda\\rho} -\\frac{\\ell_H}{2} >0$.\n\n\\item\\label{t:cvg_seq}\nSuppose that $z_0\\in \\Ima(B)$, that $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, and that $H$ is bounded below. \nThen the sequence $(\\mathbf{x}_n,{y}_n,{z}_n)_{n\\in \\mathbb{N}}$ is bounded and $\\mathbf{x}_{n+1}-\\mathbf{x}_n\\to 0$, $y_{n+1}-y_n\\to 0$, $z_{n+1}-z_n\\to 0$, and $\\mathbf{Ax}_n +By_n \\to b$ as $n\\to +\\infty$. Furthermore, for every cluster point $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ of $(\\mathbf{x}_n,{y}_n,{z}_n)_{n\\in \\mathbb{N}}$, it holds that\n\\begin{align*}\n\\mathbf{A}\\overline{\\mathbf{x}} +b\\overline{y} =b \\text{~~and~~} \\lim_{n \\to +\\infty} F(\\mathbf{x}_{n },y_{n}) =\\lim_{n \\to +\\infty}\\mathcal{L}_\\rho(\\mathbf{x}_{n },y_{n},z_{n}) =\\mathcal{L}_\\rho(\\overline{\\mathbf{x}},\\overline{y},\\overline{z}) =F(\\overline{\\mathbf{x}},\\overline{y}),    \n\\end{align*}\nand that if $G$ is strictly differentiable and, for each $i \\in \\{1,\\dots,m\\}$, $\\nabla \\phi_i$ is $\\ell_{\\phi}$-Lipschitz continuous, then $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ is a stationary point of $\\mathcal{L}_\\rho$.\n\\end{enumerate}\n\\end{theorem}","start":30177,"end":32134}]},{"id":"S039","semantic_type":"definition","kind":"definition","labels":["t:cvg","t:cvg_decrease","t:cvg_seq"],"declared_external":false,"source_only":false,"title":"离散势与下降常数","source_units":[{"raw_tex":"Suppose that Assumption~\\ref{a:standing} holds. Let $(\\mathbf{x}_n,y_n,z_n)_{n\\in \\mathbb{N}}$ be the sequence generated by Algorithm~\\ref{algo:BPLA} and, for each $n\\in \\mathbb{N}^*$, define\n\\begin{align*}\nL_n =\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) +c\\|y_n-y_{n-1}\\|^2, \\text{~where~} c=\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}.     \n\\end{align*}\n","start":30234,"end":30607},{"raw_tex":"where $\\delta_{\\mathbf{x}} =\\frac{\\mu\\alpha -\\ell_P -\\beta}{2} > 0$ and $\\delta_y =\\frac{\\lambda\\rho}{2} -\\frac{(\\ell_H+2\\nu\\ell_{\\psi})^2}{\\lambda\\rho} -\\frac{\\ell_H}{2} >0$.\n","start":30855,"end":31031}]},{"id":"S040","semantic_type":"assumption","kind":"assumption","labels":["t:cvg","t:cvg_decrease","t:cvg_seq"],"declared_external":false,"source_only":false,"title":"有界性分支条件","source_units":[{"raw_tex":"\\item\\label{t:cvg_seq}\nSuppose that $z_0\\in \\Ima(B)$, that $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, and that $H$ is bounded below. \n","start":31032,"end":31188}]},{"id":"S041","semantic_type":"assumption","kind":"assumption","labels":["t:cvg","t:cvg_decrease","t:cvg_seq"],"declared_external":false,"source_only":false,"title":"驻点分支附加条件","source_units":[{"raw_tex":"and that if $G$ is strictly differentiable and, for each $i \\in \\{1,\\dots,m\\}$, $\\nabla \\phi_i$ is $\\ell_{\\phi}$-Lipschitz continuous, then $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ is a stationary point of $\\mathcal{L}_\\rho$.\n","start":31868,"end":32105}]},{"id":"S042","semantic_type":"proof","kind":"proof","labels":["eq:B'z","eq:BTZy","eq:LL","eq:L_n_bounded","eq:from x-update","eq:kn+","eq:update_step_x","eq:uppper_zn","eq:xx","eq:yy","eq:z-update","eq:zImB","eq:zy","eq:zz"],"declared_external":false,"source_only":false,"title":"子序列定理完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{t:cvg_decrease}: Let $n\\in \\mathbb{N}$. The update of $z_{n+1}$ yields\n\\begin{align}\\label{eq:z-update}\nz_{n+1} -z_n =\\rho(\\mathbf{Ax}_{n+1}+By_{n+1}-b).   \n\\end{align}\nBy the optimality condition for the update of $y_{n+1}$ in Step~\\ref{step:A2_2} of Algorithm~\\ref{algo:BPLA}, \n\\begin{align*}\n0 &= \\nabla H(y_{n+1}) + B^\\top z_n +\\rho B^\\top(\\mathbf{Ax}_{n+1} +By_{n+1} -b) +\\nu_n(\\nabla \\psi(y_{n+1}) - \\nabla \\psi(y_n)),\n\\end{align*}\nwhich combined with \\eqref{eq:z-update} yields\n\\begin{align}\\label{eq:B'z}\nB^\\top z_{n+1} =-\\nabla H(y_{n+1}) -\\nu_n(\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)).  \n\\end{align}\nFrom the definition of the augmented Lagrangian, we have that\n\\begin{align}\\label{eq:zz}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n}) &= \\langle z_{n+1}-z_n, \\mathbf{Ax}_{n+1}+By_{n+1}-b \\rangle \\notag\\\\\n&= \\frac{1}{\\rho}\\|z_{n+1}-z_n\\|^2 \n\\end{align}\nand that\n\\begin{align*}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_n,z_n) &= H(y_{n+1}) - H(y_n) + \\langle z_n, B(y_{n+1}-y_n) \\rangle \\\\\n&\\qquad + \\frac{\\rho}{2}(\\|\\mathbf{Ax}_{n+1}+By_{n+1}-b\\|^2 - \\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2).  \n\\end{align*}\nWriting $\\mathbf{Ax}_{n+1}+By_{n}-b =(\\mathbf{Ax}_{n+1}+By_{n+1}-b) -(By_{n+1}- By_n)$, we obtain that\n\\begin{align*}\n&\\|\\mathbf{Ax}_{n+1}+By_{n+1}-b\\|^2 - \\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2 \\\\\n&= -\\|B(y_{n+1}-y_n)\\|^2 +2\\langle  \\mathbf{Ax}_{n+1}+By_{n+1}-b, B(y_{n+1}-y_n)\\rangle \\\\\n&= -\\|B(y_{n+1}-y_n)\\|^2 +\\frac{2}{\\rho}\\langle  z_{n+1}-z_{n}, B(y_{n+1}-y_n)\\rangle,\n\\end{align*}\nand so\n\\begin{align*}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_n,z_n) = H(y_{n+1}) - H(y_n) +\\langle z_{n+1}, B(y_{n+1}-y_n) \\rangle -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2.\n\\end{align*}\nIn view of \\eqref{eq:B'z},\n\\begin{align*}\n\\langle z_{n+1}, B(y_{n+1}-y_n) \\rangle &=\\langle B^\\top z_{n+1}, y_{n+1}-y_n \\rangle \\\\\n&=-\\langle \\nabla H(y_{n+1}), y_{n+1}-y_n \\rangle -\\nu_n\\langle \\nabla \\psi(y_{n+1})- \\nabla \\psi(y_n), y_{n+1}-y_n \\rangle \\\\\n&\\leq -H(y_{n+1}) +H(y_n) +\\frac{\\ell_H}{2}\\|y_{n+1}-y_n\\|^2,\n\\end{align*}\nwhere the last inequality is from the $\\ell_H$-Lipschitz continuity of $\\nabla H$ and Lemma~\\ref{l:diff}\\ref{l:diff_descent}, and from the monotonicity of $\\nabla \\psi$. \nWe therefore obtain that\n\\begin{align}\\label{eq:yy}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_n,z_n) \\leq \\frac{\\ell_H}{2}\\|y_{n+1}-y_n\\|^2 -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2.    \n\\end{align}\nNext, by the update of $x_{i,n+1}$ in Step~\\ref{step:A2_2} of Algorithm~\\ref{algo:BPLA}, for all $i \\in \\{1,\\dots,m\\}$ and all $x_i \\in \\mathbb{R}^{d_i}$,\n\\begin{align}\\label{eq:update_step_x}\n&f_i(x_{i,n+1}) +\\langle \\nabla_i P(\\mathbf{x}_n) -g_{i,n}, x_{i,n+1} \\rangle +\\langle z_n, A_{i}x_{i,n+1} \\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,n+1}(x_{i,n+1}) +By_n -b\\|^2 +\\mu_n D_{\\phi_i}(x_{i,n+1},x_{i,n}) \\notag \\\\\n&\\leq  f_i(x_i) +\\langle \\nabla_i P(\\mathbf{x}_n) -g_{i,n}, x_i \\rangle +\\langle z_n, A_{i}x_i\\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,n+1}(x_i) +By_n -b\\|^2 +\\mu_n D_{\\phi_i}(x_i,x_{i,n}).\n\\end{align}\nLetting $x_i =x_{i,n}$, using Proposition~\\ref{p:Bregman}\\ref{p:Bregman_1}\\&\\ref{p:Bregman_3}, and summing up over $i \\in \\{1,\\dots,m\\}$, we derive that\n\\begin{align}\\label{eq:from x-update}\n&\\sum_{i=1}^m f_i(x_{i,n+1}) +\\langle \\nabla P(\\mathbf{x}_n) -g_n, \\mathbf{x}_{n+1} \\rangle +\\langle z_n, \\mathbf{Ax}_{n+1} \\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{x}_{n+1} +By_n -b\\|^2 +\\frac{\\mu\\alpha}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 \\notag\\\\\n&\\leq \\sum_{i=1}^m f_i(x_{i,n}) +\\langle \\nabla P(\\mathbf{x}_n) -g_n, \\mathbf{x}_{n} \\rangle +\\langle z_n, \\mathbf{Ax}_{n} \\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{x}_n +By_n -b\\|^2.\n\\end{align}\nOn the other hand,\n\\begin{align}\\label{eq:xx}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_n,z_n) \\notag\\\\\n&= \\sum_{i=1}^m f_i(x_{i,n+1}) -\\sum_{i=1}^m f_i(x_{i,n}) +P(\\mathbf{x}_{n+1}) -P(\\mathbf{x}_{n}) -G(\\mathbf{x}_{n+1}) +G(\\mathbf{x}_{n}) \\notag\\\\ \n&\\quad +\\langle z_n, \\mathbf{Ax}_{n+1}-\\mathbf{Ax}_{n}\\rangle +\\frac{\\rho}{2}\\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2 -\\frac{\\rho}{2}\\|\\mathbf{Ax}_{n}+By_{n}-b\\|^2 \\notag\\\\\n&\\leq \\sum_{i=1}^m f_i(x_{i,n+1}) -\\sum_{i=1}^m f_i(x_{i,n}) +\\langle \\nabla P(\\mathbf{x}_n), \\mathbf{x}_{n+1}-\\mathbf{x}_n \\rangle + \\frac{\\ell_P}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_n\\|^2 - \\langle g_n, \\mathbf{x}_{n+1} -\\mathbf{x}_{n} \\rangle + \\frac{\\beta}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 \\notag\\\\\n&\\quad +\\langle z_n, \\mathbf{Ax}_{n+1}-\\mathbf{Ax}_{n}\\rangle  + \\frac{\\rho}{2}\\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2 - \\frac{\\rho}{2}\\|\\mathbf{Ax}_{n}+By_{n}-b\\|^2 \\notag\\\\\n&\\leq -\\frac{\\mu\\alpha -\\ell_P -\\beta}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 =-\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2,\n\\end{align}\nwhere the first inequality is obtained by using Lemma~\\ref{l:diff}\\ref{l:diff_descent} on the $\\ell_P$-Lipschitz continuity of $\\nabla P$ and \\cite[Lemma~4.1]{BDL22} on the $\\beta$-weak convexity of $G$, while the last inequality follows from \\eqref{eq:from x-update}. Summing up three relations \\eqref{eq:zz}, \\eqref{eq:yy}, and \\eqref{eq:xx} yields\n\\begin{align}\\label{eq:LL}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) - \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) \\notag\\\\\n&\\leq -\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\frac{\\ell_H}{2}\\|y_{n+1}-y_n\\|^2 -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2 +\\frac{1}{\\rho}\\|z_{n+1}-z_n\\|^2.\n\\end{align}\n\nNow, since $\\Ima(\\mathbf{A}) \\cup \\{b\\} \\subseteq \\Ima(B)$, it follows from \\eqref{eq:z-update} that\n\\begin{align}\\label{eq:zImB}\nz_{n+1}-z_n \\in \\Ima(B)    \n\\end{align}\nwhich, by Lemma~\\ref{l:norm}\\ref{l:norm_M'x}, yields \n\\begin{align*}\n\\|z_{n+1} -z_n\\| \\leq \\frac{1}{\\sqrt{\\lambda}}\\|B^\\top z_{n+1} -B^\\top z_n\\|.\n\\end{align*}\nLet $n\\in \\mathbb{N}^*$. We derive from \\eqref{eq:B'z} and the Lipschitz continuity of $\\nabla H$ and $\\nabla \\psi$ that\n\\begin{align}\\label{eq:BTZy}\n\\|B^\\top z_{n+1} -B^\\top z_n\\| &\\leq \\|\\nabla H(y_{n+1}) -\\nabla H(y_n)\\| +\\nu_n\\|\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)\\| +\\nu_{n-1}\\|\\nabla \\psi(y_n) - \\nabla \\psi(y_{n-1})\\| \\notag \\\\ \n&\\leq (\\ell_H +\\nu\\ell_{\\psi})\\|y_{n+1} -y_n\\| +\\nu\\ell_{\\psi}\\|y_n -y_{n-1}\\|,\n\\end{align}\nand thus, \n\\begin{align}\\label{eq:zy}\n\\|z_{n+1} -z_n\\| \\leq \\frac{1}{\\sqrt{\\lambda}}\\|B^\\top z_{n+1} -B^\\top z_n\\| \\leq \\frac{1}{\\sqrt{\\lambda}}((\\ell_H +\\nu\\ell_{\\psi})\\|y_{n+1} -y_n\\| +\\nu\\ell_{\\psi}\\|y_n -y_{n-1}\\|).\n\\end{align}\nApplying Cauchy--Schwarz inequality to two vectors $u =(\\sqrt{\\ell_H+\\nu\\ell_{\\psi}},\\sqrt{\\nu\\ell_{\\psi}})$ and $v =(\\sqrt{\\ell_H+\\nu\\ell_{\\psi}}\\|y_{n+1} -y_n\\|,\\sqrt{\\nu\\ell_{\\psi}}\\|y_n -y_{n-1}\\|)$, we obtain that\n\\begin{align*}\n\\|z_{n+1} -z_n\\|^2 \\leq \\frac{\\ell_H +2\\nu\\ell_{\\psi}}{\\lambda}((\\ell_H +\\nu\\ell_{\\psi})\\|y_{n+1} -y_n\\|^2 +\\nu\\ell_{\\psi}\\|y_n -y_{n-1}\\|^2).    \n\\end{align*}\nBy combining this with \\eqref{eq:LL} and using Lemma~\\ref{l:norm}\\ref{l:norm_Mx},\n\\begin{align*}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) - \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) \\notag\\\\\n&\\leq -\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\left(\\frac{(\\ell_H+2\\nu\\ell_{\\psi})(\\ell_H+\\nu\\ell_{\\psi})}{\\lambda\\rho} +\\frac{\\ell_H}{2}\\right)\\|y_{n+1}-y_n\\|^2 \\\\ \n&\\qquad -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2 +\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2 \\\\\n&\\leq -\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 -\\left(\\frac{\\lambda\\rho}{2} -\\frac{(\\ell_H+2\\nu\\ell_{\\psi})(\\ell_H+\\nu\\ell_{\\psi})}{\\lambda\\rho} -\\frac{\\ell_H}{2}\\right)\\|y_{n+1}-y_n\\|^2 \\\\\n&\\qquad +\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2,\n\\end{align*}\nor equivalently,\n\\begin{align*}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) +\\frac{(\\ell_H +2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_{n+1}-y_{n}\\|^2 + \\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\delta_y\\|y_{n+1}-y_n\\|^2 \\\\\n&\\leq \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) +\\frac{(\\ell_H +2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2,\n\\end{align*}\nwhich proves \\eqref{eq:decrease}. Here, we note that $\\delta_{\\mathbf{x}} =\\frac{\\mu\\alpha -\\ell_P -\\beta}{2} > 0$ since $\\mu > \\frac{\\ell_P+\\beta}{\\alpha}$ and that $\\delta_y =\\frac{\\lambda\\rho}{2} -\\frac{(\\ell_H+2\\nu\\ell_{\\psi})^2}{\\lambda\\rho} -\\frac{\\ell_H}{2} >0$ since $\\rho > \\frac{\\ell_H +\\sqrt{\\ell_H^2 +8(\\ell_H +2\\nu\\ell_{\\psi})^2}}{2\\lambda}$.\n\n\\ref{t:cvg_seq}: Let $n\\in \\mathbb{N}^*$. As $z_0 \\in \\Ima(B)$, we have from \\eqref{eq:zImB} that $z_n \\in \\Ima (B)$. Using Lemma~\\ref{l:norm}\\ref{l:norm_M'x}, \\eqref{eq:B'z}, and the Lipschitz continuity of $\\psi$ yields\n\\begin{align*}\n\\sqrt{\\lambda}\\|z_n\\|\\leq \\|B^\\top z_n\\| \\leq\\|\\nabla H(y_n)\\|+ \\nu_n\\|\\nabla \\psi(y_{n}) -\\nabla \\psi(y_{n-1})\\| \n\\leq \\|\\nabla H(y_n)\\|+ \\nu \\ell_{\\psi}\\|y_{n}-y_{n-1}\\|.\n\\end{align*}\nFor $\\omega\\in \\mathbb{R}_{++}$ to be chosen later, invoking Cauchy--Schwarz inequality with two vectors $u =(\\sqrt{\\omega},\\sqrt{\\nu\\ell_{\\psi}})$ and $v =(\\frac{1}{\\sqrt{\\omega}}\\|\\nabla H(y_n)\\|,\\sqrt{\\nu\\ell_{\\psi}}\\|y_n-y_{n-1}\\|)$, we have that\n\\begin{align}\\label{eq:uppper_zn}\n\\|z_n\\|^2 &\\leq \\frac{\\omega +\\nu\\ell_{\\psi}}{\\lambda} \\left(\\frac{1}{\\omega}\\|\\nabla H(y_n)\\|^2 +\\nu\\ell_{\\psi}\\|y_n-y_{n-1}\\|^2\\right) \\notag \\\\\n&= \\frac{1}{\\lambda}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right)\\|\\nabla H(y_n)\\|^2 +\\frac{(\\omega +\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda}\\|y_n-y_{n-1}\\|^2.\n\\end{align}\nCombining this with the fact that\n\\begin{align*}\n\\langle z_n, \\mathbf{Ax}_n+By_n-b \\rangle +\\frac{\\rho}{2}\\|\\mathbf{Ax}_n+By_n-b\\|^2 =\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} +\\mathbf{Ax}_n+By_n-b \\right\\|^2 -\\frac{1}{2\\rho}\\|z_n\\|^2,\n\\end{align*}\nwe deduce that\n\\begin{align*}\nL_n &=\\mathcal{L}_\\rho(\\mathbf{x}_n,y_n,z_n) + c\\|y_n-y_{n-1}\\|^2 \\\\\n&\\geq \\sum_{i=1}^m f_i(x_{i,n}) +H(y_n) +P(\\mathbf{x}_n) -G(\\mathbf{x}_n) \\\\ \n&\\quad +\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} + \\mathbf{Ax}_n+By_n-b \\right\\|^2 -\\frac{1}{2\\lambda\\rho}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right)\\|\\nabla H(y_n)\\|^2 +\\left(c -\\frac{(\\omega +\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{2\\lambda\\rho}\\right)\\|y_n-y_{n-1}\\|^2 \\\\\n&= \\left(\\sum_{i=1}^m f_i(x_{i,n}) +P(\\mathbf{x}_n) -G(\\mathbf{x}_n)\\right) +\\left(H(y_n) -\\frac{1}{2\\ell_H}\\|\\nabla H(y_n)\\|^2\\right) +\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} + \\mathbf{Ax}_n+By_n-b \\right\\|^2 \\\\ \n&\\quad +\\left(\\frac{1}{2\\ell_H} -\\frac{1}{2\\lambda\\rho}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right)\\right)\\|\\nabla H(y_n)\\|^2 +\\frac{\\nu\\ell_{\\psi}}{2\\lambda\\rho}(2\\ell_H +3\\nu\\ell_{\\psi} -\\omega)\\|y_n-y_{n-1}\\|^2.\n\\end{align*}\nNow, we note that\n\\begin{align*}\n\\rho > \\frac{\\ell_H +\\sqrt{\\ell_H^2 +8(\\ell_H +2\\nu\\ell_{\\psi})^2}}{2\\lambda} \\geq \\frac{\\ell_H +\\sqrt{\\ell_H^2 +8\\ell_H^2}}{2\\lambda} =\\frac{2\\ell_H}{\\lambda},    \n\\end{align*}\nand so $\\frac{1}{2\\ell_H} >\\frac{1}{\\lambda\\rho}$. Choosing $\\omega =1$ if $\\nu\\ell_{\\psi} =0$, and $\\omega =\\nu\\ell_{\\psi}$ if $\\nu\\ell_{\\psi} >0$, it holds that\n\\begin{align*}\n&\\theta :=\\frac{1}{2\\ell_H} -\\frac{1}{2\\lambda\\rho}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right) \\geq \\frac{1}{2\\ell_H} -\\frac{1}{2\\lambda\\rho}(1 +1) >0 \\text{~~and~~} \\\\\n&\\frac{\\nu\\ell_{\\psi}}{2\\lambda\\rho}(2\\ell_H +3\\nu\\ell_{\\psi} -\\omega) \\geq \\frac{\\nu\\ell_{\\psi}}{2\\lambda\\rho}(3\\nu\\ell_{\\psi} -\\omega) =\\frac{\\nu^2\\ell_{\\psi}^2}{\\lambda\\rho}. \n\\end{align*}\nThis together with \\ref{t:cvg_decrease} yields\n\\begin{align}\\label{eq:L_n_bounded}\nL_1 \\geq L_n &\\geq \\left(\\sum_{i=1}^m f_i(x_{i,n}) +P(\\mathbf{x}_n) -G(\\mathbf{x}_n)\\right) +\\left(H(y_n) -\\frac{1}{2\\ell_H}\\|\\nabla H(y_n)\\|^2\\right) +\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} + \\mathbf{Ax}_n+By_n-b \\right\\|^2 \\notag \\\\ \n&\\quad +\\theta\\|\\nabla H(y_n)\\|^2 +\\frac{\\nu^2\\ell_{\\psi}^2}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2,\n\\end{align}\nin which the first term on the right hand side is bounded below since $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, the second term is bounded below due to {the assumption on $H$ and Lemma~\\ref{l:diff}\\ref{l:diff_bounded}}, while the remaining terms are nonnegative due to their positive coefficients. Therefore, all the terms are bounded. Again, as $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, the boundedness of the first term implies that $(\\mathbf{x}_{n})_{n\\in \\mathbb{N}}$ is bounded. Since the last two terms are bounded, it follows from \\eqref{eq:uppper_zn} that $(z_n)_{n\\in \\mathbb{N}}$ is bounded. Since $z_{n+1}=z_n+\\rho(\\mathbf{Ax}_{n+1}+By_{n+1}-b)$ and $B$ has full column rank, we also obtain that $(y_n)_{n\\in \\mathbb{N}}$ is bounded. Consequently, the sequence $(\\mathbf{x}_n,y_n,z_n)_{n\\in \\mathbb{N}}$ is bounded.\n\nOn the other hand, since all the terms on the right hand side of \\eqref{eq:L_n_bounded} are bounded below, so is the sequence $(L_n)_{n\\in\\mathbb{N}}$. Combining with the nonincreasing property of $(L_n)_{n\\in\\mathbb{N}}$ due to \\ref{t:cvg_decrease}, it follows that $(L_n)_{n\\in\\mathbb{N}}$ is convergent. By rearranging \\eqref{eq:decrease} and performing telescoping, we then have that\n\\begin{align*}\n\\sum_{n=1}^{+\\infty}\\delta_y \\|y_{n+1}-y_n\\|^2 + \\delta_{\\mathbf{x}}\\sum_{n=1}^{+\\infty}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 \\leq \\sum_{n=1}^{+\\infty}(L_n - L_{n+1}) = L_1 - \\lim_{n\\to+\\infty} L_n < +\\infty,\n\\end{align*}\nwhich implies that $\\sum_{n=1}^{+\\infty}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_n\\|^2 < +\\infty$, $\\sum_{n=1}^{+\\infty}\\|y_{n+1}-y_n\\|^2 <+\\infty$, and by \\eqref{eq:zy}, $\\sum_{n=1}^{+\\infty}\\|z_{n+1}-z_n\\|^2 < +\\infty$. As a result, $\\mathbf{x}_{n+1}-\\mathbf{x}_{n} \\to 0$, $y_{n+1}-y_n \\to 0$, $z_{n+1}-z_n \\to 0$, and $\\mathbf{Ax}_n +By_n =(z_n -z_{n-1})/\\rho +b \\to b$ as $n \\to +\\infty$. It then follows from the definition of $L_n$ that the sequence $(\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}))_{n\\in\\mathbb{N}}$ is also convergent with $\\lim_{n\\to+\\infty} \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) =\\lim_{n\\to+\\infty}L_n$.\n\nLet $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ be a cluster point of the sequence  $(\\mathbf{x}_n,y_n,z_n)_{n\\in \\mathbb{N}}$. Then there exists a subsequence $(\\mathbf{x}_{k_n},y_{k_n},z_{k_n})_{n\\in \\mathbb{N}}$ that converges to $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$. In view of \\eqref{eq:update_step_x}, letting $n =k_n$ and using Proposition~\\ref{p:Bregman}\\ref{p:Bregman_3}\\&\\ref{p:Bregman_4}, we obtain that, for all $i \\in \\{1,\\dots,m\\}$ and all $x_i\\in \\mathbb{R}^{d_i}$,\n\\begin{align}\\label{eq:kn+}\n&f_i(x_{i,k_n+1}) +\\langle \\nabla_i P(\\mathbf{x}_{k_n})- g_{i,k_n}, x_{i,k_n+1} -x_i \\rangle + \\langle z_{k_n}, A_i x_{i,k_n+1} - A_ix_i\\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,k_n+1}(x_{i,k_n+1}) +By_{k_n} -b \\|^2 \\notag \\\\\n&\\quad  - \\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,k_n+1}(x_{i}) +By_{k_n} -b \\|^2 +\\frac{\\mu\\alpha}{2}\\|x_{i,k_n+1}-x_{i,k_n}\\|^2 - \\frac{\\nu\\ell_{\\phi}}{2}\\|x_{i}-x_{i,k_n}\\|^2 \\leq f_i(x_{i}).\n\\end{align}\nSince $\\nabla P$ is continuous, $\\nabla P(\\mathbf{x}_{k_n}) \\to \\nabla P(\\overline{\\mathbf{x}})$ as $n\\to +\\infty$. As $G +\\frac{\\beta}{2}\\|\\cdot\\|^2$ is a continuous convex function, it follows from \\cite[Example~9.14]{Rockafellar1998} that $G$ is locally Lipschitz continuous. Since $\\mathbf{x}_{k_n} \\to \\overline{\\mathbf{x}}$ as $n\\to +\\infty$, using Lemma~\\ref{l:upsemicont} and passing to a subsequence if necessary, we can and do assume that $g_{k_n}\\to \\overline{g} \\in \\partial_L G (\\overline{\\mathbf{x}})$ as $n\\to +\\infty$. Now, for each $i \\in \\{1,\\dots,m\\}$, letting $x_i =\\overline{x}_i$ and $n\\to +\\infty$ in \\eqref{eq:kn+} with noting that $x_{i,k_n+1} \\to \\overline{x}_i$, $y_{k_n} \\to \\overline{y}$, and $z_{k_n} \\to \\overline{z}$, we have that $\\limsup_{n \\to +\\infty} f_i(x_{i,k_n+1}) \\leq f_i(\\overline{x}_i)$, which together with the lower semicontinuity of $f_i$ yields\n\\begin{align*}\n\\lim_{n \\to +\\infty} f_i(x_{i,k_n+1}) = f_i(\\overline{x}_i).\n\\end{align*}\nAs $H$, $P$, and $G$ are continuous, it follows that $\\lim_{n \\to +\\infty}\\mathcal{L}_\\rho(\\mathbf{x}_{k_n +1 },y_{k_n+1},z_{k_n +1}) = \\mathcal{L}_\\rho(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$. Since the sequence $(\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}))_{n\\in\\mathbb{N}}$ is convergent, we deduce that\n\\begin{align*}\n\\lim_{n \\to +\\infty}\\mathcal{L}_\\rho(\\mathbf{x}_{n },y_{n},z_{n}) =\\mathcal{L}_\\rho(\\overline{\\mathbf{x}},\\overline{y},\\overline{z}).    \n\\end{align*}\n\nWe next show that $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ is a stationary point of $\\mathcal{L}_\\rho$. In \\eqref{eq:z-update} and \\eqref{eq:B'z} with $n$ replaced by $k_n-1$, letting $n\\to +\\infty$ and using the coninuity of $\\nabla H$ and the Lipschitz continuity of $\\nabla \\psi$, we obtain that $0 =\\mathbf{A\\overline{x}} + B\\overline{y} -b$ and $0 =\\nabla H(\\overline{y}) + B^\\top \\overline{z}$. In turn, letting $n \\to +\\infty$ in \\eqref{eq:kn+}, we have for all $i \\in \\{1,\\dots,m\\}$ and all $x_i\\in \\mathbb{R}^{d_i}$ that\n\\begin{align*}\n&f_i(\\overline{x}_i) +\\langle \\nabla_i P(\\overline{\\mathbf{x}})-\\overline{g}_i, \\overline{x}_i-x_i \\rangle + \\langle \\overline{z}, A_i\\overline{x}_i-A_ix_i \\rangle\\\\\n&\\quad +\\frac{\\rho}{2}\\|A_1\\overline{x}_1 +\\dots +A_{i-1}\\overline{x}_{i-1} +A_i\\overline{x}_i +A_{i+1}\\overline{x}_{i+1}+\\dots+A_m \\overline{x}_{m} +B\\overline{y} -b \\|^2 \\\\\n&\\quad -\\frac{\\rho}{2}\\|A_1\\overline{x}_1 +\\dots +A_{i-1}\\overline{x}_{i-1} +A_ix_i +A_{i+1}\\overline{x}_{i+1}+\\dots+A_m \\overline{x}_{m} +B\\overline{y} -b \\|^2 -\\frac{\\nu\\ell_{ \\phi}}{2}\\|x_i-\\overline{x}_i\\|^2 \\leq f_i(x_i),\n\\end{align*}\nwhich can be written as\n\\begin{align*}\nh_i(\\overline{x}_i) \\leq h_i(x_i),    \n\\end{align*}\nwhere $h_i(x_i) =f_i(x_i) +\\langle \\nabla_i P(\\overline{\\mathbf{x}}) -\\overline{g}_i, x_i \\rangle + \\langle \\overline{z}, A_ix_i \\rangle +\\frac{\\rho}{2}\\|A_1\\overline{x}_1 +\\dots +A_{i-1}\\overline{x}_{i-1} +A_ix_i +A_{i+1}\\overline{x}_{i+1}+\\dots+A_m \\overline{x}_{m} +B\\overline{y} -b \\|^2 +\\frac{\\nu\\ell_{\\phi}}{2}\\|x_i-\\overline{x}_i\\|^2$. It follows that, for all $i \\in \\{1,\\dots,m\\}$,\n\\begin{align*}\n0 &\\in \\partial_L h_i(\\overline{x}_i) =\\partial_L (f_i)(\\overline{x}_i) +\\nabla_i P(\\overline{\\mathbf{x}})- \\overline{g}_i +A_i^\\top \\overline{z} +\\rho A_i^\\top (\\mathbf{A}\\overline{\\mathbf{x}}+B\\overline{y}-b).\n\\end{align*}\nSince $\\mathbf{A}\\overline{\\mathbf{x}}+B\\overline{y}-b =0$, we obtain that\n\\begin{align*}\n0 \\in \\partial_L (f_1)(\\overline{x}_1)\\times \\cdots \\times \\partial_L (f_m)(\\overline{x}_m) +\\nabla P(\\overline{\\mathbf{x}}) -\\overline{g} +\\mathbf{A}^\\top \\overline{z},\n\\end{align*}\nwhich completes the proof due to Lemma~\\ref{l:cal}\\ref{l:cal_sum}\\&\\ref{l:cal_sep} and the strict differentiability of $G$.\n\\end{proof}","start":32135,"end":50863}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":["eq:B'z","eq:z-update"],"declared_external":false,"source_only":false,"title":"乘子更新与转置恒等式","source_units":[{"raw_tex":"\\ref{t:cvg_decrease}: Let $n\\in \\mathbb{N}$. The update of $z_{n+1}$ yields\n\\begin{align}\\label{eq:z-update}\nz_{n+1} -z_n =\\rho(\\mathbf{Ax}_{n+1}+By_{n+1}-b).   \n\\end{align}\nBy the optimality condition for the update of $y_{n+1}$ in Step~\\ref{step:A2_2} of Algorithm~\\ref{algo:BPLA}, \n\\begin{align*}\n0 &= \\nabla H(y_{n+1}) + B^\\top z_n +\\rho B^\\top(\\mathbf{Ax}_{n+1} +By_{n+1} -b) +\\nu_n(\\nabla \\psi(y_{n+1}) - \\nabla \\psi(y_n)),\n\\end{align*}\nwhich combined with \\eqref{eq:z-update} yields\n\\begin{align}\\label{eq:B'z}\nB^\\top z_{n+1} =-\\nabla H(y_{n+1}) -\\nu_n(\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)).  \n\\end{align}\n","start":32149,"end":32764}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":["eq:zz"],"declared_external":false,"source_only":false,"title":"乘子步能量增量","source_units":[{"raw_tex":"From the definition of the augmented Lagrangian, we have that\n\\begin{align}\\label{eq:zz}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n}) &= \\langle z_{n+1}-z_n, \\mathbf{Ax}_{n+1}+By_{n+1}-b \\rangle \\notag\\\\\n&= \\frac{1}{\\rho}\\|z_{n+1}-z_n\\|^2 \n\\end{align}\n","start":32764,"end":33072}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["eq:yy"],"declared_external":false,"source_only":false,"title":"原始y步下降","source_units":[{"raw_tex":"and that\n\\begin{align*}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_n,z_n) &= H(y_{n+1}) - H(y_n) + \\langle z_n, B(y_{n+1}-y_n) \\rangle \\\\\n&\\qquad + \\frac{\\rho}{2}(\\|\\mathbf{Ax}_{n+1}+By_{n+1}-b\\|^2 - \\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2).  \n\\end{align*}\nWriting $\\mathbf{Ax}_{n+1}+By_{n}-b =(\\mathbf{Ax}_{n+1}+By_{n+1}-b) -(By_{n+1}- By_n)$, we obtain that\n\\begin{align*}\n&\\|\\mathbf{Ax}_{n+1}+By_{n+1}-b\\|^2 - \\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2 \\\\\n&= -\\|B(y_{n+1}-y_n)\\|^2 +2\\langle  \\mathbf{Ax}_{n+1}+By_{n+1}-b, B(y_{n+1}-y_n)\\rangle \\\\\n&= -\\|B(y_{n+1}-y_n)\\|^2 +\\frac{2}{\\rho}\\langle  z_{n+1}-z_{n}, B(y_{n+1}-y_n)\\rangle,\n\\end{align*}\nand so\n\\begin{align*}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_n,z_n) = H(y_{n+1}) - H(y_n) +\\langle z_{n+1}, B(y_{n+1}-y_n) \\rangle -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2.\n\\end{align*}\nIn view of \\eqref{eq:B'z},\n\\begin{align*}\n\\langle z_{n+1}, B(y_{n+1}-y_n) \\rangle &=\\langle B^\\top z_{n+1}, y_{n+1}-y_n \\rangle \\\\\n&=-\\langle \\nabla H(y_{n+1}), y_{n+1}-y_n \\rangle -\\nu_n\\langle \\nabla \\psi(y_{n+1})- \\nabla \\psi(y_n), y_{n+1}-y_n \\rangle \\\\\n&\\leq -H(y_{n+1}) +H(y_n) +\\frac{\\ell_H}{2}\\|y_{n+1}-y_n\\|^2,\n\\end{align*}\nwhere the last inequality is from the $\\ell_H$-Lipschitz continuity of $\\nabla H$ and Lemma~\\ref{l:diff}\\ref{l:diff_descent}, and from the monotonicity of $\\nabla \\psi$. \nWe therefore obtain that\n\\begin{align}\\label{eq:yy}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_n,z_n) \\leq \\frac{\\ell_H}{2}\\|y_{n+1}-y_n\\|^2 -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2.    \n\\end{align}\n","start":33072,"end":34711}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["eq:from x-update","eq:update_step_x"],"declared_external":false,"source_only":false,"title":"分块极小值比较","source_units":[{"raw_tex":"Next, by the update of $x_{i,n+1}$ in Step~\\ref{step:A2_2} of Algorithm~\\ref{algo:BPLA}, for all $i \\in \\{1,\\dots,m\\}$ and all $x_i \\in \\mathbb{R}^{d_i}$,\n\\begin{align}\\label{eq:update_step_x}\n&f_i(x_{i,n+1}) +\\langle \\nabla_i P(\\mathbf{x}_n) -g_{i,n}, x_{i,n+1} \\rangle +\\langle z_n, A_{i}x_{i,n+1} \\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,n+1}(x_{i,n+1}) +By_n -b\\|^2 +\\mu_n D_{\\phi_i}(x_{i,n+1},x_{i,n}) \\notag \\\\\n&\\leq  f_i(x_i) +\\langle \\nabla_i P(\\mathbf{x}_n) -g_{i,n}, x_i \\rangle +\\langle z_n, A_{i}x_i\\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,n+1}(x_i) +By_n -b\\|^2 +\\mu_n D_{\\phi_i}(x_i,x_{i,n}).\n\\end{align}\nLetting $x_i =x_{i,n}$, using Proposition~\\ref{p:Bregman}\\ref{p:Bregman_1}\\&\\ref{p:Bregman_3}, and summing up over $i \\in \\{1,\\dots,m\\}$, we derive that\n\\begin{align}\\label{eq:from x-update}\n&\\sum_{i=1}^m f_i(x_{i,n+1}) +\\langle \\nabla P(\\mathbf{x}_n) -g_n, \\mathbf{x}_{n+1} \\rangle +\\langle z_n, \\mathbf{Ax}_{n+1} \\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{x}_{n+1} +By_n -b\\|^2 +\\frac{\\mu\\alpha}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 \\notag\\\\\n&\\leq \\sum_{i=1}^m f_i(x_{i,n}) +\\langle \\nabla P(\\mathbf{x}_n) -g_n, \\mathbf{x}_{n} \\rangle +\\langle z_n, \\mathbf{Ax}_{n} \\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{x}_n +By_n -b\\|^2.\n\\end{align}\n","start":34711,"end":35990}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":["eq:xx"],"declared_external":false,"source_only":false,"title":"原始x步下降","source_units":[{"raw_tex":"On the other hand,\n\\begin{align}\\label{eq:xx}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_n,z_n) \\notag\\\\\n&= \\sum_{i=1}^m f_i(x_{i,n+1}) -\\sum_{i=1}^m f_i(x_{i,n}) +P(\\mathbf{x}_{n+1}) -P(\\mathbf{x}_{n}) -G(\\mathbf{x}_{n+1}) +G(\\mathbf{x}_{n}) \\notag\\\\ \n&\\quad +\\langle z_n, \\mathbf{Ax}_{n+1}-\\mathbf{Ax}_{n}\\rangle +\\frac{\\rho}{2}\\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2 -\\frac{\\rho}{2}\\|\\mathbf{Ax}_{n}+By_{n}-b\\|^2 \\notag\\\\\n&\\leq \\sum_{i=1}^m f_i(x_{i,n+1}) -\\sum_{i=1}^m f_i(x_{i,n}) +\\langle \\nabla P(\\mathbf{x}_n), \\mathbf{x}_{n+1}-\\mathbf{x}_n \\rangle + \\frac{\\ell_P}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_n\\|^2 - \\langle g_n, \\mathbf{x}_{n+1} -\\mathbf{x}_{n} \\rangle + \\frac{\\beta}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 \\notag\\\\\n&\\quad +\\langle z_n, \\mathbf{Ax}_{n+1}-\\mathbf{Ax}_{n}\\rangle  + \\frac{\\rho}{2}\\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2 - \\frac{\\rho}{2}\\|\\mathbf{Ax}_{n}+By_{n}-b\\|^2 \\notag\\\\\n&\\leq -\\frac{\\mu\\alpha -\\ell_P -\\beta}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 =-\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2,\n\\end{align}\nwhere the first inequality is obtained by using Lemma~\\ref{l:diff}\\ref{l:diff_descent} on the $\\ell_P$-Lipschitz continuity of $\\nabla P$ and \\cite[Lemma~4.1]{BDL22} on the $\\beta$-weak convexity of $G$, while the last inequality follows from \\eqref{eq:from x-update}. Summing up three relations \\eqref{eq:zz}, \\eqref{eq:yy}, and \\eqref{eq:xx} yields\n","start":35990,"end":37422}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":["eq:LL"],"declared_external":false,"source_only":false,"title":"三步能量合并","source_units":[{"raw_tex":"where the first inequality is obtained by using Lemma~\\ref{l:diff}\\ref{l:diff_descent} on the $\\ell_P$-Lipschitz continuity of $\\nabla P$ and \\cite[Lemma~4.1]{BDL22} on the $\\beta$-weak convexity of $G$, while the last inequality follows from \\eqref{eq:from x-update}. Summing up three relations \\eqref{eq:zz}, \\eqref{eq:yy}, and \\eqref{eq:xx} yields\n\\begin{align}\\label{eq:LL}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) - \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) \\notag\\\\\n&\\leq -\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\frac{\\ell_H}{2}\\|y_{n+1}-y_n\\|^2 -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2 +\\frac{1}{\\rho}\\|z_{n+1}-z_n\\|^2.\n\\end{align}\n","start":37071,"end":37738}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["eq:BTZy","eq:zImB","eq:zy"],"declared_external":false,"source_only":false,"title":"乘子增量受y控制","source_units":[{"raw_tex":"Now, since $\\Ima(\\mathbf{A}) \\cup \\{b\\} \\subseteq \\Ima(B)$, it follows from \\eqref{eq:z-update} that\n\\begin{align}\\label{eq:zImB}\nz_{n+1}-z_n \\in \\Ima(B)    \n\\end{align}\nwhich, by Lemma~\\ref{l:norm}\\ref{l:norm_M'x}, yields \n\\begin{align*}\n\\|z_{n+1} -z_n\\| \\leq \\frac{1}{\\sqrt{\\lambda}}\\|B^\\top z_{n+1} -B^\\top z_n\\|.\n\\end{align*}\nLet $n\\in \\mathbb{N}^*$. We derive from \\eqref{eq:B'z} and the Lipschitz continuity of $\\nabla H$ and $\\nabla \\psi$ that\n\\begin{align}\\label{eq:BTZy}\n\\|B^\\top z_{n+1} -B^\\top z_n\\| &\\leq \\|\\nabla H(y_{n+1}) -\\nabla H(y_n)\\| +\\nu_n\\|\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)\\| +\\nu_{n-1}\\|\\nabla \\psi(y_n) - \\nabla \\psi(y_{n-1})\\| \\notag \\\\ \n&\\leq (\\ell_H +\\nu\\ell_{\\psi})\\|y_{n+1} -y_n\\| +\\nu\\ell_{\\psi}\\|y_n -y_{n-1}\\|,\n\\end{align}\nand thus, \n\\begin{align}\\label{eq:zy}\n\\|z_{n+1} -z_n\\| \\leq \\frac{1}{\\sqrt{\\lambda}}\\|B^\\top z_{n+1} -B^\\top z_n\\| \\leq \\frac{1}{\\sqrt{\\lambda}}((\\ell_H +\\nu\\ell_{\\psi})\\|y_{n+1} -y_n\\| +\\nu\\ell_{\\psi}\\|y_n -y_{n-1}\\|).\n\\end{align}\n","start":37739,"end":38731}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"正系数充分下降","source_units":[{"raw_tex":"Applying Cauchy--Schwarz inequality to two vectors $u =(\\sqrt{\\ell_H+\\nu\\ell_{\\psi}},\\sqrt{\\nu\\ell_{\\psi}})$ and $v =(\\sqrt{\\ell_H+\\nu\\ell_{\\psi}}\\|y_{n+1} -y_n\\|,\\sqrt{\\nu\\ell_{\\psi}}\\|y_n -y_{n-1}\\|)$, we obtain that\n\\begin{align*}\n\\|z_{n+1} -z_n\\|^2 \\leq \\frac{\\ell_H +2\\nu\\ell_{\\psi}}{\\lambda}((\\ell_H +\\nu\\ell_{\\psi})\\|y_{n+1} -y_n\\|^2 +\\nu\\ell_{\\psi}\\|y_n -y_{n-1}\\|^2).    \n\\end{align*}\nBy combining this with \\eqref{eq:LL} and using Lemma~\\ref{l:norm}\\ref{l:norm_Mx},\n\\begin{align*}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) - \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) \\notag\\\\\n&\\leq -\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\left(\\frac{(\\ell_H+2\\nu\\ell_{\\psi})(\\ell_H+\\nu\\ell_{\\psi})}{\\lambda\\rho} +\\frac{\\ell_H}{2}\\right)\\|y_{n+1}-y_n\\|^2 \\\\ \n&\\qquad -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2 +\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2 \\\\\n&\\leq -\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 -\\left(\\frac{\\lambda\\rho}{2} -\\frac{(\\ell_H+2\\nu\\ell_{\\psi})(\\ell_H+\\nu\\ell_{\\psi})}{\\lambda\\rho} -\\frac{\\ell_H}{2}\\right)\\|y_{n+1}-y_n\\|^2 \\\\\n&\\qquad +\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2,\n\\end{align*}\nor equivalently,\n\\begin{align*}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) +\\frac{(\\ell_H +2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_{n+1}-y_{n}\\|^2 + \\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\delta_y\\|y_{n+1}-y_n\\|^2 \\\\\n&\\leq \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) +\\frac{(\\ell_H +2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2,\n\\end{align*}\nwhich proves \\eqref{eq:decrease}. Here, we note that $\\delta_{\\mathbf{x}} =\\frac{\\mu\\alpha -\\ell_P -\\beta}{2} > 0$ since $\\mu > \\frac{\\ell_P+\\beta}{\\alpha}$ and that $\\delta_y =\\frac{\\lambda\\rho}{2} -\\frac{(\\ell_H+2\\nu\\ell_{\\psi})^2}{\\lambda\\rho} -\\frac{\\ell_H}{2} >0$ since $\\rho > \\frac{\\ell_H +\\sqrt{\\ell_H^2 +8(\\ell_H +2\\nu\\ell_{\\psi})^2}}{2\\lambda}$.\n","start":38731,"end":40697}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":["eq:uppper_zn"],"declared_external":false,"source_only":false,"title":"乘子绝对量估计","source_units":[{"raw_tex":"\\ref{t:cvg_seq}: Let $n\\in \\mathbb{N}^*$. As $z_0 \\in \\Ima(B)$, we have from \\eqref{eq:zImB} that $z_n \\in \\Ima (B)$. Using Lemma~\\ref{l:norm}\\ref{l:norm_M'x}, \\eqref{eq:B'z}, and the Lipschitz continuity of $\\psi$ yields\n\\begin{align*}\n\\sqrt{\\lambda}\\|z_n\\|\\leq \\|B^\\top z_n\\| \\leq\\|\\nabla H(y_n)\\|+ \\nu_n\\|\\nabla \\psi(y_{n}) -\\nabla \\psi(y_{n-1})\\| \n\\leq \\|\\nabla H(y_n)\\|+ \\nu \\ell_{\\psi}\\|y_{n}-y_{n-1}\\|.\n\\end{align*}\nFor $\\omega\\in \\mathbb{R}_{++}$ to be chosen later, invoking Cauchy--Schwarz inequality with two vectors $u =(\\sqrt{\\omega},\\sqrt{\\nu\\ell_{\\psi}})$ and $v =(\\frac{1}{\\sqrt{\\omega}}\\|\\nabla H(y_n)\\|,\\sqrt{\\nu\\ell_{\\psi}}\\|y_n-y_{n-1}\\|)$, we have that\n\\begin{align}\\label{eq:uppper_zn}\n\\|z_n\\|^2 &\\leq \\frac{\\omega +\\nu\\ell_{\\psi}}{\\lambda} \\left(\\frac{1}{\\omega}\\|\\nabla H(y_n)\\|^2 +\\nu\\ell_{\\psi}\\|y_n-y_{n-1}\\|^2\\right) \\notag \\\\\n&= \\frac{1}{\\lambda}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right)\\|\\nabla H(y_n)\\|^2 +\\frac{(\\omega +\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda}\\|y_n-y_{n-1}\\|^2.\n\\end{align}\n","start":40698,"end":41724}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":["eq:L_n_bounded"],"declared_external":false,"source_only":false,"title":"势函数下界分解","source_units":[{"raw_tex":"Combining this with the fact that\n\\begin{align*}\n\\langle z_n, \\mathbf{Ax}_n+By_n-b \\rangle +\\frac{\\rho}{2}\\|\\mathbf{Ax}_n+By_n-b\\|^2 =\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} +\\mathbf{Ax}_n+By_n-b \\right\\|^2 -\\frac{1}{2\\rho}\\|z_n\\|^2,\n\\end{align*}\nwe deduce that\n\\begin{align*}\nL_n &=\\mathcal{L}_\\rho(\\mathbf{x}_n,y_n,z_n) + c\\|y_n-y_{n-1}\\|^2 \\\\\n&\\geq \\sum_{i=1}^m f_i(x_{i,n}) +H(y_n) +P(\\mathbf{x}_n) -G(\\mathbf{x}_n) \\\\ \n&\\quad +\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} + \\mathbf{Ax}_n+By_n-b \\right\\|^2 -\\frac{1}{2\\lambda\\rho}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right)\\|\\nabla H(y_n)\\|^2 +\\left(c -\\frac{(\\omega +\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{2\\lambda\\rho}\\right)\\|y_n-y_{n-1}\\|^2 \\\\\n&= \\left(\\sum_{i=1}^m f_i(x_{i,n}) +P(\\mathbf{x}_n) -G(\\mathbf{x}_n)\\right) +\\left(H(y_n) -\\frac{1}{2\\ell_H}\\|\\nabla H(y_n)\\|^2\\right) +\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} + \\mathbf{Ax}_n+By_n-b \\right\\|^2 \\\\ \n&\\quad +\\left(\\frac{1}{2\\ell_H} -\\frac{1}{2\\lambda\\rho}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right)\\right)\\|\\nabla H(y_n)\\|^2 +\\frac{\\nu\\ell_{\\psi}}{2\\lambda\\rho}(2\\ell_H +3\\nu\\ell_{\\psi} -\\omega)\\|y_n-y_{n-1}\\|^2.\n\\end{align*}\nNow, we note that\n\\begin{align*}\n\\rho > \\frac{\\ell_H +\\sqrt{\\ell_H^2 +8(\\ell_H +2\\nu\\ell_{\\psi})^2}}{2\\lambda} \\geq \\frac{\\ell_H +\\sqrt{\\ell_H^2 +8\\ell_H^2}}{2\\lambda} =\\frac{2\\ell_H}{\\lambda},    \n\\end{align*}\nand so $\\frac{1}{2\\ell_H} >\\frac{1}{\\lambda\\rho}$. Choosing $\\omega =1$ if $\\nu\\ell_{\\psi} =0$, and $\\omega =\\nu\\ell_{\\psi}$ if $\\nu\\ell_{\\psi} >0$, it holds that\n\\begin{align*}\n&\\theta :=\\frac{1}{2\\ell_H} -\\frac{1}{2\\lambda\\rho}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right) \\geq \\frac{1}{2\\ell_H} -\\frac{1}{2\\lambda\\rho}(1 +1) >0 \\text{~~and~~} \\\\\n&\\frac{\\nu\\ell_{\\psi}}{2\\lambda\\rho}(2\\ell_H +3\\nu\\ell_{\\psi} -\\omega) \\geq \\frac{\\nu\\ell_{\\psi}}{2\\lambda\\rho}(3\\nu\\ell_{\\psi} -\\omega) =\\frac{\\nu^2\\ell_{\\psi}^2}{\\lambda\\rho}. \n\\end{align*}\nThis together with \\ref{t:cvg_decrease} yields\n\\begin{align}\\label{eq:L_n_bounded}\nL_1 \\geq L_n &\\geq \\left(\\sum_{i=1}^m f_i(x_{i,n}) +P(\\mathbf{x}_n) -G(\\mathbf{x}_n)\\right) +\\left(H(y_n) -\\frac{1}{2\\ell_H}\\|\\nabla H(y_n)\\|^2\\right) +\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} + \\mathbf{Ax}_n+By_n-b \\right\\|^2 \\notag \\\\ \n&\\quad +\\theta\\|\\nabla H(y_n)\\|^2 +\\frac{\\nu^2\\ell_{\\psi}^2}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2,\n\\end{align}\n","start":41724,"end":44030}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"全迭代有界性","source_units":[{"raw_tex":"in which the first term on the right hand side is bounded below since $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, the second term is bounded below due to {the assumption on $H$ and Lemma~\\ref{l:diff}\\ref{l:diff_bounded}}, while the remaining terms are nonnegative due to their positive coefficients. Therefore, all the terms are bounded. Again, as $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, the boundedness of the first term implies that $(\\mathbf{x}_{n})_{n\\in \\mathbb{N}}$ is bounded. Since the last two terms are bounded, it follows from \\eqref{eq:uppper_zn} that $(z_n)_{n\\in \\mathbb{N}}$ is bounded. Since $z_{n+1}=z_n+\\rho(\\mathbf{Ax}_{n+1}+By_{n+1}-b)$ and $B$ has full column rank, we also obtain that $(y_n)_{n\\in \\mathbb{N}}$ is bounded. Consequently, the sequence $(\\mathbf{x}_n,y_n,z_n)_{n\\in \\mathbb{N}}$ is bounded.\n\n","start":44030,"end":44904}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"平方可和与可行性","source_units":[{"raw_tex":"On the other hand, since all the terms on the right hand side of \\eqref{eq:L_n_bounded} are bounded below, so is the sequence $(L_n)_{n\\in\\mathbb{N}}$. Combining with the nonincreasing property of $(L_n)_{n\\in\\mathbb{N}}$ due to \\ref{t:cvg_decrease}, it follows that $(L_n)_{n\\in\\mathbb{N}}$ is convergent. By rearranging \\eqref{eq:decrease} and performing telescoping, we then have that\n\\begin{align*}\n\\sum_{n=1}^{+\\infty}\\delta_y \\|y_{n+1}-y_n\\|^2 + \\delta_{\\mathbf{x}}\\sum_{n=1}^{+\\infty}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 \\leq \\sum_{n=1}^{+\\infty}(L_n - L_{n+1}) = L_1 - \\lim_{n\\to+\\infty} L_n < +\\infty,\n\\end{align*}\nwhich implies that $\\sum_{n=1}^{+\\infty}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_n\\|^2 < +\\infty$, $\\sum_{n=1}^{+\\infty}\\|y_{n+1}-y_n\\|^2 <+\\infty$, and by \\eqref{eq:zy}, $\\sum_{n=1}^{+\\infty}\\|z_{n+1}-z_n\\|^2 < +\\infty$. As a result, $\\mathbf{x}_{n+1}-\\mathbf{x}_{n} \\to 0$, $y_{n+1}-y_n \\to 0$, $z_{n+1}-z_n \\to 0$, and $\\mathbf{Ax}_n +By_n =(z_n -z_{n-1})/\\rho +b \\to b$ as $n \\to +\\infty$. It then follows from the definition of $L_n$ that the sequence $(\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}))_{n\\in\\mathbb{N}}$ is also convergent with $\\lim_{n\\to+\\infty} \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) =\\lim_{n\\to+\\infty}L_n$.\n","start":44904,"end":46155}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["eq:kn+"],"declared_external":false,"source_only":false,"title":"聚点函数值一致","source_units":[{"raw_tex":"Let $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ be a cluster point of the sequence  $(\\mathbf{x}_n,y_n,z_n)_{n\\in \\mathbb{N}}$. Then there exists a subsequence $(\\mathbf{x}_{k_n},y_{k_n},z_{k_n})_{n\\in \\mathbb{N}}$ that converges to $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$. In view of \\eqref{eq:update_step_x}, letting $n =k_n$ and using Proposition~\\ref{p:Bregman}\\ref{p:Bregman_3}\\&\\ref{p:Bregman_4}, we obtain that, for all $i \\in \\{1,\\dots,m\\}$ and all $x_i\\in \\mathbb{R}^{d_i}$,\n\\begin{align}\\label{eq:kn+}\n&f_i(x_{i,k_n+1}) +\\langle \\nabla_i P(\\mathbf{x}_{k_n})- g_{i,k_n}, x_{i,k_n+1} -x_i \\rangle + \\langle z_{k_n}, A_i x_{i,k_n+1} - A_ix_i\\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,k_n+1}(x_{i,k_n+1}) +By_{k_n} -b \\|^2 \\notag \\\\\n&\\quad  - \\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,k_n+1}(x_{i}) +By_{k_n} -b \\|^2 +\\frac{\\mu\\alpha}{2}\\|x_{i,k_n+1}-x_{i,k_n}\\|^2 - \\frac{\\nu\\ell_{\\phi}}{2}\\|x_{i}-x_{i,k_n}\\|^2 \\leq f_i(x_{i}).\n\\end{align}\nSince $\\nabla P$ is continuous, $\\nabla P(\\mathbf{x}_{k_n}) \\to \\nabla P(\\overline{\\mathbf{x}})$ as $n\\to +\\infty$. As $G +\\frac{\\beta}{2}\\|\\cdot\\|^2$ is a continuous convex function, it follows from \\cite[Example~9.14]{Rockafellar1998} that $G$ is locally Lipschitz continuous. Since $\\mathbf{x}_{k_n} \\to \\overline{\\mathbf{x}}$ as $n\\to +\\infty$, using Lemma~\\ref{l:upsemicont} and passing to a subsequence if necessary, we can and do assume that $g_{k_n}\\to \\overline{g} \\in \\partial_L G (\\overline{\\mathbf{x}})$ as $n\\to +\\infty$. Now, for each $i \\in \\{1,\\dots,m\\}$, letting $x_i =\\overline{x}_i$ and $n\\to +\\infty$ in \\eqref{eq:kn+} with noting that $x_{i,k_n+1} \\to \\overline{x}_i$, $y_{k_n} \\to \\overline{y}$, and $z_{k_n} \\to \\overline{z}$, we have that $\\limsup_{n \\to +\\infty} f_i(x_{i,k_n+1}) \\leq f_i(\\overline{x}_i)$, which together with the lower semicontinuity of $f_i$ yields\n\\begin{align*}\n\\lim_{n \\to +\\infty} f_i(x_{i,k_n+1}) = f_i(\\overline{x}_i).\n\\end{align*}\nAs $H$, $P$, and $G$ are continuous, it follows that $\\lim_{n \\to +\\infty}\\mathcal{L}_\\rho(\\mathbf{x}_{k_n +1 },y_{k_n+1},z_{k_n +1}) = \\mathcal{L}_\\rho(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$. Since the sequence $(\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}))_{n\\in\\mathbb{N}}$ is convergent, we deduce that\n\\begin{align*}\n\\lim_{n \\to +\\infty}\\mathcal{L}_\\rho(\\mathbf{x}_{n },y_{n},z_{n}) =\\mathcal{L}_\\rho(\\overline{\\mathbf{x}},\\overline{y},\\overline{z}).    \n\\end{align*}\n","start":46156,"end":48595}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"聚点驻点性","source_units":[{"raw_tex":"We next show that $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ is a stationary point of $\\mathcal{L}_\\rho$. In \\eqref{eq:z-update} and \\eqref{eq:B'z} with $n$ replaced by $k_n-1$, letting $n\\to +\\infty$ and using the coninuity of $\\nabla H$ and the Lipschitz continuity of $\\nabla \\psi$, we obtain that $0 =\\mathbf{A\\overline{x}} + B\\overline{y} -b$ and $0 =\\nabla H(\\overline{y}) + B^\\top \\overline{z}$. In turn, letting $n \\to +\\infty$ in \\eqref{eq:kn+}, we have for all $i \\in \\{1,\\dots,m\\}$ and all $x_i\\in \\mathbb{R}^{d_i}$ that\n\\begin{align*}\n&f_i(\\overline{x}_i) +\\langle \\nabla_i P(\\overline{\\mathbf{x}})-\\overline{g}_i, \\overline{x}_i-x_i \\rangle + \\langle \\overline{z}, A_i\\overline{x}_i-A_ix_i \\rangle\\\\\n&\\quad +\\frac{\\rho}{2}\\|A_1\\overline{x}_1 +\\dots +A_{i-1}\\overline{x}_{i-1} +A_i\\overline{x}_i +A_{i+1}\\overline{x}_{i+1}+\\dots+A_m \\overline{x}_{m} +B\\overline{y} -b \\|^2 \\\\\n&\\quad -\\frac{\\rho}{2}\\|A_1\\overline{x}_1 +\\dots +A_{i-1}\\overline{x}_{i-1} +A_ix_i +A_{i+1}\\overline{x}_{i+1}+\\dots+A_m \\overline{x}_{m} +B\\overline{y} -b \\|^2 -\\frac{\\nu\\ell_{ \\phi}}{2}\\|x_i-\\overline{x}_i\\|^2 \\leq f_i(x_i),\n\\end{align*}\nwhich can be written as\n\\begin{align*}\nh_i(\\overline{x}_i) \\leq h_i(x_i),    \n\\end{align*}\nwhere $h_i(x_i) =f_i(x_i) +\\langle \\nabla_i P(\\overline{\\mathbf{x}}) -\\overline{g}_i, x_i \\rangle + \\langle \\overline{z}, A_ix_i \\rangle +\\frac{\\rho}{2}\\|A_1\\overline{x}_1 +\\dots +A_{i-1}\\overline{x}_{i-1} +A_ix_i +A_{i+1}\\overline{x}_{i+1}+\\dots+A_m \\overline{x}_{m} +B\\overline{y} -b \\|^2 +\\frac{\\nu\\ell_{\\phi}}{2}\\|x_i-\\overline{x}_i\\|^2$. It follows that, for all $i \\in \\{1,\\dots,m\\}$,\n\\begin{align*}\n0 &\\in \\partial_L h_i(\\overline{x}_i) =\\partial_L (f_i)(\\overline{x}_i) +\\nabla_i P(\\overline{\\mathbf{x}})- \\overline{g}_i +A_i^\\top \\overline{z} +\\rho A_i^\\top (\\mathbf{A}\\overline{\\mathbf{x}}+B\\overline{y}-b).\n\\end{align*}\nSince $\\mathbf{A}\\overline{\\mathbf{x}}+B\\overline{y}-b =0$, we obtain that\n\\begin{align*}\n0 \\in \\partial_L (f_1)(\\overline{x}_1)\\times \\cdots \\times \\partial_L (f_m)(\\overline{x}_m) +\\nabla P(\\overline{\\mathbf{x}}) -\\overline{g} +\\mathbf{A}^\\top \\overline{z},\n\\end{align*}\nwhich completes the proof due to Lemma~\\ref{l:cal}\\ref{l:cal_sum}\\&\\ref{l:cal_sep} and the strict differentiability of $G$.\n","start":48596,"end":50852}]},{"id":"S057","semantic_type":"proof","kind":"proof","labels":["eq:auxilary_decrease","eq:dxi","eq:dy","eq:dzdt"],"declared_external":false,"source_only":false,"title":"主定理完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nFor each $n\\in \\mathbb{N}$, set $w_n=(\\mathbf{x}_{n+1},y_{n+1},z_{n+1},y_n)$ and $\\Delta_n =\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\| +\\|y_{n+2}-y_{n+1}\\|$. Let $n \\in \\mathbb{N}^*$. According to Theorem~\\ref{t:cvg}, we have that\n\\begin{align}\\label{eq:auxilary_decrease}\n\\mathcal{F}(w_{n+1}) +\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\|^2 + \\delta_y \\|y_{n+2}-y_{n+1}\\|^2 \\leq \\mathcal{F}(w_{n}),\n\\end{align}\nthat $w_{n+1} - w_n \\to 0$ as $n\\to +\\infty$, that $(w_n)_{n\\in\\mathbb{N}}$ is bounded, that any of its cluster point $\\overline{w}=(\\overline{\\mathbf{x}},\\overline{y},\\overline{z},\\overline{y})$ is a stationary point of \\eqref{eq:prob}, and also that $\\mathcal{F}(w_{n}) \\to \\mathcal{F}(\\overline{w}) = \\mathcal{L}_{\\rho}(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ as $n\\to +\\infty$. Notice that\n\\begin{align*}\n\\Delta_n^2 \\leq \\left(\\frac{1}{\\delta_{\\mathbf{x}}} +\\frac{1}{\\delta_y}\\right) \\left(\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\|^2 + \\delta_y \\|y_{n+2}-y_{n+1}\\|^2\\right),    \n\\end{align*}\nwhich together with \\eqref{eq:auxilary_decrease} yields\n\\begin{align*}\n\\mathcal{F}(w_{n+1}) +C_1\\Delta_n^2 \\leq \\mathcal{F}(w_{n}), \\text{~~where~} C_1 =\\frac{\\delta_{\\mathbf{x}}\\delta_y}{\\delta_{\\mathbf{x}}+\\delta_y}.\n\\end{align*}\n\nNext, we have that\n\\begin{align*}\n\\partial_L ^\\mathbf{x} \\mathcal{F}(w_n) = \\partial_L \\left( \\sum_{i=1}^{m} f_i\\right)(\\mathbf{x}_{n+1}) + \\nabla P(\\mathbf{x}_{n+1}) - \\nabla G(\\mathbf{x}_{n+1}) + \\mathbf{A}^\\top z_{n+1} + \\rho \\mathbf{A}^\\top(\\mathbf{Ax}_{n+1} + By_{n+1} -b)    \n\\end{align*}\nand, for all $i \\in \\{1,\\dots,m\\}$,\n\\begin{align*}\n\\partial_L ^{x_i} \\mathcal{F}(w_n) = \\partial_L f_i(x_{i,n+1})+\\nabla_i P(\\mathbf{x}_{n+1}) - \\nabla_i G(\\mathbf{x}_{n+1}) + A_i^\\top z_{n+1} + \\rho A_i^\\top(\\mathbf{Ax}_{n+1} + By_{n+1} -b).    \n\\end{align*}\nBy revoking the optimality condition for the update of $x_{i,n+1}$ in Step~\\ref{step:A2_2} of Algorithm~\\ref{algo:BPLA},\n\\begin{align*}\n0 &\\in \\partial_L f_i(x_{i,n+1}) +\\nabla_i P(\\mathbf{x}_{n}) -  \\nabla_i G(\\mathbf{x}_n) + A_i^\\top z_n   +\\rho A_i^\\top(\\mathbf{A}\\mathbf{u}_{i,n+1}(x_{i,n+1} ) +By_n -b ) \\\\\n&\\quad +\\mu_n(\\nabla \\phi_i (x_{i,n+1})- \\nabla \\phi_i (x_{i,n})),    \n\\end{align*}\nwhich combined with the above equality implies that\n\\begin{align*}\n&(\\nabla_i P(\\mathbf{x}_{n+1}) -\\nabla_i P(\\mathbf{x}_{n})) -(\\nabla_i G(\\mathbf{x}_{n+1}) -\\nabla_i G(\\mathbf{x}_n)) +A_i^\\top(z_{n+1}-z_n) -\\mu_n(\\nabla \\phi_i (x_{i,n+1}) -\\nabla \\phi_i (x_{i,n})) \\\\\n&\\quad +\\rho A_i^\\top (A_{i+1}x_{i+1,n+1} +\\dots +A_mx_{m,n+1} -A_{i+1}x_{i+1,n} -\\dots -A_mx_{m,n} +By_{n+1} -By_n) \\in \\partial_L ^{x_i} \\mathcal{F}(w_n).\n\\end{align*}\nTherefore,\n\\begin{align}\\label{eq:dxi}\n\\dist(0,\\partial_L ^{x_i} \\mathcal{F}(w_n)) \n&\\leq \\|\\nabla_i P(\\mathbf{x}_{n+1}) -\\nabla_i P(\\mathbf{x}_{n})\\| +\\|\\nabla_i G(\\mathbf{x}_{n+1}) -\\nabla_i G(\\mathbf{x}_n)\\| \\notag \\\\\n&\\quad +\\|A_i^\\top(z_{n+1}-z_n)\\| +\\mu_n\\|\\nabla \\phi_i (x_{i,n+1}) -\\nabla \\phi_i (x_{i,n})\\| \\notag \\\\\n&\\quad +\\rho \\|A_i^\\top (A_{i+1}x_{i+1,n+1} +\\dots +A_mx_{m,n+1} -A_{i+1}x_{i+1,n} -\\dots -A_mx_{m,n} +By_{n+1} -By_n\\| \\notag \\\\ \n&\\leq \\ell_P\\|\\mathbf{x}_{n+1} - \\mathbf{x}_{n}\\| +\\ell_G\\|\\mathbf{x}_{n} - \\mathbf{x}_{n+1}\\| +\\|A_i^\\top\\|\\|z_{n+1}-z_n\\| +\\mu_n \\ell_{\\phi}\\|x_{i,n+1}-x_{i,n}\\| \\notag \\\\\n&\\quad +\\rho\\sum_{j=i+1}^{m} \\|A_i^\\top\\| \\|A_j\\| \\|x_{j,n+1}-x_{j,n}\\| +\\rho\\|A_i^\\top\\| \\|B\\| \\|y_{n+1}-y_n\\| \\notag \\\\\n&\\leq  \\left(\\ell_P +\\ell_G +\\mu_n\\ell_{\\phi} +\\rho\\sum_{j=i+1}^{m} \\|A_i^\\top\\| \\|A_j\\|\\right)\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\| +\\|A_i^\\top\\|\\|z_{n+1}-z_n\\| \\notag \\\\\n&\\quad +\\rho\\|A_i^\\top\\| \\|B\\| \\|y_{n+1}-y_n\\|.\n\\end{align}\n\nWe now see that\n\\begin{align*}\n\\partial_L ^y \\mathcal{F}(w_n) =\\nabla H(y_{n+1}) + B^\\top z_{n+1} +\\rho B^\\top(\\mathbf{Ax}_{n+1} + By_{n+1} -b) +2c(y_{n+1}-y_{n})    \n\\end{align*}\nand from the optimality condition for the update of $y_{n+1}$ in Algorithm~\\ref{algo:BPLA} that \n\\begin{align*}\n0 &= \\nabla H(y_{n+1}) + B^\\top z_n +\\rho B^\\top(\\mathbf{Ax}_{n+1} +By_{n+1} -b) +\\nu_n(\\nabla \\psi(y_{n+1}) - \\nabla \\psi(y_n)).\n\\end{align*}\nThis leads to\n\\begin{align*}\n\\partial_L ^y \\mathcal{F}(w_n) &=B^{\\top}(z_{n+1}-z_n) -\\nu_n(\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)) + 2c(y_{n+1}-y_{n}),   \n\\end{align*}\nand so\n\\begin{align}\\label{eq:dy}\n\\dist(0,\\partial_L ^{y} \\mathcal{F}(w_n)) \n&\\leq \\|B^{\\top}(z_{n+1}-z_n)\\| +\\nu_n\\|\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)\\| + 2c\\|y_{n+1}-y_{n})\\| \\notag \\\\\n&\\leq \\|B^{\\top}(z_{n+1}-z_n)\\| +(\\nu\\ell_\\psi +2c)\\|y_{n+1}-y_{n})\\|.\n\\end{align}\n\nSince $\\partial_L ^z \\mathcal{F}(w_n) =\\mathbf{Ax}_{n+1} + By_{n+1} -b =\\frac{1}{\\rho}(z_{n+1} -z_n)$ and\n$\\partial_L ^t \\mathcal{F}(w_n) =-2c(y_{n+1}-y_{n})$, it holds that\n\\begin{align}\\label{eq:dzdt}\n\\dist(0,\\partial_L ^{z} \\mathcal{F}(w_n)) =\\frac{1}{\\rho}\\|z_{n+1}-z_n\\| \\text{~~and~~}\n\\dist(0,\\partial_L ^{t} \\mathcal{F}(w_n)) =2c\\|y_{n+1}-y_n\\|.    \n\\end{align}\nCombining \\eqref{eq:dxi}, \\eqref{eq:dy}, and \\eqref{eq:dzdt}, noting that $\\limsup_{n\\to +\\infty} \\mu_n =\\overline{\\mu} <+\\infty$, and using \\eqref{eq:BTZy} and \\eqref{eq:zy} in the proof of Theorem~\\ref{t:cvg}, we derive that there exists $n_0\\in \\mathbb{N}^*$ and $C_2 \\in \\mathbb{R}_{++}$ such that, for all $n\\geq n_0$, \n\n\\begin{align*}\n\\dist(0, \\partial_L \\mathcal{F}(w_n)) \n\\leq C_2 \\left(\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\| +\\|y_{n+1}-y_n\\| +\\|y_{n}-y_{n-1}\\|\\right) \n\\leq C_2(\\Delta_{n-1} +\\Delta_{n-2}).\n\\end{align*}\n\n\\ref{t:full_cvg}: We see that all the conditions in the \\emph{abstract convergence} framework \\cite[Theorem~5.1]{BDL22} are satisfied with $I =\\{1,2\\}$, $\\lambda_1=\\lambda_2=1/2$, $\\alpha_n \\equiv C_1$, $\\beta_n \\equiv 1/(2C_2)$, and $\\varepsilon_n \\equiv 0 $. By \\cite[Theorem~5.1(i)]{BDL22}, \n\\begin{align*}\n\\sum_{n=0}^{+\\infty} (\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\|+\\|y_{n+2}-y_{n+1}\\|) =\\sum_{n=0}^{+\\infty} \\Delta_n < +\\infty,\n\\end{align*}\nwhich implies that $\\sum_{n=0}^{+\\infty} \\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\| <+\\infty$ and $\\sum_{n=0}^{+\\infty} \\|y_{n+1}-y_{n}\\| < +\\infty$. Together with \\eqref{eq:zy}, we derive that $\\sum_{n=0}^{+\\infty} \\|z_{n+1}-z_n\\|<+\\infty$, and hence \n\\begin{align*}\n\\sum_{n=0}^{+\\infty} \\|(\\mathbf{x}_{n+1},y_{n+1},z_{n+1})-(\\mathbf{x}_{n},y_{n},z_{n})\\| < +\\infty,\n\\end{align*}\nwhich yields the convergence of $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ to $(\\mathbf{x}^{*},y^{*},z^{*})$. In view of Theorem~\\ref{t:cvg}\\ref{t:cvg_seq}, $(\\mathbf{x}^{*},y^{*},z^{*})$ is a stationary point of $\\mathcal{L}_\\rho$.\n\n\\ref{t:full_rate_finite}: This follows from the arguments in the proof of \\cite[Theorem~5.1]{BDL22} and \\cite[Theorem~2(i)]{Attouch2007}. \n\n\\ref{t:full_rate_linear}: According to \\cite[Theorem~5.1(iv)]{BDL22}, there exist $\\gamma_0\\in \\mathbb{R}_{++}$ and $\\zeta\\in \\left(0,1\\right)$ such that, for all $n\\in \\mathbb{N}^*$, \n\\begin{align*}\n\\|(\\mathbf{x}_{n},y_{n})-(\\mathbf{x}^{*},y^{*})\\|\\leq\\gamma_0 \\zeta^{\\frac{n}{2}} \\text{~~and~~}\n|\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*)| \\leq \\gamma_0\\zeta^n.\n\\end{align*}\nWe derive that, for all $n\\in \\mathbb{N}^*$, $\\|\\mathbf{x}_n-\\mathbf{x}^{*}\\|\\leq \\gamma_0\\zeta^{\\frac{n}{2}}$ and $\\|y_n-y^{*}\\|\\leq \\gamma_0\\zeta^{\\frac{n}{2}}$, which yields \n\\begin{align*}\n\\|y_n-y_{n-1}\\| \\leq \\|y_n-y^{*}\\|+\\|y_{n-1}-y^{*}\\| \\leq (1+\\zeta^{-\\frac{1}{2}})\\gamma_0\\zeta^{\\frac{n}{2}}.    \n\\end{align*}\nBy passing to the limit in \\eqref{eq:B'z}, $B^\\top z^{*}=-\\nabla H(y^{*})$. Following the same steps as in \\eqref{eq:BTZy} and \\eqref{eq:zy}, we obtain that, for all $n\\in \\mathbb{N}^*$,\n\\begin{align*}\n\\|z_n-z^{*}\\| &\\leq \\frac{\\ell_H}{\\sqrt{\\lambda}}\\|y_n-y^{*}\\| + \\frac{\\nu \\ell_{\\psi}}{\\sqrt{\\lambda}}\\|y_{n}-y_{n-1}\\|\\leq \\gamma_1\\zeta^{\\frac{n}{2}}, \\text{~~where~} \\gamma_1 :=\\frac{\\ell_H\\gamma_0+(1+\\zeta^{-\\frac{1}{2}})\\nu\\ell_{\\psi}\\gamma_0}{\\sqrt{\\lambda}}.\n\\end{align*}\nConsequently,\n\\begin{align*}\n&\\|(\\mathbf{x}_{n},y_{n},z_{n})-(\\mathbf{x}^{*},y^{*},z^{*})\\| \\leq \\|\\mathbf{x}_{n}-\\mathbf{x}^{*}\\| + \\|y_{n}-y^{*}\\| + \\|z_{n}-z^{*}\\|\\leq (2\\gamma_0 +\\gamma_1)\\zeta^{\\frac{n}{2}}, \\\\\n&\\|\\mathbf{Ax}_n+By_n-b\\| =\\frac{1}{\\rho}\\|z_n-z_{n-1}\\| \\leq \\frac{1}{\\rho}(\\|z_n-z^{*}\\| +\\|z_{n-1}-z^{*}\\|) \\leq \\frac{(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho} \\zeta^{\\frac{n}{2}}, \\\\\n\\text{and~} &\\|z_n\\| \\leq \\|z_n-z^{*}\\| + \\|z^{*}\\| \\leq \\gamma_1 \\zeta^{\\frac{n}{2}} + \\|z^{*}\\|.\n\\end{align*}\n\nWe now deduce from the definition of $\\mathcal{F}$ that\n\\begin{align*}\n|\\mathcal{L}_{\\rho}(\\mathbf{x}_{n},y_{n},z_{n})- \\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})|&= \\left|\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*) - c\\|y_n-y_{n-1}\\|^2\\right|\\\\\n&\\leq |\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*)| + c\\|y_n-y_{n-1}\\|^2\\\\\n&\\leq \\gamma_2\\zeta^n, \\text{~~where~} \\gamma_2 :=\\gamma_0 + c(1+\\zeta^{-\\frac{1}{2}})^2\\gamma_0^2\n\\end{align*}\nand from the definition of $\\mathcal{L}_\\rho$ that \n\\begin{align*}\n|F(\\mathbf{x}_n,y_n)-F(\\mathbf{x}^{*},y^{*})| &= |\\mathcal{L}_{\\rho}(\\mathbf{x}_n,y_n,z_n)-\\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})-\\langle z_n, \\mathbf{Ax}_n + By_n-b\\rangle - \\frac{\\rho}{2}\\|\\mathbf{Ax}_n + By_n-b\\|^2| \\\\\n&\\leq |\\mathcal{L}_{\\rho}(\\mathbf{x}_n,y_n,z_n)-\\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})| + \\|z_n\\|\\|\\mathbf{Ax}_n + By_n-b\\| + \\frac{\\rho}{2}\\|\\mathbf{Ax}_n + By_n-b\\|^2  \\\\ \n&\\leq \\gamma_2\\zeta^n + \\left(\\gamma_1 \\zeta^{\\frac{n}{2}} + \\|z^{*}\\|\\right)\\frac{(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho} \\zeta^{\\frac{n}{2}}+ \\frac{(1+\\zeta^{-\\frac{1}{2}})^2\\gamma_1^2}{2\\rho}\\zeta^n \\\\\n&\\leq \\left(\\gamma_2 +\\frac{(3+\\zeta^{-\\frac{1}{2}})(1+\\zeta^{-\\frac{1}{2}})\\gamma_1^2}{2\\rho} +\\frac{\\|z^{*}\\|(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho}\\right)\\zeta^{\\frac{n}{2}},\n\\end{align*}\nwhere the last inequality follows from the fact that $\\zeta^n \\leq \\zeta^{\\frac{n}{2}}$ since $\\zeta \\in (0,1)$. By letting \n\\begin{align*}\n\\overline{\\gamma} =\\max \\left\\{ 2\\gamma_0+\\gamma_1, \\frac{(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho}, \\gamma_2 +\\frac{(3+\\zeta^{-\\frac{1}{2}})(1+\\zeta^{-\\frac{1}{2}})\\gamma_1^2}{2\\rho} +\\frac{\\|z^{*}\\|(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho} \\right\\} \n\\end{align*}\nand increasing it if necessary, we arrive at the conclusion.\n\n\\ref{t:full_rate_sublinear}: Following the arguments in \\cite[Theorem~5.1(iv)]{BDL22} and \\cite[Theorem~2(iii)]{Attouch2007}, we find $\\gamma_0 \\in \\mathbb{R}_{++}$ such that, for all $n\\in \\mathbb{N}^*$, \n\\begin{align*}\n\\|(\\mathbf{x}_{n},y_{n})-(\\mathbf{x}^{*},y^{*})\\|\\leq \\gamma_0 n^{-\\frac{1-\\kappa}{2\\kappa-1}} \\text{~~and~~}\n|\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*)| \\leq \\gamma_0 n^{-\\frac{2-2\\kappa}{2\\kappa-1}}.\n\\end{align*}\nSimilar to \\ref{t:full_rate_linear}, for all $n\\geq 2$, since $n-1\\geq \\frac{1}{2}n$ and $\\frac{1-\\kappa}{2\\kappa-1} >0$, we derive that \n\\begin{align*}\n&\\|(\\mathbf{x}_{n},y_{n},z_{n})-(\\mathbf{x}^{*},y^{*},z^{*})\\| \\leq (2\\gamma_0+\\gamma_1) n^{-\\frac{1-\\kappa}{2\\kappa-1}},\\\\\n&\\|\\mathbf{Ax}_n+By_n-b\\|\\leq \\frac{(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1}{\\rho} n^{-\\frac{1-\\kappa}{2\\kappa-1}},\\\\\n&|\\mathcal{L}_{\\rho}(\\mathbf{x}_{n},y_{n},z_{n})- \\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})|\\leq \\gamma_2 n^{-\\frac{2-2\\kappa}{2\\kappa-1}},\n\\end{align*}\nand that\n\\begin{align*}\n|F(\\mathbf{x}_n,y_n)-F(\\mathbf{x}^{*},y^{*})| &\\leq \\gamma_2 n^{-\\frac{2-2\\kappa}{2\\kappa-1}}+ \\left(\\gamma_1 n^{-\\frac{1-\\kappa}{2\\kappa-1}} + \\|z^{*}\\|\\right)\\frac{(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1}{\\rho} n^{-\\frac{1-\\kappa}{2\\kappa-1}} + \\frac{(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})^2\\gamma_1^2}{2\\rho} n^{-\\frac{2-2\\kappa}{2\\kappa-1}}\\\\\n&\\leq \\left(\\gamma_2 +\\frac{(3 +2^{\\frac{1-\\kappa}{2\\kappa-1}})(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1^2}{2\\rho} +\\frac{\\|z^{*}\\|(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1}{\\rho} \\right)n^{-\\frac{1-\\kappa}{2\\kappa-1}},\n\\end{align*}\nwhere $\\gamma_1 :=\\frac{\\ell_H\\gamma_0 +(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\nu\\ell_{\\psi}\\gamma_0}{\\sqrt{\\lambda}}$, $\\gamma_2 :=\\gamma_0 +c(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})^2\\gamma_0^2$, and the last inequality follows from the fact that $n^{-\\frac{2-2\\kappa}{2\\kappa-1}}<n^{-\\frac{1-\\kappa}{2\\kappa-1}}$ for $\\kappa\\in(\\frac{1}{2},1)$. By setting $\\overline{\\gamma}$ in the same way as in \\ref{t:full_rate_linear}, we obtain the conclusion.\n\\end{proof}","start":55931,"end":68289}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":["eq:auxilary_decrease"],"declared_external":false,"source_only":false,"title":"扩展序列与继承性质","source_units":[{"raw_tex":"For each $n\\in \\mathbb{N}$, set $w_n=(\\mathbf{x}_{n+1},y_{n+1},z_{n+1},y_n)$ and $\\Delta_n =\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\| +\\|y_{n+2}-y_{n+1}\\|$. Let $n \\in \\mathbb{N}^*$. According to Theorem~\\ref{t:cvg}, we have that\n\\begin{align}\\label{eq:auxilary_decrease}\n\\mathcal{F}(w_{n+1}) +\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\|^2 + \\delta_y \\|y_{n+2}-y_{n+1}\\|^2 \\leq \\mathcal{F}(w_{n}),\n\\end{align}\nthat $w_{n+1} - w_n \\to 0$ as $n\\to +\\infty$, that $(w_n)_{n\\in\\mathbb{N}}$ is bounded, that any of its cluster point $\\overline{w}=(\\overline{\\mathbf{x}},\\overline{y},\\overline{z},\\overline{y})$ is a stationary point of \\eqref{eq:prob}, and also that $\\mathcal{F}(w_{n}) \\to \\mathcal{F}(\\overline{w}) = \\mathcal{L}_{\\rho}(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ as $n\\to +\\infty$. Notice that\n","start":55945,"end":56768}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"标量步长下降条件","source_units":[{"raw_tex":"\\begin{align*}\n\\Delta_n^2 \\leq \\left(\\frac{1}{\\delta_{\\mathbf{x}}} +\\frac{1}{\\delta_y}\\right) \\left(\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\|^2 + \\delta_y \\|y_{n+2}-y_{n+1}\\|^2\\right),    \n\\end{align*}\nwhich together with \\eqref{eq:auxilary_decrease} yields\n\\begin{align*}\n\\mathcal{F}(w_{n+1}) +C_1\\Delta_n^2 \\leq \\mathcal{F}(w_{n}), \\text{~~where~} C_1 =\\frac{\\delta_{\\mathbf{x}}\\delta_y}{\\delta_{\\mathbf{x}}+\\delta_y}.\n\\end{align*}\n","start":56768,"end":57217}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"构造x块次梯度","source_units":[{"raw_tex":"Next, we have that\n\\begin{align*}\n\\partial_L ^\\mathbf{x} \\mathcal{F}(w_n) = \\partial_L \\left( \\sum_{i=1}^{m} f_i\\right)(\\mathbf{x}_{n+1}) + \\nabla P(\\mathbf{x}_{n+1}) - \\nabla G(\\mathbf{x}_{n+1}) + \\mathbf{A}^\\top z_{n+1} + \\rho \\mathbf{A}^\\top(\\mathbf{Ax}_{n+1} + By_{n+1} -b)    \n\\end{align*}\nand, for all $i \\in \\{1,\\dots,m\\}$,\n\\begin{align*}\n\\partial_L ^{x_i} \\mathcal{F}(w_n) = \\partial_L f_i(x_{i,n+1})+\\nabla_i P(\\mathbf{x}_{n+1}) - \\nabla_i G(\\mathbf{x}_{n+1}) + A_i^\\top z_{n+1} + \\rho A_i^\\top(\\mathbf{Ax}_{n+1} + By_{n+1} -b).    \n\\end{align*}\nBy revoking the optimality condition for the update of $x_{i,n+1}$ in Step~\\ref{step:A2_2} of Algorithm~\\ref{algo:BPLA},\n\\begin{align*}\n0 &\\in \\partial_L f_i(x_{i,n+1}) +\\nabla_i P(\\mathbf{x}_{n}) -  \\nabla_i G(\\mathbf{x}_n) + A_i^\\top z_n   +\\rho A_i^\\top(\\mathbf{A}\\mathbf{u}_{i,n+1}(x_{i,n+1} ) +By_n -b ) \\\\\n&\\quad +\\mu_n(\\nabla \\phi_i (x_{i,n+1})- \\nabla \\phi_i (x_{i,n})),    \n\\end{align*}\nwhich combined with the above equality implies that\n\\begin{align*}\n&(\\nabla_i P(\\mathbf{x}_{n+1}) -\\nabla_i P(\\mathbf{x}_{n})) -(\\nabla_i G(\\mathbf{x}_{n+1}) -\\nabla_i G(\\mathbf{x}_n)) +A_i^\\top(z_{n+1}-z_n) -\\mu_n(\\nabla \\phi_i (x_{i,n+1}) -\\nabla \\phi_i (x_{i,n})) \\\\\n&\\quad +\\rho A_i^\\top (A_{i+1}x_{i+1,n+1} +\\dots +A_mx_{m,n+1} -A_{i+1}x_{i+1,n} -\\dots -A_mx_{m,n} +By_{n+1} -By_n) \\in \\partial_L ^{x_i} \\mathcal{F}(w_n).\n\\end{align*}\n","start":57218,"end":58609}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":["eq:dxi"],"declared_external":false,"source_only":false,"title":"x块次梯度误差界","source_units":[{"raw_tex":"Therefore,\n\\begin{align}\\label{eq:dxi}\n\\dist(0,\\partial_L ^{x_i} \\mathcal{F}(w_n)) \n&\\leq \\|\\nabla_i P(\\mathbf{x}_{n+1}) -\\nabla_i P(\\mathbf{x}_{n})\\| +\\|\\nabla_i G(\\mathbf{x}_{n+1}) -\\nabla_i G(\\mathbf{x}_n)\\| \\notag \\\\\n&\\quad +\\|A_i^\\top(z_{n+1}-z_n)\\| +\\mu_n\\|\\nabla \\phi_i (x_{i,n+1}) -\\nabla \\phi_i (x_{i,n})\\| \\notag \\\\\n&\\quad +\\rho \\|A_i^\\top (A_{i+1}x_{i+1,n+1} +\\dots +A_mx_{m,n+1} -A_{i+1}x_{i+1,n} -\\dots -A_mx_{m,n} +By_{n+1} -By_n\\| \\notag \\\\ \n&\\leq \\ell_P\\|\\mathbf{x}_{n+1} - \\mathbf{x}_{n}\\| +\\ell_G\\|\\mathbf{x}_{n} - \\mathbf{x}_{n+1}\\| +\\|A_i^\\top\\|\\|z_{n+1}-z_n\\| +\\mu_n \\ell_{\\phi}\\|x_{i,n+1}-x_{i,n}\\| \\notag \\\\\n&\\quad +\\rho\\sum_{j=i+1}^{m} \\|A_i^\\top\\| \\|A_j\\| \\|x_{j,n+1}-x_{j,n}\\| +\\rho\\|A_i^\\top\\| \\|B\\| \\|y_{n+1}-y_n\\| \\notag \\\\\n&\\leq  \\left(\\ell_P +\\ell_G +\\mu_n\\ell_{\\phi} +\\rho\\sum_{j=i+1}^{m} \\|A_i^\\top\\| \\|A_j\\|\\right)\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\| +\\|A_i^\\top\\|\\|z_{n+1}-z_n\\| \\notag \\\\\n&\\quad +\\rho\\|A_i^\\top\\| \\|B\\| \\|y_{n+1}-y_n\\|.\n\\end{align}\n","start":58609,"end":59592}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":["eq:dy"],"declared_external":false,"source_only":false,"title":"y块次梯度误差界","source_units":[{"raw_tex":"We now see that\n\\begin{align*}\n\\partial_L ^y \\mathcal{F}(w_n) =\\nabla H(y_{n+1}) + B^\\top z_{n+1} +\\rho B^\\top(\\mathbf{Ax}_{n+1} + By_{n+1} -b) +2c(y_{n+1}-y_{n})    \n\\end{align*}\nand from the optimality condition for the update of $y_{n+1}$ in Algorithm~\\ref{algo:BPLA} that \n\\begin{align*}\n0 &= \\nabla H(y_{n+1}) + B^\\top z_n +\\rho B^\\top(\\mathbf{Ax}_{n+1} +By_{n+1} -b) +\\nu_n(\\nabla \\psi(y_{n+1}) - \\nabla \\psi(y_n)).\n\\end{align*}\nThis leads to\n\\begin{align*}\n\\partial_L ^y \\mathcal{F}(w_n) &=B^{\\top}(z_{n+1}-z_n) -\\nu_n(\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)) + 2c(y_{n+1}-y_{n}),   \n\\end{align*}\nand so\n\\begin{align}\\label{eq:dy}\n\\dist(0,\\partial_L ^{y} \\mathcal{F}(w_n)) \n&\\leq \\|B^{\\top}(z_{n+1}-z_n)\\| +\\nu_n\\|\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)\\| + 2c\\|y_{n+1}-y_{n})\\| \\notag \\\\\n&\\leq \\|B^{\\top}(z_{n+1}-z_n)\\| +(\\nu\\ell_\\psi +2c)\\|y_{n+1}-y_{n})\\|.\n\\end{align}\n","start":59593,"end":60470}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":["eq:dzdt"],"declared_external":false,"source_only":false,"title":"乘子与记忆变量误差","source_units":[{"raw_tex":"Since $\\partial_L ^z \\mathcal{F}(w_n) =\\mathbf{Ax}_{n+1} + By_{n+1} -b =\\frac{1}{\\rho}(z_{n+1} -z_n)$ and\n$\\partial_L ^t \\mathcal{F}(w_n) =-2c(y_{n+1}-y_{n})$, it holds that\n\\begin{align}\\label{eq:dzdt}\n\\dist(0,\\partial_L ^{z} \\mathcal{F}(w_n)) =\\frac{1}{\\rho}\\|z_{n+1}-z_n\\| \\text{~~and~~}\n\\dist(0,\\partial_L ^{t} \\mathcal{F}(w_n)) =2c\\|y_{n+1}-y_n\\|.    \n\\end{align}\n","start":60471,"end":60840}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"延迟相对误差条件","source_units":[{"raw_tex":"Combining \\eqref{eq:dxi}, \\eqref{eq:dy}, and \\eqref{eq:dzdt}, noting that $\\limsup_{n\\to +\\infty} \\mu_n =\\overline{\\mu} <+\\infty$, and using \\eqref{eq:BTZy} and \\eqref{eq:zy} in the proof of Theorem~\\ref{t:cvg}, we derive that there exists $n_0\\in \\mathbb{N}^*$ and $C_2 \\in \\mathbb{R}_{++}$ such that, for all $n\\geq n_0$, \n\n\\begin{align*}\n\\dist(0, \\partial_L \\mathcal{F}(w_n)) \n\\leq C_2 \\left(\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\| +\\|y_{n+1}-y_n\\| +\\|y_{n}-y_{n-1}\\|\\right) \n\\leq C_2(\\Delta_{n-1} +\\Delta_{n-2}).\n\\end{align*}\n","start":60840,"end":61366}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"抽象KL有限长度","source_units":[{"raw_tex":"\\ref{t:full_cvg}: We see that all the conditions in the \\emph{abstract convergence} framework \\cite[Theorem~5.1]{BDL22} are satisfied with $I =\\{1,2\\}$, $\\lambda_1=\\lambda_2=1/2$, $\\alpha_n \\equiv C_1$, $\\beta_n \\equiv 1/(2C_2)$, and $\\varepsilon_n \\equiv 0 $. By \\cite[Theorem~5.1(i)]{BDL22}, \n\\begin{align*}\n\\sum_{n=0}^{+\\infty} (\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\|+\\|y_{n+2}-y_{n+1}\\|) =\\sum_{n=0}^{+\\infty} \\Delta_n < +\\infty,\n\\end{align*}\n","start":61367,"end":61813}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"全变量有限长度与收敛","source_units":[{"raw_tex":"which implies that $\\sum_{n=0}^{+\\infty} \\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\| <+\\infty$ and $\\sum_{n=0}^{+\\infty} \\|y_{n+1}-y_{n}\\| < +\\infty$. Together with \\eqref{eq:zy}, we derive that $\\sum_{n=0}^{+\\infty} \\|z_{n+1}-z_n\\|<+\\infty$, and hence \n\\begin{align*}\n\\sum_{n=0}^{+\\infty} \\|(\\mathbf{x}_{n+1},y_{n+1},z_{n+1})-(\\mathbf{x}_{n},y_{n},z_{n})\\| < +\\infty,\n\\end{align*}\nwhich yields the convergence of $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ to $(\\mathbf{x}^{*},y^{*},z^{*})$. In view of Theorem~\\ref{t:cvg}\\ref{t:cvg_seq}, $(\\mathbf{x}^{*},y^{*},z^{*})$ is a stationary point of $\\mathcal{L}_\\rho$.\n","start":61813,"end":62425}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"零指数有限步收敛","source_units":[{"raw_tex":"\n\\ref{t:full_rate_finite}: This follows from the arguments in the proof of \\cite[Theorem~5.1]{BDL22} and \\cite[Theorem~2(i)]{Attouch2007}. \n","start":62425,"end":62565}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"线性速率基础估计","source_units":[{"raw_tex":"\\ref{t:full_rate_linear}: According to \\cite[Theorem~5.1(iv)]{BDL22}, there exist $\\gamma_0\\in \\mathbb{R}_{++}$ and $\\zeta\\in \\left(0,1\\right)$ such that, for all $n\\in \\mathbb{N}^*$, \n\\begin{align*}\n\\|(\\mathbf{x}_{n},y_{n})-(\\mathbf{x}^{*},y^{*})\\|\\leq\\gamma_0 \\zeta^{\\frac{n}{2}} \\text{~~and~~}\n|\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*)| \\leq \\gamma_0\\zeta^n.\n\\end{align*}\n","start":62566,"end":62992}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"线性乘子速率","source_units":[{"raw_tex":"We derive that, for all $n\\in \\mathbb{N}^*$, $\\|\\mathbf{x}_n-\\mathbf{x}^{*}\\|\\leq \\gamma_0\\zeta^{\\frac{n}{2}}$ and $\\|y_n-y^{*}\\|\\leq \\gamma_0\\zeta^{\\frac{n}{2}}$, which yields \n\\begin{align*}\n\\|y_n-y_{n-1}\\| \\leq \\|y_n-y^{*}\\|+\\|y_{n-1}-y^{*}\\| \\leq (1+\\zeta^{-\\frac{1}{2}})\\gamma_0\\zeta^{\\frac{n}{2}}.    \n\\end{align*}\nBy passing to the limit in \\eqref{eq:B'z}, $B^\\top z^{*}=-\\nabla H(y^{*})$. Following the same steps as in \\eqref{eq:BTZy} and \\eqref{eq:zy}, we obtain that, for all $n\\in \\mathbb{N}^*$,\n\\begin{align*}\n\\|z_n-z^{*}\\| &\\leq \\frac{\\ell_H}{\\sqrt{\\lambda}}\\|y_n-y^{*}\\| + \\frac{\\nu \\ell_{\\psi}}{\\sqrt{\\lambda}}\\|y_{n}-y_{n-1}\\|\\leq \\gamma_1\\zeta^{\\frac{n}{2}}, \\text{~~where~} \\gamma_1 :=\\frac{\\ell_H\\gamma_0+(1+\\zeta^{-\\frac{1}{2}})\\nu\\ell_{\\psi}\\gamma_0}{\\sqrt{\\lambda}}.\n\\end{align*}\n","start":62992,"end":63795}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"线性迭代与可行性速率","source_units":[{"raw_tex":"Consequently,\n\\begin{align*}\n&\\|(\\mathbf{x}_{n},y_{n},z_{n})-(\\mathbf{x}^{*},y^{*},z^{*})\\| \\leq \\|\\mathbf{x}_{n}-\\mathbf{x}^{*}\\| + \\|y_{n}-y^{*}\\| + \\|z_{n}-z^{*}\\|\\leq (2\\gamma_0 +\\gamma_1)\\zeta^{\\frac{n}{2}}, \\\\\n&\\|\\mathbf{Ax}_n+By_n-b\\| =\\frac{1}{\\rho}\\|z_n-z_{n-1}\\| \\leq \\frac{1}{\\rho}(\\|z_n-z^{*}\\| +\\|z_{n-1}-z^{*}\\|) \\leq \\frac{(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho} \\zeta^{\\frac{n}{2}}, \\\\\n\\text{and~} &\\|z_n\\| \\leq \\|z_n-z^{*}\\| + \\|z^{*}\\| \\leq \\gamma_1 \\zeta^{\\frac{n}{2}} + \\|z^{*}\\|.\n\\end{align*}\n","start":63795,"end":64309}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"线性拉格朗日值速率","source_units":[{"raw_tex":"We now deduce from the definition of $\\mathcal{F}$ that\n\\begin{align*}\n|\\mathcal{L}_{\\rho}(\\mathbf{x}_{n},y_{n},z_{n})- \\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})|&= \\left|\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*) - c\\|y_n-y_{n-1}\\|^2\\right|\\\\\n&\\leq |\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*)| + c\\|y_n-y_{n-1}\\|^2\\\\\n&\\leq \\gamma_2\\zeta^n, \\text{~~where~} \\gamma_2 :=\\gamma_0 + c(1+\\zeta^{-\\frac{1}{2}})^2\\gamma_0^2\n\\end{align*}\n","start":64310,"end":64843}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"线性目标值速率","source_units":[{"raw_tex":"and from the definition of $\\mathcal{L}_\\rho$ that \n\\begin{align*}\n|F(\\mathbf{x}_n,y_n)-F(\\mathbf{x}^{*},y^{*})| &= |\\mathcal{L}_{\\rho}(\\mathbf{x}_n,y_n,z_n)-\\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})-\\langle z_n, \\mathbf{Ax}_n + By_n-b\\rangle - \\frac{\\rho}{2}\\|\\mathbf{Ax}_n + By_n-b\\|^2| \\\\\n&\\leq |\\mathcal{L}_{\\rho}(\\mathbf{x}_n,y_n,z_n)-\\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})| + \\|z_n\\|\\|\\mathbf{Ax}_n + By_n-b\\| + \\frac{\\rho}{2}\\|\\mathbf{Ax}_n + By_n-b\\|^2  \\\\ \n&\\leq \\gamma_2\\zeta^n + \\left(\\gamma_1 \\zeta^{\\frac{n}{2}} + \\|z^{*}\\|\\right)\\frac{(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho} \\zeta^{\\frac{n}{2}}+ \\frac{(1+\\zeta^{-\\frac{1}{2}})^2\\gamma_1^2}{2\\rho}\\zeta^n \\\\\n&\\leq \\left(\\gamma_2 +\\frac{(3+\\zeta^{-\\frac{1}{2}})(1+\\zeta^{-\\frac{1}{2}})\\gamma_1^2}{2\\rho} +\\frac{\\|z^{*}\\|(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho}\\right)\\zeta^{\\frac{n}{2}},\n\\end{align*}\nwhere the last inequality follows from the fact that $\\zeta^n \\leq \\zeta^{\\frac{n}{2}}$ since $\\zeta \\in (0,1)$. By letting \n\\begin{align*}\n\\overline{\\gamma} =\\max \\left\\{ 2\\gamma_0+\\gamma_1, \\frac{(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho}, \\gamma_2 +\\frac{(3+\\zeta^{-\\frac{1}{2}})(1+\\zeta^{-\\frac{1}{2}})\\gamma_1^2}{2\\rho} +\\frac{\\|z^{*}\\|(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho} \\right\\} \n\\end{align*}\nand increasing it if necessary, we arrive at the conclusion.\n","start":64843,"end":66180}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"次线性基础估计","source_units":[{"raw_tex":"\\ref{t:full_rate_sublinear}: Following the arguments in \\cite[Theorem~5.1(iv)]{BDL22} and \\cite[Theorem~2(iii)]{Attouch2007}, we find $\\gamma_0 \\in \\mathbb{R}_{++}$ such that, for all $n\\in \\mathbb{N}^*$, \n\\begin{align*}\n\\|(\\mathbf{x}_{n},y_{n})-(\\mathbf{x}^{*},y^{*})\\|\\leq \\gamma_0 n^{-\\frac{1-\\kappa}{2\\kappa-1}} \\text{~~and~~}\n|\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*)| \\leq \\gamma_0 n^{-\\frac{2-2\\kappa}{2\\kappa-1}}.\n\\end{align*}\n","start":66181,"end":66667}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"次线性速率传递","source_units":[{"raw_tex":"Similar to \\ref{t:full_rate_linear}, for all $n\\geq 2$, since $n-1\\geq \\frac{1}{2}n$ and $\\frac{1-\\kappa}{2\\kappa-1} >0$, we derive that \n\\begin{align*}\n&\\|(\\mathbf{x}_{n},y_{n},z_{n})-(\\mathbf{x}^{*},y^{*},z^{*})\\| \\leq (2\\gamma_0+\\gamma_1) n^{-\\frac{1-\\kappa}{2\\kappa-1}},\\\\\n&\\|\\mathbf{Ax}_n+By_n-b\\|\\leq \\frac{(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1}{\\rho} n^{-\\frac{1-\\kappa}{2\\kappa-1}},\\\\\n&|\\mathcal{L}_{\\rho}(\\mathbf{x}_{n},y_{n},z_{n})- \\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})|\\leq \\gamma_2 n^{-\\frac{2-2\\kappa}{2\\kappa-1}},\n\\end{align*}\n","start":66667,"end":67224}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"次线性目标值速率","source_units":[{"raw_tex":"and that\n\\begin{align*}\n|F(\\mathbf{x}_n,y_n)-F(\\mathbf{x}^{*},y^{*})| &\\leq \\gamma_2 n^{-\\frac{2-2\\kappa}{2\\kappa-1}}+ \\left(\\gamma_1 n^{-\\frac{1-\\kappa}{2\\kappa-1}} + \\|z^{*}\\|\\right)\\frac{(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1}{\\rho} n^{-\\frac{1-\\kappa}{2\\kappa-1}} + \\frac{(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})^2\\gamma_1^2}{2\\rho} n^{-\\frac{2-2\\kappa}{2\\kappa-1}}\\\\\n&\\leq \\left(\\gamma_2 +\\frac{(3 +2^{\\frac{1-\\kappa}{2\\kappa-1}})(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1^2}{2\\rho} +\\frac{\\|z^{*}\\|(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1}{\\rho} \\right)n^{-\\frac{1-\\kappa}{2\\kappa-1}},\n\\end{align*}\nwhere $\\gamma_1 :=\\frac{\\ell_H\\gamma_0 +(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\nu\\ell_{\\psi}\\gamma_0}{\\sqrt{\\lambda}}$, $\\gamma_2 :=\\gamma_0 +c(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})^2\\gamma_0^2$, and the last inequality follows from the fact that $n^{-\\frac{2-2\\kappa}{2\\kappa-1}}<n^{-\\frac{1-\\kappa}{2\\kappa-1}}$ for $\\kappa\\in(\\frac{1}{2},1)$. By setting $\\overline{\\gamma}$ in the same way as in \\ref{t:full_rate_linear}, we obtain the conclusion.\n","start":67224,"end":68278}]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决列表：Mordukhovich (2006), Proposition 1.107(ii)、Theorem 3.36；Rockafellar–Wets (1998), Proposition 10.5；原文第248行明确引用。"],"source_only":false,"title":"外部次微分演算","source_units":[]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决列表：Pham et al. (2023), Tan2023, Lemma 2.2；原文第258行引用。"],"source_only":false,"title":"外部次梯度聚点引理","source_units":[]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决列表：Bregman (1967), Section 1；原文第286行引用。"],"source_only":false,"title":"Bregman原文性质","source_units":[]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决列表：Nesterov (2018), Lemma 1.2.3、Equation (1.2.19)；原文第306、308行引用。"],"source_only":false,"title":"Nesterov光滑估计","source_units":[]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决列表：Boţ–Dao–Li (2022), BDL22, Lemma 4.1；原文第535行引用。"],"source_only":false,"title":"弱凸次梯度下界","source_units":[]},{"id":"S081","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决列表：Rockafellar–Wets (1998), Example 9.14；原文第626行引用。"],"source_only":false,"title":"有限凸函数局部Lipschitz","source_units":[]},{"id":"S082","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决列表：Boţ–Dao–Li (2023), BDL23, Lemma 4.5及其证明；原文第439行引用。"],"source_only":false,"title":"外部对称矩阵谱估计","source_units":[]},{"id":"S083","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决列表：Boţ–Dao–Li (2022), BDL22, Theorem 5.1及其证明；原文第791、801、803、842行引用。"],"source_only":false,"title":"抽象KL收敛与速率","source_units":[]},{"id":"S084","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决列表：Attouch–Bolte (2007), Theorem 2(i),(iii)及相关证明；原文第801、842行引用。"],"source_only":false,"title":"有限终止与幂次论证","source_units":[]},{"id":"S085","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决列表已声明：Standard finite-dimensional compactness, spectral theory, Fermat's rule, convex-gradient monotonicity, and Cauchy–Schwarz inequality；完备性为implicit standard fact made explicit by annotator。"],"source_only":false,"title":"标准有限维分析工具","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S018","edges":[{"source_id":"S001","target_id":"S013","kind":"supports"},{"source_id":"S007","target_id":"S013","kind":"supports"},{"source_id":"S006","target_id":"S013","kind":"supports"},{"source_id":"S002","target_id":"S011","kind":"supports"},{"source_id":"S005","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S016","kind":"supports"},{"source_id":"S039","target_id":"S016","kind":"supports"},{"source_id":"S076","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S077","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S078","target_id":"S028","kind":"supports"},{"source_id":"S004","target_id":"S028","kind":"supports"},{"source_id":"S006","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S024","kind":"supports"},{"source_id":"S028","target_id":"S025","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S079","target_id":"S032","kind":"supports"},{"source_id":"S029","target_id":"S033","kind":"supports"},{"source_id":"S079","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S032","target_id":"S029","kind":"supports"},{"source_id":"S033","target_id":"S030","kind":"supports"},{"source_id":"S082","target_id":"S037","kind":"supports"},{"source_id":"S085","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S034","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S013","target_id":"S043","kind":"supports"},{"source_id":"S006","target_id":"S043","kind":"supports"},{"source_id":"S085","target_id":"S043","kind":"supports"},{"source_id":"S011","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S011","target_id":"S045","kind":"supports"},{"source_id":"S043","target_id":"S045","kind":"supports"},{"source_id":"S029","target_id":"S045","kind":"supports"},{"source_id":"S008","target_id":"S045","kind":"supports"},{"source_id":"S013","target_id":"S045","kind":"supports"},{"source_id":"S085","target_id":"S045","kind":"supports"},{"source_id":"S013","target_id":"S046","kind":"supports"},{"source_id":"S024","target_id":"S046","kind":"supports"},{"source_id":"S026","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S029","target_id":"S047","kind":"supports"},{"source_id":"S080","target_id":"S047","kind":"supports"},{"source_id":"S008","target_id":"S047","kind":"supports"},{"source_id":"S011","target_id":"S047","kind":"supports"},{"source_id":"S044","target_id":"S048","kind":"supports"},{"source_id":"S045","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S009","target_id":"S049","kind":"supports"},{"source_id":"S010","target_id":"S049","kind":"supports"},{"source_id":"S043","target_id":"S049","kind":"supports"},{"source_id":"S036","target_id":"S049","kind":"supports"},{"source_id":"S008","target_id":"S049","kind":"supports"},{"source_id":"S013","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S034","target_id":"S050","kind":"supports"},{"source_id":"S010","target_id":"S050","kind":"supports"},{"source_id":"S013","target_id":"S050","kind":"supports"},{"source_id":"S039","target_id":"S050","kind":"supports"},{"source_id":"S085","target_id":"S050","kind":"supports"},{"source_id":"S040","target_id":"S051","kind":"supports"},{"source_id":"S049","target_id":"S051","kind":"supports"},{"source_id":"S043","target_id":"S051","kind":"supports"},{"source_id":"S036","target_id":"S051","kind":"supports"},{"source_id":"S010","target_id":"S051","kind":"supports"},{"source_id":"S013","target_id":"S051","kind":"supports"},{"source_id":"S085","target_id":"S051","kind":"supports"},{"source_id":"S011","target_id":"S052","kind":"supports"},{"source_id":"S039","target_id":"S052","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S013","target_id":"S052","kind":"supports"},{"source_id":"S050","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S051","target_id":"S053","kind":"supports"},{"source_id":"S040","target_id":"S053","kind":"supports"},{"source_id":"S004","target_id":"S053","kind":"supports"},{"source_id":"S030","target_id":"S053","kind":"supports"},{"source_id":"S010","target_id":"S053","kind":"supports"},{"source_id":"S043","target_id":"S053","kind":"supports"},{"source_id":"S050","target_id":"S054","kind":"supports"},{"source_id":"S052","target_id":"S054","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S049","target_id":"S054","kind":"supports"},{"source_id":"S043","target_id":"S054","kind":"supports"},{"source_id":"S039","target_id":"S054","kind":"supports"},{"source_id":"S046","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S026","target_id":"S055","kind":"supports"},{"source_id":"S027","target_id":"S055","kind":"supports"},{"source_id":"S022","target_id":"S055","kind":"supports"},{"source_id":"S081","target_id":"S055","kind":"supports"},{"source_id":"S008","target_id":"S055","kind":"supports"},{"source_id":"S002","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S043","target_id":"S056","kind":"supports"},{"source_id":"S054","target_id":"S056","kind":"supports"},{"source_id":"S041","target_id":"S056","kind":"supports"},{"source_id":"S019","target_id":"S056","kind":"supports"},{"source_id":"S020","target_id":"S056","kind":"supports"},{"source_id":"S012","target_id":"S056","kind":"supports"},{"source_id":"S085","target_id":"S056","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S044","target_id":"S042","kind":"supports"},{"source_id":"S045","target_id":"S042","kind":"supports"},{"source_id":"S046","target_id":"S042","kind":"supports"},{"source_id":"S047","target_id":"S042","kind":"supports"},{"source_id":"S048","target_id":"S042","kind":"supports"},{"source_id":"S049","target_id":"S042","kind":"supports"},{"source_id":"S050","target_id":"S042","kind":"supports"},{"source_id":"S051","target_id":"S042","kind":"supports"},{"source_id":"S052","target_id":"S042","kind":"supports"},{"source_id":"S053","target_id":"S042","kind":"supports"},{"source_id":"S054","target_id":"S042","kind":"supports"},{"source_id":"S055","target_id":"S042","kind":"supports"},{"source_id":"S056","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S058","kind":"supports"},{"source_id":"S015","target_id":"S058","kind":"supports"},{"source_id":"S016","target_id":"S058","kind":"supports"},{"source_id":"S039","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S085","target_id":"S059","kind":"supports"},{"source_id":"S005","target_id":"S060","kind":"supports"},{"source_id":"S016","target_id":"S060","kind":"supports"},{"source_id":"S013","target_id":"S060","kind":"supports"},{"source_id":"S015","target_id":"S060","kind":"supports"},{"source_id":"S019","target_id":"S060","kind":"supports"},{"source_id":"S020","target_id":"S060","kind":"supports"},{"source_id":"S085","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S008","target_id":"S061","kind":"supports"},{"source_id":"S015","target_id":"S061","kind":"supports"},{"source_id":"S016","target_id":"S062","kind":"supports"},{"source_id":"S013","target_id":"S062","kind":"supports"},{"source_id":"S016","target_id":"S063","kind":"supports"},{"source_id":"S013","target_id":"S063","kind":"supports"},{"source_id":"S061","target_id":"S064","kind":"supports"},{"source_id":"S062","target_id":"S064","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S049","target_id":"S064","kind":"supports"},{"source_id":"S015","target_id":"S064","kind":"supports"},{"source_id":"S019","target_id":"S064","kind":"supports"},{"source_id":"S020","target_id":"S064","kind":"supports"},{"source_id":"S058","target_id":"S065","kind":"supports"},{"source_id":"S059","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S014","target_id":"S065","kind":"supports"},{"source_id":"S017","target_id":"S065","kind":"supports"},{"source_id":"S083","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S049","target_id":"S066","kind":"supports"},{"source_id":"S038","target_id":"S066","kind":"supports"},{"source_id":"S015","target_id":"S066","kind":"supports"},{"source_id":"S085","target_id":"S066","kind":"supports"},{"source_id":"S058","target_id":"S067","kind":"supports"},{"source_id":"S059","target_id":"S067","kind":"supports"},{"source_id":"S064","target_id":"S067","kind":"supports"},{"source_id":"S017","target_id":"S067","kind":"supports"},{"source_id":"S083","target_id":"S067","kind":"supports"},{"source_id":"S084","target_id":"S067","kind":"supports"},{"source_id":"S058","target_id":"S068","kind":"supports"},{"source_id":"S059","target_id":"S068","kind":"supports"},{"source_id":"S064","target_id":"S068","kind":"supports"},{"source_id":"S017","target_id":"S068","kind":"supports"},{"source_id":"S083","target_id":"S068","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S066","target_id":"S069","kind":"supports"},{"source_id":"S043","target_id":"S069","kind":"supports"},{"source_id":"S049","target_id":"S069","kind":"supports"},{"source_id":"S051","target_id":"S069","kind":"supports"},{"source_id":"S008","target_id":"S069","kind":"supports"},{"source_id":"S013","target_id":"S069","kind":"supports"},{"source_id":"S036","target_id":"S069","kind":"supports"},{"source_id":"S010","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S070","kind":"supports"},{"source_id":"S069","target_id":"S070","kind":"supports"},{"source_id":"S043","target_id":"S070","kind":"supports"},{"source_id":"S068","target_id":"S071","kind":"supports"},{"source_id":"S069","target_id":"S071","kind":"supports"},{"source_id":"S016","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S066","target_id":"S072","kind":"supports"},{"source_id":"S011","target_id":"S072","kind":"supports"},{"source_id":"S058","target_id":"S073","kind":"supports"},{"source_id":"S059","target_id":"S073","kind":"supports"},{"source_id":"S064","target_id":"S073","kind":"supports"},{"source_id":"S017","target_id":"S073","kind":"supports"},{"source_id":"S083","target_id":"S073","kind":"supports"},{"source_id":"S084","target_id":"S073","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S069","target_id":"S074","kind":"supports"},{"source_id":"S070","target_id":"S074","kind":"supports"},{"source_id":"S071","target_id":"S074","kind":"supports"},{"source_id":"S043","target_id":"S074","kind":"supports"},{"source_id":"S049","target_id":"S074","kind":"supports"},{"source_id":"S016","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S011","target_id":"S075","kind":"supports"},{"source_id":"S066","target_id":"S075","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S059","target_id":"S057","kind":"supports"},{"source_id":"S060","target_id":"S057","kind":"supports"},{"source_id":"S061","target_id":"S057","kind":"supports"},{"source_id":"S062","target_id":"S057","kind":"supports"},{"source_id":"S063","target_id":"S057","kind":"supports"},{"source_id":"S064","target_id":"S057","kind":"supports"},{"source_id":"S065","target_id":"S057","kind":"supports"},{"source_id":"S066","target_id":"S057","kind":"supports"},{"source_id":"S067","target_id":"S057","kind":"supports"},{"source_id":"S068","target_id":"S057","kind":"supports"},{"source_id":"S069","target_id":"S057","kind":"supports"},{"source_id":"S070","target_id":"S057","kind":"supports"},{"source_id":"S071","target_id":"S057","kind":"supports"},{"source_id":"S072","target_id":"S057","kind":"supports"},{"source_id":"S073","target_id":"S057","kind":"supports"},{"source_id":"S074","target_id":"S057","kind":"supports"},{"source_id":"S075","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S018","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S018","edges":[{"source_id":"S001","target_id":"S013","kind":"supports"},{"source_id":"S007","target_id":"S013","kind":"supports"},{"source_id":"S006","target_id":"S013","kind":"supports"},{"source_id":"S002","target_id":"S011","kind":"supports"},{"source_id":"S005","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S016","kind":"supports"},{"source_id":"S039","target_id":"S016","kind":"supports"},{"source_id":"S076","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S077","target_id":"S022","kind":"supports"},{"source_id":"S078","target_id":"S028","kind":"supports"},{"source_id":"S004","target_id":"S028","kind":"supports"},{"source_id":"S006","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S024","kind":"supports"},{"source_id":"S028","target_id":"S025","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S079","target_id":"S032","kind":"supports"},{"source_id":"S029","target_id":"S033","kind":"supports"},{"source_id":"S079","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S032","target_id":"S029","kind":"supports"},{"source_id":"S033","target_id":"S030","kind":"supports"},{"source_id":"S082","target_id":"S037","kind":"supports"},{"source_id":"S085","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S034","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S013","target_id":"S043","kind":"supports"},{"source_id":"S006","target_id":"S043","kind":"supports"},{"source_id":"S085","target_id":"S043","kind":"supports"},{"source_id":"S011","target_id":"S038","kind":"supports"},{"source_id":"S043","target_id":"S038","kind":"supports"},{"source_id":"S029","target_id":"S038","kind":"supports"},{"source_id":"S008","target_id":"S038","kind":"supports"},{"source_id":"S013","target_id":"S038","kind":"supports"},{"source_id":"S085","target_id":"S038","kind":"supports"},{"source_id":"S024","target_id":"S038","kind":"supports"},{"source_id":"S026","target_id":"S038","kind":"supports"},{"source_id":"S080","target_id":"S038","kind":"supports"},{"source_id":"S009","target_id":"S049","kind":"supports"},{"source_id":"S010","target_id":"S049","kind":"supports"},{"source_id":"S043","target_id":"S049","kind":"supports"},{"source_id":"S036","target_id":"S049","kind":"supports"},{"source_id":"S008","target_id":"S049","kind":"supports"},{"source_id":"S013","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S038","kind":"supports"},{"source_id":"S034","target_id":"S038","kind":"supports"},{"source_id":"S010","target_id":"S038","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S051","kind":"supports"},{"source_id":"S049","target_id":"S051","kind":"supports"},{"source_id":"S043","target_id":"S051","kind":"supports"},{"source_id":"S036","target_id":"S051","kind":"supports"},{"source_id":"S010","target_id":"S051","kind":"supports"},{"source_id":"S013","target_id":"S051","kind":"supports"},{"source_id":"S085","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S004","target_id":"S038","kind":"supports"},{"source_id":"S030","target_id":"S038","kind":"supports"},{"source_id":"S027","target_id":"S038","kind":"supports"},{"source_id":"S022","target_id":"S038","kind":"supports"},{"source_id":"S081","target_id":"S038","kind":"supports"},{"source_id":"S002","target_id":"S038","kind":"supports"},{"source_id":"S041","target_id":"S038","kind":"supports"},{"source_id":"S019","target_id":"S038","kind":"supports"},{"source_id":"S020","target_id":"S038","kind":"supports"},{"source_id":"S012","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S058","kind":"supports"},{"source_id":"S015","target_id":"S058","kind":"supports"},{"source_id":"S016","target_id":"S058","kind":"supports"},{"source_id":"S039","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S085","target_id":"S059","kind":"supports"},{"source_id":"S005","target_id":"S060","kind":"supports"},{"source_id":"S016","target_id":"S060","kind":"supports"},{"source_id":"S013","target_id":"S060","kind":"supports"},{"source_id":"S015","target_id":"S060","kind":"supports"},{"source_id":"S019","target_id":"S060","kind":"supports"},{"source_id":"S020","target_id":"S060","kind":"supports"},{"source_id":"S085","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S008","target_id":"S061","kind":"supports"},{"source_id":"S015","target_id":"S061","kind":"supports"},{"source_id":"S016","target_id":"S062","kind":"supports"},{"source_id":"S013","target_id":"S062","kind":"supports"},{"source_id":"S016","target_id":"S063","kind":"supports"},{"source_id":"S013","target_id":"S063","kind":"supports"},{"source_id":"S061","target_id":"S064","kind":"supports"},{"source_id":"S062","target_id":"S064","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S049","target_id":"S064","kind":"supports"},{"source_id":"S015","target_id":"S064","kind":"supports"},{"source_id":"S019","target_id":"S064","kind":"supports"},{"source_id":"S020","target_id":"S064","kind":"supports"},{"source_id":"S058","target_id":"S065","kind":"supports"},{"source_id":"S059","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S014","target_id":"S065","kind":"supports"},{"source_id":"S017","target_id":"S065","kind":"supports"},{"source_id":"S083","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S049","target_id":"S066","kind":"supports"},{"source_id":"S038","target_id":"S066","kind":"supports"},{"source_id":"S015","target_id":"S066","kind":"supports"},{"source_id":"S085","target_id":"S066","kind":"supports"},{"source_id":"S058","target_id":"S067","kind":"supports"},{"source_id":"S059","target_id":"S067","kind":"supports"},{"source_id":"S064","target_id":"S067","kind":"supports"},{"source_id":"S017","target_id":"S067","kind":"supports"},{"source_id":"S083","target_id":"S067","kind":"supports"},{"source_id":"S084","target_id":"S067","kind":"supports"},{"source_id":"S058","target_id":"S068","kind":"supports"},{"source_id":"S059","target_id":"S068","kind":"supports"},{"source_id":"S064","target_id":"S068","kind":"supports"},{"source_id":"S017","target_id":"S068","kind":"supports"},{"source_id":"S083","target_id":"S068","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S066","target_id":"S069","kind":"supports"},{"source_id":"S043","target_id":"S069","kind":"supports"},{"source_id":"S049","target_id":"S069","kind":"supports"},{"source_id":"S051","target_id":"S069","kind":"supports"},{"source_id":"S008","target_id":"S069","kind":"supports"},{"source_id":"S013","target_id":"S069","kind":"supports"},{"source_id":"S036","target_id":"S069","kind":"supports"},{"source_id":"S010","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S070","kind":"supports"},{"source_id":"S069","target_id":"S070","kind":"supports"},{"source_id":"S043","target_id":"S070","kind":"supports"},{"source_id":"S068","target_id":"S071","kind":"supports"},{"source_id":"S069","target_id":"S071","kind":"supports"},{"source_id":"S016","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S066","target_id":"S072","kind":"supports"},{"source_id":"S011","target_id":"S072","kind":"supports"},{"source_id":"S058","target_id":"S073","kind":"supports"},{"source_id":"S059","target_id":"S073","kind":"supports"},{"source_id":"S064","target_id":"S073","kind":"supports"},{"source_id":"S017","target_id":"S073","kind":"supports"},{"source_id":"S083","target_id":"S073","kind":"supports"},{"source_id":"S084","target_id":"S073","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S069","target_id":"S074","kind":"supports"},{"source_id":"S070","target_id":"S074","kind":"supports"},{"source_id":"S071","target_id":"S074","kind":"supports"},{"source_id":"S043","target_id":"S074","kind":"supports"},{"source_id":"S049","target_id":"S074","kind":"supports"},{"source_id":"S016","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S011","target_id":"S075","kind":"supports"},{"source_id":"S066","target_id":"S075","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S059","target_id":"S057","kind":"supports"},{"source_id":"S060","target_id":"S057","kind":"supports"},{"source_id":"S061","target_id":"S057","kind":"supports"},{"source_id":"S062","target_id":"S057","kind":"supports"},{"source_id":"S063","target_id":"S057","kind":"supports"},{"source_id":"S064","target_id":"S057","kind":"supports"},{"source_id":"S065","target_id":"S057","kind":"supports"},{"source_id":"S066","target_id":"S057","kind":"supports"},{"source_id":"S067","target_id":"S057","kind":"supports"},{"source_id":"S068","target_id":"S057","kind":"supports"},{"source_id":"S069","target_id":"S057","kind":"supports"},{"source_id":"S070","target_id":"S057","kind":"supports"},{"source_id":"S071","target_id":"S057","kind":"supports"},{"source_id":"S072","target_id":"S057","kind":"supports"},{"source_id":"S073","target_id":"S057","kind":"supports"},{"source_id":"S074","target_id":"S057","kind":"supports"},{"source_id":"S075","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S018","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"基本空间与指标","source_units":[{"raw_tex":"Throughout this paper, the set of nonnegative integers is denoted by $\\mathbb{N}$, the set of real numbers by $\\mathbb{R}$, the set of nonnegative real numbers by $\\mathbb{R}_+$, and the set of the positive real numbers by $\\mathbb{R}_{++}$. We use $\\mathbb{R}^d$ to denote the $d$-dimensional Euclidean space with inner product $\\langle \\cdot, \\cdot \\rangle$ and Euclidean norm $\\|\\cdot\\|$. For a given matrix $M$, $\\Ima(M)$ denotes\nits image.\n\n","start":5195,"end":5641}]},{"id":"S002","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:prob"],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"线性约束复合问题","source_units":[{"raw_tex":"We consider the linearly constrained composite optimization problem\n\\begin{align}\\label{eq:prob}\n\\min_{x_i \\in \\mathbb{R}^{d_i}, y\\in \\mathbb{R}^q} ~F(\\mathbf{x},y) :=\\sum_{i=1}^m f_i(x_i)+H(y)+P(\\mathbf{x})-G(\\mathbf{x}) \\quad\\text{s.t.~~}  \\sum_{i=1}^{m}A_ix_i + By=b,\n\\end{align}\nwhere $\\mathbf{x}=(x_1,\\dots,x_m)$; for each $i\\in \\{1,\\dots,m\\}$, $f_i: \\mathbb{R}^{d_i} \\to (-\\infty,+\\infty]$ is a proper lower semicontinuous function; $H:\\mathbb{R}^q \\to \\mathbb{R}$ is a differentiable (possibly\nnonconvex) function whose gradient is Lipschitz continuous; $P:\\mathbb{R}^{d_1}\\times \\dots \\times \\mathbb{R}^{d_m} \\to \\mathbb{R}$ is another differentiable (possibly nonconvex) function whose gradient is Lipschitz continuous; $G:\\mathbb{R}^{d_1}\\times \\dots \\times \\mathbb{R}^{d_m} \\to \\mathbb{R}$ is a continuous (possibly nonsmooth) weakly convex function, $A_i \\in \\mathbb{R}^{p\\times d_i}$, $B\\in \\mathbb{R}^{p\\times q}$, and $b \\in \\mathbb{R}^p$. Problem \\eqref{eq:prob} covers two important models in the literature. Firstly, when $G \\equiv 0$, problem \\eqref{eq:prob} reduces to the form\n","start":5641,"end":6739}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"凸性与强制性","source_units":[{"raw_tex":"Let $f\\colon \\mathbb{R}^d\\to \\left(-\\infty,+\\infty\\right]$. The \\emph{domain} of $f$ is $\\dom f :=\\{x\\in \\mathbb{R}^d: f(x) <+\\infty\\}$ and the \\emph{epigraph} of $f$ is $\\epi f := \\{(x,\\xi)\\in \\mathbb{R}^d\\times \\mathbb{R}: f(x)\\leq \\xi\\}$. The function $f$ is said to be \\emph{proper} if $\\dom f \\neq \\varnothing$ and it never takes the value $-\\infty$, \\emph{lower semicontinuous} if its epigraph is closed, and \\emph{convex} if its epigraph is convex. The function $f$ is said to be $\\alpha$-\\emph{strongly convex} ($\\alpha\\in \\mathbb{R}_+$) if $f-\\frac{\\alpha}{2}\\|\\cdot\\|^2$ is convex, and $\\beta$-\\emph{weakly convex} ($\\beta\\in \\mathbb{R}_+$) if $f+\\frac{\\beta}{2}\\|\\cdot\\|^2$ is convex. We say that $f$ is \\emph{coercive} if $f(x)\\to +\\infty$ as $\\|x\\|\\to +\\infty$.\n\n","start":14840,"end":15616}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"极限次微分","source_units":[{"raw_tex":"Let $x\\in \\mathbb{R}^d$ with $|f(x)| <+\\infty$. The \\emph{regular} (or \\emph{Fr\\'echet}) \\emph{subdifferential} of $f$ at $x$ is defined by\n\\begin{equation*}\n\\widehat{\\partial} f(x) :=\\left\\{x^*\\in \\mathbb{R}^d:\\; \\liminf_{y\\to x}\\frac{f(y)-f(x)-\\langle x^*,y-x\\rangle}{\\|y-x\\|}\\geq 0\\right\\}\n\\end{equation*}\nand the \\emph{limiting} (or \\emph{Mordukhovich}) \\emph{subdifferential} of $f$ at $x$ is defined by\n\\begin{equation*}\n\\partial_L f(x) :=\\left\\{x^*\\in \\mathbb{R}^d:\\; \\exists x_n\\stackrel{f}{\\to}x,\\; x_n^*\\to x^* \\text{~~with~~} x_n^*\\in \\widehat{\\partial} f(x_n)\\right\\},\n\\end{equation*}\nwhere the notation $y\\stackrel{f}{\\to}x$ means $y\\to x$ with $f(y)\\to f(x)$. In the case where $\\lvert f(x) \\rvert =+\\infty$, both regular subdifferential and limiting subdifferential of $f$ at $x$ are defined to be the empty set. The \\emph{domain} of $\\partial_L f$ is given by $\\dom \\partial_L f :=\\{x\\in \\mathbb{R}^d: \\partial_L f(x) \\neq \\varnothing\\}$. \nWe now collect some important properties of the limiting subdifferential.\n","start":15616,"end":16646}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"Bregman距离","source_units":[{"raw_tex":"We recall the concept of the well-known Bregman distance in a real Hilbert space $\\mathcal{H}$, which was introduced in \\cite{Bregman}. Given a differentiable convex function $\\phi: \\mathcal{H} \\to \\mathbb{R}$, the \\emph{Bregman distance} associated with $\\phi$ is defined as \n\\begin{align*}\n\\forall u, v\\in \\mathcal{H},\\quad D_{\\phi}(u,v)=\\phi(u)-\\phi(v)-\\langle \\nabla\\phi (v),u-v\\rangle,\n\\end{align*}\nWhen $\\phi(x)=\\|x\\|^2$, then $D_{\\phi}(u,v)=\\|u-v\\|^2$. When $\\phi(x)=x^\\top M x$, where $M$ is a positive semidefinite matrix, then $D_{\\phi}(u,v) = \\|u-v\\|^2_M :=(u-v)^\\top M (u-v)$. When $\\mathcal{H}=\\mathbb{R}^{p\\times d}$ and $\\phi(X)=\\|X\\|_F^2$, then $D_{\\phi}(U,V) =\\|U-V\\|_F^2$ \\cite{Dhillon2008}. Some useful properties of the Bregman distance are listed in the following proposition.\n","start":18239,"end":19037}]},{"id":"S007","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"分块与谱记号","source_units":[{"raw_tex":"We first fix some notations which are used throughout the paper from now on. Let $\\mathbf{A}=[A_1|A_2|\\dots|A_m]$ and let $\\mathbf{I}_{q\\times q}$ be the $q\\times q$ identity matrix. Then $\\mathbf{Ax}=\\sum_{i=1}^{m}A_ix_i$. We also denote by $\\partial_L^{x}$ the limiting subdifferential with respect to the $x$-variable, and by $\\nabla_i f$ the $i^{th}$ block of the gradient vector $\\nabla f$. Given a matrix $M$, $\\lambda_{\\min}(M)$ denotes its smallest eigenvalue while $\\lambda_{\\max}(M)$ denotes its largest eigenvalue. The following assumptions are used in our convergence analysis\n","start":21422,"end":22011}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","labels":["a:standing","a:standing_image","a:standing_rank"],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"光滑性与弱凸性","source_units":[{"raw_tex":"\\begin{assumption}[Standing assumptions]\n\\label{a:standing}\n~\n\\begin{enumerate}\n\\item \n$\\nabla H$ is $\\ell_H$-Lipschitz continuous, $\\nabla P$ is $\\ell_P$-Lipschitz continuous, and $G$ is $\\beta$-weakly convex.\n","start":22011,"end":22222}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","labels":["a:standing","a:standing_image","a:standing_rank"],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"像空间包含条件","source_units":[{"raw_tex":"\\item\\label{a:standing_image} \n$\\Ima(\\mathbf{A}) \\bigcup \\{b \\}\\subseteq \\Ima(B)$.\n","start":22222,"end":22305}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"矩阵满列秩","source_units":[{"raw_tex":"\\item\\label{a:standing_rank} \n $\\lambda :=\\lambda_{\\min}(B^\\top B) >0$ (equivalently, $B$ has full column rank).\n\\end{enumerate}\n\\end{assumption}\n","start":22305,"end":22451}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"增广拉格朗日函数","source_units":[{"raw_tex":"Recall that the augmented Lagrangian of problem \\eqref{eq:prob} is given by\n\\begin{align*}\n\\mathcal{L}_\\rho(\\mathbf{x},y,z)=\\sum_{i=1}^m f_i(x_i)+H(y)+P(\\mathbf{x})-G(\\mathbf{x}) +  \\langle z, \\mathbf{Ax}+By-b \\rangle +\\frac{\\rho}{2}\\|\\mathbf{Ax}+By-b\\|^2,\n\\end{align*}\n","start":22747,"end":23017}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["eq:optimality_conditions"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"驻点条件","source_units":[{"raw_tex":"where $z\\in \\mathbb{R}^p$ and $\\rho\\in \\mathbb{R}_{++}$. Now, we present our splitting algorithm with guaranteed global convergence to a stationary point $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ of $\\mathcal{L}_\\rho$, i.e., $0 \\in \\partial_L \\mathcal{L}_\\rho(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$, or equivalently,\n\\begin{equation}\\label{eq:optimality_conditions}\n\\begin{aligned}\n&0 \\in \\partial_L\\left(\\sum_{i=1}^{m}f_i -G\\right)(\\overline{\\mathbf{x}}) +\\nabla P(\\overline{\\mathbf{x}}) +\\mathbf{A}^\\top \\overline{z},\\\\\n&0 =\\nabla H(\\overline{y}) +B^\\top \\overline{z}, \\\\\n&0 =\\mathbf{A\\overline{x}} +B\\overline{y} -b.\n\\end{aligned}\n\\end{equation}\n","start":23017,"end":23687}]},{"id":"S013","semantic_type":"algorithm","kind":"algorithm","labels":["algo:BPLA","step:A2_1","step:A2_2"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"BPL-ADMM迭代","source_units":[{"raw_tex":"\\begin{tcolorbox}[\n\tleft=0pt,right=0pt,top=0pt,bottom=0pt,\n\tcolback=blue!10!white, colframe=blue!50!white,\n  \tboxrule=0.2pt,\n  \tbreakable]\n\\begin{algorithm}[Bregman proximal linearized ADMM (BPL-ADMM)]\n\\label{algo:BPLA}\n\\step{}\\label{step:A2_1}\nSet $n=0$. Choose $\\mathbf{x}_0 =(x_{1,0}, \\dots, x_{m,0}) \\in \\mathbb{R}^{d_1}\\times \\dots \\times \\mathbb{R}^{d_m}$, $y_0 \\in \\mathbb{R}^q$, and $z_0 \\in \\mathbb{R}^p$. Let $\\phi_i: \\mathbb{R}^{d_i} \\to \\mathbb{R}$, $i \\in \\{1,\\dots,m\\}$, be differentiable $\\alpha$-strongly convex functions with $\\alpha \\in (0, +\\infty)$, and let $\\psi: \\mathbb{R}^q \\to \\mathbb{R}$ be a differentiable convex function with $\\ell_{\\psi}$-Lipschitz continuous gradient. Let \n\\begin{align*}\n\\mu \\in \\left(\\frac{\\ell_P+\\beta}{\\alpha}, +\\infty\\right),\\;\\; \\nu \\in \\left[0, +\\infty\\right), \\text{~~and~~} \\rho \\in \\left(\\frac{\\ell_H +\\sqrt{\\ell_H^2 +8(\\ell_H +2\\nu\\ell_{\\psi})^2}}{2\\lambda}, +\\infty\\right).    \n\\end{align*}\n\n\\step{}\\label{step:A2_2}\nCalculate $\\nabla P(\\mathbf{x}_n)=(\\nabla_1 P(\\mathbf{x}_n),\\dots,\\nabla_m P(\\mathbf{x}_n))$ and $g_{n}=(g_{1,n},\\dots,g_{m,n}) \\in \\partial_L G(\\mathbf{x}_{n})$. Let $\\mu_n \\in [\\mu, +\\infty)$ and $\\nu_n \\in [0, \\nu]$. For each $i \\in \\{1,\\dots,m\\}$, define $\\mathbf{u}_{i,n+1}: \\mathbb{R}^{d_i}\\to \\mathbb{R}^{d_1}\\times \\dots \\times \\mathbb{R}^{d_m}$ by $\\mathbf{u}_{i,n+1}(x_i) =(x_{1,n+1}, \\dots, x_{i-1,n+1}, x_i, x_{i+1,n}, \\dots, x_{m,n})$, and find\n\\begin{align*}\nx_{i,n+1} &\\in  \\argmin_{x_i \\in \\mathbb{R}^{d_i}} \\left(f_i(x_i) +\\langle \\nabla_i P(\\mathbf{x}_n) -g_{i,n}, x_i \\rangle + \\langle z_n, A_ix_i\\rangle+\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,n+1}(x_i)+By_n-b\\|^2 +\\mu_n D_{\\phi_i}(x_i,x_{i,n})\\right),\\\\\ny_{n+1} &\\in  \\argmin_{y \\in \\mathbb{R}^q} \\left(H(y) + \\langle z_n, By\\rangle+\\frac{\\rho}{2}\\|\\mathbf{Ax}_{n+1} + By-b \\|^2 +\\nu_n D_{\\psi}(y,y_n) \\right),\\\\\nz_{n+1} &=z_n+\\rho(\\mathbf{Ax}_{n+1}+By_{n+1}-b),\n\\end{align*}\nwhere $\\mathbf{x}_{n+1} =(x_{1,n+1}, \\dots, x_{m,n+1})$.\n\\step{}\nIf a termination criterion is not met, set $n =n+1$ and go to Step~\\ref{step:A2_2}.\n\\end{algorithm}\n\\end{tcolorbox}\n","start":23689,"end":25795}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"KL性质与指数","source_units":[{"raw_tex":"We now establish the convergence of the full sequence generated by Algorithm~\\ref{algo:BPLA}. Recall that a proper lower semicontinuous function $\\mathcal{F}\\colon \\mathcal{H}\\to \\left(-\\infty, +\\infty\\right]$ satisfies the \\emph{Kurdyka--Łojasiewicz (KL) property} \\cite{Kurdyka1998,Loja63} at $\\overline{x} \\in \\dom \\partial_L \\mathcal{F}$ if there are $\\eta \\in (0, +\\infty]$, a neighborhood $V$ of $\\overline{x}$, and a continuous concave function $\\varphi: \\left[0, \\eta\\right) \\to \\mathbb{R}_+$ such that $\\varphi$ is continuously differentiable with $\\varphi' > 0$ on $(0, \\eta)$, $\\varphi(0) = 0$, and, for all $x \\in V$ with $\n\\mathcal{F}(\\overline{x}) < \\mathcal{F}(x) < \\mathcal{F}(\\overline{x}) + \\eta$, \n\\begin{equation*}\n\\varphi'(\\mathcal{F}(x) -\\mathcal{F}(\\overline{x})) \\dist(0, \\partial_L \\mathcal{F}(x)) \\geq 1.   \n\\end{equation*}\nWe say that $\\mathcal{F}$ is a \\emph{KL function} if it satisfies the KL property at any point in $\\dom \\partial_L \\mathcal{F}$. If $\\mathcal{F}$ satisfies the KL property at $\\overline{x} \\in \\dom \\partial_L \\mathcal{F}$, in which the corresponding function $\\varphi$ can be chosen as $\\varphi(t) = c t ^{1 - \\lambda}$ for some $c \\in \\mathbb{R}_{++}$ and $\\lambda \\in [0, 1)$, then $\\mathcal{F}$ is said to satisfy the \\emph{KL property at $\\overline{x}$ with exponent $\\lambda$}. The function $\\mathcal{F}$ is called a \\emph{KL function with exponent $\\lambda$} if it is a KL function and has the same exponent $\\lambda$ at any $x \\in \\dom \\partial_L \\mathcal{F}$. Now we will present the proof of the full sequential convergence in the following theorem.\n","start":51504,"end":53113}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","labels":["t:full","t:full_cvg","t:full_rate","t:full_rate_finite","t:full_rate_linear","t:full_rate_sublinear"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"主定理附加条件","source_units":[{"raw_tex":"Suppose that Assumption~\\ref{a:standing} holds, that $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, that $H$ is bounded below, that $G$ is differentiable with $\\ell_G$-Lipschitz continuous gradient, and that, for all $i\\in \\{1, \\dots, m\\}$, $\\nabla{\\phi_i}$ is $\\ell_{\\phi}$-Lipschitz continuous. Let $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ be the sequence generated by Algorithm~\\ref{algo:BPLA} with $z_0\\in \\Ima(B)$ and $\\limsup_{n\\to +\\infty} \\mu_n =\\overline{\\mu} <+\\infty$.\n","start":53177,"end":53682}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":["t:full","t:full_cvg","t:full_rate","t:full_rate_finite","t:full_rate_linear","t:full_rate_sublinear"],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"扩展势函数","source_units":[{"raw_tex":"Define\n\\begin{align*}\n\\mathcal{F}(\\mathbf{x},y,z,t):=\\mathcal{L}_\\rho(\\mathbf{x},y,z) +c\\|y-t\\|^2, \\text{~~where~} c=\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}.\n\\end{align*}\n","start":53682,"end":53872}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","labels":["t:full","t:full_cvg","t:full_rate","t:full_rate_finite","t:full_rate_linear","t:full_rate_sublinear"],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"KL分支假设","source_units":[{"raw_tex":"Suppose that $\\mathcal{F}$ is a KL function. Then the sequence $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ converges to a stationary point $(\\mathbf{x}^{*},y^{*},z^{*})$ of $\\mathcal{L}_\\rho$ and \\begin{align*}\n","start":53915,"end":54130},{"raw_tex":"Suppose that $\\mathcal{F}$ is a KL function with exponent $\\kappa\\in [0,1)$. Then the following hold:\n","start":54269,"end":54371}]},{"id":"S018","semantic_type":"theorem","kind":"theorem","labels":["t:full","t:full_cvg","t:full_rate","t:full_rate_finite","t:full_rate_linear","t:full_rate_sublinear"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"完整收敛与KL速率","source_units":[{"raw_tex":"\\begin{theorem}[Full sequential convergence]\n\\label{t:full}\nSuppose that Assumption~\\ref{a:standing} holds, that $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, that $H$ is bounded below, that $G$ is differentiable with $\\ell_G$-Lipschitz continuous gradient, and that, for all $i\\in \\{1, \\dots, m\\}$, $\\nabla{\\phi_i}$ is $\\ell_{\\phi}$-Lipschitz continuous. Let $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ be the sequence generated by Algorithm~\\ref{algo:BPLA} with $z_0\\in \\Ima(B)$ and $\\limsup_{n\\to +\\infty} \\mu_n =\\overline{\\mu} <+\\infty$.\nDefine\n\\begin{align*}\n\\mathcal{F}(\\mathbf{x},y,z,t):=\\mathcal{L}_\\rho(\\mathbf{x},y,z) +c\\|y-t\\|^2, \\text{~~where~} c=\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}.\n\\end{align*}\n\\begin{enumerate}\n\\item\\label{t:full_cvg} \nSuppose that $\\mathcal{F}$ is a KL function. Then the sequence $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ converges to a stationary point $(\\mathbf{x}^{*},y^{*},z^{*})$ of $\\mathcal{L}_\\rho$ and \\begin{align*}\n\\sum_{n=0}^{+\\infty} \\|(\\mathbf{x}_{n+1},y_{n+1},z_{n+1})-(\\mathbf{x}_{n},y_{n},z_{n})\\| < +\\infty.\n\\end{align*}\n\\item\\label{t:full_rate} \nSuppose that $\\mathcal{F}$ is a KL function with exponent $\\kappa\\in [0,1)$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{t:full_rate_finite}\nIf $\\kappa =0$, then $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ converges to $(\\mathbf{x}^{*},y^{*},z^{*})$ in a finite number of steps. \n\\item\\label{t:full_rate_linear}\nIf $\\kappa\\in (0,\\frac{1}{2}]$, then there exist $\\overline{\\gamma}\\in \\mathbb{R}_{++}$ and $\\zeta\\in \\left(0,1\\right)$ such that, for all $n\\in \\mathbb{N}$, \n\\begin{align*}\n&\\|(\\mathbf{x}_{n},y_{n},z_{n})-(\\mathbf{x}^{*},y^{*},z^{*})\\| \\leq \\overline{\\gamma}\\zeta^{\\frac{n}{2}}, \\\\\n&\\|\\mathbf{Ax}_n+By_n-b\\| \\leq \\overline{\\gamma} \\zeta^{\\frac{n}{2}}, \\\\\n&\\lvert \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) -\\mathcal{L}_\\rho(\\mathbf{x}^{*},y^{*},z^{*}) \\rvert \\leq \\overline{\\gamma}\\zeta^n, \\\\\n\\text{and~} &|F(\\mathbf{x}_n,y_n)-F(\\mathbf{x}^{*},y^{*})| \\leq \\overline{\\gamma} \\zeta^{\\frac{n}{2}}.\n\\end{align*}\n\\item\\label{t:full_rate_sublinear}\nIf $\\kappa\\in (\\frac{1}{2},1)$, then there exists $\\overline{\\gamma}\\in \\mathbb{R}_{++}$ such that, for all $n\\in \\mathbb{N}$,\n\\begin{align*}\n&\\|(\\mathbf{x}_{n},y_{n},z_{n})-(\\mathbf{x}^{*},y^{*},z^{*})\\| \\leq \\overline{\\gamma} n^{-\\frac{1-\\kappa}{2\\kappa-1}}, \\\\\n&\\|\\mathbf{Ax}_n+By_n-b\\| \\leq \\overline{\\gamma} n^{-\\frac{1-\\kappa}{2\\kappa-1}}, \\\\\n&\\lvert \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) -\\mathcal{L}_\\rho(\\mathbf{x}^{*},y^{*},z^{*})\\rvert \\leq \\overline{\\gamma} n^{-\\frac{2-2\\kappa}{2\\kappa-1}}, \\\\\n\\text{and~} &|F(\\mathbf{x}_n,y_n)-F(\\mathbf{x}^{*},y^{*})| \\leq \\overline{\\gamma} n^{-\\frac{1-\\kappa}{2\\kappa-1}}.\n\\end{align*}\n\\end{enumerate}\n\\end{enumerate}\n\\end{theorem}","start":53117,"end":55930}]},{"id":"S019","semantic_type":"lemma","kind":"lemma","labels":["l:cal","l:cal_sep","l:cal_sum"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"次微分求和法则","source_units":[{"raw_tex":"\\begin{lemma}[Calculus rules]\n\\label{l:cal}\nLet $f, g\\colon \\mathbb{R}^d \\to (-\\infty, +\\infty]$ be proper lower semicontinuous functions, and let $x \\in \\mathbb{R}^d$. \n\\begin{enumerate}\n\\item\\label{l:cal_sum} \n(Sum rule) Suppose that $f$ is finite at $x$ and $g$ is locally Lipschitz around $\\overline{x}$. Then $\\partial_L (f + g)(\\overline{x}) \\subseteq \\partial_L f(\\overline{x})+\\partial_L g(\\overline{x})$. Moreover, if $g$ is strictly differentiable at $\\overline{x}$, then $\\partial_L(f + g)(\\overline{x}) = \\partial_Lf(\\overline{x}) + \\nabla g(\\overline{x})$.\n","start":16647,"end":17217}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","labels":["l:cal","l:cal_sep","l:cal_sum"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"可分和次微分","source_units":[{"raw_tex":"Let $f, g\\colon \\mathbb{R}^d \\to (-\\infty, +\\infty]$ be proper lower semicontinuous functions, and let $x \\in \\mathbb{R}^d$. \n\\begin{enumerate}\n","start":16691,"end":16835},{"raw_tex":"\\item\\label{l:cal_sep} \n(Separable sum rule) If $f(\\mathbf{x})=\\sum_{i=1}^{m}f_i(x_i)$ with $\\mathbf{x}=(x_1,\\dots,x_m)$, then $\\partial_L f(\\mathbf{x}) = \\partial_L f_1(x_1) \\times \\dots \\times \\partial_L f_m(x_m)$. \n\\end{enumerate}\n\\end{lemma}","start":17217,"end":17462}]},{"id":"S021","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"求和法则作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{l:cal_sum} follows from \\cite[Proposition~1.107(ii) and Theorem~3.36]{Mordukhovich2006}, while \\ref{l:cal_sep} follows from \\cite[Proposition~10.5]{Rockafellar1998}.\n\\end{proof}","start":17463,"end":17659}]},{"id":"S022","semantic_type":"lemma","kind":"lemma","labels":["l:upsemicont"],"member_ids":["S022","S023"],"declared_external":false,"source_only":false,"title":"次梯度聚点封闭性","source_units":[{"raw_tex":"\\begin{lemma}[Upper semicontinuity of subdifferential]\n\\label{l:upsemicont}\nLet $f\\colon \\mathbb{R}^d \\to [-\\infty, +\\infty]$ be Lipschitz continuous around $x \\in \\mathbb{R}^d$ with $|f(x)| <+\\infty$, and consider sequences $(x_n)_{n\\in \\mathbb{N}}$ and $(x_n^*)_{n\\in \\mathbb{N}}$ in $\\mathbb{R}^d$ such that $x_n \\to x$ and, for all ${n\\in \\mathbb{N}}$, $x_n^* \\in \\partial_L f(x_n)$. \nThen $(x_n^*)_{n\\in \\mathbb{N}}$ is bounded and its cluster points are contained in $\\partial_L f(x)$.\n\\end{lemma}","start":17662,"end":18165},{"raw_tex":"\\begin{proof}\nThis follows from \\cite[Lemma 2.2]{Tan2023}.\n\\end{proof}","start":18166,"end":18236}]},{"id":"S024","semantic_type":"proposition","kind":"proposition","labels":["p:Bregman","p:Bregman_1","p:Bregman_2","p:Bregman_3","p:Bregman_4"],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"Bregman非负性","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{p:Bregman}\nLet $\\phi: \\mathcal{H} \\to \\mathbb{R}$ be a differentiable convex function and $D_{\\phi}$ is the Bregman distance associated with $\\phi$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{p:Bregman_1} \nFor all $u, v \\in \\mathcal{H}$, $D_{\\phi}(u, v)\\geq 0$, and $D_{\\phi}(u, v)=0$ if and only if $u = v$. \n","start":19038,"end":19387}]},{"id":"S025","semantic_type":"proposition","kind":"proposition","labels":["p:Bregman","p:Bregman_1","p:Bregman_2","p:Bregman_3","p:Bregman_4"],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"第一变量凸性","source_units":[{"raw_tex":"Let $\\phi: \\mathcal{H} \\to \\mathbb{R}$ be a differentiable convex function and $D_{\\phi}$ is the Bregman distance associated with $\\phi$. Then the following hold:\n","start":19076,"end":19239},{"raw_tex":"\n\\item\\label{p:Bregman_2} \nFor each $v \\in \\mathcal{H}$, $D_{\\phi}(\\cdot, v)$ is convex. \n","start":19387,"end":19477}]},{"id":"S026","semantic_type":"proposition","kind":"proposition","labels":["p:Bregman","p:Bregman_1","p:Bregman_2","p:Bregman_3","p:Bregman_4"],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"Bregman二次下界","source_units":[{"raw_tex":"Let $\\phi: \\mathcal{H} \\to \\mathbb{R}$ be a differentiable convex function and $D_{\\phi}$ is the Bregman distance associated with $\\phi$. Then the following hold:\n","start":19076,"end":19239},{"raw_tex":"\n\\item\\label{p:Bregman_3} \nIf $\\phi$ is $\\alpha$-strongly convex,  then, for all $u, v \\in \\mathcal{H}$, $D_{\\phi}(u, v) \\geq \\frac{\\alpha}{2} \\|u-v\\|^2$. \n","start":19477,"end":19633}]},{"id":"S027","semantic_type":"proposition","kind":"proposition","labels":["p:Bregman","p:Bregman_1","p:Bregman_2","p:Bregman_3","p:Bregman_4"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"Bregman二次上界","source_units":[{"raw_tex":"Let $\\phi: \\mathcal{H} \\to \\mathbb{R}$ be a differentiable convex function and $D_{\\phi}$ is the Bregman distance associated with $\\phi$. Then the following hold:\n","start":19076,"end":19239},{"raw_tex":"\n\\item\\label{p:Bregman_4} \nIf $\\nabla \\phi$ is $\\ell_{\\phi}$-Lipschitz continuous, then, for all $u, v \\in \\mathcal{H}$, $D_{\\phi}(u,v)\\leq \\frac{\\ell_{\\phi}}{2}\\|u-v\\|^2$. \n\\end{enumerate}\n\\end{proposition}","start":19633,"end":19840}]},{"id":"S028","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"Bregman性质作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{p:Bregman_1} is given in \\cite[Section 1]{Bregman}. \\ref{p:Bregman_2}, \\ref{p:Bregman_3}, and \\ref{p:Bregman_4} can be easily verified by using the definitions. \n\\end{proof}","start":19841,"end":20033}]},{"id":"S029","semantic_type":"lemma","kind":"lemma","labels":["l:diff","l:diff_bounded","l:diff_descent"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"光滑下降估计","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{l:diff}\nLet $h\\colon \\mathbb{R}^d\\to \\mathbb{R}$ be a differentiable function with $\\ell$-Lipschitz continuous gradient. Then the following hold:\n\\begin{enumerate}\n\\item\\label{l:diff_descent}\nFor all $x, y\\in \\mathbb{R}^d$, $|h(y) -h(x) -\\langle \\nabla h(x), y -x\\rangle|\\leq \\frac{\\ell}{2}\\|y -x\\|^2$.\n","start":20164,"end":20488}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","labels":["l:diff","l:diff_bounded","l:diff_descent"],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"梯度校正项有下界","source_units":[{"raw_tex":"Let $h\\colon \\mathbb{R}^d\\to \\mathbb{R}$ be a differentiable function with $\\ell$-Lipschitz continuous gradient. Then the following hold:\n","start":20193,"end":20331},{"raw_tex":"\\item\\label{l:diff_bounded}\nIf $h$ is bounded below and $\\ell >0$, then\n\\begin{align*}\n\\inf_{x\\in \\mathbb{R}^d} \\left(h(x) -\\frac{1}{2\\ell}\\|\\nabla h(x)\\|^2\\right) >-\\infty.    \n\\end{align*}\n\\end{enumerate}  \n\\end{lemma}","start":20488,"end":20708}]},{"id":"S031","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"光滑估计完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{l:diff_descent}: This follows from \\cite[Lemma~1.2.3]{Nesterov2018}.\n\n\\ref{l:diff_bounded}: Let $x\\in \\mathbb{R}^d$. Following the argument of \\cite[Equation~(1.2.19)]{Nesterov2018}, we apply \\ref{l:diff_descent} with $y =x -\\frac{1}{\\ell}\\nabla h(x)$ to obtain that\n\\begin{align*}\nh\\left(x -\\frac{1}{\\ell}\\nabla h(x)\\right) &\\leq h(x) +\\langle \\nabla h(x), y -x\\rangle +\\frac{\\ell}{2}\\|y -x\\|^2 \\\\\n&= h(x) -\\frac{1}{\\ell}\\|\\nabla h(x)\\|^2 +\\frac{1}{2\\ell}\\|\\nabla h(x)\\|^2 \\\\\n&= h(x) -\\frac{1}{2\\ell}\\|\\nabla h(x)\\|^2,\n\\end{align*}\nwhich together with the boundedness of $h$ implies the conclusion.\n\\end{proof}","start":20709,"end":21339}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"援引下降引理","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{l:diff_descent}: This follows from \\cite[Lemma~1.2.3]{Nesterov2018}.\n\n","start":20709,"end":20798}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"负梯度一步估计","source_units":[{"raw_tex":"\\ref{l:diff_bounded}: Let $x\\in \\mathbb{R}^d$. Following the argument of \\cite[Equation~(1.2.19)]{Nesterov2018}, we apply \\ref{l:diff_descent} with $y =x -\\frac{1}{\\ell}\\nabla h(x)$ to obtain that\n\\begin{align*}\nh\\left(x -\\frac{1}{\\ell}\\nabla h(x)\\right) &\\leq h(x) +\\langle \\nabla h(x), y -x\\rangle +\\frac{\\ell}{2}\\|y -x\\|^2 \\\\\n&= h(x) -\\frac{1}{\\ell}\\|\\nabla h(x)\\|^2 +\\frac{1}{2\\ell}\\|\\nabla h(x)\\|^2 \\\\\n&= h(x) -\\frac{1}{2\\ell}\\|\\nabla h(x)\\|^2,\n\\end{align*}\nwhich together with the boundedness of $h$ implies the conclusion.\n","start":20798,"end":21328}]},{"id":"S034","semantic_type":"lemma","kind":"lemma","labels":["l:norm","l:norm_M'x","l:norm_Mx"],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"矩阵正向谱下界","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{l:norm}\nLet $L$ be a symmetric matrix in $\\mathbb{R}^{d\\times d}$ and let $M$ be a matrix in $\\mathbb{R}^{p\\times d}$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{l:norm_Mx}\nFor all $x\\in \\mathbb{R}^d$, $\\sqrt{\\lambda_{\\min}(M^\\top M)}\\|x\\| \\leq \\|Mx\\|$.\n","start":28813,"end":29100}]},{"id":"S035","semantic_type":"lemma","kind":"lemma","labels":["l:norm","l:norm_M'x","l:norm_Mx"],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"对称矩阵谱估计","source_units":[{"raw_tex":"Let $L$ be a symmetric matrix in $\\mathbb{R}^{d\\times d}$ and let $M$ be a matrix in $\\mathbb{R}^{p\\times d}$. Then the following hold:\n","start":28842,"end":28978},{"raw_tex":"\\item\\label{l:norm_M'x}\nFor all $x\\in \\mathbb{R}^d$, $\\gamma(x^\\top Lx) \\leq \\|Lx\\|^2$, where $\\gamma =\\min\\{|\\lambda|: \\lambda \\text{~is a nonzero eigenvalue of~} L\\}$. Consequently, for all $z\\in \\Ima(M)$, $\\sqrt{\\lambda_*(M^\\top M)}\\|z\\| \\leq \\|M^\\top z\\|$, where $\\lambda_*(M^\\top M)$ is the smallest strictly positive eigenvalue of $M^\\top M$. \n","start":29100,"end":29450}]},{"id":"S036","semantic_type":"lemma","kind":"lemma","labels":["l:norm","l:norm_M'x","l:norm_Mx"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"像空间转置谱下界","source_units":[{"raw_tex":"Let $L$ be a symmetric matrix in $\\mathbb{R}^{d\\times d}$ and let $M$ be a matrix in $\\mathbb{R}^{p\\times d}$. Then the following hold:\n","start":28842,"end":28978},{"raw_tex":"\\item\\label{l:norm_M'x}\nFor all $x\\in \\mathbb{R}^d$, $\\gamma(x^\\top Lx) \\leq \\|Lx\\|^2$, where $\\gamma =\\min\\{|\\lambda|: \\lambda \\text{~is a nonzero eigenvalue of~} L\\}$. Consequently, for all $z\\in \\Ima(M)$, $\\sqrt{\\lambda_*(M^\\top M)}\\|z\\| \\leq \\|M^\\top z\\|$, where $\\lambda_*(M^\\top M)$ is the smallest strictly positive eigenvalue of $M^\\top M$. \n\\end{enumerate}\n\\end{lemma}","start":29100,"end":29477}]},{"id":"S037","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S037"],"declared_external":false,"source_only":true,"title":"谱引理源文完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{l:norm_Mx}: This follows from the fact that $\\|Mx\\|^2 =x^\\top (M^\\top M)x$.\n\n\\ref{l:norm_M'x}: \nThe first assertion follows from \\cite[Lemma~4.5]{BDL23} and its proof. Now, let $z\\in \\Ima(M)$. Then there exists $x\\in \\mathbb{R}^d$ such that $z =Mx$. Since $M^\\top M$ is a symmetric matrix in $\\mathbb{R}^{d\\times d}$ with all eigenvalues being nonnegative, we have from the first assertion that\n\\begin{align*}\n\\lambda_*(M^\\top M)\\|z\\|^2 =\\lambda_*(M^\\top M)(x^\\top M^\\top Mx) \\leq \\|M^\\top Mx\\|^2 =\\|M^\\top z\\|^2,    \n\\end{align*}\nwhich completes the proof.\n\\end{proof}\n","start":29478,"end":30067}]},{"id":"S038","semantic_type":"theorem","kind":"theorem","labels":["eq:decrease","t:cvg","t:cvg_decrease","t:cvg_seq"],"member_ids":["S038","S042","S044","S045","S046","S047","S048","S050","S052","S053","S054","S055","S056"],"declared_external":false,"source_only":false,"title":"子序列收敛定理","source_units":[{"raw_tex":"\\begin{theorem}[Subsequential convergence]\n\\label{t:cvg}\nSuppose that Assumption~\\ref{a:standing} holds. Let $(\\mathbf{x}_n,y_n,z_n)_{n\\in \\mathbb{N}}$ be the sequence generated by Algorithm~\\ref{algo:BPLA} and, for each $n\\in \\mathbb{N}^*$, define\n\\begin{align*}\nL_n =\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) +c\\|y_n-y_{n-1}\\|^2, \\text{~where~} c=\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}.     \n\\end{align*}\nThen the following hold:\n\\begin{enumerate}\n\\item\\label{t:cvg_decrease}\nFor all $n\\in \\mathbb{N}^*$,\n\\begin{align}\\label{eq:decrease}\nL_{n+1} +\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\delta_y\\|y_{n+1}-y_n\\|^2 \\leq L_n,\n\\end{align}\nwhere $\\delta_{\\mathbf{x}} =\\frac{\\mu\\alpha -\\ell_P -\\beta}{2} > 0$ and $\\delta_y =\\frac{\\lambda\\rho}{2} -\\frac{(\\ell_H+2\\nu\\ell_{\\psi})^2}{\\lambda\\rho} -\\frac{\\ell_H}{2} >0$.\n\n\\item\\label{t:cvg_seq}\nSuppose that $z_0\\in \\Ima(B)$, that $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, and that $H$ is bounded below. \nThen the sequence $(\\mathbf{x}_n,{y}_n,{z}_n)_{n\\in \\mathbb{N}}$ is bounded and $\\mathbf{x}_{n+1}-\\mathbf{x}_n\\to 0$, $y_{n+1}-y_n\\to 0$, $z_{n+1}-z_n\\to 0$, and $\\mathbf{Ax}_n +By_n \\to b$ as $n\\to +\\infty$. Furthermore, for every cluster point $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ of $(\\mathbf{x}_n,{y}_n,{z}_n)_{n\\in \\mathbb{N}}$, it holds that\n\\begin{align*}\n\\mathbf{A}\\overline{\\mathbf{x}} +b\\overline{y} =b \\text{~~and~~} \\lim_{n \\to +\\infty} F(\\mathbf{x}_{n },y_{n}) =\\lim_{n \\to +\\infty}\\mathcal{L}_\\rho(\\mathbf{x}_{n },y_{n},z_{n}) =\\mathcal{L}_\\rho(\\overline{\\mathbf{x}},\\overline{y},\\overline{z}) =F(\\overline{\\mathbf{x}},\\overline{y}),    \n\\end{align*}\nand that if $G$ is strictly differentiable and, for each $i \\in \\{1,\\dots,m\\}$, $\\nabla \\phi_i$ is $\\ell_{\\phi}$-Lipschitz continuous, then $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ is a stationary point of $\\mathcal{L}_\\rho$.\n\\end{enumerate}\n\\end{theorem}","start":30177,"end":32134},{"raw_tex":"\\begin{proof}\n\\ref{t:cvg_decrease}: Let $n\\in \\mathbb{N}$. The update of $z_{n+1}$ yields\n\\begin{align}\\label{eq:z-update}\nz_{n+1} -z_n =\\rho(\\mathbf{Ax}_{n+1}+By_{n+1}-b).   \n\\end{align}\nBy the optimality condition for the update of $y_{n+1}$ in Step~\\ref{step:A2_2} of Algorithm~\\ref{algo:BPLA}, \n\\begin{align*}\n0 &= \\nabla H(y_{n+1}) + B^\\top z_n +\\rho B^\\top(\\mathbf{Ax}_{n+1} +By_{n+1} -b) +\\nu_n(\\nabla \\psi(y_{n+1}) - \\nabla \\psi(y_n)),\n\\end{align*}\nwhich combined with \\eqref{eq:z-update} yields\n\\begin{align}\\label{eq:B'z}\nB^\\top z_{n+1} =-\\nabla H(y_{n+1}) -\\nu_n(\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)).  \n\\end{align}\nFrom the definition of the augmented Lagrangian, we have that\n\\begin{align}\\label{eq:zz}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n}) &= \\langle z_{n+1}-z_n, \\mathbf{Ax}_{n+1}+By_{n+1}-b \\rangle \\notag\\\\\n&= \\frac{1}{\\rho}\\|z_{n+1}-z_n\\|^2 \n\\end{align}\nand that\n\\begin{align*}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_n,z_n) &= H(y_{n+1}) - H(y_n) + \\langle z_n, B(y_{n+1}-y_n) \\rangle \\\\\n&\\qquad + \\frac{\\rho}{2}(\\|\\mathbf{Ax}_{n+1}+By_{n+1}-b\\|^2 - \\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2).  \n\\end{align*}\nWriting $\\mathbf{Ax}_{n+1}+By_{n}-b =(\\mathbf{Ax}_{n+1}+By_{n+1}-b) -(By_{n+1}- By_n)$, we obtain that\n\\begin{align*}\n&\\|\\mathbf{Ax}_{n+1}+By_{n+1}-b\\|^2 - \\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2 \\\\\n&= -\\|B(y_{n+1}-y_n)\\|^2 +2\\langle  \\mathbf{Ax}_{n+1}+By_{n+1}-b, B(y_{n+1}-y_n)\\rangle \\\\\n&= -\\|B(y_{n+1}-y_n)\\|^2 +\\frac{2}{\\rho}\\langle  z_{n+1}-z_{n}, B(y_{n+1}-y_n)\\rangle,\n\\end{align*}\nand so\n\\begin{align*}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_n,z_n) = H(y_{n+1}) - H(y_n) +\\langle z_{n+1}, B(y_{n+1}-y_n) \\rangle -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2.\n\\end{align*}\nIn view of \\eqref{eq:B'z},\n\\begin{align*}\n\\langle z_{n+1}, B(y_{n+1}-y_n) \\rangle &=\\langle B^\\top z_{n+1}, y_{n+1}-y_n \\rangle \\\\\n&=-\\langle \\nabla H(y_{n+1}), y_{n+1}-y_n \\rangle -\\nu_n\\langle \\nabla \\psi(y_{n+1})- \\nabla \\psi(y_n), y_{n+1}-y_n \\rangle \\\\\n&\\leq -H(y_{n+1}) +H(y_n) +\\frac{\\ell_H}{2}\\|y_{n+1}-y_n\\|^2,\n\\end{align*}\nwhere the last inequality is from the $\\ell_H$-Lipschitz continuity of $\\nabla H$ and Lemma~\\ref{l:diff}\\ref{l:diff_descent}, and from the monotonicity of $\\nabla \\psi$. \nWe therefore obtain that\n\\begin{align}\\label{eq:yy}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_n,z_n) \\leq \\frac{\\ell_H}{2}\\|y_{n+1}-y_n\\|^2 -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2.    \n\\end{align}\nNext, by the update of $x_{i,n+1}$ in Step~\\ref{step:A2_2} of Algorithm~\\ref{algo:BPLA}, for all $i \\in \\{1,\\dots,m\\}$ and all $x_i \\in \\mathbb{R}^{d_i}$,\n\\begin{align}\\label{eq:update_step_x}\n&f_i(x_{i,n+1}) +\\langle \\nabla_i P(\\mathbf{x}_n) -g_{i,n}, x_{i,n+1} \\rangle +\\langle z_n, A_{i}x_{i,n+1} \\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,n+1}(x_{i,n+1}) +By_n -b\\|^2 +\\mu_n D_{\\phi_i}(x_{i,n+1},x_{i,n}) \\notag \\\\\n&\\leq  f_i(x_i) +\\langle \\nabla_i P(\\mathbf{x}_n) -g_{i,n}, x_i \\rangle +\\langle z_n, A_{i}x_i\\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,n+1}(x_i) +By_n -b\\|^2 +\\mu_n D_{\\phi_i}(x_i,x_{i,n}).\n\\end{align}\nLetting $x_i =x_{i,n}$, using Proposition~\\ref{p:Bregman}\\ref{p:Bregman_1}\\&\\ref{p:Bregman_3}, and summing up over $i \\in \\{1,\\dots,m\\}$, we derive that\n\\begin{align}\\label{eq:from x-update}\n&\\sum_{i=1}^m f_i(x_{i,n+1}) +\\langle \\nabla P(\\mathbf{x}_n) -g_n, \\mathbf{x}_{n+1} \\rangle +\\langle z_n, \\mathbf{Ax}_{n+1} \\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{x}_{n+1} +By_n -b\\|^2 +\\frac{\\mu\\alpha}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 \\notag\\\\\n&\\leq \\sum_{i=1}^m f_i(x_{i,n}) +\\langle \\nabla P(\\mathbf{x}_n) -g_n, \\mathbf{x}_{n} \\rangle +\\langle z_n, \\mathbf{Ax}_{n} \\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{x}_n +By_n -b\\|^2.\n\\end{align}\nOn the other hand,\n\\begin{align}\\label{eq:xx}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_n,z_n) \\notag\\\\\n&= \\sum_{i=1}^m f_i(x_{i,n+1}) -\\sum_{i=1}^m f_i(x_{i,n}) +P(\\mathbf{x}_{n+1}) -P(\\mathbf{x}_{n}) -G(\\mathbf{x}_{n+1}) +G(\\mathbf{x}_{n}) \\notag\\\\ \n&\\quad +\\langle z_n, \\mathbf{Ax}_{n+1}-\\mathbf{Ax}_{n}\\rangle +\\frac{\\rho}{2}\\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2 -\\frac{\\rho}{2}\\|\\mathbf{Ax}_{n}+By_{n}-b\\|^2 \\notag\\\\\n&\\leq \\sum_{i=1}^m f_i(x_{i,n+1}) -\\sum_{i=1}^m f_i(x_{i,n}) +\\langle \\nabla P(\\mathbf{x}_n), \\mathbf{x}_{n+1}-\\mathbf{x}_n \\rangle + \\frac{\\ell_P}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_n\\|^2 - \\langle g_n, \\mathbf{x}_{n+1} -\\mathbf{x}_{n} \\rangle + \\frac{\\beta}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 \\notag\\\\\n&\\quad +\\langle z_n, \\mathbf{Ax}_{n+1}-\\mathbf{Ax}_{n}\\rangle  + \\frac{\\rho}{2}\\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2 - \\frac{\\rho}{2}\\|\\mathbf{Ax}_{n}+By_{n}-b\\|^2 \\notag\\\\\n&\\leq -\\frac{\\mu\\alpha -\\ell_P -\\beta}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 =-\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2,\n\\end{align}\nwhere the first inequality is obtained by using Lemma~\\ref{l:diff}\\ref{l:diff_descent} on the $\\ell_P$-Lipschitz continuity of $\\nabla P$ and \\cite[Lemma~4.1]{BDL22} on the $\\beta$-weak convexity of $G$, while the last inequality follows from \\eqref{eq:from x-update}. Summing up three relations \\eqref{eq:zz}, \\eqref{eq:yy}, and \\eqref{eq:xx} yields\n\\begin{align}\\label{eq:LL}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) - \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) \\notag\\\\\n&\\leq -\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\frac{\\ell_H}{2}\\|y_{n+1}-y_n\\|^2 -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2 +\\frac{1}{\\rho}\\|z_{n+1}-z_n\\|^2.\n\\end{align}\n\nNow, since $\\Ima(\\mathbf{A}) \\cup \\{b\\} \\subseteq \\Ima(B)$, it follows from \\eqref{eq:z-update} that\n\\begin{align}\\label{eq:zImB}\nz_{n+1}-z_n \\in \\Ima(B)    \n\\end{align}\nwhich, by Lemma~\\ref{l:norm}\\ref{l:norm_M'x}, yields \n\\begin{align*}\n\\|z_{n+1} -z_n\\| \\leq \\frac{1}{\\sqrt{\\lambda}}\\|B^\\top z_{n+1} -B^\\top z_n\\|.\n\\end{align*}\nLet $n\\in \\mathbb{N}^*$. We derive from \\eqref{eq:B'z} and the Lipschitz continuity of $\\nabla H$ and $\\nabla \\psi$ that\n\\begin{align}\\label{eq:BTZy}\n\\|B^\\top z_{n+1} -B^\\top z_n\\| &\\leq \\|\\nabla H(y_{n+1}) -\\nabla H(y_n)\\| +\\nu_n\\|\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)\\| +\\nu_{n-1}\\|\\nabla \\psi(y_n) - \\nabla \\psi(y_{n-1})\\| \\notag \\\\ \n&\\leq (\\ell_H +\\nu\\ell_{\\psi})\\|y_{n+1} -y_n\\| +\\nu\\ell_{\\psi}\\|y_n -y_{n-1}\\|,\n\\end{align}\nand thus, \n\\begin{align}\\label{eq:zy}\n\\|z_{n+1} -z_n\\| \\leq \\frac{1}{\\sqrt{\\lambda}}\\|B^\\top z_{n+1} -B^\\top z_n\\| \\leq \\frac{1}{\\sqrt{\\lambda}}((\\ell_H +\\nu\\ell_{\\psi})\\|y_{n+1} -y_n\\| +\\nu\\ell_{\\psi}\\|y_n -y_{n-1}\\|).\n\\end{align}\nApplying Cauchy--Schwarz inequality to two vectors $u =(\\sqrt{\\ell_H+\\nu\\ell_{\\psi}},\\sqrt{\\nu\\ell_{\\psi}})$ and $v =(\\sqrt{\\ell_H+\\nu\\ell_{\\psi}}\\|y_{n+1} -y_n\\|,\\sqrt{\\nu\\ell_{\\psi}}\\|y_n -y_{n-1}\\|)$, we obtain that\n\\begin{align*}\n\\|z_{n+1} -z_n\\|^2 \\leq \\frac{\\ell_H +2\\nu\\ell_{\\psi}}{\\lambda}((\\ell_H +\\nu\\ell_{\\psi})\\|y_{n+1} -y_n\\|^2 +\\nu\\ell_{\\psi}\\|y_n -y_{n-1}\\|^2).    \n\\end{align*}\nBy combining this with \\eqref{eq:LL} and using Lemma~\\ref{l:norm}\\ref{l:norm_Mx},\n\\begin{align*}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) - \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) \\notag\\\\\n&\\leq -\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\left(\\frac{(\\ell_H+2\\nu\\ell_{\\psi})(\\ell_H+\\nu\\ell_{\\psi})}{\\lambda\\rho} +\\frac{\\ell_H}{2}\\right)\\|y_{n+1}-y_n\\|^2 \\\\ \n&\\qquad -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2 +\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2 \\\\\n&\\leq -\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 -\\left(\\frac{\\lambda\\rho}{2} -\\frac{(\\ell_H+2\\nu\\ell_{\\psi})(\\ell_H+\\nu\\ell_{\\psi})}{\\lambda\\rho} -\\frac{\\ell_H}{2}\\right)\\|y_{n+1}-y_n\\|^2 \\\\\n&\\qquad +\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2,\n\\end{align*}\nor equivalently,\n\\begin{align*}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) +\\frac{(\\ell_H +2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_{n+1}-y_{n}\\|^2 + \\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\delta_y\\|y_{n+1}-y_n\\|^2 \\\\\n&\\leq \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) +\\frac{(\\ell_H +2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2,\n\\end{align*}\nwhich proves \\eqref{eq:decrease}. Here, we note that $\\delta_{\\mathbf{x}} =\\frac{\\mu\\alpha -\\ell_P -\\beta}{2} > 0$ since $\\mu > \\frac{\\ell_P+\\beta}{\\alpha}$ and that $\\delta_y =\\frac{\\lambda\\rho}{2} -\\frac{(\\ell_H+2\\nu\\ell_{\\psi})^2}{\\lambda\\rho} -\\frac{\\ell_H}{2} >0$ since $\\rho > \\frac{\\ell_H +\\sqrt{\\ell_H^2 +8(\\ell_H +2\\nu\\ell_{\\psi})^2}}{2\\lambda}$.\n\n\\ref{t:cvg_seq}: Let $n\\in \\mathbb{N}^*$. As $z_0 \\in \\Ima(B)$, we have from \\eqref{eq:zImB} that $z_n \\in \\Ima (B)$. Using Lemma~\\ref{l:norm}\\ref{l:norm_M'x}, \\eqref{eq:B'z}, and the Lipschitz continuity of $\\psi$ yields\n\\begin{align*}\n\\sqrt{\\lambda}\\|z_n\\|\\leq \\|B^\\top z_n\\| \\leq\\|\\nabla H(y_n)\\|+ \\nu_n\\|\\nabla \\psi(y_{n}) -\\nabla \\psi(y_{n-1})\\| \n\\leq \\|\\nabla H(y_n)\\|+ \\nu \\ell_{\\psi}\\|y_{n}-y_{n-1}\\|.\n\\end{align*}\nFor $\\omega\\in \\mathbb{R}_{++}$ to be chosen later, invoking Cauchy--Schwarz inequality with two vectors $u =(\\sqrt{\\omega},\\sqrt{\\nu\\ell_{\\psi}})$ and $v =(\\frac{1}{\\sqrt{\\omega}}\\|\\nabla H(y_n)\\|,\\sqrt{\\nu\\ell_{\\psi}}\\|y_n-y_{n-1}\\|)$, we have that\n\\begin{align}\\label{eq:uppper_zn}\n\\|z_n\\|^2 &\\leq \\frac{\\omega +\\nu\\ell_{\\psi}}{\\lambda} \\left(\\frac{1}{\\omega}\\|\\nabla H(y_n)\\|^2 +\\nu\\ell_{\\psi}\\|y_n-y_{n-1}\\|^2\\right) \\notag \\\\\n&= \\frac{1}{\\lambda}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right)\\|\\nabla H(y_n)\\|^2 +\\frac{(\\omega +\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda}\\|y_n-y_{n-1}\\|^2.\n\\end{align}\nCombining this with the fact that\n\\begin{align*}\n\\langle z_n, \\mathbf{Ax}_n+By_n-b \\rangle +\\frac{\\rho}{2}\\|\\mathbf{Ax}_n+By_n-b\\|^2 =\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} +\\mathbf{Ax}_n+By_n-b \\right\\|^2 -\\frac{1}{2\\rho}\\|z_n\\|^2,\n\\end{align*}\nwe deduce that\n\\begin{align*}\nL_n &=\\mathcal{L}_\\rho(\\mathbf{x}_n,y_n,z_n) + c\\|y_n-y_{n-1}\\|^2 \\\\\n&\\geq \\sum_{i=1}^m f_i(x_{i,n}) +H(y_n) +P(\\mathbf{x}_n) -G(\\mathbf{x}_n) \\\\ \n&\\quad +\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} + \\mathbf{Ax}_n+By_n-b \\right\\|^2 -\\frac{1}{2\\lambda\\rho}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right)\\|\\nabla H(y_n)\\|^2 +\\left(c -\\frac{(\\omega +\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{2\\lambda\\rho}\\right)\\|y_n-y_{n-1}\\|^2 \\\\\n&= \\left(\\sum_{i=1}^m f_i(x_{i,n}) +P(\\mathbf{x}_n) -G(\\mathbf{x}_n)\\right) +\\left(H(y_n) -\\frac{1}{2\\ell_H}\\|\\nabla H(y_n)\\|^2\\right) +\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} + \\mathbf{Ax}_n+By_n-b \\right\\|^2 \\\\ \n&\\quad +\\left(\\frac{1}{2\\ell_H} -\\frac{1}{2\\lambda\\rho}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right)\\right)\\|\\nabla H(y_n)\\|^2 +\\frac{\\nu\\ell_{\\psi}}{2\\lambda\\rho}(2\\ell_H +3\\nu\\ell_{\\psi} -\\omega)\\|y_n-y_{n-1}\\|^2.\n\\end{align*}\nNow, we note that\n\\begin{align*}\n\\rho > \\frac{\\ell_H +\\sqrt{\\ell_H^2 +8(\\ell_H +2\\nu\\ell_{\\psi})^2}}{2\\lambda} \\geq \\frac{\\ell_H +\\sqrt{\\ell_H^2 +8\\ell_H^2}}{2\\lambda} =\\frac{2\\ell_H}{\\lambda},    \n\\end{align*}\nand so $\\frac{1}{2\\ell_H} >\\frac{1}{\\lambda\\rho}$. Choosing $\\omega =1$ if $\\nu\\ell_{\\psi} =0$, and $\\omega =\\nu\\ell_{\\psi}$ if $\\nu\\ell_{\\psi} >0$, it holds that\n\\begin{align*}\n&\\theta :=\\frac{1}{2\\ell_H} -\\frac{1}{2\\lambda\\rho}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right) \\geq \\frac{1}{2\\ell_H} -\\frac{1}{2\\lambda\\rho}(1 +1) >0 \\text{~~and~~} \\\\\n&\\frac{\\nu\\ell_{\\psi}}{2\\lambda\\rho}(2\\ell_H +3\\nu\\ell_{\\psi} -\\omega) \\geq \\frac{\\nu\\ell_{\\psi}}{2\\lambda\\rho}(3\\nu\\ell_{\\psi} -\\omega) =\\frac{\\nu^2\\ell_{\\psi}^2}{\\lambda\\rho}. \n\\end{align*}\nThis together with \\ref{t:cvg_decrease} yields\n\\begin{align}\\label{eq:L_n_bounded}\nL_1 \\geq L_n &\\geq \\left(\\sum_{i=1}^m f_i(x_{i,n}) +P(\\mathbf{x}_n) -G(\\mathbf{x}_n)\\right) +\\left(H(y_n) -\\frac{1}{2\\ell_H}\\|\\nabla H(y_n)\\|^2\\right) +\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} + \\mathbf{Ax}_n+By_n-b \\right\\|^2 \\notag \\\\ \n&\\quad +\\theta\\|\\nabla H(y_n)\\|^2 +\\frac{\\nu^2\\ell_{\\psi}^2}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2,\n\\end{align}\nin which the first term on the right hand side is bounded below since $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, the second term is bounded below due to {the assumption on $H$ and Lemma~\\ref{l:diff}\\ref{l:diff_bounded}}, while the remaining terms are nonnegative due to their positive coefficients. Therefore, all the terms are bounded. Again, as $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, the boundedness of the first term implies that $(\\mathbf{x}_{n})_{n\\in \\mathbb{N}}$ is bounded. Since the last two terms are bounded, it follows from \\eqref{eq:uppper_zn} that $(z_n)_{n\\in \\mathbb{N}}$ is bounded. Since $z_{n+1}=z_n+\\rho(\\mathbf{Ax}_{n+1}+By_{n+1}-b)$ and $B$ has full column rank, we also obtain that $(y_n)_{n\\in \\mathbb{N}}$ is bounded. Consequently, the sequence $(\\mathbf{x}_n,y_n,z_n)_{n\\in \\mathbb{N}}$ is bounded.\n\nOn the other hand, since all the terms on the right hand side of \\eqref{eq:L_n_bounded} are bounded below, so is the sequence $(L_n)_{n\\in\\mathbb{N}}$. Combining with the nonincreasing property of $(L_n)_{n\\in\\mathbb{N}}$ due to \\ref{t:cvg_decrease}, it follows that $(L_n)_{n\\in\\mathbb{N}}$ is convergent. By rearranging \\eqref{eq:decrease} and performing telescoping, we then have that\n\\begin{align*}\n\\sum_{n=1}^{+\\infty}\\delta_y \\|y_{n+1}-y_n\\|^2 + \\delta_{\\mathbf{x}}\\sum_{n=1}^{+\\infty}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 \\leq \\sum_{n=1}^{+\\infty}(L_n - L_{n+1}) = L_1 - \\lim_{n\\to+\\infty} L_n < +\\infty,\n\\end{align*}\nwhich implies that $\\sum_{n=1}^{+\\infty}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_n\\|^2 < +\\infty$, $\\sum_{n=1}^{+\\infty}\\|y_{n+1}-y_n\\|^2 <+\\infty$, and by \\eqref{eq:zy}, $\\sum_{n=1}^{+\\infty}\\|z_{n+1}-z_n\\|^2 < +\\infty$. As a result, $\\mathbf{x}_{n+1}-\\mathbf{x}_{n} \\to 0$, $y_{n+1}-y_n \\to 0$, $z_{n+1}-z_n \\to 0$, and $\\mathbf{Ax}_n +By_n =(z_n -z_{n-1})/\\rho +b \\to b$ as $n \\to +\\infty$. It then follows from the definition of $L_n$ that the sequence $(\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}))_{n\\in\\mathbb{N}}$ is also convergent with $\\lim_{n\\to+\\infty} \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) =\\lim_{n\\to+\\infty}L_n$.\n\nLet $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ be a cluster point of the sequence  $(\\mathbf{x}_n,y_n,z_n)_{n\\in \\mathbb{N}}$. Then there exists a subsequence $(\\mathbf{x}_{k_n},y_{k_n},z_{k_n})_{n\\in \\mathbb{N}}$ that converges to $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$. In view of \\eqref{eq:update_step_x}, letting $n =k_n$ and using Proposition~\\ref{p:Bregman}\\ref{p:Bregman_3}\\&\\ref{p:Bregman_4}, we obtain that, for all $i \\in \\{1,\\dots,m\\}$ and all $x_i\\in \\mathbb{R}^{d_i}$,\n\\begin{align}\\label{eq:kn+}\n&f_i(x_{i,k_n+1}) +\\langle \\nabla_i P(\\mathbf{x}_{k_n})- g_{i,k_n}, x_{i,k_n+1} -x_i \\rangle + \\langle z_{k_n}, A_i x_{i,k_n+1} - A_ix_i\\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,k_n+1}(x_{i,k_n+1}) +By_{k_n} -b \\|^2 \\notag \\\\\n&\\quad  - \\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,k_n+1}(x_{i}) +By_{k_n} -b \\|^2 +\\frac{\\mu\\alpha}{2}\\|x_{i,k_n+1}-x_{i,k_n}\\|^2 - \\frac{\\nu\\ell_{\\phi}}{2}\\|x_{i}-x_{i,k_n}\\|^2 \\leq f_i(x_{i}).\n\\end{align}\nSince $\\nabla P$ is continuous, $\\nabla P(\\mathbf{x}_{k_n}) \\to \\nabla P(\\overline{\\mathbf{x}})$ as $n\\to +\\infty$. As $G +\\frac{\\beta}{2}\\|\\cdot\\|^2$ is a continuous convex function, it follows from \\cite[Example~9.14]{Rockafellar1998} that $G$ is locally Lipschitz continuous. Since $\\mathbf{x}_{k_n} \\to \\overline{\\mathbf{x}}$ as $n\\to +\\infty$, using Lemma~\\ref{l:upsemicont} and passing to a subsequence if necessary, we can and do assume that $g_{k_n}\\to \\overline{g} \\in \\partial_L G (\\overline{\\mathbf{x}})$ as $n\\to +\\infty$. Now, for each $i \\in \\{1,\\dots,m\\}$, letting $x_i =\\overline{x}_i$ and $n\\to +\\infty$ in \\eqref{eq:kn+} with noting that $x_{i,k_n+1} \\to \\overline{x}_i$, $y_{k_n} \\to \\overline{y}$, and $z_{k_n} \\to \\overline{z}$, we have that $\\limsup_{n \\to +\\infty} f_i(x_{i,k_n+1}) \\leq f_i(\\overline{x}_i)$, which together with the lower semicontinuity of $f_i$ yields\n\\begin{align*}\n\\lim_{n \\to +\\infty} f_i(x_{i,k_n+1}) = f_i(\\overline{x}_i).\n\\end{align*}\nAs $H$, $P$, and $G$ are continuous, it follows that $\\lim_{n \\to +\\infty}\\mathcal{L}_\\rho(\\mathbf{x}_{k_n +1 },y_{k_n+1},z_{k_n +1}) = \\mathcal{L}_\\rho(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$. Since the sequence $(\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}))_{n\\in\\mathbb{N}}$ is convergent, we deduce that\n\\begin{align*}\n\\lim_{n \\to +\\infty}\\mathcal{L}_\\rho(\\mathbf{x}_{n },y_{n},z_{n}) =\\mathcal{L}_\\rho(\\overline{\\mathbf{x}},\\overline{y},\\overline{z}).    \n\\end{align*}\n\nWe next show that $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ is a stationary point of $\\mathcal{L}_\\rho$. In \\eqref{eq:z-update} and \\eqref{eq:B'z} with $n$ replaced by $k_n-1$, letting $n\\to +\\infty$ and using the coninuity of $\\nabla H$ and the Lipschitz continuity of $\\nabla \\psi$, we obtain that $0 =\\mathbf{A\\overline{x}} + B\\overline{y} -b$ and $0 =\\nabla H(\\overline{y}) + B^\\top \\overline{z}$. In turn, letting $n \\to +\\infty$ in \\eqref{eq:kn+}, we have for all $i \\in \\{1,\\dots,m\\}$ and all $x_i\\in \\mathbb{R}^{d_i}$ that\n\\begin{align*}\n&f_i(\\overline{x}_i) +\\langle \\nabla_i P(\\overline{\\mathbf{x}})-\\overline{g}_i, \\overline{x}_i-x_i \\rangle + \\langle \\overline{z}, A_i\\overline{x}_i-A_ix_i \\rangle\\\\\n&\\quad +\\frac{\\rho}{2}\\|A_1\\overline{x}_1 +\\dots +A_{i-1}\\overline{x}_{i-1} +A_i\\overline{x}_i +A_{i+1}\\overline{x}_{i+1}+\\dots+A_m \\overline{x}_{m} +B\\overline{y} -b \\|^2 \\\\\n&\\quad -\\frac{\\rho}{2}\\|A_1\\overline{x}_1 +\\dots +A_{i-1}\\overline{x}_{i-1} +A_ix_i +A_{i+1}\\overline{x}_{i+1}+\\dots+A_m \\overline{x}_{m} +B\\overline{y} -b \\|^2 -\\frac{\\nu\\ell_{ \\phi}}{2}\\|x_i-\\overline{x}_i\\|^2 \\leq f_i(x_i),\n\\end{align*}\nwhich can be written as\n\\begin{align*}\nh_i(\\overline{x}_i) \\leq h_i(x_i),    \n\\end{align*}\nwhere $h_i(x_i) =f_i(x_i) +\\langle \\nabla_i P(\\overline{\\mathbf{x}}) -\\overline{g}_i, x_i \\rangle + \\langle \\overline{z}, A_ix_i \\rangle +\\frac{\\rho}{2}\\|A_1\\overline{x}_1 +\\dots +A_{i-1}\\overline{x}_{i-1} +A_ix_i +A_{i+1}\\overline{x}_{i+1}+\\dots+A_m \\overline{x}_{m} +B\\overline{y} -b \\|^2 +\\frac{\\nu\\ell_{\\phi}}{2}\\|x_i-\\overline{x}_i\\|^2$. It follows that, for all $i \\in \\{1,\\dots,m\\}$,\n\\begin{align*}\n0 &\\in \\partial_L h_i(\\overline{x}_i) =\\partial_L (f_i)(\\overline{x}_i) +\\nabla_i P(\\overline{\\mathbf{x}})- \\overline{g}_i +A_i^\\top \\overline{z} +\\rho A_i^\\top (\\mathbf{A}\\overline{\\mathbf{x}}+B\\overline{y}-b).\n\\end{align*}\nSince $\\mathbf{A}\\overline{\\mathbf{x}}+B\\overline{y}-b =0$, we obtain that\n\\begin{align*}\n0 \\in \\partial_L (f_1)(\\overline{x}_1)\\times \\cdots \\times \\partial_L (f_m)(\\overline{x}_m) +\\nabla P(\\overline{\\mathbf{x}}) -\\overline{g} +\\mathbf{A}^\\top \\overline{z},\n\\end{align*}\nwhich completes the proof due to Lemma~\\ref{l:cal}\\ref{l:cal_sum}\\&\\ref{l:cal_sep} and the strict differentiability of $G$.\n\\end{proof}","start":32135,"end":50863},{"raw_tex":"From the definition of the augmented Lagrangian, we have that\n\\begin{align}\\label{eq:zz}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n}) &= \\langle z_{n+1}-z_n, \\mathbf{Ax}_{n+1}+By_{n+1}-b \\rangle \\notag\\\\\n&= \\frac{1}{\\rho}\\|z_{n+1}-z_n\\|^2 \n\\end{align}\n","start":32764,"end":33072},{"raw_tex":"and that\n\\begin{align*}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_n,z_n) &= H(y_{n+1}) - H(y_n) + \\langle z_n, B(y_{n+1}-y_n) \\rangle \\\\\n&\\qquad + \\frac{\\rho}{2}(\\|\\mathbf{Ax}_{n+1}+By_{n+1}-b\\|^2 - \\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2).  \n\\end{align*}\nWriting $\\mathbf{Ax}_{n+1}+By_{n}-b =(\\mathbf{Ax}_{n+1}+By_{n+1}-b) -(By_{n+1}- By_n)$, we obtain that\n\\begin{align*}\n&\\|\\mathbf{Ax}_{n+1}+By_{n+1}-b\\|^2 - \\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2 \\\\\n&= -\\|B(y_{n+1}-y_n)\\|^2 +2\\langle  \\mathbf{Ax}_{n+1}+By_{n+1}-b, B(y_{n+1}-y_n)\\rangle \\\\\n&= -\\|B(y_{n+1}-y_n)\\|^2 +\\frac{2}{\\rho}\\langle  z_{n+1}-z_{n}, B(y_{n+1}-y_n)\\rangle,\n\\end{align*}\nand so\n\\begin{align*}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_n,z_n) = H(y_{n+1}) - H(y_n) +\\langle z_{n+1}, B(y_{n+1}-y_n) \\rangle -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2.\n\\end{align*}\nIn view of \\eqref{eq:B'z},\n\\begin{align*}\n\\langle z_{n+1}, B(y_{n+1}-y_n) \\rangle &=\\langle B^\\top z_{n+1}, y_{n+1}-y_n \\rangle \\\\\n&=-\\langle \\nabla H(y_{n+1}), y_{n+1}-y_n \\rangle -\\nu_n\\langle \\nabla \\psi(y_{n+1})- \\nabla \\psi(y_n), y_{n+1}-y_n \\rangle \\\\\n&\\leq -H(y_{n+1}) +H(y_n) +\\frac{\\ell_H}{2}\\|y_{n+1}-y_n\\|^2,\n\\end{align*}\nwhere the last inequality is from the $\\ell_H$-Lipschitz continuity of $\\nabla H$ and Lemma~\\ref{l:diff}\\ref{l:diff_descent}, and from the monotonicity of $\\nabla \\psi$. \nWe therefore obtain that\n\\begin{align}\\label{eq:yy}\n\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_n,z_n) \\leq \\frac{\\ell_H}{2}\\|y_{n+1}-y_n\\|^2 -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2.    \n\\end{align}\n","start":33072,"end":34711},{"raw_tex":"Next, by the update of $x_{i,n+1}$ in Step~\\ref{step:A2_2} of Algorithm~\\ref{algo:BPLA}, for all $i \\in \\{1,\\dots,m\\}$ and all $x_i \\in \\mathbb{R}^{d_i}$,\n\\begin{align}\\label{eq:update_step_x}\n&f_i(x_{i,n+1}) +\\langle \\nabla_i P(\\mathbf{x}_n) -g_{i,n}, x_{i,n+1} \\rangle +\\langle z_n, A_{i}x_{i,n+1} \\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,n+1}(x_{i,n+1}) +By_n -b\\|^2 +\\mu_n D_{\\phi_i}(x_{i,n+1},x_{i,n}) \\notag \\\\\n&\\leq  f_i(x_i) +\\langle \\nabla_i P(\\mathbf{x}_n) -g_{i,n}, x_i \\rangle +\\langle z_n, A_{i}x_i\\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,n+1}(x_i) +By_n -b\\|^2 +\\mu_n D_{\\phi_i}(x_i,x_{i,n}).\n\\end{align}\nLetting $x_i =x_{i,n}$, using Proposition~\\ref{p:Bregman}\\ref{p:Bregman_1}\\&\\ref{p:Bregman_3}, and summing up over $i \\in \\{1,\\dots,m\\}$, we derive that\n\\begin{align}\\label{eq:from x-update}\n&\\sum_{i=1}^m f_i(x_{i,n+1}) +\\langle \\nabla P(\\mathbf{x}_n) -g_n, \\mathbf{x}_{n+1} \\rangle +\\langle z_n, \\mathbf{Ax}_{n+1} \\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{x}_{n+1} +By_n -b\\|^2 +\\frac{\\mu\\alpha}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 \\notag\\\\\n&\\leq \\sum_{i=1}^m f_i(x_{i,n}) +\\langle \\nabla P(\\mathbf{x}_n) -g_n, \\mathbf{x}_{n} \\rangle +\\langle z_n, \\mathbf{Ax}_{n} \\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{x}_n +By_n -b\\|^2.\n\\end{align}\n","start":34711,"end":35990},{"raw_tex":"On the other hand,\n\\begin{align}\\label{eq:xx}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n},z_n) -\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_n,z_n) \\notag\\\\\n&= \\sum_{i=1}^m f_i(x_{i,n+1}) -\\sum_{i=1}^m f_i(x_{i,n}) +P(\\mathbf{x}_{n+1}) -P(\\mathbf{x}_{n}) -G(\\mathbf{x}_{n+1}) +G(\\mathbf{x}_{n}) \\notag\\\\ \n&\\quad +\\langle z_n, \\mathbf{Ax}_{n+1}-\\mathbf{Ax}_{n}\\rangle +\\frac{\\rho}{2}\\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2 -\\frac{\\rho}{2}\\|\\mathbf{Ax}_{n}+By_{n}-b\\|^2 \\notag\\\\\n&\\leq \\sum_{i=1}^m f_i(x_{i,n+1}) -\\sum_{i=1}^m f_i(x_{i,n}) +\\langle \\nabla P(\\mathbf{x}_n), \\mathbf{x}_{n+1}-\\mathbf{x}_n \\rangle + \\frac{\\ell_P}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_n\\|^2 - \\langle g_n, \\mathbf{x}_{n+1} -\\mathbf{x}_{n} \\rangle + \\frac{\\beta}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 \\notag\\\\\n&\\quad +\\langle z_n, \\mathbf{Ax}_{n+1}-\\mathbf{Ax}_{n}\\rangle  + \\frac{\\rho}{2}\\|\\mathbf{Ax}_{n+1}+By_{n}-b\\|^2 - \\frac{\\rho}{2}\\|\\mathbf{Ax}_{n}+By_{n}-b\\|^2 \\notag\\\\\n&\\leq -\\frac{\\mu\\alpha -\\ell_P -\\beta}{2}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 =-\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2,\n\\end{align}\nwhere the first inequality is obtained by using Lemma~\\ref{l:diff}\\ref{l:diff_descent} on the $\\ell_P$-Lipschitz continuity of $\\nabla P$ and \\cite[Lemma~4.1]{BDL22} on the $\\beta$-weak convexity of $G$, while the last inequality follows from \\eqref{eq:from x-update}. Summing up three relations \\eqref{eq:zz}, \\eqref{eq:yy}, and \\eqref{eq:xx} yields\n","start":35990,"end":37422},{"raw_tex":"where the first inequality is obtained by using Lemma~\\ref{l:diff}\\ref{l:diff_descent} on the $\\ell_P$-Lipschitz continuity of $\\nabla P$ and \\cite[Lemma~4.1]{BDL22} on the $\\beta$-weak convexity of $G$, while the last inequality follows from \\eqref{eq:from x-update}. Summing up three relations \\eqref{eq:zz}, \\eqref{eq:yy}, and \\eqref{eq:xx} yields\n\\begin{align}\\label{eq:LL}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) - \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) \\notag\\\\\n&\\leq -\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\frac{\\ell_H}{2}\\|y_{n+1}-y_n\\|^2 -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2 +\\frac{1}{\\rho}\\|z_{n+1}-z_n\\|^2.\n\\end{align}\n","start":37071,"end":37738},{"raw_tex":"Applying Cauchy--Schwarz inequality to two vectors $u =(\\sqrt{\\ell_H+\\nu\\ell_{\\psi}},\\sqrt{\\nu\\ell_{\\psi}})$ and $v =(\\sqrt{\\ell_H+\\nu\\ell_{\\psi}}\\|y_{n+1} -y_n\\|,\\sqrt{\\nu\\ell_{\\psi}}\\|y_n -y_{n-1}\\|)$, we obtain that\n\\begin{align*}\n\\|z_{n+1} -z_n\\|^2 \\leq \\frac{\\ell_H +2\\nu\\ell_{\\psi}}{\\lambda}((\\ell_H +\\nu\\ell_{\\psi})\\|y_{n+1} -y_n\\|^2 +\\nu\\ell_{\\psi}\\|y_n -y_{n-1}\\|^2).    \n\\end{align*}\nBy combining this with \\eqref{eq:LL} and using Lemma~\\ref{l:norm}\\ref{l:norm_Mx},\n\\begin{align*}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) - \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) \\notag\\\\\n&\\leq -\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\left(\\frac{(\\ell_H+2\\nu\\ell_{\\psi})(\\ell_H+\\nu\\ell_{\\psi})}{\\lambda\\rho} +\\frac{\\ell_H}{2}\\right)\\|y_{n+1}-y_n\\|^2 \\\\ \n&\\qquad -\\frac{\\rho}{2}\\|B(y_{n+1}-y_n)\\|^2 +\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2 \\\\\n&\\leq -\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 -\\left(\\frac{\\lambda\\rho}{2} -\\frac{(\\ell_H+2\\nu\\ell_{\\psi})(\\ell_H+\\nu\\ell_{\\psi})}{\\lambda\\rho} -\\frac{\\ell_H}{2}\\right)\\|y_{n+1}-y_n\\|^2 \\\\\n&\\qquad +\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2,\n\\end{align*}\nor equivalently,\n\\begin{align*}\n&\\mathcal{L}_\\rho(\\mathbf{x}_{n+1},y_{n+1},z_{n+1}) +\\frac{(\\ell_H +2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_{n+1}-y_{n}\\|^2 + \\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 +\\delta_y\\|y_{n+1}-y_n\\|^2 \\\\\n&\\leq \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) +\\frac{(\\ell_H +2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2,\n\\end{align*}\nwhich proves \\eqref{eq:decrease}. Here, we note that $\\delta_{\\mathbf{x}} =\\frac{\\mu\\alpha -\\ell_P -\\beta}{2} > 0$ since $\\mu > \\frac{\\ell_P+\\beta}{\\alpha}$ and that $\\delta_y =\\frac{\\lambda\\rho}{2} -\\frac{(\\ell_H+2\\nu\\ell_{\\psi})^2}{\\lambda\\rho} -\\frac{\\ell_H}{2} >0$ since $\\rho > \\frac{\\ell_H +\\sqrt{\\ell_H^2 +8(\\ell_H +2\\nu\\ell_{\\psi})^2}}{2\\lambda}$.\n","start":38731,"end":40697},{"raw_tex":"Combining this with the fact that\n\\begin{align*}\n\\langle z_n, \\mathbf{Ax}_n+By_n-b \\rangle +\\frac{\\rho}{2}\\|\\mathbf{Ax}_n+By_n-b\\|^2 =\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} +\\mathbf{Ax}_n+By_n-b \\right\\|^2 -\\frac{1}{2\\rho}\\|z_n\\|^2,\n\\end{align*}\nwe deduce that\n\\begin{align*}\nL_n &=\\mathcal{L}_\\rho(\\mathbf{x}_n,y_n,z_n) + c\\|y_n-y_{n-1}\\|^2 \\\\\n&\\geq \\sum_{i=1}^m f_i(x_{i,n}) +H(y_n) +P(\\mathbf{x}_n) -G(\\mathbf{x}_n) \\\\ \n&\\quad +\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} + \\mathbf{Ax}_n+By_n-b \\right\\|^2 -\\frac{1}{2\\lambda\\rho}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right)\\|\\nabla H(y_n)\\|^2 +\\left(c -\\frac{(\\omega +\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{2\\lambda\\rho}\\right)\\|y_n-y_{n-1}\\|^2 \\\\\n&= \\left(\\sum_{i=1}^m f_i(x_{i,n}) +P(\\mathbf{x}_n) -G(\\mathbf{x}_n)\\right) +\\left(H(y_n) -\\frac{1}{2\\ell_H}\\|\\nabla H(y_n)\\|^2\\right) +\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} + \\mathbf{Ax}_n+By_n-b \\right\\|^2 \\\\ \n&\\quad +\\left(\\frac{1}{2\\ell_H} -\\frac{1}{2\\lambda\\rho}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right)\\right)\\|\\nabla H(y_n)\\|^2 +\\frac{\\nu\\ell_{\\psi}}{2\\lambda\\rho}(2\\ell_H +3\\nu\\ell_{\\psi} -\\omega)\\|y_n-y_{n-1}\\|^2.\n\\end{align*}\nNow, we note that\n\\begin{align*}\n\\rho > \\frac{\\ell_H +\\sqrt{\\ell_H^2 +8(\\ell_H +2\\nu\\ell_{\\psi})^2}}{2\\lambda} \\geq \\frac{\\ell_H +\\sqrt{\\ell_H^2 +8\\ell_H^2}}{2\\lambda} =\\frac{2\\ell_H}{\\lambda},    \n\\end{align*}\nand so $\\frac{1}{2\\ell_H} >\\frac{1}{\\lambda\\rho}$. Choosing $\\omega =1$ if $\\nu\\ell_{\\psi} =0$, and $\\omega =\\nu\\ell_{\\psi}$ if $\\nu\\ell_{\\psi} >0$, it holds that\n\\begin{align*}\n&\\theta :=\\frac{1}{2\\ell_H} -\\frac{1}{2\\lambda\\rho}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right) \\geq \\frac{1}{2\\ell_H} -\\frac{1}{2\\lambda\\rho}(1 +1) >0 \\text{~~and~~} \\\\\n&\\frac{\\nu\\ell_{\\psi}}{2\\lambda\\rho}(2\\ell_H +3\\nu\\ell_{\\psi} -\\omega) \\geq \\frac{\\nu\\ell_{\\psi}}{2\\lambda\\rho}(3\\nu\\ell_{\\psi} -\\omega) =\\frac{\\nu^2\\ell_{\\psi}^2}{\\lambda\\rho}. \n\\end{align*}\nThis together with \\ref{t:cvg_decrease} yields\n\\begin{align}\\label{eq:L_n_bounded}\nL_1 \\geq L_n &\\geq \\left(\\sum_{i=1}^m f_i(x_{i,n}) +P(\\mathbf{x}_n) -G(\\mathbf{x}_n)\\right) +\\left(H(y_n) -\\frac{1}{2\\ell_H}\\|\\nabla H(y_n)\\|^2\\right) +\\frac{\\rho}{2}\\left\\| \\frac{z_n}{\\rho} + \\mathbf{Ax}_n+By_n-b \\right\\|^2 \\notag \\\\ \n&\\quad +\\theta\\|\\nabla H(y_n)\\|^2 +\\frac{\\nu^2\\ell_{\\psi}^2}{\\lambda\\rho}\\|y_n-y_{n-1}\\|^2,\n\\end{align}\n","start":41724,"end":44030},{"raw_tex":"in which the first term on the right hand side is bounded below since $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, the second term is bounded below due to {the assumption on $H$ and Lemma~\\ref{l:diff}\\ref{l:diff_bounded}}, while the remaining terms are nonnegative due to their positive coefficients. Therefore, all the terms are bounded. Again, as $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, the boundedness of the first term implies that $(\\mathbf{x}_{n})_{n\\in \\mathbb{N}}$ is bounded. Since the last two terms are bounded, it follows from \\eqref{eq:uppper_zn} that $(z_n)_{n\\in \\mathbb{N}}$ is bounded. Since $z_{n+1}=z_n+\\rho(\\mathbf{Ax}_{n+1}+By_{n+1}-b)$ and $B$ has full column rank, we also obtain that $(y_n)_{n\\in \\mathbb{N}}$ is bounded. Consequently, the sequence $(\\mathbf{x}_n,y_n,z_n)_{n\\in \\mathbb{N}}$ is bounded.\n\n","start":44030,"end":44904},{"raw_tex":"On the other hand, since all the terms on the right hand side of \\eqref{eq:L_n_bounded} are bounded below, so is the sequence $(L_n)_{n\\in\\mathbb{N}}$. Combining with the nonincreasing property of $(L_n)_{n\\in\\mathbb{N}}$ due to \\ref{t:cvg_decrease}, it follows that $(L_n)_{n\\in\\mathbb{N}}$ is convergent. By rearranging \\eqref{eq:decrease} and performing telescoping, we then have that\n\\begin{align*}\n\\sum_{n=1}^{+\\infty}\\delta_y \\|y_{n+1}-y_n\\|^2 + \\delta_{\\mathbf{x}}\\sum_{n=1}^{+\\infty}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\|^2 \\leq \\sum_{n=1}^{+\\infty}(L_n - L_{n+1}) = L_1 - \\lim_{n\\to+\\infty} L_n < +\\infty,\n\\end{align*}\nwhich implies that $\\sum_{n=1}^{+\\infty}\\|\\mathbf{x}_{n+1}-\\mathbf{x}_n\\|^2 < +\\infty$, $\\sum_{n=1}^{+\\infty}\\|y_{n+1}-y_n\\|^2 <+\\infty$, and by \\eqref{eq:zy}, $\\sum_{n=1}^{+\\infty}\\|z_{n+1}-z_n\\|^2 < +\\infty$. As a result, $\\mathbf{x}_{n+1}-\\mathbf{x}_{n} \\to 0$, $y_{n+1}-y_n \\to 0$, $z_{n+1}-z_n \\to 0$, and $\\mathbf{Ax}_n +By_n =(z_n -z_{n-1})/\\rho +b \\to b$ as $n \\to +\\infty$. It then follows from the definition of $L_n$ that the sequence $(\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}))_{n\\in\\mathbb{N}}$ is also convergent with $\\lim_{n\\to+\\infty} \\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) =\\lim_{n\\to+\\infty}L_n$.\n","start":44904,"end":46155},{"raw_tex":"Let $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ be a cluster point of the sequence  $(\\mathbf{x}_n,y_n,z_n)_{n\\in \\mathbb{N}}$. Then there exists a subsequence $(\\mathbf{x}_{k_n},y_{k_n},z_{k_n})_{n\\in \\mathbb{N}}$ that converges to $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$. In view of \\eqref{eq:update_step_x}, letting $n =k_n$ and using Proposition~\\ref{p:Bregman}\\ref{p:Bregman_3}\\&\\ref{p:Bregman_4}, we obtain that, for all $i \\in \\{1,\\dots,m\\}$ and all $x_i\\in \\mathbb{R}^{d_i}$,\n\\begin{align}\\label{eq:kn+}\n&f_i(x_{i,k_n+1}) +\\langle \\nabla_i P(\\mathbf{x}_{k_n})- g_{i,k_n}, x_{i,k_n+1} -x_i \\rangle + \\langle z_{k_n}, A_i x_{i,k_n+1} - A_ix_i\\rangle +\\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,k_n+1}(x_{i,k_n+1}) +By_{k_n} -b \\|^2 \\notag \\\\\n&\\quad  - \\frac{\\rho}{2}\\|\\mathbf{A}\\mathbf{u}_{i,k_n+1}(x_{i}) +By_{k_n} -b \\|^2 +\\frac{\\mu\\alpha}{2}\\|x_{i,k_n+1}-x_{i,k_n}\\|^2 - \\frac{\\nu\\ell_{\\phi}}{2}\\|x_{i}-x_{i,k_n}\\|^2 \\leq f_i(x_{i}).\n\\end{align}\nSince $\\nabla P$ is continuous, $\\nabla P(\\mathbf{x}_{k_n}) \\to \\nabla P(\\overline{\\mathbf{x}})$ as $n\\to +\\infty$. As $G +\\frac{\\beta}{2}\\|\\cdot\\|^2$ is a continuous convex function, it follows from \\cite[Example~9.14]{Rockafellar1998} that $G$ is locally Lipschitz continuous. Since $\\mathbf{x}_{k_n} \\to \\overline{\\mathbf{x}}$ as $n\\to +\\infty$, using Lemma~\\ref{l:upsemicont} and passing to a subsequence if necessary, we can and do assume that $g_{k_n}\\to \\overline{g} \\in \\partial_L G (\\overline{\\mathbf{x}})$ as $n\\to +\\infty$. Now, for each $i \\in \\{1,\\dots,m\\}$, letting $x_i =\\overline{x}_i$ and $n\\to +\\infty$ in \\eqref{eq:kn+} with noting that $x_{i,k_n+1} \\to \\overline{x}_i$, $y_{k_n} \\to \\overline{y}$, and $z_{k_n} \\to \\overline{z}$, we have that $\\limsup_{n \\to +\\infty} f_i(x_{i,k_n+1}) \\leq f_i(\\overline{x}_i)$, which together with the lower semicontinuity of $f_i$ yields\n\\begin{align*}\n\\lim_{n \\to +\\infty} f_i(x_{i,k_n+1}) = f_i(\\overline{x}_i).\n\\end{align*}\nAs $H$, $P$, and $G$ are continuous, it follows that $\\lim_{n \\to +\\infty}\\mathcal{L}_\\rho(\\mathbf{x}_{k_n +1 },y_{k_n+1},z_{k_n +1}) = \\mathcal{L}_\\rho(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$. Since the sequence $(\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}))_{n\\in\\mathbb{N}}$ is convergent, we deduce that\n\\begin{align*}\n\\lim_{n \\to +\\infty}\\mathcal{L}_\\rho(\\mathbf{x}_{n },y_{n},z_{n}) =\\mathcal{L}_\\rho(\\overline{\\mathbf{x}},\\overline{y},\\overline{z}).    \n\\end{align*}\n","start":46156,"end":48595},{"raw_tex":"We next show that $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ is a stationary point of $\\mathcal{L}_\\rho$. In \\eqref{eq:z-update} and \\eqref{eq:B'z} with $n$ replaced by $k_n-1$, letting $n\\to +\\infty$ and using the coninuity of $\\nabla H$ and the Lipschitz continuity of $\\nabla \\psi$, we obtain that $0 =\\mathbf{A\\overline{x}} + B\\overline{y} -b$ and $0 =\\nabla H(\\overline{y}) + B^\\top \\overline{z}$. In turn, letting $n \\to +\\infty$ in \\eqref{eq:kn+}, we have for all $i \\in \\{1,\\dots,m\\}$ and all $x_i\\in \\mathbb{R}^{d_i}$ that\n\\begin{align*}\n&f_i(\\overline{x}_i) +\\langle \\nabla_i P(\\overline{\\mathbf{x}})-\\overline{g}_i, \\overline{x}_i-x_i \\rangle + \\langle \\overline{z}, A_i\\overline{x}_i-A_ix_i \\rangle\\\\\n&\\quad +\\frac{\\rho}{2}\\|A_1\\overline{x}_1 +\\dots +A_{i-1}\\overline{x}_{i-1} +A_i\\overline{x}_i +A_{i+1}\\overline{x}_{i+1}+\\dots+A_m \\overline{x}_{m} +B\\overline{y} -b \\|^2 \\\\\n&\\quad -\\frac{\\rho}{2}\\|A_1\\overline{x}_1 +\\dots +A_{i-1}\\overline{x}_{i-1} +A_ix_i +A_{i+1}\\overline{x}_{i+1}+\\dots+A_m \\overline{x}_{m} +B\\overline{y} -b \\|^2 -\\frac{\\nu\\ell_{ \\phi}}{2}\\|x_i-\\overline{x}_i\\|^2 \\leq f_i(x_i),\n\\end{align*}\nwhich can be written as\n\\begin{align*}\nh_i(\\overline{x}_i) \\leq h_i(x_i),    \n\\end{align*}\nwhere $h_i(x_i) =f_i(x_i) +\\langle \\nabla_i P(\\overline{\\mathbf{x}}) -\\overline{g}_i, x_i \\rangle + \\langle \\overline{z}, A_ix_i \\rangle +\\frac{\\rho}{2}\\|A_1\\overline{x}_1 +\\dots +A_{i-1}\\overline{x}_{i-1} +A_ix_i +A_{i+1}\\overline{x}_{i+1}+\\dots+A_m \\overline{x}_{m} +B\\overline{y} -b \\|^2 +\\frac{\\nu\\ell_{\\phi}}{2}\\|x_i-\\overline{x}_i\\|^2$. It follows that, for all $i \\in \\{1,\\dots,m\\}$,\n\\begin{align*}\n0 &\\in \\partial_L h_i(\\overline{x}_i) =\\partial_L (f_i)(\\overline{x}_i) +\\nabla_i P(\\overline{\\mathbf{x}})- \\overline{g}_i +A_i^\\top \\overline{z} +\\rho A_i^\\top (\\mathbf{A}\\overline{\\mathbf{x}}+B\\overline{y}-b).\n\\end{align*}\nSince $\\mathbf{A}\\overline{\\mathbf{x}}+B\\overline{y}-b =0$, we obtain that\n\\begin{align*}\n0 \\in \\partial_L (f_1)(\\overline{x}_1)\\times \\cdots \\times \\partial_L (f_m)(\\overline{x}_m) +\\nabla P(\\overline{\\mathbf{x}}) -\\overline{g} +\\mathbf{A}^\\top \\overline{z},\n\\end{align*}\nwhich completes the proof due to Lemma~\\ref{l:cal}\\ref{l:cal_sum}\\&\\ref{l:cal_sep} and the strict differentiability of $G$.\n","start":48596,"end":50852}]},{"id":"S039","semantic_type":"definition","kind":"definition","labels":["t:cvg","t:cvg_decrease","t:cvg_seq"],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"离散势与下降常数","source_units":[{"raw_tex":"Suppose that Assumption~\\ref{a:standing} holds. Let $(\\mathbf{x}_n,y_n,z_n)_{n\\in \\mathbb{N}}$ be the sequence generated by Algorithm~\\ref{algo:BPLA} and, for each $n\\in \\mathbb{N}^*$, define\n\\begin{align*}\nL_n =\\mathcal{L}_\\rho(\\mathbf{x}_{n},y_{n},z_{n}) +c\\|y_n-y_{n-1}\\|^2, \\text{~where~} c=\\frac{(\\ell_H+2\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda\\rho}.     \n\\end{align*}\n","start":30234,"end":30607},{"raw_tex":"where $\\delta_{\\mathbf{x}} =\\frac{\\mu\\alpha -\\ell_P -\\beta}{2} > 0$ and $\\delta_y =\\frac{\\lambda\\rho}{2} -\\frac{(\\ell_H+2\\nu\\ell_{\\psi})^2}{\\lambda\\rho} -\\frac{\\ell_H}{2} >0$.\n","start":30855,"end":31031}]},{"id":"S040","semantic_type":"assumption","kind":"assumption","labels":["t:cvg","t:cvg_decrease","t:cvg_seq"],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"有界性分支条件","source_units":[{"raw_tex":"\\item\\label{t:cvg_seq}\nSuppose that $z_0\\in \\Ima(B)$, that $\\sum_{i=1}^m f_i(x_i)+P(\\mathbf{x})-G(\\mathbf{x})$ is coercive, and that $H$ is bounded below. \n","start":31032,"end":31188}]},{"id":"S041","semantic_type":"assumption","kind":"assumption","labels":["t:cvg","t:cvg_decrease","t:cvg_seq"],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"驻点分支附加条件","source_units":[{"raw_tex":"and that if $G$ is strictly differentiable and, for each $i \\in \\{1,\\dots,m\\}$, $\\nabla \\phi_i$ is $\\ell_{\\phi}$-Lipschitz continuous, then $(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ is a stationary point of $\\mathcal{L}_\\rho$.\n","start":31868,"end":32105}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":["eq:B'z","eq:z-update"],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"乘子更新与转置恒等式","source_units":[{"raw_tex":"\\ref{t:cvg_decrease}: Let $n\\in \\mathbb{N}$. The update of $z_{n+1}$ yields\n\\begin{align}\\label{eq:z-update}\nz_{n+1} -z_n =\\rho(\\mathbf{Ax}_{n+1}+By_{n+1}-b).   \n\\end{align}\nBy the optimality condition for the update of $y_{n+1}$ in Step~\\ref{step:A2_2} of Algorithm~\\ref{algo:BPLA}, \n\\begin{align*}\n0 &= \\nabla H(y_{n+1}) + B^\\top z_n +\\rho B^\\top(\\mathbf{Ax}_{n+1} +By_{n+1} -b) +\\nu_n(\\nabla \\psi(y_{n+1}) - \\nabla \\psi(y_n)),\n\\end{align*}\nwhich combined with \\eqref{eq:z-update} yields\n\\begin{align}\\label{eq:B'z}\nB^\\top z_{n+1} =-\\nabla H(y_{n+1}) -\\nu_n(\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)).  \n\\end{align}\n","start":32149,"end":32764}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["eq:BTZy","eq:zImB","eq:zy"],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"乘子增量受y控制","source_units":[{"raw_tex":"Now, since $\\Ima(\\mathbf{A}) \\cup \\{b\\} \\subseteq \\Ima(B)$, it follows from \\eqref{eq:z-update} that\n\\begin{align}\\label{eq:zImB}\nz_{n+1}-z_n \\in \\Ima(B)    \n\\end{align}\nwhich, by Lemma~\\ref{l:norm}\\ref{l:norm_M'x}, yields \n\\begin{align*}\n\\|z_{n+1} -z_n\\| \\leq \\frac{1}{\\sqrt{\\lambda}}\\|B^\\top z_{n+1} -B^\\top z_n\\|.\n\\end{align*}\nLet $n\\in \\mathbb{N}^*$. We derive from \\eqref{eq:B'z} and the Lipschitz continuity of $\\nabla H$ and $\\nabla \\psi$ that\n\\begin{align}\\label{eq:BTZy}\n\\|B^\\top z_{n+1} -B^\\top z_n\\| &\\leq \\|\\nabla H(y_{n+1}) -\\nabla H(y_n)\\| +\\nu_n\\|\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)\\| +\\nu_{n-1}\\|\\nabla \\psi(y_n) - \\nabla \\psi(y_{n-1})\\| \\notag \\\\ \n&\\leq (\\ell_H +\\nu\\ell_{\\psi})\\|y_{n+1} -y_n\\| +\\nu\\ell_{\\psi}\\|y_n -y_{n-1}\\|,\n\\end{align}\nand thus, \n\\begin{align}\\label{eq:zy}\n\\|z_{n+1} -z_n\\| \\leq \\frac{1}{\\sqrt{\\lambda}}\\|B^\\top z_{n+1} -B^\\top z_n\\| \\leq \\frac{1}{\\sqrt{\\lambda}}((\\ell_H +\\nu\\ell_{\\psi})\\|y_{n+1} -y_n\\| +\\nu\\ell_{\\psi}\\|y_n -y_{n-1}\\|).\n\\end{align}\n","start":37739,"end":38731}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":["eq:uppper_zn"],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"乘子绝对量估计","source_units":[{"raw_tex":"\\ref{t:cvg_seq}: Let $n\\in \\mathbb{N}^*$. As $z_0 \\in \\Ima(B)$, we have from \\eqref{eq:zImB} that $z_n \\in \\Ima (B)$. Using Lemma~\\ref{l:norm}\\ref{l:norm_M'x}, \\eqref{eq:B'z}, and the Lipschitz continuity of $\\psi$ yields\n\\begin{align*}\n\\sqrt{\\lambda}\\|z_n\\|\\leq \\|B^\\top z_n\\| \\leq\\|\\nabla H(y_n)\\|+ \\nu_n\\|\\nabla \\psi(y_{n}) -\\nabla \\psi(y_{n-1})\\| \n\\leq \\|\\nabla H(y_n)\\|+ \\nu \\ell_{\\psi}\\|y_{n}-y_{n-1}\\|.\n\\end{align*}\nFor $\\omega\\in \\mathbb{R}_{++}$ to be chosen later, invoking Cauchy--Schwarz inequality with two vectors $u =(\\sqrt{\\omega},\\sqrt{\\nu\\ell_{\\psi}})$ and $v =(\\frac{1}{\\sqrt{\\omega}}\\|\\nabla H(y_n)\\|,\\sqrt{\\nu\\ell_{\\psi}}\\|y_n-y_{n-1}\\|)$, we have that\n\\begin{align}\\label{eq:uppper_zn}\n\\|z_n\\|^2 &\\leq \\frac{\\omega +\\nu\\ell_{\\psi}}{\\lambda} \\left(\\frac{1}{\\omega}\\|\\nabla H(y_n)\\|^2 +\\nu\\ell_{\\psi}\\|y_n-y_{n-1}\\|^2\\right) \\notag \\\\\n&= \\frac{1}{\\lambda}\\left(1 +\\frac{\\nu\\ell_{\\psi}}{\\omega}\\right)\\|\\nabla H(y_n)\\|^2 +\\frac{(\\omega +\\nu\\ell_{\\psi})\\nu\\ell_{\\psi}}{\\lambda}\\|y_n-y_{n-1}\\|^2.\n\\end{align}\n","start":40698,"end":41724}]},{"id":"S057","semantic_type":"proof","kind":"proof","labels":["eq:auxilary_decrease","eq:dxi","eq:dy","eq:dzdt"],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"主定理完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nFor each $n\\in \\mathbb{N}$, set $w_n=(\\mathbf{x}_{n+1},y_{n+1},z_{n+1},y_n)$ and $\\Delta_n =\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\| +\\|y_{n+2}-y_{n+1}\\|$. Let $n \\in \\mathbb{N}^*$. According to Theorem~\\ref{t:cvg}, we have that\n\\begin{align}\\label{eq:auxilary_decrease}\n\\mathcal{F}(w_{n+1}) +\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\|^2 + \\delta_y \\|y_{n+2}-y_{n+1}\\|^2 \\leq \\mathcal{F}(w_{n}),\n\\end{align}\nthat $w_{n+1} - w_n \\to 0$ as $n\\to +\\infty$, that $(w_n)_{n\\in\\mathbb{N}}$ is bounded, that any of its cluster point $\\overline{w}=(\\overline{\\mathbf{x}},\\overline{y},\\overline{z},\\overline{y})$ is a stationary point of \\eqref{eq:prob}, and also that $\\mathcal{F}(w_{n}) \\to \\mathcal{F}(\\overline{w}) = \\mathcal{L}_{\\rho}(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ as $n\\to +\\infty$. Notice that\n\\begin{align*}\n\\Delta_n^2 \\leq \\left(\\frac{1}{\\delta_{\\mathbf{x}}} +\\frac{1}{\\delta_y}\\right) \\left(\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\|^2 + \\delta_y \\|y_{n+2}-y_{n+1}\\|^2\\right),    \n\\end{align*}\nwhich together with \\eqref{eq:auxilary_decrease} yields\n\\begin{align*}\n\\mathcal{F}(w_{n+1}) +C_1\\Delta_n^2 \\leq \\mathcal{F}(w_{n}), \\text{~~where~} C_1 =\\frac{\\delta_{\\mathbf{x}}\\delta_y}{\\delta_{\\mathbf{x}}+\\delta_y}.\n\\end{align*}\n\nNext, we have that\n\\begin{align*}\n\\partial_L ^\\mathbf{x} \\mathcal{F}(w_n) = \\partial_L \\left( \\sum_{i=1}^{m} f_i\\right)(\\mathbf{x}_{n+1}) + \\nabla P(\\mathbf{x}_{n+1}) - \\nabla G(\\mathbf{x}_{n+1}) + \\mathbf{A}^\\top z_{n+1} + \\rho \\mathbf{A}^\\top(\\mathbf{Ax}_{n+1} + By_{n+1} -b)    \n\\end{align*}\nand, for all $i \\in \\{1,\\dots,m\\}$,\n\\begin{align*}\n\\partial_L ^{x_i} \\mathcal{F}(w_n) = \\partial_L f_i(x_{i,n+1})+\\nabla_i P(\\mathbf{x}_{n+1}) - \\nabla_i G(\\mathbf{x}_{n+1}) + A_i^\\top z_{n+1} + \\rho A_i^\\top(\\mathbf{Ax}_{n+1} + By_{n+1} -b).    \n\\end{align*}\nBy revoking the optimality condition for the update of $x_{i,n+1}$ in Step~\\ref{step:A2_2} of Algorithm~\\ref{algo:BPLA},\n\\begin{align*}\n0 &\\in \\partial_L f_i(x_{i,n+1}) +\\nabla_i P(\\mathbf{x}_{n}) -  \\nabla_i G(\\mathbf{x}_n) + A_i^\\top z_n   +\\rho A_i^\\top(\\mathbf{A}\\mathbf{u}_{i,n+1}(x_{i,n+1} ) +By_n -b ) \\\\\n&\\quad +\\mu_n(\\nabla \\phi_i (x_{i,n+1})- \\nabla \\phi_i (x_{i,n})),    \n\\end{align*}\nwhich combined with the above equality implies that\n\\begin{align*}\n&(\\nabla_i P(\\mathbf{x}_{n+1}) -\\nabla_i P(\\mathbf{x}_{n})) -(\\nabla_i G(\\mathbf{x}_{n+1}) -\\nabla_i G(\\mathbf{x}_n)) +A_i^\\top(z_{n+1}-z_n) -\\mu_n(\\nabla \\phi_i (x_{i,n+1}) -\\nabla \\phi_i (x_{i,n})) \\\\\n&\\quad +\\rho A_i^\\top (A_{i+1}x_{i+1,n+1} +\\dots +A_mx_{m,n+1} -A_{i+1}x_{i+1,n} -\\dots -A_mx_{m,n} +By_{n+1} -By_n) \\in \\partial_L ^{x_i} \\mathcal{F}(w_n).\n\\end{align*}\nTherefore,\n\\begin{align}\\label{eq:dxi}\n\\dist(0,\\partial_L ^{x_i} \\mathcal{F}(w_n)) \n&\\leq \\|\\nabla_i P(\\mathbf{x}_{n+1}) -\\nabla_i P(\\mathbf{x}_{n})\\| +\\|\\nabla_i G(\\mathbf{x}_{n+1}) -\\nabla_i G(\\mathbf{x}_n)\\| \\notag \\\\\n&\\quad +\\|A_i^\\top(z_{n+1}-z_n)\\| +\\mu_n\\|\\nabla \\phi_i (x_{i,n+1}) -\\nabla \\phi_i (x_{i,n})\\| \\notag \\\\\n&\\quad +\\rho \\|A_i^\\top (A_{i+1}x_{i+1,n+1} +\\dots +A_mx_{m,n+1} -A_{i+1}x_{i+1,n} -\\dots -A_mx_{m,n} +By_{n+1} -By_n\\| \\notag \\\\ \n&\\leq \\ell_P\\|\\mathbf{x}_{n+1} - \\mathbf{x}_{n}\\| +\\ell_G\\|\\mathbf{x}_{n} - \\mathbf{x}_{n+1}\\| +\\|A_i^\\top\\|\\|z_{n+1}-z_n\\| +\\mu_n \\ell_{\\phi}\\|x_{i,n+1}-x_{i,n}\\| \\notag \\\\\n&\\quad +\\rho\\sum_{j=i+1}^{m} \\|A_i^\\top\\| \\|A_j\\| \\|x_{j,n+1}-x_{j,n}\\| +\\rho\\|A_i^\\top\\| \\|B\\| \\|y_{n+1}-y_n\\| \\notag \\\\\n&\\leq  \\left(\\ell_P +\\ell_G +\\mu_n\\ell_{\\phi} +\\rho\\sum_{j=i+1}^{m} \\|A_i^\\top\\| \\|A_j\\|\\right)\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\| +\\|A_i^\\top\\|\\|z_{n+1}-z_n\\| \\notag \\\\\n&\\quad +\\rho\\|A_i^\\top\\| \\|B\\| \\|y_{n+1}-y_n\\|.\n\\end{align}\n\nWe now see that\n\\begin{align*}\n\\partial_L ^y \\mathcal{F}(w_n) =\\nabla H(y_{n+1}) + B^\\top z_{n+1} +\\rho B^\\top(\\mathbf{Ax}_{n+1} + By_{n+1} -b) +2c(y_{n+1}-y_{n})    \n\\end{align*}\nand from the optimality condition for the update of $y_{n+1}$ in Algorithm~\\ref{algo:BPLA} that \n\\begin{align*}\n0 &= \\nabla H(y_{n+1}) + B^\\top z_n +\\rho B^\\top(\\mathbf{Ax}_{n+1} +By_{n+1} -b) +\\nu_n(\\nabla \\psi(y_{n+1}) - \\nabla \\psi(y_n)).\n\\end{align*}\nThis leads to\n\\begin{align*}\n\\partial_L ^y \\mathcal{F}(w_n) &=B^{\\top}(z_{n+1}-z_n) -\\nu_n(\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)) + 2c(y_{n+1}-y_{n}),   \n\\end{align*}\nand so\n\\begin{align}\\label{eq:dy}\n\\dist(0,\\partial_L ^{y} \\mathcal{F}(w_n)) \n&\\leq \\|B^{\\top}(z_{n+1}-z_n)\\| +\\nu_n\\|\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)\\| + 2c\\|y_{n+1}-y_{n})\\| \\notag \\\\\n&\\leq \\|B^{\\top}(z_{n+1}-z_n)\\| +(\\nu\\ell_\\psi +2c)\\|y_{n+1}-y_{n})\\|.\n\\end{align}\n\nSince $\\partial_L ^z \\mathcal{F}(w_n) =\\mathbf{Ax}_{n+1} + By_{n+1} -b =\\frac{1}{\\rho}(z_{n+1} -z_n)$ and\n$\\partial_L ^t \\mathcal{F}(w_n) =-2c(y_{n+1}-y_{n})$, it holds that\n\\begin{align}\\label{eq:dzdt}\n\\dist(0,\\partial_L ^{z} \\mathcal{F}(w_n)) =\\frac{1}{\\rho}\\|z_{n+1}-z_n\\| \\text{~~and~~}\n\\dist(0,\\partial_L ^{t} \\mathcal{F}(w_n)) =2c\\|y_{n+1}-y_n\\|.    \n\\end{align}\nCombining \\eqref{eq:dxi}, \\eqref{eq:dy}, and \\eqref{eq:dzdt}, noting that $\\limsup_{n\\to +\\infty} \\mu_n =\\overline{\\mu} <+\\infty$, and using \\eqref{eq:BTZy} and \\eqref{eq:zy} in the proof of Theorem~\\ref{t:cvg}, we derive that there exists $n_0\\in \\mathbb{N}^*$ and $C_2 \\in \\mathbb{R}_{++}$ such that, for all $n\\geq n_0$, \n\n\\begin{align*}\n\\dist(0, \\partial_L \\mathcal{F}(w_n)) \n\\leq C_2 \\left(\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\| +\\|y_{n+1}-y_n\\| +\\|y_{n}-y_{n-1}\\|\\right) \n\\leq C_2(\\Delta_{n-1} +\\Delta_{n-2}).\n\\end{align*}\n\n\\ref{t:full_cvg}: We see that all the conditions in the \\emph{abstract convergence} framework \\cite[Theorem~5.1]{BDL22} are satisfied with $I =\\{1,2\\}$, $\\lambda_1=\\lambda_2=1/2$, $\\alpha_n \\equiv C_1$, $\\beta_n \\equiv 1/(2C_2)$, and $\\varepsilon_n \\equiv 0 $. By \\cite[Theorem~5.1(i)]{BDL22}, \n\\begin{align*}\n\\sum_{n=0}^{+\\infty} (\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\|+\\|y_{n+2}-y_{n+1}\\|) =\\sum_{n=0}^{+\\infty} \\Delta_n < +\\infty,\n\\end{align*}\nwhich implies that $\\sum_{n=0}^{+\\infty} \\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\| <+\\infty$ and $\\sum_{n=0}^{+\\infty} \\|y_{n+1}-y_{n}\\| < +\\infty$. Together with \\eqref{eq:zy}, we derive that $\\sum_{n=0}^{+\\infty} \\|z_{n+1}-z_n\\|<+\\infty$, and hence \n\\begin{align*}\n\\sum_{n=0}^{+\\infty} \\|(\\mathbf{x}_{n+1},y_{n+1},z_{n+1})-(\\mathbf{x}_{n},y_{n},z_{n})\\| < +\\infty,\n\\end{align*}\nwhich yields the convergence of $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ to $(\\mathbf{x}^{*},y^{*},z^{*})$. In view of Theorem~\\ref{t:cvg}\\ref{t:cvg_seq}, $(\\mathbf{x}^{*},y^{*},z^{*})$ is a stationary point of $\\mathcal{L}_\\rho$.\n\n\\ref{t:full_rate_finite}: This follows from the arguments in the proof of \\cite[Theorem~5.1]{BDL22} and \\cite[Theorem~2(i)]{Attouch2007}. \n\n\\ref{t:full_rate_linear}: According to \\cite[Theorem~5.1(iv)]{BDL22}, there exist $\\gamma_0\\in \\mathbb{R}_{++}$ and $\\zeta\\in \\left(0,1\\right)$ such that, for all $n\\in \\mathbb{N}^*$, \n\\begin{align*}\n\\|(\\mathbf{x}_{n},y_{n})-(\\mathbf{x}^{*},y^{*})\\|\\leq\\gamma_0 \\zeta^{\\frac{n}{2}} \\text{~~and~~}\n|\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*)| \\leq \\gamma_0\\zeta^n.\n\\end{align*}\nWe derive that, for all $n\\in \\mathbb{N}^*$, $\\|\\mathbf{x}_n-\\mathbf{x}^{*}\\|\\leq \\gamma_0\\zeta^{\\frac{n}{2}}$ and $\\|y_n-y^{*}\\|\\leq \\gamma_0\\zeta^{\\frac{n}{2}}$, which yields \n\\begin{align*}\n\\|y_n-y_{n-1}\\| \\leq \\|y_n-y^{*}\\|+\\|y_{n-1}-y^{*}\\| \\leq (1+\\zeta^{-\\frac{1}{2}})\\gamma_0\\zeta^{\\frac{n}{2}}.    \n\\end{align*}\nBy passing to the limit in \\eqref{eq:B'z}, $B^\\top z^{*}=-\\nabla H(y^{*})$. Following the same steps as in \\eqref{eq:BTZy} and \\eqref{eq:zy}, we obtain that, for all $n\\in \\mathbb{N}^*$,\n\\begin{align*}\n\\|z_n-z^{*}\\| &\\leq \\frac{\\ell_H}{\\sqrt{\\lambda}}\\|y_n-y^{*}\\| + \\frac{\\nu \\ell_{\\psi}}{\\sqrt{\\lambda}}\\|y_{n}-y_{n-1}\\|\\leq \\gamma_1\\zeta^{\\frac{n}{2}}, \\text{~~where~} \\gamma_1 :=\\frac{\\ell_H\\gamma_0+(1+\\zeta^{-\\frac{1}{2}})\\nu\\ell_{\\psi}\\gamma_0}{\\sqrt{\\lambda}}.\n\\end{align*}\nConsequently,\n\\begin{align*}\n&\\|(\\mathbf{x}_{n},y_{n},z_{n})-(\\mathbf{x}^{*},y^{*},z^{*})\\| \\leq \\|\\mathbf{x}_{n}-\\mathbf{x}^{*}\\| + \\|y_{n}-y^{*}\\| + \\|z_{n}-z^{*}\\|\\leq (2\\gamma_0 +\\gamma_1)\\zeta^{\\frac{n}{2}}, \\\\\n&\\|\\mathbf{Ax}_n+By_n-b\\| =\\frac{1}{\\rho}\\|z_n-z_{n-1}\\| \\leq \\frac{1}{\\rho}(\\|z_n-z^{*}\\| +\\|z_{n-1}-z^{*}\\|) \\leq \\frac{(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho} \\zeta^{\\frac{n}{2}}, \\\\\n\\text{and~} &\\|z_n\\| \\leq \\|z_n-z^{*}\\| + \\|z^{*}\\| \\leq \\gamma_1 \\zeta^{\\frac{n}{2}} + \\|z^{*}\\|.\n\\end{align*}\n\nWe now deduce from the definition of $\\mathcal{F}$ that\n\\begin{align*}\n|\\mathcal{L}_{\\rho}(\\mathbf{x}_{n},y_{n},z_{n})- \\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})|&= \\left|\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*) - c\\|y_n-y_{n-1}\\|^2\\right|\\\\\n&\\leq |\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*)| + c\\|y_n-y_{n-1}\\|^2\\\\\n&\\leq \\gamma_2\\zeta^n, \\text{~~where~} \\gamma_2 :=\\gamma_0 + c(1+\\zeta^{-\\frac{1}{2}})^2\\gamma_0^2\n\\end{align*}\nand from the definition of $\\mathcal{L}_\\rho$ that \n\\begin{align*}\n|F(\\mathbf{x}_n,y_n)-F(\\mathbf{x}^{*},y^{*})| &= |\\mathcal{L}_{\\rho}(\\mathbf{x}_n,y_n,z_n)-\\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})-\\langle z_n, \\mathbf{Ax}_n + By_n-b\\rangle - \\frac{\\rho}{2}\\|\\mathbf{Ax}_n + By_n-b\\|^2| \\\\\n&\\leq |\\mathcal{L}_{\\rho}(\\mathbf{x}_n,y_n,z_n)-\\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})| + \\|z_n\\|\\|\\mathbf{Ax}_n + By_n-b\\| + \\frac{\\rho}{2}\\|\\mathbf{Ax}_n + By_n-b\\|^2  \\\\ \n&\\leq \\gamma_2\\zeta^n + \\left(\\gamma_1 \\zeta^{\\frac{n}{2}} + \\|z^{*}\\|\\right)\\frac{(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho} \\zeta^{\\frac{n}{2}}+ \\frac{(1+\\zeta^{-\\frac{1}{2}})^2\\gamma_1^2}{2\\rho}\\zeta^n \\\\\n&\\leq \\left(\\gamma_2 +\\frac{(3+\\zeta^{-\\frac{1}{2}})(1+\\zeta^{-\\frac{1}{2}})\\gamma_1^2}{2\\rho} +\\frac{\\|z^{*}\\|(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho}\\right)\\zeta^{\\frac{n}{2}},\n\\end{align*}\nwhere the last inequality follows from the fact that $\\zeta^n \\leq \\zeta^{\\frac{n}{2}}$ since $\\zeta \\in (0,1)$. By letting \n\\begin{align*}\n\\overline{\\gamma} =\\max \\left\\{ 2\\gamma_0+\\gamma_1, \\frac{(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho}, \\gamma_2 +\\frac{(3+\\zeta^{-\\frac{1}{2}})(1+\\zeta^{-\\frac{1}{2}})\\gamma_1^2}{2\\rho} +\\frac{\\|z^{*}\\|(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho} \\right\\} \n\\end{align*}\nand increasing it if necessary, we arrive at the conclusion.\n\n\\ref{t:full_rate_sublinear}: Following the arguments in \\cite[Theorem~5.1(iv)]{BDL22} and \\cite[Theorem~2(iii)]{Attouch2007}, we find $\\gamma_0 \\in \\mathbb{R}_{++}$ such that, for all $n\\in \\mathbb{N}^*$, \n\\begin{align*}\n\\|(\\mathbf{x}_{n},y_{n})-(\\mathbf{x}^{*},y^{*})\\|\\leq \\gamma_0 n^{-\\frac{1-\\kappa}{2\\kappa-1}} \\text{~~and~~}\n|\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*)| \\leq \\gamma_0 n^{-\\frac{2-2\\kappa}{2\\kappa-1}}.\n\\end{align*}\nSimilar to \\ref{t:full_rate_linear}, for all $n\\geq 2$, since $n-1\\geq \\frac{1}{2}n$ and $\\frac{1-\\kappa}{2\\kappa-1} >0$, we derive that \n\\begin{align*}\n&\\|(\\mathbf{x}_{n},y_{n},z_{n})-(\\mathbf{x}^{*},y^{*},z^{*})\\| \\leq (2\\gamma_0+\\gamma_1) n^{-\\frac{1-\\kappa}{2\\kappa-1}},\\\\\n&\\|\\mathbf{Ax}_n+By_n-b\\|\\leq \\frac{(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1}{\\rho} n^{-\\frac{1-\\kappa}{2\\kappa-1}},\\\\\n&|\\mathcal{L}_{\\rho}(\\mathbf{x}_{n},y_{n},z_{n})- \\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})|\\leq \\gamma_2 n^{-\\frac{2-2\\kappa}{2\\kappa-1}},\n\\end{align*}\nand that\n\\begin{align*}\n|F(\\mathbf{x}_n,y_n)-F(\\mathbf{x}^{*},y^{*})| &\\leq \\gamma_2 n^{-\\frac{2-2\\kappa}{2\\kappa-1}}+ \\left(\\gamma_1 n^{-\\frac{1-\\kappa}{2\\kappa-1}} + \\|z^{*}\\|\\right)\\frac{(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1}{\\rho} n^{-\\frac{1-\\kappa}{2\\kappa-1}} + \\frac{(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})^2\\gamma_1^2}{2\\rho} n^{-\\frac{2-2\\kappa}{2\\kappa-1}}\\\\\n&\\leq \\left(\\gamma_2 +\\frac{(3 +2^{\\frac{1-\\kappa}{2\\kappa-1}})(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1^2}{2\\rho} +\\frac{\\|z^{*}\\|(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1}{\\rho} \\right)n^{-\\frac{1-\\kappa}{2\\kappa-1}},\n\\end{align*}\nwhere $\\gamma_1 :=\\frac{\\ell_H\\gamma_0 +(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\nu\\ell_{\\psi}\\gamma_0}{\\sqrt{\\lambda}}$, $\\gamma_2 :=\\gamma_0 +c(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})^2\\gamma_0^2$, and the last inequality follows from the fact that $n^{-\\frac{2-2\\kappa}{2\\kappa-1}}<n^{-\\frac{1-\\kappa}{2\\kappa-1}}$ for $\\kappa\\in(\\frac{1}{2},1)$. By setting $\\overline{\\gamma}$ in the same way as in \\ref{t:full_rate_linear}, we obtain the conclusion.\n\\end{proof}","start":55931,"end":68289}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":["eq:auxilary_decrease"],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"扩展序列与继承性质","source_units":[{"raw_tex":"For each $n\\in \\mathbb{N}$, set $w_n=(\\mathbf{x}_{n+1},y_{n+1},z_{n+1},y_n)$ and $\\Delta_n =\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\| +\\|y_{n+2}-y_{n+1}\\|$. Let $n \\in \\mathbb{N}^*$. According to Theorem~\\ref{t:cvg}, we have that\n\\begin{align}\\label{eq:auxilary_decrease}\n\\mathcal{F}(w_{n+1}) +\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\|^2 + \\delta_y \\|y_{n+2}-y_{n+1}\\|^2 \\leq \\mathcal{F}(w_{n}),\n\\end{align}\nthat $w_{n+1} - w_n \\to 0$ as $n\\to +\\infty$, that $(w_n)_{n\\in\\mathbb{N}}$ is bounded, that any of its cluster point $\\overline{w}=(\\overline{\\mathbf{x}},\\overline{y},\\overline{z},\\overline{y})$ is a stationary point of \\eqref{eq:prob}, and also that $\\mathcal{F}(w_{n}) \\to \\mathcal{F}(\\overline{w}) = \\mathcal{L}_{\\rho}(\\overline{\\mathbf{x}},\\overline{y},\\overline{z})$ as $n\\to +\\infty$. Notice that\n","start":55945,"end":56768}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"标量步长下降条件","source_units":[{"raw_tex":"\\begin{align*}\n\\Delta_n^2 \\leq \\left(\\frac{1}{\\delta_{\\mathbf{x}}} +\\frac{1}{\\delta_y}\\right) \\left(\\delta_{\\mathbf{x}}\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\|^2 + \\delta_y \\|y_{n+2}-y_{n+1}\\|^2\\right),    \n\\end{align*}\nwhich together with \\eqref{eq:auxilary_decrease} yields\n\\begin{align*}\n\\mathcal{F}(w_{n+1}) +C_1\\Delta_n^2 \\leq \\mathcal{F}(w_{n}), \\text{~~where~} C_1 =\\frac{\\delta_{\\mathbf{x}}\\delta_y}{\\delta_{\\mathbf{x}}+\\delta_y}.\n\\end{align*}\n","start":56768,"end":57217}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"构造x块次梯度","source_units":[{"raw_tex":"Next, we have that\n\\begin{align*}\n\\partial_L ^\\mathbf{x} \\mathcal{F}(w_n) = \\partial_L \\left( \\sum_{i=1}^{m} f_i\\right)(\\mathbf{x}_{n+1}) + \\nabla P(\\mathbf{x}_{n+1}) - \\nabla G(\\mathbf{x}_{n+1}) + \\mathbf{A}^\\top z_{n+1} + \\rho \\mathbf{A}^\\top(\\mathbf{Ax}_{n+1} + By_{n+1} -b)    \n\\end{align*}\nand, for all $i \\in \\{1,\\dots,m\\}$,\n\\begin{align*}\n\\partial_L ^{x_i} \\mathcal{F}(w_n) = \\partial_L f_i(x_{i,n+1})+\\nabla_i P(\\mathbf{x}_{n+1}) - \\nabla_i G(\\mathbf{x}_{n+1}) + A_i^\\top z_{n+1} + \\rho A_i^\\top(\\mathbf{Ax}_{n+1} + By_{n+1} -b).    \n\\end{align*}\nBy revoking the optimality condition for the update of $x_{i,n+1}$ in Step~\\ref{step:A2_2} of Algorithm~\\ref{algo:BPLA},\n\\begin{align*}\n0 &\\in \\partial_L f_i(x_{i,n+1}) +\\nabla_i P(\\mathbf{x}_{n}) -  \\nabla_i G(\\mathbf{x}_n) + A_i^\\top z_n   +\\rho A_i^\\top(\\mathbf{A}\\mathbf{u}_{i,n+1}(x_{i,n+1} ) +By_n -b ) \\\\\n&\\quad +\\mu_n(\\nabla \\phi_i (x_{i,n+1})- \\nabla \\phi_i (x_{i,n})),    \n\\end{align*}\nwhich combined with the above equality implies that\n\\begin{align*}\n&(\\nabla_i P(\\mathbf{x}_{n+1}) -\\nabla_i P(\\mathbf{x}_{n})) -(\\nabla_i G(\\mathbf{x}_{n+1}) -\\nabla_i G(\\mathbf{x}_n)) +A_i^\\top(z_{n+1}-z_n) -\\mu_n(\\nabla \\phi_i (x_{i,n+1}) -\\nabla \\phi_i (x_{i,n})) \\\\\n&\\quad +\\rho A_i^\\top (A_{i+1}x_{i+1,n+1} +\\dots +A_mx_{m,n+1} -A_{i+1}x_{i+1,n} -\\dots -A_mx_{m,n} +By_{n+1} -By_n) \\in \\partial_L ^{x_i} \\mathcal{F}(w_n).\n\\end{align*}\n","start":57218,"end":58609}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":["eq:dxi"],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"x块次梯度误差界","source_units":[{"raw_tex":"Therefore,\n\\begin{align}\\label{eq:dxi}\n\\dist(0,\\partial_L ^{x_i} \\mathcal{F}(w_n)) \n&\\leq \\|\\nabla_i P(\\mathbf{x}_{n+1}) -\\nabla_i P(\\mathbf{x}_{n})\\| +\\|\\nabla_i G(\\mathbf{x}_{n+1}) -\\nabla_i G(\\mathbf{x}_n)\\| \\notag \\\\\n&\\quad +\\|A_i^\\top(z_{n+1}-z_n)\\| +\\mu_n\\|\\nabla \\phi_i (x_{i,n+1}) -\\nabla \\phi_i (x_{i,n})\\| \\notag \\\\\n&\\quad +\\rho \\|A_i^\\top (A_{i+1}x_{i+1,n+1} +\\dots +A_mx_{m,n+1} -A_{i+1}x_{i+1,n} -\\dots -A_mx_{m,n} +By_{n+1} -By_n\\| \\notag \\\\ \n&\\leq \\ell_P\\|\\mathbf{x}_{n+1} - \\mathbf{x}_{n}\\| +\\ell_G\\|\\mathbf{x}_{n} - \\mathbf{x}_{n+1}\\| +\\|A_i^\\top\\|\\|z_{n+1}-z_n\\| +\\mu_n \\ell_{\\phi}\\|x_{i,n+1}-x_{i,n}\\| \\notag \\\\\n&\\quad +\\rho\\sum_{j=i+1}^{m} \\|A_i^\\top\\| \\|A_j\\| \\|x_{j,n+1}-x_{j,n}\\| +\\rho\\|A_i^\\top\\| \\|B\\| \\|y_{n+1}-y_n\\| \\notag \\\\\n&\\leq  \\left(\\ell_P +\\ell_G +\\mu_n\\ell_{\\phi} +\\rho\\sum_{j=i+1}^{m} \\|A_i^\\top\\| \\|A_j\\|\\right)\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\| +\\|A_i^\\top\\|\\|z_{n+1}-z_n\\| \\notag \\\\\n&\\quad +\\rho\\|A_i^\\top\\| \\|B\\| \\|y_{n+1}-y_n\\|.\n\\end{align}\n","start":58609,"end":59592}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":["eq:dy"],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"y块次梯度误差界","source_units":[{"raw_tex":"We now see that\n\\begin{align*}\n\\partial_L ^y \\mathcal{F}(w_n) =\\nabla H(y_{n+1}) + B^\\top z_{n+1} +\\rho B^\\top(\\mathbf{Ax}_{n+1} + By_{n+1} -b) +2c(y_{n+1}-y_{n})    \n\\end{align*}\nand from the optimality condition for the update of $y_{n+1}$ in Algorithm~\\ref{algo:BPLA} that \n\\begin{align*}\n0 &= \\nabla H(y_{n+1}) + B^\\top z_n +\\rho B^\\top(\\mathbf{Ax}_{n+1} +By_{n+1} -b) +\\nu_n(\\nabla \\psi(y_{n+1}) - \\nabla \\psi(y_n)).\n\\end{align*}\nThis leads to\n\\begin{align*}\n\\partial_L ^y \\mathcal{F}(w_n) &=B^{\\top}(z_{n+1}-z_n) -\\nu_n(\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)) + 2c(y_{n+1}-y_{n}),   \n\\end{align*}\nand so\n\\begin{align}\\label{eq:dy}\n\\dist(0,\\partial_L ^{y} \\mathcal{F}(w_n)) \n&\\leq \\|B^{\\top}(z_{n+1}-z_n)\\| +\\nu_n\\|\\nabla \\psi(y_{n+1}) -\\nabla \\psi(y_n)\\| + 2c\\|y_{n+1}-y_{n})\\| \\notag \\\\\n&\\leq \\|B^{\\top}(z_{n+1}-z_n)\\| +(\\nu\\ell_\\psi +2c)\\|y_{n+1}-y_{n})\\|.\n\\end{align}\n","start":59593,"end":60470}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":["eq:dzdt"],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"乘子与记忆变量误差","source_units":[{"raw_tex":"Since $\\partial_L ^z \\mathcal{F}(w_n) =\\mathbf{Ax}_{n+1} + By_{n+1} -b =\\frac{1}{\\rho}(z_{n+1} -z_n)$ and\n$\\partial_L ^t \\mathcal{F}(w_n) =-2c(y_{n+1}-y_{n})$, it holds that\n\\begin{align}\\label{eq:dzdt}\n\\dist(0,\\partial_L ^{z} \\mathcal{F}(w_n)) =\\frac{1}{\\rho}\\|z_{n+1}-z_n\\| \\text{~~and~~}\n\\dist(0,\\partial_L ^{t} \\mathcal{F}(w_n)) =2c\\|y_{n+1}-y_n\\|.    \n\\end{align}\n","start":60471,"end":60840}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S064"],"declared_external":false,"source_only":false,"title":"延迟相对误差条件","source_units":[{"raw_tex":"Combining \\eqref{eq:dxi}, \\eqref{eq:dy}, and \\eqref{eq:dzdt}, noting that $\\limsup_{n\\to +\\infty} \\mu_n =\\overline{\\mu} <+\\infty$, and using \\eqref{eq:BTZy} and \\eqref{eq:zy} in the proof of Theorem~\\ref{t:cvg}, we derive that there exists $n_0\\in \\mathbb{N}^*$ and $C_2 \\in \\mathbb{R}_{++}$ such that, for all $n\\geq n_0$, \n\n\\begin{align*}\n\\dist(0, \\partial_L \\mathcal{F}(w_n)) \n\\leq C_2 \\left(\\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\| +\\|y_{n+1}-y_n\\| +\\|y_{n}-y_{n-1}\\|\\right) \n\\leq C_2(\\Delta_{n-1} +\\Delta_{n-2}).\n\\end{align*}\n","start":60840,"end":61366}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S065"],"declared_external":false,"source_only":false,"title":"抽象KL有限长度","source_units":[{"raw_tex":"\\ref{t:full_cvg}: We see that all the conditions in the \\emph{abstract convergence} framework \\cite[Theorem~5.1]{BDL22} are satisfied with $I =\\{1,2\\}$, $\\lambda_1=\\lambda_2=1/2$, $\\alpha_n \\equiv C_1$, $\\beta_n \\equiv 1/(2C_2)$, and $\\varepsilon_n \\equiv 0 $. By \\cite[Theorem~5.1(i)]{BDL22}, \n\\begin{align*}\n\\sum_{n=0}^{+\\infty} (\\|\\mathbf{x}_{n+2}-\\mathbf{x}_{n+1}\\|+\\|y_{n+2}-y_{n+1}\\|) =\\sum_{n=0}^{+\\infty} \\Delta_n < +\\infty,\n\\end{align*}\n","start":61367,"end":61813}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S066"],"declared_external":false,"source_only":false,"title":"全变量有限长度与收敛","source_units":[{"raw_tex":"which implies that $\\sum_{n=0}^{+\\infty} \\|\\mathbf{x}_{n+1}-\\mathbf{x}_{n}\\| <+\\infty$ and $\\sum_{n=0}^{+\\infty} \\|y_{n+1}-y_{n}\\| < +\\infty$. Together with \\eqref{eq:zy}, we derive that $\\sum_{n=0}^{+\\infty} \\|z_{n+1}-z_n\\|<+\\infty$, and hence \n\\begin{align*}\n\\sum_{n=0}^{+\\infty} \\|(\\mathbf{x}_{n+1},y_{n+1},z_{n+1})-(\\mathbf{x}_{n},y_{n},z_{n})\\| < +\\infty,\n\\end{align*}\nwhich yields the convergence of $(\\mathbf{x}_{n},y_{n},z_{n})_{n\\in\\mathbb{N}}$ to $(\\mathbf{x}^{*},y^{*},z^{*})$. In view of Theorem~\\ref{t:cvg}\\ref{t:cvg_seq}, $(\\mathbf{x}^{*},y^{*},z^{*})$ is a stationary point of $\\mathcal{L}_\\rho$.\n","start":61813,"end":62425}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S067"],"declared_external":false,"source_only":false,"title":"零指数有限步收敛","source_units":[{"raw_tex":"\n\\ref{t:full_rate_finite}: This follows from the arguments in the proof of \\cite[Theorem~5.1]{BDL22} and \\cite[Theorem~2(i)]{Attouch2007}. \n","start":62425,"end":62565}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S068"],"declared_external":false,"source_only":false,"title":"线性速率基础估计","source_units":[{"raw_tex":"\\ref{t:full_rate_linear}: According to \\cite[Theorem~5.1(iv)]{BDL22}, there exist $\\gamma_0\\in \\mathbb{R}_{++}$ and $\\zeta\\in \\left(0,1\\right)$ such that, for all $n\\in \\mathbb{N}^*$, \n\\begin{align*}\n\\|(\\mathbf{x}_{n},y_{n})-(\\mathbf{x}^{*},y^{*})\\|\\leq\\gamma_0 \\zeta^{\\frac{n}{2}} \\text{~~and~~}\n|\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*)| \\leq \\gamma_0\\zeta^n.\n\\end{align*}\n","start":62566,"end":62992}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S069"],"declared_external":false,"source_only":false,"title":"线性乘子速率","source_units":[{"raw_tex":"We derive that, for all $n\\in \\mathbb{N}^*$, $\\|\\mathbf{x}_n-\\mathbf{x}^{*}\\|\\leq \\gamma_0\\zeta^{\\frac{n}{2}}$ and $\\|y_n-y^{*}\\|\\leq \\gamma_0\\zeta^{\\frac{n}{2}}$, which yields \n\\begin{align*}\n\\|y_n-y_{n-1}\\| \\leq \\|y_n-y^{*}\\|+\\|y_{n-1}-y^{*}\\| \\leq (1+\\zeta^{-\\frac{1}{2}})\\gamma_0\\zeta^{\\frac{n}{2}}.    \n\\end{align*}\nBy passing to the limit in \\eqref{eq:B'z}, $B^\\top z^{*}=-\\nabla H(y^{*})$. Following the same steps as in \\eqref{eq:BTZy} and \\eqref{eq:zy}, we obtain that, for all $n\\in \\mathbb{N}^*$,\n\\begin{align*}\n\\|z_n-z^{*}\\| &\\leq \\frac{\\ell_H}{\\sqrt{\\lambda}}\\|y_n-y^{*}\\| + \\frac{\\nu \\ell_{\\psi}}{\\sqrt{\\lambda}}\\|y_{n}-y_{n-1}\\|\\leq \\gamma_1\\zeta^{\\frac{n}{2}}, \\text{~~where~} \\gamma_1 :=\\frac{\\ell_H\\gamma_0+(1+\\zeta^{-\\frac{1}{2}})\\nu\\ell_{\\psi}\\gamma_0}{\\sqrt{\\lambda}}.\n\\end{align*}\n","start":62992,"end":63795}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S070"],"declared_external":false,"source_only":false,"title":"线性迭代与可行性速率","source_units":[{"raw_tex":"Consequently,\n\\begin{align*}\n&\\|(\\mathbf{x}_{n},y_{n},z_{n})-(\\mathbf{x}^{*},y^{*},z^{*})\\| \\leq \\|\\mathbf{x}_{n}-\\mathbf{x}^{*}\\| + \\|y_{n}-y^{*}\\| + \\|z_{n}-z^{*}\\|\\leq (2\\gamma_0 +\\gamma_1)\\zeta^{\\frac{n}{2}}, \\\\\n&\\|\\mathbf{Ax}_n+By_n-b\\| =\\frac{1}{\\rho}\\|z_n-z_{n-1}\\| \\leq \\frac{1}{\\rho}(\\|z_n-z^{*}\\| +\\|z_{n-1}-z^{*}\\|) \\leq \\frac{(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho} \\zeta^{\\frac{n}{2}}, \\\\\n\\text{and~} &\\|z_n\\| \\leq \\|z_n-z^{*}\\| + \\|z^{*}\\| \\leq \\gamma_1 \\zeta^{\\frac{n}{2}} + \\|z^{*}\\|.\n\\end{align*}\n","start":63795,"end":64309}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S071"],"declared_external":false,"source_only":false,"title":"线性拉格朗日值速率","source_units":[{"raw_tex":"We now deduce from the definition of $\\mathcal{F}$ that\n\\begin{align*}\n|\\mathcal{L}_{\\rho}(\\mathbf{x}_{n},y_{n},z_{n})- \\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})|&= \\left|\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*) - c\\|y_n-y_{n-1}\\|^2\\right|\\\\\n&\\leq |\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*)| + c\\|y_n-y_{n-1}\\|^2\\\\\n&\\leq \\gamma_2\\zeta^n, \\text{~~where~} \\gamma_2 :=\\gamma_0 + c(1+\\zeta^{-\\frac{1}{2}})^2\\gamma_0^2\n\\end{align*}\n","start":64310,"end":64843}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S072"],"declared_external":false,"source_only":false,"title":"线性目标值速率","source_units":[{"raw_tex":"and from the definition of $\\mathcal{L}_\\rho$ that \n\\begin{align*}\n|F(\\mathbf{x}_n,y_n)-F(\\mathbf{x}^{*},y^{*})| &= |\\mathcal{L}_{\\rho}(\\mathbf{x}_n,y_n,z_n)-\\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})-\\langle z_n, \\mathbf{Ax}_n + By_n-b\\rangle - \\frac{\\rho}{2}\\|\\mathbf{Ax}_n + By_n-b\\|^2| \\\\\n&\\leq |\\mathcal{L}_{\\rho}(\\mathbf{x}_n,y_n,z_n)-\\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})| + \\|z_n\\|\\|\\mathbf{Ax}_n + By_n-b\\| + \\frac{\\rho}{2}\\|\\mathbf{Ax}_n + By_n-b\\|^2  \\\\ \n&\\leq \\gamma_2\\zeta^n + \\left(\\gamma_1 \\zeta^{\\frac{n}{2}} + \\|z^{*}\\|\\right)\\frac{(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho} \\zeta^{\\frac{n}{2}}+ \\frac{(1+\\zeta^{-\\frac{1}{2}})^2\\gamma_1^2}{2\\rho}\\zeta^n \\\\\n&\\leq \\left(\\gamma_2 +\\frac{(3+\\zeta^{-\\frac{1}{2}})(1+\\zeta^{-\\frac{1}{2}})\\gamma_1^2}{2\\rho} +\\frac{\\|z^{*}\\|(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho}\\right)\\zeta^{\\frac{n}{2}},\n\\end{align*}\nwhere the last inequality follows from the fact that $\\zeta^n \\leq \\zeta^{\\frac{n}{2}}$ since $\\zeta \\in (0,1)$. By letting \n\\begin{align*}\n\\overline{\\gamma} =\\max \\left\\{ 2\\gamma_0+\\gamma_1, \\frac{(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho}, \\gamma_2 +\\frac{(3+\\zeta^{-\\frac{1}{2}})(1+\\zeta^{-\\frac{1}{2}})\\gamma_1^2}{2\\rho} +\\frac{\\|z^{*}\\|(1+\\zeta^{-\\frac{1}{2}})\\gamma_1}{\\rho} \\right\\} \n\\end{align*}\nand increasing it if necessary, we arrive at the conclusion.\n","start":64843,"end":66180}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S073"],"declared_external":false,"source_only":false,"title":"次线性基础估计","source_units":[{"raw_tex":"\\ref{t:full_rate_sublinear}: Following the arguments in \\cite[Theorem~5.1(iv)]{BDL22} and \\cite[Theorem~2(iii)]{Attouch2007}, we find $\\gamma_0 \\in \\mathbb{R}_{++}$ such that, for all $n\\in \\mathbb{N}^*$, \n\\begin{align*}\n\\|(\\mathbf{x}_{n},y_{n})-(\\mathbf{x}^{*},y^{*})\\|\\leq \\gamma_0 n^{-\\frac{1-\\kappa}{2\\kappa-1}} \\text{~~and~~}\n|\\mathcal{F}(\\mathbf{x}_{n},y_{n},z_{n},y_{n-1})-\\mathcal{F}(\\mathbf{x}^{*},y^{*},z^{*},y^*)| \\leq \\gamma_0 n^{-\\frac{2-2\\kappa}{2\\kappa-1}}.\n\\end{align*}\n","start":66181,"end":66667}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S074"],"declared_external":false,"source_only":false,"title":"次线性速率传递","source_units":[{"raw_tex":"Similar to \\ref{t:full_rate_linear}, for all $n\\geq 2$, since $n-1\\geq \\frac{1}{2}n$ and $\\frac{1-\\kappa}{2\\kappa-1} >0$, we derive that \n\\begin{align*}\n&\\|(\\mathbf{x}_{n},y_{n},z_{n})-(\\mathbf{x}^{*},y^{*},z^{*})\\| \\leq (2\\gamma_0+\\gamma_1) n^{-\\frac{1-\\kappa}{2\\kappa-1}},\\\\\n&\\|\\mathbf{Ax}_n+By_n-b\\|\\leq \\frac{(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1}{\\rho} n^{-\\frac{1-\\kappa}{2\\kappa-1}},\\\\\n&|\\mathcal{L}_{\\rho}(\\mathbf{x}_{n},y_{n},z_{n})- \\mathcal{L}_{\\rho}(\\mathbf{x}^{*},y^{*},z^{*})|\\leq \\gamma_2 n^{-\\frac{2-2\\kappa}{2\\kappa-1}},\n\\end{align*}\n","start":66667,"end":67224}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S075"],"declared_external":false,"source_only":false,"title":"次线性目标值速率","source_units":[{"raw_tex":"and that\n\\begin{align*}\n|F(\\mathbf{x}_n,y_n)-F(\\mathbf{x}^{*},y^{*})| &\\leq \\gamma_2 n^{-\\frac{2-2\\kappa}{2\\kappa-1}}+ \\left(\\gamma_1 n^{-\\frac{1-\\kappa}{2\\kappa-1}} + \\|z^{*}\\|\\right)\\frac{(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1}{\\rho} n^{-\\frac{1-\\kappa}{2\\kappa-1}} + \\frac{(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})^2\\gamma_1^2}{2\\rho} n^{-\\frac{2-2\\kappa}{2\\kappa-1}}\\\\\n&\\leq \\left(\\gamma_2 +\\frac{(3 +2^{\\frac{1-\\kappa}{2\\kappa-1}})(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1^2}{2\\rho} +\\frac{\\|z^{*}\\|(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\gamma_1}{\\rho} \\right)n^{-\\frac{1-\\kappa}{2\\kappa-1}},\n\\end{align*}\nwhere $\\gamma_1 :=\\frac{\\ell_H\\gamma_0 +(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})\\nu\\ell_{\\psi}\\gamma_0}{\\sqrt{\\lambda}}$, $\\gamma_2 :=\\gamma_0 +c(1 +2^{\\frac{1-\\kappa}{2\\kappa-1}})^2\\gamma_0^2$, and the last inequality follows from the fact that $n^{-\\frac{2-2\\kappa}{2\\kappa-1}}<n^{-\\frac{1-\\kappa}{2\\kappa-1}}$ for $\\kappa\\in(\\frac{1}{2},1)$. By setting $\\overline{\\gamma}$ in the same way as in \\ref{t:full_rate_linear}, we obtain the conclusion.\n","start":67224,"end":68278}]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S076"],"declared_external":true,"declaration_text":["原先决列表：Mordukhovich (2006), Proposition 1.107(ii)、Theorem 3.36；Rockafellar–Wets (1998), Proposition 10.5；原文第248行明确引用。"],"source_only":false,"title":"外部次微分演算","source_units":[]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S077"],"declared_external":true,"declaration_text":["原先决列表：Pham et al. (2023), Tan2023, Lemma 2.2；原文第258行引用。"],"source_only":false,"title":"外部次梯度聚点引理","source_units":[]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S078"],"declared_external":true,"declaration_text":["原先决列表：Bregman (1967), Section 1；原文第286行引用。"],"source_only":false,"title":"Bregman原文性质","source_units":[]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S079"],"declared_external":true,"declaration_text":["原先决列表：Nesterov (2018), Lemma 1.2.3、Equation (1.2.19)；原文第306、308行引用。"],"source_only":false,"title":"Nesterov光滑估计","source_units":[]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S080"],"declared_external":true,"declaration_text":["原先决列表：Boţ–Dao–Li (2022), BDL22, Lemma 4.1；原文第535行引用。"],"source_only":false,"title":"弱凸次梯度下界","source_units":[]},{"id":"S081","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S081"],"declared_external":true,"declaration_text":["原先决列表：Rockafellar–Wets (1998), Example 9.14；原文第626行引用。"],"source_only":false,"title":"有限凸函数局部Lipschitz","source_units":[]},{"id":"S082","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S082"],"declared_external":true,"declaration_text":["原先决列表：Boţ–Dao–Li (2023), BDL23, Lemma 4.5及其证明；原文第439行引用。"],"source_only":false,"title":"外部对称矩阵谱估计","source_units":[]},{"id":"S083","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S083"],"declared_external":true,"declaration_text":["原先决列表：Boţ–Dao–Li (2022), BDL22, Theorem 5.1及其证明；原文第791、801、803、842行引用。"],"source_only":false,"title":"抽象KL收敛与速率","source_units":[]},{"id":"S084","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S084"],"declared_external":true,"declaration_text":["原先决列表：Attouch–Bolte (2007), Theorem 2(i),(iii)及相关证明；原文第801、842行引用。"],"source_only":false,"title":"有限终止与幂次论证","source_units":[]},{"id":"S085","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S085"],"declared_external":true,"declaration_text":["原先决列表已声明：Standard finite-dimensional compactness, spectral theory, Fermat's rule, convex-gradient monotonicity, and Cauchy–Schwarz inequality；完备性为implicit standard fact made explicit by annotator。"],"source_only":false,"title":"标准有限维分析工具","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S022","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S038","S043":"S043","S044":"S038","S045":"S038","S046":"S038","S047":"S038","S048":"S038","S049":"S049","S050":"S038","S051":"S051","S052":"S038","S053":"S038","S054":"S038","S055":"S038","S056":"S038","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076","S077":"S077","S078":"S078","S079":"S079","S080":"S080","S081":"S081","S082":"S082","S083":"S083","S084":"S084","S085":"S085"}}}}
{"record_id":"1303.1859v4","paper":{"title":"A Cyclic Douglas-Rachford Iteration Scheme","authors":["Borwein, Jonathan M.","Tam, Matthew K."],"arxiv_id":"1303.1859","version":"v4","journal":"Journal of Optimization Theory and Applications","abstract_url":"https://arxiv.org/abs/1303.1859v4","pdf_url":"https://arxiv.org/pdf/1303.1859v4","doi":"10.1007/s10957-013-0381-x"},"context":{"raw_tex":"Throughout this paper,\n\\begin{framed}\n  $$\\mathcal H\\text{ is a real Hilbert space with inner product }\\langle\\cdot,\\cdot\\rangle$$\n\\end{framed}\n\\noindent and induced norm $\\|\\cdot\\|$. We use $\\wto$ to denote weak convergence. \\vspace{1em}\n\nWe consider the $N$-set convex feasibility problem:\n\\begin{framed}\n \\begin{equation}\n  \\text{Find~}~x\\in \\bigcap_{i=1}^N C_i\\neq\\emptyset~\\text{ where }C_i\\subseteq\\mathcal H\\text{ are closed and convex}. \\label{eq:cfp}\n \\end{equation}\n\\end{framed}\n\nGiven a set $S\\subseteq\\mathcal H$ and point $x\\in\\mathcal H$, the \\emph{best approximation} to $x$ from $S$ is a point $p\\in S$ such that\n $$\\|p-x\\|=d(x,S):=\\inf_{s\\in S}\\|x-s\\|.$$\nIf for every $x\\in\\mathcal H$ there exists such a $p$, then $S$ is said to be \\emph{proximal}. Additionally, if $p$ is always unique then $S$ is said to be \\emph{Chebyshev}. In the latter case, the \\emph{projection} onto $S$ is the operator $P_S:\\mathcal H\\to S$ which maps $x$ to its unique nearest point in $S$ and we write $P_S(x)=p$. The \\emph{reflection} about $S$ is the operator $R_S:\\mathcal H\\to\\mathcal H$ defined by $R_S:=2P_S-I$ where $I$ denotes the \\emph{identity} operator which maps any $x\\in\\mathcal H$ to itself.\n\n\n\\begin{fact} \\label{fact:projections}\n Let $C\\subseteq\\mathcal H$ be non-empty closed and convex. Then:\n  \\begin{enumerate}[(i)]\n   \\item $C$ is Chebyshev.\n   \\item (Characterization of projections)\n  $$P_C(x)=p \\iff \\langle x-p,c-p\\rangle\\leq 0 \\text{ for all }c\\in C.$$\n   \\item (Characterization of reflections)\n  $$R_C(x)=r \\iff \\langle x-r,c-r\\rangle\\leq\\frac{1}{2}\\|x-r\\|^2 \\text{ for all }c\\in C.$$\n   \\item (Translation formula) For $y\\in\\mathcal H$, $P_{y+C}(x)=y+P_C(x-y)$.\n   \\item (Dilation formula) For $0\\neq\\lambda\\in\\mathbb R$, $P_{\\lambda C}(x)=\\lambda P_C(x/\\lambda)$.\n   \\item If $C$ is a subspace then $P_C$ is linear.\n   \\item If $C$ is an affine subspace then $P_C$ is affine.\n  \\end{enumerate}\n\\end{fact}\n\n\nGiven $A,B\\subseteq\\mathcal H$  we define the \\emph{$2$-set Douglas--Rachford} operator $T_{A,B}:\\mathcal H\\to\\mathcal H$ by\n\\begin{equation}\\label{tab}\n T_{A,B}:=\\frac{I+R_BR_A}{2}.\n\\end{equation}\nNote that $ T_{A,B}$ and  $T_{B,A}$ are typically distinct, while for an affine set $A$ we have $T_{A,A}=I$.\n\n\nLet $T:\\mathcal H\\to\\mathcal H$. We recall that $T$ is \\emph{asymptotically regular} if $T^{n}x-T^{n+1}x\\to 0$, in norm, for all $x\\in\\mathcal H$.  We denote the set of \\emph{fixed points} of $T$ by $\\Fix T=\\{x:Tx=x\\}$. Let $D\\subseteq\\mathcal H$ and $T:D\\to\\mathcal H$. We say $T$ is \\emph{nonexpansive} if\n   $$\\|Tx-Ty\\|\\leq\\|x-y\\|\\text{ for all }x,y\\in D$$\n(i.e. $1$-Lipschitz). We say $T$ is \\emph{firmly nonexpansive} if\n   $$\\|Tx-Ty\\|^2+\\|(I-T)x-(I-T)y\\|^2\\leq\\|x-y\\|^2\\text{ for all }x,y\\in D.$$\nIt immediately follows that every firmly nonexpansive mapping is nonexpansive.\n\n\n\\begin{fact} \\label{fact:nonexpansive}\n Let $A,B\\subseteq\\mathcal H$ be closed and convex. Then $P_A$ is firmly nonexpansive, $R_A$ is nonexpansive and $T_{A,B}$ is firmly nonexpansive.\n\\end{fact}\n\n\nThe class of nonexpansive mappings is closed under convex combinations, compositions, etc. The class of firmly nonexpansive mappings is, however,  not so well behaved. For example, even  the composition of two projections onto subspaces need not be firmly nonexpansive (see \\cite[Example~4.2.5]{bauschke1997method}).\n\n\n\\begin{lemma}\\label{lem:asymregular}\n Let $T_i:\\mathcal H\\to\\mathcal H$ be firmly nonexpansive, for each $i$, and define $T:=T_r\\dots T_2\\,T_1$. If $\\Fix T\\neq\\emptyset$ then $T$ is asymptotically regular.\n\\end{lemma}\n\n\n\\begin{lemma} \\label{lem:fixedpoints}\n Let $T_i:\\mathcal H\\to\\mathcal H$ be firmly nonexpansive, for each $i$, and define $T:=T_r\\dots T_2\\,T_1$. If $\\bigcap_{i=1}^r\\Fix T_i\\neq\\emptyset$ then $\\Fix T=\\bigcap_{i=1}^r\\Fix T_i$.\n\\end{lemma}\n\n\n\\begin{theorem}[Opial] \\label{thm:opial}\n Let $T:\\mathcal H\\to\\mathcal H$ be nonexpansive, asymptotically regular, and $\\Fix T\\neq\\emptyset$. Then for any $x_0\\in\\mathcal H$, $T^nx_0$  converges weakly to an element of $\\Fix T$.\n\\end{theorem}\n\n\nIn addition, when $T$ is linear, the limit can be identified and  convergence is in norm.\n\n\\begin{theorem} \\label{thm:affine}\n Let $T:\\mathcal H\\to\\mathcal H$ be linear, nonexpansive and asymptotically regular. Then for any $x_0\\in\\mathcal H$, in norm,\n   $$\\lim_{n\\to\\infty}T^nx_0 = P_{\\Fix T}x_0.$$\n\\end{theorem}\n\n\n\\begin{corollary} \\label{cor:cvgt}\n Let $T_i:\\mathcal H\\to\\mathcal H$ be firmly nonexpansive, for each $i$, with $\\bigcap_{i=1}^r\\Fix T_i\\neq\\emptyset$ and define $T:=T_r\\dots T_2T_1$. Then for any $x_0\\in\\mathcal H$, $T^nx_0$ converges weakly to an element of $\\Fix T=\\bigcap_{i=1}^N\\Fix T_i$. Moreover, if $T$ is linear, then $T^nx_0$ converges, in norm, to $P_{\\Fix T}x_0$.\n\\end{corollary}\n\nWe are now ready to introduce our first new projection algorithm, the \\emph{cyclic Douglas--Rachford} iteration scheme. Let $C_1,C_2,\\dots,C_N\\subseteq\\mathcal H$ and define $T_{[C_1\\,C_2\\,\\dots\\,C_N]}:\\mathcal H\\to\\mathcal H$ by\n  \\begin{align*}\n   T_{[C_1\\,C_2\\,\\dots\\,C_N]}\n   &:= T_{C_N,C_1}T_{C_{N-1},C_N}\\dots T_{C_2,C_3}T_{C_1,C_2} \\\\\n   &= \\left(\\frac{I+R_{C_1}R_{C_N}}{2}\\right) \\left(\\frac{I+R_{C_N}R_{C_{N-1}}}{2}\\right)\\dots \\left(\\frac{I+R_{C_3}R_{C_2}}{2}\\right)\\left(\\frac{I+R_{C_2}R_{C_1}}{2}\\right).\n  \\end{align*}\nGiven $x_0\\in\\mathcal H$, the \\emph{cyclic Douglas--Rachford} method iterates by  repeatedly setting $${x_{n+1}=T_{[C_1\\,C_2\\,\\dots\\,C_N]}x_n}.$$\n\n\n\\begin{framed}\n Where there is no ambiguity, we take indices modulo $N$, and abbreviate $T_{C_i,C_j}$ by $T_{i,j}$, and $T_{[C_1\\,C_2\\,\\dots\\,C_N]}$ by $T_{[1\\,2\\,\\dots\\,N]}$. In particular, $T_{0,1}:=T_{N,1},\\, T_{N,N+1}:=T_{N,1},\\, C_0:=C_N$ and $C_{N+1}:=C_1$.\n\\end{framed}\n","units":[{"raw_tex":"Throughout this paper,\n\\begin{framed}\n  $$\\mathcal H\\text{ is a real Hilbert space with inner product }\\langle\\cdot,\\cdot\\rangle$$\n\\end{framed}\n\\noindent and induced norm $\\|\\cdot\\|$. We use $\\wto$ to denote weak convergence. \\vspace{1em}\n\nWe consider the $N$-set convex feasibility problem:\n\\begin{framed}\n \\begin{equation}\n  \\text{Find~}~x\\in \\bigcap_{i=1}^N C_i\\neq\\emptyset~\\text{ where }C_i\\subseteq\\mathcal H\\text{ are closed and convex}. \\label{eq:cfp}\n \\end{equation}\n\\end{framed}\n\nGiven a set $S\\subseteq\\mathcal H$ and point $x\\in\\mathcal H$, the \\emph{best approximation} to $x$ from $S$ is a point $p\\in S$ such that\n $$\\|p-x\\|=d(x,S):=\\inf_{s\\in S}\\|x-s\\|.$$\nIf for every $x\\in\\mathcal H$ there exists such a $p$, then $S$ is said to be \\emph{proximal}. Additionally, if $p$ is always unique then $S$ is said to be \\emph{Chebyshev}. In the latter case, the \\emph{projection} onto $S$ is the operator $P_S:\\mathcal H\\to S$ which maps $x$ to its unique nearest point in $S$ and we write $P_S(x)=p$. The \\emph{reflection} about $S$ is the operator $R_S:\\mathcal H\\to\\mathcal H$ defined by $R_S:=2P_S-I$ where $I$ denotes the \\emph{identity} operator which maps any $x\\in\\mathcal H$ to itself.\n","start":7051,"end":8254},{"raw_tex":"\\begin{fact} \\label{fact:projections}\n Let $C\\subseteq\\mathcal H$ be non-empty closed and convex. Then:\n  \\begin{enumerate}[(i)]\n   \\item $C$ is Chebyshev.\n   \\item (Characterization of projections)\n  $$P_C(x)=p \\iff \\langle x-p,c-p\\rangle\\leq 0 \\text{ for all }c\\in C.$$\n   \\item (Characterization of reflections)\n  $$R_C(x)=r \\iff \\langle x-r,c-r\\rangle\\leq\\frac{1}{2}\\|x-r\\|^2 \\text{ for all }c\\in C.$$\n   \\item (Translation formula) For $y\\in\\mathcal H$, $P_{y+C}(x)=y+P_C(x-y)$.\n   \\item (Dilation formula) For $0\\neq\\lambda\\in\\mathbb R$, $P_{\\lambda C}(x)=\\lambda P_C(x/\\lambda)$.\n   \\item If $C$ is a subspace then $P_C$ is linear.\n   \\item If $C$ is an affine subspace then $P_C$ is affine.\n  \\end{enumerate}\n\\end{fact}\n","start":8255,"end":8983},{"raw_tex":"Given $A,B\\subseteq\\mathcal H$  we define the \\emph{$2$-set Douglas--Rachford} operator $T_{A,B}:\\mathcal H\\to\\mathcal H$ by\n\\begin{equation}\\label{tab}\n T_{A,B}:=\\frac{I+R_BR_A}{2}.\n\\end{equation}\nNote that $ T_{A,B}$ and  $T_{B,A}$ are typically distinct, while for an affine set $A$ we have $T_{A,A}=I$.\n","start":9316,"end":9623},{"raw_tex":"Let $T:\\mathcal H\\to\\mathcal H$. We recall that $T$ is \\emph{asymptotically regular} if $T^{n}x-T^{n+1}x\\to 0$, in norm, for all $x\\in\\mathcal H$.  We denote the set of \\emph{fixed points} of $T$ by $\\Fix T=\\{x:Tx=x\\}$. Let $D\\subseteq\\mathcal H$ and $T:D\\to\\mathcal H$. We say $T$ is \\emph{nonexpansive} if\n   $$\\|Tx-Ty\\|\\leq\\|x-y\\|\\text{ for all }x,y\\in D$$\n(i.e. $1$-Lipschitz). We say $T$ is \\emph{firmly nonexpansive} if\n   $$\\|Tx-Ty\\|^2+\\|(I-T)x-(I-T)y\\|^2\\leq\\|x-y\\|^2\\text{ for all }x,y\\in D.$$\nIt immediately follows that every firmly nonexpansive mapping is nonexpansive.\n","start":10403,"end":10985},{"raw_tex":"\\begin{fact} \\label{fact:nonexpansive}\n Let $A,B\\subseteq\\mathcal H$ be closed and convex. Then $P_A$ is firmly nonexpansive, $R_A$ is nonexpansive and $T_{A,B}$ is firmly nonexpansive.\n\\end{fact}\n","start":10986,"end":11183},{"raw_tex":"The class of nonexpansive mappings is closed under convex combinations, compositions, etc. The class of firmly nonexpansive mappings is, however,  not so well behaved. For example, even  the composition of two projections onto subspaces need not be firmly nonexpansive (see \\cite[Example~4.2.5]{bauschke1997method}).\n","start":11360,"end":11677},{"raw_tex":"\\begin{lemma}\\label{lem:asymregular}\n Let $T_i:\\mathcal H\\to\\mathcal H$ be firmly nonexpansive, for each $i$, and define $T:=T_r\\dots T_2\\,T_1$. If $\\Fix T\\neq\\emptyset$ then $T$ is asymptotically regular.\n\\end{lemma}\n","start":12396,"end":12614},{"raw_tex":"\\begin{lemma} \\label{lem:fixedpoints}\n Let $T_i:\\mathcal H\\to\\mathcal H$ be firmly nonexpansive, for each $i$, and define $T:=T_r\\dots T_2\\,T_1$. If $\\bigcap_{i=1}^r\\Fix T_i\\neq\\emptyset$ then $\\Fix T=\\bigcap_{i=1}^r\\Fix T_i$.\n\\end{lemma}\n","start":13092,"end":13331},{"raw_tex":"\\begin{theorem}[Opial] \\label{thm:opial}\n Let $T:\\mathcal H\\to\\mathcal H$ be nonexpansive, asymptotically regular, and $\\Fix T\\neq\\emptyset$. Then for any $x_0\\in\\mathcal H$, $T^nx_0$  converges weakly to an element of $\\Fix T$.\n\\end{theorem}\n","start":13600,"end":13843},{"raw_tex":"In addition, when $T$ is linear, the limit can be identified and  convergence is in norm.\n\n\\begin{theorem} \\label{thm:affine}\n Let $T:\\mathcal H\\to\\mathcal H$ be linear, nonexpansive and asymptotically regular. Then for any $x_0\\in\\mathcal H$, in norm,\n   $$\\lim_{n\\to\\infty}T^nx_0 = P_{\\Fix T}x_0.$$\n\\end{theorem}\n","start":13961,"end":14276},{"raw_tex":"\\begin{corollary} \\label{cor:cvgt}\n Let $T_i:\\mathcal H\\to\\mathcal H$ be firmly nonexpansive, for each $i$, with $\\bigcap_{i=1}^r\\Fix T_i\\neq\\emptyset$ and define $T:=T_r\\dots T_2T_1$. Then for any $x_0\\in\\mathcal H$, $T^nx_0$ converges weakly to an element of $\\Fix T=\\bigcap_{i=1}^N\\Fix T_i$. Moreover, if $T$ is linear, then $T^nx_0$ converges, in norm, to $P_{\\Fix T}x_0$.\n\\end{corollary}","start":14876,"end":15268},{"raw_tex":"We are now ready to introduce our first new projection algorithm, the \\emph{cyclic Douglas--Rachford} iteration scheme. Let $C_1,C_2,\\dots,C_N\\subseteq\\mathcal H$ and define $T_{[C_1\\,C_2\\,\\dots\\,C_N]}:\\mathcal H\\to\\mathcal H$ by\n  \\begin{align*}\n   T_{[C_1\\,C_2\\,\\dots\\,C_N]}\n   &:= T_{C_N,C_1}T_{C_{N-1},C_N}\\dots T_{C_2,C_3}T_{C_1,C_2} \\\\\n   &= \\left(\\frac{I+R_{C_1}R_{C_N}}{2}\\right) \\left(\\frac{I+R_{C_N}R_{C_{N-1}}}{2}\\right)\\dots \\left(\\frac{I+R_{C_3}R_{C_2}}{2}\\right)\\left(\\frac{I+R_{C_2}R_{C_1}}{2}\\right).\n  \\end{align*}\nGiven $x_0\\in\\mathcal H$, the \\emph{cyclic Douglas--Rachford} method iterates by  repeatedly setting $${x_{n+1}=T_{[C_1\\,C_2\\,\\dots\\,C_N]}x_n}.$$\n","start":15771,"end":16449},{"raw_tex":"\\begin{framed}\n Where there is no ambiguity, we take indices modulo $N$, and abbreviate $T_{C_i,C_j}$ by $T_{i,j}$, and $T_{[C_1\\,C_2\\,\\dots\\,C_N]}$ by $T_{[1\\,2\\,\\dots\\,N]}$. In particular, $T_{0,1}:=T_{N,1},\\, T_{N,N+1}:=T_{N,1},\\, C_0:=C_N$ and $C_{N+1}:=C_1$.\n\\end{framed}\n","start":16749,"end":17026}]},"statement":{"raw_tex":"\\begin{theorem}[Cyclic Douglas--Rachford] \\label{thm:main}\n  Let $C_1,C_2,\\dots,C_N\\subseteq\\mathcal H$ be closed and convex sets with a nonempty intersection. For any $x_0\\in\\mathcal H$, the sequence $T^n_{[1\\,2\\,\\dots\\,N]}x_0$ converges weakly to a point $x$ such that $P_{C_i}x = P_{C_j}x$, for all indices $i,j$. Moreover, $P_{C_j}x\\in \\bigcap_{i=1}^NC_i$, for each index $j$.\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}[Cyclic Douglas--Rachford] \\label{thm:main}\n  Let $C_1,C_2,\\dots,C_N\\subseteq\\mathcal H$ be closed and convex sets with a nonempty intersection. For any $x_0\\in\\mathcal H$, the sequence $T^n_{[1\\,2\\,\\dots\\,N]}x_0$ converges weakly to a point $x$ such that $P_{C_i}x = P_{C_j}x$, for all indices $i,j$. Moreover, $P_{C_j}x\\in \\bigcap_{i=1}^NC_i$, for each index $j$.\n\\end{theorem}","start":17630,"end":18024}]},"proof":{"raw_tex":"\\begin{lemma} \\label{lem:drfp}\n Let $A,B\\subseteq\\mathcal H$ be closed and convex with nonempty intersection. Then \n  $$P_A\\Fix T_{A,B}= A\\cap B.$$\n\\end{lemma}\n\n\\begin{proof}\n $P_A\\Fix T_{A,B}\\subseteq A\\cap B$ since\n \\begin{equation*}\n  x\\in\\Fix T_{A,B}\\iff\\frac{x+R_{B}R_{A}x}{2}=x \\iff P_{A}x=P_{B}R_{A}x\\in A\\cap B.\n \\end{equation*}\n It is straightforward to check the reverse inclusion.\n\\end{proof}\n\n\\begin{proof}\nBy Fact~\\ref{fact:nonexpansive}, $T_{i,i+1}$ is firmly nonexpansive, for each $i$. Further,\n$$\\bigcap_{i=1}^N\\Fix T_{i,i+1}\\supseteq\\bigcap_{i=1}^NC_i\\neq\\emptyset.$$\nBy Corollary~\\ref{cor:cvgt}, $T^n_{[1\\,2\\,\\dots\\,N]}x_0$ converges weakly to a point $x\\in\\Fix T_{[1\\,2\\,\\dots\\,N]}=\\bigcap_{i=1}^N\\Fix T_{i,i+1}$. By Lemma~\\ref{lem:drfp}, $P_{C_{i}}x\\in C_{i+1}$, for each $i$. Now we compute\n \\begin{align*}\n  \\frac{1}{2}\\sum_{i=1}^N \\|P_{C_{i}}x-P_{C_{i-1}}x\\|^2\n  &= \\langle x,0\\rangle+\\frac{1}{2}\\sum_{i=1}^N \\left( \\|P_{C_{i}}x\\|^2-2\\langle P_{C_{i}}x,P_{C_{i-1}}x\\rangle +\\|P_{C_{i-1}}x\\|^2\\right)\\\\\n  &= \\left\\langle x, \\sum_{i=1}^N (P_{C_{i-1}}x-P_{C_{i}}x)\\right\\rangle -\\sum_{i=1}^N\\langle P_{C_{i}}x,P_{C_{i-1}}x\\rangle+ \\sum_{i=1}^N \\|P_{C_{i}}x\\|^2 \\\\\n  &=\\sum_{i=1}^N \\langle x-P_{C_{i}}x,P_{C_{i-1}}x-P_{C_{i}}x\\rangle \\stackrel{\\text{Fact~\\ref{fact:projections}}}{\\leq} 0.\n \\end{align*}\nThus, $P_{C_{i}}x=P_{C_{i-1}}x$, for each $i$; and we are done. \n\\end{proof}","units":[{"raw_tex":"\\begin{lemma} \\label{lem:drfp}\n Let $A,B\\subseteq\\mathcal H$ be closed and convex with nonempty intersection. Then \n  $$P_A\\Fix T_{A,B}= A\\cap B.$$\n\\end{lemma}","start":17116,"end":17275},{"raw_tex":"\\begin{proof}\n $P_A\\Fix T_{A,B}\\subseteq A\\cap B$ since\n \\begin{equation*}\n  x\\in\\Fix T_{A,B}\\iff\\frac{x+R_{B}R_{A}x}{2}=x \\iff P_{A}x=P_{B}R_{A}x\\in A\\cap B.\n \\end{equation*}\n It is straightforward to check the reverse inclusion.\n\\end{proof}","start":17276,"end":17518},{"raw_tex":"\\begin{proof}\nBy Fact~\\ref{fact:nonexpansive}, $T_{i,i+1}$ is firmly nonexpansive, for each $i$. Further,\n$$\\bigcap_{i=1}^N\\Fix T_{i,i+1}\\supseteq\\bigcap_{i=1}^NC_i\\neq\\emptyset.$$\nBy Corollary~\\ref{cor:cvgt}, $T^n_{[1\\,2\\,\\dots\\,N]}x_0$ converges weakly to a point $x\\in\\Fix T_{[1\\,2\\,\\dots\\,N]}=\\bigcap_{i=1}^N\\Fix T_{i,i+1}$. By Lemma~\\ref{lem:drfp}, $P_{C_{i}}x\\in C_{i+1}$, for each $i$. Now we compute\n \\begin{align*}\n  \\frac{1}{2}\\sum_{i=1}^N \\|P_{C_{i}}x-P_{C_{i-1}}x\\|^2\n  &= \\langle x,0\\rangle+\\frac{1}{2}\\sum_{i=1}^N \\left( \\|P_{C_{i}}x\\|^2-2\\langle P_{C_{i}}x,P_{C_{i-1}}x\\rangle +\\|P_{C_{i-1}}x\\|^2\\right)\\\\\n  &= \\left\\langle x, \\sum_{i=1}^N (P_{C_{i-1}}x-P_{C_{i}}x)\\right\\rangle -\\sum_{i=1}^N\\langle P_{C_{i}}x,P_{C_{i-1}}x\\rangle+ \\sum_{i=1}^N \\|P_{C_{i}}x\\|^2 \\\\\n  &=\\sum_{i=1}^N \\langle x-P_{C_{i}}x,P_{C_{i-1}}x-P_{C_{i}}x\\rangle \\stackrel{\\text{Fact~\\ref{fact:projections}}}{\\leq} 0.\n \\end{align*}\nThus, $P_{C_{i}}x=P_{C_{i-1}}x$, for each $i$; and we are done. \n\\end{proof}","start":18026,"end":19020}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"实希尔伯特空间","source_units":[{"raw_tex":"Throughout this paper,\n\\begin{framed}\n  $$\\mathcal H\\text{ is a real Hilbert space with inner product }\\langle\\cdot,\\cdot\\rangle$$\n\\end{framed}\n\\noindent and induced norm $\\|\\cdot\\|$. We use $\\wto$ to denote weak convergence. \\vspace{1em}\n\n","start":7051,"end":7291}]},{"id":"S002","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:cfp"],"declared_external":false,"source_only":false,"title":"有限集凸可行性问题","source_units":[{"raw_tex":"We consider the $N$-set convex feasibility problem:\n\\begin{framed}\n \\begin{equation}\n  \\text{Find~}~x\\in \\bigcap_{i=1}^N C_i\\neq\\emptyset~\\text{ where }C_i\\subseteq\\mathcal H\\text{ are closed and convex}. \\label{eq:cfp}\n \\end{equation}\n\\end{framed}\n\n","start":7291,"end":7541}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"最佳逼近与距离","source_units":[{"raw_tex":"Given a set $S\\subseteq\\mathcal H$ and point $x\\in\\mathcal H$, the \\emph{best approximation} to $x$ from $S$ is a point $p\\in S$ such that\n $$\\|p-x\\|=d(x,S):=\\inf_{s\\in S}\\|x-s\\|.$$\n","start":7541,"end":7723}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"投影与切比雪夫集","source_units":[{"raw_tex":"If for every $x\\in\\mathcal H$ there exists such a $p$, then $S$ is said to be \\emph{proximal}. Additionally, if $p$ is always unique then $S$ is said to be \\emph{Chebyshev}. In the latter case, the \\emph{projection} onto $S$ is the operator $P_S:\\mathcal H\\to S$ which maps $x$ to its unique nearest point in $S$ and we write $P_S(x)=p$. The \\emph{reflection} about $S$ is the operator $R_S:\\mathcal H\\to\\mathcal H$ defined by $R_S:=2P_S-I$ where $I$ denotes the \\emph{identity} operator which maps any $x\\in\\mathcal H$ to itself.\n","start":7723,"end":8254}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"反射与恒等算子","source_units":[{"raw_tex":"If for every $x\\in\\mathcal H$ there exists such a $p$, then $S$ is said to be \\emph{proximal}. Additionally, if $p$ is always unique then $S$ is said to be \\emph{Chebyshev}. In the latter case, the \\emph{projection} onto $S$ is the operator $P_S:\\mathcal H\\to S$ which maps $x$ to its unique nearest point in $S$ and we write $P_S(x)=p$. The \\emph{reflection} about $S$ is the operator $R_S:\\mathcal H\\to\\mathcal H$ defined by $R_S:=2P_S-I$ where $I$ denotes the \\emph{identity} operator which maps any $x\\in\\mathcal H$ to itself.\n","start":7723,"end":8254}]},{"id":"S006","semantic_type":"external_fact","kind":"external_fact","labels":["fact:projections"],"declared_external":true,"declaration_text":["原声明允许的 B0006 标准投影理论；原文 L101 引 Bauschke–Combettes, Theorem 3.14 等。引用式证明未纳入所选 P 单元。"],"source_only":false,"title":"闭凸投影存在唯一","source_units":[{"raw_tex":"\\begin{fact} \\label{fact:projections}\n Let $C\\subseteq\\mathcal H$ be non-empty closed and convex. Then:\n  \\begin{enumerate}[(i)]\n   \\item $C$ is Chebyshev.\n","start":8255,"end":8411}]},{"id":"S007","semantic_type":"external_fact","kind":"external_fact","labels":["fact:projections"],"declared_external":true,"declaration_text":["原声明允许的 B0006 标准投影理论；原文 L101 引 Bauschke–Combettes, Theorem 3.14、Proposition 3.17、Corollary 3.20 及其他教材。"],"source_only":false,"title":"投影变分刻画","source_units":[{"raw_tex":"   \\item (Characterization of projections)\n  $$P_C(x)=p \\iff \\langle x-p,c-p\\rangle\\leq 0 \\text{ for all }c\\in C.$$\n","start":8411,"end":8527}]},{"id":"S008","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原声明允许的 B0006；原文 L101 给出标准投影理论的教材出处。"],"source_only":false,"title":"其余投影性质","source_units":[{"raw_tex":"   \\item (Characterization of reflections)\n  $$R_C(x)=r \\iff \\langle x-r,c-r\\rangle\\leq\\frac{1}{2}\\|x-r\\|^2 \\text{ for all }c\\in C.$$\n   \\item (Translation formula) For $y\\in\\mathcal H$, $P_{y+C}(x)=y+P_C(x-y)$.\n   \\item (Dilation formula) For $0\\neq\\lambda\\in\\mathbb R$, $P_{\\lambda C}(x)=\\lambda P_C(x/\\lambda)$.\n   \\item If $C$ is a subspace then $P_C$ is linear.\n   \\item If $C$ is an affine subspace then $P_C$ is affine.\n  \\end{enumerate}\n\\end{fact}\n","start":8527,"end":8983}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":["tab"],"declared_external":false,"source_only":false,"title":"两集道格拉斯–拉赫福德算子","source_units":[{"raw_tex":"Given $A,B\\subseteq\\mathcal H$  we define the \\emph{$2$-set Douglas--Rachford} operator $T_{A,B}:\\mathcal H\\to\\mathcal H$ by\n\\begin{equation}\\label{tab}\n T_{A,B}:=\\frac{I+R_BR_A}{2}.\n\\end{equation}\n","start":9316,"end":9514}]},{"id":"S010","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"算子次序的补充说明","source_units":[{"raw_tex":"Note that $ T_{A,B}$ and  $T_{B,A}$ are typically distinct, while for an affine set $A$ we have $T_{A,A}=I$.\n","start":9514,"end":9623}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"渐近正则性","source_units":[{"raw_tex":"Let $T:\\mathcal H\\to\\mathcal H$. We recall that $T$ is \\emph{asymptotically regular} if $T^{n}x-T^{n+1}x\\to 0$, in norm, for all $x\\in\\mathcal H$.  We denote the set of \\emph{fixed points} of $T$ by $\\Fix T=\\{x:Tx=x\\}$. Let $D\\subseteq\\mathcal H$ and $T:D\\to\\mathcal H$. We say $T$ is \\emph{nonexpansive} if\n","start":10403,"end":10711}]},{"id":"S012","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"固定点集合","source_units":[{"raw_tex":"Let $T:\\mathcal H\\to\\mathcal H$. We recall that $T$ is \\emph{asymptotically regular} if $T^{n}x-T^{n+1}x\\to 0$, in norm, for all $x\\in\\mathcal H$.  We denote the set of \\emph{fixed points} of $T$ by $\\Fix T=\\{x:Tx=x\\}$. Let $D\\subseteq\\mathcal H$ and $T:D\\to\\mathcal H$. We say $T$ is \\emph{nonexpansive} if\n","start":10403,"end":10711}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"非扩张映射","source_units":[{"raw_tex":"Let $T:\\mathcal H\\to\\mathcal H$. We recall that $T$ is \\emph{asymptotically regular} if $T^{n}x-T^{n+1}x\\to 0$, in norm, for all $x\\in\\mathcal H$.  We denote the set of \\emph{fixed points} of $T$ by $\\Fix T=\\{x:Tx=x\\}$. Let $D\\subseteq\\mathcal H$ and $T:D\\to\\mathcal H$. We say $T$ is \\emph{nonexpansive} if\n   $$\\|Tx-Ty\\|\\leq\\|x-y\\|\\text{ for all }x,y\\in D$$\n","start":10403,"end":10763}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"牢固非扩张映射","source_units":[{"raw_tex":"(i.e. $1$-Lipschitz). We say $T$ is \\emph{firmly nonexpansive} if\n   $$\\|Tx-Ty\\|^2+\\|(I-T)x-(I-T)y\\|^2\\leq\\|x-y\\|^2\\text{ for all }x,y\\in D.$$\n","start":10763,"end":10906}]},{"id":"S015","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"牢固非扩张蕴含非扩张","source_units":[{"raw_tex":"It immediately follows that every firmly nonexpansive mapping is nonexpansive.\n","start":10906,"end":10985}]},{"id":"S016","semantic_type":"external_fact","kind":"external_fact","labels":["fact:nonexpansive"],"declared_external":true,"declaration_text":["原声明允许的 B0014；原文 L131 引 Bauschke–Combettes, Proposition 4.8、Corollary 4.10、Remark 4.24 等。"],"source_only":false,"title":"投影与反射的扩张性质","source_units":[{"raw_tex":"\\begin{fact} \\label{fact:nonexpansive}\n Let $A,B\\subseteq\\mathcal H$ be closed and convex. Then $P_A$ is firmly nonexpansive, $R_A$ is nonexpansive and $T_{A,B}$ is firmly nonexpansive.\n\\end{fact}\n","start":10986,"end":11183}]},{"id":"S017","semantic_type":"external_fact","kind":"external_fact","labels":["fact:nonexpansive"],"declared_external":true,"declaration_text":["原声明允许的 B0014 标准牢固非扩张理论；原文 L131 引 Bauschke–Combettes, Proposition 4.8、Corollary 4.10、Remark 4.24。"],"source_only":false,"title":"两集算子牢固非扩张","source_units":[{"raw_tex":"\\begin{fact} \\label{fact:nonexpansive}\n Let $A,B\\subseteq\\mathcal H$ be closed and convex. Then $P_A$ is firmly nonexpansive, $R_A$ is nonexpansive and $T_{A,B}$ is firmly nonexpansive.\n\\end{fact}\n","start":10986,"end":11183}]},{"id":"S018","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件清单明确允许 Elementary Hilbert space algebra and closure of nonexpansive mappings under composition；作者在 L134 陈述该封闭性。"],"source_only":false,"title":"非扩张复合封闭性","source_units":[{"raw_tex":"The class of nonexpansive mappings is closed under convex combinations, compositions, etc. The class of firmly nonexpansive mappings is, however,  not so well behaved. For example, even  the composition of two projections onto subspaces need not be firmly nonexpansive (see \\cite[Example~4.2.5]{bauschke1997method}).\n","start":11360,"end":11677}]},{"id":"S019","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"牢固非扩张复合的警示","source_units":[{"raw_tex":"The class of nonexpansive mappings is closed under convex combinations, compositions, etc. The class of firmly nonexpansive mappings is, however,  not so well behaved. For example, even  the composition of two projections onto subspaces need not be firmly nonexpansive (see \\cite[Example~4.2.5]{bauschke1997method}).\n","start":11360,"end":11677}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","labels":["lem:asymregular"],"declared_external":true,"declaration_text":["原声明允许的 B0018；原文 L153 引 Bauschke–Combettes, Theorem 5.22。引用式证明不属于所选 P 单元。"],"source_only":false,"title":"牢固非扩张复合的渐近正则性","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:asymregular}\n Let $T_i:\\mathcal H\\to\\mathcal H$ be firmly nonexpansive, for each $i$, and define $T:=T_r\\dots T_2\\,T_1$. If $\\Fix T\\neq\\emptyset$ then $T$ is asymptotically regular.\n\\end{lemma}\n","start":12396,"end":12614}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","labels":["lem:fixedpoints"],"declared_external":true,"declaration_text":["原声明允许的 B0021；原文 L166 引 Bauschke–Combettes, Corollary 4.37。"],"source_only":false,"title":"复合的共同固定点刻画","source_units":[{"raw_tex":"\\begin{lemma} \\label{lem:fixedpoints}\n Let $T_i:\\mathcal H\\to\\mathcal H$ be firmly nonexpansive, for each $i$, and define $T:=T_r\\dots T_2\\,T_1$. If $\\bigcap_{i=1}^r\\Fix T_i\\neq\\emptyset$ then $\\Fix T=\\bigcap_{i=1}^r\\Fix T_i$.\n\\end{lemma}\n","start":13092,"end":13331}]},{"id":"S022","semantic_type":"theorem","kind":"theorem","labels":["thm:opial"],"declared_external":true,"declaration_text":["原声明允许的 Opial 定理 B0023；原文 L176 引 Opial (1967) 或 Bauschke–Combettes, Theorem 5.13。"],"source_only":false,"title":"奥皮亚尔弱收敛定理","source_units":[{"raw_tex":"\\begin{theorem}[Opial] \\label{thm:opial}\n Let $T:\\mathcal H\\to\\mathcal H$ be nonexpansive, asymptotically regular, and $\\Fix T\\neq\\emptyset$. Then for any $x_0\\in\\mathcal H$, $T^nx_0$  converges weakly to an element of $\\Fix T$.\n\\end{theorem}\n","start":13600,"end":13843}]},{"id":"S023","semantic_type":"theorem","kind":"theorem","labels":["thm:affine"],"declared_external":true,"declaration_text":["原声明允许的 B0025；原文 L186 引 Bauschke–Combettes, Proposition 5.27。"],"source_only":false,"title":"线性算子的范数收敛","source_units":[{"raw_tex":"In addition, when $T$ is linear, the limit can be identified and  convergence is in norm.\n\n\\begin{theorem} \\label{thm:affine}\n Let $T:\\mathcal H\\to\\mathcal H$ be linear, nonexpansive and asymptotically regular. Then for any $x_0\\in\\mathcal H$, in norm,\n   $$\\lim_{n\\to\\infty}T^nx_0 = P_{\\Fix T}x_0.$$\n\\end{theorem}\n","start":13961,"end":14276}]},{"id":"S024","semantic_type":"corollary","kind":"corollary","labels":["cor:cvgt"],"declared_external":false,"source_only":false,"title":"牢固非扩张复合收敛推论","source_units":[{"raw_tex":"\\begin{corollary} \\label{cor:cvgt}\n Let $T_i:\\mathcal H\\to\\mathcal H$ be firmly nonexpansive, for each $i$, with $\\bigcap_{i=1}^r\\Fix T_i\\neq\\emptyset$ and define $T:=T_r\\dots T_2T_1$. Then for any $x_0\\in\\mathcal H$, $T^nx_0$ converges weakly to an element of $\\Fix T=\\bigcap_{i=1}^N\\Fix T_i$. Moreover, if $T$ is linear, then $T^nx_0$ converges, in norm, to $P_{\\Fix T}x_0$.\n\\end{corollary}","start":14876,"end":15268}]},{"id":"S025","semantic_type":"assumption","kind":"assumption","labels":["cor:cvgt"],"declared_external":false,"source_only":false,"title":"复合推论的基础假设","source_units":[{"raw_tex":" Let $T_i:\\mathcal H\\to\\mathcal H$ be firmly nonexpansive, for each $i$, with $\\bigcap_{i=1}^r\\Fix T_i\\neq\\emptyset$ and define $T:=T_r\\dots T_2T_1$. Then for any $x_0\\in\\mathcal H$, $T^nx_0$ converges weakly to an element of $\\Fix T=\\bigcap_{i=1}^N\\Fix T_i$. Moreover, if $T$ is linear, then $T^nx_0$ converges, in norm, to $P_{\\Fix T}x_0$.\n","start":14911,"end":15253}]},{"id":"S026","semantic_type":"assumption","kind":"assumption","labels":["cor:cvgt"],"declared_external":false,"source_only":false,"title":"附加线性假设","source_units":[{"raw_tex":" Let $T_i:\\mathcal H\\to\\mathcal H$ be firmly nonexpansive, for each $i$, with $\\bigcap_{i=1}^r\\Fix T_i\\neq\\emptyset$ and define $T:=T_r\\dots T_2T_1$. Then for any $x_0\\in\\mathcal H$, $T^nx_0$ converges weakly to an element of $\\Fix T=\\bigcap_{i=1}^N\\Fix T_i$. Moreover, if $T$ is linear, then $T^nx_0$ converges, in norm, to $P_{\\Fix T}x_0$.\n","start":14911,"end":15253}]},{"id":"S027","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"复合收敛推论的作者证明","source_units":[{"raw_tex":"\\begin{proof}\n Since $T$ is the composition of nonexpansive operators, $T$ is nonexpansive. By  Lemma~\\ref{lem:fixedpoints}, $\\Fix T\\neq\\emptyset$. By Lemma~\\ref{lem:asymregular}, $T$ is asymptotically regular.  The result now follows by Theorem~\\ref{thm:opial} and Theorem~\\ref{thm:affine}. \n\\end{proof}\n","start":15269,"end":15574}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"复合算子非扩张","source_units":[{"raw_tex":" Since $T$ is the composition of nonexpansive operators, $T$ is nonexpansive. By  Lemma~\\ref{lem:fixedpoints}, $\\Fix T\\neq\\emptyset$. By Lemma~\\ref{lem:asymregular}, $T$ is asymptotically regular.  The result now follows by Theorem~\\ref{thm:opial} and Theorem~\\ref{thm:affine}. \n","start":15283,"end":15562}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"复合具有共同固定点","source_units":[{"raw_tex":" Since $T$ is the composition of nonexpansive operators, $T$ is nonexpansive. By  Lemma~\\ref{lem:fixedpoints}, $\\Fix T\\neq\\emptyset$. By Lemma~\\ref{lem:asymregular}, $T$ is asymptotically regular.  The result now follows by Theorem~\\ref{thm:opial} and Theorem~\\ref{thm:affine}. \n","start":15283,"end":15562}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"复合渐近正则","source_units":[{"raw_tex":" Since $T$ is the composition of nonexpansive operators, $T$ is nonexpansive. By  Lemma~\\ref{lem:fixedpoints}, $\\Fix T\\neq\\emptyset$. By Lemma~\\ref{lem:asymregular}, $T$ is asymptotically regular.  The result now follows by Theorem~\\ref{thm:opial} and Theorem~\\ref{thm:affine}. \n","start":15283,"end":15562}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"应用奥皮亚尔定理","source_units":[{"raw_tex":" Since $T$ is the composition of nonexpansive operators, $T$ is nonexpansive. By  Lemma~\\ref{lem:fixedpoints}, $\\Fix T\\neq\\emptyset$. By Lemma~\\ref{lem:asymregular}, $T$ is asymptotically regular.  The result now follows by Theorem~\\ref{thm:opial} and Theorem~\\ref{thm:affine}. \n","start":15283,"end":15562}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"线性分支的范数极限","source_units":[{"raw_tex":" Since $T$ is the composition of nonexpansive operators, $T$ is nonexpansive. By  Lemma~\\ref{lem:fixedpoints}, $\\Fix T\\neq\\emptyset$. By Lemma~\\ref{lem:asymregular}, $T$ is asymptotically regular.  The result now follows by Theorem~\\ref{thm:opial} and Theorem~\\ref{thm:affine}. \n","start":15283,"end":15562}]},{"id":"S033","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"循环算子的复合次序","source_units":[{"raw_tex":"We are now ready to introduce our first new projection algorithm, the \\emph{cyclic Douglas--Rachford} iteration scheme. Let $C_1,C_2,\\dots,C_N\\subseteq\\mathcal H$ and define $T_{[C_1\\,C_2\\,\\dots\\,C_N]}:\\mathcal H\\to\\mathcal H$ by\n  \\begin{align*}\n   T_{[C_1\\,C_2\\,\\dots\\,C_N]}\n   &:= T_{C_N,C_1}T_{C_{N-1},C_N}\\dots T_{C_2,C_3}T_{C_1,C_2} \\\\\n   &= \\left(\\frac{I+R_{C_1}R_{C_N}}{2}\\right) \\left(\\frac{I+R_{C_N}R_{C_{N-1}}}{2}\\right)\\dots \\left(\\frac{I+R_{C_3}R_{C_2}}{2}\\right)\\left(\\frac{I+R_{C_2}R_{C_1}}{2}\\right).\n  \\end{align*}\n","start":15771,"end":16303}]},{"id":"S034","semantic_type":"algorithm","kind":"algorithm","labels":[],"declared_external":false,"source_only":false,"title":"循环迭代规则","source_units":[{"raw_tex":"Given $x_0\\in\\mathcal H$, the \\emph{cyclic Douglas--Rachford} method iterates by  repeatedly setting $${x_{n+1}=T_{[C_1\\,C_2\\,\\dots\\,C_N]}x_n}.$$\n","start":16303,"end":16449}]},{"id":"S035","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"模编号与算子简写","source_units":[{"raw_tex":"\\begin{framed}\n Where there is no ambiguity, we take indices modulo $N$, and abbreviate $T_{C_i,C_j}$ by $T_{i,j}$, and $T_{[C_1\\,C_2\\,\\dots\\,C_N]}$ by $T_{[1\\,2\\,\\dots\\,N]}$. In particular, $T_{0,1}:=T_{N,1},\\, T_{N,N+1}:=T_{N,1},\\, C_0:=C_N$ and $C_{N+1}:=C_1$.\n\\end{framed}\n","start":16749,"end":17026}]},{"id":"S036","semantic_type":"lemma","kind":"lemma","labels":["lem:drfp"],"declared_external":false,"source_only":false,"title":"两集固定点的投影像","source_units":[{"raw_tex":"\\begin{lemma} \\label{lem:drfp}\n Let $A,B\\subseteq\\mathcal H$ be closed and convex with nonempty intersection. Then \n  $$P_A\\Fix T_{A,B}= A\\cap B.$$\n\\end{lemma}","start":17116,"end":17275}]},{"id":"S037","semantic_type":"assumption","kind":"assumption","labels":["lem:drfp"],"declared_external":false,"source_only":false,"title":"固定点引理的集合假设","source_units":[{"raw_tex":" Let $A,B\\subseteq\\mathcal H$ be closed and convex with nonempty intersection. Then \n","start":17147,"end":17232}]},{"id":"S038","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"固定点投影引理的作者证明","source_units":[{"raw_tex":"\\begin{proof}\n $P_A\\Fix T_{A,B}\\subseteq A\\cap B$ since\n \\begin{equation*}\n  x\\in\\Fix T_{A,B}\\iff\\frac{x+R_{B}R_{A}x}{2}=x \\iff P_{A}x=P_{B}R_{A}x\\in A\\cap B.\n \\end{equation*}\n It is straightforward to check the reverse inclusion.\n\\end{proof}","start":17276,"end":17518}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"固定点投影属于两集交","source_units":[{"raw_tex":" $P_A\\Fix T_{A,B}\\subseteq A\\cap B$ since\n \\begin{equation*}\n  x\\in\\Fix T_{A,B}\\iff\\frac{x+R_{B}R_{A}x}{2}=x \\iff P_{A}x=P_{B}R_{A}x\\in A\\cap B.\n \\end{equation*}\n","start":17290,"end":17452}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"反向包含的直接验证","source_units":[{"raw_tex":" It is straightforward to check the reverse inclusion.\n","start":17452,"end":17507}]},{"id":"S041","semantic_type":"theorem","kind":"theorem","labels":["thm:main"],"declared_external":false,"source_only":false,"title":"循环道格拉斯–拉赫福德收敛","source_units":[{"raw_tex":"\\begin{theorem}[Cyclic Douglas--Rachford] \\label{thm:main}\n  Let $C_1,C_2,\\dots,C_N\\subseteq\\mathcal H$ be closed and convex sets with a nonempty intersection. For any $x_0\\in\\mathcal H$, the sequence $T^n_{[1\\,2\\,\\dots\\,N]}x_0$ converges weakly to a point $x$ such that $P_{C_i}x = P_{C_j}x$, for all indices $i,j$. Moreover, $P_{C_j}x\\in \\bigcap_{i=1}^NC_i$, for each index $j$.\n\\end{theorem}","start":17630,"end":18024}]},{"id":"S042","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"循环收敛定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nBy Fact~\\ref{fact:nonexpansive}, $T_{i,i+1}$ is firmly nonexpansive, for each $i$. Further,\n$$\\bigcap_{i=1}^N\\Fix T_{i,i+1}\\supseteq\\bigcap_{i=1}^NC_i\\neq\\emptyset.$$\nBy Corollary~\\ref{cor:cvgt}, $T^n_{[1\\,2\\,\\dots\\,N]}x_0$ converges weakly to a point $x\\in\\Fix T_{[1\\,2\\,\\dots\\,N]}=\\bigcap_{i=1}^N\\Fix T_{i,i+1}$. By Lemma~\\ref{lem:drfp}, $P_{C_{i}}x\\in C_{i+1}$, for each $i$. Now we compute\n \\begin{align*}\n  \\frac{1}{2}\\sum_{i=1}^N \\|P_{C_{i}}x-P_{C_{i-1}}x\\|^2\n  &= \\langle x,0\\rangle+\\frac{1}{2}\\sum_{i=1}^N \\left( \\|P_{C_{i}}x\\|^2-2\\langle P_{C_{i}}x,P_{C_{i-1}}x\\rangle +\\|P_{C_{i-1}}x\\|^2\\right)\\\\\n  &= \\left\\langle x, \\sum_{i=1}^N (P_{C_{i-1}}x-P_{C_{i}}x)\\right\\rangle -\\sum_{i=1}^N\\langle P_{C_{i}}x,P_{C_{i-1}}x\\rangle+ \\sum_{i=1}^N \\|P_{C_{i}}x\\|^2 \\\\\n  &=\\sum_{i=1}^N \\langle x-P_{C_{i}}x,P_{C_{i-1}}x-P_{C_{i}}x\\rangle \\stackrel{\\text{Fact~\\ref{fact:projections}}}{\\leq} 0.\n \\end{align*}\nThus, $P_{C_{i}}x=P_{C_{i-1}}x$, for each $i$; and we are done. \n\\end{proof}","start":18026,"end":19020}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"相邻因子牢固非扩张","source_units":[{"raw_tex":"By Fact~\\ref{fact:nonexpansive}, $T_{i,i+1}$ is firmly nonexpansive, for each $i$. Further,\n","start":18040,"end":18132}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"可行点提供共同固定点","source_units":[{"raw_tex":"$$\\bigcap_{i=1}^N\\Fix T_{i,i+1}\\supseteq\\bigcap_{i=1}^NC_i\\neq\\emptyset.$$\n","start":18132,"end":18207}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"取得共同固定点弱极限","source_units":[{"raw_tex":"By Corollary~\\ref{cor:cvgt}, $T^n_{[1\\,2\\,\\dots\\,N]}x_0$ converges weakly to a point $x\\in\\Fix T_{[1\\,2\\,\\dots\\,N]}=\\bigcap_{i=1}^N\\Fix T_{i,i+1}$. By Lemma~\\ref{lem:drfp}, $P_{C_{i}}x\\in C_{i+1}$, for each $i$. Now we compute\n","start":18207,"end":18434}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"投影落入下一约束集","source_units":[{"raw_tex":"By Corollary~\\ref{cor:cvgt}, $T^n_{[1\\,2\\,\\dots\\,N]}x_0$ converges weakly to a point $x\\in\\Fix T_{[1\\,2\\,\\dots\\,N]}=\\bigcap_{i=1}^N\\Fix T_{i,i+1}$. By Lemma~\\ref{lem:drfp}, $P_{C_{i}}x\\in C_{i+1}$, for each $i$. Now we compute\n","start":18207,"end":18434}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"循环平方距离的代数展开","source_units":[{"raw_tex":" \\begin{align*}\n  \\frac{1}{2}\\sum_{i=1}^N \\|P_{C_{i}}x-P_{C_{i-1}}x\\|^2\n  &= \\langle x,0\\rangle+\\frac{1}{2}\\sum_{i=1}^N \\left( \\|P_{C_{i}}x\\|^2-2\\langle P_{C_{i}}x,P_{C_{i-1}}x\\rangle +\\|P_{C_{i-1}}x\\|^2\\right)\\\\\n  &= \\left\\langle x, \\sum_{i=1}^N (P_{C_{i-1}}x-P_{C_{i}}x)\\right\\rangle -\\sum_{i=1}^N\\langle P_{C_{i}}x,P_{C_{i-1}}x\\rangle+ \\sum_{i=1}^N \\|P_{C_{i}}x\\|^2 \\\\\n","start":18434,"end":18806}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"投影不等式使总和非正","source_units":[{"raw_tex":"  &=\\sum_{i=1}^N \\langle x-P_{C_{i}}x,P_{C_{i-1}}x-P_{C_{i}}x\\rangle \\stackrel{\\text{Fact~\\ref{fact:projections}}}{\\leq} 0.\n \\end{align*}\n","start":18806,"end":18944}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"所有投影一致","source_units":[{"raw_tex":"Thus, $P_{C_{i}}x=P_{C_{i-1}}x$, for each $i$; and we are done. \n","start":18944,"end":19009}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"共同投影满足全部约束","source_units":[{"raw_tex":"Thus, $P_{C_{i}}x=P_{C_{i-1}}x$, for each $i$; and we are done. \n","start":18944,"end":19009}]},{"id":"S051","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件清单允许 Elementary Hilbert space algebra；具体代数规则为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"希尔伯特空间有限和代数","source_units":[]},{"id":"S052","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；属于原清单允许的初等希尔伯特空间代数及其有限和推论。"],"source_only":false,"title":"非负有限和与相等传递","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S041","edges":[{"source_id":"S001","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S005","target_id":"S009","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S012","target_id":"S021","kind":"supports"},{"source_id":"S014","target_id":"S021","kind":"supports"},{"source_id":"S011","target_id":"S022","kind":"supports"},{"source_id":"S013","target_id":"S022","kind":"supports"},{"source_id":"S012","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S028","kind":"supports"},{"source_id":"S015","target_id":"S028","kind":"supports"},{"source_id":"S018","target_id":"S028","kind":"supports"},{"source_id":"S025","target_id":"S029","kind":"supports"},{"source_id":"S021","target_id":"S029","kind":"supports"},{"source_id":"S025","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S020","target_id":"S030","kind":"supports"},{"source_id":"S011","target_id":"S030","kind":"supports"},{"source_id":"S013","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S031","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S022","target_id":"S031","kind":"supports"},{"source_id":"S026","target_id":"S032","kind":"supports"},{"source_id":"S028","target_id":"S032","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S023","target_id":"S032","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S027","kind":"supports"},{"source_id":"S031","target_id":"S027","kind":"supports"},{"source_id":"S032","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S024","kind":"supports"},{"source_id":"S009","target_id":"S033","kind":"supports"},{"source_id":"S005","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S009","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S006","target_id":"S039","kind":"supports"},{"source_id":"S004","target_id":"S039","kind":"supports"},{"source_id":"S005","target_id":"S039","kind":"supports"},{"source_id":"S009","target_id":"S039","kind":"supports"},{"source_id":"S012","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S004","target_id":"S040","kind":"supports"},{"source_id":"S005","target_id":"S040","kind":"supports"},{"source_id":"S009","target_id":"S040","kind":"supports"},{"source_id":"S012","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S033","target_id":"S041","kind":"supports"},{"source_id":"S034","target_id":"S041","kind":"supports"},{"source_id":"S035","target_id":"S041","kind":"supports"},{"source_id":"S004","target_id":"S041","kind":"supports"},{"source_id":"S001","target_id":"S041","kind":"supports"},{"source_id":"S002","target_id":"S043","kind":"supports"},{"source_id":"S017","target_id":"S043","kind":"supports"},{"source_id":"S035","target_id":"S043","kind":"supports"},{"source_id":"S002","target_id":"S044","kind":"supports"},{"source_id":"S006","target_id":"S044","kind":"supports"},{"source_id":"S004","target_id":"S044","kind":"supports"},{"source_id":"S005","target_id":"S044","kind":"supports"},{"source_id":"S009","target_id":"S044","kind":"supports"},{"source_id":"S012","target_id":"S044","kind":"supports"},{"source_id":"S035","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S033","target_id":"S045","kind":"supports"},{"source_id":"S034","target_id":"S045","kind":"supports"},{"source_id":"S035","target_id":"S045","kind":"supports"},{"source_id":"S024","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S002","target_id":"S046","kind":"supports"},{"source_id":"S035","target_id":"S046","kind":"supports"},{"source_id":"S036","target_id":"S046","kind":"supports"},{"source_id":"S035","target_id":"S047","kind":"supports"},{"source_id":"S051","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S051","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S035","target_id":"S048","kind":"supports"},{"source_id":"S007","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S052","target_id":"S049","kind":"supports"},{"source_id":"S035","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S046","target_id":"S050","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S044","target_id":"S042","kind":"supports"},{"source_id":"S045","target_id":"S042","kind":"supports"},{"source_id":"S046","target_id":"S042","kind":"supports"},{"source_id":"S047","target_id":"S042","kind":"supports"},{"source_id":"S048","target_id":"S042","kind":"supports"},{"source_id":"S049","target_id":"S042","kind":"supports"},{"source_id":"S050","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S041","edges":[{"source_id":"S001","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S005","target_id":"S009","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S012","target_id":"S021","kind":"supports"},{"source_id":"S014","target_id":"S021","kind":"supports"},{"source_id":"S011","target_id":"S022","kind":"supports"},{"source_id":"S013","target_id":"S022","kind":"supports"},{"source_id":"S012","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S015","target_id":"S024","kind":"supports"},{"source_id":"S018","target_id":"S024","kind":"supports"},{"source_id":"S021","target_id":"S024","kind":"supports"},{"source_id":"S020","target_id":"S024","kind":"supports"},{"source_id":"S011","target_id":"S024","kind":"supports"},{"source_id":"S013","target_id":"S024","kind":"supports"},{"source_id":"S022","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S009","target_id":"S033","kind":"supports"},{"source_id":"S005","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S009","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S006","target_id":"S036","kind":"supports"},{"source_id":"S004","target_id":"S036","kind":"supports"},{"source_id":"S005","target_id":"S036","kind":"supports"},{"source_id":"S009","target_id":"S036","kind":"supports"},{"source_id":"S012","target_id":"S036","kind":"supports"},{"source_id":"S033","target_id":"S041","kind":"supports"},{"source_id":"S034","target_id":"S041","kind":"supports"},{"source_id":"S035","target_id":"S041","kind":"supports"},{"source_id":"S004","target_id":"S041","kind":"supports"},{"source_id":"S001","target_id":"S041","kind":"supports"},{"source_id":"S002","target_id":"S043","kind":"supports"},{"source_id":"S017","target_id":"S043","kind":"supports"},{"source_id":"S035","target_id":"S043","kind":"supports"},{"source_id":"S002","target_id":"S044","kind":"supports"},{"source_id":"S006","target_id":"S044","kind":"supports"},{"source_id":"S004","target_id":"S044","kind":"supports"},{"source_id":"S005","target_id":"S044","kind":"supports"},{"source_id":"S009","target_id":"S044","kind":"supports"},{"source_id":"S012","target_id":"S044","kind":"supports"},{"source_id":"S035","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S033","target_id":"S045","kind":"supports"},{"source_id":"S034","target_id":"S045","kind":"supports"},{"source_id":"S035","target_id":"S045","kind":"supports"},{"source_id":"S024","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S002","target_id":"S046","kind":"supports"},{"source_id":"S035","target_id":"S046","kind":"supports"},{"source_id":"S036","target_id":"S046","kind":"supports"},{"source_id":"S035","target_id":"S047","kind":"supports"},{"source_id":"S051","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S051","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S035","target_id":"S048","kind":"supports"},{"source_id":"S007","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S052","target_id":"S049","kind":"supports"},{"source_id":"S035","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S046","target_id":"S050","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S044","target_id":"S042","kind":"supports"},{"source_id":"S045","target_id":"S042","kind":"supports"},{"source_id":"S046","target_id":"S042","kind":"supports"},{"source_id":"S047","target_id":"S042","kind":"supports"},{"source_id":"S048","target_id":"S042","kind":"supports"},{"source_id":"S049","target_id":"S042","kind":"supports"},{"source_id":"S050","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"实希尔伯特空间","source_units":[{"raw_tex":"Throughout this paper,\n\\begin{framed}\n  $$\\mathcal H\\text{ is a real Hilbert space with inner product }\\langle\\cdot,\\cdot\\rangle$$\n\\end{framed}\n\\noindent and induced norm $\\|\\cdot\\|$. We use $\\wto$ to denote weak convergence. \\vspace{1em}\n\n","start":7051,"end":7291}]},{"id":"S002","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:cfp"],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"有限集凸可行性问题","source_units":[{"raw_tex":"We consider the $N$-set convex feasibility problem:\n\\begin{framed}\n \\begin{equation}\n  \\text{Find~}~x\\in \\bigcap_{i=1}^N C_i\\neq\\emptyset~\\text{ where }C_i\\subseteq\\mathcal H\\text{ are closed and convex}. \\label{eq:cfp}\n \\end{equation}\n\\end{framed}\n\n","start":7291,"end":7541}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"最佳逼近与距离","source_units":[{"raw_tex":"Given a set $S\\subseteq\\mathcal H$ and point $x\\in\\mathcal H$, the \\emph{best approximation} to $x$ from $S$ is a point $p\\in S$ such that\n $$\\|p-x\\|=d(x,S):=\\inf_{s\\in S}\\|x-s\\|.$$\n","start":7541,"end":7723}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"投影与切比雪夫集","source_units":[{"raw_tex":"If for every $x\\in\\mathcal H$ there exists such a $p$, then $S$ is said to be \\emph{proximal}. Additionally, if $p$ is always unique then $S$ is said to be \\emph{Chebyshev}. In the latter case, the \\emph{projection} onto $S$ is the operator $P_S:\\mathcal H\\to S$ which maps $x$ to its unique nearest point in $S$ and we write $P_S(x)=p$. The \\emph{reflection} about $S$ is the operator $R_S:\\mathcal H\\to\\mathcal H$ defined by $R_S:=2P_S-I$ where $I$ denotes the \\emph{identity} operator which maps any $x\\in\\mathcal H$ to itself.\n","start":7723,"end":8254}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"反射与恒等算子","source_units":[{"raw_tex":"If for every $x\\in\\mathcal H$ there exists such a $p$, then $S$ is said to be \\emph{proximal}. Additionally, if $p$ is always unique then $S$ is said to be \\emph{Chebyshev}. In the latter case, the \\emph{projection} onto $S$ is the operator $P_S:\\mathcal H\\to S$ which maps $x$ to its unique nearest point in $S$ and we write $P_S(x)=p$. The \\emph{reflection} about $S$ is the operator $R_S:\\mathcal H\\to\\mathcal H$ defined by $R_S:=2P_S-I$ where $I$ denotes the \\emph{identity} operator which maps any $x\\in\\mathcal H$ to itself.\n","start":7723,"end":8254}]},{"id":"S006","semantic_type":"external_fact","kind":"external_fact","labels":["fact:projections"],"member_ids":["S006"],"declared_external":true,"declaration_text":["原声明允许的 B0006 标准投影理论；原文 L101 引 Bauschke–Combettes, Theorem 3.14 等。引用式证明未纳入所选 P 单元。"],"source_only":false,"title":"闭凸投影存在唯一","source_units":[{"raw_tex":"\\begin{fact} \\label{fact:projections}\n Let $C\\subseteq\\mathcal H$ be non-empty closed and convex. Then:\n  \\begin{enumerate}[(i)]\n   \\item $C$ is Chebyshev.\n","start":8255,"end":8411}]},{"id":"S007","semantic_type":"external_fact","kind":"external_fact","labels":["fact:projections"],"member_ids":["S007"],"declared_external":true,"declaration_text":["原声明允许的 B0006 标准投影理论；原文 L101 引 Bauschke–Combettes, Theorem 3.14、Proposition 3.17、Corollary 3.20 及其他教材。"],"source_only":false,"title":"投影变分刻画","source_units":[{"raw_tex":"   \\item (Characterization of projections)\n  $$P_C(x)=p \\iff \\langle x-p,c-p\\rangle\\leq 0 \\text{ for all }c\\in C.$$\n","start":8411,"end":8527}]},{"id":"S008","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S008"],"declared_external":true,"declaration_text":["原声明允许的 B0006；原文 L101 给出标准投影理论的教材出处。"],"source_only":false,"title":"其余投影性质","source_units":[{"raw_tex":"   \\item (Characterization of reflections)\n  $$R_C(x)=r \\iff \\langle x-r,c-r\\rangle\\leq\\frac{1}{2}\\|x-r\\|^2 \\text{ for all }c\\in C.$$\n   \\item (Translation formula) For $y\\in\\mathcal H$, $P_{y+C}(x)=y+P_C(x-y)$.\n   \\item (Dilation formula) For $0\\neq\\lambda\\in\\mathbb R$, $P_{\\lambda C}(x)=\\lambda P_C(x/\\lambda)$.\n   \\item If $C$ is a subspace then $P_C$ is linear.\n   \\item If $C$ is an affine subspace then $P_C$ is affine.\n  \\end{enumerate}\n\\end{fact}\n","start":8527,"end":8983}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":["tab"],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"两集道格拉斯–拉赫福德算子","source_units":[{"raw_tex":"Given $A,B\\subseteq\\mathcal H$  we define the \\emph{$2$-set Douglas--Rachford} operator $T_{A,B}:\\mathcal H\\to\\mathcal H$ by\n\\begin{equation}\\label{tab}\n T_{A,B}:=\\frac{I+R_BR_A}{2}.\n\\end{equation}\n","start":9316,"end":9514}]},{"id":"S010","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"算子次序的补充说明","source_units":[{"raw_tex":"Note that $ T_{A,B}$ and  $T_{B,A}$ are typically distinct, while for an affine set $A$ we have $T_{A,A}=I$.\n","start":9514,"end":9623}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"渐近正则性","source_units":[{"raw_tex":"Let $T:\\mathcal H\\to\\mathcal H$. We recall that $T$ is \\emph{asymptotically regular} if $T^{n}x-T^{n+1}x\\to 0$, in norm, for all $x\\in\\mathcal H$.  We denote the set of \\emph{fixed points} of $T$ by $\\Fix T=\\{x:Tx=x\\}$. Let $D\\subseteq\\mathcal H$ and $T:D\\to\\mathcal H$. We say $T$ is \\emph{nonexpansive} if\n","start":10403,"end":10711}]},{"id":"S012","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"固定点集合","source_units":[{"raw_tex":"Let $T:\\mathcal H\\to\\mathcal H$. We recall that $T$ is \\emph{asymptotically regular} if $T^{n}x-T^{n+1}x\\to 0$, in norm, for all $x\\in\\mathcal H$.  We denote the set of \\emph{fixed points} of $T$ by $\\Fix T=\\{x:Tx=x\\}$. Let $D\\subseteq\\mathcal H$ and $T:D\\to\\mathcal H$. We say $T$ is \\emph{nonexpansive} if\n","start":10403,"end":10711}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"非扩张映射","source_units":[{"raw_tex":"Let $T:\\mathcal H\\to\\mathcal H$. We recall that $T$ is \\emph{asymptotically regular} if $T^{n}x-T^{n+1}x\\to 0$, in norm, for all $x\\in\\mathcal H$.  We denote the set of \\emph{fixed points} of $T$ by $\\Fix T=\\{x:Tx=x\\}$. Let $D\\subseteq\\mathcal H$ and $T:D\\to\\mathcal H$. We say $T$ is \\emph{nonexpansive} if\n   $$\\|Tx-Ty\\|\\leq\\|x-y\\|\\text{ for all }x,y\\in D$$\n","start":10403,"end":10763}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"牢固非扩张映射","source_units":[{"raw_tex":"(i.e. $1$-Lipschitz). We say $T$ is \\emph{firmly nonexpansive} if\n   $$\\|Tx-Ty\\|^2+\\|(I-T)x-(I-T)y\\|^2\\leq\\|x-y\\|^2\\text{ for all }x,y\\in D.$$\n","start":10763,"end":10906}]},{"id":"S015","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"牢固非扩张蕴含非扩张","source_units":[{"raw_tex":"It immediately follows that every firmly nonexpansive mapping is nonexpansive.\n","start":10906,"end":10985}]},{"id":"S016","semantic_type":"external_fact","kind":"external_fact","labels":["fact:nonexpansive"],"member_ids":["S016"],"declared_external":true,"declaration_text":["原声明允许的 B0014；原文 L131 引 Bauschke–Combettes, Proposition 4.8、Corollary 4.10、Remark 4.24 等。"],"source_only":false,"title":"投影与反射的扩张性质","source_units":[{"raw_tex":"\\begin{fact} \\label{fact:nonexpansive}\n Let $A,B\\subseteq\\mathcal H$ be closed and convex. Then $P_A$ is firmly nonexpansive, $R_A$ is nonexpansive and $T_{A,B}$ is firmly nonexpansive.\n\\end{fact}\n","start":10986,"end":11183}]},{"id":"S017","semantic_type":"external_fact","kind":"external_fact","labels":["fact:nonexpansive"],"member_ids":["S017"],"declared_external":true,"declaration_text":["原声明允许的 B0014 标准牢固非扩张理论；原文 L131 引 Bauschke–Combettes, Proposition 4.8、Corollary 4.10、Remark 4.24。"],"source_only":false,"title":"两集算子牢固非扩张","source_units":[{"raw_tex":"\\begin{fact} \\label{fact:nonexpansive}\n Let $A,B\\subseteq\\mathcal H$ be closed and convex. Then $P_A$ is firmly nonexpansive, $R_A$ is nonexpansive and $T_{A,B}$ is firmly nonexpansive.\n\\end{fact}\n","start":10986,"end":11183}]},{"id":"S018","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S018"],"declared_external":true,"declaration_text":["原先决条件清单明确允许 Elementary Hilbert space algebra and closure of nonexpansive mappings under composition；作者在 L134 陈述该封闭性。"],"source_only":false,"title":"非扩张复合封闭性","source_units":[{"raw_tex":"The class of nonexpansive mappings is closed under convex combinations, compositions, etc. The class of firmly nonexpansive mappings is, however,  not so well behaved. For example, even  the composition of two projections onto subspaces need not be firmly nonexpansive (see \\cite[Example~4.2.5]{bauschke1997method}).\n","start":11360,"end":11677}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","labels":["lem:asymregular"],"member_ids":["S020"],"declared_external":true,"declaration_text":["原声明允许的 B0018；原文 L153 引 Bauschke–Combettes, Theorem 5.22。引用式证明不属于所选 P 单元。"],"source_only":false,"title":"牢固非扩张复合的渐近正则性","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:asymregular}\n Let $T_i:\\mathcal H\\to\\mathcal H$ be firmly nonexpansive, for each $i$, and define $T:=T_r\\dots T_2\\,T_1$. If $\\Fix T\\neq\\emptyset$ then $T$ is asymptotically regular.\n\\end{lemma}\n","start":12396,"end":12614}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","labels":["lem:fixedpoints"],"member_ids":["S021"],"declared_external":true,"declaration_text":["原声明允许的 B0021；原文 L166 引 Bauschke–Combettes, Corollary 4.37。"],"source_only":false,"title":"复合的共同固定点刻画","source_units":[{"raw_tex":"\\begin{lemma} \\label{lem:fixedpoints}\n Let $T_i:\\mathcal H\\to\\mathcal H$ be firmly nonexpansive, for each $i$, and define $T:=T_r\\dots T_2\\,T_1$. If $\\bigcap_{i=1}^r\\Fix T_i\\neq\\emptyset$ then $\\Fix T=\\bigcap_{i=1}^r\\Fix T_i$.\n\\end{lemma}\n","start":13092,"end":13331}]},{"id":"S022","semantic_type":"theorem","kind":"theorem","labels":["thm:opial"],"member_ids":["S022"],"declared_external":true,"declaration_text":["原声明允许的 Opial 定理 B0023；原文 L176 引 Opial (1967) 或 Bauschke–Combettes, Theorem 5.13。"],"source_only":false,"title":"奥皮亚尔弱收敛定理","source_units":[{"raw_tex":"\\begin{theorem}[Opial] \\label{thm:opial}\n Let $T:\\mathcal H\\to\\mathcal H$ be nonexpansive, asymptotically regular, and $\\Fix T\\neq\\emptyset$. Then for any $x_0\\in\\mathcal H$, $T^nx_0$  converges weakly to an element of $\\Fix T$.\n\\end{theorem}\n","start":13600,"end":13843}]},{"id":"S023","semantic_type":"theorem","kind":"theorem","labels":["thm:affine"],"member_ids":["S023"],"declared_external":true,"declaration_text":["原声明允许的 B0025；原文 L186 引 Bauschke–Combettes, Proposition 5.27。"],"source_only":false,"title":"线性算子的范数收敛","source_units":[{"raw_tex":"In addition, when $T$ is linear, the limit can be identified and  convergence is in norm.\n\n\\begin{theorem} \\label{thm:affine}\n Let $T:\\mathcal H\\to\\mathcal H$ be linear, nonexpansive and asymptotically regular. Then for any $x_0\\in\\mathcal H$, in norm,\n   $$\\lim_{n\\to\\infty}T^nx_0 = P_{\\Fix T}x_0.$$\n\\end{theorem}\n","start":13961,"end":14276}]},{"id":"S024","semantic_type":"corollary","kind":"corollary","labels":["cor:cvgt"],"member_ids":["S024","S027","S028","S029","S030","S031","S032"],"declared_external":false,"source_only":false,"title":"牢固非扩张复合收敛推论","source_units":[{"raw_tex":"\\begin{corollary} \\label{cor:cvgt}\n Let $T_i:\\mathcal H\\to\\mathcal H$ be firmly nonexpansive, for each $i$, with $\\bigcap_{i=1}^r\\Fix T_i\\neq\\emptyset$ and define $T:=T_r\\dots T_2T_1$. Then for any $x_0\\in\\mathcal H$, $T^nx_0$ converges weakly to an element of $\\Fix T=\\bigcap_{i=1}^N\\Fix T_i$. Moreover, if $T$ is linear, then $T^nx_0$ converges, in norm, to $P_{\\Fix T}x_0$.\n\\end{corollary}","start":14876,"end":15268},{"raw_tex":"\\begin{proof}\n Since $T$ is the composition of nonexpansive operators, $T$ is nonexpansive. By  Lemma~\\ref{lem:fixedpoints}, $\\Fix T\\neq\\emptyset$. By Lemma~\\ref{lem:asymregular}, $T$ is asymptotically regular.  The result now follows by Theorem~\\ref{thm:opial} and Theorem~\\ref{thm:affine}. \n\\end{proof}\n","start":15269,"end":15574},{"raw_tex":" Since $T$ is the composition of nonexpansive operators, $T$ is nonexpansive. By  Lemma~\\ref{lem:fixedpoints}, $\\Fix T\\neq\\emptyset$. By Lemma~\\ref{lem:asymregular}, $T$ is asymptotically regular.  The result now follows by Theorem~\\ref{thm:opial} and Theorem~\\ref{thm:affine}. \n","start":15283,"end":15562}]},{"id":"S025","semantic_type":"assumption","kind":"assumption","labels":["cor:cvgt"],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"复合推论的基础假设","source_units":[{"raw_tex":" Let $T_i:\\mathcal H\\to\\mathcal H$ be firmly nonexpansive, for each $i$, with $\\bigcap_{i=1}^r\\Fix T_i\\neq\\emptyset$ and define $T:=T_r\\dots T_2T_1$. Then for any $x_0\\in\\mathcal H$, $T^nx_0$ converges weakly to an element of $\\Fix T=\\bigcap_{i=1}^N\\Fix T_i$. Moreover, if $T$ is linear, then $T^nx_0$ converges, in norm, to $P_{\\Fix T}x_0$.\n","start":14911,"end":15253}]},{"id":"S026","semantic_type":"assumption","kind":"assumption","labels":["cor:cvgt"],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"附加线性假设","source_units":[{"raw_tex":" Let $T_i:\\mathcal H\\to\\mathcal H$ be firmly nonexpansive, for each $i$, with $\\bigcap_{i=1}^r\\Fix T_i\\neq\\emptyset$ and define $T:=T_r\\dots T_2T_1$. Then for any $x_0\\in\\mathcal H$, $T^nx_0$ converges weakly to an element of $\\Fix T=\\bigcap_{i=1}^N\\Fix T_i$. Moreover, if $T$ is linear, then $T^nx_0$ converges, in norm, to $P_{\\Fix T}x_0$.\n","start":14911,"end":15253}]},{"id":"S033","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"循环算子的复合次序","source_units":[{"raw_tex":"We are now ready to introduce our first new projection algorithm, the \\emph{cyclic Douglas--Rachford} iteration scheme. Let $C_1,C_2,\\dots,C_N\\subseteq\\mathcal H$ and define $T_{[C_1\\,C_2\\,\\dots\\,C_N]}:\\mathcal H\\to\\mathcal H$ by\n  \\begin{align*}\n   T_{[C_1\\,C_2\\,\\dots\\,C_N]}\n   &:= T_{C_N,C_1}T_{C_{N-1},C_N}\\dots T_{C_2,C_3}T_{C_1,C_2} \\\\\n   &= \\left(\\frac{I+R_{C_1}R_{C_N}}{2}\\right) \\left(\\frac{I+R_{C_N}R_{C_{N-1}}}{2}\\right)\\dots \\left(\\frac{I+R_{C_3}R_{C_2}}{2}\\right)\\left(\\frac{I+R_{C_2}R_{C_1}}{2}\\right).\n  \\end{align*}\n","start":15771,"end":16303}]},{"id":"S034","semantic_type":"algorithm","kind":"algorithm","labels":[],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"循环迭代规则","source_units":[{"raw_tex":"Given $x_0\\in\\mathcal H$, the \\emph{cyclic Douglas--Rachford} method iterates by  repeatedly setting $${x_{n+1}=T_{[C_1\\,C_2\\,\\dots\\,C_N]}x_n}.$$\n","start":16303,"end":16449}]},{"id":"S035","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"模编号与算子简写","source_units":[{"raw_tex":"\\begin{framed}\n Where there is no ambiguity, we take indices modulo $N$, and abbreviate $T_{C_i,C_j}$ by $T_{i,j}$, and $T_{[C_1\\,C_2\\,\\dots\\,C_N]}$ by $T_{[1\\,2\\,\\dots\\,N]}$. In particular, $T_{0,1}:=T_{N,1},\\, T_{N,N+1}:=T_{N,1},\\, C_0:=C_N$ and $C_{N+1}:=C_1$.\n\\end{framed}\n","start":16749,"end":17026}]},{"id":"S036","semantic_type":"lemma","kind":"lemma","labels":["lem:drfp"],"member_ids":["S036","S038","S039","S040"],"declared_external":false,"source_only":false,"title":"两集固定点的投影像","source_units":[{"raw_tex":"\\begin{lemma} \\label{lem:drfp}\n Let $A,B\\subseteq\\mathcal H$ be closed and convex with nonempty intersection. Then \n  $$P_A\\Fix T_{A,B}= A\\cap B.$$\n\\end{lemma}","start":17116,"end":17275},{"raw_tex":"\\begin{proof}\n $P_A\\Fix T_{A,B}\\subseteq A\\cap B$ since\n \\begin{equation*}\n  x\\in\\Fix T_{A,B}\\iff\\frac{x+R_{B}R_{A}x}{2}=x \\iff P_{A}x=P_{B}R_{A}x\\in A\\cap B.\n \\end{equation*}\n It is straightforward to check the reverse inclusion.\n\\end{proof}","start":17276,"end":17518},{"raw_tex":" $P_A\\Fix T_{A,B}\\subseteq A\\cap B$ since\n \\begin{equation*}\n  x\\in\\Fix T_{A,B}\\iff\\frac{x+R_{B}R_{A}x}{2}=x \\iff P_{A}x=P_{B}R_{A}x\\in A\\cap B.\n \\end{equation*}\n","start":17290,"end":17452},{"raw_tex":" It is straightforward to check the reverse inclusion.\n","start":17452,"end":17507}]},{"id":"S037","semantic_type":"assumption","kind":"assumption","labels":["lem:drfp"],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"固定点引理的集合假设","source_units":[{"raw_tex":" Let $A,B\\subseteq\\mathcal H$ be closed and convex with nonempty intersection. Then \n","start":17147,"end":17232}]},{"id":"S041","semantic_type":"theorem","kind":"theorem","labels":["thm:main"],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"循环道格拉斯–拉赫福德收敛","source_units":[{"raw_tex":"\\begin{theorem}[Cyclic Douglas--Rachford] \\label{thm:main}\n  Let $C_1,C_2,\\dots,C_N\\subseteq\\mathcal H$ be closed and convex sets with a nonempty intersection. For any $x_0\\in\\mathcal H$, the sequence $T^n_{[1\\,2\\,\\dots\\,N]}x_0$ converges weakly to a point $x$ such that $P_{C_i}x = P_{C_j}x$, for all indices $i,j$. Moreover, $P_{C_j}x\\in \\bigcap_{i=1}^NC_i$, for each index $j$.\n\\end{theorem}","start":17630,"end":18024}]},{"id":"S042","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"循环收敛定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nBy Fact~\\ref{fact:nonexpansive}, $T_{i,i+1}$ is firmly nonexpansive, for each $i$. Further,\n$$\\bigcap_{i=1}^N\\Fix T_{i,i+1}\\supseteq\\bigcap_{i=1}^NC_i\\neq\\emptyset.$$\nBy Corollary~\\ref{cor:cvgt}, $T^n_{[1\\,2\\,\\dots\\,N]}x_0$ converges weakly to a point $x\\in\\Fix T_{[1\\,2\\,\\dots\\,N]}=\\bigcap_{i=1}^N\\Fix T_{i,i+1}$. By Lemma~\\ref{lem:drfp}, $P_{C_{i}}x\\in C_{i+1}$, for each $i$. Now we compute\n \\begin{align*}\n  \\frac{1}{2}\\sum_{i=1}^N \\|P_{C_{i}}x-P_{C_{i-1}}x\\|^2\n  &= \\langle x,0\\rangle+\\frac{1}{2}\\sum_{i=1}^N \\left( \\|P_{C_{i}}x\\|^2-2\\langle P_{C_{i}}x,P_{C_{i-1}}x\\rangle +\\|P_{C_{i-1}}x\\|^2\\right)\\\\\n  &= \\left\\langle x, \\sum_{i=1}^N (P_{C_{i-1}}x-P_{C_{i}}x)\\right\\rangle -\\sum_{i=1}^N\\langle P_{C_{i}}x,P_{C_{i-1}}x\\rangle+ \\sum_{i=1}^N \\|P_{C_{i}}x\\|^2 \\\\\n  &=\\sum_{i=1}^N \\langle x-P_{C_{i}}x,P_{C_{i-1}}x-P_{C_{i}}x\\rangle \\stackrel{\\text{Fact~\\ref{fact:projections}}}{\\leq} 0.\n \\end{align*}\nThus, $P_{C_{i}}x=P_{C_{i-1}}x$, for each $i$; and we are done. \n\\end{proof}","start":18026,"end":19020}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"相邻因子牢固非扩张","source_units":[{"raw_tex":"By Fact~\\ref{fact:nonexpansive}, $T_{i,i+1}$ is firmly nonexpansive, for each $i$. Further,\n","start":18040,"end":18132}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"可行点提供共同固定点","source_units":[{"raw_tex":"$$\\bigcap_{i=1}^N\\Fix T_{i,i+1}\\supseteq\\bigcap_{i=1}^NC_i\\neq\\emptyset.$$\n","start":18132,"end":18207}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"取得共同固定点弱极限","source_units":[{"raw_tex":"By Corollary~\\ref{cor:cvgt}, $T^n_{[1\\,2\\,\\dots\\,N]}x_0$ converges weakly to a point $x\\in\\Fix T_{[1\\,2\\,\\dots\\,N]}=\\bigcap_{i=1}^N\\Fix T_{i,i+1}$. By Lemma~\\ref{lem:drfp}, $P_{C_{i}}x\\in C_{i+1}$, for each $i$. Now we compute\n","start":18207,"end":18434}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"投影落入下一约束集","source_units":[{"raw_tex":"By Corollary~\\ref{cor:cvgt}, $T^n_{[1\\,2\\,\\dots\\,N]}x_0$ converges weakly to a point $x\\in\\Fix T_{[1\\,2\\,\\dots\\,N]}=\\bigcap_{i=1}^N\\Fix T_{i,i+1}$. By Lemma~\\ref{lem:drfp}, $P_{C_{i}}x\\in C_{i+1}$, for each $i$. Now we compute\n","start":18207,"end":18434}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"循环平方距离的代数展开","source_units":[{"raw_tex":" \\begin{align*}\n  \\frac{1}{2}\\sum_{i=1}^N \\|P_{C_{i}}x-P_{C_{i-1}}x\\|^2\n  &= \\langle x,0\\rangle+\\frac{1}{2}\\sum_{i=1}^N \\left( \\|P_{C_{i}}x\\|^2-2\\langle P_{C_{i}}x,P_{C_{i-1}}x\\rangle +\\|P_{C_{i-1}}x\\|^2\\right)\\\\\n  &= \\left\\langle x, \\sum_{i=1}^N (P_{C_{i-1}}x-P_{C_{i}}x)\\right\\rangle -\\sum_{i=1}^N\\langle P_{C_{i}}x,P_{C_{i-1}}x\\rangle+ \\sum_{i=1}^N \\|P_{C_{i}}x\\|^2 \\\\\n","start":18434,"end":18806}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"投影不等式使总和非正","source_units":[{"raw_tex":"  &=\\sum_{i=1}^N \\langle x-P_{C_{i}}x,P_{C_{i-1}}x-P_{C_{i}}x\\rangle \\stackrel{\\text{Fact~\\ref{fact:projections}}}{\\leq} 0.\n \\end{align*}\n","start":18806,"end":18944}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"所有投影一致","source_units":[{"raw_tex":"Thus, $P_{C_{i}}x=P_{C_{i-1}}x$, for each $i$; and we are done. \n","start":18944,"end":19009}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"共同投影满足全部约束","source_units":[{"raw_tex":"Thus, $P_{C_{i}}x=P_{C_{i-1}}x$, for each $i$; and we are done. \n","start":18944,"end":19009}]},{"id":"S051","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S051"],"declared_external":true,"declaration_text":["原先决条件清单允许 Elementary Hilbert space algebra；具体代数规则为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"希尔伯特空间有限和代数","source_units":[]},{"id":"S052","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S052"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；属于原清单允许的初等希尔伯特空间代数及其有限和推论。"],"source_only":false,"title":"非负有限和与相等传递","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S024","S028":"S024","S029":"S024","S030":"S024","S031":"S024","S032":"S024","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S036","S039":"S036","S040":"S036","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052"}}}}
{"record_id":"1807.05810v2","paper":{"title":"Union Averaged Operators with Applications to Proximal Algorithms for Min-Convex Functions","authors":["Dao, Minh N.","Tam, Matthew K."],"arxiv_id":"1807.05810","version":"v2","journal":"Journal of Optimization Theory and Applications","abstract_url":"https://arxiv.org/abs/1807.05810v2","pdf_url":"https://arxiv.org/pdf/1807.05810v2","doi":"10.1007/s10957-018-1443-x"},"context":{"raw_tex":"In this section, we recall the necessary preliminaries for the subsequent sections. Unless stated otherwise, throughout this work we assume that\n\\begin{equation*}\n\\fcolorbox{gray!30}{gray!30}{  X\\text{ is a (real) finite-dimensional Hilbert space}  }\n\\end{equation*}\nwith inner-product $\\langle\\cdot,\\cdot\\rangle$ and induced norm $\\|\\cdot\\|$.\n\n\n\\begin{definition}[Averaged nonexpansive operators]\n\\label{def:averaged}\nA single-valued operator $T\\colon X\\to X$ is said to be \\emph{nonexpansive} if \n\\begin{equation*}\n\\forall x,y\\in X,\\quad \\|Tx-Ty\\|\\leq \\|x-y\\|,\n\\end{equation*}\nand \\emph{$\\alpha$-averaged nonexpansive} if $\\alpha\\in(0,1)$ and\n\\begin{equation*}\n\\forall x,y\\in X,\\quad \\|Tx-Ty\\|^2 +\\frac{1-\\alpha}{\\alpha}\\|(\\Id-T)x-(\\Id-T)y\\|^2 \\leq \\|x-y\\|^2.\n\\end{equation*}\nWe say $T$ is \\emph{averaged nonexpansive} if there is an $\\alpha\\in(0,1)$ such that $T$ is $\\alpha$-averaged nonexpansive.\n\\end{definition}    \n\n\n\\begin{proposition}[Characterizations of averaged nonexpansiveness]\n\\label{prop:AN characterizations}\nLet $T\\colon X\\to X$ be an operator and let $\\alpha\\in(0,1)$. The following assertions are equivalent.\n\\begin{enumerate}[label =(\\alph*)]\n\\item $T$ is $\\alpha$-averaged nonexpansive.\n\\item $T =(1-\\alpha)\\Id+\\alpha R$ for some nonexpansive operator $R\\colon X\\to X$.\n\\item $(1-1/\\alpha)\\Id+(1/\\alpha)T$ is nonexpansive.\n\\end{enumerate}\n\\end{proposition}\n\n\\begin{definition}[Outer semicontinuity]\nLet $X$ and $Y$ be Hilbert spaces. A set-valued map $\\phi\\colon X\\setto Y$ is \\emph{outer semicontinuous (osc)} at $\\bar{x}$ if\n\\begin{equation*}\n\\phi(\\bar{x}) \\supseteq \\Limsup_{x\\to\\bar{x}}\\phi(x) :=\\{y\\in Y:\\exists\\,x_n\\to \\bar{x},\\,\\exists\\,y_n\\to y\\text{ with }y_n\\in \\phi(x_n)\\}. \n\\end{equation*}  \nThat is, the limit supremum is understood in the sense of the Painlev\\'e--Kuratowski outer limit on $X\\times Y$.\t\n\\end{definition}\n\n\n\\begin{definition}[Union averaged nonexpansive operators]\\label{def:union averaged}\nA set-valued operator $T\\colon X\\setto X$ is said to be \\emph{union $\\alpha$-averaged nonexpansive} (resp.\\ \\emph{union nonexpansive}) if $T$ can be expressed in the form\n\\begin{equation}\\label{eq:def union averaged}\n\\forall x\\in X,\\quad T(x) =\\{T_i(x): i\\in\\varphi(x)\\},\n\\end{equation}\nwhere $I$ is a finite index set, $\\{T_i\\}_{i\\in I}$ is a collection of $\\alpha$-averaged nonexpansive (resp.\\ nonexpansive) operators on $X$, and $\\varphi\\colon X\\setto I$, called an \\emph{active selector}, is an osc operator with nonempty values.\n\nAs before, we say $T$ is \\emph{union averaged nonexpansive} if there is an $\\alpha\\in(0,1)$ such that $T$ is union $\\alpha$-averaged nonexpansive.\n\\end{definition}\n\n\nThe following proposition is the union averaged nonexpansive analogue of Proposition~\\ref{prop:AN characterizations} and offers equivalent characterizations of union averaged nonexpansiveness. In what follows, the sum of two or more sets is understood in the sense of the \\emph{Minkowski sum}.\n\n\nFor set-valued operators such as those introduced in Definition~\\ref{def:union averaged}, we distinguish two different notions for fixed points which are both the same in the single-valued case. The \\emph{fixed point set} of $T$ is denoted by $\\Fix T :=\\{x:x\\in T(x)\\}$, and the \\emph{strong fixed point set} of $T$ is given by $\\StrFix T :=\\{x:T(x) =\\{x\\}\\}$. \n\n\nIn the following proposition, we take a closer look the structure of a class of set-valued operators which includes union averaged nonexpansive operators as a special case. Given a set-valued operator $T\\colon X\\setto X$, its \\emph{single-valued set}, denoted by \n\\begin{equation*}\n\\Sing T :=\\{x\\in X: T(x)\\text{~is a singleton}\\},\n\\end{equation*}\nis the set of points at which $T$ is single-valued.\n\n\nThe condition that each element in $I$ must appear infinitely often in a sequence is $(i_n)_{n\\in\\mathbb{N}}\\subseteq I$ is generalized in the following definition.\n\\begin{definition}[Admissible sequences]\nLet $I$ and $I^*$ be nonempty finite sets with $I^*\\subseteq I$. A sequence $(i_n)_{n\\in\\mathbb{N}}\\subseteq I$ is \\emph{admissible in $I^*$} if every element of $I^*$ appears  infinitely often in $(i_n)_{n\\in\\mathbb{N}}$.\n\\end{definition}\n\n\nUsing the results of the previous subsection, we now turn our attention convergence of iterations based on union nonexpansive operators. Throughout this subsection, we fix a particular representation for the considered operator. Let $T\\colon X\\setto X$ denote a union nonexpansive (resp.\\ union $\\alpha$-averaged nonexpansive) operator which we assume to be represented as\n\\begin{equation}\\label{eq:a particular uan op}\nT(x) =\\{T_i(x):i\\in\\varphi(x)\\},\n\\end{equation}\nwhere $I$ is a finite index set, $\\{T_i\\}_{i\\in I}$ is a collection of nonexpansive (resp.\\ $\\alpha$-averaged nonexpansive) operators on $X$, and $\\varphi\\colon X\\setto I$ is the osc, nonempty-valued active selector. Fixing this representation is convenient because, in general, the representation of a union nonexpansive (resp.\\ union $\\alpha$-averaged nonexpansive) operator need not be unique and allows us to avoid repetition.\n\nCorresponding to the representation \\eqref{eq:a particular uan op}, we define the \\emph{radius of attraction} of $T$ at a point $x^*\\in X$ as\n\\begin{equation}\\label{eq:radius of attraction}\nr(x^*;T) :=\\sup\\{\\delta>0: \\forall x\\in \\mathbb{B}(x^*;\\delta),\\ \\varphi(x)\\subseteq\\varphi(x^*)\\}\n\\end{equation}\nHere we note that radius of attraction is nonzero for a union nonexpansive operator (and hence too for a union averaged nonexpansive operator). In fact, we have that\n\\begin{equation*}\n\\forall x^*\\in X,\\quad r(x^*;T)\\in (0,+\\infty]\n\\end{equation*}\nas is shown in the following proposition.\n\n\n\\begin{proposition}\\label{prop:prop2.1 Tam17}\nLet $\\varphi\\colon X\\setto I$ for a finite set $I$ and let $x^*\\in X$. Then $\\varphi$ is outer semicontinuous at $x^*$ if and only if there exists $\\delta>0$ such that\n\\begin{equation*}\n\\forall x\\in\\mathbb{B}(x^*;\\delta),\\quad \\varphi(x)\\subseteq\\varphi(x^*).\n\\end{equation*}\nConsequently, if $T\\colon X\\setto X$ is a union nonexpansive operator, then $r(x^*;T)\\in (0,+\\infty]$. \n\\end{proposition}","units":[{"raw_tex":"In this section, we recall the necessary preliminaries for the subsequent sections. Unless stated otherwise, throughout this work we assume that\n\\begin{equation*}\n\\fcolorbox{gray!30}{gray!30}{  X\\text{ is a (real) finite-dimensional Hilbert space}  }\n\\end{equation*}\nwith inner-product $\\langle\\cdot,\\cdot\\rangle$ and induced norm $\\|\\cdot\\|$.\n","start":7427,"end":7771},{"raw_tex":"\\begin{definition}[Averaged nonexpansive operators]\n\\label{def:averaged}\nA single-valued operator $T\\colon X\\to X$ is said to be \\emph{nonexpansive} if \n\\begin{equation*}\n\\forall x,y\\in X,\\quad \\|Tx-Ty\\|\\leq \\|x-y\\|,\n\\end{equation*}\nand \\emph{$\\alpha$-averaged nonexpansive} if $\\alpha\\in(0,1)$ and\n\\begin{equation*}\n\\forall x,y\\in X,\\quad \\|Tx-Ty\\|^2 +\\frac{1-\\alpha}{\\alpha}\\|(\\Id-T)x-(\\Id-T)y\\|^2 \\leq \\|x-y\\|^2.\n\\end{equation*}\nWe say $T$ is \\emph{averaged nonexpansive} if there is an $\\alpha\\in(0,1)$ such that $T$ is $\\alpha$-averaged nonexpansive.\n\\end{definition}    \n","start":8045,"end":8622},{"raw_tex":"\\begin{proposition}[Characterizations of averaged nonexpansiveness]\n\\label{prop:AN characterizations}\nLet $T\\colon X\\to X$ be an operator and let $\\alpha\\in(0,1)$. The following assertions are equivalent.\n\\begin{enumerate}[label =(\\alph*)]\n\\item $T$ is $\\alpha$-averaged nonexpansive.\n\\item $T =(1-\\alpha)\\Id+\\alpha R$ for some nonexpansive operator $R\\colon X\\to X$.\n\\item $(1-1/\\alpha)\\Id+(1/\\alpha)T$ is nonexpansive.\n\\end{enumerate}\n\\end{proposition}","start":8820,"end":9274},{"raw_tex":"\\begin{definition}[Outer semicontinuity]\nLet $X$ and $Y$ be Hilbert spaces. A set-valued map $\\phi\\colon X\\setto Y$ is \\emph{outer semicontinuous (osc)} at $\\bar{x}$ if\n\\begin{equation*}\n\\phi(\\bar{x}) \\supseteq \\Limsup_{x\\to\\bar{x}}\\phi(x) :=\\{y\\in Y:\\exists\\,x_n\\to \\bar{x},\\,\\exists\\,y_n\\to y\\text{ with }y_n\\in \\phi(x_n)\\}. \n\\end{equation*}  \nThat is, the limit supremum is understood in the sense of the Painlev\\'e--Kuratowski outer limit on $X\\times Y$.\t\n\\end{definition}\n","start":11302,"end":11779},{"raw_tex":"\\begin{definition}[Union averaged nonexpansive operators]\\label{def:union averaged}\nA set-valued operator $T\\colon X\\setto X$ is said to be \\emph{union $\\alpha$-averaged nonexpansive} (resp.\\ \\emph{union nonexpansive}) if $T$ can be expressed in the form\n\\begin{equation}\\label{eq:def union averaged}\n\\forall x\\in X,\\quad T(x) =\\{T_i(x): i\\in\\varphi(x)\\},\n\\end{equation}\nwhere $I$ is a finite index set, $\\{T_i\\}_{i\\in I}$ is a collection of $\\alpha$-averaged nonexpansive (resp.\\ nonexpansive) operators on $X$, and $\\varphi\\colon X\\setto I$, called an \\emph{active selector}, is an osc operator with nonempty values.\n\nAs before, we say $T$ is \\emph{union averaged nonexpansive} if there is an $\\alpha\\in(0,1)$ such that $T$ is union $\\alpha$-averaged nonexpansive.\n\\end{definition}\n","start":13786,"end":14570},{"raw_tex":"The following proposition is the union averaged nonexpansive analogue of Proposition~\\ref{prop:AN characterizations} and offers equivalent characterizations of union averaged nonexpansiveness. In what follows, the sum of two or more sets is understood in the sense of the \\emph{Minkowski sum}.\n","start":15789,"end":16083},{"raw_tex":"For set-valued operators such as those introduced in Definition~\\ref{def:union averaged}, we distinguish two different notions for fixed points which are both the same in the single-valued case. The \\emph{fixed point set} of $T$ is denoted by $\\Fix T :=\\{x:x\\in T(x)\\}$, and the \\emph{strong fixed point set} of $T$ is given by $\\StrFix T :=\\{x:T(x) =\\{x\\}\\}$. \n\n\nIn the following proposition, we take a closer look the structure of a class of set-valued operators which includes union averaged nonexpansive operators as a special case. Given a set-valued operator $T\\colon X\\setto X$, its \\emph{single-valued set}, denoted by \n\\begin{equation*}\n\\Sing T :=\\{x\\in X: T(x)\\text{~is a singleton}\\},\n\\end{equation*}\nis the set of points at which $T$ is single-valued.\n","start":23098,"end":23862},{"raw_tex":"The condition that each element in $I$ must appear infinitely often in a sequence is $(i_n)_{n\\in\\mathbb{N}}\\subseteq I$ is generalized in the following definition.\n\\begin{definition}[Admissible sequences]\nLet $I$ and $I^*$ be nonempty finite sets with $I^*\\subseteq I$. A sequence $(i_n)_{n\\in\\mathbb{N}}\\subseteq I$ is \\emph{admissible in $I^*$} if every element of $I^*$ appears  infinitely often in $(i_n)_{n\\in\\mathbb{N}}$.\n\\end{definition}\n","start":28137,"end":28583},{"raw_tex":"Using the results of the previous subsection, we now turn our attention convergence of iterations based on union nonexpansive operators. Throughout this subsection, we fix a particular representation for the considered operator. Let $T\\colon X\\setto X$ denote a union nonexpansive (resp.\\ union $\\alpha$-averaged nonexpansive) operator which we assume to be represented as\n\\begin{equation}\\label{eq:a particular uan op}\nT(x) =\\{T_i(x):i\\in\\varphi(x)\\},\n\\end{equation}\nwhere $I$ is a finite index set, $\\{T_i\\}_{i\\in I}$ is a collection of nonexpansive (resp.\\ $\\alpha$-averaged nonexpansive) operators on $X$, and $\\varphi\\colon X\\setto I$ is the osc, nonempty-valued active selector. Fixing this representation is convenient because, in general, the representation of a union nonexpansive (resp.\\ union $\\alpha$-averaged nonexpansive) operator need not be unique and allows us to avoid repetition.\n\nCorresponding to the representation \\eqref{eq:a particular uan op}, we define the \\emph{radius of attraction} of $T$ at a point $x^*\\in X$ as\n\\begin{equation}\\label{eq:radius of attraction}\nr(x^*;T) :=\\sup\\{\\delta>0: \\forall x\\in \\mathbb{B}(x^*;\\delta),\\ \\varphi(x)\\subseteq\\varphi(x^*)\\}\n\\end{equation}\nHere we note that radius of attraction is nonzero for a union nonexpansive operator (and hence too for a union averaged nonexpansive operator). In fact, we have that\n\\begin{equation*}\n\\forall x^*\\in X,\\quad r(x^*;T)\\in (0,+\\infty]\n\\end{equation*}\nas is shown in the following proposition.\n","start":33037,"end":34530},{"raw_tex":"\\begin{proposition}\\label{prop:prop2.1 Tam17}\nLet $\\varphi\\colon X\\setto I$ for a finite set $I$ and let $x^*\\in X$. Then $\\varphi$ is outer semicontinuous at $x^*$ if and only if there exists $\\delta>0$ such that\n\\begin{equation*}\n\\forall x\\in\\mathbb{B}(x^*;\\delta),\\quad \\varphi(x)\\subseteq\\varphi(x^*).\n\\end{equation*}\nConsequently, if $T\\colon X\\setto X$ is a union nonexpansive operator, then $r(x^*;T)\\in (0,+\\infty]$. \n\\end{proposition}","start":34531,"end":34974}]},"statement":{"raw_tex":"\\begin{corollary}[Local convergence of union averaged nonexpansive iterations]\\label{cor:union averaged}\nLet $T\\colon X\\setto X$ be union $\\alpha$-averaged nonexpansive operator with $x^*\\in\\StrFix T$ and let $(\\lambda_n)_{n\\in\\mathbb{N}}$ be a sequence in $(0, 1/\\alpha]$ with $\\liminf_{n\\to\\infty} \\lambda_n(1/\\alpha-\\lambda_n)> 0$. Denote $r :=r(x^*;T)\\in(0,+\\infty]$ and consider $(x_n)_{n\\in\\mathbb{N}}$ with $x_0\\in \\inte\\mathbb{B}(x^*;r)$ satisfying\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1}\\in (1-\\lambda_n)x_n+\\lambda_n T(x_n).\n\\end{equation*} \nThen $(x_n)_{n\\in\\mathbb{N}}$ converges to a point $\\overline{x}\\in\\Fix T\\cap \\mathbb{B}(x^*;r)$. \n\\end{corollary}","units":[{"raw_tex":"\\begin{corollary}[Local convergence of union averaged nonexpansive iterations]\\label{cor:union averaged}\nLet $T\\colon X\\setto X$ be union $\\alpha$-averaged nonexpansive operator with $x^*\\in\\StrFix T$ and let $(\\lambda_n)_{n\\in\\mathbb{N}}$ be a sequence in $(0, 1/\\alpha]$ with $\\liminf_{n\\to\\infty} \\lambda_n(1/\\alpha-\\lambda_n)> 0$. Denote $r :=r(x^*;T)\\in(0,+\\infty]$ and consider $(x_n)_{n\\in\\mathbb{N}}$ with $x_0\\in \\inte\\mathbb{B}(x^*;r)$ satisfying\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1}\\in (1-\\lambda_n)x_n+\\lambda_n T(x_n).\n\\end{equation*} \nThen $(x_n)_{n\\in\\mathbb{N}}$ converges to a point $\\overline{x}\\in\\Fix T\\cap \\mathbb{B}(x^*;r)$. \n\\end{corollary}","start":39877,"end":40558}]},"proof":{"raw_tex":"\\begin{proposition}[Equivalent characterizations of union averaged nonexpansiveness]\n\\label{prop:averaged}\nLet $T\\colon X\\setto X$ be a set-valued operator and let $\\alpha\\in(0,1)$. The following assertions are equivalent.\n\\begin{enumerate}[label =(\\alph*)]\n\\item $T$ is union $\\alpha$-averaged nonexpansive.\n\\item\\label{it:prop averaged 2} $T =(1-\\alpha)\\Id+\\alpha R$ for some union nonexpansive operator $R\\colon X\\setto X$.\n\\item $(1-1/\\alpha)\\Id+(1/\\alpha)T$ is union nonexpansive.\n\\end{enumerate}\n\\end{proposition}\n\n\\begin{proof}\nFollows by combining Definition~\\ref{def:union averaged} with Proposition~\\ref{prop:AN characterizations}.\n\\end{proof}\n\n\\begin{proposition}[Active selectors, fixed points, single-valuedness]\\label{prop:fixed points}\nLet $T\\colon X\\setto X$ be a set-valued operator given by\n\\begin{equation*}\n\\forall x\\in X,\\quad T(x) =\\{T_i(x): i\\in \\varphi(x)\\}\n\\end{equation*}\nwhere $I$ is a finite index set, $\\varphi\\colon X\\setto I$ is an osc operator with nonempty values, and $\\{T_i\\}_{i\\in I}$ is a collection of single-valued operators on $X$. The following assertions hold.\n\\begin{enumerate}[label =(\\alph*)]\n\\item\\label{it:fixed points_X} For each $i\\in I$, the set $\\varphi^{-1}(i)$ is closed. The space $X$ can be represented as\n\\begin{equation}\n\\label{eq:X is a union}\nX =\\bigcup_{i\\in I}\\varphi^{-1}(i).\n\\end{equation}    \n\\item\\label{it:fixed points_Fix} $x\\in\\Fix T$ if and only if there exists $i\\in I$ such that $x\\in\\varphi^{-1}(i)\\cap\\Fix T_i$. Consequently, we have\n\\begin{equation*}\n\\Fix T =\\bigcup_{i\\in I}\\left[\\varphi^{-1}(i)\\cap\\Fix T_i\\right]. \n\\end{equation*}\n\\item\\label{it:fixed points_StrFix} $x\\in\\StrFix T$ if and only if \n\\begin{equation*}\nx\\in \\bigcap_{i\\in\\varphi(x)}\\Fix T_i =\\bigcap_{i\\in\\varphi(x)} \\left[\\varphi^{-1}(i)\\cap\\Fix T_i\\right].\n\\end{equation*}\n\\item\\label{it:fixed points_StrFixSing} The strong fixed point set is the intersection of the fixed point set and the single-valued set, that is,\n\\begin{equation*}\n\\StrFix T =\\Fix T\\cap \\Sing T.\n\\end{equation*}\n\\item\\label{it:fixed points_Sing} If in addition $\\{T_i\\}_{i\\in I}$ is a collection of continuous operators, then $T$ is ocs. Consequently,\n\\begin{equation*}\n\\Sing T =\\left\\{x\\in X:\\Limsup_{x'\\to x}T(x')\\text{ is singleton}\\right\\}.\n\\end{equation*}\n\\end{enumerate}\n\\end{proposition}\n\n\\begin{proof}\n\\ref{it:fixed points_X}: Let $i\\in I$ and consider a sequence $(x_n)_{n\\in\\mathbb{N}}\\subseteq\\varphi^{-1}(i)$ such that $x_n\\to\\overline{x}$. The definition of the inverse implies that $i\\in\\varphi(x_n)$ for all $n\\in\\mathbb{N}$ and, since $\\varphi$ is osc, it holds that $i\\in \\varphi(\\overline{x})$, which shows that $\\varphi^{-1}(i)$ is closed and proves the first claim. The fact that \\eqref{eq:X is a union} follows from the assumption that $\\varphi(x)\\neq \\varnothing$ for all $x\\in X$.\n\n\\ref{it:fixed points_Fix}: $x\\in\\Fix T\\iff\\exists i\\in \\varphi(x)$ such that $x\\in\\Fix T_i\\iff\\exists i\\in I$ such that $x\\in\\varphi^{-1}(i)\\cap\\Fix T_i.$ \n\n\\ref{it:fixed points_StrFix}: $x\\in\\StrFix T\\iff x\\in\\Fix T_i$ for all $i\\in\\varphi(x)\\iff x\\in\\varphi^{-1}(i)\\cap\\Fix T_i$ for all $i\\in\\varphi(x)$.\n\n\\ref{it:fixed points_StrFixSing}: Immediate from the respective definitions.\n\n\\ref{it:fixed points_Sing}: Fix $x\\in X$. We first show that $T$ is osc. To this end, take $x_n\\to x$ and $y_n\\to y$ with $y_n\\in T(x_n)$. Then the definition of $T$ ensures the existence of sequence $i_n\\in \\varphi(x_n)$ such that $y_n =T_{i_n}(x_n)\\in T(x_n)$. Using the pigeonhole princple, we pass to a subsequence so that $y_{k_n} =T_i(x_{k_n})$ for fixed $i\\in I$. The osc of $\\varphi$ implies that $i\\in\\varphi(x)$ and continuity of $T_i$ gives\n\\begin{equation*}\ny =\\lim_{n\\to\\infty}y_{k_n} =\\lim_{n\\to\\infty}T_i(x_{k_n}) =T_i(x)\\in T(x),\n\\end{equation*}\nwhich proves osc. The claimed formula for the singleton set follows since $x\\in\\Sing T\\iff T(x)$ is singleton, and osc implies that $T(x) =\\Limsup_{x'\\to x}T(x')$. \n\\end{proof}\n\n\\begin{theorem}[Krasnosel'ski\\u{\\i}--Mann iterations with admissible control]\n\\label{thm:KM iterations with admissible control}\nLet $I$ be a finite index set and let $\\{T_i\\}_{i\\in I}$ be a collection of nonexpansive operators on $X$ with a common fixed point. Define a sequence $(x_n)_{n\\in\\mathbb{N}}$ with starting point $x_0\\in X$ according to\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1} :=(1-\\lambda_n)x_n+\\lambda_n T_{i_n}(x_n),\n\\end{equation*}\nwhere $(i_n)_{n\\in\\mathbb{N}}$ is admissible in $I$, and $(\\lambda_n)_{n\\in\\mathbb{N}}$ is in $(0, 1]$ with $\\liminf_{n\\to\\infty}\\lambda_n(1-\\lambda_n)> 0$.\nThen $(x_n)_{n\\in\\mathbb{N}}$ converges to a point $\\overline{x}\\in\\cap_{i\\in I} \\Fix T_i$.\n\\end{theorem}\n\n\\begin{proof}\nSince $\\liminf_{n\\to\\infty}\\lambda_n(1-\\lambda_n)> 0$, there exists $n_0\\in \\mathbb{N}$ such that $\\inf_{n\\geq n_0}\\lambda_n(1-\\lambda_n)> 0$. By relabeling if necessary, we may assume without loss of generality that $\\varepsilon :=\\inf_{n\\in\\mathbb{N}}\\lambda_n(1-\\lambda_n)> 0$. \nLet $x\\in \\cap_{i\\in I} \\Fix T_i$ be arbitrary. Then, for all $n\\in\\mathbb{N}$, nonexpansiveness of $T_{i_n}$ yields\n\\begin{subequations} \n\\begin{align*}\n\\|x_{n+1}-x\\|^2& =\\|(1-\\lambda_n)(x_n-x)+\\lambda_n(T_{i_n}(x_n)-x)\\|^2 \\\\\n& =(1-\\lambda_n)\\|x_n-x\\|^2+\\lambda_n\\|T_{i_n}(x_n)-x\\|^2-\\lambda_n(1-\\lambda_n)\\|x_n-T_{i_n}(x_n)\\|^2 \\\\\n&\\leq \\|x_n-x\\|^2-\\varepsilon\\|x_n-T_{i_n}(x_n)\\|^2.\n\\end{align*}\n\\end{subequations}\nIt follows that $(\\|x_n-x\\|^2)_{n\\in\\mathbb{N}}$ is monotone nonincreasing, and hence convergent. Consequently, the sequence $(x_n)_{n\\in\\mathbb{N}}$ is bounded and \n\\begin{equation}\\label{eq:pre-asymp}\nx_n-T_{i_n}(x_n)\\to 0 \\quad\\text{as~} n\\to\\infty.\n\\end{equation}\nNow, let $\\overline{x}\\in X$ be a cluster point of $(x_n)_{n\\in\\mathbb{N}}$. Then $(x_n)_{n\\in\\mathbb{N}}$ contains a convergent subsequence, say $(x_{k_n})_{n\\in\\mathbb{N}}$, with limit $\\overline{x}$. \nWe claim that $\\overline{x}$ is an element of $\\cap_{i\\in I} \\Fix T_i$. To this end suppose, by way of a contradiction, that $\\overline{x}\\not\\in\\cap_{i\\in I} \\Fix T_i$.\nBy passing to a further subsequence if necessary, we deduce that the sequence $(t_n)_{n\\in\\mathbb{N}}$ defined by\n\\begin{equation*}\nt_n :=\\min\\{p\\in\\{k_n,\\dots,k_{n+1}-1\\}:\\overline{x}\\not\\in\\Fix T_{i_p}\\}, \n\\end{equation*}\nis well defined.\nSince $I$ is finite, by passing to yet another subsequence if necessary, we may assume that $i_{t_n} =\\ell$ for all $n\\in\\mathbb{N}$ for some fixed index $\\ell\\in I$. \nTogether with \\eqref{eq:pre-asymp}, we deduce that\n\\begin{equation*}\nx_{t_n}\\to \\overline{x} \\text{~~~and~~~} (\\Id-T_\\ell)(x_{t_n})\\to 0 \\quad\\text{as~} n\\to\\infty,\n\\end{equation*}\nwhich, due to nonexpansiveness of $T_\\ell$, implies that $\\overline{x}\\in\\Fix T_\\ell$. \nThus a contradiction is obtained, and conclude that $\\overline{x}\\in\\cap_{i\\in I} \\Fix T_i$. We then have that $(\\|x_n-\\overline{x}\\|)_{n\\in\\mathbb{N}}$ is monotone nonincreasing and $\\|x_{k_n}-\\overline{x}\\|\\to 0$, and thus the conclusion follows.\n\\end{proof}\n\n\\begin{proposition}[Radius of attraction at strong fixed points]\\label{prop:radius}\nLet $T\\colon X\\setto X$ be a union nonexpansive operator with $x^*\\in\\StrFix T$. Then $r :=r(x^*;T)\\in (0,+\\infty]$ and\n\\begin{equation}\\label{eq:attraction nonexpansive}\n\\forall x\\in \\inte\\mathbb{B}(x^*;r),\\,\\forall y\\in T(x),\\quad \n\\|y-x^*\\|\\leq \\|x-x^*\\|.\n\\end{equation}\nFurthermore, for any $\\lambda\\in (0, 1]$, we have\n\\begin{equation}\\label{eq:attraction averaged}\n\\forall x\\in \\inte\\mathbb{B}(x^*;r),\\,\\forall y\\in (1-\\lambda)x+\\lambda T(x),\\quad \n\\|y-x^*\\|^2+\\frac{1-\\lambda}{\\lambda}\\|x-y\\|^2\\leq \\|x-x^*\\|^2.\n\\end{equation}\n\\end{proposition}\n\n\\begin{proof}\nSince $\\varphi$ is osc and its range, $I$, is a finite set, Proposition~\\ref{prop:prop2.1 Tam17} implies that $r :=r(x^*;T)\\in(0,+\\infty]$. Since $x^*\\in\\StrFix T$, Proposition~\\ref{prop:fixed points}\\ref{it:fixed points_StrFix} implies that $x^*\\in\\cap_{i\\in\\varphi(x*)}\\Fix T_i$ and, since each $T_i$ is nonexpansive, we have\n\\begin{equation}\n\\label{eq:nonexpansive}\n\\forall i\\in\\varphi(x^*),\\, \\forall x\\in X,\\quad \\|T_i(x)-x^*\\|^2\\leq \\|x-x^*\\|^2.\n\\end{equation}\nIn particular, for any $x\\in \\inte\\mathbb{B}(x^*;r)$ and $y\\in T(x)$, there exists an $i\\in \\varphi(x)\\subseteq \\varphi(x^*)$ such that $y =T_i(x)$. Consequently, \\eqref{eq:attraction nonexpansive} follows from \\eqref{eq:nonexpansive}.\n\nFurthermore, for any $\\lambda\\in (0, 1]$, the operator $S_i :=(1-\\lambda)\\Id+\\lambda T_i$ is $\\lambda$-averaged nonexpansive by Proposition~\\ref{prop:averaged} and we therefore have that\n\\begin{equation}\n\\label{eq:averaged}\n\\forall i\\in\\varphi(x^*),\\, \\forall x\\in X,\\quad \\|S_i(x)-x^*\\|^2+\\frac{1-\\lambda}{\\lambda}\\|x-S_i(x)\\|^2\\leq \\|x-x^*\\|^2.\n\\end{equation}\nIn particular, if $x\\in\\inte\\mathbb{B}(x^*;r)$ and $y\\in (1-\\lambda)x+\\lambda T(x)$, then there exists $i\\in \\varphi(x)\\subseteq \\varphi(x^*)$ such that $y =S_i(x)$. As before, \\eqref{eq:attraction averaged} follows from \\eqref{eq:averaged}.\n\\end{proof}\n\n\\begin{theorem}[Local convergence of union nonexpansive iterations]\\label{thm:union}\nLet $T\\colon X\\setto X$ be a union nonexpansive operator with $x^*\\in\\StrFix T$ and let $(\\lambda_n)_{n\\in\\mathbb{N}}$ be a sequence in $(0, 1]$ with $\\liminf_{n\\to\\infty}\\lambda_n(1-\\lambda_n)> 0$. Denote $r :=r(x^*;T)\\in(0,+\\infty]$ and consider a sequence $(x_n)_{n\\in\\mathbb{N}}$ with $x_0\\in \\inte\\mathbb{B}(x^*;r)$ satisfying\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1}\\in (1-\\lambda_n)x_n+\\lambda_n T(x_n).\n\\end{equation*} \nThen $(x_n)_{n\\in\\mathbb{N}}$ converges to a point $\\overline{x}\\in\\Fix T\\cap \\mathbb{B}(x^*;r)$. \n\\end{theorem}\n\n\\begin{proof}\nWe first observe from Proposition~\\ref{prop:radius} that, for all $n\\in\\mathbb{N}$, $x_n\\in \\inte\\mathbb{B}(x^*;r)$ and \n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad \\|x_{n+1}-x^*\\|^2+\\frac{1-\\lambda_n}{\\lambda_n}\\|x_n-x_{n+1}\\|^2\\leq \\|x_n-x^*\\|^2\n\\end{equation*}\nwith convention that $\\frac{1-\\lambda_n}{\\lambda_n} =0$ if $\\lambda_n =0$.\nBy the definition of $T$, there is a sequence of indices $(i_n)_{n\\in\\mathbb{N}}\\subseteq I$ such that\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1} =(1-\\lambda_n)x_n+\\lambda_n T_{i_n}(x_n)\\quad\\text{and}\\quad i_n\\in \\varphi(x_n)\\subseteq\\varphi(x^*),\n\\end{equation*}\nwhere the last inclusion is a consequence of Proposition~\\ref{prop:prop2.1 Tam17} and the fact that $(x_n)_{n\\in\\mathbb{N}}\\subseteq\\inte\\mathbb{B}(x^*;r)$.\n\nLet $I^*$ denote the set of admissible indices in the sequence $(i_n)_{n\\in\\mathbb{N}}$. Then $I^*\\subseteq \\varphi(x^*)$ which,  together with Proposition~\\ref{prop:fixed points}\\ref{it:fixed points_StrFix} applied to $x^*\\in \\StrFix T$, yields\n\\begin{equation*}\nx^*\\in \\bigcap_{i\\in\\varphi(x*)}\\Fix T_i\\subseteq \\bigcap_{i\\in I^*}\\Fix T_i.\n\\end{equation*}\nThat is, $\\{T_i\\}_{i\\in I^*}$ is a collection of nonexpansive operators with a common fixed point. By applying Theorem~\\ref{thm:KM iterations with admissible control}, $x_n\\to\\overline{x}\\in\\cap_{i\\in I^*}\\Fix T_i$. Since, for all $n\\in\\mathbb{N}$, $x_n\\in \\mathbb{B}(x^*;r)$, it also holds that $\\overline{x}\\in \\mathbb{B}(x^*;r)$. Finally, for any $i\\in I^*$, there exists a subsequence $i_{k_n}\\to i$. Since $x_{k_n}\\to\\overline{x}$ and $\\varphi$ is osc with $i_{k_n}\\in\\varphi(x_{n_k})$, we deduce that $i\\in\\varphi(\\overline{x})$. By Proposition~\\ref{prop:fixed points}\\ref{it:fixed points_Fix},  it follows that $\\overline{x}\\in\\Fix T$ as was claimed.\n\\end{proof}\n\n\\begin{proof}\nIt follows from Proposition~\\ref{prop:averaged} that $R :=(1-1/\\alpha)\\Id+(1/\\alpha)T$ is union nonexpansive. Under the fixed representation of $T$ in \\eqref{eq:a particular uan op}, we have $r(x^*;R) =r(x^*;T)$. Moreover, $\\StrFix R =\\StrFix T$, $\\Fix R =\\Fix T$, and, for all $n\\in\\mathbb{N}$, $x_{n+1} =(1-\\alpha\\lambda_n)x_n+\\alpha\\lambda_n R(x)$. The result now follows from Theorem~\\ref{thm:union} applied to $R$ and $(\\alpha\\lambda_n)_{n\\in\\mathbb{N}}$. \n\\end{proof}","units":[{"raw_tex":"\\begin{proposition}[Equivalent characterizations of union averaged nonexpansiveness]\n\\label{prop:averaged}\nLet $T\\colon X\\setto X$ be a set-valued operator and let $\\alpha\\in(0,1)$. The following assertions are equivalent.\n\\begin{enumerate}[label =(\\alph*)]\n\\item $T$ is union $\\alpha$-averaged nonexpansive.\n\\item\\label{it:prop averaged 2} $T =(1-\\alpha)\\Id+\\alpha R$ for some union nonexpansive operator $R\\colon X\\setto X$.\n\\item $(1-1/\\alpha)\\Id+(1/\\alpha)T$ is union nonexpansive.\n\\end{enumerate}\n\\end{proposition}","start":16083,"end":16602},{"raw_tex":"\\begin{proof}\nFollows by combining Definition~\\ref{def:union averaged} with Proposition~\\ref{prop:AN characterizations}.\n\\end{proof}","start":16603,"end":16735},{"raw_tex":"\\begin{proposition}[Active selectors, fixed points, single-valuedness]\\label{prop:fixed points}\nLet $T\\colon X\\setto X$ be a set-valued operator given by\n\\begin{equation*}\n\\forall x\\in X,\\quad T(x) =\\{T_i(x): i\\in \\varphi(x)\\}\n\\end{equation*}\nwhere $I$ is a finite index set, $\\varphi\\colon X\\setto I$ is an osc operator with nonempty values, and $\\{T_i\\}_{i\\in I}$ is a collection of single-valued operators on $X$. The following assertions hold.\n\\begin{enumerate}[label =(\\alph*)]\n\\item\\label{it:fixed points_X} For each $i\\in I$, the set $\\varphi^{-1}(i)$ is closed. The space $X$ can be represented as\n\\begin{equation}\n\\label{eq:X is a union}\nX =\\bigcup_{i\\in I}\\varphi^{-1}(i).\n\\end{equation}    \n\\item\\label{it:fixed points_Fix} $x\\in\\Fix T$ if and only if there exists $i\\in I$ such that $x\\in\\varphi^{-1}(i)\\cap\\Fix T_i$. Consequently, we have\n\\begin{equation*}\n\\Fix T =\\bigcup_{i\\in I}\\left[\\varphi^{-1}(i)\\cap\\Fix T_i\\right]. \n\\end{equation*}\n\\item\\label{it:fixed points_StrFix} $x\\in\\StrFix T$ if and only if \n\\begin{equation*}\nx\\in \\bigcap_{i\\in\\varphi(x)}\\Fix T_i =\\bigcap_{i\\in\\varphi(x)} \\left[\\varphi^{-1}(i)\\cap\\Fix T_i\\right].\n\\end{equation*}\n\\item\\label{it:fixed points_StrFixSing} The strong fixed point set is the intersection of the fixed point set and the single-valued set, that is,\n\\begin{equation*}\n\\StrFix T =\\Fix T\\cap \\Sing T.\n\\end{equation*}\n\\item\\label{it:fixed points_Sing} If in addition $\\{T_i\\}_{i\\in I}$ is a collection of continuous operators, then $T$ is ocs. Consequently,\n\\begin{equation*}\n\\Sing T =\\left\\{x\\in X:\\Limsup_{x'\\to x}T(x')\\text{ is singleton}\\right\\}.\n\\end{equation*}\n\\end{enumerate}\n\\end{proposition}","start":23862,"end":25516},{"raw_tex":"\\begin{proof}\n\\ref{it:fixed points_X}: Let $i\\in I$ and consider a sequence $(x_n)_{n\\in\\mathbb{N}}\\subseteq\\varphi^{-1}(i)$ such that $x_n\\to\\overline{x}$. The definition of the inverse implies that $i\\in\\varphi(x_n)$ for all $n\\in\\mathbb{N}$ and, since $\\varphi$ is osc, it holds that $i\\in \\varphi(\\overline{x})$, which shows that $\\varphi^{-1}(i)$ is closed and proves the first claim. The fact that \\eqref{eq:X is a union} follows from the assumption that $\\varphi(x)\\neq \\varnothing$ for all $x\\in X$.\n\n\\ref{it:fixed points_Fix}: $x\\in\\Fix T\\iff\\exists i\\in \\varphi(x)$ such that $x\\in\\Fix T_i\\iff\\exists i\\in I$ such that $x\\in\\varphi^{-1}(i)\\cap\\Fix T_i.$ \n\n\\ref{it:fixed points_StrFix}: $x\\in\\StrFix T\\iff x\\in\\Fix T_i$ for all $i\\in\\varphi(x)\\iff x\\in\\varphi^{-1}(i)\\cap\\Fix T_i$ for all $i\\in\\varphi(x)$.\n\n\\ref{it:fixed points_StrFixSing}: Immediate from the respective definitions.\n\n\\ref{it:fixed points_Sing}: Fix $x\\in X$. We first show that $T$ is osc. To this end, take $x_n\\to x$ and $y_n\\to y$ with $y_n\\in T(x_n)$. Then the definition of $T$ ensures the existence of sequence $i_n\\in \\varphi(x_n)$ such that $y_n =T_{i_n}(x_n)\\in T(x_n)$. Using the pigeonhole princple, we pass to a subsequence so that $y_{k_n} =T_i(x_{k_n})$ for fixed $i\\in I$. The osc of $\\varphi$ implies that $i\\in\\varphi(x)$ and continuity of $T_i$ gives\n\\begin{equation*}\ny =\\lim_{n\\to\\infty}y_{k_n} =\\lim_{n\\to\\infty}T_i(x_{k_n}) =T_i(x)\\in T(x),\n\\end{equation*}\nwhich proves osc. The claimed formula for the singleton set follows since $x\\in\\Sing T\\iff T(x)$ is singleton, and osc implies that $T(x) =\\Limsup_{x'\\to x}T(x')$. \n\\end{proof}","start":25517,"end":27150},{"raw_tex":"\\begin{theorem}[Krasnosel'ski\\u{\\i}--Mann iterations with admissible control]\n\\label{thm:KM iterations with admissible control}\nLet $I$ be a finite index set and let $\\{T_i\\}_{i\\in I}$ be a collection of nonexpansive operators on $X$ with a common fixed point. Define a sequence $(x_n)_{n\\in\\mathbb{N}}$ with starting point $x_0\\in X$ according to\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1} :=(1-\\lambda_n)x_n+\\lambda_n T_{i_n}(x_n),\n\\end{equation*}\nwhere $(i_n)_{n\\in\\mathbb{N}}$ is admissible in $I$, and $(\\lambda_n)_{n\\in\\mathbb{N}}$ is in $(0, 1]$ with $\\liminf_{n\\to\\infty}\\lambda_n(1-\\lambda_n)> 0$.\nThen $(x_n)_{n\\in\\mathbb{N}}$ converges to a point $\\overline{x}\\in\\cap_{i\\in I} \\Fix T_i$.\n\\end{theorem}","start":28810,"end":29534},{"raw_tex":"\\begin{proof}\nSince $\\liminf_{n\\to\\infty}\\lambda_n(1-\\lambda_n)> 0$, there exists $n_0\\in \\mathbb{N}$ such that $\\inf_{n\\geq n_0}\\lambda_n(1-\\lambda_n)> 0$. By relabeling if necessary, we may assume without loss of generality that $\\varepsilon :=\\inf_{n\\in\\mathbb{N}}\\lambda_n(1-\\lambda_n)> 0$. \nLet $x\\in \\cap_{i\\in I} \\Fix T_i$ be arbitrary. Then, for all $n\\in\\mathbb{N}$, nonexpansiveness of $T_{i_n}$ yields\n\\begin{subequations} \n\\begin{align*}\n\\|x_{n+1}-x\\|^2& =\\|(1-\\lambda_n)(x_n-x)+\\lambda_n(T_{i_n}(x_n)-x)\\|^2 \\\\\n& =(1-\\lambda_n)\\|x_n-x\\|^2+\\lambda_n\\|T_{i_n}(x_n)-x\\|^2-\\lambda_n(1-\\lambda_n)\\|x_n-T_{i_n}(x_n)\\|^2 \\\\\n&\\leq \\|x_n-x\\|^2-\\varepsilon\\|x_n-T_{i_n}(x_n)\\|^2.\n\\end{align*}\n\\end{subequations}\nIt follows that $(\\|x_n-x\\|^2)_{n\\in\\mathbb{N}}$ is monotone nonincreasing, and hence convergent. Consequently, the sequence $(x_n)_{n\\in\\mathbb{N}}$ is bounded and \n\\begin{equation}\\label{eq:pre-asymp}\nx_n-T_{i_n}(x_n)\\to 0 \\quad\\text{as~} n\\to\\infty.\n\\end{equation}\nNow, let $\\overline{x}\\in X$ be a cluster point of $(x_n)_{n\\in\\mathbb{N}}$. Then $(x_n)_{n\\in\\mathbb{N}}$ contains a convergent subsequence, say $(x_{k_n})_{n\\in\\mathbb{N}}$, with limit $\\overline{x}$. \nWe claim that $\\overline{x}$ is an element of $\\cap_{i\\in I} \\Fix T_i$. To this end suppose, by way of a contradiction, that $\\overline{x}\\not\\in\\cap_{i\\in I} \\Fix T_i$.\nBy passing to a further subsequence if necessary, we deduce that the sequence $(t_n)_{n\\in\\mathbb{N}}$ defined by\n\\begin{equation*}\nt_n :=\\min\\{p\\in\\{k_n,\\dots,k_{n+1}-1\\}:\\overline{x}\\not\\in\\Fix T_{i_p}\\}, \n\\end{equation*}\nis well defined.\nSince $I$ is finite, by passing to yet another subsequence if necessary, we may assume that $i_{t_n} =\\ell$ for all $n\\in\\mathbb{N}$ for some fixed index $\\ell\\in I$. \nTogether with \\eqref{eq:pre-asymp}, we deduce that\n\\begin{equation*}\nx_{t_n}\\to \\overline{x} \\text{~~~and~~~} (\\Id-T_\\ell)(x_{t_n})\\to 0 \\quad\\text{as~} n\\to\\infty,\n\\end{equation*}\nwhich, due to nonexpansiveness of $T_\\ell$, implies that $\\overline{x}\\in\\Fix T_\\ell$. \nThus a contradiction is obtained, and conclude that $\\overline{x}\\in\\cap_{i\\in I} \\Fix T_i$. We then have that $(\\|x_n-\\overline{x}\\|)_{n\\in\\mathbb{N}}$ is monotone nonincreasing and $\\|x_{k_n}-\\overline{x}\\|\\to 0$, and thus the conclusion follows.\n\\end{proof}","start":29535,"end":31830},{"raw_tex":"\\begin{proposition}[Radius of attraction at strong fixed points]\\label{prop:radius}\nLet $T\\colon X\\setto X$ be a union nonexpansive operator with $x^*\\in\\StrFix T$. Then $r :=r(x^*;T)\\in (0,+\\infty]$ and\n\\begin{equation}\\label{eq:attraction nonexpansive}\n\\forall x\\in \\inte\\mathbb{B}(x^*;r),\\,\\forall y\\in T(x),\\quad \n\\|y-x^*\\|\\leq \\|x-x^*\\|.\n\\end{equation}\nFurthermore, for any $\\lambda\\in (0, 1]$, we have\n\\begin{equation}\\label{eq:attraction averaged}\n\\forall x\\in \\inte\\mathbb{B}(x^*;r),\\,\\forall y\\in (1-\\lambda)x+\\lambda T(x),\\quad \n\\|y-x^*\\|^2+\\frac{1-\\lambda}{\\lambda}\\|x-y\\|^2\\leq \\|x-x^*\\|^2.\n\\end{equation}\n\\end{proposition}","start":35256,"end":35891},{"raw_tex":"\\begin{proof}\nSince $\\varphi$ is osc and its range, $I$, is a finite set, Proposition~\\ref{prop:prop2.1 Tam17} implies that $r :=r(x^*;T)\\in(0,+\\infty]$. Since $x^*\\in\\StrFix T$, Proposition~\\ref{prop:fixed points}\\ref{it:fixed points_StrFix} implies that $x^*\\in\\cap_{i\\in\\varphi(x*)}\\Fix T_i$ and, since each $T_i$ is nonexpansive, we have\n\\begin{equation}\n\\label{eq:nonexpansive}\n\\forall i\\in\\varphi(x^*),\\, \\forall x\\in X,\\quad \\|T_i(x)-x^*\\|^2\\leq \\|x-x^*\\|^2.\n\\end{equation}\nIn particular, for any $x\\in \\inte\\mathbb{B}(x^*;r)$ and $y\\in T(x)$, there exists an $i\\in \\varphi(x)\\subseteq \\varphi(x^*)$ such that $y =T_i(x)$. Consequently, \\eqref{eq:attraction nonexpansive} follows from \\eqref{eq:nonexpansive}.\n\nFurthermore, for any $\\lambda\\in (0, 1]$, the operator $S_i :=(1-\\lambda)\\Id+\\lambda T_i$ is $\\lambda$-averaged nonexpansive by Proposition~\\ref{prop:averaged} and we therefore have that\n\\begin{equation}\n\\label{eq:averaged}\n\\forall i\\in\\varphi(x^*),\\, \\forall x\\in X,\\quad \\|S_i(x)-x^*\\|^2+\\frac{1-\\lambda}{\\lambda}\\|x-S_i(x)\\|^2\\leq \\|x-x^*\\|^2.\n\\end{equation}\nIn particular, if $x\\in\\inte\\mathbb{B}(x^*;r)$ and $y\\in (1-\\lambda)x+\\lambda T(x)$, then there exists $i\\in \\varphi(x)\\subseteq \\varphi(x^*)$ such that $y =S_i(x)$. As before, \\eqref{eq:attraction averaged} follows from \\eqref{eq:averaged}.\n\\end{proof}","start":35892,"end":37225},{"raw_tex":"\\begin{theorem}[Local convergence of union nonexpansive iterations]\\label{thm:union}\nLet $T\\colon X\\setto X$ be a union nonexpansive operator with $x^*\\in\\StrFix T$ and let $(\\lambda_n)_{n\\in\\mathbb{N}}$ be a sequence in $(0, 1]$ with $\\liminf_{n\\to\\infty}\\lambda_n(1-\\lambda_n)> 0$. Denote $r :=r(x^*;T)\\in(0,+\\infty]$ and consider a sequence $(x_n)_{n\\in\\mathbb{N}}$ with $x_0\\in \\inte\\mathbb{B}(x^*;r)$ satisfying\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1}\\in (1-\\lambda_n)x_n+\\lambda_n T(x_n).\n\\end{equation*} \nThen $(x_n)_{n\\in\\mathbb{N}}$ converges to a point $\\overline{x}\\in\\Fix T\\cap \\mathbb{B}(x^*;r)$. \n\\end{theorem}","start":37416,"end":38055},{"raw_tex":"\\begin{proof}\nWe first observe from Proposition~\\ref{prop:radius} that, for all $n\\in\\mathbb{N}$, $x_n\\in \\inte\\mathbb{B}(x^*;r)$ and \n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad \\|x_{n+1}-x^*\\|^2+\\frac{1-\\lambda_n}{\\lambda_n}\\|x_n-x_{n+1}\\|^2\\leq \\|x_n-x^*\\|^2\n\\end{equation*}\nwith convention that $\\frac{1-\\lambda_n}{\\lambda_n} =0$ if $\\lambda_n =0$.\nBy the definition of $T$, there is a sequence of indices $(i_n)_{n\\in\\mathbb{N}}\\subseteq I$ such that\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1} =(1-\\lambda_n)x_n+\\lambda_n T_{i_n}(x_n)\\quad\\text{and}\\quad i_n\\in \\varphi(x_n)\\subseteq\\varphi(x^*),\n\\end{equation*}\nwhere the last inclusion is a consequence of Proposition~\\ref{prop:prop2.1 Tam17} and the fact that $(x_n)_{n\\in\\mathbb{N}}\\subseteq\\inte\\mathbb{B}(x^*;r)$.\n\nLet $I^*$ denote the set of admissible indices in the sequence $(i_n)_{n\\in\\mathbb{N}}$. Then $I^*\\subseteq \\varphi(x^*)$ which,  together with Proposition~\\ref{prop:fixed points}\\ref{it:fixed points_StrFix} applied to $x^*\\in \\StrFix T$, yields\n\\begin{equation*}\nx^*\\in \\bigcap_{i\\in\\varphi(x*)}\\Fix T_i\\subseteq \\bigcap_{i\\in I^*}\\Fix T_i.\n\\end{equation*}\nThat is, $\\{T_i\\}_{i\\in I^*}$ is a collection of nonexpansive operators with a common fixed point. By applying Theorem~\\ref{thm:KM iterations with admissible control}, $x_n\\to\\overline{x}\\in\\cap_{i\\in I^*}\\Fix T_i$. Since, for all $n\\in\\mathbb{N}$, $x_n\\in \\mathbb{B}(x^*;r)$, it also holds that $\\overline{x}\\in \\mathbb{B}(x^*;r)$. Finally, for any $i\\in I^*$, there exists a subsequence $i_{k_n}\\to i$. Since $x_{k_n}\\to\\overline{x}$ and $\\varphi$ is osc with $i_{k_n}\\in\\varphi(x_{n_k})$, we deduce that $i\\in\\varphi(\\overline{x})$. By Proposition~\\ref{prop:fixed points}\\ref{it:fixed points_Fix},  it follows that $\\overline{x}\\in\\Fix T$ as was claimed.\n\\end{proof}","start":38056,"end":39874},{"raw_tex":"\\begin{proof}\nIt follows from Proposition~\\ref{prop:averaged} that $R :=(1-1/\\alpha)\\Id+(1/\\alpha)T$ is union nonexpansive. Under the fixed representation of $T$ in \\eqref{eq:a particular uan op}, we have $r(x^*;R) =r(x^*;T)$. Moreover, $\\StrFix R =\\StrFix T$, $\\Fix R =\\Fix T$, and, for all $n\\in\\mathbb{N}$, $x_{n+1} =(1-\\alpha\\lambda_n)x_n+\\alpha\\lambda_n R(x)$. The result now follows from Theorem~\\ref{thm:union} applied to $R$ and $(\\alpha\\lambda_n)_{n\\in\\mathbb{N}}$. \n\\end{proof}","start":40559,"end":41046}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"有限维实希尔伯特空间","source_units":[{"raw_tex":"In this section, we recall the necessary preliminaries for the subsequent sections. Unless stated otherwise, throughout this work we assume that\n\\begin{equation*}\n\\fcolorbox{gray!30}{gray!30}{  X\\text{ is a (real) finite-dimensional Hilbert space}  }\n\\end{equation*}\nwith inner-product $\\langle\\cdot,\\cdot\\rangle$ and induced norm $\\|\\cdot\\|$.\n","start":7427,"end":7771}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":["def:averaged"],"declared_external":false,"source_only":false,"title":"非扩张算子","source_units":[{"raw_tex":"\\begin{definition}[Averaged nonexpansive operators]\n\\label{def:averaged}\nA single-valued operator $T\\colon X\\to X$ is said to be \\emph{nonexpansive} if \n\\begin{equation*}\n\\forall x,y\\in X,\\quad \\|Tx-Ty\\|\\leq \\|x-y\\|,\n\\end{equation*}\n","start":8045,"end":8278}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"平均非扩张算子","source_units":[{"raw_tex":"and \\emph{$\\alpha$-averaged nonexpansive} if $\\alpha\\in(0,1)$ and\n\\begin{equation*}\n\\forall x,y\\in X,\\quad \\|Tx-Ty\\|^2 +\\frac{1-\\alpha}{\\alpha}\\|(\\Id-T)x-(\\Id-T)y\\|^2 \\leq \\|x-y\\|^2.\n\\end{equation*}\nWe say $T$ is \\emph{averaged nonexpansive} if there is an $\\alpha\\in(0,1)$ such that $T$ is $\\alpha$-averaged nonexpansive.\n\\end{definition}    \n","start":8278,"end":8622}]},{"id":"S004","semantic_type":"proposition","kind":"proposition","labels":["prop:AN characterizations"],"declared_external":false,"source_only":false,"title":"平均性的仿射刻画","source_units":[{"raw_tex":"\\begin{proposition}[Characterizations of averaged nonexpansiveness]\n\\label{prop:AN characterizations}\nLet $T\\colon X\\to X$ be an operator and let $\\alpha\\in(0,1)$. The following assertions are equivalent.\n\\begin{enumerate}[label =(\\alph*)]\n\\item $T$ is $\\alpha$-averaged nonexpansive.\n\\item $T =(1-\\alpha)\\Id+\\alpha R$ for some nonexpansive operator $R\\colon X\\to X$.\n\\item $(1-1/\\alpha)\\Id+(1/\\alpha)T$ is nonexpansive.\n\\end{enumerate}\n\\end{proposition}","start":8820,"end":9274}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"外半连续与序列外极限","source_units":[{"raw_tex":"\\begin{definition}[Outer semicontinuity]\nLet $X$ and $Y$ be Hilbert spaces. A set-valued map $\\phi\\colon X\\setto Y$ is \\emph{outer semicontinuous (osc)} at $\\bar{x}$ if\n\\begin{equation*}\n\\phi(\\bar{x}) \\supseteq \\Limsup_{x\\to\\bar{x}}\\phi(x) :=\\{y\\in Y:\\exists\\,x_n\\to \\bar{x},\\,\\exists\\,y_n\\to y\\text{ with }y_n\\in \\phi(x_n)\\}. \n\\end{equation*}  \nThat is, the limit supremum is understood in the sense of the Painlev\\'e--Kuratowski outer limit on $X\\times Y$.\t\n\\end{definition}\n","start":11302,"end":11779}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["def:union averaged","eq:def union averaged"],"declared_external":false,"source_only":false,"title":"并平均算子与活动选择器","source_units":[{"raw_tex":"\\begin{definition}[Union averaged nonexpansive operators]\\label{def:union averaged}\nA set-valued operator $T\\colon X\\setto X$ is said to be \\emph{union $\\alpha$-averaged nonexpansive} (resp.\\ \\emph{union nonexpansive}) if $T$ can be expressed in the form\n\\begin{equation}\\label{eq:def union averaged}\n\\forall x\\in X,\\quad T(x) =\\{T_i(x): i\\in\\varphi(x)\\},\n\\end{equation}\nwhere $I$ is a finite index set, $\\{T_i\\}_{i\\in I}$ is a collection of $\\alpha$-averaged nonexpansive (resp.\\ nonexpansive) operators on $X$, and $\\varphi\\colon X\\setto I$, called an \\emph{active selector}, is an osc operator with nonempty values.\n\nAs before, we say $T$ is \\emph{union averaged nonexpansive} if there is an $\\alpha\\in(0,1)$ such that $T$ is union $\\alpha$-averaged nonexpansive.\n\\end{definition}\n","start":13786,"end":14570}]},{"id":"S007","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"集合值运算约定","source_units":[{"raw_tex":"The following proposition is the union averaged nonexpansive analogue of Proposition~\\ref{prop:AN characterizations} and offers equivalent characterizations of union averaged nonexpansiveness. In what follows, the sum of two or more sets is understood in the sense of the \\emph{Minkowski sum}.\n","start":15789,"end":16083}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"不动点与强不动点","source_units":[{"raw_tex":"For set-valued operators such as those introduced in Definition~\\ref{def:union averaged}, we distinguish two different notions for fixed points which are both the same in the single-valued case. The \\emph{fixed point set} of $T$ is denoted by $\\Fix T :=\\{x:x\\in T(x)\\}$, and the \\emph{strong fixed point set} of $T$ is given by $\\StrFix T :=\\{x:T(x) =\\{x\\}\\}$. \n\n\n","start":23098,"end":23462}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"单值点集","source_units":[{"raw_tex":"In the following proposition, we take a closer look the structure of a class of set-valued operators which includes union averaged nonexpansive operators as a special case. Given a set-valued operator $T\\colon X\\setto X$, its \\emph{single-valued set}, denoted by \n\\begin{equation*}\n\\Sing T :=\\{x\\in X: T(x)\\text{~is a singleton}\\},\n\\end{equation*}\nis the set of points at which $T$ is single-valued.\n","start":23462,"end":23862}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"容许指标序列","source_units":[{"raw_tex":"The condition that each element in $I$ must appear infinitely often in a sequence is $(i_n)_{n\\in\\mathbb{N}}\\subseteq I$ is generalized in the following definition.\n\\begin{definition}[Admissible sequences]\nLet $I$ and $I^*$ be nonempty finite sets with $I^*\\subseteq I$. A sequence $(i_n)_{n\\in\\mathbb{N}}\\subseteq I$ is \\emph{admissible in $I^*$} if every element of $I^*$ appears  infinitely often in $(i_n)_{n\\in\\mathbb{N}}$.\n\\end{definition}\n","start":28137,"end":28583}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","labels":["eq:a particular uan op"],"declared_external":false,"source_only":false,"title":"固定分支表示","source_units":[{"raw_tex":"Using the results of the previous subsection, we now turn our attention convergence of iterations based on union nonexpansive operators. Throughout this subsection, we fix a particular representation for the considered operator. Let $T\\colon X\\setto X$ denote a union nonexpansive (resp.\\ union $\\alpha$-averaged nonexpansive) operator which we assume to be represented as\n\\begin{equation}\\label{eq:a particular uan op}\nT(x) =\\{T_i(x):i\\in\\varphi(x)\\},\n\\end{equation}\nwhere $I$ is a finite index set, $\\{T_i\\}_{i\\in I}$ is a collection of nonexpansive (resp.\\ $\\alpha$-averaged nonexpansive) operators on $X$, and $\\varphi\\colon X\\setto I$ is the osc, nonempty-valued active selector. Fixing this representation is convenient because, in general, the representation of a union nonexpansive (resp.\\ union $\\alpha$-averaged nonexpansive) operator need not be unique and allows us to avoid repetition.\n\n","start":33037,"end":33937}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["eq:radius of attraction"],"declared_external":false,"source_only":false,"title":"表示相关的吸引半径","source_units":[{"raw_tex":"Corresponding to the representation \\eqref{eq:a particular uan op}, we define the \\emph{radius of attraction} of $T$ at a point $x^*\\in X$ as\n\\begin{equation}\\label{eq:radius of attraction}\nr(x^*;T) :=\\sup\\{\\delta>0: \\forall x\\in \\mathbb{B}(x^*;\\delta),\\ \\varphi(x)\\subseteq\\varphi(x^*)\\}\n\\end{equation}\n","start":33937,"end":34241}]},{"id":"S013","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"吸引半径为正","source_units":[{"raw_tex":"Here we note that radius of attraction is nonzero for a union nonexpansive operator (and hence too for a union averaged nonexpansive operator). In fact, we have that\n\\begin{equation*}\n\\forall x^*\\in X,\\quad r(x^*;T)\\in (0,+\\infty]\n\\end{equation*}\nas is shown in the following proposition.\n","start":34241,"end":34530}]},{"id":"S014","semantic_type":"proposition","kind":"proposition","labels":["prop:prop2.1 Tam17"],"declared_external":false,"source_only":false,"title":"有限值选择器的局部包含","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:prop2.1 Tam17}\nLet $\\varphi\\colon X\\setto I$ for a finite set $I$ and let $x^*\\in X$. Then $\\varphi$ is outer semicontinuous at $x^*$ if and only if there exists $\\delta>0$ such that\n\\begin{equation*}\n\\forall x\\in\\mathbb{B}(x^*;\\delta),\\quad \\varphi(x)\\subseteq\\varphi(x^*).\n\\end{equation*}\nConsequently, if $T\\colon X\\setto X$ is a union nonexpansive operator, then $r(x^*;T)\\in (0,+\\infty]$. \n\\end{proposition}","start":34531,"end":34974}]},{"id":"S015","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisites：Bauschke and Combettes (2017), Definition 4.33 and Proposition 4.35；原文第144行引用。"],"source_only":false,"title":"单值平均性的外部刻画","source_units":[]},{"id":"S016","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"单值刻画的引文证明","source_units":[{"raw_tex":"\\begin{proof}\nFollows by combining \\cite[Definition~4.33 and Proposition~4.35]{BauCom17}.\n\\end{proof}\n","start":9275,"end":9377}]},{"id":"S017","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisites：Tam (2018), Proposition 1；原文第472行以键 Tam17 引用，参考文献第1294–1296行标明2018年出版。"],"source_only":false,"title":"有限选择器的外部命题","source_units":[]},{"id":"S018","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"局部包含的引文证明","source_units":[{"raw_tex":"\\begin{proof}\nThis follows from \\cite[Proposition~1]{Tam17}.\n\\end{proof}\n","start":34975,"end":35048}]},{"id":"S019","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisites：Standard Hilbert-space squared-norm identities；作者在第407–409行直接使用，未另列引文。"],"source_only":false,"title":"平方范数恒等式","source_units":[]},{"id":"S020","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisites：Standard finite-dimensional compactness；原文第412–416行使用。"],"source_only":false,"title":"有限维有界序列的聚点","source_units":[]},{"id":"S021","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisites：Standard finite-index subsequence arguments；有限尾部排除是 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"有限指标的子列与尾部事实","source_units":[]},{"id":"S022","semantic_type":"proposition","kind":"proposition","labels":["it:prop averaged 2","prop:averaged"],"declared_external":false,"source_only":false,"title":"并平均性的仿射刻画","source_units":[{"raw_tex":"\\begin{proposition}[Equivalent characterizations of union averaged nonexpansiveness]\n\\label{prop:averaged}\nLet $T\\colon X\\setto X$ be a set-valued operator and let $\\alpha\\in(0,1)$. The following assertions are equivalent.\n\\begin{enumerate}[label =(\\alph*)]\n\\item $T$ is union $\\alpha$-averaged nonexpansive.\n\\item\\label{it:prop averaged 2} $T =(1-\\alpha)\\Id+\\alpha R$ for some union nonexpansive operator $R\\colon X\\setto X$.\n\\item $(1-1/\\alpha)\\Id+(1/\\alpha)T$ is union nonexpansive.\n\\end{enumerate}\n\\end{proposition}","start":16083,"end":16602}]},{"id":"S023","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"并平均刻画的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nFollows by combining Definition~\\ref{def:union averaged} with Proposition~\\ref{prop:AN characterizations}.\n\\end{proof}","start":16603,"end":16735}]},{"id":"S024","semantic_type":"assumption","kind":"assumption","labels":["it:fixed points_Fix","it:fixed points_Sing","it:fixed points_StrFix","it:fixed points_StrFixSing","it:fixed points_X","prop:fixed points"],"declared_external":false,"source_only":false,"title":"一般有限分支表示的假设","source_units":[{"raw_tex":"\\begin{proposition}[Active selectors, fixed points, single-valuedness]\\label{prop:fixed points}\nLet $T\\colon X\\setto X$ be a set-valued operator given by\n\\begin{equation*}\n\\forall x\\in X,\\quad T(x) =\\{T_i(x): i\\in \\varphi(x)\\}\n\\end{equation*}\nwhere $I$ is a finite index set, $\\varphi\\colon X\\setto I$ is an osc operator with nonempty values, and $\\{T_i\\}_{i\\in I}$ is a collection of single-valued operators on $X$. The following assertions hold.\n\\begin{enumerate}[label =(\\alph*)]\n","start":23862,"end":24345}]},{"id":"S025","semantic_type":"proposition","kind":"proposition","labels":["eq:X is a union","it:fixed points_Fix","it:fixed points_Sing","it:fixed points_StrFix","it:fixed points_StrFixSing","it:fixed points_X","prop:fixed points"],"declared_external":false,"source_only":false,"title":"活动区域闭合并覆盖空间","source_units":[{"raw_tex":"\\item\\label{it:fixed points_X} For each $i\\in I$, the set $\\varphi^{-1}(i)$ is closed. The space $X$ can be represented as\n\\begin{equation}\n\\label{eq:X is a union}\nX =\\bigcup_{i\\in I}\\varphi^{-1}(i).\n\\end{equation}    \n","start":24345,"end":24564}]},{"id":"S026","semantic_type":"proposition","kind":"proposition","labels":["it:fixed points_Fix","it:fixed points_Sing","it:fixed points_StrFix","it:fixed points_StrFixSing","it:fixed points_X","prop:fixed points"],"declared_external":false,"source_only":false,"title":"不动点的活动分支刻画","source_units":[{"raw_tex":"\\item\\label{it:fixed points_Fix} $x\\in\\Fix T$ if and only if there exists $i\\in I$ such that $x\\in\\varphi^{-1}(i)\\cap\\Fix T_i$. Consequently, we have\n\\begin{equation*}\n\\Fix T =\\bigcup_{i\\in I}\\left[\\varphi^{-1}(i)\\cap\\Fix T_i\\right]. \n\\end{equation*}\n","start":24564,"end":24815}]},{"id":"S027","semantic_type":"proposition","kind":"proposition","labels":["it:fixed points_Fix","it:fixed points_Sing","it:fixed points_StrFix","it:fixed points_StrFixSing","it:fixed points_X","prop:fixed points"],"declared_external":false,"source_only":false,"title":"强不动点的共同分支刻画","source_units":[{"raw_tex":"\\item\\label{it:fixed points_StrFix} $x\\in\\StrFix T$ if and only if \n\\begin{equation*}\nx\\in \\bigcap_{i\\in\\varphi(x)}\\Fix T_i =\\bigcap_{i\\in\\varphi(x)} \\left[\\varphi^{-1}(i)\\cap\\Fix T_i\\right].\n\\end{equation*}\n","start":24815,"end":25023}]},{"id":"S028","semantic_type":"proposition","kind":"proposition","labels":["it:fixed points_Fix","it:fixed points_Sing","it:fixed points_StrFix","it:fixed points_StrFixSing","it:fixed points_X","prop:fixed points"],"declared_external":false,"source_only":false,"title":"强不动点与单值性的交集","source_units":[{"raw_tex":"\\item\\label{it:fixed points_StrFixSing} The strong fixed point set is the intersection of the fixed point set and the single-valued set, that is,\n\\begin{equation*}\n\\StrFix T =\\Fix T\\cap \\Sing T.\n\\end{equation*}\n","start":25023,"end":25234}]},{"id":"S029","semantic_type":"proposition","kind":"proposition","labels":["it:fixed points_Fix","it:fixed points_Sing","it:fixed points_StrFix","it:fixed points_StrFixSing","it:fixed points_X","prop:fixed points"],"declared_external":false,"source_only":false,"title":"连续分支的外半连续性","source_units":[{"raw_tex":"\\item\\label{it:fixed points_Sing} If in addition $\\{T_i\\}_{i\\in I}$ is a collection of continuous operators, then $T$ is ocs. Consequently,\n\\begin{equation*}\n\\Sing T =\\left\\{x\\in X:\\Limsup_{x'\\to x}T(x')\\text{ is singleton}\\right\\}.\n\\end{equation*}\n\\end{enumerate}\n\\end{proposition}","start":25234,"end":25516}]},{"id":"S030","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"分支结构命题的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{it:fixed points_X}: Let $i\\in I$ and consider a sequence $(x_n)_{n\\in\\mathbb{N}}\\subseteq\\varphi^{-1}(i)$ such that $x_n\\to\\overline{x}$. The definition of the inverse implies that $i\\in\\varphi(x_n)$ for all $n\\in\\mathbb{N}$ and, since $\\varphi$ is osc, it holds that $i\\in \\varphi(\\overline{x})$, which shows that $\\varphi^{-1}(i)$ is closed and proves the first claim. The fact that \\eqref{eq:X is a union} follows from the assumption that $\\varphi(x)\\neq \\varnothing$ for all $x\\in X$.\n\n\\ref{it:fixed points_Fix}: $x\\in\\Fix T\\iff\\exists i\\in \\varphi(x)$ such that $x\\in\\Fix T_i\\iff\\exists i\\in I$ such that $x\\in\\varphi^{-1}(i)\\cap\\Fix T_i.$ \n\n\\ref{it:fixed points_StrFix}: $x\\in\\StrFix T\\iff x\\in\\Fix T_i$ for all $i\\in\\varphi(x)\\iff x\\in\\varphi^{-1}(i)\\cap\\Fix T_i$ for all $i\\in\\varphi(x)$.\n\n\\ref{it:fixed points_StrFixSing}: Immediate from the respective definitions.\n\n\\ref{it:fixed points_Sing}: Fix $x\\in X$. We first show that $T$ is osc. To this end, take $x_n\\to x$ and $y_n\\to y$ with $y_n\\in T(x_n)$. Then the definition of $T$ ensures the existence of sequence $i_n\\in \\varphi(x_n)$ such that $y_n =T_{i_n}(x_n)\\in T(x_n)$. Using the pigeonhole princple, we pass to a subsequence so that $y_{k_n} =T_i(x_{k_n})$ for fixed $i\\in I$. The osc of $\\varphi$ implies that $i\\in\\varphi(x)$ and continuity of $T_i$ gives\n\\begin{equation*}\ny =\\lim_{n\\to\\infty}y_{k_n} =\\lim_{n\\to\\infty}T_i(x_{k_n}) =T_i(x)\\in T(x),\n\\end{equation*}\nwhich proves osc. The claimed formula for the singleton set follows since $x\\in\\Sing T\\iff T(x)$ is singleton, and osc implies that $T(x) =\\Limsup_{x'\\to x}T(x')$. \n\\end{proof}","start":25517,"end":27150}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"由选择器外半连续得到闭性","source_units":[{"raw_tex":"\\ref{it:fixed points_X}: Let $i\\in I$ and consider a sequence $(x_n)_{n\\in\\mathbb{N}}\\subseteq\\varphi^{-1}(i)$ such that $x_n\\to\\overline{x}$. The definition of the inverse implies that $i\\in\\varphi(x_n)$ for all $n\\in\\mathbb{N}$ and, since $\\varphi$ is osc, it holds that $i\\in \\varphi(\\overline{x})$, which shows that $\\varphi^{-1}(i)$ is closed and proves the first claim. The fact that \\eqref{eq:X is a union} follows from the assumption that $\\varphi(x)\\neq \\varnothing$ for all $x\\in X$.\n","start":25531,"end":26025}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"展开普通不动点定义","source_units":[{"raw_tex":"\\ref{it:fixed points_Fix}: $x\\in\\Fix T\\iff\\exists i\\in \\varphi(x)$ such that $x\\in\\Fix T_i\\iff\\exists i\\in I$ such that $x\\in\\varphi^{-1}(i)\\cap\\Fix T_i.$ \n","start":26026,"end":26182}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"展开强不动点定义","source_units":[{"raw_tex":"\\ref{it:fixed points_StrFix}: $x\\in\\StrFix T\\iff x\\in\\Fix T_i$ for all $i\\in\\varphi(x)\\iff x\\in\\varphi^{-1}(i)\\cap\\Fix T_i$ for all $i\\in\\varphi(x)$.\n","start":26183,"end":26333}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"由定义得到单值交集关系","source_units":[{"raw_tex":"\\ref{it:fixed points_StrFixSing}: Immediate from the respective definitions.\n","start":26334,"end":26411}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"固定分支子列传递极限","source_units":[{"raw_tex":"\\ref{it:fixed points_Sing}: Fix $x\\in X$. We first show that $T$ is osc. To this end, take $x_n\\to x$ and $y_n\\to y$ with $y_n\\in T(x_n)$. Then the definition of $T$ ensures the existence of sequence $i_n\\in \\varphi(x_n)$ such that $y_n =T_{i_n}(x_n)\\in T(x_n)$. Using the pigeonhole princple, we pass to a subsequence so that $y_{k_n} =T_i(x_{k_n})$ for fixed $i\\in I$. The osc of $\\varphi$ implies that $i\\in\\varphi(x)$ and continuity of $T_i$ gives\n\\begin{equation*}\ny =\\lim_{n\\to\\infty}y_{k_n} =\\lim_{n\\to\\infty}T_i(x_{k_n}) =T_i(x)\\in T(x),\n\\end{equation*}\nwhich proves osc. The claimed formula for the singleton set follows since $x\\in\\Sing T\\iff T(x)$ is singleton, and osc implies that $T(x) =\\Limsup_{x'\\to x}T(x')$. \n","start":26412,"end":27139}]},{"id":"S036","semantic_type":"theorem","kind":"theorem","labels":["thm:KM iterations with admissible control"],"declared_external":false,"source_only":false,"title":"容许控制下的 KM 收敛","source_units":[{"raw_tex":"\\begin{theorem}[Krasnosel'ski\\u{\\i}--Mann iterations with admissible control]\n\\label{thm:KM iterations with admissible control}\nLet $I$ be a finite index set and let $\\{T_i\\}_{i\\in I}$ be a collection of nonexpansive operators on $X$ with a common fixed point. Define a sequence $(x_n)_{n\\in\\mathbb{N}}$ with starting point $x_0\\in X$ according to\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1} :=(1-\\lambda_n)x_n+\\lambda_n T_{i_n}(x_n),\n\\end{equation*}\nwhere $(i_n)_{n\\in\\mathbb{N}}$ is admissible in $I$, and $(\\lambda_n)_{n\\in\\mathbb{N}}$ is in $(0, 1]$ with $\\liminf_{n\\to\\infty}\\lambda_n(1-\\lambda_n)> 0$.\nThen $(x_n)_{n\\in\\mathbb{N}}$ converges to a point $\\overline{x}\\in\\cap_{i\\in I} \\Fix T_i$.\n\\end{theorem}","start":28810,"end":29534}]},{"id":"S037","semantic_type":"assumption","kind":"assumption","labels":["thm:KM iterations with admissible control"],"declared_external":false,"source_only":false,"title":"KM 算子族与迭代条件","source_units":[{"raw_tex":"Let $I$ be a finite index set and let $\\{T_i\\}_{i\\in I}$ be a collection of nonexpansive operators on $X$ with a common fixed point. Define a sequence $(x_n)_{n\\in\\mathbb{N}}$ with starting point $x_0\\in X$ according to\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1} :=(1-\\lambda_n)x_n+\\lambda_n T_{i_n}(x_n),\n\\end{equation*}\nwhere $(i_n)_{n\\in\\mathbb{N}}$ is admissible in $I$, and $(\\lambda_n)_{n\\in\\mathbb{N}}$ is in $(0, 1]$ with $\\liminf_{n\\to\\infty}\\lambda_n(1-\\lambda_n)> 0$.\n","start":28938,"end":29429}]},{"id":"S038","semantic_type":"proof","kind":"proof","labels":["eq:pre-asymp"],"declared_external":false,"source_only":false,"title":"KM 收敛的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nSince $\\liminf_{n\\to\\infty}\\lambda_n(1-\\lambda_n)> 0$, there exists $n_0\\in \\mathbb{N}$ such that $\\inf_{n\\geq n_0}\\lambda_n(1-\\lambda_n)> 0$. By relabeling if necessary, we may assume without loss of generality that $\\varepsilon :=\\inf_{n\\in\\mathbb{N}}\\lambda_n(1-\\lambda_n)> 0$. \nLet $x\\in \\cap_{i\\in I} \\Fix T_i$ be arbitrary. Then, for all $n\\in\\mathbb{N}$, nonexpansiveness of $T_{i_n}$ yields\n\\begin{subequations} \n\\begin{align*}\n\\|x_{n+1}-x\\|^2& =\\|(1-\\lambda_n)(x_n-x)+\\lambda_n(T_{i_n}(x_n)-x)\\|^2 \\\\\n& =(1-\\lambda_n)\\|x_n-x\\|^2+\\lambda_n\\|T_{i_n}(x_n)-x\\|^2-\\lambda_n(1-\\lambda_n)\\|x_n-T_{i_n}(x_n)\\|^2 \\\\\n&\\leq \\|x_n-x\\|^2-\\varepsilon\\|x_n-T_{i_n}(x_n)\\|^2.\n\\end{align*}\n\\end{subequations}\nIt follows that $(\\|x_n-x\\|^2)_{n\\in\\mathbb{N}}$ is monotone nonincreasing, and hence convergent. Consequently, the sequence $(x_n)_{n\\in\\mathbb{N}}$ is bounded and \n\\begin{equation}\\label{eq:pre-asymp}\nx_n-T_{i_n}(x_n)\\to 0 \\quad\\text{as~} n\\to\\infty.\n\\end{equation}\nNow, let $\\overline{x}\\in X$ be a cluster point of $(x_n)_{n\\in\\mathbb{N}}$. Then $(x_n)_{n\\in\\mathbb{N}}$ contains a convergent subsequence, say $(x_{k_n})_{n\\in\\mathbb{N}}$, with limit $\\overline{x}$. \nWe claim that $\\overline{x}$ is an element of $\\cap_{i\\in I} \\Fix T_i$. To this end suppose, by way of a contradiction, that $\\overline{x}\\not\\in\\cap_{i\\in I} \\Fix T_i$.\nBy passing to a further subsequence if necessary, we deduce that the sequence $(t_n)_{n\\in\\mathbb{N}}$ defined by\n\\begin{equation*}\nt_n :=\\min\\{p\\in\\{k_n,\\dots,k_{n+1}-1\\}:\\overline{x}\\not\\in\\Fix T_{i_p}\\}, \n\\end{equation*}\nis well defined.\nSince $I$ is finite, by passing to yet another subsequence if necessary, we may assume that $i_{t_n} =\\ell$ for all $n\\in\\mathbb{N}$ for some fixed index $\\ell\\in I$. \nTogether with \\eqref{eq:pre-asymp}, we deduce that\n\\begin{equation*}\nx_{t_n}\\to \\overline{x} \\text{~~~and~~~} (\\Id-T_\\ell)(x_{t_n})\\to 0 \\quad\\text{as~} n\\to\\infty,\n\\end{equation*}\nwhich, due to nonexpansiveness of $T_\\ell$, implies that $\\overline{x}\\in\\Fix T_\\ell$. \nThus a contradiction is obtained, and conclude that $\\overline{x}\\in\\cap_{i\\in I} \\Fix T_i$. We then have that $(\\|x_n-\\overline{x}\\|)_{n\\in\\mathbb{N}}$ is monotone nonincreasing and $\\|x_{k_n}-\\overline{x}\\|\\to 0$, and thus the conclusion follows.\n\\end{proof}","start":29535,"end":31830}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"删除前缀取得统一参数界","source_units":[{"raw_tex":"Since $\\liminf_{n\\to\\infty}\\lambda_n(1-\\lambda_n)> 0$, there exists $n_0\\in \\mathbb{N}$ such that $\\inf_{n\\geq n_0}\\lambda_n(1-\\lambda_n)> 0$. By relabeling if necessary, we may assume without loss of generality that $\\varepsilon :=\\inf_{n\\in\\mathbb{N}}\\lambda_n(1-\\lambda_n)> 0$. \n","start":29549,"end":29831}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"共同不动点的平方距离下降","source_units":[{"raw_tex":"Let $x\\in \\cap_{i\\in I} \\Fix T_i$ be arbitrary. Then, for all $n\\in\\mathbb{N}$, nonexpansiveness of $T_{i_n}$ yields\n\\begin{subequations} \n\\begin{align*}\n\\|x_{n+1}-x\\|^2& =\\|(1-\\lambda_n)(x_n-x)+\\lambda_n(T_{i_n}(x_n)-x)\\|^2 \\\\\n& =(1-\\lambda_n)\\|x_n-x\\|^2+\\lambda_n\\|T_{i_n}(x_n)-x\\|^2-\\lambda_n(1-\\lambda_n)\\|x_n-T_{i_n}(x_n)\\|^2 \\\\\n&\\leq \\|x_n-x\\|^2-\\varepsilon\\|x_n-T_{i_n}(x_n)\\|^2.\n\\end{align*}\n\\end{subequations}\n","start":29831,"end":30250}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pre-asymp"],"declared_external":false,"source_only":false,"title":"有界性与分支残差消失","source_units":[{"raw_tex":"It follows that $(\\|x_n-x\\|^2)_{n\\in\\mathbb{N}}$ is monotone nonincreasing, and hence convergent. Consequently, the sequence $(x_n)_{n\\in\\mathbb{N}}$ is bounded and \n\\begin{equation}\\label{eq:pre-asymp}\nx_n-T_{i_n}(x_n)\\to 0 \\quad\\text{as~} n\\to\\infty.\n\\end{equation}\n","start":30250,"end":30518}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"聚点反证与首个违例指标","source_units":[{"raw_tex":"Now, let $\\overline{x}\\in X$ be a cluster point of $(x_n)_{n\\in\\mathbb{N}}$. Then $(x_n)_{n\\in\\mathbb{N}}$ contains a convergent subsequence, say $(x_{k_n})_{n\\in\\mathbb{N}}$, with limit $\\overline{x}$. \nWe claim that $\\overline{x}$ is an element of $\\cap_{i\\in I} \\Fix T_i$. To this end suppose, by way of a contradiction, that $\\overline{x}\\not\\in\\cap_{i\\in I} \\Fix T_i$.\nBy passing to a further subsequence if necessary, we deduce that the sequence $(t_n)_{n\\in\\mathbb{N}}$ defined by\n\\begin{equation*}\nt_n :=\\min\\{p\\in\\{k_n,\\dots,k_{n+1}-1\\}:\\overline{x}\\not\\in\\Fix T_{i_p}\\}, \n\\end{equation*}\nis well defined.\nSince $I$ is finite, by passing to yet another subsequence if necessary, we may assume that $i_{t_n} =\\ell$ for all $n\\in\\mathbb{N}$ for some fixed index $\\ell\\in I$. \n","start":30518,"end":31301}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"残差极限导致矛盾","source_units":[{"raw_tex":"Together with \\eqref{eq:pre-asymp}, we deduce that\n\\begin{equation*}\nx_{t_n}\\to \\overline{x} \\text{~~~and~~~} (\\Id-T_\\ell)(x_{t_n})\\to 0 \\quad\\text{as~} n\\to\\infty,\n\\end{equation*}\nwhich, due to nonexpansiveness of $T_\\ell$, implies that $\\overline{x}\\in\\Fix T_\\ell$. \n","start":31301,"end":31570}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"由共同聚点得到整列收敛","source_units":[{"raw_tex":"Thus a contradiction is obtained, and conclude that $\\overline{x}\\in\\cap_{i\\in I} \\Fix T_i$. We then have that $(\\|x_n-\\overline{x}\\|)_{n\\in\\mathbb{N}}$ is monotone nonincreasing and $\\|x_{k_n}-\\overline{x}\\|\\to 0$, and thus the conclusion follows.\n","start":31570,"end":31819}]},{"id":"S045","semantic_type":"assumption","kind":"assumption","labels":["prop:radius"],"declared_external":false,"source_only":false,"title":"半径命题的强不动点假设","source_units":[{"raw_tex":"Let $T\\colon X\\setto X$ be a union nonexpansive operator with $x^*\\in\\StrFix T$. Then $r :=r(x^*;T)\\in (0,+\\infty]$ and\n","start":35340,"end":35460}]},{"id":"S046","semantic_type":"proposition","kind":"proposition","labels":["eq:attraction nonexpansive","prop:radius"],"declared_external":false,"source_only":false,"title":"强不动点附近的准非扩张性","source_units":[{"raw_tex":"\\begin{proposition}[Radius of attraction at strong fixed points]\\label{prop:radius}\nLet $T\\colon X\\setto X$ be a union nonexpansive operator with $x^*\\in\\StrFix T$. Then $r :=r(x^*;T)\\in (0,+\\infty]$ and\n\\begin{equation}\\label{eq:attraction nonexpansive}\n\\forall x\\in \\inte\\mathbb{B}(x^*;r),\\,\\forall y\\in T(x),\\quad \n\\|y-x^*\\|\\leq \\|x-x^*\\|.\n\\end{equation}\n","start":35256,"end":35614}]},{"id":"S047","semantic_type":"proposition","kind":"proposition","labels":["eq:attraction averaged","prop:radius"],"declared_external":false,"source_only":false,"title":"松弛迭代的局部下降估计","source_units":[{"raw_tex":"Furthermore, for any $\\lambda\\in (0, 1]$, we have\n\\begin{equation}\\label{eq:attraction averaged}\n\\forall x\\in \\inte\\mathbb{B}(x^*;r),\\,\\forall y\\in (1-\\lambda)x+\\lambda T(x),\\quad \n\\|y-x^*\\|^2+\\frac{1-\\lambda}{\\lambda}\\|x-y\\|^2\\leq \\|x-x^*\\|^2.\n\\end{equation}\n\\end{proposition}","start":35614,"end":35891}]},{"id":"S048","semantic_type":"proof","kind":"proof","labels":["eq:averaged","eq:nonexpansive"],"declared_external":false,"source_only":false,"title":"局部下降性质的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nSince $\\varphi$ is osc and its range, $I$, is a finite set, Proposition~\\ref{prop:prop2.1 Tam17} implies that $r :=r(x^*;T)\\in(0,+\\infty]$. Since $x^*\\in\\StrFix T$, Proposition~\\ref{prop:fixed points}\\ref{it:fixed points_StrFix} implies that $x^*\\in\\cap_{i\\in\\varphi(x*)}\\Fix T_i$ and, since each $T_i$ is nonexpansive, we have\n\\begin{equation}\n\\label{eq:nonexpansive}\n\\forall i\\in\\varphi(x^*),\\, \\forall x\\in X,\\quad \\|T_i(x)-x^*\\|^2\\leq \\|x-x^*\\|^2.\n\\end{equation}\nIn particular, for any $x\\in \\inte\\mathbb{B}(x^*;r)$ and $y\\in T(x)$, there exists an $i\\in \\varphi(x)\\subseteq \\varphi(x^*)$ such that $y =T_i(x)$. Consequently, \\eqref{eq:attraction nonexpansive} follows from \\eqref{eq:nonexpansive}.\n\nFurthermore, for any $\\lambda\\in (0, 1]$, the operator $S_i :=(1-\\lambda)\\Id+\\lambda T_i$ is $\\lambda$-averaged nonexpansive by Proposition~\\ref{prop:averaged} and we therefore have that\n\\begin{equation}\n\\label{eq:averaged}\n\\forall i\\in\\varphi(x^*),\\, \\forall x\\in X,\\quad \\|S_i(x)-x^*\\|^2+\\frac{1-\\lambda}{\\lambda}\\|x-S_i(x)\\|^2\\leq \\|x-x^*\\|^2.\n\\end{equation}\nIn particular, if $x\\in\\inte\\mathbb{B}(x^*;r)$ and $y\\in (1-\\lambda)x+\\lambda T(x)$, then there exists $i\\in \\varphi(x)\\subseteq \\varphi(x^*)$ such that $y =S_i(x)$. As before, \\eqref{eq:attraction averaged} follows from \\eqref{eq:averaged}.\n\\end{proof}","start":35892,"end":37225}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["eq:nonexpansive"],"declared_external":false,"source_only":false,"title":"活动分支的共同参考不动点","source_units":[{"raw_tex":"Since $\\varphi$ is osc and its range, $I$, is a finite set, Proposition~\\ref{prop:prop2.1 Tam17} implies that $r :=r(x^*;T)\\in(0,+\\infty]$. Since $x^*\\in\\StrFix T$, Proposition~\\ref{prop:fixed points}\\ref{it:fixed points_StrFix} implies that $x^*\\in\\cap_{i\\in\\varphi(x*)}\\Fix T_i$ and, since each $T_i$ is nonexpansive, we have\n\\begin{equation}\n\\label{eq:nonexpansive}\n\\forall i\\in\\varphi(x^*),\\, \\forall x\\in X,\\quad \\|T_i(x)-x^*\\|^2\\leq \\|x-x^*\\|^2.\n\\end{equation}\n","start":35906,"end":36373}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"将分支距离估计传给集合取值","source_units":[{"raw_tex":"In particular, for any $x\\in \\inte\\mathbb{B}(x^*;r)$ and $y\\in T(x)$, there exists an $i\\in \\varphi(x)\\subseteq \\varphi(x^*)$ such that $y =T_i(x)$. Consequently, \\eqref{eq:attraction nonexpansive} follows from \\eqref{eq:nonexpansive}.\n","start":36373,"end":36609}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":["eq:averaged"],"declared_external":false,"source_only":false,"title":"松弛分支的平均性估计","source_units":[{"raw_tex":"Furthermore, for any $\\lambda\\in (0, 1]$, the operator $S_i :=(1-\\lambda)\\Id+\\lambda T_i$ is $\\lambda$-averaged nonexpansive by Proposition~\\ref{prop:averaged} and we therefore have that\n\\begin{equation}\n\\label{eq:averaged}\n\\forall i\\in\\varphi(x^*),\\, \\forall x\\in X,\\quad \\|S_i(x)-x^*\\|^2+\\frac{1-\\lambda}{\\lambda}\\|x-S_i(x)\\|^2\\leq \\|x-x^*\\|^2.\n\\end{equation}\n","start":36610,"end":36972}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"将松弛分支估计传给迭代","source_units":[{"raw_tex":"In particular, if $x\\in\\inte\\mathbb{B}(x^*;r)$ and $y\\in (1-\\lambda)x+\\lambda T(x)$, then there exists $i\\in \\varphi(x)\\subseteq \\varphi(x^*)$ such that $y =S_i(x)$. As before, \\eqref{eq:attraction averaged} follows from \\eqref{eq:averaged}.\n","start":36972,"end":37214}]},{"id":"S053","semantic_type":"theorem","kind":"theorem","labels":["thm:union"],"declared_external":false,"source_only":false,"title":"并非扩张迭代的局部收敛","source_units":[{"raw_tex":"\\begin{theorem}[Local convergence of union nonexpansive iterations]\\label{thm:union}\nLet $T\\colon X\\setto X$ be a union nonexpansive operator with $x^*\\in\\StrFix T$ and let $(\\lambda_n)_{n\\in\\mathbb{N}}$ be a sequence in $(0, 1]$ with $\\liminf_{n\\to\\infty}\\lambda_n(1-\\lambda_n)> 0$. Denote $r :=r(x^*;T)\\in(0,+\\infty]$ and consider a sequence $(x_n)_{n\\in\\mathbb{N}}$ with $x_0\\in \\inte\\mathbb{B}(x^*;r)$ satisfying\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1}\\in (1-\\lambda_n)x_n+\\lambda_n T(x_n).\n\\end{equation*} \nThen $(x_n)_{n\\in\\mathbb{N}}$ converges to a point $\\overline{x}\\in\\Fix T\\cap \\mathbb{B}(x^*;r)$. \n\\end{theorem}","start":37416,"end":38055}]},{"id":"S054","semantic_type":"assumption","kind":"assumption","labels":["thm:union"],"declared_external":false,"source_only":false,"title":"并非扩张迭代的输入条件","source_units":[{"raw_tex":"Let $T\\colon X\\setto X$ be a union nonexpansive operator with $x^*\\in\\StrFix T$ and let $(\\lambda_n)_{n\\in\\mathbb{N}}$ be a sequence in $(0, 1]$ with $\\liminf_{n\\to\\infty}\\lambda_n(1-\\lambda_n)> 0$. Denote $r :=r(x^*;T)\\in(0,+\\infty]$ and consider a sequence $(x_n)_{n\\in\\mathbb{N}}$ with $x_0\\in \\inte\\mathbb{B}(x^*;r)$ satisfying\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1}\\in (1-\\lambda_n)x_n+\\lambda_n T(x_n).\n\\end{equation*} \n","start":37501,"end":37943}]},{"id":"S055","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"并非扩张局部收敛的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nWe first observe from Proposition~\\ref{prop:radius} that, for all $n\\in\\mathbb{N}$, $x_n\\in \\inte\\mathbb{B}(x^*;r)$ and \n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad \\|x_{n+1}-x^*\\|^2+\\frac{1-\\lambda_n}{\\lambda_n}\\|x_n-x_{n+1}\\|^2\\leq \\|x_n-x^*\\|^2\n\\end{equation*}\nwith convention that $\\frac{1-\\lambda_n}{\\lambda_n} =0$ if $\\lambda_n =0$.\nBy the definition of $T$, there is a sequence of indices $(i_n)_{n\\in\\mathbb{N}}\\subseteq I$ such that\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1} =(1-\\lambda_n)x_n+\\lambda_n T_{i_n}(x_n)\\quad\\text{and}\\quad i_n\\in \\varphi(x_n)\\subseteq\\varphi(x^*),\n\\end{equation*}\nwhere the last inclusion is a consequence of Proposition~\\ref{prop:prop2.1 Tam17} and the fact that $(x_n)_{n\\in\\mathbb{N}}\\subseteq\\inte\\mathbb{B}(x^*;r)$.\n\nLet $I^*$ denote the set of admissible indices in the sequence $(i_n)_{n\\in\\mathbb{N}}$. Then $I^*\\subseteq \\varphi(x^*)$ which,  together with Proposition~\\ref{prop:fixed points}\\ref{it:fixed points_StrFix} applied to $x^*\\in \\StrFix T$, yields\n\\begin{equation*}\nx^*\\in \\bigcap_{i\\in\\varphi(x*)}\\Fix T_i\\subseteq \\bigcap_{i\\in I^*}\\Fix T_i.\n\\end{equation*}\nThat is, $\\{T_i\\}_{i\\in I^*}$ is a collection of nonexpansive operators with a common fixed point. By applying Theorem~\\ref{thm:KM iterations with admissible control}, $x_n\\to\\overline{x}\\in\\cap_{i\\in I^*}\\Fix T_i$. Since, for all $n\\in\\mathbb{N}$, $x_n\\in \\mathbb{B}(x^*;r)$, it also holds that $\\overline{x}\\in \\mathbb{B}(x^*;r)$. Finally, for any $i\\in I^*$, there exists a subsequence $i_{k_n}\\to i$. Since $x_{k_n}\\to\\overline{x}$ and $\\varphi$ is osc with $i_{k_n}\\in\\varphi(x_{n_k})$, we deduce that $i\\in\\varphi(\\overline{x})$. By Proposition~\\ref{prop:fixed points}\\ref{it:fixed points_Fix},  it follows that $\\overline{x}\\in\\Fix T$ as was claimed.\n\\end{proof}","start":38056,"end":39874}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"吸引球内部的不变性","source_units":[{"raw_tex":"We first observe from Proposition~\\ref{prop:radius} that, for all $n\\in\\mathbb{N}$, $x_n\\in \\inte\\mathbb{B}(x^*;r)$ and \n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad \\|x_{n+1}-x^*\\|^2+\\frac{1-\\lambda_n}{\\lambda_n}\\|x_n-x_{n+1}\\|^2\\leq \\|x_n-x^*\\|^2\n\\end{equation*}\nwith convention that $\\frac{1-\\lambda_n}{\\lambda_n} =0$ if $\\lambda_n =0$.\n","start":38070,"end":38412}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"选择活动分支指标","source_units":[{"raw_tex":"By the definition of $T$, there is a sequence of indices $(i_n)_{n\\in\\mathbb{N}}\\subseteq I$ such that\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1} =(1-\\lambda_n)x_n+\\lambda_n T_{i_n}(x_n)\\quad\\text{and}\\quad i_n\\in \\varphi(x_n)\\subseteq\\varphi(x^*),\n\\end{equation*}\nwhere the last inclusion is a consequence of Proposition~\\ref{prop:prop2.1 Tam17} and the fact that $(x_n)_{n\\in\\mathbb{N}}\\subseteq\\inte\\mathbb{B}(x^*;r)$.\n","start":38412,"end":38846}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"常返分支共享强参考不动点","source_units":[{"raw_tex":"Let $I^*$ denote the set of admissible indices in the sequence $(i_n)_{n\\in\\mathbb{N}}$. Then $I^*\\subseteq \\varphi(x^*)$ which,  together with Proposition~\\ref{prop:fixed points}\\ref{it:fixed points_StrFix} applied to $x^*\\in \\StrFix T$, yields\n\\begin{equation*}\nx^*\\in \\bigcap_{i\\in\\varphi(x*)}\\Fix T_i\\subseteq \\bigcap_{i\\in I^*}\\Fix T_i.\n\\end{equation*}\n","start":38847,"end":39205}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"对常返分支应用 KM 定理","source_units":[{"raw_tex":"That is, $\\{T_i\\}_{i\\in I^*}$ is a collection of nonexpansive operators with a common fixed point. By applying Theorem~\\ref{thm:KM iterations with admissible control}, $x_n\\to\\overline{x}\\in\\cap_{i\\in I^*}\\Fix T_i$. Since, for all $n\\in\\mathbb{N}$, $x_n\\in \\mathbb{B}(x^*;r)$, it also holds that $\\overline{x}\\in \\mathbb{B}(x^*;r)$. Finally, for any $i\\in I^*$, there exists a subsequence $i_{k_n}\\to i$. Since $x_{k_n}\\to\\overline{x}$ and $\\varphi$ is osc with $i_{k_n}\\in\\varphi(x_{n_k})$, we deduce that $i\\in\\varphi(\\overline{x})$. By Proposition~\\ref{prop:fixed points}\\ref{it:fixed points_Fix},  it follows that $\\overline{x}\\in\\Fix T$ as was claimed.\n","start":39205,"end":39863}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"极限活动性与集合值不动点","source_units":[{"raw_tex":"That is, $\\{T_i\\}_{i\\in I^*}$ is a collection of nonexpansive operators with a common fixed point. By applying Theorem~\\ref{thm:KM iterations with admissible control}, $x_n\\to\\overline{x}\\in\\cap_{i\\in I^*}\\Fix T_i$. Since, for all $n\\in\\mathbb{N}$, $x_n\\in \\mathbb{B}(x^*;r)$, it also holds that $\\overline{x}\\in \\mathbb{B}(x^*;r)$. Finally, for any $i\\in I^*$, there exists a subsequence $i_{k_n}\\to i$. Since $x_{k_n}\\to\\overline{x}$ and $\\varphi$ is osc with $i_{k_n}\\in\\varphi(x_{n_k})$, we deduce that $i\\in\\varphi(\\overline{x})$. By Proposition~\\ref{prop:fixed points}\\ref{it:fixed points_Fix},  it follows that $\\overline{x}\\in\\Fix T$ as was claimed.\n","start":39205,"end":39863}]},{"id":"S061","semantic_type":"corollary","kind":"corollary","labels":["cor:union averaged"],"declared_external":false,"source_only":false,"title":"并平均迭代的局部收敛","source_units":[{"raw_tex":"\\begin{corollary}[Local convergence of union averaged nonexpansive iterations]\\label{cor:union averaged}\nLet $T\\colon X\\setto X$ be union $\\alpha$-averaged nonexpansive operator with $x^*\\in\\StrFix T$ and let $(\\lambda_n)_{n\\in\\mathbb{N}}$ be a sequence in $(0, 1/\\alpha]$ with $\\liminf_{n\\to\\infty} \\lambda_n(1/\\alpha-\\lambda_n)> 0$. Denote $r :=r(x^*;T)\\in(0,+\\infty]$ and consider $(x_n)_{n\\in\\mathbb{N}}$ with $x_0\\in \\inte\\mathbb{B}(x^*;r)$ satisfying\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1}\\in (1-\\lambda_n)x_n+\\lambda_n T(x_n).\n\\end{equation*} \nThen $(x_n)_{n\\in\\mathbb{N}}$ converges to a point $\\overline{x}\\in\\Fix T\\cap \\mathbb{B}(x^*;r)$. \n\\end{corollary}","start":39877,"end":40558}]},{"id":"S062","semantic_type":"assumption","kind":"assumption","labels":["cor:union averaged"],"declared_external":false,"source_only":false,"title":"主推论的局部与参数条件","source_units":[{"raw_tex":"Let $T\\colon X\\setto X$ be union $\\alpha$-averaged nonexpansive operator with $x^*\\in\\StrFix T$ and let $(\\lambda_n)_{n\\in\\mathbb{N}}$ be a sequence in $(0, 1/\\alpha]$ with $\\liminf_{n\\to\\infty} \\lambda_n(1/\\alpha-\\lambda_n)> 0$. Denote $r :=r(x^*;T)\\in(0,+\\infty]$ and consider $(x_n)_{n\\in\\mathbb{N}}$ with $x_0\\in \\inte\\mathbb{B}(x^*;r)$ satisfying\n","start":39982,"end":40334}]},{"id":"S063","semantic_type":"algorithm","kind":"algorithm","labels":["cor:union averaged"],"declared_external":false,"source_only":false,"title":"集合值松弛迭代","source_units":[{"raw_tex":"\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1}\\in (1-\\lambda_n)x_n+\\lambda_n T(x_n).\n\\end{equation*} \n","start":40334,"end":40444}]},{"id":"S064","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"并平均局部收敛的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nIt follows from Proposition~\\ref{prop:averaged} that $R :=(1-1/\\alpha)\\Id+(1/\\alpha)T$ is union nonexpansive. Under the fixed representation of $T$ in \\eqref{eq:a particular uan op}, we have $r(x^*;R) =r(x^*;T)$. Moreover, $\\StrFix R =\\StrFix T$, $\\Fix R =\\Fix T$, and, for all $n\\in\\mathbb{N}$, $x_{n+1} =(1-\\alpha\\lambda_n)x_n+\\alpha\\lambda_n R(x)$. The result now follows from Theorem~\\ref{thm:union} applied to $R$ and $(\\alpha\\lambda_n)_{n\\in\\mathbb{N}}$. \n\\end{proof}","start":40559,"end":41046}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"变换为并非扩张算子","source_units":[{"raw_tex":"It follows from Proposition~\\ref{prop:averaged} that $R :=(1-1/\\alpha)\\Id+(1/\\alpha)T$ is union nonexpansive. Under the fixed representation of $T$ in \\eqref{eq:a particular uan op}, we have $r(x^*;R) =r(x^*;T)$. Moreover, $\\StrFix R =\\StrFix T$, $\\Fix R =\\Fix T$, and, for all $n\\in\\mathbb{N}$, $x_{n+1} =(1-\\alpha\\lambda_n)x_n+\\alpha\\lambda_n R(x)$. The result now follows from Theorem~\\ref{thm:union} applied to $R$ and $(\\alpha\\lambda_n)_{n\\in\\mathbb{N}}$. \n","start":40573,"end":41035}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"保留选择器与吸引半径","source_units":[{"raw_tex":"It follows from Proposition~\\ref{prop:averaged} that $R :=(1-1/\\alpha)\\Id+(1/\\alpha)T$ is union nonexpansive. Under the fixed representation of $T$ in \\eqref{eq:a particular uan op}, we have $r(x^*;R) =r(x^*;T)$. Moreover, $\\StrFix R =\\StrFix T$, $\\Fix R =\\Fix T$, and, for all $n\\in\\mathbb{N}$, $x_{n+1} =(1-\\alpha\\lambda_n)x_n+\\alpha\\lambda_n R(x)$. The result now follows from Theorem~\\ref{thm:union} applied to $R$ and $(\\alpha\\lambda_n)_{n\\in\\mathbb{N}}$. \n","start":40573,"end":41035}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"保留普通与强不动点","source_units":[{"raw_tex":"It follows from Proposition~\\ref{prop:averaged} that $R :=(1-1/\\alpha)\\Id+(1/\\alpha)T$ is union nonexpansive. Under the fixed representation of $T$ in \\eqref{eq:a particular uan op}, we have $r(x^*;R) =r(x^*;T)$. Moreover, $\\StrFix R =\\StrFix T$, $\\Fix R =\\Fix T$, and, for all $n\\in\\mathbb{N}$, $x_{n+1} =(1-\\alpha\\lambda_n)x_n+\\alpha\\lambda_n R(x)$. The result now follows from Theorem~\\ref{thm:union} applied to $R$ and $(\\alpha\\lambda_n)_{n\\in\\mathbb{N}}$. \n","start":40573,"end":41035}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"重写变换后的松弛递推","source_units":[{"raw_tex":"It follows from Proposition~\\ref{prop:averaged} that $R :=(1-1/\\alpha)\\Id+(1/\\alpha)T$ is union nonexpansive. Under the fixed representation of $T$ in \\eqref{eq:a particular uan op}, we have $r(x^*;R) =r(x^*;T)$. Moreover, $\\StrFix R =\\StrFix T$, $\\Fix R =\\Fix T$, and, for all $n\\in\\mathbb{N}$, $x_{n+1} =(1-\\alpha\\lambda_n)x_n+\\alpha\\lambda_n R(x)$. The result now follows from Theorem~\\ref{thm:union} applied to $R$ and $(\\alpha\\lambda_n)_{n\\in\\mathbb{N}}$. \n","start":40573,"end":41035}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"应用并非扩张局部收敛定理","source_units":[{"raw_tex":"It follows from Proposition~\\ref{prop:averaged} that $R :=(1-1/\\alpha)\\Id+(1/\\alpha)T$ is union nonexpansive. Under the fixed representation of $T$ in \\eqref{eq:a particular uan op}, we have $r(x^*;R) =r(x^*;T)$. Moreover, $\\StrFix R =\\StrFix T$, $\\Fix R =\\Fix T$, and, for all $n\\in\\mathbb{N}$, $x_{n+1} =(1-\\alpha\\lambda_n)x_n+\\alpha\\lambda_n R(x)$. The result now follows from Theorem~\\ref{thm:union} applied to $R$ and $(\\alpha\\lambda_n)_{n\\in\\mathbb{N}}$. \n","start":40573,"end":41035}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S061","edges":[{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S004","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S014","kind":"supports"},{"source_id":"S005","target_id":"S014","kind":"supports"},{"source_id":"S011","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S002","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S002","target_id":"S006","kind":"supports"},{"source_id":"S003","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S006","target_id":"S023","kind":"supports"},{"source_id":"S004","target_id":"S023","kind":"supports"},{"source_id":"S007","target_id":"S022","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S031","kind":"supports"},{"source_id":"S005","target_id":"S031","kind":"supports"},{"source_id":"S024","target_id":"S032","kind":"supports"},{"source_id":"S008","target_id":"S032","kind":"supports"},{"source_id":"S024","target_id":"S033","kind":"supports"},{"source_id":"S008","target_id":"S033","kind":"supports"},{"source_id":"S008","target_id":"S034","kind":"supports"},{"source_id":"S009","target_id":"S034","kind":"supports"},{"source_id":"S024","target_id":"S035","kind":"supports"},{"source_id":"S005","target_id":"S035","kind":"supports"},{"source_id":"S009","target_id":"S035","kind":"supports"},{"source_id":"S021","target_id":"S035","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S030","kind":"supports"},{"source_id":"S033","target_id":"S030","kind":"supports"},{"source_id":"S034","target_id":"S030","kind":"supports"},{"source_id":"S035","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S025","kind":"supports"},{"source_id":"S030","target_id":"S026","kind":"supports"},{"source_id":"S030","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S010","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S002","target_id":"S040","kind":"supports"},{"source_id":"S019","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S001","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S020","target_id":"S042","kind":"supports"},{"source_id":"S037","target_id":"S042","kind":"supports"},{"source_id":"S010","target_id":"S042","kind":"supports"},{"source_id":"S021","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S041","target_id":"S043","kind":"supports"},{"source_id":"S002","target_id":"S043","kind":"supports"},{"source_id":"S040","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S042","target_id":"S044","kind":"supports"},{"source_id":"S040","target_id":"S044","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S041","target_id":"S038","kind":"supports"},{"source_id":"S042","target_id":"S038","kind":"supports"},{"source_id":"S043","target_id":"S038","kind":"supports"},{"source_id":"S044","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S045","target_id":"S049","kind":"supports"},{"source_id":"S011","target_id":"S049","kind":"supports"},{"source_id":"S012","target_id":"S049","kind":"supports"},{"source_id":"S014","target_id":"S049","kind":"supports"},{"source_id":"S027","target_id":"S049","kind":"supports"},{"source_id":"S002","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S011","target_id":"S050","kind":"supports"},{"source_id":"S012","target_id":"S050","kind":"supports"},{"source_id":"S022","target_id":"S051","kind":"supports"},{"source_id":"S049","target_id":"S051","kind":"supports"},{"source_id":"S003","target_id":"S051","kind":"supports"},{"source_id":"S002","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S011","target_id":"S052","kind":"supports"},{"source_id":"S012","target_id":"S052","kind":"supports"},{"source_id":"S007","target_id":"S052","kind":"supports"},{"source_id":"S049","target_id":"S048","kind":"supports"},{"source_id":"S050","target_id":"S048","kind":"supports"},{"source_id":"S051","target_id":"S048","kind":"supports"},{"source_id":"S052","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S054","target_id":"S056","kind":"supports"},{"source_id":"S047","target_id":"S056","kind":"supports"},{"source_id":"S054","target_id":"S057","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S011","target_id":"S057","kind":"supports"},{"source_id":"S012","target_id":"S057","kind":"supports"},{"source_id":"S014","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S054","target_id":"S058","kind":"supports"},{"source_id":"S027","target_id":"S058","kind":"supports"},{"source_id":"S010","target_id":"S058","kind":"supports"},{"source_id":"S021","target_id":"S058","kind":"supports"},{"source_id":"S036","target_id":"S059","kind":"supports"},{"source_id":"S057","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S054","target_id":"S059","kind":"supports"},{"source_id":"S056","target_id":"S059","kind":"supports"},{"source_id":"S021","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S057","target_id":"S060","kind":"supports"},{"source_id":"S058","target_id":"S060","kind":"supports"},{"source_id":"S005","target_id":"S060","kind":"supports"},{"source_id":"S026","target_id":"S060","kind":"supports"},{"source_id":"S021","target_id":"S060","kind":"supports"},{"source_id":"S056","target_id":"S055","kind":"supports"},{"source_id":"S057","target_id":"S055","kind":"supports"},{"source_id":"S058","target_id":"S055","kind":"supports"},{"source_id":"S059","target_id":"S055","kind":"supports"},{"source_id":"S060","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S053","kind":"supports"},{"source_id":"S006","target_id":"S062","kind":"supports"},{"source_id":"S003","target_id":"S062","kind":"supports"},{"source_id":"S008","target_id":"S062","kind":"supports"},{"source_id":"S011","target_id":"S062","kind":"supports"},{"source_id":"S012","target_id":"S062","kind":"supports"},{"source_id":"S007","target_id":"S063","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S022","target_id":"S065","kind":"supports"},{"source_id":"S062","target_id":"S065","kind":"supports"},{"source_id":"S007","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S011","target_id":"S066","kind":"supports"},{"source_id":"S012","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S067","kind":"supports"},{"source_id":"S008","target_id":"S067","kind":"supports"},{"source_id":"S007","target_id":"S067","kind":"supports"},{"source_id":"S062","target_id":"S067","kind":"supports"},{"source_id":"S063","target_id":"S068","kind":"supports"},{"source_id":"S065","target_id":"S068","kind":"supports"},{"source_id":"S062","target_id":"S068","kind":"supports"},{"source_id":"S007","target_id":"S068","kind":"supports"},{"source_id":"S053","target_id":"S069","kind":"supports"},{"source_id":"S065","target_id":"S069","kind":"supports"},{"source_id":"S066","target_id":"S069","kind":"supports"},{"source_id":"S067","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S062","target_id":"S069","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S064","kind":"supports"},{"source_id":"S067","target_id":"S064","kind":"supports"},{"source_id":"S068","target_id":"S064","kind":"supports"},{"source_id":"S069","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S061","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S061","edges":[{"source_id":"S015","target_id":"S004","kind":"supports"},{"source_id":"S017","target_id":"S014","kind":"supports"},{"source_id":"S005","target_id":"S014","kind":"supports"},{"source_id":"S011","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S002","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S002","target_id":"S006","kind":"supports"},{"source_id":"S003","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S006","target_id":"S022","kind":"supports"},{"source_id":"S004","target_id":"S022","kind":"supports"},{"source_id":"S007","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S031","kind":"supports"},{"source_id":"S005","target_id":"S031","kind":"supports"},{"source_id":"S024","target_id":"S032","kind":"supports"},{"source_id":"S008","target_id":"S032","kind":"supports"},{"source_id":"S024","target_id":"S033","kind":"supports"},{"source_id":"S008","target_id":"S033","kind":"supports"},{"source_id":"S008","target_id":"S034","kind":"supports"},{"source_id":"S009","target_id":"S034","kind":"supports"},{"source_id":"S024","target_id":"S035","kind":"supports"},{"source_id":"S005","target_id":"S035","kind":"supports"},{"source_id":"S009","target_id":"S035","kind":"supports"},{"source_id":"S021","target_id":"S035","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S030","kind":"supports"},{"source_id":"S033","target_id":"S030","kind":"supports"},{"source_id":"S034","target_id":"S030","kind":"supports"},{"source_id":"S035","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S025","kind":"supports"},{"source_id":"S030","target_id":"S026","kind":"supports"},{"source_id":"S030","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S010","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S002","target_id":"S036","kind":"supports"},{"source_id":"S019","target_id":"S036","kind":"supports"},{"source_id":"S001","target_id":"S036","kind":"supports"},{"source_id":"S020","target_id":"S036","kind":"supports"},{"source_id":"S010","target_id":"S036","kind":"supports"},{"source_id":"S021","target_id":"S036","kind":"supports"},{"source_id":"S045","target_id":"S049","kind":"supports"},{"source_id":"S011","target_id":"S049","kind":"supports"},{"source_id":"S012","target_id":"S049","kind":"supports"},{"source_id":"S014","target_id":"S049","kind":"supports"},{"source_id":"S027","target_id":"S049","kind":"supports"},{"source_id":"S002","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S011","target_id":"S050","kind":"supports"},{"source_id":"S012","target_id":"S050","kind":"supports"},{"source_id":"S022","target_id":"S051","kind":"supports"},{"source_id":"S049","target_id":"S051","kind":"supports"},{"source_id":"S003","target_id":"S051","kind":"supports"},{"source_id":"S002","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S011","target_id":"S052","kind":"supports"},{"source_id":"S012","target_id":"S052","kind":"supports"},{"source_id":"S007","target_id":"S052","kind":"supports"},{"source_id":"S049","target_id":"S048","kind":"supports"},{"source_id":"S050","target_id":"S048","kind":"supports"},{"source_id":"S051","target_id":"S048","kind":"supports"},{"source_id":"S052","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S047","target_id":"S053","kind":"supports"},{"source_id":"S011","target_id":"S053","kind":"supports"},{"source_id":"S012","target_id":"S053","kind":"supports"},{"source_id":"S014","target_id":"S053","kind":"supports"},{"source_id":"S027","target_id":"S053","kind":"supports"},{"source_id":"S010","target_id":"S053","kind":"supports"},{"source_id":"S021","target_id":"S053","kind":"supports"},{"source_id":"S036","target_id":"S053","kind":"supports"},{"source_id":"S005","target_id":"S053","kind":"supports"},{"source_id":"S026","target_id":"S053","kind":"supports"},{"source_id":"S006","target_id":"S062","kind":"supports"},{"source_id":"S003","target_id":"S062","kind":"supports"},{"source_id":"S008","target_id":"S062","kind":"supports"},{"source_id":"S011","target_id":"S062","kind":"supports"},{"source_id":"S012","target_id":"S062","kind":"supports"},{"source_id":"S007","target_id":"S063","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S022","target_id":"S065","kind":"supports"},{"source_id":"S062","target_id":"S065","kind":"supports"},{"source_id":"S007","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S011","target_id":"S066","kind":"supports"},{"source_id":"S012","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S067","kind":"supports"},{"source_id":"S008","target_id":"S067","kind":"supports"},{"source_id":"S007","target_id":"S067","kind":"supports"},{"source_id":"S062","target_id":"S067","kind":"supports"},{"source_id":"S063","target_id":"S068","kind":"supports"},{"source_id":"S065","target_id":"S068","kind":"supports"},{"source_id":"S062","target_id":"S068","kind":"supports"},{"source_id":"S007","target_id":"S068","kind":"supports"},{"source_id":"S053","target_id":"S069","kind":"supports"},{"source_id":"S065","target_id":"S069","kind":"supports"},{"source_id":"S066","target_id":"S069","kind":"supports"},{"source_id":"S067","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S062","target_id":"S069","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S064","kind":"supports"},{"source_id":"S067","target_id":"S064","kind":"supports"},{"source_id":"S068","target_id":"S064","kind":"supports"},{"source_id":"S069","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S061","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"有限维实希尔伯特空间","source_units":[{"raw_tex":"In this section, we recall the necessary preliminaries for the subsequent sections. Unless stated otherwise, throughout this work we assume that\n\\begin{equation*}\n\\fcolorbox{gray!30}{gray!30}{  X\\text{ is a (real) finite-dimensional Hilbert space}  }\n\\end{equation*}\nwith inner-product $\\langle\\cdot,\\cdot\\rangle$ and induced norm $\\|\\cdot\\|$.\n","start":7427,"end":7771}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":["def:averaged"],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"非扩张算子","source_units":[{"raw_tex":"\\begin{definition}[Averaged nonexpansive operators]\n\\label{def:averaged}\nA single-valued operator $T\\colon X\\to X$ is said to be \\emph{nonexpansive} if \n\\begin{equation*}\n\\forall x,y\\in X,\\quad \\|Tx-Ty\\|\\leq \\|x-y\\|,\n\\end{equation*}\n","start":8045,"end":8278}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"平均非扩张算子","source_units":[{"raw_tex":"and \\emph{$\\alpha$-averaged nonexpansive} if $\\alpha\\in(0,1)$ and\n\\begin{equation*}\n\\forall x,y\\in X,\\quad \\|Tx-Ty\\|^2 +\\frac{1-\\alpha}{\\alpha}\\|(\\Id-T)x-(\\Id-T)y\\|^2 \\leq \\|x-y\\|^2.\n\\end{equation*}\nWe say $T$ is \\emph{averaged nonexpansive} if there is an $\\alpha\\in(0,1)$ such that $T$ is $\\alpha$-averaged nonexpansive.\n\\end{definition}    \n","start":8278,"end":8622}]},{"id":"S004","semantic_type":"proposition","kind":"proposition","labels":["prop:AN characterizations"],"member_ids":["S004","S016"],"declared_external":false,"source_only":false,"title":"平均性的仿射刻画","source_units":[{"raw_tex":"\\begin{proposition}[Characterizations of averaged nonexpansiveness]\n\\label{prop:AN characterizations}\nLet $T\\colon X\\to X$ be an operator and let $\\alpha\\in(0,1)$. The following assertions are equivalent.\n\\begin{enumerate}[label =(\\alph*)]\n\\item $T$ is $\\alpha$-averaged nonexpansive.\n\\item $T =(1-\\alpha)\\Id+\\alpha R$ for some nonexpansive operator $R\\colon X\\to X$.\n\\item $(1-1/\\alpha)\\Id+(1/\\alpha)T$ is nonexpansive.\n\\end{enumerate}\n\\end{proposition}","start":8820,"end":9274},{"raw_tex":"\\begin{proof}\nFollows by combining \\cite[Definition~4.33 and Proposition~4.35]{BauCom17}.\n\\end{proof}\n","start":9275,"end":9377}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"外半连续与序列外极限","source_units":[{"raw_tex":"\\begin{definition}[Outer semicontinuity]\nLet $X$ and $Y$ be Hilbert spaces. A set-valued map $\\phi\\colon X\\setto Y$ is \\emph{outer semicontinuous (osc)} at $\\bar{x}$ if\n\\begin{equation*}\n\\phi(\\bar{x}) \\supseteq \\Limsup_{x\\to\\bar{x}}\\phi(x) :=\\{y\\in Y:\\exists\\,x_n\\to \\bar{x},\\,\\exists\\,y_n\\to y\\text{ with }y_n\\in \\phi(x_n)\\}. \n\\end{equation*}  \nThat is, the limit supremum is understood in the sense of the Painlev\\'e--Kuratowski outer limit on $X\\times Y$.\t\n\\end{definition}\n","start":11302,"end":11779}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["def:union averaged","eq:def union averaged"],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"并平均算子与活动选择器","source_units":[{"raw_tex":"\\begin{definition}[Union averaged nonexpansive operators]\\label{def:union averaged}\nA set-valued operator $T\\colon X\\setto X$ is said to be \\emph{union $\\alpha$-averaged nonexpansive} (resp.\\ \\emph{union nonexpansive}) if $T$ can be expressed in the form\n\\begin{equation}\\label{eq:def union averaged}\n\\forall x\\in X,\\quad T(x) =\\{T_i(x): i\\in\\varphi(x)\\},\n\\end{equation}\nwhere $I$ is a finite index set, $\\{T_i\\}_{i\\in I}$ is a collection of $\\alpha$-averaged nonexpansive (resp.\\ nonexpansive) operators on $X$, and $\\varphi\\colon X\\setto I$, called an \\emph{active selector}, is an osc operator with nonempty values.\n\nAs before, we say $T$ is \\emph{union averaged nonexpansive} if there is an $\\alpha\\in(0,1)$ such that $T$ is union $\\alpha$-averaged nonexpansive.\n\\end{definition}\n","start":13786,"end":14570}]},{"id":"S007","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"集合值运算约定","source_units":[{"raw_tex":"The following proposition is the union averaged nonexpansive analogue of Proposition~\\ref{prop:AN characterizations} and offers equivalent characterizations of union averaged nonexpansiveness. In what follows, the sum of two or more sets is understood in the sense of the \\emph{Minkowski sum}.\n","start":15789,"end":16083}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"不动点与强不动点","source_units":[{"raw_tex":"For set-valued operators such as those introduced in Definition~\\ref{def:union averaged}, we distinguish two different notions for fixed points which are both the same in the single-valued case. The \\emph{fixed point set} of $T$ is denoted by $\\Fix T :=\\{x:x\\in T(x)\\}$, and the \\emph{strong fixed point set} of $T$ is given by $\\StrFix T :=\\{x:T(x) =\\{x\\}\\}$. \n\n\n","start":23098,"end":23462}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"单值点集","source_units":[{"raw_tex":"In the following proposition, we take a closer look the structure of a class of set-valued operators which includes union averaged nonexpansive operators as a special case. Given a set-valued operator $T\\colon X\\setto X$, its \\emph{single-valued set}, denoted by \n\\begin{equation*}\n\\Sing T :=\\{x\\in X: T(x)\\text{~is a singleton}\\},\n\\end{equation*}\nis the set of points at which $T$ is single-valued.\n","start":23462,"end":23862}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"容许指标序列","source_units":[{"raw_tex":"The condition that each element in $I$ must appear infinitely often in a sequence is $(i_n)_{n\\in\\mathbb{N}}\\subseteq I$ is generalized in the following definition.\n\\begin{definition}[Admissible sequences]\nLet $I$ and $I^*$ be nonempty finite sets with $I^*\\subseteq I$. A sequence $(i_n)_{n\\in\\mathbb{N}}\\subseteq I$ is \\emph{admissible in $I^*$} if every element of $I^*$ appears  infinitely often in $(i_n)_{n\\in\\mathbb{N}}$.\n\\end{definition}\n","start":28137,"end":28583}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","labels":["eq:a particular uan op"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"固定分支表示","source_units":[{"raw_tex":"Using the results of the previous subsection, we now turn our attention convergence of iterations based on union nonexpansive operators. Throughout this subsection, we fix a particular representation for the considered operator. Let $T\\colon X\\setto X$ denote a union nonexpansive (resp.\\ union $\\alpha$-averaged nonexpansive) operator which we assume to be represented as\n\\begin{equation}\\label{eq:a particular uan op}\nT(x) =\\{T_i(x):i\\in\\varphi(x)\\},\n\\end{equation}\nwhere $I$ is a finite index set, $\\{T_i\\}_{i\\in I}$ is a collection of nonexpansive (resp.\\ $\\alpha$-averaged nonexpansive) operators on $X$, and $\\varphi\\colon X\\setto I$ is the osc, nonempty-valued active selector. Fixing this representation is convenient because, in general, the representation of a union nonexpansive (resp.\\ union $\\alpha$-averaged nonexpansive) operator need not be unique and allows us to avoid repetition.\n\n","start":33037,"end":33937}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["eq:radius of attraction"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"表示相关的吸引半径","source_units":[{"raw_tex":"Corresponding to the representation \\eqref{eq:a particular uan op}, we define the \\emph{radius of attraction} of $T$ at a point $x^*\\in X$ as\n\\begin{equation}\\label{eq:radius of attraction}\nr(x^*;T) :=\\sup\\{\\delta>0: \\forall x\\in \\mathbb{B}(x^*;\\delta),\\ \\varphi(x)\\subseteq\\varphi(x^*)\\}\n\\end{equation}\n","start":33937,"end":34241}]},{"id":"S013","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"吸引半径为正","source_units":[{"raw_tex":"Here we note that radius of attraction is nonzero for a union nonexpansive operator (and hence too for a union averaged nonexpansive operator). In fact, we have that\n\\begin{equation*}\n\\forall x^*\\in X,\\quad r(x^*;T)\\in (0,+\\infty]\n\\end{equation*}\nas is shown in the following proposition.\n","start":34241,"end":34530}]},{"id":"S014","semantic_type":"proposition","kind":"proposition","labels":["prop:prop2.1 Tam17"],"member_ids":["S014","S018"],"declared_external":false,"source_only":false,"title":"有限值选择器的局部包含","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:prop2.1 Tam17}\nLet $\\varphi\\colon X\\setto I$ for a finite set $I$ and let $x^*\\in X$. Then $\\varphi$ is outer semicontinuous at $x^*$ if and only if there exists $\\delta>0$ such that\n\\begin{equation*}\n\\forall x\\in\\mathbb{B}(x^*;\\delta),\\quad \\varphi(x)\\subseteq\\varphi(x^*).\n\\end{equation*}\nConsequently, if $T\\colon X\\setto X$ is a union nonexpansive operator, then $r(x^*;T)\\in (0,+\\infty]$. \n\\end{proposition}","start":34531,"end":34974},{"raw_tex":"\\begin{proof}\nThis follows from \\cite[Proposition~1]{Tam17}.\n\\end{proof}\n","start":34975,"end":35048}]},{"id":"S015","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S015"],"declared_external":true,"declaration_text":["原始 prerequisites：Bauschke and Combettes (2017), Definition 4.33 and Proposition 4.35；原文第144行引用。"],"source_only":false,"title":"单值平均性的外部刻画","source_units":[]},{"id":"S017","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S017"],"declared_external":true,"declaration_text":["原始 prerequisites：Tam (2018), Proposition 1；原文第472行以键 Tam17 引用，参考文献第1294–1296行标明2018年出版。"],"source_only":false,"title":"有限选择器的外部命题","source_units":[]},{"id":"S019","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S019"],"declared_external":true,"declaration_text":["原始 prerequisites：Standard Hilbert-space squared-norm identities；作者在第407–409行直接使用，未另列引文。"],"source_only":false,"title":"平方范数恒等式","source_units":[]},{"id":"S020","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S020"],"declared_external":true,"declaration_text":["原始 prerequisites：Standard finite-dimensional compactness；原文第412–416行使用。"],"source_only":false,"title":"有限维有界序列的聚点","source_units":[]},{"id":"S021","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S021"],"declared_external":true,"declaration_text":["原始 prerequisites：Standard finite-index subsequence arguments；有限尾部排除是 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"有限指标的子列与尾部事实","source_units":[]},{"id":"S022","semantic_type":"proposition","kind":"proposition","labels":["it:prop averaged 2","prop:averaged"],"member_ids":["S022","S023"],"declared_external":false,"source_only":false,"title":"并平均性的仿射刻画","source_units":[{"raw_tex":"\\begin{proposition}[Equivalent characterizations of union averaged nonexpansiveness]\n\\label{prop:averaged}\nLet $T\\colon X\\setto X$ be a set-valued operator and let $\\alpha\\in(0,1)$. The following assertions are equivalent.\n\\begin{enumerate}[label =(\\alph*)]\n\\item $T$ is union $\\alpha$-averaged nonexpansive.\n\\item\\label{it:prop averaged 2} $T =(1-\\alpha)\\Id+\\alpha R$ for some union nonexpansive operator $R\\colon X\\setto X$.\n\\item $(1-1/\\alpha)\\Id+(1/\\alpha)T$ is union nonexpansive.\n\\end{enumerate}\n\\end{proposition}","start":16083,"end":16602},{"raw_tex":"\\begin{proof}\nFollows by combining Definition~\\ref{def:union averaged} with Proposition~\\ref{prop:AN characterizations}.\n\\end{proof}","start":16603,"end":16735}]},{"id":"S024","semantic_type":"assumption","kind":"assumption","labels":["it:fixed points_Fix","it:fixed points_Sing","it:fixed points_StrFix","it:fixed points_StrFixSing","it:fixed points_X","prop:fixed points"],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"一般有限分支表示的假设","source_units":[{"raw_tex":"\\begin{proposition}[Active selectors, fixed points, single-valuedness]\\label{prop:fixed points}\nLet $T\\colon X\\setto X$ be a set-valued operator given by\n\\begin{equation*}\n\\forall x\\in X,\\quad T(x) =\\{T_i(x): i\\in \\varphi(x)\\}\n\\end{equation*}\nwhere $I$ is a finite index set, $\\varphi\\colon X\\setto I$ is an osc operator with nonempty values, and $\\{T_i\\}_{i\\in I}$ is a collection of single-valued operators on $X$. The following assertions hold.\n\\begin{enumerate}[label =(\\alph*)]\n","start":23862,"end":24345}]},{"id":"S025","semantic_type":"proposition","kind":"proposition","labels":["eq:X is a union","it:fixed points_Fix","it:fixed points_Sing","it:fixed points_StrFix","it:fixed points_StrFixSing","it:fixed points_X","prop:fixed points"],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"活动区域闭合并覆盖空间","source_units":[{"raw_tex":"\\item\\label{it:fixed points_X} For each $i\\in I$, the set $\\varphi^{-1}(i)$ is closed. The space $X$ can be represented as\n\\begin{equation}\n\\label{eq:X is a union}\nX =\\bigcup_{i\\in I}\\varphi^{-1}(i).\n\\end{equation}    \n","start":24345,"end":24564}]},{"id":"S026","semantic_type":"proposition","kind":"proposition","labels":["it:fixed points_Fix","it:fixed points_Sing","it:fixed points_StrFix","it:fixed points_StrFixSing","it:fixed points_X","prop:fixed points"],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"不动点的活动分支刻画","source_units":[{"raw_tex":"\\item\\label{it:fixed points_Fix} $x\\in\\Fix T$ if and only if there exists $i\\in I$ such that $x\\in\\varphi^{-1}(i)\\cap\\Fix T_i$. Consequently, we have\n\\begin{equation*}\n\\Fix T =\\bigcup_{i\\in I}\\left[\\varphi^{-1}(i)\\cap\\Fix T_i\\right]. \n\\end{equation*}\n","start":24564,"end":24815}]},{"id":"S027","semantic_type":"proposition","kind":"proposition","labels":["it:fixed points_Fix","it:fixed points_Sing","it:fixed points_StrFix","it:fixed points_StrFixSing","it:fixed points_X","prop:fixed points"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"强不动点的共同分支刻画","source_units":[{"raw_tex":"\\item\\label{it:fixed points_StrFix} $x\\in\\StrFix T$ if and only if \n\\begin{equation*}\nx\\in \\bigcap_{i\\in\\varphi(x)}\\Fix T_i =\\bigcap_{i\\in\\varphi(x)} \\left[\\varphi^{-1}(i)\\cap\\Fix T_i\\right].\n\\end{equation*}\n","start":24815,"end":25023}]},{"id":"S028","semantic_type":"proposition","kind":"proposition","labels":["it:fixed points_Fix","it:fixed points_Sing","it:fixed points_StrFix","it:fixed points_StrFixSing","it:fixed points_X","prop:fixed points"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"强不动点与单值性的交集","source_units":[{"raw_tex":"\\item\\label{it:fixed points_StrFixSing} The strong fixed point set is the intersection of the fixed point set and the single-valued set, that is,\n\\begin{equation*}\n\\StrFix T =\\Fix T\\cap \\Sing T.\n\\end{equation*}\n","start":25023,"end":25234}]},{"id":"S029","semantic_type":"proposition","kind":"proposition","labels":["it:fixed points_Fix","it:fixed points_Sing","it:fixed points_StrFix","it:fixed points_StrFixSing","it:fixed points_X","prop:fixed points"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"连续分支的外半连续性","source_units":[{"raw_tex":"\\item\\label{it:fixed points_Sing} If in addition $\\{T_i\\}_{i\\in I}$ is a collection of continuous operators, then $T$ is ocs. Consequently,\n\\begin{equation*}\n\\Sing T =\\left\\{x\\in X:\\Limsup_{x'\\to x}T(x')\\text{ is singleton}\\right\\}.\n\\end{equation*}\n\\end{enumerate}\n\\end{proposition}","start":25234,"end":25516}]},{"id":"S030","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"分支结构命题的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{it:fixed points_X}: Let $i\\in I$ and consider a sequence $(x_n)_{n\\in\\mathbb{N}}\\subseteq\\varphi^{-1}(i)$ such that $x_n\\to\\overline{x}$. The definition of the inverse implies that $i\\in\\varphi(x_n)$ for all $n\\in\\mathbb{N}$ and, since $\\varphi$ is osc, it holds that $i\\in \\varphi(\\overline{x})$, which shows that $\\varphi^{-1}(i)$ is closed and proves the first claim. The fact that \\eqref{eq:X is a union} follows from the assumption that $\\varphi(x)\\neq \\varnothing$ for all $x\\in X$.\n\n\\ref{it:fixed points_Fix}: $x\\in\\Fix T\\iff\\exists i\\in \\varphi(x)$ such that $x\\in\\Fix T_i\\iff\\exists i\\in I$ such that $x\\in\\varphi^{-1}(i)\\cap\\Fix T_i.$ \n\n\\ref{it:fixed points_StrFix}: $x\\in\\StrFix T\\iff x\\in\\Fix T_i$ for all $i\\in\\varphi(x)\\iff x\\in\\varphi^{-1}(i)\\cap\\Fix T_i$ for all $i\\in\\varphi(x)$.\n\n\\ref{it:fixed points_StrFixSing}: Immediate from the respective definitions.\n\n\\ref{it:fixed points_Sing}: Fix $x\\in X$. We first show that $T$ is osc. To this end, take $x_n\\to x$ and $y_n\\to y$ with $y_n\\in T(x_n)$. Then the definition of $T$ ensures the existence of sequence $i_n\\in \\varphi(x_n)$ such that $y_n =T_{i_n}(x_n)\\in T(x_n)$. Using the pigeonhole princple, we pass to a subsequence so that $y_{k_n} =T_i(x_{k_n})$ for fixed $i\\in I$. The osc of $\\varphi$ implies that $i\\in\\varphi(x)$ and continuity of $T_i$ gives\n\\begin{equation*}\ny =\\lim_{n\\to\\infty}y_{k_n} =\\lim_{n\\to\\infty}T_i(x_{k_n}) =T_i(x)\\in T(x),\n\\end{equation*}\nwhich proves osc. The claimed formula for the singleton set follows since $x\\in\\Sing T\\iff T(x)$ is singleton, and osc implies that $T(x) =\\Limsup_{x'\\to x}T(x')$. \n\\end{proof}","start":25517,"end":27150}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"由选择器外半连续得到闭性","source_units":[{"raw_tex":"\\ref{it:fixed points_X}: Let $i\\in I$ and consider a sequence $(x_n)_{n\\in\\mathbb{N}}\\subseteq\\varphi^{-1}(i)$ such that $x_n\\to\\overline{x}$. The definition of the inverse implies that $i\\in\\varphi(x_n)$ for all $n\\in\\mathbb{N}$ and, since $\\varphi$ is osc, it holds that $i\\in \\varphi(\\overline{x})$, which shows that $\\varphi^{-1}(i)$ is closed and proves the first claim. The fact that \\eqref{eq:X is a union} follows from the assumption that $\\varphi(x)\\neq \\varnothing$ for all $x\\in X$.\n","start":25531,"end":26025}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"展开普通不动点定义","source_units":[{"raw_tex":"\\ref{it:fixed points_Fix}: $x\\in\\Fix T\\iff\\exists i\\in \\varphi(x)$ such that $x\\in\\Fix T_i\\iff\\exists i\\in I$ such that $x\\in\\varphi^{-1}(i)\\cap\\Fix T_i.$ \n","start":26026,"end":26182}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"展开强不动点定义","source_units":[{"raw_tex":"\\ref{it:fixed points_StrFix}: $x\\in\\StrFix T\\iff x\\in\\Fix T_i$ for all $i\\in\\varphi(x)\\iff x\\in\\varphi^{-1}(i)\\cap\\Fix T_i$ for all $i\\in\\varphi(x)$.\n","start":26183,"end":26333}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"由定义得到单值交集关系","source_units":[{"raw_tex":"\\ref{it:fixed points_StrFixSing}: Immediate from the respective definitions.\n","start":26334,"end":26411}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"固定分支子列传递极限","source_units":[{"raw_tex":"\\ref{it:fixed points_Sing}: Fix $x\\in X$. We first show that $T$ is osc. To this end, take $x_n\\to x$ and $y_n\\to y$ with $y_n\\in T(x_n)$. Then the definition of $T$ ensures the existence of sequence $i_n\\in \\varphi(x_n)$ such that $y_n =T_{i_n}(x_n)\\in T(x_n)$. Using the pigeonhole princple, we pass to a subsequence so that $y_{k_n} =T_i(x_{k_n})$ for fixed $i\\in I$. The osc of $\\varphi$ implies that $i\\in\\varphi(x)$ and continuity of $T_i$ gives\n\\begin{equation*}\ny =\\lim_{n\\to\\infty}y_{k_n} =\\lim_{n\\to\\infty}T_i(x_{k_n}) =T_i(x)\\in T(x),\n\\end{equation*}\nwhich proves osc. The claimed formula for the singleton set follows since $x\\in\\Sing T\\iff T(x)$ is singleton, and osc implies that $T(x) =\\Limsup_{x'\\to x}T(x')$. \n","start":26412,"end":27139}]},{"id":"S036","semantic_type":"theorem","kind":"theorem","labels":["thm:KM iterations with admissible control"],"member_ids":["S036","S038","S039","S040","S041","S042","S043","S044"],"declared_external":false,"source_only":false,"title":"容许控制下的 KM 收敛","source_units":[{"raw_tex":"\\begin{theorem}[Krasnosel'ski\\u{\\i}--Mann iterations with admissible control]\n\\label{thm:KM iterations with admissible control}\nLet $I$ be a finite index set and let $\\{T_i\\}_{i\\in I}$ be a collection of nonexpansive operators on $X$ with a common fixed point. Define a sequence $(x_n)_{n\\in\\mathbb{N}}$ with starting point $x_0\\in X$ according to\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1} :=(1-\\lambda_n)x_n+\\lambda_n T_{i_n}(x_n),\n\\end{equation*}\nwhere $(i_n)_{n\\in\\mathbb{N}}$ is admissible in $I$, and $(\\lambda_n)_{n\\in\\mathbb{N}}$ is in $(0, 1]$ with $\\liminf_{n\\to\\infty}\\lambda_n(1-\\lambda_n)> 0$.\nThen $(x_n)_{n\\in\\mathbb{N}}$ converges to a point $\\overline{x}\\in\\cap_{i\\in I} \\Fix T_i$.\n\\end{theorem}","start":28810,"end":29534},{"raw_tex":"\\begin{proof}\nSince $\\liminf_{n\\to\\infty}\\lambda_n(1-\\lambda_n)> 0$, there exists $n_0\\in \\mathbb{N}$ such that $\\inf_{n\\geq n_0}\\lambda_n(1-\\lambda_n)> 0$. By relabeling if necessary, we may assume without loss of generality that $\\varepsilon :=\\inf_{n\\in\\mathbb{N}}\\lambda_n(1-\\lambda_n)> 0$. \nLet $x\\in \\cap_{i\\in I} \\Fix T_i$ be arbitrary. Then, for all $n\\in\\mathbb{N}$, nonexpansiveness of $T_{i_n}$ yields\n\\begin{subequations} \n\\begin{align*}\n\\|x_{n+1}-x\\|^2& =\\|(1-\\lambda_n)(x_n-x)+\\lambda_n(T_{i_n}(x_n)-x)\\|^2 \\\\\n& =(1-\\lambda_n)\\|x_n-x\\|^2+\\lambda_n\\|T_{i_n}(x_n)-x\\|^2-\\lambda_n(1-\\lambda_n)\\|x_n-T_{i_n}(x_n)\\|^2 \\\\\n&\\leq \\|x_n-x\\|^2-\\varepsilon\\|x_n-T_{i_n}(x_n)\\|^2.\n\\end{align*}\n\\end{subequations}\nIt follows that $(\\|x_n-x\\|^2)_{n\\in\\mathbb{N}}$ is monotone nonincreasing, and hence convergent. Consequently, the sequence $(x_n)_{n\\in\\mathbb{N}}$ is bounded and \n\\begin{equation}\\label{eq:pre-asymp}\nx_n-T_{i_n}(x_n)\\to 0 \\quad\\text{as~} n\\to\\infty.\n\\end{equation}\nNow, let $\\overline{x}\\in X$ be a cluster point of $(x_n)_{n\\in\\mathbb{N}}$. Then $(x_n)_{n\\in\\mathbb{N}}$ contains a convergent subsequence, say $(x_{k_n})_{n\\in\\mathbb{N}}$, with limit $\\overline{x}$. \nWe claim that $\\overline{x}$ is an element of $\\cap_{i\\in I} \\Fix T_i$. To this end suppose, by way of a contradiction, that $\\overline{x}\\not\\in\\cap_{i\\in I} \\Fix T_i$.\nBy passing to a further subsequence if necessary, we deduce that the sequence $(t_n)_{n\\in\\mathbb{N}}$ defined by\n\\begin{equation*}\nt_n :=\\min\\{p\\in\\{k_n,\\dots,k_{n+1}-1\\}:\\overline{x}\\not\\in\\Fix T_{i_p}\\}, \n\\end{equation*}\nis well defined.\nSince $I$ is finite, by passing to yet another subsequence if necessary, we may assume that $i_{t_n} =\\ell$ for all $n\\in\\mathbb{N}$ for some fixed index $\\ell\\in I$. \nTogether with \\eqref{eq:pre-asymp}, we deduce that\n\\begin{equation*}\nx_{t_n}\\to \\overline{x} \\text{~~~and~~~} (\\Id-T_\\ell)(x_{t_n})\\to 0 \\quad\\text{as~} n\\to\\infty,\n\\end{equation*}\nwhich, due to nonexpansiveness of $T_\\ell$, implies that $\\overline{x}\\in\\Fix T_\\ell$. \nThus a contradiction is obtained, and conclude that $\\overline{x}\\in\\cap_{i\\in I} \\Fix T_i$. We then have that $(\\|x_n-\\overline{x}\\|)_{n\\in\\mathbb{N}}$ is monotone nonincreasing and $\\|x_{k_n}-\\overline{x}\\|\\to 0$, and thus the conclusion follows.\n\\end{proof}","start":29535,"end":31830},{"raw_tex":"Since $\\liminf_{n\\to\\infty}\\lambda_n(1-\\lambda_n)> 0$, there exists $n_0\\in \\mathbb{N}$ such that $\\inf_{n\\geq n_0}\\lambda_n(1-\\lambda_n)> 0$. By relabeling if necessary, we may assume without loss of generality that $\\varepsilon :=\\inf_{n\\in\\mathbb{N}}\\lambda_n(1-\\lambda_n)> 0$. \n","start":29549,"end":29831},{"raw_tex":"Let $x\\in \\cap_{i\\in I} \\Fix T_i$ be arbitrary. Then, for all $n\\in\\mathbb{N}$, nonexpansiveness of $T_{i_n}$ yields\n\\begin{subequations} \n\\begin{align*}\n\\|x_{n+1}-x\\|^2& =\\|(1-\\lambda_n)(x_n-x)+\\lambda_n(T_{i_n}(x_n)-x)\\|^2 \\\\\n& =(1-\\lambda_n)\\|x_n-x\\|^2+\\lambda_n\\|T_{i_n}(x_n)-x\\|^2-\\lambda_n(1-\\lambda_n)\\|x_n-T_{i_n}(x_n)\\|^2 \\\\\n&\\leq \\|x_n-x\\|^2-\\varepsilon\\|x_n-T_{i_n}(x_n)\\|^2.\n\\end{align*}\n\\end{subequations}\n","start":29831,"end":30250},{"raw_tex":"It follows that $(\\|x_n-x\\|^2)_{n\\in\\mathbb{N}}$ is monotone nonincreasing, and hence convergent. Consequently, the sequence $(x_n)_{n\\in\\mathbb{N}}$ is bounded and \n\\begin{equation}\\label{eq:pre-asymp}\nx_n-T_{i_n}(x_n)\\to 0 \\quad\\text{as~} n\\to\\infty.\n\\end{equation}\n","start":30250,"end":30518},{"raw_tex":"Now, let $\\overline{x}\\in X$ be a cluster point of $(x_n)_{n\\in\\mathbb{N}}$. Then $(x_n)_{n\\in\\mathbb{N}}$ contains a convergent subsequence, say $(x_{k_n})_{n\\in\\mathbb{N}}$, with limit $\\overline{x}$. \nWe claim that $\\overline{x}$ is an element of $\\cap_{i\\in I} \\Fix T_i$. To this end suppose, by way of a contradiction, that $\\overline{x}\\not\\in\\cap_{i\\in I} \\Fix T_i$.\nBy passing to a further subsequence if necessary, we deduce that the sequence $(t_n)_{n\\in\\mathbb{N}}$ defined by\n\\begin{equation*}\nt_n :=\\min\\{p\\in\\{k_n,\\dots,k_{n+1}-1\\}:\\overline{x}\\not\\in\\Fix T_{i_p}\\}, \n\\end{equation*}\nis well defined.\nSince $I$ is finite, by passing to yet another subsequence if necessary, we may assume that $i_{t_n} =\\ell$ for all $n\\in\\mathbb{N}$ for some fixed index $\\ell\\in I$. \n","start":30518,"end":31301},{"raw_tex":"Together with \\eqref{eq:pre-asymp}, we deduce that\n\\begin{equation*}\nx_{t_n}\\to \\overline{x} \\text{~~~and~~~} (\\Id-T_\\ell)(x_{t_n})\\to 0 \\quad\\text{as~} n\\to\\infty,\n\\end{equation*}\nwhich, due to nonexpansiveness of $T_\\ell$, implies that $\\overline{x}\\in\\Fix T_\\ell$. \n","start":31301,"end":31570},{"raw_tex":"Thus a contradiction is obtained, and conclude that $\\overline{x}\\in\\cap_{i\\in I} \\Fix T_i$. We then have that $(\\|x_n-\\overline{x}\\|)_{n\\in\\mathbb{N}}$ is monotone nonincreasing and $\\|x_{k_n}-\\overline{x}\\|\\to 0$, and thus the conclusion follows.\n","start":31570,"end":31819}]},{"id":"S037","semantic_type":"assumption","kind":"assumption","labels":["thm:KM iterations with admissible control"],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"KM 算子族与迭代条件","source_units":[{"raw_tex":"Let $I$ be a finite index set and let $\\{T_i\\}_{i\\in I}$ be a collection of nonexpansive operators on $X$ with a common fixed point. Define a sequence $(x_n)_{n\\in\\mathbb{N}}$ with starting point $x_0\\in X$ according to\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1} :=(1-\\lambda_n)x_n+\\lambda_n T_{i_n}(x_n),\n\\end{equation*}\nwhere $(i_n)_{n\\in\\mathbb{N}}$ is admissible in $I$, and $(\\lambda_n)_{n\\in\\mathbb{N}}$ is in $(0, 1]$ with $\\liminf_{n\\to\\infty}\\lambda_n(1-\\lambda_n)> 0$.\n","start":28938,"end":29429}]},{"id":"S045","semantic_type":"assumption","kind":"assumption","labels":["prop:radius"],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"半径命题的强不动点假设","source_units":[{"raw_tex":"Let $T\\colon X\\setto X$ be a union nonexpansive operator with $x^*\\in\\StrFix T$. Then $r :=r(x^*;T)\\in (0,+\\infty]$ and\n","start":35340,"end":35460}]},{"id":"S046","semantic_type":"proposition","kind":"proposition","labels":["eq:attraction nonexpansive","prop:radius"],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"强不动点附近的准非扩张性","source_units":[{"raw_tex":"\\begin{proposition}[Radius of attraction at strong fixed points]\\label{prop:radius}\nLet $T\\colon X\\setto X$ be a union nonexpansive operator with $x^*\\in\\StrFix T$. Then $r :=r(x^*;T)\\in (0,+\\infty]$ and\n\\begin{equation}\\label{eq:attraction nonexpansive}\n\\forall x\\in \\inte\\mathbb{B}(x^*;r),\\,\\forall y\\in T(x),\\quad \n\\|y-x^*\\|\\leq \\|x-x^*\\|.\n\\end{equation}\n","start":35256,"end":35614}]},{"id":"S047","semantic_type":"proposition","kind":"proposition","labels":["eq:attraction averaged","prop:radius"],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"松弛迭代的局部下降估计","source_units":[{"raw_tex":"Furthermore, for any $\\lambda\\in (0, 1]$, we have\n\\begin{equation}\\label{eq:attraction averaged}\n\\forall x\\in \\inte\\mathbb{B}(x^*;r),\\,\\forall y\\in (1-\\lambda)x+\\lambda T(x),\\quad \n\\|y-x^*\\|^2+\\frac{1-\\lambda}{\\lambda}\\|x-y\\|^2\\leq \\|x-x^*\\|^2.\n\\end{equation}\n\\end{proposition}","start":35614,"end":35891}]},{"id":"S048","semantic_type":"proof","kind":"proof","labels":["eq:averaged","eq:nonexpansive"],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"局部下降性质的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nSince $\\varphi$ is osc and its range, $I$, is a finite set, Proposition~\\ref{prop:prop2.1 Tam17} implies that $r :=r(x^*;T)\\in(0,+\\infty]$. Since $x^*\\in\\StrFix T$, Proposition~\\ref{prop:fixed points}\\ref{it:fixed points_StrFix} implies that $x^*\\in\\cap_{i\\in\\varphi(x*)}\\Fix T_i$ and, since each $T_i$ is nonexpansive, we have\n\\begin{equation}\n\\label{eq:nonexpansive}\n\\forall i\\in\\varphi(x^*),\\, \\forall x\\in X,\\quad \\|T_i(x)-x^*\\|^2\\leq \\|x-x^*\\|^2.\n\\end{equation}\nIn particular, for any $x\\in \\inte\\mathbb{B}(x^*;r)$ and $y\\in T(x)$, there exists an $i\\in \\varphi(x)\\subseteq \\varphi(x^*)$ such that $y =T_i(x)$. Consequently, \\eqref{eq:attraction nonexpansive} follows from \\eqref{eq:nonexpansive}.\n\nFurthermore, for any $\\lambda\\in (0, 1]$, the operator $S_i :=(1-\\lambda)\\Id+\\lambda T_i$ is $\\lambda$-averaged nonexpansive by Proposition~\\ref{prop:averaged} and we therefore have that\n\\begin{equation}\n\\label{eq:averaged}\n\\forall i\\in\\varphi(x^*),\\, \\forall x\\in X,\\quad \\|S_i(x)-x^*\\|^2+\\frac{1-\\lambda}{\\lambda}\\|x-S_i(x)\\|^2\\leq \\|x-x^*\\|^2.\n\\end{equation}\nIn particular, if $x\\in\\inte\\mathbb{B}(x^*;r)$ and $y\\in (1-\\lambda)x+\\lambda T(x)$, then there exists $i\\in \\varphi(x)\\subseteq \\varphi(x^*)$ such that $y =S_i(x)$. As before, \\eqref{eq:attraction averaged} follows from \\eqref{eq:averaged}.\n\\end{proof}","start":35892,"end":37225}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["eq:nonexpansive"],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"活动分支的共同参考不动点","source_units":[{"raw_tex":"Since $\\varphi$ is osc and its range, $I$, is a finite set, Proposition~\\ref{prop:prop2.1 Tam17} implies that $r :=r(x^*;T)\\in(0,+\\infty]$. Since $x^*\\in\\StrFix T$, Proposition~\\ref{prop:fixed points}\\ref{it:fixed points_StrFix} implies that $x^*\\in\\cap_{i\\in\\varphi(x*)}\\Fix T_i$ and, since each $T_i$ is nonexpansive, we have\n\\begin{equation}\n\\label{eq:nonexpansive}\n\\forall i\\in\\varphi(x^*),\\, \\forall x\\in X,\\quad \\|T_i(x)-x^*\\|^2\\leq \\|x-x^*\\|^2.\n\\end{equation}\n","start":35906,"end":36373}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"将分支距离估计传给集合取值","source_units":[{"raw_tex":"In particular, for any $x\\in \\inte\\mathbb{B}(x^*;r)$ and $y\\in T(x)$, there exists an $i\\in \\varphi(x)\\subseteq \\varphi(x^*)$ such that $y =T_i(x)$. Consequently, \\eqref{eq:attraction nonexpansive} follows from \\eqref{eq:nonexpansive}.\n","start":36373,"end":36609}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":["eq:averaged"],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"松弛分支的平均性估计","source_units":[{"raw_tex":"Furthermore, for any $\\lambda\\in (0, 1]$, the operator $S_i :=(1-\\lambda)\\Id+\\lambda T_i$ is $\\lambda$-averaged nonexpansive by Proposition~\\ref{prop:averaged} and we therefore have that\n\\begin{equation}\n\\label{eq:averaged}\n\\forall i\\in\\varphi(x^*),\\, \\forall x\\in X,\\quad \\|S_i(x)-x^*\\|^2+\\frac{1-\\lambda}{\\lambda}\\|x-S_i(x)\\|^2\\leq \\|x-x^*\\|^2.\n\\end{equation}\n","start":36610,"end":36972}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"将松弛分支估计传给迭代","source_units":[{"raw_tex":"In particular, if $x\\in\\inte\\mathbb{B}(x^*;r)$ and $y\\in (1-\\lambda)x+\\lambda T(x)$, then there exists $i\\in \\varphi(x)\\subseteq \\varphi(x^*)$ such that $y =S_i(x)$. As before, \\eqref{eq:attraction averaged} follows from \\eqref{eq:averaged}.\n","start":36972,"end":37214}]},{"id":"S053","semantic_type":"theorem","kind":"theorem","labels":["thm:union"],"member_ids":["S053","S055","S056","S057","S058","S059","S060"],"declared_external":false,"source_only":false,"title":"并非扩张迭代的局部收敛","source_units":[{"raw_tex":"\\begin{theorem}[Local convergence of union nonexpansive iterations]\\label{thm:union}\nLet $T\\colon X\\setto X$ be a union nonexpansive operator with $x^*\\in\\StrFix T$ and let $(\\lambda_n)_{n\\in\\mathbb{N}}$ be a sequence in $(0, 1]$ with $\\liminf_{n\\to\\infty}\\lambda_n(1-\\lambda_n)> 0$. Denote $r :=r(x^*;T)\\in(0,+\\infty]$ and consider a sequence $(x_n)_{n\\in\\mathbb{N}}$ with $x_0\\in \\inte\\mathbb{B}(x^*;r)$ satisfying\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1}\\in (1-\\lambda_n)x_n+\\lambda_n T(x_n).\n\\end{equation*} \nThen $(x_n)_{n\\in\\mathbb{N}}$ converges to a point $\\overline{x}\\in\\Fix T\\cap \\mathbb{B}(x^*;r)$. \n\\end{theorem}","start":37416,"end":38055},{"raw_tex":"\\begin{proof}\nWe first observe from Proposition~\\ref{prop:radius} that, for all $n\\in\\mathbb{N}$, $x_n\\in \\inte\\mathbb{B}(x^*;r)$ and \n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad \\|x_{n+1}-x^*\\|^2+\\frac{1-\\lambda_n}{\\lambda_n}\\|x_n-x_{n+1}\\|^2\\leq \\|x_n-x^*\\|^2\n\\end{equation*}\nwith convention that $\\frac{1-\\lambda_n}{\\lambda_n} =0$ if $\\lambda_n =0$.\nBy the definition of $T$, there is a sequence of indices $(i_n)_{n\\in\\mathbb{N}}\\subseteq I$ such that\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1} =(1-\\lambda_n)x_n+\\lambda_n T_{i_n}(x_n)\\quad\\text{and}\\quad i_n\\in \\varphi(x_n)\\subseteq\\varphi(x^*),\n\\end{equation*}\nwhere the last inclusion is a consequence of Proposition~\\ref{prop:prop2.1 Tam17} and the fact that $(x_n)_{n\\in\\mathbb{N}}\\subseteq\\inte\\mathbb{B}(x^*;r)$.\n\nLet $I^*$ denote the set of admissible indices in the sequence $(i_n)_{n\\in\\mathbb{N}}$. Then $I^*\\subseteq \\varphi(x^*)$ which,  together with Proposition~\\ref{prop:fixed points}\\ref{it:fixed points_StrFix} applied to $x^*\\in \\StrFix T$, yields\n\\begin{equation*}\nx^*\\in \\bigcap_{i\\in\\varphi(x*)}\\Fix T_i\\subseteq \\bigcap_{i\\in I^*}\\Fix T_i.\n\\end{equation*}\nThat is, $\\{T_i\\}_{i\\in I^*}$ is a collection of nonexpansive operators with a common fixed point. By applying Theorem~\\ref{thm:KM iterations with admissible control}, $x_n\\to\\overline{x}\\in\\cap_{i\\in I^*}\\Fix T_i$. Since, for all $n\\in\\mathbb{N}$, $x_n\\in \\mathbb{B}(x^*;r)$, it also holds that $\\overline{x}\\in \\mathbb{B}(x^*;r)$. Finally, for any $i\\in I^*$, there exists a subsequence $i_{k_n}\\to i$. Since $x_{k_n}\\to\\overline{x}$ and $\\varphi$ is osc with $i_{k_n}\\in\\varphi(x_{n_k})$, we deduce that $i\\in\\varphi(\\overline{x})$. By Proposition~\\ref{prop:fixed points}\\ref{it:fixed points_Fix},  it follows that $\\overline{x}\\in\\Fix T$ as was claimed.\n\\end{proof}","start":38056,"end":39874},{"raw_tex":"We first observe from Proposition~\\ref{prop:radius} that, for all $n\\in\\mathbb{N}$, $x_n\\in \\inte\\mathbb{B}(x^*;r)$ and \n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad \\|x_{n+1}-x^*\\|^2+\\frac{1-\\lambda_n}{\\lambda_n}\\|x_n-x_{n+1}\\|^2\\leq \\|x_n-x^*\\|^2\n\\end{equation*}\nwith convention that $\\frac{1-\\lambda_n}{\\lambda_n} =0$ if $\\lambda_n =0$.\n","start":38070,"end":38412},{"raw_tex":"By the definition of $T$, there is a sequence of indices $(i_n)_{n\\in\\mathbb{N}}\\subseteq I$ such that\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1} =(1-\\lambda_n)x_n+\\lambda_n T_{i_n}(x_n)\\quad\\text{and}\\quad i_n\\in \\varphi(x_n)\\subseteq\\varphi(x^*),\n\\end{equation*}\nwhere the last inclusion is a consequence of Proposition~\\ref{prop:prop2.1 Tam17} and the fact that $(x_n)_{n\\in\\mathbb{N}}\\subseteq\\inte\\mathbb{B}(x^*;r)$.\n","start":38412,"end":38846},{"raw_tex":"Let $I^*$ denote the set of admissible indices in the sequence $(i_n)_{n\\in\\mathbb{N}}$. Then $I^*\\subseteq \\varphi(x^*)$ which,  together with Proposition~\\ref{prop:fixed points}\\ref{it:fixed points_StrFix} applied to $x^*\\in \\StrFix T$, yields\n\\begin{equation*}\nx^*\\in \\bigcap_{i\\in\\varphi(x*)}\\Fix T_i\\subseteq \\bigcap_{i\\in I^*}\\Fix T_i.\n\\end{equation*}\n","start":38847,"end":39205},{"raw_tex":"That is, $\\{T_i\\}_{i\\in I^*}$ is a collection of nonexpansive operators with a common fixed point. By applying Theorem~\\ref{thm:KM iterations with admissible control}, $x_n\\to\\overline{x}\\in\\cap_{i\\in I^*}\\Fix T_i$. Since, for all $n\\in\\mathbb{N}$, $x_n\\in \\mathbb{B}(x^*;r)$, it also holds that $\\overline{x}\\in \\mathbb{B}(x^*;r)$. Finally, for any $i\\in I^*$, there exists a subsequence $i_{k_n}\\to i$. Since $x_{k_n}\\to\\overline{x}$ and $\\varphi$ is osc with $i_{k_n}\\in\\varphi(x_{n_k})$, we deduce that $i\\in\\varphi(\\overline{x})$. By Proposition~\\ref{prop:fixed points}\\ref{it:fixed points_Fix},  it follows that $\\overline{x}\\in\\Fix T$ as was claimed.\n","start":39205,"end":39863}]},{"id":"S054","semantic_type":"assumption","kind":"assumption","labels":["thm:union"],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"并非扩张迭代的输入条件","source_units":[{"raw_tex":"Let $T\\colon X\\setto X$ be a union nonexpansive operator with $x^*\\in\\StrFix T$ and let $(\\lambda_n)_{n\\in\\mathbb{N}}$ be a sequence in $(0, 1]$ with $\\liminf_{n\\to\\infty}\\lambda_n(1-\\lambda_n)> 0$. Denote $r :=r(x^*;T)\\in(0,+\\infty]$ and consider a sequence $(x_n)_{n\\in\\mathbb{N}}$ with $x_0\\in \\inte\\mathbb{B}(x^*;r)$ satisfying\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1}\\in (1-\\lambda_n)x_n+\\lambda_n T(x_n).\n\\end{equation*} \n","start":37501,"end":37943}]},{"id":"S061","semantic_type":"corollary","kind":"corollary","labels":["cor:union averaged"],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"并平均迭代的局部收敛","source_units":[{"raw_tex":"\\begin{corollary}[Local convergence of union averaged nonexpansive iterations]\\label{cor:union averaged}\nLet $T\\colon X\\setto X$ be union $\\alpha$-averaged nonexpansive operator with $x^*\\in\\StrFix T$ and let $(\\lambda_n)_{n\\in\\mathbb{N}}$ be a sequence in $(0, 1/\\alpha]$ with $\\liminf_{n\\to\\infty} \\lambda_n(1/\\alpha-\\lambda_n)> 0$. Denote $r :=r(x^*;T)\\in(0,+\\infty]$ and consider $(x_n)_{n\\in\\mathbb{N}}$ with $x_0\\in \\inte\\mathbb{B}(x^*;r)$ satisfying\n\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1}\\in (1-\\lambda_n)x_n+\\lambda_n T(x_n).\n\\end{equation*} \nThen $(x_n)_{n\\in\\mathbb{N}}$ converges to a point $\\overline{x}\\in\\Fix T\\cap \\mathbb{B}(x^*;r)$. \n\\end{corollary}","start":39877,"end":40558}]},{"id":"S062","semantic_type":"assumption","kind":"assumption","labels":["cor:union averaged"],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"主推论的局部与参数条件","source_units":[{"raw_tex":"Let $T\\colon X\\setto X$ be union $\\alpha$-averaged nonexpansive operator with $x^*\\in\\StrFix T$ and let $(\\lambda_n)_{n\\in\\mathbb{N}}$ be a sequence in $(0, 1/\\alpha]$ with $\\liminf_{n\\to\\infty} \\lambda_n(1/\\alpha-\\lambda_n)> 0$. Denote $r :=r(x^*;T)\\in(0,+\\infty]$ and consider $(x_n)_{n\\in\\mathbb{N}}$ with $x_0\\in \\inte\\mathbb{B}(x^*;r)$ satisfying\n","start":39982,"end":40334}]},{"id":"S063","semantic_type":"algorithm","kind":"algorithm","labels":["cor:union averaged"],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"集合值松弛迭代","source_units":[{"raw_tex":"\\begin{equation*}\n\\forall n\\in\\mathbb{N},\\quad x_{n+1}\\in (1-\\lambda_n)x_n+\\lambda_n T(x_n).\n\\end{equation*} \n","start":40334,"end":40444}]},{"id":"S064","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S064"],"declared_external":false,"source_only":false,"title":"并平均局部收敛的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nIt follows from Proposition~\\ref{prop:averaged} that $R :=(1-1/\\alpha)\\Id+(1/\\alpha)T$ is union nonexpansive. Under the fixed representation of $T$ in \\eqref{eq:a particular uan op}, we have $r(x^*;R) =r(x^*;T)$. Moreover, $\\StrFix R =\\StrFix T$, $\\Fix R =\\Fix T$, and, for all $n\\in\\mathbb{N}$, $x_{n+1} =(1-\\alpha\\lambda_n)x_n+\\alpha\\lambda_n R(x)$. The result now follows from Theorem~\\ref{thm:union} applied to $R$ and $(\\alpha\\lambda_n)_{n\\in\\mathbb{N}}$. \n\\end{proof}","start":40559,"end":41046}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S065"],"declared_external":false,"source_only":false,"title":"变换为并非扩张算子","source_units":[{"raw_tex":"It follows from Proposition~\\ref{prop:averaged} that $R :=(1-1/\\alpha)\\Id+(1/\\alpha)T$ is union nonexpansive. Under the fixed representation of $T$ in \\eqref{eq:a particular uan op}, we have $r(x^*;R) =r(x^*;T)$. Moreover, $\\StrFix R =\\StrFix T$, $\\Fix R =\\Fix T$, and, for all $n\\in\\mathbb{N}$, $x_{n+1} =(1-\\alpha\\lambda_n)x_n+\\alpha\\lambda_n R(x)$. The result now follows from Theorem~\\ref{thm:union} applied to $R$ and $(\\alpha\\lambda_n)_{n\\in\\mathbb{N}}$. \n","start":40573,"end":41035}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S066"],"declared_external":false,"source_only":false,"title":"保留选择器与吸引半径","source_units":[{"raw_tex":"It follows from Proposition~\\ref{prop:averaged} that $R :=(1-1/\\alpha)\\Id+(1/\\alpha)T$ is union nonexpansive. Under the fixed representation of $T$ in \\eqref{eq:a particular uan op}, we have $r(x^*;R) =r(x^*;T)$. Moreover, $\\StrFix R =\\StrFix T$, $\\Fix R =\\Fix T$, and, for all $n\\in\\mathbb{N}$, $x_{n+1} =(1-\\alpha\\lambda_n)x_n+\\alpha\\lambda_n R(x)$. The result now follows from Theorem~\\ref{thm:union} applied to $R$ and $(\\alpha\\lambda_n)_{n\\in\\mathbb{N}}$. \n","start":40573,"end":41035}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S067"],"declared_external":false,"source_only":false,"title":"保留普通与强不动点","source_units":[{"raw_tex":"It follows from Proposition~\\ref{prop:averaged} that $R :=(1-1/\\alpha)\\Id+(1/\\alpha)T$ is union nonexpansive. Under the fixed representation of $T$ in \\eqref{eq:a particular uan op}, we have $r(x^*;R) =r(x^*;T)$. Moreover, $\\StrFix R =\\StrFix T$, $\\Fix R =\\Fix T$, and, for all $n\\in\\mathbb{N}$, $x_{n+1} =(1-\\alpha\\lambda_n)x_n+\\alpha\\lambda_n R(x)$. The result now follows from Theorem~\\ref{thm:union} applied to $R$ and $(\\alpha\\lambda_n)_{n\\in\\mathbb{N}}$. \n","start":40573,"end":41035}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S068"],"declared_external":false,"source_only":false,"title":"重写变换后的松弛递推","source_units":[{"raw_tex":"It follows from Proposition~\\ref{prop:averaged} that $R :=(1-1/\\alpha)\\Id+(1/\\alpha)T$ is union nonexpansive. Under the fixed representation of $T$ in \\eqref{eq:a particular uan op}, we have $r(x^*;R) =r(x^*;T)$. Moreover, $\\StrFix R =\\StrFix T$, $\\Fix R =\\Fix T$, and, for all $n\\in\\mathbb{N}$, $x_{n+1} =(1-\\alpha\\lambda_n)x_n+\\alpha\\lambda_n R(x)$. The result now follows from Theorem~\\ref{thm:union} applied to $R$ and $(\\alpha\\lambda_n)_{n\\in\\mathbb{N}}$. \n","start":40573,"end":41035}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S069"],"declared_external":false,"source_only":false,"title":"应用并非扩张局部收敛定理","source_units":[{"raw_tex":"It follows from Proposition~\\ref{prop:averaged} that $R :=(1-1/\\alpha)\\Id+(1/\\alpha)T$ is union nonexpansive. Under the fixed representation of $T$ in \\eqref{eq:a particular uan op}, we have $r(x^*;R) =r(x^*;T)$. Moreover, $\\StrFix R =\\StrFix T$, $\\Fix R =\\Fix T$, and, for all $n\\in\\mathbb{N}$, $x_{n+1} =(1-\\alpha\\lambda_n)x_n+\\alpha\\lambda_n R(x)$. The result now follows from Theorem~\\ref{thm:union} applied to $R$ and $(\\alpha\\lambda_n)_{n\\in\\mathbb{N}}$. \n","start":40573,"end":41035}]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S004","S017":"S017","S018":"S014","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S022","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S036","S039":"S036","S040":"S036","S041":"S036","S042":"S036","S043":"S036","S044":"S036","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S053","S056":"S053","S057":"S053","S058":"S053","S059":"S053","S060":"S053","S061":"S061","S062":"S062","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S069"}}}}
{"record_id":"2207.06750v2","paper":{"title":"On the Approximation of Unbounded Convex Sets by Polyhedra","authors":["Dörfler, Daniel"],"arxiv_id":"2207.06750","version":"v2","journal":"Journal of Optimization Theory and Applications","abstract_url":"https://arxiv.org/abs/2207.06750v2","pdf_url":"https://arxiv.org/pdf/2207.06750v2","doi":"10.1007/s10957-022-02020-3"},"context":{"raw_tex":"Throughout this article we denote by $\\cl C$, $\\interior C$, $\\recc\nC$, $\\conv C$, and $\\cone C$ the closure, interior, recession cone,\nconvex hull and conical hull of a set $C$, respectively. A compact\nconvex set with nonempty interior is called a \\textit{convex\nbody}. The Euclidean ball with radius $r$ centered at a point~${c \\in\n\\R^n}$ is denoted by~$B_r(c)$. A point $c$ of a convex set $C$ is\ncalled an \\emph{extreme point} of $C$, if $C \\setminus\\{c\\}$ is\nconvex. The set of extreme points of $C$ is denoted by $\\ext\nC$. Extreme points are exactly the points of $C$ that cannot be\nwritten as a proper convex combination of elements of $C$\n\\cite[p.~162]{Roc70}. For finite sets $V, D \\subseteq \\R^n$, the set\n\\begin{equation}\\label{vrep}\nP = \\conv V + \\cone D\n\\end{equation}\nis called a \\emph{polyhedron}. The plus sign denotes Minkowski\naddition. The sets $V,D$ in \\eqref{vrep} are called a\n\\emph{$V$-representation} of $P$ as it is expressed in terms of its\nvertices and directions. A polyhedron can equivalently be expressed as\na finite intersection of closed halfspaces \\cite[Theorem 19.1]{Roc70},\ni.e.\n\\begin{equation}\\label{hrep}\nP = \\{x \\in \\R^n \\mid Ax \\leq b\\}\n\\end{equation}\nfor a matrix $A \\in \\R^{m \\times n}$ and a vector $b \\in \\R^m$. The\ndata $(A,b)$ are called a \\emph{$H$-representation} of $P$.  The\nextreme points of a polyhedron $P$ are called \\emph{vertices} of $P$\nand are denoted by~$\\vertices P$. For symmetric matrices\n$A_0,A_1,\\dots,A_n$ of arbitrary fixed size we define\n\\begin{equation}\n\\Ab(x) := \\sum_{i=1}^n x_iA_i \\quad \\text{and} \\quad \\A(x) := \\Ab(x) +\nA_0,\n\\end{equation}\ni.e. a linear combination of the $A_i$ and a translation of $\\Ab(x)$\nby $A_0$, respectively. We denote by $A \\mg 0$ ($A \\mgeq 0$) positive\n(semi-)definiteness of the symmetric matrix $A$. A set of the form\n$\\{x \\in \\R^n \\mid \\A(x) \\mgeq 0\\}$ is called a\n\\emph{spectrahedron}. Spectrahedra are a generalization of polyhedra\nfor which many geometric properties of polyhedra generalize nicely,\ne.g. the recession cone of a spectrahedron $C$ is obtained as $\\{x \\in\n\\R^n \\mid \\Ab(x) \\mgeq 0 \\}$, see \\cite{GR95}, whereas the recession\ncone of a polyhedron in $H$-representation is $\\{x \\in \\R^n \\mid Ax\n\\leq 0\\}$. Given a cone $K$ the set $\\polar{K} = \\{y \\in \\R^n \\mid\n\\forall x \\in K: x\\T y \\leq 0\\}$ is called the \\emph{polar cone} of\n$K$. The polar $\\polar{(\\recc{C})}$ of the recession cone of a\nspectrahedron $C$ is computed as\n\\begin{equation}\\label{EPCS}\n\\polar{(\\recc{C})} = \\cl \\left\\lbrace \\left( -A_1 \\cdot X, \\dots, -A_n\n\\cdot X \\right)\\T \\mid X \\mgeq 0 \\right\\rbrace, \n\\end{equation}\nwhere $A_i \\cdot X$ means the trace of the matrix product $A_iX$, see\n\\cite[Section 3]{GR95}. A cone $K$ is called \\emph{polyhedral} if ${K\n= \\cone D}$ for some finite set $D$ and \\emph{pointed} if ${K \\cap\n(-K) = \\{0\\}}$. A set whose recession cone is pointed is called\n\\emph{line-free}. It is well known, that a closed convex set contains\nan extreme point if and only if it is line-free, see \\cite[Corollary\n18.5.3]{Roc70}. Given nonempty sets ${C_1, C_2 \\subseteq \\R^n}$ the\n\\emph{excess of $C_1$ over $C_2$}, ${\\exc{C_1}{C_2}}$, is defined as\n\\begin{equation}\n\\exc{C_1}{C_2} = \\adjustlimits \\sup_{c_1 \\in C_1} \\inf_{c_2 \\in C_2}\n\\norm{c_1-c_2},\n\\end{equation}\nwhere~$\\norm{\\cdot}$ denotes the Euclidean norm. The \\emph{Hausdorff\ndistance between $C_1$ and $C_2$}, ${\\haus{C_1}{C_2}}$, is then\nexpressed as\n\\begin{equation}\n\\haus{C_1}{C_2} = \\max \\{\\exc{C_1}{C_2}, \\exc{C_2}{C_1}\\}.\n\\end{equation}\nIt is well known, that the Hausdorff distance defines a metric on the\nspace of nonempty compact subsets of $\\R^n$. Between unbounded sets\nthe Hausdorff distance may be infinite. A polyhedron $P$ is called an\n\\textit{$\\varepsilon$-approximation} of a convex set $C$ if\n$\\haus{P}{C} \\leq \\varepsilon$.\n\n\n\\begin{definition}\n  Given nonempty closed convex cones~${K_1, K_2 \\subseteq \\R^n}$,\n  the \\emph{truncated Hausdorff distance between~$K_1$\n    and~$K_2$},~${\\thaus{K_1}{K_2}}$, is defined as\n  \\begin{equation}\n    \\thaus{K_1}{K_2} := \\haus{K_1 \\cap B_1(0)}{K_2 \\cap B_1(0)}.\n  \\end{equation}\n\\end{definition}\n\n\n\\begin{definition}\\label{DEDA}\n  Given a nonempty closed convex and line-free set~${C \\subseteq\n    \\R^n}$, a line-free polyhedron~$P$ is called an\n  \\emph{${\\ed}$-approximation of~$C$} if\n  \\begin{enumerate}[(i)]\n  \\item $\\exc{\\vertices P}{C} \\leq \\varepsilon$,\n  \\item $\\thaus{\\recc{P}}{\\recc{C}} \\leq \\delta$,\n  \\item $P \\supseteq C$.\n  \\end{enumerate}\n\\end{definition}\n\n\n\\begin{definition}\n  A sequence~$\\{C^{\\nu}\\}_{\\nu \\in \\mathbb{N}}$ of subsets of~$\\R^n$ is said\n  to \\emph{converge to~${C \\subseteq \\R^n}$ in the sense of\n    Painlevé-Kuratowski}, denoted by~${C^{\\nu} \\rightarrow C}$, if the\n  following equalities hold:\n  \\begin{equation*}\n    \\begin{aligned}\n      C &= \\left\\lbrace x \\in \\R^n \\middle\\vert \\begin{matrix} \\text{for all open\n          neighbourhoods } U \\text{ of } x,\\\\ U \\cap C^{\\nu}  \\neq \\emptyset\n          \\text{ for large enough } \\nu\\end{matrix} \\right\\rbrace \\\\\n      &= \\left\\lbrace x \\in \\R^n \\middle\\vert \\begin{matrix} \\text{for all open\n        neighbourhoods } U \\text{ of } x,\\\\ U \\cap C^{\\nu} \\neq \\emptyset\n        \\text{ for infinitely many }\\nu\\end{matrix} \\right\\rbrace.\n    \\end{aligned}\n  \\end{equation*}\n\\end{definition}\n\n\n\\begin{theorem}[{see \\cite[p.~120]{RW98}}]\\label{TPKCHD}\n  A sequence~$C^{\\nu}$ of nonempty closed and convex sets converges to~$C$\n  in the sense of Painlevé-Kuratowski if and only if there exist~$x\n  \\in \\R^n$ and~$r_0 \\in \\R$, such that for all~$r \\geq r_0$ it holds, that\n  \\begin{equation}\n    \\haus{C^{\\nu} \\cap B_r(x)}{C \\cap B_r(x)} \\rightarrow 0.\n  \\end{equation}\n\\end{theorem}\n","units":[{"raw_tex":"Throughout this article we denote by $\\cl C$, $\\interior C$, $\\recc\nC$, $\\conv C$, and $\\cone C$ the closure, interior, recession cone,\nconvex hull and conical hull of a set $C$, respectively. A compact\nconvex set with nonempty interior is called a \\textit{convex\nbody}. The Euclidean ball with radius $r$ centered at a point~${c \\in\n\\R^n}$ is denoted by~$B_r(c)$. A point $c$ of a convex set $C$ is\ncalled an \\emph{extreme point} of $C$, if $C \\setminus\\{c\\}$ is\nconvex. The set of extreme points of $C$ is denoted by $\\ext\nC$. Extreme points are exactly the points of $C$ that cannot be\nwritten as a proper convex combination of elements of $C$\n\\cite[p.~162]{Roc70}. For finite sets $V, D \\subseteq \\R^n$, the set\n\\begin{equation}\\label{vrep}\nP = \\conv V + \\cone D\n\\end{equation}\nis called a \\emph{polyhedron}. The plus sign denotes Minkowski\naddition. The sets $V,D$ in \\eqref{vrep} are called a\n\\emph{$V$-representation} of $P$ as it is expressed in terms of its\nvertices and directions. A polyhedron can equivalently be expressed as\na finite intersection of closed halfspaces \\cite[Theorem 19.1]{Roc70},\ni.e.\n\\begin{equation}\\label{hrep}\nP = \\{x \\in \\R^n \\mid Ax \\leq b\\}\n\\end{equation}\nfor a matrix $A \\in \\R^{m \\times n}$ and a vector $b \\in \\R^m$. The\ndata $(A,b)$ are called a \\emph{$H$-representation} of $P$.  The\nextreme points of a polyhedron $P$ are called \\emph{vertices} of $P$\nand are denoted by~$\\vertices P$. For symmetric matrices\n$A_0,A_1,\\dots,A_n$ of arbitrary fixed size we define\n\\begin{equation}\n\\Ab(x) := \\sum_{i=1}^n x_iA_i \\quad \\text{and} \\quad \\A(x) := \\Ab(x) +\nA_0,\n\\end{equation}\ni.e. a linear combination of the $A_i$ and a translation of $\\Ab(x)$\nby $A_0$, respectively. We denote by $A \\mg 0$ ($A \\mgeq 0$) positive\n(semi-)definiteness of the symmetric matrix $A$. A set of the form\n$\\{x \\in \\R^n \\mid \\A(x) \\mgeq 0\\}$ is called a\n\\emph{spectrahedron}. Spectrahedra are a generalization of polyhedra\nfor which many geometric properties of polyhedra generalize nicely,\ne.g. the recession cone of a spectrahedron $C$ is obtained as $\\{x \\in\n\\R^n \\mid \\Ab(x) \\mgeq 0 \\}$, see \\cite{GR95}, whereas the recession\ncone of a polyhedron in $H$-representation is $\\{x \\in \\R^n \\mid Ax\n\\leq 0\\}$. Given a cone $K$ the set $\\polar{K} = \\{y \\in \\R^n \\mid\n\\forall x \\in K: x\\T y \\leq 0\\}$ is called the \\emph{polar cone} of\n$K$. The polar $\\polar{(\\recc{C})}$ of the recession cone of a\nspectrahedron $C$ is computed as\n\\begin{equation}\\label{EPCS}\n\\polar{(\\recc{C})} = \\cl \\left\\lbrace \\left( -A_1 \\cdot X, \\dots, -A_n\n\\cdot X \\right)\\T \\mid X \\mgeq 0 \\right\\rbrace, \n\\end{equation}\nwhere $A_i \\cdot X$ means the trace of the matrix product $A_iX$, see\n\\cite[Section 3]{GR95}. A cone $K$ is called \\emph{polyhedral} if ${K\n= \\cone D}$ for some finite set $D$ and \\emph{pointed} if ${K \\cap\n(-K) = \\{0\\}}$. A set whose recession cone is pointed is called\n\\emph{line-free}. It is well known, that a closed convex set contains\nan extreme point if and only if it is line-free, see \\cite[Corollary\n18.5.3]{Roc70}. Given nonempty sets ${C_1, C_2 \\subseteq \\R^n}$ the\n\\emph{excess of $C_1$ over $C_2$}, ${\\exc{C_1}{C_2}}$, is defined as\n\\begin{equation}\n\\exc{C_1}{C_2} = \\adjustlimits \\sup_{c_1 \\in C_1} \\inf_{c_2 \\in C_2}\n\\norm{c_1-c_2},\n\\end{equation}\nwhere~$\\norm{\\cdot}$ denotes the Euclidean norm. The \\emph{Hausdorff\ndistance between $C_1$ and $C_2$}, ${\\haus{C_1}{C_2}}$, is then\nexpressed as\n\\begin{equation}\n\\haus{C_1}{C_2} = \\max \\{\\exc{C_1}{C_2}, \\exc{C_2}{C_1}\\}.\n\\end{equation}\nIt is well known, that the Hausdorff distance defines a metric on the\nspace of nonempty compact subsets of $\\R^n$. Between unbounded sets\nthe Hausdorff distance may be infinite. A polyhedron $P$ is called an\n\\textit{$\\varepsilon$-approximation} of a convex set $C$ if\n$\\haus{P}{C} \\leq \\varepsilon$.\n","start":9679,"end":13487},{"raw_tex":"\\begin{definition}\n  Given nonempty closed convex cones~${K_1, K_2 \\subseteq \\R^n}$,\n  the \\emph{truncated Hausdorff distance between~$K_1$\n    and~$K_2$},~${\\thaus{K_1}{K_2}}$, is defined as\n  \\begin{equation}\n    \\thaus{K_1}{K_2} := \\haus{K_1 \\cap B_1(0)}{K_2 \\cap B_1(0)}.\n  \\end{equation}\n\\end{definition}\n","start":22126,"end":22436},{"raw_tex":"\\begin{definition}\\label{DEDA}\n  Given a nonempty closed convex and line-free set~${C \\subseteq\n    \\R^n}$, a line-free polyhedron~$P$ is called an\n  \\emph{${\\ed}$-approximation of~$C$} if\n  \\begin{enumerate}[(i)]\n  \\item $\\exc{\\vertices P}{C} \\leq \\varepsilon$,\n  \\item $\\thaus{\\recc{P}}{\\recc{C}} \\leq \\delta$,\n  \\item $P \\supseteq C$.\n  \\end{enumerate}\n\\end{definition}\n","start":22952,"end":23325},{"raw_tex":"\\begin{definition}\n  A sequence~$\\{C^{\\nu}\\}_{\\nu \\in \\mathbb{N}}$ of subsets of~$\\R^n$ is said\n  to \\emph{converge to~${C \\subseteq \\R^n}$ in the sense of\n    Painlevé-Kuratowski}, denoted by~${C^{\\nu} \\rightarrow C}$, if the\n  following equalities hold:\n  \\begin{equation*}\n    \\begin{aligned}\n      C &= \\left\\lbrace x \\in \\R^n \\middle\\vert \\begin{matrix} \\text{for all open\n          neighbourhoods } U \\text{ of } x,\\\\ U \\cap C^{\\nu}  \\neq \\emptyset\n          \\text{ for large enough } \\nu\\end{matrix} \\right\\rbrace \\\\\n      &= \\left\\lbrace x \\in \\R^n \\middle\\vert \\begin{matrix} \\text{for all open\n        neighbourhoods } U \\text{ of } x,\\\\ U \\cap C^{\\nu} \\neq \\emptyset\n        \\text{ for infinitely many }\\nu\\end{matrix} \\right\\rbrace.\n    \\end{aligned}\n  \\end{equation*}\n\\end{definition}\n","start":24952,"end":25750},{"raw_tex":"\\begin{theorem}[{see \\cite[p.~120]{RW98}}]\\label{TPKCHD}\n  A sequence~$C^{\\nu}$ of nonempty closed and convex sets converges to~$C$\n  in the sense of Painlevé-Kuratowski if and only if there exist~$x\n  \\in \\R^n$ and~$r_0 \\in \\R$, such that for all~$r \\geq r_0$ it holds, that\n  \\begin{equation}\n    \\haus{C^{\\nu} \\cap B_r(x)}{C \\cap B_r(x)} \\rightarrow 0.\n  \\end{equation}\n\\end{theorem}\n","start":26924,"end":27311}]},"statement":{"raw_tex":"\\begin{theorem}\\label{TPKC}\n  Let $C \\subseteq \\R^n$ be nonempty closed convex and\n  line-free. For $\\nu \\in \\mathbb{N}$ let~$P^{\\nu}$ be\n  an~$(\\varepsilon^{\\nu},\\delta^{\\nu})$-approximation of\n  $C$. If~${(\\varepsilon^{\\nu},\\delta^{\\nu}) \\rightarrow (0,0)}$,\n  then~${P^{\\nu} \\rightarrow C}$ in the sense of Painlevé-Kuratowski.\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\\label{TPKC}\n  Let $C \\subseteq \\R^n$ be nonempty closed convex and\n  line-free. For $\\nu \\in \\mathbb{N}$ let~$P^{\\nu}$ be\n  an~$(\\varepsilon^{\\nu},\\delta^{\\nu})$-approximation of\n  $C$. If~${(\\varepsilon^{\\nu},\\delta^{\\nu}) \\rightarrow (0,0)}$,\n  then~${P^{\\nu} \\rightarrow C}$ in the sense of Painlevé-Kuratowski.\n\\end{theorem}","start":36459,"end":36803}]},"proof":{"raw_tex":"\\begin{proposition}\\label{PHDA}\n  Let $C \\subseteq \\R^n$ be nonempty closed convex and line-free\n  and let~$P$ be an~$\\ed$-approximation of~$C$. Then for every~$x \\in\n  \\conv \\vertices P$ and~$r \\geq \\varepsilon$ it holds true, that\n  \\begin{equation}\n    \\haus{P \\cap B_r(x)}{C \\cap B_r(x)} \\leq 2\\left(\n      \\varepsilon+\\delta\\left(r+\\norm{x-v}\\right)\\right)\n  \\end{equation}\n  for some~$v \\in \\conv\\vertices P$. In particular, if~$\\haus{P \\cap\n    B_r(x)}{C \\cap B_r(x)}$ is attained as~$\\norm{p-c}$ with~$p \\in P$,\n  then~$p = v + td$ for some~$d \\in \\recc{P}$ and~$t \\geq 0$.\n\\end{proposition}\n\n\\begin{proof}\n  Denote by $\\bar{P}$, $\\bar{C}$, and $V$, $P \\cap B_r(x)$, $C \\cap\n  B_r(x)$, and $\\conv \\vertices P$, respectively. Since~$\\bar{P},\n  \\bar{C}$ are nonempty convex and compact,~$\\haus{\\bar{P}}{\\bar{C}} =\n  \\norm{p^*-c^*}$ for some~$p^* \\in \\bar{P}$ and~$c^* \\in\n  \\bar{C}$. For~$\\lambda \\in [0,1]$ let~$z(\\lambda) = \\lambda p^* +\n  (1-\\lambda) x$. We distinguish two cases. First, assume~$p^* \\in\n  V$. Then~$z(\\lambda) \\in V$ for every~${\\lambda \\in [0,1]}$ and, due\n  to (i) in Definition \\ref{DEDA}, there is~${c_{\\lambda} \\in \\bar{C}}$\n  with~${\\norm{z(\\lambda)-c_{\\lambda}} \\leq\n    \\varepsilon}$. If~$\\norm{p^*-x} \\leq \\varepsilon$ then\n  \\[ \\norm{p^*-c^*} \\leq \\norm{p^*-c_0} \\leq \\norm{p^*-x} +\n    \\norm{x-c_0} \\leq 2\\varepsilon. \\] If~$\\norm{p^*-x} > \\varepsilon$,\n  set ${\\lambda}^* = \\frac{\\varepsilon}{\\norm{p^*-x}}$. Similarly, we\n  have\n  \\[ \\norm{p^*-c^*} \\leq \\norm{p^*-c_{{\\lambda}^*}} \\leq\n    \\norm{p^*-z({\\lambda}^*)} + \\norm{z({\\lambda}^*)-c_{{\\lambda}^*}} \\leq\n    2\\varepsilon. \\] Now, assume~${p^* \\notin V}$. Then there\n  exists~$\\bar{\\lambda} \\in (0,1)$, such that~$z(\\bar{\\lambda}) \\in V$\n  and~$z(\\lambda) \\notin V$ for all $\\lambda \\in\n  (\\bar{\\lambda},1]$. For~${\\lambda \\in (\\bar{\\lambda},1]}$,\n  $z(\\lambda)$ can be written as ${z(\\lambda) = v_{\\lambda} +\n    t_{\\lambda}d_{\\lambda}}$ with $v_{\\lambda} = \\argmin \\{\n      \\norm{z(\\bar{\\lambda}) - v} \\;\\mid\\;$ $v \\in \\left(\\{z(\\lambda)\\} -\n        \\recc{P}\\right) \\cap V \\}$, $t_{\\lambda} \\geq 0$ and\n  $d_{\\lambda} \\in \\recc{P}$, $\\norm{d_{\\lambda}} = 1$. By the\n  definition of $\\ed$-approximation there exist~$c_{\\lambda} \\in C$\n  and~$\\bar{d_{\\lambda}} \\in \\recc{C}$ such\n  that~$\\norm{v_{\\lambda}-c_{\\lambda}} \\leq \\varepsilon$\n  and~$\\norm{d_{\\lambda}-\\bar{d_{\\lambda}}} \\leq \\delta$. Now we have\n  \\begin{equation}\\label{proof_PHDA_1}\n    \\norm{z(\\lambda) - (c_{\\lambda} + t_{\\lambda}\\bar{d_{\\lambda}})} \\leq\n    \\norm{v_{\\lambda}-c_{\\lambda}} + t_{\\lambda}\n    \\norm{d_{\\lambda}-\\bar{d_{\\lambda}}} \\leq \\varepsilon +\n    t_{\\lambda}\\delta.\n  \\end{equation}\n  Furthermore, $t_{\\lambda} =\n  \\norm{z(\\lambda)-v_{\\lambda}}$, because $\\norm{d_{\\lambda}} =\n  1$. Hence,\n  \\begin{equation}\\label{proof_PHDA_2}\n    \\begin{aligned} t_{\\lambda} &\\leq\n      \\norm{z(\\lambda)-z(\\bar{\\lambda})} + \\norm{z(\\bar{\\lambda}) -\n        v_{\\lambda}} \\leq \\norm{z(\\lambda)-z(\\bar{\\lambda})} +\n      \\norm{z(\\bar{\\lambda})-v_1} \\\\ &\\leq\n      \\norm{z(\\lambda)-z(\\bar{\\lambda})} + \\norm{z(\\bar{\\lambda})-x} +\n      \\norm{x -v_1} \\\\ &= \\norm{z(\\lambda)-x} + \\norm{x-v_1}\\\\ &\\leq r\n      + \\norm{x-v_1}.\n    \\end{aligned}\n  \\end{equation}\nIf $\\frac{r-\\varepsilon}{r+\\norm{x-v_1}} < \\delta \\leq 1$ then \\[\n    \\norm{p^*-c^*} \\leq \\norm{p^*-c_0} \\leq \\norm{p^*-x} +\n    \\norm{x-c_0} \\leq r + \\varepsilon \\leq 2(\\varepsilon + \\delta\n    (r+\\norm{x-v_1})). \\] Otherwise, the last inequality is\n  violated. In this case, let~${\\lambda}^* = 1-\\frac{\\varepsilon +\n    \\delta(r\\norm{x-v_1})}{\\norm{p^*-x}}$. If~${\\lambda}^* \\leq\n  \\bar{\\lambda}$ then $z({\\lambda}^*) \\in V$ and there\n  exists~$c_{{\\lambda}^*} \\in C$ with~$\\norm{z({\\lambda}^*) -\n    c_{{\\lambda}^*}} \\leq \\varepsilon$. Therefore,\n  \\[ \\norm{p^*-c^*} \\leq \\norm{p^*-z({\\lambda}^*)} +\n    \\norm{z({\\lambda}^*)-c_{{\\lambda}^*}} \\leq 2 \\varepsilon + \\delta\n    (r+\\norm{x-v_1}). \\] If~${\\lambda}^* > \\bar{\\lambda}$ then,\n  according to \\eqref{proof_PHDA_1} and \\eqref{proof_PHDA_2}, there\n  exists~$c \\in C$ such that $\\norm{z({\\lambda}^*)-c} \\leq \\varepsilon\n  + \\delta(r+\\norm{x-v_1})$. Altogether this yields \\[ \\norm{p^*-c^*}\n    \\leq \\norm{p^*-c} \\leq \\norm{p^*-z({\\lambda}^*)} +\n    \\norm{z({\\lambda}^*)-c} \\leq 2 \\left( \\varepsilon + \\delta\n      (r+\\norm{x-v_1})\\right), \\] which completes the proof.\n\\end{proof}\n\n\\begin{lemma}\\label{LBS}\n  Let $C \\subseteq \\R^n$ be nonempty closed and convex and let there\n  be sequences~$v^{\\nu}$,~$r^{\\nu}$ such that~$\\inf_{c \\in C}\n  \\norm{v^{\\nu}-c} \\rightarrow 0$, $\\inf_{r \\in \\recc{C}}\n  \\norm{r^{\\nu}-r} \\rightarrow 0$, and $v^{\\nu}+r^{\\nu} \\in\n    B_M(x)$ for some $M \\geq 0$ and $x \\in \\R^n$. If $C$ is\n  line-free, then $\\norm{v^{\\nu}}$ is bounded.\n\\end{lemma}\n\n\\begin{proof}\n  Assume that $\\norm{v^{\\nu}}$ is unbounded. This implies\n  that~$\\norm{r^{\\nu}}$ is also unbounded and $\\recc{C} \\neq\n  \\{0\\}$. Without loss of generality let $r^{\\nu} \\neq 0$ for all\n  $\\nu$. Then $d^{\\nu} := r^{\\nu}/\\norm{r^{\\nu}}$ is bounded and has a\n  convergent subsequence. Without loss of generality we can assume\n  that $d^{\\nu} \\rightarrow d \\in \\recc{C}$. We will show, that $-d\n  \\in \\recc{C}$. Therefore let $c \\in C$, $t \\geq 0$ and define $y^{\\nu} =\n  \\argmin_{y \\in C} \\norm{v^{\\nu}-y}$. By the triangle inequality it\n  holds true, that\n  \\begin{equation}\\label{proof_LBS}\n    \\norm{c-r^{\\nu}-y^{\\nu}} \\leq M + \\norm{c-x} +\n    \\norm{v^{\\nu}-y^{\\nu}} =: \\ols{M}^{\\nu}.\n  \\end{equation}\n  Note, that $\\ols{M}^{\\nu}$ is bounded from above by some $\\ols{M}$, because\n  $\\norm{v^{\\nu}-y^{\\nu}} \\rightarrow 0$. For every $T \\geq 0$ there\n  exists some $\\nu_T$, such that $\\norm{r^{\\nu_T}} \\geq T$. Let $T\n  \\geq t$ and define\n  \\[ \\bar{y} := \\frac{t}{\\norm{r^{\\nu_T}}} y^{\\nu_T} + \\left(\n      1-\\frac{t}{\\norm{r^{\\nu_T}}} \\right) c \\in C. \\]\n  Putting it all together one gets\n  \\begin{equation*}\n    \\begin{aligned}\n      \\inf_{y \\in C} \\norm{(c-td^{\\nu_T})-y} &\\leq\n      \\norm{(c-td^{\\nu_T})-\\bar{y}} \\\\\n      &= \\norm{\\frac{t}{\\norm{r^{\\nu_T}}}c\n        -td^{\\nu_T}-\\frac{t}{\\norm{r^{\\nu_T}}}y^{\\nu_T}} \\\\\n      &= \\frac{t}{\\norm{r^{\\nu_T}}} \\norm{c-r^{\\nu_T}-y^{\\nu_T}} \\\\\n      &\\leq \\frac{t}{T} \\ols{M},\n    \\end{aligned}\n  \\end{equation*}\n  where the last inequality holds due to \\eqref{proof_LBS} and\n  boundedness of $\\ols{M}^{\\nu}$. Since $C$ is closed and $d^{\\nu}\n  \\rightarrow d \\in \\recc{C}$, taking the limit $T \\rightarrow\n  +\\infty$ yields that $c-td \\in C$. This is a contradiction to the\n  pointedness of $\\recc{C}$.\n\\end{proof}\n\n\\begin{proposition}\\label{PEP}\n  Let $C \\subseteq \\R^n$ be nonempty closed convex and\n  line-free. For $\\nu \\in \\mathbb{N}$ let~$P^{\\nu}$ be an\n  $(\\varepsilon^{\\nu},\\delta^{\\nu})$-approximation of\n  $C$. If~${(\\varepsilon^{\\nu},\\delta^{\\nu}) \\rightarrow (0,0)}$, then\n  for every extreme point~$c$ of $C$ there exists a sequence~${x^{\\nu}\n    \\rightarrow c}$ such that $x^{\\nu} \\in \\conv \\vertices P^{\\nu}$.\n\\end{proposition}\n\n\\begin{proof} Since $C$ is line-free, it has at least one extreme\n  point. Let $c$ be one such extreme point. Assume that for every\n  sequence $x^{\\nu}$ with~${x^{\\nu} \\in \\conv \\vertices P^{\\nu}}$ there\n  exists a $\\gamma > 0$, such that~${\\norm{x^{\\nu}-c} > \\gamma}$ for\n  infinitely many $\\nu$. Then, without loss of generality, there exists\n  one such sequence such that~$\\norm{x^{\\nu}-c} > \\gamma$ for every\n  $\\nu$ and, since $C \\subseteq P^{\\nu}$, $c=x^{\\nu}+r^{\\nu}$ for some\n  $r^{\\nu} \\in \\recc{P^{\\nu}}$. By Lemma \\ref{LBS} it holds that\n  $\\norm{x^{\\nu}}$ and $\\norm{r^{\\nu}}$ are bounded. Then there exist\n  subsequences $x^{\\nu_k}$, $r^{\\nu_k}$ such that\n  \\[ x^{\\nu_k} \\rightarrow x \\in C, \\quad r^{\\nu_k} \\rightarrow r \\in\n    \\recc{C}. \\] Note that $r \\neq 0$, because $\\norm{r^{\\nu}} > \\gamma$\n  for all $\\nu$. Finally,\n  \\[ c=x+r=\\frac{1}{2}\\left( x+2r \\right) + \\frac{1}{2} x. \\] This is\n  a contradiction to $c$ being an extreme point of $C$.\n\\end{proof}\n\n\\begin{proof}\n  By Theorem \\ref{TPKCHD} we must show that there exist $c \\in \\R^n$\n  and $r_0 \\geq 0$ such that for all $r \\geq r_0$ it holds $\\haus{P^{\\nu} \\cap B_r(c)}{C \\cap B_r(c)}\n  \\rightarrow 0$. Let~${r \\geq \\max_{\\nu \\in\n      \\mathbb{N}} \\varepsilon^{\\nu}}$ and let $c$ be an extreme point of $C$,\n  which exists, because $C$ contains no lines. By Proposition \\ref{PEP}\n  there exists a sequence~${x^{\\nu} \\rightarrow c}$ with~${x^{\\nu} \\in\n    \\conv \\vertices P^{\\nu}}$. Applying the triangle inequality and\n  Proposition \\ref{PHDA} yields\n  \\begin{equation*}\n    \\begin{aligned}\n      & & &  \\haus{P^{\\nu} \\cap B_r(c)}{C \\cap B_r(c)} \\\\\n      & \\leq & & \\haus{P^{\\nu} \\cap B_r(c)}{P^{\\nu} \\cap B_r(x^{\\nu})}\n      \\\\ & & + & \\haus{P^{\\nu} \\cap B_r(x^{\\nu})}{C \\cap B_r(x^{\\nu})}\n      \\\\ & & + & \\haus{C \\cap B_r(x^{\\nu})}{C \\cap B_r(c)} \\\\\n      & \\leq & & \\haus{P^{\\nu} \\cap B_r(c)}{P^{\\nu} \\cap B_r(x^{\\nu})}\n      \\\\ & & + & 2(\\varepsilon^{\\nu}+\\delta^{\\nu}(r+\\norm{x^{\\nu}-v^{\\nu}}))\n      \\\\ & & + & \\haus{C \\cap B_r(x^{\\nu})}{C \\cap B_r(c)},\n   \\end{aligned}\n \\end{equation*}\n for some $v^{\\nu} \\in \\conv \\vertices P^{\\nu}$. The first and third\n term in this sum converge to zero as~${x^{\\nu} \\rightarrow c}$. It\n remains to show that~$\\norm{x^{\\nu}-v^{\\nu}}$ is\n bounded. Since $C \\subseteq P^{\\nu}$, the distance ${\\haus{P^{\\nu}\n   \\cap B_r(x^{\\nu})}{C \\cap B_r(x^{\\nu})}}$ is attained as\n $\\exc{P^{\\nu} \\cap B_r(x^{\\nu})}{C \\cap B_r(x^{\\nu})}$. Let the\n supremum be attained by~$p^{\\nu} \\in P^{\\nu}$. Then $p^{\\nu} =\n   v^{\\nu} + d^{\\nu}$ for some~$d^{\\nu} \\in \\recc{P^{\\nu}}$. It holds\n \\[ \\norm{p^{\\nu}-c} \\leq \\norm{p^{\\nu}-x^{\\nu}} + \\norm{x^{\\nu}-c}\n   \\leq r + \\max_{\\nu \\in \\mathbb{N}} \\norm{x^{\\nu}-c} < +\\infty, \\]\n i.e. $v^{\\nu}+d^{\\nu} \\in B_M(c)$ for some $M \\geq 0$. Therefore, the\n sequence $\\norm{v^{\\nu}}$ is bounded according to Lemma\n \\ref{LBS}. Hence, $\\norm{x^{\\nu}-v^{\\nu}}$ is also bounded\n and $\\haus{P^{\\nu} \\cap B_r(c)}{C \\cap B_r(c)} \\rightarrow 0$,\n which was to be proved.\n\\end{proof}","units":[{"raw_tex":"\\begin{proposition}\\label{PHDA}\n  Let $C \\subseteq \\R^n$ be nonempty closed convex and line-free\n  and let~$P$ be an~$\\ed$-approximation of~$C$. Then for every~$x \\in\n  \\conv \\vertices P$ and~$r \\geq \\varepsilon$ it holds true, that\n  \\begin{equation}\n    \\haus{P \\cap B_r(x)}{C \\cap B_r(x)} \\leq 2\\left(\n      \\varepsilon+\\delta\\left(r+\\norm{x-v}\\right)\\right)\n  \\end{equation}\n  for some~$v \\in \\conv\\vertices P$. In particular, if~$\\haus{P \\cap\n    B_r(x)}{C \\cap B_r(x)}$ is attained as~$\\norm{p-c}$ with~$p \\in P$,\n  then~$p = v + td$ for some~$d \\in \\recc{P}$ and~$t \\geq 0$.\n\\end{proposition}","start":27980,"end":28579},{"raw_tex":"\\begin{proof}\n  Denote by $\\bar{P}$, $\\bar{C}$, and $V$, $P \\cap B_r(x)$, $C \\cap\n  B_r(x)$, and $\\conv \\vertices P$, respectively. Since~$\\bar{P},\n  \\bar{C}$ are nonempty convex and compact,~$\\haus{\\bar{P}}{\\bar{C}} =\n  \\norm{p^*-c^*}$ for some~$p^* \\in \\bar{P}$ and~$c^* \\in\n  \\bar{C}$. For~$\\lambda \\in [0,1]$ let~$z(\\lambda) = \\lambda p^* +\n  (1-\\lambda) x$. We distinguish two cases. First, assume~$p^* \\in\n  V$. Then~$z(\\lambda) \\in V$ for every~${\\lambda \\in [0,1]}$ and, due\n  to (i) in Definition \\ref{DEDA}, there is~${c_{\\lambda} \\in \\bar{C}}$\n  with~${\\norm{z(\\lambda)-c_{\\lambda}} \\leq\n    \\varepsilon}$. If~$\\norm{p^*-x} \\leq \\varepsilon$ then\n  \\[ \\norm{p^*-c^*} \\leq \\norm{p^*-c_0} \\leq \\norm{p^*-x} +\n    \\norm{x-c_0} \\leq 2\\varepsilon. \\] If~$\\norm{p^*-x} > \\varepsilon$,\n  set ${\\lambda}^* = \\frac{\\varepsilon}{\\norm{p^*-x}}$. Similarly, we\n  have\n  \\[ \\norm{p^*-c^*} \\leq \\norm{p^*-c_{{\\lambda}^*}} \\leq\n    \\norm{p^*-z({\\lambda}^*)} + \\norm{z({\\lambda}^*)-c_{{\\lambda}^*}} \\leq\n    2\\varepsilon. \\] Now, assume~${p^* \\notin V}$. Then there\n  exists~$\\bar{\\lambda} \\in (0,1)$, such that~$z(\\bar{\\lambda}) \\in V$\n  and~$z(\\lambda) \\notin V$ for all $\\lambda \\in\n  (\\bar{\\lambda},1]$. For~${\\lambda \\in (\\bar{\\lambda},1]}$,\n  $z(\\lambda)$ can be written as ${z(\\lambda) = v_{\\lambda} +\n    t_{\\lambda}d_{\\lambda}}$ with $v_{\\lambda} = \\argmin \\{\n      \\norm{z(\\bar{\\lambda}) - v} \\;\\mid\\;$ $v \\in \\left(\\{z(\\lambda)\\} -\n        \\recc{P}\\right) \\cap V \\}$, $t_{\\lambda} \\geq 0$ and\n  $d_{\\lambda} \\in \\recc{P}$, $\\norm{d_{\\lambda}} = 1$. By the\n  definition of $\\ed$-approximation there exist~$c_{\\lambda} \\in C$\n  and~$\\bar{d_{\\lambda}} \\in \\recc{C}$ such\n  that~$\\norm{v_{\\lambda}-c_{\\lambda}} \\leq \\varepsilon$\n  and~$\\norm{d_{\\lambda}-\\bar{d_{\\lambda}}} \\leq \\delta$. Now we have\n  \\begin{equation}\\label{proof_PHDA_1}\n    \\norm{z(\\lambda) - (c_{\\lambda} + t_{\\lambda}\\bar{d_{\\lambda}})} \\leq\n    \\norm{v_{\\lambda}-c_{\\lambda}} + t_{\\lambda}\n    \\norm{d_{\\lambda}-\\bar{d_{\\lambda}}} \\leq \\varepsilon +\n    t_{\\lambda}\\delta.\n  \\end{equation}\n  Furthermore, $t_{\\lambda} =\n  \\norm{z(\\lambda)-v_{\\lambda}}$, because $\\norm{d_{\\lambda}} =\n  1$. Hence,\n  \\begin{equation}\\label{proof_PHDA_2}\n    \\begin{aligned} t_{\\lambda} &\\leq\n      \\norm{z(\\lambda)-z(\\bar{\\lambda})} + \\norm{z(\\bar{\\lambda}) -\n        v_{\\lambda}} \\leq \\norm{z(\\lambda)-z(\\bar{\\lambda})} +\n      \\norm{z(\\bar{\\lambda})-v_1} \\\\ &\\leq\n      \\norm{z(\\lambda)-z(\\bar{\\lambda})} + \\norm{z(\\bar{\\lambda})-x} +\n      \\norm{x -v_1} \\\\ &= \\norm{z(\\lambda)-x} + \\norm{x-v_1}\\\\ &\\leq r\n      + \\norm{x-v_1}.\n    \\end{aligned}\n  \\end{equation}\nIf $\\frac{r-\\varepsilon}{r+\\norm{x-v_1}} < \\delta \\leq 1$ then \\[\n    \\norm{p^*-c^*} \\leq \\norm{p^*-c_0} \\leq \\norm{p^*-x} +\n    \\norm{x-c_0} \\leq r + \\varepsilon \\leq 2(\\varepsilon + \\delta\n    (r+\\norm{x-v_1})). \\] Otherwise, the last inequality is\n  violated. In this case, let~${\\lambda}^* = 1-\\frac{\\varepsilon +\n    \\delta(r\\norm{x-v_1})}{\\norm{p^*-x}}$. If~${\\lambda}^* \\leq\n  \\bar{\\lambda}$ then $z({\\lambda}^*) \\in V$ and there\n  exists~$c_{{\\lambda}^*} \\in C$ with~$\\norm{z({\\lambda}^*) -\n    c_{{\\lambda}^*}} \\leq \\varepsilon$. Therefore,\n  \\[ \\norm{p^*-c^*} \\leq \\norm{p^*-z({\\lambda}^*)} +\n    \\norm{z({\\lambda}^*)-c_{{\\lambda}^*}} \\leq 2 \\varepsilon + \\delta\n    (r+\\norm{x-v_1}). \\] If~${\\lambda}^* > \\bar{\\lambda}$ then,\n  according to \\eqref{proof_PHDA_1} and \\eqref{proof_PHDA_2}, there\n  exists~$c \\in C$ such that $\\norm{z({\\lambda}^*)-c} \\leq \\varepsilon\n  + \\delta(r+\\norm{x-v_1})$. Altogether this yields \\[ \\norm{p^*-c^*}\n    \\leq \\norm{p^*-c} \\leq \\norm{p^*-z({\\lambda}^*)} +\n    \\norm{z({\\lambda}^*)-c} \\leq 2 \\left( \\varepsilon + \\delta\n      (r+\\norm{x-v_1})\\right), \\] which completes the proof.\n\\end{proof}","start":28581,"end":32337},{"raw_tex":"\\begin{lemma}\\label{LBS}\n  Let $C \\subseteq \\R^n$ be nonempty closed and convex and let there\n  be sequences~$v^{\\nu}$,~$r^{\\nu}$ such that~$\\inf_{c \\in C}\n  \\norm{v^{\\nu}-c} \\rightarrow 0$, $\\inf_{r \\in \\recc{C}}\n  \\norm{r^{\\nu}-r} \\rightarrow 0$, and $v^{\\nu}+r^{\\nu} \\in\n    B_M(x)$ for some $M \\geq 0$ and $x \\in \\R^n$. If $C$ is\n  line-free, then $\\norm{v^{\\nu}}$ is bounded.\n\\end{lemma}","start":32411,"end":32803},{"raw_tex":"\\begin{proof}\n  Assume that $\\norm{v^{\\nu}}$ is unbounded. This implies\n  that~$\\norm{r^{\\nu}}$ is also unbounded and $\\recc{C} \\neq\n  \\{0\\}$. Without loss of generality let $r^{\\nu} \\neq 0$ for all\n  $\\nu$. Then $d^{\\nu} := r^{\\nu}/\\norm{r^{\\nu}}$ is bounded and has a\n  convergent subsequence. Without loss of generality we can assume\n  that $d^{\\nu} \\rightarrow d \\in \\recc{C}$. We will show, that $-d\n  \\in \\recc{C}$. Therefore let $c \\in C$, $t \\geq 0$ and define $y^{\\nu} =\n  \\argmin_{y \\in C} \\norm{v^{\\nu}-y}$. By the triangle inequality it\n  holds true, that\n  \\begin{equation}\\label{proof_LBS}\n    \\norm{c-r^{\\nu}-y^{\\nu}} \\leq M + \\norm{c-x} +\n    \\norm{v^{\\nu}-y^{\\nu}} =: \\ols{M}^{\\nu}.\n  \\end{equation}\n  Note, that $\\ols{M}^{\\nu}$ is bounded from above by some $\\ols{M}$, because\n  $\\norm{v^{\\nu}-y^{\\nu}} \\rightarrow 0$. For every $T \\geq 0$ there\n  exists some $\\nu_T$, such that $\\norm{r^{\\nu_T}} \\geq T$. Let $T\n  \\geq t$ and define\n  \\[ \\bar{y} := \\frac{t}{\\norm{r^{\\nu_T}}} y^{\\nu_T} + \\left(\n      1-\\frac{t}{\\norm{r^{\\nu_T}}} \\right) c \\in C. \\]\n  Putting it all together one gets\n  \\begin{equation*}\n    \\begin{aligned}\n      \\inf_{y \\in C} \\norm{(c-td^{\\nu_T})-y} &\\leq\n      \\norm{(c-td^{\\nu_T})-\\bar{y}} \\\\\n      &= \\norm{\\frac{t}{\\norm{r^{\\nu_T}}}c\n        -td^{\\nu_T}-\\frac{t}{\\norm{r^{\\nu_T}}}y^{\\nu_T}} \\\\\n      &= \\frac{t}{\\norm{r^{\\nu_T}}} \\norm{c-r^{\\nu_T}-y^{\\nu_T}} \\\\\n      &\\leq \\frac{t}{T} \\ols{M},\n    \\end{aligned}\n  \\end{equation*}\n  where the last inequality holds due to \\eqref{proof_LBS} and\n  boundedness of $\\ols{M}^{\\nu}$. Since $C$ is closed and $d^{\\nu}\n  \\rightarrow d \\in \\recc{C}$, taking the limit $T \\rightarrow\n  +\\infty$ yields that $c-td \\in C$. This is a contradiction to the\n  pointedness of $\\recc{C}$.\n\\end{proof}","start":32805,"end":34580},{"raw_tex":"\\begin{proposition}\\label{PEP}\n  Let $C \\subseteq \\R^n$ be nonempty closed convex and\n  line-free. For $\\nu \\in \\mathbb{N}$ let~$P^{\\nu}$ be an\n  $(\\varepsilon^{\\nu},\\delta^{\\nu})$-approximation of\n  $C$. If~${(\\varepsilon^{\\nu},\\delta^{\\nu}) \\rightarrow (0,0)}$, then\n  for every extreme point~$c$ of $C$ there exists a sequence~${x^{\\nu}\n    \\rightarrow c}$ such that $x^{\\nu} \\in \\conv \\vertices P^{\\nu}$.\n\\end{proposition}","start":35010,"end":35436},{"raw_tex":"\\begin{proof} Since $C$ is line-free, it has at least one extreme\n  point. Let $c$ be one such extreme point. Assume that for every\n  sequence $x^{\\nu}$ with~${x^{\\nu} \\in \\conv \\vertices P^{\\nu}}$ there\n  exists a $\\gamma > 0$, such that~${\\norm{x^{\\nu}-c} > \\gamma}$ for\n  infinitely many $\\nu$. Then, without loss of generality, there exists\n  one such sequence such that~$\\norm{x^{\\nu}-c} > \\gamma$ for every\n  $\\nu$ and, since $C \\subseteq P^{\\nu}$, $c=x^{\\nu}+r^{\\nu}$ for some\n  $r^{\\nu} \\in \\recc{P^{\\nu}}$. By Lemma \\ref{LBS} it holds that\n  $\\norm{x^{\\nu}}$ and $\\norm{r^{\\nu}}$ are bounded. Then there exist\n  subsequences $x^{\\nu_k}$, $r^{\\nu_k}$ such that\n  \\[ x^{\\nu_k} \\rightarrow x \\in C, \\quad r^{\\nu_k} \\rightarrow r \\in\n    \\recc{C}. \\] Note that $r \\neq 0$, because $\\norm{r^{\\nu}} > \\gamma$\n  for all $\\nu$. Finally,\n  \\[ c=x+r=\\frac{1}{2}\\left( x+2r \\right) + \\frac{1}{2} x. \\] This is\n  a contradiction to $c$ being an extreme point of $C$.\n\\end{proof}","start":35438,"end":36413},{"raw_tex":"\\begin{proof}\n  By Theorem \\ref{TPKCHD} we must show that there exist $c \\in \\R^n$\n  and $r_0 \\geq 0$ such that for all $r \\geq r_0$ it holds $\\haus{P^{\\nu} \\cap B_r(c)}{C \\cap B_r(c)}\n  \\rightarrow 0$. Let~${r \\geq \\max_{\\nu \\in\n      \\mathbb{N}} \\varepsilon^{\\nu}}$ and let $c$ be an extreme point of $C$,\n  which exists, because $C$ contains no lines. By Proposition \\ref{PEP}\n  there exists a sequence~${x^{\\nu} \\rightarrow c}$ with~${x^{\\nu} \\in\n    \\conv \\vertices P^{\\nu}}$. Applying the triangle inequality and\n  Proposition \\ref{PHDA} yields\n  \\begin{equation*}\n    \\begin{aligned}\n      & & &  \\haus{P^{\\nu} \\cap B_r(c)}{C \\cap B_r(c)} \\\\\n      & \\leq & & \\haus{P^{\\nu} \\cap B_r(c)}{P^{\\nu} \\cap B_r(x^{\\nu})}\n      \\\\ & & + & \\haus{P^{\\nu} \\cap B_r(x^{\\nu})}{C \\cap B_r(x^{\\nu})}\n      \\\\ & & + & \\haus{C \\cap B_r(x^{\\nu})}{C \\cap B_r(c)} \\\\\n      & \\leq & & \\haus{P^{\\nu} \\cap B_r(c)}{P^{\\nu} \\cap B_r(x^{\\nu})}\n      \\\\ & & + & 2(\\varepsilon^{\\nu}+\\delta^{\\nu}(r+\\norm{x^{\\nu}-v^{\\nu}}))\n      \\\\ & & + & \\haus{C \\cap B_r(x^{\\nu})}{C \\cap B_r(c)},\n   \\end{aligned}\n \\end{equation*}\n for some $v^{\\nu} \\in \\conv \\vertices P^{\\nu}$. The first and third\n term in this sum converge to zero as~${x^{\\nu} \\rightarrow c}$. It\n remains to show that~$\\norm{x^{\\nu}-v^{\\nu}}$ is\n bounded. Since $C \\subseteq P^{\\nu}$, the distance ${\\haus{P^{\\nu}\n   \\cap B_r(x^{\\nu})}{C \\cap B_r(x^{\\nu})}}$ is attained as\n $\\exc{P^{\\nu} \\cap B_r(x^{\\nu})}{C \\cap B_r(x^{\\nu})}$. Let the\n supremum be attained by~$p^{\\nu} \\in P^{\\nu}$. Then $p^{\\nu} =\n   v^{\\nu} + d^{\\nu}$ for some~$d^{\\nu} \\in \\recc{P^{\\nu}}$. It holds\n \\[ \\norm{p^{\\nu}-c} \\leq \\norm{p^{\\nu}-x^{\\nu}} + \\norm{x^{\\nu}-c}\n   \\leq r + \\max_{\\nu \\in \\mathbb{N}} \\norm{x^{\\nu}-c} < +\\infty, \\]\n i.e. $v^{\\nu}+d^{\\nu} \\in B_M(c)$ for some $M \\geq 0$. Therefore, the\n sequence $\\norm{v^{\\nu}}$ is bounded according to Lemma\n \\ref{LBS}. Hence, $\\norm{x^{\\nu}-v^{\\nu}}$ is also bounded\n and $\\haus{P^{\\nu} \\cap B_r(c)}{C \\cap B_r(c)} \\rightarrow 0$,\n which was to be proved.\n\\end{proof}","start":36805,"end":38839}]},"nodes":[{"id":"S001","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"集合运算与欧氏球","source_units":[{"raw_tex":"Throughout this article we denote by $\\cl C$, $\\interior C$, $\\recc\nC$, $\\conv C$, and $\\cone C$ the closure, interior, recession cone,\nconvex hull and conical hull of a set $C$, respectively. A compact\nconvex set with nonempty interior is called a \\textit{convex\nbody}. The Euclidean ball with radius $r$ centered at a point~${c \\in\n\\R^n}$ is denoted by~$B_r(c)$. A point $c$ of a convex set $C$ is\n","start":9679,"end":10079}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"极点及非平凡凸组合","source_units":[{"raw_tex":"\\R^n}$ is denoted by~$B_r(c)$. A point $c$ of a convex set $C$ is\ncalled an \\emph{extreme point} of $C$, if $C \\setminus\\{c\\}$ is\nconvex. The set of extreme points of $C$ is denoted by $\\ext\nC$. Extreme points are exactly the points of $C$ that cannot be\nwritten as a proper convex combination of elements of $C$\n\\cite[p.~162]{Roc70}. For finite sets $V, D \\subseteq \\R^n$, the set\n","start":10013,"end":10395}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":["hrep","vrep"],"declared_external":false,"source_only":false,"title":"多面体表示与顶点","source_units":[{"raw_tex":"\\cite[p.~162]{Roc70}. For finite sets $V, D \\subseteq \\R^n$, the set\n\\begin{equation}\\label{vrep}\nP = \\conv V + \\cone D\n\\end{equation}\nis called a \\emph{polyhedron}. The plus sign denotes Minkowski\naddition. The sets $V,D$ in \\eqref{vrep} are called a\n\\emph{$V$-representation} of $P$ as it is expressed in terms of its\nvertices and directions. A polyhedron can equivalently be expressed as\na finite intersection of closed halfspaces \\cite[Theorem 19.1]{Roc70},\ni.e.\n\\begin{equation}\\label{hrep}\nP = \\{x \\in \\R^n \\mid Ax \\leq b\\}\n\\end{equation}\nfor a matrix $A \\in \\R^{m \\times n}$ and a vector $b \\in \\R^m$. The\ndata $(A,b)$ are called a \\emph{$H$-representation} of $P$.  The\nextreme points of a polyhedron $P$ are called \\emph{vertices} of $P$\nand are denoted by~$\\vertices P$. For symmetric matrices\n","start":10326,"end":11130}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["EPCS"],"declared_external":false,"source_only":false,"title":"谱面体与极锥背景","source_units":[{"raw_tex":"and are denoted by~$\\vertices P$. For symmetric matrices\n$A_0,A_1,\\dots,A_n$ of arbitrary fixed size we define\n\\begin{equation}\n\\Ab(x) := \\sum_{i=1}^n x_iA_i \\quad \\text{and} \\quad \\A(x) := \\Ab(x) +\nA_0,\n\\end{equation}\ni.e. a linear combination of the $A_i$ and a translation of $\\Ab(x)$\nby $A_0$, respectively. We denote by $A \\mg 0$ ($A \\mgeq 0$) positive\n(semi-)definiteness of the symmetric matrix $A$. A set of the form\n$\\{x \\in \\R^n \\mid \\A(x) \\mgeq 0\\}$ is called a\n\\emph{spectrahedron}. Spectrahedra are a generalization of polyhedra\nfor which many geometric properties of polyhedra generalize nicely,\ne.g. the recession cone of a spectrahedron $C$ is obtained as $\\{x \\in\n\\R^n \\mid \\Ab(x) \\mgeq 0 \\}$, see \\cite{GR95}, whereas the recession\ncone of a polyhedron in $H$-representation is $\\{x \\in \\R^n \\mid Ax\n\\leq 0\\}$. Given a cone $K$ the set $\\polar{K} = \\{y \\in \\R^n \\mid\n\\forall x \\in K: x\\T y \\leq 0\\}$ is called the \\emph{polar cone} of\n$K$. The polar $\\polar{(\\recc{C})}$ of the recession cone of a\nspectrahedron $C$ is computed as\n\\begin{equation}\\label{EPCS}\n\\polar{(\\recc{C})} = \\cl \\left\\lbrace \\left( -A_1 \\cdot X, \\dots, -A_n\n\\cdot X \\right)\\T \\mid X \\mgeq 0 \\right\\rbrace, \n\\end{equation}\nwhere $A_i \\cdot X$ means the trace of the matrix product $A_iX$, see\n\\cite[Section 3]{GR95}. A cone $K$ is called \\emph{polyhedral} if ${K\n","start":11073,"end":12426}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"尖锥与无直线集合","source_units":[{"raw_tex":"\\cite[Section 3]{GR95}. A cone $K$ is called \\emph{polyhedral} if ${K\n= \\cone D}$ for some finite set $D$ and \\emph{pointed} if ${K \\cap\n(-K) = \\{0\\}}$. A set whose recession cone is pointed is called\n\\emph{line-free}. It is well known, that a closed convex set contains\n","start":12356,"end":12627}]},{"id":"S006","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表明确允许：存在极点的标准结论，Roc70, Corollary 18.5.3；源文第249–251行。"],"source_only":false,"title":"无直线闭凸集存在极点","source_units":[{"raw_tex":"\\emph{line-free}. It is well known, that a closed convex set contains\nan extreme point if and only if it is line-free, see \\cite[Corollary\n18.5.3]{Roc70}. Given nonempty sets ${C_1, C_2 \\subseteq \\R^n}$ the\n","start":12557,"end":12764}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"有向超距","source_units":[{"raw_tex":"18.5.3]{Roc70}. Given nonempty sets ${C_1, C_2 \\subseteq \\R^n}$ the\n\\emph{excess of $C_1$ over $C_2$}, ${\\exc{C_1}{C_2}}$, is defined as\n\\begin{equation}\n\\exc{C_1}{C_2} = \\adjustlimits \\sup_{c_1 \\in C_1} \\inf_{c_2 \\in C_2}\n\\norm{c_1-c_2},\n\\end{equation}\nwhere~$\\norm{\\cdot}$ denotes the Euclidean norm. The \\emph{Hausdorff\n","start":12696,"end":13019}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"Hausdorff 距离","source_units":[{"raw_tex":"where~$\\norm{\\cdot}$ denotes the Euclidean norm. The \\emph{Hausdorff\ndistance between $C_1$ and $C_2$}, ${\\haus{C_1}{C_2}}$, is then\nexpressed as\n\\begin{equation}\n\\haus{C_1}{C_2} = \\max \\{\\exc{C_1}{C_2}, \\exc{C_2}{C_1}\\}.\n\\end{equation}\nIt is well known, that the Hausdorff distance defines a metric on the\nspace of nonempty compact subsets of $\\R^n$. Between unbounded sets\nthe Hausdorff distance may be infinite. A polyhedron $P$ is called an\n","start":12950,"end":13395}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"传统单误差逼近","source_units":[{"raw_tex":"the Hausdorff distance may be infinite. A polyhedron $P$ is called an\n\\textit{$\\varepsilon$-approximation} of a convex set $C$ if\n$\\haus{P}{C} \\leq \\varepsilon$.\n","start":13325,"end":13487}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"锥的截断距离","source_units":[{"raw_tex":"\\begin{definition}\n  Given nonempty closed convex cones~${K_1, K_2 \\subseteq \\R^n}$,\n  the \\emph{truncated Hausdorff distance between~$K_1$\n    and~$K_2$},~${\\thaus{K_1}{K_2}}$, is defined as\n  \\begin{equation}\n    \\thaus{K_1}{K_2} := \\haus{K_1 \\cap B_1(0)}{K_2 \\cap B_1(0)}.\n  \\end{equation}\n\\end{definition}\n","start":22126,"end":22436}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":["DEDA"],"declared_external":false,"source_only":false,"title":"双误差多面体逼近","source_units":[{"raw_tex":"\\begin{definition}\\label{DEDA}\n  Given a nonempty closed convex and line-free set~${C \\subseteq\n    \\R^n}$, a line-free polyhedron~$P$ is called an\n  \\emph{${\\ed}$-approximation of~$C$} if\n  \\begin{enumerate}[(i)]\n  \\item $\\exc{\\vertices P}{C} \\leq \\varepsilon$,\n  \\item $\\thaus{\\recc{P}}{\\recc{C}} \\leq \\delta$,\n  \\item $P \\supseteq C$.\n  \\end{enumerate}\n\\end{definition}\n","start":22952,"end":23325}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["DEDA"],"declared_external":false,"source_only":false,"title":"顶点误差条件","source_units":[{"raw_tex":"  \\begin{enumerate}[(i)]\n  \\item $\\exc{\\vertices P}{C} \\leq \\varepsilon$,\n","start":23141,"end":23215}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["DEDA"],"declared_external":false,"source_only":false,"title":"衰退锥误差条件","source_units":[{"raw_tex":"  \\item $\\thaus{\\recc{P}}{\\recc{C}} \\leq \\delta$,\n","start":23215,"end":23265}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":["DEDA"],"declared_external":false,"source_only":false,"title":"外包含条件","source_units":[{"raw_tex":"  \\item $P \\supseteq C$.\n","start":23265,"end":23290}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"集合序列的极限定义","source_units":[{"raw_tex":"\\begin{definition}\n  A sequence~$\\{C^{\\nu}\\}_{\\nu \\in \\mathbb{N}}$ of subsets of~$\\R^n$ is said\n  to \\emph{converge to~${C \\subseteq \\R^n}$ in the sense of\n    Painlevé-Kuratowski}, denoted by~${C^{\\nu} \\rightarrow C}$, if the\n  following equalities hold:\n  \\begin{equation*}\n    \\begin{aligned}\n      C &= \\left\\lbrace x \\in \\R^n \\middle\\vert \\begin{matrix} \\text{for all open\n          neighbourhoods } U \\text{ of } x,\\\\ U \\cap C^{\\nu}  \\neq \\emptyset\n          \\text{ for large enough } \\nu\\end{matrix} \\right\\rbrace \\\\\n      &= \\left\\lbrace x \\in \\R^n \\middle\\vert \\begin{matrix} \\text{for all open\n        neighbourhoods } U \\text{ of } x,\\\\ U \\cap C^{\\nu} \\neq \\emptyset\n        \\text{ for infinitely many }\\nu\\end{matrix} \\right\\rbrace.\n    \\end{aligned}\n  \\end{equation*}\n\\end{definition}\n","start":24952,"end":25750}]},{"id":"S016","semantic_type":"theorem","kind":"theorem","labels":["TPKCHD"],"declared_external":true,"declaration_text":["原始前提列表明确允许的 B0031：RW98, p.120 的收敛刻画；文内仅重述，无文内证明。"],"source_only":false,"title":"球截断收敛刻画","source_units":[{"raw_tex":"\\begin{theorem}[{see \\cite[p.~120]{RW98}}]\\label{TPKCHD}\n  A sequence~$C^{\\nu}$ of nonempty closed and convex sets converges to~$C$\n  in the sense of Painlevé-Kuratowski if and only if there exist~$x\n  \\in \\R^n$ and~$r_0 \\in \\R$, such that for all~$r \\geq r_0$ it holds, that\n  \\begin{equation}\n    \\haus{C^{\\nu} \\cap B_r(x)}{C \\cap B_r(x)} \\rightarrow 0.\n  \\end{equation}\n\\end{theorem}\n","start":26924,"end":27311}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","labels":["TPKC"],"declared_external":false,"source_only":false,"title":"主定理的共同假设","source_units":[{"raw_tex":"  Let $C \\subseteq \\R^n$ be nonempty closed convex and\n  line-free. For $\\nu \\in \\mathbb{N}$ let~$P^{\\nu}$ be\n  an~$(\\varepsilon^{\\nu},\\delta^{\\nu})$-approximation of\n  $C$. If~${(\\varepsilon^{\\nu},\\delta^{\\nu}) \\rightarrow (0,0)}$,\n","start":36487,"end":36720}]},{"id":"S018","semantic_type":"theorem","kind":"theorem","labels":["TPKC"],"declared_external":false,"source_only":false,"title":"双误差逼近的集合收敛","source_units":[{"raw_tex":"\\begin{theorem}\\label{TPKC}\n  Let $C \\subseteq \\R^n$ be nonempty closed convex and\n  line-free. For $\\nu \\in \\mathbb{N}$ let~$P^{\\nu}$ be\n  an~$(\\varepsilon^{\\nu},\\delta^{\\nu})$-approximation of\n  $C$. If~${(\\varepsilon^{\\nu},\\delta^{\\nu}) \\rightarrow (0,0)}$,\n  then~${P^{\\nu} \\rightarrow C}$ in the sense of Painlevé-Kuratowski.\n\\end{theorem}","start":36459,"end":36803}]},{"id":"S019","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表中的标准有限维凸分析：到凸集距离函数的凸性；由标注者显式列出。"],"source_only":false,"title":"到凸集距离的凸性","source_units":[]},{"id":"S020","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表中的标准有限维凸分析：最近点投影。"],"source_only":false,"title":"闭凸集上的最近点","source_units":[]},{"id":"S021","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表中的标准有限维凸分析：紧性和子列论证。"],"source_only":false,"title":"有限维紧性与子列","source_units":[]},{"id":"S022","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表中的标准有限维凸分析：闭衰退锥；方向相加性质是该对象的标准定义性背景。"],"source_only":false,"title":"闭凸衰退锥的性质","source_units":[]},{"id":"S023","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表明确允许：无直线多面体分解为顶点凸包与衰退锥之和。"],"source_only":false,"title":"无直线多面体的分解","source_units":[]},{"id":"S024","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator：实数和有限维向量序列的基本极限运算；不是源文单独陈述的辅助定理。"],"source_only":false,"title":"有界乘子的极限运算","source_units":[]},{"id":"S025","semantic_type":"proposition","kind":"proposition","labels":["PHDA"],"declared_external":false,"source_only":false,"title":"球截断的误差上界","source_units":[{"raw_tex":"\\begin{proposition}\\label{PHDA}\n  Let $C \\subseteq \\R^n$ be nonempty closed convex and line-free\n  and let~$P$ be an~$\\ed$-approximation of~$C$. Then for every~$x \\in\n  \\conv \\vertices P$ and~$r \\geq \\varepsilon$ it holds true, that\n  \\begin{equation}\n    \\haus{P \\cap B_r(x)}{C \\cap B_r(x)} \\leq 2\\left(\n      \\varepsilon+\\delta\\left(r+\\norm{x-v}\\right)\\right)\n  \\end{equation}\n  for some~$v \\in \\conv\\vertices P$. In particular, if~$\\haus{P \\cap\n    B_r(x)}{C \\cap B_r(x)}$ is attained as~$\\norm{p-c}$ with~$p \\in P$,\n  then~$p = v + td$ for some~$d \\in \\recc{P}$ and~$t \\geq 0$.\n\\end{proposition}","start":27980,"end":28579}]},{"id":"S026","semantic_type":"assumption","kind":"assumption","labels":["PHDA"],"declared_external":false,"source_only":false,"title":"截断估计的局部条件","source_units":[{"raw_tex":"  Let $C \\subseteq \\R^n$ be nonempty closed convex and line-free\n  and let~$P$ be an~$\\ed$-approximation of~$C$. Then for every~$x \\in\n  \\conv \\vertices P$ and~$r \\geq \\varepsilon$ it holds true, that\n","start":28012,"end":28213}]},{"id":"S027","semantic_type":"proof","kind":"proof","labels":["proof_PHDA_1","proof_PHDA_2"],"declared_external":false,"source_only":false,"title":"截断估计的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n  Denote by $\\bar{P}$, $\\bar{C}$, and $V$, $P \\cap B_r(x)$, $C \\cap\n  B_r(x)$, and $\\conv \\vertices P$, respectively. Since~$\\bar{P},\n  \\bar{C}$ are nonempty convex and compact,~$\\haus{\\bar{P}}{\\bar{C}} =\n  \\norm{p^*-c^*}$ for some~$p^* \\in \\bar{P}$ and~$c^* \\in\n  \\bar{C}$. For~$\\lambda \\in [0,1]$ let~$z(\\lambda) = \\lambda p^* +\n  (1-\\lambda) x$. We distinguish two cases. First, assume~$p^* \\in\n  V$. Then~$z(\\lambda) \\in V$ for every~${\\lambda \\in [0,1]}$ and, due\n  to (i) in Definition \\ref{DEDA}, there is~${c_{\\lambda} \\in \\bar{C}}$\n  with~${\\norm{z(\\lambda)-c_{\\lambda}} \\leq\n    \\varepsilon}$. If~$\\norm{p^*-x} \\leq \\varepsilon$ then\n  \\[ \\norm{p^*-c^*} \\leq \\norm{p^*-c_0} \\leq \\norm{p^*-x} +\n    \\norm{x-c_0} \\leq 2\\varepsilon. \\] If~$\\norm{p^*-x} > \\varepsilon$,\n  set ${\\lambda}^* = \\frac{\\varepsilon}{\\norm{p^*-x}}$. Similarly, we\n  have\n  \\[ \\norm{p^*-c^*} \\leq \\norm{p^*-c_{{\\lambda}^*}} \\leq\n    \\norm{p^*-z({\\lambda}^*)} + \\norm{z({\\lambda}^*)-c_{{\\lambda}^*}} \\leq\n    2\\varepsilon. \\] Now, assume~${p^* \\notin V}$. Then there\n  exists~$\\bar{\\lambda} \\in (0,1)$, such that~$z(\\bar{\\lambda}) \\in V$\n  and~$z(\\lambda) \\notin V$ for all $\\lambda \\in\n  (\\bar{\\lambda},1]$. For~${\\lambda \\in (\\bar{\\lambda},1]}$,\n  $z(\\lambda)$ can be written as ${z(\\lambda) = v_{\\lambda} +\n    t_{\\lambda}d_{\\lambda}}$ with $v_{\\lambda} = \\argmin \\{\n      \\norm{z(\\bar{\\lambda}) - v} \\;\\mid\\;$ $v \\in \\left(\\{z(\\lambda)\\} -\n        \\recc{P}\\right) \\cap V \\}$, $t_{\\lambda} \\geq 0$ and\n  $d_{\\lambda} \\in \\recc{P}$, $\\norm{d_{\\lambda}} = 1$. By the\n  definition of $\\ed$-approximation there exist~$c_{\\lambda} \\in C$\n  and~$\\bar{d_{\\lambda}} \\in \\recc{C}$ such\n  that~$\\norm{v_{\\lambda}-c_{\\lambda}} \\leq \\varepsilon$\n  and~$\\norm{d_{\\lambda}-\\bar{d_{\\lambda}}} \\leq \\delta$. Now we have\n  \\begin{equation}\\label{proof_PHDA_1}\n    \\norm{z(\\lambda) - (c_{\\lambda} + t_{\\lambda}\\bar{d_{\\lambda}})} \\leq\n    \\norm{v_{\\lambda}-c_{\\lambda}} + t_{\\lambda}\n    \\norm{d_{\\lambda}-\\bar{d_{\\lambda}}} \\leq \\varepsilon +\n    t_{\\lambda}\\delta.\n  \\end{equation}\n  Furthermore, $t_{\\lambda} =\n  \\norm{z(\\lambda)-v_{\\lambda}}$, because $\\norm{d_{\\lambda}} =\n  1$. Hence,\n  \\begin{equation}\\label{proof_PHDA_2}\n    \\begin{aligned} t_{\\lambda} &\\leq\n      \\norm{z(\\lambda)-z(\\bar{\\lambda})} + \\norm{z(\\bar{\\lambda}) -\n        v_{\\lambda}} \\leq \\norm{z(\\lambda)-z(\\bar{\\lambda})} +\n      \\norm{z(\\bar{\\lambda})-v_1} \\\\ &\\leq\n      \\norm{z(\\lambda)-z(\\bar{\\lambda})} + \\norm{z(\\bar{\\lambda})-x} +\n      \\norm{x -v_1} \\\\ &= \\norm{z(\\lambda)-x} + \\norm{x-v_1}\\\\ &\\leq r\n      + \\norm{x-v_1}.\n    \\end{aligned}\n  \\end{equation}\nIf $\\frac{r-\\varepsilon}{r+\\norm{x-v_1}} < \\delta \\leq 1$ then \\[\n    \\norm{p^*-c^*} \\leq \\norm{p^*-c_0} \\leq \\norm{p^*-x} +\n    \\norm{x-c_0} \\leq r + \\varepsilon \\leq 2(\\varepsilon + \\delta\n    (r+\\norm{x-v_1})). \\] Otherwise, the last inequality is\n  violated. In this case, let~${\\lambda}^* = 1-\\frac{\\varepsilon +\n    \\delta(r\\norm{x-v_1})}{\\norm{p^*-x}}$. If~${\\lambda}^* \\leq\n  \\bar{\\lambda}$ then $z({\\lambda}^*) \\in V$ and there\n  exists~$c_{{\\lambda}^*} \\in C$ with~$\\norm{z({\\lambda}^*) -\n    c_{{\\lambda}^*}} \\leq \\varepsilon$. Therefore,\n  \\[ \\norm{p^*-c^*} \\leq \\norm{p^*-z({\\lambda}^*)} +\n    \\norm{z({\\lambda}^*)-c_{{\\lambda}^*}} \\leq 2 \\varepsilon + \\delta\n    (r+\\norm{x-v_1}). \\] If~${\\lambda}^* > \\bar{\\lambda}$ then,\n  according to \\eqref{proof_PHDA_1} and \\eqref{proof_PHDA_2}, there\n  exists~$c \\in C$ such that $\\norm{z({\\lambda}^*)-c} \\leq \\varepsilon\n  + \\delta(r+\\norm{x-v_1})$. Altogether this yields \\[ \\norm{p^*-c^*}\n    \\leq \\norm{p^*-c} \\leq \\norm{p^*-z({\\lambda}^*)} +\n    \\norm{z({\\lambda}^*)-c} \\leq 2 \\left( \\varepsilon + \\delta\n      (r+\\norm{x-v_1})\\right), \\] which completes the proof.\n\\end{proof}","start":28581,"end":32337}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"截断集合与距离取得点","source_units":[{"raw_tex":"  Denote by $\\bar{P}$, $\\bar{C}$, and $V$, $P \\cap B_r(x)$, $C \\cap\n  B_r(x)$, and $\\conv \\vertices P$, respectively. Since~$\\bar{P},\n  \\bar{C}$ are nonempty convex and compact,~$\\haus{\\bar{P}}{\\bar{C}} =\n  \\norm{p^*-c^*}$ for some~$p^* \\in \\bar{P}$ and~$c^* \\in\n  \\bar{C}$. For~$\\lambda \\in [0,1]$ let~$z(\\lambda) = \\lambda p^* +\n  (1-\\lambda) x$. We distinguish two cases. First, assume~$p^* \\in\n","start":28595,"end":28993}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"取得点位于顶点凸包的情形","source_units":[{"raw_tex":"  (1-\\lambda) x$. We distinguish two cases. First, assume~$p^* \\in\n  V$. Then~$z(\\lambda) \\in V$ for every~${\\lambda \\in [0,1]}$ and, due\n  to (i) in Definition \\ref{DEDA}, there is~${c_{\\lambda} \\in \\bar{C}}$\n  with~${\\norm{z(\\lambda)-c_{\\lambda}} \\leq\n    \\varepsilon}$. If~$\\norm{p^*-x} \\leq \\varepsilon$ then\n  \\[ \\norm{p^*-c^*} \\leq \\norm{p^*-c_0} \\leq \\norm{p^*-x} +\n    \\norm{x-c_0} \\leq 2\\varepsilon. \\] If~$\\norm{p^*-x} > \\varepsilon$,\n  set ${\\lambda}^* = \\frac{\\varepsilon}{\\norm{p^*-x}}$. Similarly, we\n  have\n  \\[ \\norm{p^*-c^*} \\leq \\norm{p^*-c_{{\\lambda}^*}} \\leq\n    \\norm{p^*-z({\\lambda}^*)} + \\norm{z({\\lambda}^*)-c_{{\\lambda}^*}} \\leq\n    2\\varepsilon. \\] Now, assume~${p^* \\notin V}$. Then there\n","start":28926,"end":29642}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"离开顶点凸包后的分解","source_units":[{"raw_tex":"    2\\varepsilon. \\] Now, assume~${p^* \\notin V}$. Then there\n  exists~$\\bar{\\lambda} \\in (0,1)$, such that~$z(\\bar{\\lambda}) \\in V$\n  and~$z(\\lambda) \\notin V$ for all $\\lambda \\in\n  (\\bar{\\lambda},1]$. For~${\\lambda \\in (\\bar{\\lambda},1]}$,\n  $z(\\lambda)$ can be written as ${z(\\lambda) = v_{\\lambda} +\n    t_{\\lambda}d_{\\lambda}}$ with $v_{\\lambda} = \\argmin \\{\n      \\norm{z(\\bar{\\lambda}) - v} \\;\\mid\\;$ $v \\in \\left(\\{z(\\lambda)\\} -\n        \\recc{P}\\right) \\cap V \\}$, $t_{\\lambda} \\geq 0$ and\n  $d_{\\lambda} \\in \\recc{P}$, $\\norm{d_{\\lambda}} = 1$. By the\n","start":29580,"end":30143}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":["proof_PHDA_1"],"declared_external":false,"source_only":false,"title":"点误差与方向误差合成","source_units":[{"raw_tex":"  definition of $\\ed$-approximation there exist~$c_{\\lambda} \\in C$\n  and~$\\bar{d_{\\lambda}} \\in \\recc{C}$ such\n  that~$\\norm{v_{\\lambda}-c_{\\lambda}} \\leq \\varepsilon$\n  and~$\\norm{d_{\\lambda}-\\bar{d_{\\lambda}}} \\leq \\delta$. Now we have\n  \\begin{equation}\\label{proof_PHDA_1}\n    \\norm{z(\\lambda) - (c_{\\lambda} + t_{\\lambda}\\bar{d_{\\lambda}})} \\leq\n    \\norm{v_{\\lambda}-c_{\\lambda}} + t_{\\lambda}\n    \\norm{d_{\\lambda}-\\bar{d_{\\lambda}}} \\leq \\varepsilon +\n    t_{\\lambda}\\delta.\n  \\end{equation}\n","start":30143,"end":30644}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":["proof_PHDA_2"],"declared_external":false,"source_only":false,"title":"衰退系数的半径上界","source_units":[{"raw_tex":"  Furthermore, $t_{\\lambda} =\n  \\norm{z(\\lambda)-v_{\\lambda}}$, because $\\norm{d_{\\lambda}} =\n  1$. Hence,\n  \\begin{equation}\\label{proof_PHDA_2}\n    \\begin{aligned} t_{\\lambda} &\\leq\n      \\norm{z(\\lambda)-z(\\bar{\\lambda})} + \\norm{z(\\bar{\\lambda}) -\n        v_{\\lambda}} \\leq \\norm{z(\\lambda)-z(\\bar{\\lambda})} +\n      \\norm{z(\\bar{\\lambda})-v_1} \\\\ &\\leq\n      \\norm{z(\\lambda)-z(\\bar{\\lambda})} + \\norm{z(\\bar{\\lambda})-x} +\n      \\norm{x -v_1} \\\\ &= \\norm{z(\\lambda)-x} + \\norm{x-v_1}\\\\ &\\leq r\n      + \\norm{x-v_1}.\n    \\end{aligned}\n  \\end{equation}\n","start":30644,"end":31201}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"较大方向容差的分支","source_units":[{"raw_tex":"If $\\frac{r-\\varepsilon}{r+\\norm{x-v_1}} < \\delta \\leq 1$ then \\[\n    \\norm{p^*-c^*} \\leq \\norm{p^*-c_0} \\leq \\norm{p^*-x} +\n    \\norm{x-c_0} \\leq r + \\varepsilon \\leq 2(\\varepsilon + \\delta\n    (r+\\norm{x-v_1})). \\] Otherwise, the last inequality is\n","start":31201,"end":31452}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"再次选取连线参数","source_units":[{"raw_tex":"  violated. In this case, let~${\\lambda}^* = 1-\\frac{\\varepsilon +\n    \\delta(r\\norm{x-v_1})}{\\norm{p^*-x}}$. If~${\\lambda}^* \\leq\n  \\bar{\\lambda}$ then $z({\\lambda}^*) \\in V$ and there\n  exists~$c_{{\\lambda}^*} \\in C$ with~$\\norm{z({\\lambda}^*) -\n    c_{{\\lambda}^*}} \\leq \\varepsilon$. Therefore,\n  \\[ \\norm{p^*-c^*} \\leq \\norm{p^*-z({\\lambda}^*)} +\n    \\norm{z({\\lambda}^*)-c_{{\\lambda}^*}} \\leq 2 \\varepsilon + \\delta\n    (r+\\norm{x-v_1}). \\] If~${\\lambda}^* > \\bar{\\lambda}$ then,\n","start":31452,"end":31938}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"合并两个内部估计","source_units":[{"raw_tex":"    (r+\\norm{x-v_1}). \\] If~${\\lambda}^* > \\bar{\\lambda}$ then,\n  according to \\eqref{proof_PHDA_1} and \\eqref{proof_PHDA_2}, there\n  exists~$c \\in C$ such that $\\norm{z({\\lambda}^*)-c} \\leq \\varepsilon\n  + \\delta(r+\\norm{x-v_1})$. Altogether this yields \\[ \\norm{p^*-c^*}\n    \\leq \\norm{p^*-c} \\leq \\norm{p^*-z({\\lambda}^*)} +\n    \\norm{z({\\lambda}^*)-c} \\leq 2 \\left( \\varepsilon + \\delta\n      (r+\\norm{x-v_1})\\right), \\] which completes the proof.\n","start":31874,"end":32326}]},{"id":"S036","semantic_type":"lemma","kind":"lemma","labels":["LBS"],"declared_external":false,"source_only":false,"title":"无直线性保证分解有界","source_units":[{"raw_tex":"\\begin{lemma}\\label{LBS}\n  Let $C \\subseteq \\R^n$ be nonempty closed and convex and let there\n  be sequences~$v^{\\nu}$,~$r^{\\nu}$ such that~$\\inf_{c \\in C}\n  \\norm{v^{\\nu}-c} \\rightarrow 0$, $\\inf_{r \\in \\recc{C}}\n  \\norm{r^{\\nu}-r} \\rightarrow 0$, and $v^{\\nu}+r^{\\nu} \\in\n    B_M(x)$ for some $M \\geq 0$ and $x \\in \\R^n$. If $C$ is\n  line-free, then $\\norm{v^{\\nu}}$ is bounded.\n\\end{lemma}","start":32411,"end":32803}]},{"id":"S037","semantic_type":"assumption","kind":"assumption","labels":["LBS"],"declared_external":false,"source_only":false,"title":"有界性引理的精确前提","source_units":[{"raw_tex":"  Let $C \\subseteq \\R^n$ be nonempty closed and convex and let there\n  be sequences~$v^{\\nu}$,~$r^{\\nu}$ such that~$\\inf_{c \\in C}\n  \\norm{v^{\\nu}-c} \\rightarrow 0$, $\\inf_{r \\in \\recc{C}}\n  \\norm{r^{\\nu}-r} \\rightarrow 0$, and $v^{\\nu}+r^{\\nu} \\in\n    B_M(x)$ for some $M \\geq 0$ and $x \\in \\R^n$. If $C$ is\n  line-free, then $\\norm{v^{\\nu}}$ is bounded.\n","start":32436,"end":32792}]},{"id":"S038","semantic_type":"proof","kind":"proof","labels":["proof_LBS"],"declared_external":false,"source_only":false,"title":"有界性引理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n  Assume that $\\norm{v^{\\nu}}$ is unbounded. This implies\n  that~$\\norm{r^{\\nu}}$ is also unbounded and $\\recc{C} \\neq\n  \\{0\\}$. Without loss of generality let $r^{\\nu} \\neq 0$ for all\n  $\\nu$. Then $d^{\\nu} := r^{\\nu}/\\norm{r^{\\nu}}$ is bounded and has a\n  convergent subsequence. Without loss of generality we can assume\n  that $d^{\\nu} \\rightarrow d \\in \\recc{C}$. We will show, that $-d\n  \\in \\recc{C}$. Therefore let $c \\in C$, $t \\geq 0$ and define $y^{\\nu} =\n  \\argmin_{y \\in C} \\norm{v^{\\nu}-y}$. By the triangle inequality it\n  holds true, that\n  \\begin{equation}\\label{proof_LBS}\n    \\norm{c-r^{\\nu}-y^{\\nu}} \\leq M + \\norm{c-x} +\n    \\norm{v^{\\nu}-y^{\\nu}} =: \\ols{M}^{\\nu}.\n  \\end{equation}\n  Note, that $\\ols{M}^{\\nu}$ is bounded from above by some $\\ols{M}$, because\n  $\\norm{v^{\\nu}-y^{\\nu}} \\rightarrow 0$. For every $T \\geq 0$ there\n  exists some $\\nu_T$, such that $\\norm{r^{\\nu_T}} \\geq T$. Let $T\n  \\geq t$ and define\n  \\[ \\bar{y} := \\frac{t}{\\norm{r^{\\nu_T}}} y^{\\nu_T} + \\left(\n      1-\\frac{t}{\\norm{r^{\\nu_T}}} \\right) c \\in C. \\]\n  Putting it all together one gets\n  \\begin{equation*}\n    \\begin{aligned}\n      \\inf_{y \\in C} \\norm{(c-td^{\\nu_T})-y} &\\leq\n      \\norm{(c-td^{\\nu_T})-\\bar{y}} \\\\\n      &= \\norm{\\frac{t}{\\norm{r^{\\nu_T}}}c\n        -td^{\\nu_T}-\\frac{t}{\\norm{r^{\\nu_T}}}y^{\\nu_T}} \\\\\n      &= \\frac{t}{\\norm{r^{\\nu_T}}} \\norm{c-r^{\\nu_T}-y^{\\nu_T}} \\\\\n      &\\leq \\frac{t}{T} \\ols{M},\n    \\end{aligned}\n  \\end{equation*}\n  where the last inequality holds due to \\eqref{proof_LBS} and\n  boundedness of $\\ols{M}^{\\nu}$. Since $C$ is closed and $d^{\\nu}\n  \\rightarrow d \\in \\recc{C}$, taking the limit $T \\rightarrow\n  +\\infty$ yields that $c-td \\in C$. This is a contradiction to the\n  pointedness of $\\recc{C}$.\n\\end{proof}","start":32805,"end":34580}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"抽取非零极限方向","source_units":[{"raw_tex":"  Assume that $\\norm{v^{\\nu}}$ is unbounded. This implies\n  that~$\\norm{r^{\\nu}}$ is also unbounded and $\\recc{C} \\neq\n  \\{0\\}$. Without loss of generality let $r^{\\nu} \\neq 0$ for all\n  $\\nu$. Then $d^{\\nu} := r^{\\nu}/\\norm{r^{\\nu}}$ is bounded and has a\n  convergent subsequence. Without loss of generality we can assume\n  that $d^{\\nu} \\rightarrow d \\in \\recc{C}$. We will show, that $-d\n  \\in \\recc{C}$. Therefore let $c \\in C$, $t \\geq 0$ and define $y^{\\nu} =\n","start":32819,"end":33285}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":["proof_LBS"],"declared_external":false,"source_only":false,"title":"最近点的统一误差界","source_units":[{"raw_tex":"  \\in \\recc{C}$. Therefore let $c \\in C$, $t \\geq 0$ and define $y^{\\nu} =\n  \\argmin_{y \\in C} \\norm{v^{\\nu}-y}$. By the triangle inequality it\n  holds true, that\n  \\begin{equation}\\label{proof_LBS}\n    \\norm{c-r^{\\nu}-y^{\\nu}} \\leq M + \\norm{c-x} +\n    \\norm{v^{\\nu}-y^{\\nu}} =: \\ols{M}^{\\nu}.\n  \\end{equation}\n  Note, that $\\ols{M}^{\\nu}$ is bounded from above by some $\\ols{M}$, because\n  $\\norm{v^{\\nu}-y^{\\nu}} \\rightarrow 0$. For every $T \\geq 0$ there\n","start":33210,"end":33669}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"凸组合逼近反向射线","source_units":[{"raw_tex":"  $\\norm{v^{\\nu}-y^{\\nu}} \\rightarrow 0$. For every $T \\geq 0$ there\n  exists some $\\nu_T$, such that $\\norm{r^{\\nu_T}} \\geq T$. Let $T\n  \\geq t$ and define\n  \\[ \\bar{y} := \\frac{t}{\\norm{r^{\\nu_T}}} y^{\\nu_T} + \\left(\n      1-\\frac{t}{\\norm{r^{\\nu_T}}} \\right) c \\in C. \\]\n  Putting it all together one gets\n  \\begin{equation*}\n    \\begin{aligned}\n      \\inf_{y \\in C} \\norm{(c-td^{\\nu_T})-y} &\\leq\n      \\norm{(c-td^{\\nu_T})-\\bar{y}} \\\\\n      &= \\norm{\\frac{t}{\\norm{r^{\\nu_T}}}c\n        -td^{\\nu_T}-\\frac{t}{\\norm{r^{\\nu_T}}}y^{\\nu_T}} \\\\\n      &= \\frac{t}{\\norm{r^{\\nu_T}}} \\norm{c-r^{\\nu_T}-y^{\\nu_T}} \\\\\n      &\\leq \\frac{t}{T} \\ols{M},\n    \\end{aligned}\n  \\end{equation*}\n","start":33600,"end":34279}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"反向衰退方向违反尖性","source_units":[{"raw_tex":"  where the last inequality holds due to \\eqref{proof_LBS} and\n  boundedness of $\\ols{M}^{\\nu}$. Since $C$ is closed and $d^{\\nu}\n  \\rightarrow d \\in \\recc{C}$, taking the limit $T \\rightarrow\n  +\\infty$ yields that $c-td \\in C$. This is a contradiction to the\n  pointedness of $\\recc{C}$.\n","start":34279,"end":34569}]},{"id":"S043","semantic_type":"proposition","kind":"proposition","labels":["PEP"],"declared_external":false,"source_only":false,"title":"极点的顶点凸包逼近","source_units":[{"raw_tex":"\\begin{proposition}\\label{PEP}\n  Let $C \\subseteq \\R^n$ be nonempty closed convex and\n  line-free. For $\\nu \\in \\mathbb{N}$ let~$P^{\\nu}$ be an\n  $(\\varepsilon^{\\nu},\\delta^{\\nu})$-approximation of\n  $C$. If~${(\\varepsilon^{\\nu},\\delta^{\\nu}) \\rightarrow (0,0)}$, then\n  for every extreme point~$c$ of $C$ there exists a sequence~${x^{\\nu}\n    \\rightarrow c}$ such that $x^{\\nu} \\in \\conv \\vertices P^{\\nu}$.\n\\end{proposition}","start":35010,"end":35436}]},{"id":"S044","semantic_type":"assumption","kind":"assumption","labels":["PEP"],"declared_external":false,"source_only":false,"title":"极点逼近命题的条件","source_units":[{"raw_tex":"  Let $C \\subseteq \\R^n$ be nonempty closed convex and\n  line-free. For $\\nu \\in \\mathbb{N}$ let~$P^{\\nu}$ be an\n  $(\\varepsilon^{\\nu},\\delta^{\\nu})$-approximation of\n  $C$. If~${(\\varepsilon^{\\nu},\\delta^{\\nu}) \\rightarrow (0,0)}$, then\n  for every extreme point~$c$ of $C$ there exists a sequence~${x^{\\nu}\n","start":35041,"end":35350}]},{"id":"S045","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"极点逼近的完整作者证明","source_units":[{"raw_tex":"\\begin{proof} Since $C$ is line-free, it has at least one extreme\n  point. Let $c$ be one such extreme point. Assume that for every\n  sequence $x^{\\nu}$ with~${x^{\\nu} \\in \\conv \\vertices P^{\\nu}}$ there\n  exists a $\\gamma > 0$, such that~${\\norm{x^{\\nu}-c} > \\gamma}$ for\n  infinitely many $\\nu$. Then, without loss of generality, there exists\n  one such sequence such that~$\\norm{x^{\\nu}-c} > \\gamma$ for every\n  $\\nu$ and, since $C \\subseteq P^{\\nu}$, $c=x^{\\nu}+r^{\\nu}$ for some\n  $r^{\\nu} \\in \\recc{P^{\\nu}}$. By Lemma \\ref{LBS} it holds that\n  $\\norm{x^{\\nu}}$ and $\\norm{r^{\\nu}}$ are bounded. Then there exist\n  subsequences $x^{\\nu_k}$, $r^{\\nu_k}$ such that\n  \\[ x^{\\nu_k} \\rightarrow x \\in C, \\quad r^{\\nu_k} \\rightarrow r \\in\n    \\recc{C}. \\] Note that $r \\neq 0$, because $\\norm{r^{\\nu}} > \\gamma$\n  for all $\\nu$. Finally,\n  \\[ c=x+r=\\frac{1}{2}\\left( x+2r \\right) + \\frac{1}{2} x. \\] This is\n  a contradiction to $c$ being an extreme point of $C$.\n\\end{proof}","start":35438,"end":36413}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"固定间隔与极点分解","source_units":[{"raw_tex":"\\begin{proof} Since $C$ is line-free, it has at least one extreme\n  point. Let $c$ be one such extreme point. Assume that for every\n  sequence $x^{\\nu}$ with~${x^{\\nu} \\in \\conv \\vertices P^{\\nu}}$ there\n  exists a $\\gamma > 0$, such that~${\\norm{x^{\\nu}-c} > \\gamma}$ for\n  infinitely many $\\nu$. Then, without loss of generality, there exists\n  one such sequence such that~$\\norm{x^{\\nu}-c} > \\gamma$ for every\n  $\\nu$ and, since $C \\subseteq P^{\\nu}$, $c=x^{\\nu}+r^{\\nu}$ for some\n  $r^{\\nu} \\in \\recc{P^{\\nu}}$. By Lemma \\ref{LBS} it holds that\n","start":35438,"end":35987}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"引用引理断言分解有界","source_units":[{"raw_tex":"  $r^{\\nu} \\in \\recc{P^{\\nu}}$. By Lemma \\ref{LBS} it holds that\n  $\\norm{x^{\\nu}}$ and $\\norm{r^{\\nu}}$ are bounded. Then there exist\n","start":35922,"end":36057}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"抽取固定集合中的极限","source_units":[{"raw_tex":"  $\\norm{x^{\\nu}}$ and $\\norm{r^{\\nu}}$ are bounded. Then there exist\n  subsequences $x^{\\nu_k}$, $r^{\\nu_k}$ such that\n  \\[ x^{\\nu_k} \\rightarrow x \\in C, \\quad r^{\\nu_k} \\rightarrow r \\in\n    \\recc{C}. \\] Note that $r \\neq 0$, because $\\norm{r^{\\nu}} > \\gamma$\n","start":35987,"end":36250}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"非零衰退部分破坏极点性","source_units":[{"raw_tex":"    \\recc{C}. \\] Note that $r \\neq 0$, because $\\norm{r^{\\nu}} > \\gamma$\n  for all $\\nu$. Finally,\n  \\[ c=x+r=\\frac{1}{2}\\left( x+2r \\right) + \\frac{1}{2} x. \\] This is\n  a contradiction to $c$ being an extreme point of $C$.\n","start":36177,"end":36402}]},{"id":"S050","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"主收敛定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n  By Theorem \\ref{TPKCHD} we must show that there exist $c \\in \\R^n$\n  and $r_0 \\geq 0$ such that for all $r \\geq r_0$ it holds $\\haus{P^{\\nu} \\cap B_r(c)}{C \\cap B_r(c)}\n  \\rightarrow 0$. Let~${r \\geq \\max_{\\nu \\in\n      \\mathbb{N}} \\varepsilon^{\\nu}}$ and let $c$ be an extreme point of $C$,\n  which exists, because $C$ contains no lines. By Proposition \\ref{PEP}\n  there exists a sequence~${x^{\\nu} \\rightarrow c}$ with~${x^{\\nu} \\in\n    \\conv \\vertices P^{\\nu}}$. Applying the triangle inequality and\n  Proposition \\ref{PHDA} yields\n  \\begin{equation*}\n    \\begin{aligned}\n      & & &  \\haus{P^{\\nu} \\cap B_r(c)}{C \\cap B_r(c)} \\\\\n      & \\leq & & \\haus{P^{\\nu} \\cap B_r(c)}{P^{\\nu} \\cap B_r(x^{\\nu})}\n      \\\\ & & + & \\haus{P^{\\nu} \\cap B_r(x^{\\nu})}{C \\cap B_r(x^{\\nu})}\n      \\\\ & & + & \\haus{C \\cap B_r(x^{\\nu})}{C \\cap B_r(c)} \\\\\n      & \\leq & & \\haus{P^{\\nu} \\cap B_r(c)}{P^{\\nu} \\cap B_r(x^{\\nu})}\n      \\\\ & & + & 2(\\varepsilon^{\\nu}+\\delta^{\\nu}(r+\\norm{x^{\\nu}-v^{\\nu}}))\n      \\\\ & & + & \\haus{C \\cap B_r(x^{\\nu})}{C \\cap B_r(c)},\n   \\end{aligned}\n \\end{equation*}\n for some $v^{\\nu} \\in \\conv \\vertices P^{\\nu}$. The first and third\n term in this sum converge to zero as~${x^{\\nu} \\rightarrow c}$. It\n remains to show that~$\\norm{x^{\\nu}-v^{\\nu}}$ is\n bounded. Since $C \\subseteq P^{\\nu}$, the distance ${\\haus{P^{\\nu}\n   \\cap B_r(x^{\\nu})}{C \\cap B_r(x^{\\nu})}}$ is attained as\n $\\exc{P^{\\nu} \\cap B_r(x^{\\nu})}{C \\cap B_r(x^{\\nu})}$. Let the\n supremum be attained by~$p^{\\nu} \\in P^{\\nu}$. Then $p^{\\nu} =\n   v^{\\nu} + d^{\\nu}$ for some~$d^{\\nu} \\in \\recc{P^{\\nu}}$. It holds\n \\[ \\norm{p^{\\nu}-c} \\leq \\norm{p^{\\nu}-x^{\\nu}} + \\norm{x^{\\nu}-c}\n   \\leq r + \\max_{\\nu \\in \\mathbb{N}} \\norm{x^{\\nu}-c} < +\\infty, \\]\n i.e. $v^{\\nu}+d^{\\nu} \\in B_M(c)$ for some $M \\geq 0$. Therefore, the\n sequence $\\norm{v^{\\nu}}$ is bounded according to Lemma\n \\ref{LBS}. Hence, $\\norm{x^{\\nu}-v^{\\nu}}$ is also bounded\n and $\\haus{P^{\\nu} \\cap B_r(c)}{C \\cap B_r(c)} \\rightarrow 0$,\n which was to be proved.\n\\end{proof}","start":36805,"end":38839}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"约化到固定中心的球截断","source_units":[{"raw_tex":"  By Theorem \\ref{TPKCHD} we must show that there exist $c \\in \\R^n$\n  and $r_0 \\geq 0$ such that for all $r \\geq r_0$ it holds $\\haus{P^{\\nu} \\cap B_r(c)}{C \\cap B_r(c)}\n  \\rightarrow 0$. Let~${r \\geq \\max_{\\nu \\in\n","start":36819,"end":37035}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"极点中心与逼近中心","source_units":[{"raw_tex":"  \\rightarrow 0$. Let~${r \\geq \\max_{\\nu \\in\n      \\mathbb{N}} \\varepsilon^{\\nu}}$ and let $c$ be an extreme point of $C$,\n  which exists, because $C$ contains no lines. By Proposition \\ref{PEP}\n  there exists a sequence~${x^{\\nu} \\rightarrow c}$ with~${x^{\\nu} \\in\n    \\conv \\vertices P^{\\nu}}$. Applying the triangle inequality and\n","start":36990,"end":37324}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"移动中心的三项估计","source_units":[{"raw_tex":"    \\conv \\vertices P^{\\nu}}$. Applying the triangle inequality and\n  Proposition \\ref{PHDA} yields\n  \\begin{equation*}\n    \\begin{aligned}\n      & & &  \\haus{P^{\\nu} \\cap B_r(c)}{C \\cap B_r(c)} \\\\\n      & \\leq & & \\haus{P^{\\nu} \\cap B_r(c)}{P^{\\nu} \\cap B_r(x^{\\nu})}\n      \\\\ & & + & \\haus{P^{\\nu} \\cap B_r(x^{\\nu})}{C \\cap B_r(x^{\\nu})}\n      \\\\ & & + & \\haus{C \\cap B_r(x^{\\nu})}{C \\cap B_r(c)} \\\\\n      & \\leq & & \\haus{P^{\\nu} \\cap B_r(c)}{P^{\\nu} \\cap B_r(x^{\\nu})}\n      \\\\ & & + & 2(\\varepsilon^{\\nu}+\\delta^{\\nu}(r+\\norm{x^{\\nu}-v^{\\nu}}))\n      \\\\ & & + & \\haus{C \\cap B_r(x^{\\nu})}{C \\cap B_r(c)},\n   \\end{aligned}\n \\end{equation*}\n for some $v^{\\nu} \\in \\conv \\vertices P^{\\nu}$. The first and third\n","start":37256,"end":37969}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"作者断言换中心项趋零","source_units":[{"raw_tex":" for some $v^{\\nu} \\in \\conv \\vertices P^{\\nu}$. The first and third\n term in this sum converge to zero as~${x^{\\nu} \\rightarrow c}$. It\n","start":37900,"end":38037}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"最坏截断点与相应分解","source_units":[{"raw_tex":" remains to show that~$\\norm{x^{\\nu}-v^{\\nu}}$ is\n bounded. Since $C \\subseteq P^{\\nu}$, the distance ${\\haus{P^{\\nu}\n   \\cap B_r(x^{\\nu})}{C \\cap B_r(x^{\\nu})}}$ is attained as\n $\\exc{P^{\\nu} \\cap B_r(x^{\\nu})}{C \\cap B_r(x^{\\nu})}$. Let the\n supremum be attained by~$p^{\\nu} \\in P^{\\nu}$. Then $p^{\\nu} =\n   v^{\\nu} + d^{\\nu}$ for some~$d^{\\nu} \\in \\recc{P^{\\nu}}$. It holds\n","start":38037,"end":38414}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"取得点落在统一固定球","source_units":[{"raw_tex":" \\[ \\norm{p^{\\nu}-c} \\leq \\norm{p^{\\nu}-x^{\\nu}} + \\norm{x^{\\nu}-c}\n   \\leq r + \\max_{\\nu \\in \\mathbb{N}} \\norm{x^{\\nu}-c} < +\\infty, \\]\n i.e. $v^{\\nu}+d^{\\nu} \\in B_M(c)$ for some $M \\geq 0$. Therefore, the\n","start":38414,"end":38622}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"再次调用分解有界性引理","source_units":[{"raw_tex":" i.e. $v^{\\nu}+d^{\\nu} \\in B_M(c)$ for some $M \\geq 0$. Therefore, the\n sequence $\\norm{v^{\\nu}}$ is bounded according to Lemma\n \\ref{LBS}. Hence, $\\norm{x^{\\nu}-v^{\\nu}}$ is also bounded\n","start":38551,"end":38739}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"误差趋零并结束约化","source_units":[{"raw_tex":" \\ref{LBS}. Hence, $\\norm{x^{\\nu}-v^{\\nu}}$ is also bounded\n and $\\haus{P^{\\nu} \\cap B_r(c)}{C \\cap B_r(c)} \\rightarrow 0$,\n which was to be proved.\n","start":38679,"end":38828}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S018","edges":[{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S010","kind":"supports"},{"source_id":"S001","target_id":"S010","kind":"supports"},{"source_id":"S003","target_id":"S012","kind":"supports"},{"source_id":"S007","target_id":"S012","kind":"supports"},{"source_id":"S010","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S011","kind":"supports"},{"source_id":"S013","target_id":"S011","kind":"supports"},{"source_id":"S014","target_id":"S011","kind":"supports"},{"source_id":"S005","target_id":"S011","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S008","target_id":"S016","kind":"supports"},{"source_id":"S001","target_id":"S016","kind":"supports"},{"source_id":"S011","target_id":"S017","kind":"supports"},{"source_id":"S011","target_id":"S018","kind":"supports"},{"source_id":"S015","target_id":"S018","kind":"supports"},{"source_id":"S011","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S028","kind":"supports"},{"source_id":"S012","target_id":"S028","kind":"supports"},{"source_id":"S014","target_id":"S028","kind":"supports"},{"source_id":"S019","target_id":"S028","kind":"supports"},{"source_id":"S020","target_id":"S028","kind":"supports"},{"source_id":"S021","target_id":"S028","kind":"supports"},{"source_id":"S008","target_id":"S028","kind":"supports"},{"source_id":"S012","target_id":"S029","kind":"supports"},{"source_id":"S019","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S026","target_id":"S030","kind":"supports"},{"source_id":"S028","target_id":"S030","kind":"supports"},{"source_id":"S023","target_id":"S030","kind":"supports"},{"source_id":"S020","target_id":"S030","kind":"supports"},{"source_id":"S021","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S012","target_id":"S031","kind":"supports"},{"source_id":"S019","target_id":"S031","kind":"supports"},{"source_id":"S013","target_id":"S031","kind":"supports"},{"source_id":"S010","target_id":"S031","kind":"supports"},{"source_id":"S022","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S028","target_id":"S032","kind":"supports"},{"source_id":"S026","target_id":"S033","kind":"supports"},{"source_id":"S012","target_id":"S033","kind":"supports"},{"source_id":"S019","target_id":"S033","kind":"supports"},{"source_id":"S028","target_id":"S033","kind":"supports"},{"source_id":"S030","target_id":"S034","kind":"supports"},{"source_id":"S012","target_id":"S034","kind":"supports"},{"source_id":"S019","target_id":"S034","kind":"supports"},{"source_id":"S028","target_id":"S034","kind":"supports"},{"source_id":"S031","target_id":"S035","kind":"supports"},{"source_id":"S032","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S027","kind":"supports"},{"source_id":"S031","target_id":"S027","kind":"supports"},{"source_id":"S032","target_id":"S027","kind":"supports"},{"source_id":"S033","target_id":"S027","kind":"supports"},{"source_id":"S034","target_id":"S027","kind":"supports"},{"source_id":"S035","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S025","kind":"supports"},{"source_id":"S005","target_id":"S037","kind":"supports"},{"source_id":"S001","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S021","target_id":"S039","kind":"supports"},{"source_id":"S022","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S020","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S037","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S039","target_id":"S042","kind":"supports"},{"source_id":"S021","target_id":"S042","kind":"supports"},{"source_id":"S022","target_id":"S042","kind":"supports"},{"source_id":"S005","target_id":"S042","kind":"supports"},{"source_id":"S037","target_id":"S042","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S041","target_id":"S038","kind":"supports"},{"source_id":"S042","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S011","target_id":"S044","kind":"supports"},{"source_id":"S002","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S046","kind":"supports"},{"source_id":"S006","target_id":"S046","kind":"supports"},{"source_id":"S014","target_id":"S046","kind":"supports"},{"source_id":"S023","target_id":"S046","kind":"supports"},{"source_id":"S036","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S044","target_id":"S047","kind":"supports"},{"source_id":"S012","target_id":"S047","kind":"supports"},{"source_id":"S019","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S021","target_id":"S048","kind":"supports"},{"source_id":"S012","target_id":"S048","kind":"supports"},{"source_id":"S019","target_id":"S048","kind":"supports"},{"source_id":"S013","target_id":"S048","kind":"supports"},{"source_id":"S010","target_id":"S048","kind":"supports"},{"source_id":"S022","target_id":"S048","kind":"supports"},{"source_id":"S044","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S002","target_id":"S049","kind":"supports"},{"source_id":"S022","target_id":"S049","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S048","target_id":"S045","kind":"supports"},{"source_id":"S049","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"},{"source_id":"S016","target_id":"S051","kind":"supports"},{"source_id":"S017","target_id":"S051","kind":"supports"},{"source_id":"S017","target_id":"S052","kind":"supports"},{"source_id":"S006","target_id":"S052","kind":"supports"},{"source_id":"S043","target_id":"S052","kind":"supports"},{"source_id":"S024","target_id":"S052","kind":"supports"},{"source_id":"S008","target_id":"S053","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S025","target_id":"S053","kind":"supports"},{"source_id":"S017","target_id":"S053","kind":"supports"},{"source_id":"S052","target_id":"S054","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S014","target_id":"S055","kind":"supports"},{"source_id":"S008","target_id":"S055","kind":"supports"},{"source_id":"S007","target_id":"S055","kind":"supports"},{"source_id":"S021","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S025","target_id":"S055","kind":"supports"},{"source_id":"S023","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S052","target_id":"S056","kind":"supports"},{"source_id":"S024","target_id":"S056","kind":"supports"},{"source_id":"S036","target_id":"S057","kind":"supports"},{"source_id":"S055","target_id":"S057","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S017","target_id":"S057","kind":"supports"},{"source_id":"S012","target_id":"S057","kind":"supports"},{"source_id":"S019","target_id":"S057","kind":"supports"},{"source_id":"S053","target_id":"S057","kind":"supports"},{"source_id":"S052","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S053","target_id":"S058","kind":"supports"},{"source_id":"S054","target_id":"S058","kind":"supports"},{"source_id":"S017","target_id":"S058","kind":"supports"},{"source_id":"S024","target_id":"S058","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S050","kind":"supports"},{"source_id":"S054","target_id":"S050","kind":"supports"},{"source_id":"S055","target_id":"S050","kind":"supports"},{"source_id":"S056","target_id":"S050","kind":"supports"},{"source_id":"S057","target_id":"S050","kind":"supports"},{"source_id":"S058","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S018","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S018","edges":[{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S010","kind":"supports"},{"source_id":"S001","target_id":"S010","kind":"supports"},{"source_id":"S003","target_id":"S012","kind":"supports"},{"source_id":"S007","target_id":"S012","kind":"supports"},{"source_id":"S010","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S011","kind":"supports"},{"source_id":"S013","target_id":"S011","kind":"supports"},{"source_id":"S014","target_id":"S011","kind":"supports"},{"source_id":"S005","target_id":"S011","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S008","target_id":"S016","kind":"supports"},{"source_id":"S001","target_id":"S016","kind":"supports"},{"source_id":"S011","target_id":"S017","kind":"supports"},{"source_id":"S011","target_id":"S018","kind":"supports"},{"source_id":"S015","target_id":"S018","kind":"supports"},{"source_id":"S011","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S012","target_id":"S025","kind":"supports"},{"source_id":"S014","target_id":"S025","kind":"supports"},{"source_id":"S019","target_id":"S025","kind":"supports"},{"source_id":"S020","target_id":"S025","kind":"supports"},{"source_id":"S021","target_id":"S025","kind":"supports"},{"source_id":"S008","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S013","target_id":"S025","kind":"supports"},{"source_id":"S010","target_id":"S025","kind":"supports"},{"source_id":"S022","target_id":"S025","kind":"supports"},{"source_id":"S005","target_id":"S037","kind":"supports"},{"source_id":"S001","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S021","target_id":"S036","kind":"supports"},{"source_id":"S022","target_id":"S036","kind":"supports"},{"source_id":"S020","target_id":"S036","kind":"supports"},{"source_id":"S005","target_id":"S036","kind":"supports"},{"source_id":"S011","target_id":"S044","kind":"supports"},{"source_id":"S002","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S006","target_id":"S043","kind":"supports"},{"source_id":"S014","target_id":"S043","kind":"supports"},{"source_id":"S023","target_id":"S043","kind":"supports"},{"source_id":"S036","target_id":"S043","kind":"supports"},{"source_id":"S012","target_id":"S043","kind":"supports"},{"source_id":"S019","target_id":"S043","kind":"supports"},{"source_id":"S021","target_id":"S043","kind":"supports"},{"source_id":"S013","target_id":"S043","kind":"supports"},{"source_id":"S010","target_id":"S043","kind":"supports"},{"source_id":"S022","target_id":"S043","kind":"supports"},{"source_id":"S002","target_id":"S043","kind":"supports"},{"source_id":"S016","target_id":"S051","kind":"supports"},{"source_id":"S017","target_id":"S051","kind":"supports"},{"source_id":"S017","target_id":"S052","kind":"supports"},{"source_id":"S006","target_id":"S052","kind":"supports"},{"source_id":"S043","target_id":"S052","kind":"supports"},{"source_id":"S024","target_id":"S052","kind":"supports"},{"source_id":"S008","target_id":"S053","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S025","target_id":"S053","kind":"supports"},{"source_id":"S017","target_id":"S053","kind":"supports"},{"source_id":"S052","target_id":"S054","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S014","target_id":"S055","kind":"supports"},{"source_id":"S008","target_id":"S055","kind":"supports"},{"source_id":"S007","target_id":"S055","kind":"supports"},{"source_id":"S021","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S025","target_id":"S055","kind":"supports"},{"source_id":"S023","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S052","target_id":"S056","kind":"supports"},{"source_id":"S024","target_id":"S056","kind":"supports"},{"source_id":"S036","target_id":"S057","kind":"supports"},{"source_id":"S055","target_id":"S057","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S017","target_id":"S057","kind":"supports"},{"source_id":"S012","target_id":"S057","kind":"supports"},{"source_id":"S019","target_id":"S057","kind":"supports"},{"source_id":"S053","target_id":"S057","kind":"supports"},{"source_id":"S052","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S053","target_id":"S058","kind":"supports"},{"source_id":"S054","target_id":"S058","kind":"supports"},{"source_id":"S017","target_id":"S058","kind":"supports"},{"source_id":"S024","target_id":"S058","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S050","kind":"supports"},{"source_id":"S054","target_id":"S050","kind":"supports"},{"source_id":"S055","target_id":"S050","kind":"supports"},{"source_id":"S056","target_id":"S050","kind":"supports"},{"source_id":"S057","target_id":"S050","kind":"supports"},{"source_id":"S058","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S018","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"集合运算与欧氏球","source_units":[{"raw_tex":"Throughout this article we denote by $\\cl C$, $\\interior C$, $\\recc\nC$, $\\conv C$, and $\\cone C$ the closure, interior, recession cone,\nconvex hull and conical hull of a set $C$, respectively. A compact\nconvex set with nonempty interior is called a \\textit{convex\nbody}. The Euclidean ball with radius $r$ centered at a point~${c \\in\n\\R^n}$ is denoted by~$B_r(c)$. A point $c$ of a convex set $C$ is\n","start":9679,"end":10079}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"极点及非平凡凸组合","source_units":[{"raw_tex":"\\R^n}$ is denoted by~$B_r(c)$. A point $c$ of a convex set $C$ is\ncalled an \\emph{extreme point} of $C$, if $C \\setminus\\{c\\}$ is\nconvex. The set of extreme points of $C$ is denoted by $\\ext\nC$. Extreme points are exactly the points of $C$ that cannot be\nwritten as a proper convex combination of elements of $C$\n\\cite[p.~162]{Roc70}. For finite sets $V, D \\subseteq \\R^n$, the set\n","start":10013,"end":10395}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":["hrep","vrep"],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"多面体表示与顶点","source_units":[{"raw_tex":"\\cite[p.~162]{Roc70}. For finite sets $V, D \\subseteq \\R^n$, the set\n\\begin{equation}\\label{vrep}\nP = \\conv V + \\cone D\n\\end{equation}\nis called a \\emph{polyhedron}. The plus sign denotes Minkowski\naddition. The sets $V,D$ in \\eqref{vrep} are called a\n\\emph{$V$-representation} of $P$ as it is expressed in terms of its\nvertices and directions. A polyhedron can equivalently be expressed as\na finite intersection of closed halfspaces \\cite[Theorem 19.1]{Roc70},\ni.e.\n\\begin{equation}\\label{hrep}\nP = \\{x \\in \\R^n \\mid Ax \\leq b\\}\n\\end{equation}\nfor a matrix $A \\in \\R^{m \\times n}$ and a vector $b \\in \\R^m$. The\ndata $(A,b)$ are called a \\emph{$H$-representation} of $P$.  The\nextreme points of a polyhedron $P$ are called \\emph{vertices} of $P$\nand are denoted by~$\\vertices P$. For symmetric matrices\n","start":10326,"end":11130}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["EPCS"],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"谱面体与极锥背景","source_units":[{"raw_tex":"and are denoted by~$\\vertices P$. For symmetric matrices\n$A_0,A_1,\\dots,A_n$ of arbitrary fixed size we define\n\\begin{equation}\n\\Ab(x) := \\sum_{i=1}^n x_iA_i \\quad \\text{and} \\quad \\A(x) := \\Ab(x) +\nA_0,\n\\end{equation}\ni.e. a linear combination of the $A_i$ and a translation of $\\Ab(x)$\nby $A_0$, respectively. We denote by $A \\mg 0$ ($A \\mgeq 0$) positive\n(semi-)definiteness of the symmetric matrix $A$. A set of the form\n$\\{x \\in \\R^n \\mid \\A(x) \\mgeq 0\\}$ is called a\n\\emph{spectrahedron}. Spectrahedra are a generalization of polyhedra\nfor which many geometric properties of polyhedra generalize nicely,\ne.g. the recession cone of a spectrahedron $C$ is obtained as $\\{x \\in\n\\R^n \\mid \\Ab(x) \\mgeq 0 \\}$, see \\cite{GR95}, whereas the recession\ncone of a polyhedron in $H$-representation is $\\{x \\in \\R^n \\mid Ax\n\\leq 0\\}$. Given a cone $K$ the set $\\polar{K} = \\{y \\in \\R^n \\mid\n\\forall x \\in K: x\\T y \\leq 0\\}$ is called the \\emph{polar cone} of\n$K$. The polar $\\polar{(\\recc{C})}$ of the recession cone of a\nspectrahedron $C$ is computed as\n\\begin{equation}\\label{EPCS}\n\\polar{(\\recc{C})} = \\cl \\left\\lbrace \\left( -A_1 \\cdot X, \\dots, -A_n\n\\cdot X \\right)\\T \\mid X \\mgeq 0 \\right\\rbrace, \n\\end{equation}\nwhere $A_i \\cdot X$ means the trace of the matrix product $A_iX$, see\n\\cite[Section 3]{GR95}. A cone $K$ is called \\emph{polyhedral} if ${K\n","start":11073,"end":12426}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"尖锥与无直线集合","source_units":[{"raw_tex":"\\cite[Section 3]{GR95}. A cone $K$ is called \\emph{polyhedral} if ${K\n= \\cone D}$ for some finite set $D$ and \\emph{pointed} if ${K \\cap\n(-K) = \\{0\\}}$. A set whose recession cone is pointed is called\n\\emph{line-free}. It is well known, that a closed convex set contains\n","start":12356,"end":12627}]},{"id":"S006","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S006"],"declared_external":true,"declaration_text":["原始前提列表明确允许：存在极点的标准结论，Roc70, Corollary 18.5.3；源文第249–251行。"],"source_only":false,"title":"无直线闭凸集存在极点","source_units":[{"raw_tex":"\\emph{line-free}. It is well known, that a closed convex set contains\nan extreme point if and only if it is line-free, see \\cite[Corollary\n18.5.3]{Roc70}. Given nonempty sets ${C_1, C_2 \\subseteq \\R^n}$ the\n","start":12557,"end":12764}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"有向超距","source_units":[{"raw_tex":"18.5.3]{Roc70}. Given nonempty sets ${C_1, C_2 \\subseteq \\R^n}$ the\n\\emph{excess of $C_1$ over $C_2$}, ${\\exc{C_1}{C_2}}$, is defined as\n\\begin{equation}\n\\exc{C_1}{C_2} = \\adjustlimits \\sup_{c_1 \\in C_1} \\inf_{c_2 \\in C_2}\n\\norm{c_1-c_2},\n\\end{equation}\nwhere~$\\norm{\\cdot}$ denotes the Euclidean norm. The \\emph{Hausdorff\n","start":12696,"end":13019}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"Hausdorff 距离","source_units":[{"raw_tex":"where~$\\norm{\\cdot}$ denotes the Euclidean norm. The \\emph{Hausdorff\ndistance between $C_1$ and $C_2$}, ${\\haus{C_1}{C_2}}$, is then\nexpressed as\n\\begin{equation}\n\\haus{C_1}{C_2} = \\max \\{\\exc{C_1}{C_2}, \\exc{C_2}{C_1}\\}.\n\\end{equation}\nIt is well known, that the Hausdorff distance defines a metric on the\nspace of nonempty compact subsets of $\\R^n$. Between unbounded sets\nthe Hausdorff distance may be infinite. A polyhedron $P$ is called an\n","start":12950,"end":13395}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"传统单误差逼近","source_units":[{"raw_tex":"the Hausdorff distance may be infinite. A polyhedron $P$ is called an\n\\textit{$\\varepsilon$-approximation} of a convex set $C$ if\n$\\haus{P}{C} \\leq \\varepsilon$.\n","start":13325,"end":13487}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"锥的截断距离","source_units":[{"raw_tex":"\\begin{definition}\n  Given nonempty closed convex cones~${K_1, K_2 \\subseteq \\R^n}$,\n  the \\emph{truncated Hausdorff distance between~$K_1$\n    and~$K_2$},~${\\thaus{K_1}{K_2}}$, is defined as\n  \\begin{equation}\n    \\thaus{K_1}{K_2} := \\haus{K_1 \\cap B_1(0)}{K_2 \\cap B_1(0)}.\n  \\end{equation}\n\\end{definition}\n","start":22126,"end":22436}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":["DEDA"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"双误差多面体逼近","source_units":[{"raw_tex":"\\begin{definition}\\label{DEDA}\n  Given a nonempty closed convex and line-free set~${C \\subseteq\n    \\R^n}$, a line-free polyhedron~$P$ is called an\n  \\emph{${\\ed}$-approximation of~$C$} if\n  \\begin{enumerate}[(i)]\n  \\item $\\exc{\\vertices P}{C} \\leq \\varepsilon$,\n  \\item $\\thaus{\\recc{P}}{\\recc{C}} \\leq \\delta$,\n  \\item $P \\supseteq C$.\n  \\end{enumerate}\n\\end{definition}\n","start":22952,"end":23325}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["DEDA"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"顶点误差条件","source_units":[{"raw_tex":"  \\begin{enumerate}[(i)]\n  \\item $\\exc{\\vertices P}{C} \\leq \\varepsilon$,\n","start":23141,"end":23215}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["DEDA"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"衰退锥误差条件","source_units":[{"raw_tex":"  \\item $\\thaus{\\recc{P}}{\\recc{C}} \\leq \\delta$,\n","start":23215,"end":23265}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":["DEDA"],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"外包含条件","source_units":[{"raw_tex":"  \\item $P \\supseteq C$.\n","start":23265,"end":23290}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"集合序列的极限定义","source_units":[{"raw_tex":"\\begin{definition}\n  A sequence~$\\{C^{\\nu}\\}_{\\nu \\in \\mathbb{N}}$ of subsets of~$\\R^n$ is said\n  to \\emph{converge to~${C \\subseteq \\R^n}$ in the sense of\n    Painlevé-Kuratowski}, denoted by~${C^{\\nu} \\rightarrow C}$, if the\n  following equalities hold:\n  \\begin{equation*}\n    \\begin{aligned}\n      C &= \\left\\lbrace x \\in \\R^n \\middle\\vert \\begin{matrix} \\text{for all open\n          neighbourhoods } U \\text{ of } x,\\\\ U \\cap C^{\\nu}  \\neq \\emptyset\n          \\text{ for large enough } \\nu\\end{matrix} \\right\\rbrace \\\\\n      &= \\left\\lbrace x \\in \\R^n \\middle\\vert \\begin{matrix} \\text{for all open\n        neighbourhoods } U \\text{ of } x,\\\\ U \\cap C^{\\nu} \\neq \\emptyset\n        \\text{ for infinitely many }\\nu\\end{matrix} \\right\\rbrace.\n    \\end{aligned}\n  \\end{equation*}\n\\end{definition}\n","start":24952,"end":25750}]},{"id":"S016","semantic_type":"theorem","kind":"theorem","labels":["TPKCHD"],"member_ids":["S016"],"declared_external":true,"declaration_text":["原始前提列表明确允许的 B0031：RW98, p.120 的收敛刻画；文内仅重述，无文内证明。"],"source_only":false,"title":"球截断收敛刻画","source_units":[{"raw_tex":"\\begin{theorem}[{see \\cite[p.~120]{RW98}}]\\label{TPKCHD}\n  A sequence~$C^{\\nu}$ of nonempty closed and convex sets converges to~$C$\n  in the sense of Painlevé-Kuratowski if and only if there exist~$x\n  \\in \\R^n$ and~$r_0 \\in \\R$, such that for all~$r \\geq r_0$ it holds, that\n  \\begin{equation}\n    \\haus{C^{\\nu} \\cap B_r(x)}{C \\cap B_r(x)} \\rightarrow 0.\n  \\end{equation}\n\\end{theorem}\n","start":26924,"end":27311}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","labels":["TPKC"],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"主定理的共同假设","source_units":[{"raw_tex":"  Let $C \\subseteq \\R^n$ be nonempty closed convex and\n  line-free. For $\\nu \\in \\mathbb{N}$ let~$P^{\\nu}$ be\n  an~$(\\varepsilon^{\\nu},\\delta^{\\nu})$-approximation of\n  $C$. If~${(\\varepsilon^{\\nu},\\delta^{\\nu}) \\rightarrow (0,0)}$,\n","start":36487,"end":36720}]},{"id":"S018","semantic_type":"theorem","kind":"theorem","labels":["TPKC"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"双误差逼近的集合收敛","source_units":[{"raw_tex":"\\begin{theorem}\\label{TPKC}\n  Let $C \\subseteq \\R^n$ be nonempty closed convex and\n  line-free. For $\\nu \\in \\mathbb{N}$ let~$P^{\\nu}$ be\n  an~$(\\varepsilon^{\\nu},\\delta^{\\nu})$-approximation of\n  $C$. If~${(\\varepsilon^{\\nu},\\delta^{\\nu}) \\rightarrow (0,0)}$,\n  then~${P^{\\nu} \\rightarrow C}$ in the sense of Painlevé-Kuratowski.\n\\end{theorem}","start":36459,"end":36803}]},{"id":"S019","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S019"],"declared_external":true,"declaration_text":["原始前提列表中的标准有限维凸分析：到凸集距离函数的凸性；由标注者显式列出。"],"source_only":false,"title":"到凸集距离的凸性","source_units":[]},{"id":"S020","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S020"],"declared_external":true,"declaration_text":["原始前提列表中的标准有限维凸分析：最近点投影。"],"source_only":false,"title":"闭凸集上的最近点","source_units":[]},{"id":"S021","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S021"],"declared_external":true,"declaration_text":["原始前提列表中的标准有限维凸分析：紧性和子列论证。"],"source_only":false,"title":"有限维紧性与子列","source_units":[]},{"id":"S022","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S022"],"declared_external":true,"declaration_text":["原始前提列表中的标准有限维凸分析：闭衰退锥；方向相加性质是该对象的标准定义性背景。"],"source_only":false,"title":"闭凸衰退锥的性质","source_units":[]},{"id":"S023","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S023"],"declared_external":true,"declaration_text":["原始前提列表明确允许：无直线多面体分解为顶点凸包与衰退锥之和。"],"source_only":false,"title":"无直线多面体的分解","source_units":[]},{"id":"S024","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S024"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator：实数和有限维向量序列的基本极限运算；不是源文单独陈述的辅助定理。"],"source_only":false,"title":"有界乘子的极限运算","source_units":[]},{"id":"S025","semantic_type":"proposition","kind":"proposition","labels":["PHDA"],"member_ids":["S025","S027","S028","S029","S030","S031","S032","S033","S034","S035"],"declared_external":false,"source_only":false,"title":"球截断的误差上界","source_units":[{"raw_tex":"\\begin{proposition}\\label{PHDA}\n  Let $C \\subseteq \\R^n$ be nonempty closed convex and line-free\n  and let~$P$ be an~$\\ed$-approximation of~$C$. Then for every~$x \\in\n  \\conv \\vertices P$ and~$r \\geq \\varepsilon$ it holds true, that\n  \\begin{equation}\n    \\haus{P \\cap B_r(x)}{C \\cap B_r(x)} \\leq 2\\left(\n      \\varepsilon+\\delta\\left(r+\\norm{x-v}\\right)\\right)\n  \\end{equation}\n  for some~$v \\in \\conv\\vertices P$. In particular, if~$\\haus{P \\cap\n    B_r(x)}{C \\cap B_r(x)}$ is attained as~$\\norm{p-c}$ with~$p \\in P$,\n  then~$p = v + td$ for some~$d \\in \\recc{P}$ and~$t \\geq 0$.\n\\end{proposition}","start":27980,"end":28579},{"raw_tex":"\\begin{proof}\n  Denote by $\\bar{P}$, $\\bar{C}$, and $V$, $P \\cap B_r(x)$, $C \\cap\n  B_r(x)$, and $\\conv \\vertices P$, respectively. Since~$\\bar{P},\n  \\bar{C}$ are nonempty convex and compact,~$\\haus{\\bar{P}}{\\bar{C}} =\n  \\norm{p^*-c^*}$ for some~$p^* \\in \\bar{P}$ and~$c^* \\in\n  \\bar{C}$. For~$\\lambda \\in [0,1]$ let~$z(\\lambda) = \\lambda p^* +\n  (1-\\lambda) x$. We distinguish two cases. First, assume~$p^* \\in\n  V$. Then~$z(\\lambda) \\in V$ for every~${\\lambda \\in [0,1]}$ and, due\n  to (i) in Definition \\ref{DEDA}, there is~${c_{\\lambda} \\in \\bar{C}}$\n  with~${\\norm{z(\\lambda)-c_{\\lambda}} \\leq\n    \\varepsilon}$. If~$\\norm{p^*-x} \\leq \\varepsilon$ then\n  \\[ \\norm{p^*-c^*} \\leq \\norm{p^*-c_0} \\leq \\norm{p^*-x} +\n    \\norm{x-c_0} \\leq 2\\varepsilon. \\] If~$\\norm{p^*-x} > \\varepsilon$,\n  set ${\\lambda}^* = \\frac{\\varepsilon}{\\norm{p^*-x}}$. Similarly, we\n  have\n  \\[ \\norm{p^*-c^*} \\leq \\norm{p^*-c_{{\\lambda}^*}} \\leq\n    \\norm{p^*-z({\\lambda}^*)} + \\norm{z({\\lambda}^*)-c_{{\\lambda}^*}} \\leq\n    2\\varepsilon. \\] Now, assume~${p^* \\notin V}$. Then there\n  exists~$\\bar{\\lambda} \\in (0,1)$, such that~$z(\\bar{\\lambda}) \\in V$\n  and~$z(\\lambda) \\notin V$ for all $\\lambda \\in\n  (\\bar{\\lambda},1]$. For~${\\lambda \\in (\\bar{\\lambda},1]}$,\n  $z(\\lambda)$ can be written as ${z(\\lambda) = v_{\\lambda} +\n    t_{\\lambda}d_{\\lambda}}$ with $v_{\\lambda} = \\argmin \\{\n      \\norm{z(\\bar{\\lambda}) - v} \\;\\mid\\;$ $v \\in \\left(\\{z(\\lambda)\\} -\n        \\recc{P}\\right) \\cap V \\}$, $t_{\\lambda} \\geq 0$ and\n  $d_{\\lambda} \\in \\recc{P}$, $\\norm{d_{\\lambda}} = 1$. By the\n  definition of $\\ed$-approximation there exist~$c_{\\lambda} \\in C$\n  and~$\\bar{d_{\\lambda}} \\in \\recc{C}$ such\n  that~$\\norm{v_{\\lambda}-c_{\\lambda}} \\leq \\varepsilon$\n  and~$\\norm{d_{\\lambda}-\\bar{d_{\\lambda}}} \\leq \\delta$. Now we have\n  \\begin{equation}\\label{proof_PHDA_1}\n    \\norm{z(\\lambda) - (c_{\\lambda} + t_{\\lambda}\\bar{d_{\\lambda}})} \\leq\n    \\norm{v_{\\lambda}-c_{\\lambda}} + t_{\\lambda}\n    \\norm{d_{\\lambda}-\\bar{d_{\\lambda}}} \\leq \\varepsilon +\n    t_{\\lambda}\\delta.\n  \\end{equation}\n  Furthermore, $t_{\\lambda} =\n  \\norm{z(\\lambda)-v_{\\lambda}}$, because $\\norm{d_{\\lambda}} =\n  1$. Hence,\n  \\begin{equation}\\label{proof_PHDA_2}\n    \\begin{aligned} t_{\\lambda} &\\leq\n      \\norm{z(\\lambda)-z(\\bar{\\lambda})} + \\norm{z(\\bar{\\lambda}) -\n        v_{\\lambda}} \\leq \\norm{z(\\lambda)-z(\\bar{\\lambda})} +\n      \\norm{z(\\bar{\\lambda})-v_1} \\\\ &\\leq\n      \\norm{z(\\lambda)-z(\\bar{\\lambda})} + \\norm{z(\\bar{\\lambda})-x} +\n      \\norm{x -v_1} \\\\ &= \\norm{z(\\lambda)-x} + \\norm{x-v_1}\\\\ &\\leq r\n      + \\norm{x-v_1}.\n    \\end{aligned}\n  \\end{equation}\nIf $\\frac{r-\\varepsilon}{r+\\norm{x-v_1}} < \\delta \\leq 1$ then \\[\n    \\norm{p^*-c^*} \\leq \\norm{p^*-c_0} \\leq \\norm{p^*-x} +\n    \\norm{x-c_0} \\leq r + \\varepsilon \\leq 2(\\varepsilon + \\delta\n    (r+\\norm{x-v_1})). \\] Otherwise, the last inequality is\n  violated. In this case, let~${\\lambda}^* = 1-\\frac{\\varepsilon +\n    \\delta(r\\norm{x-v_1})}{\\norm{p^*-x}}$. If~${\\lambda}^* \\leq\n  \\bar{\\lambda}$ then $z({\\lambda}^*) \\in V$ and there\n  exists~$c_{{\\lambda}^*} \\in C$ with~$\\norm{z({\\lambda}^*) -\n    c_{{\\lambda}^*}} \\leq \\varepsilon$. Therefore,\n  \\[ \\norm{p^*-c^*} \\leq \\norm{p^*-z({\\lambda}^*)} +\n    \\norm{z({\\lambda}^*)-c_{{\\lambda}^*}} \\leq 2 \\varepsilon + \\delta\n    (r+\\norm{x-v_1}). \\] If~${\\lambda}^* > \\bar{\\lambda}$ then,\n  according to \\eqref{proof_PHDA_1} and \\eqref{proof_PHDA_2}, there\n  exists~$c \\in C$ such that $\\norm{z({\\lambda}^*)-c} \\leq \\varepsilon\n  + \\delta(r+\\norm{x-v_1})$. Altogether this yields \\[ \\norm{p^*-c^*}\n    \\leq \\norm{p^*-c} \\leq \\norm{p^*-z({\\lambda}^*)} +\n    \\norm{z({\\lambda}^*)-c} \\leq 2 \\left( \\varepsilon + \\delta\n      (r+\\norm{x-v_1})\\right), \\] which completes the proof.\n\\end{proof}","start":28581,"end":32337},{"raw_tex":"  Denote by $\\bar{P}$, $\\bar{C}$, and $V$, $P \\cap B_r(x)$, $C \\cap\n  B_r(x)$, and $\\conv \\vertices P$, respectively. Since~$\\bar{P},\n  \\bar{C}$ are nonempty convex and compact,~$\\haus{\\bar{P}}{\\bar{C}} =\n  \\norm{p^*-c^*}$ for some~$p^* \\in \\bar{P}$ and~$c^* \\in\n  \\bar{C}$. For~$\\lambda \\in [0,1]$ let~$z(\\lambda) = \\lambda p^* +\n  (1-\\lambda) x$. We distinguish two cases. First, assume~$p^* \\in\n","start":28595,"end":28993},{"raw_tex":"  (1-\\lambda) x$. We distinguish two cases. First, assume~$p^* \\in\n  V$. Then~$z(\\lambda) \\in V$ for every~${\\lambda \\in [0,1]}$ and, due\n  to (i) in Definition \\ref{DEDA}, there is~${c_{\\lambda} \\in \\bar{C}}$\n  with~${\\norm{z(\\lambda)-c_{\\lambda}} \\leq\n    \\varepsilon}$. If~$\\norm{p^*-x} \\leq \\varepsilon$ then\n  \\[ \\norm{p^*-c^*} \\leq \\norm{p^*-c_0} \\leq \\norm{p^*-x} +\n    \\norm{x-c_0} \\leq 2\\varepsilon. \\] If~$\\norm{p^*-x} > \\varepsilon$,\n  set ${\\lambda}^* = \\frac{\\varepsilon}{\\norm{p^*-x}}$. Similarly, we\n  have\n  \\[ \\norm{p^*-c^*} \\leq \\norm{p^*-c_{{\\lambda}^*}} \\leq\n    \\norm{p^*-z({\\lambda}^*)} + \\norm{z({\\lambda}^*)-c_{{\\lambda}^*}} \\leq\n    2\\varepsilon. \\] Now, assume~${p^* \\notin V}$. Then there\n","start":28926,"end":29642},{"raw_tex":"    2\\varepsilon. \\] Now, assume~${p^* \\notin V}$. Then there\n  exists~$\\bar{\\lambda} \\in (0,1)$, such that~$z(\\bar{\\lambda}) \\in V$\n  and~$z(\\lambda) \\notin V$ for all $\\lambda \\in\n  (\\bar{\\lambda},1]$. For~${\\lambda \\in (\\bar{\\lambda},1]}$,\n  $z(\\lambda)$ can be written as ${z(\\lambda) = v_{\\lambda} +\n    t_{\\lambda}d_{\\lambda}}$ with $v_{\\lambda} = \\argmin \\{\n      \\norm{z(\\bar{\\lambda}) - v} \\;\\mid\\;$ $v \\in \\left(\\{z(\\lambda)\\} -\n        \\recc{P}\\right) \\cap V \\}$, $t_{\\lambda} \\geq 0$ and\n  $d_{\\lambda} \\in \\recc{P}$, $\\norm{d_{\\lambda}} = 1$. By the\n","start":29580,"end":30143},{"raw_tex":"  definition of $\\ed$-approximation there exist~$c_{\\lambda} \\in C$\n  and~$\\bar{d_{\\lambda}} \\in \\recc{C}$ such\n  that~$\\norm{v_{\\lambda}-c_{\\lambda}} \\leq \\varepsilon$\n  and~$\\norm{d_{\\lambda}-\\bar{d_{\\lambda}}} \\leq \\delta$. Now we have\n  \\begin{equation}\\label{proof_PHDA_1}\n    \\norm{z(\\lambda) - (c_{\\lambda} + t_{\\lambda}\\bar{d_{\\lambda}})} \\leq\n    \\norm{v_{\\lambda}-c_{\\lambda}} + t_{\\lambda}\n    \\norm{d_{\\lambda}-\\bar{d_{\\lambda}}} \\leq \\varepsilon +\n    t_{\\lambda}\\delta.\n  \\end{equation}\n","start":30143,"end":30644},{"raw_tex":"  Furthermore, $t_{\\lambda} =\n  \\norm{z(\\lambda)-v_{\\lambda}}$, because $\\norm{d_{\\lambda}} =\n  1$. Hence,\n  \\begin{equation}\\label{proof_PHDA_2}\n    \\begin{aligned} t_{\\lambda} &\\leq\n      \\norm{z(\\lambda)-z(\\bar{\\lambda})} + \\norm{z(\\bar{\\lambda}) -\n        v_{\\lambda}} \\leq \\norm{z(\\lambda)-z(\\bar{\\lambda})} +\n      \\norm{z(\\bar{\\lambda})-v_1} \\\\ &\\leq\n      \\norm{z(\\lambda)-z(\\bar{\\lambda})} + \\norm{z(\\bar{\\lambda})-x} +\n      \\norm{x -v_1} \\\\ &= \\norm{z(\\lambda)-x} + \\norm{x-v_1}\\\\ &\\leq r\n      + \\norm{x-v_1}.\n    \\end{aligned}\n  \\end{equation}\n","start":30644,"end":31201},{"raw_tex":"If $\\frac{r-\\varepsilon}{r+\\norm{x-v_1}} < \\delta \\leq 1$ then \\[\n    \\norm{p^*-c^*} \\leq \\norm{p^*-c_0} \\leq \\norm{p^*-x} +\n    \\norm{x-c_0} \\leq r + \\varepsilon \\leq 2(\\varepsilon + \\delta\n    (r+\\norm{x-v_1})). \\] Otherwise, the last inequality is\n","start":31201,"end":31452},{"raw_tex":"  violated. In this case, let~${\\lambda}^* = 1-\\frac{\\varepsilon +\n    \\delta(r\\norm{x-v_1})}{\\norm{p^*-x}}$. If~${\\lambda}^* \\leq\n  \\bar{\\lambda}$ then $z({\\lambda}^*) \\in V$ and there\n  exists~$c_{{\\lambda}^*} \\in C$ with~$\\norm{z({\\lambda}^*) -\n    c_{{\\lambda}^*}} \\leq \\varepsilon$. Therefore,\n  \\[ \\norm{p^*-c^*} \\leq \\norm{p^*-z({\\lambda}^*)} +\n    \\norm{z({\\lambda}^*)-c_{{\\lambda}^*}} \\leq 2 \\varepsilon + \\delta\n    (r+\\norm{x-v_1}). \\] If~${\\lambda}^* > \\bar{\\lambda}$ then,\n","start":31452,"end":31938},{"raw_tex":"    (r+\\norm{x-v_1}). \\] If~${\\lambda}^* > \\bar{\\lambda}$ then,\n  according to \\eqref{proof_PHDA_1} and \\eqref{proof_PHDA_2}, there\n  exists~$c \\in C$ such that $\\norm{z({\\lambda}^*)-c} \\leq \\varepsilon\n  + \\delta(r+\\norm{x-v_1})$. Altogether this yields \\[ \\norm{p^*-c^*}\n    \\leq \\norm{p^*-c} \\leq \\norm{p^*-z({\\lambda}^*)} +\n    \\norm{z({\\lambda}^*)-c} \\leq 2 \\left( \\varepsilon + \\delta\n      (r+\\norm{x-v_1})\\right), \\] which completes the proof.\n","start":31874,"end":32326}]},{"id":"S026","semantic_type":"assumption","kind":"assumption","labels":["PHDA"],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"截断估计的局部条件","source_units":[{"raw_tex":"  Let $C \\subseteq \\R^n$ be nonempty closed convex and line-free\n  and let~$P$ be an~$\\ed$-approximation of~$C$. Then for every~$x \\in\n  \\conv \\vertices P$ and~$r \\geq \\varepsilon$ it holds true, that\n","start":28012,"end":28213}]},{"id":"S036","semantic_type":"lemma","kind":"lemma","labels":["LBS"],"member_ids":["S036","S038","S039","S040","S041","S042"],"declared_external":false,"source_only":false,"title":"无直线性保证分解有界","source_units":[{"raw_tex":"\\begin{lemma}\\label{LBS}\n  Let $C \\subseteq \\R^n$ be nonempty closed and convex and let there\n  be sequences~$v^{\\nu}$,~$r^{\\nu}$ such that~$\\inf_{c \\in C}\n  \\norm{v^{\\nu}-c} \\rightarrow 0$, $\\inf_{r \\in \\recc{C}}\n  \\norm{r^{\\nu}-r} \\rightarrow 0$, and $v^{\\nu}+r^{\\nu} \\in\n    B_M(x)$ for some $M \\geq 0$ and $x \\in \\R^n$. If $C$ is\n  line-free, then $\\norm{v^{\\nu}}$ is bounded.\n\\end{lemma}","start":32411,"end":32803},{"raw_tex":"\\begin{proof}\n  Assume that $\\norm{v^{\\nu}}$ is unbounded. This implies\n  that~$\\norm{r^{\\nu}}$ is also unbounded and $\\recc{C} \\neq\n  \\{0\\}$. Without loss of generality let $r^{\\nu} \\neq 0$ for all\n  $\\nu$. Then $d^{\\nu} := r^{\\nu}/\\norm{r^{\\nu}}$ is bounded and has a\n  convergent subsequence. Without loss of generality we can assume\n  that $d^{\\nu} \\rightarrow d \\in \\recc{C}$. We will show, that $-d\n  \\in \\recc{C}$. Therefore let $c \\in C$, $t \\geq 0$ and define $y^{\\nu} =\n  \\argmin_{y \\in C} \\norm{v^{\\nu}-y}$. By the triangle inequality it\n  holds true, that\n  \\begin{equation}\\label{proof_LBS}\n    \\norm{c-r^{\\nu}-y^{\\nu}} \\leq M + \\norm{c-x} +\n    \\norm{v^{\\nu}-y^{\\nu}} =: \\ols{M}^{\\nu}.\n  \\end{equation}\n  Note, that $\\ols{M}^{\\nu}$ is bounded from above by some $\\ols{M}$, because\n  $\\norm{v^{\\nu}-y^{\\nu}} \\rightarrow 0$. For every $T \\geq 0$ there\n  exists some $\\nu_T$, such that $\\norm{r^{\\nu_T}} \\geq T$. Let $T\n  \\geq t$ and define\n  \\[ \\bar{y} := \\frac{t}{\\norm{r^{\\nu_T}}} y^{\\nu_T} + \\left(\n      1-\\frac{t}{\\norm{r^{\\nu_T}}} \\right) c \\in C. \\]\n  Putting it all together one gets\n  \\begin{equation*}\n    \\begin{aligned}\n      \\inf_{y \\in C} \\norm{(c-td^{\\nu_T})-y} &\\leq\n      \\norm{(c-td^{\\nu_T})-\\bar{y}} \\\\\n      &= \\norm{\\frac{t}{\\norm{r^{\\nu_T}}}c\n        -td^{\\nu_T}-\\frac{t}{\\norm{r^{\\nu_T}}}y^{\\nu_T}} \\\\\n      &= \\frac{t}{\\norm{r^{\\nu_T}}} \\norm{c-r^{\\nu_T}-y^{\\nu_T}} \\\\\n      &\\leq \\frac{t}{T} \\ols{M},\n    \\end{aligned}\n  \\end{equation*}\n  where the last inequality holds due to \\eqref{proof_LBS} and\n  boundedness of $\\ols{M}^{\\nu}$. Since $C$ is closed and $d^{\\nu}\n  \\rightarrow d \\in \\recc{C}$, taking the limit $T \\rightarrow\n  +\\infty$ yields that $c-td \\in C$. This is a contradiction to the\n  pointedness of $\\recc{C}$.\n\\end{proof}","start":32805,"end":34580},{"raw_tex":"  Assume that $\\norm{v^{\\nu}}$ is unbounded. This implies\n  that~$\\norm{r^{\\nu}}$ is also unbounded and $\\recc{C} \\neq\n  \\{0\\}$. Without loss of generality let $r^{\\nu} \\neq 0$ for all\n  $\\nu$. Then $d^{\\nu} := r^{\\nu}/\\norm{r^{\\nu}}$ is bounded and has a\n  convergent subsequence. Without loss of generality we can assume\n  that $d^{\\nu} \\rightarrow d \\in \\recc{C}$. We will show, that $-d\n  \\in \\recc{C}$. Therefore let $c \\in C$, $t \\geq 0$ and define $y^{\\nu} =\n","start":32819,"end":33285},{"raw_tex":"  \\in \\recc{C}$. Therefore let $c \\in C$, $t \\geq 0$ and define $y^{\\nu} =\n  \\argmin_{y \\in C} \\norm{v^{\\nu}-y}$. By the triangle inequality it\n  holds true, that\n  \\begin{equation}\\label{proof_LBS}\n    \\norm{c-r^{\\nu}-y^{\\nu}} \\leq M + \\norm{c-x} +\n    \\norm{v^{\\nu}-y^{\\nu}} =: \\ols{M}^{\\nu}.\n  \\end{equation}\n  Note, that $\\ols{M}^{\\nu}$ is bounded from above by some $\\ols{M}$, because\n  $\\norm{v^{\\nu}-y^{\\nu}} \\rightarrow 0$. For every $T \\geq 0$ there\n","start":33210,"end":33669},{"raw_tex":"  $\\norm{v^{\\nu}-y^{\\nu}} \\rightarrow 0$. For every $T \\geq 0$ there\n  exists some $\\nu_T$, such that $\\norm{r^{\\nu_T}} \\geq T$. Let $T\n  \\geq t$ and define\n  \\[ \\bar{y} := \\frac{t}{\\norm{r^{\\nu_T}}} y^{\\nu_T} + \\left(\n      1-\\frac{t}{\\norm{r^{\\nu_T}}} \\right) c \\in C. \\]\n  Putting it all together one gets\n  \\begin{equation*}\n    \\begin{aligned}\n      \\inf_{y \\in C} \\norm{(c-td^{\\nu_T})-y} &\\leq\n      \\norm{(c-td^{\\nu_T})-\\bar{y}} \\\\\n      &= \\norm{\\frac{t}{\\norm{r^{\\nu_T}}}c\n        -td^{\\nu_T}-\\frac{t}{\\norm{r^{\\nu_T}}}y^{\\nu_T}} \\\\\n      &= \\frac{t}{\\norm{r^{\\nu_T}}} \\norm{c-r^{\\nu_T}-y^{\\nu_T}} \\\\\n      &\\leq \\frac{t}{T} \\ols{M},\n    \\end{aligned}\n  \\end{equation*}\n","start":33600,"end":34279},{"raw_tex":"  where the last inequality holds due to \\eqref{proof_LBS} and\n  boundedness of $\\ols{M}^{\\nu}$. Since $C$ is closed and $d^{\\nu}\n  \\rightarrow d \\in \\recc{C}$, taking the limit $T \\rightarrow\n  +\\infty$ yields that $c-td \\in C$. This is a contradiction to the\n  pointedness of $\\recc{C}$.\n","start":34279,"end":34569}]},{"id":"S037","semantic_type":"assumption","kind":"assumption","labels":["LBS"],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"有界性引理的精确前提","source_units":[{"raw_tex":"  Let $C \\subseteq \\R^n$ be nonempty closed and convex and let there\n  be sequences~$v^{\\nu}$,~$r^{\\nu}$ such that~$\\inf_{c \\in C}\n  \\norm{v^{\\nu}-c} \\rightarrow 0$, $\\inf_{r \\in \\recc{C}}\n  \\norm{r^{\\nu}-r} \\rightarrow 0$, and $v^{\\nu}+r^{\\nu} \\in\n    B_M(x)$ for some $M \\geq 0$ and $x \\in \\R^n$. If $C$ is\n  line-free, then $\\norm{v^{\\nu}}$ is bounded.\n","start":32436,"end":32792}]},{"id":"S043","semantic_type":"proposition","kind":"proposition","labels":["PEP"],"member_ids":["S043","S045","S046","S047","S048","S049"],"declared_external":false,"source_only":false,"title":"极点的顶点凸包逼近","source_units":[{"raw_tex":"\\begin{proposition}\\label{PEP}\n  Let $C \\subseteq \\R^n$ be nonempty closed convex and\n  line-free. For $\\nu \\in \\mathbb{N}$ let~$P^{\\nu}$ be an\n  $(\\varepsilon^{\\nu},\\delta^{\\nu})$-approximation of\n  $C$. If~${(\\varepsilon^{\\nu},\\delta^{\\nu}) \\rightarrow (0,0)}$, then\n  for every extreme point~$c$ of $C$ there exists a sequence~${x^{\\nu}\n    \\rightarrow c}$ such that $x^{\\nu} \\in \\conv \\vertices P^{\\nu}$.\n\\end{proposition}","start":35010,"end":35436},{"raw_tex":"\\begin{proof} Since $C$ is line-free, it has at least one extreme\n  point. Let $c$ be one such extreme point. Assume that for every\n  sequence $x^{\\nu}$ with~${x^{\\nu} \\in \\conv \\vertices P^{\\nu}}$ there\n  exists a $\\gamma > 0$, such that~${\\norm{x^{\\nu}-c} > \\gamma}$ for\n  infinitely many $\\nu$. Then, without loss of generality, there exists\n  one such sequence such that~$\\norm{x^{\\nu}-c} > \\gamma$ for every\n  $\\nu$ and, since $C \\subseteq P^{\\nu}$, $c=x^{\\nu}+r^{\\nu}$ for some\n  $r^{\\nu} \\in \\recc{P^{\\nu}}$. By Lemma \\ref{LBS} it holds that\n","start":35438,"end":35987},{"raw_tex":"\\begin{proof} Since $C$ is line-free, it has at least one extreme\n  point. Let $c$ be one such extreme point. Assume that for every\n  sequence $x^{\\nu}$ with~${x^{\\nu} \\in \\conv \\vertices P^{\\nu}}$ there\n  exists a $\\gamma > 0$, such that~${\\norm{x^{\\nu}-c} > \\gamma}$ for\n  infinitely many $\\nu$. Then, without loss of generality, there exists\n  one such sequence such that~$\\norm{x^{\\nu}-c} > \\gamma$ for every\n  $\\nu$ and, since $C \\subseteq P^{\\nu}$, $c=x^{\\nu}+r^{\\nu}$ for some\n  $r^{\\nu} \\in \\recc{P^{\\nu}}$. By Lemma \\ref{LBS} it holds that\n  $\\norm{x^{\\nu}}$ and $\\norm{r^{\\nu}}$ are bounded. Then there exist\n  subsequences $x^{\\nu_k}$, $r^{\\nu_k}$ such that\n  \\[ x^{\\nu_k} \\rightarrow x \\in C, \\quad r^{\\nu_k} \\rightarrow r \\in\n    \\recc{C}. \\] Note that $r \\neq 0$, because $\\norm{r^{\\nu}} > \\gamma$\n  for all $\\nu$. Finally,\n  \\[ c=x+r=\\frac{1}{2}\\left( x+2r \\right) + \\frac{1}{2} x. \\] This is\n  a contradiction to $c$ being an extreme point of $C$.\n\\end{proof}","start":35438,"end":36413},{"raw_tex":"  $r^{\\nu} \\in \\recc{P^{\\nu}}$. By Lemma \\ref{LBS} it holds that\n  $\\norm{x^{\\nu}}$ and $\\norm{r^{\\nu}}$ are bounded. Then there exist\n","start":35922,"end":36057},{"raw_tex":"  $\\norm{x^{\\nu}}$ and $\\norm{r^{\\nu}}$ are bounded. Then there exist\n  subsequences $x^{\\nu_k}$, $r^{\\nu_k}$ such that\n  \\[ x^{\\nu_k} \\rightarrow x \\in C, \\quad r^{\\nu_k} \\rightarrow r \\in\n    \\recc{C}. \\] Note that $r \\neq 0$, because $\\norm{r^{\\nu}} > \\gamma$\n","start":35987,"end":36250},{"raw_tex":"    \\recc{C}. \\] Note that $r \\neq 0$, because $\\norm{r^{\\nu}} > \\gamma$\n  for all $\\nu$. Finally,\n  \\[ c=x+r=\\frac{1}{2}\\left( x+2r \\right) + \\frac{1}{2} x. \\] This is\n  a contradiction to $c$ being an extreme point of $C$.\n","start":36177,"end":36402}]},{"id":"S044","semantic_type":"assumption","kind":"assumption","labels":["PEP"],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"极点逼近命题的条件","source_units":[{"raw_tex":"  Let $C \\subseteq \\R^n$ be nonempty closed convex and\n  line-free. For $\\nu \\in \\mathbb{N}$ let~$P^{\\nu}$ be an\n  $(\\varepsilon^{\\nu},\\delta^{\\nu})$-approximation of\n  $C$. If~${(\\varepsilon^{\\nu},\\delta^{\\nu}) \\rightarrow (0,0)}$, then\n  for every extreme point~$c$ of $C$ there exists a sequence~${x^{\\nu}\n","start":35041,"end":35350}]},{"id":"S050","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"主收敛定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n  By Theorem \\ref{TPKCHD} we must show that there exist $c \\in \\R^n$\n  and $r_0 \\geq 0$ such that for all $r \\geq r_0$ it holds $\\haus{P^{\\nu} \\cap B_r(c)}{C \\cap B_r(c)}\n  \\rightarrow 0$. Let~${r \\geq \\max_{\\nu \\in\n      \\mathbb{N}} \\varepsilon^{\\nu}}$ and let $c$ be an extreme point of $C$,\n  which exists, because $C$ contains no lines. By Proposition \\ref{PEP}\n  there exists a sequence~${x^{\\nu} \\rightarrow c}$ with~${x^{\\nu} \\in\n    \\conv \\vertices P^{\\nu}}$. Applying the triangle inequality and\n  Proposition \\ref{PHDA} yields\n  \\begin{equation*}\n    \\begin{aligned}\n      & & &  \\haus{P^{\\nu} \\cap B_r(c)}{C \\cap B_r(c)} \\\\\n      & \\leq & & \\haus{P^{\\nu} \\cap B_r(c)}{P^{\\nu} \\cap B_r(x^{\\nu})}\n      \\\\ & & + & \\haus{P^{\\nu} \\cap B_r(x^{\\nu})}{C \\cap B_r(x^{\\nu})}\n      \\\\ & & + & \\haus{C \\cap B_r(x^{\\nu})}{C \\cap B_r(c)} \\\\\n      & \\leq & & \\haus{P^{\\nu} \\cap B_r(c)}{P^{\\nu} \\cap B_r(x^{\\nu})}\n      \\\\ & & + & 2(\\varepsilon^{\\nu}+\\delta^{\\nu}(r+\\norm{x^{\\nu}-v^{\\nu}}))\n      \\\\ & & + & \\haus{C \\cap B_r(x^{\\nu})}{C \\cap B_r(c)},\n   \\end{aligned}\n \\end{equation*}\n for some $v^{\\nu} \\in \\conv \\vertices P^{\\nu}$. The first and third\n term in this sum converge to zero as~${x^{\\nu} \\rightarrow c}$. It\n remains to show that~$\\norm{x^{\\nu}-v^{\\nu}}$ is\n bounded. Since $C \\subseteq P^{\\nu}$, the distance ${\\haus{P^{\\nu}\n   \\cap B_r(x^{\\nu})}{C \\cap B_r(x^{\\nu})}}$ is attained as\n $\\exc{P^{\\nu} \\cap B_r(x^{\\nu})}{C \\cap B_r(x^{\\nu})}$. Let the\n supremum be attained by~$p^{\\nu} \\in P^{\\nu}$. Then $p^{\\nu} =\n   v^{\\nu} + d^{\\nu}$ for some~$d^{\\nu} \\in \\recc{P^{\\nu}}$. It holds\n \\[ \\norm{p^{\\nu}-c} \\leq \\norm{p^{\\nu}-x^{\\nu}} + \\norm{x^{\\nu}-c}\n   \\leq r + \\max_{\\nu \\in \\mathbb{N}} \\norm{x^{\\nu}-c} < +\\infty, \\]\n i.e. $v^{\\nu}+d^{\\nu} \\in B_M(c)$ for some $M \\geq 0$. Therefore, the\n sequence $\\norm{v^{\\nu}}$ is bounded according to Lemma\n \\ref{LBS}. Hence, $\\norm{x^{\\nu}-v^{\\nu}}$ is also bounded\n and $\\haus{P^{\\nu} \\cap B_r(c)}{C \\cap B_r(c)} \\rightarrow 0$,\n which was to be proved.\n\\end{proof}","start":36805,"end":38839}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"约化到固定中心的球截断","source_units":[{"raw_tex":"  By Theorem \\ref{TPKCHD} we must show that there exist $c \\in \\R^n$\n  and $r_0 \\geq 0$ such that for all $r \\geq r_0$ it holds $\\haus{P^{\\nu} \\cap B_r(c)}{C \\cap B_r(c)}\n  \\rightarrow 0$. Let~${r \\geq \\max_{\\nu \\in\n","start":36819,"end":37035}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"极点中心与逼近中心","source_units":[{"raw_tex":"  \\rightarrow 0$. Let~${r \\geq \\max_{\\nu \\in\n      \\mathbb{N}} \\varepsilon^{\\nu}}$ and let $c$ be an extreme point of $C$,\n  which exists, because $C$ contains no lines. By Proposition \\ref{PEP}\n  there exists a sequence~${x^{\\nu} \\rightarrow c}$ with~${x^{\\nu} \\in\n    \\conv \\vertices P^{\\nu}}$. Applying the triangle inequality and\n","start":36990,"end":37324}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"移动中心的三项估计","source_units":[{"raw_tex":"    \\conv \\vertices P^{\\nu}}$. Applying the triangle inequality and\n  Proposition \\ref{PHDA} yields\n  \\begin{equation*}\n    \\begin{aligned}\n      & & &  \\haus{P^{\\nu} \\cap B_r(c)}{C \\cap B_r(c)} \\\\\n      & \\leq & & \\haus{P^{\\nu} \\cap B_r(c)}{P^{\\nu} \\cap B_r(x^{\\nu})}\n      \\\\ & & + & \\haus{P^{\\nu} \\cap B_r(x^{\\nu})}{C \\cap B_r(x^{\\nu})}\n      \\\\ & & + & \\haus{C \\cap B_r(x^{\\nu})}{C \\cap B_r(c)} \\\\\n      & \\leq & & \\haus{P^{\\nu} \\cap B_r(c)}{P^{\\nu} \\cap B_r(x^{\\nu})}\n      \\\\ & & + & 2(\\varepsilon^{\\nu}+\\delta^{\\nu}(r+\\norm{x^{\\nu}-v^{\\nu}}))\n      \\\\ & & + & \\haus{C \\cap B_r(x^{\\nu})}{C \\cap B_r(c)},\n   \\end{aligned}\n \\end{equation*}\n for some $v^{\\nu} \\in \\conv \\vertices P^{\\nu}$. The first and third\n","start":37256,"end":37969}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"作者断言换中心项趋零","source_units":[{"raw_tex":" for some $v^{\\nu} \\in \\conv \\vertices P^{\\nu}$. The first and third\n term in this sum converge to zero as~${x^{\\nu} \\rightarrow c}$. It\n","start":37900,"end":38037}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"最坏截断点与相应分解","source_units":[{"raw_tex":" remains to show that~$\\norm{x^{\\nu}-v^{\\nu}}$ is\n bounded. Since $C \\subseteq P^{\\nu}$, the distance ${\\haus{P^{\\nu}\n   \\cap B_r(x^{\\nu})}{C \\cap B_r(x^{\\nu})}}$ is attained as\n $\\exc{P^{\\nu} \\cap B_r(x^{\\nu})}{C \\cap B_r(x^{\\nu})}$. Let the\n supremum be attained by~$p^{\\nu} \\in P^{\\nu}$. Then $p^{\\nu} =\n   v^{\\nu} + d^{\\nu}$ for some~$d^{\\nu} \\in \\recc{P^{\\nu}}$. It holds\n","start":38037,"end":38414}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"取得点落在统一固定球","source_units":[{"raw_tex":" \\[ \\norm{p^{\\nu}-c} \\leq \\norm{p^{\\nu}-x^{\\nu}} + \\norm{x^{\\nu}-c}\n   \\leq r + \\max_{\\nu \\in \\mathbb{N}} \\norm{x^{\\nu}-c} < +\\infty, \\]\n i.e. $v^{\\nu}+d^{\\nu} \\in B_M(c)$ for some $M \\geq 0$. Therefore, the\n","start":38414,"end":38622}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"再次调用分解有界性引理","source_units":[{"raw_tex":" i.e. $v^{\\nu}+d^{\\nu} \\in B_M(c)$ for some $M \\geq 0$. Therefore, the\n sequence $\\norm{v^{\\nu}}$ is bounded according to Lemma\n \\ref{LBS}. Hence, $\\norm{x^{\\nu}-v^{\\nu}}$ is also bounded\n","start":38551,"end":38739}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"误差趋零并结束约化","source_units":[{"raw_tex":" \\ref{LBS}. Hence, $\\norm{x^{\\nu}-v^{\\nu}}$ is also bounded\n and $\\haus{P^{\\nu} \\cap B_r(c)}{C \\cap B_r(c)} \\rightarrow 0$,\n which was to be proved.\n","start":38679,"end":38828}]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S025","S028":"S025","S029":"S025","S030":"S025","S031":"S025","S032":"S025","S033":"S025","S034":"S025","S035":"S025","S036":"S036","S037":"S037","S038":"S036","S039":"S036","S040":"S036","S041":"S036","S042":"S036","S043":"S043","S044":"S044","S045":"S043","S046":"S043","S047":"S043","S048":"S043","S049":"S043","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058"}}}}
{"record_id":"2003.02400v2","paper":{"title":"Bounds for the tracking error of first-order online optimization methods","authors":["Madden, Liam","Becker, Stephen","Dall'Anese, Emiliano"],"arxiv_id":"2003.02400","version":"v2","journal":"Journal of Optimization Theory and Applications","abstract_url":"https://arxiv.org/abs/2003.02400v2","pdf_url":"https://arxiv.org/pdf/2003.02400v2","doi":"10.1007/s10957-021-01836-9"},"context":{"raw_tex":"This section reviews useful results from convex analysis~\\cite{Beck,Combettes,Nesterov,Nesterov2,Boyd,Bubeck}, and introduces key definitions that will be used throughout the paper. The paper considers functions on a Hilbert space $\\Hi$ with corresponding norm $\\|\\cdot\\|$. We assume all functions, $f_t: \\Hi \\rightarrow \\R$,  are  proper, lower semicontinuous, convex, and strongly smooth; $t\\in\\N\\cup\\{0\\}$ denotes the time index~\\cite{SPMag,popkov2005gradient,SimonettoGlobalsip2014,mokhtari2016online}. The discretized time-varying optimization problem of interest is the sequence of convex problems (one for each $t$): \n\\begin{align}\n\\label{eq:time-varying-opt}\n   \\min_{x\\in\\Hi} f_t(x) \\,, \\hspace{.5cm} t \\in\\N\\cup\\{0\\}\n\\end{align}\nalong with a time-varying minimizer set. In particular, assume that the minimizer set, denoted as $X_t\\opt$, is nonempty for each $t$. Let $x_t\\opt$ denote an element of $X_t\\opt$ and $f_t\\opt=f_t(x_t\\opt)$. % the optimal value function.\nWe denote the set of sequences of $L$-strongly smooth functions by $\\SS'(L)$. For function sequences that are additionally $\\mu$-strongly convex, $X_t\\opt$ contains only one point; therefore, we can measure the temporal variability of the optimal solution trajectory of~\\eqref{eq:time-varying-opt} with the sequence \n\\begin{align*}\n    \\sigma_t :=\\|x_{t+1}\\opt-x_t\\opt\\| , \\hspace{.5cm} t\\in \\N\\cup\\{0\\}.\n\\end{align*}\nWe define $\\SS(\\kappa^{-1},L,\\sigma)$, where $\\kappa$ is the condition number, as the set of $L$-strongly smooth, $\\kappa^{-1}L$-strongly convex functions sequences with $\\sigma_t \\leq \\sigma$ for all $t$. For both $\\SS'(L)$ and $\\SS(\\kappa^{-1},L,\\sigma)$, the Hilbert space, along with its dimension, is left implicit since the results are independent of it.\n\n\nDue to the temporal variability, we cannot expect any of the error sequences to converge to zero in general. Thus, we will characterize the performance of online algorithms via bounds on the limit supremum of the errors, which we term ``tracking'' error, rather than bounds on the convergence rate (to the tracking error ball).\n\n\nIn this section, we present a long-step Nesterov's method; the term ``long-steps'' refers to the fact that the algorithm takes a certain number of steps using the same stale function before catching up to the most recent function and repeating. For this particular long-step Nesterov's method, we are able to prove upper bounds on the tracking error (on the other hand, no bounds for the online Nesterov's method are yet available, and are the subject of current efforts). \n\nThe specific sequence constructed by $\\OLNM(T)$ is defined in Algorithm~\\ref{algo:OLNM}.\n\\begin{algorithm}[h]\n\\begin{algorithmic}[1]\n\\Require $x_0$\n\\State $y_0\\gets x_0$, $z_0\\gets x_0$, $a_0\\gets 1$\n\\For{$t=1,2,\\ldots$} %\\Comment{We have the answer if r is 0}\n          \\State $z_{t+1} =y_t-\\frac{1}{L}\\nabla f_{T\\lfloor t/T \\rfloor}(y_t)$\n \\If{     $T \\not| ~ t+1$ }\n \\State $a_{t+1} = \\frac{1+\\sqrt{1+4a_t^2}}{2}$\n \\State $y_{t+1} = z_{t+1} +\\frac{a_t-1}{a_{t+1}}(z_{t+1}-z_t)$\n \\State $x_{t+1} = x_t$ \\Comment{so $x_{t+1} = x_t = x_{T\\lfloor t/T\\rfloor}$}\n \\ElsIf{$T\\mid t+1$}\n \\State $a_{t+1} = 1$\n \\State $y_{t+1} = z_{t+1}$\n \\State $x_{t+1} = z_{t+1}$\n \\EndIf\n\\EndFor\n\\end{algorithmic}\n\\caption{$\\OLNM(T)$}\n\\label{algo:OLNM}\n\\end{algorithm}\n\nThe method can be extended to the composite case by applying the proximal operator to the $z$ iterates. Theorem \\ref{thm:onlineNesterov} gives an upper bound on the tracking iterate error of $\\OLNM$, using results from \\cite[Thm. 10.34]{Beck}.\n","units":[{"raw_tex":"This section reviews useful results from convex analysis~\\cite{Beck,Combettes,Nesterov,Nesterov2,Boyd,Bubeck}, and introduces key definitions that will be used throughout the paper. The paper considers functions on a Hilbert space $\\Hi$ with corresponding norm $\\|\\cdot\\|$. We assume all functions, $f_t: \\Hi \\rightarrow \\R$,  are  proper, lower semicontinuous, convex, and strongly smooth; $t\\in\\N\\cup\\{0\\}$ denotes the time index~\\cite{SPMag,popkov2005gradient,SimonettoGlobalsip2014,mokhtari2016online}. The discretized time-varying optimization problem of interest is the sequence of convex problems (one for each $t$): \n\\begin{align}\n\\label{eq:time-varying-opt}\n   \\min_{x\\in\\Hi} f_t(x) \\,, \\hspace{.5cm} t \\in\\N\\cup\\{0\\}\n\\end{align}\nalong with a time-varying minimizer set. In particular, assume that the minimizer set, denoted as $X_t\\opt$, is nonempty for each $t$. Let $x_t\\opt$ denote an element of $X_t\\opt$ and $f_t\\opt=f_t(x_t\\opt)$. % the optimal value function.\nWe denote the set of sequences of $L$-strongly smooth functions by $\\SS'(L)$. For function sequences that are additionally $\\mu$-strongly convex, $X_t\\opt$ contains only one point; therefore, we can measure the temporal variability of the optimal solution trajectory of~\\eqref{eq:time-varying-opt} with the sequence \n\\begin{align*}\n    \\sigma_t :=\\|x_{t+1}\\opt-x_t\\opt\\| , \\hspace{.5cm} t\\in \\N\\cup\\{0\\}.\n\\end{align*}\nWe define $\\SS(\\kappa^{-1},L,\\sigma)$, where $\\kappa$ is the condition number, as the set of $L$-strongly smooth, $\\kappa^{-1}L$-strongly convex functions sequences with $\\sigma_t \\leq \\sigma$ for all $t$. For both $\\SS'(L)$ and $\\SS(\\kappa^{-1},L,\\sigma)$, the Hilbert space, along with its dimension, is left implicit since the results are independent of it.\n","start":6131,"end":7887},{"raw_tex":"Due to the temporal variability, we cannot expect any of the error sequences to converge to zero in general. Thus, we will characterize the performance of online algorithms via bounds on the limit supremum of the errors, which we term ``tracking'' error, rather than bounds on the convergence rate (to the tracking error ball).\n","start":8915,"end":9243},{"raw_tex":"In this section, we present a long-step Nesterov's method; the term ``long-steps'' refers to the fact that the algorithm takes a certain number of steps using the same stale function before catching up to the most recent function and repeating. For this particular long-step Nesterov's method, we are able to prove upper bounds on the tracking error (on the other hand, no bounds for the online Nesterov's method are yet available, and are the subject of current efforts). \n\nThe specific sequence constructed by $\\OLNM(T)$ is defined in Algorithm~\\ref{algo:OLNM}.\n\\begin{algorithm}[h]\n\\begin{algorithmic}[1]\n\\Require $x_0$\n\\State $y_0\\gets x_0$, $z_0\\gets x_0$, $a_0\\gets 1$\n\\For{$t=1,2,\\ldots$} %\\Comment{We have the answer if r is 0}\n          \\State $z_{t+1} =y_t-\\frac{1}{L}\\nabla f_{T\\lfloor t/T \\rfloor}(y_t)$\n \\If{     $T \\not| ~ t+1$ }\n \\State $a_{t+1} = \\frac{1+\\sqrt{1+4a_t^2}}{2}$\n \\State $y_{t+1} = z_{t+1} +\\frac{a_t-1}{a_{t+1}}(z_{t+1}-z_t)$\n \\State $x_{t+1} = x_t$ \\Comment{so $x_{t+1} = x_t = x_{T\\lfloor t/T\\rfloor}$}\n \\ElsIf{$T\\mid t+1$}\n \\State $a_{t+1} = 1$\n \\State $y_{t+1} = z_{t+1}$\n \\State $x_{t+1} = z_{t+1}$\n \\EndIf\n\\EndFor\n\\end{algorithmic}\n\\caption{$\\OLNM(T)$}\n\\label{algo:OLNM}\n\\end{algorithm}\n\nThe method can be extended to the composite case by applying the proximal operator to the $z$ iterates. Theorem \\ref{thm:onlineNesterov} gives an upper bound on the tracking iterate error of $\\OLNM$, using results from \\cite[Thm. 10.34]{Beck}.\n","start":32969,"end":34437}]},"statement":{"raw_tex":"\\begin{theorem} \\label{thm:onlineNesterov}\nIf $(f_t)\\in \\SS(\\kappa^{-1},L,\\sigma)$, then, given $x_0$, $\\OLNM(T)$ where $T=c\\sqrt{\\kappa}$ for $c>2$ such that $c\\sqrt{\\kappa}\\in\\N$, constructs a sequence $(x_t)$ such that\n\\begin{align} \\label{eq:OLNM}\n    \\limsup_{t\\to\\infty}\\|x_t-x_t\\opt\\|\\leq \\frac{2c(c-1)}{c-2}\\sqrt{\\kappa}\\sigma.\n\\end{align}\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem} \\label{thm:onlineNesterov}\nIf $(f_t)\\in \\SS(\\kappa^{-1},L,\\sigma)$, then, given $x_0$, $\\OLNM(T)$ where $T=c\\sqrt{\\kappa}$ for $c>2$ such that $c\\sqrt{\\kappa}\\in\\N$, constructs a sequence $(x_t)$ such that\n\\begin{align} \\label{eq:OLNM}\n    \\limsup_{t\\to\\infty}\\|x_t-x_t\\opt\\|\\leq \\frac{2c(c-1)}{c-2}\\sqrt{\\kappa}\\sigma.\n\\end{align}\n\\end{theorem}","start":34438,"end":34799}]},"proof":{"raw_tex":"\\begin{proof}\nFirst, via standard batch optimization bounds, we have\n\\begin{align*}\n    \\|x_{kT}-x_{(k-1)T}\\opt\\|\\leq \\frac{2\\sqrt{\\kappa}}{T}\\|x_{(k-1)T}-x_{(k-1)T}\\opt\\|.\n\\end{align*}\nThus,\n\\begin{align*}\n    \\|x_{kT}-x_{kT}\\opt\\|&\\leq\\frac{2\\sqrt{\\kappa}}{T}\\|x_{(k-1)T}-x_{(k-1)T}\\opt\\|+T\\sigma\\\\\n    &\\leq \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^k\\|x_0-x_0\\opt\\|+\\frac{T\\sigma}{1-\\frac{2\\sqrt{\\kappa}}{T}}\\\\\n    &= \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^k\\|x_0-x_0\\opt\\|+\\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}\n\\end{align*}\nand so\n\\begin{align*}\n    \\|x_t-x_t\\opt\\| &= \\|x_{T\\lfloor t/T \\rfloor}-x_t\\opt\\|\\\\\n    &\\leq \\|x_{T\\lfloor t/T \\rfloor}-x_{T\\lfloor t/T \\rfloor}\\opt\\|+\\|x_t\\opt-x_{T\\lfloor t/T \\rfloor}\\opt\\|\\\\\n    &\\leq \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^{\\lfloor t/T \\rfloor}\\|x_0-x_0\\opt\\|+\\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}+T\\sigma.\n\\end{align*}\nThen, taking the limit supremum, we get\n\\begin{align*}\n    \\limsup_{t\\to\\infty}\\|x_t-x_t\\opt\\| &\\leq \\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}+T\\sigma\\\\\n    &= \\frac{2T\\sigma\\left(T-\\sqrt{\\kappa}\\right)}{T-2\\sqrt{\\kappa}}\\\\\n    &= \\frac{2c(c-1)}{c-2}\\sqrt{\\kappa}\\sigma.\n\\end{align*}\n\\end{proof}","units":[{"raw_tex":"\\begin{proof}\nFirst, via standard batch optimization bounds, we have\n\\begin{align*}\n    \\|x_{kT}-x_{(k-1)T}\\opt\\|\\leq \\frac{2\\sqrt{\\kappa}}{T}\\|x_{(k-1)T}-x_{(k-1)T}\\opt\\|.\n\\end{align*}\nThus,\n\\begin{align*}\n    \\|x_{kT}-x_{kT}\\opt\\|&\\leq\\frac{2\\sqrt{\\kappa}}{T}\\|x_{(k-1)T}-x_{(k-1)T}\\opt\\|+T\\sigma\\\\\n    &\\leq \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^k\\|x_0-x_0\\opt\\|+\\frac{T\\sigma}{1-\\frac{2\\sqrt{\\kappa}}{T}}\\\\\n    &= \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^k\\|x_0-x_0\\opt\\|+\\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}\n\\end{align*}\nand so\n\\begin{align*}\n    \\|x_t-x_t\\opt\\| &= \\|x_{T\\lfloor t/T \\rfloor}-x_t\\opt\\|\\\\\n    &\\leq \\|x_{T\\lfloor t/T \\rfloor}-x_{T\\lfloor t/T \\rfloor}\\opt\\|+\\|x_t\\opt-x_{T\\lfloor t/T \\rfloor}\\opt\\|\\\\\n    &\\leq \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^{\\lfloor t/T \\rfloor}\\|x_0-x_0\\opt\\|+\\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}+T\\sigma.\n\\end{align*}\nThen, taking the limit supremum, we get\n\\begin{align*}\n    \\limsup_{t\\to\\infty}\\|x_t-x_t\\opt\\| &\\leq \\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}+T\\sigma\\\\\n    &= \\frac{2T\\sigma\\left(T-\\sqrt{\\kappa}\\right)}{T-2\\sqrt{\\kappa}}\\\\\n    &= \\frac{2c(c-1)}{c-2}\\sqrt{\\kappa}\\sigma.\n\\end{align*}\n\\end{proof}","start":34800,"end":35946}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:time-varying-opt"],"declared_external":false,"source_only":false,"title":"时变希尔伯特空间优化","source_units":[{"raw_tex":"This section reviews useful results from convex analysis~\\cite{Beck,Combettes,Nesterov,Nesterov2,Boyd,Bubeck}, and introduces key definitions that will be used throughout the paper. The paper considers functions on a Hilbert space $\\Hi$ with corresponding norm $\\|\\cdot\\|$. We assume all functions, $f_t: \\Hi \\rightarrow \\R$,  are  proper, lower semicontinuous, convex, and strongly smooth; $t\\in\\N\\cup\\{0\\}$ denotes the time index~\\cite{SPMag,popkov2005gradient,SimonettoGlobalsip2014,mokhtari2016online}. The discretized time-varying optimization problem of interest is the sequence of convex problems (one for each $t$): \n\\begin{align}\n\\label{eq:time-varying-opt}\n   \\min_{x\\in\\Hi} f_t(x) \\,, \\hspace{.5cm} t \\in\\N\\cup\\{0\\}\n\\end{align}\n","start":6131,"end":6870}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"目标函数基本条件","source_units":[{"raw_tex":"This section reviews useful results from convex analysis~\\cite{Beck,Combettes,Nesterov,Nesterov2,Boyd,Bubeck}, and introduces key definitions that will be used throughout the paper. The paper considers functions on a Hilbert space $\\Hi$ with corresponding norm $\\|\\cdot\\|$. We assume all functions, $f_t: \\Hi \\rightarrow \\R$,  are  proper, lower semicontinuous, convex, and strongly smooth; $t\\in\\N\\cup\\{0\\}$ denotes the time index~\\cite{SPMag,popkov2005gradient,SimonettoGlobalsip2014,mokhtari2016online}. The discretized time-varying optimization problem of interest is the sequence of convex problems (one for each $t$): \n","start":6131,"end":6756}]},{"id":"S003","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"最优点与最优值","source_units":[{"raw_tex":"along with a time-varying minimizer set. In particular, assume that the minimizer set, denoted as $X_t\\opt$, is nonempty for each $t$. Let $x_t\\opt$ denote an element of $X_t\\opt$ and $f_t\\opt=f_t(x_t\\opt)$. % the optimal value function.\n","start":6870,"end":7108}]},{"id":"S004","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"强凸性下最优点唯一","source_units":[{"raw_tex":"We denote the set of sequences of $L$-strongly smooth functions by $\\SS'(L)$. For function sequences that are additionally $\\mu$-strongly convex, $X_t\\opt$ contains only one point; therefore, we can measure the temporal variability of the optimal solution trajectory of~\\eqref{eq:time-varying-opt} with the sequence \n","start":7108,"end":7425}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"仅光滑序列类","source_units":[{"raw_tex":"We denote the set of sequences of $L$-strongly smooth functions by $\\SS'(L)$. For function sequences that are additionally $\\mu$-strongly convex, $X_t\\opt$ contains only one point; therefore, we can measure the temporal variability of the optimal solution trajectory of~\\eqref{eq:time-varying-opt} with the sequence \n","start":7108,"end":7425}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"逐时最优点变化量","source_units":[{"raw_tex":"\\begin{align*}\n    \\sigma_t :=\\|x_{t+1}\\opt-x_t\\opt\\| , \\hspace{.5cm} t\\in \\N\\cup\\{0\\}.\n\\end{align*}\n","start":7425,"end":7526}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"统一强凸时变函数类","source_units":[{"raw_tex":"We define $\\SS(\\kappa^{-1},L,\\sigma)$, where $\\kappa$ is the condition number, as the set of $L$-strongly smooth, $\\kappa^{-1}L$-strongly convex functions sequences with $\\sigma_t \\leq \\sigma$ for all $t$. For both $\\SS'(L)$ and $\\SS(\\kappa^{-1},L,\\sigma)$, the Hilbert space, along with its dimension, is left implicit since the results are independent of it.\n","start":7526,"end":7887}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"跟踪误差指标","source_units":[{"raw_tex":"Due to the temporal variability, we cannot expect any of the error sequences to converge to zero in general. Thus, we will characterize the performance of online algorithms via bounds on the limit supremum of the errors, which we term ``tracking'' error, rather than bounds on the convergence rate (to the tracking error ball).\n","start":8915,"end":9243}]},{"id":"S009","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"长步法的设计说明","source_units":[{"raw_tex":"In this section, we present a long-step Nesterov's method; the term ``long-steps'' refers to the fact that the algorithm takes a certain number of steps using the same stale function before catching up to the most recent function and repeating. For this particular long-step Nesterov's method, we are able to prove upper bounds on the tracking error (on the other hand, no bounds for the online Nesterov's method are yet available, and are the subject of current efforts). \n\n","start":32969,"end":33444}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","labels":["algo:OLNM"],"declared_external":false,"source_only":false,"title":"在线长步 Nesterov 算法","source_units":[{"raw_tex":"The specific sequence constructed by $\\OLNM(T)$ is defined in Algorithm~\\ref{algo:OLNM}.\n\\begin{algorithm}[h]\n\\begin{algorithmic}[1]\n\\Require $x_0$\n\\State $y_0\\gets x_0$, $z_0\\gets x_0$, $a_0\\gets 1$\n\\For{$t=1,2,\\ldots$} %\\Comment{We have the answer if r is 0}\n          \\State $z_{t+1} =y_t-\\frac{1}{L}\\nabla f_{T\\lfloor t/T \\rfloor}(y_t)$\n \\If{     $T \\not| ~ t+1$ }\n \\State $a_{t+1} = \\frac{1+\\sqrt{1+4a_t^2}}{2}$\n \\State $y_{t+1} = z_{t+1} +\\frac{a_t-1}{a_{t+1}}(z_{t+1}-z_t)$\n \\State $x_{t+1} = x_t$ \\Comment{so $x_{t+1} = x_t = x_{T\\lfloor t/T\\rfloor}$}\n \\ElsIf{$T\\mid t+1$}\n \\State $a_{t+1} = 1$\n \\State $y_{t+1} = z_{t+1}$\n \\State $x_{t+1} = z_{t+1}$\n \\EndIf\n\\EndFor\n\\end{algorithmic}\n\\caption{$\\OLNM(T)$}\n\\label{algo:OLNM}\n\\end{algorithm}\n","start":33444,"end":34192}]},{"id":"S011","semantic_type":"algorithm","kind":"algorithm","labels":["algo:OLNM"],"declared_external":false,"source_only":false,"title":"初始化与循环起点","source_units":[{"raw_tex":"\\Require $x_0$\n\\State $y_0\\gets x_0$, $z_0\\gets x_0$, $a_0\\gets 1$\n\\For{$t=1,2,\\ldots$} %\\Comment{We have the answer if r is 0}\n","start":33577,"end":33705}]},{"id":"S012","semantic_type":"algorithm","kind":"algorithm","labels":["algo:OLNM"],"declared_external":false,"source_only":false,"title":"固定批首目标的梯度步","source_units":[{"raw_tex":"          \\State $z_{t+1} =y_t-\\frac{1}{L}\\nabla f_{T\\lfloor t/T \\rfloor}(y_t)$\n","start":33705,"end":33785}]},{"id":"S013","semantic_type":"algorithm","kind":"algorithm","labels":["algo:OLNM"],"declared_external":false,"source_only":false,"title":"批内加速状态更新","source_units":[{"raw_tex":" \\If{     $T \\not| ~ t+1$ }\n \\State $a_{t+1} = \\frac{1+\\sqrt{1+4a_t^2}}{2}$\n \\State $y_{t+1} = z_{t+1} +\\frac{a_t-1}{a_{t+1}}(z_{t+1}-z_t)$\n","start":33785,"end":33925}]},{"id":"S014","semantic_type":"algorithm","kind":"algorithm","labels":["algo:OLNM"],"declared_external":false,"source_only":false,"title":"批内保持输出不变","source_units":[{"raw_tex":" \\State $x_{t+1} = x_t$ \\Comment{so $x_{t+1} = x_t = x_{T\\lfloor t/T\\rfloor}$}\n","start":33925,"end":34004}]},{"id":"S015","semantic_type":"algorithm","kind":"algorithm","labels":["algo:OLNM"],"declared_external":false,"source_only":false,"title":"批末提交与重新启动","source_units":[{"raw_tex":" \\ElsIf{$T\\mid t+1$}\n \\State $a_{t+1} = 1$\n \\State $y_{t+1} = z_{t+1}$\n \\State $x_{t+1} = z_{t+1}$\n \\EndIf\n","start":34004,"end":34111}]},{"id":"S016","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"复合目标的扩展说明","source_units":[{"raw_tex":"\nThe method can be extended to the composite case by applying the proximal operator to the $z$ iterates. Theorem \\ref{thm:onlineNesterov} gives an upper bound on the tracking iterate error of $\\OLNM$, using results from \\cite[Thm. 10.34]{Beck}.\n","start":34192,"end":34437}]},{"id":"S017","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第1项：Beck, Theorem 10.34；原文行377引用 \\cite[Thm. 10.34]{Beck}，用于行386–388。"],"source_only":false,"title":"Beck 批量加速收敛界","source_units":[{"raw_tex":"The method can be extended to the composite case by applying the proximal operator to the $z$ iterates. Theorem \\ref{thm:onlineNesterov} gives an upper bound on the tracking iterate error of $\\OLNM$, using results from \\cite[Thm. 10.34]{Beck}.\n","start":34193,"end":34437}]},{"id":"S018","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第2项中的 Standard strong-convexity distance/error inequality。"],"source_only":false,"title":"强凸函数的距离误差界","source_units":[]},{"id":"S019","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第2项中的 triangle inequality。"],"source_only":false,"title":"范数三角不等式","source_units":[]},{"id":"S020","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第2项中的 geometric-series estimates。"],"source_only":false,"title":"收缩递推与几何级数","source_units":[]},{"id":"S021","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；用于原文行400–404的极限步骤。"],"source_only":false,"title":"衰减项的极限上确界","source_units":[]},{"id":"S022","semantic_type":"theorem","kind":"theorem","labels":["eq:OLNM","thm:onlineNesterov"],"declared_external":false,"source_only":false,"title":"OLNM 跟踪误差上界","source_units":[{"raw_tex":"\\begin{theorem} \\label{thm:onlineNesterov}\nIf $(f_t)\\in \\SS(\\kappa^{-1},L,\\sigma)$, then, given $x_0$, $\\OLNM(T)$ where $T=c\\sqrt{\\kappa}$ for $c>2$ such that $c\\sqrt{\\kappa}\\in\\N$, constructs a sequence $(x_t)$ such that\n\\begin{align} \\label{eq:OLNM}\n    \\limsup_{t\\to\\infty}\\|x_t-x_t\\opt\\|\\leq \\frac{2c(c-1)}{c-2}\\sqrt{\\kappa}\\sigma.\n\\end{align}\n\\end{theorem}","start":34438,"end":34799}]},{"id":"S023","semantic_type":"assumption","kind":"assumption","labels":["thm:onlineNesterov"],"declared_external":false,"source_only":false,"title":"批长与比例参数限制","source_units":[{"raw_tex":"If $(f_t)\\in \\SS(\\kappa^{-1},L,\\sigma)$, then, given $x_0$, $\\OLNM(T)$ where $T=c\\sqrt{\\kappa}$ for $c>2$ such that $c\\sqrt{\\kappa}\\in\\N$, constructs a sequence $(x_t)$ such that\n","start":34481,"end":34660}]},{"id":"S024","semantic_type":"assumption","kind":"assumption","labels":["thm:onlineNesterov"],"declared_external":false,"source_only":false,"title":"选定序列属于强凸类","source_units":[{"raw_tex":"If $(f_t)\\in \\SS(\\kappa^{-1},L,\\sigma)$, then, given $x_0$, $\\OLNM(T)$ where $T=c\\sqrt{\\kappa}$ for $c>2$ such that $c\\sqrt{\\kappa}\\in\\N$, constructs a sequence $(x_t)$ such that\n","start":34481,"end":34660}]},{"id":"S025","semantic_type":"derived_fact","kind":"derived_fact","labels":["thm:onlineNesterov"],"declared_external":false,"source_only":false,"title":"收缩系数与分母为正","source_units":[{"raw_tex":"If $(f_t)\\in \\SS(\\kappa^{-1},L,\\sigma)$, then, given $x_0$, $\\OLNM(T)$ where $T=c\\sqrt{\\kappa}$ for $c>2$ such that $c\\sqrt{\\kappa}\\in\\N$, constructs a sequence $(x_t)$ such that\n","start":34481,"end":34660},{"raw_tex":"    &\\leq \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^k\\|x_0-x_0\\opt\\|+\\frac{T\\sigma}{1-\\frac{2\\sqrt{\\kappa}}{T}}\\\\\n    &= \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^k\\|x_0-x_0\\opt\\|+\\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}\n","start":35101,"end":35308}]},{"id":"S026","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"跟踪上界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nFirst, via standard batch optimization bounds, we have\n\\begin{align*}\n    \\|x_{kT}-x_{(k-1)T}\\opt\\|\\leq \\frac{2\\sqrt{\\kappa}}{T}\\|x_{(k-1)T}-x_{(k-1)T}\\opt\\|.\n\\end{align*}\nThus,\n\\begin{align*}\n    \\|x_{kT}-x_{kT}\\opt\\|&\\leq\\frac{2\\sqrt{\\kappa}}{T}\\|x_{(k-1)T}-x_{(k-1)T}\\opt\\|+T\\sigma\\\\\n    &\\leq \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^k\\|x_0-x_0\\opt\\|+\\frac{T\\sigma}{1-\\frac{2\\sqrt{\\kappa}}{T}}\\\\\n    &= \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^k\\|x_0-x_0\\opt\\|+\\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}\n\\end{align*}\nand so\n\\begin{align*}\n    \\|x_t-x_t\\opt\\| &= \\|x_{T\\lfloor t/T \\rfloor}-x_t\\opt\\|\\\\\n    &\\leq \\|x_{T\\lfloor t/T \\rfloor}-x_{T\\lfloor t/T \\rfloor}\\opt\\|+\\|x_t\\opt-x_{T\\lfloor t/T \\rfloor}\\opt\\|\\\\\n    &\\leq \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^{\\lfloor t/T \\rfloor}\\|x_0-x_0\\opt\\|+\\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}+T\\sigma.\n\\end{align*}\nThen, taking the limit supremum, we get\n\\begin{align*}\n    \\limsup_{t\\to\\infty}\\|x_t-x_t\\opt\\| &\\leq \\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}+T\\sigma\\\\\n    &= \\frac{2T\\sigma\\left(T-\\sqrt{\\kappa}\\right)}{T-2\\sqrt{\\kappa}}\\\\\n    &= \\frac{2c(c-1)}{c-2}\\sqrt{\\kappa}\\sigma.\n\\end{align*}\n\\end{proof}","start":34800,"end":35946}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"固定旧目标的批末收缩","source_units":[{"raw_tex":"First, via standard batch optimization bounds, we have\n\\begin{align*}\n    \\|x_{kT}-x_{(k-1)T}\\opt\\|\\leq \\frac{2\\sqrt{\\kappa}}{T}\\|x_{(k-1)T}-x_{(k-1)T}\\opt\\|.\n\\end{align*}\n","start":34814,"end":34986}]},{"id":"S028","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"整批最优轨迹移动界","source_units":[{"raw_tex":"    \\|x_{kT}-x_{kT}\\opt\\|&\\leq\\frac{2\\sqrt{\\kappa}}{T}\\|x_{(k-1)T}-x_{(k-1)T}\\opt\\|+T\\sigma\\\\\n","start":35007,"end":35101}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"建立批末跟踪递推","source_units":[{"raw_tex":"Thus,\n\\begin{align*}\n    \\|x_{kT}-x_{kT}\\opt\\|&\\leq\\frac{2\\sqrt{\\kappa}}{T}\\|x_{(k-1)T}-x_{(k-1)T}\\opt\\|+T\\sigma\\\\\n","start":34986,"end":35101}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"展开递推并估计级数","source_units":[{"raw_tex":"    &\\leq \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^k\\|x_0-x_0\\opt\\|+\\frac{T\\sigma}{1-\\frac{2\\sqrt{\\kappa}}{T}}\\\\\n","start":35101,"end":35211}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"整理批末稳态项","source_units":[{"raw_tex":"    &= \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^k\\|x_0-x_0\\opt\\|+\\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}\n\\end{align*}\n","start":35211,"end":35321}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"任意时刻使用批首输出","source_units":[{"raw_tex":"and so\n\\begin{align*}\n    \\|x_t-x_t\\opt\\| &= \\|x_{T\\lfloor t/T \\rfloor}-x_t\\opt\\|\\\\\n","start":35321,"end":35405}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"拆分任意时刻误差","source_units":[{"raw_tex":"    &\\leq \\|x_{T\\lfloor t/T \\rfloor}-x_{T\\lfloor t/T \\rfloor}\\opt\\|+\\|x_t\\opt-x_{T\\lfloor t/T \\rfloor}\\opt\\|\\\\\n","start":35405,"end":35516}]},{"id":"S034","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"批内最优轨迹移动界","source_units":[{"raw_tex":"    &\\leq \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^{\\lfloor t/T \\rfloor}\\|x_0-x_0\\opt\\|+\\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}+T\\sigma.\n","start":35516,"end":35645}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"获得全时刻跟踪界","source_units":[{"raw_tex":"    &\\leq \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^{\\lfloor t/T \\rfloor}\\|x_0-x_0\\opt\\|+\\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}+T\\sigma.\n\\end{align*}\n","start":35516,"end":35658}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"取极限上确界","source_units":[{"raw_tex":"Then, taking the limit supremum, we get\n\\begin{align*}\n    \\limsup_{t\\to\\infty}\\|x_t-x_t\\opt\\| &\\leq \\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}+T\\sigma\\\\\n","start":35658,"end":35804}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"合并稳态误差项","source_units":[{"raw_tex":"    &= \\frac{2T\\sigma\\left(T-\\sqrt{\\kappa}\\right)}{T-2\\sqrt{\\kappa}}\\\\\n","start":35804,"end":35875}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"代入批长得最终常数","source_units":[{"raw_tex":"    &= \\frac{2c(c-1)}{c-2}\\sqrt{\\kappa}\\sigma.\n\\end{align*}\n","start":35875,"end":35935}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S022","edges":[{"source_id":"S001","target_id":"S006","kind":"supports"},{"source_id":"S003","target_id":"S006","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S011","target_id":"S010","kind":"supports"},{"source_id":"S012","target_id":"S010","kind":"supports"},{"source_id":"S013","target_id":"S010","kind":"supports"},{"source_id":"S014","target_id":"S010","kind":"supports"},{"source_id":"S015","target_id":"S010","kind":"supports"},{"source_id":"S007","target_id":"S024","kind":"supports"},{"source_id":"S001","target_id":"S022","kind":"supports"},{"source_id":"S003","target_id":"S022","kind":"supports"},{"source_id":"S007","target_id":"S022","kind":"supports"},{"source_id":"S008","target_id":"S022","kind":"supports"},{"source_id":"S010","target_id":"S022","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S017","target_id":"S027","kind":"supports"},{"source_id":"S018","target_id":"S027","kind":"supports"},{"source_id":"S024","target_id":"S027","kind":"supports"},{"source_id":"S011","target_id":"S027","kind":"supports"},{"source_id":"S012","target_id":"S027","kind":"supports"},{"source_id":"S013","target_id":"S027","kind":"supports"},{"source_id":"S015","target_id":"S027","kind":"supports"},{"source_id":"S024","target_id":"S028","kind":"supports"},{"source_id":"S006","target_id":"S028","kind":"supports"},{"source_id":"S019","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S019","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S025","target_id":"S030","kind":"supports"},{"source_id":"S020","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S025","target_id":"S031","kind":"supports"},{"source_id":"S011","target_id":"S032","kind":"supports"},{"source_id":"S014","target_id":"S032","kind":"supports"},{"source_id":"S015","target_id":"S032","kind":"supports"},{"source_id":"S023","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S019","target_id":"S033","kind":"supports"},{"source_id":"S024","target_id":"S034","kind":"supports"},{"source_id":"S006","target_id":"S034","kind":"supports"},{"source_id":"S023","target_id":"S034","kind":"supports"},{"source_id":"S019","target_id":"S034","kind":"supports"},{"source_id":"S031","target_id":"S035","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S025","target_id":"S036","kind":"supports"},{"source_id":"S023","target_id":"S036","kind":"supports"},{"source_id":"S021","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S025","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S023","target_id":"S038","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S029","target_id":"S026","kind":"supports"},{"source_id":"S030","target_id":"S026","kind":"supports"},{"source_id":"S031","target_id":"S026","kind":"supports"},{"source_id":"S032","target_id":"S026","kind":"supports"},{"source_id":"S033","target_id":"S026","kind":"supports"},{"source_id":"S035","target_id":"S026","kind":"supports"},{"source_id":"S036","target_id":"S026","kind":"supports"},{"source_id":"S037","target_id":"S026","kind":"supports"},{"source_id":"S038","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S022","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S022","edges":[{"source_id":"S001","target_id":"S006","kind":"supports"},{"source_id":"S003","target_id":"S006","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S011","target_id":"S010","kind":"supports"},{"source_id":"S012","target_id":"S010","kind":"supports"},{"source_id":"S013","target_id":"S010","kind":"supports"},{"source_id":"S014","target_id":"S010","kind":"supports"},{"source_id":"S015","target_id":"S010","kind":"supports"},{"source_id":"S007","target_id":"S024","kind":"supports"},{"source_id":"S001","target_id":"S022","kind":"supports"},{"source_id":"S003","target_id":"S022","kind":"supports"},{"source_id":"S007","target_id":"S022","kind":"supports"},{"source_id":"S008","target_id":"S022","kind":"supports"},{"source_id":"S010","target_id":"S022","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S017","target_id":"S027","kind":"supports"},{"source_id":"S018","target_id":"S027","kind":"supports"},{"source_id":"S024","target_id":"S027","kind":"supports"},{"source_id":"S011","target_id":"S027","kind":"supports"},{"source_id":"S012","target_id":"S027","kind":"supports"},{"source_id":"S013","target_id":"S027","kind":"supports"},{"source_id":"S015","target_id":"S027","kind":"supports"},{"source_id":"S024","target_id":"S028","kind":"supports"},{"source_id":"S006","target_id":"S028","kind":"supports"},{"source_id":"S019","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S019","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S025","target_id":"S030","kind":"supports"},{"source_id":"S020","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S025","target_id":"S031","kind":"supports"},{"source_id":"S011","target_id":"S032","kind":"supports"},{"source_id":"S014","target_id":"S032","kind":"supports"},{"source_id":"S015","target_id":"S032","kind":"supports"},{"source_id":"S023","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S019","target_id":"S033","kind":"supports"},{"source_id":"S024","target_id":"S034","kind":"supports"},{"source_id":"S006","target_id":"S034","kind":"supports"},{"source_id":"S023","target_id":"S034","kind":"supports"},{"source_id":"S019","target_id":"S034","kind":"supports"},{"source_id":"S031","target_id":"S035","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S025","target_id":"S036","kind":"supports"},{"source_id":"S023","target_id":"S036","kind":"supports"},{"source_id":"S021","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S025","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S023","target_id":"S038","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S029","target_id":"S026","kind":"supports"},{"source_id":"S030","target_id":"S026","kind":"supports"},{"source_id":"S031","target_id":"S026","kind":"supports"},{"source_id":"S032","target_id":"S026","kind":"supports"},{"source_id":"S033","target_id":"S026","kind":"supports"},{"source_id":"S035","target_id":"S026","kind":"supports"},{"source_id":"S036","target_id":"S026","kind":"supports"},{"source_id":"S037","target_id":"S026","kind":"supports"},{"source_id":"S038","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S022","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:time-varying-opt"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"时变希尔伯特空间优化","source_units":[{"raw_tex":"This section reviews useful results from convex analysis~\\cite{Beck,Combettes,Nesterov,Nesterov2,Boyd,Bubeck}, and introduces key definitions that will be used throughout the paper. The paper considers functions on a Hilbert space $\\Hi$ with corresponding norm $\\|\\cdot\\|$. We assume all functions, $f_t: \\Hi \\rightarrow \\R$,  are  proper, lower semicontinuous, convex, and strongly smooth; $t\\in\\N\\cup\\{0\\}$ denotes the time index~\\cite{SPMag,popkov2005gradient,SimonettoGlobalsip2014,mokhtari2016online}. The discretized time-varying optimization problem of interest is the sequence of convex problems (one for each $t$): \n\\begin{align}\n\\label{eq:time-varying-opt}\n   \\min_{x\\in\\Hi} f_t(x) \\,, \\hspace{.5cm} t \\in\\N\\cup\\{0\\}\n\\end{align}\n","start":6131,"end":6870}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"目标函数基本条件","source_units":[{"raw_tex":"This section reviews useful results from convex analysis~\\cite{Beck,Combettes,Nesterov,Nesterov2,Boyd,Bubeck}, and introduces key definitions that will be used throughout the paper. The paper considers functions on a Hilbert space $\\Hi$ with corresponding norm $\\|\\cdot\\|$. We assume all functions, $f_t: \\Hi \\rightarrow \\R$,  are  proper, lower semicontinuous, convex, and strongly smooth; $t\\in\\N\\cup\\{0\\}$ denotes the time index~\\cite{SPMag,popkov2005gradient,SimonettoGlobalsip2014,mokhtari2016online}. The discretized time-varying optimization problem of interest is the sequence of convex problems (one for each $t$): \n","start":6131,"end":6756}]},{"id":"S003","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"最优点与最优值","source_units":[{"raw_tex":"along with a time-varying minimizer set. In particular, assume that the minimizer set, denoted as $X_t\\opt$, is nonempty for each $t$. Let $x_t\\opt$ denote an element of $X_t\\opt$ and $f_t\\opt=f_t(x_t\\opt)$. % the optimal value function.\n","start":6870,"end":7108}]},{"id":"S004","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"强凸性下最优点唯一","source_units":[{"raw_tex":"We denote the set of sequences of $L$-strongly smooth functions by $\\SS'(L)$. For function sequences that are additionally $\\mu$-strongly convex, $X_t\\opt$ contains only one point; therefore, we can measure the temporal variability of the optimal solution trajectory of~\\eqref{eq:time-varying-opt} with the sequence \n","start":7108,"end":7425}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"仅光滑序列类","source_units":[{"raw_tex":"We denote the set of sequences of $L$-strongly smooth functions by $\\SS'(L)$. For function sequences that are additionally $\\mu$-strongly convex, $X_t\\opt$ contains only one point; therefore, we can measure the temporal variability of the optimal solution trajectory of~\\eqref{eq:time-varying-opt} with the sequence \n","start":7108,"end":7425}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"逐时最优点变化量","source_units":[{"raw_tex":"\\begin{align*}\n    \\sigma_t :=\\|x_{t+1}\\opt-x_t\\opt\\| , \\hspace{.5cm} t\\in \\N\\cup\\{0\\}.\n\\end{align*}\n","start":7425,"end":7526}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"统一强凸时变函数类","source_units":[{"raw_tex":"We define $\\SS(\\kappa^{-1},L,\\sigma)$, where $\\kappa$ is the condition number, as the set of $L$-strongly smooth, $\\kappa^{-1}L$-strongly convex functions sequences with $\\sigma_t \\leq \\sigma$ for all $t$. For both $\\SS'(L)$ and $\\SS(\\kappa^{-1},L,\\sigma)$, the Hilbert space, along with its dimension, is left implicit since the results are independent of it.\n","start":7526,"end":7887}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"跟踪误差指标","source_units":[{"raw_tex":"Due to the temporal variability, we cannot expect any of the error sequences to converge to zero in general. Thus, we will characterize the performance of online algorithms via bounds on the limit supremum of the errors, which we term ``tracking'' error, rather than bounds on the convergence rate (to the tracking error ball).\n","start":8915,"end":9243}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","labels":["algo:OLNM"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"在线长步 Nesterov 算法","source_units":[{"raw_tex":"The specific sequence constructed by $\\OLNM(T)$ is defined in Algorithm~\\ref{algo:OLNM}.\n\\begin{algorithm}[h]\n\\begin{algorithmic}[1]\n\\Require $x_0$\n\\State $y_0\\gets x_0$, $z_0\\gets x_0$, $a_0\\gets 1$\n\\For{$t=1,2,\\ldots$} %\\Comment{We have the answer if r is 0}\n          \\State $z_{t+1} =y_t-\\frac{1}{L}\\nabla f_{T\\lfloor t/T \\rfloor}(y_t)$\n \\If{     $T \\not| ~ t+1$ }\n \\State $a_{t+1} = \\frac{1+\\sqrt{1+4a_t^2}}{2}$\n \\State $y_{t+1} = z_{t+1} +\\frac{a_t-1}{a_{t+1}}(z_{t+1}-z_t)$\n \\State $x_{t+1} = x_t$ \\Comment{so $x_{t+1} = x_t = x_{T\\lfloor t/T\\rfloor}$}\n \\ElsIf{$T\\mid t+1$}\n \\State $a_{t+1} = 1$\n \\State $y_{t+1} = z_{t+1}$\n \\State $x_{t+1} = z_{t+1}$\n \\EndIf\n\\EndFor\n\\end{algorithmic}\n\\caption{$\\OLNM(T)$}\n\\label{algo:OLNM}\n\\end{algorithm}\n","start":33444,"end":34192}]},{"id":"S011","semantic_type":"algorithm","kind":"algorithm","labels":["algo:OLNM"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"初始化与循环起点","source_units":[{"raw_tex":"\\Require $x_0$\n\\State $y_0\\gets x_0$, $z_0\\gets x_0$, $a_0\\gets 1$\n\\For{$t=1,2,\\ldots$} %\\Comment{We have the answer if r is 0}\n","start":33577,"end":33705}]},{"id":"S012","semantic_type":"algorithm","kind":"algorithm","labels":["algo:OLNM"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"固定批首目标的梯度步","source_units":[{"raw_tex":"          \\State $z_{t+1} =y_t-\\frac{1}{L}\\nabla f_{T\\lfloor t/T \\rfloor}(y_t)$\n","start":33705,"end":33785}]},{"id":"S013","semantic_type":"algorithm","kind":"algorithm","labels":["algo:OLNM"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"批内加速状态更新","source_units":[{"raw_tex":" \\If{     $T \\not| ~ t+1$ }\n \\State $a_{t+1} = \\frac{1+\\sqrt{1+4a_t^2}}{2}$\n \\State $y_{t+1} = z_{t+1} +\\frac{a_t-1}{a_{t+1}}(z_{t+1}-z_t)$\n","start":33785,"end":33925}]},{"id":"S014","semantic_type":"algorithm","kind":"algorithm","labels":["algo:OLNM"],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"批内保持输出不变","source_units":[{"raw_tex":" \\State $x_{t+1} = x_t$ \\Comment{so $x_{t+1} = x_t = x_{T\\lfloor t/T\\rfloor}$}\n","start":33925,"end":34004}]},{"id":"S015","semantic_type":"algorithm","kind":"algorithm","labels":["algo:OLNM"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"批末提交与重新启动","source_units":[{"raw_tex":" \\ElsIf{$T\\mid t+1$}\n \\State $a_{t+1} = 1$\n \\State $y_{t+1} = z_{t+1}$\n \\State $x_{t+1} = z_{t+1}$\n \\EndIf\n","start":34004,"end":34111}]},{"id":"S017","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S017"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第1项：Beck, Theorem 10.34；原文行377引用 \\cite[Thm. 10.34]{Beck}，用于行386–388。"],"source_only":false,"title":"Beck 批量加速收敛界","source_units":[{"raw_tex":"The method can be extended to the composite case by applying the proximal operator to the $z$ iterates. Theorem \\ref{thm:onlineNesterov} gives an upper bound on the tracking iterate error of $\\OLNM$, using results from \\cite[Thm. 10.34]{Beck}.\n","start":34193,"end":34437}]},{"id":"S018","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S018"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第2项中的 Standard strong-convexity distance/error inequality。"],"source_only":false,"title":"强凸函数的距离误差界","source_units":[]},{"id":"S019","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S019"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第2项中的 triangle inequality。"],"source_only":false,"title":"范数三角不等式","source_units":[]},{"id":"S020","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S020"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第2项中的 geometric-series estimates。"],"source_only":false,"title":"收缩递推与几何级数","source_units":[]},{"id":"S021","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S021"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；用于原文行400–404的极限步骤。"],"source_only":false,"title":"衰减项的极限上确界","source_units":[]},{"id":"S022","semantic_type":"theorem","kind":"theorem","labels":["eq:OLNM","thm:onlineNesterov"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"OLNM 跟踪误差上界","source_units":[{"raw_tex":"\\begin{theorem} \\label{thm:onlineNesterov}\nIf $(f_t)\\in \\SS(\\kappa^{-1},L,\\sigma)$, then, given $x_0$, $\\OLNM(T)$ where $T=c\\sqrt{\\kappa}$ for $c>2$ such that $c\\sqrt{\\kappa}\\in\\N$, constructs a sequence $(x_t)$ such that\n\\begin{align} \\label{eq:OLNM}\n    \\limsup_{t\\to\\infty}\\|x_t-x_t\\opt\\|\\leq \\frac{2c(c-1)}{c-2}\\sqrt{\\kappa}\\sigma.\n\\end{align}\n\\end{theorem}","start":34438,"end":34799}]},{"id":"S023","semantic_type":"assumption","kind":"assumption","labels":["thm:onlineNesterov"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"批长与比例参数限制","source_units":[{"raw_tex":"If $(f_t)\\in \\SS(\\kappa^{-1},L,\\sigma)$, then, given $x_0$, $\\OLNM(T)$ where $T=c\\sqrt{\\kappa}$ for $c>2$ such that $c\\sqrt{\\kappa}\\in\\N$, constructs a sequence $(x_t)$ such that\n","start":34481,"end":34660}]},{"id":"S024","semantic_type":"assumption","kind":"assumption","labels":["thm:onlineNesterov"],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"选定序列属于强凸类","source_units":[{"raw_tex":"If $(f_t)\\in \\SS(\\kappa^{-1},L,\\sigma)$, then, given $x_0$, $\\OLNM(T)$ where $T=c\\sqrt{\\kappa}$ for $c>2$ such that $c\\sqrt{\\kappa}\\in\\N$, constructs a sequence $(x_t)$ such that\n","start":34481,"end":34660}]},{"id":"S025","semantic_type":"derived_fact","kind":"derived_fact","labels":["thm:onlineNesterov"],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"收缩系数与分母为正","source_units":[{"raw_tex":"If $(f_t)\\in \\SS(\\kappa^{-1},L,\\sigma)$, then, given $x_0$, $\\OLNM(T)$ where $T=c\\sqrt{\\kappa}$ for $c>2$ such that $c\\sqrt{\\kappa}\\in\\N$, constructs a sequence $(x_t)$ such that\n","start":34481,"end":34660},{"raw_tex":"    &\\leq \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^k\\|x_0-x_0\\opt\\|+\\frac{T\\sigma}{1-\\frac{2\\sqrt{\\kappa}}{T}}\\\\\n    &= \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^k\\|x_0-x_0\\opt\\|+\\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}\n","start":35101,"end":35308}]},{"id":"S026","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"跟踪上界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nFirst, via standard batch optimization bounds, we have\n\\begin{align*}\n    \\|x_{kT}-x_{(k-1)T}\\opt\\|\\leq \\frac{2\\sqrt{\\kappa}}{T}\\|x_{(k-1)T}-x_{(k-1)T}\\opt\\|.\n\\end{align*}\nThus,\n\\begin{align*}\n    \\|x_{kT}-x_{kT}\\opt\\|&\\leq\\frac{2\\sqrt{\\kappa}}{T}\\|x_{(k-1)T}-x_{(k-1)T}\\opt\\|+T\\sigma\\\\\n    &\\leq \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^k\\|x_0-x_0\\opt\\|+\\frac{T\\sigma}{1-\\frac{2\\sqrt{\\kappa}}{T}}\\\\\n    &= \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^k\\|x_0-x_0\\opt\\|+\\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}\n\\end{align*}\nand so\n\\begin{align*}\n    \\|x_t-x_t\\opt\\| &= \\|x_{T\\lfloor t/T \\rfloor}-x_t\\opt\\|\\\\\n    &\\leq \\|x_{T\\lfloor t/T \\rfloor}-x_{T\\lfloor t/T \\rfloor}\\opt\\|+\\|x_t\\opt-x_{T\\lfloor t/T \\rfloor}\\opt\\|\\\\\n    &\\leq \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^{\\lfloor t/T \\rfloor}\\|x_0-x_0\\opt\\|+\\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}+T\\sigma.\n\\end{align*}\nThen, taking the limit supremum, we get\n\\begin{align*}\n    \\limsup_{t\\to\\infty}\\|x_t-x_t\\opt\\| &\\leq \\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}+T\\sigma\\\\\n    &= \\frac{2T\\sigma\\left(T-\\sqrt{\\kappa}\\right)}{T-2\\sqrt{\\kappa}}\\\\\n    &= \\frac{2c(c-1)}{c-2}\\sqrt{\\kappa}\\sigma.\n\\end{align*}\n\\end{proof}","start":34800,"end":35946}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"固定旧目标的批末收缩","source_units":[{"raw_tex":"First, via standard batch optimization bounds, we have\n\\begin{align*}\n    \\|x_{kT}-x_{(k-1)T}\\opt\\|\\leq \\frac{2\\sqrt{\\kappa}}{T}\\|x_{(k-1)T}-x_{(k-1)T}\\opt\\|.\n\\end{align*}\n","start":34814,"end":34986}]},{"id":"S028","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"整批最优轨迹移动界","source_units":[{"raw_tex":"    \\|x_{kT}-x_{kT}\\opt\\|&\\leq\\frac{2\\sqrt{\\kappa}}{T}\\|x_{(k-1)T}-x_{(k-1)T}\\opt\\|+T\\sigma\\\\\n","start":35007,"end":35101}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"建立批末跟踪递推","source_units":[{"raw_tex":"Thus,\n\\begin{align*}\n    \\|x_{kT}-x_{kT}\\opt\\|&\\leq\\frac{2\\sqrt{\\kappa}}{T}\\|x_{(k-1)T}-x_{(k-1)T}\\opt\\|+T\\sigma\\\\\n","start":34986,"end":35101}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"展开递推并估计级数","source_units":[{"raw_tex":"    &\\leq \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^k\\|x_0-x_0\\opt\\|+\\frac{T\\sigma}{1-\\frac{2\\sqrt{\\kappa}}{T}}\\\\\n","start":35101,"end":35211}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"整理批末稳态项","source_units":[{"raw_tex":"    &= \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^k\\|x_0-x_0\\opt\\|+\\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}\n\\end{align*}\n","start":35211,"end":35321}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"任意时刻使用批首输出","source_units":[{"raw_tex":"and so\n\\begin{align*}\n    \\|x_t-x_t\\opt\\| &= \\|x_{T\\lfloor t/T \\rfloor}-x_t\\opt\\|\\\\\n","start":35321,"end":35405}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"拆分任意时刻误差","source_units":[{"raw_tex":"    &\\leq \\|x_{T\\lfloor t/T \\rfloor}-x_{T\\lfloor t/T \\rfloor}\\opt\\|+\\|x_t\\opt-x_{T\\lfloor t/T \\rfloor}\\opt\\|\\\\\n","start":35405,"end":35516}]},{"id":"S034","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"批内最优轨迹移动界","source_units":[{"raw_tex":"    &\\leq \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^{\\lfloor t/T \\rfloor}\\|x_0-x_0\\opt\\|+\\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}+T\\sigma.\n","start":35516,"end":35645}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"获得全时刻跟踪界","source_units":[{"raw_tex":"    &\\leq \\left(\\frac{2\\sqrt{\\kappa}}{T}\\right)^{\\lfloor t/T \\rfloor}\\|x_0-x_0\\opt\\|+\\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}+T\\sigma.\n\\end{align*}\n","start":35516,"end":35658}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"取极限上确界","source_units":[{"raw_tex":"Then, taking the limit supremum, we get\n\\begin{align*}\n    \\limsup_{t\\to\\infty}\\|x_t-x_t\\opt\\| &\\leq \\frac{T^2\\sigma}{T-2\\sqrt{\\kappa}}+T\\sigma\\\\\n","start":35658,"end":35804}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"合并稳态误差项","source_units":[{"raw_tex":"    &= \\frac{2T\\sigma\\left(T-\\sqrt{\\kappa}\\right)}{T-2\\sqrt{\\kappa}}\\\\\n","start":35804,"end":35875}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"代入批长得最终常数","source_units":[{"raw_tex":"    &= \\frac{2c(c-1)}{c-2}\\sqrt{\\kappa}\\sigma.\n\\end{align*}\n","start":35875,"end":35935}]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038"}}}}
{"record_id":"2203.00252v4","paper":{"title":"Bregman three-operator splitting methods","authors":["Jiang, Xin","Vandenberghe, Lieven"],"arxiv_id":"2203.00252","version":"v4","journal":"Journal of Optimization Theory and Applications","abstract_url":"https://arxiv.org/abs/2203.00252v4","pdf_url":"https://arxiv.org/pdf/2203.00252v4","doi":"10.1007/s10957-022-02125-9"},"context":{"raw_tex":"We discuss proximal splitting methods for optimization problems \nin the form\n\\begin{equation} \\label{e-prob}\n  \\mbox{minimize} \\quad f(x) + g(Ax) + h(x),\n\\end{equation}\nwhere $f$, $g$, and $h$ are convex functions, and $h$ is differentiable.\n\n\nWe use the notation $\\inprod{x}{y} = x^Ty$ for the\nstandard inner product of vectors $x$ and $y$, and\n$\\|x\\| = \\inprod{x}{x}^{1/2}$ for the Euclidean norm of a vector\n$x$.  Other norms will be distinguished by a subscript.\n\n\\subsection{Problem formulation}\n\nIn~\\eqref{e-prob} the vector $x$ is an $n$-vector and\n$A$ is an $m\\times n$ matrix.  The functions $f$, $g$, $h$\nare closed and convex, and $h$ is differentiable, \\ie,  \n\\[\nh(x) \\geq h(x') + \\inprod{\\nabla h(x')}{x-x'} \\quad\n\\mbox{for all $x,x' \\in \\dom h$},\n\\]\nwhere $\\dom h$ is an open convex set.  We assume that $f+h$ and $g$\nare proper, \\ie, have nonempty domains.\n\n\nThe dual of problem~\\eqref{e-prob} is\n\\BEQ \\label{e-prob-dual}\n \\mbox{maximize}\\quad  {-(f+h)^*(-A^Tz) - g^*(z)},\n\\EEQ\nwhere $(f+h)^*$ and $g^*$ are the conjugates of $f+h$ and $g$:\n\\[\n (f+h)^*(w) = \\sup_x{(\\inprod{w}{x} - f(x) - h(x))}, \\qquad\n g^*(z) = \\sup_y{(\\inprod{z}{y} - g(y))}.\n\\]\n\n\nThe primal--dual optimality conditions for~\\eqref{e-prob}\nand~\\eqref{e-prob-dual} are\n\\[\n 0 \\in \\partial f(x) + \\nabla h(x) + A^T z, \\qquad\n 0 \\in \\partial g^*(z) - Ax.\n\\]\nHere $\\partial f$ and $\\partial g^*$ are the subdifferentials of\n$f$ and  $g^*$.  We often write the optimality conditions as\n\\begin{equation} \\label{e-opt-cond}\n  0 \\in \\begin{bmatrix} 0 & A^T \\\\ -A & 0 \\end{bmatrix}\n  \\begin{bmatrix} x \\\\ z \\end{bmatrix} + \\begin{bmatrix}\n    \\partial f(x)+\\nabla h(x) \\\\ \\partial g^\\ast(z)\n\\end{bmatrix}.\n\\end{equation}\nThroughout the paper, we assume that the optimality \nconditions~\\eqref{e-opt-cond} are solvable.\n\nWe will refer to the convex--concave function\n\\[\n\\cL(x,z) = f(x)+h(x)+ \\inprod{z}{Ax} -g^\\ast(z)\n\\]\nas the \\emph{Lagrangian} of~\\eqref{e-prob}.\nWe follow the convention that $\\cL(x,z) = +\\infty$ if $x\\not\\in \\dom(f+h)$\nand $\\cL(x,z) = -\\infty$ if $x\\in\\dom(f+h)$ and $z\\not\\in \\dom g^*$.\n\n\nLet $\\phi$ be a convex function with a domain that has nonempty\ninterior, and assume $\\phi$ is continuous on $\\dom \\phi$ and \ncontinuously differentiable on $\\intr(\\dom \\phi)$.\nThe \\emph{generalized distance} (or \\emph{Bregman distance})\ngenerated by the \\emph{kernel function} $\\phi$ is defined as the function\n\\[\nd(x,y) = \\phi(x)-\\phi(y)-\\inprod{\\nabla \\phi(y)}{x-y},\n\\]\nwith domain $\\dom d=\\dom \\phi \\times \\intr(\\dom \\phi)$.\nThe corresponding \\emph{Bregman proximal operator} of a function $f$ is \n\\begin{align}\n\\prox_f^\\phi(y,a)\n  &= \\argmin_x{(f(x)+\\inprod{a}{x}+d(x,y))} \\label{e-prox-opt} \\\\\n&= \\argmin_x{(f(x)+\\inprod{a}{x}+\\phi(x)\n   -\\inprod{\\nabla \\phi(y)}{x})}. \\label{e-prox-opt2}\n\\end{align}\nIt is assumed that for every $a$ and every $y \\in \\intr(\\dom\\phi)$\nthe minimizer $\\hat x=\\prox_f^\\phi(y,a)$ is unique and in\n$\\intr(\\dom \\phi)$.\n\n\nFrom the expression~\\eqref{e-prox-opt2} we see that \n$\\hat x=\\prox_f^\\phi(y,a)$\nsatisfies\n\\[\n\\nabla \\phi(y) - \\nabla \\phi(\\hat x) - a \\in \\partial f(\\hat x).\n\\]\nEquivalently, by definition of subgradient,\n\\begin{align}\nf(x) + \\inprod{a}{x}\n&\\geq f(\\hat x)+\\inprod{a}{\\hat x}\n  +\\inprod{\\nabla\\phi(y)-\\nabla\\phi(\\hat x)}{x-\\hat x} \\nonumber \\\\\n&= f(\\hat x)+\\inprod{a}{\\hat x}+d(\\hat x, y)+d(x,\\hat x)-d(x,y)\n  \\label{e-prox-cond}\n\\end{align}\nfor all $x\\in \\dom f \\cap \\dom \\phi$.\n\n\n\\paragraph{Assumptions} \nThroughout Section~\\ref{s-bcv} we make the following assumptions.\nThe kernel functions $\\phi_\\primal$ and $\\phi_\\dual$ are \n1-strongly convex with respect to norms $\\|\\cdot\\|_\\primal$ and \n$\\|\\cdot\\|_\\dual$, respectively:\n\\begin{equation} \\label{e-bcv-str-cvx}\nd_\\primal(x,x^\\prime) \\geq \\frac{1}{2}\\|x-x^\\prime\\|^2_\\primal, \\qquad\nd_\\dual(z,z^\\prime) \\geq \\frac{1}{2}\\|z-z^\\prime\\|^2_\\dual\n\\end{equation}\nfor all $(x,x') \\in \\dom d_\\primal$ and $(z,z') \\in \\dom d_\\dual$.\nThe assumption that the strong convexity constants are equal to one\ncan be made without loss of generality, by scaling the norms (or distances)\nif needed.\nWe also assume that the function $L\\phi_\\primal-h$ is convex for some \n$L>0$.  More precisely, $\\dom \\phi_\\primal \\subseteq \\dom h$ and\n\\BEQ \\label{e-bcv-lip}\nh(x)-h(x^\\prime)-\\inprod{\\nabla h(x^\\prime)}{x-x^\\prime}\n\\leq Ld_\\primal (x,x^\\prime) \\quad \\mbox{for all\\ }\n(x,x^\\prime) \\in \\dom d_\\primal.\n\\EEQ\nNote that this assumption is looser than\nthe one in~\\cite[Equation~(4)]{ChP:16}.\nWe denote by $\\|A\\|$ the matrix norm\n\\BEQ \\label{e-norm}\n\\|A\\|=\\sup_{u \\neq 0, v \\neq 0}\n    \\frac{\\inprod{v}{Au}}{\\|v\\|_\\dual \\|u\\|_\\primal}\n = \\sup_{u\\neq 0} \\frac{\\|Au\\|_{\\dual,*}}{\\|u\\|_\\primal}\n = \\sup_{v\\neq 0} \\frac{\\|A^Tv\\|_{\\primal,*}}{\\|v\\|_\\dual},\n\\EEQ\nwhere $\\|\\cdot\\|_{\\primal,*}$ and $\\|\\cdot\\|_{\\dual,*}$ are\nthe dual norms of $\\|\\cdot\\|_\\primal$ and $\\|\\cdot\\|_\\dual$. \n\nIt is also assumed that the primal--dual optimality \nconditions~\\eqref{e-opt-cond} have a solution $(x^\\star, z^\\star)$\nwith $x^\\star \\in \\dom \\phi_\\primal$ and \n$z^\\star \\in \\dom \\phi_\\dual$.  \n\n\nWe introduce four possible primal--dual kernel functions: the functions\n\\[\n\\phi_+(x,z) = \\frac{1}{\\tau}\\phi_\\primal(x)\n+\\frac{1}{\\sigma}\\phi_\\dual(z)+\\inprod{z}{Ax}, \\qquad\n\\phi_-(x,z) = \\frac{1}{\\tau}\\phi_\\primal(x)\n+\\frac{1}{\\sigma}\\phi_\\dual(z)-\\inprod{z}{Ax},\n\\]\nwhere $\\sigma,\\tau > 0$, and the functions \n\\[\n \\phi_\\mathrm{dcv}(x,z) = \\phi_+(x,z) - h(x), \\qquad \n \\phi_\\mathrm{pcv}(x,z) = \\phi_-(x,z) - h(x).\n\\]\nThe subscripts in $\\phi_+$ and $\\phi_-$ \nrefer to the sign of the inner product term $\\inprod{z}{Ax}$.\nThe subscripts in $\\phi_\\mathrm{pcv}$ and $\\phi_\\mathrm{dcv}$\nindicate the algorithm (Bregman primal or dual Condat-V\\~u) for which\nthese distances will be relevant. \nIf these kernel functions are convex,\nthey generate the following Bregman distances.\nThe distances generated by $\\phi_+$ and $\\phi_-$ are\n\\begin{align}\nd_+ (x,z;x^\\prime,z^\\prime) &= \n  \\frac{1}{\\tau}d_\\primal(x,x^\\prime)+\\frac{1}{\\sigma}d_\\dual(z,z^\\prime)\n  +\\inprod{z-z^\\prime}{A(x-x^\\prime)} \\nonumber \\\\\nd_-(x,z;x^\\prime,z^\\prime) &=\n  \\frac{1}{\\tau}d_\\primal(x,x^\\prime)+\\frac{1}{\\sigma}d_\\dual(z,z^\\prime)\n  -\\inprod{z-z^\\prime}{A(x-x^\\prime)}, \\label{e-bcv-dminus}\n\\end{align}\nrespectively, and the distances generated by $\\phi_\\mathrm{dcv}$ and \n$\\phi_\\mathrm{pcv}$ are  \n\\begin{align*}\nd_\\mathrm{dcv}(x,z;x^\\prime,z^\\prime) &= \n  d_+ (x,z;x^\\prime,z^\\prime) \n  -h(x)+h(x^\\prime)+\\inprod{\\nabla h(x')}{x-x^\\prime} \\\\\nd_\\mathrm{pcv}(x,z;x^\\prime,z^\\prime) &= \n  d_- (x,z;x^\\prime,z^\\prime) \n  -h(x)+h(x^\\prime)+\\inprod{\\nabla h(x')}{x-x^\\prime}.\n\\end{align*}\n\n\nThe algorithm is restricted to problems in the equality constrained\nform\n\\begin{equation} \\label{e-lcp-prob}\n \\begin{array}{ll}\n   \\mbox{minimize} & f(x) + h(x) \\\\\n   \\mbox{subject to} & Ax=b.\n \\end{array}\n\\end{equation}\nThis is a special case of~\\eqref{e-prob} with $g=\\delta_{\\{b\\}}$,\nthe indicator function of the singleton $\\{b\\}$.\n\n\n\\paragraph{Assumptions}\nWe make the same assumptions as in Section~\\ref{s-bcv-ppm}, but define\n\\[\n\\phi_\\dual (z) = \\frac{1}{2} \\|z\\|^2, \\qquad\nd_\\dual (z,z') = \\frac{1}{2} \\|z-z'\\|^2, \\qquad\n\\|z\\|_\\dual = \\|z\\|,\n\\]\nwhere $\\|\\cdot\\|$ is the Euclidean norm.\nThe matrix norm $\\|A\\|$ is defined accordingly as\n\\[\n\\|A\\|=\\sup_{u \\neq 0, v \\neq 0}\n  \\frac{\\inprod{v}{Au}}{\\|v\\| \\|u\\|_\\primal}\n= \\sup_{u \\neq 0} \\frac{\\|Au\\|}{\\|u\\|_\\primal}\n= \\sup_{v \\neq 0} \\frac{\\|A^Tv\\|_{\\primal,*}}{\\|v\\|}.\n\\]\n\n\nThe algorithm uses the following iteration,\nwith starting points $x^{(0)} \\in \\intr(\\dom \\phi_\\primal) \\cap \\dom h$\nand $z^{(-1)} = z^{(0)}$:\n\\begin{subequations} \\label{e-ls}\n\\begin{align}\n\\bar z^{(k+1)} &= z^{(k)} + \\theta_k (z^{(k)} - z^{(k-1)}) \n  \\label{e-ls-a} \\\\\nx^{(k+1)} &= \\prox_{\\tau_k f}^{\\phi_\\primal} \\big(\n  x^{(k)}, \\tau_k (A^T\\bar z^{(k+1)}+\\nabla h(x^{(k)}))\n  \\big) \\label{e-ls-b} \\\\\nz^{(k+1)} &= z^{(k)} + \\sigma_k (Ax^{(k+1)}-b). \\label{e-ls-c}\n\\end{align}\n\\end{subequations}\n\n\nIn the line search algorithm, the parameters \n$\\theta_k$, $\\tau_k$, $\\sigma_k$ are determined by a\nbacktracking search.\nAt the start of the algorithm, we set $\\tau_{-1}$ and $\\sigma_{-1}$ to\nsome positive values.\nTo start the search in iteration $k$ we choose\n$\\bar\\theta_k \\geq 1$.  For $i=0,1,2,\\ldots$,\nwe set $\\theta_k = 2^{-i}\\bar\\theta_k$,\n$\\tau_k=\\theta_k\\tau_{k-1}$, $\\sigma_k=\\theta_k\\sigma_{k-1}$,\nand compute $\\bar z_{k+1}$, $x_{k+1}$, $z_{k+1}$ using~\\eqref{e-ls}.\nFor some $\\delta \\in (0,1]$, if\n\\BEA\n& & \\inprod{z^{(k+1)}-\\bar z^{(k+1)}}{A(x^{(k+1)}-x^{(k)})}\n  +h(x^{(k+1)})-h(x^{(k)})-\\inprod{\\nabla h(x^{(k)})}{x^{(k+1)}-x^{(k)}}\n  \\nonumber \\\\\n&\\leq & \\frac{\\delta^2}{\\tau_k} d_\\primal(x^{(k+1)}, x^{(k)})\n    + \\frac{1}{2\\sigma_k} \\|\\bar z^{(k+1)} - z^{(k+1)}\\|^2,\n    \\label{e-ls-cond}\n\\EEA\nwe accept the computed iterates $\\bar z^{(k+1)}$, $x^{(k+1)}$, $z^{(k+1)}$\nand parameters $\\theta_k$, $\\sigma_k$, $\\tau_k$, and terminate the \nbacktracking search.\nIf~\\eqref{e-ls-cond} does not hold, we increment $i$ and continue\nthe backtracking search.\n\n\n\\BEQ \\label{e-ls-param-lbnd}\n  \\tau_k \\geq \\tau_\\mathrm{min} \\triangleq \\min{\\Big\\{\\tau_{-1},\n  \\frac{-L+\\sqrt{L^2+4\\delta^2\\beta\\|A\\|^2}}{4\\beta\\|A\\|^2}\\Big\\}},\n  \\qquad\n  \\sigma_k \\geq \\sigma_\\mathrm{min} \\triangleq \\beta\\tau_\\mathrm{min},\n\\EEQ\n\nwhere $\\beta=\\sigma_{-1}/\\tau_{-1}$.\nThe lower bounds imply that the backtracking eventually terminates\nwith positive stepsizes $\\sigma_k$ and $\\tau_k$.\n\n\n\\begin{align}\n\\cL(x^{(k+1)},z)-\\cL(x, \\bar z^{(k+1)})\n  &\\leq \\frac{1}{\\tau_k} \n  \\left(d_\\primal(x,x^{(k)}) - d_\\primal(x,x^{(k+1)}) -\n  (1-\\delta^2) d_\\primal(x^{(k+1)},x^{(k)})\\right) \\nonumber \\\\\n&\\phantom{=} + \\frac{1}{2\\sigma_k} \\left(\n  \\|z-z^{(k)}\\|^2 - \\|z-z^{(k+1)}\\|^2 - \n  \\|\\bar z^{(k+1)} - z^{(k)}\\|^2 \\right) \\label{e-ls-conv-i}\n\\end{align}\n\nWe define the averaged primal and dual sequences\n\\[\nx^{(k)}_\\avg = \\frac{1}{\\sum_{i=1}^k \\tau_{i-1}}\n\\sum_{i=1}^k \\tau_{i-1}x^{(i)}, \\qquad\n\\bar z^{(k)}_\\avg = \\frac{1}{\\sum_{i=1}^k \\tau_{i-1}}\n\\sum_{i=1}^k \\tau_{i-1} \\bar z^{(i)}\n\\]\nfor $k \\geq 1$.\n","units":[{"raw_tex":"We discuss proximal splitting methods for optimization problems \nin the form\n\\begin{equation} \\label{e-prob}\n  \\mbox{minimize} \\quad f(x) + g(Ax) + h(x),\n\\end{equation}\nwhere $f$, $g$, and $h$ are convex functions, and $h$ is differentiable.\n","start":2548,"end":2790},{"raw_tex":"We use the notation $\\inprod{x}{y} = x^Ty$ for the\nstandard inner product of vectors $x$ and $y$, and\n$\\|x\\| = \\inprod{x}{x}^{1/2}$ for the Euclidean norm of a vector\n$x$.  Other norms will be distinguished by a subscript.\n\n\\subsection{Problem formulation}\n\nIn~\\eqref{e-prob} the vector $x$ is an $n$-vector and\n$A$ is an $m\\times n$ matrix.  The functions $f$, $g$, $h$\nare closed and convex, and $h$ is differentiable, \\ie,  \n\\[\nh(x) \\geq h(x') + \\inprod{\\nabla h(x')}{x-x'} \\quad\n\\mbox{for all $x,x' \\in \\dom h$},\n\\]\nwhere $\\dom h$ is an open convex set.  We assume that $f+h$ and $g$\nare proper, \\ie, have nonempty domains.\n","start":8657,"end":9285},{"raw_tex":"The dual of problem~\\eqref{e-prob} is\n\\BEQ \\label{e-prob-dual}\n \\mbox{maximize}\\quad  {-(f+h)^*(-A^Tz) - g^*(z)},\n\\EEQ\nwhere $(f+h)^*$ and $g^*$ are the conjugates of $f+h$ and $g$:\n\\[\n (f+h)^*(w) = \\sup_x{(\\inprod{w}{x} - f(x) - h(x))}, \\qquad\n g^*(z) = \\sup_y{(\\inprod{z}{y} - g(y))}.\n\\]\n","start":10003,"end":10293},{"raw_tex":"The primal--dual optimality conditions for~\\eqref{e-prob}\nand~\\eqref{e-prob-dual} are\n\\[\n 0 \\in \\partial f(x) + \\nabla h(x) + A^T z, \\qquad\n 0 \\in \\partial g^*(z) - Ax.\n\\]\nHere $\\partial f$ and $\\partial g^*$ are the subdifferentials of\n$f$ and  $g^*$.  We often write the optimality conditions as\n\\begin{equation} \\label{e-opt-cond}\n  0 \\in \\begin{bmatrix} 0 & A^T \\\\ -A & 0 \\end{bmatrix}\n  \\begin{bmatrix} x \\\\ z \\end{bmatrix} + \\begin{bmatrix}\n    \\partial f(x)+\\nabla h(x) \\\\ \\partial g^\\ast(z)\n\\end{bmatrix}.\n\\end{equation}\nThroughout the paper, we assume that the optimality \nconditions~\\eqref{e-opt-cond} are solvable.\n\nWe will refer to the convex--concave function\n\\[\n\\cL(x,z) = f(x)+h(x)+ \\inprod{z}{Ax} -g^\\ast(z)\n\\]\nas the \\emph{Lagrangian} of~\\eqref{e-prob}.\nWe follow the convention that $\\cL(x,z) = +\\infty$ if $x\\not\\in \\dom(f+h)$\nand $\\cL(x,z) = -\\infty$ if $x\\in\\dom(f+h)$ and $z\\not\\in \\dom g^*$.\n","start":10457,"end":11372},{"raw_tex":"Let $\\phi$ be a convex function with a domain that has nonempty\ninterior, and assume $\\phi$ is continuous on $\\dom \\phi$ and \ncontinuously differentiable on $\\intr(\\dom \\phi)$.\nThe \\emph{generalized distance} (or \\emph{Bregman distance})\ngenerated by the \\emph{kernel function} $\\phi$ is defined as the function\n\\[\nd(x,y) = \\phi(x)-\\phi(y)-\\inprod{\\nabla \\phi(y)}{x-y},\n\\]\nwith domain $\\dom d=\\dom \\phi \\times \\intr(\\dom \\phi)$.\nThe corresponding \\emph{Bregman proximal operator} of a function $f$ is \n\\begin{align}\n\\prox_f^\\phi(y,a)\n  &= \\argmin_x{(f(x)+\\inprod{a}{x}+d(x,y))} \\label{e-prox-opt} \\\\\n&= \\argmin_x{(f(x)+\\inprod{a}{x}+\\phi(x)\n   -\\inprod{\\nabla \\phi(y)}{x})}. \\label{e-prox-opt2}\n\\end{align}\nIt is assumed that for every $a$ and every $y \\in \\intr(\\dom\\phi)$\nthe minimizer $\\hat x=\\prox_f^\\phi(y,a)$ is unique and in\n$\\intr(\\dom \\phi)$.\n","start":37479,"end":38331},{"raw_tex":"From the expression~\\eqref{e-prox-opt2} we see that \n$\\hat x=\\prox_f^\\phi(y,a)$\nsatisfies\n\\[\n\\nabla \\phi(y) - \\nabla \\phi(\\hat x) - a \\in \\partial f(\\hat x).\n\\]\nEquivalently, by definition of subgradient,\n\\begin{align}\nf(x) + \\inprod{a}{x}\n&\\geq f(\\hat x)+\\inprod{a}{\\hat x}\n  +\\inprod{\\nabla\\phi(y)-\\nabla\\phi(\\hat x)}{x-\\hat x} \\nonumber \\\\\n&= f(\\hat x)+\\inprod{a}{\\hat x}+d(\\hat x, y)+d(x,\\hat x)-d(x,y)\n  \\label{e-prox-cond}\n\\end{align}\nfor all $x\\in \\dom f \\cap \\dom \\phi$.\n","start":39609,"end":40088},{"raw_tex":"\\paragraph{Assumptions} \nThroughout Section~\\ref{s-bcv} we make the following assumptions.\nThe kernel functions $\\phi_\\primal$ and $\\phi_\\dual$ are \n1-strongly convex with respect to norms $\\|\\cdot\\|_\\primal$ and \n$\\|\\cdot\\|_\\dual$, respectively:\n\\begin{equation} \\label{e-bcv-str-cvx}\nd_\\primal(x,x^\\prime) \\geq \\frac{1}{2}\\|x-x^\\prime\\|^2_\\primal, \\qquad\nd_\\dual(z,z^\\prime) \\geq \\frac{1}{2}\\|z-z^\\prime\\|^2_\\dual\n\\end{equation}\nfor all $(x,x') \\in \\dom d_\\primal$ and $(z,z') \\in \\dom d_\\dual$.\nThe assumption that the strong convexity constants are equal to one\ncan be made without loss of generality, by scaling the norms (or distances)\nif needed.\nWe also assume that the function $L\\phi_\\primal-h$ is convex for some \n$L>0$.  More precisely, $\\dom \\phi_\\primal \\subseteq \\dom h$ and\n\\BEQ \\label{e-bcv-lip}\nh(x)-h(x^\\prime)-\\inprod{\\nabla h(x^\\prime)}{x-x^\\prime}\n\\leq Ld_\\primal (x,x^\\prime) \\quad \\mbox{for all\\ }\n(x,x^\\prime) \\in \\dom d_\\primal.\n\\EEQ\nNote that this assumption is looser than\nthe one in~\\cite[Equation~(4)]{ChP:16}.\nWe denote by $\\|A\\|$ the matrix norm\n\\BEQ \\label{e-norm}\n\\|A\\|=\\sup_{u \\neq 0, v \\neq 0}\n    \\frac{\\inprod{v}{Au}}{\\|v\\|_\\dual \\|u\\|_\\primal}\n = \\sup_{u\\neq 0} \\frac{\\|Au\\|_{\\dual,*}}{\\|u\\|_\\primal}\n = \\sup_{v\\neq 0} \\frac{\\|A^Tv\\|_{\\primal,*}}{\\|v\\|_\\dual},\n\\EEQ\nwhere $\\|\\cdot\\|_{\\primal,*}$ and $\\|\\cdot\\|_{\\dual,*}$ are\nthe dual norms of $\\|\\cdot\\|_\\primal$ and $\\|\\cdot\\|_\\dual$. \n\nIt is also assumed that the primal--dual optimality \nconditions~\\eqref{e-opt-cond} have a solution $(x^\\star, z^\\star)$\nwith $x^\\star \\in \\dom \\phi_\\primal$ and \n$z^\\star \\in \\dom \\phi_\\dual$.  \n","start":42221,"end":43843},{"raw_tex":"We introduce four possible primal--dual kernel functions: the functions\n\\[\n\\phi_+(x,z) = \\frac{1}{\\tau}\\phi_\\primal(x)\n+\\frac{1}{\\sigma}\\phi_\\dual(z)+\\inprod{z}{Ax}, \\qquad\n\\phi_-(x,z) = \\frac{1}{\\tau}\\phi_\\primal(x)\n+\\frac{1}{\\sigma}\\phi_\\dual(z)-\\inprod{z}{Ax},\n\\]\nwhere $\\sigma,\\tau > 0$, and the functions \n\\[\n \\phi_\\mathrm{dcv}(x,z) = \\phi_+(x,z) - h(x), \\qquad \n \\phi_\\mathrm{pcv}(x,z) = \\phi_-(x,z) - h(x).\n\\]\nThe subscripts in $\\phi_+$ and $\\phi_-$ \nrefer to the sign of the inner product term $\\inprod{z}{Ax}$.\nThe subscripts in $\\phi_\\mathrm{pcv}$ and $\\phi_\\mathrm{dcv}$\nindicate the algorithm (Bregman primal or dual Condat-V\\~u) for which\nthese distances will be relevant. \nIf these kernel functions are convex,\nthey generate the following Bregman distances.\nThe distances generated by $\\phi_+$ and $\\phi_-$ are\n\\begin{align}\nd_+ (x,z;x^\\prime,z^\\prime) &= \n  \\frac{1}{\\tau}d_\\primal(x,x^\\prime)+\\frac{1}{\\sigma}d_\\dual(z,z^\\prime)\n  +\\inprod{z-z^\\prime}{A(x-x^\\prime)} \\nonumber \\\\\nd_-(x,z;x^\\prime,z^\\prime) &=\n  \\frac{1}{\\tau}d_\\primal(x,x^\\prime)+\\frac{1}{\\sigma}d_\\dual(z,z^\\prime)\n  -\\inprod{z-z^\\prime}{A(x-x^\\prime)}, \\label{e-bcv-dminus}\n\\end{align}\nrespectively, and the distances generated by $\\phi_\\mathrm{dcv}$ and \n$\\phi_\\mathrm{pcv}$ are  \n\\begin{align*}\nd_\\mathrm{dcv}(x,z;x^\\prime,z^\\prime) &= \n  d_+ (x,z;x^\\prime,z^\\prime) \n  -h(x)+h(x^\\prime)+\\inprod{\\nabla h(x')}{x-x^\\prime} \\\\\nd_\\mathrm{pcv}(x,z;x^\\prime,z^\\prime) &= \n  d_- (x,z;x^\\prime,z^\\prime) \n  -h(x)+h(x^\\prime)+\\inprod{\\nabla h(x')}{x-x^\\prime}.\n\\end{align*}\n","start":45215,"end":46769},{"raw_tex":"The algorithm is restricted to problems in the equality constrained\nform\n\\begin{equation} \\label{e-lcp-prob}\n \\begin{array}{ll}\n   \\mbox{minimize} & f(x) + h(x) \\\\\n   \\mbox{subject to} & Ax=b.\n \\end{array}\n\\end{equation}\nThis is a special case of~\\eqref{e-prob} with $g=\\delta_{\\{b\\}}$,\nthe indicator function of the singleton $\\{b\\}$.\n","start":76128,"end":76464},{"raw_tex":"\\paragraph{Assumptions}\nWe make the same assumptions as in Section~\\ref{s-bcv-ppm}, but define\n\\[\n\\phi_\\dual (z) = \\frac{1}{2} \\|z\\|^2, \\qquad\nd_\\dual (z,z') = \\frac{1}{2} \\|z-z'\\|^2, \\qquad\n\\|z\\|_\\dual = \\|z\\|,\n\\]\nwhere $\\|\\cdot\\|$ is the Euclidean norm.\nThe matrix norm $\\|A\\|$ is defined accordingly as\n\\[\n\\|A\\|=\\sup_{u \\neq 0, v \\neq 0}\n  \\frac{\\inprod{v}{Au}}{\\|v\\| \\|u\\|_\\primal}\n= \\sup_{u \\neq 0} \\frac{\\|Au\\|}{\\|u\\|_\\primal}\n= \\sup_{v \\neq 0} \\frac{\\|A^Tv\\|_{\\primal,*}}{\\|v\\|}.\n\\]\n","start":76744,"end":77234},{"raw_tex":"The algorithm uses the following iteration,\nwith starting points $x^{(0)} \\in \\intr(\\dom \\phi_\\primal) \\cap \\dom h$\nand $z^{(-1)} = z^{(0)}$:\n\\begin{subequations} \\label{e-ls}\n\\begin{align}\n\\bar z^{(k+1)} &= z^{(k)} + \\theta_k (z^{(k)} - z^{(k-1)}) \n  \\label{e-ls-a} \\\\\nx^{(k+1)} &= \\prox_{\\tau_k f}^{\\phi_\\primal} \\big(\n  x^{(k)}, \\tau_k (A^T\\bar z^{(k+1)}+\\nabla h(x^{(k)}))\n  \\big) \\label{e-ls-b} \\\\\nz^{(k+1)} &= z^{(k)} + \\sigma_k (Ax^{(k+1)}-b). \\label{e-ls-c}\n\\end{align}\n\\end{subequations}\n","start":77277,"end":77774},{"raw_tex":"In the line search algorithm, the parameters \n$\\theta_k$, $\\tau_k$, $\\sigma_k$ are determined by a\nbacktracking search.\nAt the start of the algorithm, we set $\\tau_{-1}$ and $\\sigma_{-1}$ to\nsome positive values.\nTo start the search in iteration $k$ we choose\n$\\bar\\theta_k \\geq 1$.  For $i=0,1,2,\\ldots$,\nwe set $\\theta_k = 2^{-i}\\bar\\theta_k$,\n$\\tau_k=\\theta_k\\tau_{k-1}$, $\\sigma_k=\\theta_k\\sigma_{k-1}$,\nand compute $\\bar z_{k+1}$, $x_{k+1}$, $z_{k+1}$ using~\\eqref{e-ls}.\nFor some $\\delta \\in (0,1]$, if\n\\BEA\n& & \\inprod{z^{(k+1)}-\\bar z^{(k+1)}}{A(x^{(k+1)}-x^{(k)})}\n  +h(x^{(k+1)})-h(x^{(k)})-\\inprod{\\nabla h(x^{(k)})}{x^{(k+1)}-x^{(k)}}\n  \\nonumber \\\\\n&\\leq & \\frac{\\delta^2}{\\tau_k} d_\\primal(x^{(k+1)}, x^{(k)})\n    + \\frac{1}{2\\sigma_k} \\|\\bar z^{(k+1)} - z^{(k+1)}\\|^2,\n    \\label{e-ls-cond}\n\\EEA\nwe accept the computed iterates $\\bar z^{(k+1)}$, $x^{(k+1)}$, $z^{(k+1)}$\nand parameters $\\theta_k$, $\\sigma_k$, $\\tau_k$, and terminate the \nbacktracking search.\nIf~\\eqref{e-ls-cond} does not hold, we increment $i$ and continue\nthe backtracking search.\n","start":78232,"end":79298},{"raw_tex":"\\BEQ \\label{e-ls-param-lbnd}\n  \\tau_k \\geq \\tau_\\mathrm{min} \\triangleq \\min{\\Big\\{\\tau_{-1},\n  \\frac{-L+\\sqrt{L^2+4\\delta^2\\beta\\|A\\|^2}}{4\\beta\\|A\\|^2}\\Big\\}},\n  \\qquad\n  \\sigma_k \\geq \\sigma_\\mathrm{min} \\triangleq \\beta\\tau_\\mathrm{min},\n\\EEQ","start":80009,"end":80255},{"raw_tex":"where $\\beta=\\sigma_{-1}/\\tau_{-1}$.\nThe lower bounds imply that the backtracking eventually terminates\nwith positive stepsizes $\\sigma_k$ and $\\tau_k$.\n","start":80256,"end":80409},{"raw_tex":"\\begin{align}\n\\cL(x^{(k+1)},z)-\\cL(x, \\bar z^{(k+1)})\n  &\\leq \\frac{1}{\\tau_k} \n  \\left(d_\\primal(x,x^{(k)}) - d_\\primal(x,x^{(k+1)}) -\n  (1-\\delta^2) d_\\primal(x^{(k+1)},x^{(k)})\\right) \\nonumber \\\\\n&\\phantom{=} + \\frac{1}{2\\sigma_k} \\left(\n  \\|z-z^{(k)}\\|^2 - \\|z-z^{(k+1)}\\|^2 - \n  \\|\\bar z^{(k+1)} - z^{(k)}\\|^2 \\right) \\label{e-ls-conv-i}\n\\end{align}","start":82048,"end":82403},{"raw_tex":"We define the averaged primal and dual sequences\n\\[\nx^{(k)}_\\avg = \\frac{1}{\\sum_{i=1}^k \\tau_{i-1}}\n\\sum_{i=1}^k \\tau_{i-1}x^{(i)}, \\qquad\n\\bar z^{(k)}_\\avg = \\frac{1}{\\sum_{i=1}^k \\tau_{i-1}}\n\\sum_{i=1}^k \\tau_{i-1} \\bar z^{(i)}\n\\]\nfor $k \\geq 1$.\n","start":87154,"end":87404}]},"statement":{"raw_tex":"We show that\n\\begin{align}\n\\cL(x^{(k)}_\\avg,z) - \\cL(x,\\bar z^{(k)}_\\avg)\n  &\\leq \\frac{1}{\\sum_{i=1}^k \\tau_{i-1}}\n  \\big(d_\\primal(x,x^{(0)}) + \\frac{1}{2\\beta} \\|z-z^{(0)}\\|^2\\big)\n  \\label{e-ls-conv} \\\\ \n&\\leq \\frac{1}{k \\tau_\\mathrm{min}}\n  \\big(d_\\primal(x,x^{(0)}) + \\frac{1}{2\\beta} \\|z-z^{(0)}\\|^2\\big)\n  \\label{e-ls-conv-1/k}\n\\end{align}\nfor all $x \\in \\dom f \\cap \\dom \\phi_\\primal$ and all $z$.\nThis holds for any choice of $\\delta \\in (0,1]$ in~\\eqref{e-ls-cond}.\n","units":[{"raw_tex":"We show that\n\\begin{align}\n\\cL(x^{(k)}_\\avg,z) - \\cL(x,\\bar z^{(k)}_\\avg)\n  &\\leq \\frac{1}{\\sum_{i=1}^k \\tau_{i-1}}\n  \\big(d_\\primal(x,x^{(0)}) + \\frac{1}{2\\beta} \\|z-z^{(0)}\\|^2\\big)\n  \\label{e-ls-conv} \\\\ \n&\\leq \\frac{1}{k \\tau_\\mathrm{min}}\n  \\big(d_\\primal(x,x^{(0)}) + \\frac{1}{2\\beta} \\|z-z^{(0)}\\|^2\\big)\n  \\label{e-ls-conv-1/k}\n\\end{align}\nfor all $x \\in \\dom f \\cap \\dom \\phi_\\primal$ and all $z$.\nThis holds for any choice of $\\delta \\in (0,1]$ in~\\eqref{e-ls-cond}.\n","start":87404,"end":87881}]},"proof":{"raw_tex":"\\begin{proof}\nTo show that the kernel functions $\\phi_+$ and $\\phi_-$ are convex,\nwe show that $d_+$ and $d_-$ are nonnegative.\nSuppose $\\sigma\\tau \\|A\\|^2 \\leq \\delta_1 \\delta_2$\nwith $\\delta_1,\\delta_2 > 0$.\nThen~\\eqref{e-bcv-str-cvx} and the arithmetic--geometric mean inequality\nimply that\n\\begin{align}\n\\left| \\inprod{z-z^\\prime}{A(x-x^\\prime)}\\right|\n  &\\leq \\|A\\| \\|z-z'\\|_\\dual \\|x-x'\\|_\\primal  \\nonumber \\\\\n&\\leq \\sqrt{\\frac{\\delta_1 \\delta_2}{\\sigma \\tau}}\n\\|z-z'\\|_\\dual \\|x-x'\\|_\\primal \\nonumber \\\\\n&\\leq \\frac{\\delta_1}{2\\tau} \\|x-x'\\|_\\primal^2  +\n  \\frac{\\delta_2}{2\\sigma} \\|z-z'\\|_\\primal^2 \\nonumber \\\\\n&\\leq \\frac{\\delta_1}{\\tau} d_\\primal (x,x') +\n  \\frac{\\delta_2}{\\sigma} d_\\dual (z,z'). \\label{e-bcv-delta}\n\\end{align}\nTherefore,\n\\begin{align*}\nd_{\\pm} (x,z; x',z')\n  &= \\frac{1}{\\tau} d_\\primal(x,x^\\prime) + \\frac{1}{\\sigma}\n  d_\\dual(z,z^\\prime) \\pm \\inprod{z-z^\\prime}{A(x-x^\\prime)} \\\\\n&\\geq \\frac{1-\\delta_1}{\\tau} d_\\primal (x,x') +\n  \\frac{1-\\delta_2}{\\sigma} d_\\dual (z,z') \\\\\n&\\geq \\frac{1-\\delta_1}{2\\tau} \\|x-x'\\|_\\primal^2 +\n  \\frac{1-\\delta_2}{2\\sigma} \\|z-z'\\|_\\dual^2.\n\\end{align*}\nWith $\\delta_1=\\delta_2=1$, this shows convexity of $\\phi_+$ and $\\phi_-$;\nwith $\\delta_1=\\delta_2<1$, strong convexity.\nSimilarly,\n\\begin{align*}\nd_\\mathrm{dcv/pcv} (x,z; x', z')\n  &= d_{\\pm} (x,z; x',z') - h(x) + h(x') + \\inprod{\\nabla h(x')}{x-x'} \\\\\n&\\geq \\frac{1-\\tau L-\\delta_1}{\\tau} d_\\primal (x,x^\\prime) +\n  \\frac{1-\\delta_2}{\\sigma} d_\\dual (z,z^\\prime).\n\\end{align*}\nWith $\\delta_1=1-\\tau L$ and $\\delta_2=1$, this shows convexity of\n$\\phi_\\mathrm{pcv}$ and $\\phi_\\mathrm{dcv}$;\nwith $\\delta_1=\\delta-\\tau L$ and $\\delta_2=\\delta < 1$,\nstrong convexity.\n\\end{proof}\n\n\\begin{proof}\nApplying~\\eqref{e-bcv-delta} with $\\tau=\\tau_k$, $\\sigma=\\sigma_k$, $\\delta_1=\\delta^2-\\tau_k L$ and $\\delta_2=1$,\ntogether with the Lipschitz condition~\\eqref{e-bcv-lip},\nwe see that the backtracking condition~\\eqref{e-ls-cond} holds\nat iteration $k$ if $0 < \\delta < 1$ and\n\\[\n\\tau_k \\sigma_k\\|A\\|^2 + \\tau_k L \\leq \\delta^2.\n\\]\nThen mathematical induction can be used to prove~\\eqref{e-ls-param-lbnd}.\nThe two lower bounds~\\eqref{e-ls-param-lbnd} hold at $k=0$\nby the definition of $\\tau_\\mathrm{min}$ and $\\sigma_\\mathrm{min}$.\nNow assume $\\tau_{k-1} \\geq \\tau_\\mathrm{min}$,\n$\\sigma_{k-1} \\geq \\sigma_\\mathrm{min}$,\nand consider the $k$th iteration.\nThe first attempt of $\\theta_k$ is $\\theta_k=\\bar \\theta_k \\geq 1$.\nIf this value is accepted, then\n\\[\n\\tau_k = \\bar\\theta_k \\tau_{k-1} \\geq \\tau_{k-1} \\geq \\tau_\\mathrm{min},\n\\qquad\n\\sigma_k = \\bar\\theta_k \\sigma_{k-1} \\geq \\sigma_{k-1} \\geq\n  \\sigma_\\mathrm{min}.\n\\]\nOtherwise, one or more backtracking steps are needed.\nDenote by $\\tilde \\theta_k$ the last rejected value.\nThen $\\tilde\\theta_k^2\\tau_{k-1}^2\\beta\\|A\\|^2 + \n\\tilde\\theta_k \\tau_{k-1} L >\\delta^2$\nand the accepted $\\theta_k$ satisfies\n\\[\n\\theta_k = \\frac{\\tilde \\theta_k}{2} \\geq\n\\frac{-L+\\sqrt{L^2+4\\delta^2\\beta\\|A\\|^2}}%\n  {4\\tau_{k-1}\\beta\\|A\\|^2}.\n\\]\nTherefore,\n\\[\n\\tau_k=\\theta_k\\tau_{k-1} >\n\\frac{-L+\\sqrt{L^2+4\\delta^2\\beta\\|A\\|^2}}{4\\beta\\|A\\|^2}, \\qquad\n\\sigma_k=\\beta\\tau_k \\geq \\beta\\tau_\\mathrm{min}. \\qedhere\n\\]\n\\end{proof}\n\n\\begin{proof}\nThe optimality condition for the primal prox-operator~\\eqref{e-ls-b}\ngives\n\\BEAS\n\\lefteqn{f(x^{(k+1)})-f(x)} \\\\\n&\\leq & \\frac{1}{\\tau_k}\n  \\big(d_\\primal(x,x^{(k)})-d_\\primal(x,x^{(k+1)})-\n  d_\\primal(x^{(k+1)}, x^{(k)})\\big)\n  +\\inprod{A^T\\bar z^{(k+1)}+\\nabla h(x^{(k)})}{x-x^{(k+1)}}\n\\EEAS\nfor all $x\\in \\dom f \\cap \\dom\\phi_\\primal$.\nHence\n\\BEA\n\\lefteqn{f(x^{(k+1)}) + h(x^{(k+1)}) - f(x) - h(x)} \\nonumber \\\\\n  &\\leq & \\frac{1}{\\tau_k} (d_\\primal(x,x^{(k)}) - \n  d_\\primal(x,x^{(k+1)}) - d_\\primal(x^{(k+1)}, x^{(k)}))\n  + \\inprod{A^T\\bar z_{k+1}}{x - x^{(k+1)}} \\nonumber \\\\\n& &\\mbox{} + h(x^{(k+1)})-h(x)+\\inprod{\\nabla h(x^{(k)})}{x-x^{(k+1)}}\n  \\nonumber \\\\\n&\\leq & \\frac{1}{\\tau_k} (d_\\primal(x,x^{(k)}) - \n d_\\primal(x,x^{(k+1)}) - d_\\primal(x^{(k+1)}, x^{(k)}))\n  + \\inprod{A^T\\bar z^{(k+1)}}{x - x^{(k+1)}} \\nonumber \\\\\n& &\\mbox{} +h(x^{(k+1)})-h(x^{(k)})\n  -\\inprod{\\nabla h(x^{(k)})}{x^{(k+1)}-x^{(k)}}.\n  \\label{e-ls-conv-1}\n\\EEA\nThe second inequality follows from the convexity of $h$, \\ie,\n$h(x) \\geq h(x^{(k)}) + \\inprod{\\nabla h(x^{(k)})}{x-x^{(k)}}$.\nThe dual update~\\eqref{e-ls-c} implies that\n\\BEQ \\label{e-ls-conv-2}\n\\inprod{z-z^{(k+1)}}{Ax^{(k+1)}-b}\n= \\frac{1}{\\sigma_k} \\inprod{z-z^{(k+1)}}{z^{(k+1)}-z^{(k)}}\n\\quad \\mbox{for all $z$.}\n\\EEQ\nThis equality at $k=i-1$ is\n\\begin{align}\n\\inprod{z-z^{(i)}}{Ax^{(i)}-b}\n  &= \\frac{1}{\\sigma_{i-1}} \\inprod{z-z^{(i)}}{z^{(i)}-z^{(i-1)}} \n  \\nonumber \\\\\n&= \\frac{1}{2\\sigma_{i-1}} \\left( \\|z-z^{(i-1)}\\|^2\n  - \\|z-z^{(i)}\\|^2 - \\|z^{(i)} - z^{(i-1)} \\|^2 \\right).\n  \\label{e-ls-conv-3}\n\\end{align}\nThe equality~\\eqref{e-ls-conv-2} at $k=i-2$ is\n\\begin{align*}\n\\inprod{z-z^{(i-1)}}{Ax^{(i-1)}-b}\n  &= \\frac{1}{\\sigma_{i-2}} \\inprod{z-z^{(i-1)}}{z^{(i-1)}-z^{(i-2)}} \\\\\n&= \\frac{\\theta_{i-1}}{\\sigma_{i-1}}\n  \\inprod{z-z^{(i-1)}}{z^{(i-1)}-z^{(i-2)}} \\\\\n&= \\frac{1}{\\sigma_{i-1}} \\inprod{z-z^{(i-1)}}{\\bar z^{(i)}-z^{(i-1)}}.\n\\end{align*}\nWe evaluate this at $z=z^{(i)}$ and add it to the equality\nat $z=z^{(i-2)}$ multiplied by $\\theta_{i-1}$:\n\\begin{align}\n\\inprod{z^{(i)}-\\bar z^{(i)}}{Ax^{(i-1)}-b}\n  &= \\frac{1}{\\sigma_{i-1}}\n  \\inprod{z^{(i)}-\\bar z^{(i)}}{\\bar z^{(i)} - z^{(i-1)}} \\nonumber \\\\\n&= \\frac{1}{2\\sigma_{i-1}} \\left(\\|z^{(i)}-z^{(i-1)}\\|^2\n  -\\|z^{(i)}-\\bar z^{(i)}\\|^2-\\|\\bar z^{(i)}-z^{(i-1)}\\|^2 \\right).\n  \\label{e-ls-conv-4}\n\\end{align}\nNow we combine~\\eqref{e-ls-conv-1} for $k=i-1$,\nwith~\\eqref{e-ls-conv-3} and~\\eqref{e-ls-conv-4}. For $i \\geq 1$,\n\\BEAS\n\\lefteqn{\\cL(x^{(i)},z) - \\cL(x, \\bar z^{(i)})} \\nonumber \\\\\n&=& f(x^{(i)})+h(x^{(i)})+\\inprod{z}{Ax^{(i)}-b}\n  -f(x)-h(x)-\\inprod{\\bar z^{(i)}}{Ax-b} \\nonumber \\\\\n&\\leq & \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)})-d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big)\n  + \\inprod{A^T\\bar z^{(i)}}{x-x^{(i)}} + \\inprod{z}{Ax^{(i)}-b} \\nonumber\\\\\n& &\\mbox{} - \\inprod{\\bar z^{(i)}}{Ax-b} + h(x^{(i)}) - h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)}) \n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big)\n  + \\inprod{z-\\bar z^{(i)}}{Ax^{(i)}-b} \\nonumber \\\\\n& &\\mbox{} +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big) \\nonumber \\\\\n& & \\mbox{} +\\inprod{z^{(i)}-\\bar z^{(i)}}{A(x^{(i)}-x^{(i-1)})}\n  + \\inprod{z-z^{(i)}}{Ax^{(i)}-b}\n  + \\inprod{z^{(i)}-\\bar z^{(i)}}{Ax^{(i-1)}-b} \\nonumber \\\\\n& &\\mbox{} +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)}))\\Big) \\nonumber \\\\\n& &\\mbox{} +\\frac{1}{2\\sigma_{i-1}}\n  \\Big(\\|z-z^{(i-1)}\\|^2-\\|z-z^{(i)}\\|^2-\\|\\bar z^{(i)}-z^{(i-1)}\\|^2\n  -\\|\\bar z^{(i)}-z^{(i)}\\|^2\\Big) \\nonumber \\\\\n& &\\mbox{}+\\inprod{A^T(z^{(i)}-\\bar z^{(i)})}{x^{(i)}-x^{(i-1)}}\n  +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)})} \\\\\n&\\leq & \\frac{1}{\\tau_{i-1}} \n  \\left(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)}) -\n  (1-\\delta^2) d_\\primal(x^{(i)},x^{(i-1)})\\right) \\nonumber \\\\\n& &\\mbox{} + \\frac{1}{2\\sigma_{i-1}} \\left(\n  \\|z-z^{(i-1)}\\|^2 - \\|z-z^{(i)}\\|^2 - \n  \\|\\bar z^{(i)} - z^{(i-1)}\\|^2 \\right),\n\\EEAS\nwhich is the desired result~\\eqref{e-ls-conv-i}.\nThe first inequality follows from~\\eqref{e-ls-conv-1}.\nIn the second last step we substitute~\\eqref{e-ls-conv-3}\nand~\\eqref{e-ls-conv-4}.\nThe last step uses the line search exit condition~\\eqref{e-ls-cond}\nat $k=i-1$.\n\\end{proof}\n\n\\begin{proof}\nFrom~\\eqref{e-ls-conv-i},\n\\[\n\\cL(x^{(i)},z)-\\cL(x, \\bar z^{(i)}) \\leq \\frac{1}{\\tau_{i-1}}\n\\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n+ \\frac{1}{2\\beta} \\|z-z^{(i-1)}\\|^2\n- \\frac{1}{2\\beta} \\|z-z^{(i)}\\|^2 \\Big).\n\\]\nSince $\\cL(u,v)$ is convex in $u$ and affine in $v$,\n\\BEA\n\\lefteqn{(\\sum_{i=1}^k \\tau_{i-1}) \n  \\big(\\cL(x^{(k)}_\\avg, z)-\\cL(x, \\bar z^{(k)}_\\avg)\\big)} \\nonumber \\\\\n&\\leq &\\sum_{i=1}^k \\tau_{i-1} (\\cL(x^{(i)}, z)-\\cL(x, \\bar z^{(i)})) \n  \\nonumber \\\\\n&\\leq & d_\\primal(x,x^{(0)}) - d_\\primal(x,x^{(k)}) +\n  \\frac{1}{2\\beta} (\\|z-z^{(0)}\\|^2 - \\|z-z^{(k)}\\|^2) \\nonumber \\\\\n&\\leq & d_\\primal(x,x^{(0)}) + \\frac{1}{2\\beta} \\|z-z^{(0)}\\|^2.\n  \\label{e-ls-conv-k}\n\\EEA\nDividing by $\\sum_{i=1}^k \\tau_{i-1}$ gives~\\eqref{e-ls-conv}.\n\\end{proof}","units":[{"raw_tex":"\\begin{proof}\nTo show that the kernel functions $\\phi_+$ and $\\phi_-$ are convex,\nwe show that $d_+$ and $d_-$ are nonnegative.\nSuppose $\\sigma\\tau \\|A\\|^2 \\leq \\delta_1 \\delta_2$\nwith $\\delta_1,\\delta_2 > 0$.\nThen~\\eqref{e-bcv-str-cvx} and the arithmetic--geometric mean inequality\nimply that\n\\begin{align}\n\\left| \\inprod{z-z^\\prime}{A(x-x^\\prime)}\\right|\n  &\\leq \\|A\\| \\|z-z'\\|_\\dual \\|x-x'\\|_\\primal  \\nonumber \\\\\n&\\leq \\sqrt{\\frac{\\delta_1 \\delta_2}{\\sigma \\tau}}\n\\|z-z'\\|_\\dual \\|x-x'\\|_\\primal \\nonumber \\\\\n&\\leq \\frac{\\delta_1}{2\\tau} \\|x-x'\\|_\\primal^2  +\n  \\frac{\\delta_2}{2\\sigma} \\|z-z'\\|_\\primal^2 \\nonumber \\\\\n&\\leq \\frac{\\delta_1}{\\tau} d_\\primal (x,x') +\n  \\frac{\\delta_2}{\\sigma} d_\\dual (z,z'). \\label{e-bcv-delta}\n\\end{align}\nTherefore,\n\\begin{align*}\nd_{\\pm} (x,z; x',z')\n  &= \\frac{1}{\\tau} d_\\primal(x,x^\\prime) + \\frac{1}{\\sigma}\n  d_\\dual(z,z^\\prime) \\pm \\inprod{z-z^\\prime}{A(x-x^\\prime)} \\\\\n&\\geq \\frac{1-\\delta_1}{\\tau} d_\\primal (x,x') +\n  \\frac{1-\\delta_2}{\\sigma} d_\\dual (z,z') \\\\\n&\\geq \\frac{1-\\delta_1}{2\\tau} \\|x-x'\\|_\\primal^2 +\n  \\frac{1-\\delta_2}{2\\sigma} \\|z-z'\\|_\\dual^2.\n\\end{align*}\nWith $\\delta_1=\\delta_2=1$, this shows convexity of $\\phi_+$ and $\\phi_-$;\nwith $\\delta_1=\\delta_2<1$, strong convexity.\nSimilarly,\n\\begin{align*}\nd_\\mathrm{dcv/pcv} (x,z; x', z')\n  &= d_{\\pm} (x,z; x',z') - h(x) + h(x') + \\inprod{\\nabla h(x')}{x-x'} \\\\\n&\\geq \\frac{1-\\tau L-\\delta_1}{\\tau} d_\\primal (x,x^\\prime) +\n  \\frac{1-\\delta_2}{\\sigma} d_\\dual (z,z^\\prime).\n\\end{align*}\nWith $\\delta_1=1-\\tau L$ and $\\delta_2=1$, this shows convexity of\n$\\phi_\\mathrm{pcv}$ and $\\phi_\\mathrm{dcv}$;\nwith $\\delta_1=\\delta-\\tau L$ and $\\delta_2=\\delta < 1$,\nstrong convexity.\n\\end{proof}","start":47108,"end":48808},{"raw_tex":"\\begin{proof}\nApplying~\\eqref{e-bcv-delta} with $\\tau=\\tau_k$, $\\sigma=\\sigma_k$, $\\delta_1=\\delta^2-\\tau_k L$ and $\\delta_2=1$,\ntogether with the Lipschitz condition~\\eqref{e-bcv-lip},\nwe see that the backtracking condition~\\eqref{e-ls-cond} holds\nat iteration $k$ if $0 < \\delta < 1$ and\n\\[\n\\tau_k \\sigma_k\\|A\\|^2 + \\tau_k L \\leq \\delta^2.\n\\]\nThen mathematical induction can be used to prove~\\eqref{e-ls-param-lbnd}.\nThe two lower bounds~\\eqref{e-ls-param-lbnd} hold at $k=0$\nby the definition of $\\tau_\\mathrm{min}$ and $\\sigma_\\mathrm{min}$.\nNow assume $\\tau_{k-1} \\geq \\tau_\\mathrm{min}$,\n$\\sigma_{k-1} \\geq \\sigma_\\mathrm{min}$,\nand consider the $k$th iteration.\nThe first attempt of $\\theta_k$ is $\\theta_k=\\bar \\theta_k \\geq 1$.\nIf this value is accepted, then\n\\[\n\\tau_k = \\bar\\theta_k \\tau_{k-1} \\geq \\tau_{k-1} \\geq \\tau_\\mathrm{min},\n\\qquad\n\\sigma_k = \\bar\\theta_k \\sigma_{k-1} \\geq \\sigma_{k-1} \\geq\n  \\sigma_\\mathrm{min}.\n\\]\nOtherwise, one or more backtracking steps are needed.\nDenote by $\\tilde \\theta_k$ the last rejected value.\nThen $\\tilde\\theta_k^2\\tau_{k-1}^2\\beta\\|A\\|^2 + \n\\tilde\\theta_k \\tau_{k-1} L >\\delta^2$\nand the accepted $\\theta_k$ satisfies\n\\[\n\\theta_k = \\frac{\\tilde \\theta_k}{2} \\geq\n\\frac{-L+\\sqrt{L^2+4\\delta^2\\beta\\|A\\|^2}}%\n  {4\\tau_{k-1}\\beta\\|A\\|^2}.\n\\]\nTherefore,\n\\[\n\\tau_k=\\theta_k\\tau_{k-1} >\n\\frac{-L+\\sqrt{L^2+4\\delta^2\\beta\\|A\\|^2}}{4\\beta\\|A\\|^2}, \\qquad\n\\sigma_k=\\beta\\tau_k \\geq \\beta\\tau_\\mathrm{min}. \\qedhere\n\\]\n\\end{proof}","start":80409,"end":81883},{"raw_tex":"\\begin{proof}\nThe optimality condition for the primal prox-operator~\\eqref{e-ls-b}\ngives\n\\BEAS\n\\lefteqn{f(x^{(k+1)})-f(x)} \\\\\n&\\leq & \\frac{1}{\\tau_k}\n  \\big(d_\\primal(x,x^{(k)})-d_\\primal(x,x^{(k+1)})-\n  d_\\primal(x^{(k+1)}, x^{(k)})\\big)\n  +\\inprod{A^T\\bar z^{(k+1)}+\\nabla h(x^{(k)})}{x-x^{(k+1)}}\n\\EEAS\nfor all $x\\in \\dom f \\cap \\dom\\phi_\\primal$.\nHence\n\\BEA\n\\lefteqn{f(x^{(k+1)}) + h(x^{(k+1)}) - f(x) - h(x)} \\nonumber \\\\\n  &\\leq & \\frac{1}{\\tau_k} (d_\\primal(x,x^{(k)}) - \n  d_\\primal(x,x^{(k+1)}) - d_\\primal(x^{(k+1)}, x^{(k)}))\n  + \\inprod{A^T\\bar z_{k+1}}{x - x^{(k+1)}} \\nonumber \\\\\n& &\\mbox{} + h(x^{(k+1)})-h(x)+\\inprod{\\nabla h(x^{(k)})}{x-x^{(k+1)}}\n  \\nonumber \\\\\n&\\leq & \\frac{1}{\\tau_k} (d_\\primal(x,x^{(k)}) - \n d_\\primal(x,x^{(k+1)}) - d_\\primal(x^{(k+1)}, x^{(k)}))\n  + \\inprod{A^T\\bar z^{(k+1)}}{x - x^{(k+1)}} \\nonumber \\\\\n& &\\mbox{} +h(x^{(k+1)})-h(x^{(k)})\n  -\\inprod{\\nabla h(x^{(k)})}{x^{(k+1)}-x^{(k)}}.\n  \\label{e-ls-conv-1}\n\\EEA\nThe second inequality follows from the convexity of $h$, \\ie,\n$h(x) \\geq h(x^{(k)}) + \\inprod{\\nabla h(x^{(k)})}{x-x^{(k)}}$.\nThe dual update~\\eqref{e-ls-c} implies that\n\\BEQ \\label{e-ls-conv-2}\n\\inprod{z-z^{(k+1)}}{Ax^{(k+1)}-b}\n= \\frac{1}{\\sigma_k} \\inprod{z-z^{(k+1)}}{z^{(k+1)}-z^{(k)}}\n\\quad \\mbox{for all $z$.}\n\\EEQ\nThis equality at $k=i-1$ is\n\\begin{align}\n\\inprod{z-z^{(i)}}{Ax^{(i)}-b}\n  &= \\frac{1}{\\sigma_{i-1}} \\inprod{z-z^{(i)}}{z^{(i)}-z^{(i-1)}} \n  \\nonumber \\\\\n&= \\frac{1}{2\\sigma_{i-1}} \\left( \\|z-z^{(i-1)}\\|^2\n  - \\|z-z^{(i)}\\|^2 - \\|z^{(i)} - z^{(i-1)} \\|^2 \\right).\n  \\label{e-ls-conv-3}\n\\end{align}\nThe equality~\\eqref{e-ls-conv-2} at $k=i-2$ is\n\\begin{align*}\n\\inprod{z-z^{(i-1)}}{Ax^{(i-1)}-b}\n  &= \\frac{1}{\\sigma_{i-2}} \\inprod{z-z^{(i-1)}}{z^{(i-1)}-z^{(i-2)}} \\\\\n&= \\frac{\\theta_{i-1}}{\\sigma_{i-1}}\n  \\inprod{z-z^{(i-1)}}{z^{(i-1)}-z^{(i-2)}} \\\\\n&= \\frac{1}{\\sigma_{i-1}} \\inprod{z-z^{(i-1)}}{\\bar z^{(i)}-z^{(i-1)}}.\n\\end{align*}\nWe evaluate this at $z=z^{(i)}$ and add it to the equality\nat $z=z^{(i-2)}$ multiplied by $\\theta_{i-1}$:\n\\begin{align}\n\\inprod{z^{(i)}-\\bar z^{(i)}}{Ax^{(i-1)}-b}\n  &= \\frac{1}{\\sigma_{i-1}}\n  \\inprod{z^{(i)}-\\bar z^{(i)}}{\\bar z^{(i)} - z^{(i-1)}} \\nonumber \\\\\n&= \\frac{1}{2\\sigma_{i-1}} \\left(\\|z^{(i)}-z^{(i-1)}\\|^2\n  -\\|z^{(i)}-\\bar z^{(i)}\\|^2-\\|\\bar z^{(i)}-z^{(i-1)}\\|^2 \\right).\n  \\label{e-ls-conv-4}\n\\end{align}\nNow we combine~\\eqref{e-ls-conv-1} for $k=i-1$,\nwith~\\eqref{e-ls-conv-3} and~\\eqref{e-ls-conv-4}. For $i \\geq 1$,\n\\BEAS\n\\lefteqn{\\cL(x^{(i)},z) - \\cL(x, \\bar z^{(i)})} \\nonumber \\\\\n&=& f(x^{(i)})+h(x^{(i)})+\\inprod{z}{Ax^{(i)}-b}\n  -f(x)-h(x)-\\inprod{\\bar z^{(i)}}{Ax-b} \\nonumber \\\\\n&\\leq & \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)})-d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big)\n  + \\inprod{A^T\\bar z^{(i)}}{x-x^{(i)}} + \\inprod{z}{Ax^{(i)}-b} \\nonumber\\\\\n& &\\mbox{} - \\inprod{\\bar z^{(i)}}{Ax-b} + h(x^{(i)}) - h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)}) \n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big)\n  + \\inprod{z-\\bar z^{(i)}}{Ax^{(i)}-b} \\nonumber \\\\\n& &\\mbox{} +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big) \\nonumber \\\\\n& & \\mbox{} +\\inprod{z^{(i)}-\\bar z^{(i)}}{A(x^{(i)}-x^{(i-1)})}\n  + \\inprod{z-z^{(i)}}{Ax^{(i)}-b}\n  + \\inprod{z^{(i)}-\\bar z^{(i)}}{Ax^{(i-1)}-b} \\nonumber \\\\\n& &\\mbox{} +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)}))\\Big) \\nonumber \\\\\n& &\\mbox{} +\\frac{1}{2\\sigma_{i-1}}\n  \\Big(\\|z-z^{(i-1)}\\|^2-\\|z-z^{(i)}\\|^2-\\|\\bar z^{(i)}-z^{(i-1)}\\|^2\n  -\\|\\bar z^{(i)}-z^{(i)}\\|^2\\Big) \\nonumber \\\\\n& &\\mbox{}+\\inprod{A^T(z^{(i)}-\\bar z^{(i)})}{x^{(i)}-x^{(i-1)}}\n  +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)})} \\\\\n&\\leq & \\frac{1}{\\tau_{i-1}} \n  \\left(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)}) -\n  (1-\\delta^2) d_\\primal(x^{(i)},x^{(i-1)})\\right) \\nonumber \\\\\n& &\\mbox{} + \\frac{1}{2\\sigma_{i-1}} \\left(\n  \\|z-z^{(i-1)}\\|^2 - \\|z-z^{(i)}\\|^2 - \n  \\|\\bar z^{(i)} - z^{(i-1)}\\|^2 \\right),\n\\EEAS\nwhich is the desired result~\\eqref{e-ls-conv-i}.\nThe first inequality follows from~\\eqref{e-ls-conv-1}.\nIn the second last step we substitute~\\eqref{e-ls-conv-3}\nand~\\eqref{e-ls-conv-4}.\nThe last step uses the line search exit condition~\\eqref{e-ls-cond}\nat $k=i-1$.\n\\end{proof}","start":82509,"end":87115},{"raw_tex":"\\begin{proof}\nFrom~\\eqref{e-ls-conv-i},\n\\[\n\\cL(x^{(i)},z)-\\cL(x, \\bar z^{(i)}) \\leq \\frac{1}{\\tau_{i-1}}\n\\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n+ \\frac{1}{2\\beta} \\|z-z^{(i-1)}\\|^2\n- \\frac{1}{2\\beta} \\|z-z^{(i)}\\|^2 \\Big).\n\\]\nSince $\\cL(u,v)$ is convex in $u$ and affine in $v$,\n\\BEA\n\\lefteqn{(\\sum_{i=1}^k \\tau_{i-1}) \n  \\big(\\cL(x^{(k)}_\\avg, z)-\\cL(x, \\bar z^{(k)}_\\avg)\\big)} \\nonumber \\\\\n&\\leq &\\sum_{i=1}^k \\tau_{i-1} (\\cL(x^{(i)}, z)-\\cL(x, \\bar z^{(i)})) \n  \\nonumber \\\\\n&\\leq & d_\\primal(x,x^{(0)}) - d_\\primal(x,x^{(k)}) +\n  \\frac{1}{2\\beta} (\\|z-z^{(0)}\\|^2 - \\|z-z^{(k)}\\|^2) \\nonumber \\\\\n&\\leq & d_\\primal(x,x^{(0)}) + \\frac{1}{2\\beta} \\|z-z^{(0)}\\|^2.\n  \\label{e-ls-conv-k}\n\\EEA\nDividing by $\\sum_{i=1}^k \\tau_{i-1}$ gives~\\eqref{e-ls-conv}.\n\\end{proof}","start":88061,"end":88840}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["e-prob"],"declared_external":false,"source_only":false,"title":"复合凸优化问题","source_units":[{"raw_tex":"We discuss proximal splitting methods for optimization problems \nin the form\n\\begin{equation} \\label{e-prob}\n  \\mbox{minimize} \\quad f(x) + g(Ax) + h(x),\n\\end{equation}\nwhere $f$, $g$, and $h$ are convex functions, and $h$ is differentiable.\n","start":2548,"end":2790}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"内积与范数记号","source_units":[{"raw_tex":"We use the notation $\\inprod{x}{y} = x^Ty$ for the\nstandard inner product of vectors $x$ and $y$, and\n$\\|x\\| = \\inprod{x}{x}^{1/2}$ for the Euclidean norm of a vector\n$x$.  Other norms will be distinguished by a subscript.\n\n\\subsection{Problem formulation}\n\n","start":8657,"end":8915}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"函数与维数条件","source_units":[{"raw_tex":"In~\\eqref{e-prob} the vector $x$ is an $n$-vector and\n$A$ is an $m\\times n$ matrix.  The functions $f$, $g$, $h$\nare closed and convex, and $h$ is differentiable, \\ie,  \n\\[\nh(x) \\geq h(x') + \\inprod{\\nabla h(x')}{x-x'} \\quad\n\\mbox{for all $x,x' \\in \\dom h$},\n\\]\nwhere $\\dom h$ is an open convex set.  We assume that $f+h$ and $g$\nare proper, \\ie, have nonempty domains.\n","start":8915,"end":9285}]},{"id":"S004","semantic_type":"problem_setting","kind":"problem_setting","labels":["e-prob-dual"],"declared_external":false,"source_only":false,"title":"一般对偶问题","source_units":[{"raw_tex":"The dual of problem~\\eqref{e-prob} is\n\\BEQ \\label{e-prob-dual}\n \\mbox{maximize}\\quad  {-(f+h)^*(-A^Tz) - g^*(z)},\n\\EEQ\n","start":10003,"end":10122}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"凸共轭定义","source_units":[{"raw_tex":"where $(f+h)^*$ and $g^*$ are the conjugates of $f+h$ and $g$:\n\\[\n (f+h)^*(w) = \\sup_x{(\\inprod{w}{x} - f(x) - h(x))}, \\qquad\n g^*(z) = \\sup_y{(\\inprod{z}{y} - g(y))}.\n\\]\n","start":10122,"end":10293}]},{"id":"S006","semantic_type":"derived_fact","kind":"derived_fact","labels":["e-opt-cond"],"declared_external":false,"source_only":false,"title":"原始对偶最优性条件","source_units":[{"raw_tex":"The primal--dual optimality conditions for~\\eqref{e-prob}\nand~\\eqref{e-prob-dual} are\n\\[\n 0 \\in \\partial f(x) + \\nabla h(x) + A^T z, \\qquad\n 0 \\in \\partial g^*(z) - Ax.\n\\]\nHere $\\partial f$ and $\\partial g^*$ are the subdifferentials of\n$f$ and  $g^*$.  We often write the optimality conditions as\n\\begin{equation} \\label{e-opt-cond}\n  0 \\in \\begin{bmatrix} 0 & A^T \\\\ -A & 0 \\end{bmatrix}\n  \\begin{bmatrix} x \\\\ z \\end{bmatrix} + \\begin{bmatrix}\n    \\partial f(x)+\\nabla h(x) \\\\ \\partial g^\\ast(z)\n\\end{bmatrix}.\n\\end{equation}\n","start":10457,"end":10986}]},{"id":"S007","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"最优性系统可解","source_units":[{"raw_tex":"Throughout the paper, we assume that the optimality \nconditions~\\eqref{e-opt-cond} are solvable.\n\n","start":10986,"end":11084}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"一般拉格朗日函数","source_units":[{"raw_tex":"We will refer to the convex--concave function\n\\[\n\\cL(x,z) = f(x)+h(x)+ \\inprod{z}{Ax} -g^\\ast(z)\n\\]\nas the \\emph{Lagrangian} of~\\eqref{e-prob}.\nWe follow the convention that $\\cL(x,z) = +\\infty$ if $x\\not\\in \\dom(f+h)$\nand $\\cL(x,z) = -\\infty$ if $x\\in\\dom(f+h)$ and $z\\not\\in \\dom g^*$.\n","start":11084,"end":11372}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"Bregman核与距离","source_units":[{"raw_tex":"Let $\\phi$ be a convex function with a domain that has nonempty\ninterior, and assume $\\phi$ is continuous on $\\dom \\phi$ and \ncontinuously differentiable on $\\intr(\\dom \\phi)$.\nThe \\emph{generalized distance} (or \\emph{Bregman distance})\ngenerated by the \\emph{kernel function} $\\phi$ is defined as the function\n\\[\nd(x,y) = \\phi(x)-\\phi(y)-\\inprod{\\nabla \\phi(y)}{x-y},\n\\]\nwith domain $\\dom d=\\dom \\phi \\times \\intr(\\dom \\phi)$.\n","start":37479,"end":37908}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":["e-prox-opt","e-prox-opt2"],"declared_external":false,"source_only":false,"title":"Bregman近端映射","source_units":[{"raw_tex":"The corresponding \\emph{Bregman proximal operator} of a function $f$ is \n\\begin{align}\n\\prox_f^\\phi(y,a)\n  &= \\argmin_x{(f(x)+\\inprod{a}{x}+d(x,y))} \\label{e-prox-opt} \\\\\n&= \\argmin_x{(f(x)+\\inprod{a}{x}+\\phi(x)\n   -\\inprod{\\nabla \\phi(y)}{x})}. \\label{e-prox-opt2}\n\\end{align}\n","start":37908,"end":38186}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"近端更新适定性","source_units":[{"raw_tex":"It is assumed that for every $a$ and every $y \\in \\intr(\\dom\\phi)$\nthe minimizer $\\hat x=\\prox_f^\\phi(y,a)$ is unique and in\n$\\intr(\\dom \\phi)$.\n","start":38186,"end":38331}]},{"id":"S012","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"近端次梯度条件","source_units":[{"raw_tex":"From the expression~\\eqref{e-prox-opt2} we see that \n$\\hat x=\\prox_f^\\phi(y,a)$\nsatisfies\n\\[\n\\nabla \\phi(y) - \\nabla \\phi(\\hat x) - a \\in \\partial f(\\hat x).\n\\]\n","start":39609,"end":39770}]},{"id":"S013","semantic_type":"derived_fact","kind":"derived_fact","labels":["e-prox-cond"],"declared_external":false,"source_only":false,"title":"近端三点不等式","source_units":[{"raw_tex":"Equivalently, by definition of subgradient,\n\\begin{align}\nf(x) + \\inprod{a}{x}\n&\\geq f(\\hat x)+\\inprod{a}{\\hat x}\n  +\\inprod{\\nabla\\phi(y)-\\nabla\\phi(\\hat x)}{x-\\hat x} \\nonumber \\\\\n&= f(\\hat x)+\\inprod{a}{\\hat x}+d(\\hat x, y)+d(x,\\hat x)-d(x,y)\n  \\label{e-prox-cond}\n\\end{align}\nfor all $x\\in \\dom f \\cap \\dom \\phi$.\n","start":39770,"end":40088}]},{"id":"S014","semantic_type":"assumption","kind":"assumption","labels":["e-bcv-str-cvx"],"declared_external":false,"source_only":false,"title":"核的一致强凸性","source_units":[{"raw_tex":"\\paragraph{Assumptions} \nThroughout Section~\\ref{s-bcv} we make the following assumptions.\nThe kernel functions $\\phi_\\primal$ and $\\phi_\\dual$ are \n1-strongly convex with respect to norms $\\|\\cdot\\|_\\primal$ and \n$\\|\\cdot\\|_\\dual$, respectively:\n\\begin{equation} \\label{e-bcv-str-cvx}\nd_\\primal(x,x^\\prime) \\geq \\frac{1}{2}\\|x-x^\\prime\\|^2_\\primal, \\qquad\nd_\\dual(z,z^\\prime) \\geq \\frac{1}{2}\\|z-z^\\prime\\|^2_\\dual\n\\end{equation}\nfor all $(x,x') \\in \\dom d_\\primal$ and $(z,z') \\in \\dom d_\\dual$.\nThe assumption that the strong convexity constants are equal to one\ncan be made without loss of generality, by scaling the norms (or distances)\nif needed.\n","start":42221,"end":42874}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","labels":["e-bcv-lip"],"declared_external":false,"source_only":false,"title":"相对光滑性","source_units":[{"raw_tex":"We also assume that the function $L\\phi_\\primal-h$ is convex for some \n$L>0$.  More precisely, $\\dom \\phi_\\primal \\subseteq \\dom h$ and\n\\BEQ \\label{e-bcv-lip}\nh(x)-h(x^\\prime)-\\inprod{\\nabla h(x^\\prime)}{x-x^\\prime}\n\\leq Ld_\\primal (x,x^\\prime) \\quad \\mbox{for all\\ }\n(x,x^\\prime) \\in \\dom d_\\primal.\n\\EEQ\nNote that this assumption is looser than\nthe one in~\\cite[Equation~(4)]{ChP:16}.\n","start":42874,"end":43261}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":["e-norm"],"declared_external":false,"source_only":false,"title":"原始对偶矩阵范数","source_units":[{"raw_tex":"We denote by $\\|A\\|$ the matrix norm\n\\BEQ \\label{e-norm}\n\\|A\\|=\\sup_{u \\neq 0, v \\neq 0}\n    \\frac{\\inprod{v}{Au}}{\\|v\\|_\\dual \\|u\\|_\\primal}\n = \\sup_{u\\neq 0} \\frac{\\|Au\\|_{\\dual,*}}{\\|u\\|_\\primal}\n = \\sup_{v\\neq 0} \\frac{\\|A^Tv\\|_{\\primal,*}}{\\|v\\|_\\dual},\n\\EEQ\nwhere $\\|\\cdot\\|_{\\primal,*}$ and $\\|\\cdot\\|_{\\dual,*}$ are\nthe dual norms of $\\|\\cdot\\|_\\primal$ and $\\|\\cdot\\|_\\dual$. \n","start":43261,"end":43647}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"核域内存在最优解","source_units":[{"raw_tex":"\nIt is also assumed that the primal--dual optimality \nconditions~\\eqref{e-opt-cond} have a solution $(x^\\star, z^\\star)$\nwith $x^\\star \\in \\dom \\phi_\\primal$ and \n$z^\\star \\in \\dom \\phi_\\dual$.  \n","start":43647,"end":43843}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"四种耦合核","source_units":[{"raw_tex":"We introduce four possible primal--dual kernel functions: the functions\n\\[\n\\phi_+(x,z) = \\frac{1}{\\tau}\\phi_\\primal(x)\n+\\frac{1}{\\sigma}\\phi_\\dual(z)+\\inprod{z}{Ax}, \\qquad\n\\phi_-(x,z) = \\frac{1}{\\tau}\\phi_\\primal(x)\n+\\frac{1}{\\sigma}\\phi_\\dual(z)-\\inprod{z}{Ax},\n\\]\nwhere $\\sigma,\\tau > 0$, and the functions \n\\[\n \\phi_\\mathrm{dcv}(x,z) = \\phi_+(x,z) - h(x), \\qquad \n \\phi_\\mathrm{pcv}(x,z) = \\phi_-(x,z) - h(x).\n\\]\nThe subscripts in $\\phi_+$ and $\\phi_-$ \nrefer to the sign of the inner product term $\\inprod{z}{Ax}$.\nThe subscripts in $\\phi_\\mathrm{pcv}$ and $\\phi_\\mathrm{dcv}$\nindicate the algorithm (Bregman primal or dual Condat-V\\~u) for which\nthese distances will be relevant. \n","start":45215,"end":45902}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":["e-bcv-dminus"],"declared_external":false,"source_only":false,"title":"正负耦合距离","source_units":[{"raw_tex":"If these kernel functions are convex,\nthey generate the following Bregman distances.\nThe distances generated by $\\phi_+$ and $\\phi_-$ are\n\\begin{align}\nd_+ (x,z;x^\\prime,z^\\prime) &= \n  \\frac{1}{\\tau}d_\\primal(x,x^\\prime)+\\frac{1}{\\sigma}d_\\dual(z,z^\\prime)\n  +\\inprod{z-z^\\prime}{A(x-x^\\prime)} \\nonumber \\\\\nd_-(x,z;x^\\prime,z^\\prime) &=\n  \\frac{1}{\\tau}d_\\primal(x,x^\\prime)+\\frac{1}{\\sigma}d_\\dual(z,z^\\prime)\n  -\\inprod{z-z^\\prime}{A(x-x^\\prime)}, \\label{e-bcv-dminus}\n\\end{align}\n","start":45902,"end":46387}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"减去光滑项的距离","source_units":[{"raw_tex":"respectively, and the distances generated by $\\phi_\\mathrm{dcv}$ and \n$\\phi_\\mathrm{pcv}$ are  \n\\begin{align*}\nd_\\mathrm{dcv}(x,z;x^\\prime,z^\\prime) &= \n  d_+ (x,z;x^\\prime,z^\\prime) \n  -h(x)+h(x^\\prime)+\\inprod{\\nabla h(x')}{x-x^\\prime} \\\\\nd_\\mathrm{pcv}(x,z;x^\\prime,z^\\prime) &= \n  d_- (x,z;x^\\prime,z^\\prime) \n  -h(x)+h(x^\\prime)+\\inprod{\\nabla h(x')}{x-x^\\prime}.\n\\end{align*}\n","start":46387,"end":46769}]},{"id":"S021","semantic_type":"problem_setting","kind":"problem_setting","labels":["e-lcp-prob"],"declared_external":false,"source_only":false,"title":"等式约束专门问题","source_units":[{"raw_tex":"The algorithm is restricted to problems in the equality constrained\nform\n\\begin{equation} \\label{e-lcp-prob}\n \\begin{array}{ll}\n   \\mbox{minimize} & f(x) + h(x) \\\\\n   \\mbox{subject to} & Ax=b.\n \\end{array}\n\\end{equation}\nThis is a special case of~\\eqref{e-prob} with $g=\\delta_{\\{b\\}}$,\nthe indicator function of the singleton $\\{b\\}$.\n","start":76128,"end":76464}]},{"id":"S022","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"对偶空间采用欧氏核","source_units":[{"raw_tex":"\\paragraph{Assumptions}\nWe make the same assumptions as in Section~\\ref{s-bcv-ppm}, but define\n\\[\n\\phi_\\dual (z) = \\frac{1}{2} \\|z\\|^2, \\qquad\nd_\\dual (z,z') = \\frac{1}{2} \\|z-z'\\|^2, \\qquad\n\\|z\\|_\\dual = \\|z\\|,\n\\]\nwhere $\\|\\cdot\\|$ is the Euclidean norm.\n","start":76744,"end":77000}]},{"id":"S023","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"线搜索中的矩阵范数","source_units":[{"raw_tex":"The matrix norm $\\|A\\|$ is defined accordingly as\n\\[\n\\|A\\|=\\sup_{u \\neq 0, v \\neq 0}\n  \\frac{\\inprod{v}{Au}}{\\|v\\| \\|u\\|_\\primal}\n= \\sup_{u \\neq 0} \\frac{\\|Au\\|}{\\|u\\|_\\primal}\n= \\sup_{v \\neq 0} \\frac{\\|A^Tv\\|_{\\primal,*}}{\\|v\\|}.\n\\]\n","start":77000,"end":77234}]},{"id":"S024","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"迭代初值","source_units":[{"raw_tex":"The algorithm uses the following iteration,\nwith starting points $x^{(0)} \\in \\intr(\\dom \\phi_\\primal) \\cap \\dom h$\nand $z^{(-1)} = z^{(0)}$:\n\\begin{subequations} \\label{e-ls}\n","start":77277,"end":77453}]},{"id":"S025","semantic_type":"algorithm","kind":"algorithm","labels":["e-ls-a","e-ls-b","e-ls-c"],"declared_external":false,"source_only":false,"title":"对偶外推更新","source_units":[{"raw_tex":"\\begin{align}\n\\bar z^{(k+1)} &= z^{(k)} + \\theta_k (z^{(k)} - z^{(k-1)}) \n  \\label{e-ls-a} \\\\\n","start":77453,"end":77547}]},{"id":"S026","semantic_type":"algorithm","kind":"algorithm","labels":["e-ls-a","e-ls-b","e-ls-c"],"declared_external":false,"source_only":false,"title":"原始近端更新","source_units":[{"raw_tex":"x^{(k+1)} &= \\prox_{\\tau_k f}^{\\phi_\\primal} \\big(\n  x^{(k)}, \\tau_k (A^T\\bar z^{(k+1)}+\\nabla h(x^{(k)}))\n  \\big) \\label{e-ls-b} \\\\\n","start":77547,"end":77680}]},{"id":"S027","semantic_type":"algorithm","kind":"algorithm","labels":[],"declared_external":false,"source_only":false,"title":"对偶残差更新","source_units":[{"raw_tex":"z^{(k+1)} &= z^{(k)} + \\sigma_k (Ax^{(k+1)}-b). \\label{e-ls-c}\n\\end{align}\n\\end{subequations}\n","start":77680,"end":77774}]},{"id":"S028","semantic_type":"algorithm","kind":"algorithm","labels":[],"declared_external":false,"source_only":false,"title":"回溯参数生成","source_units":[{"raw_tex":"In the line search algorithm, the parameters \n$\\theta_k$, $\\tau_k$, $\\sigma_k$ are determined by a\nbacktracking search.\nAt the start of the algorithm, we set $\\tau_{-1}$ and $\\sigma_{-1}$ to\nsome positive values.\nTo start the search in iteration $k$ we choose\n$\\bar\\theta_k \\geq 1$.  For $i=0,1,2,\\ldots$,\nwe set $\\theta_k = 2^{-i}\\bar\\theta_k$,\n$\\tau_k=\\theta_k\\tau_{k-1}$, $\\sigma_k=\\theta_k\\sigma_{k-1}$,\nand compute $\\bar z_{k+1}$, $x_{k+1}$, $z_{k+1}$ using~\\eqref{e-ls}.\n","start":78232,"end":78709}]},{"id":"S029","semantic_type":"algorithm","kind":"algorithm","labels":["e-ls-cond"],"declared_external":false,"source_only":false,"title":"回溯接受不等式","source_units":[{"raw_tex":"For some $\\delta \\in (0,1]$, if\n\\BEA\n& & \\inprod{z^{(k+1)}-\\bar z^{(k+1)}}{A(x^{(k+1)}-x^{(k)})}\n  +h(x^{(k+1)})-h(x^{(k)})-\\inprod{\\nabla h(x^{(k)})}{x^{(k+1)}-x^{(k)}}\n  \\nonumber \\\\\n&\\leq & \\frac{\\delta^2}{\\tau_k} d_\\primal(x^{(k+1)}, x^{(k)})\n    + \\frac{1}{2\\sigma_k} \\|\\bar z^{(k+1)} - z^{(k+1)}\\|^2,\n    \\label{e-ls-cond}\n\\EEA\n","start":78709,"end":79043}]},{"id":"S030","semantic_type":"algorithm","kind":"algorithm","labels":[],"declared_external":false,"source_only":false,"title":"接受与减半规则","source_units":[{"raw_tex":"we accept the computed iterates $\\bar z^{(k+1)}$, $x^{(k+1)}$, $z^{(k+1)}$\nand parameters $\\theta_k$, $\\sigma_k$, $\\tau_k$, and terminate the \nbacktracking search.\nIf~\\eqref{e-ls-cond} does not hold, we increment $i$ and continue\nthe backtracking search.\n","start":79043,"end":79298}]},{"id":"S031","semantic_type":"lemma","kind":"lemma","labels":["e-ls-param-lbnd"],"declared_external":false,"source_only":false,"title":"步长统一正下界","source_units":[{"raw_tex":"\\BEQ \\label{e-ls-param-lbnd}\n  \\tau_k \\geq \\tau_\\mathrm{min} \\triangleq \\min{\\Big\\{\\tau_{-1},\n  \\frac{-L+\\sqrt{L^2+4\\delta^2\\beta\\|A\\|^2}}{4\\beta\\|A\\|^2}\\Big\\}},\n  \\qquad\n  \\sigma_k \\geq \\sigma_\\mathrm{min} \\triangleq \\beta\\tau_\\mathrm{min},\n\\EEQ","start":80009,"end":80255}]},{"id":"S032","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"初始步长比","source_units":[{"raw_tex":"where $\\beta=\\sigma_{-1}/\\tau_{-1}$.\n","start":80256,"end":80293}]},{"id":"S033","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"回溯终止的作者说明","source_units":[{"raw_tex":"The lower bounds imply that the backtracking eventually terminates\nwith positive stepsizes $\\sigma_k$ and $\\tau_k$.\n","start":80293,"end":80409}]},{"id":"S034","semantic_type":"lemma","kind":"lemma","labels":["e-ls-conv-i"],"declared_external":false,"source_only":false,"title":"线搜索单步不等式","source_units":[{"raw_tex":"\\begin{align}\n\\cL(x^{(k+1)},z)-\\cL(x, \\bar z^{(k+1)})\n  &\\leq \\frac{1}{\\tau_k} \n  \\left(d_\\primal(x,x^{(k)}) - d_\\primal(x,x^{(k+1)}) -\n  (1-\\delta^2) d_\\primal(x^{(k+1)},x^{(k)})\\right) \\nonumber \\\\\n&\\phantom{=} + \\frac{1}{2\\sigma_k} \\left(\n  \\|z-z^{(k)}\\|^2 - \\|z-z^{(k+1)}\\|^2 - \n  \\|\\bar z^{(k+1)} - z^{(k)}\\|^2 \\right) \\label{e-ls-conv-i}\n\\end{align}","start":82048,"end":82403}]},{"id":"S035","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":true,"title":"单步界的比较点范围","source_units":[{"raw_tex":"for all $x \\in \\dom f \\cap \\dom \\phi_\\primal$ and all $z$.\n","start":82404,"end":82463}]},{"id":"S036","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":true,"title":"等式约束拉格朗日函数","source_units":[{"raw_tex":"Here $\\cL(x,z)=f(x)+h(x)+\\inprod{z}{Ax-b}$.  \n","start":82463,"end":82509}]},{"id":"S037","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"步长加权遍历平均","source_units":[{"raw_tex":"We define the averaged primal and dual sequences\n\\[\nx^{(k)}_\\avg = \\frac{1}{\\sum_{i=1}^k \\tau_{i-1}}\n\\sum_{i=1}^k \\tau_{i-1}x^{(i)}, \\qquad\n\\bar z^{(k)}_\\avg = \\frac{1}{\\sum_{i=1}^k \\tau_{i-1}}\n\\sum_{i=1}^k \\tau_{i-1} \\bar z^{(i)}\n\\]\nfor $k \\geq 1$.\n","start":87154,"end":87404}]},{"id":"S038","semantic_type":"proposition","kind":"proposition","labels":["e-bcv-param"],"declared_external":false,"source_only":true,"title":"耦合核的凸性条件","source_units":[{"raw_tex":"We now show that $\\phi_+$ and $\\phi_-$ are convex if\n\\[\n\\sigma\\tau \\|A\\|^2 \\leq 1 \n\\]\nand strongly convex if $\\sigma\\tau\\|A\\|^2 < 1$,\nand that the functions $\\phi_\\mathrm{dcv}$ and \n$\\phi_\\mathrm{pcv}$ are convex if\n\\BEQ \\label{e-bcv-param}\n\\sigma\\tau \\|A\\|^2 + \\tau L \\leq 1\n\\EEQ\nand strongly convex if $\\sigma\\tau \\|A\\|^2 + \\tau L < 1$.\n","start":46769,"end":47108}]},{"id":"S039","semantic_type":"lemma","kind":"lemma","labels":["e-bcv-delta"],"declared_external":false,"source_only":false,"title":"耦合交叉项控制","source_units":[{"raw_tex":"Suppose $\\sigma\\tau \\|A\\|^2 \\leq \\delta_1 \\delta_2$\nwith $\\delta_1,\\delta_2 > 0$.\nThen~\\eqref{e-bcv-str-cvx} and the arithmetic--geometric mean inequality\nimply that\n\\begin{align}\n\\left| \\inprod{z-z^\\prime}{A(x-x^\\prime)}\\right|\n  &\\leq \\|A\\| \\|z-z'\\|_\\dual \\|x-x'\\|_\\primal  \\nonumber \\\\\n&\\leq \\sqrt{\\frac{\\delta_1 \\delta_2}{\\sigma \\tau}}\n\\|z-z'\\|_\\dual \\|x-x'\\|_\\primal \\nonumber \\\\\n&\\leq \\frac{\\delta_1}{2\\tau} \\|x-x'\\|_\\primal^2  +\n  \\frac{\\delta_2}{2\\sigma} \\|z-z'\\|_\\primal^2 \\nonumber \\\\\n&\\leq \\frac{\\delta_1}{\\tau} d_\\primal (x,x') +\n  \\frac{\\delta_2}{\\sigma} d_\\dual (z,z'). \\label{e-bcv-delta}\n\\end{align}\n","start":47236,"end":47852}]},{"id":"S040","semantic_type":"proof","kind":"proof","labels":["e-bcv-delta"],"declared_external":false,"source_only":false,"title":"耦合核凸性的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nTo show that the kernel functions $\\phi_+$ and $\\phi_-$ are convex,\nwe show that $d_+$ and $d_-$ are nonnegative.\nSuppose $\\sigma\\tau \\|A\\|^2 \\leq \\delta_1 \\delta_2$\nwith $\\delta_1,\\delta_2 > 0$.\nThen~\\eqref{e-bcv-str-cvx} and the arithmetic--geometric mean inequality\nimply that\n\\begin{align}\n\\left| \\inprod{z-z^\\prime}{A(x-x^\\prime)}\\right|\n  &\\leq \\|A\\| \\|z-z'\\|_\\dual \\|x-x'\\|_\\primal  \\nonumber \\\\\n&\\leq \\sqrt{\\frac{\\delta_1 \\delta_2}{\\sigma \\tau}}\n\\|z-z'\\|_\\dual \\|x-x'\\|_\\primal \\nonumber \\\\\n&\\leq \\frac{\\delta_1}{2\\tau} \\|x-x'\\|_\\primal^2  +\n  \\frac{\\delta_2}{2\\sigma} \\|z-z'\\|_\\primal^2 \\nonumber \\\\\n&\\leq \\frac{\\delta_1}{\\tau} d_\\primal (x,x') +\n  \\frac{\\delta_2}{\\sigma} d_\\dual (z,z'). \\label{e-bcv-delta}\n\\end{align}\nTherefore,\n\\begin{align*}\nd_{\\pm} (x,z; x',z')\n  &= \\frac{1}{\\tau} d_\\primal(x,x^\\prime) + \\frac{1}{\\sigma}\n  d_\\dual(z,z^\\prime) \\pm \\inprod{z-z^\\prime}{A(x-x^\\prime)} \\\\\n&\\geq \\frac{1-\\delta_1}{\\tau} d_\\primal (x,x') +\n  \\frac{1-\\delta_2}{\\sigma} d_\\dual (z,z') \\\\\n&\\geq \\frac{1-\\delta_1}{2\\tau} \\|x-x'\\|_\\primal^2 +\n  \\frac{1-\\delta_2}{2\\sigma} \\|z-z'\\|_\\dual^2.\n\\end{align*}\nWith $\\delta_1=\\delta_2=1$, this shows convexity of $\\phi_+$ and $\\phi_-$;\nwith $\\delta_1=\\delta_2<1$, strong convexity.\nSimilarly,\n\\begin{align*}\nd_\\mathrm{dcv/pcv} (x,z; x', z')\n  &= d_{\\pm} (x,z; x',z') - h(x) + h(x') + \\inprod{\\nabla h(x')}{x-x'} \\\\\n&\\geq \\frac{1-\\tau L-\\delta_1}{\\tau} d_\\primal (x,x^\\prime) +\n  \\frac{1-\\delta_2}{\\sigma} d_\\dual (z,z^\\prime).\n\\end{align*}\nWith $\\delta_1=1-\\tau L$ and $\\delta_2=1$, this shows convexity of\n$\\phi_\\mathrm{pcv}$ and $\\phi_\\mathrm{dcv}$;\nwith $\\delta_1=\\delta-\\tau L$ and $\\delta_2=\\delta < 1$,\nstrong convexity.\n\\end{proof}","start":47108,"end":48808}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":["e-bcv-delta"],"declared_external":false,"source_only":false,"title":"范数与算术几何均值估计","source_units":[{"raw_tex":"\\begin{proof}\nTo show that the kernel functions $\\phi_+$ and $\\phi_-$ are convex,\nwe show that $d_+$ and $d_-$ are nonnegative.\nSuppose $\\sigma\\tau \\|A\\|^2 \\leq \\delta_1 \\delta_2$\nwith $\\delta_1,\\delta_2 > 0$.\nThen~\\eqref{e-bcv-str-cvx} and the arithmetic--geometric mean inequality\nimply that\n\\begin{align}\n\\left| \\inprod{z-z^\\prime}{A(x-x^\\prime)}\\right|\n  &\\leq \\|A\\| \\|z-z'\\|_\\dual \\|x-x'\\|_\\primal  \\nonumber \\\\\n&\\leq \\sqrt{\\frac{\\delta_1 \\delta_2}{\\sigma \\tau}}\n\\|z-z'\\|_\\dual \\|x-x'\\|_\\primal \\nonumber \\\\\n&\\leq \\frac{\\delta_1}{2\\tau} \\|x-x'\\|_\\primal^2  +\n  \\frac{\\delta_2}{2\\sigma} \\|z-z'\\|_\\primal^2 \\nonumber \\\\\n&\\leq \\frac{\\delta_1}{\\tau} d_\\primal (x,x') +\n  \\frac{\\delta_2}{\\sigma} d_\\dual (z,z'). \\label{e-bcv-delta}\n\\end{align}\n","start":47108,"end":47852}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"正负核的距离下界","source_units":[{"raw_tex":"Therefore,\n\\begin{align*}\nd_{\\pm} (x,z; x',z')\n  &= \\frac{1}{\\tau} d_\\primal(x,x^\\prime) + \\frac{1}{\\sigma}\n  d_\\dual(z,z^\\prime) \\pm \\inprod{z-z^\\prime}{A(x-x^\\prime)} \\\\\n&\\geq \\frac{1-\\delta_1}{\\tau} d_\\primal (x,x') +\n  \\frac{1-\\delta_2}{\\sigma} d_\\dual (z,z') \\\\\n&\\geq \\frac{1-\\delta_1}{2\\tau} \\|x-x'\\|_\\primal^2 +\n  \\frac{1-\\delta_2}{2\\sigma} \\|z-z'\\|_\\dual^2.\n\\end{align*}\nWith $\\delta_1=\\delta_2=1$, this shows convexity of $\\phi_+$ and $\\phi_-$;\nwith $\\delta_1=\\delta_2<1$, strong convexity.\n","start":47852,"end":48352}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"修正核的距离下界","source_units":[{"raw_tex":"Similarly,\n\\begin{align*}\nd_\\mathrm{dcv/pcv} (x,z; x', z')\n  &= d_{\\pm} (x,z; x',z') - h(x) + h(x') + \\inprod{\\nabla h(x')}{x-x'} \\\\\n&\\geq \\frac{1-\\tau L-\\delta_1}{\\tau} d_\\primal (x,x^\\prime) +\n  \\frac{1-\\delta_2}{\\sigma} d_\\dual (z,z^\\prime).\n\\end{align*}\nWith $\\delta_1=1-\\tau L$ and $\\delta_2=1$, this shows convexity of\n$\\phi_\\mathrm{pcv}$ and $\\phi_\\mathrm{dcv}$;\nwith $\\delta_1=\\delta-\\tau L$ and $\\delta_2=\\delta < 1$,\nstrong convexity.\n\\end{proof}","start":48352,"end":48808}]},{"id":"S044","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"步长下界的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nApplying~\\eqref{e-bcv-delta} with $\\tau=\\tau_k$, $\\sigma=\\sigma_k$, $\\delta_1=\\delta^2-\\tau_k L$ and $\\delta_2=1$,\ntogether with the Lipschitz condition~\\eqref{e-bcv-lip},\nwe see that the backtracking condition~\\eqref{e-ls-cond} holds\nat iteration $k$ if $0 < \\delta < 1$ and\n\\[\n\\tau_k \\sigma_k\\|A\\|^2 + \\tau_k L \\leq \\delta^2.\n\\]\nThen mathematical induction can be used to prove~\\eqref{e-ls-param-lbnd}.\nThe two lower bounds~\\eqref{e-ls-param-lbnd} hold at $k=0$\nby the definition of $\\tau_\\mathrm{min}$ and $\\sigma_\\mathrm{min}$.\nNow assume $\\tau_{k-1} \\geq \\tau_\\mathrm{min}$,\n$\\sigma_{k-1} \\geq \\sigma_\\mathrm{min}$,\nand consider the $k$th iteration.\nThe first attempt of $\\theta_k$ is $\\theta_k=\\bar \\theta_k \\geq 1$.\nIf this value is accepted, then\n\\[\n\\tau_k = \\bar\\theta_k \\tau_{k-1} \\geq \\tau_{k-1} \\geq \\tau_\\mathrm{min},\n\\qquad\n\\sigma_k = \\bar\\theta_k \\sigma_{k-1} \\geq \\sigma_{k-1} \\geq\n  \\sigma_\\mathrm{min}.\n\\]\nOtherwise, one or more backtracking steps are needed.\nDenote by $\\tilde \\theta_k$ the last rejected value.\nThen $\\tilde\\theta_k^2\\tau_{k-1}^2\\beta\\|A\\|^2 + \n\\tilde\\theta_k \\tau_{k-1} L >\\delta^2$\nand the accepted $\\theta_k$ satisfies\n\\[\n\\theta_k = \\frac{\\tilde \\theta_k}{2} \\geq\n\\frac{-L+\\sqrt{L^2+4\\delta^2\\beta\\|A\\|^2}}%\n  {4\\tau_{k-1}\\beta\\|A\\|^2}.\n\\]\nTherefore,\n\\[\n\\tau_k=\\theta_k\\tau_{k-1} >\n\\frac{-L+\\sqrt{L^2+4\\delta^2\\beta\\|A\\|^2}}{4\\beta\\|A\\|^2}, \\qquad\n\\sigma_k=\\beta\\tau_k \\geq \\beta\\tau_\\mathrm{min}. \\qedhere\n\\]\n\\end{proof}","start":80409,"end":81883}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"保证回溯接受的充分条件","source_units":[{"raw_tex":"\\begin{proof}\nApplying~\\eqref{e-bcv-delta} with $\\tau=\\tau_k$, $\\sigma=\\sigma_k$, $\\delta_1=\\delta^2-\\tau_k L$ and $\\delta_2=1$,\ntogether with the Lipschitz condition~\\eqref{e-bcv-lip},\nwe see that the backtracking condition~\\eqref{e-ls-cond} holds\nat iteration $k$ if $0 < \\delta < 1$ and\n\\[\n\\tau_k \\sigma_k\\|A\\|^2 + \\tau_k L \\leq \\delta^2.\n\\]\n","start":80409,"end":80754}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"下界归纳的设置","source_units":[{"raw_tex":"Then mathematical induction can be used to prove~\\eqref{e-ls-param-lbnd}.\nThe two lower bounds~\\eqref{e-ls-param-lbnd} hold at $k=0$\nby the definition of $\\tau_\\mathrm{min}$ and $\\sigma_\\mathrm{min}$.\nNow assume $\\tau_{k-1} \\geq \\tau_\\mathrm{min}$,\n$\\sigma_{k-1} \\geq \\sigma_\\mathrm{min}$,\nand consider the $k$th iteration.\n","start":80754,"end":81078}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"首次试探被接受","source_units":[{"raw_tex":"The first attempt of $\\theta_k$ is $\\theta_k=\\bar \\theta_k \\geq 1$.\nIf this value is accepted, then\n\\[\n\\tau_k = \\bar\\theta_k \\tau_{k-1} \\geq \\tau_{k-1} \\geq \\tau_\\mathrm{min},\n\\qquad\n\\sigma_k = \\bar\\theta_k \\sigma_{k-1} \\geq \\sigma_{k-1} \\geq\n  \\sigma_\\mathrm{min}.\n\\]\n","start":81078,"end":81347}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"最后拒绝值的二次界","source_units":[{"raw_tex":"Otherwise, one or more backtracking steps are needed.\nDenote by $\\tilde \\theta_k$ the last rejected value.\nThen $\\tilde\\theta_k^2\\tau_{k-1}^2\\beta\\|A\\|^2 + \n\\tilde\\theta_k \\tau_{k-1} L >\\delta^2$\nand the accepted $\\theta_k$ satisfies\n\\[\n\\theta_k = \\frac{\\tilde \\theta_k}{2} \\geq\n\\frac{-L+\\sqrt{L^2+4\\delta^2\\beta\\|A\\|^2}}%\n  {4\\tau_{k-1}\\beta\\|A\\|^2}.\n\\]\n","start":81347,"end":81702}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"接受步长满足统一下界","source_units":[{"raw_tex":"Therefore,\n\\[\n\\tau_k=\\theta_k\\tau_{k-1} >\n\\frac{-L+\\sqrt{L^2+4\\delta^2\\beta\\|A\\|^2}}{4\\beta\\|A\\|^2}, \\qquad\n\\sigma_k=\\beta\\tau_k \\geq \\beta\\tau_\\mathrm{min}. \\qedhere\n\\]\n\\end{proof}","start":81702,"end":81883}]},{"id":"S050","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"步长比保持不变","source_units":[{"raw_tex":"we set $\\theta_k = 2^{-i}\\bar\\theta_k$,\n$\\tau_k=\\theta_k\\tau_{k-1}$, $\\sigma_k=\\theta_k\\sigma_{k-1}$,\n","start":78538,"end":78640},{"raw_tex":"where $\\beta=\\sigma_{-1}/\\tau_{-1}$.\n","start":80256,"end":80293}]},{"id":"S051","semantic_type":"proof","kind":"proof","labels":["e-ls-conv-1","e-ls-conv-2","e-ls-conv-3","e-ls-conv-4"],"declared_external":false,"source_only":false,"title":"单步不等式的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nThe optimality condition for the primal prox-operator~\\eqref{e-ls-b}\ngives\n\\BEAS\n\\lefteqn{f(x^{(k+1)})-f(x)} \\\\\n&\\leq & \\frac{1}{\\tau_k}\n  \\big(d_\\primal(x,x^{(k)})-d_\\primal(x,x^{(k+1)})-\n  d_\\primal(x^{(k+1)}, x^{(k)})\\big)\n  +\\inprod{A^T\\bar z^{(k+1)}+\\nabla h(x^{(k)})}{x-x^{(k+1)}}\n\\EEAS\nfor all $x\\in \\dom f \\cap \\dom\\phi_\\primal$.\nHence\n\\BEA\n\\lefteqn{f(x^{(k+1)}) + h(x^{(k+1)}) - f(x) - h(x)} \\nonumber \\\\\n  &\\leq & \\frac{1}{\\tau_k} (d_\\primal(x,x^{(k)}) - \n  d_\\primal(x,x^{(k+1)}) - d_\\primal(x^{(k+1)}, x^{(k)}))\n  + \\inprod{A^T\\bar z_{k+1}}{x - x^{(k+1)}} \\nonumber \\\\\n& &\\mbox{} + h(x^{(k+1)})-h(x)+\\inprod{\\nabla h(x^{(k)})}{x-x^{(k+1)}}\n  \\nonumber \\\\\n&\\leq & \\frac{1}{\\tau_k} (d_\\primal(x,x^{(k)}) - \n d_\\primal(x,x^{(k+1)}) - d_\\primal(x^{(k+1)}, x^{(k)}))\n  + \\inprod{A^T\\bar z^{(k+1)}}{x - x^{(k+1)}} \\nonumber \\\\\n& &\\mbox{} +h(x^{(k+1)})-h(x^{(k)})\n  -\\inprod{\\nabla h(x^{(k)})}{x^{(k+1)}-x^{(k)}}.\n  \\label{e-ls-conv-1}\n\\EEA\nThe second inequality follows from the convexity of $h$, \\ie,\n$h(x) \\geq h(x^{(k)}) + \\inprod{\\nabla h(x^{(k)})}{x-x^{(k)}}$.\nThe dual update~\\eqref{e-ls-c} implies that\n\\BEQ \\label{e-ls-conv-2}\n\\inprod{z-z^{(k+1)}}{Ax^{(k+1)}-b}\n= \\frac{1}{\\sigma_k} \\inprod{z-z^{(k+1)}}{z^{(k+1)}-z^{(k)}}\n\\quad \\mbox{for all $z$.}\n\\EEQ\nThis equality at $k=i-1$ is\n\\begin{align}\n\\inprod{z-z^{(i)}}{Ax^{(i)}-b}\n  &= \\frac{1}{\\sigma_{i-1}} \\inprod{z-z^{(i)}}{z^{(i)}-z^{(i-1)}} \n  \\nonumber \\\\\n&= \\frac{1}{2\\sigma_{i-1}} \\left( \\|z-z^{(i-1)}\\|^2\n  - \\|z-z^{(i)}\\|^2 - \\|z^{(i)} - z^{(i-1)} \\|^2 \\right).\n  \\label{e-ls-conv-3}\n\\end{align}\nThe equality~\\eqref{e-ls-conv-2} at $k=i-2$ is\n\\begin{align*}\n\\inprod{z-z^{(i-1)}}{Ax^{(i-1)}-b}\n  &= \\frac{1}{\\sigma_{i-2}} \\inprod{z-z^{(i-1)}}{z^{(i-1)}-z^{(i-2)}} \\\\\n&= \\frac{\\theta_{i-1}}{\\sigma_{i-1}}\n  \\inprod{z-z^{(i-1)}}{z^{(i-1)}-z^{(i-2)}} \\\\\n&= \\frac{1}{\\sigma_{i-1}} \\inprod{z-z^{(i-1)}}{\\bar z^{(i)}-z^{(i-1)}}.\n\\end{align*}\nWe evaluate this at $z=z^{(i)}$ and add it to the equality\nat $z=z^{(i-2)}$ multiplied by $\\theta_{i-1}$:\n\\begin{align}\n\\inprod{z^{(i)}-\\bar z^{(i)}}{Ax^{(i-1)}-b}\n  &= \\frac{1}{\\sigma_{i-1}}\n  \\inprod{z^{(i)}-\\bar z^{(i)}}{\\bar z^{(i)} - z^{(i-1)}} \\nonumber \\\\\n&= \\frac{1}{2\\sigma_{i-1}} \\left(\\|z^{(i)}-z^{(i-1)}\\|^2\n  -\\|z^{(i)}-\\bar z^{(i)}\\|^2-\\|\\bar z^{(i)}-z^{(i-1)}\\|^2 \\right).\n  \\label{e-ls-conv-4}\n\\end{align}\nNow we combine~\\eqref{e-ls-conv-1} for $k=i-1$,\nwith~\\eqref{e-ls-conv-3} and~\\eqref{e-ls-conv-4}. For $i \\geq 1$,\n\\BEAS\n\\lefteqn{\\cL(x^{(i)},z) - \\cL(x, \\bar z^{(i)})} \\nonumber \\\\\n&=& f(x^{(i)})+h(x^{(i)})+\\inprod{z}{Ax^{(i)}-b}\n  -f(x)-h(x)-\\inprod{\\bar z^{(i)}}{Ax-b} \\nonumber \\\\\n&\\leq & \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)})-d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big)\n  + \\inprod{A^T\\bar z^{(i)}}{x-x^{(i)}} + \\inprod{z}{Ax^{(i)}-b} \\nonumber\\\\\n& &\\mbox{} - \\inprod{\\bar z^{(i)}}{Ax-b} + h(x^{(i)}) - h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)}) \n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big)\n  + \\inprod{z-\\bar z^{(i)}}{Ax^{(i)}-b} \\nonumber \\\\\n& &\\mbox{} +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big) \\nonumber \\\\\n& & \\mbox{} +\\inprod{z^{(i)}-\\bar z^{(i)}}{A(x^{(i)}-x^{(i-1)})}\n  + \\inprod{z-z^{(i)}}{Ax^{(i)}-b}\n  + \\inprod{z^{(i)}-\\bar z^{(i)}}{Ax^{(i-1)}-b} \\nonumber \\\\\n& &\\mbox{} +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)}))\\Big) \\nonumber \\\\\n& &\\mbox{} +\\frac{1}{2\\sigma_{i-1}}\n  \\Big(\\|z-z^{(i-1)}\\|^2-\\|z-z^{(i)}\\|^2-\\|\\bar z^{(i)}-z^{(i-1)}\\|^2\n  -\\|\\bar z^{(i)}-z^{(i)}\\|^2\\Big) \\nonumber \\\\\n& &\\mbox{}+\\inprod{A^T(z^{(i)}-\\bar z^{(i)})}{x^{(i)}-x^{(i-1)}}\n  +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)})} \\\\\n&\\leq & \\frac{1}{\\tau_{i-1}} \n  \\left(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)}) -\n  (1-\\delta^2) d_\\primal(x^{(i)},x^{(i-1)})\\right) \\nonumber \\\\\n& &\\mbox{} + \\frac{1}{2\\sigma_{i-1}} \\left(\n  \\|z-z^{(i-1)}\\|^2 - \\|z-z^{(i)}\\|^2 - \n  \\|\\bar z^{(i)} - z^{(i-1)}\\|^2 \\right),\n\\EEAS\nwhich is the desired result~\\eqref{e-ls-conv-i}.\nThe first inequality follows from~\\eqref{e-ls-conv-1}.\nIn the second last step we substitute~\\eqref{e-ls-conv-3}\nand~\\eqref{e-ls-conv-4}.\nThe last step uses the line search exit condition~\\eqref{e-ls-cond}\nat $k=i-1$.\n\\end{proof}","start":82509,"end":87115}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"应用原始近端最优性","source_units":[{"raw_tex":"\\begin{proof}\nThe optimality condition for the primal prox-operator~\\eqref{e-ls-b}\ngives\n\\BEAS\n\\lefteqn{f(x^{(k+1)})-f(x)} \\\\\n&\\leq & \\frac{1}{\\tau_k}\n  \\big(d_\\primal(x,x^{(k)})-d_\\primal(x,x^{(k+1)})-\n  d_\\primal(x^{(k+1)}, x^{(k)})\\big)\n  +\\inprod{A^T\\bar z^{(k+1)}+\\nabla h(x^{(k)})}{x-x^{(k+1)}}\n\\EEAS\nfor all $x\\in \\dom f \\cap \\dom\\phi_\\primal$.\n","start":82509,"end":82861}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":["e-ls-conv-1"],"declared_external":false,"source_only":false,"title":"加入光滑目标项","source_units":[{"raw_tex":"Hence\n\\BEA\n\\lefteqn{f(x^{(k+1)}) + h(x^{(k+1)}) - f(x) - h(x)} \\nonumber \\\\\n  &\\leq & \\frac{1}{\\tau_k} (d_\\primal(x,x^{(k)}) - \n  d_\\primal(x,x^{(k+1)}) - d_\\primal(x^{(k+1)}, x^{(k)}))\n  + \\inprod{A^T\\bar z_{k+1}}{x - x^{(k+1)}} \\nonumber \\\\\n& &\\mbox{} + h(x^{(k+1)})-h(x)+\\inprod{\\nabla h(x^{(k)})}{x-x^{(k+1)}}\n  \\nonumber \\\\\n&\\leq & \\frac{1}{\\tau_k} (d_\\primal(x,x^{(k)}) - \n d_\\primal(x,x^{(k+1)}) - d_\\primal(x^{(k+1)}, x^{(k)}))\n  + \\inprod{A^T\\bar z^{(k+1)}}{x - x^{(k+1)}} \\nonumber \\\\\n& &\\mbox{} +h(x^{(k+1)})-h(x^{(k)})\n  -\\inprod{\\nabla h(x^{(k)})}{x^{(k+1)}-x^{(k)}}.\n  \\label{e-ls-conv-1}\n\\EEA\nThe second inequality follows from the convexity of $h$, \\ie,\n$h(x) \\geq h(x^{(k)}) + \\inprod{\\nabla h(x^{(k)})}{x-x^{(k)}}$.\n","start":82861,"end":83595}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":["e-ls-conv-2"],"declared_external":false,"source_only":false,"title":"残差与对偶增量恒等式","source_units":[{"raw_tex":"The dual update~\\eqref{e-ls-c} implies that\n\\BEQ \\label{e-ls-conv-2}\n\\inprod{z-z^{(k+1)}}{Ax^{(k+1)}-b}\n= \\frac{1}{\\sigma_k} \\inprod{z-z^{(k+1)}}{z^{(k+1)}-z^{(k)}}\n\\quad \\mbox{for all $z$.}\n\\EEQ\n","start":83595,"end":83791}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["e-ls-conv-3"],"declared_external":false,"source_only":false,"title":"对偶增量的平方差形式","source_units":[{"raw_tex":"This equality at $k=i-1$ is\n\\begin{align}\n\\inprod{z-z^{(i)}}{Ax^{(i)}-b}\n  &= \\frac{1}{\\sigma_{i-1}} \\inprod{z-z^{(i)}}{z^{(i)}-z^{(i-1)}} \n  \\nonumber \\\\\n&= \\frac{1}{2\\sigma_{i-1}} \\left( \\|z-z^{(i-1)}\\|^2\n  - \\|z-z^{(i)}\\|^2 - \\|z^{(i)} - z^{(i-1)} \\|^2 \\right).\n  \\label{e-ls-conv-3}\n\\end{align}\n","start":83791,"end":84090}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"前一残差与外推增量","source_units":[{"raw_tex":"The equality~\\eqref{e-ls-conv-2} at $k=i-2$ is\n\\begin{align*}\n\\inprod{z-z^{(i-1)}}{Ax^{(i-1)}-b}\n  &= \\frac{1}{\\sigma_{i-2}} \\inprod{z-z^{(i-1)}}{z^{(i-1)}-z^{(i-2)}} \\\\\n&= \\frac{\\theta_{i-1}}{\\sigma_{i-1}}\n  \\inprod{z-z^{(i-1)}}{z^{(i-1)}-z^{(i-2)}} \\\\\n&= \\frac{1}{\\sigma_{i-1}} \\inprod{z-z^{(i-1)}}{\\bar z^{(i)}-z^{(i-1)}}.\n\\end{align*}\n","start":84090,"end":84429}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":["e-ls-conv-4"],"declared_external":false,"source_only":false,"title":"外推残差的平方恒等式","source_units":[{"raw_tex":"We evaluate this at $z=z^{(i)}$ and add it to the equality\nat $z=z^{(i-2)}$ multiplied by $\\theta_{i-1}$:\n\\begin{align}\n\\inprod{z^{(i)}-\\bar z^{(i)}}{Ax^{(i-1)}-b}\n  &= \\frac{1}{\\sigma_{i-1}}\n  \\inprod{z^{(i)}-\\bar z^{(i)}}{\\bar z^{(i)} - z^{(i-1)}} \\nonumber \\\\\n&= \\frac{1}{2\\sigma_{i-1}} \\left(\\|z^{(i)}-z^{(i-1)}\\|^2\n  -\\|z^{(i)}-\\bar z^{(i)}\\|^2-\\|\\bar z^{(i)}-z^{(i-1)}\\|^2 \\right).\n  \\label{e-ls-conv-4}\n\\end{align}\n","start":84429,"end":84851}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"合并拉格朗日差与平方差","source_units":[{"raw_tex":"Now we combine~\\eqref{e-ls-conv-1} for $k=i-1$,\nwith~\\eqref{e-ls-conv-3} and~\\eqref{e-ls-conv-4}. For $i \\geq 1$,\n\\BEAS\n\\lefteqn{\\cL(x^{(i)},z) - \\cL(x, \\bar z^{(i)})} \\nonumber \\\\\n&=& f(x^{(i)})+h(x^{(i)})+\\inprod{z}{Ax^{(i)}-b}\n  -f(x)-h(x)-\\inprod{\\bar z^{(i)}}{Ax-b} \\nonumber \\\\\n&\\leq & \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)})-d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big)\n  + \\inprod{A^T\\bar z^{(i)}}{x-x^{(i)}} + \\inprod{z}{Ax^{(i)}-b} \\nonumber\\\\\n& &\\mbox{} - \\inprod{\\bar z^{(i)}}{Ax-b} + h(x^{(i)}) - h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)}) \n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big)\n  + \\inprod{z-\\bar z^{(i)}}{Ax^{(i)}-b} \\nonumber \\\\\n& &\\mbox{} +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big) \\nonumber \\\\\n& & \\mbox{} +\\inprod{z^{(i)}-\\bar z^{(i)}}{A(x^{(i)}-x^{(i-1)})}\n  + \\inprod{z-z^{(i)}}{Ax^{(i)}-b}\n  + \\inprod{z^{(i)}-\\bar z^{(i)}}{Ax^{(i-1)}-b} \\nonumber \\\\\n& &\\mbox{} +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)}))\\Big) \\nonumber \\\\\n& &\\mbox{} +\\frac{1}{2\\sigma_{i-1}}\n  \\Big(\\|z-z^{(i-1)}\\|^2-\\|z-z^{(i)}\\|^2-\\|\\bar z^{(i)}-z^{(i-1)}\\|^2\n  -\\|\\bar z^{(i)}-z^{(i)}\\|^2\\Big) \\nonumber \\\\\n& &\\mbox{}+\\inprod{A^T(z^{(i)}-\\bar z^{(i)})}{x^{(i)}-x^{(i-1)}}\n  +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)})} \\\\\n","start":84851,"end":86554}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"接受条件消去余项","source_units":[{"raw_tex":"&\\leq & \\frac{1}{\\tau_{i-1}} \n  \\left(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)}) -\n  (1-\\delta^2) d_\\primal(x^{(i)},x^{(i-1)})\\right) \\nonumber \\\\\n& &\\mbox{} + \\frac{1}{2\\sigma_{i-1}} \\left(\n  \\|z-z^{(i-1)}\\|^2 - \\|z-z^{(i)}\\|^2 - \n  \\|\\bar z^{(i)} - z^{(i-1)}\\|^2 \\right),\n\\EEAS\nwhich is the desired result~\\eqref{e-ls-conv-i}.\nThe first inequality follows from~\\eqref{e-ls-conv-1}.\nIn the second last step we substitute~\\eqref{e-ls-conv-3}\nand~\\eqref{e-ls-conv-4}.\nThe last step uses the line search exit condition~\\eqref{e-ls-cond}\nat $k=i-1$.\n\\end{proof}","start":86554,"end":87115}]},{"id":"S060","semantic_type":"proof","kind":"proof","labels":["e-ls-conv-k"],"declared_external":false,"source_only":false,"title":"加权遍历界的完整主证明","source_units":[{"raw_tex":"\\begin{proof}\nFrom~\\eqref{e-ls-conv-i},\n\\[\n\\cL(x^{(i)},z)-\\cL(x, \\bar z^{(i)}) \\leq \\frac{1}{\\tau_{i-1}}\n\\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n+ \\frac{1}{2\\beta} \\|z-z^{(i-1)}\\|^2\n- \\frac{1}{2\\beta} \\|z-z^{(i)}\\|^2 \\Big).\n\\]\nSince $\\cL(u,v)$ is convex in $u$ and affine in $v$,\n\\BEA\n\\lefteqn{(\\sum_{i=1}^k \\tau_{i-1}) \n  \\big(\\cL(x^{(k)}_\\avg, z)-\\cL(x, \\bar z^{(k)}_\\avg)\\big)} \\nonumber \\\\\n&\\leq &\\sum_{i=1}^k \\tau_{i-1} (\\cL(x^{(i)}, z)-\\cL(x, \\bar z^{(i)})) \n  \\nonumber \\\\\n&\\leq & d_\\primal(x,x^{(0)}) - d_\\primal(x,x^{(k)}) +\n  \\frac{1}{2\\beta} (\\|z-z^{(0)}\\|^2 - \\|z-z^{(k)}\\|^2) \\nonumber \\\\\n&\\leq & d_\\primal(x,x^{(0)}) + \\frac{1}{2\\beta} \\|z-z^{(0)}\\|^2.\n  \\label{e-ls-conv-k}\n\\EEA\nDividing by $\\sum_{i=1}^k \\tau_{i-1}$ gives~\\eqref{e-ls-conv}.\n\\end{proof}","start":88061,"end":88840}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"删去单步耗散项","source_units":[{"raw_tex":"\\begin{proof}\nFrom~\\eqref{e-ls-conv-i},\n\\[\n\\cL(x^{(i)},z)-\\cL(x, \\bar z^{(i)}) \\leq \\frac{1}{\\tau_{i-1}}\n\\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n+ \\frac{1}{2\\beta} \\|z-z^{(i-1)}\\|^2\n- \\frac{1}{2\\beta} \\|z-z^{(i)}\\|^2 \\Big).\n\\]\n","start":88061,"end":88299}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"加权平均的凸性估计","source_units":[{"raw_tex":"Since $\\cL(u,v)$ is convex in $u$ and affine in $v$,\n\\BEA\n\\lefteqn{(\\sum_{i=1}^k \\tau_{i-1}) \n  \\big(\\cL(x^{(k)}_\\avg, z)-\\cL(x, \\bar z^{(k)}_\\avg)\\big)} \\nonumber \\\\\n&\\leq &\\sum_{i=1}^k \\tau_{i-1} (\\cL(x^{(i)}, z)-\\cL(x, \\bar z^{(i)})) \n  \\nonumber \\\\\n","start":88299,"end":88552}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":["e-ls-conv-k"],"declared_external":false,"source_only":false,"title":"距离差望远镜求和","source_units":[{"raw_tex":"&\\leq & d_\\primal(x,x^{(0)}) - d_\\primal(x,x^{(k)}) +\n  \\frac{1}{2\\beta} (\\|z-z^{(0)}\\|^2 - \\|z-z^{(k)}\\|^2) \\nonumber \\\\\n","start":88552,"end":88674}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"删去末端非负距离","source_units":[{"raw_tex":"&\\leq & d_\\primal(x,x^{(0)}) + \\frac{1}{2\\beta} \\|z-z^{(0)}\\|^2.\n  \\label{e-ls-conv-k}\n\\EEA\n","start":88674,"end":88766}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"除以累计步长","source_units":[{"raw_tex":"Dividing by $\\sum_{i=1}^k \\tau_{i-1}$ gives~\\eqref{e-ls-conv}.\n\\end{proof}","start":88766,"end":88840}]},{"id":"S066","semantic_type":"proof","kind":"proof","labels":["e-bcv-delta","e-ls-conv-1","e-ls-conv-2","e-ls-conv-3","e-ls-conv-4","e-ls-conv-k"],"declared_external":false,"source_only":false,"title":"遍历收敛的完整作者证明包","source_units":[{"raw_tex":"\\begin{proof}\nTo show that the kernel functions $\\phi_+$ and $\\phi_-$ are convex,\nwe show that $d_+$ and $d_-$ are nonnegative.\nSuppose $\\sigma\\tau \\|A\\|^2 \\leq \\delta_1 \\delta_2$\nwith $\\delta_1,\\delta_2 > 0$.\nThen~\\eqref{e-bcv-str-cvx} and the arithmetic--geometric mean inequality\nimply that\n\\begin{align}\n\\left| \\inprod{z-z^\\prime}{A(x-x^\\prime)}\\right|\n  &\\leq \\|A\\| \\|z-z'\\|_\\dual \\|x-x'\\|_\\primal  \\nonumber \\\\\n&\\leq \\sqrt{\\frac{\\delta_1 \\delta_2}{\\sigma \\tau}}\n\\|z-z'\\|_\\dual \\|x-x'\\|_\\primal \\nonumber \\\\\n&\\leq \\frac{\\delta_1}{2\\tau} \\|x-x'\\|_\\primal^2  +\n  \\frac{\\delta_2}{2\\sigma} \\|z-z'\\|_\\primal^2 \\nonumber \\\\\n&\\leq \\frac{\\delta_1}{\\tau} d_\\primal (x,x') +\n  \\frac{\\delta_2}{\\sigma} d_\\dual (z,z'). \\label{e-bcv-delta}\n\\end{align}\nTherefore,\n\\begin{align*}\nd_{\\pm} (x,z; x',z')\n  &= \\frac{1}{\\tau} d_\\primal(x,x^\\prime) + \\frac{1}{\\sigma}\n  d_\\dual(z,z^\\prime) \\pm \\inprod{z-z^\\prime}{A(x-x^\\prime)} \\\\\n&\\geq \\frac{1-\\delta_1}{\\tau} d_\\primal (x,x') +\n  \\frac{1-\\delta_2}{\\sigma} d_\\dual (z,z') \\\\\n&\\geq \\frac{1-\\delta_1}{2\\tau} \\|x-x'\\|_\\primal^2 +\n  \\frac{1-\\delta_2}{2\\sigma} \\|z-z'\\|_\\dual^2.\n\\end{align*}\nWith $\\delta_1=\\delta_2=1$, this shows convexity of $\\phi_+$ and $\\phi_-$;\nwith $\\delta_1=\\delta_2<1$, strong convexity.\nSimilarly,\n\\begin{align*}\nd_\\mathrm{dcv/pcv} (x,z; x', z')\n  &= d_{\\pm} (x,z; x',z') - h(x) + h(x') + \\inprod{\\nabla h(x')}{x-x'} \\\\\n&\\geq \\frac{1-\\tau L-\\delta_1}{\\tau} d_\\primal (x,x^\\prime) +\n  \\frac{1-\\delta_2}{\\sigma} d_\\dual (z,z^\\prime).\n\\end{align*}\nWith $\\delta_1=1-\\tau L$ and $\\delta_2=1$, this shows convexity of\n$\\phi_\\mathrm{pcv}$ and $\\phi_\\mathrm{dcv}$;\nwith $\\delta_1=\\delta-\\tau L$ and $\\delta_2=\\delta < 1$,\nstrong convexity.\n\\end{proof}","start":47108,"end":48808},{"raw_tex":"\\begin{proof}\nApplying~\\eqref{e-bcv-delta} with $\\tau=\\tau_k$, $\\sigma=\\sigma_k$, $\\delta_1=\\delta^2-\\tau_k L$ and $\\delta_2=1$,\ntogether with the Lipschitz condition~\\eqref{e-bcv-lip},\nwe see that the backtracking condition~\\eqref{e-ls-cond} holds\nat iteration $k$ if $0 < \\delta < 1$ and\n\\[\n\\tau_k \\sigma_k\\|A\\|^2 + \\tau_k L \\leq \\delta^2.\n\\]\nThen mathematical induction can be used to prove~\\eqref{e-ls-param-lbnd}.\nThe two lower bounds~\\eqref{e-ls-param-lbnd} hold at $k=0$\nby the definition of $\\tau_\\mathrm{min}$ and $\\sigma_\\mathrm{min}$.\nNow assume $\\tau_{k-1} \\geq \\tau_\\mathrm{min}$,\n$\\sigma_{k-1} \\geq \\sigma_\\mathrm{min}$,\nand consider the $k$th iteration.\nThe first attempt of $\\theta_k$ is $\\theta_k=\\bar \\theta_k \\geq 1$.\nIf this value is accepted, then\n\\[\n\\tau_k = \\bar\\theta_k \\tau_{k-1} \\geq \\tau_{k-1} \\geq \\tau_\\mathrm{min},\n\\qquad\n\\sigma_k = \\bar\\theta_k \\sigma_{k-1} \\geq \\sigma_{k-1} \\geq\n  \\sigma_\\mathrm{min}.\n\\]\nOtherwise, one or more backtracking steps are needed.\nDenote by $\\tilde \\theta_k$ the last rejected value.\nThen $\\tilde\\theta_k^2\\tau_{k-1}^2\\beta\\|A\\|^2 + \n\\tilde\\theta_k \\tau_{k-1} L >\\delta^2$\nand the accepted $\\theta_k$ satisfies\n\\[\n\\theta_k = \\frac{\\tilde \\theta_k}{2} \\geq\n\\frac{-L+\\sqrt{L^2+4\\delta^2\\beta\\|A\\|^2}}%\n  {4\\tau_{k-1}\\beta\\|A\\|^2}.\n\\]\nTherefore,\n\\[\n\\tau_k=\\theta_k\\tau_{k-1} >\n\\frac{-L+\\sqrt{L^2+4\\delta^2\\beta\\|A\\|^2}}{4\\beta\\|A\\|^2}, \\qquad\n\\sigma_k=\\beta\\tau_k \\geq \\beta\\tau_\\mathrm{min}. \\qedhere\n\\]\n\\end{proof}","start":80409,"end":81883},{"raw_tex":"\\begin{proof}\nThe optimality condition for the primal prox-operator~\\eqref{e-ls-b}\ngives\n\\BEAS\n\\lefteqn{f(x^{(k+1)})-f(x)} \\\\\n&\\leq & \\frac{1}{\\tau_k}\n  \\big(d_\\primal(x,x^{(k)})-d_\\primal(x,x^{(k+1)})-\n  d_\\primal(x^{(k+1)}, x^{(k)})\\big)\n  +\\inprod{A^T\\bar z^{(k+1)}+\\nabla h(x^{(k)})}{x-x^{(k+1)}}\n\\EEAS\nfor all $x\\in \\dom f \\cap \\dom\\phi_\\primal$.\nHence\n\\BEA\n\\lefteqn{f(x^{(k+1)}) + h(x^{(k+1)}) - f(x) - h(x)} \\nonumber \\\\\n  &\\leq & \\frac{1}{\\tau_k} (d_\\primal(x,x^{(k)}) - \n  d_\\primal(x,x^{(k+1)}) - d_\\primal(x^{(k+1)}, x^{(k)}))\n  + \\inprod{A^T\\bar z_{k+1}}{x - x^{(k+1)}} \\nonumber \\\\\n& &\\mbox{} + h(x^{(k+1)})-h(x)+\\inprod{\\nabla h(x^{(k)})}{x-x^{(k+1)}}\n  \\nonumber \\\\\n&\\leq & \\frac{1}{\\tau_k} (d_\\primal(x,x^{(k)}) - \n d_\\primal(x,x^{(k+1)}) - d_\\primal(x^{(k+1)}, x^{(k)}))\n  + \\inprod{A^T\\bar z^{(k+1)}}{x - x^{(k+1)}} \\nonumber \\\\\n& &\\mbox{} +h(x^{(k+1)})-h(x^{(k)})\n  -\\inprod{\\nabla h(x^{(k)})}{x^{(k+1)}-x^{(k)}}.\n  \\label{e-ls-conv-1}\n\\EEA\nThe second inequality follows from the convexity of $h$, \\ie,\n$h(x) \\geq h(x^{(k)}) + \\inprod{\\nabla h(x^{(k)})}{x-x^{(k)}}$.\nThe dual update~\\eqref{e-ls-c} implies that\n\\BEQ \\label{e-ls-conv-2}\n\\inprod{z-z^{(k+1)}}{Ax^{(k+1)}-b}\n= \\frac{1}{\\sigma_k} \\inprod{z-z^{(k+1)}}{z^{(k+1)}-z^{(k)}}\n\\quad \\mbox{for all $z$.}\n\\EEQ\nThis equality at $k=i-1$ is\n\\begin{align}\n\\inprod{z-z^{(i)}}{Ax^{(i)}-b}\n  &= \\frac{1}{\\sigma_{i-1}} \\inprod{z-z^{(i)}}{z^{(i)}-z^{(i-1)}} \n  \\nonumber \\\\\n&= \\frac{1}{2\\sigma_{i-1}} \\left( \\|z-z^{(i-1)}\\|^2\n  - \\|z-z^{(i)}\\|^2 - \\|z^{(i)} - z^{(i-1)} \\|^2 \\right).\n  \\label{e-ls-conv-3}\n\\end{align}\nThe equality~\\eqref{e-ls-conv-2} at $k=i-2$ is\n\\begin{align*}\n\\inprod{z-z^{(i-1)}}{Ax^{(i-1)}-b}\n  &= \\frac{1}{\\sigma_{i-2}} \\inprod{z-z^{(i-1)}}{z^{(i-1)}-z^{(i-2)}} \\\\\n&= \\frac{\\theta_{i-1}}{\\sigma_{i-1}}\n  \\inprod{z-z^{(i-1)}}{z^{(i-1)}-z^{(i-2)}} \\\\\n&= \\frac{1}{\\sigma_{i-1}} \\inprod{z-z^{(i-1)}}{\\bar z^{(i)}-z^{(i-1)}}.\n\\end{align*}\nWe evaluate this at $z=z^{(i)}$ and add it to the equality\nat $z=z^{(i-2)}$ multiplied by $\\theta_{i-1}$:\n\\begin{align}\n\\inprod{z^{(i)}-\\bar z^{(i)}}{Ax^{(i-1)}-b}\n  &= \\frac{1}{\\sigma_{i-1}}\n  \\inprod{z^{(i)}-\\bar z^{(i)}}{\\bar z^{(i)} - z^{(i-1)}} \\nonumber \\\\\n&= \\frac{1}{2\\sigma_{i-1}} \\left(\\|z^{(i)}-z^{(i-1)}\\|^2\n  -\\|z^{(i)}-\\bar z^{(i)}\\|^2-\\|\\bar z^{(i)}-z^{(i-1)}\\|^2 \\right).\n  \\label{e-ls-conv-4}\n\\end{align}\nNow we combine~\\eqref{e-ls-conv-1} for $k=i-1$,\nwith~\\eqref{e-ls-conv-3} and~\\eqref{e-ls-conv-4}. For $i \\geq 1$,\n\\BEAS\n\\lefteqn{\\cL(x^{(i)},z) - \\cL(x, \\bar z^{(i)})} \\nonumber \\\\\n&=& f(x^{(i)})+h(x^{(i)})+\\inprod{z}{Ax^{(i)}-b}\n  -f(x)-h(x)-\\inprod{\\bar z^{(i)}}{Ax-b} \\nonumber \\\\\n&\\leq & \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)})-d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big)\n  + \\inprod{A^T\\bar z^{(i)}}{x-x^{(i)}} + \\inprod{z}{Ax^{(i)}-b} \\nonumber\\\\\n& &\\mbox{} - \\inprod{\\bar z^{(i)}}{Ax-b} + h(x^{(i)}) - h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)}) \n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big)\n  + \\inprod{z-\\bar z^{(i)}}{Ax^{(i)}-b} \\nonumber \\\\\n& &\\mbox{} +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big) \\nonumber \\\\\n& & \\mbox{} +\\inprod{z^{(i)}-\\bar z^{(i)}}{A(x^{(i)}-x^{(i-1)})}\n  + \\inprod{z-z^{(i)}}{Ax^{(i)}-b}\n  + \\inprod{z^{(i)}-\\bar z^{(i)}}{Ax^{(i-1)}-b} \\nonumber \\\\\n& &\\mbox{} +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)}))\\Big) \\nonumber \\\\\n& &\\mbox{} +\\frac{1}{2\\sigma_{i-1}}\n  \\Big(\\|z-z^{(i-1)}\\|^2-\\|z-z^{(i)}\\|^2-\\|\\bar z^{(i)}-z^{(i-1)}\\|^2\n  -\\|\\bar z^{(i)}-z^{(i)}\\|^2\\Big) \\nonumber \\\\\n& &\\mbox{}+\\inprod{A^T(z^{(i)}-\\bar z^{(i)})}{x^{(i)}-x^{(i-1)}}\n  +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)})} \\\\\n&\\leq & \\frac{1}{\\tau_{i-1}} \n  \\left(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)}) -\n  (1-\\delta^2) d_\\primal(x^{(i)},x^{(i-1)})\\right) \\nonumber \\\\\n& &\\mbox{} + \\frac{1}{2\\sigma_{i-1}} \\left(\n  \\|z-z^{(i-1)}\\|^2 - \\|z-z^{(i)}\\|^2 - \n  \\|\\bar z^{(i)} - z^{(i-1)}\\|^2 \\right),\n\\EEAS\nwhich is the desired result~\\eqref{e-ls-conv-i}.\nThe first inequality follows from~\\eqref{e-ls-conv-1}.\nIn the second last step we substitute~\\eqref{e-ls-conv-3}\nand~\\eqref{e-ls-conv-4}.\nThe last step uses the line search exit condition~\\eqref{e-ls-cond}\nat $k=i-1$.\n\\end{proof}","start":82509,"end":87115},{"raw_tex":"\\begin{proof}\nFrom~\\eqref{e-ls-conv-i},\n\\[\n\\cL(x^{(i)},z)-\\cL(x, \\bar z^{(i)}) \\leq \\frac{1}{\\tau_{i-1}}\n\\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n+ \\frac{1}{2\\beta} \\|z-z^{(i-1)}\\|^2\n- \\frac{1}{2\\beta} \\|z-z^{(i)}\\|^2 \\Big).\n\\]\nSince $\\cL(u,v)$ is convex in $u$ and affine in $v$,\n\\BEA\n\\lefteqn{(\\sum_{i=1}^k \\tau_{i-1}) \n  \\big(\\cL(x^{(k)}_\\avg, z)-\\cL(x, \\bar z^{(k)}_\\avg)\\big)} \\nonumber \\\\\n&\\leq &\\sum_{i=1}^k \\tau_{i-1} (\\cL(x^{(i)}, z)-\\cL(x, \\bar z^{(i)})) \n  \\nonumber \\\\\n&\\leq & d_\\primal(x,x^{(0)}) - d_\\primal(x,x^{(k)}) +\n  \\frac{1}{2\\beta} (\\|z-z^{(0)}\\|^2 - \\|z-z^{(k)}\\|^2) \\nonumber \\\\\n&\\leq & d_\\primal(x,x^{(0)}) + \\frac{1}{2\\beta} \\|z-z^{(0)}\\|^2.\n  \\label{e-ls-conv-k}\n\\EEA\nDividing by $\\sum_{i=1}^k \\tau_{i-1}$ gives~\\eqref{e-ls-conv}.\n\\end{proof}","start":88061,"end":88840}]},{"id":"S067","semantic_type":"proposition","kind":"proposition","labels":["e-ls-conv","e-ls-conv-1/k"],"declared_external":false,"source_only":false,"title":"回溯算法的遍历收敛界","source_units":[{"raw_tex":"We show that\n\\begin{align}\n\\cL(x^{(k)}_\\avg,z) - \\cL(x,\\bar z^{(k)}_\\avg)\n  &\\leq \\frac{1}{\\sum_{i=1}^k \\tau_{i-1}}\n  \\big(d_\\primal(x,x^{(0)}) + \\frac{1}{2\\beta} \\|z-z^{(0)}\\|^2\\big)\n  \\label{e-ls-conv} \\\\ \n&\\leq \\frac{1}{k \\tau_\\mathrm{min}}\n  \\big(d_\\primal(x,x^{(0)}) + \\frac{1}{2\\beta} \\|z-z^{(0)}\\|^2\\big)\n  \\label{e-ls-conv-1/k}\n\\end{align}\nfor all $x \\in \\dom f \\cap \\dom \\phi_\\primal$ and all $z$.\nThis holds for any choice of $\\delta \\in (0,1]$ in~\\eqref{e-ls-cond}.\n","start":87404,"end":87881}]},{"id":"S068","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第二项明确允许 arithmetic–geometric mean inequality；第1115行也明确点名。"],"source_only":false,"title":"算术几何均值不等式","source_units":[]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第二项明确允许 Jensen's inequality。"],"source_only":false,"title":"加权Jensen不等式","source_units":[]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第二项明确允许 nonnegativity of Bregman distances。"],"source_only":false,"title":"Bregman距离非负","source_units":[]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；原文第1997–1998、2015–2016行直接使用，未给外部引用。"],"source_only":false,"title":"欧氏三点平方恒等式","source_units":[]},{"id":"S072","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；作者在第1111–1112、1137–1138、1146–1149行直接采用该判据。"],"source_only":false,"title":"距离下界与核凸性判据","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S067","edges":[{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S009","target_id":"S013","kind":"supports"},{"source_id":"S009","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S009","target_id":"S020","kind":"supports"},{"source_id":"S016","target_id":"S041","kind":"supports"},{"source_id":"S014","target_id":"S041","kind":"supports"},{"source_id":"S068","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S039","kind":"supports"},{"source_id":"S019","target_id":"S042","kind":"supports"},{"source_id":"S039","target_id":"S042","kind":"supports"},{"source_id":"S014","target_id":"S042","kind":"supports"},{"source_id":"S072","target_id":"S042","kind":"supports"},{"source_id":"S020","target_id":"S043","kind":"supports"},{"source_id":"S039","target_id":"S043","kind":"supports"},{"source_id":"S015","target_id":"S043","kind":"supports"},{"source_id":"S014","target_id":"S043","kind":"supports"},{"source_id":"S072","target_id":"S043","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S042","target_id":"S040","kind":"supports"},{"source_id":"S043","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S016","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S021","target_id":"S036","kind":"supports"},{"source_id":"S005","target_id":"S036","kind":"supports"},{"source_id":"S008","target_id":"S036","kind":"supports"},{"source_id":"S010","target_id":"S026","kind":"supports"},{"source_id":"S024","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S050","kind":"supports"},{"source_id":"S032","target_id":"S050","kind":"supports"},{"source_id":"S039","target_id":"S045","kind":"supports"},{"source_id":"S015","target_id":"S045","kind":"supports"},{"source_id":"S022","target_id":"S045","kind":"supports"},{"source_id":"S023","target_id":"S045","kind":"supports"},{"source_id":"S029","target_id":"S045","kind":"supports"},{"source_id":"S028","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S045","target_id":"S048","kind":"supports"},{"source_id":"S050","target_id":"S048","kind":"supports"},{"source_id":"S028","target_id":"S048","kind":"supports"},{"source_id":"S030","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S044","kind":"supports"},{"source_id":"S049","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S033","kind":"supports"},{"source_id":"S026","target_id":"S052","kind":"supports"},{"source_id":"S013","target_id":"S052","kind":"supports"},{"source_id":"S011","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S003","target_id":"S053","kind":"supports"},{"source_id":"S027","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S071","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S056","kind":"supports"},{"source_id":"S028","target_id":"S056","kind":"supports"},{"source_id":"S025","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S025","target_id":"S057","kind":"supports"},{"source_id":"S071","target_id":"S057","kind":"supports"},{"source_id":"S053","target_id":"S058","kind":"supports"},{"source_id":"S055","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S036","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S029","target_id":"S059","kind":"supports"},{"source_id":"S030","target_id":"S059","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S051","kind":"supports"},{"source_id":"S055","target_id":"S051","kind":"supports"},{"source_id":"S056","target_id":"S051","kind":"supports"},{"source_id":"S057","target_id":"S051","kind":"supports"},{"source_id":"S058","target_id":"S051","kind":"supports"},{"source_id":"S059","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S034","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S061","kind":"supports"},{"source_id":"S050","target_id":"S061","kind":"supports"},{"source_id":"S029","target_id":"S061","kind":"supports"},{"source_id":"S070","target_id":"S061","kind":"supports"},{"source_id":"S037","target_id":"S062","kind":"supports"},{"source_id":"S036","target_id":"S062","kind":"supports"},{"source_id":"S003","target_id":"S062","kind":"supports"},{"source_id":"S028","target_id":"S062","kind":"supports"},{"source_id":"S069","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S063","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S070","target_id":"S064","kind":"supports"},{"source_id":"S050","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S028","target_id":"S065","kind":"supports"},{"source_id":"S037","target_id":"S065","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S060","kind":"supports"},{"source_id":"S063","target_id":"S060","kind":"supports"},{"source_id":"S064","target_id":"S060","kind":"supports"},{"source_id":"S065","target_id":"S060","kind":"supports"},{"source_id":"S037","target_id":"S067","kind":"supports"},{"source_id":"S032","target_id":"S067","kind":"supports"},{"source_id":"S031","target_id":"S067","kind":"supports"},{"source_id":"S029","target_id":"S067","kind":"supports"},{"source_id":"S036","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S040","target_id":"S066","kind":"supports"},{"source_id":"S044","target_id":"S066","kind":"supports"},{"source_id":"S051","target_id":"S066","kind":"supports"},{"source_id":"S060","target_id":"S066","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S067","edges":[{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S009","target_id":"S013","kind":"supports"},{"source_id":"S009","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S009","target_id":"S020","kind":"supports"},{"source_id":"S016","target_id":"S041","kind":"supports"},{"source_id":"S014","target_id":"S041","kind":"supports"},{"source_id":"S068","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S039","kind":"supports"},{"source_id":"S019","target_id":"S038","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S014","target_id":"S038","kind":"supports"},{"source_id":"S072","target_id":"S038","kind":"supports"},{"source_id":"S020","target_id":"S038","kind":"supports"},{"source_id":"S015","target_id":"S038","kind":"supports"},{"source_id":"S041","target_id":"S038","kind":"supports"},{"source_id":"S016","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S021","target_id":"S036","kind":"supports"},{"source_id":"S005","target_id":"S036","kind":"supports"},{"source_id":"S008","target_id":"S036","kind":"supports"},{"source_id":"S010","target_id":"S026","kind":"supports"},{"source_id":"S024","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S050","kind":"supports"},{"source_id":"S032","target_id":"S050","kind":"supports"},{"source_id":"S039","target_id":"S031","kind":"supports"},{"source_id":"S015","target_id":"S031","kind":"supports"},{"source_id":"S022","target_id":"S031","kind":"supports"},{"source_id":"S023","target_id":"S031","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S028","target_id":"S031","kind":"supports"},{"source_id":"S050","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S026","target_id":"S034","kind":"supports"},{"source_id":"S013","target_id":"S034","kind":"supports"},{"source_id":"S011","target_id":"S034","kind":"supports"},{"source_id":"S003","target_id":"S034","kind":"supports"},{"source_id":"S027","target_id":"S034","kind":"supports"},{"source_id":"S071","target_id":"S034","kind":"supports"},{"source_id":"S028","target_id":"S034","kind":"supports"},{"source_id":"S025","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S034","kind":"supports"},{"source_id":"S029","target_id":"S034","kind":"supports"},{"source_id":"S030","target_id":"S034","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S061","kind":"supports"},{"source_id":"S050","target_id":"S061","kind":"supports"},{"source_id":"S029","target_id":"S061","kind":"supports"},{"source_id":"S070","target_id":"S061","kind":"supports"},{"source_id":"S037","target_id":"S062","kind":"supports"},{"source_id":"S036","target_id":"S062","kind":"supports"},{"source_id":"S003","target_id":"S062","kind":"supports"},{"source_id":"S028","target_id":"S062","kind":"supports"},{"source_id":"S069","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S063","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S070","target_id":"S064","kind":"supports"},{"source_id":"S050","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S028","target_id":"S065","kind":"supports"},{"source_id":"S037","target_id":"S065","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S060","kind":"supports"},{"source_id":"S063","target_id":"S060","kind":"supports"},{"source_id":"S064","target_id":"S060","kind":"supports"},{"source_id":"S065","target_id":"S060","kind":"supports"},{"source_id":"S037","target_id":"S067","kind":"supports"},{"source_id":"S032","target_id":"S067","kind":"supports"},{"source_id":"S031","target_id":"S067","kind":"supports"},{"source_id":"S029","target_id":"S067","kind":"supports"},{"source_id":"S036","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S038","target_id":"S066","kind":"supports"},{"source_id":"S031","target_id":"S066","kind":"supports"},{"source_id":"S034","target_id":"S066","kind":"supports"},{"source_id":"S060","target_id":"S066","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["e-prob"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"复合凸优化问题","source_units":[{"raw_tex":"We discuss proximal splitting methods for optimization problems \nin the form\n\\begin{equation} \\label{e-prob}\n  \\mbox{minimize} \\quad f(x) + g(Ax) + h(x),\n\\end{equation}\nwhere $f$, $g$, and $h$ are convex functions, and $h$ is differentiable.\n","start":2548,"end":2790}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"内积与范数记号","source_units":[{"raw_tex":"We use the notation $\\inprod{x}{y} = x^Ty$ for the\nstandard inner product of vectors $x$ and $y$, and\n$\\|x\\| = \\inprod{x}{x}^{1/2}$ for the Euclidean norm of a vector\n$x$.  Other norms will be distinguished by a subscript.\n\n\\subsection{Problem formulation}\n\n","start":8657,"end":8915}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"函数与维数条件","source_units":[{"raw_tex":"In~\\eqref{e-prob} the vector $x$ is an $n$-vector and\n$A$ is an $m\\times n$ matrix.  The functions $f$, $g$, $h$\nare closed and convex, and $h$ is differentiable, \\ie,  \n\\[\nh(x) \\geq h(x') + \\inprod{\\nabla h(x')}{x-x'} \\quad\n\\mbox{for all $x,x' \\in \\dom h$},\n\\]\nwhere $\\dom h$ is an open convex set.  We assume that $f+h$ and $g$\nare proper, \\ie, have nonempty domains.\n","start":8915,"end":9285}]},{"id":"S004","semantic_type":"problem_setting","kind":"problem_setting","labels":["e-prob-dual"],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"一般对偶问题","source_units":[{"raw_tex":"The dual of problem~\\eqref{e-prob} is\n\\BEQ \\label{e-prob-dual}\n \\mbox{maximize}\\quad  {-(f+h)^*(-A^Tz) - g^*(z)},\n\\EEQ\n","start":10003,"end":10122}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"凸共轭定义","source_units":[{"raw_tex":"where $(f+h)^*$ and $g^*$ are the conjugates of $f+h$ and $g$:\n\\[\n (f+h)^*(w) = \\sup_x{(\\inprod{w}{x} - f(x) - h(x))}, \\qquad\n g^*(z) = \\sup_y{(\\inprod{z}{y} - g(y))}.\n\\]\n","start":10122,"end":10293}]},{"id":"S006","semantic_type":"derived_fact","kind":"derived_fact","labels":["e-opt-cond"],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"原始对偶最优性条件","source_units":[{"raw_tex":"The primal--dual optimality conditions for~\\eqref{e-prob}\nand~\\eqref{e-prob-dual} are\n\\[\n 0 \\in \\partial f(x) + \\nabla h(x) + A^T z, \\qquad\n 0 \\in \\partial g^*(z) - Ax.\n\\]\nHere $\\partial f$ and $\\partial g^*$ are the subdifferentials of\n$f$ and  $g^*$.  We often write the optimality conditions as\n\\begin{equation} \\label{e-opt-cond}\n  0 \\in \\begin{bmatrix} 0 & A^T \\\\ -A & 0 \\end{bmatrix}\n  \\begin{bmatrix} x \\\\ z \\end{bmatrix} + \\begin{bmatrix}\n    \\partial f(x)+\\nabla h(x) \\\\ \\partial g^\\ast(z)\n\\end{bmatrix}.\n\\end{equation}\n","start":10457,"end":10986}]},{"id":"S007","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"最优性系统可解","source_units":[{"raw_tex":"Throughout the paper, we assume that the optimality \nconditions~\\eqref{e-opt-cond} are solvable.\n\n","start":10986,"end":11084}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"一般拉格朗日函数","source_units":[{"raw_tex":"We will refer to the convex--concave function\n\\[\n\\cL(x,z) = f(x)+h(x)+ \\inprod{z}{Ax} -g^\\ast(z)\n\\]\nas the \\emph{Lagrangian} of~\\eqref{e-prob}.\nWe follow the convention that $\\cL(x,z) = +\\infty$ if $x\\not\\in \\dom(f+h)$\nand $\\cL(x,z) = -\\infty$ if $x\\in\\dom(f+h)$ and $z\\not\\in \\dom g^*$.\n","start":11084,"end":11372}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"Bregman核与距离","source_units":[{"raw_tex":"Let $\\phi$ be a convex function with a domain that has nonempty\ninterior, and assume $\\phi$ is continuous on $\\dom \\phi$ and \ncontinuously differentiable on $\\intr(\\dom \\phi)$.\nThe \\emph{generalized distance} (or \\emph{Bregman distance})\ngenerated by the \\emph{kernel function} $\\phi$ is defined as the function\n\\[\nd(x,y) = \\phi(x)-\\phi(y)-\\inprod{\\nabla \\phi(y)}{x-y},\n\\]\nwith domain $\\dom d=\\dom \\phi \\times \\intr(\\dom \\phi)$.\n","start":37479,"end":37908}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":["e-prox-opt","e-prox-opt2"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"Bregman近端映射","source_units":[{"raw_tex":"The corresponding \\emph{Bregman proximal operator} of a function $f$ is \n\\begin{align}\n\\prox_f^\\phi(y,a)\n  &= \\argmin_x{(f(x)+\\inprod{a}{x}+d(x,y))} \\label{e-prox-opt} \\\\\n&= \\argmin_x{(f(x)+\\inprod{a}{x}+\\phi(x)\n   -\\inprod{\\nabla \\phi(y)}{x})}. \\label{e-prox-opt2}\n\\end{align}\n","start":37908,"end":38186}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"近端更新适定性","source_units":[{"raw_tex":"It is assumed that for every $a$ and every $y \\in \\intr(\\dom\\phi)$\nthe minimizer $\\hat x=\\prox_f^\\phi(y,a)$ is unique and in\n$\\intr(\\dom \\phi)$.\n","start":38186,"end":38331}]},{"id":"S012","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"近端次梯度条件","source_units":[{"raw_tex":"From the expression~\\eqref{e-prox-opt2} we see that \n$\\hat x=\\prox_f^\\phi(y,a)$\nsatisfies\n\\[\n\\nabla \\phi(y) - \\nabla \\phi(\\hat x) - a \\in \\partial f(\\hat x).\n\\]\n","start":39609,"end":39770}]},{"id":"S013","semantic_type":"derived_fact","kind":"derived_fact","labels":["e-prox-cond"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"近端三点不等式","source_units":[{"raw_tex":"Equivalently, by definition of subgradient,\n\\begin{align}\nf(x) + \\inprod{a}{x}\n&\\geq f(\\hat x)+\\inprod{a}{\\hat x}\n  +\\inprod{\\nabla\\phi(y)-\\nabla\\phi(\\hat x)}{x-\\hat x} \\nonumber \\\\\n&= f(\\hat x)+\\inprod{a}{\\hat x}+d(\\hat x, y)+d(x,\\hat x)-d(x,y)\n  \\label{e-prox-cond}\n\\end{align}\nfor all $x\\in \\dom f \\cap \\dom \\phi$.\n","start":39770,"end":40088}]},{"id":"S014","semantic_type":"assumption","kind":"assumption","labels":["e-bcv-str-cvx"],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"核的一致强凸性","source_units":[{"raw_tex":"\\paragraph{Assumptions} \nThroughout Section~\\ref{s-bcv} we make the following assumptions.\nThe kernel functions $\\phi_\\primal$ and $\\phi_\\dual$ are \n1-strongly convex with respect to norms $\\|\\cdot\\|_\\primal$ and \n$\\|\\cdot\\|_\\dual$, respectively:\n\\begin{equation} \\label{e-bcv-str-cvx}\nd_\\primal(x,x^\\prime) \\geq \\frac{1}{2}\\|x-x^\\prime\\|^2_\\primal, \\qquad\nd_\\dual(z,z^\\prime) \\geq \\frac{1}{2}\\|z-z^\\prime\\|^2_\\dual\n\\end{equation}\nfor all $(x,x') \\in \\dom d_\\primal$ and $(z,z') \\in \\dom d_\\dual$.\nThe assumption that the strong convexity constants are equal to one\ncan be made without loss of generality, by scaling the norms (or distances)\nif needed.\n","start":42221,"end":42874}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","labels":["e-bcv-lip"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"相对光滑性","source_units":[{"raw_tex":"We also assume that the function $L\\phi_\\primal-h$ is convex for some \n$L>0$.  More precisely, $\\dom \\phi_\\primal \\subseteq \\dom h$ and\n\\BEQ \\label{e-bcv-lip}\nh(x)-h(x^\\prime)-\\inprod{\\nabla h(x^\\prime)}{x-x^\\prime}\n\\leq Ld_\\primal (x,x^\\prime) \\quad \\mbox{for all\\ }\n(x,x^\\prime) \\in \\dom d_\\primal.\n\\EEQ\nNote that this assumption is looser than\nthe one in~\\cite[Equation~(4)]{ChP:16}.\n","start":42874,"end":43261}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":["e-norm"],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"原始对偶矩阵范数","source_units":[{"raw_tex":"We denote by $\\|A\\|$ the matrix norm\n\\BEQ \\label{e-norm}\n\\|A\\|=\\sup_{u \\neq 0, v \\neq 0}\n    \\frac{\\inprod{v}{Au}}{\\|v\\|_\\dual \\|u\\|_\\primal}\n = \\sup_{u\\neq 0} \\frac{\\|Au\\|_{\\dual,*}}{\\|u\\|_\\primal}\n = \\sup_{v\\neq 0} \\frac{\\|A^Tv\\|_{\\primal,*}}{\\|v\\|_\\dual},\n\\EEQ\nwhere $\\|\\cdot\\|_{\\primal,*}$ and $\\|\\cdot\\|_{\\dual,*}$ are\nthe dual norms of $\\|\\cdot\\|_\\primal$ and $\\|\\cdot\\|_\\dual$. \n","start":43261,"end":43647}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"核域内存在最优解","source_units":[{"raw_tex":"\nIt is also assumed that the primal--dual optimality \nconditions~\\eqref{e-opt-cond} have a solution $(x^\\star, z^\\star)$\nwith $x^\\star \\in \\dom \\phi_\\primal$ and \n$z^\\star \\in \\dom \\phi_\\dual$.  \n","start":43647,"end":43843}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"四种耦合核","source_units":[{"raw_tex":"We introduce four possible primal--dual kernel functions: the functions\n\\[\n\\phi_+(x,z) = \\frac{1}{\\tau}\\phi_\\primal(x)\n+\\frac{1}{\\sigma}\\phi_\\dual(z)+\\inprod{z}{Ax}, \\qquad\n\\phi_-(x,z) = \\frac{1}{\\tau}\\phi_\\primal(x)\n+\\frac{1}{\\sigma}\\phi_\\dual(z)-\\inprod{z}{Ax},\n\\]\nwhere $\\sigma,\\tau > 0$, and the functions \n\\[\n \\phi_\\mathrm{dcv}(x,z) = \\phi_+(x,z) - h(x), \\qquad \n \\phi_\\mathrm{pcv}(x,z) = \\phi_-(x,z) - h(x).\n\\]\nThe subscripts in $\\phi_+$ and $\\phi_-$ \nrefer to the sign of the inner product term $\\inprod{z}{Ax}$.\nThe subscripts in $\\phi_\\mathrm{pcv}$ and $\\phi_\\mathrm{dcv}$\nindicate the algorithm (Bregman primal or dual Condat-V\\~u) for which\nthese distances will be relevant. \n","start":45215,"end":45902}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":["e-bcv-dminus"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"正负耦合距离","source_units":[{"raw_tex":"If these kernel functions are convex,\nthey generate the following Bregman distances.\nThe distances generated by $\\phi_+$ and $\\phi_-$ are\n\\begin{align}\nd_+ (x,z;x^\\prime,z^\\prime) &= \n  \\frac{1}{\\tau}d_\\primal(x,x^\\prime)+\\frac{1}{\\sigma}d_\\dual(z,z^\\prime)\n  +\\inprod{z-z^\\prime}{A(x-x^\\prime)} \\nonumber \\\\\nd_-(x,z;x^\\prime,z^\\prime) &=\n  \\frac{1}{\\tau}d_\\primal(x,x^\\prime)+\\frac{1}{\\sigma}d_\\dual(z,z^\\prime)\n  -\\inprod{z-z^\\prime}{A(x-x^\\prime)}, \\label{e-bcv-dminus}\n\\end{align}\n","start":45902,"end":46387}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"减去光滑项的距离","source_units":[{"raw_tex":"respectively, and the distances generated by $\\phi_\\mathrm{dcv}$ and \n$\\phi_\\mathrm{pcv}$ are  \n\\begin{align*}\nd_\\mathrm{dcv}(x,z;x^\\prime,z^\\prime) &= \n  d_+ (x,z;x^\\prime,z^\\prime) \n  -h(x)+h(x^\\prime)+\\inprod{\\nabla h(x')}{x-x^\\prime} \\\\\nd_\\mathrm{pcv}(x,z;x^\\prime,z^\\prime) &= \n  d_- (x,z;x^\\prime,z^\\prime) \n  -h(x)+h(x^\\prime)+\\inprod{\\nabla h(x')}{x-x^\\prime}.\n\\end{align*}\n","start":46387,"end":46769}]},{"id":"S021","semantic_type":"problem_setting","kind":"problem_setting","labels":["e-lcp-prob"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"等式约束专门问题","source_units":[{"raw_tex":"The algorithm is restricted to problems in the equality constrained\nform\n\\begin{equation} \\label{e-lcp-prob}\n \\begin{array}{ll}\n   \\mbox{minimize} & f(x) + h(x) \\\\\n   \\mbox{subject to} & Ax=b.\n \\end{array}\n\\end{equation}\nThis is a special case of~\\eqref{e-prob} with $g=\\delta_{\\{b\\}}$,\nthe indicator function of the singleton $\\{b\\}$.\n","start":76128,"end":76464}]},{"id":"S022","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"对偶空间采用欧氏核","source_units":[{"raw_tex":"\\paragraph{Assumptions}\nWe make the same assumptions as in Section~\\ref{s-bcv-ppm}, but define\n\\[\n\\phi_\\dual (z) = \\frac{1}{2} \\|z\\|^2, \\qquad\nd_\\dual (z,z') = \\frac{1}{2} \\|z-z'\\|^2, \\qquad\n\\|z\\|_\\dual = \\|z\\|,\n\\]\nwhere $\\|\\cdot\\|$ is the Euclidean norm.\n","start":76744,"end":77000}]},{"id":"S023","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"线搜索中的矩阵范数","source_units":[{"raw_tex":"The matrix norm $\\|A\\|$ is defined accordingly as\n\\[\n\\|A\\|=\\sup_{u \\neq 0, v \\neq 0}\n  \\frac{\\inprod{v}{Au}}{\\|v\\| \\|u\\|_\\primal}\n= \\sup_{u \\neq 0} \\frac{\\|Au\\|}{\\|u\\|_\\primal}\n= \\sup_{v \\neq 0} \\frac{\\|A^Tv\\|_{\\primal,*}}{\\|v\\|}.\n\\]\n","start":77000,"end":77234}]},{"id":"S024","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"迭代初值","source_units":[{"raw_tex":"The algorithm uses the following iteration,\nwith starting points $x^{(0)} \\in \\intr(\\dom \\phi_\\primal) \\cap \\dom h$\nand $z^{(-1)} = z^{(0)}$:\n\\begin{subequations} \\label{e-ls}\n","start":77277,"end":77453}]},{"id":"S025","semantic_type":"algorithm","kind":"algorithm","labels":["e-ls-a","e-ls-b","e-ls-c"],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"对偶外推更新","source_units":[{"raw_tex":"\\begin{align}\n\\bar z^{(k+1)} &= z^{(k)} + \\theta_k (z^{(k)} - z^{(k-1)}) \n  \\label{e-ls-a} \\\\\n","start":77453,"end":77547}]},{"id":"S026","semantic_type":"algorithm","kind":"algorithm","labels":["e-ls-a","e-ls-b","e-ls-c"],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"原始近端更新","source_units":[{"raw_tex":"x^{(k+1)} &= \\prox_{\\tau_k f}^{\\phi_\\primal} \\big(\n  x^{(k)}, \\tau_k (A^T\\bar z^{(k+1)}+\\nabla h(x^{(k)}))\n  \\big) \\label{e-ls-b} \\\\\n","start":77547,"end":77680}]},{"id":"S027","semantic_type":"algorithm","kind":"algorithm","labels":[],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"对偶残差更新","source_units":[{"raw_tex":"z^{(k+1)} &= z^{(k)} + \\sigma_k (Ax^{(k+1)}-b). \\label{e-ls-c}\n\\end{align}\n\\end{subequations}\n","start":77680,"end":77774}]},{"id":"S028","semantic_type":"algorithm","kind":"algorithm","labels":[],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"回溯参数生成","source_units":[{"raw_tex":"In the line search algorithm, the parameters \n$\\theta_k$, $\\tau_k$, $\\sigma_k$ are determined by a\nbacktracking search.\nAt the start of the algorithm, we set $\\tau_{-1}$ and $\\sigma_{-1}$ to\nsome positive values.\nTo start the search in iteration $k$ we choose\n$\\bar\\theta_k \\geq 1$.  For $i=0,1,2,\\ldots$,\nwe set $\\theta_k = 2^{-i}\\bar\\theta_k$,\n$\\tau_k=\\theta_k\\tau_{k-1}$, $\\sigma_k=\\theta_k\\sigma_{k-1}$,\nand compute $\\bar z_{k+1}$, $x_{k+1}$, $z_{k+1}$ using~\\eqref{e-ls}.\n","start":78232,"end":78709}]},{"id":"S029","semantic_type":"algorithm","kind":"algorithm","labels":["e-ls-cond"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"回溯接受不等式","source_units":[{"raw_tex":"For some $\\delta \\in (0,1]$, if\n\\BEA\n& & \\inprod{z^{(k+1)}-\\bar z^{(k+1)}}{A(x^{(k+1)}-x^{(k)})}\n  +h(x^{(k+1)})-h(x^{(k)})-\\inprod{\\nabla h(x^{(k)})}{x^{(k+1)}-x^{(k)}}\n  \\nonumber \\\\\n&\\leq & \\frac{\\delta^2}{\\tau_k} d_\\primal(x^{(k+1)}, x^{(k)})\n    + \\frac{1}{2\\sigma_k} \\|\\bar z^{(k+1)} - z^{(k+1)}\\|^2,\n    \\label{e-ls-cond}\n\\EEA\n","start":78709,"end":79043}]},{"id":"S030","semantic_type":"algorithm","kind":"algorithm","labels":[],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"接受与减半规则","source_units":[{"raw_tex":"we accept the computed iterates $\\bar z^{(k+1)}$, $x^{(k+1)}$, $z^{(k+1)}$\nand parameters $\\theta_k$, $\\sigma_k$, $\\tau_k$, and terminate the \nbacktracking search.\nIf~\\eqref{e-ls-cond} does not hold, we increment $i$ and continue\nthe backtracking search.\n","start":79043,"end":79298}]},{"id":"S031","semantic_type":"lemma","kind":"lemma","labels":["e-ls-param-lbnd"],"member_ids":["S031","S044","S045","S046","S047","S048","S049"],"declared_external":false,"source_only":false,"title":"步长统一正下界","source_units":[{"raw_tex":"\\BEQ \\label{e-ls-param-lbnd}\n  \\tau_k \\geq \\tau_\\mathrm{min} \\triangleq \\min{\\Big\\{\\tau_{-1},\n  \\frac{-L+\\sqrt{L^2+4\\delta^2\\beta\\|A\\|^2}}{4\\beta\\|A\\|^2}\\Big\\}},\n  \\qquad\n  \\sigma_k \\geq \\sigma_\\mathrm{min} \\triangleq \\beta\\tau_\\mathrm{min},\n\\EEQ","start":80009,"end":80255},{"raw_tex":"\\begin{proof}\nApplying~\\eqref{e-bcv-delta} with $\\tau=\\tau_k$, $\\sigma=\\sigma_k$, $\\delta_1=\\delta^2-\\tau_k L$ and $\\delta_2=1$,\ntogether with the Lipschitz condition~\\eqref{e-bcv-lip},\nwe see that the backtracking condition~\\eqref{e-ls-cond} holds\nat iteration $k$ if $0 < \\delta < 1$ and\n\\[\n\\tau_k \\sigma_k\\|A\\|^2 + \\tau_k L \\leq \\delta^2.\n\\]\n","start":80409,"end":80754},{"raw_tex":"\\begin{proof}\nApplying~\\eqref{e-bcv-delta} with $\\tau=\\tau_k$, $\\sigma=\\sigma_k$, $\\delta_1=\\delta^2-\\tau_k L$ and $\\delta_2=1$,\ntogether with the Lipschitz condition~\\eqref{e-bcv-lip},\nwe see that the backtracking condition~\\eqref{e-ls-cond} holds\nat iteration $k$ if $0 < \\delta < 1$ and\n\\[\n\\tau_k \\sigma_k\\|A\\|^2 + \\tau_k L \\leq \\delta^2.\n\\]\nThen mathematical induction can be used to prove~\\eqref{e-ls-param-lbnd}.\nThe two lower bounds~\\eqref{e-ls-param-lbnd} hold at $k=0$\nby the definition of $\\tau_\\mathrm{min}$ and $\\sigma_\\mathrm{min}$.\nNow assume $\\tau_{k-1} \\geq \\tau_\\mathrm{min}$,\n$\\sigma_{k-1} \\geq \\sigma_\\mathrm{min}$,\nand consider the $k$th iteration.\nThe first attempt of $\\theta_k$ is $\\theta_k=\\bar \\theta_k \\geq 1$.\nIf this value is accepted, then\n\\[\n\\tau_k = \\bar\\theta_k \\tau_{k-1} \\geq \\tau_{k-1} \\geq \\tau_\\mathrm{min},\n\\qquad\n\\sigma_k = \\bar\\theta_k \\sigma_{k-1} \\geq \\sigma_{k-1} \\geq\n  \\sigma_\\mathrm{min}.\n\\]\nOtherwise, one or more backtracking steps are needed.\nDenote by $\\tilde \\theta_k$ the last rejected value.\nThen $\\tilde\\theta_k^2\\tau_{k-1}^2\\beta\\|A\\|^2 + \n\\tilde\\theta_k \\tau_{k-1} L >\\delta^2$\nand the accepted $\\theta_k$ satisfies\n\\[\n\\theta_k = \\frac{\\tilde \\theta_k}{2} \\geq\n\\frac{-L+\\sqrt{L^2+4\\delta^2\\beta\\|A\\|^2}}%\n  {4\\tau_{k-1}\\beta\\|A\\|^2}.\n\\]\nTherefore,\n\\[\n\\tau_k=\\theta_k\\tau_{k-1} >\n\\frac{-L+\\sqrt{L^2+4\\delta^2\\beta\\|A\\|^2}}{4\\beta\\|A\\|^2}, \\qquad\n\\sigma_k=\\beta\\tau_k \\geq \\beta\\tau_\\mathrm{min}. \\qedhere\n\\]\n\\end{proof}","start":80409,"end":81883},{"raw_tex":"Then mathematical induction can be used to prove~\\eqref{e-ls-param-lbnd}.\nThe two lower bounds~\\eqref{e-ls-param-lbnd} hold at $k=0$\nby the definition of $\\tau_\\mathrm{min}$ and $\\sigma_\\mathrm{min}$.\nNow assume $\\tau_{k-1} \\geq \\tau_\\mathrm{min}$,\n$\\sigma_{k-1} \\geq \\sigma_\\mathrm{min}$,\nand consider the $k$th iteration.\n","start":80754,"end":81078},{"raw_tex":"The first attempt of $\\theta_k$ is $\\theta_k=\\bar \\theta_k \\geq 1$.\nIf this value is accepted, then\n\\[\n\\tau_k = \\bar\\theta_k \\tau_{k-1} \\geq \\tau_{k-1} \\geq \\tau_\\mathrm{min},\n\\qquad\n\\sigma_k = \\bar\\theta_k \\sigma_{k-1} \\geq \\sigma_{k-1} \\geq\n  \\sigma_\\mathrm{min}.\n\\]\n","start":81078,"end":81347},{"raw_tex":"Otherwise, one or more backtracking steps are needed.\nDenote by $\\tilde \\theta_k$ the last rejected value.\nThen $\\tilde\\theta_k^2\\tau_{k-1}^2\\beta\\|A\\|^2 + \n\\tilde\\theta_k \\tau_{k-1} L >\\delta^2$\nand the accepted $\\theta_k$ satisfies\n\\[\n\\theta_k = \\frac{\\tilde \\theta_k}{2} \\geq\n\\frac{-L+\\sqrt{L^2+4\\delta^2\\beta\\|A\\|^2}}%\n  {4\\tau_{k-1}\\beta\\|A\\|^2}.\n\\]\n","start":81347,"end":81702},{"raw_tex":"Therefore,\n\\[\n\\tau_k=\\theta_k\\tau_{k-1} >\n\\frac{-L+\\sqrt{L^2+4\\delta^2\\beta\\|A\\|^2}}{4\\beta\\|A\\|^2}, \\qquad\n\\sigma_k=\\beta\\tau_k \\geq \\beta\\tau_\\mathrm{min}. \\qedhere\n\\]\n\\end{proof}","start":81702,"end":81883}]},{"id":"S032","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"初始步长比","source_units":[{"raw_tex":"where $\\beta=\\sigma_{-1}/\\tau_{-1}$.\n","start":80256,"end":80293}]},{"id":"S034","semantic_type":"lemma","kind":"lemma","labels":["e-ls-conv-i"],"member_ids":["S034","S051","S052","S053","S054","S055","S056","S057","S058","S059"],"declared_external":false,"source_only":false,"title":"线搜索单步不等式","source_units":[{"raw_tex":"\\begin{align}\n\\cL(x^{(k+1)},z)-\\cL(x, \\bar z^{(k+1)})\n  &\\leq \\frac{1}{\\tau_k} \n  \\left(d_\\primal(x,x^{(k)}) - d_\\primal(x,x^{(k+1)}) -\n  (1-\\delta^2) d_\\primal(x^{(k+1)},x^{(k)})\\right) \\nonumber \\\\\n&\\phantom{=} + \\frac{1}{2\\sigma_k} \\left(\n  \\|z-z^{(k)}\\|^2 - \\|z-z^{(k+1)}\\|^2 - \n  \\|\\bar z^{(k+1)} - z^{(k)}\\|^2 \\right) \\label{e-ls-conv-i}\n\\end{align}","start":82048,"end":82403},{"raw_tex":"\\begin{proof}\nThe optimality condition for the primal prox-operator~\\eqref{e-ls-b}\ngives\n\\BEAS\n\\lefteqn{f(x^{(k+1)})-f(x)} \\\\\n&\\leq & \\frac{1}{\\tau_k}\n  \\big(d_\\primal(x,x^{(k)})-d_\\primal(x,x^{(k+1)})-\n  d_\\primal(x^{(k+1)}, x^{(k)})\\big)\n  +\\inprod{A^T\\bar z^{(k+1)}+\\nabla h(x^{(k)})}{x-x^{(k+1)}}\n\\EEAS\nfor all $x\\in \\dom f \\cap \\dom\\phi_\\primal$.\n","start":82509,"end":82861},{"raw_tex":"\\begin{proof}\nThe optimality condition for the primal prox-operator~\\eqref{e-ls-b}\ngives\n\\BEAS\n\\lefteqn{f(x^{(k+1)})-f(x)} \\\\\n&\\leq & \\frac{1}{\\tau_k}\n  \\big(d_\\primal(x,x^{(k)})-d_\\primal(x,x^{(k+1)})-\n  d_\\primal(x^{(k+1)}, x^{(k)})\\big)\n  +\\inprod{A^T\\bar z^{(k+1)}+\\nabla h(x^{(k)})}{x-x^{(k+1)}}\n\\EEAS\nfor all $x\\in \\dom f \\cap \\dom\\phi_\\primal$.\nHence\n\\BEA\n\\lefteqn{f(x^{(k+1)}) + h(x^{(k+1)}) - f(x) - h(x)} \\nonumber \\\\\n  &\\leq & \\frac{1}{\\tau_k} (d_\\primal(x,x^{(k)}) - \n  d_\\primal(x,x^{(k+1)}) - d_\\primal(x^{(k+1)}, x^{(k)}))\n  + \\inprod{A^T\\bar z_{k+1}}{x - x^{(k+1)}} \\nonumber \\\\\n& &\\mbox{} + h(x^{(k+1)})-h(x)+\\inprod{\\nabla h(x^{(k)})}{x-x^{(k+1)}}\n  \\nonumber \\\\\n&\\leq & \\frac{1}{\\tau_k} (d_\\primal(x,x^{(k)}) - \n d_\\primal(x,x^{(k+1)}) - d_\\primal(x^{(k+1)}, x^{(k)}))\n  + \\inprod{A^T\\bar z^{(k+1)}}{x - x^{(k+1)}} \\nonumber \\\\\n& &\\mbox{} +h(x^{(k+1)})-h(x^{(k)})\n  -\\inprod{\\nabla h(x^{(k)})}{x^{(k+1)}-x^{(k)}}.\n  \\label{e-ls-conv-1}\n\\EEA\nThe second inequality follows from the convexity of $h$, \\ie,\n$h(x) \\geq h(x^{(k)}) + \\inprod{\\nabla h(x^{(k)})}{x-x^{(k)}}$.\nThe dual update~\\eqref{e-ls-c} implies that\n\\BEQ \\label{e-ls-conv-2}\n\\inprod{z-z^{(k+1)}}{Ax^{(k+1)}-b}\n= \\frac{1}{\\sigma_k} \\inprod{z-z^{(k+1)}}{z^{(k+1)}-z^{(k)}}\n\\quad \\mbox{for all $z$.}\n\\EEQ\nThis equality at $k=i-1$ is\n\\begin{align}\n\\inprod{z-z^{(i)}}{Ax^{(i)}-b}\n  &= \\frac{1}{\\sigma_{i-1}} \\inprod{z-z^{(i)}}{z^{(i)}-z^{(i-1)}} \n  \\nonumber \\\\\n&= \\frac{1}{2\\sigma_{i-1}} \\left( \\|z-z^{(i-1)}\\|^2\n  - \\|z-z^{(i)}\\|^2 - \\|z^{(i)} - z^{(i-1)} \\|^2 \\right).\n  \\label{e-ls-conv-3}\n\\end{align}\nThe equality~\\eqref{e-ls-conv-2} at $k=i-2$ is\n\\begin{align*}\n\\inprod{z-z^{(i-1)}}{Ax^{(i-1)}-b}\n  &= \\frac{1}{\\sigma_{i-2}} \\inprod{z-z^{(i-1)}}{z^{(i-1)}-z^{(i-2)}} \\\\\n&= \\frac{\\theta_{i-1}}{\\sigma_{i-1}}\n  \\inprod{z-z^{(i-1)}}{z^{(i-1)}-z^{(i-2)}} \\\\\n&= \\frac{1}{\\sigma_{i-1}} \\inprod{z-z^{(i-1)}}{\\bar z^{(i)}-z^{(i-1)}}.\n\\end{align*}\nWe evaluate this at $z=z^{(i)}$ and add it to the equality\nat $z=z^{(i-2)}$ multiplied by $\\theta_{i-1}$:\n\\begin{align}\n\\inprod{z^{(i)}-\\bar z^{(i)}}{Ax^{(i-1)}-b}\n  &= \\frac{1}{\\sigma_{i-1}}\n  \\inprod{z^{(i)}-\\bar z^{(i)}}{\\bar z^{(i)} - z^{(i-1)}} \\nonumber \\\\\n&= \\frac{1}{2\\sigma_{i-1}} \\left(\\|z^{(i)}-z^{(i-1)}\\|^2\n  -\\|z^{(i)}-\\bar z^{(i)}\\|^2-\\|\\bar z^{(i)}-z^{(i-1)}\\|^2 \\right).\n  \\label{e-ls-conv-4}\n\\end{align}\nNow we combine~\\eqref{e-ls-conv-1} for $k=i-1$,\nwith~\\eqref{e-ls-conv-3} and~\\eqref{e-ls-conv-4}. For $i \\geq 1$,\n\\BEAS\n\\lefteqn{\\cL(x^{(i)},z) - \\cL(x, \\bar z^{(i)})} \\nonumber \\\\\n&=& f(x^{(i)})+h(x^{(i)})+\\inprod{z}{Ax^{(i)}-b}\n  -f(x)-h(x)-\\inprod{\\bar z^{(i)}}{Ax-b} \\nonumber \\\\\n&\\leq & \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)})-d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big)\n  + \\inprod{A^T\\bar z^{(i)}}{x-x^{(i)}} + \\inprod{z}{Ax^{(i)}-b} \\nonumber\\\\\n& &\\mbox{} - \\inprod{\\bar z^{(i)}}{Ax-b} + h(x^{(i)}) - h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)}) \n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big)\n  + \\inprod{z-\\bar z^{(i)}}{Ax^{(i)}-b} \\nonumber \\\\\n& &\\mbox{} +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big) \\nonumber \\\\\n& & \\mbox{} +\\inprod{z^{(i)}-\\bar z^{(i)}}{A(x^{(i)}-x^{(i-1)})}\n  + \\inprod{z-z^{(i)}}{Ax^{(i)}-b}\n  + \\inprod{z^{(i)}-\\bar z^{(i)}}{Ax^{(i-1)}-b} \\nonumber \\\\\n& &\\mbox{} +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)}))\\Big) \\nonumber \\\\\n& &\\mbox{} +\\frac{1}{2\\sigma_{i-1}}\n  \\Big(\\|z-z^{(i-1)}\\|^2-\\|z-z^{(i)}\\|^2-\\|\\bar z^{(i)}-z^{(i-1)}\\|^2\n  -\\|\\bar z^{(i)}-z^{(i)}\\|^2\\Big) \\nonumber \\\\\n& &\\mbox{}+\\inprod{A^T(z^{(i)}-\\bar z^{(i)})}{x^{(i)}-x^{(i-1)}}\n  +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)})} \\\\\n&\\leq & \\frac{1}{\\tau_{i-1}} \n  \\left(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)}) -\n  (1-\\delta^2) d_\\primal(x^{(i)},x^{(i-1)})\\right) \\nonumber \\\\\n& &\\mbox{} + \\frac{1}{2\\sigma_{i-1}} \\left(\n  \\|z-z^{(i-1)}\\|^2 - \\|z-z^{(i)}\\|^2 - \n  \\|\\bar z^{(i)} - z^{(i-1)}\\|^2 \\right),\n\\EEAS\nwhich is the desired result~\\eqref{e-ls-conv-i}.\nThe first inequality follows from~\\eqref{e-ls-conv-1}.\nIn the second last step we substitute~\\eqref{e-ls-conv-3}\nand~\\eqref{e-ls-conv-4}.\nThe last step uses the line search exit condition~\\eqref{e-ls-cond}\nat $k=i-1$.\n\\end{proof}","start":82509,"end":87115},{"raw_tex":"Hence\n\\BEA\n\\lefteqn{f(x^{(k+1)}) + h(x^{(k+1)}) - f(x) - h(x)} \\nonumber \\\\\n  &\\leq & \\frac{1}{\\tau_k} (d_\\primal(x,x^{(k)}) - \n  d_\\primal(x,x^{(k+1)}) - d_\\primal(x^{(k+1)}, x^{(k)}))\n  + \\inprod{A^T\\bar z_{k+1}}{x - x^{(k+1)}} \\nonumber \\\\\n& &\\mbox{} + h(x^{(k+1)})-h(x)+\\inprod{\\nabla h(x^{(k)})}{x-x^{(k+1)}}\n  \\nonumber \\\\\n&\\leq & \\frac{1}{\\tau_k} (d_\\primal(x,x^{(k)}) - \n d_\\primal(x,x^{(k+1)}) - d_\\primal(x^{(k+1)}, x^{(k)}))\n  + \\inprod{A^T\\bar z^{(k+1)}}{x - x^{(k+1)}} \\nonumber \\\\\n& &\\mbox{} +h(x^{(k+1)})-h(x^{(k)})\n  -\\inprod{\\nabla h(x^{(k)})}{x^{(k+1)}-x^{(k)}}.\n  \\label{e-ls-conv-1}\n\\EEA\nThe second inequality follows from the convexity of $h$, \\ie,\n$h(x) \\geq h(x^{(k)}) + \\inprod{\\nabla h(x^{(k)})}{x-x^{(k)}}$.\n","start":82861,"end":83595},{"raw_tex":"The dual update~\\eqref{e-ls-c} implies that\n\\BEQ \\label{e-ls-conv-2}\n\\inprod{z-z^{(k+1)}}{Ax^{(k+1)}-b}\n= \\frac{1}{\\sigma_k} \\inprod{z-z^{(k+1)}}{z^{(k+1)}-z^{(k)}}\n\\quad \\mbox{for all $z$.}\n\\EEQ\n","start":83595,"end":83791},{"raw_tex":"This equality at $k=i-1$ is\n\\begin{align}\n\\inprod{z-z^{(i)}}{Ax^{(i)}-b}\n  &= \\frac{1}{\\sigma_{i-1}} \\inprod{z-z^{(i)}}{z^{(i)}-z^{(i-1)}} \n  \\nonumber \\\\\n&= \\frac{1}{2\\sigma_{i-1}} \\left( \\|z-z^{(i-1)}\\|^2\n  - \\|z-z^{(i)}\\|^2 - \\|z^{(i)} - z^{(i-1)} \\|^2 \\right).\n  \\label{e-ls-conv-3}\n\\end{align}\n","start":83791,"end":84090},{"raw_tex":"The equality~\\eqref{e-ls-conv-2} at $k=i-2$ is\n\\begin{align*}\n\\inprod{z-z^{(i-1)}}{Ax^{(i-1)}-b}\n  &= \\frac{1}{\\sigma_{i-2}} \\inprod{z-z^{(i-1)}}{z^{(i-1)}-z^{(i-2)}} \\\\\n&= \\frac{\\theta_{i-1}}{\\sigma_{i-1}}\n  \\inprod{z-z^{(i-1)}}{z^{(i-1)}-z^{(i-2)}} \\\\\n&= \\frac{1}{\\sigma_{i-1}} \\inprod{z-z^{(i-1)}}{\\bar z^{(i)}-z^{(i-1)}}.\n\\end{align*}\n","start":84090,"end":84429},{"raw_tex":"We evaluate this at $z=z^{(i)}$ and add it to the equality\nat $z=z^{(i-2)}$ multiplied by $\\theta_{i-1}$:\n\\begin{align}\n\\inprod{z^{(i)}-\\bar z^{(i)}}{Ax^{(i-1)}-b}\n  &= \\frac{1}{\\sigma_{i-1}}\n  \\inprod{z^{(i)}-\\bar z^{(i)}}{\\bar z^{(i)} - z^{(i-1)}} \\nonumber \\\\\n&= \\frac{1}{2\\sigma_{i-1}} \\left(\\|z^{(i)}-z^{(i-1)}\\|^2\n  -\\|z^{(i)}-\\bar z^{(i)}\\|^2-\\|\\bar z^{(i)}-z^{(i-1)}\\|^2 \\right).\n  \\label{e-ls-conv-4}\n\\end{align}\n","start":84429,"end":84851},{"raw_tex":"Now we combine~\\eqref{e-ls-conv-1} for $k=i-1$,\nwith~\\eqref{e-ls-conv-3} and~\\eqref{e-ls-conv-4}. For $i \\geq 1$,\n\\BEAS\n\\lefteqn{\\cL(x^{(i)},z) - \\cL(x, \\bar z^{(i)})} \\nonumber \\\\\n&=& f(x^{(i)})+h(x^{(i)})+\\inprod{z}{Ax^{(i)}-b}\n  -f(x)-h(x)-\\inprod{\\bar z^{(i)}}{Ax-b} \\nonumber \\\\\n&\\leq & \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)})-d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big)\n  + \\inprod{A^T\\bar z^{(i)}}{x-x^{(i)}} + \\inprod{z}{Ax^{(i)}-b} \\nonumber\\\\\n& &\\mbox{} - \\inprod{\\bar z^{(i)}}{Ax-b} + h(x^{(i)}) - h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)}) \n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big)\n  + \\inprod{z-\\bar z^{(i)}}{Ax^{(i)}-b} \\nonumber \\\\\n& &\\mbox{} +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big) \\nonumber \\\\\n& & \\mbox{} +\\inprod{z^{(i)}-\\bar z^{(i)}}{A(x^{(i)}-x^{(i-1)})}\n  + \\inprod{z-z^{(i)}}{Ax^{(i)}-b}\n  + \\inprod{z^{(i)}-\\bar z^{(i)}}{Ax^{(i-1)}-b} \\nonumber \\\\\n& &\\mbox{} +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)}))\\Big) \\nonumber \\\\\n& &\\mbox{} +\\frac{1}{2\\sigma_{i-1}}\n  \\Big(\\|z-z^{(i-1)}\\|^2-\\|z-z^{(i)}\\|^2-\\|\\bar z^{(i)}-z^{(i-1)}\\|^2\n  -\\|\\bar z^{(i)}-z^{(i)}\\|^2\\Big) \\nonumber \\\\\n& &\\mbox{}+\\inprod{A^T(z^{(i)}-\\bar z^{(i)})}{x^{(i)}-x^{(i-1)}}\n  +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)})} \\\\\n","start":84851,"end":86554},{"raw_tex":"&\\leq & \\frac{1}{\\tau_{i-1}} \n  \\left(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)}) -\n  (1-\\delta^2) d_\\primal(x^{(i)},x^{(i-1)})\\right) \\nonumber \\\\\n& &\\mbox{} + \\frac{1}{2\\sigma_{i-1}} \\left(\n  \\|z-z^{(i-1)}\\|^2 - \\|z-z^{(i)}\\|^2 - \n  \\|\\bar z^{(i)} - z^{(i-1)}\\|^2 \\right),\n\\EEAS\nwhich is the desired result~\\eqref{e-ls-conv-i}.\nThe first inequality follows from~\\eqref{e-ls-conv-1}.\nIn the second last step we substitute~\\eqref{e-ls-conv-3}\nand~\\eqref{e-ls-conv-4}.\nThe last step uses the line search exit condition~\\eqref{e-ls-cond}\nat $k=i-1$.\n\\end{proof}","start":86554,"end":87115}]},{"id":"S035","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S035"],"declared_external":false,"source_only":true,"title":"单步界的比较点范围","source_units":[{"raw_tex":"for all $x \\in \\dom f \\cap \\dom \\phi_\\primal$ and all $z$.\n","start":82404,"end":82463}]},{"id":"S036","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S036"],"declared_external":false,"source_only":true,"title":"等式约束拉格朗日函数","source_units":[{"raw_tex":"Here $\\cL(x,z)=f(x)+h(x)+\\inprod{z}{Ax-b}$.  \n","start":82463,"end":82509}]},{"id":"S037","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"步长加权遍历平均","source_units":[{"raw_tex":"We define the averaged primal and dual sequences\n\\[\nx^{(k)}_\\avg = \\frac{1}{\\sum_{i=1}^k \\tau_{i-1}}\n\\sum_{i=1}^k \\tau_{i-1}x^{(i)}, \\qquad\n\\bar z^{(k)}_\\avg = \\frac{1}{\\sum_{i=1}^k \\tau_{i-1}}\n\\sum_{i=1}^k \\tau_{i-1} \\bar z^{(i)}\n\\]\nfor $k \\geq 1$.\n","start":87154,"end":87404}]},{"id":"S038","semantic_type":"proposition","kind":"proposition","labels":["e-bcv-param"],"member_ids":["S038","S040","S042","S043"],"declared_external":false,"source_only":true,"title":"耦合核的凸性条件","source_units":[{"raw_tex":"We now show that $\\phi_+$ and $\\phi_-$ are convex if\n\\[\n\\sigma\\tau \\|A\\|^2 \\leq 1 \n\\]\nand strongly convex if $\\sigma\\tau\\|A\\|^2 < 1$,\nand that the functions $\\phi_\\mathrm{dcv}$ and \n$\\phi_\\mathrm{pcv}$ are convex if\n\\BEQ \\label{e-bcv-param}\n\\sigma\\tau \\|A\\|^2 + \\tau L \\leq 1\n\\EEQ\nand strongly convex if $\\sigma\\tau \\|A\\|^2 + \\tau L < 1$.\n","start":46769,"end":47108},{"raw_tex":"\\begin{proof}\nTo show that the kernel functions $\\phi_+$ and $\\phi_-$ are convex,\nwe show that $d_+$ and $d_-$ are nonnegative.\nSuppose $\\sigma\\tau \\|A\\|^2 \\leq \\delta_1 \\delta_2$\nwith $\\delta_1,\\delta_2 > 0$.\nThen~\\eqref{e-bcv-str-cvx} and the arithmetic--geometric mean inequality\nimply that\n\\begin{align}\n\\left| \\inprod{z-z^\\prime}{A(x-x^\\prime)}\\right|\n  &\\leq \\|A\\| \\|z-z'\\|_\\dual \\|x-x'\\|_\\primal  \\nonumber \\\\\n&\\leq \\sqrt{\\frac{\\delta_1 \\delta_2}{\\sigma \\tau}}\n\\|z-z'\\|_\\dual \\|x-x'\\|_\\primal \\nonumber \\\\\n&\\leq \\frac{\\delta_1}{2\\tau} \\|x-x'\\|_\\primal^2  +\n  \\frac{\\delta_2}{2\\sigma} \\|z-z'\\|_\\primal^2 \\nonumber \\\\\n&\\leq \\frac{\\delta_1}{\\tau} d_\\primal (x,x') +\n  \\frac{\\delta_2}{\\sigma} d_\\dual (z,z'). \\label{e-bcv-delta}\n\\end{align}\nTherefore,\n\\begin{align*}\nd_{\\pm} (x,z; x',z')\n  &= \\frac{1}{\\tau} d_\\primal(x,x^\\prime) + \\frac{1}{\\sigma}\n  d_\\dual(z,z^\\prime) \\pm \\inprod{z-z^\\prime}{A(x-x^\\prime)} \\\\\n&\\geq \\frac{1-\\delta_1}{\\tau} d_\\primal (x,x') +\n  \\frac{1-\\delta_2}{\\sigma} d_\\dual (z,z') \\\\\n&\\geq \\frac{1-\\delta_1}{2\\tau} \\|x-x'\\|_\\primal^2 +\n  \\frac{1-\\delta_2}{2\\sigma} \\|z-z'\\|_\\dual^2.\n\\end{align*}\nWith $\\delta_1=\\delta_2=1$, this shows convexity of $\\phi_+$ and $\\phi_-$;\nwith $\\delta_1=\\delta_2<1$, strong convexity.\nSimilarly,\n\\begin{align*}\nd_\\mathrm{dcv/pcv} (x,z; x', z')\n  &= d_{\\pm} (x,z; x',z') - h(x) + h(x') + \\inprod{\\nabla h(x')}{x-x'} \\\\\n&\\geq \\frac{1-\\tau L-\\delta_1}{\\tau} d_\\primal (x,x^\\prime) +\n  \\frac{1-\\delta_2}{\\sigma} d_\\dual (z,z^\\prime).\n\\end{align*}\nWith $\\delta_1=1-\\tau L$ and $\\delta_2=1$, this shows convexity of\n$\\phi_\\mathrm{pcv}$ and $\\phi_\\mathrm{dcv}$;\nwith $\\delta_1=\\delta-\\tau L$ and $\\delta_2=\\delta < 1$,\nstrong convexity.\n\\end{proof}","start":47108,"end":48808},{"raw_tex":"Therefore,\n\\begin{align*}\nd_{\\pm} (x,z; x',z')\n  &= \\frac{1}{\\tau} d_\\primal(x,x^\\prime) + \\frac{1}{\\sigma}\n  d_\\dual(z,z^\\prime) \\pm \\inprod{z-z^\\prime}{A(x-x^\\prime)} \\\\\n&\\geq \\frac{1-\\delta_1}{\\tau} d_\\primal (x,x') +\n  \\frac{1-\\delta_2}{\\sigma} d_\\dual (z,z') \\\\\n&\\geq \\frac{1-\\delta_1}{2\\tau} \\|x-x'\\|_\\primal^2 +\n  \\frac{1-\\delta_2}{2\\sigma} \\|z-z'\\|_\\dual^2.\n\\end{align*}\nWith $\\delta_1=\\delta_2=1$, this shows convexity of $\\phi_+$ and $\\phi_-$;\nwith $\\delta_1=\\delta_2<1$, strong convexity.\n","start":47852,"end":48352},{"raw_tex":"Similarly,\n\\begin{align*}\nd_\\mathrm{dcv/pcv} (x,z; x', z')\n  &= d_{\\pm} (x,z; x',z') - h(x) + h(x') + \\inprod{\\nabla h(x')}{x-x'} \\\\\n&\\geq \\frac{1-\\tau L-\\delta_1}{\\tau} d_\\primal (x,x^\\prime) +\n  \\frac{1-\\delta_2}{\\sigma} d_\\dual (z,z^\\prime).\n\\end{align*}\nWith $\\delta_1=1-\\tau L$ and $\\delta_2=1$, this shows convexity of\n$\\phi_\\mathrm{pcv}$ and $\\phi_\\mathrm{dcv}$;\nwith $\\delta_1=\\delta-\\tau L$ and $\\delta_2=\\delta < 1$,\nstrong convexity.\n\\end{proof}","start":48352,"end":48808}]},{"id":"S039","semantic_type":"lemma","kind":"lemma","labels":["e-bcv-delta"],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"耦合交叉项控制","source_units":[{"raw_tex":"Suppose $\\sigma\\tau \\|A\\|^2 \\leq \\delta_1 \\delta_2$\nwith $\\delta_1,\\delta_2 > 0$.\nThen~\\eqref{e-bcv-str-cvx} and the arithmetic--geometric mean inequality\nimply that\n\\begin{align}\n\\left| \\inprod{z-z^\\prime}{A(x-x^\\prime)}\\right|\n  &\\leq \\|A\\| \\|z-z'\\|_\\dual \\|x-x'\\|_\\primal  \\nonumber \\\\\n&\\leq \\sqrt{\\frac{\\delta_1 \\delta_2}{\\sigma \\tau}}\n\\|z-z'\\|_\\dual \\|x-x'\\|_\\primal \\nonumber \\\\\n&\\leq \\frac{\\delta_1}{2\\tau} \\|x-x'\\|_\\primal^2  +\n  \\frac{\\delta_2}{2\\sigma} \\|z-z'\\|_\\primal^2 \\nonumber \\\\\n&\\leq \\frac{\\delta_1}{\\tau} d_\\primal (x,x') +\n  \\frac{\\delta_2}{\\sigma} d_\\dual (z,z'). \\label{e-bcv-delta}\n\\end{align}\n","start":47236,"end":47852}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":["e-bcv-delta"],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"范数与算术几何均值估计","source_units":[{"raw_tex":"\\begin{proof}\nTo show that the kernel functions $\\phi_+$ and $\\phi_-$ are convex,\nwe show that $d_+$ and $d_-$ are nonnegative.\nSuppose $\\sigma\\tau \\|A\\|^2 \\leq \\delta_1 \\delta_2$\nwith $\\delta_1,\\delta_2 > 0$.\nThen~\\eqref{e-bcv-str-cvx} and the arithmetic--geometric mean inequality\nimply that\n\\begin{align}\n\\left| \\inprod{z-z^\\prime}{A(x-x^\\prime)}\\right|\n  &\\leq \\|A\\| \\|z-z'\\|_\\dual \\|x-x'\\|_\\primal  \\nonumber \\\\\n&\\leq \\sqrt{\\frac{\\delta_1 \\delta_2}{\\sigma \\tau}}\n\\|z-z'\\|_\\dual \\|x-x'\\|_\\primal \\nonumber \\\\\n&\\leq \\frac{\\delta_1}{2\\tau} \\|x-x'\\|_\\primal^2  +\n  \\frac{\\delta_2}{2\\sigma} \\|z-z'\\|_\\primal^2 \\nonumber \\\\\n&\\leq \\frac{\\delta_1}{\\tau} d_\\primal (x,x') +\n  \\frac{\\delta_2}{\\sigma} d_\\dual (z,z'). \\label{e-bcv-delta}\n\\end{align}\n","start":47108,"end":47852}]},{"id":"S050","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"步长比保持不变","source_units":[{"raw_tex":"we set $\\theta_k = 2^{-i}\\bar\\theta_k$,\n$\\tau_k=\\theta_k\\tau_{k-1}$, $\\sigma_k=\\theta_k\\sigma_{k-1}$,\n","start":78538,"end":78640},{"raw_tex":"where $\\beta=\\sigma_{-1}/\\tau_{-1}$.\n","start":80256,"end":80293}]},{"id":"S060","semantic_type":"proof","kind":"proof","labels":["e-ls-conv-k"],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"加权遍历界的完整主证明","source_units":[{"raw_tex":"\\begin{proof}\nFrom~\\eqref{e-ls-conv-i},\n\\[\n\\cL(x^{(i)},z)-\\cL(x, \\bar z^{(i)}) \\leq \\frac{1}{\\tau_{i-1}}\n\\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n+ \\frac{1}{2\\beta} \\|z-z^{(i-1)}\\|^2\n- \\frac{1}{2\\beta} \\|z-z^{(i)}\\|^2 \\Big).\n\\]\nSince $\\cL(u,v)$ is convex in $u$ and affine in $v$,\n\\BEA\n\\lefteqn{(\\sum_{i=1}^k \\tau_{i-1}) \n  \\big(\\cL(x^{(k)}_\\avg, z)-\\cL(x, \\bar z^{(k)}_\\avg)\\big)} \\nonumber \\\\\n&\\leq &\\sum_{i=1}^k \\tau_{i-1} (\\cL(x^{(i)}, z)-\\cL(x, \\bar z^{(i)})) \n  \\nonumber \\\\\n&\\leq & d_\\primal(x,x^{(0)}) - d_\\primal(x,x^{(k)}) +\n  \\frac{1}{2\\beta} (\\|z-z^{(0)}\\|^2 - \\|z-z^{(k)}\\|^2) \\nonumber \\\\\n&\\leq & d_\\primal(x,x^{(0)}) + \\frac{1}{2\\beta} \\|z-z^{(0)}\\|^2.\n  \\label{e-ls-conv-k}\n\\EEA\nDividing by $\\sum_{i=1}^k \\tau_{i-1}$ gives~\\eqref{e-ls-conv}.\n\\end{proof}","start":88061,"end":88840}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"删去单步耗散项","source_units":[{"raw_tex":"\\begin{proof}\nFrom~\\eqref{e-ls-conv-i},\n\\[\n\\cL(x^{(i)},z)-\\cL(x, \\bar z^{(i)}) \\leq \\frac{1}{\\tau_{i-1}}\n\\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n+ \\frac{1}{2\\beta} \\|z-z^{(i-1)}\\|^2\n- \\frac{1}{2\\beta} \\|z-z^{(i)}\\|^2 \\Big).\n\\]\n","start":88061,"end":88299}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"加权平均的凸性估计","source_units":[{"raw_tex":"Since $\\cL(u,v)$ is convex in $u$ and affine in $v$,\n\\BEA\n\\lefteqn{(\\sum_{i=1}^k \\tau_{i-1}) \n  \\big(\\cL(x^{(k)}_\\avg, z)-\\cL(x, \\bar z^{(k)}_\\avg)\\big)} \\nonumber \\\\\n&\\leq &\\sum_{i=1}^k \\tau_{i-1} (\\cL(x^{(i)}, z)-\\cL(x, \\bar z^{(i)})) \n  \\nonumber \\\\\n","start":88299,"end":88552}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":["e-ls-conv-k"],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"距离差望远镜求和","source_units":[{"raw_tex":"&\\leq & d_\\primal(x,x^{(0)}) - d_\\primal(x,x^{(k)}) +\n  \\frac{1}{2\\beta} (\\|z-z^{(0)}\\|^2 - \\|z-z^{(k)}\\|^2) \\nonumber \\\\\n","start":88552,"end":88674}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S064"],"declared_external":false,"source_only":false,"title":"删去末端非负距离","source_units":[{"raw_tex":"&\\leq & d_\\primal(x,x^{(0)}) + \\frac{1}{2\\beta} \\|z-z^{(0)}\\|^2.\n  \\label{e-ls-conv-k}\n\\EEA\n","start":88674,"end":88766}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S065"],"declared_external":false,"source_only":false,"title":"除以累计步长","source_units":[{"raw_tex":"Dividing by $\\sum_{i=1}^k \\tau_{i-1}$ gives~\\eqref{e-ls-conv}.\n\\end{proof}","start":88766,"end":88840}]},{"id":"S066","semantic_type":"proof","kind":"proof","labels":["e-bcv-delta","e-ls-conv-1","e-ls-conv-2","e-ls-conv-3","e-ls-conv-4","e-ls-conv-k"],"member_ids":["S066"],"declared_external":false,"source_only":false,"title":"遍历收敛的完整作者证明包","source_units":[{"raw_tex":"\\begin{proof}\nTo show that the kernel functions $\\phi_+$ and $\\phi_-$ are convex,\nwe show that $d_+$ and $d_-$ are nonnegative.\nSuppose $\\sigma\\tau \\|A\\|^2 \\leq \\delta_1 \\delta_2$\nwith $\\delta_1,\\delta_2 > 0$.\nThen~\\eqref{e-bcv-str-cvx} and the arithmetic--geometric mean inequality\nimply that\n\\begin{align}\n\\left| \\inprod{z-z^\\prime}{A(x-x^\\prime)}\\right|\n  &\\leq \\|A\\| \\|z-z'\\|_\\dual \\|x-x'\\|_\\primal  \\nonumber \\\\\n&\\leq \\sqrt{\\frac{\\delta_1 \\delta_2}{\\sigma \\tau}}\n\\|z-z'\\|_\\dual \\|x-x'\\|_\\primal \\nonumber \\\\\n&\\leq \\frac{\\delta_1}{2\\tau} \\|x-x'\\|_\\primal^2  +\n  \\frac{\\delta_2}{2\\sigma} \\|z-z'\\|_\\primal^2 \\nonumber \\\\\n&\\leq \\frac{\\delta_1}{\\tau} d_\\primal (x,x') +\n  \\frac{\\delta_2}{\\sigma} d_\\dual (z,z'). \\label{e-bcv-delta}\n\\end{align}\nTherefore,\n\\begin{align*}\nd_{\\pm} (x,z; x',z')\n  &= \\frac{1}{\\tau} d_\\primal(x,x^\\prime) + \\frac{1}{\\sigma}\n  d_\\dual(z,z^\\prime) \\pm \\inprod{z-z^\\prime}{A(x-x^\\prime)} \\\\\n&\\geq \\frac{1-\\delta_1}{\\tau} d_\\primal (x,x') +\n  \\frac{1-\\delta_2}{\\sigma} d_\\dual (z,z') \\\\\n&\\geq \\frac{1-\\delta_1}{2\\tau} \\|x-x'\\|_\\primal^2 +\n  \\frac{1-\\delta_2}{2\\sigma} \\|z-z'\\|_\\dual^2.\n\\end{align*}\nWith $\\delta_1=\\delta_2=1$, this shows convexity of $\\phi_+$ and $\\phi_-$;\nwith $\\delta_1=\\delta_2<1$, strong convexity.\nSimilarly,\n\\begin{align*}\nd_\\mathrm{dcv/pcv} (x,z; x', z')\n  &= d_{\\pm} (x,z; x',z') - h(x) + h(x') + \\inprod{\\nabla h(x')}{x-x'} \\\\\n&\\geq \\frac{1-\\tau L-\\delta_1}{\\tau} d_\\primal (x,x^\\prime) +\n  \\frac{1-\\delta_2}{\\sigma} d_\\dual (z,z^\\prime).\n\\end{align*}\nWith $\\delta_1=1-\\tau L$ and $\\delta_2=1$, this shows convexity of\n$\\phi_\\mathrm{pcv}$ and $\\phi_\\mathrm{dcv}$;\nwith $\\delta_1=\\delta-\\tau L$ and $\\delta_2=\\delta < 1$,\nstrong convexity.\n\\end{proof}","start":47108,"end":48808},{"raw_tex":"\\begin{proof}\nApplying~\\eqref{e-bcv-delta} with $\\tau=\\tau_k$, $\\sigma=\\sigma_k$, $\\delta_1=\\delta^2-\\tau_k L$ and $\\delta_2=1$,\ntogether with the Lipschitz condition~\\eqref{e-bcv-lip},\nwe see that the backtracking condition~\\eqref{e-ls-cond} holds\nat iteration $k$ if $0 < \\delta < 1$ and\n\\[\n\\tau_k \\sigma_k\\|A\\|^2 + \\tau_k L \\leq \\delta^2.\n\\]\nThen mathematical induction can be used to prove~\\eqref{e-ls-param-lbnd}.\nThe two lower bounds~\\eqref{e-ls-param-lbnd} hold at $k=0$\nby the definition of $\\tau_\\mathrm{min}$ and $\\sigma_\\mathrm{min}$.\nNow assume $\\tau_{k-1} \\geq \\tau_\\mathrm{min}$,\n$\\sigma_{k-1} \\geq \\sigma_\\mathrm{min}$,\nand consider the $k$th iteration.\nThe first attempt of $\\theta_k$ is $\\theta_k=\\bar \\theta_k \\geq 1$.\nIf this value is accepted, then\n\\[\n\\tau_k = \\bar\\theta_k \\tau_{k-1} \\geq \\tau_{k-1} \\geq \\tau_\\mathrm{min},\n\\qquad\n\\sigma_k = \\bar\\theta_k \\sigma_{k-1} \\geq \\sigma_{k-1} \\geq\n  \\sigma_\\mathrm{min}.\n\\]\nOtherwise, one or more backtracking steps are needed.\nDenote by $\\tilde \\theta_k$ the last rejected value.\nThen $\\tilde\\theta_k^2\\tau_{k-1}^2\\beta\\|A\\|^2 + \n\\tilde\\theta_k \\tau_{k-1} L >\\delta^2$\nand the accepted $\\theta_k$ satisfies\n\\[\n\\theta_k = \\frac{\\tilde \\theta_k}{2} \\geq\n\\frac{-L+\\sqrt{L^2+4\\delta^2\\beta\\|A\\|^2}}%\n  {4\\tau_{k-1}\\beta\\|A\\|^2}.\n\\]\nTherefore,\n\\[\n\\tau_k=\\theta_k\\tau_{k-1} >\n\\frac{-L+\\sqrt{L^2+4\\delta^2\\beta\\|A\\|^2}}{4\\beta\\|A\\|^2}, \\qquad\n\\sigma_k=\\beta\\tau_k \\geq \\beta\\tau_\\mathrm{min}. \\qedhere\n\\]\n\\end{proof}","start":80409,"end":81883},{"raw_tex":"\\begin{proof}\nThe optimality condition for the primal prox-operator~\\eqref{e-ls-b}\ngives\n\\BEAS\n\\lefteqn{f(x^{(k+1)})-f(x)} \\\\\n&\\leq & \\frac{1}{\\tau_k}\n  \\big(d_\\primal(x,x^{(k)})-d_\\primal(x,x^{(k+1)})-\n  d_\\primal(x^{(k+1)}, x^{(k)})\\big)\n  +\\inprod{A^T\\bar z^{(k+1)}+\\nabla h(x^{(k)})}{x-x^{(k+1)}}\n\\EEAS\nfor all $x\\in \\dom f \\cap \\dom\\phi_\\primal$.\nHence\n\\BEA\n\\lefteqn{f(x^{(k+1)}) + h(x^{(k+1)}) - f(x) - h(x)} \\nonumber \\\\\n  &\\leq & \\frac{1}{\\tau_k} (d_\\primal(x,x^{(k)}) - \n  d_\\primal(x,x^{(k+1)}) - d_\\primal(x^{(k+1)}, x^{(k)}))\n  + \\inprod{A^T\\bar z_{k+1}}{x - x^{(k+1)}} \\nonumber \\\\\n& &\\mbox{} + h(x^{(k+1)})-h(x)+\\inprod{\\nabla h(x^{(k)})}{x-x^{(k+1)}}\n  \\nonumber \\\\\n&\\leq & \\frac{1}{\\tau_k} (d_\\primal(x,x^{(k)}) - \n d_\\primal(x,x^{(k+1)}) - d_\\primal(x^{(k+1)}, x^{(k)}))\n  + \\inprod{A^T\\bar z^{(k+1)}}{x - x^{(k+1)}} \\nonumber \\\\\n& &\\mbox{} +h(x^{(k+1)})-h(x^{(k)})\n  -\\inprod{\\nabla h(x^{(k)})}{x^{(k+1)}-x^{(k)}}.\n  \\label{e-ls-conv-1}\n\\EEA\nThe second inequality follows from the convexity of $h$, \\ie,\n$h(x) \\geq h(x^{(k)}) + \\inprod{\\nabla h(x^{(k)})}{x-x^{(k)}}$.\nThe dual update~\\eqref{e-ls-c} implies that\n\\BEQ \\label{e-ls-conv-2}\n\\inprod{z-z^{(k+1)}}{Ax^{(k+1)}-b}\n= \\frac{1}{\\sigma_k} \\inprod{z-z^{(k+1)}}{z^{(k+1)}-z^{(k)}}\n\\quad \\mbox{for all $z$.}\n\\EEQ\nThis equality at $k=i-1$ is\n\\begin{align}\n\\inprod{z-z^{(i)}}{Ax^{(i)}-b}\n  &= \\frac{1}{\\sigma_{i-1}} \\inprod{z-z^{(i)}}{z^{(i)}-z^{(i-1)}} \n  \\nonumber \\\\\n&= \\frac{1}{2\\sigma_{i-1}} \\left( \\|z-z^{(i-1)}\\|^2\n  - \\|z-z^{(i)}\\|^2 - \\|z^{(i)} - z^{(i-1)} \\|^2 \\right).\n  \\label{e-ls-conv-3}\n\\end{align}\nThe equality~\\eqref{e-ls-conv-2} at $k=i-2$ is\n\\begin{align*}\n\\inprod{z-z^{(i-1)}}{Ax^{(i-1)}-b}\n  &= \\frac{1}{\\sigma_{i-2}} \\inprod{z-z^{(i-1)}}{z^{(i-1)}-z^{(i-2)}} \\\\\n&= \\frac{\\theta_{i-1}}{\\sigma_{i-1}}\n  \\inprod{z-z^{(i-1)}}{z^{(i-1)}-z^{(i-2)}} \\\\\n&= \\frac{1}{\\sigma_{i-1}} \\inprod{z-z^{(i-1)}}{\\bar z^{(i)}-z^{(i-1)}}.\n\\end{align*}\nWe evaluate this at $z=z^{(i)}$ and add it to the equality\nat $z=z^{(i-2)}$ multiplied by $\\theta_{i-1}$:\n\\begin{align}\n\\inprod{z^{(i)}-\\bar z^{(i)}}{Ax^{(i-1)}-b}\n  &= \\frac{1}{\\sigma_{i-1}}\n  \\inprod{z^{(i)}-\\bar z^{(i)}}{\\bar z^{(i)} - z^{(i-1)}} \\nonumber \\\\\n&= \\frac{1}{2\\sigma_{i-1}} \\left(\\|z^{(i)}-z^{(i-1)}\\|^2\n  -\\|z^{(i)}-\\bar z^{(i)}\\|^2-\\|\\bar z^{(i)}-z^{(i-1)}\\|^2 \\right).\n  \\label{e-ls-conv-4}\n\\end{align}\nNow we combine~\\eqref{e-ls-conv-1} for $k=i-1$,\nwith~\\eqref{e-ls-conv-3} and~\\eqref{e-ls-conv-4}. For $i \\geq 1$,\n\\BEAS\n\\lefteqn{\\cL(x^{(i)},z) - \\cL(x, \\bar z^{(i)})} \\nonumber \\\\\n&=& f(x^{(i)})+h(x^{(i)})+\\inprod{z}{Ax^{(i)}-b}\n  -f(x)-h(x)-\\inprod{\\bar z^{(i)}}{Ax-b} \\nonumber \\\\\n&\\leq & \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)})-d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big)\n  + \\inprod{A^T\\bar z^{(i)}}{x-x^{(i)}} + \\inprod{z}{Ax^{(i)}-b} \\nonumber\\\\\n& &\\mbox{} - \\inprod{\\bar z^{(i)}}{Ax-b} + h(x^{(i)}) - h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)}) \n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big)\n  + \\inprod{z-\\bar z^{(i)}}{Ax^{(i)}-b} \\nonumber \\\\\n& &\\mbox{} +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)})\\Big) \\nonumber \\\\\n& & \\mbox{} +\\inprod{z^{(i)}-\\bar z^{(i)}}{A(x^{(i)}-x^{(i-1)})}\n  + \\inprod{z-z^{(i)}}{Ax^{(i)}-b}\n  + \\inprod{z^{(i)}-\\bar z^{(i)}}{Ax^{(i-1)}-b} \\nonumber \\\\\n& &\\mbox{} +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)}} \\nonumber \\\\\n&=& \\frac{1}{\\tau_{i-1}}\n  \\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n  - d_\\primal(x^{(i)},x^{(i-1)}))\\Big) \\nonumber \\\\\n& &\\mbox{} +\\frac{1}{2\\sigma_{i-1}}\n  \\Big(\\|z-z^{(i-1)}\\|^2-\\|z-z^{(i)}\\|^2-\\|\\bar z^{(i)}-z^{(i-1)}\\|^2\n  -\\|\\bar z^{(i)}-z^{(i)}\\|^2\\Big) \\nonumber \\\\\n& &\\mbox{}+\\inprod{A^T(z^{(i)}-\\bar z^{(i)})}{x^{(i)}-x^{(i-1)}}\n  +h(x^{(i)})-h(x^{(i-1)})\n  -\\inprod{\\nabla h(x^{(i-1)})}{x^{(i)}-x^{(i-1)})} \\\\\n&\\leq & \\frac{1}{\\tau_{i-1}} \n  \\left(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)}) -\n  (1-\\delta^2) d_\\primal(x^{(i)},x^{(i-1)})\\right) \\nonumber \\\\\n& &\\mbox{} + \\frac{1}{2\\sigma_{i-1}} \\left(\n  \\|z-z^{(i-1)}\\|^2 - \\|z-z^{(i)}\\|^2 - \n  \\|\\bar z^{(i)} - z^{(i-1)}\\|^2 \\right),\n\\EEAS\nwhich is the desired result~\\eqref{e-ls-conv-i}.\nThe first inequality follows from~\\eqref{e-ls-conv-1}.\nIn the second last step we substitute~\\eqref{e-ls-conv-3}\nand~\\eqref{e-ls-conv-4}.\nThe last step uses the line search exit condition~\\eqref{e-ls-cond}\nat $k=i-1$.\n\\end{proof}","start":82509,"end":87115},{"raw_tex":"\\begin{proof}\nFrom~\\eqref{e-ls-conv-i},\n\\[\n\\cL(x^{(i)},z)-\\cL(x, \\bar z^{(i)}) \\leq \\frac{1}{\\tau_{i-1}}\n\\Big(d_\\primal(x,x^{(i-1)}) - d_\\primal(x,x^{(i)})\n+ \\frac{1}{2\\beta} \\|z-z^{(i-1)}\\|^2\n- \\frac{1}{2\\beta} \\|z-z^{(i)}\\|^2 \\Big).\n\\]\nSince $\\cL(u,v)$ is convex in $u$ and affine in $v$,\n\\BEA\n\\lefteqn{(\\sum_{i=1}^k \\tau_{i-1}) \n  \\big(\\cL(x^{(k)}_\\avg, z)-\\cL(x, \\bar z^{(k)}_\\avg)\\big)} \\nonumber \\\\\n&\\leq &\\sum_{i=1}^k \\tau_{i-1} (\\cL(x^{(i)}, z)-\\cL(x, \\bar z^{(i)})) \n  \\nonumber \\\\\n&\\leq & d_\\primal(x,x^{(0)}) - d_\\primal(x,x^{(k)}) +\n  \\frac{1}{2\\beta} (\\|z-z^{(0)}\\|^2 - \\|z-z^{(k)}\\|^2) \\nonumber \\\\\n&\\leq & d_\\primal(x,x^{(0)}) + \\frac{1}{2\\beta} \\|z-z^{(0)}\\|^2.\n  \\label{e-ls-conv-k}\n\\EEA\nDividing by $\\sum_{i=1}^k \\tau_{i-1}$ gives~\\eqref{e-ls-conv}.\n\\end{proof}","start":88061,"end":88840}]},{"id":"S067","semantic_type":"proposition","kind":"proposition","labels":["e-ls-conv","e-ls-conv-1/k"],"member_ids":["S067"],"declared_external":false,"source_only":false,"title":"回溯算法的遍历收敛界","source_units":[{"raw_tex":"We show that\n\\begin{align}\n\\cL(x^{(k)}_\\avg,z) - \\cL(x,\\bar z^{(k)}_\\avg)\n  &\\leq \\frac{1}{\\sum_{i=1}^k \\tau_{i-1}}\n  \\big(d_\\primal(x,x^{(0)}) + \\frac{1}{2\\beta} \\|z-z^{(0)}\\|^2\\big)\n  \\label{e-ls-conv} \\\\ \n&\\leq \\frac{1}{k \\tau_\\mathrm{min}}\n  \\big(d_\\primal(x,x^{(0)}) + \\frac{1}{2\\beta} \\|z-z^{(0)}\\|^2\\big)\n  \\label{e-ls-conv-1/k}\n\\end{align}\nfor all $x \\in \\dom f \\cap \\dom \\phi_\\primal$ and all $z$.\nThis holds for any choice of $\\delta \\in (0,1]$ in~\\eqref{e-ls-cond}.\n","start":87404,"end":87881}]},{"id":"S068","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S068"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第二项明确允许 arithmetic–geometric mean inequality；第1115行也明确点名。"],"source_only":false,"title":"算术几何均值不等式","source_units":[]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S069"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第二项明确允许 Jensen's inequality。"],"source_only":false,"title":"加权Jensen不等式","source_units":[]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S070"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第二项明确允许 nonnegativity of Bregman distances。"],"source_only":false,"title":"Bregman距离非负","source_units":[]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S071"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；原文第1997–1998、2015–2016行直接使用，未给外部引用。"],"source_only":false,"title":"欧氏三点平方恒等式","source_units":[]},{"id":"S072","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S072"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；作者在第1111–1112、1137–1138、1146–1149行直接采用该判据。"],"source_only":false,"title":"距离下界与核凸性判据","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S038","S041":"S041","S042":"S038","S043":"S038","S044":"S031","S045":"S031","S046":"S031","S047":"S031","S048":"S031","S049":"S031","S050":"S050","S051":"S034","S052":"S034","S053":"S034","S054":"S034","S055":"S034","S056":"S034","S057":"S034","S058":"S034","S059":"S034","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072"}}}}
{"record_id":"1905.02671v3","paper":{"title":"Minimizing Uniformly Convex Functions by Cubic Regularization of Newton Method","authors":["Doikov, Nikita","Nesterov, Yurii"],"arxiv_id":"1905.02671","version":"v3","journal":"Journal of Optimization Theory and Applications","abstract_url":"https://arxiv.org/abs/1905.02671v3","pdf_url":"https://arxiv.org/pdf/1905.02671v3","doi":"10.1007/s10957-021-01838-7"},"context":{"raw_tex":"Let us start from some notation. In what follows, we denote\nby $\\E$ a finite-dimensional real vector space and by\n$\\E^{*}$ its dual space, which is a space of linear\nfunctions on $\\E$. The value of function $s \\in \\E^{*}$ at\npoint $x \\in \\E$ is denoted by $\\langle s, x \\rangle$.\nLet us fix some linear self-adjoint positive-definite operator\n$B: \\E \\to \\E^{*}$ and introduce the following Euclidean norms in the\nprimal and dual spaces:\n$$\n\\ba{rcl}\n\\|x\\| & \\defeq &  \\langle Bx, x \\rangle^{1/2}, \\quad x \\in \\E,\n\\qquad \\|s\\|_{*} \\; \\defeq \\; \\langle s, B^{-1}s \\rangle^{1/2},\n\\quad s \\in \\E^{*}.\n\\ea\n$$\nFor any linear operator $A: \\E \\to \\E^{*}$, its norm is induced in a\nstandard way:\n$$\n\\ba{rcl}\n\\|A\\| & \\defeq & \\max\\limits_{x \\in \\E} \\bigl\\{ \\|Ax\\|_{*} \\; | \\;\n\\|x\\| \\leq 1 \\bigr\\}.\n\\ea\n$$\nOur goal is to solve the convex optimization problem\nin the composite form:\n\\beq \\label{MainProblem}\n\\ba{rcl}\n\\min\\limits_{x \\in \\dom F} F(x) & := & f(x) + h(x),\n\\ea\n\\eeq\nwhere $f$ is a twice differentiable on its open domain\nuniformly convex function, and $h$ is a \\textit{simple}\nclosed convex function with $\\dom h \\subseteq \\dom f$.\n\\textit{Simple} means that all auxiliary subproblems with\nan explicit presence of $h$ are easily solvable.\n\nFor a smooth function $f$, its gradient at point $x$ is\ndenoted by $\\nabla f(x) \\in \\E^{*}$,  and its Hessian is\ndenoted by $\\nabla^2 f(x) : \\E \\to \\E^{*}$. For convex but\nnot necessary differentiable function $h$, we denote by\n$\\partial h(x) \\subset \\E^{*}$ its subdifferential at the\npoint $x \\in \\dom h$.\n\nWe say that differentiable function $f$ is \\textit{uniformly convex}\nof degree $p \\geq 2$ on a convex set $C \\subseteq \\dom f$\nif for some constant $\\sigma > 0$ it satisfies inequality\n\\beq \\label{UniformlyConvexDefinition}\n\\ba{rcl}\nf(y) & \\; \\geq \\; & f(x) + \\langle \\nabla f(x), y - x \\rangle +\n\\frac{\\sigma}{p}\\|y - x\\|^p, \\qquad x, y \\in C.\n\\ea\n\\eeq\nUniformly convex functions of degree $p = 2$ are known as\n\\textit{strongly convex}. If\ninequality~\\eqref{UniformlyConvexDefinition} holds with\n$\\sigma = 0$, the function $f$ is called just \\textit{convex}.\nThe following convenient condition\nis sufficient for function $f$ to be uniformly convex on a convex set\n$C \\subseteq \\dom f$:\n\\begin{lemma}(Lemma 1 in~\\cite{nesterov2008accelerating})\n\tLet for some $\\sigma > 0$ and $p \\geq 2$\n\tthe following inequality holds:\n\t\\beq \\label{UniformlyConvexSufficient}\n\t\\ba{rcl}\n\t\\langle \\nabla f(x) - \\nabla f(y), x - y \\rangle & \\; \\geq \\; &\n\t\\sigma \\|x - y\\|^p, \\qquad x, y \\in C.\n\t\\ea\n\t\\eeq\n\tThen, function $f$ is uniformly convex of degree $p$ on set $C$ with\n\tparameter $\\sigma$.\n\\end{lemma}\nFrom now on, we assume\n$\nC \\; := \\; \\dom F \\; \\subseteq \\; \\dom f.\n$\nBy the composite representation~\\eqref{MainProblem},\nwe have for every $x \\in \\dom F$ and for\nall $F'(x) \\in \\partial F(x)$:\n\\begin{equation} \\label{UniformlyConvexLowerBound}\n\\ba{rcl}\nF(y) & \\geq  & F(x) + \\langle\nF'(x), y - x \\rangle + \\frac{\\sigma}{p}\\|x - y\\|^p, \\qquad\ny \\in \\dom F.\n\\ea\n\\end{equation}\nTherefore, if $\\sigma > 0$, then we can have only one\npoint $x^{*} \\in \\dom F$ with $F(x^{*}) = F^{*}$, which\nalways exists for $F$ being uniformly convex and closed.\n\n\nIt is reasonable to define the best possible constant~$\\sigma$ in\ninequality~\\eqref{UniformlyConvexSufficient} for a certain degree~$p$.\nThis leads us to a system of constants:\n\\begin{equation} \\label{SigmaDefinition}\n\\ba{rcl}\n\\sigmafp & \\; \\defeq \\; & \\inf\\limits_{\\substack{x, y \\, \\in \\, \\dom F \\\\\n\t\tx \\not= y }} \\frac{\\langle \\nabla f(x) - \\nabla f(y), x - y \\rangle }\n{\\|x - y\\|^p}, \\qquad p \\geq 2.\n\\ea\n\\end{equation}\nWe prefer to use\ninequality~\\eqref{UniformlyConvexSufficient} for the\ndefinition of $\\sigmafp$, instead\nof~\\eqref{UniformlyConvexDefinition}, because of its\nsymmetry in $x$ and $y$. Note that the value $\\sigmafp$\nalso depends on the domain of $F$. However, we omit this\ndependence in our notation since it is always clear from\nthe context.\n\n\nFor a twice-differentiable function $f$, we say\nthat it has \\textit{H\\\"older continuous Hessian} of degree\n$\\nu \\in [0, 1]$ on a convex set $C \\subseteq \\dom f$, if for some\nconstant $\\mathcal{H}$, it\nholds:\n\\begin{equation} \\label{HolderHessianDefinition}\n\\ba{rcl}\n\\| \\nabla^2 f(x) - \\nabla^2 f(y) \\| & \\leq &\n\\mathcal{H} \\|x - y\\|^{\\nu}, \\qquad x, y \\in C.\n\\ea\n\\end{equation}\nTwo simple consequences of~\\eqref{HolderHessianDefinition}\nare as follows:\n\\beq \\label{HolderHessianGradBound}\n\\ba{rcl}\n\\| \\nabla f(y) - \\nabla f(x) - \\nabla^2 f(x)(y - x) \\|_{*} \n& \\leq &\n \\frac{\\H \\|x - y\\|^{1 + \\nu}}{1 + \\nu},\n\\ea\n\\eeq\n\\beq \\label{HolderHessianFuncBound}\n\\ba{rcl}\n| f(y) - Q(x; y) |  & \\leq & \n\\frac{\\H \\|x - y\\|^{2 + \\nu}}{(1 + \\nu)(2 + \\nu)},\n\\ea\n\\eeq\nwhere $Q(x; y)$ is the quadratic model of $f$ at the point\n$x$:\n$$\n\\ba{rcl}\nQ(x; y) & \\; \\defeq \\; & f(x) + \\langle \\nabla f(x), y - x \\rangle +\n\\frac{1}{2} \\langle \\nabla^2 f(x) (y - x), y - x \\rangle.\n\\ea\n$$\n\nIn order to characterize the level of smoothness of\nfunction $f$ on the set $C := \\dom F$, let us define\nthe system of H\\\"older constants (see\n\\cite{grapiglia2017regularized}):\n\\begin{equation} \\label{HfnuDefinition}\n\\ba{rcl}\n\\Hfnu & \\; \\defeq &\\; \\sup\\limits_{\\substack{x, y \\in \\dom F \\\\ x \\not= y}}\n\\frac{\\| \\nabla^2 f(x) - \\nabla^2 f(y) \\|}{\\|x - y\\|^{\\nu}},\n\\qquad \\nu \\in [0, 1].\n\\ea\n\\end{equation}\nWe allow $\\Hfnu$ to be equal to $+\\infty$ for some $\\nu$.\n\n\nHence, for a twice-differentiable convex function $f$ with\n$\\inf_{\\nu \\in [0, 1]} \\Hfnu > 0$, we can define only one\nmeaningful \\textit{condition number} of degree $\\nu \\in\n[0, 1]$:\n\\begin{equation} \\label{ConditionNumberDefinition}\n\\ba{rcl}\n\\gamma_f(\\nu) &\\; \\defeq \\;& \\frac{\\sigma_f(2 + \\nu)}{\\Hfnu}.\n\\ea\n\\end{equation}\nIf for some particular $\\nu$ we have $\\Hfnu = +\\infty$, then by our\ndefinition: $\\gamma_f(\\nu) = 0$.\n\n\nLet us start from the case when we know that for a specific\n$\\nu \\in [0, 1]$ function $f$ has H\\\"older continuous Hessian:\n$\\H_f(\\nu) < +\\infty$. Then, from~\\eqref{HolderHessianFuncBound}, we\nhave the global upper bound for the objective function:\n\\begin{displaymath}\n\\ba{rcl}\nF(y) & \\; \\leq \\; & M_{\\nu, H}(x; y) \\;\\defeq \\; Q(x; y)\n+ \\frac{H \\|x - y\\|^{2 + \\nu}}{(1 + \\nu)(2 + \\nu)} + h(y),\n\\qquad x, y \\in \\dom F,\n\\ea\n\\end{displaymath}\nwhere $H > 0$ is large enough: $H \\geq \\Hfnu$. Thus, it is\nnatural to employ the minimum of a regularized quadratic\nmodel:\n\\begin{displaymath}\n\\ba{c}\nT_{\\nu, H}(x) \\; \\defeq \\;\n\\argmin\\limits_{y \\in \\dom F} M_{\\nu, H}(x; y), \\qquad M_{\\nu, H}^{*}(x)\n\\; \\defeq \\; \\min\\limits_{y \\in \\dom F} M_{\\nu, H}(x; y),\n\\ea\n\\end{displaymath}\n\n\nUnfortunately, in practice it is difficult to decide on an\nappropriate value of $\\nu \\in [0, 1]$ with $\\Hfnu <\n+\\infty$. Therefore, it is interesting to develop the\n\\textit{universal methods} which are not based on some\nparticular parameters. Recently, it was\nshown~\\cite{grapiglia2017regularized} that one good\nchoice for such universal scheme is the cubic\nregularization of the Newton Method\n\\cite{nesterov2006cubic}. This is actually the\nprocess~\\eqref{GeneralRegularizedNewton} with the fixed\nparameter $\\nu = 1$. For this choice, in the rest part of\nthe paper we omit the corresponding index in the\ndefinitions of all necessary objects:\n$M_H(x; y) := M_{1, H}(x; y)$, \n$T_H(x) := T_{1, H}(x)$, and $M_H^{*}(x) := M_{1, H}^{*}(x) = M_H(x; T_H(x))$.\nThe adaptive scheme of our method with dynamic estimation\nof the constant $H$ is as follows.\n\n\n\\vspace*{5pt}\n\\noindent\n\\hspace*{5pt}\n\\lfbox{\n\t\\hspace*{4pt}\n\t\\centering\n\t\\begin{minipage}[b]{0.85\\linewidth} \\label{MainAlgorithm}\n\t\t\\begin{algorithm}[H] \n\t\t  \t\\caption{\\small \\textbf{Adaptive Cubic\n\t\t\t\t\tRegularization of Newton Method}}\n\t\t\t\\vspace*{6pt}\n\t\t\t\\noindent\\makebox[\\linewidth]{\\rule{1.097\\linewidth}{0.4pt}}\n\t\t\t\\begin{algorithmic}[1]\n\t\t\t\t%\\vspace{5pt}\n\t\t\t\t\\Require Choose $x_0 \\in \\dom F$, $H_0 > 0$.\n\t\t\t\t\\vspace*{2pt}\n\t\t\t\t\\Ensure $k \\geq 0$. \\vspace*{2pt}\n\t\t\t\t\\State Find the minimal integer $i_k \\geq 0 \\;$\n\t\t\t\tsuch that $F(T_{H_k 2^{i_k}}(x_k)) \\leq M^{*}_{H_k\n\t\t\t\t\t2^{i_k}} (x_k)$. %\\vspace*{1pt}\n\t\t\t\t\\State Perform the Cubic Step: $x_{k + 1}  = T_{H_k\n\t\t\t\t\t2^{i_k}}(x_k)$. \\vspace*{1pt}\n\t\t\t\t\\State Set $H_{k + 1} := 2^{i_k - 1} H_k $.\n\t\t\t\t\\vspace*{5pt}\n\t\t\t\\end{algorithmic}\n\t\t\\end{algorithm}\n\t\\end{minipage}\n\t\\hspace*{4pt}\n}\n\\vspace*{1pt}\n\n\nLet us present the main properties of the composite Cubic\nNewton step $x \\mapsto T_H(x)$. Denote\n$$\nr_H(x) \\defeq \\|T_H(x) - x\\|.\n$$\nSince point $T_H(x)$ is a minimum of strictly convex\nfunction $M_H(x;\\cdot)$, it satisfies the following\nfirst-order optimality condition:\n\\beq \\label{CubicStepStationary}\n\\ba{cl}\n&\\bigl \\la \\nabla f(x) + \\nabla^2 f(x)(T_H(x) - x) +\n\\tfrac{H r_H(x)}{2}B(T_H(x) - x), y - T_H(x) \\bigr \\ra  \\; + \\\\\n\\\\\n& \\quad h(y) \\; \\geq \\; h(T_H(x)),\n\\qquad y \\in \\dom F.\n\\ea\n\\eeq \nIn other words, the vector\n$$\n\\ba{rcl}\nh'(T_H(x)) & \\defeq & -\\nabla f(x) - \\nabla^2 f(x)(T_H(x) -\nx) - \\frac{H r_H(x)}{2}B(T_H(x) - x)\n\\ea\n$$\nbelongs to the subdifferential of $h$:\n\\begin{equation} \\label{CubicStepSubdiff}\n\\ba{rcl}\nh'(T_H(x)) &\\; \\in \\; & \\partial h(T_H(x)).\n\\ea\n\\end{equation}\nComputation of a point $T = T_H(x)$, satisfying condition\n\\eqref{CubicStepSubdiff}, requires some standard techniques\nof Convex Optimization and Linear Algebra\n(see~\\cite{nesterov2006cubic,nesterov2019implementable,agarwal2017finding,carmon2016gradient}). Arithmetical\ncomplexity of such a procedure is usually similar to that\nof the standard Newton step.\n\nPlugging into~\\eqref{CubicStepStationary} $y := x \\in\n\\dom F$, we get:\n\\begin{align} \n&\\la \\nabla f(x), x - T_H(x) \\ra \\label{StationaryConsequence} \\\\\n&\\geq \\; \\la \\nabla^2 f(x) (T_H(x) - x), T_H(x) - x \\ra +\n\\tfrac{H r_H^3(x)}{2} + h(T_H(x)) - h(x). \\notag\n\\end{align}\nThus, we obtain the following bound for the minimal value\n$M_H^{*}(x)$ of the cubic model:\n\\beq \\label{CubicModelMinimumExpression}\n\\ba{rcl}\nM_H^{*}(x) \n& \\overset{\\eqref{StationaryConsequence}}{\\leq} &\nf(x) -\n\\tfrac{1}{2}\\la \\nabla^2 f(x)(T_H(x) - x), T_H(x) - x \\ra -\n\\tfrac{H r_H^3(x)}{3} + h(x) \\\\[5pt]\n& = &\nF(x) - \\tfrac{1}{2}\\la \\nabla^2 f(x)(T_H(x) - x),\nT_H(x) - x \\ra - \\tfrac{Hr_H^3(x)}{3}. \n\\ea\n\\eeq\n\n\nIf for some value $\\nu \\in [0, 1]$ the Hessian is H\\\"older continuous:\n$\\Hfnu < +\\infty$, then by \\eqref{HolderHessianGradBound}\nand~\\eqref{CubicStepSubdiff} we get the following bound for\nthe subgradient:\n$$\nF'(T_H(x)) \\defeq \\nabla f(T_H(x)) + h'(T_H(x))\n$$\nat the new point:\n\\beq \\label{CubicGradNewPoint}\n\\ba{cl}\n& \\|  F'(T_H(x)) \\|_{*} \\\\[5pt]\n& \\leq \\;\n  \\| \\nabla f(T_H(x)) - \\nabla f(x) - \\nabla^2 f(x)\n(T_H(x) - x)\\|_{*} + \\tfrac{H r^2_H(x)}{2} \\\\[5pt]\n& \\stackrel{\\eqref{HolderHessianGradBound}}{\\leq}\n\\tfrac{\\Hfnu\n\tr^{1 + \\nu}_H(x)}{1 + \\nu}  +  \\tfrac{H r^2_H(x)}{2} \n\\; = \\; \nr^{1 + \\nu}_H(x) \\cdot \\Bigl(  \\tfrac{\\Hfnu}{1 + \\nu} + \n\\tfrac{Hr_H^{1 - \\nu}(x)}{2} \\Bigr). \n\\ea\n\\eeq\n\n\nDenote by $\\kappa_f(\\nu)$ the following auxiliary value:\n\\begin{equation} \\label{KappaDefinition}\n\\ba{rcl}\n\\kappa_f(\\nu)\n& \\; \\defeq \\; & \\frac{  \\Hfnu^{\\frac{2}{1 + \\nu}}   }\n{ (\\sigmafnu)^{\\frac{1 - \\nu}{(1 + \\nu)(2 + \\nu)}} }\n\\cdot\n\\frac{6 \\cdot (8 + \\nu)^{\\frac{1 - \\nu}{1 + \\nu}} }\n{ \\left((1 + \\nu)(2 + \\nu)\\right)^{\\frac{2}{1 + \\nu}} }\n\\cdot\n\\bigl( \\frac{1 + \\nu}{2 + \\nu} \\bigr)^{\\frac{1 - \\nu}{2 + \\nu}}, \\;\n\\nu \\in [0, 1].\n\\ea\n\\end{equation}\n","units":[{"raw_tex":"Let us start from some notation. In what follows, we denote\nby $\\E$ a finite-dimensional real vector space and by\n$\\E^{*}$ its dual space, which is a space of linear\nfunctions on $\\E$. The value of function $s \\in \\E^{*}$ at\npoint $x \\in \\E$ is denoted by $\\langle s, x \\rangle$.\nLet us fix some linear self-adjoint positive-definite operator\n$B: \\E \\to \\E^{*}$ and introduce the following Euclidean norms in the\nprimal and dual spaces:\n$$\n\\ba{rcl}\n\\|x\\| & \\defeq &  \\langle Bx, x \\rangle^{1/2}, \\quad x \\in \\E,\n\\qquad \\|s\\|_{*} \\; \\defeq \\; \\langle s, B^{-1}s \\rangle^{1/2},\n\\quad s \\in \\E^{*}.\n\\ea\n$$\nFor any linear operator $A: \\E \\to \\E^{*}$, its norm is induced in a\nstandard way:\n$$\n\\ba{rcl}\n\\|A\\| & \\defeq & \\max\\limits_{x \\in \\E} \\bigl\\{ \\|Ax\\|_{*} \\; | \\;\n\\|x\\| \\leq 1 \\bigr\\}.\n\\ea\n$$\nOur goal is to solve the convex optimization problem\nin the composite form:\n\\beq \\label{MainProblem}\n\\ba{rcl}\n\\min\\limits_{x \\in \\dom F} F(x) & := & f(x) + h(x),\n\\ea\n\\eeq\nwhere $f$ is a twice differentiable on its open domain\nuniformly convex function, and $h$ is a \\textit{simple}\nclosed convex function with $\\dom h \\subseteq \\dom f$.\n\\textit{Simple} means that all auxiliary subproblems with\nan explicit presence of $h$ are easily solvable.\n\nFor a smooth function $f$, its gradient at point $x$ is\ndenoted by $\\nabla f(x) \\in \\E^{*}$,  and its Hessian is\ndenoted by $\\nabla^2 f(x) : \\E \\to \\E^{*}$. For convex but\nnot necessary differentiable function $h$, we denote by\n$\\partial h(x) \\subset \\E^{*}$ its subdifferential at the\npoint $x \\in \\dom h$.\n\nWe say that differentiable function $f$ is \\textit{uniformly convex}\nof degree $p \\geq 2$ on a convex set $C \\subseteq \\dom f$\nif for some constant $\\sigma > 0$ it satisfies inequality\n\\beq \\label{UniformlyConvexDefinition}\n\\ba{rcl}\nf(y) & \\; \\geq \\; & f(x) + \\langle \\nabla f(x), y - x \\rangle +\n\\frac{\\sigma}{p}\\|y - x\\|^p, \\qquad x, y \\in C.\n\\ea\n\\eeq\nUniformly convex functions of degree $p = 2$ are known as\n\\textit{strongly convex}. If\ninequality~\\eqref{UniformlyConvexDefinition} holds with\n$\\sigma = 0$, the function $f$ is called just \\textit{convex}.\nThe following convenient condition\nis sufficient for function $f$ to be uniformly convex on a convex set\n$C \\subseteq \\dom f$:\n\\begin{lemma}(Lemma 1 in~\\cite{nesterov2008accelerating})\n\tLet for some $\\sigma > 0$ and $p \\geq 2$\n\tthe following inequality holds:\n\t\\beq \\label{UniformlyConvexSufficient}\n\t\\ba{rcl}\n\t\\langle \\nabla f(x) - \\nabla f(y), x - y \\rangle & \\; \\geq \\; &\n\t\\sigma \\|x - y\\|^p, \\qquad x, y \\in C.\n\t\\ea\n\t\\eeq\n\tThen, function $f$ is uniformly convex of degree $p$ on set $C$ with\n\tparameter $\\sigma$.\n\\end{lemma}\nFrom now on, we assume\n$\nC \\; := \\; \\dom F \\; \\subseteq \\; \\dom f.\n$\nBy the composite representation~\\eqref{MainProblem},\nwe have for every $x \\in \\dom F$ and for\nall $F'(x) \\in \\partial F(x)$:\n\\begin{equation} \\label{UniformlyConvexLowerBound}\n\\ba{rcl}\nF(y) & \\geq  & F(x) + \\langle\nF'(x), y - x \\rangle + \\frac{\\sigma}{p}\\|x - y\\|^p, \\qquad\ny \\in \\dom F.\n\\ea\n\\end{equation}\nTherefore, if $\\sigma > 0$, then we can have only one\npoint $x^{*} \\in \\dom F$ with $F(x^{*}) = F^{*}$, which\nalways exists for $F$ being uniformly convex and closed.\n","start":8361,"end":11541},{"raw_tex":"It is reasonable to define the best possible constant~$\\sigma$ in\ninequality~\\eqref{UniformlyConvexSufficient} for a certain degree~$p$.\nThis leads us to a system of constants:\n\\begin{equation} \\label{SigmaDefinition}\n\\ba{rcl}\n\\sigmafp & \\; \\defeq \\; & \\inf\\limits_{\\substack{x, y \\, \\in \\, \\dom F \\\\\n\t\tx \\not= y }} \\frac{\\langle \\nabla f(x) - \\nabla f(y), x - y \\rangle }\n{\\|x - y\\|^p}, \\qquad p \\geq 2.\n\\ea\n\\end{equation}\nWe prefer to use\ninequality~\\eqref{UniformlyConvexSufficient} for the\ndefinition of $\\sigmafp$, instead\nof~\\eqref{UniformlyConvexDefinition}, because of its\nsymmetry in $x$ and $y$. Note that the value $\\sigmafp$\nalso depends on the domain of $F$. However, we omit this\ndependence in our notation since it is always clear from\nthe context.\n","start":12288,"end":13052},{"raw_tex":"For a twice-differentiable function $f$, we say\nthat it has \\textit{H\\\"older continuous Hessian} of degree\n$\\nu \\in [0, 1]$ on a convex set $C \\subseteq \\dom f$, if for some\nconstant $\\mathcal{H}$, it\nholds:\n\\begin{equation} \\label{HolderHessianDefinition}\n\\ba{rcl}\n\\| \\nabla^2 f(x) - \\nabla^2 f(y) \\| & \\leq &\n\\mathcal{H} \\|x - y\\|^{\\nu}, \\qquad x, y \\in C.\n\\ea\n\\end{equation}\nTwo simple consequences of~\\eqref{HolderHessianDefinition}\nare as follows:\n\\beq \\label{HolderHessianGradBound}\n\\ba{rcl}\n\\| \\nabla f(y) - \\nabla f(x) - \\nabla^2 f(x)(y - x) \\|_{*} \n& \\leq &\n \\frac{\\H \\|x - y\\|^{1 + \\nu}}{1 + \\nu},\n\\ea\n\\eeq\n\\beq \\label{HolderHessianFuncBound}\n\\ba{rcl}\n| f(y) - Q(x; y) |  & \\leq & \n\\frac{\\H \\|x - y\\|^{2 + \\nu}}{(1 + \\nu)(2 + \\nu)},\n\\ea\n\\eeq\nwhere $Q(x; y)$ is the quadratic model of $f$ at the point\n$x$:\n$$\n\\ba{rcl}\nQ(x; y) & \\; \\defeq \\; & f(x) + \\langle \\nabla f(x), y - x \\rangle +\n\\frac{1}{2} \\langle \\nabla^2 f(x) (y - x), y - x \\rangle.\n\\ea\n$$\n\nIn order to characterize the level of smoothness of\nfunction $f$ on the set $C := \\dom F$, let us define\nthe system of H\\\"older constants (see\n\\cite{grapiglia2017regularized}):\n\\begin{equation} \\label{HfnuDefinition}\n\\ba{rcl}\n\\Hfnu & \\; \\defeq &\\; \\sup\\limits_{\\substack{x, y \\in \\dom F \\\\ x \\not= y}}\n\\frac{\\| \\nabla^2 f(x) - \\nabla^2 f(y) \\|}{\\|x - y\\|^{\\nu}},\n\\qquad \\nu \\in [0, 1].\n\\ea\n\\end{equation}\nWe allow $\\Hfnu$ to be equal to $+\\infty$ for some $\\nu$.\n","start":13435,"end":14861},{"raw_tex":"Hence, for a twice-differentiable convex function $f$ with\n$\\inf_{\\nu \\in [0, 1]} \\Hfnu > 0$, we can define only one\nmeaningful \\textit{condition number} of degree $\\nu \\in\n[0, 1]$:\n\\begin{equation} \\label{ConditionNumberDefinition}\n\\ba{rcl}\n\\gamma_f(\\nu) &\\; \\defeq \\;& \\frac{\\sigma_f(2 + \\nu)}{\\Hfnu}.\n\\ea\n\\end{equation}\nIf for some particular $\\nu$ we have $\\Hfnu = +\\infty$, then by our\ndefinition: $\\gamma_f(\\nu) = 0$.\n","start":20006,"end":20430},{"raw_tex":"Let us start from the case when we know that for a specific\n$\\nu \\in [0, 1]$ function $f$ has H\\\"older continuous Hessian:\n$\\H_f(\\nu) < +\\infty$. Then, from~\\eqref{HolderHessianFuncBound}, we\nhave the global upper bound for the objective function:\n\\begin{displaymath}\n\\ba{rcl}\nF(y) & \\; \\leq \\; & M_{\\nu, H}(x; y) \\;\\defeq \\; Q(x; y)\n+ \\frac{H \\|x - y\\|^{2 + \\nu}}{(1 + \\nu)(2 + \\nu)} + h(y),\n\\qquad x, y \\in \\dom F,\n\\ea\n\\end{displaymath}\nwhere $H > 0$ is large enough: $H \\geq \\Hfnu$. Thus, it is\nnatural to employ the minimum of a regularized quadratic\nmodel:\n\\begin{displaymath}\n\\ba{c}\nT_{\\nu, H}(x) \\; \\defeq \\;\n\\argmin\\limits_{y \\in \\dom F} M_{\\nu, H}(x; y), \\qquad M_{\\nu, H}^{*}(x)\n\\; \\defeq \\; \\min\\limits_{y \\in \\dom F} M_{\\nu, H}(x; y),\n\\ea\n\\end{displaymath}\n","start":34020,"end":34789},{"raw_tex":"Unfortunately, in practice it is difficult to decide on an\nappropriate value of $\\nu \\in [0, 1]$ with $\\Hfnu <\n+\\infty$. Therefore, it is interesting to develop the\n\\textit{universal methods} which are not based on some\nparticular parameters. Recently, it was\nshown~\\cite{grapiglia2017regularized} that one good\nchoice for such universal scheme is the cubic\nregularization of the Newton Method\n\\cite{nesterov2006cubic}. This is actually the\nprocess~\\eqref{GeneralRegularizedNewton} with the fixed\nparameter $\\nu = 1$. For this choice, in the rest part of\nthe paper we omit the corresponding index in the\ndefinitions of all necessary objects:\n$M_H(x; y) := M_{1, H}(x; y)$, \n$T_H(x) := T_{1, H}(x)$, and $M_H^{*}(x) := M_{1, H}^{*}(x) = M_H(x; T_H(x))$.\nThe adaptive scheme of our method with dynamic estimation\nof the constant $H$ is as follows.\n","start":37376,"end":38222},{"raw_tex":"\\vspace*{5pt}\n\\noindent\n\\hspace*{5pt}\n\\lfbox{\n\t\\hspace*{4pt}\n\t\\centering\n\t\\begin{minipage}[b]{0.85\\linewidth} \\label{MainAlgorithm}\n\t\t\\begin{algorithm}[H] \n\t\t  \t\\caption{\\small \\textbf{Adaptive Cubic\n\t\t\t\t\tRegularization of Newton Method}}\n\t\t\t\\vspace*{6pt}\n\t\t\t\\noindent\\makebox[\\linewidth]{\\rule{1.097\\linewidth}{0.4pt}}\n\t\t\t\\begin{algorithmic}[1]\n\t\t\t\t%\\vspace{5pt}\n\t\t\t\t\\Require Choose $x_0 \\in \\dom F$, $H_0 > 0$.\n\t\t\t\t\\vspace*{2pt}\n\t\t\t\t\\Ensure $k \\geq 0$. \\vspace*{2pt}\n\t\t\t\t\\State Find the minimal integer $i_k \\geq 0 \\;$\n\t\t\t\tsuch that $F(T_{H_k 2^{i_k}}(x_k)) \\leq M^{*}_{H_k\n\t\t\t\t\t2^{i_k}} (x_k)$. %\\vspace*{1pt}\n\t\t\t\t\\State Perform the Cubic Step: $x_{k + 1}  = T_{H_k\n\t\t\t\t\t2^{i_k}}(x_k)$. \\vspace*{1pt}\n\t\t\t\t\\State Set $H_{k + 1} := 2^{i_k - 1} H_k $.\n\t\t\t\t\\vspace*{5pt}\n\t\t\t\\end{algorithmic}\n\t\t\\end{algorithm}\n\t\\end{minipage}\n\t\\hspace*{4pt}\n}\n\\vspace*{1pt}\n","start":38223,"end":39079},{"raw_tex":"Let us present the main properties of the composite Cubic\nNewton step $x \\mapsto T_H(x)$. Denote\n$$\nr_H(x) \\defeq \\|T_H(x) - x\\|.\n$$\nSince point $T_H(x)$ is a minimum of strictly convex\nfunction $M_H(x;\\cdot)$, it satisfies the following\nfirst-order optimality condition:\n\\beq \\label{CubicStepStationary}\n\\ba{cl}\n&\\bigl \\la \\nabla f(x) + \\nabla^2 f(x)(T_H(x) - x) +\n\\tfrac{H r_H(x)}{2}B(T_H(x) - x), y - T_H(x) \\bigr \\ra  \\; + \\\\\n\\\\\n& \\quad h(y) \\; \\geq \\; h(T_H(x)),\n\\qquad y \\in \\dom F.\n\\ea\n\\eeq \nIn other words, the vector\n$$\n\\ba{rcl}\nh'(T_H(x)) & \\defeq & -\\nabla f(x) - \\nabla^2 f(x)(T_H(x) -\nx) - \\frac{H r_H(x)}{2}B(T_H(x) - x)\n\\ea\n$$\nbelongs to the subdifferential of $h$:\n\\begin{equation} \\label{CubicStepSubdiff}\n\\ba{rcl}\nh'(T_H(x)) &\\; \\in \\; & \\partial h(T_H(x)).\n\\ea\n\\end{equation}\nComputation of a point $T = T_H(x)$, satisfying condition\n\\eqref{CubicStepSubdiff}, requires some standard techniques\nof Convex Optimization and Linear Algebra\n(see~\\cite{nesterov2006cubic,nesterov2019implementable,agarwal2017finding,carmon2016gradient}). Arithmetical\ncomplexity of such a procedure is usually similar to that\nof the standard Newton step.\n\nPlugging into~\\eqref{CubicStepStationary} $y := x \\in\n\\dom F$, we get:\n\\begin{align} \n&\\la \\nabla f(x), x - T_H(x) \\ra \\label{StationaryConsequence} \\\\\n&\\geq \\; \\la \\nabla^2 f(x) (T_H(x) - x), T_H(x) - x \\ra +\n\\tfrac{H r_H^3(x)}{2} + h(T_H(x)) - h(x). \\notag\n\\end{align}\nThus, we obtain the following bound for the minimal value\n$M_H^{*}(x)$ of the cubic model:\n\\beq \\label{CubicModelMinimumExpression}\n\\ba{rcl}\nM_H^{*}(x) \n& \\overset{\\eqref{StationaryConsequence}}{\\leq} &\nf(x) -\n\\tfrac{1}{2}\\la \\nabla^2 f(x)(T_H(x) - x), T_H(x) - x \\ra -\n\\tfrac{H r_H^3(x)}{3} + h(x) \\\\[5pt]\n& = &\nF(x) - \\tfrac{1}{2}\\la \\nabla^2 f(x)(T_H(x) - x),\nT_H(x) - x \\ra - \\tfrac{Hr_H^3(x)}{3}. \n\\ea\n\\eeq\n\n\nIf for some value $\\nu \\in [0, 1]$ the Hessian is H\\\"older continuous:\n$\\Hfnu < +\\infty$, then by \\eqref{HolderHessianGradBound}\nand~\\eqref{CubicStepSubdiff} we get the following bound for\nthe subgradient:\n$$\nF'(T_H(x)) \\defeq \\nabla f(T_H(x)) + h'(T_H(x))\n$$\nat the new point:\n\\beq \\label{CubicGradNewPoint}\n\\ba{cl}\n& \\|  F'(T_H(x)) \\|_{*} \\\\[5pt]\n& \\leq \\;\n  \\| \\nabla f(T_H(x)) - \\nabla f(x) - \\nabla^2 f(x)\n(T_H(x) - x)\\|_{*} + \\tfrac{H r^2_H(x)}{2} \\\\[5pt]\n& \\stackrel{\\eqref{HolderHessianGradBound}}{\\leq}\n\\tfrac{\\Hfnu\n\tr^{1 + \\nu}_H(x)}{1 + \\nu}  +  \\tfrac{H r^2_H(x)}{2} \n\\; = \\; \nr^{1 + \\nu}_H(x) \\cdot \\Bigl(  \\tfrac{\\Hfnu}{1 + \\nu} + \n\\tfrac{Hr_H^{1 - \\nu}(x)}{2} \\Bigr). \n\\ea\n\\eeq\n","start":39080,"end":41610},{"raw_tex":"Denote by $\\kappa_f(\\nu)$ the following auxiliary value:\n\\begin{equation} \\label{KappaDefinition}\n\\ba{rcl}\n\\kappa_f(\\nu)\n& \\; \\defeq \\; & \\frac{  \\Hfnu^{\\frac{2}{1 + \\nu}}   }\n{ (\\sigmafnu)^{\\frac{1 - \\nu}{(1 + \\nu)(2 + \\nu)}} }\n\\cdot\n\\frac{6 \\cdot (8 + \\nu)^{\\frac{1 - \\nu}{1 + \\nu}} }\n{ \\left((1 + \\nu)(2 + \\nu)\\right)^{\\frac{2}{1 + \\nu}} }\n\\cdot\n\\bigl( \\frac{1 + \\nu}{2 + \\nu} \\bigr)^{\\frac{1 - \\nu}{2 + \\nu}}, \\;\n\\nu \\in [0, 1].\n\\ea\n\\end{equation}\n","start":51894,"end":52346}]},"statement":{"raw_tex":"\\begin{theorem} \\label{TheoremLinearRateUniformly} Assume that for a\n\tfixed $\\nu \\in [0, 1]$ we have $0 < \\Hfnu < +\\infty$ and\n\t$\\sigmafnu > 0$. Let parameter $H_0$ in Algorithm 1 be\n\tsmall enough:\n\t\\begin{equation} \\label{HolderChoosingH}\n\t\\ba{rcl}\n\tH_0 & \\; \\leq \\; & \\frac{ \\kappa_f(\\nu) }{ \\left( F(x_0) - F^{*}\n\t\t\\right)^{(1 - \\nu) / (2 + \\nu)  } },\n\t\\ea\n\t\\end{equation}\n\twhere $\\kappa_f(\\nu)$ is defined\n\tby~\\eqref{KappaDefinition}. Let the sequence $\\{ x_k \\}_{k\n\t\t= 0}^K$ generated by the method satisfy condition:\n\t\\begin{equation} \\label{HolderConditionFuncEps}\n\tF(T_{H_k2^{j}}(x_k)) - F^{*} \\; \\geq \\; \\varepsilon \\; > \\; 0, \\qquad\n\t\\quad 0 \\leq j \\leq i_k, \\quad 0 \\leq k \\leq K - 1.\n\t\\end{equation}\n\tThen, for every $0 \\leq k \\leq K - 1$, we have:\n\t\\beq \\label{HolderMainResult}\n\t\\ba{cl}\n\t& F(x_{k + 1}) - F^{*}  \\leq \\\\[5pt]\n\t& \\bigl( 1 - \\min\\bigl\\{  \\frac{ (2 + \\nu)\\left( (1 + \\nu) (2\n\t\t+ \\nu) \\right)^{ 1 / (1 + \\nu) }  \n\t    \\left( \\gammafnu  \\right)^{\\frac{1}{1 +\n\t    \t\t\\nu}}   }{ (1 + \\nu)6^{3/2} \\cdot 2^{1/2} \\cdot (8\n\t\t+ \\nu)^{ (1 - \\nu) / (2 + 2 \\nu) } } , \\frac{1}{2} \\bigr\\}  \\bigr) \\cdot \n\t  \\left( F(x_{k}) - F^{*} \\right).\n\t\\ea\n\t\\eeq\n\tTherefore, the rate of convergence is linear, and\n\t\\begin{displaymath}\n\t\\ba{rcl}\n\tK & \\; \\leq \\; & \\max\\bigl\\{  \\left( \\gammafnu \\right)^{\\frac{-1}{1\n\t\t\t+ \\nu}} \\cdot \\frac{1 + \\nu}{2 + \\nu}  \\cdot \\frac{6^{3/2} \\cdot\n\t\t2^{1/2} \\cdot (8 + \\nu)^{(1 - \\nu) / (2 + 2\\nu)} }{ \\left( (1 +\n\t\t\\nu)(2 + \\nu) \\right)^{1 / (1 + \\nu)}  } , \\; 1 \\bigr\\} \\cdot \\log\n\t\\frac{ F(x_0) - F^{*} }{\\varepsilon}.\n\t\\ea\n\t\\end{displaymath}\n\tMoreover, we have the following bound for the total number of oracle\n\tcalls $N_K$ during the first $K$ iterations:\n\t\\begin{equation} \\label{HolderOracleCallsBound}\n\t\\ba{rcl}\n\tN_K & \\; \\leq \\; & 2K  +  \\log_2\n\t\\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1 - \\nu) / (2 + \\nu)}} -\n\t\\log_2 H_0.\n\t\\ea\n\t\\end{equation}\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem} \\label{TheoremLinearRateUniformly} Assume that for a\n\tfixed $\\nu \\in [0, 1]$ we have $0 < \\Hfnu < +\\infty$ and\n\t$\\sigmafnu > 0$. Let parameter $H_0$ in Algorithm 1 be\n\tsmall enough:\n\t\\begin{equation} \\label{HolderChoosingH}\n\t\\ba{rcl}\n\tH_0 & \\; \\leq \\; & \\frac{ \\kappa_f(\\nu) }{ \\left( F(x_0) - F^{*}\n\t\t\\right)^{(1 - \\nu) / (2 + \\nu)  } },\n\t\\ea\n\t\\end{equation}\n\twhere $\\kappa_f(\\nu)$ is defined\n\tby~\\eqref{KappaDefinition}. Let the sequence $\\{ x_k \\}_{k\n\t\t= 0}^K$ generated by the method satisfy condition:\n\t\\begin{equation} \\label{HolderConditionFuncEps}\n\tF(T_{H_k2^{j}}(x_k)) - F^{*} \\; \\geq \\; \\varepsilon \\; > \\; 0, \\qquad\n\t\\quad 0 \\leq j \\leq i_k, \\quad 0 \\leq k \\leq K - 1.\n\t\\end{equation}\n\tThen, for every $0 \\leq k \\leq K - 1$, we have:\n\t\\beq \\label{HolderMainResult}\n\t\\ba{cl}\n\t& F(x_{k + 1}) - F^{*}  \\leq \\\\[5pt]\n\t& \\bigl( 1 - \\min\\bigl\\{  \\frac{ (2 + \\nu)\\left( (1 + \\nu) (2\n\t\t+ \\nu) \\right)^{ 1 / (1 + \\nu) }  \n\t    \\left( \\gammafnu  \\right)^{\\frac{1}{1 +\n\t    \t\t\\nu}}   }{ (1 + \\nu)6^{3/2} \\cdot 2^{1/2} \\cdot (8\n\t\t+ \\nu)^{ (1 - \\nu) / (2 + 2 \\nu) } } , \\frac{1}{2} \\bigr\\}  \\bigr) \\cdot \n\t  \\left( F(x_{k}) - F^{*} \\right).\n\t\\ea\n\t\\eeq\n\tTherefore, the rate of convergence is linear, and\n\t\\begin{displaymath}\n\t\\ba{rcl}\n\tK & \\; \\leq \\; & \\max\\bigl\\{  \\left( \\gammafnu \\right)^{\\frac{-1}{1\n\t\t\t+ \\nu}} \\cdot \\frac{1 + \\nu}{2 + \\nu}  \\cdot \\frac{6^{3/2} \\cdot\n\t\t2^{1/2} \\cdot (8 + \\nu)^{(1 - \\nu) / (2 + 2\\nu)} }{ \\left( (1 +\n\t\t\\nu)(2 + \\nu) \\right)^{1 / (1 + \\nu)}  } , \\; 1 \\bigr\\} \\cdot \\log\n\t\\frac{ F(x_0) - F^{*} }{\\varepsilon}.\n\t\\ea\n\t\\end{displaymath}\n\tMoreover, we have the following bound for the total number of oracle\n\tcalls $N_K$ during the first $K$ iterations:\n\t\\begin{equation} \\label{HolderOracleCallsBound}\n\t\\ba{rcl}\n\tN_K & \\; \\leq \\; & 2K  +  \\log_2\n\t\\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1 - \\nu) / (2 + \\nu)}} -\n\t\\log_2 H_0.\n\t\\ea\n\t\\end{equation}\n\\end{theorem}","start":55337,"end":57236}]},"proof":{"raw_tex":"\\begin{lemma} \\label{UniformlyConvexConsequence}\n\tLet $f$ be uniformly convex of degree $p \\geq 2$ with\n\tconstant $\\sigma > 0$ on set $\\dom F$. Then, for every $x\n\t\\in \\dom F$ and for all $F'(x) \\in \\partial F(x)$ we have\n\t\\begin{equation} \\label{UniformlyConvexGradientBound}\n\t\\ba{rcl}\n\tF(x) - F^{*} & \\; \\leq \\; & \\frac{p - 1}{p} \\left( \\frac{1}{\\sigma}\n\t\\right)^{\\frac{1}{p - 1}} \\|F'(x)\\|_{*}^{\\frac{p}{p - 1}}.\n\t\\ea\n\t\\end{equation}\n\\end{lemma}\n\n\\begin{proof}\n\t\\noindent\n\tIn view of~\\eqref{UniformlyConvexLowerBound}, \n\tbound \\eqref{UniformlyConvexGradientBound}\n\tfollows as in the proof of Lemma~3 \n\tin~\\cite{nesterov2008accelerating}.\n\\qed\n\\end{proof}\n\n\\begin{proof}\n\tDenote $\\kappa(x) \\equiv \\sum_{i = 1}^m e^{\\langle a_i, x \\rangle}$.\n\tLet us fix arbitrary $x, y \\in \\E$ and direction $h \\in \\E$. Then,\n\tstraightforward computation gives:\n\t$$\n\t\\ba{rcl}\n\t\\langle \\nabla f(x), h \\rangle &\\; = \\;& \\frac{1}{\\kappa(x)}\n\t\\sum_{i = 1}^m e^{\\langle a_i, x \\rangle} \\langle a_i, h \\rangle, \\\\[6pt]\n\t\\langle \\nabla^2 f(x)h, h \\rangle &\\; =\\;&\n\t \\frac{1}{\\kappa(x)} \\sum_{i = 1}^m e^{\\langle a_i, x \\rangle}\n\t\\langle\n\ta_i, h \\rangle^2 - \\bigl( \\frac{1}{\\kappa(x)} \\sum_{i = 1}^m\n\te^{\\langle a_i, x \\rangle} \\langle a_i, h \\rangle  \\bigr)^2 \\\\[5pt]\n\t&\\; = \\;& \\frac{1}{\\kappa(x)} \\sum_{i = 1}^m e^{\\langle a_i, x \\rangle}\n\t\\left( \\langle\n\ta_i, h \\rangle - \\langle \\nabla f(x), h \\rangle\n\t\\right)^2 \\; \\geq \\; 0.\n\t\\ea\n\t$$\n\tHence, we get\n\t$$\n\t\\ba{rcl}\n\t\\| \\nabla^2 f(x) \\| & = & \\max\\limits_{\\|h\\| \\leq 1} \\langle \\nabla^2 f(x)\n\th, h \\rangle \\;\n\t\\leq \\;  \\max\\limits_{\\|h\\| \\leq 1} \\sum_{i = 1}^m \\langle a_i, h \\rangle^2\n\t\\;\n\t= \\; \\max\\limits_{\\|h\\| \\leq 1} \\|h\\|^2 \\; = \\; 1.\n\t\\ea\n\t$$\n\tSince all Hessians of function $f$ are positive definite,\n\twe conclude that $\\Hf(0) \\leq 1$. Inequality $\\Hf(1) \\leq\n\t2$ can be easily obtained from the following\n\trepresentation of the third derivative:\n\t$$\n\t\\ba{rcl}\n\tf'''(x)[h,h,h]  & \\; = \\; & {1 \\over \\kappa(x)}\n\t\\sum\\limits_{i=1}^m e^{\\la a_i, x \\ra} \\left( \\la a_i, h\n\t\\ra - \\la \\nabla f(x), h \\ra \\right)^3\\\\\n\t\\\\\n\t& \\;  \\leq \\; & \\la \\nabla^2 f(x) h, h \\ra \\max\\limits_{1 \\leq\n\t\ti,j \\leq m } \\la a_i - a_j , h \\ra \\; \\leq \\; 2 \\| h \\|^3.\n\t\\ea\n\t$$\n\\qed\n\\end{proof}\n\n\\begin{lemma} \\label{LemmaStepHolder}\n\tLet $0 < \\Hfnu < +\\infty$ and $\\sigmafnu > 0$ for some\n\t$\\nu \\in [0,1]$. Then, for arbitrary $x \\in \\dom F$ and $H\n\t> 0$ we have:\n\t\\beq \\label{OneStepProgressHolder}\n\t\\ba{cl}\n\t& F(x) - M_H^{*}(x)  \\\\[5pt] \n\t& \\quad \\geq \\; \\min\\Bigl[ \\left( F(x) -\n\tF^{*} \\right) \\cdot \\frac{(1 + \\nu)}{(2 + \\nu)} \\cdot\n\t\\min\\bigl\\{\\bigl( \\frac{(1 + \\nu) \\gammafnu}{2(2 + \\nu)}  \\bigr)^{\\frac{1}{1 + \\nu}}, \n\t\\; 1\\bigr\\} ,   \\\\[5pt]\n\t& \\; \\qquad \\qquad \\left( F(T_H(x)) - F^{*} \\right)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}}\n\t\\cdot \\bigl ( \\frac{2 + \\nu}{1 + \\nu}  \\bigr)^{\\frac{3(1 + \\nu)}{2(2\n\t\t\t+ \\nu)}} \\cdot \\frac{(\\sigmafnu)^{\\frac{3}{2(2 + \\nu)}}}{ 3 \\sqrt{H}\n\t}  \\;  \\Bigr]. \n\t\\ea\n\t\\eeq\n\\end{lemma}\n\n\\begin{proof}\n\tLet us consider two cases. 1) \n\t$H r_H^{1 - \\nu}(x) \\leq \\frac{2 \\Hfnu}{1 + \\nu}$.\n\t\tThen, for arbitrary $y \\in \\dom F$, we have:\n\t\t$$\n\t\t\\ba{rcl}\n\t\tM_H^{*}(x) \n\t\t& := &  Q(x; T_H(x)) +\n\t\t\\frac{H}{6}\\|T_H(x) - x\\|^3 + h(T_H(x)) \\\\[7pt]\n\t\t& \\leq &  Q(x; y) + \\frac{H r_H^{1 - \\nu}(x) \\|y -\n\t\t\tx\\|^{2 + \\nu}}{2 (2 + \\nu) } + h(y) \\\\[5pt]\n\t\t& \\overset{\\eqref{HolderHessianFuncBound}}{\\leq} &  \n\t\tF(y) +\n\t\t\\frac{\\Hfnu \\|y - x\\|^{2 + \\nu}}{(1 + \\nu)(2 + \\nu)} +\n\t\t\\frac{H r_H^{1 - \\nu}(x) \\|y - x\\|^{2 + \\nu}}{2 (2 + \\nu) }\n\t\t\\\\[7pt]\n\t\t& \\leq &\n\t\t F(y) +  \\frac{2\\Hfnu \\|y - x\\|^{2 +\n\t\t\\nu}}{(1 + \\nu)(2 + \\nu)},\n\t\t\\ea\n\t\t$$\n\t\twhere the first inequality follows from the fact, that\n\t\t$$\n\t\t\\ba{rcl}\n\t\tT_H(x) & = & \\argmin\\limits_{y \\in \\dom F} \\bigl\\{ Q(x; y)\n\t\t+ \\frac{H r_H^{1 - \\nu}(x) \\|y -\n\t\t\tx\\|^{2 + \\nu}}{2 (2 + \\nu) } + h(y) \\bigr\\}.\n\t\t\\ea\n\t\t$$\n\t\tLet us restrict $y$ to the segment: $y = \\alpha x^{*} + (1 -\n\t\t\\alpha) x,$ with $\\alpha \\in [0, 1]$. Taking into account the\n\t\tuniform convexity, we get:\n\t\t$$\n\t\t\\ba{rcl}\n\t\tM_H^{*}(x) & \\leq &  F(x) - \\alpha \\left( F(x) -\n\t\tF^{*} \\right) + \\alpha^{2 + \\nu} \\frac{2\\Hfnu \\|x^{*} - x\\|^{2 +\n\t\t\t\t\\nu}} {(1 + \\nu)(2 + \\nu)} \\quad \\\\[5pt]\n\t\t& \\overset{\\eqref{UniformlyConvexLowerBound}}{\\leq} &  \n\t\tF(x)\n\t\t-  \\Bigl(\\alpha  - \\alpha^{2 + \\nu} \\frac{2\\Hfnu}{  (1 + \\nu)\n\t\t\t\\sigmafnu}\n\t\t\\Bigr) \\left( F(x) - F^{*} \\right).\n\t\t\\ea\n\t\t$$\n\t\tThe minimum of the right-hand side is attained at\n\t\t$\n\t\t\\alpha^{*} = \\min\\bigl\\{\\frac{(1 + \\nu)\\gammafnu}{2(2 + \\nu)}, 1 \\bigr\\}^{\\frac{1}{1 + \\nu}}.\n\t\t$\n\t\tPlugging this value into the bound, we have:\n\t\t\\begin{displaymath}\n\t\t\\ba{rcl}\n\t\tM^{*}_H(x) & \\leq & F(x)  -   \\min\\bigl\\{ \\bigl(\n\t\t\\frac{(1 + \\nu)\\gammafnu}{2(2 + \\nu)}   \\bigr)^{1 / (1 + \\nu)}\n\t\t, \\; 1 \\bigr\\} \\cdot \\frac{(1 + \\nu)}{(2 + \\nu)} \\cdot \\left(\n\t\tF(x) - F^{*} \\right),\n\t\t\\ea\n\t\t\\end{displaymath}\n\t\tand this is the first argument of the minimum in\n\t\t\\eqref{OneStepProgressHolder}.\n\t\t\n\t\t2) $H r_H^{1 - \\nu}(x) > \\frac{2 \\Hfnu}{1 + \\nu}.$\n\t\tBy~\\eqref{CubicGradNewPoint}, we have the bound:\n\t\t\\begin{equation} \\label{NextPointGradBound2}\n\t\t\\ba{rcl}\n\t\t\\| F'(T_H(x)) \\|_{*} & \\; < \\; & Hr_H^2(x).\n\t\t\\ea\n\t\t\\end{equation}\n\t\tUsing the fact that $\\nabla^2 f(x) \\succeq 0$, we get the second argument of the\n\t\tminimum:\n\t$$\n\t\\ba{rcl}\n\tF(x) - M_H^{*}(x)\n\t& \\overset{\\eqref{CubicModelMinimumExpression}}{\\geq} &\n\t\\frac{Hr^3_H(x)}{3} \\quad  \\overset{ \\eqref{NextPointGradBound2}\n\t}{\\geq} \\quad \\frac{\\| F'(T_H(x)) \\|_{*}^{\\frac32}}{3 \\sqrt{H}}\n\t\\\\[5pt]\n\t& \\overset{\\eqref{UniformlyConvexGradientBound}}{\\geq} &\n\t \\left(\n\t\\frac{2 + \\nu}{1 + \\nu}  \\right)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}}\n\t\\cdot \\frac{(\\sigmafnu)^{\\frac{3}{2(2 + \\nu)}}}{ 3 \\sqrt{H} } \\cdot\n\t\\left( F(T_H(x)) - F^{*} \\right)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}}.\n\t\\ea\n\t$$\n\\qed\n\\end{proof}\n\n\\begin{lemma} \\label{LemmaUpHolder} Assume that for a fixed\n\t$x \\in \\dom F$ the parameter $H > 0$ is such that:\n\t\\begin{equation} \\label{LemmaUpHolderCondition}\n\t\\ba{rcl}\n\tF(T_H(x)) & \\; > \\; & M^{*}_H(x).\n\t\\ea\n\t\\end{equation}\n\tIf for some $\\nu \\in [0, 1]$, we have $\\sigmafnu > 0$, then it holds:\n\t\\begin{equation} \\label{LemmaUpHolder3}\n\t\\ba{rcl}\n\tH \\left( F(T_{2H}(x)) - F^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}}\n\t& \\; < \\; & \\kappa_f(\\nu).\n\t\\ea\n\t\\end{equation}\n\\end{lemma}\n\n\\begin{proof}\n\tFirstly, let us prove that from~\\eqref{LemmaUpHolderCondition} we\n\thave:\n\t\\begin{equation} \\label{LemmaUpHolderBound1}\n\t\\ba{rcl}\n\tH r^{1 - \\nu}_H(x) & \\; < \\; & \\frac{6 \\Hfnu}{(1 + \\nu)(2 + \\nu)}.\n\t\\ea\n\t\\end{equation}\n\tAssuming by contradiction, $H r^{1 - \\nu}_H(x) \\; \\geq \\; \\frac{6\n\t\t\\Hfnu}{(1 + \\nu)(2 + \\nu)}$, we get:\n\t$$\n\t\\ba{rcl}\n\tM_H^{*}(x)  & := & \\frac{H \\|T_H(x) - x\\|^3}{6} + Q(x;\n\tT_H(x)) + h(T_H(x)) \\\\[5pt]\n\t& \\geq & \\frac{\\Hfnu \\|T_H(x) - x\\|^{2 + \\nu}}{(1 + \\nu)(2 +\n\t\t\\nu)} + Q(x; T_H(x)) + h(T_H(x)) \\\\[5pt]\n\t& \\overset{\\eqref{HolderHessianFuncBound}}{\\geq} & F(T_H(x)),\n\t\\ea\n\t$$\n\twhich contradicts~\\eqref{LemmaUpHolderCondition}.\n\tSecondly, by its definition, $M^*_H(x)$ is a concave function of $H$.\n\tTherefore, its derivative\n\t$\n\t{d \\over d H} M^*_H(x)  =  {1 \\over 6} r_H^3(x)\n\t$\n\tis non-increasing. Hence, it holds:\n\t\\beq \\label{LemmaUpHolderBound2}\n\t\\ba{rcl}\n\tr_{2H}(x) & \\leq & r_H(x)\n\t\\overset{\\eqref{LemmaUpHolderBound1}}{<} \n\t\\bigl( \\frac{6 \\Hfnu}{(1 + \\nu)(2 + \\nu) H} \\bigr)^{\\frac{1}{1 - \\nu}}.\n\t\\ea\n\t\\eeq\n\tFinally, by the smoothness and the uniform convexity, we obtain:\n\t$$\n\t\\ba{cl}\n\t& H \\left( F(T_{2H}(x)) - F^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}}\n\t\\;\\; \\refLE{UniformlyConvexGradientBound} \\;\\;\n\tH \\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\| F'(T_{2H}(x)) \\|_{*}^{\\frac{1 - \\nu}{1 + \\nu}}  \\\\[5pt]\n\t& \\quad \\refLE{CubicGradNewPoint} \n\tH \\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\Bigl(\n\tr^{1 +\\nu}_{2H}(x) \\cdot \\left(  \\frac{\\Hfnu}{1 + \\nu} + H r^{1 -\n\t\t\\nu}_{2H}(x) \\right)\n\t\\Bigr)^{\\frac{1 - \\nu}{1 + \\nu}} \\\\[5pt]\n\t& \\quad \\overset{\\eqref{LemmaUpHolderBound2}}{<} \n\tH \\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\Bigl( \n\tr^{1+\\nu}_{2H}(x) \\cdot \\frac{(8 + \\nu) \\Hfnu}{(1 + \\nu)(2 + \\nu)}\n\t\\Bigr)^{\\frac{1 - \\nu}{1 + \\nu}} \\\\[5pt]\n\t& \\quad \\overset{\\eqref{LemmaUpHolderBound2}}{<} \n\t\t\\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\left( \\frac{\\Hfnu}{(1 + \\nu)(2 + \\nu)}  \\right)^{\\frac{2}{1 + \\nu}}\n\t6  (8 + \\nu)^{\\frac{1 - \\nu}{1 + \\nu}}\n\t\\; =: \\; \\kappa_f(\\nu).\n\t\\ea\n\t$$\n\\qed\n\\end{proof}\n\n\\begin{proof}\n\tThe proof is based on Lemmas~\\ref{LemmaStepHolder}\n\tand~\\ref{LemmaUpHolder}, and monotonicity of the sequence\n\t$\\bigl\\{ F(x_k)  \\bigr\\}_{k \\geq 0}$.\n\t\tFirstly, we need to show that every iteration of the method is\n\twell-defined. Namely, we are going to verify that for a fixed $0 \\leq\n\tk \\leq K-1$, there exists a finite integer $\\ell \\geq 0$ such that\n\teither $F(T_{H_k 2^{\\ell}}(x_k) ) \\leq M_{H_k 2^{\\ell}}^{*}(x_k)$ or\n\t$F(T_{H_k 2^{\\ell + 1}}(x_k)) - F^{*} < \\varepsilon$.\n\t\tIndeed, let us set\n\t\\begin{equation} \\label{Th2HLowerBound}\n\t\\ba{rcl}\n\t\\ell := \\max\\left\\{0,  \\log_2 \\left \\lceil\n\t\\frac{\\kappa_f(\\nu)}{H_k \\varepsilon^{(1 - \\nu) / (2 + \\nu)} }\n\t\\right\\rceil  \\right\\}, \n\t& \\quad \\text{and} \\quad &\n\tH := H_k\n\t2^{\\ell}  \\geq  \\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1 - \\nu) / (2\n\t\t\t+ \\nu)}}.\n\t\\ea\n\t\\end{equation}\n\tThen, if we have both $F(T_{H}(x_k)) > M_{H}^{*}(x_k)$ and\n\t$F(T_{2H}(x_k)) - F^{*}\n\t\\geq \\varepsilon$, we get by Lemma \\ref{LemmaUpHolder}:\n\t\\begin{displaymath}\n\t\\ba{rcl}\n\tH & \\; \\overset{\\eqref{LemmaUpHolder3}}{<} \\; &\n\t\\frac{\\kappa_f(\\nu)}{\\left( F(T_{2H}(x_k)) - F^{*}  \\right)^{(1 - \\nu)\n\t\t\t/ (2 + \\nu)}  } \\; \\leq \\; \\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1\n\t\t\t- \\nu) / (2 + \\nu)}},\n\t\\ea\n\t\\end{displaymath}\n\twhich contradicts~\\eqref{Th2HLowerBound}. Therefore, if we are unable\n\tto find the value $0 \\leq i_k \\leq \\ell$ (see line~1 of\n\tAlgorithm) in a finite number of steps, that only means we have\n\talready solved the problem up to accuracy $\\varepsilon$.\n\t\n\t\n\t\n\tNow, let us show that for every $0 \\leq k \\leq K$ it holds:\n\t\\begin{equation} \\label{Th2UsefulBound}\n\t\\ba{rcl}\n\tH_k \\left( F(x_k) - F^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t& \\leq &\n\t\\max \\left\\{ \\kappa_f(\\nu), \\; H_0 \\left( F(x_0) - F^{*}\n\t\\right)^{\\frac{1 - \\nu}{2 + \\nu}} \\right\\}.\n\t\\ea\n\t\\end{equation}\n\tThis inequality is obviously valid for $k = 0$. Assume it\n\tis also valid for some $k \\geq 0$. Then, by definition of\n\t$H_{k + 1}$ (see line~3 of Algorithm), we have $H_{k +\n\t\t1} = H_k 2^{i_k - 1}$. \n\tThere are two cases.\n\t\t1) $i_k = 0$. Then, $H_{k + 1} < H_k$. By monotonicity of\n\t\t$\\bigl\\{ F(x_k) \\bigr\\}_{k \\geq 0}$ and by induction, we get:\n\t\t$$\n\t\t\\ba{rcl}\n\t\tH_{k + 1} \\left( F(x_{k + 1}) - F^{*} \\right)^{\\frac{1 - \\nu}{2 +\n\t\t\t\t\\nu}}  & \\; < \\; & H_k \\left( F(x_k) - F^{*} \\right)^{\\frac{1 -\n\t\t\t\t\\nu}{2 + \\nu}} \\\\[5pt]\n\t\t& \\; \\leq \\; & \n\t\t\\max \\left\\{ \\kappa_f(\\nu), \\; H_0 \\left( F(x_0) -\n\t\tF^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \\right\\}.\n\t\t\\ea\n\t\t$$\n\t\t\n\t\t2) $i_k > 0$. Then, applying Lemma~\\ref{LemmaUpHolder} with $H\n\t\t:= H_k 2^{i_k - 1} = H_{k + 1}$ and $x :=\n\t\tx_{k}$, we have:\n\t\t\\begin{displaymath}\n\t\t\\ba{rcl}\n\t\tH_{k + 1} \\left( F(x_{k + 1}) - F^{*} \\right)^{\\frac{1 - \\nu}{2 +\n\t\t\t\t\\nu}} & \\; = \\; & H \\left( F(T_{2H}(x)) - F^{*}\n\t\t\\right)^{\\frac{1 - \\nu}{2 + \\nu}} \\;\n\t\t\\overset{\\eqref{LemmaUpHolder3}}{\\leq} \\; \\kappa_f(\\nu).\n\t\t\\ea\n\t\t\\end{displaymath}\n\tThus, \\eqref{Th2UsefulBound} is true by induction.\n\tChoosing $H_0$ small enough~\\eqref{HolderChoosingH}, we have:\n\t\\begin{equation} \\label{HolderHtGoodBound}\n\t\\ba{rcl}\n\t2H_{k} \\left( F(x_k) - F^{*}\\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t& \\; \\leq \\; & \n\t2\\kappa_f(\\nu), \\qquad 0 \\leq k \\leq K.\n\t\\ea\n\t\\end{equation}\n\tFrom Lemma~\\ref{LemmaStepHolder} we know, that one of the two\n\tfollowing estimates is true (denote $\\delta_k := F(x_k) - F^{*}$):\n\t\n\t\n\t\\begin{enumerate}\n\t\t\\item[1)] $F(x_k) - F(x_{k + 1})  \\geq \\alpha \\cdot \\delta_k\n\t\t \\;  \\Leftrightarrow  \\;  \\delta_{k + 1}  \\leq (1 -\n\t\t\\alpha) \\cdot \\delta_k, \\; $ \\textit{or}\n\t\t\n\t\t\\item[2)] $F(x_k) - F(x_{k + 1})  \\geq \\beta \\cdot \\delta_{k +\n\t\t\t1}  \\; \\Leftrightarrow \\;  \\delta_{k + 1}\n\t \t\\leq (1 + \\beta)^{-1} \\delta_k  \\leq  (1 - \\min\\{\\beta,\n\t\t1\\} / 2) \\cdot \\delta_k$,\n\t\\end{enumerate}\n\twhere\n\t$\n\t\\alpha  \n\t := \n\t \\frac{1 + \\nu}{2 + \\nu} \\cdot\n\t\\min\\bigl\\{\\bigl(  \\frac{(1 + \\nu) \\gammafnu}{2(2 + \\nu)} \\bigr)^{\n\t\t\\frac{1}{1 + \\nu}}, \\; 1\\bigr\\},\n\t$\n\tand\n\t$$\n\t\\ba{rcl}\n\t\\beta \n\t & := & \n\t\\bigl( \\frac{2 + \\nu}{1 + \\nu}\n\t\\bigr)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}} \\cdot\n\t\\frac{(\\sigmafnu)^{\\frac{3}{2(2 + \\nu)}}}{3 (2 \\kappa_f(\\nu))^{1/2}\n\t}\n\t\\; \\refEQ{KappaDefinition} \\;\n\t\\frac{2 + \\nu}{1 + \\nu} \\cdot \\frac{ 2^{1/2} \\cdot\n\t\t\\left( (1 + \\nu) (2 + \\nu)  \\right)^{\\frac{1}{1 + \\nu}} }{6^{3/2}\n\t\t\\cdot (8 + \\nu)^{(1 - \\nu) / (2 + 2\\nu)}} \\cdot \\gammafnu^{\\frac{1}{1\n\t\t\t+ \\nu}}.\n\t\\ea\n\t$$\n\tIt remains to notice, that $\\alpha \\geq \\min \\bigl\\{\\beta, 1\\bigr\\} /\n\t2$. Thus, we obtain~\\eqref{HolderMainResult}.\n\t\n\tFinally, let us estimate the total number of the oracle calls $N_K$\n\tduring the first $K$ iterations. At each\n\titeration, the oracle is called $i_k + 1$ times, and we have $H_{k +\n\t\t1} = H_k 2^{i_k - 1}$. Therefore,\n\t$$\n\t\\ba{rcl}\n\tN_K & \\; = \\; & \\sum_{k = 0}^{K - 1} (i_k + 1) \\; = \\; \\sum_{k = 0}^{K -\n\t\t1} \\left( \\log_2 \\frac{H_{k + 1}}{H_k} + 2\\right) \\\\[5pt]\n\t& \\; = \\; & 2K + \\log_2\n\tH_{K} - \\log_2 H_0\n\t\\; \\overset{\\eqref{HolderHtGoodBound},\n\t\t\\eqref{HolderConditionFuncEps}}{\\leq} \\;\n\t2K + \\log_2\n\t\\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1 - \\nu) / (2 + \\nu)}} - \\log_2\n\tH_0.\n\t\\ea\n\t$$\n\\qed\n\\end{proof}","units":[{"raw_tex":"\\begin{lemma} \\label{UniformlyConvexConsequence}\n\tLet $f$ be uniformly convex of degree $p \\geq 2$ with\n\tconstant $\\sigma > 0$ on set $\\dom F$. Then, for every $x\n\t\\in \\dom F$ and for all $F'(x) \\in \\partial F(x)$ we have\n\t\\begin{equation} \\label{UniformlyConvexGradientBound}\n\t\\ba{rcl}\n\tF(x) - F^{*} & \\; \\leq \\; & \\frac{p - 1}{p} \\left( \\frac{1}{\\sigma}\n\t\\right)^{\\frac{1}{p - 1}} \\|F'(x)\\|_{*}^{\\frac{p}{p - 1}}.\n\t\\ea\n\t\\end{equation}\n\\end{lemma}","start":11630,"end":12078},{"raw_tex":"\\begin{proof}\n\t\\noindent\n\tIn view of~\\eqref{UniformlyConvexLowerBound}, \n\tbound \\eqref{UniformlyConvexGradientBound}\n\tfollows as in the proof of Lemma~3 \n\tin~\\cite{nesterov2008accelerating}.\n\\qed\n\\end{proof}","start":12079,"end":12286},{"raw_tex":"\\begin{proof}\n\tDenote $\\kappa(x) \\equiv \\sum_{i = 1}^m e^{\\langle a_i, x \\rangle}$.\n\tLet us fix arbitrary $x, y \\in \\E$ and direction $h \\in \\E$. Then,\n\tstraightforward computation gives:\n\t$$\n\t\\ba{rcl}\n\t\\langle \\nabla f(x), h \\rangle &\\; = \\;& \\frac{1}{\\kappa(x)}\n\t\\sum_{i = 1}^m e^{\\langle a_i, x \\rangle} \\langle a_i, h \\rangle, \\\\[6pt]\n\t\\langle \\nabla^2 f(x)h, h \\rangle &\\; =\\;&\n\t \\frac{1}{\\kappa(x)} \\sum_{i = 1}^m e^{\\langle a_i, x \\rangle}\n\t\\langle\n\ta_i, h \\rangle^2 - \\bigl( \\frac{1}{\\kappa(x)} \\sum_{i = 1}^m\n\te^{\\langle a_i, x \\rangle} \\langle a_i, h \\rangle  \\bigr)^2 \\\\[5pt]\n\t&\\; = \\;& \\frac{1}{\\kappa(x)} \\sum_{i = 1}^m e^{\\langle a_i, x \\rangle}\n\t\\left( \\langle\n\ta_i, h \\rangle - \\langle \\nabla f(x), h \\rangle\n\t\\right)^2 \\; \\geq \\; 0.\n\t\\ea\n\t$$\n\tHence, we get\n\t$$\n\t\\ba{rcl}\n\t\\| \\nabla^2 f(x) \\| & = & \\max\\limits_{\\|h\\| \\leq 1} \\langle \\nabla^2 f(x)\n\th, h \\rangle \\;\n\t\\leq \\;  \\max\\limits_{\\|h\\| \\leq 1} \\sum_{i = 1}^m \\langle a_i, h \\rangle^2\n\t\\;\n\t= \\; \\max\\limits_{\\|h\\| \\leq 1} \\|h\\|^2 \\; = \\; 1.\n\t\\ea\n\t$$\n\tSince all Hessians of function $f$ are positive definite,\n\twe conclude that $\\Hf(0) \\leq 1$. Inequality $\\Hf(1) \\leq\n\t2$ can be easily obtained from the following\n\trepresentation of the third derivative:\n\t$$\n\t\\ba{rcl}\n\tf'''(x)[h,h,h]  & \\; = \\; & {1 \\over \\kappa(x)}\n\t\\sum\\limits_{i=1}^m e^{\\la a_i, x \\ra} \\left( \\la a_i, h\n\t\\ra - \\la \\nabla f(x), h \\ra \\right)^3\\\\\n\t\\\\\n\t& \\;  \\leq \\; & \\la \\nabla^2 f(x) h, h \\ra \\max\\limits_{1 \\leq\n\t\ti,j \\leq m } \\la a_i - a_j , h \\ra \\; \\leq \\; 2 \\| h \\|^3.\n\t\\ea\n\t$$\n\\qed\n\\end{proof}","start":16648,"end":18193},{"raw_tex":"\\begin{lemma} \\label{LemmaStepHolder}\n\tLet $0 < \\Hfnu < +\\infty$ and $\\sigmafnu > 0$ for some\n\t$\\nu \\in [0,1]$. Then, for arbitrary $x \\in \\dom F$ and $H\n\t> 0$ we have:\n\t\\beq \\label{OneStepProgressHolder}\n\t\\ba{cl}\n\t& F(x) - M_H^{*}(x)  \\\\[5pt] \n\t& \\quad \\geq \\; \\min\\Bigl[ \\left( F(x) -\n\tF^{*} \\right) \\cdot \\frac{(1 + \\nu)}{(2 + \\nu)} \\cdot\n\t\\min\\bigl\\{\\bigl( \\frac{(1 + \\nu) \\gammafnu}{2(2 + \\nu)}  \\bigr)^{\\frac{1}{1 + \\nu}}, \n\t\\; 1\\bigr\\} ,   \\\\[5pt]\n\t& \\; \\qquad \\qquad \\left( F(T_H(x)) - F^{*} \\right)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}}\n\t\\cdot \\bigl ( \\frac{2 + \\nu}{1 + \\nu}  \\bigr)^{\\frac{3(1 + \\nu)}{2(2\n\t\t\t+ \\nu)}} \\cdot \\frac{(\\sigmafnu)^{\\frac{3}{2(2 + \\nu)}}}{ 3 \\sqrt{H}\n\t}  \\;  \\Bigr]. \n\t\\ea\n\t\\eeq\n\\end{lemma}","start":48378,"end":49099},{"raw_tex":"\\begin{proof}\n\tLet us consider two cases. 1) \n\t$H r_H^{1 - \\nu}(x) \\leq \\frac{2 \\Hfnu}{1 + \\nu}$.\n\t\tThen, for arbitrary $y \\in \\dom F$, we have:\n\t\t$$\n\t\t\\ba{rcl}\n\t\tM_H^{*}(x) \n\t\t& := &  Q(x; T_H(x)) +\n\t\t\\frac{H}{6}\\|T_H(x) - x\\|^3 + h(T_H(x)) \\\\[7pt]\n\t\t& \\leq &  Q(x; y) + \\frac{H r_H^{1 - \\nu}(x) \\|y -\n\t\t\tx\\|^{2 + \\nu}}{2 (2 + \\nu) } + h(y) \\\\[5pt]\n\t\t& \\overset{\\eqref{HolderHessianFuncBound}}{\\leq} &  \n\t\tF(y) +\n\t\t\\frac{\\Hfnu \\|y - x\\|^{2 + \\nu}}{(1 + \\nu)(2 + \\nu)} +\n\t\t\\frac{H r_H^{1 - \\nu}(x) \\|y - x\\|^{2 + \\nu}}{2 (2 + \\nu) }\n\t\t\\\\[7pt]\n\t\t& \\leq &\n\t\t F(y) +  \\frac{2\\Hfnu \\|y - x\\|^{2 +\n\t\t\\nu}}{(1 + \\nu)(2 + \\nu)},\n\t\t\\ea\n\t\t$$\n\t\twhere the first inequality follows from the fact, that\n\t\t$$\n\t\t\\ba{rcl}\n\t\tT_H(x) & = & \\argmin\\limits_{y \\in \\dom F} \\bigl\\{ Q(x; y)\n\t\t+ \\frac{H r_H^{1 - \\nu}(x) \\|y -\n\t\t\tx\\|^{2 + \\nu}}{2 (2 + \\nu) } + h(y) \\bigr\\}.\n\t\t\\ea\n\t\t$$\n\t\tLet us restrict $y$ to the segment: $y = \\alpha x^{*} + (1 -\n\t\t\\alpha) x,$ with $\\alpha \\in [0, 1]$. Taking into account the\n\t\tuniform convexity, we get:\n\t\t$$\n\t\t\\ba{rcl}\n\t\tM_H^{*}(x) & \\leq &  F(x) - \\alpha \\left( F(x) -\n\t\tF^{*} \\right) + \\alpha^{2 + \\nu} \\frac{2\\Hfnu \\|x^{*} - x\\|^{2 +\n\t\t\t\t\\nu}} {(1 + \\nu)(2 + \\nu)} \\quad \\\\[5pt]\n\t\t& \\overset{\\eqref{UniformlyConvexLowerBound}}{\\leq} &  \n\t\tF(x)\n\t\t-  \\Bigl(\\alpha  - \\alpha^{2 + \\nu} \\frac{2\\Hfnu}{  (1 + \\nu)\n\t\t\t\\sigmafnu}\n\t\t\\Bigr) \\left( F(x) - F^{*} \\right).\n\t\t\\ea\n\t\t$$\n\t\tThe minimum of the right-hand side is attained at\n\t\t$\n\t\t\\alpha^{*} = \\min\\bigl\\{\\frac{(1 + \\nu)\\gammafnu}{2(2 + \\nu)}, 1 \\bigr\\}^{\\frac{1}{1 + \\nu}}.\n\t\t$\n\t\tPlugging this value into the bound, we have:\n\t\t\\begin{displaymath}\n\t\t\\ba{rcl}\n\t\tM^{*}_H(x) & \\leq & F(x)  -   \\min\\bigl\\{ \\bigl(\n\t\t\\frac{(1 + \\nu)\\gammafnu}{2(2 + \\nu)}   \\bigr)^{1 / (1 + \\nu)}\n\t\t, \\; 1 \\bigr\\} \\cdot \\frac{(1 + \\nu)}{(2 + \\nu)} \\cdot \\left(\n\t\tF(x) - F^{*} \\right),\n\t\t\\ea\n\t\t\\end{displaymath}\n\t\tand this is the first argument of the minimum in\n\t\t\\eqref{OneStepProgressHolder}.\n\t\t\n\t\t2) $H r_H^{1 - \\nu}(x) > \\frac{2 \\Hfnu}{1 + \\nu}.$\n\t\tBy~\\eqref{CubicGradNewPoint}, we have the bound:\n\t\t\\begin{equation} \\label{NextPointGradBound2}\n\t\t\\ba{rcl}\n\t\t\\| F'(T_H(x)) \\|_{*} & \\; < \\; & Hr_H^2(x).\n\t\t\\ea\n\t\t\\end{equation}\n\t\tUsing the fact that $\\nabla^2 f(x) \\succeq 0$, we get the second argument of the\n\t\tminimum:\n\t$$\n\t\\ba{rcl}\n\tF(x) - M_H^{*}(x)\n\t& \\overset{\\eqref{CubicModelMinimumExpression}}{\\geq} &\n\t\\frac{Hr^3_H(x)}{3} \\quad  \\overset{ \\eqref{NextPointGradBound2}\n\t}{\\geq} \\quad \\frac{\\| F'(T_H(x)) \\|_{*}^{\\frac32}}{3 \\sqrt{H}}\n\t\\\\[5pt]\n\t& \\overset{\\eqref{UniformlyConvexGradientBound}}{\\geq} &\n\t \\left(\n\t\\frac{2 + \\nu}{1 + \\nu}  \\right)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}}\n\t\\cdot \\frac{(\\sigmafnu)^{\\frac{3}{2(2 + \\nu)}}}{ 3 \\sqrt{H} } \\cdot\n\t\\left( F(T_H(x)) - F^{*} \\right)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}}.\n\t\\ea\n\t$$\n\\qed\n\\end{proof}","start":49100,"end":51890},{"raw_tex":"\\begin{lemma} \\label{LemmaUpHolder} Assume that for a fixed\n\t$x \\in \\dom F$ the parameter $H > 0$ is such that:\n\t\\begin{equation} \\label{LemmaUpHolderCondition}\n\t\\ba{rcl}\n\tF(T_H(x)) & \\; > \\; & M^{*}_H(x).\n\t\\ea\n\t\\end{equation}\n\tIf for some $\\nu \\in [0, 1]$, we have $\\sigmafnu > 0$, then it holds:\n\t\\begin{equation} \\label{LemmaUpHolder3}\n\t\\ba{rcl}\n\tH \\left( F(T_{2H}(x)) - F^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}}\n\t& \\; < \\; & \\kappa_f(\\nu).\n\t\\ea\n\t\\end{equation}\n\\end{lemma}","start":52438,"end":52912},{"raw_tex":"\\begin{proof}\n\tFirstly, let us prove that from~\\eqref{LemmaUpHolderCondition} we\n\thave:\n\t\\begin{equation} \\label{LemmaUpHolderBound1}\n\t\\ba{rcl}\n\tH r^{1 - \\nu}_H(x) & \\; < \\; & \\frac{6 \\Hfnu}{(1 + \\nu)(2 + \\nu)}.\n\t\\ea\n\t\\end{equation}\n\tAssuming by contradiction, $H r^{1 - \\nu}_H(x) \\; \\geq \\; \\frac{6\n\t\t\\Hfnu}{(1 + \\nu)(2 + \\nu)}$, we get:\n\t$$\n\t\\ba{rcl}\n\tM_H^{*}(x)  & := & \\frac{H \\|T_H(x) - x\\|^3}{6} + Q(x;\n\tT_H(x)) + h(T_H(x)) \\\\[5pt]\n\t& \\geq & \\frac{\\Hfnu \\|T_H(x) - x\\|^{2 + \\nu}}{(1 + \\nu)(2 +\n\t\t\\nu)} + Q(x; T_H(x)) + h(T_H(x)) \\\\[5pt]\n\t& \\overset{\\eqref{HolderHessianFuncBound}}{\\geq} & F(T_H(x)),\n\t\\ea\n\t$$\n\twhich contradicts~\\eqref{LemmaUpHolderCondition}.\n\tSecondly, by its definition, $M^*_H(x)$ is a concave function of $H$.\n\tTherefore, its derivative\n\t$\n\t{d \\over d H} M^*_H(x)  =  {1 \\over 6} r_H^3(x)\n\t$\n\tis non-increasing. Hence, it holds:\n\t\\beq \\label{LemmaUpHolderBound2}\n\t\\ba{rcl}\n\tr_{2H}(x) & \\leq & r_H(x)\n\t\\overset{\\eqref{LemmaUpHolderBound1}}{<} \n\t\\bigl( \\frac{6 \\Hfnu}{(1 + \\nu)(2 + \\nu) H} \\bigr)^{\\frac{1}{1 - \\nu}}.\n\t\\ea\n\t\\eeq\n\tFinally, by the smoothness and the uniform convexity, we obtain:\n\t$$\n\t\\ba{cl}\n\t& H \\left( F(T_{2H}(x)) - F^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}}\n\t\\;\\; \\refLE{UniformlyConvexGradientBound} \\;\\;\n\tH \\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\| F'(T_{2H}(x)) \\|_{*}^{\\frac{1 - \\nu}{1 + \\nu}}  \\\\[5pt]\n\t& \\quad \\refLE{CubicGradNewPoint} \n\tH \\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\Bigl(\n\tr^{1 +\\nu}_{2H}(x) \\cdot \\left(  \\frac{\\Hfnu}{1 + \\nu} + H r^{1 -\n\t\t\\nu}_{2H}(x) \\right)\n\t\\Bigr)^{\\frac{1 - \\nu}{1 + \\nu}} \\\\[5pt]\n\t& \\quad \\overset{\\eqref{LemmaUpHolderBound2}}{<} \n\tH \\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\Bigl( \n\tr^{1+\\nu}_{2H}(x) \\cdot \\frac{(8 + \\nu) \\Hfnu}{(1 + \\nu)(2 + \\nu)}\n\t\\Bigr)^{\\frac{1 - \\nu}{1 + \\nu}} \\\\[5pt]\n\t& \\quad \\overset{\\eqref{LemmaUpHolderBound2}}{<} \n\t\t\\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\left( \\frac{\\Hfnu}{(1 + \\nu)(2 + \\nu)}  \\right)^{\\frac{2}{1 + \\nu}}\n\t6  (8 + \\nu)^{\\frac{1 - \\nu}{1 + \\nu}}\n\t\\; =: \\; \\kappa_f(\\nu).\n\t\\ea\n\t$$\n\\qed\n\\end{proof}","start":52913,"end":55281},{"raw_tex":"\\begin{proof}\n\tThe proof is based on Lemmas~\\ref{LemmaStepHolder}\n\tand~\\ref{LemmaUpHolder}, and monotonicity of the sequence\n\t$\\bigl\\{ F(x_k)  \\bigr\\}_{k \\geq 0}$.\n\t\tFirstly, we need to show that every iteration of the method is\n\twell-defined. Namely, we are going to verify that for a fixed $0 \\leq\n\tk \\leq K-1$, there exists a finite integer $\\ell \\geq 0$ such that\n\teither $F(T_{H_k 2^{\\ell}}(x_k) ) \\leq M_{H_k 2^{\\ell}}^{*}(x_k)$ or\n\t$F(T_{H_k 2^{\\ell + 1}}(x_k)) - F^{*} < \\varepsilon$.\n\t\tIndeed, let us set\n\t\\begin{equation} \\label{Th2HLowerBound}\n\t\\ba{rcl}\n\t\\ell := \\max\\left\\{0,  \\log_2 \\left \\lceil\n\t\\frac{\\kappa_f(\\nu)}{H_k \\varepsilon^{(1 - \\nu) / (2 + \\nu)} }\n\t\\right\\rceil  \\right\\}, \n\t& \\quad \\text{and} \\quad &\n\tH := H_k\n\t2^{\\ell}  \\geq  \\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1 - \\nu) / (2\n\t\t\t+ \\nu)}}.\n\t\\ea\n\t\\end{equation}\n\tThen, if we have both $F(T_{H}(x_k)) > M_{H}^{*}(x_k)$ and\n\t$F(T_{2H}(x_k)) - F^{*}\n\t\\geq \\varepsilon$, we get by Lemma \\ref{LemmaUpHolder}:\n\t\\begin{displaymath}\n\t\\ba{rcl}\n\tH & \\; \\overset{\\eqref{LemmaUpHolder3}}{<} \\; &\n\t\\frac{\\kappa_f(\\nu)}{\\left( F(T_{2H}(x_k)) - F^{*}  \\right)^{(1 - \\nu)\n\t\t\t/ (2 + \\nu)}  } \\; \\leq \\; \\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1\n\t\t\t- \\nu) / (2 + \\nu)}},\n\t\\ea\n\t\\end{displaymath}\n\twhich contradicts~\\eqref{Th2HLowerBound}. Therefore, if we are unable\n\tto find the value $0 \\leq i_k \\leq \\ell$ (see line~1 of\n\tAlgorithm) in a finite number of steps, that only means we have\n\talready solved the problem up to accuracy $\\varepsilon$.\n\t\n\t\n\t\n\tNow, let us show that for every $0 \\leq k \\leq K$ it holds:\n\t\\begin{equation} \\label{Th2UsefulBound}\n\t\\ba{rcl}\n\tH_k \\left( F(x_k) - F^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t& \\leq &\n\t\\max \\left\\{ \\kappa_f(\\nu), \\; H_0 \\left( F(x_0) - F^{*}\n\t\\right)^{\\frac{1 - \\nu}{2 + \\nu}} \\right\\}.\n\t\\ea\n\t\\end{equation}\n\tThis inequality is obviously valid for $k = 0$. Assume it\n\tis also valid for some $k \\geq 0$. Then, by definition of\n\t$H_{k + 1}$ (see line~3 of Algorithm), we have $H_{k +\n\t\t1} = H_k 2^{i_k - 1}$. \n\tThere are two cases.\n\t\t1) $i_k = 0$. Then, $H_{k + 1} < H_k$. By monotonicity of\n\t\t$\\bigl\\{ F(x_k) \\bigr\\}_{k \\geq 0}$ and by induction, we get:\n\t\t$$\n\t\t\\ba{rcl}\n\t\tH_{k + 1} \\left( F(x_{k + 1}) - F^{*} \\right)^{\\frac{1 - \\nu}{2 +\n\t\t\t\t\\nu}}  & \\; < \\; & H_k \\left( F(x_k) - F^{*} \\right)^{\\frac{1 -\n\t\t\t\t\\nu}{2 + \\nu}} \\\\[5pt]\n\t\t& \\; \\leq \\; & \n\t\t\\max \\left\\{ \\kappa_f(\\nu), \\; H_0 \\left( F(x_0) -\n\t\tF^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \\right\\}.\n\t\t\\ea\n\t\t$$\n\t\t\n\t\t2) $i_k > 0$. Then, applying Lemma~\\ref{LemmaUpHolder} with $H\n\t\t:= H_k 2^{i_k - 1} = H_{k + 1}$ and $x :=\n\t\tx_{k}$, we have:\n\t\t\\begin{displaymath}\n\t\t\\ba{rcl}\n\t\tH_{k + 1} \\left( F(x_{k + 1}) - F^{*} \\right)^{\\frac{1 - \\nu}{2 +\n\t\t\t\t\\nu}} & \\; = \\; & H \\left( F(T_{2H}(x)) - F^{*}\n\t\t\\right)^{\\frac{1 - \\nu}{2 + \\nu}} \\;\n\t\t\\overset{\\eqref{LemmaUpHolder3}}{\\leq} \\; \\kappa_f(\\nu).\n\t\t\\ea\n\t\t\\end{displaymath}\n\tThus, \\eqref{Th2UsefulBound} is true by induction.\n\tChoosing $H_0$ small enough~\\eqref{HolderChoosingH}, we have:\n\t\\begin{equation} \\label{HolderHtGoodBound}\n\t\\ba{rcl}\n\t2H_{k} \\left( F(x_k) - F^{*}\\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t& \\; \\leq \\; & \n\t2\\kappa_f(\\nu), \\qquad 0 \\leq k \\leq K.\n\t\\ea\n\t\\end{equation}\n\tFrom Lemma~\\ref{LemmaStepHolder} we know, that one of the two\n\tfollowing estimates is true (denote $\\delta_k := F(x_k) - F^{*}$):\n\t\n\t\n\t\\begin{enumerate}\n\t\t\\item[1)] $F(x_k) - F(x_{k + 1})  \\geq \\alpha \\cdot \\delta_k\n\t\t \\;  \\Leftrightarrow  \\;  \\delta_{k + 1}  \\leq (1 -\n\t\t\\alpha) \\cdot \\delta_k, \\; $ \\textit{or}\n\t\t\n\t\t\\item[2)] $F(x_k) - F(x_{k + 1})  \\geq \\beta \\cdot \\delta_{k +\n\t\t\t1}  \\; \\Leftrightarrow \\;  \\delta_{k + 1}\n\t \t\\leq (1 + \\beta)^{-1} \\delta_k  \\leq  (1 - \\min\\{\\beta,\n\t\t1\\} / 2) \\cdot \\delta_k$,\n\t\\end{enumerate}\n\twhere\n\t$\n\t\\alpha  \n\t := \n\t \\frac{1 + \\nu}{2 + \\nu} \\cdot\n\t\\min\\bigl\\{\\bigl(  \\frac{(1 + \\nu) \\gammafnu}{2(2 + \\nu)} \\bigr)^{\n\t\t\\frac{1}{1 + \\nu}}, \\; 1\\bigr\\},\n\t$\n\tand\n\t$$\n\t\\ba{rcl}\n\t\\beta \n\t & := & \n\t\\bigl( \\frac{2 + \\nu}{1 + \\nu}\n\t\\bigr)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}} \\cdot\n\t\\frac{(\\sigmafnu)^{\\frac{3}{2(2 + \\nu)}}}{3 (2 \\kappa_f(\\nu))^{1/2}\n\t}\n\t\\; \\refEQ{KappaDefinition} \\;\n\t\\frac{2 + \\nu}{1 + \\nu} \\cdot \\frac{ 2^{1/2} \\cdot\n\t\t\\left( (1 + \\nu) (2 + \\nu)  \\right)^{\\frac{1}{1 + \\nu}} }{6^{3/2}\n\t\t\\cdot (8 + \\nu)^{(1 - \\nu) / (2 + 2\\nu)}} \\cdot \\gammafnu^{\\frac{1}{1\n\t\t\t+ \\nu}}.\n\t\\ea\n\t$$\n\tIt remains to notice, that $\\alpha \\geq \\min \\bigl\\{\\beta, 1\\bigr\\} /\n\t2$. Thus, we obtain~\\eqref{HolderMainResult}.\n\t\n\tFinally, let us estimate the total number of the oracle calls $N_K$\n\tduring the first $K$ iterations. At each\n\titeration, the oracle is called $i_k + 1$ times, and we have $H_{k +\n\t\t1} = H_k 2^{i_k - 1}$. Therefore,\n\t$$\n\t\\ba{rcl}\n\tN_K & \\; = \\; & \\sum_{k = 0}^{K - 1} (i_k + 1) \\; = \\; \\sum_{k = 0}^{K -\n\t\t1} \\left( \\log_2 \\frac{H_{k + 1}}{H_k} + 2\\right) \\\\[5pt]\n\t& \\; = \\; & 2K + \\log_2\n\tH_{K} - \\log_2 H_0\n\t\\; \\overset{\\eqref{HolderHtGoodBound},\n\t\t\\eqref{HolderConditionFuncEps}}{\\leq} \\;\n\t2K + \\log_2\n\t\\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1 - \\nu) / (2 + \\nu)}} - \\log_2\n\tH_0.\n\t\\ea\n\t$$\n\\qed\n\\end{proof}","start":57237,"end":62279}]},"nodes":[{"id":"S001","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"原对偶空间与范数","source_units":[{"raw_tex":"Let us start from some notation. In what follows, we denote\nby $\\E$ a finite-dimensional real vector space and by\n$\\E^{*}$ its dual space, which is a space of linear\nfunctions on $\\E$. The value of function $s \\in \\E^{*}$ at\npoint $x \\in \\E$ is denoted by $\\langle s, x \\rangle$.\nLet us fix some linear self-adjoint positive-definite operator\n$B: \\E \\to \\E^{*}$ and introduce the following Euclidean norms in the\nprimal and dual spaces:\n$$\n\\ba{rcl}\n\\|x\\| & \\defeq &  \\langle Bx, x \\rangle^{1/2}, \\quad x \\in \\E,\n\\qquad \\|s\\|_{*} \\; \\defeq \\; \\langle s, B^{-1}s \\rangle^{1/2},\n\\quad s \\in \\E^{*}.\n\\ea\n$$\nFor any linear operator $A: \\E \\to \\E^{*}$, its norm is induced in a\nstandard way:\n$$\n\\ba{rcl}\n\\|A\\| & \\defeq & \\max\\limits_{x \\in \\E} \\bigl\\{ \\|Ax\\|_{*} \\; | \\;\n\\|x\\| \\leq 1 \\bigr\\}.\n\\ea\n$$\n","start":8361,"end":9155}]},{"id":"S002","semantic_type":"problem_setting","kind":"problem_setting","labels":["MainProblem"],"declared_external":false,"source_only":false,"title":"复合凸优化问题","source_units":[{"raw_tex":"Our goal is to solve the convex optimization problem\nin the composite form:\n\\beq \\label{MainProblem}\n\\ba{rcl}\n\\min\\limits_{x \\in \\dom F} F(x) & := & f(x) + h(x),\n\\ea\n\\eeq\nwhere $f$ is a twice differentiable on its open domain\nuniformly convex function, and $h$ is a \\textit{simple}\nclosed convex function with $\\dom h \\subseteq \\dom f$.\n\\textit{Simple} means that all auxiliary subproblems with\nan explicit presence of $h$ are easily solvable.\n","start":9155,"end":9599}]},{"id":"S003","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"导数与次微分","source_units":[{"raw_tex":"\nFor a smooth function $f$, its gradient at point $x$ is\ndenoted by $\\nabla f(x) \\in \\E^{*}$,  and its Hessian is\ndenoted by $\\nabla^2 f(x) : \\E \\to \\E^{*}$. For convex but\nnot necessary differentiable function $h$, we denote by\n$\\partial h(x) \\subset \\E^{*}$ its subdifferential at the\npoint $x \\in \\dom h$.\n\n","start":9599,"end":9909}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["UniformlyConvexDefinition"],"declared_external":false,"source_only":false,"title":"一致凸性定义","source_units":[{"raw_tex":"We say that differentiable function $f$ is \\textit{uniformly convex}\nof degree $p \\geq 2$ on a convex set $C \\subseteq \\dom f$\nif for some constant $\\sigma > 0$ it satisfies inequality\n\\beq \\label{UniformlyConvexDefinition}\n\\ba{rcl}\nf(y) & \\; \\geq \\; & f(x) + \\langle \\nabla f(x), y - x \\rangle +\n\\frac{\\sigma}{p}\\|y - x\\|^p, \\qquad x, y \\in C.\n\\ea\n\\eeq\nUniformly convex functions of degree $p = 2$ are known as\n\\textit{strongly convex}. If\ninequality~\\eqref{UniformlyConvexDefinition} holds with\n$\\sigma = 0$, the function $f$ is called just \\textit{convex}.\n","start":9909,"end":10469}]},{"id":"S005","semantic_type":"external_fact","kind":"external_fact","labels":["UniformlyConvexSufficient"],"declared_external":true,"declaration_text":["原前提清单明确允许 B0007；原文引用 Nesterov (2008), Accelerating the cubic regularization of Newton's method on convex problems, Lemma 1。"],"source_only":false,"title":"梯度单调性充分条件","source_units":[{"raw_tex":"The following convenient condition\nis sufficient for function $f$ to be uniformly convex on a convex set\n$C \\subseteq \\dom f$:\n\\begin{lemma}(Lemma 1 in~\\cite{nesterov2008accelerating})\n\tLet for some $\\sigma > 0$ and $p \\geq 2$\n\tthe following inequality holds:\n\t\\beq \\label{UniformlyConvexSufficient}\n\t\\ba{rcl}\n\t\\langle \\nabla f(x) - \\nabla f(y), x - y \\rangle & \\; \\geq \\; &\n\t\\sigma \\|x - y\\|^p, \\qquad x, y \\in C.\n\t\\ea\n\t\\eeq\n\tThen, function $f$ is uniformly convex of degree $p$ on set $C$ with\n\tparameter $\\sigma$.\n\\end{lemma}\n","start":10469,"end":10998}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"一致凸性作用域","source_units":[{"raw_tex":"From now on, we assume\n$\nC \\; := \\; \\dom F \\; \\subseteq \\; \\dom f.\n$\n","start":10998,"end":11067}]},{"id":"S007","semantic_type":"derived_fact","kind":"derived_fact","labels":["UniformlyConvexLowerBound"],"declared_external":false,"source_only":false,"title":"复合目标一致凸下界","source_units":[{"raw_tex":"By the composite representation~\\eqref{MainProblem},\nwe have for every $x \\in \\dom F$ and for\nall $F'(x) \\in \\partial F(x)$:\n\\begin{equation} \\label{UniformlyConvexLowerBound}\n\\ba{rcl}\nF(y) & \\geq  & F(x) + \\langle\nF'(x), y - x \\rangle + \\frac{\\sigma}{p}\\|x - y\\|^p, \\qquad\ny \\in \\dom F.\n\\ea\n\\end{equation}\n","start":11067,"end":11374}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"最优解与最优值","source_units":[{"raw_tex":"Therefore, if $\\sigma > 0$, then we can have only one\npoint $x^{*} \\in \\dom F$ with $F(x^{*}) = F^{*}$, which\nalways exists for $F$ being uniformly convex and closed.\n","start":11374,"end":11541}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":["SigmaDefinition"],"declared_external":false,"source_only":false,"title":"一致凸梯度模量","source_units":[{"raw_tex":"It is reasonable to define the best possible constant~$\\sigma$ in\ninequality~\\eqref{UniformlyConvexSufficient} for a certain degree~$p$.\nThis leads us to a system of constants:\n\\begin{equation} \\label{SigmaDefinition}\n\\ba{rcl}\n\\sigmafp & \\; \\defeq \\; & \\inf\\limits_{\\substack{x, y \\, \\in \\, \\dom F \\\\\n\t\tx \\not= y }} \\frac{\\langle \\nabla f(x) - \\nabla f(y), x - y \\rangle }\n{\\|x - y\\|^p}, \\qquad p \\geq 2.\n\\ea\n\\end{equation}\nWe prefer to use\ninequality~\\eqref{UniformlyConvexSufficient} for the\ndefinition of $\\sigmafp$, instead\nof~\\eqref{UniformlyConvexDefinition}, because of its\nsymmetry in $x$ and $y$. Note that the value $\\sigmafp$\nalso depends on the domain of $F$. However, we omit this\ndependence in our notation since it is always clear from\nthe context.\n","start":12288,"end":13052}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":["HolderHessianDefinition"],"declared_external":false,"source_only":false,"title":"海森矩阵霍尔德条件","source_units":[{"raw_tex":"For a twice-differentiable function $f$, we say\nthat it has \\textit{H\\\"older continuous Hessian} of degree\n$\\nu \\in [0, 1]$ on a convex set $C \\subseteq \\dom f$, if for some\nconstant $\\mathcal{H}$, it\nholds:\n\\begin{equation} \\label{HolderHessianDefinition}\n\\ba{rcl}\n\\| \\nabla^2 f(x) - \\nabla^2 f(y) \\| & \\leq &\n\\mathcal{H} \\|x - y\\|^{\\nu}, \\qquad x, y \\in C.\n\\ea\n\\end{equation}\n","start":13435,"end":13813}]},{"id":"S011","semantic_type":"derived_fact","kind":"derived_fact","labels":["HolderHessianGradBound"],"declared_external":false,"source_only":false,"title":"梯度泰勒余项","source_units":[{"raw_tex":"Two simple consequences of~\\eqref{HolderHessianDefinition}\nare as follows:\n\\beq \\label{HolderHessianGradBound}\n\\ba{rcl}\n\\| \\nabla f(y) - \\nabla f(x) - \\nabla^2 f(x)(y - x) \\|_{*} \n& \\leq &\n \\frac{\\H \\|x - y\\|^{1 + \\nu}}{1 + \\nu},\n\\ea\n\\eeq\n","start":13813,"end":14052}]},{"id":"S012","semantic_type":"derived_fact","kind":"derived_fact","labels":["HolderHessianFuncBound"],"declared_external":false,"source_only":false,"title":"函数泰勒余项","source_units":[{"raw_tex":"\\beq \\label{HolderHessianFuncBound}\n\\ba{rcl}\n| f(y) - Q(x; y) |  & \\leq & \n\\frac{\\H \\|x - y\\|^{2 + \\nu}}{(1 + \\nu)(2 + \\nu)},\n\\ea\n\\eeq\n","start":14052,"end":14187}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"局部二次模型","source_units":[{"raw_tex":"where $Q(x; y)$ is the quadratic model of $f$ at the point\n$x$:\n$$\n\\ba{rcl}\nQ(x; y) & \\; \\defeq \\; & f(x) + \\langle \\nabla f(x), y - x \\rangle +\n\\frac{1}{2} \\langle \\nabla^2 f(x) (y - x), y - x \\rangle.\n\\ea\n$$\n","start":14187,"end":14397}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":["HfnuDefinition"],"declared_external":false,"source_only":false,"title":"霍尔德最优模量","source_units":[{"raw_tex":"\nIn order to characterize the level of smoothness of\nfunction $f$ on the set $C := \\dom F$, let us define\nthe system of H\\\"older constants (see\n\\cite{grapiglia2017regularized}):\n\\begin{equation} \\label{HfnuDefinition}\n\\ba{rcl}\n\\Hfnu & \\; \\defeq &\\; \\sup\\limits_{\\substack{x, y \\in \\dom F \\\\ x \\not= y}}\n\\frac{\\| \\nabla^2 f(x) - \\nabla^2 f(y) \\|}{\\|x - y\\|^{\\nu}},\n\\qquad \\nu \\in [0, 1].\n\\ea\n\\end{equation}\nWe allow $\\Hfnu$ to be equal to $+\\infty$ for some $\\nu$.\n","start":14397,"end":14861}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":["ConditionNumberDefinition"],"declared_external":false,"source_only":false,"title":"二阶条件数","source_units":[{"raw_tex":"Hence, for a twice-differentiable convex function $f$ with\n$\\inf_{\\nu \\in [0, 1]} \\Hfnu > 0$, we can define only one\nmeaningful \\textit{condition number} of degree $\\nu \\in\n[0, 1]$:\n\\begin{equation} \\label{ConditionNumberDefinition}\n\\ba{rcl}\n\\gamma_f(\\nu) &\\; \\defeq \\;& \\frac{\\sigma_f(2 + \\nu)}{\\Hfnu}.\n\\ea\n\\end{equation}\nIf for some particular $\\nu$ we have $\\Hfnu = +\\infty$, then by our\ndefinition: $\\gamma_f(\\nu) = 0$.\n","start":20006,"end":20430}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"一般正则化模型","source_units":[{"raw_tex":"Let us start from the case when we know that for a specific\n$\\nu \\in [0, 1]$ function $f$ has H\\\"older continuous Hessian:\n$\\H_f(\\nu) < +\\infty$. Then, from~\\eqref{HolderHessianFuncBound}, we\nhave the global upper bound for the objective function:\n\\begin{displaymath}\n\\ba{rcl}\nF(y) & \\; \\leq \\; & M_{\\nu, H}(x; y) \\;\\defeq \\; Q(x; y)\n+ \\frac{H \\|x - y\\|^{2 + \\nu}}{(1 + \\nu)(2 + \\nu)} + h(y),\n\\qquad x, y \\in \\dom F,\n\\ea\n\\end{displaymath}\nwhere $H > 0$ is large enough: $H \\geq \\Hfnu$. Thus, it is\n","start":34020,"end":34518}]},{"id":"S017","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"模型步与最小值","source_units":[{"raw_tex":"natural to employ the minimum of a regularized quadratic\nmodel:\n\\begin{displaymath}\n\\ba{c}\nT_{\\nu, H}(x) \\; \\defeq \\;\n\\argmin\\limits_{y \\in \\dom F} M_{\\nu, H}(x; y), \\qquad M_{\\nu, H}^{*}(x)\n\\; \\defeq \\; \\min\\limits_{y \\in \\dom F} M_{\\nu, H}(x; y),\n\\ea\n\\end{displaymath}\n","start":34518,"end":34789}]},{"id":"S018","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"采用三次模型的动机","source_units":[{"raw_tex":"Unfortunately, in practice it is difficult to decide on an\nappropriate value of $\\nu \\in [0, 1]$ with $\\Hfnu <\n+\\infty$. Therefore, it is interesting to develop the\n\\textit{universal methods} which are not based on some\nparticular parameters. Recently, it was\nshown~\\cite{grapiglia2017regularized} that one good\nchoice for such universal scheme is the cubic\nregularization of the Newton Method\n\\cite{nesterov2006cubic}. This is actually the\n","start":37376,"end":37817}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"三次模型简记","source_units":[{"raw_tex":"process~\\eqref{GeneralRegularizedNewton} with the fixed\nparameter $\\nu = 1$. For this choice, in the rest part of\nthe paper we omit the corresponding index in the\ndefinitions of all necessary objects:\n$M_H(x; y) := M_{1, H}(x; y)$, \n$T_H(x) := T_{1, H}(x)$, and $M_H^{*}(x) := M_{1, H}^{*}(x) = M_H(x; T_H(x))$.\nThe adaptive scheme of our method with dynamic estimation\nof the constant $H$ is as follows.\n","start":37817,"end":38222}]},{"id":"S020","semantic_type":"algorithm","kind":"algorithm","labels":[],"declared_external":false,"source_only":false,"title":"自适应三次牛顿法","source_units":[{"raw_tex":"\\vspace*{5pt}\n\\noindent\n\\hspace*{5pt}\n\\lfbox{\n\t\\hspace*{4pt}\n\t\\centering\n\t\\begin{minipage}[b]{0.85\\linewidth} \\label{MainAlgorithm}\n\t\t\\begin{algorithm}[H] \n\t\t  \t\\caption{\\small \\textbf{Adaptive Cubic\n\t\t\t\t\tRegularization of Newton Method}}\n\t\t\t\\vspace*{6pt}\n\t\t\t\\noindent\\makebox[\\linewidth]{\\rule{1.097\\linewidth}{0.4pt}}\n\t\t\t\\begin{algorithmic}[1]\n\t\t\t\t%\\vspace{5pt}\n\t\t\t\t\\Require Choose $x_0 \\in \\dom F$, $H_0 > 0$.\n\t\t\t\t\\vspace*{2pt}\n\t\t\t\t\\Ensure $k \\geq 0$. \\vspace*{2pt}\n\t\t\t\t\\State Find the minimal integer $i_k \\geq 0 \\;$\n\t\t\t\tsuch that $F(T_{H_k 2^{i_k}}(x_k)) \\leq M^{*}_{H_k\n\t\t\t\t\t2^{i_k}} (x_k)$. %\\vspace*{1pt}\n\t\t\t\t\\State Perform the Cubic Step: $x_{k + 1}  = T_{H_k\n\t\t\t\t\t2^{i_k}}(x_k)$. \\vspace*{1pt}\n\t\t\t\t\\State Set $H_{k + 1} := 2^{i_k - 1} H_k $.\n\t\t\t\t\\vspace*{5pt}\n\t\t\t\\end{algorithmic}\n\t\t\\end{algorithm}\n\t\\end{minipage}\n\t\\hspace*{4pt}\n}\n\\vspace*{1pt}\n","start":38223,"end":39079}]},{"id":"S021","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"三次步长半径","source_units":[{"raw_tex":"Let us present the main properties of the composite Cubic\nNewton step $x \\mapsto T_H(x)$. Denote\n$$\nr_H(x) \\defeq \\|T_H(x) - x\\|.\n$$\n","start":39080,"end":39213}]},{"id":"S022","semantic_type":"derived_fact","kind":"derived_fact","labels":["CubicStepStationary"],"declared_external":false,"source_only":false,"title":"三次步最优性条件","source_units":[{"raw_tex":"Since point $T_H(x)$ is a minimum of strictly convex\nfunction $M_H(x;\\cdot)$, it satisfies the following\nfirst-order optimality condition:\n\\beq \\label{CubicStepStationary}\n\\ba{cl}\n&\\bigl \\la \\nabla f(x) + \\nabla^2 f(x)(T_H(x) - x) +\n\\tfrac{H r_H(x)}{2}B(T_H(x) - x), y - T_H(x) \\bigr \\ra  \\; + \\\\\n\\\\\n& \\quad h(y) \\; \\geq \\; h(T_H(x)),\n\\qquad y \\in \\dom F.\n\\ea\n\\eeq \n","start":39213,"end":39579}]},{"id":"S023","semantic_type":"derived_fact","kind":"derived_fact","labels":["CubicStepSubdiff"],"declared_external":false,"source_only":false,"title":"构造非光滑项次梯度","source_units":[{"raw_tex":"In other words, the vector\n$$\n\\ba{rcl}\nh'(T_H(x)) & \\defeq & -\\nabla f(x) - \\nabla^2 f(x)(T_H(x) -\nx) - \\frac{H r_H(x)}{2}B(T_H(x) - x)\n\\ea\n$$\nbelongs to the subdifferential of $h$:\n\\begin{equation} \\label{CubicStepSubdiff}\n\\ba{rcl}\nh'(T_H(x)) &\\; \\in \\; & \\partial h(T_H(x)).\n\\ea\n\\end{equation}\n","start":39579,"end":39875}]},{"id":"S024","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"子问题计算代价说明","source_units":[{"raw_tex":"Computation of a point $T = T_H(x)$, satisfying condition\n\\eqref{CubicStepSubdiff}, requires some standard techniques\nof Convex Optimization and Linear Algebra\n(see~\\cite{nesterov2006cubic,nesterov2019implementable,agarwal2017finding,carmon2016gradient}). Arithmetical\ncomplexity of such a procedure is usually similar to that\nof the standard Newton step.\n","start":39875,"end":40231}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":["StationaryConsequence"],"declared_external":false,"source_only":false,"title":"代入当前点","source_units":[{"raw_tex":"\nPlugging into~\\eqref{CubicStepStationary} $y := x \\in\n\\dom F$, we get:\n\\begin{align} \n&\\la \\nabla f(x), x - T_H(x) \\ra \\label{StationaryConsequence} \\\\\n&\\geq \\; \\la \\nabla^2 f(x) (T_H(x) - x), T_H(x) - x \\ra +\n\\tfrac{H r_H^3(x)}{2} + h(T_H(x)) - h(x). \\notag\n\\end{align}\n","start":40231,"end":40503}]},{"id":"S026","semantic_type":"derived_fact","kind":"derived_fact","labels":["CubicModelMinimumExpression"],"declared_external":false,"source_only":false,"title":"三次模型下降界","source_units":[{"raw_tex":"Thus, we obtain the following bound for the minimal value\n$M_H^{*}(x)$ of the cubic model:\n\\beq \\label{CubicModelMinimumExpression}\n\\ba{rcl}\nM_H^{*}(x) \n& \\overset{\\eqref{StationaryConsequence}}{\\leq} &\nf(x) -\n\\tfrac{1}{2}\\la \\nabla^2 f(x)(T_H(x) - x), T_H(x) - x \\ra -\n\\tfrac{H r_H^3(x)}{3} + h(x) \\\\[5pt]\n& = &\nF(x) - \\tfrac{1}{2}\\la \\nabla^2 f(x)(T_H(x) - x),\nT_H(x) - x \\ra - \\tfrac{Hr_H^3(x)}{3}. \n\\ea\n\\eeq\n\n\n","start":40503,"end":40917}]},{"id":"S027","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"新点目标次梯度","source_units":[{"raw_tex":"If for some value $\\nu \\in [0, 1]$ the Hessian is H\\\"older continuous:\n$\\Hfnu < +\\infty$, then by \\eqref{HolderHessianGradBound}\nand~\\eqref{CubicStepSubdiff} we get the following bound for\nthe subgradient:\n$$\nF'(T_H(x)) \\defeq \\nabla f(T_H(x)) + h'(T_H(x))\n$$\nat the new point:\n","start":40917,"end":41195}]},{"id":"S028","semantic_type":"derived_fact","kind":"derived_fact","labels":["CubicGradNewPoint"],"declared_external":false,"source_only":false,"title":"新点次梯度估计","source_units":[{"raw_tex":"\\beq \\label{CubicGradNewPoint}\n\\ba{cl}\n& \\|  F'(T_H(x)) \\|_{*} \\\\[5pt]\n& \\leq \\;\n  \\| \\nabla f(T_H(x)) - \\nabla f(x) - \\nabla^2 f(x)\n(T_H(x) - x)\\|_{*} + \\tfrac{H r^2_H(x)}{2} \\\\[5pt]\n& \\stackrel{\\eqref{HolderHessianGradBound}}{\\leq}\n\\tfrac{\\Hfnu\n\tr^{1 + \\nu}_H(x)}{1 + \\nu}  +  \\tfrac{H r^2_H(x)}{2} \n\\; = \\; \nr^{1 + \\nu}_H(x) \\cdot \\Bigl(  \\tfrac{\\Hfnu}{1 + \\nu} + \n\\tfrac{Hr_H^{1 - \\nu}(x)}{2} \\Bigr). \n\\ea\n\\eeq\n","start":41195,"end":41610}]},{"id":"S029","semantic_type":"definition","kind":"definition","labels":["KappaDefinition"],"declared_external":false,"source_only":false,"title":"参数控制常数","source_units":[{"raw_tex":"Denote by $\\kappa_f(\\nu)$ the following auxiliary value:\n\\begin{equation} \\label{KappaDefinition}\n\\ba{rcl}\n\\kappa_f(\\nu)\n& \\; \\defeq \\; & \\frac{  \\Hfnu^{\\frac{2}{1 + \\nu}}   }\n{ (\\sigmafnu)^{\\frac{1 - \\nu}{(1 + \\nu)(2 + \\nu)}} }\n\\cdot\n\\frac{6 \\cdot (8 + \\nu)^{\\frac{1 - \\nu}{1 + \\nu}} }\n{ \\left((1 + \\nu)(2 + \\nu)\\right)^{\\frac{2}{1 + \\nu}} }\n\\cdot\n\\bigl( \\frac{1 + \\nu}{2 + \\nu} \\bigr)^{\\frac{1 - \\nu}{2 + \\nu}}, \\;\n\\nu \\in [0, 1].\n\\ea\n\\end{equation}\n","start":51894,"end":52346}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","labels":["UniformlyConvexConsequence","UniformlyConvexGradientBound"],"declared_external":false,"source_only":false,"title":"次梯度控制函数残差","source_units":[{"raw_tex":"\\begin{lemma} \\label{UniformlyConvexConsequence}\n\tLet $f$ be uniformly convex of degree $p \\geq 2$ with\n\tconstant $\\sigma > 0$ on set $\\dom F$. Then, for every $x\n\t\\in \\dom F$ and for all $F'(x) \\in \\partial F(x)$ we have\n\t\\begin{equation} \\label{UniformlyConvexGradientBound}\n\t\\ba{rcl}\n\tF(x) - F^{*} & \\; \\leq \\; & \\frac{p - 1}{p} \\left( \\frac{1}{\\sigma}\n\t\\right)^{\\frac{1}{p - 1}} \\|F'(x)\\|_{*}^{\\frac{p}{p - 1}}.\n\t\\ea\n\t\\end{equation}\n\\end{lemma}","start":11630,"end":12078}]},{"id":"S031","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"残差界的作者引用证明","source_units":[{"raw_tex":"\\begin{proof}\n\t\\noindent\n\tIn view of~\\eqref{UniformlyConvexLowerBound}, \n\tbound \\eqref{UniformlyConvexGradientBound}\n\tfollows as in the proof of Lemma~3 \n\tin~\\cite{nesterov2008accelerating}.\n\\qed\n\\end{proof}","start":12079,"end":12286}]},{"id":"S032","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原前提清单：Nesterov (2008), Lemma 3 proof argument invoked by B0012 for the residual bound; not reproduced in full。原文使用位置为第 327–330 行。"],"source_only":false,"title":"外部残差界证明论证","source_units":[]},{"id":"S033","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"误收录的例证完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\tDenote $\\kappa(x) \\equiv \\sum_{i = 1}^m e^{\\langle a_i, x \\rangle}$.\n\tLet us fix arbitrary $x, y \\in \\E$ and direction $h \\in \\E$. Then,\n\tstraightforward computation gives:\n\t$$\n\t\\ba{rcl}\n\t\\langle \\nabla f(x), h \\rangle &\\; = \\;& \\frac{1}{\\kappa(x)}\n\t\\sum_{i = 1}^m e^{\\langle a_i, x \\rangle} \\langle a_i, h \\rangle, \\\\[6pt]\n\t\\langle \\nabla^2 f(x)h, h \\rangle &\\; =\\;&\n\t \\frac{1}{\\kappa(x)} \\sum_{i = 1}^m e^{\\langle a_i, x \\rangle}\n\t\\langle\n\ta_i, h \\rangle^2 - \\bigl( \\frac{1}{\\kappa(x)} \\sum_{i = 1}^m\n\te^{\\langle a_i, x \\rangle} \\langle a_i, h \\rangle  \\bigr)^2 \\\\[5pt]\n\t&\\; = \\;& \\frac{1}{\\kappa(x)} \\sum_{i = 1}^m e^{\\langle a_i, x \\rangle}\n\t\\left( \\langle\n\ta_i, h \\rangle - \\langle \\nabla f(x), h \\rangle\n\t\\right)^2 \\; \\geq \\; 0.\n\t\\ea\n\t$$\n\tHence, we get\n\t$$\n\t\\ba{rcl}\n\t\\| \\nabla^2 f(x) \\| & = & \\max\\limits_{\\|h\\| \\leq 1} \\langle \\nabla^2 f(x)\n\th, h \\rangle \\;\n\t\\leq \\;  \\max\\limits_{\\|h\\| \\leq 1} \\sum_{i = 1}^m \\langle a_i, h \\rangle^2\n\t\\;\n\t= \\; \\max\\limits_{\\|h\\| \\leq 1} \\|h\\|^2 \\; = \\; 1.\n\t\\ea\n\t$$\n\tSince all Hessians of function $f$ are positive definite,\n\twe conclude that $\\Hf(0) \\leq 1$. Inequality $\\Hf(1) \\leq\n\t2$ can be easily obtained from the following\n\trepresentation of the third derivative:\n\t$$\n\t\\ba{rcl}\n\tf'''(x)[h,h,h]  & \\; = \\; & {1 \\over \\kappa(x)}\n\t\\sum\\limits_{i=1}^m e^{\\la a_i, x \\ra} \\left( \\la a_i, h\n\t\\ra - \\la \\nabla f(x), h \\ra \\right)^3\\\\\n\t\\\\\n\t& \\;  \\leq \\; & \\la \\nabla^2 f(x) h, h \\ra \\max\\limits_{1 \\leq\n\t\ti,j \\leq m } \\la a_i - a_j , h \\ra \\; \\leq \\; 2 \\| h \\|^3.\n\t\\ea\n\t$$\n\\qed\n\\end{proof}","start":16648,"end":18193}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"例证导数计算","source_units":[{"raw_tex":"\\begin{proof}\n\tDenote $\\kappa(x) \\equiv \\sum_{i = 1}^m e^{\\langle a_i, x \\rangle}$.\n\tLet us fix arbitrary $x, y \\in \\E$ and direction $h \\in \\E$. Then,\n\tstraightforward computation gives:\n\t$$\n\t\\ba{rcl}\n\t\\langle \\nabla f(x), h \\rangle &\\; = \\;& \\frac{1}{\\kappa(x)}\n\t\\sum_{i = 1}^m e^{\\langle a_i, x \\rangle} \\langle a_i, h \\rangle, \\\\[6pt]\n\t\\langle \\nabla^2 f(x)h, h \\rangle &\\; =\\;&\n\t \\frac{1}{\\kappa(x)} \\sum_{i = 1}^m e^{\\langle a_i, x \\rangle}\n\t\\langle\n\ta_i, h \\rangle^2 - \\bigl( \\frac{1}{\\kappa(x)} \\sum_{i = 1}^m\n\te^{\\langle a_i, x \\rangle} \\langle a_i, h \\rangle  \\bigr)^2 \\\\[5pt]\n\t&\\; = \\;& \\frac{1}{\\kappa(x)} \\sum_{i = 1}^m e^{\\langle a_i, x \\rangle}\n\t\\left( \\langle\n\ta_i, h \\rangle - \\langle \\nabla f(x), h \\rangle\n\t\\right)^2 \\; \\geq \\; 0.\n\t\\ea\n\t$$\n","start":16648,"end":17407}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"例证零次光滑界","source_units":[{"raw_tex":"\tHence, we get\n\t$$\n\t\\ba{rcl}\n\t\\| \\nabla^2 f(x) \\| & = & \\max\\limits_{\\|h\\| \\leq 1} \\langle \\nabla^2 f(x)\n\th, h \\rangle \\;\n\t\\leq \\;  \\max\\limits_{\\|h\\| \\leq 1} \\sum_{i = 1}^m \\langle a_i, h \\rangle^2\n\t\\;\n\t= \\; \\max\\limits_{\\|h\\| \\leq 1} \\|h\\|^2 \\; = \\; 1.\n\t\\ea\n\t$$\n\tSince all Hessians of function $f$ are positive definite,\n\twe conclude that $\\Hf(0) \\leq 1$. Inequality $\\Hf(1) \\leq\n\t2$ can be easily obtained from the following\n","start":17407,"end":17835}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"例证一阶光滑界","source_units":[{"raw_tex":"\trepresentation of the third derivative:\n\t$$\n\t\\ba{rcl}\n\tf'''(x)[h,h,h]  & \\; = \\; & {1 \\over \\kappa(x)}\n\t\\sum\\limits_{i=1}^m e^{\\la a_i, x \\ra} \\left( \\la a_i, h\n\t\\ra - \\la \\nabla f(x), h \\ra \\right)^3\\\\\n\t\\\\\n\t& \\;  \\leq \\; & \\la \\nabla^2 f(x) h, h \\ra \\max\\limits_{1 \\leq\n\t\ti,j \\leq m } \\la a_i - a_j , h \\ra \\; \\leq \\; 2 \\| h \\|^3.\n\t\\ea\n\t$$\n\\qed\n\\end{proof}","start":17835,"end":18193}]},{"id":"S037","semantic_type":"lemma","kind":"lemma","labels":["LemmaStepHolder","OneStepProgressHolder"],"declared_external":false,"source_only":false,"title":"三次模型的双分支进展","source_units":[{"raw_tex":"\\begin{lemma} \\label{LemmaStepHolder}\n\tLet $0 < \\Hfnu < +\\infty$ and $\\sigmafnu > 0$ for some\n\t$\\nu \\in [0,1]$. Then, for arbitrary $x \\in \\dom F$ and $H\n\t> 0$ we have:\n\t\\beq \\label{OneStepProgressHolder}\n\t\\ba{cl}\n\t& F(x) - M_H^{*}(x)  \\\\[5pt] \n\t& \\quad \\geq \\; \\min\\Bigl[ \\left( F(x) -\n\tF^{*} \\right) \\cdot \\frac{(1 + \\nu)}{(2 + \\nu)} \\cdot\n\t\\min\\bigl\\{\\bigl( \\frac{(1 + \\nu) \\gammafnu}{2(2 + \\nu)}  \\bigr)^{\\frac{1}{1 + \\nu}}, \n\t\\; 1\\bigr\\} ,   \\\\[5pt]\n\t& \\; \\qquad \\qquad \\left( F(T_H(x)) - F^{*} \\right)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}}\n\t\\cdot \\bigl ( \\frac{2 + \\nu}{1 + \\nu}  \\bigr)^{\\frac{3(1 + \\nu)}{2(2\n\t\t\t+ \\nu)}} \\cdot \\frac{(\\sigmafnu)^{\\frac{3}{2(2 + \\nu)}}}{ 3 \\sqrt{H}\n\t}  \\;  \\Bigr]. \n\t\\ea\n\t\\eeq\n\\end{lemma}","start":48378,"end":49099}]},{"id":"S038","semantic_type":"proof","kind":"proof","labels":["NextPointGradBound2"],"declared_external":false,"source_only":false,"title":"双分支进展的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\tLet us consider two cases. 1) \n\t$H r_H^{1 - \\nu}(x) \\leq \\frac{2 \\Hfnu}{1 + \\nu}$.\n\t\tThen, for arbitrary $y \\in \\dom F$, we have:\n\t\t$$\n\t\t\\ba{rcl}\n\t\tM_H^{*}(x) \n\t\t& := &  Q(x; T_H(x)) +\n\t\t\\frac{H}{6}\\|T_H(x) - x\\|^3 + h(T_H(x)) \\\\[7pt]\n\t\t& \\leq &  Q(x; y) + \\frac{H r_H^{1 - \\nu}(x) \\|y -\n\t\t\tx\\|^{2 + \\nu}}{2 (2 + \\nu) } + h(y) \\\\[5pt]\n\t\t& \\overset{\\eqref{HolderHessianFuncBound}}{\\leq} &  \n\t\tF(y) +\n\t\t\\frac{\\Hfnu \\|y - x\\|^{2 + \\nu}}{(1 + \\nu)(2 + \\nu)} +\n\t\t\\frac{H r_H^{1 - \\nu}(x) \\|y - x\\|^{2 + \\nu}}{2 (2 + \\nu) }\n\t\t\\\\[7pt]\n\t\t& \\leq &\n\t\t F(y) +  \\frac{2\\Hfnu \\|y - x\\|^{2 +\n\t\t\\nu}}{(1 + \\nu)(2 + \\nu)},\n\t\t\\ea\n\t\t$$\n\t\twhere the first inequality follows from the fact, that\n\t\t$$\n\t\t\\ba{rcl}\n\t\tT_H(x) & = & \\argmin\\limits_{y \\in \\dom F} \\bigl\\{ Q(x; y)\n\t\t+ \\frac{H r_H^{1 - \\nu}(x) \\|y -\n\t\t\tx\\|^{2 + \\nu}}{2 (2 + \\nu) } + h(y) \\bigr\\}.\n\t\t\\ea\n\t\t$$\n\t\tLet us restrict $y$ to the segment: $y = \\alpha x^{*} + (1 -\n\t\t\\alpha) x,$ with $\\alpha \\in [0, 1]$. Taking into account the\n\t\tuniform convexity, we get:\n\t\t$$\n\t\t\\ba{rcl}\n\t\tM_H^{*}(x) & \\leq &  F(x) - \\alpha \\left( F(x) -\n\t\tF^{*} \\right) + \\alpha^{2 + \\nu} \\frac{2\\Hfnu \\|x^{*} - x\\|^{2 +\n\t\t\t\t\\nu}} {(1 + \\nu)(2 + \\nu)} \\quad \\\\[5pt]\n\t\t& \\overset{\\eqref{UniformlyConvexLowerBound}}{\\leq} &  \n\t\tF(x)\n\t\t-  \\Bigl(\\alpha  - \\alpha^{2 + \\nu} \\frac{2\\Hfnu}{  (1 + \\nu)\n\t\t\t\\sigmafnu}\n\t\t\\Bigr) \\left( F(x) - F^{*} \\right).\n\t\t\\ea\n\t\t$$\n\t\tThe minimum of the right-hand side is attained at\n\t\t$\n\t\t\\alpha^{*} = \\min\\bigl\\{\\frac{(1 + \\nu)\\gammafnu}{2(2 + \\nu)}, 1 \\bigr\\}^{\\frac{1}{1 + \\nu}}.\n\t\t$\n\t\tPlugging this value into the bound, we have:\n\t\t\\begin{displaymath}\n\t\t\\ba{rcl}\n\t\tM^{*}_H(x) & \\leq & F(x)  -   \\min\\bigl\\{ \\bigl(\n\t\t\\frac{(1 + \\nu)\\gammafnu}{2(2 + \\nu)}   \\bigr)^{1 / (1 + \\nu)}\n\t\t, \\; 1 \\bigr\\} \\cdot \\frac{(1 + \\nu)}{(2 + \\nu)} \\cdot \\left(\n\t\tF(x) - F^{*} \\right),\n\t\t\\ea\n\t\t\\end{displaymath}\n\t\tand this is the first argument of the minimum in\n\t\t\\eqref{OneStepProgressHolder}.\n\t\t\n\t\t2) $H r_H^{1 - \\nu}(x) > \\frac{2 \\Hfnu}{1 + \\nu}.$\n\t\tBy~\\eqref{CubicGradNewPoint}, we have the bound:\n\t\t\\begin{equation} \\label{NextPointGradBound2}\n\t\t\\ba{rcl}\n\t\t\\| F'(T_H(x)) \\|_{*} & \\; < \\; & Hr_H^2(x).\n\t\t\\ea\n\t\t\\end{equation}\n\t\tUsing the fact that $\\nabla^2 f(x) \\succeq 0$, we get the second argument of the\n\t\tminimum:\n\t$$\n\t\\ba{rcl}\n\tF(x) - M_H^{*}(x)\n\t& \\overset{\\eqref{CubicModelMinimumExpression}}{\\geq} &\n\t\\frac{Hr^3_H(x)}{3} \\quad  \\overset{ \\eqref{NextPointGradBound2}\n\t}{\\geq} \\quad \\frac{\\| F'(T_H(x)) \\|_{*}^{\\frac32}}{3 \\sqrt{H}}\n\t\\\\[5pt]\n\t& \\overset{\\eqref{UniformlyConvexGradientBound}}{\\geq} &\n\t \\left(\n\t\\frac{2 + \\nu}{1 + \\nu}  \\right)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}}\n\t\\cdot \\frac{(\\sigmafnu)^{\\frac{3}{2(2 + \\nu)}}}{ 3 \\sqrt{H} } \\cdot\n\t\\left( F(T_H(x)) - F^{*} \\right)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}}.\n\t\\ea\n\t$$\n\\qed\n\\end{proof}","start":49100,"end":51890}]},{"id":"S039","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"小参数分支条件","source_units":[{"raw_tex":"\\begin{proof}\n\tLet us consider two cases. 1) \n\t$H r_H^{1 - \\nu}(x) \\leq \\frac{2 \\Hfnu}{1 + \\nu}$.\n\t\tThen, for arbitrary $y \\in \\dom F$, we have:\n","start":49100,"end":49245}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"同一点最小化重加权模型","source_units":[{"raw_tex":"\t\twhere the first inequality follows from the fact, that\n\t\t$$\n\t\t\\ba{rcl}\n\t\tT_H(x) & = & \\argmin\\limits_{y \\in \\dom F} \\bigl\\{ Q(x; y)\n\t\t+ \\frac{H r_H^{1 - \\nu}(x) \\|y -\n\t\t\tx\\|^{2 + \\nu}}{2 (2 + \\nu) } + h(y) \\bigr\\}.\n\t\t\\ea\n\t\t$$\n","start":49733,"end":49961}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"小参数模型比较","source_units":[{"raw_tex":"\t\t$$\n\t\t\\ba{rcl}\n\t\tM_H^{*}(x) \n\t\t& := &  Q(x; T_H(x)) +\n\t\t\\frac{H}{6}\\|T_H(x) - x\\|^3 + h(T_H(x)) \\\\[7pt]\n\t\t& \\leq &  Q(x; y) + \\frac{H r_H^{1 - \\nu}(x) \\|y -\n\t\t\tx\\|^{2 + \\nu}}{2 (2 + \\nu) } + h(y) \\\\[5pt]\n\t\t& \\overset{\\eqref{HolderHessianFuncBound}}{\\leq} &  \n\t\tF(y) +\n\t\t\\frac{\\Hfnu \\|y - x\\|^{2 + \\nu}}{(1 + \\nu)(2 + \\nu)} +\n\t\t\\frac{H r_H^{1 - \\nu}(x) \\|y - x\\|^{2 + \\nu}}{2 (2 + \\nu) }\n\t\t\\\\[7pt]\n\t\t& \\leq &\n\t\t F(y) +  \\frac{2\\Hfnu \\|y - x\\|^{2 +\n\t\t\\nu}}{(1 + \\nu)(2 + \\nu)},\n\t\t\\ea\n\t\t$$\n","start":49245,"end":49733}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"沿最优点线段比较","source_units":[{"raw_tex":"\t\tLet us restrict $y$ to the segment: $y = \\alpha x^{*} + (1 -\n\t\t\\alpha) x,$ with $\\alpha \\in [0, 1]$. Taking into account the\n\t\tuniform convexity, we get:\n\t\t$$\n\t\t\\ba{rcl}\n\t\tM_H^{*}(x) & \\leq &  F(x) - \\alpha \\left( F(x) -\n\t\tF^{*} \\right) + \\alpha^{2 + \\nu} \\frac{2\\Hfnu \\|x^{*} - x\\|^{2 +\n\t\t\t\t\\nu}} {(1 + \\nu)(2 + \\nu)} \\quad \\\\[5pt]\n\t\t& \\overset{\\eqref{UniformlyConvexLowerBound}}{\\leq} &  \n\t\tF(x)\n\t\t-  \\Bigl(\\alpha  - \\alpha^{2 + \\nu} \\frac{2\\Hfnu}{  (1 + \\nu)\n\t\t\t\\sigmafnu}\n\t\t\\Bigr) \\left( F(x) - F^{*} \\right).\n\t\t\\ea\n\t\t$$\n","start":49961,"end":50488}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"优化线段参数","source_units":[{"raw_tex":"\t\tThe minimum of the right-hand side is attained at\n\t\t$\n\t\t\\alpha^{*} = \\min\\bigl\\{\\frac{(1 + \\nu)\\gammafnu}{2(2 + \\nu)}, 1 \\bigr\\}^{\\frac{1}{1 + \\nu}}.\n\t\t$\n\t\tPlugging this value into the bound, we have:\n\t\t\\begin{displaymath}\n\t\t\\ba{rcl}\n\t\tM^{*}_H(x) & \\leq & F(x)  -   \\min\\bigl\\{ \\bigl(\n\t\t\\frac{(1 + \\nu)\\gammafnu}{2(2 + \\nu)}   \\bigr)^{1 / (1 + \\nu)}\n\t\t, \\; 1 \\bigr\\} \\cdot \\frac{(1 + \\nu)}{(2 + \\nu)} \\cdot \\left(\n\t\tF(x) - F^{*} \\right),\n\t\t\\ea\n\t\t\\end{displaymath}\n\t\tand this is the first argument of the minimum in\n\t\t\\eqref{OneStepProgressHolder}.\n\t\t\n","start":50488,"end":51041}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":["NextPointGradBound2"],"declared_external":false,"source_only":false,"title":"大参数分支次梯度界","source_units":[{"raw_tex":"\t\t2) $H r_H^{1 - \\nu}(x) > \\frac{2 \\Hfnu}{1 + \\nu}.$\n\t\tBy~\\eqref{CubicGradNewPoint}, we have the bound:\n\t\t\\begin{equation} \\label{NextPointGradBound2}\n\t\t\\ba{rcl}\n\t\t\\| F'(T_H(x)) \\|_{*} & \\; < \\; & Hr_H^2(x).\n\t\t\\ea\n\t\t\\end{equation}\n","start":51041,"end":51272}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"大参数分支残差进展","source_units":[{"raw_tex":"\t\tUsing the fact that $\\nabla^2 f(x) \\succeq 0$, we get the second argument of the\n\t\tminimum:\n\t$$\n\t\\ba{rcl}\n\tF(x) - M_H^{*}(x)\n\t& \\overset{\\eqref{CubicModelMinimumExpression}}{\\geq} &\n\t\\frac{Hr^3_H(x)}{3} \\quad  \\overset{ \\eqref{NextPointGradBound2}\n\t}{\\geq} \\quad \\frac{\\| F'(T_H(x)) \\|_{*}^{\\frac32}}{3 \\sqrt{H}}\n\t\\\\[5pt]\n\t& \\overset{\\eqref{UniformlyConvexGradientBound}}{\\geq} &\n\t \\left(\n\t\\frac{2 + \\nu}{1 + \\nu}  \\right)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}}\n\t\\cdot \\frac{(\\sigmafnu)^{\\frac{3}{2(2 + \\nu)}}}{ 3 \\sqrt{H} } \\cdot\n\t\\left( F(T_H(x)) - F^{*} \\right)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}}.\n\t\\ea\n\t$$\n\\qed\n\\end{proof}","start":51272,"end":51890}]},{"id":"S046","semantic_type":"lemma","kind":"lemma","labels":["LemmaUpHolder","LemmaUpHolder3","LemmaUpHolderCondition"],"declared_external":false,"source_only":false,"title":"失败试探后的参数界","source_units":[{"raw_tex":"\\begin{lemma} \\label{LemmaUpHolder} Assume that for a fixed\n\t$x \\in \\dom F$ the parameter $H > 0$ is such that:\n\t\\begin{equation} \\label{LemmaUpHolderCondition}\n\t\\ba{rcl}\n\tF(T_H(x)) & \\; > \\; & M^{*}_H(x).\n\t\\ea\n\t\\end{equation}\n\tIf for some $\\nu \\in [0, 1]$, we have $\\sigmafnu > 0$, then it holds:\n\t\\begin{equation} \\label{LemmaUpHolder3}\n\t\\ba{rcl}\n\tH \\left( F(T_{2H}(x)) - F^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}}\n\t& \\; < \\; & \\kappa_f(\\nu).\n\t\\ea\n\t\\end{equation}\n\\end{lemma}","start":52438,"end":52912}]},{"id":"S047","semantic_type":"assumption","kind":"assumption","labels":["LemmaUpHolder","LemmaUpHolderCondition"],"declared_external":false,"source_only":false,"title":"参数引理的失败条件","source_units":[{"raw_tex":"\\begin{lemma} \\label{LemmaUpHolder} Assume that for a fixed\n\t$x \\in \\dom F$ the parameter $H > 0$ is such that:\n\t\\begin{equation} \\label{LemmaUpHolderCondition}\n\t\\ba{rcl}\n\tF(T_H(x)) & \\; > \\; & M^{*}_H(x).\n\t\\ea\n\t\\end{equation}\n\tIf for some $\\nu \\in [0, 1]$, we have $\\sigmafnu > 0$, then it holds:\n","start":52438,"end":52736}]},{"id":"S048","semantic_type":"proof","kind":"proof","labels":["LemmaUpHolderBound1","LemmaUpHolderBound2"],"declared_external":false,"source_only":false,"title":"失败试探参数界完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\tFirstly, let us prove that from~\\eqref{LemmaUpHolderCondition} we\n\thave:\n\t\\begin{equation} \\label{LemmaUpHolderBound1}\n\t\\ba{rcl}\n\tH r^{1 - \\nu}_H(x) & \\; < \\; & \\frac{6 \\Hfnu}{(1 + \\nu)(2 + \\nu)}.\n\t\\ea\n\t\\end{equation}\n\tAssuming by contradiction, $H r^{1 - \\nu}_H(x) \\; \\geq \\; \\frac{6\n\t\t\\Hfnu}{(1 + \\nu)(2 + \\nu)}$, we get:\n\t$$\n\t\\ba{rcl}\n\tM_H^{*}(x)  & := & \\frac{H \\|T_H(x) - x\\|^3}{6} + Q(x;\n\tT_H(x)) + h(T_H(x)) \\\\[5pt]\n\t& \\geq & \\frac{\\Hfnu \\|T_H(x) - x\\|^{2 + \\nu}}{(1 + \\nu)(2 +\n\t\t\\nu)} + Q(x; T_H(x)) + h(T_H(x)) \\\\[5pt]\n\t& \\overset{\\eqref{HolderHessianFuncBound}}{\\geq} & F(T_H(x)),\n\t\\ea\n\t$$\n\twhich contradicts~\\eqref{LemmaUpHolderCondition}.\n\tSecondly, by its definition, $M^*_H(x)$ is a concave function of $H$.\n\tTherefore, its derivative\n\t$\n\t{d \\over d H} M^*_H(x)  =  {1 \\over 6} r_H^3(x)\n\t$\n\tis non-increasing. Hence, it holds:\n\t\\beq \\label{LemmaUpHolderBound2}\n\t\\ba{rcl}\n\tr_{2H}(x) & \\leq & r_H(x)\n\t\\overset{\\eqref{LemmaUpHolderBound1}}{<} \n\t\\bigl( \\frac{6 \\Hfnu}{(1 + \\nu)(2 + \\nu) H} \\bigr)^{\\frac{1}{1 - \\nu}}.\n\t\\ea\n\t\\eeq\n\tFinally, by the smoothness and the uniform convexity, we obtain:\n\t$$\n\t\\ba{cl}\n\t& H \\left( F(T_{2H}(x)) - F^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}}\n\t\\;\\; \\refLE{UniformlyConvexGradientBound} \\;\\;\n\tH \\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\| F'(T_{2H}(x)) \\|_{*}^{\\frac{1 - \\nu}{1 + \\nu}}  \\\\[5pt]\n\t& \\quad \\refLE{CubicGradNewPoint} \n\tH \\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\Bigl(\n\tr^{1 +\\nu}_{2H}(x) \\cdot \\left(  \\frac{\\Hfnu}{1 + \\nu} + H r^{1 -\n\t\t\\nu}_{2H}(x) \\right)\n\t\\Bigr)^{\\frac{1 - \\nu}{1 + \\nu}} \\\\[5pt]\n\t& \\quad \\overset{\\eqref{LemmaUpHolderBound2}}{<} \n\tH \\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\Bigl( \n\tr^{1+\\nu}_{2H}(x) \\cdot \\frac{(8 + \\nu) \\Hfnu}{(1 + \\nu)(2 + \\nu)}\n\t\\Bigr)^{\\frac{1 - \\nu}{1 + \\nu}} \\\\[5pt]\n\t& \\quad \\overset{\\eqref{LemmaUpHolderBound2}}{<} \n\t\t\\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\left( \\frac{\\Hfnu}{(1 + \\nu)(2 + \\nu)}  \\right)^{\\frac{2}{1 + \\nu}}\n\t6  (8 + \\nu)^{\\frac{1 - \\nu}{1 + \\nu}}\n\t\\; =: \\; \\kappa_f(\\nu).\n\t\\ea\n\t$$\n\\qed\n\\end{proof}","start":52913,"end":55281}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["LemmaUpHolderBound1"],"declared_external":false,"source_only":false,"title":"失败迫使正则项偏小","source_units":[{"raw_tex":"\\begin{proof}\n\tFirstly, let us prove that from~\\eqref{LemmaUpHolderCondition} we\n\thave:\n\t\\begin{equation} \\label{LemmaUpHolderBound1}\n\t\\ba{rcl}\n\tH r^{1 - \\nu}_H(x) & \\; < \\; & \\frac{6 \\Hfnu}{(1 + \\nu)(2 + \\nu)}.\n\t\\ea\n\t\\end{equation}\n\tAssuming by contradiction, $H r^{1 - \\nu}_H(x) \\; \\geq \\; \\frac{6\n\t\t\\Hfnu}{(1 + \\nu)(2 + \\nu)}$, we get:\n\t$$\n\t\\ba{rcl}\n\tM_H^{*}(x)  & := & \\frac{H \\|T_H(x) - x\\|^3}{6} + Q(x;\n\tT_H(x)) + h(T_H(x)) \\\\[5pt]\n\t& \\geq & \\frac{\\Hfnu \\|T_H(x) - x\\|^{2 + \\nu}}{(1 + \\nu)(2 +\n\t\t\\nu)} + Q(x; T_H(x)) + h(T_H(x)) \\\\[5pt]\n\t& \\overset{\\eqref{HolderHessianFuncBound}}{\\geq} & F(T_H(x)),\n\t\\ea\n\t$$\n\twhich contradicts~\\eqref{LemmaUpHolderCondition}.\n","start":52913,"end":53579}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"参数最小值的凹性","source_units":[{"raw_tex":"\tSecondly, by its definition, $M^*_H(x)$ is a concave function of $H$.\n\tTherefore, its derivative\n\t$\n\t{d \\over d H} M^*_H(x)  =  {1 \\over 6} r_H^3(x)\n\t$\n\tis non-increasing. Hence, it holds:\n","start":53579,"end":53769}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":["LemmaUpHolderBound2"],"declared_external":false,"source_only":false,"title":"加倍参数后的半径界","source_units":[{"raw_tex":"\t\\beq \\label{LemmaUpHolderBound2}\n\t\\ba{rcl}\n\tr_{2H}(x) & \\leq & r_H(x)\n\t\\overset{\\eqref{LemmaUpHolderBound1}}{<} \n\t\\bigl( \\frac{6 \\Hfnu}{(1 + \\nu)(2 + \\nu) H} \\bigr)^{\\frac{1}{1 - \\nu}}.\n\t\\ea\n\t\\eeq\n","start":53769,"end":53967}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"串联残差界计算常数","source_units":[{"raw_tex":"\tFinally, by the smoothness and the uniform convexity, we obtain:\n\t$$\n\t\\ba{cl}\n\t& H \\left( F(T_{2H}(x)) - F^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}}\n\t\\;\\; \\refLE{UniformlyConvexGradientBound} \\;\\;\n\tH \\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\| F'(T_{2H}(x)) \\|_{*}^{\\frac{1 - \\nu}{1 + \\nu}}  \\\\[5pt]\n\t& \\quad \\refLE{CubicGradNewPoint} \n\tH \\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\Bigl(\n\tr^{1 +\\nu}_{2H}(x) \\cdot \\left(  \\frac{\\Hfnu}{1 + \\nu} + H r^{1 -\n\t\t\\nu}_{2H}(x) \\right)\n\t\\Bigr)^{\\frac{1 - \\nu}{1 + \\nu}} \\\\[5pt]\n\t& \\quad \\overset{\\eqref{LemmaUpHolderBound2}}{<} \n\tH \\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\Bigl( \n\tr^{1+\\nu}_{2H}(x) \\cdot \\frac{(8 + \\nu) \\Hfnu}{(1 + \\nu)(2 + \\nu)}\n\t\\Bigr)^{\\frac{1 - \\nu}{1 + \\nu}} \\\\[5pt]\n\t& \\quad \\overset{\\eqref{LemmaUpHolderBound2}}{<} \n\t\t\\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\left( \\frac{\\Hfnu}{(1 + \\nu)(2 + \\nu)}  \\right)^{\\frac{2}{1 + \\nu}}\n\t6  (8 + \\nu)^{\\frac{1 - \\nu}{1 + \\nu}}\n\t\\; =: \\; \\kappa_f(\\nu).\n\t\\ea\n\t$$\n\\qed\n\\end{proof}","start":53967,"end":55281}]},{"id":"S053","semantic_type":"assumption","kind":"assumption","labels":["TheoremLinearRateUniformly"],"declared_external":false,"source_only":false,"title":"主定理非退化条件","source_units":[{"raw_tex":"\\begin{theorem} \\label{TheoremLinearRateUniformly} Assume that for a\n\tfixed $\\nu \\in [0, 1]$ we have $0 < \\Hfnu < +\\infty$ and\n\t$\\sigmafnu > 0$. Let parameter $H_0$ in Algorithm 1 be\n\tsmall enough:\n","start":55337,"end":55535}]},{"id":"S054","semantic_type":"assumption","kind":"assumption","labels":["HolderChoosingH","TheoremLinearRateUniformly"],"declared_external":false,"source_only":false,"title":"初始正则参数限制","source_units":[{"raw_tex":"\t\\begin{equation} \\label{HolderChoosingH}\n\t\\ba{rcl}\n\tH_0 & \\; \\leq \\; & \\frac{ \\kappa_f(\\nu) }{ \\left( F(x_0) - F^{*}\n\t\t\\right)^{(1 - \\nu) / (2 + \\nu)  } },\n\t\\ea\n\t\\end{equation}\n\twhere $\\kappa_f(\\nu)$ is defined\n\tby~\\eqref{KappaDefinition}. Let the sequence $\\{ x_k \\}_{k\n\t\t= 0}^K$ generated by the method satisfy condition:\n","start":55535,"end":55860}]},{"id":"S055","semantic_type":"assumption","kind":"assumption","labels":["HolderConditionFuncEps","TheoremLinearRateUniformly"],"declared_external":false,"source_only":false,"title":"所有试探点尚未达标","source_units":[{"raw_tex":"\t\\begin{equation} \\label{HolderConditionFuncEps}\n\tF(T_{H_k2^{j}}(x_k)) - F^{*} \\; \\geq \\; \\varepsilon \\; > \\; 0, \\qquad\n\t\\quad 0 \\leq j \\leq i_k, \\quad 0 \\leq k \\leq K - 1.\n\t\\end{equation}\n","start":55860,"end":56049}]},{"id":"S056","semantic_type":"theorem","kind":"theorem","labels":["HolderChoosingH","HolderConditionFuncEps","HolderMainResult","HolderOracleCallsBound","TheoremLinearRateUniformly"],"declared_external":false,"source_only":false,"title":"自适应三次牛顿全局线性界","source_units":[{"raw_tex":"\\begin{theorem} \\label{TheoremLinearRateUniformly} Assume that for a\n\tfixed $\\nu \\in [0, 1]$ we have $0 < \\Hfnu < +\\infty$ and\n\t$\\sigmafnu > 0$. Let parameter $H_0$ in Algorithm 1 be\n\tsmall enough:\n\t\\begin{equation} \\label{HolderChoosingH}\n\t\\ba{rcl}\n\tH_0 & \\; \\leq \\; & \\frac{ \\kappa_f(\\nu) }{ \\left( F(x_0) - F^{*}\n\t\t\\right)^{(1 - \\nu) / (2 + \\nu)  } },\n\t\\ea\n\t\\end{equation}\n\twhere $\\kappa_f(\\nu)$ is defined\n\tby~\\eqref{KappaDefinition}. Let the sequence $\\{ x_k \\}_{k\n\t\t= 0}^K$ generated by the method satisfy condition:\n\t\\begin{equation} \\label{HolderConditionFuncEps}\n\tF(T_{H_k2^{j}}(x_k)) - F^{*} \\; \\geq \\; \\varepsilon \\; > \\; 0, \\qquad\n\t\\quad 0 \\leq j \\leq i_k, \\quad 0 \\leq k \\leq K - 1.\n\t\\end{equation}\n\tThen, for every $0 \\leq k \\leq K - 1$, we have:\n\t\\beq \\label{HolderMainResult}\n\t\\ba{cl}\n\t& F(x_{k + 1}) - F^{*}  \\leq \\\\[5pt]\n\t& \\bigl( 1 - \\min\\bigl\\{  \\frac{ (2 + \\nu)\\left( (1 + \\nu) (2\n\t\t+ \\nu) \\right)^{ 1 / (1 + \\nu) }  \n\t    \\left( \\gammafnu  \\right)^{\\frac{1}{1 +\n\t    \t\t\\nu}}   }{ (1 + \\nu)6^{3/2} \\cdot 2^{1/2} \\cdot (8\n\t\t+ \\nu)^{ (1 - \\nu) / (2 + 2 \\nu) } } , \\frac{1}{2} \\bigr\\}  \\bigr) \\cdot \n\t  \\left( F(x_{k}) - F^{*} \\right).\n\t\\ea\n\t\\eeq\n\tTherefore, the rate of convergence is linear, and\n\t\\begin{displaymath}\n\t\\ba{rcl}\n\tK & \\; \\leq \\; & \\max\\bigl\\{  \\left( \\gammafnu \\right)^{\\frac{-1}{1\n\t\t\t+ \\nu}} \\cdot \\frac{1 + \\nu}{2 + \\nu}  \\cdot \\frac{6^{3/2} \\cdot\n\t\t2^{1/2} \\cdot (8 + \\nu)^{(1 - \\nu) / (2 + 2\\nu)} }{ \\left( (1 +\n\t\t\\nu)(2 + \\nu) \\right)^{1 / (1 + \\nu)}  } , \\; 1 \\bigr\\} \\cdot \\log\n\t\\frac{ F(x_0) - F^{*} }{\\varepsilon}.\n\t\\ea\n\t\\end{displaymath}\n\tMoreover, we have the following bound for the total number of oracle\n\tcalls $N_K$ during the first $K$ iterations:\n\t\\begin{equation} \\label{HolderOracleCallsBound}\n\t\\ba{rcl}\n\tN_K & \\; \\leq \\; & 2K  +  \\log_2\n\t\\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1 - \\nu) / (2 + \\nu)}} -\n\t\\log_2 H_0.\n\t\\ea\n\t\\end{equation}\n\\end{theorem}","start":55337,"end":57236}]},{"id":"S057","semantic_type":"derived_fact","kind":"derived_fact","labels":["TheoremLinearRateUniformly"],"declared_external":false,"source_only":false,"title":"作者所列迭代数结论","source_units":[{"raw_tex":"\tTherefore, the rate of convergence is linear, and\n\t\\begin{displaymath}\n\t\\ba{rcl}\n\tK & \\; \\leq \\; & \\max\\bigl\\{  \\left( \\gammafnu \\right)^{\\frac{-1}{1\n\t\t\t+ \\nu}} \\cdot \\frac{1 + \\nu}{2 + \\nu}  \\cdot \\frac{6^{3/2} \\cdot\n\t\t2^{1/2} \\cdot (8 + \\nu)^{(1 - \\nu) / (2 + 2\\nu)} }{ \\left( (1 +\n\t\t\\nu)(2 + \\nu) \\right)^{1 / (1 + \\nu)}  } , \\; 1 \\bigr\\} \\cdot \\log\n\t\\frac{ F(x_0) - F^{*} }{\\varepsilon}.\n\t\\ea\n\t\\end{displaymath}\n","start":56502,"end":56919}]},{"id":"S058","semantic_type":"proof","kind":"proof","labels":["HolderHtGoodBound","Th2HLowerBound","Th2UsefulBound"],"declared_external":false,"source_only":false,"title":"主定理完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\tThe proof is based on Lemmas~\\ref{LemmaStepHolder}\n\tand~\\ref{LemmaUpHolder}, and monotonicity of the sequence\n\t$\\bigl\\{ F(x_k)  \\bigr\\}_{k \\geq 0}$.\n\t\tFirstly, we need to show that every iteration of the method is\n\twell-defined. Namely, we are going to verify that for a fixed $0 \\leq\n\tk \\leq K-1$, there exists a finite integer $\\ell \\geq 0$ such that\n\teither $F(T_{H_k 2^{\\ell}}(x_k) ) \\leq M_{H_k 2^{\\ell}}^{*}(x_k)$ or\n\t$F(T_{H_k 2^{\\ell + 1}}(x_k)) - F^{*} < \\varepsilon$.\n\t\tIndeed, let us set\n\t\\begin{equation} \\label{Th2HLowerBound}\n\t\\ba{rcl}\n\t\\ell := \\max\\left\\{0,  \\log_2 \\left \\lceil\n\t\\frac{\\kappa_f(\\nu)}{H_k \\varepsilon^{(1 - \\nu) / (2 + \\nu)} }\n\t\\right\\rceil  \\right\\}, \n\t& \\quad \\text{and} \\quad &\n\tH := H_k\n\t2^{\\ell}  \\geq  \\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1 - \\nu) / (2\n\t\t\t+ \\nu)}}.\n\t\\ea\n\t\\end{equation}\n\tThen, if we have both $F(T_{H}(x_k)) > M_{H}^{*}(x_k)$ and\n\t$F(T_{2H}(x_k)) - F^{*}\n\t\\geq \\varepsilon$, we get by Lemma \\ref{LemmaUpHolder}:\n\t\\begin{displaymath}\n\t\\ba{rcl}\n\tH & \\; \\overset{\\eqref{LemmaUpHolder3}}{<} \\; &\n\t\\frac{\\kappa_f(\\nu)}{\\left( F(T_{2H}(x_k)) - F^{*}  \\right)^{(1 - \\nu)\n\t\t\t/ (2 + \\nu)}  } \\; \\leq \\; \\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1\n\t\t\t- \\nu) / (2 + \\nu)}},\n\t\\ea\n\t\\end{displaymath}\n\twhich contradicts~\\eqref{Th2HLowerBound}. Therefore, if we are unable\n\tto find the value $0 \\leq i_k \\leq \\ell$ (see line~1 of\n\tAlgorithm) in a finite number of steps, that only means we have\n\talready solved the problem up to accuracy $\\varepsilon$.\n\t\n\t\n\t\n\tNow, let us show that for every $0 \\leq k \\leq K$ it holds:\n\t\\begin{equation} \\label{Th2UsefulBound}\n\t\\ba{rcl}\n\tH_k \\left( F(x_k) - F^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t& \\leq &\n\t\\max \\left\\{ \\kappa_f(\\nu), \\; H_0 \\left( F(x_0) - F^{*}\n\t\\right)^{\\frac{1 - \\nu}{2 + \\nu}} \\right\\}.\n\t\\ea\n\t\\end{equation}\n\tThis inequality is obviously valid for $k = 0$. Assume it\n\tis also valid for some $k \\geq 0$. Then, by definition of\n\t$H_{k + 1}$ (see line~3 of Algorithm), we have $H_{k +\n\t\t1} = H_k 2^{i_k - 1}$. \n\tThere are two cases.\n\t\t1) $i_k = 0$. Then, $H_{k + 1} < H_k$. By monotonicity of\n\t\t$\\bigl\\{ F(x_k) \\bigr\\}_{k \\geq 0}$ and by induction, we get:\n\t\t$$\n\t\t\\ba{rcl}\n\t\tH_{k + 1} \\left( F(x_{k + 1}) - F^{*} \\right)^{\\frac{1 - \\nu}{2 +\n\t\t\t\t\\nu}}  & \\; < \\; & H_k \\left( F(x_k) - F^{*} \\right)^{\\frac{1 -\n\t\t\t\t\\nu}{2 + \\nu}} \\\\[5pt]\n\t\t& \\; \\leq \\; & \n\t\t\\max \\left\\{ \\kappa_f(\\nu), \\; H_0 \\left( F(x_0) -\n\t\tF^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \\right\\}.\n\t\t\\ea\n\t\t$$\n\t\t\n\t\t2) $i_k > 0$. Then, applying Lemma~\\ref{LemmaUpHolder} with $H\n\t\t:= H_k 2^{i_k - 1} = H_{k + 1}$ and $x :=\n\t\tx_{k}$, we have:\n\t\t\\begin{displaymath}\n\t\t\\ba{rcl}\n\t\tH_{k + 1} \\left( F(x_{k + 1}) - F^{*} \\right)^{\\frac{1 - \\nu}{2 +\n\t\t\t\t\\nu}} & \\; = \\; & H \\left( F(T_{2H}(x)) - F^{*}\n\t\t\\right)^{\\frac{1 - \\nu}{2 + \\nu}} \\;\n\t\t\\overset{\\eqref{LemmaUpHolder3}}{\\leq} \\; \\kappa_f(\\nu).\n\t\t\\ea\n\t\t\\end{displaymath}\n\tThus, \\eqref{Th2UsefulBound} is true by induction.\n\tChoosing $H_0$ small enough~\\eqref{HolderChoosingH}, we have:\n\t\\begin{equation} \\label{HolderHtGoodBound}\n\t\\ba{rcl}\n\t2H_{k} \\left( F(x_k) - F^{*}\\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t& \\; \\leq \\; & \n\t2\\kappa_f(\\nu), \\qquad 0 \\leq k \\leq K.\n\t\\ea\n\t\\end{equation}\n\tFrom Lemma~\\ref{LemmaStepHolder} we know, that one of the two\n\tfollowing estimates is true (denote $\\delta_k := F(x_k) - F^{*}$):\n\t\n\t\n\t\\begin{enumerate}\n\t\t\\item[1)] $F(x_k) - F(x_{k + 1})  \\geq \\alpha \\cdot \\delta_k\n\t\t \\;  \\Leftrightarrow  \\;  \\delta_{k + 1}  \\leq (1 -\n\t\t\\alpha) \\cdot \\delta_k, \\; $ \\textit{or}\n\t\t\n\t\t\\item[2)] $F(x_k) - F(x_{k + 1})  \\geq \\beta \\cdot \\delta_{k +\n\t\t\t1}  \\; \\Leftrightarrow \\;  \\delta_{k + 1}\n\t \t\\leq (1 + \\beta)^{-1} \\delta_k  \\leq  (1 - \\min\\{\\beta,\n\t\t1\\} / 2) \\cdot \\delta_k$,\n\t\\end{enumerate}\n\twhere\n\t$\n\t\\alpha  \n\t := \n\t \\frac{1 + \\nu}{2 + \\nu} \\cdot\n\t\\min\\bigl\\{\\bigl(  \\frac{(1 + \\nu) \\gammafnu}{2(2 + \\nu)} \\bigr)^{\n\t\t\\frac{1}{1 + \\nu}}, \\; 1\\bigr\\},\n\t$\n\tand\n\t$$\n\t\\ba{rcl}\n\t\\beta \n\t & := & \n\t\\bigl( \\frac{2 + \\nu}{1 + \\nu}\n\t\\bigr)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}} \\cdot\n\t\\frac{(\\sigmafnu)^{\\frac{3}{2(2 + \\nu)}}}{3 (2 \\kappa_f(\\nu))^{1/2}\n\t}\n\t\\; \\refEQ{KappaDefinition} \\;\n\t\\frac{2 + \\nu}{1 + \\nu} \\cdot \\frac{ 2^{1/2} \\cdot\n\t\t\\left( (1 + \\nu) (2 + \\nu)  \\right)^{\\frac{1}{1 + \\nu}} }{6^{3/2}\n\t\t\\cdot (8 + \\nu)^{(1 - \\nu) / (2 + 2\\nu)}} \\cdot \\gammafnu^{\\frac{1}{1\n\t\t\t+ \\nu}}.\n\t\\ea\n\t$$\n\tIt remains to notice, that $\\alpha \\geq \\min \\bigl\\{\\beta, 1\\bigr\\} /\n\t2$. Thus, we obtain~\\eqref{HolderMainResult}.\n\t\n\tFinally, let us estimate the total number of the oracle calls $N_K$\n\tduring the first $K$ iterations. At each\n\titeration, the oracle is called $i_k + 1$ times, and we have $H_{k +\n\t\t1} = H_k 2^{i_k - 1}$. Therefore,\n\t$$\n\t\\ba{rcl}\n\tN_K & \\; = \\; & \\sum_{k = 0}^{K - 1} (i_k + 1) \\; = \\; \\sum_{k = 0}^{K -\n\t\t1} \\left( \\log_2 \\frac{H_{k + 1}}{H_k} + 2\\right) \\\\[5pt]\n\t& \\; = \\; & 2K + \\log_2\n\tH_{K} - \\log_2 H_0\n\t\\; \\overset{\\eqref{HolderHtGoodBound},\n\t\t\\eqref{HolderConditionFuncEps}}{\\leq} \\;\n\t2K + \\log_2\n\t\\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1 - \\nu) / (2 + \\nu)}} - \\log_2\n\tH_0.\n\t\\ea\n\t$$\n\\qed\n\\end{proof}","start":57237,"end":62279}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"主证明路线声明","source_units":[{"raw_tex":"\\begin{proof}\n\tThe proof is based on Lemmas~\\ref{LemmaStepHolder}\n\tand~\\ref{LemmaUpHolder}, and monotonicity of the sequence\n\t$\\bigl\\{ F(x_k)  \\bigr\\}_{k \\geq 0}$.\n","start":57237,"end":57401}]},{"id":"S060","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"接受迭代的目标单调性","source_units":[{"raw_tex":"\tThe proof is based on Lemmas~\\ref{LemmaStepHolder}\n\tand~\\ref{LemmaUpHolder}, and monotonicity of the sequence\n\t$\\bigl\\{ F(x_k)  \\bigr\\}_{k \\geq 0}$.\n","start":57251,"end":57401},{"raw_tex":"\t\t1) $i_k = 0$. Then, $H_{k + 1} < H_k$. By monotonicity of\n\t\t$\\bigl\\{ F(x_k) \\bigr\\}_{k \\geq 0}$ and by induction, we get:\n\t\t$$\n\t\t\\ba{rcl}\n\t\tH_{k + 1} \\left( F(x_{k + 1}) - F^{*} \\right)^{\\frac{1 - \\nu}{2 +\n\t\t\t\t\\nu}}  & \\; < \\; & H_k \\left( F(x_k) - F^{*} \\right)^{\\frac{1 -\n\t\t\t\t\\nu}{2 + \\nu}} \\\\[5pt]\n\t\t& \\; \\leq \\; & \n\t\t\\max \\left\\{ \\kappa_f(\\nu), \\; H_0 \\left( F(x_0) -\n\t\tF^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \\right\\}.\n\t\t\\ea\n\t\t$$\n","start":59271,"end":59708}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":["Th2HLowerBound"],"declared_external":false,"source_only":false,"title":"构造有限试探阈值","source_units":[{"raw_tex":"\t\tFirstly, we need to show that every iteration of the method is\n\twell-defined. Namely, we are going to verify that for a fixed $0 \\leq\n\tk \\leq K-1$, there exists a finite integer $\\ell \\geq 0$ such that\n\teither $F(T_{H_k 2^{\\ell}}(x_k) ) \\leq M_{H_k 2^{\\ell}}^{*}(x_k)$ or\n\t$F(T_{H_k 2^{\\ell + 1}}(x_k)) - F^{*} < \\varepsilon$.\n\t\tIndeed, let us set\n\t\\begin{equation} \\label{Th2HLowerBound}\n\t\\ba{rcl}\n\t\\ell := \\max\\left\\{0,  \\log_2 \\left \\lceil\n\t\\frac{\\kappa_f(\\nu)}{H_k \\varepsilon^{(1 - \\nu) / (2 + \\nu)} }\n\t\\right\\rceil  \\right\\}, \n\t& \\quad \\text{and} \\quad &\n\tH := H_k\n\t2^{\\ell}  \\geq  \\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1 - \\nu) / (2\n\t\t\t+ \\nu)}}.\n\t\\ea\n\t\\end{equation}\n","start":57401,"end":58074}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"失败与未达标导致矛盾","source_units":[{"raw_tex":"\tThen, if we have both $F(T_{H}(x_k)) > M_{H}^{*}(x_k)$ and\n\t$F(T_{2H}(x_k)) - F^{*}\n\t\\geq \\varepsilon$, we get by Lemma \\ref{LemmaUpHolder}:\n\t\\begin{displaymath}\n\t\\ba{rcl}\n\tH & \\; \\overset{\\eqref{LemmaUpHolder3}}{<} \\; &\n\t\\frac{\\kappa_f(\\nu)}{\\left( F(T_{2H}(x_k)) - F^{*}  \\right)^{(1 - \\nu)\n\t\t\t/ (2 + \\nu)}  } \\; \\leq \\; \\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1\n\t\t\t- \\nu) / (2 + \\nu)}},\n\t\\ea\n\t\\end{displaymath}\n\twhich contradicts~\\eqref{Th2HLowerBound}. Therefore, if we are unable\n\tto find the value $0 \\leq i_k \\leq \\ell$ (see line~1 of\n\tAlgorithm) in a finite number of steps, that only means we have\n\talready solved the problem up to accuracy $\\varepsilon$.\n","start":58074,"end":58735}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":["Th2UsefulBound"],"declared_external":false,"source_only":false,"title":"乘积不变量与归纳起点","source_units":[{"raw_tex":"\t\n\t\n\t\n\tNow, let us show that for every $0 \\leq k \\leq K$ it holds:\n\t\\begin{equation} \\label{Th2UsefulBound}\n\t\\ba{rcl}\n\tH_k \\left( F(x_k) - F^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t& \\leq &\n\t\\max \\left\\{ \\kappa_f(\\nu), \\; H_0 \\left( F(x_0) - F^{*}\n\t\\right)^{\\frac{1 - \\nu}{2 + \\nu}} \\right\\}.\n\t\\ea\n\t\\end{equation}\n\tThis inequality is obviously valid for $k = 0$. Assume it\n\tis also valid for some $k \\geq 0$. Then, by definition of\n\t$H_{k + 1}$ (see line~3 of Algorithm), we have $H_{k +\n\t\t1} = H_k 2^{i_k - 1}$. \n\tThere are two cases.\n","start":58735,"end":59271}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"直接接受时保持不变量","source_units":[{"raw_tex":"\t\t1) $i_k = 0$. Then, $H_{k + 1} < H_k$. By monotonicity of\n\t\t$\\bigl\\{ F(x_k) \\bigr\\}_{k \\geq 0}$ and by induction, we get:\n\t\t$$\n\t\t\\ba{rcl}\n\t\tH_{k + 1} \\left( F(x_{k + 1}) - F^{*} \\right)^{\\frac{1 - \\nu}{2 +\n\t\t\t\t\\nu}}  & \\; < \\; & H_k \\left( F(x_k) - F^{*} \\right)^{\\frac{1 -\n\t\t\t\t\\nu}{2 + \\nu}} \\\\[5pt]\n\t\t& \\; \\leq \\; & \n\t\t\\max \\left\\{ \\kappa_f(\\nu), \\; H_0 \\left( F(x_0) -\n\t\tF^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \\right\\}.\n\t\t\\ea\n\t\t$$\n\t\t\n","start":59271,"end":59711}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"存在失败试探时控制参数","source_units":[{"raw_tex":"\t\t2) $i_k > 0$. Then, applying Lemma~\\ref{LemmaUpHolder} with $H\n\t\t:= H_k 2^{i_k - 1} = H_{k + 1}$ and $x :=\n\t\tx_{k}$, we have:\n\t\t\\begin{displaymath}\n\t\t\\ba{rcl}\n\t\tH_{k + 1} \\left( F(x_{k + 1}) - F^{*} \\right)^{\\frac{1 - \\nu}{2 +\n\t\t\t\t\\nu}} & \\; = \\; & H \\left( F(T_{2H}(x)) - F^{*}\n\t\t\\right)^{\\frac{1 - \\nu}{2 + \\nu}} \\;\n\t\t\\overset{\\eqref{LemmaUpHolder3}}{\\leq} \\; \\kappa_f(\\nu).\n\t\t\\ea\n\t\t\\end{displaymath}\n","start":59711,"end":60116}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":["HolderHtGoodBound"],"declared_external":false,"source_only":false,"title":"闭合归纳并代入初始化","source_units":[{"raw_tex":"\tThus, \\eqref{Th2UsefulBound} is true by induction.\n\tChoosing $H_0$ small enough~\\eqref{HolderChoosingH}, we have:\n\t\\begin{equation} \\label{HolderHtGoodBound}\n\t\\ba{rcl}\n\t2H_{k} \\left( F(x_k) - F^{*}\\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t& \\; \\leq \\; & \n\t2\\kappa_f(\\nu), \\qquad 0 \\leq k \\leq K.\n\t\\ea\n\t\\end{equation}\n","start":60116,"end":60428}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"将模型进展转为两种收缩","source_units":[{"raw_tex":"\tFrom Lemma~\\ref{LemmaStepHolder} we know, that one of the two\n\tfollowing estimates is true (denote $\\delta_k := F(x_k) - F^{*}$):\n\t\n\t\n\t\\begin{enumerate}\n","start":60428,"end":60582}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"当前残差分支的收缩","source_units":[{"raw_tex":"\t\t\\item[1)] $F(x_k) - F(x_{k + 1})  \\geq \\alpha \\cdot \\delta_k\n\t\t \\;  \\Leftrightarrow  \\;  \\delta_{k + 1}  \\leq (1 -\n\t\t\\alpha) \\cdot \\delta_k, \\; $ \\textit{or}\n\t\t\n","start":60582,"end":60745},{"raw_tex":"\twhere\n\t$\n\t\\alpha  \n\t := \n\t \\frac{1 + \\nu}{2 + \\nu} \\cdot\n\t\\min\\bigl\\{\\bigl(  \\frac{(1 + \\nu) \\gammafnu}{2(2 + \\nu)} \\bigr)^{\n\t\t\\frac{1}{1 + \\nu}}, \\; 1\\bigr\\},\n\t$\n","start":60959,"end":61123}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"下一点残差分支的收缩","source_units":[{"raw_tex":"\t\t\\item[2)] $F(x_k) - F(x_{k + 1})  \\geq \\beta \\cdot \\delta_{k +\n\t\t\t1}  \\; \\Leftrightarrow \\;  \\delta_{k + 1}\n\t \t\\leq (1 + \\beta)^{-1} \\delta_k  \\leq  (1 - \\min\\{\\beta,\n\t\t1\\} / 2) \\cdot \\delta_k$,\n\t\\end{enumerate}\n","start":60745,"end":60959}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"消去残差并化简系数","source_units":[{"raw_tex":"\tand\n\t$$\n\t\\ba{rcl}\n\t\\beta \n\t & := & \n\t\\bigl( \\frac{2 + \\nu}{1 + \\nu}\n\t\\bigr)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}} \\cdot\n\t\\frac{(\\sigmafnu)^{\\frac{3}{2(2 + \\nu)}}}{3 (2 \\kappa_f(\\nu))^{1/2}\n\t}\n\t\\; \\refEQ{KappaDefinition} \\;\n\t\\frac{2 + \\nu}{1 + \\nu} \\cdot \\frac{ 2^{1/2} \\cdot\n\t\t\\left( (1 + \\nu) (2 + \\nu)  \\right)^{\\frac{1}{1 + \\nu}} }{6^{3/2}\n\t\t\\cdot (8 + \\nu)^{(1 - \\nu) / (2 + 2\\nu)}} \\cdot \\gammafnu^{\\frac{1}{1\n\t\t\t+ \\nu}}.\n\t\\ea\n\t$$\n","start":61123,"end":61554}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"统一得到线性收缩","source_units":[{"raw_tex":"\tIt remains to notice, that $\\alpha \\geq \\min \\bigl\\{\\beta, 1\\bigr\\} /\n\t2$. Thus, we obtain~\\eqref{HolderMainResult}.\n\t\n","start":61554,"end":61674}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"逐次试探的调用计数","source_units":[{"raw_tex":"\tFinally, let us estimate the total number of the oracle calls $N_K$\n\tduring the first $K$ iterations. At each\n\titeration, the oracle is called $i_k + 1$ times, and we have $H_{k +\n\t\t1} = H_k 2^{i_k - 1}$. Therefore,\n\t$$\n\t\\ba{rcl}\n\tN_K & \\; = \\; & \\sum_{k = 0}^{K - 1} (i_k + 1) \\; = \\; \\sum_{k = 0}^{K -\n\t\t1} \\left( \\log_2 \\frac{H_{k + 1}}{H_k} + 2\\right) \\\\[5pt]\n\t& \\; = \\; & 2K + \\log_2\n\tH_{K} - \\log_2 H_0\n","start":61674,"end":62084}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"以残差下界截断参数","source_units":[{"raw_tex":"\t\\; \\overset{\\eqref{HolderHtGoodBound},\n\t\t\\eqref{HolderConditionFuncEps}}{\\leq} \\;\n\t2K + \\log_2\n\t\\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1 - \\nu) / (2 + \\nu)}} - \\log_2\n\tH_0.\n\t\\ea\n\t$$\n\\qed\n\\end{proof}","start":62084,"end":62279}]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原前提清单：Standard convex subdifferential calculus, first-order optimality, dual-norm inequalities；其中海森矩阵半正定性及最优点零次梯度为 annotator 明示的 implicit standard fact。"],"source_only":false,"title":"标准凸分析与对偶范数","source_units":[]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原前提清单：Standard ... Hölder Taylor remainder estimates。"],"source_only":false,"title":"霍尔德泰勒余项原理","source_units":[]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原前提清单：Standard differentiation of a parameterized minimum with a unique minimizer, used at lines 1533–1538。"],"source_only":false,"title":"参数化最小值求导","source_units":[]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；原前提清单未逐项列出几何衰减的对数计数。"],"source_only":false,"title":"几何衰减与对数计数","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S056","edges":[{"source_id":"S001","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S001","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S002","target_id":"S007","kind":"supports"},{"source_id":"S004","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S074","target_id":"S007","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S006","target_id":"S009","kind":"supports"},{"source_id":"S003","target_id":"S009","kind":"supports"},{"source_id":"S001","target_id":"S009","kind":"supports"},{"source_id":"S009","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S075","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S075","target_id":"S012","kind":"supports"},{"source_id":"S010","target_id":"S014","kind":"supports"},{"source_id":"S006","target_id":"S014","kind":"supports"},{"source_id":"S001","target_id":"S014","kind":"supports"},{"source_id":"S009","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S012","target_id":"S016","kind":"supports"},{"source_id":"S014","target_id":"S016","kind":"supports"},{"source_id":"S013","target_id":"S016","kind":"supports"},{"source_id":"S002","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S019","kind":"supports"},{"source_id":"S017","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S002","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S001","target_id":"S021","kind":"supports"},{"source_id":"S019","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S002","target_id":"S022","kind":"supports"},{"source_id":"S074","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S074","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S019","target_id":"S026","kind":"supports"},{"source_id":"S013","target_id":"S026","kind":"supports"},{"source_id":"S023","target_id":"S027","kind":"supports"},{"source_id":"S002","target_id":"S027","kind":"supports"},{"source_id":"S074","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S023","target_id":"S028","kind":"supports"},{"source_id":"S011","target_id":"S028","kind":"supports"},{"source_id":"S014","target_id":"S028","kind":"supports"},{"source_id":"S001","target_id":"S028","kind":"supports"},{"source_id":"S074","target_id":"S028","kind":"supports"},{"source_id":"S014","target_id":"S029","kind":"supports"},{"source_id":"S009","target_id":"S029","kind":"supports"},{"source_id":"S007","target_id":"S031","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S035","target_id":"S033","kind":"supports"},{"source_id":"S036","target_id":"S033","kind":"supports"},{"source_id":"S022","target_id":"S040","kind":"supports"},{"source_id":"S074","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S012","target_id":"S041","kind":"supports"},{"source_id":"S014","target_id":"S041","kind":"supports"},{"source_id":"S019","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S007","target_id":"S042","kind":"supports"},{"source_id":"S008","target_id":"S042","kind":"supports"},{"source_id":"S009","target_id":"S042","kind":"supports"},{"source_id":"S005","target_id":"S042","kind":"supports"},{"source_id":"S074","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S015","target_id":"S043","kind":"supports"},{"source_id":"S028","target_id":"S044","kind":"supports"},{"source_id":"S026","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S030","target_id":"S045","kind":"supports"},{"source_id":"S009","target_id":"S045","kind":"supports"},{"source_id":"S005","target_id":"S045","kind":"supports"},{"source_id":"S074","target_id":"S045","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S041","target_id":"S038","kind":"supports"},{"source_id":"S042","target_id":"S038","kind":"supports"},{"source_id":"S043","target_id":"S038","kind":"supports"},{"source_id":"S044","target_id":"S038","kind":"supports"},{"source_id":"S045","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S012","target_id":"S049","kind":"supports"},{"source_id":"S014","target_id":"S049","kind":"supports"},{"source_id":"S019","target_id":"S049","kind":"supports"},{"source_id":"S019","target_id":"S050","kind":"supports"},{"source_id":"S017","target_id":"S050","kind":"supports"},{"source_id":"S021","target_id":"S050","kind":"supports"},{"source_id":"S076","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S049","target_id":"S051","kind":"supports"},{"source_id":"S030","target_id":"S052","kind":"supports"},{"source_id":"S028","target_id":"S052","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S009","target_id":"S052","kind":"supports"},{"source_id":"S005","target_id":"S052","kind":"supports"},{"source_id":"S029","target_id":"S052","kind":"supports"},{"source_id":"S049","target_id":"S048","kind":"supports"},{"source_id":"S050","target_id":"S048","kind":"supports"},{"source_id":"S051","target_id":"S048","kind":"supports"},{"source_id":"S052","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S014","target_id":"S053","kind":"supports"},{"source_id":"S009","target_id":"S053","kind":"supports"},{"source_id":"S020","target_id":"S053","kind":"supports"},{"source_id":"S029","target_id":"S054","kind":"supports"},{"source_id":"S008","target_id":"S054","kind":"supports"},{"source_id":"S020","target_id":"S054","kind":"supports"},{"source_id":"S020","target_id":"S055","kind":"supports"},{"source_id":"S019","target_id":"S055","kind":"supports"},{"source_id":"S008","target_id":"S055","kind":"supports"},{"source_id":"S020","target_id":"S060","kind":"supports"},{"source_id":"S026","target_id":"S060","kind":"supports"},{"source_id":"S074","target_id":"S060","kind":"supports"},{"source_id":"S020","target_id":"S061","kind":"supports"},{"source_id":"S053","target_id":"S061","kind":"supports"},{"source_id":"S029","target_id":"S061","kind":"supports"},{"source_id":"S055","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S046","target_id":"S062","kind":"supports"},{"source_id":"S053","target_id":"S062","kind":"supports"},{"source_id":"S020","target_id":"S063","kind":"supports"},{"source_id":"S029","target_id":"S063","kind":"supports"},{"source_id":"S008","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S060","target_id":"S064","kind":"supports"},{"source_id":"S020","target_id":"S064","kind":"supports"},{"source_id":"S053","target_id":"S064","kind":"supports"},{"source_id":"S055","target_id":"S064","kind":"supports"},{"source_id":"S020","target_id":"S065","kind":"supports"},{"source_id":"S046","target_id":"S065","kind":"supports"},{"source_id":"S053","target_id":"S065","kind":"supports"},{"source_id":"S063","target_id":"S066","kind":"supports"},{"source_id":"S064","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S054","target_id":"S066","kind":"supports"},{"source_id":"S037","target_id":"S067","kind":"supports"},{"source_id":"S020","target_id":"S067","kind":"supports"},{"source_id":"S053","target_id":"S067","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S015","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S070","kind":"supports"},{"source_id":"S066","target_id":"S070","kind":"supports"},{"source_id":"S020","target_id":"S070","kind":"supports"},{"source_id":"S029","target_id":"S070","kind":"supports"},{"source_id":"S015","target_id":"S070","kind":"supports"},{"source_id":"S067","target_id":"S069","kind":"supports"},{"source_id":"S070","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S071","kind":"supports"},{"source_id":"S069","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S053","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S057","kind":"supports"},{"source_id":"S055","target_id":"S057","kind":"supports"},{"source_id":"S077","target_id":"S057","kind":"supports"},{"source_id":"S020","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S066","target_id":"S073","kind":"supports"},{"source_id":"S055","target_id":"S073","kind":"supports"},{"source_id":"S059","target_id":"S058","kind":"supports"},{"source_id":"S061","target_id":"S058","kind":"supports"},{"source_id":"S062","target_id":"S058","kind":"supports"},{"source_id":"S063","target_id":"S058","kind":"supports"},{"source_id":"S064","target_id":"S058","kind":"supports"},{"source_id":"S065","target_id":"S058","kind":"supports"},{"source_id":"S066","target_id":"S058","kind":"supports"},{"source_id":"S067","target_id":"S058","kind":"supports"},{"source_id":"S068","target_id":"S058","kind":"supports"},{"source_id":"S069","target_id":"S058","kind":"supports"},{"source_id":"S070","target_id":"S058","kind":"supports"},{"source_id":"S071","target_id":"S058","kind":"supports"},{"source_id":"S072","target_id":"S058","kind":"supports"},{"source_id":"S073","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S056","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S056","edges":[{"source_id":"S001","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S001","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S002","target_id":"S007","kind":"supports"},{"source_id":"S004","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S074","target_id":"S007","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S006","target_id":"S009","kind":"supports"},{"source_id":"S003","target_id":"S009","kind":"supports"},{"source_id":"S001","target_id":"S009","kind":"supports"},{"source_id":"S009","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S075","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S075","target_id":"S012","kind":"supports"},{"source_id":"S010","target_id":"S014","kind":"supports"},{"source_id":"S006","target_id":"S014","kind":"supports"},{"source_id":"S001","target_id":"S014","kind":"supports"},{"source_id":"S009","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S012","target_id":"S016","kind":"supports"},{"source_id":"S014","target_id":"S016","kind":"supports"},{"source_id":"S013","target_id":"S016","kind":"supports"},{"source_id":"S002","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S019","kind":"supports"},{"source_id":"S017","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S002","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S001","target_id":"S021","kind":"supports"},{"source_id":"S019","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S002","target_id":"S022","kind":"supports"},{"source_id":"S074","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S074","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S019","target_id":"S026","kind":"supports"},{"source_id":"S013","target_id":"S026","kind":"supports"},{"source_id":"S023","target_id":"S027","kind":"supports"},{"source_id":"S002","target_id":"S027","kind":"supports"},{"source_id":"S074","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S023","target_id":"S028","kind":"supports"},{"source_id":"S011","target_id":"S028","kind":"supports"},{"source_id":"S014","target_id":"S028","kind":"supports"},{"source_id":"S001","target_id":"S028","kind":"supports"},{"source_id":"S074","target_id":"S028","kind":"supports"},{"source_id":"S014","target_id":"S029","kind":"supports"},{"source_id":"S009","target_id":"S029","kind":"supports"},{"source_id":"S007","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S030","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S035","target_id":"S033","kind":"supports"},{"source_id":"S036","target_id":"S033","kind":"supports"},{"source_id":"S022","target_id":"S037","kind":"supports"},{"source_id":"S074","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S037","kind":"supports"},{"source_id":"S012","target_id":"S037","kind":"supports"},{"source_id":"S014","target_id":"S037","kind":"supports"},{"source_id":"S019","target_id":"S037","kind":"supports"},{"source_id":"S007","target_id":"S037","kind":"supports"},{"source_id":"S008","target_id":"S037","kind":"supports"},{"source_id":"S009","target_id":"S037","kind":"supports"},{"source_id":"S005","target_id":"S037","kind":"supports"},{"source_id":"S015","target_id":"S037","kind":"supports"},{"source_id":"S028","target_id":"S037","kind":"supports"},{"source_id":"S026","target_id":"S037","kind":"supports"},{"source_id":"S030","target_id":"S037","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S012","target_id":"S046","kind":"supports"},{"source_id":"S014","target_id":"S046","kind":"supports"},{"source_id":"S019","target_id":"S046","kind":"supports"},{"source_id":"S017","target_id":"S046","kind":"supports"},{"source_id":"S021","target_id":"S046","kind":"supports"},{"source_id":"S076","target_id":"S046","kind":"supports"},{"source_id":"S030","target_id":"S046","kind":"supports"},{"source_id":"S028","target_id":"S046","kind":"supports"},{"source_id":"S009","target_id":"S046","kind":"supports"},{"source_id":"S005","target_id":"S046","kind":"supports"},{"source_id":"S029","target_id":"S046","kind":"supports"},{"source_id":"S014","target_id":"S053","kind":"supports"},{"source_id":"S009","target_id":"S053","kind":"supports"},{"source_id":"S020","target_id":"S053","kind":"supports"},{"source_id":"S029","target_id":"S054","kind":"supports"},{"source_id":"S008","target_id":"S054","kind":"supports"},{"source_id":"S020","target_id":"S054","kind":"supports"},{"source_id":"S020","target_id":"S055","kind":"supports"},{"source_id":"S019","target_id":"S055","kind":"supports"},{"source_id":"S008","target_id":"S055","kind":"supports"},{"source_id":"S020","target_id":"S060","kind":"supports"},{"source_id":"S026","target_id":"S060","kind":"supports"},{"source_id":"S074","target_id":"S060","kind":"supports"},{"source_id":"S020","target_id":"S061","kind":"supports"},{"source_id":"S053","target_id":"S061","kind":"supports"},{"source_id":"S029","target_id":"S061","kind":"supports"},{"source_id":"S055","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S046","target_id":"S062","kind":"supports"},{"source_id":"S053","target_id":"S062","kind":"supports"},{"source_id":"S020","target_id":"S063","kind":"supports"},{"source_id":"S029","target_id":"S063","kind":"supports"},{"source_id":"S008","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S060","target_id":"S064","kind":"supports"},{"source_id":"S020","target_id":"S064","kind":"supports"},{"source_id":"S053","target_id":"S064","kind":"supports"},{"source_id":"S055","target_id":"S064","kind":"supports"},{"source_id":"S020","target_id":"S065","kind":"supports"},{"source_id":"S046","target_id":"S065","kind":"supports"},{"source_id":"S053","target_id":"S065","kind":"supports"},{"source_id":"S063","target_id":"S066","kind":"supports"},{"source_id":"S064","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S054","target_id":"S066","kind":"supports"},{"source_id":"S037","target_id":"S067","kind":"supports"},{"source_id":"S020","target_id":"S067","kind":"supports"},{"source_id":"S053","target_id":"S067","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S015","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S070","kind":"supports"},{"source_id":"S066","target_id":"S070","kind":"supports"},{"source_id":"S020","target_id":"S070","kind":"supports"},{"source_id":"S029","target_id":"S070","kind":"supports"},{"source_id":"S015","target_id":"S070","kind":"supports"},{"source_id":"S067","target_id":"S069","kind":"supports"},{"source_id":"S070","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S071","kind":"supports"},{"source_id":"S069","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S053","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S057","kind":"supports"},{"source_id":"S055","target_id":"S057","kind":"supports"},{"source_id":"S077","target_id":"S057","kind":"supports"},{"source_id":"S020","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S066","target_id":"S073","kind":"supports"},{"source_id":"S055","target_id":"S073","kind":"supports"},{"source_id":"S059","target_id":"S058","kind":"supports"},{"source_id":"S061","target_id":"S058","kind":"supports"},{"source_id":"S062","target_id":"S058","kind":"supports"},{"source_id":"S063","target_id":"S058","kind":"supports"},{"source_id":"S064","target_id":"S058","kind":"supports"},{"source_id":"S065","target_id":"S058","kind":"supports"},{"source_id":"S066","target_id":"S058","kind":"supports"},{"source_id":"S067","target_id":"S058","kind":"supports"},{"source_id":"S068","target_id":"S058","kind":"supports"},{"source_id":"S069","target_id":"S058","kind":"supports"},{"source_id":"S070","target_id":"S058","kind":"supports"},{"source_id":"S071","target_id":"S058","kind":"supports"},{"source_id":"S072","target_id":"S058","kind":"supports"},{"source_id":"S073","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S056","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"原对偶空间与范数","source_units":[{"raw_tex":"Let us start from some notation. In what follows, we denote\nby $\\E$ a finite-dimensional real vector space and by\n$\\E^{*}$ its dual space, which is a space of linear\nfunctions on $\\E$. The value of function $s \\in \\E^{*}$ at\npoint $x \\in \\E$ is denoted by $\\langle s, x \\rangle$.\nLet us fix some linear self-adjoint positive-definite operator\n$B: \\E \\to \\E^{*}$ and introduce the following Euclidean norms in the\nprimal and dual spaces:\n$$\n\\ba{rcl}\n\\|x\\| & \\defeq &  \\langle Bx, x \\rangle^{1/2}, \\quad x \\in \\E,\n\\qquad \\|s\\|_{*} \\; \\defeq \\; \\langle s, B^{-1}s \\rangle^{1/2},\n\\quad s \\in \\E^{*}.\n\\ea\n$$\nFor any linear operator $A: \\E \\to \\E^{*}$, its norm is induced in a\nstandard way:\n$$\n\\ba{rcl}\n\\|A\\| & \\defeq & \\max\\limits_{x \\in \\E} \\bigl\\{ \\|Ax\\|_{*} \\; | \\;\n\\|x\\| \\leq 1 \\bigr\\}.\n\\ea\n$$\n","start":8361,"end":9155}]},{"id":"S002","semantic_type":"problem_setting","kind":"problem_setting","labels":["MainProblem"],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"复合凸优化问题","source_units":[{"raw_tex":"Our goal is to solve the convex optimization problem\nin the composite form:\n\\beq \\label{MainProblem}\n\\ba{rcl}\n\\min\\limits_{x \\in \\dom F} F(x) & := & f(x) + h(x),\n\\ea\n\\eeq\nwhere $f$ is a twice differentiable on its open domain\nuniformly convex function, and $h$ is a \\textit{simple}\nclosed convex function with $\\dom h \\subseteq \\dom f$.\n\\textit{Simple} means that all auxiliary subproblems with\nan explicit presence of $h$ are easily solvable.\n","start":9155,"end":9599}]},{"id":"S003","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"导数与次微分","source_units":[{"raw_tex":"\nFor a smooth function $f$, its gradient at point $x$ is\ndenoted by $\\nabla f(x) \\in \\E^{*}$,  and its Hessian is\ndenoted by $\\nabla^2 f(x) : \\E \\to \\E^{*}$. For convex but\nnot necessary differentiable function $h$, we denote by\n$\\partial h(x) \\subset \\E^{*}$ its subdifferential at the\npoint $x \\in \\dom h$.\n\n","start":9599,"end":9909}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["UniformlyConvexDefinition"],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"一致凸性定义","source_units":[{"raw_tex":"We say that differentiable function $f$ is \\textit{uniformly convex}\nof degree $p \\geq 2$ on a convex set $C \\subseteq \\dom f$\nif for some constant $\\sigma > 0$ it satisfies inequality\n\\beq \\label{UniformlyConvexDefinition}\n\\ba{rcl}\nf(y) & \\; \\geq \\; & f(x) + \\langle \\nabla f(x), y - x \\rangle +\n\\frac{\\sigma}{p}\\|y - x\\|^p, \\qquad x, y \\in C.\n\\ea\n\\eeq\nUniformly convex functions of degree $p = 2$ are known as\n\\textit{strongly convex}. If\ninequality~\\eqref{UniformlyConvexDefinition} holds with\n$\\sigma = 0$, the function $f$ is called just \\textit{convex}.\n","start":9909,"end":10469}]},{"id":"S005","semantic_type":"external_fact","kind":"external_fact","labels":["UniformlyConvexSufficient"],"member_ids":["S005"],"declared_external":true,"declaration_text":["原前提清单明确允许 B0007；原文引用 Nesterov (2008), Accelerating the cubic regularization of Newton's method on convex problems, Lemma 1。"],"source_only":false,"title":"梯度单调性充分条件","source_units":[{"raw_tex":"The following convenient condition\nis sufficient for function $f$ to be uniformly convex on a convex set\n$C \\subseteq \\dom f$:\n\\begin{lemma}(Lemma 1 in~\\cite{nesterov2008accelerating})\n\tLet for some $\\sigma > 0$ and $p \\geq 2$\n\tthe following inequality holds:\n\t\\beq \\label{UniformlyConvexSufficient}\n\t\\ba{rcl}\n\t\\langle \\nabla f(x) - \\nabla f(y), x - y \\rangle & \\; \\geq \\; &\n\t\\sigma \\|x - y\\|^p, \\qquad x, y \\in C.\n\t\\ea\n\t\\eeq\n\tThen, function $f$ is uniformly convex of degree $p$ on set $C$ with\n\tparameter $\\sigma$.\n\\end{lemma}\n","start":10469,"end":10998}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"一致凸性作用域","source_units":[{"raw_tex":"From now on, we assume\n$\nC \\; := \\; \\dom F \\; \\subseteq \\; \\dom f.\n$\n","start":10998,"end":11067}]},{"id":"S007","semantic_type":"derived_fact","kind":"derived_fact","labels":["UniformlyConvexLowerBound"],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"复合目标一致凸下界","source_units":[{"raw_tex":"By the composite representation~\\eqref{MainProblem},\nwe have for every $x \\in \\dom F$ and for\nall $F'(x) \\in \\partial F(x)$:\n\\begin{equation} \\label{UniformlyConvexLowerBound}\n\\ba{rcl}\nF(y) & \\geq  & F(x) + \\langle\nF'(x), y - x \\rangle + \\frac{\\sigma}{p}\\|x - y\\|^p, \\qquad\ny \\in \\dom F.\n\\ea\n\\end{equation}\n","start":11067,"end":11374}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"最优解与最优值","source_units":[{"raw_tex":"Therefore, if $\\sigma > 0$, then we can have only one\npoint $x^{*} \\in \\dom F$ with $F(x^{*}) = F^{*}$, which\nalways exists for $F$ being uniformly convex and closed.\n","start":11374,"end":11541}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":["SigmaDefinition"],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"一致凸梯度模量","source_units":[{"raw_tex":"It is reasonable to define the best possible constant~$\\sigma$ in\ninequality~\\eqref{UniformlyConvexSufficient} for a certain degree~$p$.\nThis leads us to a system of constants:\n\\begin{equation} \\label{SigmaDefinition}\n\\ba{rcl}\n\\sigmafp & \\; \\defeq \\; & \\inf\\limits_{\\substack{x, y \\, \\in \\, \\dom F \\\\\n\t\tx \\not= y }} \\frac{\\langle \\nabla f(x) - \\nabla f(y), x - y \\rangle }\n{\\|x - y\\|^p}, \\qquad p \\geq 2.\n\\ea\n\\end{equation}\nWe prefer to use\ninequality~\\eqref{UniformlyConvexSufficient} for the\ndefinition of $\\sigmafp$, instead\nof~\\eqref{UniformlyConvexDefinition}, because of its\nsymmetry in $x$ and $y$. Note that the value $\\sigmafp$\nalso depends on the domain of $F$. However, we omit this\ndependence in our notation since it is always clear from\nthe context.\n","start":12288,"end":13052}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":["HolderHessianDefinition"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"海森矩阵霍尔德条件","source_units":[{"raw_tex":"For a twice-differentiable function $f$, we say\nthat it has \\textit{H\\\"older continuous Hessian} of degree\n$\\nu \\in [0, 1]$ on a convex set $C \\subseteq \\dom f$, if for some\nconstant $\\mathcal{H}$, it\nholds:\n\\begin{equation} \\label{HolderHessianDefinition}\n\\ba{rcl}\n\\| \\nabla^2 f(x) - \\nabla^2 f(y) \\| & \\leq &\n\\mathcal{H} \\|x - y\\|^{\\nu}, \\qquad x, y \\in C.\n\\ea\n\\end{equation}\n","start":13435,"end":13813}]},{"id":"S011","semantic_type":"derived_fact","kind":"derived_fact","labels":["HolderHessianGradBound"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"梯度泰勒余项","source_units":[{"raw_tex":"Two simple consequences of~\\eqref{HolderHessianDefinition}\nare as follows:\n\\beq \\label{HolderHessianGradBound}\n\\ba{rcl}\n\\| \\nabla f(y) - \\nabla f(x) - \\nabla^2 f(x)(y - x) \\|_{*} \n& \\leq &\n \\frac{\\H \\|x - y\\|^{1 + \\nu}}{1 + \\nu},\n\\ea\n\\eeq\n","start":13813,"end":14052}]},{"id":"S012","semantic_type":"derived_fact","kind":"derived_fact","labels":["HolderHessianFuncBound"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"函数泰勒余项","source_units":[{"raw_tex":"\\beq \\label{HolderHessianFuncBound}\n\\ba{rcl}\n| f(y) - Q(x; y) |  & \\leq & \n\\frac{\\H \\|x - y\\|^{2 + \\nu}}{(1 + \\nu)(2 + \\nu)},\n\\ea\n\\eeq\n","start":14052,"end":14187}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"局部二次模型","source_units":[{"raw_tex":"where $Q(x; y)$ is the quadratic model of $f$ at the point\n$x$:\n$$\n\\ba{rcl}\nQ(x; y) & \\; \\defeq \\; & f(x) + \\langle \\nabla f(x), y - x \\rangle +\n\\frac{1}{2} \\langle \\nabla^2 f(x) (y - x), y - x \\rangle.\n\\ea\n$$\n","start":14187,"end":14397}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":["HfnuDefinition"],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"霍尔德最优模量","source_units":[{"raw_tex":"\nIn order to characterize the level of smoothness of\nfunction $f$ on the set $C := \\dom F$, let us define\nthe system of H\\\"older constants (see\n\\cite{grapiglia2017regularized}):\n\\begin{equation} \\label{HfnuDefinition}\n\\ba{rcl}\n\\Hfnu & \\; \\defeq &\\; \\sup\\limits_{\\substack{x, y \\in \\dom F \\\\ x \\not= y}}\n\\frac{\\| \\nabla^2 f(x) - \\nabla^2 f(y) \\|}{\\|x - y\\|^{\\nu}},\n\\qquad \\nu \\in [0, 1].\n\\ea\n\\end{equation}\nWe allow $\\Hfnu$ to be equal to $+\\infty$ for some $\\nu$.\n","start":14397,"end":14861}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":["ConditionNumberDefinition"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"二阶条件数","source_units":[{"raw_tex":"Hence, for a twice-differentiable convex function $f$ with\n$\\inf_{\\nu \\in [0, 1]} \\Hfnu > 0$, we can define only one\nmeaningful \\textit{condition number} of degree $\\nu \\in\n[0, 1]$:\n\\begin{equation} \\label{ConditionNumberDefinition}\n\\ba{rcl}\n\\gamma_f(\\nu) &\\; \\defeq \\;& \\frac{\\sigma_f(2 + \\nu)}{\\Hfnu}.\n\\ea\n\\end{equation}\nIf for some particular $\\nu$ we have $\\Hfnu = +\\infty$, then by our\ndefinition: $\\gamma_f(\\nu) = 0$.\n","start":20006,"end":20430}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"一般正则化模型","source_units":[{"raw_tex":"Let us start from the case when we know that for a specific\n$\\nu \\in [0, 1]$ function $f$ has H\\\"older continuous Hessian:\n$\\H_f(\\nu) < +\\infty$. Then, from~\\eqref{HolderHessianFuncBound}, we\nhave the global upper bound for the objective function:\n\\begin{displaymath}\n\\ba{rcl}\nF(y) & \\; \\leq \\; & M_{\\nu, H}(x; y) \\;\\defeq \\; Q(x; y)\n+ \\frac{H \\|x - y\\|^{2 + \\nu}}{(1 + \\nu)(2 + \\nu)} + h(y),\n\\qquad x, y \\in \\dom F,\n\\ea\n\\end{displaymath}\nwhere $H > 0$ is large enough: $H \\geq \\Hfnu$. Thus, it is\n","start":34020,"end":34518}]},{"id":"S017","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"模型步与最小值","source_units":[{"raw_tex":"natural to employ the minimum of a regularized quadratic\nmodel:\n\\begin{displaymath}\n\\ba{c}\nT_{\\nu, H}(x) \\; \\defeq \\;\n\\argmin\\limits_{y \\in \\dom F} M_{\\nu, H}(x; y), \\qquad M_{\\nu, H}^{*}(x)\n\\; \\defeq \\; \\min\\limits_{y \\in \\dom F} M_{\\nu, H}(x; y),\n\\ea\n\\end{displaymath}\n","start":34518,"end":34789}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"三次模型简记","source_units":[{"raw_tex":"process~\\eqref{GeneralRegularizedNewton} with the fixed\nparameter $\\nu = 1$. For this choice, in the rest part of\nthe paper we omit the corresponding index in the\ndefinitions of all necessary objects:\n$M_H(x; y) := M_{1, H}(x; y)$, \n$T_H(x) := T_{1, H}(x)$, and $M_H^{*}(x) := M_{1, H}^{*}(x) = M_H(x; T_H(x))$.\nThe adaptive scheme of our method with dynamic estimation\nof the constant $H$ is as follows.\n","start":37817,"end":38222}]},{"id":"S020","semantic_type":"algorithm","kind":"algorithm","labels":[],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"自适应三次牛顿法","source_units":[{"raw_tex":"\\vspace*{5pt}\n\\noindent\n\\hspace*{5pt}\n\\lfbox{\n\t\\hspace*{4pt}\n\t\\centering\n\t\\begin{minipage}[b]{0.85\\linewidth} \\label{MainAlgorithm}\n\t\t\\begin{algorithm}[H] \n\t\t  \t\\caption{\\small \\textbf{Adaptive Cubic\n\t\t\t\t\tRegularization of Newton Method}}\n\t\t\t\\vspace*{6pt}\n\t\t\t\\noindent\\makebox[\\linewidth]{\\rule{1.097\\linewidth}{0.4pt}}\n\t\t\t\\begin{algorithmic}[1]\n\t\t\t\t%\\vspace{5pt}\n\t\t\t\t\\Require Choose $x_0 \\in \\dom F$, $H_0 > 0$.\n\t\t\t\t\\vspace*{2pt}\n\t\t\t\t\\Ensure $k \\geq 0$. \\vspace*{2pt}\n\t\t\t\t\\State Find the minimal integer $i_k \\geq 0 \\;$\n\t\t\t\tsuch that $F(T_{H_k 2^{i_k}}(x_k)) \\leq M^{*}_{H_k\n\t\t\t\t\t2^{i_k}} (x_k)$. %\\vspace*{1pt}\n\t\t\t\t\\State Perform the Cubic Step: $x_{k + 1}  = T_{H_k\n\t\t\t\t\t2^{i_k}}(x_k)$. \\vspace*{1pt}\n\t\t\t\t\\State Set $H_{k + 1} := 2^{i_k - 1} H_k $.\n\t\t\t\t\\vspace*{5pt}\n\t\t\t\\end{algorithmic}\n\t\t\\end{algorithm}\n\t\\end{minipage}\n\t\\hspace*{4pt}\n}\n\\vspace*{1pt}\n","start":38223,"end":39079}]},{"id":"S021","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"三次步长半径","source_units":[{"raw_tex":"Let us present the main properties of the composite Cubic\nNewton step $x \\mapsto T_H(x)$. Denote\n$$\nr_H(x) \\defeq \\|T_H(x) - x\\|.\n$$\n","start":39080,"end":39213}]},{"id":"S022","semantic_type":"derived_fact","kind":"derived_fact","labels":["CubicStepStationary"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"三次步最优性条件","source_units":[{"raw_tex":"Since point $T_H(x)$ is a minimum of strictly convex\nfunction $M_H(x;\\cdot)$, it satisfies the following\nfirst-order optimality condition:\n\\beq \\label{CubicStepStationary}\n\\ba{cl}\n&\\bigl \\la \\nabla f(x) + \\nabla^2 f(x)(T_H(x) - x) +\n\\tfrac{H r_H(x)}{2}B(T_H(x) - x), y - T_H(x) \\bigr \\ra  \\; + \\\\\n\\\\\n& \\quad h(y) \\; \\geq \\; h(T_H(x)),\n\\qquad y \\in \\dom F.\n\\ea\n\\eeq \n","start":39213,"end":39579}]},{"id":"S023","semantic_type":"derived_fact","kind":"derived_fact","labels":["CubicStepSubdiff"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"构造非光滑项次梯度","source_units":[{"raw_tex":"In other words, the vector\n$$\n\\ba{rcl}\nh'(T_H(x)) & \\defeq & -\\nabla f(x) - \\nabla^2 f(x)(T_H(x) -\nx) - \\frac{H r_H(x)}{2}B(T_H(x) - x)\n\\ea\n$$\nbelongs to the subdifferential of $h$:\n\\begin{equation} \\label{CubicStepSubdiff}\n\\ba{rcl}\nh'(T_H(x)) &\\; \\in \\; & \\partial h(T_H(x)).\n\\ea\n\\end{equation}\n","start":39579,"end":39875}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":["StationaryConsequence"],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"代入当前点","source_units":[{"raw_tex":"\nPlugging into~\\eqref{CubicStepStationary} $y := x \\in\n\\dom F$, we get:\n\\begin{align} \n&\\la \\nabla f(x), x - T_H(x) \\ra \\label{StationaryConsequence} \\\\\n&\\geq \\; \\la \\nabla^2 f(x) (T_H(x) - x), T_H(x) - x \\ra +\n\\tfrac{H r_H^3(x)}{2} + h(T_H(x)) - h(x). \\notag\n\\end{align}\n","start":40231,"end":40503}]},{"id":"S026","semantic_type":"derived_fact","kind":"derived_fact","labels":["CubicModelMinimumExpression"],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"三次模型下降界","source_units":[{"raw_tex":"Thus, we obtain the following bound for the minimal value\n$M_H^{*}(x)$ of the cubic model:\n\\beq \\label{CubicModelMinimumExpression}\n\\ba{rcl}\nM_H^{*}(x) \n& \\overset{\\eqref{StationaryConsequence}}{\\leq} &\nf(x) -\n\\tfrac{1}{2}\\la \\nabla^2 f(x)(T_H(x) - x), T_H(x) - x \\ra -\n\\tfrac{H r_H^3(x)}{3} + h(x) \\\\[5pt]\n& = &\nF(x) - \\tfrac{1}{2}\\la \\nabla^2 f(x)(T_H(x) - x),\nT_H(x) - x \\ra - \\tfrac{Hr_H^3(x)}{3}. \n\\ea\n\\eeq\n\n\n","start":40503,"end":40917}]},{"id":"S027","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"新点目标次梯度","source_units":[{"raw_tex":"If for some value $\\nu \\in [0, 1]$ the Hessian is H\\\"older continuous:\n$\\Hfnu < +\\infty$, then by \\eqref{HolderHessianGradBound}\nand~\\eqref{CubicStepSubdiff} we get the following bound for\nthe subgradient:\n$$\nF'(T_H(x)) \\defeq \\nabla f(T_H(x)) + h'(T_H(x))\n$$\nat the new point:\n","start":40917,"end":41195}]},{"id":"S028","semantic_type":"derived_fact","kind":"derived_fact","labels":["CubicGradNewPoint"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"新点次梯度估计","source_units":[{"raw_tex":"\\beq \\label{CubicGradNewPoint}\n\\ba{cl}\n& \\|  F'(T_H(x)) \\|_{*} \\\\[5pt]\n& \\leq \\;\n  \\| \\nabla f(T_H(x)) - \\nabla f(x) - \\nabla^2 f(x)\n(T_H(x) - x)\\|_{*} + \\tfrac{H r^2_H(x)}{2} \\\\[5pt]\n& \\stackrel{\\eqref{HolderHessianGradBound}}{\\leq}\n\\tfrac{\\Hfnu\n\tr^{1 + \\nu}_H(x)}{1 + \\nu}  +  \\tfrac{H r^2_H(x)}{2} \n\\; = \\; \nr^{1 + \\nu}_H(x) \\cdot \\Bigl(  \\tfrac{\\Hfnu}{1 + \\nu} + \n\\tfrac{Hr_H^{1 - \\nu}(x)}{2} \\Bigr). \n\\ea\n\\eeq\n","start":41195,"end":41610}]},{"id":"S029","semantic_type":"definition","kind":"definition","labels":["KappaDefinition"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"参数控制常数","source_units":[{"raw_tex":"Denote by $\\kappa_f(\\nu)$ the following auxiliary value:\n\\begin{equation} \\label{KappaDefinition}\n\\ba{rcl}\n\\kappa_f(\\nu)\n& \\; \\defeq \\; & \\frac{  \\Hfnu^{\\frac{2}{1 + \\nu}}   }\n{ (\\sigmafnu)^{\\frac{1 - \\nu}{(1 + \\nu)(2 + \\nu)}} }\n\\cdot\n\\frac{6 \\cdot (8 + \\nu)^{\\frac{1 - \\nu}{1 + \\nu}} }\n{ \\left((1 + \\nu)(2 + \\nu)\\right)^{\\frac{2}{1 + \\nu}} }\n\\cdot\n\\bigl( \\frac{1 + \\nu}{2 + \\nu} \\bigr)^{\\frac{1 - \\nu}{2 + \\nu}}, \\;\n\\nu \\in [0, 1].\n\\ea\n\\end{equation}\n","start":51894,"end":52346}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","labels":["UniformlyConvexConsequence","UniformlyConvexGradientBound"],"member_ids":["S030","S031"],"declared_external":false,"source_only":false,"title":"次梯度控制函数残差","source_units":[{"raw_tex":"\\begin{lemma} \\label{UniformlyConvexConsequence}\n\tLet $f$ be uniformly convex of degree $p \\geq 2$ with\n\tconstant $\\sigma > 0$ on set $\\dom F$. Then, for every $x\n\t\\in \\dom F$ and for all $F'(x) \\in \\partial F(x)$ we have\n\t\\begin{equation} \\label{UniformlyConvexGradientBound}\n\t\\ba{rcl}\n\tF(x) - F^{*} & \\; \\leq \\; & \\frac{p - 1}{p} \\left( \\frac{1}{\\sigma}\n\t\\right)^{\\frac{1}{p - 1}} \\|F'(x)\\|_{*}^{\\frac{p}{p - 1}}.\n\t\\ea\n\t\\end{equation}\n\\end{lemma}","start":11630,"end":12078},{"raw_tex":"\\begin{proof}\n\t\\noindent\n\tIn view of~\\eqref{UniformlyConvexLowerBound}, \n\tbound \\eqref{UniformlyConvexGradientBound}\n\tfollows as in the proof of Lemma~3 \n\tin~\\cite{nesterov2008accelerating}.\n\\qed\n\\end{proof}","start":12079,"end":12286}]},{"id":"S032","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S032"],"declared_external":true,"declaration_text":["原前提清单：Nesterov (2008), Lemma 3 proof argument invoked by B0012 for the residual bound; not reproduced in full。原文使用位置为第 327–330 行。"],"source_only":false,"title":"外部残差界证明论证","source_units":[]},{"id":"S033","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"误收录的例证完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\tDenote $\\kappa(x) \\equiv \\sum_{i = 1}^m e^{\\langle a_i, x \\rangle}$.\n\tLet us fix arbitrary $x, y \\in \\E$ and direction $h \\in \\E$. Then,\n\tstraightforward computation gives:\n\t$$\n\t\\ba{rcl}\n\t\\langle \\nabla f(x), h \\rangle &\\; = \\;& \\frac{1}{\\kappa(x)}\n\t\\sum_{i = 1}^m e^{\\langle a_i, x \\rangle} \\langle a_i, h \\rangle, \\\\[6pt]\n\t\\langle \\nabla^2 f(x)h, h \\rangle &\\; =\\;&\n\t \\frac{1}{\\kappa(x)} \\sum_{i = 1}^m e^{\\langle a_i, x \\rangle}\n\t\\langle\n\ta_i, h \\rangle^2 - \\bigl( \\frac{1}{\\kappa(x)} \\sum_{i = 1}^m\n\te^{\\langle a_i, x \\rangle} \\langle a_i, h \\rangle  \\bigr)^2 \\\\[5pt]\n\t&\\; = \\;& \\frac{1}{\\kappa(x)} \\sum_{i = 1}^m e^{\\langle a_i, x \\rangle}\n\t\\left( \\langle\n\ta_i, h \\rangle - \\langle \\nabla f(x), h \\rangle\n\t\\right)^2 \\; \\geq \\; 0.\n\t\\ea\n\t$$\n\tHence, we get\n\t$$\n\t\\ba{rcl}\n\t\\| \\nabla^2 f(x) \\| & = & \\max\\limits_{\\|h\\| \\leq 1} \\langle \\nabla^2 f(x)\n\th, h \\rangle \\;\n\t\\leq \\;  \\max\\limits_{\\|h\\| \\leq 1} \\sum_{i = 1}^m \\langle a_i, h \\rangle^2\n\t\\;\n\t= \\; \\max\\limits_{\\|h\\| \\leq 1} \\|h\\|^2 \\; = \\; 1.\n\t\\ea\n\t$$\n\tSince all Hessians of function $f$ are positive definite,\n\twe conclude that $\\Hf(0) \\leq 1$. Inequality $\\Hf(1) \\leq\n\t2$ can be easily obtained from the following\n\trepresentation of the third derivative:\n\t$$\n\t\\ba{rcl}\n\tf'''(x)[h,h,h]  & \\; = \\; & {1 \\over \\kappa(x)}\n\t\\sum\\limits_{i=1}^m e^{\\la a_i, x \\ra} \\left( \\la a_i, h\n\t\\ra - \\la \\nabla f(x), h \\ra \\right)^3\\\\\n\t\\\\\n\t& \\;  \\leq \\; & \\la \\nabla^2 f(x) h, h \\ra \\max\\limits_{1 \\leq\n\t\ti,j \\leq m } \\la a_i - a_j , h \\ra \\; \\leq \\; 2 \\| h \\|^3.\n\t\\ea\n\t$$\n\\qed\n\\end{proof}","start":16648,"end":18193}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"例证导数计算","source_units":[{"raw_tex":"\\begin{proof}\n\tDenote $\\kappa(x) \\equiv \\sum_{i = 1}^m e^{\\langle a_i, x \\rangle}$.\n\tLet us fix arbitrary $x, y \\in \\E$ and direction $h \\in \\E$. Then,\n\tstraightforward computation gives:\n\t$$\n\t\\ba{rcl}\n\t\\langle \\nabla f(x), h \\rangle &\\; = \\;& \\frac{1}{\\kappa(x)}\n\t\\sum_{i = 1}^m e^{\\langle a_i, x \\rangle} \\langle a_i, h \\rangle, \\\\[6pt]\n\t\\langle \\nabla^2 f(x)h, h \\rangle &\\; =\\;&\n\t \\frac{1}{\\kappa(x)} \\sum_{i = 1}^m e^{\\langle a_i, x \\rangle}\n\t\\langle\n\ta_i, h \\rangle^2 - \\bigl( \\frac{1}{\\kappa(x)} \\sum_{i = 1}^m\n\te^{\\langle a_i, x \\rangle} \\langle a_i, h \\rangle  \\bigr)^2 \\\\[5pt]\n\t&\\; = \\;& \\frac{1}{\\kappa(x)} \\sum_{i = 1}^m e^{\\langle a_i, x \\rangle}\n\t\\left( \\langle\n\ta_i, h \\rangle - \\langle \\nabla f(x), h \\rangle\n\t\\right)^2 \\; \\geq \\; 0.\n\t\\ea\n\t$$\n","start":16648,"end":17407}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"例证零次光滑界","source_units":[{"raw_tex":"\tHence, we get\n\t$$\n\t\\ba{rcl}\n\t\\| \\nabla^2 f(x) \\| & = & \\max\\limits_{\\|h\\| \\leq 1} \\langle \\nabla^2 f(x)\n\th, h \\rangle \\;\n\t\\leq \\;  \\max\\limits_{\\|h\\| \\leq 1} \\sum_{i = 1}^m \\langle a_i, h \\rangle^2\n\t\\;\n\t= \\; \\max\\limits_{\\|h\\| \\leq 1} \\|h\\|^2 \\; = \\; 1.\n\t\\ea\n\t$$\n\tSince all Hessians of function $f$ are positive definite,\n\twe conclude that $\\Hf(0) \\leq 1$. Inequality $\\Hf(1) \\leq\n\t2$ can be easily obtained from the following\n","start":17407,"end":17835}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"例证一阶光滑界","source_units":[{"raw_tex":"\trepresentation of the third derivative:\n\t$$\n\t\\ba{rcl}\n\tf'''(x)[h,h,h]  & \\; = \\; & {1 \\over \\kappa(x)}\n\t\\sum\\limits_{i=1}^m e^{\\la a_i, x \\ra} \\left( \\la a_i, h\n\t\\ra - \\la \\nabla f(x), h \\ra \\right)^3\\\\\n\t\\\\\n\t& \\;  \\leq \\; & \\la \\nabla^2 f(x) h, h \\ra \\max\\limits_{1 \\leq\n\t\ti,j \\leq m } \\la a_i - a_j , h \\ra \\; \\leq \\; 2 \\| h \\|^3.\n\t\\ea\n\t$$\n\\qed\n\\end{proof}","start":17835,"end":18193}]},{"id":"S037","semantic_type":"lemma","kind":"lemma","labels":["LemmaStepHolder","OneStepProgressHolder"],"member_ids":["S037","S038","S040","S041","S042","S043","S044","S045"],"declared_external":false,"source_only":false,"title":"三次模型的双分支进展","source_units":[{"raw_tex":"\\begin{lemma} \\label{LemmaStepHolder}\n\tLet $0 < \\Hfnu < +\\infty$ and $\\sigmafnu > 0$ for some\n\t$\\nu \\in [0,1]$. Then, for arbitrary $x \\in \\dom F$ and $H\n\t> 0$ we have:\n\t\\beq \\label{OneStepProgressHolder}\n\t\\ba{cl}\n\t& F(x) - M_H^{*}(x)  \\\\[5pt] \n\t& \\quad \\geq \\; \\min\\Bigl[ \\left( F(x) -\n\tF^{*} \\right) \\cdot \\frac{(1 + \\nu)}{(2 + \\nu)} \\cdot\n\t\\min\\bigl\\{\\bigl( \\frac{(1 + \\nu) \\gammafnu}{2(2 + \\nu)}  \\bigr)^{\\frac{1}{1 + \\nu}}, \n\t\\; 1\\bigr\\} ,   \\\\[5pt]\n\t& \\; \\qquad \\qquad \\left( F(T_H(x)) - F^{*} \\right)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}}\n\t\\cdot \\bigl ( \\frac{2 + \\nu}{1 + \\nu}  \\bigr)^{\\frac{3(1 + \\nu)}{2(2\n\t\t\t+ \\nu)}} \\cdot \\frac{(\\sigmafnu)^{\\frac{3}{2(2 + \\nu)}}}{ 3 \\sqrt{H}\n\t}  \\;  \\Bigr]. \n\t\\ea\n\t\\eeq\n\\end{lemma}","start":48378,"end":49099},{"raw_tex":"\\begin{proof}\n\tLet us consider two cases. 1) \n\t$H r_H^{1 - \\nu}(x) \\leq \\frac{2 \\Hfnu}{1 + \\nu}$.\n\t\tThen, for arbitrary $y \\in \\dom F$, we have:\n\t\t$$\n\t\t\\ba{rcl}\n\t\tM_H^{*}(x) \n\t\t& := &  Q(x; T_H(x)) +\n\t\t\\frac{H}{6}\\|T_H(x) - x\\|^3 + h(T_H(x)) \\\\[7pt]\n\t\t& \\leq &  Q(x; y) + \\frac{H r_H^{1 - \\nu}(x) \\|y -\n\t\t\tx\\|^{2 + \\nu}}{2 (2 + \\nu) } + h(y) \\\\[5pt]\n\t\t& \\overset{\\eqref{HolderHessianFuncBound}}{\\leq} &  \n\t\tF(y) +\n\t\t\\frac{\\Hfnu \\|y - x\\|^{2 + \\nu}}{(1 + \\nu)(2 + \\nu)} +\n\t\t\\frac{H r_H^{1 - \\nu}(x) \\|y - x\\|^{2 + \\nu}}{2 (2 + \\nu) }\n\t\t\\\\[7pt]\n\t\t& \\leq &\n\t\t F(y) +  \\frac{2\\Hfnu \\|y - x\\|^{2 +\n\t\t\\nu}}{(1 + \\nu)(2 + \\nu)},\n\t\t\\ea\n\t\t$$\n\t\twhere the first inequality follows from the fact, that\n\t\t$$\n\t\t\\ba{rcl}\n\t\tT_H(x) & = & \\argmin\\limits_{y \\in \\dom F} \\bigl\\{ Q(x; y)\n\t\t+ \\frac{H r_H^{1 - \\nu}(x) \\|y -\n\t\t\tx\\|^{2 + \\nu}}{2 (2 + \\nu) } + h(y) \\bigr\\}.\n\t\t\\ea\n\t\t$$\n\t\tLet us restrict $y$ to the segment: $y = \\alpha x^{*} + (1 -\n\t\t\\alpha) x,$ with $\\alpha \\in [0, 1]$. Taking into account the\n\t\tuniform convexity, we get:\n\t\t$$\n\t\t\\ba{rcl}\n\t\tM_H^{*}(x) & \\leq &  F(x) - \\alpha \\left( F(x) -\n\t\tF^{*} \\right) + \\alpha^{2 + \\nu} \\frac{2\\Hfnu \\|x^{*} - x\\|^{2 +\n\t\t\t\t\\nu}} {(1 + \\nu)(2 + \\nu)} \\quad \\\\[5pt]\n\t\t& \\overset{\\eqref{UniformlyConvexLowerBound}}{\\leq} &  \n\t\tF(x)\n\t\t-  \\Bigl(\\alpha  - \\alpha^{2 + \\nu} \\frac{2\\Hfnu}{  (1 + \\nu)\n\t\t\t\\sigmafnu}\n\t\t\\Bigr) \\left( F(x) - F^{*} \\right).\n\t\t\\ea\n\t\t$$\n\t\tThe minimum of the right-hand side is attained at\n\t\t$\n\t\t\\alpha^{*} = \\min\\bigl\\{\\frac{(1 + \\nu)\\gammafnu}{2(2 + \\nu)}, 1 \\bigr\\}^{\\frac{1}{1 + \\nu}}.\n\t\t$\n\t\tPlugging this value into the bound, we have:\n\t\t\\begin{displaymath}\n\t\t\\ba{rcl}\n\t\tM^{*}_H(x) & \\leq & F(x)  -   \\min\\bigl\\{ \\bigl(\n\t\t\\frac{(1 + \\nu)\\gammafnu}{2(2 + \\nu)}   \\bigr)^{1 / (1 + \\nu)}\n\t\t, \\; 1 \\bigr\\} \\cdot \\frac{(1 + \\nu)}{(2 + \\nu)} \\cdot \\left(\n\t\tF(x) - F^{*} \\right),\n\t\t\\ea\n\t\t\\end{displaymath}\n\t\tand this is the first argument of the minimum in\n\t\t\\eqref{OneStepProgressHolder}.\n\t\t\n\t\t2) $H r_H^{1 - \\nu}(x) > \\frac{2 \\Hfnu}{1 + \\nu}.$\n\t\tBy~\\eqref{CubicGradNewPoint}, we have the bound:\n\t\t\\begin{equation} \\label{NextPointGradBound2}\n\t\t\\ba{rcl}\n\t\t\\| F'(T_H(x)) \\|_{*} & \\; < \\; & Hr_H^2(x).\n\t\t\\ea\n\t\t\\end{equation}\n\t\tUsing the fact that $\\nabla^2 f(x) \\succeq 0$, we get the second argument of the\n\t\tminimum:\n\t$$\n\t\\ba{rcl}\n\tF(x) - M_H^{*}(x)\n\t& \\overset{\\eqref{CubicModelMinimumExpression}}{\\geq} &\n\t\\frac{Hr^3_H(x)}{3} \\quad  \\overset{ \\eqref{NextPointGradBound2}\n\t}{\\geq} \\quad \\frac{\\| F'(T_H(x)) \\|_{*}^{\\frac32}}{3 \\sqrt{H}}\n\t\\\\[5pt]\n\t& \\overset{\\eqref{UniformlyConvexGradientBound}}{\\geq} &\n\t \\left(\n\t\\frac{2 + \\nu}{1 + \\nu}  \\right)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}}\n\t\\cdot \\frac{(\\sigmafnu)^{\\frac{3}{2(2 + \\nu)}}}{ 3 \\sqrt{H} } \\cdot\n\t\\left( F(T_H(x)) - F^{*} \\right)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}}.\n\t\\ea\n\t$$\n\\qed\n\\end{proof}","start":49100,"end":51890},{"raw_tex":"\t\t$$\n\t\t\\ba{rcl}\n\t\tM_H^{*}(x) \n\t\t& := &  Q(x; T_H(x)) +\n\t\t\\frac{H}{6}\\|T_H(x) - x\\|^3 + h(T_H(x)) \\\\[7pt]\n\t\t& \\leq &  Q(x; y) + \\frac{H r_H^{1 - \\nu}(x) \\|y -\n\t\t\tx\\|^{2 + \\nu}}{2 (2 + \\nu) } + h(y) \\\\[5pt]\n\t\t& \\overset{\\eqref{HolderHessianFuncBound}}{\\leq} &  \n\t\tF(y) +\n\t\t\\frac{\\Hfnu \\|y - x\\|^{2 + \\nu}}{(1 + \\nu)(2 + \\nu)} +\n\t\t\\frac{H r_H^{1 - \\nu}(x) \\|y - x\\|^{2 + \\nu}}{2 (2 + \\nu) }\n\t\t\\\\[7pt]\n\t\t& \\leq &\n\t\t F(y) +  \\frac{2\\Hfnu \\|y - x\\|^{2 +\n\t\t\\nu}}{(1 + \\nu)(2 + \\nu)},\n\t\t\\ea\n\t\t$$\n","start":49245,"end":49733},{"raw_tex":"\t\twhere the first inequality follows from the fact, that\n\t\t$$\n\t\t\\ba{rcl}\n\t\tT_H(x) & = & \\argmin\\limits_{y \\in \\dom F} \\bigl\\{ Q(x; y)\n\t\t+ \\frac{H r_H^{1 - \\nu}(x) \\|y -\n\t\t\tx\\|^{2 + \\nu}}{2 (2 + \\nu) } + h(y) \\bigr\\}.\n\t\t\\ea\n\t\t$$\n","start":49733,"end":49961},{"raw_tex":"\t\tLet us restrict $y$ to the segment: $y = \\alpha x^{*} + (1 -\n\t\t\\alpha) x,$ with $\\alpha \\in [0, 1]$. Taking into account the\n\t\tuniform convexity, we get:\n\t\t$$\n\t\t\\ba{rcl}\n\t\tM_H^{*}(x) & \\leq &  F(x) - \\alpha \\left( F(x) -\n\t\tF^{*} \\right) + \\alpha^{2 + \\nu} \\frac{2\\Hfnu \\|x^{*} - x\\|^{2 +\n\t\t\t\t\\nu}} {(1 + \\nu)(2 + \\nu)} \\quad \\\\[5pt]\n\t\t& \\overset{\\eqref{UniformlyConvexLowerBound}}{\\leq} &  \n\t\tF(x)\n\t\t-  \\Bigl(\\alpha  - \\alpha^{2 + \\nu} \\frac{2\\Hfnu}{  (1 + \\nu)\n\t\t\t\\sigmafnu}\n\t\t\\Bigr) \\left( F(x) - F^{*} \\right).\n\t\t\\ea\n\t\t$$\n","start":49961,"end":50488},{"raw_tex":"\t\tThe minimum of the right-hand side is attained at\n\t\t$\n\t\t\\alpha^{*} = \\min\\bigl\\{\\frac{(1 + \\nu)\\gammafnu}{2(2 + \\nu)}, 1 \\bigr\\}^{\\frac{1}{1 + \\nu}}.\n\t\t$\n\t\tPlugging this value into the bound, we have:\n\t\t\\begin{displaymath}\n\t\t\\ba{rcl}\n\t\tM^{*}_H(x) & \\leq & F(x)  -   \\min\\bigl\\{ \\bigl(\n\t\t\\frac{(1 + \\nu)\\gammafnu}{2(2 + \\nu)}   \\bigr)^{1 / (1 + \\nu)}\n\t\t, \\; 1 \\bigr\\} \\cdot \\frac{(1 + \\nu)}{(2 + \\nu)} \\cdot \\left(\n\t\tF(x) - F^{*} \\right),\n\t\t\\ea\n\t\t\\end{displaymath}\n\t\tand this is the first argument of the minimum in\n\t\t\\eqref{OneStepProgressHolder}.\n\t\t\n","start":50488,"end":51041},{"raw_tex":"\t\t2) $H r_H^{1 - \\nu}(x) > \\frac{2 \\Hfnu}{1 + \\nu}.$\n\t\tBy~\\eqref{CubicGradNewPoint}, we have the bound:\n\t\t\\begin{equation} \\label{NextPointGradBound2}\n\t\t\\ba{rcl}\n\t\t\\| F'(T_H(x)) \\|_{*} & \\; < \\; & Hr_H^2(x).\n\t\t\\ea\n\t\t\\end{equation}\n","start":51041,"end":51272},{"raw_tex":"\t\tUsing the fact that $\\nabla^2 f(x) \\succeq 0$, we get the second argument of the\n\t\tminimum:\n\t$$\n\t\\ba{rcl}\n\tF(x) - M_H^{*}(x)\n\t& \\overset{\\eqref{CubicModelMinimumExpression}}{\\geq} &\n\t\\frac{Hr^3_H(x)}{3} \\quad  \\overset{ \\eqref{NextPointGradBound2}\n\t}{\\geq} \\quad \\frac{\\| F'(T_H(x)) \\|_{*}^{\\frac32}}{3 \\sqrt{H}}\n\t\\\\[5pt]\n\t& \\overset{\\eqref{UniformlyConvexGradientBound}}{\\geq} &\n\t \\left(\n\t\\frac{2 + \\nu}{1 + \\nu}  \\right)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}}\n\t\\cdot \\frac{(\\sigmafnu)^{\\frac{3}{2(2 + \\nu)}}}{ 3 \\sqrt{H} } \\cdot\n\t\\left( F(T_H(x)) - F^{*} \\right)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}}.\n\t\\ea\n\t$$\n\\qed\n\\end{proof}","start":51272,"end":51890}]},{"id":"S039","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"小参数分支条件","source_units":[{"raw_tex":"\\begin{proof}\n\tLet us consider two cases. 1) \n\t$H r_H^{1 - \\nu}(x) \\leq \\frac{2 \\Hfnu}{1 + \\nu}$.\n\t\tThen, for arbitrary $y \\in \\dom F$, we have:\n","start":49100,"end":49245}]},{"id":"S046","semantic_type":"lemma","kind":"lemma","labels":["LemmaUpHolder","LemmaUpHolder3","LemmaUpHolderCondition"],"member_ids":["S046","S048","S049","S050","S051","S052"],"declared_external":false,"source_only":false,"title":"失败试探后的参数界","source_units":[{"raw_tex":"\\begin{lemma} \\label{LemmaUpHolder} Assume that for a fixed\n\t$x \\in \\dom F$ the parameter $H > 0$ is such that:\n\t\\begin{equation} \\label{LemmaUpHolderCondition}\n\t\\ba{rcl}\n\tF(T_H(x)) & \\; > \\; & M^{*}_H(x).\n\t\\ea\n\t\\end{equation}\n\tIf for some $\\nu \\in [0, 1]$, we have $\\sigmafnu > 0$, then it holds:\n\t\\begin{equation} \\label{LemmaUpHolder3}\n\t\\ba{rcl}\n\tH \\left( F(T_{2H}(x)) - F^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}}\n\t& \\; < \\; & \\kappa_f(\\nu).\n\t\\ea\n\t\\end{equation}\n\\end{lemma}","start":52438,"end":52912},{"raw_tex":"\\begin{proof}\n\tFirstly, let us prove that from~\\eqref{LemmaUpHolderCondition} we\n\thave:\n\t\\begin{equation} \\label{LemmaUpHolderBound1}\n\t\\ba{rcl}\n\tH r^{1 - \\nu}_H(x) & \\; < \\; & \\frac{6 \\Hfnu}{(1 + \\nu)(2 + \\nu)}.\n\t\\ea\n\t\\end{equation}\n\tAssuming by contradiction, $H r^{1 - \\nu}_H(x) \\; \\geq \\; \\frac{6\n\t\t\\Hfnu}{(1 + \\nu)(2 + \\nu)}$, we get:\n\t$$\n\t\\ba{rcl}\n\tM_H^{*}(x)  & := & \\frac{H \\|T_H(x) - x\\|^3}{6} + Q(x;\n\tT_H(x)) + h(T_H(x)) \\\\[5pt]\n\t& \\geq & \\frac{\\Hfnu \\|T_H(x) - x\\|^{2 + \\nu}}{(1 + \\nu)(2 +\n\t\t\\nu)} + Q(x; T_H(x)) + h(T_H(x)) \\\\[5pt]\n\t& \\overset{\\eqref{HolderHessianFuncBound}}{\\geq} & F(T_H(x)),\n\t\\ea\n\t$$\n\twhich contradicts~\\eqref{LemmaUpHolderCondition}.\n","start":52913,"end":53579},{"raw_tex":"\\begin{proof}\n\tFirstly, let us prove that from~\\eqref{LemmaUpHolderCondition} we\n\thave:\n\t\\begin{equation} \\label{LemmaUpHolderBound1}\n\t\\ba{rcl}\n\tH r^{1 - \\nu}_H(x) & \\; < \\; & \\frac{6 \\Hfnu}{(1 + \\nu)(2 + \\nu)}.\n\t\\ea\n\t\\end{equation}\n\tAssuming by contradiction, $H r^{1 - \\nu}_H(x) \\; \\geq \\; \\frac{6\n\t\t\\Hfnu}{(1 + \\nu)(2 + \\nu)}$, we get:\n\t$$\n\t\\ba{rcl}\n\tM_H^{*}(x)  & := & \\frac{H \\|T_H(x) - x\\|^3}{6} + Q(x;\n\tT_H(x)) + h(T_H(x)) \\\\[5pt]\n\t& \\geq & \\frac{\\Hfnu \\|T_H(x) - x\\|^{2 + \\nu}}{(1 + \\nu)(2 +\n\t\t\\nu)} + Q(x; T_H(x)) + h(T_H(x)) \\\\[5pt]\n\t& \\overset{\\eqref{HolderHessianFuncBound}}{\\geq} & F(T_H(x)),\n\t\\ea\n\t$$\n\twhich contradicts~\\eqref{LemmaUpHolderCondition}.\n\tSecondly, by its definition, $M^*_H(x)$ is a concave function of $H$.\n\tTherefore, its derivative\n\t$\n\t{d \\over d H} M^*_H(x)  =  {1 \\over 6} r_H^3(x)\n\t$\n\tis non-increasing. Hence, it holds:\n\t\\beq \\label{LemmaUpHolderBound2}\n\t\\ba{rcl}\n\tr_{2H}(x) & \\leq & r_H(x)\n\t\\overset{\\eqref{LemmaUpHolderBound1}}{<} \n\t\\bigl( \\frac{6 \\Hfnu}{(1 + \\nu)(2 + \\nu) H} \\bigr)^{\\frac{1}{1 - \\nu}}.\n\t\\ea\n\t\\eeq\n\tFinally, by the smoothness and the uniform convexity, we obtain:\n\t$$\n\t\\ba{cl}\n\t& H \\left( F(T_{2H}(x)) - F^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}}\n\t\\;\\; \\refLE{UniformlyConvexGradientBound} \\;\\;\n\tH \\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\| F'(T_{2H}(x)) \\|_{*}^{\\frac{1 - \\nu}{1 + \\nu}}  \\\\[5pt]\n\t& \\quad \\refLE{CubicGradNewPoint} \n\tH \\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\Bigl(\n\tr^{1 +\\nu}_{2H}(x) \\cdot \\left(  \\frac{\\Hfnu}{1 + \\nu} + H r^{1 -\n\t\t\\nu}_{2H}(x) \\right)\n\t\\Bigr)^{\\frac{1 - \\nu}{1 + \\nu}} \\\\[5pt]\n\t& \\quad \\overset{\\eqref{LemmaUpHolderBound2}}{<} \n\tH \\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\Bigl( \n\tr^{1+\\nu}_{2H}(x) \\cdot \\frac{(8 + \\nu) \\Hfnu}{(1 + \\nu)(2 + \\nu)}\n\t\\Bigr)^{\\frac{1 - \\nu}{1 + \\nu}} \\\\[5pt]\n\t& \\quad \\overset{\\eqref{LemmaUpHolderBound2}}{<} \n\t\t\\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\left( \\frac{\\Hfnu}{(1 + \\nu)(2 + \\nu)}  \\right)^{\\frac{2}{1 + \\nu}}\n\t6  (8 + \\nu)^{\\frac{1 - \\nu}{1 + \\nu}}\n\t\\; =: \\; \\kappa_f(\\nu).\n\t\\ea\n\t$$\n\\qed\n\\end{proof}","start":52913,"end":55281},{"raw_tex":"\tSecondly, by its definition, $M^*_H(x)$ is a concave function of $H$.\n\tTherefore, its derivative\n\t$\n\t{d \\over d H} M^*_H(x)  =  {1 \\over 6} r_H^3(x)\n\t$\n\tis non-increasing. Hence, it holds:\n","start":53579,"end":53769},{"raw_tex":"\t\\beq \\label{LemmaUpHolderBound2}\n\t\\ba{rcl}\n\tr_{2H}(x) & \\leq & r_H(x)\n\t\\overset{\\eqref{LemmaUpHolderBound1}}{<} \n\t\\bigl( \\frac{6 \\Hfnu}{(1 + \\nu)(2 + \\nu) H} \\bigr)^{\\frac{1}{1 - \\nu}}.\n\t\\ea\n\t\\eeq\n","start":53769,"end":53967},{"raw_tex":"\tFinally, by the smoothness and the uniform convexity, we obtain:\n\t$$\n\t\\ba{cl}\n\t& H \\left( F(T_{2H}(x)) - F^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}}\n\t\\;\\; \\refLE{UniformlyConvexGradientBound} \\;\\;\n\tH \\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\| F'(T_{2H}(x)) \\|_{*}^{\\frac{1 - \\nu}{1 + \\nu}}  \\\\[5pt]\n\t& \\quad \\refLE{CubicGradNewPoint} \n\tH \\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\Bigl(\n\tr^{1 +\\nu}_{2H}(x) \\cdot \\left(  \\frac{\\Hfnu}{1 + \\nu} + H r^{1 -\n\t\t\\nu}_{2H}(x) \\right)\n\t\\Bigr)^{\\frac{1 - \\nu}{1 + \\nu}} \\\\[5pt]\n\t& \\quad \\overset{\\eqref{LemmaUpHolderBound2}}{<} \n\tH \\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\Bigl( \n\tr^{1+\\nu}_{2H}(x) \\cdot \\frac{(8 + \\nu) \\Hfnu}{(1 + \\nu)(2 + \\nu)}\n\t\\Bigr)^{\\frac{1 - \\nu}{1 + \\nu}} \\\\[5pt]\n\t& \\quad \\overset{\\eqref{LemmaUpHolderBound2}}{<} \n\t\t\\left(\n\t\\frac{1 + \\nu}{2 + \\nu} \\bigl( \\frac{1}{\\sigmafnu}\n\t\\bigr)^{\\frac{1}{1 + \\nu}}  \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t\\left( \\frac{\\Hfnu}{(1 + \\nu)(2 + \\nu)}  \\right)^{\\frac{2}{1 + \\nu}}\n\t6  (8 + \\nu)^{\\frac{1 - \\nu}{1 + \\nu}}\n\t\\; =: \\; \\kappa_f(\\nu).\n\t\\ea\n\t$$\n\\qed\n\\end{proof}","start":53967,"end":55281}]},{"id":"S047","semantic_type":"assumption","kind":"assumption","labels":["LemmaUpHolder","LemmaUpHolderCondition"],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"参数引理的失败条件","source_units":[{"raw_tex":"\\begin{lemma} \\label{LemmaUpHolder} Assume that for a fixed\n\t$x \\in \\dom F$ the parameter $H > 0$ is such that:\n\t\\begin{equation} \\label{LemmaUpHolderCondition}\n\t\\ba{rcl}\n\tF(T_H(x)) & \\; > \\; & M^{*}_H(x).\n\t\\ea\n\t\\end{equation}\n\tIf for some $\\nu \\in [0, 1]$, we have $\\sigmafnu > 0$, then it holds:\n","start":52438,"end":52736}]},{"id":"S053","semantic_type":"assumption","kind":"assumption","labels":["TheoremLinearRateUniformly"],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"主定理非退化条件","source_units":[{"raw_tex":"\\begin{theorem} \\label{TheoremLinearRateUniformly} Assume that for a\n\tfixed $\\nu \\in [0, 1]$ we have $0 < \\Hfnu < +\\infty$ and\n\t$\\sigmafnu > 0$. Let parameter $H_0$ in Algorithm 1 be\n\tsmall enough:\n","start":55337,"end":55535}]},{"id":"S054","semantic_type":"assumption","kind":"assumption","labels":["HolderChoosingH","TheoremLinearRateUniformly"],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"初始正则参数限制","source_units":[{"raw_tex":"\t\\begin{equation} \\label{HolderChoosingH}\n\t\\ba{rcl}\n\tH_0 & \\; \\leq \\; & \\frac{ \\kappa_f(\\nu) }{ \\left( F(x_0) - F^{*}\n\t\t\\right)^{(1 - \\nu) / (2 + \\nu)  } },\n\t\\ea\n\t\\end{equation}\n\twhere $\\kappa_f(\\nu)$ is defined\n\tby~\\eqref{KappaDefinition}. Let the sequence $\\{ x_k \\}_{k\n\t\t= 0}^K$ generated by the method satisfy condition:\n","start":55535,"end":55860}]},{"id":"S055","semantic_type":"assumption","kind":"assumption","labels":["HolderConditionFuncEps","TheoremLinearRateUniformly"],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"所有试探点尚未达标","source_units":[{"raw_tex":"\t\\begin{equation} \\label{HolderConditionFuncEps}\n\tF(T_{H_k2^{j}}(x_k)) - F^{*} \\; \\geq \\; \\varepsilon \\; > \\; 0, \\qquad\n\t\\quad 0 \\leq j \\leq i_k, \\quad 0 \\leq k \\leq K - 1.\n\t\\end{equation}\n","start":55860,"end":56049}]},{"id":"S056","semantic_type":"theorem","kind":"theorem","labels":["HolderChoosingH","HolderConditionFuncEps","HolderMainResult","HolderOracleCallsBound","TheoremLinearRateUniformly"],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"自适应三次牛顿全局线性界","source_units":[{"raw_tex":"\\begin{theorem} \\label{TheoremLinearRateUniformly} Assume that for a\n\tfixed $\\nu \\in [0, 1]$ we have $0 < \\Hfnu < +\\infty$ and\n\t$\\sigmafnu > 0$. Let parameter $H_0$ in Algorithm 1 be\n\tsmall enough:\n\t\\begin{equation} \\label{HolderChoosingH}\n\t\\ba{rcl}\n\tH_0 & \\; \\leq \\; & \\frac{ \\kappa_f(\\nu) }{ \\left( F(x_0) - F^{*}\n\t\t\\right)^{(1 - \\nu) / (2 + \\nu)  } },\n\t\\ea\n\t\\end{equation}\n\twhere $\\kappa_f(\\nu)$ is defined\n\tby~\\eqref{KappaDefinition}. Let the sequence $\\{ x_k \\}_{k\n\t\t= 0}^K$ generated by the method satisfy condition:\n\t\\begin{equation} \\label{HolderConditionFuncEps}\n\tF(T_{H_k2^{j}}(x_k)) - F^{*} \\; \\geq \\; \\varepsilon \\; > \\; 0, \\qquad\n\t\\quad 0 \\leq j \\leq i_k, \\quad 0 \\leq k \\leq K - 1.\n\t\\end{equation}\n\tThen, for every $0 \\leq k \\leq K - 1$, we have:\n\t\\beq \\label{HolderMainResult}\n\t\\ba{cl}\n\t& F(x_{k + 1}) - F^{*}  \\leq \\\\[5pt]\n\t& \\bigl( 1 - \\min\\bigl\\{  \\frac{ (2 + \\nu)\\left( (1 + \\nu) (2\n\t\t+ \\nu) \\right)^{ 1 / (1 + \\nu) }  \n\t    \\left( \\gammafnu  \\right)^{\\frac{1}{1 +\n\t    \t\t\\nu}}   }{ (1 + \\nu)6^{3/2} \\cdot 2^{1/2} \\cdot (8\n\t\t+ \\nu)^{ (1 - \\nu) / (2 + 2 \\nu) } } , \\frac{1}{2} \\bigr\\}  \\bigr) \\cdot \n\t  \\left( F(x_{k}) - F^{*} \\right).\n\t\\ea\n\t\\eeq\n\tTherefore, the rate of convergence is linear, and\n\t\\begin{displaymath}\n\t\\ba{rcl}\n\tK & \\; \\leq \\; & \\max\\bigl\\{  \\left( \\gammafnu \\right)^{\\frac{-1}{1\n\t\t\t+ \\nu}} \\cdot \\frac{1 + \\nu}{2 + \\nu}  \\cdot \\frac{6^{3/2} \\cdot\n\t\t2^{1/2} \\cdot (8 + \\nu)^{(1 - \\nu) / (2 + 2\\nu)} }{ \\left( (1 +\n\t\t\\nu)(2 + \\nu) \\right)^{1 / (1 + \\nu)}  } , \\; 1 \\bigr\\} \\cdot \\log\n\t\\frac{ F(x_0) - F^{*} }{\\varepsilon}.\n\t\\ea\n\t\\end{displaymath}\n\tMoreover, we have the following bound for the total number of oracle\n\tcalls $N_K$ during the first $K$ iterations:\n\t\\begin{equation} \\label{HolderOracleCallsBound}\n\t\\ba{rcl}\n\tN_K & \\; \\leq \\; & 2K  +  \\log_2\n\t\\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1 - \\nu) / (2 + \\nu)}} -\n\t\\log_2 H_0.\n\t\\ea\n\t\\end{equation}\n\\end{theorem}","start":55337,"end":57236}]},{"id":"S057","semantic_type":"derived_fact","kind":"derived_fact","labels":["TheoremLinearRateUniformly"],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"作者所列迭代数结论","source_units":[{"raw_tex":"\tTherefore, the rate of convergence is linear, and\n\t\\begin{displaymath}\n\t\\ba{rcl}\n\tK & \\; \\leq \\; & \\max\\bigl\\{  \\left( \\gammafnu \\right)^{\\frac{-1}{1\n\t\t\t+ \\nu}} \\cdot \\frac{1 + \\nu}{2 + \\nu}  \\cdot \\frac{6^{3/2} \\cdot\n\t\t2^{1/2} \\cdot (8 + \\nu)^{(1 - \\nu) / (2 + 2\\nu)} }{ \\left( (1 +\n\t\t\\nu)(2 + \\nu) \\right)^{1 / (1 + \\nu)}  } , \\; 1 \\bigr\\} \\cdot \\log\n\t\\frac{ F(x_0) - F^{*} }{\\varepsilon}.\n\t\\ea\n\t\\end{displaymath}\n","start":56502,"end":56919}]},{"id":"S058","semantic_type":"proof","kind":"proof","labels":["HolderHtGoodBound","Th2HLowerBound","Th2UsefulBound"],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"主定理完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\tThe proof is based on Lemmas~\\ref{LemmaStepHolder}\n\tand~\\ref{LemmaUpHolder}, and monotonicity of the sequence\n\t$\\bigl\\{ F(x_k)  \\bigr\\}_{k \\geq 0}$.\n\t\tFirstly, we need to show that every iteration of the method is\n\twell-defined. Namely, we are going to verify that for a fixed $0 \\leq\n\tk \\leq K-1$, there exists a finite integer $\\ell \\geq 0$ such that\n\teither $F(T_{H_k 2^{\\ell}}(x_k) ) \\leq M_{H_k 2^{\\ell}}^{*}(x_k)$ or\n\t$F(T_{H_k 2^{\\ell + 1}}(x_k)) - F^{*} < \\varepsilon$.\n\t\tIndeed, let us set\n\t\\begin{equation} \\label{Th2HLowerBound}\n\t\\ba{rcl}\n\t\\ell := \\max\\left\\{0,  \\log_2 \\left \\lceil\n\t\\frac{\\kappa_f(\\nu)}{H_k \\varepsilon^{(1 - \\nu) / (2 + \\nu)} }\n\t\\right\\rceil  \\right\\}, \n\t& \\quad \\text{and} \\quad &\n\tH := H_k\n\t2^{\\ell}  \\geq  \\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1 - \\nu) / (2\n\t\t\t+ \\nu)}}.\n\t\\ea\n\t\\end{equation}\n\tThen, if we have both $F(T_{H}(x_k)) > M_{H}^{*}(x_k)$ and\n\t$F(T_{2H}(x_k)) - F^{*}\n\t\\geq \\varepsilon$, we get by Lemma \\ref{LemmaUpHolder}:\n\t\\begin{displaymath}\n\t\\ba{rcl}\n\tH & \\; \\overset{\\eqref{LemmaUpHolder3}}{<} \\; &\n\t\\frac{\\kappa_f(\\nu)}{\\left( F(T_{2H}(x_k)) - F^{*}  \\right)^{(1 - \\nu)\n\t\t\t/ (2 + \\nu)}  } \\; \\leq \\; \\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1\n\t\t\t- \\nu) / (2 + \\nu)}},\n\t\\ea\n\t\\end{displaymath}\n\twhich contradicts~\\eqref{Th2HLowerBound}. Therefore, if we are unable\n\tto find the value $0 \\leq i_k \\leq \\ell$ (see line~1 of\n\tAlgorithm) in a finite number of steps, that only means we have\n\talready solved the problem up to accuracy $\\varepsilon$.\n\t\n\t\n\t\n\tNow, let us show that for every $0 \\leq k \\leq K$ it holds:\n\t\\begin{equation} \\label{Th2UsefulBound}\n\t\\ba{rcl}\n\tH_k \\left( F(x_k) - F^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t& \\leq &\n\t\\max \\left\\{ \\kappa_f(\\nu), \\; H_0 \\left( F(x_0) - F^{*}\n\t\\right)^{\\frac{1 - \\nu}{2 + \\nu}} \\right\\}.\n\t\\ea\n\t\\end{equation}\n\tThis inequality is obviously valid for $k = 0$. Assume it\n\tis also valid for some $k \\geq 0$. Then, by definition of\n\t$H_{k + 1}$ (see line~3 of Algorithm), we have $H_{k +\n\t\t1} = H_k 2^{i_k - 1}$. \n\tThere are two cases.\n\t\t1) $i_k = 0$. Then, $H_{k + 1} < H_k$. By monotonicity of\n\t\t$\\bigl\\{ F(x_k) \\bigr\\}_{k \\geq 0}$ and by induction, we get:\n\t\t$$\n\t\t\\ba{rcl}\n\t\tH_{k + 1} \\left( F(x_{k + 1}) - F^{*} \\right)^{\\frac{1 - \\nu}{2 +\n\t\t\t\t\\nu}}  & \\; < \\; & H_k \\left( F(x_k) - F^{*} \\right)^{\\frac{1 -\n\t\t\t\t\\nu}{2 + \\nu}} \\\\[5pt]\n\t\t& \\; \\leq \\; & \n\t\t\\max \\left\\{ \\kappa_f(\\nu), \\; H_0 \\left( F(x_0) -\n\t\tF^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \\right\\}.\n\t\t\\ea\n\t\t$$\n\t\t\n\t\t2) $i_k > 0$. Then, applying Lemma~\\ref{LemmaUpHolder} with $H\n\t\t:= H_k 2^{i_k - 1} = H_{k + 1}$ and $x :=\n\t\tx_{k}$, we have:\n\t\t\\begin{displaymath}\n\t\t\\ba{rcl}\n\t\tH_{k + 1} \\left( F(x_{k + 1}) - F^{*} \\right)^{\\frac{1 - \\nu}{2 +\n\t\t\t\t\\nu}} & \\; = \\; & H \\left( F(T_{2H}(x)) - F^{*}\n\t\t\\right)^{\\frac{1 - \\nu}{2 + \\nu}} \\;\n\t\t\\overset{\\eqref{LemmaUpHolder3}}{\\leq} \\; \\kappa_f(\\nu).\n\t\t\\ea\n\t\t\\end{displaymath}\n\tThus, \\eqref{Th2UsefulBound} is true by induction.\n\tChoosing $H_0$ small enough~\\eqref{HolderChoosingH}, we have:\n\t\\begin{equation} \\label{HolderHtGoodBound}\n\t\\ba{rcl}\n\t2H_{k} \\left( F(x_k) - F^{*}\\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t& \\; \\leq \\; & \n\t2\\kappa_f(\\nu), \\qquad 0 \\leq k \\leq K.\n\t\\ea\n\t\\end{equation}\n\tFrom Lemma~\\ref{LemmaStepHolder} we know, that one of the two\n\tfollowing estimates is true (denote $\\delta_k := F(x_k) - F^{*}$):\n\t\n\t\n\t\\begin{enumerate}\n\t\t\\item[1)] $F(x_k) - F(x_{k + 1})  \\geq \\alpha \\cdot \\delta_k\n\t\t \\;  \\Leftrightarrow  \\;  \\delta_{k + 1}  \\leq (1 -\n\t\t\\alpha) \\cdot \\delta_k, \\; $ \\textit{or}\n\t\t\n\t\t\\item[2)] $F(x_k) - F(x_{k + 1})  \\geq \\beta \\cdot \\delta_{k +\n\t\t\t1}  \\; \\Leftrightarrow \\;  \\delta_{k + 1}\n\t \t\\leq (1 + \\beta)^{-1} \\delta_k  \\leq  (1 - \\min\\{\\beta,\n\t\t1\\} / 2) \\cdot \\delta_k$,\n\t\\end{enumerate}\n\twhere\n\t$\n\t\\alpha  \n\t := \n\t \\frac{1 + \\nu}{2 + \\nu} \\cdot\n\t\\min\\bigl\\{\\bigl(  \\frac{(1 + \\nu) \\gammafnu}{2(2 + \\nu)} \\bigr)^{\n\t\t\\frac{1}{1 + \\nu}}, \\; 1\\bigr\\},\n\t$\n\tand\n\t$$\n\t\\ba{rcl}\n\t\\beta \n\t & := & \n\t\\bigl( \\frac{2 + \\nu}{1 + \\nu}\n\t\\bigr)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}} \\cdot\n\t\\frac{(\\sigmafnu)^{\\frac{3}{2(2 + \\nu)}}}{3 (2 \\kappa_f(\\nu))^{1/2}\n\t}\n\t\\; \\refEQ{KappaDefinition} \\;\n\t\\frac{2 + \\nu}{1 + \\nu} \\cdot \\frac{ 2^{1/2} \\cdot\n\t\t\\left( (1 + \\nu) (2 + \\nu)  \\right)^{\\frac{1}{1 + \\nu}} }{6^{3/2}\n\t\t\\cdot (8 + \\nu)^{(1 - \\nu) / (2 + 2\\nu)}} \\cdot \\gammafnu^{\\frac{1}{1\n\t\t\t+ \\nu}}.\n\t\\ea\n\t$$\n\tIt remains to notice, that $\\alpha \\geq \\min \\bigl\\{\\beta, 1\\bigr\\} /\n\t2$. Thus, we obtain~\\eqref{HolderMainResult}.\n\t\n\tFinally, let us estimate the total number of the oracle calls $N_K$\n\tduring the first $K$ iterations. At each\n\titeration, the oracle is called $i_k + 1$ times, and we have $H_{k +\n\t\t1} = H_k 2^{i_k - 1}$. Therefore,\n\t$$\n\t\\ba{rcl}\n\tN_K & \\; = \\; & \\sum_{k = 0}^{K - 1} (i_k + 1) \\; = \\; \\sum_{k = 0}^{K -\n\t\t1} \\left( \\log_2 \\frac{H_{k + 1}}{H_k} + 2\\right) \\\\[5pt]\n\t& \\; = \\; & 2K + \\log_2\n\tH_{K} - \\log_2 H_0\n\t\\; \\overset{\\eqref{HolderHtGoodBound},\n\t\t\\eqref{HolderConditionFuncEps}}{\\leq} \\;\n\t2K + \\log_2\n\t\\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1 - \\nu) / (2 + \\nu)}} - \\log_2\n\tH_0.\n\t\\ea\n\t$$\n\\qed\n\\end{proof}","start":57237,"end":62279}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"主证明路线声明","source_units":[{"raw_tex":"\\begin{proof}\n\tThe proof is based on Lemmas~\\ref{LemmaStepHolder}\n\tand~\\ref{LemmaUpHolder}, and monotonicity of the sequence\n\t$\\bigl\\{ F(x_k)  \\bigr\\}_{k \\geq 0}$.\n","start":57237,"end":57401}]},{"id":"S060","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"接受迭代的目标单调性","source_units":[{"raw_tex":"\tThe proof is based on Lemmas~\\ref{LemmaStepHolder}\n\tand~\\ref{LemmaUpHolder}, and monotonicity of the sequence\n\t$\\bigl\\{ F(x_k)  \\bigr\\}_{k \\geq 0}$.\n","start":57251,"end":57401},{"raw_tex":"\t\t1) $i_k = 0$. Then, $H_{k + 1} < H_k$. By monotonicity of\n\t\t$\\bigl\\{ F(x_k) \\bigr\\}_{k \\geq 0}$ and by induction, we get:\n\t\t$$\n\t\t\\ba{rcl}\n\t\tH_{k + 1} \\left( F(x_{k + 1}) - F^{*} \\right)^{\\frac{1 - \\nu}{2 +\n\t\t\t\t\\nu}}  & \\; < \\; & H_k \\left( F(x_k) - F^{*} \\right)^{\\frac{1 -\n\t\t\t\t\\nu}{2 + \\nu}} \\\\[5pt]\n\t\t& \\; \\leq \\; & \n\t\t\\max \\left\\{ \\kappa_f(\\nu), \\; H_0 \\left( F(x_0) -\n\t\tF^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \\right\\}.\n\t\t\\ea\n\t\t$$\n","start":59271,"end":59708}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":["Th2HLowerBound"],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"构造有限试探阈值","source_units":[{"raw_tex":"\t\tFirstly, we need to show that every iteration of the method is\n\twell-defined. Namely, we are going to verify that for a fixed $0 \\leq\n\tk \\leq K-1$, there exists a finite integer $\\ell \\geq 0$ such that\n\teither $F(T_{H_k 2^{\\ell}}(x_k) ) \\leq M_{H_k 2^{\\ell}}^{*}(x_k)$ or\n\t$F(T_{H_k 2^{\\ell + 1}}(x_k)) - F^{*} < \\varepsilon$.\n\t\tIndeed, let us set\n\t\\begin{equation} \\label{Th2HLowerBound}\n\t\\ba{rcl}\n\t\\ell := \\max\\left\\{0,  \\log_2 \\left \\lceil\n\t\\frac{\\kappa_f(\\nu)}{H_k \\varepsilon^{(1 - \\nu) / (2 + \\nu)} }\n\t\\right\\rceil  \\right\\}, \n\t& \\quad \\text{and} \\quad &\n\tH := H_k\n\t2^{\\ell}  \\geq  \\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1 - \\nu) / (2\n\t\t\t+ \\nu)}}.\n\t\\ea\n\t\\end{equation}\n","start":57401,"end":58074}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"失败与未达标导致矛盾","source_units":[{"raw_tex":"\tThen, if we have both $F(T_{H}(x_k)) > M_{H}^{*}(x_k)$ and\n\t$F(T_{2H}(x_k)) - F^{*}\n\t\\geq \\varepsilon$, we get by Lemma \\ref{LemmaUpHolder}:\n\t\\begin{displaymath}\n\t\\ba{rcl}\n\tH & \\; \\overset{\\eqref{LemmaUpHolder3}}{<} \\; &\n\t\\frac{\\kappa_f(\\nu)}{\\left( F(T_{2H}(x_k)) - F^{*}  \\right)^{(1 - \\nu)\n\t\t\t/ (2 + \\nu)}  } \\; \\leq \\; \\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1\n\t\t\t- \\nu) / (2 + \\nu)}},\n\t\\ea\n\t\\end{displaymath}\n\twhich contradicts~\\eqref{Th2HLowerBound}. Therefore, if we are unable\n\tto find the value $0 \\leq i_k \\leq \\ell$ (see line~1 of\n\tAlgorithm) in a finite number of steps, that only means we have\n\talready solved the problem up to accuracy $\\varepsilon$.\n","start":58074,"end":58735}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":["Th2UsefulBound"],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"乘积不变量与归纳起点","source_units":[{"raw_tex":"\t\n\t\n\t\n\tNow, let us show that for every $0 \\leq k \\leq K$ it holds:\n\t\\begin{equation} \\label{Th2UsefulBound}\n\t\\ba{rcl}\n\tH_k \\left( F(x_k) - F^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t& \\leq &\n\t\\max \\left\\{ \\kappa_f(\\nu), \\; H_0 \\left( F(x_0) - F^{*}\n\t\\right)^{\\frac{1 - \\nu}{2 + \\nu}} \\right\\}.\n\t\\ea\n\t\\end{equation}\n\tThis inequality is obviously valid for $k = 0$. Assume it\n\tis also valid for some $k \\geq 0$. Then, by definition of\n\t$H_{k + 1}$ (see line~3 of Algorithm), we have $H_{k +\n\t\t1} = H_k 2^{i_k - 1}$. \n\tThere are two cases.\n","start":58735,"end":59271}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S064"],"declared_external":false,"source_only":false,"title":"直接接受时保持不变量","source_units":[{"raw_tex":"\t\t1) $i_k = 0$. Then, $H_{k + 1} < H_k$. By monotonicity of\n\t\t$\\bigl\\{ F(x_k) \\bigr\\}_{k \\geq 0}$ and by induction, we get:\n\t\t$$\n\t\t\\ba{rcl}\n\t\tH_{k + 1} \\left( F(x_{k + 1}) - F^{*} \\right)^{\\frac{1 - \\nu}{2 +\n\t\t\t\t\\nu}}  & \\; < \\; & H_k \\left( F(x_k) - F^{*} \\right)^{\\frac{1 -\n\t\t\t\t\\nu}{2 + \\nu}} \\\\[5pt]\n\t\t& \\; \\leq \\; & \n\t\t\\max \\left\\{ \\kappa_f(\\nu), \\; H_0 \\left( F(x_0) -\n\t\tF^{*} \\right)^{\\frac{1 - \\nu}{2 + \\nu}} \\right\\}.\n\t\t\\ea\n\t\t$$\n\t\t\n","start":59271,"end":59711}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S065"],"declared_external":false,"source_only":false,"title":"存在失败试探时控制参数","source_units":[{"raw_tex":"\t\t2) $i_k > 0$. Then, applying Lemma~\\ref{LemmaUpHolder} with $H\n\t\t:= H_k 2^{i_k - 1} = H_{k + 1}$ and $x :=\n\t\tx_{k}$, we have:\n\t\t\\begin{displaymath}\n\t\t\\ba{rcl}\n\t\tH_{k + 1} \\left( F(x_{k + 1}) - F^{*} \\right)^{\\frac{1 - \\nu}{2 +\n\t\t\t\t\\nu}} & \\; = \\; & H \\left( F(T_{2H}(x)) - F^{*}\n\t\t\\right)^{\\frac{1 - \\nu}{2 + \\nu}} \\;\n\t\t\\overset{\\eqref{LemmaUpHolder3}}{\\leq} \\; \\kappa_f(\\nu).\n\t\t\\ea\n\t\t\\end{displaymath}\n","start":59711,"end":60116}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":["HolderHtGoodBound"],"member_ids":["S066"],"declared_external":false,"source_only":false,"title":"闭合归纳并代入初始化","source_units":[{"raw_tex":"\tThus, \\eqref{Th2UsefulBound} is true by induction.\n\tChoosing $H_0$ small enough~\\eqref{HolderChoosingH}, we have:\n\t\\begin{equation} \\label{HolderHtGoodBound}\n\t\\ba{rcl}\n\t2H_{k} \\left( F(x_k) - F^{*}\\right)^{\\frac{1 - \\nu}{2 + \\nu}} \n\t& \\; \\leq \\; & \n\t2\\kappa_f(\\nu), \\qquad 0 \\leq k \\leq K.\n\t\\ea\n\t\\end{equation}\n","start":60116,"end":60428}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S067"],"declared_external":false,"source_only":false,"title":"将模型进展转为两种收缩","source_units":[{"raw_tex":"\tFrom Lemma~\\ref{LemmaStepHolder} we know, that one of the two\n\tfollowing estimates is true (denote $\\delta_k := F(x_k) - F^{*}$):\n\t\n\t\n\t\\begin{enumerate}\n","start":60428,"end":60582}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S068"],"declared_external":false,"source_only":false,"title":"当前残差分支的收缩","source_units":[{"raw_tex":"\t\t\\item[1)] $F(x_k) - F(x_{k + 1})  \\geq \\alpha \\cdot \\delta_k\n\t\t \\;  \\Leftrightarrow  \\;  \\delta_{k + 1}  \\leq (1 -\n\t\t\\alpha) \\cdot \\delta_k, \\; $ \\textit{or}\n\t\t\n","start":60582,"end":60745},{"raw_tex":"\twhere\n\t$\n\t\\alpha  \n\t := \n\t \\frac{1 + \\nu}{2 + \\nu} \\cdot\n\t\\min\\bigl\\{\\bigl(  \\frac{(1 + \\nu) \\gammafnu}{2(2 + \\nu)} \\bigr)^{\n\t\t\\frac{1}{1 + \\nu}}, \\; 1\\bigr\\},\n\t$\n","start":60959,"end":61123}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S069"],"declared_external":false,"source_only":false,"title":"下一点残差分支的收缩","source_units":[{"raw_tex":"\t\t\\item[2)] $F(x_k) - F(x_{k + 1})  \\geq \\beta \\cdot \\delta_{k +\n\t\t\t1}  \\; \\Leftrightarrow \\;  \\delta_{k + 1}\n\t \t\\leq (1 + \\beta)^{-1} \\delta_k  \\leq  (1 - \\min\\{\\beta,\n\t\t1\\} / 2) \\cdot \\delta_k$,\n\t\\end{enumerate}\n","start":60745,"end":60959}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S070"],"declared_external":false,"source_only":false,"title":"消去残差并化简系数","source_units":[{"raw_tex":"\tand\n\t$$\n\t\\ba{rcl}\n\t\\beta \n\t & := & \n\t\\bigl( \\frac{2 + \\nu}{1 + \\nu}\n\t\\bigr)^{\\frac{3(1 + \\nu)}{2(2 + \\nu)}} \\cdot\n\t\\frac{(\\sigmafnu)^{\\frac{3}{2(2 + \\nu)}}}{3 (2 \\kappa_f(\\nu))^{1/2}\n\t}\n\t\\; \\refEQ{KappaDefinition} \\;\n\t\\frac{2 + \\nu}{1 + \\nu} \\cdot \\frac{ 2^{1/2} \\cdot\n\t\t\\left( (1 + \\nu) (2 + \\nu)  \\right)^{\\frac{1}{1 + \\nu}} }{6^{3/2}\n\t\t\\cdot (8 + \\nu)^{(1 - \\nu) / (2 + 2\\nu)}} \\cdot \\gammafnu^{\\frac{1}{1\n\t\t\t+ \\nu}}.\n\t\\ea\n\t$$\n","start":61123,"end":61554}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S071"],"declared_external":false,"source_only":false,"title":"统一得到线性收缩","source_units":[{"raw_tex":"\tIt remains to notice, that $\\alpha \\geq \\min \\bigl\\{\\beta, 1\\bigr\\} /\n\t2$. Thus, we obtain~\\eqref{HolderMainResult}.\n\t\n","start":61554,"end":61674}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S072"],"declared_external":false,"source_only":false,"title":"逐次试探的调用计数","source_units":[{"raw_tex":"\tFinally, let us estimate the total number of the oracle calls $N_K$\n\tduring the first $K$ iterations. At each\n\titeration, the oracle is called $i_k + 1$ times, and we have $H_{k +\n\t\t1} = H_k 2^{i_k - 1}$. Therefore,\n\t$$\n\t\\ba{rcl}\n\tN_K & \\; = \\; & \\sum_{k = 0}^{K - 1} (i_k + 1) \\; = \\; \\sum_{k = 0}^{K -\n\t\t1} \\left( \\log_2 \\frac{H_{k + 1}}{H_k} + 2\\right) \\\\[5pt]\n\t& \\; = \\; & 2K + \\log_2\n\tH_{K} - \\log_2 H_0\n","start":61674,"end":62084}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S073"],"declared_external":false,"source_only":false,"title":"以残差下界截断参数","source_units":[{"raw_tex":"\t\\; \\overset{\\eqref{HolderHtGoodBound},\n\t\t\\eqref{HolderConditionFuncEps}}{\\leq} \\;\n\t2K + \\log_2\n\t\\frac{\\kappa_f(\\nu)}{\\varepsilon^{(1 - \\nu) / (2 + \\nu)}} - \\log_2\n\tH_0.\n\t\\ea\n\t$$\n\\qed\n\\end{proof}","start":62084,"end":62279}]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S074"],"declared_external":true,"declaration_text":["原前提清单：Standard convex subdifferential calculus, first-order optimality, dual-norm inequalities；其中海森矩阵半正定性及最优点零次梯度为 annotator 明示的 implicit standard fact。"],"source_only":false,"title":"标准凸分析与对偶范数","source_units":[]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S075"],"declared_external":true,"declaration_text":["原前提清单：Standard ... Hölder Taylor remainder estimates。"],"source_only":false,"title":"霍尔德泰勒余项原理","source_units":[]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S076"],"declared_external":true,"declaration_text":["原前提清单：Standard differentiation of a parameterized minimum with a unique minimizer, used at lines 1533–1538。"],"source_only":false,"title":"参数化最小值求导","source_units":[]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S077"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；原前提清单未逐项列出几何衰减的对数计数。"],"source_only":false,"title":"几何衰减与对数计数","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S030","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S037","S039":"S039","S040":"S037","S041":"S037","S042":"S037","S043":"S037","S044":"S037","S045":"S037","S046":"S046","S047":"S047","S048":"S046","S049":"S046","S050":"S046","S051":"S046","S052":"S046","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076","S077":"S077"}}}}
{"record_id":"1309.7002v3","paper":{"title":"Quantitative Characterizations of Regularity Properties of Collections of Sets","authors":["Kruger, Alexander","Thao, Nguyen Hieu"],"arxiv_id":"1309.7002","version":"v3","journal":"Journal of Optimization Theory and Applications","abstract_url":"https://arxiv.org/abs/1309.7002v3","pdf_url":"https://arxiv.org/pdf/1309.7002v3","doi":"10.1007/s10957-014-0556-0"},"context":{"raw_tex":"Our basic notation is standard; cf. \\cite{Mor06.1,RocWet98}.\r\nFor a normed linear space $X$, its topological dual is denoted $X^*$, while $\\langle\\cdot,\\cdot\\rangle$ denotes the bilinear form defining the pairing between the two spaces.\r\nThe closed unit ball in a normed space is denoted $\\B $, $B_\\delta(x)$ stands for the closed ball with radius $\\delta$ and centre $x$.\r\nProducts of normed spaces will be considered with the maximum type norms, if not specified otherwise.\r\n\r\nThe Fr\\'echet normal cone to a set $\\Omega\\subset X$ at $x\\in \\Omega$ and the Fr\\'echet subdifferential of a function $f:X \\to\\R_\\infty:=\\R\\cup\\{+\\infty\\}$ at a point $x$ with $f(x)<\\infty$ are defined, respectively, by\r\n\\begin{equation*}\r\nN_{\\Omega}(x):= \\left\\{x^* \\in X^* :\\; \\limsup_{u\\to{x},\\,u\\in\\Omega\\setminus\\{x\\}} \\frac {\\langle x^*,u-x \\rangle}{\\|u-x\\|} \\leq 0 \\right\\},\r\n\\end{equation*}\r\n\\begin{equation*}\r\n\\partial f(x):=\\left\\{x^*\\in X^*:\\; \\liminf_{\\substack{u\\to x,\\,u\\neq x}} \\frac{f(u)-f(x)-\\langle x^*,u-x\\rangle}{\\norm{u-x}}\\ge 0\\right\\}.\r\n\\end{equation*}\r\n\r\nFor a given set $\\Omega\\subset X$, the distance function associated with $\\Omega$ is defined by\r\n$$\r\nd(x,\\Omega):=\\inf_{\\omega\\in\\Omega}\\norm{x-\\omega},\\; \\forall x\\in X.\r\n$$\r\n\r\nIn the sequel, $\\bold{\\Omega}$ stands for a collection of $m$ $(m\\ge 2)$ sets $\\Omega_1,\\ldots,\\Omega_m$ in a normed linear space $X$, and we assume the existence of a point $\\bar x\\in \\bigcap_{i=1}^m\\Omega_i$.\r\n\n\nThe next definition introduces several regularity properties of $\\bold{\\Omega}$ at $\\bar x$.\r\n\r\n\\begin{dfn}\\label{HolderUR}\r\n\\begin{enumerate}\r\n\\item\r\n$\\bold{\\Omega}$ is semiregular at $\\bar x$ iff there exist positive numbers $\\alpha$ and $\\delta$ such that\r\n\\begin{equation}\\label{HR}\r\n\\bigcap_{i=1}^m(\\Omega_i-x_i)\\  \\bigcap\\ B_{\\rho}(\\bar x) \\neq \\emptyset\r\n\\end{equation}\r\nfor all $\\rho \\in ]0,\\delta[$ and all\r\n$x_i\\in{X}$ $(i=1,\\ldots,m)$ such that $\\max\\limits_{1\\leq i\\leq m}\\|x_i\\| \\le \\alpha\\rho$.\r\n\\item\r\n$\\bold{\\Omega}$ is subregular at $\\bar x$ iff there exist positive numbers $\\alpha$ and $\\delta$ such that\r\n\\begin{equation}\\label{Hlr'}\r\n\\bigcap_{i=1}^m\\left(\\Omega_i+ (\\alpha\\rho)\\B \\right)\\bigcap B_{\\delta}(\\bar x)\\subseteq\r\n\\left(\\bigcap_{i=1}^m \\Omega_i\\right)+\\rho\\B\r\n\\end{equation}\r\nfor all $\\rho\\in ]0,\\delta[$.\r\n\\item\r\n$\\bold{\\Omega}$ is uniformly regular at $\\bar x$ iff there exist positive numbers $\\alpha$ and $\\delta$ such that\r\n\\begin{equation}\\label{HUR}\r\n\\bigcap_{i=1}^m(\\Omega_i-\\omega_i-x_i) \\bigcap(\\rho\\B )\\neq \\emptyset\r\n\\end{equation}\r\nfor all $\\rho \\in ]0,\\delta[$, $\\omega_i \\in \\Omega_i \\cap B_{\\delta}(\\bar{x})$, and all $x_i\\in{X}$ $(i=1,\\ldots,m)$ such that $\\max\\limits_{1\\leq i\\leq m}\\|x_i\\| \\le \\alpha\\rho$.\r\n\\end{enumerate}\r\n\\end{dfn}\r\n\n\nThe regularity properties in Definition~\\ref{HolderUR} can be equivalently defined using the following nonnegative constants which provide quantitative characterizations of these properties:\r\n\\begin{gather}\\label{01.1}\r\n\\theta[\\bold{\\Omega}](\\bar{x}):= \\liminf_{\\rho \\downarrow 0} \\dfrac{\\theta_{\\rho}[\\bold{\\Omega}](\\bar{x})}{\\rho},\r\n\\\\\\label{01.2}\r\n\\zeta[\\bold{\\Omega}](\\bar{x}):= \\lim_{\\delta\\downarrow 0}\\inf_{0<\\rho<\\delta} \\dfrac{\\zeta_{\\rho,\\delta}[\\bold{\\Omega}](\\bar{x})} {\\rho},\r\n\\\\\\label{01.3}\r\n\\hat{\\theta}[\\bold{\\Omega}](\\bar{x}):= \\liminf_{\\omega_i\\TO{\\Omega_i}\\bar{x},\\rho\\downarrow0} \\dfrac{\\theta_{\\rho}[\\Omega_1-\\omega_1,\\ldots,\\Omega_m-\\omega_m](0)}{\\rho},\r\n\\end{gather}\r\nwhere, for $\\rho>0$ and $\\delta>0$,\r\n\\begin{gather}\\label{02.1}\r\n\\theta_{\\rho}[\\bold{\\Omega}](\\bar{x}):= \\sup\\left\\{r \\ge 0 :\\; \\bigcap_{i=1}^m (\\Omega_i - x_i) \\bigcap B_{\\rho}(\\bar{x}) \\neq \\emptyset,\\; \\forall x_i \\in r\\B \\right\\},\r\n\\\\\\label{02.2}\r\n\\zeta_{\\rho,\\delta}[\\bold{\\Omega}](\\bar{x}):= \\sup\\left\\{r\\ge0 :\\;\\bigcap_{i=1}^m (\\Omega_i+ r\\B )\\bigcap B_{\\delta}(\\bar x)\\subseteq\r\n\\bigcap_{i=1}^m \\Omega_i+\\rho\\B \\right\\}.\r\n\\end{gather}\r\n\n\n\\begin{prop}\\label{pr2}\r\n\\begin{align}\\label{newcon}\r\n\\theta[\\bold{\\Omega}](\\bar{x})&= \\liminf_{\\substack{x_i\\to 0\\; (1\\le i\\le m)\\\\ \\bar x\\notin \\bigcap_{i=1}^m(\\Omega_i-x_i)}}\r\n\\frac{\\max_{1\\le i\\le m}\\norm{x_i}}{d\\Big{(}\\bar x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\Big{)}},\r\n\\\\\\label{errcon'}\r\n\\zeta[\\bold{\\Omega}](\\bar{x})&= \\quad\\liminf_{\\substack{x\\to\\bar{x}\\\\ x\\notin \\bigcap_{i=1}^m\\Omega_i}}\r\n\\quad\\frac{\\max_{1\\le i\\le m} d(x,\\Omega_i)}{d\\Big{(}x,\\bigcap_{i=1}^m\\Omega_i\\Big{)}}\r\n\\\\\\notag\r\n&=\\liminf_{\\substack{x\\to\\bar{x}\\\\ \\omega_i\\TO{\\Omega_i}\\bar x\\; (1\\le i\\le m)\\\\ x\\notin \\bigcap_{i=1}^m\\Omega_i}}\\frac{\\max_{1\\le i\\le m}\\norm{\\omega_i-x}}{d\\Big{(}x,\\bigcap_{i=1}^m\\Omega_i\\Big{)}},\r\n\\\\\\label{vartheta}\r\n\\hat{\\theta}[\\bold{\\Omega}](\\bar{x})&= \\liminf_{\\substack{x\\to\\bar{x}\\\\x_i \\to 0\\;(1\\le i\\le m)\\\\x\\notin\\bigcap_{i=1}^m(\\Omega_i-x_i)}} \\frac{\\max_{1\\le i\\le m}d(x+x_i,\\Omega_i)} {d\\Big{(}x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\Big{)}}\r\n\\\\\\notag\r\n&= \\liminf_{\\substack{x\\to\\bar{x}\\\\x_i\\to0,\\; \\omega_i\\TO{\\Omega_i} \\bar x\\;(1\\le i\\le m)\\\\ x\\notin\\bigcap_{i=1}^m(\\Omega_i-x_i)}} \\frac{\\max_{1\\le i\\le m} \\norm{x+x_i-\\omega_i}} {d\\Big{(}x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\Big{)}}.\r\n\\end{align}\r\n\\end{prop}\n\nThis section discusses dual characterizations of regularity properties of a collection of sets $\\bold{\\Omega}:=\\{\\Omega_1,\\ldots,\\Omega_m\\}$ at $\\bar x\\in \\bigcap_{i=1}^m\\Omega_i$.\r\nWe are going to use the notation ${\\widehat\\Omega:= \\Omega_1\\times\\ldots\\times\\Omega_m\\subset X^m}$.\r\n\\sloppy\r\n\r\nRecall that the (normalized) \\emph{duality mapping} \\cite[Definition~3.2.6]{Lucc06} $J$ between a normed space $Y$ and its dual $Y^*$ is defined as\r\n$$\r\nJ(y):=\\left\\{y^*\\in \\mathbb{S}_{Y^*}:\\; \\langle y^*,y\\rangle=\\norm{y}\\right\\},\\; \\forall y\\in Y.\r\n$$\r\nNote that $J(-y)=-J(y)$.\r\n\n\nThe following simple fact of convex analysis is well known (cf., e.g., \\cite[Corollary~2.4.16]{Zal02}).\r\n\r\n\\begin{lem}\\label{ll01}\r\nLet $(Y,\\|\\cdot\\|)$ be a normed space.\r\n\\begin{enumerate}\r\n\\item\r\n$\\sd\\|\\cdot\\|(y)=J(y)$ for any $y\\ne0$.\r\n\\item\r\n$\\sd\\|\\cdot\\|(0)=\\B^*$.\r\n\\end{enumerate}\r\n\\end{lem}\r\n\n\nIn this section, along with the maximum type norm on $X^{m+1}=X\\times X^m$, we are going to use another one depending on a parameter $\\rho>0$ and defined as follows:\r\n\\begin{equation}\\label{norm}\r\n\\norm{(x,\\hat x)}_{\\rho} :=\\max\\left\\{\\norm{x},\\rho \\norm{\\hat x}\\right\\},\\quad x\\in X,\\;\\hat x\\in X^m.\r\n\\end{equation}\r\nIt is easy to check that the corresponding dual norm has the following representation:\r\n\\begin{equation}\\label{normd}\r\n\\norm{(x^*,\\hat x^*)}_{\\rho} =\\|x^*\\|+\\rho^{-1}\\|\\hat x^*\\|,\\quad x^*\\in X^*,\\;\\hat x^*\\in (X^m)^*.\r\n\\end{equation}\r\nNote that if, in \\eqref{norm} and \\eqref{normd}, $\\hat x= (x_1,\\ldots,x_m)$ and $\\hat x^*= (x_1^*,\\ldots,x_m^*)$ with $x_i\\in X$ and $x_i^*\\in X^*$ ($i=1,2,\\ldots,m$), then $\\|\\hat x\\|=\\max_{1\\le i\\le m}\\|x_i\\|$ and $\\|\\hat x^*\\|=\\sum_{i=1}^m\\|x_i^*\\|$.\r\n\n\nThe proof of the main theorem of this section relies heavily on two fundamental results of variational analysis: the \\emph{Ekeland variational principle} (Ekeland \\cite{Eke74}; cf., e.g., \\cite[Theorem~2.1]{Kru03.1}, \\cite[Theorem 2.26]{Mor06.1}) and  the \\emph{fuzzy} (\\emph{approximate}) \\emph{sum rule} (Fabian \\cite{Fab89}; cf., e.g., \\cite[Rule~2.2]{Kru03.1}, \\cite[Theorem 2.33]{Mor06.1}).\r\nBelow we provide these results for completeness.\r\n\r\n\\begin{lem}[Ekeland variational principle] \\label{l01}\r\nSuppose $X$ is a complete metric space, and $f: X\\to\\R_\\infty$ is lower semicontinuous and bounded from below, $\\varepsilon>0, \\lambda>0$. If\r\n$$\r\nf(v)<\\inf_X f + \\varepsilon,\r\n$$\r\nthen there exists $x\\in X$ such that\r\n\r\n(a) $d(x,v)<\\lambda $,\r\n\r\n(b) $f(x)\\le f(v)$,\r\n\r\n(c) $f(u)+(\\varepsilon/\\lambda)d(u,x)\\ge f(x)$ for all $u\\in X$.\r\n\\end{lem}\r\n\r\n\\begin{lem}[Fuzzy sum rule] \\label{l02}\r\nSuppose $X$ is Asplund,\r\n$f_1: X \\to\\R$ is Lipschitz continuous and\r\n$f_2: X \\to\\R_\\infty$\r\nis lower semicontinuous in a neighbourhood of $\\bar x$ with $f_2(\\bar x)<\\infty$.\r\nThen, for any $\\varepsilon>0$, there exist $x_1,x_2\\in X$ with $\\|x_i-\\bar x\\|<\\varepsilon$, ${|f_i(x_i)-f_i(\\bar x)|<\\varepsilon}$ $(i=1,2)$ such that\r\n$$\r\n\\partial (f_1+f_2) (\\bar x) \\subset \\partial f_1(x_1) +\\partial f_2(x_2) + \\varepsilon\\B^\\ast.\r\n$$\r\n\\end{lem}\r\n\n\nObserve that constant \\eqref{errcon'} can be rewritten as\r\n\\begin{equation}\\label{errcon}\r\n\\zeta[\\bold{\\Omega}](\\bar{x})= \\liminflarge_{\\substack{x\\to\\bar{x},\\, \\omega_i\\to\\bar x\\; (1\\le i\\le m)\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\\\x\\notin \\bigcap_{i=1}^m\\Omega_i}} \\frac{f(x,\\hat\\omega)} {d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right)}\r\n\\end{equation}\r\nwith function $f:X^{m+1}\\to{\\R}_\\infty :={\\R}\\cup\\{+\\infty\\}$ defined as\r\n\\begin{equation}\\label{f}\r\nf(x,\\hat x)=\\max_{1\\le i\\le m} \\norm{x-x_i} + \\delta_{\\widehat\\Omega}(\\hat x),\\quad x\\in X,\\; \\hat x:=(x_1,\\ldots,x_m)\\in X^m,\r\n\\end{equation}\r\nwhere $\\delta_{\\widehat\\Omega}$ is the indicator function of $\\widehat\\Omega$: $\\delta_{\\widehat\\Omega}(\\hat x)=0$ if $\\hat x\\in\\widehat\\Omega$ and $\\delta_{\\widehat\\Omega}(\\hat x)=+\\infty$ otherwise.\r\n","units":[{"raw_tex":"Our basic notation is standard; cf. \\cite{Mor06.1,RocWet98}.\r\nFor a normed linear space $X$, its topological dual is denoted $X^*$, while $\\langle\\cdot,\\cdot\\rangle$ denotes the bilinear form defining the pairing between the two spaces.\r\nThe closed unit ball in a normed space is denoted $\\B $, $B_\\delta(x)$ stands for the closed ball with radius $\\delta$ and centre $x$.\r\nProducts of normed spaces will be considered with the maximum type norms, if not specified otherwise.\r\n\r\nThe Fr\\'echet normal cone to a set $\\Omega\\subset X$ at $x\\in \\Omega$ and the Fr\\'echet subdifferential of a function $f:X \\to\\R_\\infty:=\\R\\cup\\{+\\infty\\}$ at a point $x$ with $f(x)<\\infty$ are defined, respectively, by\r\n\\begin{equation*}\r\nN_{\\Omega}(x):= \\left\\{x^* \\in X^* :\\; \\limsup_{u\\to{x},\\,u\\in\\Omega\\setminus\\{x\\}} \\frac {\\langle x^*,u-x \\rangle}{\\|u-x\\|} \\leq 0 \\right\\},\r\n\\end{equation*}\r\n\\begin{equation*}\r\n\\partial f(x):=\\left\\{x^*\\in X^*:\\; \\liminf_{\\substack{u\\to x,\\,u\\neq x}} \\frac{f(u)-f(x)-\\langle x^*,u-x\\rangle}{\\norm{u-x}}\\ge 0\\right\\}.\r\n\\end{equation*}\r\n\r\nFor a given set $\\Omega\\subset X$, the distance function associated with $\\Omega$ is defined by\r\n$$\r\nd(x,\\Omega):=\\inf_{\\omega\\in\\Omega}\\norm{x-\\omega},\\; \\forall x\\in X.\r\n$$\r\n\r\nIn the sequel, $\\bold{\\Omega}$ stands for a collection of $m$ $(m\\ge 2)$ sets $\\Omega_1,\\ldots,\\Omega_m$ in a normed linear space $X$, and we assume the existence of a point $\\bar x\\in \\bigcap_{i=1}^m\\Omega_i$.\r\n","start":6233,"end":7681},{"raw_tex":"The next definition introduces several regularity properties of $\\bold{\\Omega}$ at $\\bar x$.\r\n\r\n\\begin{dfn}\\label{HolderUR}\r\n\\begin{enumerate}\r\n\\item\r\n$\\bold{\\Omega}$ is semiregular at $\\bar x$ iff there exist positive numbers $\\alpha$ and $\\delta$ such that\r\n\\begin{equation}\\label{HR}\r\n\\bigcap_{i=1}^m(\\Omega_i-x_i)\\  \\bigcap\\ B_{\\rho}(\\bar x) \\neq \\emptyset\r\n\\end{equation}\r\nfor all $\\rho \\in ]0,\\delta[$ and all\r\n$x_i\\in{X}$ $(i=1,\\ldots,m)$ such that $\\max\\limits_{1\\leq i\\leq m}\\|x_i\\| \\le \\alpha\\rho$.\r\n\\item\r\n$\\bold{\\Omega}$ is subregular at $\\bar x$ iff there exist positive numbers $\\alpha$ and $\\delta$ such that\r\n\\begin{equation}\\label{Hlr'}\r\n\\bigcap_{i=1}^m\\left(\\Omega_i+ (\\alpha\\rho)\\B \\right)\\bigcap B_{\\delta}(\\bar x)\\subseteq\r\n\\left(\\bigcap_{i=1}^m \\Omega_i\\right)+\\rho\\B\r\n\\end{equation}\r\nfor all $\\rho\\in ]0,\\delta[$.\r\n\\item\r\n$\\bold{\\Omega}$ is uniformly regular at $\\bar x$ iff there exist positive numbers $\\alpha$ and $\\delta$ such that\r\n\\begin{equation}\\label{HUR}\r\n\\bigcap_{i=1}^m(\\Omega_i-\\omega_i-x_i) \\bigcap(\\rho\\B )\\neq \\emptyset\r\n\\end{equation}\r\nfor all $\\rho \\in ]0,\\delta[$, $\\omega_i \\in \\Omega_i \\cap B_{\\delta}(\\bar{x})$, and all $x_i\\in{X}$ $(i=1,\\ldots,m)$ such that $\\max\\limits_{1\\leq i\\leq m}\\|x_i\\| \\le \\alpha\\rho$.\r\n\\end{enumerate}\r\n\\end{dfn}\r\n","start":7931,"end":9217},{"raw_tex":"The regularity properties in Definition~\\ref{HolderUR} can be equivalently defined using the following nonnegative constants which provide quantitative characterizations of these properties:\r\n\\begin{gather}\\label{01.1}\r\n\\theta[\\bold{\\Omega}](\\bar{x}):= \\liminf_{\\rho \\downarrow 0} \\dfrac{\\theta_{\\rho}[\\bold{\\Omega}](\\bar{x})}{\\rho},\r\n\\\\\\label{01.2}\r\n\\zeta[\\bold{\\Omega}](\\bar{x}):= \\lim_{\\delta\\downarrow 0}\\inf_{0<\\rho<\\delta} \\dfrac{\\zeta_{\\rho,\\delta}[\\bold{\\Omega}](\\bar{x})} {\\rho},\r\n\\\\\\label{01.3}\r\n\\hat{\\theta}[\\bold{\\Omega}](\\bar{x}):= \\liminf_{\\omega_i\\TO{\\Omega_i}\\bar{x},\\rho\\downarrow0} \\dfrac{\\theta_{\\rho}[\\Omega_1-\\omega_1,\\ldots,\\Omega_m-\\omega_m](0)}{\\rho},\r\n\\end{gather}\r\nwhere, for $\\rho>0$ and $\\delta>0$,\r\n\\begin{gather}\\label{02.1}\r\n\\theta_{\\rho}[\\bold{\\Omega}](\\bar{x}):= \\sup\\left\\{r \\ge 0 :\\; \\bigcap_{i=1}^m (\\Omega_i - x_i) \\bigcap B_{\\rho}(\\bar{x}) \\neq \\emptyset,\\; \\forall x_i \\in r\\B \\right\\},\r\n\\\\\\label{02.2}\r\n\\zeta_{\\rho,\\delta}[\\bold{\\Omega}](\\bar{x}):= \\sup\\left\\{r\\ge0 :\\;\\bigcap_{i=1}^m (\\Omega_i+ r\\B )\\bigcap B_{\\delta}(\\bar x)\\subseteq\r\n\\bigcap_{i=1}^m \\Omega_i+\\rho\\B \\right\\}.\r\n\\end{gather}\r\n","start":10444,"end":11579},{"raw_tex":"\\begin{prop}\\label{pr2}\r\n\\begin{align}\\label{newcon}\r\n\\theta[\\bold{\\Omega}](\\bar{x})&= \\liminf_{\\substack{x_i\\to 0\\; (1\\le i\\le m)\\\\ \\bar x\\notin \\bigcap_{i=1}^m(\\Omega_i-x_i)}}\r\n\\frac{\\max_{1\\le i\\le m}\\norm{x_i}}{d\\Big{(}\\bar x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\Big{)}},\r\n\\\\\\label{errcon'}\r\n\\zeta[\\bold{\\Omega}](\\bar{x})&= \\quad\\liminf_{\\substack{x\\to\\bar{x}\\\\ x\\notin \\bigcap_{i=1}^m\\Omega_i}}\r\n\\quad\\frac{\\max_{1\\le i\\le m} d(x,\\Omega_i)}{d\\Big{(}x,\\bigcap_{i=1}^m\\Omega_i\\Big{)}}\r\n\\\\\\notag\r\n&=\\liminf_{\\substack{x\\to\\bar{x}\\\\ \\omega_i\\TO{\\Omega_i}\\bar x\\; (1\\le i\\le m)\\\\ x\\notin \\bigcap_{i=1}^m\\Omega_i}}\\frac{\\max_{1\\le i\\le m}\\norm{\\omega_i-x}}{d\\Big{(}x,\\bigcap_{i=1}^m\\Omega_i\\Big{)}},\r\n\\\\\\label{vartheta}\r\n\\hat{\\theta}[\\bold{\\Omega}](\\bar{x})&= \\liminf_{\\substack{x\\to\\bar{x}\\\\x_i \\to 0\\;(1\\le i\\le m)\\\\x\\notin\\bigcap_{i=1}^m(\\Omega_i-x_i)}} \\frac{\\max_{1\\le i\\le m}d(x+x_i,\\Omega_i)} {d\\Big{(}x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\Big{)}}\r\n\\\\\\notag\r\n&= \\liminf_{\\substack{x\\to\\bar{x}\\\\x_i\\to0,\\; \\omega_i\\TO{\\Omega_i} \\bar x\\;(1\\le i\\le m)\\\\ x\\notin\\bigcap_{i=1}^m(\\Omega_i-x_i)}} \\frac{\\max_{1\\le i\\le m} \\norm{x+x_i-\\omega_i}} {d\\Big{(}x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\Big{)}}.\r\n\\end{align}\r\n\\end{prop}","start":21228,"end":22433},{"raw_tex":"This section discusses dual characterizations of regularity properties of a collection of sets $\\bold{\\Omega}:=\\{\\Omega_1,\\ldots,\\Omega_m\\}$ at $\\bar x\\in \\bigcap_{i=1}^m\\Omega_i$.\r\nWe are going to use the notation ${\\widehat\\Omega:= \\Omega_1\\times\\ldots\\times\\Omega_m\\subset X^m}$.\r\n\\sloppy\r\n\r\nRecall that the (normalized) \\emph{duality mapping} \\cite[Definition~3.2.6]{Lucc06} $J$ between a normed space $Y$ and its dual $Y^*$ is defined as\r\n$$\r\nJ(y):=\\left\\{y^*\\in \\mathbb{S}_{Y^*}:\\; \\langle y^*,y\\rangle=\\norm{y}\\right\\},\\; \\forall y\\in Y.\r\n$$\r\nNote that $J(-y)=-J(y)$.\r\n","start":28342,"end":28918},{"raw_tex":"The following simple fact of convex analysis is well known (cf., e.g., \\cite[Corollary~2.4.16]{Zal02}).\r\n\r\n\\begin{lem}\\label{ll01}\r\nLet $(Y,\\|\\cdot\\|)$ be a normed space.\r\n\\begin{enumerate}\r\n\\item\r\n$\\sd\\|\\cdot\\|(y)=J(y)$ for any $y\\ne0$.\r\n\\item\r\n$\\sd\\|\\cdot\\|(0)=\\B^*$.\r\n\\end{enumerate}\r\n\\end{lem}\r\n","start":28920,"end":29219},{"raw_tex":"In this section, along with the maximum type norm on $X^{m+1}=X\\times X^m$, we are going to use another one depending on a parameter $\\rho>0$ and defined as follows:\r\n\\begin{equation}\\label{norm}\r\n\\norm{(x,\\hat x)}_{\\rho} :=\\max\\left\\{\\norm{x},\\rho \\norm{\\hat x}\\right\\},\\quad x\\in X,\\;\\hat x\\in X^m.\r\n\\end{equation}\r\nIt is easy to check that the corresponding dual norm has the following representation:\r\n\\begin{equation}\\label{normd}\r\n\\norm{(x^*,\\hat x^*)}_{\\rho} =\\|x^*\\|+\\rho^{-1}\\|\\hat x^*\\|,\\quad x^*\\in X^*,\\;\\hat x^*\\in (X^m)^*.\r\n\\end{equation}\r\nNote that if, in \\eqref{norm} and \\eqref{normd}, $\\hat x= (x_1,\\ldots,x_m)$ and $\\hat x^*= (x_1^*,\\ldots,x_m^*)$ with $x_i\\in X$ and $x_i^*\\in X^*$ ($i=1,2,\\ldots,m$), then $\\|\\hat x\\|=\\max_{1\\le i\\le m}\\|x_i\\|$ and $\\|\\hat x^*\\|=\\sum_{i=1}^m\\|x_i^*\\|$.\r\n","start":31029,"end":31838},{"raw_tex":"The proof of the main theorem of this section relies heavily on two fundamental results of variational analysis: the \\emph{Ekeland variational principle} (Ekeland \\cite{Eke74}; cf., e.g., \\cite[Theorem~2.1]{Kru03.1}, \\cite[Theorem 2.26]{Mor06.1}) and  the \\emph{fuzzy} (\\emph{approximate}) \\emph{sum rule} (Fabian \\cite{Fab89}; cf., e.g., \\cite[Rule~2.2]{Kru03.1}, \\cite[Theorem 2.33]{Mor06.1}).\r\nBelow we provide these results for completeness.\r\n\r\n\\begin{lem}[Ekeland variational principle] \\label{l01}\r\nSuppose $X$ is a complete metric space, and $f: X\\to\\R_\\infty$ is lower semicontinuous and bounded from below, $\\varepsilon>0, \\lambda>0$. If\r\n$$\r\nf(v)<\\inf_X f + \\varepsilon,\r\n$$\r\nthen there exists $x\\in X$ such that\r\n\r\n(a) $d(x,v)<\\lambda $,\r\n\r\n(b) $f(x)\\le f(v)$,\r\n\r\n(c) $f(u)+(\\varepsilon/\\lambda)d(u,x)\\ge f(x)$ for all $u\\in X$.\r\n\\end{lem}\r\n\r\n\\begin{lem}[Fuzzy sum rule] \\label{l02}\r\nSuppose $X$ is Asplund,\r\n$f_1: X \\to\\R$ is Lipschitz continuous and\r\n$f_2: X \\to\\R_\\infty$\r\nis lower semicontinuous in a neighbourhood of $\\bar x$ with $f_2(\\bar x)<\\infty$.\r\nThen, for any $\\varepsilon>0$, there exist $x_1,x_2\\in X$ with $\\|x_i-\\bar x\\|<\\varepsilon$, ${|f_i(x_i)-f_i(\\bar x)|<\\varepsilon}$ $(i=1,2)$ such that\r\n$$\r\n\\partial (f_1+f_2) (\\bar x) \\subset \\partial f_1(x_1) +\\partial f_2(x_2) + \\varepsilon\\B^\\ast.\r\n$$\r\n\\end{lem}\r\n","start":33628,"end":34966},{"raw_tex":"Observe that constant \\eqref{errcon'} can be rewritten as\r\n\\begin{equation}\\label{errcon}\r\n\\zeta[\\bold{\\Omega}](\\bar{x})= \\liminflarge_{\\substack{x\\to\\bar{x},\\, \\omega_i\\to\\bar x\\; (1\\le i\\le m)\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\\\x\\notin \\bigcap_{i=1}^m\\Omega_i}} \\frac{f(x,\\hat\\omega)} {d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right)}\r\n\\end{equation}\r\nwith function $f:X^{m+1}\\to{\\R}_\\infty :={\\R}\\cup\\{+\\infty\\}$ defined as\r\n\\begin{equation}\\label{f}\r\nf(x,\\hat x)=\\max_{1\\le i\\le m} \\norm{x-x_i} + \\delta_{\\widehat\\Omega}(\\hat x),\\quad x\\in X,\\; \\hat x:=(x_1,\\ldots,x_m)\\in X^m,\r\n\\end{equation}\r\nwhere $\\delta_{\\widehat\\Omega}$ is the indicator function of $\\widehat\\Omega$: $\\delta_{\\widehat\\Omega}(\\hat x)=0$ if $\\hat x\\in\\widehat\\Omega$ and $\\delta_{\\widehat\\Omega}(\\hat x)=+\\infty$ otherwise.\r\n","start":36578,"end":37377}]},"statement":{"raw_tex":"\\begin{thm}\\label{T3.1}\r\nLet $X$ be an Asplund space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n\\begin{enumerate}\r\n\\item\r\n$\\bold{\\Omega}$ is subregular at $\\bar x$ if there exist positive numbers $\\alpha$ and $\\delta$ such that, for any $\\rho\\in ]0,\\delta[$, $x\\in B_{\\rho}(\\bar x)$, $\\omega_i\\in\\Omega_i\\cap B_{\\rho}(x)$ $(i=1,\\ldots,m)$ with $\\omega_i\\neq x$ for some ${i\\in\\{1,\\ldots,m\\}}$, there is an $\\varepsilon>0$ such that, for any $x'\\in B_{\\varepsilon}(x)$, $\\omega'_i\\in\\Omega_i\\cap B_{\\varepsilon}(\\omega_i)$, $x_{i}^*\\in N_{\\Omega_i}(\\omega_i') +\\rho\\B^*$ $(i=1,\\ldots,m)$\r\nsatisfying\r\n\\begin{gather}\\notag\r\nx_i^*=0\r\n\\quad\\mbox{if}\\quad\r\n\\norm{x'-\\omega_i'}<\\max_{1\\le j\\le m} \\norm{x'-\\omega_j'},\r\n\\\\\\notag\r\n\\langle x_i^*,x'-\\omega_i'\\rangle\r\n\\ge\\|x_i^*\\|(\\|x'-\\omega_i'\\|-\\varepsilon),\r\n\\\\\\label{=1}\r\n\\sum_{i=1}^m\\|x_i^*\\|=1,\r\n\\end{gather}\r\nit holds\r\n\\begin{gather}\\label{>al}\r\n\\left\\|\\sum\\limits_{i=1}^m{x}_i^*\\right\\|>\\alpha.\r\n\\end{gather}\r\n\\item\r\n$\\bold{\\Omega}$ is uniformly regular at $\\bar x$ if and only if there are positive numbers $\\alpha$ and $\\delta$ such that\r\n\\eqref{>al} holds true\r\nfor all $\\omega_i\\in\\Omega_i\\cap{B}_\\delta(\\bar{x})$ and\r\n$x_i^*\\in{N}_{\\Omega_i}(\\omega_i)$ $(i=1,\\ldots,m)$ satisfying \\eqref{=1}. \\end{enumerate}\r\n\\end{thm}","units":[{"raw_tex":"\\begin{thm}\\label{T3.1}\r\nLet $X$ be an Asplund space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n\\begin{enumerate}\r\n\\item\r\n$\\bold{\\Omega}$ is subregular at $\\bar x$ if there exist positive numbers $\\alpha$ and $\\delta$ such that, for any $\\rho\\in ]0,\\delta[$, $x\\in B_{\\rho}(\\bar x)$, $\\omega_i\\in\\Omega_i\\cap B_{\\rho}(x)$ $(i=1,\\ldots,m)$ with $\\omega_i\\neq x$ for some ${i\\in\\{1,\\ldots,m\\}}$, there is an $\\varepsilon>0$ such that, for any $x'\\in B_{\\varepsilon}(x)$, $\\omega'_i\\in\\Omega_i\\cap B_{\\varepsilon}(\\omega_i)$, $x_{i}^*\\in N_{\\Omega_i}(\\omega_i') +\\rho\\B^*$ $(i=1,\\ldots,m)$\r\nsatisfying\r\n\\begin{gather}\\notag\r\nx_i^*=0\r\n\\quad\\mbox{if}\\quad\r\n\\norm{x'-\\omega_i'}<\\max_{1\\le j\\le m} \\norm{x'-\\omega_j'},\r\n\\\\\\notag\r\n\\langle x_i^*,x'-\\omega_i'\\rangle\r\n\\ge\\|x_i^*\\|(\\|x'-\\omega_i'\\|-\\varepsilon),\r\n\\\\\\label{=1}\r\n\\sum_{i=1}^m\\|x_i^*\\|=1,\r\n\\end{gather}\r\nit holds\r\n\\begin{gather}\\label{>al}\r\n\\left\\|\\sum\\limits_{i=1}^m{x}_i^*\\right\\|>\\alpha.\r\n\\end{gather}\r\n\\item\r\n$\\bold{\\Omega}$ is uniformly regular at $\\bar x$ if and only if there are positive numbers $\\alpha$ and $\\delta$ such that\r\n\\eqref{>al} holds true\r\nfor all $\\omega_i\\in\\Omega_i\\cap{B}_\\delta(\\bar{x})$ and\r\n$x_i^*\\in{N}_{\\Omega_i}(\\omega_i)$ $(i=1,\\ldots,m)$ satisfying \\eqref{=1}. \\end{enumerate}\r\n\\end{thm}","start":35060,"end":36337}]},"proof":{"raw_tex":"\\begin{prop}\\label{theorem11}\r\n\\begin{enumerate}\r\n\\item\r\n$\\bold{\\Omega}$ is semiregular at $\\bar x$ if and only if $\\theta[\\bold{\\Omega}](\\bar{x})>0$. Moreover, $\\theta[\\bold{\\Omega}](\\bar{x})$ is the exact upper bound of all numbers $\\alpha$ such that (\\ref{HR}) is satisfied.\r\n\\item\r\n$\\bold{\\Omega}$ is subregular at $\\bar x$ if and only if $\\zeta[\\bold{\\Omega}](\\bar{x})>0$. Moreover, $\\zeta[\\bold{\\Omega}](\\bar{x})$ is the exact upper bound of all numbers $\\alpha$ such that (\\ref{Hlr'}) is satisfied.\r\n\\item\r\n$\\bold{\\Omega}$ is uniformly regular at $\\bar x$ if and only if $\\hat{\\theta}[\\bold{\\Omega}](\\bar{x})>0$. Moreover, $\\hat{\\theta}[\\bold{\\Omega}](\\bar{x})$ is the exact upper bound of all numbers $\\alpha$ such that (\\ref{HUR}) is satisfied.\r\n\\end{enumerate}\r\n\\end{prop}\n\n\\begin{prop}\\label{prJ}\r\nFor each $(x_1,\\ldots,x_m)\\in X^m$,\r\n\\begin{multline*}\r\nJ(x_1,\\ldots,x_m)= \\Biggl\\{(x_1^*,\\ldots,x_m^*)\\in(X^*)^m:\\; \\sum_{i=1}^m\\|x_i^*\\|=1;\\mbox{ either } x_i^*=0\\\\\r\n\\mbox{or }\\left(\\|x_i\\|=\\max_{1\\le j\\le m} \\|x_j\\|,\\; x_i^*\\in\\|x_i^*\\| J(x_i)\\right)\\; (i=1,\\ldots,m)\\Biggr\\}.\r\n\\end{multline*}\r\n\\end{prop}\n\n\\begin{proof}\r\nLet $\\hat x:=(x_1,\\ldots,x_m)\\in X^m$, $\\hat x^*:=(x_1^*,\\ldots,x_m^*)\\in(X^*)^m$.\r\nThen\r\n$${\\|\\hat x\\|=\\max_{1\\le i\\le m} \\|x_i\\|},\\quad \\|\\hat x^*\\|=\\sum_{i=1}^m\\|x_i^*\\|,\\quad \\langle \\hat x^*,\\hat x\\rangle=\\sum_{i=1}^m\\langle x_i^*,x_i\\rangle.$$\r\nSuppose $\\|\\hat x^*\\|=1$, i.e., $\\sum_{i=1}^m\\|x_i^*\\|=1$.\r\nThen $\\hat x^*\\in J(\\hat x)$ if and only if ${\\sum_{i=1}^m\\langle x_i^*,x_i\\rangle=\\|\\hat x\\|}$.\r\nIn its turn, the last equality holds true if and only if ${\\langle x_i^*,x_i\\rangle=\\|x_i^*\\|\\cdot\\|\\hat x\\|}$ for all $i=1,\\ldots,m$.\r\n\r\nIndeed, if $\\langle x_i^*,x_i\\rangle=\\|x_i^*\\|\\cdot\\|\\hat x\\|$ for all $i=1,\\ldots,m$, then adding these $m$ equalities, we obtain $\\sum_{i=1}^m\\langle x_i^*,x_i\\rangle=\\|\\hat x\\|$.\r\nConversely, if $\\langle x_i^*,x_i\\rangle\\ne\\|x_i^*\\|\\cdot\\|\\hat x\\|$, i.e., ${\\langle x_i^*,x_i\\rangle<\\|x_i^*\\|\\cdot\\|\\hat x\\|}$ for some $i\\in\\{1,\\ldots,m\\}$, then\r\n$$\r\n\\sum_{j=1}^m\\langle x_j^*,x_j\\rangle =\\langle x_i^*,x_i\\rangle+\\sum_{j\\ne i}\\langle x_j^*,x_j\\rangle <\\|x_i^*\\|\\cdot\\|\\hat x\\|+\\|\\hat x\\|\\sum_{j\\ne i} \\|x_j^*\\|=\\|\\hat x\\|.\r\n$$\r\n\r\nFinally, $\\langle x_i^*,x_i\\rangle=\\|x_i^*\\|\\cdot\\|\\hat x\\|$ for some $i\\in\\{1,\\ldots,m\\}$ if and only if either $\\|x_i\\|=\\|\\hat x\\|$ and $x_i^*\\in\\|x_i^*\\| J(x_i)$ or $x_i^*=0$.\r\n\\qed\\end{proof}\n\n\\begin{lem}\\label{ll02}\r\nLet $X$ be a normed space and\r\n$\\varphi(u,\\hat u)=\\|(u-u_1,\\ldots,u-u_m)\\|$, $u\\in X$, $\\hat u:=(u_1,\\ldots,u_m)\\in X^m$.\r\nSuppose $x\\in X$, $\\hat x:=(x_1,\\ldots,x_m)\\in X^m$, and ${\\hat v:=(x-x_1,\\ldots,x-x_m)}\\ne0$. Then\r\n\\begin{align*}\r\n\\sd\\varphi(x,\\hat x)\\subseteq\\bigl\\{\\left(x^*,\\hat x^* =(x_1^*,\\ldots,x_m^*)\\right)\\in &X^*\\times (X^*)^m :\\;\\\\\r\n&-\\hat x^*\\in J(\\hat v),\\; x^*=-(x_{1}^*+\\ldots+x_{m}^*)\\bigr\\}.\r\n\\end{align*}\r\n\\end{lem}\n\n\\begin{proof}\r\n%Observe that $\\varphi$ is a composition of the norm on $X^m$ and the linear continuous mapping $(x,\\hat x)\\mapsto(x_1-x,\\ldots,x_m-x)$.\r\nLet $\\left(x^*,\\hat x^* =(x_1^*,\\ldots,x_m^*)\\right)\\in \\sd\\varphi(x,\\hat x)$, i.e.,\r\n$$\r\n\\|(u-u_1,\\ldots,u-u_m)\\|-\\|(x-x_1,\\ldots,x-x_m)\\|\\ge \\langle x^*,u-x\\rangle+\\sum_{i=1}^m\\langle x_i^*,u_i-x_i\\rangle\r\n$$\r\nfor any $u\\in X$ and $\\hat u:=(u_1,\\ldots,u_m)\\in X^m$.\r\nIn particular, with $u=x$ and $u_i=x_i-x_i'$ ($i=1,\\ldots,m$) for an arbitrary $\\hat x':=(x_1',\\ldots,x_m')\\in X^m$, we have\r\n$$\r\n\\|\\hat v+\\hat x'\\|-\\|\\hat v\\|\\ge-\\langle\\hat x^*,\\hat x'\\rangle,\r\n$$\r\ni.e., $-\\hat x^*\\in J(\\hat v)$.\r\nSimilarly, with $u=x+x'$ and $u_i=x_i+x'$ ($i=1,\\ldots,m$) for an arbitrary $x'\\in X$, we have\r\n$$\r\n\\left\\langle x^*+\\sum_{i=1}^mx_i^*,x'\\right\\rangle\\le0,\r\n$$\r\nand consequently $x^*+x_{1}^*+\\ldots+x_{m}^*=0$.\r\n\\qed\\end{proof}\n\n\\begin{lem}\\label{ll03}\r\nLet $X$ be a normed space and $\\hat\\omega:=(\\omega_1,\\ldots,\\omega_m)\\in\\widehat\\Omega$.\r\nThen ${N_{\\widehat\\Omega}(\\hat\\omega) =N_{\\Omega_1}(\\omega_1)\\times\\ldots\\times N_{\\Omega_m}(\\omega_m)}$.\r\n\\end{lem}\n\n\\begin{proof}\r\nfollows directly from the definition of the Fr\\'echet normal cone.\r\n\\qed\\end{proof}\n\n\\begin{prop}\\label{T3.1.1}\r\nLet $X$ be a Banach space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n\\begin{enumerate}\r\n\\item\r\n$\\hat\\zeta[\\bold{\\Omega}](\\bar x)\\le \\zeta[\\bold{\\Omega}](\\bar{x})$, where\r\n\\begin{equation}\\label{strslo}\r\n\\hat\\zeta[\\bold{\\Omega}](\\bar x):= \\lim_{\\rho\\downarrow0} \\inflarge_{\\substack{\\norm{x-\\bar x}<\\rho\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega)\r\n\\end{equation}\r\nand, for $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega$,\r\n\\begin{equation}\\label{rhoslo}\r\n\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega):= \\limsup_{\\substack{(u,\\hat v)\\to(x,\\hat\\omega)\\\\(u,\\hat v)\\neq (x,\\hat\\omega)\\\\ \\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega}} \\frac{\\left(\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}-\\max\\limits_{1\\le i\\le m} \\norm{u-v_i}\\right)_+}{\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho}}.\r\n\\end{equation}\r\n\\item\r\nIf $\\hat\\zeta[\\bold{\\Omega}](\\bar x)>0$, then $\\bold{\\Omega}$ is subregular at $\\bar x$.\r\n\\end{enumerate}\r\n\\end{prop}\n\n\\begin{proof}\r\n(i)\r\nLet $\\zeta[\\bold{\\Omega}](\\bar{x})<\\alpha<\\infty$. Choose a $\\rho\\in ]0,1[$ and set\r\n\\begin{equation}\\label{28}\r\n\\eta:= \\min\\left\\{\\frac{\\rho}{2},\\frac{\\rho}{\\alpha},\\rho^{\\frac{2}{\\rho}}\\right\\}.\r\n\\end{equation}\r\nBy \\eqref{errcon}, there are $x'\\in B_{\\eta}(\\bar x)$ and $\\hat\\omega'=(\\omega_1',\\ldots,\\omega_m')\\in \\widehat\\Omega$ such that\r\n\\begin{equation}\\label{29}\r\n   0<f(x',\\hat\\omega')<\\alpha d\\left(x',\\bigcap_{i=1}^m\\Omega_i\\right).\r\n\\end{equation}\r\nDenote $\\varepsilon:=f(x',\\hat\\omega')$ and $\\mu:=d\\left(x',\\bigcap_{i=1}^m\\Omega_i\\right)$. Then $\\mu\\le \\norm{x'-\\bar x}\\le \\eta\\le \\frac{\\rho}{2}<1$.\r\nObserve that $f$ is lower semicontinuous.\r\nApplying to $f$ Lemma~\\ref{l01} with $\\varepsilon$ as above and\r\n\\begin{equation}\\label{30}\r\n\\lambda:=\\mu(1-\\mu^{\\frac{\\rho}{2-\\rho}}),\r\n\\end{equation}\r\nwe find points $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in X^{m}$ such that\r\n\\begin{equation}\\label{31}\r\n   \\norm{(x,\\hat\\omega)-(x',\\hat\\omega')}_{\\rho}<\\lambda,\\; f(x,\\hat\\omega)\\le f(x',\\hat\\omega'),\r\n\\end{equation}\r\nand\r\n\\begin{equation}\\label{32}\r\nf(u,\\hat v)+\\frac{\\varepsilon}{\\lambda}\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho} \\ge f(x,\\hat\\omega),\r\n\\end{equation}\r\nfor all $(u,\\hat v)\\in X\\times X^{m}$.\r\nThanks to \\eqref{31}, \\eqref{30}, \\eqref{28}, and \\eqref{29}, we have\r\n$$\r\n\\norm{x-x'}<\\lambda <\\mu \\le \\norm{x'-\\bar x},\r\n$$\r\n\\begin{equation}\\label{33}\r\n   d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right)\\ge d\\left(x',\\bigcap_{i=1}^m\\Omega_i\\right)- \\norm{x-x'}\\ge \\mu-\\lambda =\\mu^{\\frac{2}{2-\\rho}},\r\n\\end{equation}\r\n\\begin{equation}\\label{34}\r\n\\norm{x-\\bar x}\\le \\norm{x-x'}+\\norm{x'-\\bar x}<2\\norm{x'-\\bar x}\\le 2\\eta\\le\\rho,\r\n\\end{equation}\r\n\\begin{equation}\\label{35}\r\nf(x,\\hat\\omega)\\le f(x',\\hat\\omega')<\\alpha\\mu\\le \\alpha\\eta\\le\\rho.\r\n\\end{equation}\r\nIt follows from \\eqref{33}, \\eqref{34}, and \\eqref{35} that\r\n$$\r\n\\norm{x-\\bar x}<\\rho,\\; \\hat\\omega\\in \\widehat\\Omega,\\; 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho.\r\n$$\r\nObserve that $\\mu^{\\frac{\\rho}{2-\\rho}}\\le \\eta^{\\frac{\\rho}{2-\\rho}}< \\eta^{\\frac{\\rho}{2}}\\le \\rho$, and consequently, by \\eqref{29} and \\eqref{30},\r\n$$\r\n\\frac{\\varepsilon}{\\lambda} <\\frac{\\alpha\\mu}{\\lambda} =\\frac{\\alpha}{1-\\mu^{\\frac{\\rho}{2-\\rho}}} <\\frac{\\alpha}{1-\\rho}.\r\n$$\r\nThanks to \\eqref{32} and \\eqref{f}, we have\r\n$$\r\n\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}-\\max\\limits_{1\\le i\\le m} \\norm{u-v_i}\\le \\frac{\\alpha}{1-\\rho}\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho}\r\n$$\r\nfor all $u\\in X$ and $\\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega$.\r\nIt follows that $\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega)\\le \\dfrac{\\alpha}{1-\\rho}$ and consequently\r\n$$\r\n\\inflarge_{\\substack{\\norm{x-\\bar x}<\\rho\\\\\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega)\\le \\frac{\\alpha}{1-\\rho}.\r\n$$\r\nTaking limits in the last inequality as $\\rho\\downarrow 0$ and $\\alpha\\to \\zeta[\\bold{\\Omega}](\\bar{x})$ yields the claimed inequality.\r\n\r\n(ii) follows from (i) and Proposition~\\ref{theorem11}~(ii).\r\n\\qed\\end{proof}\n\n\\begin{prop}\\label{T3.1.2}\r\nLet $X$ be an Asplund space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n\\begin{enumerate}\r\n\\item\r\n$\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)\\le \\hat\\zeta[\\bold{\\Omega}](\\bar x)$, where $\\hat\\zeta[\\bold{\\Omega}](\\bar x)$ is given by \\eqref{strslo},\r\n\\begin{equation}\\label{strsubslo'}\r\n\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x):= \\lim_{\\rho\\downarrow 0} \\inflarge_{\\substack{\\norm{x-\\bar x}<\\rho\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x,\\hat\\omega)\r\n\\end{equation}\r\nand, for $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega$,\r\n\\begin{equation}\\label{subrhoslo}\r\n\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x,\\hat\\omega):= \\inflarge_{\\substack{(x^*,\\hat y^*)\\in\\partial f(x,\\hat\\omega)\\\\ \\norm{\\hat y^*}<\\rho}}\\norm{x^*}\r\n\\end{equation}\r\n(with the convention that the infimum over the empty set equals $+\\infty$).\r\n\\item\r\nIf $\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)>0$, then $\\bold{\\Omega}$ is subregular at $\\bar x$.\r\n\\end{enumerate}\r\n\\end{prop}\n\n\\begin{proof}\r\nLet $\\hat\\zeta[\\bold{\\Omega}](\\bar x)< \\alpha<\\infty$.\r\nChoose a $\\beta\\in ]\\hat\\zeta[\\bold{\\Omega}](\\bar x),\\alpha[$ and an arbitrary ${\\rho>0}$.\r\nSet ${\\rho'=\\min\\{1,\\alpha^{-1}\\}\\rho}$.\r\nBy \\eqref{strslo} and \\eqref{rhoslo}, one can find points $x\\in X$ and ${\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega}$ such that $\\norm{x-\\bar x}<\\rho'$, $0<\\max_{1\\le i\\le m} \\norm{\\omega_i-x}<\\rho'$, and\r\n$$\r\n\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}-\\max\\limits_{1\\le i\\le m} \\norm{u-v_i}\\le \\beta\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho'}\r\n$$\r\nfor all $(u,\\hat v)$ with $\\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega$ near $(x,\\hat\\omega)$.\r\nIn other words, $(x,\\hat\\omega)$ is a local minimizer of the function\r\n$$\r\n(u,\\hat v)\\mapsto \\max\\limits_{1\\le i\\le m} \\norm{u-v_i}+\\beta\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho'}\r\n$$\r\nsubject to $\\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega$.\r\nBy definition \\eqref{f}, this means that $(x,\\hat\\omega)$ minimizes locally the function\r\n$$\r\n(u,\\hat v)\\mapsto f(u,\\hat v)+\\beta\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho'},\r\n$$\r\nand consequently its Fr\\'echet subdifferential at $(x,\\hat\\omega)$ contains zero.\r\nTake an\r\n$$\r\n\\varepsilon\\in\\Bigl]0,\\min\\{\\rho-\\norm{x-\\bar x},\\rho-\\max_{1\\le i\\le m} \\norm{x-\\omega_i},\\alpha-\\beta\\}\\Bigr[.\r\n$$\r\nApplying Lemma~\\ref{l02} and Lemma~\\ref{ll01}~(ii), we can find points $x'\\in X$, ${\\hat\\omega'=(\\omega_1',\\ldots,\\omega_m')\\in \\widehat\\Omega}$, and $(x^*,\\hat y^*)\\in \\partial f(x',\\hat\\omega')$ such that $$\\norm{x'-x}<\\varepsilon,\\quad 0<\\max_{1\\le i\\le m} \\norm{x'-\\omega_i'}\\le \\max_{1\\le i\\le m} \\norm{x-\\omega_i}+\\varepsilon,$$\r\n$$\\mbox{and}\\quad\\norm{(x^*,\\hat y^*)}_{\\rho'} =\\|x^*\\|+\\|\\hat y^*\\|/\\rho' <\\beta+\\varepsilon.$$\r\nIt follows that\r\n$$\\norm{x'-\\bar x}<\\rho,\\;\\; 0<\\max_{1\\le i\\le m} \\norm{x'-\\omega_i'}<\\rho,\\;\\; \\norm{x^*}<\\alpha,\\;\\mbox{ and }\\; \\norm{\\hat y^*}<\\rho'\\alpha\\le \\rho.$$\r\nHence, $\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x',\\hat\\omega')<\\alpha$, and consequently $\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)<\\alpha$.\r\nBy letting $\\alpha\\to \\hat\\zeta[\\bold{\\Omega}](\\bar x)$, we obtain the claimed inequality.\r\n\r\n(ii) follows from (i) and Proposition~\\ref{T3.1.1}~(ii).\r\n\\qed\\end{proof}\n\n\\begin{prop}\\label{T3.1.3}\r\nLet $X$ be an Asplund space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n\\begin{enumerate}\r\n\\item\r\n$\\hat\\zeta^*_2[\\bold{\\Omega}](\\bar x)\\le \\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)$, where $\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)$ is given by \\eqref{strsubslo'},\r\n\\begin{equation}\\label{c2}\r\n\\hat\\zeta^{*}_2[\\bold{\\Omega}](\\bar x):= \\lim_{\\rho\\downarrow 0} \\inf_{\\substack{\\norm{x-\\bar x}<\\rho\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\lim_{\\varepsilon\\downarrow 0} \\inf_{\\substack{\\norm{x'-x}<\\varepsilon\\\\ \\hat\\omega'\\in \\widehat\\Omega\\\\ \\norm{\\hat\\omega'-\\hat\\omega}<\\varepsilon}}\r\n\\zeta_{\\rho,\\varepsilon,2}^{*}[\\bold{\\Omega}] (x',\\hat\\omega')\r\n\\end{equation}\r\nand, for $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega$ with $(x-\\omega_1,\\ldots,x-\\omega_m)\\ne0$,\r\n\\begin{align}\r\n\\zeta_{\\rho,\\varepsilon,2}^{*}[\\bold{\\Omega}](x,\\hat\\omega):= \\inf\\Biggl\\{\r\n\\left\\|\\sum\\limits_{i=1}^m{x}_i^*\\right\\|:\\,\r\n&\r\nx_{i}^*\\in N_{\\Omega_i}(\\omega_i) +\\rho\\B^*\\quad (i=1,\\ldots,m),\r\n\\notag\\\\&\r\nx_i^*=0\r\n\\quad\\mbox{if}\\quad\r\n\\norm{x-\\omega_i}<\\max_{1\\le j\\le m} \\norm{x-\\omega_j},\r\n\\notag\\\\&\r\n\\langle x_i^*,x-\\omega_i\\rangle\r\n\\ge\\|x_i^*\\|(\\|x-\\omega_i\\|-\\varepsilon),\r\n\\notag\\\\&\r\n\\sum_{i=1}^m\\|x_i^*\\|=1\r\n\\Biggr\\}.\\label{c2r}\r\n\\end{align}\r\n\\item\r\nIf $\\hat\\zeta^*_2[\\bold{\\Omega}](\\bar x)>0$, then $\\bold{\\Omega}$ is subregular at $\\bar x$.\r\n\\end{enumerate}\r\n\\end{prop}\n\n\\begin{proof}\r\n(i)\r\nLet $\\rho>0$, $x\\in X$, $\\hat\\omega:= (\\omega_1,\\ldots,\\omega_m)\\in\\widehat\\Omega$ with $\\|x-\\bar x\\|<\\rho$, ${0<\\max_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}$, $(u^*,\\hat v^*) \\in\\partial f(x,\\hat\\omega)$, where $f$ is given by \\eqref{f}, and $\\|\\hat v^*\\|<\\rho$.\r\nDenote $\\hat v:=(x-\\omega_1,\\ldots,x-\\omega_m)$.\r\nThen $0<\\|\\hat v\\|<\\rho$.\r\nObserve that function $f$ is the sum of two functions on $X^{m+1}$:\r\n$$(x,\\hat x)\\mapsto \\varphi(x,\\hat x):= \\|(x-x_1,\\ldots,x-x_m)\\|\\quad\\mbox{and}\\quad(x,\\hat x) \\mapsto\\delta_{\\widehat\\Omega}(\\hat x),$$\r\nwhere $\\hat x:=(x_1,\\ldots,x_m)$ and $\\delta_{\\widehat\\Omega}$ is the indicator function of $\\widehat\\Omega$.\r\nThe first function is Lipschitz continuous while the second one is lower semicontinuous.\r\nOne can apply Lemma~\\ref{l02}.\r\nFor any $\\varepsilon>0$, there exist points $x'\\in X$, $\\hat x:=(x_1,\\ldots,x_m)\\in X^m$, $\\hat\\omega':=(\\omega_1',\\ldots,\\omega_m') \\in\\widehat\\Omega$, $\\left(x^*,\\hat y^*\\right)\\in\\partial \\varphi(x',\\hat x)$, and $\\hat\\omega^*\\in N_{\\widehat\\Omega}(\\hat\\omega')$ such that\r\n\\begin{gather}\\notag\r\n\\|x'-x\\|<\\varepsilon,\\quad \\|\\hat x-\\hat\\omega\\|<\\frac{\\varepsilon}{4},\\quad \\|\\hat\\omega'-\\hat\\omega\\|<\\frac{\\varepsilon}{4},\r\n\\\\\\label{est}\r\n\\|(u^*,\\hat v^*)-(x^*,\\hat y^*)-(0,\\hat\\omega^*)\\|<\\varepsilon.\r\n\\end{gather}\r\nTaking a smaller $\\varepsilon$ if necessary, one can ensure that $\\hat v':=(x'-\\omega_1',\\ldots,x'-\\omega_m')\\ne0$, $\\hat v'':=(x'-x_1,\\ldots,x'-x_m)\\ne0$, $\\|\\hat v^*\\|+\\varepsilon<\\rho$\r\nand,\r\nfor any $i=1,\\ldots,m$, ${\\norm{x'-x_i}<\\max_{1\\le j\\le m} \\norm{x'-x_j}}$ if and only if ${\\norm{x'-\\omega_i'}<\\max_{1\\le j\\le m} \\norm{x'-\\omega_j'}}$.\r\nBy Lemma~\\ref{ll02},\r\n$$\\hat x^*:=-\\hat y^*\\in J(\\hat v'')\r\n\\quad\\mbox{and}\\quad\r\nx^*=x_{1}^*+\\ldots+x_{m}^*,$$\r\nwhere\r\n$\\hat x^*=(x_1^*,\\ldots,x_m^*)$.\r\nBy Proposition~\\ref{prJ},\r\n\\begin{gather*}\r\n\\sum_{i=1}^m\\|x_i^*\\|=1,\r\n\\\\\r\nx_i^*=0\r\n\\quad\\mbox{if}\\quad\r\n\\norm{x'-\\omega_i'}<\\max_{1\\le j\\le m} \\norm{x'-\\omega_j'},\r\n\\end{gather*}\r\n\\begin{align*}\r\n\\langle x_i^*,x'-\\omega_i'\\rangle\r\n&\\ge\\langle x_i^*,x'-x_i\\rangle-\\|x_i^*\\|\\,\\|x_i-\\omega_i'\\| =\\|x_i^*\\|(\\|x'-x_i\\|-\\|x_i-\\omega_i'\\|)\r\n\\\\&\r\n\\ge\\|x_i^*\\|(\\|x'-\\omega_i'\\|-2\\|x_i-\\omega_i'\\|)\r\n\\ge\\|x_i^*\\|(\\|x'-\\omega_i'\\|-\\varepsilon) \\quad(i=1,\\ldots,m).\r\n\\end{align*}\r\nInequality \\eqref{est} yields the estimates:\r\n$\\|u^*\\|>\\left\\|{x}^*\\right\\|-\\varepsilon$, $\\left\\|\\hat x^*-\\hat\\omega^*\\right\\|<\\|\\hat v^*\\|+\\varepsilon<\\rho$,\r\nand consequently\r\n$$\r\n\\|u^*\\|> \\left\\|\\sum\\limits_{i=1}^m{x}_i^*\\right\\|-\\varepsilon, \\quad \\hat x^*\\in N_{\\widehat\\Omega}(\\hat\\omega') +\\rho\\B_m^*.\r\n$$\r\nIt follows from Lemma~\\ref{ll03} and definitions \\eqref{subrhoslo} and \\eqref{c2r} that\r\n$$\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x,\\hat\\omega)\\ge \\zeta_{\\rho,\\varepsilon,2}^*[\\bold{\\Omega}] (x',\\hat\\omega')-\\varepsilon.$$\r\nThe claimed inequality is a consequence of the last one and definitions \\eqref{strsubslo'} and \\eqref{c2}.\r\n\r\n(ii) follows from (i) and Proposition~\\ref{T3.1.2}~(ii).\r\n\\qed\\end{proof}\n\n\\begin{proof}\\emph{of Theorem~\\ref{T3.1}}\r\n(i) follows from Proposition~\\ref{T3.1.3}~(ii) and definitions \\eqref{c2} and \\eqref{c2r}.\r\n\r\n(ii) is a consequence of \\cite[Theorem~4]{Kru09.1}.\r\n\\qed\\end{proof}","units":[{"raw_tex":"\\begin{prop}\\label{theorem11}\r\n\\begin{enumerate}\r\n\\item\r\n$\\bold{\\Omega}$ is semiregular at $\\bar x$ if and only if $\\theta[\\bold{\\Omega}](\\bar{x})>0$. Moreover, $\\theta[\\bold{\\Omega}](\\bar{x})$ is the exact upper bound of all numbers $\\alpha$ such that (\\ref{HR}) is satisfied.\r\n\\item\r\n$\\bold{\\Omega}$ is subregular at $\\bar x$ if and only if $\\zeta[\\bold{\\Omega}](\\bar{x})>0$. Moreover, $\\zeta[\\bold{\\Omega}](\\bar{x})$ is the exact upper bound of all numbers $\\alpha$ such that (\\ref{Hlr'}) is satisfied.\r\n\\item\r\n$\\bold{\\Omega}$ is uniformly regular at $\\bar x$ if and only if $\\hat{\\theta}[\\bold{\\Omega}](\\bar{x})>0$. Moreover, $\\hat{\\theta}[\\bold{\\Omega}](\\bar{x})$ is the exact upper bound of all numbers $\\alpha$ such that (\\ref{HUR}) is satisfied.\r\n\\end{enumerate}\r\n\\end{prop}","start":11647,"end":12429},{"raw_tex":"\\begin{prop}\\label{prJ}\r\nFor each $(x_1,\\ldots,x_m)\\in X^m$,\r\n\\begin{multline*}\r\nJ(x_1,\\ldots,x_m)= \\Biggl\\{(x_1^*,\\ldots,x_m^*)\\in(X^*)^m:\\; \\sum_{i=1}^m\\|x_i^*\\|=1;\\mbox{ either } x_i^*=0\\\\\r\n\\mbox{or }\\left(\\|x_i\\|=\\max_{1\\le j\\le m} \\|x_j\\|,\\; x_i^*\\in\\|x_i^*\\| J(x_i)\\right)\\; (i=1,\\ldots,m)\\Biggr\\}.\r\n\\end{multline*}\r\n\\end{prop}","start":29399,"end":29732},{"raw_tex":"\\begin{proof}\r\nLet $\\hat x:=(x_1,\\ldots,x_m)\\in X^m$, $\\hat x^*:=(x_1^*,\\ldots,x_m^*)\\in(X^*)^m$.\r\nThen\r\n$${\\|\\hat x\\|=\\max_{1\\le i\\le m} \\|x_i\\|},\\quad \\|\\hat x^*\\|=\\sum_{i=1}^m\\|x_i^*\\|,\\quad \\langle \\hat x^*,\\hat x\\rangle=\\sum_{i=1}^m\\langle x_i^*,x_i\\rangle.$$\r\nSuppose $\\|\\hat x^*\\|=1$, i.e., $\\sum_{i=1}^m\\|x_i^*\\|=1$.\r\nThen $\\hat x^*\\in J(\\hat x)$ if and only if ${\\sum_{i=1}^m\\langle x_i^*,x_i\\rangle=\\|\\hat x\\|}$.\r\nIn its turn, the last equality holds true if and only if ${\\langle x_i^*,x_i\\rangle=\\|x_i^*\\|\\cdot\\|\\hat x\\|}$ for all $i=1,\\ldots,m$.\r\n\r\nIndeed, if $\\langle x_i^*,x_i\\rangle=\\|x_i^*\\|\\cdot\\|\\hat x\\|$ for all $i=1,\\ldots,m$, then adding these $m$ equalities, we obtain $\\sum_{i=1}^m\\langle x_i^*,x_i\\rangle=\\|\\hat x\\|$.\r\nConversely, if $\\langle x_i^*,x_i\\rangle\\ne\\|x_i^*\\|\\cdot\\|\\hat x\\|$, i.e., ${\\langle x_i^*,x_i\\rangle<\\|x_i^*\\|\\cdot\\|\\hat x\\|}$ for some $i\\in\\{1,\\ldots,m\\}$, then\r\n$$\r\n\\sum_{j=1}^m\\langle x_j^*,x_j\\rangle =\\langle x_i^*,x_i\\rangle+\\sum_{j\\ne i}\\langle x_j^*,x_j\\rangle <\\|x_i^*\\|\\cdot\\|\\hat x\\|+\\|\\hat x\\|\\sum_{j\\ne i} \\|x_j^*\\|=\\|\\hat x\\|.\r\n$$\r\n\r\nFinally, $\\langle x_i^*,x_i\\rangle=\\|x_i^*\\|\\cdot\\|\\hat x\\|$ for some $i\\in\\{1,\\ldots,m\\}$ if and only if either $\\|x_i\\|=\\|\\hat x\\|$ and $x_i^*\\in\\|x_i^*\\| J(x_i)$ or $x_i^*=0$.\r\n\\qed\\end{proof}","start":29734,"end":31025},{"raw_tex":"\\begin{lem}\\label{ll02}\r\nLet $X$ be a normed space and\r\n$\\varphi(u,\\hat u)=\\|(u-u_1,\\ldots,u-u_m)\\|$, $u\\in X$, $\\hat u:=(u_1,\\ldots,u_m)\\in X^m$.\r\nSuppose $x\\in X$, $\\hat x:=(x_1,\\ldots,x_m)\\in X^m$, and ${\\hat v:=(x-x_1,\\ldots,x-x_m)}\\ne0$. Then\r\n\\begin{align*}\r\n\\sd\\varphi(x,\\hat x)\\subseteq\\bigl\\{\\left(x^*,\\hat x^* =(x_1^*,\\ldots,x_m^*)\\right)\\in &X^*\\times (X^*)^m :\\;\\\\\r\n&-\\hat x^*\\in J(\\hat v),\\; x^*=-(x_{1}^*+\\ldots+x_{m}^*)\\bigr\\}.\r\n\\end{align*}\r\n\\end{lem}","start":31939,"end":32406},{"raw_tex":"\\begin{proof}\r\n%Observe that $\\varphi$ is a composition of the norm on $X^m$ and the linear continuous mapping $(x,\\hat x)\\mapsto(x_1-x,\\ldots,x_m-x)$.\r\nLet $\\left(x^*,\\hat x^* =(x_1^*,\\ldots,x_m^*)\\right)\\in \\sd\\varphi(x,\\hat x)$, i.e.,\r\n$$\r\n\\|(u-u_1,\\ldots,u-u_m)\\|-\\|(x-x_1,\\ldots,x-x_m)\\|\\ge \\langle x^*,u-x\\rangle+\\sum_{i=1}^m\\langle x_i^*,u_i-x_i\\rangle\r\n$$\r\nfor any $u\\in X$ and $\\hat u:=(u_1,\\ldots,u_m)\\in X^m$.\r\nIn particular, with $u=x$ and $u_i=x_i-x_i'$ ($i=1,\\ldots,m$) for an arbitrary $\\hat x':=(x_1',\\ldots,x_m')\\in X^m$, we have\r\n$$\r\n\\|\\hat v+\\hat x'\\|-\\|\\hat v\\|\\ge-\\langle\\hat x^*,\\hat x'\\rangle,\r\n$$\r\ni.e., $-\\hat x^*\\in J(\\hat v)$.\r\nSimilarly, with $u=x+x'$ and $u_i=x_i+x'$ ($i=1,\\ldots,m$) for an arbitrary $x'\\in X$, we have\r\n$$\r\n\\left\\langle x^*+\\sum_{i=1}^mx_i^*,x'\\right\\rangle\\le0,\r\n$$\r\nand consequently $x^*+x_{1}^*+\\ldots+x_{m}^*=0$.\r\n\\qed\\end{proof}","start":32408,"end":33289},{"raw_tex":"\\begin{lem}\\label{ll03}\r\nLet $X$ be a normed space and $\\hat\\omega:=(\\omega_1,\\ldots,\\omega_m)\\in\\widehat\\Omega$.\r\nThen ${N_{\\widehat\\Omega}(\\hat\\omega) =N_{\\Omega_1}(\\omega_1)\\times\\ldots\\times N_{\\Omega_m}(\\omega_m)}$.\r\n\\end{lem}","start":33293,"end":33524},{"raw_tex":"\\begin{proof}\r\nfollows directly from the definition of the Fr\\'echet normal cone.\r\n\\qed\\end{proof}","start":33526,"end":33624},{"raw_tex":"\\begin{prop}\\label{T3.1.1}\r\nLet $X$ be a Banach space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n\\begin{enumerate}\r\n\\item\r\n$\\hat\\zeta[\\bold{\\Omega}](\\bar x)\\le \\zeta[\\bold{\\Omega}](\\bar{x})$, where\r\n\\begin{equation}\\label{strslo}\r\n\\hat\\zeta[\\bold{\\Omega}](\\bar x):= \\lim_{\\rho\\downarrow0} \\inflarge_{\\substack{\\norm{x-\\bar x}<\\rho\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega)\r\n\\end{equation}\r\nand, for $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega$,\r\n\\begin{equation}\\label{rhoslo}\r\n\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega):= \\limsup_{\\substack{(u,\\hat v)\\to(x,\\hat\\omega)\\\\(u,\\hat v)\\neq (x,\\hat\\omega)\\\\ \\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega}} \\frac{\\left(\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}-\\max\\limits_{1\\le i\\le m} \\norm{u-v_i}\\right)_+}{\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho}}.\r\n\\end{equation}\r\n\\item\r\nIf $\\hat\\zeta[\\bold{\\Omega}](\\bar x)>0$, then $\\bold{\\Omega}$ is subregular at $\\bar x$.\r\n\\end{enumerate}\r\n\\end{prop}","start":37379,"end":38457},{"raw_tex":"\\begin{proof}\r\n(i)\r\nLet $\\zeta[\\bold{\\Omega}](\\bar{x})<\\alpha<\\infty$. Choose a $\\rho\\in ]0,1[$ and set\r\n\\begin{equation}\\label{28}\r\n\\eta:= \\min\\left\\{\\frac{\\rho}{2},\\frac{\\rho}{\\alpha},\\rho^{\\frac{2}{\\rho}}\\right\\}.\r\n\\end{equation}\r\nBy \\eqref{errcon}, there are $x'\\in B_{\\eta}(\\bar x)$ and $\\hat\\omega'=(\\omega_1',\\ldots,\\omega_m')\\in \\widehat\\Omega$ such that\r\n\\begin{equation}\\label{29}\r\n   0<f(x',\\hat\\omega')<\\alpha d\\left(x',\\bigcap_{i=1}^m\\Omega_i\\right).\r\n\\end{equation}\r\nDenote $\\varepsilon:=f(x',\\hat\\omega')$ and $\\mu:=d\\left(x',\\bigcap_{i=1}^m\\Omega_i\\right)$. Then $\\mu\\le \\norm{x'-\\bar x}\\le \\eta\\le \\frac{\\rho}{2}<1$.\r\nObserve that $f$ is lower semicontinuous.\r\nApplying to $f$ Lemma~\\ref{l01} with $\\varepsilon$ as above and\r\n\\begin{equation}\\label{30}\r\n\\lambda:=\\mu(1-\\mu^{\\frac{\\rho}{2-\\rho}}),\r\n\\end{equation}\r\nwe find points $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in X^{m}$ such that\r\n\\begin{equation}\\label{31}\r\n   \\norm{(x,\\hat\\omega)-(x',\\hat\\omega')}_{\\rho}<\\lambda,\\; f(x,\\hat\\omega)\\le f(x',\\hat\\omega'),\r\n\\end{equation}\r\nand\r\n\\begin{equation}\\label{32}\r\nf(u,\\hat v)+\\frac{\\varepsilon}{\\lambda}\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho} \\ge f(x,\\hat\\omega),\r\n\\end{equation}\r\nfor all $(u,\\hat v)\\in X\\times X^{m}$.\r\nThanks to \\eqref{31}, \\eqref{30}, \\eqref{28}, and \\eqref{29}, we have\r\n$$\r\n\\norm{x-x'}<\\lambda <\\mu \\le \\norm{x'-\\bar x},\r\n$$\r\n\\begin{equation}\\label{33}\r\n   d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right)\\ge d\\left(x',\\bigcap_{i=1}^m\\Omega_i\\right)- \\norm{x-x'}\\ge \\mu-\\lambda =\\mu^{\\frac{2}{2-\\rho}},\r\n\\end{equation}\r\n\\begin{equation}\\label{34}\r\n\\norm{x-\\bar x}\\le \\norm{x-x'}+\\norm{x'-\\bar x}<2\\norm{x'-\\bar x}\\le 2\\eta\\le\\rho,\r\n\\end{equation}\r\n\\begin{equation}\\label{35}\r\nf(x,\\hat\\omega)\\le f(x',\\hat\\omega')<\\alpha\\mu\\le \\alpha\\eta\\le\\rho.\r\n\\end{equation}\r\nIt follows from \\eqref{33}, \\eqref{34}, and \\eqref{35} that\r\n$$\r\n\\norm{x-\\bar x}<\\rho,\\; \\hat\\omega\\in \\widehat\\Omega,\\; 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho.\r\n$$\r\nObserve that $\\mu^{\\frac{\\rho}{2-\\rho}}\\le \\eta^{\\frac{\\rho}{2-\\rho}}< \\eta^{\\frac{\\rho}{2}}\\le \\rho$, and consequently, by \\eqref{29} and \\eqref{30},\r\n$$\r\n\\frac{\\varepsilon}{\\lambda} <\\frac{\\alpha\\mu}{\\lambda} =\\frac{\\alpha}{1-\\mu^{\\frac{\\rho}{2-\\rho}}} <\\frac{\\alpha}{1-\\rho}.\r\n$$\r\nThanks to \\eqref{32} and \\eqref{f}, we have\r\n$$\r\n\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}-\\max\\limits_{1\\le i\\le m} \\norm{u-v_i}\\le \\frac{\\alpha}{1-\\rho}\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho}\r\n$$\r\nfor all $u\\in X$ and $\\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega$.\r\nIt follows that $\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega)\\le \\dfrac{\\alpha}{1-\\rho}$ and consequently\r\n$$\r\n\\inflarge_{\\substack{\\norm{x-\\bar x}<\\rho\\\\\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega)\\le \\frac{\\alpha}{1-\\rho}.\r\n$$\r\nTaking limits in the last inequality as $\\rho\\downarrow 0$ and $\\alpha\\to \\zeta[\\bold{\\Omega}](\\bar{x})$ yields the claimed inequality.\r\n\r\n(ii) follows from (i) and Proposition~\\ref{theorem11}~(ii).\r\n\\qed\\end{proof}","start":38461,"end":41548},{"raw_tex":"\\begin{prop}\\label{T3.1.2}\r\nLet $X$ be an Asplund space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n\\begin{enumerate}\r\n\\item\r\n$\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)\\le \\hat\\zeta[\\bold{\\Omega}](\\bar x)$, where $\\hat\\zeta[\\bold{\\Omega}](\\bar x)$ is given by \\eqref{strslo},\r\n\\begin{equation}\\label{strsubslo'}\r\n\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x):= \\lim_{\\rho\\downarrow 0} \\inflarge_{\\substack{\\norm{x-\\bar x}<\\rho\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x,\\hat\\omega)\r\n\\end{equation}\r\nand, for $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega$,\r\n\\begin{equation}\\label{subrhoslo}\r\n\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x,\\hat\\omega):= \\inflarge_{\\substack{(x^*,\\hat y^*)\\in\\partial f(x,\\hat\\omega)\\\\ \\norm{\\hat y^*}<\\rho}}\\norm{x^*}\r\n\\end{equation}\r\n(with the convention that the infimum over the empty set equals $+\\infty$).\r\n\\item\r\nIf $\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)>0$, then $\\bold{\\Omega}$ is subregular at $\\bar x$.\r\n\\end{enumerate}\r\n\\end{prop}","start":41552,"end":42630},{"raw_tex":"\\begin{proof}\r\nLet $\\hat\\zeta[\\bold{\\Omega}](\\bar x)< \\alpha<\\infty$.\r\nChoose a $\\beta\\in ]\\hat\\zeta[\\bold{\\Omega}](\\bar x),\\alpha[$ and an arbitrary ${\\rho>0}$.\r\nSet ${\\rho'=\\min\\{1,\\alpha^{-1}\\}\\rho}$.\r\nBy \\eqref{strslo} and \\eqref{rhoslo}, one can find points $x\\in X$ and ${\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega}$ such that $\\norm{x-\\bar x}<\\rho'$, $0<\\max_{1\\le i\\le m} \\norm{\\omega_i-x}<\\rho'$, and\r\n$$\r\n\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}-\\max\\limits_{1\\le i\\le m} \\norm{u-v_i}\\le \\beta\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho'}\r\n$$\r\nfor all $(u,\\hat v)$ with $\\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega$ near $(x,\\hat\\omega)$.\r\nIn other words, $(x,\\hat\\omega)$ is a local minimizer of the function\r\n$$\r\n(u,\\hat v)\\mapsto \\max\\limits_{1\\le i\\le m} \\norm{u-v_i}+\\beta\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho'}\r\n$$\r\nsubject to $\\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega$.\r\nBy definition \\eqref{f}, this means that $(x,\\hat\\omega)$ minimizes locally the function\r\n$$\r\n(u,\\hat v)\\mapsto f(u,\\hat v)+\\beta\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho'},\r\n$$\r\nand consequently its Fr\\'echet subdifferential at $(x,\\hat\\omega)$ contains zero.\r\nTake an\r\n$$\r\n\\varepsilon\\in\\Bigl]0,\\min\\{\\rho-\\norm{x-\\bar x},\\rho-\\max_{1\\le i\\le m} \\norm{x-\\omega_i},\\alpha-\\beta\\}\\Bigr[.\r\n$$\r\nApplying Lemma~\\ref{l02} and Lemma~\\ref{ll01}~(ii), we can find points $x'\\in X$, ${\\hat\\omega'=(\\omega_1',\\ldots,\\omega_m')\\in \\widehat\\Omega}$, and $(x^*,\\hat y^*)\\in \\partial f(x',\\hat\\omega')$ such that $$\\norm{x'-x}<\\varepsilon,\\quad 0<\\max_{1\\le i\\le m} \\norm{x'-\\omega_i'}\\le \\max_{1\\le i\\le m} \\norm{x-\\omega_i}+\\varepsilon,$$\r\n$$\\mbox{and}\\quad\\norm{(x^*,\\hat y^*)}_{\\rho'} =\\|x^*\\|+\\|\\hat y^*\\|/\\rho' <\\beta+\\varepsilon.$$\r\nIt follows that\r\n$$\\norm{x'-\\bar x}<\\rho,\\;\\; 0<\\max_{1\\le i\\le m} \\norm{x'-\\omega_i'}<\\rho,\\;\\; \\norm{x^*}<\\alpha,\\;\\mbox{ and }\\; \\norm{\\hat y^*}<\\rho'\\alpha\\le \\rho.$$\r\nHence, $\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x',\\hat\\omega')<\\alpha$, and consequently $\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)<\\alpha$.\r\nBy letting $\\alpha\\to \\hat\\zeta[\\bold{\\Omega}](\\bar x)$, we obtain the claimed inequality.\r\n\r\n(ii) follows from (i) and Proposition~\\ref{T3.1.1}~(ii).\r\n\\qed\\end{proof}","start":42634,"end":44826},{"raw_tex":"\\begin{prop}\\label{T3.1.3}\r\nLet $X$ be an Asplund space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n\\begin{enumerate}\r\n\\item\r\n$\\hat\\zeta^*_2[\\bold{\\Omega}](\\bar x)\\le \\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)$, where $\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)$ is given by \\eqref{strsubslo'},\r\n\\begin{equation}\\label{c2}\r\n\\hat\\zeta^{*}_2[\\bold{\\Omega}](\\bar x):= \\lim_{\\rho\\downarrow 0} \\inf_{\\substack{\\norm{x-\\bar x}<\\rho\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\lim_{\\varepsilon\\downarrow 0} \\inf_{\\substack{\\norm{x'-x}<\\varepsilon\\\\ \\hat\\omega'\\in \\widehat\\Omega\\\\ \\norm{\\hat\\omega'-\\hat\\omega}<\\varepsilon}}\r\n\\zeta_{\\rho,\\varepsilon,2}^{*}[\\bold{\\Omega}] (x',\\hat\\omega')\r\n\\end{equation}\r\nand, for $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega$ with $(x-\\omega_1,\\ldots,x-\\omega_m)\\ne0$,\r\n\\begin{align}\r\n\\zeta_{\\rho,\\varepsilon,2}^{*}[\\bold{\\Omega}](x,\\hat\\omega):= \\inf\\Biggl\\{\r\n\\left\\|\\sum\\limits_{i=1}^m{x}_i^*\\right\\|:\\,\r\n&\r\nx_{i}^*\\in N_{\\Omega_i}(\\omega_i) +\\rho\\B^*\\quad (i=1,\\ldots,m),\r\n\\notag\\\\&\r\nx_i^*=0\r\n\\quad\\mbox{if}\\quad\r\n\\norm{x-\\omega_i}<\\max_{1\\le j\\le m} \\norm{x-\\omega_j},\r\n\\notag\\\\&\r\n\\langle x_i^*,x-\\omega_i\\rangle\r\n\\ge\\|x_i^*\\|(\\|x-\\omega_i\\|-\\varepsilon),\r\n\\notag\\\\&\r\n\\sum_{i=1}^m\\|x_i^*\\|=1\r\n\\Biggr\\}.\\label{c2r}\r\n\\end{align}\r\n\\item\r\nIf $\\hat\\zeta^*_2[\\bold{\\Omega}](\\bar x)>0$, then $\\bold{\\Omega}$ is subregular at $\\bar x$.\r\n\\end{enumerate}\r\n\\end{prop}","start":44830,"end":46305},{"raw_tex":"\\begin{proof}\r\n(i)\r\nLet $\\rho>0$, $x\\in X$, $\\hat\\omega:= (\\omega_1,\\ldots,\\omega_m)\\in\\widehat\\Omega$ with $\\|x-\\bar x\\|<\\rho$, ${0<\\max_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}$, $(u^*,\\hat v^*) \\in\\partial f(x,\\hat\\omega)$, where $f$ is given by \\eqref{f}, and $\\|\\hat v^*\\|<\\rho$.\r\nDenote $\\hat v:=(x-\\omega_1,\\ldots,x-\\omega_m)$.\r\nThen $0<\\|\\hat v\\|<\\rho$.\r\nObserve that function $f$ is the sum of two functions on $X^{m+1}$:\r\n$$(x,\\hat x)\\mapsto \\varphi(x,\\hat x):= \\|(x-x_1,\\ldots,x-x_m)\\|\\quad\\mbox{and}\\quad(x,\\hat x) \\mapsto\\delta_{\\widehat\\Omega}(\\hat x),$$\r\nwhere $\\hat x:=(x_1,\\ldots,x_m)$ and $\\delta_{\\widehat\\Omega}$ is the indicator function of $\\widehat\\Omega$.\r\nThe first function is Lipschitz continuous while the second one is lower semicontinuous.\r\nOne can apply Lemma~\\ref{l02}.\r\nFor any $\\varepsilon>0$, there exist points $x'\\in X$, $\\hat x:=(x_1,\\ldots,x_m)\\in X^m$, $\\hat\\omega':=(\\omega_1',\\ldots,\\omega_m') \\in\\widehat\\Omega$, $\\left(x^*,\\hat y^*\\right)\\in\\partial \\varphi(x',\\hat x)$, and $\\hat\\omega^*\\in N_{\\widehat\\Omega}(\\hat\\omega')$ such that\r\n\\begin{gather}\\notag\r\n\\|x'-x\\|<\\varepsilon,\\quad \\|\\hat x-\\hat\\omega\\|<\\frac{\\varepsilon}{4},\\quad \\|\\hat\\omega'-\\hat\\omega\\|<\\frac{\\varepsilon}{4},\r\n\\\\\\label{est}\r\n\\|(u^*,\\hat v^*)-(x^*,\\hat y^*)-(0,\\hat\\omega^*)\\|<\\varepsilon.\r\n\\end{gather}\r\nTaking a smaller $\\varepsilon$ if necessary, one can ensure that $\\hat v':=(x'-\\omega_1',\\ldots,x'-\\omega_m')\\ne0$, $\\hat v'':=(x'-x_1,\\ldots,x'-x_m)\\ne0$, $\\|\\hat v^*\\|+\\varepsilon<\\rho$\r\nand,\r\nfor any $i=1,\\ldots,m$, ${\\norm{x'-x_i}<\\max_{1\\le j\\le m} \\norm{x'-x_j}}$ if and only if ${\\norm{x'-\\omega_i'}<\\max_{1\\le j\\le m} \\norm{x'-\\omega_j'}}$.\r\nBy Lemma~\\ref{ll02},\r\n$$\\hat x^*:=-\\hat y^*\\in J(\\hat v'')\r\n\\quad\\mbox{and}\\quad\r\nx^*=x_{1}^*+\\ldots+x_{m}^*,$$\r\nwhere\r\n$\\hat x^*=(x_1^*,\\ldots,x_m^*)$.\r\nBy Proposition~\\ref{prJ},\r\n\\begin{gather*}\r\n\\sum_{i=1}^m\\|x_i^*\\|=1,\r\n\\\\\r\nx_i^*=0\r\n\\quad\\mbox{if}\\quad\r\n\\norm{x'-\\omega_i'}<\\max_{1\\le j\\le m} \\norm{x'-\\omega_j'},\r\n\\end{gather*}\r\n\\begin{align*}\r\n\\langle x_i^*,x'-\\omega_i'\\rangle\r\n&\\ge\\langle x_i^*,x'-x_i\\rangle-\\|x_i^*\\|\\,\\|x_i-\\omega_i'\\| =\\|x_i^*\\|(\\|x'-x_i\\|-\\|x_i-\\omega_i'\\|)\r\n\\\\&\r\n\\ge\\|x_i^*\\|(\\|x'-\\omega_i'\\|-2\\|x_i-\\omega_i'\\|)\r\n\\ge\\|x_i^*\\|(\\|x'-\\omega_i'\\|-\\varepsilon) \\quad(i=1,\\ldots,m).\r\n\\end{align*}\r\nInequality \\eqref{est} yields the estimates:\r\n$\\|u^*\\|>\\left\\|{x}^*\\right\\|-\\varepsilon$, $\\left\\|\\hat x^*-\\hat\\omega^*\\right\\|<\\|\\hat v^*\\|+\\varepsilon<\\rho$,\r\nand consequently\r\n$$\r\n\\|u^*\\|> \\left\\|\\sum\\limits_{i=1}^m{x}_i^*\\right\\|-\\varepsilon, \\quad \\hat x^*\\in N_{\\widehat\\Omega}(\\hat\\omega') +\\rho\\B_m^*.\r\n$$\r\nIt follows from Lemma~\\ref{ll03} and definitions \\eqref{subrhoslo} and \\eqref{c2r} that\r\n$$\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x,\\hat\\omega)\\ge \\zeta_{\\rho,\\varepsilon,2}^*[\\bold{\\Omega}] (x',\\hat\\omega')-\\varepsilon.$$\r\nThe claimed inequality is a consequence of the last one and definitions \\eqref{strsubslo'} and \\eqref{c2}.\r\n\r\n(ii) follows from (i) and Proposition~\\ref{T3.1.2}~(ii).\r\n\\qed\\end{proof}","start":46309,"end":49318},{"raw_tex":"\\begin{proof}\\emph{of Theorem~\\ref{T3.1}}\r\n(i) follows from Proposition~\\ref{T3.1.3}~(ii) and definitions \\eqref{c2} and \\eqref{c2r}.\r\n\r\n(ii) is a consequence of \\cite[Theorem~4]{Kru09.1}.\r\n\\qed\\end{proof}","start":49322,"end":49527}]},"nodes":[{"id":"S001","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"空间、对偶与乘积范数","source_units":[{"raw_tex":"Our basic notation is standard; cf. \\cite{Mor06.1,RocWet98}.\r\nFor a normed linear space $X$, its topological dual is denoted $X^*$, while $\\langle\\cdot,\\cdot\\rangle$ denotes the bilinear form defining the pairing between the two spaces.\r\nThe closed unit ball in a normed space is denoted $\\B $, $B_\\delta(x)$ stands for the closed ball with radius $\\delta$ and centre $x$.\r\nProducts of normed spaces will be considered with the maximum type norms, if not specified otherwise.\r\n\r\n","start":6233,"end":6712}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"Fréchet法锥","source_units":[{"raw_tex":"The Fr\\'echet normal cone to a set $\\Omega\\subset X$ at $x\\in \\Omega$ and the Fr\\'echet subdifferential of a function $f:X \\to\\R_\\infty:=\\R\\cup\\{+\\infty\\}$ at a point $x$ with $f(x)<\\infty$ are defined, respectively, by\r\n\\begin{equation*}\r\nN_{\\Omega}(x):= \\left\\{x^* \\in X^* :\\; \\limsup_{u\\to{x},\\,u\\in\\Omega\\setminus\\{x\\}} \\frac {\\langle x^*,u-x \\rangle}{\\|u-x\\|} \\leq 0 \\right\\},\r\n\\end{equation*}\r\n","start":6712,"end":7112}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"Fréchet次微分","source_units":[{"raw_tex":"\\begin{equation*}\r\n\\partial f(x):=\\left\\{x^*\\in X^*:\\; \\liminf_{\\substack{u\\to x,\\,u\\neq x}} \\frac{f(u)-f(x)-\\langle x^*,u-x\\rangle}{\\norm{u-x}}\\ge 0\\right\\}.\r\n\\end{equation*}\r\n","start":7112,"end":7289}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"集合距离","source_units":[{"raw_tex":"\r\nFor a given set $\\Omega\\subset X$, the distance function associated with $\\Omega$ is defined by\r\n$$\r\nd(x,\\Omega):=\\inf_{\\omega\\in\\Omega}\\norm{x-\\omega},\\; \\forall x\\in X.\r\n$$\r\n","start":7289,"end":7467}]},{"id":"S005","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"有限集合族与公共点","source_units":[{"raw_tex":"\r\nIn the sequel, $\\bold{\\Omega}$ stands for a collection of $m$ $(m\\ge 2)$ sets $\\Omega_1,\\ldots,\\Omega_m$ in a normed linear space $X$, and we assume the existence of a point $\\bar x\\in \\bigcap_{i=1}^m\\Omega_i$.\r\n","start":7467,"end":7681}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["HR"],"declared_external":false,"source_only":false,"title":"半正则性定义","source_units":[{"raw_tex":"The next definition introduces several regularity properties of $\\bold{\\Omega}$ at $\\bar x$.\r\n\r\n\\begin{dfn}\\label{HolderUR}\r\n\\begin{enumerate}\r\n\\item\r\n$\\bold{\\Omega}$ is semiregular at $\\bar x$ iff there exist positive numbers $\\alpha$ and $\\delta$ such that\r\n\\begin{equation}\\label{HR}\r\n\\bigcap_{i=1}^m(\\Omega_i-x_i)\\  \\bigcap\\ B_{\\rho}(\\bar x) \\neq \\emptyset\r\n\\end{equation}\r\nfor all $\\rho \\in ]0,\\delta[$ and all\r\n$x_i\\in{X}$ $(i=1,\\ldots,m)$ such that $\\max\\limits_{1\\leq i\\leq m}\\|x_i\\| \\le \\alpha\\rho$.\r\n","start":7931,"end":8441}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":["Hlr'","HolderUR"],"declared_external":false,"source_only":false,"title":"次正则性定义","source_units":[{"raw_tex":"\\item\r\n$\\bold{\\Omega}$ is subregular at $\\bar x$ iff there exist positive numbers $\\alpha$ and $\\delta$ such that\r\n\\begin{equation}\\label{Hlr'}\r\n\\bigcap_{i=1}^m\\left(\\Omega_i+ (\\alpha\\rho)\\B \\right)\\bigcap B_{\\delta}(\\bar x)\\subseteq\r\n\\left(\\bigcap_{i=1}^m \\Omega_i\\right)+\\rho\\B\r\n\\end{equation}\r\nfor all $\\rho\\in ]0,\\delta[$.\r\n","start":8441,"end":8769}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":["HUR"],"declared_external":false,"source_only":false,"title":"一致正则性定义","source_units":[{"raw_tex":"\\item\r\n$\\bold{\\Omega}$ is uniformly regular at $\\bar x$ iff there exist positive numbers $\\alpha$ and $\\delta$ such that\r\n\\begin{equation}\\label{HUR}\r\n\\bigcap_{i=1}^m(\\Omega_i-\\omega_i-x_i) \\bigcap(\\rho\\B )\\neq \\emptyset\r\n\\end{equation}\r\nfor all $\\rho \\in ]0,\\delta[$, $\\omega_i \\in \\Omega_i \\cap B_{\\delta}(\\bar{x})$, and all $x_i\\in{X}$ $(i=1,\\ldots,m)$ such that $\\max\\limits_{1\\leq i\\leq m}\\|x_i\\| \\le \\alpha\\rho$.\r\n\\end{enumerate}\r\n\\end{dfn}\r\n","start":8769,"end":9217}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"半正则模量","source_units":[{"raw_tex":"The regularity properties in Definition~\\ref{HolderUR} can be equivalently defined using the following nonnegative constants which provide quantitative characterizations of these properties:\r\n\\begin{gather}\\label{01.1}\r\n\\theta[\\bold{\\Omega}](\\bar{x}):= \\liminf_{\\rho \\downarrow 0} \\dfrac{\\theta_{\\rho}[\\bold{\\Omega}](\\bar{x})}{\\rho},\r\n","start":10444,"end":10779},{"raw_tex":"where, for $\\rho>0$ and $\\delta>0$,\r\n\\begin{gather}\\label{02.1}\r\n\\theta_{\\rho}[\\bold{\\Omega}](\\bar{x}):= \\sup\\left\\{r \\ge 0 :\\; \\bigcap_{i=1}^m (\\Omega_i - x_i) \\bigcap B_{\\rho}(\\bar{x}) \\neq \\emptyset,\\; \\forall x_i \\in r\\B \\right\\},\r\n","start":11135,"end":11371}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":["01.1","01.2","01.3"],"declared_external":false,"source_only":false,"title":"次正则模量","source_units":[{"raw_tex":"\\\\\\label{01.2}\r\n\\zeta[\\bold{\\Omega}](\\bar{x}):= \\lim_{\\delta\\downarrow 0}\\inf_{0<\\rho<\\delta} \\dfrac{\\zeta_{\\rho,\\delta}[\\bold{\\Omega}](\\bar{x})} {\\rho},\r\n","start":10779,"end":10934},{"raw_tex":"\\\\\\label{02.2}\r\n\\zeta_{\\rho,\\delta}[\\bold{\\Omega}](\\bar{x}):= \\sup\\left\\{r\\ge0 :\\;\\bigcap_{i=1}^m (\\Omega_i+ r\\B )\\bigcap B_{\\delta}(\\bar x)\\subseteq\r\n\\bigcap_{i=1}^m \\Omega_i+\\rho\\B \\right\\}.\r\n\\end{gather}\r\n","start":11371,"end":11579}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"一致正则模量","source_units":[{"raw_tex":"\\\\\\label{01.3}\r\n\\hat{\\theta}[\\bold{\\Omega}](\\bar{x}):= \\liminf_{\\omega_i\\TO{\\Omega_i}\\bar{x},\\rho\\downarrow0} \\dfrac{\\theta_{\\rho}[\\Omega_1-\\omega_1,\\ldots,\\Omega_m-\\omega_m](0)}{\\rho},\r\n\\end{gather}\r\n","start":10934,"end":11135}]},{"id":"S012","semantic_type":"proposition","kind":"proposition","labels":["theorem11"],"declared_external":false,"source_only":false,"title":"半正则性的模量判据","source_units":[{"raw_tex":"\\begin{prop}\\label{theorem11}\r\n\\begin{enumerate}\r\n\\item\r\n$\\bold{\\Omega}$ is semiregular at $\\bar x$ if and only if $\\theta[\\bold{\\Omega}](\\bar{x})>0$. Moreover, $\\theta[\\bold{\\Omega}](\\bar{x})$ is the exact upper bound of all numbers $\\alpha$ such that (\\ref{HR}) is satisfied.\r\n","start":11647,"end":11926}]},{"id":"S013","semantic_type":"proposition","kind":"proposition","labels":["theorem11"],"declared_external":false,"source_only":false,"title":"次正则性的模量判据","source_units":[{"raw_tex":"\\item\r\n$\\bold{\\Omega}$ is subregular at $\\bar x$ if and only if $\\zeta[\\bold{\\Omega}](\\bar{x})>0$. Moreover, $\\zeta[\\bold{\\Omega}](\\bar{x})$ is the exact upper bound of all numbers $\\alpha$ such that (\\ref{Hlr'}) is satisfied.\r\n","start":11926,"end":12154}]},{"id":"S014","semantic_type":"proposition","kind":"proposition","labels":["theorem11"],"declared_external":false,"source_only":false,"title":"一致正则性的模量判据","source_units":[{"raw_tex":"\\item\r\n$\\bold{\\Omega}$ is uniformly regular at $\\bar x$ if and only if $\\hat{\\theta}[\\bold{\\Omega}](\\bar{x})>0$. Moreover, $\\hat{\\theta}[\\bold{\\Omega}](\\bar{x})$ is the exact upper bound of all numbers $\\alpha$ such that (\\ref{HUR}) is satisfied.\r\n\\end{enumerate}\r\n\\end{prop}","start":12154,"end":12429}]},{"id":"S015","semantic_type":"proposition","kind":"proposition","labels":["errcon'","newcon","pr2","vartheta"],"declared_external":false,"source_only":false,"title":"次正则模量的度量表示","source_units":[{"raw_tex":"\\\\\\label{errcon'}\r\n\\zeta[\\bold{\\Omega}](\\bar{x})&= \\quad\\liminf_{\\substack{x\\to\\bar{x}\\\\ x\\notin \\bigcap_{i=1}^m\\Omega_i}}\r\n\\quad\\frac{\\max_{1\\le i\\le m} d(x,\\Omega_i)}{d\\Big{(}x,\\bigcap_{i=1}^m\\Omega_i\\Big{)}}\r\n\\\\\\notag\r\n&=\\liminf_{\\substack{x\\to\\bar{x}\\\\ \\omega_i\\TO{\\Omega_i}\\bar x\\; (1\\le i\\le m)\\\\ x\\notin \\bigcap_{i=1}^m\\Omega_i}}\\frac{\\max_{1\\le i\\le m}\\norm{\\omega_i-x}}{d\\Big{(}x,\\bigcap_{i=1}^m\\Omega_i\\Big{)}},\r\n","start":21498,"end":21921}]},{"id":"S016","semantic_type":"proposition","kind":"proposition","labels":["pr2"],"declared_external":false,"source_only":false,"title":"其他正则模量的度量表示","source_units":[{"raw_tex":"\\begin{prop}\\label{pr2}\r\n\\begin{align}\\label{newcon}\r\n\\theta[\\bold{\\Omega}](\\bar{x})&= \\liminf_{\\substack{x_i\\to 0\\; (1\\le i\\le m)\\\\ \\bar x\\notin \\bigcap_{i=1}^m(\\Omega_i-x_i)}}\r\n\\frac{\\max_{1\\le i\\le m}\\norm{x_i}}{d\\Big{(}\\bar x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\Big{)}},\r\n","start":21228,"end":21498},{"raw_tex":"\\\\\\label{vartheta}\r\n\\hat{\\theta}[\\bold{\\Omega}](\\bar{x})&= \\liminf_{\\substack{x\\to\\bar{x}\\\\x_i \\to 0\\;(1\\le i\\le m)\\\\x\\notin\\bigcap_{i=1}^m(\\Omega_i-x_i)}} \\frac{\\max_{1\\le i\\le m}d(x+x_i,\\Omega_i)} {d\\Big{(}x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\Big{)}}\r\n\\\\\\notag\r\n&= \\liminf_{\\substack{x\\to\\bar{x}\\\\x_i\\to0,\\; \\omega_i\\TO{\\Omega_i} \\bar x\\;(1\\le i\\le m)\\\\ x\\notin\\bigcap_{i=1}^m(\\Omega_i-x_i)}} \\frac{\\max_{1\\le i\\le m} \\norm{x+x_i-\\omega_i}} {d\\Big{(}x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\Big{)}}.\r\n\\end{align}\r\n\\end{prop}","start":21921,"end":22433}]},{"id":"S017","semantic_type":"proof","kind":"proof","labels":["new1"],"declared_external":false,"source_only":true,"title":"度量表示的作者完整证明","source_units":[{"raw_tex":"\\begin{proof}\r\n\\emph{Equality \\eqref{newcon}}.\r\nLet $\\xi$ stand for the \\RHS\\ of \\eqref{newcon}.\r\nSuppose that $\\xi>0$ and fix an arbitrary number $\\gamma\\in ]0,\\xi[$.\r\nThen there is a number $\\delta>0$ such that\r\n\\begin{equation}\\label{new1}\r\n\\gamma d\\left(\\bar x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\right)\\le \\max_{1\\le i\\le m}\\norm{x_i},\\; \\forall x_i\\in \\delta\\B \\; (i=1,\\ldots,m).\r\n\\end{equation}\r\nChoose a number $\\alpha\\in ]0,\\gamma[$ and set $\\delta'=\\frac{\\delta}{\\alpha}$.\r\nThen, for any $\\rho\\in ]0,\\delta'[$ and $x_i\\in (\\alpha\\rho)\\B$ $(i=1,\\ldots,m)$, it holds\r\n$\\max_{1\\le i\\le m}\\norm{x_i}\\le \\alpha\\rho\\le \\alpha\\delta'=\\delta$.\r\nHence, \\eqref{new1} yields\r\n$$\r\nd\\left(\\bar x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\right)\\le \\frac{1}{\\gamma}\\max_{1\\le i\\le m}\\norm{x_i}\\le \\frac{\\alpha}{\\gamma}\\rho<\\rho.\r\n$$\r\nThis implies \\eqref{HR} and consequently $\\theta[\\bold{\\Omega}](\\bar{x})\\ge\\alpha$.\r\nTaking into account that $\\alpha$ can be arbitrarily close to $\\xi$, we obtain $\\theta[\\bold{\\Omega}](\\bar{x})\\ge \\xi$.\r\n\r\nConversely, suppose that $\\theta[\\bold{\\Omega}](\\bar{x})>0$ and fix an arbitrary number $\\alpha\\in ]0,\\theta[\\bold{\\Omega}](\\bar{x})[$.\r\nThen there is a number $\\delta>0$ such that \\eqref{HR} is satisfied for all $\\rho\\in ]0,\\delta[$ and $x_i\\in (\\alpha\\rho)\\B$ $(i=1,\\ldots,m)$.\r\nChoose a positive $\\delta'<\\alpha\\delta$.\r\nFor any $x_i\\in \\delta'\\B \\; (i=1,\\ldots,m)$, it holds $\\max_{1\\le i\\le m}\\norm{x_i}<\\alpha\\delta$.\r\nPick up a $\\rho\\in]0,\\delta[$ such that $\\max_{1\\le i\\le m}\\norm{x_i}=\\alpha\\rho$.\r\nThen \\eqref{HR} yields\r\n$$\r\n\\alpha d\\left(\\bar x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\right)\\le \\alpha\\rho=\\max_{1\\le i\\le m}\\norm{x_i}.\r\n$$\r\nThis implies $\\xi\\ge \\alpha$. Since $\\alpha$ can be arbitrarily close to $\\theta[\\bold{\\Omega}](\\bar{x})$, we deduce\r\n$\\xi\\ge \\theta[\\bold{\\Omega}](\\bar{x})$.\r\n\r\n\\emph{Equality \\eqref{errcon'}}.\r\nLet $\\xi$ stand for the \\RHS\\ of \\eqref{errcon'}.\r\nSuppose that $\\xi>0$ and fix an arbitrary number $\\alpha\\in ]0,\\xi[$. Then there is a number $\\delta>0$ such that\r\n\\begin{equation*}\r\n\\alpha d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right) \\le \\max_{1\\le i\\le m} d(x,\\Omega_i),\\; \\forall x \\in B_{\\delta}(\\bar{x}).\r\n\\end{equation*}\r\nIf $x\\in\\bigcap_{i=1}^m\\left(\\Omega_i+ (\\alpha\\rho)\\B \\right)\\bigcap B_{\\delta}(\\bar x)$ for some $\\rho\\in ]0,\\delta[$, then $\\max_{1\\le i\\le m} d(x,\\Omega_i)\\le\\alpha\\rho$, and consequently $d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right) \\le\\rho$, i.e., $\\zeta_{\\rho,\\delta}[\\bold{\\Omega}](\\bar{x}) \\ge \\alpha\\rho$.\r\nHence, $\\zeta[\\bold{\\Omega}](\\bar{x})\\ge \\alpha$. Since $\\alpha$ can be arbitrarily close to $\\xi$, we obtain $\\zeta[\\bold{\\Omega}](\\bar{x})\\ge \\xi$.\r\n\r\nConversely, suppose that $\\zeta[\\bold{\\Omega}](\\bar{x})>0$ and fix any $\\alpha\\in ]0,\\zeta[\\bold{\\Omega}](\\bar{x})[$.\r\nThen there is a number $\\delta>0$ such that\r\n\\eqref{Hlr'} is satisfied for all $\\rho\\in ]0,\\delta[$.\r\nChoose a positive number $\\delta'<\\min\\{\\alpha\\delta,\\delta\\}$.\r\nFor any $x \\in B_{\\delta'}(\\bar{x})$, it holds\r\n$$\r\n\\max_{1\\le i\\le m}d(x,\\Omega_i)\\le \\norm{x-\\bar x}\\le \\delta'<\\alpha\\delta.\r\n$$\r\nChoose a $\\rho\\in]0,\\delta[$ such that $\\max_{1\\le i\\le m}d(x,\\Omega_i)=\\alpha\\rho$.\r\nThen, by \\eqref{Hlr'},\r\n$$\r\n\\alpha d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right) \\le \\alpha\\rho =\\max_{1\\le i\\le m}d(x,\\Omega_i).\r\n$$\r\nHence, $\\alpha\\le\\xi$.\r\nBy letting $\\alpha\\to\\zeta[\\bold{\\Omega}](\\bar{x})$, we obtain $\\zeta[\\bold{\\Omega}](\\bar{x})\\le\\xi$.\r\n\r\n\\emph{Equality \\eqref{vartheta}} has been proved in \\cite[Theorem 1]{Kru05.1}.\r\n\\qed\r\n\\end{proof}\r\n","start":22437,"end":25948}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","labels":["new1"],"declared_external":false,"source_only":true,"title":"半正则度量表示的双向估计","source_units":[{"raw_tex":"\\emph{Equality \\eqref{newcon}}.\r\nLet $\\xi$ stand for the \\RHS\\ of \\eqref{newcon}.\r\nSuppose that $\\xi>0$ and fix an arbitrary number $\\gamma\\in ]0,\\xi[$.\r\nThen there is a number $\\delta>0$ such that\r\n\\begin{equation}\\label{new1}\r\n\\gamma d\\left(\\bar x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\right)\\le \\max_{1\\le i\\le m}\\norm{x_i},\\; \\forall x_i\\in \\delta\\B \\; (i=1,\\ldots,m).\r\n\\end{equation}\r\nChoose a number $\\alpha\\in ]0,\\gamma[$ and set $\\delta'=\\frac{\\delta}{\\alpha}$.\r\nThen, for any $\\rho\\in ]0,\\delta'[$ and $x_i\\in (\\alpha\\rho)\\B$ $(i=1,\\ldots,m)$, it holds\r\n$\\max_{1\\le i\\le m}\\norm{x_i}\\le \\alpha\\rho\\le \\alpha\\delta'=\\delta$.\r\nHence, \\eqref{new1} yields\r\n$$\r\nd\\left(\\bar x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\right)\\le \\frac{1}{\\gamma}\\max_{1\\le i\\le m}\\norm{x_i}\\le \\frac{\\alpha}{\\gamma}\\rho<\\rho.\r\n$$\r\nThis implies \\eqref{HR} and consequently $\\theta[\\bold{\\Omega}](\\bar{x})\\ge\\alpha$.\r\nTaking into account that $\\alpha$ can be arbitrarily close to $\\xi$, we obtain $\\theta[\\bold{\\Omega}](\\bar{x})\\ge \\xi$.\r\n\r\nConversely, suppose that $\\theta[\\bold{\\Omega}](\\bar{x})>0$ and fix an arbitrary number $\\alpha\\in ]0,\\theta[\\bold{\\Omega}](\\bar{x})[$.\r\nThen there is a number $\\delta>0$ such that \\eqref{HR} is satisfied for all $\\rho\\in ]0,\\delta[$ and $x_i\\in (\\alpha\\rho)\\B$ $(i=1,\\ldots,m)$.\r\nChoose a positive $\\delta'<\\alpha\\delta$.\r\nFor any $x_i\\in \\delta'\\B \\; (i=1,\\ldots,m)$, it holds $\\max_{1\\le i\\le m}\\norm{x_i}<\\alpha\\delta$.\r\nPick up a $\\rho\\in]0,\\delta[$ such that $\\max_{1\\le i\\le m}\\norm{x_i}=\\alpha\\rho$.\r\nThen \\eqref{HR} yields\r\n$$\r\n\\alpha d\\left(\\bar x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\right)\\le \\alpha\\rho=\\max_{1\\le i\\le m}\\norm{x_i}.\r\n$$\r\nThis implies $\\xi\\ge \\alpha$. Since $\\alpha$ can be arbitrarily close to $\\theta[\\bold{\\Omega}](\\bar{x})$, we deduce\r\n$\\xi\\ge \\theta[\\bold{\\Omega}](\\bar{x})$.\r\n","start":22452,"end":24261}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"次正则度量表示的双向估计","source_units":[{"raw_tex":"\\emph{Equality \\eqref{errcon'}}.\r\nLet $\\xi$ stand for the \\RHS\\ of \\eqref{errcon'}.\r\nSuppose that $\\xi>0$ and fix an arbitrary number $\\alpha\\in ]0,\\xi[$. Then there is a number $\\delta>0$ such that\r\n\\begin{equation*}\r\n\\alpha d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right) \\le \\max_{1\\le i\\le m} d(x,\\Omega_i),\\; \\forall x \\in B_{\\delta}(\\bar{x}).\r\n\\end{equation*}\r\nIf $x\\in\\bigcap_{i=1}^m\\left(\\Omega_i+ (\\alpha\\rho)\\B \\right)\\bigcap B_{\\delta}(\\bar x)$ for some $\\rho\\in ]0,\\delta[$, then $\\max_{1\\le i\\le m} d(x,\\Omega_i)\\le\\alpha\\rho$, and consequently $d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right) \\le\\rho$, i.e., $\\zeta_{\\rho,\\delta}[\\bold{\\Omega}](\\bar{x}) \\ge \\alpha\\rho$.\r\nHence, $\\zeta[\\bold{\\Omega}](\\bar{x})\\ge \\alpha$. Since $\\alpha$ can be arbitrarily close to $\\xi$, we obtain $\\zeta[\\bold{\\Omega}](\\bar{x})\\ge \\xi$.\r\n\r\nConversely, suppose that $\\zeta[\\bold{\\Omega}](\\bar{x})>0$ and fix any $\\alpha\\in ]0,\\zeta[\\bold{\\Omega}](\\bar{x})[$.\r\nThen there is a number $\\delta>0$ such that\r\n\\eqref{Hlr'} is satisfied for all $\\rho\\in ]0,\\delta[$.\r\nChoose a positive number $\\delta'<\\min\\{\\alpha\\delta,\\delta\\}$.\r\nFor any $x \\in B_{\\delta'}(\\bar{x})$, it holds\r\n$$\r\n\\max_{1\\le i\\le m}d(x,\\Omega_i)\\le \\norm{x-\\bar x}\\le \\delta'<\\alpha\\delta.\r\n$$\r\nChoose a $\\rho\\in]0,\\delta[$ such that $\\max_{1\\le i\\le m}d(x,\\Omega_i)=\\alpha\\rho$.\r\nThen, by \\eqref{Hlr'},\r\n$$\r\n\\alpha d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right) \\le \\alpha\\rho =\\max_{1\\le i\\le m}d(x,\\Omega_i).\r\n$$\r\nHence, $\\alpha\\le\\xi$.\r\nBy letting $\\alpha\\to\\zeta[\\bold{\\Omega}](\\bar{x})$, we obtain $\\zeta[\\bold{\\Omega}](\\bar{x})\\le\\xi$.\r\n","start":24263,"end":25847}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"一致正则表示的外部引用","source_units":[{"raw_tex":"\\emph{Equality \\eqref{vartheta}} has been proved in \\cite[Theorem 1]{Kru05.1}.\r\n\\qed\r\n\\end{proof}\r\n","start":25849,"end":25948}]},{"id":"S021","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"集合族的乘积集合","source_units":[{"raw_tex":"This section discusses dual characterizations of regularity properties of a collection of sets $\\bold{\\Omega}:=\\{\\Omega_1,\\ldots,\\Omega_m\\}$ at $\\bar x\\in \\bigcap_{i=1}^m\\Omega_i$.\r\nWe are going to use the notation ${\\widehat\\Omega:= \\Omega_1\\times\\ldots\\times\\Omega_m\\subset X^m}$.\r\n\\sloppy\r\n","start":28342,"end":28635}]},{"id":"S022","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"归一化对偶映射","source_units":[{"raw_tex":"\r\nRecall that the (normalized) \\emph{duality mapping} \\cite[Definition~3.2.6]{Lucc06} $J$ between a normed space $Y$ and its dual $Y^*$ is defined as\r\n$$\r\nJ(y):=\\left\\{y^*\\in \\mathbb{S}_{Y^*}:\\; \\langle y^*,y\\rangle=\\norm{y}\\right\\},\\; \\forall y\\in Y.\r\n$$\r\nNote that $J(-y)=-J(y)$.\r\n","start":28635,"end":28918}]},{"id":"S023","semantic_type":"external_fact","kind":"external_fact","labels":["ll01"],"declared_external":true,"declaration_text":["原前提列表中的B0066；原文第503行引用 Zălinescu (2002), Corollary 2.4.16。"],"source_only":false,"title":"范数的次微分公式","source_units":[{"raw_tex":"The following simple fact of convex analysis is well known (cf., e.g., \\cite[Corollary~2.4.16]{Zal02}).\r\n\r\n\\begin{lem}\\label{ll01}\r\nLet $(Y,\\|\\cdot\\|)$ be a normed space.\r\n\\begin{enumerate}\r\n\\item\r\n$\\sd\\|\\cdot\\|(y)=J(y)$ for any $y\\ne0$.\r\n\\item\r\n$\\sd\\|\\cdot\\|(0)=\\B^*$.\r\n\\end{enumerate}\r\n\\end{lem}\r\n","start":28920,"end":29219}]},{"id":"S024","semantic_type":"definition","kind":"definition","labels":["norm"],"declared_external":false,"source_only":false,"title":"参数化乘积范数","source_units":[{"raw_tex":"In this section, along with the maximum type norm on $X^{m+1}=X\\times X^m$, we are going to use another one depending on a parameter $\\rho>0$ and defined as follows:\r\n\\begin{equation}\\label{norm}\r\n\\norm{(x,\\hat x)}_{\\rho} :=\\max\\left\\{\\norm{x},\\rho \\norm{\\hat x}\\right\\},\\quad x\\in X,\\;\\hat x\\in X^m.\r\n\\end{equation}\r\n","start":31029,"end":31347}]},{"id":"S025","semantic_type":"derived_fact","kind":"derived_fact","labels":["normd"],"declared_external":false,"source_only":false,"title":"参数范数的对偶表示","source_units":[{"raw_tex":"It is easy to check that the corresponding dual norm has the following representation:\r\n\\begin{equation}\\label{normd}\r\n\\norm{(x^*,\\hat x^*)}_{\\rho} =\\|x^*\\|+\\rho^{-1}\\|\\hat x^*\\|,\\quad x^*\\in X^*,\\;\\hat x^*\\in (X^m)^*.\r\n\\end{equation}\r\nNote that if, in \\eqref{norm} and \\eqref{normd}, $\\hat x= (x_1,\\ldots,x_m)$ and $\\hat x^*= (x_1^*,\\ldots,x_m^*)$ with $x_i\\in X$ and $x_i^*\\in X^*$ ($i=1,2,\\ldots,m$), then $\\|\\hat x\\|=\\max_{1\\le i\\le m}\\|x_i\\|$ and $\\|\\hat x^*\\|=\\sum_{i=1}^m\\|x_i^*\\|$.\r\n","start":31347,"end":31838}]},{"id":"S026","semantic_type":"external_fact","kind":"external_fact","labels":["l01"],"declared_external":true,"declaration_text":["原前提列表中的B0082；原文引用 Ekeland (1974)、Kruger (2003), Theorem 2.1、Mordukhovich (2006), Theorem 2.26。"],"source_only":false,"title":"Ekeland变分原理","source_units":[{"raw_tex":"The proof of the main theorem of this section relies heavily on two fundamental results of variational analysis: the \\emph{Ekeland variational principle} (Ekeland \\cite{Eke74}; cf., e.g., \\cite[Theorem~2.1]{Kru03.1}, \\cite[Theorem 2.26]{Mor06.1}) and  the \\emph{fuzzy} (\\emph{approximate}) \\emph{sum rule} (Fabian \\cite{Fab89}; cf., e.g., \\cite[Rule~2.2]{Kru03.1}, \\cite[Theorem 2.33]{Mor06.1}).\r\nBelow we provide these results for completeness.\r\n\r\n\\begin{lem}[Ekeland variational principle] \\label{l01}\r\nSuppose $X$ is a complete metric space, and $f: X\\to\\R_\\infty$ is lower semicontinuous and bounded from below, $\\varepsilon>0, \\lambda>0$. If\r\n$$\r\nf(v)<\\inf_X f + \\varepsilon,\r\n$$\r\nthen there exists $x\\in X$ such that\r\n\r\n(a) $d(x,v)<\\lambda $,\r\n\r\n(b) $f(x)\\le f(v)$,\r\n\r\n(c) $f(u)+(\\varepsilon/\\lambda)d(u,x)\\ge f(x)$ for all $u\\in X$.\r\n\\end{lem}\r\n","start":33628,"end":34480}]},{"id":"S027","semantic_type":"external_fact","kind":"external_fact","labels":["l02"],"declared_external":true,"declaration_text":["原前提列表中的B0084；第589行引用 Fabian (1989)、Kruger (2003), Rule 2.2、Mordukhovich (2006), Theorem 2.33。"],"source_only":false,"title":"Asplund空间模糊和法则","source_units":[{"raw_tex":"\r\n\\begin{lem}[Fuzzy sum rule] \\label{l02}\r\nSuppose $X$ is Asplund,\r\n$f_1: X \\to\\R$ is Lipschitz continuous and\r\n$f_2: X \\to\\R_\\infty$\r\nis lower semicontinuous in a neighbourhood of $\\bar x$ with $f_2(\\bar x)<\\infty$.\r\nThen, for any $\\varepsilon>0$, there exist $x_1,x_2\\in X$ with $\\|x_i-\\bar x\\|<\\varepsilon$, ${|f_i(x_i)-f_i(\\bar x)|<\\varepsilon}$ $(i=1,2)$ such that\r\n$$\r\n\\partial (f_1+f_2) (\\bar x) \\subset \\partial f_1(x_1) +\\partial f_2(x_2) + \\varepsilon\\B^\\ast.\r\n$$\r\n\\end{lem}\r\n","start":34480,"end":34966}]},{"id":"S028","semantic_type":"definition","kind":"definition","labels":["f"],"declared_external":false,"source_only":false,"title":"扩展值残差函数","source_units":[{"raw_tex":"with function $f:X^{m+1}\\to{\\R}_\\infty :={\\R}\\cup\\{+\\infty\\}$ defined as\r\n\\begin{equation}\\label{f}\r\nf(x,\\hat x)=\\max_{1\\le i\\le m} \\norm{x-x_i} + \\delta_{\\widehat\\Omega}(\\hat x),\\quad x\\in X,\\; \\hat x:=(x_1,\\ldots,x_m)\\in X^m,\r\n\\end{equation}\r\nwhere $\\delta_{\\widehat\\Omega}$ is the indicator function of $\\widehat\\Omega$: $\\delta_{\\widehat\\Omega}(\\hat x)=0$ if $\\hat x\\in\\widehat\\Omega$ and $\\delta_{\\widehat\\Omega}(\\hat x)=+\\infty$ otherwise.\r\n","start":36930,"end":37377}]},{"id":"S029","semantic_type":"derived_fact","kind":"derived_fact","labels":["errcon"],"declared_external":false,"source_only":false,"title":"以残差函数重写模量","source_units":[{"raw_tex":"Observe that constant \\eqref{errcon'} can be rewritten as\r\n\\begin{equation}\\label{errcon}\r\n\\zeta[\\bold{\\Omega}](\\bar{x})= \\liminflarge_{\\substack{x\\to\\bar{x},\\, \\omega_i\\to\\bar x\\; (1\\le i\\le m)\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\\\x\\notin \\bigcap_{i=1}^m\\Omega_i}} \\frac{f(x,\\hat\\omega)} {d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right)}\r\n\\end{equation}\r\n","start":36578,"end":36930}]},{"id":"S030","semantic_type":"proposition","kind":"proposition","labels":["prJ"],"declared_external":false,"source_only":false,"title":"最大范数的对偶映射","source_units":[{"raw_tex":"\\begin{prop}\\label{prJ}\r\nFor each $(x_1,\\ldots,x_m)\\in X^m$,\r\n\\begin{multline*}\r\nJ(x_1,\\ldots,x_m)= \\Biggl\\{(x_1^*,\\ldots,x_m^*)\\in(X^*)^m:\\; \\sum_{i=1}^m\\|x_i^*\\|=1;\\mbox{ either } x_i^*=0\\\\\r\n\\mbox{or }\\left(\\|x_i\\|=\\max_{1\\le j\\le m} \\|x_j\\|,\\; x_i^*\\in\\|x_i^*\\| J(x_i)\\right)\\; (i=1,\\ldots,m)\\Biggr\\}.\r\n\\end{multline*}\r\n\\end{prop}","start":29399,"end":29732}]},{"id":"S031","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"乘积对偶映射的完整证明","source_units":[{"raw_tex":"\\begin{proof}\r\nLet $\\hat x:=(x_1,\\ldots,x_m)\\in X^m$, $\\hat x^*:=(x_1^*,\\ldots,x_m^*)\\in(X^*)^m$.\r\nThen\r\n$${\\|\\hat x\\|=\\max_{1\\le i\\le m} \\|x_i\\|},\\quad \\|\\hat x^*\\|=\\sum_{i=1}^m\\|x_i^*\\|,\\quad \\langle \\hat x^*,\\hat x\\rangle=\\sum_{i=1}^m\\langle x_i^*,x_i\\rangle.$$\r\nSuppose $\\|\\hat x^*\\|=1$, i.e., $\\sum_{i=1}^m\\|x_i^*\\|=1$.\r\nThen $\\hat x^*\\in J(\\hat x)$ if and only if ${\\sum_{i=1}^m\\langle x_i^*,x_i\\rangle=\\|\\hat x\\|}$.\r\nIn its turn, the last equality holds true if and only if ${\\langle x_i^*,x_i\\rangle=\\|x_i^*\\|\\cdot\\|\\hat x\\|}$ for all $i=1,\\ldots,m$.\r\n\r\nIndeed, if $\\langle x_i^*,x_i\\rangle=\\|x_i^*\\|\\cdot\\|\\hat x\\|$ for all $i=1,\\ldots,m$, then adding these $m$ equalities, we obtain $\\sum_{i=1}^m\\langle x_i^*,x_i\\rangle=\\|\\hat x\\|$.\r\nConversely, if $\\langle x_i^*,x_i\\rangle\\ne\\|x_i^*\\|\\cdot\\|\\hat x\\|$, i.e., ${\\langle x_i^*,x_i\\rangle<\\|x_i^*\\|\\cdot\\|\\hat x\\|}$ for some $i\\in\\{1,\\ldots,m\\}$, then\r\n$$\r\n\\sum_{j=1}^m\\langle x_j^*,x_j\\rangle =\\langle x_i^*,x_i\\rangle+\\sum_{j\\ne i}\\langle x_j^*,x_j\\rangle <\\|x_i^*\\|\\cdot\\|\\hat x\\|+\\|\\hat x\\|\\sum_{j\\ne i} \\|x_j^*\\|=\\|\\hat x\\|.\r\n$$\r\n\r\nFinally, $\\langle x_i^*,x_i\\rangle=\\|x_i^*\\|\\cdot\\|\\hat x\\|$ for some $i\\in\\{1,\\ldots,m\\}$ if and only if either $\\|x_i\\|=\\|\\hat x\\|$ and $x_i^*\\in\\|x_i^*\\| J(x_i)$ or $x_i^*=0$.\r\n\\qed\\end{proof}","start":29734,"end":31025}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"配对取等的坐标形式","source_units":[{"raw_tex":"Let $\\hat x:=(x_1,\\ldots,x_m)\\in X^m$, $\\hat x^*:=(x_1^*,\\ldots,x_m^*)\\in(X^*)^m$.\r\nThen\r\n$${\\|\\hat x\\|=\\max_{1\\le i\\le m} \\|x_i\\|},\\quad \\|\\hat x^*\\|=\\sum_{i=1}^m\\|x_i^*\\|,\\quad \\langle \\hat x^*,\\hat x\\rangle=\\sum_{i=1}^m\\langle x_i^*,x_i\\rangle.$$\r\nSuppose $\\|\\hat x^*\\|=1$, i.e., $\\sum_{i=1}^m\\|x_i^*\\|=1$.\r\nThen $\\hat x^*\\in J(\\hat x)$ if and only if ${\\sum_{i=1}^m\\langle x_i^*,x_i\\rangle=\\|\\hat x\\|}$.\r\nIn its turn, the last equality holds true if and only if ${\\langle x_i^*,x_i\\rangle=\\|x_i^*\\|\\cdot\\|\\hat x\\|}$ for all $i=1,\\ldots,m$.\r\n","start":29749,"end":30294}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"逐坐标取等与活跃性","source_units":[{"raw_tex":"\r\nIndeed, if $\\langle x_i^*,x_i\\rangle=\\|x_i^*\\|\\cdot\\|\\hat x\\|$ for all $i=1,\\ldots,m$, then adding these $m$ equalities, we obtain $\\sum_{i=1}^m\\langle x_i^*,x_i\\rangle=\\|\\hat x\\|$.\r\nConversely, if $\\langle x_i^*,x_i\\rangle\\ne\\|x_i^*\\|\\cdot\\|\\hat x\\|$, i.e., ${\\langle x_i^*,x_i\\rangle<\\|x_i^*\\|\\cdot\\|\\hat x\\|}$ for some $i\\in\\{1,\\ldots,m\\}$, then\r\n$$\r\n\\sum_{j=1}^m\\langle x_j^*,x_j\\rangle =\\langle x_i^*,x_i\\rangle+\\sum_{j\\ne i}\\langle x_j^*,x_j\\rangle <\\|x_i^*\\|\\cdot\\|\\hat x\\|+\\|\\hat x\\|\\sum_{j\\ne i} \\|x_j^*\\|=\\|\\hat x\\|.\r\n$$\r\n\r\nFinally, $\\langle x_i^*,x_i\\rangle=\\|x_i^*\\|\\cdot\\|\\hat x\\|$ for some $i\\in\\{1,\\ldots,m\\}$ if and only if either $\\|x_i\\|=\\|\\hat x\\|$ and $x_i^*\\in\\|x_i^*\\| J(x_i)$ or $x_i^*=0$.\r\n\\qed\\end{proof}","start":30294,"end":31025}]},{"id":"S034","semantic_type":"lemma","kind":"lemma","labels":["ll02"],"declared_external":false,"source_only":false,"title":"残差范数的次微分约束","source_units":[{"raw_tex":"\\begin{lem}\\label{ll02}\r\nLet $X$ be a normed space and\r\n$\\varphi(u,\\hat u)=\\|(u-u_1,\\ldots,u-u_m)\\|$, $u\\in X$, $\\hat u:=(u_1,\\ldots,u_m)\\in X^m$.\r\nSuppose $x\\in X$, $\\hat x:=(x_1,\\ldots,x_m)\\in X^m$, and ${\\hat v:=(x-x_1,\\ldots,x-x_m)}\\ne0$. Then\r\n\\begin{align*}\r\n\\sd\\varphi(x,\\hat x)\\subseteq\\bigl\\{\\left(x^*,\\hat x^* =(x_1^*,\\ldots,x_m^*)\\right)\\in &X^*\\times (X^*)^m :\\;\\\\\r\n&-\\hat x^*\\in J(\\hat v),\\; x^*=-(x_{1}^*+\\ldots+x_{m}^*)\\bigr\\}.\r\n\\end{align*}\r\n\\end{lem}","start":31939,"end":32406}]},{"id":"S035","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"残差次微分的完整证明","source_units":[{"raw_tex":"\\begin{proof}\r\n%Observe that $\\varphi$ is a composition of the norm on $X^m$ and the linear continuous mapping $(x,\\hat x)\\mapsto(x_1-x,\\ldots,x_m-x)$.\r\nLet $\\left(x^*,\\hat x^* =(x_1^*,\\ldots,x_m^*)\\right)\\in \\sd\\varphi(x,\\hat x)$, i.e.,\r\n$$\r\n\\|(u-u_1,\\ldots,u-u_m)\\|-\\|(x-x_1,\\ldots,x-x_m)\\|\\ge \\langle x^*,u-x\\rangle+\\sum_{i=1}^m\\langle x_i^*,u_i-x_i\\rangle\r\n$$\r\nfor any $u\\in X$ and $\\hat u:=(u_1,\\ldots,u_m)\\in X^m$.\r\nIn particular, with $u=x$ and $u_i=x_i-x_i'$ ($i=1,\\ldots,m$) for an arbitrary $\\hat x':=(x_1',\\ldots,x_m')\\in X^m$, we have\r\n$$\r\n\\|\\hat v+\\hat x'\\|-\\|\\hat v\\|\\ge-\\langle\\hat x^*,\\hat x'\\rangle,\r\n$$\r\ni.e., $-\\hat x^*\\in J(\\hat v)$.\r\nSimilarly, with $u=x+x'$ and $u_i=x_i+x'$ ($i=1,\\ldots,m$) for an arbitrary $x'\\in X$, we have\r\n$$\r\n\\left\\langle x^*+\\sum_{i=1}^mx_i^*,x'\\right\\rangle\\le0,\r\n$$\r\nand consequently $x^*+x_{1}^*+\\ldots+x_{m}^*=0$.\r\n\\qed\\end{proof}","start":32408,"end":33289}]},{"id":"S036","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"未启用的复合映射说明","source_units":[{"raw_tex":"%Observe that $\\varphi$ is a composition of the norm on $X^m$ and the linear continuous mapping $(x,\\hat x)\\mapsto(x_1-x,\\ldots,x_m-x)$.\r\n","start":32423,"end":32561}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"固定公共变量识别对偶方向","source_units":[{"raw_tex":"Let $\\left(x^*,\\hat x^* =(x_1^*,\\ldots,x_m^*)\\right)\\in \\sd\\varphi(x,\\hat x)$, i.e.,\r\n$$\r\n\\|(u-u_1,\\ldots,u-u_m)\\|-\\|(x-x_1,\\ldots,x-x_m)\\|\\ge \\langle x^*,u-x\\rangle+\\sum_{i=1}^m\\langle x_i^*,u_i-x_i\\rangle\r\n$$\r\nfor any $u\\in X$ and $\\hat u:=(u_1,\\ldots,u_m)\\in X^m$.\r\nIn particular, with $u=x$ and $u_i=x_i-x_i'$ ($i=1,\\ldots,m$) for an arbitrary $\\hat x':=(x_1',\\ldots,x_m')\\in X^m$, we have\r\n$$\r\n\\|\\hat v+\\hat x'\\|-\\|\\hat v\\|\\ge-\\langle\\hat x^*,\\hat x'\\rangle,\r\n$$\r\ni.e., $-\\hat x^*\\in J(\\hat v)$.\r\n","start":32561,"end":33063}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"同步平移消去分量和","source_units":[{"raw_tex":"Similarly, with $u=x+x'$ and $u_i=x_i+x'$ ($i=1,\\ldots,m$) for an arbitrary $x'\\in X$, we have\r\n$$\r\n\\left\\langle x^*+\\sum_{i=1}^mx_i^*,x'\\right\\rangle\\le0,\r\n$$\r\nand consequently $x^*+x_{1}^*+\\ldots+x_{m}^*=0$.\r\n\\qed\\end{proof}","start":33063,"end":33289}]},{"id":"S039","semantic_type":"lemma","kind":"lemma","labels":["ll03"],"declared_external":false,"source_only":false,"title":"乘积集合的法锥分解","source_units":[{"raw_tex":"\\begin{lem}\\label{ll03}\r\nLet $X$ be a normed space and $\\hat\\omega:=(\\omega_1,\\ldots,\\omega_m)\\in\\widehat\\Omega$.\r\nThen ${N_{\\widehat\\Omega}(\\hat\\omega) =N_{\\Omega_1}(\\omega_1)\\times\\ldots\\times N_{\\Omega_m}(\\omega_m)}$.\r\n\\end{lem}","start":33293,"end":33524}]},{"id":"S040","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"乘积法锥的作者证明","source_units":[{"raw_tex":"\\begin{proof}\r\nfollows directly from the definition of the Fr\\'echet normal cone.\r\n\\qed\\end{proof}","start":33526,"end":33624}]},{"id":"S041","semantic_type":"assumption","kind":"assumption","labels":["T3.1.1"],"declared_external":false,"source_only":false,"title":"斜率估计的Banach假设","source_units":[{"raw_tex":"Let $X$ be a Banach space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n","start":37407,"end":37480}]},{"id":"S042","semantic_type":"proposition","kind":"proposition","labels":["T3.1.1","rhoslo","strslo"],"declared_external":false,"source_only":false,"title":"原始斜率给出模量下界","source_units":[{"raw_tex":"\\begin{prop}\\label{T3.1.1}\r\nLet $X$ be a Banach space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n\\begin{enumerate}\r\n\\item\r\n$\\hat\\zeta[\\bold{\\Omega}](\\bar x)\\le \\zeta[\\bold{\\Omega}](\\bar{x})$, where\r\n\\begin{equation}\\label{strslo}\r\n\\hat\\zeta[\\bold{\\Omega}](\\bar x):= \\lim_{\\rho\\downarrow0} \\inflarge_{\\substack{\\norm{x-\\bar x}<\\rho\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega)\r\n\\end{equation}\r\nand, for $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega$,\r\n\\begin{equation}\\label{rhoslo}\r\n\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega):= \\limsup_{\\substack{(u,\\hat v)\\to(x,\\hat\\omega)\\\\(u,\\hat v)\\neq (x,\\hat\\omega)\\\\ \\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega}} \\frac{\\left(\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}-\\max\\limits_{1\\le i\\le m} \\norm{u-v_i}\\right)_+}{\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho}}.\r\n\\end{equation}\r\n\\item\r\nIf $\\hat\\zeta[\\bold{\\Omega}](\\bar x)>0$, then $\\bold{\\Omega}$ is subregular at $\\bar x$.\r\n\\end{enumerate}\r\n\\end{prop}","start":37379,"end":38457}]},{"id":"S043","semantic_type":"definition","kind":"definition","labels":["T3.1.1","rhoslo","strslo"],"declared_external":false,"source_only":false,"title":"局部下降斜率与严格模量","source_units":[{"raw_tex":"\\begin{equation}\\label{strslo}\r\n\\hat\\zeta[\\bold{\\Omega}](\\bar x):= \\lim_{\\rho\\downarrow0} \\inflarge_{\\substack{\\norm{x-\\bar x}<\\rho\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega)\r\n\\end{equation}\r\nand, for $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega$,\r\n\\begin{equation}\\label{rhoslo}\r\n\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega):= \\limsup_{\\substack{(u,\\hat v)\\to(x,\\hat\\omega)\\\\(u,\\hat v)\\neq (x,\\hat\\omega)\\\\ \\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega}} \\frac{\\left(\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}-\\max\\limits_{1\\le i\\le m} \\norm{u-v_i}\\right)_+}{\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho}}.\r\n\\end{equation}\r\n","start":37582,"end":38333}]},{"id":"S044","semantic_type":"proof","kind":"proof","labels":["28","29","30","31","32","33","34","35"],"declared_external":false,"source_only":false,"title":"原始斜率估计的完整证明","source_units":[{"raw_tex":"\\begin{proof}\r\n(i)\r\nLet $\\zeta[\\bold{\\Omega}](\\bar{x})<\\alpha<\\infty$. Choose a $\\rho\\in ]0,1[$ and set\r\n\\begin{equation}\\label{28}\r\n\\eta:= \\min\\left\\{\\frac{\\rho}{2},\\frac{\\rho}{\\alpha},\\rho^{\\frac{2}{\\rho}}\\right\\}.\r\n\\end{equation}\r\nBy \\eqref{errcon}, there are $x'\\in B_{\\eta}(\\bar x)$ and $\\hat\\omega'=(\\omega_1',\\ldots,\\omega_m')\\in \\widehat\\Omega$ such that\r\n\\begin{equation}\\label{29}\r\n   0<f(x',\\hat\\omega')<\\alpha d\\left(x',\\bigcap_{i=1}^m\\Omega_i\\right).\r\n\\end{equation}\r\nDenote $\\varepsilon:=f(x',\\hat\\omega')$ and $\\mu:=d\\left(x',\\bigcap_{i=1}^m\\Omega_i\\right)$. Then $\\mu\\le \\norm{x'-\\bar x}\\le \\eta\\le \\frac{\\rho}{2}<1$.\r\nObserve that $f$ is lower semicontinuous.\r\nApplying to $f$ Lemma~\\ref{l01} with $\\varepsilon$ as above and\r\n\\begin{equation}\\label{30}\r\n\\lambda:=\\mu(1-\\mu^{\\frac{\\rho}{2-\\rho}}),\r\n\\end{equation}\r\nwe find points $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in X^{m}$ such that\r\n\\begin{equation}\\label{31}\r\n   \\norm{(x,\\hat\\omega)-(x',\\hat\\omega')}_{\\rho}<\\lambda,\\; f(x,\\hat\\omega)\\le f(x',\\hat\\omega'),\r\n\\end{equation}\r\nand\r\n\\begin{equation}\\label{32}\r\nf(u,\\hat v)+\\frac{\\varepsilon}{\\lambda}\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho} \\ge f(x,\\hat\\omega),\r\n\\end{equation}\r\nfor all $(u,\\hat v)\\in X\\times X^{m}$.\r\nThanks to \\eqref{31}, \\eqref{30}, \\eqref{28}, and \\eqref{29}, we have\r\n$$\r\n\\norm{x-x'}<\\lambda <\\mu \\le \\norm{x'-\\bar x},\r\n$$\r\n\\begin{equation}\\label{33}\r\n   d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right)\\ge d\\left(x',\\bigcap_{i=1}^m\\Omega_i\\right)- \\norm{x-x'}\\ge \\mu-\\lambda =\\mu^{\\frac{2}{2-\\rho}},\r\n\\end{equation}\r\n\\begin{equation}\\label{34}\r\n\\norm{x-\\bar x}\\le \\norm{x-x'}+\\norm{x'-\\bar x}<2\\norm{x'-\\bar x}\\le 2\\eta\\le\\rho,\r\n\\end{equation}\r\n\\begin{equation}\\label{35}\r\nf(x,\\hat\\omega)\\le f(x',\\hat\\omega')<\\alpha\\mu\\le \\alpha\\eta\\le\\rho.\r\n\\end{equation}\r\nIt follows from \\eqref{33}, \\eqref{34}, and \\eqref{35} that\r\n$$\r\n\\norm{x-\\bar x}<\\rho,\\; \\hat\\omega\\in \\widehat\\Omega,\\; 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho.\r\n$$\r\nObserve that $\\mu^{\\frac{\\rho}{2-\\rho}}\\le \\eta^{\\frac{\\rho}{2-\\rho}}< \\eta^{\\frac{\\rho}{2}}\\le \\rho$, and consequently, by \\eqref{29} and \\eqref{30},\r\n$$\r\n\\frac{\\varepsilon}{\\lambda} <\\frac{\\alpha\\mu}{\\lambda} =\\frac{\\alpha}{1-\\mu^{\\frac{\\rho}{2-\\rho}}} <\\frac{\\alpha}{1-\\rho}.\r\n$$\r\nThanks to \\eqref{32} and \\eqref{f}, we have\r\n$$\r\n\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}-\\max\\limits_{1\\le i\\le m} \\norm{u-v_i}\\le \\frac{\\alpha}{1-\\rho}\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho}\r\n$$\r\nfor all $u\\in X$ and $\\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega$.\r\nIt follows that $\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega)\\le \\dfrac{\\alpha}{1-\\rho}$ and consequently\r\n$$\r\n\\inflarge_{\\substack{\\norm{x-\\bar x}<\\rho\\\\\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega)\\le \\frac{\\alpha}{1-\\rho}.\r\n$$\r\nTaking limits in the last inequality as $\\rho\\downarrow 0$ and $\\alpha\\to \\zeta[\\bold{\\Omega}](\\bar{x})$ yields the claimed inequality.\r\n\r\n(ii) follows from (i) and Proposition~\\ref{theorem11}~(ii).\r\n\\qed\\end{proof}","start":38461,"end":41548}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["28","29"],"declared_external":false,"source_only":false,"title":"选取小残差点与尺度","source_units":[{"raw_tex":"(i)\r\nLet $\\zeta[\\bold{\\Omega}](\\bar{x})<\\alpha<\\infty$. Choose a $\\rho\\in ]0,1[$ and set\r\n\\begin{equation}\\label{28}\r\n\\eta:= \\min\\left\\{\\frac{\\rho}{2},\\frac{\\rho}{\\alpha},\\rho^{\\frac{2}{\\rho}}\\right\\}.\r\n\\end{equation}\r\nBy \\eqref{errcon}, there are $x'\\in B_{\\eta}(\\bar x)$ and $\\hat\\omega'=(\\omega_1',\\ldots,\\omega_m')\\in \\widehat\\Omega$ such that\r\n\\begin{equation}\\label{29}\r\n   0<f(x',\\hat\\omega')<\\alpha d\\left(x',\\bigcap_{i=1}^m\\Omega_i\\right).\r\n\\end{equation}\r\nDenote $\\varepsilon:=f(x',\\hat\\omega')$ and $\\mu:=d\\left(x',\\bigcap_{i=1}^m\\Omega_i\\right)$. Then $\\mu\\le \\norm{x'-\\bar x}\\le \\eta\\le \\frac{\\rho}{2}<1$.\r\n","start":38476,"end":39096}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["30","31","32"],"declared_external":false,"source_only":false,"title":"Ekeland产生扰动极小点","source_units":[{"raw_tex":"Observe that $f$ is lower semicontinuous.\r\nApplying to $f$ Lemma~\\ref{l01} with $\\varepsilon$ as above and\r\n\\begin{equation}\\label{30}\r\n\\lambda:=\\mu(1-\\mu^{\\frac{\\rho}{2-\\rho}}),\r\n\\end{equation}\r\nwe find points $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in X^{m}$ such that\r\n\\begin{equation}\\label{31}\r\n   \\norm{(x,\\hat\\omega)-(x',\\hat\\omega')}_{\\rho}<\\lambda,\\; f(x,\\hat\\omega)\\le f(x',\\hat\\omega'),\r\n\\end{equation}\r\nand\r\n\\begin{equation}\\label{32}\r\nf(u,\\hat v)+\\frac{\\varepsilon}{\\lambda}\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho} \\ge f(x,\\hat\\omega),\r\n\\end{equation}\r\nfor all $(u,\\hat v)\\in X\\times X^{m}$.\r\n","start":39096,"end":39713}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":["33","34","35"],"declared_external":false,"source_only":false,"title":"验证新点的局部可行性","source_units":[{"raw_tex":"Thanks to \\eqref{31}, \\eqref{30}, \\eqref{28}, and \\eqref{29}, we have\r\n$$\r\n\\norm{x-x'}<\\lambda <\\mu \\le \\norm{x'-\\bar x},\r\n$$\r\n\\begin{equation}\\label{33}\r\n   d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right)\\ge d\\left(x',\\bigcap_{i=1}^m\\Omega_i\\right)- \\norm{x-x'}\\ge \\mu-\\lambda =\\mu^{\\frac{2}{2-\\rho}},\r\n\\end{equation}\r\n\\begin{equation}\\label{34}\r\n\\norm{x-\\bar x}\\le \\norm{x-x'}+\\norm{x'-\\bar x}<2\\norm{x'-\\bar x}\\le 2\\eta\\le\\rho,\r\n\\end{equation}\r\n\\begin{equation}\\label{35}\r\nf(x,\\hat\\omega)\\le f(x',\\hat\\omega')<\\alpha\\mu\\le \\alpha\\eta\\le\\rho.\r\n\\end{equation}\r\nIt follows from \\eqref{33}, \\eqref{34}, and \\eqref{35} that\r\n$$\r\n\\norm{x-\\bar x}<\\rho,\\; \\hat\\omega\\in \\widehat\\Omega,\\; 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho.\r\n$$\r\n","start":39713,"end":40445}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"斜率上界与极限","source_units":[{"raw_tex":"Observe that $\\mu^{\\frac{\\rho}{2-\\rho}}\\le \\eta^{\\frac{\\rho}{2-\\rho}}< \\eta^{\\frac{\\rho}{2}}\\le \\rho$, and consequently, by \\eqref{29} and \\eqref{30},\r\n$$\r\n\\frac{\\varepsilon}{\\lambda} <\\frac{\\alpha\\mu}{\\lambda} =\\frac{\\alpha}{1-\\mu^{\\frac{\\rho}{2-\\rho}}} <\\frac{\\alpha}{1-\\rho}.\r\n$$\r\nThanks to \\eqref{32} and \\eqref{f}, we have\r\n$$\r\n\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}-\\max\\limits_{1\\le i\\le m} \\norm{u-v_i}\\le \\frac{\\alpha}{1-\\rho}\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho}\r\n$$\r\nfor all $u\\in X$ and $\\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega$.\r\nIt follows that $\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega)\\le \\dfrac{\\alpha}{1-\\rho}$ and consequently\r\n$$\r\n\\inflarge_{\\substack{\\norm{x-\\bar x}<\\rho\\\\\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega)\\le \\frac{\\alpha}{1-\\rho}.\r\n$$\r\nTaking limits in the last inequality as $\\rho\\downarrow 0$ and $\\alpha\\to \\zeta[\\bold{\\Omega}](\\bar{x})$ yields the claimed inequality.\r\n","start":40445,"end":41470}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"正斜率推出次正则","source_units":[{"raw_tex":"\r\n(ii) follows from (i) and Proposition~\\ref{theorem11}~(ii).\r\n\\qed\\end{proof}","start":41470,"end":41548}]},{"id":"S050","semantic_type":"proposition","kind":"proposition","labels":["T3.1.2","strsubslo'","subrhoslo"],"declared_external":false,"source_only":false,"title":"次微分模量的下界","source_units":[{"raw_tex":"\\begin{prop}\\label{T3.1.2}\r\nLet $X$ be an Asplund space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n\\begin{enumerate}\r\n\\item\r\n$\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)\\le \\hat\\zeta[\\bold{\\Omega}](\\bar x)$, where $\\hat\\zeta[\\bold{\\Omega}](\\bar x)$ is given by \\eqref{strslo},\r\n\\begin{equation}\\label{strsubslo'}\r\n\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x):= \\lim_{\\rho\\downarrow 0} \\inflarge_{\\substack{\\norm{x-\\bar x}<\\rho\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x,\\hat\\omega)\r\n\\end{equation}\r\nand, for $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega$,\r\n\\begin{equation}\\label{subrhoslo}\r\n\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x,\\hat\\omega):= \\inflarge_{\\substack{(x^*,\\hat y^*)\\in\\partial f(x,\\hat\\omega)\\\\ \\norm{\\hat y^*}<\\rho}}\\norm{x^*}\r\n\\end{equation}\r\n(with the convention that the infimum over the empty set equals $+\\infty$).\r\n\\item\r\nIf $\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)>0$, then $\\bold{\\Omega}$ is subregular at $\\bar x$.\r\n\\end{enumerate}\r\n\\end{prop}","start":41552,"end":42630}]},{"id":"S051","semantic_type":"definition","kind":"definition","labels":["T3.1.2","strsubslo'","subrhoslo"],"declared_external":false,"source_only":false,"title":"次微分局部模量","source_units":[{"raw_tex":"\\begin{equation}\\label{strsubslo'}\r\n\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x):= \\lim_{\\rho\\downarrow 0} \\inflarge_{\\substack{\\norm{x-\\bar x}<\\rho\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x,\\hat\\omega)\r\n\\end{equation}\r\nand, for $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega$,\r\n\\begin{equation}\\label{subrhoslo}\r\n\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x,\\hat\\omega):= \\inflarge_{\\substack{(x^*,\\hat y^*)\\in\\partial f(x,\\hat\\omega)\\\\ \\norm{\\hat y^*}<\\rho}}\\norm{x^*}\r\n\\end{equation}\r\n(with the convention that the infimum over the empty set equals $+\\infty$).\r\n","start":41827,"end":42502}]},{"id":"S052","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"次微分模量估计的完整证明","source_units":[{"raw_tex":"\\begin{proof}\r\nLet $\\hat\\zeta[\\bold{\\Omega}](\\bar x)< \\alpha<\\infty$.\r\nChoose a $\\beta\\in ]\\hat\\zeta[\\bold{\\Omega}](\\bar x),\\alpha[$ and an arbitrary ${\\rho>0}$.\r\nSet ${\\rho'=\\min\\{1,\\alpha^{-1}\\}\\rho}$.\r\nBy \\eqref{strslo} and \\eqref{rhoslo}, one can find points $x\\in X$ and ${\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega}$ such that $\\norm{x-\\bar x}<\\rho'$, $0<\\max_{1\\le i\\le m} \\norm{\\omega_i-x}<\\rho'$, and\r\n$$\r\n\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}-\\max\\limits_{1\\le i\\le m} \\norm{u-v_i}\\le \\beta\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho'}\r\n$$\r\nfor all $(u,\\hat v)$ with $\\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega$ near $(x,\\hat\\omega)$.\r\nIn other words, $(x,\\hat\\omega)$ is a local minimizer of the function\r\n$$\r\n(u,\\hat v)\\mapsto \\max\\limits_{1\\le i\\le m} \\norm{u-v_i}+\\beta\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho'}\r\n$$\r\nsubject to $\\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega$.\r\nBy definition \\eqref{f}, this means that $(x,\\hat\\omega)$ minimizes locally the function\r\n$$\r\n(u,\\hat v)\\mapsto f(u,\\hat v)+\\beta\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho'},\r\n$$\r\nand consequently its Fr\\'echet subdifferential at $(x,\\hat\\omega)$ contains zero.\r\nTake an\r\n$$\r\n\\varepsilon\\in\\Bigl]0,\\min\\{\\rho-\\norm{x-\\bar x},\\rho-\\max_{1\\le i\\le m} \\norm{x-\\omega_i},\\alpha-\\beta\\}\\Bigr[.\r\n$$\r\nApplying Lemma~\\ref{l02} and Lemma~\\ref{ll01}~(ii), we can find points $x'\\in X$, ${\\hat\\omega'=(\\omega_1',\\ldots,\\omega_m')\\in \\widehat\\Omega}$, and $(x^*,\\hat y^*)\\in \\partial f(x',\\hat\\omega')$ such that $$\\norm{x'-x}<\\varepsilon,\\quad 0<\\max_{1\\le i\\le m} \\norm{x'-\\omega_i'}\\le \\max_{1\\le i\\le m} \\norm{x-\\omega_i}+\\varepsilon,$$\r\n$$\\mbox{and}\\quad\\norm{(x^*,\\hat y^*)}_{\\rho'} =\\|x^*\\|+\\|\\hat y^*\\|/\\rho' <\\beta+\\varepsilon.$$\r\nIt follows that\r\n$$\\norm{x'-\\bar x}<\\rho,\\;\\; 0<\\max_{1\\le i\\le m} \\norm{x'-\\omega_i'}<\\rho,\\;\\; \\norm{x^*}<\\alpha,\\;\\mbox{ and }\\; \\norm{\\hat y^*}<\\rho'\\alpha\\le \\rho.$$\r\nHence, $\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x',\\hat\\omega')<\\alpha$, and consequently $\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)<\\alpha$.\r\nBy letting $\\alpha\\to \\hat\\zeta[\\bold{\\Omega}](\\bar x)$, we obtain the claimed inequality.\r\n\r\n(ii) follows from (i) and Proposition~\\ref{T3.1.1}~(ii).\r\n\\qed\\end{proof}","start":42634,"end":44826}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"斜率界转为局部极小","source_units":[{"raw_tex":"Let $\\hat\\zeta[\\bold{\\Omega}](\\bar x)< \\alpha<\\infty$.\r\nChoose a $\\beta\\in ]\\hat\\zeta[\\bold{\\Omega}](\\bar x),\\alpha[$ and an arbitrary ${\\rho>0}$.\r\nSet ${\\rho'=\\min\\{1,\\alpha^{-1}\\}\\rho}$.\r\nBy \\eqref{strslo} and \\eqref{rhoslo}, one can find points $x\\in X$ and ${\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega}$ such that $\\norm{x-\\bar x}<\\rho'$, $0<\\max_{1\\le i\\le m} \\norm{\\omega_i-x}<\\rho'$, and\r\n$$\r\n\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}-\\max\\limits_{1\\le i\\le m} \\norm{u-v_i}\\le \\beta\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho'}\r\n$$\r\nfor all $(u,\\hat v)$ with $\\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega$ near $(x,\\hat\\omega)$.\r\nIn other words, $(x,\\hat\\omega)$ is a local minimizer of the function\r\n$$\r\n(u,\\hat v)\\mapsto \\max\\limits_{1\\le i\\le m} \\norm{u-v_i}+\\beta\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho'}\r\n$$\r\nsubject to $\\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega$.\r\nBy definition \\eqref{f}, this means that $(x,\\hat\\omega)$ minimizes locally the function\r\n$$\r\n(u,\\hat v)\\mapsto f(u,\\hat v)+\\beta\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho'},\r\n$$\r\nand consequently its Fr\\'echet subdifferential at $(x,\\hat\\omega)$ contains zero.\r\n","start":42649,"end":43792}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"模糊分解产生小次梯度","source_units":[{"raw_tex":"Take an\r\n$$\r\n\\varepsilon\\in\\Bigl]0,\\min\\{\\rho-\\norm{x-\\bar x},\\rho-\\max_{1\\le i\\le m} \\norm{x-\\omega_i},\\alpha-\\beta\\}\\Bigr[.\r\n$$\r\nApplying Lemma~\\ref{l02} and Lemma~\\ref{ll01}~(ii), we can find points $x'\\in X$, ${\\hat\\omega'=(\\omega_1',\\ldots,\\omega_m')\\in \\widehat\\Omega}$, and $(x^*,\\hat y^*)\\in \\partial f(x',\\hat\\omega')$ such that $$\\norm{x'-x}<\\varepsilon,\\quad 0<\\max_{1\\le i\\le m} \\norm{x'-\\omega_i'}\\le \\max_{1\\le i\\le m} \\norm{x-\\omega_i}+\\varepsilon,$$\r\n$$\\mbox{and}\\quad\\norm{(x^*,\\hat y^*)}_{\\rho'} =\\|x^*\\|+\\|\\hat y^*\\|/\\rho' <\\beta+\\varepsilon.$$\r\n","start":43792,"end":44357}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"分量控制与模量极限","source_units":[{"raw_tex":"It follows that\r\n$$\\norm{x'-\\bar x}<\\rho,\\;\\; 0<\\max_{1\\le i\\le m} \\norm{x'-\\omega_i'}<\\rho,\\;\\; \\norm{x^*}<\\alpha,\\;\\mbox{ and }\\; \\norm{\\hat y^*}<\\rho'\\alpha\\le \\rho.$$\r\nHence, $\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x',\\hat\\omega')<\\alpha$, and consequently $\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)<\\alpha$.\r\nBy letting $\\alpha\\to \\hat\\zeta[\\bold{\\Omega}](\\bar x)$, we obtain the claimed inequality.\r\n","start":44357,"end":44751}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"正次微分模量推出次正则","source_units":[{"raw_tex":"\r\n(ii) follows from (i) and Proposition~\\ref{T3.1.1}~(ii).\r\n\\qed\\end{proof}","start":44751,"end":44826}]},{"id":"S057","semantic_type":"proposition","kind":"proposition","labels":["T3.1.3","c2","c2r"],"declared_external":false,"source_only":false,"title":"近似法向量模量的下界","source_units":[{"raw_tex":"\\begin{prop}\\label{T3.1.3}\r\nLet $X$ be an Asplund space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n\\begin{enumerate}\r\n\\item\r\n$\\hat\\zeta^*_2[\\bold{\\Omega}](\\bar x)\\le \\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)$, where $\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)$ is given by \\eqref{strsubslo'},\r\n\\begin{equation}\\label{c2}\r\n\\hat\\zeta^{*}_2[\\bold{\\Omega}](\\bar x):= \\lim_{\\rho\\downarrow 0} \\inf_{\\substack{\\norm{x-\\bar x}<\\rho\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\lim_{\\varepsilon\\downarrow 0} \\inf_{\\substack{\\norm{x'-x}<\\varepsilon\\\\ \\hat\\omega'\\in \\widehat\\Omega\\\\ \\norm{\\hat\\omega'-\\hat\\omega}<\\varepsilon}}\r\n\\zeta_{\\rho,\\varepsilon,2}^{*}[\\bold{\\Omega}] (x',\\hat\\omega')\r\n\\end{equation}\r\nand, for $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega$ with $(x-\\omega_1,\\ldots,x-\\omega_m)\\ne0$,\r\n\\begin{align}\r\n\\zeta_{\\rho,\\varepsilon,2}^{*}[\\bold{\\Omega}](x,\\hat\\omega):= \\inf\\Biggl\\{\r\n\\left\\|\\sum\\limits_{i=1}^m{x}_i^*\\right\\|:\\,\r\n&\r\nx_{i}^*\\in N_{\\Omega_i}(\\omega_i) +\\rho\\B^*\\quad (i=1,\\ldots,m),\r\n\\notag\\\\&\r\nx_i^*=0\r\n\\quad\\mbox{if}\\quad\r\n\\norm{x-\\omega_i}<\\max_{1\\le j\\le m} \\norm{x-\\omega_j},\r\n\\notag\\\\&\r\n\\langle x_i^*,x-\\omega_i\\rangle\r\n\\ge\\|x_i^*\\|(\\|x-\\omega_i\\|-\\varepsilon),\r\n\\notag\\\\&\r\n\\sum_{i=1}^m\\|x_i^*\\|=1\r\n\\Biggr\\}.\\label{c2r}\r\n\\end{align}\r\n\\item\r\nIf $\\hat\\zeta^*_2[\\bold{\\Omega}](\\bar x)>0$, then $\\bold{\\Omega}$ is subregular at $\\bar x$.\r\n\\end{enumerate}\r\n\\end{prop}","start":44830,"end":46305}]},{"id":"S058","semantic_type":"definition","kind":"definition","labels":["T3.1.3","c2","c2r"],"declared_external":false,"source_only":false,"title":"归一化近似法向量模量","source_units":[{"raw_tex":"\\begin{equation}\\label{c2}\r\n\\hat\\zeta^{*}_2[\\bold{\\Omega}](\\bar x):= \\lim_{\\rho\\downarrow 0} \\inf_{\\substack{\\norm{x-\\bar x}<\\rho\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\lim_{\\varepsilon\\downarrow 0} \\inf_{\\substack{\\norm{x'-x}<\\varepsilon\\\\ \\hat\\omega'\\in \\widehat\\Omega\\\\ \\norm{\\hat\\omega'-\\hat\\omega}<\\varepsilon}}\r\n\\zeta_{\\rho,\\varepsilon,2}^{*}[\\bold{\\Omega}] (x',\\hat\\omega')\r\n\\end{equation}\r\nand, for $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega$ with $(x-\\omega_1,\\ldots,x-\\omega_m)\\ne0$,\r\n\\begin{align}\r\n\\zeta_{\\rho,\\varepsilon,2}^{*}[\\bold{\\Omega}](x,\\hat\\omega):= \\inf\\Biggl\\{\r\n\\left\\|\\sum\\limits_{i=1}^m{x}_i^*\\right\\|:\\,\r\n&\r\nx_{i}^*\\in N_{\\Omega_i}(\\omega_i) +\\rho\\B^*\\quad (i=1,\\ldots,m),\r\n\\notag\\\\&\r\nx_i^*=0\r\n\\quad\\mbox{if}\\quad\r\n\\norm{x-\\omega_i}<\\max_{1\\le j\\le m} \\norm{x-\\omega_j},\r\n\\notag\\\\&\r\n\\langle x_i^*,x-\\omega_i\\rangle\r\n\\ge\\|x_i^*\\|(\\|x-\\omega_i\\|-\\varepsilon),\r\n\\notag\\\\&\r\n\\sum_{i=1}^m\\|x_i^*\\|=1\r\n\\Biggr\\}.\\label{c2r}\r\n\\end{align}\r\n","start":45117,"end":46177}]},{"id":"S059","semantic_type":"proof","kind":"proof","labels":["est"],"declared_external":false,"source_only":false,"title":"近似法向量估计的完整证明","source_units":[{"raw_tex":"\\begin{proof}\r\n(i)\r\nLet $\\rho>0$, $x\\in X$, $\\hat\\omega:= (\\omega_1,\\ldots,\\omega_m)\\in\\widehat\\Omega$ with $\\|x-\\bar x\\|<\\rho$, ${0<\\max_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}$, $(u^*,\\hat v^*) \\in\\partial f(x,\\hat\\omega)$, where $f$ is given by \\eqref{f}, and $\\|\\hat v^*\\|<\\rho$.\r\nDenote $\\hat v:=(x-\\omega_1,\\ldots,x-\\omega_m)$.\r\nThen $0<\\|\\hat v\\|<\\rho$.\r\nObserve that function $f$ is the sum of two functions on $X^{m+1}$:\r\n$$(x,\\hat x)\\mapsto \\varphi(x,\\hat x):= \\|(x-x_1,\\ldots,x-x_m)\\|\\quad\\mbox{and}\\quad(x,\\hat x) \\mapsto\\delta_{\\widehat\\Omega}(\\hat x),$$\r\nwhere $\\hat x:=(x_1,\\ldots,x_m)$ and $\\delta_{\\widehat\\Omega}$ is the indicator function of $\\widehat\\Omega$.\r\nThe first function is Lipschitz continuous while the second one is lower semicontinuous.\r\nOne can apply Lemma~\\ref{l02}.\r\nFor any $\\varepsilon>0$, there exist points $x'\\in X$, $\\hat x:=(x_1,\\ldots,x_m)\\in X^m$, $\\hat\\omega':=(\\omega_1',\\ldots,\\omega_m') \\in\\widehat\\Omega$, $\\left(x^*,\\hat y^*\\right)\\in\\partial \\varphi(x',\\hat x)$, and $\\hat\\omega^*\\in N_{\\widehat\\Omega}(\\hat\\omega')$ such that\r\n\\begin{gather}\\notag\r\n\\|x'-x\\|<\\varepsilon,\\quad \\|\\hat x-\\hat\\omega\\|<\\frac{\\varepsilon}{4},\\quad \\|\\hat\\omega'-\\hat\\omega\\|<\\frac{\\varepsilon}{4},\r\n\\\\\\label{est}\r\n\\|(u^*,\\hat v^*)-(x^*,\\hat y^*)-(0,\\hat\\omega^*)\\|<\\varepsilon.\r\n\\end{gather}\r\nTaking a smaller $\\varepsilon$ if necessary, one can ensure that $\\hat v':=(x'-\\omega_1',\\ldots,x'-\\omega_m')\\ne0$, $\\hat v'':=(x'-x_1,\\ldots,x'-x_m)\\ne0$, $\\|\\hat v^*\\|+\\varepsilon<\\rho$\r\nand,\r\nfor any $i=1,\\ldots,m$, ${\\norm{x'-x_i}<\\max_{1\\le j\\le m} \\norm{x'-x_j}}$ if and only if ${\\norm{x'-\\omega_i'}<\\max_{1\\le j\\le m} \\norm{x'-\\omega_j'}}$.\r\nBy Lemma~\\ref{ll02},\r\n$$\\hat x^*:=-\\hat y^*\\in J(\\hat v'')\r\n\\quad\\mbox{and}\\quad\r\nx^*=x_{1}^*+\\ldots+x_{m}^*,$$\r\nwhere\r\n$\\hat x^*=(x_1^*,\\ldots,x_m^*)$.\r\nBy Proposition~\\ref{prJ},\r\n\\begin{gather*}\r\n\\sum_{i=1}^m\\|x_i^*\\|=1,\r\n\\\\\r\nx_i^*=0\r\n\\quad\\mbox{if}\\quad\r\n\\norm{x'-\\omega_i'}<\\max_{1\\le j\\le m} \\norm{x'-\\omega_j'},\r\n\\end{gather*}\r\n\\begin{align*}\r\n\\langle x_i^*,x'-\\omega_i'\\rangle\r\n&\\ge\\langle x_i^*,x'-x_i\\rangle-\\|x_i^*\\|\\,\\|x_i-\\omega_i'\\| =\\|x_i^*\\|(\\|x'-x_i\\|-\\|x_i-\\omega_i'\\|)\r\n\\\\&\r\n\\ge\\|x_i^*\\|(\\|x'-\\omega_i'\\|-2\\|x_i-\\omega_i'\\|)\r\n\\ge\\|x_i^*\\|(\\|x'-\\omega_i'\\|-\\varepsilon) \\quad(i=1,\\ldots,m).\r\n\\end{align*}\r\nInequality \\eqref{est} yields the estimates:\r\n$\\|u^*\\|>\\left\\|{x}^*\\right\\|-\\varepsilon$, $\\left\\|\\hat x^*-\\hat\\omega^*\\right\\|<\\|\\hat v^*\\|+\\varepsilon<\\rho$,\r\nand consequently\r\n$$\r\n\\|u^*\\|> \\left\\|\\sum\\limits_{i=1}^m{x}_i^*\\right\\|-\\varepsilon, \\quad \\hat x^*\\in N_{\\widehat\\Omega}(\\hat\\omega') +\\rho\\B_m^*.\r\n$$\r\nIt follows from Lemma~\\ref{ll03} and definitions \\eqref{subrhoslo} and \\eqref{c2r} that\r\n$$\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x,\\hat\\omega)\\ge \\zeta_{\\rho,\\varepsilon,2}^*[\\bold{\\Omega}] (x',\\hat\\omega')-\\varepsilon.$$\r\nThe claimed inequality is a consequence of the last one and definitions \\eqref{strsubslo'} and \\eqref{c2}.\r\n\r\n(ii) follows from (i) and Proposition~\\ref{T3.1.2}~(ii).\r\n\\qed\\end{proof}","start":46309,"end":49318}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"分解残差函数","source_units":[{"raw_tex":"(i)\r\nLet $\\rho>0$, $x\\in X$, $\\hat\\omega:= (\\omega_1,\\ldots,\\omega_m)\\in\\widehat\\Omega$ with $\\|x-\\bar x\\|<\\rho$, ${0<\\max_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}$, $(u^*,\\hat v^*) \\in\\partial f(x,\\hat\\omega)$, where $f$ is given by \\eqref{f}, and $\\|\\hat v^*\\|<\\rho$.\r\nDenote $\\hat v:=(x-\\omega_1,\\ldots,x-\\omega_m)$.\r\nThen $0<\\|\\hat v\\|<\\rho$.\r\nObserve that function $f$ is the sum of two functions on $X^{m+1}$:\r\n$$(x,\\hat x)\\mapsto \\varphi(x,\\hat x):= \\|(x-x_1,\\ldots,x-x_m)\\|\\quad\\mbox{and}\\quad(x,\\hat x) \\mapsto\\delta_{\\widehat\\Omega}(\\hat x),$$\r\nwhere $\\hat x:=(x_1,\\ldots,x_m)$ and $\\delta_{\\widehat\\Omega}$ is the indicator function of $\\widehat\\Omega$.\r\nThe first function is Lipschitz continuous while the second one is lower semicontinuous.\r\n","start":46324,"end":47077}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":["est"],"declared_external":false,"source_only":false,"title":"近邻次梯度与法向量分解","source_units":[{"raw_tex":"One can apply Lemma~\\ref{l02}.\r\nFor any $\\varepsilon>0$, there exist points $x'\\in X$, $\\hat x:=(x_1,\\ldots,x_m)\\in X^m$, $\\hat\\omega':=(\\omega_1',\\ldots,\\omega_m') \\in\\widehat\\Omega$, $\\left(x^*,\\hat y^*\\right)\\in\\partial \\varphi(x',\\hat x)$, and $\\hat\\omega^*\\in N_{\\widehat\\Omega}(\\hat\\omega')$ such that\r\n\\begin{gather}\\notag\r\n\\|x'-x\\|<\\varepsilon,\\quad \\|\\hat x-\\hat\\omega\\|<\\frac{\\varepsilon}{4},\\quad \\|\\hat\\omega'-\\hat\\omega\\|<\\frac{\\varepsilon}{4},\r\n\\\\\\label{est}\r\n\\|(u^*,\\hat v^*)-(x^*,\\hat y^*)-(0,\\hat\\omega^*)\\|<\\varepsilon.\r\n\\end{gather}\r\n","start":47077,"end":47630}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"活跃指标一致性的作者断言","source_units":[{"raw_tex":"Taking a smaller $\\varepsilon$ if necessary, one can ensure that $\\hat v':=(x'-\\omega_1',\\ldots,x'-\\omega_m')\\ne0$, $\\hat v'':=(x'-x_1,\\ldots,x'-x_m)\\ne0$, $\\|\\hat v^*\\|+\\varepsilon<\\rho$\r\nand,\r\nfor any $i=1,\\ldots,m$, ${\\norm{x'-x_i}<\\max_{1\\le j\\le m} \\norm{x'-x_j}}$ if and only if ${\\norm{x'-\\omega_i'}<\\max_{1\\le j\\le m} \\norm{x'-\\omega_j'}}$.\r\n","start":47630,"end":47980}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"归一化与非活跃坐标消失","source_units":[{"raw_tex":"By Lemma~\\ref{ll02},\r\n$$\\hat x^*:=-\\hat y^*\\in J(\\hat v'')\r\n\\quad\\mbox{and}\\quad\r\nx^*=x_{1}^*+\\ldots+x_{m}^*,$$\r\nwhere\r\n$\\hat x^*=(x_1^*,\\ldots,x_m^*)$.\r\nBy Proposition~\\ref{prJ},\r\n\\begin{gather*}\r\n\\sum_{i=1}^m\\|x_i^*\\|=1,\r\n\\\\\r\nx_i^*=0\r\n\\quad\\mbox{if}\\quad\r\n\\norm{x'-\\omega_i'}<\\max_{1\\le j\\le m} \\norm{x'-\\omega_j'},\r\n\\end{gather*}\r\n","start":47980,"end":48314}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"近似配对取等估计","source_units":[{"raw_tex":"\\begin{align*}\r\n\\langle x_i^*,x'-\\omega_i'\\rangle\r\n&\\ge\\langle x_i^*,x'-x_i\\rangle-\\|x_i^*\\|\\,\\|x_i-\\omega_i'\\| =\\|x_i^*\\|(\\|x'-x_i\\|-\\|x_i-\\omega_i'\\|)\r\n\\\\&\r\n\\ge\\|x_i^*\\|(\\|x'-\\omega_i'\\|-2\\|x_i-\\omega_i'\\|)\r\n\\ge\\|x_i^*\\|(\\|x'-\\omega_i'\\|-\\varepsilon) \\quad(i=1,\\ldots,m).\r\n\\end{align*}\r\n","start":48314,"end":48603}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"近似法向性与向量和控制","source_units":[{"raw_tex":"Inequality \\eqref{est} yields the estimates:\r\n$\\|u^*\\|>\\left\\|{x}^*\\right\\|-\\varepsilon$, $\\left\\|\\hat x^*-\\hat\\omega^*\\right\\|<\\|\\hat v^*\\|+\\varepsilon<\\rho$,\r\nand consequently\r\n$$\r\n\\|u^*\\|> \\left\\|\\sum\\limits_{i=1}^m{x}_i^*\\right\\|-\\varepsilon, \\quad \\hat x^*\\in N_{\\widehat\\Omega}(\\hat\\omega') +\\rho\\B_m^*.\r\n$$\r\n","start":48603,"end":48918}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"局部法向模量比较与极限","source_units":[{"raw_tex":"It follows from Lemma~\\ref{ll03} and definitions \\eqref{subrhoslo} and \\eqref{c2r} that\r\n$$\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x,\\hat\\omega)\\ge \\zeta_{\\rho,\\varepsilon,2}^*[\\bold{\\Omega}] (x',\\hat\\omega')-\\varepsilon.$$\r\nThe claimed inequality is a consequence of the last one and definitions \\eqref{strsubslo'} and \\eqref{c2}.\r\n","start":48918,"end":49243}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"正法向模量推出次正则","source_units":[{"raw_tex":"\r\n(ii) follows from (i) and Proposition~\\ref{T3.1.2}~(ii).\r\n\\qed\\end{proof}","start":49243,"end":49318}]},{"id":"S068","semantic_type":"assumption","kind":"assumption","labels":["T3.1","T3.1.2","T3.1.3"],"declared_external":false,"source_only":false,"title":"Asplund与闭集假设","source_units":[{"raw_tex":"Let $X$ be an Asplund space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n","start":35085,"end":35160},{"raw_tex":"Let $X$ be an Asplund space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n","start":41580,"end":41655},{"raw_tex":"Let $X$ be an Asplund space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n","start":44858,"end":44933}]},{"id":"S069","semantic_type":"assumption","kind":"assumption","labels":["=1",">al","T3.1"],"declared_external":false,"source_only":false,"title":"次正则性的法向充分条件","source_units":[{"raw_tex":"$\\bold{\\Omega}$ is subregular at $\\bar x$ if there exist positive numbers $\\alpha$ and $\\delta$ such that, for any $\\rho\\in ]0,\\delta[$, $x\\in B_{\\rho}(\\bar x)$, $\\omega_i\\in\\Omega_i\\cap B_{\\rho}(x)$ $(i=1,\\ldots,m)$ with $\\omega_i\\neq x$ for some ${i\\in\\{1,\\ldots,m\\}}$, there is an $\\varepsilon>0$ such that, for any $x'\\in B_{\\varepsilon}(x)$, $\\omega'_i\\in\\Omega_i\\cap B_{\\varepsilon}(\\omega_i)$, $x_{i}^*\\in N_{\\Omega_i}(\\omega_i') +\\rho\\B^*$ $(i=1,\\ldots,m)$\r\nsatisfying\r\n\\begin{gather}\\notag\r\nx_i^*=0\r\n\\quad\\mbox{if}\\quad\r\n\\norm{x'-\\omega_i'}<\\max_{1\\le j\\le m} \\norm{x'-\\omega_j'},\r\n\\\\\\notag\r\n\\langle x_i^*,x'-\\omega_i'\\rangle\r\n\\ge\\|x_i^*\\|(\\|x'-\\omega_i'\\|-\\varepsilon),\r\n\\\\\\label{=1}\r\n\\sum_{i=1}^m\\|x_i^*\\|=1,\r\n\\end{gather}\r\nit holds\r\n\\begin{gather}\\label{>al}\r\n\\left\\|\\sum\\limits_{i=1}^m{x}_i^*\\right\\|>\\alpha.\r\n\\end{gather}\r\n","start":35186,"end":36023}]},{"id":"S070","semantic_type":"theorem","kind":"theorem","labels":["=1",">al","T3.1"],"declared_external":false,"source_only":false,"title":"集合族正则性的对偶判据","source_units":[{"raw_tex":"\\begin{thm}\\label{T3.1}\r\nLet $X$ be an Asplund space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n\\begin{enumerate}\r\n\\item\r\n$\\bold{\\Omega}$ is subregular at $\\bar x$ if there exist positive numbers $\\alpha$ and $\\delta$ such that, for any $\\rho\\in ]0,\\delta[$, $x\\in B_{\\rho}(\\bar x)$, $\\omega_i\\in\\Omega_i\\cap B_{\\rho}(x)$ $(i=1,\\ldots,m)$ with $\\omega_i\\neq x$ for some ${i\\in\\{1,\\ldots,m\\}}$, there is an $\\varepsilon>0$ such that, for any $x'\\in B_{\\varepsilon}(x)$, $\\omega'_i\\in\\Omega_i\\cap B_{\\varepsilon}(\\omega_i)$, $x_{i}^*\\in N_{\\Omega_i}(\\omega_i') +\\rho\\B^*$ $(i=1,\\ldots,m)$\r\nsatisfying\r\n\\begin{gather}\\notag\r\nx_i^*=0\r\n\\quad\\mbox{if}\\quad\r\n\\norm{x'-\\omega_i'}<\\max_{1\\le j\\le m} \\norm{x'-\\omega_j'},\r\n\\\\\\notag\r\n\\langle x_i^*,x'-\\omega_i'\\rangle\r\n\\ge\\|x_i^*\\|(\\|x'-\\omega_i'\\|-\\varepsilon),\r\n\\\\\\label{=1}\r\n\\sum_{i=1}^m\\|x_i^*\\|=1,\r\n\\end{gather}\r\nit holds\r\n\\begin{gather}\\label{>al}\r\n\\left\\|\\sum\\limits_{i=1}^m{x}_i^*\\right\\|>\\alpha.\r\n\\end{gather}\r\n\\item\r\n$\\bold{\\Omega}$ is uniformly regular at $\\bar x$ if and only if there are positive numbers $\\alpha$ and $\\delta$ such that\r\n\\eqref{>al} holds true\r\nfor all $\\omega_i\\in\\Omega_i\\cap{B}_\\delta(\\bar{x})$ and\r\n$x_i^*\\in{N}_{\\Omega_i}(\\omega_i)$ $(i=1,\\ldots,m)$ satisfying \\eqref{=1}. \\end{enumerate}\r\n\\end{thm}","start":35060,"end":36337}]},{"id":"S071","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\\emph{of Theorem~\\ref{T3.1}}\r\n(i) follows from Proposition~\\ref{T3.1.3}~(ii) and definitions \\eqref{c2} and \\eqref{c2r}.\r\n\r\n(ii) is a consequence of \\cite[Theorem~4]{Kru09.1}.\r\n\\qed\\end{proof}","start":49322,"end":49527}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"法向条件转为正模量","source_units":[{"raw_tex":"(i) follows from Proposition~\\ref{T3.1.3}~(ii) and definitions \\eqref{c2} and \\eqref{c2r}.\r\n","start":49365,"end":49457}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"一致正则判据的外部应用","source_units":[{"raw_tex":"\r\n(ii) is a consequence of \\cite[Theorem~4]{Kru09.1}.\r\n\\qed\\end{proof}","start":49457,"end":49527}]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原前提列表中的Basic convex subdifferential calculus；由标注者显式列出其在第564—568行的具体用途。"],"source_only":false,"title":"凸次梯度的全局不等式","source_units":[]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原前提列表中的the subdifferential Fermat rule；由标注者明确化。"],"source_only":false,"title":"次微分Fermat规则","source_units":[]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原前提列表中的identification of indicator subdifferentials with normal cones；由标注者明确其乘积形式。"],"source_only":false,"title":"指示函数次微分与法锥","source_units":[]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原前提列表中的Completeness of Asplund spaces and preservation under finite products and equivalent norms；Banach有限乘积完备性为implicit standard fact made explicit by annotator。"],"source_only":false,"title":"完备性与Asplund稳定性","source_units":[]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原前提列表：Kruger (2009), Theorem 4；原文第876行引用Kru09.1，书目信息见第1272—1274行。"],"source_only":false,"title":"Kruger一致正则定理","source_units":[]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原前提列表：Kruger (2005), Theorem 1；原文第453行引用Kru05.1，书目信息见第1264—1266行。"],"source_only":false,"title":"Kruger度量表示定理","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S070","edges":[{"source_id":"S006","target_id":"S012","kind":"supports"},{"source_id":"S009","target_id":"S012","kind":"supports"},{"source_id":"S007","target_id":"S013","kind":"supports"},{"source_id":"S010","target_id":"S013","kind":"supports"},{"source_id":"S008","target_id":"S014","kind":"supports"},{"source_id":"S011","target_id":"S014","kind":"supports"},{"source_id":"S006","target_id":"S018","kind":"supports"},{"source_id":"S009","target_id":"S018","kind":"supports"},{"source_id":"S004","target_id":"S018","kind":"supports"},{"source_id":"S007","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S019","kind":"supports"},{"source_id":"S004","target_id":"S019","kind":"supports"},{"source_id":"S005","target_id":"S019","kind":"supports"},{"source_id":"S079","target_id":"S020","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S017","kind":"supports"},{"source_id":"S020","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S001","target_id":"S032","kind":"supports"},{"source_id":"S022","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S022","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S001","target_id":"S025","kind":"supports"},{"source_id":"S074","target_id":"S037","kind":"supports"},{"source_id":"S003","target_id":"S037","kind":"supports"},{"source_id":"S023","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S002","target_id":"S040","kind":"supports"},{"source_id":"S021","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S024","target_id":"S043","kind":"supports"},{"source_id":"S021","target_id":"S043","kind":"supports"},{"source_id":"S029","target_id":"S045","kind":"supports"},{"source_id":"S004","target_id":"S045","kind":"supports"},{"source_id":"S005","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S026","target_id":"S046","kind":"supports"},{"source_id":"S041","target_id":"S046","kind":"supports"},{"source_id":"S028","target_id":"S046","kind":"supports"},{"source_id":"S024","target_id":"S046","kind":"supports"},{"source_id":"S077","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S004","target_id":"S047","kind":"supports"},{"source_id":"S045","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S028","target_id":"S048","kind":"supports"},{"source_id":"S043","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S013","target_id":"S049","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S044","kind":"supports"},{"source_id":"S049","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S042","kind":"supports"},{"source_id":"S028","target_id":"S051","kind":"supports"},{"source_id":"S003","target_id":"S051","kind":"supports"},{"source_id":"S043","target_id":"S053","kind":"supports"},{"source_id":"S024","target_id":"S053","kind":"supports"},{"source_id":"S028","target_id":"S053","kind":"supports"},{"source_id":"S075","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S027","target_id":"S054","kind":"supports"},{"source_id":"S023","target_id":"S054","kind":"supports"},{"source_id":"S025","target_id":"S054","kind":"supports"},{"source_id":"S068","target_id":"S054","kind":"supports"},{"source_id":"S077","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S051","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S042","target_id":"S056","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S055","target_id":"S052","kind":"supports"},{"source_id":"S056","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S002","target_id":"S058","kind":"supports"},{"source_id":"S021","target_id":"S058","kind":"supports"},{"source_id":"S051","target_id":"S060","kind":"supports"},{"source_id":"S028","target_id":"S060","kind":"supports"},{"source_id":"S068","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S027","target_id":"S061","kind":"supports"},{"source_id":"S076","target_id":"S061","kind":"supports"},{"source_id":"S068","target_id":"S061","kind":"supports"},{"source_id":"S077","target_id":"S061","kind":"supports"},{"source_id":"S060","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S063","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S034","target_id":"S063","kind":"supports"},{"source_id":"S030","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S061","target_id":"S064","kind":"supports"},{"source_id":"S061","target_id":"S065","kind":"supports"},{"source_id":"S063","target_id":"S065","kind":"supports"},{"source_id":"S062","target_id":"S065","kind":"supports"},{"source_id":"S063","target_id":"S066","kind":"supports"},{"source_id":"S064","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S039","target_id":"S066","kind":"supports"},{"source_id":"S051","target_id":"S066","kind":"supports"},{"source_id":"S058","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S050","target_id":"S067","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S061","target_id":"S059","kind":"supports"},{"source_id":"S062","target_id":"S059","kind":"supports"},{"source_id":"S063","target_id":"S059","kind":"supports"},{"source_id":"S064","target_id":"S059","kind":"supports"},{"source_id":"S065","target_id":"S059","kind":"supports"},{"source_id":"S066","target_id":"S059","kind":"supports"},{"source_id":"S067","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S057","kind":"supports"},{"source_id":"S069","target_id":"S072","kind":"supports"},{"source_id":"S058","target_id":"S072","kind":"supports"},{"source_id":"S057","target_id":"S072","kind":"supports"},{"source_id":"S068","target_id":"S072","kind":"supports"},{"source_id":"S078","target_id":"S073","kind":"supports"},{"source_id":"S068","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S071","kind":"supports"},{"source_id":"S073","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S070","kind":"supports"},{"source_id":"S005","target_id":"S070","kind":"supports"},{"source_id":"S007","target_id":"S070","kind":"supports"},{"source_id":"S008","target_id":"S070","kind":"supports"},{"source_id":"S002","target_id":"S070","kind":"supports"},{"source_id":"S001","target_id":"S070","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S070","edges":[{"source_id":"S006","target_id":"S012","kind":"supports"},{"source_id":"S009","target_id":"S012","kind":"supports"},{"source_id":"S007","target_id":"S013","kind":"supports"},{"source_id":"S010","target_id":"S013","kind":"supports"},{"source_id":"S008","target_id":"S014","kind":"supports"},{"source_id":"S011","target_id":"S014","kind":"supports"},{"source_id":"S006","target_id":"S018","kind":"supports"},{"source_id":"S009","target_id":"S018","kind":"supports"},{"source_id":"S004","target_id":"S018","kind":"supports"},{"source_id":"S007","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S019","kind":"supports"},{"source_id":"S004","target_id":"S019","kind":"supports"},{"source_id":"S005","target_id":"S019","kind":"supports"},{"source_id":"S079","target_id":"S020","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S017","kind":"supports"},{"source_id":"S020","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S001","target_id":"S030","kind":"supports"},{"source_id":"S022","target_id":"S030","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S001","target_id":"S025","kind":"supports"},{"source_id":"S074","target_id":"S034","kind":"supports"},{"source_id":"S003","target_id":"S034","kind":"supports"},{"source_id":"S023","target_id":"S034","kind":"supports"},{"source_id":"S002","target_id":"S039","kind":"supports"},{"source_id":"S021","target_id":"S039","kind":"supports"},{"source_id":"S024","target_id":"S043","kind":"supports"},{"source_id":"S021","target_id":"S043","kind":"supports"},{"source_id":"S029","target_id":"S042","kind":"supports"},{"source_id":"S004","target_id":"S042","kind":"supports"},{"source_id":"S005","target_id":"S042","kind":"supports"},{"source_id":"S026","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S028","target_id":"S042","kind":"supports"},{"source_id":"S024","target_id":"S042","kind":"supports"},{"source_id":"S077","target_id":"S042","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S013","target_id":"S042","kind":"supports"},{"source_id":"S028","target_id":"S051","kind":"supports"},{"source_id":"S003","target_id":"S051","kind":"supports"},{"source_id":"S043","target_id":"S050","kind":"supports"},{"source_id":"S024","target_id":"S050","kind":"supports"},{"source_id":"S028","target_id":"S050","kind":"supports"},{"source_id":"S075","target_id":"S050","kind":"supports"},{"source_id":"S027","target_id":"S050","kind":"supports"},{"source_id":"S023","target_id":"S050","kind":"supports"},{"source_id":"S025","target_id":"S050","kind":"supports"},{"source_id":"S068","target_id":"S050","kind":"supports"},{"source_id":"S077","target_id":"S050","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S042","target_id":"S050","kind":"supports"},{"source_id":"S002","target_id":"S058","kind":"supports"},{"source_id":"S021","target_id":"S058","kind":"supports"},{"source_id":"S051","target_id":"S057","kind":"supports"},{"source_id":"S028","target_id":"S057","kind":"supports"},{"source_id":"S068","target_id":"S057","kind":"supports"},{"source_id":"S027","target_id":"S057","kind":"supports"},{"source_id":"S076","target_id":"S057","kind":"supports"},{"source_id":"S077","target_id":"S057","kind":"supports"},{"source_id":"S034","target_id":"S057","kind":"supports"},{"source_id":"S030","target_id":"S057","kind":"supports"},{"source_id":"S039","target_id":"S057","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S050","target_id":"S057","kind":"supports"},{"source_id":"S069","target_id":"S072","kind":"supports"},{"source_id":"S058","target_id":"S072","kind":"supports"},{"source_id":"S057","target_id":"S072","kind":"supports"},{"source_id":"S068","target_id":"S072","kind":"supports"},{"source_id":"S078","target_id":"S073","kind":"supports"},{"source_id":"S068","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S071","kind":"supports"},{"source_id":"S073","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S070","kind":"supports"},{"source_id":"S005","target_id":"S070","kind":"supports"},{"source_id":"S007","target_id":"S070","kind":"supports"},{"source_id":"S008","target_id":"S070","kind":"supports"},{"source_id":"S002","target_id":"S070","kind":"supports"},{"source_id":"S001","target_id":"S070","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"空间、对偶与乘积范数","source_units":[{"raw_tex":"Our basic notation is standard; cf. \\cite{Mor06.1,RocWet98}.\r\nFor a normed linear space $X$, its topological dual is denoted $X^*$, while $\\langle\\cdot,\\cdot\\rangle$ denotes the bilinear form defining the pairing between the two spaces.\r\nThe closed unit ball in a normed space is denoted $\\B $, $B_\\delta(x)$ stands for the closed ball with radius $\\delta$ and centre $x$.\r\nProducts of normed spaces will be considered with the maximum type norms, if not specified otherwise.\r\n\r\n","start":6233,"end":6712}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"Fréchet法锥","source_units":[{"raw_tex":"The Fr\\'echet normal cone to a set $\\Omega\\subset X$ at $x\\in \\Omega$ and the Fr\\'echet subdifferential of a function $f:X \\to\\R_\\infty:=\\R\\cup\\{+\\infty\\}$ at a point $x$ with $f(x)<\\infty$ are defined, respectively, by\r\n\\begin{equation*}\r\nN_{\\Omega}(x):= \\left\\{x^* \\in X^* :\\; \\limsup_{u\\to{x},\\,u\\in\\Omega\\setminus\\{x\\}} \\frac {\\langle x^*,u-x \\rangle}{\\|u-x\\|} \\leq 0 \\right\\},\r\n\\end{equation*}\r\n","start":6712,"end":7112}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"Fréchet次微分","source_units":[{"raw_tex":"\\begin{equation*}\r\n\\partial f(x):=\\left\\{x^*\\in X^*:\\; \\liminf_{\\substack{u\\to x,\\,u\\neq x}} \\frac{f(u)-f(x)-\\langle x^*,u-x\\rangle}{\\norm{u-x}}\\ge 0\\right\\}.\r\n\\end{equation*}\r\n","start":7112,"end":7289}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"集合距离","source_units":[{"raw_tex":"\r\nFor a given set $\\Omega\\subset X$, the distance function associated with $\\Omega$ is defined by\r\n$$\r\nd(x,\\Omega):=\\inf_{\\omega\\in\\Omega}\\norm{x-\\omega},\\; \\forall x\\in X.\r\n$$\r\n","start":7289,"end":7467}]},{"id":"S005","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"有限集合族与公共点","source_units":[{"raw_tex":"\r\nIn the sequel, $\\bold{\\Omega}$ stands for a collection of $m$ $(m\\ge 2)$ sets $\\Omega_1,\\ldots,\\Omega_m$ in a normed linear space $X$, and we assume the existence of a point $\\bar x\\in \\bigcap_{i=1}^m\\Omega_i$.\r\n","start":7467,"end":7681}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["HR"],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"半正则性定义","source_units":[{"raw_tex":"The next definition introduces several regularity properties of $\\bold{\\Omega}$ at $\\bar x$.\r\n\r\n\\begin{dfn}\\label{HolderUR}\r\n\\begin{enumerate}\r\n\\item\r\n$\\bold{\\Omega}$ is semiregular at $\\bar x$ iff there exist positive numbers $\\alpha$ and $\\delta$ such that\r\n\\begin{equation}\\label{HR}\r\n\\bigcap_{i=1}^m(\\Omega_i-x_i)\\  \\bigcap\\ B_{\\rho}(\\bar x) \\neq \\emptyset\r\n\\end{equation}\r\nfor all $\\rho \\in ]0,\\delta[$ and all\r\n$x_i\\in{X}$ $(i=1,\\ldots,m)$ such that $\\max\\limits_{1\\leq i\\leq m}\\|x_i\\| \\le \\alpha\\rho$.\r\n","start":7931,"end":8441}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":["Hlr'","HolderUR"],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"次正则性定义","source_units":[{"raw_tex":"\\item\r\n$\\bold{\\Omega}$ is subregular at $\\bar x$ iff there exist positive numbers $\\alpha$ and $\\delta$ such that\r\n\\begin{equation}\\label{Hlr'}\r\n\\bigcap_{i=1}^m\\left(\\Omega_i+ (\\alpha\\rho)\\B \\right)\\bigcap B_{\\delta}(\\bar x)\\subseteq\r\n\\left(\\bigcap_{i=1}^m \\Omega_i\\right)+\\rho\\B\r\n\\end{equation}\r\nfor all $\\rho\\in ]0,\\delta[$.\r\n","start":8441,"end":8769}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":["HUR"],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"一致正则性定义","source_units":[{"raw_tex":"\\item\r\n$\\bold{\\Omega}$ is uniformly regular at $\\bar x$ iff there exist positive numbers $\\alpha$ and $\\delta$ such that\r\n\\begin{equation}\\label{HUR}\r\n\\bigcap_{i=1}^m(\\Omega_i-\\omega_i-x_i) \\bigcap(\\rho\\B )\\neq \\emptyset\r\n\\end{equation}\r\nfor all $\\rho \\in ]0,\\delta[$, $\\omega_i \\in \\Omega_i \\cap B_{\\delta}(\\bar{x})$, and all $x_i\\in{X}$ $(i=1,\\ldots,m)$ such that $\\max\\limits_{1\\leq i\\leq m}\\|x_i\\| \\le \\alpha\\rho$.\r\n\\end{enumerate}\r\n\\end{dfn}\r\n","start":8769,"end":9217}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"半正则模量","source_units":[{"raw_tex":"The regularity properties in Definition~\\ref{HolderUR} can be equivalently defined using the following nonnegative constants which provide quantitative characterizations of these properties:\r\n\\begin{gather}\\label{01.1}\r\n\\theta[\\bold{\\Omega}](\\bar{x}):= \\liminf_{\\rho \\downarrow 0} \\dfrac{\\theta_{\\rho}[\\bold{\\Omega}](\\bar{x})}{\\rho},\r\n","start":10444,"end":10779},{"raw_tex":"where, for $\\rho>0$ and $\\delta>0$,\r\n\\begin{gather}\\label{02.1}\r\n\\theta_{\\rho}[\\bold{\\Omega}](\\bar{x}):= \\sup\\left\\{r \\ge 0 :\\; \\bigcap_{i=1}^m (\\Omega_i - x_i) \\bigcap B_{\\rho}(\\bar{x}) \\neq \\emptyset,\\; \\forall x_i \\in r\\B \\right\\},\r\n","start":11135,"end":11371}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":["01.1","01.2","01.3"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"次正则模量","source_units":[{"raw_tex":"\\\\\\label{01.2}\r\n\\zeta[\\bold{\\Omega}](\\bar{x}):= \\lim_{\\delta\\downarrow 0}\\inf_{0<\\rho<\\delta} \\dfrac{\\zeta_{\\rho,\\delta}[\\bold{\\Omega}](\\bar{x})} {\\rho},\r\n","start":10779,"end":10934},{"raw_tex":"\\\\\\label{02.2}\r\n\\zeta_{\\rho,\\delta}[\\bold{\\Omega}](\\bar{x}):= \\sup\\left\\{r\\ge0 :\\;\\bigcap_{i=1}^m (\\Omega_i+ r\\B )\\bigcap B_{\\delta}(\\bar x)\\subseteq\r\n\\bigcap_{i=1}^m \\Omega_i+\\rho\\B \\right\\}.\r\n\\end{gather}\r\n","start":11371,"end":11579}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"一致正则模量","source_units":[{"raw_tex":"\\\\\\label{01.3}\r\n\\hat{\\theta}[\\bold{\\Omega}](\\bar{x}):= \\liminf_{\\omega_i\\TO{\\Omega_i}\\bar{x},\\rho\\downarrow0} \\dfrac{\\theta_{\\rho}[\\Omega_1-\\omega_1,\\ldots,\\Omega_m-\\omega_m](0)}{\\rho},\r\n\\end{gather}\r\n","start":10934,"end":11135}]},{"id":"S012","semantic_type":"proposition","kind":"proposition","labels":["theorem11"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"半正则性的模量判据","source_units":[{"raw_tex":"\\begin{prop}\\label{theorem11}\r\n\\begin{enumerate}\r\n\\item\r\n$\\bold{\\Omega}$ is semiregular at $\\bar x$ if and only if $\\theta[\\bold{\\Omega}](\\bar{x})>0$. Moreover, $\\theta[\\bold{\\Omega}](\\bar{x})$ is the exact upper bound of all numbers $\\alpha$ such that (\\ref{HR}) is satisfied.\r\n","start":11647,"end":11926}]},{"id":"S013","semantic_type":"proposition","kind":"proposition","labels":["theorem11"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"次正则性的模量判据","source_units":[{"raw_tex":"\\item\r\n$\\bold{\\Omega}$ is subregular at $\\bar x$ if and only if $\\zeta[\\bold{\\Omega}](\\bar{x})>0$. Moreover, $\\zeta[\\bold{\\Omega}](\\bar{x})$ is the exact upper bound of all numbers $\\alpha$ such that (\\ref{Hlr'}) is satisfied.\r\n","start":11926,"end":12154}]},{"id":"S014","semantic_type":"proposition","kind":"proposition","labels":["theorem11"],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"一致正则性的模量判据","source_units":[{"raw_tex":"\\item\r\n$\\bold{\\Omega}$ is uniformly regular at $\\bar x$ if and only if $\\hat{\\theta}[\\bold{\\Omega}](\\bar{x})>0$. Moreover, $\\hat{\\theta}[\\bold{\\Omega}](\\bar{x})$ is the exact upper bound of all numbers $\\alpha$ such that (\\ref{HUR}) is satisfied.\r\n\\end{enumerate}\r\n\\end{prop}","start":12154,"end":12429}]},{"id":"S015","semantic_type":"proposition","kind":"proposition","labels":["errcon'","newcon","pr2","vartheta"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"次正则模量的度量表示","source_units":[{"raw_tex":"\\\\\\label{errcon'}\r\n\\zeta[\\bold{\\Omega}](\\bar{x})&= \\quad\\liminf_{\\substack{x\\to\\bar{x}\\\\ x\\notin \\bigcap_{i=1}^m\\Omega_i}}\r\n\\quad\\frac{\\max_{1\\le i\\le m} d(x,\\Omega_i)}{d\\Big{(}x,\\bigcap_{i=1}^m\\Omega_i\\Big{)}}\r\n\\\\\\notag\r\n&=\\liminf_{\\substack{x\\to\\bar{x}\\\\ \\omega_i\\TO{\\Omega_i}\\bar x\\; (1\\le i\\le m)\\\\ x\\notin \\bigcap_{i=1}^m\\Omega_i}}\\frac{\\max_{1\\le i\\le m}\\norm{\\omega_i-x}}{d\\Big{(}x,\\bigcap_{i=1}^m\\Omega_i\\Big{)}},\r\n","start":21498,"end":21921}]},{"id":"S016","semantic_type":"proposition","kind":"proposition","labels":["pr2"],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"其他正则模量的度量表示","source_units":[{"raw_tex":"\\begin{prop}\\label{pr2}\r\n\\begin{align}\\label{newcon}\r\n\\theta[\\bold{\\Omega}](\\bar{x})&= \\liminf_{\\substack{x_i\\to 0\\; (1\\le i\\le m)\\\\ \\bar x\\notin \\bigcap_{i=1}^m(\\Omega_i-x_i)}}\r\n\\frac{\\max_{1\\le i\\le m}\\norm{x_i}}{d\\Big{(}\\bar x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\Big{)}},\r\n","start":21228,"end":21498},{"raw_tex":"\\\\\\label{vartheta}\r\n\\hat{\\theta}[\\bold{\\Omega}](\\bar{x})&= \\liminf_{\\substack{x\\to\\bar{x}\\\\x_i \\to 0\\;(1\\le i\\le m)\\\\x\\notin\\bigcap_{i=1}^m(\\Omega_i-x_i)}} \\frac{\\max_{1\\le i\\le m}d(x+x_i,\\Omega_i)} {d\\Big{(}x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\Big{)}}\r\n\\\\\\notag\r\n&= \\liminf_{\\substack{x\\to\\bar{x}\\\\x_i\\to0,\\; \\omega_i\\TO{\\Omega_i} \\bar x\\;(1\\le i\\le m)\\\\ x\\notin\\bigcap_{i=1}^m(\\Omega_i-x_i)}} \\frac{\\max_{1\\le i\\le m} \\norm{x+x_i-\\omega_i}} {d\\Big{(}x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\Big{)}}.\r\n\\end{align}\r\n\\end{prop}","start":21921,"end":22433}]},{"id":"S017","semantic_type":"proof","kind":"proof","labels":["new1"],"member_ids":["S017"],"declared_external":false,"source_only":true,"title":"度量表示的作者完整证明","source_units":[{"raw_tex":"\\begin{proof}\r\n\\emph{Equality \\eqref{newcon}}.\r\nLet $\\xi$ stand for the \\RHS\\ of \\eqref{newcon}.\r\nSuppose that $\\xi>0$ and fix an arbitrary number $\\gamma\\in ]0,\\xi[$.\r\nThen there is a number $\\delta>0$ such that\r\n\\begin{equation}\\label{new1}\r\n\\gamma d\\left(\\bar x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\right)\\le \\max_{1\\le i\\le m}\\norm{x_i},\\; \\forall x_i\\in \\delta\\B \\; (i=1,\\ldots,m).\r\n\\end{equation}\r\nChoose a number $\\alpha\\in ]0,\\gamma[$ and set $\\delta'=\\frac{\\delta}{\\alpha}$.\r\nThen, for any $\\rho\\in ]0,\\delta'[$ and $x_i\\in (\\alpha\\rho)\\B$ $(i=1,\\ldots,m)$, it holds\r\n$\\max_{1\\le i\\le m}\\norm{x_i}\\le \\alpha\\rho\\le \\alpha\\delta'=\\delta$.\r\nHence, \\eqref{new1} yields\r\n$$\r\nd\\left(\\bar x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\right)\\le \\frac{1}{\\gamma}\\max_{1\\le i\\le m}\\norm{x_i}\\le \\frac{\\alpha}{\\gamma}\\rho<\\rho.\r\n$$\r\nThis implies \\eqref{HR} and consequently $\\theta[\\bold{\\Omega}](\\bar{x})\\ge\\alpha$.\r\nTaking into account that $\\alpha$ can be arbitrarily close to $\\xi$, we obtain $\\theta[\\bold{\\Omega}](\\bar{x})\\ge \\xi$.\r\n\r\nConversely, suppose that $\\theta[\\bold{\\Omega}](\\bar{x})>0$ and fix an arbitrary number $\\alpha\\in ]0,\\theta[\\bold{\\Omega}](\\bar{x})[$.\r\nThen there is a number $\\delta>0$ such that \\eqref{HR} is satisfied for all $\\rho\\in ]0,\\delta[$ and $x_i\\in (\\alpha\\rho)\\B$ $(i=1,\\ldots,m)$.\r\nChoose a positive $\\delta'<\\alpha\\delta$.\r\nFor any $x_i\\in \\delta'\\B \\; (i=1,\\ldots,m)$, it holds $\\max_{1\\le i\\le m}\\norm{x_i}<\\alpha\\delta$.\r\nPick up a $\\rho\\in]0,\\delta[$ such that $\\max_{1\\le i\\le m}\\norm{x_i}=\\alpha\\rho$.\r\nThen \\eqref{HR} yields\r\n$$\r\n\\alpha d\\left(\\bar x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\right)\\le \\alpha\\rho=\\max_{1\\le i\\le m}\\norm{x_i}.\r\n$$\r\nThis implies $\\xi\\ge \\alpha$. Since $\\alpha$ can be arbitrarily close to $\\theta[\\bold{\\Omega}](\\bar{x})$, we deduce\r\n$\\xi\\ge \\theta[\\bold{\\Omega}](\\bar{x})$.\r\n\r\n\\emph{Equality \\eqref{errcon'}}.\r\nLet $\\xi$ stand for the \\RHS\\ of \\eqref{errcon'}.\r\nSuppose that $\\xi>0$ and fix an arbitrary number $\\alpha\\in ]0,\\xi[$. Then there is a number $\\delta>0$ such that\r\n\\begin{equation*}\r\n\\alpha d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right) \\le \\max_{1\\le i\\le m} d(x,\\Omega_i),\\; \\forall x \\in B_{\\delta}(\\bar{x}).\r\n\\end{equation*}\r\nIf $x\\in\\bigcap_{i=1}^m\\left(\\Omega_i+ (\\alpha\\rho)\\B \\right)\\bigcap B_{\\delta}(\\bar x)$ for some $\\rho\\in ]0,\\delta[$, then $\\max_{1\\le i\\le m} d(x,\\Omega_i)\\le\\alpha\\rho$, and consequently $d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right) \\le\\rho$, i.e., $\\zeta_{\\rho,\\delta}[\\bold{\\Omega}](\\bar{x}) \\ge \\alpha\\rho$.\r\nHence, $\\zeta[\\bold{\\Omega}](\\bar{x})\\ge \\alpha$. Since $\\alpha$ can be arbitrarily close to $\\xi$, we obtain $\\zeta[\\bold{\\Omega}](\\bar{x})\\ge \\xi$.\r\n\r\nConversely, suppose that $\\zeta[\\bold{\\Omega}](\\bar{x})>0$ and fix any $\\alpha\\in ]0,\\zeta[\\bold{\\Omega}](\\bar{x})[$.\r\nThen there is a number $\\delta>0$ such that\r\n\\eqref{Hlr'} is satisfied for all $\\rho\\in ]0,\\delta[$.\r\nChoose a positive number $\\delta'<\\min\\{\\alpha\\delta,\\delta\\}$.\r\nFor any $x \\in B_{\\delta'}(\\bar{x})$, it holds\r\n$$\r\n\\max_{1\\le i\\le m}d(x,\\Omega_i)\\le \\norm{x-\\bar x}\\le \\delta'<\\alpha\\delta.\r\n$$\r\nChoose a $\\rho\\in]0,\\delta[$ such that $\\max_{1\\le i\\le m}d(x,\\Omega_i)=\\alpha\\rho$.\r\nThen, by \\eqref{Hlr'},\r\n$$\r\n\\alpha d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right) \\le \\alpha\\rho =\\max_{1\\le i\\le m}d(x,\\Omega_i).\r\n$$\r\nHence, $\\alpha\\le\\xi$.\r\nBy letting $\\alpha\\to\\zeta[\\bold{\\Omega}](\\bar{x})$, we obtain $\\zeta[\\bold{\\Omega}](\\bar{x})\\le\\xi$.\r\n\r\n\\emph{Equality \\eqref{vartheta}} has been proved in \\cite[Theorem 1]{Kru05.1}.\r\n\\qed\r\n\\end{proof}\r\n","start":22437,"end":25948}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","labels":["new1"],"member_ids":["S018"],"declared_external":false,"source_only":true,"title":"半正则度量表示的双向估计","source_units":[{"raw_tex":"\\emph{Equality \\eqref{newcon}}.\r\nLet $\\xi$ stand for the \\RHS\\ of \\eqref{newcon}.\r\nSuppose that $\\xi>0$ and fix an arbitrary number $\\gamma\\in ]0,\\xi[$.\r\nThen there is a number $\\delta>0$ such that\r\n\\begin{equation}\\label{new1}\r\n\\gamma d\\left(\\bar x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\right)\\le \\max_{1\\le i\\le m}\\norm{x_i},\\; \\forall x_i\\in \\delta\\B \\; (i=1,\\ldots,m).\r\n\\end{equation}\r\nChoose a number $\\alpha\\in ]0,\\gamma[$ and set $\\delta'=\\frac{\\delta}{\\alpha}$.\r\nThen, for any $\\rho\\in ]0,\\delta'[$ and $x_i\\in (\\alpha\\rho)\\B$ $(i=1,\\ldots,m)$, it holds\r\n$\\max_{1\\le i\\le m}\\norm{x_i}\\le \\alpha\\rho\\le \\alpha\\delta'=\\delta$.\r\nHence, \\eqref{new1} yields\r\n$$\r\nd\\left(\\bar x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\right)\\le \\frac{1}{\\gamma}\\max_{1\\le i\\le m}\\norm{x_i}\\le \\frac{\\alpha}{\\gamma}\\rho<\\rho.\r\n$$\r\nThis implies \\eqref{HR} and consequently $\\theta[\\bold{\\Omega}](\\bar{x})\\ge\\alpha$.\r\nTaking into account that $\\alpha$ can be arbitrarily close to $\\xi$, we obtain $\\theta[\\bold{\\Omega}](\\bar{x})\\ge \\xi$.\r\n\r\nConversely, suppose that $\\theta[\\bold{\\Omega}](\\bar{x})>0$ and fix an arbitrary number $\\alpha\\in ]0,\\theta[\\bold{\\Omega}](\\bar{x})[$.\r\nThen there is a number $\\delta>0$ such that \\eqref{HR} is satisfied for all $\\rho\\in ]0,\\delta[$ and $x_i\\in (\\alpha\\rho)\\B$ $(i=1,\\ldots,m)$.\r\nChoose a positive $\\delta'<\\alpha\\delta$.\r\nFor any $x_i\\in \\delta'\\B \\; (i=1,\\ldots,m)$, it holds $\\max_{1\\le i\\le m}\\norm{x_i}<\\alpha\\delta$.\r\nPick up a $\\rho\\in]0,\\delta[$ such that $\\max_{1\\le i\\le m}\\norm{x_i}=\\alpha\\rho$.\r\nThen \\eqref{HR} yields\r\n$$\r\n\\alpha d\\left(\\bar x,\\bigcap_{i=1}^m(\\Omega_i-x_i)\\right)\\le \\alpha\\rho=\\max_{1\\le i\\le m}\\norm{x_i}.\r\n$$\r\nThis implies $\\xi\\ge \\alpha$. Since $\\alpha$ can be arbitrarily close to $\\theta[\\bold{\\Omega}](\\bar{x})$, we deduce\r\n$\\xi\\ge \\theta[\\bold{\\Omega}](\\bar{x})$.\r\n","start":22452,"end":24261}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S019"],"declared_external":false,"source_only":true,"title":"次正则度量表示的双向估计","source_units":[{"raw_tex":"\\emph{Equality \\eqref{errcon'}}.\r\nLet $\\xi$ stand for the \\RHS\\ of \\eqref{errcon'}.\r\nSuppose that $\\xi>0$ and fix an arbitrary number $\\alpha\\in ]0,\\xi[$. Then there is a number $\\delta>0$ such that\r\n\\begin{equation*}\r\n\\alpha d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right) \\le \\max_{1\\le i\\le m} d(x,\\Omega_i),\\; \\forall x \\in B_{\\delta}(\\bar{x}).\r\n\\end{equation*}\r\nIf $x\\in\\bigcap_{i=1}^m\\left(\\Omega_i+ (\\alpha\\rho)\\B \\right)\\bigcap B_{\\delta}(\\bar x)$ for some $\\rho\\in ]0,\\delta[$, then $\\max_{1\\le i\\le m} d(x,\\Omega_i)\\le\\alpha\\rho$, and consequently $d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right) \\le\\rho$, i.e., $\\zeta_{\\rho,\\delta}[\\bold{\\Omega}](\\bar{x}) \\ge \\alpha\\rho$.\r\nHence, $\\zeta[\\bold{\\Omega}](\\bar{x})\\ge \\alpha$. Since $\\alpha$ can be arbitrarily close to $\\xi$, we obtain $\\zeta[\\bold{\\Omega}](\\bar{x})\\ge \\xi$.\r\n\r\nConversely, suppose that $\\zeta[\\bold{\\Omega}](\\bar{x})>0$ and fix any $\\alpha\\in ]0,\\zeta[\\bold{\\Omega}](\\bar{x})[$.\r\nThen there is a number $\\delta>0$ such that\r\n\\eqref{Hlr'} is satisfied for all $\\rho\\in ]0,\\delta[$.\r\nChoose a positive number $\\delta'<\\min\\{\\alpha\\delta,\\delta\\}$.\r\nFor any $x \\in B_{\\delta'}(\\bar{x})$, it holds\r\n$$\r\n\\max_{1\\le i\\le m}d(x,\\Omega_i)\\le \\norm{x-\\bar x}\\le \\delta'<\\alpha\\delta.\r\n$$\r\nChoose a $\\rho\\in]0,\\delta[$ such that $\\max_{1\\le i\\le m}d(x,\\Omega_i)=\\alpha\\rho$.\r\nThen, by \\eqref{Hlr'},\r\n$$\r\n\\alpha d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right) \\le \\alpha\\rho =\\max_{1\\le i\\le m}d(x,\\Omega_i).\r\n$$\r\nHence, $\\alpha\\le\\xi$.\r\nBy letting $\\alpha\\to\\zeta[\\bold{\\Omega}](\\bar{x})$, we obtain $\\zeta[\\bold{\\Omega}](\\bar{x})\\le\\xi$.\r\n","start":24263,"end":25847}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S020"],"declared_external":false,"source_only":true,"title":"一致正则表示的外部引用","source_units":[{"raw_tex":"\\emph{Equality \\eqref{vartheta}} has been proved in \\cite[Theorem 1]{Kru05.1}.\r\n\\qed\r\n\\end{proof}\r\n","start":25849,"end":25948}]},{"id":"S021","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"集合族的乘积集合","source_units":[{"raw_tex":"This section discusses dual characterizations of regularity properties of a collection of sets $\\bold{\\Omega}:=\\{\\Omega_1,\\ldots,\\Omega_m\\}$ at $\\bar x\\in \\bigcap_{i=1}^m\\Omega_i$.\r\nWe are going to use the notation ${\\widehat\\Omega:= \\Omega_1\\times\\ldots\\times\\Omega_m\\subset X^m}$.\r\n\\sloppy\r\n","start":28342,"end":28635}]},{"id":"S022","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"归一化对偶映射","source_units":[{"raw_tex":"\r\nRecall that the (normalized) \\emph{duality mapping} \\cite[Definition~3.2.6]{Lucc06} $J$ between a normed space $Y$ and its dual $Y^*$ is defined as\r\n$$\r\nJ(y):=\\left\\{y^*\\in \\mathbb{S}_{Y^*}:\\; \\langle y^*,y\\rangle=\\norm{y}\\right\\},\\; \\forall y\\in Y.\r\n$$\r\nNote that $J(-y)=-J(y)$.\r\n","start":28635,"end":28918}]},{"id":"S023","semantic_type":"external_fact","kind":"external_fact","labels":["ll01"],"member_ids":["S023"],"declared_external":true,"declaration_text":["原前提列表中的B0066；原文第503行引用 Zălinescu (2002), Corollary 2.4.16。"],"source_only":false,"title":"范数的次微分公式","source_units":[{"raw_tex":"The following simple fact of convex analysis is well known (cf., e.g., \\cite[Corollary~2.4.16]{Zal02}).\r\n\r\n\\begin{lem}\\label{ll01}\r\nLet $(Y,\\|\\cdot\\|)$ be a normed space.\r\n\\begin{enumerate}\r\n\\item\r\n$\\sd\\|\\cdot\\|(y)=J(y)$ for any $y\\ne0$.\r\n\\item\r\n$\\sd\\|\\cdot\\|(0)=\\B^*$.\r\n\\end{enumerate}\r\n\\end{lem}\r\n","start":28920,"end":29219}]},{"id":"S024","semantic_type":"definition","kind":"definition","labels":["norm"],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"参数化乘积范数","source_units":[{"raw_tex":"In this section, along with the maximum type norm on $X^{m+1}=X\\times X^m$, we are going to use another one depending on a parameter $\\rho>0$ and defined as follows:\r\n\\begin{equation}\\label{norm}\r\n\\norm{(x,\\hat x)}_{\\rho} :=\\max\\left\\{\\norm{x},\\rho \\norm{\\hat x}\\right\\},\\quad x\\in X,\\;\\hat x\\in X^m.\r\n\\end{equation}\r\n","start":31029,"end":31347}]},{"id":"S025","semantic_type":"derived_fact","kind":"derived_fact","labels":["normd"],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"参数范数的对偶表示","source_units":[{"raw_tex":"It is easy to check that the corresponding dual norm has the following representation:\r\n\\begin{equation}\\label{normd}\r\n\\norm{(x^*,\\hat x^*)}_{\\rho} =\\|x^*\\|+\\rho^{-1}\\|\\hat x^*\\|,\\quad x^*\\in X^*,\\;\\hat x^*\\in (X^m)^*.\r\n\\end{equation}\r\nNote that if, in \\eqref{norm} and \\eqref{normd}, $\\hat x= (x_1,\\ldots,x_m)$ and $\\hat x^*= (x_1^*,\\ldots,x_m^*)$ with $x_i\\in X$ and $x_i^*\\in X^*$ ($i=1,2,\\ldots,m$), then $\\|\\hat x\\|=\\max_{1\\le i\\le m}\\|x_i\\|$ and $\\|\\hat x^*\\|=\\sum_{i=1}^m\\|x_i^*\\|$.\r\n","start":31347,"end":31838}]},{"id":"S026","semantic_type":"external_fact","kind":"external_fact","labels":["l01"],"member_ids":["S026"],"declared_external":true,"declaration_text":["原前提列表中的B0082；原文引用 Ekeland (1974)、Kruger (2003), Theorem 2.1、Mordukhovich (2006), Theorem 2.26。"],"source_only":false,"title":"Ekeland变分原理","source_units":[{"raw_tex":"The proof of the main theorem of this section relies heavily on two fundamental results of variational analysis: the \\emph{Ekeland variational principle} (Ekeland \\cite{Eke74}; cf., e.g., \\cite[Theorem~2.1]{Kru03.1}, \\cite[Theorem 2.26]{Mor06.1}) and  the \\emph{fuzzy} (\\emph{approximate}) \\emph{sum rule} (Fabian \\cite{Fab89}; cf., e.g., \\cite[Rule~2.2]{Kru03.1}, \\cite[Theorem 2.33]{Mor06.1}).\r\nBelow we provide these results for completeness.\r\n\r\n\\begin{lem}[Ekeland variational principle] \\label{l01}\r\nSuppose $X$ is a complete metric space, and $f: X\\to\\R_\\infty$ is lower semicontinuous and bounded from below, $\\varepsilon>0, \\lambda>0$. If\r\n$$\r\nf(v)<\\inf_X f + \\varepsilon,\r\n$$\r\nthen there exists $x\\in X$ such that\r\n\r\n(a) $d(x,v)<\\lambda $,\r\n\r\n(b) $f(x)\\le f(v)$,\r\n\r\n(c) $f(u)+(\\varepsilon/\\lambda)d(u,x)\\ge f(x)$ for all $u\\in X$.\r\n\\end{lem}\r\n","start":33628,"end":34480}]},{"id":"S027","semantic_type":"external_fact","kind":"external_fact","labels":["l02"],"member_ids":["S027"],"declared_external":true,"declaration_text":["原前提列表中的B0084；第589行引用 Fabian (1989)、Kruger (2003), Rule 2.2、Mordukhovich (2006), Theorem 2.33。"],"source_only":false,"title":"Asplund空间模糊和法则","source_units":[{"raw_tex":"\r\n\\begin{lem}[Fuzzy sum rule] \\label{l02}\r\nSuppose $X$ is Asplund,\r\n$f_1: X \\to\\R$ is Lipschitz continuous and\r\n$f_2: X \\to\\R_\\infty$\r\nis lower semicontinuous in a neighbourhood of $\\bar x$ with $f_2(\\bar x)<\\infty$.\r\nThen, for any $\\varepsilon>0$, there exist $x_1,x_2\\in X$ with $\\|x_i-\\bar x\\|<\\varepsilon$, ${|f_i(x_i)-f_i(\\bar x)|<\\varepsilon}$ $(i=1,2)$ such that\r\n$$\r\n\\partial (f_1+f_2) (\\bar x) \\subset \\partial f_1(x_1) +\\partial f_2(x_2) + \\varepsilon\\B^\\ast.\r\n$$\r\n\\end{lem}\r\n","start":34480,"end":34966}]},{"id":"S028","semantic_type":"definition","kind":"definition","labels":["f"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"扩展值残差函数","source_units":[{"raw_tex":"with function $f:X^{m+1}\\to{\\R}_\\infty :={\\R}\\cup\\{+\\infty\\}$ defined as\r\n\\begin{equation}\\label{f}\r\nf(x,\\hat x)=\\max_{1\\le i\\le m} \\norm{x-x_i} + \\delta_{\\widehat\\Omega}(\\hat x),\\quad x\\in X,\\; \\hat x:=(x_1,\\ldots,x_m)\\in X^m,\r\n\\end{equation}\r\nwhere $\\delta_{\\widehat\\Omega}$ is the indicator function of $\\widehat\\Omega$: $\\delta_{\\widehat\\Omega}(\\hat x)=0$ if $\\hat x\\in\\widehat\\Omega$ and $\\delta_{\\widehat\\Omega}(\\hat x)=+\\infty$ otherwise.\r\n","start":36930,"end":37377}]},{"id":"S029","semantic_type":"derived_fact","kind":"derived_fact","labels":["errcon"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"以残差函数重写模量","source_units":[{"raw_tex":"Observe that constant \\eqref{errcon'} can be rewritten as\r\n\\begin{equation}\\label{errcon}\r\n\\zeta[\\bold{\\Omega}](\\bar{x})= \\liminflarge_{\\substack{x\\to\\bar{x},\\, \\omega_i\\to\\bar x\\; (1\\le i\\le m)\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\\\x\\notin \\bigcap_{i=1}^m\\Omega_i}} \\frac{f(x,\\hat\\omega)} {d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right)}\r\n\\end{equation}\r\n","start":36578,"end":36930}]},{"id":"S030","semantic_type":"proposition","kind":"proposition","labels":["prJ"],"member_ids":["S030","S031","S032","S033"],"declared_external":false,"source_only":false,"title":"最大范数的对偶映射","source_units":[{"raw_tex":"\\begin{prop}\\label{prJ}\r\nFor each $(x_1,\\ldots,x_m)\\in X^m$,\r\n\\begin{multline*}\r\nJ(x_1,\\ldots,x_m)= \\Biggl\\{(x_1^*,\\ldots,x_m^*)\\in(X^*)^m:\\; \\sum_{i=1}^m\\|x_i^*\\|=1;\\mbox{ either } x_i^*=0\\\\\r\n\\mbox{or }\\left(\\|x_i\\|=\\max_{1\\le j\\le m} \\|x_j\\|,\\; x_i^*\\in\\|x_i^*\\| J(x_i)\\right)\\; (i=1,\\ldots,m)\\Biggr\\}.\r\n\\end{multline*}\r\n\\end{prop}","start":29399,"end":29732},{"raw_tex":"\\begin{proof}\r\nLet $\\hat x:=(x_1,\\ldots,x_m)\\in X^m$, $\\hat x^*:=(x_1^*,\\ldots,x_m^*)\\in(X^*)^m$.\r\nThen\r\n$${\\|\\hat x\\|=\\max_{1\\le i\\le m} \\|x_i\\|},\\quad \\|\\hat x^*\\|=\\sum_{i=1}^m\\|x_i^*\\|,\\quad \\langle \\hat x^*,\\hat x\\rangle=\\sum_{i=1}^m\\langle x_i^*,x_i\\rangle.$$\r\nSuppose $\\|\\hat x^*\\|=1$, i.e., $\\sum_{i=1}^m\\|x_i^*\\|=1$.\r\nThen $\\hat x^*\\in J(\\hat x)$ if and only if ${\\sum_{i=1}^m\\langle x_i^*,x_i\\rangle=\\|\\hat x\\|}$.\r\nIn its turn, the last equality holds true if and only if ${\\langle x_i^*,x_i\\rangle=\\|x_i^*\\|\\cdot\\|\\hat x\\|}$ for all $i=1,\\ldots,m$.\r\n\r\nIndeed, if $\\langle x_i^*,x_i\\rangle=\\|x_i^*\\|\\cdot\\|\\hat x\\|$ for all $i=1,\\ldots,m$, then adding these $m$ equalities, we obtain $\\sum_{i=1}^m\\langle x_i^*,x_i\\rangle=\\|\\hat x\\|$.\r\nConversely, if $\\langle x_i^*,x_i\\rangle\\ne\\|x_i^*\\|\\cdot\\|\\hat x\\|$, i.e., ${\\langle x_i^*,x_i\\rangle<\\|x_i^*\\|\\cdot\\|\\hat x\\|}$ for some $i\\in\\{1,\\ldots,m\\}$, then\r\n$$\r\n\\sum_{j=1}^m\\langle x_j^*,x_j\\rangle =\\langle x_i^*,x_i\\rangle+\\sum_{j\\ne i}\\langle x_j^*,x_j\\rangle <\\|x_i^*\\|\\cdot\\|\\hat x\\|+\\|\\hat x\\|\\sum_{j\\ne i} \\|x_j^*\\|=\\|\\hat x\\|.\r\n$$\r\n\r\nFinally, $\\langle x_i^*,x_i\\rangle=\\|x_i^*\\|\\cdot\\|\\hat x\\|$ for some $i\\in\\{1,\\ldots,m\\}$ if and only if either $\\|x_i\\|=\\|\\hat x\\|$ and $x_i^*\\in\\|x_i^*\\| J(x_i)$ or $x_i^*=0$.\r\n\\qed\\end{proof}","start":29734,"end":31025},{"raw_tex":"Let $\\hat x:=(x_1,\\ldots,x_m)\\in X^m$, $\\hat x^*:=(x_1^*,\\ldots,x_m^*)\\in(X^*)^m$.\r\nThen\r\n$${\\|\\hat x\\|=\\max_{1\\le i\\le m} \\|x_i\\|},\\quad \\|\\hat x^*\\|=\\sum_{i=1}^m\\|x_i^*\\|,\\quad \\langle \\hat x^*,\\hat x\\rangle=\\sum_{i=1}^m\\langle x_i^*,x_i\\rangle.$$\r\nSuppose $\\|\\hat x^*\\|=1$, i.e., $\\sum_{i=1}^m\\|x_i^*\\|=1$.\r\nThen $\\hat x^*\\in J(\\hat x)$ if and only if ${\\sum_{i=1}^m\\langle x_i^*,x_i\\rangle=\\|\\hat x\\|}$.\r\nIn its turn, the last equality holds true if and only if ${\\langle x_i^*,x_i\\rangle=\\|x_i^*\\|\\cdot\\|\\hat x\\|}$ for all $i=1,\\ldots,m$.\r\n","start":29749,"end":30294},{"raw_tex":"\r\nIndeed, if $\\langle x_i^*,x_i\\rangle=\\|x_i^*\\|\\cdot\\|\\hat x\\|$ for all $i=1,\\ldots,m$, then adding these $m$ equalities, we obtain $\\sum_{i=1}^m\\langle x_i^*,x_i\\rangle=\\|\\hat x\\|$.\r\nConversely, if $\\langle x_i^*,x_i\\rangle\\ne\\|x_i^*\\|\\cdot\\|\\hat x\\|$, i.e., ${\\langle x_i^*,x_i\\rangle<\\|x_i^*\\|\\cdot\\|\\hat x\\|}$ for some $i\\in\\{1,\\ldots,m\\}$, then\r\n$$\r\n\\sum_{j=1}^m\\langle x_j^*,x_j\\rangle =\\langle x_i^*,x_i\\rangle+\\sum_{j\\ne i}\\langle x_j^*,x_j\\rangle <\\|x_i^*\\|\\cdot\\|\\hat x\\|+\\|\\hat x\\|\\sum_{j\\ne i} \\|x_j^*\\|=\\|\\hat x\\|.\r\n$$\r\n\r\nFinally, $\\langle x_i^*,x_i\\rangle=\\|x_i^*\\|\\cdot\\|\\hat x\\|$ for some $i\\in\\{1,\\ldots,m\\}$ if and only if either $\\|x_i\\|=\\|\\hat x\\|$ and $x_i^*\\in\\|x_i^*\\| J(x_i)$ or $x_i^*=0$.\r\n\\qed\\end{proof}","start":30294,"end":31025}]},{"id":"S034","semantic_type":"lemma","kind":"lemma","labels":["ll02"],"member_ids":["S034","S035","S037","S038"],"declared_external":false,"source_only":false,"title":"残差范数的次微分约束","source_units":[{"raw_tex":"\\begin{lem}\\label{ll02}\r\nLet $X$ be a normed space and\r\n$\\varphi(u,\\hat u)=\\|(u-u_1,\\ldots,u-u_m)\\|$, $u\\in X$, $\\hat u:=(u_1,\\ldots,u_m)\\in X^m$.\r\nSuppose $x\\in X$, $\\hat x:=(x_1,\\ldots,x_m)\\in X^m$, and ${\\hat v:=(x-x_1,\\ldots,x-x_m)}\\ne0$. Then\r\n\\begin{align*}\r\n\\sd\\varphi(x,\\hat x)\\subseteq\\bigl\\{\\left(x^*,\\hat x^* =(x_1^*,\\ldots,x_m^*)\\right)\\in &X^*\\times (X^*)^m :\\;\\\\\r\n&-\\hat x^*\\in J(\\hat v),\\; x^*=-(x_{1}^*+\\ldots+x_{m}^*)\\bigr\\}.\r\n\\end{align*}\r\n\\end{lem}","start":31939,"end":32406},{"raw_tex":"\\begin{proof}\r\n%Observe that $\\varphi$ is a composition of the norm on $X^m$ and the linear continuous mapping $(x,\\hat x)\\mapsto(x_1-x,\\ldots,x_m-x)$.\r\nLet $\\left(x^*,\\hat x^* =(x_1^*,\\ldots,x_m^*)\\right)\\in \\sd\\varphi(x,\\hat x)$, i.e.,\r\n$$\r\n\\|(u-u_1,\\ldots,u-u_m)\\|-\\|(x-x_1,\\ldots,x-x_m)\\|\\ge \\langle x^*,u-x\\rangle+\\sum_{i=1}^m\\langle x_i^*,u_i-x_i\\rangle\r\n$$\r\nfor any $u\\in X$ and $\\hat u:=(u_1,\\ldots,u_m)\\in X^m$.\r\nIn particular, with $u=x$ and $u_i=x_i-x_i'$ ($i=1,\\ldots,m$) for an arbitrary $\\hat x':=(x_1',\\ldots,x_m')\\in X^m$, we have\r\n$$\r\n\\|\\hat v+\\hat x'\\|-\\|\\hat v\\|\\ge-\\langle\\hat x^*,\\hat x'\\rangle,\r\n$$\r\ni.e., $-\\hat x^*\\in J(\\hat v)$.\r\nSimilarly, with $u=x+x'$ and $u_i=x_i+x'$ ($i=1,\\ldots,m$) for an arbitrary $x'\\in X$, we have\r\n$$\r\n\\left\\langle x^*+\\sum_{i=1}^mx_i^*,x'\\right\\rangle\\le0,\r\n$$\r\nand consequently $x^*+x_{1}^*+\\ldots+x_{m}^*=0$.\r\n\\qed\\end{proof}","start":32408,"end":33289},{"raw_tex":"Let $\\left(x^*,\\hat x^* =(x_1^*,\\ldots,x_m^*)\\right)\\in \\sd\\varphi(x,\\hat x)$, i.e.,\r\n$$\r\n\\|(u-u_1,\\ldots,u-u_m)\\|-\\|(x-x_1,\\ldots,x-x_m)\\|\\ge \\langle x^*,u-x\\rangle+\\sum_{i=1}^m\\langle x_i^*,u_i-x_i\\rangle\r\n$$\r\nfor any $u\\in X$ and $\\hat u:=(u_1,\\ldots,u_m)\\in X^m$.\r\nIn particular, with $u=x$ and $u_i=x_i-x_i'$ ($i=1,\\ldots,m$) for an arbitrary $\\hat x':=(x_1',\\ldots,x_m')\\in X^m$, we have\r\n$$\r\n\\|\\hat v+\\hat x'\\|-\\|\\hat v\\|\\ge-\\langle\\hat x^*,\\hat x'\\rangle,\r\n$$\r\ni.e., $-\\hat x^*\\in J(\\hat v)$.\r\n","start":32561,"end":33063},{"raw_tex":"Similarly, with $u=x+x'$ and $u_i=x_i+x'$ ($i=1,\\ldots,m$) for an arbitrary $x'\\in X$, we have\r\n$$\r\n\\left\\langle x^*+\\sum_{i=1}^mx_i^*,x'\\right\\rangle\\le0,\r\n$$\r\nand consequently $x^*+x_{1}^*+\\ldots+x_{m}^*=0$.\r\n\\qed\\end{proof}","start":33063,"end":33289}]},{"id":"S039","semantic_type":"lemma","kind":"lemma","labels":["ll03"],"member_ids":["S039","S040"],"declared_external":false,"source_only":false,"title":"乘积集合的法锥分解","source_units":[{"raw_tex":"\\begin{lem}\\label{ll03}\r\nLet $X$ be a normed space and $\\hat\\omega:=(\\omega_1,\\ldots,\\omega_m)\\in\\widehat\\Omega$.\r\nThen ${N_{\\widehat\\Omega}(\\hat\\omega) =N_{\\Omega_1}(\\omega_1)\\times\\ldots\\times N_{\\Omega_m}(\\omega_m)}$.\r\n\\end{lem}","start":33293,"end":33524},{"raw_tex":"\\begin{proof}\r\nfollows directly from the definition of the Fr\\'echet normal cone.\r\n\\qed\\end{proof}","start":33526,"end":33624}]},{"id":"S041","semantic_type":"assumption","kind":"assumption","labels":["T3.1.1"],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"斜率估计的Banach假设","source_units":[{"raw_tex":"Let $X$ be a Banach space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n","start":37407,"end":37480}]},{"id":"S042","semantic_type":"proposition","kind":"proposition","labels":["T3.1.1","rhoslo","strslo"],"member_ids":["S042","S044","S045","S046","S047","S048","S049"],"declared_external":false,"source_only":false,"title":"原始斜率给出模量下界","source_units":[{"raw_tex":"\\begin{prop}\\label{T3.1.1}\r\nLet $X$ be a Banach space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n\\begin{enumerate}\r\n\\item\r\n$\\hat\\zeta[\\bold{\\Omega}](\\bar x)\\le \\zeta[\\bold{\\Omega}](\\bar{x})$, where\r\n\\begin{equation}\\label{strslo}\r\n\\hat\\zeta[\\bold{\\Omega}](\\bar x):= \\lim_{\\rho\\downarrow0} \\inflarge_{\\substack{\\norm{x-\\bar x}<\\rho\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega)\r\n\\end{equation}\r\nand, for $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega$,\r\n\\begin{equation}\\label{rhoslo}\r\n\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega):= \\limsup_{\\substack{(u,\\hat v)\\to(x,\\hat\\omega)\\\\(u,\\hat v)\\neq (x,\\hat\\omega)\\\\ \\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega}} \\frac{\\left(\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}-\\max\\limits_{1\\le i\\le m} \\norm{u-v_i}\\right)_+}{\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho}}.\r\n\\end{equation}\r\n\\item\r\nIf $\\hat\\zeta[\\bold{\\Omega}](\\bar x)>0$, then $\\bold{\\Omega}$ is subregular at $\\bar x$.\r\n\\end{enumerate}\r\n\\end{prop}","start":37379,"end":38457},{"raw_tex":"\\begin{proof}\r\n(i)\r\nLet $\\zeta[\\bold{\\Omega}](\\bar{x})<\\alpha<\\infty$. Choose a $\\rho\\in ]0,1[$ and set\r\n\\begin{equation}\\label{28}\r\n\\eta:= \\min\\left\\{\\frac{\\rho}{2},\\frac{\\rho}{\\alpha},\\rho^{\\frac{2}{\\rho}}\\right\\}.\r\n\\end{equation}\r\nBy \\eqref{errcon}, there are $x'\\in B_{\\eta}(\\bar x)$ and $\\hat\\omega'=(\\omega_1',\\ldots,\\omega_m')\\in \\widehat\\Omega$ such that\r\n\\begin{equation}\\label{29}\r\n   0<f(x',\\hat\\omega')<\\alpha d\\left(x',\\bigcap_{i=1}^m\\Omega_i\\right).\r\n\\end{equation}\r\nDenote $\\varepsilon:=f(x',\\hat\\omega')$ and $\\mu:=d\\left(x',\\bigcap_{i=1}^m\\Omega_i\\right)$. Then $\\mu\\le \\norm{x'-\\bar x}\\le \\eta\\le \\frac{\\rho}{2}<1$.\r\nObserve that $f$ is lower semicontinuous.\r\nApplying to $f$ Lemma~\\ref{l01} with $\\varepsilon$ as above and\r\n\\begin{equation}\\label{30}\r\n\\lambda:=\\mu(1-\\mu^{\\frac{\\rho}{2-\\rho}}),\r\n\\end{equation}\r\nwe find points $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in X^{m}$ such that\r\n\\begin{equation}\\label{31}\r\n   \\norm{(x,\\hat\\omega)-(x',\\hat\\omega')}_{\\rho}<\\lambda,\\; f(x,\\hat\\omega)\\le f(x',\\hat\\omega'),\r\n\\end{equation}\r\nand\r\n\\begin{equation}\\label{32}\r\nf(u,\\hat v)+\\frac{\\varepsilon}{\\lambda}\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho} \\ge f(x,\\hat\\omega),\r\n\\end{equation}\r\nfor all $(u,\\hat v)\\in X\\times X^{m}$.\r\nThanks to \\eqref{31}, \\eqref{30}, \\eqref{28}, and \\eqref{29}, we have\r\n$$\r\n\\norm{x-x'}<\\lambda <\\mu \\le \\norm{x'-\\bar x},\r\n$$\r\n\\begin{equation}\\label{33}\r\n   d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right)\\ge d\\left(x',\\bigcap_{i=1}^m\\Omega_i\\right)- \\norm{x-x'}\\ge \\mu-\\lambda =\\mu^{\\frac{2}{2-\\rho}},\r\n\\end{equation}\r\n\\begin{equation}\\label{34}\r\n\\norm{x-\\bar x}\\le \\norm{x-x'}+\\norm{x'-\\bar x}<2\\norm{x'-\\bar x}\\le 2\\eta\\le\\rho,\r\n\\end{equation}\r\n\\begin{equation}\\label{35}\r\nf(x,\\hat\\omega)\\le f(x',\\hat\\omega')<\\alpha\\mu\\le \\alpha\\eta\\le\\rho.\r\n\\end{equation}\r\nIt follows from \\eqref{33}, \\eqref{34}, and \\eqref{35} that\r\n$$\r\n\\norm{x-\\bar x}<\\rho,\\; \\hat\\omega\\in \\widehat\\Omega,\\; 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho.\r\n$$\r\nObserve that $\\mu^{\\frac{\\rho}{2-\\rho}}\\le \\eta^{\\frac{\\rho}{2-\\rho}}< \\eta^{\\frac{\\rho}{2}}\\le \\rho$, and consequently, by \\eqref{29} and \\eqref{30},\r\n$$\r\n\\frac{\\varepsilon}{\\lambda} <\\frac{\\alpha\\mu}{\\lambda} =\\frac{\\alpha}{1-\\mu^{\\frac{\\rho}{2-\\rho}}} <\\frac{\\alpha}{1-\\rho}.\r\n$$\r\nThanks to \\eqref{32} and \\eqref{f}, we have\r\n$$\r\n\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}-\\max\\limits_{1\\le i\\le m} \\norm{u-v_i}\\le \\frac{\\alpha}{1-\\rho}\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho}\r\n$$\r\nfor all $u\\in X$ and $\\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega$.\r\nIt follows that $\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega)\\le \\dfrac{\\alpha}{1-\\rho}$ and consequently\r\n$$\r\n\\inflarge_{\\substack{\\norm{x-\\bar x}<\\rho\\\\\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega)\\le \\frac{\\alpha}{1-\\rho}.\r\n$$\r\nTaking limits in the last inequality as $\\rho\\downarrow 0$ and $\\alpha\\to \\zeta[\\bold{\\Omega}](\\bar{x})$ yields the claimed inequality.\r\n\r\n(ii) follows from (i) and Proposition~\\ref{theorem11}~(ii).\r\n\\qed\\end{proof}","start":38461,"end":41548},{"raw_tex":"(i)\r\nLet $\\zeta[\\bold{\\Omega}](\\bar{x})<\\alpha<\\infty$. Choose a $\\rho\\in ]0,1[$ and set\r\n\\begin{equation}\\label{28}\r\n\\eta:= \\min\\left\\{\\frac{\\rho}{2},\\frac{\\rho}{\\alpha},\\rho^{\\frac{2}{\\rho}}\\right\\}.\r\n\\end{equation}\r\nBy \\eqref{errcon}, there are $x'\\in B_{\\eta}(\\bar x)$ and $\\hat\\omega'=(\\omega_1',\\ldots,\\omega_m')\\in \\widehat\\Omega$ such that\r\n\\begin{equation}\\label{29}\r\n   0<f(x',\\hat\\omega')<\\alpha d\\left(x',\\bigcap_{i=1}^m\\Omega_i\\right).\r\n\\end{equation}\r\nDenote $\\varepsilon:=f(x',\\hat\\omega')$ and $\\mu:=d\\left(x',\\bigcap_{i=1}^m\\Omega_i\\right)$. Then $\\mu\\le \\norm{x'-\\bar x}\\le \\eta\\le \\frac{\\rho}{2}<1$.\r\n","start":38476,"end":39096},{"raw_tex":"Observe that $f$ is lower semicontinuous.\r\nApplying to $f$ Lemma~\\ref{l01} with $\\varepsilon$ as above and\r\n\\begin{equation}\\label{30}\r\n\\lambda:=\\mu(1-\\mu^{\\frac{\\rho}{2-\\rho}}),\r\n\\end{equation}\r\nwe find points $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in X^{m}$ such that\r\n\\begin{equation}\\label{31}\r\n   \\norm{(x,\\hat\\omega)-(x',\\hat\\omega')}_{\\rho}<\\lambda,\\; f(x,\\hat\\omega)\\le f(x',\\hat\\omega'),\r\n\\end{equation}\r\nand\r\n\\begin{equation}\\label{32}\r\nf(u,\\hat v)+\\frac{\\varepsilon}{\\lambda}\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho} \\ge f(x,\\hat\\omega),\r\n\\end{equation}\r\nfor all $(u,\\hat v)\\in X\\times X^{m}$.\r\n","start":39096,"end":39713},{"raw_tex":"Thanks to \\eqref{31}, \\eqref{30}, \\eqref{28}, and \\eqref{29}, we have\r\n$$\r\n\\norm{x-x'}<\\lambda <\\mu \\le \\norm{x'-\\bar x},\r\n$$\r\n\\begin{equation}\\label{33}\r\n   d\\left(x,\\bigcap_{i=1}^m\\Omega_i\\right)\\ge d\\left(x',\\bigcap_{i=1}^m\\Omega_i\\right)- \\norm{x-x'}\\ge \\mu-\\lambda =\\mu^{\\frac{2}{2-\\rho}},\r\n\\end{equation}\r\n\\begin{equation}\\label{34}\r\n\\norm{x-\\bar x}\\le \\norm{x-x'}+\\norm{x'-\\bar x}<2\\norm{x'-\\bar x}\\le 2\\eta\\le\\rho,\r\n\\end{equation}\r\n\\begin{equation}\\label{35}\r\nf(x,\\hat\\omega)\\le f(x',\\hat\\omega')<\\alpha\\mu\\le \\alpha\\eta\\le\\rho.\r\n\\end{equation}\r\nIt follows from \\eqref{33}, \\eqref{34}, and \\eqref{35} that\r\n$$\r\n\\norm{x-\\bar x}<\\rho,\\; \\hat\\omega\\in \\widehat\\Omega,\\; 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho.\r\n$$\r\n","start":39713,"end":40445},{"raw_tex":"Observe that $\\mu^{\\frac{\\rho}{2-\\rho}}\\le \\eta^{\\frac{\\rho}{2-\\rho}}< \\eta^{\\frac{\\rho}{2}}\\le \\rho$, and consequently, by \\eqref{29} and \\eqref{30},\r\n$$\r\n\\frac{\\varepsilon}{\\lambda} <\\frac{\\alpha\\mu}{\\lambda} =\\frac{\\alpha}{1-\\mu^{\\frac{\\rho}{2-\\rho}}} <\\frac{\\alpha}{1-\\rho}.\r\n$$\r\nThanks to \\eqref{32} and \\eqref{f}, we have\r\n$$\r\n\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}-\\max\\limits_{1\\le i\\le m} \\norm{u-v_i}\\le \\frac{\\alpha}{1-\\rho}\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho}\r\n$$\r\nfor all $u\\in X$ and $\\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega$.\r\nIt follows that $\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega)\\le \\dfrac{\\alpha}{1-\\rho}$ and consequently\r\n$$\r\n\\inflarge_{\\substack{\\norm{x-\\bar x}<\\rho\\\\\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega)\\le \\frac{\\alpha}{1-\\rho}.\r\n$$\r\nTaking limits in the last inequality as $\\rho\\downarrow 0$ and $\\alpha\\to \\zeta[\\bold{\\Omega}](\\bar{x})$ yields the claimed inequality.\r\n","start":40445,"end":41470},{"raw_tex":"\r\n(ii) follows from (i) and Proposition~\\ref{theorem11}~(ii).\r\n\\qed\\end{proof}","start":41470,"end":41548}]},{"id":"S043","semantic_type":"definition","kind":"definition","labels":["T3.1.1","rhoslo","strslo"],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"局部下降斜率与严格模量","source_units":[{"raw_tex":"\\begin{equation}\\label{strslo}\r\n\\hat\\zeta[\\bold{\\Omega}](\\bar x):= \\lim_{\\rho\\downarrow0} \\inflarge_{\\substack{\\norm{x-\\bar x}<\\rho\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega)\r\n\\end{equation}\r\nand, for $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega$,\r\n\\begin{equation}\\label{rhoslo}\r\n\\zeta_{\\rho}[\\bold{\\Omega}](x,\\hat\\omega):= \\limsup_{\\substack{(u,\\hat v)\\to(x,\\hat\\omega)\\\\(u,\\hat v)\\neq (x,\\hat\\omega)\\\\ \\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega}} \\frac{\\left(\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}-\\max\\limits_{1\\le i\\le m} \\norm{u-v_i}\\right)_+}{\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho}}.\r\n\\end{equation}\r\n","start":37582,"end":38333}]},{"id":"S050","semantic_type":"proposition","kind":"proposition","labels":["T3.1.2","strsubslo'","subrhoslo"],"member_ids":["S050","S052","S053","S054","S055","S056"],"declared_external":false,"source_only":false,"title":"次微分模量的下界","source_units":[{"raw_tex":"\\begin{prop}\\label{T3.1.2}\r\nLet $X$ be an Asplund space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n\\begin{enumerate}\r\n\\item\r\n$\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)\\le \\hat\\zeta[\\bold{\\Omega}](\\bar x)$, where $\\hat\\zeta[\\bold{\\Omega}](\\bar x)$ is given by \\eqref{strslo},\r\n\\begin{equation}\\label{strsubslo'}\r\n\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x):= \\lim_{\\rho\\downarrow 0} \\inflarge_{\\substack{\\norm{x-\\bar x}<\\rho\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x,\\hat\\omega)\r\n\\end{equation}\r\nand, for $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega$,\r\n\\begin{equation}\\label{subrhoslo}\r\n\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x,\\hat\\omega):= \\inflarge_{\\substack{(x^*,\\hat y^*)\\in\\partial f(x,\\hat\\omega)\\\\ \\norm{\\hat y^*}<\\rho}}\\norm{x^*}\r\n\\end{equation}\r\n(with the convention that the infimum over the empty set equals $+\\infty$).\r\n\\item\r\nIf $\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)>0$, then $\\bold{\\Omega}$ is subregular at $\\bar x$.\r\n\\end{enumerate}\r\n\\end{prop}","start":41552,"end":42630},{"raw_tex":"\\begin{proof}\r\nLet $\\hat\\zeta[\\bold{\\Omega}](\\bar x)< \\alpha<\\infty$.\r\nChoose a $\\beta\\in ]\\hat\\zeta[\\bold{\\Omega}](\\bar x),\\alpha[$ and an arbitrary ${\\rho>0}$.\r\nSet ${\\rho'=\\min\\{1,\\alpha^{-1}\\}\\rho}$.\r\nBy \\eqref{strslo} and \\eqref{rhoslo}, one can find points $x\\in X$ and ${\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega}$ such that $\\norm{x-\\bar x}<\\rho'$, $0<\\max_{1\\le i\\le m} \\norm{\\omega_i-x}<\\rho'$, and\r\n$$\r\n\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}-\\max\\limits_{1\\le i\\le m} \\norm{u-v_i}\\le \\beta\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho'}\r\n$$\r\nfor all $(u,\\hat v)$ with $\\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega$ near $(x,\\hat\\omega)$.\r\nIn other words, $(x,\\hat\\omega)$ is a local minimizer of the function\r\n$$\r\n(u,\\hat v)\\mapsto \\max\\limits_{1\\le i\\le m} \\norm{u-v_i}+\\beta\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho'}\r\n$$\r\nsubject to $\\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega$.\r\nBy definition \\eqref{f}, this means that $(x,\\hat\\omega)$ minimizes locally the function\r\n$$\r\n(u,\\hat v)\\mapsto f(u,\\hat v)+\\beta\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho'},\r\n$$\r\nand consequently its Fr\\'echet subdifferential at $(x,\\hat\\omega)$ contains zero.\r\nTake an\r\n$$\r\n\\varepsilon\\in\\Bigl]0,\\min\\{\\rho-\\norm{x-\\bar x},\\rho-\\max_{1\\le i\\le m} \\norm{x-\\omega_i},\\alpha-\\beta\\}\\Bigr[.\r\n$$\r\nApplying Lemma~\\ref{l02} and Lemma~\\ref{ll01}~(ii), we can find points $x'\\in X$, ${\\hat\\omega'=(\\omega_1',\\ldots,\\omega_m')\\in \\widehat\\Omega}$, and $(x^*,\\hat y^*)\\in \\partial f(x',\\hat\\omega')$ such that $$\\norm{x'-x}<\\varepsilon,\\quad 0<\\max_{1\\le i\\le m} \\norm{x'-\\omega_i'}\\le \\max_{1\\le i\\le m} \\norm{x-\\omega_i}+\\varepsilon,$$\r\n$$\\mbox{and}\\quad\\norm{(x^*,\\hat y^*)}_{\\rho'} =\\|x^*\\|+\\|\\hat y^*\\|/\\rho' <\\beta+\\varepsilon.$$\r\nIt follows that\r\n$$\\norm{x'-\\bar x}<\\rho,\\;\\; 0<\\max_{1\\le i\\le m} \\norm{x'-\\omega_i'}<\\rho,\\;\\; \\norm{x^*}<\\alpha,\\;\\mbox{ and }\\; \\norm{\\hat y^*}<\\rho'\\alpha\\le \\rho.$$\r\nHence, $\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x',\\hat\\omega')<\\alpha$, and consequently $\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)<\\alpha$.\r\nBy letting $\\alpha\\to \\hat\\zeta[\\bold{\\Omega}](\\bar x)$, we obtain the claimed inequality.\r\n\r\n(ii) follows from (i) and Proposition~\\ref{T3.1.1}~(ii).\r\n\\qed\\end{proof}","start":42634,"end":44826},{"raw_tex":"Let $\\hat\\zeta[\\bold{\\Omega}](\\bar x)< \\alpha<\\infty$.\r\nChoose a $\\beta\\in ]\\hat\\zeta[\\bold{\\Omega}](\\bar x),\\alpha[$ and an arbitrary ${\\rho>0}$.\r\nSet ${\\rho'=\\min\\{1,\\alpha^{-1}\\}\\rho}$.\r\nBy \\eqref{strslo} and \\eqref{rhoslo}, one can find points $x\\in X$ and ${\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega}$ such that $\\norm{x-\\bar x}<\\rho'$, $0<\\max_{1\\le i\\le m} \\norm{\\omega_i-x}<\\rho'$, and\r\n$$\r\n\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}-\\max\\limits_{1\\le i\\le m} \\norm{u-v_i}\\le \\beta\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho'}\r\n$$\r\nfor all $(u,\\hat v)$ with $\\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega$ near $(x,\\hat\\omega)$.\r\nIn other words, $(x,\\hat\\omega)$ is a local minimizer of the function\r\n$$\r\n(u,\\hat v)\\mapsto \\max\\limits_{1\\le i\\le m} \\norm{u-v_i}+\\beta\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho'}\r\n$$\r\nsubject to $\\hat v=(v_1,\\ldots,v_m)\\in \\widehat\\Omega$.\r\nBy definition \\eqref{f}, this means that $(x,\\hat\\omega)$ minimizes locally the function\r\n$$\r\n(u,\\hat v)\\mapsto f(u,\\hat v)+\\beta\\norm{(u,\\hat v)-(x,\\hat\\omega)}_{\\rho'},\r\n$$\r\nand consequently its Fr\\'echet subdifferential at $(x,\\hat\\omega)$ contains zero.\r\n","start":42649,"end":43792},{"raw_tex":"Take an\r\n$$\r\n\\varepsilon\\in\\Bigl]0,\\min\\{\\rho-\\norm{x-\\bar x},\\rho-\\max_{1\\le i\\le m} \\norm{x-\\omega_i},\\alpha-\\beta\\}\\Bigr[.\r\n$$\r\nApplying Lemma~\\ref{l02} and Lemma~\\ref{ll01}~(ii), we can find points $x'\\in X$, ${\\hat\\omega'=(\\omega_1',\\ldots,\\omega_m')\\in \\widehat\\Omega}$, and $(x^*,\\hat y^*)\\in \\partial f(x',\\hat\\omega')$ such that $$\\norm{x'-x}<\\varepsilon,\\quad 0<\\max_{1\\le i\\le m} \\norm{x'-\\omega_i'}\\le \\max_{1\\le i\\le m} \\norm{x-\\omega_i}+\\varepsilon,$$\r\n$$\\mbox{and}\\quad\\norm{(x^*,\\hat y^*)}_{\\rho'} =\\|x^*\\|+\\|\\hat y^*\\|/\\rho' <\\beta+\\varepsilon.$$\r\n","start":43792,"end":44357},{"raw_tex":"It follows that\r\n$$\\norm{x'-\\bar x}<\\rho,\\;\\; 0<\\max_{1\\le i\\le m} \\norm{x'-\\omega_i'}<\\rho,\\;\\; \\norm{x^*}<\\alpha,\\;\\mbox{ and }\\; \\norm{\\hat y^*}<\\rho'\\alpha\\le \\rho.$$\r\nHence, $\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x',\\hat\\omega')<\\alpha$, and consequently $\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)<\\alpha$.\r\nBy letting $\\alpha\\to \\hat\\zeta[\\bold{\\Omega}](\\bar x)$, we obtain the claimed inequality.\r\n","start":44357,"end":44751},{"raw_tex":"\r\n(ii) follows from (i) and Proposition~\\ref{T3.1.1}~(ii).\r\n\\qed\\end{proof}","start":44751,"end":44826}]},{"id":"S051","semantic_type":"definition","kind":"definition","labels":["T3.1.2","strsubslo'","subrhoslo"],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"次微分局部模量","source_units":[{"raw_tex":"\\begin{equation}\\label{strsubslo'}\r\n\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x):= \\lim_{\\rho\\downarrow 0} \\inflarge_{\\substack{\\norm{x-\\bar x}<\\rho\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x,\\hat\\omega)\r\n\\end{equation}\r\nand, for $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega$,\r\n\\begin{equation}\\label{subrhoslo}\r\n\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x,\\hat\\omega):= \\inflarge_{\\substack{(x^*,\\hat y^*)\\in\\partial f(x,\\hat\\omega)\\\\ \\norm{\\hat y^*}<\\rho}}\\norm{x^*}\r\n\\end{equation}\r\n(with the convention that the infimum over the empty set equals $+\\infty$).\r\n","start":41827,"end":42502}]},{"id":"S057","semantic_type":"proposition","kind":"proposition","labels":["T3.1.3","c2","c2r"],"member_ids":["S057","S059","S060","S061","S062","S063","S064","S065","S066","S067"],"declared_external":false,"source_only":false,"title":"近似法向量模量的下界","source_units":[{"raw_tex":"\\begin{prop}\\label{T3.1.3}\r\nLet $X$ be an Asplund space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n\\begin{enumerate}\r\n\\item\r\n$\\hat\\zeta^*_2[\\bold{\\Omega}](\\bar x)\\le \\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)$, where $\\hat\\zeta^*_1[\\bold{\\Omega}](\\bar x)$ is given by \\eqref{strsubslo'},\r\n\\begin{equation}\\label{c2}\r\n\\hat\\zeta^{*}_2[\\bold{\\Omega}](\\bar x):= \\lim_{\\rho\\downarrow 0} \\inf_{\\substack{\\norm{x-\\bar x}<\\rho\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\lim_{\\varepsilon\\downarrow 0} \\inf_{\\substack{\\norm{x'-x}<\\varepsilon\\\\ \\hat\\omega'\\in \\widehat\\Omega\\\\ \\norm{\\hat\\omega'-\\hat\\omega}<\\varepsilon}}\r\n\\zeta_{\\rho,\\varepsilon,2}^{*}[\\bold{\\Omega}] (x',\\hat\\omega')\r\n\\end{equation}\r\nand, for $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega$ with $(x-\\omega_1,\\ldots,x-\\omega_m)\\ne0$,\r\n\\begin{align}\r\n\\zeta_{\\rho,\\varepsilon,2}^{*}[\\bold{\\Omega}](x,\\hat\\omega):= \\inf\\Biggl\\{\r\n\\left\\|\\sum\\limits_{i=1}^m{x}_i^*\\right\\|:\\,\r\n&\r\nx_{i}^*\\in N_{\\Omega_i}(\\omega_i) +\\rho\\B^*\\quad (i=1,\\ldots,m),\r\n\\notag\\\\&\r\nx_i^*=0\r\n\\quad\\mbox{if}\\quad\r\n\\norm{x-\\omega_i}<\\max_{1\\le j\\le m} \\norm{x-\\omega_j},\r\n\\notag\\\\&\r\n\\langle x_i^*,x-\\omega_i\\rangle\r\n\\ge\\|x_i^*\\|(\\|x-\\omega_i\\|-\\varepsilon),\r\n\\notag\\\\&\r\n\\sum_{i=1}^m\\|x_i^*\\|=1\r\n\\Biggr\\}.\\label{c2r}\r\n\\end{align}\r\n\\item\r\nIf $\\hat\\zeta^*_2[\\bold{\\Omega}](\\bar x)>0$, then $\\bold{\\Omega}$ is subregular at $\\bar x$.\r\n\\end{enumerate}\r\n\\end{prop}","start":44830,"end":46305},{"raw_tex":"\\begin{proof}\r\n(i)\r\nLet $\\rho>0$, $x\\in X$, $\\hat\\omega:= (\\omega_1,\\ldots,\\omega_m)\\in\\widehat\\Omega$ with $\\|x-\\bar x\\|<\\rho$, ${0<\\max_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}$, $(u^*,\\hat v^*) \\in\\partial f(x,\\hat\\omega)$, where $f$ is given by \\eqref{f}, and $\\|\\hat v^*\\|<\\rho$.\r\nDenote $\\hat v:=(x-\\omega_1,\\ldots,x-\\omega_m)$.\r\nThen $0<\\|\\hat v\\|<\\rho$.\r\nObserve that function $f$ is the sum of two functions on $X^{m+1}$:\r\n$$(x,\\hat x)\\mapsto \\varphi(x,\\hat x):= \\|(x-x_1,\\ldots,x-x_m)\\|\\quad\\mbox{and}\\quad(x,\\hat x) \\mapsto\\delta_{\\widehat\\Omega}(\\hat x),$$\r\nwhere $\\hat x:=(x_1,\\ldots,x_m)$ and $\\delta_{\\widehat\\Omega}$ is the indicator function of $\\widehat\\Omega$.\r\nThe first function is Lipschitz continuous while the second one is lower semicontinuous.\r\nOne can apply Lemma~\\ref{l02}.\r\nFor any $\\varepsilon>0$, there exist points $x'\\in X$, $\\hat x:=(x_1,\\ldots,x_m)\\in X^m$, $\\hat\\omega':=(\\omega_1',\\ldots,\\omega_m') \\in\\widehat\\Omega$, $\\left(x^*,\\hat y^*\\right)\\in\\partial \\varphi(x',\\hat x)$, and $\\hat\\omega^*\\in N_{\\widehat\\Omega}(\\hat\\omega')$ such that\r\n\\begin{gather}\\notag\r\n\\|x'-x\\|<\\varepsilon,\\quad \\|\\hat x-\\hat\\omega\\|<\\frac{\\varepsilon}{4},\\quad \\|\\hat\\omega'-\\hat\\omega\\|<\\frac{\\varepsilon}{4},\r\n\\\\\\label{est}\r\n\\|(u^*,\\hat v^*)-(x^*,\\hat y^*)-(0,\\hat\\omega^*)\\|<\\varepsilon.\r\n\\end{gather}\r\nTaking a smaller $\\varepsilon$ if necessary, one can ensure that $\\hat v':=(x'-\\omega_1',\\ldots,x'-\\omega_m')\\ne0$, $\\hat v'':=(x'-x_1,\\ldots,x'-x_m)\\ne0$, $\\|\\hat v^*\\|+\\varepsilon<\\rho$\r\nand,\r\nfor any $i=1,\\ldots,m$, ${\\norm{x'-x_i}<\\max_{1\\le j\\le m} \\norm{x'-x_j}}$ if and only if ${\\norm{x'-\\omega_i'}<\\max_{1\\le j\\le m} \\norm{x'-\\omega_j'}}$.\r\nBy Lemma~\\ref{ll02},\r\n$$\\hat x^*:=-\\hat y^*\\in J(\\hat v'')\r\n\\quad\\mbox{and}\\quad\r\nx^*=x_{1}^*+\\ldots+x_{m}^*,$$\r\nwhere\r\n$\\hat x^*=(x_1^*,\\ldots,x_m^*)$.\r\nBy Proposition~\\ref{prJ},\r\n\\begin{gather*}\r\n\\sum_{i=1}^m\\|x_i^*\\|=1,\r\n\\\\\r\nx_i^*=0\r\n\\quad\\mbox{if}\\quad\r\n\\norm{x'-\\omega_i'}<\\max_{1\\le j\\le m} \\norm{x'-\\omega_j'},\r\n\\end{gather*}\r\n\\begin{align*}\r\n\\langle x_i^*,x'-\\omega_i'\\rangle\r\n&\\ge\\langle x_i^*,x'-x_i\\rangle-\\|x_i^*\\|\\,\\|x_i-\\omega_i'\\| =\\|x_i^*\\|(\\|x'-x_i\\|-\\|x_i-\\omega_i'\\|)\r\n\\\\&\r\n\\ge\\|x_i^*\\|(\\|x'-\\omega_i'\\|-2\\|x_i-\\omega_i'\\|)\r\n\\ge\\|x_i^*\\|(\\|x'-\\omega_i'\\|-\\varepsilon) \\quad(i=1,\\ldots,m).\r\n\\end{align*}\r\nInequality \\eqref{est} yields the estimates:\r\n$\\|u^*\\|>\\left\\|{x}^*\\right\\|-\\varepsilon$, $\\left\\|\\hat x^*-\\hat\\omega^*\\right\\|<\\|\\hat v^*\\|+\\varepsilon<\\rho$,\r\nand consequently\r\n$$\r\n\\|u^*\\|> \\left\\|\\sum\\limits_{i=1}^m{x}_i^*\\right\\|-\\varepsilon, \\quad \\hat x^*\\in N_{\\widehat\\Omega}(\\hat\\omega') +\\rho\\B_m^*.\r\n$$\r\nIt follows from Lemma~\\ref{ll03} and definitions \\eqref{subrhoslo} and \\eqref{c2r} that\r\n$$\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x,\\hat\\omega)\\ge \\zeta_{\\rho,\\varepsilon,2}^*[\\bold{\\Omega}] (x',\\hat\\omega')-\\varepsilon.$$\r\nThe claimed inequality is a consequence of the last one and definitions \\eqref{strsubslo'} and \\eqref{c2}.\r\n\r\n(ii) follows from (i) and Proposition~\\ref{T3.1.2}~(ii).\r\n\\qed\\end{proof}","start":46309,"end":49318},{"raw_tex":"(i)\r\nLet $\\rho>0$, $x\\in X$, $\\hat\\omega:= (\\omega_1,\\ldots,\\omega_m)\\in\\widehat\\Omega$ with $\\|x-\\bar x\\|<\\rho$, ${0<\\max_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}$, $(u^*,\\hat v^*) \\in\\partial f(x,\\hat\\omega)$, where $f$ is given by \\eqref{f}, and $\\|\\hat v^*\\|<\\rho$.\r\nDenote $\\hat v:=(x-\\omega_1,\\ldots,x-\\omega_m)$.\r\nThen $0<\\|\\hat v\\|<\\rho$.\r\nObserve that function $f$ is the sum of two functions on $X^{m+1}$:\r\n$$(x,\\hat x)\\mapsto \\varphi(x,\\hat x):= \\|(x-x_1,\\ldots,x-x_m)\\|\\quad\\mbox{and}\\quad(x,\\hat x) \\mapsto\\delta_{\\widehat\\Omega}(\\hat x),$$\r\nwhere $\\hat x:=(x_1,\\ldots,x_m)$ and $\\delta_{\\widehat\\Omega}$ is the indicator function of $\\widehat\\Omega$.\r\nThe first function is Lipschitz continuous while the second one is lower semicontinuous.\r\n","start":46324,"end":47077},{"raw_tex":"One can apply Lemma~\\ref{l02}.\r\nFor any $\\varepsilon>0$, there exist points $x'\\in X$, $\\hat x:=(x_1,\\ldots,x_m)\\in X^m$, $\\hat\\omega':=(\\omega_1',\\ldots,\\omega_m') \\in\\widehat\\Omega$, $\\left(x^*,\\hat y^*\\right)\\in\\partial \\varphi(x',\\hat x)$, and $\\hat\\omega^*\\in N_{\\widehat\\Omega}(\\hat\\omega')$ such that\r\n\\begin{gather}\\notag\r\n\\|x'-x\\|<\\varepsilon,\\quad \\|\\hat x-\\hat\\omega\\|<\\frac{\\varepsilon}{4},\\quad \\|\\hat\\omega'-\\hat\\omega\\|<\\frac{\\varepsilon}{4},\r\n\\\\\\label{est}\r\n\\|(u^*,\\hat v^*)-(x^*,\\hat y^*)-(0,\\hat\\omega^*)\\|<\\varepsilon.\r\n\\end{gather}\r\n","start":47077,"end":47630},{"raw_tex":"Taking a smaller $\\varepsilon$ if necessary, one can ensure that $\\hat v':=(x'-\\omega_1',\\ldots,x'-\\omega_m')\\ne0$, $\\hat v'':=(x'-x_1,\\ldots,x'-x_m)\\ne0$, $\\|\\hat v^*\\|+\\varepsilon<\\rho$\r\nand,\r\nfor any $i=1,\\ldots,m$, ${\\norm{x'-x_i}<\\max_{1\\le j\\le m} \\norm{x'-x_j}}$ if and only if ${\\norm{x'-\\omega_i'}<\\max_{1\\le j\\le m} \\norm{x'-\\omega_j'}}$.\r\n","start":47630,"end":47980},{"raw_tex":"By Lemma~\\ref{ll02},\r\n$$\\hat x^*:=-\\hat y^*\\in J(\\hat v'')\r\n\\quad\\mbox{and}\\quad\r\nx^*=x_{1}^*+\\ldots+x_{m}^*,$$\r\nwhere\r\n$\\hat x^*=(x_1^*,\\ldots,x_m^*)$.\r\nBy Proposition~\\ref{prJ},\r\n\\begin{gather*}\r\n\\sum_{i=1}^m\\|x_i^*\\|=1,\r\n\\\\\r\nx_i^*=0\r\n\\quad\\mbox{if}\\quad\r\n\\norm{x'-\\omega_i'}<\\max_{1\\le j\\le m} \\norm{x'-\\omega_j'},\r\n\\end{gather*}\r\n","start":47980,"end":48314},{"raw_tex":"\\begin{align*}\r\n\\langle x_i^*,x'-\\omega_i'\\rangle\r\n&\\ge\\langle x_i^*,x'-x_i\\rangle-\\|x_i^*\\|\\,\\|x_i-\\omega_i'\\| =\\|x_i^*\\|(\\|x'-x_i\\|-\\|x_i-\\omega_i'\\|)\r\n\\\\&\r\n\\ge\\|x_i^*\\|(\\|x'-\\omega_i'\\|-2\\|x_i-\\omega_i'\\|)\r\n\\ge\\|x_i^*\\|(\\|x'-\\omega_i'\\|-\\varepsilon) \\quad(i=1,\\ldots,m).\r\n\\end{align*}\r\n","start":48314,"end":48603},{"raw_tex":"Inequality \\eqref{est} yields the estimates:\r\n$\\|u^*\\|>\\left\\|{x}^*\\right\\|-\\varepsilon$, $\\left\\|\\hat x^*-\\hat\\omega^*\\right\\|<\\|\\hat v^*\\|+\\varepsilon<\\rho$,\r\nand consequently\r\n$$\r\n\\|u^*\\|> \\left\\|\\sum\\limits_{i=1}^m{x}_i^*\\right\\|-\\varepsilon, \\quad \\hat x^*\\in N_{\\widehat\\Omega}(\\hat\\omega') +\\rho\\B_m^*.\r\n$$\r\n","start":48603,"end":48918},{"raw_tex":"It follows from Lemma~\\ref{ll03} and definitions \\eqref{subrhoslo} and \\eqref{c2r} that\r\n$$\\zeta_{\\rho,1}^*[\\bold{\\Omega}](x,\\hat\\omega)\\ge \\zeta_{\\rho,\\varepsilon,2}^*[\\bold{\\Omega}] (x',\\hat\\omega')-\\varepsilon.$$\r\nThe claimed inequality is a consequence of the last one and definitions \\eqref{strsubslo'} and \\eqref{c2}.\r\n","start":48918,"end":49243},{"raw_tex":"\r\n(ii) follows from (i) and Proposition~\\ref{T3.1.2}~(ii).\r\n\\qed\\end{proof}","start":49243,"end":49318}]},{"id":"S058","semantic_type":"definition","kind":"definition","labels":["T3.1.3","c2","c2r"],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"归一化近似法向量模量","source_units":[{"raw_tex":"\\begin{equation}\\label{c2}\r\n\\hat\\zeta^{*}_2[\\bold{\\Omega}](\\bar x):= \\lim_{\\rho\\downarrow 0} \\inf_{\\substack{\\norm{x-\\bar x}<\\rho\\\\ \\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega\\\\ 0<\\max\\limits_{1\\le i\\le m} \\norm{x-\\omega_i}<\\rho}}\r\n\\lim_{\\varepsilon\\downarrow 0} \\inf_{\\substack{\\norm{x'-x}<\\varepsilon\\\\ \\hat\\omega'\\in \\widehat\\Omega\\\\ \\norm{\\hat\\omega'-\\hat\\omega}<\\varepsilon}}\r\n\\zeta_{\\rho,\\varepsilon,2}^{*}[\\bold{\\Omega}] (x',\\hat\\omega')\r\n\\end{equation}\r\nand, for $x\\in X$ and $\\hat\\omega=(\\omega_1,\\ldots,\\omega_m)\\in \\widehat\\Omega$ with $(x-\\omega_1,\\ldots,x-\\omega_m)\\ne0$,\r\n\\begin{align}\r\n\\zeta_{\\rho,\\varepsilon,2}^{*}[\\bold{\\Omega}](x,\\hat\\omega):= \\inf\\Biggl\\{\r\n\\left\\|\\sum\\limits_{i=1}^m{x}_i^*\\right\\|:\\,\r\n&\r\nx_{i}^*\\in N_{\\Omega_i}(\\omega_i) +\\rho\\B^*\\quad (i=1,\\ldots,m),\r\n\\notag\\\\&\r\nx_i^*=0\r\n\\quad\\mbox{if}\\quad\r\n\\norm{x-\\omega_i}<\\max_{1\\le j\\le m} \\norm{x-\\omega_j},\r\n\\notag\\\\&\r\n\\langle x_i^*,x-\\omega_i\\rangle\r\n\\ge\\|x_i^*\\|(\\|x-\\omega_i\\|-\\varepsilon),\r\n\\notag\\\\&\r\n\\sum_{i=1}^m\\|x_i^*\\|=1\r\n\\Biggr\\}.\\label{c2r}\r\n\\end{align}\r\n","start":45117,"end":46177}]},{"id":"S068","semantic_type":"assumption","kind":"assumption","labels":["T3.1","T3.1.2","T3.1.3"],"member_ids":["S068"],"declared_external":false,"source_only":false,"title":"Asplund与闭集假设","source_units":[{"raw_tex":"Let $X$ be an Asplund space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n","start":35085,"end":35160},{"raw_tex":"Let $X$ be an Asplund space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n","start":41580,"end":41655},{"raw_tex":"Let $X$ be an Asplund space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n","start":44858,"end":44933}]},{"id":"S069","semantic_type":"assumption","kind":"assumption","labels":["=1",">al","T3.1"],"member_ids":["S069"],"declared_external":false,"source_only":false,"title":"次正则性的法向充分条件","source_units":[{"raw_tex":"$\\bold{\\Omega}$ is subregular at $\\bar x$ if there exist positive numbers $\\alpha$ and $\\delta$ such that, for any $\\rho\\in ]0,\\delta[$, $x\\in B_{\\rho}(\\bar x)$, $\\omega_i\\in\\Omega_i\\cap B_{\\rho}(x)$ $(i=1,\\ldots,m)$ with $\\omega_i\\neq x$ for some ${i\\in\\{1,\\ldots,m\\}}$, there is an $\\varepsilon>0$ such that, for any $x'\\in B_{\\varepsilon}(x)$, $\\omega'_i\\in\\Omega_i\\cap B_{\\varepsilon}(\\omega_i)$, $x_{i}^*\\in N_{\\Omega_i}(\\omega_i') +\\rho\\B^*$ $(i=1,\\ldots,m)$\r\nsatisfying\r\n\\begin{gather}\\notag\r\nx_i^*=0\r\n\\quad\\mbox{if}\\quad\r\n\\norm{x'-\\omega_i'}<\\max_{1\\le j\\le m} \\norm{x'-\\omega_j'},\r\n\\\\\\notag\r\n\\langle x_i^*,x'-\\omega_i'\\rangle\r\n\\ge\\|x_i^*\\|(\\|x'-\\omega_i'\\|-\\varepsilon),\r\n\\\\\\label{=1}\r\n\\sum_{i=1}^m\\|x_i^*\\|=1,\r\n\\end{gather}\r\nit holds\r\n\\begin{gather}\\label{>al}\r\n\\left\\|\\sum\\limits_{i=1}^m{x}_i^*\\right\\|>\\alpha.\r\n\\end{gather}\r\n","start":35186,"end":36023}]},{"id":"S070","semantic_type":"theorem","kind":"theorem","labels":["=1",">al","T3.1"],"member_ids":["S070"],"declared_external":false,"source_only":false,"title":"集合族正则性的对偶判据","source_units":[{"raw_tex":"\\begin{thm}\\label{T3.1}\r\nLet $X$ be an Asplund space and $\\Omega_1$, \\ldots, $\\Omega_m$ be closed.\r\n\\begin{enumerate}\r\n\\item\r\n$\\bold{\\Omega}$ is subregular at $\\bar x$ if there exist positive numbers $\\alpha$ and $\\delta$ such that, for any $\\rho\\in ]0,\\delta[$, $x\\in B_{\\rho}(\\bar x)$, $\\omega_i\\in\\Omega_i\\cap B_{\\rho}(x)$ $(i=1,\\ldots,m)$ with $\\omega_i\\neq x$ for some ${i\\in\\{1,\\ldots,m\\}}$, there is an $\\varepsilon>0$ such that, for any $x'\\in B_{\\varepsilon}(x)$, $\\omega'_i\\in\\Omega_i\\cap B_{\\varepsilon}(\\omega_i)$, $x_{i}^*\\in N_{\\Omega_i}(\\omega_i') +\\rho\\B^*$ $(i=1,\\ldots,m)$\r\nsatisfying\r\n\\begin{gather}\\notag\r\nx_i^*=0\r\n\\quad\\mbox{if}\\quad\r\n\\norm{x'-\\omega_i'}<\\max_{1\\le j\\le m} \\norm{x'-\\omega_j'},\r\n\\\\\\notag\r\n\\langle x_i^*,x'-\\omega_i'\\rangle\r\n\\ge\\|x_i^*\\|(\\|x'-\\omega_i'\\|-\\varepsilon),\r\n\\\\\\label{=1}\r\n\\sum_{i=1}^m\\|x_i^*\\|=1,\r\n\\end{gather}\r\nit holds\r\n\\begin{gather}\\label{>al}\r\n\\left\\|\\sum\\limits_{i=1}^m{x}_i^*\\right\\|>\\alpha.\r\n\\end{gather}\r\n\\item\r\n$\\bold{\\Omega}$ is uniformly regular at $\\bar x$ if and only if there are positive numbers $\\alpha$ and $\\delta$ such that\r\n\\eqref{>al} holds true\r\nfor all $\\omega_i\\in\\Omega_i\\cap{B}_\\delta(\\bar{x})$ and\r\n$x_i^*\\in{N}_{\\Omega_i}(\\omega_i)$ $(i=1,\\ldots,m)$ satisfying \\eqref{=1}. \\end{enumerate}\r\n\\end{thm}","start":35060,"end":36337}]},{"id":"S071","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S071"],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\\emph{of Theorem~\\ref{T3.1}}\r\n(i) follows from Proposition~\\ref{T3.1.3}~(ii) and definitions \\eqref{c2} and \\eqref{c2r}.\r\n\r\n(ii) is a consequence of \\cite[Theorem~4]{Kru09.1}.\r\n\\qed\\end{proof}","start":49322,"end":49527}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S072"],"declared_external":false,"source_only":false,"title":"法向条件转为正模量","source_units":[{"raw_tex":"(i) follows from Proposition~\\ref{T3.1.3}~(ii) and definitions \\eqref{c2} and \\eqref{c2r}.\r\n","start":49365,"end":49457}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S073"],"declared_external":false,"source_only":false,"title":"一致正则判据的外部应用","source_units":[{"raw_tex":"\r\n(ii) is a consequence of \\cite[Theorem~4]{Kru09.1}.\r\n\\qed\\end{proof}","start":49457,"end":49527}]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S074"],"declared_external":true,"declaration_text":["原前提列表中的Basic convex subdifferential calculus；由标注者显式列出其在第564—568行的具体用途。"],"source_only":false,"title":"凸次梯度的全局不等式","source_units":[]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S075"],"declared_external":true,"declaration_text":["原前提列表中的the subdifferential Fermat rule；由标注者明确化。"],"source_only":false,"title":"次微分Fermat规则","source_units":[]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S076"],"declared_external":true,"declaration_text":["原前提列表中的identification of indicator subdifferentials with normal cones；由标注者明确其乘积形式。"],"source_only":false,"title":"指示函数次微分与法锥","source_units":[]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S077"],"declared_external":true,"declaration_text":["原前提列表中的Completeness of Asplund spaces and preservation under finite products and equivalent norms；Banach有限乘积完备性为implicit standard fact made explicit by annotator。"],"source_only":false,"title":"完备性与Asplund稳定性","source_units":[]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S078"],"declared_external":true,"declaration_text":["原前提列表：Kruger (2009), Theorem 4；原文第876行引用Kru09.1，书目信息见第1272—1274行。"],"source_only":false,"title":"Kruger一致正则定理","source_units":[]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S079"],"declared_external":true,"declaration_text":["原前提列表：Kruger (2005), Theorem 1；原文第453行引用Kru05.1，书目信息见第1264—1266行。"],"source_only":false,"title":"Kruger度量表示定理","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S030","S032":"S030","S033":"S030","S034":"S034","S035":"S034","S037":"S034","S038":"S034","S039":"S039","S040":"S039","S041":"S041","S042":"S042","S043":"S043","S044":"S042","S045":"S042","S046":"S042","S047":"S042","S048":"S042","S049":"S042","S050":"S050","S051":"S051","S052":"S050","S053":"S050","S054":"S050","S055":"S050","S056":"S050","S057":"S057","S058":"S058","S059":"S057","S060":"S057","S061":"S057","S062":"S057","S063":"S057","S064":"S057","S065":"S057","S066":"S057","S067":"S057","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076","S077":"S077","S078":"S078","S079":"S079"}}}}
{"record_id":"1209.0652v2","paper":{"title":"Necessary and sufficient conditions of solution uniqueness in $\\ell_1$ minimization","authors":["Zhang, Hui","Yin, Wotao","Cheng, Lizhi"],"arxiv_id":"1209.0652","version":"v2","journal":"Journal of Optimization Theory and Applications","abstract_url":"https://arxiv.org/abs/1209.0652v2","pdf_url":"https://arxiv.org/pdf/1209.0652v2","doi":"10.1007/s10957-014-0581-z"},"context":{"raw_tex":"Let $x\\in\\RR^n$ be the decision variable. This paper studies the unique solutions of the  $\\ell_1$ minimization problems  including the basis pursuit problem  \\cite{CDS}\n\\begin{subequations}\n\\label{eq:qp}\n\\beq\\label{eq:bp}\n\\min\\|x\\|_1\\quad\\text{s.t.}~Ax=b\n\\eeq\nand  convex problems\n\\begin{align}\n\\label{eq:qpa}\n&\\min f_1(Ax-b)+\\lambda \\|x\\|_1,\\\\\n\\label{eq:qpb}\n&\\min \\|x\\|_1,\\quad\\text{s.t.}~f_2(Ax-b)\\leq \\sigma, \\\\\n\\label{eq:qpc}\n&\\min f_3(Ax-b), \\quad\\text{s.t.}~  \\|x\\|_1\\leq \\tau,\n\\end{align}\n\\end{subequations}\nwhere $\\lambda,\\sigma,\\tau>0$ are scalar parameters,  $A$ is a matrix, and $f_i(x), i=1, 2, 3$ are strictly convex functions. The Lasso problem \\cite{Ti} is a special case of  problem \\eqref{eq:qpa} or \\eqref{eq:qpc} while the basis pursuit denoising problem \\cite{CDS} is a special case of  problem \\eqref{eq:qpb} all with $f_i(\\cdot)=\\frac{1}{2}\\|\\cdot\\|_2^2,~ i=1, 2, 3$.\n\n\nWe let $X$, $X_\\lambda$, $Y_\\sigma$, and $Z_\\tau$ denote the sets of solutions to  problems \\eqref{eq:bp}--\\eqref{eq:qpc}, respectively. We let $a_i$ be the $i$th column of $A$ and $x_i$ be the $i$th entry of $x$. Given an index set $I$, we frequently use $A_I$ as the submatrix of $A$ formed by its columns $a_i$,  $i\\in I$ and  $x_I$ as the subvector of $x$ formed by entries $x_i$, $i\\in I$.\n\nOur analysis makes the following assumptions:\n\n\\begin{assumption}\\label{assmp1}\nMatrix $A$ has full row rank.\n\\end{assumption}\n\\begin{assumption}\\label{assmp2}\nThe solution sets $X$, $X_\\lambda$, $Y_\\sigma$, and $Z_\\tau$ of problems \\eqref{eq:bp}--\\eqref{eq:qpc}, respectively, are nonempty.\n\\end{assumption}\n\\begin{assumption}\\label{assmp3} In problems \\eqref{eq:qpa}--\\eqref{eq:qpc}, functions $f_1,f_2,f_3$ are strictly convex. In addition,  the constraint of problem \\eqref{eq:qpc} is bounding, namely, $\\tau \\le \\inf \\{\\|x\\|_1: f_3(Ax-b) = f_3^*\\}$, where $f_3^*:=\\min_{y\\in\\RR^n} f(Ay- b)$.\n\\end{assumption}\nAssumptions \\ref{assmp1} and \\ref{assmp2} are standard. If Assumption \\ref{assmp1} does not hold and $Ax=b$ is consistent, the problems can be simplified; specifically, one can decompose $A=\\begin{bmatrix}A_1\\\\ A_2\\end{bmatrix}$ and $b=\\begin{bmatrix}b_1\\\\ b_2\\end{bmatrix}$ so that and  $A_1$ has full row rank equal to $\\mathrm{rank}(A)$, and one can replace the constraints $Ax = b$ by $A_1x = b_1$ and introduce functions $\\bar{f}_i$ so that $\\bar{f}_i(A_1x - b_1) \\equiv f_i(Ax-b)$, $i=1,2,3$. Assumption \\ref{assmp2} guarantees that the solutions of problems \\eqref{eq:bp}--\\eqref{eq:qpc} can be attained so the discussion of solution uniqueness makes sense.  The strict convexity of $f_1,f_2,f_3$ and the restriction on $\\tau$ in Assumption \\ref{assmp3} are  also fairly basic toward solution uniqueness. Strict convexity rules out piece-wise linearity. (Note that $f_1,f_2,f_3$ can still be non-differentiable.) If the restriction on $\\tau$ is removed, the solution uniqueness of  problem \\eqref{eq:qpc} becomes solely  up to $f_3(Ax-b)$, independent of $\\|x\\|_1$. % and Figure \\ref{?} illustrates this point by an example.\n\nFor a given vector $x^*$, the following conditions on matrix $A$ is the key to solution uniqueness, and its sufficiency has been established in \\cite{Fu}.\n\n\\begin{condition}\\label{cond:unique}  Under the definitions  $I:=\\supp(x^*)\\subseteq\\{1,\\ldots,n\\}$ and $s:=\\sign(x_I^*)$, matrix $A\\in\\mathbb{R}^{m\\times n}$ has the following properties:\n\\begin{enumerate}\n\\item submatrix $A_{I}$ has full column rank, and\n\\item there is  $y \\in \\mathbb{R}^m$ obeying  $A_{I}^Ty=s$ and $\\|A_{I^c}^Ty\\|_\\infty <1$.\n\\end{enumerate}\n\\end{condition}\n\n\nWe establish Theorem \\ref{thm:unique} in three steps. Our first step proves the theorem for problem \\eqref{eq:bp} only. Since the only difference between this part and Theorem \\ref{thm:bp} is the conditions $Ax^*=b$, we  prove Theorem \\ref{thm:bp} first. In the second step, for problems \\eqref{eq:qpa}--\\eqref{eq:qpc}, we show that both $\\|x\\|_1$ and $Ax-b$ are constant for $x$ over the solution sets $X_\\lambda, Y_\\sigma,Z_\\tau$, respectively. Finally,  we prove Theorem \\ref{thm:unique}  for problems \\eqref{eq:qpa}--\\eqref{eq:qpc}.\n","units":[{"raw_tex":"Let $x\\in\\RR^n$ be the decision variable. This paper studies the unique solutions of the  $\\ell_1$ minimization problems  including the basis pursuit problem  \\cite{CDS}\n\\begin{subequations}\n\\label{eq:qp}\n\\beq\\label{eq:bp}\n\\min\\|x\\|_1\\quad\\text{s.t.}~Ax=b\n\\eeq\nand  convex problems\n\\begin{align}\n\\label{eq:qpa}\n&\\min f_1(Ax-b)+\\lambda \\|x\\|_1,\\\\\n\\label{eq:qpb}\n&\\min \\|x\\|_1,\\quad\\text{s.t.}~f_2(Ax-b)\\leq \\sigma, \\\\\n\\label{eq:qpc}\n&\\min f_3(Ax-b), \\quad\\text{s.t.}~  \\|x\\|_1\\leq \\tau,\n\\end{align}\n\\end{subequations}\nwhere $\\lambda,\\sigma,\\tau>0$ are scalar parameters,  $A$ is a matrix, and $f_i(x), i=1, 2, 3$ are strictly convex functions. The Lasso problem \\cite{Ti} is a special case of  problem \\eqref{eq:qpa} or \\eqref{eq:qpc} while the basis pursuit denoising problem \\cite{CDS} is a special case of  problem \\eqref{eq:qpb} all with $f_i(\\cdot)=\\frac{1}{2}\\|\\cdot\\|_2^2,~ i=1, 2, 3$.\n","start":9278,"end":10170},{"raw_tex":"We let $X$, $X_\\lambda$, $Y_\\sigma$, and $Z_\\tau$ denote the sets of solutions to  problems \\eqref{eq:bp}--\\eqref{eq:qpc}, respectively. We let $a_i$ be the $i$th column of $A$ and $x_i$ be the $i$th entry of $x$. Given an index set $I$, we frequently use $A_I$ as the submatrix of $A$ formed by its columns $a_i$,  $i\\in I$ and  $x_I$ as the subvector of $x$ formed by entries $x_i$, $i\\in I$.\n\nOur analysis makes the following assumptions:\n\n\\begin{assumption}\\label{assmp1}\nMatrix $A$ has full row rank.\n\\end{assumption}\n\\begin{assumption}\\label{assmp2}\nThe solution sets $X$, $X_\\lambda$, $Y_\\sigma$, and $Z_\\tau$ of problems \\eqref{eq:bp}--\\eqref{eq:qpc}, respectively, are nonempty.\n\\end{assumption}\n\\begin{assumption}\\label{assmp3} In problems \\eqref{eq:qpa}--\\eqref{eq:qpc}, functions $f_1,f_2,f_3$ are strictly convex. In addition,  the constraint of problem \\eqref{eq:qpc} is bounding, namely, $\\tau \\le \\inf \\{\\|x\\|_1: f_3(Ax-b) = f_3^*\\}$, where $f_3^*:=\\min_{y\\in\\RR^n} f(Ay- b)$.\n\\end{assumption}\nAssumptions \\ref{assmp1} and \\ref{assmp2} are standard. If Assumption \\ref{assmp1} does not hold and $Ax=b$ is consistent, the problems can be simplified; specifically, one can decompose $A=\\begin{bmatrix}A_1\\\\ A_2\\end{bmatrix}$ and $b=\\begin{bmatrix}b_1\\\\ b_2\\end{bmatrix}$ so that and  $A_1$ has full row rank equal to $\\mathrm{rank}(A)$, and one can replace the constraints $Ax = b$ by $A_1x = b_1$ and introduce functions $\\bar{f}_i$ so that $\\bar{f}_i(A_1x - b_1) \\equiv f_i(Ax-b)$, $i=1,2,3$. Assumption \\ref{assmp2} guarantees that the solutions of problems \\eqref{eq:bp}--\\eqref{eq:qpc} can be attained so the discussion of solution uniqueness makes sense.  The strict convexity of $f_1,f_2,f_3$ and the restriction on $\\tau$ in Assumption \\ref{assmp3} are  also fairly basic toward solution uniqueness. Strict convexity rules out piece-wise linearity. (Note that $f_1,f_2,f_3$ can still be non-differentiable.) If the restriction on $\\tau$ is removed, the solution uniqueness of  problem \\eqref{eq:qpc} becomes solely  up to $f_3(Ax-b)$, independent of $\\|x\\|_1$. % and Figure \\ref{?} illustrates this point by an example.\n\nFor a given vector $x^*$, the following conditions on matrix $A$ is the key to solution uniqueness, and its sufficiency has been established in \\cite{Fu}.\n\n\\begin{condition}\\label{cond:unique}  Under the definitions  $I:=\\supp(x^*)\\subseteq\\{1,\\ldots,n\\}$ and $s:=\\sign(x_I^*)$, matrix $A\\in\\mathbb{R}^{m\\times n}$ has the following properties:\n\\begin{enumerate}\n\\item submatrix $A_{I}$ has full column rank, and\n\\item there is  $y \\in \\mathbb{R}^m$ obeying  $A_{I}^Ty=s$ and $\\|A_{I^c}^Ty\\|_\\infty <1$.\n\\end{enumerate}\n\\end{condition}\n","start":12330,"end":15009},{"raw_tex":"We establish Theorem \\ref{thm:unique} in three steps. Our first step proves the theorem for problem \\eqref{eq:bp} only. Since the only difference between this part and Theorem \\ref{thm:bp} is the conditions $Ax^*=b$, we  prove Theorem \\ref{thm:bp} first. In the second step, for problems \\eqref{eq:qpa}--\\eqref{eq:qpc}, we show that both $\\|x\\|_1$ and $Ax-b$ are constant for $x$ over the solution sets $X_\\lambda, Y_\\sigma,Z_\\tau$, respectively. Finally,  we prove Theorem \\ref{thm:unique}  for problems \\eqref{eq:qpa}--\\eqref{eq:qpc}.\n","start":22058,"end":22595}]},"statement":{"raw_tex":"\\begin{theorem}[Solution uniqueness]\\label{thm:unique} Under Assumptions \\ref{assmp1}--\\ref{assmp3}, given that $x^*$ is a solution  to problem \\eqref{eq:bp}, \\eqref{eq:qpa}, \\eqref{eq:qpb}, or \\eqref{eq:qpc}, $x^*$ is  the unique solution if and only if Condition \\ref{cond:unique} holds.\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}[Solution uniqueness]\\label{thm:unique} Under Assumptions \\ref{assmp1}--\\ref{assmp3}, given that $x^*$ is a solution  to problem \\eqref{eq:bp}, \\eqref{eq:qpa}, \\eqref{eq:qpb}, or \\eqref{eq:qpc}, $x^*$ is  the unique solution if and only if Condition \\ref{cond:unique} holds.\n\\end{theorem}","start":15151,"end":15454}]},"proof":{"raw_tex":"\\begin{theorem}[Basis pursuit  unique optimality]\\label{thm:bp} Under Assumptions \\ref{assmp1}--\\ref{assmp2},\n$x^*\\in\\mathbb{R}^n$ is the unique solution to problem \\eqref{eq:bp} if and only if $Ax^*=b$ and Condition \\ref{cond:unique} is satisfied.\n\\end{theorem}\n\n\\begin{IEEEproof}[Proof of Theorem \\ref{thm:bp}] We frequently use the notions  $I=\\supp(x^*)$ and $s=\\sign(x_I^*)$ below.\n\\medskip\n\n``$\\Longleftarrow$''. This part has been shown in \\cite{Fu, Tr}. For completeness, we give a  proof. Let $y$ satisfy Condition \\ref{cond:unique}, part 2, and let $x\\in\\RR^n$ be an arbitrary vector satisfying $Ax = b$ and $x\\not=x^*$. We shall show $\\|x^*\\|_1<\\|x\\|_1$.\n\nSince $A_I$ has full column rank and $x\\not=x^*$, we have $\\supp(x)\\not=I$; otherwise from $A_I x^*_I =b= A_I x_I$, we would get $x_I^* = x_I$ and  thus the contradiction $x^*=x$.\n\nFrom $\\supp(x)\\not=I$, we  get $b^T y < \\|x\\|_1$. To see this, let $J := \\supp(x) \\setminus I$, which is a non-empty subset of $I^c$. From Condition \\ref{cond:unique}, we have $\\|A_I^T y\\|_\\infty =1$ and $\\|A_J^T y\\|_\\infty < 1$, and thus\n\\begin{align*}\n\\langle x_I, A_I^T y\\rangle&\\le \\|x_I\\|_1\\cdot\\|A_I^T y\\|_\\infty\\le\\|x_I\\|_1,\\\\\n\\langle x_J, A_J^T y\\rangle&\\le \\|x_J\\|_1\\cdot\\|A_J^T y\\|_\\infty<\\|x_J\\|_1,\n\\end{align*}\n(the last inequality is ``$<$'' not ``$\\le$'') which lead to\n\\begin{eqnarray*}\nb^T y = \\langle x, A^T y\\rangle& = &\\langle x_I, A_I^T y\\rangle+ \\langle x_J, A_J^T y\\rangle\\\\\n &<& \\|x_I\\|_1  +\\|x_J\\|_1 = \\|x\\|_1.\n\\end{eqnarray*}\n\nOn the other hand, we have\n$$\n\\|x^*\\|_1 = \\langle x^*_I, \\sign(x^*_I)\\rangle = \\langle x^*_I, A_I^T y\\rangle = \\langle A_I x^*_I,y\\rangle = b^T y\n$$\nand thus $\\|x^*\\|_1=b^T y<\\|x\\|_1$.%, from which it follows $x^*$ is the unique solution to \\eqref{eq:bp}.\n\\medskip\n\n\\noindent``$\\Longrightarrow$''. Assume that $x^*$ is the unique solution to \\eqref{eq:bp}. Obviously, $Ax^*=b$.\n\nIt is easy to obtain Condition \\ref{cond:unique}, part 1. Suppose it does not hold. Then, $A_I$ has a nontrivial null space, and perturbing $x_I^*$ along the null space will change the objective $\\|x_I^*\\|_1 = s^T x^*_I$ while maintaining $A_I x_I^*=b$; hence, this  perturbing breaks the unique optimality of $x^*$.\n(In more details, there exists a \\emph{nonzero} vector $d\\in\\RR^{n}$ such that $A_I d_I = \\vzero$ and $d_{I^c}=\\vzero$. For any scalar $\\alpha$ near zero, we have $\\sign(x^*_I+\\alpha d_I)=\\sign(x_I^*)=s$ and thus $\\|x^*+\\alpha d\\|_1= s^T (x^*_I+\\alpha d_I)=s^T x^*_I+\\alpha( s^T d_I)=\\|x^*\\|_1+\\alpha( s^T d_I)$. Since $x^*$ is the unique solution, we must have $\\|x^*_I+\\alpha d\\|_1> \\|x^*\\|_1$ or, equivalently, $\\alpha( s^T d_I)>0$ whenever $\\alpha\\not=0$. This is impossible as we can perturb $\\alpha$ around 0 both ways.)\n\nIt remains to construct a vector $y$ for Condition \\ref{cond:unique}, part 2.\nOur construction is based on  the strong convexity relation between a linear program (called the primal problem) and its dual problem, namely, if one problem has a solution, so does the other, and the two solutions must give the same objective value. (For the interested reader, this result follows from the Hahn-Banach separation theorem, also from the theorem of alternatives \\cite{Da}. Alternatively, it can be  obtain constructively via the Simplex method; specifically, whenever a primal solution exists, the Simplex method terminates in a finite number of steps with a primal-dual solution pair.)\n\nThe strong duality relation holds between  \\eqref{eq:bp} and its dual problem\n\\beq\\label{eq:bpdual}\n\\max_{p\\in\\RR^m} b^Tp\\quad\\text{s.t.}~ \\|A^Tp\\|_\\infty \\le 1\n\\eeq\nbecause  \\eqref{eq:bp} and \\eqref{eq:bpdual}, as a primal-dual pair, are equivalent to the primal-dual linear programs\n\\begin{subequations}\\label{eq:pdpair}\n\\begin{align}\n\\label{eq:pdpaira}\n\\min_{u,v\\in\\RR^n} \\vone^Tu + \\vone^T v & \\quad \\text{s.t.}~A u - Av = b, ~u\\ge 0, ~v\\ge 0,\\\\\n\\label{eq:pdpairb}\n\\max_{q\\in \\RR^m} b^T q& \\quad \\text{s.t.} -\\vone\\le A^T q\\le \\vone,\n\\end{align}\n\\end{subequations}\nrespectively, where the strong duality relation holds between \\eqref{eq:pdpaira} and \\eqref{eq:pdpairb}. By ``equivalent'', we mean that one can obtain solutions from each other by the rules:\n$$\\begin{array}{ll}\n\\text{given}~u^*,v^*, & \\text{obtain}~ x^* = u^* - v^*\\\\\n\\text{given}~x^*, & \\text{obtain}~ u^*= \\max(x^*,\\vzero), ~v^*= \\max(-x^*,\\vzero),\\\\\n\\text{given}~q^*, & \\text{obtain}~ p^*= q^*,\\\\\n\\text{given}~p^*, & \\text{obtain}~ q^*= p^*.\n\\end{array}$$\nTherefore, since \\eqref{eq:bp} has solution $x^*$, there exists a solution $y^*$ to \\eqref{eq:bpdual}, which satisfies $\\|x^*\\|_1 = b^T y^*$ and $\\|A^T y^*\\|_\\infty\\le 1$. (One can obtain such $y^*$ from the  Hahn-Banach separation theorem or the theorem of alternatives rather directly.) However,  $y^*$ may \\emph{not} obey $\\|A_{I^c}^Ty^*\\|_\\infty <1$. We shall perturb $y^*$ so that $\\|A_{I^c}^Ty^*\\|_\\infty <1$.\n\nTo prepare for the perturbation, we let $L:=\\{i\\in I^c:a_i^Ty^* = -1 \\}$ and $U:=\\{i\\in I^c:a_i^Ty^* = 1\\}$. Our goal is to perturb $y^*$ so that  $-1<a_i^Ty^*<1$ for $i\\in L\\cup U$ and $y^*$ remains optimal to  \\eqref{eq:bpdual}.\nTo this end, consider for a fixed $\\alpha>0$ and $t:=\\|x^*\\|_1$, the linear program\n\\beq\\label{eq:iprimal}\n\\min_{x\\in\\RR^n}  \\sum_{i\\in L}\\alpha x_i-\\sum_{i\\in U}\\alpha x_i,\\quad\\text{s.t.}~Ax=b,~\\|x\\|_1\\le t.\n\\eeq\nSince $x^*$ is the unique solution to \\eqref{eq:bp}, it is  the unique feasible solution to problem \\eqref{eq:iprimal}, so problem \\eqref{eq:iprimal} has the optimal objective value $\\sum_{i\\in U}\\alpha x_i^* - \\sum_{i\\in L}\\alpha x_i^*=0$.\nBy setting up equivalent linear programs like what has been done for \\eqref{eq:bp} and \\eqref{eq:bpdual} above,  the strong duality relation holds between \\eqref{eq:iprimal} and its dual problem\n\\beq\\label{eq:idual}\n\\max_{p\\in\\RR^m,q\\in \\RR} b^T p - tq,\\quad\\text{s.t.}~\\|A^Tp - \\alpha r\\|_\\infty \\le q,~q\\ge 0,\n\\eeq\nwhere  $r\\in\\RR^n$ is given by\n$$r_i = \\begin{cases}1,&i\\in L,\\\\\n-1,&i\\in U,\\\\\n0,&\\text{otherwise}.\n\\end{cases}$$\nTherefore, \\eqref{eq:idual} has a solution $(p^*,q^*)$ satisfying $b^T p^*- tq^*= 0$.\n\nAccording to the last constraint of \\eqref{eq:idual}, we have $q^*\\ge 0$, which we split into two cases:  $q^*=0$ and  $q^*>0$.\n\\begin{itemize}\n\\item[i)] If $q^*=0$, we have $A^T p^*=\\alpha r$ and $b^T p^*=0$.\n\\item[ii)] If $q^*>0$, we let $r^*:=p^*/q^*$, which satisfies $b^T r^* =t=  \\|x^*\\|_1$ and $\\|A^Tr^* - \\frac{\\alpha}{q^*}r\\|_\\infty \\le 1$, or equivalently, $-\\vone+\\frac{\\alpha}{q^*}r\\le A^Tr^*  \\le \\vone+\\frac{\\alpha}{q^*}r$.\n\\end{itemize}\nNow we perturb $y^*$. Solve \\eqref{eq:idual}  with a sufficiently small $\\alpha > 0$ and obtain a solution $(p^*,q^*)$. If case i) occurs, we let $y^* \\gets y^* +p^*$; otherwise, we let $y^* \\gets \\frac{1}{2}(y^* +r^*)$. In both cases,\n\\begin{itemize}\n\\item $b^T y^*$ is unchanged, still equal to $\\|x^*\\|_1$;\n\\item $-1<a_i^Ty^*<1$ holds for $i\\in L\\cup U$ after the perturbation;\n\\item for each $i \\not \\in L\\cup U$, if $a_j^Ty^*\\in[-1,1]$ or $a_j^Ty^*\\in(-1,1)$ holds before the perturbation, the same holds after the perturbation;\n\\end{itemize}\nTherefore, after the perturbation, $y^*$ satisfies:\n\\begin{itemize}\n\\item[1)] $b^T y^* = \\|x^*\\|_1$,\n\\item[2)] $\\|A_I^T y^*\\|_\\infty\\le 1$, and\n\\item[3)] $\\|A_{I^c}^T y^*\\|_\\infty <1$.\n\\end{itemize}\nFrom 1) and 2) it follows\n\\begin{itemize}\n\\item[4)] $A_I^T y = \\sign(x_I^*)$\n\n\\end{itemize}\nsince $\\|x_I^*\\|_1 = \\|x^*\\|_1 = b^T y^* = \\langle A_I x_I^*,y^*\\rangle = \\langle x_I^*, A_I^T y^*\\rangle\\le \\|x_I^*\\|_1 \\|A^T y^*\\|_\\infty \\le \\|x_I^*\\|_1$ and thus $\\langle x_I^*, A_I^T y^*\\rangle = \\|x_I^*\\|_1$, which dictates 4). From 3) and 4), Condition \\ref{cond:unique}, part 2, holds with $y=y^*$.\n\\end{IEEEproof}\n\n\\begin{IEEEproof}[Proof of Theorem \\ref{thm:unique} for problem \\eqref{eq:bp}]\nThe above proof for Theorem \\ref{thm:bp} also serves the proof of  Theorem \\ref{thm:unique} for problem \\eqref{eq:bp} since $Ax^*=b$ is  involved only in the optimality part, not the uniqueness part.\n\\end{IEEEproof}\n\n\\begin{lemma}\\label{lem01}\nLet $f$ be a strictly convex function. If $f(Ax-b)+\\|x\\|_1$ is constant on a convex set $S$, then both $Ax-b$ and $\\|x\\|_1$ are  constant on $S$.\n\\end{lemma}\n\n\\begin{IEEEproof}\nIt suffices to prove the case where $S$ has more one point. Let $x_1$ and $x_2$ be any two  different points in $S$. Consider the line segment $L$ connecting $x_1$ and $x_2$. Since  $X$ is convex, we have $L\\subset X$ and that   $f(Ax-b)+\\|x\\|_1$ is constant on $L$.  On one hand, $\\|x\\|_1$ is piece-wise linear over $L$; on the other hand, the strict convexity of $f$ makes it impossible for $f(Ax-b)$ to be piece-wise linear over $L$ unless $Ax_1-b=Ax_2-b$. Hence, we have $Ax_1-b=Ax_2-b$ and thus $f(Ax_1-b)=f(Ax_2-b)$, from which it follows  $\\|x_1\\|_1= \\|x_2\\|_1$. Since $x_1$ and $x_2$ are arbitrary two points in $S$, the lemma is proved.\n\\end{IEEEproof}\n\n\\begin{lemma}\\label{lem02}\nUnder Assumptions \\ref{assmp2} and \\ref{assmp3}, the following statements for problems \\eqref{eq:qpa}--\\eqref{eq:qpc} hold\n\\begin{itemize}\n\\item[1)]  $X_\\lambda, Y_\\sigma$ and $Z_\\tau$ are convex;\n\\item[2)]  In problem \\eqref{eq:qpa}, $Ax-b$ and $\\|x\\|_1$ are constant for all $x\\in X_\\lambda$;\n\\item[3)]  Part 2) holds for problem \\eqref{eq:qpb} and $Y_\\sigma$;\n\\item[4)]  Part 2) holds for problem \\eqref{eq:qpc} and $Z_\\tau$.\n\\end{itemize}\n\\end{lemma}\n\n\\begin{IEEEproof}\nAssumption \\ref{assmp2} makes sure that $X_\\lambda,Y_\\sigma,Z_\\tau$ are all non-empty.\n\n1) As a well-known result, the  set of solutions of a convex program is convex.\n\n2) Since $f_1(Ax-b)+\\lambda \\|x\\|_1$ is constant over $x\\in X_\\lambda$ and $f_1$ is strictly convex by Assumption \\ref{assmp3}, the result follows directly from Lemma \\ref{lem01}.\n\n3) If $\\vzero\\in Y_\\sigma$, then the optimal objective is $\\|\\vzero\\|_1= 0$; hence, $Y_\\lambda=\\{\\vzero\\}$ and the results hold trivially. Suppose $\\vzero\\not\\in Y_\\sigma$. Since the optimal objective $\\|x\\|_1$ is constant for all $x\\in Y_\\sigma$ and $f_2$ is strictly convex by Assumption \\ref{assmp3}, to prove this part in light of Lemma \\ref{lem01}, we shall show $f_2(Ax-b)=\\sigma$ for all $x\\in Y_\\sigma$.\n\nAssume that there is $\\hat{x}\\in Y_\\sigma$ such that $f_2(A\\hat{x}-b)<\\sigma$. Since $f_2(Ax-b)$ is convex and thus continuous in $x$, there exists a non-empty ball $\\cB$ centered at $\\hat{x}$ with a sufficiently small radius $\\rho>0$ so that $f_2(A\\bar{x}-b)<\\sigma$ for all $\\bar{x}\\in\\cB$.  Let $\\alpha = \\min\\{\\frac{\\rho}{2\\cdot \\|\\hat{x}\\|_2},\\frac{1}{2}\\}\\in (0,1)$. We have $(1-\\alpha)\\hat{x}\\in\\cB$ and  $\\|(1-\\alpha)\\hat{x}\\|_1= (1-\\alpha)\\|\\hat{x}\\|_1<\\|\\hat{x}\\|_1$, so $(1-\\alpha)\\hat{x}$ is both feasible and achieving an objective value lower than the optimal one. Contradiction.\n\n4) By Assumption \\ref{assmp3}, we have $\\|x\\|_1=\\tau$ for all $x\\in Z_\\tau$; otherwise, there exists $\\bar{x}\\in Z_\\tau$ such that  $\\tau>f_3(A\\bar{x}-b)\\ge \\inf \\{\\|x\\|_1: f_3(Ax-b) = f_3^*\\}$, contradicting Assumption \\ref{assmp3}. Since the optimal objective $f_3(Ax-b)$ is constant for all $x\\in Z_\\tau$ and $f_3$ is strictly convex by Assumption \\ref{assmp3},  the result follows from Lemma \\ref{lem01}.\n\\end{IEEEproof}\n\n\\begin{IEEEproof}[Proof of Theorem \\ref{thm:unique} for problems \\eqref{eq:qpa}--\\eqref{eq:qpc}]\nThis proof exploits Lemma \\ref{lem02}. Since the results of Lemma \\ref{lem02} are are identical for problems \\eqref{eq:qpa}--\\eqref{eq:qpc}, we  present the proof  for problem \\eqref{eq:qpa}. The proofs for the other two problems are similar.\n\nFrom Assumption \\ref{assmp3}, $X_\\lambda$ is nonempty so we pick $x^*\\in X_\\lambda$. Let $b^* = Ax^*$, which is independent of the choice of $x^*$ according to Lemma \\ref{lem02}. We introduce the linear program\n\\beq\\label{eq:dd}\n\\min \\|x\\|_1,\\quad\\text{s.t.}~Ax= b^*,\n\\eeq\nand let $X^*$ denote its solution set.\n\nNow, we show that $X_\\lambda = X^*$. Since $Ax = Ax^*$ and $\\|x\\|_1 = \\|x^*\\|_1$ for all $x\\in X_\\lambda$ and conversely any $x$ obeying $Ax=Ax^*$ and $\\|x\\|_1=\\|x^*\\|_1$  belongs to $X_\\lambda$, it is suffices to show that $\\|{x}\\|_1 = \\|x^*\\|_1$ for any ${x}\\in X^*$. Assuming this does \\emph{not} hold, then since problem \\eqref{eq:dd}  has  $x^*$ as a feasible solution and has a finite objective,  we have a nonempty $X^*$ and there exists $\\bar{x}\\in X^*$ satisfying $\\|\\bar{x}\\|_1 < \\|x^*\\|_1$. But,  $f(A\\bar{x}-b) = f(b^*-b)=f(Ax^*-b)$ and $\\|\\bar{x}\\|_1 < \\|x^*\\|_1$ mean that $\\bar{x}$ is a strictly better solution to problem \\eqref{eq:qpa} than $x^*$, contradicting the assumption $x^*\\in X_\\lambda$.\n\nSince $X_\\lambda = X^*$,  $x^*$ is the unique solution to problem \\eqref{eq:qpa} if and only if it is the same to problem \\eqref{eq:dd}. Since problem \\eqref{eq:dd} is in the same form of problem \\eqref{eq:bp}, applying the part of Theorem \\ref{thm:unique} for problem \\eqref{eq:bp}, which is already proved, we conclude that  $x^*$ is the unique solution to problem \\eqref{eq:qpa} if and only if Condition \\ref{cond:unique} holds.\n\\end{IEEEproof}","units":[{"raw_tex":"\\begin{theorem}[Basis pursuit  unique optimality]\\label{thm:bp} Under Assumptions \\ref{assmp1}--\\ref{assmp2},\n$x^*\\in\\mathbb{R}^n$ is the unique solution to problem \\eqref{eq:bp} if and only if $Ax^*=b$ and Condition \\ref{cond:unique} is satisfied.\n\\end{theorem}","start":15694,"end":15956},{"raw_tex":"\\begin{IEEEproof}[Proof of Theorem \\ref{thm:bp}] We frequently use the notions  $I=\\supp(x^*)$ and $s=\\sign(x_I^*)$ below.\n\\medskip\n\n``$\\Longleftarrow$''. This part has been shown in \\cite{Fu, Tr}. For completeness, we give a  proof. Let $y$ satisfy Condition \\ref{cond:unique}, part 2, and let $x\\in\\RR^n$ be an arbitrary vector satisfying $Ax = b$ and $x\\not=x^*$. We shall show $\\|x^*\\|_1<\\|x\\|_1$.\n\nSince $A_I$ has full column rank and $x\\not=x^*$, we have $\\supp(x)\\not=I$; otherwise from $A_I x^*_I =b= A_I x_I$, we would get $x_I^* = x_I$ and  thus the contradiction $x^*=x$.\n\nFrom $\\supp(x)\\not=I$, we  get $b^T y < \\|x\\|_1$. To see this, let $J := \\supp(x) \\setminus I$, which is a non-empty subset of $I^c$. From Condition \\ref{cond:unique}, we have $\\|A_I^T y\\|_\\infty =1$ and $\\|A_J^T y\\|_\\infty < 1$, and thus\n\\begin{align*}\n\\langle x_I, A_I^T y\\rangle&\\le \\|x_I\\|_1\\cdot\\|A_I^T y\\|_\\infty\\le\\|x_I\\|_1,\\\\\n\\langle x_J, A_J^T y\\rangle&\\le \\|x_J\\|_1\\cdot\\|A_J^T y\\|_\\infty<\\|x_J\\|_1,\n\\end{align*}\n(the last inequality is ``$<$'' not ``$\\le$'') which lead to\n\\begin{eqnarray*}\nb^T y = \\langle x, A^T y\\rangle& = &\\langle x_I, A_I^T y\\rangle+ \\langle x_J, A_J^T y\\rangle\\\\\n &<& \\|x_I\\|_1  +\\|x_J\\|_1 = \\|x\\|_1.\n\\end{eqnarray*}\n\nOn the other hand, we have\n$$\n\\|x^*\\|_1 = \\langle x^*_I, \\sign(x^*_I)\\rangle = \\langle x^*_I, A_I^T y\\rangle = \\langle A_I x^*_I,y\\rangle = b^T y\n$$\nand thus $\\|x^*\\|_1=b^T y<\\|x\\|_1$.%, from which it follows $x^*$ is the unique solution to \\eqref{eq:bp}.\n\\medskip\n\n\\noindent``$\\Longrightarrow$''. Assume that $x^*$ is the unique solution to \\eqref{eq:bp}. Obviously, $Ax^*=b$.\n\nIt is easy to obtain Condition \\ref{cond:unique}, part 1. Suppose it does not hold. Then, $A_I$ has a nontrivial null space, and perturbing $x_I^*$ along the null space will change the objective $\\|x_I^*\\|_1 = s^T x^*_I$ while maintaining $A_I x_I^*=b$; hence, this  perturbing breaks the unique optimality of $x^*$.\n(In more details, there exists a \\emph{nonzero} vector $d\\in\\RR^{n}$ such that $A_I d_I = \\vzero$ and $d_{I^c}=\\vzero$. For any scalar $\\alpha$ near zero, we have $\\sign(x^*_I+\\alpha d_I)=\\sign(x_I^*)=s$ and thus $\\|x^*+\\alpha d\\|_1= s^T (x^*_I+\\alpha d_I)=s^T x^*_I+\\alpha( s^T d_I)=\\|x^*\\|_1+\\alpha( s^T d_I)$. Since $x^*$ is the unique solution, we must have $\\|x^*_I+\\alpha d\\|_1> \\|x^*\\|_1$ or, equivalently, $\\alpha( s^T d_I)>0$ whenever $\\alpha\\not=0$. This is impossible as we can perturb $\\alpha$ around 0 both ways.)\n\nIt remains to construct a vector $y$ for Condition \\ref{cond:unique}, part 2.\nOur construction is based on  the strong convexity relation between a linear program (called the primal problem) and its dual problem, namely, if one problem has a solution, so does the other, and the two solutions must give the same objective value. (For the interested reader, this result follows from the Hahn-Banach separation theorem, also from the theorem of alternatives \\cite{Da}. Alternatively, it can be  obtain constructively via the Simplex method; specifically, whenever a primal solution exists, the Simplex method terminates in a finite number of steps with a primal-dual solution pair.)\n\nThe strong duality relation holds between  \\eqref{eq:bp} and its dual problem\n\\beq\\label{eq:bpdual}\n\\max_{p\\in\\RR^m} b^Tp\\quad\\text{s.t.}~ \\|A^Tp\\|_\\infty \\le 1\n\\eeq\nbecause  \\eqref{eq:bp} and \\eqref{eq:bpdual}, as a primal-dual pair, are equivalent to the primal-dual linear programs\n\\begin{subequations}\\label{eq:pdpair}\n\\begin{align}\n\\label{eq:pdpaira}\n\\min_{u,v\\in\\RR^n} \\vone^Tu + \\vone^T v & \\quad \\text{s.t.}~A u - Av = b, ~u\\ge 0, ~v\\ge 0,\\\\\n\\label{eq:pdpairb}\n\\max_{q\\in \\RR^m} b^T q& \\quad \\text{s.t.} -\\vone\\le A^T q\\le \\vone,\n\\end{align}\n\\end{subequations}\nrespectively, where the strong duality relation holds between \\eqref{eq:pdpaira} and \\eqref{eq:pdpairb}. By ``equivalent'', we mean that one can obtain solutions from each other by the rules:\n$$\\begin{array}{ll}\n\\text{given}~u^*,v^*, & \\text{obtain}~ x^* = u^* - v^*\\\\\n\\text{given}~x^*, & \\text{obtain}~ u^*= \\max(x^*,\\vzero), ~v^*= \\max(-x^*,\\vzero),\\\\\n\\text{given}~q^*, & \\text{obtain}~ p^*= q^*,\\\\\n\\text{given}~p^*, & \\text{obtain}~ q^*= p^*.\n\\end{array}$$\nTherefore, since \\eqref{eq:bp} has solution $x^*$, there exists a solution $y^*$ to \\eqref{eq:bpdual}, which satisfies $\\|x^*\\|_1 = b^T y^*$ and $\\|A^T y^*\\|_\\infty\\le 1$. (One can obtain such $y^*$ from the  Hahn-Banach separation theorem or the theorem of alternatives rather directly.) However,  $y^*$ may \\emph{not} obey $\\|A_{I^c}^Ty^*\\|_\\infty <1$. We shall perturb $y^*$ so that $\\|A_{I^c}^Ty^*\\|_\\infty <1$.\n\nTo prepare for the perturbation, we let $L:=\\{i\\in I^c:a_i^Ty^* = -1 \\}$ and $U:=\\{i\\in I^c:a_i^Ty^* = 1\\}$. Our goal is to perturb $y^*$ so that  $-1<a_i^Ty^*<1$ for $i\\in L\\cup U$ and $y^*$ remains optimal to  \\eqref{eq:bpdual}.\nTo this end, consider for a fixed $\\alpha>0$ and $t:=\\|x^*\\|_1$, the linear program\n\\beq\\label{eq:iprimal}\n\\min_{x\\in\\RR^n}  \\sum_{i\\in L}\\alpha x_i-\\sum_{i\\in U}\\alpha x_i,\\quad\\text{s.t.}~Ax=b,~\\|x\\|_1\\le t.\n\\eeq\nSince $x^*$ is the unique solution to \\eqref{eq:bp}, it is  the unique feasible solution to problem \\eqref{eq:iprimal}, so problem \\eqref{eq:iprimal} has the optimal objective value $\\sum_{i\\in U}\\alpha x_i^* - \\sum_{i\\in L}\\alpha x_i^*=0$.\nBy setting up equivalent linear programs like what has been done for \\eqref{eq:bp} and \\eqref{eq:bpdual} above,  the strong duality relation holds between \\eqref{eq:iprimal} and its dual problem\n\\beq\\label{eq:idual}\n\\max_{p\\in\\RR^m,q\\in \\RR} b^T p - tq,\\quad\\text{s.t.}~\\|A^Tp - \\alpha r\\|_\\infty \\le q,~q\\ge 0,\n\\eeq\nwhere  $r\\in\\RR^n$ is given by\n$$r_i = \\begin{cases}1,&i\\in L,\\\\\n-1,&i\\in U,\\\\\n0,&\\text{otherwise}.\n\\end{cases}$$\nTherefore, \\eqref{eq:idual} has a solution $(p^*,q^*)$ satisfying $b^T p^*- tq^*= 0$.\n\nAccording to the last constraint of \\eqref{eq:idual}, we have $q^*\\ge 0$, which we split into two cases:  $q^*=0$ and  $q^*>0$.\n\\begin{itemize}\n\\item[i)] If $q^*=0$, we have $A^T p^*=\\alpha r$ and $b^T p^*=0$.\n\\item[ii)] If $q^*>0$, we let $r^*:=p^*/q^*$, which satisfies $b^T r^* =t=  \\|x^*\\|_1$ and $\\|A^Tr^* - \\frac{\\alpha}{q^*}r\\|_\\infty \\le 1$, or equivalently, $-\\vone+\\frac{\\alpha}{q^*}r\\le A^Tr^*  \\le \\vone+\\frac{\\alpha}{q^*}r$.\n\\end{itemize}\nNow we perturb $y^*$. Solve \\eqref{eq:idual}  with a sufficiently small $\\alpha > 0$ and obtain a solution $(p^*,q^*)$. If case i) occurs, we let $y^* \\gets y^* +p^*$; otherwise, we let $y^* \\gets \\frac{1}{2}(y^* +r^*)$. In both cases,\n\\begin{itemize}\n\\item $b^T y^*$ is unchanged, still equal to $\\|x^*\\|_1$;\n\\item $-1<a_i^Ty^*<1$ holds for $i\\in L\\cup U$ after the perturbation;\n\\item for each $i \\not \\in L\\cup U$, if $a_j^Ty^*\\in[-1,1]$ or $a_j^Ty^*\\in(-1,1)$ holds before the perturbation, the same holds after the perturbation;\n\\end{itemize}\nTherefore, after the perturbation, $y^*$ satisfies:\n\\begin{itemize}\n\\item[1)] $b^T y^* = \\|x^*\\|_1$,\n\\item[2)] $\\|A_I^T y^*\\|_\\infty\\le 1$, and\n\\item[3)] $\\|A_{I^c}^T y^*\\|_\\infty <1$.\n\\end{itemize}\nFrom 1) and 2) it follows\n\\begin{itemize}\n\\item[4)] $A_I^T y = \\sign(x_I^*)$\n\n\\end{itemize}\nsince $\\|x_I^*\\|_1 = \\|x^*\\|_1 = b^T y^* = \\langle A_I x_I^*,y^*\\rangle = \\langle x_I^*, A_I^T y^*\\rangle\\le \\|x_I^*\\|_1 \\|A^T y^*\\|_\\infty \\le \\|x_I^*\\|_1$ and thus $\\langle x_I^*, A_I^T y^*\\rangle = \\|x_I^*\\|_1$, which dictates 4). From 3) and 4), Condition \\ref{cond:unique}, part 2, holds with $y=y^*$.\n\\end{IEEEproof}","start":22595,"end":30001},{"raw_tex":"\\begin{IEEEproof}[Proof of Theorem \\ref{thm:unique} for problem \\eqref{eq:bp}]\nThe above proof for Theorem \\ref{thm:bp} also serves the proof of  Theorem \\ref{thm:unique} for problem \\eqref{eq:bp} since $Ax^*=b$ is  involved only in the optimality part, not the uniqueness part.\n\\end{IEEEproof}","start":30003,"end":30297},{"raw_tex":"\\begin{lemma}\\label{lem01}\nLet $f$ be a strictly convex function. If $f(Ax-b)+\\|x\\|_1$ is constant on a convex set $S$, then both $Ax-b$ and $\\|x\\|_1$ are  constant on $S$.\n\\end{lemma}","start":30415,"end":30599},{"raw_tex":"\\begin{IEEEproof}\nIt suffices to prove the case where $S$ has more one point. Let $x_1$ and $x_2$ be any two  different points in $S$. Consider the line segment $L$ connecting $x_1$ and $x_2$. Since  $X$ is convex, we have $L\\subset X$ and that   $f(Ax-b)+\\|x\\|_1$ is constant on $L$.  On one hand, $\\|x\\|_1$ is piece-wise linear over $L$; on the other hand, the strict convexity of $f$ makes it impossible for $f(Ax-b)$ to be piece-wise linear over $L$ unless $Ax_1-b=Ax_2-b$. Hence, we have $Ax_1-b=Ax_2-b$ and thus $f(Ax_1-b)=f(Ax_2-b)$, from which it follows  $\\|x_1\\|_1= \\|x_2\\|_1$. Since $x_1$ and $x_2$ are arbitrary two points in $S$, the lemma is proved.\n\\end{IEEEproof}","start":30600,"end":31279},{"raw_tex":"\\begin{lemma}\\label{lem02}\nUnder Assumptions \\ref{assmp2} and \\ref{assmp3}, the following statements for problems \\eqref{eq:qpa}--\\eqref{eq:qpc} hold\n\\begin{itemize}\n\\item[1)]  $X_\\lambda, Y_\\sigma$ and $Z_\\tau$ are convex;\n\\item[2)]  In problem \\eqref{eq:qpa}, $Ax-b$ and $\\|x\\|_1$ are constant for all $x\\in X_\\lambda$;\n\\item[3)]  Part 2) holds for problem \\eqref{eq:qpb} and $Y_\\sigma$;\n\\item[4)]  Part 2) holds for problem \\eqref{eq:qpc} and $Z_\\tau$.\n\\end{itemize}\n\\end{lemma}","start":31315,"end":31796},{"raw_tex":"\\begin{IEEEproof}\nAssumption \\ref{assmp2} makes sure that $X_\\lambda,Y_\\sigma,Z_\\tau$ are all non-empty.\n\n1) As a well-known result, the  set of solutions of a convex program is convex.\n\n2) Since $f_1(Ax-b)+\\lambda \\|x\\|_1$ is constant over $x\\in X_\\lambda$ and $f_1$ is strictly convex by Assumption \\ref{assmp3}, the result follows directly from Lemma \\ref{lem01}.\n\n3) If $\\vzero\\in Y_\\sigma$, then the optimal objective is $\\|\\vzero\\|_1= 0$; hence, $Y_\\lambda=\\{\\vzero\\}$ and the results hold trivially. Suppose $\\vzero\\not\\in Y_\\sigma$. Since the optimal objective $\\|x\\|_1$ is constant for all $x\\in Y_\\sigma$ and $f_2$ is strictly convex by Assumption \\ref{assmp3}, to prove this part in light of Lemma \\ref{lem01}, we shall show $f_2(Ax-b)=\\sigma$ for all $x\\in Y_\\sigma$.\n\nAssume that there is $\\hat{x}\\in Y_\\sigma$ such that $f_2(A\\hat{x}-b)<\\sigma$. Since $f_2(Ax-b)$ is convex and thus continuous in $x$, there exists a non-empty ball $\\cB$ centered at $\\hat{x}$ with a sufficiently small radius $\\rho>0$ so that $f_2(A\\bar{x}-b)<\\sigma$ for all $\\bar{x}\\in\\cB$.  Let $\\alpha = \\min\\{\\frac{\\rho}{2\\cdot \\|\\hat{x}\\|_2},\\frac{1}{2}\\}\\in (0,1)$. We have $(1-\\alpha)\\hat{x}\\in\\cB$ and  $\\|(1-\\alpha)\\hat{x}\\|_1= (1-\\alpha)\\|\\hat{x}\\|_1<\\|\\hat{x}\\|_1$, so $(1-\\alpha)\\hat{x}$ is both feasible and achieving an objective value lower than the optimal one. Contradiction.\n\n4) By Assumption \\ref{assmp3}, we have $\\|x\\|_1=\\tau$ for all $x\\in Z_\\tau$; otherwise, there exists $\\bar{x}\\in Z_\\tau$ such that  $\\tau>f_3(A\\bar{x}-b)\\ge \\inf \\{\\|x\\|_1: f_3(Ax-b) = f_3^*\\}$, contradicting Assumption \\ref{assmp3}. Since the optimal objective $f_3(Ax-b)$ is constant for all $x\\in Z_\\tau$ and $f_3$ is strictly convex by Assumption \\ref{assmp3},  the result follows from Lemma \\ref{lem01}.\n\\end{IEEEproof}","start":31797,"end":33597},{"raw_tex":"\\begin{IEEEproof}[Proof of Theorem \\ref{thm:unique} for problems \\eqref{eq:qpa}--\\eqref{eq:qpc}]\nThis proof exploits Lemma \\ref{lem02}. Since the results of Lemma \\ref{lem02} are are identical for problems \\eqref{eq:qpa}--\\eqref{eq:qpc}, we  present the proof  for problem \\eqref{eq:qpa}. The proofs for the other two problems are similar.\n\nFrom Assumption \\ref{assmp3}, $X_\\lambda$ is nonempty so we pick $x^*\\in X_\\lambda$. Let $b^* = Ax^*$, which is independent of the choice of $x^*$ according to Lemma \\ref{lem02}. We introduce the linear program\n\\beq\\label{eq:dd}\n\\min \\|x\\|_1,\\quad\\text{s.t.}~Ax= b^*,\n\\eeq\nand let $X^*$ denote its solution set.\n\nNow, we show that $X_\\lambda = X^*$. Since $Ax = Ax^*$ and $\\|x\\|_1 = \\|x^*\\|_1$ for all $x\\in X_\\lambda$ and conversely any $x$ obeying $Ax=Ax^*$ and $\\|x\\|_1=\\|x^*\\|_1$  belongs to $X_\\lambda$, it is suffices to show that $\\|{x}\\|_1 = \\|x^*\\|_1$ for any ${x}\\in X^*$. Assuming this does \\emph{not} hold, then since problem \\eqref{eq:dd}  has  $x^*$ as a feasible solution and has a finite objective,  we have a nonempty $X^*$ and there exists $\\bar{x}\\in X^*$ satisfying $\\|\\bar{x}\\|_1 < \\|x^*\\|_1$. But,  $f(A\\bar{x}-b) = f(b^*-b)=f(Ax^*-b)$ and $\\|\\bar{x}\\|_1 < \\|x^*\\|_1$ mean that $\\bar{x}$ is a strictly better solution to problem \\eqref{eq:qpa} than $x^*$, contradicting the assumption $x^*\\in X_\\lambda$.\n\nSince $X_\\lambda = X^*$,  $x^*$ is the unique solution to problem \\eqref{eq:qpa} if and only if it is the same to problem \\eqref{eq:dd}. Since problem \\eqref{eq:dd} is in the same form of problem \\eqref{eq:bp}, applying the part of Theorem \\ref{thm:unique} for problem \\eqref{eq:bp}, which is already proved, we conclude that  $x^*$ is the unique solution to problem \\eqref{eq:qpa} if and only if Condition \\ref{cond:unique} holds.\n\\end{IEEEproof}","start":33599,"end":35415}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:bp"],"declared_external":false,"source_only":false,"title":"基追踪模型","source_units":[{"raw_tex":"Let $x\\in\\RR^n$ be the decision variable. This paper studies the unique solutions of the  $\\ell_1$ minimization problems  including the basis pursuit problem  \\cite{CDS}\n\\begin{subequations}\n\\label{eq:qp}\n\\beq\\label{eq:bp}\n\\min\\|x\\|_1\\quad\\text{s.t.}~Ax=b\n\\eeq\nand  convex problems\n","start":9278,"end":9560}]},{"id":"S002","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:qpa","eq:qpb","eq:qpc"],"declared_external":false,"source_only":false,"title":"惩罚型模型","source_units":[{"raw_tex":"\\begin{align}\n\\label{eq:qpa}\n&\\min f_1(Ax-b)+\\lambda \\|x\\|_1,\\\\\n","start":9560,"end":9624}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:qpa","eq:qpb","eq:qpc"],"declared_external":false,"source_only":false,"title":"损失约束模型","source_units":[{"raw_tex":"\\label{eq:qpb}\n&\\min \\|x\\|_1,\\quad\\text{s.t.}~f_2(Ax-b)\\leq \\sigma, \\\\\n","start":9624,"end":9695}]},{"id":"S004","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"范数约束模型","source_units":[{"raw_tex":"\\label{eq:qpc}\n&\\min f_3(Ax-b), \\quad\\text{s.t.}~  \\|x\\|_1\\leq \\tau,\n\\end{align}\n\\end{subequations}\n","start":9695,"end":9795}]},{"id":"S005","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"参数与损失范围","source_units":[{"raw_tex":"where $\\lambda,\\sigma,\\tau>0$ are scalar parameters,  $A$ is a matrix, and $f_i(x), i=1, 2, 3$ are strictly convex functions. The Lasso problem \\cite{Ti} is a special case of  problem \\eqref{eq:qpa} or \\eqref{eq:qpc} while the basis pursuit denoising problem \\cite{CDS} is a special case of  problem \\eqref{eq:qpb} all with $f_i(\\cdot)=\\frac{1}{2}\\|\\cdot\\|_2^2,~ i=1, 2, 3$.\n","start":9795,"end":10170}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"解集与子矩阵记号","source_units":[{"raw_tex":"We let $X$, $X_\\lambda$, $Y_\\sigma$, and $Z_\\tau$ denote the sets of solutions to  problems \\eqref{eq:bp}--\\eqref{eq:qpc}, respectively. We let $a_i$ be the $i$th column of $A$ and $x_i$ be the $i$th entry of $x$. Given an index set $I$, we frequently use $A_I$ as the submatrix of $A$ formed by its columns $a_i$,  $i\\in I$ and  $x_I$ as the subvector of $x$ formed by entries $x_i$, $i\\in I$.\n\n","start":12330,"end":12726}]},{"id":"S007","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"假设引入","source_units":[{"raw_tex":"Our analysis makes the following assumptions:\n\n","start":12726,"end":12773}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","labels":["assmp1"],"declared_external":false,"source_only":false,"title":"矩阵满行秩","source_units":[{"raw_tex":"\\begin{assumption}\\label{assmp1}\nMatrix $A$ has full row rank.\n\\end{assumption}\n","start":12773,"end":12853}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","labels":["assmp2"],"declared_external":false,"source_only":false,"title":"各模型存在解","source_units":[{"raw_tex":"\\begin{assumption}\\label{assmp2}\nThe solution sets $X$, $X_\\lambda$, $Y_\\sigma$, and $Z_\\tau$ of problems \\eqref{eq:bp}--\\eqref{eq:qpc}, respectively, are nonempty.\n\\end{assumption}\n","start":12853,"end":13035}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","labels":["assmp3"],"declared_external":false,"source_only":false,"title":"损失严格凸","source_units":[{"raw_tex":"\\begin{assumption}\\label{assmp3} In problems \\eqref{eq:qpa}--\\eqref{eq:qpc}, functions $f_1,f_2,f_3$ are strictly convex. In addition,  the constraint of problem \\eqref{eq:qpc} is bounding, namely, $\\tau \\le \\inf \\{\\|x\\|_1: f_3(Ax-b) = f_3^*\\}$, where $f_3^*:=\\min_{y\\in\\RR^n} f(Ay- b)$.\n\\end{assumption}\n","start":13035,"end":13340}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","labels":["assmp3"],"declared_external":false,"source_only":false,"title":"范数约束的界限","source_units":[{"raw_tex":"\\begin{assumption}\\label{assmp3} In problems \\eqref{eq:qpa}--\\eqref{eq:qpc}, functions $f_1,f_2,f_3$ are strictly convex. In addition,  the constraint of problem \\eqref{eq:qpc} is bounding, namely, $\\tau \\le \\inf \\{\\|x\\|_1: f_3(Ax-b) = f_3^*\\}$, where $f_3^*:=\\min_{y\\in\\RR^n} f(Ay- b)$.\n\\end{assumption}\n","start":13035,"end":13340}]},{"id":"S012","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"假设的化简与动机","source_units":[{"raw_tex":"Assumptions \\ref{assmp1} and \\ref{assmp2} are standard. If Assumption \\ref{assmp1} does not hold and $Ax=b$ is consistent, the problems can be simplified; specifically, one can decompose $A=\\begin{bmatrix}A_1\\\\ A_2\\end{bmatrix}$ and $b=\\begin{bmatrix}b_1\\\\ b_2\\end{bmatrix}$ so that and  $A_1$ has full row rank equal to $\\mathrm{rank}(A)$, and one can replace the constraints $Ax = b$ by $A_1x = b_1$ and introduce functions $\\bar{f}_i$ so that $\\bar{f}_i(A_1x - b_1) \\equiv f_i(Ax-b)$, $i=1,2,3$. Assumption \\ref{assmp2} guarantees that the solutions of problems \\eqref{eq:bp}--\\eqref{eq:qpc} can be attained so the discussion of solution uniqueness makes sense.  The strict convexity of $f_1,f_2,f_3$ and the restriction on $\\tau$ in Assumption \\ref{assmp3} are  also fairly basic toward solution uniqueness. Strict convexity rules out piece-wise linearity. (Note that $f_1,f_2,f_3$ can still be non-differentiable.) If the restriction on $\\tau$ is removed, the solution uniqueness of  problem \\eqref{eq:qpc} becomes solely  up to $f_3(Ax-b)$, independent of $\\|x\\|_1$. % and Figure \\ref{?} illustrates this point by an example.\n\n","start":13340,"end":14473}]},{"id":"S013","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"条件的既有充分性","source_units":[{"raw_tex":"For a given vector $x^*$, the following conditions on matrix $A$ is the key to solution uniqueness, and its sufficiency has been established in \\cite{Fu}.\n\n","start":14473,"end":14629}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":["cond:unique"],"declared_external":false,"source_only":false,"title":"支撑与符号","source_units":[{"raw_tex":"\\begin{condition}\\label{cond:unique}  Under the definitions  $I:=\\supp(x^*)\\subseteq\\{1,\\ldots,n\\}$ and $s:=\\sign(x_I^*)$, matrix $A\\in\\mathbb{R}^{m\\times n}$ has the following properties:\n\\begin{enumerate}\n","start":14629,"end":14836}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":["cond:unique"],"declared_external":false,"source_only":false,"title":"唯一性条件的秩部分","source_units":[{"raw_tex":"\\item submatrix $A_{I}$ has full column rank, and\n","start":14836,"end":14886}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"唯一性条件的严格证书","source_units":[{"raw_tex":"\\item there is  $y \\in \\mathbb{R}^m$ obeying  $A_{I}^Ty=s$ and $\\|A_{I^c}^Ty\\|_\\infty <1$.\n\\end{enumerate}\n\\end{condition}\n","start":14886,"end":15009}]},{"id":"S017","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"三阶段证明路线","source_units":[{"raw_tex":"We establish Theorem \\ref{thm:unique} in three steps. Our first step proves the theorem for problem \\eqref{eq:bp} only. Since the only difference between this part and Theorem \\ref{thm:bp} is the conditions $Ax^*=b$, we  prove Theorem \\ref{thm:bp} first. In the second step, for problems \\eqref{eq:qpa}--\\eqref{eq:qpc}, we show that both $\\|x\\|_1$ and $Ax-b$ are constant for $x$ over the solution sets $X_\\lambda, Y_\\sigma,Z_\\tau$, respectively. Finally,  we prove Theorem \\ref{thm:unique}  for problems \\eqref{eq:qpa}--\\eqref{eq:qpc}.\n","start":22058,"end":22595}]},{"id":"S018","semantic_type":"theorem","kind":"theorem","labels":["thm:unique"],"declared_external":false,"source_only":false,"title":"四类模型的唯一性刻画","source_units":[{"raw_tex":"\\begin{theorem}[Solution uniqueness]\\label{thm:unique} Under Assumptions \\ref{assmp1}--\\ref{assmp3}, given that $x^*$ is a solution  to problem \\eqref{eq:bp}, \\eqref{eq:qpa}, \\eqref{eq:qpb}, or \\eqref{eq:qpc}, $x^*$ is  the unique solution if and only if Condition \\ref{cond:unique} holds.\n\\end{theorem}","start":15151,"end":15454}]},{"id":"S019","semantic_type":"theorem","kind":"theorem","labels":["thm:bp"],"declared_external":false,"source_only":false,"title":"基追踪的唯一最优性","source_units":[{"raw_tex":"\\begin{theorem}[Basis pursuit  unique optimality]\\label{thm:bp} Under Assumptions \\ref{assmp1}--\\ref{assmp2},\n$x^*\\in\\mathbb{R}^n$ is the unique solution to problem \\eqref{eq:bp} if and only if $Ax^*=b$ and Condition \\ref{cond:unique} is satisfied.\n\\end{theorem}","start":15694,"end":15956}]},{"id":"S020","semantic_type":"proof","kind":"proof","labels":["eq:bpdual","eq:idual","eq:iprimal","eq:pdpair","eq:pdpaira","eq:pdpairb"],"declared_external":false,"source_only":false,"title":"基追踪定理的完整证明","source_units":[{"raw_tex":"\\begin{IEEEproof}[Proof of Theorem \\ref{thm:bp}] We frequently use the notions  $I=\\supp(x^*)$ and $s=\\sign(x_I^*)$ below.\n\\medskip\n\n``$\\Longleftarrow$''. This part has been shown in \\cite{Fu, Tr}. For completeness, we give a  proof. Let $y$ satisfy Condition \\ref{cond:unique}, part 2, and let $x\\in\\RR^n$ be an arbitrary vector satisfying $Ax = b$ and $x\\not=x^*$. We shall show $\\|x^*\\|_1<\\|x\\|_1$.\n\nSince $A_I$ has full column rank and $x\\not=x^*$, we have $\\supp(x)\\not=I$; otherwise from $A_I x^*_I =b= A_I x_I$, we would get $x_I^* = x_I$ and  thus the contradiction $x^*=x$.\n\nFrom $\\supp(x)\\not=I$, we  get $b^T y < \\|x\\|_1$. To see this, let $J := \\supp(x) \\setminus I$, which is a non-empty subset of $I^c$. From Condition \\ref{cond:unique}, we have $\\|A_I^T y\\|_\\infty =1$ and $\\|A_J^T y\\|_\\infty < 1$, and thus\n\\begin{align*}\n\\langle x_I, A_I^T y\\rangle&\\le \\|x_I\\|_1\\cdot\\|A_I^T y\\|_\\infty\\le\\|x_I\\|_1,\\\\\n\\langle x_J, A_J^T y\\rangle&\\le \\|x_J\\|_1\\cdot\\|A_J^T y\\|_\\infty<\\|x_J\\|_1,\n\\end{align*}\n(the last inequality is ``$<$'' not ``$\\le$'') which lead to\n\\begin{eqnarray*}\nb^T y = \\langle x, A^T y\\rangle& = &\\langle x_I, A_I^T y\\rangle+ \\langle x_J, A_J^T y\\rangle\\\\\n &<& \\|x_I\\|_1  +\\|x_J\\|_1 = \\|x\\|_1.\n\\end{eqnarray*}\n\nOn the other hand, we have\n$$\n\\|x^*\\|_1 = \\langle x^*_I, \\sign(x^*_I)\\rangle = \\langle x^*_I, A_I^T y\\rangle = \\langle A_I x^*_I,y\\rangle = b^T y\n$$\nand thus $\\|x^*\\|_1=b^T y<\\|x\\|_1$.%, from which it follows $x^*$ is the unique solution to \\eqref{eq:bp}.\n\\medskip\n\n\\noindent``$\\Longrightarrow$''. Assume that $x^*$ is the unique solution to \\eqref{eq:bp}. Obviously, $Ax^*=b$.\n\nIt is easy to obtain Condition \\ref{cond:unique}, part 1. Suppose it does not hold. Then, $A_I$ has a nontrivial null space, and perturbing $x_I^*$ along the null space will change the objective $\\|x_I^*\\|_1 = s^T x^*_I$ while maintaining $A_I x_I^*=b$; hence, this  perturbing breaks the unique optimality of $x^*$.\n(In more details, there exists a \\emph{nonzero} vector $d\\in\\RR^{n}$ such that $A_I d_I = \\vzero$ and $d_{I^c}=\\vzero$. For any scalar $\\alpha$ near zero, we have $\\sign(x^*_I+\\alpha d_I)=\\sign(x_I^*)=s$ and thus $\\|x^*+\\alpha d\\|_1= s^T (x^*_I+\\alpha d_I)=s^T x^*_I+\\alpha( s^T d_I)=\\|x^*\\|_1+\\alpha( s^T d_I)$. Since $x^*$ is the unique solution, we must have $\\|x^*_I+\\alpha d\\|_1> \\|x^*\\|_1$ or, equivalently, $\\alpha( s^T d_I)>0$ whenever $\\alpha\\not=0$. This is impossible as we can perturb $\\alpha$ around 0 both ways.)\n\nIt remains to construct a vector $y$ for Condition \\ref{cond:unique}, part 2.\nOur construction is based on  the strong convexity relation between a linear program (called the primal problem) and its dual problem, namely, if one problem has a solution, so does the other, and the two solutions must give the same objective value. (For the interested reader, this result follows from the Hahn-Banach separation theorem, also from the theorem of alternatives \\cite{Da}. Alternatively, it can be  obtain constructively via the Simplex method; specifically, whenever a primal solution exists, the Simplex method terminates in a finite number of steps with a primal-dual solution pair.)\n\nThe strong duality relation holds between  \\eqref{eq:bp} and its dual problem\n\\beq\\label{eq:bpdual}\n\\max_{p\\in\\RR^m} b^Tp\\quad\\text{s.t.}~ \\|A^Tp\\|_\\infty \\le 1\n\\eeq\nbecause  \\eqref{eq:bp} and \\eqref{eq:bpdual}, as a primal-dual pair, are equivalent to the primal-dual linear programs\n\\begin{subequations}\\label{eq:pdpair}\n\\begin{align}\n\\label{eq:pdpaira}\n\\min_{u,v\\in\\RR^n} \\vone^Tu + \\vone^T v & \\quad \\text{s.t.}~A u - Av = b, ~u\\ge 0, ~v\\ge 0,\\\\\n\\label{eq:pdpairb}\n\\max_{q\\in \\RR^m} b^T q& \\quad \\text{s.t.} -\\vone\\le A^T q\\le \\vone,\n\\end{align}\n\\end{subequations}\nrespectively, where the strong duality relation holds between \\eqref{eq:pdpaira} and \\eqref{eq:pdpairb}. By ``equivalent'', we mean that one can obtain solutions from each other by the rules:\n$$\\begin{array}{ll}\n\\text{given}~u^*,v^*, & \\text{obtain}~ x^* = u^* - v^*\\\\\n\\text{given}~x^*, & \\text{obtain}~ u^*= \\max(x^*,\\vzero), ~v^*= \\max(-x^*,\\vzero),\\\\\n\\text{given}~q^*, & \\text{obtain}~ p^*= q^*,\\\\\n\\text{given}~p^*, & \\text{obtain}~ q^*= p^*.\n\\end{array}$$\nTherefore, since \\eqref{eq:bp} has solution $x^*$, there exists a solution $y^*$ to \\eqref{eq:bpdual}, which satisfies $\\|x^*\\|_1 = b^T y^*$ and $\\|A^T y^*\\|_\\infty\\le 1$. (One can obtain such $y^*$ from the  Hahn-Banach separation theorem or the theorem of alternatives rather directly.) However,  $y^*$ may \\emph{not} obey $\\|A_{I^c}^Ty^*\\|_\\infty <1$. We shall perturb $y^*$ so that $\\|A_{I^c}^Ty^*\\|_\\infty <1$.\n\nTo prepare for the perturbation, we let $L:=\\{i\\in I^c:a_i^Ty^* = -1 \\}$ and $U:=\\{i\\in I^c:a_i^Ty^* = 1\\}$. Our goal is to perturb $y^*$ so that  $-1<a_i^Ty^*<1$ for $i\\in L\\cup U$ and $y^*$ remains optimal to  \\eqref{eq:bpdual}.\nTo this end, consider for a fixed $\\alpha>0$ and $t:=\\|x^*\\|_1$, the linear program\n\\beq\\label{eq:iprimal}\n\\min_{x\\in\\RR^n}  \\sum_{i\\in L}\\alpha x_i-\\sum_{i\\in U}\\alpha x_i,\\quad\\text{s.t.}~Ax=b,~\\|x\\|_1\\le t.\n\\eeq\nSince $x^*$ is the unique solution to \\eqref{eq:bp}, it is  the unique feasible solution to problem \\eqref{eq:iprimal}, so problem \\eqref{eq:iprimal} has the optimal objective value $\\sum_{i\\in U}\\alpha x_i^* - \\sum_{i\\in L}\\alpha x_i^*=0$.\nBy setting up equivalent linear programs like what has been done for \\eqref{eq:bp} and \\eqref{eq:bpdual} above,  the strong duality relation holds between \\eqref{eq:iprimal} and its dual problem\n\\beq\\label{eq:idual}\n\\max_{p\\in\\RR^m,q\\in \\RR} b^T p - tq,\\quad\\text{s.t.}~\\|A^Tp - \\alpha r\\|_\\infty \\le q,~q\\ge 0,\n\\eeq\nwhere  $r\\in\\RR^n$ is given by\n$$r_i = \\begin{cases}1,&i\\in L,\\\\\n-1,&i\\in U,\\\\\n0,&\\text{otherwise}.\n\\end{cases}$$\nTherefore, \\eqref{eq:idual} has a solution $(p^*,q^*)$ satisfying $b^T p^*- tq^*= 0$.\n\nAccording to the last constraint of \\eqref{eq:idual}, we have $q^*\\ge 0$, which we split into two cases:  $q^*=0$ and  $q^*>0$.\n\\begin{itemize}\n\\item[i)] If $q^*=0$, we have $A^T p^*=\\alpha r$ and $b^T p^*=0$.\n\\item[ii)] If $q^*>0$, we let $r^*:=p^*/q^*$, which satisfies $b^T r^* =t=  \\|x^*\\|_1$ and $\\|A^Tr^* - \\frac{\\alpha}{q^*}r\\|_\\infty \\le 1$, or equivalently, $-\\vone+\\frac{\\alpha}{q^*}r\\le A^Tr^*  \\le \\vone+\\frac{\\alpha}{q^*}r$.\n\\end{itemize}\nNow we perturb $y^*$. Solve \\eqref{eq:idual}  with a sufficiently small $\\alpha > 0$ and obtain a solution $(p^*,q^*)$. If case i) occurs, we let $y^* \\gets y^* +p^*$; otherwise, we let $y^* \\gets \\frac{1}{2}(y^* +r^*)$. In both cases,\n\\begin{itemize}\n\\item $b^T y^*$ is unchanged, still equal to $\\|x^*\\|_1$;\n\\item $-1<a_i^Ty^*<1$ holds for $i\\in L\\cup U$ after the perturbation;\n\\item for each $i \\not \\in L\\cup U$, if $a_j^Ty^*\\in[-1,1]$ or $a_j^Ty^*\\in(-1,1)$ holds before the perturbation, the same holds after the perturbation;\n\\end{itemize}\nTherefore, after the perturbation, $y^*$ satisfies:\n\\begin{itemize}\n\\item[1)] $b^T y^* = \\|x^*\\|_1$,\n\\item[2)] $\\|A_I^T y^*\\|_\\infty\\le 1$, and\n\\item[3)] $\\|A_{I^c}^T y^*\\|_\\infty <1$.\n\\end{itemize}\nFrom 1) and 2) it follows\n\\begin{itemize}\n\\item[4)] $A_I^T y = \\sign(x_I^*)$\n\n\\end{itemize}\nsince $\\|x_I^*\\|_1 = \\|x^*\\|_1 = b^T y^* = \\langle A_I x_I^*,y^*\\rangle = \\langle x_I^*, A_I^T y^*\\rangle\\le \\|x_I^*\\|_1 \\|A^T y^*\\|_\\infty \\le \\|x_I^*\\|_1$ and thus $\\langle x_I^*, A_I^T y^*\\rangle = \\|x_I^*\\|_1$, which dictates 4). From 3) and 4), Condition \\ref{cond:unique}, part 2, holds with $y=y^*$.\n\\end{IEEEproof}","start":22595,"end":30001}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"declared_external":false,"source_only":false,"title":"充分性分支的比较对象","source_units":[{"raw_tex":"\\begin{IEEEproof}[Proof of Theorem \\ref{thm:bp}] We frequently use the notions  $I=\\supp(x^*)$ and $s=\\sign(x_I^*)$ below.\n\\medskip\n\n``$\\Longleftarrow$''. This part has been shown in \\cite{Fu, Tr}. For completeness, we give a  proof. Let $y$ satisfy Condition \\ref{cond:unique}, part 2, and let $x\\in\\RR^n$ be an arbitrary vector satisfying $Ax = b$ and $x\\not=x^*$. We shall show $\\|x^*\\|_1<\\|x\\|_1$.\n","start":22595,"end":22997}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"declared_external":false,"source_only":false,"title":"排除同支撑竞争解","source_units":[{"raw_tex":"Since $A_I$ has full column rank and $x\\not=x^*$, we have $\\supp(x)\\not=I$; otherwise from $A_I x^*_I =b= A_I x_I$, we would get $x_I^* = x_I$ and  thus the contradiction $x^*=x$.\n","start":22998,"end":23178}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"declared_external":false,"source_only":false,"title":"竞争解的严格对偶界","source_units":[{"raw_tex":"From $\\supp(x)\\not=I$, we  get $b^T y < \\|x\\|_1$. To see this, let $J := \\supp(x) \\setminus I$, which is a non-empty subset of $I^c$. From Condition \\ref{cond:unique}, we have $\\|A_I^T y\\|_\\infty =1$ and $\\|A_J^T y\\|_\\infty < 1$, and thus\n\\begin{align*}\n\\langle x_I, A_I^T y\\rangle&\\le \\|x_I\\|_1\\cdot\\|A_I^T y\\|_\\infty\\le\\|x_I\\|_1,\\\\\n\\langle x_J, A_J^T y\\rangle&\\le \\|x_J\\|_1\\cdot\\|A_J^T y\\|_\\infty<\\|x_J\\|_1,\n\\end{align*}\n(the last inequality is ``$<$'' not ``$\\le$'') which lead to\n\\begin{eqnarray*}\nb^T y = \\langle x, A^T y\\rangle& = &\\langle x_I, A_I^T y\\rangle+ \\langle x_J, A_J^T y\\rangle\\\\\n &<& \\|x_I\\|_1  +\\|x_J\\|_1 = \\|x\\|_1.\n\\end{eqnarray*}\n","start":23179,"end":23830}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"declared_external":false,"source_only":false,"title":"候选解达到对偶值","source_units":[{"raw_tex":"On the other hand, we have\n$$\n\\|x^*\\|_1 = \\langle x^*_I, \\sign(x^*_I)\\rangle = \\langle x^*_I, A_I^T y\\rangle = \\langle A_I x^*_I,y\\rangle = b^T y\n$$\n","start":23831,"end":23980}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"declared_external":false,"source_only":false,"title":"充分性的严格范数比较","source_units":[{"raw_tex":"and thus $\\|x^*\\|_1=b^T y<\\|x\\|_1$.%, from which it follows $x^*$ is the unique solution to \\eqref{eq:bp}.\n\\medskip\n","start":23980,"end":24096}]},{"id":"S026","semantic_type":"assumption","kind":"assumption","labels":["eq:pdpair"],"declared_external":false,"source_only":false,"title":"必要性分支的唯一解假设","source_units":[{"raw_tex":"\n\\noindent``$\\Longrightarrow$''. Assume that $x^*$ is the unique solution to \\eqref{eq:bp}. Obviously, $Ax^*=b$.\n","start":24096,"end":24209}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"declared_external":false,"source_only":false,"title":"双向零空间扰动","source_units":[{"raw_tex":"It is easy to obtain Condition \\ref{cond:unique}, part 1. Suppose it does not hold. Then, $A_I$ has a nontrivial null space, and perturbing $x_I^*$ along the null space will change the objective $\\|x_I^*\\|_1 = s^T x^*_I$ while maintaining $A_I x_I^*=b$; hence, this  perturbing breaks the unique optimality of $x^*$.\n(In more details, there exists a \\emph{nonzero} vector $d\\in\\RR^{n}$ such that $A_I d_I = \\vzero$ and $d_{I^c}=\\vzero$. For any scalar $\\alpha$ near zero, we have $\\sign(x^*_I+\\alpha d_I)=\\sign(x_I^*)=s$ and thus $\\|x^*+\\alpha d\\|_1= s^T (x^*_I+\\alpha d_I)=s^T x^*_I+\\alpha( s^T d_I)=\\|x^*\\|_1+\\alpha( s^T d_I)$. Since $x^*$ is the unique solution, we must have $\\|x^*_I+\\alpha d\\|_1> \\|x^*\\|_1$ or, equivalently, $\\alpha( s^T d_I)>0$ whenever $\\alpha\\not=0$. This is impossible as we can perturb $\\alpha$ around 0 both ways.)\n","start":24210,"end":25054}]},{"id":"S028","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:bpdual","eq:pdpair"],"declared_external":false,"source_only":false,"title":"基追踪对偶问题","source_units":[{"raw_tex":"The strong duality relation holds between  \\eqref{eq:bp} and its dual problem\n\\beq\\label{eq:bpdual}\n\\max_{p\\in\\RR^m} b^Tp\\quad\\text{s.t.}~ \\|A^Tp\\|_\\infty \\le 1\n\\eeq\n","start":25737,"end":25903}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair","eq:pdpaira","eq:pdpairb"],"declared_external":false,"source_only":false,"title":"正负分解的线性规划对","source_units":[{"raw_tex":"because  \\eqref{eq:bp} and \\eqref{eq:bpdual}, as a primal-dual pair, are equivalent to the primal-dual linear programs\n\\begin{subequations}\\label{eq:pdpair}\n\\begin{align}\n\\label{eq:pdpaira}\n\\min_{u,v\\in\\RR^n} \\vone^Tu + \\vone^T v & \\quad \\text{s.t.}~A u - Av = b, ~u\\ge 0, ~v\\ge 0,\\\\\n\\label{eq:pdpairb}\n\\max_{q\\in \\RR^m} b^T q& \\quad \\text{s.t.} -\\vone\\le A^T q\\le \\vone,\n\\end{align}\n\\end{subequations}\nrespectively, where the strong duality relation holds between \\eqref{eq:pdpaira} and \\eqref{eq:pdpairb}. By ``equivalent'', we mean that one can obtain solutions from each other by the rules:\n$$\\begin{array}{ll}\n\\text{given}~u^*,v^*, & \\text{obtain}~ x^* = u^* - v^*\\\\\n\\text{given}~x^*, & \\text{obtain}~ u^*= \\max(x^*,\\vzero), ~v^*= \\max(-x^*,\\vzero),\\\\\n\\text{given}~q^*, & \\text{obtain}~ p^*= q^*,\\\\\n\\text{given}~p^*, & \\text{obtain}~ q^*= p^*.\n\\end{array}$$\n","start":25903,"end":26766}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"declared_external":false,"source_only":false,"title":"取得非严格对偶证书","source_units":[{"raw_tex":"Therefore, since \\eqref{eq:bp} has solution $x^*$, there exists a solution $y^*$ to \\eqref{eq:bpdual}, which satisfies $\\|x^*\\|_1 = b^T y^*$ and $\\|A^T y^*\\|_\\infty\\le 1$. (One can obtain such $y^*$ from the  Hahn-Banach separation theorem or the theorem of alternatives rather directly.) However,  $y^*$ may \\emph{not} obey $\\|A_{I^c}^Ty^*\\|_\\infty <1$. We shall perturb $y^*$ so that $\\|A_{I^c}^Ty^*\\|_\\infty <1$.\n\n","start":26766,"end":27183}]},{"id":"S031","semantic_type":"definition","kind":"definition","labels":["eq:pdpair"],"declared_external":false,"source_only":false,"title":"支撑外饱和指标","source_units":[{"raw_tex":"To prepare for the perturbation, we let $L:=\\{i\\in I^c:a_i^Ty^* = -1 \\}$ and $U:=\\{i\\in I^c:a_i^Ty^* = 1\\}$. Our goal is to perturb $y^*$ so that  $-1<a_i^Ty^*<1$ for $i\\in L\\cup U$ and $y^*$ remains optimal to  \\eqref{eq:bpdual}.\n","start":27183,"end":27414}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":["eq:iprimal","eq:pdpair"],"declared_external":false,"source_only":false,"title":"辅助原问题的单点可行域","source_units":[{"raw_tex":"To this end, consider for a fixed $\\alpha>0$ and $t:=\\|x^*\\|_1$, the linear program\n\\beq\\label{eq:iprimal}\n\\min_{x\\in\\RR^n}  \\sum_{i\\in L}\\alpha x_i-\\sum_{i\\in U}\\alpha x_i,\\quad\\text{s.t.}~Ax=b,~\\|x\\|_1\\le t.\n\\eeq\nSince $x^*$ is the unique solution to \\eqref{eq:bp}, it is  the unique feasible solution to problem \\eqref{eq:iprimal}, so problem \\eqref{eq:iprimal} has the optimal objective value $\\sum_{i\\in U}\\alpha x_i^* - \\sum_{i\\in L}\\alpha x_i^*=0$.\n","start":27414,"end":27870}]},{"id":"S033","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:idual","eq:pdpair"],"declared_external":false,"source_only":false,"title":"辅助对偶与方向向量","source_units":[{"raw_tex":"By setting up equivalent linear programs like what has been done for \\eqref{eq:bp} and \\eqref{eq:bpdual} above,  the strong duality relation holds between \\eqref{eq:iprimal} and its dual problem\n\\beq\\label{eq:idual}\n\\max_{p\\in\\RR^m,q\\in \\RR} b^T p - tq,\\quad\\text{s.t.}~\\|A^Tp - \\alpha r\\|_\\infty \\le q,~q\\ge 0,\n\\eeq\nwhere  $r\\in\\RR^n$ is given by\n$$r_i = \\begin{cases}1,&i\\in L,\\\\\n-1,&i\\in U,\\\\\n0,&\\text{otherwise}.\n\\end{cases}$$\n","start":27870,"end":28301}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"declared_external":false,"source_only":false,"title":"辅助对偶达到零值","source_units":[{"raw_tex":"Therefore, \\eqref{eq:idual} has a solution $(p^*,q^*)$ satisfying $b^T p^*- tq^*= 0$.\n\n","start":28301,"end":28388}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"declared_external":false,"source_only":false,"title":"按辅助乘子分情形","source_units":[{"raw_tex":"According to the last constraint of \\eqref{eq:idual}, we have $q^*\\ge 0$, which we split into two cases:  $q^*=0$ and  $q^*>0$.\n\\begin{itemize}\n\\item[i)] If $q^*=0$, we have $A^T p^*=\\alpha r$ and $b^T p^*=0$.\n\\item[ii)] If $q^*>0$, we let $r^*:=p^*/q^*$, which satisfies $b^T r^* =t=  \\|x^*\\|_1$ and $\\|A^Tr^* - \\frac{\\alpha}{q^*}r\\|_\\infty \\le 1$, or equivalently, $-\\vone+\\frac{\\alpha}{q^*}r\\le A^Tr^*  \\le \\vone+\\frac{\\alpha}{q^*}r$.\n\\end{itemize}\n","start":28388,"end":28840}]},{"id":"S036","semantic_type":"algorithm","kind":"algorithm","labels":["eq:pdpair"],"declared_external":false,"source_only":false,"title":"严格化对偶解的更新","source_units":[{"raw_tex":"Now we perturb $y^*$. Solve \\eqref{eq:idual}  with a sufficiently small $\\alpha > 0$ and obtain a solution $(p^*,q^*)$. If case i) occurs, we let $y^* \\gets y^* +p^*$; otherwise, we let $y^* \\gets \\frac{1}{2}(y^* +r^*)$. In both cases,\n\\begin{itemize}\n\\item $b^T y^*$ is unchanged, still equal to $\\|x^*\\|_1$;\n\\item $-1<a_i^Ty^*<1$ holds for $i\\in L\\cup U$ after the perturbation;\n\\item for each $i \\not \\in L\\cup U$, if $a_j^Ty^*\\in[-1,1]$ or $a_j^Ty^*\\in(-1,1)$ holds before the perturbation, the same holds after the perturbation;\n\\end{itemize}\n","start":28840,"end":29388}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"declared_external":false,"source_only":false,"title":"更新后的证书界","source_units":[{"raw_tex":"Therefore, after the perturbation, $y^*$ satisfies:\n\\begin{itemize}\n\\item[1)] $b^T y^* = \\|x^*\\|_1$,\n\\item[2)] $\\|A_I^T y^*\\|_\\infty\\le 1$, and\n\\item[3)] $\\|A_{I^c}^T y^*\\|_\\infty <1$.\n\\end{itemize}\n","start":29388,"end":29587}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"declared_external":false,"source_only":false,"title":"由等号恢复支撑符号","source_units":[{"raw_tex":"From 1) and 2) it follows\n\\begin{itemize}\n\\item[4)] $A_I^T y = \\sign(x_I^*)$\n\n\\end{itemize}\nsince $\\|x_I^*\\|_1 = \\|x^*\\|_1 = b^T y^* = \\langle A_I x_I^*,y^*\\rangle = \\langle x_I^*, A_I^T y^*\\rangle\\le \\|x_I^*\\|_1 \\|A^T y^*\\|_\\infty \\le \\|x_I^*\\|_1$ and thus $\\langle x_I^*, A_I^T y^*\\rangle = \\|x_I^*\\|_1$, which dictates 4). From 3) and 4), Condition \\ref{cond:unique}, part 2, holds with $y=y^*$.\n\\end{IEEEproof}","start":29587,"end":30001}]},{"id":"S039","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"主定理的基追踪部分证明","source_units":[{"raw_tex":"\\begin{IEEEproof}[Proof of Theorem \\ref{thm:unique} for problem \\eqref{eq:bp}]\nThe above proof for Theorem \\ref{thm:bp} also serves the proof of  Theorem \\ref{thm:unique} for problem \\eqref{eq:bp} since $Ax^*=b$ is  involved only in the optimality part, not the uniqueness part.\n\\end{IEEEproof}","start":30003,"end":30297}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"给定解时复用基追踪定理","source_units":[{"raw_tex":"The above proof for Theorem \\ref{thm:bp} also serves the proof of  Theorem \\ref{thm:unique} for problem \\eqref{eq:bp} since $Ax^*=b$ is  involved only in the optimality part, not the uniqueness part.\n","start":30082,"end":30282}]},{"id":"S041","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"已证明的基追踪唯一性分支","source_units":[{"raw_tex":"The above proof for Theorem \\ref{thm:bp} also serves the proof of  Theorem \\ref{thm:unique} for problem \\eqref{eq:bp} since $Ax^*=b$ is  involved only in the optimality part, not the uniqueness part.\n","start":30082,"end":30282}]},{"id":"S042","semantic_type":"lemma","kind":"lemma","labels":["lem01"],"declared_external":false,"source_only":false,"title":"凸集上常值和的分离","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem01}\nLet $f$ be a strictly convex function. If $f(Ax-b)+\\|x\\|_1$ is constant on a convex set $S$, then both $Ax-b$ and $\\|x\\|_1$ are  constant on $S$.\n\\end{lemma}","start":30415,"end":30599}]},{"id":"S043","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"常值和引理的完整证明","source_units":[{"raw_tex":"\\begin{IEEEproof}\nIt suffices to prove the case where $S$ has more one point. Let $x_1$ and $x_2$ be any two  different points in $S$. Consider the line segment $L$ connecting $x_1$ and $x_2$. Since  $X$ is convex, we have $L\\subset X$ and that   $f(Ax-b)+\\|x\\|_1$ is constant on $L$.  On one hand, $\\|x\\|_1$ is piece-wise linear over $L$; on the other hand, the strict convexity of $f$ makes it impossible for $f(Ax-b)$ to be piece-wise linear over $L$ unless $Ax_1-b=Ax_2-b$. Hence, we have $Ax_1-b=Ax_2-b$ and thus $f(Ax_1-b)=f(Ax_2-b)$, from which it follows  $\\|x_1\\|_1= \\|x_2\\|_1$. Since $x_1$ and $x_2$ are arbitrary two points in $S$, the lemma is proved.\n\\end{IEEEproof}","start":30600,"end":31279}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"任意两点间的常值线段","source_units":[{"raw_tex":"It suffices to prove the case where $S$ has more one point. Let $x_1$ and $x_2$ be any two  different points in $S$. Consider the line segment $L$ connecting $x_1$ and $x_2$. Since  $X$ is convex, we have $L\\subset X$ and that   $f(Ax-b)+\\|x\\|_1$ is constant on $L$.  On one hand, $\\|x\\|_1$ is piece-wise linear over $L$; on the other hand, the strict convexity of $f$ makes it impossible for $f(Ax-b)$ to be piece-wise linear over $L$ unless $Ax_1-b=Ax_2-b$. Hence, we have $Ax_1-b=Ax_2-b$ and thus $f(Ax_1-b)=f(Ax_2-b)$, from which it follows  $\\|x_1\\|_1= \\|x_2\\|_1$. Since $x_1$ and $x_2$ are arbitrary two points in $S$, the lemma is proved.\n","start":30618,"end":31264}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"分段线性排除不同残差","source_units":[{"raw_tex":"It suffices to prove the case where $S$ has more one point. Let $x_1$ and $x_2$ be any two  different points in $S$. Consider the line segment $L$ connecting $x_1$ and $x_2$. Since  $X$ is convex, we have $L\\subset X$ and that   $f(Ax-b)+\\|x\\|_1$ is constant on $L$.  On one hand, $\\|x\\|_1$ is piece-wise linear over $L$; on the other hand, the strict convexity of $f$ makes it impossible for $f(Ax-b)$ to be piece-wise linear over $L$ unless $Ax_1-b=Ax_2-b$. Hence, we have $Ax_1-b=Ax_2-b$ and thus $f(Ax_1-b)=f(Ax_2-b)$, from which it follows  $\\|x_1\\|_1= \\|x_2\\|_1$. Since $x_1$ and $x_2$ are arbitrary two points in $S$, the lemma is proved.\n","start":30618,"end":31264}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"由残差相同得到范数相同","source_units":[{"raw_tex":"It suffices to prove the case where $S$ has more one point. Let $x_1$ and $x_2$ be any two  different points in $S$. Consider the line segment $L$ connecting $x_1$ and $x_2$. Since  $X$ is convex, we have $L\\subset X$ and that   $f(Ax-b)+\\|x\\|_1$ is constant on $L$.  On one hand, $\\|x\\|_1$ is piece-wise linear over $L$; on the other hand, the strict convexity of $f$ makes it impossible for $f(Ax-b)$ to be piece-wise linear over $L$ unless $Ax_1-b=Ax_2-b$. Hence, we have $Ax_1-b=Ax_2-b$ and thus $f(Ax_1-b)=f(Ax_2-b)$, from which it follows  $\\|x_1\\|_1= \\|x_2\\|_1$. Since $x_1$ and $x_2$ are arbitrary two points in $S$, the lemma is proved.\n","start":30618,"end":31264}]},{"id":"S047","semantic_type":"lemma","kind":"lemma","labels":["lem02"],"declared_external":false,"source_only":false,"title":"三类模型解集的恒定性","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem02}\nUnder Assumptions \\ref{assmp2} and \\ref{assmp3}, the following statements for problems \\eqref{eq:qpa}--\\eqref{eq:qpc} hold\n\\begin{itemize}\n\\item[1)]  $X_\\lambda, Y_\\sigma$ and $Z_\\tau$ are convex;\n\\item[2)]  In problem \\eqref{eq:qpa}, $Ax-b$ and $\\|x\\|_1$ are constant for all $x\\in X_\\lambda$;\n\\item[3)]  Part 2) holds for problem \\eqref{eq:qpb} and $Y_\\sigma$;\n\\item[4)]  Part 2) holds for problem \\eqref{eq:qpc} and $Z_\\tau$.\n\\end{itemize}\n\\end{lemma}","start":31315,"end":31796}]},{"id":"S048","semantic_type":"derived_fact","kind":"derived_fact","labels":["lem02"],"declared_external":false,"source_only":false,"title":"三个解集均凸","source_units":[{"raw_tex":"\\item[1)]  $X_\\lambda, Y_\\sigma$ and $Z_\\tau$ are convex;\n","start":31481,"end":31539}]},{"id":"S049","semantic_type":"derived_fact","kind":"derived_fact","labels":["lem02"],"declared_external":false,"source_only":false,"title":"惩罚型解集上的两个常值","source_units":[{"raw_tex":"\\item[2)]  In problem \\eqref{eq:qpa}, $Ax-b$ and $\\|x\\|_1$ are constant for all $x\\in X_\\lambda$;\n","start":31539,"end":31637}]},{"id":"S050","semantic_type":"derived_fact","kind":"derived_fact","labels":["lem02"],"declared_external":false,"source_only":false,"title":"损失约束解集上的两个常值","source_units":[{"raw_tex":"\\item[3)]  Part 2) holds for problem \\eqref{eq:qpb} and $Y_\\sigma$;\n","start":31637,"end":31705}]},{"id":"S051","semantic_type":"derived_fact","kind":"derived_fact","labels":["lem02"],"declared_external":false,"source_only":false,"title":"范数约束解集上的两个常值","source_units":[{"raw_tex":"\\item[4)]  Part 2) holds for problem \\eqref{eq:qpc} and $Z_\\tau$.\n","start":31705,"end":31771}]},{"id":"S052","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"解集恒定性引理的完整证明","source_units":[{"raw_tex":"\\begin{IEEEproof}\nAssumption \\ref{assmp2} makes sure that $X_\\lambda,Y_\\sigma,Z_\\tau$ are all non-empty.\n\n1) As a well-known result, the  set of solutions of a convex program is convex.\n\n2) Since $f_1(Ax-b)+\\lambda \\|x\\|_1$ is constant over $x\\in X_\\lambda$ and $f_1$ is strictly convex by Assumption \\ref{assmp3}, the result follows directly from Lemma \\ref{lem01}.\n\n3) If $\\vzero\\in Y_\\sigma$, then the optimal objective is $\\|\\vzero\\|_1= 0$; hence, $Y_\\lambda=\\{\\vzero\\}$ and the results hold trivially. Suppose $\\vzero\\not\\in Y_\\sigma$. Since the optimal objective $\\|x\\|_1$ is constant for all $x\\in Y_\\sigma$ and $f_2$ is strictly convex by Assumption \\ref{assmp3}, to prove this part in light of Lemma \\ref{lem01}, we shall show $f_2(Ax-b)=\\sigma$ for all $x\\in Y_\\sigma$.\n\nAssume that there is $\\hat{x}\\in Y_\\sigma$ such that $f_2(A\\hat{x}-b)<\\sigma$. Since $f_2(Ax-b)$ is convex and thus continuous in $x$, there exists a non-empty ball $\\cB$ centered at $\\hat{x}$ with a sufficiently small radius $\\rho>0$ so that $f_2(A\\bar{x}-b)<\\sigma$ for all $\\bar{x}\\in\\cB$.  Let $\\alpha = \\min\\{\\frac{\\rho}{2\\cdot \\|\\hat{x}\\|_2},\\frac{1}{2}\\}\\in (0,1)$. We have $(1-\\alpha)\\hat{x}\\in\\cB$ and  $\\|(1-\\alpha)\\hat{x}\\|_1= (1-\\alpha)\\|\\hat{x}\\|_1<\\|\\hat{x}\\|_1$, so $(1-\\alpha)\\hat{x}$ is both feasible and achieving an objective value lower than the optimal one. Contradiction.\n\n4) By Assumption \\ref{assmp3}, we have $\\|x\\|_1=\\tau$ for all $x\\in Z_\\tau$; otherwise, there exists $\\bar{x}\\in Z_\\tau$ such that  $\\tau>f_3(A\\bar{x}-b)\\ge \\inf \\{\\|x\\|_1: f_3(Ax-b) = f_3^*\\}$, contradicting Assumption \\ref{assmp3}. Since the optimal objective $f_3(Ax-b)$ is constant for all $x\\in Z_\\tau$ and $f_3$ is strictly convex by Assumption \\ref{assmp3},  the result follows from Lemma \\ref{lem01}.\n\\end{IEEEproof}","start":31797,"end":33597}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"确认三个解集非空","source_units":[{"raw_tex":"Assumption \\ref{assmp2} makes sure that $X_\\lambda,Y_\\sigma,Z_\\tau$ are all non-empty.\n\n","start":31815,"end":31903}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"调用凸规划解集凸性","source_units":[{"raw_tex":"1) As a well-known result, the  set of solutions of a convex program is convex.\n\n","start":31903,"end":31984}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"惩罚目标恒定后应用引理","source_units":[{"raw_tex":"2) Since $f_1(Ax-b)+\\lambda \\|x\\|_1$ is constant over $x\\in X_\\lambda$ and $f_1$ is strictly convex by Assumption \\ref{assmp3}, the result follows directly from Lemma \\ref{lem01}.\n\n","start":31984,"end":32165}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"损失约束模型的零解分支","source_units":[{"raw_tex":"3) If $\\vzero\\in Y_\\sigma$, then the optimal objective is $\\|\\vzero\\|_1= 0$; hence, $Y_\\lambda=\\{\\vzero\\}$ and the results hold trivially. Suppose $\\vzero\\not\\in Y_\\sigma$. Since the optimal objective $\\|x\\|_1$ is constant for all $x\\in Y_\\sigma$ and $f_2$ is strictly convex by Assumption \\ref{assmp3}, to prove this part in light of Lemma \\ref{lem01}, we shall show $f_2(Ax-b)=\\sigma$ for all $x\\in Y_\\sigma$.\n\n","start":32165,"end":32578}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"向原点缩放排除松弛约束","source_units":[{"raw_tex":"Assume that there is $\\hat{x}\\in Y_\\sigma$ such that $f_2(A\\hat{x}-b)<\\sigma$. Since $f_2(Ax-b)$ is convex and thus continuous in $x$, there exists a non-empty ball $\\cB$ centered at $\\hat{x}$ with a sufficiently small radius $\\rho>0$ so that $f_2(A\\bar{x}-b)<\\sigma$ for all $\\bar{x}\\in\\cB$.  Let $\\alpha = \\min\\{\\frac{\\rho}{2\\cdot \\|\\hat{x}\\|_2},\\frac{1}{2}\\}\\in (0,1)$. We have $(1-\\alpha)\\hat{x}\\in\\cB$ and  $\\|(1-\\alpha)\\hat{x}\\|_1= (1-\\alpha)\\|\\hat{x}\\|_1<\\|\\hat{x}\\|_1$, so $(1-\\alpha)\\hat{x}$ is both feasible and achieving an objective value lower than the optimal one. Contradiction.\n\n","start":32578,"end":33173}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"损失约束模型应用常值引理","source_units":[{"raw_tex":"3) If $\\vzero\\in Y_\\sigma$, then the optimal objective is $\\|\\vzero\\|_1= 0$; hence, $Y_\\lambda=\\{\\vzero\\}$ and the results hold trivially. Suppose $\\vzero\\not\\in Y_\\sigma$. Since the optimal objective $\\|x\\|_1$ is constant for all $x\\in Y_\\sigma$ and $f_2$ is strictly convex by Assumption \\ref{assmp3}, to prove this part in light of Lemma \\ref{lem01}, we shall show $f_2(Ax-b)=\\sigma$ for all $x\\in Y_\\sigma$.\n\nAssume that there is $\\hat{x}\\in Y_\\sigma$ such that $f_2(A\\hat{x}-b)<\\sigma$. Since $f_2(Ax-b)$ is convex and thus continuous in $x$, there exists a non-empty ball $\\cB$ centered at $\\hat{x}$ with a sufficiently small radius $\\rho>0$ so that $f_2(A\\bar{x}-b)<\\sigma$ for all $\\bar{x}\\in\\cB$.  Let $\\alpha = \\min\\{\\frac{\\rho}{2\\cdot \\|\\hat{x}\\|_2},\\frac{1}{2}\\}\\in (0,1)$. We have $(1-\\alpha)\\hat{x}\\in\\cB$ and  $\\|(1-\\alpha)\\hat{x}\\|_1= (1-\\alpha)\\|\\hat{x}\\|_1<\\|\\hat{x}\\|_1$, so $(1-\\alpha)\\hat{x}$ is both feasible and achieving an objective value lower than the optimal one. Contradiction.\n","start":32165,"end":33172}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"作者宣称范数约束活跃","source_units":[{"raw_tex":"4) By Assumption \\ref{assmp3}, we have $\\|x\\|_1=\\tau$ for all $x\\in Z_\\tau$; otherwise, there exists $\\bar{x}\\in Z_\\tau$ such that  $\\tau>f_3(A\\bar{x}-b)\\ge \\inf \\{\\|x\\|_1: f_3(Ax-b) = f_3^*\\}$, contradicting Assumption \\ref{assmp3}. Since the optimal objective $f_3(Ax-b)$ is constant for all $x\\in Z_\\tau$ and $f_3$ is strictly convex by Assumption \\ref{assmp3},  the result follows from Lemma \\ref{lem01}.\n","start":33173,"end":33582}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"范数约束模型应用常值引理","source_units":[{"raw_tex":"4) By Assumption \\ref{assmp3}, we have $\\|x\\|_1=\\tau$ for all $x\\in Z_\\tau$; otherwise, there exists $\\bar{x}\\in Z_\\tau$ such that  $\\tau>f_3(A\\bar{x}-b)\\ge \\inf \\{\\|x\\|_1: f_3(Ax-b) = f_3^*\\}$, contradicting Assumption \\ref{assmp3}. Since the optimal objective $f_3(Ax-b)$ is constant for all $x\\in Z_\\tau$ and $f_3$ is strictly convex by Assumption \\ref{assmp3},  the result follows from Lemma \\ref{lem01}.\n","start":33173,"end":33582}]},{"id":"S061","semantic_type":"proof","kind":"proof","labels":["eq:dd"],"declared_external":false,"source_only":false,"title":"主定理其余模型的完整证明","source_units":[{"raw_tex":"\\begin{IEEEproof}[Proof of Theorem \\ref{thm:unique} for problems \\eqref{eq:qpa}--\\eqref{eq:qpc}]\nThis proof exploits Lemma \\ref{lem02}. Since the results of Lemma \\ref{lem02} are are identical for problems \\eqref{eq:qpa}--\\eqref{eq:qpc}, we  present the proof  for problem \\eqref{eq:qpa}. The proofs for the other two problems are similar.\n\nFrom Assumption \\ref{assmp3}, $X_\\lambda$ is nonempty so we pick $x^*\\in X_\\lambda$. Let $b^* = Ax^*$, which is independent of the choice of $x^*$ according to Lemma \\ref{lem02}. We introduce the linear program\n\\beq\\label{eq:dd}\n\\min \\|x\\|_1,\\quad\\text{s.t.}~Ax= b^*,\n\\eeq\nand let $X^*$ denote its solution set.\n\nNow, we show that $X_\\lambda = X^*$. Since $Ax = Ax^*$ and $\\|x\\|_1 = \\|x^*\\|_1$ for all $x\\in X_\\lambda$ and conversely any $x$ obeying $Ax=Ax^*$ and $\\|x\\|_1=\\|x^*\\|_1$  belongs to $X_\\lambda$, it is suffices to show that $\\|{x}\\|_1 = \\|x^*\\|_1$ for any ${x}\\in X^*$. Assuming this does \\emph{not} hold, then since problem \\eqref{eq:dd}  has  $x^*$ as a feasible solution and has a finite objective,  we have a nonempty $X^*$ and there exists $\\bar{x}\\in X^*$ satisfying $\\|\\bar{x}\\|_1 < \\|x^*\\|_1$. But,  $f(A\\bar{x}-b) = f(b^*-b)=f(Ax^*-b)$ and $\\|\\bar{x}\\|_1 < \\|x^*\\|_1$ mean that $\\bar{x}$ is a strictly better solution to problem \\eqref{eq:qpa} than $x^*$, contradicting the assumption $x^*\\in X_\\lambda$.\n\nSince $X_\\lambda = X^*$,  $x^*$ is the unique solution to problem \\eqref{eq:qpa} if and only if it is the same to problem \\eqref{eq:dd}. Since problem \\eqref{eq:dd} is in the same form of problem \\eqref{eq:bp}, applying the part of Theorem \\ref{thm:unique} for problem \\eqref{eq:bp}, which is already proved, we conclude that  $x^*$ is the unique solution to problem \\eqref{eq:qpa} if and only if Condition \\ref{cond:unique} holds.\n\\end{IEEEproof}","start":33599,"end":35415}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"其余模型共用归约的声明","source_units":[{"raw_tex":"\\begin{IEEEproof}[Proof of Theorem \\ref{thm:unique} for problems \\eqref{eq:qpa}--\\eqref{eq:qpc}]\nThis proof exploits Lemma \\ref{lem02}. Since the results of Lemma \\ref{lem02} are are identical for problems \\eqref{eq:qpa}--\\eqref{eq:qpc}, we  present the proof  for problem \\eqref{eq:qpa}. The proofs for the other two problems are similar.\n\n","start":33599,"end":33940}]},{"id":"S063","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:dd"],"declared_external":false,"source_only":false,"title":"固定最优测量的基追踪","source_units":[{"raw_tex":"From Assumption \\ref{assmp3}, $X_\\lambda$ is nonempty so we pick $x^*\\in X_\\lambda$. Let $b^* = Ax^*$, which is independent of the choice of $x^*$ according to Lemma \\ref{lem02}. We introduce the linear program\n\\beq\\label{eq:dd}\n\\min \\|x\\|_1,\\quad\\text{s.t.}~Ax= b^*,\n\\eeq\nand let $X^*$ denote its solution set.\n\n","start":33940,"end":34253}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"原解集的测量与范数刻画","source_units":[{"raw_tex":"Now, we show that $X_\\lambda = X^*$. Since $Ax = Ax^*$ and $\\|x\\|_1 = \\|x^*\\|_1$ for all $x\\in X_\\lambda$ and conversely any $x$ obeying $Ax=Ax^*$ and $\\|x\\|_1=\\|x^*\\|_1$  belongs to $X_\\lambda$, it is suffices to show that $\\|{x}\\|_1 = \\|x^*\\|_1$ for any ${x}\\in X^*$. Assuming this does \\emph{not} hold, then since problem \\eqref{eq:dd}  has  $x^*$ as a feasible solution and has a finite objective,  we have a nonempty $X^*$ and there exists $\\bar{x}\\in X^*$ satisfying $\\|\\bar{x}\\|_1 < \\|x^*\\|_1$. But,  $f(A\\bar{x}-b) = f(b^*-b)=f(Ax^*-b)$ and $\\|\\bar{x}\\|_1 < \\|x^*\\|_1$ mean that $\\bar{x}$ is a strictly better solution to problem \\eqref{eq:qpa} than $x^*$, contradicting the assumption $x^*\\in X_\\lambda$.\n","start":34253,"end":34967}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"排除基追踪中的更小范数","source_units":[{"raw_tex":"Now, we show that $X_\\lambda = X^*$. Since $Ax = Ax^*$ and $\\|x\\|_1 = \\|x^*\\|_1$ for all $x\\in X_\\lambda$ and conversely any $x$ obeying $Ax=Ax^*$ and $\\|x\\|_1=\\|x^*\\|_1$  belongs to $X_\\lambda$, it is suffices to show that $\\|{x}\\|_1 = \\|x^*\\|_1$ for any ${x}\\in X^*$. Assuming this does \\emph{not} hold, then since problem \\eqref{eq:dd}  has  $x^*$ as a feasible solution and has a finite objective,  we have a nonempty $X^*$ and there exists $\\bar{x}\\in X^*$ satisfying $\\|\\bar{x}\\|_1 < \\|x^*\\|_1$. But,  $f(A\\bar{x}-b) = f(b^*-b)=f(Ax^*-b)$ and $\\|\\bar{x}\\|_1 < \\|x^*\\|_1$ mean that $\\bar{x}$ is a strictly better solution to problem \\eqref{eq:qpa} than $x^*$, contradicting the assumption $x^*\\in X_\\lambda$.\n","start":34253,"end":34967}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"两个解集相等","source_units":[{"raw_tex":"Now, we show that $X_\\lambda = X^*$. Since $Ax = Ax^*$ and $\\|x\\|_1 = \\|x^*\\|_1$ for all $x\\in X_\\lambda$ and conversely any $x$ obeying $Ax=Ax^*$ and $\\|x\\|_1=\\|x^*\\|_1$  belongs to $X_\\lambda$, it is suffices to show that $\\|{x}\\|_1 = \\|x^*\\|_1$ for any ${x}\\in X^*$. Assuming this does \\emph{not} hold, then since problem \\eqref{eq:dd}  has  $x^*$ as a feasible solution and has a finite objective,  we have a nonempty $X^*$ and there exists $\\bar{x}\\in X^*$ satisfying $\\|\\bar{x}\\|_1 < \\|x^*\\|_1$. But,  $f(A\\bar{x}-b) = f(b^*-b)=f(Ax^*-b)$ and $\\|\\bar{x}\\|_1 < \\|x^*\\|_1$ mean that $\\bar{x}$ is a strictly better solution to problem \\eqref{eq:qpa} than $x^*$, contradicting the assumption $x^*\\in X_\\lambda$.\n\nSince $X_\\lambda = X^*$,  $x^*$ is the unique solution to problem \\eqref{eq:qpa} if and only if it is the same to problem \\eqref{eq:dd}. Since problem \\eqref{eq:dd} is in the same form of problem \\eqref{eq:bp}, applying the part of Theorem \\ref{thm:unique} for problem \\eqref{eq:bp}, which is already proved, we conclude that  $x^*$ is the unique solution to problem \\eqref{eq:qpa} if and only if Condition \\ref{cond:unique} holds.\n","start":34253,"end":35400}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"转移唯一性并调用已证分支","source_units":[{"raw_tex":"Since $X_\\lambda = X^*$,  $x^*$ is the unique solution to problem \\eqref{eq:qpa} if and only if it is the same to problem \\eqref{eq:dd}. Since problem \\eqref{eq:dd} is in the same form of problem \\eqref{eq:bp}, applying the part of Theorem \\ref{thm:unique} for problem \\eqref{eq:bp}, which is already proved, we conclude that  $x^*$ is the unique solution to problem \\eqref{eq:qpa} if and only if Condition \\ref{cond:unique} holds.\n\\end{IEEEproof}","start":34968,"end":35415}]},{"id":"S068","semantic_type":"external_fact","kind":"external_fact","labels":["eq:pdpair"],"declared_external":true,"declaration_text":["原始先修列表第1项；原文第417行引用 Dantzig [Da]，书目信息位于第621行。"],"source_only":false,"title":"线性规划强对偶与达到性","source_units":[{"raw_tex":"It remains to construct a vector $y$ for Condition \\ref{cond:unique}, part 2.\nOur construction is based on  the strong convexity relation between a linear program (called the primal problem) and its dual problem, namely, if one problem has a solution, so does the other, and the two solutions must give the same objective value. (For the interested reader, this result follows from the Hahn-Banach separation theorem, also from the theorem of alternatives \\cite{Da}. Alternatively, it can be  obtain constructively via the Simplex method; specifically, whenever a primal solution exists, the Simplex method terminates in a finite number of steps with a primal-dual solution pair.)\n","start":25055,"end":25736}]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先修列表第2项：有限维凸分析中的最小化解集凸性；作者第507行明确称为已知结果。"],"source_only":false,"title":"凸规划解集为凸集","source_units":[]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先修列表第2项：有限凸函数连续性；原文第513行用“convex and thus continuous”调用。"],"source_only":false,"title":"有限凸函数的连续性","source_units":[]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先修列表第2项：可行且强制的范数最小化达到最优值；作者第527行省略此标准依据，由标注者显式化。"],"source_only":false,"title":"可行范数最小化的达到性","source_units":[]},{"id":"S072","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先修列表第3项：标准范数对偶不等式；基本范数性质作为作者隐含使用的标准事实由标注者显式化。"],"source_only":false,"title":"范数对偶界与基本范数性质","source_units":[]},{"id":"S073","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先修列表第3项：初等线性代数；局部符号稳定及坐标等号性质为作者隐含使用的标准事实，由标注者显式化。"],"source_only":false,"title":"秩、局部符号与等号性质","source_units":[]},{"id":"S074","semantic_type":"assumption","kind":"assumption","labels":["lem01"],"declared_external":false,"source_only":false,"title":"引理1的局部前提","source_units":[{"raw_tex":"Let $f$ be a strictly convex function. If $f(Ax-b)+\\|x\\|_1$ is constant on a convex set $S$, then both $Ax-b$ and $\\|x\\|_1$ are  constant on $S$.\n","start":30442,"end":30588}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S018","edges":[{"source_id":"S001","target_id":"S018","kind":"supports"},{"source_id":"S002","target_id":"S018","kind":"supports"},{"source_id":"S003","target_id":"S018","kind":"supports"},{"source_id":"S004","target_id":"S018","kind":"supports"},{"source_id":"S005","target_id":"S018","kind":"supports"},{"source_id":"S006","target_id":"S018","kind":"supports"},{"source_id":"S014","target_id":"S018","kind":"supports"},{"source_id":"S015","target_id":"S018","kind":"supports"},{"source_id":"S016","target_id":"S018","kind":"supports"},{"source_id":"S008","target_id":"S018","kind":"supports"},{"source_id":"S009","target_id":"S018","kind":"supports"},{"source_id":"S010","target_id":"S018","kind":"supports"},{"source_id":"S011","target_id":"S018","kind":"supports"},{"source_id":"S001","target_id":"S019","kind":"supports"},{"source_id":"S008","target_id":"S019","kind":"supports"},{"source_id":"S009","target_id":"S019","kind":"supports"},{"source_id":"S014","target_id":"S019","kind":"supports"},{"source_id":"S015","target_id":"S019","kind":"supports"},{"source_id":"S016","target_id":"S019","kind":"supports"},{"source_id":"S016","target_id":"S021","kind":"supports"},{"source_id":"S015","target_id":"S021","kind":"supports"},{"source_id":"S001","target_id":"S021","kind":"supports"},{"source_id":"S014","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S015","target_id":"S022","kind":"supports"},{"source_id":"S073","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S016","target_id":"S023","kind":"supports"},{"source_id":"S072","target_id":"S023","kind":"supports"},{"source_id":"S021","target_id":"S024","kind":"supports"},{"source_id":"S016","target_id":"S024","kind":"supports"},{"source_id":"S014","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S014","target_id":"S027","kind":"supports"},{"source_id":"S073","target_id":"S027","kind":"supports"},{"source_id":"S001","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S026","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S068","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S014","target_id":"S031","kind":"supports"},{"source_id":"S026","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S014","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S031","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S068","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S030","target_id":"S036","kind":"supports"},{"source_id":"S031","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S033","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S031","target_id":"S037","kind":"supports"},{"source_id":"S030","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S026","target_id":"S038","kind":"supports"},{"source_id":"S014","target_id":"S038","kind":"supports"},{"source_id":"S072","target_id":"S038","kind":"supports"},{"source_id":"S073","target_id":"S038","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S020","kind":"supports"},{"source_id":"S023","target_id":"S020","kind":"supports"},{"source_id":"S024","target_id":"S020","kind":"supports"},{"source_id":"S025","target_id":"S020","kind":"supports"},{"source_id":"S026","target_id":"S020","kind":"supports"},{"source_id":"S027","target_id":"S020","kind":"supports"},{"source_id":"S028","target_id":"S020","kind":"supports"},{"source_id":"S029","target_id":"S020","kind":"supports"},{"source_id":"S030","target_id":"S020","kind":"supports"},{"source_id":"S031","target_id":"S020","kind":"supports"},{"source_id":"S032","target_id":"S020","kind":"supports"},{"source_id":"S033","target_id":"S020","kind":"supports"},{"source_id":"S034","target_id":"S020","kind":"supports"},{"source_id":"S035","target_id":"S020","kind":"supports"},{"source_id":"S036","target_id":"S020","kind":"supports"},{"source_id":"S037","target_id":"S020","kind":"supports"},{"source_id":"S038","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S040","kind":"supports"},{"source_id":"S001","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S074","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"},{"source_id":"S046","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S009","target_id":"S047","kind":"supports"},{"source_id":"S010","target_id":"S047","kind":"supports"},{"source_id":"S011","target_id":"S047","kind":"supports"},{"source_id":"S002","target_id":"S047","kind":"supports"},{"source_id":"S003","target_id":"S047","kind":"supports"},{"source_id":"S004","target_id":"S047","kind":"supports"},{"source_id":"S006","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S047","target_id":"S050","kind":"supports"},{"source_id":"S047","target_id":"S051","kind":"supports"},{"source_id":"S009","target_id":"S053","kind":"supports"},{"source_id":"S002","target_id":"S054","kind":"supports"},{"source_id":"S003","target_id":"S054","kind":"supports"},{"source_id":"S004","target_id":"S054","kind":"supports"},{"source_id":"S005","target_id":"S054","kind":"supports"},{"source_id":"S069","target_id":"S054","kind":"supports"},{"source_id":"S042","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S010","target_id":"S055","kind":"supports"},{"source_id":"S002","target_id":"S055","kind":"supports"},{"source_id":"S005","target_id":"S055","kind":"supports"},{"source_id":"S003","target_id":"S056","kind":"supports"},{"source_id":"S072","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S003","target_id":"S057","kind":"supports"},{"source_id":"S010","target_id":"S057","kind":"supports"},{"source_id":"S070","target_id":"S057","kind":"supports"},{"source_id":"S072","target_id":"S057","kind":"supports"},{"source_id":"S056","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S054","target_id":"S058","kind":"supports"},{"source_id":"S010","target_id":"S058","kind":"supports"},{"source_id":"S042","target_id":"S058","kind":"supports"},{"source_id":"S011","target_id":"S059","kind":"supports"},{"source_id":"S004","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S054","target_id":"S060","kind":"supports"},{"source_id":"S010","target_id":"S060","kind":"supports"},{"source_id":"S004","target_id":"S060","kind":"supports"},{"source_id":"S042","target_id":"S060","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S055","target_id":"S052","kind":"supports"},{"source_id":"S056","target_id":"S052","kind":"supports"},{"source_id":"S057","target_id":"S052","kind":"supports"},{"source_id":"S058","target_id":"S052","kind":"supports"},{"source_id":"S059","target_id":"S052","kind":"supports"},{"source_id":"S060","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S047","kind":"supports"},{"source_id":"S049","target_id":"S062","kind":"supports"},{"source_id":"S050","target_id":"S062","kind":"supports"},{"source_id":"S051","target_id":"S062","kind":"supports"},{"source_id":"S009","target_id":"S063","kind":"supports"},{"source_id":"S049","target_id":"S063","kind":"supports"},{"source_id":"S006","target_id":"S063","kind":"supports"},{"source_id":"S049","target_id":"S064","kind":"supports"},{"source_id":"S002","target_id":"S064","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S063","target_id":"S065","kind":"supports"},{"source_id":"S002","target_id":"S065","kind":"supports"},{"source_id":"S005","target_id":"S065","kind":"supports"},{"source_id":"S071","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S063","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S063","target_id":"S067","kind":"supports"},{"source_id":"S041","target_id":"S067","kind":"supports"},{"source_id":"S015","target_id":"S067","kind":"supports"},{"source_id":"S016","target_id":"S067","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S063","target_id":"S061","kind":"supports"},{"source_id":"S064","target_id":"S061","kind":"supports"},{"source_id":"S065","target_id":"S061","kind":"supports"},{"source_id":"S066","target_id":"S061","kind":"supports"},{"source_id":"S067","target_id":"S061","kind":"supports"},{"source_id":"S039","target_id":"S018","kind":"supports"},{"source_id":"S061","target_id":"S018","kind":"supports"},{"source_id":"S074","target_id":"S044","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S018","edges":[{"source_id":"S001","target_id":"S018","kind":"supports"},{"source_id":"S002","target_id":"S018","kind":"supports"},{"source_id":"S003","target_id":"S018","kind":"supports"},{"source_id":"S004","target_id":"S018","kind":"supports"},{"source_id":"S005","target_id":"S018","kind":"supports"},{"source_id":"S006","target_id":"S018","kind":"supports"},{"source_id":"S014","target_id":"S018","kind":"supports"},{"source_id":"S015","target_id":"S018","kind":"supports"},{"source_id":"S016","target_id":"S018","kind":"supports"},{"source_id":"S008","target_id":"S018","kind":"supports"},{"source_id":"S009","target_id":"S018","kind":"supports"},{"source_id":"S010","target_id":"S018","kind":"supports"},{"source_id":"S011","target_id":"S018","kind":"supports"},{"source_id":"S001","target_id":"S019","kind":"supports"},{"source_id":"S008","target_id":"S019","kind":"supports"},{"source_id":"S009","target_id":"S019","kind":"supports"},{"source_id":"S014","target_id":"S019","kind":"supports"},{"source_id":"S015","target_id":"S019","kind":"supports"},{"source_id":"S016","target_id":"S019","kind":"supports"},{"source_id":"S016","target_id":"S021","kind":"supports"},{"source_id":"S015","target_id":"S021","kind":"supports"},{"source_id":"S001","target_id":"S021","kind":"supports"},{"source_id":"S014","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S015","target_id":"S022","kind":"supports"},{"source_id":"S073","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S016","target_id":"S023","kind":"supports"},{"source_id":"S072","target_id":"S023","kind":"supports"},{"source_id":"S021","target_id":"S024","kind":"supports"},{"source_id":"S016","target_id":"S024","kind":"supports"},{"source_id":"S014","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S014","target_id":"S027","kind":"supports"},{"source_id":"S073","target_id":"S027","kind":"supports"},{"source_id":"S001","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S026","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S068","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S014","target_id":"S031","kind":"supports"},{"source_id":"S026","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S014","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S031","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S068","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S030","target_id":"S036","kind":"supports"},{"source_id":"S031","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S033","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S031","target_id":"S037","kind":"supports"},{"source_id":"S030","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S026","target_id":"S038","kind":"supports"},{"source_id":"S014","target_id":"S038","kind":"supports"},{"source_id":"S072","target_id":"S038","kind":"supports"},{"source_id":"S073","target_id":"S038","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S020","kind":"supports"},{"source_id":"S023","target_id":"S020","kind":"supports"},{"source_id":"S024","target_id":"S020","kind":"supports"},{"source_id":"S025","target_id":"S020","kind":"supports"},{"source_id":"S026","target_id":"S020","kind":"supports"},{"source_id":"S027","target_id":"S020","kind":"supports"},{"source_id":"S028","target_id":"S020","kind":"supports"},{"source_id":"S029","target_id":"S020","kind":"supports"},{"source_id":"S030","target_id":"S020","kind":"supports"},{"source_id":"S031","target_id":"S020","kind":"supports"},{"source_id":"S032","target_id":"S020","kind":"supports"},{"source_id":"S033","target_id":"S020","kind":"supports"},{"source_id":"S034","target_id":"S020","kind":"supports"},{"source_id":"S035","target_id":"S020","kind":"supports"},{"source_id":"S036","target_id":"S020","kind":"supports"},{"source_id":"S037","target_id":"S020","kind":"supports"},{"source_id":"S038","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S040","kind":"supports"},{"source_id":"S001","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S074","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"},{"source_id":"S046","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S009","target_id":"S047","kind":"supports"},{"source_id":"S010","target_id":"S047","kind":"supports"},{"source_id":"S011","target_id":"S047","kind":"supports"},{"source_id":"S002","target_id":"S047","kind":"supports"},{"source_id":"S003","target_id":"S047","kind":"supports"},{"source_id":"S004","target_id":"S047","kind":"supports"},{"source_id":"S006","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S047","target_id":"S050","kind":"supports"},{"source_id":"S047","target_id":"S051","kind":"supports"},{"source_id":"S009","target_id":"S053","kind":"supports"},{"source_id":"S002","target_id":"S054","kind":"supports"},{"source_id":"S003","target_id":"S054","kind":"supports"},{"source_id":"S004","target_id":"S054","kind":"supports"},{"source_id":"S005","target_id":"S054","kind":"supports"},{"source_id":"S069","target_id":"S054","kind":"supports"},{"source_id":"S042","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S010","target_id":"S055","kind":"supports"},{"source_id":"S002","target_id":"S055","kind":"supports"},{"source_id":"S005","target_id":"S055","kind":"supports"},{"source_id":"S003","target_id":"S056","kind":"supports"},{"source_id":"S072","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S003","target_id":"S057","kind":"supports"},{"source_id":"S010","target_id":"S057","kind":"supports"},{"source_id":"S070","target_id":"S057","kind":"supports"},{"source_id":"S072","target_id":"S057","kind":"supports"},{"source_id":"S056","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S054","target_id":"S058","kind":"supports"},{"source_id":"S010","target_id":"S058","kind":"supports"},{"source_id":"S042","target_id":"S058","kind":"supports"},{"source_id":"S011","target_id":"S059","kind":"supports"},{"source_id":"S004","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S054","target_id":"S060","kind":"supports"},{"source_id":"S010","target_id":"S060","kind":"supports"},{"source_id":"S004","target_id":"S060","kind":"supports"},{"source_id":"S042","target_id":"S060","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S055","target_id":"S052","kind":"supports"},{"source_id":"S056","target_id":"S052","kind":"supports"},{"source_id":"S057","target_id":"S052","kind":"supports"},{"source_id":"S058","target_id":"S052","kind":"supports"},{"source_id":"S059","target_id":"S052","kind":"supports"},{"source_id":"S060","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S047","kind":"supports"},{"source_id":"S049","target_id":"S062","kind":"supports"},{"source_id":"S050","target_id":"S062","kind":"supports"},{"source_id":"S051","target_id":"S062","kind":"supports"},{"source_id":"S009","target_id":"S063","kind":"supports"},{"source_id":"S049","target_id":"S063","kind":"supports"},{"source_id":"S006","target_id":"S063","kind":"supports"},{"source_id":"S049","target_id":"S064","kind":"supports"},{"source_id":"S002","target_id":"S064","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S063","target_id":"S065","kind":"supports"},{"source_id":"S002","target_id":"S065","kind":"supports"},{"source_id":"S005","target_id":"S065","kind":"supports"},{"source_id":"S071","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S063","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S063","target_id":"S067","kind":"supports"},{"source_id":"S041","target_id":"S067","kind":"supports"},{"source_id":"S015","target_id":"S067","kind":"supports"},{"source_id":"S016","target_id":"S067","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S063","target_id":"S061","kind":"supports"},{"source_id":"S064","target_id":"S061","kind":"supports"},{"source_id":"S065","target_id":"S061","kind":"supports"},{"source_id":"S066","target_id":"S061","kind":"supports"},{"source_id":"S067","target_id":"S061","kind":"supports"},{"source_id":"S039","target_id":"S018","kind":"supports"},{"source_id":"S061","target_id":"S018","kind":"supports"},{"source_id":"S074","target_id":"S044","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:bp"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"基追踪模型","source_units":[{"raw_tex":"Let $x\\in\\RR^n$ be the decision variable. This paper studies the unique solutions of the  $\\ell_1$ minimization problems  including the basis pursuit problem  \\cite{CDS}\n\\begin{subequations}\n\\label{eq:qp}\n\\beq\\label{eq:bp}\n\\min\\|x\\|_1\\quad\\text{s.t.}~Ax=b\n\\eeq\nand  convex problems\n","start":9278,"end":9560}]},{"id":"S002","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:qpa","eq:qpb","eq:qpc"],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"惩罚型模型","source_units":[{"raw_tex":"\\begin{align}\n\\label{eq:qpa}\n&\\min f_1(Ax-b)+\\lambda \\|x\\|_1,\\\\\n","start":9560,"end":9624}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:qpa","eq:qpb","eq:qpc"],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"损失约束模型","source_units":[{"raw_tex":"\\label{eq:qpb}\n&\\min \\|x\\|_1,\\quad\\text{s.t.}~f_2(Ax-b)\\leq \\sigma, \\\\\n","start":9624,"end":9695}]},{"id":"S004","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"范数约束模型","source_units":[{"raw_tex":"\\label{eq:qpc}\n&\\min f_3(Ax-b), \\quad\\text{s.t.}~  \\|x\\|_1\\leq \\tau,\n\\end{align}\n\\end{subequations}\n","start":9695,"end":9795}]},{"id":"S005","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"参数与损失范围","source_units":[{"raw_tex":"where $\\lambda,\\sigma,\\tau>0$ are scalar parameters,  $A$ is a matrix, and $f_i(x), i=1, 2, 3$ are strictly convex functions. The Lasso problem \\cite{Ti} is a special case of  problem \\eqref{eq:qpa} or \\eqref{eq:qpc} while the basis pursuit denoising problem \\cite{CDS} is a special case of  problem \\eqref{eq:qpb} all with $f_i(\\cdot)=\\frac{1}{2}\\|\\cdot\\|_2^2,~ i=1, 2, 3$.\n","start":9795,"end":10170}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"解集与子矩阵记号","source_units":[{"raw_tex":"We let $X$, $X_\\lambda$, $Y_\\sigma$, and $Z_\\tau$ denote the sets of solutions to  problems \\eqref{eq:bp}--\\eqref{eq:qpc}, respectively. We let $a_i$ be the $i$th column of $A$ and $x_i$ be the $i$th entry of $x$. Given an index set $I$, we frequently use $A_I$ as the submatrix of $A$ formed by its columns $a_i$,  $i\\in I$ and  $x_I$ as the subvector of $x$ formed by entries $x_i$, $i\\in I$.\n\n","start":12330,"end":12726}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","labels":["assmp1"],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"矩阵满行秩","source_units":[{"raw_tex":"\\begin{assumption}\\label{assmp1}\nMatrix $A$ has full row rank.\n\\end{assumption}\n","start":12773,"end":12853}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","labels":["assmp2"],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"各模型存在解","source_units":[{"raw_tex":"\\begin{assumption}\\label{assmp2}\nThe solution sets $X$, $X_\\lambda$, $Y_\\sigma$, and $Z_\\tau$ of problems \\eqref{eq:bp}--\\eqref{eq:qpc}, respectively, are nonempty.\n\\end{assumption}\n","start":12853,"end":13035}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","labels":["assmp3"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"损失严格凸","source_units":[{"raw_tex":"\\begin{assumption}\\label{assmp3} In problems \\eqref{eq:qpa}--\\eqref{eq:qpc}, functions $f_1,f_2,f_3$ are strictly convex. In addition,  the constraint of problem \\eqref{eq:qpc} is bounding, namely, $\\tau \\le \\inf \\{\\|x\\|_1: f_3(Ax-b) = f_3^*\\}$, where $f_3^*:=\\min_{y\\in\\RR^n} f(Ay- b)$.\n\\end{assumption}\n","start":13035,"end":13340}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","labels":["assmp3"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"范数约束的界限","source_units":[{"raw_tex":"\\begin{assumption}\\label{assmp3} In problems \\eqref{eq:qpa}--\\eqref{eq:qpc}, functions $f_1,f_2,f_3$ are strictly convex. In addition,  the constraint of problem \\eqref{eq:qpc} is bounding, namely, $\\tau \\le \\inf \\{\\|x\\|_1: f_3(Ax-b) = f_3^*\\}$, where $f_3^*:=\\min_{y\\in\\RR^n} f(Ay- b)$.\n\\end{assumption}\n","start":13035,"end":13340}]},{"id":"S012","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"假设的化简与动机","source_units":[{"raw_tex":"Assumptions \\ref{assmp1} and \\ref{assmp2} are standard. If Assumption \\ref{assmp1} does not hold and $Ax=b$ is consistent, the problems can be simplified; specifically, one can decompose $A=\\begin{bmatrix}A_1\\\\ A_2\\end{bmatrix}$ and $b=\\begin{bmatrix}b_1\\\\ b_2\\end{bmatrix}$ so that and  $A_1$ has full row rank equal to $\\mathrm{rank}(A)$, and one can replace the constraints $Ax = b$ by $A_1x = b_1$ and introduce functions $\\bar{f}_i$ so that $\\bar{f}_i(A_1x - b_1) \\equiv f_i(Ax-b)$, $i=1,2,3$. Assumption \\ref{assmp2} guarantees that the solutions of problems \\eqref{eq:bp}--\\eqref{eq:qpc} can be attained so the discussion of solution uniqueness makes sense.  The strict convexity of $f_1,f_2,f_3$ and the restriction on $\\tau$ in Assumption \\ref{assmp3} are  also fairly basic toward solution uniqueness. Strict convexity rules out piece-wise linearity. (Note that $f_1,f_2,f_3$ can still be non-differentiable.) If the restriction on $\\tau$ is removed, the solution uniqueness of  problem \\eqref{eq:qpc} becomes solely  up to $f_3(Ax-b)$, independent of $\\|x\\|_1$. % and Figure \\ref{?} illustrates this point by an example.\n\n","start":13340,"end":14473}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":["cond:unique"],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"支撑与符号","source_units":[{"raw_tex":"\\begin{condition}\\label{cond:unique}  Under the definitions  $I:=\\supp(x^*)\\subseteq\\{1,\\ldots,n\\}$ and $s:=\\sign(x_I^*)$, matrix $A\\in\\mathbb{R}^{m\\times n}$ has the following properties:\n\\begin{enumerate}\n","start":14629,"end":14836}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":["cond:unique"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"唯一性条件的秩部分","source_units":[{"raw_tex":"\\item submatrix $A_{I}$ has full column rank, and\n","start":14836,"end":14886}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"唯一性条件的严格证书","source_units":[{"raw_tex":"\\item there is  $y \\in \\mathbb{R}^m$ obeying  $A_{I}^Ty=s$ and $\\|A_{I^c}^Ty\\|_\\infty <1$.\n\\end{enumerate}\n\\end{condition}\n","start":14886,"end":15009}]},{"id":"S018","semantic_type":"theorem","kind":"theorem","labels":["thm:unique"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"四类模型的唯一性刻画","source_units":[{"raw_tex":"\\begin{theorem}[Solution uniqueness]\\label{thm:unique} Under Assumptions \\ref{assmp1}--\\ref{assmp3}, given that $x^*$ is a solution  to problem \\eqref{eq:bp}, \\eqref{eq:qpa}, \\eqref{eq:qpb}, or \\eqref{eq:qpc}, $x^*$ is  the unique solution if and only if Condition \\ref{cond:unique} holds.\n\\end{theorem}","start":15151,"end":15454}]},{"id":"S019","semantic_type":"theorem","kind":"theorem","labels":["thm:bp"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"基追踪的唯一最优性","source_units":[{"raw_tex":"\\begin{theorem}[Basis pursuit  unique optimality]\\label{thm:bp} Under Assumptions \\ref{assmp1}--\\ref{assmp2},\n$x^*\\in\\mathbb{R}^n$ is the unique solution to problem \\eqref{eq:bp} if and only if $Ax^*=b$ and Condition \\ref{cond:unique} is satisfied.\n\\end{theorem}","start":15694,"end":15956}]},{"id":"S020","semantic_type":"proof","kind":"proof","labels":["eq:bpdual","eq:idual","eq:iprimal","eq:pdpair","eq:pdpaira","eq:pdpairb"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"基追踪定理的完整证明","source_units":[{"raw_tex":"\\begin{IEEEproof}[Proof of Theorem \\ref{thm:bp}] We frequently use the notions  $I=\\supp(x^*)$ and $s=\\sign(x_I^*)$ below.\n\\medskip\n\n``$\\Longleftarrow$''. This part has been shown in \\cite{Fu, Tr}. For completeness, we give a  proof. Let $y$ satisfy Condition \\ref{cond:unique}, part 2, and let $x\\in\\RR^n$ be an arbitrary vector satisfying $Ax = b$ and $x\\not=x^*$. We shall show $\\|x^*\\|_1<\\|x\\|_1$.\n\nSince $A_I$ has full column rank and $x\\not=x^*$, we have $\\supp(x)\\not=I$; otherwise from $A_I x^*_I =b= A_I x_I$, we would get $x_I^* = x_I$ and  thus the contradiction $x^*=x$.\n\nFrom $\\supp(x)\\not=I$, we  get $b^T y < \\|x\\|_1$. To see this, let $J := \\supp(x) \\setminus I$, which is a non-empty subset of $I^c$. From Condition \\ref{cond:unique}, we have $\\|A_I^T y\\|_\\infty =1$ and $\\|A_J^T y\\|_\\infty < 1$, and thus\n\\begin{align*}\n\\langle x_I, A_I^T y\\rangle&\\le \\|x_I\\|_1\\cdot\\|A_I^T y\\|_\\infty\\le\\|x_I\\|_1,\\\\\n\\langle x_J, A_J^T y\\rangle&\\le \\|x_J\\|_1\\cdot\\|A_J^T y\\|_\\infty<\\|x_J\\|_1,\n\\end{align*}\n(the last inequality is ``$<$'' not ``$\\le$'') which lead to\n\\begin{eqnarray*}\nb^T y = \\langle x, A^T y\\rangle& = &\\langle x_I, A_I^T y\\rangle+ \\langle x_J, A_J^T y\\rangle\\\\\n &<& \\|x_I\\|_1  +\\|x_J\\|_1 = \\|x\\|_1.\n\\end{eqnarray*}\n\nOn the other hand, we have\n$$\n\\|x^*\\|_1 = \\langle x^*_I, \\sign(x^*_I)\\rangle = \\langle x^*_I, A_I^T y\\rangle = \\langle A_I x^*_I,y\\rangle = b^T y\n$$\nand thus $\\|x^*\\|_1=b^T y<\\|x\\|_1$.%, from which it follows $x^*$ is the unique solution to \\eqref{eq:bp}.\n\\medskip\n\n\\noindent``$\\Longrightarrow$''. Assume that $x^*$ is the unique solution to \\eqref{eq:bp}. Obviously, $Ax^*=b$.\n\nIt is easy to obtain Condition \\ref{cond:unique}, part 1. Suppose it does not hold. Then, $A_I$ has a nontrivial null space, and perturbing $x_I^*$ along the null space will change the objective $\\|x_I^*\\|_1 = s^T x^*_I$ while maintaining $A_I x_I^*=b$; hence, this  perturbing breaks the unique optimality of $x^*$.\n(In more details, there exists a \\emph{nonzero} vector $d\\in\\RR^{n}$ such that $A_I d_I = \\vzero$ and $d_{I^c}=\\vzero$. For any scalar $\\alpha$ near zero, we have $\\sign(x^*_I+\\alpha d_I)=\\sign(x_I^*)=s$ and thus $\\|x^*+\\alpha d\\|_1= s^T (x^*_I+\\alpha d_I)=s^T x^*_I+\\alpha( s^T d_I)=\\|x^*\\|_1+\\alpha( s^T d_I)$. Since $x^*$ is the unique solution, we must have $\\|x^*_I+\\alpha d\\|_1> \\|x^*\\|_1$ or, equivalently, $\\alpha( s^T d_I)>0$ whenever $\\alpha\\not=0$. This is impossible as we can perturb $\\alpha$ around 0 both ways.)\n\nIt remains to construct a vector $y$ for Condition \\ref{cond:unique}, part 2.\nOur construction is based on  the strong convexity relation between a linear program (called the primal problem) and its dual problem, namely, if one problem has a solution, so does the other, and the two solutions must give the same objective value. (For the interested reader, this result follows from the Hahn-Banach separation theorem, also from the theorem of alternatives \\cite{Da}. Alternatively, it can be  obtain constructively via the Simplex method; specifically, whenever a primal solution exists, the Simplex method terminates in a finite number of steps with a primal-dual solution pair.)\n\nThe strong duality relation holds between  \\eqref{eq:bp} and its dual problem\n\\beq\\label{eq:bpdual}\n\\max_{p\\in\\RR^m} b^Tp\\quad\\text{s.t.}~ \\|A^Tp\\|_\\infty \\le 1\n\\eeq\nbecause  \\eqref{eq:bp} and \\eqref{eq:bpdual}, as a primal-dual pair, are equivalent to the primal-dual linear programs\n\\begin{subequations}\\label{eq:pdpair}\n\\begin{align}\n\\label{eq:pdpaira}\n\\min_{u,v\\in\\RR^n} \\vone^Tu + \\vone^T v & \\quad \\text{s.t.}~A u - Av = b, ~u\\ge 0, ~v\\ge 0,\\\\\n\\label{eq:pdpairb}\n\\max_{q\\in \\RR^m} b^T q& \\quad \\text{s.t.} -\\vone\\le A^T q\\le \\vone,\n\\end{align}\n\\end{subequations}\nrespectively, where the strong duality relation holds between \\eqref{eq:pdpaira} and \\eqref{eq:pdpairb}. By ``equivalent'', we mean that one can obtain solutions from each other by the rules:\n$$\\begin{array}{ll}\n\\text{given}~u^*,v^*, & \\text{obtain}~ x^* = u^* - v^*\\\\\n\\text{given}~x^*, & \\text{obtain}~ u^*= \\max(x^*,\\vzero), ~v^*= \\max(-x^*,\\vzero),\\\\\n\\text{given}~q^*, & \\text{obtain}~ p^*= q^*,\\\\\n\\text{given}~p^*, & \\text{obtain}~ q^*= p^*.\n\\end{array}$$\nTherefore, since \\eqref{eq:bp} has solution $x^*$, there exists a solution $y^*$ to \\eqref{eq:bpdual}, which satisfies $\\|x^*\\|_1 = b^T y^*$ and $\\|A^T y^*\\|_\\infty\\le 1$. (One can obtain such $y^*$ from the  Hahn-Banach separation theorem or the theorem of alternatives rather directly.) However,  $y^*$ may \\emph{not} obey $\\|A_{I^c}^Ty^*\\|_\\infty <1$. We shall perturb $y^*$ so that $\\|A_{I^c}^Ty^*\\|_\\infty <1$.\n\nTo prepare for the perturbation, we let $L:=\\{i\\in I^c:a_i^Ty^* = -1 \\}$ and $U:=\\{i\\in I^c:a_i^Ty^* = 1\\}$. Our goal is to perturb $y^*$ so that  $-1<a_i^Ty^*<1$ for $i\\in L\\cup U$ and $y^*$ remains optimal to  \\eqref{eq:bpdual}.\nTo this end, consider for a fixed $\\alpha>0$ and $t:=\\|x^*\\|_1$, the linear program\n\\beq\\label{eq:iprimal}\n\\min_{x\\in\\RR^n}  \\sum_{i\\in L}\\alpha x_i-\\sum_{i\\in U}\\alpha x_i,\\quad\\text{s.t.}~Ax=b,~\\|x\\|_1\\le t.\n\\eeq\nSince $x^*$ is the unique solution to \\eqref{eq:bp}, it is  the unique feasible solution to problem \\eqref{eq:iprimal}, so problem \\eqref{eq:iprimal} has the optimal objective value $\\sum_{i\\in U}\\alpha x_i^* - \\sum_{i\\in L}\\alpha x_i^*=0$.\nBy setting up equivalent linear programs like what has been done for \\eqref{eq:bp} and \\eqref{eq:bpdual} above,  the strong duality relation holds between \\eqref{eq:iprimal} and its dual problem\n\\beq\\label{eq:idual}\n\\max_{p\\in\\RR^m,q\\in \\RR} b^T p - tq,\\quad\\text{s.t.}~\\|A^Tp - \\alpha r\\|_\\infty \\le q,~q\\ge 0,\n\\eeq\nwhere  $r\\in\\RR^n$ is given by\n$$r_i = \\begin{cases}1,&i\\in L,\\\\\n-1,&i\\in U,\\\\\n0,&\\text{otherwise}.\n\\end{cases}$$\nTherefore, \\eqref{eq:idual} has a solution $(p^*,q^*)$ satisfying $b^T p^*- tq^*= 0$.\n\nAccording to the last constraint of \\eqref{eq:idual}, we have $q^*\\ge 0$, which we split into two cases:  $q^*=0$ and  $q^*>0$.\n\\begin{itemize}\n\\item[i)] If $q^*=0$, we have $A^T p^*=\\alpha r$ and $b^T p^*=0$.\n\\item[ii)] If $q^*>0$, we let $r^*:=p^*/q^*$, which satisfies $b^T r^* =t=  \\|x^*\\|_1$ and $\\|A^Tr^* - \\frac{\\alpha}{q^*}r\\|_\\infty \\le 1$, or equivalently, $-\\vone+\\frac{\\alpha}{q^*}r\\le A^Tr^*  \\le \\vone+\\frac{\\alpha}{q^*}r$.\n\\end{itemize}\nNow we perturb $y^*$. Solve \\eqref{eq:idual}  with a sufficiently small $\\alpha > 0$ and obtain a solution $(p^*,q^*)$. If case i) occurs, we let $y^* \\gets y^* +p^*$; otherwise, we let $y^* \\gets \\frac{1}{2}(y^* +r^*)$. In both cases,\n\\begin{itemize}\n\\item $b^T y^*$ is unchanged, still equal to $\\|x^*\\|_1$;\n\\item $-1<a_i^Ty^*<1$ holds for $i\\in L\\cup U$ after the perturbation;\n\\item for each $i \\not \\in L\\cup U$, if $a_j^Ty^*\\in[-1,1]$ or $a_j^Ty^*\\in(-1,1)$ holds before the perturbation, the same holds after the perturbation;\n\\end{itemize}\nTherefore, after the perturbation, $y^*$ satisfies:\n\\begin{itemize}\n\\item[1)] $b^T y^* = \\|x^*\\|_1$,\n\\item[2)] $\\|A_I^T y^*\\|_\\infty\\le 1$, and\n\\item[3)] $\\|A_{I^c}^T y^*\\|_\\infty <1$.\n\\end{itemize}\nFrom 1) and 2) it follows\n\\begin{itemize}\n\\item[4)] $A_I^T y = \\sign(x_I^*)$\n\n\\end{itemize}\nsince $\\|x_I^*\\|_1 = \\|x^*\\|_1 = b^T y^* = \\langle A_I x_I^*,y^*\\rangle = \\langle x_I^*, A_I^T y^*\\rangle\\le \\|x_I^*\\|_1 \\|A^T y^*\\|_\\infty \\le \\|x_I^*\\|_1$ and thus $\\langle x_I^*, A_I^T y^*\\rangle = \\|x_I^*\\|_1$, which dictates 4). From 3) and 4), Condition \\ref{cond:unique}, part 2, holds with $y=y^*$.\n\\end{IEEEproof}","start":22595,"end":30001}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"充分性分支的比较对象","source_units":[{"raw_tex":"\\begin{IEEEproof}[Proof of Theorem \\ref{thm:bp}] We frequently use the notions  $I=\\supp(x^*)$ and $s=\\sign(x_I^*)$ below.\n\\medskip\n\n``$\\Longleftarrow$''. This part has been shown in \\cite{Fu, Tr}. For completeness, we give a  proof. Let $y$ satisfy Condition \\ref{cond:unique}, part 2, and let $x\\in\\RR^n$ be an arbitrary vector satisfying $Ax = b$ and $x\\not=x^*$. We shall show $\\|x^*\\|_1<\\|x\\|_1$.\n","start":22595,"end":22997}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"排除同支撑竞争解","source_units":[{"raw_tex":"Since $A_I$ has full column rank and $x\\not=x^*$, we have $\\supp(x)\\not=I$; otherwise from $A_I x^*_I =b= A_I x_I$, we would get $x_I^* = x_I$ and  thus the contradiction $x^*=x$.\n","start":22998,"end":23178}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"竞争解的严格对偶界","source_units":[{"raw_tex":"From $\\supp(x)\\not=I$, we  get $b^T y < \\|x\\|_1$. To see this, let $J := \\supp(x) \\setminus I$, which is a non-empty subset of $I^c$. From Condition \\ref{cond:unique}, we have $\\|A_I^T y\\|_\\infty =1$ and $\\|A_J^T y\\|_\\infty < 1$, and thus\n\\begin{align*}\n\\langle x_I, A_I^T y\\rangle&\\le \\|x_I\\|_1\\cdot\\|A_I^T y\\|_\\infty\\le\\|x_I\\|_1,\\\\\n\\langle x_J, A_J^T y\\rangle&\\le \\|x_J\\|_1\\cdot\\|A_J^T y\\|_\\infty<\\|x_J\\|_1,\n\\end{align*}\n(the last inequality is ``$<$'' not ``$\\le$'') which lead to\n\\begin{eqnarray*}\nb^T y = \\langle x, A^T y\\rangle& = &\\langle x_I, A_I^T y\\rangle+ \\langle x_J, A_J^T y\\rangle\\\\\n &<& \\|x_I\\|_1  +\\|x_J\\|_1 = \\|x\\|_1.\n\\end{eqnarray*}\n","start":23179,"end":23830}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"候选解达到对偶值","source_units":[{"raw_tex":"On the other hand, we have\n$$\n\\|x^*\\|_1 = \\langle x^*_I, \\sign(x^*_I)\\rangle = \\langle x^*_I, A_I^T y\\rangle = \\langle A_I x^*_I,y\\rangle = b^T y\n$$\n","start":23831,"end":23980}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"充分性的严格范数比较","source_units":[{"raw_tex":"and thus $\\|x^*\\|_1=b^T y<\\|x\\|_1$.%, from which it follows $x^*$ is the unique solution to \\eqref{eq:bp}.\n\\medskip\n","start":23980,"end":24096}]},{"id":"S026","semantic_type":"assumption","kind":"assumption","labels":["eq:pdpair"],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"必要性分支的唯一解假设","source_units":[{"raw_tex":"\n\\noindent``$\\Longrightarrow$''. Assume that $x^*$ is the unique solution to \\eqref{eq:bp}. Obviously, $Ax^*=b$.\n","start":24096,"end":24209}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"双向零空间扰动","source_units":[{"raw_tex":"It is easy to obtain Condition \\ref{cond:unique}, part 1. Suppose it does not hold. Then, $A_I$ has a nontrivial null space, and perturbing $x_I^*$ along the null space will change the objective $\\|x_I^*\\|_1 = s^T x^*_I$ while maintaining $A_I x_I^*=b$; hence, this  perturbing breaks the unique optimality of $x^*$.\n(In more details, there exists a \\emph{nonzero} vector $d\\in\\RR^{n}$ such that $A_I d_I = \\vzero$ and $d_{I^c}=\\vzero$. For any scalar $\\alpha$ near zero, we have $\\sign(x^*_I+\\alpha d_I)=\\sign(x_I^*)=s$ and thus $\\|x^*+\\alpha d\\|_1= s^T (x^*_I+\\alpha d_I)=s^T x^*_I+\\alpha( s^T d_I)=\\|x^*\\|_1+\\alpha( s^T d_I)$. Since $x^*$ is the unique solution, we must have $\\|x^*_I+\\alpha d\\|_1> \\|x^*\\|_1$ or, equivalently, $\\alpha( s^T d_I)>0$ whenever $\\alpha\\not=0$. This is impossible as we can perturb $\\alpha$ around 0 both ways.)\n","start":24210,"end":25054}]},{"id":"S028","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:bpdual","eq:pdpair"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"基追踪对偶问题","source_units":[{"raw_tex":"The strong duality relation holds between  \\eqref{eq:bp} and its dual problem\n\\beq\\label{eq:bpdual}\n\\max_{p\\in\\RR^m} b^Tp\\quad\\text{s.t.}~ \\|A^Tp\\|_\\infty \\le 1\n\\eeq\n","start":25737,"end":25903}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair","eq:pdpaira","eq:pdpairb"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"正负分解的线性规划对","source_units":[{"raw_tex":"because  \\eqref{eq:bp} and \\eqref{eq:bpdual}, as a primal-dual pair, are equivalent to the primal-dual linear programs\n\\begin{subequations}\\label{eq:pdpair}\n\\begin{align}\n\\label{eq:pdpaira}\n\\min_{u,v\\in\\RR^n} \\vone^Tu + \\vone^T v & \\quad \\text{s.t.}~A u - Av = b, ~u\\ge 0, ~v\\ge 0,\\\\\n\\label{eq:pdpairb}\n\\max_{q\\in \\RR^m} b^T q& \\quad \\text{s.t.} -\\vone\\le A^T q\\le \\vone,\n\\end{align}\n\\end{subequations}\nrespectively, where the strong duality relation holds between \\eqref{eq:pdpaira} and \\eqref{eq:pdpairb}. By ``equivalent'', we mean that one can obtain solutions from each other by the rules:\n$$\\begin{array}{ll}\n\\text{given}~u^*,v^*, & \\text{obtain}~ x^* = u^* - v^*\\\\\n\\text{given}~x^*, & \\text{obtain}~ u^*= \\max(x^*,\\vzero), ~v^*= \\max(-x^*,\\vzero),\\\\\n\\text{given}~q^*, & \\text{obtain}~ p^*= q^*,\\\\\n\\text{given}~p^*, & \\text{obtain}~ q^*= p^*.\n\\end{array}$$\n","start":25903,"end":26766}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"取得非严格对偶证书","source_units":[{"raw_tex":"Therefore, since \\eqref{eq:bp} has solution $x^*$, there exists a solution $y^*$ to \\eqref{eq:bpdual}, which satisfies $\\|x^*\\|_1 = b^T y^*$ and $\\|A^T y^*\\|_\\infty\\le 1$. (One can obtain such $y^*$ from the  Hahn-Banach separation theorem or the theorem of alternatives rather directly.) However,  $y^*$ may \\emph{not} obey $\\|A_{I^c}^Ty^*\\|_\\infty <1$. We shall perturb $y^*$ so that $\\|A_{I^c}^Ty^*\\|_\\infty <1$.\n\n","start":26766,"end":27183}]},{"id":"S031","semantic_type":"definition","kind":"definition","labels":["eq:pdpair"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"支撑外饱和指标","source_units":[{"raw_tex":"To prepare for the perturbation, we let $L:=\\{i\\in I^c:a_i^Ty^* = -1 \\}$ and $U:=\\{i\\in I^c:a_i^Ty^* = 1\\}$. Our goal is to perturb $y^*$ so that  $-1<a_i^Ty^*<1$ for $i\\in L\\cup U$ and $y^*$ remains optimal to  \\eqref{eq:bpdual}.\n","start":27183,"end":27414}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":["eq:iprimal","eq:pdpair"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"辅助原问题的单点可行域","source_units":[{"raw_tex":"To this end, consider for a fixed $\\alpha>0$ and $t:=\\|x^*\\|_1$, the linear program\n\\beq\\label{eq:iprimal}\n\\min_{x\\in\\RR^n}  \\sum_{i\\in L}\\alpha x_i-\\sum_{i\\in U}\\alpha x_i,\\quad\\text{s.t.}~Ax=b,~\\|x\\|_1\\le t.\n\\eeq\nSince $x^*$ is the unique solution to \\eqref{eq:bp}, it is  the unique feasible solution to problem \\eqref{eq:iprimal}, so problem \\eqref{eq:iprimal} has the optimal objective value $\\sum_{i\\in U}\\alpha x_i^* - \\sum_{i\\in L}\\alpha x_i^*=0$.\n","start":27414,"end":27870}]},{"id":"S033","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:idual","eq:pdpair"],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"辅助对偶与方向向量","source_units":[{"raw_tex":"By setting up equivalent linear programs like what has been done for \\eqref{eq:bp} and \\eqref{eq:bpdual} above,  the strong duality relation holds between \\eqref{eq:iprimal} and its dual problem\n\\beq\\label{eq:idual}\n\\max_{p\\in\\RR^m,q\\in \\RR} b^T p - tq,\\quad\\text{s.t.}~\\|A^Tp - \\alpha r\\|_\\infty \\le q,~q\\ge 0,\n\\eeq\nwhere  $r\\in\\RR^n$ is given by\n$$r_i = \\begin{cases}1,&i\\in L,\\\\\n-1,&i\\in U,\\\\\n0,&\\text{otherwise}.\n\\end{cases}$$\n","start":27870,"end":28301}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"辅助对偶达到零值","source_units":[{"raw_tex":"Therefore, \\eqref{eq:idual} has a solution $(p^*,q^*)$ satisfying $b^T p^*- tq^*= 0$.\n\n","start":28301,"end":28388}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"按辅助乘子分情形","source_units":[{"raw_tex":"According to the last constraint of \\eqref{eq:idual}, we have $q^*\\ge 0$, which we split into two cases:  $q^*=0$ and  $q^*>0$.\n\\begin{itemize}\n\\item[i)] If $q^*=0$, we have $A^T p^*=\\alpha r$ and $b^T p^*=0$.\n\\item[ii)] If $q^*>0$, we let $r^*:=p^*/q^*$, which satisfies $b^T r^* =t=  \\|x^*\\|_1$ and $\\|A^Tr^* - \\frac{\\alpha}{q^*}r\\|_\\infty \\le 1$, or equivalently, $-\\vone+\\frac{\\alpha}{q^*}r\\le A^Tr^*  \\le \\vone+\\frac{\\alpha}{q^*}r$.\n\\end{itemize}\n","start":28388,"end":28840}]},{"id":"S036","semantic_type":"algorithm","kind":"algorithm","labels":["eq:pdpair"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"严格化对偶解的更新","source_units":[{"raw_tex":"Now we perturb $y^*$. Solve \\eqref{eq:idual}  with a sufficiently small $\\alpha > 0$ and obtain a solution $(p^*,q^*)$. If case i) occurs, we let $y^* \\gets y^* +p^*$; otherwise, we let $y^* \\gets \\frac{1}{2}(y^* +r^*)$. In both cases,\n\\begin{itemize}\n\\item $b^T y^*$ is unchanged, still equal to $\\|x^*\\|_1$;\n\\item $-1<a_i^Ty^*<1$ holds for $i\\in L\\cup U$ after the perturbation;\n\\item for each $i \\not \\in L\\cup U$, if $a_j^Ty^*\\in[-1,1]$ or $a_j^Ty^*\\in(-1,1)$ holds before the perturbation, the same holds after the perturbation;\n\\end{itemize}\n","start":28840,"end":29388}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"更新后的证书界","source_units":[{"raw_tex":"Therefore, after the perturbation, $y^*$ satisfies:\n\\begin{itemize}\n\\item[1)] $b^T y^* = \\|x^*\\|_1$,\n\\item[2)] $\\|A_I^T y^*\\|_\\infty\\le 1$, and\n\\item[3)] $\\|A_{I^c}^T y^*\\|_\\infty <1$.\n\\end{itemize}\n","start":29388,"end":29587}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":["eq:pdpair"],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"由等号恢复支撑符号","source_units":[{"raw_tex":"From 1) and 2) it follows\n\\begin{itemize}\n\\item[4)] $A_I^T y = \\sign(x_I^*)$\n\n\\end{itemize}\nsince $\\|x_I^*\\|_1 = \\|x^*\\|_1 = b^T y^* = \\langle A_I x_I^*,y^*\\rangle = \\langle x_I^*, A_I^T y^*\\rangle\\le \\|x_I^*\\|_1 \\|A^T y^*\\|_\\infty \\le \\|x_I^*\\|_1$ and thus $\\langle x_I^*, A_I^T y^*\\rangle = \\|x_I^*\\|_1$, which dictates 4). From 3) and 4), Condition \\ref{cond:unique}, part 2, holds with $y=y^*$.\n\\end{IEEEproof}","start":29587,"end":30001}]},{"id":"S039","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"主定理的基追踪部分证明","source_units":[{"raw_tex":"\\begin{IEEEproof}[Proof of Theorem \\ref{thm:unique} for problem \\eqref{eq:bp}]\nThe above proof for Theorem \\ref{thm:bp} also serves the proof of  Theorem \\ref{thm:unique} for problem \\eqref{eq:bp} since $Ax^*=b$ is  involved only in the optimality part, not the uniqueness part.\n\\end{IEEEproof}","start":30003,"end":30297}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"给定解时复用基追踪定理","source_units":[{"raw_tex":"The above proof for Theorem \\ref{thm:bp} also serves the proof of  Theorem \\ref{thm:unique} for problem \\eqref{eq:bp} since $Ax^*=b$ is  involved only in the optimality part, not the uniqueness part.\n","start":30082,"end":30282}]},{"id":"S041","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"已证明的基追踪唯一性分支","source_units":[{"raw_tex":"The above proof for Theorem \\ref{thm:bp} also serves the proof of  Theorem \\ref{thm:unique} for problem \\eqref{eq:bp} since $Ax^*=b$ is  involved only in the optimality part, not the uniqueness part.\n","start":30082,"end":30282}]},{"id":"S042","semantic_type":"lemma","kind":"lemma","labels":["lem01"],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"凸集上常值和的分离","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem01}\nLet $f$ be a strictly convex function. If $f(Ax-b)+\\|x\\|_1$ is constant on a convex set $S$, then both $Ax-b$ and $\\|x\\|_1$ are  constant on $S$.\n\\end{lemma}","start":30415,"end":30599}]},{"id":"S043","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"常值和引理的完整证明","source_units":[{"raw_tex":"\\begin{IEEEproof}\nIt suffices to prove the case where $S$ has more one point. Let $x_1$ and $x_2$ be any two  different points in $S$. Consider the line segment $L$ connecting $x_1$ and $x_2$. Since  $X$ is convex, we have $L\\subset X$ and that   $f(Ax-b)+\\|x\\|_1$ is constant on $L$.  On one hand, $\\|x\\|_1$ is piece-wise linear over $L$; on the other hand, the strict convexity of $f$ makes it impossible for $f(Ax-b)$ to be piece-wise linear over $L$ unless $Ax_1-b=Ax_2-b$. Hence, we have $Ax_1-b=Ax_2-b$ and thus $f(Ax_1-b)=f(Ax_2-b)$, from which it follows  $\\|x_1\\|_1= \\|x_2\\|_1$. Since $x_1$ and $x_2$ are arbitrary two points in $S$, the lemma is proved.\n\\end{IEEEproof}","start":30600,"end":31279}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"任意两点间的常值线段","source_units":[{"raw_tex":"It suffices to prove the case where $S$ has more one point. Let $x_1$ and $x_2$ be any two  different points in $S$. Consider the line segment $L$ connecting $x_1$ and $x_2$. Since  $X$ is convex, we have $L\\subset X$ and that   $f(Ax-b)+\\|x\\|_1$ is constant on $L$.  On one hand, $\\|x\\|_1$ is piece-wise linear over $L$; on the other hand, the strict convexity of $f$ makes it impossible for $f(Ax-b)$ to be piece-wise linear over $L$ unless $Ax_1-b=Ax_2-b$. Hence, we have $Ax_1-b=Ax_2-b$ and thus $f(Ax_1-b)=f(Ax_2-b)$, from which it follows  $\\|x_1\\|_1= \\|x_2\\|_1$. Since $x_1$ and $x_2$ are arbitrary two points in $S$, the lemma is proved.\n","start":30618,"end":31264}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"分段线性排除不同残差","source_units":[{"raw_tex":"It suffices to prove the case where $S$ has more one point. Let $x_1$ and $x_2$ be any two  different points in $S$. Consider the line segment $L$ connecting $x_1$ and $x_2$. Since  $X$ is convex, we have $L\\subset X$ and that   $f(Ax-b)+\\|x\\|_1$ is constant on $L$.  On one hand, $\\|x\\|_1$ is piece-wise linear over $L$; on the other hand, the strict convexity of $f$ makes it impossible for $f(Ax-b)$ to be piece-wise linear over $L$ unless $Ax_1-b=Ax_2-b$. Hence, we have $Ax_1-b=Ax_2-b$ and thus $f(Ax_1-b)=f(Ax_2-b)$, from which it follows  $\\|x_1\\|_1= \\|x_2\\|_1$. Since $x_1$ and $x_2$ are arbitrary two points in $S$, the lemma is proved.\n","start":30618,"end":31264}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"由残差相同得到范数相同","source_units":[{"raw_tex":"It suffices to prove the case where $S$ has more one point. Let $x_1$ and $x_2$ be any two  different points in $S$. Consider the line segment $L$ connecting $x_1$ and $x_2$. Since  $X$ is convex, we have $L\\subset X$ and that   $f(Ax-b)+\\|x\\|_1$ is constant on $L$.  On one hand, $\\|x\\|_1$ is piece-wise linear over $L$; on the other hand, the strict convexity of $f$ makes it impossible for $f(Ax-b)$ to be piece-wise linear over $L$ unless $Ax_1-b=Ax_2-b$. Hence, we have $Ax_1-b=Ax_2-b$ and thus $f(Ax_1-b)=f(Ax_2-b)$, from which it follows  $\\|x_1\\|_1= \\|x_2\\|_1$. Since $x_1$ and $x_2$ are arbitrary two points in $S$, the lemma is proved.\n","start":30618,"end":31264}]},{"id":"S047","semantic_type":"lemma","kind":"lemma","labels":["lem02"],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"三类模型解集的恒定性","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem02}\nUnder Assumptions \\ref{assmp2} and \\ref{assmp3}, the following statements for problems \\eqref{eq:qpa}--\\eqref{eq:qpc} hold\n\\begin{itemize}\n\\item[1)]  $X_\\lambda, Y_\\sigma$ and $Z_\\tau$ are convex;\n\\item[2)]  In problem \\eqref{eq:qpa}, $Ax-b$ and $\\|x\\|_1$ are constant for all $x\\in X_\\lambda$;\n\\item[3)]  Part 2) holds for problem \\eqref{eq:qpb} and $Y_\\sigma$;\n\\item[4)]  Part 2) holds for problem \\eqref{eq:qpc} and $Z_\\tau$.\n\\end{itemize}\n\\end{lemma}","start":31315,"end":31796}]},{"id":"S048","semantic_type":"derived_fact","kind":"derived_fact","labels":["lem02"],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"三个解集均凸","source_units":[{"raw_tex":"\\item[1)]  $X_\\lambda, Y_\\sigma$ and $Z_\\tau$ are convex;\n","start":31481,"end":31539}]},{"id":"S049","semantic_type":"derived_fact","kind":"derived_fact","labels":["lem02"],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"惩罚型解集上的两个常值","source_units":[{"raw_tex":"\\item[2)]  In problem \\eqref{eq:qpa}, $Ax-b$ and $\\|x\\|_1$ are constant for all $x\\in X_\\lambda$;\n","start":31539,"end":31637}]},{"id":"S050","semantic_type":"derived_fact","kind":"derived_fact","labels":["lem02"],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"损失约束解集上的两个常值","source_units":[{"raw_tex":"\\item[3)]  Part 2) holds for problem \\eqref{eq:qpb} and $Y_\\sigma$;\n","start":31637,"end":31705}]},{"id":"S051","semantic_type":"derived_fact","kind":"derived_fact","labels":["lem02"],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"范数约束解集上的两个常值","source_units":[{"raw_tex":"\\item[4)]  Part 2) holds for problem \\eqref{eq:qpc} and $Z_\\tau$.\n","start":31705,"end":31771}]},{"id":"S052","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"解集恒定性引理的完整证明","source_units":[{"raw_tex":"\\begin{IEEEproof}\nAssumption \\ref{assmp2} makes sure that $X_\\lambda,Y_\\sigma,Z_\\tau$ are all non-empty.\n\n1) As a well-known result, the  set of solutions of a convex program is convex.\n\n2) Since $f_1(Ax-b)+\\lambda \\|x\\|_1$ is constant over $x\\in X_\\lambda$ and $f_1$ is strictly convex by Assumption \\ref{assmp3}, the result follows directly from Lemma \\ref{lem01}.\n\n3) If $\\vzero\\in Y_\\sigma$, then the optimal objective is $\\|\\vzero\\|_1= 0$; hence, $Y_\\lambda=\\{\\vzero\\}$ and the results hold trivially. Suppose $\\vzero\\not\\in Y_\\sigma$. Since the optimal objective $\\|x\\|_1$ is constant for all $x\\in Y_\\sigma$ and $f_2$ is strictly convex by Assumption \\ref{assmp3}, to prove this part in light of Lemma \\ref{lem01}, we shall show $f_2(Ax-b)=\\sigma$ for all $x\\in Y_\\sigma$.\n\nAssume that there is $\\hat{x}\\in Y_\\sigma$ such that $f_2(A\\hat{x}-b)<\\sigma$. Since $f_2(Ax-b)$ is convex and thus continuous in $x$, there exists a non-empty ball $\\cB$ centered at $\\hat{x}$ with a sufficiently small radius $\\rho>0$ so that $f_2(A\\bar{x}-b)<\\sigma$ for all $\\bar{x}\\in\\cB$.  Let $\\alpha = \\min\\{\\frac{\\rho}{2\\cdot \\|\\hat{x}\\|_2},\\frac{1}{2}\\}\\in (0,1)$. We have $(1-\\alpha)\\hat{x}\\in\\cB$ and  $\\|(1-\\alpha)\\hat{x}\\|_1= (1-\\alpha)\\|\\hat{x}\\|_1<\\|\\hat{x}\\|_1$, so $(1-\\alpha)\\hat{x}$ is both feasible and achieving an objective value lower than the optimal one. Contradiction.\n\n4) By Assumption \\ref{assmp3}, we have $\\|x\\|_1=\\tau$ for all $x\\in Z_\\tau$; otherwise, there exists $\\bar{x}\\in Z_\\tau$ such that  $\\tau>f_3(A\\bar{x}-b)\\ge \\inf \\{\\|x\\|_1: f_3(Ax-b) = f_3^*\\}$, contradicting Assumption \\ref{assmp3}. Since the optimal objective $f_3(Ax-b)$ is constant for all $x\\in Z_\\tau$ and $f_3$ is strictly convex by Assumption \\ref{assmp3},  the result follows from Lemma \\ref{lem01}.\n\\end{IEEEproof}","start":31797,"end":33597}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"确认三个解集非空","source_units":[{"raw_tex":"Assumption \\ref{assmp2} makes sure that $X_\\lambda,Y_\\sigma,Z_\\tau$ are all non-empty.\n\n","start":31815,"end":31903}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"调用凸规划解集凸性","source_units":[{"raw_tex":"1) As a well-known result, the  set of solutions of a convex program is convex.\n\n","start":31903,"end":31984}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"惩罚目标恒定后应用引理","source_units":[{"raw_tex":"2) Since $f_1(Ax-b)+\\lambda \\|x\\|_1$ is constant over $x\\in X_\\lambda$ and $f_1$ is strictly convex by Assumption \\ref{assmp3}, the result follows directly from Lemma \\ref{lem01}.\n\n","start":31984,"end":32165}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"损失约束模型的零解分支","source_units":[{"raw_tex":"3) If $\\vzero\\in Y_\\sigma$, then the optimal objective is $\\|\\vzero\\|_1= 0$; hence, $Y_\\lambda=\\{\\vzero\\}$ and the results hold trivially. Suppose $\\vzero\\not\\in Y_\\sigma$. Since the optimal objective $\\|x\\|_1$ is constant for all $x\\in Y_\\sigma$ and $f_2$ is strictly convex by Assumption \\ref{assmp3}, to prove this part in light of Lemma \\ref{lem01}, we shall show $f_2(Ax-b)=\\sigma$ for all $x\\in Y_\\sigma$.\n\n","start":32165,"end":32578}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"向原点缩放排除松弛约束","source_units":[{"raw_tex":"Assume that there is $\\hat{x}\\in Y_\\sigma$ such that $f_2(A\\hat{x}-b)<\\sigma$. Since $f_2(Ax-b)$ is convex and thus continuous in $x$, there exists a non-empty ball $\\cB$ centered at $\\hat{x}$ with a sufficiently small radius $\\rho>0$ so that $f_2(A\\bar{x}-b)<\\sigma$ for all $\\bar{x}\\in\\cB$.  Let $\\alpha = \\min\\{\\frac{\\rho}{2\\cdot \\|\\hat{x}\\|_2},\\frac{1}{2}\\}\\in (0,1)$. We have $(1-\\alpha)\\hat{x}\\in\\cB$ and  $\\|(1-\\alpha)\\hat{x}\\|_1= (1-\\alpha)\\|\\hat{x}\\|_1<\\|\\hat{x}\\|_1$, so $(1-\\alpha)\\hat{x}$ is both feasible and achieving an objective value lower than the optimal one. Contradiction.\n\n","start":32578,"end":33173}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"损失约束模型应用常值引理","source_units":[{"raw_tex":"3) If $\\vzero\\in Y_\\sigma$, then the optimal objective is $\\|\\vzero\\|_1= 0$; hence, $Y_\\lambda=\\{\\vzero\\}$ and the results hold trivially. Suppose $\\vzero\\not\\in Y_\\sigma$. Since the optimal objective $\\|x\\|_1$ is constant for all $x\\in Y_\\sigma$ and $f_2$ is strictly convex by Assumption \\ref{assmp3}, to prove this part in light of Lemma \\ref{lem01}, we shall show $f_2(Ax-b)=\\sigma$ for all $x\\in Y_\\sigma$.\n\nAssume that there is $\\hat{x}\\in Y_\\sigma$ such that $f_2(A\\hat{x}-b)<\\sigma$. Since $f_2(Ax-b)$ is convex and thus continuous in $x$, there exists a non-empty ball $\\cB$ centered at $\\hat{x}$ with a sufficiently small radius $\\rho>0$ so that $f_2(A\\bar{x}-b)<\\sigma$ for all $\\bar{x}\\in\\cB$.  Let $\\alpha = \\min\\{\\frac{\\rho}{2\\cdot \\|\\hat{x}\\|_2},\\frac{1}{2}\\}\\in (0,1)$. We have $(1-\\alpha)\\hat{x}\\in\\cB$ and  $\\|(1-\\alpha)\\hat{x}\\|_1= (1-\\alpha)\\|\\hat{x}\\|_1<\\|\\hat{x}\\|_1$, so $(1-\\alpha)\\hat{x}$ is both feasible and achieving an objective value lower than the optimal one. Contradiction.\n","start":32165,"end":33172}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"作者宣称范数约束活跃","source_units":[{"raw_tex":"4) By Assumption \\ref{assmp3}, we have $\\|x\\|_1=\\tau$ for all $x\\in Z_\\tau$; otherwise, there exists $\\bar{x}\\in Z_\\tau$ such that  $\\tau>f_3(A\\bar{x}-b)\\ge \\inf \\{\\|x\\|_1: f_3(Ax-b) = f_3^*\\}$, contradicting Assumption \\ref{assmp3}. Since the optimal objective $f_3(Ax-b)$ is constant for all $x\\in Z_\\tau$ and $f_3$ is strictly convex by Assumption \\ref{assmp3},  the result follows from Lemma \\ref{lem01}.\n","start":33173,"end":33582}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"范数约束模型应用常值引理","source_units":[{"raw_tex":"4) By Assumption \\ref{assmp3}, we have $\\|x\\|_1=\\tau$ for all $x\\in Z_\\tau$; otherwise, there exists $\\bar{x}\\in Z_\\tau$ such that  $\\tau>f_3(A\\bar{x}-b)\\ge \\inf \\{\\|x\\|_1: f_3(Ax-b) = f_3^*\\}$, contradicting Assumption \\ref{assmp3}. Since the optimal objective $f_3(Ax-b)$ is constant for all $x\\in Z_\\tau$ and $f_3$ is strictly convex by Assumption \\ref{assmp3},  the result follows from Lemma \\ref{lem01}.\n","start":33173,"end":33582}]},{"id":"S061","semantic_type":"proof","kind":"proof","labels":["eq:dd"],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"主定理其余模型的完整证明","source_units":[{"raw_tex":"\\begin{IEEEproof}[Proof of Theorem \\ref{thm:unique} for problems \\eqref{eq:qpa}--\\eqref{eq:qpc}]\nThis proof exploits Lemma \\ref{lem02}. Since the results of Lemma \\ref{lem02} are are identical for problems \\eqref{eq:qpa}--\\eqref{eq:qpc}, we  present the proof  for problem \\eqref{eq:qpa}. The proofs for the other two problems are similar.\n\nFrom Assumption \\ref{assmp3}, $X_\\lambda$ is nonempty so we pick $x^*\\in X_\\lambda$. Let $b^* = Ax^*$, which is independent of the choice of $x^*$ according to Lemma \\ref{lem02}. We introduce the linear program\n\\beq\\label{eq:dd}\n\\min \\|x\\|_1,\\quad\\text{s.t.}~Ax= b^*,\n\\eeq\nand let $X^*$ denote its solution set.\n\nNow, we show that $X_\\lambda = X^*$. Since $Ax = Ax^*$ and $\\|x\\|_1 = \\|x^*\\|_1$ for all $x\\in X_\\lambda$ and conversely any $x$ obeying $Ax=Ax^*$ and $\\|x\\|_1=\\|x^*\\|_1$  belongs to $X_\\lambda$, it is suffices to show that $\\|{x}\\|_1 = \\|x^*\\|_1$ for any ${x}\\in X^*$. Assuming this does \\emph{not} hold, then since problem \\eqref{eq:dd}  has  $x^*$ as a feasible solution and has a finite objective,  we have a nonempty $X^*$ and there exists $\\bar{x}\\in X^*$ satisfying $\\|\\bar{x}\\|_1 < \\|x^*\\|_1$. But,  $f(A\\bar{x}-b) = f(b^*-b)=f(Ax^*-b)$ and $\\|\\bar{x}\\|_1 < \\|x^*\\|_1$ mean that $\\bar{x}$ is a strictly better solution to problem \\eqref{eq:qpa} than $x^*$, contradicting the assumption $x^*\\in X_\\lambda$.\n\nSince $X_\\lambda = X^*$,  $x^*$ is the unique solution to problem \\eqref{eq:qpa} if and only if it is the same to problem \\eqref{eq:dd}. Since problem \\eqref{eq:dd} is in the same form of problem \\eqref{eq:bp}, applying the part of Theorem \\ref{thm:unique} for problem \\eqref{eq:bp}, which is already proved, we conclude that  $x^*$ is the unique solution to problem \\eqref{eq:qpa} if and only if Condition \\ref{cond:unique} holds.\n\\end{IEEEproof}","start":33599,"end":35415}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"其余模型共用归约的声明","source_units":[{"raw_tex":"\\begin{IEEEproof}[Proof of Theorem \\ref{thm:unique} for problems \\eqref{eq:qpa}--\\eqref{eq:qpc}]\nThis proof exploits Lemma \\ref{lem02}. Since the results of Lemma \\ref{lem02} are are identical for problems \\eqref{eq:qpa}--\\eqref{eq:qpc}, we  present the proof  for problem \\eqref{eq:qpa}. The proofs for the other two problems are similar.\n\n","start":33599,"end":33940}]},{"id":"S063","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:dd"],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"固定最优测量的基追踪","source_units":[{"raw_tex":"From Assumption \\ref{assmp3}, $X_\\lambda$ is nonempty so we pick $x^*\\in X_\\lambda$. Let $b^* = Ax^*$, which is independent of the choice of $x^*$ according to Lemma \\ref{lem02}. We introduce the linear program\n\\beq\\label{eq:dd}\n\\min \\|x\\|_1,\\quad\\text{s.t.}~Ax= b^*,\n\\eeq\nand let $X^*$ denote its solution set.\n\n","start":33940,"end":34253}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S064"],"declared_external":false,"source_only":false,"title":"原解集的测量与范数刻画","source_units":[{"raw_tex":"Now, we show that $X_\\lambda = X^*$. Since $Ax = Ax^*$ and $\\|x\\|_1 = \\|x^*\\|_1$ for all $x\\in X_\\lambda$ and conversely any $x$ obeying $Ax=Ax^*$ and $\\|x\\|_1=\\|x^*\\|_1$  belongs to $X_\\lambda$, it is suffices to show that $\\|{x}\\|_1 = \\|x^*\\|_1$ for any ${x}\\in X^*$. Assuming this does \\emph{not} hold, then since problem \\eqref{eq:dd}  has  $x^*$ as a feasible solution and has a finite objective,  we have a nonempty $X^*$ and there exists $\\bar{x}\\in X^*$ satisfying $\\|\\bar{x}\\|_1 < \\|x^*\\|_1$. But,  $f(A\\bar{x}-b) = f(b^*-b)=f(Ax^*-b)$ and $\\|\\bar{x}\\|_1 < \\|x^*\\|_1$ mean that $\\bar{x}$ is a strictly better solution to problem \\eqref{eq:qpa} than $x^*$, contradicting the assumption $x^*\\in X_\\lambda$.\n","start":34253,"end":34967}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S065"],"declared_external":false,"source_only":false,"title":"排除基追踪中的更小范数","source_units":[{"raw_tex":"Now, we show that $X_\\lambda = X^*$. Since $Ax = Ax^*$ and $\\|x\\|_1 = \\|x^*\\|_1$ for all $x\\in X_\\lambda$ and conversely any $x$ obeying $Ax=Ax^*$ and $\\|x\\|_1=\\|x^*\\|_1$  belongs to $X_\\lambda$, it is suffices to show that $\\|{x}\\|_1 = \\|x^*\\|_1$ for any ${x}\\in X^*$. Assuming this does \\emph{not} hold, then since problem \\eqref{eq:dd}  has  $x^*$ as a feasible solution and has a finite objective,  we have a nonempty $X^*$ and there exists $\\bar{x}\\in X^*$ satisfying $\\|\\bar{x}\\|_1 < \\|x^*\\|_1$. But,  $f(A\\bar{x}-b) = f(b^*-b)=f(Ax^*-b)$ and $\\|\\bar{x}\\|_1 < \\|x^*\\|_1$ mean that $\\bar{x}$ is a strictly better solution to problem \\eqref{eq:qpa} than $x^*$, contradicting the assumption $x^*\\in X_\\lambda$.\n","start":34253,"end":34967}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S066"],"declared_external":false,"source_only":false,"title":"两个解集相等","source_units":[{"raw_tex":"Now, we show that $X_\\lambda = X^*$. Since $Ax = Ax^*$ and $\\|x\\|_1 = \\|x^*\\|_1$ for all $x\\in X_\\lambda$ and conversely any $x$ obeying $Ax=Ax^*$ and $\\|x\\|_1=\\|x^*\\|_1$  belongs to $X_\\lambda$, it is suffices to show that $\\|{x}\\|_1 = \\|x^*\\|_1$ for any ${x}\\in X^*$. Assuming this does \\emph{not} hold, then since problem \\eqref{eq:dd}  has  $x^*$ as a feasible solution and has a finite objective,  we have a nonempty $X^*$ and there exists $\\bar{x}\\in X^*$ satisfying $\\|\\bar{x}\\|_1 < \\|x^*\\|_1$. But,  $f(A\\bar{x}-b) = f(b^*-b)=f(Ax^*-b)$ and $\\|\\bar{x}\\|_1 < \\|x^*\\|_1$ mean that $\\bar{x}$ is a strictly better solution to problem \\eqref{eq:qpa} than $x^*$, contradicting the assumption $x^*\\in X_\\lambda$.\n\nSince $X_\\lambda = X^*$,  $x^*$ is the unique solution to problem \\eqref{eq:qpa} if and only if it is the same to problem \\eqref{eq:dd}. Since problem \\eqref{eq:dd} is in the same form of problem \\eqref{eq:bp}, applying the part of Theorem \\ref{thm:unique} for problem \\eqref{eq:bp}, which is already proved, we conclude that  $x^*$ is the unique solution to problem \\eqref{eq:qpa} if and only if Condition \\ref{cond:unique} holds.\n","start":34253,"end":35400}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S067"],"declared_external":false,"source_only":false,"title":"转移唯一性并调用已证分支","source_units":[{"raw_tex":"Since $X_\\lambda = X^*$,  $x^*$ is the unique solution to problem \\eqref{eq:qpa} if and only if it is the same to problem \\eqref{eq:dd}. Since problem \\eqref{eq:dd} is in the same form of problem \\eqref{eq:bp}, applying the part of Theorem \\ref{thm:unique} for problem \\eqref{eq:bp}, which is already proved, we conclude that  $x^*$ is the unique solution to problem \\eqref{eq:qpa} if and only if Condition \\ref{cond:unique} holds.\n\\end{IEEEproof}","start":34968,"end":35415}]},{"id":"S068","semantic_type":"external_fact","kind":"external_fact","labels":["eq:pdpair"],"member_ids":["S068"],"declared_external":true,"declaration_text":["原始先修列表第1项；原文第417行引用 Dantzig [Da]，书目信息位于第621行。"],"source_only":false,"title":"线性规划强对偶与达到性","source_units":[{"raw_tex":"It remains to construct a vector $y$ for Condition \\ref{cond:unique}, part 2.\nOur construction is based on  the strong convexity relation between a linear program (called the primal problem) and its dual problem, namely, if one problem has a solution, so does the other, and the two solutions must give the same objective value. (For the interested reader, this result follows from the Hahn-Banach separation theorem, also from the theorem of alternatives \\cite{Da}. Alternatively, it can be  obtain constructively via the Simplex method; specifically, whenever a primal solution exists, the Simplex method terminates in a finite number of steps with a primal-dual solution pair.)\n","start":25055,"end":25736}]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S069"],"declared_external":true,"declaration_text":["原始先修列表第2项：有限维凸分析中的最小化解集凸性；作者第507行明确称为已知结果。"],"source_only":false,"title":"凸规划解集为凸集","source_units":[]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S070"],"declared_external":true,"declaration_text":["原始先修列表第2项：有限凸函数连续性；原文第513行用“convex and thus continuous”调用。"],"source_only":false,"title":"有限凸函数的连续性","source_units":[]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S071"],"declared_external":true,"declaration_text":["原始先修列表第2项：可行且强制的范数最小化达到最优值；作者第527行省略此标准依据，由标注者显式化。"],"source_only":false,"title":"可行范数最小化的达到性","source_units":[]},{"id":"S072","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S072"],"declared_external":true,"declaration_text":["原始先修列表第3项：标准范数对偶不等式；基本范数性质作为作者隐含使用的标准事实由标注者显式化。"],"source_only":false,"title":"范数对偶界与基本范数性质","source_units":[]},{"id":"S073","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S073"],"declared_external":true,"declaration_text":["原始先修列表第3项：初等线性代数；局部符号稳定及坐标等号性质为作者隐含使用的标准事实，由标注者显式化。"],"source_only":false,"title":"秩、局部符号与等号性质","source_units":[]},{"id":"S074","semantic_type":"assumption","kind":"assumption","labels":["lem01"],"member_ids":["S074"],"declared_external":false,"source_only":false,"title":"引理1的局部前提","source_units":[{"raw_tex":"Let $f$ be a strictly convex function. If $f(Ax-b)+\\|x\\|_1$ is constant on a convex set $S$, then both $Ax-b$ and $\\|x\\|_1$ are  constant on $S$.\n","start":30442,"end":30588}]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S014":"S014","S015":"S015","S016":"S016","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074"}}}}
{"record_id":"1912.12153v1","paper":{"title":"Accelerated iterative regularization via dual diagonal descent","authors":["Calatroni, Luca","Garrigos, Guillaume","Rosasco, Lorenzo","Villa, Silvia"],"arxiv_id":"1912.12153","version":"v1","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/1912.12153v1","pdf_url":"https://arxiv.org/pdf/1912.12153v1","doi":"10.1137/19M1308888"},"context":{"raw_tex":"We are interested in solving the linear inverse problem:\n\\begin{equation}   \\label{eq:inverse_problem}\n\\text{find }\\quad \\xtrue \\in \\Xx \\quad \\text{s.t.}\\quad A \\xtrue = \\ytrue,\n\\end{equation}\nwhere $A: \\mathcal{X}\\to\\mathcal{Y}$ is a bounded linear operator between two Hilbert spaces $\\mathcal{X}$ and $\\Yy$,\nand $ \\ytrue \\in \\mathcal{Y}$ can be seen as  a given  measurement of some unknown $\\xtrue \\in \\Xx$ we want to recover.\nIn general, the inverse problem \\eqref{eq:inverse_problem} is ill-posed as its solution (if it exists) may lack some fundamental properties like uniqueness or stability. \nA standard approach   is assuming the desired $\\xtrue$ to be well-approximated by\n \\cite{EngHanNeu96,BenBur18}:\n\\begin{equation}\\label{eq:primal}\n   \\tag*{\\textbf{($\\bm{P_0(\\ytrue)}$)}} \\text{find\\ }\n      \\xsol \\in  \\argmin  \\left\\{ R(x) \\text{ s.t. } x\\in\\underset{x' \\in \\Xx}{\\argmin}~ \\ell(Ax';  \\ytrue) \\right\\}, \n\\end{equation} \ncalled the primal problem in the following.  Here,  $R$ is a  regularization function enforcing some \\emph{a-priori} knowledge on  the desired solution $\\xtrue$, while $\\ell: \\Yy^2\\to\\R\\cup \\{+\\infty\\}$ is a data-fit function. In practice, the  data  is  subject to noise  due to, e.g., possible transmission and/or acquisition problems. As a consequence, we only have access to an inexact version $\\ynoisy$ of $\\ytrue$. \n\n\nWe begin fixing the  notation. Let $\\Hh$ be a Hilbert space with scalar product  $\\langle\\cdot,\\cdot\\rangle$ and associated norm $\\|\\cdot\\|$. Given $y\\in\\Hh$, and $\\varrho\\in\\mathbb{R}_+$ and $\\mathbb{B}(y,\\varrho)$ is the open ball of center $y$ and radius $\\varrho$.\nwe denote by $\\Gamma_0(\\Hh)$ the set of proper, convex and lower semi-continuous functions from $\\Hh$ to $]-\\infty,+\\infty]$.\nWe say that  $f \\in \\Gamma_0(\\Hh)$ is $\\sigma$-{\\em strongly convex}  if \n$f-{\\sigma}\\Vert \\cdot \\Vert^2/2 \\in \\Gamma_0(\\Hh)$, with  $\\sigma \\in\\left]0,+\\infty\\right[$.\nWe recall that the {\\em subdifferential} of  $f \\in \\Gamma_0(\\Hh)$  is the multi-valued operator \n$\\partial f : \\Hh \\rightarrow 2^{\\Hh}$ defined by setting\n\\begin{equation}  \\label{def:subdiff}\n(\\forall x\\in\\Hh)\\quad \\partial f(x) := \\left\\{u \\in \\Hh: f(x') - f(x) - \\langle u , x' - x \\rangle \\geq 0 \\text{ for all } x' \\in \\Hh \\right\\}.\n\\end{equation}\nNote that if $f$ is Gateaux differentiable at $x \\in \\Hh$, then $\\partial f(x)=\\{\\nabla f(x) \\}$, see, e.g.~, \\cite[Proposition 17.31 i)]{BauschkeCombettes2017}.\nWe also recall that for all  $x\\in \\Hh$ and $\\tau>0$ the proximity operator $\\prox_{\\tau f} : \\Hh \\rightarrow \\Hh$  of $f\\in\\Gamma_0(\\Hh)$ with parameter $\\tau$ is defined by:\n\\begin{equation}\\label{def:proximal operator}\n    \\prox_{\\tau f}(x) =\n    (I + \\partial f)^{-1}(x) = \n    \\argmin\\limits_{x' \\in \\Hh} \\left\\{ f(x') + \\frac{1}{2\\tau}\\Vert x' - x \\Vert^2 \\right\\}.\n\\end{equation}\nGiven $f \\in \\Gamma_0(\\Hh)$, we will denote by $f^* : \\Hh \\rightarrow [-\\infty,+\\infty]$   the {\\em Fenchel conjugate} of $f$\n\\[\n(\\forall u\\in\\Hh)\\quad f^*(u):=  \\sup_{x\\in \\Hh}  \\left\\{\\langle u,x\\rangle_{\\Hh} -f(x) \\right\\}.\n\\]\nThe Fenchel conjugate of $f$ belongs to $\\Gamma_0(\\Hh)$, and if $f$ is $\\sigma$-strongly convex then $f^*$ is differentiable at any point, with a $\\sigma^{-1}$-Lipschitz continuous gradient, see, e.g. \\cite[Theorem 18.15]{BauschkeCombettes2017}. \nFurthermore, the following property holds, see \\cite[Theorem 16.23]{BauschkeCombettes2017}: \n\\[\n(\\forall (x,u)\\in \\Hh^2)\\quad u \\in \\partial f(x) \\Leftrightarrow x \\in \\partial f^*(u).\n\\] \nFinally, given two real sequences $(a_k)_{k\\geq 1}$ and $(b_k)_{k\\geq 1}$, we will write $a_k = O(b_k)$ whenever there exists a positive constant $M>0$ such that $a_k \\leq M b_k$ for all $k \\geq 1$. We will use the more precise notation $a_k = \\Theta(b_k)$ if both conditions $a_k = O(b_k)$ and $b_k = O(a_k)$ hold.\nNote also that we will use the same notation  $\\|\\cdot\\|$ and $\\langle\\cdot,\\cdot\\rangle$ for the norm and the scalar product in every Hilbert space we consider. \n\n\nWe make   the following assumptions on the data-fit $\\ell$ and the regularizer $R$: \n\\begin{enumerate}[label=\\textbf{$(L_\\arabic*)$}]\n\t\\item \\label{assumption:data-fit convex and coercive}\n\tFor all $y \\in \\Yy$, $\\ell_y:=\\ell(\\cdot,y)\\in \\Gamma_0(\\Yy) $ and it is coercive.\n\t\\item \\label{assumption:data-fit not flat} \n\tFor all $(y_1,y_2)\\in \\Yy^2, ~\\ell(y_1,y_2)=0 \\iff y_1=y_2$.\n\t\\item \\label{assumption:data-fit conditionned locally}\n\t there exists\n     $q\\in[1,+\\infty[$, $\\varrho \\in ]0,+\\infty]$, $ \\gamma \\in ]0,+\\infty[$ such that\n\t$$\n\t\\forall y \\in \\mathbb{B}( \\ytrue,\\varrho),\\quad \\frac{\\gamma}{q}\\Vert y-\\ytrue\\Vert^q \\leq \\ell(y,\\ytrue). \n\t$$ \n\\end{enumerate}\n\\begin{enumerate}[label=\\textbf{$(R_\\arabic*)$}]\t\n\t\\item \\label{H:R strongly convex} $R$ is $\\sigma$-strongly convex, with $\\sigma \\in \\left]0,+\\infty\\right[$,  \n\t\\item \\label{assumption:AR2}   $\\partial R(\\xsol) \\cap \\im A^* \\neq \\emptyset$.\t\n\\end{enumerate}\nIf assumption \\ref{assumption:data-fit conditionned locally} is satisfied, we say that $\\ell_{\\ytrue}$ is \\emph{locally\n\t$q$-conditioned} at $\\bar{y}$. If \\ref{assumption:data-fit conditionned locally} is satisfied with $\\rho=+\\infty$, then we say that $\\ell_{\\ytrue}$  is $q$-conditioned at $\\bar{y}$.\n\n\nA class of methods to solve \\eqref{eq:inverse_problem}  consider the   problem \n\\begin{equation}   \\label{eq:primal_char}\n    \\text{find }  \\xsol \\in  \\argmin_{x \\in \\Xx}  \\left\\{ R(x) + \\delta_{\\bar{y}}(Ax) \\right\\},\n\\end{equation}\nwhere the constraint \\eqref{eq:inverse_problem} is encoded by the indicator function $\\delta_{\\bar{y}}$. Using Fenchel-Rockafeller duality,  \nwe derive the corresponding dual problem:\n\\begin{equation}   \\label{eq:Dual}\n\\text{find } u^\\dagger \\quad\\text{ s.t. }\\quad u^\\dagger \\in \\argmin_{u\\in \\mathcal{Y}}\\  \\left\\{ d_0(u) := R^*(-A^*u) + \\langle \\bar{y}, u \\rangle \\right\\}\n    \\tag*{\\textbf{($\\bm{D_0}$)}}.\n\\end{equation}\n\n\nThe dual problem of \\ref{eq:primal penalized} is\n\\begin{equation}   \\label{eq:DualReg}\n\\text{find } u_\\lambda \\in \\argmin_{u\\in \\mathcal{Y}}\\ \\left\\{d_\\lambda(u) := R^*(-A^*u) + \\frac{1}{\\lambda}\\ell^*(\\lambda u;\\ynoisy)\\right\\}.   \\tag*{\\textbf{($\\bm{D_\\lambda}$)}}\n\\end{equation} \n\n\n\\begin{equation}\\label{alg:fw_accel}\nu_0 = u_1 \\in \\Yy, \\quad \\text{ compute for } k \\geq 1 \\quad \n\\begin{cases}\n w_k & = u_k + \\alpha_k(u_k-u_{k-1}), \\\\\n u_{k+1} &= \\mathrm{prox}_{\\frac{\\tau}{\\lambda_{k}} \\ell^*_{\\bar{y}}(\\lambda_k\\cdot)}\\left(w_k - \\tau\\nabla \\RA(w_k)\\right),\\\\\n    x_{k+1} & = \\nabla R^*(-A^* u_{k+1}). \n\\end{cases}\n\\tag*{\\textbf{($\\bm{I3D}$)}}\n\\end{equation}\nThis algorithm depends on three parameters: the stepsize $\\tau >0$, the relaxation parameters $(\\lambda_k)_k$ and the friction parameters $(\\alpha_k)_k$.\nThe stepsize will be chosen depending on the value of the Lipschitz constant of $\\nabla \\RA$.\nFor the relaxation parameters, we will consider a discrete analogue of the assumption \\ref{requirement:lambda1:cont} formulated in the continuous setting.\nFor the friction parameters $\\alpha_k$, we will allow more general values not necessarily corresponding to $\\alpha_k = 1 -\\alpha/k$ as  above. We gather  the requirements on the   parameters in the following assumptions:\n\\begin{enumerate}[label=\\textbf{$(P_\\arabic*)$}]\n\t\\item \\label{assumption:parameter:stepsize}\n\t$\\tau \\in \\left(0,\\frac{\\sigma^2}{\\Vert A \\Vert^2}\\right]$, where $\\sigma > 0$ is defined in assumption \\ref{H:R strongly convex}.\n\t\\item \\label{assumption:parameter:friction}\n\t$\\alpha_k$ is non-negative and for every $k\\geq 1$, $t_k:=1+\\sum_{i=k}^{+\\infty}\\prod_{j=k}^i \\alpha_j$ is finite,  with $t_k = \\Theta(k)$.\n\t\\item \\label{assumption:parameter:relaxation}\n\t$(\\lambda_k)$ is a strictly positive non-increasing sequence such that $\\lim_{k\\to\\infty} \\lambda_k = 0$. \n\tMoreover, by defining\n\t\\begin{equation}   \\label{def:biglambda}\n\t\\Lambda:=\n\t\\begin{cases}\n\t\\sum_{k\\geq 1} t_{k+1} \\lambda_k^{1/(q-1)}\n\t& \\text{ if } q>1, \\\\\n\t0\n\t& \\text{ if } q=1, \n\t\\end{cases}\n\t\\end{equation}\n\twe have that $\\Lambda< + \\infty$.\n\t\\item \\label{assumption:parameter:technical}\n\tFor some $u^\\dagger \\in \\argmin d_0$, we have  $\\lambda_0 \\Vert u^\\dagger \\Vert \\leq \\gamma \\varrho^{q-1} /q$.\n\\end{enumerate}\n\n\n Following \\cite{AttouchTikAccel}, we require the sequence of friction parameters $(\\alpha_k)$ to satisfy  \\ref{assumption:parameter:friction},  a particular summability property  guaranteeing a technical condition crucial in the following. We summarize such a requirement and the resulting condition in the following lemma.\n\n\\begin{lemma}[ {{\\cite[Lemma 2.1]{AttouchTikAccel} }}]   \\label{lemma:sequence}\nAssume that $(\\alpha_k)$ is non-negative and satisfies\n\\begin{equation}  \\label{eq:choice:alpha}\n\\sum_{i=k}^{+\\infty}\\prod_{j=k}^i \\alpha_j<+\\infty,\\quad\\text{for every }k\\geq 1.\n\\end{equation}\nThen, the sequence defined by\n\\begin{equation} \\label{eq:choice:tk}\nt_k:=1+\\sum_{i=k}^{+\\infty}\\prod_{j=k}^i \\alpha_j\n\\end{equation}\nis well-defined, and satisfies for every $k\\geq 1$ the following properties:\n\\begin{equation}   \\label{eq:properties_sequences_alphak_tk}\n1+\\alpha_k t_{k+1} = t_k, \\qquad t^2_{k+1}-t^2_k\\leq t_{k+1}.\n\\end{equation}\n\\end{lemma}\n\n\nWe first consider $\\Rr_A $, $\\ell^*_y$ defined in \\eqref{eq:composite} and on the diagonal dual function $d_{\\lambda}$ and its limit $d_0$  defined in \\ref{eq:DualReg} and \\ref{eq:Dual}, respectively. For similar results see also \\cite{Garrigos2017}.\n","units":[{"raw_tex":"We are interested in solving the linear inverse problem:\n\\begin{equation}   \\label{eq:inverse_problem}\n\\text{find }\\quad \\xtrue \\in \\Xx \\quad \\text{s.t.}\\quad A \\xtrue = \\ytrue,\n\\end{equation}\nwhere $A: \\mathcal{X}\\to\\mathcal{Y}$ is a bounded linear operator between two Hilbert spaces $\\mathcal{X}$ and $\\Yy$,\nand $ \\ytrue \\in \\mathcal{Y}$ can be seen as  a given  measurement of some unknown $\\xtrue \\in \\Xx$ we want to recover.\nIn general, the inverse problem \\eqref{eq:inverse_problem} is ill-posed as its solution (if it exists) may lack some fundamental properties like uniqueness or stability. \nA standard approach   is assuming the desired $\\xtrue$ to be well-approximated by\n \\cite{EngHanNeu96,BenBur18}:\n\\begin{equation}\\label{eq:primal}\n   \\tag*{\\textbf{($\\bm{P_0(\\ytrue)}$)}} \\text{find\\ }\n      \\xsol \\in  \\argmin  \\left\\{ R(x) \\text{ s.t. } x\\in\\underset{x' \\in \\Xx}{\\argmin}~ \\ell(Ax';  \\ytrue) \\right\\}, \n\\end{equation} \ncalled the primal problem in the following.  Here,  $R$ is a  regularization function enforcing some \\emph{a-priori} knowledge on  the desired solution $\\xtrue$, while $\\ell: \\Yy^2\\to\\R\\cup \\{+\\infty\\}$ is a data-fit function. In practice, the  data  is  subject to noise  due to, e.g., possible transmission and/or acquisition problems. As a consequence, we only have access to an inexact version $\\ynoisy$ of $\\ytrue$. \n","start":20669,"end":22028},{"raw_tex":"We begin fixing the  notation. Let $\\Hh$ be a Hilbert space with scalar product  $\\langle\\cdot,\\cdot\\rangle$ and associated norm $\\|\\cdot\\|$. Given $y\\in\\Hh$, and $\\varrho\\in\\mathbb{R}_+$ and $\\mathbb{B}(y,\\varrho)$ is the open ball of center $y$ and radius $\\varrho$.\nwe denote by $\\Gamma_0(\\Hh)$ the set of proper, convex and lower semi-continuous functions from $\\Hh$ to $]-\\infty,+\\infty]$.\nWe say that  $f \\in \\Gamma_0(\\Hh)$ is $\\sigma$-{\\em strongly convex}  if \n$f-{\\sigma}\\Vert \\cdot \\Vert^2/2 \\in \\Gamma_0(\\Hh)$, with  $\\sigma \\in\\left]0,+\\infty\\right[$.\nWe recall that the {\\em subdifferential} of  $f \\in \\Gamma_0(\\Hh)$  is the multi-valued operator \n$\\partial f : \\Hh \\rightarrow 2^{\\Hh}$ defined by setting\n\\begin{equation}  \\label{def:subdiff}\n(\\forall x\\in\\Hh)\\quad \\partial f(x) := \\left\\{u \\in \\Hh: f(x') - f(x) - \\langle u , x' - x \\rangle \\geq 0 \\text{ for all } x' \\in \\Hh \\right\\}.\n\\end{equation}\nNote that if $f$ is Gateaux differentiable at $x \\in \\Hh$, then $\\partial f(x)=\\{\\nabla f(x) \\}$, see, e.g.~, \\cite[Proposition 17.31 i)]{BauschkeCombettes2017}.\nWe also recall that for all  $x\\in \\Hh$ and $\\tau>0$ the proximity operator $\\prox_{\\tau f} : \\Hh \\rightarrow \\Hh$  of $f\\in\\Gamma_0(\\Hh)$ with parameter $\\tau$ is defined by:\n\\begin{equation}\\label{def:proximal operator}\n    \\prox_{\\tau f}(x) =\n    (I + \\partial f)^{-1}(x) = \n    \\argmin\\limits_{x' \\in \\Hh} \\left\\{ f(x') + \\frac{1}{2\\tau}\\Vert x' - x \\Vert^2 \\right\\}.\n\\end{equation}\nGiven $f \\in \\Gamma_0(\\Hh)$, we will denote by $f^* : \\Hh \\rightarrow [-\\infty,+\\infty]$   the {\\em Fenchel conjugate} of $f$\n\\[\n(\\forall u\\in\\Hh)\\quad f^*(u):=  \\sup_{x\\in \\Hh}  \\left\\{\\langle u,x\\rangle_{\\Hh} -f(x) \\right\\}.\n\\]\nThe Fenchel conjugate of $f$ belongs to $\\Gamma_0(\\Hh)$, and if $f$ is $\\sigma$-strongly convex then $f^*$ is differentiable at any point, with a $\\sigma^{-1}$-Lipschitz continuous gradient, see, e.g. \\cite[Theorem 18.15]{BauschkeCombettes2017}. \nFurthermore, the following property holds, see \\cite[Theorem 16.23]{BauschkeCombettes2017}: \n\\[\n(\\forall (x,u)\\in \\Hh^2)\\quad u \\in \\partial f(x) \\Leftrightarrow x \\in \\partial f^*(u).\n\\] \nFinally, given two real sequences $(a_k)_{k\\geq 1}$ and $(b_k)_{k\\geq 1}$, we will write $a_k = O(b_k)$ whenever there exists a positive constant $M>0$ such that $a_k \\leq M b_k$ for all $k \\geq 1$. We will use the more precise notation $a_k = \\Theta(b_k)$ if both conditions $a_k = O(b_k)$ and $b_k = O(a_k)$ hold.\nNote also that we will use the same notation  $\\|\\cdot\\|$ and $\\langle\\cdot,\\cdot\\rangle$ for the norm and the scalar product in every Hilbert space we consider. \n","start":28119,"end":30731},{"raw_tex":"We make   the following assumptions on the data-fit $\\ell$ and the regularizer $R$: \n\\begin{enumerate}[label=\\textbf{$(L_\\arabic*)$}]\n\t\\item \\label{assumption:data-fit convex and coercive}\n\tFor all $y \\in \\Yy$, $\\ell_y:=\\ell(\\cdot,y)\\in \\Gamma_0(\\Yy) $ and it is coercive.\n\t\\item \\label{assumption:data-fit not flat} \n\tFor all $(y_1,y_2)\\in \\Yy^2, ~\\ell(y_1,y_2)=0 \\iff y_1=y_2$.\n\t\\item \\label{assumption:data-fit conditionned locally}\n\t there exists\n     $q\\in[1,+\\infty[$, $\\varrho \\in ]0,+\\infty]$, $ \\gamma \\in ]0,+\\infty[$ such that\n\t$$\n\t\\forall y \\in \\mathbb{B}( \\ytrue,\\varrho),\\quad \\frac{\\gamma}{q}\\Vert y-\\ytrue\\Vert^q \\leq \\ell(y,\\ytrue). \n\t$$ \n\\end{enumerate}\n\\begin{enumerate}[label=\\textbf{$(R_\\arabic*)$}]\t\n\t\\item \\label{H:R strongly convex} $R$ is $\\sigma$-strongly convex, with $\\sigma \\in \\left]0,+\\infty\\right[$,  \n\t\\item \\label{assumption:AR2}   $\\partial R(\\xsol) \\cap \\im A^* \\neq \\emptyset$.\t\n\\end{enumerate}\nIf assumption \\ref{assumption:data-fit conditionned locally} is satisfied, we say that $\\ell_{\\ytrue}$ is \\emph{locally\n\t$q$-conditioned} at $\\bar{y}$. If \\ref{assumption:data-fit conditionned locally} is satisfied with $\\rho=+\\infty$, then we say that $\\ell_{\\ytrue}$  is $q$-conditioned at $\\bar{y}$.\n","start":30762,"end":31997},{"raw_tex":"A class of methods to solve \\eqref{eq:inverse_problem}  consider the   problem \n\\begin{equation}   \\label{eq:primal_char}\n    \\text{find }  \\xsol \\in  \\argmin_{x \\in \\Xx}  \\left\\{ R(x) + \\delta_{\\bar{y}}(Ax) \\right\\},\n\\end{equation}\nwhere the constraint \\eqref{eq:inverse_problem} is encoded by the indicator function $\\delta_{\\bar{y}}$. Using Fenchel-Rockafeller duality,  \nwe derive the corresponding dual problem:\n\\begin{equation}   \\label{eq:Dual}\n\\text{find } u^\\dagger \\quad\\text{ s.t. }\\quad u^\\dagger \\in \\argmin_{u\\in \\mathcal{Y}}\\  \\left\\{ d_0(u) := R^*(-A^*u) + \\langle \\bar{y}, u \\rangle \\right\\}\n    \\tag*{\\textbf{($\\bm{D_0}$)}}.\n\\end{equation}\n","start":36263,"end":36921},{"raw_tex":"The dual problem of \\ref{eq:primal penalized} is\n\\begin{equation}   \\label{eq:DualReg}\n\\text{find } u_\\lambda \\in \\argmin_{u\\in \\mathcal{Y}}\\ \\left\\{d_\\lambda(u) := R^*(-A^*u) + \\frac{1}{\\lambda}\\ell^*(\\lambda u;\\ynoisy)\\right\\}.   \\tag*{\\textbf{($\\bm{D_\\lambda}$)}}\n\\end{equation} \n","start":39699,"end":39982},{"raw_tex":"\\begin{equation}\\label{alg:fw_accel}\nu_0 = u_1 \\in \\Yy, \\quad \\text{ compute for } k \\geq 1 \\quad \n\\begin{cases}\n w_k & = u_k + \\alpha_k(u_k-u_{k-1}), \\\\\n u_{k+1} &= \\mathrm{prox}_{\\frac{\\tau}{\\lambda_{k}} \\ell^*_{\\bar{y}}(\\lambda_k\\cdot)}\\left(w_k - \\tau\\nabla \\RA(w_k)\\right),\\\\\n    x_{k+1} & = \\nabla R^*(-A^* u_{k+1}). \n\\end{cases}\n\\tag*{\\textbf{($\\bm{I3D}$)}}\n\\end{equation}\nThis algorithm depends on three parameters: the stepsize $\\tau >0$, the relaxation parameters $(\\lambda_k)_k$ and the friction parameters $(\\alpha_k)_k$.\nThe stepsize will be chosen depending on the value of the Lipschitz constant of $\\nabla \\RA$.\nFor the relaxation parameters, we will consider a discrete analogue of the assumption \\ref{requirement:lambda1:cont} formulated in the continuous setting.\nFor the friction parameters $\\alpha_k$, we will allow more general values not necessarily corresponding to $\\alpha_k = 1 -\\alpha/k$ as  above. We gather  the requirements on the   parameters in the following assumptions:\n\\begin{enumerate}[label=\\textbf{$(P_\\arabic*)$}]\n\t\\item \\label{assumption:parameter:stepsize}\n\t$\\tau \\in \\left(0,\\frac{\\sigma^2}{\\Vert A \\Vert^2}\\right]$, where $\\sigma > 0$ is defined in assumption \\ref{H:R strongly convex}.\n\t\\item \\label{assumption:parameter:friction}\n\t$\\alpha_k$ is non-negative and for every $k\\geq 1$, $t_k:=1+\\sum_{i=k}^{+\\infty}\\prod_{j=k}^i \\alpha_j$ is finite,  with $t_k = \\Theta(k)$.\n\t\\item \\label{assumption:parameter:relaxation}\n\t$(\\lambda_k)$ is a strictly positive non-increasing sequence such that $\\lim_{k\\to\\infty} \\lambda_k = 0$. \n\tMoreover, by defining\n\t\\begin{equation}   \\label{def:biglambda}\n\t\\Lambda:=\n\t\\begin{cases}\n\t\\sum_{k\\geq 1} t_{k+1} \\lambda_k^{1/(q-1)}\n\t& \\text{ if } q>1, \\\\\n\t0\n\t& \\text{ if } q=1, \n\t\\end{cases}\n\t\\end{equation}\n\twe have that $\\Lambda< + \\infty$.\n\t\\item \\label{assumption:parameter:technical}\n\tFor some $u^\\dagger \\in \\argmin d_0$, we have  $\\lambda_0 \\Vert u^\\dagger \\Vert \\leq \\gamma \\varrho^{q-1} /q$.\n\\end{enumerate}\n","start":56423,"end":58414},{"raw_tex":" Following \\cite{AttouchTikAccel}, we require the sequence of friction parameters $(\\alpha_k)$ to satisfy  \\ref{assumption:parameter:friction},  a particular summability property  guaranteeing a technical condition crucial in the following. We summarize such a requirement and the resulting condition in the following lemma.\n\n\\begin{lemma}[ {{\\cite[Lemma 2.1]{AttouchTikAccel} }}]   \\label{lemma:sequence}\nAssume that $(\\alpha_k)$ is non-negative and satisfies\n\\begin{equation}  \\label{eq:choice:alpha}\n\\sum_{i=k}^{+\\infty}\\prod_{j=k}^i \\alpha_j<+\\infty,\\quad\\text{for every }k\\geq 1.\n\\end{equation}\nThen, the sequence defined by\n\\begin{equation} \\label{eq:choice:tk}\nt_k:=1+\\sum_{i=k}^{+\\infty}\\prod_{j=k}^i \\alpha_j\n\\end{equation}\nis well-defined, and satisfies for every $k\\geq 1$ the following properties:\n\\begin{equation}   \\label{eq:properties_sequences_alphak_tk}\n1+\\alpha_k t_{k+1} = t_k, \\qquad t^2_{k+1}-t^2_k\\leq t_{k+1}.\n\\end{equation}\n\\end{lemma}\n","start":59791,"end":60751},{"raw_tex":"We first consider $\\Rr_A $, $\\ell^*_y$ defined in \\eqref{eq:composite} and on the diagonal dual function $d_{\\lambda}$ and its limit $d_0$  defined in \\ref{eq:DualReg} and \\ref{eq:Dual}, respectively. For similar results see also \\cite{Garrigos2017}.\n","start":114878,"end":115129}]},"statement":{"raw_tex":"\\begin{theorem}[Fast convergence]\\label{T:discrete convergence rates noiseless}\nLet the assumptions \\ref{assumption:data-fit convex and coercive}-\\ref{assumption:data-fit conditionned locally}, \\ref{H:R strongly convex}-\\ref{assumption:AR2}, \\ref{assumption:parameter:stepsize}-\\ref{assumption:parameter:technical} hold true.\nLet $(x_k)$ and $(u_k)$ be the sequences generated by algorithm \\ref{alg:fw_accel}.\nThen, there exists $C\\in\\left]0,+\\infty\\right[$ such that\n\\begin{equation}  \\label{eq:rate}%\\label{eq:rate:primal:it}\n d_{{\\lambda_k}}(u_k) - \\inf  d_0  \\leq \\frac{C}{t_k^2} \n \\quad \\text{ and } \\quad \n \\| x_k - x^\\dagger \\| \\leq \\frac{\\sqrt{2C}}{\\sqrt{\\sigma}t_k}.\n\\end{equation}\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}[Fast convergence]\\label{T:discrete convergence rates noiseless}\nLet the assumptions \\ref{assumption:data-fit convex and coercive}-\\ref{assumption:data-fit conditionned locally}, \\ref{H:R strongly convex}-\\ref{assumption:AR2}, \\ref{assumption:parameter:stepsize}-\\ref{assumption:parameter:technical} hold true.\nLet $(x_k)$ and $(u_k)$ be the sequences generated by algorithm \\ref{alg:fw_accel}.\nThen, there exists $C\\in\\left]0,+\\infty\\right[$ such that\n\\begin{equation}  \\label{eq:rate}%\\label{eq:rate:primal:it}\n d_{{\\lambda_k}}(u_k) - \\inf  d_0  \\leq \\frac{C}{t_k^2} \n \\quad \\text{ and } \\quad \n \\| x_k - x^\\dagger \\| \\leq \\frac{\\sqrt{2C}}{\\sqrt{\\sigma}t_k}.\n\\end{equation}\n\\end{theorem}","start":62973,"end":63677}]},"proof":{"raw_tex":"\\begin{proof}\nLet $u^\\dagger\\in \\argmin d_0$ be the minimizer of $d_0$ defined in assumption \\ref{assumption:parameter:technical}, and\ndefine, for every $k\\geq 1$,  the discrete Lyapunov energy function:\n\\begin{equation}  \\label{eq:discrete energy}\n\\mathcal{E}(k):=t_k^2 \\Big( d_{{\\lambda_k}}(u_k)-\\inf d_0\\Big) + \\frac{1}{2\\tau}\\|z_k-u^\\dagger\\|^2,\n\\end{equation}\nwhere $z_k$ is defined as:\n\\begin{equation}  \\label{eq:def_zk:noisefree}\nz_k:=u_{k-1} +t_k(u_k-u_{k-1}).\n\\end{equation}\nOur goal is to get an estimate on the decay of $\\mathcal{E}$ along time.\nMore precisely, we will show that for every $k\\geq 1$\n\\begin{equation}  \\label{eq:condition}\n\\mathcal{E}(k+1) - \\mathcal{E}(k)\n\\leq  t_{k+1} \\Big(d_{{\\lambda_k}}(u^\\dagger) -  \\inf d_0\\Big),\n\\end{equation}\nwhich can be seen as a discrete analogue of \\eqref{proof:ccd2}, and from which the desired accelerated convergence rates will follow in a straightforward manner.\n\nFor simplicity, let us denote by $\\ell_k$  the function defined by setting\n\\begin{equation}  \\label{eq:shortand_notation_Dk}\n(\\forall u\\in \\mathcal{Y})\\quad \\ell_k(u)=\\lambda_k^{-1}{\\ell}_{\\bar{y}}^*(\\lambda_k u).\n\\end{equation} \nTo prove \\eqref{eq:condition}, we define for every $k\\geq 1$ the operator $G_k:\\mathcal{Y}\\to \\mathcal{Y}$ as\n\\begin{equation}  \\label{eq:fw_operator}\nG_k(z)  := \\frac{1}{\\tau}\\Big(z-\\text{prox}_{\\tau \\ell_k}(z-\\tau \\nabla  \\RA(z))\\Big)\n\\end{equation}\nand notice that the proximal step of  \\ref{alg:fw_accel} can be written in terms of $G_k$ as $u_{k+1}=w_k - \\tau G_k(w_k)$. The descent lemma yields (see, e.g., \\cite{AttouchTikAccel,ChambolleDossal2015})\n\\begin{equation}  \\label{eq:descent}\n    d_{\\lambda_k}(w-\\tau G_k(w)) \\leq  d_{\\lambda_k}(u) + \\langle G_k(w), w-u\\rangle - \\frac{\\tau}{2}\\|G_k(w)\\|^2, \\quad \\text{for all }w,u\\in\\mathcal{Y},\n\\end{equation}\nEvaluating \\eqref{eq:descent} for $u=u_{k}$ and $w=w_k$, we get\n\\begin{equation}  \\label{descent:first}\nd_{{\\lambda_k}}(u_{k+1}) \\leq  d_{{\\lambda_k}}(u_{k}) + \\langle G_k(w_k), w_k - u_{k}\\rangle - \\frac{\\tau}{2}\\|G_k(w_k)\\|^2.\n\\end{equation}\nEvaluating \\eqref{eq:descent} for $u=u^\\dagger$ and $w=w_k$ we derive\n \\begin{equation}  \\label{descent:second}\nd_{{\\lambda_k}}(u_{k+1}) \\leq  d_{{\\lambda_k}}(u^\\dagger) + \\langle G_k(w_k), w_k -u^\\dagger\\rangle - \\frac{\\tau}{2}\\|G_k(w_k)\\|^2.\n\\end{equation}\nWe now multiply \\eqref{descent:first} by $t_{k+1}-1$ and we add it to \\eqref{descent:second}, and we obtain\n\\begin{align}\nt_{k+1} d_{{\\lambda_k}}(u_{k+1}) \n&\\leq  (t_{k+1}-1)d_{{\\lambda_k}}(u_{k}) +  d_{{\\lambda_k}}(u^\\dagger)  \\nonumber\\\\\n\\label{ineq:1}&+ \\langle  G_k(w_k), (t_{k+1}-1)(w_k-u_{k}) + (w_{k} - u^\\dagger)\\rangle  -\\frac{\\tau}{2}t_{k+1}\\|  G_k(w_k)\\|^2 . \n\\end{align}\nAs an immediate consequence of Lemma \\ref{lemma:sequence}, we observe the following property:\n\\begin{align}  \\label{eq:property_zk}\n(t_{k+1}-1)(w_k-u_{k}) + w_k & = u_{k} + t_{k+1}(w_k -u_{k}) \\notag\\\\\n& = u_{k} + t_{k+1} \\alpha_k(u_{k} -u_{k-1}) \\\\\n& = u_{k-1} + (1+ t_{k+1} \\alpha_k)(u_{k} -u_{k-1}) \\notag\\\\\n& = u_{k-1} +t_k(u_{k} -u_{k-1}) = z_k. \\notag\n\\end{align}\nThanks to \\eqref{eq:def_zk:noisefree}, the fact that $z_k-\\tau t_{k+1}G_k(w_k)=z_{k+1}$ and the previous equality, we can reorder the terms in \\eqref{ineq:1} and rewrite it as\n\\begin{align}\nt_{k+1} ( d_{{\\lambda_k}}(u_{k+1}) -  d_{{\\lambda_k}}(u^\\dagger)) \n& \\leq  (t_{k+1}-1) (d_{{\\lambda_k}}(u_{k}) -  d_{{\\lambda_k}}(u^\\dagger)) \\notag\\\\\n& + \\frac{1}{2\\tau t_{k+1}}\\left(\\|z_{k}-u^\\dagger\\|^2-\\|z_{k+1}-u^\\dagger\\|^2\\right). \\notag \n\\end{align}\nWe now multiply everything by $t_{k+1}$, re-arrange and get\n\\begin{align}  \\label{descent:third}\n& t^2_{k+1} ( d_{{\\lambda_k}}(u_{k+1}) -  d_{{\\lambda_k}}(u^\\dagger))   + \\frac{1}{2\\tau}\n\\|z_{k+1}-u^\\dagger\\|^2   \\\\\n&\\leq  (t^2_{k+1}-t_{k+1}) (d_{{\\lambda_k}}(u_{k}) -  d_{\\lambda_k}(u^\\dagger)) + \\frac{1}{2\\tau } \\|z_{k}-u^\\dagger\\|^2.  \\notag\n\\end{align}\nEquivalently:\n\\begin{align}  \\label{descent:third2}\n& t^2_{k+1} \\Big( d_{{\\lambda_k}}(u_{k+1}) -  d_{{\\lambda_k}}(u^\\dagger) \\Big)   + \\frac{1}{2\\tau}\n\\|z_{k+1}-u^\\dagger\\|^2 \\nonumber  \\\\\n&\\leq  t^2_k \\Big(d_{{\\lambda_k}}(u_{k}) -  d_{{\\lambda_k}}(u^\\dagger)\\Big) +   (t^2_{k+1}-t_{k+1} - t_{k}^2) \\Big( d_{{\\lambda_k}}(u_{k}) -  d_{{\\lambda_k}}(u^\\dagger) \\Big)+ \\frac{1}{2\\tau } \\|z_{k}-u^\\dagger\\|^2. \\notag\n\\end{align}\nTo get the desired terms, we first use on the left-hand side  the monotonicity property of the function $ d_{{\\lambda_k}}(\\cdot)$ as a function of $k$ (see  Lemma  \\ref{L:functional properties}\\ref{L:functional properties:decreasing function sequence}) and then add and subtract in the parentheses  the term $\\inf d_0$, thus getting:\n\\begin{align}  \\label{descent:fifth}\n&t^2_{k+1} \\Big( d_{{\\lambda_{k+1}}}(u_{k+1}) -   \\inf d_0 \\Big)   + \\frac{1}{2\\tau}\n\\|z_{k+1}-u^\\dagger\\|^2   \\notag \\\\\n&\\leq  t^2_k \\Big(d_{{\\lambda_{k}}}(u_{k}) - \\inf d_0 \\Big) +   (t^2_{k+1}-t_{k+1} - t_{k}^2) \\Big( d_{{\\lambda_{k}}}(u_{k}) -  \\inf d_0\\Big) \\\\\n& +t_{k+1} \\Big(d_{{\\lambda_{k}}}(u^\\dagger) -  \\inf d_0\\Big)   +  \\frac{1}{2\\tau } \\|z_{k}-u^\\dagger\\|^2. \\notag\n\\end{align}\nAfter rearranging and using the definition of $\\mathcal{E}$ in \\eqref{eq:discrete energy}, we deduce:\n\\begin{align}  \\label{descent:fifth}\n\\mathcal{E}(k+1) +    (t^2_{k}+t_{k+1} - t_{k+1}^2) \\Big( d_{{\\lambda_{k}}}(u_{k}) -  \\inf d_0\\Big) \\leq  \\mathcal{E}(k) + t_{k+1} \\Big(d_{{\\lambda_{k}}}(u^\\dagger) -  \\inf d_0\\Big). \\nonumber\n\\end{align}\n\n\\noindent Thanks to \\eqref{eq:properties_sequences_alphak_tk} and  Lemma~\\ref{L:functional properties}\\ref{L:functional properties:decreasing function sequence}, we can neglect the second term  in the left-hand side of the above inequality,  finally getting the desired  \\eqref{eq:condition}.\nIterating \\eqref{eq:condition}  recursively gives\n\\begin{align}  \\label{eq:decay_complete}\n\\mathcal{E}(k) \\leq \\mathcal{E}(1) +\\sum_{j=1}^{k-1}t_{j+1}\\Big( d_{\\lambda_{j}}(u^\\dagger)-\\inf d_0 \\Big).\n\\end{align}\nTo  bound the sum appearing on the right hand side, we need to analyze the residuals $r_j:= d_{\\lambda_{j}}(u^\\dagger) -\\inf d_0$. Similarly as for the estimation obtained in the continuous case, we can use for this purpose the property in Lemma \\ref{L:functional properties}\\ref{L:functional properties:diagonal rates via conditioning} and get that for some fixed constant $c$ independent on $j$ (defined analogously as in \\eqref{proof:ccd3}), we have\n$$\nr_j\\leq c \\lambda_j^{\\frac{1}{q-1}},\\qquad\\text{for every }j\\geq 1.\n$$\nBy assumption \\ref{assumption:parameter:relaxation}, with $\\Lambda$ as  in \\eqref{def:biglambda}, we thus conclude that\n$$\n\\sum_{j=1}^{k-1} t_{j+1} r_j \\leq  c \\sum_{j=1}^{k-1} t_{j+1} \\lambda_j^{\\frac{1}{q-1}}  \\leq c \\Lambda < +\\infty\n$$\nThis allows us to deduce from \\eqref{eq:decay_complete} the convergence rate on the dual values in \\eqref{eq:rate}, by taking $C = \\Ee(1) + c \\Lambda$.\nFinally, the convergence rate on the primal iterates  in \\eqref{eq:rate} follows from  Lemma \\ref{L:functional properties}\\ref{L:functional properties:from dual value to primal iterates}.\n\\end{proof}\n\n\\begin{lemma}\\label{L:functional properties}\nUnder the assumptions \\ref{assumption:data-fit convex and coercive}-\\ref{assumption:data-fit conditionned locally} and \\ref{H:R strongly convex}-\\ref{assumption:AR2},  we have:\n\\begin{enumerate}[label=(\\roman*)]\n\t\\item\\label{L:functional properties:R_A Lipschitz} $\\Rr_A$ is differentiable and $\\nabla  \\RA$ is Lipschitz continuous, with Lipschitz constant equal to $\\sigma^{-1} \\Vert A \\Vert^2$.\n\t\\item\\label{L:functional properties:L at 0} $\\forall y \\in \\Yy$, $\\ell^*_{y}(0)=0$ and $\\partial \\ell^*_y(0)=\\{y\\}$.\n\t\\item\\label{L:functional properties:argmin d_0 nonempty} There holds: $\\argmin  d_0 \\neq \\emptyset$.\n \t\\item\\label{L:functional properties:decreasing function sequence} $\\forall u \\in \\Yy$, the function $\\lambda\\in [0,+\\infty) \\mapsto \\ d_{\\lambda}(u)$ is nondecreasing. \n\t\\item\\label{L:functional properties:from dual value to primal iterates} $\\forall t >0, \\forall u \\in \\Yy, \\ \\text{ if }\n{x}:=\\nabla R^*(-A^*u),$ then $\n \\frac{\\sigma}{2} \\Vert {x} - x^\\dagger \\Vert^2 \\leq d_0(u) - \\inf d_0.$ \n \t\\item\\label{L:functional properties:diagonal rates via conditioning} $\\forall u^\\dagger \\in \\argmin  d_0$, if  $\\lambda \\Vert u^\\dagger \\Vert \\leq \\frac{\\gamma}{q} \\varrho^{q-1}$, then\n \t\\begin{equation}\n \t    d_{\\lambda}(u^\\dagger) - \\inf  d_0 \\leq\n \t    \\begin{cases}\n \t     0 & \\text{ if } q=1, \\\\\n \t     (1-\\frac{1}{q}) {\\gamma^{-1/(q-1)}} \\Vert u^\\dagger \\Vert^{q/(q-1)} \\lambda^{1/(q-1)} & \\text{ if } q>1.\n \t    \\end{cases}\n \t\\end{equation}\n\\end{enumerate}\n\\end{lemma}\n\n\\begin{proof}\n\\ref{L:functional properties:R_A Lipschitz}: follows  from the strong convexity of $R$, see, e.g.,  \n\\cite[Theorem 18.15]{BauschkeCombettes2017}.\\\\\n\\ref{L:functional properties:L at 0}: it is a simple consequence of the properties of the Fenchel transform as it can be found, e.g., in \n \\cite[Proposition 13.10(i) \\& Corollary 16.30]{BauschkeCombettes2017}.\\\\\n\\ref{L:functional properties:argmin d_0 nonempty} and \\ref{L:functional properties:from dual value to primal iterates}: follow from \\cite[Lemma 5]{Garrigos2017} by simply taking $f=R$ and $g=\\delta_{\\{\\bar y\\}}$, while property \\ref{L:functional properties:decreasing function sequence} has been proved in \\cite[Proposition 2(i)]{Garrigos2017}.\\\\\n\\ref{L:functional properties:diagonal rates via conditioning}: it is enough to verify that $\\ell_{\\bar y}(\\cdot)$ is $q$-well conditioned in the sense of \\cite[Definition 1]{Garrigos2017}, while assumption \\ref{assumption:data-fit conditionned locally} holds only locally.\nTo check this, we introduce the function $\\psi : \\R \\rightarrow \\R$ defined for the $\\varrho>0$ appearing in \\ref{assumption:data-fit conditionned locally}  by:\n\\begin{equation}\n \\psi  t \\mapsto\n\\begin{cases}\n \\frac{\\gamma}{q}\\vert t \\vert^q  & \\text{ if } \\vert t \\vert \\leq \\varrho, \\\\\n \\frac{\\gamma}{q} \\varrho^{q-1} \\vert t \\vert & \\text{ if } \\vert t \\vert > \\varrho.\n\\end{cases}\n\\end{equation}\nFrom \\ref{assumption:data-fit conditionned locally}, we easily  deduce that $\\ell_{\\bar y}(y) \\geq \\psi(\\Vert y- \\bar y \\Vert)$ for all $y \\in \\Yy$ (see \\cite[Corollary 3.4.2]{ZalinescuBook2002}).\nNote that $\\psi$ is not convex for $q >1$, so in this case we consider instead the function\n\\begin{equation}\n m: \\R \\rightarrow \\R , \\quad t \\mapsto \n\\begin{cases}\n \\frac{\\gamma}{q}\\vert t \\vert^q  & \\text{ if } \\vert t \\vert \\leq q^{1/(1-q)} \\varrho, \\\\\n \\frac{\\gamma}{q}\\varrho^{q-1}\\vert t \\vert \n -  \\frac{\\gamma}{q^{\\frac{q}{q-1}}}\\varrho^q (1 - \\frac{1}{q})  & \\text{ if } \\vert t \\vert > q^{1/(1-q)} \\varrho,\n\\end{cases}\n\\end{equation}\nand define $m:=\\psi$ for $q=1$.\nIt is an easy exercise to verify that $m$ is indeed a convex function on $\\R$, and that $m(w) \\leq \\psi(w)$ for all $w\\in\\R$.\nNow, we can make use of \\cite[Lemma 2]{Garrigos2017},\nwhich tells us that $d_\\lambda(u) - \\inf d_0 \\leq \\lambda^{-1} m^*(\\Vert u \\Vert \\lambda)$.\nThe desired result  now follows from the computation of the Fenchel transform of $m$.\nIf $q=1$, we have that $m(t) = \\gamma \\vert t \\vert$, so classic Fenchel calculus entails that $m^*$ is $\\delta_{[-\\gamma,\\gamma]}$, the indicator function of $[-\\gamma,\\gamma]$.\nIf $q>1$, easy computations show that $m^*$ reads\n\\begin{equation}\n m^* : \\R \\rightarrow \\R, \\quad s \\mapsto\n\\begin{cases}\n (1 - \\frac{1}{q}) \\gamma^{-1/(q-1)}\\vert s \\vert^{\\frac{q}{q-1}}  & \\text{ if } \\vert s \\vert \\leq \\frac{\\gamma}{q} \\varrho^{q-1}, \\\\\n + \\infty  & \\text{ if } \\vert s \\vert > \\frac{\\gamma}{q} \\varrho^{q-1}.\n\\end{cases}\n\\end{equation}\nBy now applying \\cite[Lemma 2]{Garrigos2017} we  conclude. \n\\end{proof}","units":[{"raw_tex":"\\begin{proof}\nLet $u^\\dagger\\in \\argmin d_0$ be the minimizer of $d_0$ defined in assumption \\ref{assumption:parameter:technical}, and\ndefine, for every $k\\geq 1$,  the discrete Lyapunov energy function:\n\\begin{equation}  \\label{eq:discrete energy}\n\\mathcal{E}(k):=t_k^2 \\Big( d_{{\\lambda_k}}(u_k)-\\inf d_0\\Big) + \\frac{1}{2\\tau}\\|z_k-u^\\dagger\\|^2,\n\\end{equation}\nwhere $z_k$ is defined as:\n\\begin{equation}  \\label{eq:def_zk:noisefree}\nz_k:=u_{k-1} +t_k(u_k-u_{k-1}).\n\\end{equation}\nOur goal is to get an estimate on the decay of $\\mathcal{E}$ along time.\nMore precisely, we will show that for every $k\\geq 1$\n\\begin{equation}  \\label{eq:condition}\n\\mathcal{E}(k+1) - \\mathcal{E}(k)\n\\leq  t_{k+1} \\Big(d_{{\\lambda_k}}(u^\\dagger) -  \\inf d_0\\Big),\n\\end{equation}\nwhich can be seen as a discrete analogue of \\eqref{proof:ccd2}, and from which the desired accelerated convergence rates will follow in a straightforward manner.\n\nFor simplicity, let us denote by $\\ell_k$  the function defined by setting\n\\begin{equation}  \\label{eq:shortand_notation_Dk}\n(\\forall u\\in \\mathcal{Y})\\quad \\ell_k(u)=\\lambda_k^{-1}{\\ell}_{\\bar{y}}^*(\\lambda_k u).\n\\end{equation} \nTo prove \\eqref{eq:condition}, we define for every $k\\geq 1$ the operator $G_k:\\mathcal{Y}\\to \\mathcal{Y}$ as\n\\begin{equation}  \\label{eq:fw_operator}\nG_k(z)  := \\frac{1}{\\tau}\\Big(z-\\text{prox}_{\\tau \\ell_k}(z-\\tau \\nabla  \\RA(z))\\Big)\n\\end{equation}\nand notice that the proximal step of  \\ref{alg:fw_accel} can be written in terms of $G_k$ as $u_{k+1}=w_k - \\tau G_k(w_k)$. The descent lemma yields (see, e.g., \\cite{AttouchTikAccel,ChambolleDossal2015})\n\\begin{equation}  \\label{eq:descent}\n    d_{\\lambda_k}(w-\\tau G_k(w)) \\leq  d_{\\lambda_k}(u) + \\langle G_k(w), w-u\\rangle - \\frac{\\tau}{2}\\|G_k(w)\\|^2, \\quad \\text{for all }w,u\\in\\mathcal{Y},\n\\end{equation}\nEvaluating \\eqref{eq:descent} for $u=u_{k}$ and $w=w_k$, we get\n\\begin{equation}  \\label{descent:first}\nd_{{\\lambda_k}}(u_{k+1}) \\leq  d_{{\\lambda_k}}(u_{k}) + \\langle G_k(w_k), w_k - u_{k}\\rangle - \\frac{\\tau}{2}\\|G_k(w_k)\\|^2.\n\\end{equation}\nEvaluating \\eqref{eq:descent} for $u=u^\\dagger$ and $w=w_k$ we derive\n \\begin{equation}  \\label{descent:second}\nd_{{\\lambda_k}}(u_{k+1}) \\leq  d_{{\\lambda_k}}(u^\\dagger) + \\langle G_k(w_k), w_k -u^\\dagger\\rangle - \\frac{\\tau}{2}\\|G_k(w_k)\\|^2.\n\\end{equation}\nWe now multiply \\eqref{descent:first} by $t_{k+1}-1$ and we add it to \\eqref{descent:second}, and we obtain\n\\begin{align}\nt_{k+1} d_{{\\lambda_k}}(u_{k+1}) \n&\\leq  (t_{k+1}-1)d_{{\\lambda_k}}(u_{k}) +  d_{{\\lambda_k}}(u^\\dagger)  \\nonumber\\\\\n\\label{ineq:1}&+ \\langle  G_k(w_k), (t_{k+1}-1)(w_k-u_{k}) + (w_{k} - u^\\dagger)\\rangle  -\\frac{\\tau}{2}t_{k+1}\\|  G_k(w_k)\\|^2 . \n\\end{align}\nAs an immediate consequence of Lemma \\ref{lemma:sequence}, we observe the following property:\n\\begin{align}  \\label{eq:property_zk}\n(t_{k+1}-1)(w_k-u_{k}) + w_k & = u_{k} + t_{k+1}(w_k -u_{k}) \\notag\\\\\n& = u_{k} + t_{k+1} \\alpha_k(u_{k} -u_{k-1}) \\\\\n& = u_{k-1} + (1+ t_{k+1} \\alpha_k)(u_{k} -u_{k-1}) \\notag\\\\\n& = u_{k-1} +t_k(u_{k} -u_{k-1}) = z_k. \\notag\n\\end{align}\nThanks to \\eqref{eq:def_zk:noisefree}, the fact that $z_k-\\tau t_{k+1}G_k(w_k)=z_{k+1}$ and the previous equality, we can reorder the terms in \\eqref{ineq:1} and rewrite it as\n\\begin{align}\nt_{k+1} ( d_{{\\lambda_k}}(u_{k+1}) -  d_{{\\lambda_k}}(u^\\dagger)) \n& \\leq  (t_{k+1}-1) (d_{{\\lambda_k}}(u_{k}) -  d_{{\\lambda_k}}(u^\\dagger)) \\notag\\\\\n& + \\frac{1}{2\\tau t_{k+1}}\\left(\\|z_{k}-u^\\dagger\\|^2-\\|z_{k+1}-u^\\dagger\\|^2\\right). \\notag \n\\end{align}\nWe now multiply everything by $t_{k+1}$, re-arrange and get\n\\begin{align}  \\label{descent:third}\n& t^2_{k+1} ( d_{{\\lambda_k}}(u_{k+1}) -  d_{{\\lambda_k}}(u^\\dagger))   + \\frac{1}{2\\tau}\n\\|z_{k+1}-u^\\dagger\\|^2   \\\\\n&\\leq  (t^2_{k+1}-t_{k+1}) (d_{{\\lambda_k}}(u_{k}) -  d_{\\lambda_k}(u^\\dagger)) + \\frac{1}{2\\tau } \\|z_{k}-u^\\dagger\\|^2.  \\notag\n\\end{align}\nEquivalently:\n\\begin{align}  \\label{descent:third2}\n& t^2_{k+1} \\Big( d_{{\\lambda_k}}(u_{k+1}) -  d_{{\\lambda_k}}(u^\\dagger) \\Big)   + \\frac{1}{2\\tau}\n\\|z_{k+1}-u^\\dagger\\|^2 \\nonumber  \\\\\n&\\leq  t^2_k \\Big(d_{{\\lambda_k}}(u_{k}) -  d_{{\\lambda_k}}(u^\\dagger)\\Big) +   (t^2_{k+1}-t_{k+1} - t_{k}^2) \\Big( d_{{\\lambda_k}}(u_{k}) -  d_{{\\lambda_k}}(u^\\dagger) \\Big)+ \\frac{1}{2\\tau } \\|z_{k}-u^\\dagger\\|^2. \\notag\n\\end{align}\nTo get the desired terms, we first use on the left-hand side  the monotonicity property of the function $ d_{{\\lambda_k}}(\\cdot)$ as a function of $k$ (see  Lemma  \\ref{L:functional properties}\\ref{L:functional properties:decreasing function sequence}) and then add and subtract in the parentheses  the term $\\inf d_0$, thus getting:\n\\begin{align}  \\label{descent:fifth}\n&t^2_{k+1} \\Big( d_{{\\lambda_{k+1}}}(u_{k+1}) -   \\inf d_0 \\Big)   + \\frac{1}{2\\tau}\n\\|z_{k+1}-u^\\dagger\\|^2   \\notag \\\\\n&\\leq  t^2_k \\Big(d_{{\\lambda_{k}}}(u_{k}) - \\inf d_0 \\Big) +   (t^2_{k+1}-t_{k+1} - t_{k}^2) \\Big( d_{{\\lambda_{k}}}(u_{k}) -  \\inf d_0\\Big) \\\\\n& +t_{k+1} \\Big(d_{{\\lambda_{k}}}(u^\\dagger) -  \\inf d_0\\Big)   +  \\frac{1}{2\\tau } \\|z_{k}-u^\\dagger\\|^2. \\notag\n\\end{align}\nAfter rearranging and using the definition of $\\mathcal{E}$ in \\eqref{eq:discrete energy}, we deduce:\n\\begin{align}  \\label{descent:fifth}\n\\mathcal{E}(k+1) +    (t^2_{k}+t_{k+1} - t_{k+1}^2) \\Big( d_{{\\lambda_{k}}}(u_{k}) -  \\inf d_0\\Big) \\leq  \\mathcal{E}(k) + t_{k+1} \\Big(d_{{\\lambda_{k}}}(u^\\dagger) -  \\inf d_0\\Big). \\nonumber\n\\end{align}\n\n\\noindent Thanks to \\eqref{eq:properties_sequences_alphak_tk} and  Lemma~\\ref{L:functional properties}\\ref{L:functional properties:decreasing function sequence}, we can neglect the second term  in the left-hand side of the above inequality,  finally getting the desired  \\eqref{eq:condition}.\nIterating \\eqref{eq:condition}  recursively gives\n\\begin{align}  \\label{eq:decay_complete}\n\\mathcal{E}(k) \\leq \\mathcal{E}(1) +\\sum_{j=1}^{k-1}t_{j+1}\\Big( d_{\\lambda_{j}}(u^\\dagger)-\\inf d_0 \\Big).\n\\end{align}\nTo  bound the sum appearing on the right hand side, we need to analyze the residuals $r_j:= d_{\\lambda_{j}}(u^\\dagger) -\\inf d_0$. Similarly as for the estimation obtained in the continuous case, we can use for this purpose the property in Lemma \\ref{L:functional properties}\\ref{L:functional properties:diagonal rates via conditioning} and get that for some fixed constant $c$ independent on $j$ (defined analogously as in \\eqref{proof:ccd3}), we have\n$$\nr_j\\leq c \\lambda_j^{\\frac{1}{q-1}},\\qquad\\text{for every }j\\geq 1.\n$$\nBy assumption \\ref{assumption:parameter:relaxation}, with $\\Lambda$ as  in \\eqref{def:biglambda}, we thus conclude that\n$$\n\\sum_{j=1}^{k-1} t_{j+1} r_j \\leq  c \\sum_{j=1}^{k-1} t_{j+1} \\lambda_j^{\\frac{1}{q-1}}  \\leq c \\Lambda < +\\infty\n$$\nThis allows us to deduce from \\eqref{eq:decay_complete} the convergence rate on the dual values in \\eqref{eq:rate}, by taking $C = \\Ee(1) + c \\Lambda$.\nFinally, the convergence rate on the primal iterates  in \\eqref{eq:rate} follows from  Lemma \\ref{L:functional properties}\\ref{L:functional properties:from dual value to primal iterates}.\n\\end{proof}","start":63678,"end":70715},{"raw_tex":"\\begin{lemma}\\label{L:functional properties}\nUnder the assumptions \\ref{assumption:data-fit convex and coercive}-\\ref{assumption:data-fit conditionned locally} and \\ref{H:R strongly convex}-\\ref{assumption:AR2},  we have:\n\\begin{enumerate}[label=(\\roman*)]\n\t\\item\\label{L:functional properties:R_A Lipschitz} $\\Rr_A$ is differentiable and $\\nabla  \\RA$ is Lipschitz continuous, with Lipschitz constant equal to $\\sigma^{-1} \\Vert A \\Vert^2$.\n\t\\item\\label{L:functional properties:L at 0} $\\forall y \\in \\Yy$, $\\ell^*_{y}(0)=0$ and $\\partial \\ell^*_y(0)=\\{y\\}$.\n\t\\item\\label{L:functional properties:argmin d_0 nonempty} There holds: $\\argmin  d_0 \\neq \\emptyset$.\n \t\\item\\label{L:functional properties:decreasing function sequence} $\\forall u \\in \\Yy$, the function $\\lambda\\in [0,+\\infty) \\mapsto \\ d_{\\lambda}(u)$ is nondecreasing. \n\t\\item\\label{L:functional properties:from dual value to primal iterates} $\\forall t >0, \\forall u \\in \\Yy, \\ \\text{ if }\n{x}:=\\nabla R^*(-A^*u),$ then $\n \\frac{\\sigma}{2} \\Vert {x} - x^\\dagger \\Vert^2 \\leq d_0(u) - \\inf d_0.$ \n \t\\item\\label{L:functional properties:diagonal rates via conditioning} $\\forall u^\\dagger \\in \\argmin  d_0$, if  $\\lambda \\Vert u^\\dagger \\Vert \\leq \\frac{\\gamma}{q} \\varrho^{q-1}$, then\n \t\\begin{equation}\n \t    d_{\\lambda}(u^\\dagger) - \\inf  d_0 \\leq\n \t    \\begin{cases}\n \t     0 & \\text{ if } q=1, \\\\\n \t     (1-\\frac{1}{q}) {\\gamma^{-1/(q-1)}} \\Vert u^\\dagger \\Vert^{q/(q-1)} \\lambda^{1/(q-1)} & \\text{ if } q>1.\n \t    \\end{cases}\n \t\\end{equation}\n\\end{enumerate}\n\\end{lemma}","start":115130,"end":116667},{"raw_tex":"\\begin{proof}\n\\ref{L:functional properties:R_A Lipschitz}: follows  from the strong convexity of $R$, see, e.g.,  \n\\cite[Theorem 18.15]{BauschkeCombettes2017}.\\\\\n\\ref{L:functional properties:L at 0}: it is a simple consequence of the properties of the Fenchel transform as it can be found, e.g., in \n \\cite[Proposition 13.10(i) \\& Corollary 16.30]{BauschkeCombettes2017}.\\\\\n\\ref{L:functional properties:argmin d_0 nonempty} and \\ref{L:functional properties:from dual value to primal iterates}: follow from \\cite[Lemma 5]{Garrigos2017} by simply taking $f=R$ and $g=\\delta_{\\{\\bar y\\}}$, while property \\ref{L:functional properties:decreasing function sequence} has been proved in \\cite[Proposition 2(i)]{Garrigos2017}.\\\\\n\\ref{L:functional properties:diagonal rates via conditioning}: it is enough to verify that $\\ell_{\\bar y}(\\cdot)$ is $q$-well conditioned in the sense of \\cite[Definition 1]{Garrigos2017}, while assumption \\ref{assumption:data-fit conditionned locally} holds only locally.\nTo check this, we introduce the function $\\psi : \\R \\rightarrow \\R$ defined for the $\\varrho>0$ appearing in \\ref{assumption:data-fit conditionned locally}  by:\n\\begin{equation}\n \\psi  t \\mapsto\n\\begin{cases}\n \\frac{\\gamma}{q}\\vert t \\vert^q  & \\text{ if } \\vert t \\vert \\leq \\varrho, \\\\\n \\frac{\\gamma}{q} \\varrho^{q-1} \\vert t \\vert & \\text{ if } \\vert t \\vert > \\varrho.\n\\end{cases}\n\\end{equation}\nFrom \\ref{assumption:data-fit conditionned locally}, we easily  deduce that $\\ell_{\\bar y}(y) \\geq \\psi(\\Vert y- \\bar y \\Vert)$ for all $y \\in \\Yy$ (see \\cite[Corollary 3.4.2]{ZalinescuBook2002}).\nNote that $\\psi$ is not convex for $q >1$, so in this case we consider instead the function\n\\begin{equation}\n m: \\R \\rightarrow \\R , \\quad t \\mapsto \n\\begin{cases}\n \\frac{\\gamma}{q}\\vert t \\vert^q  & \\text{ if } \\vert t \\vert \\leq q^{1/(1-q)} \\varrho, \\\\\n \\frac{\\gamma}{q}\\varrho^{q-1}\\vert t \\vert \n -  \\frac{\\gamma}{q^{\\frac{q}{q-1}}}\\varrho^q (1 - \\frac{1}{q})  & \\text{ if } \\vert t \\vert > q^{1/(1-q)} \\varrho,\n\\end{cases}\n\\end{equation}\nand define $m:=\\psi$ for $q=1$.\nIt is an easy exercise to verify that $m$ is indeed a convex function on $\\R$, and that $m(w) \\leq \\psi(w)$ for all $w\\in\\R$.\nNow, we can make use of \\cite[Lemma 2]{Garrigos2017},\nwhich tells us that $d_\\lambda(u) - \\inf d_0 \\leq \\lambda^{-1} m^*(\\Vert u \\Vert \\lambda)$.\nThe desired result  now follows from the computation of the Fenchel transform of $m$.\nIf $q=1$, we have that $m(t) = \\gamma \\vert t \\vert$, so classic Fenchel calculus entails that $m^*$ is $\\delta_{[-\\gamma,\\gamma]}$, the indicator function of $[-\\gamma,\\gamma]$.\nIf $q>1$, easy computations show that $m^*$ reads\n\\begin{equation}\n m^* : \\R \\rightarrow \\R, \\quad s \\mapsto\n\\begin{cases}\n (1 - \\frac{1}{q}) \\gamma^{-1/(q-1)}\\vert s \\vert^{\\frac{q}{q-1}}  & \\text{ if } \\vert s \\vert \\leq \\frac{\\gamma}{q} \\varrho^{q-1}, \\\\\n + \\infty  & \\text{ if } \\vert s \\vert > \\frac{\\gamma}{q} \\varrho^{q-1}.\n\\end{cases}\n\\end{equation}\nBy now applying \\cite[Lemma 2]{Garrigos2017} we  conclude. \n\\end{proof}","start":116668,"end":119700}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:inverse_problem"],"declared_external":false,"source_only":false,"title":"线性逆问题","source_units":[{"raw_tex":"We are interested in solving the linear inverse problem:\n\\begin{equation}   \\label{eq:inverse_problem}\n\\text{find }\\quad \\xtrue \\in \\Xx \\quad \\text{s.t.}\\quad A \\xtrue = \\ytrue,\n\\end{equation}\nwhere $A: \\mathcal{X}\\to\\mathcal{Y}$ is a bounded linear operator between two Hilbert spaces $\\mathcal{X}$ and $\\Yy$,\nand $ \\ytrue \\in \\mathcal{Y}$ can be seen as  a given  measurement of some unknown $\\xtrue \\in \\Xx$ we want to recover.\nIn general, the inverse problem \\eqref{eq:inverse_problem} is ill-posed as its solution (if it exists) may lack some fundamental properties like uniqueness or stability. \n","start":20669,"end":21271}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":["eq:primal"],"declared_external":false,"source_only":false,"title":"分层原问题","source_units":[{"raw_tex":"A standard approach   is assuming the desired $\\xtrue$ to be well-approximated by\n \\cite{EngHanNeu96,BenBur18}:\n\\begin{equation}\\label{eq:primal}\n   \\tag*{\\textbf{($\\bm{P_0(\\ytrue)}$)}} \\text{find\\ }\n      \\xsol \\in  \\argmin  \\left\\{ R(x) \\text{ s.t. } x\\in\\underset{x' \\in \\Xx}{\\argmin}~ \\ell(Ax';  \\ytrue) \\right\\}, \n\\end{equation} \ncalled the primal problem in the following.  Here,  $R$ is a  regularization function enforcing some \\emph{a-priori} knowledge on  the desired solution $\\xtrue$, while $\\ell: \\Yy^2\\to\\R\\cup \\{+\\infty\\}$ is a data-fit function. In practice, the  data  is  subject to noise  due to, e.g., possible transmission and/or acquisition problems. As a consequence, we only have access to an inexact version $\\ynoisy$ of $\\ytrue$. \n","start":21271,"end":22028}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"观测噪声背景","source_units":[{"raw_tex":"called the primal problem in the following.  Here,  $R$ is a  regularization function enforcing some \\emph{a-priori} knowledge on  the desired solution $\\xtrue$, while $\\ell: \\Yy^2\\to\\R\\cup \\{+\\infty\\}$ is a data-fit function. In practice, the  data  is  subject to noise  due to, e.g., possible transmission and/or acquisition problems. As a consequence, we only have access to an inexact version $\\ynoisy$ of $\\ytrue$. \n","start":21606,"end":22028}]},{"id":"S004","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"Hilbert 空间与凸函数类","source_units":[{"raw_tex":"We begin fixing the  notation. Let $\\Hh$ be a Hilbert space with scalar product  $\\langle\\cdot,\\cdot\\rangle$ and associated norm $\\|\\cdot\\|$. Given $y\\in\\Hh$, and $\\varrho\\in\\mathbb{R}_+$ and $\\mathbb{B}(y,\\varrho)$ is the open ball of center $y$ and radius $\\varrho$.\nwe denote by $\\Gamma_0(\\Hh)$ the set of proper, convex and lower semi-continuous functions from $\\Hh$ to $]-\\infty,+\\infty]$.\n","start":28119,"end":28514}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"强凸性","source_units":[{"raw_tex":"We say that  $f \\in \\Gamma_0(\\Hh)$ is $\\sigma$-{\\em strongly convex}  if \n$f-{\\sigma}\\Vert \\cdot \\Vert^2/2 \\in \\Gamma_0(\\Hh)$, with  $\\sigma \\in\\left]0,+\\infty\\right[$.\n","start":28514,"end":28683}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["def:subdiff"],"declared_external":false,"source_only":false,"title":"次微分","source_units":[{"raw_tex":"We recall that the {\\em subdifferential} of  $f \\in \\Gamma_0(\\Hh)$  is the multi-valued operator \n$\\partial f : \\Hh \\rightarrow 2^{\\Hh}$ defined by setting\n\\begin{equation}  \\label{def:subdiff}\n(\\forall x\\in\\Hh)\\quad \\partial f(x) := \\left\\{u \\in \\Hh: f(x') - f(x) - \\langle u , x' - x \\rangle \\geq 0 \\text{ for all } x' \\in \\Hh \\right\\}.\n\\end{equation}\nNote that if $f$ is Gateaux differentiable at $x \\in \\Hh$, then $\\partial f(x)=\\{\\nabla f(x) \\}$, see, e.g.~, \\cite[Proposition 17.31 i)]{BauschkeCombettes2017}.\n","start":28683,"end":29199}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":["def:proximal operator"],"declared_external":false,"source_only":false,"title":"近端算子","source_units":[{"raw_tex":"We also recall that for all  $x\\in \\Hh$ and $\\tau>0$ the proximity operator $\\prox_{\\tau f} : \\Hh \\rightarrow \\Hh$  of $f\\in\\Gamma_0(\\Hh)$ with parameter $\\tau$ is defined by:\n\\begin{equation}\\label{def:proximal operator}\n    \\prox_{\\tau f}(x) =\n    (I + \\partial f)^{-1}(x) = \n    \\argmin\\limits_{x' \\in \\Hh} \\left\\{ f(x') + \\frac{1}{2\\tau}\\Vert x' - x \\Vert^2 \\right\\}.\n\\end{equation}\n","start":29199,"end":29586}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"Fenchel 共轭","source_units":[{"raw_tex":"Given $f \\in \\Gamma_0(\\Hh)$, we will denote by $f^* : \\Hh \\rightarrow [-\\infty,+\\infty]$   the {\\em Fenchel conjugate} of $f$\n\\[\n(\\forall u\\in\\Hh)\\quad f^*(u):=  \\sup_{x\\in \\Hh}  \\left\\{\\langle u,x\\rangle_{\\Hh} -f(x) \\right\\}.\n\\]\n","start":29586,"end":29816}]},{"id":"S009","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先修列表：BauschkeCombettes2017, Theorem 18.15；正文第724行重述该标准结论。"],"source_only":false,"title":"强凸共轭的光滑性","source_units":[{"raw_tex":"The Fenchel conjugate of $f$ belongs to $\\Gamma_0(\\Hh)$, and if $f$ is $\\sigma$-strongly convex then $f^*$ is differentiable at any point, with a $\\sigma^{-1}$-Lipschitz continuous gradient, see, e.g. \\cite[Theorem 18.15]{BauschkeCombettes2017}. \n","start":29816,"end":30063}]},{"id":"S010","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先修列表中的 standard convex conjugacy；正文明确引用 BauschkeCombettes2017, Theorem 16.23。"],"source_only":false,"title":"共轭次梯度互逆","source_units":[{"raw_tex":"Furthermore, the following property holds, see \\cite[Theorem 16.23]{BauschkeCombettes2017}: \n\\[\n(\\forall (x,u)\\in \\Hh^2)\\quad u \\in \\partial f(x) \\Leftrightarrow x \\in \\partial f^*(u).\n\\] \n","start":30063,"end":30252}]},{"id":"S011","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"渐近阶与统一范数","source_units":[{"raw_tex":"Finally, given two real sequences $(a_k)_{k\\geq 1}$ and $(b_k)_{k\\geq 1}$, we will write $a_k = O(b_k)$ whenever there exists a positive constant $M>0$ such that $a_k \\leq M b_k$ for all $k \\geq 1$. We will use the more precise notation $a_k = \\Theta(b_k)$ if both conditions $a_k = O(b_k)$ and $b_k = O(a_k)$ hold.\nNote also that we will use the same notation  $\\|\\cdot\\|$ and $\\langle\\cdot,\\cdot\\rangle$ for the norm and the scalar product in every Hilbert space we consider. \n","start":30252,"end":30731}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"拟合项的凸性与强制性","source_units":[{"raw_tex":"We make   the following assumptions on the data-fit $\\ell$ and the regularizer $R$: \n\\begin{enumerate}[label=\\textbf{$(L_\\arabic*)$}]\n\t\\item \\label{assumption:data-fit convex and coercive}\n\tFor all $y \\in \\Yy$, $\\ell_y:=\\ell(\\cdot,y)\\in \\Gamma_0(\\Yy) $ and it is coercive.\n","start":30762,"end":31035}]},{"id":"S013","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"拟合项的唯一零点","source_units":[{"raw_tex":"\t\\item \\label{assumption:data-fit not flat} \n\tFor all $(y_1,y_2)\\in \\Yy^2, ~\\ell(y_1,y_2)=0 \\iff y_1=y_2$.\n","start":31035,"end":31142}]},{"id":"S014","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"局部条件下界","source_units":[{"raw_tex":"\t\\item \\label{assumption:data-fit conditionned locally}\n\t there exists\n     $q\\in[1,+\\infty[$, $\\varrho \\in ]0,+\\infty]$, $ \\gamma \\in ]0,+\\infty[$ such that\n\t$$\n\t\\forall y \\in \\mathbb{B}( \\ytrue,\\varrho),\\quad \\frac{\\gamma}{q}\\Vert y-\\ytrue\\Vert^q \\leq \\ell(y,\\ytrue). \n\t$$ \n\\end{enumerate}\n","start":31142,"end":31434}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"正则项强凸","source_units":[{"raw_tex":"\\begin{enumerate}[label=\\textbf{$(R_\\arabic*)$}]\t\n\t\\item \\label{H:R strongly convex} $R$ is $\\sigma$-strongly convex, with $\\sigma \\in \\left]0,+\\infty\\right[$,  \n","start":31434,"end":31596}]},{"id":"S016","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"源条件","source_units":[{"raw_tex":"\t\\item \\label{assumption:AR2}   $\\partial R(\\xsol) \\cap \\im A^* \\neq \\emptyset$.\t\n\\end{enumerate}\n","start":31596,"end":31694}]},{"id":"S017","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"局部与全局条件性","source_units":[{"raw_tex":"If assumption \\ref{assumption:data-fit conditionned locally} is satisfied, we say that $\\ell_{\\ytrue}$ is \\emph{locally\n\t$q$-conditioned} at $\\bar{y}$. If \\ref{assumption:data-fit conditionned locally} is satisfied with $\\rho=+\\infty$, then we say that $\\ell_{\\ytrue}$  is $q$-conditioned at $\\bar{y}$.\n","start":31694,"end":31997}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":["eq:primal_char"],"declared_external":false,"source_only":false,"title":"精确约束原问题","source_units":[{"raw_tex":"A class of methods to solve \\eqref{eq:inverse_problem}  consider the   problem \n\\begin{equation}   \\label{eq:primal_char}\n    \\text{find }  \\xsol \\in  \\argmin_{x \\in \\Xx}  \\left\\{ R(x) + \\delta_{\\bar{y}}(Ax) \\right\\},\n\\end{equation}\nwhere the constraint \\eqref{eq:inverse_problem} is encoded by the indicator function $\\delta_{\\bar{y}}$. Using Fenchel-Rockafeller duality,  \n","start":36263,"end":36638}]},{"id":"S019","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:Dual"],"declared_external":false,"source_only":false,"title":"极限对偶目标","source_units":[{"raw_tex":"where the constraint \\eqref{eq:inverse_problem} is encoded by the indicator function $\\delta_{\\bar{y}}$. Using Fenchel-Rockafeller duality,  \nwe derive the corresponding dual problem:\n\\begin{equation}   \\label{eq:Dual}\n\\text{find } u^\\dagger \\quad\\text{ s.t. }\\quad u^\\dagger \\in \\argmin_{u\\in \\mathcal{Y}}\\  \\left\\{ d_0(u) := R^*(-A^*u) + \\langle \\bar{y}, u \\rangle \\right\\}\n    \\tag*{\\textbf{($\\bm{D_0}$)}}.\n\\end{equation}\n","start":36496,"end":36921}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":["eq:DualReg"],"declared_external":false,"source_only":false,"title":"正则化对偶目标","source_units":[{"raw_tex":"The dual problem of \\ref{eq:primal penalized} is\n\\begin{equation}   \\label{eq:DualReg}\n\\text{find } u_\\lambda \\in \\argmin_{u\\in \\mathcal{Y}}\\ \\left\\{d_\\lambda(u) := R^*(-A^*u) + \\frac{1}{\\lambda}\\ell^*(\\lambda u;\\ynoisy)\\right\\}.   \\tag*{\\textbf{($\\bm{D_\\lambda}$)}}\n\\end{equation} \n","start":39699,"end":39982}]},{"id":"S021","semantic_type":"notation","kind":"notation","labels":["eq:composite"],"declared_external":false,"source_only":true,"title":"复合光滑项缩写","source_units":[{"raw_tex":"From now on we will use the following shorthand notation: \n\\begin{equation} \\label{eq:composite}\n   \\RA: = R^* \\circ (-A^*),\\qquad \\ell^*_{\\bar{y}}(\\cdot): = \\ell(\\cdot,\\bar{y})^*\n\\end{equation}\nso that the composite dual function $d_\\lambda$ can be written as $d_{\\lambda}(u) =  \\RA(u) + \\lambda^{-1} \\ell^*_{\\bar{y}}(\\lambda u)$, for every $u\\in\\mathcal{Y}$.\n","start":48647,"end":49008}]},{"id":"S022","semantic_type":"definition","kind":"definition","labels":["proof:ccd3"],"declared_external":false,"source_only":true,"title":"残差常数","source_units":[{"raw_tex":"where the constant $c$ is defined as:\n\\begin{equation}\\label{proof:ccd3}\n    c :=\n    \\begin{cases}\n    0 \\quad &\\quad \\text{ if }q=1,\\\\\n    (1-(1/q))\\gamma^{-1/(q-1)}\\Vert u^\\dagger \\Vert^{q/(q-1)} \\quad &\\quad \\text{ if } q >1,\n    \\end{cases}\n\\end{equation}\nand it is finite in both cases.\n","start":52273,"end":52566}]},{"id":"S023","semantic_type":"algorithm","kind":"algorithm","labels":["alg:fw_accel"],"declared_external":false,"source_only":false,"title":"惯性对偶对角下降","source_units":[{"raw_tex":"\\begin{equation}\\label{alg:fw_accel}\nu_0 = u_1 \\in \\Yy, \\quad \\text{ compute for } k \\geq 1 \\quad \n\\begin{cases}\n w_k & = u_k + \\alpha_k(u_k-u_{k-1}), \\\\\n u_{k+1} &= \\mathrm{prox}_{\\frac{\\tau}{\\lambda_{k}} \\ell^*_{\\bar{y}}(\\lambda_k\\cdot)}\\left(w_k - \\tau\\nabla \\RA(w_k)\\right),\\\\\n    x_{k+1} & = \\nabla R^*(-A^* u_{k+1}). \n\\end{cases}\n\\tag*{\\textbf{($\\bm{I3D}$)}}\n\\end{equation}\nThis algorithm depends on three parameters: the stepsize $\\tau >0$, the relaxation parameters $(\\lambda_k)_k$ and the friction parameters $(\\alpha_k)_k$.\n","start":56423,"end":56957}]},{"id":"S024","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"参数选择说明","source_units":[{"raw_tex":"The stepsize will be chosen depending on the value of the Lipschitz constant of $\\nabla \\RA$.\nFor the relaxation parameters, we will consider a discrete analogue of the assumption \\ref{requirement:lambda1:cont} formulated in the continuous setting.\nFor the friction parameters $\\alpha_k$, we will allow more general values not necessarily corresponding to $\\alpha_k = 1 -\\alpha/k$ as  above. We gather  the requirements on the   parameters in the following assumptions:\n\\begin{enumerate}[label=\\textbf{$(P_\\arabic*)$}]\n","start":56957,"end":57476}]},{"id":"S025","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"步长限制","source_units":[{"raw_tex":"\t\\item \\label{assumption:parameter:stepsize}\n\t$\\tau \\in \\left(0,\\frac{\\sigma^2}{\\Vert A \\Vert^2}\\right]$, where $\\sigma > 0$ is defined in assumption \\ref{H:R strongly convex}.\n","start":57476,"end":57653}]},{"id":"S026","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"摩擦与时间权重","source_units":[{"raw_tex":"\t\\item \\label{assumption:parameter:friction}\n\t$\\alpha_k$ is non-negative and for every $k\\geq 1$, $t_k:=1+\\sum_{i=k}^{+\\infty}\\prod_{j=k}^i \\alpha_j$ is finite,  with $t_k = \\Theta(k)$.\n","start":57653,"end":57839}]},{"id":"S027","semantic_type":"assumption","kind":"assumption","labels":["def:biglambda"],"declared_external":false,"source_only":false,"title":"对角参数与加权可和性","source_units":[{"raw_tex":"\t\\item \\label{assumption:parameter:relaxation}\n\t$(\\lambda_k)$ is a strictly positive non-increasing sequence such that $\\lim_{k\\to\\infty} \\lambda_k = 0$. \n\tMoreover, by defining\n\t\\begin{equation}   \\label{def:biglambda}\n\t\\Lambda:=\n\t\\begin{cases}\n\t\\sum_{k\\geq 1} t_{k+1} \\lambda_k^{1/(q-1)}\n\t& \\text{ if } q>1, \\\\\n\t0\n\t& \\text{ if } q=1, \n\t\\end{cases}\n\t\\end{equation}\n\twe have that $\\Lambda< + \\infty$.\n","start":57839,"end":58240}]},{"id":"S028","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"初始对偶尺度限制","source_units":[{"raw_tex":"\t\\item \\label{assumption:parameter:technical}\n\tFor some $u^\\dagger \\in \\argmin d_0$, we have  $\\lambda_0 \\Vert u^\\dagger \\Vert \\leq \\gamma \\varrho^{q-1} /q$.\n\\end{enumerate}\n","start":58240,"end":58414}]},{"id":"S029","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"序列先修的引入","source_units":[{"raw_tex":" Following \\cite{AttouchTikAccel}, we require the sequence of friction parameters $(\\alpha_k)$ to satisfy  \\ref{assumption:parameter:friction},  a particular summability property  guaranteeing a technical condition crucial in the following. We summarize such a requirement and the resulting condition in the following lemma.\n\n","start":59791,"end":60117}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","labels":["eq:choice:alpha","eq:choice:tk","eq:properties_sequences_alphak_tk","lemma:sequence"],"declared_external":true,"declaration_text":["原始先修列表：AttouchTikAccel, Lemma 2.1，作为 B0069 重述，文内无证明。"],"source_only":false,"title":"引用的序列引理","source_units":[{"raw_tex":"\\begin{lemma}[ {{\\cite[Lemma 2.1]{AttouchTikAccel} }}]   \\label{lemma:sequence}\nAssume that $(\\alpha_k)$ is non-negative and satisfies\n\\begin{equation}  \\label{eq:choice:alpha}\n\\sum_{i=k}^{+\\infty}\\prod_{j=k}^i \\alpha_j<+\\infty,\\quad\\text{for every }k\\geq 1.\n\\end{equation}\nThen, the sequence defined by\n\\begin{equation} \\label{eq:choice:tk}\nt_k:=1+\\sum_{i=k}^{+\\infty}\\prod_{j=k}^i \\alpha_j\n\\end{equation}\nis well-defined, and satisfies for every $k\\geq 1$ the following properties:\n\\begin{equation}   \\label{eq:properties_sequences_alphak_tk}\n1+\\alpha_k t_{k+1} = t_k, \\qquad t^2_{k+1}-t^2_k\\leq t_{k+1}.\n\\end{equation}\n\\end{lemma}\n","start":60117,"end":60751}]},{"id":"S031","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:properties_sequences_alphak_tk","lemma:sequence"],"declared_external":false,"source_only":false,"title":"时间权重一次恒等式","source_units":[{"raw_tex":"\\begin{equation}   \\label{eq:properties_sequences_alphak_tk}\n1+\\alpha_k t_{k+1} = t_k, \\qquad t^2_{k+1}-t^2_k\\leq t_{k+1}.\n\\end{equation}\n","start":60601,"end":60739}]},{"id":"S032","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:properties_sequences_alphak_tk","lemma:sequence"],"declared_external":false,"source_only":false,"title":"时间权重二次上界","source_units":[{"raw_tex":"\\begin{equation}   \\label{eq:properties_sequences_alphak_tk}\n1+\\alpha_k t_{k+1} = t_k, \\qquad t^2_{k+1}-t^2_k\\leq t_{k+1}.\n\\end{equation}\n","start":60601,"end":60739}]},{"id":"S033","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"附录对象回指","source_units":[{"raw_tex":"We first consider $\\Rr_A $, $\\ell^*_y$ defined in \\eqref{eq:composite} and on the diagonal dual function $d_{\\lambda}$ and its limit $d_0$  defined in \\ref{eq:DualReg} and \\ref{eq:Dual}, respectively. For similar results see also \\cite{Garrigos2017}.\n","start":114878,"end":115129}]},{"id":"S034","semantic_type":"theorem","kind":"theorem","labels":["T:discrete convergence rates noiseless","eq:rate"],"declared_external":false,"source_only":false,"title":"精确数据快速收敛","source_units":[{"raw_tex":"\\begin{theorem}[Fast convergence]\\label{T:discrete convergence rates noiseless}\nLet the assumptions \\ref{assumption:data-fit convex and coercive}-\\ref{assumption:data-fit conditionned locally}, \\ref{H:R strongly convex}-\\ref{assumption:AR2}, \\ref{assumption:parameter:stepsize}-\\ref{assumption:parameter:technical} hold true.\nLet $(x_k)$ and $(u_k)$ be the sequences generated by algorithm \\ref{alg:fw_accel}.\nThen, there exists $C\\in\\left]0,+\\infty\\right[$ such that\n\\begin{equation}  \\label{eq:rate}%\\label{eq:rate:primal:it}\n d_{{\\lambda_k}}(u_k) - \\inf  d_0  \\leq \\frac{C}{t_k^2} \n \\quad \\text{ and } \\quad \n \\| x_k - x^\\dagger \\| \\leq \\frac{\\sqrt{2C}}{\\sqrt{\\sigma}t_k}.\n\\end{equation}\n\\end{theorem}","start":62973,"end":63677}]},{"id":"S035","semantic_type":"proof","kind":"proof","labels":["descent:fifth","descent:first","descent:second","descent:third","descent:third2","eq:condition","eq:decay_complete","eq:def_zk:noisefree","eq:descent","eq:discrete energy","eq:fw_operator","eq:property_zk","eq:shortand_notation_Dk","ineq:1"],"declared_external":false,"source_only":false,"title":"快速收敛的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $u^\\dagger\\in \\argmin d_0$ be the minimizer of $d_0$ defined in assumption \\ref{assumption:parameter:technical}, and\ndefine, for every $k\\geq 1$,  the discrete Lyapunov energy function:\n\\begin{equation}  \\label{eq:discrete energy}\n\\mathcal{E}(k):=t_k^2 \\Big( d_{{\\lambda_k}}(u_k)-\\inf d_0\\Big) + \\frac{1}{2\\tau}\\|z_k-u^\\dagger\\|^2,\n\\end{equation}\nwhere $z_k$ is defined as:\n\\begin{equation}  \\label{eq:def_zk:noisefree}\nz_k:=u_{k-1} +t_k(u_k-u_{k-1}).\n\\end{equation}\nOur goal is to get an estimate on the decay of $\\mathcal{E}$ along time.\nMore precisely, we will show that for every $k\\geq 1$\n\\begin{equation}  \\label{eq:condition}\n\\mathcal{E}(k+1) - \\mathcal{E}(k)\n\\leq  t_{k+1} \\Big(d_{{\\lambda_k}}(u^\\dagger) -  \\inf d_0\\Big),\n\\end{equation}\nwhich can be seen as a discrete analogue of \\eqref{proof:ccd2}, and from which the desired accelerated convergence rates will follow in a straightforward manner.\n\nFor simplicity, let us denote by $\\ell_k$  the function defined by setting\n\\begin{equation}  \\label{eq:shortand_notation_Dk}\n(\\forall u\\in \\mathcal{Y})\\quad \\ell_k(u)=\\lambda_k^{-1}{\\ell}_{\\bar{y}}^*(\\lambda_k u).\n\\end{equation} \nTo prove \\eqref{eq:condition}, we define for every $k\\geq 1$ the operator $G_k:\\mathcal{Y}\\to \\mathcal{Y}$ as\n\\begin{equation}  \\label{eq:fw_operator}\nG_k(z)  := \\frac{1}{\\tau}\\Big(z-\\text{prox}_{\\tau \\ell_k}(z-\\tau \\nabla  \\RA(z))\\Big)\n\\end{equation}\nand notice that the proximal step of  \\ref{alg:fw_accel} can be written in terms of $G_k$ as $u_{k+1}=w_k - \\tau G_k(w_k)$. The descent lemma yields (see, e.g., \\cite{AttouchTikAccel,ChambolleDossal2015})\n\\begin{equation}  \\label{eq:descent}\n    d_{\\lambda_k}(w-\\tau G_k(w)) \\leq  d_{\\lambda_k}(u) + \\langle G_k(w), w-u\\rangle - \\frac{\\tau}{2}\\|G_k(w)\\|^2, \\quad \\text{for all }w,u\\in\\mathcal{Y},\n\\end{equation}\nEvaluating \\eqref{eq:descent} for $u=u_{k}$ and $w=w_k$, we get\n\\begin{equation}  \\label{descent:first}\nd_{{\\lambda_k}}(u_{k+1}) \\leq  d_{{\\lambda_k}}(u_{k}) + \\langle G_k(w_k), w_k - u_{k}\\rangle - \\frac{\\tau}{2}\\|G_k(w_k)\\|^2.\n\\end{equation}\nEvaluating \\eqref{eq:descent} for $u=u^\\dagger$ and $w=w_k$ we derive\n \\begin{equation}  \\label{descent:second}\nd_{{\\lambda_k}}(u_{k+1}) \\leq  d_{{\\lambda_k}}(u^\\dagger) + \\langle G_k(w_k), w_k -u^\\dagger\\rangle - \\frac{\\tau}{2}\\|G_k(w_k)\\|^2.\n\\end{equation}\nWe now multiply \\eqref{descent:first} by $t_{k+1}-1$ and we add it to \\eqref{descent:second}, and we obtain\n\\begin{align}\nt_{k+1} d_{{\\lambda_k}}(u_{k+1}) \n&\\leq  (t_{k+1}-1)d_{{\\lambda_k}}(u_{k}) +  d_{{\\lambda_k}}(u^\\dagger)  \\nonumber\\\\\n\\label{ineq:1}&+ \\langle  G_k(w_k), (t_{k+1}-1)(w_k-u_{k}) + (w_{k} - u^\\dagger)\\rangle  -\\frac{\\tau}{2}t_{k+1}\\|  G_k(w_k)\\|^2 . \n\\end{align}\nAs an immediate consequence of Lemma \\ref{lemma:sequence}, we observe the following property:\n\\begin{align}  \\label{eq:property_zk}\n(t_{k+1}-1)(w_k-u_{k}) + w_k & = u_{k} + t_{k+1}(w_k -u_{k}) \\notag\\\\\n& = u_{k} + t_{k+1} \\alpha_k(u_{k} -u_{k-1}) \\\\\n& = u_{k-1} + (1+ t_{k+1} \\alpha_k)(u_{k} -u_{k-1}) \\notag\\\\\n& = u_{k-1} +t_k(u_{k} -u_{k-1}) = z_k. \\notag\n\\end{align}\nThanks to \\eqref{eq:def_zk:noisefree}, the fact that $z_k-\\tau t_{k+1}G_k(w_k)=z_{k+1}$ and the previous equality, we can reorder the terms in \\eqref{ineq:1} and rewrite it as\n\\begin{align}\nt_{k+1} ( d_{{\\lambda_k}}(u_{k+1}) -  d_{{\\lambda_k}}(u^\\dagger)) \n& \\leq  (t_{k+1}-1) (d_{{\\lambda_k}}(u_{k}) -  d_{{\\lambda_k}}(u^\\dagger)) \\notag\\\\\n& + \\frac{1}{2\\tau t_{k+1}}\\left(\\|z_{k}-u^\\dagger\\|^2-\\|z_{k+1}-u^\\dagger\\|^2\\right). \\notag \n\\end{align}\nWe now multiply everything by $t_{k+1}$, re-arrange and get\n\\begin{align}  \\label{descent:third}\n& t^2_{k+1} ( d_{{\\lambda_k}}(u_{k+1}) -  d_{{\\lambda_k}}(u^\\dagger))   + \\frac{1}{2\\tau}\n\\|z_{k+1}-u^\\dagger\\|^2   \\\\\n&\\leq  (t^2_{k+1}-t_{k+1}) (d_{{\\lambda_k}}(u_{k}) -  d_{\\lambda_k}(u^\\dagger)) + \\frac{1}{2\\tau } \\|z_{k}-u^\\dagger\\|^2.  \\notag\n\\end{align}\nEquivalently:\n\\begin{align}  \\label{descent:third2}\n& t^2_{k+1} \\Big( d_{{\\lambda_k}}(u_{k+1}) -  d_{{\\lambda_k}}(u^\\dagger) \\Big)   + \\frac{1}{2\\tau}\n\\|z_{k+1}-u^\\dagger\\|^2 \\nonumber  \\\\\n&\\leq  t^2_k \\Big(d_{{\\lambda_k}}(u_{k}) -  d_{{\\lambda_k}}(u^\\dagger)\\Big) +   (t^2_{k+1}-t_{k+1} - t_{k}^2) \\Big( d_{{\\lambda_k}}(u_{k}) -  d_{{\\lambda_k}}(u^\\dagger) \\Big)+ \\frac{1}{2\\tau } \\|z_{k}-u^\\dagger\\|^2. \\notag\n\\end{align}\nTo get the desired terms, we first use on the left-hand side  the monotonicity property of the function $ d_{{\\lambda_k}}(\\cdot)$ as a function of $k$ (see  Lemma  \\ref{L:functional properties}\\ref{L:functional properties:decreasing function sequence}) and then add and subtract in the parentheses  the term $\\inf d_0$, thus getting:\n\\begin{align}  \\label{descent:fifth}\n&t^2_{k+1} \\Big( d_{{\\lambda_{k+1}}}(u_{k+1}) -   \\inf d_0 \\Big)   + \\frac{1}{2\\tau}\n\\|z_{k+1}-u^\\dagger\\|^2   \\notag \\\\\n&\\leq  t^2_k \\Big(d_{{\\lambda_{k}}}(u_{k}) - \\inf d_0 \\Big) +   (t^2_{k+1}-t_{k+1} - t_{k}^2) \\Big( d_{{\\lambda_{k}}}(u_{k}) -  \\inf d_0\\Big) \\\\\n& +t_{k+1} \\Big(d_{{\\lambda_{k}}}(u^\\dagger) -  \\inf d_0\\Big)   +  \\frac{1}{2\\tau } \\|z_{k}-u^\\dagger\\|^2. \\notag\n\\end{align}\nAfter rearranging and using the definition of $\\mathcal{E}$ in \\eqref{eq:discrete energy}, we deduce:\n\\begin{align}  \\label{descent:fifth}\n\\mathcal{E}(k+1) +    (t^2_{k}+t_{k+1} - t_{k+1}^2) \\Big( d_{{\\lambda_{k}}}(u_{k}) -  \\inf d_0\\Big) \\leq  \\mathcal{E}(k) + t_{k+1} \\Big(d_{{\\lambda_{k}}}(u^\\dagger) -  \\inf d_0\\Big). \\nonumber\n\\end{align}\n\n\\noindent Thanks to \\eqref{eq:properties_sequences_alphak_tk} and  Lemma~\\ref{L:functional properties}\\ref{L:functional properties:decreasing function sequence}, we can neglect the second term  in the left-hand side of the above inequality,  finally getting the desired  \\eqref{eq:condition}.\nIterating \\eqref{eq:condition}  recursively gives\n\\begin{align}  \\label{eq:decay_complete}\n\\mathcal{E}(k) \\leq \\mathcal{E}(1) +\\sum_{j=1}^{k-1}t_{j+1}\\Big( d_{\\lambda_{j}}(u^\\dagger)-\\inf d_0 \\Big).\n\\end{align}\nTo  bound the sum appearing on the right hand side, we need to analyze the residuals $r_j:= d_{\\lambda_{j}}(u^\\dagger) -\\inf d_0$. Similarly as for the estimation obtained in the continuous case, we can use for this purpose the property in Lemma \\ref{L:functional properties}\\ref{L:functional properties:diagonal rates via conditioning} and get that for some fixed constant $c$ independent on $j$ (defined analogously as in \\eqref{proof:ccd3}), we have\n$$\nr_j\\leq c \\lambda_j^{\\frac{1}{q-1}},\\qquad\\text{for every }j\\geq 1.\n$$\nBy assumption \\ref{assumption:parameter:relaxation}, with $\\Lambda$ as  in \\eqref{def:biglambda}, we thus conclude that\n$$\n\\sum_{j=1}^{k-1} t_{j+1} r_j \\leq  c \\sum_{j=1}^{k-1} t_{j+1} \\lambda_j^{\\frac{1}{q-1}}  \\leq c \\Lambda < +\\infty\n$$\nThis allows us to deduce from \\eqref{eq:decay_complete} the convergence rate on the dual values in \\eqref{eq:rate}, by taking $C = \\Ee(1) + c \\Lambda$.\nFinally, the convergence rate on the primal iterates  in \\eqref{eq:rate} follows from  Lemma \\ref{L:functional properties}\\ref{L:functional properties:from dual value to primal iterates}.\n\\end{proof}","start":63678,"end":70715}]},{"id":"S036","semantic_type":"definition","kind":"definition","labels":["eq:discrete energy"],"declared_external":false,"source_only":false,"title":"离散 Lyapunov 能量","source_units":[{"raw_tex":"Let $u^\\dagger\\in \\argmin d_0$ be the minimizer of $d_0$ defined in assumption \\ref{assumption:parameter:technical}, and\ndefine, for every $k\\geq 1$,  the discrete Lyapunov energy function:\n\\begin{equation}  \\label{eq:discrete energy}\n\\mathcal{E}(k):=t_k^2 \\Big( d_{{\\lambda_k}}(u_k)-\\inf d_0\\Big) + \\frac{1}{2\\tau}\\|z_k-u^\\dagger\\|^2,\n\\end{equation}\n","start":63692,"end":64043}]},{"id":"S037","semantic_type":"definition","kind":"definition","labels":["eq:def_zk:noisefree"],"declared_external":false,"source_only":false,"title":"惯性辅助变量","source_units":[{"raw_tex":"where $z_k$ is defined as:\n\\begin{equation}  \\label{eq:def_zk:noisefree}\nz_k:=u_{k-1} +t_k(u_k-u_{k-1}).\n\\end{equation}\n","start":64043,"end":64163}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":["eq:condition"],"declared_external":false,"source_only":false,"title":"提出能量递推目标","source_units":[{"raw_tex":"Our goal is to get an estimate on the decay of $\\mathcal{E}$ along time.\nMore precisely, we will show that for every $k\\geq 1$\n\\begin{equation}  \\label{eq:condition}\n\\mathcal{E}(k+1) - \\mathcal{E}(k)\n\\leq  t_{k+1} \\Big(d_{{\\lambda_k}}(u^\\dagger) -  \\inf d_0\\Big),\n\\end{equation}\nwhich can be seen as a discrete analogue of \\eqref{proof:ccd2}, and from which the desired accelerated convergence rates will follow in a straightforward manner.\n","start":64163,"end":64604}]},{"id":"S039","semantic_type":"notation","kind":"notation","labels":["eq:shortand_notation_Dk"],"declared_external":false,"source_only":false,"title":"缩放共轭项","source_units":[{"raw_tex":"For simplicity, let us denote by $\\ell_k$  the function defined by setting\n\\begin{equation}  \\label{eq:shortand_notation_Dk}\n(\\forall u\\in \\mathcal{Y})\\quad \\ell_k(u)=\\lambda_k^{-1}{\\ell}_{\\bar{y}}^*(\\lambda_k u).\n\\end{equation} \n","start":64605,"end":64835}]},{"id":"S040","semantic_type":"definition","kind":"definition","labels":["eq:fw_operator"],"declared_external":false,"source_only":false,"title":"前向后向残差映射","source_units":[{"raw_tex":"To prove \\eqref{eq:condition}, we define for every $k\\geq 1$ the operator $G_k:\\mathcal{Y}\\to \\mathcal{Y}$ as\n\\begin{equation}  \\label{eq:fw_operator}\nG_k(z)  := \\frac{1}{\\tau}\\Big(z-\\text{prox}_{\\tau \\ell_k}(z-\\tau \\nabla  \\RA(z))\\Big)\n\\end{equation}\nand notice that the proximal step of  \\ref{alg:fw_accel} can be written in terms of $G_k$ as $u_{k+1}=w_k - \\tau G_k(w_k)$. The descent lemma yields (see, e.g., \\cite{AttouchTikAccel,ChambolleDossal2015})\n","start":64835,"end":65292}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":["eq:descent"],"declared_external":false,"source_only":false,"title":"调用复合下降不等式","source_units":[{"raw_tex":"and notice that the proximal step of  \\ref{alg:fw_accel} can be written in terms of $G_k$ as $u_{k+1}=w_k - \\tau G_k(w_k)$. The descent lemma yields (see, e.g., \\cite{AttouchTikAccel,ChambolleDossal2015})\n\\begin{equation}  \\label{eq:descent}\n    d_{\\lambda_k}(w-\\tau G_k(w)) \\leq  d_{\\lambda_k}(u) + \\langle G_k(w), w-u\\rangle - \\frac{\\tau}{2}\\|G_k(w)\\|^2, \\quad \\text{for all }w,u\\in\\mathcal{Y},\n\\end{equation}\n","start":65087,"end":65499}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":["descent:first"],"declared_external":false,"source_only":false,"title":"与当前迭代点比较","source_units":[{"raw_tex":"Evaluating \\eqref{eq:descent} for $u=u_{k}$ and $w=w_k$, we get\n\\begin{equation}  \\label{descent:first}\nd_{{\\lambda_k}}(u_{k+1}) \\leq  d_{{\\lambda_k}}(u_{k}) + \\langle G_k(w_k), w_k - u_{k}\\rangle - \\frac{\\tau}{2}\\|G_k(w_k)\\|^2.\n\\end{equation}\n","start":65499,"end":65743}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":["descent:second"],"declared_external":false,"source_only":false,"title":"与对偶极小点比较","source_units":[{"raw_tex":"Evaluating \\eqref{eq:descent} for $u=u^\\dagger$ and $w=w_k$ we derive\n \\begin{equation}  \\label{descent:second}\nd_{{\\lambda_k}}(u_{k+1}) \\leq  d_{{\\lambda_k}}(u^\\dagger) + \\langle G_k(w_k), w_k -u^\\dagger\\rangle - \\frac{\\tau}{2}\\|G_k(w_k)\\|^2.\n\\end{equation}\n","start":65743,"end":66002}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":["ineq:1"],"declared_external":false,"source_only":false,"title":"加权合并下降估计","source_units":[{"raw_tex":"We now multiply \\eqref{descent:first} by $t_{k+1}-1$ and we add it to \\eqref{descent:second}, and we obtain\n\\begin{align}\nt_{k+1} d_{{\\lambda_k}}(u_{k+1}) \n&\\leq  (t_{k+1}-1)d_{{\\lambda_k}}(u_{k}) +  d_{{\\lambda_k}}(u^\\dagger)  \\nonumber\\\\\n\\label{ineq:1}&+ \\langle  G_k(w_k), (t_{k+1}-1)(w_k-u_{k}) + (w_{k} - u^\\dagger)\\rangle  -\\frac{\\tau}{2}t_{k+1}\\|  G_k(w_k)\\|^2 . \n\\end{align}\n","start":66002,"end":66385}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["eq:property_zk"],"declared_external":false,"source_only":false,"title":"识别惯性线性组合","source_units":[{"raw_tex":"As an immediate consequence of Lemma \\ref{lemma:sequence}, we observe the following property:\n\\begin{align}  \\label{eq:property_zk}\n(t_{k+1}-1)(w_k-u_{k}) + w_k & = u_{k} + t_{k+1}(w_k -u_{k}) \\notag\\\\\n& = u_{k} + t_{k+1} \\alpha_k(u_{k} -u_{k-1}) \\\\\n& = u_{k-1} + (1+ t_{k+1} \\alpha_k)(u_{k} -u_{k-1}) \\notag\\\\\n& = u_{k-1} +t_k(u_{k} -u_{k-1}) = z_k. \\notag\n\\end{align}\n","start":66385,"end":66755}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"改写为平方范数差","source_units":[{"raw_tex":"Thanks to \\eqref{eq:def_zk:noisefree}, the fact that $z_k-\\tau t_{k+1}G_k(w_k)=z_{k+1}$ and the previous equality, we can reorder the terms in \\eqref{ineq:1} and rewrite it as\n\\begin{align}\nt_{k+1} ( d_{{\\lambda_k}}(u_{k+1}) -  d_{{\\lambda_k}}(u^\\dagger)) \n& \\leq  (t_{k+1}-1) (d_{{\\lambda_k}}(u_{k}) -  d_{{\\lambda_k}}(u^\\dagger)) \\notag\\\\\n& + \\frac{1}{2\\tau t_{k+1}}\\left(\\|z_{k}-u^\\dagger\\|^2-\\|z_{k+1}-u^\\dagger\\|^2\\right). \\notag \n\\end{align}\n","start":66755,"end":67203}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":["descent:third"],"declared_external":false,"source_only":false,"title":"建立平方权重估计","source_units":[{"raw_tex":"We now multiply everything by $t_{k+1}$, re-arrange and get\n\\begin{align}  \\label{descent:third}\n& t^2_{k+1} ( d_{{\\lambda_k}}(u_{k+1}) -  d_{{\\lambda_k}}(u^\\dagger))   + \\frac{1}{2\\tau}\n\\|z_{k+1}-u^\\dagger\\|^2   \\\\\n&\\leq  (t^2_{k+1}-t_{k+1}) (d_{{\\lambda_k}}(u_{k}) -  d_{\\lambda_k}(u^\\dagger)) + \\frac{1}{2\\tau } \\|z_{k}-u^\\dagger\\|^2.  \\notag\n\\end{align}\n","start":67203,"end":67561}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":["descent:third2"],"declared_external":false,"source_only":false,"title":"分离权重余项","source_units":[{"raw_tex":"Equivalently:\n\\begin{align}  \\label{descent:third2}\n& t^2_{k+1} \\Big( d_{{\\lambda_k}}(u_{k+1}) -  d_{{\\lambda_k}}(u^\\dagger) \\Big)   + \\frac{1}{2\\tau}\n\\|z_{k+1}-u^\\dagger\\|^2 \\nonumber  \\\\\n&\\leq  t^2_k \\Big(d_{{\\lambda_k}}(u_{k}) -  d_{{\\lambda_k}}(u^\\dagger)\\Big) +   (t^2_{k+1}-t_{k+1} - t_{k}^2) \\Big( d_{{\\lambda_k}}(u_{k}) -  d_{{\\lambda_k}}(u^\\dagger) \\Big)+ \\frac{1}{2\\tau } \\|z_{k}-u^\\dagger\\|^2. \\notag\n\\end{align}\n","start":67561,"end":67985}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["descent:fifth"],"declared_external":false,"source_only":false,"title":"更换对角目标并移基准","source_units":[{"raw_tex":"To get the desired terms, we first use on the left-hand side  the monotonicity property of the function $ d_{{\\lambda_k}}(\\cdot)$ as a function of $k$ (see  Lemma  \\ref{L:functional properties}\\ref{L:functional properties:decreasing function sequence}) and then add and subtract in the parentheses  the term $\\inf d_0$, thus getting:\n\\begin{align}  \\label{descent:fifth}\n&t^2_{k+1} \\Big( d_{{\\lambda_{k+1}}}(u_{k+1}) -   \\inf d_0 \\Big)   + \\frac{1}{2\\tau}\n\\|z_{k+1}-u^\\dagger\\|^2   \\notag \\\\\n&\\leq  t^2_k \\Big(d_{{\\lambda_{k}}}(u_{k}) - \\inf d_0 \\Big) +   (t^2_{k+1}-t_{k+1} - t_{k}^2) \\Big( d_{{\\lambda_{k}}}(u_{k}) -  \\inf d_0\\Big) \\\\\n& +t_{k+1} \\Big(d_{{\\lambda_{k}}}(u^\\dagger) -  \\inf d_0\\Big)   +  \\frac{1}{2\\tau } \\|z_{k}-u^\\dagger\\|^2. \\notag\n\\end{align}\n","start":67985,"end":68748}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":["descent:fifth"],"declared_external":false,"source_only":false,"title":"写成能量加余项","source_units":[{"raw_tex":"After rearranging and using the definition of $\\mathcal{E}$ in \\eqref{eq:discrete energy}, we deduce:\n\\begin{align}  \\label{descent:fifth}\n\\mathcal{E}(k+1) +    (t^2_{k}+t_{k+1} - t_{k+1}^2) \\Big( d_{{\\lambda_{k}}}(u_{k}) -  \\inf d_0\\Big) \\leq  \\mathcal{E}(k) + t_{k+1} \\Big(d_{{\\lambda_{k}}}(u^\\dagger) -  \\inf d_0\\Big). \\nonumber\n\\end{align}\n","start":68748,"end":69092}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"丢弃非负余项","source_units":[{"raw_tex":"\\noindent Thanks to \\eqref{eq:properties_sequences_alphak_tk} and  Lemma~\\ref{L:functional properties}\\ref{L:functional properties:decreasing function sequence}, we can neglect the second term  in the left-hand side of the above inequality,  finally getting the desired  \\eqref{eq:condition}.\n","start":69093,"end":69386}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":["eq:decay_complete"],"declared_external":false,"source_only":false,"title":"递归累加能量界","source_units":[{"raw_tex":"Iterating \\eqref{eq:condition}  recursively gives\n\\begin{align}  \\label{eq:decay_complete}\n\\mathcal{E}(k) \\leq \\mathcal{E}(1) +\\sum_{j=1}^{k-1}t_{j+1}\\Big( d_{\\lambda_{j}}(u^\\dagger)-\\inf d_0 \\Big).\n\\end{align}\n","start":69386,"end":69597}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"统一控制对角残差","source_units":[{"raw_tex":"To  bound the sum appearing on the right hand side, we need to analyze the residuals $r_j:= d_{\\lambda_{j}}(u^\\dagger) -\\inf d_0$. Similarly as for the estimation obtained in the continuous case, we can use for this purpose the property in Lemma \\ref{L:functional properties}\\ref{L:functional properties:diagonal rates via conditioning} and get that for some fixed constant $c$ independent on $j$ (defined analogously as in \\eqref{proof:ccd3}), we have\n$$\nr_j\\leq c \\lambda_j^{\\frac{1}{q-1}},\\qquad\\text{for every }j\\geq 1.\n$$\n","start":69597,"end":70124}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"残差加权和有界","source_units":[{"raw_tex":"By assumption \\ref{assumption:parameter:relaxation}, with $\\Lambda$ as  in \\eqref{def:biglambda}, we thus conclude that\n$$\n\\sum_{j=1}^{k-1} t_{j+1} r_j \\leq  c \\sum_{j=1}^{k-1} t_{j+1} \\lambda_j^{\\frac{1}{q-1}}  \\leq c \\Lambda < +\\infty\n$$\n","start":70124,"end":70364}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"提取对偶收敛率","source_units":[{"raw_tex":"This allows us to deduce from \\eqref{eq:decay_complete} the convergence rate on the dual values in \\eqref{eq:rate}, by taking $C = \\Ee(1) + c \\Lambda$.\n","start":70364,"end":70516}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"转换原变量误差界","source_units":[{"raw_tex":"Finally, the convergence rate on the primal iterates  in \\eqref{eq:rate} follows from  Lemma \\ref{L:functional properties}\\ref{L:functional properties:from dual value to primal iterates}.\n","start":70516,"end":70704}]},{"id":"S057","semantic_type":"lemma","kind":"lemma","labels":["L:functional properties","L:functional properties:L at 0","L:functional properties:R_A Lipschitz","L:functional properties:argmin d_0 nonempty","L:functional properties:decreasing function sequence","L:functional properties:diagonal rates via conditioning","L:functional properties:from dual value to primal iterates"],"declared_external":false,"source_only":false,"title":"对偶函数的六项性质","source_units":[{"raw_tex":"\\begin{lemma}\\label{L:functional properties}\nUnder the assumptions \\ref{assumption:data-fit convex and coercive}-\\ref{assumption:data-fit conditionned locally} and \\ref{H:R strongly convex}-\\ref{assumption:AR2},  we have:\n\\begin{enumerate}[label=(\\roman*)]\n\t\\item\\label{L:functional properties:R_A Lipschitz} $\\Rr_A$ is differentiable and $\\nabla  \\RA$ is Lipschitz continuous, with Lipschitz constant equal to $\\sigma^{-1} \\Vert A \\Vert^2$.\n\t\\item\\label{L:functional properties:L at 0} $\\forall y \\in \\Yy$, $\\ell^*_{y}(0)=0$ and $\\partial \\ell^*_y(0)=\\{y\\}$.\n\t\\item\\label{L:functional properties:argmin d_0 nonempty} There holds: $\\argmin  d_0 \\neq \\emptyset$.\n \t\\item\\label{L:functional properties:decreasing function sequence} $\\forall u \\in \\Yy$, the function $\\lambda\\in [0,+\\infty) \\mapsto \\ d_{\\lambda}(u)$ is nondecreasing. \n\t\\item\\label{L:functional properties:from dual value to primal iterates} $\\forall t >0, \\forall u \\in \\Yy, \\ \\text{ if }\n{x}:=\\nabla R^*(-A^*u),$ then $\n \\frac{\\sigma}{2} \\Vert {x} - x^\\dagger \\Vert^2 \\leq d_0(u) - \\inf d_0.$ \n \t\\item\\label{L:functional properties:diagonal rates via conditioning} $\\forall u^\\dagger \\in \\argmin  d_0$, if  $\\lambda \\Vert u^\\dagger \\Vert \\leq \\frac{\\gamma}{q} \\varrho^{q-1}$, then\n \t\\begin{equation}\n \t    d_{\\lambda}(u^\\dagger) - \\inf  d_0 \\leq\n \t    \\begin{cases}\n \t     0 & \\text{ if } q=1, \\\\\n \t     (1-\\frac{1}{q}) {\\gamma^{-1/(q-1)}} \\Vert u^\\dagger \\Vert^{q/(q-1)} \\lambda^{1/(q-1)} & \\text{ if } q>1.\n \t    \\end{cases}\n \t\\end{equation}\n\\end{enumerate}\n\\end{lemma}","start":115130,"end":116667}]},{"id":"S058","semantic_type":"lemma","kind":"lemma","labels":["L:functional properties","L:functional properties:L at 0","L:functional properties:R_A Lipschitz","L:functional properties:argmin d_0 nonempty","L:functional properties:decreasing function sequence","L:functional properties:diagonal rates via conditioning","L:functional properties:from dual value to primal iterates"],"declared_external":false,"source_only":false,"title":"复合项的 Lipschitz 梯度","source_units":[{"raw_tex":"\t\\item\\label{L:functional properties:R_A Lipschitz} $\\Rr_A$ is differentiable and $\\nabla  \\RA$ is Lipschitz continuous, with Lipschitz constant equal to $\\sigma^{-1} \\Vert A \\Vert^2$.\n","start":115387,"end":115572}]},{"id":"S059","semantic_type":"lemma","kind":"lemma","labels":["L:functional properties","L:functional properties:L at 0","L:functional properties:R_A Lipschitz","L:functional properties:argmin d_0 nonempty","L:functional properties:decreasing function sequence","L:functional properties:diagonal rates via conditioning","L:functional properties:from dual value to primal iterates"],"declared_external":false,"source_only":false,"title":"数据共轭在零处的性质","source_units":[{"raw_tex":"\t\\item\\label{L:functional properties:L at 0} $\\forall y \\in \\Yy$, $\\ell^*_{y}(0)=0$ and $\\partial \\ell^*_y(0)=\\{y\\}$.\n","start":115572,"end":115690}]},{"id":"S060","semantic_type":"lemma","kind":"lemma","labels":["L:functional properties","L:functional properties:L at 0","L:functional properties:R_A Lipschitz","L:functional properties:argmin d_0 nonempty","L:functional properties:decreasing function sequence","L:functional properties:diagonal rates via conditioning","L:functional properties:from dual value to primal iterates"],"declared_external":false,"source_only":false,"title":"极限对偶解存在","source_units":[{"raw_tex":"\t\\item\\label{L:functional properties:argmin d_0 nonempty} There holds: $\\argmin  d_0 \\neq \\emptyset$.\n","start":115690,"end":115792}]},{"id":"S061","semantic_type":"lemma","kind":"lemma","labels":["L:functional properties","L:functional properties:L at 0","L:functional properties:R_A Lipschitz","L:functional properties:argmin d_0 nonempty","L:functional properties:decreasing function sequence","L:functional properties:diagonal rates via conditioning","L:functional properties:from dual value to primal iterates"],"declared_external":false,"source_only":false,"title":"对偶目标随参数单调","source_units":[{"raw_tex":" \t\\item\\label{L:functional properties:decreasing function sequence} $\\forall u \\in \\Yy$, the function $\\lambda\\in [0,+\\infty) \\mapsto \\ d_{\\lambda}(u)$ is nondecreasing. \n","start":115792,"end":115963}]},{"id":"S062","semantic_type":"lemma","kind":"lemma","labels":["L:functional properties","L:functional properties:L at 0","L:functional properties:R_A Lipschitz","L:functional properties:argmin d_0 nonempty","L:functional properties:decreasing function sequence","L:functional properties:diagonal rates via conditioning","L:functional properties:from dual value to primal iterates"],"declared_external":false,"source_only":false,"title":"对偶间隙控制原误差","source_units":[{"raw_tex":"\t\\item\\label{L:functional properties:from dual value to primal iterates} $\\forall t >0, \\forall u \\in \\Yy, \\ \\text{ if }\n{x}:=\\nabla R^*(-A^*u),$ then $\n \\frac{\\sigma}{2} \\Vert {x} - x^\\dagger \\Vert^2 \\leq d_0(u) - \\inf d_0.$ \n","start":115963,"end":116190}]},{"id":"S063","semantic_type":"lemma","kind":"lemma","labels":["L:functional properties","L:functional properties:L at 0","L:functional properties:R_A Lipschitz","L:functional properties:argmin d_0 nonempty","L:functional properties:decreasing function sequence","L:functional properties:diagonal rates via conditioning","L:functional properties:from dual value to primal iterates"],"declared_external":false,"source_only":false,"title":"条件性给出的残差界","source_units":[{"raw_tex":" \t\\item\\label{L:functional properties:diagonal rates via conditioning} $\\forall u^\\dagger \\in \\argmin  d_0$, if  $\\lambda \\Vert u^\\dagger \\Vert \\leq \\frac{\\gamma}{q} \\varrho^{q-1}$, then\n \t\\begin{equation}\n \t    d_{\\lambda}(u^\\dagger) - \\inf  d_0 \\leq\n \t    \\begin{cases}\n \t     0 & \\text{ if } q=1, \\\\\n \t     (1-\\frac{1}{q}) {\\gamma^{-1/(q-1)}} \\Vert u^\\dagger \\Vert^{q/(q-1)} \\lambda^{1/(q-1)} & \\text{ if } q>1.\n \t    \\end{cases}\n \t\\end{equation}\n","start":116190,"end":116640}]},{"id":"S064","semantic_type":"assumption","kind":"assumption","labels":["L:functional properties","L:functional properties:L at 0","L:functional properties:R_A Lipschitz","L:functional properties:argmin d_0 nonempty","L:functional properties:decreasing function sequence","L:functional properties:diagonal rates via conditioning","L:functional properties:from dual value to primal iterates"],"declared_external":false,"source_only":false,"title":"残差结论的局部适用域","source_units":[{"raw_tex":" \t\\item\\label{L:functional properties:diagonal rates via conditioning} $\\forall u^\\dagger \\in \\argmin  d_0$, if  $\\lambda \\Vert u^\\dagger \\Vert \\leq \\frac{\\gamma}{q} \\varrho^{q-1}$, then\n","start":116190,"end":116377}]},{"id":"S065","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"六项性质的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{L:functional properties:R_A Lipschitz}: follows  from the strong convexity of $R$, see, e.g.,  \n\\cite[Theorem 18.15]{BauschkeCombettes2017}.\\\\\n\\ref{L:functional properties:L at 0}: it is a simple consequence of the properties of the Fenchel transform as it can be found, e.g., in \n \\cite[Proposition 13.10(i) \\& Corollary 16.30]{BauschkeCombettes2017}.\\\\\n\\ref{L:functional properties:argmin d_0 nonempty} and \\ref{L:functional properties:from dual value to primal iterates}: follow from \\cite[Lemma 5]{Garrigos2017} by simply taking $f=R$ and $g=\\delta_{\\{\\bar y\\}}$, while property \\ref{L:functional properties:decreasing function sequence} has been proved in \\cite[Proposition 2(i)]{Garrigos2017}.\\\\\n\\ref{L:functional properties:diagonal rates via conditioning}: it is enough to verify that $\\ell_{\\bar y}(\\cdot)$ is $q$-well conditioned in the sense of \\cite[Definition 1]{Garrigos2017}, while assumption \\ref{assumption:data-fit conditionned locally} holds only locally.\nTo check this, we introduce the function $\\psi : \\R \\rightarrow \\R$ defined for the $\\varrho>0$ appearing in \\ref{assumption:data-fit conditionned locally}  by:\n\\begin{equation}\n \\psi  t \\mapsto\n\\begin{cases}\n \\frac{\\gamma}{q}\\vert t \\vert^q  & \\text{ if } \\vert t \\vert \\leq \\varrho, \\\\\n \\frac{\\gamma}{q} \\varrho^{q-1} \\vert t \\vert & \\text{ if } \\vert t \\vert > \\varrho.\n\\end{cases}\n\\end{equation}\nFrom \\ref{assumption:data-fit conditionned locally}, we easily  deduce that $\\ell_{\\bar y}(y) \\geq \\psi(\\Vert y- \\bar y \\Vert)$ for all $y \\in \\Yy$ (see \\cite[Corollary 3.4.2]{ZalinescuBook2002}).\nNote that $\\psi$ is not convex for $q >1$, so in this case we consider instead the function\n\\begin{equation}\n m: \\R \\rightarrow \\R , \\quad t \\mapsto \n\\begin{cases}\n \\frac{\\gamma}{q}\\vert t \\vert^q  & \\text{ if } \\vert t \\vert \\leq q^{1/(1-q)} \\varrho, \\\\\n \\frac{\\gamma}{q}\\varrho^{q-1}\\vert t \\vert \n -  \\frac{\\gamma}{q^{\\frac{q}{q-1}}}\\varrho^q (1 - \\frac{1}{q})  & \\text{ if } \\vert t \\vert > q^{1/(1-q)} \\varrho,\n\\end{cases}\n\\end{equation}\nand define $m:=\\psi$ for $q=1$.\nIt is an easy exercise to verify that $m$ is indeed a convex function on $\\R$, and that $m(w) \\leq \\psi(w)$ for all $w\\in\\R$.\nNow, we can make use of \\cite[Lemma 2]{Garrigos2017},\nwhich tells us that $d_\\lambda(u) - \\inf d_0 \\leq \\lambda^{-1} m^*(\\Vert u \\Vert \\lambda)$.\nThe desired result  now follows from the computation of the Fenchel transform of $m$.\nIf $q=1$, we have that $m(t) = \\gamma \\vert t \\vert$, so classic Fenchel calculus entails that $m^*$ is $\\delta_{[-\\gamma,\\gamma]}$, the indicator function of $[-\\gamma,\\gamma]$.\nIf $q>1$, easy computations show that $m^*$ reads\n\\begin{equation}\n m^* : \\R \\rightarrow \\R, \\quad s \\mapsto\n\\begin{cases}\n (1 - \\frac{1}{q}) \\gamma^{-1/(q-1)}\\vert s \\vert^{\\frac{q}{q-1}}  & \\text{ if } \\vert s \\vert \\leq \\frac{\\gamma}{q} \\varrho^{q-1}, \\\\\n + \\infty  & \\text{ if } \\vert s \\vert > \\frac{\\gamma}{q} \\varrho^{q-1}.\n\\end{cases}\n\\end{equation}\nBy now applying \\cite[Lemma 2]{Garrigos2017} we  conclude. \n\\end{proof}","start":116668,"end":119700}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"由强凸性得到光滑性","source_units":[{"raw_tex":"\\ref{L:functional properties:R_A Lipschitz}: follows  from the strong convexity of $R$, see, e.g.,  \n\\cite[Theorem 18.15]{BauschkeCombettes2017}.\\\\\n","start":116682,"end":116830}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"引用共轭零点性质","source_units":[{"raw_tex":"\\ref{L:functional properties:L at 0}: it is a simple consequence of the properties of the Fenchel transform as it can be found, e.g., in \n \\cite[Proposition 13.10(i) \\& Corollary 16.30]{BauschkeCombettes2017}.\\\\\n","start":116830,"end":117042}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"特化对偶误差引理","source_units":[{"raw_tex":"\\ref{L:functional properties:argmin d_0 nonempty} and \\ref{L:functional properties:from dual value to primal iterates}: follow from \\cite[Lemma 5]{Garrigos2017} by simply taking $f=R$ and $g=\\delta_{\\{\\bar y\\}}$, while property \\ref{L:functional properties:decreasing function sequence} has been proved in \\cite[Proposition 2(i)]{Garrigos2017}.\\\\\n","start":117042,"end":117389}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"引用参数单调性","source_units":[{"raw_tex":"\\ref{L:functional properties:argmin d_0 nonempty} and \\ref{L:functional properties:from dual value to primal iterates}: follow from \\cite[Lemma 5]{Garrigos2017} by simply taking $f=R$ and $g=\\delta_{\\{\\bar y\\}}$, while property \\ref{L:functional properties:decreasing function sequence} has been proved in \\cite[Proposition 2(i)]{Garrigos2017}.\\\\\n","start":117042,"end":117389}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"转向全局良态条件","source_units":[{"raw_tex":"\\ref{L:functional properties:diagonal rates via conditioning}: it is enough to verify that $\\ell_{\\bar y}(\\cdot)$ is $q$-well conditioned in the sense of \\cite[Definition 1]{Garrigos2017}, while assumption \\ref{assumption:data-fit conditionned locally} holds only locally.\n","start":117389,"end":117662}]},{"id":"S071","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"局部幂次与线性延拓","source_units":[{"raw_tex":"To check this, we introduce the function $\\psi : \\R \\rightarrow \\R$ defined for the $\\varrho>0$ appearing in \\ref{assumption:data-fit conditionned locally}  by:\n\\begin{equation}\n \\psi  t \\mapsto\n\\begin{cases}\n \\frac{\\gamma}{q}\\vert t \\vert^q  & \\text{ if } \\vert t \\vert \\leq \\varrho, \\\\\n \\frac{\\gamma}{q} \\varrho^{q-1} \\vert t \\vert & \\text{ if } \\vert t \\vert > \\varrho.\n\\end{cases}\n\\end{equation}\n","start":117662,"end":118062}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"获得全局径向下界","source_units":[{"raw_tex":"From \\ref{assumption:data-fit conditionned locally}, we easily  deduce that $\\ell_{\\bar y}(y) \\geq \\psi(\\Vert y- \\bar y \\Vert)$ for all $y \\in \\Yy$ (see \\cite[Corollary 3.4.2]{ZalinescuBook2002}).\n","start":118062,"end":118259}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"构造凸下界","source_units":[{"raw_tex":"Note that $\\psi$ is not convex for $q >1$, so in this case we consider instead the function\n\\begin{equation}\n m: \\R \\rightarrow \\R , \\quad t \\mapsto \n\\begin{cases}\n \\frac{\\gamma}{q}\\vert t \\vert^q  & \\text{ if } \\vert t \\vert \\leq q^{1/(1-q)} \\varrho, \\\\\n \\frac{\\gamma}{q}\\varrho^{q-1}\\vert t \\vert \n -  \\frac{\\gamma}{q^{\\frac{q}{q-1}}}\\varrho^q (1 - \\frac{1}{q})  & \\text{ if } \\vert t \\vert > q^{1/(1-q)} \\varrho,\n\\end{cases}\n\\end{equation}\nand define $m:=\\psi$ for $q=1$.\nIt is an easy exercise to verify that $m$ is indeed a convex function on $\\R$, and that $m(w) \\leq \\psi(w)$ for all $w\\in\\R$.\n","start":118259,"end":118860}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"调用标量共轭残差界","source_units":[{"raw_tex":"Now, we can make use of \\cite[Lemma 2]{Garrigos2017},\nwhich tells us that $d_\\lambda(u) - \\inf d_0 \\leq \\lambda^{-1} m^*(\\Vert u \\Vert \\lambda)$.\nThe desired result  now follows from the computation of the Fenchel transform of $m$.\n","start":118860,"end":119092}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"线性条件性的共轭","source_units":[{"raw_tex":"If $q=1$, we have that $m(t) = \\gamma \\vert t \\vert$, so classic Fenchel calculus entails that $m^*$ is $\\delta_{[-\\gamma,\\gamma]}$, the indicator function of $[-\\gamma,\\gamma]$.\n","start":119092,"end":119271}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"幂次条件性的共轭","source_units":[{"raw_tex":"If $q>1$, easy computations show that $m^*$ reads\n\\begin{equation}\n m^* : \\R \\rightarrow \\R, \\quad s \\mapsto\n\\begin{cases}\n (1 - \\frac{1}{q}) \\gamma^{-1/(q-1)}\\vert s \\vert^{\\frac{q}{q-1}}  & \\text{ if } \\vert s \\vert \\leq \\frac{\\gamma}{q} \\varrho^{q-1}, \\\\\n + \\infty  & \\text{ if } \\vert s \\vert > \\frac{\\gamma}{q} \\varrho^{q-1}.\n\\end{cases}\n\\end{equation}\n","start":119271,"end":119629}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"完成分情况残差估计","source_units":[{"raw_tex":"By now applying \\cite[Lemma 2]{Garrigos2017} we  conclude. \n","start":119629,"end":119689}]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先修列表：第1188–1191行引用的 AttouchTikAccel 与 ChambolleDossal2015 的 forward-backward descent inequality。"],"source_only":false,"title":"前向后向下降引理","source_units":[]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先修列表：BauschkeCombettes2017, Proposition 13.10(i) 与 Corollary 16.30；第1946–1947行明确调用。"],"source_only":false,"title":"Fenchel 零点与极小点规则","source_units":[]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先修列表：Garrigos2017, Lemma 5；第1948行取 f=R、g=\\delta_{\\{\\bar y\\}}。"],"source_only":false,"title":"源条件下的对偶误差引理","source_units":[]},{"id":"S081","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先修列表：Garrigos2017, Proposition 2(i)；第1948行明确调用。"],"source_only":false,"title":"对角目标的参数单调性","source_units":[]},{"id":"S082","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先修列表：ZalinescuBook2002, Corollary 3.4.2；第1958行明确调用。"],"source_only":false,"title":"凸函数局部下界延拓","source_units":[]},{"id":"S083","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先修列表：Garrigos2017, Lemma 2；第1970–1971及1982行调用。"],"source_only":false,"title":"由标量下界到对偶残差","source_units":[]},{"id":"S084","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先修列表：Hilbert-space norm identities；implicit standard fact made explicit by annotator。"],"source_only":false,"title":"Hilbert 平方范数恒等式","source_units":[]},{"id":"S085","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先修列表：standard convex conjugacy、subgradient calculus；implicit standard fact made explicit by annotator。"],"source_only":false,"title":"复合梯度与标量共轭规则","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S034","edges":[{"source_id":"S001","target_id":"S019","kind":"supports"},{"source_id":"S008","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S014","target_id":"S017","kind":"supports"},{"source_id":"S007","target_id":"S023","kind":"supports"},{"source_id":"S021","target_id":"S023","kind":"supports"},{"source_id":"S005","target_id":"S015","kind":"supports"},{"source_id":"S011","target_id":"S026","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S021","target_id":"S033","kind":"supports"},{"source_id":"S020","target_id":"S033","kind":"supports"},{"source_id":"S019","target_id":"S033","kind":"supports"},{"source_id":"S012","target_id":"S034","kind":"supports"},{"source_id":"S013","target_id":"S034","kind":"supports"},{"source_id":"S014","target_id":"S034","kind":"supports"},{"source_id":"S015","target_id":"S034","kind":"supports"},{"source_id":"S016","target_id":"S034","kind":"supports"},{"source_id":"S025","target_id":"S034","kind":"supports"},{"source_id":"S026","target_id":"S034","kind":"supports"},{"source_id":"S027","target_id":"S034","kind":"supports"},{"source_id":"S028","target_id":"S034","kind":"supports"},{"source_id":"S023","target_id":"S034","kind":"supports"},{"source_id":"S028","target_id":"S036","kind":"supports"},{"source_id":"S019","target_id":"S036","kind":"supports"},{"source_id":"S020","target_id":"S036","kind":"supports"},{"source_id":"S026","target_id":"S036","kind":"supports"},{"source_id":"S025","target_id":"S036","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S021","target_id":"S039","kind":"supports"},{"source_id":"S027","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S007","target_id":"S040","kind":"supports"},{"source_id":"S021","target_id":"S040","kind":"supports"},{"source_id":"S025","target_id":"S040","kind":"supports"},{"source_id":"S023","target_id":"S040","kind":"supports"},{"source_id":"S078","target_id":"S041","kind":"supports"},{"source_id":"S058","target_id":"S041","kind":"supports"},{"source_id":"S025","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S043","kind":"supports"},{"source_id":"S040","target_id":"S043","kind":"supports"},{"source_id":"S028","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S026","target_id":"S044","kind":"supports"},{"source_id":"S031","target_id":"S045","kind":"supports"},{"source_id":"S023","target_id":"S045","kind":"supports"},{"source_id":"S037","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S037","target_id":"S046","kind":"supports"},{"source_id":"S040","target_id":"S046","kind":"supports"},{"source_id":"S084","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S026","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S061","target_id":"S049","kind":"supports"},{"source_id":"S027","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S036","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S032","target_id":"S051","kind":"supports"},{"source_id":"S061","target_id":"S051","kind":"supports"},{"source_id":"S019","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S063","target_id":"S053","kind":"supports"},{"source_id":"S028","target_id":"S053","kind":"supports"},{"source_id":"S027","target_id":"S053","kind":"supports"},{"source_id":"S022","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S027","target_id":"S054","kind":"supports"},{"source_id":"S052","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S036","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S062","target_id":"S056","kind":"supports"},{"source_id":"S061","target_id":"S056","kind":"supports"},{"source_id":"S023","target_id":"S056","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S035","kind":"supports"},{"source_id":"S039","target_id":"S035","kind":"supports"},{"source_id":"S040","target_id":"S035","kind":"supports"},{"source_id":"S041","target_id":"S035","kind":"supports"},{"source_id":"S042","target_id":"S035","kind":"supports"},{"source_id":"S043","target_id":"S035","kind":"supports"},{"source_id":"S044","target_id":"S035","kind":"supports"},{"source_id":"S045","target_id":"S035","kind":"supports"},{"source_id":"S046","target_id":"S035","kind":"supports"},{"source_id":"S047","target_id":"S035","kind":"supports"},{"source_id":"S048","target_id":"S035","kind":"supports"},{"source_id":"S049","target_id":"S035","kind":"supports"},{"source_id":"S050","target_id":"S035","kind":"supports"},{"source_id":"S051","target_id":"S035","kind":"supports"},{"source_id":"S052","target_id":"S035","kind":"supports"},{"source_id":"S053","target_id":"S035","kind":"supports"},{"source_id":"S054","target_id":"S035","kind":"supports"},{"source_id":"S055","target_id":"S035","kind":"supports"},{"source_id":"S056","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S012","target_id":"S057","kind":"supports"},{"source_id":"S013","target_id":"S057","kind":"supports"},{"source_id":"S014","target_id":"S057","kind":"supports"},{"source_id":"S015","target_id":"S057","kind":"supports"},{"source_id":"S016","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S059","kind":"supports"},{"source_id":"S057","target_id":"S060","kind":"supports"},{"source_id":"S057","target_id":"S061","kind":"supports"},{"source_id":"S057","target_id":"S062","kind":"supports"},{"source_id":"S057","target_id":"S063","kind":"supports"},{"source_id":"S015","target_id":"S066","kind":"supports"},{"source_id":"S009","target_id":"S066","kind":"supports"},{"source_id":"S021","target_id":"S066","kind":"supports"},{"source_id":"S001","target_id":"S066","kind":"supports"},{"source_id":"S085","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S058","kind":"supports"},{"source_id":"S079","target_id":"S067","kind":"supports"},{"source_id":"S012","target_id":"S067","kind":"supports"},{"source_id":"S013","target_id":"S067","kind":"supports"},{"source_id":"S014","target_id":"S067","kind":"supports"},{"source_id":"S008","target_id":"S067","kind":"supports"},{"source_id":"S067","target_id":"S059","kind":"supports"},{"source_id":"S080","target_id":"S068","kind":"supports"},{"source_id":"S015","target_id":"S068","kind":"supports"},{"source_id":"S016","target_id":"S068","kind":"supports"},{"source_id":"S018","target_id":"S068","kind":"supports"},{"source_id":"S019","target_id":"S068","kind":"supports"},{"source_id":"S068","target_id":"S060","kind":"supports"},{"source_id":"S068","target_id":"S062","kind":"supports"},{"source_id":"S081","target_id":"S069","kind":"supports"},{"source_id":"S020","target_id":"S069","kind":"supports"},{"source_id":"S019","target_id":"S069","kind":"supports"},{"source_id":"S069","target_id":"S061","kind":"supports"},{"source_id":"S014","target_id":"S070","kind":"supports"},{"source_id":"S014","target_id":"S071","kind":"supports"},{"source_id":"S012","target_id":"S072","kind":"supports"},{"source_id":"S014","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S082","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S073","kind":"supports"},{"source_id":"S014","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S083","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S075","kind":"supports"},{"source_id":"S008","target_id":"S075","kind":"supports"},{"source_id":"S085","target_id":"S075","kind":"supports"},{"source_id":"S073","target_id":"S076","kind":"supports"},{"source_id":"S008","target_id":"S076","kind":"supports"},{"source_id":"S085","target_id":"S076","kind":"supports"},{"source_id":"S074","target_id":"S077","kind":"supports"},{"source_id":"S075","target_id":"S077","kind":"supports"},{"source_id":"S076","target_id":"S077","kind":"supports"},{"source_id":"S064","target_id":"S077","kind":"supports"},{"source_id":"S077","target_id":"S063","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"},{"source_id":"S067","target_id":"S065","kind":"supports"},{"source_id":"S068","target_id":"S065","kind":"supports"},{"source_id":"S069","target_id":"S065","kind":"supports"},{"source_id":"S070","target_id":"S065","kind":"supports"},{"source_id":"S071","target_id":"S065","kind":"supports"},{"source_id":"S072","target_id":"S065","kind":"supports"},{"source_id":"S073","target_id":"S065","kind":"supports"},{"source_id":"S074","target_id":"S065","kind":"supports"},{"source_id":"S075","target_id":"S065","kind":"supports"},{"source_id":"S076","target_id":"S065","kind":"supports"},{"source_id":"S077","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S057","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S034","edges":[{"source_id":"S001","target_id":"S019","kind":"supports"},{"source_id":"S008","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S014","target_id":"S017","kind":"supports"},{"source_id":"S007","target_id":"S023","kind":"supports"},{"source_id":"S021","target_id":"S023","kind":"supports"},{"source_id":"S005","target_id":"S015","kind":"supports"},{"source_id":"S011","target_id":"S026","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S021","target_id":"S033","kind":"supports"},{"source_id":"S020","target_id":"S033","kind":"supports"},{"source_id":"S019","target_id":"S033","kind":"supports"},{"source_id":"S012","target_id":"S034","kind":"supports"},{"source_id":"S013","target_id":"S034","kind":"supports"},{"source_id":"S014","target_id":"S034","kind":"supports"},{"source_id":"S015","target_id":"S034","kind":"supports"},{"source_id":"S016","target_id":"S034","kind":"supports"},{"source_id":"S025","target_id":"S034","kind":"supports"},{"source_id":"S026","target_id":"S034","kind":"supports"},{"source_id":"S027","target_id":"S034","kind":"supports"},{"source_id":"S028","target_id":"S034","kind":"supports"},{"source_id":"S023","target_id":"S034","kind":"supports"},{"source_id":"S028","target_id":"S036","kind":"supports"},{"source_id":"S019","target_id":"S036","kind":"supports"},{"source_id":"S020","target_id":"S036","kind":"supports"},{"source_id":"S026","target_id":"S036","kind":"supports"},{"source_id":"S025","target_id":"S036","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S021","target_id":"S039","kind":"supports"},{"source_id":"S027","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S007","target_id":"S040","kind":"supports"},{"source_id":"S021","target_id":"S040","kind":"supports"},{"source_id":"S025","target_id":"S040","kind":"supports"},{"source_id":"S023","target_id":"S040","kind":"supports"},{"source_id":"S078","target_id":"S041","kind":"supports"},{"source_id":"S058","target_id":"S041","kind":"supports"},{"source_id":"S025","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S043","kind":"supports"},{"source_id":"S040","target_id":"S043","kind":"supports"},{"source_id":"S028","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S026","target_id":"S044","kind":"supports"},{"source_id":"S031","target_id":"S045","kind":"supports"},{"source_id":"S023","target_id":"S045","kind":"supports"},{"source_id":"S037","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S037","target_id":"S046","kind":"supports"},{"source_id":"S040","target_id":"S046","kind":"supports"},{"source_id":"S084","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S026","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S061","target_id":"S049","kind":"supports"},{"source_id":"S027","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S036","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S032","target_id":"S051","kind":"supports"},{"source_id":"S061","target_id":"S051","kind":"supports"},{"source_id":"S019","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S063","target_id":"S053","kind":"supports"},{"source_id":"S028","target_id":"S053","kind":"supports"},{"source_id":"S027","target_id":"S053","kind":"supports"},{"source_id":"S022","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S027","target_id":"S054","kind":"supports"},{"source_id":"S052","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S036","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S062","target_id":"S056","kind":"supports"},{"source_id":"S061","target_id":"S056","kind":"supports"},{"source_id":"S023","target_id":"S056","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S035","kind":"supports"},{"source_id":"S039","target_id":"S035","kind":"supports"},{"source_id":"S040","target_id":"S035","kind":"supports"},{"source_id":"S041","target_id":"S035","kind":"supports"},{"source_id":"S042","target_id":"S035","kind":"supports"},{"source_id":"S043","target_id":"S035","kind":"supports"},{"source_id":"S044","target_id":"S035","kind":"supports"},{"source_id":"S045","target_id":"S035","kind":"supports"},{"source_id":"S046","target_id":"S035","kind":"supports"},{"source_id":"S047","target_id":"S035","kind":"supports"},{"source_id":"S048","target_id":"S035","kind":"supports"},{"source_id":"S049","target_id":"S035","kind":"supports"},{"source_id":"S050","target_id":"S035","kind":"supports"},{"source_id":"S051","target_id":"S035","kind":"supports"},{"source_id":"S052","target_id":"S035","kind":"supports"},{"source_id":"S053","target_id":"S035","kind":"supports"},{"source_id":"S054","target_id":"S035","kind":"supports"},{"source_id":"S055","target_id":"S035","kind":"supports"},{"source_id":"S056","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S012","target_id":"S057","kind":"supports"},{"source_id":"S013","target_id":"S057","kind":"supports"},{"source_id":"S014","target_id":"S057","kind":"supports"},{"source_id":"S015","target_id":"S057","kind":"supports"},{"source_id":"S016","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S059","kind":"supports"},{"source_id":"S057","target_id":"S060","kind":"supports"},{"source_id":"S057","target_id":"S061","kind":"supports"},{"source_id":"S057","target_id":"S062","kind":"supports"},{"source_id":"S057","target_id":"S063","kind":"supports"},{"source_id":"S015","target_id":"S066","kind":"supports"},{"source_id":"S009","target_id":"S066","kind":"supports"},{"source_id":"S021","target_id":"S066","kind":"supports"},{"source_id":"S001","target_id":"S066","kind":"supports"},{"source_id":"S085","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S058","kind":"supports"},{"source_id":"S079","target_id":"S067","kind":"supports"},{"source_id":"S012","target_id":"S067","kind":"supports"},{"source_id":"S013","target_id":"S067","kind":"supports"},{"source_id":"S014","target_id":"S067","kind":"supports"},{"source_id":"S008","target_id":"S067","kind":"supports"},{"source_id":"S067","target_id":"S059","kind":"supports"},{"source_id":"S080","target_id":"S068","kind":"supports"},{"source_id":"S015","target_id":"S068","kind":"supports"},{"source_id":"S016","target_id":"S068","kind":"supports"},{"source_id":"S018","target_id":"S068","kind":"supports"},{"source_id":"S019","target_id":"S068","kind":"supports"},{"source_id":"S068","target_id":"S060","kind":"supports"},{"source_id":"S068","target_id":"S062","kind":"supports"},{"source_id":"S081","target_id":"S069","kind":"supports"},{"source_id":"S020","target_id":"S069","kind":"supports"},{"source_id":"S019","target_id":"S069","kind":"supports"},{"source_id":"S069","target_id":"S061","kind":"supports"},{"source_id":"S014","target_id":"S070","kind":"supports"},{"source_id":"S014","target_id":"S071","kind":"supports"},{"source_id":"S012","target_id":"S072","kind":"supports"},{"source_id":"S014","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S082","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S073","kind":"supports"},{"source_id":"S014","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S083","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S075","kind":"supports"},{"source_id":"S008","target_id":"S075","kind":"supports"},{"source_id":"S085","target_id":"S075","kind":"supports"},{"source_id":"S073","target_id":"S076","kind":"supports"},{"source_id":"S008","target_id":"S076","kind":"supports"},{"source_id":"S085","target_id":"S076","kind":"supports"},{"source_id":"S074","target_id":"S077","kind":"supports"},{"source_id":"S075","target_id":"S077","kind":"supports"},{"source_id":"S076","target_id":"S077","kind":"supports"},{"source_id":"S064","target_id":"S077","kind":"supports"},{"source_id":"S077","target_id":"S063","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"},{"source_id":"S067","target_id":"S065","kind":"supports"},{"source_id":"S068","target_id":"S065","kind":"supports"},{"source_id":"S069","target_id":"S065","kind":"supports"},{"source_id":"S070","target_id":"S065","kind":"supports"},{"source_id":"S071","target_id":"S065","kind":"supports"},{"source_id":"S072","target_id":"S065","kind":"supports"},{"source_id":"S073","target_id":"S065","kind":"supports"},{"source_id":"S074","target_id":"S065","kind":"supports"},{"source_id":"S075","target_id":"S065","kind":"supports"},{"source_id":"S076","target_id":"S065","kind":"supports"},{"source_id":"S077","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S057","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:inverse_problem"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"线性逆问题","source_units":[{"raw_tex":"We are interested in solving the linear inverse problem:\n\\begin{equation}   \\label{eq:inverse_problem}\n\\text{find }\\quad \\xtrue \\in \\Xx \\quad \\text{s.t.}\\quad A \\xtrue = \\ytrue,\n\\end{equation}\nwhere $A: \\mathcal{X}\\to\\mathcal{Y}$ is a bounded linear operator between two Hilbert spaces $\\mathcal{X}$ and $\\Yy$,\nand $ \\ytrue \\in \\mathcal{Y}$ can be seen as  a given  measurement of some unknown $\\xtrue \\in \\Xx$ we want to recover.\nIn general, the inverse problem \\eqref{eq:inverse_problem} is ill-posed as its solution (if it exists) may lack some fundamental properties like uniqueness or stability. \n","start":20669,"end":21271}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":["eq:primal"],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"分层原问题","source_units":[{"raw_tex":"A standard approach   is assuming the desired $\\xtrue$ to be well-approximated by\n \\cite{EngHanNeu96,BenBur18}:\n\\begin{equation}\\label{eq:primal}\n   \\tag*{\\textbf{($\\bm{P_0(\\ytrue)}$)}} \\text{find\\ }\n      \\xsol \\in  \\argmin  \\left\\{ R(x) \\text{ s.t. } x\\in\\underset{x' \\in \\Xx}{\\argmin}~ \\ell(Ax';  \\ytrue) \\right\\}, \n\\end{equation} \ncalled the primal problem in the following.  Here,  $R$ is a  regularization function enforcing some \\emph{a-priori} knowledge on  the desired solution $\\xtrue$, while $\\ell: \\Yy^2\\to\\R\\cup \\{+\\infty\\}$ is a data-fit function. In practice, the  data  is  subject to noise  due to, e.g., possible transmission and/or acquisition problems. As a consequence, we only have access to an inexact version $\\ynoisy$ of $\\ytrue$. \n","start":21271,"end":22028}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"观测噪声背景","source_units":[{"raw_tex":"called the primal problem in the following.  Here,  $R$ is a  regularization function enforcing some \\emph{a-priori} knowledge on  the desired solution $\\xtrue$, while $\\ell: \\Yy^2\\to\\R\\cup \\{+\\infty\\}$ is a data-fit function. In practice, the  data  is  subject to noise  due to, e.g., possible transmission and/or acquisition problems. As a consequence, we only have access to an inexact version $\\ynoisy$ of $\\ytrue$. \n","start":21606,"end":22028}]},{"id":"S004","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"Hilbert 空间与凸函数类","source_units":[{"raw_tex":"We begin fixing the  notation. Let $\\Hh$ be a Hilbert space with scalar product  $\\langle\\cdot,\\cdot\\rangle$ and associated norm $\\|\\cdot\\|$. Given $y\\in\\Hh$, and $\\varrho\\in\\mathbb{R}_+$ and $\\mathbb{B}(y,\\varrho)$ is the open ball of center $y$ and radius $\\varrho$.\nwe denote by $\\Gamma_0(\\Hh)$ the set of proper, convex and lower semi-continuous functions from $\\Hh$ to $]-\\infty,+\\infty]$.\n","start":28119,"end":28514}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"强凸性","source_units":[{"raw_tex":"We say that  $f \\in \\Gamma_0(\\Hh)$ is $\\sigma$-{\\em strongly convex}  if \n$f-{\\sigma}\\Vert \\cdot \\Vert^2/2 \\in \\Gamma_0(\\Hh)$, with  $\\sigma \\in\\left]0,+\\infty\\right[$.\n","start":28514,"end":28683}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["def:subdiff"],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"次微分","source_units":[{"raw_tex":"We recall that the {\\em subdifferential} of  $f \\in \\Gamma_0(\\Hh)$  is the multi-valued operator \n$\\partial f : \\Hh \\rightarrow 2^{\\Hh}$ defined by setting\n\\begin{equation}  \\label{def:subdiff}\n(\\forall x\\in\\Hh)\\quad \\partial f(x) := \\left\\{u \\in \\Hh: f(x') - f(x) - \\langle u , x' - x \\rangle \\geq 0 \\text{ for all } x' \\in \\Hh \\right\\}.\n\\end{equation}\nNote that if $f$ is Gateaux differentiable at $x \\in \\Hh$, then $\\partial f(x)=\\{\\nabla f(x) \\}$, see, e.g.~, \\cite[Proposition 17.31 i)]{BauschkeCombettes2017}.\n","start":28683,"end":29199}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":["def:proximal operator"],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"近端算子","source_units":[{"raw_tex":"We also recall that for all  $x\\in \\Hh$ and $\\tau>0$ the proximity operator $\\prox_{\\tau f} : \\Hh \\rightarrow \\Hh$  of $f\\in\\Gamma_0(\\Hh)$ with parameter $\\tau$ is defined by:\n\\begin{equation}\\label{def:proximal operator}\n    \\prox_{\\tau f}(x) =\n    (I + \\partial f)^{-1}(x) = \n    \\argmin\\limits_{x' \\in \\Hh} \\left\\{ f(x') + \\frac{1}{2\\tau}\\Vert x' - x \\Vert^2 \\right\\}.\n\\end{equation}\n","start":29199,"end":29586}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"Fenchel 共轭","source_units":[{"raw_tex":"Given $f \\in \\Gamma_0(\\Hh)$, we will denote by $f^* : \\Hh \\rightarrow [-\\infty,+\\infty]$   the {\\em Fenchel conjugate} of $f$\n\\[\n(\\forall u\\in\\Hh)\\quad f^*(u):=  \\sup_{x\\in \\Hh}  \\left\\{\\langle u,x\\rangle_{\\Hh} -f(x) \\right\\}.\n\\]\n","start":29586,"end":29816}]},{"id":"S009","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S009"],"declared_external":true,"declaration_text":["原始先修列表：BauschkeCombettes2017, Theorem 18.15；正文第724行重述该标准结论。"],"source_only":false,"title":"强凸共轭的光滑性","source_units":[{"raw_tex":"The Fenchel conjugate of $f$ belongs to $\\Gamma_0(\\Hh)$, and if $f$ is $\\sigma$-strongly convex then $f^*$ is differentiable at any point, with a $\\sigma^{-1}$-Lipschitz continuous gradient, see, e.g. \\cite[Theorem 18.15]{BauschkeCombettes2017}. \n","start":29816,"end":30063}]},{"id":"S010","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S010"],"declared_external":true,"declaration_text":["原始先修列表中的 standard convex conjugacy；正文明确引用 BauschkeCombettes2017, Theorem 16.23。"],"source_only":false,"title":"共轭次梯度互逆","source_units":[{"raw_tex":"Furthermore, the following property holds, see \\cite[Theorem 16.23]{BauschkeCombettes2017}: \n\\[\n(\\forall (x,u)\\in \\Hh^2)\\quad u \\in \\partial f(x) \\Leftrightarrow x \\in \\partial f^*(u).\n\\] \n","start":30063,"end":30252}]},{"id":"S011","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"渐近阶与统一范数","source_units":[{"raw_tex":"Finally, given two real sequences $(a_k)_{k\\geq 1}$ and $(b_k)_{k\\geq 1}$, we will write $a_k = O(b_k)$ whenever there exists a positive constant $M>0$ such that $a_k \\leq M b_k$ for all $k \\geq 1$. We will use the more precise notation $a_k = \\Theta(b_k)$ if both conditions $a_k = O(b_k)$ and $b_k = O(a_k)$ hold.\nNote also that we will use the same notation  $\\|\\cdot\\|$ and $\\langle\\cdot,\\cdot\\rangle$ for the norm and the scalar product in every Hilbert space we consider. \n","start":30252,"end":30731}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"拟合项的凸性与强制性","source_units":[{"raw_tex":"We make   the following assumptions on the data-fit $\\ell$ and the regularizer $R$: \n\\begin{enumerate}[label=\\textbf{$(L_\\arabic*)$}]\n\t\\item \\label{assumption:data-fit convex and coercive}\n\tFor all $y \\in \\Yy$, $\\ell_y:=\\ell(\\cdot,y)\\in \\Gamma_0(\\Yy) $ and it is coercive.\n","start":30762,"end":31035}]},{"id":"S013","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"拟合项的唯一零点","source_units":[{"raw_tex":"\t\\item \\label{assumption:data-fit not flat} \n\tFor all $(y_1,y_2)\\in \\Yy^2, ~\\ell(y_1,y_2)=0 \\iff y_1=y_2$.\n","start":31035,"end":31142}]},{"id":"S014","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"局部条件下界","source_units":[{"raw_tex":"\t\\item \\label{assumption:data-fit conditionned locally}\n\t there exists\n     $q\\in[1,+\\infty[$, $\\varrho \\in ]0,+\\infty]$, $ \\gamma \\in ]0,+\\infty[$ such that\n\t$$\n\t\\forall y \\in \\mathbb{B}( \\ytrue,\\varrho),\\quad \\frac{\\gamma}{q}\\Vert y-\\ytrue\\Vert^q \\leq \\ell(y,\\ytrue). \n\t$$ \n\\end{enumerate}\n","start":31142,"end":31434}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"正则项强凸","source_units":[{"raw_tex":"\\begin{enumerate}[label=\\textbf{$(R_\\arabic*)$}]\t\n\t\\item \\label{H:R strongly convex} $R$ is $\\sigma$-strongly convex, with $\\sigma \\in \\left]0,+\\infty\\right[$,  \n","start":31434,"end":31596}]},{"id":"S016","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"源条件","source_units":[{"raw_tex":"\t\\item \\label{assumption:AR2}   $\\partial R(\\xsol) \\cap \\im A^* \\neq \\emptyset$.\t\n\\end{enumerate}\n","start":31596,"end":31694}]},{"id":"S017","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"局部与全局条件性","source_units":[{"raw_tex":"If assumption \\ref{assumption:data-fit conditionned locally} is satisfied, we say that $\\ell_{\\ytrue}$ is \\emph{locally\n\t$q$-conditioned} at $\\bar{y}$. If \\ref{assumption:data-fit conditionned locally} is satisfied with $\\rho=+\\infty$, then we say that $\\ell_{\\ytrue}$  is $q$-conditioned at $\\bar{y}$.\n","start":31694,"end":31997}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":["eq:primal_char"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"精确约束原问题","source_units":[{"raw_tex":"A class of methods to solve \\eqref{eq:inverse_problem}  consider the   problem \n\\begin{equation}   \\label{eq:primal_char}\n    \\text{find }  \\xsol \\in  \\argmin_{x \\in \\Xx}  \\left\\{ R(x) + \\delta_{\\bar{y}}(Ax) \\right\\},\n\\end{equation}\nwhere the constraint \\eqref{eq:inverse_problem} is encoded by the indicator function $\\delta_{\\bar{y}}$. Using Fenchel-Rockafeller duality,  \n","start":36263,"end":36638}]},{"id":"S019","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:Dual"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"极限对偶目标","source_units":[{"raw_tex":"where the constraint \\eqref{eq:inverse_problem} is encoded by the indicator function $\\delta_{\\bar{y}}$. Using Fenchel-Rockafeller duality,  \nwe derive the corresponding dual problem:\n\\begin{equation}   \\label{eq:Dual}\n\\text{find } u^\\dagger \\quad\\text{ s.t. }\\quad u^\\dagger \\in \\argmin_{u\\in \\mathcal{Y}}\\  \\left\\{ d_0(u) := R^*(-A^*u) + \\langle \\bar{y}, u \\rangle \\right\\}\n    \\tag*{\\textbf{($\\bm{D_0}$)}}.\n\\end{equation}\n","start":36496,"end":36921}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":["eq:DualReg"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"正则化对偶目标","source_units":[{"raw_tex":"The dual problem of \\ref{eq:primal penalized} is\n\\begin{equation}   \\label{eq:DualReg}\n\\text{find } u_\\lambda \\in \\argmin_{u\\in \\mathcal{Y}}\\ \\left\\{d_\\lambda(u) := R^*(-A^*u) + \\frac{1}{\\lambda}\\ell^*(\\lambda u;\\ynoisy)\\right\\}.   \\tag*{\\textbf{($\\bm{D_\\lambda}$)}}\n\\end{equation} \n","start":39699,"end":39982}]},{"id":"S021","semantic_type":"notation","kind":"notation","labels":["eq:composite"],"member_ids":["S021"],"declared_external":false,"source_only":true,"title":"复合光滑项缩写","source_units":[{"raw_tex":"From now on we will use the following shorthand notation: \n\\begin{equation} \\label{eq:composite}\n   \\RA: = R^* \\circ (-A^*),\\qquad \\ell^*_{\\bar{y}}(\\cdot): = \\ell(\\cdot,\\bar{y})^*\n\\end{equation}\nso that the composite dual function $d_\\lambda$ can be written as $d_{\\lambda}(u) =  \\RA(u) + \\lambda^{-1} \\ell^*_{\\bar{y}}(\\lambda u)$, for every $u\\in\\mathcal{Y}$.\n","start":48647,"end":49008}]},{"id":"S022","semantic_type":"definition","kind":"definition","labels":["proof:ccd3"],"member_ids":["S022"],"declared_external":false,"source_only":true,"title":"残差常数","source_units":[{"raw_tex":"where the constant $c$ is defined as:\n\\begin{equation}\\label{proof:ccd3}\n    c :=\n    \\begin{cases}\n    0 \\quad &\\quad \\text{ if }q=1,\\\\\n    (1-(1/q))\\gamma^{-1/(q-1)}\\Vert u^\\dagger \\Vert^{q/(q-1)} \\quad &\\quad \\text{ if } q >1,\n    \\end{cases}\n\\end{equation}\nand it is finite in both cases.\n","start":52273,"end":52566}]},{"id":"S023","semantic_type":"algorithm","kind":"algorithm","labels":["alg:fw_accel"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"惯性对偶对角下降","source_units":[{"raw_tex":"\\begin{equation}\\label{alg:fw_accel}\nu_0 = u_1 \\in \\Yy, \\quad \\text{ compute for } k \\geq 1 \\quad \n\\begin{cases}\n w_k & = u_k + \\alpha_k(u_k-u_{k-1}), \\\\\n u_{k+1} &= \\mathrm{prox}_{\\frac{\\tau}{\\lambda_{k}} \\ell^*_{\\bar{y}}(\\lambda_k\\cdot)}\\left(w_k - \\tau\\nabla \\RA(w_k)\\right),\\\\\n    x_{k+1} & = \\nabla R^*(-A^* u_{k+1}). \n\\end{cases}\n\\tag*{\\textbf{($\\bm{I3D}$)}}\n\\end{equation}\nThis algorithm depends on three parameters: the stepsize $\\tau >0$, the relaxation parameters $(\\lambda_k)_k$ and the friction parameters $(\\alpha_k)_k$.\n","start":56423,"end":56957}]},{"id":"S025","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"步长限制","source_units":[{"raw_tex":"\t\\item \\label{assumption:parameter:stepsize}\n\t$\\tau \\in \\left(0,\\frac{\\sigma^2}{\\Vert A \\Vert^2}\\right]$, where $\\sigma > 0$ is defined in assumption \\ref{H:R strongly convex}.\n","start":57476,"end":57653}]},{"id":"S026","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"摩擦与时间权重","source_units":[{"raw_tex":"\t\\item \\label{assumption:parameter:friction}\n\t$\\alpha_k$ is non-negative and for every $k\\geq 1$, $t_k:=1+\\sum_{i=k}^{+\\infty}\\prod_{j=k}^i \\alpha_j$ is finite,  with $t_k = \\Theta(k)$.\n","start":57653,"end":57839}]},{"id":"S027","semantic_type":"assumption","kind":"assumption","labels":["def:biglambda"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"对角参数与加权可和性","source_units":[{"raw_tex":"\t\\item \\label{assumption:parameter:relaxation}\n\t$(\\lambda_k)$ is a strictly positive non-increasing sequence such that $\\lim_{k\\to\\infty} \\lambda_k = 0$. \n\tMoreover, by defining\n\t\\begin{equation}   \\label{def:biglambda}\n\t\\Lambda:=\n\t\\begin{cases}\n\t\\sum_{k\\geq 1} t_{k+1} \\lambda_k^{1/(q-1)}\n\t& \\text{ if } q>1, \\\\\n\t0\n\t& \\text{ if } q=1, \n\t\\end{cases}\n\t\\end{equation}\n\twe have that $\\Lambda< + \\infty$.\n","start":57839,"end":58240}]},{"id":"S028","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"初始对偶尺度限制","source_units":[{"raw_tex":"\t\\item \\label{assumption:parameter:technical}\n\tFor some $u^\\dagger \\in \\argmin d_0$, we have  $\\lambda_0 \\Vert u^\\dagger \\Vert \\leq \\gamma \\varrho^{q-1} /q$.\n\\end{enumerate}\n","start":58240,"end":58414}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","labels":["eq:choice:alpha","eq:choice:tk","eq:properties_sequences_alphak_tk","lemma:sequence"],"member_ids":["S030"],"declared_external":true,"declaration_text":["原始先修列表：AttouchTikAccel, Lemma 2.1，作为 B0069 重述，文内无证明。"],"source_only":false,"title":"引用的序列引理","source_units":[{"raw_tex":"\\begin{lemma}[ {{\\cite[Lemma 2.1]{AttouchTikAccel} }}]   \\label{lemma:sequence}\nAssume that $(\\alpha_k)$ is non-negative and satisfies\n\\begin{equation}  \\label{eq:choice:alpha}\n\\sum_{i=k}^{+\\infty}\\prod_{j=k}^i \\alpha_j<+\\infty,\\quad\\text{for every }k\\geq 1.\n\\end{equation}\nThen, the sequence defined by\n\\begin{equation} \\label{eq:choice:tk}\nt_k:=1+\\sum_{i=k}^{+\\infty}\\prod_{j=k}^i \\alpha_j\n\\end{equation}\nis well-defined, and satisfies for every $k\\geq 1$ the following properties:\n\\begin{equation}   \\label{eq:properties_sequences_alphak_tk}\n1+\\alpha_k t_{k+1} = t_k, \\qquad t^2_{k+1}-t^2_k\\leq t_{k+1}.\n\\end{equation}\n\\end{lemma}\n","start":60117,"end":60751}]},{"id":"S031","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:properties_sequences_alphak_tk","lemma:sequence"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"时间权重一次恒等式","source_units":[{"raw_tex":"\\begin{equation}   \\label{eq:properties_sequences_alphak_tk}\n1+\\alpha_k t_{k+1} = t_k, \\qquad t^2_{k+1}-t^2_k\\leq t_{k+1}.\n\\end{equation}\n","start":60601,"end":60739}]},{"id":"S032","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:properties_sequences_alphak_tk","lemma:sequence"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"时间权重二次上界","source_units":[{"raw_tex":"\\begin{equation}   \\label{eq:properties_sequences_alphak_tk}\n1+\\alpha_k t_{k+1} = t_k, \\qquad t^2_{k+1}-t^2_k\\leq t_{k+1}.\n\\end{equation}\n","start":60601,"end":60739}]},{"id":"S033","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"附录对象回指","source_units":[{"raw_tex":"We first consider $\\Rr_A $, $\\ell^*_y$ defined in \\eqref{eq:composite} and on the diagonal dual function $d_{\\lambda}$ and its limit $d_0$  defined in \\ref{eq:DualReg} and \\ref{eq:Dual}, respectively. For similar results see also \\cite{Garrigos2017}.\n","start":114878,"end":115129}]},{"id":"S034","semantic_type":"theorem","kind":"theorem","labels":["T:discrete convergence rates noiseless","eq:rate"],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"精确数据快速收敛","source_units":[{"raw_tex":"\\begin{theorem}[Fast convergence]\\label{T:discrete convergence rates noiseless}\nLet the assumptions \\ref{assumption:data-fit convex and coercive}-\\ref{assumption:data-fit conditionned locally}, \\ref{H:R strongly convex}-\\ref{assumption:AR2}, \\ref{assumption:parameter:stepsize}-\\ref{assumption:parameter:technical} hold true.\nLet $(x_k)$ and $(u_k)$ be the sequences generated by algorithm \\ref{alg:fw_accel}.\nThen, there exists $C\\in\\left]0,+\\infty\\right[$ such that\n\\begin{equation}  \\label{eq:rate}%\\label{eq:rate:primal:it}\n d_{{\\lambda_k}}(u_k) - \\inf  d_0  \\leq \\frac{C}{t_k^2} \n \\quad \\text{ and } \\quad \n \\| x_k - x^\\dagger \\| \\leq \\frac{\\sqrt{2C}}{\\sqrt{\\sigma}t_k}.\n\\end{equation}\n\\end{theorem}","start":62973,"end":63677}]},{"id":"S035","semantic_type":"proof","kind":"proof","labels":["descent:fifth","descent:first","descent:second","descent:third","descent:third2","eq:condition","eq:decay_complete","eq:def_zk:noisefree","eq:descent","eq:discrete energy","eq:fw_operator","eq:property_zk","eq:shortand_notation_Dk","ineq:1"],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"快速收敛的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $u^\\dagger\\in \\argmin d_0$ be the minimizer of $d_0$ defined in assumption \\ref{assumption:parameter:technical}, and\ndefine, for every $k\\geq 1$,  the discrete Lyapunov energy function:\n\\begin{equation}  \\label{eq:discrete energy}\n\\mathcal{E}(k):=t_k^2 \\Big( d_{{\\lambda_k}}(u_k)-\\inf d_0\\Big) + \\frac{1}{2\\tau}\\|z_k-u^\\dagger\\|^2,\n\\end{equation}\nwhere $z_k$ is defined as:\n\\begin{equation}  \\label{eq:def_zk:noisefree}\nz_k:=u_{k-1} +t_k(u_k-u_{k-1}).\n\\end{equation}\nOur goal is to get an estimate on the decay of $\\mathcal{E}$ along time.\nMore precisely, we will show that for every $k\\geq 1$\n\\begin{equation}  \\label{eq:condition}\n\\mathcal{E}(k+1) - \\mathcal{E}(k)\n\\leq  t_{k+1} \\Big(d_{{\\lambda_k}}(u^\\dagger) -  \\inf d_0\\Big),\n\\end{equation}\nwhich can be seen as a discrete analogue of \\eqref{proof:ccd2}, and from which the desired accelerated convergence rates will follow in a straightforward manner.\n\nFor simplicity, let us denote by $\\ell_k$  the function defined by setting\n\\begin{equation}  \\label{eq:shortand_notation_Dk}\n(\\forall u\\in \\mathcal{Y})\\quad \\ell_k(u)=\\lambda_k^{-1}{\\ell}_{\\bar{y}}^*(\\lambda_k u).\n\\end{equation} \nTo prove \\eqref{eq:condition}, we define for every $k\\geq 1$ the operator $G_k:\\mathcal{Y}\\to \\mathcal{Y}$ as\n\\begin{equation}  \\label{eq:fw_operator}\nG_k(z)  := \\frac{1}{\\tau}\\Big(z-\\text{prox}_{\\tau \\ell_k}(z-\\tau \\nabla  \\RA(z))\\Big)\n\\end{equation}\nand notice that the proximal step of  \\ref{alg:fw_accel} can be written in terms of $G_k$ as $u_{k+1}=w_k - \\tau G_k(w_k)$. The descent lemma yields (see, e.g., \\cite{AttouchTikAccel,ChambolleDossal2015})\n\\begin{equation}  \\label{eq:descent}\n    d_{\\lambda_k}(w-\\tau G_k(w)) \\leq  d_{\\lambda_k}(u) + \\langle G_k(w), w-u\\rangle - \\frac{\\tau}{2}\\|G_k(w)\\|^2, \\quad \\text{for all }w,u\\in\\mathcal{Y},\n\\end{equation}\nEvaluating \\eqref{eq:descent} for $u=u_{k}$ and $w=w_k$, we get\n\\begin{equation}  \\label{descent:first}\nd_{{\\lambda_k}}(u_{k+1}) \\leq  d_{{\\lambda_k}}(u_{k}) + \\langle G_k(w_k), w_k - u_{k}\\rangle - \\frac{\\tau}{2}\\|G_k(w_k)\\|^2.\n\\end{equation}\nEvaluating \\eqref{eq:descent} for $u=u^\\dagger$ and $w=w_k$ we derive\n \\begin{equation}  \\label{descent:second}\nd_{{\\lambda_k}}(u_{k+1}) \\leq  d_{{\\lambda_k}}(u^\\dagger) + \\langle G_k(w_k), w_k -u^\\dagger\\rangle - \\frac{\\tau}{2}\\|G_k(w_k)\\|^2.\n\\end{equation}\nWe now multiply \\eqref{descent:first} by $t_{k+1}-1$ and we add it to \\eqref{descent:second}, and we obtain\n\\begin{align}\nt_{k+1} d_{{\\lambda_k}}(u_{k+1}) \n&\\leq  (t_{k+1}-1)d_{{\\lambda_k}}(u_{k}) +  d_{{\\lambda_k}}(u^\\dagger)  \\nonumber\\\\\n\\label{ineq:1}&+ \\langle  G_k(w_k), (t_{k+1}-1)(w_k-u_{k}) + (w_{k} - u^\\dagger)\\rangle  -\\frac{\\tau}{2}t_{k+1}\\|  G_k(w_k)\\|^2 . \n\\end{align}\nAs an immediate consequence of Lemma \\ref{lemma:sequence}, we observe the following property:\n\\begin{align}  \\label{eq:property_zk}\n(t_{k+1}-1)(w_k-u_{k}) + w_k & = u_{k} + t_{k+1}(w_k -u_{k}) \\notag\\\\\n& = u_{k} + t_{k+1} \\alpha_k(u_{k} -u_{k-1}) \\\\\n& = u_{k-1} + (1+ t_{k+1} \\alpha_k)(u_{k} -u_{k-1}) \\notag\\\\\n& = u_{k-1} +t_k(u_{k} -u_{k-1}) = z_k. \\notag\n\\end{align}\nThanks to \\eqref{eq:def_zk:noisefree}, the fact that $z_k-\\tau t_{k+1}G_k(w_k)=z_{k+1}$ and the previous equality, we can reorder the terms in \\eqref{ineq:1} and rewrite it as\n\\begin{align}\nt_{k+1} ( d_{{\\lambda_k}}(u_{k+1}) -  d_{{\\lambda_k}}(u^\\dagger)) \n& \\leq  (t_{k+1}-1) (d_{{\\lambda_k}}(u_{k}) -  d_{{\\lambda_k}}(u^\\dagger)) \\notag\\\\\n& + \\frac{1}{2\\tau t_{k+1}}\\left(\\|z_{k}-u^\\dagger\\|^2-\\|z_{k+1}-u^\\dagger\\|^2\\right). \\notag \n\\end{align}\nWe now multiply everything by $t_{k+1}$, re-arrange and get\n\\begin{align}  \\label{descent:third}\n& t^2_{k+1} ( d_{{\\lambda_k}}(u_{k+1}) -  d_{{\\lambda_k}}(u^\\dagger))   + \\frac{1}{2\\tau}\n\\|z_{k+1}-u^\\dagger\\|^2   \\\\\n&\\leq  (t^2_{k+1}-t_{k+1}) (d_{{\\lambda_k}}(u_{k}) -  d_{\\lambda_k}(u^\\dagger)) + \\frac{1}{2\\tau } \\|z_{k}-u^\\dagger\\|^2.  \\notag\n\\end{align}\nEquivalently:\n\\begin{align}  \\label{descent:third2}\n& t^2_{k+1} \\Big( d_{{\\lambda_k}}(u_{k+1}) -  d_{{\\lambda_k}}(u^\\dagger) \\Big)   + \\frac{1}{2\\tau}\n\\|z_{k+1}-u^\\dagger\\|^2 \\nonumber  \\\\\n&\\leq  t^2_k \\Big(d_{{\\lambda_k}}(u_{k}) -  d_{{\\lambda_k}}(u^\\dagger)\\Big) +   (t^2_{k+1}-t_{k+1} - t_{k}^2) \\Big( d_{{\\lambda_k}}(u_{k}) -  d_{{\\lambda_k}}(u^\\dagger) \\Big)+ \\frac{1}{2\\tau } \\|z_{k}-u^\\dagger\\|^2. \\notag\n\\end{align}\nTo get the desired terms, we first use on the left-hand side  the monotonicity property of the function $ d_{{\\lambda_k}}(\\cdot)$ as a function of $k$ (see  Lemma  \\ref{L:functional properties}\\ref{L:functional properties:decreasing function sequence}) and then add and subtract in the parentheses  the term $\\inf d_0$, thus getting:\n\\begin{align}  \\label{descent:fifth}\n&t^2_{k+1} \\Big( d_{{\\lambda_{k+1}}}(u_{k+1}) -   \\inf d_0 \\Big)   + \\frac{1}{2\\tau}\n\\|z_{k+1}-u^\\dagger\\|^2   \\notag \\\\\n&\\leq  t^2_k \\Big(d_{{\\lambda_{k}}}(u_{k}) - \\inf d_0 \\Big) +   (t^2_{k+1}-t_{k+1} - t_{k}^2) \\Big( d_{{\\lambda_{k}}}(u_{k}) -  \\inf d_0\\Big) \\\\\n& +t_{k+1} \\Big(d_{{\\lambda_{k}}}(u^\\dagger) -  \\inf d_0\\Big)   +  \\frac{1}{2\\tau } \\|z_{k}-u^\\dagger\\|^2. \\notag\n\\end{align}\nAfter rearranging and using the definition of $\\mathcal{E}$ in \\eqref{eq:discrete energy}, we deduce:\n\\begin{align}  \\label{descent:fifth}\n\\mathcal{E}(k+1) +    (t^2_{k}+t_{k+1} - t_{k+1}^2) \\Big( d_{{\\lambda_{k}}}(u_{k}) -  \\inf d_0\\Big) \\leq  \\mathcal{E}(k) + t_{k+1} \\Big(d_{{\\lambda_{k}}}(u^\\dagger) -  \\inf d_0\\Big). \\nonumber\n\\end{align}\n\n\\noindent Thanks to \\eqref{eq:properties_sequences_alphak_tk} and  Lemma~\\ref{L:functional properties}\\ref{L:functional properties:decreasing function sequence}, we can neglect the second term  in the left-hand side of the above inequality,  finally getting the desired  \\eqref{eq:condition}.\nIterating \\eqref{eq:condition}  recursively gives\n\\begin{align}  \\label{eq:decay_complete}\n\\mathcal{E}(k) \\leq \\mathcal{E}(1) +\\sum_{j=1}^{k-1}t_{j+1}\\Big( d_{\\lambda_{j}}(u^\\dagger)-\\inf d_0 \\Big).\n\\end{align}\nTo  bound the sum appearing on the right hand side, we need to analyze the residuals $r_j:= d_{\\lambda_{j}}(u^\\dagger) -\\inf d_0$. Similarly as for the estimation obtained in the continuous case, we can use for this purpose the property in Lemma \\ref{L:functional properties}\\ref{L:functional properties:diagonal rates via conditioning} and get that for some fixed constant $c$ independent on $j$ (defined analogously as in \\eqref{proof:ccd3}), we have\n$$\nr_j\\leq c \\lambda_j^{\\frac{1}{q-1}},\\qquad\\text{for every }j\\geq 1.\n$$\nBy assumption \\ref{assumption:parameter:relaxation}, with $\\Lambda$ as  in \\eqref{def:biglambda}, we thus conclude that\n$$\n\\sum_{j=1}^{k-1} t_{j+1} r_j \\leq  c \\sum_{j=1}^{k-1} t_{j+1} \\lambda_j^{\\frac{1}{q-1}}  \\leq c \\Lambda < +\\infty\n$$\nThis allows us to deduce from \\eqref{eq:decay_complete} the convergence rate on the dual values in \\eqref{eq:rate}, by taking $C = \\Ee(1) + c \\Lambda$.\nFinally, the convergence rate on the primal iterates  in \\eqref{eq:rate} follows from  Lemma \\ref{L:functional properties}\\ref{L:functional properties:from dual value to primal iterates}.\n\\end{proof}","start":63678,"end":70715}]},{"id":"S036","semantic_type":"definition","kind":"definition","labels":["eq:discrete energy"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"离散 Lyapunov 能量","source_units":[{"raw_tex":"Let $u^\\dagger\\in \\argmin d_0$ be the minimizer of $d_0$ defined in assumption \\ref{assumption:parameter:technical}, and\ndefine, for every $k\\geq 1$,  the discrete Lyapunov energy function:\n\\begin{equation}  \\label{eq:discrete energy}\n\\mathcal{E}(k):=t_k^2 \\Big( d_{{\\lambda_k}}(u_k)-\\inf d_0\\Big) + \\frac{1}{2\\tau}\\|z_k-u^\\dagger\\|^2,\n\\end{equation}\n","start":63692,"end":64043}]},{"id":"S037","semantic_type":"definition","kind":"definition","labels":["eq:def_zk:noisefree"],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"惯性辅助变量","source_units":[{"raw_tex":"where $z_k$ is defined as:\n\\begin{equation}  \\label{eq:def_zk:noisefree}\nz_k:=u_{k-1} +t_k(u_k-u_{k-1}).\n\\end{equation}\n","start":64043,"end":64163}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":["eq:condition"],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"提出能量递推目标","source_units":[{"raw_tex":"Our goal is to get an estimate on the decay of $\\mathcal{E}$ along time.\nMore precisely, we will show that for every $k\\geq 1$\n\\begin{equation}  \\label{eq:condition}\n\\mathcal{E}(k+1) - \\mathcal{E}(k)\n\\leq  t_{k+1} \\Big(d_{{\\lambda_k}}(u^\\dagger) -  \\inf d_0\\Big),\n\\end{equation}\nwhich can be seen as a discrete analogue of \\eqref{proof:ccd2}, and from which the desired accelerated convergence rates will follow in a straightforward manner.\n","start":64163,"end":64604}]},{"id":"S039","semantic_type":"notation","kind":"notation","labels":["eq:shortand_notation_Dk"],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"缩放共轭项","source_units":[{"raw_tex":"For simplicity, let us denote by $\\ell_k$  the function defined by setting\n\\begin{equation}  \\label{eq:shortand_notation_Dk}\n(\\forall u\\in \\mathcal{Y})\\quad \\ell_k(u)=\\lambda_k^{-1}{\\ell}_{\\bar{y}}^*(\\lambda_k u).\n\\end{equation} \n","start":64605,"end":64835}]},{"id":"S040","semantic_type":"definition","kind":"definition","labels":["eq:fw_operator"],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"前向后向残差映射","source_units":[{"raw_tex":"To prove \\eqref{eq:condition}, we define for every $k\\geq 1$ the operator $G_k:\\mathcal{Y}\\to \\mathcal{Y}$ as\n\\begin{equation}  \\label{eq:fw_operator}\nG_k(z)  := \\frac{1}{\\tau}\\Big(z-\\text{prox}_{\\tau \\ell_k}(z-\\tau \\nabla  \\RA(z))\\Big)\n\\end{equation}\nand notice that the proximal step of  \\ref{alg:fw_accel} can be written in terms of $G_k$ as $u_{k+1}=w_k - \\tau G_k(w_k)$. The descent lemma yields (see, e.g., \\cite{AttouchTikAccel,ChambolleDossal2015})\n","start":64835,"end":65292}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":["eq:descent"],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"调用复合下降不等式","source_units":[{"raw_tex":"and notice that the proximal step of  \\ref{alg:fw_accel} can be written in terms of $G_k$ as $u_{k+1}=w_k - \\tau G_k(w_k)$. The descent lemma yields (see, e.g., \\cite{AttouchTikAccel,ChambolleDossal2015})\n\\begin{equation}  \\label{eq:descent}\n    d_{\\lambda_k}(w-\\tau G_k(w)) \\leq  d_{\\lambda_k}(u) + \\langle G_k(w), w-u\\rangle - \\frac{\\tau}{2}\\|G_k(w)\\|^2, \\quad \\text{for all }w,u\\in\\mathcal{Y},\n\\end{equation}\n","start":65087,"end":65499}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":["descent:first"],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"与当前迭代点比较","source_units":[{"raw_tex":"Evaluating \\eqref{eq:descent} for $u=u_{k}$ and $w=w_k$, we get\n\\begin{equation}  \\label{descent:first}\nd_{{\\lambda_k}}(u_{k+1}) \\leq  d_{{\\lambda_k}}(u_{k}) + \\langle G_k(w_k), w_k - u_{k}\\rangle - \\frac{\\tau}{2}\\|G_k(w_k)\\|^2.\n\\end{equation}\n","start":65499,"end":65743}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":["descent:second"],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"与对偶极小点比较","source_units":[{"raw_tex":"Evaluating \\eqref{eq:descent} for $u=u^\\dagger$ and $w=w_k$ we derive\n \\begin{equation}  \\label{descent:second}\nd_{{\\lambda_k}}(u_{k+1}) \\leq  d_{{\\lambda_k}}(u^\\dagger) + \\langle G_k(w_k), w_k -u^\\dagger\\rangle - \\frac{\\tau}{2}\\|G_k(w_k)\\|^2.\n\\end{equation}\n","start":65743,"end":66002}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":["ineq:1"],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"加权合并下降估计","source_units":[{"raw_tex":"We now multiply \\eqref{descent:first} by $t_{k+1}-1$ and we add it to \\eqref{descent:second}, and we obtain\n\\begin{align}\nt_{k+1} d_{{\\lambda_k}}(u_{k+1}) \n&\\leq  (t_{k+1}-1)d_{{\\lambda_k}}(u_{k}) +  d_{{\\lambda_k}}(u^\\dagger)  \\nonumber\\\\\n\\label{ineq:1}&+ \\langle  G_k(w_k), (t_{k+1}-1)(w_k-u_{k}) + (w_{k} - u^\\dagger)\\rangle  -\\frac{\\tau}{2}t_{k+1}\\|  G_k(w_k)\\|^2 . \n\\end{align}\n","start":66002,"end":66385}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["eq:property_zk"],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"识别惯性线性组合","source_units":[{"raw_tex":"As an immediate consequence of Lemma \\ref{lemma:sequence}, we observe the following property:\n\\begin{align}  \\label{eq:property_zk}\n(t_{k+1}-1)(w_k-u_{k}) + w_k & = u_{k} + t_{k+1}(w_k -u_{k}) \\notag\\\\\n& = u_{k} + t_{k+1} \\alpha_k(u_{k} -u_{k-1}) \\\\\n& = u_{k-1} + (1+ t_{k+1} \\alpha_k)(u_{k} -u_{k-1}) \\notag\\\\\n& = u_{k-1} +t_k(u_{k} -u_{k-1}) = z_k. \\notag\n\\end{align}\n","start":66385,"end":66755}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"改写为平方范数差","source_units":[{"raw_tex":"Thanks to \\eqref{eq:def_zk:noisefree}, the fact that $z_k-\\tau t_{k+1}G_k(w_k)=z_{k+1}$ and the previous equality, we can reorder the terms in \\eqref{ineq:1} and rewrite it as\n\\begin{align}\nt_{k+1} ( d_{{\\lambda_k}}(u_{k+1}) -  d_{{\\lambda_k}}(u^\\dagger)) \n& \\leq  (t_{k+1}-1) (d_{{\\lambda_k}}(u_{k}) -  d_{{\\lambda_k}}(u^\\dagger)) \\notag\\\\\n& + \\frac{1}{2\\tau t_{k+1}}\\left(\\|z_{k}-u^\\dagger\\|^2-\\|z_{k+1}-u^\\dagger\\|^2\\right). \\notag \n\\end{align}\n","start":66755,"end":67203}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":["descent:third"],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"建立平方权重估计","source_units":[{"raw_tex":"We now multiply everything by $t_{k+1}$, re-arrange and get\n\\begin{align}  \\label{descent:third}\n& t^2_{k+1} ( d_{{\\lambda_k}}(u_{k+1}) -  d_{{\\lambda_k}}(u^\\dagger))   + \\frac{1}{2\\tau}\n\\|z_{k+1}-u^\\dagger\\|^2   \\\\\n&\\leq  (t^2_{k+1}-t_{k+1}) (d_{{\\lambda_k}}(u_{k}) -  d_{\\lambda_k}(u^\\dagger)) + \\frac{1}{2\\tau } \\|z_{k}-u^\\dagger\\|^2.  \\notag\n\\end{align}\n","start":67203,"end":67561}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":["descent:third2"],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"分离权重余项","source_units":[{"raw_tex":"Equivalently:\n\\begin{align}  \\label{descent:third2}\n& t^2_{k+1} \\Big( d_{{\\lambda_k}}(u_{k+1}) -  d_{{\\lambda_k}}(u^\\dagger) \\Big)   + \\frac{1}{2\\tau}\n\\|z_{k+1}-u^\\dagger\\|^2 \\nonumber  \\\\\n&\\leq  t^2_k \\Big(d_{{\\lambda_k}}(u_{k}) -  d_{{\\lambda_k}}(u^\\dagger)\\Big) +   (t^2_{k+1}-t_{k+1} - t_{k}^2) \\Big( d_{{\\lambda_k}}(u_{k}) -  d_{{\\lambda_k}}(u^\\dagger) \\Big)+ \\frac{1}{2\\tau } \\|z_{k}-u^\\dagger\\|^2. \\notag\n\\end{align}\n","start":67561,"end":67985}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["descent:fifth"],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"更换对角目标并移基准","source_units":[{"raw_tex":"To get the desired terms, we first use on the left-hand side  the monotonicity property of the function $ d_{{\\lambda_k}}(\\cdot)$ as a function of $k$ (see  Lemma  \\ref{L:functional properties}\\ref{L:functional properties:decreasing function sequence}) and then add and subtract in the parentheses  the term $\\inf d_0$, thus getting:\n\\begin{align}  \\label{descent:fifth}\n&t^2_{k+1} \\Big( d_{{\\lambda_{k+1}}}(u_{k+1}) -   \\inf d_0 \\Big)   + \\frac{1}{2\\tau}\n\\|z_{k+1}-u^\\dagger\\|^2   \\notag \\\\\n&\\leq  t^2_k \\Big(d_{{\\lambda_{k}}}(u_{k}) - \\inf d_0 \\Big) +   (t^2_{k+1}-t_{k+1} - t_{k}^2) \\Big( d_{{\\lambda_{k}}}(u_{k}) -  \\inf d_0\\Big) \\\\\n& +t_{k+1} \\Big(d_{{\\lambda_{k}}}(u^\\dagger) -  \\inf d_0\\Big)   +  \\frac{1}{2\\tau } \\|z_{k}-u^\\dagger\\|^2. \\notag\n\\end{align}\n","start":67985,"end":68748}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":["descent:fifth"],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"写成能量加余项","source_units":[{"raw_tex":"After rearranging and using the definition of $\\mathcal{E}$ in \\eqref{eq:discrete energy}, we deduce:\n\\begin{align}  \\label{descent:fifth}\n\\mathcal{E}(k+1) +    (t^2_{k}+t_{k+1} - t_{k+1}^2) \\Big( d_{{\\lambda_{k}}}(u_{k}) -  \\inf d_0\\Big) \\leq  \\mathcal{E}(k) + t_{k+1} \\Big(d_{{\\lambda_{k}}}(u^\\dagger) -  \\inf d_0\\Big). \\nonumber\n\\end{align}\n","start":68748,"end":69092}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"丢弃非负余项","source_units":[{"raw_tex":"\\noindent Thanks to \\eqref{eq:properties_sequences_alphak_tk} and  Lemma~\\ref{L:functional properties}\\ref{L:functional properties:decreasing function sequence}, we can neglect the second term  in the left-hand side of the above inequality,  finally getting the desired  \\eqref{eq:condition}.\n","start":69093,"end":69386}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":["eq:decay_complete"],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"递归累加能量界","source_units":[{"raw_tex":"Iterating \\eqref{eq:condition}  recursively gives\n\\begin{align}  \\label{eq:decay_complete}\n\\mathcal{E}(k) \\leq \\mathcal{E}(1) +\\sum_{j=1}^{k-1}t_{j+1}\\Big( d_{\\lambda_{j}}(u^\\dagger)-\\inf d_0 \\Big).\n\\end{align}\n","start":69386,"end":69597}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"统一控制对角残差","source_units":[{"raw_tex":"To  bound the sum appearing on the right hand side, we need to analyze the residuals $r_j:= d_{\\lambda_{j}}(u^\\dagger) -\\inf d_0$. Similarly as for the estimation obtained in the continuous case, we can use for this purpose the property in Lemma \\ref{L:functional properties}\\ref{L:functional properties:diagonal rates via conditioning} and get that for some fixed constant $c$ independent on $j$ (defined analogously as in \\eqref{proof:ccd3}), we have\n$$\nr_j\\leq c \\lambda_j^{\\frac{1}{q-1}},\\qquad\\text{for every }j\\geq 1.\n$$\n","start":69597,"end":70124}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"残差加权和有界","source_units":[{"raw_tex":"By assumption \\ref{assumption:parameter:relaxation}, with $\\Lambda$ as  in \\eqref{def:biglambda}, we thus conclude that\n$$\n\\sum_{j=1}^{k-1} t_{j+1} r_j \\leq  c \\sum_{j=1}^{k-1} t_{j+1} \\lambda_j^{\\frac{1}{q-1}}  \\leq c \\Lambda < +\\infty\n$$\n","start":70124,"end":70364}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"提取对偶收敛率","source_units":[{"raw_tex":"This allows us to deduce from \\eqref{eq:decay_complete} the convergence rate on the dual values in \\eqref{eq:rate}, by taking $C = \\Ee(1) + c \\Lambda$.\n","start":70364,"end":70516}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"转换原变量误差界","source_units":[{"raw_tex":"Finally, the convergence rate on the primal iterates  in \\eqref{eq:rate} follows from  Lemma \\ref{L:functional properties}\\ref{L:functional properties:from dual value to primal iterates}.\n","start":70516,"end":70704}]},{"id":"S057","semantic_type":"lemma","kind":"lemma","labels":["L:functional properties","L:functional properties:L at 0","L:functional properties:R_A Lipschitz","L:functional properties:argmin d_0 nonempty","L:functional properties:decreasing function sequence","L:functional properties:diagonal rates via conditioning","L:functional properties:from dual value to primal iterates"],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"对偶函数的六项性质","source_units":[{"raw_tex":"\\begin{lemma}\\label{L:functional properties}\nUnder the assumptions \\ref{assumption:data-fit convex and coercive}-\\ref{assumption:data-fit conditionned locally} and \\ref{H:R strongly convex}-\\ref{assumption:AR2},  we have:\n\\begin{enumerate}[label=(\\roman*)]\n\t\\item\\label{L:functional properties:R_A Lipschitz} $\\Rr_A$ is differentiable and $\\nabla  \\RA$ is Lipschitz continuous, with Lipschitz constant equal to $\\sigma^{-1} \\Vert A \\Vert^2$.\n\t\\item\\label{L:functional properties:L at 0} $\\forall y \\in \\Yy$, $\\ell^*_{y}(0)=0$ and $\\partial \\ell^*_y(0)=\\{y\\}$.\n\t\\item\\label{L:functional properties:argmin d_0 nonempty} There holds: $\\argmin  d_0 \\neq \\emptyset$.\n \t\\item\\label{L:functional properties:decreasing function sequence} $\\forall u \\in \\Yy$, the function $\\lambda\\in [0,+\\infty) \\mapsto \\ d_{\\lambda}(u)$ is nondecreasing. \n\t\\item\\label{L:functional properties:from dual value to primal iterates} $\\forall t >0, \\forall u \\in \\Yy, \\ \\text{ if }\n{x}:=\\nabla R^*(-A^*u),$ then $\n \\frac{\\sigma}{2} \\Vert {x} - x^\\dagger \\Vert^2 \\leq d_0(u) - \\inf d_0.$ \n \t\\item\\label{L:functional properties:diagonal rates via conditioning} $\\forall u^\\dagger \\in \\argmin  d_0$, if  $\\lambda \\Vert u^\\dagger \\Vert \\leq \\frac{\\gamma}{q} \\varrho^{q-1}$, then\n \t\\begin{equation}\n \t    d_{\\lambda}(u^\\dagger) - \\inf  d_0 \\leq\n \t    \\begin{cases}\n \t     0 & \\text{ if } q=1, \\\\\n \t     (1-\\frac{1}{q}) {\\gamma^{-1/(q-1)}} \\Vert u^\\dagger \\Vert^{q/(q-1)} \\lambda^{1/(q-1)} & \\text{ if } q>1.\n \t    \\end{cases}\n \t\\end{equation}\n\\end{enumerate}\n\\end{lemma}","start":115130,"end":116667}]},{"id":"S058","semantic_type":"lemma","kind":"lemma","labels":["L:functional properties","L:functional properties:L at 0","L:functional properties:R_A Lipschitz","L:functional properties:argmin d_0 nonempty","L:functional properties:decreasing function sequence","L:functional properties:diagonal rates via conditioning","L:functional properties:from dual value to primal iterates"],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"复合项的 Lipschitz 梯度","source_units":[{"raw_tex":"\t\\item\\label{L:functional properties:R_A Lipschitz} $\\Rr_A$ is differentiable and $\\nabla  \\RA$ is Lipschitz continuous, with Lipschitz constant equal to $\\sigma^{-1} \\Vert A \\Vert^2$.\n","start":115387,"end":115572}]},{"id":"S059","semantic_type":"lemma","kind":"lemma","labels":["L:functional properties","L:functional properties:L at 0","L:functional properties:R_A Lipschitz","L:functional properties:argmin d_0 nonempty","L:functional properties:decreasing function sequence","L:functional properties:diagonal rates via conditioning","L:functional properties:from dual value to primal iterates"],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"数据共轭在零处的性质","source_units":[{"raw_tex":"\t\\item\\label{L:functional properties:L at 0} $\\forall y \\in \\Yy$, $\\ell^*_{y}(0)=0$ and $\\partial \\ell^*_y(0)=\\{y\\}$.\n","start":115572,"end":115690}]},{"id":"S060","semantic_type":"lemma","kind":"lemma","labels":["L:functional properties","L:functional properties:L at 0","L:functional properties:R_A Lipschitz","L:functional properties:argmin d_0 nonempty","L:functional properties:decreasing function sequence","L:functional properties:diagonal rates via conditioning","L:functional properties:from dual value to primal iterates"],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"极限对偶解存在","source_units":[{"raw_tex":"\t\\item\\label{L:functional properties:argmin d_0 nonempty} There holds: $\\argmin  d_0 \\neq \\emptyset$.\n","start":115690,"end":115792}]},{"id":"S061","semantic_type":"lemma","kind":"lemma","labels":["L:functional properties","L:functional properties:L at 0","L:functional properties:R_A Lipschitz","L:functional properties:argmin d_0 nonempty","L:functional properties:decreasing function sequence","L:functional properties:diagonal rates via conditioning","L:functional properties:from dual value to primal iterates"],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"对偶目标随参数单调","source_units":[{"raw_tex":" \t\\item\\label{L:functional properties:decreasing function sequence} $\\forall u \\in \\Yy$, the function $\\lambda\\in [0,+\\infty) \\mapsto \\ d_{\\lambda}(u)$ is nondecreasing. \n","start":115792,"end":115963}]},{"id":"S062","semantic_type":"lemma","kind":"lemma","labels":["L:functional properties","L:functional properties:L at 0","L:functional properties:R_A Lipschitz","L:functional properties:argmin d_0 nonempty","L:functional properties:decreasing function sequence","L:functional properties:diagonal rates via conditioning","L:functional properties:from dual value to primal iterates"],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"对偶间隙控制原误差","source_units":[{"raw_tex":"\t\\item\\label{L:functional properties:from dual value to primal iterates} $\\forall t >0, \\forall u \\in \\Yy, \\ \\text{ if }\n{x}:=\\nabla R^*(-A^*u),$ then $\n \\frac{\\sigma}{2} \\Vert {x} - x^\\dagger \\Vert^2 \\leq d_0(u) - \\inf d_0.$ \n","start":115963,"end":116190}]},{"id":"S063","semantic_type":"lemma","kind":"lemma","labels":["L:functional properties","L:functional properties:L at 0","L:functional properties:R_A Lipschitz","L:functional properties:argmin d_0 nonempty","L:functional properties:decreasing function sequence","L:functional properties:diagonal rates via conditioning","L:functional properties:from dual value to primal iterates"],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"条件性给出的残差界","source_units":[{"raw_tex":" \t\\item\\label{L:functional properties:diagonal rates via conditioning} $\\forall u^\\dagger \\in \\argmin  d_0$, if  $\\lambda \\Vert u^\\dagger \\Vert \\leq \\frac{\\gamma}{q} \\varrho^{q-1}$, then\n \t\\begin{equation}\n \t    d_{\\lambda}(u^\\dagger) - \\inf  d_0 \\leq\n \t    \\begin{cases}\n \t     0 & \\text{ if } q=1, \\\\\n \t     (1-\\frac{1}{q}) {\\gamma^{-1/(q-1)}} \\Vert u^\\dagger \\Vert^{q/(q-1)} \\lambda^{1/(q-1)} & \\text{ if } q>1.\n \t    \\end{cases}\n \t\\end{equation}\n","start":116190,"end":116640}]},{"id":"S064","semantic_type":"assumption","kind":"assumption","labels":["L:functional properties","L:functional properties:L at 0","L:functional properties:R_A Lipschitz","L:functional properties:argmin d_0 nonempty","L:functional properties:decreasing function sequence","L:functional properties:diagonal rates via conditioning","L:functional properties:from dual value to primal iterates"],"member_ids":["S064"],"declared_external":false,"source_only":false,"title":"残差结论的局部适用域","source_units":[{"raw_tex":" \t\\item\\label{L:functional properties:diagonal rates via conditioning} $\\forall u^\\dagger \\in \\argmin  d_0$, if  $\\lambda \\Vert u^\\dagger \\Vert \\leq \\frac{\\gamma}{q} \\varrho^{q-1}$, then\n","start":116190,"end":116377}]},{"id":"S065","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S065"],"declared_external":false,"source_only":false,"title":"六项性质的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{L:functional properties:R_A Lipschitz}: follows  from the strong convexity of $R$, see, e.g.,  \n\\cite[Theorem 18.15]{BauschkeCombettes2017}.\\\\\n\\ref{L:functional properties:L at 0}: it is a simple consequence of the properties of the Fenchel transform as it can be found, e.g., in \n \\cite[Proposition 13.10(i) \\& Corollary 16.30]{BauschkeCombettes2017}.\\\\\n\\ref{L:functional properties:argmin d_0 nonempty} and \\ref{L:functional properties:from dual value to primal iterates}: follow from \\cite[Lemma 5]{Garrigos2017} by simply taking $f=R$ and $g=\\delta_{\\{\\bar y\\}}$, while property \\ref{L:functional properties:decreasing function sequence} has been proved in \\cite[Proposition 2(i)]{Garrigos2017}.\\\\\n\\ref{L:functional properties:diagonal rates via conditioning}: it is enough to verify that $\\ell_{\\bar y}(\\cdot)$ is $q$-well conditioned in the sense of \\cite[Definition 1]{Garrigos2017}, while assumption \\ref{assumption:data-fit conditionned locally} holds only locally.\nTo check this, we introduce the function $\\psi : \\R \\rightarrow \\R$ defined for the $\\varrho>0$ appearing in \\ref{assumption:data-fit conditionned locally}  by:\n\\begin{equation}\n \\psi  t \\mapsto\n\\begin{cases}\n \\frac{\\gamma}{q}\\vert t \\vert^q  & \\text{ if } \\vert t \\vert \\leq \\varrho, \\\\\n \\frac{\\gamma}{q} \\varrho^{q-1} \\vert t \\vert & \\text{ if } \\vert t \\vert > \\varrho.\n\\end{cases}\n\\end{equation}\nFrom \\ref{assumption:data-fit conditionned locally}, we easily  deduce that $\\ell_{\\bar y}(y) \\geq \\psi(\\Vert y- \\bar y \\Vert)$ for all $y \\in \\Yy$ (see \\cite[Corollary 3.4.2]{ZalinescuBook2002}).\nNote that $\\psi$ is not convex for $q >1$, so in this case we consider instead the function\n\\begin{equation}\n m: \\R \\rightarrow \\R , \\quad t \\mapsto \n\\begin{cases}\n \\frac{\\gamma}{q}\\vert t \\vert^q  & \\text{ if } \\vert t \\vert \\leq q^{1/(1-q)} \\varrho, \\\\\n \\frac{\\gamma}{q}\\varrho^{q-1}\\vert t \\vert \n -  \\frac{\\gamma}{q^{\\frac{q}{q-1}}}\\varrho^q (1 - \\frac{1}{q})  & \\text{ if } \\vert t \\vert > q^{1/(1-q)} \\varrho,\n\\end{cases}\n\\end{equation}\nand define $m:=\\psi$ for $q=1$.\nIt is an easy exercise to verify that $m$ is indeed a convex function on $\\R$, and that $m(w) \\leq \\psi(w)$ for all $w\\in\\R$.\nNow, we can make use of \\cite[Lemma 2]{Garrigos2017},\nwhich tells us that $d_\\lambda(u) - \\inf d_0 \\leq \\lambda^{-1} m^*(\\Vert u \\Vert \\lambda)$.\nThe desired result  now follows from the computation of the Fenchel transform of $m$.\nIf $q=1$, we have that $m(t) = \\gamma \\vert t \\vert$, so classic Fenchel calculus entails that $m^*$ is $\\delta_{[-\\gamma,\\gamma]}$, the indicator function of $[-\\gamma,\\gamma]$.\nIf $q>1$, easy computations show that $m^*$ reads\n\\begin{equation}\n m^* : \\R \\rightarrow \\R, \\quad s \\mapsto\n\\begin{cases}\n (1 - \\frac{1}{q}) \\gamma^{-1/(q-1)}\\vert s \\vert^{\\frac{q}{q-1}}  & \\text{ if } \\vert s \\vert \\leq \\frac{\\gamma}{q} \\varrho^{q-1}, \\\\\n + \\infty  & \\text{ if } \\vert s \\vert > \\frac{\\gamma}{q} \\varrho^{q-1}.\n\\end{cases}\n\\end{equation}\nBy now applying \\cite[Lemma 2]{Garrigos2017} we  conclude. \n\\end{proof}","start":116668,"end":119700}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S066"],"declared_external":false,"source_only":false,"title":"由强凸性得到光滑性","source_units":[{"raw_tex":"\\ref{L:functional properties:R_A Lipschitz}: follows  from the strong convexity of $R$, see, e.g.,  \n\\cite[Theorem 18.15]{BauschkeCombettes2017}.\\\\\n","start":116682,"end":116830}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S067"],"declared_external":false,"source_only":false,"title":"引用共轭零点性质","source_units":[{"raw_tex":"\\ref{L:functional properties:L at 0}: it is a simple consequence of the properties of the Fenchel transform as it can be found, e.g., in \n \\cite[Proposition 13.10(i) \\& Corollary 16.30]{BauschkeCombettes2017}.\\\\\n","start":116830,"end":117042}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S068"],"declared_external":false,"source_only":false,"title":"特化对偶误差引理","source_units":[{"raw_tex":"\\ref{L:functional properties:argmin d_0 nonempty} and \\ref{L:functional properties:from dual value to primal iterates}: follow from \\cite[Lemma 5]{Garrigos2017} by simply taking $f=R$ and $g=\\delta_{\\{\\bar y\\}}$, while property \\ref{L:functional properties:decreasing function sequence} has been proved in \\cite[Proposition 2(i)]{Garrigos2017}.\\\\\n","start":117042,"end":117389}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S069"],"declared_external":false,"source_only":false,"title":"引用参数单调性","source_units":[{"raw_tex":"\\ref{L:functional properties:argmin d_0 nonempty} and \\ref{L:functional properties:from dual value to primal iterates}: follow from \\cite[Lemma 5]{Garrigos2017} by simply taking $f=R$ and $g=\\delta_{\\{\\bar y\\}}$, while property \\ref{L:functional properties:decreasing function sequence} has been proved in \\cite[Proposition 2(i)]{Garrigos2017}.\\\\\n","start":117042,"end":117389}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S070"],"declared_external":false,"source_only":false,"title":"转向全局良态条件","source_units":[{"raw_tex":"\\ref{L:functional properties:diagonal rates via conditioning}: it is enough to verify that $\\ell_{\\bar y}(\\cdot)$ is $q$-well conditioned in the sense of \\cite[Definition 1]{Garrigos2017}, while assumption \\ref{assumption:data-fit conditionned locally} holds only locally.\n","start":117389,"end":117662}]},{"id":"S071","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S071"],"declared_external":false,"source_only":false,"title":"局部幂次与线性延拓","source_units":[{"raw_tex":"To check this, we introduce the function $\\psi : \\R \\rightarrow \\R$ defined for the $\\varrho>0$ appearing in \\ref{assumption:data-fit conditionned locally}  by:\n\\begin{equation}\n \\psi  t \\mapsto\n\\begin{cases}\n \\frac{\\gamma}{q}\\vert t \\vert^q  & \\text{ if } \\vert t \\vert \\leq \\varrho, \\\\\n \\frac{\\gamma}{q} \\varrho^{q-1} \\vert t \\vert & \\text{ if } \\vert t \\vert > \\varrho.\n\\end{cases}\n\\end{equation}\n","start":117662,"end":118062}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S072"],"declared_external":false,"source_only":false,"title":"获得全局径向下界","source_units":[{"raw_tex":"From \\ref{assumption:data-fit conditionned locally}, we easily  deduce that $\\ell_{\\bar y}(y) \\geq \\psi(\\Vert y- \\bar y \\Vert)$ for all $y \\in \\Yy$ (see \\cite[Corollary 3.4.2]{ZalinescuBook2002}).\n","start":118062,"end":118259}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S073"],"declared_external":false,"source_only":false,"title":"构造凸下界","source_units":[{"raw_tex":"Note that $\\psi$ is not convex for $q >1$, so in this case we consider instead the function\n\\begin{equation}\n m: \\R \\rightarrow \\R , \\quad t \\mapsto \n\\begin{cases}\n \\frac{\\gamma}{q}\\vert t \\vert^q  & \\text{ if } \\vert t \\vert \\leq q^{1/(1-q)} \\varrho, \\\\\n \\frac{\\gamma}{q}\\varrho^{q-1}\\vert t \\vert \n -  \\frac{\\gamma}{q^{\\frac{q}{q-1}}}\\varrho^q (1 - \\frac{1}{q})  & \\text{ if } \\vert t \\vert > q^{1/(1-q)} \\varrho,\n\\end{cases}\n\\end{equation}\nand define $m:=\\psi$ for $q=1$.\nIt is an easy exercise to verify that $m$ is indeed a convex function on $\\R$, and that $m(w) \\leq \\psi(w)$ for all $w\\in\\R$.\n","start":118259,"end":118860}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S074"],"declared_external":false,"source_only":false,"title":"调用标量共轭残差界","source_units":[{"raw_tex":"Now, we can make use of \\cite[Lemma 2]{Garrigos2017},\nwhich tells us that $d_\\lambda(u) - \\inf d_0 \\leq \\lambda^{-1} m^*(\\Vert u \\Vert \\lambda)$.\nThe desired result  now follows from the computation of the Fenchel transform of $m$.\n","start":118860,"end":119092}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S075"],"declared_external":false,"source_only":false,"title":"线性条件性的共轭","source_units":[{"raw_tex":"If $q=1$, we have that $m(t) = \\gamma \\vert t \\vert$, so classic Fenchel calculus entails that $m^*$ is $\\delta_{[-\\gamma,\\gamma]}$, the indicator function of $[-\\gamma,\\gamma]$.\n","start":119092,"end":119271}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S076"],"declared_external":false,"source_only":false,"title":"幂次条件性的共轭","source_units":[{"raw_tex":"If $q>1$, easy computations show that $m^*$ reads\n\\begin{equation}\n m^* : \\R \\rightarrow \\R, \\quad s \\mapsto\n\\begin{cases}\n (1 - \\frac{1}{q}) \\gamma^{-1/(q-1)}\\vert s \\vert^{\\frac{q}{q-1}}  & \\text{ if } \\vert s \\vert \\leq \\frac{\\gamma}{q} \\varrho^{q-1}, \\\\\n + \\infty  & \\text{ if } \\vert s \\vert > \\frac{\\gamma}{q} \\varrho^{q-1}.\n\\end{cases}\n\\end{equation}\n","start":119271,"end":119629}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S077"],"declared_external":false,"source_only":false,"title":"完成分情况残差估计","source_units":[{"raw_tex":"By now applying \\cite[Lemma 2]{Garrigos2017} we  conclude. \n","start":119629,"end":119689}]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S078"],"declared_external":true,"declaration_text":["原始先修列表：第1188–1191行引用的 AttouchTikAccel 与 ChambolleDossal2015 的 forward-backward descent inequality。"],"source_only":false,"title":"前向后向下降引理","source_units":[]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S079"],"declared_external":true,"declaration_text":["原始先修列表：BauschkeCombettes2017, Proposition 13.10(i) 与 Corollary 16.30；第1946–1947行明确调用。"],"source_only":false,"title":"Fenchel 零点与极小点规则","source_units":[]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S080"],"declared_external":true,"declaration_text":["原始先修列表：Garrigos2017, Lemma 5；第1948行取 f=R、g=\\delta_{\\{\\bar y\\}}。"],"source_only":false,"title":"源条件下的对偶误差引理","source_units":[]},{"id":"S081","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S081"],"declared_external":true,"declaration_text":["原始先修列表：Garrigos2017, Proposition 2(i)；第1948行明确调用。"],"source_only":false,"title":"对角目标的参数单调性","source_units":[]},{"id":"S082","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S082"],"declared_external":true,"declaration_text":["原始先修列表：ZalinescuBook2002, Corollary 3.4.2；第1958行明确调用。"],"source_only":false,"title":"凸函数局部下界延拓","source_units":[]},{"id":"S083","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S083"],"declared_external":true,"declaration_text":["原始先修列表：Garrigos2017, Lemma 2；第1970–1971及1982行调用。"],"source_only":false,"title":"由标量下界到对偶残差","source_units":[]},{"id":"S084","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S084"],"declared_external":true,"declaration_text":["原始先修列表：Hilbert-space norm identities；implicit standard fact made explicit by annotator。"],"source_only":false,"title":"Hilbert 平方范数恒等式","source_units":[]},{"id":"S085","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S085"],"declared_external":true,"declaration_text":["原始先修列表：standard convex conjugacy、subgradient calculus；implicit standard fact made explicit by annotator。"],"source_only":false,"title":"复合梯度与标量共轭规则","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076","S077":"S077","S078":"S078","S079":"S079","S080":"S080","S081":"S081","S082":"S082","S083":"S083","S084":"S084","S085":"S085"}}}}
{"record_id":"1705.06019v2","paper":{"title":"Regularizing with Bregman-Moreau envelopes","authors":["Bauschke, Heinz H.","Dao, Minh N.","Lindstrom, Scott B."],"arxiv_id":"1705.06019","version":"v2","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/1705.06019v2","pdf_url":"https://arxiv.org/pdf/1705.06019v2","doi":"10.1137/17M1130745"},"context":{"raw_tex":"We assume throughout that\n\\begin{empheq}[box=\\mybluebox]{equation}\n\\text{$X := \\RR^J$,}\n\\end{empheq}\nwhich we equip with \nthe standard inner product $\\scal{\\cdot}{\\cdot}$ and \nthe induced Euclidean norm $\\|\\cdot\\|$.\n\n\nNow assume that \n\\begin{empheq}[box=\\mybluebox]{equation}\n\\text{$f \\colon X \\to\\left]-\\infty, +\\infty\\right]$ is convex and\ndifferentiable on $U := \\intdom f \\neq\\varnothing$.}\n\\end{empheq}\nThe \\emph{Bregman distance}\\footnote{Note that $D_f$ is\n\\emph{not} a distance in the sense of metric topology; however,\nthis naming convention is now ubiquitous.}  associated with $f$, \nfirst explored by Bregman in \\cite{Bre67} (see also\n\\cite{CZ97}), is \n\\begin{equation} \n\\label{e:Df}\nD_f \\colon X \\times X \\to \\left[0, +\\infty\\right] \\colon (x,y) \\mapsto\n\\begin{cases}\nf(x) -f(y) -\\scal{\\nabla f(y)}{x -y}, &\\text{~if~} y\\in U; \\\\\n+\\infty, &\\text{~otherwise}.\n\\end{cases}\n\\end{equation} \n\n\nHowever, because different assignments of $f$ may allow for\ncases in which \n$D_f(x,y)\\neq D_f(y,x)$, we actually are led to consider\n\\emph{two} envelopes:\nthe \\emph{left} and \\emph{right} \n\\emph{Bregman--Moreau envelopes} are defined by\n\\begin{equation}\n\\label{e:bba}\n\\benv[\\gamma]{\\theta} \\colon X\\to \\left[-\\infty, +\\infty\\right] \\colon\ny\\mapsto \\inf_{x\\in X} \\big(\\theta(x) +\\frac{1}{\\gamma}D_f(x,y)\\big)\n\\end{equation}\nand \n\\begin{equation}\n\\label{e:fba}\n\\fenv[\\gamma]{\\theta} \\colon X \\to \\left[-\\infty, +\\infty\\right] \\colon\nx \\mapsto\n\\inf_{y\\in X} \\big(\\theta(y) +\\frac{1}{\\gamma}D_f(x,y)\\big),\n\\end{equation}\nrespectively. \n\n\n\\begin{proposition}\n\\label{p:12.22i}\nLet $\\theta \\colon X \\to \\left]-\\infty, +\\infty\\right]$, \nlet $\\gamma \\in \\RPP$, and let $\\mu \\in \\RPP$.\nThen $\\benv[\\mu]{\\gamma\\theta} =\\gamma\\benv[\\gamma\\mu]{\\theta}$ \nand $\\fenv[\\mu]{\\gamma\\theta} =\\gamma\\fenv[\\gamma\\mu]{\\theta}$.\n\\end{proposition}\n\nDenote by $\\Gamma_0(X)$ the set of all proper lower semicontinuous convex functions from $X$ to $\\left]-\\infty, +\\infty\\right]$. \nFrom now on, we strengthen our assumptions by requiring that \n\\begin{empheq}[box=\\mybluebox]{equation}\n\\label{e:fLegendre}\n\\text{$f\\in\\Gamma_0(X)$ is a convex function of Legendre type and\n$U := \\intdom f$.}\n\\end{empheq}\nThis will allow us to obtain a quite satisfying theory in which\nthe envelopes are convex functions. \nNote that $f$ is essentially smooth and essentially strictly convex in the sense of \\cite[Section~26]{Roc70}.\nIt is well known that\n\\begin{equation}\n\\label{e:Leggy}\n\\nabla f\\colon U\\to U^*:= \\intdom f^*\n\\text{~is a homeomorphism with~}\n\\big(\\nabla f\\big)^{-1} = \\nabla f^*.\n\\end{equation}\nWe will also work with the following standard assumptions:\n\\begin{itemize}\n\\item[\\bfseries A1] \n$\\nabla^2 f$ exists and is continuous on $U$;\n\\item[\\bfseries A2] \n$D_f$ is \\emph{jointly convex}, i.e., convex on $X\\times X$;\n\\item[\\bfseries A3] \n$(\\forall x\\in U)$ $D_f(x,\\cdot)$ is strictly convex on $U$;\n\\item[\\bfseries A4] \n$(\\forall x\\in U)$ $D_f(x,\\cdot)$ is coercive, i.e., \n$D_f(x, y) \\to +\\infty$ as $\\|y\\| \\to +\\infty$.\n\\end{itemize}\nWe also henceforth assume that \n\\begin{empheq}[box=\\mybluebox]{equation}\n\\text{all assumptions \\textbf{A1}--\\textbf{A4} hold.}\n\\end{empheq}\n\n\n\\begin{fact} \n\\label{f:coeR}\nLet $\\theta\\in\\Gamma_0(X)$ be such that $\\dom\\theta\\cap U\\neq\\varnothing$ and let $\\gamma \\in \\RPP$.\nConsider the following properties:\n\\begin{itemize}\n\\item[\\rm (a)]\n$U\\cap \\dom\\theta$ is bounded.\n\\item[\\rm (b)]\n$\\inf\\theta(U)>-\\infty$.\n\\item[\\rm (c)]\n$f$ is supercoercive, i.e., \n$f(x)/\\|x\\| \\to +\\infty$ as $\\|x\\| \\to +\\infty$.\n\\item[\\rm (d)]\n$(\\forall x\\in U)$ $D_f(x,\\cdot)$ is supercoercive.\n\\end{itemize}\nThen the following hold:\n\\begin{enumerate}\n\\item \\label{l:coeRi}\nIf any of the conditions (a), (b), or (c) holds, then \n\\begin{equation} \\label{e:bcoeR}\n(\\forall y \\in U) \\quad \\theta(\\cdot) + \\frac{1}{\\gamma}D_f(\\cdot,y) \\;\\;\n\\text{is coercive}\n\\end{equation}\nor, equivalently,\n\\begin{equation}\n\\frac{1}{\\gamma}\\ran\\nabla f \\subseteq \\intdom\\left(\\frac{1}{\\gamma}f +\\theta\\right)^*.\n\\label{e:brilliantissimo}\n\\end{equation}\n\n\\item \\label{l:coeRii}\nIf any of the conditions (a), (b), or (d) holds, then \n\\begin{equation} \\label{e:fcoeR}\n(\\forall x \\in U) \\quad \\theta(\\cdot) + \\frac{1}{\\gamma}D_f(x,\\cdot) \\;\\;\n\\text{is coercive.}\n\\end{equation}\n\\end{enumerate}\n\\end{fact}\n\n\\begin{proposition} \n\\label{p:exact} \nLet $\\theta \\colon X \\to \\left]-\\infty, +\\infty\\right]$ be convex\nand such that $U\\cap \\dom\\theta\\neq\\varnothing$, and let $\\gamma \\in \\RPP$.\nThen the following hold:\n\\begin{enumerate}\n\\item \n\\label{p:exact_left}\n$\\benv[\\gamma]{\\theta}$ is convex and continuous on $U$, and\n\t\\begin{enumerate}\n\t\\item \n\tif \\eqref{e:bcoeR} holds, i.e., $(\\forall y \\in U)$ $\\theta(\\cdot) +\\frac{1}{\\gamma}D_f(\\cdot,y)$ is coercive, \n\tthen $\\benv[\\gamma]{\\theta}$ is proper; \n\t\\item\n\tif $\\theta\\in\\Gamma_0(X)$ and $\\theta(\\cdot) +\\frac{1}{\\gamma}D_f(\\cdot,y)$ is coercive for a given $y\\in U$, \n\tthen there exists a unique point $z\\in U$ such that $\\benv[\\gamma]{\\theta}(y)=\\theta(z) +\\frac{1}{\\gamma}D_f(z,y)$.\n\t\\end{enumerate}\n\n\\item \n\\label{p:exact_right} \n$\\fenv[\\gamma]{\\theta}$ is convex and continuous on $U$, and\n\t\\begin{enumerate}\n\t\\item \n\tif \\eqref{e:fcoeR} holds, i.e., $(\\forall x \\in U)$ $\\theta(\\cdot) +\\frac{1}{\\gamma}D_f(x,\\cdot)$ is coercive, \n\tthen $\\fenv[\\gamma]{\\theta}$ is proper; \n\t\\item\n\tif $\\theta\\in\\Gamma_0(X)$ and $\\theta(\\cdot) +\\frac{1}{\\gamma}D_f(x,\\cdot)$ is coercive for a given $x\\in U$, \n\tthen there exists a unique point $z\\in U$ such that $\\fenv[\\gamma]{\\theta}(z)=\\theta(z) +\\frac{1}{\\gamma}D_f(x,z)$.\n\t\\end{enumerate}\n\\end{enumerate}\n\\end{proposition}\n\n\\begin{definition}[Bregman proximity operators] \n\\label{d:prox}\nLet $\\theta\\in\\Gamma_0(X)$ be such that \n$U\\cap \\dom\\theta\\neq\\varnothing$.\nIf \\eqref{e:bcoeR} holds for $\\gamma =1$, then \nthe \\emph{left proximity operator} associated with $\\theta$ is\n\\begin{equation} \\label{e:dbprox}\n\\bprox{\\theta} \\colon U \\to U \\colon\ny\\mapsto\n\\underset{x\\in X}{\\operatorname{argmin}}\\;\\: \\big(\\theta(x) +D_f(x,y)\\big).\n\\end{equation}\nIf \\eqref{e:fcoeR} holds for $\\gamma =1$, then \nthe \\emph{right proximity operator} associated with $\\theta$ is\n\\begin{equation} \\label{e:dfprox}\n\\fprox{\\theta}\\colon U \\to  U \\colon\nx \\mapsto\n\\underset{y\\in X}{\\operatorname{argmin}}\\;\\: \\big(\\theta(y) +D_f(x,y)\\big).\n\\end{equation}\n\\end{definition}\n\n\nGiven a closed convex subset $C$ of $X$ with $C\\cap U \\neq\\varnothing$, we have that $\\iota_C \\in \\Gamma_0(X)$, $\\dom\\iota_C =C$, \nand hence $U\\cap \\dom\\iota_C =U\\cap C \\neq\\varnothing$ and also $\\inf \\iota_C(U) =0 >-\\infty$, \nwhich together with \\cref{f:coeR} imply that \\eqref{e:bcoeR} and \\eqref{e:fcoeR} hold for all $\\gamma \\in \\RPP$. \nThis leads to the following definition. \n\n\\begin{definition}[Bregman projectors]\n\\label{d:BregProj}\nLet $C$ be a closed convex subset of $X$ such that \n$U\\cap C\\neq\\varnothing$. \nThen $\\bproj{C} :=\\bprox{\\iota_C}$ is the \\emph{left Bregman projector} onto $C$\nand $\\fproj{C} :=\\fprox{\\iota_C}$ is the \\emph{right Bregman projector} onto $C$.\n\\end{definition}\n\n\n\\begin{remark}\n\\label{r:prox}\nLet $\\theta\\in\\Gamma_0(X)$ be such that \n$U\\cap \\dom\\theta\\neq\\varnothing$ and let $\\gamma \\in \\RPP$.\n\\cref{p:12.22i} implies that \n$\\benv{\\gamma\\theta} =\\gamma\\benv[\\gamma]{\\theta}$ \nand $\\fenv{\\gamma\\theta} =\\gamma\\fenv[\\gamma]{\\theta}$. \nWe thus derive from the definition that if \\eqref{e:bcoeR} holds, then\n\\begin{subequations}\n\\label{e:benv-bproxall}\n\\begin{equation}\n\\label{e:benv-bprox}\n\\benv[\\gamma]{\\theta}(y) =\\theta\\big(\\bprox{\\gamma\\theta}(y)\\big) \n+\\frac{1}{\\gamma}D_f\\big(\\bprox{\\gamma\\theta}(y), y\\big)\n\\end{equation}\nand, by combining with \\cref{p:12.9}\\ref{p:12.9_left},\n\\begin{equation}\n\\label{e:benv-bprox'}\n\\theta\\big(\\bprox{\\gamma\\theta}(y)\\big) \\leq \\benv[\\gamma]{\\theta}(y) \\leq \\theta(y).\n\\end{equation}\n\\end{subequations}\nSimilarly, if \\eqref{e:fcoeR} holds, then\n\\begin{subequations}\n\\label{e:fenv-fproxall}\n\\begin{equation}\n\\label{e:fenv-fprox}\n\\fenv[\\gamma]{\\theta}(x) =\\theta\\big(\\fprox{\\gamma\\theta}(x)\\big) +\n\\frac{1}{\\gamma}D_f\\big(x, \\fprox{\\gamma\\theta}(x)\\big)\n\\end{equation}\nand \n\\begin{equation}\n\\theta\\big(\\fprox{\\gamma\\theta}(x)\\big) \\leq \\fenv[\\gamma]{\\theta}(x) \\leq \\theta(x).\n\\end{equation}\n\\end{subequations}\n\\end{remark}\n\n\\begin{proposition}\n\\label{p:prox}\nLet $\\theta\\in\\Gamma_0(X)$ be such that $\\dom\\theta\\cap U\\neq\\varnothing$ and let $\\gamma \\in \\RPP$.\n\\begin{enumerate}\n\\item \n\\label{p:prox_left}\nSuppose that \\eqref{e:bcoeR} holds.\nThen for every $(x,y) \\in U\\times U$, \nthe following conditions are equivalent:\n\t\\begin{enumerate}\n\t\\item \n\t$x =\\bprox{\\gamma\\theta}(y)$,\n\t\\item \n\t$0 \\in \\gamma\\partial\\theta(x) +\\nabla f(x) -\\nabla f(y)$,\n\t\\item \n\t$(\\forall z \\in X)\\quad \\scal{\\nabla f(y) -\\nabla f(x)}{z -x} +\\gamma\\theta(x) \\leq \\gamma\\theta(z)$.\n\t\\end{enumerate}\nMoreover, \n\\begin{equation} \\label{e:pro}\n\\bprox{\\gamma\\theta}=(\\nabla f +\\gamma\\partial\\theta)^{-1}\\circ \\nabla f \n\\end{equation}\nis continuous on $U$. \n\\item \n\\label{p:prox_right}\nSuppose that \\eqref{e:fcoeR} holds. \nThen for every $(x,y) \\in U\\times U$, \nthe following conditions are equivalent:\n\t\\begin{enumerate}\n\t\\item \n\t$y =\\fprox{\\gamma\\theta}(x)$,\n\t\\item \n\t$0 \\in \\gamma\\partial\\theta(y) +\\nabla^2 f(y)(y -x)$,\n\t\\item \n\t$(\\forall z \\in X)\\quad \\scal{\\nabla^2 f(y)(x -y)}{z -y} +\\gamma\\theta(y) \\leq \\gamma\\theta(z)$.\n\t\\end{enumerate}\nMoreover, $\\fprox{\\gamma\\theta}$ is continuous on $U$. \n\\end{enumerate}\n\\end{proposition}\n\n\\begin{lemma}\n\\label{l:cvg}\nLet $C$ be a compact subset of a Hausdorff space $\\mathcal X$,\nlet $\\phi\\colon \\mathcal X \\to \\left[-\\infty, +\\infty\\right]$ be lower \nsemicontinuous,\nlet $(x_a)_{a\\in A}$ be a net in $C$,\nand suppose that $\\phi(x_a) \\to \\inf\\phi(\\mathcal X)$. \nThen $\\argmin\\phi \\neq\\varnothing$ \nand all cluster points of $(x_a)_{a\\in A}$ lie in $\\argmin\\phi$.\nConsequently, if $\\phi$ attains its minimum at a unique point $u$, then $x_a \\to u$.\n\\end{lemma}","units":[{"raw_tex":"We assume throughout that\n\\begin{empheq}[box=\\mybluebox]{equation}\n\\text{$X := \\RR^J$,}\n\\end{empheq}\nwhich we equip with \nthe standard inner product $\\scal{\\cdot}{\\cdot}$ and \nthe induced Euclidean norm $\\|\\cdot\\|$.\n","start":7236,"end":7452},{"raw_tex":"Now assume that \n\\begin{empheq}[box=\\mybluebox]{equation}\n\\text{$f \\colon X \\to\\left]-\\infty, +\\infty\\right]$ is convex and\ndifferentiable on $U := \\intdom f \\neq\\varnothing$.}\n\\end{empheq}\nThe \\emph{Bregman distance}\\footnote{Note that $D_f$ is\n\\emph{not} a distance in the sense of metric topology; however,\nthis naming convention is now ubiquitous.}  associated with $f$, \nfirst explored by Bregman in \\cite{Bre67} (see also\n\\cite{CZ97}), is \n\\begin{equation} \n\\label{e:Df}\nD_f \\colon X \\times X \\to \\left[0, +\\infty\\right] \\colon (x,y) \\mapsto\n\\begin{cases}\nf(x) -f(y) -\\scal{\\nabla f(y)}{x -y}, &\\text{~if~} y\\in U; \\\\\n+\\infty, &\\text{~otherwise}.\n\\end{cases}\n\\end{equation} \n","start":8646,"end":9327},{"raw_tex":"However, because different assignments of $f$ may allow for\ncases in which \n$D_f(x,y)\\neq D_f(y,x)$, we actually are led to consider\n\\emph{two} envelopes:\nthe \\emph{left} and \\emph{right} \n\\emph{Bregman--Moreau envelopes} are defined by\n\\begin{equation}\n\\label{e:bba}\n\\benv[\\gamma]{\\theta} \\colon X\\to \\left[-\\infty, +\\infty\\right] \\colon\ny\\mapsto \\inf_{x\\in X} \\big(\\theta(x) +\\frac{1}{\\gamma}D_f(x,y)\\big)\n\\end{equation}\nand \n\\begin{equation}\n\\label{e:fba}\n\\fenv[\\gamma]{\\theta} \\colon X \\to \\left[-\\infty, +\\infty\\right] \\colon\nx \\mapsto\n\\inf_{y\\in X} \\big(\\theta(y) +\\frac{1}{\\gamma}D_f(x,y)\\big),\n\\end{equation}\nrespectively. \n","start":10141,"end":10773},{"raw_tex":"\\begin{proposition}\n\\label{p:12.22i}\nLet $\\theta \\colon X \\to \\left]-\\infty, +\\infty\\right]$, \nlet $\\gamma \\in \\RPP$, and let $\\mu \\in \\RPP$.\nThen $\\benv[\\mu]{\\gamma\\theta} =\\gamma\\benv[\\gamma\\mu]{\\theta}$ \nand $\\fenv[\\mu]{\\gamma\\theta} =\\gamma\\fenv[\\gamma\\mu]{\\theta}$.\n\\end{proposition}","start":12824,"end":13112},{"raw_tex":"Denote by $\\Gamma_0(X)$ the set of all proper lower semicontinuous convex functions from $X$ to $\\left]-\\infty, +\\infty\\right]$. \nFrom now on, we strengthen our assumptions by requiring that \n\\begin{empheq}[box=\\mybluebox]{equation}\n\\label{e:fLegendre}\n\\text{$f\\in\\Gamma_0(X)$ is a convex function of Legendre type and\n$U := \\intdom f$.}\n\\end{empheq}\nThis will allow us to obtain a quite satisfying theory in which\nthe envelopes are convex functions. \nNote that $f$ is essentially smooth and essentially strictly convex in the sense of \\cite[Section~26]{Roc70}.\nIt is well known that\n\\begin{equation}\n\\label{e:Leggy}\n\\nabla f\\colon U\\to U^*:= \\intdom f^*\n\\text{~is a homeomorphism with~}\n\\big(\\nabla f\\big)^{-1} = \\nabla f^*.\n\\end{equation}\nWe will also work with the following standard assumptions:\n\\begin{itemize}\n\\item[\\bfseries A1] \n$\\nabla^2 f$ exists and is continuous on $U$;\n\\item[\\bfseries A2] \n$D_f$ is \\emph{jointly convex}, i.e., convex on $X\\times X$;\n\\item[\\bfseries A3] \n$(\\forall x\\in U)$ $D_f(x,\\cdot)$ is strictly convex on $U$;\n\\item[\\bfseries A4] \n$(\\forall x\\in U)$ $D_f(x,\\cdot)$ is coercive, i.e., \n$D_f(x, y) \\to +\\infty$ as $\\|y\\| \\to +\\infty$.\n\\end{itemize}\nWe also henceforth assume that \n\\begin{empheq}[box=\\mybluebox]{equation}\n\\text{all assumptions \\textbf{A1}--\\textbf{A4} hold.}\n\\end{empheq}\n","start":16624,"end":17948},{"raw_tex":"\\begin{fact} \n\\label{f:coeR}\nLet $\\theta\\in\\Gamma_0(X)$ be such that $\\dom\\theta\\cap U\\neq\\varnothing$ and let $\\gamma \\in \\RPP$.\nConsider the following properties:\n\\begin{itemize}\n\\item[\\rm (a)]\n$U\\cap \\dom\\theta$ is bounded.\n\\item[\\rm (b)]\n$\\inf\\theta(U)>-\\infty$.\n\\item[\\rm (c)]\n$f$ is supercoercive, i.e., \n$f(x)/\\|x\\| \\to +\\infty$ as $\\|x\\| \\to +\\infty$.\n\\item[\\rm (d)]\n$(\\forall x\\in U)$ $D_f(x,\\cdot)$ is supercoercive.\n\\end{itemize}\nThen the following hold:\n\\begin{enumerate}\n\\item \\label{l:coeRi}\nIf any of the conditions (a), (b), or (c) holds, then \n\\begin{equation} \\label{e:bcoeR}\n(\\forall y \\in U) \\quad \\theta(\\cdot) + \\frac{1}{\\gamma}D_f(\\cdot,y) \\;\\;\n\\text{is coercive}\n\\end{equation}\nor, equivalently,\n\\begin{equation}\n\\frac{1}{\\gamma}\\ran\\nabla f \\subseteq \\intdom\\left(\\frac{1}{\\gamma}f +\\theta\\right)^*.\n\\label{e:brilliantissimo}\n\\end{equation}\n\n\\item \\label{l:coeRii}\nIf any of the conditions (a), (b), or (d) holds, then \n\\begin{equation} \\label{e:fcoeR}\n(\\forall x \\in U) \\quad \\theta(\\cdot) + \\frac{1}{\\gamma}D_f(x,\\cdot) \\;\\;\n\\text{is coercive.}\n\\end{equation}\n\\end{enumerate}\n\\end{fact}","start":21812,"end":22925},{"raw_tex":"\\begin{proposition} \n\\label{p:exact} \nLet $\\theta \\colon X \\to \\left]-\\infty, +\\infty\\right]$ be convex\nand such that $U\\cap \\dom\\theta\\neq\\varnothing$, and let $\\gamma \\in \\RPP$.\nThen the following hold:\n\\begin{enumerate}\n\\item \n\\label{p:exact_left}\n$\\benv[\\gamma]{\\theta}$ is convex and continuous on $U$, and\n\t\\begin{enumerate}\n\t\\item \n\tif \\eqref{e:bcoeR} holds, i.e., $(\\forall y \\in U)$ $\\theta(\\cdot) +\\frac{1}{\\gamma}D_f(\\cdot,y)$ is coercive, \n\tthen $\\benv[\\gamma]{\\theta}$ is proper; \n\t\\item\n\tif $\\theta\\in\\Gamma_0(X)$ and $\\theta(\\cdot) +\\frac{1}{\\gamma}D_f(\\cdot,y)$ is coercive for a given $y\\in U$, \n\tthen there exists a unique point $z\\in U$ such that $\\benv[\\gamma]{\\theta}(y)=\\theta(z) +\\frac{1}{\\gamma}D_f(z,y)$.\n\t\\end{enumerate}\n\n\\item \n\\label{p:exact_right} \n$\\fenv[\\gamma]{\\theta}$ is convex and continuous on $U$, and\n\t\\begin{enumerate}\n\t\\item \n\tif \\eqref{e:fcoeR} holds, i.e., $(\\forall x \\in U)$ $\\theta(\\cdot) +\\frac{1}{\\gamma}D_f(x,\\cdot)$ is coercive, \n\tthen $\\fenv[\\gamma]{\\theta}$ is proper; \n\t\\item\n\tif $\\theta\\in\\Gamma_0(X)$ and $\\theta(\\cdot) +\\frac{1}{\\gamma}D_f(x,\\cdot)$ is coercive for a given $x\\in U$, \n\tthen there exists a unique point $z\\in U$ such that $\\fenv[\\gamma]{\\theta}(z)=\\theta(z) +\\frac{1}{\\gamma}D_f(x,z)$.\n\t\\end{enumerate}\n\\end{enumerate}\n\\end{proposition}","start":23247,"end":24554},{"raw_tex":"\\begin{definition}[Bregman proximity operators] \n\\label{d:prox}\nLet $\\theta\\in\\Gamma_0(X)$ be such that \n$U\\cap \\dom\\theta\\neq\\varnothing$.\nIf \\eqref{e:bcoeR} holds for $\\gamma =1$, then \nthe \\emph{left proximity operator} associated with $\\theta$ is\n\\begin{equation} \\label{e:dbprox}\n\\bprox{\\theta} \\colon U \\to U \\colon\ny\\mapsto\n\\underset{x\\in X}{\\operatorname{argmin}}\\;\\: \\big(\\theta(x) +D_f(x,y)\\big).\n\\end{equation}\nIf \\eqref{e:fcoeR} holds for $\\gamma =1$, then \nthe \\emph{right proximity operator} associated with $\\theta$ is\n\\begin{equation} \\label{e:dfprox}\n\\fprox{\\theta}\\colon U \\to  U \\colon\nx \\mapsto\n\\underset{y\\in X}{\\operatorname{argmin}}\\;\\: \\big(\\theta(y) +D_f(x,y)\\big).\n\\end{equation}\n\\end{definition}\n","start":24842,"end":25565},{"raw_tex":"Given a closed convex subset $C$ of $X$ with $C\\cap U \\neq\\varnothing$, we have that $\\iota_C \\in \\Gamma_0(X)$, $\\dom\\iota_C =C$, \nand hence $U\\cap \\dom\\iota_C =U\\cap C \\neq\\varnothing$ and also $\\inf \\iota_C(U) =0 >-\\infty$, \nwhich together with \\cref{f:coeR} imply that \\eqref{e:bcoeR} and \\eqref{e:fcoeR} hold for all $\\gamma \\in \\RPP$. \nThis leads to the following definition. \n\n\\begin{definition}[Bregman projectors]\n\\label{d:BregProj}\nLet $C$ be a closed convex subset of $X$ such that \n$U\\cap C\\neq\\varnothing$. \nThen $\\bproj{C} :=\\bprox{\\iota_C}$ is the \\emph{left Bregman projector} onto $C$\nand $\\fproj{C} :=\\fprox{\\iota_C}$ is the \\emph{right Bregman projector} onto $C$.\n\\end{definition}\n","start":26129,"end":26829},{"raw_tex":"\\begin{remark}\n\\label{r:prox}\nLet $\\theta\\in\\Gamma_0(X)$ be such that \n$U\\cap \\dom\\theta\\neq\\varnothing$ and let $\\gamma \\in \\RPP$.\n\\cref{p:12.22i} implies that \n$\\benv{\\gamma\\theta} =\\gamma\\benv[\\gamma]{\\theta}$ \nand $\\fenv{\\gamma\\theta} =\\gamma\\fenv[\\gamma]{\\theta}$. \nWe thus derive from the definition that if \\eqref{e:bcoeR} holds, then\n\\begin{subequations}\n\\label{e:benv-bproxall}\n\\begin{equation}\n\\label{e:benv-bprox}\n\\benv[\\gamma]{\\theta}(y) =\\theta\\big(\\bprox{\\gamma\\theta}(y)\\big) \n+\\frac{1}{\\gamma}D_f\\big(\\bprox{\\gamma\\theta}(y), y\\big)\n\\end{equation}\nand, by combining with \\cref{p:12.9}\\ref{p:12.9_left},\n\\begin{equation}\n\\label{e:benv-bprox'}\n\\theta\\big(\\bprox{\\gamma\\theta}(y)\\big) \\leq \\benv[\\gamma]{\\theta}(y) \\leq \\theta(y).\n\\end{equation}\n\\end{subequations}\nSimilarly, if \\eqref{e:fcoeR} holds, then\n\\begin{subequations}\n\\label{e:fenv-fproxall}\n\\begin{equation}\n\\label{e:fenv-fprox}\n\\fenv[\\gamma]{\\theta}(x) =\\theta\\big(\\fprox{\\gamma\\theta}(x)\\big) +\n\\frac{1}{\\gamma}D_f\\big(x, \\fprox{\\gamma\\theta}(x)\\big)\n\\end{equation}\nand \n\\begin{equation}\n\\theta\\big(\\fprox{\\gamma\\theta}(x)\\big) \\leq \\fenv[\\gamma]{\\theta}(x) \\leq \\theta(x).\n\\end{equation}\n\\end{subequations}\n\\end{remark}","start":29283,"end":30479},{"raw_tex":"\\begin{proposition}\n\\label{p:prox}\nLet $\\theta\\in\\Gamma_0(X)$ be such that $\\dom\\theta\\cap U\\neq\\varnothing$ and let $\\gamma \\in \\RPP$.\n\\begin{enumerate}\n\\item \n\\label{p:prox_left}\nSuppose that \\eqref{e:bcoeR} holds.\nThen for every $(x,y) \\in U\\times U$, \nthe following conditions are equivalent:\n\t\\begin{enumerate}\n\t\\item \n\t$x =\\bprox{\\gamma\\theta}(y)$,\n\t\\item \n\t$0 \\in \\gamma\\partial\\theta(x) +\\nabla f(x) -\\nabla f(y)$,\n\t\\item \n\t$(\\forall z \\in X)\\quad \\scal{\\nabla f(y) -\\nabla f(x)}{z -x} +\\gamma\\theta(x) \\leq \\gamma\\theta(z)$.\n\t\\end{enumerate}\nMoreover, \n\\begin{equation} \\label{e:pro}\n\\bprox{\\gamma\\theta}=(\\nabla f +\\gamma\\partial\\theta)^{-1}\\circ \\nabla f \n\\end{equation}\nis continuous on $U$. \n\\item \n\\label{p:prox_right}\nSuppose that \\eqref{e:fcoeR} holds. \nThen for every $(x,y) \\in U\\times U$, \nthe following conditions are equivalent:\n\t\\begin{enumerate}\n\t\\item \n\t$y =\\fprox{\\gamma\\theta}(x)$,\n\t\\item \n\t$0 \\in \\gamma\\partial\\theta(y) +\\nabla^2 f(y)(y -x)$,\n\t\\item \n\t$(\\forall z \\in X)\\quad \\scal{\\nabla^2 f(y)(x -y)}{z -y} +\\gamma\\theta(y) \\leq \\gamma\\theta(z)$.\n\t\\end{enumerate}\nMoreover, $\\fprox{\\gamma\\theta}$ is continuous on $U$. \n\\end{enumerate}\n\\end{proposition}","start":31727,"end":32910},{"raw_tex":"\\begin{lemma}\n\\label{l:cvg}\nLet $C$ be a compact subset of a Hausdorff space $\\mathcal X$,\nlet $\\phi\\colon \\mathcal X \\to \\left[-\\infty, +\\infty\\right]$ be lower \nsemicontinuous,\nlet $(x_a)_{a\\in A}$ be a net in $C$,\nand suppose that $\\phi(x_a) \\to \\inf\\phi(\\mathcal X)$. \nThen $\\argmin\\phi \\neq\\varnothing$ \nand all cluster points of $(x_a)_{a\\in A}$ lie in $\\argmin\\phi$.\nConsequently, if $\\phi$ attains its minimum at a unique point $u$, then $x_a \\to u$.\n\\end{lemma}","start":40270,"end":40740}]},"statement":{"raw_tex":"\\begin{theorem}\n\\label{p:limprox+}\nLet $\\theta\\in\\Gamma_0(X)$ be coercive such that \n$U\\cap \\dom\\theta\\neq\\varnothing$ and let $x, y \\in U$.\nThen the following hold:\n\\begin{enumerate}\n\\item \n\\label{p:limprox+_left}\nThe net $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ is bounded \nwith all cluster points as $\\gamma \\uparrow +\\infty$ lying in $\\argmin\\theta$. \nMoreover,\n\t\\begin{enumerate}\n\t\\item\\label{p:limprox+_leftA} \n\tif $\\argmin\\theta$ is a singleton, \n\tthen $\\bprox{\\gamma\\theta}(y) \\to \\argmin\\theta$ as $\\gamma \\uparrow +\\infty$;\n\t\\item\\label{p:limprox+_leftP} \n\tif $\\argmin\\theta \\subseteq U$, \n\tthen $\\bprox{\\gamma\\theta}(y) \\to \\bproj{\\argmin\\theta}y$ as $\\gamma \\uparrow +\\infty$. \n\t\\end{enumerate}\n\n\\item \n\\label{p:limprox+_right}\nThe net $(\\fprox{\\gamma\\theta}(x))_{\\gamma \\in \\RPP}$ is bounded \nwith all cluster points as $\\gamma \\uparrow +\\infty$ lying in $\\argmin\\theta$. \nMoreover,\n\t\\begin{enumerate}\n\t\\item \n\tif $\\argmin\\theta$ is a singleton, \n\tthen $\\fprox{\\gamma\\theta}(x) \\to \\argmin\\theta$ as $\\gamma \\uparrow +\\infty$;\n\t\\item\\label{p:limprox+_rightP} \n\tif $\\argmin\\theta \\subseteq U$, \n\tthen $\\fprox{\\gamma\\theta}(x) \\to \\fproj{\\argmin\\theta}x$ as $\\gamma \\uparrow +\\infty$.  \n\t\\end{enumerate}\n\\end{enumerate}\t\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\n\\label{p:limprox+}\nLet $\\theta\\in\\Gamma_0(X)$ be coercive such that \n$U\\cap \\dom\\theta\\neq\\varnothing$ and let $x, y \\in U$.\nThen the following hold:\n\\begin{enumerate}\n\\item \n\\label{p:limprox+_left}\nThe net $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ is bounded \nwith all cluster points as $\\gamma \\uparrow +\\infty$ lying in $\\argmin\\theta$. \nMoreover,\n\t\\begin{enumerate}\n\t\\item\\label{p:limprox+_leftA} \n\tif $\\argmin\\theta$ is a singleton, \n\tthen $\\bprox{\\gamma\\theta}(y) \\to \\argmin\\theta$ as $\\gamma \\uparrow +\\infty$;\n\t\\item\\label{p:limprox+_leftP} \n\tif $\\argmin\\theta \\subseteq U$, \n\tthen $\\bprox{\\gamma\\theta}(y) \\to \\bproj{\\argmin\\theta}y$ as $\\gamma \\uparrow +\\infty$. \n\t\\end{enumerate}\n\n\\item \n\\label{p:limprox+_right}\nThe net $(\\fprox{\\gamma\\theta}(x))_{\\gamma \\in \\RPP}$ is bounded \nwith all cluster points as $\\gamma \\uparrow +\\infty$ lying in $\\argmin\\theta$. \nMoreover,\n\t\\begin{enumerate}\n\t\\item \n\tif $\\argmin\\theta$ is a singleton, \n\tthen $\\fprox{\\gamma\\theta}(x) \\to \\argmin\\theta$ as $\\gamma \\uparrow +\\infty$;\n\t\\item\\label{p:limprox+_rightP} \n\tif $\\argmin\\theta \\subseteq U$, \n\tthen $\\fprox{\\gamma\\theta}(x) \\to \\fproj{\\argmin\\theta}x$ as $\\gamma \\uparrow +\\infty$.  \n\t\\end{enumerate}\n\\end{enumerate}\t\n\\end{theorem}","start":47939,"end":49189}]},"proof":{"raw_tex":"\\begin{proposition} \n\\label{p:12.9} \nLet $\\theta \\colon X \\to \\left]-\\infty, +\\infty\\right]$ be such that $U\\cap \\dom\\theta\\neq\\varnothing$ and let $\\gamma \\in \\RPP$.\nThen the following hold:\n\\begin{enumerate}\n\\item \n\\label{p:12.9_left}\n$\\dom\\benv[\\gamma]{\\theta}=U$, \nand $(\\forall y\\in U)(\\forall\\mu \\in \\left]\\gamma, +\\infty\\right[)$ \n$\\inf \\theta(X) \\leq \\benv[\\mu]{\\theta}(y) \\leq \\benv[\\gamma]{\\theta}(y) \\leq \\theta(y)$.\nConsequently, $\\inf \\theta(X) \\leq \\inf \\benv[\\gamma]{\\theta}(X) \\leq \\inf \\theta(U)$, \nand $\\benv[\\gamma]{\\theta}(y) \\downarrow \\inf \\theta(X)$ as $\\gamma \\uparrow +\\infty$.\n\\item \n\\label{p:12.9_right}\n$\\dom\\fenv[\\gamma]{\\theta}=\\dom f$, \nand $(\\forall x \\in U)(\\forall\\mu \\in \\left]\\gamma, +\\infty\\right[)$ \n$\\inf \\theta(X) \\leq \\fenv[\\mu]{\\theta}(x) \\leq \\fenv[\\gamma]{\\theta}(x) \\leq \\theta(x)$.\nConsequently, $\\inf \\theta(X) \\leq \\inf \\fenv[\\gamma]{\\theta}(X) \\leq \\inf \\theta(U)$, \nand $\\fenv[\\gamma]{\\theta}(x) \\downarrow \\inf \\theta(X)$ as $\\gamma \\uparrow +\\infty$.\n\\end{enumerate}\n\\end{proposition}\n\n\\begin{proof}\n\\ref{p:12.9_left}: We first show that $\\dom\\benv[\\gamma]{\\theta}=U$.\nLet $y \\in \\dom\\benv[\\gamma]{\\theta}$. \nThen $\\benv[\\gamma]{\\theta}(y) =\\inf_{x\\in X} \\big(\\theta(x) +\\frac{1}{\\gamma}D_f(x,y)\\big) <+\\infty$,\nand hence there exists $x \\in X$ such that $\\theta(x) +\\frac{1}{\\gamma}D_f(x,y) <+\\infty$.\nSince $\\theta(x) >-\\infty$, this yields $y \\in U$.\n\nFrom now on, let $y \\in U$, and pick $u \\in \\dom\\theta \\cap U$. \nThen $-f(y) <+\\infty$, $\\|\\nabla f(y)\\| <+\\infty$, $f(u) <+\\infty$, $\\theta(u) <+\\infty$, and\n\\begin{subequations}\n\\begin{align}\n\\benv[\\gamma]{\\theta}(y) &=\\inf_{x\\in X} \\Big(\\theta(x) +\\frac{1}{\\gamma}\\big(f(x) -f(y) -\\scal{\\nabla f(y)}{x -y}\\big)\\Big), \\\\ \n&\\leq \\theta(u) +\\frac{1}{\\gamma}\\big(f(u) -f(y) -\\scal{\\nabla f(y)}{u -y}\\big) <+\\infty,\n\\end{align}\n\\end{subequations} \nwhich gives $y \\in \\dom\\benv[\\gamma]{\\theta}$. Hence, $\\dom \\benv[\\gamma]{\\theta} =U$. \n\nNext, let $\\mu \\in \\left]\\gamma, +\\infty\\right[$. \nThen $\\frac{1}{\\mu} < \\frac{1}{\\gamma}$, \n$\\theta \\leq \\theta +\\frac{1}{\\mu}D_f(\\cdot,y) \\leq \\theta +\\frac{1}{\\gamma}D_f(\\cdot,y)$, and so\n\\begin{equation}\n\\inf_{x \\in X} \\theta(x) \\leq \\benv[\\mu]{\\theta}(y) \\leq \\benv[\\gamma]{\\theta}(y) =\\inf_{x \\in X} \\big(\\theta(x) +\\frac{1}{\\gamma}D_f(x,y)\\big)\n\\leq \\theta(y) +\\frac{1}{\\gamma}D_f(y,y) =\\theta(y). \n\\end{equation}\nTherefore,\n\\begin{equation}\n\\label{e:170428a}\n\\inf \\theta(X) \\leq \\benv[\\mu]{\\theta}(y) \\leq \\benv[\\gamma]{\\theta}(y) \\leq \\theta(y).\n\\end{equation}\nTaking now the infimum over $y\\in U$ yields\n$\\inf \\theta(X) \\leq \\inf \\benv[\\gamma]{\\theta}(X) \\leq \\inf \\theta(U)$.\nConsequently, \n\\begin{equation}\n\\inf \\theta(X) \\leq \\varliminf_{\\gamma \\to+\\infty}\n\\benv[\\gamma]{\\theta}(y).\n\\end{equation}\nOn the other hand, $(\\forall x \\in X)$ $\\benv[\\gamma]{\\theta}(y) \\leq \\theta(x) +\\frac{1}{\\gamma}D_f(x,y)$, \nwhich implies that $(\\forall x \\in X)$ $\\varlimsup_{\\gamma\n\\to+\\infty} \\benv[\\gamma]{\\theta}(y) \\leq \\theta(x)$ and thus\n$\\varlimsup_{\\gamma \\to+\\infty} \\benv[\\gamma]{\\theta}(y) \\leq\n\\inf \\theta(X)$. \nAltogether,\n$\\lim_{\\gamma\\to+\\infty} \\benv[\\gamma]{\\theta}(y) =\n\\inf \\theta(X)$ and the conclusion follows from\n\\eqref{e:170428a}. \n\n\\ref{p:12.9_right}: This is similar to \\ref{p:12.9_left}. \n\\end{proof}\n\n\\begin{corollary}\n\\label{c:BregProj}\nLet $C$ be a closed convex subset of $X$ such that \n$U\\cap C\\neq\\varnothing$, \nlet $(x,y) \\in U\\times U$, \nand let $p \\in U\\cap C$.\nThen the following hold:\n\\begin{enumerate}\n\\item \n\\label{c:BregProj_left}\n$p =\\bproj{C}y \\ \\Leftrightarrow \\ (\\forall z \\in C)\\; \n\\scal{\\nabla f(y) -\\nabla f(p)}{z -p} \\leq 0$. \n\\item \n\\label{c:BregProj_right}\n$p =\\fproj{C}x \\ \\Leftrightarrow \\ (\\forall z \\in C)\\; \n\\scal{\\nabla^2 f(p)(x -p)}{z -p} \\leq 0$. \n\\end{enumerate}\n\\end{corollary}\n\n\\begin{proof}\nIn light of \\cref{d:BregProj}, we \napply \\cref{p:prox} \n(see also \\cite[Proposition~3.16]{BB97}).\n\\end{proof}\n\n\\begin{proposition}\n\\label{p:limthetaprox}\nLet $\\theta\\in\\Gamma_0(X)$ be such that $U\\cap \\dom\\theta\\neq\\varnothing$ and let $x, y \\in U$.\nThen the following hold:\n\\begin{enumerate}\n\\item \n\\label{p:limthetaprox_left}\nIf \\eqref{e:bcoeR} holds for all $\\gamma\\in\\RPP$, \nthen $\\theta(\\bprox{\\gamma\\theta}(y)) \\to \\inf \\theta(X)$ as $\\gamma \\uparrow +\\infty$.\n\\item \n\\label{p:limthetaprox_right}\nIf \\eqref{e:fcoeR} holds for all $\\gamma\\in \\RPP$, then  \n$\\theta(\\fprox{\\gamma\\theta}(x)) \\to \\inf \\theta(X)$ as $\\gamma \\uparrow +\\infty$.\n\\end{enumerate}\t\n\\end{proposition}\n\n\\begin{proof}\nWe shall just prove \\ref{p:limthetaprox_left} \nbecause the proof of \\ref{p:limthetaprox_right} is similar.\nAssume that \\eqref{e:bcoeR} holds for all \n$\\gamma \\in \\RPP$.\nCombining \\eqref{e:benv-bprox'} with \\cref{p:12.9}\\ref{p:12.9_left} yields\n\\begin{equation}\n\\inf\\theta(X) \\leq \\theta\\big(\\bprox{\\gamma\\theta}(y)\\big) \\leq \\benv[\\gamma]{\\theta}(y) \\to \\inf\\theta(X) \n\\quad\\text{as}\\quad \\gamma \\uparrow +\\infty,\n\\end{equation} \nwhich implies that $\\theta(\\bprox{\\gamma\\theta}(y)) \\to \n\\inf\\theta(X)$ as $\\gamma \\uparrow +\\infty$. \n\\end{proof}\n\n\\begin{proof}\nFirst, by assumption, \\cite[Proposition~11.15(i)]{BC17} gives $\\argmin\\theta \\neq\\varnothing$. \nThis combined with \\cite[Lemma~1.24 and Corollary~8.5]{BC17} implies that \n$\\argmin\\theta =\\lev{\\inf\\theta(X)}\\theta$ is a nonempty closed convex subset of $X$.\nNow since $\\theta$ is coercive and since $D_f\\geq 0$, we immediately get that  \n\\eqref{e:bcoeR} and \\eqref{e:fcoeR} hold for all $\\gamma \\in \\RPP$.\n\n\\ref{p:limprox+_left}: It follows from \\eqref{e:benv-bprox'} that   \n\\begin{equation}\n\\label{e:bounded}\n(\\forall \\gamma \\in \\RPP)\\quad \\bprox{\\gamma\\theta}(y) \\in \\lev{\\theta(y)}\\theta,\n\\end{equation}\nand then\nfrom the coercivity of $\\theta$ and \\cite[Proposition~11.12]{BC17} that $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ is bounded.\nIn turn, \\cref{p:limthetaprox}\\ref{p:limthetaprox_left} and \\cref{l:cvg} imply that\nall cluster points of $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ as $\\gamma \\uparrow +\\infty$ lie in $\\argmin\\theta$, \nand we get \\ref{p:limprox+_leftA}.\n\nNow assume that $\\argmin\\theta \\subseteq U$. \nLet $y'$ be a cluster point of $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ as $\\gamma \\uparrow +\\infty$.  \nThen $y' \\in \\argmin\\theta \\subseteq U$ and there exists a sequence $(\\gamma_n)_\\nnn$ in $\\RPP$ such that \n$\\gamma_n \\uparrow +\\infty$ and $\\bprox{\\gamma_n\\theta}(y) \\to y'$ as $n \\to +\\infty$.\nLet $z \\in \\argmin\\theta$.  \nWe have $(\\forall\\nnn)$ $\\theta(z) \\leq \\theta(\\bprox{\\gamma_n\\theta}(y))$, \nand by Proposition~\\ref{p:prox}\\ref{p:prox_left},\n\\begin{equation}\n\\scal{\\nabla f(y) -\\nabla f\\big(\\bprox{\\gamma_n\\theta}(y)\\big)}{z -\\bprox{\\gamma_n\\theta}(y)} \n\\leq \\gamma_n\\Big(\\theta(z)\n-\\theta\\big(\\bprox{\\gamma_n\\theta}(y)\\big)\\Big) \\leq 0.\n\\end{equation}\nTaking the limit as $n \\to +\\infty$ and using the continuity of $\\nabla f$ yield\n\\begin{equation}\n\\scal{\\nabla f(y) -\\nabla f(y')}{z -y'} \\leq 0.\n\\end{equation}\nSince $z \\in \\argmin\\theta$ was chosen arbitrarily \nand since $\\argmin\\theta$ is a closed convex subset of $X$ with \n$U\\cap \\argmin\\theta \\neq\\varnothing$, \nin view of \\cref{c:BregProj}\\ref{c:BregProj_left}, $y' =\\bproj{\\argmin\\theta}y$, \nand $\\bproj{\\argmin\\theta}y$ is thus the only cluster point of $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ \nas $\\gamma \\uparrow +\\infty$.\nHence, \\ref{p:limprox+_leftP} holds.\n\n\\ref{p:limprox+_right}: \nThe proof is similar to the one of \\ref{p:limprox+_left}.\n\\end{proof}","units":[{"raw_tex":"\\begin{proposition} \n\\label{p:12.9} \nLet $\\theta \\colon X \\to \\left]-\\infty, +\\infty\\right]$ be such that $U\\cap \\dom\\theta\\neq\\varnothing$ and let $\\gamma \\in \\RPP$.\nThen the following hold:\n\\begin{enumerate}\n\\item \n\\label{p:12.9_left}\n$\\dom\\benv[\\gamma]{\\theta}=U$, \nand $(\\forall y\\in U)(\\forall\\mu \\in \\left]\\gamma, +\\infty\\right[)$ \n$\\inf \\theta(X) \\leq \\benv[\\mu]{\\theta}(y) \\leq \\benv[\\gamma]{\\theta}(y) \\leq \\theta(y)$.\nConsequently, $\\inf \\theta(X) \\leq \\inf \\benv[\\gamma]{\\theta}(X) \\leq \\inf \\theta(U)$, \nand $\\benv[\\gamma]{\\theta}(y) \\downarrow \\inf \\theta(X)$ as $\\gamma \\uparrow +\\infty$.\n\\item \n\\label{p:12.9_right}\n$\\dom\\fenv[\\gamma]{\\theta}=\\dom f$, \nand $(\\forall x \\in U)(\\forall\\mu \\in \\left]\\gamma, +\\infty\\right[)$ \n$\\inf \\theta(X) \\leq \\fenv[\\mu]{\\theta}(x) \\leq \\fenv[\\gamma]{\\theta}(x) \\leq \\theta(x)$.\nConsequently, $\\inf \\theta(X) \\leq \\inf \\fenv[\\gamma]{\\theta}(X) \\leq \\inf \\theta(U)$, \nand $\\fenv[\\gamma]{\\theta}(x) \\downarrow \\inf \\theta(X)$ as $\\gamma \\uparrow +\\infty$.\n\\end{enumerate}\n\\end{proposition}","start":13355,"end":14391},{"raw_tex":"\\begin{proof}\n\\ref{p:12.9_left}: We first show that $\\dom\\benv[\\gamma]{\\theta}=U$.\nLet $y \\in \\dom\\benv[\\gamma]{\\theta}$. \nThen $\\benv[\\gamma]{\\theta}(y) =\\inf_{x\\in X} \\big(\\theta(x) +\\frac{1}{\\gamma}D_f(x,y)\\big) <+\\infty$,\nand hence there exists $x \\in X$ such that $\\theta(x) +\\frac{1}{\\gamma}D_f(x,y) <+\\infty$.\nSince $\\theta(x) >-\\infty$, this yields $y \\in U$.\n\nFrom now on, let $y \\in U$, and pick $u \\in \\dom\\theta \\cap U$. \nThen $-f(y) <+\\infty$, $\\|\\nabla f(y)\\| <+\\infty$, $f(u) <+\\infty$, $\\theta(u) <+\\infty$, and\n\\begin{subequations}\n\\begin{align}\n\\benv[\\gamma]{\\theta}(y) &=\\inf_{x\\in X} \\Big(\\theta(x) +\\frac{1}{\\gamma}\\big(f(x) -f(y) -\\scal{\\nabla f(y)}{x -y}\\big)\\Big), \\\\ \n&\\leq \\theta(u) +\\frac{1}{\\gamma}\\big(f(u) -f(y) -\\scal{\\nabla f(y)}{u -y}\\big) <+\\infty,\n\\end{align}\n\\end{subequations} \nwhich gives $y \\in \\dom\\benv[\\gamma]{\\theta}$. Hence, $\\dom \\benv[\\gamma]{\\theta} =U$. \n\nNext, let $\\mu \\in \\left]\\gamma, +\\infty\\right[$. \nThen $\\frac{1}{\\mu} < \\frac{1}{\\gamma}$, \n$\\theta \\leq \\theta +\\frac{1}{\\mu}D_f(\\cdot,y) \\leq \\theta +\\frac{1}{\\gamma}D_f(\\cdot,y)$, and so\n\\begin{equation}\n\\inf_{x \\in X} \\theta(x) \\leq \\benv[\\mu]{\\theta}(y) \\leq \\benv[\\gamma]{\\theta}(y) =\\inf_{x \\in X} \\big(\\theta(x) +\\frac{1}{\\gamma}D_f(x,y)\\big)\n\\leq \\theta(y) +\\frac{1}{\\gamma}D_f(y,y) =\\theta(y). \n\\end{equation}\nTherefore,\n\\begin{equation}\n\\label{e:170428a}\n\\inf \\theta(X) \\leq \\benv[\\mu]{\\theta}(y) \\leq \\benv[\\gamma]{\\theta}(y) \\leq \\theta(y).\n\\end{equation}\nTaking now the infimum over $y\\in U$ yields\n$\\inf \\theta(X) \\leq \\inf \\benv[\\gamma]{\\theta}(X) \\leq \\inf \\theta(U)$.\nConsequently, \n\\begin{equation}\n\\inf \\theta(X) \\leq \\varliminf_{\\gamma \\to+\\infty}\n\\benv[\\gamma]{\\theta}(y).\n\\end{equation}\nOn the other hand, $(\\forall x \\in X)$ $\\benv[\\gamma]{\\theta}(y) \\leq \\theta(x) +\\frac{1}{\\gamma}D_f(x,y)$, \nwhich implies that $(\\forall x \\in X)$ $\\varlimsup_{\\gamma\n\\to+\\infty} \\benv[\\gamma]{\\theta}(y) \\leq \\theta(x)$ and thus\n$\\varlimsup_{\\gamma \\to+\\infty} \\benv[\\gamma]{\\theta}(y) \\leq\n\\inf \\theta(X)$. \nAltogether,\n$\\lim_{\\gamma\\to+\\infty} \\benv[\\gamma]{\\theta}(y) =\n\\inf \\theta(X)$ and the conclusion follows from\n\\eqref{e:170428a}. \n\n\\ref{p:12.9_right}: This is similar to \\ref{p:12.9_left}. \n\\end{proof}","start":14392,"end":16620},{"raw_tex":"\\begin{corollary}\n\\label{c:BregProj}\nLet $C$ be a closed convex subset of $X$ such that \n$U\\cap C\\neq\\varnothing$, \nlet $(x,y) \\in U\\times U$, \nand let $p \\in U\\cap C$.\nThen the following hold:\n\\begin{enumerate}\n\\item \n\\label{c:BregProj_left}\n$p =\\bproj{C}y \\ \\Leftrightarrow \\ (\\forall z \\in C)\\; \n\\scal{\\nabla f(y) -\\nabla f(p)}{z -p} \\leq 0$. \n\\item \n\\label{c:BregProj_right}\n$p =\\fproj{C}x \\ \\Leftrightarrow \\ (\\forall z \\in C)\\; \n\\scal{\\nabla^2 f(p)(x -p)}{z -p} \\leq 0$. \n\\end{enumerate}\n\\end{corollary}","start":34473,"end":34982},{"raw_tex":"\\begin{proof}\nIn light of \\cref{d:BregProj}, we \napply \\cref{p:prox} \n(see also \\cite[Proposition~3.16]{BB97}).\n\\end{proof}","start":34983,"end":35106},{"raw_tex":"\\begin{proposition}\n\\label{p:limthetaprox}\nLet $\\theta\\in\\Gamma_0(X)$ be such that $U\\cap \\dom\\theta\\neq\\varnothing$ and let $x, y \\in U$.\nThen the following hold:\n\\begin{enumerate}\n\\item \n\\label{p:limthetaprox_left}\nIf \\eqref{e:bcoeR} holds for all $\\gamma\\in\\RPP$, \nthen $\\theta(\\bprox{\\gamma\\theta}(y)) \\to \\inf \\theta(X)$ as $\\gamma \\uparrow +\\infty$.\n\\item \n\\label{p:limthetaprox_right}\nIf \\eqref{e:fcoeR} holds for all $\\gamma\\in \\RPP$, then  \n$\\theta(\\fprox{\\gamma\\theta}(x)) \\to \\inf \\theta(X)$ as $\\gamma \\uparrow +\\infty$.\n\\end{enumerate}\t\n\\end{proposition}","start":46810,"end":47377},{"raw_tex":"\\begin{proof}\nWe shall just prove \\ref{p:limthetaprox_left} \nbecause the proof of \\ref{p:limthetaprox_right} is similar.\nAssume that \\eqref{e:bcoeR} holds for all \n$\\gamma \\in \\RPP$.\nCombining \\eqref{e:benv-bprox'} with \\cref{p:12.9}\\ref{p:12.9_left} yields\n\\begin{equation}\n\\inf\\theta(X) \\leq \\theta\\big(\\bprox{\\gamma\\theta}(y)\\big) \\leq \\benv[\\gamma]{\\theta}(y) \\to \\inf\\theta(X) \n\\quad\\text{as}\\quad \\gamma \\uparrow +\\infty,\n\\end{equation} \nwhich implies that $\\theta(\\bprox{\\gamma\\theta}(y)) \\to \n\\inf\\theta(X)$ as $\\gamma \\uparrow +\\infty$. \n\\end{proof}","start":47378,"end":47936},{"raw_tex":"\\begin{proof}\nFirst, by assumption, \\cite[Proposition~11.15(i)]{BC17} gives $\\argmin\\theta \\neq\\varnothing$. \nThis combined with \\cite[Lemma~1.24 and Corollary~8.5]{BC17} implies that \n$\\argmin\\theta =\\lev{\\inf\\theta(X)}\\theta$ is a nonempty closed convex subset of $X$.\nNow since $\\theta$ is coercive and since $D_f\\geq 0$, we immediately get that  \n\\eqref{e:bcoeR} and \\eqref{e:fcoeR} hold for all $\\gamma \\in \\RPP$.\n\n\\ref{p:limprox+_left}: It follows from \\eqref{e:benv-bprox'} that   \n\\begin{equation}\n\\label{e:bounded}\n(\\forall \\gamma \\in \\RPP)\\quad \\bprox{\\gamma\\theta}(y) \\in \\lev{\\theta(y)}\\theta,\n\\end{equation}\nand then\nfrom the coercivity of $\\theta$ and \\cite[Proposition~11.12]{BC17} that $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ is bounded.\nIn turn, \\cref{p:limthetaprox}\\ref{p:limthetaprox_left} and \\cref{l:cvg} imply that\nall cluster points of $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ as $\\gamma \\uparrow +\\infty$ lie in $\\argmin\\theta$, \nand we get \\ref{p:limprox+_leftA}.\n\nNow assume that $\\argmin\\theta \\subseteq U$. \nLet $y'$ be a cluster point of $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ as $\\gamma \\uparrow +\\infty$.  \nThen $y' \\in \\argmin\\theta \\subseteq U$ and there exists a sequence $(\\gamma_n)_\\nnn$ in $\\RPP$ such that \n$\\gamma_n \\uparrow +\\infty$ and $\\bprox{\\gamma_n\\theta}(y) \\to y'$ as $n \\to +\\infty$.\nLet $z \\in \\argmin\\theta$.  \nWe have $(\\forall\\nnn)$ $\\theta(z) \\leq \\theta(\\bprox{\\gamma_n\\theta}(y))$, \nand by Proposition~\\ref{p:prox}\\ref{p:prox_left},\n\\begin{equation}\n\\scal{\\nabla f(y) -\\nabla f\\big(\\bprox{\\gamma_n\\theta}(y)\\big)}{z -\\bprox{\\gamma_n\\theta}(y)} \n\\leq \\gamma_n\\Big(\\theta(z)\n-\\theta\\big(\\bprox{\\gamma_n\\theta}(y)\\big)\\Big) \\leq 0.\n\\end{equation}\nTaking the limit as $n \\to +\\infty$ and using the continuity of $\\nabla f$ yield\n\\begin{equation}\n\\scal{\\nabla f(y) -\\nabla f(y')}{z -y'} \\leq 0.\n\\end{equation}\nSince $z \\in \\argmin\\theta$ was chosen arbitrarily \nand since $\\argmin\\theta$ is a closed convex subset of $X$ with \n$U\\cap \\argmin\\theta \\neq\\varnothing$, \nin view of \\cref{c:BregProj}\\ref{c:BregProj_left}, $y' =\\bproj{\\argmin\\theta}y$, \nand $\\bproj{\\argmin\\theta}y$ is thus the only cluster point of $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ \nas $\\gamma \\uparrow +\\infty$.\nHence, \\ref{p:limprox+_leftP} holds.\n\n\\ref{p:limprox+_right}: \nThe proof is similar to the one of \\ref{p:limprox+_left}.\n\\end{proof}","start":49190,"end":51580}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"有限维欧氏空间","source_units":[{"raw_tex":"We assume throughout that\n\\begin{empheq}[box=\\mybluebox]{equation}\n\\text{$X := \\RR^J$,}\n\\end{empheq}\nwhich we equip with \nthe standard inner product $\\scal{\\cdot}{\\cdot}$ and \nthe induced Euclidean norm $\\|\\cdot\\|$.\n","start":7236,"end":7452}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"生成函数的初始条件","source_units":[{"raw_tex":"Now assume that \n\\begin{empheq}[box=\\mybluebox]{equation}\n\\text{$f \\colon X \\to\\left]-\\infty, +\\infty\\right]$ is convex and\ndifferentiable on $U := \\intdom f \\neq\\varnothing$.}\n\\end{empheq}\n","start":8646,"end":8836}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":["e:Df"],"declared_external":false,"source_only":false,"title":"Bregman 距离","source_units":[{"raw_tex":"The \\emph{Bregman distance}\\footnote{Note that $D_f$ is\n\\emph{not} a distance in the sense of metric topology; however,\nthis naming convention is now ubiquitous.}  associated with $f$, \nfirst explored by Bregman in \\cite{Bre67} (see also\n\\cite{CZ97}), is \n\\begin{equation} \n\\label{e:Df}\nD_f \\colon X \\times X \\to \\left[0, +\\infty\\right] \\colon (x,y) \\mapsto\n\\begin{cases}\nf(x) -f(y) -\\scal{\\nabla f(y)}{x -y}, &\\text{~if~} y\\in U; \\\\\n+\\infty, &\\text{~otherwise}.\n\\end{cases}\n\\end{equation} \n","start":8836,"end":9327}]},{"id":"S004","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"左右方向的动机","source_units":[{"raw_tex":"However, because different assignments of $f$ may allow for\ncases in which \n$D_f(x,y)\\neq D_f(y,x)$, we actually are led to consider\n\\emph{two} envelopes:\nthe \\emph{left} and \\emph{right} \n\\emph{Bregman--Moreau envelopes} are defined by\n","start":10141,"end":10378}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":["e:bba"],"declared_external":false,"source_only":false,"title":"左 Bregman 包络","source_units":[{"raw_tex":"\\begin{equation}\n\\label{e:bba}\n\\benv[\\gamma]{\\theta} \\colon X\\to \\left[-\\infty, +\\infty\\right] \\colon\ny\\mapsto \\inf_{x\\in X} \\big(\\theta(x) +\\frac{1}{\\gamma}D_f(x,y)\\big)\n\\end{equation}\nand \n","start":10378,"end":10569}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["e:fba"],"declared_external":false,"source_only":false,"title":"右 Bregman 包络","source_units":[{"raw_tex":"\\begin{equation}\n\\label{e:fba}\n\\fenv[\\gamma]{\\theta} \\colon X \\to \\left[-\\infty, +\\infty\\right] \\colon\nx \\mapsto\n\\inf_{y\\in X} \\big(\\theta(y) +\\frac{1}{\\gamma}D_f(x,y)\\big),\n\\end{equation}\nrespectively. \n","start":10569,"end":10773}]},{"id":"S007","semantic_type":"proposition","kind":"proposition","labels":["p:12.22i"],"declared_external":false,"source_only":false,"title":"包络的正数缩放","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{p:12.22i}\nLet $\\theta \\colon X \\to \\left]-\\infty, +\\infty\\right]$, \nlet $\\gamma \\in \\RPP$, and let $\\mu \\in \\RPP$.\nThen $\\benv[\\mu]{\\gamma\\theta} =\\gamma\\benv[\\gamma\\mu]{\\theta}$ \nand $\\fenv[\\mu]{\\gamma\\theta} =\\gamma\\fenv[\\gamma\\mu]{\\theta}$.\n\\end{proposition}","start":12824,"end":13112}]},{"id":"S008","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"缩放命题的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nThis is analogous to the proof of \\cite[Proposition~12.22(i)]{BC17}.\n\\end{proof}\n","start":13113,"end":13208}]},{"id":"S009","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件列表：BC17, Proposition 12.22(i)；原文 L395 仅按其证明类比。"],"source_only":false,"title":"经典包络缩放论证","source_units":[]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"闭真凸函数类","source_units":[{"raw_tex":"Denote by $\\Gamma_0(X)$ the set of all proper lower semicontinuous convex functions from $X$ to $\\left]-\\infty, +\\infty\\right]$. \n","start":16624,"end":16754}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"Legendre 站立假设","source_units":[{"raw_tex":"From now on, we strengthen our assumptions by requiring that \n\\begin{empheq}[box=\\mybluebox]{equation}\n\\label{e:fLegendre}\n\\text{$f\\in\\Gamma_0(X)$ is a convex function of Legendre type and\n$U := \\intdom f$.}\n\\end{empheq}\nThis will allow us to obtain a quite satisfying theory in which\nthe envelopes are convex functions. \nNote that $f$ is essentially smooth and essentially strictly convex in the sense of \\cite[Section~26]{Roc70}.\n","start":16754,"end":17186}]},{"id":"S012","semantic_type":"derived_fact","kind":"derived_fact","labels":["e:Leggy"],"declared_external":false,"source_only":false,"title":"梯度同胚","source_units":[{"raw_tex":"It is well known that\n\\begin{equation}\n\\label{e:Leggy}\n\\nabla f\\colon U\\to U^*:= \\intdom f^*\n\\text{~is a homeomorphism with~}\n\\big(\\nabla f\\big)^{-1} = \\nabla f^*.\n\\end{equation}\n","start":17186,"end":17365}]},{"id":"S013","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"连续 Hessian 假设","source_units":[{"raw_tex":"We will also work with the following standard assumptions:\n\\begin{itemize}\n\\item[\\bfseries A1] \n$\\nabla^2 f$ exists and is continuous on $U$;\n","start":17365,"end":17507}]},{"id":"S014","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"联合凸性假设","source_units":[{"raw_tex":"\\item[\\bfseries A2] \n$D_f$ is \\emph{jointly convex}, i.e., convex on $X\\times X$;\n","start":17507,"end":17589}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"右变量严格凸假设","source_units":[{"raw_tex":"\\item[\\bfseries A3] \n$(\\forall x\\in U)$ $D_f(x,\\cdot)$ is strictly convex on $U$;\n","start":17589,"end":17671}]},{"id":"S016","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"右变量强制性假设","source_units":[{"raw_tex":"\\item[\\bfseries A4] \n$(\\forall x\\in U)$ $D_f(x,\\cdot)$ is coercive, i.e., \n$D_f(x, y) \\to +\\infty$ as $\\|y\\| \\to +\\infty$.\n\\end{itemize}\nWe also henceforth assume that \n\\begin{empheq}[box=\\mybluebox]{equation}\n\\text{all assumptions \\textbf{A1}--\\textbf{A4} hold.}\n\\end{empheq}\n","start":17671,"end":17948}]},{"id":"S017","semantic_type":"proposition","kind":"proposition","labels":["e:bcoeR","e:brilliantissimo","e:fcoeR","f:coeR","l:coeRi","l:coeRii"],"declared_external":false,"source_only":false,"title":"正则化目标的强制性条件","source_units":[{"raw_tex":"\\begin{fact} \n\\label{f:coeR}\nLet $\\theta\\in\\Gamma_0(X)$ be such that $\\dom\\theta\\cap U\\neq\\varnothing$ and let $\\gamma \\in \\RPP$.\nConsider the following properties:\n\\begin{itemize}\n\\item[\\rm (a)]\n$U\\cap \\dom\\theta$ is bounded.\n\\item[\\rm (b)]\n$\\inf\\theta(U)>-\\infty$.\n\\item[\\rm (c)]\n$f$ is supercoercive, i.e., \n$f(x)/\\|x\\| \\to +\\infty$ as $\\|x\\| \\to +\\infty$.\n\\item[\\rm (d)]\n$(\\forall x\\in U)$ $D_f(x,\\cdot)$ is supercoercive.\n\\end{itemize}\nThen the following hold:\n\\begin{enumerate}\n\\item \\label{l:coeRi}\nIf any of the conditions (a), (b), or (c) holds, then \n\\begin{equation} \\label{e:bcoeR}\n(\\forall y \\in U) \\quad \\theta(\\cdot) + \\frac{1}{\\gamma}D_f(\\cdot,y) \\;\\;\n\\text{is coercive}\n\\end{equation}\nor, equivalently,\n\\begin{equation}\n\\frac{1}{\\gamma}\\ran\\nabla f \\subseteq \\intdom\\left(\\frac{1}{\\gamma}f +\\theta\\right)^*.\n\\label{e:brilliantissimo}\n\\end{equation}\n\n\\item \\label{l:coeRii}\nIf any of the conditions (a), (b), or (d) holds, then \n\\begin{equation} \\label{e:fcoeR}\n(\\forall x \\in U) \\quad \\theta(\\cdot) + \\frac{1}{\\gamma}D_f(x,\\cdot) \\;\\;\n\\text{is coercive.}\n\\end{equation}\n\\end{enumerate}\n\\end{fact}","start":21812,"end":22925}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":["e:bcoeR","f:coeR","l:coeRi","l:coeRii"],"declared_external":false,"source_only":false,"title":"左强制性条件","source_units":[{"raw_tex":"\\begin{equation} \\label{e:bcoeR}\n(\\forall y \\in U) \\quad \\theta(\\cdot) + \\frac{1}{\\gamma}D_f(\\cdot,y) \\;\\;\n\\text{is coercive}\n\\end{equation}\n","start":22373,"end":22514}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":["e:fcoeR","f:coeR","l:coeRi","l:coeRii"],"declared_external":false,"source_only":false,"title":"右强制性条件","source_units":[{"raw_tex":"\\begin{equation} \\label{e:fcoeR}\n(\\forall x \\in U) \\quad \\theta(\\cdot) + \\frac{1}{\\gamma}D_f(x,\\cdot) \\;\\;\n\\text{is coercive.}\n\\end{equation}\n","start":22757,"end":22899}]},{"id":"S020","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"强制性事实的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nSince $\\frac{1}{\\gamma}D_f =D_{\\frac{1}{\\gamma}f}$,\nthe result follows from \\cite[Lemma~2.12]{BCN06} \napplied to $\\tfrac{1}{\\gamma}f$.\n\\end{proof}\n","start":22926,"end":23087}]},{"id":"S021","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件列表：BCN06, Lemma 2.12；原文 L671 明确引用。"],"source_only":false,"title":"外部强制性引理","source_units":[]},{"id":"S022","semantic_type":"proposition","kind":"proposition","labels":["p:exact","p:exact_left","p:exact_right"],"declared_external":false,"source_only":false,"title":"包络性质与唯一取到","source_units":[{"raw_tex":"\\begin{proposition} \n\\label{p:exact} \nLet $\\theta \\colon X \\to \\left]-\\infty, +\\infty\\right]$ be convex\nand such that $U\\cap \\dom\\theta\\neq\\varnothing$, and let $\\gamma \\in \\RPP$.\nThen the following hold:\n\\begin{enumerate}\n\\item \n\\label{p:exact_left}\n$\\benv[\\gamma]{\\theta}$ is convex and continuous on $U$, and\n\t\\begin{enumerate}\n\t\\item \n\tif \\eqref{e:bcoeR} holds, i.e., $(\\forall y \\in U)$ $\\theta(\\cdot) +\\frac{1}{\\gamma}D_f(\\cdot,y)$ is coercive, \n\tthen $\\benv[\\gamma]{\\theta}$ is proper; \n\t\\item\n\tif $\\theta\\in\\Gamma_0(X)$ and $\\theta(\\cdot) +\\frac{1}{\\gamma}D_f(\\cdot,y)$ is coercive for a given $y\\in U$, \n\tthen there exists a unique point $z\\in U$ such that $\\benv[\\gamma]{\\theta}(y)=\\theta(z) +\\frac{1}{\\gamma}D_f(z,y)$.\n\t\\end{enumerate}\n\n\\item \n\\label{p:exact_right} \n$\\fenv[\\gamma]{\\theta}$ is convex and continuous on $U$, and\n\t\\begin{enumerate}\n\t\\item \n\tif \\eqref{e:fcoeR} holds, i.e., $(\\forall x \\in U)$ $\\theta(\\cdot) +\\frac{1}{\\gamma}D_f(x,\\cdot)$ is coercive, \n\tthen $\\fenv[\\gamma]{\\theta}$ is proper; \n\t\\item\n\tif $\\theta\\in\\Gamma_0(X)$ and $\\theta(\\cdot) +\\frac{1}{\\gamma}D_f(x,\\cdot)$ is coercive for a given $x\\in U$, \n\tthen there exists a unique point $z\\in U$ such that $\\fenv[\\gamma]{\\theta}(z)=\\theta(z) +\\frac{1}{\\gamma}D_f(x,z)$.\n\t\\end{enumerate}\n\\end{enumerate}\n\\end{proposition}","start":23247,"end":24554}]},{"id":"S023","semantic_type":"proposition","kind":"proposition","labels":["p:exact","p:exact_left","p:exact_right"],"declared_external":false,"source_only":false,"title":"左近端唯一取到","source_units":[{"raw_tex":"\tif $\\theta\\in\\Gamma_0(X)$ and $\\theta(\\cdot) +\\frac{1}{\\gamma}D_f(\\cdot,y)$ is coercive for a given $y\\in U$, \n\tthen there exists a unique point $z\\in U$ such that $\\benv[\\gamma]{\\theta}(y)=\\theta(z) +\\frac{1}{\\gamma}D_f(z,y)$.\n","start":23748,"end":23977}]},{"id":"S024","semantic_type":"proposition","kind":"proposition","labels":["p:exact","p:exact_left","p:exact_right"],"declared_external":false,"source_only":false,"title":"右近端取到的原文表述","source_units":[{"raw_tex":"\tif $\\theta\\in\\Gamma_0(X)$ and $\\theta(\\cdot) +\\frac{1}{\\gamma}D_f(x,\\cdot)$ is coercive for a given $x\\in U$, \n\tthen there exists a unique point $z\\in U$ such that $\\fenv[\\gamma]{\\theta}(z)=\\theta(z) +\\frac{1}{\\gamma}D_f(x,z)$.\n","start":24275,"end":24504}]},{"id":"S025","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"唯一取到命题的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nSince $\\frac{1}{\\gamma}D_f =D_{\\frac{1}{\\gamma}f}$,\nthe result follows from\n \\cite[Propositions~3.4 and 3.5]{BCN06}\napplied to $\\tfrac{1}{\\gamma}f$.\n\\end{proof}\n","start":24555,"end":24730}]},{"id":"S026","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件列表：BCN06, Propositions 3.4 and 3.5；原文 L714 引用。"],"source_only":false,"title":"外部包络与取到命题","source_units":[]},{"id":"S027","semantic_type":"definition","kind":"definition","labels":["d:prox","e:dbprox"],"declared_external":false,"source_only":false,"title":"左近端映射","source_units":[{"raw_tex":"\\begin{definition}[Bregman proximity operators] \n\\label{d:prox}\nLet $\\theta\\in\\Gamma_0(X)$ be such that \n$U\\cap \\dom\\theta\\neq\\varnothing$.\nIf \\eqref{e:bcoeR} holds for $\\gamma =1$, then \nthe \\emph{left proximity operator} associated with $\\theta$ is\n\\begin{equation} \\label{e:dbprox}\n\\bprox{\\theta} \\colon U \\to U \\colon\ny\\mapsto\n\\underset{x\\in X}{\\operatorname{argmin}}\\;\\: \\big(\\theta(x) +D_f(x,y)\\big).\n\\end{equation}\n","start":24842,"end":25264}]},{"id":"S028","semantic_type":"definition","kind":"definition","labels":["e:dfprox"],"declared_external":false,"source_only":false,"title":"右近端映射","source_units":[{"raw_tex":"If \\eqref{e:fcoeR} holds for $\\gamma =1$, then \nthe \\emph{right proximity operator} associated with $\\theta$ is\n\\begin{equation} \\label{e:dfprox}\n\\fprox{\\theta}\\colon U \\to  U \\colon\nx \\mapsto\n\\underset{y\\in X}{\\operatorname{argmin}}\\;\\: \\big(\\theta(y) +D_f(x,y)\\big).\n\\end{equation}\n\\end{definition}\n","start":25264,"end":25565}]},{"id":"S029","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"指标函数保证投影适定","source_units":[{"raw_tex":"Given a closed convex subset $C$ of $X$ with $C\\cap U \\neq\\varnothing$, we have that $\\iota_C \\in \\Gamma_0(X)$, $\\dom\\iota_C =C$, \nand hence $U\\cap \\dom\\iota_C =U\\cap C \\neq\\varnothing$ and also $\\inf \\iota_C(U) =0 >-\\infty$, \nwhich together with \\cref{f:coeR} imply that \\eqref{e:bcoeR} and \\eqref{e:fcoeR} hold for all $\\gamma \\in \\RPP$. \nThis leads to the following definition. \n","start":26129,"end":26511}]},{"id":"S030","semantic_type":"definition","kind":"definition","labels":["d:BregProj"],"declared_external":false,"source_only":false,"title":"左右 Bregman 投影","source_units":[{"raw_tex":"\n\\begin{definition}[Bregman projectors]\n\\label{d:BregProj}\nLet $C$ be a closed convex subset of $X$ such that \n$U\\cap C\\neq\\varnothing$. \nThen $\\bproj{C} :=\\bprox{\\iota_C}$ is the \\emph{left Bregman projector} onto $C$\nand $\\fproj{C} :=\\fprox{\\iota_C}$ is the \\emph{right Bregman projector} onto $C$.\n\\end{definition}\n","start":26511,"end":26829}]},{"id":"S031","semantic_type":"derived_fact","kind":"derived_fact","labels":["e:benv-bproxall","e:fenv-fproxall","r:prox"],"declared_external":false,"source_only":false,"title":"参数与目标缩放换算","source_units":[{"raw_tex":"\\begin{remark}\n\\label{r:prox}\nLet $\\theta\\in\\Gamma_0(X)$ be such that \n$U\\cap \\dom\\theta\\neq\\varnothing$ and let $\\gamma \\in \\RPP$.\n\\cref{p:12.22i} implies that \n$\\benv{\\gamma\\theta} =\\gamma\\benv[\\gamma]{\\theta}$ \nand $\\fenv{\\gamma\\theta} =\\gamma\\fenv[\\gamma]{\\theta}$. \n","start":29283,"end":29554}]},{"id":"S032","semantic_type":"derived_fact","kind":"derived_fact","labels":["e:benv-bprox","e:benv-bproxall","e:fenv-fproxall","r:prox"],"declared_external":false,"source_only":false,"title":"左包络的近端分解","source_units":[{"raw_tex":"We thus derive from the definition that if \\eqref{e:bcoeR} holds, then\n\\begin{subequations}\n\\label{e:benv-bproxall}\n\\begin{equation}\n\\label{e:benv-bprox}\n\\benv[\\gamma]{\\theta}(y) =\\theta\\big(\\bprox{\\gamma\\theta}(y)\\big) \n+\\frac{1}{\\gamma}D_f\\big(\\bprox{\\gamma\\theta}(y), y\\big)\n\\end{equation}\n","start":29554,"end":29847}]},{"id":"S033","semantic_type":"derived_fact","kind":"derived_fact","labels":["e:benv-bprox'","e:benv-bproxall","e:fenv-fproxall","r:prox"],"declared_external":false,"source_only":false,"title":"左近端目标值夹逼","source_units":[{"raw_tex":"and, by combining with \\cref{p:12.9}\\ref{p:12.9_left},\n\\begin{equation}\n\\label{e:benv-bprox'}\n\\theta\\big(\\bprox{\\gamma\\theta}(y)\\big) \\leq \\benv[\\gamma]{\\theta}(y) \\leq \\theta(y).\n\\end{equation}\n\\end{subequations}\n","start":29847,"end":30061}]},{"id":"S034","semantic_type":"derived_fact","kind":"derived_fact","labels":["e:benv-bproxall","e:fenv-fprox","e:fenv-fproxall","r:prox"],"declared_external":false,"source_only":false,"title":"右包络的近端分解","source_units":[{"raw_tex":"Similarly, if \\eqref{e:fcoeR} holds, then\n\\begin{subequations}\n\\label{e:fenv-fproxall}\n\\begin{equation}\n\\label{e:fenv-fprox}\n\\fenv[\\gamma]{\\theta}(x) =\\theta\\big(\\fprox{\\gamma\\theta}(x)\\big) +\n\\frac{1}{\\gamma}D_f\\big(x, \\fprox{\\gamma\\theta}(x)\\big)\n\\end{equation}\n","start":30061,"end":30325}]},{"id":"S035","semantic_type":"derived_fact","kind":"derived_fact","labels":["e:benv-bproxall","e:fenv-fproxall","r:prox"],"declared_external":false,"source_only":false,"title":"右近端目标值夹逼","source_units":[{"raw_tex":"and \n\\begin{equation}\n\\theta\\big(\\fprox{\\gamma\\theta}(x)\\big) \\leq \\fenv[\\gamma]{\\theta}(x) \\leq \\theta(x).\n\\end{equation}\n\\end{subequations}\n\\end{remark}","start":30325,"end":30479}]},{"id":"S036","semantic_type":"proposition","kind":"proposition","labels":["e:pro","p:prox","p:prox_left","p:prox_right"],"declared_external":false,"source_only":false,"title":"近端映射的变分刻画","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{p:prox}\nLet $\\theta\\in\\Gamma_0(X)$ be such that $\\dom\\theta\\cap U\\neq\\varnothing$ and let $\\gamma \\in \\RPP$.\n\\begin{enumerate}\n\\item \n\\label{p:prox_left}\nSuppose that \\eqref{e:bcoeR} holds.\nThen for every $(x,y) \\in U\\times U$, \nthe following conditions are equivalent:\n\t\\begin{enumerate}\n\t\\item \n\t$x =\\bprox{\\gamma\\theta}(y)$,\n\t\\item \n\t$0 \\in \\gamma\\partial\\theta(x) +\\nabla f(x) -\\nabla f(y)$,\n\t\\item \n\t$(\\forall z \\in X)\\quad \\scal{\\nabla f(y) -\\nabla f(x)}{z -x} +\\gamma\\theta(x) \\leq \\gamma\\theta(z)$.\n\t\\end{enumerate}\nMoreover, \n\\begin{equation} \\label{e:pro}\n\\bprox{\\gamma\\theta}=(\\nabla f +\\gamma\\partial\\theta)^{-1}\\circ \\nabla f \n\\end{equation}\nis continuous on $U$. \n\\item \n\\label{p:prox_right}\nSuppose that \\eqref{e:fcoeR} holds. \nThen for every $(x,y) \\in U\\times U$, \nthe following conditions are equivalent:\n\t\\begin{enumerate}\n\t\\item \n\t$y =\\fprox{\\gamma\\theta}(x)$,\n\t\\item \n\t$0 \\in \\gamma\\partial\\theta(y) +\\nabla^2 f(y)(y -x)$,\n\t\\item \n\t$(\\forall z \\in X)\\quad \\scal{\\nabla^2 f(y)(x -y)}{z -y} +\\gamma\\theta(y) \\leq \\gamma\\theta(z)$.\n\t\\end{enumerate}\nMoreover, $\\fprox{\\gamma\\theta}$ is continuous on $U$. \n\\end{enumerate}\n\\end{proposition}","start":31727,"end":32910}]},{"id":"S037","semantic_type":"proposition","kind":"proposition","labels":["p:prox","p:prox_left","p:prox_right"],"declared_external":false,"source_only":false,"title":"左近端变分不等式","source_units":[{"raw_tex":"Suppose that \\eqref{e:bcoeR} holds.\nThen for every $(x,y) \\in U\\times U$, \nthe following conditions are equivalent:\n\t\\begin{enumerate}\n\t\\item \n\t$x =\\bprox{\\gamma\\theta}(y)$,\n\t\\item \n\t$0 \\in \\gamma\\partial\\theta(x) +\\nabla f(x) -\\nabla f(y)$,\n\t\\item \n\t$(\\forall z \\in X)\\quad \\scal{\\nabla f(y) -\\nabla f(x)}{z -x} +\\gamma\\theta(x) \\leq \\gamma\\theta(z)$.\n","start":31908,"end":32261}]},{"id":"S038","semantic_type":"proposition","kind":"proposition","labels":["p:prox","p:prox_left","p:prox_right"],"declared_external":false,"source_only":false,"title":"右近端变分不等式","source_units":[{"raw_tex":"Suppose that \\eqref{e:fcoeR} holds. \nThen for every $(x,y) \\in U\\times U$, \nthe following conditions are equivalent:\n\t\\begin{enumerate}\n\t\\item \n\t$y =\\fprox{\\gamma\\theta}(x)$,\n\t\\item \n\t$0 \\in \\gamma\\partial\\theta(y) +\\nabla^2 f(y)(y -x)$,\n\t\\item \n\t$(\\forall z \\in X)\\quad \\scal{\\nabla^2 f(y)(x -y)}{z -y} +\\gamma\\theta(y) \\leq \\gamma\\theta(z)$.\n","start":32460,"end":32804}]},{"id":"S039","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"近端刻画的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nApply \\cite[Proposition~3.10]{BCN06} to $\\gamma\\theta$.\n\\end{proof}\n","start":32911,"end":32993}]},{"id":"S040","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件列表：BCN06, Proposition 3.10；原文 L956 引用。"],"source_only":false,"title":"外部近端刻画命题","source_units":[]},{"id":"S041","semantic_type":"lemma","kind":"lemma","labels":["l:cvg"],"declared_external":false,"source_only":false,"title":"紧集内极小化网的聚点","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{l:cvg}\nLet $C$ be a compact subset of a Hausdorff space $\\mathcal X$,\nlet $\\phi\\colon \\mathcal X \\to \\left[-\\infty, +\\infty\\right]$ be lower \nsemicontinuous,\nlet $(x_a)_{a\\in A}$ be a net in $C$,\nand suppose that $\\phi(x_a) \\to \\inf\\phi(\\mathcal X)$. \nThen $\\argmin\\phi \\neq\\varnothing$ \nand all cluster points of $(x_a)_{a\\in A}$ lie in $\\argmin\\phi$.\nConsequently, if $\\phi$ attains its minimum at a unique point $u$, then $x_a \\to u$.\n\\end{lemma}","start":40270,"end":40740}]},{"id":"S042","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"极小化网引理的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nThis follows from the lower semicontinuity of $\\phi$ and \\cite[Lemma 1.14]{BC17}.\n\\end{proof}\n","start":40741,"end":40849}]},{"id":"S043","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件列表：BC17, Lemma 1.14；原文 L1169 引用。"],"source_only":false,"title":"紧性论证的外部引理","source_units":[]},{"id":"S044","semantic_type":"proposition","kind":"proposition","labels":["p:12.9","p:12.9_left","p:12.9_right"],"declared_external":false,"source_only":false,"title":"包络域与大参数极限","source_units":[{"raw_tex":"\\begin{proposition} \n\\label{p:12.9} \nLet $\\theta \\colon X \\to \\left]-\\infty, +\\infty\\right]$ be such that $U\\cap \\dom\\theta\\neq\\varnothing$ and let $\\gamma \\in \\RPP$.\nThen the following hold:\n\\begin{enumerate}\n\\item \n\\label{p:12.9_left}\n$\\dom\\benv[\\gamma]{\\theta}=U$, \nand $(\\forall y\\in U)(\\forall\\mu \\in \\left]\\gamma, +\\infty\\right[)$ \n$\\inf \\theta(X) \\leq \\benv[\\mu]{\\theta}(y) \\leq \\benv[\\gamma]{\\theta}(y) \\leq \\theta(y)$.\nConsequently, $\\inf \\theta(X) \\leq \\inf \\benv[\\gamma]{\\theta}(X) \\leq \\inf \\theta(U)$, \nand $\\benv[\\gamma]{\\theta}(y) \\downarrow \\inf \\theta(X)$ as $\\gamma \\uparrow +\\infty$.\n\\item \n\\label{p:12.9_right}\n$\\dom\\fenv[\\gamma]{\\theta}=\\dom f$, \nand $(\\forall x \\in U)(\\forall\\mu \\in \\left]\\gamma, +\\infty\\right[)$ \n$\\inf \\theta(X) \\leq \\fenv[\\mu]{\\theta}(x) \\leq \\fenv[\\gamma]{\\theta}(x) \\leq \\theta(x)$.\nConsequently, $\\inf \\theta(X) \\leq \\inf \\fenv[\\gamma]{\\theta}(X) \\leq \\inf \\theta(U)$, \nand $\\fenv[\\gamma]{\\theta}(x) \\downarrow \\inf \\theta(X)$ as $\\gamma \\uparrow +\\infty$.\n\\end{enumerate}\n\\end{proposition}","start":13355,"end":14391}]},{"id":"S045","semantic_type":"assumption","kind":"assumption","labels":["p:12.9","p:12.9_left","p:12.9_right"],"declared_external":false,"source_only":false,"title":"早期包络命题的局部假设","source_units":[{"raw_tex":"Let $\\theta \\colon X \\to \\left]-\\infty, +\\infty\\right]$ be such that $U\\cap \\dom\\theta\\neq\\varnothing$ and let $\\gamma \\in \\RPP$.\n","start":13392,"end":13522}]},{"id":"S046","semantic_type":"proposition","kind":"proposition","labels":["p:12.9","p:12.9_left","p:12.9_right"],"declared_external":false,"source_only":false,"title":"左包络的域与序界","source_units":[{"raw_tex":"\\item \n\\label{p:12.9_left}\n$\\dom\\benv[\\gamma]{\\theta}=U$, \nand $(\\forall y\\in U)(\\forall\\mu \\in \\left]\\gamma, +\\infty\\right[)$ \n$\\inf \\theta(X) \\leq \\benv[\\mu]{\\theta}(y) \\leq \\benv[\\gamma]{\\theta}(y) \\leq \\theta(y)$.\nConsequently, $\\inf \\theta(X) \\leq \\inf \\benv[\\gamma]{\\theta}(X) \\leq \\inf \\theta(U)$, \n","start":13565,"end":13871}]},{"id":"S047","semantic_type":"proposition","kind":"proposition","labels":["p:12.9","p:12.9_left","p:12.9_right"],"declared_external":false,"source_only":false,"title":"左包络的大参数极限","source_units":[{"raw_tex":"and $\\benv[\\gamma]{\\theta}(y) \\downarrow \\inf \\theta(X)$ as $\\gamma \\uparrow +\\infty$.\n","start":13871,"end":13958}]},{"id":"S048","semantic_type":"proposition","kind":"proposition","labels":["p:12.9","p:12.9_left","p:12.9_right"],"declared_external":false,"source_only":false,"title":"右包络的域与序界","source_units":[{"raw_tex":"\\item \n\\label{p:12.9_right}\n$\\dom\\fenv[\\gamma]{\\theta}=\\dom f$, \nand $(\\forall x \\in U)(\\forall\\mu \\in \\left]\\gamma, +\\infty\\right[)$ \n$\\inf \\theta(X) \\leq \\fenv[\\mu]{\\theta}(x) \\leq \\fenv[\\gamma]{\\theta}(x) \\leq \\theta(x)$.\nConsequently, $\\inf \\theta(X) \\leq \\inf \\fenv[\\gamma]{\\theta}(X) \\leq \\inf \\theta(U)$, \n","start":13958,"end":14271}]},{"id":"S049","semantic_type":"proposition","kind":"proposition","labels":["p:12.9","p:12.9_left","p:12.9_right"],"declared_external":false,"source_only":false,"title":"右包络的大参数极限","source_units":[{"raw_tex":"and $\\fenv[\\gamma]{\\theta}(x) \\downarrow \\inf \\theta(X)$ as $\\gamma \\uparrow +\\infty$.\n","start":14271,"end":14358}]},{"id":"S050","semantic_type":"proof","kind":"proof","labels":["e:170428a"],"declared_external":false,"source_only":false,"title":"包络域与极限的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{p:12.9_left}: We first show that $\\dom\\benv[\\gamma]{\\theta}=U$.\nLet $y \\in \\dom\\benv[\\gamma]{\\theta}$. \nThen $\\benv[\\gamma]{\\theta}(y) =\\inf_{x\\in X} \\big(\\theta(x) +\\frac{1}{\\gamma}D_f(x,y)\\big) <+\\infty$,\nand hence there exists $x \\in X$ such that $\\theta(x) +\\frac{1}{\\gamma}D_f(x,y) <+\\infty$.\nSince $\\theta(x) >-\\infty$, this yields $y \\in U$.\n\nFrom now on, let $y \\in U$, and pick $u \\in \\dom\\theta \\cap U$. \nThen $-f(y) <+\\infty$, $\\|\\nabla f(y)\\| <+\\infty$, $f(u) <+\\infty$, $\\theta(u) <+\\infty$, and\n\\begin{subequations}\n\\begin{align}\n\\benv[\\gamma]{\\theta}(y) &=\\inf_{x\\in X} \\Big(\\theta(x) +\\frac{1}{\\gamma}\\big(f(x) -f(y) -\\scal{\\nabla f(y)}{x -y}\\big)\\Big), \\\\ \n&\\leq \\theta(u) +\\frac{1}{\\gamma}\\big(f(u) -f(y) -\\scal{\\nabla f(y)}{u -y}\\big) <+\\infty,\n\\end{align}\n\\end{subequations} \nwhich gives $y \\in \\dom\\benv[\\gamma]{\\theta}$. Hence, $\\dom \\benv[\\gamma]{\\theta} =U$. \n\nNext, let $\\mu \\in \\left]\\gamma, +\\infty\\right[$. \nThen $\\frac{1}{\\mu} < \\frac{1}{\\gamma}$, \n$\\theta \\leq \\theta +\\frac{1}{\\mu}D_f(\\cdot,y) \\leq \\theta +\\frac{1}{\\gamma}D_f(\\cdot,y)$, and so\n\\begin{equation}\n\\inf_{x \\in X} \\theta(x) \\leq \\benv[\\mu]{\\theta}(y) \\leq \\benv[\\gamma]{\\theta}(y) =\\inf_{x \\in X} \\big(\\theta(x) +\\frac{1}{\\gamma}D_f(x,y)\\big)\n\\leq \\theta(y) +\\frac{1}{\\gamma}D_f(y,y) =\\theta(y). \n\\end{equation}\nTherefore,\n\\begin{equation}\n\\label{e:170428a}\n\\inf \\theta(X) \\leq \\benv[\\mu]{\\theta}(y) \\leq \\benv[\\gamma]{\\theta}(y) \\leq \\theta(y).\n\\end{equation}\nTaking now the infimum over $y\\in U$ yields\n$\\inf \\theta(X) \\leq \\inf \\benv[\\gamma]{\\theta}(X) \\leq \\inf \\theta(U)$.\nConsequently, \n\\begin{equation}\n\\inf \\theta(X) \\leq \\varliminf_{\\gamma \\to+\\infty}\n\\benv[\\gamma]{\\theta}(y).\n\\end{equation}\nOn the other hand, $(\\forall x \\in X)$ $\\benv[\\gamma]{\\theta}(y) \\leq \\theta(x) +\\frac{1}{\\gamma}D_f(x,y)$, \nwhich implies that $(\\forall x \\in X)$ $\\varlimsup_{\\gamma\n\\to+\\infty} \\benv[\\gamma]{\\theta}(y) \\leq \\theta(x)$ and thus\n$\\varlimsup_{\\gamma \\to+\\infty} \\benv[\\gamma]{\\theta}(y) \\leq\n\\inf \\theta(X)$. \nAltogether,\n$\\lim_{\\gamma\\to+\\infty} \\benv[\\gamma]{\\theta}(y) =\n\\inf \\theta(X)$ and the conclusion follows from\n\\eqref{e:170428a}. \n\n\\ref{p:12.9_right}: This is similar to \\ref{p:12.9_left}. \n\\end{proof}","start":14392,"end":16620}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"确定左包络的域","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{p:12.9_left}: We first show that $\\dom\\benv[\\gamma]{\\theta}=U$.\nLet $y \\in \\dom\\benv[\\gamma]{\\theta}$. \nThen $\\benv[\\gamma]{\\theta}(y) =\\inf_{x\\in X} \\big(\\theta(x) +\\frac{1}{\\gamma}D_f(x,y)\\big) <+\\infty$,\nand hence there exists $x \\in X$ such that $\\theta(x) +\\frac{1}{\\gamma}D_f(x,y) <+\\infty$.\nSince $\\theta(x) >-\\infty$, this yields $y \\in U$.\n\nFrom now on, let $y \\in U$, and pick $u \\in \\dom\\theta \\cap U$. \nThen $-f(y) <+\\infty$, $\\|\\nabla f(y)\\| <+\\infty$, $f(u) <+\\infty$, $\\theta(u) <+\\infty$, and\n\\begin{subequations}\n\\begin{align}\n\\benv[\\gamma]{\\theta}(y) &=\\inf_{x\\in X} \\Big(\\theta(x) +\\frac{1}{\\gamma}\\big(f(x) -f(y) -\\scal{\\nabla f(y)}{x -y}\\big)\\Big), \\\\ \n&\\leq \\theta(u) +\\frac{1}{\\gamma}\\big(f(u) -f(y) -\\scal{\\nabla f(y)}{u -y}\\big) <+\\infty,\n\\end{align}\n\\end{subequations} \nwhich gives $y \\in \\dom\\benv[\\gamma]{\\theta}$. Hence, $\\dom \\benv[\\gamma]{\\theta} =U$. \n","start":14392,"end":15295}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":["e:170428a"],"declared_external":false,"source_only":false,"title":"建立参数单调性与序界","source_units":[{"raw_tex":"\nNext, let $\\mu \\in \\left]\\gamma, +\\infty\\right[$. \nThen $\\frac{1}{\\mu} < \\frac{1}{\\gamma}$, \n$\\theta \\leq \\theta +\\frac{1}{\\mu}D_f(\\cdot,y) \\leq \\theta +\\frac{1}{\\gamma}D_f(\\cdot,y)$, and so\n\\begin{equation}\n\\inf_{x \\in X} \\theta(x) \\leq \\benv[\\mu]{\\theta}(y) \\leq \\benv[\\gamma]{\\theta}(y) =\\inf_{x \\in X} \\big(\\theta(x) +\\frac{1}{\\gamma}D_f(x,y)\\big)\n\\leq \\theta(y) +\\frac{1}{\\gamma}D_f(y,y) =\\theta(y). \n\\end{equation}\nTherefore,\n\\begin{equation}\n\\label{e:170428a}\n\\inf \\theta(X) \\leq \\benv[\\mu]{\\theta}(y) \\leq \\benv[\\gamma]{\\theta}(y) \\leq \\theta(y).\n\\end{equation}\nTaking now the infimum over $y\\in U$ yields\n$\\inf \\theta(X) \\leq \\inf \\benv[\\gamma]{\\theta}(X) \\leq \\inf \\theta(U)$.\n","start":15295,"end":15983}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"极限下界","source_units":[{"raw_tex":"Consequently, \n\\begin{equation}\n\\inf \\theta(X) \\leq \\varliminf_{\\gamma \\to+\\infty}\n\\benv[\\gamma]{\\theta}(y).\n\\end{equation}\n","start":15983,"end":16107}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"任意点比较与极限结论","source_units":[{"raw_tex":"On the other hand, $(\\forall x \\in X)$ $\\benv[\\gamma]{\\theta}(y) \\leq \\theta(x) +\\frac{1}{\\gamma}D_f(x,y)$, \nwhich implies that $(\\forall x \\in X)$ $\\varlimsup_{\\gamma\n\\to+\\infty} \\benv[\\gamma]{\\theta}(y) \\leq \\theta(x)$ and thus\n$\\varlimsup_{\\gamma \\to+\\infty} \\benv[\\gamma]{\\theta}(y) \\leq\n\\inf \\theta(X)$. \nAltogether,\n$\\lim_{\\gamma\\to+\\infty} \\benv[\\gamma]{\\theta}(y) =\n\\inf \\theta(X)$ and the conclusion follows from\n\\eqref{e:170428a}. \n","start":16107,"end":16549}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"右包络证明的类比声明","source_units":[{"raw_tex":"\n\\ref{p:12.9_right}: This is similar to \\ref{p:12.9_left}. \n\\end{proof}","start":16549,"end":16620}]},{"id":"S056","semantic_type":"corollary","kind":"corollary","labels":["c:BregProj","c:BregProj_left","c:BregProj_right"],"declared_external":false,"source_only":false,"title":"Bregman 投影的变分刻画","source_units":[{"raw_tex":"\\begin{corollary}\n\\label{c:BregProj}\nLet $C$ be a closed convex subset of $X$ such that \n$U\\cap C\\neq\\varnothing$, \nlet $(x,y) \\in U\\times U$, \nand let $p \\in U\\cap C$.\nThen the following hold:\n\\begin{enumerate}\n\\item \n\\label{c:BregProj_left}\n$p =\\bproj{C}y \\ \\Leftrightarrow \\ (\\forall z \\in C)\\; \n\\scal{\\nabla f(y) -\\nabla f(p)}{z -p} \\leq 0$. \n\\item \n\\label{c:BregProj_right}\n$p =\\fproj{C}x \\ \\Leftrightarrow \\ (\\forall z \\in C)\\; \n\\scal{\\nabla^2 f(p)(x -p)}{z -p} \\leq 0$. \n\\end{enumerate}\n\\end{corollary}","start":34473,"end":34982}]},{"id":"S057","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"投影刻画的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nIn light of \\cref{d:BregProj}, we \napply \\cref{p:prox} \n(see also \\cite[Proposition~3.16]{BB97}).\n\\end{proof}","start":34983,"end":35106}]},{"id":"S058","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件列表：Standard indicator-function and convex subdifferential conventions。"],"source_only":false,"title":"指标函数与次微分约定","source_units":[]},{"id":"S059","semantic_type":"proposition","kind":"proposition","labels":["p:limthetaprox","p:limthetaprox_left","p:limthetaprox_right"],"declared_external":false,"source_only":false,"title":"大参数近端目标值极限","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{p:limthetaprox}\nLet $\\theta\\in\\Gamma_0(X)$ be such that $U\\cap \\dom\\theta\\neq\\varnothing$ and let $x, y \\in U$.\nThen the following hold:\n\\begin{enumerate}\n\\item \n\\label{p:limthetaprox_left}\nIf \\eqref{e:bcoeR} holds for all $\\gamma\\in\\RPP$, \nthen $\\theta(\\bprox{\\gamma\\theta}(y)) \\to \\inf \\theta(X)$ as $\\gamma \\uparrow +\\infty$.\n\\item \n\\label{p:limthetaprox_right}\nIf \\eqref{e:fcoeR} holds for all $\\gamma\\in \\RPP$, then  \n$\\theta(\\fprox{\\gamma\\theta}(x)) \\to \\inf \\theta(X)$ as $\\gamma \\uparrow +\\infty$.\n\\end{enumerate}\t\n\\end{proposition}","start":46810,"end":47377}]},{"id":"S060","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"目标值极限的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nWe shall just prove \\ref{p:limthetaprox_left} \nbecause the proof of \\ref{p:limthetaprox_right} is similar.\nAssume that \\eqref{e:bcoeR} holds for all \n$\\gamma \\in \\RPP$.\nCombining \\eqref{e:benv-bprox'} with \\cref{p:12.9}\\ref{p:12.9_left} yields\n\\begin{equation}\n\\inf\\theta(X) \\leq \\theta\\big(\\bprox{\\gamma\\theta}(y)\\big) \\leq \\benv[\\gamma]{\\theta}(y) \\to \\inf\\theta(X) \n\\quad\\text{as}\\quad \\gamma \\uparrow +\\infty,\n\\end{equation} \nwhich implies that $\\theta(\\bprox{\\gamma\\theta}(y)) \\to \n\\inf\\theta(X)$ as $\\gamma \\uparrow +\\infty$. \n\\end{proof}","start":47378,"end":47936}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"左近端目标值的夹逼","source_units":[{"raw_tex":"Assume that \\eqref{e:bcoeR} holds for all \n$\\gamma \\in \\RPP$.\nCombining \\eqref{e:benv-bprox'} with \\cref{p:12.9}\\ref{p:12.9_left} yields\n\\begin{equation}\n\\inf\\theta(X) \\leq \\theta\\big(\\bprox{\\gamma\\theta}(y)\\big) \\leq \\benv[\\gamma]{\\theta}(y) \\to \\inf\\theta(X) \n\\quad\\text{as}\\quad \\gamma \\uparrow +\\infty,\n\\end{equation} \nwhich implies that $\\theta(\\bprox{\\gamma\\theta}(y)) \\to \n\\inf\\theta(X)$ as $\\gamma \\uparrow +\\infty$. \n","start":47499,"end":47925}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"右目标值极限的类比声明","source_units":[{"raw_tex":"\\begin{proof}\nWe shall just prove \\ref{p:limthetaprox_left} \nbecause the proof of \\ref{p:limthetaprox_right} is similar.\n","start":47378,"end":47499},{"raw_tex":"\\end{proof}","start":47925,"end":47936}]},{"id":"S063","semantic_type":"theorem","kind":"theorem","labels":["p:limprox+","p:limprox+_left","p:limprox+_leftA","p:limprox+_leftP","p:limprox+_right","p:limprox+_rightP"],"declared_external":false,"source_only":false,"title":"大参数近端点趋向极小点与投影","source_units":[{"raw_tex":"\\begin{theorem}\n\\label{p:limprox+}\nLet $\\theta\\in\\Gamma_0(X)$ be coercive such that \n$U\\cap \\dom\\theta\\neq\\varnothing$ and let $x, y \\in U$.\nThen the following hold:\n\\begin{enumerate}\n\\item \n\\label{p:limprox+_left}\nThe net $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ is bounded \nwith all cluster points as $\\gamma \\uparrow +\\infty$ lying in $\\argmin\\theta$. \nMoreover,\n\t\\begin{enumerate}\n\t\\item\\label{p:limprox+_leftA} \n\tif $\\argmin\\theta$ is a singleton, \n\tthen $\\bprox{\\gamma\\theta}(y) \\to \\argmin\\theta$ as $\\gamma \\uparrow +\\infty$;\n\t\\item\\label{p:limprox+_leftP} \n\tif $\\argmin\\theta \\subseteq U$, \n\tthen $\\bprox{\\gamma\\theta}(y) \\to \\bproj{\\argmin\\theta}y$ as $\\gamma \\uparrow +\\infty$. \n\t\\end{enumerate}\n\n\\item \n\\label{p:limprox+_right}\nThe net $(\\fprox{\\gamma\\theta}(x))_{\\gamma \\in \\RPP}$ is bounded \nwith all cluster points as $\\gamma \\uparrow +\\infty$ lying in $\\argmin\\theta$. \nMoreover,\n\t\\begin{enumerate}\n\t\\item \n\tif $\\argmin\\theta$ is a singleton, \n\tthen $\\fprox{\\gamma\\theta}(x) \\to \\argmin\\theta$ as $\\gamma \\uparrow +\\infty$;\n\t\\item\\label{p:limprox+_rightP} \n\tif $\\argmin\\theta \\subseteq U$, \n\tthen $\\fprox{\\gamma\\theta}(x) \\to \\fproj{\\argmin\\theta}x$ as $\\gamma \\uparrow +\\infty$.  \n\t\\end{enumerate}\n\\end{enumerate}\t\n\\end{theorem}","start":47939,"end":49189}]},{"id":"S064","semantic_type":"assumption","kind":"assumption","labels":["p:limprox+","p:limprox+_left","p:limprox+_leftA","p:limprox+_leftP","p:limprox+_right","p:limprox+_rightP"],"declared_external":false,"source_only":false,"title":"主定理共同假设","source_units":[{"raw_tex":"Let $\\theta\\in\\Gamma_0(X)$ be coercive such that \n$U\\cap \\dom\\theta\\neq\\varnothing$ and let $x, y \\in U$.\n","start":47974,"end":48080}]},{"id":"S065","semantic_type":"assumption","kind":"assumption","labels":["p:limprox+","p:limprox+_left","p:limprox+_leftA","p:limprox+_leftP","p:limprox+_right","p:limprox+_rightP"],"declared_external":false,"source_only":false,"title":"投影极限的内域条件","source_units":[{"raw_tex":"\t\\item\\label{p:limprox+_leftP} \n\tif $\\argmin\\theta \\subseteq U$, \n\tthen $\\bprox{\\gamma\\theta}(y) \\to \\bproj{\\argmin\\theta}y$ as $\\gamma \\uparrow +\\infty$. \n","start":48478,"end":48634},{"raw_tex":"\t\\item\\label{p:limprox+_rightP} \n\tif $\\argmin\\theta \\subseteq U$, \n\tthen $\\fprox{\\gamma\\theta}(x) \\to \\fproj{\\argmin\\theta}x$ as $\\gamma \\uparrow +\\infty$.  \n","start":48984,"end":49142}]},{"id":"S066","semantic_type":"proof","kind":"proof","labels":["e:bounded"],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nFirst, by assumption, \\cite[Proposition~11.15(i)]{BC17} gives $\\argmin\\theta \\neq\\varnothing$. \nThis combined with \\cite[Lemma~1.24 and Corollary~8.5]{BC17} implies that \n$\\argmin\\theta =\\lev{\\inf\\theta(X)}\\theta$ is a nonempty closed convex subset of $X$.\nNow since $\\theta$ is coercive and since $D_f\\geq 0$, we immediately get that  \n\\eqref{e:bcoeR} and \\eqref{e:fcoeR} hold for all $\\gamma \\in \\RPP$.\n\n\\ref{p:limprox+_left}: It follows from \\eqref{e:benv-bprox'} that   \n\\begin{equation}\n\\label{e:bounded}\n(\\forall \\gamma \\in \\RPP)\\quad \\bprox{\\gamma\\theta}(y) \\in \\lev{\\theta(y)}\\theta,\n\\end{equation}\nand then\nfrom the coercivity of $\\theta$ and \\cite[Proposition~11.12]{BC17} that $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ is bounded.\nIn turn, \\cref{p:limthetaprox}\\ref{p:limthetaprox_left} and \\cref{l:cvg} imply that\nall cluster points of $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ as $\\gamma \\uparrow +\\infty$ lie in $\\argmin\\theta$, \nand we get \\ref{p:limprox+_leftA}.\n\nNow assume that $\\argmin\\theta \\subseteq U$. \nLet $y'$ be a cluster point of $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ as $\\gamma \\uparrow +\\infty$.  \nThen $y' \\in \\argmin\\theta \\subseteq U$ and there exists a sequence $(\\gamma_n)_\\nnn$ in $\\RPP$ such that \n$\\gamma_n \\uparrow +\\infty$ and $\\bprox{\\gamma_n\\theta}(y) \\to y'$ as $n \\to +\\infty$.\nLet $z \\in \\argmin\\theta$.  \nWe have $(\\forall\\nnn)$ $\\theta(z) \\leq \\theta(\\bprox{\\gamma_n\\theta}(y))$, \nand by Proposition~\\ref{p:prox}\\ref{p:prox_left},\n\\begin{equation}\n\\scal{\\nabla f(y) -\\nabla f\\big(\\bprox{\\gamma_n\\theta}(y)\\big)}{z -\\bprox{\\gamma_n\\theta}(y)} \n\\leq \\gamma_n\\Big(\\theta(z)\n-\\theta\\big(\\bprox{\\gamma_n\\theta}(y)\\big)\\Big) \\leq 0.\n\\end{equation}\nTaking the limit as $n \\to +\\infty$ and using the continuity of $\\nabla f$ yield\n\\begin{equation}\n\\scal{\\nabla f(y) -\\nabla f(y')}{z -y'} \\leq 0.\n\\end{equation}\nSince $z \\in \\argmin\\theta$ was chosen arbitrarily \nand since $\\argmin\\theta$ is a closed convex subset of $X$ with \n$U\\cap \\argmin\\theta \\neq\\varnothing$, \nin view of \\cref{c:BregProj}\\ref{c:BregProj_left}, $y' =\\bproj{\\argmin\\theta}y$, \nand $\\bproj{\\argmin\\theta}y$ is thus the only cluster point of $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ \nas $\\gamma \\uparrow +\\infty$.\nHence, \\ref{p:limprox+_leftP} holds.\n\n\\ref{p:limprox+_right}: \nThe proof is similar to the one of \\ref{p:limprox+_left}.\n\\end{proof}","start":49190,"end":51580}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"极小点存在","source_units":[{"raw_tex":"\\begin{proof}\nFirst, by assumption, \\cite[Proposition~11.15(i)]{BC17} gives $\\argmin\\theta \\neq\\varnothing$. \n","start":49190,"end":49300}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"极小点集非空闭凸","source_units":[{"raw_tex":"This combined with \\cite[Lemma~1.24 and Corollary~8.5]{BC17} implies that \n$\\argmin\\theta =\\lev{\\inf\\theta(X)}\\theta$ is a nonempty closed convex subset of $X$.\n","start":49300,"end":49461}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"全部正参数的左右强制性","source_units":[{"raw_tex":"Now since $\\theta$ is coercive and since $D_f\\geq 0$, we immediately get that  \n\\eqref{e:bcoeR} and \\eqref{e:fcoeR} hold for all $\\gamma \\in \\RPP$.\n\n","start":49461,"end":49610}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":["e:bounded"],"declared_external":false,"source_only":false,"title":"左近端网的子水平有界性","source_units":[{"raw_tex":"\\ref{p:limprox+_left}: It follows from \\eqref{e:benv-bprox'} that   \n\\begin{equation}\n\\label{e:bounded}\n(\\forall \\gamma \\in \\RPP)\\quad \\bprox{\\gamma\\theta}(y) \\in \\lev{\\theta(y)}\\theta,\n\\end{equation}\nand then\nfrom the coercivity of $\\theta$ and \\cite[Proposition~11.12]{BC17} that $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ is bounded.\n","start":49610,"end":49950}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"聚点极小性与单点收敛","source_units":[{"raw_tex":"In turn, \\cref{p:limthetaprox}\\ref{p:limthetaprox_left} and \\cref{l:cvg} imply that\nall cluster points of $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ as $\\gamma \\uparrow +\\infty$ lie in $\\argmin\\theta$, \nand we get \\ref{p:limprox+_leftA}.\n\n","start":49950,"end":50192}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"内域聚点与参数子序列","source_units":[{"raw_tex":"Now assume that $\\argmin\\theta \\subseteq U$. \nLet $y'$ be a cluster point of $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ as $\\gamma \\uparrow +\\infty$.  \nThen $y' \\in \\argmin\\theta \\subseteq U$ and there exists a sequence $(\\gamma_n)_\\nnn$ in $\\RPP$ such that \n$\\gamma_n \\uparrow +\\infty$ and $\\bprox{\\gamma_n\\theta}(y) \\to y'$ as $n \\to +\\infty$.\nLet $z \\in \\argmin\\theta$.  \nWe have $(\\forall\\nnn)$ $\\theta(z) \\leq \\theta(\\bprox{\\gamma_n\\theta}(y))$, \n","start":50192,"end":50647}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"极小点比较消去目标差","source_units":[{"raw_tex":"and by Proposition~\\ref{p:prox}\\ref{p:prox_left},\n\\begin{equation}\n\\scal{\\nabla f(y) -\\nabla f\\big(\\bprox{\\gamma_n\\theta}(y)\\big)}{z -\\bprox{\\gamma_n\\theta}(y)} \n\\leq \\gamma_n\\Big(\\theta(z)\n-\\theta\\big(\\bprox{\\gamma_n\\theta}(y)\\big)\\Big) \\leq 0.\n\\end{equation}\n","start":50647,"end":50908}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"梯度不等式取极限","source_units":[{"raw_tex":"Taking the limit as $n \\to +\\infty$ and using the continuity of $\\nabla f$ yield\n\\begin{equation}\n\\scal{\\nabla f(y) -\\nabla f(y')}{z -y'} \\leq 0.\n\\end{equation}\n","start":50908,"end":51069}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"识别投影并得到全网收敛","source_units":[{"raw_tex":"Since $z \\in \\argmin\\theta$ was chosen arbitrarily \nand since $\\argmin\\theta$ is a closed convex subset of $X$ with \n$U\\cap \\argmin\\theta \\neq\\varnothing$, \nin view of \\cref{c:BregProj}\\ref{c:BregProj_left}, $y' =\\bproj{\\argmin\\theta}y$, \nand $\\bproj{\\argmin\\theta}y$ is thus the only cluster point of $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ \nas $\\gamma \\uparrow +\\infty$.\nHence, \\ref{p:limprox+_leftP} holds.\n","start":51069,"end":51485}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"右侧主结论的类比证明","source_units":[{"raw_tex":"\n\\ref{p:limprox+_right}: \nThe proof is similar to the one of \\ref{p:limprox+_left}.\n\\end{proof}","start":51485,"end":51580}]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件列表：BC17, Proposition 11.15(i)；原文 L1382 引用。"],"source_only":false,"title":"强制闭真凸函数取到极小值","source_units":[]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件列表：BC17, Lemma 1.24 and Corollary 8.5；原文 L1383 引用。"],"source_only":false,"title":"子水平集的闭性与凸性","source_units":[]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件列表：BC17, Proposition 11.12；原文 L1394 引用。"],"source_only":false,"title":"强制函数的有限子水平有界","source_units":[]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先决条件列表：Standard finite-dimensional compactness and cluster-point arguments for nets；这些省略背景由标注者显式化。"],"source_only":false,"title":"有限维紧性与网聚点背景","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S063","edges":[{"source_id":"S001","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S003","target_id":"S006","kind":"supports"},{"source_id":"S009","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S007","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S003","target_id":"S018","kind":"supports"},{"source_id":"S010","target_id":"S018","kind":"supports"},{"source_id":"S003","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S019","kind":"supports"},{"source_id":"S003","target_id":"S020","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S017","kind":"supports"},{"source_id":"S003","target_id":"S025","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S027","kind":"supports"},{"source_id":"S018","target_id":"S027","kind":"supports"},{"source_id":"S024","target_id":"S028","kind":"supports"},{"source_id":"S019","target_id":"S028","kind":"supports"},{"source_id":"S058","target_id":"S029","kind":"supports"},{"source_id":"S017","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S027","target_id":"S030","kind":"supports"},{"source_id":"S028","target_id":"S030","kind":"supports"},{"source_id":"S007","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S027","target_id":"S032","kind":"supports"},{"source_id":"S023","target_id":"S032","kind":"supports"},{"source_id":"S005","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S003","target_id":"S033","kind":"supports"},{"source_id":"S046","target_id":"S033","kind":"supports"},{"source_id":"S031","target_id":"S034","kind":"supports"},{"source_id":"S028","target_id":"S034","kind":"supports"},{"source_id":"S006","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S003","target_id":"S035","kind":"supports"},{"source_id":"S048","target_id":"S035","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S036","kind":"supports"},{"source_id":"S039","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S045","target_id":"S051","kind":"supports"},{"source_id":"S003","target_id":"S051","kind":"supports"},{"source_id":"S005","target_id":"S051","kind":"supports"},{"source_id":"S045","target_id":"S052","kind":"supports"},{"source_id":"S003","target_id":"S052","kind":"supports"},{"source_id":"S005","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S005","target_id":"S054","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S052","target_id":"S054","kind":"supports"},{"source_id":"S051","target_id":"S055","kind":"supports"},{"source_id":"S052","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S006","target_id":"S055","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S050","kind":"supports"},{"source_id":"S054","target_id":"S050","kind":"supports"},{"source_id":"S055","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S044","kind":"supports"},{"source_id":"S050","target_id":"S046","kind":"supports"},{"source_id":"S050","target_id":"S047","kind":"supports"},{"source_id":"S050","target_id":"S048","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S030","target_id":"S057","kind":"supports"},{"source_id":"S037","target_id":"S057","kind":"supports"},{"source_id":"S038","target_id":"S057","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S033","target_id":"S061","kind":"supports"},{"source_id":"S047","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S035","target_id":"S062","kind":"supports"},{"source_id":"S049","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S064","target_id":"S067","kind":"supports"},{"source_id":"S077","target_id":"S067","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S064","target_id":"S068","kind":"supports"},{"source_id":"S078","target_id":"S068","kind":"supports"},{"source_id":"S064","target_id":"S069","kind":"supports"},{"source_id":"S003","target_id":"S069","kind":"supports"},{"source_id":"S064","target_id":"S070","kind":"supports"},{"source_id":"S033","target_id":"S070","kind":"supports"},{"source_id":"S079","target_id":"S070","kind":"supports"},{"source_id":"S069","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S059","target_id":"S071","kind":"supports"},{"source_id":"S041","target_id":"S071","kind":"supports"},{"source_id":"S080","target_id":"S071","kind":"supports"},{"source_id":"S065","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S080","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S069","target_id":"S073","kind":"supports"},{"source_id":"S037","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S012","target_id":"S074","kind":"supports"},{"source_id":"S068","target_id":"S075","kind":"supports"},{"source_id":"S065","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S056","target_id":"S075","kind":"supports"},{"source_id":"S030","target_id":"S075","kind":"supports"},{"source_id":"S070","target_id":"S075","kind":"supports"},{"source_id":"S080","target_id":"S075","kind":"supports"},{"source_id":"S070","target_id":"S076","kind":"supports"},{"source_id":"S071","target_id":"S076","kind":"supports"},{"source_id":"S072","target_id":"S076","kind":"supports"},{"source_id":"S073","target_id":"S076","kind":"supports"},{"source_id":"S074","target_id":"S076","kind":"supports"},{"source_id":"S075","target_id":"S076","kind":"supports"},{"source_id":"S035","target_id":"S076","kind":"supports"},{"source_id":"S059","target_id":"S076","kind":"supports"},{"source_id":"S041","target_id":"S076","kind":"supports"},{"source_id":"S038","target_id":"S076","kind":"supports"},{"source_id":"S056","target_id":"S076","kind":"supports"},{"source_id":"S013","target_id":"S076","kind":"supports"},{"source_id":"S067","target_id":"S066","kind":"supports"},{"source_id":"S068","target_id":"S066","kind":"supports"},{"source_id":"S069","target_id":"S066","kind":"supports"},{"source_id":"S070","target_id":"S066","kind":"supports"},{"source_id":"S071","target_id":"S066","kind":"supports"},{"source_id":"S072","target_id":"S066","kind":"supports"},{"source_id":"S073","target_id":"S066","kind":"supports"},{"source_id":"S074","target_id":"S066","kind":"supports"},{"source_id":"S075","target_id":"S066","kind":"supports"},{"source_id":"S076","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S063","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S063","edges":[{"source_id":"S001","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S003","target_id":"S006","kind":"supports"},{"source_id":"S009","target_id":"S007","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S003","target_id":"S018","kind":"supports"},{"source_id":"S010","target_id":"S018","kind":"supports"},{"source_id":"S003","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S019","kind":"supports"},{"source_id":"S003","target_id":"S017","kind":"supports"},{"source_id":"S021","target_id":"S017","kind":"supports"},{"source_id":"S003","target_id":"S025","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S027","kind":"supports"},{"source_id":"S018","target_id":"S027","kind":"supports"},{"source_id":"S024","target_id":"S028","kind":"supports"},{"source_id":"S019","target_id":"S028","kind":"supports"},{"source_id":"S058","target_id":"S029","kind":"supports"},{"source_id":"S017","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S027","target_id":"S030","kind":"supports"},{"source_id":"S028","target_id":"S030","kind":"supports"},{"source_id":"S007","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S027","target_id":"S032","kind":"supports"},{"source_id":"S023","target_id":"S032","kind":"supports"},{"source_id":"S005","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S003","target_id":"S033","kind":"supports"},{"source_id":"S046","target_id":"S033","kind":"supports"},{"source_id":"S031","target_id":"S034","kind":"supports"},{"source_id":"S028","target_id":"S034","kind":"supports"},{"source_id":"S006","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S003","target_id":"S035","kind":"supports"},{"source_id":"S048","target_id":"S035","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S036","kind":"supports"},{"source_id":"S039","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S043","target_id":"S041","kind":"supports"},{"source_id":"S045","target_id":"S051","kind":"supports"},{"source_id":"S003","target_id":"S051","kind":"supports"},{"source_id":"S005","target_id":"S051","kind":"supports"},{"source_id":"S045","target_id":"S052","kind":"supports"},{"source_id":"S003","target_id":"S052","kind":"supports"},{"source_id":"S005","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S005","target_id":"S054","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S052","target_id":"S054","kind":"supports"},{"source_id":"S051","target_id":"S055","kind":"supports"},{"source_id":"S052","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S006","target_id":"S055","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S050","kind":"supports"},{"source_id":"S054","target_id":"S050","kind":"supports"},{"source_id":"S055","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S044","kind":"supports"},{"source_id":"S050","target_id":"S046","kind":"supports"},{"source_id":"S050","target_id":"S047","kind":"supports"},{"source_id":"S050","target_id":"S048","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S030","target_id":"S056","kind":"supports"},{"source_id":"S037","target_id":"S056","kind":"supports"},{"source_id":"S038","target_id":"S056","kind":"supports"},{"source_id":"S058","target_id":"S056","kind":"supports"},{"source_id":"S033","target_id":"S059","kind":"supports"},{"source_id":"S047","target_id":"S059","kind":"supports"},{"source_id":"S035","target_id":"S059","kind":"supports"},{"source_id":"S049","target_id":"S059","kind":"supports"},{"source_id":"S064","target_id":"S067","kind":"supports"},{"source_id":"S077","target_id":"S067","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S064","target_id":"S068","kind":"supports"},{"source_id":"S078","target_id":"S068","kind":"supports"},{"source_id":"S064","target_id":"S069","kind":"supports"},{"source_id":"S003","target_id":"S069","kind":"supports"},{"source_id":"S064","target_id":"S070","kind":"supports"},{"source_id":"S033","target_id":"S070","kind":"supports"},{"source_id":"S079","target_id":"S070","kind":"supports"},{"source_id":"S069","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S059","target_id":"S071","kind":"supports"},{"source_id":"S041","target_id":"S071","kind":"supports"},{"source_id":"S080","target_id":"S071","kind":"supports"},{"source_id":"S065","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S080","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S069","target_id":"S073","kind":"supports"},{"source_id":"S037","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S012","target_id":"S074","kind":"supports"},{"source_id":"S068","target_id":"S075","kind":"supports"},{"source_id":"S065","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S056","target_id":"S075","kind":"supports"},{"source_id":"S030","target_id":"S075","kind":"supports"},{"source_id":"S070","target_id":"S075","kind":"supports"},{"source_id":"S080","target_id":"S075","kind":"supports"},{"source_id":"S070","target_id":"S076","kind":"supports"},{"source_id":"S071","target_id":"S076","kind":"supports"},{"source_id":"S072","target_id":"S076","kind":"supports"},{"source_id":"S073","target_id":"S076","kind":"supports"},{"source_id":"S074","target_id":"S076","kind":"supports"},{"source_id":"S075","target_id":"S076","kind":"supports"},{"source_id":"S035","target_id":"S076","kind":"supports"},{"source_id":"S059","target_id":"S076","kind":"supports"},{"source_id":"S041","target_id":"S076","kind":"supports"},{"source_id":"S038","target_id":"S076","kind":"supports"},{"source_id":"S056","target_id":"S076","kind":"supports"},{"source_id":"S013","target_id":"S076","kind":"supports"},{"source_id":"S067","target_id":"S066","kind":"supports"},{"source_id":"S068","target_id":"S066","kind":"supports"},{"source_id":"S069","target_id":"S066","kind":"supports"},{"source_id":"S070","target_id":"S066","kind":"supports"},{"source_id":"S071","target_id":"S066","kind":"supports"},{"source_id":"S072","target_id":"S066","kind":"supports"},{"source_id":"S073","target_id":"S066","kind":"supports"},{"source_id":"S074","target_id":"S066","kind":"supports"},{"source_id":"S075","target_id":"S066","kind":"supports"},{"source_id":"S076","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S063","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"有限维欧氏空间","source_units":[{"raw_tex":"We assume throughout that\n\\begin{empheq}[box=\\mybluebox]{equation}\n\\text{$X := \\RR^J$,}\n\\end{empheq}\nwhich we equip with \nthe standard inner product $\\scal{\\cdot}{\\cdot}$ and \nthe induced Euclidean norm $\\|\\cdot\\|$.\n","start":7236,"end":7452}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"生成函数的初始条件","source_units":[{"raw_tex":"Now assume that \n\\begin{empheq}[box=\\mybluebox]{equation}\n\\text{$f \\colon X \\to\\left]-\\infty, +\\infty\\right]$ is convex and\ndifferentiable on $U := \\intdom f \\neq\\varnothing$.}\n\\end{empheq}\n","start":8646,"end":8836}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":["e:Df"],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"Bregman 距离","source_units":[{"raw_tex":"The \\emph{Bregman distance}\\footnote{Note that $D_f$ is\n\\emph{not} a distance in the sense of metric topology; however,\nthis naming convention is now ubiquitous.}  associated with $f$, \nfirst explored by Bregman in \\cite{Bre67} (see also\n\\cite{CZ97}), is \n\\begin{equation} \n\\label{e:Df}\nD_f \\colon X \\times X \\to \\left[0, +\\infty\\right] \\colon (x,y) \\mapsto\n\\begin{cases}\nf(x) -f(y) -\\scal{\\nabla f(y)}{x -y}, &\\text{~if~} y\\in U; \\\\\n+\\infty, &\\text{~otherwise}.\n\\end{cases}\n\\end{equation} \n","start":8836,"end":9327}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":["e:bba"],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"左 Bregman 包络","source_units":[{"raw_tex":"\\begin{equation}\n\\label{e:bba}\n\\benv[\\gamma]{\\theta} \\colon X\\to \\left[-\\infty, +\\infty\\right] \\colon\ny\\mapsto \\inf_{x\\in X} \\big(\\theta(x) +\\frac{1}{\\gamma}D_f(x,y)\\big)\n\\end{equation}\nand \n","start":10378,"end":10569}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["e:fba"],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"右 Bregman 包络","source_units":[{"raw_tex":"\\begin{equation}\n\\label{e:fba}\n\\fenv[\\gamma]{\\theta} \\colon X \\to \\left[-\\infty, +\\infty\\right] \\colon\nx \\mapsto\n\\inf_{y\\in X} \\big(\\theta(y) +\\frac{1}{\\gamma}D_f(x,y)\\big),\n\\end{equation}\nrespectively. \n","start":10569,"end":10773}]},{"id":"S007","semantic_type":"proposition","kind":"proposition","labels":["p:12.22i"],"member_ids":["S007","S008"],"declared_external":false,"source_only":false,"title":"包络的正数缩放","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{p:12.22i}\nLet $\\theta \\colon X \\to \\left]-\\infty, +\\infty\\right]$, \nlet $\\gamma \\in \\RPP$, and let $\\mu \\in \\RPP$.\nThen $\\benv[\\mu]{\\gamma\\theta} =\\gamma\\benv[\\gamma\\mu]{\\theta}$ \nand $\\fenv[\\mu]{\\gamma\\theta} =\\gamma\\fenv[\\gamma\\mu]{\\theta}$.\n\\end{proposition}","start":12824,"end":13112},{"raw_tex":"\\begin{proof}\nThis is analogous to the proof of \\cite[Proposition~12.22(i)]{BC17}.\n\\end{proof}\n","start":13113,"end":13208}]},{"id":"S009","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S009"],"declared_external":true,"declaration_text":["原始先决条件列表：BC17, Proposition 12.22(i)；原文 L395 仅按其证明类比。"],"source_only":false,"title":"经典包络缩放论证","source_units":[]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"闭真凸函数类","source_units":[{"raw_tex":"Denote by $\\Gamma_0(X)$ the set of all proper lower semicontinuous convex functions from $X$ to $\\left]-\\infty, +\\infty\\right]$. \n","start":16624,"end":16754}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"Legendre 站立假设","source_units":[{"raw_tex":"From now on, we strengthen our assumptions by requiring that \n\\begin{empheq}[box=\\mybluebox]{equation}\n\\label{e:fLegendre}\n\\text{$f\\in\\Gamma_0(X)$ is a convex function of Legendre type and\n$U := \\intdom f$.}\n\\end{empheq}\nThis will allow us to obtain a quite satisfying theory in which\nthe envelopes are convex functions. \nNote that $f$ is essentially smooth and essentially strictly convex in the sense of \\cite[Section~26]{Roc70}.\n","start":16754,"end":17186}]},{"id":"S012","semantic_type":"derived_fact","kind":"derived_fact","labels":["e:Leggy"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"梯度同胚","source_units":[{"raw_tex":"It is well known that\n\\begin{equation}\n\\label{e:Leggy}\n\\nabla f\\colon U\\to U^*:= \\intdom f^*\n\\text{~is a homeomorphism with~}\n\\big(\\nabla f\\big)^{-1} = \\nabla f^*.\n\\end{equation}\n","start":17186,"end":17365}]},{"id":"S013","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"连续 Hessian 假设","source_units":[{"raw_tex":"We will also work with the following standard assumptions:\n\\begin{itemize}\n\\item[\\bfseries A1] \n$\\nabla^2 f$ exists and is continuous on $U$;\n","start":17365,"end":17507}]},{"id":"S014","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"联合凸性假设","source_units":[{"raw_tex":"\\item[\\bfseries A2] \n$D_f$ is \\emph{jointly convex}, i.e., convex on $X\\times X$;\n","start":17507,"end":17589}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"右变量严格凸假设","source_units":[{"raw_tex":"\\item[\\bfseries A3] \n$(\\forall x\\in U)$ $D_f(x,\\cdot)$ is strictly convex on $U$;\n","start":17589,"end":17671}]},{"id":"S016","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"右变量强制性假设","source_units":[{"raw_tex":"\\item[\\bfseries A4] \n$(\\forall x\\in U)$ $D_f(x,\\cdot)$ is coercive, i.e., \n$D_f(x, y) \\to +\\infty$ as $\\|y\\| \\to +\\infty$.\n\\end{itemize}\nWe also henceforth assume that \n\\begin{empheq}[box=\\mybluebox]{equation}\n\\text{all assumptions \\textbf{A1}--\\textbf{A4} hold.}\n\\end{empheq}\n","start":17671,"end":17948}]},{"id":"S017","semantic_type":"proposition","kind":"proposition","labels":["e:bcoeR","e:brilliantissimo","e:fcoeR","f:coeR","l:coeRi","l:coeRii"],"member_ids":["S017","S020"],"declared_external":false,"source_only":false,"title":"正则化目标的强制性条件","source_units":[{"raw_tex":"\\begin{fact} \n\\label{f:coeR}\nLet $\\theta\\in\\Gamma_0(X)$ be such that $\\dom\\theta\\cap U\\neq\\varnothing$ and let $\\gamma \\in \\RPP$.\nConsider the following properties:\n\\begin{itemize}\n\\item[\\rm (a)]\n$U\\cap \\dom\\theta$ is bounded.\n\\item[\\rm (b)]\n$\\inf\\theta(U)>-\\infty$.\n\\item[\\rm (c)]\n$f$ is supercoercive, i.e., \n$f(x)/\\|x\\| \\to +\\infty$ as $\\|x\\| \\to +\\infty$.\n\\item[\\rm (d)]\n$(\\forall x\\in U)$ $D_f(x,\\cdot)$ is supercoercive.\n\\end{itemize}\nThen the following hold:\n\\begin{enumerate}\n\\item \\label{l:coeRi}\nIf any of the conditions (a), (b), or (c) holds, then \n\\begin{equation} \\label{e:bcoeR}\n(\\forall y \\in U) \\quad \\theta(\\cdot) + \\frac{1}{\\gamma}D_f(\\cdot,y) \\;\\;\n\\text{is coercive}\n\\end{equation}\nor, equivalently,\n\\begin{equation}\n\\frac{1}{\\gamma}\\ran\\nabla f \\subseteq \\intdom\\left(\\frac{1}{\\gamma}f +\\theta\\right)^*.\n\\label{e:brilliantissimo}\n\\end{equation}\n\n\\item \\label{l:coeRii}\nIf any of the conditions (a), (b), or (d) holds, then \n\\begin{equation} \\label{e:fcoeR}\n(\\forall x \\in U) \\quad \\theta(\\cdot) + \\frac{1}{\\gamma}D_f(x,\\cdot) \\;\\;\n\\text{is coercive.}\n\\end{equation}\n\\end{enumerate}\n\\end{fact}","start":21812,"end":22925},{"raw_tex":"\\begin{proof}\nSince $\\frac{1}{\\gamma}D_f =D_{\\frac{1}{\\gamma}f}$,\nthe result follows from \\cite[Lemma~2.12]{BCN06} \napplied to $\\tfrac{1}{\\gamma}f$.\n\\end{proof}\n","start":22926,"end":23087}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":["e:bcoeR","f:coeR","l:coeRi","l:coeRii"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"左强制性条件","source_units":[{"raw_tex":"\\begin{equation} \\label{e:bcoeR}\n(\\forall y \\in U) \\quad \\theta(\\cdot) + \\frac{1}{\\gamma}D_f(\\cdot,y) \\;\\;\n\\text{is coercive}\n\\end{equation}\n","start":22373,"end":22514}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":["e:fcoeR","f:coeR","l:coeRi","l:coeRii"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"右强制性条件","source_units":[{"raw_tex":"\\begin{equation} \\label{e:fcoeR}\n(\\forall x \\in U) \\quad \\theta(\\cdot) + \\frac{1}{\\gamma}D_f(x,\\cdot) \\;\\;\n\\text{is coercive.}\n\\end{equation}\n","start":22757,"end":22899}]},{"id":"S021","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S021"],"declared_external":true,"declaration_text":["原始先决条件列表：BCN06, Lemma 2.12；原文 L671 明确引用。"],"source_only":false,"title":"外部强制性引理","source_units":[]},{"id":"S022","semantic_type":"proposition","kind":"proposition","labels":["p:exact","p:exact_left","p:exact_right"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"包络性质与唯一取到","source_units":[{"raw_tex":"\\begin{proposition} \n\\label{p:exact} \nLet $\\theta \\colon X \\to \\left]-\\infty, +\\infty\\right]$ be convex\nand such that $U\\cap \\dom\\theta\\neq\\varnothing$, and let $\\gamma \\in \\RPP$.\nThen the following hold:\n\\begin{enumerate}\n\\item \n\\label{p:exact_left}\n$\\benv[\\gamma]{\\theta}$ is convex and continuous on $U$, and\n\t\\begin{enumerate}\n\t\\item \n\tif \\eqref{e:bcoeR} holds, i.e., $(\\forall y \\in U)$ $\\theta(\\cdot) +\\frac{1}{\\gamma}D_f(\\cdot,y)$ is coercive, \n\tthen $\\benv[\\gamma]{\\theta}$ is proper; \n\t\\item\n\tif $\\theta\\in\\Gamma_0(X)$ and $\\theta(\\cdot) +\\frac{1}{\\gamma}D_f(\\cdot,y)$ is coercive for a given $y\\in U$, \n\tthen there exists a unique point $z\\in U$ such that $\\benv[\\gamma]{\\theta}(y)=\\theta(z) +\\frac{1}{\\gamma}D_f(z,y)$.\n\t\\end{enumerate}\n\n\\item \n\\label{p:exact_right} \n$\\fenv[\\gamma]{\\theta}$ is convex and continuous on $U$, and\n\t\\begin{enumerate}\n\t\\item \n\tif \\eqref{e:fcoeR} holds, i.e., $(\\forall x \\in U)$ $\\theta(\\cdot) +\\frac{1}{\\gamma}D_f(x,\\cdot)$ is coercive, \n\tthen $\\fenv[\\gamma]{\\theta}$ is proper; \n\t\\item\n\tif $\\theta\\in\\Gamma_0(X)$ and $\\theta(\\cdot) +\\frac{1}{\\gamma}D_f(x,\\cdot)$ is coercive for a given $x\\in U$, \n\tthen there exists a unique point $z\\in U$ such that $\\fenv[\\gamma]{\\theta}(z)=\\theta(z) +\\frac{1}{\\gamma}D_f(x,z)$.\n\t\\end{enumerate}\n\\end{enumerate}\n\\end{proposition}","start":23247,"end":24554}]},{"id":"S023","semantic_type":"proposition","kind":"proposition","labels":["p:exact","p:exact_left","p:exact_right"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"左近端唯一取到","source_units":[{"raw_tex":"\tif $\\theta\\in\\Gamma_0(X)$ and $\\theta(\\cdot) +\\frac{1}{\\gamma}D_f(\\cdot,y)$ is coercive for a given $y\\in U$, \n\tthen there exists a unique point $z\\in U$ such that $\\benv[\\gamma]{\\theta}(y)=\\theta(z) +\\frac{1}{\\gamma}D_f(z,y)$.\n","start":23748,"end":23977}]},{"id":"S024","semantic_type":"proposition","kind":"proposition","labels":["p:exact","p:exact_left","p:exact_right"],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"右近端取到的原文表述","source_units":[{"raw_tex":"\tif $\\theta\\in\\Gamma_0(X)$ and $\\theta(\\cdot) +\\frac{1}{\\gamma}D_f(x,\\cdot)$ is coercive for a given $x\\in U$, \n\tthen there exists a unique point $z\\in U$ such that $\\fenv[\\gamma]{\\theta}(z)=\\theta(z) +\\frac{1}{\\gamma}D_f(x,z)$.\n","start":24275,"end":24504}]},{"id":"S025","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S025"],"declared_external":false,"source_only":true,"title":"唯一取到命题的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nSince $\\frac{1}{\\gamma}D_f =D_{\\frac{1}{\\gamma}f}$,\nthe result follows from\n \\cite[Propositions~3.4 and 3.5]{BCN06}\napplied to $\\tfrac{1}{\\gamma}f$.\n\\end{proof}\n","start":24555,"end":24730}]},{"id":"S026","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S026"],"declared_external":true,"declaration_text":["原始先决条件列表：BCN06, Propositions 3.4 and 3.5；原文 L714 引用。"],"source_only":false,"title":"外部包络与取到命题","source_units":[]},{"id":"S027","semantic_type":"definition","kind":"definition","labels":["d:prox","e:dbprox"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"左近端映射","source_units":[{"raw_tex":"\\begin{definition}[Bregman proximity operators] \n\\label{d:prox}\nLet $\\theta\\in\\Gamma_0(X)$ be such that \n$U\\cap \\dom\\theta\\neq\\varnothing$.\nIf \\eqref{e:bcoeR} holds for $\\gamma =1$, then \nthe \\emph{left proximity operator} associated with $\\theta$ is\n\\begin{equation} \\label{e:dbprox}\n\\bprox{\\theta} \\colon U \\to U \\colon\ny\\mapsto\n\\underset{x\\in X}{\\operatorname{argmin}}\\;\\: \\big(\\theta(x) +D_f(x,y)\\big).\n\\end{equation}\n","start":24842,"end":25264}]},{"id":"S028","semantic_type":"definition","kind":"definition","labels":["e:dfprox"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"右近端映射","source_units":[{"raw_tex":"If \\eqref{e:fcoeR} holds for $\\gamma =1$, then \nthe \\emph{right proximity operator} associated with $\\theta$ is\n\\begin{equation} \\label{e:dfprox}\n\\fprox{\\theta}\\colon U \\to  U \\colon\nx \\mapsto\n\\underset{y\\in X}{\\operatorname{argmin}}\\;\\: \\big(\\theta(y) +D_f(x,y)\\big).\n\\end{equation}\n\\end{definition}\n","start":25264,"end":25565}]},{"id":"S029","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"指标函数保证投影适定","source_units":[{"raw_tex":"Given a closed convex subset $C$ of $X$ with $C\\cap U \\neq\\varnothing$, we have that $\\iota_C \\in \\Gamma_0(X)$, $\\dom\\iota_C =C$, \nand hence $U\\cap \\dom\\iota_C =U\\cap C \\neq\\varnothing$ and also $\\inf \\iota_C(U) =0 >-\\infty$, \nwhich together with \\cref{f:coeR} imply that \\eqref{e:bcoeR} and \\eqref{e:fcoeR} hold for all $\\gamma \\in \\RPP$. \nThis leads to the following definition. \n","start":26129,"end":26511}]},{"id":"S030","semantic_type":"definition","kind":"definition","labels":["d:BregProj"],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"左右 Bregman 投影","source_units":[{"raw_tex":"\n\\begin{definition}[Bregman projectors]\n\\label{d:BregProj}\nLet $C$ be a closed convex subset of $X$ such that \n$U\\cap C\\neq\\varnothing$. \nThen $\\bproj{C} :=\\bprox{\\iota_C}$ is the \\emph{left Bregman projector} onto $C$\nand $\\fproj{C} :=\\fprox{\\iota_C}$ is the \\emph{right Bregman projector} onto $C$.\n\\end{definition}\n","start":26511,"end":26829}]},{"id":"S031","semantic_type":"derived_fact","kind":"derived_fact","labels":["e:benv-bproxall","e:fenv-fproxall","r:prox"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"参数与目标缩放换算","source_units":[{"raw_tex":"\\begin{remark}\n\\label{r:prox}\nLet $\\theta\\in\\Gamma_0(X)$ be such that \n$U\\cap \\dom\\theta\\neq\\varnothing$ and let $\\gamma \\in \\RPP$.\n\\cref{p:12.22i} implies that \n$\\benv{\\gamma\\theta} =\\gamma\\benv[\\gamma]{\\theta}$ \nand $\\fenv{\\gamma\\theta} =\\gamma\\fenv[\\gamma]{\\theta}$. \n","start":29283,"end":29554}]},{"id":"S032","semantic_type":"derived_fact","kind":"derived_fact","labels":["e:benv-bprox","e:benv-bproxall","e:fenv-fproxall","r:prox"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"左包络的近端分解","source_units":[{"raw_tex":"We thus derive from the definition that if \\eqref{e:bcoeR} holds, then\n\\begin{subequations}\n\\label{e:benv-bproxall}\n\\begin{equation}\n\\label{e:benv-bprox}\n\\benv[\\gamma]{\\theta}(y) =\\theta\\big(\\bprox{\\gamma\\theta}(y)\\big) \n+\\frac{1}{\\gamma}D_f\\big(\\bprox{\\gamma\\theta}(y), y\\big)\n\\end{equation}\n","start":29554,"end":29847}]},{"id":"S033","semantic_type":"derived_fact","kind":"derived_fact","labels":["e:benv-bprox'","e:benv-bproxall","e:fenv-fproxall","r:prox"],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"左近端目标值夹逼","source_units":[{"raw_tex":"and, by combining with \\cref{p:12.9}\\ref{p:12.9_left},\n\\begin{equation}\n\\label{e:benv-bprox'}\n\\theta\\big(\\bprox{\\gamma\\theta}(y)\\big) \\leq \\benv[\\gamma]{\\theta}(y) \\leq \\theta(y).\n\\end{equation}\n\\end{subequations}\n","start":29847,"end":30061}]},{"id":"S034","semantic_type":"derived_fact","kind":"derived_fact","labels":["e:benv-bproxall","e:fenv-fprox","e:fenv-fproxall","r:prox"],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"右包络的近端分解","source_units":[{"raw_tex":"Similarly, if \\eqref{e:fcoeR} holds, then\n\\begin{subequations}\n\\label{e:fenv-fproxall}\n\\begin{equation}\n\\label{e:fenv-fprox}\n\\fenv[\\gamma]{\\theta}(x) =\\theta\\big(\\fprox{\\gamma\\theta}(x)\\big) +\n\\frac{1}{\\gamma}D_f\\big(x, \\fprox{\\gamma\\theta}(x)\\big)\n\\end{equation}\n","start":30061,"end":30325}]},{"id":"S035","semantic_type":"derived_fact","kind":"derived_fact","labels":["e:benv-bproxall","e:fenv-fproxall","r:prox"],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"右近端目标值夹逼","source_units":[{"raw_tex":"and \n\\begin{equation}\n\\theta\\big(\\fprox{\\gamma\\theta}(x)\\big) \\leq \\fenv[\\gamma]{\\theta}(x) \\leq \\theta(x).\n\\end{equation}\n\\end{subequations}\n\\end{remark}","start":30325,"end":30479}]},{"id":"S036","semantic_type":"proposition","kind":"proposition","labels":["e:pro","p:prox","p:prox_left","p:prox_right"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"近端映射的变分刻画","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{p:prox}\nLet $\\theta\\in\\Gamma_0(X)$ be such that $\\dom\\theta\\cap U\\neq\\varnothing$ and let $\\gamma \\in \\RPP$.\n\\begin{enumerate}\n\\item \n\\label{p:prox_left}\nSuppose that \\eqref{e:bcoeR} holds.\nThen for every $(x,y) \\in U\\times U$, \nthe following conditions are equivalent:\n\t\\begin{enumerate}\n\t\\item \n\t$x =\\bprox{\\gamma\\theta}(y)$,\n\t\\item \n\t$0 \\in \\gamma\\partial\\theta(x) +\\nabla f(x) -\\nabla f(y)$,\n\t\\item \n\t$(\\forall z \\in X)\\quad \\scal{\\nabla f(y) -\\nabla f(x)}{z -x} +\\gamma\\theta(x) \\leq \\gamma\\theta(z)$.\n\t\\end{enumerate}\nMoreover, \n\\begin{equation} \\label{e:pro}\n\\bprox{\\gamma\\theta}=(\\nabla f +\\gamma\\partial\\theta)^{-1}\\circ \\nabla f \n\\end{equation}\nis continuous on $U$. \n\\item \n\\label{p:prox_right}\nSuppose that \\eqref{e:fcoeR} holds. \nThen for every $(x,y) \\in U\\times U$, \nthe following conditions are equivalent:\n\t\\begin{enumerate}\n\t\\item \n\t$y =\\fprox{\\gamma\\theta}(x)$,\n\t\\item \n\t$0 \\in \\gamma\\partial\\theta(y) +\\nabla^2 f(y)(y -x)$,\n\t\\item \n\t$(\\forall z \\in X)\\quad \\scal{\\nabla^2 f(y)(x -y)}{z -y} +\\gamma\\theta(y) \\leq \\gamma\\theta(z)$.\n\t\\end{enumerate}\nMoreover, $\\fprox{\\gamma\\theta}$ is continuous on $U$. \n\\end{enumerate}\n\\end{proposition}","start":31727,"end":32910}]},{"id":"S037","semantic_type":"proposition","kind":"proposition","labels":["p:prox","p:prox_left","p:prox_right"],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"左近端变分不等式","source_units":[{"raw_tex":"Suppose that \\eqref{e:bcoeR} holds.\nThen for every $(x,y) \\in U\\times U$, \nthe following conditions are equivalent:\n\t\\begin{enumerate}\n\t\\item \n\t$x =\\bprox{\\gamma\\theta}(y)$,\n\t\\item \n\t$0 \\in \\gamma\\partial\\theta(x) +\\nabla f(x) -\\nabla f(y)$,\n\t\\item \n\t$(\\forall z \\in X)\\quad \\scal{\\nabla f(y) -\\nabla f(x)}{z -x} +\\gamma\\theta(x) \\leq \\gamma\\theta(z)$.\n","start":31908,"end":32261}]},{"id":"S038","semantic_type":"proposition","kind":"proposition","labels":["p:prox","p:prox_left","p:prox_right"],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"右近端变分不等式","source_units":[{"raw_tex":"Suppose that \\eqref{e:fcoeR} holds. \nThen for every $(x,y) \\in U\\times U$, \nthe following conditions are equivalent:\n\t\\begin{enumerate}\n\t\\item \n\t$y =\\fprox{\\gamma\\theta}(x)$,\n\t\\item \n\t$0 \\in \\gamma\\partial\\theta(y) +\\nabla^2 f(y)(y -x)$,\n\t\\item \n\t$(\\forall z \\in X)\\quad \\scal{\\nabla^2 f(y)(x -y)}{z -y} +\\gamma\\theta(y) \\leq \\gamma\\theta(z)$.\n","start":32460,"end":32804}]},{"id":"S039","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S039"],"declared_external":false,"source_only":true,"title":"近端刻画的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nApply \\cite[Proposition~3.10]{BCN06} to $\\gamma\\theta$.\n\\end{proof}\n","start":32911,"end":32993}]},{"id":"S040","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S040"],"declared_external":true,"declaration_text":["原始先决条件列表：BCN06, Proposition 3.10；原文 L956 引用。"],"source_only":false,"title":"外部近端刻画命题","source_units":[]},{"id":"S041","semantic_type":"lemma","kind":"lemma","labels":["l:cvg"],"member_ids":["S041","S042"],"declared_external":false,"source_only":false,"title":"紧集内极小化网的聚点","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{l:cvg}\nLet $C$ be a compact subset of a Hausdorff space $\\mathcal X$,\nlet $\\phi\\colon \\mathcal X \\to \\left[-\\infty, +\\infty\\right]$ be lower \nsemicontinuous,\nlet $(x_a)_{a\\in A}$ be a net in $C$,\nand suppose that $\\phi(x_a) \\to \\inf\\phi(\\mathcal X)$. \nThen $\\argmin\\phi \\neq\\varnothing$ \nand all cluster points of $(x_a)_{a\\in A}$ lie in $\\argmin\\phi$.\nConsequently, if $\\phi$ attains its minimum at a unique point $u$, then $x_a \\to u$.\n\\end{lemma}","start":40270,"end":40740},{"raw_tex":"\\begin{proof}\nThis follows from the lower semicontinuity of $\\phi$ and \\cite[Lemma 1.14]{BC17}.\n\\end{proof}\n","start":40741,"end":40849}]},{"id":"S043","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S043"],"declared_external":true,"declaration_text":["原始先决条件列表：BC17, Lemma 1.14；原文 L1169 引用。"],"source_only":false,"title":"紧性论证的外部引理","source_units":[]},{"id":"S044","semantic_type":"proposition","kind":"proposition","labels":["p:12.9","p:12.9_left","p:12.9_right"],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"包络域与大参数极限","source_units":[{"raw_tex":"\\begin{proposition} \n\\label{p:12.9} \nLet $\\theta \\colon X \\to \\left]-\\infty, +\\infty\\right]$ be such that $U\\cap \\dom\\theta\\neq\\varnothing$ and let $\\gamma \\in \\RPP$.\nThen the following hold:\n\\begin{enumerate}\n\\item \n\\label{p:12.9_left}\n$\\dom\\benv[\\gamma]{\\theta}=U$, \nand $(\\forall y\\in U)(\\forall\\mu \\in \\left]\\gamma, +\\infty\\right[)$ \n$\\inf \\theta(X) \\leq \\benv[\\mu]{\\theta}(y) \\leq \\benv[\\gamma]{\\theta}(y) \\leq \\theta(y)$.\nConsequently, $\\inf \\theta(X) \\leq \\inf \\benv[\\gamma]{\\theta}(X) \\leq \\inf \\theta(U)$, \nand $\\benv[\\gamma]{\\theta}(y) \\downarrow \\inf \\theta(X)$ as $\\gamma \\uparrow +\\infty$.\n\\item \n\\label{p:12.9_right}\n$\\dom\\fenv[\\gamma]{\\theta}=\\dom f$, \nand $(\\forall x \\in U)(\\forall\\mu \\in \\left]\\gamma, +\\infty\\right[)$ \n$\\inf \\theta(X) \\leq \\fenv[\\mu]{\\theta}(x) \\leq \\fenv[\\gamma]{\\theta}(x) \\leq \\theta(x)$.\nConsequently, $\\inf \\theta(X) \\leq \\inf \\fenv[\\gamma]{\\theta}(X) \\leq \\inf \\theta(U)$, \nand $\\fenv[\\gamma]{\\theta}(x) \\downarrow \\inf \\theta(X)$ as $\\gamma \\uparrow +\\infty$.\n\\end{enumerate}\n\\end{proposition}","start":13355,"end":14391}]},{"id":"S045","semantic_type":"assumption","kind":"assumption","labels":["p:12.9","p:12.9_left","p:12.9_right"],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"早期包络命题的局部假设","source_units":[{"raw_tex":"Let $\\theta \\colon X \\to \\left]-\\infty, +\\infty\\right]$ be such that $U\\cap \\dom\\theta\\neq\\varnothing$ and let $\\gamma \\in \\RPP$.\n","start":13392,"end":13522}]},{"id":"S046","semantic_type":"proposition","kind":"proposition","labels":["p:12.9","p:12.9_left","p:12.9_right"],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"左包络的域与序界","source_units":[{"raw_tex":"\\item \n\\label{p:12.9_left}\n$\\dom\\benv[\\gamma]{\\theta}=U$, \nand $(\\forall y\\in U)(\\forall\\mu \\in \\left]\\gamma, +\\infty\\right[)$ \n$\\inf \\theta(X) \\leq \\benv[\\mu]{\\theta}(y) \\leq \\benv[\\gamma]{\\theta}(y) \\leq \\theta(y)$.\nConsequently, $\\inf \\theta(X) \\leq \\inf \\benv[\\gamma]{\\theta}(X) \\leq \\inf \\theta(U)$, \n","start":13565,"end":13871}]},{"id":"S047","semantic_type":"proposition","kind":"proposition","labels":["p:12.9","p:12.9_left","p:12.9_right"],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"左包络的大参数极限","source_units":[{"raw_tex":"and $\\benv[\\gamma]{\\theta}(y) \\downarrow \\inf \\theta(X)$ as $\\gamma \\uparrow +\\infty$.\n","start":13871,"end":13958}]},{"id":"S048","semantic_type":"proposition","kind":"proposition","labels":["p:12.9","p:12.9_left","p:12.9_right"],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"右包络的域与序界","source_units":[{"raw_tex":"\\item \n\\label{p:12.9_right}\n$\\dom\\fenv[\\gamma]{\\theta}=\\dom f$, \nand $(\\forall x \\in U)(\\forall\\mu \\in \\left]\\gamma, +\\infty\\right[)$ \n$\\inf \\theta(X) \\leq \\fenv[\\mu]{\\theta}(x) \\leq \\fenv[\\gamma]{\\theta}(x) \\leq \\theta(x)$.\nConsequently, $\\inf \\theta(X) \\leq \\inf \\fenv[\\gamma]{\\theta}(X) \\leq \\inf \\theta(U)$, \n","start":13958,"end":14271}]},{"id":"S049","semantic_type":"proposition","kind":"proposition","labels":["p:12.9","p:12.9_left","p:12.9_right"],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"右包络的大参数极限","source_units":[{"raw_tex":"and $\\fenv[\\gamma]{\\theta}(x) \\downarrow \\inf \\theta(X)$ as $\\gamma \\uparrow +\\infty$.\n","start":14271,"end":14358}]},{"id":"S050","semantic_type":"proof","kind":"proof","labels":["e:170428a"],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"包络域与极限的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{p:12.9_left}: We first show that $\\dom\\benv[\\gamma]{\\theta}=U$.\nLet $y \\in \\dom\\benv[\\gamma]{\\theta}$. \nThen $\\benv[\\gamma]{\\theta}(y) =\\inf_{x\\in X} \\big(\\theta(x) +\\frac{1}{\\gamma}D_f(x,y)\\big) <+\\infty$,\nand hence there exists $x \\in X$ such that $\\theta(x) +\\frac{1}{\\gamma}D_f(x,y) <+\\infty$.\nSince $\\theta(x) >-\\infty$, this yields $y \\in U$.\n\nFrom now on, let $y \\in U$, and pick $u \\in \\dom\\theta \\cap U$. \nThen $-f(y) <+\\infty$, $\\|\\nabla f(y)\\| <+\\infty$, $f(u) <+\\infty$, $\\theta(u) <+\\infty$, and\n\\begin{subequations}\n\\begin{align}\n\\benv[\\gamma]{\\theta}(y) &=\\inf_{x\\in X} \\Big(\\theta(x) +\\frac{1}{\\gamma}\\big(f(x) -f(y) -\\scal{\\nabla f(y)}{x -y}\\big)\\Big), \\\\ \n&\\leq \\theta(u) +\\frac{1}{\\gamma}\\big(f(u) -f(y) -\\scal{\\nabla f(y)}{u -y}\\big) <+\\infty,\n\\end{align}\n\\end{subequations} \nwhich gives $y \\in \\dom\\benv[\\gamma]{\\theta}$. Hence, $\\dom \\benv[\\gamma]{\\theta} =U$. \n\nNext, let $\\mu \\in \\left]\\gamma, +\\infty\\right[$. \nThen $\\frac{1}{\\mu} < \\frac{1}{\\gamma}$, \n$\\theta \\leq \\theta +\\frac{1}{\\mu}D_f(\\cdot,y) \\leq \\theta +\\frac{1}{\\gamma}D_f(\\cdot,y)$, and so\n\\begin{equation}\n\\inf_{x \\in X} \\theta(x) \\leq \\benv[\\mu]{\\theta}(y) \\leq \\benv[\\gamma]{\\theta}(y) =\\inf_{x \\in X} \\big(\\theta(x) +\\frac{1}{\\gamma}D_f(x,y)\\big)\n\\leq \\theta(y) +\\frac{1}{\\gamma}D_f(y,y) =\\theta(y). \n\\end{equation}\nTherefore,\n\\begin{equation}\n\\label{e:170428a}\n\\inf \\theta(X) \\leq \\benv[\\mu]{\\theta}(y) \\leq \\benv[\\gamma]{\\theta}(y) \\leq \\theta(y).\n\\end{equation}\nTaking now the infimum over $y\\in U$ yields\n$\\inf \\theta(X) \\leq \\inf \\benv[\\gamma]{\\theta}(X) \\leq \\inf \\theta(U)$.\nConsequently, \n\\begin{equation}\n\\inf \\theta(X) \\leq \\varliminf_{\\gamma \\to+\\infty}\n\\benv[\\gamma]{\\theta}(y).\n\\end{equation}\nOn the other hand, $(\\forall x \\in X)$ $\\benv[\\gamma]{\\theta}(y) \\leq \\theta(x) +\\frac{1}{\\gamma}D_f(x,y)$, \nwhich implies that $(\\forall x \\in X)$ $\\varlimsup_{\\gamma\n\\to+\\infty} \\benv[\\gamma]{\\theta}(y) \\leq \\theta(x)$ and thus\n$\\varlimsup_{\\gamma \\to+\\infty} \\benv[\\gamma]{\\theta}(y) \\leq\n\\inf \\theta(X)$. \nAltogether,\n$\\lim_{\\gamma\\to+\\infty} \\benv[\\gamma]{\\theta}(y) =\n\\inf \\theta(X)$ and the conclusion follows from\n\\eqref{e:170428a}. \n\n\\ref{p:12.9_right}: This is similar to \\ref{p:12.9_left}. \n\\end{proof}","start":14392,"end":16620}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"确定左包络的域","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{p:12.9_left}: We first show that $\\dom\\benv[\\gamma]{\\theta}=U$.\nLet $y \\in \\dom\\benv[\\gamma]{\\theta}$. \nThen $\\benv[\\gamma]{\\theta}(y) =\\inf_{x\\in X} \\big(\\theta(x) +\\frac{1}{\\gamma}D_f(x,y)\\big) <+\\infty$,\nand hence there exists $x \\in X$ such that $\\theta(x) +\\frac{1}{\\gamma}D_f(x,y) <+\\infty$.\nSince $\\theta(x) >-\\infty$, this yields $y \\in U$.\n\nFrom now on, let $y \\in U$, and pick $u \\in \\dom\\theta \\cap U$. \nThen $-f(y) <+\\infty$, $\\|\\nabla f(y)\\| <+\\infty$, $f(u) <+\\infty$, $\\theta(u) <+\\infty$, and\n\\begin{subequations}\n\\begin{align}\n\\benv[\\gamma]{\\theta}(y) &=\\inf_{x\\in X} \\Big(\\theta(x) +\\frac{1}{\\gamma}\\big(f(x) -f(y) -\\scal{\\nabla f(y)}{x -y}\\big)\\Big), \\\\ \n&\\leq \\theta(u) +\\frac{1}{\\gamma}\\big(f(u) -f(y) -\\scal{\\nabla f(y)}{u -y}\\big) <+\\infty,\n\\end{align}\n\\end{subequations} \nwhich gives $y \\in \\dom\\benv[\\gamma]{\\theta}$. Hence, $\\dom \\benv[\\gamma]{\\theta} =U$. \n","start":14392,"end":15295}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":["e:170428a"],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"建立参数单调性与序界","source_units":[{"raw_tex":"\nNext, let $\\mu \\in \\left]\\gamma, +\\infty\\right[$. \nThen $\\frac{1}{\\mu} < \\frac{1}{\\gamma}$, \n$\\theta \\leq \\theta +\\frac{1}{\\mu}D_f(\\cdot,y) \\leq \\theta +\\frac{1}{\\gamma}D_f(\\cdot,y)$, and so\n\\begin{equation}\n\\inf_{x \\in X} \\theta(x) \\leq \\benv[\\mu]{\\theta}(y) \\leq \\benv[\\gamma]{\\theta}(y) =\\inf_{x \\in X} \\big(\\theta(x) +\\frac{1}{\\gamma}D_f(x,y)\\big)\n\\leq \\theta(y) +\\frac{1}{\\gamma}D_f(y,y) =\\theta(y). \n\\end{equation}\nTherefore,\n\\begin{equation}\n\\label{e:170428a}\n\\inf \\theta(X) \\leq \\benv[\\mu]{\\theta}(y) \\leq \\benv[\\gamma]{\\theta}(y) \\leq \\theta(y).\n\\end{equation}\nTaking now the infimum over $y\\in U$ yields\n$\\inf \\theta(X) \\leq \\inf \\benv[\\gamma]{\\theta}(X) \\leq \\inf \\theta(U)$.\n","start":15295,"end":15983}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"极限下界","source_units":[{"raw_tex":"Consequently, \n\\begin{equation}\n\\inf \\theta(X) \\leq \\varliminf_{\\gamma \\to+\\infty}\n\\benv[\\gamma]{\\theta}(y).\n\\end{equation}\n","start":15983,"end":16107}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"任意点比较与极限结论","source_units":[{"raw_tex":"On the other hand, $(\\forall x \\in X)$ $\\benv[\\gamma]{\\theta}(y) \\leq \\theta(x) +\\frac{1}{\\gamma}D_f(x,y)$, \nwhich implies that $(\\forall x \\in X)$ $\\varlimsup_{\\gamma\n\\to+\\infty} \\benv[\\gamma]{\\theta}(y) \\leq \\theta(x)$ and thus\n$\\varlimsup_{\\gamma \\to+\\infty} \\benv[\\gamma]{\\theta}(y) \\leq\n\\inf \\theta(X)$. \nAltogether,\n$\\lim_{\\gamma\\to+\\infty} \\benv[\\gamma]{\\theta}(y) =\n\\inf \\theta(X)$ and the conclusion follows from\n\\eqref{e:170428a}. \n","start":16107,"end":16549}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"右包络证明的类比声明","source_units":[{"raw_tex":"\n\\ref{p:12.9_right}: This is similar to \\ref{p:12.9_left}. \n\\end{proof}","start":16549,"end":16620}]},{"id":"S056","semantic_type":"corollary","kind":"corollary","labels":["c:BregProj","c:BregProj_left","c:BregProj_right"],"member_ids":["S056","S057"],"declared_external":false,"source_only":false,"title":"Bregman 投影的变分刻画","source_units":[{"raw_tex":"\\begin{corollary}\n\\label{c:BregProj}\nLet $C$ be a closed convex subset of $X$ such that \n$U\\cap C\\neq\\varnothing$, \nlet $(x,y) \\in U\\times U$, \nand let $p \\in U\\cap C$.\nThen the following hold:\n\\begin{enumerate}\n\\item \n\\label{c:BregProj_left}\n$p =\\bproj{C}y \\ \\Leftrightarrow \\ (\\forall z \\in C)\\; \n\\scal{\\nabla f(y) -\\nabla f(p)}{z -p} \\leq 0$. \n\\item \n\\label{c:BregProj_right}\n$p =\\fproj{C}x \\ \\Leftrightarrow \\ (\\forall z \\in C)\\; \n\\scal{\\nabla^2 f(p)(x -p)}{z -p} \\leq 0$. \n\\end{enumerate}\n\\end{corollary}","start":34473,"end":34982},{"raw_tex":"\\begin{proof}\nIn light of \\cref{d:BregProj}, we \napply \\cref{p:prox} \n(see also \\cite[Proposition~3.16]{BB97}).\n\\end{proof}","start":34983,"end":35106}]},{"id":"S058","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S058"],"declared_external":true,"declaration_text":["原始先决条件列表：Standard indicator-function and convex subdifferential conventions。"],"source_only":false,"title":"指标函数与次微分约定","source_units":[]},{"id":"S059","semantic_type":"proposition","kind":"proposition","labels":["p:limthetaprox","p:limthetaprox_left","p:limthetaprox_right"],"member_ids":["S059","S060","S061","S062"],"declared_external":false,"source_only":false,"title":"大参数近端目标值极限","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{p:limthetaprox}\nLet $\\theta\\in\\Gamma_0(X)$ be such that $U\\cap \\dom\\theta\\neq\\varnothing$ and let $x, y \\in U$.\nThen the following hold:\n\\begin{enumerate}\n\\item \n\\label{p:limthetaprox_left}\nIf \\eqref{e:bcoeR} holds for all $\\gamma\\in\\RPP$, \nthen $\\theta(\\bprox{\\gamma\\theta}(y)) \\to \\inf \\theta(X)$ as $\\gamma \\uparrow +\\infty$.\n\\item \n\\label{p:limthetaprox_right}\nIf \\eqref{e:fcoeR} holds for all $\\gamma\\in \\RPP$, then  \n$\\theta(\\fprox{\\gamma\\theta}(x)) \\to \\inf \\theta(X)$ as $\\gamma \\uparrow +\\infty$.\n\\end{enumerate}\t\n\\end{proposition}","start":46810,"end":47377},{"raw_tex":"\\begin{proof}\nWe shall just prove \\ref{p:limthetaprox_left} \nbecause the proof of \\ref{p:limthetaprox_right} is similar.\n","start":47378,"end":47499},{"raw_tex":"\\begin{proof}\nWe shall just prove \\ref{p:limthetaprox_left} \nbecause the proof of \\ref{p:limthetaprox_right} is similar.\nAssume that \\eqref{e:bcoeR} holds for all \n$\\gamma \\in \\RPP$.\nCombining \\eqref{e:benv-bprox'} with \\cref{p:12.9}\\ref{p:12.9_left} yields\n\\begin{equation}\n\\inf\\theta(X) \\leq \\theta\\big(\\bprox{\\gamma\\theta}(y)\\big) \\leq \\benv[\\gamma]{\\theta}(y) \\to \\inf\\theta(X) \n\\quad\\text{as}\\quad \\gamma \\uparrow +\\infty,\n\\end{equation} \nwhich implies that $\\theta(\\bprox{\\gamma\\theta}(y)) \\to \n\\inf\\theta(X)$ as $\\gamma \\uparrow +\\infty$. \n\\end{proof}","start":47378,"end":47936},{"raw_tex":"Assume that \\eqref{e:bcoeR} holds for all \n$\\gamma \\in \\RPP$.\nCombining \\eqref{e:benv-bprox'} with \\cref{p:12.9}\\ref{p:12.9_left} yields\n\\begin{equation}\n\\inf\\theta(X) \\leq \\theta\\big(\\bprox{\\gamma\\theta}(y)\\big) \\leq \\benv[\\gamma]{\\theta}(y) \\to \\inf\\theta(X) \n\\quad\\text{as}\\quad \\gamma \\uparrow +\\infty,\n\\end{equation} \nwhich implies that $\\theta(\\bprox{\\gamma\\theta}(y)) \\to \n\\inf\\theta(X)$ as $\\gamma \\uparrow +\\infty$. \n","start":47499,"end":47925},{"raw_tex":"\\end{proof}","start":47925,"end":47936}]},{"id":"S063","semantic_type":"theorem","kind":"theorem","labels":["p:limprox+","p:limprox+_left","p:limprox+_leftA","p:limprox+_leftP","p:limprox+_right","p:limprox+_rightP"],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"大参数近端点趋向极小点与投影","source_units":[{"raw_tex":"\\begin{theorem}\n\\label{p:limprox+}\nLet $\\theta\\in\\Gamma_0(X)$ be coercive such that \n$U\\cap \\dom\\theta\\neq\\varnothing$ and let $x, y \\in U$.\nThen the following hold:\n\\begin{enumerate}\n\\item \n\\label{p:limprox+_left}\nThe net $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ is bounded \nwith all cluster points as $\\gamma \\uparrow +\\infty$ lying in $\\argmin\\theta$. \nMoreover,\n\t\\begin{enumerate}\n\t\\item\\label{p:limprox+_leftA} \n\tif $\\argmin\\theta$ is a singleton, \n\tthen $\\bprox{\\gamma\\theta}(y) \\to \\argmin\\theta$ as $\\gamma \\uparrow +\\infty$;\n\t\\item\\label{p:limprox+_leftP} \n\tif $\\argmin\\theta \\subseteq U$, \n\tthen $\\bprox{\\gamma\\theta}(y) \\to \\bproj{\\argmin\\theta}y$ as $\\gamma \\uparrow +\\infty$. \n\t\\end{enumerate}\n\n\\item \n\\label{p:limprox+_right}\nThe net $(\\fprox{\\gamma\\theta}(x))_{\\gamma \\in \\RPP}$ is bounded \nwith all cluster points as $\\gamma \\uparrow +\\infty$ lying in $\\argmin\\theta$. \nMoreover,\n\t\\begin{enumerate}\n\t\\item \n\tif $\\argmin\\theta$ is a singleton, \n\tthen $\\fprox{\\gamma\\theta}(x) \\to \\argmin\\theta$ as $\\gamma \\uparrow +\\infty$;\n\t\\item\\label{p:limprox+_rightP} \n\tif $\\argmin\\theta \\subseteq U$, \n\tthen $\\fprox{\\gamma\\theta}(x) \\to \\fproj{\\argmin\\theta}x$ as $\\gamma \\uparrow +\\infty$.  \n\t\\end{enumerate}\n\\end{enumerate}\t\n\\end{theorem}","start":47939,"end":49189}]},{"id":"S064","semantic_type":"assumption","kind":"assumption","labels":["p:limprox+","p:limprox+_left","p:limprox+_leftA","p:limprox+_leftP","p:limprox+_right","p:limprox+_rightP"],"member_ids":["S064"],"declared_external":false,"source_only":false,"title":"主定理共同假设","source_units":[{"raw_tex":"Let $\\theta\\in\\Gamma_0(X)$ be coercive such that \n$U\\cap \\dom\\theta\\neq\\varnothing$ and let $x, y \\in U$.\n","start":47974,"end":48080}]},{"id":"S065","semantic_type":"assumption","kind":"assumption","labels":["p:limprox+","p:limprox+_left","p:limprox+_leftA","p:limprox+_leftP","p:limprox+_right","p:limprox+_rightP"],"member_ids":["S065"],"declared_external":false,"source_only":false,"title":"投影极限的内域条件","source_units":[{"raw_tex":"\t\\item\\label{p:limprox+_leftP} \n\tif $\\argmin\\theta \\subseteq U$, \n\tthen $\\bprox{\\gamma\\theta}(y) \\to \\bproj{\\argmin\\theta}y$ as $\\gamma \\uparrow +\\infty$. \n","start":48478,"end":48634},{"raw_tex":"\t\\item\\label{p:limprox+_rightP} \n\tif $\\argmin\\theta \\subseteq U$, \n\tthen $\\fprox{\\gamma\\theta}(x) \\to \\fproj{\\argmin\\theta}x$ as $\\gamma \\uparrow +\\infty$.  \n","start":48984,"end":49142}]},{"id":"S066","semantic_type":"proof","kind":"proof","labels":["e:bounded"],"member_ids":["S066"],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nFirst, by assumption, \\cite[Proposition~11.15(i)]{BC17} gives $\\argmin\\theta \\neq\\varnothing$. \nThis combined with \\cite[Lemma~1.24 and Corollary~8.5]{BC17} implies that \n$\\argmin\\theta =\\lev{\\inf\\theta(X)}\\theta$ is a nonempty closed convex subset of $X$.\nNow since $\\theta$ is coercive and since $D_f\\geq 0$, we immediately get that  \n\\eqref{e:bcoeR} and \\eqref{e:fcoeR} hold for all $\\gamma \\in \\RPP$.\n\n\\ref{p:limprox+_left}: It follows from \\eqref{e:benv-bprox'} that   \n\\begin{equation}\n\\label{e:bounded}\n(\\forall \\gamma \\in \\RPP)\\quad \\bprox{\\gamma\\theta}(y) \\in \\lev{\\theta(y)}\\theta,\n\\end{equation}\nand then\nfrom the coercivity of $\\theta$ and \\cite[Proposition~11.12]{BC17} that $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ is bounded.\nIn turn, \\cref{p:limthetaprox}\\ref{p:limthetaprox_left} and \\cref{l:cvg} imply that\nall cluster points of $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ as $\\gamma \\uparrow +\\infty$ lie in $\\argmin\\theta$, \nand we get \\ref{p:limprox+_leftA}.\n\nNow assume that $\\argmin\\theta \\subseteq U$. \nLet $y'$ be a cluster point of $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ as $\\gamma \\uparrow +\\infty$.  \nThen $y' \\in \\argmin\\theta \\subseteq U$ and there exists a sequence $(\\gamma_n)_\\nnn$ in $\\RPP$ such that \n$\\gamma_n \\uparrow +\\infty$ and $\\bprox{\\gamma_n\\theta}(y) \\to y'$ as $n \\to +\\infty$.\nLet $z \\in \\argmin\\theta$.  \nWe have $(\\forall\\nnn)$ $\\theta(z) \\leq \\theta(\\bprox{\\gamma_n\\theta}(y))$, \nand by Proposition~\\ref{p:prox}\\ref{p:prox_left},\n\\begin{equation}\n\\scal{\\nabla f(y) -\\nabla f\\big(\\bprox{\\gamma_n\\theta}(y)\\big)}{z -\\bprox{\\gamma_n\\theta}(y)} \n\\leq \\gamma_n\\Big(\\theta(z)\n-\\theta\\big(\\bprox{\\gamma_n\\theta}(y)\\big)\\Big) \\leq 0.\n\\end{equation}\nTaking the limit as $n \\to +\\infty$ and using the continuity of $\\nabla f$ yield\n\\begin{equation}\n\\scal{\\nabla f(y) -\\nabla f(y')}{z -y'} \\leq 0.\n\\end{equation}\nSince $z \\in \\argmin\\theta$ was chosen arbitrarily \nand since $\\argmin\\theta$ is a closed convex subset of $X$ with \n$U\\cap \\argmin\\theta \\neq\\varnothing$, \nin view of \\cref{c:BregProj}\\ref{c:BregProj_left}, $y' =\\bproj{\\argmin\\theta}y$, \nand $\\bproj{\\argmin\\theta}y$ is thus the only cluster point of $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ \nas $\\gamma \\uparrow +\\infty$.\nHence, \\ref{p:limprox+_leftP} holds.\n\n\\ref{p:limprox+_right}: \nThe proof is similar to the one of \\ref{p:limprox+_left}.\n\\end{proof}","start":49190,"end":51580}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S067"],"declared_external":false,"source_only":false,"title":"极小点存在","source_units":[{"raw_tex":"\\begin{proof}\nFirst, by assumption, \\cite[Proposition~11.15(i)]{BC17} gives $\\argmin\\theta \\neq\\varnothing$. \n","start":49190,"end":49300}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S068"],"declared_external":false,"source_only":false,"title":"极小点集非空闭凸","source_units":[{"raw_tex":"This combined with \\cite[Lemma~1.24 and Corollary~8.5]{BC17} implies that \n$\\argmin\\theta =\\lev{\\inf\\theta(X)}\\theta$ is a nonempty closed convex subset of $X$.\n","start":49300,"end":49461}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S069"],"declared_external":false,"source_only":false,"title":"全部正参数的左右强制性","source_units":[{"raw_tex":"Now since $\\theta$ is coercive and since $D_f\\geq 0$, we immediately get that  \n\\eqref{e:bcoeR} and \\eqref{e:fcoeR} hold for all $\\gamma \\in \\RPP$.\n\n","start":49461,"end":49610}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":["e:bounded"],"member_ids":["S070"],"declared_external":false,"source_only":false,"title":"左近端网的子水平有界性","source_units":[{"raw_tex":"\\ref{p:limprox+_left}: It follows from \\eqref{e:benv-bprox'} that   \n\\begin{equation}\n\\label{e:bounded}\n(\\forall \\gamma \\in \\RPP)\\quad \\bprox{\\gamma\\theta}(y) \\in \\lev{\\theta(y)}\\theta,\n\\end{equation}\nand then\nfrom the coercivity of $\\theta$ and \\cite[Proposition~11.12]{BC17} that $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ is bounded.\n","start":49610,"end":49950}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S071"],"declared_external":false,"source_only":false,"title":"聚点极小性与单点收敛","source_units":[{"raw_tex":"In turn, \\cref{p:limthetaprox}\\ref{p:limthetaprox_left} and \\cref{l:cvg} imply that\nall cluster points of $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ as $\\gamma \\uparrow +\\infty$ lie in $\\argmin\\theta$, \nand we get \\ref{p:limprox+_leftA}.\n\n","start":49950,"end":50192}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S072"],"declared_external":false,"source_only":false,"title":"内域聚点与参数子序列","source_units":[{"raw_tex":"Now assume that $\\argmin\\theta \\subseteq U$. \nLet $y'$ be a cluster point of $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ as $\\gamma \\uparrow +\\infty$.  \nThen $y' \\in \\argmin\\theta \\subseteq U$ and there exists a sequence $(\\gamma_n)_\\nnn$ in $\\RPP$ such that \n$\\gamma_n \\uparrow +\\infty$ and $\\bprox{\\gamma_n\\theta}(y) \\to y'$ as $n \\to +\\infty$.\nLet $z \\in \\argmin\\theta$.  \nWe have $(\\forall\\nnn)$ $\\theta(z) \\leq \\theta(\\bprox{\\gamma_n\\theta}(y))$, \n","start":50192,"end":50647}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S073"],"declared_external":false,"source_only":false,"title":"极小点比较消去目标差","source_units":[{"raw_tex":"and by Proposition~\\ref{p:prox}\\ref{p:prox_left},\n\\begin{equation}\n\\scal{\\nabla f(y) -\\nabla f\\big(\\bprox{\\gamma_n\\theta}(y)\\big)}{z -\\bprox{\\gamma_n\\theta}(y)} \n\\leq \\gamma_n\\Big(\\theta(z)\n-\\theta\\big(\\bprox{\\gamma_n\\theta}(y)\\big)\\Big) \\leq 0.\n\\end{equation}\n","start":50647,"end":50908}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S074"],"declared_external":false,"source_only":false,"title":"梯度不等式取极限","source_units":[{"raw_tex":"Taking the limit as $n \\to +\\infty$ and using the continuity of $\\nabla f$ yield\n\\begin{equation}\n\\scal{\\nabla f(y) -\\nabla f(y')}{z -y'} \\leq 0.\n\\end{equation}\n","start":50908,"end":51069}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S075"],"declared_external":false,"source_only":false,"title":"识别投影并得到全网收敛","source_units":[{"raw_tex":"Since $z \\in \\argmin\\theta$ was chosen arbitrarily \nand since $\\argmin\\theta$ is a closed convex subset of $X$ with \n$U\\cap \\argmin\\theta \\neq\\varnothing$, \nin view of \\cref{c:BregProj}\\ref{c:BregProj_left}, $y' =\\bproj{\\argmin\\theta}y$, \nand $\\bproj{\\argmin\\theta}y$ is thus the only cluster point of $(\\bprox{\\gamma\\theta}(y))_{\\gamma \\in \\RPP}$ \nas $\\gamma \\uparrow +\\infty$.\nHence, \\ref{p:limprox+_leftP} holds.\n","start":51069,"end":51485}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S076"],"declared_external":false,"source_only":false,"title":"右侧主结论的类比证明","source_units":[{"raw_tex":"\n\\ref{p:limprox+_right}: \nThe proof is similar to the one of \\ref{p:limprox+_left}.\n\\end{proof}","start":51485,"end":51580}]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S077"],"declared_external":true,"declaration_text":["原始先决条件列表：BC17, Proposition 11.15(i)；原文 L1382 引用。"],"source_only":false,"title":"强制闭真凸函数取到极小值","source_units":[]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S078"],"declared_external":true,"declaration_text":["原始先决条件列表：BC17, Lemma 1.24 and Corollary 8.5；原文 L1383 引用。"],"source_only":false,"title":"子水平集的闭性与凸性","source_units":[]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S079"],"declared_external":true,"declaration_text":["原始先决条件列表：BC17, Proposition 11.12；原文 L1394 引用。"],"source_only":false,"title":"强制函数的有限子水平有界","source_units":[]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S080"],"declared_external":true,"declaration_text":["原始先决条件列表：Standard finite-dimensional compactness and cluster-point arguments for nets；这些省略背景由标注者显式化。"],"source_only":false,"title":"有限维紧性与网聚点背景","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S005":"S005","S006":"S006","S007":"S007","S008":"S007","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S017","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S041","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S056","S058":"S058","S059":"S059","S060":"S059","S061":"S059","S062":"S059","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076","S077":"S077","S078":"S078","S079":"S079","S080":"S080"}}}}
{"record_id":"2205.11093v2","paper":{"title":"Accelerated Minimax Algorithms Flock Together","authors":["Yoon, TaeHo","Ryu, Ernest K."],"arxiv_id":"2205.11093","version":"v2","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/2205.11093v2","pdf_url":"https://arxiv.org/pdf/2205.11093v2","doi":"10.1137/22M1504597"},"context":{"raw_tex":"Minimax optimization problems of the form\n\\begin{align}\n\\label{eqn:minimax_problem}\n    \\underset{x \\in \\cX}{\\textrm{minimize}} \\,\\, \\underset{y \\in \\cY}{\\textrm{maximize}} \\,\\,\\,\\, \\lagrange (x,y)\n\\end{align}\n\n\nA function $\\lagrange(\\cdot, \\cdot): \\cX \\times \\cY \\to \\reals$ is \\emph{convex-concave} if $\\cX$ and $\\cY$ are convex sets, $\\lagrange(x, y)$ is convex as a function of $x$ for all $y$, and $\\lagrange(x, y)$ is concave as a function of $y$ for all $x$.\nA point $(x_\\star, y_\\star) \\in \\cX \\times \\cY$ is a \\emph{saddle point}, or \\emph{(minimax) solution} of~\\eqref{eqn:minimax_problem} if $\\lagrange(x_\\star, y) \\le \\lagrange(x_\\star, y_\\star) \\le \\lagrange(x, y_\\star)$ for all $(x,y) \\in \\cX \\times \\cY$.\n\nAn operator $\\opA$ on $\\reals^d$, denoted $\\opA\\colon \\reals^d \\rightrightarrows \\reals^d$, is a set-valued function, i.e., $\\opA(z)\\subseteq\\reals^d$ for all $z \\in \\reals^d$.\nFor simplicity, we often write $\\opA z = \\opA(z)$.\nIf $\\opA z$ contains exactly one element for all $z\\in\\reals^d$, then we say $\\opA$ is \\emph{single-valued} and view it as a function.\nAn operator $\\opA\\colon \\reals^d \\rightrightarrows \\reals^d$ is \\emph{monotone} if\n\\begin{align*}\n    \\langle \\opA z - \\opA z', z - z' \\rangle \\ge 0, \\qquad \\forall \\, z,z'\\in\\reals^d ,\n\\end{align*}\nwhere the notation means that $\\langle u - v, z - z' \\rangle \\ge 0$ for any $u \\in \\opA z$ and $v \\in \\opA z'$.\nFor $\\mu > 0$, an operator $\\opA$ is $\\mu$-\\emph{strongly monotone} if\n\\begin{align*}\n    \\langle \\opA z - \\opA z', z - z' \\rangle \\ge \\mu \\|z-z'\\|^2, \\qquad \\forall \\, z,z'\\in\\reals^d .\n\\end{align*}\n\n\nFor differentiable and convex-concave $\\lagrange\\colon\\reals^n \\times \\reals^m\\rightarrow\\reals$, we define\n\\[\n    \\sop\\lagrange (x,y) = (\\nabla_x \\lagrange (x,y), -\\nabla_y \\lagrange (x,y))\n\\]\nand call it the \\emph{saddle operator} of $\\lagrange$.\nThen $\\sop \\lagrange$ is monotone \\cite{Rockafellar1970_monotone}, and $z_\\star = (x_\\star, y_\\star)$ is a solution of~\\eqref{eqn:minimax_problem} if and only if $\\sop \\lagrange (z_\\star) = 0$.\n\n\nWe design the novel algorithm \\emph{\\textbf{S}trongly \\textbf{M}onotone \\textbf{E}xtra \\textbf{A}nchored \\textbf{G}radient\\textbf{\\texttt{+}}}\\footnote{The ``\\texttt{+}'' follows the nomenclature of \\cite{DiakonikolasDaskalakisJordan2021_efficient}, where the two-time-scale EG was named EG\\texttt{+}.} \n\\begin{align}\n\\label{eqn:SM-EAG+}\n\\begin{aligned}\n    z_{k+1/2} & = \\beta_k z_0 + (1-\\beta_k) z_k - \\eta_k \\alpha \\opB z_k \\\\\n    z_{k+1}   & = \\beta_k z_0 + (1-\\beta_k) z_k - \\alpha \\opB z_{k+1/2} ,\n\\end{aligned}\n\\tag{SM-EAG\\texttt{+}}\n\\end{align}\nwhere $\\opB=\\sop\\lagrange$, $\\beta_k = \\frac{1}{\\sum_{j=0}^k (1+2\\alpha\\mu)^j}$, $\\eta_k = \\frac{1-\\beta_k}{1+2\\alpha\\mu}$, and $0 < \\alpha \\le \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2}$, to approximate OC-Halpern in the MP sense. \\ref{eqn:SM-EAG+} inherits the accelerated rate of OC-Halpern and achieves the fastest known rate in the setup with respect to the gradient norm, improving upon the prior rates of EG and OG by factors of 8 and 4, respectively.\n(OC-Halpern is not a forward algorithm as it uses the proximal operator $\\JB$, while \\ref{eqn:SM-EAG+} is a forward algorithm using evaluations of $\\opB$.)\n","units":[{"raw_tex":"Minimax optimization problems of the form\n\\begin{align}\n\\label{eqn:minimax_problem}\n    \\underset{x \\in \\cX}{\\textrm{minimize}} \\,\\, \\underset{y \\in \\cY}{\\textrm{maximize}} \\,\\,\\,\\, \\lagrange (x,y)\n\\end{align}\n","start":14562,"end":14772},{"raw_tex":"A function $\\lagrange(\\cdot, \\cdot): \\cX \\times \\cY \\to \\reals$ is \\emph{convex-concave} if $\\cX$ and $\\cY$ are convex sets, $\\lagrange(x, y)$ is convex as a function of $x$ for all $y$, and $\\lagrange(x, y)$ is concave as a function of $y$ for all $x$.\nA point $(x_\\star, y_\\star) \\in \\cX \\times \\cY$ is a \\emph{saddle point}, or \\emph{(minimax) solution} of~\\eqref{eqn:minimax_problem} if $\\lagrange(x_\\star, y) \\le \\lagrange(x_\\star, y_\\star) \\le \\lagrange(x, y_\\star)$ for all $(x,y) \\in \\cX \\times \\cY$.\n\nAn operator $\\opA$ on $\\reals^d$, denoted $\\opA\\colon \\reals^d \\rightrightarrows \\reals^d$, is a set-valued function, i.e., $\\opA(z)\\subseteq\\reals^d$ for all $z \\in \\reals^d$.\nFor simplicity, we often write $\\opA z = \\opA(z)$.\nIf $\\opA z$ contains exactly one element for all $z\\in\\reals^d$, then we say $\\opA$ is \\emph{single-valued} and view it as a function.\nAn operator $\\opA\\colon \\reals^d \\rightrightarrows \\reals^d$ is \\emph{monotone} if\n\\begin{align*}\n    \\langle \\opA z - \\opA z', z - z' \\rangle \\ge 0, \\qquad \\forall \\, z,z'\\in\\reals^d ,\n\\end{align*}\nwhere the notation means that $\\langle u - v, z - z' \\rangle \\ge 0$ for any $u \\in \\opA z$ and $v \\in \\opA z'$.\nFor $\\mu > 0$, an operator $\\opA$ is $\\mu$-\\emph{strongly monotone} if\n\\begin{align*}\n    \\langle \\opA z - \\opA z', z - z' \\rangle \\ge \\mu \\|z-z'\\|^2, \\qquad \\forall \\, z,z'\\in\\reals^d .\n\\end{align*}\n","start":28121,"end":29505},{"raw_tex":"For differentiable and convex-concave $\\lagrange\\colon\\reals^n \\times \\reals^m\\rightarrow\\reals$, we define\n\\[\n    \\sop\\lagrange (x,y) = (\\nabla_x \\lagrange (x,y), -\\nabla_y \\lagrange (x,y))\n\\]\nand call it the \\emph{saddle operator} of $\\lagrange$.\nThen $\\sop \\lagrange$ is monotone \\cite{Rockafellar1970_monotone}, and $z_\\star = (x_\\star, y_\\star)$ is a solution of~\\eqref{eqn:minimax_problem} if and only if $\\sop \\lagrange (z_\\star) = 0$.\n","start":31511,"end":31954},{"raw_tex":"We design the novel algorithm \\emph{\\textbf{S}trongly \\textbf{M}onotone \\textbf{E}xtra \\textbf{A}nchored \\textbf{G}radient\\textbf{\\texttt{+}}}\\footnote{The ``\\texttt{+}'' follows the nomenclature of \\cite{DiakonikolasDaskalakisJordan2021_efficient}, where the two-time-scale EG was named EG\\texttt{+}.} \n\\begin{align}\n\\label{eqn:SM-EAG+}\n\\begin{aligned}\n    z_{k+1/2} & = \\beta_k z_0 + (1-\\beta_k) z_k - \\eta_k \\alpha \\opB z_k \\\\\n    z_{k+1}   & = \\beta_k z_0 + (1-\\beta_k) z_k - \\alpha \\opB z_{k+1/2} ,\n\\end{aligned}\n\\tag{SM-EAG\\texttt{+}}\n\\end{align}\nwhere $\\opB=\\sop\\lagrange$, $\\beta_k = \\frac{1}{\\sum_{j=0}^k (1+2\\alpha\\mu)^j}$, $\\eta_k = \\frac{1-\\beta_k}{1+2\\alpha\\mu}$, and $0 < \\alpha \\le \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2}$, to approximate OC-Halpern in the MP sense. \\ref{eqn:SM-EAG+} inherits the accelerated rate of OC-Halpern and achieves the fastest known rate in the setup with respect to the gradient norm, improving upon the prior rates of EG and OG by factors of 8 and 4, respectively.\n(OC-Halpern is not a forward algorithm as it uses the proximal operator $\\JB$, while \\ref{eqn:SM-EAG+} is a forward algorithm using evaluations of $\\opB$.)\n","start":40779,"end":41941}]},"statement":{"raw_tex":"\\begin{theorem}[Fast rate of \\ref{eqn:SM-EAG+}]\n\\label{thm:sm-EAG}\nLet $\\opB\\colon \\reals^d \\to \\reals^d$ be $\\mu$-strongly monotone and $L$-Lipschitz with $0 < \\mu \\le L$.\nLet $z_\\star$ be the zero of $\\opB$.\nFor $\\alpha \\in \\big( 0, \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2} \\big]$, SM-EAG\\texttt{+} exhibits the rate\n\\begin{align*}\n    \\|\\opB z_k\\|^2 \\le \\frac{\\left(\\sqrt{1+2\\alpha\\mu} + 1\\right)^2}{\\alpha^2 \\left( \\sum_{j=0}^{k-1} (1+2\\alpha\\mu)^{\\frac{j}{2}} \\right)^2} \\|z_0 - z_\\star\\|^2 ,\n    \\qquad\\text{for } k=1,2,\\dots.\n\\end{align*}\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}[Fast rate of \\ref{eqn:SM-EAG+}]\n\\label{thm:sm-EAG}\nLet $\\opB\\colon \\reals^d \\to \\reals^d$ be $\\mu$-strongly monotone and $L$-Lipschitz with $0 < \\mu \\le L$.\nLet $z_\\star$ be the zero of $\\opB$.\nFor $\\alpha \\in \\big( 0, \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2} \\big]$, SM-EAG\\texttt{+} exhibits the rate\n\\begin{align*}\n    \\|\\opB z_k\\|^2 \\le \\frac{\\left(\\sqrt{1+2\\alpha\\mu} + 1\\right)^2}{\\alpha^2 \\left( \\sum_{j=0}^{k-1} (1+2\\alpha\\mu)^{\\frac{j}{2}} \\right)^2} \\|z_0 - z_\\star\\|^2 ,\n    \\qquad\\text{for } k=1,2,\\dots.\n\\end{align*}\n\\end{theorem}","start":44629,"end":45184}]},"proof":{"raw_tex":"\\begin{lemma}\n\\label{lem:sm-EAG_Lyapunov}\nLet $\\opB\\colon \\reals^d \\to \\reals^d$ be a $\\mu$-strongly monotone, $L$-Lipschitz operator ($0 < \\mu \\le L$), and let\n$\\{z_k\\}_{k=0,1,\\dots}$ be the iterates of \\ref{eqn:SM-EAG+} with $\\beta_k = \\frac{1}{\\sum_{j=0}^k (1+2\\alpha\\mu)^j}$.\nThen, the quantities\n\\begin{align*}\n    V_k = p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k - \\mu (z_k - z_0), z_k - z_0 \\rangle + \\left( \\frac{1}{2\\alpha} + \\mu \\right) \\|z_0-z_\\star\\|^2\n\\end{align*}\nwith $p_0 = q_0 = 0$ and $q_k = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{1}{\\beta_{k-1}}$, $p_k = \\frac{\\eta_k \\alpha q_k}{2 \\beta_k}$ for $k=1,2,\\dots$ satisfy $V_k \\ge 0$, $V_0 - V_1 \\ge \\frac{\\alpha (1+2\\alpha\\mu-\\alpha^2 L^2)}{2}\\|\\opB z_0\\|^2$, and\n\\begin{align*}\n    V_k - V_{k+1} \\ge \\frac{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2) q_k}{2\\beta_k (1-\\beta_k)} \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2\n\\end{align*}\nfor $k=1,2,\\dots$.\nIn particular, if $\\alpha \\in \\big( 0, \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2} \\big]$, $V_k$ is a Lyapunov function for the algorithm, and if $\\alpha \\in \\big( 0, \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2} \\big)$, we additionally have the summability result\n\\begin{align}\n    \\sum_{k=0}^\\infty (1+2\\alpha\\mu)^k \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2 \\le \\frac{1+2\\alpha\\mu}{\\alpha^2 (1 + 2\\alpha\\mu - \\alpha^2 L^2)} \\|z_0 - z_\\star\\|^2  < \\infty .\n    \\label{eqn:sm-EAG_summability}\n\\end{align}\n\\end{lemma}\n\n\\subsection{Proof of \\texorpdfstring{\\cref{thm:sm-EAG}}{Theorem 4.2}}\nBecause the term $\\left( \\frac{1}{2\\alpha}+\\mu \\right) \\|z_0-z_\\star\\|^2$ subtracted from $V_k$ to obtain $W_k$ does not depend on $k$, clearly $W_k$ inherits nonincreasingness from $V_k$, i.e., $W_k \\le \\cdots \\le W_0 = 0$.\nTherefore, given that $q_k < \\lambda < q_k + 4\\mu p_k$, we have\n\\begin{align}\n    & \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 \\ge W_k + \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 \\ge \\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right) \\|\\opB z_k\\|^2 \\nonumber \\\\\n    & \\implies \\|\\opB z_k\\|^2 \\le \\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right)^{-1} \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 . \\label{eqn:sm-EAG_convergence_rate}\n\\end{align}\nOne can straightforwardly verify that \n\\begin{align}\n    \\lambda_\\star = (q_k (q_k + 4\\mu p_k))^{1/2}\n    \\label{eqn:sm-EAG_lambda}\n\\end{align}\nlies between $q_k$ and $q_k + 4\\mu p_k$, and minimizes the factor $\\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right)^{-1} \\frac{q_k \\lambda \\mu}{\\lambda - q_k}$ in the right hand side of \\eqref{eqn:sm-EAG_convergence_rate}.\nPlugging in the explicit expressions $q_k = \\frac{(1+2\\alpha\\mu)^k - 1}{2\\alpha\\mu (1+2\\alpha\\mu)^{k-1}}$ and $p_k = \\frac{\\alpha}{2}(1+2\\alpha\\mu)^{k-1} q_k^2 = \\frac{((1+2\\alpha\\mu)^k - 1)^2}{8\\alpha\\mu^2 (1+2\\alpha\\mu)^{k-1}}$ (which uses \\eqref{eqn:sm-EAG_pkqk}) into~\\eqref{eqn:sm-EAG_lambda}, we obtain $\\lambda_\\star = (1+2\\alpha\\mu)^{\\frac{k}{2}} q_k$ and\n\\begin{align*}\n    \\left( p_k - \\frac{\\lambda_\\star - q_k}{4\\mu} \\right)^{-1} \\frac{q_k \\lambda_\\star \\mu}{\\lambda_\\star - q_k} & = \\frac{4\\mu^2}{((1+2\\alpha\\mu)^{\\frac{k}{2}} - 1)^2} \n    = \\frac{(\\sqrt{1+2\\alpha\\mu}+1)^2}{\\alpha^2 \\left( \\sum_{j=0}^{k-1} (1+2\\alpha\\mu)^{\\frac{j}{2}} \\right)^2} .\n\\end{align*}\nTherefore, \\eqref{eqn:sm-EAG_convergence_rate} with $\\lambda = \\lambda_\\star$ concludes the proof.\n\n\n\n\n\n\\subsection{Proof of \\texorpdfstring{\\cref{lem:sm-EAG_Lyapunov}}{Lemma 4.4}}\nWhen $k=0$, we have $z_{1/2} = z_0$, $z_1 = z_0 - \\alpha \\opB z_0$, $V_0 = \\left( \\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2$, and\n\\begin{align*}\n    V_1 & = \\frac{\\alpha}{2}\\|\\opB z_1\\|^2 + \\langle \\opB z_1 - \\mu(z_1 - z_0), z_1 - z_0 \\rangle + \\left(\\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2 \\nonumber \\\\\n    & = \\frac{\\alpha}{2}\\|\\opB z_1\\|^2 - \\alpha \\langle \\opB z_1, \\opB z_0 \\rangle - \\alpha^2 \\mu \\|\\opB z_0\\|^2 + \\left(\\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2 . \n\\end{align*}\nBecause $\\opB$ is $L$-Lipschitz,\n\\begin{align*}\n    0 & \\le L^2 \\|z_1 - z_0\\|^2 - \\|\\opB z_1 - \\opB z_0\\|^2 = (\\alpha^2 L^2 - 1) \\|\\opB z_0\\|^2 + 2\\langle \\opB z_0, \\opB z_1 \\rangle - \\|\\opB z_1\\|^2 .\n\\end{align*}\nTherefore,\n\\begin{align*}\n    V_0 - V_1 & = \\alpha^2 \\mu \\|\\opB z_0\\|^2 + \\alpha \\langle \\opB z_0, \\opB z_1 \\rangle - \\frac{\\alpha}{2} \\|\\opB z_1\\|^2 \\\\\n    & = \\frac{\\alpha}{2} (1+2\\alpha\\mu -\\alpha^2 L^2) \\|\\opB z_0\\|^2 + \\frac{\\alpha}{2} \\left( (\\alpha^2 L^2 - 1) \\|\\opB z_0\\|^2 + 2\\langle \\opB z_0, \\opB z_1 \\rangle - \\|\\opB z_1\\|^2 \\right) \\\\\n    & \\ge \\frac{\\alpha}{2} (1+2\\alpha\\mu -\\alpha^2 L^2) \\|\\opB z_0\\|^2 \\\\\n    & = \\frac{\\alpha}{2} (1+2\\alpha\\mu -\\alpha^2 L^2) \\|\\eta_0 \\opB z_0 - \\opB z_{1/2}\\|^2 \n\\end{align*}\nwhere the last equality holds because $\\eta_0 = 0$ and $z_{1/2} = z_0$.\n\nNow suppose $k\\ge 1$. For \\ref{eqn:SM-EAG+}, we have the identities\n\\begin{align}\n    z_{k+1} - z_0 & = (1-\\beta_k) (z_k - z_0) - \\alpha \\opB z_{k+1/2} \\label{eqn:sm_EAG+_zkp1_minus_z0} \\\\\n    z_k - z_{k+1} & = \\beta_k (z_k - z_0) + \\alpha \\opB z_{k+1/2} \\label{eqn:SM-EAG+_zk_minus_zkp1} \\\\\n    z_{k+1/2} - z_{k+1} & = \\alpha \\opB z_{k+1/2} - \\eta_k \\alpha \\opB z_k . \\label{eqn:sm_EAG+_zkhalf_minus_zkp1}\n\\end{align}\nNote that $\\beta_k = \\frac{1}{\\sum_{j=0}^{k} (1+2\\alpha\\mu)^j}$ satisfies\n\\begin{align*}\n    \\frac{1}{\\beta_{k}} = \\sum_{j=0}^{k} (1+2\\alpha\\mu)^j = 1 + (1+2\\alpha\\mu) \\sum_{j=0}^{k-1} (1+2\\alpha\\mu)^j = 1 + \\frac{1+2\\alpha\\mu}{\\beta_{k-1}} \n\\end{align*}\nfor $k=1,2,\\dots$, so $q_k = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{1}{\\beta_{k-1}}$ satisfies the recurrence relation\n\\begin{align}\n    q_{k+1} = \\frac{q_k}{\\beta_k} \\frac{\\beta_{k-1}}{(1+2\\alpha\\mu)} =\n    \\frac{q_k}{\\beta_k} \\frac{1}{\\frac{1}{\\beta_k} - 1} = \\frac{q_k}{1-\\beta_k}\n    \\label{eqn:sm-EAG_recurrence_q_k}\n\\end{align}\nfor $k=1,2,\\dots$.\nAdditionally, because $\\eta_k = \\frac{1-\\beta_k}{1+2\\alpha\\mu}$ and $p_k = \\frac{\\eta_k \\alpha q_k}{2\\beta_k}$,\n\\begin{align}\n    p_k = \\frac{\\alpha (1-\\beta_k) q_k}{2\\beta_k (1+2\\alpha\\mu)} = \\frac{\\alpha}{2} \\left( \\frac{1}{\\beta_k} - 1 \\right) \\frac{q_k}{1 + 2\\alpha\\mu} = \\frac{\\alpha}{2} \\frac{1}{\\beta_{k-1}} q_k = \\frac{\\alpha}{2} (1+2\\alpha\\mu)^{k-1} q_k^2 \n    \\label{eqn:sm-EAG_pkqk}\n\\end{align}\nand\n\\begin{align}\n    p_{k+1} & = \\frac{\\eta_{k+1}\\alpha q_{k+1}}{2\\beta_{k+1}} = \\frac{\\alpha q_{k+1}}{2\\beta_{k+1}} \\frac{1-\\beta_{k+1}}{1+2\\alpha\\mu} = \\frac{\\alpha q_k}{2(1-\\beta_k)} \\frac{1-\\beta_{k+1}}{(1+2\\alpha\\mu)\\beta_{k+1}} \\nonumber \\\\\n    & = \\frac{\\alpha q_k}{2(1-\\beta_k)} \\frac{1}{1+2\\alpha\\mu} \\left( \\frac{1}{\\beta_{k+1}} - 1 \\right) = \\frac{\\alpha q_k}{2\\beta_k (1-\\beta_k)} . \n    \\label{eqn:sm-EAG_recurrence_p_k}\n\\end{align}\nThe identities~\\eqref{eqn:sm-EAG_recurrence_q_k} through \\eqref{eqn:sm-EAG_recurrence_p_k} will be useful for the subsequent arguments.\n\nUsing $\\mu$-strong monotonicity of $\\opB$ and the identities~\\eqref{eqn:sm_EAG+_zkp1_minus_z0}, \\eqref{eqn:SM-EAG+_zk_minus_zkp1} and \\eqref{eqn:sm-EAG_recurrence_q_k}, we have\n\\begin{align}\n    V_k - V_{k+1} & \\ge V_k - V_{k+1} - \\frac{q_k}{\\beta_k} \\langle z_k - z_{k+1}, \\opB z_k - \\opB z_{k+1} - \\mu(z_k - z_{k+1}) \\rangle \\nonumber \\\\\n    & = p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k , z_k-z_0 \\rangle - q_k \\mu \\|z_k - z_0\\|^2 \\nonumber \\\\\n    & \\quad - p_{k+1} \\|\\opB z_{k+1}\\|^2 - q_{k+1} \\langle \\opB z_{k+1} , (1-\\beta_k)(z_k - z_0) - \\alpha \\opB z_{k+1/2} \\rangle \\nonumber \\\\\n    & \\quad + q_{k+1} \\mu \\left\\| (1-\\beta_k) (z_k - z_0) - \\alpha \\opB z_{k+1/2} \\right\\|^2 \\nonumber \\\\\n    & \\quad - \\frac{q_k}{\\beta_k} \\langle \\beta_k (z_k - z_0) + \\alpha \\opB z_{k+1/2} , \\opB z_k - \\opB z_{k+1} \\rangle \\nonumber \\\\\n    & \\quad + \\frac{q_k \\mu}{\\beta_k} \\left\\| \\beta_k (z_k - z_0) + \\alpha \\opB z_{k+1/2} \\right\\|^2 \\nonumber \\\\\n    & \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 - \\frac{\\alpha q_k}{\\beta_k} \\langle \\opB z_k, \\opB z_{k+1/2} \\rangle + \\mu \\alpha^2 \\left( \\frac{q_k}{\\beta_k} + q_{k+1} \\right) \\|\\opB z_{k+1/2}\\|^2 \\\\\n        & \\quad + \\alpha \\left( \\frac{q_k}{\\beta_k} + q_{k+1} \\right) \\langle \\opB z_{k+1/2} , \\opB z_{k+1} \\rangle - p_{k+1} \\|\\opB z_{k+1}\\|^2 \\\\\n        & \\quad + 2\\mu\\alpha \\left( q_k - (1-\\beta_k)q_{k+1} \\right) \\langle \\opB z_{k+1/2}, z_k - z_0 \\rangle \\\\\n        & \\quad + (q_k - (1-\\beta_k) q_{k+1}) \\langle \\opB z_{k+1}, z_k - z_0 \\rangle \\\\\n        & \\quad + \\mu (1-\\beta_k) ((1-\\beta_k) q_{k+1} - q_k) \\|z_k - z_0\\|^2 \n    \\end{aligned} \\nonumber \\\\\n    & \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 - \\frac{\\alpha q_k}{\\beta_k} \\langle \\opB z_k, \\opB z_{k+1/2} \\rangle + \\frac{\\mu \\alpha^2 q_k}{\\beta_k (1-\\beta_k)} \\|\\opB z_{k+1/2}\\|^2 \\\\\n        & \\quad + \\frac{\\alpha q_k}{\\beta_k (1-\\beta_k)} \\langle \\opB z_{k+1/2} , \\opB z_{k+1} \\rangle - p_{k+1} \\|\\opB z_{k+1}\\|^2 .\n    \\end{aligned} \\label{eqn:sm-EAG_Lyapunov_monotone}\n\\end{align}\nNow because $\\opB$ is $L$-Lipschitz,\n\\begin{align}\n\\label{eqn:sm-EAG_Lyapunov_Lipschitz}\n    \\|\\opB z_{k+1/2} - \\opB z_{k+1}\\|^2 \\le L^2 \\|z_{k+1/2} - z_{k+1}\\|^2 = L^2 \\|\\alpha \\opB z_{k+1/2} - \\eta_k \\alpha \\opB z_k\\|^2 \n\\end{align}\nwhere we applied the identity~\\eqref{eqn:sm_EAG+_zkhalf_minus_zkp1} for the last equality.\nMultiplying $p_{k+1}$ to \\eqref{eqn:sm-EAG_Lyapunov_Lipschitz} and then subtracting it from \\eqref{eqn:sm-EAG_Lyapunov_monotone}, we obtain\n\\begin{align}\n    V_k - V_{k+1} & \\ge ( p_k - p_{k+1} \\eta_k^2 \\alpha^2 L^2) \\|\\opB z_k\\|^2 - \\left( \\frac{\\alpha q_k}{\\beta_k} - 2p_{k+1} \\eta_k \\alpha^2 L^2 \\right) \\langle \\opB z_k, \\opB z_{k+1/2} \\rangle \\nonumber \\\\\n    & \\quad + \\left( \\frac{\\mu\\alpha^2 q_k}{\\beta_k (1-\\beta_k)} + p_{k+1}(1-\\alpha^2 L^2) \\right) \\|\\opB z_{k+1/2}\\|^2 \\nonumber \\\\\n    & \\quad + \\left( \\frac{\\alpha q_k}{\\beta_k (1-\\beta_k)} - 2p_{k+1} \\right) \\langle \\opB z_{k+1/2} , \\opB z_{k+1} \\rangle \\nonumber \\\\\n    & = \\frac{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2) q_k}{2\\beta_k (1-\\beta_k)} \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2\n    \\label{eqn:sm-EAG_summable_term}\n\\end{align}\nwhere the last equality follows from the identities $\\eta_k = \\frac{1-\\beta_k}{1+2\\alpha\\mu}$, $p_k = \\frac{\\eta_k \\alpha q_k}{2\\beta_k}$ and \\eqref{eqn:sm-EAG_recurrence_p_k}.\nAs $1 + 2\\alpha\\mu - \\alpha^2 L^2 \\ge 0$ if $0 < \\alpha \\le \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2}$, for this range of $\\alpha$,\n\\begin{align*}\n    V_k - V_{k+1} & \\ge \\frac{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2)}{2} (1+2\\alpha\\mu)^k \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2 \\ge 0,\n\\end{align*}\nwhere we used $\\frac{q_k}{\\beta_k (1-\\beta_k)} = \\frac{q_{k+1}}{\\beta_k} > (1+2\\alpha\\mu)^k$.\nNow it only remains to show that $V_N \\ge 0$ holds for any $N \\in \\mathbb{N}$; if that is the case, then\n\\begin{align*}\n    & \\sum_{k=0}^N (1+2\\alpha\\mu)^k \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2 \\le \\frac{2}{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2)} (V_0 - V_{N+1}) \\\\\n    & \\quad \\le \\frac{2}{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2)} \\left( \\frac{1}{2\\alpha} + \\mu \\right) \\|z_0 - z_\\star\\|^2 = \\frac{1+2\\alpha\\mu}{\\alpha^2 (1 + 2\\alpha\\mu - \\alpha^2 L^2)} \\|z_0 - z_\\star\\|^2 ,\n\\end{align*}\nand taking $N\\to\\infty$, we obtain~\\eqref{eqn:sm-EAG_summability}.\n\nConsider the following shifted version of $V_k$:\n\\begin{align*}\n    W_k = V_k - \\left(\\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2 = p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k - \\mu (z_k - z_0), z_k - z_0 \\rangle .\n\\end{align*}\nBecause $\\opB$ is $\\mu$-strongly monotone, if $\\lambda \\ge 0$ and $\\lambda \\ne q_k$,\n\\begin{align}\n    W_k & \\ge p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k - \\mu (z_k - z_0), z_k - z_0 \\rangle - \\lambda \\langle \\opB z_k - \\mu (z_k - z_\\star), z_k - z_\\star \\rangle \\nonumber \\\\\n    &\n    \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 + \\langle \\opB z_k , (q_k - \\lambda) z_k - q_k z_0 + \\lambda z_\\star \\rangle \\\\\n        & \\quad + \\mu(\\lambda-q_k) \\|z_k\\|^2 + 2 \\mu \\langle z_k, q_k z_0 - \\lambda z_\\star \\rangle - q_k \\mu \\|z_0\\|^2 + \\lambda\\mu \\|z_\\star\\|^2  \n    \\end{aligned} \\nonumber \\\\\n    &\n    \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 + \\langle \\opB z_k , (q_k - \\lambda) z_k - q_k z_0 + \\lambda z_\\star \\rangle \\\\\n        & \\quad + \\frac{\\mu}{\\lambda - q_k} \\left\\| (\\lambda - q_k) z_k + q_k z_0 - \\lambda z_\\star \\right\\|^2 - \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 .\n    \\end{aligned} \\nonumber \\\\ \n    &\n    \\begin{aligned}\n        & = \\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right) \\|\\opB z_k\\|^2 + \\frac{\\mu}{\\lambda - q_k} \\left\\| \\frac{\\lambda - q_k}{2\\mu} \\opB z_k - (\\lambda - q_k) z_k - q_k z_0 + \\lambda z_\\star \\right\\|^2 \\\\\n        & \\quad - \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 .\n    \\end{aligned} \\nonumber\n\\end{align}\nAdding $\\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2$ to the both sides and taking $\\lambda = q_k + 4\\mu p_k$, we obtain \n\\begin{align*}\n    0 & \\le W_k + \\left( \\frac{q_k^2}{4p_k} + \\mu q_k \\right) \\|z_0 - z_\\star\\|^2 = V_k ,\n\\end{align*}\nwhere the last equality follows by applying \\eqref{eqn:sm-EAG_pkqk} and then plugging in the expression $q_k = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{1}{\\beta_{k-1}} = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{(1+2\\alpha\\mu)^k - 1}{2\\alpha\\mu}$.\n\n\n","units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:sm-EAG_Lyapunov}\nLet $\\opB\\colon \\reals^d \\to \\reals^d$ be a $\\mu$-strongly monotone, $L$-Lipschitz operator ($0 < \\mu \\le L$), and let\n$\\{z_k\\}_{k=0,1,\\dots}$ be the iterates of \\ref{eqn:SM-EAG+} with $\\beta_k = \\frac{1}{\\sum_{j=0}^k (1+2\\alpha\\mu)^j}$.\nThen, the quantities\n\\begin{align*}\n    V_k = p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k - \\mu (z_k - z_0), z_k - z_0 \\rangle + \\left( \\frac{1}{2\\alpha} + \\mu \\right) \\|z_0-z_\\star\\|^2\n\\end{align*}\nwith $p_0 = q_0 = 0$ and $q_k = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{1}{\\beta_{k-1}}$, $p_k = \\frac{\\eta_k \\alpha q_k}{2 \\beta_k}$ for $k=1,2,\\dots$ satisfy $V_k \\ge 0$, $V_0 - V_1 \\ge \\frac{\\alpha (1+2\\alpha\\mu-\\alpha^2 L^2)}{2}\\|\\opB z_0\\|^2$, and\n\\begin{align*}\n    V_k - V_{k+1} \\ge \\frac{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2) q_k}{2\\beta_k (1-\\beta_k)} \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2\n\\end{align*}\nfor $k=1,2,\\dots$.\nIn particular, if $\\alpha \\in \\big( 0, \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2} \\big]$, $V_k$ is a Lyapunov function for the algorithm, and if $\\alpha \\in \\big( 0, \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2} \\big)$, we additionally have the summability result\n\\begin{align}\n    \\sum_{k=0}^\\infty (1+2\\alpha\\mu)^k \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2 \\le \\frac{1+2\\alpha\\mu}{\\alpha^2 (1 + 2\\alpha\\mu - \\alpha^2 L^2)} \\|z_0 - z_\\star\\|^2  < \\infty .\n    \\label{eqn:sm-EAG_summability}\n\\end{align}\n\\end{lemma}","start":52374,"end":53778},{"raw_tex":"\\subsection{Proof of \\texorpdfstring{\\cref{thm:sm-EAG}}{Theorem 4.2}}\nBecause the term $\\left( \\frac{1}{2\\alpha}+\\mu \\right) \\|z_0-z_\\star\\|^2$ subtracted from $V_k$ to obtain $W_k$ does not depend on $k$, clearly $W_k$ inherits nonincreasingness from $V_k$, i.e., $W_k \\le \\cdots \\le W_0 = 0$.\nTherefore, given that $q_k < \\lambda < q_k + 4\\mu p_k$, we have\n\\begin{align}\n    & \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 \\ge W_k + \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 \\ge \\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right) \\|\\opB z_k\\|^2 \\nonumber \\\\\n    & \\implies \\|\\opB z_k\\|^2 \\le \\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right)^{-1} \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 . \\label{eqn:sm-EAG_convergence_rate}\n\\end{align}\nOne can straightforwardly verify that \n\\begin{align}\n    \\lambda_\\star = (q_k (q_k + 4\\mu p_k))^{1/2}\n    \\label{eqn:sm-EAG_lambda}\n\\end{align}\nlies between $q_k$ and $q_k + 4\\mu p_k$, and minimizes the factor $\\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right)^{-1} \\frac{q_k \\lambda \\mu}{\\lambda - q_k}$ in the right hand side of \\eqref{eqn:sm-EAG_convergence_rate}.\nPlugging in the explicit expressions $q_k = \\frac{(1+2\\alpha\\mu)^k - 1}{2\\alpha\\mu (1+2\\alpha\\mu)^{k-1}}$ and $p_k = \\frac{\\alpha}{2}(1+2\\alpha\\mu)^{k-1} q_k^2 = \\frac{((1+2\\alpha\\mu)^k - 1)^2}{8\\alpha\\mu^2 (1+2\\alpha\\mu)^{k-1}}$ (which uses \\eqref{eqn:sm-EAG_pkqk}) into~\\eqref{eqn:sm-EAG_lambda}, we obtain $\\lambda_\\star = (1+2\\alpha\\mu)^{\\frac{k}{2}} q_k$ and\n\\begin{align*}\n    \\left( p_k - \\frac{\\lambda_\\star - q_k}{4\\mu} \\right)^{-1} \\frac{q_k \\lambda_\\star \\mu}{\\lambda_\\star - q_k} & = \\frac{4\\mu^2}{((1+2\\alpha\\mu)^{\\frac{k}{2}} - 1)^2} \n    = \\frac{(\\sqrt{1+2\\alpha\\mu}+1)^2}{\\alpha^2 \\left( \\sum_{j=0}^{k-1} (1+2\\alpha\\mu)^{\\frac{j}{2}} \\right)^2} .\n\\end{align*}\nTherefore, \\eqref{eqn:sm-EAG_convergence_rate} with $\\lambda = \\lambda_\\star$ concludes the proof.\n\n\n\n","start":53865,"end":55789},{"raw_tex":"\\subsection{Proof of \\texorpdfstring{\\cref{lem:sm-EAG_Lyapunov}}{Lemma 4.4}}\nWhen $k=0$, we have $z_{1/2} = z_0$, $z_1 = z_0 - \\alpha \\opB z_0$, $V_0 = \\left( \\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2$, and\n\\begin{align*}\n    V_1 & = \\frac{\\alpha}{2}\\|\\opB z_1\\|^2 + \\langle \\opB z_1 - \\mu(z_1 - z_0), z_1 - z_0 \\rangle + \\left(\\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2 \\nonumber \\\\\n    & = \\frac{\\alpha}{2}\\|\\opB z_1\\|^2 - \\alpha \\langle \\opB z_1, \\opB z_0 \\rangle - \\alpha^2 \\mu \\|\\opB z_0\\|^2 + \\left(\\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2 . \n\\end{align*}\nBecause $\\opB$ is $L$-Lipschitz,\n\\begin{align*}\n    0 & \\le L^2 \\|z_1 - z_0\\|^2 - \\|\\opB z_1 - \\opB z_0\\|^2 = (\\alpha^2 L^2 - 1) \\|\\opB z_0\\|^2 + 2\\langle \\opB z_0, \\opB z_1 \\rangle - \\|\\opB z_1\\|^2 .\n\\end{align*}\nTherefore,\n\\begin{align*}\n    V_0 - V_1 & = \\alpha^2 \\mu \\|\\opB z_0\\|^2 + \\alpha \\langle \\opB z_0, \\opB z_1 \\rangle - \\frac{\\alpha}{2} \\|\\opB z_1\\|^2 \\\\\n    & = \\frac{\\alpha}{2} (1+2\\alpha\\mu -\\alpha^2 L^2) \\|\\opB z_0\\|^2 + \\frac{\\alpha}{2} \\left( (\\alpha^2 L^2 - 1) \\|\\opB z_0\\|^2 + 2\\langle \\opB z_0, \\opB z_1 \\rangle - \\|\\opB z_1\\|^2 \\right) \\\\\n    & \\ge \\frac{\\alpha}{2} (1+2\\alpha\\mu -\\alpha^2 L^2) \\|\\opB z_0\\|^2 \\\\\n    & = \\frac{\\alpha}{2} (1+2\\alpha\\mu -\\alpha^2 L^2) \\|\\eta_0 \\opB z_0 - \\opB z_{1/2}\\|^2 \n\\end{align*}\nwhere the last equality holds because $\\eta_0 = 0$ and $z_{1/2} = z_0$.\n\nNow suppose $k\\ge 1$. For \\ref{eqn:SM-EAG+}, we have the identities\n\\begin{align}\n    z_{k+1} - z_0 & = (1-\\beta_k) (z_k - z_0) - \\alpha \\opB z_{k+1/2} \\label{eqn:sm_EAG+_zkp1_minus_z0} \\\\\n    z_k - z_{k+1} & = \\beta_k (z_k - z_0) + \\alpha \\opB z_{k+1/2} \\label{eqn:SM-EAG+_zk_minus_zkp1} \\\\\n    z_{k+1/2} - z_{k+1} & = \\alpha \\opB z_{k+1/2} - \\eta_k \\alpha \\opB z_k . \\label{eqn:sm_EAG+_zkhalf_minus_zkp1}\n\\end{align}\nNote that $\\beta_k = \\frac{1}{\\sum_{j=0}^{k} (1+2\\alpha\\mu)^j}$ satisfies\n\\begin{align*}\n    \\frac{1}{\\beta_{k}} = \\sum_{j=0}^{k} (1+2\\alpha\\mu)^j = 1 + (1+2\\alpha\\mu) \\sum_{j=0}^{k-1} (1+2\\alpha\\mu)^j = 1 + \\frac{1+2\\alpha\\mu}{\\beta_{k-1}} \n\\end{align*}\nfor $k=1,2,\\dots$, so $q_k = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{1}{\\beta_{k-1}}$ satisfies the recurrence relation\n\\begin{align}\n    q_{k+1} = \\frac{q_k}{\\beta_k} \\frac{\\beta_{k-1}}{(1+2\\alpha\\mu)} =\n    \\frac{q_k}{\\beta_k} \\frac{1}{\\frac{1}{\\beta_k} - 1} = \\frac{q_k}{1-\\beta_k}\n    \\label{eqn:sm-EAG_recurrence_q_k}\n\\end{align}\nfor $k=1,2,\\dots$.\nAdditionally, because $\\eta_k = \\frac{1-\\beta_k}{1+2\\alpha\\mu}$ and $p_k = \\frac{\\eta_k \\alpha q_k}{2\\beta_k}$,\n\\begin{align}\n    p_k = \\frac{\\alpha (1-\\beta_k) q_k}{2\\beta_k (1+2\\alpha\\mu)} = \\frac{\\alpha}{2} \\left( \\frac{1}{\\beta_k} - 1 \\right) \\frac{q_k}{1 + 2\\alpha\\mu} = \\frac{\\alpha}{2} \\frac{1}{\\beta_{k-1}} q_k = \\frac{\\alpha}{2} (1+2\\alpha\\mu)^{k-1} q_k^2 \n    \\label{eqn:sm-EAG_pkqk}\n\\end{align}\nand\n\\begin{align}\n    p_{k+1} & = \\frac{\\eta_{k+1}\\alpha q_{k+1}}{2\\beta_{k+1}} = \\frac{\\alpha q_{k+1}}{2\\beta_{k+1}} \\frac{1-\\beta_{k+1}}{1+2\\alpha\\mu} = \\frac{\\alpha q_k}{2(1-\\beta_k)} \\frac{1-\\beta_{k+1}}{(1+2\\alpha\\mu)\\beta_{k+1}} \\nonumber \\\\\n    & = \\frac{\\alpha q_k}{2(1-\\beta_k)} \\frac{1}{1+2\\alpha\\mu} \\left( \\frac{1}{\\beta_{k+1}} - 1 \\right) = \\frac{\\alpha q_k}{2\\beta_k (1-\\beta_k)} . \n    \\label{eqn:sm-EAG_recurrence_p_k}\n\\end{align}\nThe identities~\\eqref{eqn:sm-EAG_recurrence_q_k} through \\eqref{eqn:sm-EAG_recurrence_p_k} will be useful for the subsequent arguments.\n\nUsing $\\mu$-strong monotonicity of $\\opB$ and the identities~\\eqref{eqn:sm_EAG+_zkp1_minus_z0}, \\eqref{eqn:SM-EAG+_zk_minus_zkp1} and \\eqref{eqn:sm-EAG_recurrence_q_k}, we have\n\\begin{align}\n    V_k - V_{k+1} & \\ge V_k - V_{k+1} - \\frac{q_k}{\\beta_k} \\langle z_k - z_{k+1}, \\opB z_k - \\opB z_{k+1} - \\mu(z_k - z_{k+1}) \\rangle \\nonumber \\\\\n    & = p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k , z_k-z_0 \\rangle - q_k \\mu \\|z_k - z_0\\|^2 \\nonumber \\\\\n    & \\quad - p_{k+1} \\|\\opB z_{k+1}\\|^2 - q_{k+1} \\langle \\opB z_{k+1} , (1-\\beta_k)(z_k - z_0) - \\alpha \\opB z_{k+1/2} \\rangle \\nonumber \\\\\n    & \\quad + q_{k+1} \\mu \\left\\| (1-\\beta_k) (z_k - z_0) - \\alpha \\opB z_{k+1/2} \\right\\|^2 \\nonumber \\\\\n    & \\quad - \\frac{q_k}{\\beta_k} \\langle \\beta_k (z_k - z_0) + \\alpha \\opB z_{k+1/2} , \\opB z_k - \\opB z_{k+1} \\rangle \\nonumber \\\\\n    & \\quad + \\frac{q_k \\mu}{\\beta_k} \\left\\| \\beta_k (z_k - z_0) + \\alpha \\opB z_{k+1/2} \\right\\|^2 \\nonumber \\\\\n    & \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 - \\frac{\\alpha q_k}{\\beta_k} \\langle \\opB z_k, \\opB z_{k+1/2} \\rangle + \\mu \\alpha^2 \\left( \\frac{q_k}{\\beta_k} + q_{k+1} \\right) \\|\\opB z_{k+1/2}\\|^2 \\\\\n        & \\quad + \\alpha \\left( \\frac{q_k}{\\beta_k} + q_{k+1} \\right) \\langle \\opB z_{k+1/2} , \\opB z_{k+1} \\rangle - p_{k+1} \\|\\opB z_{k+1}\\|^2 \\\\\n        & \\quad + 2\\mu\\alpha \\left( q_k - (1-\\beta_k)q_{k+1} \\right) \\langle \\opB z_{k+1/2}, z_k - z_0 \\rangle \\\\\n        & \\quad + (q_k - (1-\\beta_k) q_{k+1}) \\langle \\opB z_{k+1}, z_k - z_0 \\rangle \\\\\n        & \\quad + \\mu (1-\\beta_k) ((1-\\beta_k) q_{k+1} - q_k) \\|z_k - z_0\\|^2 \n    \\end{aligned} \\nonumber \\\\\n    & \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 - \\frac{\\alpha q_k}{\\beta_k} \\langle \\opB z_k, \\opB z_{k+1/2} \\rangle + \\frac{\\mu \\alpha^2 q_k}{\\beta_k (1-\\beta_k)} \\|\\opB z_{k+1/2}\\|^2 \\\\\n        & \\quad + \\frac{\\alpha q_k}{\\beta_k (1-\\beta_k)} \\langle \\opB z_{k+1/2} , \\opB z_{k+1} \\rangle - p_{k+1} \\|\\opB z_{k+1}\\|^2 .\n    \\end{aligned} \\label{eqn:sm-EAG_Lyapunov_monotone}\n\\end{align}\nNow because $\\opB$ is $L$-Lipschitz,\n\\begin{align}\n\\label{eqn:sm-EAG_Lyapunov_Lipschitz}\n    \\|\\opB z_{k+1/2} - \\opB z_{k+1}\\|^2 \\le L^2 \\|z_{k+1/2} - z_{k+1}\\|^2 = L^2 \\|\\alpha \\opB z_{k+1/2} - \\eta_k \\alpha \\opB z_k\\|^2 \n\\end{align}\nwhere we applied the identity~\\eqref{eqn:sm_EAG+_zkhalf_minus_zkp1} for the last equality.\nMultiplying $p_{k+1}$ to \\eqref{eqn:sm-EAG_Lyapunov_Lipschitz} and then subtracting it from \\eqref{eqn:sm-EAG_Lyapunov_monotone}, we obtain\n\\begin{align}\n    V_k - V_{k+1} & \\ge ( p_k - p_{k+1} \\eta_k^2 \\alpha^2 L^2) \\|\\opB z_k\\|^2 - \\left( \\frac{\\alpha q_k}{\\beta_k} - 2p_{k+1} \\eta_k \\alpha^2 L^2 \\right) \\langle \\opB z_k, \\opB z_{k+1/2} \\rangle \\nonumber \\\\\n    & \\quad + \\left( \\frac{\\mu\\alpha^2 q_k}{\\beta_k (1-\\beta_k)} + p_{k+1}(1-\\alpha^2 L^2) \\right) \\|\\opB z_{k+1/2}\\|^2 \\nonumber \\\\\n    & \\quad + \\left( \\frac{\\alpha q_k}{\\beta_k (1-\\beta_k)} - 2p_{k+1} \\right) \\langle \\opB z_{k+1/2} , \\opB z_{k+1} \\rangle \\nonumber \\\\\n    & = \\frac{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2) q_k}{2\\beta_k (1-\\beta_k)} \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2\n    \\label{eqn:sm-EAG_summable_term}\n\\end{align}\nwhere the last equality follows from the identities $\\eta_k = \\frac{1-\\beta_k}{1+2\\alpha\\mu}$, $p_k = \\frac{\\eta_k \\alpha q_k}{2\\beta_k}$ and \\eqref{eqn:sm-EAG_recurrence_p_k}.\nAs $1 + 2\\alpha\\mu - \\alpha^2 L^2 \\ge 0$ if $0 < \\alpha \\le \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2}$, for this range of $\\alpha$,\n\\begin{align*}\n    V_k - V_{k+1} & \\ge \\frac{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2)}{2} (1+2\\alpha\\mu)^k \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2 \\ge 0,\n\\end{align*}\nwhere we used $\\frac{q_k}{\\beta_k (1-\\beta_k)} = \\frac{q_{k+1}}{\\beta_k} > (1+2\\alpha\\mu)^k$.\nNow it only remains to show that $V_N \\ge 0$ holds for any $N \\in \\mathbb{N}$; if that is the case, then\n\\begin{align*}\n    & \\sum_{k=0}^N (1+2\\alpha\\mu)^k \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2 \\le \\frac{2}{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2)} (V_0 - V_{N+1}) \\\\\n    & \\quad \\le \\frac{2}{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2)} \\left( \\frac{1}{2\\alpha} + \\mu \\right) \\|z_0 - z_\\star\\|^2 = \\frac{1+2\\alpha\\mu}{\\alpha^2 (1 + 2\\alpha\\mu - \\alpha^2 L^2)} \\|z_0 - z_\\star\\|^2 ,\n\\end{align*}\nand taking $N\\to\\infty$, we obtain~\\eqref{eqn:sm-EAG_summability}.\n\nConsider the following shifted version of $V_k$:\n\\begin{align*}\n    W_k = V_k - \\left(\\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2 = p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k - \\mu (z_k - z_0), z_k - z_0 \\rangle .\n\\end{align*}\nBecause $\\opB$ is $\\mu$-strongly monotone, if $\\lambda \\ge 0$ and $\\lambda \\ne q_k$,\n\\begin{align}\n    W_k & \\ge p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k - \\mu (z_k - z_0), z_k - z_0 \\rangle - \\lambda \\langle \\opB z_k - \\mu (z_k - z_\\star), z_k - z_\\star \\rangle \\nonumber \\\\\n    &\n    \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 + \\langle \\opB z_k , (q_k - \\lambda) z_k - q_k z_0 + \\lambda z_\\star \\rangle \\\\\n        & \\quad + \\mu(\\lambda-q_k) \\|z_k\\|^2 + 2 \\mu \\langle z_k, q_k z_0 - \\lambda z_\\star \\rangle - q_k \\mu \\|z_0\\|^2 + \\lambda\\mu \\|z_\\star\\|^2  \n    \\end{aligned} \\nonumber \\\\\n    &\n    \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 + \\langle \\opB z_k , (q_k - \\lambda) z_k - q_k z_0 + \\lambda z_\\star \\rangle \\\\\n        & \\quad + \\frac{\\mu}{\\lambda - q_k} \\left\\| (\\lambda - q_k) z_k + q_k z_0 - \\lambda z_\\star \\right\\|^2 - \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 .\n    \\end{aligned} \\nonumber \\\\ \n    &\n    \\begin{aligned}\n        & = \\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right) \\|\\opB z_k\\|^2 + \\frac{\\mu}{\\lambda - q_k} \\left\\| \\frac{\\lambda - q_k}{2\\mu} \\opB z_k - (\\lambda - q_k) z_k - q_k z_0 + \\lambda z_\\star \\right\\|^2 \\\\\n        & \\quad - \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 .\n    \\end{aligned} \\nonumber\n\\end{align}\nAdding $\\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2$ to the both sides and taking $\\lambda = q_k + 4\\mu p_k$, we obtain \n\\begin{align*}\n    0 & \\le W_k + \\left( \\frac{q_k^2}{4p_k} + \\mu q_k \\right) \\|z_0 - z_\\star\\|^2 = V_k ,\n\\end{align*}\nwhere the last equality follows by applying \\eqref{eqn:sm-EAG_pkqk} and then plugging in the expression $q_k = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{1}{\\beta_{k-1}} = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{(1+2\\alpha\\mu)^k - 1}{2\\alpha\\mu}$.\n\n\n","start":90433,"end":100120}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eqn:minimax_problem"],"declared_external":false,"source_only":false,"title":"极小极大问题","source_units":[{"raw_tex":"Minimax optimization problems of the form\n\\begin{align}\n\\label{eqn:minimax_problem}\n    \\underset{x \\in \\cX}{\\textrm{minimize}} \\,\\, \\underset{y \\in \\cY}{\\textrm{maximize}} \\,\\,\\,\\, \\lagrange (x,y)\n\\end{align}\n","start":14562,"end":14772}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"凸凹函数","source_units":[{"raw_tex":"A function $\\lagrange(\\cdot, \\cdot): \\cX \\times \\cY \\to \\reals$ is \\emph{convex-concave} if $\\cX$ and $\\cY$ are convex sets, $\\lagrange(x, y)$ is convex as a function of $x$ for all $y$, and $\\lagrange(x, y)$ is concave as a function of $y$ for all $x$.\n","start":28121,"end":28375}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"鞍点与极小极大解","source_units":[{"raw_tex":"A point $(x_\\star, y_\\star) \\in \\cX \\times \\cY$ is a \\emph{saddle point}, or \\emph{(minimax) solution} of~\\eqref{eqn:minimax_problem} if $\\lagrange(x_\\star, y) \\le \\lagrange(x_\\star, y_\\star) \\le \\lagrange(x, y_\\star)$ for all $(x,y) \\in \\cX \\times \\cY$.\n\n","start":28375,"end":28631}]},{"id":"S004","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"算子记号与单值性","source_units":[{"raw_tex":"An operator $\\opA$ on $\\reals^d$, denoted $\\opA\\colon \\reals^d \\rightrightarrows \\reals^d$, is a set-valued function, i.e., $\\opA(z)\\subseteq\\reals^d$ for all $z \\in \\reals^d$.\nFor simplicity, we often write $\\opA z = \\opA(z)$.\nIf $\\opA z$ contains exactly one element for all $z\\in\\reals^d$, then we say $\\opA$ is \\emph{single-valued} and view it as a function.\n","start":28631,"end":28994}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"单调算子","source_units":[{"raw_tex":"An operator $\\opA\\colon \\reals^d \\rightrightarrows \\reals^d$ is \\emph{monotone} if\n\\begin{align*}\n    \\langle \\opA z - \\opA z', z - z' \\rangle \\ge 0, \\qquad \\forall \\, z,z'\\in\\reals^d ,\n\\end{align*}\nwhere the notation means that $\\langle u - v, z - z' \\rangle \\ge 0$ for any $u \\in \\opA z$ and $v \\in \\opA z'$.\n","start":28994,"end":29305}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"强单调算子","source_units":[{"raw_tex":"For $\\mu > 0$, an operator $\\opA$ is $\\mu$-\\emph{strongly monotone} if\n\\begin{align*}\n    \\langle \\opA z - \\opA z', z - z' \\rangle \\ge \\mu \\|z-z'\\|^2, \\qquad \\forall \\, z,z'\\in\\reals^d .\n\\end{align*}\n","start":29305,"end":29505}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"鞍算子","source_units":[{"raw_tex":"For differentiable and convex-concave $\\lagrange\\colon\\reals^n \\times \\reals^m\\rightarrow\\reals$, we define\n\\[\n    \\sop\\lagrange (x,y) = (\\nabla_x \\lagrange (x,y), -\\nabla_y \\lagrange (x,y))\n\\]\nand call it the \\emph{saddle operator} of $\\lagrange$.\n","start":31511,"end":31760}]},{"id":"S008","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原文第627行引用 Rockafellar1970_monotone；此项是来源明确的背景事实，不属于任务列出的两类基础先决事实，且未用于强单调算子收敛证明。"],"source_only":false,"title":"鞍算子的单调性","source_units":[{"raw_tex":"Then $\\sop \\lagrange$ is monotone \\cite{Rockafellar1970_monotone}, and $z_\\star = (x_\\star, y_\\star)$ is a solution of~\\eqref{eqn:minimax_problem} if and only if $\\sop \\lagrange (z_\\star) = 0$.\n","start":31760,"end":31954}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"零点对应鞍点","source_units":[{"raw_tex":"Then $\\sop \\lagrange$ is monotone \\cite{Rockafellar1970_monotone}, and $z_\\star = (x_\\star, y_\\star)$ is a solution of~\\eqref{eqn:minimax_problem} if and only if $\\sop \\lagrange (z_\\star) = 0$.\n","start":31760,"end":31954}]},{"id":"S010","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"算法名称来源","source_units":[{"raw_tex":"We design the novel algorithm \\emph{\\textbf{S}trongly \\textbf{M}onotone \\textbf{E}xtra \\textbf{A}nchored \\textbf{G}radient\\textbf{\\texttt{+}}}\\footnote{The ``\\texttt{+}'' follows the nomenclature of \\cite{DiakonikolasDaskalakisJordan2021_efficient}, where the two-time-scale EG was named EG\\texttt{+}.} \n","start":40779,"end":41083}]},{"id":"S011","semantic_type":"algorithm","kind":"algorithm","labels":["eqn:SM-EAG+"],"declared_external":false,"source_only":false,"title":"SM-EAG+ 两步迭代","source_units":[{"raw_tex":"\\begin{align}\n\\label{eqn:SM-EAG+}\n\\begin{aligned}\n    z_{k+1/2} & = \\beta_k z_0 + (1-\\beta_k) z_k - \\eta_k \\alpha \\opB z_k \\\\\n    z_{k+1}   & = \\beta_k z_0 + (1-\\beta_k) z_k - \\alpha \\opB z_{k+1/2} ,\n\\end{aligned}\n\\tag{SM-EAG\\texttt{+}}\n\\end{align}\n","start":41083,"end":41332}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"迭代参数与步长","source_units":[{"raw_tex":"where $\\opB=\\sop\\lagrange$, $\\beta_k = \\frac{1}{\\sum_{j=0}^k (1+2\\alpha\\mu)^j}$, $\\eta_k = \\frac{1-\\beta_k}{1+2\\alpha\\mu}$, and $0 < \\alpha \\le \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2}$, to approximate OC-Halpern in the MP sense. \\ref{eqn:SM-EAG+} inherits the accelerated rate of OC-Halpern and achieves the fastest known rate in the setup with respect to the gradient norm, improving upon the prior rates of EG and OG by factors of 8 and 4, respectively.\n","start":41332,"end":41785}]},{"id":"S013","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"继承速率与算法比较","source_units":[{"raw_tex":"where $\\opB=\\sop\\lagrange$, $\\beta_k = \\frac{1}{\\sum_{j=0}^k (1+2\\alpha\\mu)^j}$, $\\eta_k = \\frac{1-\\beta_k}{1+2\\alpha\\mu}$, and $0 < \\alpha \\le \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2}$, to approximate OC-Halpern in the MP sense. \\ref{eqn:SM-EAG+} inherits the accelerated rate of OC-Halpern and achieves the fastest known rate in the setup with respect to the gradient norm, improving upon the prior rates of EG and OG by factors of 8 and 4, respectively.\n","start":41332,"end":41785}]},{"id":"S014","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"前向算法定位","source_units":[{"raw_tex":"(OC-Halpern is not a forward algorithm as it uses the proximal operator $\\JB$, while \\ref{eqn:SM-EAG+} is a forward algorithm using evaluations of $\\opB$.)\n","start":41785,"end":41941}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","labels":["thm:sm-EAG"],"declared_external":false,"source_only":false,"title":"算子假设与零点","source_units":[{"raw_tex":"Let $\\opB\\colon \\reals^d \\to \\reals^d$ be $\\mu$-strongly monotone and $L$-Lipschitz with $0 < \\mu \\le L$.\nLet $z_\\star$ be the zero of $\\opB$.\n","start":44696,"end":44839}]},{"id":"S016","semantic_type":"theorem","kind":"theorem","labels":["thm:sm-EAG"],"declared_external":false,"source_only":false,"title":"SM-EAG+ 显式加速速率","source_units":[{"raw_tex":"\\begin{theorem}[Fast rate of \\ref{eqn:SM-EAG+}]\n\\label{thm:sm-EAG}\nLet $\\opB\\colon \\reals^d \\to \\reals^d$ be $\\mu$-strongly monotone and $L$-Lipschitz with $0 < \\mu \\le L$.\nLet $z_\\star$ be the zero of $\\opB$.\nFor $\\alpha \\in \\big( 0, \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2} \\big]$, SM-EAG\\texttt{+} exhibits the rate\n\\begin{align*}\n    \\|\\opB z_k\\|^2 \\le \\frac{\\left(\\sqrt{1+2\\alpha\\mu} + 1\\right)^2}{\\alpha^2 \\left( \\sum_{j=0}^{k-1} (1+2\\alpha\\mu)^{\\frac{j}{2}} \\right)^2} \\|z_0 - z_\\star\\|^2 ,\n    \\qquad\\text{for } k=1,2,\\dots.\n\\end{align*}\n\\end{theorem}","start":44629,"end":45184}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","labels":["lem:sm-EAG_Lyapunov"],"declared_external":false,"source_only":false,"title":"势函数引理的设置","source_units":[{"raw_tex":"Let $\\opB\\colon \\reals^d \\to \\reals^d$ be a $\\mu$-strongly monotone, $L$-Lipschitz operator ($0 < \\mu \\le L$), and let\n$\\{z_k\\}_{k=0,1,\\dots}$ be the iterates of \\ref{eqn:SM-EAG+} with $\\beta_k = \\frac{1}{\\sum_{j=0}^k (1+2\\alpha\\mu)^j}$.\n","start":52416,"end":52654}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":["lem:sm-EAG_Lyapunov"],"declared_external":false,"source_only":false,"title":"带常数的势函数","source_units":[{"raw_tex":"Then, the quantities\n\\begin{align*}\n    V_k = p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k - \\mu (z_k - z_0), z_k - z_0 \\rangle + \\left( \\frac{1}{2\\alpha} + \\mu \\right) \\|z_0-z_\\star\\|^2\n\\end{align*}\n","start":52654,"end":52851}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":["lem:sm-EAG_Lyapunov"],"declared_external":false,"source_only":false,"title":"势函数系数","source_units":[{"raw_tex":"with $p_0 = q_0 = 0$ and $q_k = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{1}{\\beta_{k-1}}$, $p_k = \\frac{\\eta_k \\alpha q_k}{2 \\beta_k}$ for $k=1,2,\\dots$ satisfy $V_k \\ge 0$, $V_0 - V_1 \\ge \\frac{\\alpha (1+2\\alpha\\mu-\\alpha^2 L^2)}{2}\\|\\opB z_0\\|^2$, and\n","start":52851,"end":53101}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","labels":["eqn:sm-EAG_summability","lem:sm-EAG_Lyapunov"],"declared_external":false,"source_only":false,"title":"势函数下降与可和性","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:sm-EAG_Lyapunov}\nLet $\\opB\\colon \\reals^d \\to \\reals^d$ be a $\\mu$-strongly monotone, $L$-Lipschitz operator ($0 < \\mu \\le L$), and let\n$\\{z_k\\}_{k=0,1,\\dots}$ be the iterates of \\ref{eqn:SM-EAG+} with $\\beta_k = \\frac{1}{\\sum_{j=0}^k (1+2\\alpha\\mu)^j}$.\nThen, the quantities\n\\begin{align*}\n    V_k = p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k - \\mu (z_k - z_0), z_k - z_0 \\rangle + \\left( \\frac{1}{2\\alpha} + \\mu \\right) \\|z_0-z_\\star\\|^2\n\\end{align*}\nwith $p_0 = q_0 = 0$ and $q_k = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{1}{\\beta_{k-1}}$, $p_k = \\frac{\\eta_k \\alpha q_k}{2 \\beta_k}$ for $k=1,2,\\dots$ satisfy $V_k \\ge 0$, $V_0 - V_1 \\ge \\frac{\\alpha (1+2\\alpha\\mu-\\alpha^2 L^2)}{2}\\|\\opB z_0\\|^2$, and\n\\begin{align*}\n    V_k - V_{k+1} \\ge \\frac{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2) q_k}{2\\beta_k (1-\\beta_k)} \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2\n\\end{align*}\nfor $k=1,2,\\dots$.\nIn particular, if $\\alpha \\in \\big( 0, \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2} \\big]$, $V_k$ is a Lyapunov function for the algorithm, and if $\\alpha \\in \\big( 0, \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2} \\big)$, we additionally have the summability result\n\\begin{align}\n    \\sum_{k=0}^\\infty (1+2\\alpha\\mu)^k \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2 \\le \\frac{1+2\\alpha\\mu}{\\alpha^2 (1 + 2\\alpha\\mu - \\alpha^2 L^2)} \\|z_0 - z_\\star\\|^2  < \\infty .\n    \\label{eqn:sm-EAG_summability}\n\\end{align}\n\\end{lemma}","start":52374,"end":53778}]},{"id":"S021","semantic_type":"derived_fact","kind":"derived_fact","labels":["lem:sm-EAG_Lyapunov"],"declared_external":false,"source_only":false,"title":"势函数非负与初始下降","source_units":[{"raw_tex":"with $p_0 = q_0 = 0$ and $q_k = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{1}{\\beta_{k-1}}$, $p_k = \\frac{\\eta_k \\alpha q_k}{2 \\beta_k}$ for $k=1,2,\\dots$ satisfy $V_k \\ge 0$, $V_0 - V_1 \\ge \\frac{\\alpha (1+2\\alpha\\mu-\\alpha^2 L^2)}{2}\\|\\opB z_0\\|^2$, and\n","start":52851,"end":53101}]},{"id":"S022","semantic_type":"derived_fact","kind":"derived_fact","labels":["lem:sm-EAG_Lyapunov"],"declared_external":false,"source_only":false,"title":"一般迭代下降界","source_units":[{"raw_tex":"\\begin{align*}\n    V_k - V_{k+1} \\ge \\frac{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2) q_k}{2\\beta_k (1-\\beta_k)} \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2\n\\end{align*}\nfor $k=1,2,\\dots$.\n","start":53101,"end":53281}]},{"id":"S023","semantic_type":"derived_fact","kind":"derived_fact","labels":["lem:sm-EAG_Lyapunov"],"declared_external":false,"source_only":false,"title":"闭步长区间内不增","source_units":[{"raw_tex":"In particular, if $\\alpha \\in \\big( 0, \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2} \\big]$, $V_k$ is a Lyapunov function for the algorithm, and if $\\alpha \\in \\big( 0, \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2} \\big)$, we additionally have the summability result\n","start":53281,"end":53530}]},{"id":"S024","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:sm-EAG_summability","lem:sm-EAG_Lyapunov"],"declared_external":false,"source_only":false,"title":"严格步长下的加权可和界","source_units":[{"raw_tex":"In particular, if $\\alpha \\in \\big( 0, \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2} \\big]$, $V_k$ is a Lyapunov function for the algorithm, and if $\\alpha \\in \\big( 0, \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2} \\big)$, we additionally have the summability result\n\\begin{align}\n    \\sum_{k=0}^\\infty (1+2\\alpha\\mu)^k \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2 \\le \\frac{1+2\\alpha\\mu}{\\alpha^2 (1 + 2\\alpha\\mu - \\alpha^2 L^2)} \\|z_0 - z_\\star\\|^2  < \\infty .\n    \\label{eqn:sm-EAG_summability}\n\\end{align}\n","start":53281,"end":53767}]},{"id":"S025","semantic_type":"proof","kind":"proof","labels":["eqn:sm-EAG_convergence_rate","eqn:sm-EAG_lambda"],"declared_external":false,"source_only":false,"title":"主定理完整作者证明","source_units":[{"raw_tex":"\\subsection{Proof of \\texorpdfstring{\\cref{thm:sm-EAG}}{Theorem 4.2}}\nBecause the term $\\left( \\frac{1}{2\\alpha}+\\mu \\right) \\|z_0-z_\\star\\|^2$ subtracted from $V_k$ to obtain $W_k$ does not depend on $k$, clearly $W_k$ inherits nonincreasingness from $V_k$, i.e., $W_k \\le \\cdots \\le W_0 = 0$.\nTherefore, given that $q_k < \\lambda < q_k + 4\\mu p_k$, we have\n\\begin{align}\n    & \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 \\ge W_k + \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 \\ge \\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right) \\|\\opB z_k\\|^2 \\nonumber \\\\\n    & \\implies \\|\\opB z_k\\|^2 \\le \\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right)^{-1} \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 . \\label{eqn:sm-EAG_convergence_rate}\n\\end{align}\nOne can straightforwardly verify that \n\\begin{align}\n    \\lambda_\\star = (q_k (q_k + 4\\mu p_k))^{1/2}\n    \\label{eqn:sm-EAG_lambda}\n\\end{align}\nlies between $q_k$ and $q_k + 4\\mu p_k$, and minimizes the factor $\\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right)^{-1} \\frac{q_k \\lambda \\mu}{\\lambda - q_k}$ in the right hand side of \\eqref{eqn:sm-EAG_convergence_rate}.\nPlugging in the explicit expressions $q_k = \\frac{(1+2\\alpha\\mu)^k - 1}{2\\alpha\\mu (1+2\\alpha\\mu)^{k-1}}$ and $p_k = \\frac{\\alpha}{2}(1+2\\alpha\\mu)^{k-1} q_k^2 = \\frac{((1+2\\alpha\\mu)^k - 1)^2}{8\\alpha\\mu^2 (1+2\\alpha\\mu)^{k-1}}$ (which uses \\eqref{eqn:sm-EAG_pkqk}) into~\\eqref{eqn:sm-EAG_lambda}, we obtain $\\lambda_\\star = (1+2\\alpha\\mu)^{\\frac{k}{2}} q_k$ and\n\\begin{align*}\n    \\left( p_k - \\frac{\\lambda_\\star - q_k}{4\\mu} \\right)^{-1} \\frac{q_k \\lambda_\\star \\mu}{\\lambda_\\star - q_k} & = \\frac{4\\mu^2}{((1+2\\alpha\\mu)^{\\frac{k}{2}} - 1)^2} \n    = \\frac{(\\sqrt{1+2\\alpha\\mu}+1)^2}{\\alpha^2 \\left( \\sum_{j=0}^{k-1} (1+2\\alpha\\mu)^{\\frac{j}{2}} \\right)^2} .\n\\end{align*}\nTherefore, \\eqref{eqn:sm-EAG_convergence_rate} with $\\lambda = \\lambda_\\star$ concludes the proof.\n\n\n\n","start":53865,"end":55789}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"平移后势函数不增","source_units":[{"raw_tex":"Because the term $\\left( \\frac{1}{2\\alpha}+\\mu \\right) \\|z_0-z_\\star\\|^2$ subtracted from $V_k$ to obtain $W_k$ does not depend on $k$, clearly $W_k$ inherits nonincreasingness from $V_k$, i.e., $W_k \\le \\cdots \\le W_0 = 0$.\n","start":53935,"end":54160}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:sm-EAG_convergence_rate"],"declared_external":false,"source_only":false,"title":"参数化残差上界","source_units":[{"raw_tex":"Therefore, given that $q_k < \\lambda < q_k + 4\\mu p_k$, we have\n\\begin{align}\n    & \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 \\ge W_k + \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 \\ge \\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right) \\|\\opB z_k\\|^2 \\nonumber \\\\\n    & \\implies \\|\\opB z_k\\|^2 \\le \\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right)^{-1} \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 . \\label{eqn:sm-EAG_convergence_rate}\n\\end{align}\n","start":54160,"end":54645}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:sm-EAG_lambda"],"declared_external":false,"source_only":false,"title":"选择最优标量参数","source_units":[{"raw_tex":"One can straightforwardly verify that \n\\begin{align}\n    \\lambda_\\star = (q_k (q_k + 4\\mu p_k))^{1/2}\n    \\label{eqn:sm-EAG_lambda}\n\\end{align}\nlies between $q_k$ and $q_k + 4\\mu p_k$, and minimizes the factor $\\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right)^{-1} \\frac{q_k \\lambda \\mu}{\\lambda - q_k}$ in the right hand side of \\eqref{eqn:sm-EAG_convergence_rate}.\n","start":54645,"end":55011}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"代入系数闭式","source_units":[{"raw_tex":"Plugging in the explicit expressions $q_k = \\frac{(1+2\\alpha\\mu)^k - 1}{2\\alpha\\mu (1+2\\alpha\\mu)^{k-1}}$ and $p_k = \\frac{\\alpha}{2}(1+2\\alpha\\mu)^{k-1} q_k^2 = \\frac{((1+2\\alpha\\mu)^k - 1)^2}{8\\alpha\\mu^2 (1+2\\alpha\\mu)^{k-1}}$ (which uses \\eqref{eqn:sm-EAG_pkqk}) into~\\eqref{eqn:sm-EAG_lambda}, we obtain $\\lambda_\\star = (1+2\\alpha\\mu)^{\\frac{k}{2}} q_k$ and\n","start":55011,"end":55375}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"化简最优速率系数","source_units":[{"raw_tex":"\\begin{align*}\n    \\left( p_k - \\frac{\\lambda_\\star - q_k}{4\\mu} \\right)^{-1} \\frac{q_k \\lambda_\\star \\mu}{\\lambda_\\star - q_k} & = \\frac{4\\mu^2}{((1+2\\alpha\\mu)^{\\frac{k}{2}} - 1)^2} \n    = \\frac{(\\sqrt{1+2\\alpha\\mu}+1)^2}{\\alpha^2 \\left( \\sum_{j=0}^{k-1} (1+2\\alpha\\mu)^{\\frac{j}{2}} \\right)^2} .\n\\end{align*}\n","start":55375,"end":55687}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"代回并结束主证明","source_units":[{"raw_tex":"Therefore, \\eqref{eqn:sm-EAG_convergence_rate} with $\\lambda = \\lambda_\\star$ concludes the proof.\n\n\n\n","start":55687,"end":55789}]},{"id":"S032","semantic_type":"proof","kind":"proof","labels":["eqn:SM-EAG+_zk_minus_zkp1","eqn:sm-EAG_Lyapunov_Lipschitz","eqn:sm-EAG_Lyapunov_monotone","eqn:sm-EAG_pkqk","eqn:sm-EAG_recurrence_p_k","eqn:sm-EAG_recurrence_q_k","eqn:sm-EAG_summable_term","eqn:sm_EAG+_zkhalf_minus_zkp1","eqn:sm_EAG+_zkp1_minus_z0"],"declared_external":false,"source_only":false,"title":"势函数引理完整作者证明","source_units":[{"raw_tex":"\\subsection{Proof of \\texorpdfstring{\\cref{lem:sm-EAG_Lyapunov}}{Lemma 4.4}}\nWhen $k=0$, we have $z_{1/2} = z_0$, $z_1 = z_0 - \\alpha \\opB z_0$, $V_0 = \\left( \\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2$, and\n\\begin{align*}\n    V_1 & = \\frac{\\alpha}{2}\\|\\opB z_1\\|^2 + \\langle \\opB z_1 - \\mu(z_1 - z_0), z_1 - z_0 \\rangle + \\left(\\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2 \\nonumber \\\\\n    & = \\frac{\\alpha}{2}\\|\\opB z_1\\|^2 - \\alpha \\langle \\opB z_1, \\opB z_0 \\rangle - \\alpha^2 \\mu \\|\\opB z_0\\|^2 + \\left(\\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2 . \n\\end{align*}\nBecause $\\opB$ is $L$-Lipschitz,\n\\begin{align*}\n    0 & \\le L^2 \\|z_1 - z_0\\|^2 - \\|\\opB z_1 - \\opB z_0\\|^2 = (\\alpha^2 L^2 - 1) \\|\\opB z_0\\|^2 + 2\\langle \\opB z_0, \\opB z_1 \\rangle - \\|\\opB z_1\\|^2 .\n\\end{align*}\nTherefore,\n\\begin{align*}\n    V_0 - V_1 & = \\alpha^2 \\mu \\|\\opB z_0\\|^2 + \\alpha \\langle \\opB z_0, \\opB z_1 \\rangle - \\frac{\\alpha}{2} \\|\\opB z_1\\|^2 \\\\\n    & = \\frac{\\alpha}{2} (1+2\\alpha\\mu -\\alpha^2 L^2) \\|\\opB z_0\\|^2 + \\frac{\\alpha}{2} \\left( (\\alpha^2 L^2 - 1) \\|\\opB z_0\\|^2 + 2\\langle \\opB z_0, \\opB z_1 \\rangle - \\|\\opB z_1\\|^2 \\right) \\\\\n    & \\ge \\frac{\\alpha}{2} (1+2\\alpha\\mu -\\alpha^2 L^2) \\|\\opB z_0\\|^2 \\\\\n    & = \\frac{\\alpha}{2} (1+2\\alpha\\mu -\\alpha^2 L^2) \\|\\eta_0 \\opB z_0 - \\opB z_{1/2}\\|^2 \n\\end{align*}\nwhere the last equality holds because $\\eta_0 = 0$ and $z_{1/2} = z_0$.\n\nNow suppose $k\\ge 1$. For \\ref{eqn:SM-EAG+}, we have the identities\n\\begin{align}\n    z_{k+1} - z_0 & = (1-\\beta_k) (z_k - z_0) - \\alpha \\opB z_{k+1/2} \\label{eqn:sm_EAG+_zkp1_minus_z0} \\\\\n    z_k - z_{k+1} & = \\beta_k (z_k - z_0) + \\alpha \\opB z_{k+1/2} \\label{eqn:SM-EAG+_zk_minus_zkp1} \\\\\n    z_{k+1/2} - z_{k+1} & = \\alpha \\opB z_{k+1/2} - \\eta_k \\alpha \\opB z_k . \\label{eqn:sm_EAG+_zkhalf_minus_zkp1}\n\\end{align}\nNote that $\\beta_k = \\frac{1}{\\sum_{j=0}^{k} (1+2\\alpha\\mu)^j}$ satisfies\n\\begin{align*}\n    \\frac{1}{\\beta_{k}} = \\sum_{j=0}^{k} (1+2\\alpha\\mu)^j = 1 + (1+2\\alpha\\mu) \\sum_{j=0}^{k-1} (1+2\\alpha\\mu)^j = 1 + \\frac{1+2\\alpha\\mu}{\\beta_{k-1}} \n\\end{align*}\nfor $k=1,2,\\dots$, so $q_k = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{1}{\\beta_{k-1}}$ satisfies the recurrence relation\n\\begin{align}\n    q_{k+1} = \\frac{q_k}{\\beta_k} \\frac{\\beta_{k-1}}{(1+2\\alpha\\mu)} =\n    \\frac{q_k}{\\beta_k} \\frac{1}{\\frac{1}{\\beta_k} - 1} = \\frac{q_k}{1-\\beta_k}\n    \\label{eqn:sm-EAG_recurrence_q_k}\n\\end{align}\nfor $k=1,2,\\dots$.\nAdditionally, because $\\eta_k = \\frac{1-\\beta_k}{1+2\\alpha\\mu}$ and $p_k = \\frac{\\eta_k \\alpha q_k}{2\\beta_k}$,\n\\begin{align}\n    p_k = \\frac{\\alpha (1-\\beta_k) q_k}{2\\beta_k (1+2\\alpha\\mu)} = \\frac{\\alpha}{2} \\left( \\frac{1}{\\beta_k} - 1 \\right) \\frac{q_k}{1 + 2\\alpha\\mu} = \\frac{\\alpha}{2} \\frac{1}{\\beta_{k-1}} q_k = \\frac{\\alpha}{2} (1+2\\alpha\\mu)^{k-1} q_k^2 \n    \\label{eqn:sm-EAG_pkqk}\n\\end{align}\nand\n\\begin{align}\n    p_{k+1} & = \\frac{\\eta_{k+1}\\alpha q_{k+1}}{2\\beta_{k+1}} = \\frac{\\alpha q_{k+1}}{2\\beta_{k+1}} \\frac{1-\\beta_{k+1}}{1+2\\alpha\\mu} = \\frac{\\alpha q_k}{2(1-\\beta_k)} \\frac{1-\\beta_{k+1}}{(1+2\\alpha\\mu)\\beta_{k+1}} \\nonumber \\\\\n    & = \\frac{\\alpha q_k}{2(1-\\beta_k)} \\frac{1}{1+2\\alpha\\mu} \\left( \\frac{1}{\\beta_{k+1}} - 1 \\right) = \\frac{\\alpha q_k}{2\\beta_k (1-\\beta_k)} . \n    \\label{eqn:sm-EAG_recurrence_p_k}\n\\end{align}\nThe identities~\\eqref{eqn:sm-EAG_recurrence_q_k} through \\eqref{eqn:sm-EAG_recurrence_p_k} will be useful for the subsequent arguments.\n\nUsing $\\mu$-strong monotonicity of $\\opB$ and the identities~\\eqref{eqn:sm_EAG+_zkp1_minus_z0}, \\eqref{eqn:SM-EAG+_zk_minus_zkp1} and \\eqref{eqn:sm-EAG_recurrence_q_k}, we have\n\\begin{align}\n    V_k - V_{k+1} & \\ge V_k - V_{k+1} - \\frac{q_k}{\\beta_k} \\langle z_k - z_{k+1}, \\opB z_k - \\opB z_{k+1} - \\mu(z_k - z_{k+1}) \\rangle \\nonumber \\\\\n    & = p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k , z_k-z_0 \\rangle - q_k \\mu \\|z_k - z_0\\|^2 \\nonumber \\\\\n    & \\quad - p_{k+1} \\|\\opB z_{k+1}\\|^2 - q_{k+1} \\langle \\opB z_{k+1} , (1-\\beta_k)(z_k - z_0) - \\alpha \\opB z_{k+1/2} \\rangle \\nonumber \\\\\n    & \\quad + q_{k+1} \\mu \\left\\| (1-\\beta_k) (z_k - z_0) - \\alpha \\opB z_{k+1/2} \\right\\|^2 \\nonumber \\\\\n    & \\quad - \\frac{q_k}{\\beta_k} \\langle \\beta_k (z_k - z_0) + \\alpha \\opB z_{k+1/2} , \\opB z_k - \\opB z_{k+1} \\rangle \\nonumber \\\\\n    & \\quad + \\frac{q_k \\mu}{\\beta_k} \\left\\| \\beta_k (z_k - z_0) + \\alpha \\opB z_{k+1/2} \\right\\|^2 \\nonumber \\\\\n    & \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 - \\frac{\\alpha q_k}{\\beta_k} \\langle \\opB z_k, \\opB z_{k+1/2} \\rangle + \\mu \\alpha^2 \\left( \\frac{q_k}{\\beta_k} + q_{k+1} \\right) \\|\\opB z_{k+1/2}\\|^2 \\\\\n        & \\quad + \\alpha \\left( \\frac{q_k}{\\beta_k} + q_{k+1} \\right) \\langle \\opB z_{k+1/2} , \\opB z_{k+1} \\rangle - p_{k+1} \\|\\opB z_{k+1}\\|^2 \\\\\n        & \\quad + 2\\mu\\alpha \\left( q_k - (1-\\beta_k)q_{k+1} \\right) \\langle \\opB z_{k+1/2}, z_k - z_0 \\rangle \\\\\n        & \\quad + (q_k - (1-\\beta_k) q_{k+1}) \\langle \\opB z_{k+1}, z_k - z_0 \\rangle \\\\\n        & \\quad + \\mu (1-\\beta_k) ((1-\\beta_k) q_{k+1} - q_k) \\|z_k - z_0\\|^2 \n    \\end{aligned} \\nonumber \\\\\n    & \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 - \\frac{\\alpha q_k}{\\beta_k} \\langle \\opB z_k, \\opB z_{k+1/2} \\rangle + \\frac{\\mu \\alpha^2 q_k}{\\beta_k (1-\\beta_k)} \\|\\opB z_{k+1/2}\\|^2 \\\\\n        & \\quad + \\frac{\\alpha q_k}{\\beta_k (1-\\beta_k)} \\langle \\opB z_{k+1/2} , \\opB z_{k+1} \\rangle - p_{k+1} \\|\\opB z_{k+1}\\|^2 .\n    \\end{aligned} \\label{eqn:sm-EAG_Lyapunov_monotone}\n\\end{align}\nNow because $\\opB$ is $L$-Lipschitz,\n\\begin{align}\n\\label{eqn:sm-EAG_Lyapunov_Lipschitz}\n    \\|\\opB z_{k+1/2} - \\opB z_{k+1}\\|^2 \\le L^2 \\|z_{k+1/2} - z_{k+1}\\|^2 = L^2 \\|\\alpha \\opB z_{k+1/2} - \\eta_k \\alpha \\opB z_k\\|^2 \n\\end{align}\nwhere we applied the identity~\\eqref{eqn:sm_EAG+_zkhalf_minus_zkp1} for the last equality.\nMultiplying $p_{k+1}$ to \\eqref{eqn:sm-EAG_Lyapunov_Lipschitz} and then subtracting it from \\eqref{eqn:sm-EAG_Lyapunov_monotone}, we obtain\n\\begin{align}\n    V_k - V_{k+1} & \\ge ( p_k - p_{k+1} \\eta_k^2 \\alpha^2 L^2) \\|\\opB z_k\\|^2 - \\left( \\frac{\\alpha q_k}{\\beta_k} - 2p_{k+1} \\eta_k \\alpha^2 L^2 \\right) \\langle \\opB z_k, \\opB z_{k+1/2} \\rangle \\nonumber \\\\\n    & \\quad + \\left( \\frac{\\mu\\alpha^2 q_k}{\\beta_k (1-\\beta_k)} + p_{k+1}(1-\\alpha^2 L^2) \\right) \\|\\opB z_{k+1/2}\\|^2 \\nonumber \\\\\n    & \\quad + \\left( \\frac{\\alpha q_k}{\\beta_k (1-\\beta_k)} - 2p_{k+1} \\right) \\langle \\opB z_{k+1/2} , \\opB z_{k+1} \\rangle \\nonumber \\\\\n    & = \\frac{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2) q_k}{2\\beta_k (1-\\beta_k)} \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2\n    \\label{eqn:sm-EAG_summable_term}\n\\end{align}\nwhere the last equality follows from the identities $\\eta_k = \\frac{1-\\beta_k}{1+2\\alpha\\mu}$, $p_k = \\frac{\\eta_k \\alpha q_k}{2\\beta_k}$ and \\eqref{eqn:sm-EAG_recurrence_p_k}.\nAs $1 + 2\\alpha\\mu - \\alpha^2 L^2 \\ge 0$ if $0 < \\alpha \\le \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2}$, for this range of $\\alpha$,\n\\begin{align*}\n    V_k - V_{k+1} & \\ge \\frac{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2)}{2} (1+2\\alpha\\mu)^k \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2 \\ge 0,\n\\end{align*}\nwhere we used $\\frac{q_k}{\\beta_k (1-\\beta_k)} = \\frac{q_{k+1}}{\\beta_k} > (1+2\\alpha\\mu)^k$.\nNow it only remains to show that $V_N \\ge 0$ holds for any $N \\in \\mathbb{N}$; if that is the case, then\n\\begin{align*}\n    & \\sum_{k=0}^N (1+2\\alpha\\mu)^k \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2 \\le \\frac{2}{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2)} (V_0 - V_{N+1}) \\\\\n    & \\quad \\le \\frac{2}{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2)} \\left( \\frac{1}{2\\alpha} + \\mu \\right) \\|z_0 - z_\\star\\|^2 = \\frac{1+2\\alpha\\mu}{\\alpha^2 (1 + 2\\alpha\\mu - \\alpha^2 L^2)} \\|z_0 - z_\\star\\|^2 ,\n\\end{align*}\nand taking $N\\to\\infty$, we obtain~\\eqref{eqn:sm-EAG_summability}.\n\nConsider the following shifted version of $V_k$:\n\\begin{align*}\n    W_k = V_k - \\left(\\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2 = p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k - \\mu (z_k - z_0), z_k - z_0 \\rangle .\n\\end{align*}\nBecause $\\opB$ is $\\mu$-strongly monotone, if $\\lambda \\ge 0$ and $\\lambda \\ne q_k$,\n\\begin{align}\n    W_k & \\ge p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k - \\mu (z_k - z_0), z_k - z_0 \\rangle - \\lambda \\langle \\opB z_k - \\mu (z_k - z_\\star), z_k - z_\\star \\rangle \\nonumber \\\\\n    &\n    \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 + \\langle \\opB z_k , (q_k - \\lambda) z_k - q_k z_0 + \\lambda z_\\star \\rangle \\\\\n        & \\quad + \\mu(\\lambda-q_k) \\|z_k\\|^2 + 2 \\mu \\langle z_k, q_k z_0 - \\lambda z_\\star \\rangle - q_k \\mu \\|z_0\\|^2 + \\lambda\\mu \\|z_\\star\\|^2  \n    \\end{aligned} \\nonumber \\\\\n    &\n    \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 + \\langle \\opB z_k , (q_k - \\lambda) z_k - q_k z_0 + \\lambda z_\\star \\rangle \\\\\n        & \\quad + \\frac{\\mu}{\\lambda - q_k} \\left\\| (\\lambda - q_k) z_k + q_k z_0 - \\lambda z_\\star \\right\\|^2 - \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 .\n    \\end{aligned} \\nonumber \\\\ \n    &\n    \\begin{aligned}\n        & = \\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right) \\|\\opB z_k\\|^2 + \\frac{\\mu}{\\lambda - q_k} \\left\\| \\frac{\\lambda - q_k}{2\\mu} \\opB z_k - (\\lambda - q_k) z_k - q_k z_0 + \\lambda z_\\star \\right\\|^2 \\\\\n        & \\quad - \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 .\n    \\end{aligned} \\nonumber\n\\end{align}\nAdding $\\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2$ to the both sides and taking $\\lambda = q_k + 4\\mu p_k$, we obtain \n\\begin{align*}\n    0 & \\le W_k + \\left( \\frac{q_k^2}{4p_k} + \\mu q_k \\right) \\|z_0 - z_\\star\\|^2 = V_k ,\n\\end{align*}\nwhere the last equality follows by applying \\eqref{eqn:sm-EAG_pkqk} and then plugging in the expression $q_k = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{1}{\\beta_{k-1}} = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{(1+2\\alpha\\mu)^k - 1}{2\\alpha\\mu}$.\n\n\n","start":90433,"end":100120}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"初始迭代与势函数值","source_units":[{"raw_tex":"When $k=0$, we have $z_{1/2} = z_0$, $z_1 = z_0 - \\alpha \\opB z_0$, $V_0 = \\left( \\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2$, and\n\\begin{align*}\n    V_1 & = \\frac{\\alpha}{2}\\|\\opB z_1\\|^2 + \\langle \\opB z_1 - \\mu(z_1 - z_0), z_1 - z_0 \\rangle + \\left(\\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2 \\nonumber \\\\\n    & = \\frac{\\alpha}{2}\\|\\opB z_1\\|^2 - \\alpha \\langle \\opB z_1, \\opB z_0 \\rangle - \\alpha^2 \\mu \\|\\opB z_0\\|^2 + \\left(\\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2 . \n\\end{align*}\n","start":90510,"end":91022}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"初始步的 Lipschitz 约束","source_units":[{"raw_tex":"Because $\\opB$ is $L$-Lipschitz,\n\\begin{align*}\n    0 & \\le L^2 \\|z_1 - z_0\\|^2 - \\|\\opB z_1 - \\opB z_0\\|^2 = (\\alpha^2 L^2 - 1) \\|\\opB z_0\\|^2 + 2\\langle \\opB z_0, \\opB z_1 \\rangle - \\|\\opB z_1\\|^2 .\n\\end{align*}\n","start":91022,"end":91236}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"得到初始下降界","source_units":[{"raw_tex":"Therefore,\n\\begin{align*}\n    V_0 - V_1 & = \\alpha^2 \\mu \\|\\opB z_0\\|^2 + \\alpha \\langle \\opB z_0, \\opB z_1 \\rangle - \\frac{\\alpha}{2} \\|\\opB z_1\\|^2 \\\\\n    & = \\frac{\\alpha}{2} (1+2\\alpha\\mu -\\alpha^2 L^2) \\|\\opB z_0\\|^2 + \\frac{\\alpha}{2} \\left( (\\alpha^2 L^2 - 1) \\|\\opB z_0\\|^2 + 2\\langle \\opB z_0, \\opB z_1 \\rangle - \\|\\opB z_1\\|^2 \\right) \\\\\n    & \\ge \\frac{\\alpha}{2} (1+2\\alpha\\mu -\\alpha^2 L^2) \\|\\opB z_0\\|^2 \\\\\n    & = \\frac{\\alpha}{2} (1+2\\alpha\\mu -\\alpha^2 L^2) \\|\\eta_0 \\opB z_0 - \\opB z_{1/2}\\|^2 \n\\end{align*}\nwhere the last equality holds because $\\eta_0 = 0$ and $z_{1/2} = z_0$.\n","start":91236,"end":91835}]},{"id":"S036","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:SM-EAG+_zk_minus_zkp1","eqn:sm_EAG+_zkhalf_minus_zkp1","eqn:sm_EAG+_zkp1_minus_z0"],"declared_external":false,"source_only":false,"title":"三条迭代差恒等式","source_units":[{"raw_tex":"Now suppose $k\\ge 1$. For \\ref{eqn:SM-EAG+}, we have the identities\n\\begin{align}\n    z_{k+1} - z_0 & = (1-\\beta_k) (z_k - z_0) - \\alpha \\opB z_{k+1/2} \\label{eqn:sm_EAG+_zkp1_minus_z0} \\\\\n    z_k - z_{k+1} & = \\beta_k (z_k - z_0) + \\alpha \\opB z_{k+1/2} \\label{eqn:SM-EAG+_zk_minus_zkp1} \\\\\n    z_{k+1/2} - z_{k+1} & = \\alpha \\opB z_{k+1/2} - \\eta_k \\alpha \\opB z_k . \\label{eqn:sm_EAG+_zkhalf_minus_zkp1}\n\\end{align}\n","start":91836,"end":92255}]},{"id":"S037","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"锚定系数的递推","source_units":[{"raw_tex":"Note that $\\beta_k = \\frac{1}{\\sum_{j=0}^{k} (1+2\\alpha\\mu)^j}$ satisfies\n\\begin{align*}\n    \\frac{1}{\\beta_{k}} = \\sum_{j=0}^{k} (1+2\\alpha\\mu)^j = 1 + (1+2\\alpha\\mu) \\sum_{j=0}^{k-1} (1+2\\alpha\\mu)^j = 1 + \\frac{1+2\\alpha\\mu}{\\beta_{k-1}} \n\\end{align*}\n","start":92255,"end":92510}]},{"id":"S038","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:sm-EAG_recurrence_q_k"],"declared_external":false,"source_only":false,"title":"q 系数递推","source_units":[{"raw_tex":"for $k=1,2,\\dots$, so $q_k = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{1}{\\beta_{k-1}}$ satisfies the recurrence relation\n\\begin{align}\n    q_{k+1} = \\frac{q_k}{\\beta_k} \\frac{\\beta_{k-1}}{(1+2\\alpha\\mu)} =\n    \\frac{q_k}{\\beta_k} \\frac{1}{\\frac{1}{\\beta_k} - 1} = \\frac{q_k}{1-\\beta_k}\n    \\label{eqn:sm-EAG_recurrence_q_k}\n\\end{align}\nfor $k=1,2,\\dots$.\n","start":92510,"end":92861}]},{"id":"S039","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:sm-EAG_pkqk"],"declared_external":false,"source_only":false,"title":"p 与 q 的关键恒等式","source_units":[{"raw_tex":"Additionally, because $\\eta_k = \\frac{1-\\beta_k}{1+2\\alpha\\mu}$ and $p_k = \\frac{\\eta_k \\alpha q_k}{2\\beta_k}$,\n\\begin{align}\n    p_k = \\frac{\\alpha (1-\\beta_k) q_k}{2\\beta_k (1+2\\alpha\\mu)} = \\frac{\\alpha}{2} \\left( \\frac{1}{\\beta_k} - 1 \\right) \\frac{q_k}{1 + 2\\alpha\\mu} = \\frac{\\alpha}{2} \\frac{1}{\\beta_{k-1}} q_k = \\frac{\\alpha}{2} (1+2\\alpha\\mu)^{k-1} q_k^2 \n    \\label{eqn:sm-EAG_pkqk}\n\\end{align}\n","start":92861,"end":93267}]},{"id":"S040","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:sm-EAG_recurrence_p_k"],"declared_external":false,"source_only":false,"title":"下一步 p 系数","source_units":[{"raw_tex":"and\n\\begin{align}\n    p_{k+1} & = \\frac{\\eta_{k+1}\\alpha q_{k+1}}{2\\beta_{k+1}} = \\frac{\\alpha q_{k+1}}{2\\beta_{k+1}} \\frac{1-\\beta_{k+1}}{1+2\\alpha\\mu} = \\frac{\\alpha q_k}{2(1-\\beta_k)} \\frac{1-\\beta_{k+1}}{(1+2\\alpha\\mu)\\beta_{k+1}} \\nonumber \\\\\n    & = \\frac{\\alpha q_k}{2(1-\\beta_k)} \\frac{1}{1+2\\alpha\\mu} \\left( \\frac{1}{\\beta_{k+1}} - 1 \\right) = \\frac{\\alpha q_k}{2\\beta_k (1-\\beta_k)} . \n    \\label{eqn:sm-EAG_recurrence_p_k}\n\\end{align}\nThe identities~\\eqref{eqn:sm-EAG_recurrence_q_k} through \\eqref{eqn:sm-EAG_recurrence_p_k} will be useful for the subsequent arguments.\n","start":93267,"end":93850}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"加入强单调非负项","source_units":[{"raw_tex":"Using $\\mu$-strong monotonicity of $\\opB$ and the identities~\\eqref{eqn:sm_EAG+_zkp1_minus_z0}, \\eqref{eqn:SM-EAG+_zk_minus_zkp1} and \\eqref{eqn:sm-EAG_recurrence_q_k}, we have\n\\begin{align}\n    V_k - V_{k+1} & \\ge V_k - V_{k+1} - \\frac{q_k}{\\beta_k} \\langle z_k - z_{k+1}, \\opB z_k - \\opB z_{k+1} - \\mu(z_k - z_{k+1}) \\rangle \\nonumber \\\\\n    & = p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k , z_k-z_0 \\rangle - q_k \\mu \\|z_k - z_0\\|^2 \\nonumber \\\\\n    & \\quad - p_{k+1} \\|\\opB z_{k+1}\\|^2 - q_{k+1} \\langle \\opB z_{k+1} , (1-\\beta_k)(z_k - z_0) - \\alpha \\opB z_{k+1/2} \\rangle \\nonumber \\\\\n    & \\quad + q_{k+1} \\mu \\left\\| (1-\\beta_k) (z_k - z_0) - \\alpha \\opB z_{k+1/2} \\right\\|^2 \\nonumber \\\\\n    & \\quad - \\frac{q_k}{\\beta_k} \\langle \\beta_k (z_k - z_0) + \\alpha \\opB z_{k+1/2} , \\opB z_k - \\opB z_{k+1} \\rangle \\nonumber \\\\\n    & \\quad + \\frac{q_k \\mu}{\\beta_k} \\left\\| \\beta_k (z_k - z_0) + \\alpha \\opB z_{k+1/2} \\right\\|^2 \\nonumber \\\\\n    & \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 - \\frac{\\alpha q_k}{\\beta_k} \\langle \\opB z_k, \\opB z_{k+1/2} \\rangle + \\mu \\alpha^2 \\left( \\frac{q_k}{\\beta_k} + q_{k+1} \\right) \\|\\opB z_{k+1/2}\\|^2 \\\\\n        & \\quad + \\alpha \\left( \\frac{q_k}{\\beta_k} + q_{k+1} \\right) \\langle \\opB z_{k+1/2} , \\opB z_{k+1} \\rangle - p_{k+1} \\|\\opB z_{k+1}\\|^2 \\\\\n        & \\quad + 2\\mu\\alpha \\left( q_k - (1-\\beta_k)q_{k+1} \\right) \\langle \\opB z_{k+1/2}, z_k - z_0 \\rangle \\\\\n        & \\quad + (q_k - (1-\\beta_k) q_{k+1}) \\langle \\opB z_{k+1}, z_k - z_0 \\rangle \\\\\n        & \\quad + \\mu (1-\\beta_k) ((1-\\beta_k) q_{k+1} - q_k) \\|z_k - z_0\\|^2 \n    \\end{aligned} \\nonumber \\\\\n","start":93851,"end":95461}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"消去锚点交叉项","source_units":[{"raw_tex":"    & \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 - \\frac{\\alpha q_k}{\\beta_k} \\langle \\opB z_k, \\opB z_{k+1/2} \\rangle + \\frac{\\mu \\alpha^2 q_k}{\\beta_k (1-\\beta_k)} \\|\\opB z_{k+1/2}\\|^2 \\\\\n        & \\quad + \\frac{\\alpha q_k}{\\beta_k (1-\\beta_k)} \\langle \\opB z_{k+1/2} , \\opB z_{k+1} \\rangle - p_{k+1} \\|\\opB z_{k+1}\\|^2 .\n    \\end{aligned} \\label{eqn:sm-EAG_Lyapunov_monotone}\n\\end{align}\n","start":95461,"end":95856}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:sm-EAG_Lyapunov_Lipschitz"],"declared_external":false,"source_only":false,"title":"中间点的 Lipschitz 估计","source_units":[{"raw_tex":"Now because $\\opB$ is $L$-Lipschitz,\n\\begin{align}\n\\label{eqn:sm-EAG_Lyapunov_Lipschitz}\n    \\|\\opB z_{k+1/2} - \\opB z_{k+1}\\|^2 \\le L^2 \\|z_{k+1/2} - z_{k+1}\\|^2 = L^2 \\|\\alpha \\opB z_{k+1/2} - \\eta_k \\alpha \\opB z_k\\|^2 \n\\end{align}\nwhere we applied the identity~\\eqref{eqn:sm_EAG+_zkhalf_minus_zkp1} for the last equality.\n","start":95856,"end":96182}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:sm-EAG_summable_term"],"declared_external":false,"source_only":false,"title":"组合为误差平方下降","source_units":[{"raw_tex":"Multiplying $p_{k+1}$ to \\eqref{eqn:sm-EAG_Lyapunov_Lipschitz} and then subtracting it from \\eqref{eqn:sm-EAG_Lyapunov_monotone}, we obtain\n\\begin{align}\n    V_k - V_{k+1} & \\ge ( p_k - p_{k+1} \\eta_k^2 \\alpha^2 L^2) \\|\\opB z_k\\|^2 - \\left( \\frac{\\alpha q_k}{\\beta_k} - 2p_{k+1} \\eta_k \\alpha^2 L^2 \\right) \\langle \\opB z_k, \\opB z_{k+1/2} \\rangle \\nonumber \\\\\n    & \\quad + \\left( \\frac{\\mu\\alpha^2 q_k}{\\beta_k (1-\\beta_k)} + p_{k+1}(1-\\alpha^2 L^2) \\right) \\|\\opB z_{k+1/2}\\|^2 \\nonumber \\\\\n    & \\quad + \\left( \\frac{\\alpha q_k}{\\beta_k (1-\\beta_k)} - 2p_{k+1} \\right) \\langle \\opB z_{k+1/2} , \\opB z_{k+1} \\rangle \\nonumber \\\\\n    & = \\frac{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2) q_k}{2\\beta_k (1-\\beta_k)} \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2\n    \\label{eqn:sm-EAG_summable_term}\n\\end{align}\nwhere the last equality follows from the identities $\\eta_k = \\frac{1-\\beta_k}{1+2\\alpha\\mu}$, $p_k = \\frac{\\eta_k \\alpha q_k}{2\\beta_k}$ and \\eqref{eqn:sm-EAG_recurrence_p_k}.\n","start":96182,"end":97159}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"步长保证非负下降","source_units":[{"raw_tex":"As $1 + 2\\alpha\\mu - \\alpha^2 L^2 \\ge 0$ if $0 < \\alpha \\le \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2}$, for this range of $\\alpha$,\n\\begin{align*}\n    V_k - V_{k+1} & \\ge \\frac{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2)}{2} (1+2\\alpha\\mu)^k \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2 \\ge 0,\n\\end{align*}\nwhere we used $\\frac{q_k}{\\beta_k (1-\\beta_k)} = \\frac{q_{k+1}}{\\beta_k} > (1+2\\alpha\\mu)^k$.\n","start":97159,"end":97544}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"条件性望远镜求和","source_units":[{"raw_tex":"Now it only remains to show that $V_N \\ge 0$ holds for any $N \\in \\mathbb{N}$; if that is the case, then\n\\begin{align*}\n    & \\sum_{k=0}^N (1+2\\alpha\\mu)^k \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2 \\le \\frac{2}{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2)} (V_0 - V_{N+1}) \\\\\n    & \\quad \\le \\frac{2}{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2)} \\left( \\frac{1}{2\\alpha} + \\mu \\right) \\|z_0 - z_\\star\\|^2 = \\frac{1+2\\alpha\\mu}{\\alpha^2 (1 + 2\\alpha\\mu - \\alpha^2 L^2)} \\|z_0 - z_\\star\\|^2 ,\n\\end{align*}\nand taking $N\\to\\infty$, we obtain~\\eqref{eqn:sm-EAG_summability}.\n","start":97544,"end":98101}]},{"id":"S047","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"平移势函数","source_units":[{"raw_tex":"Consider the following shifted version of $V_k$:\n\\begin{align*}\n    W_k = V_k - \\left(\\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2 = p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k - \\mu (z_k - z_0), z_k - z_0 \\rangle .\n\\end{align*}\n","start":98102,"end":98335}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"相对零点应用强单调性","source_units":[{"raw_tex":"Because $\\opB$ is $\\mu$-strongly monotone, if $\\lambda \\ge 0$ and $\\lambda \\ne q_k$,\n\\begin{align}\n    W_k & \\ge p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k - \\mu (z_k - z_0), z_k - z_0 \\rangle - \\lambda \\langle \\opB z_k - \\mu (z_k - z_\\star), z_k - z_\\star \\rangle \\nonumber \\\\\n    &\n    \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 + \\langle \\opB z_k , (q_k - \\lambda) z_k - q_k z_0 + \\lambda z_\\star \\rangle \\\\\n        & \\quad + \\mu(\\lambda-q_k) \\|z_k\\|^2 + 2 \\mu \\langle z_k, q_k z_0 - \\lambda z_\\star \\rangle - q_k \\mu \\|z_0\\|^2 + \\lambda\\mu \\|z_\\star\\|^2  \n    \\end{aligned} \\nonumber \\\\\n","start":98335,"end":98929}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"第一次配方","source_units":[{"raw_tex":"    &\n    \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 + \\langle \\opB z_k , (q_k - \\lambda) z_k - q_k z_0 + \\lambda z_\\star \\rangle \\\\\n        & \\quad + \\frac{\\mu}{\\lambda - q_k} \\left\\| (\\lambda - q_k) z_k + q_k z_0 - \\lambda z_\\star \\right\\|^2 - \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 .\n    \\end{aligned} \\nonumber \\\\ \n","start":98929,"end":99271}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"算子残差的配方下界","source_units":[{"raw_tex":"    &\n    \\begin{aligned}\n        & = \\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right) \\|\\opB z_k\\|^2 + \\frac{\\mu}{\\lambda - q_k} \\left\\| \\frac{\\lambda - q_k}{2\\mu} \\opB z_k - (\\lambda - q_k) z_k - q_k z_0 + \\lambda z_\\star \\right\\|^2 \\\\\n        & \\quad - \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 .\n    \\end{aligned} \\nonumber\n\\end{align}\n","start":99271,"end":99626}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"特殊参数证明非负","source_units":[{"raw_tex":"Adding $\\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2$ to the both sides and taking $\\lambda = q_k + 4\\mu p_k$, we obtain \n\\begin{align*}\n    0 & \\le W_k + \\left( \\frac{q_k^2}{4p_k} + \\mu q_k \\right) \\|z_0 - z_\\star\\|^2 = V_k ,\n\\end{align*}\nwhere the last equality follows by applying \\eqref{eqn:sm-EAG_pkqk} and then plugging in the expression $q_k = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{1}{\\beta_{k-1}} = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{(1+2\\alpha\\mu)^k - 1}{2\\alpha\\mu}$.\n\n\n","start":99626,"end":100120}]},{"id":"S052","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["任务原始先决事实列表：Standard Euclidean inner-product identities, Young's inequality, and completing the square。"],"source_only":false,"title":"欧氏内积恒等式","source_units":[]},{"id":"S053","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["任务原始先决事实列表中的 completing the square。"],"source_only":false,"title":"二次式配方","source_units":[]},{"id":"S054","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["任务原始先决事实列表：Elementary geometric-series identities and scalar minimization。"],"source_only":false,"title":"有限几何级数","source_units":[]},{"id":"S055","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["任务原始先决事实列表中的 scalar minimization。"],"source_only":false,"title":"一维系数最小化","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S016","edges":[{"source_id":"S001","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S007","kind":"supports"},{"source_id":"S001","target_id":"S009","kind":"supports"},{"source_id":"S003","target_id":"S009","kind":"supports"},{"source_id":"S007","target_id":"S009","kind":"supports"},{"source_id":"S004","target_id":"S015","kind":"supports"},{"source_id":"S006","target_id":"S015","kind":"supports"},{"source_id":"S011","target_id":"S016","kind":"supports"},{"source_id":"S012","target_id":"S016","kind":"supports"},{"source_id":"S011","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S018","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S011","target_id":"S033","kind":"supports"},{"source_id":"S012","target_id":"S033","kind":"supports"},{"source_id":"S018","target_id":"S033","kind":"supports"},{"source_id":"S019","target_id":"S033","kind":"supports"},{"source_id":"S017","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S052","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S012","target_id":"S035","kind":"supports"},{"source_id":"S052","target_id":"S035","kind":"supports"},{"source_id":"S011","target_id":"S036","kind":"supports"},{"source_id":"S012","target_id":"S037","kind":"supports"},{"source_id":"S054","target_id":"S037","kind":"supports"},{"source_id":"S019","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S012","target_id":"S039","kind":"supports"},{"source_id":"S019","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S012","target_id":"S040","kind":"supports"},{"source_id":"S019","target_id":"S040","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S006","target_id":"S041","kind":"supports"},{"source_id":"S017","target_id":"S041","kind":"supports"},{"source_id":"S018","target_id":"S041","kind":"supports"},{"source_id":"S036","target_id":"S041","kind":"supports"},{"source_id":"S038","target_id":"S041","kind":"supports"},{"source_id":"S052","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S038","target_id":"S042","kind":"supports"},{"source_id":"S017","target_id":"S043","kind":"supports"},{"source_id":"S036","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S040","target_id":"S044","kind":"supports"},{"source_id":"S019","target_id":"S044","kind":"supports"},{"source_id":"S012","target_id":"S044","kind":"supports"},{"source_id":"S052","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S012","target_id":"S045","kind":"supports"},{"source_id":"S019","target_id":"S045","kind":"supports"},{"source_id":"S038","target_id":"S045","kind":"supports"},{"source_id":"S018","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S006","target_id":"S048","kind":"supports"},{"source_id":"S017","target_id":"S048","kind":"supports"},{"source_id":"S015","target_id":"S048","kind":"supports"},{"source_id":"S052","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S053","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S039","target_id":"S051","kind":"supports"},{"source_id":"S019","target_id":"S051","kind":"supports"},{"source_id":"S012","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S051","kind":"supports"},{"source_id":"S033","target_id":"S021","kind":"supports"},{"source_id":"S035","target_id":"S021","kind":"supports"},{"source_id":"S051","target_id":"S021","kind":"supports"},{"source_id":"S044","target_id":"S022","kind":"supports"},{"source_id":"S035","target_id":"S023","kind":"supports"},{"source_id":"S045","target_id":"S023","kind":"supports"},{"source_id":"S033","target_id":"S023","kind":"supports"},{"source_id":"S051","target_id":"S023","kind":"supports"},{"source_id":"S035","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S033","target_id":"S046","kind":"supports"},{"source_id":"S051","target_id":"S046","kind":"supports"},{"source_id":"S012","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S024","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S032","kind":"supports"},{"source_id":"S035","target_id":"S032","kind":"supports"},{"source_id":"S036","target_id":"S032","kind":"supports"},{"source_id":"S037","target_id":"S032","kind":"supports"},{"source_id":"S038","target_id":"S032","kind":"supports"},{"source_id":"S039","target_id":"S032","kind":"supports"},{"source_id":"S040","target_id":"S032","kind":"supports"},{"source_id":"S041","target_id":"S032","kind":"supports"},{"source_id":"S042","target_id":"S032","kind":"supports"},{"source_id":"S043","target_id":"S032","kind":"supports"},{"source_id":"S044","target_id":"S032","kind":"supports"},{"source_id":"S045","target_id":"S032","kind":"supports"},{"source_id":"S046","target_id":"S032","kind":"supports"},{"source_id":"S047","target_id":"S032","kind":"supports"},{"source_id":"S048","target_id":"S032","kind":"supports"},{"source_id":"S049","target_id":"S032","kind":"supports"},{"source_id":"S050","target_id":"S032","kind":"supports"},{"source_id":"S051","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S020","kind":"supports"},{"source_id":"S023","target_id":"S026","kind":"supports"},{"source_id":"S047","target_id":"S026","kind":"supports"},{"source_id":"S019","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S050","target_id":"S027","kind":"supports"},{"source_id":"S019","target_id":"S027","kind":"supports"},{"source_id":"S012","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S019","target_id":"S028","kind":"supports"},{"source_id":"S012","target_id":"S028","kind":"supports"},{"source_id":"S055","target_id":"S028","kind":"supports"},{"source_id":"S019","target_id":"S029","kind":"supports"},{"source_id":"S012","target_id":"S029","kind":"supports"},{"source_id":"S039","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S054","target_id":"S029","kind":"supports"},{"source_id":"S027","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S054","target_id":"S030","kind":"supports"},{"source_id":"S027","target_id":"S031","kind":"supports"},{"source_id":"S028","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S027","target_id":"S025","kind":"supports"},{"source_id":"S028","target_id":"S025","kind":"supports"},{"source_id":"S029","target_id":"S025","kind":"supports"},{"source_id":"S030","target_id":"S025","kind":"supports"},{"source_id":"S031","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S016","kind":"supports"},{"source_id":"S006","target_id":"S017","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S016","edges":[{"source_id":"S001","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S007","kind":"supports"},{"source_id":"S001","target_id":"S009","kind":"supports"},{"source_id":"S003","target_id":"S009","kind":"supports"},{"source_id":"S007","target_id":"S009","kind":"supports"},{"source_id":"S004","target_id":"S015","kind":"supports"},{"source_id":"S006","target_id":"S015","kind":"supports"},{"source_id":"S011","target_id":"S016","kind":"supports"},{"source_id":"S012","target_id":"S016","kind":"supports"},{"source_id":"S011","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S018","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S011","target_id":"S033","kind":"supports"},{"source_id":"S012","target_id":"S033","kind":"supports"},{"source_id":"S018","target_id":"S033","kind":"supports"},{"source_id":"S019","target_id":"S033","kind":"supports"},{"source_id":"S017","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S052","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S012","target_id":"S035","kind":"supports"},{"source_id":"S052","target_id":"S035","kind":"supports"},{"source_id":"S011","target_id":"S036","kind":"supports"},{"source_id":"S012","target_id":"S037","kind":"supports"},{"source_id":"S054","target_id":"S037","kind":"supports"},{"source_id":"S019","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S012","target_id":"S039","kind":"supports"},{"source_id":"S019","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S012","target_id":"S040","kind":"supports"},{"source_id":"S019","target_id":"S040","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S006","target_id":"S041","kind":"supports"},{"source_id":"S017","target_id":"S041","kind":"supports"},{"source_id":"S018","target_id":"S041","kind":"supports"},{"source_id":"S036","target_id":"S041","kind":"supports"},{"source_id":"S038","target_id":"S041","kind":"supports"},{"source_id":"S052","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S038","target_id":"S042","kind":"supports"},{"source_id":"S017","target_id":"S043","kind":"supports"},{"source_id":"S036","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S040","target_id":"S044","kind":"supports"},{"source_id":"S019","target_id":"S044","kind":"supports"},{"source_id":"S012","target_id":"S044","kind":"supports"},{"source_id":"S052","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S012","target_id":"S045","kind":"supports"},{"source_id":"S019","target_id":"S045","kind":"supports"},{"source_id":"S038","target_id":"S045","kind":"supports"},{"source_id":"S018","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S006","target_id":"S048","kind":"supports"},{"source_id":"S017","target_id":"S048","kind":"supports"},{"source_id":"S015","target_id":"S048","kind":"supports"},{"source_id":"S052","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S053","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S039","target_id":"S051","kind":"supports"},{"source_id":"S019","target_id":"S051","kind":"supports"},{"source_id":"S012","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S051","kind":"supports"},{"source_id":"S033","target_id":"S021","kind":"supports"},{"source_id":"S035","target_id":"S021","kind":"supports"},{"source_id":"S051","target_id":"S021","kind":"supports"},{"source_id":"S044","target_id":"S022","kind":"supports"},{"source_id":"S035","target_id":"S023","kind":"supports"},{"source_id":"S045","target_id":"S023","kind":"supports"},{"source_id":"S033","target_id":"S023","kind":"supports"},{"source_id":"S051","target_id":"S023","kind":"supports"},{"source_id":"S035","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S033","target_id":"S046","kind":"supports"},{"source_id":"S051","target_id":"S046","kind":"supports"},{"source_id":"S012","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S024","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S032","kind":"supports"},{"source_id":"S035","target_id":"S032","kind":"supports"},{"source_id":"S036","target_id":"S032","kind":"supports"},{"source_id":"S037","target_id":"S032","kind":"supports"},{"source_id":"S038","target_id":"S032","kind":"supports"},{"source_id":"S039","target_id":"S032","kind":"supports"},{"source_id":"S040","target_id":"S032","kind":"supports"},{"source_id":"S041","target_id":"S032","kind":"supports"},{"source_id":"S042","target_id":"S032","kind":"supports"},{"source_id":"S043","target_id":"S032","kind":"supports"},{"source_id":"S044","target_id":"S032","kind":"supports"},{"source_id":"S045","target_id":"S032","kind":"supports"},{"source_id":"S046","target_id":"S032","kind":"supports"},{"source_id":"S047","target_id":"S032","kind":"supports"},{"source_id":"S048","target_id":"S032","kind":"supports"},{"source_id":"S049","target_id":"S032","kind":"supports"},{"source_id":"S050","target_id":"S032","kind":"supports"},{"source_id":"S051","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S020","kind":"supports"},{"source_id":"S023","target_id":"S026","kind":"supports"},{"source_id":"S047","target_id":"S026","kind":"supports"},{"source_id":"S019","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S050","target_id":"S027","kind":"supports"},{"source_id":"S019","target_id":"S027","kind":"supports"},{"source_id":"S012","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S019","target_id":"S028","kind":"supports"},{"source_id":"S012","target_id":"S028","kind":"supports"},{"source_id":"S055","target_id":"S028","kind":"supports"},{"source_id":"S019","target_id":"S029","kind":"supports"},{"source_id":"S012","target_id":"S029","kind":"supports"},{"source_id":"S039","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S054","target_id":"S029","kind":"supports"},{"source_id":"S027","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S054","target_id":"S030","kind":"supports"},{"source_id":"S027","target_id":"S031","kind":"supports"},{"source_id":"S028","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S027","target_id":"S025","kind":"supports"},{"source_id":"S028","target_id":"S025","kind":"supports"},{"source_id":"S029","target_id":"S025","kind":"supports"},{"source_id":"S030","target_id":"S025","kind":"supports"},{"source_id":"S031","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S016","kind":"supports"},{"source_id":"S006","target_id":"S017","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eqn:minimax_problem"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"极小极大问题","source_units":[{"raw_tex":"Minimax optimization problems of the form\n\\begin{align}\n\\label{eqn:minimax_problem}\n    \\underset{x \\in \\cX}{\\textrm{minimize}} \\,\\, \\underset{y \\in \\cY}{\\textrm{maximize}} \\,\\,\\,\\, \\lagrange (x,y)\n\\end{align}\n","start":14562,"end":14772}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"凸凹函数","source_units":[{"raw_tex":"A function $\\lagrange(\\cdot, \\cdot): \\cX \\times \\cY \\to \\reals$ is \\emph{convex-concave} if $\\cX$ and $\\cY$ are convex sets, $\\lagrange(x, y)$ is convex as a function of $x$ for all $y$, and $\\lagrange(x, y)$ is concave as a function of $y$ for all $x$.\n","start":28121,"end":28375}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"鞍点与极小极大解","source_units":[{"raw_tex":"A point $(x_\\star, y_\\star) \\in \\cX \\times \\cY$ is a \\emph{saddle point}, or \\emph{(minimax) solution} of~\\eqref{eqn:minimax_problem} if $\\lagrange(x_\\star, y) \\le \\lagrange(x_\\star, y_\\star) \\le \\lagrange(x, y_\\star)$ for all $(x,y) \\in \\cX \\times \\cY$.\n\n","start":28375,"end":28631}]},{"id":"S004","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"算子记号与单值性","source_units":[{"raw_tex":"An operator $\\opA$ on $\\reals^d$, denoted $\\opA\\colon \\reals^d \\rightrightarrows \\reals^d$, is a set-valued function, i.e., $\\opA(z)\\subseteq\\reals^d$ for all $z \\in \\reals^d$.\nFor simplicity, we often write $\\opA z = \\opA(z)$.\nIf $\\opA z$ contains exactly one element for all $z\\in\\reals^d$, then we say $\\opA$ is \\emph{single-valued} and view it as a function.\n","start":28631,"end":28994}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"单调算子","source_units":[{"raw_tex":"An operator $\\opA\\colon \\reals^d \\rightrightarrows \\reals^d$ is \\emph{monotone} if\n\\begin{align*}\n    \\langle \\opA z - \\opA z', z - z' \\rangle \\ge 0, \\qquad \\forall \\, z,z'\\in\\reals^d ,\n\\end{align*}\nwhere the notation means that $\\langle u - v, z - z' \\rangle \\ge 0$ for any $u \\in \\opA z$ and $v \\in \\opA z'$.\n","start":28994,"end":29305}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"强单调算子","source_units":[{"raw_tex":"For $\\mu > 0$, an operator $\\opA$ is $\\mu$-\\emph{strongly monotone} if\n\\begin{align*}\n    \\langle \\opA z - \\opA z', z - z' \\rangle \\ge \\mu \\|z-z'\\|^2, \\qquad \\forall \\, z,z'\\in\\reals^d .\n\\end{align*}\n","start":29305,"end":29505}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"鞍算子","source_units":[{"raw_tex":"For differentiable and convex-concave $\\lagrange\\colon\\reals^n \\times \\reals^m\\rightarrow\\reals$, we define\n\\[\n    \\sop\\lagrange (x,y) = (\\nabla_x \\lagrange (x,y), -\\nabla_y \\lagrange (x,y))\n\\]\nand call it the \\emph{saddle operator} of $\\lagrange$.\n","start":31511,"end":31760}]},{"id":"S008","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S008"],"declared_external":true,"declaration_text":["原文第627行引用 Rockafellar1970_monotone；此项是来源明确的背景事实，不属于任务列出的两类基础先决事实，且未用于强单调算子收敛证明。"],"source_only":false,"title":"鞍算子的单调性","source_units":[{"raw_tex":"Then $\\sop \\lagrange$ is monotone \\cite{Rockafellar1970_monotone}, and $z_\\star = (x_\\star, y_\\star)$ is a solution of~\\eqref{eqn:minimax_problem} if and only if $\\sop \\lagrange (z_\\star) = 0$.\n","start":31760,"end":31954}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"零点对应鞍点","source_units":[{"raw_tex":"Then $\\sop \\lagrange$ is monotone \\cite{Rockafellar1970_monotone}, and $z_\\star = (x_\\star, y_\\star)$ is a solution of~\\eqref{eqn:minimax_problem} if and only if $\\sop \\lagrange (z_\\star) = 0$.\n","start":31760,"end":31954}]},{"id":"S010","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"算法名称来源","source_units":[{"raw_tex":"We design the novel algorithm \\emph{\\textbf{S}trongly \\textbf{M}onotone \\textbf{E}xtra \\textbf{A}nchored \\textbf{G}radient\\textbf{\\texttt{+}}}\\footnote{The ``\\texttt{+}'' follows the nomenclature of \\cite{DiakonikolasDaskalakisJordan2021_efficient}, where the two-time-scale EG was named EG\\texttt{+}.} \n","start":40779,"end":41083}]},{"id":"S011","semantic_type":"algorithm","kind":"algorithm","labels":["eqn:SM-EAG+"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"SM-EAG+ 两步迭代","source_units":[{"raw_tex":"\\begin{align}\n\\label{eqn:SM-EAG+}\n\\begin{aligned}\n    z_{k+1/2} & = \\beta_k z_0 + (1-\\beta_k) z_k - \\eta_k \\alpha \\opB z_k \\\\\n    z_{k+1}   & = \\beta_k z_0 + (1-\\beta_k) z_k - \\alpha \\opB z_{k+1/2} ,\n\\end{aligned}\n\\tag{SM-EAG\\texttt{+}}\n\\end{align}\n","start":41083,"end":41332}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"迭代参数与步长","source_units":[{"raw_tex":"where $\\opB=\\sop\\lagrange$, $\\beta_k = \\frac{1}{\\sum_{j=0}^k (1+2\\alpha\\mu)^j}$, $\\eta_k = \\frac{1-\\beta_k}{1+2\\alpha\\mu}$, and $0 < \\alpha \\le \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2}$, to approximate OC-Halpern in the MP sense. \\ref{eqn:SM-EAG+} inherits the accelerated rate of OC-Halpern and achieves the fastest known rate in the setup with respect to the gradient norm, improving upon the prior rates of EG and OG by factors of 8 and 4, respectively.\n","start":41332,"end":41785}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","labels":["thm:sm-EAG"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"算子假设与零点","source_units":[{"raw_tex":"Let $\\opB\\colon \\reals^d \\to \\reals^d$ be $\\mu$-strongly monotone and $L$-Lipschitz with $0 < \\mu \\le L$.\nLet $z_\\star$ be the zero of $\\opB$.\n","start":44696,"end":44839}]},{"id":"S016","semantic_type":"theorem","kind":"theorem","labels":["thm:sm-EAG"],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"SM-EAG+ 显式加速速率","source_units":[{"raw_tex":"\\begin{theorem}[Fast rate of \\ref{eqn:SM-EAG+}]\n\\label{thm:sm-EAG}\nLet $\\opB\\colon \\reals^d \\to \\reals^d$ be $\\mu$-strongly monotone and $L$-Lipschitz with $0 < \\mu \\le L$.\nLet $z_\\star$ be the zero of $\\opB$.\nFor $\\alpha \\in \\big( 0, \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2} \\big]$, SM-EAG\\texttt{+} exhibits the rate\n\\begin{align*}\n    \\|\\opB z_k\\|^2 \\le \\frac{\\left(\\sqrt{1+2\\alpha\\mu} + 1\\right)^2}{\\alpha^2 \\left( \\sum_{j=0}^{k-1} (1+2\\alpha\\mu)^{\\frac{j}{2}} \\right)^2} \\|z_0 - z_\\star\\|^2 ,\n    \\qquad\\text{for } k=1,2,\\dots.\n\\end{align*}\n\\end{theorem}","start":44629,"end":45184}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","labels":["lem:sm-EAG_Lyapunov"],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"势函数引理的设置","source_units":[{"raw_tex":"Let $\\opB\\colon \\reals^d \\to \\reals^d$ be a $\\mu$-strongly monotone, $L$-Lipschitz operator ($0 < \\mu \\le L$), and let\n$\\{z_k\\}_{k=0,1,\\dots}$ be the iterates of \\ref{eqn:SM-EAG+} with $\\beta_k = \\frac{1}{\\sum_{j=0}^k (1+2\\alpha\\mu)^j}$.\n","start":52416,"end":52654}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":["lem:sm-EAG_Lyapunov"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"带常数的势函数","source_units":[{"raw_tex":"Then, the quantities\n\\begin{align*}\n    V_k = p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k - \\mu (z_k - z_0), z_k - z_0 \\rangle + \\left( \\frac{1}{2\\alpha} + \\mu \\right) \\|z_0-z_\\star\\|^2\n\\end{align*}\n","start":52654,"end":52851}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":["lem:sm-EAG_Lyapunov"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"势函数系数","source_units":[{"raw_tex":"with $p_0 = q_0 = 0$ and $q_k = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{1}{\\beta_{k-1}}$, $p_k = \\frac{\\eta_k \\alpha q_k}{2 \\beta_k}$ for $k=1,2,\\dots$ satisfy $V_k \\ge 0$, $V_0 - V_1 \\ge \\frac{\\alpha (1+2\\alpha\\mu-\\alpha^2 L^2)}{2}\\|\\opB z_0\\|^2$, and\n","start":52851,"end":53101}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","labels":["eqn:sm-EAG_summability","lem:sm-EAG_Lyapunov"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"势函数下降与可和性","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{lem:sm-EAG_Lyapunov}\nLet $\\opB\\colon \\reals^d \\to \\reals^d$ be a $\\mu$-strongly monotone, $L$-Lipschitz operator ($0 < \\mu \\le L$), and let\n$\\{z_k\\}_{k=0,1,\\dots}$ be the iterates of \\ref{eqn:SM-EAG+} with $\\beta_k = \\frac{1}{\\sum_{j=0}^k (1+2\\alpha\\mu)^j}$.\nThen, the quantities\n\\begin{align*}\n    V_k = p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k - \\mu (z_k - z_0), z_k - z_0 \\rangle + \\left( \\frac{1}{2\\alpha} + \\mu \\right) \\|z_0-z_\\star\\|^2\n\\end{align*}\nwith $p_0 = q_0 = 0$ and $q_k = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{1}{\\beta_{k-1}}$, $p_k = \\frac{\\eta_k \\alpha q_k}{2 \\beta_k}$ for $k=1,2,\\dots$ satisfy $V_k \\ge 0$, $V_0 - V_1 \\ge \\frac{\\alpha (1+2\\alpha\\mu-\\alpha^2 L^2)}{2}\\|\\opB z_0\\|^2$, and\n\\begin{align*}\n    V_k - V_{k+1} \\ge \\frac{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2) q_k}{2\\beta_k (1-\\beta_k)} \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2\n\\end{align*}\nfor $k=1,2,\\dots$.\nIn particular, if $\\alpha \\in \\big( 0, \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2} \\big]$, $V_k$ is a Lyapunov function for the algorithm, and if $\\alpha \\in \\big( 0, \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2} \\big)$, we additionally have the summability result\n\\begin{align}\n    \\sum_{k=0}^\\infty (1+2\\alpha\\mu)^k \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2 \\le \\frac{1+2\\alpha\\mu}{\\alpha^2 (1 + 2\\alpha\\mu - \\alpha^2 L^2)} \\|z_0 - z_\\star\\|^2  < \\infty .\n    \\label{eqn:sm-EAG_summability}\n\\end{align}\n\\end{lemma}","start":52374,"end":53778}]},{"id":"S021","semantic_type":"derived_fact","kind":"derived_fact","labels":["lem:sm-EAG_Lyapunov"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"势函数非负与初始下降","source_units":[{"raw_tex":"with $p_0 = q_0 = 0$ and $q_k = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{1}{\\beta_{k-1}}$, $p_k = \\frac{\\eta_k \\alpha q_k}{2 \\beta_k}$ for $k=1,2,\\dots$ satisfy $V_k \\ge 0$, $V_0 - V_1 \\ge \\frac{\\alpha (1+2\\alpha\\mu-\\alpha^2 L^2)}{2}\\|\\opB z_0\\|^2$, and\n","start":52851,"end":53101}]},{"id":"S022","semantic_type":"derived_fact","kind":"derived_fact","labels":["lem:sm-EAG_Lyapunov"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"一般迭代下降界","source_units":[{"raw_tex":"\\begin{align*}\n    V_k - V_{k+1} \\ge \\frac{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2) q_k}{2\\beta_k (1-\\beta_k)} \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2\n\\end{align*}\nfor $k=1,2,\\dots$.\n","start":53101,"end":53281}]},{"id":"S023","semantic_type":"derived_fact","kind":"derived_fact","labels":["lem:sm-EAG_Lyapunov"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"闭步长区间内不增","source_units":[{"raw_tex":"In particular, if $\\alpha \\in \\big( 0, \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2} \\big]$, $V_k$ is a Lyapunov function for the algorithm, and if $\\alpha \\in \\big( 0, \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2} \\big)$, we additionally have the summability result\n","start":53281,"end":53530}]},{"id":"S024","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:sm-EAG_summability","lem:sm-EAG_Lyapunov"],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"严格步长下的加权可和界","source_units":[{"raw_tex":"In particular, if $\\alpha \\in \\big( 0, \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2} \\big]$, $V_k$ is a Lyapunov function for the algorithm, and if $\\alpha \\in \\big( 0, \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2} \\big)$, we additionally have the summability result\n\\begin{align}\n    \\sum_{k=0}^\\infty (1+2\\alpha\\mu)^k \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2 \\le \\frac{1+2\\alpha\\mu}{\\alpha^2 (1 + 2\\alpha\\mu - \\alpha^2 L^2)} \\|z_0 - z_\\star\\|^2  < \\infty .\n    \\label{eqn:sm-EAG_summability}\n\\end{align}\n","start":53281,"end":53767}]},{"id":"S025","semantic_type":"proof","kind":"proof","labels":["eqn:sm-EAG_convergence_rate","eqn:sm-EAG_lambda"],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"主定理完整作者证明","source_units":[{"raw_tex":"\\subsection{Proof of \\texorpdfstring{\\cref{thm:sm-EAG}}{Theorem 4.2}}\nBecause the term $\\left( \\frac{1}{2\\alpha}+\\mu \\right) \\|z_0-z_\\star\\|^2$ subtracted from $V_k$ to obtain $W_k$ does not depend on $k$, clearly $W_k$ inherits nonincreasingness from $V_k$, i.e., $W_k \\le \\cdots \\le W_0 = 0$.\nTherefore, given that $q_k < \\lambda < q_k + 4\\mu p_k$, we have\n\\begin{align}\n    & \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 \\ge W_k + \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 \\ge \\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right) \\|\\opB z_k\\|^2 \\nonumber \\\\\n    & \\implies \\|\\opB z_k\\|^2 \\le \\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right)^{-1} \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 . \\label{eqn:sm-EAG_convergence_rate}\n\\end{align}\nOne can straightforwardly verify that \n\\begin{align}\n    \\lambda_\\star = (q_k (q_k + 4\\mu p_k))^{1/2}\n    \\label{eqn:sm-EAG_lambda}\n\\end{align}\nlies between $q_k$ and $q_k + 4\\mu p_k$, and minimizes the factor $\\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right)^{-1} \\frac{q_k \\lambda \\mu}{\\lambda - q_k}$ in the right hand side of \\eqref{eqn:sm-EAG_convergence_rate}.\nPlugging in the explicit expressions $q_k = \\frac{(1+2\\alpha\\mu)^k - 1}{2\\alpha\\mu (1+2\\alpha\\mu)^{k-1}}$ and $p_k = \\frac{\\alpha}{2}(1+2\\alpha\\mu)^{k-1} q_k^2 = \\frac{((1+2\\alpha\\mu)^k - 1)^2}{8\\alpha\\mu^2 (1+2\\alpha\\mu)^{k-1}}$ (which uses \\eqref{eqn:sm-EAG_pkqk}) into~\\eqref{eqn:sm-EAG_lambda}, we obtain $\\lambda_\\star = (1+2\\alpha\\mu)^{\\frac{k}{2}} q_k$ and\n\\begin{align*}\n    \\left( p_k - \\frac{\\lambda_\\star - q_k}{4\\mu} \\right)^{-1} \\frac{q_k \\lambda_\\star \\mu}{\\lambda_\\star - q_k} & = \\frac{4\\mu^2}{((1+2\\alpha\\mu)^{\\frac{k}{2}} - 1)^2} \n    = \\frac{(\\sqrt{1+2\\alpha\\mu}+1)^2}{\\alpha^2 \\left( \\sum_{j=0}^{k-1} (1+2\\alpha\\mu)^{\\frac{j}{2}} \\right)^2} .\n\\end{align*}\nTherefore, \\eqref{eqn:sm-EAG_convergence_rate} with $\\lambda = \\lambda_\\star$ concludes the proof.\n\n\n\n","start":53865,"end":55789}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"平移后势函数不增","source_units":[{"raw_tex":"Because the term $\\left( \\frac{1}{2\\alpha}+\\mu \\right) \\|z_0-z_\\star\\|^2$ subtracted from $V_k$ to obtain $W_k$ does not depend on $k$, clearly $W_k$ inherits nonincreasingness from $V_k$, i.e., $W_k \\le \\cdots \\le W_0 = 0$.\n","start":53935,"end":54160}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:sm-EAG_convergence_rate"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"参数化残差上界","source_units":[{"raw_tex":"Therefore, given that $q_k < \\lambda < q_k + 4\\mu p_k$, we have\n\\begin{align}\n    & \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 \\ge W_k + \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 \\ge \\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right) \\|\\opB z_k\\|^2 \\nonumber \\\\\n    & \\implies \\|\\opB z_k\\|^2 \\le \\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right)^{-1} \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 . \\label{eqn:sm-EAG_convergence_rate}\n\\end{align}\n","start":54160,"end":54645}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:sm-EAG_lambda"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"选择最优标量参数","source_units":[{"raw_tex":"One can straightforwardly verify that \n\\begin{align}\n    \\lambda_\\star = (q_k (q_k + 4\\mu p_k))^{1/2}\n    \\label{eqn:sm-EAG_lambda}\n\\end{align}\nlies between $q_k$ and $q_k + 4\\mu p_k$, and minimizes the factor $\\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right)^{-1} \\frac{q_k \\lambda \\mu}{\\lambda - q_k}$ in the right hand side of \\eqref{eqn:sm-EAG_convergence_rate}.\n","start":54645,"end":55011}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"代入系数闭式","source_units":[{"raw_tex":"Plugging in the explicit expressions $q_k = \\frac{(1+2\\alpha\\mu)^k - 1}{2\\alpha\\mu (1+2\\alpha\\mu)^{k-1}}$ and $p_k = \\frac{\\alpha}{2}(1+2\\alpha\\mu)^{k-1} q_k^2 = \\frac{((1+2\\alpha\\mu)^k - 1)^2}{8\\alpha\\mu^2 (1+2\\alpha\\mu)^{k-1}}$ (which uses \\eqref{eqn:sm-EAG_pkqk}) into~\\eqref{eqn:sm-EAG_lambda}, we obtain $\\lambda_\\star = (1+2\\alpha\\mu)^{\\frac{k}{2}} q_k$ and\n","start":55011,"end":55375}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"化简最优速率系数","source_units":[{"raw_tex":"\\begin{align*}\n    \\left( p_k - \\frac{\\lambda_\\star - q_k}{4\\mu} \\right)^{-1} \\frac{q_k \\lambda_\\star \\mu}{\\lambda_\\star - q_k} & = \\frac{4\\mu^2}{((1+2\\alpha\\mu)^{\\frac{k}{2}} - 1)^2} \n    = \\frac{(\\sqrt{1+2\\alpha\\mu}+1)^2}{\\alpha^2 \\left( \\sum_{j=0}^{k-1} (1+2\\alpha\\mu)^{\\frac{j}{2}} \\right)^2} .\n\\end{align*}\n","start":55375,"end":55687}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"代回并结束主证明","source_units":[{"raw_tex":"Therefore, \\eqref{eqn:sm-EAG_convergence_rate} with $\\lambda = \\lambda_\\star$ concludes the proof.\n\n\n\n","start":55687,"end":55789}]},{"id":"S032","semantic_type":"proof","kind":"proof","labels":["eqn:SM-EAG+_zk_minus_zkp1","eqn:sm-EAG_Lyapunov_Lipschitz","eqn:sm-EAG_Lyapunov_monotone","eqn:sm-EAG_pkqk","eqn:sm-EAG_recurrence_p_k","eqn:sm-EAG_recurrence_q_k","eqn:sm-EAG_summable_term","eqn:sm_EAG+_zkhalf_minus_zkp1","eqn:sm_EAG+_zkp1_minus_z0"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"势函数引理完整作者证明","source_units":[{"raw_tex":"\\subsection{Proof of \\texorpdfstring{\\cref{lem:sm-EAG_Lyapunov}}{Lemma 4.4}}\nWhen $k=0$, we have $z_{1/2} = z_0$, $z_1 = z_0 - \\alpha \\opB z_0$, $V_0 = \\left( \\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2$, and\n\\begin{align*}\n    V_1 & = \\frac{\\alpha}{2}\\|\\opB z_1\\|^2 + \\langle \\opB z_1 - \\mu(z_1 - z_0), z_1 - z_0 \\rangle + \\left(\\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2 \\nonumber \\\\\n    & = \\frac{\\alpha}{2}\\|\\opB z_1\\|^2 - \\alpha \\langle \\opB z_1, \\opB z_0 \\rangle - \\alpha^2 \\mu \\|\\opB z_0\\|^2 + \\left(\\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2 . \n\\end{align*}\nBecause $\\opB$ is $L$-Lipschitz,\n\\begin{align*}\n    0 & \\le L^2 \\|z_1 - z_0\\|^2 - \\|\\opB z_1 - \\opB z_0\\|^2 = (\\alpha^2 L^2 - 1) \\|\\opB z_0\\|^2 + 2\\langle \\opB z_0, \\opB z_1 \\rangle - \\|\\opB z_1\\|^2 .\n\\end{align*}\nTherefore,\n\\begin{align*}\n    V_0 - V_1 & = \\alpha^2 \\mu \\|\\opB z_0\\|^2 + \\alpha \\langle \\opB z_0, \\opB z_1 \\rangle - \\frac{\\alpha}{2} \\|\\opB z_1\\|^2 \\\\\n    & = \\frac{\\alpha}{2} (1+2\\alpha\\mu -\\alpha^2 L^2) \\|\\opB z_0\\|^2 + \\frac{\\alpha}{2} \\left( (\\alpha^2 L^2 - 1) \\|\\opB z_0\\|^2 + 2\\langle \\opB z_0, \\opB z_1 \\rangle - \\|\\opB z_1\\|^2 \\right) \\\\\n    & \\ge \\frac{\\alpha}{2} (1+2\\alpha\\mu -\\alpha^2 L^2) \\|\\opB z_0\\|^2 \\\\\n    & = \\frac{\\alpha}{2} (1+2\\alpha\\mu -\\alpha^2 L^2) \\|\\eta_0 \\opB z_0 - \\opB z_{1/2}\\|^2 \n\\end{align*}\nwhere the last equality holds because $\\eta_0 = 0$ and $z_{1/2} = z_0$.\n\nNow suppose $k\\ge 1$. For \\ref{eqn:SM-EAG+}, we have the identities\n\\begin{align}\n    z_{k+1} - z_0 & = (1-\\beta_k) (z_k - z_0) - \\alpha \\opB z_{k+1/2} \\label{eqn:sm_EAG+_zkp1_minus_z0} \\\\\n    z_k - z_{k+1} & = \\beta_k (z_k - z_0) + \\alpha \\opB z_{k+1/2} \\label{eqn:SM-EAG+_zk_minus_zkp1} \\\\\n    z_{k+1/2} - z_{k+1} & = \\alpha \\opB z_{k+1/2} - \\eta_k \\alpha \\opB z_k . \\label{eqn:sm_EAG+_zkhalf_minus_zkp1}\n\\end{align}\nNote that $\\beta_k = \\frac{1}{\\sum_{j=0}^{k} (1+2\\alpha\\mu)^j}$ satisfies\n\\begin{align*}\n    \\frac{1}{\\beta_{k}} = \\sum_{j=0}^{k} (1+2\\alpha\\mu)^j = 1 + (1+2\\alpha\\mu) \\sum_{j=0}^{k-1} (1+2\\alpha\\mu)^j = 1 + \\frac{1+2\\alpha\\mu}{\\beta_{k-1}} \n\\end{align*}\nfor $k=1,2,\\dots$, so $q_k = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{1}{\\beta_{k-1}}$ satisfies the recurrence relation\n\\begin{align}\n    q_{k+1} = \\frac{q_k}{\\beta_k} \\frac{\\beta_{k-1}}{(1+2\\alpha\\mu)} =\n    \\frac{q_k}{\\beta_k} \\frac{1}{\\frac{1}{\\beta_k} - 1} = \\frac{q_k}{1-\\beta_k}\n    \\label{eqn:sm-EAG_recurrence_q_k}\n\\end{align}\nfor $k=1,2,\\dots$.\nAdditionally, because $\\eta_k = \\frac{1-\\beta_k}{1+2\\alpha\\mu}$ and $p_k = \\frac{\\eta_k \\alpha q_k}{2\\beta_k}$,\n\\begin{align}\n    p_k = \\frac{\\alpha (1-\\beta_k) q_k}{2\\beta_k (1+2\\alpha\\mu)} = \\frac{\\alpha}{2} \\left( \\frac{1}{\\beta_k} - 1 \\right) \\frac{q_k}{1 + 2\\alpha\\mu} = \\frac{\\alpha}{2} \\frac{1}{\\beta_{k-1}} q_k = \\frac{\\alpha}{2} (1+2\\alpha\\mu)^{k-1} q_k^2 \n    \\label{eqn:sm-EAG_pkqk}\n\\end{align}\nand\n\\begin{align}\n    p_{k+1} & = \\frac{\\eta_{k+1}\\alpha q_{k+1}}{2\\beta_{k+1}} = \\frac{\\alpha q_{k+1}}{2\\beta_{k+1}} \\frac{1-\\beta_{k+1}}{1+2\\alpha\\mu} = \\frac{\\alpha q_k}{2(1-\\beta_k)} \\frac{1-\\beta_{k+1}}{(1+2\\alpha\\mu)\\beta_{k+1}} \\nonumber \\\\\n    & = \\frac{\\alpha q_k}{2(1-\\beta_k)} \\frac{1}{1+2\\alpha\\mu} \\left( \\frac{1}{\\beta_{k+1}} - 1 \\right) = \\frac{\\alpha q_k}{2\\beta_k (1-\\beta_k)} . \n    \\label{eqn:sm-EAG_recurrence_p_k}\n\\end{align}\nThe identities~\\eqref{eqn:sm-EAG_recurrence_q_k} through \\eqref{eqn:sm-EAG_recurrence_p_k} will be useful for the subsequent arguments.\n\nUsing $\\mu$-strong monotonicity of $\\opB$ and the identities~\\eqref{eqn:sm_EAG+_zkp1_minus_z0}, \\eqref{eqn:SM-EAG+_zk_minus_zkp1} and \\eqref{eqn:sm-EAG_recurrence_q_k}, we have\n\\begin{align}\n    V_k - V_{k+1} & \\ge V_k - V_{k+1} - \\frac{q_k}{\\beta_k} \\langle z_k - z_{k+1}, \\opB z_k - \\opB z_{k+1} - \\mu(z_k - z_{k+1}) \\rangle \\nonumber \\\\\n    & = p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k , z_k-z_0 \\rangle - q_k \\mu \\|z_k - z_0\\|^2 \\nonumber \\\\\n    & \\quad - p_{k+1} \\|\\opB z_{k+1}\\|^2 - q_{k+1} \\langle \\opB z_{k+1} , (1-\\beta_k)(z_k - z_0) - \\alpha \\opB z_{k+1/2} \\rangle \\nonumber \\\\\n    & \\quad + q_{k+1} \\mu \\left\\| (1-\\beta_k) (z_k - z_0) - \\alpha \\opB z_{k+1/2} \\right\\|^2 \\nonumber \\\\\n    & \\quad - \\frac{q_k}{\\beta_k} \\langle \\beta_k (z_k - z_0) + \\alpha \\opB z_{k+1/2} , \\opB z_k - \\opB z_{k+1} \\rangle \\nonumber \\\\\n    & \\quad + \\frac{q_k \\mu}{\\beta_k} \\left\\| \\beta_k (z_k - z_0) + \\alpha \\opB z_{k+1/2} \\right\\|^2 \\nonumber \\\\\n    & \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 - \\frac{\\alpha q_k}{\\beta_k} \\langle \\opB z_k, \\opB z_{k+1/2} \\rangle + \\mu \\alpha^2 \\left( \\frac{q_k}{\\beta_k} + q_{k+1} \\right) \\|\\opB z_{k+1/2}\\|^2 \\\\\n        & \\quad + \\alpha \\left( \\frac{q_k}{\\beta_k} + q_{k+1} \\right) \\langle \\opB z_{k+1/2} , \\opB z_{k+1} \\rangle - p_{k+1} \\|\\opB z_{k+1}\\|^2 \\\\\n        & \\quad + 2\\mu\\alpha \\left( q_k - (1-\\beta_k)q_{k+1} \\right) \\langle \\opB z_{k+1/2}, z_k - z_0 \\rangle \\\\\n        & \\quad + (q_k - (1-\\beta_k) q_{k+1}) \\langle \\opB z_{k+1}, z_k - z_0 \\rangle \\\\\n        & \\quad + \\mu (1-\\beta_k) ((1-\\beta_k) q_{k+1} - q_k) \\|z_k - z_0\\|^2 \n    \\end{aligned} \\nonumber \\\\\n    & \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 - \\frac{\\alpha q_k}{\\beta_k} \\langle \\opB z_k, \\opB z_{k+1/2} \\rangle + \\frac{\\mu \\alpha^2 q_k}{\\beta_k (1-\\beta_k)} \\|\\opB z_{k+1/2}\\|^2 \\\\\n        & \\quad + \\frac{\\alpha q_k}{\\beta_k (1-\\beta_k)} \\langle \\opB z_{k+1/2} , \\opB z_{k+1} \\rangle - p_{k+1} \\|\\opB z_{k+1}\\|^2 .\n    \\end{aligned} \\label{eqn:sm-EAG_Lyapunov_monotone}\n\\end{align}\nNow because $\\opB$ is $L$-Lipschitz,\n\\begin{align}\n\\label{eqn:sm-EAG_Lyapunov_Lipschitz}\n    \\|\\opB z_{k+1/2} - \\opB z_{k+1}\\|^2 \\le L^2 \\|z_{k+1/2} - z_{k+1}\\|^2 = L^2 \\|\\alpha \\opB z_{k+1/2} - \\eta_k \\alpha \\opB z_k\\|^2 \n\\end{align}\nwhere we applied the identity~\\eqref{eqn:sm_EAG+_zkhalf_minus_zkp1} for the last equality.\nMultiplying $p_{k+1}$ to \\eqref{eqn:sm-EAG_Lyapunov_Lipschitz} and then subtracting it from \\eqref{eqn:sm-EAG_Lyapunov_monotone}, we obtain\n\\begin{align}\n    V_k - V_{k+1} & \\ge ( p_k - p_{k+1} \\eta_k^2 \\alpha^2 L^2) \\|\\opB z_k\\|^2 - \\left( \\frac{\\alpha q_k}{\\beta_k} - 2p_{k+1} \\eta_k \\alpha^2 L^2 \\right) \\langle \\opB z_k, \\opB z_{k+1/2} \\rangle \\nonumber \\\\\n    & \\quad + \\left( \\frac{\\mu\\alpha^2 q_k}{\\beta_k (1-\\beta_k)} + p_{k+1}(1-\\alpha^2 L^2) \\right) \\|\\opB z_{k+1/2}\\|^2 \\nonumber \\\\\n    & \\quad + \\left( \\frac{\\alpha q_k}{\\beta_k (1-\\beta_k)} - 2p_{k+1} \\right) \\langle \\opB z_{k+1/2} , \\opB z_{k+1} \\rangle \\nonumber \\\\\n    & = \\frac{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2) q_k}{2\\beta_k (1-\\beta_k)} \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2\n    \\label{eqn:sm-EAG_summable_term}\n\\end{align}\nwhere the last equality follows from the identities $\\eta_k = \\frac{1-\\beta_k}{1+2\\alpha\\mu}$, $p_k = \\frac{\\eta_k \\alpha q_k}{2\\beta_k}$ and \\eqref{eqn:sm-EAG_recurrence_p_k}.\nAs $1 + 2\\alpha\\mu - \\alpha^2 L^2 \\ge 0$ if $0 < \\alpha \\le \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2}$, for this range of $\\alpha$,\n\\begin{align*}\n    V_k - V_{k+1} & \\ge \\frac{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2)}{2} (1+2\\alpha\\mu)^k \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2 \\ge 0,\n\\end{align*}\nwhere we used $\\frac{q_k}{\\beta_k (1-\\beta_k)} = \\frac{q_{k+1}}{\\beta_k} > (1+2\\alpha\\mu)^k$.\nNow it only remains to show that $V_N \\ge 0$ holds for any $N \\in \\mathbb{N}$; if that is the case, then\n\\begin{align*}\n    & \\sum_{k=0}^N (1+2\\alpha\\mu)^k \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2 \\le \\frac{2}{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2)} (V_0 - V_{N+1}) \\\\\n    & \\quad \\le \\frac{2}{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2)} \\left( \\frac{1}{2\\alpha} + \\mu \\right) \\|z_0 - z_\\star\\|^2 = \\frac{1+2\\alpha\\mu}{\\alpha^2 (1 + 2\\alpha\\mu - \\alpha^2 L^2)} \\|z_0 - z_\\star\\|^2 ,\n\\end{align*}\nand taking $N\\to\\infty$, we obtain~\\eqref{eqn:sm-EAG_summability}.\n\nConsider the following shifted version of $V_k$:\n\\begin{align*}\n    W_k = V_k - \\left(\\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2 = p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k - \\mu (z_k - z_0), z_k - z_0 \\rangle .\n\\end{align*}\nBecause $\\opB$ is $\\mu$-strongly monotone, if $\\lambda \\ge 0$ and $\\lambda \\ne q_k$,\n\\begin{align}\n    W_k & \\ge p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k - \\mu (z_k - z_0), z_k - z_0 \\rangle - \\lambda \\langle \\opB z_k - \\mu (z_k - z_\\star), z_k - z_\\star \\rangle \\nonumber \\\\\n    &\n    \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 + \\langle \\opB z_k , (q_k - \\lambda) z_k - q_k z_0 + \\lambda z_\\star \\rangle \\\\\n        & \\quad + \\mu(\\lambda-q_k) \\|z_k\\|^2 + 2 \\mu \\langle z_k, q_k z_0 - \\lambda z_\\star \\rangle - q_k \\mu \\|z_0\\|^2 + \\lambda\\mu \\|z_\\star\\|^2  \n    \\end{aligned} \\nonumber \\\\\n    &\n    \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 + \\langle \\opB z_k , (q_k - \\lambda) z_k - q_k z_0 + \\lambda z_\\star \\rangle \\\\\n        & \\quad + \\frac{\\mu}{\\lambda - q_k} \\left\\| (\\lambda - q_k) z_k + q_k z_0 - \\lambda z_\\star \\right\\|^2 - \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 .\n    \\end{aligned} \\nonumber \\\\ \n    &\n    \\begin{aligned}\n        & = \\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right) \\|\\opB z_k\\|^2 + \\frac{\\mu}{\\lambda - q_k} \\left\\| \\frac{\\lambda - q_k}{2\\mu} \\opB z_k - (\\lambda - q_k) z_k - q_k z_0 + \\lambda z_\\star \\right\\|^2 \\\\\n        & \\quad - \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 .\n    \\end{aligned} \\nonumber\n\\end{align}\nAdding $\\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2$ to the both sides and taking $\\lambda = q_k + 4\\mu p_k$, we obtain \n\\begin{align*}\n    0 & \\le W_k + \\left( \\frac{q_k^2}{4p_k} + \\mu q_k \\right) \\|z_0 - z_\\star\\|^2 = V_k ,\n\\end{align*}\nwhere the last equality follows by applying \\eqref{eqn:sm-EAG_pkqk} and then plugging in the expression $q_k = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{1}{\\beta_{k-1}} = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{(1+2\\alpha\\mu)^k - 1}{2\\alpha\\mu}$.\n\n\n","start":90433,"end":100120}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"初始迭代与势函数值","source_units":[{"raw_tex":"When $k=0$, we have $z_{1/2} = z_0$, $z_1 = z_0 - \\alpha \\opB z_0$, $V_0 = \\left( \\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2$, and\n\\begin{align*}\n    V_1 & = \\frac{\\alpha}{2}\\|\\opB z_1\\|^2 + \\langle \\opB z_1 - \\mu(z_1 - z_0), z_1 - z_0 \\rangle + \\left(\\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2 \\nonumber \\\\\n    & = \\frac{\\alpha}{2}\\|\\opB z_1\\|^2 - \\alpha \\langle \\opB z_1, \\opB z_0 \\rangle - \\alpha^2 \\mu \\|\\opB z_0\\|^2 + \\left(\\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2 . \n\\end{align*}\n","start":90510,"end":91022}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"初始步的 Lipschitz 约束","source_units":[{"raw_tex":"Because $\\opB$ is $L$-Lipschitz,\n\\begin{align*}\n    0 & \\le L^2 \\|z_1 - z_0\\|^2 - \\|\\opB z_1 - \\opB z_0\\|^2 = (\\alpha^2 L^2 - 1) \\|\\opB z_0\\|^2 + 2\\langle \\opB z_0, \\opB z_1 \\rangle - \\|\\opB z_1\\|^2 .\n\\end{align*}\n","start":91022,"end":91236}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"得到初始下降界","source_units":[{"raw_tex":"Therefore,\n\\begin{align*}\n    V_0 - V_1 & = \\alpha^2 \\mu \\|\\opB z_0\\|^2 + \\alpha \\langle \\opB z_0, \\opB z_1 \\rangle - \\frac{\\alpha}{2} \\|\\opB z_1\\|^2 \\\\\n    & = \\frac{\\alpha}{2} (1+2\\alpha\\mu -\\alpha^2 L^2) \\|\\opB z_0\\|^2 + \\frac{\\alpha}{2} \\left( (\\alpha^2 L^2 - 1) \\|\\opB z_0\\|^2 + 2\\langle \\opB z_0, \\opB z_1 \\rangle - \\|\\opB z_1\\|^2 \\right) \\\\\n    & \\ge \\frac{\\alpha}{2} (1+2\\alpha\\mu -\\alpha^2 L^2) \\|\\opB z_0\\|^2 \\\\\n    & = \\frac{\\alpha}{2} (1+2\\alpha\\mu -\\alpha^2 L^2) \\|\\eta_0 \\opB z_0 - \\opB z_{1/2}\\|^2 \n\\end{align*}\nwhere the last equality holds because $\\eta_0 = 0$ and $z_{1/2} = z_0$.\n","start":91236,"end":91835}]},{"id":"S036","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:SM-EAG+_zk_minus_zkp1","eqn:sm_EAG+_zkhalf_minus_zkp1","eqn:sm_EAG+_zkp1_minus_z0"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"三条迭代差恒等式","source_units":[{"raw_tex":"Now suppose $k\\ge 1$. For \\ref{eqn:SM-EAG+}, we have the identities\n\\begin{align}\n    z_{k+1} - z_0 & = (1-\\beta_k) (z_k - z_0) - \\alpha \\opB z_{k+1/2} \\label{eqn:sm_EAG+_zkp1_minus_z0} \\\\\n    z_k - z_{k+1} & = \\beta_k (z_k - z_0) + \\alpha \\opB z_{k+1/2} \\label{eqn:SM-EAG+_zk_minus_zkp1} \\\\\n    z_{k+1/2} - z_{k+1} & = \\alpha \\opB z_{k+1/2} - \\eta_k \\alpha \\opB z_k . \\label{eqn:sm_EAG+_zkhalf_minus_zkp1}\n\\end{align}\n","start":91836,"end":92255}]},{"id":"S037","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"锚定系数的递推","source_units":[{"raw_tex":"Note that $\\beta_k = \\frac{1}{\\sum_{j=0}^{k} (1+2\\alpha\\mu)^j}$ satisfies\n\\begin{align*}\n    \\frac{1}{\\beta_{k}} = \\sum_{j=0}^{k} (1+2\\alpha\\mu)^j = 1 + (1+2\\alpha\\mu) \\sum_{j=0}^{k-1} (1+2\\alpha\\mu)^j = 1 + \\frac{1+2\\alpha\\mu}{\\beta_{k-1}} \n\\end{align*}\n","start":92255,"end":92510}]},{"id":"S038","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:sm-EAG_recurrence_q_k"],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"q 系数递推","source_units":[{"raw_tex":"for $k=1,2,\\dots$, so $q_k = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{1}{\\beta_{k-1}}$ satisfies the recurrence relation\n\\begin{align}\n    q_{k+1} = \\frac{q_k}{\\beta_k} \\frac{\\beta_{k-1}}{(1+2\\alpha\\mu)} =\n    \\frac{q_k}{\\beta_k} \\frac{1}{\\frac{1}{\\beta_k} - 1} = \\frac{q_k}{1-\\beta_k}\n    \\label{eqn:sm-EAG_recurrence_q_k}\n\\end{align}\nfor $k=1,2,\\dots$.\n","start":92510,"end":92861}]},{"id":"S039","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:sm-EAG_pkqk"],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"p 与 q 的关键恒等式","source_units":[{"raw_tex":"Additionally, because $\\eta_k = \\frac{1-\\beta_k}{1+2\\alpha\\mu}$ and $p_k = \\frac{\\eta_k \\alpha q_k}{2\\beta_k}$,\n\\begin{align}\n    p_k = \\frac{\\alpha (1-\\beta_k) q_k}{2\\beta_k (1+2\\alpha\\mu)} = \\frac{\\alpha}{2} \\left( \\frac{1}{\\beta_k} - 1 \\right) \\frac{q_k}{1 + 2\\alpha\\mu} = \\frac{\\alpha}{2} \\frac{1}{\\beta_{k-1}} q_k = \\frac{\\alpha}{2} (1+2\\alpha\\mu)^{k-1} q_k^2 \n    \\label{eqn:sm-EAG_pkqk}\n\\end{align}\n","start":92861,"end":93267}]},{"id":"S040","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:sm-EAG_recurrence_p_k"],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"下一步 p 系数","source_units":[{"raw_tex":"and\n\\begin{align}\n    p_{k+1} & = \\frac{\\eta_{k+1}\\alpha q_{k+1}}{2\\beta_{k+1}} = \\frac{\\alpha q_{k+1}}{2\\beta_{k+1}} \\frac{1-\\beta_{k+1}}{1+2\\alpha\\mu} = \\frac{\\alpha q_k}{2(1-\\beta_k)} \\frac{1-\\beta_{k+1}}{(1+2\\alpha\\mu)\\beta_{k+1}} \\nonumber \\\\\n    & = \\frac{\\alpha q_k}{2(1-\\beta_k)} \\frac{1}{1+2\\alpha\\mu} \\left( \\frac{1}{\\beta_{k+1}} - 1 \\right) = \\frac{\\alpha q_k}{2\\beta_k (1-\\beta_k)} . \n    \\label{eqn:sm-EAG_recurrence_p_k}\n\\end{align}\nThe identities~\\eqref{eqn:sm-EAG_recurrence_q_k} through \\eqref{eqn:sm-EAG_recurrence_p_k} will be useful for the subsequent arguments.\n","start":93267,"end":93850}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"加入强单调非负项","source_units":[{"raw_tex":"Using $\\mu$-strong monotonicity of $\\opB$ and the identities~\\eqref{eqn:sm_EAG+_zkp1_minus_z0}, \\eqref{eqn:SM-EAG+_zk_minus_zkp1} and \\eqref{eqn:sm-EAG_recurrence_q_k}, we have\n\\begin{align}\n    V_k - V_{k+1} & \\ge V_k - V_{k+1} - \\frac{q_k}{\\beta_k} \\langle z_k - z_{k+1}, \\opB z_k - \\opB z_{k+1} - \\mu(z_k - z_{k+1}) \\rangle \\nonumber \\\\\n    & = p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k , z_k-z_0 \\rangle - q_k \\mu \\|z_k - z_0\\|^2 \\nonumber \\\\\n    & \\quad - p_{k+1} \\|\\opB z_{k+1}\\|^2 - q_{k+1} \\langle \\opB z_{k+1} , (1-\\beta_k)(z_k - z_0) - \\alpha \\opB z_{k+1/2} \\rangle \\nonumber \\\\\n    & \\quad + q_{k+1} \\mu \\left\\| (1-\\beta_k) (z_k - z_0) - \\alpha \\opB z_{k+1/2} \\right\\|^2 \\nonumber \\\\\n    & \\quad - \\frac{q_k}{\\beta_k} \\langle \\beta_k (z_k - z_0) + \\alpha \\opB z_{k+1/2} , \\opB z_k - \\opB z_{k+1} \\rangle \\nonumber \\\\\n    & \\quad + \\frac{q_k \\mu}{\\beta_k} \\left\\| \\beta_k (z_k - z_0) + \\alpha \\opB z_{k+1/2} \\right\\|^2 \\nonumber \\\\\n    & \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 - \\frac{\\alpha q_k}{\\beta_k} \\langle \\opB z_k, \\opB z_{k+1/2} \\rangle + \\mu \\alpha^2 \\left( \\frac{q_k}{\\beta_k} + q_{k+1} \\right) \\|\\opB z_{k+1/2}\\|^2 \\\\\n        & \\quad + \\alpha \\left( \\frac{q_k}{\\beta_k} + q_{k+1} \\right) \\langle \\opB z_{k+1/2} , \\opB z_{k+1} \\rangle - p_{k+1} \\|\\opB z_{k+1}\\|^2 \\\\\n        & \\quad + 2\\mu\\alpha \\left( q_k - (1-\\beta_k)q_{k+1} \\right) \\langle \\opB z_{k+1/2}, z_k - z_0 \\rangle \\\\\n        & \\quad + (q_k - (1-\\beta_k) q_{k+1}) \\langle \\opB z_{k+1}, z_k - z_0 \\rangle \\\\\n        & \\quad + \\mu (1-\\beta_k) ((1-\\beta_k) q_{k+1} - q_k) \\|z_k - z_0\\|^2 \n    \\end{aligned} \\nonumber \\\\\n","start":93851,"end":95461}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"消去锚点交叉项","source_units":[{"raw_tex":"    & \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 - \\frac{\\alpha q_k}{\\beta_k} \\langle \\opB z_k, \\opB z_{k+1/2} \\rangle + \\frac{\\mu \\alpha^2 q_k}{\\beta_k (1-\\beta_k)} \\|\\opB z_{k+1/2}\\|^2 \\\\\n        & \\quad + \\frac{\\alpha q_k}{\\beta_k (1-\\beta_k)} \\langle \\opB z_{k+1/2} , \\opB z_{k+1} \\rangle - p_{k+1} \\|\\opB z_{k+1}\\|^2 .\n    \\end{aligned} \\label{eqn:sm-EAG_Lyapunov_monotone}\n\\end{align}\n","start":95461,"end":95856}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:sm-EAG_Lyapunov_Lipschitz"],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"中间点的 Lipschitz 估计","source_units":[{"raw_tex":"Now because $\\opB$ is $L$-Lipschitz,\n\\begin{align}\n\\label{eqn:sm-EAG_Lyapunov_Lipschitz}\n    \\|\\opB z_{k+1/2} - \\opB z_{k+1}\\|^2 \\le L^2 \\|z_{k+1/2} - z_{k+1}\\|^2 = L^2 \\|\\alpha \\opB z_{k+1/2} - \\eta_k \\alpha \\opB z_k\\|^2 \n\\end{align}\nwhere we applied the identity~\\eqref{eqn:sm_EAG+_zkhalf_minus_zkp1} for the last equality.\n","start":95856,"end":96182}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:sm-EAG_summable_term"],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"组合为误差平方下降","source_units":[{"raw_tex":"Multiplying $p_{k+1}$ to \\eqref{eqn:sm-EAG_Lyapunov_Lipschitz} and then subtracting it from \\eqref{eqn:sm-EAG_Lyapunov_monotone}, we obtain\n\\begin{align}\n    V_k - V_{k+1} & \\ge ( p_k - p_{k+1} \\eta_k^2 \\alpha^2 L^2) \\|\\opB z_k\\|^2 - \\left( \\frac{\\alpha q_k}{\\beta_k} - 2p_{k+1} \\eta_k \\alpha^2 L^2 \\right) \\langle \\opB z_k, \\opB z_{k+1/2} \\rangle \\nonumber \\\\\n    & \\quad + \\left( \\frac{\\mu\\alpha^2 q_k}{\\beta_k (1-\\beta_k)} + p_{k+1}(1-\\alpha^2 L^2) \\right) \\|\\opB z_{k+1/2}\\|^2 \\nonumber \\\\\n    & \\quad + \\left( \\frac{\\alpha q_k}{\\beta_k (1-\\beta_k)} - 2p_{k+1} \\right) \\langle \\opB z_{k+1/2} , \\opB z_{k+1} \\rangle \\nonumber \\\\\n    & = \\frac{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2) q_k}{2\\beta_k (1-\\beta_k)} \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2\n    \\label{eqn:sm-EAG_summable_term}\n\\end{align}\nwhere the last equality follows from the identities $\\eta_k = \\frac{1-\\beta_k}{1+2\\alpha\\mu}$, $p_k = \\frac{\\eta_k \\alpha q_k}{2\\beta_k}$ and \\eqref{eqn:sm-EAG_recurrence_p_k}.\n","start":96182,"end":97159}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"步长保证非负下降","source_units":[{"raw_tex":"As $1 + 2\\alpha\\mu - \\alpha^2 L^2 \\ge 0$ if $0 < \\alpha \\le \\frac{\\sqrt{L^2 + \\mu^2} + \\mu}{L^2}$, for this range of $\\alpha$,\n\\begin{align*}\n    V_k - V_{k+1} & \\ge \\frac{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2)}{2} (1+2\\alpha\\mu)^k \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2 \\ge 0,\n\\end{align*}\nwhere we used $\\frac{q_k}{\\beta_k (1-\\beta_k)} = \\frac{q_{k+1}}{\\beta_k} > (1+2\\alpha\\mu)^k$.\n","start":97159,"end":97544}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"条件性望远镜求和","source_units":[{"raw_tex":"Now it only remains to show that $V_N \\ge 0$ holds for any $N \\in \\mathbb{N}$; if that is the case, then\n\\begin{align*}\n    & \\sum_{k=0}^N (1+2\\alpha\\mu)^k \\|\\eta_k \\opB z_k - \\opB z_{k+1/2}\\|^2 \\le \\frac{2}{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2)} (V_0 - V_{N+1}) \\\\\n    & \\quad \\le \\frac{2}{\\alpha (1 + 2\\alpha\\mu - \\alpha^2 L^2)} \\left( \\frac{1}{2\\alpha} + \\mu \\right) \\|z_0 - z_\\star\\|^2 = \\frac{1+2\\alpha\\mu}{\\alpha^2 (1 + 2\\alpha\\mu - \\alpha^2 L^2)} \\|z_0 - z_\\star\\|^2 ,\n\\end{align*}\nand taking $N\\to\\infty$, we obtain~\\eqref{eqn:sm-EAG_summability}.\n","start":97544,"end":98101}]},{"id":"S047","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"平移势函数","source_units":[{"raw_tex":"Consider the following shifted version of $V_k$:\n\\begin{align*}\n    W_k = V_k - \\left(\\frac{1}{2\\alpha} + \\mu\\right) \\|z_0 - z_\\star\\|^2 = p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k - \\mu (z_k - z_0), z_k - z_0 \\rangle .\n\\end{align*}\n","start":98102,"end":98335}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"相对零点应用强单调性","source_units":[{"raw_tex":"Because $\\opB$ is $\\mu$-strongly monotone, if $\\lambda \\ge 0$ and $\\lambda \\ne q_k$,\n\\begin{align}\n    W_k & \\ge p_k \\|\\opB z_k\\|^2 + q_k \\langle \\opB z_k - \\mu (z_k - z_0), z_k - z_0 \\rangle - \\lambda \\langle \\opB z_k - \\mu (z_k - z_\\star), z_k - z_\\star \\rangle \\nonumber \\\\\n    &\n    \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 + \\langle \\opB z_k , (q_k - \\lambda) z_k - q_k z_0 + \\lambda z_\\star \\rangle \\\\\n        & \\quad + \\mu(\\lambda-q_k) \\|z_k\\|^2 + 2 \\mu \\langle z_k, q_k z_0 - \\lambda z_\\star \\rangle - q_k \\mu \\|z_0\\|^2 + \\lambda\\mu \\|z_\\star\\|^2  \n    \\end{aligned} \\nonumber \\\\\n","start":98335,"end":98929}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"第一次配方","source_units":[{"raw_tex":"    &\n    \\begin{aligned}\n        & = p_k \\|\\opB z_k\\|^2 + \\langle \\opB z_k , (q_k - \\lambda) z_k - q_k z_0 + \\lambda z_\\star \\rangle \\\\\n        & \\quad + \\frac{\\mu}{\\lambda - q_k} \\left\\| (\\lambda - q_k) z_k + q_k z_0 - \\lambda z_\\star \\right\\|^2 - \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 .\n    \\end{aligned} \\nonumber \\\\ \n","start":98929,"end":99271}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"算子残差的配方下界","source_units":[{"raw_tex":"    &\n    \\begin{aligned}\n        & = \\left( p_k - \\frac{\\lambda - q_k}{4\\mu} \\right) \\|\\opB z_k\\|^2 + \\frac{\\mu}{\\lambda - q_k} \\left\\| \\frac{\\lambda - q_k}{2\\mu} \\opB z_k - (\\lambda - q_k) z_k - q_k z_0 + \\lambda z_\\star \\right\\|^2 \\\\\n        & \\quad - \\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2 .\n    \\end{aligned} \\nonumber\n\\end{align}\n","start":99271,"end":99626}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"特殊参数证明非负","source_units":[{"raw_tex":"Adding $\\frac{q_k \\lambda \\mu}{\\lambda - q_k} \\|z_0 - z_\\star\\|^2$ to the both sides and taking $\\lambda = q_k + 4\\mu p_k$, we obtain \n\\begin{align*}\n    0 & \\le W_k + \\left( \\frac{q_k^2}{4p_k} + \\mu q_k \\right) \\|z_0 - z_\\star\\|^2 = V_k ,\n\\end{align*}\nwhere the last equality follows by applying \\eqref{eqn:sm-EAG_pkqk} and then plugging in the expression $q_k = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{1}{\\beta_{k-1}} = \\frac{1}{(1+2\\alpha\\mu)^{k-1}} \\frac{(1+2\\alpha\\mu)^k - 1}{2\\alpha\\mu}$.\n\n\n","start":99626,"end":100120}]},{"id":"S052","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S052"],"declared_external":true,"declaration_text":["任务原始先决事实列表：Standard Euclidean inner-product identities, Young's inequality, and completing the square。"],"source_only":false,"title":"欧氏内积恒等式","source_units":[]},{"id":"S053","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S053"],"declared_external":true,"declaration_text":["任务原始先决事实列表中的 completing the square。"],"source_only":false,"title":"二次式配方","source_units":[]},{"id":"S054","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S054"],"declared_external":true,"declaration_text":["任务原始先决事实列表：Elementary geometric-series identities and scalar minimization。"],"source_only":false,"title":"有限几何级数","source_units":[]},{"id":"S055","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S055"],"declared_external":true,"declaration_text":["任务原始先决事实列表中的 scalar minimization。"],"source_only":false,"title":"一维系数最小化","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055"}}}}
{"record_id":"2211.04889v4","paper":{"title":"Exponential convergence of sum-of-squares hierarchies for trigonometric polynomials","authors":["Bach, Francis","Rudi, Alessandro"],"arxiv_id":"2211.04889","version":"v4","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/2211.04889v4","pdf_url":"https://arxiv.org/pdf/2211.04889v4","doi":"10.1137/22m1540818"},"context":{"raw_tex":"\\paragraph{Periodic functions and trigonometric polynomials}\n We consider $1$-periodic continuous functions $f$ on $\\rb^d$, which we restrict to  $f: [0,1]^d \\to \\rb$, with summable Fourier series, that is, for which the ``F-norm'':\n \\[\n \\| f\\|_{\\rm F} = \\sum_{\\omega \\in \\zb^d} | \\hat{f}(\\omega)|\n \\]\n is finite, where $\\ds \\hat{f}(\\omega) = \\int_{[0,1]^d} f(x) e^{-2i \\pi \\omega^\\top x} dx $ is the Fourier series of $f$. We can then represent such functions as sums of complex exponentials $f(x) = \\sum_{\\omega \\in \\zb^d} \\hat{f}(\\omega) e^{2i\\pi \\omega^\\top x}$, where the series is uniformly convergent. A key property of the F-norm is its relationship with the $L_\\infty$-norm, that is, $\\| f\\|_\\infty \\leqslant \\|f\\|_{\\rm F}$.\n \n We consider real-valued functions $f$, that is, such that $\\hat{f}(-\\omega) = \\hat{f}^\\ast(\\omega)$ for all $\\omega \\in \\zb^d$. This implies we can write $f(x)$ as real linear combinations of $\\cos 2\\pi \\omega^\\top x$ and $\\sin 2\\pi \\omega^\\top x$, and thus as a linear combination of monomials in $\\cos 2\\pi x_1,\\dots,\\cos 2\\pi x_d,$ $ \\sin 2\\pi x_1,\\dots,\\sin 2\\pi x_d$. This includes, but is not limited to,  trigonometric polynomials of degree $2r$, which corresponds to functions with vanishing Fourier series coefficients $\\hat{f}(\\omega)$ for $\\|\\omega\\|_\\infty > 2r$, that is,\n \\[\n f(x) = \\sum_{\\| \\omega \\|_\\infty  \\leqslant 2r} \\hat{f}(\\omega) e^{2i\\pi \\omega^\\top x}.\n \\]\n{We denote by $x_\\ast$ any minimizer of $f$ on $[0,1]^d$ and by $f_\\ast$ the minimal value (which does not depend on the chosen minimizer).}\n\\paragraph{Hierarchies of SOS optimization problems}\n We consider the maximization of $c$ such that $f-c$ is a sum of squares of trigonometric polynomials of degree $s$. We denote the optimal value by $c_\\ast(f,s)$. The principle behind SOS hierarchies is that when $f$ is a trigonometric polynomial, this optimization problem can be solved as a finite-dimensional semi-definite programming (SDP) problem that we describe in \\mysec{sdp} and thus be solved with a variety of algorithms~(see, e.g.,~\\cite{helmberg1996interior}).\n \n If $f$ is a trigonometric polynomial of degree $2r$ with  $r \\leqslant s$, then the value is finite, and we always have $c_\\ast(f,s) \\leqslant  \\inf_{x \\in [0,1]^d} f(x) = f_\\ast$. Our main goal is to provide a bound:\n \\BEQ\n \\label{eq:EPS}\n0 \\leqslant  \\inf_{x \\in [0,1]^d} f(x) -c_\\ast(f,s)  \\leqslant \\varepsilon(f,s),\n \\EEQ\n depending on simple properties of $f$, and that tends to zero when $s$ tends to $+\\infty$ with an explicit dependence in $s$.\n","units":[{"raw_tex":"\\paragraph{Periodic functions and trigonometric polynomials}\n We consider $1$-periodic continuous functions $f$ on $\\rb^d$, which we restrict to  $f: [0,1]^d \\to \\rb$, with summable Fourier series, that is, for which the ``F-norm'':\n \\[\n \\| f\\|_{\\rm F} = \\sum_{\\omega \\in \\zb^d} | \\hat{f}(\\omega)|\n \\]\n is finite, where $\\ds \\hat{f}(\\omega) = \\int_{[0,1]^d} f(x) e^{-2i \\pi \\omega^\\top x} dx $ is the Fourier series of $f$. We can then represent such functions as sums of complex exponentials $f(x) = \\sum_{\\omega \\in \\zb^d} \\hat{f}(\\omega) e^{2i\\pi \\omega^\\top x}$, where the series is uniformly convergent. A key property of the F-norm is its relationship with the $L_\\infty$-norm, that is, $\\| f\\|_\\infty \\leqslant \\|f\\|_{\\rm F}$.\n \n We consider real-valued functions $f$, that is, such that $\\hat{f}(-\\omega) = \\hat{f}^\\ast(\\omega)$ for all $\\omega \\in \\zb^d$. This implies we can write $f(x)$ as real linear combinations of $\\cos 2\\pi \\omega^\\top x$ and $\\sin 2\\pi \\omega^\\top x$, and thus as a linear combination of monomials in $\\cos 2\\pi x_1,\\dots,\\cos 2\\pi x_d,$ $ \\sin 2\\pi x_1,\\dots,\\sin 2\\pi x_d$. This includes, but is not limited to,  trigonometric polynomials of degree $2r$, which corresponds to functions with vanishing Fourier series coefficients $\\hat{f}(\\omega)$ for $\\|\\omega\\|_\\infty > 2r$, that is,\n \\[\n f(x) = \\sum_{\\| \\omega \\|_\\infty  \\leqslant 2r} \\hat{f}(\\omega) e^{2i\\pi \\omega^\\top x}.\n \\]\n{We denote by $x_\\ast$ any minimizer of $f$ on $[0,1]^d$ and by $f_\\ast$ the minimal value (which does not depend on the chosen minimizer).}\n\\paragraph{Hierarchies of SOS optimization problems}\n We consider the maximization of $c$ such that $f-c$ is a sum of squares of trigonometric polynomials of degree $s$. We denote the optimal value by $c_\\ast(f,s)$. The principle behind SOS hierarchies is that when $f$ is a trigonometric polynomial, this optimization problem can be solved as a finite-dimensional semi-definite programming (SDP) problem that we describe in \\mysec{sdp} and thus be solved with a variety of algorithms~(see, e.g.,~\\cite{helmberg1996interior}).\n \n If $f$ is a trigonometric polynomial of degree $2r$ with  $r \\leqslant s$, then the value is finite, and we always have $c_\\ast(f,s) \\leqslant  \\inf_{x \\in [0,1]^d} f(x) = f_\\ast$. Our main goal is to provide a bound:\n \\BEQ\n \\label{eq:EPS}\n0 \\leqslant  \\inf_{x \\in [0,1]^d} f(x) -c_\\ast(f,s)  \\leqslant \\varepsilon(f,s),\n \\EEQ\n depending on simple properties of $f$, and that tends to zero when $s$ tends to $+\\infty$ with an explicit dependence in $s$.\n","start":8821,"end":11366}]},"statement":{"raw_tex":"\\begin{theorem}\n\\label{theo:nocond}\nFor any trigonometric polynomial $f$ of degree less than $2r$, we have, for any $s \\geqslant 3r$, and for $\\bar{f} = \\hat{f}(0)$ the mean-value of $f$:\n\\[\n\\varepsilon(f,s) \\leqslant  \\| f - \\bar{f}\\|_{\\rm F}\\cdot   \n\\Big[  \\Big( 1 - \\frac{6r^2}{s^2} \\Big)^{-d} - 1 \\Big] \\sim_{s \\to +\\infty} \\| f - \\bar{f} \\|_{\\rm F}\\cdot    \\frac{ 6 r^2 d }{s^2}.\n\\]\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\n\\label{theo:nocond}\nFor any trigonometric polynomial $f$ of degree less than $2r$, we have, for any $s \\geqslant 3r$, and for $\\bar{f} = \\hat{f}(0)$ the mean-value of $f$:\n\\[\n\\varepsilon(f,s) \\leqslant  \\| f - \\bar{f}\\|_{\\rm F}\\cdot   \n\\Big[  \\Big( 1 - \\frac{6r^2}{s^2} \\Big)^{-d} - 1 \\Big] \\sim_{s \\to +\\infty} \\| f - \\bar{f} \\|_{\\rm F}\\cdot    \\frac{ 6 r^2 d }{s^2}.\n\\]\n\\end{theorem}","start":23628,"end":24029}]},"proof":{"raw_tex":"\\begin{proof}\n We here follow the proof technique of~\\cite{fang2021sum,laurent2022effective} based on integral operators by adapting it to trigonometric polynomials of degree $2r$, which are easier to deal with than spherical harmonics or regular polynomials through the use of Fourier series. We consider the following integral operator on $1$-periodic functions on  $ [0,1]^d$ to $ \\mathbb{R}$, defined as \n \\BEQ\n \\label{eq:Th}\n Th(x) = \\int_{[0, 1]^d} |q(x-y)|^2 h(y) dy,\n \\EEQ\n for a well-chosen $1$-periodic function $ q$ which is a trigonometric polynomial of degree~$ s$. \n {The function $x \\mapsto |q(x-y)|^2$ is an element of the finite-dimensional cone of SOS polynomials of degree $s$, thus,} by design, if $ h$ is a non-negative function, then $ Th$ is a sum of squares of polynomials of degree less than $ s$. We will find $ h$ such that $ Th = f - f_\\ast+b$ for a constant $ b \\geqslant 0$, for $f_\\ast$ the minimal value of $f$, which will prove the result, since then $f = f_\\ast - b + Th$, and $f_\\ast-b$ is smaller than the value of the SOS relaxation $c_\\ast(f,s)$, leading to $f_\\ast - c_\\ast(f,s) \\leqslant b$.\n\nIn the Fourier domain, since convolutions lead to pointwise multiplication and vice-versa, we have for all $\\omega \\in \\zb^d$, where $\\hat{q} \\ast \\hat{q}(\\omega)$ is a shorthand for $(\\hat{q} \\ast \\hat{q})(\\omega)$ : \n\\[\n\\widehat{Th}(\\omega) = \\hat{q} \\ast \\hat{q}(\\omega) \\cdot \\hat{h}(\\omega),\n\\]\n and thus, the candidate $ h$ is defined by its Fourier series, which is equal to zero for $ \\| \\omega\\|_\\infty > 2r$, and to \n \\[ \\frac{\\hat{f}(\\omega)+(b-f_\\ast) 1_{\\omega = 0}}{ \\hat{q}  \\ast \\hat{q}(\\omega)}\\]\n  otherwise. If we impose that $ \\hat{q}  \\ast \\hat{q}(0)=1$, we then have \n\\BEAS\n   f -f_\\ast + b  - h &  = &    \\sum_{\\omega  \\in \\zb^d} \\hat{f}(\\omega) \\Big(  1  - \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)} \\Big) \\exp( 2i \\pi \\omega^\\top \\cdot) \\\\\n& = &    \\sum_{\\omega  \\neq 0 } \\hat{f}(\\omega) \\Big(  1  - \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)} \\Big) \\exp( 2i \\pi \\omega^\\top \\cdot) .\\EEAS\n We then get:\n$ \\| f -f_\\ast + b  - h\\|_\\infty =  \\big\\| \\sum_{\\omega  \\neq 0} \\hat{f}(\\omega) \\big(  1  - \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)} \\big) \\exp( 2i \\pi \\omega^\\top \\cdot) \\big\\|_\\infty.$\n\nUsing that $\\|\\cdot\\|_{\\infty} \\leqslant \\| \\cdot\\|_{\\rm F}$, we get: \n\\[ \\| f -f_\\ast + b \\, - h\\|_\\infty \\leqslant \\!\\sum_{\\omega \\neq 0}| \\hat{f}(\\omega)| \\cdot  \\! \\max_{ \\|\\omega\\|_\\infty \\leqslant 2r}  \\Big|   \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)}  \\, - 1\\Big|\n\\leqslant \\| f - \\bar{f}\\|_{\\rm F}\\cdot  \\max_{ \\|\\omega\\|_\\infty \\leqslant 2r}  \\Big|   \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)}  \\, - 1\\Big| .\n\\]\nThe goal is now to find a good function $ q:[0,1]^d \\to \\rb$ with Fourier support within the  ball of radius $ s$, so that $ \\hat{q}  \\ast  \\hat{q}(\\omega)$ is close to $ 1$ for $ \\| \\omega\\|_\\infty \\leqslant 2 r$, and simply check when $ \\| f -f_\\ast + b \\, - h\\|_\\infty \\leqslant b$.\n\nA simple candidate is \n$\\hat{q}(\\omega)   = \\frac{1}{(2s+1)^{d/2}} 1_{\\|\\omega\\|_\\infty \\leqslant s}$, {based on a ``box kernel''};  we can then compute the convolution and obtain that  $\\hat{q}\\ast \\hat{q}(\\omega) = \\prod_{i=1}^d \\big( 1 - \\frac{|\\omega_i|}{2s+1} \\big)_+ \\geqslant \\big( 1 - \\frac{2r}{2s+1} \\big)^d, $ leading to $b=  \\| f - \\bar{f}\\|_{\\rm F} \\cdot \\big[  \\big( 1 - \\frac{2r}{2s+1} \\big)^{-d} - 1 \\big].$ \nWhen $ s$ goes to infinity, we have the equivalent $ b  \\sim  \\| f - \\bar{f}\\|_{\\rm F}\\cdot    \\frac{ rd }{s} = O(1/s)$, which thus converges to zero, but at a slow rate.\n\n\nA better candidate leads to a rate in $ O(1/s^2)$ (like in~\\cite{fang2021sum,laurent2022effective}), {is based on a a``triangular kernel''} as:\n\\[ \\hat{q}(\\omega) = a \\prod_{i=1}^d \\Big( 1 - \\frac{|\\omega_i|}{s} \\Big)_+, \\]\nwith $a$ a normalizing constant. \nA tedious computation including sums of powers of consecutive integers, {detailed in Appendix~\\ref{app:tedious}}, leads to, for any $\\|\\omega\\|_\\infty\\leqslant s$ (note that $\\hat{q}\\ast \\hat{q}(\\omega)$ is only equal to zero for $\\|\\omega\\|_\\infty>2s$),\n\\BEQ\n\\label{eq:tedious} \\hat{q}\\ast \\hat{q}(\\omega)\n= a^2 \\prod_{i=1}^d \\Big[\n\\frac{2s}{3}+\\frac{1}{3s}  - \\frac{\\omega_i^2}{s} + \\frac{|\\omega_i|}{2s^2}( \\omega_i^2 - 1) \\Big].\n\\EEQ\nThus we need $a^2 = \\frac{1}{(\\frac{2s}{3} + \\frac{1}{3s})^d}$ to get $ \\hat{q}\\ast \\hat{q}(0)=1$ and thus \n\\[\n\\hat{q}\\ast \\hat{q}(\\omega) \n\\geqslant  \\prod_{i=1}^d \\Big( 1- \\frac{1}{\\frac{2s}{3} + \\frac{1}{3s}} \\frac{\\omega_i^2}{s}\\Big)_+\n\\geqslant \\prod_{i=1}^d \\Big( 1- \\frac{3\\omega_i^2}{2s^2}\\Big)_+,\n\\]\n which is greater than $\\big( 1- \\frac{6r^2}{s^2}\\big)_+^d$,\nwhen in addition $\\|\\omega\\|_\\infty \\leqslant 2r $.\nThis leads to, for $ s \\geqslant 3 r \\geqslant \\sqrt{6} r $, \n\\[ b  \\leqslant  \\| f - \\bar{f}\\|_{\\rm F}\\cdot   \n\\Big[  \\Big( 1 - \\frac{6r^2}{s^2} \\Big)^{-d} - 1 \\Big] \\sim  \\| f -  \\bar{f}\\|_{\\rm F}\\cdot    \\frac{ 6 r^2 d }{s^2}.\n\\]\n{Above, the asymptotic equivalent is taken with $s$ tending to infinity, with $r$ and $d$ being fixed.}\n\\end{proof}","units":[{"raw_tex":"\\begin{proof}\n We here follow the proof technique of~\\cite{fang2021sum,laurent2022effective} based on integral operators by adapting it to trigonometric polynomials of degree $2r$, which are easier to deal with than spherical harmonics or regular polynomials through the use of Fourier series. We consider the following integral operator on $1$-periodic functions on  $ [0,1]^d$ to $ \\mathbb{R}$, defined as \n \\BEQ\n \\label{eq:Th}\n Th(x) = \\int_{[0, 1]^d} |q(x-y)|^2 h(y) dy,\n \\EEQ\n for a well-chosen $1$-periodic function $ q$ which is a trigonometric polynomial of degree~$ s$. \n {The function $x \\mapsto |q(x-y)|^2$ is an element of the finite-dimensional cone of SOS polynomials of degree $s$, thus,} by design, if $ h$ is a non-negative function, then $ Th$ is a sum of squares of polynomials of degree less than $ s$. We will find $ h$ such that $ Th = f - f_\\ast+b$ for a constant $ b \\geqslant 0$, for $f_\\ast$ the minimal value of $f$, which will prove the result, since then $f = f_\\ast - b + Th$, and $f_\\ast-b$ is smaller than the value of the SOS relaxation $c_\\ast(f,s)$, leading to $f_\\ast - c_\\ast(f,s) \\leqslant b$.\n\nIn the Fourier domain, since convolutions lead to pointwise multiplication and vice-versa, we have for all $\\omega \\in \\zb^d$, where $\\hat{q} \\ast \\hat{q}(\\omega)$ is a shorthand for $(\\hat{q} \\ast \\hat{q})(\\omega)$ : \n\\[\n\\widehat{Th}(\\omega) = \\hat{q} \\ast \\hat{q}(\\omega) \\cdot \\hat{h}(\\omega),\n\\]\n and thus, the candidate $ h$ is defined by its Fourier series, which is equal to zero for $ \\| \\omega\\|_\\infty > 2r$, and to \n \\[ \\frac{\\hat{f}(\\omega)+(b-f_\\ast) 1_{\\omega = 0}}{ \\hat{q}  \\ast \\hat{q}(\\omega)}\\]\n  otherwise. If we impose that $ \\hat{q}  \\ast \\hat{q}(0)=1$, we then have \n\\BEAS\n   f -f_\\ast + b  - h &  = &    \\sum_{\\omega  \\in \\zb^d} \\hat{f}(\\omega) \\Big(  1  - \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)} \\Big) \\exp( 2i \\pi \\omega^\\top \\cdot) \\\\\n& = &    \\sum_{\\omega  \\neq 0 } \\hat{f}(\\omega) \\Big(  1  - \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)} \\Big) \\exp( 2i \\pi \\omega^\\top \\cdot) .\\EEAS\n We then get:\n$ \\| f -f_\\ast + b  - h\\|_\\infty =  \\big\\| \\sum_{\\omega  \\neq 0} \\hat{f}(\\omega) \\big(  1  - \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)} \\big) \\exp( 2i \\pi \\omega^\\top \\cdot) \\big\\|_\\infty.$\n\nUsing that $\\|\\cdot\\|_{\\infty} \\leqslant \\| \\cdot\\|_{\\rm F}$, we get: \n\\[ \\| f -f_\\ast + b \\, - h\\|_\\infty \\leqslant \\!\\sum_{\\omega \\neq 0}| \\hat{f}(\\omega)| \\cdot  \\! \\max_{ \\|\\omega\\|_\\infty \\leqslant 2r}  \\Big|   \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)}  \\, - 1\\Big|\n\\leqslant \\| f - \\bar{f}\\|_{\\rm F}\\cdot  \\max_{ \\|\\omega\\|_\\infty \\leqslant 2r}  \\Big|   \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)}  \\, - 1\\Big| .\n\\]\nThe goal is now to find a good function $ q:[0,1]^d \\to \\rb$ with Fourier support within the  ball of radius $ s$, so that $ \\hat{q}  \\ast  \\hat{q}(\\omega)$ is close to $ 1$ for $ \\| \\omega\\|_\\infty \\leqslant 2 r$, and simply check when $ \\| f -f_\\ast + b \\, - h\\|_\\infty \\leqslant b$.\n\nA simple candidate is \n$\\hat{q}(\\omega)   = \\frac{1}{(2s+1)^{d/2}} 1_{\\|\\omega\\|_\\infty \\leqslant s}$, {based on a ``box kernel''};  we can then compute the convolution and obtain that  $\\hat{q}\\ast \\hat{q}(\\omega) = \\prod_{i=1}^d \\big( 1 - \\frac{|\\omega_i|}{2s+1} \\big)_+ \\geqslant \\big( 1 - \\frac{2r}{2s+1} \\big)^d, $ leading to $b=  \\| f - \\bar{f}\\|_{\\rm F} \\cdot \\big[  \\big( 1 - \\frac{2r}{2s+1} \\big)^{-d} - 1 \\big].$ \nWhen $ s$ goes to infinity, we have the equivalent $ b  \\sim  \\| f - \\bar{f}\\|_{\\rm F}\\cdot    \\frac{ rd }{s} = O(1/s)$, which thus converges to zero, but at a slow rate.\n\n\nA better candidate leads to a rate in $ O(1/s^2)$ (like in~\\cite{fang2021sum,laurent2022effective}), {is based on a a``triangular kernel''} as:\n\\[ \\hat{q}(\\omega) = a \\prod_{i=1}^d \\Big( 1 - \\frac{|\\omega_i|}{s} \\Big)_+, \\]\nwith $a$ a normalizing constant. \nA tedious computation including sums of powers of consecutive integers, {detailed in Appendix~\\ref{app:tedious}}, leads to, for any $\\|\\omega\\|_\\infty\\leqslant s$ (note that $\\hat{q}\\ast \\hat{q}(\\omega)$ is only equal to zero for $\\|\\omega\\|_\\infty>2s$),\n\\BEQ\n\\label{eq:tedious} \\hat{q}\\ast \\hat{q}(\\omega)\n= a^2 \\prod_{i=1}^d \\Big[\n\\frac{2s}{3}+\\frac{1}{3s}  - \\frac{\\omega_i^2}{s} + \\frac{|\\omega_i|}{2s^2}( \\omega_i^2 - 1) \\Big].\n\\EEQ\nThus we need $a^2 = \\frac{1}{(\\frac{2s}{3} + \\frac{1}{3s})^d}$ to get $ \\hat{q}\\ast \\hat{q}(0)=1$ and thus \n\\[\n\\hat{q}\\ast \\hat{q}(\\omega) \n\\geqslant  \\prod_{i=1}^d \\Big( 1- \\frac{1}{\\frac{2s}{3} + \\frac{1}{3s}} \\frac{\\omega_i^2}{s}\\Big)_+\n\\geqslant \\prod_{i=1}^d \\Big( 1- \\frac{3\\omega_i^2}{2s^2}\\Big)_+,\n\\]\n which is greater than $\\big( 1- \\frac{6r^2}{s^2}\\big)_+^d$,\nwhen in addition $\\|\\omega\\|_\\infty \\leqslant 2r $.\nThis leads to, for $ s \\geqslant 3 r \\geqslant \\sqrt{6} r $, \n\\[ b  \\leqslant  \\| f - \\bar{f}\\|_{\\rm F}\\cdot   \n\\Big[  \\Big( 1 - \\frac{6r^2}{s^2} \\Big)^{-d} - 1 \\Big] \\sim  \\| f -  \\bar{f}\\|_{\\rm F}\\cdot    \\frac{ 6 r^2 d }{s^2}.\n\\]\n{Above, the asymptotic equivalent is taken with $s$ tending to infinity, with $r$ and $d$ being fixed.}\n\\end{proof}","start":24030,"end":29033}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"周期函数优化场景","source_units":[{"raw_tex":"\\paragraph{Periodic functions and trigonometric polynomials}\n We consider $1$-periodic continuous functions $f$ on $\\rb^d$, which we restrict to  $f: [0,1]^d \\to \\rb$, with summable Fourier series, that is, for which the ``F-norm'':\n","start":8821,"end":9054}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"Fourier 系数与 F 范数","source_units":[{"raw_tex":" \\[\n \\| f\\|_{\\rm F} = \\sum_{\\omega \\in \\zb^d} | \\hat{f}(\\omega)|\n \\]\n is finite, where $\\ds \\hat{f}(\\omega) = \\int_{[0,1]^d} f(x) e^{-2i \\pi \\omega^\\top x} dx $ is the Fourier series of $f$. We can then represent such functions as sums of complex exponentials $f(x) = \\sum_{\\omega \\in \\zb^d} \\hat{f}(\\omega) e^{2i\\pi \\omega^\\top x}$, where the series is uniformly convergent. A key property of the F-norm is its relationship with the $L_\\infty$-norm, that is, $\\| f\\|_\\infty \\leqslant \\|f\\|_{\\rm F}$.\n","start":9054,"end":9555}]},{"id":"S003","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"F 范数控制一致范数","source_units":[{"raw_tex":" is finite, where $\\ds \\hat{f}(\\omega) = \\int_{[0,1]^d} f(x) e^{-2i \\pi \\omega^\\top x} dx $ is the Fourier series of $f$. We can then represent such functions as sums of complex exponentials $f(x) = \\sum_{\\omega \\in \\zb^d} \\hat{f}(\\omega) e^{2i\\pi \\omega^\\top x}$, where the series is uniformly convergent. A key property of the F-norm is its relationship with the $L_\\infty$-norm, that is, $\\| f\\|_\\infty \\leqslant \\|f\\|_{\\rm F}$.\n","start":9123,"end":9555}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"实值性与共轭对称","source_units":[{"raw_tex":" We consider real-valued functions $f$, that is, such that $\\hat{f}(-\\omega) = \\hat{f}^\\ast(\\omega)$ for all $\\omega \\in \\zb^d$. This implies we can write $f(x)$ as real linear combinations of $\\cos 2\\pi \\omega^\\top x$ and $\\sin 2\\pi \\omega^\\top x$, and thus as a linear combination of monomials in $\\cos 2\\pi x_1,\\dots,\\cos 2\\pi x_d,$ $ \\sin 2\\pi x_1,\\dots,\\sin 2\\pi x_d$. This includes, but is not limited to,  trigonometric polynomials of degree $2r$, which corresponds to functions with vanishing Fourier series coefficients $\\hat{f}(\\omega)$ for $\\|\\omega\\|_\\infty > 2r$, that is,\n","start":9557,"end":10143}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"三角多项式次数","source_units":[{"raw_tex":" We consider real-valued functions $f$, that is, such that $\\hat{f}(-\\omega) = \\hat{f}^\\ast(\\omega)$ for all $\\omega \\in \\zb^d$. This implies we can write $f(x)$ as real linear combinations of $\\cos 2\\pi \\omega^\\top x$ and $\\sin 2\\pi \\omega^\\top x$, and thus as a linear combination of monomials in $\\cos 2\\pi x_1,\\dots,\\cos 2\\pi x_d,$ $ \\sin 2\\pi x_1,\\dots,\\sin 2\\pi x_d$. This includes, but is not limited to,  trigonometric polynomials of degree $2r$, which corresponds to functions with vanishing Fourier series coefficients $\\hat{f}(\\omega)$ for $\\|\\omega\\|_\\infty > 2r$, that is,\n \\[\n f(x) = \\sum_{\\| \\omega \\|_\\infty  \\leqslant 2r} \\hat{f}(\\omega) e^{2i\\pi \\omega^\\top x}.\n \\]\n","start":9557,"end":10241}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"最小值及最小点","source_units":[{"raw_tex":"{We denote by $x_\\ast$ any minimizer of $f$ on $[0,1]^d$ and by $f_\\ast$ the minimal value (which does not depend on the chosen minimizer).}\n","start":10241,"end":10382}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"SOS 层级与最优值","source_units":[{"raw_tex":"\\paragraph{Hierarchies of SOS optimization problems}\n We consider the maximization of $c$ such that $f-c$ is a sum of squares of trigonometric polynomials of degree $s$. We denote the optimal value by $c_\\ast(f,s)$. The principle behind SOS hierarchies is that when $f$ is a trigonometric polynomial, this optimization problem can be solved as a finite-dimensional semi-definite programming (SDP) problem that we describe in \\mysec{sdp} and thus be solved with a variety of algorithms~(see, e.g.,~\\cite{helmberg1996interior}).\n","start":10382,"end":10909}]},{"id":"S008","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"半定规划计算背景","source_units":[{"raw_tex":" We consider the maximization of $c$ such that $f-c$ is a sum of squares of trigonometric polynomials of degree $s$. We denote the optimal value by $c_\\ast(f,s)$. The principle behind SOS hierarchies is that when $f$ is a trigonometric polynomial, this optimization problem can be solved as a finite-dimensional semi-definite programming (SDP) problem that we describe in \\mysec{sdp} and thus be solved with a variety of algorithms~(see, e.g.,~\\cite{helmberg1996interior}).\n","start":10435,"end":10909}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"松弛值有限且为下界","source_units":[{"raw_tex":" If $f$ is a trigonometric polynomial of degree $2r$ with  $r \\leqslant s$, then the value is finite, and we always have $c_\\ast(f,s) \\leqslant  \\inf_{x \\in [0,1]^d} f(x) = f_\\ast$. Our main goal is to provide a bound:\n","start":10911,"end":11130}]},{"id":"S010","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:EPS"],"declared_external":false,"source_only":false,"title":"层级误差控制目标","source_units":[{"raw_tex":" \n If $f$ is a trigonometric polynomial of degree $2r$ with  $r \\leqslant s$, then the value is finite, and we always have $c_\\ast(f,s) \\leqslant  \\inf_{x \\in [0,1]^d} f(x) = f_\\ast$. Our main goal is to provide a bound:\n \\BEQ\n \\label{eq:EPS}\n0 \\leqslant  \\inf_{x \\in [0,1]^d} f(x) -c_\\ast(f,s)  \\leqslant \\varepsilon(f,s),\n \\EEQ\n depending on simple properties of $f$, and that tends to zero when $s$ tends to $+\\infty$ with an explicit dependence in $s$.\n","start":10909,"end":11366}]},{"id":"S011","semantic_type":"theorem","kind":"theorem","labels":["theo:nocond"],"declared_external":false,"source_only":false,"title":"无附加条件的二阶收敛","source_units":[{"raw_tex":"\\begin{theorem}\n\\label{theo:nocond}\nFor any trigonometric polynomial $f$ of degree less than $2r$, we have, for any $s \\geqslant 3r$, and for $\\bar{f} = \\hat{f}(0)$ the mean-value of $f$:\n\\[\n\\varepsilon(f,s) \\leqslant  \\| f - \\bar{f}\\|_{\\rm F}\\cdot   \n\\Big[  \\Big( 1 - \\frac{6r^2}{s^2} \\Big)^{-d} - 1 \\Big] \\sim_{s \\to +\\infty} \\| f - \\bar{f} \\|_{\\rm F}\\cdot    \\frac{ 6 r^2 d }{s^2}.\n\\]\n\\end{theorem}","start":23628,"end":24029}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","labels":["sec:nocond","theo:nocond"],"declared_external":false,"source_only":false,"title":"次数及层级参数限制","source_units":[{"raw_tex":"For any trigonometric polynomial $f$ of degree less than $2r$, we have, for any $s \\geqslant 3r$, and for $\\bar{f} = \\hat{f}(0)$ the mean-value of $f$:\n","start":23664,"end":23816}]},{"id":"S013","semantic_type":"notation","kind":"notation","labels":["sec:nocond","theo:nocond"],"declared_external":false,"source_only":false,"title":"均值与非零频率部分","source_units":[{"raw_tex":"For any trigonometric polynomial $f$ of degree less than $2r$, we have, for any $s \\geqslant 3r$, and for $\\bar{f} = \\hat{f}(0)$ the mean-value of $f$:\n","start":23664,"end":23816}]},{"id":"S014","semantic_type":"proof","kind":"proof","labels":["eq:Th","eq:tedious"],"declared_external":false,"source_only":false,"title":"主定理完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n We here follow the proof technique of~\\cite{fang2021sum,laurent2022effective} based on integral operators by adapting it to trigonometric polynomials of degree $2r$, which are easier to deal with than spherical harmonics or regular polynomials through the use of Fourier series. We consider the following integral operator on $1$-periodic functions on  $ [0,1]^d$ to $ \\mathbb{R}$, defined as \n \\BEQ\n \\label{eq:Th}\n Th(x) = \\int_{[0, 1]^d} |q(x-y)|^2 h(y) dy,\n \\EEQ\n for a well-chosen $1$-periodic function $ q$ which is a trigonometric polynomial of degree~$ s$. \n {The function $x \\mapsto |q(x-y)|^2$ is an element of the finite-dimensional cone of SOS polynomials of degree $s$, thus,} by design, if $ h$ is a non-negative function, then $ Th$ is a sum of squares of polynomials of degree less than $ s$. We will find $ h$ such that $ Th = f - f_\\ast+b$ for a constant $ b \\geqslant 0$, for $f_\\ast$ the minimal value of $f$, which will prove the result, since then $f = f_\\ast - b + Th$, and $f_\\ast-b$ is smaller than the value of the SOS relaxation $c_\\ast(f,s)$, leading to $f_\\ast - c_\\ast(f,s) \\leqslant b$.\n\nIn the Fourier domain, since convolutions lead to pointwise multiplication and vice-versa, we have for all $\\omega \\in \\zb^d$, where $\\hat{q} \\ast \\hat{q}(\\omega)$ is a shorthand for $(\\hat{q} \\ast \\hat{q})(\\omega)$ : \n\\[\n\\widehat{Th}(\\omega) = \\hat{q} \\ast \\hat{q}(\\omega) \\cdot \\hat{h}(\\omega),\n\\]\n and thus, the candidate $ h$ is defined by its Fourier series, which is equal to zero for $ \\| \\omega\\|_\\infty > 2r$, and to \n \\[ \\frac{\\hat{f}(\\omega)+(b-f_\\ast) 1_{\\omega = 0}}{ \\hat{q}  \\ast \\hat{q}(\\omega)}\\]\n  otherwise. If we impose that $ \\hat{q}  \\ast \\hat{q}(0)=1$, we then have \n\\BEAS\n   f -f_\\ast + b  - h &  = &    \\sum_{\\omega  \\in \\zb^d} \\hat{f}(\\omega) \\Big(  1  - \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)} \\Big) \\exp( 2i \\pi \\omega^\\top \\cdot) \\\\\n& = &    \\sum_{\\omega  \\neq 0 } \\hat{f}(\\omega) \\Big(  1  - \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)} \\Big) \\exp( 2i \\pi \\omega^\\top \\cdot) .\\EEAS\n We then get:\n$ \\| f -f_\\ast + b  - h\\|_\\infty =  \\big\\| \\sum_{\\omega  \\neq 0} \\hat{f}(\\omega) \\big(  1  - \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)} \\big) \\exp( 2i \\pi \\omega^\\top \\cdot) \\big\\|_\\infty.$\n\nUsing that $\\|\\cdot\\|_{\\infty} \\leqslant \\| \\cdot\\|_{\\rm F}$, we get: \n\\[ \\| f -f_\\ast + b \\, - h\\|_\\infty \\leqslant \\!\\sum_{\\omega \\neq 0}| \\hat{f}(\\omega)| \\cdot  \\! \\max_{ \\|\\omega\\|_\\infty \\leqslant 2r}  \\Big|   \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)}  \\, - 1\\Big|\n\\leqslant \\| f - \\bar{f}\\|_{\\rm F}\\cdot  \\max_{ \\|\\omega\\|_\\infty \\leqslant 2r}  \\Big|   \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)}  \\, - 1\\Big| .\n\\]\nThe goal is now to find a good function $ q:[0,1]^d \\to \\rb$ with Fourier support within the  ball of radius $ s$, so that $ \\hat{q}  \\ast  \\hat{q}(\\omega)$ is close to $ 1$ for $ \\| \\omega\\|_\\infty \\leqslant 2 r$, and simply check when $ \\| f -f_\\ast + b \\, - h\\|_\\infty \\leqslant b$.\n\nA simple candidate is \n$\\hat{q}(\\omega)   = \\frac{1}{(2s+1)^{d/2}} 1_{\\|\\omega\\|_\\infty \\leqslant s}$, {based on a ``box kernel''};  we can then compute the convolution and obtain that  $\\hat{q}\\ast \\hat{q}(\\omega) = \\prod_{i=1}^d \\big( 1 - \\frac{|\\omega_i|}{2s+1} \\big)_+ \\geqslant \\big( 1 - \\frac{2r}{2s+1} \\big)^d, $ leading to $b=  \\| f - \\bar{f}\\|_{\\rm F} \\cdot \\big[  \\big( 1 - \\frac{2r}{2s+1} \\big)^{-d} - 1 \\big].$ \nWhen $ s$ goes to infinity, we have the equivalent $ b  \\sim  \\| f - \\bar{f}\\|_{\\rm F}\\cdot    \\frac{ rd }{s} = O(1/s)$, which thus converges to zero, but at a slow rate.\n\n\nA better candidate leads to a rate in $ O(1/s^2)$ (like in~\\cite{fang2021sum,laurent2022effective}), {is based on a a``triangular kernel''} as:\n\\[ \\hat{q}(\\omega) = a \\prod_{i=1}^d \\Big( 1 - \\frac{|\\omega_i|}{s} \\Big)_+, \\]\nwith $a$ a normalizing constant. \nA tedious computation including sums of powers of consecutive integers, {detailed in Appendix~\\ref{app:tedious}}, leads to, for any $\\|\\omega\\|_\\infty\\leqslant s$ (note that $\\hat{q}\\ast \\hat{q}(\\omega)$ is only equal to zero for $\\|\\omega\\|_\\infty>2s$),\n\\BEQ\n\\label{eq:tedious} \\hat{q}\\ast \\hat{q}(\\omega)\n= a^2 \\prod_{i=1}^d \\Big[\n\\frac{2s}{3}+\\frac{1}{3s}  - \\frac{\\omega_i^2}{s} + \\frac{|\\omega_i|}{2s^2}( \\omega_i^2 - 1) \\Big].\n\\EEQ\nThus we need $a^2 = \\frac{1}{(\\frac{2s}{3} + \\frac{1}{3s})^d}$ to get $ \\hat{q}\\ast \\hat{q}(0)=1$ and thus \n\\[\n\\hat{q}\\ast \\hat{q}(\\omega) \n\\geqslant  \\prod_{i=1}^d \\Big( 1- \\frac{1}{\\frac{2s}{3} + \\frac{1}{3s}} \\frac{\\omega_i^2}{s}\\Big)_+\n\\geqslant \\prod_{i=1}^d \\Big( 1- \\frac{3\\omega_i^2}{2s^2}\\Big)_+,\n\\]\n which is greater than $\\big( 1- \\frac{6r^2}{s^2}\\big)_+^d$,\nwhen in addition $\\|\\omega\\|_\\infty \\leqslant 2r $.\nThis leads to, for $ s \\geqslant 3 r \\geqslant \\sqrt{6} r $, \n\\[ b  \\leqslant  \\| f - \\bar{f}\\|_{\\rm F}\\cdot   \n\\Big[  \\Big( 1 - \\frac{6r^2}{s^2} \\Big)^{-d} - 1 \\Big] \\sim  \\| f -  \\bar{f}\\|_{\\rm F}\\cdot    \\frac{ 6 r^2 d }{s^2}.\n\\]\n{Above, the asymptotic equivalent is taken with $s$ tending to infinity, with $r$ and $d$ being fixed.}\n\\end{proof}","start":24030,"end":29033}]},{"id":"S015","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"积分算子方法来源","source_units":[{"raw_tex":"\\begin{proof}\n We here follow the proof technique of~\\cite{fang2021sum,laurent2022effective} based on integral operators by adapting it to trigonometric polynomials of degree $2r$, which are easier to deal with than spherical harmonics or regular polynomials through the use of Fourier series. We consider the following integral operator on $1$-periodic functions on  $ [0,1]^d$ to $ \\mathbb{R}$, defined as \n","start":24030,"end":24439}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":["eq:Th"],"declared_external":false,"source_only":false,"title":"平方核积分算子","source_units":[{"raw_tex":" We here follow the proof technique of~\\cite{fang2021sum,laurent2022effective} based on integral operators by adapting it to trigonometric polynomials of degree $2r$, which are easier to deal with than spherical harmonics or regular polynomials through the use of Fourier series. We consider the following integral operator on $1$-periodic functions on  $ [0,1]^d$ to $ \\mathbb{R}$, defined as \n \\BEQ\n \\label{eq:Th}\n Th(x) = \\int_{[0, 1]^d} |q(x-y)|^2 h(y) dy,\n \\EEQ\n for a well-chosen $1$-periodic function $ q$ which is a trigonometric polynomial of degree~$ s$. \n","start":24044,"end":24610}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"非负输入产生平方和","source_units":[{"raw_tex":" {The function $x \\mapsto |q(x-y)|^2$ is an element of the finite-dimensional cone of SOS polynomials of degree $s$, thus,} by design, if $ h$ is a non-negative function, then $ Th$ is a sum of squares of polynomials of degree less than $ s$. We will find $ h$ such that $ Th = f - f_\\ast+b$ for a constant $ b \\geqslant 0$, for $f_\\ast$ the minimal value of $f$, which will prove the result, since then $f = f_\\ast - b + Th$, and $f_\\ast-b$ is smaller than the value of the SOS relaxation $c_\\ast(f,s)$, leading to $f_\\ast - c_\\ast(f,s) \\leqslant b$.\n","start":24610,"end":25162}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"证书转为松弛误差界","source_units":[{"raw_tex":" {The function $x \\mapsto |q(x-y)|^2$ is an element of the finite-dimensional cone of SOS polynomials of degree $s$, thus,} by design, if $ h$ is a non-negative function, then $ Th$ is a sum of squares of polynomials of degree less than $ s$. We will find $ h$ such that $ Th = f - f_\\ast+b$ for a constant $ b \\geqslant 0$, for $f_\\ast$ the minimal value of $f$, which will prove the result, since then $f = f_\\ast - b + Th$, and $f_\\ast-b$ is smaller than the value of the SOS relaxation $c_\\ast(f,s)$, leading to $f_\\ast - c_\\ast(f,s) \\leqslant b$.\n","start":24610,"end":25162}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"卷积算子的 Fourier 乘子","source_units":[{"raw_tex":"\nIn the Fourier domain, since convolutions lead to pointwise multiplication and vice-versa, we have for all $\\omega \\in \\zb^d$, where $\\hat{q} \\ast \\hat{q}(\\omega)$ is a shorthand for $(\\hat{q} \\ast \\hat{q})(\\omega)$ : \n\\[\n\\widehat{Th}(\\omega) = \\hat{q} \\ast \\hat{q}(\\omega) \\cdot \\hat{h}(\\omega),\n\\]\n","start":25162,"end":25463}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"按乘子反演构造输入","source_units":[{"raw_tex":" and thus, the candidate $ h$ is defined by its Fourier series, which is equal to zero for $ \\| \\omega\\|_\\infty > 2r$, and to \n \\[ \\frac{\\hat{f}(\\omega)+(b-f_\\ast) 1_{\\omega = 0}}{ \\hat{q}  \\ast \\hat{q}(\\omega)}\\]\n","start":25463,"end":25677}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"归一化消去常数频率","source_units":[{"raw_tex":"  otherwise. If we impose that $ \\hat{q}  \\ast \\hat{q}(0)=1$, we then have \n\\BEAS\n   f -f_\\ast + b  - h &  = &    \\sum_{\\omega  \\in \\zb^d} \\hat{f}(\\omega) \\Big(  1  - \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)} \\Big) \\exp( 2i \\pi \\omega^\\top \\cdot) \\\\\n& = &    \\sum_{\\omega  \\neq 0 } \\hat{f}(\\omega) \\Big(  1  - \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)} \\Big) \\exp( 2i \\pi \\omega^\\top \\cdot) .\\EEAS\n We then get:\n$ \\| f -f_\\ast + b  - h\\|_\\infty =  \\big\\| \\sum_{\\omega  \\neq 0} \\hat{f}(\\omega) \\big(  1  - \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)} \\big) \\exp( 2i \\pi \\omega^\\top \\cdot) \\big\\|_\\infty.$\n","start":25677,"end":26268}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"反演误差的统一上界","source_units":[{"raw_tex":"\nUsing that $\\|\\cdot\\|_{\\infty} \\leqslant \\| \\cdot\\|_{\\rm F}$, we get: \n\\[ \\| f -f_\\ast + b \\, - h\\|_\\infty \\leqslant \\!\\sum_{\\omega \\neq 0}| \\hat{f}(\\omega)| \\cdot  \\! \\max_{ \\|\\omega\\|_\\infty \\leqslant 2r}  \\Big|   \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)}  \\, - 1\\Big|\n\\leqslant \\| f - \\bar{f}\\|_{\\rm F}\\cdot  \\max_{ \\|\\omega\\|_\\infty \\leqslant 2r}  \\Big|   \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)}  \\, - 1\\Big| .\n\\]\n","start":26268,"end":26683}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"非负性检查条件","source_units":[{"raw_tex":"The goal is now to find a good function $ q:[0,1]^d \\to \\rb$ with Fourier support within the  ball of radius $ s$, so that $ \\hat{q}  \\ast  \\hat{q}(\\omega)$ is close to $ 1$ for $ \\| \\omega\\|_\\infty \\leqslant 2 r$, and simply check when $ \\| f -f_\\ast + b \\, - h\\|_\\infty \\leqslant b$.\n\n","start":26683,"end":26970}]},{"id":"S024","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"箱形核的较慢比较界","source_units":[{"raw_tex":"A simple candidate is \n$\\hat{q}(\\omega)   = \\frac{1}{(2s+1)^{d/2}} 1_{\\|\\omega\\|_\\infty \\leqslant s}$, {based on a ``box kernel''};  we can then compute the convolution and obtain that  $\\hat{q}\\ast \\hat{q}(\\omega) = \\prod_{i=1}^d \\big( 1 - \\frac{|\\omega_i|}{2s+1} \\big)_+ \\geqslant \\big( 1 - \\frac{2r}{2s+1} \\big)^d, $ leading to $b=  \\| f - \\bar{f}\\|_{\\rm F} \\cdot \\big[  \\big( 1 - \\frac{2r}{2s+1} \\big)^{-d} - 1 \\big].$ \nWhen $ s$ goes to infinity, we have the equivalent $ b  \\sim  \\| f - \\bar{f}\\|_{\\rm F}\\cdot    \\frac{ rd }{s} = O(1/s)$, which thus converges to zero, but at a slow rate.\n\n\n","start":26970,"end":27567}]},{"id":"S025","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"张量积三角核","source_units":[{"raw_tex":"A better candidate leads to a rate in $ O(1/s^2)$ (like in~\\cite{fang2021sum,laurent2022effective}), {is based on a a``triangular kernel''} as:\n\\[ \\hat{q}(\\omega) = a \\prod_{i=1}^d \\Big( 1 - \\frac{|\\omega_i|}{s} \\Big)_+, \\]\nwith $a$ a normalizing constant. \n","start":27567,"end":27825}]},{"id":"S026","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:tedious"],"declared_external":false,"source_only":false,"title":"三角核卷积显式公式","source_units":[{"raw_tex":"A tedious computation including sums of powers of consecutive integers, {detailed in Appendix~\\ref{app:tedious}}, leads to, for any $\\|\\omega\\|_\\infty\\leqslant s$ (note that $\\hat{q}\\ast \\hat{q}(\\omega)$ is only equal to zero for $\\|\\omega\\|_\\infty>2s$),\n\\BEQ\n\\label{eq:tedious} \\hat{q}\\ast \\hat{q}(\\omega)\n= a^2 \\prod_{i=1}^d \\Big[\n\\frac{2s}{3}+\\frac{1}{3s}  - \\frac{\\omega_i^2}{s} + \\frac{|\\omega_i|}{2s^2}( \\omega_i^2 - 1) \\Big].\n\\EEQ\n","start":27825,"end":28263}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"确定核的归一化常数","source_units":[{"raw_tex":"Thus we need $a^2 = \\frac{1}{(\\frac{2s}{3} + \\frac{1}{3s})^d}$ to get $ \\hat{q}\\ast \\hat{q}(0)=1$ and thus \n","start":28263,"end":28371}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"卷积乘子的二次下界","source_units":[{"raw_tex":"\\[\n\\hat{q}\\ast \\hat{q}(\\omega) \n\\geqslant  \\prod_{i=1}^d \\Big( 1- \\frac{1}{\\frac{2s}{3} + \\frac{1}{3s}} \\frac{\\omega_i^2}{s}\\Big)_+\n\\geqslant \\prod_{i=1}^d \\Big( 1- \\frac{3\\omega_i^2}{2s^2}\\Big)_+,\n\\]\n","start":28371,"end":28572}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"目标频带上的严格正性","source_units":[{"raw_tex":" which is greater than $\\big( 1- \\frac{6r^2}{s^2}\\big)_+^d$,\nwhen in addition $\\|\\omega\\|_\\infty \\leqslant 2r $.\nThis leads to, for $ s \\geqslant 3 r \\geqslant \\sqrt{6} r $, \n","start":28572,"end":28747}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"校正常数的定量控制","source_units":[{"raw_tex":"This leads to, for $ s \\geqslant 3 r \\geqslant \\sqrt{6} r $, \n\\[ b  \\leqslant  \\| f - \\bar{f}\\|_{\\rm F}\\cdot   \n\\Big[  \\Big( 1 - \\frac{6r^2}{s^2} \\Big)^{-d} - 1 \\Big] \\sim  \\| f -  \\bar{f}\\|_{\\rm F}\\cdot    \\frac{ 6 r^2 d }{s^2}.\n\\]\n","start":28685,"end":28918}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"闭合非负输入与 SOS 证书","source_units":[{"raw_tex":" {The function $x \\mapsto |q(x-y)|^2$ is an element of the finite-dimensional cone of SOS polynomials of degree $s$, thus,} by design, if $ h$ is a non-negative function, then $ Th$ is a sum of squares of polynomials of degree less than $ s$. We will find $ h$ such that $ Th = f - f_\\ast+b$ for a constant $ b \\geqslant 0$, for $f_\\ast$ the minimal value of $f$, which will prove the result, since then $f = f_\\ast - b + Th$, and $f_\\ast-b$ is smaller than the value of the SOS relaxation $c_\\ast(f,s)$, leading to $f_\\ast - c_\\ast(f,s) \\leqslant b$.\n","start":24610,"end":25162},{"raw_tex":"The goal is now to find a good function $ q:[0,1]^d \\to \\rb$ with Fourier support within the  ball of radius $ s$, so that $ \\hat{q}  \\ast  \\hat{q}(\\omega)$ is close to $ 1$ for $ \\| \\omega\\|_\\infty \\leqslant 2 r$, and simply check when $ \\| f -f_\\ast + b \\, - h\\|_\\infty \\leqslant b$.\n","start":26683,"end":26969},{"raw_tex":"This leads to, for $ s \\geqslant 3 r \\geqslant \\sqrt{6} r $, \n\\[ b  \\leqslant  \\| f - \\bar{f}\\|_{\\rm F}\\cdot   \n\\Big[  \\Big( 1 - \\frac{6r^2}{s^2} \\Big)^{-d} - 1 \\Big] \\sim  \\| f -  \\bar{f}\\|_{\\rm F}\\cdot    \\frac{ 6 r^2 d }{s^2}.\n\\]\n","start":28685,"end":28918}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"固定维度次数的渐近项","source_units":[{"raw_tex":"\\[ b  \\leqslant  \\| f - \\bar{f}\\|_{\\rm F}\\cdot   \n\\Big[  \\Big( 1 - \\frac{6r^2}{s^2} \\Big)^{-d} - 1 \\Big] \\sim  \\| f -  \\bar{f}\\|_{\\rm F}\\cdot    \\frac{ 6 r^2 d }{s^2}.\n\\]\n{Above, the asymptotic equivalent is taken with $s$ tending to infinity, with $r$ and $d$ being fixed.}\n\\end{proof}","start":28747,"end":29033}]},{"id":"S033","semantic_type":"lemma","kind":"lemma","labels":[],"declared_external":false,"source_only":true,"title":"一维三角序列卷积恒等式","source_units":[{"raw_tex":"{Given the function $b: \\mathbb{Z} \\to \\rb$ defined as $b(\\omega)  = ( s - | \\omega |)_+$, we  need to compute the convolution $b\\! \\ast\\! b(\\omega)$ for $|\\omega| \\leqslant s$. Since $b$ is even, so is $b \\!\\ast\\! b$ and we can thus consider $\\omega \\in [0,s]$. We want to show that $b \\!\\ast\\! b(\\omega) = \\frac{s(2s^2+1)}{3}  - \\frac{\\omega}{2} - s \\omega^2 + \\frac{\\omega^3}{2}$.}\n","start":64845,"end":65230}]},{"id":"S034","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":true,"title":"附录的一维三角序列","source_units":[{"raw_tex":"{Given the function $b: \\mathbb{Z} \\to \\rb$ defined as $b(\\omega)  = ( s - | \\omega |)_+$, we  need to compute the convolution $b\\! \\ast\\! b(\\omega)$ for $|\\omega| \\leqslant s$. Since $b$ is even, so is $b \\!\\ast\\! b$ and we can thus consider $\\omega \\in [0,s]$. We want to show that $b \\!\\ast\\! b(\\omega) = \\frac{s(2s^2+1)}{3}  - \\frac{\\omega}{2} - s \\omega^2 + \\frac{\\omega^3}{2}$.}\n","start":64845,"end":65230}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"利用偶性缩减频率范围","source_units":[{"raw_tex":"{Given the function $b: \\mathbb{Z} \\to \\rb$ defined as $b(\\omega)  = ( s - | \\omega |)_+$, we  need to compute the convolution $b\\! \\ast\\! b(\\omega)$ for $|\\omega| \\leqslant s$. Since $b$ is even, so is $b \\!\\ast\\! b$ and we can thus consider $\\omega \\in [0,s]$. We want to show that $b \\!\\ast\\! b(\\omega) = \\frac{s(2s^2+1)}{3}  - \\frac{\\omega}{2} - s \\omega^2 + \\frac{\\omega^3}{2}$.}\n","start":64845,"end":65230}]},{"id":"S036","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"一维卷积恒等式作者证明","source_units":[{"raw_tex":"{We can split the sum \n$b \\!\\ast\\! b(\\omega) = \\sum_{i\\in \\mathbb{Z}} b(i) b(\\omega - i) $ as follows,\n\\BEAS\n&   & \\sum_{i = \\omega - s}^0 ( s + i) ( s - \\omega + i)\n+\n\\sum_{i= 1}^\\omega ( s - i) ( s - \\omega + i)\n+\n\\sum_{i = \\omega +1 }^s ( s  - i) ( s - i + \\omega)\n\\\\\n& = & \\sum_{i = \\omega - s}^0 \\!\\! \\big[ i^2 + (2s-\\omega)i + s(s-\\omega) \\big]\n+\n\\sum_{i = 1}^\\omega \\big[ - i^2 +\\omega i + s(s-\\omega) \\big]\n\\\\[-.35cm]\n& &\\hspace*{7cm}\n+\n\\sum_{i = \\omega +1 }^s \\!\\! \\big[ i^2 - (2s+\\omega)i + s(s+\\omega) \\big].\n\\EEAS\nThen, \nusing $\\sum_{i=1}^t \\! i = \\frac{t(t+1)}{2}= \\frac{1}{2}(t^2+t)$ and $\\sum_{i=1}^t \\! i^2 = \\frac{t(t+1)(2t+1)}{6} = \\frac{1}{6}(2 t^3+3t^2 + t)$, we get:\n\\BEAS\n&  & \\frac{1}{6}(s-\\omega)(s-\\omega+1) ( 2s-2\\omega+1)\n- \\frac{1}{2} (s-\\omega)(s-\\omega+1) ( 2s-\\omega)\\\\\n& & + s(s-\\omega)(s-\\omega+1)   -  2 \\frac{1}{6} \\omega(\\omega+1)( 2\\omega+1)\n+ \\frac{1}{2} \\omega(\\omega+1)(2s+2\\omega) + s(s-\\omega) \\omega \\\\\n& & + \\frac{1}{6} s(s+1)( 2s+1)\n- \\frac{1}{2} s(s +1) ( 2s+\\omega) + s(s+\\omega)(s-\\omega),\n\\EEAS\nleading to\n\\BEAS\n& & \\frac{1}{6} (s-\\omega)(s-\\omega+1) (  \\omega+1 - 4s )\n+ s(s-\\omega) ( 2s +\\omega + 1  ) \\\\\n& & + \\frac{1}{6}(2 s^3+3s^2 +s ) - \\frac{1}{3}(2 \\omega^3+3\\omega^2 +\\omega )\n+( \\omega^2+\\omega)(s+\\omega) - \\frac{1}{2}(s^2+s)(2s+\\omega) \\\\\n& = &\\big[   \\frac{1}{6}   -\\frac{2}{3} + 1\\big] \\omega^3 +\\big[  \\frac{1}{6}(  {1-4s-s-1-s}) -s - 1+ s+1\\big]\\omega^2  \\\\\n& & +\n\\big[ \\frac{1}{6}(  (s+1)(4s\\!-\\!1)-s(1\\!-\\!4s)+s(s\\!+\\!1)) + s^2 -s(2s\\!+\\!1) - \\frac{1}{3}+s  - \\frac{1}{2}(s^2\\!+\\! s) \\big] \\omega  \\\\\n& & + \\big[ \ns(s+1)\\frac{1}{6}(1-4s)+s^2(2s+1)+\\frac{1}{6}(2 s^3+3s^2 +s ) -s(s^2+s)\n\\big]\n\\\\\n& = &\\frac{1}{2}\\omega^3 - s \\omega^2 - \\frac{1}{2}\\omega +\\frac{s}{3} + \\frac{2}{3}s^3. \n\\EEAS\n","start":65232,"end":66990}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"将卷积拆成三个求和区间","source_units":[{"raw_tex":"{We can split the sum \n$b \\!\\ast\\! b(\\omega) = \\sum_{i\\in \\mathbb{Z}} b(i) b(\\omega - i) $ as follows,\n\\BEAS\n&   & \\sum_{i = \\omega - s}^0 ( s + i) ( s - \\omega + i)\n+\n\\sum_{i= 1}^\\omega ( s - i) ( s - \\omega + i)\n+\n\\sum_{i = \\omega +1 }^s ( s  - i) ( s - i + \\omega)\n\\\\\n& = & \\sum_{i = \\omega - s}^0 \\!\\! \\big[ i^2 + (2s-\\omega)i + s(s-\\omega) \\big]\n+\n\\sum_{i = 1}^\\omega \\big[ - i^2 +\\omega i + s(s-\\omega) \\big]\n\\\\[-.35cm]\n& &\\hspace*{7cm}\n+\n\\sum_{i = \\omega +1 }^s \\!\\! \\big[ i^2 - (2s+\\omega)i + s(s+\\omega) \\big].\n\\EEAS\n","start":65232,"end":65758}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"代入整数和及平方和","source_units":[{"raw_tex":"Then, \nusing $\\sum_{i=1}^t \\! i = \\frac{t(t+1)}{2}= \\frac{1}{2}(t^2+t)$ and $\\sum_{i=1}^t \\! i^2 = \\frac{t(t+1)(2t+1)}{6} = \\frac{1}{6}(2 t^3+3t^2 + t)$, we get:\n\\BEAS\n&  & \\frac{1}{6}(s-\\omega)(s-\\omega+1) ( 2s-2\\omega+1)\n- \\frac{1}{2} (s-\\omega)(s-\\omega+1) ( 2s-\\omega)\\\\\n& & + s(s-\\omega)(s-\\omega+1)   -  2 \\frac{1}{6} \\omega(\\omega+1)( 2\\omega+1)\n+ \\frac{1}{2} \\omega(\\omega+1)(2s+2\\omega) + s(s-\\omega) \\omega \\\\\n& & + \\frac{1}{6} s(s+1)( 2s+1)\n- \\frac{1}{2} s(s +1) ( 2s+\\omega) + s(s+\\omega)(s-\\omega),\n\\EEAS\n","start":65758,"end":66276}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"整理卷积多项式系数","source_units":[{"raw_tex":"leading to\n\\BEAS\n& & \\frac{1}{6} (s-\\omega)(s-\\omega+1) (  \\omega+1 - 4s )\n+ s(s-\\omega) ( 2s +\\omega + 1  ) \\\\\n& & + \\frac{1}{6}(2 s^3+3s^2 +s ) - \\frac{1}{3}(2 \\omega^3+3\\omega^2 +\\omega )\n+( \\omega^2+\\omega)(s+\\omega) - \\frac{1}{2}(s^2+s)(2s+\\omega) \\\\\n& = &\\big[   \\frac{1}{6}   -\\frac{2}{3} + 1\\big] \\omega^3 +\\big[  \\frac{1}{6}(  {1-4s-s-1-s}) -s - 1+ s+1\\big]\\omega^2  \\\\\n& & +\n\\big[ \\frac{1}{6}(  (s+1)(4s\\!-\\!1)-s(1\\!-\\!4s)+s(s\\!+\\!1)) + s^2 -s(2s\\!+\\!1) - \\frac{1}{3}+s  - \\frac{1}{2}(s^2\\!+\\! s) \\big] \\omega  \\\\\n& & + \\big[ \ns(s+1)\\frac{1}{6}(1-4s)+s^2(2s+1)+\\frac{1}{6}(2 s^3+3s^2 +s ) -s(s^2+s)\n\\big]\n\\\\\n& = &\\frac{1}{2}\\omega^3 - s \\omega^2 - \\frac{1}{2}\\omega +\\frac{s}{3} + \\frac{2}{3}s^3. \n\\EEAS\n","start":66276,"end":66990}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"由一维卷积恢复张量积","source_units":[{"raw_tex":"To get \\eq{tedious}, we then use $\\hat{q} \\ast \\hat{q}(\\omega) = a^2 \\prod_{i=1}^d \\frac{1}{s^2} b \\!\\ast\\! b(|\\omega_i|)$.}\n","start":66990,"end":67115}]},{"id":"S041","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"多维卷积公式完整作者推导","source_units":[{"raw_tex":"{We can split the sum \n$b \\!\\ast\\! b(\\omega) = \\sum_{i\\in \\mathbb{Z}} b(i) b(\\omega - i) $ as follows,\n\\BEAS\n&   & \\sum_{i = \\omega - s}^0 ( s + i) ( s - \\omega + i)\n+\n\\sum_{i= 1}^\\omega ( s - i) ( s - \\omega + i)\n+\n\\sum_{i = \\omega +1 }^s ( s  - i) ( s - i + \\omega)\n\\\\\n& = & \\sum_{i = \\omega - s}^0 \\!\\! \\big[ i^2 + (2s-\\omega)i + s(s-\\omega) \\big]\n+\n\\sum_{i = 1}^\\omega \\big[ - i^2 +\\omega i + s(s-\\omega) \\big]\n\\\\[-.35cm]\n& &\\hspace*{7cm}\n+\n\\sum_{i = \\omega +1 }^s \\!\\! \\big[ i^2 - (2s+\\omega)i + s(s+\\omega) \\big].\n\\EEAS\nThen, \nusing $\\sum_{i=1}^t \\! i = \\frac{t(t+1)}{2}= \\frac{1}{2}(t^2+t)$ and $\\sum_{i=1}^t \\! i^2 = \\frac{t(t+1)(2t+1)}{6} = \\frac{1}{6}(2 t^3+3t^2 + t)$, we get:\n\\BEAS\n&  & \\frac{1}{6}(s-\\omega)(s-\\omega+1) ( 2s-2\\omega+1)\n- \\frac{1}{2} (s-\\omega)(s-\\omega+1) ( 2s-\\omega)\\\\\n& & + s(s-\\omega)(s-\\omega+1)   -  2 \\frac{1}{6} \\omega(\\omega+1)( 2\\omega+1)\n+ \\frac{1}{2} \\omega(\\omega+1)(2s+2\\omega) + s(s-\\omega) \\omega \\\\\n& & + \\frac{1}{6} s(s+1)( 2s+1)\n- \\frac{1}{2} s(s +1) ( 2s+\\omega) + s(s+\\omega)(s-\\omega),\n\\EEAS\nleading to\n\\BEAS\n& & \\frac{1}{6} (s-\\omega)(s-\\omega+1) (  \\omega+1 - 4s )\n+ s(s-\\omega) ( 2s +\\omega + 1  ) \\\\\n& & + \\frac{1}{6}(2 s^3+3s^2 +s ) - \\frac{1}{3}(2 \\omega^3+3\\omega^2 +\\omega )\n+( \\omega^2+\\omega)(s+\\omega) - \\frac{1}{2}(s^2+s)(2s+\\omega) \\\\\n& = &\\big[   \\frac{1}{6}   -\\frac{2}{3} + 1\\big] \\omega^3 +\\big[  \\frac{1}{6}(  {1-4s-s-1-s}) -s - 1+ s+1\\big]\\omega^2  \\\\\n& & +\n\\big[ \\frac{1}{6}(  (s+1)(4s\\!-\\!1)-s(1\\!-\\!4s)+s(s\\!+\\!1)) + s^2 -s(2s\\!+\\!1) - \\frac{1}{3}+s  - \\frac{1}{2}(s^2\\!+\\! s) \\big] \\omega  \\\\\n& & + \\big[ \ns(s+1)\\frac{1}{6}(1-4s)+s^2(2s+1)+\\frac{1}{6}(2 s^3+3s^2 +s ) -s(s^2+s)\n\\big]\n\\\\\n& = &\\frac{1}{2}\\omega^3 - s \\omega^2 - \\frac{1}{2}\\omega +\\frac{s}{3} + \\frac{2}{3}s^3. \n\\EEAS\nTo get \\eq{tedious}, we then use $\\hat{q} \\ast \\hat{q}(\\omega) = a^2 \\prod_{i=1}^d \\frac{1}{s^2} b \\!\\ast\\! b(|\\omega_i|)$.}\n","start":65232,"end":67115}]},{"id":"S042","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisite：Standard Fourier convolution identities and factorization of tensor-product convolutions."],"source_only":false,"title":"周期 Fourier 卷积规则","source_units":[]},{"id":"S043","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisite：Standard Fourier convolution identities and factorization of tensor-product convolutions."],"source_only":false,"title":"张量积卷积因子分解","source_units":[]},{"id":"S044","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisite：Integration of elements of a closed finite-dimensional SOS cone preserves membership in that cone."],"source_only":false,"title":"闭 SOS 锥对非负积分封闭","source_units":[]},{"id":"S045","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisite：Elementary formulas for sums of consecutive integers and their squares."],"source_only":false,"title":"连续整数幂和公式","source_units":[]},{"id":"S046","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator：离散 Cauchy–Schwarz 不等式及平移不改变 ℓ2 范数。"],"source_only":false,"title":"归一化自卷积的上界","source_units":[]},{"id":"S047","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator：实函数在 t=0 的一阶 Taylor 展开。"],"source_only":false,"title":"固定指数的一阶渐近展开","source_units":[]},{"id":"R_C01","semantic_type":"retained_source","kind":"retained_source","labels":[],"declared_external":false,"source_only":false,"title":"原文保留片段","source_units":[{"raw_tex":" \n","start":9555,"end":9557}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S011","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S007","target_id":"S009","kind":"supports"},{"source_id":"S006","target_id":"S009","kind":"supports"},{"source_id":"S006","target_id":"S010","kind":"supports"},{"source_id":"S007","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S005","target_id":"S011","kind":"supports"},{"source_id":"S007","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S013","target_id":"S011","kind":"supports"},{"source_id":"S002","target_id":"S013","kind":"supports"},{"source_id":"S001","target_id":"S016","kind":"supports"},{"source_id":"S005","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S007","target_id":"S017","kind":"supports"},{"source_id":"S044","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S007","target_id":"S018","kind":"supports"},{"source_id":"S006","target_id":"S018","kind":"supports"},{"source_id":"S016","target_id":"S019","kind":"supports"},{"source_id":"S042","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S005","target_id":"S020","kind":"supports"},{"source_id":"S006","target_id":"S020","kind":"supports"},{"source_id":"S012","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S002","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S003","target_id":"S022","kind":"supports"},{"source_id":"S005","target_id":"S022","kind":"supports"},{"source_id":"S013","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S006","target_id":"S023","kind":"supports"},{"source_id":"S002","target_id":"S025","kind":"supports"},{"source_id":"S005","target_id":"S025","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S042","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S037","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S045","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S039","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S040","kind":"supports"},{"source_id":"S034","target_id":"S040","kind":"supports"},{"source_id":"S025","target_id":"S040","kind":"supports"},{"source_id":"S043","target_id":"S040","kind":"supports"},{"source_id":"S036","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S012","target_id":"S029","kind":"supports"},{"source_id":"S025","target_id":"S030","kind":"supports"},{"source_id":"S027","target_id":"S030","kind":"supports"},{"source_id":"S046","target_id":"S030","kind":"supports"},{"source_id":"S022","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S019","target_id":"S031","kind":"supports"},{"source_id":"S020","target_id":"S031","kind":"supports"},{"source_id":"S025","target_id":"S031","kind":"supports"},{"source_id":"S027","target_id":"S031","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S004","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S023","target_id":"S031","kind":"supports"},{"source_id":"S018","target_id":"S031","kind":"supports"},{"source_id":"S009","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S047","target_id":"S032","kind":"supports"},{"source_id":"S016","target_id":"S014","kind":"supports"},{"source_id":"S017","target_id":"S014","kind":"supports"},{"source_id":"S018","target_id":"S014","kind":"supports"},{"source_id":"S019","target_id":"S014","kind":"supports"},{"source_id":"S020","target_id":"S014","kind":"supports"},{"source_id":"S021","target_id":"S014","kind":"supports"},{"source_id":"S022","target_id":"S014","kind":"supports"},{"source_id":"S023","target_id":"S014","kind":"supports"},{"source_id":"S025","target_id":"S014","kind":"supports"},{"source_id":"S026","target_id":"S014","kind":"supports"},{"source_id":"S027","target_id":"S014","kind":"supports"},{"source_id":"S028","target_id":"S014","kind":"supports"},{"source_id":"S029","target_id":"S014","kind":"supports"},{"source_id":"S030","target_id":"S014","kind":"supports"},{"source_id":"S031","target_id":"S014","kind":"supports"},{"source_id":"S032","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S011","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S011","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S007","target_id":"S009","kind":"supports"},{"source_id":"S006","target_id":"S009","kind":"supports"},{"source_id":"S006","target_id":"S010","kind":"supports"},{"source_id":"S007","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S005","target_id":"S011","kind":"supports"},{"source_id":"S007","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S013","target_id":"S011","kind":"supports"},{"source_id":"S002","target_id":"S013","kind":"supports"},{"source_id":"S001","target_id":"S016","kind":"supports"},{"source_id":"S005","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S007","target_id":"S017","kind":"supports"},{"source_id":"S044","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S007","target_id":"S018","kind":"supports"},{"source_id":"S006","target_id":"S018","kind":"supports"},{"source_id":"S016","target_id":"S019","kind":"supports"},{"source_id":"S042","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S005","target_id":"S020","kind":"supports"},{"source_id":"S006","target_id":"S020","kind":"supports"},{"source_id":"S012","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S002","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S003","target_id":"S022","kind":"supports"},{"source_id":"S005","target_id":"S022","kind":"supports"},{"source_id":"S013","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S006","target_id":"S023","kind":"supports"},{"source_id":"S002","target_id":"S025","kind":"supports"},{"source_id":"S005","target_id":"S025","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S042","target_id":"S033","kind":"supports"},{"source_id":"S045","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S040","kind":"supports"},{"source_id":"S034","target_id":"S040","kind":"supports"},{"source_id":"S025","target_id":"S040","kind":"supports"},{"source_id":"S043","target_id":"S040","kind":"supports"},{"source_id":"S033","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S012","target_id":"S029","kind":"supports"},{"source_id":"S025","target_id":"S030","kind":"supports"},{"source_id":"S027","target_id":"S030","kind":"supports"},{"source_id":"S046","target_id":"S030","kind":"supports"},{"source_id":"S022","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S019","target_id":"S031","kind":"supports"},{"source_id":"S020","target_id":"S031","kind":"supports"},{"source_id":"S025","target_id":"S031","kind":"supports"},{"source_id":"S027","target_id":"S031","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S004","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S023","target_id":"S031","kind":"supports"},{"source_id":"S018","target_id":"S031","kind":"supports"},{"source_id":"S009","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S047","target_id":"S032","kind":"supports"},{"source_id":"S016","target_id":"S014","kind":"supports"},{"source_id":"S017","target_id":"S014","kind":"supports"},{"source_id":"S018","target_id":"S014","kind":"supports"},{"source_id":"S019","target_id":"S014","kind":"supports"},{"source_id":"S020","target_id":"S014","kind":"supports"},{"source_id":"S021","target_id":"S014","kind":"supports"},{"source_id":"S022","target_id":"S014","kind":"supports"},{"source_id":"S023","target_id":"S014","kind":"supports"},{"source_id":"S025","target_id":"S014","kind":"supports"},{"source_id":"S026","target_id":"S014","kind":"supports"},{"source_id":"S027","target_id":"S014","kind":"supports"},{"source_id":"S028","target_id":"S014","kind":"supports"},{"source_id":"S029","target_id":"S014","kind":"supports"},{"source_id":"S030","target_id":"S014","kind":"supports"},{"source_id":"S031","target_id":"S014","kind":"supports"},{"source_id":"S032","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S011","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"周期函数优化场景","source_units":[{"raw_tex":"\\paragraph{Periodic functions and trigonometric polynomials}\n We consider $1$-periodic continuous functions $f$ on $\\rb^d$, which we restrict to  $f: [0,1]^d \\to \\rb$, with summable Fourier series, that is, for which the ``F-norm'':\n","start":8821,"end":9054}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"Fourier 系数与 F 范数","source_units":[{"raw_tex":" \\[\n \\| f\\|_{\\rm F} = \\sum_{\\omega \\in \\zb^d} | \\hat{f}(\\omega)|\n \\]\n is finite, where $\\ds \\hat{f}(\\omega) = \\int_{[0,1]^d} f(x) e^{-2i \\pi \\omega^\\top x} dx $ is the Fourier series of $f$. We can then represent such functions as sums of complex exponentials $f(x) = \\sum_{\\omega \\in \\zb^d} \\hat{f}(\\omega) e^{2i\\pi \\omega^\\top x}$, where the series is uniformly convergent. A key property of the F-norm is its relationship with the $L_\\infty$-norm, that is, $\\| f\\|_\\infty \\leqslant \\|f\\|_{\\rm F}$.\n","start":9054,"end":9555}]},{"id":"S003","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"F 范数控制一致范数","source_units":[{"raw_tex":" is finite, where $\\ds \\hat{f}(\\omega) = \\int_{[0,1]^d} f(x) e^{-2i \\pi \\omega^\\top x} dx $ is the Fourier series of $f$. We can then represent such functions as sums of complex exponentials $f(x) = \\sum_{\\omega \\in \\zb^d} \\hat{f}(\\omega) e^{2i\\pi \\omega^\\top x}$, where the series is uniformly convergent. A key property of the F-norm is its relationship with the $L_\\infty$-norm, that is, $\\| f\\|_\\infty \\leqslant \\|f\\|_{\\rm F}$.\n","start":9123,"end":9555}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"实值性与共轭对称","source_units":[{"raw_tex":" We consider real-valued functions $f$, that is, such that $\\hat{f}(-\\omega) = \\hat{f}^\\ast(\\omega)$ for all $\\omega \\in \\zb^d$. This implies we can write $f(x)$ as real linear combinations of $\\cos 2\\pi \\omega^\\top x$ and $\\sin 2\\pi \\omega^\\top x$, and thus as a linear combination of monomials in $\\cos 2\\pi x_1,\\dots,\\cos 2\\pi x_d,$ $ \\sin 2\\pi x_1,\\dots,\\sin 2\\pi x_d$. This includes, but is not limited to,  trigonometric polynomials of degree $2r$, which corresponds to functions with vanishing Fourier series coefficients $\\hat{f}(\\omega)$ for $\\|\\omega\\|_\\infty > 2r$, that is,\n","start":9557,"end":10143}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"三角多项式次数","source_units":[{"raw_tex":" We consider real-valued functions $f$, that is, such that $\\hat{f}(-\\omega) = \\hat{f}^\\ast(\\omega)$ for all $\\omega \\in \\zb^d$. This implies we can write $f(x)$ as real linear combinations of $\\cos 2\\pi \\omega^\\top x$ and $\\sin 2\\pi \\omega^\\top x$, and thus as a linear combination of monomials in $\\cos 2\\pi x_1,\\dots,\\cos 2\\pi x_d,$ $ \\sin 2\\pi x_1,\\dots,\\sin 2\\pi x_d$. This includes, but is not limited to,  trigonometric polynomials of degree $2r$, which corresponds to functions with vanishing Fourier series coefficients $\\hat{f}(\\omega)$ for $\\|\\omega\\|_\\infty > 2r$, that is,\n \\[\n f(x) = \\sum_{\\| \\omega \\|_\\infty  \\leqslant 2r} \\hat{f}(\\omega) e^{2i\\pi \\omega^\\top x}.\n \\]\n","start":9557,"end":10241}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"最小值及最小点","source_units":[{"raw_tex":"{We denote by $x_\\ast$ any minimizer of $f$ on $[0,1]^d$ and by $f_\\ast$ the minimal value (which does not depend on the chosen minimizer).}\n","start":10241,"end":10382}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"SOS 层级与最优值","source_units":[{"raw_tex":"\\paragraph{Hierarchies of SOS optimization problems}\n We consider the maximization of $c$ such that $f-c$ is a sum of squares of trigonometric polynomials of degree $s$. We denote the optimal value by $c_\\ast(f,s)$. The principle behind SOS hierarchies is that when $f$ is a trigonometric polynomial, this optimization problem can be solved as a finite-dimensional semi-definite programming (SDP) problem that we describe in \\mysec{sdp} and thus be solved with a variety of algorithms~(see, e.g.,~\\cite{helmberg1996interior}).\n","start":10382,"end":10909}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"松弛值有限且为下界","source_units":[{"raw_tex":" If $f$ is a trigonometric polynomial of degree $2r$ with  $r \\leqslant s$, then the value is finite, and we always have $c_\\ast(f,s) \\leqslant  \\inf_{x \\in [0,1]^d} f(x) = f_\\ast$. Our main goal is to provide a bound:\n","start":10911,"end":11130}]},{"id":"S010","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:EPS"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"层级误差控制目标","source_units":[{"raw_tex":" \n If $f$ is a trigonometric polynomial of degree $2r$ with  $r \\leqslant s$, then the value is finite, and we always have $c_\\ast(f,s) \\leqslant  \\inf_{x \\in [0,1]^d} f(x) = f_\\ast$. Our main goal is to provide a bound:\n \\BEQ\n \\label{eq:EPS}\n0 \\leqslant  \\inf_{x \\in [0,1]^d} f(x) -c_\\ast(f,s)  \\leqslant \\varepsilon(f,s),\n \\EEQ\n depending on simple properties of $f$, and that tends to zero when $s$ tends to $+\\infty$ with an explicit dependence in $s$.\n","start":10909,"end":11366}]},{"id":"S011","semantic_type":"theorem","kind":"theorem","labels":["theo:nocond"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"无附加条件的二阶收敛","source_units":[{"raw_tex":"\\begin{theorem}\n\\label{theo:nocond}\nFor any trigonometric polynomial $f$ of degree less than $2r$, we have, for any $s \\geqslant 3r$, and for $\\bar{f} = \\hat{f}(0)$ the mean-value of $f$:\n\\[\n\\varepsilon(f,s) \\leqslant  \\| f - \\bar{f}\\|_{\\rm F}\\cdot   \n\\Big[  \\Big( 1 - \\frac{6r^2}{s^2} \\Big)^{-d} - 1 \\Big] \\sim_{s \\to +\\infty} \\| f - \\bar{f} \\|_{\\rm F}\\cdot    \\frac{ 6 r^2 d }{s^2}.\n\\]\n\\end{theorem}","start":23628,"end":24029}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","labels":["sec:nocond","theo:nocond"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"次数及层级参数限制","source_units":[{"raw_tex":"For any trigonometric polynomial $f$ of degree less than $2r$, we have, for any $s \\geqslant 3r$, and for $\\bar{f} = \\hat{f}(0)$ the mean-value of $f$:\n","start":23664,"end":23816}]},{"id":"S013","semantic_type":"notation","kind":"notation","labels":["sec:nocond","theo:nocond"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"均值与非零频率部分","source_units":[{"raw_tex":"For any trigonometric polynomial $f$ of degree less than $2r$, we have, for any $s \\geqslant 3r$, and for $\\bar{f} = \\hat{f}(0)$ the mean-value of $f$:\n","start":23664,"end":23816}]},{"id":"S014","semantic_type":"proof","kind":"proof","labels":["eq:Th","eq:tedious"],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"主定理完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n We here follow the proof technique of~\\cite{fang2021sum,laurent2022effective} based on integral operators by adapting it to trigonometric polynomials of degree $2r$, which are easier to deal with than spherical harmonics or regular polynomials through the use of Fourier series. We consider the following integral operator on $1$-periodic functions on  $ [0,1]^d$ to $ \\mathbb{R}$, defined as \n \\BEQ\n \\label{eq:Th}\n Th(x) = \\int_{[0, 1]^d} |q(x-y)|^2 h(y) dy,\n \\EEQ\n for a well-chosen $1$-periodic function $ q$ which is a trigonometric polynomial of degree~$ s$. \n {The function $x \\mapsto |q(x-y)|^2$ is an element of the finite-dimensional cone of SOS polynomials of degree $s$, thus,} by design, if $ h$ is a non-negative function, then $ Th$ is a sum of squares of polynomials of degree less than $ s$. We will find $ h$ such that $ Th = f - f_\\ast+b$ for a constant $ b \\geqslant 0$, for $f_\\ast$ the minimal value of $f$, which will prove the result, since then $f = f_\\ast - b + Th$, and $f_\\ast-b$ is smaller than the value of the SOS relaxation $c_\\ast(f,s)$, leading to $f_\\ast - c_\\ast(f,s) \\leqslant b$.\n\nIn the Fourier domain, since convolutions lead to pointwise multiplication and vice-versa, we have for all $\\omega \\in \\zb^d$, where $\\hat{q} \\ast \\hat{q}(\\omega)$ is a shorthand for $(\\hat{q} \\ast \\hat{q})(\\omega)$ : \n\\[\n\\widehat{Th}(\\omega) = \\hat{q} \\ast \\hat{q}(\\omega) \\cdot \\hat{h}(\\omega),\n\\]\n and thus, the candidate $ h$ is defined by its Fourier series, which is equal to zero for $ \\| \\omega\\|_\\infty > 2r$, and to \n \\[ \\frac{\\hat{f}(\\omega)+(b-f_\\ast) 1_{\\omega = 0}}{ \\hat{q}  \\ast \\hat{q}(\\omega)}\\]\n  otherwise. If we impose that $ \\hat{q}  \\ast \\hat{q}(0)=1$, we then have \n\\BEAS\n   f -f_\\ast + b  - h &  = &    \\sum_{\\omega  \\in \\zb^d} \\hat{f}(\\omega) \\Big(  1  - \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)} \\Big) \\exp( 2i \\pi \\omega^\\top \\cdot) \\\\\n& = &    \\sum_{\\omega  \\neq 0 } \\hat{f}(\\omega) \\Big(  1  - \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)} \\Big) \\exp( 2i \\pi \\omega^\\top \\cdot) .\\EEAS\n We then get:\n$ \\| f -f_\\ast + b  - h\\|_\\infty =  \\big\\| \\sum_{\\omega  \\neq 0} \\hat{f}(\\omega) \\big(  1  - \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)} \\big) \\exp( 2i \\pi \\omega^\\top \\cdot) \\big\\|_\\infty.$\n\nUsing that $\\|\\cdot\\|_{\\infty} \\leqslant \\| \\cdot\\|_{\\rm F}$, we get: \n\\[ \\| f -f_\\ast + b \\, - h\\|_\\infty \\leqslant \\!\\sum_{\\omega \\neq 0}| \\hat{f}(\\omega)| \\cdot  \\! \\max_{ \\|\\omega\\|_\\infty \\leqslant 2r}  \\Big|   \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)}  \\, - 1\\Big|\n\\leqslant \\| f - \\bar{f}\\|_{\\rm F}\\cdot  \\max_{ \\|\\omega\\|_\\infty \\leqslant 2r}  \\Big|   \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)}  \\, - 1\\Big| .\n\\]\nThe goal is now to find a good function $ q:[0,1]^d \\to \\rb$ with Fourier support within the  ball of radius $ s$, so that $ \\hat{q}  \\ast  \\hat{q}(\\omega)$ is close to $ 1$ for $ \\| \\omega\\|_\\infty \\leqslant 2 r$, and simply check when $ \\| f -f_\\ast + b \\, - h\\|_\\infty \\leqslant b$.\n\nA simple candidate is \n$\\hat{q}(\\omega)   = \\frac{1}{(2s+1)^{d/2}} 1_{\\|\\omega\\|_\\infty \\leqslant s}$, {based on a ``box kernel''};  we can then compute the convolution and obtain that  $\\hat{q}\\ast \\hat{q}(\\omega) = \\prod_{i=1}^d \\big( 1 - \\frac{|\\omega_i|}{2s+1} \\big)_+ \\geqslant \\big( 1 - \\frac{2r}{2s+1} \\big)^d, $ leading to $b=  \\| f - \\bar{f}\\|_{\\rm F} \\cdot \\big[  \\big( 1 - \\frac{2r}{2s+1} \\big)^{-d} - 1 \\big].$ \nWhen $ s$ goes to infinity, we have the equivalent $ b  \\sim  \\| f - \\bar{f}\\|_{\\rm F}\\cdot    \\frac{ rd }{s} = O(1/s)$, which thus converges to zero, but at a slow rate.\n\n\nA better candidate leads to a rate in $ O(1/s^2)$ (like in~\\cite{fang2021sum,laurent2022effective}), {is based on a a``triangular kernel''} as:\n\\[ \\hat{q}(\\omega) = a \\prod_{i=1}^d \\Big( 1 - \\frac{|\\omega_i|}{s} \\Big)_+, \\]\nwith $a$ a normalizing constant. \nA tedious computation including sums of powers of consecutive integers, {detailed in Appendix~\\ref{app:tedious}}, leads to, for any $\\|\\omega\\|_\\infty\\leqslant s$ (note that $\\hat{q}\\ast \\hat{q}(\\omega)$ is only equal to zero for $\\|\\omega\\|_\\infty>2s$),\n\\BEQ\n\\label{eq:tedious} \\hat{q}\\ast \\hat{q}(\\omega)\n= a^2 \\prod_{i=1}^d \\Big[\n\\frac{2s}{3}+\\frac{1}{3s}  - \\frac{\\omega_i^2}{s} + \\frac{|\\omega_i|}{2s^2}( \\omega_i^2 - 1) \\Big].\n\\EEQ\nThus we need $a^2 = \\frac{1}{(\\frac{2s}{3} + \\frac{1}{3s})^d}$ to get $ \\hat{q}\\ast \\hat{q}(0)=1$ and thus \n\\[\n\\hat{q}\\ast \\hat{q}(\\omega) \n\\geqslant  \\prod_{i=1}^d \\Big( 1- \\frac{1}{\\frac{2s}{3} + \\frac{1}{3s}} \\frac{\\omega_i^2}{s}\\Big)_+\n\\geqslant \\prod_{i=1}^d \\Big( 1- \\frac{3\\omega_i^2}{2s^2}\\Big)_+,\n\\]\n which is greater than $\\big( 1- \\frac{6r^2}{s^2}\\big)_+^d$,\nwhen in addition $\\|\\omega\\|_\\infty \\leqslant 2r $.\nThis leads to, for $ s \\geqslant 3 r \\geqslant \\sqrt{6} r $, \n\\[ b  \\leqslant  \\| f - \\bar{f}\\|_{\\rm F}\\cdot   \n\\Big[  \\Big( 1 - \\frac{6r^2}{s^2} \\Big)^{-d} - 1 \\Big] \\sim  \\| f -  \\bar{f}\\|_{\\rm F}\\cdot    \\frac{ 6 r^2 d }{s^2}.\n\\]\n{Above, the asymptotic equivalent is taken with $s$ tending to infinity, with $r$ and $d$ being fixed.}\n\\end{proof}","start":24030,"end":29033}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":["eq:Th"],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"平方核积分算子","source_units":[{"raw_tex":" We here follow the proof technique of~\\cite{fang2021sum,laurent2022effective} based on integral operators by adapting it to trigonometric polynomials of degree $2r$, which are easier to deal with than spherical harmonics or regular polynomials through the use of Fourier series. We consider the following integral operator on $1$-periodic functions on  $ [0,1]^d$ to $ \\mathbb{R}$, defined as \n \\BEQ\n \\label{eq:Th}\n Th(x) = \\int_{[0, 1]^d} |q(x-y)|^2 h(y) dy,\n \\EEQ\n for a well-chosen $1$-periodic function $ q$ which is a trigonometric polynomial of degree~$ s$. \n","start":24044,"end":24610}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"非负输入产生平方和","source_units":[{"raw_tex":" {The function $x \\mapsto |q(x-y)|^2$ is an element of the finite-dimensional cone of SOS polynomials of degree $s$, thus,} by design, if $ h$ is a non-negative function, then $ Th$ is a sum of squares of polynomials of degree less than $ s$. We will find $ h$ such that $ Th = f - f_\\ast+b$ for a constant $ b \\geqslant 0$, for $f_\\ast$ the minimal value of $f$, which will prove the result, since then $f = f_\\ast - b + Th$, and $f_\\ast-b$ is smaller than the value of the SOS relaxation $c_\\ast(f,s)$, leading to $f_\\ast - c_\\ast(f,s) \\leqslant b$.\n","start":24610,"end":25162}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"证书转为松弛误差界","source_units":[{"raw_tex":" {The function $x \\mapsto |q(x-y)|^2$ is an element of the finite-dimensional cone of SOS polynomials of degree $s$, thus,} by design, if $ h$ is a non-negative function, then $ Th$ is a sum of squares of polynomials of degree less than $ s$. We will find $ h$ such that $ Th = f - f_\\ast+b$ for a constant $ b \\geqslant 0$, for $f_\\ast$ the minimal value of $f$, which will prove the result, since then $f = f_\\ast - b + Th$, and $f_\\ast-b$ is smaller than the value of the SOS relaxation $c_\\ast(f,s)$, leading to $f_\\ast - c_\\ast(f,s) \\leqslant b$.\n","start":24610,"end":25162}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"卷积算子的 Fourier 乘子","source_units":[{"raw_tex":"\nIn the Fourier domain, since convolutions lead to pointwise multiplication and vice-versa, we have for all $\\omega \\in \\zb^d$, where $\\hat{q} \\ast \\hat{q}(\\omega)$ is a shorthand for $(\\hat{q} \\ast \\hat{q})(\\omega)$ : \n\\[\n\\widehat{Th}(\\omega) = \\hat{q} \\ast \\hat{q}(\\omega) \\cdot \\hat{h}(\\omega),\n\\]\n","start":25162,"end":25463}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"按乘子反演构造输入","source_units":[{"raw_tex":" and thus, the candidate $ h$ is defined by its Fourier series, which is equal to zero for $ \\| \\omega\\|_\\infty > 2r$, and to \n \\[ \\frac{\\hat{f}(\\omega)+(b-f_\\ast) 1_{\\omega = 0}}{ \\hat{q}  \\ast \\hat{q}(\\omega)}\\]\n","start":25463,"end":25677}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"归一化消去常数频率","source_units":[{"raw_tex":"  otherwise. If we impose that $ \\hat{q}  \\ast \\hat{q}(0)=1$, we then have \n\\BEAS\n   f -f_\\ast + b  - h &  = &    \\sum_{\\omega  \\in \\zb^d} \\hat{f}(\\omega) \\Big(  1  - \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)} \\Big) \\exp( 2i \\pi \\omega^\\top \\cdot) \\\\\n& = &    \\sum_{\\omega  \\neq 0 } \\hat{f}(\\omega) \\Big(  1  - \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)} \\Big) \\exp( 2i \\pi \\omega^\\top \\cdot) .\\EEAS\n We then get:\n$ \\| f -f_\\ast + b  - h\\|_\\infty =  \\big\\| \\sum_{\\omega  \\neq 0} \\hat{f}(\\omega) \\big(  1  - \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)} \\big) \\exp( 2i \\pi \\omega^\\top \\cdot) \\big\\|_\\infty.$\n","start":25677,"end":26268}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"反演误差的统一上界","source_units":[{"raw_tex":"\nUsing that $\\|\\cdot\\|_{\\infty} \\leqslant \\| \\cdot\\|_{\\rm F}$, we get: \n\\[ \\| f -f_\\ast + b \\, - h\\|_\\infty \\leqslant \\!\\sum_{\\omega \\neq 0}| \\hat{f}(\\omega)| \\cdot  \\! \\max_{ \\|\\omega\\|_\\infty \\leqslant 2r}  \\Big|   \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)}  \\, - 1\\Big|\n\\leqslant \\| f - \\bar{f}\\|_{\\rm F}\\cdot  \\max_{ \\|\\omega\\|_\\infty \\leqslant 2r}  \\Big|   \\frac{1}{\\hat{q} \\ast \\hat{q}(\\omega)}  \\, - 1\\Big| .\n\\]\n","start":26268,"end":26683}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"非负性检查条件","source_units":[{"raw_tex":"The goal is now to find a good function $ q:[0,1]^d \\to \\rb$ with Fourier support within the  ball of radius $ s$, so that $ \\hat{q}  \\ast  \\hat{q}(\\omega)$ is close to $ 1$ for $ \\| \\omega\\|_\\infty \\leqslant 2 r$, and simply check when $ \\| f -f_\\ast + b \\, - h\\|_\\infty \\leqslant b$.\n\n","start":26683,"end":26970}]},{"id":"S024","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"箱形核的较慢比较界","source_units":[{"raw_tex":"A simple candidate is \n$\\hat{q}(\\omega)   = \\frac{1}{(2s+1)^{d/2}} 1_{\\|\\omega\\|_\\infty \\leqslant s}$, {based on a ``box kernel''};  we can then compute the convolution and obtain that  $\\hat{q}\\ast \\hat{q}(\\omega) = \\prod_{i=1}^d \\big( 1 - \\frac{|\\omega_i|}{2s+1} \\big)_+ \\geqslant \\big( 1 - \\frac{2r}{2s+1} \\big)^d, $ leading to $b=  \\| f - \\bar{f}\\|_{\\rm F} \\cdot \\big[  \\big( 1 - \\frac{2r}{2s+1} \\big)^{-d} - 1 \\big].$ \nWhen $ s$ goes to infinity, we have the equivalent $ b  \\sim  \\| f - \\bar{f}\\|_{\\rm F}\\cdot    \\frac{ rd }{s} = O(1/s)$, which thus converges to zero, but at a slow rate.\n\n\n","start":26970,"end":27567}]},{"id":"S025","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"张量积三角核","source_units":[{"raw_tex":"A better candidate leads to a rate in $ O(1/s^2)$ (like in~\\cite{fang2021sum,laurent2022effective}), {is based on a a``triangular kernel''} as:\n\\[ \\hat{q}(\\omega) = a \\prod_{i=1}^d \\Big( 1 - \\frac{|\\omega_i|}{s} \\Big)_+, \\]\nwith $a$ a normalizing constant. \n","start":27567,"end":27825}]},{"id":"S026","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:tedious"],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"三角核卷积显式公式","source_units":[{"raw_tex":"A tedious computation including sums of powers of consecutive integers, {detailed in Appendix~\\ref{app:tedious}}, leads to, for any $\\|\\omega\\|_\\infty\\leqslant s$ (note that $\\hat{q}\\ast \\hat{q}(\\omega)$ is only equal to zero for $\\|\\omega\\|_\\infty>2s$),\n\\BEQ\n\\label{eq:tedious} \\hat{q}\\ast \\hat{q}(\\omega)\n= a^2 \\prod_{i=1}^d \\Big[\n\\frac{2s}{3}+\\frac{1}{3s}  - \\frac{\\omega_i^2}{s} + \\frac{|\\omega_i|}{2s^2}( \\omega_i^2 - 1) \\Big].\n\\EEQ\n","start":27825,"end":28263}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"确定核的归一化常数","source_units":[{"raw_tex":"Thus we need $a^2 = \\frac{1}{(\\frac{2s}{3} + \\frac{1}{3s})^d}$ to get $ \\hat{q}\\ast \\hat{q}(0)=1$ and thus \n","start":28263,"end":28371}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"卷积乘子的二次下界","source_units":[{"raw_tex":"\\[\n\\hat{q}\\ast \\hat{q}(\\omega) \n\\geqslant  \\prod_{i=1}^d \\Big( 1- \\frac{1}{\\frac{2s}{3} + \\frac{1}{3s}} \\frac{\\omega_i^2}{s}\\Big)_+\n\\geqslant \\prod_{i=1}^d \\Big( 1- \\frac{3\\omega_i^2}{2s^2}\\Big)_+,\n\\]\n","start":28371,"end":28572}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"目标频带上的严格正性","source_units":[{"raw_tex":" which is greater than $\\big( 1- \\frac{6r^2}{s^2}\\big)_+^d$,\nwhen in addition $\\|\\omega\\|_\\infty \\leqslant 2r $.\nThis leads to, for $ s \\geqslant 3 r \\geqslant \\sqrt{6} r $, \n","start":28572,"end":28747}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"校正常数的定量控制","source_units":[{"raw_tex":"This leads to, for $ s \\geqslant 3 r \\geqslant \\sqrt{6} r $, \n\\[ b  \\leqslant  \\| f - \\bar{f}\\|_{\\rm F}\\cdot   \n\\Big[  \\Big( 1 - \\frac{6r^2}{s^2} \\Big)^{-d} - 1 \\Big] \\sim  \\| f -  \\bar{f}\\|_{\\rm F}\\cdot    \\frac{ 6 r^2 d }{s^2}.\n\\]\n","start":28685,"end":28918}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"闭合非负输入与 SOS 证书","source_units":[{"raw_tex":" {The function $x \\mapsto |q(x-y)|^2$ is an element of the finite-dimensional cone of SOS polynomials of degree $s$, thus,} by design, if $ h$ is a non-negative function, then $ Th$ is a sum of squares of polynomials of degree less than $ s$. We will find $ h$ such that $ Th = f - f_\\ast+b$ for a constant $ b \\geqslant 0$, for $f_\\ast$ the minimal value of $f$, which will prove the result, since then $f = f_\\ast - b + Th$, and $f_\\ast-b$ is smaller than the value of the SOS relaxation $c_\\ast(f,s)$, leading to $f_\\ast - c_\\ast(f,s) \\leqslant b$.\n","start":24610,"end":25162},{"raw_tex":"The goal is now to find a good function $ q:[0,1]^d \\to \\rb$ with Fourier support within the  ball of radius $ s$, so that $ \\hat{q}  \\ast  \\hat{q}(\\omega)$ is close to $ 1$ for $ \\| \\omega\\|_\\infty \\leqslant 2 r$, and simply check when $ \\| f -f_\\ast + b \\, - h\\|_\\infty \\leqslant b$.\n","start":26683,"end":26969},{"raw_tex":"This leads to, for $ s \\geqslant 3 r \\geqslant \\sqrt{6} r $, \n\\[ b  \\leqslant  \\| f - \\bar{f}\\|_{\\rm F}\\cdot   \n\\Big[  \\Big( 1 - \\frac{6r^2}{s^2} \\Big)^{-d} - 1 \\Big] \\sim  \\| f -  \\bar{f}\\|_{\\rm F}\\cdot    \\frac{ 6 r^2 d }{s^2}.\n\\]\n","start":28685,"end":28918}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"固定维度次数的渐近项","source_units":[{"raw_tex":"\\[ b  \\leqslant  \\| f - \\bar{f}\\|_{\\rm F}\\cdot   \n\\Big[  \\Big( 1 - \\frac{6r^2}{s^2} \\Big)^{-d} - 1 \\Big] \\sim  \\| f -  \\bar{f}\\|_{\\rm F}\\cdot    \\frac{ 6 r^2 d }{s^2}.\n\\]\n{Above, the asymptotic equivalent is taken with $s$ tending to infinity, with $r$ and $d$ being fixed.}\n\\end{proof}","start":28747,"end":29033}]},{"id":"S033","semantic_type":"lemma","kind":"lemma","labels":[],"member_ids":["S033","S035","S036","S037","S038","S039"],"declared_external":false,"source_only":true,"title":"一维三角序列卷积恒等式","source_units":[{"raw_tex":"{Given the function $b: \\mathbb{Z} \\to \\rb$ defined as $b(\\omega)  = ( s - | \\omega |)_+$, we  need to compute the convolution $b\\! \\ast\\! b(\\omega)$ for $|\\omega| \\leqslant s$. Since $b$ is even, so is $b \\!\\ast\\! b$ and we can thus consider $\\omega \\in [0,s]$. We want to show that $b \\!\\ast\\! b(\\omega) = \\frac{s(2s^2+1)}{3}  - \\frac{\\omega}{2} - s \\omega^2 + \\frac{\\omega^3}{2}$.}\n","start":64845,"end":65230},{"raw_tex":"{We can split the sum \n$b \\!\\ast\\! b(\\omega) = \\sum_{i\\in \\mathbb{Z}} b(i) b(\\omega - i) $ as follows,\n\\BEAS\n&   & \\sum_{i = \\omega - s}^0 ( s + i) ( s - \\omega + i)\n+\n\\sum_{i= 1}^\\omega ( s - i) ( s - \\omega + i)\n+\n\\sum_{i = \\omega +1 }^s ( s  - i) ( s - i + \\omega)\n\\\\\n& = & \\sum_{i = \\omega - s}^0 \\!\\! \\big[ i^2 + (2s-\\omega)i + s(s-\\omega) \\big]\n+\n\\sum_{i = 1}^\\omega \\big[ - i^2 +\\omega i + s(s-\\omega) \\big]\n\\\\[-.35cm]\n& &\\hspace*{7cm}\n+\n\\sum_{i = \\omega +1 }^s \\!\\! \\big[ i^2 - (2s+\\omega)i + s(s+\\omega) \\big].\n\\EEAS\n","start":65232,"end":65758},{"raw_tex":"{We can split the sum \n$b \\!\\ast\\! b(\\omega) = \\sum_{i\\in \\mathbb{Z}} b(i) b(\\omega - i) $ as follows,\n\\BEAS\n&   & \\sum_{i = \\omega - s}^0 ( s + i) ( s - \\omega + i)\n+\n\\sum_{i= 1}^\\omega ( s - i) ( s - \\omega + i)\n+\n\\sum_{i = \\omega +1 }^s ( s  - i) ( s - i + \\omega)\n\\\\\n& = & \\sum_{i = \\omega - s}^0 \\!\\! \\big[ i^2 + (2s-\\omega)i + s(s-\\omega) \\big]\n+\n\\sum_{i = 1}^\\omega \\big[ - i^2 +\\omega i + s(s-\\omega) \\big]\n\\\\[-.35cm]\n& &\\hspace*{7cm}\n+\n\\sum_{i = \\omega +1 }^s \\!\\! \\big[ i^2 - (2s+\\omega)i + s(s+\\omega) \\big].\n\\EEAS\nThen, \nusing $\\sum_{i=1}^t \\! i = \\frac{t(t+1)}{2}= \\frac{1}{2}(t^2+t)$ and $\\sum_{i=1}^t \\! i^2 = \\frac{t(t+1)(2t+1)}{6} = \\frac{1}{6}(2 t^3+3t^2 + t)$, we get:\n\\BEAS\n&  & \\frac{1}{6}(s-\\omega)(s-\\omega+1) ( 2s-2\\omega+1)\n- \\frac{1}{2} (s-\\omega)(s-\\omega+1) ( 2s-\\omega)\\\\\n& & + s(s-\\omega)(s-\\omega+1)   -  2 \\frac{1}{6} \\omega(\\omega+1)( 2\\omega+1)\n+ \\frac{1}{2} \\omega(\\omega+1)(2s+2\\omega) + s(s-\\omega) \\omega \\\\\n& & + \\frac{1}{6} s(s+1)( 2s+1)\n- \\frac{1}{2} s(s +1) ( 2s+\\omega) + s(s+\\omega)(s-\\omega),\n\\EEAS\nleading to\n\\BEAS\n& & \\frac{1}{6} (s-\\omega)(s-\\omega+1) (  \\omega+1 - 4s )\n+ s(s-\\omega) ( 2s +\\omega + 1  ) \\\\\n& & + \\frac{1}{6}(2 s^3+3s^2 +s ) - \\frac{1}{3}(2 \\omega^3+3\\omega^2 +\\omega )\n+( \\omega^2+\\omega)(s+\\omega) - \\frac{1}{2}(s^2+s)(2s+\\omega) \\\\\n& = &\\big[   \\frac{1}{6}   -\\frac{2}{3} + 1\\big] \\omega^3 +\\big[  \\frac{1}{6}(  {1-4s-s-1-s}) -s - 1+ s+1\\big]\\omega^2  \\\\\n& & +\n\\big[ \\frac{1}{6}(  (s+1)(4s\\!-\\!1)-s(1\\!-\\!4s)+s(s\\!+\\!1)) + s^2 -s(2s\\!+\\!1) - \\frac{1}{3}+s  - \\frac{1}{2}(s^2\\!+\\! s) \\big] \\omega  \\\\\n& & + \\big[ \ns(s+1)\\frac{1}{6}(1-4s)+s^2(2s+1)+\\frac{1}{6}(2 s^3+3s^2 +s ) -s(s^2+s)\n\\big]\n\\\\\n& = &\\frac{1}{2}\\omega^3 - s \\omega^2 - \\frac{1}{2}\\omega +\\frac{s}{3} + \\frac{2}{3}s^3. \n\\EEAS\n","start":65232,"end":66990},{"raw_tex":"Then, \nusing $\\sum_{i=1}^t \\! i = \\frac{t(t+1)}{2}= \\frac{1}{2}(t^2+t)$ and $\\sum_{i=1}^t \\! i^2 = \\frac{t(t+1)(2t+1)}{6} = \\frac{1}{6}(2 t^3+3t^2 + t)$, we get:\n\\BEAS\n&  & \\frac{1}{6}(s-\\omega)(s-\\omega+1) ( 2s-2\\omega+1)\n- \\frac{1}{2} (s-\\omega)(s-\\omega+1) ( 2s-\\omega)\\\\\n& & + s(s-\\omega)(s-\\omega+1)   -  2 \\frac{1}{6} \\omega(\\omega+1)( 2\\omega+1)\n+ \\frac{1}{2} \\omega(\\omega+1)(2s+2\\omega) + s(s-\\omega) \\omega \\\\\n& & + \\frac{1}{6} s(s+1)( 2s+1)\n- \\frac{1}{2} s(s +1) ( 2s+\\omega) + s(s+\\omega)(s-\\omega),\n\\EEAS\n","start":65758,"end":66276},{"raw_tex":"leading to\n\\BEAS\n& & \\frac{1}{6} (s-\\omega)(s-\\omega+1) (  \\omega+1 - 4s )\n+ s(s-\\omega) ( 2s +\\omega + 1  ) \\\\\n& & + \\frac{1}{6}(2 s^3+3s^2 +s ) - \\frac{1}{3}(2 \\omega^3+3\\omega^2 +\\omega )\n+( \\omega^2+\\omega)(s+\\omega) - \\frac{1}{2}(s^2+s)(2s+\\omega) \\\\\n& = &\\big[   \\frac{1}{6}   -\\frac{2}{3} + 1\\big] \\omega^3 +\\big[  \\frac{1}{6}(  {1-4s-s-1-s}) -s - 1+ s+1\\big]\\omega^2  \\\\\n& & +\n\\big[ \\frac{1}{6}(  (s+1)(4s\\!-\\!1)-s(1\\!-\\!4s)+s(s\\!+\\!1)) + s^2 -s(2s\\!+\\!1) - \\frac{1}{3}+s  - \\frac{1}{2}(s^2\\!+\\! s) \\big] \\omega  \\\\\n& & + \\big[ \ns(s+1)\\frac{1}{6}(1-4s)+s^2(2s+1)+\\frac{1}{6}(2 s^3+3s^2 +s ) -s(s^2+s)\n\\big]\n\\\\\n& = &\\frac{1}{2}\\omega^3 - s \\omega^2 - \\frac{1}{2}\\omega +\\frac{s}{3} + \\frac{2}{3}s^3. \n\\EEAS\n","start":66276,"end":66990}]},{"id":"S034","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S034"],"declared_external":false,"source_only":true,"title":"附录的一维三角序列","source_units":[{"raw_tex":"{Given the function $b: \\mathbb{Z} \\to \\rb$ defined as $b(\\omega)  = ( s - | \\omega |)_+$, we  need to compute the convolution $b\\! \\ast\\! b(\\omega)$ for $|\\omega| \\leqslant s$. Since $b$ is even, so is $b \\!\\ast\\! b$ and we can thus consider $\\omega \\in [0,s]$. We want to show that $b \\!\\ast\\! b(\\omega) = \\frac{s(2s^2+1)}{3}  - \\frac{\\omega}{2} - s \\omega^2 + \\frac{\\omega^3}{2}$.}\n","start":64845,"end":65230}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S040"],"declared_external":false,"source_only":true,"title":"由一维卷积恢复张量积","source_units":[{"raw_tex":"To get \\eq{tedious}, we then use $\\hat{q} \\ast \\hat{q}(\\omega) = a^2 \\prod_{i=1}^d \\frac{1}{s^2} b \\!\\ast\\! b(|\\omega_i|)$.}\n","start":66990,"end":67115}]},{"id":"S041","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S041"],"declared_external":false,"source_only":true,"title":"多维卷积公式完整作者推导","source_units":[{"raw_tex":"{We can split the sum \n$b \\!\\ast\\! b(\\omega) = \\sum_{i\\in \\mathbb{Z}} b(i) b(\\omega - i) $ as follows,\n\\BEAS\n&   & \\sum_{i = \\omega - s}^0 ( s + i) ( s - \\omega + i)\n+\n\\sum_{i= 1}^\\omega ( s - i) ( s - \\omega + i)\n+\n\\sum_{i = \\omega +1 }^s ( s  - i) ( s - i + \\omega)\n\\\\\n& = & \\sum_{i = \\omega - s}^0 \\!\\! \\big[ i^2 + (2s-\\omega)i + s(s-\\omega) \\big]\n+\n\\sum_{i = 1}^\\omega \\big[ - i^2 +\\omega i + s(s-\\omega) \\big]\n\\\\[-.35cm]\n& &\\hspace*{7cm}\n+\n\\sum_{i = \\omega +1 }^s \\!\\! \\big[ i^2 - (2s+\\omega)i + s(s+\\omega) \\big].\n\\EEAS\nThen, \nusing $\\sum_{i=1}^t \\! i = \\frac{t(t+1)}{2}= \\frac{1}{2}(t^2+t)$ and $\\sum_{i=1}^t \\! i^2 = \\frac{t(t+1)(2t+1)}{6} = \\frac{1}{6}(2 t^3+3t^2 + t)$, we get:\n\\BEAS\n&  & \\frac{1}{6}(s-\\omega)(s-\\omega+1) ( 2s-2\\omega+1)\n- \\frac{1}{2} (s-\\omega)(s-\\omega+1) ( 2s-\\omega)\\\\\n& & + s(s-\\omega)(s-\\omega+1)   -  2 \\frac{1}{6} \\omega(\\omega+1)( 2\\omega+1)\n+ \\frac{1}{2} \\omega(\\omega+1)(2s+2\\omega) + s(s-\\omega) \\omega \\\\\n& & + \\frac{1}{6} s(s+1)( 2s+1)\n- \\frac{1}{2} s(s +1) ( 2s+\\omega) + s(s+\\omega)(s-\\omega),\n\\EEAS\nleading to\n\\BEAS\n& & \\frac{1}{6} (s-\\omega)(s-\\omega+1) (  \\omega+1 - 4s )\n+ s(s-\\omega) ( 2s +\\omega + 1  ) \\\\\n& & + \\frac{1}{6}(2 s^3+3s^2 +s ) - \\frac{1}{3}(2 \\omega^3+3\\omega^2 +\\omega )\n+( \\omega^2+\\omega)(s+\\omega) - \\frac{1}{2}(s^2+s)(2s+\\omega) \\\\\n& = &\\big[   \\frac{1}{6}   -\\frac{2}{3} + 1\\big] \\omega^3 +\\big[  \\frac{1}{6}(  {1-4s-s-1-s}) -s - 1+ s+1\\big]\\omega^2  \\\\\n& & +\n\\big[ \\frac{1}{6}(  (s+1)(4s\\!-\\!1)-s(1\\!-\\!4s)+s(s\\!+\\!1)) + s^2 -s(2s\\!+\\!1) - \\frac{1}{3}+s  - \\frac{1}{2}(s^2\\!+\\! s) \\big] \\omega  \\\\\n& & + \\big[ \ns(s+1)\\frac{1}{6}(1-4s)+s^2(2s+1)+\\frac{1}{6}(2 s^3+3s^2 +s ) -s(s^2+s)\n\\big]\n\\\\\n& = &\\frac{1}{2}\\omega^3 - s \\omega^2 - \\frac{1}{2}\\omega +\\frac{s}{3} + \\frac{2}{3}s^3. \n\\EEAS\nTo get \\eq{tedious}, we then use $\\hat{q} \\ast \\hat{q}(\\omega) = a^2 \\prod_{i=1}^d \\frac{1}{s^2} b \\!\\ast\\! b(|\\omega_i|)$.}\n","start":65232,"end":67115}]},{"id":"S042","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S042"],"declared_external":true,"declaration_text":["原始 prerequisite：Standard Fourier convolution identities and factorization of tensor-product convolutions."],"source_only":false,"title":"周期 Fourier 卷积规则","source_units":[]},{"id":"S043","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S043"],"declared_external":true,"declaration_text":["原始 prerequisite：Standard Fourier convolution identities and factorization of tensor-product convolutions."],"source_only":false,"title":"张量积卷积因子分解","source_units":[]},{"id":"S044","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S044"],"declared_external":true,"declaration_text":["原始 prerequisite：Integration of elements of a closed finite-dimensional SOS cone preserves membership in that cone."],"source_only":false,"title":"闭 SOS 锥对非负积分封闭","source_units":[]},{"id":"S045","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S045"],"declared_external":true,"declaration_text":["原始 prerequisite：Elementary formulas for sums of consecutive integers and their squares."],"source_only":false,"title":"连续整数幂和公式","source_units":[]},{"id":"S046","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S046"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator：离散 Cauchy–Schwarz 不等式及平移不改变 ℓ2 范数。"],"source_only":false,"title":"归一化自卷积的上界","source_units":[]},{"id":"S047","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S047"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator：实函数在 t=0 的一阶 Taylor 展开。"],"source_only":false,"title":"固定指数的一阶渐近展开","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S033","S036":"S033","S037":"S033","S038":"S033","S039":"S033","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047"}}}}
{"record_id":"2101.03067v2","paper":{"title":"Exponential Decay of Sensitivity in Graph-Structured Nonlinear Programs","authors":["Shin, Sungho","Anitescu, Mihai","Zavala, Victor M."],"arxiv_id":"2101.03067","version":"v2","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/2101.03067v2","pdf_url":"https://arxiv.org/pdf/2101.03067v2","doi":"10.1137/21M1391079"},"context":{"raw_tex":"{\\it Notation:} The set of real numbers and the set of integers are denoted by $\\mathbb{R}$ and $\\mathbb{I}$, respectively. We define $\\mathbb{I}_{A}:=\\mathbb{I}\\cap A$, where $A$ is a set; $\\mathbb{I}_{>0}:=\\mathbb{I}\\cap(0,\\infty)$; $\\mathbb{I}_{\\geq 0}:=\\mathbb{I}\\cap[0,\\infty)$; $\\mathbb{R}_{>0}:=(0,\\infty)$; and $\\mathbb{R}_{\\geq 0}:=[0,\\infty)$. For $A\\subseteq X$, $f:X\\rightarrow Y$, and $x\\in X$, where $X$ and $Y$ are linear spaces, $A+x := \\{x'+x:x'\\in A\\}$ and $f(X):=\\{f(x)\\in Y:x\\in X\\}$. Vectors are treated as column vectors. We use the syntax: $[M_1;\\cdots;M_n]:=[M_1^\\top\\,\\cdots\\,M_n^\\top]^\\top$; $\\{M_i\\}_{i\\in U}:=[M_{i_1}; \\cdots; M_{i_{m}}]$; $\\{M_{i,j}\\}_{i\\in U,j\\in V}:=\\{\\{M_{i,j}^\\top\\}_{j\\in V}^\\top\\}_{i\\in U}$, where, $U=\\{i_1<\\cdots<i_{m}\\}$ and $V=\\{j_1<\\cdots<j_{n}\\}$ are strictly ordered sets. Furthermore, $v[i]$ is the $i$-th component of vector $v$; $M[i,j]$ is the $(i,j)$-th component of matrix $M$; $v[I]:=\\{v[i]\\}_{i\\in I}$; $M[I,J]:=\\{M[i,j]\\}_{i\\in I,j\\in J}$. For a function $\\phi:\\mathbb{R}^{n}\\rightarrow \\mathbb{R}$ and variable vectors $y\\in\\mathbb{R}^{p}$, $z\\in\\mathbb{R}^{q}$, $\\nabla^2_{yz}\\phi(x):=\\{\\frac{\\partial^2}{\\partial y[i]\\partial z[j]}\\phi(x)\\}_{i\\in\\mathbb{I}_{[1,p]},j\\in\\mathbb{I}_{[1,q]}}$. For a vector function $\\varphi:\\mathbb{R}^{n}\\rightarrow \\mathbb{R}^m$ and a variable vector $w\\in\\mathbb{R}^{s}$, $\\nabla_{w}\\varphi(x):=\\{\\frac{\\partial}{\\partial w[j]}\\varphi(x)[i]\\}_{i\\in\\mathbb{I}_{[1,m]},j\\in\\mathbb{I}_{[1,s]}}$. We use the shorthand notation $\\nabla^2\\phi(x):=\\nabla^2_{xx}\\phi(x)$, $\\nabla\\varphi(x):=\\nabla_x\\varphi(x)$. Vector 2-norms and induced 2-norms of matrices are denoted by $\\Vert\\cdot\\Vert$. For matrices $A$ and $B$, $A\\succ(\\succeq) B$ indicates that $A-B$ is positive (semi)-definite while $A>(\\geq) B$ denotes a componentwise inequality. The identity matrix is denoted as $\\bI$ and the zero matrix or vector are denoted as $\\bzero$. Specific notations will be introduced at the first appearance. {\\blue We denote the largest and smallest eigenvalues of a symmetric matrix and non-trivial singular values of general matrices by $\\olambda(\\cdot)$, $\\ulambda(\\cdot)$, $\\osigma(\\cdot)$, and $\\usigma(\\cdot)$}.\n\n{\\it Settings:} We study gsNLPs of the following form: \n\\begin{subequations}\\label{eqn:prob}\n  \\begin{align}\n    \\min_{\\{x_i\\in\\mathbb{R}^{r_i}\\}_{i\\in \\cV}}\\;& \\sum_{i\\in \\cV}f_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]})\\label{eqn:prob-obj}\\\\\n    \\st\\;& c^E_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]}) = 0,\\;i\\in \\cV,\\;(y^E_i)\\label{eqn:prob-eq}\\\\\n                                                  &c^I_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]}) \\geq 0,\\;i\\in \\cV,\\;(y^I_i)\\label{eqn:prob-ineq},\n  \\end{align}\n\\end{subequations}\nwhere $\\cG=(\\cV,\\cE)$ is an undirected graph, $\\cV$ is a strictly ordered node set, $\\cE\\subseteq \\{\\{i,j\\}\\subseteq \\cV: i\\neq j\\}$ is the edge set, and $\\cN_\\cG[i]:=\\{j\\in \\cV:\\{i,j\\}\\in \\cE\\}\\cup\\{i\\}$ denotes the closed neighborhood\\footnote{Note that closed neighborhood is different from the standard notion of neighborhood (also called open neighborhood), defined as $\\cN_\\cG(i):=\\{j\\in \\cV:\\{i,j\\}\\in \\cE\\}$. The closed neighborhood of a node contains itself whereas the open neighborhood does not.} of $i\\in \\cV$. For each node $i\\in \\cV$, $x_i\\in\\mathbb{R}^{r_i}$ is the primal variable; $p_i\\in\\mathbb{R}^{l_i}$ is the data; $f_i:\\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{r_j}\\times\\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{l_j}\\rightarrow \\mathbb{R}$ is the objective function; $c^E_i:\\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{r_j}\\times \\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{l_j}\\rightarrow \\mathbb{R}^{m^E_i}$ is the equality constraint function; $c^I_i:\\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{r_j}\\times \\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{l_j}\\rightarrow \\mathbb{R}^{m^I_i}$ is the inequality constraint function; and $y^E_i\\in\\mathbb{R}^{m^E_i}$ and $y^I_i\\in\\mathbb{R}^{m^I_i}$ are the dual variables associated with \\eqref{eqn:prob-eq} and \\eqref{eqn:prob-ineq}. The objective and constraint functions of node $i$ depend on the variables $\\{x_j\\}_{j\\in \\cN_\\cG[i]}$ and data $\\{p_j\\}_{j\\in \\cN_\\cG[i]}$ of the neighboring nodes. That is, node $i$ is coupled algebraically only to its neighbors, and the topology of such connectivity is dictated by $\\cG$.\n\n{\\blue\n  We can express \\eqref{eqn:prob} as a standard NLP of the following form:\n \\begin{subequations}\n   \\begin{align}\n     P(\\bp):\\;\\min_{\\bx}\\;&\\bff(\\bx;\\bp)\\\\\n     \\st\\;&\\bc^E(\\bx;\\bp)=\\bzero,\\; (\\by^E)\\\\\n                          &\\bc^I(\\bx;\\bp)\\geq \\bzero,\\;\\;(\\by^I).\n   \\end{align}\n \\end{subequations}\n Here, we compact the notation as: $\\bx:=\\{x_i\\}_{i\\in \\cV}$; $\\by^E:=\\{y^E_i\\}_{i\\in \\cV}$; $\\by^I:=\\{y^I_i\\}_{i\\in \\cV}$; $\\bp:=\\{p_i\\}_{i\\in \\cV}$; $\\br=\\sum_{i\\in \\cV} r_i$; $\\bm^E=\\sum_{i\\in \\cV} m^E_i$; $\\bm^I=\\sum_{i\\in \\cV} m^I_i$; $\\bl = \\sum_{i\\in \\cV}l_i$; $\\bff(\\bx;\\bp):=\\sum_{i\\in \\cV}f_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]})$; $\\bc^E(\\bx;\\bp):=\\{c^E_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]})\\}_{i\\in \\cV}$; and $\\bc^I(\\bx;\\bp):=\\{c^I_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]})\\}_{i\\in \\cV}$. We also define $c_i(\\cdot):=[c^E_i(\\cdot);c^I_i(\\cdot)]$; $y_i:=[y^E_i;y^I_i]$; $z_i:=[x_i;y_i]$; $m_i=m^E_i+m^I_i$; $n_i=r_i+m_i$; $\\by:=\\{y_i\\}_{i\\in \\cV}$; $\\bz:=\\{z_i\\}_{i\\in \\cV}$;  $\\bm=\\sum_{i\\in \\cV} m_i$; $\\bn=\\sum_{i\\in \\cV} n_i$; and $\\bc(\\bx;\\bp):=\\{c_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]})\\}_{i\\in \\cV}$.\n}\n\n\n\\begin{definition}[Graph Distance and Diameter]\\label{def:distance}\n  The distance $d_\\cG(i,j)$ between nodes $i,j\\in \\cV$ on graph $\\cG=(\\cV,\\cE)$ is the number of edges in the shortest path connecting them; {if a shortest path does not exist, the distance is $+\\infty$}. Furthermore, the diameter $D_\\cG$ of $\\cG$ is the longest distance between any pair of nodes in $\\cV$.\n\\end{definition}\n\\begin{proposition}\\label{prop:metric}\n  The distance $d_\\cG:\\cV\\times \\cV \\rightarrow \\mathbb{I}_{\\geq 0}{\\cup\\{\\infty\\}}$ is a metric on $\\cV$; that is, (a) $d_\\cG(i,j)\\geq 0$ for any $i,j\\in \\cV$; (b) $i=j$ if and only if $d_\\cG(i,j)=0$; (c) $d_\\cG(i,j)=d_\\cG(j,i)$ for any $i,j\\in \\cV$; (d) $d_\\cG(i,j)\\leq d_\\cG(i,k)+d_\\cG(k,j)$ for any $i,j,k\\in \\cV$.\n\\end{proposition}\nThe proof of this result is straightforward and thus omitted. {Note that the shortest path does not exist when the graph is not connected. } \n\n{A graph-structured matrix $X\\in\\mathbb{R}^{m\\times n}$ is indexed by a graph $\\cG=(\\cV,\\cE)$ and a pair of index set families $\\cI:=\\{I_{i}\\}_{i\\in \\cV}$ and $\\cJ:=\\{J_{i}\\}_{i\\in \\cV}$ that partition\\footnote{In this paper, we call a family $\\{X_1,\\cdots,X_k\\}$ of subsets of $X$ to be a partition if $\\bigcup_{k=1}^K X_k=X$ and $X_1,\\cdots,X_k$ are disjoint; here, {\\it we allow $X_k$ to be empty sets}. Note that this differs from the standard definition of a partition, where the nonemptiness of $X_k$ is enforced. This modification allows us to handle nodes with empty variables, constraints, or data, which is useful in many settings.} $\\mathbb{I}_{[1,m]}$ and $\\mathbb{I}_{[1,n]}$, respectively. We refer to $X_{[i][j]}$ as the $[i][j]$-block of matrix $X$, which can later be associated with the node pair $(i,j)$. It should be noted that the short-hand notation $X_{[i][j]}:=X[I_i,J_j]$ assumes that the proper graph and index set families $(\\cG,\\cI,\\cJ)$ are introduced or can be inferred from the context. We are now ready to introduce the concept of graph-induced {\\it bandwidth}.} \n\n\\begin{definition}[Graph-Induced Bandwidth]\\label{def:bandwidth}\n  Consider a matrix $X\\in\\mathbb{R}^{m\\times n}$, a graph $\\cG=(\\cV,\\cE)$, and index sets $\\cI=\\{I_i\\}_{i\\in \\cV}$, $\\cJ=\\{J_i\\}_{i\\in \\cV}$ that partition $\\mathbb{I}_{[1,m]}$ and $\\mathbb{I}_{[1,n]}$, respectively. Matrix $X$ is said to have bandwidth $B$ induced by $(\\cG,\\cI,\\cJ)$, if $B$ is the smallest nonnegative integer such that $X_{[i][j]}=\\bzero$ for any $i,j\\in \\cV$ with $d_\\cG(i,j)> B$, where $X_{[i][j]}:=X[I_i,J_j]$.\n\\end{definition}\n\n\nWe use $\\bz^\\star \\in\\mathbb{R}^{\\bn}$ to denote the primal-dual solution of $P(\\bp^\\star)$. We refer to $\\bp^\\star$ as the base data and $\\bz^\\star$ as the base solution. We denote the primal and dual components of $\\bz^\\star=\\{z^\\star_i\\}_{i\\in\\cV}=\\{[x^\\star_i;y^\\star_i]\\}_{i\\in\\cV}$ as $\\bx^\\star:=\\{x^\\star_i\\}_{i\\in\\cV}$ and $\\by^\\star=\\{y^\\star_i\\}_{i\\in\\cV}$, respectively. We say that $\\bz^\\star$ is a primal-dual solution if $\\bx^\\star$ satisfies the first-order optimality conditions with Lagrange multiplier $\\by^\\star$ (see \\cite{nocedal2006numerical}). We now make key assumptions that are necessary to establish our main results. \n\n\\begin{assumption}[Twice Continuous Differentiability of Functions]\\label{ass:c2}\n  The functions $\\bff:\\mathbb{R}^{\\br}\\times \\mathbb{R}^{\\bl}\\rightarrow \\mathbb{R}$ and $\\bc:\\mathbb{R}^{\\br}\\times \\mathbb{R}^{\\bl}\\rightarrow \\mathbb{R}^{\\bm}$ are twice continuously differentiable in the neighborhood of $[\\bx^\\star;\\bp^\\star]$.\n\\end{assumption}\n\n\\begin{assumption}[Regularity of Solution]\\label{ass:regularity}\n  The base solution $\\bz^\\star$ satisfies SSOSC and LICQ for $P(\\bp^\\star)$ with base data $\\bp^\\star$.\n\\end{assumption}\nWe recall that SSOSC requires positive definiteness of the reduced Hessian of the Lagrangian at $\\bz^\\star$. The reduced Hessian is the Hessian projected on the null space defined by equality constraints and active inequality constraints with nonzero duals. LICQ requires that the constraint Jacobian defined by equality and active inequality constraints are linearly independent at $\\bz^\\star$. These requirements are stated as: \n\\begin{align*}\n  ReH(\\nabla^2_{\\bx\\bx}\\cL(\\bz^\\star;\\bp^\\star),\\nabla_{\\bx}\\bc(\\bx^\\star;\\bp^\\star)[\\cA^0(\\bp^\\star),:]) &\\succ \\bzero\\tag{SSOSC}\\\\\n  \\usigma(\\nabla_{\\bx}\\bc(\\bx^\\star;\\bp^\\star)[\\cA^1(\\bp^\\star),:]) &> 0 \\tag{LICQ}.\n\\end{align*}\nHere, $\\cL(\\bz;\\bp) := \\bff(\\bx;\\bp)-\\by^\\top \\bc(\\bx;\\bp)$ is the Lagrange function, $ReH(H,A):=Z^\\top HZ$ is the reduced Hessian, where $Z$ is a null-space matrix of $A$, and\n\\begin{align}\\label{eqn:active-set-defs}\n  \\cA^0(\\bp^\\star)&:=\\cA^E\\cup\\{i\\in\\cA^I:\\bc(\\bx^\\star)[i]=0,\\;\\by^\\star[i]\\neq 0\\}\\\\\n  \\cA^1(\\bp^\\star)&:=\\cA^E\\cup\\{i\\in\\cA^I:\\bc(\\bx^\\star)[i]=0\\},\n\\end{align}\nwhere $\\cA^E$ and $\\cA^I$ are the set of equality and inequality constraint indices. Hereafter, we call a solution of gsNLP regular if the solution satisfies SSOSC and LICQ. Note that SSOSC and LICQ are standard assumptions used in NLP sensitivity theory \\cite{robinson1980strongly}. We now state the classical sensitivity result.\n\n\nUnder Lemma \\ref{lem:nlp-sensitivity}, the rate of change $D_{\\bq}\\bz^\\dag(\\bp)$ of the primal-dual solution of $P(\\bp)$ (for a given direction $\\bq$) can be quantified by using the solution of $QP(\\bp,\\bq)$. For given $\\bp$ and $\\bq$, the parameters in $QP(\\bp,\\bq)$ can be evaluated explicitly and thus its solution can be calculated. As such, Lemma \\ref{lem:nlp-sensitivity} provides a computational procedure to evaluate primal-dual solution sensitivity.\n\nWe now further analyze $QP(\\bp,\\bq)$; its first-order conditions are as follows:\n\\begin{subequations}\\label{eqn:kkt}\n  \\begin{align}\n    \\bQ(\\bp)\\bxi + \\bS(\\bp)\\bq + \\bA(\\bp)[\\cA^{1}(\\bp),:]^\\top \\bbeta &= \\bzero\\label{eqn:kkt-du}\\\\\n    \\bA(\\bp)[i,:]\\bxi + \\bB(\\bp)[i,:]\\bq &= 0,\\; i\\in\\cA^0(\\bp)\\label{eqn:kkt-pr-1}\\\\\n    \\bA(\\bp)[i,:]\\bxi + \\bB(\\bp)[i,:]\\bq &\\geq 0,\\;i\\in\\cA^1(\\bp)\\setminus\\cA^0(\\bp)\\label{eqn:kkt-pr-2}\\\\\n    \\bbeta[i] &\\geq 0,\\;i\\in\\cA^1(\\bp)\\setminus\\cA^0(\\bp)\\\\\n    \\left(\\bA(\\bp)[i,:]\\bxi + \\bB(\\bp)[i,:]\\bq \\right)\\bbeta[i]&=0,\\; i\\in\\cA^1(\\bp)\\setminus\\cA^0(\\bp).\\label{eqn:kkt-compl}\n  \\end{align}\n\\end{subequations}\nUnder SSOSC and LICQ for $P(\\bp^\\star)$ at $\\bz^\\star$, these conditions are necessary and sufficient for optimality. From the complementarity condition \\eqref{eqn:kkt-compl}, one can observe that there exists $\\cA^0(\\bp)\\subseteq\\cA'(\\bp,\\bq)\\subseteq \\cA^1(\\bp)$ such that:\n\\begin{subequations}\\label{eqn:compl-explicit}\n\\begin{align}\n  \\bA(\\bp)[i,:]\\bxi + \\bB(\\bp)[i,:]\\bq &= 0,\\; i\\in \\cA'(\\bp,\\bq)\\\\\n  \\bbeta[i] &= 0,\\; i\\in \\cA^1(\\bp)\\setminus \\cA'(\\bp,\\bq).\n\\end{align}\n\\end{subequations}\nare satisfied at $\\bxi^\\dag(\\bp,\\bq)$, $\\bbeta^\\dag(\\bp,\\bq)$. Thus, from \\eqref{eqn:kkt}, \\eqref{eqn:compl-explicit}, and $\\bbeta[i]=0$ for $i\\in\\mathbb{I}_{[1,\\bm]}\\setminus \\cA^1(\\bp)$ (from Lemma \\ref{lem:nlp-sensitivity}), we have:\n\\begin{align}\\label{eqn:linear-form-0}\n  \\begin{aligned}\n  \\begin{bmatrix}\n    \\bQ(\\bp)& \\bA(\\bp)[\\cA'(\\bp,\\bq),:]^\\top\\\\\n    \\bA(\\bp)[\\cA'(\\bp,\\bq),:]\n  \\end{bmatrix}\n  \\begin{bmatrix}\n    \\bxi\\\\\n    \\bbeta[\\cA'(\\bp,\\bq)]\n  \\end{bmatrix} &= -\n  \\begin{bmatrix}\n    \\bS(\\bp)\\\\  \n    \\bB(\\bp)[\\cA'(\\bp,\\bq),:]\n  \\end{bmatrix}\\bq\\\\\n  \\bbeta [\\mathbb{I}_{[1,\\bm]}\\setminus \\cA'(\\bp,\\bq)] &= \\bzero.\n\\end{aligned}\n\\end{align}\n% The linear equation \\eqref{eqn:linear-form-0} provides a relationship between $[\\bxi^\\dag(\\bp,\\bq);\\bbeta^\\dag(\\bp,\\bq)]$ and $\\bq$; however, it does not provide an explicit relationship  because $\\cA'(\\bp,\\bq)$ depends on $\\bq$.\nBy rearranging $[\\bxi^\\dag(\\bp,\\bq);\\bbeta^\\dag(\\bp,\\bq)]$ one can obtain $\\bzeta^\\dag(\\bp)=\\{[\\xi_i^\\dag(\\bp,\\bq);\\eta_i^\\dag(\\bp,\\bq)]\\}_{i\\in\\cV}$, where $\\bxi^\\dag(\\bp,\\bq)=\\{\\xi_i^\\dag(\\bp,\\bq)\\}_{i\\in\\cV}$, $\\bbeta^\\dag(\\bp,\\bq)=\\{\\eta_i^\\dag(\\bp,\\bq)\\}_{i\\in\\cV}$. To perform such rearrangement, we consider a permutation $\\phi:\\mathbb{I}_{[1,\\bn]}\\rightarrow\\mathbb{I}_{[1,\\bn]}$ that achieves $\\bzeta[\\phi(i)] = [\\bxi;\\bbeta][i]$. This permutation enables the following definition:\n\\begin{subequations}\\label{eqn:Bs}\n\\begin{align}\n  \\cB^0(\\bp)&:=\\phi(\\mathbb{I}_{[1,\\br]}\\cup(\\cA^0(\\bp)+\\br)),\\quad\n                      \\cB^1(\\bp):=\\phi(\\mathbb{I}_{[1,\\br]}\\cup(\\cA^1(\\bp)+\\br))\\\\\n  \\cB'(\\bp,\\bq)&:=\\phi(\\mathbb{I}_{[1,\\br]}\\cup(\\cA'(\\bp,\\bq)+\\br)).\n\\end{align}\n\\end{subequations}\nFinally, using the notation in \\eqref{eqn:Bs}, we obtain the following from \\eqref{eqn:linear-form-0}:\n\\begin{subequations}\\label{eqn:linear-form}\n\\begin{align}\n  D_{\\bq}\\bz^\\dag(\\bp)[\\cB'(\\bp,\\bq)] &= -\\left(\\bH(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)]\\right)^{-1}\\bR(\\bp)[\\cB'(\\bp,\\bq),:] \\bq,\\\\\n  D_{\\bq}\\bz^\\dag(\\bp)[\\mathbb{I}_{[1,\\bn]}\\setminus\\cB'(\\bp,\\bq)] &= \\bzero,\\label{eqn:linear-form-b}\n\\end{align}\n\\end{subequations}\nwhere:\n\\begin{align}\\label{eqn:HR}\n  \\bH(\\bp) &:= \\nabla_{\\bz\\bz}\\cL(\\bz^\\dag(\\bp);\\bp);\\quad \\bR(\\bp) := \\nabla_{\\bz\\bp}\\cL(\\bz^\\dag(\\bp);\\bp).\n\\end{align}\nHere, the nonsingularity of $\\bH(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)]$\n% is guaranteed by the fact that $\\bQ(\\bp)$ is positive definite on the null-space of $\\bA(\\bp)[\\cA^0(\\bp),:]$. This\nfollows from LICQ, SSOSC, and \\cite[Lemma 16.1]{nocedal2006numerical}.\n\n\n\\subsection{Nodal Sensitivity Result}\\label{sec:nodal}\n{We now establish the main nodal sensitivity result. First, we consider $\\bH(\\bp)$ and $\\bR(\\bp)$ as graph-structured matrices.} Specifically, we introduce the index set families $\\cI:=\\{I_i\\}_{i\\in\\cV}$ and $\\cK:=\\{K_i\\}_{i\\in\\cV}$with:\n\\begin{align}\\label{eqn:ind-1}\n  I_i &:= \\mathbb{I}_{\\sum_{j\\in \\cV,j<i}n_j+[1,n_i]},\\quad\n  K_i := \\mathbb{I}_{\\sum_{j\\in \\cV,j<i}l_j+[1,l_i]},\n\\end{align}\nand impose graph structures to $\\bH(\\bp)$ and $\\bR(\\bp)$ with $(\\cG,\\cI,\\cI)$ and $(\\cG,\\cI,\\cK)$, respectively. Note that $\\cI$ and $\\cK$ partition $\\mathbb{I}_{[1,\\bn]}$ and $\\mathbb{I}_{[1,\\bl]}$, respectively. For $i,j\\in\\cV$,\n\\begin{subequations}\\label{eqn:HRij}\n\\begin{align}\n  H_{ij}(\\bp)&:=\\nabla^2_{z_iz_j}\\cL(\\bz^\\dag(\\bp);\\bp)=(\\bH(\\bp))_{[i][j]}\\\\\n  R_{ij}(\\bp)&:=\\nabla^2_{z_ip_j}\\cL(\\bz^\\dag(\\bp);\\bp)=(\\bR(\\bp))_{[i][j]}.\n\\end{align}\n\\end{subequations}\n{We now can observe that $\\bH(\\bp)$ and $\\bR(\\bp)$ have bandwidth not greater than two. In other words, if $d_\\mathcal{G}(i,j)> 2$, $\\nabla_{z_i z_j}\\mathcal{L}(\\bz;\\bp)=0$ and $\\nabla_{z_i p_j}\\mathcal{L}(\\bz;\\bp)=0$. This is because in the graph-structured nonlinear programming setting in (1.1), we have assumed that the nodal objective and constraint functions are only dependent on the variables that are within the closed neighborhood of the associated node.}\n% The reason that the bandwidth may be greater than one is as follows. Suppose that $\\cV=\\{1,2,3\\}$ and $\\cE=\\{\\{1,2\\},\\{2,3\\}\\}$; then the constraints at node $2$ can be coupled with variables/data at node $1$ and $3$. This produces potential coupling between the variables that are within distance $2$ can have potential coupling.\n\nThe submatrices of $\\bH(\\bp)$ and $\\bR(\\bp)$ are also graph-structured. In particular, $\\bH(\\bp)[\\cB,\\cB]$ and $\\bR(\\bp)[\\cB,:]$ with $\\cB\\subseteq \\mathbb{I}_{[1,\\bn]}$ have bandwidth not greater than two induced by $(\\cG,\\cI^\\cB,\\cI^\\cB)$ and $(\\cG,\\cI^\\cB,\\cK)$, where $\\cI^\\cB:=\\{I_i\\cap\\cB\\}_{i\\in\\cV}$. We thus see that \\eqref{eqn:linear-form} involves the inverse of a {graph-induced banded} matrix. By combining Theorem \\ref{thm:inv} and Lemma \\ref{lem:nlp-sensitivity}, we obtain the following result.\n","units":[{"raw_tex":"{\\it Notation:} The set of real numbers and the set of integers are denoted by $\\mathbb{R}$ and $\\mathbb{I}$, respectively. We define $\\mathbb{I}_{A}:=\\mathbb{I}\\cap A$, where $A$ is a set; $\\mathbb{I}_{>0}:=\\mathbb{I}\\cap(0,\\infty)$; $\\mathbb{I}_{\\geq 0}:=\\mathbb{I}\\cap[0,\\infty)$; $\\mathbb{R}_{>0}:=(0,\\infty)$; and $\\mathbb{R}_{\\geq 0}:=[0,\\infty)$. For $A\\subseteq X$, $f:X\\rightarrow Y$, and $x\\in X$, where $X$ and $Y$ are linear spaces, $A+x := \\{x'+x:x'\\in A\\}$ and $f(X):=\\{f(x)\\in Y:x\\in X\\}$. Vectors are treated as column vectors. We use the syntax: $[M_1;\\cdots;M_n]:=[M_1^\\top\\,\\cdots\\,M_n^\\top]^\\top$; $\\{M_i\\}_{i\\in U}:=[M_{i_1}; \\cdots; M_{i_{m}}]$; $\\{M_{i,j}\\}_{i\\in U,j\\in V}:=\\{\\{M_{i,j}^\\top\\}_{j\\in V}^\\top\\}_{i\\in U}$, where, $U=\\{i_1<\\cdots<i_{m}\\}$ and $V=\\{j_1<\\cdots<j_{n}\\}$ are strictly ordered sets. Furthermore, $v[i]$ is the $i$-th component of vector $v$; $M[i,j]$ is the $(i,j)$-th component of matrix $M$; $v[I]:=\\{v[i]\\}_{i\\in I}$; $M[I,J]:=\\{M[i,j]\\}_{i\\in I,j\\in J}$. For a function $\\phi:\\mathbb{R}^{n}\\rightarrow \\mathbb{R}$ and variable vectors $y\\in\\mathbb{R}^{p}$, $z\\in\\mathbb{R}^{q}$, $\\nabla^2_{yz}\\phi(x):=\\{\\frac{\\partial^2}{\\partial y[i]\\partial z[j]}\\phi(x)\\}_{i\\in\\mathbb{I}_{[1,p]},j\\in\\mathbb{I}_{[1,q]}}$. For a vector function $\\varphi:\\mathbb{R}^{n}\\rightarrow \\mathbb{R}^m$ and a variable vector $w\\in\\mathbb{R}^{s}$, $\\nabla_{w}\\varphi(x):=\\{\\frac{\\partial}{\\partial w[j]}\\varphi(x)[i]\\}_{i\\in\\mathbb{I}_{[1,m]},j\\in\\mathbb{I}_{[1,s]}}$. We use the shorthand notation $\\nabla^2\\phi(x):=\\nabla^2_{xx}\\phi(x)$, $\\nabla\\varphi(x):=\\nabla_x\\varphi(x)$. Vector 2-norms and induced 2-norms of matrices are denoted by $\\Vert\\cdot\\Vert$. For matrices $A$ and $B$, $A\\succ(\\succeq) B$ indicates that $A-B$ is positive (semi)-definite while $A>(\\geq) B$ denotes a componentwise inequality. The identity matrix is denoted as $\\bI$ and the zero matrix or vector are denoted as $\\bzero$. Specific notations will be introduced at the first appearance. {\\blue We denote the largest and smallest eigenvalues of a symmetric matrix and non-trivial singular values of general matrices by $\\olambda(\\cdot)$, $\\ulambda(\\cdot)$, $\\osigma(\\cdot)$, and $\\usigma(\\cdot)$}.\n\n{\\it Settings:} We study gsNLPs of the following form: \n\\begin{subequations}\\label{eqn:prob}\n  \\begin{align}\n    \\min_{\\{x_i\\in\\mathbb{R}^{r_i}\\}_{i\\in \\cV}}\\;& \\sum_{i\\in \\cV}f_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]})\\label{eqn:prob-obj}\\\\\n    \\st\\;& c^E_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]}) = 0,\\;i\\in \\cV,\\;(y^E_i)\\label{eqn:prob-eq}\\\\\n                                                  &c^I_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]}) \\geq 0,\\;i\\in \\cV,\\;(y^I_i)\\label{eqn:prob-ineq},\n  \\end{align}\n\\end{subequations}\nwhere $\\cG=(\\cV,\\cE)$ is an undirected graph, $\\cV$ is a strictly ordered node set, $\\cE\\subseteq \\{\\{i,j\\}\\subseteq \\cV: i\\neq j\\}$ is the edge set, and $\\cN_\\cG[i]:=\\{j\\in \\cV:\\{i,j\\}\\in \\cE\\}\\cup\\{i\\}$ denotes the closed neighborhood\\footnote{Note that closed neighborhood is different from the standard notion of neighborhood (also called open neighborhood), defined as $\\cN_\\cG(i):=\\{j\\in \\cV:\\{i,j\\}\\in \\cE\\}$. The closed neighborhood of a node contains itself whereas the open neighborhood does not.} of $i\\in \\cV$. For each node $i\\in \\cV$, $x_i\\in\\mathbb{R}^{r_i}$ is the primal variable; $p_i\\in\\mathbb{R}^{l_i}$ is the data; $f_i:\\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{r_j}\\times\\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{l_j}\\rightarrow \\mathbb{R}$ is the objective function; $c^E_i:\\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{r_j}\\times \\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{l_j}\\rightarrow \\mathbb{R}^{m^E_i}$ is the equality constraint function; $c^I_i:\\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{r_j}\\times \\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{l_j}\\rightarrow \\mathbb{R}^{m^I_i}$ is the inequality constraint function; and $y^E_i\\in\\mathbb{R}^{m^E_i}$ and $y^I_i\\in\\mathbb{R}^{m^I_i}$ are the dual variables associated with \\eqref{eqn:prob-eq} and \\eqref{eqn:prob-ineq}. The objective and constraint functions of node $i$ depend on the variables $\\{x_j\\}_{j\\in \\cN_\\cG[i]}$ and data $\\{p_j\\}_{j\\in \\cN_\\cG[i]}$ of the neighboring nodes. That is, node $i$ is coupled algebraically only to its neighbors, and the topology of such connectivity is dictated by $\\cG$.\n\n{\\blue\n  We can express \\eqref{eqn:prob} as a standard NLP of the following form:\n \\begin{subequations}\n   \\begin{align}\n     P(\\bp):\\;\\min_{\\bx}\\;&\\bff(\\bx;\\bp)\\\\\n     \\st\\;&\\bc^E(\\bx;\\bp)=\\bzero,\\; (\\by^E)\\\\\n                          &\\bc^I(\\bx;\\bp)\\geq \\bzero,\\;\\;(\\by^I).\n   \\end{align}\n \\end{subequations}\n Here, we compact the notation as: $\\bx:=\\{x_i\\}_{i\\in \\cV}$; $\\by^E:=\\{y^E_i\\}_{i\\in \\cV}$; $\\by^I:=\\{y^I_i\\}_{i\\in \\cV}$; $\\bp:=\\{p_i\\}_{i\\in \\cV}$; $\\br=\\sum_{i\\in \\cV} r_i$; $\\bm^E=\\sum_{i\\in \\cV} m^E_i$; $\\bm^I=\\sum_{i\\in \\cV} m^I_i$; $\\bl = \\sum_{i\\in \\cV}l_i$; $\\bff(\\bx;\\bp):=\\sum_{i\\in \\cV}f_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]})$; $\\bc^E(\\bx;\\bp):=\\{c^E_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]})\\}_{i\\in \\cV}$; and $\\bc^I(\\bx;\\bp):=\\{c^I_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]})\\}_{i\\in \\cV}$. We also define $c_i(\\cdot):=[c^E_i(\\cdot);c^I_i(\\cdot)]$; $y_i:=[y^E_i;y^I_i]$; $z_i:=[x_i;y_i]$; $m_i=m^E_i+m^I_i$; $n_i=r_i+m_i$; $\\by:=\\{y_i\\}_{i\\in \\cV}$; $\\bz:=\\{z_i\\}_{i\\in \\cV}$;  $\\bm=\\sum_{i\\in \\cV} m_i$; $\\bn=\\sum_{i\\in \\cV} n_i$; and $\\bc(\\bx;\\bp):=\\{c_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]})\\}_{i\\in \\cV}$.\n}\n","start":13210,"end":18723},{"raw_tex":"\\begin{definition}[Graph Distance and Diameter]\\label{def:distance}\n  The distance $d_\\cG(i,j)$ between nodes $i,j\\in \\cV$ on graph $\\cG=(\\cV,\\cE)$ is the number of edges in the shortest path connecting them; {if a shortest path does not exist, the distance is $+\\infty$}. Furthermore, the diameter $D_\\cG$ of $\\cG$ is the longest distance between any pair of nodes in $\\cV$.\n\\end{definition}\n\\begin{proposition}\\label{prop:metric}\n  The distance $d_\\cG:\\cV\\times \\cV \\rightarrow \\mathbb{I}_{\\geq 0}{\\cup\\{\\infty\\}}$ is a metric on $\\cV$; that is, (a) $d_\\cG(i,j)\\geq 0$ for any $i,j\\in \\cV$; (b) $i=j$ if and only if $d_\\cG(i,j)=0$; (c) $d_\\cG(i,j)=d_\\cG(j,i)$ for any $i,j\\in \\cV$; (d) $d_\\cG(i,j)\\leq d_\\cG(i,k)+d_\\cG(k,j)$ for any $i,j,k\\in \\cV$.\n\\end{proposition}\nThe proof of this result is straightforward and thus omitted. {Note that the shortest path does not exist when the graph is not connected. } \n\n{A graph-structured matrix $X\\in\\mathbb{R}^{m\\times n}$ is indexed by a graph $\\cG=(\\cV,\\cE)$ and a pair of index set families $\\cI:=\\{I_{i}\\}_{i\\in \\cV}$ and $\\cJ:=\\{J_{i}\\}_{i\\in \\cV}$ that partition\\footnote{In this paper, we call a family $\\{X_1,\\cdots,X_k\\}$ of subsets of $X$ to be a partition if $\\bigcup_{k=1}^K X_k=X$ and $X_1,\\cdots,X_k$ are disjoint; here, {\\it we allow $X_k$ to be empty sets}. Note that this differs from the standard definition of a partition, where the nonemptiness of $X_k$ is enforced. This modification allows us to handle nodes with empty variables, constraints, or data, which is useful in many settings.} $\\mathbb{I}_{[1,m]}$ and $\\mathbb{I}_{[1,n]}$, respectively. We refer to $X_{[i][j]}$ as the $[i][j]$-block of matrix $X$, which can later be associated with the node pair $(i,j)$. It should be noted that the short-hand notation $X_{[i][j]}:=X[I_i,J_j]$ assumes that the proper graph and index set families $(\\cG,\\cI,\\cJ)$ are introduced or can be inferred from the context. We are now ready to introduce the concept of graph-induced {\\it bandwidth}.} \n\n\\begin{definition}[Graph-Induced Bandwidth]\\label{def:bandwidth}\n  Consider a matrix $X\\in\\mathbb{R}^{m\\times n}$, a graph $\\cG=(\\cV,\\cE)$, and index sets $\\cI=\\{I_i\\}_{i\\in \\cV}$, $\\cJ=\\{J_i\\}_{i\\in \\cV}$ that partition $\\mathbb{I}_{[1,m]}$ and $\\mathbb{I}_{[1,n]}$, respectively. Matrix $X$ is said to have bandwidth $B$ induced by $(\\cG,\\cI,\\cJ)$, if $B$ is the smallest nonnegative integer such that $X_{[i][j]}=\\bzero$ for any $i,j\\in \\cV$ with $d_\\cG(i,j)> B$, where $X_{[i][j]}:=X[I_i,J_j]$.\n\\end{definition}\n","start":22924,"end":25449},{"raw_tex":"We use $\\bz^\\star \\in\\mathbb{R}^{\\bn}$ to denote the primal-dual solution of $P(\\bp^\\star)$. We refer to $\\bp^\\star$ as the base data and $\\bz^\\star$ as the base solution. We denote the primal and dual components of $\\bz^\\star=\\{z^\\star_i\\}_{i\\in\\cV}=\\{[x^\\star_i;y^\\star_i]\\}_{i\\in\\cV}$ as $\\bx^\\star:=\\{x^\\star_i\\}_{i\\in\\cV}$ and $\\by^\\star=\\{y^\\star_i\\}_{i\\in\\cV}$, respectively. We say that $\\bz^\\star$ is a primal-dual solution if $\\bx^\\star$ satisfies the first-order optimality conditions with Lagrange multiplier $\\by^\\star$ (see \\cite{nocedal2006numerical}). We now make key assumptions that are necessary to establish our main results. \n\n\\begin{assumption}[Twice Continuous Differentiability of Functions]\\label{ass:c2}\n  The functions $\\bff:\\mathbb{R}^{\\br}\\times \\mathbb{R}^{\\bl}\\rightarrow \\mathbb{R}$ and $\\bc:\\mathbb{R}^{\\br}\\times \\mathbb{R}^{\\bl}\\rightarrow \\mathbb{R}^{\\bm}$ are twice continuously differentiable in the neighborhood of $[\\bx^\\star;\\bp^\\star]$.\n\\end{assumption}\n\n\\begin{assumption}[Regularity of Solution]\\label{ass:regularity}\n  The base solution $\\bz^\\star$ satisfies SSOSC and LICQ for $P(\\bp^\\star)$ with base data $\\bp^\\star$.\n\\end{assumption}\nWe recall that SSOSC requires positive definiteness of the reduced Hessian of the Lagrangian at $\\bz^\\star$. The reduced Hessian is the Hessian projected on the null space defined by equality constraints and active inequality constraints with nonzero duals. LICQ requires that the constraint Jacobian defined by equality and active inequality constraints are linearly independent at $\\bz^\\star$. These requirements are stated as: \n\\begin{align*}\n  ReH(\\nabla^2_{\\bx\\bx}\\cL(\\bz^\\star;\\bp^\\star),\\nabla_{\\bx}\\bc(\\bx^\\star;\\bp^\\star)[\\cA^0(\\bp^\\star),:]) &\\succ \\bzero\\tag{SSOSC}\\\\\n  \\usigma(\\nabla_{\\bx}\\bc(\\bx^\\star;\\bp^\\star)[\\cA^1(\\bp^\\star),:]) &> 0 \\tag{LICQ}.\n\\end{align*}\nHere, $\\cL(\\bz;\\bp) := \\bff(\\bx;\\bp)-\\by^\\top \\bc(\\bx;\\bp)$ is the Lagrange function, $ReH(H,A):=Z^\\top HZ$ is the reduced Hessian, where $Z$ is a null-space matrix of $A$, and\n\\begin{align}\\label{eqn:active-set-defs}\n  \\cA^0(\\bp^\\star)&:=\\cA^E\\cup\\{i\\in\\cA^I:\\bc(\\bx^\\star)[i]=0,\\;\\by^\\star[i]\\neq 0\\}\\\\\n  \\cA^1(\\bp^\\star)&:=\\cA^E\\cup\\{i\\in\\cA^I:\\bc(\\bx^\\star)[i]=0\\},\n\\end{align}\nwhere $\\cA^E$ and $\\cA^I$ are the set of equality and inequality constraint indices. Hereafter, we call a solution of gsNLP regular if the solution satisfies SSOSC and LICQ. Note that SSOSC and LICQ are standard assumptions used in NLP sensitivity theory \\cite{robinson1980strongly}. We now state the classical sensitivity result.\n","start":36376,"end":38949},{"raw_tex":"Under Lemma \\ref{lem:nlp-sensitivity}, the rate of change $D_{\\bq}\\bz^\\dag(\\bp)$ of the primal-dual solution of $P(\\bp)$ (for a given direction $\\bq$) can be quantified by using the solution of $QP(\\bp,\\bq)$. For given $\\bp$ and $\\bq$, the parameters in $QP(\\bp,\\bq)$ can be evaluated explicitly and thus its solution can be calculated. As such, Lemma \\ref{lem:nlp-sensitivity} provides a computational procedure to evaluate primal-dual solution sensitivity.\n\nWe now further analyze $QP(\\bp,\\bq)$; its first-order conditions are as follows:\n\\begin{subequations}\\label{eqn:kkt}\n  \\begin{align}\n    \\bQ(\\bp)\\bxi + \\bS(\\bp)\\bq + \\bA(\\bp)[\\cA^{1}(\\bp),:]^\\top \\bbeta &= \\bzero\\label{eqn:kkt-du}\\\\\n    \\bA(\\bp)[i,:]\\bxi + \\bB(\\bp)[i,:]\\bq &= 0,\\; i\\in\\cA^0(\\bp)\\label{eqn:kkt-pr-1}\\\\\n    \\bA(\\bp)[i,:]\\bxi + \\bB(\\bp)[i,:]\\bq &\\geq 0,\\;i\\in\\cA^1(\\bp)\\setminus\\cA^0(\\bp)\\label{eqn:kkt-pr-2}\\\\\n    \\bbeta[i] &\\geq 0,\\;i\\in\\cA^1(\\bp)\\setminus\\cA^0(\\bp)\\\\\n    \\left(\\bA(\\bp)[i,:]\\bxi + \\bB(\\bp)[i,:]\\bq \\right)\\bbeta[i]&=0,\\; i\\in\\cA^1(\\bp)\\setminus\\cA^0(\\bp).\\label{eqn:kkt-compl}\n  \\end{align}\n\\end{subequations}\nUnder SSOSC and LICQ for $P(\\bp^\\star)$ at $\\bz^\\star$, these conditions are necessary and sufficient for optimality. From the complementarity condition \\eqref{eqn:kkt-compl}, one can observe that there exists $\\cA^0(\\bp)\\subseteq\\cA'(\\bp,\\bq)\\subseteq \\cA^1(\\bp)$ such that:\n\\begin{subequations}\\label{eqn:compl-explicit}\n\\begin{align}\n  \\bA(\\bp)[i,:]\\bxi + \\bB(\\bp)[i,:]\\bq &= 0,\\; i\\in \\cA'(\\bp,\\bq)\\\\\n  \\bbeta[i] &= 0,\\; i\\in \\cA^1(\\bp)\\setminus \\cA'(\\bp,\\bq).\n\\end{align}\n\\end{subequations}\nare satisfied at $\\bxi^\\dag(\\bp,\\bq)$, $\\bbeta^\\dag(\\bp,\\bq)$. Thus, from \\eqref{eqn:kkt}, \\eqref{eqn:compl-explicit}, and $\\bbeta[i]=0$ for $i\\in\\mathbb{I}_{[1,\\bm]}\\setminus \\cA^1(\\bp)$ (from Lemma \\ref{lem:nlp-sensitivity}), we have:\n\\begin{align}\\label{eqn:linear-form-0}\n  \\begin{aligned}\n  \\begin{bmatrix}\n    \\bQ(\\bp)& \\bA(\\bp)[\\cA'(\\bp,\\bq),:]^\\top\\\\\n    \\bA(\\bp)[\\cA'(\\bp,\\bq),:]\n  \\end{bmatrix}\n  \\begin{bmatrix}\n    \\bxi\\\\\n    \\bbeta[\\cA'(\\bp,\\bq)]\n  \\end{bmatrix} &= -\n  \\begin{bmatrix}\n    \\bS(\\bp)\\\\  \n    \\bB(\\bp)[\\cA'(\\bp,\\bq),:]\n  \\end{bmatrix}\\bq\\\\\n  \\bbeta [\\mathbb{I}_{[1,\\bm]}\\setminus \\cA'(\\bp,\\bq)] &= \\bzero.\n\\end{aligned}\n\\end{align}\n% The linear equation \\eqref{eqn:linear-form-0} provides a relationship between $[\\bxi^\\dag(\\bp,\\bq);\\bbeta^\\dag(\\bp,\\bq)]$ and $\\bq$; however, it does not provide an explicit relationship  because $\\cA'(\\bp,\\bq)$ depends on $\\bq$.\nBy rearranging $[\\bxi^\\dag(\\bp,\\bq);\\bbeta^\\dag(\\bp,\\bq)]$ one can obtain $\\bzeta^\\dag(\\bp)=\\{[\\xi_i^\\dag(\\bp,\\bq);\\eta_i^\\dag(\\bp,\\bq)]\\}_{i\\in\\cV}$, where $\\bxi^\\dag(\\bp,\\bq)=\\{\\xi_i^\\dag(\\bp,\\bq)\\}_{i\\in\\cV}$, $\\bbeta^\\dag(\\bp,\\bq)=\\{\\eta_i^\\dag(\\bp,\\bq)\\}_{i\\in\\cV}$. To perform such rearrangement, we consider a permutation $\\phi:\\mathbb{I}_{[1,\\bn]}\\rightarrow\\mathbb{I}_{[1,\\bn]}$ that achieves $\\bzeta[\\phi(i)] = [\\bxi;\\bbeta][i]$. This permutation enables the following definition:\n\\begin{subequations}\\label{eqn:Bs}\n\\begin{align}\n  \\cB^0(\\bp)&:=\\phi(\\mathbb{I}_{[1,\\br]}\\cup(\\cA^0(\\bp)+\\br)),\\quad\n                      \\cB^1(\\bp):=\\phi(\\mathbb{I}_{[1,\\br]}\\cup(\\cA^1(\\bp)+\\br))\\\\\n  \\cB'(\\bp,\\bq)&:=\\phi(\\mathbb{I}_{[1,\\br]}\\cup(\\cA'(\\bp,\\bq)+\\br)).\n\\end{align}\n\\end{subequations}\nFinally, using the notation in \\eqref{eqn:Bs}, we obtain the following from \\eqref{eqn:linear-form-0}:\n\\begin{subequations}\\label{eqn:linear-form}\n\\begin{align}\n  D_{\\bq}\\bz^\\dag(\\bp)[\\cB'(\\bp,\\bq)] &= -\\left(\\bH(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)]\\right)^{-1}\\bR(\\bp)[\\cB'(\\bp,\\bq),:] \\bq,\\\\\n  D_{\\bq}\\bz^\\dag(\\bp)[\\mathbb{I}_{[1,\\bn]}\\setminus\\cB'(\\bp,\\bq)] &= \\bzero,\\label{eqn:linear-form-b}\n\\end{align}\n\\end{subequations}\nwhere:\n\\begin{align}\\label{eqn:HR}\n  \\bH(\\bp) &:= \\nabla_{\\bz\\bz}\\cL(\\bz^\\dag(\\bp);\\bp);\\quad \\bR(\\bp) := \\nabla_{\\bz\\bp}\\cL(\\bz^\\dag(\\bp);\\bp).\n\\end{align}\nHere, the nonsingularity of $\\bH(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)]$\n% is guaranteed by the fact that $\\bQ(\\bp)$ is positive definite on the null-space of $\\bA(\\bp)[\\cA^0(\\bp),:]$. This\nfollows from LICQ, SSOSC, and \\cite[Lemma 16.1]{nocedal2006numerical}.\n","start":42864,"end":46985},{"raw_tex":"\\subsection{Nodal Sensitivity Result}\\label{sec:nodal}\n{We now establish the main nodal sensitivity result. First, we consider $\\bH(\\bp)$ and $\\bR(\\bp)$ as graph-structured matrices.} Specifically, we introduce the index set families $\\cI:=\\{I_i\\}_{i\\in\\cV}$ and $\\cK:=\\{K_i\\}_{i\\in\\cV}$with:\n\\begin{align}\\label{eqn:ind-1}\n  I_i &:= \\mathbb{I}_{\\sum_{j\\in \\cV,j<i}n_j+[1,n_i]},\\quad\n  K_i := \\mathbb{I}_{\\sum_{j\\in \\cV,j<i}l_j+[1,l_i]},\n\\end{align}\nand impose graph structures to $\\bH(\\bp)$ and $\\bR(\\bp)$ with $(\\cG,\\cI,\\cI)$ and $(\\cG,\\cI,\\cK)$, respectively. Note that $\\cI$ and $\\cK$ partition $\\mathbb{I}_{[1,\\bn]}$ and $\\mathbb{I}_{[1,\\bl]}$, respectively. For $i,j\\in\\cV$,\n\\begin{subequations}\\label{eqn:HRij}\n\\begin{align}\n  H_{ij}(\\bp)&:=\\nabla^2_{z_iz_j}\\cL(\\bz^\\dag(\\bp);\\bp)=(\\bH(\\bp))_{[i][j]}\\\\\n  R_{ij}(\\bp)&:=\\nabla^2_{z_ip_j}\\cL(\\bz^\\dag(\\bp);\\bp)=(\\bR(\\bp))_{[i][j]}.\n\\end{align}\n\\end{subequations}\n{We now can observe that $\\bH(\\bp)$ and $\\bR(\\bp)$ have bandwidth not greater than two. In other words, if $d_\\mathcal{G}(i,j)> 2$, $\\nabla_{z_i z_j}\\mathcal{L}(\\bz;\\bp)=0$ and $\\nabla_{z_i p_j}\\mathcal{L}(\\bz;\\bp)=0$. This is because in the graph-structured nonlinear programming setting in (1.1), we have assumed that the nodal objective and constraint functions are only dependent on the variables that are within the closed neighborhood of the associated node.}\n% The reason that the bandwidth may be greater than one is as follows. Suppose that $\\cV=\\{1,2,3\\}$ and $\\cE=\\{\\{1,2\\},\\{2,3\\}\\}$; then the constraints at node $2$ can be coupled with variables/data at node $1$ and $3$. This produces potential coupling between the variables that are within distance $2$ can have potential coupling.\n\nThe submatrices of $\\bH(\\bp)$ and $\\bR(\\bp)$ are also graph-structured. In particular, $\\bH(\\bp)[\\cB,\\cB]$ and $\\bR(\\bp)[\\cB,:]$ with $\\cB\\subseteq \\mathbb{I}_{[1,\\bn]}$ have bandwidth not greater than two induced by $(\\cG,\\cI^\\cB,\\cI^\\cB)$ and $(\\cG,\\cI^\\cB,\\cK)$, where $\\cI^\\cB:=\\{I_i\\cap\\cB\\}_{i\\in\\cV}$. We thus see that \\eqref{eqn:linear-form} involves the inverse of a {graph-induced banded} matrix. By combining Theorem \\ref{thm:inv} and Lemma \\ref{lem:nlp-sensitivity}, we obtain the following result.\n","start":46986,"end":49215}]},"statement":{"raw_tex":"\\begin{theorem}[Exponential Decay of Sensitivity (EDS)]\\label{thm:main}\n  Under Assumptions \\ref{ass:c2}, \\ref{ass:regularity}, and for given $\\epsilon>0$, $\\osigma_{\\bH}\\geq\\oosigma_{\\bH}(\\bp^\\star)$, $\\osigma_{\\bR}\\geq\\oosigma_{\\bR}(\\bp^\\star)$, and $0<\\usigma_{\\bH}\\leq\\uusigma_{\\bH}(\\bp^\\star)$, where\n  \\begin{subequations}\\label{eqn:sigmas}\n    \\begin{align}\n      \\oosigma_{\\bH}(\\bp)&:=\\max\\{\\osigma(\\bH(\\bp)[\\cB,\\cB]):\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)\\}\\\\\n      \\oosigma_{\\bR}(\\bp)&:=\\max\\{\\osigma(\\bR(\\bp)[\\cB,:]):\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)\\}\\\\\n      \\uusigma_{\\bH}(\\bp)&:=\\min\\{\\usigma(\\bH(\\bp)[\\cB,\\cB]):\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)\\},\n    \\end{align}\n  \\end{subequations}\n  there exists a neighborhood $\\mathbb{P}_{\\epsilon}$ of $\\bp^\\star$ such that the following holds for any $\\bp,\\bp'\\in\\mathbb{P}_{\\epsilon}$:\n  \\begin{align}\\label{eqn:main}\n      &\\Vert z^\\dag_i(\\bp)-z^\\dag_i(\\bp')\\Vert \n      \\leq \\sum_{j\\in \\cV} \\Upsilon\\rho^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+} \\Vert p_j-p'_j\\Vert,\\quad i\\in \\cV,\n  \\end{align}\nwith $\\Upsilon :=\\dfrac{\\osigma_{\\bH}\\osigma_{\\bR}}{{\\usigma}_{\\bH}^2}+\\epsilon$ and $\\rho:=\\dfrac{\\osigma_{\\bH}^2-{\\usigma}_{\\bH}^2}{\\osigma_{\\bH}^2+{\\usigma}_{\\bH}^2}+\\epsilon$.\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}[Exponential Decay of Sensitivity (EDS)]\\label{thm:main}\n  Under Assumptions \\ref{ass:c2}, \\ref{ass:regularity}, and for given $\\epsilon>0$, $\\osigma_{\\bH}\\geq\\oosigma_{\\bH}(\\bp^\\star)$, $\\osigma_{\\bR}\\geq\\oosigma_{\\bR}(\\bp^\\star)$, and $0<\\usigma_{\\bH}\\leq\\uusigma_{\\bH}(\\bp^\\star)$, where\n  \\begin{subequations}\\label{eqn:sigmas}\n    \\begin{align}\n      \\oosigma_{\\bH}(\\bp)&:=\\max\\{\\osigma(\\bH(\\bp)[\\cB,\\cB]):\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)\\}\\\\\n      \\oosigma_{\\bR}(\\bp)&:=\\max\\{\\osigma(\\bR(\\bp)[\\cB,:]):\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)\\}\\\\\n      \\uusigma_{\\bH}(\\bp)&:=\\min\\{\\usigma(\\bH(\\bp)[\\cB,\\cB]):\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)\\},\n    \\end{align}\n  \\end{subequations}\n  there exists a neighborhood $\\mathbb{P}_{\\epsilon}$ of $\\bp^\\star$ such that the following holds for any $\\bp,\\bp'\\in\\mathbb{P}_{\\epsilon}$:\n  \\begin{align}\\label{eqn:main}\n      &\\Vert z^\\dag_i(\\bp)-z^\\dag_i(\\bp')\\Vert \n      \\leq \\sum_{j\\in \\cV} \\Upsilon\\rho^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+} \\Vert p_j-p'_j\\Vert,\\quad i\\in \\cV,\n  \\end{align}\nwith $\\Upsilon :=\\dfrac{\\osigma_{\\bH}\\osigma_{\\bR}}{{\\usigma}_{\\bH}^2}+\\epsilon$ and $\\rho:=\\dfrac{\\osigma_{\\bH}^2-{\\usigma}_{\\bH}^2}{\\osigma_{\\bH}^2+{\\usigma}_{\\bH}^2}+\\epsilon$.\n\\end{theorem}","start":52537,"end":53850}]},"proof":{"raw_tex":"\\begin{lemma}\\label{lem:bandwidth}\n  Consider $X\\in\\mathbb{R}^{m\\times n}$ with bandwidth not greater than $B_X$ induced by $(\\cG,\\cI,\\cJ)$; we have that:\n  (a) $X^\\top$ has bandwidth not greater than $B_X$ induced by $(\\cG,\\cJ,\\cI)$;\n  (b) if $Y\\in\\mathbb{R}^{m\\times n}$ has bandwidth not greater than $B_Y$ induced by $(\\cG,\\cI,\\cJ)$, then $X+Y$ has bandwidth not greater than $\\max(B_X,B_Y)$ induced by $(\\cG,\\cI,\\cJ)$;\n  (c) if $W\\in\\mathbb{R}^{n\\times k}$ has bandwidth not greater than $B_W$ induced by $(\\cG,\\cJ,\\cK)$, then $XW$ has bandwidth not greater than $B_X+B_W$ induced by $(\\cG,\\cI,\\cK)$.\n\\end{lemma}\n\n\\begin{proof}[Proof of (a)] We have that $(X^\\top)_{[i][j]}=(X^\\top){[J_i,I_j]}=(X[I_j,J_i])^\\top=(X_{[j][i]})^\\top$. From the assumption that $X$ has bandwidth not greater than $B_X$ and Proposition \\ref{prop:metric}(c), $(X^\\top)_{[i][j]}=\\bzero$ if $d_\\cG(i,j)>B_X$; therefore, $X$ has bandwidth not greater than $B_X$, and induced by $(\\cG,\\cJ,\\cI)$.\n\\end{proof}\n\n\\begin{proof}[Proof of (b)] We have that $X_{[i][j]}=\\bzero$ and $Y_{[i][j]}=\\bzero$ if $d_\\cG(i,j)>\\max(B_X,B_Y)$, which yields $(X+Y)_{[i][j]}=\\bzero$ if $d_\\cG(i,j)>\\max(B_X,B_Y)$. Thus, $X+Y$ has bandwidth not greater than $\\max(B_X,B_Y)$, and induced by $(\\cG,\\cI,\\cJ)$.\n\\end{proof}\n\n\\begin{proof}[Proof of (c)] If $d_\\cG(i,j)>B_X+B_W$, from Proposition \\ref{prop:metric}(d) we have that for any $k\\in \\cV$, $d_\\cG(i,k)>B_X$ or $d_\\cG(j,k)>B_W$. Thus, if $d_\\cG(i,j)>B_X+B_W$,  we have $(XW)_{[i][j]}=\\sum_{k\\in \\cV} X_{[i][k]}W_{[k][j]}= \\bzero$ (where the first equality comes from the block matrix multiplication law and the second equality comes from observing that $d_\\cG(i,k)>B_X$ or $d_\\cG(j,k)>B_W$). Therefore, $XW$ has bandwidth not greater than $B_X+B_W$, and induced by $(\\cG,\\cI,\\cK)$.  \n\\end{proof}\n\n\\begin{theorem}\\label{thm:inv}\n  Consider a nonsingular matrix $X\\in\\mathbb{R}^{n\\times n}$ with bandwidth not greater than $B_X\\geq 1$ induced by $(\\cG,\\cK,\\cP)$, $Y\\in\\mathbb{R}^{n\\times m}$ with bandwidth not greater than $B_Y$ induced by $(\\cG, \\cK,\\cJ)$, and $W\\in\\mathbb{R}^{\\ell\\times n}$ with bandwidth not greater than $B_W$ induced by $(\\cG, \\cI,\\cP)$; for constants $\\osigma_X\\geq \\osigma(X)$, $\\osigma_Y\\geq \\osigma(Y)$, $\\osigma_W\\geq \\osigma(W)$, and $0<\\usigma_X\\leq \\usigma(X)$, the following holds:\n  \\begin{align}\\label{eqn:inv}\n    \\Vert (WX^{-1}Y)_{[i][j]}\\Vert \\leq\n    \\frac{\\osigma_X\\osigma_Y\\osigma_W}{\\usigma_X^2}\\left(\\frac{\\osigma_X^2-\\usigma_X^2}{\\osigma_X^2+\\usigma_X^2}\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)-B_X-B_Y-B_W}{2B_X}\\right\\rceil_+} ,\\quad i,j\\in \\cV,\n  \\end{align}\n  where $(WX^{-1}Y)_{[i][j]}:=(WX^{-1}Y)[I_i,J_j]$ and $\\lceil\\cdot\\rceil_+$ is the smallest non-negative integer that is greater than or equal to the argument.\n\\end{theorem}\n\n\\begin{proof}\n  By definition of singular values and the assumptions on $\\osigma_X$, $\\osigma_Y$ , $\\osigma_W$, and $\\usigma_X$, we have $\\usigma_X^2 \\bI \\preceq\\usigma(X)^2 \\bI \\preceq X X^\\top\\preceq  \\osigma(X)^2\\bI\\preceq\\osigma_X^2 \\bI $. From this, one can obtain:\n \\begin{align}\\label{eqn:factor}\n   \\frac{\\usigma_X^2-\\osigma_X^2}{\\usigma_X^2+\\osigma_X^2} \\bI\\preceq \\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top\n   \\preceq \\frac{-\\usigma_X^2+\\osigma_X^2}{\\usigma_X^2+\\osigma_X^2}\\bI.\n \\end{align}\n By Lemma \\ref{lem:bandwidth}(c), $XX^\\top$ has bandwidth not greater than $2B_X$ induced by $(\\cG,\\cK,\\cK)$. From the nonsingularity of $XX^\\top$, we obtain the following:\n \\begin{subequations}\\label{eqn:key-1}\n   \\begin{align}\n     WX^{-1}Y &= \\frac{2}{\\usigma_X^2+\\osigma_X^2} WX^\\top \\left(\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^{-1}Y\\\\\n   % &= \\frac{2}{\\usigma_X^2+\\osigma_X^2} WX^\\top \\left(\\bI-\\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)\\right)^{-1} Y\\\\\n     &= \\frac{2}{\\usigma_X^2+\\osigma_X^2}WX^\\top \\left(\\sum_{q=0}^\\infty \\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^q\\right)Y,\\label{eqn:key-1-c}\\\\\n     &=\\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=0}^\\infty WX^\\top\\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^q Y.\n \\end{align}\n \\end{subequations}\n Here the second equality follows from \\cite[Corollary 5.6.16]{horn2012matrix} and \\eqref{eqn:factor}, and the last equality follows from the fact that the series in \\eqref{eqn:key-1-c} converges (due to \\eqref{eqn:factor}). Furthermore, from Lemma \\ref{lem:bandwidth}, we see that $WX^\\top\\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^q Y$ has bandwidth not greater than $(2q+1)B_X +B_Y+B_W$, induced by $(\\cG,\\cI,\\cJ)$. By extracting submatrices $(\\cdot)[I_i,J_j]$ from \\eqref{eqn:key-1}, we obtain:\n  \\begin{align*}\n    (WX^{-1}Y)_{[i][j]}&=\\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=q_0(i,j)}^\\infty \\left(WX^\\top\\left(\\bI-\\frac{2XX^\\top}{\\usigma_X^2+\\osigma_X^2} \\right)^q Y\\right)_{[i][j]}\n  \\end{align*}\n  where $q_0(i,j):=\\left\\lceil \\frac{d_\\cG(i,j)-B_X-B_Y-B_W}{2B_X}\\right\\rceil_+$; the summation over $q=0,\\cdots, q_0(i,j)-1$ is zero because such $q$ satisfy $(2q+1)B_X+B_Y+B_W< d_\\cG(i,j)$. Using the triangle inequality and the fact that the matrix norm of a submatrix is smaller than that of the original matrix,\n  \\begin{align}\\label{eqn:Hbound}\n    \\Vert (WX^{-1}Y)_{[i][j]} \\Vert &\\leq \\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=q_0(i,j)}^\\infty \\left\\Vert WX^\\top\\left(\\bI-\\frac{2XX^\\top}{\\usigma_X^2+\\osigma_X^2}\\right)^q Y\\right\\Vert\\\\\n    &\\leq \\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=q_0(i,j)}^\\infty \\osigma_W\\osigma_X\\left(\\frac{\\osigma_X^2-\\usigma_X^2}{\\osigma_X^2+\\usigma_X^2}\\right)^q \\osigma_Y\\nonumber\\\\\n    &\\leq \\frac{\\osigma_X\\osigma_Y\\osigma_W}{\\usigma_X^2}\\left(\\frac{\\osigma_X^2-\\usigma_X^2}{\\osigma_X^2+\\usigma_X^2}\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)-B_X-B_Y-B_W}{2B_X}\\right\\rceil_+} .\\nonumber\n  \\end{align}\n  The second inequality follows from the submultiplicativity of the matrix norm and \\eqref{eqn:factor}; and the last inequality follows from the summation of geometric series. \n\\end{proof}\n\n\\begin{lemma}\\label{lem:nlp-sensitivity}\n  Under Assumptions \\ref{ass:c2}, \\ref{ass:regularity}, there exist neighborhoods $\\mathbb{P}\\subseteq\\mathbb{R}^{\\bl}$ of $\\bp^\\star$ and $\\bbZ\\subseteq\\mathbb{R}^{\\bn}$ of $\\bz^\\star$ and a continuous function $\\bz^\\dag:\\mathbb{P}\\rightarrow \\bbZ$ such that $\\bz^\\dag(\\bp)$ is a primal-dual solution of $P(\\bp)$ that satisfies SSOSC and LICQ. Furthermore, for any $\\bq:=\\{q_i\\}_{i\\in \\cV}\\in\\mathbb{R}^{\\bl}$, the one-sided directional derivative of $\\bz^\\dag(\\cdot)$ is given by:\n  \\begin{align*}\n    D_{\\bq} \\bz^\\dag(\\bp)&:=\\lim_{h\\searrow 0}\\frac{\\bz^\\dag(\\bp+\\bq h)-\\bz^\\dag(\\bp)}{h}; \\label{eqn:one-sided-DD}\n  \\end{align*}\n  with $D_{\\bq}\\bx^\\dag(\\bp) = \\bxi^\\dag(\\bp,\\bq)$, $D_{\\bq}\\by^\\dag(\\bp,\\bq) = \\bbeta^\\dag(\\bp,\\bq)$, where $(\\bxi^\\dag(\\bp,\\bq),\\bbeta^\\dag(\\bp,\\bq)[\\cA^1(\\bp)])$ is a primal-dual solution of the quadratic program:\n  \\begin{subequations}\\label{eqn:nlp-sensitivity-0}\n    \\begin{align}\n      QP(\\bp,\\bq):\\quad\\min_{\\bxi}\\;&\\frac{1}{2}\\bxi^\\top\\bQ(\\bp)\\bxi + \\bxi^\\top\\bS(\\bp)\\bq,\\label{eqn:nlp-sensitivity-0-obj}\\\\\n      \\st\\;&\\bA(\\bp)[i,:]^\\top\\bxi + \\bB(\\bp)[i,:]\\bq = 0,\\;i\\in \\cA^0(\\bp),\\; (\\bbeta[i])\\\\\n                   &\\bA(\\bp)[i,:]^\\top\\bxi + \\bB(\\bp)[i,:]\\bq \\geq 0,\\;i\\in \\cA^1(\\bp)\\setminus \\cA^0(\\bp),\\; (\\bbeta[i])\n    \\end{align}      \n  \\end{subequations}\n  and $\\bbeta^\\dag(\\bp,\\bq)[\\mathbb{I}_{[1,\\bm]}\\setminus\\cA^1(\\bp)]=\\bzero$ (i.e., the free dual variables are fixed to zero); here,\n  \\begin{subequations}\\label{eqn:AAQA}\n    \\begin{align}\n      \\cA^0(\\bp)&:=\\cA^E\\cup\\{i\\in\\cA^I:\\bc(\\bx^\\dag(\\bp))[i]=0,\\;\\by^\\dag(\\bp)[i]\\neq 0\\}\\\\\n      \\cA^1(\\bp)&:=\\cA^E\\cup\\{i\\in\\cA^I:\\bc(\\bx^\\dag(\\bp))[i]=0\\}\\\\\n      \\bQ(\\bp)&:=\\nabla_{\\bx\\bx}\\cL(\\bz^\\dag(\\bp);\\bp);\\quad \\bS(\\bp):=\\nabla_{\\bx\\bp}\\cL(\\bz^\\dag(\\bp);\\bp)\\\\\n      \\bA(\\bp)&:=\\nabla_{\\bx}\\bc(\\bx^\\dag(\\bp);\\bp);\\quad \\bB(\\bp):=\\nabla_{\\bp}\\bc(\\bx^\\dag(\\bp);\\bp).\n    \\end{align}\n  \\end{subequations}\n  Moreover, a unique primal-dual solution of $QP(\\bp,\\bq)$ exists for any $\\bp\\in\\mathbb{P}$ and $\\bq\\in\\mathbb{R}^{\\bl}$; thus, $\\bxi^\\dag(\\cdot,\\cdot)$ and $\\bbeta^\\dag(\\cdot,\\cdot)$ are well-defined.\n\\end{lemma}\n\n\\begin{proof}\n  The results in \\cite[Theorem 2G.8]{dontchev2009implicit} ensure semidifferentiability (which guarantees continuity) of the solution of the first-order optimality conditions for $P(\\bp)$ over a certain neighborhood of $\\bp^\\star$. Furthermore, \\cite[Theorem 2G.9]{dontchev2009implicit} establishes that over a certain neighborhood $\\mathbb{P}$, the solution mapping of the generalized equation (GE) satisfies SSOSC and LICQ; that is, within $\\mathbb{P}$, the solution mapping for the GE is the solution mapping for $P(\\bp)$ at which SSOSC and LICQ are satisfied. Moreover, by \\cite[Theorem 2G.8]{dontchev2009implicit}, the one-sided directional derivative of the solution mapping for the GE (which exists for any direction $\\bq\\in\\mathbb{R}^{\\bl}$ by semidifferentiability \\cite[Theorem 2D.1]{dontchev2009implicit}) can be evaluated by using the linearized GE. The linear GE corresponds to the first-order optimality conditions for $QP(\\bp,\\bq)$ (see \\cite[Equation (35)]{dontchev2009implicit}); here, the first-order conditions are necessary and sufficient conditions for the optimality due to the convexity $QP(\\bp,\\bq)$ (guaranteed by SSOSC and LICQ of $\\bz^\\star$ for the original problem). As such, the one-sided directional derivative of the solution mapping for $P(\\bp)$ can be evaluated by solving $QP(\\bp,\\bq)$. Finally, the strong regularity of the GE at $\\bz^\\star$ guarantees that there exists a unique solution of the linearized GE, which in turn guarantees the existence of a unique solution of $QP(\\bp,\\bq)$.  \n\\end{proof}\n\n\\begin{lemma}\\label{lem:dder}\n  Suppose Assumptions \\ref{ass:c2}, \\ref{ass:regularity} hold, and suppose that, for given $\\bp\\in\\mathbb{P}$ (defined in Lemma \\ref{lem:nlp-sensitivity}) and $\\bq:=\\{q_i\\}_{i\\in \\cV}\\in\\mathbb{R}^{\\bl}$, we have $\\osigma_{\\bH}(\\bp,\\bq)$, $\\osigma_{\\bR}(\\bp,\\bq)$, and $\\usigma_{\\bH}(\\bp,\\bq)$ such that:\n  \\begin{subequations}\\label{eqn:dder-0}\n  \\begin{align}\n    \\osigma_{\\bH}(\\bp,\\bq)&\\geq \\osigma(\\bH(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)])\\\\\n    \\osigma_{\\bR}(\\bp,\\bq)&\\geq \\osigma(\\bR(\\bp)[\\cB'(\\bp,\\bq),:])\\\\\n    0<\\usigma_{\\bH}(\\bp,\\bq)&\\leq \\usigma(\\bH(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)]);\\label{eqn:dder-0-c}\n  \\end{align}\n\\end{subequations}\nthen the following holds for any $i\\in \\cV$:\n  \\begin{align*}\n    \\Vert D_{\\bq} z^\\dag_i(\\bp,\\bq)\\Vert\\leq \\sum_{j\\in \\cV}\\frac{\\osigma_{\\bH}(\\bp,\\bq)\\osigma_{\\bR}(\\bp,\\bq)}{\\usigma_{\\bH}(\\bp,\\bq)^2}\\left(\\frac{\\osigma_{\\bH}(\\bp,\\bq)^2-\\usigma_{\\bH}(\\bp,\\bq)^2}{\\osigma_{\\bH}(\\bp,\\bq)^2+\\usigma_{\\bH}(\\bp,\\bq)^2}\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert.\n  \\end{align*}\n\\end{lemma}\n\n\\begin{proof}\n  {\\blue For simplicity, we denote $\\cB'(\\bp,\\bq)$, $\\osigma_{\\bH}(\\bp,\\bq)$, $\\osigma_{\\bR}(\\bp,\\bq)$, $\\usigma_{\\bH}(\\bp,\\bq)$ by $\\cB'$, $\\osigma_{\\bH}$, $\\osigma_{\\bR}$, $\\usigma_{\\bH}$}. From that $\\bH(\\bp)[\\cB',\\cB']$ is always nonsingular, we have that $\\usigma_{\\bH}(\\bp,\\bq)$ satisfying \\eqref{eqn:dder-0-c} always exists. By inspecting the block structure of \\eqref{eqn:linear-form} we obtain:\n  \\begin{align}\\label{eqn:nlp-sensitivity-1}\n    D_{\\bq}\\bz^\\dag(\\bp)[I_i\\cap\\cB'] &= \\sum_{j\\in \\cV} -((\\bH(\\bp)[\\cB',\\cB'])^{-1}\\bR(\\bp)[\\cB',:])_{[i][j]} q_j,\n  \\end{align}\n  where $\\cI^{\\cB'}:=\\{I_i\\cap{\\cB'}\\}_{i\\in \\cV}$ and $\\cK:=\\{K_i\\}_{i\\in \\cV}$ (defined in \\eqref{eqn:ind-1}) are used for index sets. We have already established that $\\bH(\\bp)[\\cB',\\cB']$ has bandwidth not greater than two, induced by $(\\cG,\\cI^{\\cB'},\\cI^{\\cB'})$ and that $\\bR(\\bp)[\\cB',:]$ has bandwidth not greater than two, induced by $(\\cG,\\cI^{\\cB'},\\cK)$. By applying Theorem \\ref{thm:inv}, we obtain:\n  \\begin{align}\\label{eqn:nlp-sensitivity-2}\\blue\n    \\Vert((\\bH(\\bp)[\\cB',\\cB'])^{-1}\\bR(\\bp)[\\cB',:])_{[i][j]}\\Vert\\leq \\frac{\\osigma_{\\bH}\\osigma_{\\bR}}{\\usigma_{\\bH}^2}\\left(\\frac{\\osigma_{\\bH}^2-\\usigma_{\\bH}^2}{\\osigma_{\\bH}^2+\\usigma_{\\bH}^2}\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+} .\n  \\end{align}\n  Now note that\n  $\\Vert D_{\\bq}z^\\dag_i(\\bp)\\Vert \\leq \\Vert D_{\\bq}\\bz^\\dag(\\bp)[I_i\\cap\\cB'] \\Vert  + \\Vert D_{\\bq}\\bz^\\dag(\\bp)[I_i\\setminus\\cB']\\Vert$,\n  and recall from \\eqref{eqn:linear-form-b} that $D_{\\bq}\\bz^\\dag(\\bp)[I_i\\setminus\\cB'] =\\bzero$. Hence, by applying \\eqref{eqn:nlp-sensitivity-2} to \\eqref{eqn:nlp-sensitivity-1}, and applying triangle inequality, we obtain the desired result.\n\\end{proof}\n\n\\begin{proof}\n  From the continuity of $\\bz^\\dag(\\cdot)$ in the neighborhood of $\\bp^\\star$ and the continuity of $\\bc(\\cdot,\\cdot)$ in the neighborhood of $[\\bz^\\star;\\bp^\\star]$, there exists a neighborhood $\\widetilde{\\mathbb{P}}\\subseteq\\mathbb{P}$ of $\\bp^\\star$ such that, for $\\bp\\in\\widetilde{\\mathbb{P}}$ and $i\\in\\mathbb{I}_{[1,\\bm]}$, $\\bc(\\bx^\\star)[i] > 0\\Rightarrow\\bc(\\bx^\\dag(\\bp))[i] > 0$ and $\\by^\\star[i] \\neq 0\\Rightarrow\\by^\\dag(\\bp)[i] \\neq 0$. These conditions and complementarity slackness imply that for $\\bp\\in\\widetilde{\\mathbb{P}}$, we have $\\cA^0(\\bp^\\star)\\subseteq\\cA^0(\\bp)$ and $\\cA^1(\\bp)\\subseteq\\cA^1(\\bp^\\star)$; that is, $\\cB^0(\\bp^\\star)\\subseteq\\cB^0(\\bp)$ and $\\cB^1(\\bp)\\subseteq\\cB^1(\\bp^\\star)$. This and $\\cB^0(\\bp)\\subseteq\\cB'(\\bp,\\bq)\\subseteq\\cB^1(\\bp)$ yields:\n  \\begin{subequations}\\label{eqn:main-0}\n  \\begin{align}\n    \\osigma({\\bH}(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)])&\\leq \\oosigma_{\\bH}(\\bp)\\\\\n    \\osigma({\\bR}(\\bp)[\\cB'(\\bp,\\bq),:])&\\leq \\oosigma_{\\bR}(\\bp)\\\\\n    \\usigma({\\bH}(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)])&\\geq \\uusigma_{\\bH}(\\bp).\n  \\end{align}\n\\end{subequations}\n  By the twice-continuous differentiability of $\\cL(\\cdot,\\cdot)$ and the continuity of $\\bz^\\dag(\\cdot)$, we have that $\\bH(\\cdot)$ is continuous. The same holds true for its submatrices: $\\bH(\\cdot)[\\cB,\\cB]$ with $\\cB^0(\\bp^\\star)\\subseteq \\cB\\subseteq \\cB^1(\\bp^\\star)$. From the continuity of singular values with respect to its entries \\cite[Corollary 8.6.2]{golub2012matrix}, we have that $\\osigma(\\bH(\\cdot)[\\cB,\\cB])$ and $\\usigma(\\bH(\\cdot)[\\cB,\\cB])$ are continuous for any $\\cB^0(\\bp^\\star)\\subseteq \\cB\\subseteq \\cB^1(\\bp^\\star)$; accordingly, since a maximum and a minimum of a fixed and finite number of continuous functions is continuous, we have that $\\oosigma_{\\bH}(\\bp)$, $\\oosigma_{\\bR}(\\bp)$, $\\uusigma_{\\bH}(\\bp)$ are continuous with respect to $\\bp$ in $\\widetilde{\\mathbb{P}}$. Thus, there exists a convex neighborhood $\\mathbb{P}_{\\epsilon}\\subseteq\\widetilde{\\mathbb{P}}$ of $\\bp^\\star$ wherein the following are satisfied:\n  \\begin{subequations}\\label{eqn:main-1}\n  \\begin{align}\n    \\frac{\\oosigma_{\\bH}(\\bp)\\oosigma_{\\bR}(\\bp)}{\\uusigma_{\\bH}(\\bp)^2} &\\leq\n    \\frac{\\oosigma_{\\bH}(\\bp^\\star)\\oosigma_{\\bR}(\\bp^\\star)}{\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon\\\\\n    \\frac{\\oosigma_{\\bH}(\\bp)^2-\\uusigma_{\\bH}(\\bp)^2}{\\oosigma_{\\bH}(\\bp)^2+\\uusigma_{\\bH}(\\bp)^2}&\\leq\n    \\frac{\\oosigma_{\\bH}(\\bp^\\star)^2-\\uusigma_{\\bH}(\\bp^\\star)^2}{\\oosigma_{\\bH}(\\bp^\\star)^2+\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon.\n  \\end{align}\n\\end{subequations}\nHere, note that $\\uusigma_{\\bH}(\\bp^\\star)>0$ because $\\usigma(\\bH[\\cB,\\cB])>0$ holds for any $\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)$. By applying \\eqref{eqn:main-0} and \\eqref{eqn:main-1} to Lemma \\ref{lem:dder}, we obtain:\n\\begin{align}\\label{eqn:main-2}\n  \\Vert D_{\\bq} z^\\dag_i(\\bp)\\Vert&\\leq \\sum_{j\\in \\cV}\\left(\\frac{\\oosigma_{\\bH}(\\bp^\\star)\\oosigma_{\\bR}(\\bp^\\star)}{\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon\\right)\\left(\\frac{\\oosigma_{\\bH}(\\bp^\\star)^2-\\uusigma_{\\bH}(\\bp^\\star)^2}{\\oosigma_{\\bH}(\\bp^\\star)^2+\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert\\nonumber\\\\\n  &\\leq \\sum_{j\\in \\cV}\\left(\\frac{\\osigma_{\\bH}\\osigma_{\\bR}}{\\usigma_{\\bH}^2}+\\epsilon\\right)\\left(\\frac{\\osigma_{\\bH}^2-\\usigma_{\\bH}^2}{\\osigma_{\\bH}^2+\\usigma_{\\bH}^2}+\\epsilon\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert\\nonumber\\\\\n  &\\leq \\sum_{j\\in \\cV}\\Upsilon\\rho^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert,\n\\end{align}\nfor any $\\bp\\in\\mathbb{P}_{\\epsilon}$ and $\\bq\\in\\mathbb{R}^{\\bl}$. Finally, since we have chosen $\\mathbb{P}_\\epsilon$ to be convex, for any $\\bp,\\bp'\\in\\mathbb{P}_{\\epsilon}$, the line segment between $\\bp,\\bp'$ is within $\\mathbb{P}_{\\epsilon}$. Thus, we have:\n\\begin{align*}\n  \\Vert z^\\dag_i(\\bp)-z^\\dag_i(\\bp')\\Vert\n  % &\\leq\\Vert \\int_{0}^1 D_{\\bp'-\\bp}z^\\dag_i ((1-t)\\bp+t\\bp')dt\\Vert\\\\\n    &\\leq \\int_{0}^1\\Vert D_{\\bp'-\\bp}z^\\dag_i ((1-t)\\bp+t\\bp')\\Vert dt \\leq \\sum_{j\\in \\cV}\\Upsilon\\rho^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert p_j-p'_j\\Vert,\n\\end{align*}\nwhere the first inequality is from Newton-Leibniz and triangle inequality for integrals, and the last inequality follows from \\eqref{eqn:main-2}.\n\\end{proof}","units":[{"raw_tex":"\\begin{lemma}\\label{lem:bandwidth}\n  Consider $X\\in\\mathbb{R}^{m\\times n}$ with bandwidth not greater than $B_X$ induced by $(\\cG,\\cI,\\cJ)$; we have that:\n  (a) $X^\\top$ has bandwidth not greater than $B_X$ induced by $(\\cG,\\cJ,\\cI)$;\n  (b) if $Y\\in\\mathbb{R}^{m\\times n}$ has bandwidth not greater than $B_Y$ induced by $(\\cG,\\cI,\\cJ)$, then $X+Y$ has bandwidth not greater than $\\max(B_X,B_Y)$ induced by $(\\cG,\\cI,\\cJ)$;\n  (c) if $W\\in\\mathbb{R}^{n\\times k}$ has bandwidth not greater than $B_W$ induced by $(\\cG,\\cJ,\\cK)$, then $XW$ has bandwidth not greater than $B_X+B_W$ induced by $(\\cG,\\cI,\\cK)$.\n\\end{lemma}","start":27298,"end":27915},{"raw_tex":"\\begin{proof}[Proof of (a)] We have that $(X^\\top)_{[i][j]}=(X^\\top){[J_i,I_j]}=(X[I_j,J_i])^\\top=(X_{[j][i]})^\\top$. From the assumption that $X$ has bandwidth not greater than $B_X$ and Proposition \\ref{prop:metric}(c), $(X^\\top)_{[i][j]}=\\bzero$ if $d_\\cG(i,j)>B_X$; therefore, $X$ has bandwidth not greater than $B_X$, and induced by $(\\cG,\\cJ,\\cI)$.\n\\end{proof}","start":27916,"end":28282},{"raw_tex":"\\begin{proof}[Proof of (b)] We have that $X_{[i][j]}=\\bzero$ and $Y_{[i][j]}=\\bzero$ if $d_\\cG(i,j)>\\max(B_X,B_Y)$, which yields $(X+Y)_{[i][j]}=\\bzero$ if $d_\\cG(i,j)>\\max(B_X,B_Y)$. Thus, $X+Y$ has bandwidth not greater than $\\max(B_X,B_Y)$, and induced by $(\\cG,\\cI,\\cJ)$.\n\\end{proof}","start":28283,"end":28570},{"raw_tex":"\\begin{proof}[Proof of (c)] If $d_\\cG(i,j)>B_X+B_W$, from Proposition \\ref{prop:metric}(d) we have that for any $k\\in \\cV$, $d_\\cG(i,k)>B_X$ or $d_\\cG(j,k)>B_W$. Thus, if $d_\\cG(i,j)>B_X+B_W$,  we have $(XW)_{[i][j]}=\\sum_{k\\in \\cV} X_{[i][k]}W_{[k][j]}= \\bzero$ (where the first equality comes from the block matrix multiplication law and the second equality comes from observing that $d_\\cG(i,k)>B_X$ or $d_\\cG(j,k)>B_W$). Therefore, $XW$ has bandwidth not greater than $B_X+B_W$, and induced by $(\\cG,\\cI,\\cK)$.  \n\\end{proof}","start":28571,"end":29099},{"raw_tex":"\\begin{theorem}\\label{thm:inv}\n  Consider a nonsingular matrix $X\\in\\mathbb{R}^{n\\times n}$ with bandwidth not greater than $B_X\\geq 1$ induced by $(\\cG,\\cK,\\cP)$, $Y\\in\\mathbb{R}^{n\\times m}$ with bandwidth not greater than $B_Y$ induced by $(\\cG, \\cK,\\cJ)$, and $W\\in\\mathbb{R}^{\\ell\\times n}$ with bandwidth not greater than $B_W$ induced by $(\\cG, \\cI,\\cP)$; for constants $\\osigma_X\\geq \\osigma(X)$, $\\osigma_Y\\geq \\osigma(Y)$, $\\osigma_W\\geq \\osigma(W)$, and $0<\\usigma_X\\leq \\usigma(X)$, the following holds:\n  \\begin{align}\\label{eqn:inv}\n    \\Vert (WX^{-1}Y)_{[i][j]}\\Vert \\leq\n    \\frac{\\osigma_X\\osigma_Y\\osigma_W}{\\usigma_X^2}\\left(\\frac{\\osigma_X^2-\\usigma_X^2}{\\osigma_X^2+\\usigma_X^2}\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)-B_X-B_Y-B_W}{2B_X}\\right\\rceil_+} ,\\quad i,j\\in \\cV,\n  \\end{align}\n  where $(WX^{-1}Y)_{[i][j]}:=(WX^{-1}Y)[I_i,J_j]$ and $\\lceil\\cdot\\rceil_+$ is the smallest non-negative integer that is greater than or equal to the argument.\n\\end{theorem}","start":29434,"end":30412},{"raw_tex":"\\begin{proof}\n  By definition of singular values and the assumptions on $\\osigma_X$, $\\osigma_Y$ , $\\osigma_W$, and $\\usigma_X$, we have $\\usigma_X^2 \\bI \\preceq\\usigma(X)^2 \\bI \\preceq X X^\\top\\preceq  \\osigma(X)^2\\bI\\preceq\\osigma_X^2 \\bI $. From this, one can obtain:\n \\begin{align}\\label{eqn:factor}\n   \\frac{\\usigma_X^2-\\osigma_X^2}{\\usigma_X^2+\\osigma_X^2} \\bI\\preceq \\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top\n   \\preceq \\frac{-\\usigma_X^2+\\osigma_X^2}{\\usigma_X^2+\\osigma_X^2}\\bI.\n \\end{align}\n By Lemma \\ref{lem:bandwidth}(c), $XX^\\top$ has bandwidth not greater than $2B_X$ induced by $(\\cG,\\cK,\\cK)$. From the nonsingularity of $XX^\\top$, we obtain the following:\n \\begin{subequations}\\label{eqn:key-1}\n   \\begin{align}\n     WX^{-1}Y &= \\frac{2}{\\usigma_X^2+\\osigma_X^2} WX^\\top \\left(\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^{-1}Y\\\\\n   % &= \\frac{2}{\\usigma_X^2+\\osigma_X^2} WX^\\top \\left(\\bI-\\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)\\right)^{-1} Y\\\\\n     &= \\frac{2}{\\usigma_X^2+\\osigma_X^2}WX^\\top \\left(\\sum_{q=0}^\\infty \\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^q\\right)Y,\\label{eqn:key-1-c}\\\\\n     &=\\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=0}^\\infty WX^\\top\\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^q Y.\n \\end{align}\n \\end{subequations}\n Here the second equality follows from \\cite[Corollary 5.6.16]{horn2012matrix} and \\eqref{eqn:factor}, and the last equality follows from the fact that the series in \\eqref{eqn:key-1-c} converges (due to \\eqref{eqn:factor}). Furthermore, from Lemma \\ref{lem:bandwidth}, we see that $WX^\\top\\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^q Y$ has bandwidth not greater than $(2q+1)B_X +B_Y+B_W$, induced by $(\\cG,\\cI,\\cJ)$. By extracting submatrices $(\\cdot)[I_i,J_j]$ from \\eqref{eqn:key-1}, we obtain:\n  \\begin{align*}\n    (WX^{-1}Y)_{[i][j]}&=\\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=q_0(i,j)}^\\infty \\left(WX^\\top\\left(\\bI-\\frac{2XX^\\top}{\\usigma_X^2+\\osigma_X^2} \\right)^q Y\\right)_{[i][j]}\n  \\end{align*}\n  where $q_0(i,j):=\\left\\lceil \\frac{d_\\cG(i,j)-B_X-B_Y-B_W}{2B_X}\\right\\rceil_+$; the summation over $q=0,\\cdots, q_0(i,j)-1$ is zero because such $q$ satisfy $(2q+1)B_X+B_Y+B_W< d_\\cG(i,j)$. Using the triangle inequality and the fact that the matrix norm of a submatrix is smaller than that of the original matrix,\n  \\begin{align}\\label{eqn:Hbound}\n    \\Vert (WX^{-1}Y)_{[i][j]} \\Vert &\\leq \\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=q_0(i,j)}^\\infty \\left\\Vert WX^\\top\\left(\\bI-\\frac{2XX^\\top}{\\usigma_X^2+\\osigma_X^2}\\right)^q Y\\right\\Vert\\\\\n    &\\leq \\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=q_0(i,j)}^\\infty \\osigma_W\\osigma_X\\left(\\frac{\\osigma_X^2-\\usigma_X^2}{\\osigma_X^2+\\usigma_X^2}\\right)^q \\osigma_Y\\nonumber\\\\\n    &\\leq \\frac{\\osigma_X\\osigma_Y\\osigma_W}{\\usigma_X^2}\\left(\\frac{\\osigma_X^2-\\usigma_X^2}{\\osigma_X^2+\\usigma_X^2}\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)-B_X-B_Y-B_W}{2B_X}\\right\\rceil_+} .\\nonumber\n  \\end{align}\n  The second inequality follows from the submultiplicativity of the matrix norm and \\eqref{eqn:factor}; and the last inequality follows from the summation of geometric series. \n\\end{proof}","start":30413,"end":33580},{"raw_tex":"\\begin{lemma}\\label{lem:nlp-sensitivity}\n  Under Assumptions \\ref{ass:c2}, \\ref{ass:regularity}, there exist neighborhoods $\\mathbb{P}\\subseteq\\mathbb{R}^{\\bl}$ of $\\bp^\\star$ and $\\bbZ\\subseteq\\mathbb{R}^{\\bn}$ of $\\bz^\\star$ and a continuous function $\\bz^\\dag:\\mathbb{P}\\rightarrow \\bbZ$ such that $\\bz^\\dag(\\bp)$ is a primal-dual solution of $P(\\bp)$ that satisfies SSOSC and LICQ. Furthermore, for any $\\bq:=\\{q_i\\}_{i\\in \\cV}\\in\\mathbb{R}^{\\bl}$, the one-sided directional derivative of $\\bz^\\dag(\\cdot)$ is given by:\n  \\begin{align*}\n    D_{\\bq} \\bz^\\dag(\\bp)&:=\\lim_{h\\searrow 0}\\frac{\\bz^\\dag(\\bp+\\bq h)-\\bz^\\dag(\\bp)}{h}; \\label{eqn:one-sided-DD}\n  \\end{align*}\n  with $D_{\\bq}\\bx^\\dag(\\bp) = \\bxi^\\dag(\\bp,\\bq)$, $D_{\\bq}\\by^\\dag(\\bp,\\bq) = \\bbeta^\\dag(\\bp,\\bq)$, where $(\\bxi^\\dag(\\bp,\\bq),\\bbeta^\\dag(\\bp,\\bq)[\\cA^1(\\bp)])$ is a primal-dual solution of the quadratic program:\n  \\begin{subequations}\\label{eqn:nlp-sensitivity-0}\n    \\begin{align}\n      QP(\\bp,\\bq):\\quad\\min_{\\bxi}\\;&\\frac{1}{2}\\bxi^\\top\\bQ(\\bp)\\bxi + \\bxi^\\top\\bS(\\bp)\\bq,\\label{eqn:nlp-sensitivity-0-obj}\\\\\n      \\st\\;&\\bA(\\bp)[i,:]^\\top\\bxi + \\bB(\\bp)[i,:]\\bq = 0,\\;i\\in \\cA^0(\\bp),\\; (\\bbeta[i])\\\\\n                   &\\bA(\\bp)[i,:]^\\top\\bxi + \\bB(\\bp)[i,:]\\bq \\geq 0,\\;i\\in \\cA^1(\\bp)\\setminus \\cA^0(\\bp),\\; (\\bbeta[i])\n    \\end{align}      \n  \\end{subequations}\n  and $\\bbeta^\\dag(\\bp,\\bq)[\\mathbb{I}_{[1,\\bm]}\\setminus\\cA^1(\\bp)]=\\bzero$ (i.e., the free dual variables are fixed to zero); here,\n  \\begin{subequations}\\label{eqn:AAQA}\n    \\begin{align}\n      \\cA^0(\\bp)&:=\\cA^E\\cup\\{i\\in\\cA^I:\\bc(\\bx^\\dag(\\bp))[i]=0,\\;\\by^\\dag(\\bp)[i]\\neq 0\\}\\\\\n      \\cA^1(\\bp)&:=\\cA^E\\cup\\{i\\in\\cA^I:\\bc(\\bx^\\dag(\\bp))[i]=0\\}\\\\\n      \\bQ(\\bp)&:=\\nabla_{\\bx\\bx}\\cL(\\bz^\\dag(\\bp);\\bp);\\quad \\bS(\\bp):=\\nabla_{\\bx\\bp}\\cL(\\bz^\\dag(\\bp);\\bp)\\\\\n      \\bA(\\bp)&:=\\nabla_{\\bx}\\bc(\\bx^\\dag(\\bp);\\bp);\\quad \\bB(\\bp):=\\nabla_{\\bp}\\bc(\\bx^\\dag(\\bp);\\bp).\n    \\end{align}\n  \\end{subequations}\n  Moreover, a unique primal-dual solution of $QP(\\bp,\\bq)$ exists for any $\\bp\\in\\mathbb{P}$ and $\\bq\\in\\mathbb{R}^{\\bl}$; thus, $\\bxi^\\dag(\\cdot,\\cdot)$ and $\\bbeta^\\dag(\\cdot,\\cdot)$ are well-defined.\n\\end{lemma}","start":38950,"end":41114},{"raw_tex":"\\begin{proof}\n  The results in \\cite[Theorem 2G.8]{dontchev2009implicit} ensure semidifferentiability (which guarantees continuity) of the solution of the first-order optimality conditions for $P(\\bp)$ over a certain neighborhood of $\\bp^\\star$. Furthermore, \\cite[Theorem 2G.9]{dontchev2009implicit} establishes that over a certain neighborhood $\\mathbb{P}$, the solution mapping of the generalized equation (GE) satisfies SSOSC and LICQ; that is, within $\\mathbb{P}$, the solution mapping for the GE is the solution mapping for $P(\\bp)$ at which SSOSC and LICQ are satisfied. Moreover, by \\cite[Theorem 2G.8]{dontchev2009implicit}, the one-sided directional derivative of the solution mapping for the GE (which exists for any direction $\\bq\\in\\mathbb{R}^{\\bl}$ by semidifferentiability \\cite[Theorem 2D.1]{dontchev2009implicit}) can be evaluated by using the linearized GE. The linear GE corresponds to the first-order optimality conditions for $QP(\\bp,\\bq)$ (see \\cite[Equation (35)]{dontchev2009implicit}); here, the first-order conditions are necessary and sufficient conditions for the optimality due to the convexity $QP(\\bp,\\bq)$ (guaranteed by SSOSC and LICQ of $\\bz^\\star$ for the original problem). As such, the one-sided directional derivative of the solution mapping for $P(\\bp)$ can be evaluated by solving $QP(\\bp,\\bq)$. Finally, the strong regularity of the GE at $\\bz^\\star$ guarantees that there exists a unique solution of the linearized GE, which in turn guarantees the existence of a unique solution of $QP(\\bp,\\bq)$.  \n\\end{proof}","start":41310,"end":42862},{"raw_tex":"\\begin{lemma}\\label{lem:dder}\n  Suppose Assumptions \\ref{ass:c2}, \\ref{ass:regularity} hold, and suppose that, for given $\\bp\\in\\mathbb{P}$ (defined in Lemma \\ref{lem:nlp-sensitivity}) and $\\bq:=\\{q_i\\}_{i\\in \\cV}\\in\\mathbb{R}^{\\bl}$, we have $\\osigma_{\\bH}(\\bp,\\bq)$, $\\osigma_{\\bR}(\\bp,\\bq)$, and $\\usigma_{\\bH}(\\bp,\\bq)$ such that:\n  \\begin{subequations}\\label{eqn:dder-0}\n  \\begin{align}\n    \\osigma_{\\bH}(\\bp,\\bq)&\\geq \\osigma(\\bH(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)])\\\\\n    \\osigma_{\\bR}(\\bp,\\bq)&\\geq \\osigma(\\bR(\\bp)[\\cB'(\\bp,\\bq),:])\\\\\n    0<\\usigma_{\\bH}(\\bp,\\bq)&\\leq \\usigma(\\bH(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)]);\\label{eqn:dder-0-c}\n  \\end{align}\n\\end{subequations}\nthen the following holds for any $i\\in \\cV$:\n  \\begin{align*}\n    \\Vert D_{\\bq} z^\\dag_i(\\bp,\\bq)\\Vert\\leq \\sum_{j\\in \\cV}\\frac{\\osigma_{\\bH}(\\bp,\\bq)\\osigma_{\\bR}(\\bp,\\bq)}{\\usigma_{\\bH}(\\bp,\\bq)^2}\\left(\\frac{\\osigma_{\\bH}(\\bp,\\bq)^2-\\usigma_{\\bH}(\\bp,\\bq)^2}{\\osigma_{\\bH}(\\bp,\\bq)^2+\\usigma_{\\bH}(\\bp,\\bq)^2}\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert.\n  \\end{align*}\n\\end{lemma}","start":49216,"end":50305},{"raw_tex":"\\begin{proof}\n  {\\blue For simplicity, we denote $\\cB'(\\bp,\\bq)$, $\\osigma_{\\bH}(\\bp,\\bq)$, $\\osigma_{\\bR}(\\bp,\\bq)$, $\\usigma_{\\bH}(\\bp,\\bq)$ by $\\cB'$, $\\osigma_{\\bH}$, $\\osigma_{\\bR}$, $\\usigma_{\\bH}$}. From that $\\bH(\\bp)[\\cB',\\cB']$ is always nonsingular, we have that $\\usigma_{\\bH}(\\bp,\\bq)$ satisfying \\eqref{eqn:dder-0-c} always exists. By inspecting the block structure of \\eqref{eqn:linear-form} we obtain:\n  \\begin{align}\\label{eqn:nlp-sensitivity-1}\n    D_{\\bq}\\bz^\\dag(\\bp)[I_i\\cap\\cB'] &= \\sum_{j\\in \\cV} -((\\bH(\\bp)[\\cB',\\cB'])^{-1}\\bR(\\bp)[\\cB',:])_{[i][j]} q_j,\n  \\end{align}\n  where $\\cI^{\\cB'}:=\\{I_i\\cap{\\cB'}\\}_{i\\in \\cV}$ and $\\cK:=\\{K_i\\}_{i\\in \\cV}$ (defined in \\eqref{eqn:ind-1}) are used for index sets. We have already established that $\\bH(\\bp)[\\cB',\\cB']$ has bandwidth not greater than two, induced by $(\\cG,\\cI^{\\cB'},\\cI^{\\cB'})$ and that $\\bR(\\bp)[\\cB',:]$ has bandwidth not greater than two, induced by $(\\cG,\\cI^{\\cB'},\\cK)$. By applying Theorem \\ref{thm:inv}, we obtain:\n  \\begin{align}\\label{eqn:nlp-sensitivity-2}\\blue\n    \\Vert((\\bH(\\bp)[\\cB',\\cB'])^{-1}\\bR(\\bp)[\\cB',:])_{[i][j]}\\Vert\\leq \\frac{\\osigma_{\\bH}\\osigma_{\\bR}}{\\usigma_{\\bH}^2}\\left(\\frac{\\osigma_{\\bH}^2-\\usigma_{\\bH}^2}{\\osigma_{\\bH}^2+\\usigma_{\\bH}^2}\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+} .\n  \\end{align}\n  Now note that\n  $\\Vert D_{\\bq}z^\\dag_i(\\bp)\\Vert \\leq \\Vert D_{\\bq}\\bz^\\dag(\\bp)[I_i\\cap\\cB'] \\Vert  + \\Vert D_{\\bq}\\bz^\\dag(\\bp)[I_i\\setminus\\cB']\\Vert$,\n  and recall from \\eqref{eqn:linear-form-b} that $D_{\\bq}\\bz^\\dag(\\bp)[I_i\\setminus\\cB'] =\\bzero$. Hence, by applying \\eqref{eqn:nlp-sensitivity-2} to \\eqref{eqn:nlp-sensitivity-1}, and applying triangle inequality, we obtain the desired result.\n\\end{proof}","start":50306,"end":52052},{"raw_tex":"\\begin{proof}\n  From the continuity of $\\bz^\\dag(\\cdot)$ in the neighborhood of $\\bp^\\star$ and the continuity of $\\bc(\\cdot,\\cdot)$ in the neighborhood of $[\\bz^\\star;\\bp^\\star]$, there exists a neighborhood $\\widetilde{\\mathbb{P}}\\subseteq\\mathbb{P}$ of $\\bp^\\star$ such that, for $\\bp\\in\\widetilde{\\mathbb{P}}$ and $i\\in\\mathbb{I}_{[1,\\bm]}$, $\\bc(\\bx^\\star)[i] > 0\\Rightarrow\\bc(\\bx^\\dag(\\bp))[i] > 0$ and $\\by^\\star[i] \\neq 0\\Rightarrow\\by^\\dag(\\bp)[i] \\neq 0$. These conditions and complementarity slackness imply that for $\\bp\\in\\widetilde{\\mathbb{P}}$, we have $\\cA^0(\\bp^\\star)\\subseteq\\cA^0(\\bp)$ and $\\cA^1(\\bp)\\subseteq\\cA^1(\\bp^\\star)$; that is, $\\cB^0(\\bp^\\star)\\subseteq\\cB^0(\\bp)$ and $\\cB^1(\\bp)\\subseteq\\cB^1(\\bp^\\star)$. This and $\\cB^0(\\bp)\\subseteq\\cB'(\\bp,\\bq)\\subseteq\\cB^1(\\bp)$ yields:\n  \\begin{subequations}\\label{eqn:main-0}\n  \\begin{align}\n    \\osigma({\\bH}(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)])&\\leq \\oosigma_{\\bH}(\\bp)\\\\\n    \\osigma({\\bR}(\\bp)[\\cB'(\\bp,\\bq),:])&\\leq \\oosigma_{\\bR}(\\bp)\\\\\n    \\usigma({\\bH}(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)])&\\geq \\uusigma_{\\bH}(\\bp).\n  \\end{align}\n\\end{subequations}\n  By the twice-continuous differentiability of $\\cL(\\cdot,\\cdot)$ and the continuity of $\\bz^\\dag(\\cdot)$, we have that $\\bH(\\cdot)$ is continuous. The same holds true for its submatrices: $\\bH(\\cdot)[\\cB,\\cB]$ with $\\cB^0(\\bp^\\star)\\subseteq \\cB\\subseteq \\cB^1(\\bp^\\star)$. From the continuity of singular values with respect to its entries \\cite[Corollary 8.6.2]{golub2012matrix}, we have that $\\osigma(\\bH(\\cdot)[\\cB,\\cB])$ and $\\usigma(\\bH(\\cdot)[\\cB,\\cB])$ are continuous for any $\\cB^0(\\bp^\\star)\\subseteq \\cB\\subseteq \\cB^1(\\bp^\\star)$; accordingly, since a maximum and a minimum of a fixed and finite number of continuous functions is continuous, we have that $\\oosigma_{\\bH}(\\bp)$, $\\oosigma_{\\bR}(\\bp)$, $\\uusigma_{\\bH}(\\bp)$ are continuous with respect to $\\bp$ in $\\widetilde{\\mathbb{P}}$. Thus, there exists a convex neighborhood $\\mathbb{P}_{\\epsilon}\\subseteq\\widetilde{\\mathbb{P}}$ of $\\bp^\\star$ wherein the following are satisfied:\n  \\begin{subequations}\\label{eqn:main-1}\n  \\begin{align}\n    \\frac{\\oosigma_{\\bH}(\\bp)\\oosigma_{\\bR}(\\bp)}{\\uusigma_{\\bH}(\\bp)^2} &\\leq\n    \\frac{\\oosigma_{\\bH}(\\bp^\\star)\\oosigma_{\\bR}(\\bp^\\star)}{\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon\\\\\n    \\frac{\\oosigma_{\\bH}(\\bp)^2-\\uusigma_{\\bH}(\\bp)^2}{\\oosigma_{\\bH}(\\bp)^2+\\uusigma_{\\bH}(\\bp)^2}&\\leq\n    \\frac{\\oosigma_{\\bH}(\\bp^\\star)^2-\\uusigma_{\\bH}(\\bp^\\star)^2}{\\oosigma_{\\bH}(\\bp^\\star)^2+\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon.\n  \\end{align}\n\\end{subequations}\nHere, note that $\\uusigma_{\\bH}(\\bp^\\star)>0$ because $\\usigma(\\bH[\\cB,\\cB])>0$ holds for any $\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)$. By applying \\eqref{eqn:main-0} and \\eqref{eqn:main-1} to Lemma \\ref{lem:dder}, we obtain:\n\\begin{align}\\label{eqn:main-2}\n  \\Vert D_{\\bq} z^\\dag_i(\\bp)\\Vert&\\leq \\sum_{j\\in \\cV}\\left(\\frac{\\oosigma_{\\bH}(\\bp^\\star)\\oosigma_{\\bR}(\\bp^\\star)}{\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon\\right)\\left(\\frac{\\oosigma_{\\bH}(\\bp^\\star)^2-\\uusigma_{\\bH}(\\bp^\\star)^2}{\\oosigma_{\\bH}(\\bp^\\star)^2+\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert\\nonumber\\\\\n  &\\leq \\sum_{j\\in \\cV}\\left(\\frac{\\osigma_{\\bH}\\osigma_{\\bR}}{\\usigma_{\\bH}^2}+\\epsilon\\right)\\left(\\frac{\\osigma_{\\bH}^2-\\usigma_{\\bH}^2}{\\osigma_{\\bH}^2+\\usigma_{\\bH}^2}+\\epsilon\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert\\nonumber\\\\\n  &\\leq \\sum_{j\\in \\cV}\\Upsilon\\rho^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert,\n\\end{align}\nfor any $\\bp\\in\\mathbb{P}_{\\epsilon}$ and $\\bq\\in\\mathbb{R}^{\\bl}$. Finally, since we have chosen $\\mathbb{P}_\\epsilon$ to be convex, for any $\\bp,\\bp'\\in\\mathbb{P}_{\\epsilon}$, the line segment between $\\bp,\\bp'$ is within $\\mathbb{P}_{\\epsilon}$. Thus, we have:\n\\begin{align*}\n  \\Vert z^\\dag_i(\\bp)-z^\\dag_i(\\bp')\\Vert\n  % &\\leq\\Vert \\int_{0}^1 D_{\\bp'-\\bp}z^\\dag_i ((1-t)\\bp+t\\bp')dt\\Vert\\\\\n    &\\leq \\int_{0}^1\\Vert D_{\\bp'-\\bp}z^\\dag_i ((1-t)\\bp+t\\bp')\\Vert dt \\leq \\sum_{j\\in \\cV}\\Upsilon\\rho^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert p_j-p'_j\\Vert,\n\\end{align*}\nwhere the first inequality is from Newton-Leibniz and triangle inequality for integrals, and the last inequality follows from \\eqref{eqn:main-2}.\n\\end{proof}","start":53851,"end":58212}]},"nodes":[{"id":"S001","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"向量矩阵与谱记号","source_units":[{"raw_tex":"{\\it Notation:} The set of real numbers and the set of integers are denoted by $\\mathbb{R}$ and $\\mathbb{I}$, respectively. We define $\\mathbb{I}_{A}:=\\mathbb{I}\\cap A$, where $A$ is a set; $\\mathbb{I}_{>0}:=\\mathbb{I}\\cap(0,\\infty)$; $\\mathbb{I}_{\\geq 0}:=\\mathbb{I}\\cap[0,\\infty)$; $\\mathbb{R}_{>0}:=(0,\\infty)$; and $\\mathbb{R}_{\\geq 0}:=[0,\\infty)$. For $A\\subseteq X$, $f:X\\rightarrow Y$, and $x\\in X$, where $X$ and $Y$ are linear spaces, $A+x := \\{x'+x:x'\\in A\\}$ and $f(X):=\\{f(x)\\in Y:x\\in X\\}$. Vectors are treated as column vectors. We use the syntax: $[M_1;\\cdots;M_n]:=[M_1^\\top\\,\\cdots\\,M_n^\\top]^\\top$; $\\{M_i\\}_{i\\in U}:=[M_{i_1}; \\cdots; M_{i_{m}}]$; $\\{M_{i,j}\\}_{i\\in U,j\\in V}:=\\{\\{M_{i,j}^\\top\\}_{j\\in V}^\\top\\}_{i\\in U}$, where, $U=\\{i_1<\\cdots<i_{m}\\}$ and $V=\\{j_1<\\cdots<j_{n}\\}$ are strictly ordered sets. Furthermore, $v[i]$ is the $i$-th component of vector $v$; $M[i,j]$ is the $(i,j)$-th component of matrix $M$; $v[I]:=\\{v[i]\\}_{i\\in I}$; $M[I,J]:=\\{M[i,j]\\}_{i\\in I,j\\in J}$. For a function $\\phi:\\mathbb{R}^{n}\\rightarrow \\mathbb{R}$ and variable vectors $y\\in\\mathbb{R}^{p}$, $z\\in\\mathbb{R}^{q}$, $\\nabla^2_{yz}\\phi(x):=\\{\\frac{\\partial^2}{\\partial y[i]\\partial z[j]}\\phi(x)\\}_{i\\in\\mathbb{I}_{[1,p]},j\\in\\mathbb{I}_{[1,q]}}$. For a vector function $\\varphi:\\mathbb{R}^{n}\\rightarrow \\mathbb{R}^m$ and a variable vector $w\\in\\mathbb{R}^{s}$, $\\nabla_{w}\\varphi(x):=\\{\\frac{\\partial}{\\partial w[j]}\\varphi(x)[i]\\}_{i\\in\\mathbb{I}_{[1,m]},j\\in\\mathbb{I}_{[1,s]}}$. We use the shorthand notation $\\nabla^2\\phi(x):=\\nabla^2_{xx}\\phi(x)$, $\\nabla\\varphi(x):=\\nabla_x\\varphi(x)$. Vector 2-norms and induced 2-norms of matrices are denoted by $\\Vert\\cdot\\Vert$. For matrices $A$ and $B$, $A\\succ(\\succeq) B$ indicates that $A-B$ is positive (semi)-definite while $A>(\\geq) B$ denotes a componentwise inequality. The identity matrix is denoted as $\\bI$ and the zero matrix or vector are denoted as $\\bzero$. Specific notations will be introduced at the first appearance. {\\blue We denote the largest and smallest eigenvalues of a symmetric matrix and non-trivial singular values of general matrices by $\\olambda(\\cdot)$, $\\ulambda(\\cdot)$, $\\osigma(\\cdot)$, and $\\usigma(\\cdot)$}.\n\n","start":13210,"end":15419}]},{"id":"S002","semantic_type":"problem_setting","kind":"problem_setting","labels":["eqn:prob-eq","eqn:prob-ineq","eqn:prob-obj"],"declared_external":false,"source_only":false,"title":"闭邻域耦合非线性规划","source_units":[{"raw_tex":"{\\it Settings:} We study gsNLPs of the following form: \n\\begin{subequations}\\label{eqn:prob}\n  \\begin{align}\n    \\min_{\\{x_i\\in\\mathbb{R}^{r_i}\\}_{i\\in \\cV}}\\;& \\sum_{i\\in \\cV}f_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]})\\label{eqn:prob-obj}\\\\\n    \\st\\;& c^E_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]}) = 0,\\;i\\in \\cV,\\;(y^E_i)\\label{eqn:prob-eq}\\\\\n                                                  &c^I_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]}) \\geq 0,\\;i\\in \\cV,\\;(y^I_i)\\label{eqn:prob-ineq},\n  \\end{align}\n\\end{subequations}\nwhere $\\cG=(\\cV,\\cE)$ is an undirected graph, $\\cV$ is a strictly ordered node set, $\\cE\\subseteq \\{\\{i,j\\}\\subseteq \\cV: i\\neq j\\}$ is the edge set, and $\\cN_\\cG[i]:=\\{j\\in \\cV:\\{i,j\\}\\in \\cE\\}\\cup\\{i\\}$ denotes the closed neighborhood\\footnote{Note that closed neighborhood is different from the standard notion of neighborhood (also called open neighborhood), defined as $\\cN_\\cG(i):=\\{j\\in \\cV:\\{i,j\\}\\in \\cE\\}$. The closed neighborhood of a node contains itself whereas the open neighborhood does not.} of $i\\in \\cV$. For each node $i\\in \\cV$, $x_i\\in\\mathbb{R}^{r_i}$ is the primal variable; $p_i\\in\\mathbb{R}^{l_i}$ is the data; $f_i:\\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{r_j}\\times\\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{l_j}\\rightarrow \\mathbb{R}$ is the objective function; $c^E_i:\\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{r_j}\\times \\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{l_j}\\rightarrow \\mathbb{R}^{m^E_i}$ is the equality constraint function; $c^I_i:\\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{r_j}\\times \\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{l_j}\\rightarrow \\mathbb{R}^{m^I_i}$ is the inequality constraint function; and $y^E_i\\in\\mathbb{R}^{m^E_i}$ and $y^I_i\\in\\mathbb{R}^{m^I_i}$ are the dual variables associated with \\eqref{eqn:prob-eq} and \\eqref{eqn:prob-ineq}. The objective and constraint functions of node $i$ depend on the variables $\\{x_j\\}_{j\\in \\cN_\\cG[i]}$ and data $\\{p_j\\}_{j\\in \\cN_\\cG[i]}$ of the neighboring nodes. That is, node $i$ is coupled algebraically only to its neighbors, and the topology of such connectivity is dictated by $\\cG$.\n","start":15419,"end":17527}]},{"id":"S003","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"整体规划与节点原对偶变量","source_units":[{"raw_tex":"\n{\\blue\n  We can express \\eqref{eqn:prob} as a standard NLP of the following form:\n \\begin{subequations}\n   \\begin{align}\n     P(\\bp):\\;\\min_{\\bx}\\;&\\bff(\\bx;\\bp)\\\\\n     \\st\\;&\\bc^E(\\bx;\\bp)=\\bzero,\\; (\\by^E)\\\\\n                          &\\bc^I(\\bx;\\bp)\\geq \\bzero,\\;\\;(\\by^I).\n   \\end{align}\n \\end{subequations}\n Here, we compact the notation as: $\\bx:=\\{x_i\\}_{i\\in \\cV}$; $\\by^E:=\\{y^E_i\\}_{i\\in \\cV}$; $\\by^I:=\\{y^I_i\\}_{i\\in \\cV}$; $\\bp:=\\{p_i\\}_{i\\in \\cV}$; $\\br=\\sum_{i\\in \\cV} r_i$; $\\bm^E=\\sum_{i\\in \\cV} m^E_i$; $\\bm^I=\\sum_{i\\in \\cV} m^I_i$; $\\bl = \\sum_{i\\in \\cV}l_i$; $\\bff(\\bx;\\bp):=\\sum_{i\\in \\cV}f_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]})$; $\\bc^E(\\bx;\\bp):=\\{c^E_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]})\\}_{i\\in \\cV}$; and $\\bc^I(\\bx;\\bp):=\\{c^I_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]})\\}_{i\\in \\cV}$. We also define $c_i(\\cdot):=[c^E_i(\\cdot);c^I_i(\\cdot)]$; $y_i:=[y^E_i;y^I_i]$; $z_i:=[x_i;y_i]$; $m_i=m^E_i+m^I_i$; $n_i=r_i+m_i$; $\\by:=\\{y_i\\}_{i\\in \\cV}$; $\\bz:=\\{z_i\\}_{i\\in \\cV}$;  $\\bm=\\sum_{i\\in \\cV} m_i$; $\\bn=\\sum_{i\\in \\cV} n_i$; and $\\bc(\\bx;\\bp):=\\{c_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]})\\}_{i\\in \\cV}$.\n}\n","start":17527,"end":18723}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["def:distance"],"declared_external":false,"source_only":false,"title":"图距离与直径","source_units":[{"raw_tex":"\\begin{definition}[Graph Distance and Diameter]\\label{def:distance}\n  The distance $d_\\cG(i,j)$ between nodes $i,j\\in \\cV$ on graph $\\cG=(\\cV,\\cE)$ is the number of edges in the shortest path connecting them; {if a shortest path does not exist, the distance is $+\\infty$}. Furthermore, the diameter $D_\\cG$ of $\\cG$ is the longest distance between any pair of nodes in $\\cV$.\n\\end{definition}\n","start":22924,"end":23317}]},{"id":"S005","semantic_type":"proposition","kind":"proposition","labels":["prop:metric"],"declared_external":true,"declaration_text":["原始 prerequisite list 明确许可 B0013 的标准最短路径性质，并接受作者省略其证明。"],"source_only":false,"title":"最短路径距离性质","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:metric}\n  The distance $d_\\cG:\\cV\\times \\cV \\rightarrow \\mathbb{I}_{\\geq 0}{\\cup\\{\\infty\\}}$ is a metric on $\\cV$; that is, (a) $d_\\cG(i,j)\\geq 0$ for any $i,j\\in \\cV$; (b) $i=j$ if and only if $d_\\cG(i,j)=0$; (c) $d_\\cG(i,j)=d_\\cG(j,i)$ for any $i,j\\in \\cV$; (d) $d_\\cG(i,j)\\leq d_\\cG(i,k)+d_\\cG(k,j)$ for any $i,j,k\\in \\cV$.\n\\end{proposition}\nThe proof of this result is straightforward and thus omitted. {Note that the shortest path does not exist when the graph is not connected. } \n","start":23317,"end":23835}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"节点分块与空分区约定","source_units":[{"raw_tex":"\n{A graph-structured matrix $X\\in\\mathbb{R}^{m\\times n}$ is indexed by a graph $\\cG=(\\cV,\\cE)$ and a pair of index set families $\\cI:=\\{I_{i}\\}_{i\\in \\cV}$ and $\\cJ:=\\{J_{i}\\}_{i\\in \\cV}$ that partition\\footnote{In this paper, we call a family $\\{X_1,\\cdots,X_k\\}$ of subsets of $X$ to be a partition if $\\bigcup_{k=1}^K X_k=X$ and $X_1,\\cdots,X_k$ are disjoint; here, {\\it we allow $X_k$ to be empty sets}. Note that this differs from the standard definition of a partition, where the nonemptiness of $X_k$ is enforced. This modification allows us to handle nodes with empty variables, constraints, or data, which is useful in many settings.} $\\mathbb{I}_{[1,m]}$ and $\\mathbb{I}_{[1,n]}$, respectively. We refer to $X_{[i][j]}$ as the $[i][j]$-block of matrix $X$, which can later be associated with the node pair $(i,j)$. It should be noted that the short-hand notation $X_{[i][j]}:=X[I_i,J_j]$ assumes that the proper graph and index set families $(\\cG,\\cI,\\cJ)$ are introduced or can be inferred from the context. We are now ready to introduce the concept of graph-induced {\\it bandwidth}.} \n\n","start":23835,"end":24933}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":["def:bandwidth"],"declared_external":false,"source_only":false,"title":"图诱导带宽","source_units":[{"raw_tex":"\\begin{definition}[Graph-Induced Bandwidth]\\label{def:bandwidth}\n  Consider a matrix $X\\in\\mathbb{R}^{m\\times n}$, a graph $\\cG=(\\cV,\\cE)$, and index sets $\\cI=\\{I_i\\}_{i\\in \\cV}$, $\\cJ=\\{J_i\\}_{i\\in \\cV}$ that partition $\\mathbb{I}_{[1,m]}$ and $\\mathbb{I}_{[1,n]}$, respectively. Matrix $X$ is said to have bandwidth $B$ induced by $(\\cG,\\cI,\\cJ)$, if $B$ is the smallest nonnegative integer such that $X_{[i][j]}=\\bzero$ for any $i,j\\in \\cV$ with $d_\\cG(i,j)> B$, where $X_{[i][j]}:=X[I_i,J_j]$.\n\\end{definition}\n","start":24933,"end":25449}]},{"id":"S008","semantic_type":"lemma","kind":"lemma","labels":["lem:bandwidth"],"declared_external":false,"source_only":false,"title":"带宽运算法则","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:bandwidth}\n  Consider $X\\in\\mathbb{R}^{m\\times n}$ with bandwidth not greater than $B_X$ induced by $(\\cG,\\cI,\\cJ)$; we have that:\n  (a) $X^\\top$ has bandwidth not greater than $B_X$ induced by $(\\cG,\\cJ,\\cI)$;\n  (b) if $Y\\in\\mathbb{R}^{m\\times n}$ has bandwidth not greater than $B_Y$ induced by $(\\cG,\\cI,\\cJ)$, then $X+Y$ has bandwidth not greater than $\\max(B_X,B_Y)$ induced by $(\\cG,\\cI,\\cJ)$;\n  (c) if $W\\in\\mathbb{R}^{n\\times k}$ has bandwidth not greater than $B_W$ induced by $(\\cG,\\cJ,\\cK)$, then $XW$ has bandwidth not greater than $B_X+B_W$ induced by $(\\cG,\\cI,\\cK)$.\n\\end{lemma}","start":27298,"end":27915}]},{"id":"S009","semantic_type":"lemma","kind":"lemma","labels":["lem:bandwidth"],"declared_external":false,"source_only":false,"title":"转置带宽上界","source_units":[{"raw_tex":"  (a) $X^\\top$ has bandwidth not greater than $B_X$ induced by $(\\cG,\\cJ,\\cI)$;\n","start":27453,"end":27533}]},{"id":"S010","semantic_type":"lemma","kind":"lemma","labels":["lem:bandwidth"],"declared_external":false,"source_only":false,"title":"矩阵和的带宽上界","source_units":[{"raw_tex":"  (b) if $Y\\in\\mathbb{R}^{m\\times n}$ has bandwidth not greater than $B_Y$ induced by $(\\cG,\\cI,\\cJ)$, then $X+Y$ has bandwidth not greater than $\\max(B_X,B_Y)$ induced by $(\\cG,\\cI,\\cJ)$;\n","start":27533,"end":27722}]},{"id":"S011","semantic_type":"lemma","kind":"lemma","labels":["lem:bandwidth"],"declared_external":false,"source_only":false,"title":"矩阵积的带宽上界","source_units":[{"raw_tex":"  (c) if $W\\in\\mathbb{R}^{n\\times k}$ has bandwidth not greater than $B_W$ induced by $(\\cG,\\cJ,\\cK)$, then $XW$ has bandwidth not greater than $B_X+B_W$ induced by $(\\cG,\\cI,\\cK)$.\n","start":27722,"end":27904}]},{"id":"S012","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"转置带宽的作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of (a)] We have that $(X^\\top)_{[i][j]}=(X^\\top){[J_i,I_j]}=(X[I_j,J_i])^\\top=(X_{[j][i]})^\\top$. From the assumption that $X$ has bandwidth not greater than $B_X$ and Proposition \\ref{prop:metric}(c), $(X^\\top)_{[i][j]}=\\bzero$ if $d_\\cG(i,j)>B_X$; therefore, $X$ has bandwidth not greater than $B_X$, and induced by $(\\cG,\\cJ,\\cI)$.\n\\end{proof}","start":27916,"end":28282}]},{"id":"S013","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"加法带宽的作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of (b)] We have that $X_{[i][j]}=\\bzero$ and $Y_{[i][j]}=\\bzero$ if $d_\\cG(i,j)>\\max(B_X,B_Y)$, which yields $(X+Y)_{[i][j]}=\\bzero$ if $d_\\cG(i,j)>\\max(B_X,B_Y)$. Thus, $X+Y$ has bandwidth not greater than $\\max(B_X,B_Y)$, and induced by $(\\cG,\\cI,\\cJ)$.\n\\end{proof}","start":28283,"end":28570}]},{"id":"S014","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"乘法带宽的作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of (c)] If $d_\\cG(i,j)>B_X+B_W$, from Proposition \\ref{prop:metric}(d) we have that for any $k\\in \\cV$, $d_\\cG(i,k)>B_X$ or $d_\\cG(j,k)>B_W$. Thus, if $d_\\cG(i,j)>B_X+B_W$,  we have $(XW)_{[i][j]}=\\sum_{k\\in \\cV} X_{[i][k]}W_{[k][j]}= \\bzero$ (where the first equality comes from the block matrix multiplication law and the second equality comes from observing that $d_\\cG(i,k)>B_X$ or $d_\\cG(j,k)>B_W$). Therefore, $XW$ has bandwidth not greater than $B_X+B_W$, and induced by $(\\cG,\\cI,\\cK)$.  \n\\end{proof}","start":28571,"end":29099}]},{"id":"S015","semantic_type":"theorem","kind":"theorem","labels":["eqn:inv","thm:inv"],"declared_external":false,"source_only":false,"title":"逆矩阵乘积的分块衰减","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm:inv}\n  Consider a nonsingular matrix $X\\in\\mathbb{R}^{n\\times n}$ with bandwidth not greater than $B_X\\geq 1$ induced by $(\\cG,\\cK,\\cP)$, $Y\\in\\mathbb{R}^{n\\times m}$ with bandwidth not greater than $B_Y$ induced by $(\\cG, \\cK,\\cJ)$, and $W\\in\\mathbb{R}^{\\ell\\times n}$ with bandwidth not greater than $B_W$ induced by $(\\cG, \\cI,\\cP)$; for constants $\\osigma_X\\geq \\osigma(X)$, $\\osigma_Y\\geq \\osigma(Y)$, $\\osigma_W\\geq \\osigma(W)$, and $0<\\usigma_X\\leq \\usigma(X)$, the following holds:\n  \\begin{align}\\label{eqn:inv}\n    \\Vert (WX^{-1}Y)_{[i][j]}\\Vert \\leq\n    \\frac{\\osigma_X\\osigma_Y\\osigma_W}{\\usigma_X^2}\\left(\\frac{\\osigma_X^2-\\usigma_X^2}{\\osigma_X^2+\\usigma_X^2}\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)-B_X-B_Y-B_W}{2B_X}\\right\\rceil_+} ,\\quad i,j\\in \\cV,\n  \\end{align}\n  where $(WX^{-1}Y)_{[i][j]}:=(WX^{-1}Y)[I_i,J_j]$ and $\\lceil\\cdot\\rceil_+$ is the smallest non-negative integer that is greater than or equal to the argument.\n\\end{theorem}","start":29434,"end":30412}]},{"id":"S016","semantic_type":"assumption","kind":"assumption","labels":["thm:inv"],"declared_external":false,"source_only":false,"title":"逆衰减定理的矩阵条件","source_units":[{"raw_tex":"  Consider a nonsingular matrix $X\\in\\mathbb{R}^{n\\times n}$ with bandwidth not greater than $B_X\\geq 1$ induced by $(\\cG,\\cK,\\cP)$, $Y\\in\\mathbb{R}^{n\\times m}$ with bandwidth not greater than $B_Y$ induced by $(\\cG, \\cK,\\cJ)$, and $W\\in\\mathbb{R}^{\\ell\\times n}$ with bandwidth not greater than $B_W$ induced by $(\\cG, \\cI,\\cP)$; for constants $\\osigma_X\\geq \\osigma(X)$, $\\osigma_Y\\geq \\osigma(Y)$, $\\osigma_W\\geq \\osigma(W)$, and $0<\\usigma_X\\leq \\usigma(X)$, the following holds:\n","start":29465,"end":29950}]},{"id":"S017","semantic_type":"proof","kind":"proof","labels":["eqn:Hbound","eqn:factor","eqn:key-1","eqn:key-1-c"],"declared_external":false,"source_only":false,"title":"逆衰减定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n  By definition of singular values and the assumptions on $\\osigma_X$, $\\osigma_Y$ , $\\osigma_W$, and $\\usigma_X$, we have $\\usigma_X^2 \\bI \\preceq\\usigma(X)^2 \\bI \\preceq X X^\\top\\preceq  \\osigma(X)^2\\bI\\preceq\\osigma_X^2 \\bI $. From this, one can obtain:\n \\begin{align}\\label{eqn:factor}\n   \\frac{\\usigma_X^2-\\osigma_X^2}{\\usigma_X^2+\\osigma_X^2} \\bI\\preceq \\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top\n   \\preceq \\frac{-\\usigma_X^2+\\osigma_X^2}{\\usigma_X^2+\\osigma_X^2}\\bI.\n \\end{align}\n By Lemma \\ref{lem:bandwidth}(c), $XX^\\top$ has bandwidth not greater than $2B_X$ induced by $(\\cG,\\cK,\\cK)$. From the nonsingularity of $XX^\\top$, we obtain the following:\n \\begin{subequations}\\label{eqn:key-1}\n   \\begin{align}\n     WX^{-1}Y &= \\frac{2}{\\usigma_X^2+\\osigma_X^2} WX^\\top \\left(\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^{-1}Y\\\\\n   % &= \\frac{2}{\\usigma_X^2+\\osigma_X^2} WX^\\top \\left(\\bI-\\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)\\right)^{-1} Y\\\\\n     &= \\frac{2}{\\usigma_X^2+\\osigma_X^2}WX^\\top \\left(\\sum_{q=0}^\\infty \\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^q\\right)Y,\\label{eqn:key-1-c}\\\\\n     &=\\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=0}^\\infty WX^\\top\\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^q Y.\n \\end{align}\n \\end{subequations}\n Here the second equality follows from \\cite[Corollary 5.6.16]{horn2012matrix} and \\eqref{eqn:factor}, and the last equality follows from the fact that the series in \\eqref{eqn:key-1-c} converges (due to \\eqref{eqn:factor}). Furthermore, from Lemma \\ref{lem:bandwidth}, we see that $WX^\\top\\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^q Y$ has bandwidth not greater than $(2q+1)B_X +B_Y+B_W$, induced by $(\\cG,\\cI,\\cJ)$. By extracting submatrices $(\\cdot)[I_i,J_j]$ from \\eqref{eqn:key-1}, we obtain:\n  \\begin{align*}\n    (WX^{-1}Y)_{[i][j]}&=\\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=q_0(i,j)}^\\infty \\left(WX^\\top\\left(\\bI-\\frac{2XX^\\top}{\\usigma_X^2+\\osigma_X^2} \\right)^q Y\\right)_{[i][j]}\n  \\end{align*}\n  where $q_0(i,j):=\\left\\lceil \\frac{d_\\cG(i,j)-B_X-B_Y-B_W}{2B_X}\\right\\rceil_+$; the summation over $q=0,\\cdots, q_0(i,j)-1$ is zero because such $q$ satisfy $(2q+1)B_X+B_Y+B_W< d_\\cG(i,j)$. Using the triangle inequality and the fact that the matrix norm of a submatrix is smaller than that of the original matrix,\n  \\begin{align}\\label{eqn:Hbound}\n    \\Vert (WX^{-1}Y)_{[i][j]} \\Vert &\\leq \\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=q_0(i,j)}^\\infty \\left\\Vert WX^\\top\\left(\\bI-\\frac{2XX^\\top}{\\usigma_X^2+\\osigma_X^2}\\right)^q Y\\right\\Vert\\\\\n    &\\leq \\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=q_0(i,j)}^\\infty \\osigma_W\\osigma_X\\left(\\frac{\\osigma_X^2-\\usigma_X^2}{\\osigma_X^2+\\usigma_X^2}\\right)^q \\osigma_Y\\nonumber\\\\\n    &\\leq \\frac{\\osigma_X\\osigma_Y\\osigma_W}{\\usigma_X^2}\\left(\\frac{\\osigma_X^2-\\usigma_X^2}{\\osigma_X^2+\\usigma_X^2}\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)-B_X-B_Y-B_W}{2B_X}\\right\\rceil_+} .\\nonumber\n  \\end{align}\n  The second inequality follows from the submultiplicativity of the matrix norm and \\eqref{eqn:factor}; and the last inequality follows from the summation of geometric series. \n\\end{proof}","start":30413,"end":33580}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:factor","eqn:key-1"],"declared_external":false,"source_only":false,"title":"构造谱压缩界","source_units":[{"raw_tex":"  By definition of singular values and the assumptions on $\\osigma_X$, $\\osigma_Y$ , $\\osigma_W$, and $\\usigma_X$, we have $\\usigma_X^2 \\bI \\preceq\\usigma(X)^2 \\bI \\preceq X X^\\top\\preceq  \\osigma(X)^2\\bI\\preceq\\osigma_X^2 \\bI $. From this, one can obtain:\n \\begin{align}\\label{eqn:factor}\n   \\frac{\\usigma_X^2-\\osigma_X^2}{\\usigma_X^2+\\osigma_X^2} \\bI\\preceq \\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top\n   \\preceq \\frac{-\\usigma_X^2+\\osigma_X^2}{\\usigma_X^2+\\osigma_X^2}\\bI.\n \\end{align}\n","start":30427,"end":30917}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:key-1","eqn:key-1-c"],"declared_external":false,"source_only":false,"title":"逆矩阵的收敛级数表示","source_units":[{"raw_tex":" By Lemma \\ref{lem:bandwidth}(c), $XX^\\top$ has bandwidth not greater than $2B_X$ induced by $(\\cG,\\cK,\\cK)$. From the nonsingularity of $XX^\\top$, we obtain the following:\n \\begin{subequations}\\label{eqn:key-1}\n   \\begin{align}\n     WX^{-1}Y &= \\frac{2}{\\usigma_X^2+\\osigma_X^2} WX^\\top \\left(\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^{-1}Y\\\\\n   % &= \\frac{2}{\\usigma_X^2+\\osigma_X^2} WX^\\top \\left(\\bI-\\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)\\right)^{-1} Y\\\\\n     &= \\frac{2}{\\usigma_X^2+\\osigma_X^2}WX^\\top \\left(\\sum_{q=0}^\\infty \\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^q\\right)Y,\\label{eqn:key-1-c}\\\\\n     &=\\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=0}^\\infty WX^\\top\\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^q Y.\n \\end{align}\n \\end{subequations}\n Here the second equality follows from \\cite[Corollary 5.6.16]{horn2012matrix} and \\eqref{eqn:factor}, and the last equality follows from the fact that the series in \\eqref{eqn:key-1-c} converges (due to \\eqref{eqn:factor}). Furthermore, from Lemma \\ref{lem:bandwidth}, we see that $WX^\\top\\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^q Y$ has bandwidth not greater than $(2q+1)B_X +B_Y+B_W$, induced by $(\\cG,\\cI,\\cJ)$. By extracting submatrices $(\\cdot)[I_i,J_j]$ from \\eqref{eqn:key-1}, we obtain:\n","start":30917,"end":32245}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:key-1"],"declared_external":false,"source_only":false,"title":"带宽消去低阶块","source_units":[{"raw_tex":" Here the second equality follows from \\cite[Corollary 5.6.16]{horn2012matrix} and \\eqref{eqn:factor}, and the last equality follows from the fact that the series in \\eqref{eqn:key-1-c} converges (due to \\eqref{eqn:factor}). Furthermore, from Lemma \\ref{lem:bandwidth}, we see that $WX^\\top\\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^q Y$ has bandwidth not greater than $(2q+1)B_X +B_Y+B_W$, induced by $(\\cG,\\cI,\\cJ)$. By extracting submatrices $(\\cdot)[I_i,J_j]$ from \\eqref{eqn:key-1}, we obtain:\n  \\begin{align*}\n    (WX^{-1}Y)_{[i][j]}&=\\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=q_0(i,j)}^\\infty \\left(WX^\\top\\left(\\bI-\\frac{2XX^\\top}{\\usigma_X^2+\\osigma_X^2} \\right)^q Y\\right)_{[i][j]}\n  \\end{align*}\n  where $q_0(i,j):=\\left\\lceil \\frac{d_\\cG(i,j)-B_X-B_Y-B_W}{2B_X}\\right\\rceil_+$; the summation over $q=0,\\cdots, q_0(i,j)-1$ is zero because such $q$ satisfy $(2q+1)B_X+B_Y+B_W< d_\\cG(i,j)$. Using the triangle inequality and the fact that the matrix norm of a submatrix is smaller than that of the original matrix,\n","start":31730,"end":32770}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:Hbound","eqn:key-1"],"declared_external":false,"source_only":false,"title":"尾级数范数求和","source_units":[{"raw_tex":"  where $q_0(i,j):=\\left\\lceil \\frac{d_\\cG(i,j)-B_X-B_Y-B_W}{2B_X}\\right\\rceil_+$; the summation over $q=0,\\cdots, q_0(i,j)-1$ is zero because such $q$ satisfy $(2q+1)B_X+B_Y+B_W< d_\\cG(i,j)$. Using the triangle inequality and the fact that the matrix norm of a submatrix is smaller than that of the original matrix,\n  \\begin{align}\\label{eqn:Hbound}\n    \\Vert (WX^{-1}Y)_{[i][j]} \\Vert &\\leq \\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=q_0(i,j)}^\\infty \\left\\Vert WX^\\top\\left(\\bI-\\frac{2XX^\\top}{\\usigma_X^2+\\osigma_X^2}\\right)^q Y\\right\\Vert\\\\\n    &\\leq \\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=q_0(i,j)}^\\infty \\osigma_W\\osigma_X\\left(\\frac{\\osigma_X^2-\\usigma_X^2}{\\osigma_X^2+\\usigma_X^2}\\right)^q \\osigma_Y\\nonumber\\\\\n    &\\leq \\frac{\\osigma_X\\osigma_Y\\osigma_W}{\\usigma_X^2}\\left(\\frac{\\osigma_X^2-\\usigma_X^2}{\\osigma_X^2+\\usigma_X^2}\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)-B_X-B_Y-B_W}{2B_X}\\right\\rceil_+} .\\nonumber\n  \\end{align}\n  The second inequality follows from the submultiplicativity of the matrix norm and \\eqref{eqn:factor}; and the last inequality follows from the summation of geometric series. \n","start":32453,"end":33569}]},{"id":"S022","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"基准数据与基准解","source_units":[{"raw_tex":"We use $\\bz^\\star \\in\\mathbb{R}^{\\bn}$ to denote the primal-dual solution of $P(\\bp^\\star)$. We refer to $\\bp^\\star$ as the base data and $\\bz^\\star$ as the base solution. We denote the primal and dual components of $\\bz^\\star=\\{z^\\star_i\\}_{i\\in\\cV}=\\{[x^\\star_i;y^\\star_i]\\}_{i\\in\\cV}$ as $\\bx^\\star:=\\{x^\\star_i\\}_{i\\in\\cV}$ and $\\by^\\star=\\{y^\\star_i\\}_{i\\in\\cV}$, respectively. We say that $\\bz^\\star$ is a primal-dual solution if $\\bx^\\star$ satisfies the first-order optimality conditions with Lagrange multiplier $\\by^\\star$ (see \\cite{nocedal2006numerical}). We now make key assumptions that are necessary to establish our main results. \n\n","start":36376,"end":37024}]},{"id":"S023","semantic_type":"assumption","kind":"assumption","labels":["ass:c2"],"declared_external":false,"source_only":false,"title":"二阶连续可微假设","source_units":[{"raw_tex":"\\begin{assumption}[Twice Continuous Differentiability of Functions]\\label{ass:c2}\n  The functions $\\bff:\\mathbb{R}^{\\br}\\times \\mathbb{R}^{\\bl}\\rightarrow \\mathbb{R}$ and $\\bc:\\mathbb{R}^{\\br}\\times \\mathbb{R}^{\\bl}\\rightarrow \\mathbb{R}^{\\bm}$ are twice continuously differentiable in the neighborhood of $[\\bx^\\star;\\bp^\\star]$.\n\\end{assumption}\n\n","start":37024,"end":37373}]},{"id":"S024","semantic_type":"assumption","kind":"assumption","labels":["ass:regularity"],"declared_external":false,"source_only":false,"title":"基准解正则性假设","source_units":[{"raw_tex":"\\begin{assumption}[Regularity of Solution]\\label{ass:regularity}\n  The base solution $\\bz^\\star$ satisfies SSOSC and LICQ for $P(\\bp^\\star)$ with base data $\\bp^\\star$.\n\\end{assumption}\n","start":37373,"end":37559}]},{"id":"S025","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"拉格朗日函数与正则性含义","source_units":[{"raw_tex":"We recall that SSOSC requires positive definiteness of the reduced Hessian of the Lagrangian at $\\bz^\\star$. The reduced Hessian is the Hessian projected on the null space defined by equality constraints and active inequality constraints with nonzero duals. LICQ requires that the constraint Jacobian defined by equality and active inequality constraints are linearly independent at $\\bz^\\star$. These requirements are stated as: \n\\begin{align*}\n  ReH(\\nabla^2_{\\bx\\bx}\\cL(\\bz^\\star;\\bp^\\star),\\nabla_{\\bx}\\bc(\\bx^\\star;\\bp^\\star)[\\cA^0(\\bp^\\star),:]) &\\succ \\bzero\\tag{SSOSC}\\\\\n  \\usigma(\\nabla_{\\bx}\\bc(\\bx^\\star;\\bp^\\star)[\\cA^1(\\bp^\\star),:]) &> 0 \\tag{LICQ}.\n\\end{align*}\nHere, $\\cL(\\bz;\\bp) := \\bff(\\bx;\\bp)-\\by^\\top \\bc(\\bx;\\bp)$ is the Lagrange function, $ReH(H,A):=Z^\\top HZ$ is the reduced Hessian, where $Z$ is a null-space matrix of $A$, and\n","start":37559,"end":38413}]},{"id":"S026","semantic_type":"definition","kind":"definition","labels":["eqn:active-set-defs"],"declared_external":false,"source_only":false,"title":"基点活动约束集合","source_units":[{"raw_tex":"\\begin{align}\\label{eqn:active-set-defs}\n  \\cA^0(\\bp^\\star)&:=\\cA^E\\cup\\{i\\in\\cA^I:\\bc(\\bx^\\star)[i]=0,\\;\\by^\\star[i]\\neq 0\\}\\\\\n  \\cA^1(\\bp^\\star)&:=\\cA^E\\cup\\{i\\in\\cA^I:\\bc(\\bx^\\star)[i]=0\\},\n\\end{align}\nwhere $\\cA^E$ and $\\cA^I$ are the set of equality and inequality constraint indices. Hereafter, we call a solution of gsNLP regular if the solution satisfies SSOSC and LICQ. Note that SSOSC and LICQ are standard assumptions used in NLP sensitivity theory \\cite{robinson1980strongly}. We now state the classical sensitivity result.\n","start":38413,"end":38949}]},{"id":"S027","semantic_type":"lemma","kind":"lemma","labels":["eqn:AAQA","eqn:nlp-sensitivity-0","eqn:nlp-sensitivity-0-obj","eqn:one-sided-DD","lem:nlp-sensitivity"],"declared_external":false,"source_only":false,"title":"经典局部敏感性引理","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:nlp-sensitivity}\n  Under Assumptions \\ref{ass:c2}, \\ref{ass:regularity}, there exist neighborhoods $\\mathbb{P}\\subseteq\\mathbb{R}^{\\bl}$ of $\\bp^\\star$ and $\\bbZ\\subseteq\\mathbb{R}^{\\bn}$ of $\\bz^\\star$ and a continuous function $\\bz^\\dag:\\mathbb{P}\\rightarrow \\bbZ$ such that $\\bz^\\dag(\\bp)$ is a primal-dual solution of $P(\\bp)$ that satisfies SSOSC and LICQ. Furthermore, for any $\\bq:=\\{q_i\\}_{i\\in \\cV}\\in\\mathbb{R}^{\\bl}$, the one-sided directional derivative of $\\bz^\\dag(\\cdot)$ is given by:\n  \\begin{align*}\n    D_{\\bq} \\bz^\\dag(\\bp)&:=\\lim_{h\\searrow 0}\\frac{\\bz^\\dag(\\bp+\\bq h)-\\bz^\\dag(\\bp)}{h}; \\label{eqn:one-sided-DD}\n  \\end{align*}\n  with $D_{\\bq}\\bx^\\dag(\\bp) = \\bxi^\\dag(\\bp,\\bq)$, $D_{\\bq}\\by^\\dag(\\bp,\\bq) = \\bbeta^\\dag(\\bp,\\bq)$, where $(\\bxi^\\dag(\\bp,\\bq),\\bbeta^\\dag(\\bp,\\bq)[\\cA^1(\\bp)])$ is a primal-dual solution of the quadratic program:\n  \\begin{subequations}\\label{eqn:nlp-sensitivity-0}\n    \\begin{align}\n      QP(\\bp,\\bq):\\quad\\min_{\\bxi}\\;&\\frac{1}{2}\\bxi^\\top\\bQ(\\bp)\\bxi + \\bxi^\\top\\bS(\\bp)\\bq,\\label{eqn:nlp-sensitivity-0-obj}\\\\\n      \\st\\;&\\bA(\\bp)[i,:]^\\top\\bxi + \\bB(\\bp)[i,:]\\bq = 0,\\;i\\in \\cA^0(\\bp),\\; (\\bbeta[i])\\\\\n                   &\\bA(\\bp)[i,:]^\\top\\bxi + \\bB(\\bp)[i,:]\\bq \\geq 0,\\;i\\in \\cA^1(\\bp)\\setminus \\cA^0(\\bp),\\; (\\bbeta[i])\n    \\end{align}      \n  \\end{subequations}\n  and $\\bbeta^\\dag(\\bp,\\bq)[\\mathbb{I}_{[1,\\bm]}\\setminus\\cA^1(\\bp)]=\\bzero$ (i.e., the free dual variables are fixed to zero); here,\n  \\begin{subequations}\\label{eqn:AAQA}\n    \\begin{align}\n      \\cA^0(\\bp)&:=\\cA^E\\cup\\{i\\in\\cA^I:\\bc(\\bx^\\dag(\\bp))[i]=0,\\;\\by^\\dag(\\bp)[i]\\neq 0\\}\\\\\n      \\cA^1(\\bp)&:=\\cA^E\\cup\\{i\\in\\cA^I:\\bc(\\bx^\\dag(\\bp))[i]=0\\}\\\\\n      \\bQ(\\bp)&:=\\nabla_{\\bx\\bx}\\cL(\\bz^\\dag(\\bp);\\bp);\\quad \\bS(\\bp):=\\nabla_{\\bx\\bp}\\cL(\\bz^\\dag(\\bp);\\bp)\\\\\n      \\bA(\\bp)&:=\\nabla_{\\bx}\\bc(\\bx^\\dag(\\bp);\\bp);\\quad \\bB(\\bp):=\\nabla_{\\bp}\\bc(\\bx^\\dag(\\bp);\\bp).\n    \\end{align}\n  \\end{subequations}\n  Moreover, a unique primal-dual solution of $QP(\\bp,\\bq)$ exists for any $\\bp\\in\\mathbb{P}$ and $\\bq\\in\\mathbb{R}^{\\bl}$; thus, $\\bxi^\\dag(\\cdot,\\cdot)$ and $\\bbeta^\\dag(\\cdot,\\cdot)$ are well-defined.\n\\end{lemma}","start":38950,"end":41114}]},{"id":"S028","semantic_type":"definition","kind":"definition","labels":["eqn:AAQA","eqn:nlp-sensitivity-0","lem:nlp-sensitivity"],"declared_external":false,"source_only":false,"title":"移动活动集合与 QP 系数","source_units":[{"raw_tex":"  \\begin{subequations}\\label{eqn:AAQA}\n    \\begin{align}\n      \\cA^0(\\bp)&:=\\cA^E\\cup\\{i\\in\\cA^I:\\bc(\\bx^\\dag(\\bp))[i]=0,\\;\\by^\\dag(\\bp)[i]\\neq 0\\}\\\\\n      \\cA^1(\\bp)&:=\\cA^E\\cup\\{i\\in\\cA^I:\\bc(\\bx^\\dag(\\bp))[i]=0\\}\\\\\n      \\bQ(\\bp)&:=\\nabla_{\\bx\\bx}\\cL(\\bz^\\dag(\\bp);\\bp);\\quad \\bS(\\bp):=\\nabla_{\\bx\\bp}\\cL(\\bz^\\dag(\\bp);\\bp)\\\\\n      \\bA(\\bp)&:=\\nabla_{\\bx}\\bc(\\bx^\\dag(\\bp);\\bp);\\quad \\bB(\\bp):=\\nabla_{\\bp}\\bc(\\bx^\\dag(\\bp);\\bp).\n    \\end{align}\n  \\end{subequations}\n","start":40430,"end":40900}]},{"id":"S029","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"敏感性引理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n  The results in \\cite[Theorem 2G.8]{dontchev2009implicit} ensure semidifferentiability (which guarantees continuity) of the solution of the first-order optimality conditions for $P(\\bp)$ over a certain neighborhood of $\\bp^\\star$. Furthermore, \\cite[Theorem 2G.9]{dontchev2009implicit} establishes that over a certain neighborhood $\\mathbb{P}$, the solution mapping of the generalized equation (GE) satisfies SSOSC and LICQ; that is, within $\\mathbb{P}$, the solution mapping for the GE is the solution mapping for $P(\\bp)$ at which SSOSC and LICQ are satisfied. Moreover, by \\cite[Theorem 2G.8]{dontchev2009implicit}, the one-sided directional derivative of the solution mapping for the GE (which exists for any direction $\\bq\\in\\mathbb{R}^{\\bl}$ by semidifferentiability \\cite[Theorem 2D.1]{dontchev2009implicit}) can be evaluated by using the linearized GE. The linear GE corresponds to the first-order optimality conditions for $QP(\\bp,\\bq)$ (see \\cite[Equation (35)]{dontchev2009implicit}); here, the first-order conditions are necessary and sufficient conditions for the optimality due to the convexity $QP(\\bp,\\bq)$ (guaranteed by SSOSC and LICQ of $\\bz^\\star$ for the original problem). As such, the one-sided directional derivative of the solution mapping for $P(\\bp)$ can be evaluated by solving $QP(\\bp,\\bq)$. Finally, the strong regularity of the GE at $\\bz^\\star$ guarantees that there exists a unique solution of the linearized GE, which in turn guarantees the existence of a unique solution of $QP(\\bp,\\bq)$.  \n\\end{proof}","start":41310,"end":42862}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"连续解映射与邻域正则性","source_units":[{"raw_tex":"  The results in \\cite[Theorem 2G.8]{dontchev2009implicit} ensure semidifferentiability (which guarantees continuity) of the solution of the first-order optimality conditions for $P(\\bp)$ over a certain neighborhood of $\\bp^\\star$. Furthermore, \\cite[Theorem 2G.9]{dontchev2009implicit} establishes that over a certain neighborhood $\\mathbb{P}$, the solution mapping of the generalized equation (GE) satisfies SSOSC and LICQ; that is, within $\\mathbb{P}$, the solution mapping for the GE is the solution mapping for $P(\\bp)$ at which SSOSC and LICQ are satisfied. Moreover, by \\cite[Theorem 2G.8]{dontchev2009implicit}, the one-sided directional derivative of the solution mapping for the GE (which exists for any direction $\\bq\\in\\mathbb{R}^{\\bl}$ by semidifferentiability \\cite[Theorem 2D.1]{dontchev2009implicit}) can be evaluated by using the linearized GE. The linear GE corresponds to the first-order optimality conditions for $QP(\\bp,\\bq)$ (see \\cite[Equation (35)]{dontchev2009implicit}); here, the first-order conditions are necessary and sufficient conditions for the optimality due to the convexity $QP(\\bp,\\bq)$ (guaranteed by SSOSC and LICQ of $\\bz^\\star$ for the original problem). As such, the one-sided directional derivative of the solution mapping for $P(\\bp)$ can be evaluated by solving $QP(\\bp,\\bq)$. Finally, the strong regularity of the GE at $\\bz^\\star$ guarantees that there exists a unique solution of the linearized GE, which in turn guarantees the existence of a unique solution of $QP(\\bp,\\bq)$.  \n","start":41324,"end":42851}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"方向导数与线性化广义方程","source_units":[{"raw_tex":"  The results in \\cite[Theorem 2G.8]{dontchev2009implicit} ensure semidifferentiability (which guarantees continuity) of the solution of the first-order optimality conditions for $P(\\bp)$ over a certain neighborhood of $\\bp^\\star$. Furthermore, \\cite[Theorem 2G.9]{dontchev2009implicit} establishes that over a certain neighborhood $\\mathbb{P}$, the solution mapping of the generalized equation (GE) satisfies SSOSC and LICQ; that is, within $\\mathbb{P}$, the solution mapping for the GE is the solution mapping for $P(\\bp)$ at which SSOSC and LICQ are satisfied. Moreover, by \\cite[Theorem 2G.8]{dontchev2009implicit}, the one-sided directional derivative of the solution mapping for the GE (which exists for any direction $\\bq\\in\\mathbb{R}^{\\bl}$ by semidifferentiability \\cite[Theorem 2D.1]{dontchev2009implicit}) can be evaluated by using the linearized GE. The linear GE corresponds to the first-order optimality conditions for $QP(\\bp,\\bq)$ (see \\cite[Equation (35)]{dontchev2009implicit}); here, the first-order conditions are necessary and sufficient conditions for the optimality due to the convexity $QP(\\bp,\\bq)$ (guaranteed by SSOSC and LICQ of $\\bz^\\star$ for the original problem). As such, the one-sided directional derivative of the solution mapping for $P(\\bp)$ can be evaluated by solving $QP(\\bp,\\bq)$. Finally, the strong regularity of the GE at $\\bz^\\star$ guarantees that there exists a unique solution of the linearized GE, which in turn guarantees the existence of a unique solution of $QP(\\bp,\\bq)$.  \n","start":41324,"end":42851}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"强正则性与导数问题唯一性","source_units":[{"raw_tex":"  The results in \\cite[Theorem 2G.8]{dontchev2009implicit} ensure semidifferentiability (which guarantees continuity) of the solution of the first-order optimality conditions for $P(\\bp)$ over a certain neighborhood of $\\bp^\\star$. Furthermore, \\cite[Theorem 2G.9]{dontchev2009implicit} establishes that over a certain neighborhood $\\mathbb{P}$, the solution mapping of the generalized equation (GE) satisfies SSOSC and LICQ; that is, within $\\mathbb{P}$, the solution mapping for the GE is the solution mapping for $P(\\bp)$ at which SSOSC and LICQ are satisfied. Moreover, by \\cite[Theorem 2G.8]{dontchev2009implicit}, the one-sided directional derivative of the solution mapping for the GE (which exists for any direction $\\bq\\in\\mathbb{R}^{\\bl}$ by semidifferentiability \\cite[Theorem 2D.1]{dontchev2009implicit}) can be evaluated by using the linearized GE. The linear GE corresponds to the first-order optimality conditions for $QP(\\bp,\\bq)$ (see \\cite[Equation (35)]{dontchev2009implicit}); here, the first-order conditions are necessary and sufficient conditions for the optimality due to the convexity $QP(\\bp,\\bq)$ (guaranteed by SSOSC and LICQ of $\\bz^\\star$ for the original problem). As such, the one-sided directional derivative of the solution mapping for $P(\\bp)$ can be evaluated by solving $QP(\\bp,\\bq)$. Finally, the strong regularity of the GE at $\\bz^\\star$ guarantees that there exists a unique solution of the linearized GE, which in turn guarantees the existence of a unique solution of $QP(\\bp,\\bq)$.  \n","start":41324,"end":42851}]},{"id":"S033","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"QP 计算敏感性的解释","source_units":[{"raw_tex":"Under Lemma \\ref{lem:nlp-sensitivity}, the rate of change $D_{\\bq}\\bz^\\dag(\\bp)$ of the primal-dual solution of $P(\\bp)$ (for a given direction $\\bq$) can be quantified by using the solution of $QP(\\bp,\\bq)$. For given $\\bp$ and $\\bq$, the parameters in $QP(\\bp,\\bq)$ can be evaluated explicitly and thus its solution can be calculated. As such, Lemma \\ref{lem:nlp-sensitivity} provides a computational procedure to evaluate primal-dual solution sensitivity.\n\n","start":42864,"end":43324}]},{"id":"S034","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:kkt-compl","eqn:kkt-du","eqn:kkt-pr-1","eqn:kkt-pr-2"],"declared_external":false,"source_only":false,"title":"方向 QP 的最优性条件","source_units":[{"raw_tex":"We now further analyze $QP(\\bp,\\bq)$; its first-order conditions are as follows:\n\\begin{subequations}\\label{eqn:kkt}\n  \\begin{align}\n    \\bQ(\\bp)\\bxi + \\bS(\\bp)\\bq + \\bA(\\bp)[\\cA^{1}(\\bp),:]^\\top \\bbeta &= \\bzero\\label{eqn:kkt-du}\\\\\n    \\bA(\\bp)[i,:]\\bxi + \\bB(\\bp)[i,:]\\bq &= 0,\\; i\\in\\cA^0(\\bp)\\label{eqn:kkt-pr-1}\\\\\n    \\bA(\\bp)[i,:]\\bxi + \\bB(\\bp)[i,:]\\bq &\\geq 0,\\;i\\in\\cA^1(\\bp)\\setminus\\cA^0(\\bp)\\label{eqn:kkt-pr-2}\\\\\n    \\bbeta[i] &\\geq 0,\\;i\\in\\cA^1(\\bp)\\setminus\\cA^0(\\bp)\\\\\n    \\left(\\bA(\\bp)[i,:]\\bxi + \\bB(\\bp)[i,:]\\bq \\right)\\bbeta[i]&=0,\\; i\\in\\cA^1(\\bp)\\setminus\\cA^0(\\bp).\\label{eqn:kkt-compl}\n  \\end{align}\n\\end{subequations}\nUnder SSOSC and LICQ for $P(\\bp^\\star)$ at $\\bz^\\star$, these conditions are necessary and sufficient for optimality. From the complementarity condition \\eqref{eqn:kkt-compl}, one can observe that there exists $\\cA^0(\\bp)\\subseteq\\cA'(\\bp,\\bq)\\subseteq \\cA^1(\\bp)$ such that:\n","start":43324,"end":44245}]},{"id":"S035","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"互补条件选取方向活动集","source_units":[{"raw_tex":"Under SSOSC and LICQ for $P(\\bp^\\star)$ at $\\bz^\\star$, these conditions are necessary and sufficient for optimality. From the complementarity condition \\eqref{eqn:kkt-compl}, one can observe that there exists $\\cA^0(\\bp)\\subseteq\\cA'(\\bp,\\bq)\\subseteq \\cA^1(\\bp)$ such that:\n\\begin{subequations}\\label{eqn:compl-explicit}\n\\begin{align}\n  \\bA(\\bp)[i,:]\\bxi + \\bB(\\bp)[i,:]\\bq &= 0,\\; i\\in \\cA'(\\bp,\\bq)\\\\\n  \\bbeta[i] &= 0,\\; i\\in \\cA^1(\\bp)\\setminus \\cA'(\\bp,\\bq).\n\\end{align}\n\\end{subequations}\nare satisfied at $\\bxi^\\dag(\\bp,\\bq)$, $\\bbeta^\\dag(\\bp,\\bq)$. Thus, from \\eqref{eqn:kkt}, \\eqref{eqn:compl-explicit}, and $\\bbeta[i]=0$ for $i\\in\\mathbb{I}_{[1,\\bm]}\\setminus \\cA^1(\\bp)$ (from Lemma \\ref{lem:nlp-sensitivity}), we have:\n","start":43969,"end":44702}]},{"id":"S036","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:linear-form-0"],"declared_external":false,"source_only":false,"title":"活动坐标上的线性系统","source_units":[{"raw_tex":"\\begin{align}\\label{eqn:linear-form-0}\n  \\begin{aligned}\n  \\begin{bmatrix}\n    \\bQ(\\bp)& \\bA(\\bp)[\\cA'(\\bp,\\bq),:]^\\top\\\\\n    \\bA(\\bp)[\\cA'(\\bp,\\bq),:]\n  \\end{bmatrix}\n  \\begin{bmatrix}\n    \\bxi\\\\\n    \\bbeta[\\cA'(\\bp,\\bq)]\n  \\end{bmatrix} &= -\n  \\begin{bmatrix}\n    \\bS(\\bp)\\\\  \n    \\bB(\\bp)[\\cA'(\\bp,\\bq),:]\n  \\end{bmatrix}\\bq\\\\\n  \\bbeta [\\mathbb{I}_{[1,\\bm]}\\setminus \\cA'(\\bp,\\bq)] &= \\bzero.\n\\end{aligned}\n\\end{align}\n% The linear equation \\eqref{eqn:linear-form-0} provides a relationship between $[\\bxi^\\dag(\\bp,\\bq);\\bbeta^\\dag(\\bp,\\bq)]$ and $\\bq$; however, it does not provide an explicit relationship  because $\\cA'(\\bp,\\bq)$ depends on $\\bq$.\n","start":44702,"end":45356}]},{"id":"S037","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"节点重排与选中坐标集合","source_units":[{"raw_tex":"By rearranging $[\\bxi^\\dag(\\bp,\\bq);\\bbeta^\\dag(\\bp,\\bq)]$ one can obtain $\\bzeta^\\dag(\\bp)=\\{[\\xi_i^\\dag(\\bp,\\bq);\\eta_i^\\dag(\\bp,\\bq)]\\}_{i\\in\\cV}$, where $\\bxi^\\dag(\\bp,\\bq)=\\{\\xi_i^\\dag(\\bp,\\bq)\\}_{i\\in\\cV}$, $\\bbeta^\\dag(\\bp,\\bq)=\\{\\eta_i^\\dag(\\bp,\\bq)\\}_{i\\in\\cV}$. To perform such rearrangement, we consider a permutation $\\phi:\\mathbb{I}_{[1,\\bn]}\\rightarrow\\mathbb{I}_{[1,\\bn]}$ that achieves $\\bzeta[\\phi(i)] = [\\bxi;\\bbeta][i]$. This permutation enables the following definition:\n\\begin{subequations}\\label{eqn:Bs}\n\\begin{align}\n  \\cB^0(\\bp)&:=\\phi(\\mathbb{I}_{[1,\\br]}\\cup(\\cA^0(\\bp)+\\br)),\\quad\n                      \\cB^1(\\bp):=\\phi(\\mathbb{I}_{[1,\\br]}\\cup(\\cA^1(\\bp)+\\br))\\\\\n  \\cB'(\\bp,\\bq)&:=\\phi(\\mathbb{I}_{[1,\\br]}\\cup(\\cA'(\\bp,\\bq)+\\br)).\n\\end{align}\n\\end{subequations}\n","start":45356,"end":46147}]},{"id":"S038","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:HR","eqn:linear-form-b"],"declared_external":false,"source_only":false,"title":"方向导数的 Hessian 逆表示","source_units":[{"raw_tex":"Finally, using the notation in \\eqref{eqn:Bs}, we obtain the following from \\eqref{eqn:linear-form-0}:\n\\begin{subequations}\\label{eqn:linear-form}\n\\begin{align}\n  D_{\\bq}\\bz^\\dag(\\bp)[\\cB'(\\bp,\\bq)] &= -\\left(\\bH(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)]\\right)^{-1}\\bR(\\bp)[\\cB'(\\bp,\\bq),:] \\bq,\\\\\n  D_{\\bq}\\bz^\\dag(\\bp)[\\mathbb{I}_{[1,\\bn]}\\setminus\\cB'(\\bp,\\bq)] &= \\bzero,\\label{eqn:linear-form-b}\n\\end{align}\n\\end{subequations}\nwhere:\n\\begin{align}\\label{eqn:HR}\n  \\bH(\\bp) &:= \\nabla_{\\bz\\bz}\\cL(\\bz^\\dag(\\bp);\\bp);\\quad \\bR(\\bp) := \\nabla_{\\bz\\bp}\\cL(\\bz^\\dag(\\bp);\\bp).\n\\end{align}\n","start":46147,"end":46729}]},{"id":"S039","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"活动 Hessian 子矩阵非奇异","source_units":[{"raw_tex":"Here, the nonsingularity of $\\bH(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)]$\n% is guaranteed by the fact that $\\bQ(\\bp)$ is positive definite on the null-space of $\\bA(\\bp)[\\cA^0(\\bp),:]$. This\nfollows from LICQ, SSOSC, and \\cite[Lemma 16.1]{nocedal2006numerical}.\n","start":46729,"end":46985}]},{"id":"S040","semantic_type":"notation","kind":"notation","labels":["eqn:ind-1"],"declared_external":false,"source_only":false,"title":"节点坐标族与 Hessian 块","source_units":[{"raw_tex":"\\subsection{Nodal Sensitivity Result}\\label{sec:nodal}\n{We now establish the main nodal sensitivity result. First, we consider $\\bH(\\bp)$ and $\\bR(\\bp)$ as graph-structured matrices.} Specifically, we introduce the index set families $\\cI:=\\{I_i\\}_{i\\in\\cV}$ and $\\cK:=\\{K_i\\}_{i\\in\\cV}$with:\n\\begin{align}\\label{eqn:ind-1}\n  I_i &:= \\mathbb{I}_{\\sum_{j\\in \\cV,j<i}n_j+[1,n_i]},\\quad\n  K_i := \\mathbb{I}_{\\sum_{j\\in \\cV,j<i}l_j+[1,l_i]},\n\\end{align}\nand impose graph structures to $\\bH(\\bp)$ and $\\bR(\\bp)$ with $(\\cG,\\cI,\\cI)$ and $(\\cG,\\cI,\\cK)$, respectively. Note that $\\cI$ and $\\cK$ partition $\\mathbb{I}_{[1,\\bn]}$ and $\\mathbb{I}_{[1,\\bl]}$, respectively. For $i,j\\in\\cV$,\n\\begin{subequations}\\label{eqn:HRij}\n\\begin{align}\n  H_{ij}(\\bp)&:=\\nabla^2_{z_iz_j}\\cL(\\bz^\\dag(\\bp);\\bp)=(\\bH(\\bp))_{[i][j]}\\\\\n  R_{ij}(\\bp)&:=\\nabla^2_{z_ip_j}\\cL(\\bz^\\dag(\\bp);\\bp)=(\\bR(\\bp))_{[i][j]}.\n\\end{align}\n\\end{subequations}\n","start":46986,"end":47904}]},{"id":"S041","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"闭邻域耦合推出带宽二","source_units":[{"raw_tex":"{We now can observe that $\\bH(\\bp)$ and $\\bR(\\bp)$ have bandwidth not greater than two. In other words, if $d_\\mathcal{G}(i,j)> 2$, $\\nabla_{z_i z_j}\\mathcal{L}(\\bz;\\bp)=0$ and $\\nabla_{z_i p_j}\\mathcal{L}(\\bz;\\bp)=0$. This is because in the graph-structured nonlinear programming setting in (1.1), we have assumed that the nodal objective and constraint functions are only dependent on the variables that are within the closed neighborhood of the associated node.}\n% The reason that the bandwidth may be greater than one is as follows. Suppose that $\\cV=\\{1,2,3\\}$ and $\\cE=\\{\\{1,2\\},\\{2,3\\}\\}$; then the constraints at node $2$ can be coupled with variables/data at node $1$ and $3$. This produces potential coupling between the variables that are within distance $2$ can have potential coupling.\n\n","start":47904,"end":48704}]},{"id":"S042","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"限制子矩阵继承带宽","source_units":[{"raw_tex":"The submatrices of $\\bH(\\bp)$ and $\\bR(\\bp)$ are also graph-structured. In particular, $\\bH(\\bp)[\\cB,\\cB]$ and $\\bR(\\bp)[\\cB,:]$ with $\\cB\\subseteq \\mathbb{I}_{[1,\\bn]}$ have bandwidth not greater than two induced by $(\\cG,\\cI^\\cB,\\cI^\\cB)$ and $(\\cG,\\cI^\\cB,\\cK)$, where $\\cI^\\cB:=\\{I_i\\cap\\cB\\}_{i\\in\\cV}$. We thus see that \\eqref{eqn:linear-form} involves the inverse of a {graph-induced banded} matrix. By combining Theorem \\ref{thm:inv} and Lemma \\ref{lem:nlp-sensitivity}, we obtain the following result.\n","start":48704,"end":49215}]},{"id":"S043","semantic_type":"lemma","kind":"lemma","labels":["eqn:dder-0","eqn:dder-0-c","lem:dder"],"declared_external":false,"source_only":false,"title":"节点单侧方向导数界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:dder}\n  Suppose Assumptions \\ref{ass:c2}, \\ref{ass:regularity} hold, and suppose that, for given $\\bp\\in\\mathbb{P}$ (defined in Lemma \\ref{lem:nlp-sensitivity}) and $\\bq:=\\{q_i\\}_{i\\in \\cV}\\in\\mathbb{R}^{\\bl}$, we have $\\osigma_{\\bH}(\\bp,\\bq)$, $\\osigma_{\\bR}(\\bp,\\bq)$, and $\\usigma_{\\bH}(\\bp,\\bq)$ such that:\n  \\begin{subequations}\\label{eqn:dder-0}\n  \\begin{align}\n    \\osigma_{\\bH}(\\bp,\\bq)&\\geq \\osigma(\\bH(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)])\\\\\n    \\osigma_{\\bR}(\\bp,\\bq)&\\geq \\osigma(\\bR(\\bp)[\\cB'(\\bp,\\bq),:])\\\\\n    0<\\usigma_{\\bH}(\\bp,\\bq)&\\leq \\usigma(\\bH(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)]);\\label{eqn:dder-0-c}\n  \\end{align}\n\\end{subequations}\nthen the following holds for any $i\\in \\cV$:\n  \\begin{align*}\n    \\Vert D_{\\bq} z^\\dag_i(\\bp,\\bq)\\Vert\\leq \\sum_{j\\in \\cV}\\frac{\\osigma_{\\bH}(\\bp,\\bq)\\osigma_{\\bR}(\\bp,\\bq)}{\\usigma_{\\bH}(\\bp,\\bq)^2}\\left(\\frac{\\osigma_{\\bH}(\\bp,\\bq)^2-\\usigma_{\\bH}(\\bp,\\bq)^2}{\\osigma_{\\bH}(\\bp,\\bq)^2+\\usigma_{\\bH}(\\bp,\\bq)^2}\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert.\n  \\end{align*}\n\\end{lemma}","start":49216,"end":50305}]},{"id":"S044","semantic_type":"assumption","kind":"assumption","labels":["eqn:dder-0","eqn:dder-0-c","lem:dder"],"declared_external":false,"source_only":false,"title":"方向导数界的谱参数","source_units":[{"raw_tex":"  Suppose Assumptions \\ref{ass:c2}, \\ref{ass:regularity} hold, and suppose that, for given $\\bp\\in\\mathbb{P}$ (defined in Lemma \\ref{lem:nlp-sensitivity}) and $\\bq:=\\{q_i\\}_{i\\in \\cV}\\in\\mathbb{R}^{\\bl}$, we have $\\osigma_{\\bH}(\\bp,\\bq)$, $\\osigma_{\\bR}(\\bp,\\bq)$, and $\\usigma_{\\bH}(\\bp,\\bq)$ such that:\n  \\begin{subequations}\\label{eqn:dder-0}\n  \\begin{align}\n    \\osigma_{\\bH}(\\bp,\\bq)&\\geq \\osigma(\\bH(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)])\\\\\n    \\osigma_{\\bR}(\\bp,\\bq)&\\geq \\osigma(\\bR(\\bp)[\\cB'(\\bp,\\bq),:])\\\\\n    0<\\usigma_{\\bH}(\\bp,\\bq)&\\leq \\usigma(\\bH(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)]);\\label{eqn:dder-0-c}\n  \\end{align}\n\\end{subequations}\n","start":49246,"end":49893}]},{"id":"S045","semantic_type":"proof","kind":"proof","labels":["eqn:nlp-sensitivity-1","eqn:nlp-sensitivity-2"],"declared_external":false,"source_only":false,"title":"节点导数界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n  {\\blue For simplicity, we denote $\\cB'(\\bp,\\bq)$, $\\osigma_{\\bH}(\\bp,\\bq)$, $\\osigma_{\\bR}(\\bp,\\bq)$, $\\usigma_{\\bH}(\\bp,\\bq)$ by $\\cB'$, $\\osigma_{\\bH}$, $\\osigma_{\\bR}$, $\\usigma_{\\bH}$}. From that $\\bH(\\bp)[\\cB',\\cB']$ is always nonsingular, we have that $\\usigma_{\\bH}(\\bp,\\bq)$ satisfying \\eqref{eqn:dder-0-c} always exists. By inspecting the block structure of \\eqref{eqn:linear-form} we obtain:\n  \\begin{align}\\label{eqn:nlp-sensitivity-1}\n    D_{\\bq}\\bz^\\dag(\\bp)[I_i\\cap\\cB'] &= \\sum_{j\\in \\cV} -((\\bH(\\bp)[\\cB',\\cB'])^{-1}\\bR(\\bp)[\\cB',:])_{[i][j]} q_j,\n  \\end{align}\n  where $\\cI^{\\cB'}:=\\{I_i\\cap{\\cB'}\\}_{i\\in \\cV}$ and $\\cK:=\\{K_i\\}_{i\\in \\cV}$ (defined in \\eqref{eqn:ind-1}) are used for index sets. We have already established that $\\bH(\\bp)[\\cB',\\cB']$ has bandwidth not greater than two, induced by $(\\cG,\\cI^{\\cB'},\\cI^{\\cB'})$ and that $\\bR(\\bp)[\\cB',:]$ has bandwidth not greater than two, induced by $(\\cG,\\cI^{\\cB'},\\cK)$. By applying Theorem \\ref{thm:inv}, we obtain:\n  \\begin{align}\\label{eqn:nlp-sensitivity-2}\\blue\n    \\Vert((\\bH(\\bp)[\\cB',\\cB'])^{-1}\\bR(\\bp)[\\cB',:])_{[i][j]}\\Vert\\leq \\frac{\\osigma_{\\bH}\\osigma_{\\bR}}{\\usigma_{\\bH}^2}\\left(\\frac{\\osigma_{\\bH}^2-\\usigma_{\\bH}^2}{\\osigma_{\\bH}^2+\\usigma_{\\bH}^2}\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+} .\n  \\end{align}\n  Now note that\n  $\\Vert D_{\\bq}z^\\dag_i(\\bp)\\Vert \\leq \\Vert D_{\\bq}\\bz^\\dag(\\bp)[I_i\\cap\\cB'] \\Vert  + \\Vert D_{\\bq}\\bz^\\dag(\\bp)[I_i\\setminus\\cB']\\Vert$,\n  and recall from \\eqref{eqn:linear-form-b} that $D_{\\bq}\\bz^\\dag(\\bp)[I_i\\setminus\\cB'] =\\bzero$. Hence, by applying \\eqref{eqn:nlp-sensitivity-2} to \\eqref{eqn:nlp-sensitivity-1}, and applying triangle inequality, we obtain the desired result.\n\\end{proof}","start":50306,"end":52052}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:nlp-sensitivity-1"],"declared_external":false,"source_only":false,"title":"按数据节点展开方向导数","source_units":[{"raw_tex":"  {\\blue For simplicity, we denote $\\cB'(\\bp,\\bq)$, $\\osigma_{\\bH}(\\bp,\\bq)$, $\\osigma_{\\bR}(\\bp,\\bq)$, $\\usigma_{\\bH}(\\bp,\\bq)$ by $\\cB'$, $\\osigma_{\\bH}$, $\\osigma_{\\bR}$, $\\usigma_{\\bH}$}. From that $\\bH(\\bp)[\\cB',\\cB']$ is always nonsingular, we have that $\\usigma_{\\bH}(\\bp,\\bq)$ satisfying \\eqref{eqn:dder-0-c} always exists. By inspecting the block structure of \\eqref{eqn:linear-form} we obtain:\n  \\begin{align}\\label{eqn:nlp-sensitivity-1}\n    D_{\\bq}\\bz^\\dag(\\bp)[I_i\\cap\\cB'] &= \\sum_{j\\in \\cV} -((\\bH(\\bp)[\\cB',\\cB'])^{-1}\\bR(\\bp)[\\cB',:])_{[i][j]} q_j,\n  \\end{align}\n  where $\\cI^{\\cB'}:=\\{I_i\\cap{\\cB'}\\}_{i\\in \\cV}$ and $\\cK:=\\{K_i\\}_{i\\in \\cV}$ (defined in \\eqref{eqn:ind-1}) are used for index sets. We have already established that $\\bH(\\bp)[\\cB',\\cB']$ has bandwidth not greater than two, induced by $(\\cG,\\cI^{\\cB'},\\cI^{\\cB'})$ and that $\\bR(\\bp)[\\cB',:]$ has bandwidth not greater than two, induced by $(\\cG,\\cI^{\\cB'},\\cK)$. By applying Theorem \\ref{thm:inv}, we obtain:\n","start":50320,"end":51314}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:nlp-sensitivity-2"],"declared_external":false,"source_only":false,"title":"将逆衰减用于方向系数块","source_units":[{"raw_tex":"  where $\\cI^{\\cB'}:=\\{I_i\\cap{\\cB'}\\}_{i\\in \\cV}$ and $\\cK:=\\{K_i\\}_{i\\in \\cV}$ (defined in \\eqref{eqn:ind-1}) are used for index sets. We have already established that $\\bH(\\bp)[\\cB',\\cB']$ has bandwidth not greater than two, induced by $(\\cG,\\cI^{\\cB'},\\cI^{\\cB'})$ and that $\\bR(\\bp)[\\cB',:]$ has bandwidth not greater than two, induced by $(\\cG,\\cI^{\\cB'},\\cK)$. By applying Theorem \\ref{thm:inv}, we obtain:\n  \\begin{align}\\label{eqn:nlp-sensitivity-2}\\blue\n    \\Vert((\\bH(\\bp)[\\cB',\\cB'])^{-1}\\bR(\\bp)[\\cB',:])_{[i][j]}\\Vert\\leq \\frac{\\osigma_{\\bH}\\osigma_{\\bR}}{\\usigma_{\\bH}^2}\\left(\\frac{\\osigma_{\\bH}^2-\\usigma_{\\bH}^2}{\\osigma_{\\bH}^2+\\usigma_{\\bH}^2}\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+} .\n  \\end{align}\n","start":50900,"end":51638}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"恢复完整节点并求范数","source_units":[{"raw_tex":"  Now note that\n  $\\Vert D_{\\bq}z^\\dag_i(\\bp)\\Vert \\leq \\Vert D_{\\bq}\\bz^\\dag(\\bp)[I_i\\cap\\cB'] \\Vert  + \\Vert D_{\\bq}\\bz^\\dag(\\bp)[I_i\\setminus\\cB']\\Vert$,\n  and recall from \\eqref{eqn:linear-form-b} that $D_{\\bq}\\bz^\\dag(\\bp)[I_i\\setminus\\cB'] =\\bzero$. Hence, by applying \\eqref{eqn:nlp-sensitivity-2} to \\eqref{eqn:nlp-sensitivity-1}, and applying triangle inequality, we obtain the desired result.\n","start":51638,"end":52041}]},{"id":"S049","semantic_type":"theorem","kind":"theorem","labels":["eqn:main","eqn:sigmas","thm:main"],"declared_external":false,"source_only":false,"title":"指数敏感性衰减主定理","source_units":[{"raw_tex":"\\begin{theorem}[Exponential Decay of Sensitivity (EDS)]\\label{thm:main}\n  Under Assumptions \\ref{ass:c2}, \\ref{ass:regularity}, and for given $\\epsilon>0$, $\\osigma_{\\bH}\\geq\\oosigma_{\\bH}(\\bp^\\star)$, $\\osigma_{\\bR}\\geq\\oosigma_{\\bR}(\\bp^\\star)$, and $0<\\usigma_{\\bH}\\leq\\uusigma_{\\bH}(\\bp^\\star)$, where\n  \\begin{subequations}\\label{eqn:sigmas}\n    \\begin{align}\n      \\oosigma_{\\bH}(\\bp)&:=\\max\\{\\osigma(\\bH(\\bp)[\\cB,\\cB]):\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)\\}\\\\\n      \\oosigma_{\\bR}(\\bp)&:=\\max\\{\\osigma(\\bR(\\bp)[\\cB,:]):\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)\\}\\\\\n      \\uusigma_{\\bH}(\\bp)&:=\\min\\{\\usigma(\\bH(\\bp)[\\cB,\\cB]):\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)\\},\n    \\end{align}\n  \\end{subequations}\n  there exists a neighborhood $\\mathbb{P}_{\\epsilon}$ of $\\bp^\\star$ such that the following holds for any $\\bp,\\bp'\\in\\mathbb{P}_{\\epsilon}$:\n  \\begin{align}\\label{eqn:main}\n      &\\Vert z^\\dag_i(\\bp)-z^\\dag_i(\\bp')\\Vert \n      \\leq \\sum_{j\\in \\cV} \\Upsilon\\rho^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+} \\Vert p_j-p'_j\\Vert,\\quad i\\in \\cV,\n  \\end{align}\nwith $\\Upsilon :=\\dfrac{\\osigma_{\\bH}\\osigma_{\\bR}}{{\\usigma}_{\\bH}^2}+\\epsilon$ and $\\rho:=\\dfrac{\\osigma_{\\bH}^2-{\\usigma}_{\\bH}^2}{\\osigma_{\\bH}^2+{\\usigma}_{\\bH}^2}+\\epsilon$.\n\\end{theorem}","start":52537,"end":53850}]},{"id":"S050","semantic_type":"definition","kind":"definition","labels":["eqn:sigmas","thm:main"],"declared_external":false,"source_only":false,"title":"固定基点活动族的谱包络","source_units":[{"raw_tex":"  \\begin{subequations}\\label{eqn:sigmas}\n    \\begin{align}\n      \\oosigma_{\\bH}(\\bp)&:=\\max\\{\\osigma(\\bH(\\bp)[\\cB,\\cB]):\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)\\}\\\\\n      \\oosigma_{\\bR}(\\bp)&:=\\max\\{\\osigma(\\bR(\\bp)[\\cB,:]):\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)\\}\\\\\n      \\uusigma_{\\bH}(\\bp)&:=\\min\\{\\usigma(\\bH(\\bp)[\\cB,\\cB]):\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)\\},\n    \\end{align}\n  \\end{subequations}\n","start":52843,"end":53293}]},{"id":"S051","semantic_type":"assumption","kind":"assumption","labels":["eqn:sigmas","thm:main"],"declared_external":false,"source_only":false,"title":"主定理给定参数与系数","source_units":[{"raw_tex":"  Under Assumptions \\ref{ass:c2}, \\ref{ass:regularity}, and for given $\\epsilon>0$, $\\osigma_{\\bH}\\geq\\oosigma_{\\bH}(\\bp^\\star)$, $\\osigma_{\\bR}\\geq\\oosigma_{\\bR}(\\bp^\\star)$, and $0<\\usigma_{\\bH}\\leq\\uusigma_{\\bH}(\\bp^\\star)$, where\n","start":52609,"end":52843},{"raw_tex":"with $\\Upsilon :=\\dfrac{\\osigma_{\\bH}\\osigma_{\\bR}}{{\\usigma}_{\\bH}^2}+\\epsilon$ and $\\rho:=\\dfrac{\\osigma_{\\bH}^2-{\\usigma}_{\\bH}^2}{\\osigma_{\\bH}^2+{\\usigma}_{\\bH}^2}+\\epsilon$.\n","start":53657,"end":53837}]},{"id":"S052","semantic_type":"proof","kind":"proof","labels":["eqn:main-0","eqn:main-1","eqn:main-2"],"declared_external":false,"source_only":false,"title":"EDS 主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n  From the continuity of $\\bz^\\dag(\\cdot)$ in the neighborhood of $\\bp^\\star$ and the continuity of $\\bc(\\cdot,\\cdot)$ in the neighborhood of $[\\bz^\\star;\\bp^\\star]$, there exists a neighborhood $\\widetilde{\\mathbb{P}}\\subseteq\\mathbb{P}$ of $\\bp^\\star$ such that, for $\\bp\\in\\widetilde{\\mathbb{P}}$ and $i\\in\\mathbb{I}_{[1,\\bm]}$, $\\bc(\\bx^\\star)[i] > 0\\Rightarrow\\bc(\\bx^\\dag(\\bp))[i] > 0$ and $\\by^\\star[i] \\neq 0\\Rightarrow\\by^\\dag(\\bp)[i] \\neq 0$. These conditions and complementarity slackness imply that for $\\bp\\in\\widetilde{\\mathbb{P}}$, we have $\\cA^0(\\bp^\\star)\\subseteq\\cA^0(\\bp)$ and $\\cA^1(\\bp)\\subseteq\\cA^1(\\bp^\\star)$; that is, $\\cB^0(\\bp^\\star)\\subseteq\\cB^0(\\bp)$ and $\\cB^1(\\bp)\\subseteq\\cB^1(\\bp^\\star)$. This and $\\cB^0(\\bp)\\subseteq\\cB'(\\bp,\\bq)\\subseteq\\cB^1(\\bp)$ yields:\n  \\begin{subequations}\\label{eqn:main-0}\n  \\begin{align}\n    \\osigma({\\bH}(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)])&\\leq \\oosigma_{\\bH}(\\bp)\\\\\n    \\osigma({\\bR}(\\bp)[\\cB'(\\bp,\\bq),:])&\\leq \\oosigma_{\\bR}(\\bp)\\\\\n    \\usigma({\\bH}(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)])&\\geq \\uusigma_{\\bH}(\\bp).\n  \\end{align}\n\\end{subequations}\n  By the twice-continuous differentiability of $\\cL(\\cdot,\\cdot)$ and the continuity of $\\bz^\\dag(\\cdot)$, we have that $\\bH(\\cdot)$ is continuous. The same holds true for its submatrices: $\\bH(\\cdot)[\\cB,\\cB]$ with $\\cB^0(\\bp^\\star)\\subseteq \\cB\\subseteq \\cB^1(\\bp^\\star)$. From the continuity of singular values with respect to its entries \\cite[Corollary 8.6.2]{golub2012matrix}, we have that $\\osigma(\\bH(\\cdot)[\\cB,\\cB])$ and $\\usigma(\\bH(\\cdot)[\\cB,\\cB])$ are continuous for any $\\cB^0(\\bp^\\star)\\subseteq \\cB\\subseteq \\cB^1(\\bp^\\star)$; accordingly, since a maximum and a minimum of a fixed and finite number of continuous functions is continuous, we have that $\\oosigma_{\\bH}(\\bp)$, $\\oosigma_{\\bR}(\\bp)$, $\\uusigma_{\\bH}(\\bp)$ are continuous with respect to $\\bp$ in $\\widetilde{\\mathbb{P}}$. Thus, there exists a convex neighborhood $\\mathbb{P}_{\\epsilon}\\subseteq\\widetilde{\\mathbb{P}}$ of $\\bp^\\star$ wherein the following are satisfied:\n  \\begin{subequations}\\label{eqn:main-1}\n  \\begin{align}\n    \\frac{\\oosigma_{\\bH}(\\bp)\\oosigma_{\\bR}(\\bp)}{\\uusigma_{\\bH}(\\bp)^2} &\\leq\n    \\frac{\\oosigma_{\\bH}(\\bp^\\star)\\oosigma_{\\bR}(\\bp^\\star)}{\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon\\\\\n    \\frac{\\oosigma_{\\bH}(\\bp)^2-\\uusigma_{\\bH}(\\bp)^2}{\\oosigma_{\\bH}(\\bp)^2+\\uusigma_{\\bH}(\\bp)^2}&\\leq\n    \\frac{\\oosigma_{\\bH}(\\bp^\\star)^2-\\uusigma_{\\bH}(\\bp^\\star)^2}{\\oosigma_{\\bH}(\\bp^\\star)^2+\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon.\n  \\end{align}\n\\end{subequations}\nHere, note that $\\uusigma_{\\bH}(\\bp^\\star)>0$ because $\\usigma(\\bH[\\cB,\\cB])>0$ holds for any $\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)$. By applying \\eqref{eqn:main-0} and \\eqref{eqn:main-1} to Lemma \\ref{lem:dder}, we obtain:\n\\begin{align}\\label{eqn:main-2}\n  \\Vert D_{\\bq} z^\\dag_i(\\bp)\\Vert&\\leq \\sum_{j\\in \\cV}\\left(\\frac{\\oosigma_{\\bH}(\\bp^\\star)\\oosigma_{\\bR}(\\bp^\\star)}{\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon\\right)\\left(\\frac{\\oosigma_{\\bH}(\\bp^\\star)^2-\\uusigma_{\\bH}(\\bp^\\star)^2}{\\oosigma_{\\bH}(\\bp^\\star)^2+\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert\\nonumber\\\\\n  &\\leq \\sum_{j\\in \\cV}\\left(\\frac{\\osigma_{\\bH}\\osigma_{\\bR}}{\\usigma_{\\bH}^2}+\\epsilon\\right)\\left(\\frac{\\osigma_{\\bH}^2-\\usigma_{\\bH}^2}{\\osigma_{\\bH}^2+\\usigma_{\\bH}^2}+\\epsilon\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert\\nonumber\\\\\n  &\\leq \\sum_{j\\in \\cV}\\Upsilon\\rho^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert,\n\\end{align}\nfor any $\\bp\\in\\mathbb{P}_{\\epsilon}$ and $\\bq\\in\\mathbb{R}^{\\bl}$. Finally, since we have chosen $\\mathbb{P}_\\epsilon$ to be convex, for any $\\bp,\\bp'\\in\\mathbb{P}_{\\epsilon}$, the line segment between $\\bp,\\bp'$ is within $\\mathbb{P}_{\\epsilon}$. Thus, we have:\n\\begin{align*}\n  \\Vert z^\\dag_i(\\bp)-z^\\dag_i(\\bp')\\Vert\n  % &\\leq\\Vert \\int_{0}^1 D_{\\bp'-\\bp}z^\\dag_i ((1-t)\\bp+t\\bp')dt\\Vert\\\\\n    &\\leq \\int_{0}^1\\Vert D_{\\bp'-\\bp}z^\\dag_i ((1-t)\\bp+t\\bp')\\Vert dt \\leq \\sum_{j\\in \\cV}\\Upsilon\\rho^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert p_j-p'_j\\Vert,\n\\end{align*}\nwhere the first inequality is from Newton-Leibniz and triangle inequality for integrals, and the last inequality follows from \\eqref{eqn:main-2}.\n\\end{proof}","start":53851,"end":58212}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:main-0","eqn:main-1"],"declared_external":false,"source_only":false,"title":"邻域活动集夹持","source_units":[{"raw_tex":"  From the continuity of $\\bz^\\dag(\\cdot)$ in the neighborhood of $\\bp^\\star$ and the continuity of $\\bc(\\cdot,\\cdot)$ in the neighborhood of $[\\bz^\\star;\\bp^\\star]$, there exists a neighborhood $\\widetilde{\\mathbb{P}}\\subseteq\\mathbb{P}$ of $\\bp^\\star$ such that, for $\\bp\\in\\widetilde{\\mathbb{P}}$ and $i\\in\\mathbb{I}_{[1,\\bm]}$, $\\bc(\\bx^\\star)[i] > 0\\Rightarrow\\bc(\\bx^\\dag(\\bp))[i] > 0$ and $\\by^\\star[i] \\neq 0\\Rightarrow\\by^\\dag(\\bp)[i] \\neq 0$. These conditions and complementarity slackness imply that for $\\bp\\in\\widetilde{\\mathbb{P}}$, we have $\\cA^0(\\bp^\\star)\\subseteq\\cA^0(\\bp)$ and $\\cA^1(\\bp)\\subseteq\\cA^1(\\bp^\\star)$; that is, $\\cB^0(\\bp^\\star)\\subseteq\\cB^0(\\bp)$ and $\\cB^1(\\bp)\\subseteq\\cB^1(\\bp^\\star)$. This and $\\cB^0(\\bp)\\subseteq\\cB'(\\bp,\\bq)\\subseteq\\cB^1(\\bp)$ yields:\n","start":53865,"end":54662}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:main-0","eqn:main-1"],"declared_external":false,"source_only":false,"title":"选中子矩阵受谱包络控制","source_units":[{"raw_tex":"  \\begin{subequations}\\label{eqn:main-0}\n  \\begin{align}\n    \\osigma({\\bH}(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)])&\\leq \\oosigma_{\\bH}(\\bp)\\\\\n    \\osigma({\\bR}(\\bp)[\\cB'(\\bp,\\bq),:])&\\leq \\oosigma_{\\bR}(\\bp)\\\\\n    \\usigma({\\bH}(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)])&\\geq \\uusigma_{\\bH}(\\bp).\n  \\end{align}\n\\end{subequations}\n","start":54662,"end":54979}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:main-0","eqn:main-1"],"declared_external":false,"source_only":false,"title":"固定有限族的谱包络连续","source_units":[{"raw_tex":"  By the twice-continuous differentiability of $\\cL(\\cdot,\\cdot)$ and the continuity of $\\bz^\\dag(\\cdot)$, we have that $\\bH(\\cdot)$ is continuous. The same holds true for its submatrices: $\\bH(\\cdot)[\\cB,\\cB]$ with $\\cB^0(\\bp^\\star)\\subseteq \\cB\\subseteq \\cB^1(\\bp^\\star)$. From the continuity of singular values with respect to its entries \\cite[Corollary 8.6.2]{golub2012matrix}, we have that $\\osigma(\\bH(\\cdot)[\\cB,\\cB])$ and $\\usigma(\\bH(\\cdot)[\\cB,\\cB])$ are continuous for any $\\cB^0(\\bp^\\star)\\subseteq \\cB\\subseteq \\cB^1(\\bp^\\star)$; accordingly, since a maximum and a minimum of a fixed and finite number of continuous functions is continuous, we have that $\\oosigma_{\\bH}(\\bp)$, $\\oosigma_{\\bR}(\\bp)$, $\\uusigma_{\\bH}(\\bp)$ are continuous with respect to $\\bp$ in $\\widetilde{\\mathbb{P}}$. Thus, there exists a convex neighborhood $\\mathbb{P}_{\\epsilon}\\subseteq\\widetilde{\\mathbb{P}}$ of $\\bp^\\star$ wherein the following are satisfied:\n","start":54979,"end":55929}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:main-0","eqn:main-1"],"declared_external":false,"source_only":false,"title":"凸邻域中的统一比率界","source_units":[{"raw_tex":"  \\begin{subequations}\\label{eqn:main-1}\n  \\begin{align}\n    \\frac{\\oosigma_{\\bH}(\\bp)\\oosigma_{\\bR}(\\bp)}{\\uusigma_{\\bH}(\\bp)^2} &\\leq\n    \\frac{\\oosigma_{\\bH}(\\bp^\\star)\\oosigma_{\\bR}(\\bp^\\star)}{\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon\\\\\n    \\frac{\\oosigma_{\\bH}(\\bp)^2-\\uusigma_{\\bH}(\\bp)^2}{\\oosigma_{\\bH}(\\bp)^2+\\uusigma_{\\bH}(\\bp)^2}&\\leq\n    \\frac{\\oosigma_{\\bH}(\\bp^\\star)^2-\\uusigma_{\\bH}(\\bp^\\star)^2}{\\oosigma_{\\bH}(\\bp^\\star)^2+\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon.\n  \\end{align}\n\\end{subequations}\n","start":55929,"end":56439}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:main-0","eqn:main-1"],"declared_external":false,"source_only":false,"title":"基点下谱包络为正","source_units":[{"raw_tex":"Here, note that $\\uusigma_{\\bH}(\\bp^\\star)>0$ because $\\usigma(\\bH[\\cB,\\cB])>0$ holds for any $\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)$. By applying \\eqref{eqn:main-0} and \\eqref{eqn:main-1} to Lemma \\ref{lem:dder}, we obtain:\n","start":56439,"end":56680}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:main-0","eqn:main-1","eqn:main-2"],"declared_external":false,"source_only":false,"title":"统一全部方向的节点导数界","source_units":[{"raw_tex":"Here, note that $\\uusigma_{\\bH}(\\bp^\\star)>0$ because $\\usigma(\\bH[\\cB,\\cB])>0$ holds for any $\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)$. By applying \\eqref{eqn:main-0} and \\eqref{eqn:main-1} to Lemma \\ref{lem:dder}, we obtain:\n\\begin{align}\\label{eqn:main-2}\n  \\Vert D_{\\bq} z^\\dag_i(\\bp)\\Vert&\\leq \\sum_{j\\in \\cV}\\left(\\frac{\\oosigma_{\\bH}(\\bp^\\star)\\oosigma_{\\bR}(\\bp^\\star)}{\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon\\right)\\left(\\frac{\\oosigma_{\\bH}(\\bp^\\star)^2-\\uusigma_{\\bH}(\\bp^\\star)^2}{\\oosigma_{\\bH}(\\bp^\\star)^2+\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert\\nonumber\\\\\n  &\\leq \\sum_{j\\in \\cV}\\left(\\frac{\\osigma_{\\bH}\\osigma_{\\bR}}{\\usigma_{\\bH}^2}+\\epsilon\\right)\\left(\\frac{\\osigma_{\\bH}^2-\\usigma_{\\bH}^2}{\\osigma_{\\bH}^2+\\usigma_{\\bH}^2}+\\epsilon\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert\\nonumber\\\\\n  &\\leq \\sum_{j\\in \\cV}\\Upsilon\\rho^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert,\n\\end{align}\nfor any $\\bp\\in\\mathbb{P}_{\\epsilon}$ and $\\bq\\in\\mathbb{R}^{\\bl}$. Finally, since we have chosen $\\mathbb{P}_\\epsilon$ to be convex, for any $\\bp,\\bp'\\in\\mathbb{P}_{\\epsilon}$, the line segment between $\\bp,\\bp'$ is within $\\mathbb{P}_{\\epsilon}$. Thus, we have:\n","start":56439,"end":57736}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:main-0","eqn:main-1"],"declared_external":false,"source_only":false,"title":"沿数据线段积分","source_units":[{"raw_tex":"for any $\\bp\\in\\mathbb{P}_{\\epsilon}$ and $\\bq\\in\\mathbb{R}^{\\bl}$. Finally, since we have chosen $\\mathbb{P}_\\epsilon$ to be convex, for any $\\bp,\\bp'\\in\\mathbb{P}_{\\epsilon}$, the line segment between $\\bp,\\bp'$ is within $\\mathbb{P}_{\\epsilon}$. Thus, we have:\n\\begin{align*}\n  \\Vert z^\\dag_i(\\bp)-z^\\dag_i(\\bp')\\Vert\n  % &\\leq\\Vert \\int_{0}^1 D_{\\bp'-\\bp}z^\\dag_i ((1-t)\\bp+t\\bp')dt\\Vert\\\\\n    &\\leq \\int_{0}^1\\Vert D_{\\bp'-\\bp}z^\\dag_i ((1-t)\\bp+t\\bp')\\Vert dt \\leq \\sum_{j\\in \\cV}\\Upsilon\\rho^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert p_j-p'_j\\Vert,\n\\end{align*}\nwhere the first inequality is from Newton-Leibniz and triangle inequality for integrals, and the last inequality follows from \\eqref{eqn:main-2}.\n","start":57472,"end":58201}]},{"id":"S060","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisite list：Dontchev and Rockafellar, dontchev2009implicit, Theorem 2G.8；实际使用见第 383 行。"],"source_only":false,"title":"广义方程半可微敏感性结果","source_units":[]},{"id":"S061","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisite list：Dontchev and Rockafellar, dontchev2009implicit, Theorem 2G.9；实际使用见第 383 行。"],"source_only":false,"title":"邻域内正则性保持结果","source_units":[]},{"id":"S062","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisite list：Dontchev and Rockafellar, dontchev2009implicit, Theorem 2D.1；实际使用见第 383 行。"],"source_only":false,"title":"半可微性与单侧方向导数","source_units":[]},{"id":"S063","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisite list：Dontchev and Rockafellar, dontchev2009implicit, Equation (35)；实际使用见第 383 行。"],"source_only":false,"title":"线性化 GE 与 QP 条件对应","source_units":[]},{"id":"S064","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisite list：Nocedal and Wright, nocedal2006numerical, Lemma 16.1；实际引用见第 445 行。"],"source_only":false,"title":"正则 KKT 矩阵非奇异性","source_units":[]},{"id":"S065","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisite list：Horn and Johnson, horn2012matrix, Corollary 5.6.16；实际引用见第 315 行。"],"source_only":false,"title":"收敛矩阵 Neumann 级数","source_units":[]},{"id":"S066","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisite list：Golub and Van Loan, golub2012matrix, Corollary 8.6.2；主证明实际引用见第 520 行。"],"source_only":false,"title":"奇异值随矩阵连续","source_units":[]},{"id":"S067","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisite list 中的 standard matrix norm inequalities；几何级数求和及谱与范数对应为 implicit standard facts made explicit by annotator。"],"source_only":false,"title":"标准谱与矩阵范数事实","source_units":[]},{"id":"S068","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisite list：standard Newton–Leibniz along absolutely continuous solution paths；作者在第 541 行明确使用 Newton–Leibniz。"],"source_only":false,"title":"绝对连续路径的积分公式","source_units":[]},{"id":"S069","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":true,"title":"衰减解释与距离幂重参数化","source_units":[{"raw_tex":"One can observe that $\\Upsilon>0$ and $\\rho\\in(0,1)$ hold; consequently, we have that the upper bound of the nodal sensitivity decays exponentially as $d_\\cG(i,j)$ increases. We can also see that $(\\Upsilon,\\rho)$ depend on the singular values of the submatrices of $\\bH(\\bp^\\star)$, $\\bR(\\bp^\\star)$. {Specifically, it is necessary to have a reasonably good conditioning of the $\\bH(\\bp^\\star)[\\cB,\\cB]$ matrices satisfying $\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)$ in order to guarantee sufficiently fast decay. In the next section, we discuss under which conditions we can guarantee good conditioning in the $\\bH(\\bp^\\star)[\\cB,\\cB]$ matrices.}\n{\\blue Finally, after a suitable redefinition ($\\Upsilon\\leftarrow \\Upsilon\\rho^{-1}$ and $\\rho\\leftarrow \\rho^{1/4}$), one can obtain the result claimed in Section \\ref{sec:intro}; that is, \\eqref{eqn:sens} holds with the sensitivity coefficients $\\{C_{ij}=\\Upsilon\\rho^{ d_\\cG(i,j)}\\}_{i,j\\in \\cV}$. }\n","start":58214,"end":59180}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S049","edges":[{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S012","kind":"supports"},{"source_id":"S007","target_id":"S012","kind":"supports"},{"source_id":"S005","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S009","kind":"supports"},{"source_id":"S006","target_id":"S013","kind":"supports"},{"source_id":"S007","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S010","kind":"supports"},{"source_id":"S006","target_id":"S014","kind":"supports"},{"source_id":"S007","target_id":"S014","kind":"supports"},{"source_id":"S005","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S011","kind":"supports"},{"source_id":"S012","target_id":"S008","kind":"supports"},{"source_id":"S013","target_id":"S008","kind":"supports"},{"source_id":"S014","target_id":"S008","kind":"supports"},{"source_id":"S016","target_id":"S018","kind":"supports"},{"source_id":"S067","target_id":"S018","kind":"supports"},{"source_id":"S016","target_id":"S019","kind":"supports"},{"source_id":"S009","target_id":"S019","kind":"supports"},{"source_id":"S011","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S065","target_id":"S019","kind":"supports"},{"source_id":"S009","target_id":"S020","kind":"supports"},{"source_id":"S010","target_id":"S020","kind":"supports"},{"source_id":"S011","target_id":"S020","kind":"supports"},{"source_id":"S007","target_id":"S020","kind":"supports"},{"source_id":"S016","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S018","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S016","target_id":"S021","kind":"supports"},{"source_id":"S067","target_id":"S021","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S017","kind":"supports"},{"source_id":"S020","target_id":"S017","kind":"supports"},{"source_id":"S021","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S015","kind":"supports"},{"source_id":"S003","target_id":"S022","kind":"supports"},{"source_id":"S003","target_id":"S025","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S022","target_id":"S030","kind":"supports"},{"source_id":"S023","target_id":"S030","kind":"supports"},{"source_id":"S024","target_id":"S030","kind":"supports"},{"source_id":"S025","target_id":"S030","kind":"supports"},{"source_id":"S026","target_id":"S030","kind":"supports"},{"source_id":"S060","target_id":"S030","kind":"supports"},{"source_id":"S061","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S060","target_id":"S031","kind":"supports"},{"source_id":"S062","target_id":"S031","kind":"supports"},{"source_id":"S063","target_id":"S031","kind":"supports"},{"source_id":"S023","target_id":"S032","kind":"supports"},{"source_id":"S024","target_id":"S032","kind":"supports"},{"source_id":"S060","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S032","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S033","kind":"supports"},{"source_id":"S027","target_id":"S034","kind":"supports"},{"source_id":"S028","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S028","target_id":"S035","kind":"supports"},{"source_id":"S027","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S028","target_id":"S036","kind":"supports"},{"source_id":"S003","target_id":"S037","kind":"supports"},{"source_id":"S028","target_id":"S037","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S027","target_id":"S039","kind":"supports"},{"source_id":"S025","target_id":"S039","kind":"supports"},{"source_id":"S028","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S064","target_id":"S039","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S025","target_id":"S038","kind":"supports"},{"source_id":"S027","target_id":"S038","kind":"supports"},{"source_id":"S003","target_id":"S040","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S006","target_id":"S040","kind":"supports"},{"source_id":"S002","target_id":"S041","kind":"supports"},{"source_id":"S025","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S004","target_id":"S041","kind":"supports"},{"source_id":"S007","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S006","target_id":"S042","kind":"supports"},{"source_id":"S007","target_id":"S042","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S023","target_id":"S043","kind":"supports"},{"source_id":"S024","target_id":"S043","kind":"supports"},{"source_id":"S027","target_id":"S043","kind":"supports"},{"source_id":"S038","target_id":"S043","kind":"supports"},{"source_id":"S040","target_id":"S043","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S038","target_id":"S046","kind":"supports"},{"source_id":"S039","target_id":"S046","kind":"supports"},{"source_id":"S040","target_id":"S046","kind":"supports"},{"source_id":"S015","target_id":"S047","kind":"supports"},{"source_id":"S042","target_id":"S047","kind":"supports"},{"source_id":"S039","target_id":"S047","kind":"supports"},{"source_id":"S044","target_id":"S047","kind":"supports"},{"source_id":"S040","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S038","target_id":"S048","kind":"supports"},{"source_id":"S067","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S048","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"},{"source_id":"S037","target_id":"S050","kind":"supports"},{"source_id":"S038","target_id":"S050","kind":"supports"},{"source_id":"S001","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S023","target_id":"S049","kind":"supports"},{"source_id":"S024","target_id":"S049","kind":"supports"},{"source_id":"S051","target_id":"S049","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S027","target_id":"S049","kind":"supports"},{"source_id":"S040","target_id":"S049","kind":"supports"},{"source_id":"S004","target_id":"S049","kind":"supports"},{"source_id":"S027","target_id":"S053","kind":"supports"},{"source_id":"S023","target_id":"S053","kind":"supports"},{"source_id":"S022","target_id":"S053","kind":"supports"},{"source_id":"S026","target_id":"S053","kind":"supports"},{"source_id":"S028","target_id":"S053","kind":"supports"},{"source_id":"S035","target_id":"S053","kind":"supports"},{"source_id":"S037","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S050","target_id":"S054","kind":"supports"},{"source_id":"S023","target_id":"S055","kind":"supports"},{"source_id":"S025","target_id":"S055","kind":"supports"},{"source_id":"S038","target_id":"S055","kind":"supports"},{"source_id":"S027","target_id":"S055","kind":"supports"},{"source_id":"S050","target_id":"S055","kind":"supports"},{"source_id":"S066","target_id":"S055","kind":"supports"},{"source_id":"S024","target_id":"S057","kind":"supports"},{"source_id":"S025","target_id":"S057","kind":"supports"},{"source_id":"S026","target_id":"S057","kind":"supports"},{"source_id":"S037","target_id":"S057","kind":"supports"},{"source_id":"S064","target_id":"S057","kind":"supports"},{"source_id":"S050","target_id":"S057","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S051","target_id":"S056","kind":"supports"},{"source_id":"S043","target_id":"S058","kind":"supports"},{"source_id":"S054","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S051","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S068","target_id":"S059","kind":"supports"},{"source_id":"S067","target_id":"S059","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S055","target_id":"S052","kind":"supports"},{"source_id":"S056","target_id":"S052","kind":"supports"},{"source_id":"S057","target_id":"S052","kind":"supports"},{"source_id":"S058","target_id":"S052","kind":"supports"},{"source_id":"S059","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S069","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S049","edges":[{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S012","kind":"supports"},{"source_id":"S007","target_id":"S012","kind":"supports"},{"source_id":"S005","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S009","kind":"supports"},{"source_id":"S006","target_id":"S013","kind":"supports"},{"source_id":"S007","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S010","kind":"supports"},{"source_id":"S006","target_id":"S014","kind":"supports"},{"source_id":"S007","target_id":"S014","kind":"supports"},{"source_id":"S005","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S011","kind":"supports"},{"source_id":"S012","target_id":"S008","kind":"supports"},{"source_id":"S013","target_id":"S008","kind":"supports"},{"source_id":"S014","target_id":"S008","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S067","target_id":"S015","kind":"supports"},{"source_id":"S009","target_id":"S015","kind":"supports"},{"source_id":"S011","target_id":"S015","kind":"supports"},{"source_id":"S065","target_id":"S015","kind":"supports"},{"source_id":"S010","target_id":"S015","kind":"supports"},{"source_id":"S007","target_id":"S015","kind":"supports"},{"source_id":"S003","target_id":"S022","kind":"supports"},{"source_id":"S003","target_id":"S025","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S022","target_id":"S027","kind":"supports"},{"source_id":"S023","target_id":"S027","kind":"supports"},{"source_id":"S024","target_id":"S027","kind":"supports"},{"source_id":"S025","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S060","target_id":"S027","kind":"supports"},{"source_id":"S061","target_id":"S027","kind":"supports"},{"source_id":"S062","target_id":"S027","kind":"supports"},{"source_id":"S063","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S034","kind":"supports"},{"source_id":"S028","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S028","target_id":"S035","kind":"supports"},{"source_id":"S027","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S028","target_id":"S036","kind":"supports"},{"source_id":"S003","target_id":"S037","kind":"supports"},{"source_id":"S028","target_id":"S037","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S027","target_id":"S039","kind":"supports"},{"source_id":"S025","target_id":"S039","kind":"supports"},{"source_id":"S028","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S064","target_id":"S039","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S025","target_id":"S038","kind":"supports"},{"source_id":"S027","target_id":"S038","kind":"supports"},{"source_id":"S003","target_id":"S040","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S006","target_id":"S040","kind":"supports"},{"source_id":"S002","target_id":"S041","kind":"supports"},{"source_id":"S025","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S004","target_id":"S041","kind":"supports"},{"source_id":"S007","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S006","target_id":"S042","kind":"supports"},{"source_id":"S007","target_id":"S042","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S023","target_id":"S043","kind":"supports"},{"source_id":"S024","target_id":"S043","kind":"supports"},{"source_id":"S027","target_id":"S043","kind":"supports"},{"source_id":"S038","target_id":"S043","kind":"supports"},{"source_id":"S040","target_id":"S043","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S039","target_id":"S043","kind":"supports"},{"source_id":"S015","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S067","target_id":"S043","kind":"supports"},{"source_id":"S037","target_id":"S050","kind":"supports"},{"source_id":"S038","target_id":"S050","kind":"supports"},{"source_id":"S001","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S023","target_id":"S049","kind":"supports"},{"source_id":"S024","target_id":"S049","kind":"supports"},{"source_id":"S051","target_id":"S049","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S027","target_id":"S049","kind":"supports"},{"source_id":"S040","target_id":"S049","kind":"supports"},{"source_id":"S004","target_id":"S049","kind":"supports"},{"source_id":"S027","target_id":"S053","kind":"supports"},{"source_id":"S023","target_id":"S053","kind":"supports"},{"source_id":"S022","target_id":"S053","kind":"supports"},{"source_id":"S026","target_id":"S053","kind":"supports"},{"source_id":"S028","target_id":"S053","kind":"supports"},{"source_id":"S035","target_id":"S053","kind":"supports"},{"source_id":"S037","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S050","target_id":"S054","kind":"supports"},{"source_id":"S023","target_id":"S055","kind":"supports"},{"source_id":"S025","target_id":"S055","kind":"supports"},{"source_id":"S038","target_id":"S055","kind":"supports"},{"source_id":"S027","target_id":"S055","kind":"supports"},{"source_id":"S050","target_id":"S055","kind":"supports"},{"source_id":"S066","target_id":"S055","kind":"supports"},{"source_id":"S024","target_id":"S057","kind":"supports"},{"source_id":"S025","target_id":"S057","kind":"supports"},{"source_id":"S026","target_id":"S057","kind":"supports"},{"source_id":"S037","target_id":"S057","kind":"supports"},{"source_id":"S064","target_id":"S057","kind":"supports"},{"source_id":"S050","target_id":"S057","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S051","target_id":"S056","kind":"supports"},{"source_id":"S043","target_id":"S058","kind":"supports"},{"source_id":"S054","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S051","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S068","target_id":"S059","kind":"supports"},{"source_id":"S067","target_id":"S059","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S055","target_id":"S052","kind":"supports"},{"source_id":"S056","target_id":"S052","kind":"supports"},{"source_id":"S057","target_id":"S052","kind":"supports"},{"source_id":"S058","target_id":"S052","kind":"supports"},{"source_id":"S059","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S049","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"向量矩阵与谱记号","source_units":[{"raw_tex":"{\\it Notation:} The set of real numbers and the set of integers are denoted by $\\mathbb{R}$ and $\\mathbb{I}$, respectively. We define $\\mathbb{I}_{A}:=\\mathbb{I}\\cap A$, where $A$ is a set; $\\mathbb{I}_{>0}:=\\mathbb{I}\\cap(0,\\infty)$; $\\mathbb{I}_{\\geq 0}:=\\mathbb{I}\\cap[0,\\infty)$; $\\mathbb{R}_{>0}:=(0,\\infty)$; and $\\mathbb{R}_{\\geq 0}:=[0,\\infty)$. For $A\\subseteq X$, $f:X\\rightarrow Y$, and $x\\in X$, where $X$ and $Y$ are linear spaces, $A+x := \\{x'+x:x'\\in A\\}$ and $f(X):=\\{f(x)\\in Y:x\\in X\\}$. Vectors are treated as column vectors. We use the syntax: $[M_1;\\cdots;M_n]:=[M_1^\\top\\,\\cdots\\,M_n^\\top]^\\top$; $\\{M_i\\}_{i\\in U}:=[M_{i_1}; \\cdots; M_{i_{m}}]$; $\\{M_{i,j}\\}_{i\\in U,j\\in V}:=\\{\\{M_{i,j}^\\top\\}_{j\\in V}^\\top\\}_{i\\in U}$, where, $U=\\{i_1<\\cdots<i_{m}\\}$ and $V=\\{j_1<\\cdots<j_{n}\\}$ are strictly ordered sets. Furthermore, $v[i]$ is the $i$-th component of vector $v$; $M[i,j]$ is the $(i,j)$-th component of matrix $M$; $v[I]:=\\{v[i]\\}_{i\\in I}$; $M[I,J]:=\\{M[i,j]\\}_{i\\in I,j\\in J}$. For a function $\\phi:\\mathbb{R}^{n}\\rightarrow \\mathbb{R}$ and variable vectors $y\\in\\mathbb{R}^{p}$, $z\\in\\mathbb{R}^{q}$, $\\nabla^2_{yz}\\phi(x):=\\{\\frac{\\partial^2}{\\partial y[i]\\partial z[j]}\\phi(x)\\}_{i\\in\\mathbb{I}_{[1,p]},j\\in\\mathbb{I}_{[1,q]}}$. For a vector function $\\varphi:\\mathbb{R}^{n}\\rightarrow \\mathbb{R}^m$ and a variable vector $w\\in\\mathbb{R}^{s}$, $\\nabla_{w}\\varphi(x):=\\{\\frac{\\partial}{\\partial w[j]}\\varphi(x)[i]\\}_{i\\in\\mathbb{I}_{[1,m]},j\\in\\mathbb{I}_{[1,s]}}$. We use the shorthand notation $\\nabla^2\\phi(x):=\\nabla^2_{xx}\\phi(x)$, $\\nabla\\varphi(x):=\\nabla_x\\varphi(x)$. Vector 2-norms and induced 2-norms of matrices are denoted by $\\Vert\\cdot\\Vert$. For matrices $A$ and $B$, $A\\succ(\\succeq) B$ indicates that $A-B$ is positive (semi)-definite while $A>(\\geq) B$ denotes a componentwise inequality. The identity matrix is denoted as $\\bI$ and the zero matrix or vector are denoted as $\\bzero$. Specific notations will be introduced at the first appearance. {\\blue We denote the largest and smallest eigenvalues of a symmetric matrix and non-trivial singular values of general matrices by $\\olambda(\\cdot)$, $\\ulambda(\\cdot)$, $\\osigma(\\cdot)$, and $\\usigma(\\cdot)$}.\n\n","start":13210,"end":15419}]},{"id":"S002","semantic_type":"problem_setting","kind":"problem_setting","labels":["eqn:prob-eq","eqn:prob-ineq","eqn:prob-obj"],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"闭邻域耦合非线性规划","source_units":[{"raw_tex":"{\\it Settings:} We study gsNLPs of the following form: \n\\begin{subequations}\\label{eqn:prob}\n  \\begin{align}\n    \\min_{\\{x_i\\in\\mathbb{R}^{r_i}\\}_{i\\in \\cV}}\\;& \\sum_{i\\in \\cV}f_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]})\\label{eqn:prob-obj}\\\\\n    \\st\\;& c^E_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]}) = 0,\\;i\\in \\cV,\\;(y^E_i)\\label{eqn:prob-eq}\\\\\n                                                  &c^I_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]}) \\geq 0,\\;i\\in \\cV,\\;(y^I_i)\\label{eqn:prob-ineq},\n  \\end{align}\n\\end{subequations}\nwhere $\\cG=(\\cV,\\cE)$ is an undirected graph, $\\cV$ is a strictly ordered node set, $\\cE\\subseteq \\{\\{i,j\\}\\subseteq \\cV: i\\neq j\\}$ is the edge set, and $\\cN_\\cG[i]:=\\{j\\in \\cV:\\{i,j\\}\\in \\cE\\}\\cup\\{i\\}$ denotes the closed neighborhood\\footnote{Note that closed neighborhood is different from the standard notion of neighborhood (also called open neighborhood), defined as $\\cN_\\cG(i):=\\{j\\in \\cV:\\{i,j\\}\\in \\cE\\}$. The closed neighborhood of a node contains itself whereas the open neighborhood does not.} of $i\\in \\cV$. For each node $i\\in \\cV$, $x_i\\in\\mathbb{R}^{r_i}$ is the primal variable; $p_i\\in\\mathbb{R}^{l_i}$ is the data; $f_i:\\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{r_j}\\times\\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{l_j}\\rightarrow \\mathbb{R}$ is the objective function; $c^E_i:\\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{r_j}\\times \\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{l_j}\\rightarrow \\mathbb{R}^{m^E_i}$ is the equality constraint function; $c^I_i:\\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{r_j}\\times \\prod_{j\\in \\cN_\\cG[i]}\\mathbb{R}^{l_j}\\rightarrow \\mathbb{R}^{m^I_i}$ is the inequality constraint function; and $y^E_i\\in\\mathbb{R}^{m^E_i}$ and $y^I_i\\in\\mathbb{R}^{m^I_i}$ are the dual variables associated with \\eqref{eqn:prob-eq} and \\eqref{eqn:prob-ineq}. The objective and constraint functions of node $i$ depend on the variables $\\{x_j\\}_{j\\in \\cN_\\cG[i]}$ and data $\\{p_j\\}_{j\\in \\cN_\\cG[i]}$ of the neighboring nodes. That is, node $i$ is coupled algebraically only to its neighbors, and the topology of such connectivity is dictated by $\\cG$.\n","start":15419,"end":17527}]},{"id":"S003","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"整体规划与节点原对偶变量","source_units":[{"raw_tex":"\n{\\blue\n  We can express \\eqref{eqn:prob} as a standard NLP of the following form:\n \\begin{subequations}\n   \\begin{align}\n     P(\\bp):\\;\\min_{\\bx}\\;&\\bff(\\bx;\\bp)\\\\\n     \\st\\;&\\bc^E(\\bx;\\bp)=\\bzero,\\; (\\by^E)\\\\\n                          &\\bc^I(\\bx;\\bp)\\geq \\bzero,\\;\\;(\\by^I).\n   \\end{align}\n \\end{subequations}\n Here, we compact the notation as: $\\bx:=\\{x_i\\}_{i\\in \\cV}$; $\\by^E:=\\{y^E_i\\}_{i\\in \\cV}$; $\\by^I:=\\{y^I_i\\}_{i\\in \\cV}$; $\\bp:=\\{p_i\\}_{i\\in \\cV}$; $\\br=\\sum_{i\\in \\cV} r_i$; $\\bm^E=\\sum_{i\\in \\cV} m^E_i$; $\\bm^I=\\sum_{i\\in \\cV} m^I_i$; $\\bl = \\sum_{i\\in \\cV}l_i$; $\\bff(\\bx;\\bp):=\\sum_{i\\in \\cV}f_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]})$; $\\bc^E(\\bx;\\bp):=\\{c^E_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]})\\}_{i\\in \\cV}$; and $\\bc^I(\\bx;\\bp):=\\{c^I_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]})\\}_{i\\in \\cV}$. We also define $c_i(\\cdot):=[c^E_i(\\cdot);c^I_i(\\cdot)]$; $y_i:=[y^E_i;y^I_i]$; $z_i:=[x_i;y_i]$; $m_i=m^E_i+m^I_i$; $n_i=r_i+m_i$; $\\by:=\\{y_i\\}_{i\\in \\cV}$; $\\bz:=\\{z_i\\}_{i\\in \\cV}$;  $\\bm=\\sum_{i\\in \\cV} m_i$; $\\bn=\\sum_{i\\in \\cV} n_i$; and $\\bc(\\bx;\\bp):=\\{c_i(\\{x_j\\}_{j\\in \\cN_\\cG[i]};\\{p_j\\}_{j\\in \\cN_\\cG[i]})\\}_{i\\in \\cV}$.\n}\n","start":17527,"end":18723}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["def:distance"],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"图距离与直径","source_units":[{"raw_tex":"\\begin{definition}[Graph Distance and Diameter]\\label{def:distance}\n  The distance $d_\\cG(i,j)$ between nodes $i,j\\in \\cV$ on graph $\\cG=(\\cV,\\cE)$ is the number of edges in the shortest path connecting them; {if a shortest path does not exist, the distance is $+\\infty$}. Furthermore, the diameter $D_\\cG$ of $\\cG$ is the longest distance between any pair of nodes in $\\cV$.\n\\end{definition}\n","start":22924,"end":23317}]},{"id":"S005","semantic_type":"proposition","kind":"proposition","labels":["prop:metric"],"member_ids":["S005"],"declared_external":true,"declaration_text":["原始 prerequisite list 明确许可 B0013 的标准最短路径性质，并接受作者省略其证明。"],"source_only":false,"title":"最短路径距离性质","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:metric}\n  The distance $d_\\cG:\\cV\\times \\cV \\rightarrow \\mathbb{I}_{\\geq 0}{\\cup\\{\\infty\\}}$ is a metric on $\\cV$; that is, (a) $d_\\cG(i,j)\\geq 0$ for any $i,j\\in \\cV$; (b) $i=j$ if and only if $d_\\cG(i,j)=0$; (c) $d_\\cG(i,j)=d_\\cG(j,i)$ for any $i,j\\in \\cV$; (d) $d_\\cG(i,j)\\leq d_\\cG(i,k)+d_\\cG(k,j)$ for any $i,j,k\\in \\cV$.\n\\end{proposition}\nThe proof of this result is straightforward and thus omitted. {Note that the shortest path does not exist when the graph is not connected. } \n","start":23317,"end":23835}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"节点分块与空分区约定","source_units":[{"raw_tex":"\n{A graph-structured matrix $X\\in\\mathbb{R}^{m\\times n}$ is indexed by a graph $\\cG=(\\cV,\\cE)$ and a pair of index set families $\\cI:=\\{I_{i}\\}_{i\\in \\cV}$ and $\\cJ:=\\{J_{i}\\}_{i\\in \\cV}$ that partition\\footnote{In this paper, we call a family $\\{X_1,\\cdots,X_k\\}$ of subsets of $X$ to be a partition if $\\bigcup_{k=1}^K X_k=X$ and $X_1,\\cdots,X_k$ are disjoint; here, {\\it we allow $X_k$ to be empty sets}. Note that this differs from the standard definition of a partition, where the nonemptiness of $X_k$ is enforced. This modification allows us to handle nodes with empty variables, constraints, or data, which is useful in many settings.} $\\mathbb{I}_{[1,m]}$ and $\\mathbb{I}_{[1,n]}$, respectively. We refer to $X_{[i][j]}$ as the $[i][j]$-block of matrix $X$, which can later be associated with the node pair $(i,j)$. It should be noted that the short-hand notation $X_{[i][j]}:=X[I_i,J_j]$ assumes that the proper graph and index set families $(\\cG,\\cI,\\cJ)$ are introduced or can be inferred from the context. We are now ready to introduce the concept of graph-induced {\\it bandwidth}.} \n\n","start":23835,"end":24933}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":["def:bandwidth"],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"图诱导带宽","source_units":[{"raw_tex":"\\begin{definition}[Graph-Induced Bandwidth]\\label{def:bandwidth}\n  Consider a matrix $X\\in\\mathbb{R}^{m\\times n}$, a graph $\\cG=(\\cV,\\cE)$, and index sets $\\cI=\\{I_i\\}_{i\\in \\cV}$, $\\cJ=\\{J_i\\}_{i\\in \\cV}$ that partition $\\mathbb{I}_{[1,m]}$ and $\\mathbb{I}_{[1,n]}$, respectively. Matrix $X$ is said to have bandwidth $B$ induced by $(\\cG,\\cI,\\cJ)$, if $B$ is the smallest nonnegative integer such that $X_{[i][j]}=\\bzero$ for any $i,j\\in \\cV$ with $d_\\cG(i,j)> B$, where $X_{[i][j]}:=X[I_i,J_j]$.\n\\end{definition}\n","start":24933,"end":25449}]},{"id":"S008","semantic_type":"lemma","kind":"lemma","labels":["lem:bandwidth"],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"带宽运算法则","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:bandwidth}\n  Consider $X\\in\\mathbb{R}^{m\\times n}$ with bandwidth not greater than $B_X$ induced by $(\\cG,\\cI,\\cJ)$; we have that:\n  (a) $X^\\top$ has bandwidth not greater than $B_X$ induced by $(\\cG,\\cJ,\\cI)$;\n  (b) if $Y\\in\\mathbb{R}^{m\\times n}$ has bandwidth not greater than $B_Y$ induced by $(\\cG,\\cI,\\cJ)$, then $X+Y$ has bandwidth not greater than $\\max(B_X,B_Y)$ induced by $(\\cG,\\cI,\\cJ)$;\n  (c) if $W\\in\\mathbb{R}^{n\\times k}$ has bandwidth not greater than $B_W$ induced by $(\\cG,\\cJ,\\cK)$, then $XW$ has bandwidth not greater than $B_X+B_W$ induced by $(\\cG,\\cI,\\cK)$.\n\\end{lemma}","start":27298,"end":27915}]},{"id":"S009","semantic_type":"lemma","kind":"lemma","labels":["lem:bandwidth"],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"转置带宽上界","source_units":[{"raw_tex":"  (a) $X^\\top$ has bandwidth not greater than $B_X$ induced by $(\\cG,\\cJ,\\cI)$;\n","start":27453,"end":27533}]},{"id":"S010","semantic_type":"lemma","kind":"lemma","labels":["lem:bandwidth"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"矩阵和的带宽上界","source_units":[{"raw_tex":"  (b) if $Y\\in\\mathbb{R}^{m\\times n}$ has bandwidth not greater than $B_Y$ induced by $(\\cG,\\cI,\\cJ)$, then $X+Y$ has bandwidth not greater than $\\max(B_X,B_Y)$ induced by $(\\cG,\\cI,\\cJ)$;\n","start":27533,"end":27722}]},{"id":"S011","semantic_type":"lemma","kind":"lemma","labels":["lem:bandwidth"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"矩阵积的带宽上界","source_units":[{"raw_tex":"  (c) if $W\\in\\mathbb{R}^{n\\times k}$ has bandwidth not greater than $B_W$ induced by $(\\cG,\\cJ,\\cK)$, then $XW$ has bandwidth not greater than $B_X+B_W$ induced by $(\\cG,\\cI,\\cK)$.\n","start":27722,"end":27904}]},{"id":"S012","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"转置带宽的作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of (a)] We have that $(X^\\top)_{[i][j]}=(X^\\top){[J_i,I_j]}=(X[I_j,J_i])^\\top=(X_{[j][i]})^\\top$. From the assumption that $X$ has bandwidth not greater than $B_X$ and Proposition \\ref{prop:metric}(c), $(X^\\top)_{[i][j]}=\\bzero$ if $d_\\cG(i,j)>B_X$; therefore, $X$ has bandwidth not greater than $B_X$, and induced by $(\\cG,\\cJ,\\cI)$.\n\\end{proof}","start":27916,"end":28282}]},{"id":"S013","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"加法带宽的作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of (b)] We have that $X_{[i][j]}=\\bzero$ and $Y_{[i][j]}=\\bzero$ if $d_\\cG(i,j)>\\max(B_X,B_Y)$, which yields $(X+Y)_{[i][j]}=\\bzero$ if $d_\\cG(i,j)>\\max(B_X,B_Y)$. Thus, $X+Y$ has bandwidth not greater than $\\max(B_X,B_Y)$, and induced by $(\\cG,\\cI,\\cJ)$.\n\\end{proof}","start":28283,"end":28570}]},{"id":"S014","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"乘法带宽的作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of (c)] If $d_\\cG(i,j)>B_X+B_W$, from Proposition \\ref{prop:metric}(d) we have that for any $k\\in \\cV$, $d_\\cG(i,k)>B_X$ or $d_\\cG(j,k)>B_W$. Thus, if $d_\\cG(i,j)>B_X+B_W$,  we have $(XW)_{[i][j]}=\\sum_{k\\in \\cV} X_{[i][k]}W_{[k][j]}= \\bzero$ (where the first equality comes from the block matrix multiplication law and the second equality comes from observing that $d_\\cG(i,k)>B_X$ or $d_\\cG(j,k)>B_W$). Therefore, $XW$ has bandwidth not greater than $B_X+B_W$, and induced by $(\\cG,\\cI,\\cK)$.  \n\\end{proof}","start":28571,"end":29099}]},{"id":"S015","semantic_type":"theorem","kind":"theorem","labels":["eqn:inv","thm:inv"],"member_ids":["S015","S017","S018","S019","S020","S021"],"declared_external":false,"source_only":false,"title":"逆矩阵乘积的分块衰减","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm:inv}\n  Consider a nonsingular matrix $X\\in\\mathbb{R}^{n\\times n}$ with bandwidth not greater than $B_X\\geq 1$ induced by $(\\cG,\\cK,\\cP)$, $Y\\in\\mathbb{R}^{n\\times m}$ with bandwidth not greater than $B_Y$ induced by $(\\cG, \\cK,\\cJ)$, and $W\\in\\mathbb{R}^{\\ell\\times n}$ with bandwidth not greater than $B_W$ induced by $(\\cG, \\cI,\\cP)$; for constants $\\osigma_X\\geq \\osigma(X)$, $\\osigma_Y\\geq \\osigma(Y)$, $\\osigma_W\\geq \\osigma(W)$, and $0<\\usigma_X\\leq \\usigma(X)$, the following holds:\n  \\begin{align}\\label{eqn:inv}\n    \\Vert (WX^{-1}Y)_{[i][j]}\\Vert \\leq\n    \\frac{\\osigma_X\\osigma_Y\\osigma_W}{\\usigma_X^2}\\left(\\frac{\\osigma_X^2-\\usigma_X^2}{\\osigma_X^2+\\usigma_X^2}\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)-B_X-B_Y-B_W}{2B_X}\\right\\rceil_+} ,\\quad i,j\\in \\cV,\n  \\end{align}\n  where $(WX^{-1}Y)_{[i][j]}:=(WX^{-1}Y)[I_i,J_j]$ and $\\lceil\\cdot\\rceil_+$ is the smallest non-negative integer that is greater than or equal to the argument.\n\\end{theorem}","start":29434,"end":30412},{"raw_tex":"\\begin{proof}\n  By definition of singular values and the assumptions on $\\osigma_X$, $\\osigma_Y$ , $\\osigma_W$, and $\\usigma_X$, we have $\\usigma_X^2 \\bI \\preceq\\usigma(X)^2 \\bI \\preceq X X^\\top\\preceq  \\osigma(X)^2\\bI\\preceq\\osigma_X^2 \\bI $. From this, one can obtain:\n \\begin{align}\\label{eqn:factor}\n   \\frac{\\usigma_X^2-\\osigma_X^2}{\\usigma_X^2+\\osigma_X^2} \\bI\\preceq \\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top\n   \\preceq \\frac{-\\usigma_X^2+\\osigma_X^2}{\\usigma_X^2+\\osigma_X^2}\\bI.\n \\end{align}\n By Lemma \\ref{lem:bandwidth}(c), $XX^\\top$ has bandwidth not greater than $2B_X$ induced by $(\\cG,\\cK,\\cK)$. From the nonsingularity of $XX^\\top$, we obtain the following:\n \\begin{subequations}\\label{eqn:key-1}\n   \\begin{align}\n     WX^{-1}Y &= \\frac{2}{\\usigma_X^2+\\osigma_X^2} WX^\\top \\left(\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^{-1}Y\\\\\n   % &= \\frac{2}{\\usigma_X^2+\\osigma_X^2} WX^\\top \\left(\\bI-\\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)\\right)^{-1} Y\\\\\n     &= \\frac{2}{\\usigma_X^2+\\osigma_X^2}WX^\\top \\left(\\sum_{q=0}^\\infty \\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^q\\right)Y,\\label{eqn:key-1-c}\\\\\n     &=\\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=0}^\\infty WX^\\top\\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^q Y.\n \\end{align}\n \\end{subequations}\n Here the second equality follows from \\cite[Corollary 5.6.16]{horn2012matrix} and \\eqref{eqn:factor}, and the last equality follows from the fact that the series in \\eqref{eqn:key-1-c} converges (due to \\eqref{eqn:factor}). Furthermore, from Lemma \\ref{lem:bandwidth}, we see that $WX^\\top\\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^q Y$ has bandwidth not greater than $(2q+1)B_X +B_Y+B_W$, induced by $(\\cG,\\cI,\\cJ)$. By extracting submatrices $(\\cdot)[I_i,J_j]$ from \\eqref{eqn:key-1}, we obtain:\n  \\begin{align*}\n    (WX^{-1}Y)_{[i][j]}&=\\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=q_0(i,j)}^\\infty \\left(WX^\\top\\left(\\bI-\\frac{2XX^\\top}{\\usigma_X^2+\\osigma_X^2} \\right)^q Y\\right)_{[i][j]}\n  \\end{align*}\n  where $q_0(i,j):=\\left\\lceil \\frac{d_\\cG(i,j)-B_X-B_Y-B_W}{2B_X}\\right\\rceil_+$; the summation over $q=0,\\cdots, q_0(i,j)-1$ is zero because such $q$ satisfy $(2q+1)B_X+B_Y+B_W< d_\\cG(i,j)$. Using the triangle inequality and the fact that the matrix norm of a submatrix is smaller than that of the original matrix,\n  \\begin{align}\\label{eqn:Hbound}\n    \\Vert (WX^{-1}Y)_{[i][j]} \\Vert &\\leq \\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=q_0(i,j)}^\\infty \\left\\Vert WX^\\top\\left(\\bI-\\frac{2XX^\\top}{\\usigma_X^2+\\osigma_X^2}\\right)^q Y\\right\\Vert\\\\\n    &\\leq \\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=q_0(i,j)}^\\infty \\osigma_W\\osigma_X\\left(\\frac{\\osigma_X^2-\\usigma_X^2}{\\osigma_X^2+\\usigma_X^2}\\right)^q \\osigma_Y\\nonumber\\\\\n    &\\leq \\frac{\\osigma_X\\osigma_Y\\osigma_W}{\\usigma_X^2}\\left(\\frac{\\osigma_X^2-\\usigma_X^2}{\\osigma_X^2+\\usigma_X^2}\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)-B_X-B_Y-B_W}{2B_X}\\right\\rceil_+} .\\nonumber\n  \\end{align}\n  The second inequality follows from the submultiplicativity of the matrix norm and \\eqref{eqn:factor}; and the last inequality follows from the summation of geometric series. \n\\end{proof}","start":30413,"end":33580},{"raw_tex":"  By definition of singular values and the assumptions on $\\osigma_X$, $\\osigma_Y$ , $\\osigma_W$, and $\\usigma_X$, we have $\\usigma_X^2 \\bI \\preceq\\usigma(X)^2 \\bI \\preceq X X^\\top\\preceq  \\osigma(X)^2\\bI\\preceq\\osigma_X^2 \\bI $. From this, one can obtain:\n \\begin{align}\\label{eqn:factor}\n   \\frac{\\usigma_X^2-\\osigma_X^2}{\\usigma_X^2+\\osigma_X^2} \\bI\\preceq \\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top\n   \\preceq \\frac{-\\usigma_X^2+\\osigma_X^2}{\\usigma_X^2+\\osigma_X^2}\\bI.\n \\end{align}\n","start":30427,"end":30917},{"raw_tex":" By Lemma \\ref{lem:bandwidth}(c), $XX^\\top$ has bandwidth not greater than $2B_X$ induced by $(\\cG,\\cK,\\cK)$. From the nonsingularity of $XX^\\top$, we obtain the following:\n \\begin{subequations}\\label{eqn:key-1}\n   \\begin{align}\n     WX^{-1}Y &= \\frac{2}{\\usigma_X^2+\\osigma_X^2} WX^\\top \\left(\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^{-1}Y\\\\\n   % &= \\frac{2}{\\usigma_X^2+\\osigma_X^2} WX^\\top \\left(\\bI-\\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)\\right)^{-1} Y\\\\\n     &= \\frac{2}{\\usigma_X^2+\\osigma_X^2}WX^\\top \\left(\\sum_{q=0}^\\infty \\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^q\\right)Y,\\label{eqn:key-1-c}\\\\\n     &=\\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=0}^\\infty WX^\\top\\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^q Y.\n \\end{align}\n \\end{subequations}\n Here the second equality follows from \\cite[Corollary 5.6.16]{horn2012matrix} and \\eqref{eqn:factor}, and the last equality follows from the fact that the series in \\eqref{eqn:key-1-c} converges (due to \\eqref{eqn:factor}). Furthermore, from Lemma \\ref{lem:bandwidth}, we see that $WX^\\top\\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^q Y$ has bandwidth not greater than $(2q+1)B_X +B_Y+B_W$, induced by $(\\cG,\\cI,\\cJ)$. By extracting submatrices $(\\cdot)[I_i,J_j]$ from \\eqref{eqn:key-1}, we obtain:\n","start":30917,"end":32245},{"raw_tex":" Here the second equality follows from \\cite[Corollary 5.6.16]{horn2012matrix} and \\eqref{eqn:factor}, and the last equality follows from the fact that the series in \\eqref{eqn:key-1-c} converges (due to \\eqref{eqn:factor}). Furthermore, from Lemma \\ref{lem:bandwidth}, we see that $WX^\\top\\left(\\bI-\\frac{2}{\\usigma_X^2+\\osigma_X^2}XX^\\top \\right)^q Y$ has bandwidth not greater than $(2q+1)B_X +B_Y+B_W$, induced by $(\\cG,\\cI,\\cJ)$. By extracting submatrices $(\\cdot)[I_i,J_j]$ from \\eqref{eqn:key-1}, we obtain:\n  \\begin{align*}\n    (WX^{-1}Y)_{[i][j]}&=\\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=q_0(i,j)}^\\infty \\left(WX^\\top\\left(\\bI-\\frac{2XX^\\top}{\\usigma_X^2+\\osigma_X^2} \\right)^q Y\\right)_{[i][j]}\n  \\end{align*}\n  where $q_0(i,j):=\\left\\lceil \\frac{d_\\cG(i,j)-B_X-B_Y-B_W}{2B_X}\\right\\rceil_+$; the summation over $q=0,\\cdots, q_0(i,j)-1$ is zero because such $q$ satisfy $(2q+1)B_X+B_Y+B_W< d_\\cG(i,j)$. Using the triangle inequality and the fact that the matrix norm of a submatrix is smaller than that of the original matrix,\n","start":31730,"end":32770},{"raw_tex":"  where $q_0(i,j):=\\left\\lceil \\frac{d_\\cG(i,j)-B_X-B_Y-B_W}{2B_X}\\right\\rceil_+$; the summation over $q=0,\\cdots, q_0(i,j)-1$ is zero because such $q$ satisfy $(2q+1)B_X+B_Y+B_W< d_\\cG(i,j)$. Using the triangle inequality and the fact that the matrix norm of a submatrix is smaller than that of the original matrix,\n  \\begin{align}\\label{eqn:Hbound}\n    \\Vert (WX^{-1}Y)_{[i][j]} \\Vert &\\leq \\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=q_0(i,j)}^\\infty \\left\\Vert WX^\\top\\left(\\bI-\\frac{2XX^\\top}{\\usigma_X^2+\\osigma_X^2}\\right)^q Y\\right\\Vert\\\\\n    &\\leq \\frac{2}{\\usigma_X^2+\\osigma_X^2} \\sum_{q=q_0(i,j)}^\\infty \\osigma_W\\osigma_X\\left(\\frac{\\osigma_X^2-\\usigma_X^2}{\\osigma_X^2+\\usigma_X^2}\\right)^q \\osigma_Y\\nonumber\\\\\n    &\\leq \\frac{\\osigma_X\\osigma_Y\\osigma_W}{\\usigma_X^2}\\left(\\frac{\\osigma_X^2-\\usigma_X^2}{\\osigma_X^2+\\usigma_X^2}\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)-B_X-B_Y-B_W}{2B_X}\\right\\rceil_+} .\\nonumber\n  \\end{align}\n  The second inequality follows from the submultiplicativity of the matrix norm and \\eqref{eqn:factor}; and the last inequality follows from the summation of geometric series. \n","start":32453,"end":33569}]},{"id":"S016","semantic_type":"assumption","kind":"assumption","labels":["thm:inv"],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"逆衰减定理的矩阵条件","source_units":[{"raw_tex":"  Consider a nonsingular matrix $X\\in\\mathbb{R}^{n\\times n}$ with bandwidth not greater than $B_X\\geq 1$ induced by $(\\cG,\\cK,\\cP)$, $Y\\in\\mathbb{R}^{n\\times m}$ with bandwidth not greater than $B_Y$ induced by $(\\cG, \\cK,\\cJ)$, and $W\\in\\mathbb{R}^{\\ell\\times n}$ with bandwidth not greater than $B_W$ induced by $(\\cG, \\cI,\\cP)$; for constants $\\osigma_X\\geq \\osigma(X)$, $\\osigma_Y\\geq \\osigma(Y)$, $\\osigma_W\\geq \\osigma(W)$, and $0<\\usigma_X\\leq \\usigma(X)$, the following holds:\n","start":29465,"end":29950}]},{"id":"S022","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"基准数据与基准解","source_units":[{"raw_tex":"We use $\\bz^\\star \\in\\mathbb{R}^{\\bn}$ to denote the primal-dual solution of $P(\\bp^\\star)$. We refer to $\\bp^\\star$ as the base data and $\\bz^\\star$ as the base solution. We denote the primal and dual components of $\\bz^\\star=\\{z^\\star_i\\}_{i\\in\\cV}=\\{[x^\\star_i;y^\\star_i]\\}_{i\\in\\cV}$ as $\\bx^\\star:=\\{x^\\star_i\\}_{i\\in\\cV}$ and $\\by^\\star=\\{y^\\star_i\\}_{i\\in\\cV}$, respectively. We say that $\\bz^\\star$ is a primal-dual solution if $\\bx^\\star$ satisfies the first-order optimality conditions with Lagrange multiplier $\\by^\\star$ (see \\cite{nocedal2006numerical}). We now make key assumptions that are necessary to establish our main results. \n\n","start":36376,"end":37024}]},{"id":"S023","semantic_type":"assumption","kind":"assumption","labels":["ass:c2"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"二阶连续可微假设","source_units":[{"raw_tex":"\\begin{assumption}[Twice Continuous Differentiability of Functions]\\label{ass:c2}\n  The functions $\\bff:\\mathbb{R}^{\\br}\\times \\mathbb{R}^{\\bl}\\rightarrow \\mathbb{R}$ and $\\bc:\\mathbb{R}^{\\br}\\times \\mathbb{R}^{\\bl}\\rightarrow \\mathbb{R}^{\\bm}$ are twice continuously differentiable in the neighborhood of $[\\bx^\\star;\\bp^\\star]$.\n\\end{assumption}\n\n","start":37024,"end":37373}]},{"id":"S024","semantic_type":"assumption","kind":"assumption","labels":["ass:regularity"],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"基准解正则性假设","source_units":[{"raw_tex":"\\begin{assumption}[Regularity of Solution]\\label{ass:regularity}\n  The base solution $\\bz^\\star$ satisfies SSOSC and LICQ for $P(\\bp^\\star)$ with base data $\\bp^\\star$.\n\\end{assumption}\n","start":37373,"end":37559}]},{"id":"S025","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"拉格朗日函数与正则性含义","source_units":[{"raw_tex":"We recall that SSOSC requires positive definiteness of the reduced Hessian of the Lagrangian at $\\bz^\\star$. The reduced Hessian is the Hessian projected on the null space defined by equality constraints and active inequality constraints with nonzero duals. LICQ requires that the constraint Jacobian defined by equality and active inequality constraints are linearly independent at $\\bz^\\star$. These requirements are stated as: \n\\begin{align*}\n  ReH(\\nabla^2_{\\bx\\bx}\\cL(\\bz^\\star;\\bp^\\star),\\nabla_{\\bx}\\bc(\\bx^\\star;\\bp^\\star)[\\cA^0(\\bp^\\star),:]) &\\succ \\bzero\\tag{SSOSC}\\\\\n  \\usigma(\\nabla_{\\bx}\\bc(\\bx^\\star;\\bp^\\star)[\\cA^1(\\bp^\\star),:]) &> 0 \\tag{LICQ}.\n\\end{align*}\nHere, $\\cL(\\bz;\\bp) := \\bff(\\bx;\\bp)-\\by^\\top \\bc(\\bx;\\bp)$ is the Lagrange function, $ReH(H,A):=Z^\\top HZ$ is the reduced Hessian, where $Z$ is a null-space matrix of $A$, and\n","start":37559,"end":38413}]},{"id":"S026","semantic_type":"definition","kind":"definition","labels":["eqn:active-set-defs"],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"基点活动约束集合","source_units":[{"raw_tex":"\\begin{align}\\label{eqn:active-set-defs}\n  \\cA^0(\\bp^\\star)&:=\\cA^E\\cup\\{i\\in\\cA^I:\\bc(\\bx^\\star)[i]=0,\\;\\by^\\star[i]\\neq 0\\}\\\\\n  \\cA^1(\\bp^\\star)&:=\\cA^E\\cup\\{i\\in\\cA^I:\\bc(\\bx^\\star)[i]=0\\},\n\\end{align}\nwhere $\\cA^E$ and $\\cA^I$ are the set of equality and inequality constraint indices. Hereafter, we call a solution of gsNLP regular if the solution satisfies SSOSC and LICQ. Note that SSOSC and LICQ are standard assumptions used in NLP sensitivity theory \\cite{robinson1980strongly}. We now state the classical sensitivity result.\n","start":38413,"end":38949}]},{"id":"S027","semantic_type":"lemma","kind":"lemma","labels":["eqn:AAQA","eqn:nlp-sensitivity-0","eqn:nlp-sensitivity-0-obj","eqn:one-sided-DD","lem:nlp-sensitivity"],"member_ids":["S027","S029","S030","S031","S032"],"declared_external":false,"source_only":false,"title":"经典局部敏感性引理","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:nlp-sensitivity}\n  Under Assumptions \\ref{ass:c2}, \\ref{ass:regularity}, there exist neighborhoods $\\mathbb{P}\\subseteq\\mathbb{R}^{\\bl}$ of $\\bp^\\star$ and $\\bbZ\\subseteq\\mathbb{R}^{\\bn}$ of $\\bz^\\star$ and a continuous function $\\bz^\\dag:\\mathbb{P}\\rightarrow \\bbZ$ such that $\\bz^\\dag(\\bp)$ is a primal-dual solution of $P(\\bp)$ that satisfies SSOSC and LICQ. Furthermore, for any $\\bq:=\\{q_i\\}_{i\\in \\cV}\\in\\mathbb{R}^{\\bl}$, the one-sided directional derivative of $\\bz^\\dag(\\cdot)$ is given by:\n  \\begin{align*}\n    D_{\\bq} \\bz^\\dag(\\bp)&:=\\lim_{h\\searrow 0}\\frac{\\bz^\\dag(\\bp+\\bq h)-\\bz^\\dag(\\bp)}{h}; \\label{eqn:one-sided-DD}\n  \\end{align*}\n  with $D_{\\bq}\\bx^\\dag(\\bp) = \\bxi^\\dag(\\bp,\\bq)$, $D_{\\bq}\\by^\\dag(\\bp,\\bq) = \\bbeta^\\dag(\\bp,\\bq)$, where $(\\bxi^\\dag(\\bp,\\bq),\\bbeta^\\dag(\\bp,\\bq)[\\cA^1(\\bp)])$ is a primal-dual solution of the quadratic program:\n  \\begin{subequations}\\label{eqn:nlp-sensitivity-0}\n    \\begin{align}\n      QP(\\bp,\\bq):\\quad\\min_{\\bxi}\\;&\\frac{1}{2}\\bxi^\\top\\bQ(\\bp)\\bxi + \\bxi^\\top\\bS(\\bp)\\bq,\\label{eqn:nlp-sensitivity-0-obj}\\\\\n      \\st\\;&\\bA(\\bp)[i,:]^\\top\\bxi + \\bB(\\bp)[i,:]\\bq = 0,\\;i\\in \\cA^0(\\bp),\\; (\\bbeta[i])\\\\\n                   &\\bA(\\bp)[i,:]^\\top\\bxi + \\bB(\\bp)[i,:]\\bq \\geq 0,\\;i\\in \\cA^1(\\bp)\\setminus \\cA^0(\\bp),\\; (\\bbeta[i])\n    \\end{align}      \n  \\end{subequations}\n  and $\\bbeta^\\dag(\\bp,\\bq)[\\mathbb{I}_{[1,\\bm]}\\setminus\\cA^1(\\bp)]=\\bzero$ (i.e., the free dual variables are fixed to zero); here,\n  \\begin{subequations}\\label{eqn:AAQA}\n    \\begin{align}\n      \\cA^0(\\bp)&:=\\cA^E\\cup\\{i\\in\\cA^I:\\bc(\\bx^\\dag(\\bp))[i]=0,\\;\\by^\\dag(\\bp)[i]\\neq 0\\}\\\\\n      \\cA^1(\\bp)&:=\\cA^E\\cup\\{i\\in\\cA^I:\\bc(\\bx^\\dag(\\bp))[i]=0\\}\\\\\n      \\bQ(\\bp)&:=\\nabla_{\\bx\\bx}\\cL(\\bz^\\dag(\\bp);\\bp);\\quad \\bS(\\bp):=\\nabla_{\\bx\\bp}\\cL(\\bz^\\dag(\\bp);\\bp)\\\\\n      \\bA(\\bp)&:=\\nabla_{\\bx}\\bc(\\bx^\\dag(\\bp);\\bp);\\quad \\bB(\\bp):=\\nabla_{\\bp}\\bc(\\bx^\\dag(\\bp);\\bp).\n    \\end{align}\n  \\end{subequations}\n  Moreover, a unique primal-dual solution of $QP(\\bp,\\bq)$ exists for any $\\bp\\in\\mathbb{P}$ and $\\bq\\in\\mathbb{R}^{\\bl}$; thus, $\\bxi^\\dag(\\cdot,\\cdot)$ and $\\bbeta^\\dag(\\cdot,\\cdot)$ are well-defined.\n\\end{lemma}","start":38950,"end":41114},{"raw_tex":"\\begin{proof}\n  The results in \\cite[Theorem 2G.8]{dontchev2009implicit} ensure semidifferentiability (which guarantees continuity) of the solution of the first-order optimality conditions for $P(\\bp)$ over a certain neighborhood of $\\bp^\\star$. Furthermore, \\cite[Theorem 2G.9]{dontchev2009implicit} establishes that over a certain neighborhood $\\mathbb{P}$, the solution mapping of the generalized equation (GE) satisfies SSOSC and LICQ; that is, within $\\mathbb{P}$, the solution mapping for the GE is the solution mapping for $P(\\bp)$ at which SSOSC and LICQ are satisfied. Moreover, by \\cite[Theorem 2G.8]{dontchev2009implicit}, the one-sided directional derivative of the solution mapping for the GE (which exists for any direction $\\bq\\in\\mathbb{R}^{\\bl}$ by semidifferentiability \\cite[Theorem 2D.1]{dontchev2009implicit}) can be evaluated by using the linearized GE. The linear GE corresponds to the first-order optimality conditions for $QP(\\bp,\\bq)$ (see \\cite[Equation (35)]{dontchev2009implicit}); here, the first-order conditions are necessary and sufficient conditions for the optimality due to the convexity $QP(\\bp,\\bq)$ (guaranteed by SSOSC and LICQ of $\\bz^\\star$ for the original problem). As such, the one-sided directional derivative of the solution mapping for $P(\\bp)$ can be evaluated by solving $QP(\\bp,\\bq)$. Finally, the strong regularity of the GE at $\\bz^\\star$ guarantees that there exists a unique solution of the linearized GE, which in turn guarantees the existence of a unique solution of $QP(\\bp,\\bq)$.  \n\\end{proof}","start":41310,"end":42862},{"raw_tex":"  The results in \\cite[Theorem 2G.8]{dontchev2009implicit} ensure semidifferentiability (which guarantees continuity) of the solution of the first-order optimality conditions for $P(\\bp)$ over a certain neighborhood of $\\bp^\\star$. Furthermore, \\cite[Theorem 2G.9]{dontchev2009implicit} establishes that over a certain neighborhood $\\mathbb{P}$, the solution mapping of the generalized equation (GE) satisfies SSOSC and LICQ; that is, within $\\mathbb{P}$, the solution mapping for the GE is the solution mapping for $P(\\bp)$ at which SSOSC and LICQ are satisfied. Moreover, by \\cite[Theorem 2G.8]{dontchev2009implicit}, the one-sided directional derivative of the solution mapping for the GE (which exists for any direction $\\bq\\in\\mathbb{R}^{\\bl}$ by semidifferentiability \\cite[Theorem 2D.1]{dontchev2009implicit}) can be evaluated by using the linearized GE. The linear GE corresponds to the first-order optimality conditions for $QP(\\bp,\\bq)$ (see \\cite[Equation (35)]{dontchev2009implicit}); here, the first-order conditions are necessary and sufficient conditions for the optimality due to the convexity $QP(\\bp,\\bq)$ (guaranteed by SSOSC and LICQ of $\\bz^\\star$ for the original problem). As such, the one-sided directional derivative of the solution mapping for $P(\\bp)$ can be evaluated by solving $QP(\\bp,\\bq)$. Finally, the strong regularity of the GE at $\\bz^\\star$ guarantees that there exists a unique solution of the linearized GE, which in turn guarantees the existence of a unique solution of $QP(\\bp,\\bq)$.  \n","start":41324,"end":42851}]},{"id":"S028","semantic_type":"definition","kind":"definition","labels":["eqn:AAQA","eqn:nlp-sensitivity-0","lem:nlp-sensitivity"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"移动活动集合与 QP 系数","source_units":[{"raw_tex":"  \\begin{subequations}\\label{eqn:AAQA}\n    \\begin{align}\n      \\cA^0(\\bp)&:=\\cA^E\\cup\\{i\\in\\cA^I:\\bc(\\bx^\\dag(\\bp))[i]=0,\\;\\by^\\dag(\\bp)[i]\\neq 0\\}\\\\\n      \\cA^1(\\bp)&:=\\cA^E\\cup\\{i\\in\\cA^I:\\bc(\\bx^\\dag(\\bp))[i]=0\\}\\\\\n      \\bQ(\\bp)&:=\\nabla_{\\bx\\bx}\\cL(\\bz^\\dag(\\bp);\\bp);\\quad \\bS(\\bp):=\\nabla_{\\bx\\bp}\\cL(\\bz^\\dag(\\bp);\\bp)\\\\\n      \\bA(\\bp)&:=\\nabla_{\\bx}\\bc(\\bx^\\dag(\\bp);\\bp);\\quad \\bB(\\bp):=\\nabla_{\\bp}\\bc(\\bx^\\dag(\\bp);\\bp).\n    \\end{align}\n  \\end{subequations}\n","start":40430,"end":40900}]},{"id":"S034","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:kkt-compl","eqn:kkt-du","eqn:kkt-pr-1","eqn:kkt-pr-2"],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"方向 QP 的最优性条件","source_units":[{"raw_tex":"We now further analyze $QP(\\bp,\\bq)$; its first-order conditions are as follows:\n\\begin{subequations}\\label{eqn:kkt}\n  \\begin{align}\n    \\bQ(\\bp)\\bxi + \\bS(\\bp)\\bq + \\bA(\\bp)[\\cA^{1}(\\bp),:]^\\top \\bbeta &= \\bzero\\label{eqn:kkt-du}\\\\\n    \\bA(\\bp)[i,:]\\bxi + \\bB(\\bp)[i,:]\\bq &= 0,\\; i\\in\\cA^0(\\bp)\\label{eqn:kkt-pr-1}\\\\\n    \\bA(\\bp)[i,:]\\bxi + \\bB(\\bp)[i,:]\\bq &\\geq 0,\\;i\\in\\cA^1(\\bp)\\setminus\\cA^0(\\bp)\\label{eqn:kkt-pr-2}\\\\\n    \\bbeta[i] &\\geq 0,\\;i\\in\\cA^1(\\bp)\\setminus\\cA^0(\\bp)\\\\\n    \\left(\\bA(\\bp)[i,:]\\bxi + \\bB(\\bp)[i,:]\\bq \\right)\\bbeta[i]&=0,\\; i\\in\\cA^1(\\bp)\\setminus\\cA^0(\\bp).\\label{eqn:kkt-compl}\n  \\end{align}\n\\end{subequations}\nUnder SSOSC and LICQ for $P(\\bp^\\star)$ at $\\bz^\\star$, these conditions are necessary and sufficient for optimality. From the complementarity condition \\eqref{eqn:kkt-compl}, one can observe that there exists $\\cA^0(\\bp)\\subseteq\\cA'(\\bp,\\bq)\\subseteq \\cA^1(\\bp)$ such that:\n","start":43324,"end":44245}]},{"id":"S035","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"互补条件选取方向活动集","source_units":[{"raw_tex":"Under SSOSC and LICQ for $P(\\bp^\\star)$ at $\\bz^\\star$, these conditions are necessary and sufficient for optimality. From the complementarity condition \\eqref{eqn:kkt-compl}, one can observe that there exists $\\cA^0(\\bp)\\subseteq\\cA'(\\bp,\\bq)\\subseteq \\cA^1(\\bp)$ such that:\n\\begin{subequations}\\label{eqn:compl-explicit}\n\\begin{align}\n  \\bA(\\bp)[i,:]\\bxi + \\bB(\\bp)[i,:]\\bq &= 0,\\; i\\in \\cA'(\\bp,\\bq)\\\\\n  \\bbeta[i] &= 0,\\; i\\in \\cA^1(\\bp)\\setminus \\cA'(\\bp,\\bq).\n\\end{align}\n\\end{subequations}\nare satisfied at $\\bxi^\\dag(\\bp,\\bq)$, $\\bbeta^\\dag(\\bp,\\bq)$. Thus, from \\eqref{eqn:kkt}, \\eqref{eqn:compl-explicit}, and $\\bbeta[i]=0$ for $i\\in\\mathbb{I}_{[1,\\bm]}\\setminus \\cA^1(\\bp)$ (from Lemma \\ref{lem:nlp-sensitivity}), we have:\n","start":43969,"end":44702}]},{"id":"S036","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:linear-form-0"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"活动坐标上的线性系统","source_units":[{"raw_tex":"\\begin{align}\\label{eqn:linear-form-0}\n  \\begin{aligned}\n  \\begin{bmatrix}\n    \\bQ(\\bp)& \\bA(\\bp)[\\cA'(\\bp,\\bq),:]^\\top\\\\\n    \\bA(\\bp)[\\cA'(\\bp,\\bq),:]\n  \\end{bmatrix}\n  \\begin{bmatrix}\n    \\bxi\\\\\n    \\bbeta[\\cA'(\\bp,\\bq)]\n  \\end{bmatrix} &= -\n  \\begin{bmatrix}\n    \\bS(\\bp)\\\\  \n    \\bB(\\bp)[\\cA'(\\bp,\\bq),:]\n  \\end{bmatrix}\\bq\\\\\n  \\bbeta [\\mathbb{I}_{[1,\\bm]}\\setminus \\cA'(\\bp,\\bq)] &= \\bzero.\n\\end{aligned}\n\\end{align}\n% The linear equation \\eqref{eqn:linear-form-0} provides a relationship between $[\\bxi^\\dag(\\bp,\\bq);\\bbeta^\\dag(\\bp,\\bq)]$ and $\\bq$; however, it does not provide an explicit relationship  because $\\cA'(\\bp,\\bq)$ depends on $\\bq$.\n","start":44702,"end":45356}]},{"id":"S037","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"节点重排与选中坐标集合","source_units":[{"raw_tex":"By rearranging $[\\bxi^\\dag(\\bp,\\bq);\\bbeta^\\dag(\\bp,\\bq)]$ one can obtain $\\bzeta^\\dag(\\bp)=\\{[\\xi_i^\\dag(\\bp,\\bq);\\eta_i^\\dag(\\bp,\\bq)]\\}_{i\\in\\cV}$, where $\\bxi^\\dag(\\bp,\\bq)=\\{\\xi_i^\\dag(\\bp,\\bq)\\}_{i\\in\\cV}$, $\\bbeta^\\dag(\\bp,\\bq)=\\{\\eta_i^\\dag(\\bp,\\bq)\\}_{i\\in\\cV}$. To perform such rearrangement, we consider a permutation $\\phi:\\mathbb{I}_{[1,\\bn]}\\rightarrow\\mathbb{I}_{[1,\\bn]}$ that achieves $\\bzeta[\\phi(i)] = [\\bxi;\\bbeta][i]$. This permutation enables the following definition:\n\\begin{subequations}\\label{eqn:Bs}\n\\begin{align}\n  \\cB^0(\\bp)&:=\\phi(\\mathbb{I}_{[1,\\br]}\\cup(\\cA^0(\\bp)+\\br)),\\quad\n                      \\cB^1(\\bp):=\\phi(\\mathbb{I}_{[1,\\br]}\\cup(\\cA^1(\\bp)+\\br))\\\\\n  \\cB'(\\bp,\\bq)&:=\\phi(\\mathbb{I}_{[1,\\br]}\\cup(\\cA'(\\bp,\\bq)+\\br)).\n\\end{align}\n\\end{subequations}\n","start":45356,"end":46147}]},{"id":"S038","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:HR","eqn:linear-form-b"],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"方向导数的 Hessian 逆表示","source_units":[{"raw_tex":"Finally, using the notation in \\eqref{eqn:Bs}, we obtain the following from \\eqref{eqn:linear-form-0}:\n\\begin{subequations}\\label{eqn:linear-form}\n\\begin{align}\n  D_{\\bq}\\bz^\\dag(\\bp)[\\cB'(\\bp,\\bq)] &= -\\left(\\bH(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)]\\right)^{-1}\\bR(\\bp)[\\cB'(\\bp,\\bq),:] \\bq,\\\\\n  D_{\\bq}\\bz^\\dag(\\bp)[\\mathbb{I}_{[1,\\bn]}\\setminus\\cB'(\\bp,\\bq)] &= \\bzero,\\label{eqn:linear-form-b}\n\\end{align}\n\\end{subequations}\nwhere:\n\\begin{align}\\label{eqn:HR}\n  \\bH(\\bp) &:= \\nabla_{\\bz\\bz}\\cL(\\bz^\\dag(\\bp);\\bp);\\quad \\bR(\\bp) := \\nabla_{\\bz\\bp}\\cL(\\bz^\\dag(\\bp);\\bp).\n\\end{align}\n","start":46147,"end":46729}]},{"id":"S039","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"活动 Hessian 子矩阵非奇异","source_units":[{"raw_tex":"Here, the nonsingularity of $\\bH(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)]$\n% is guaranteed by the fact that $\\bQ(\\bp)$ is positive definite on the null-space of $\\bA(\\bp)[\\cA^0(\\bp),:]$. This\nfollows from LICQ, SSOSC, and \\cite[Lemma 16.1]{nocedal2006numerical}.\n","start":46729,"end":46985}]},{"id":"S040","semantic_type":"notation","kind":"notation","labels":["eqn:ind-1"],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"节点坐标族与 Hessian 块","source_units":[{"raw_tex":"\\subsection{Nodal Sensitivity Result}\\label{sec:nodal}\n{We now establish the main nodal sensitivity result. First, we consider $\\bH(\\bp)$ and $\\bR(\\bp)$ as graph-structured matrices.} Specifically, we introduce the index set families $\\cI:=\\{I_i\\}_{i\\in\\cV}$ and $\\cK:=\\{K_i\\}_{i\\in\\cV}$with:\n\\begin{align}\\label{eqn:ind-1}\n  I_i &:= \\mathbb{I}_{\\sum_{j\\in \\cV,j<i}n_j+[1,n_i]},\\quad\n  K_i := \\mathbb{I}_{\\sum_{j\\in \\cV,j<i}l_j+[1,l_i]},\n\\end{align}\nand impose graph structures to $\\bH(\\bp)$ and $\\bR(\\bp)$ with $(\\cG,\\cI,\\cI)$ and $(\\cG,\\cI,\\cK)$, respectively. Note that $\\cI$ and $\\cK$ partition $\\mathbb{I}_{[1,\\bn]}$ and $\\mathbb{I}_{[1,\\bl]}$, respectively. For $i,j\\in\\cV$,\n\\begin{subequations}\\label{eqn:HRij}\n\\begin{align}\n  H_{ij}(\\bp)&:=\\nabla^2_{z_iz_j}\\cL(\\bz^\\dag(\\bp);\\bp)=(\\bH(\\bp))_{[i][j]}\\\\\n  R_{ij}(\\bp)&:=\\nabla^2_{z_ip_j}\\cL(\\bz^\\dag(\\bp);\\bp)=(\\bR(\\bp))_{[i][j]}.\n\\end{align}\n\\end{subequations}\n","start":46986,"end":47904}]},{"id":"S041","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"闭邻域耦合推出带宽二","source_units":[{"raw_tex":"{We now can observe that $\\bH(\\bp)$ and $\\bR(\\bp)$ have bandwidth not greater than two. In other words, if $d_\\mathcal{G}(i,j)> 2$, $\\nabla_{z_i z_j}\\mathcal{L}(\\bz;\\bp)=0$ and $\\nabla_{z_i p_j}\\mathcal{L}(\\bz;\\bp)=0$. This is because in the graph-structured nonlinear programming setting in (1.1), we have assumed that the nodal objective and constraint functions are only dependent on the variables that are within the closed neighborhood of the associated node.}\n% The reason that the bandwidth may be greater than one is as follows. Suppose that $\\cV=\\{1,2,3\\}$ and $\\cE=\\{\\{1,2\\},\\{2,3\\}\\}$; then the constraints at node $2$ can be coupled with variables/data at node $1$ and $3$. This produces potential coupling between the variables that are within distance $2$ can have potential coupling.\n\n","start":47904,"end":48704}]},{"id":"S042","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"限制子矩阵继承带宽","source_units":[{"raw_tex":"The submatrices of $\\bH(\\bp)$ and $\\bR(\\bp)$ are also graph-structured. In particular, $\\bH(\\bp)[\\cB,\\cB]$ and $\\bR(\\bp)[\\cB,:]$ with $\\cB\\subseteq \\mathbb{I}_{[1,\\bn]}$ have bandwidth not greater than two induced by $(\\cG,\\cI^\\cB,\\cI^\\cB)$ and $(\\cG,\\cI^\\cB,\\cK)$, where $\\cI^\\cB:=\\{I_i\\cap\\cB\\}_{i\\in\\cV}$. We thus see that \\eqref{eqn:linear-form} involves the inverse of a {graph-induced banded} matrix. By combining Theorem \\ref{thm:inv} and Lemma \\ref{lem:nlp-sensitivity}, we obtain the following result.\n","start":48704,"end":49215}]},{"id":"S043","semantic_type":"lemma","kind":"lemma","labels":["eqn:dder-0","eqn:dder-0-c","lem:dder"],"member_ids":["S043","S045","S046","S047","S048"],"declared_external":false,"source_only":false,"title":"节点单侧方向导数界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:dder}\n  Suppose Assumptions \\ref{ass:c2}, \\ref{ass:regularity} hold, and suppose that, for given $\\bp\\in\\mathbb{P}$ (defined in Lemma \\ref{lem:nlp-sensitivity}) and $\\bq:=\\{q_i\\}_{i\\in \\cV}\\in\\mathbb{R}^{\\bl}$, we have $\\osigma_{\\bH}(\\bp,\\bq)$, $\\osigma_{\\bR}(\\bp,\\bq)$, and $\\usigma_{\\bH}(\\bp,\\bq)$ such that:\n  \\begin{subequations}\\label{eqn:dder-0}\n  \\begin{align}\n    \\osigma_{\\bH}(\\bp,\\bq)&\\geq \\osigma(\\bH(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)])\\\\\n    \\osigma_{\\bR}(\\bp,\\bq)&\\geq \\osigma(\\bR(\\bp)[\\cB'(\\bp,\\bq),:])\\\\\n    0<\\usigma_{\\bH}(\\bp,\\bq)&\\leq \\usigma(\\bH(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)]);\\label{eqn:dder-0-c}\n  \\end{align}\n\\end{subequations}\nthen the following holds for any $i\\in \\cV$:\n  \\begin{align*}\n    \\Vert D_{\\bq} z^\\dag_i(\\bp,\\bq)\\Vert\\leq \\sum_{j\\in \\cV}\\frac{\\osigma_{\\bH}(\\bp,\\bq)\\osigma_{\\bR}(\\bp,\\bq)}{\\usigma_{\\bH}(\\bp,\\bq)^2}\\left(\\frac{\\osigma_{\\bH}(\\bp,\\bq)^2-\\usigma_{\\bH}(\\bp,\\bq)^2}{\\osigma_{\\bH}(\\bp,\\bq)^2+\\usigma_{\\bH}(\\bp,\\bq)^2}\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert.\n  \\end{align*}\n\\end{lemma}","start":49216,"end":50305},{"raw_tex":"\\begin{proof}\n  {\\blue For simplicity, we denote $\\cB'(\\bp,\\bq)$, $\\osigma_{\\bH}(\\bp,\\bq)$, $\\osigma_{\\bR}(\\bp,\\bq)$, $\\usigma_{\\bH}(\\bp,\\bq)$ by $\\cB'$, $\\osigma_{\\bH}$, $\\osigma_{\\bR}$, $\\usigma_{\\bH}$}. From that $\\bH(\\bp)[\\cB',\\cB']$ is always nonsingular, we have that $\\usigma_{\\bH}(\\bp,\\bq)$ satisfying \\eqref{eqn:dder-0-c} always exists. By inspecting the block structure of \\eqref{eqn:linear-form} we obtain:\n  \\begin{align}\\label{eqn:nlp-sensitivity-1}\n    D_{\\bq}\\bz^\\dag(\\bp)[I_i\\cap\\cB'] &= \\sum_{j\\in \\cV} -((\\bH(\\bp)[\\cB',\\cB'])^{-1}\\bR(\\bp)[\\cB',:])_{[i][j]} q_j,\n  \\end{align}\n  where $\\cI^{\\cB'}:=\\{I_i\\cap{\\cB'}\\}_{i\\in \\cV}$ and $\\cK:=\\{K_i\\}_{i\\in \\cV}$ (defined in \\eqref{eqn:ind-1}) are used for index sets. We have already established that $\\bH(\\bp)[\\cB',\\cB']$ has bandwidth not greater than two, induced by $(\\cG,\\cI^{\\cB'},\\cI^{\\cB'})$ and that $\\bR(\\bp)[\\cB',:]$ has bandwidth not greater than two, induced by $(\\cG,\\cI^{\\cB'},\\cK)$. By applying Theorem \\ref{thm:inv}, we obtain:\n  \\begin{align}\\label{eqn:nlp-sensitivity-2}\\blue\n    \\Vert((\\bH(\\bp)[\\cB',\\cB'])^{-1}\\bR(\\bp)[\\cB',:])_{[i][j]}\\Vert\\leq \\frac{\\osigma_{\\bH}\\osigma_{\\bR}}{\\usigma_{\\bH}^2}\\left(\\frac{\\osigma_{\\bH}^2-\\usigma_{\\bH}^2}{\\osigma_{\\bH}^2+\\usigma_{\\bH}^2}\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+} .\n  \\end{align}\n  Now note that\n  $\\Vert D_{\\bq}z^\\dag_i(\\bp)\\Vert \\leq \\Vert D_{\\bq}\\bz^\\dag(\\bp)[I_i\\cap\\cB'] \\Vert  + \\Vert D_{\\bq}\\bz^\\dag(\\bp)[I_i\\setminus\\cB']\\Vert$,\n  and recall from \\eqref{eqn:linear-form-b} that $D_{\\bq}\\bz^\\dag(\\bp)[I_i\\setminus\\cB'] =\\bzero$. Hence, by applying \\eqref{eqn:nlp-sensitivity-2} to \\eqref{eqn:nlp-sensitivity-1}, and applying triangle inequality, we obtain the desired result.\n\\end{proof}","start":50306,"end":52052},{"raw_tex":"  {\\blue For simplicity, we denote $\\cB'(\\bp,\\bq)$, $\\osigma_{\\bH}(\\bp,\\bq)$, $\\osigma_{\\bR}(\\bp,\\bq)$, $\\usigma_{\\bH}(\\bp,\\bq)$ by $\\cB'$, $\\osigma_{\\bH}$, $\\osigma_{\\bR}$, $\\usigma_{\\bH}$}. From that $\\bH(\\bp)[\\cB',\\cB']$ is always nonsingular, we have that $\\usigma_{\\bH}(\\bp,\\bq)$ satisfying \\eqref{eqn:dder-0-c} always exists. By inspecting the block structure of \\eqref{eqn:linear-form} we obtain:\n  \\begin{align}\\label{eqn:nlp-sensitivity-1}\n    D_{\\bq}\\bz^\\dag(\\bp)[I_i\\cap\\cB'] &= \\sum_{j\\in \\cV} -((\\bH(\\bp)[\\cB',\\cB'])^{-1}\\bR(\\bp)[\\cB',:])_{[i][j]} q_j,\n  \\end{align}\n  where $\\cI^{\\cB'}:=\\{I_i\\cap{\\cB'}\\}_{i\\in \\cV}$ and $\\cK:=\\{K_i\\}_{i\\in \\cV}$ (defined in \\eqref{eqn:ind-1}) are used for index sets. We have already established that $\\bH(\\bp)[\\cB',\\cB']$ has bandwidth not greater than two, induced by $(\\cG,\\cI^{\\cB'},\\cI^{\\cB'})$ and that $\\bR(\\bp)[\\cB',:]$ has bandwidth not greater than two, induced by $(\\cG,\\cI^{\\cB'},\\cK)$. By applying Theorem \\ref{thm:inv}, we obtain:\n","start":50320,"end":51314},{"raw_tex":"  where $\\cI^{\\cB'}:=\\{I_i\\cap{\\cB'}\\}_{i\\in \\cV}$ and $\\cK:=\\{K_i\\}_{i\\in \\cV}$ (defined in \\eqref{eqn:ind-1}) are used for index sets. We have already established that $\\bH(\\bp)[\\cB',\\cB']$ has bandwidth not greater than two, induced by $(\\cG,\\cI^{\\cB'},\\cI^{\\cB'})$ and that $\\bR(\\bp)[\\cB',:]$ has bandwidth not greater than two, induced by $(\\cG,\\cI^{\\cB'},\\cK)$. By applying Theorem \\ref{thm:inv}, we obtain:\n  \\begin{align}\\label{eqn:nlp-sensitivity-2}\\blue\n    \\Vert((\\bH(\\bp)[\\cB',\\cB'])^{-1}\\bR(\\bp)[\\cB',:])_{[i][j]}\\Vert\\leq \\frac{\\osigma_{\\bH}\\osigma_{\\bR}}{\\usigma_{\\bH}^2}\\left(\\frac{\\osigma_{\\bH}^2-\\usigma_{\\bH}^2}{\\osigma_{\\bH}^2+\\usigma_{\\bH}^2}\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+} .\n  \\end{align}\n","start":50900,"end":51638},{"raw_tex":"  Now note that\n  $\\Vert D_{\\bq}z^\\dag_i(\\bp)\\Vert \\leq \\Vert D_{\\bq}\\bz^\\dag(\\bp)[I_i\\cap\\cB'] \\Vert  + \\Vert D_{\\bq}\\bz^\\dag(\\bp)[I_i\\setminus\\cB']\\Vert$,\n  and recall from \\eqref{eqn:linear-form-b} that $D_{\\bq}\\bz^\\dag(\\bp)[I_i\\setminus\\cB'] =\\bzero$. Hence, by applying \\eqref{eqn:nlp-sensitivity-2} to \\eqref{eqn:nlp-sensitivity-1}, and applying triangle inequality, we obtain the desired result.\n","start":51638,"end":52041}]},{"id":"S044","semantic_type":"assumption","kind":"assumption","labels":["eqn:dder-0","eqn:dder-0-c","lem:dder"],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"方向导数界的谱参数","source_units":[{"raw_tex":"  Suppose Assumptions \\ref{ass:c2}, \\ref{ass:regularity} hold, and suppose that, for given $\\bp\\in\\mathbb{P}$ (defined in Lemma \\ref{lem:nlp-sensitivity}) and $\\bq:=\\{q_i\\}_{i\\in \\cV}\\in\\mathbb{R}^{\\bl}$, we have $\\osigma_{\\bH}(\\bp,\\bq)$, $\\osigma_{\\bR}(\\bp,\\bq)$, and $\\usigma_{\\bH}(\\bp,\\bq)$ such that:\n  \\begin{subequations}\\label{eqn:dder-0}\n  \\begin{align}\n    \\osigma_{\\bH}(\\bp,\\bq)&\\geq \\osigma(\\bH(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)])\\\\\n    \\osigma_{\\bR}(\\bp,\\bq)&\\geq \\osigma(\\bR(\\bp)[\\cB'(\\bp,\\bq),:])\\\\\n    0<\\usigma_{\\bH}(\\bp,\\bq)&\\leq \\usigma(\\bH(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)]);\\label{eqn:dder-0-c}\n  \\end{align}\n\\end{subequations}\n","start":49246,"end":49893}]},{"id":"S049","semantic_type":"theorem","kind":"theorem","labels":["eqn:main","eqn:sigmas","thm:main"],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"指数敏感性衰减主定理","source_units":[{"raw_tex":"\\begin{theorem}[Exponential Decay of Sensitivity (EDS)]\\label{thm:main}\n  Under Assumptions \\ref{ass:c2}, \\ref{ass:regularity}, and for given $\\epsilon>0$, $\\osigma_{\\bH}\\geq\\oosigma_{\\bH}(\\bp^\\star)$, $\\osigma_{\\bR}\\geq\\oosigma_{\\bR}(\\bp^\\star)$, and $0<\\usigma_{\\bH}\\leq\\uusigma_{\\bH}(\\bp^\\star)$, where\n  \\begin{subequations}\\label{eqn:sigmas}\n    \\begin{align}\n      \\oosigma_{\\bH}(\\bp)&:=\\max\\{\\osigma(\\bH(\\bp)[\\cB,\\cB]):\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)\\}\\\\\n      \\oosigma_{\\bR}(\\bp)&:=\\max\\{\\osigma(\\bR(\\bp)[\\cB,:]):\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)\\}\\\\\n      \\uusigma_{\\bH}(\\bp)&:=\\min\\{\\usigma(\\bH(\\bp)[\\cB,\\cB]):\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)\\},\n    \\end{align}\n  \\end{subequations}\n  there exists a neighborhood $\\mathbb{P}_{\\epsilon}$ of $\\bp^\\star$ such that the following holds for any $\\bp,\\bp'\\in\\mathbb{P}_{\\epsilon}$:\n  \\begin{align}\\label{eqn:main}\n      &\\Vert z^\\dag_i(\\bp)-z^\\dag_i(\\bp')\\Vert \n      \\leq \\sum_{j\\in \\cV} \\Upsilon\\rho^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+} \\Vert p_j-p'_j\\Vert,\\quad i\\in \\cV,\n  \\end{align}\nwith $\\Upsilon :=\\dfrac{\\osigma_{\\bH}\\osigma_{\\bR}}{{\\usigma}_{\\bH}^2}+\\epsilon$ and $\\rho:=\\dfrac{\\osigma_{\\bH}^2-{\\usigma}_{\\bH}^2}{\\osigma_{\\bH}^2+{\\usigma}_{\\bH}^2}+\\epsilon$.\n\\end{theorem}","start":52537,"end":53850}]},{"id":"S050","semantic_type":"definition","kind":"definition","labels":["eqn:sigmas","thm:main"],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"固定基点活动族的谱包络","source_units":[{"raw_tex":"  \\begin{subequations}\\label{eqn:sigmas}\n    \\begin{align}\n      \\oosigma_{\\bH}(\\bp)&:=\\max\\{\\osigma(\\bH(\\bp)[\\cB,\\cB]):\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)\\}\\\\\n      \\oosigma_{\\bR}(\\bp)&:=\\max\\{\\osigma(\\bR(\\bp)[\\cB,:]):\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)\\}\\\\\n      \\uusigma_{\\bH}(\\bp)&:=\\min\\{\\usigma(\\bH(\\bp)[\\cB,\\cB]):\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)\\},\n    \\end{align}\n  \\end{subequations}\n","start":52843,"end":53293}]},{"id":"S051","semantic_type":"assumption","kind":"assumption","labels":["eqn:sigmas","thm:main"],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"主定理给定参数与系数","source_units":[{"raw_tex":"  Under Assumptions \\ref{ass:c2}, \\ref{ass:regularity}, and for given $\\epsilon>0$, $\\osigma_{\\bH}\\geq\\oosigma_{\\bH}(\\bp^\\star)$, $\\osigma_{\\bR}\\geq\\oosigma_{\\bR}(\\bp^\\star)$, and $0<\\usigma_{\\bH}\\leq\\uusigma_{\\bH}(\\bp^\\star)$, where\n","start":52609,"end":52843},{"raw_tex":"with $\\Upsilon :=\\dfrac{\\osigma_{\\bH}\\osigma_{\\bR}}{{\\usigma}_{\\bH}^2}+\\epsilon$ and $\\rho:=\\dfrac{\\osigma_{\\bH}^2-{\\usigma}_{\\bH}^2}{\\osigma_{\\bH}^2+{\\usigma}_{\\bH}^2}+\\epsilon$.\n","start":53657,"end":53837}]},{"id":"S052","semantic_type":"proof","kind":"proof","labels":["eqn:main-0","eqn:main-1","eqn:main-2"],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"EDS 主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n  From the continuity of $\\bz^\\dag(\\cdot)$ in the neighborhood of $\\bp^\\star$ and the continuity of $\\bc(\\cdot,\\cdot)$ in the neighborhood of $[\\bz^\\star;\\bp^\\star]$, there exists a neighborhood $\\widetilde{\\mathbb{P}}\\subseteq\\mathbb{P}$ of $\\bp^\\star$ such that, for $\\bp\\in\\widetilde{\\mathbb{P}}$ and $i\\in\\mathbb{I}_{[1,\\bm]}$, $\\bc(\\bx^\\star)[i] > 0\\Rightarrow\\bc(\\bx^\\dag(\\bp))[i] > 0$ and $\\by^\\star[i] \\neq 0\\Rightarrow\\by^\\dag(\\bp)[i] \\neq 0$. These conditions and complementarity slackness imply that for $\\bp\\in\\widetilde{\\mathbb{P}}$, we have $\\cA^0(\\bp^\\star)\\subseteq\\cA^0(\\bp)$ and $\\cA^1(\\bp)\\subseteq\\cA^1(\\bp^\\star)$; that is, $\\cB^0(\\bp^\\star)\\subseteq\\cB^0(\\bp)$ and $\\cB^1(\\bp)\\subseteq\\cB^1(\\bp^\\star)$. This and $\\cB^0(\\bp)\\subseteq\\cB'(\\bp,\\bq)\\subseteq\\cB^1(\\bp)$ yields:\n  \\begin{subequations}\\label{eqn:main-0}\n  \\begin{align}\n    \\osigma({\\bH}(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)])&\\leq \\oosigma_{\\bH}(\\bp)\\\\\n    \\osigma({\\bR}(\\bp)[\\cB'(\\bp,\\bq),:])&\\leq \\oosigma_{\\bR}(\\bp)\\\\\n    \\usigma({\\bH}(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)])&\\geq \\uusigma_{\\bH}(\\bp).\n  \\end{align}\n\\end{subequations}\n  By the twice-continuous differentiability of $\\cL(\\cdot,\\cdot)$ and the continuity of $\\bz^\\dag(\\cdot)$, we have that $\\bH(\\cdot)$ is continuous. The same holds true for its submatrices: $\\bH(\\cdot)[\\cB,\\cB]$ with $\\cB^0(\\bp^\\star)\\subseteq \\cB\\subseteq \\cB^1(\\bp^\\star)$. From the continuity of singular values with respect to its entries \\cite[Corollary 8.6.2]{golub2012matrix}, we have that $\\osigma(\\bH(\\cdot)[\\cB,\\cB])$ and $\\usigma(\\bH(\\cdot)[\\cB,\\cB])$ are continuous for any $\\cB^0(\\bp^\\star)\\subseteq \\cB\\subseteq \\cB^1(\\bp^\\star)$; accordingly, since a maximum and a minimum of a fixed and finite number of continuous functions is continuous, we have that $\\oosigma_{\\bH}(\\bp)$, $\\oosigma_{\\bR}(\\bp)$, $\\uusigma_{\\bH}(\\bp)$ are continuous with respect to $\\bp$ in $\\widetilde{\\mathbb{P}}$. Thus, there exists a convex neighborhood $\\mathbb{P}_{\\epsilon}\\subseteq\\widetilde{\\mathbb{P}}$ of $\\bp^\\star$ wherein the following are satisfied:\n  \\begin{subequations}\\label{eqn:main-1}\n  \\begin{align}\n    \\frac{\\oosigma_{\\bH}(\\bp)\\oosigma_{\\bR}(\\bp)}{\\uusigma_{\\bH}(\\bp)^2} &\\leq\n    \\frac{\\oosigma_{\\bH}(\\bp^\\star)\\oosigma_{\\bR}(\\bp^\\star)}{\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon\\\\\n    \\frac{\\oosigma_{\\bH}(\\bp)^2-\\uusigma_{\\bH}(\\bp)^2}{\\oosigma_{\\bH}(\\bp)^2+\\uusigma_{\\bH}(\\bp)^2}&\\leq\n    \\frac{\\oosigma_{\\bH}(\\bp^\\star)^2-\\uusigma_{\\bH}(\\bp^\\star)^2}{\\oosigma_{\\bH}(\\bp^\\star)^2+\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon.\n  \\end{align}\n\\end{subequations}\nHere, note that $\\uusigma_{\\bH}(\\bp^\\star)>0$ because $\\usigma(\\bH[\\cB,\\cB])>0$ holds for any $\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)$. By applying \\eqref{eqn:main-0} and \\eqref{eqn:main-1} to Lemma \\ref{lem:dder}, we obtain:\n\\begin{align}\\label{eqn:main-2}\n  \\Vert D_{\\bq} z^\\dag_i(\\bp)\\Vert&\\leq \\sum_{j\\in \\cV}\\left(\\frac{\\oosigma_{\\bH}(\\bp^\\star)\\oosigma_{\\bR}(\\bp^\\star)}{\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon\\right)\\left(\\frac{\\oosigma_{\\bH}(\\bp^\\star)^2-\\uusigma_{\\bH}(\\bp^\\star)^2}{\\oosigma_{\\bH}(\\bp^\\star)^2+\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert\\nonumber\\\\\n  &\\leq \\sum_{j\\in \\cV}\\left(\\frac{\\osigma_{\\bH}\\osigma_{\\bR}}{\\usigma_{\\bH}^2}+\\epsilon\\right)\\left(\\frac{\\osigma_{\\bH}^2-\\usigma_{\\bH}^2}{\\osigma_{\\bH}^2+\\usigma_{\\bH}^2}+\\epsilon\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert\\nonumber\\\\\n  &\\leq \\sum_{j\\in \\cV}\\Upsilon\\rho^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert,\n\\end{align}\nfor any $\\bp\\in\\mathbb{P}_{\\epsilon}$ and $\\bq\\in\\mathbb{R}^{\\bl}$. Finally, since we have chosen $\\mathbb{P}_\\epsilon$ to be convex, for any $\\bp,\\bp'\\in\\mathbb{P}_{\\epsilon}$, the line segment between $\\bp,\\bp'$ is within $\\mathbb{P}_{\\epsilon}$. Thus, we have:\n\\begin{align*}\n  \\Vert z^\\dag_i(\\bp)-z^\\dag_i(\\bp')\\Vert\n  % &\\leq\\Vert \\int_{0}^1 D_{\\bp'-\\bp}z^\\dag_i ((1-t)\\bp+t\\bp')dt\\Vert\\\\\n    &\\leq \\int_{0}^1\\Vert D_{\\bp'-\\bp}z^\\dag_i ((1-t)\\bp+t\\bp')\\Vert dt \\leq \\sum_{j\\in \\cV}\\Upsilon\\rho^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert p_j-p'_j\\Vert,\n\\end{align*}\nwhere the first inequality is from Newton-Leibniz and triangle inequality for integrals, and the last inequality follows from \\eqref{eqn:main-2}.\n\\end{proof}","start":53851,"end":58212}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:main-0","eqn:main-1"],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"邻域活动集夹持","source_units":[{"raw_tex":"  From the continuity of $\\bz^\\dag(\\cdot)$ in the neighborhood of $\\bp^\\star$ and the continuity of $\\bc(\\cdot,\\cdot)$ in the neighborhood of $[\\bz^\\star;\\bp^\\star]$, there exists a neighborhood $\\widetilde{\\mathbb{P}}\\subseteq\\mathbb{P}$ of $\\bp^\\star$ such that, for $\\bp\\in\\widetilde{\\mathbb{P}}$ and $i\\in\\mathbb{I}_{[1,\\bm]}$, $\\bc(\\bx^\\star)[i] > 0\\Rightarrow\\bc(\\bx^\\dag(\\bp))[i] > 0$ and $\\by^\\star[i] \\neq 0\\Rightarrow\\by^\\dag(\\bp)[i] \\neq 0$. These conditions and complementarity slackness imply that for $\\bp\\in\\widetilde{\\mathbb{P}}$, we have $\\cA^0(\\bp^\\star)\\subseteq\\cA^0(\\bp)$ and $\\cA^1(\\bp)\\subseteq\\cA^1(\\bp^\\star)$; that is, $\\cB^0(\\bp^\\star)\\subseteq\\cB^0(\\bp)$ and $\\cB^1(\\bp)\\subseteq\\cB^1(\\bp^\\star)$. This and $\\cB^0(\\bp)\\subseteq\\cB'(\\bp,\\bq)\\subseteq\\cB^1(\\bp)$ yields:\n","start":53865,"end":54662}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:main-0","eqn:main-1"],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"选中子矩阵受谱包络控制","source_units":[{"raw_tex":"  \\begin{subequations}\\label{eqn:main-0}\n  \\begin{align}\n    \\osigma({\\bH}(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)])&\\leq \\oosigma_{\\bH}(\\bp)\\\\\n    \\osigma({\\bR}(\\bp)[\\cB'(\\bp,\\bq),:])&\\leq \\oosigma_{\\bR}(\\bp)\\\\\n    \\usigma({\\bH}(\\bp)[\\cB'(\\bp,\\bq),\\cB'(\\bp,\\bq)])&\\geq \\uusigma_{\\bH}(\\bp).\n  \\end{align}\n\\end{subequations}\n","start":54662,"end":54979}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:main-0","eqn:main-1"],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"固定有限族的谱包络连续","source_units":[{"raw_tex":"  By the twice-continuous differentiability of $\\cL(\\cdot,\\cdot)$ and the continuity of $\\bz^\\dag(\\cdot)$, we have that $\\bH(\\cdot)$ is continuous. The same holds true for its submatrices: $\\bH(\\cdot)[\\cB,\\cB]$ with $\\cB^0(\\bp^\\star)\\subseteq \\cB\\subseteq \\cB^1(\\bp^\\star)$. From the continuity of singular values with respect to its entries \\cite[Corollary 8.6.2]{golub2012matrix}, we have that $\\osigma(\\bH(\\cdot)[\\cB,\\cB])$ and $\\usigma(\\bH(\\cdot)[\\cB,\\cB])$ are continuous for any $\\cB^0(\\bp^\\star)\\subseteq \\cB\\subseteq \\cB^1(\\bp^\\star)$; accordingly, since a maximum and a minimum of a fixed and finite number of continuous functions is continuous, we have that $\\oosigma_{\\bH}(\\bp)$, $\\oosigma_{\\bR}(\\bp)$, $\\uusigma_{\\bH}(\\bp)$ are continuous with respect to $\\bp$ in $\\widetilde{\\mathbb{P}}$. Thus, there exists a convex neighborhood $\\mathbb{P}_{\\epsilon}\\subseteq\\widetilde{\\mathbb{P}}$ of $\\bp^\\star$ wherein the following are satisfied:\n","start":54979,"end":55929}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:main-0","eqn:main-1"],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"凸邻域中的统一比率界","source_units":[{"raw_tex":"  \\begin{subequations}\\label{eqn:main-1}\n  \\begin{align}\n    \\frac{\\oosigma_{\\bH}(\\bp)\\oosigma_{\\bR}(\\bp)}{\\uusigma_{\\bH}(\\bp)^2} &\\leq\n    \\frac{\\oosigma_{\\bH}(\\bp^\\star)\\oosigma_{\\bR}(\\bp^\\star)}{\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon\\\\\n    \\frac{\\oosigma_{\\bH}(\\bp)^2-\\uusigma_{\\bH}(\\bp)^2}{\\oosigma_{\\bH}(\\bp)^2+\\uusigma_{\\bH}(\\bp)^2}&\\leq\n    \\frac{\\oosigma_{\\bH}(\\bp^\\star)^2-\\uusigma_{\\bH}(\\bp^\\star)^2}{\\oosigma_{\\bH}(\\bp^\\star)^2+\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon.\n  \\end{align}\n\\end{subequations}\n","start":55929,"end":56439}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:main-0","eqn:main-1"],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"基点下谱包络为正","source_units":[{"raw_tex":"Here, note that $\\uusigma_{\\bH}(\\bp^\\star)>0$ because $\\usigma(\\bH[\\cB,\\cB])>0$ holds for any $\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)$. By applying \\eqref{eqn:main-0} and \\eqref{eqn:main-1} to Lemma \\ref{lem:dder}, we obtain:\n","start":56439,"end":56680}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:main-0","eqn:main-1","eqn:main-2"],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"统一全部方向的节点导数界","source_units":[{"raw_tex":"Here, note that $\\uusigma_{\\bH}(\\bp^\\star)>0$ because $\\usigma(\\bH[\\cB,\\cB])>0$ holds for any $\\cB^0(\\bp^\\star)\\subseteq\\cB\\subseteq\\cB^1(\\bp^\\star)$. By applying \\eqref{eqn:main-0} and \\eqref{eqn:main-1} to Lemma \\ref{lem:dder}, we obtain:\n\\begin{align}\\label{eqn:main-2}\n  \\Vert D_{\\bq} z^\\dag_i(\\bp)\\Vert&\\leq \\sum_{j\\in \\cV}\\left(\\frac{\\oosigma_{\\bH}(\\bp^\\star)\\oosigma_{\\bR}(\\bp^\\star)}{\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon\\right)\\left(\\frac{\\oosigma_{\\bH}(\\bp^\\star)^2-\\uusigma_{\\bH}(\\bp^\\star)^2}{\\oosigma_{\\bH}(\\bp^\\star)^2+\\uusigma_{\\bH}(\\bp^\\star)^2}+\\epsilon\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert\\nonumber\\\\\n  &\\leq \\sum_{j\\in \\cV}\\left(\\frac{\\osigma_{\\bH}\\osigma_{\\bR}}{\\usigma_{\\bH}^2}+\\epsilon\\right)\\left(\\frac{\\osigma_{\\bH}^2-\\usigma_{\\bH}^2}{\\osigma_{\\bH}^2+\\usigma_{\\bH}^2}+\\epsilon\\right)^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert\\nonumber\\\\\n  &\\leq \\sum_{j\\in \\cV}\\Upsilon\\rho^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert q_j\\Vert,\n\\end{align}\nfor any $\\bp\\in\\mathbb{P}_{\\epsilon}$ and $\\bq\\in\\mathbb{R}^{\\bl}$. Finally, since we have chosen $\\mathbb{P}_\\epsilon$ to be convex, for any $\\bp,\\bp'\\in\\mathbb{P}_{\\epsilon}$, the line segment between $\\bp,\\bp'$ is within $\\mathbb{P}_{\\epsilon}$. Thus, we have:\n","start":56439,"end":57736}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:main-0","eqn:main-1"],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"沿数据线段积分","source_units":[{"raw_tex":"for any $\\bp\\in\\mathbb{P}_{\\epsilon}$ and $\\bq\\in\\mathbb{R}^{\\bl}$. Finally, since we have chosen $\\mathbb{P}_\\epsilon$ to be convex, for any $\\bp,\\bp'\\in\\mathbb{P}_{\\epsilon}$, the line segment between $\\bp,\\bp'$ is within $\\mathbb{P}_{\\epsilon}$. Thus, we have:\n\\begin{align*}\n  \\Vert z^\\dag_i(\\bp)-z^\\dag_i(\\bp')\\Vert\n  % &\\leq\\Vert \\int_{0}^1 D_{\\bp'-\\bp}z^\\dag_i ((1-t)\\bp+t\\bp')dt\\Vert\\\\\n    &\\leq \\int_{0}^1\\Vert D_{\\bp'-\\bp}z^\\dag_i ((1-t)\\bp+t\\bp')\\Vert dt \\leq \\sum_{j\\in \\cV}\\Upsilon\\rho^{\\left\\lceil \\frac{d_\\cG(i,j)}{4}-1\\right\\rceil_+}\\Vert p_j-p'_j\\Vert,\n\\end{align*}\nwhere the first inequality is from Newton-Leibniz and triangle inequality for integrals, and the last inequality follows from \\eqref{eqn:main-2}.\n","start":57472,"end":58201}]},{"id":"S060","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S060"],"declared_external":true,"declaration_text":["原始 prerequisite list：Dontchev and Rockafellar, dontchev2009implicit, Theorem 2G.8；实际使用见第 383 行。"],"source_only":false,"title":"广义方程半可微敏感性结果","source_units":[]},{"id":"S061","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S061"],"declared_external":true,"declaration_text":["原始 prerequisite list：Dontchev and Rockafellar, dontchev2009implicit, Theorem 2G.9；实际使用见第 383 行。"],"source_only":false,"title":"邻域内正则性保持结果","source_units":[]},{"id":"S062","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S062"],"declared_external":true,"declaration_text":["原始 prerequisite list：Dontchev and Rockafellar, dontchev2009implicit, Theorem 2D.1；实际使用见第 383 行。"],"source_only":false,"title":"半可微性与单侧方向导数","source_units":[]},{"id":"S063","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S063"],"declared_external":true,"declaration_text":["原始 prerequisite list：Dontchev and Rockafellar, dontchev2009implicit, Equation (35)；实际使用见第 383 行。"],"source_only":false,"title":"线性化 GE 与 QP 条件对应","source_units":[]},{"id":"S064","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S064"],"declared_external":true,"declaration_text":["原始 prerequisite list：Nocedal and Wright, nocedal2006numerical, Lemma 16.1；实际引用见第 445 行。"],"source_only":false,"title":"正则 KKT 矩阵非奇异性","source_units":[]},{"id":"S065","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S065"],"declared_external":true,"declaration_text":["原始 prerequisite list：Horn and Johnson, horn2012matrix, Corollary 5.6.16；实际引用见第 315 行。"],"source_only":false,"title":"收敛矩阵 Neumann 级数","source_units":[]},{"id":"S066","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S066"],"declared_external":true,"declaration_text":["原始 prerequisite list：Golub and Van Loan, golub2012matrix, Corollary 8.6.2；主证明实际引用见第 520 行。"],"source_only":false,"title":"奇异值随矩阵连续","source_units":[]},{"id":"S067","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S067"],"declared_external":true,"declaration_text":["原始 prerequisite list 中的 standard matrix norm inequalities；几何级数求和及谱与范数对应为 implicit standard facts made explicit by annotator。"],"source_only":false,"title":"标准谱与矩阵范数事实","source_units":[]},{"id":"S068","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S068"],"declared_external":true,"declaration_text":["原始 prerequisite list：standard Newton–Leibniz along absolutely continuous solution paths；作者在第 541 行明确使用 Newton–Leibniz。"],"source_only":false,"title":"绝对连续路径的积分公式","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S015","S018":"S015","S019":"S015","S020":"S015","S021":"S015","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S027","S030":"S027","S031":"S027","S032":"S027","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S043","S046":"S043","S047":"S043","S048":"S043","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068"}}}}
{"record_id":"2405.08894v4","paper":{"title":"Global weight optimization of frame structures under free-vibration eigenvalue constraints","authors":["Tyburec, Marek","Kočvara, Michal","Handa, Marouan","Zeman, Jan"],"arxiv_id":"2405.08894","version":"v4","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/2405.08894v4","pdf_url":"https://arxiv.org/pdf/2405.08894v4","doi":"10.1137/24M1713934"},"context":{"raw_tex":"Consider a polynomial $f(\\mathbf{x})$ of a degree $d$, which can be written as $f(\\mathbf{x}) = \\mathbf{p}^\\mathrm{T}\\mathbf{b}_d(\\mathbf{x})$, where\n%\n\\begin{equation}\n\\begin{aligned}\n\\mathbf{b}_d(\\mathbf{x})=&\\Big( 1 \\quad x_1 \\quad x_2 \\quad \\cdots \\quad x_1^2 \\quad x_1x_2 \\quad  \\cdots  \\quad x_1x_n \\\\ &  x_2^2 \\quad x_2x_3 \\quad  \\cdots \\quad x_n^2 \\quad \\cdots \\quad x_1^d \\quad \\cdots \\quad x_n^d \\Big)^\\mathrm{T}\n\\end{aligned}\n\\label{strandard_basis}\n\\end{equation}\n%\ndenotes the canonical monomial basis of degree up to $d$ and $\\mathbf{p} \\in \\mathbb{R}^{|\\mathbf{b}_d(\\mathbf{x})|}$ is the corresponding coefficient vector. To certify polynomial nonnegativity, we rely on the Sum-Of-Squares (SOS) representation\n\n\n{To construct these truncations, we introduce moment variables $\\mathbf{y} \\in \\mathbb{R}^{|\\mathbf{b}_{2r}(\\mathbf{x})|}$ indexed by the monomial basis $\\mathbf{b}_{2r}(\\mathbf{x})$. Further, let us define} the Riesz functional $L_\\mathbf{y} (\\mathbf{Z}(\\mathbf{x}))$ that linearizes {any} polynomial matrix $\\mathbf{Z}(\\mathbf{x})$ using $\\mathbf{y}$. Specifically, the operator is applied entry-wise and for the $(i,j)$-th component of $\\mathbf{Z}$ we write\n%\n\\begin{equation}\n    Z_{i,j}(\\mathbf{x}) = \\mathbf{p}_{{{i,j}}}^\\mathrm{T} \\mathbf{b}_{2r}(\\mathbf{x})\\quad \\Longleftrightarrow \\quad L_\\mathbf{y}(Z_{i,j}) = \\mathbf{p}_{{{i,j}}}^\\mathrm{T} \\mathbf{y}\\; \\text{ with }\\mathbf{y}\\text{ indexed in }\\mathbf{b}_{2r}(\\mathbf{x})\n\\end{equation}\n%\nwith a coefficient vector $\\mathbf{p}_{{{i,j}}}$. In this manuscript, we further adopt the notation $y_{x_i^d}$ to express the component of $\\mathbf{y}$ belonging to $x_i^d \\in \\mathbf{b}_{2r}(\\mathbf{x})$.\n\n{\nUnder Assumption \\ref{as:archimedean} (Archimedean assumption), polynomial optimization can be solved via the moment-sum-of-squares hierarchy:\n%\n\\begin{subequations}\\label{eq:general_moment}\n\\begin{align}\n    \\min_{\\mathbf{y}} \\; & L_\\mathbf{y}(f(\\mathbf{x}))\\\\\n    \\mathrm{s.t.}\\; & L_{\\mathbf{y}}\\bigl(\\mathbf{b}_{r-d_\\mathrm{G}}(\\mathbf{x}) \\mathbf{b}_{r-d_\\mathrm{G}}(\\mathbf{x})^\\mathrm{T} \\otimes \\mathbf{G}(\\mathbf{x})\\bigr) \\succeq 0, & \\text{(localizing matrix)}\\label{eq:localizing}\\\\\n    & L_{\\mathbf{y}}\\bigl(\\mathbf{b}_r(\\mathbf{x}) \\mathbf{b}_r(\\mathbf{x})^\\mathrm{T}\\bigr) \\succeq 0, & \\text{(moment matrix)}\\label{eq:moment}\n\\end{align}\n\\end{subequations}\n%\nwhere $d_\\mathrm{G}= \\lceil \\mathrm{deg}(\\mathbf{G}(\\mathbf{x}))/2 \\rceil$ (with $\\lceil \\bullet \\rceil$ denoting the ceiling function), and $\\otimes$ denotes the Kronecker product. The constraints define moment matrices $\\mathbf{M}_r(\\mathbf{y})$ and localizing matrices $\\mathbf{M}_{r-d_\\mathrm{G}} (\\mathbf{G}\\mathbf{y})$ that correspond to the SOS conditions in the dual formulation.}\n\n{Although \\eqref{eq:general_moment} is convex in the moments $\\mathbf{y}$, it is computationally intractable for $r\\rightarrow \\infty$. Instead, we solve a hierarchy of finite-dimensional truncations, starting with relaxation order $r = \\max\\left\\{d_\\mathrm{G}; \\left\\lceil\\mathrm{deg}(f(\\mathbf{x}))/2\\right\\rceil\\right\\}$ and increasing until convergence occurs. Since constraints of relaxation $r$ are included in relaxation $r+1$, we obtain a monotonic sequence of lower bounds: $f^{(r)} \\le f^{(r+1)} \\le f^* = f^{(\\infty)}$. Remarkably, the hierarchy also exhibits generic finite convergence \\cite{Nie2013}, though the convergence degree is not known \\textit{a priori}.}\n\n\n{Let $\\mathcal{L}_\\mathrm{s}$ and $\\mathcal{L}_\\mathrm{fv}$ denote the sets of indices corresponding to static and free-vibration load cases, respectively.} We investigate the global solution to topology optimization of least-weight frame structures \\eqref{eq:opt_orig_objective}, while accounting for upper bounds $\\overline{c}_j \\in \\mathbb{R}_{>0}$, $j \\in {\\mathcal{L}_\\mathrm{s}}$, on structural compliances \\eqref{eq:opt_orig_compliance} of {$\\lvert \\mathcal{L}_\\mathrm{s} \\rvert$ static} load cases and lower bounds $\\underline{\\lambda}_j \\in \\mathbb{R}_{>0}${, $j \\in \\mathcal{L}_\\mathrm{fv}$,} on the lowest nonzero free-vibration eigenvalues {of the $\\lvert \\mathcal{L}_\\mathrm{fv} \\rvert$ free-vibration load cases} expressed using the Rayleigh quotient \\eqref{eq:opt_orig_fv}. Such optimization problems are naturally formalized in terms of cross-section areas $\\mathbf{a}$, displacements $\\mathcal{U} = {\\{\\mathbf{u}_j : j \\in \\mathcal{L}_\\mathrm{s}\\}}$ with $\\mathbf{u}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}$, and normalized eigenmodes $\\mathcal{V} = {\\{\\mathbf{v}_j : j \\in \\mathcal{L}_\\mathrm{fv}\\}}$ with $\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}$ as\n%\n\\begin{subequations}\\label{eq:opt_orig}\n\t\\begin{align}\n\t\t\\min_{\\mathbf{a}\\in \\mathbb{R}^{n_{\\mathrm{e}}}\\!,\\;\n        \\mathcal{U},\\, \n        \\mathcal{V}} \n        & \\sum_{e=1}^{n_\\mathrm{e} } \\rho_e \\ell_e a_e\\label{eq:opt_orig_objective}\\\\\n\t\t\\text{s.t.}\\; & \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}} \\setminus \\mathrm{Ker}\\left( \\mathbf{M}(\\mathbf{a}) \\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\mathbf{K}_j(\\mathbf{a}) \\mathbf{v}_j}{\\mathbf{v}_j^\\mathrm{T} \\mathbf{M}_j(\\mathbf{a}) \\mathbf{v}_j} \\ge \\underline{\\lambda}_j,\\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\label{eq:opt_orig_fv}\\\\\n\t\t& \\mathbf{K}_j(\\mathbf{a}) \\mathbf{u}_j = \\mathbf{f}_j,\\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\label{eq:opt_orig_static}\\\\\n\t\t& \\mathbf{f}_j^\\mathrm{T} \\mathbf{u}_j \\le \\overline{c}_j,\\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\label{eq:opt_orig_compliance}\\\\\n\t\t& \\mathbf{a} \\ge \\mathbf{0},\\label{eq:opt_orig_nonneg}\n\t\\end{align}\n\\end{subequations}\n%\nwhere $\\mathbf{K}_j(\\mathbf{a}) \\in \\mathbb{S}^{n_{\\mathrm{dof},j}}_{\\succeq 0}$ and $\\mathbf{M}_j(\\mathbf{a}) \\in \\mathbb{S}^{n_\\mathrm{dof},j}_{\\succeq 0}$ stand for the stiffness and mass matrices, $n_{\\mathrm{dof},j}$ is the number of degrees of freedom in the $j$-th load case, and $\\mathbf{f}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}$ stands for the generalized force vector of the $j$-th load case, which includes the nodal forces and moments. In addition, $\\bm{\\ell} \\in \\mathbb{R}^{n_\\mathrm{e}}_{>0}$ and $\\bm{\\rho} \\in \\mathbb{R}^{n_\\mathrm{e}}_{>0}$ are vectors of lengths and densities of the $n_\\mathrm{e}$ elements. {Because the stiffness and mass matrices may vary across the load cases, the setting of \\eqref{eq:opt_orig} covers varying kinematic boundary conditions.}     {Notice that \\eqref{eq:opt_orig} is a non-smooth problem, as the free-vibration eigenvalue constraints \\eqref{eq:opt_orig_fv} are nondifferentiable for repeated eigenvalues.}\n\nIn bending-resistant structures such as frames, the stiffness matrix is usually a {low-degree} polynomial function of the design variables {(see Appendix \\ref{app:polydep} for more details)}, and follows from the assembly\n%\n\\begin{equation}\\label{eq:stiffness}\n\t\\mathbf{K}_j (\\mathbf{a}) = \\sum_{e=1}^{n_\\mathrm{e}} \\left[a_e \\mathbf{K}_{j,e}^{\\langle 1\\rangle} + a_e^2 \\mathbf{K}_{j,e}^{\\langle 2\\rangle} + a_e^3 \\mathbf{K}_{j,e}^{\\langle 3\\rangle}\\right],\n\\end{equation}\n%\nwhere $\\mathbf{K}_{j,e}^{\\langle i\\rangle} \\succeq 0$, $e \\in \\{1, \\dots, n_\\mathrm{e}\\}$, $i \\in \\{1,2,3\\}$, are auxiliary element stiffness matrices. In \\eqref{eq:stiffness}, the terms $ a_e \\mathbf{K}_{j,e}^{\\langle 1\\rangle}$ {typically} cover the membrane behavior, whereas the higher-order terms $a_e^2 \\mathbf{K}_{j,e}^{\\langle 2\\rangle}$ and $a_e^3 \\mathbf{K}_{j,e}^{\\langle 3\\rangle}$ capture the bending effects.\n\nFor the mass matrix $\\mathbf{M}_j(\\mathbf{a})$, we assume the form {(again, see Appendix \\ref{app:polydep})}\n%\n\\begin{equation}\\label{eq:mass}\n\t\\mathbf{M}_j (\\mathbf{a}) = \\mathbf{M}_j^{\\langle 0\\rangle} + \\sum_{e=1}^{n_\\mathrm{e}} a_e\\mathbf{M}_{j,e}^{\\langle 1\\rangle},\n\\end{equation}\n%\nin which $a_e \\mathbf{M}_{j,e}^{\\langle 1\\rangle} \\succeq 0$, $e \\in \\{1, \\dots, n_\\mathrm{e}\\}$, are the element mass matrices and $\\mathbf{M}_j^{\\langle 0\\rangle} \\succeq 0$ constitutes a non-structural mass. In this paper, we adopt stiffness-consistent mass matrices, but any other form can be used assuming it maintains a linear dependence on the cross-section areas. An extension to nonlinear terms in $\\mathbf{M}_j(\\mathbf{a})$ might be possible but is not covered in this manuscript.\n\n\nOne of the difficulties of the optimization problem \\eqref{eq:opt_orig} lies in the fact that \\eqref{eq:opt_orig_fv} and \\eqref{eq:opt_orig_static} include products of state variables $\\mathbf{u}_j$ and $\\mathbf{v}_j$ with design variables $\\mathbf{a}$. In \\cite[Proposition 2.3]{Achtziger2008}, it has been shown that the state variables $\\mathbf{u}_j$ and $\\mathbf{v}_j$ can be eliminated from the formulation, leading to an equivalent semidefinite programming reformulation in the design variables only:\n%\n\\begin{subequations}\\label{eq:opt}\n\\begin{align}\n\t\\min_{\\mathbf{a}\\in \\mathbb{R}^{n_{\\mathrm{e}}}}\\; & \\sum_{e=1}^{n_\\mathrm{e} } \\rho_e \\ell_e a_e\\label{eq:opt_objective}\\\\\n\t\\text{s.t.}\\; & \\mathbf{K}_j(\\mathbf{a}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\mathbf{a}) \\succeq 0,\\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\label{eq:opt_fv}\\\\\n\t& \\begin{pmatrix}\n\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t-\\mathbf{f}_j & \\mathbf{K}_j(\\mathbf{a})\n\t\\end{pmatrix} \\succeq 0,\\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\label{eq:opt_static}\\\\\n\t& \\mathbf{a} \\ge \\mathbf{0},\\label{eq:opt_nonneg}\n\\end{align}\n\\end{subequations}\n%\nwhere the constraints \\eqref{eq:opt_fv} and \\eqref{eq:opt_static} are matrix inequalities. These inequalities are generally nonconvex due to their polynomial dependence on $\\mathbf{a}$, and thus obtaining a (global) solution to \\eqref{eq:opt} remains challenging. {However, the non-differentiability of the free-vibration eigenvalue constraints is avoided, as the Rayleigh quotient \\eqref{eq:opt_orig_fv} is replaced by a polynomial matrix inequality \\eqref{eq:opt_fv}.}\n\nFrom the formulation \\eqref{eq:opt}, we can state the following basic properties of the feasible designs $\\hat{\\mathbf{a}}$:\n\n\n\\begin{proposition}[Static admissibility {\\cite[Proposition 1]{Tyburec2021}}]\\label{lemma:statical_admissibility}\n\tLet $$\\mathbf{\\hat{a}} \\in \\left\\{\\mathbf{a} \\; \\vert \\; \\begin{pmatrix}\n\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t-\\mathbf{f}_j & \\mathbf{K}_j(\\mathbf{a})\n\t\\end{pmatrix} \\succeq 0, \\mathbf{a}\\ge \\mathbf{0}\\right\\}$$ hold true. Then, $\\mathbf{f}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j(\\hat{\\mathbf{a}})\\right)$.\n\\end{proposition}\n\n\nIn this section, we introduce a bilevel programming reformulation of the problem \\eqref{eq:opt_orig} and provide conditions for its solvability. This reformulation will further enable us to construct feasible solutions to \\eqref{eq:opt_orig}.\n\nSetting $\\mathbf{a}(\\delta) := \\delta \\tilde{\\mathbf{a}}$ with $\\tilde{\\mathbf{a}}\\ge \\mathbf{0}$, the optimization problem \\eqref{eq:opt} becomes equivalent to a bilevel optimization problem, with the upper level seeking \\textit{admissible} ratios of the cross-section areas $\\tilde{\\mathbf{a}}$ and the lower level searching for the optimal scaling factor $\\delta$ of the fixed cross-section ratios:\n%\n\\begin{subequations}\\label{eq:bilevel}\n\t\\begin{align}\n\t\t\\min_{\\tilde{\\mathbf{a}}\\in \\mathbb{R}^{n_{\\mathrm{e}}}\\!,\\; \\delta^*\\in\\mathbb{R}}\\; & \\delta^* \\sum_{e=1}^{n_\\mathrm{e}} \\rho_e \\ell_e \\tilde{a}_e\\\\\n\t\t\\mathrm{s.t.}\\ & \\tilde{\\mathbf{a}} \\ge \\mathbf{0},\\label{eq:bilevel-atilde}\\\\\n\t\t& \\mathbf{f}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right), \\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\label{eq:bilevel_static}\\\\\n\t\t& \\mathrm{Im}\\left(\\mathbf{M}_{j}(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_{j}(\\tilde{\\mathbf{a}})\\right),\\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\label{eq:bilevel_mass}\\\\\n\t\t& \\mathbf{W}_j^\\mathrm{T}\\left[ \\sum_{e=1}^{n_\\mathrm{e}}\\tilde{a}_e\\mathbf{K}_{j,e}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\sum_{e=1}^{n_\\mathrm{e}} \\tilde{a}_e \\mathbf{M}_{j,e}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0, \\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\label{eq:bilevel-membraine}\\\\\n\t\t& \\begin{aligned}\\label{eq:bilevel-lower}\n\t\t\t\\delta^* \\in \\argmin_{\\delta\\in\\mathbb{R}}\\; & \\delta\\\\\n\t\t\t\\mathrm{s.t.}\\; & \\begin{pmatrix}\n\t\t\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t\t\t-\\mathbf{f}_j & \\mathbf{K}_j (\\delta \\tilde{\\mathbf{a}})\n\t\t\t\\end{pmatrix} \\succeq 0, \\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\\\\n\t\t\t& \\mathbf{K}_j(\\delta \\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\delta \\tilde{\\mathbf{a}}) \\succeq 0, \\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\\\\n\t\t\t& \\delta \\ge 0,\n\t\t\\end{aligned}\n\t\\end{align}\n\\end{subequations}\n%\nwhere $\\mathbf{W}_j$ is an orthonormal basis of $\\mathrm{Ker}\\left(\\sum_{e=1}^{n_\\mathrm{e}}\\left[\\mathbf{K}_{j,e}^{\\langle 2\\rangle} + \\mathbf{K}_{j,e}^{\\langle 3\\rangle}\\right]\\right)$ and the constraints \\eqref{eq:bilevel-atilde}-\\eqref{eq:bilevel-membraine} ensure that the lower-level problem \\eqref{eq:bilevel-lower} {has a non-empty interior}. In particular, these {constraints} include nonnegative components of the cross-sectional ratios \\eqref{eq:bilevel-atilde}, static \\eqref{eq:bilevel_static} and mass \\eqref{eq:bilevel_mass} admissibility of the ratios, and the pure membrane eigenvalues to be at least $\\underline{\\lambda}_j$ \\eqref{eq:bilevel-membraine}, which we will prove in Proposition \\ref{th:feasibility_cond} (that uses the notation set up in \\eqref{eq:hatnotation}). Moreover, the lower-level problem \\eqref{eq:bilevel-lower} is quasi-convex in $\\delta$, allowing one to obtain a globally optimal scaling $\\delta^*$ for fixed ratios $\\tilde{\\mathbf{a}}$ using bisection. Proving the quasi-convexity property and showing that the conditions \\eqref{eq:bilevel-atilde}--\\eqref{eq:bilevel-membraine} are sufficient and necessary for solvability of the lower-level problem \\eqref{eq:bilevel-lower} constitutes the main goal of this section.\n\nLet us first focus on the lower-level problem \\eqref{eq:bilevel-lower}. We start by expressing the lowest nonzero\n%non-singular \nfree-vibration eigenvalue function $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ using the Raleigh quotient \\cite[Definition 2.2 and Appendix A.1]{Achtziger2008} as\n%\n\\begin{equation}\\label{eq:RQ}\n\t\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) = \\min_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\left[\\delta\\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} + \\delta^2\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\delta^3\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\right]  \\mathbf{v}_j}{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\delta\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j},\n\\end{equation}\n%\nin which we adopt the simplified notation\n%\n\\begin{equation}\\label{eq:hatnotation}\n\t\\hat{\\mathbf{K}}_j^{\\langle i\\rangle}(\\tilde{\\mathbf{a}}) = \\sum_{e=1}^{n_\\mathrm{e}}\\tilde{a}_e^i\\mathbf{K}_{j,e}^{\\langle i\\rangle}, \\qquad \n\t\\hat{\\mathbf{M}}_j^{\\langle i\\rangle}(\\tilde{\\mathbf{a}}) = \\sum_{e=1}^{n_\\mathrm{e}}\\tilde{a}_e^i \\mathbf{M}_{j,e}^{\\langle i\\rangle}.\n\\end{equation}\n%\nand omit the argument of $\\hat{\\mathbf{K}}_j^{\\langle i\\rangle}$ and $\\hat{\\mathbf{M}}_j^{\\langle i\\rangle}$ \nwhen it is clear from the context, to shorten the equations.\n\nBecause $\\forall j \\in \\mathcal{L}_{\\mathrm{fv}}: \\mathrm{Im}\\left(\\mathbf{M}_{j}(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_{j}(\\tilde{\\mathbf{a}})\\right)$ (see Proposition~\\ref{prop:exists_gamma_fv}), \\eqref{eq:RQ} renders that $\\lambda_{\\min,j}>0$. Thus, to prove feasibility of the eigenvalue constraint in \\eqref{eq:bilevel-lower} with respect to $\\delta$, it remains to find the condition under which $\\sup_{\\delta} \\lambda_{\\min,j} \\ge \\underline{\\lambda}_j$. \n\nBecause of $\\delta > 0$, equation~\\eqref{eq:RQ} can further be rewritten to\n%\n\\begin{equation}\\label{eq:RQs}\n\t\\begin{multlined}\n\t\t\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) =\\\\\n\t\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}.\n\t\\end{multlined}\n\\end{equation}\n\n\nWith this result, we have shown that by increasing $\\delta$ we do not decrease $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$. {Notice that this holds up to $\\delta\\rightarrow \\infty$, as we can choose $\\Delta\\rightarrow \\infty$ in the above proof.} For the solvability of the lower-level problem \\eqref{eq:bilevel-lower}, it is crucial to guarantee that there exits a $\\delta$ such that $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) \\ge \\underline{\\lambda}_j$. In other words, we require that $\\sup_{\\delta} \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) \\ge \\underline{\\lambda}_j$. To show this, let $\\mathbf{v}_j := \\mathbf{t}_j + \\mathbf{w}_j$, where $\\mathbf{t}_j \\in \\mathrm{Im}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) + \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right) $ and $\\mathbf{w}_j \\in \\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$.\nThen, we can write the expression for the supremum.\n\n\n{Let us start with the first modification.} Since the objective function \\eqref{eq:opt_objective}, representing the weight of the structure, is a conic combination of nonnegative variables, any feasible point to the optimization problem \\eqref{eq:opt} provides an upper-bound on  the objective function value $\\overline{w}$ that also bounds the cross-section areas from above. This upper bound can be obtained by solving the convex problem \\eqref{eq:linprob}, providing the cross-sectional ratios $\\tilde{\\mathbf{a}}$, and{, subsequently,} by finding the minimal scaling factor $\\delta^*$ by solving the lower-level problem \\eqref{eq:bilevel-lower} globally using bisection. Consequently, the optimal cross-section areas are bounded as\n%\n\\begin{equation}\n\t0 \\le a_e \\le \\frac{\\overline{w}}{\\rho_e\\ell_e}, \\forall e \\in \\{1,\\dots,n_\\mathrm{e}\\}.\n\\end{equation}\n%\nFurthermore, using the substitution $a_e = \\frac{\\overline{w}}{2\\rho_e\\ell_e} (a_{\\mathrm{s},e} + 1)$, these bounds can be written as $a_{\\mathrm{s},e}^2 \\le 1$. %Similarly, we add a weight constraint to strengthen the relaxations as\n%\n%\\begin{equation}\n%    \\sum_{e=1}^{n_\\mathrm{e}} \\rho_e \\ell_e a_e \\le \\overline{w}\\quad\\longleftrightarrow\\quad 2-n_\\mathrm{e} - \\sum_{e=1}^{n_\\mathrm{e}} a_{\\mathrm{s},e} \\ge 0.\n%\\end{equation}\nSimilarly, we add a weight constraint $\\sum_{e=1}^{n_\\mathrm{e}} \\rho_e \\ell_e a_e \\le \\overline{w}$ to strengthen the relaxations.\nThe influence of this constraint on the feasible set of relaxations will be illustrated in Section~\\ref{sec:illustrative}. Using the above substitution of $a_e$, we obtain the formulation\n%\n\\begin{subequations}\\label{eq:opt_po}\n\t\\begin{align}\n\t\t\\min_{\\mathbf{a}_{{\\mathrm{s}}}\\in\\mathbb{R}^{n_\\mathrm{e}}}\\; & {\\frac{1}{2}} \\overline{w} (n_\\mathrm{e} + \\mathbf{1}^\\mathrm{T} \\mathbf{a}_\\mathrm{s})\\label{eq:opt_objective_po}\\\\\n\t\t\\text{s.t.}\\; & \\mathbf{K}_j(\\mathbf{a}_\\mathrm{s}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\mathbf{a}_\\mathrm{s}) \\succeq 0, \\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\label{eq:opt_fv_po}\\\\\n\t\t& \\begin{pmatrix}\n\t\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t\t-\\mathbf{f}_j & \\mathbf{K}_j(\\mathbf{a}_\\mathrm{s})\n\t\t\\end{pmatrix} \\succeq 0,\\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\label{eq:opt_static_po}\\\\\n            & 2-n_\\mathrm{e} - \\sum_{e=1}^{n_\\mathrm{e}} a_{\\mathrm{s},e} \\ge 0,\\label{eq:opt_weight_po}\\\\\n\t\t& 1 - a_\\mathrm{s,e}^2 \\ge 0, \\quad e \\in \\{1,\\dots,n_\\mathrm{e}\\}\\label{eq:opt_nonneg_po}\n\t\\end{align}\n\\end{subequations}\n%\n{in the scaled variables $\\mathbf{a}_{\\mathrm{s}} \\in \\mathbb{R}^{n_\\mathrm{e}}$. Here,} we slightly abuse the notation and express the stiffness and mass matrices in terms of the scaled variables $\\mathbf{a}_\\mathrm{s}$ as $\\mathbf{K}_j(\\mathbf{a}_\\mathrm{s})$ and $\\mathbf{M}_j(\\mathbf{a}_\\mathrm{s})$, respectively. {Although this compactification means that \\eqref{eq:opt} and \\eqref{eq:opt_po} do not have the same feasible set (after rescaling), they maintain the same set of optimal solutions. Thus, solution to \\eqref{eq:opt_po} also solves \\eqref{eq:opt}.}\n\n\n\\subsubsection{Feasible upper bounds from first-order moments}\nIn what follows, we show that the first-order moments obtained by solving any relaxation of \\eqref{eq:opt_po} satisfy the {feasibility conditions} set in Proposition \\ref{th:feasibility_cond}, which allows us to construct a feasible upper-bound solution to \\eqref{eq:opt}.\n\nTo show this, let us introduce the notation\n%\n\\begin{subequations}\n\t\\begin{align}\n\t\ta_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}) &= 0.5\\,{\\frac{\\overline{w}}{\\rho_e \\ell_e}} (y_{a_{\\mathrm{s},e}^1} + 1),\\label{eq:rel_a_1}\\\\\n\t\ta_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}) &= 0.25\\, {\\frac{\\overline{w}^2}{\\rho_e^2 \\ell_e^2}} \\left( y_{a_{\\mathrm{s},e}^2} + 2 y_{a_{\\mathrm{s},e}^1} + 1 \\right),\\label{eq:rel_a_2}\\\\\n\t\ta_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}) &= 0.125\\, {\\frac{\\overline{w}^3}{\\rho_e^3 \\ell_e^3}} \\left( y_{a_{\\mathrm{s},e}^3} + 3 y_{a_{\\mathrm{s},e}^2} + 3 y_{a_{\\mathrm{s},e}^1} + 1 \\right)\\label{eq:rel_a_3}\n\t\\end{align}\n\\end{subequations}\n%\nfor evaluating {expected values of the} monomials of the (unscaled) cross-section areas based on the (scaled) moments $\\mathbf{y}$. In this notation, the superscript $\\bullet^{\\langle i\\rangle}$ indicates that it corresponds to the $i$-th degree monomial of $\\bullet$. We wish to emphasize here that, generally, $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}) \\neq \\left[a_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y})\\right]^2$ and $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}) \\neq \\left[a_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y})\\right]^3$ due to the relaxation procedure.\n\nAs the first step in the proof, we need to show that replacing higher-order moments with the powers of the first-order moments does not change the rank of the stiffness matrix. Since the stiffness matrix is a conic combination of positive semidefinite matrices with the coefficients being monomials of the cross-section areas, recall \\eqref{eq:stiffness}, it suffices to show that strict positivity of $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y})$ or $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y})$ implies strict positivity of $a_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y})$.\n","units":[{"raw_tex":"Consider a polynomial $f(\\mathbf{x})$ of a degree $d$, which can be written as $f(\\mathbf{x}) = \\mathbf{p}^\\mathrm{T}\\mathbf{b}_d(\\mathbf{x})$, where\n%\n\\begin{equation}\n\\begin{aligned}\n\\mathbf{b}_d(\\mathbf{x})=&\\Big( 1 \\quad x_1 \\quad x_2 \\quad \\cdots \\quad x_1^2 \\quad x_1x_2 \\quad  \\cdots  \\quad x_1x_n \\\\ &  x_2^2 \\quad x_2x_3 \\quad  \\cdots \\quad x_n^2 \\quad \\cdots \\quad x_1^d \\quad \\cdots \\quad x_n^d \\Big)^\\mathrm{T}\n\\end{aligned}\n\\label{strandard_basis}\n\\end{equation}\n%\ndenotes the canonical monomial basis of degree up to $d$ and $\\mathbf{p} \\in \\mathbb{R}^{|\\mathbf{b}_d(\\mathbf{x})|}$ is the corresponding coefficient vector. To certify polynomial nonnegativity, we rely on the Sum-Of-Squares (SOS) representation\n","start":18904,"end":19629},{"raw_tex":"{To construct these truncations, we introduce moment variables $\\mathbf{y} \\in \\mathbb{R}^{|\\mathbf{b}_{2r}(\\mathbf{x})|}$ indexed by the monomial basis $\\mathbf{b}_{2r}(\\mathbf{x})$. Further, let us define} the Riesz functional $L_\\mathbf{y} (\\mathbf{Z}(\\mathbf{x}))$ that linearizes {any} polynomial matrix $\\mathbf{Z}(\\mathbf{x})$ using $\\mathbf{y}$. Specifically, the operator is applied entry-wise and for the $(i,j)$-th component of $\\mathbf{Z}$ we write\n%\n\\begin{equation}\n    Z_{i,j}(\\mathbf{x}) = \\mathbf{p}_{{{i,j}}}^\\mathrm{T} \\mathbf{b}_{2r}(\\mathbf{x})\\quad \\Longleftrightarrow \\quad L_\\mathbf{y}(Z_{i,j}) = \\mathbf{p}_{{{i,j}}}^\\mathrm{T} \\mathbf{y}\\; \\text{ with }\\mathbf{y}\\text{ indexed in }\\mathbf{b}_{2r}(\\mathbf{x})\n\\end{equation}\n%\nwith a coefficient vector $\\mathbf{p}_{{{i,j}}}$. In this manuscript, we further adopt the notation $y_{x_i^d}$ to express the component of $\\mathbf{y}$ belonging to $x_i^d \\in \\mathbf{b}_{2r}(\\mathbf{x})$.\n\n{\nUnder Assumption \\ref{as:archimedean} (Archimedean assumption), polynomial optimization can be solved via the moment-sum-of-squares hierarchy:\n%\n\\begin{subequations}\\label{eq:general_moment}\n\\begin{align}\n    \\min_{\\mathbf{y}} \\; & L_\\mathbf{y}(f(\\mathbf{x}))\\\\\n    \\mathrm{s.t.}\\; & L_{\\mathbf{y}}\\bigl(\\mathbf{b}_{r-d_\\mathrm{G}}(\\mathbf{x}) \\mathbf{b}_{r-d_\\mathrm{G}}(\\mathbf{x})^\\mathrm{T} \\otimes \\mathbf{G}(\\mathbf{x})\\bigr) \\succeq 0, & \\text{(localizing matrix)}\\label{eq:localizing}\\\\\n    & L_{\\mathbf{y}}\\bigl(\\mathbf{b}_r(\\mathbf{x}) \\mathbf{b}_r(\\mathbf{x})^\\mathrm{T}\\bigr) \\succeq 0, & \\text{(moment matrix)}\\label{eq:moment}\n\\end{align}\n\\end{subequations}\n%\nwhere $d_\\mathrm{G}= \\lceil \\mathrm{deg}(\\mathbf{G}(\\mathbf{x}))/2 \\rceil$ (with $\\lceil \\bullet \\rceil$ denoting the ceiling function), and $\\otimes$ denotes the Kronecker product. The constraints define moment matrices $\\mathbf{M}_r(\\mathbf{y})$ and localizing matrices $\\mathbf{M}_{r-d_\\mathrm{G}} (\\mathbf{G}\\mathbf{y})$ that correspond to the SOS conditions in the dual formulation.}\n\n{Although \\eqref{eq:general_moment} is convex in the moments $\\mathbf{y}$, it is computationally intractable for $r\\rightarrow \\infty$. Instead, we solve a hierarchy of finite-dimensional truncations, starting with relaxation order $r = \\max\\left\\{d_\\mathrm{G}; \\left\\lceil\\mathrm{deg}(f(\\mathbf{x}))/2\\right\\rceil\\right\\}$ and increasing until convergence occurs. Since constraints of relaxation $r$ are included in relaxation $r+1$, we obtain a monotonic sequence of lower bounds: $f^{(r)} \\le f^{(r+1)} \\le f^* = f^{(\\infty)}$. Remarkably, the hierarchy also exhibits generic finite convergence \\cite{Nie2013}, though the convergence degree is not known \\textit{a priori}.}\n","start":21814,"end":24518},{"raw_tex":"{Let $\\mathcal{L}_\\mathrm{s}$ and $\\mathcal{L}_\\mathrm{fv}$ denote the sets of indices corresponding to static and free-vibration load cases, respectively.} We investigate the global solution to topology optimization of least-weight frame structures \\eqref{eq:opt_orig_objective}, while accounting for upper bounds $\\overline{c}_j \\in \\mathbb{R}_{>0}$, $j \\in {\\mathcal{L}_\\mathrm{s}}$, on structural compliances \\eqref{eq:opt_orig_compliance} of {$\\lvert \\mathcal{L}_\\mathrm{s} \\rvert$ static} load cases and lower bounds $\\underline{\\lambda}_j \\in \\mathbb{R}_{>0}${, $j \\in \\mathcal{L}_\\mathrm{fv}$,} on the lowest nonzero free-vibration eigenvalues {of the $\\lvert \\mathcal{L}_\\mathrm{fv} \\rvert$ free-vibration load cases} expressed using the Rayleigh quotient \\eqref{eq:opt_orig_fv}. Such optimization problems are naturally formalized in terms of cross-section areas $\\mathbf{a}$, displacements $\\mathcal{U} = {\\{\\mathbf{u}_j : j \\in \\mathcal{L}_\\mathrm{s}\\}}$ with $\\mathbf{u}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}$, and normalized eigenmodes $\\mathcal{V} = {\\{\\mathbf{v}_j : j \\in \\mathcal{L}_\\mathrm{fv}\\}}$ with $\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}$ as\n%\n\\begin{subequations}\\label{eq:opt_orig}\n\t\\begin{align}\n\t\t\\min_{\\mathbf{a}\\in \\mathbb{R}^{n_{\\mathrm{e}}}\\!,\\;\n        \\mathcal{U},\\, \n        \\mathcal{V}} \n        & \\sum_{e=1}^{n_\\mathrm{e} } \\rho_e \\ell_e a_e\\label{eq:opt_orig_objective}\\\\\n\t\t\\text{s.t.}\\; & \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}} \\setminus \\mathrm{Ker}\\left( \\mathbf{M}(\\mathbf{a}) \\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\mathbf{K}_j(\\mathbf{a}) \\mathbf{v}_j}{\\mathbf{v}_j^\\mathrm{T} \\mathbf{M}_j(\\mathbf{a}) \\mathbf{v}_j} \\ge \\underline{\\lambda}_j,\\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\label{eq:opt_orig_fv}\\\\\n\t\t& \\mathbf{K}_j(\\mathbf{a}) \\mathbf{u}_j = \\mathbf{f}_j,\\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\label{eq:opt_orig_static}\\\\\n\t\t& \\mathbf{f}_j^\\mathrm{T} \\mathbf{u}_j \\le \\overline{c}_j,\\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\label{eq:opt_orig_compliance}\\\\\n\t\t& \\mathbf{a} \\ge \\mathbf{0},\\label{eq:opt_orig_nonneg}\n\t\\end{align}\n\\end{subequations}\n%\nwhere $\\mathbf{K}_j(\\mathbf{a}) \\in \\mathbb{S}^{n_{\\mathrm{dof},j}}_{\\succeq 0}$ and $\\mathbf{M}_j(\\mathbf{a}) \\in \\mathbb{S}^{n_\\mathrm{dof},j}_{\\succeq 0}$ stand for the stiffness and mass matrices, $n_{\\mathrm{dof},j}$ is the number of degrees of freedom in the $j$-th load case, and $\\mathbf{f}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}$ stands for the generalized force vector of the $j$-th load case, which includes the nodal forces and moments. In addition, $\\bm{\\ell} \\in \\mathbb{R}^{n_\\mathrm{e}}_{>0}$ and $\\bm{\\rho} \\in \\mathbb{R}^{n_\\mathrm{e}}_{>0}$ are vectors of lengths and densities of the $n_\\mathrm{e}$ elements. {Because the stiffness and mass matrices may vary across the load cases, the setting of \\eqref{eq:opt_orig} covers varying kinematic boundary conditions.}     {Notice that \\eqref{eq:opt_orig} is a non-smooth problem, as the free-vibration eigenvalue constraints \\eqref{eq:opt_orig_fv} are nondifferentiable for repeated eigenvalues.}\n\nIn bending-resistant structures such as frames, the stiffness matrix is usually a {low-degree} polynomial function of the design variables {(see Appendix \\ref{app:polydep} for more details)}, and follows from the assembly\n%\n\\begin{equation}\\label{eq:stiffness}\n\t\\mathbf{K}_j (\\mathbf{a}) = \\sum_{e=1}^{n_\\mathrm{e}} \\left[a_e \\mathbf{K}_{j,e}^{\\langle 1\\rangle} + a_e^2 \\mathbf{K}_{j,e}^{\\langle 2\\rangle} + a_e^3 \\mathbf{K}_{j,e}^{\\langle 3\\rangle}\\right],\n\\end{equation}\n%\nwhere $\\mathbf{K}_{j,e}^{\\langle i\\rangle} \\succeq 0$, $e \\in \\{1, \\dots, n_\\mathrm{e}\\}$, $i \\in \\{1,2,3\\}$, are auxiliary element stiffness matrices. In \\eqref{eq:stiffness}, the terms $ a_e \\mathbf{K}_{j,e}^{\\langle 1\\rangle}$ {typically} cover the membrane behavior, whereas the higher-order terms $a_e^2 \\mathbf{K}_{j,e}^{\\langle 2\\rangle}$ and $a_e^3 \\mathbf{K}_{j,e}^{\\langle 3\\rangle}$ capture the bending effects.\n\nFor the mass matrix $\\mathbf{M}_j(\\mathbf{a})$, we assume the form {(again, see Appendix \\ref{app:polydep})}\n%\n\\begin{equation}\\label{eq:mass}\n\t\\mathbf{M}_j (\\mathbf{a}) = \\mathbf{M}_j^{\\langle 0\\rangle} + \\sum_{e=1}^{n_\\mathrm{e}} a_e\\mathbf{M}_{j,e}^{\\langle 1\\rangle},\n\\end{equation}\n%\nin which $a_e \\mathbf{M}_{j,e}^{\\langle 1\\rangle} \\succeq 0$, $e \\in \\{1, \\dots, n_\\mathrm{e}\\}$, are the element mass matrices and $\\mathbf{M}_j^{\\langle 0\\rangle} \\succeq 0$ constitutes a non-structural mass. In this paper, we adopt stiffness-consistent mass matrices, but any other form can be used assuming it maintains a linear dependence on the cross-section areas. An extension to nonlinear terms in $\\mathbf{M}_j(\\mathbf{a})$ might be possible but is not covered in this manuscript.\n","start":26611,"end":31369},{"raw_tex":"One of the difficulties of the optimization problem \\eqref{eq:opt_orig} lies in the fact that \\eqref{eq:opt_orig_fv} and \\eqref{eq:opt_orig_static} include products of state variables $\\mathbf{u}_j$ and $\\mathbf{v}_j$ with design variables $\\mathbf{a}$. In \\cite[Proposition 2.3]{Achtziger2008}, it has been shown that the state variables $\\mathbf{u}_j$ and $\\mathbf{v}_j$ can be eliminated from the formulation, leading to an equivalent semidefinite programming reformulation in the design variables only:\n%\n\\begin{subequations}\\label{eq:opt}\n\\begin{align}\n\t\\min_{\\mathbf{a}\\in \\mathbb{R}^{n_{\\mathrm{e}}}}\\; & \\sum_{e=1}^{n_\\mathrm{e} } \\rho_e \\ell_e a_e\\label{eq:opt_objective}\\\\\n\t\\text{s.t.}\\; & \\mathbf{K}_j(\\mathbf{a}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\mathbf{a}) \\succeq 0,\\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\label{eq:opt_fv}\\\\\n\t& \\begin{pmatrix}\n\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t-\\mathbf{f}_j & \\mathbf{K}_j(\\mathbf{a})\n\t\\end{pmatrix} \\succeq 0,\\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\label{eq:opt_static}\\\\\n\t& \\mathbf{a} \\ge \\mathbf{0},\\label{eq:opt_nonneg}\n\\end{align}\n\\end{subequations}\n%\nwhere the constraints \\eqref{eq:opt_fv} and \\eqref{eq:opt_static} are matrix inequalities. These inequalities are generally nonconvex due to their polynomial dependence on $\\mathbf{a}$, and thus obtaining a (global) solution to \\eqref{eq:opt} remains challenging. {However, the non-differentiability of the free-vibration eigenvalue constraints is avoided, as the Rayleigh quotient \\eqref{eq:opt_orig_fv} is replaced by a polynomial matrix inequality \\eqref{eq:opt_fv}.}\n\nFrom the formulation \\eqref{eq:opt}, we can state the following basic properties of the feasible designs $\\hat{\\mathbf{a}}$:\n","start":31370,"end":33085},{"raw_tex":"\\begin{proposition}[Static admissibility {\\cite[Proposition 1]{Tyburec2021}}]\\label{lemma:statical_admissibility}\n\tLet $$\\mathbf{\\hat{a}} \\in \\left\\{\\mathbf{a} \\; \\vert \\; \\begin{pmatrix}\n\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t-\\mathbf{f}_j & \\mathbf{K}_j(\\mathbf{a})\n\t\\end{pmatrix} \\succeq 0, \\mathbf{a}\\ge \\mathbf{0}\\right\\}$$ hold true. Then, $\\mathbf{f}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j(\\hat{\\mathbf{a}})\\right)$.\n\\end{proposition}\n","start":33087,"end":33535},{"raw_tex":"In this section, we introduce a bilevel programming reformulation of the problem \\eqref{eq:opt_orig} and provide conditions for its solvability. This reformulation will further enable us to construct feasible solutions to \\eqref{eq:opt_orig}.\n\nSetting $\\mathbf{a}(\\delta) := \\delta \\tilde{\\mathbf{a}}$ with $\\tilde{\\mathbf{a}}\\ge \\mathbf{0}$, the optimization problem \\eqref{eq:opt} becomes equivalent to a bilevel optimization problem, with the upper level seeking \\textit{admissible} ratios of the cross-section areas $\\tilde{\\mathbf{a}}$ and the lower level searching for the optimal scaling factor $\\delta$ of the fixed cross-section ratios:\n%\n\\begin{subequations}\\label{eq:bilevel}\n\t\\begin{align}\n\t\t\\min_{\\tilde{\\mathbf{a}}\\in \\mathbb{R}^{n_{\\mathrm{e}}}\\!,\\; \\delta^*\\in\\mathbb{R}}\\; & \\delta^* \\sum_{e=1}^{n_\\mathrm{e}} \\rho_e \\ell_e \\tilde{a}_e\\\\\n\t\t\\mathrm{s.t.}\\ & \\tilde{\\mathbf{a}} \\ge \\mathbf{0},\\label{eq:bilevel-atilde}\\\\\n\t\t& \\mathbf{f}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right), \\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\label{eq:bilevel_static}\\\\\n\t\t& \\mathrm{Im}\\left(\\mathbf{M}_{j}(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_{j}(\\tilde{\\mathbf{a}})\\right),\\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\label{eq:bilevel_mass}\\\\\n\t\t& \\mathbf{W}_j^\\mathrm{T}\\left[ \\sum_{e=1}^{n_\\mathrm{e}}\\tilde{a}_e\\mathbf{K}_{j,e}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\sum_{e=1}^{n_\\mathrm{e}} \\tilde{a}_e \\mathbf{M}_{j,e}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0, \\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\label{eq:bilevel-membraine}\\\\\n\t\t& \\begin{aligned}\\label{eq:bilevel-lower}\n\t\t\t\\delta^* \\in \\argmin_{\\delta\\in\\mathbb{R}}\\; & \\delta\\\\\n\t\t\t\\mathrm{s.t.}\\; & \\begin{pmatrix}\n\t\t\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t\t\t-\\mathbf{f}_j & \\mathbf{K}_j (\\delta \\tilde{\\mathbf{a}})\n\t\t\t\\end{pmatrix} \\succeq 0, \\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\\\\n\t\t\t& \\mathbf{K}_j(\\delta \\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\delta \\tilde{\\mathbf{a}}) \\succeq 0, \\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\\\\n\t\t\t& \\delta \\ge 0,\n\t\t\\end{aligned}\n\t\\end{align}\n\\end{subequations}\n%\nwhere $\\mathbf{W}_j$ is an orthonormal basis of $\\mathrm{Ker}\\left(\\sum_{e=1}^{n_\\mathrm{e}}\\left[\\mathbf{K}_{j,e}^{\\langle 2\\rangle} + \\mathbf{K}_{j,e}^{\\langle 3\\rangle}\\right]\\right)$ and the constraints \\eqref{eq:bilevel-atilde}-\\eqref{eq:bilevel-membraine} ensure that the lower-level problem \\eqref{eq:bilevel-lower} {has a non-empty interior}. In particular, these {constraints} include nonnegative components of the cross-sectional ratios \\eqref{eq:bilevel-atilde}, static \\eqref{eq:bilevel_static} and mass \\eqref{eq:bilevel_mass} admissibility of the ratios, and the pure membrane eigenvalues to be at least $\\underline{\\lambda}_j$ \\eqref{eq:bilevel-membraine}, which we will prove in Proposition \\ref{th:feasibility_cond} (that uses the notation set up in \\eqref{eq:hatnotation}). Moreover, the lower-level problem \\eqref{eq:bilevel-lower} is quasi-convex in $\\delta$, allowing one to obtain a globally optimal scaling $\\delta^*$ for fixed ratios $\\tilde{\\mathbf{a}}$ using bisection. Proving the quasi-convexity property and showing that the conditions \\eqref{eq:bilevel-atilde}--\\eqref{eq:bilevel-membraine} are sufficient and necessary for solvability of the lower-level problem \\eqref{eq:bilevel-lower} constitutes the main goal of this section.\n\nLet us first focus on the lower-level problem \\eqref{eq:bilevel-lower}. We start by expressing the lowest nonzero\n%non-singular \nfree-vibration eigenvalue function $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ using the Raleigh quotient \\cite[Definition 2.2 and Appendix A.1]{Achtziger2008} as\n%\n\\begin{equation}\\label{eq:RQ}\n\t\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) = \\min_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\left[\\delta\\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} + \\delta^2\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\delta^3\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\right]  \\mathbf{v}_j}{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\delta\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j},\n\\end{equation}\n%\nin which we adopt the simplified notation\n%\n\\begin{equation}\\label{eq:hatnotation}\n\t\\hat{\\mathbf{K}}_j^{\\langle i\\rangle}(\\tilde{\\mathbf{a}}) = \\sum_{e=1}^{n_\\mathrm{e}}\\tilde{a}_e^i\\mathbf{K}_{j,e}^{\\langle i\\rangle}, \\qquad \n\t\\hat{\\mathbf{M}}_j^{\\langle i\\rangle}(\\tilde{\\mathbf{a}}) = \\sum_{e=1}^{n_\\mathrm{e}}\\tilde{a}_e^i \\mathbf{M}_{j,e}^{\\langle i\\rangle}.\n\\end{equation}\n%\nand omit the argument of $\\hat{\\mathbf{K}}_j^{\\langle i\\rangle}$ and $\\hat{\\mathbf{M}}_j^{\\langle i\\rangle}$ \nwhen it is clear from the context, to shorten the equations.\n\nBecause $\\forall j \\in \\mathcal{L}_{\\mathrm{fv}}: \\mathrm{Im}\\left(\\mathbf{M}_{j}(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_{j}(\\tilde{\\mathbf{a}})\\right)$ (see Proposition~\\ref{prop:exists_gamma_fv}), \\eqref{eq:RQ} renders that $\\lambda_{\\min,j}>0$. Thus, to prove feasibility of the eigenvalue constraint in \\eqref{eq:bilevel-lower} with respect to $\\delta$, it remains to find the condition under which $\\sup_{\\delta} \\lambda_{\\min,j} \\ge \\underline{\\lambda}_j$. \n\nBecause of $\\delta > 0$, equation~\\eqref{eq:RQ} can further be rewritten to\n%\n\\begin{equation}\\label{eq:RQs}\n\t\\begin{multlined}\n\t\t\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) =\\\\\n\t\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}.\n\t\\end{multlined}\n\\end{equation}\n","start":37715,"end":43857},{"raw_tex":"With this result, we have shown that by increasing $\\delta$ we do not decrease $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$. {Notice that this holds up to $\\delta\\rightarrow \\infty$, as we can choose $\\Delta\\rightarrow \\infty$ in the above proof.} For the solvability of the lower-level problem \\eqref{eq:bilevel-lower}, it is crucial to guarantee that there exits a $\\delta$ such that $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) \\ge \\underline{\\lambda}_j$. In other words, we require that $\\sup_{\\delta} \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) \\ge \\underline{\\lambda}_j$. To show this, let $\\mathbf{v}_j := \\mathbf{t}_j + \\mathbf{w}_j$, where $\\mathbf{t}_j \\in \\mathrm{Im}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) + \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right) $ and $\\mathbf{w}_j \\in \\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$.\nThen, we can write the expression for the supremum.\n","start":51189,"end":52352},{"raw_tex":"{Let us start with the first modification.} Since the objective function \\eqref{eq:opt_objective}, representing the weight of the structure, is a conic combination of nonnegative variables, any feasible point to the optimization problem \\eqref{eq:opt} provides an upper-bound on  the objective function value $\\overline{w}$ that also bounds the cross-section areas from above. This upper bound can be obtained by solving the convex problem \\eqref{eq:linprob}, providing the cross-sectional ratios $\\tilde{\\mathbf{a}}$, and{, subsequently,} by finding the minimal scaling factor $\\delta^*$ by solving the lower-level problem \\eqref{eq:bilevel-lower} globally using bisection. Consequently, the optimal cross-section areas are bounded as\n%\n\\begin{equation}\n\t0 \\le a_e \\le \\frac{\\overline{w}}{\\rho_e\\ell_e}, \\forall e \\in \\{1,\\dots,n_\\mathrm{e}\\}.\n\\end{equation}\n%\nFurthermore, using the substitution $a_e = \\frac{\\overline{w}}{2\\rho_e\\ell_e} (a_{\\mathrm{s},e} + 1)$, these bounds can be written as $a_{\\mathrm{s},e}^2 \\le 1$. %Similarly, we add a weight constraint to strengthen the relaxations as\n%\n%\\begin{equation}\n%    \\sum_{e=1}^{n_\\mathrm{e}} \\rho_e \\ell_e a_e \\le \\overline{w}\\quad\\longleftrightarrow\\quad 2-n_\\mathrm{e} - \\sum_{e=1}^{n_\\mathrm{e}} a_{\\mathrm{s},e} \\ge 0.\n%\\end{equation}\nSimilarly, we add a weight constraint $\\sum_{e=1}^{n_\\mathrm{e}} \\rho_e \\ell_e a_e \\le \\overline{w}$ to strengthen the relaxations.\nThe influence of this constraint on the feasible set of relaxations will be illustrated in Section~\\ref{sec:illustrative}. Using the above substitution of $a_e$, we obtain the formulation\n%\n\\begin{subequations}\\label{eq:opt_po}\n\t\\begin{align}\n\t\t\\min_{\\mathbf{a}_{{\\mathrm{s}}}\\in\\mathbb{R}^{n_\\mathrm{e}}}\\; & {\\frac{1}{2}} \\overline{w} (n_\\mathrm{e} + \\mathbf{1}^\\mathrm{T} \\mathbf{a}_\\mathrm{s})\\label{eq:opt_objective_po}\\\\\n\t\t\\text{s.t.}\\; & \\mathbf{K}_j(\\mathbf{a}_\\mathrm{s}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\mathbf{a}_\\mathrm{s}) \\succeq 0, \\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\label{eq:opt_fv_po}\\\\\n\t\t& \\begin{pmatrix}\n\t\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t\t-\\mathbf{f}_j & \\mathbf{K}_j(\\mathbf{a}_\\mathrm{s})\n\t\t\\end{pmatrix} \\succeq 0,\\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\label{eq:opt_static_po}\\\\\n            & 2-n_\\mathrm{e} - \\sum_{e=1}^{n_\\mathrm{e}} a_{\\mathrm{s},e} \\ge 0,\\label{eq:opt_weight_po}\\\\\n\t\t& 1 - a_\\mathrm{s,e}^2 \\ge 0, \\quad e \\in \\{1,\\dots,n_\\mathrm{e}\\}\\label{eq:opt_nonneg_po}\n\t\\end{align}\n\\end{subequations}\n%\n{in the scaled variables $\\mathbf{a}_{\\mathrm{s}} \\in \\mathbb{R}^{n_\\mathrm{e}}$. Here,} we slightly abuse the notation and express the stiffness and mass matrices in terms of the scaled variables $\\mathbf{a}_\\mathrm{s}$ as $\\mathbf{K}_j(\\mathbf{a}_\\mathrm{s})$ and $\\mathbf{M}_j(\\mathbf{a}_\\mathrm{s})$, respectively. {Although this compactification means that \\eqref{eq:opt} and \\eqref{eq:opt_po} do not have the same feasible set (after rescaling), they maintain the same set of optimal solutions. Thus, solution to \\eqref{eq:opt_po} also solves \\eqref{eq:opt}.}\n","start":66732,"end":69781},{"raw_tex":"\\subsubsection{Feasible upper bounds from first-order moments}\nIn what follows, we show that the first-order moments obtained by solving any relaxation of \\eqref{eq:opt_po} satisfy the {feasibility conditions} set in Proposition \\ref{th:feasibility_cond}, which allows us to construct a feasible upper-bound solution to \\eqref{eq:opt}.\n\nTo show this, let us introduce the notation\n%\n\\begin{subequations}\n\t\\begin{align}\n\t\ta_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}) &= 0.5\\,{\\frac{\\overline{w}}{\\rho_e \\ell_e}} (y_{a_{\\mathrm{s},e}^1} + 1),\\label{eq:rel_a_1}\\\\\n\t\ta_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}) &= 0.25\\, {\\frac{\\overline{w}^2}{\\rho_e^2 \\ell_e^2}} \\left( y_{a_{\\mathrm{s},e}^2} + 2 y_{a_{\\mathrm{s},e}^1} + 1 \\right),\\label{eq:rel_a_2}\\\\\n\t\ta_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}) &= 0.125\\, {\\frac{\\overline{w}^3}{\\rho_e^3 \\ell_e^3}} \\left( y_{a_{\\mathrm{s},e}^3} + 3 y_{a_{\\mathrm{s},e}^2} + 3 y_{a_{\\mathrm{s},e}^1} + 1 \\right)\\label{eq:rel_a_3}\n\t\\end{align}\n\\end{subequations}\n%\nfor evaluating {expected values of the} monomials of the (unscaled) cross-section areas based on the (scaled) moments $\\mathbf{y}$. In this notation, the superscript $\\bullet^{\\langle i\\rangle}$ indicates that it corresponds to the $i$-th degree monomial of $\\bullet$. We wish to emphasize here that, generally, $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}) \\neq \\left[a_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y})\\right]^2$ and $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}) \\neq \\left[a_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y})\\right]^3$ due to the relaxation procedure.\n\nAs the first step in the proof, we need to show that replacing higher-order moments with the powers of the first-order moments does not change the rank of the stiffness matrix. Since the stiffness matrix is a conic combination of positive semidefinite matrices with the coefficients being monomials of the cross-section areas, recall \\eqref{eq:stiffness}, it suffices to show that strict positivity of $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y})$ or $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y})$ implies strict positivity of $a_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y})$.\n","start":71353,"end":73553}]},"statement":{"raw_tex":"\\begin{proposition}\\label{prop:feasibleub}\n\tLet $\\mathbf{y}^{*(r)}$ be the optimal moments obtained by solving the $r$-th degree relaxation and set $\\tilde{\\mathbf{a}} := \\mathbf{a}_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}^{*(r)})$. Then, there exists a $\\delta$ such that $\\mathbf{a}(\\delta):=\\delta\\tilde{\\mathbf{a}}$ is feasible to the lower-level problem \\eqref{eq:bilevel-lower}.\n\\end{proposition}","units":[{"raw_tex":"\\begin{proposition}\\label{prop:feasibleub}\n\tLet $\\mathbf{y}^{*(r)}$ be the optimal moments obtained by solving the $r$-th degree relaxation and set $\\tilde{\\mathbf{a}} := \\mathbf{a}_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}^{*(r)})$. Then, there exists a $\\delta$ such that $\\mathbf{a}(\\delta):=\\delta\\tilde{\\mathbf{a}}$ is feasible to the lower-level problem \\eqref{eq:bilevel-lower}.\n\\end{proposition}","start":80750,"end":81158}]},"proof":{"raw_tex":"\\begin{proposition}[Mass admissibility]\\label{lemma:dynamic_admissibility}\n\tLet $$\\mathbf{\\hat{a}}\\in\\left\\{\\mathbf{a} \\; \\vert \\; \\mathbf{K}_j(\\mathbf{a}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\mathbf{a}) \\succeq 0, \\mathbf{a}\\ge \\mathbf{0}\\right\\}.$$ Then, $\\mathrm{Im}\\left(\\mathbf{M}_j(\\hat{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\hat{\\mathbf{a}})\\right)$.\n\\end{proposition}\n\n\\begin{proof}\n\tThe matrix inequality \\eqref{eq:opt_fv} is equivalent to\n\t%\n\t\\begin{equation}\\label{eq:inimage}\n\t\t\\mathbf{v}_j^\\mathrm{T}\\mathbf{K}_j(\\mathbf{a})\\mathbf{v}_j - \\underline{\\lambda}_j \\mathbf{v}_j^\\mathrm{T}\\mathbf{M}_j(\\mathbf{a})\\mathbf{v}_j\\ge 0\\quad \\forall \\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}.\n\t\\end{equation}\n\t%\n\tLet us now proceed by contradiction and assume that $\\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right) \\nsubseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\mathbf{a})\\right)$. Then, there must exist $\\hat{\\mathbf{v}}_j\\in\\left\\{\\mathbf{v}_j\\;\\vert\\; \\mathbf{v}_j\\in \\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right), \\mathbf{v}_j\\notin \\mathrm{Im}\\left(\\mathbf{K}_j(\\mathbf{a})\\right)\\right\\}$, for which the left-hand side of \\eqref{eq:inimage} evaluates as\n\t%\n\t\\begin{equation}\n\t\t-\\underline{\\lambda}_j \\hat{\\mathbf{v}}_j^\\mathrm{T}\\mathbf{M}_j(\\mathbf{a})\\hat{\\mathbf{v}}_j < 0,\n\t\\end{equation}\n\t%\n\twhich is strictly negative because $\\underline{\\lambda}_j>0$, $\\mathbf{M}_j(\\mathbf{a}) \\succeq 0$, and $\\hat{\\mathbf{v}}_j\\in \\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right)$, contradicting~\\eqref{eq:inimage}.\n\\end{proof}\n\n\\begin{proposition}[{Positive finite eigenvalue}]\\label{prop:exists_gamma_fv}\n\tLet $\\mathrm{Im}(\\mathbf{M}_j(\\mathbf{a})) \\subseteq \\mathrm{Im}(\\mathbf{K}_j(\\mathbf{a}))$. Then, there exists $\\hat{\\lambda}>0$ such that\n\t%\n\t$\\mathbf{K}_j(\\mathbf{a}) - \\hat{\\lambda} \\mathbf{M}_j(\\mathbf{a})\\succeq 0$.\n\\end{proposition}\n\n\\begin{proof}\nSince $\\mathbf{K}_j(\\mathbf{a}) \\in \\mathbb{S}^{{n_\\mathrm{dof},j}}_{\\succeq 0}$ and $\\mathbf{M}_j(\\mathbf{a}) \\in \\mathbb{S}^{{n_\\mathrm{dof},j}}_{\\succeq 0}$, the nonzero eigenvalues of the generalized eigenvalue problem $\\mathbf{K}_j(\\mathbf{a})\\mathbf{v}_j - \\lambda \\mathbf{M}_j(\\mathbf{a}) \\mathbf{v}_j = \\mathbf{0}$, with $\\mathbf{v}_j \\in \\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right)$, must be real and positive. Thus, setting $0<\\hat{\\lambda}= \\lambda_1$ with $\\lambda_1$ being the lowest nonzero eigenvalue, the eigenvalue constraint is satisfied.\n\\end{proof}\n\n\\begin{proposition}\\label{prop:monotonic}\nLet $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ be as defined in \\eqref{eq:RQs} and $j \\in \\mathcal{L}_\\mathrm{fv}$. Then, $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ is a non-decreasing function.\n\\begin{proof}\nLet $\\Delta > 0$; we need to show that $\\lambda_{\\min,j}((\\delta+\\Delta) \\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})\\ge 0$. Using the definition \\eqref{eq:RQs}, we have %\\textcolor{purple}{ We don't really need to write equations (3.11), it is clear enough to directly proceed to (3.12). }\n\\begin{multline}\n\t\\lambda_{\\min,j}((\\delta + \\Delta)\\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})=\\\\\n\t%\n\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + (\\delta+\\Delta) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + (\\delta+\\Delta)^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta }\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\\\\n\t%\n\t- \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}.\n\\end{multline}\n%\nSplitting the first infimum into two parts using the triangle inequality, we obtain \n%\n\\begin{multline}\\label{eq:lamdiff}\n\t\\lambda_{\\min,j}((\\delta + \\Delta)\\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})\\ge\\\\\n\t%\n\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\\\\n\t%\n\t+ \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\Delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + (2\\delta \\Delta + \\Delta^2) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}\\\\\n\t%\n\t- \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}.\n\\end{multline}\n%\nBecause $\\forall \\delta,\\Delta >0: \\frac{1}{\\delta} \\ge \\frac{1}{\\delta + \\Delta}$, it holds for all $\\mathbf{v}_j$, including the minimizing ones, that\n\\begin{equation}\n\t\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j \\ge \\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j.\n\\end{equation}\n%\nConsequently,\n% \n\\begin{multline}\n\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\\\\n\t- \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\ge 0,\n\\end{multline}\n%\nwhich allows us to simplify \\eqref{eq:lamdiff} to\n%\n\\begin{equation}\\label{eq:monotoneous}\n\\begin{multlined}\n\\lambda_{\\min,j}((\\delta + \\Delta)\\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) \\ge\\\\\n \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\\frac{\\Delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + (2\\delta \\Delta + \\Delta^2) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\ge 0.\n\\end{multlined}\n\\end{equation}\n%\nIn \\eqref{eq:monotoneous}, the nonnegativity follows from $\\Delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j \\ge 0$ and $\\left(2\\delta \\Delta + \\Delta^2\\right) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j\\ge 0$, which is because of $\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}, \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\succeq 0$ and $\\delta\\ge 0, \\Delta > 0$.\n\\end{proof}\n\\end{proposition}\n\n\\begin{proposition}\\label{prop:supremum}\nLet $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ be as defined in \\eqref{eq:RQs}, and let $\\mathbf{v}_j := \\mathbf{t}_j + \\mathbf{w}_j$ as defined above. Then, \n\\begin{equation}\\label{eq:supremum}\n\t\\sup_{\\delta} \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) = \\inf_{\\mathbf{w}_j \\in \\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{w}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{w}_j}{\\mathbf{w}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{w}_j}.\n\\end{equation}\n\\begin{proof}\nWe prove the statement by splitting $\\mathbf{v}_j$ into $\\mathbf{t}_j$ and $\\mathbf{w}_j$ and showing that it must hold that $\\mathbf{t}_j=\\mathbf{0}$.  For this splitting, the supremum is expressed as\n\\begin{equation}\\label{eq:splitting}\n\t\\begin{multlined}\n\t\t\\sup_\\delta \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})= \\\\\n\t\t\\sup_\\delta \\inf\\limits_{\\mathbf{t}_j, \\mathbf{w}_j}\n\t\t \\frac{\\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\left(\\mathbf{t}_j+\\mathbf{w}_j\\right) + \\delta\\mathbf{t}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\mathbf{t}_j + \\delta^2\\mathbf{t}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\mathbf{t}_j }{\\frac{1}{\\delta}\\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\left(\\mathbf{t}_j+\\mathbf{w}_j\\right) + \\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)}.\n\t\\end{multlined}\n\\end{equation}\n%\nBecause of the monotonicity (recall Proposition~\\ref{prop:monotonic}), $\\sup_\\delta \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) = \\lim\\limits_{\\delta\\rightarrow \\infty}\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$. Further, suppose by {contradiction} that $\\mathbf{t}_j \\neq \\mathbf{0}$ holds at the infimum. Then, $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) \\rightarrow \\infty$ as $\\delta \\rightarrow \\infty$, due to $\\mathbf{w}_j \\in \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$ and because $\\mathbf{t}_j$ cannot be in $\\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right)$. On the other hand, for $\\mathbf{t}_j = \\mathbf{0}$, $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ is finite for all $\\delta$ due to $\\mathbf{w}_j \\in \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$. Consequently, $\\mathbf{t}_j^*=\\mathbf{0}$ must hold at the infimum. After inserting $\\mathbf{t}_j^*=\\mathbf{0}$ and $\\delta^*\\rightarrow \\infty$ into \\eqref{eq:splitting}, we receive \\eqref{eq:supremum}, which completes the proof.\n\\end{proof}\n\\end{proposition}\n\n\\begin{proposition}\\label{prop:scaling}\n\tThere exists $\\delta$ such that $\\mathbf{K}_j(\\delta \\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\delta \\tilde{\\mathbf{a}}) \\succeq 0$ if and only if $\\tilde{\\mathbf{a}} \\in \\left\\{\\mathbf{a} \\;\\vert\\; \\mathbf{a}\\ge \\mathbf{0}, \\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\mathbf{a})\\right),\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0\\right\\}$, where $\\mathbf{W}_j$ denotes an orthonormal basis of $\\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$.\n\\end{proposition}\n\n\\begin{proof}\n\\fbox{$\\Rightarrow$} Let $\\mathbf{K}_j(\\delta \\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\delta \\tilde{\\mathbf{a}}) \\succeq 0$ hold. Then, $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ based on Proposition \\ref{lemma:dynamic_admissibility}. Next, we project the matrix inequality using $\\mathbf{W}_j$, yielding\n%\n\\begin{equation}\n\t\\delta \\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} \\mathbf{W}_j- \\underline{\\lambda}_j \\delta \\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{W}_j - \\underline{\\lambda}_j\\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{W}_j \\succeq 0.\n\\end{equation}\n%\nFor $\\delta>0$, this is further equivalent to\n%\n\\begin{equation}\n\t\\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} \\mathbf{W}_j- \\underline{\\lambda}_j \\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{W}_j \\succeq \\frac{1}{\\delta}\\underline{\\lambda}_j\\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{W}_j \\succeq 0.\n\\end{equation}\n%\n\\fbox{$\\Leftarrow$} Let $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ and $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$ hold. Because of the first assumption and Proposition \\ref{prop:exists_gamma_fv}, it holds that\n%\n\\begin{equation}\n\\forall \\mathbf{v}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right) \\exists \\gamma>0: \\mathbf{v}_j^\\mathrm{T}\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\mathbf{v}_j - \\gamma\\mathbf{v}_j^\\mathrm{T}\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\mathbf{v}_j \\ge 0,\n\\end{equation}\n%\nand thus also $\\mathbf{K}_j(\\tilde{\\mathbf{a}}) - \\gamma\\mathbf{M}_j(\\tilde{\\mathbf{a}}) \\succeq 0$ for $\\gamma > 0$. Consequently, it remains to show that $\\gamma\\ge \\underline{\\lambda}_j$.\n\nBecause of $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$, we also have\n%\n\\begin{equation}\\label{eq:inequality}\n\t\\forall \\mathbf{w}_j \\in \\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right):  \\mathbf{w}_j^\\mathrm{T} \\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right] \\mathbf{w}_j \\ge 0.\n\\end{equation}\n%\nIn particular, the above holds for $w_j$ minimizing the left-hand side of \\eqref{eq:inequality}, yielding\n%\n\\begin{equation}\n\t\\underline{\\lambda}_j \\le \\inf_{\\mathbf{w}_j \\in \\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{w}_j^\\mathrm{T} \\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} \\mathbf{w}_j}{\\mathbf{w}_j^\\mathrm{T} \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\mathbf{w}_j} = \\sup_{\\delta} \\lambda_{\\min,j}(\\mathbf{a}(\\delta))\n\\end{equation}\n%\nbased on Proposition \\ref{prop:supremum}.\n\\end{proof}\n\n\\begin{proposition}\\label{th:feasibility_cond}\n    The optimization problem \\eqref{eq:bilevel-lower} has a non-empty interior if and only if $\\tilde{\\mathbf{a}} \\in \\mathbb{R}^{n_{\\mathrm{e}}}$ satisfies {the following feasibility conditions:}\n    {\\begin{itemize}\n    \\item $\\tilde{\\mathbf{a}} \\ge \\mathbf{0}$,\n    \\item $\\mathbf{f}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j (\\tilde{\\mathbf{a}})\\right)$ for all $j \\in \\mathcal{L}_\\mathrm{s}$,\n    \\item $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ for all $j \\in \\mathcal{L}_\\mathrm{fv}$,\n    \\item $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$ for all $j \\in \\mathcal{L}_\\mathrm{fv}$,\n    \\end{itemize}}%\n    %\n    \\noindent where $\\mathbf{W}_j$ is an orthonormal basis of $\\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$.\n\\end{proposition}\n\n\\begin{proof}\n\tBased on \\cite[Proposition 5]{Tyburec2023}, for $\\mathbf{f}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j (\\tilde{\\mathbf{a}})\\right)$ there exists a finite $\\delta_{\\mathrm{c},j}^*>0$ such that the static matrix inequality in \\eqref{eq:bilevel-lower} is satisfied. Conversely, for $\\mathbf{f}_j \\notin \\mathrm{Im}\\left(\\mathbf{K}_j (\\tilde{\\mathbf{a}})\\right)$, the optimization problem \\eqref{eq:bilevel-lower} is infeasible.\n\t\n\tUsing Proposition \\ref{prop:scaling}, there exists $\\delta_{\\mathrm{fv},j}^*>0$ such that the free-vibration constraint is satisfied if and only if $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ and $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$.\n    Finally, there exists $\\delta^* = \\max\\{\\max_{j{\\in \\mathcal{L}_\\mathrm{s}}} \\delta^*_{\\mathrm{c},j}, \\max_{j{\\in \\mathcal{L}_\\mathrm{fv}}} \\delta^*_{\\mathrm{fv},j} \\}>0$ such that the free-vibration and compliance constraints are satisfied if and only if the {right-hand-side} conditions of the proposition are satisfied. This is (i) because we have upper bounds for the compliance $\\overline{c}_j$ and compliance is a nonincreasing function in $\\delta$ \\cite[Proposition 1]{Tyburec2023}, and (ii) because we have lower bounds for the lowest free-vibration eigenvalues $\\underline{\\lambda}_j$ and the lowest free-vibration eigenvalue function is a nondecreasing function in $\\delta$ (Proposition \\ref{prop:monotonic}).\n\\end{proof}\n\n\\begin{proposition}\\label{prop:inertia}\n\tLet $\\mathbf{y}^{*(r)}$ be the optimal moments associated with the cross-section areas $\\mathbf{a}$ obtained by solving the $r$-th degree relaxation. If $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)}) > 0$ or $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)}) > 0$, then $a_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}^{*(r)}) > 0$.\n\t%\n\t\\begin{proof}\n        By contradiction, let us assume that $a_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}^{*(r)}) = 0$, i.e., $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$. Then, we need to show that $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)}) = 0$ and $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)}) = 0$.\n\t\t\n\t\tConsider first the case of $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)})$. The moment matrix {(recall the definition in \\eqref{eq:moment})} then contains the principal submatrix\n\t\t%\n\t\t\\begin{equation}\n\t\t\t\\begin{pmatrix}\n\t\t\t\t1 & y_{a_{\\mathrm{s},e}^1}^{*(r)}\\\\\n\t\t\t\ty_{a_{\\mathrm{s},e}^1}^{*(r)} & y_{a_{\\mathrm{s},e}^2}^{*(r)}\n\t\t\t\\end{pmatrix} \\succeq 0.\n\t\t\\end{equation}\n\t\t%\n\t\tThe determinant of this matrix must be nonnegative, so that we obtain the inequality\n\t\t%\n\t\t\\begin{equation}\n\t\t\ty_{a_{\\mathrm{s},e}^2}^{*(r)} - \\left[y_{a_{\\mathrm{s},e}^1}^{*(r)}\\right]^2 \\ge 0.\n\t\t\\end{equation}\n\t\t%\n\t\tAfter inserting $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$, we obtain $y_{a_{\\mathrm{s},e}^2}^{*(r)} \\ge 1$. Due to the bound constraints \\eqref{eq:opt_nonneg_po}, $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$. Using $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$ with $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$, \\eqref{eq:rel_a_2} is then evaluated as $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)})=0$.\n\t\t\n\t\tFor the case of, $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)})$, the moment matrix contains the principal submatrix\n\t\t%\n\t\t\\begin{equation}\n\t\t\t\\begin{pmatrix}\n\t\t\t\t1 & y_{a_{\\mathrm{s},e}^1}^{*(r)} & y_{a_{\\mathrm{s},e}^2}^{*(r)}\\\\\n\t\t\t\ty_{a_{\\mathrm{s},e}^1}^{*(r)} & y_{a_{\\mathrm{s},e}^2}^{*(r)} & y_{a_{\\mathrm{s},e}^3}^{*(r)}\\\\\n\t\t\t\ty_{a_{\\mathrm{s},e}^2}^{*(r)} & y_{a_{\\mathrm{s},e}^3}^{*(r)} & y_{a_{\\mathrm{s},e}^4}^{*(r)}\n\t\t\t\\end{pmatrix} \\succeq 0.\n\t\t\\end{equation}\n\t\t%\n\t\tAfter inserting $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$ and $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$, the determinant evaluates as\n\t\t%\n\t\t\\begin{equation}\n\t\t\t- \\left(y_{a_{\\mathrm{s},e}^3}^{*(r)} + 1\\right)^2 \\ge 0,\n\t\t\\end{equation}\n\t\t%\n\t\tso that $y_{a_{\\mathrm{s},e}^3}^{*(r)}=-1$ is the only feasible solution. Inserting $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$, $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$, and $y_{a_{\\mathrm{s},e}^3}^{*(r)} = -1$ into \\eqref{eq:rel_a_3}, we again receive $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)})=0$, which finishes the proof.\n\t\\end{proof}\n\\end{proposition}\n\n\\begin{proposition}\\label{th:ub}\n\tLet $\\mathbf{y}^{*(r)}$ be the optimal moments obtained by solving the $r$-th degree relaxation and set $\\tilde{\\mathbf{a}} := \\mathbf{a}_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}^{*(r)})$. Then, there exists $\\delta$ such that\n\t%\n\t\\begin{equation}\\label{eq:scalar_fvineq}\n\t\t\\mathbf{K}_j(\\delta\\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\delta\\tilde{\\mathbf{a}}) \\succeq 0\\,.\n\t\\end{equation}\n\\end{proposition}\n\n\\begin{proof}\n\tFrom the hierarchy construction {\\eqref{eq:general_moment}}, the {localizing matrix \\eqref{eq:localizing} will always contain $L_\\mathbf{y}(1 \\otimes \\left[\\mathbf{K}_j (\\mathbf{a}) - \\underline{\\lambda} \\mathbf{M}_j(\\mathbf{a})\\right])$ as principal submatrix due to} \\eqref{eq:opt_fv}{, which} can be written in terms of the lifted variables as\n\t%\n\t\\begin{multline}\\label{eq:fvrelaxed}\n\t\t\\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)})\\right]^{\\circ\\frac{1}{2}}\\right) + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)})\\right]^{\\circ\\frac{1}{3}}\\right) \\\\\n\t\t- \\underline{\\lambda}_j\\left( \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) \\right)\\succeq 0,\n\t\\end{multline}\n\t%\n\twith $\\bullet^{\\circ q}$ raising each element in $\\bullet$ to the power of $q$. We note here that the power factors in \\eqref{eq:fvrelaxed} eliminate the powers introduced in the notation \\eqref{eq:hatnotation}, so that \\eqref{eq:fvrelaxed} is indeed linear in $\\mathbf{a}_\\mathrm{rel}^{(i)}$ for all $i$.\n \n    After projecting via an orthonormal basis of $\\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$ denoted with $\\mathbf{W}_j$, we receive\n\t%\n\t\\begin{equation}\\label{eq:rel_cond1}\n\t\t\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) \\right]\\mathbf{W}_j \\succeq \\underline{\\lambda}_j \\mathbf{W}_j^\\mathrm{T} \\hat{\\mathbf{M}}_{j}^{\\langle 0\\rangle}\\mathbf{W}_j \\succeq 0.\n\t\\end{equation}\n\t%\n\tFurther, because of Proposition \\ref{prop:inertia}, {$a_{\\mathrm{rel},e}^{\\langle 1 \\rangle} \\mathbf{K}_{j,e}^{\\langle 1\\rangle}\\neq \\mathbf{0}$} whenever {$a_{\\mathrm{rel},e}^{\\langle2\\rangle} \\mathbf{K}_{j,e}^{\\langle 2\\rangle} + a_{\\mathrm{rel},e}^{\\langle3 \\rangle} \\mathbf{K}_{j,e}^{\\langle 3\\rangle} \\neq \\mathbf{0}$}. Consequently, \n\t\\begin{multline}\\label{eq:spans}\n\t\t\\mathrm{Span}\\left(\\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} (\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 2\\rangle}\\right]^{\\circ\\frac{1}{2}}\\right) + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 3\\rangle}\\right]^{\\circ\\frac{1}{3}}\\right)\\right)\\\\ = \\mathrm{Span}\\left(\\hat{\\mathbf{K}}_j^{\\langle 1 \\rangle} (\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}(\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{{\\langle 3\\rangle}}(\\tilde{\\mathbf{a}}) \\right).\n\t\\end{multline}\n\t%\n\tThus, because \\eqref{eq:fvrelaxed} is feasible due to $\\mathbf{y}^{*(r)}$ being optimal, it holds based on Proposition \\ref{lemma:dynamic_admissibility} and based on \\eqref{eq:spans} that \n\t%\n\t\\begin{equation}\\label{eq:rel_cond2}\n\t\t\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right).\n\t\\end{equation}\n\t%\n\tUsing \\eqref{eq:rel_cond1} and \\eqref{eq:rel_cond2}, Proposition \\ref{prop:scaling} certifies existence of $\\delta$ such that the matrix inequality \\eqref{eq:scalar_fvineq} holds.\n\\end{proof}\n\n\\begin{proof}\n\tFor free-vibration constraints, the proof follows from Proposition \\ref{th:ub}. For static constraints, based on Proposition \\ref{prop:inertia}, equation~\\eqref{eq:spans} holds. Thus, since\n\t%\n\t\\begin{equation}\n\t\t\\begin{pmatrix}\n\t\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t\t-\\mathbf{f}_j & \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 2\\rangle}\\right]^{\\circ\\frac{1}{2}}\\right) + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 3\\rangle}\\right]^{\\circ\\frac{1}{3}}\\right)\n\t\t\\end{pmatrix} \\succeq 0\n\t\\end{equation}\n\t%\n\tis satisfied by optimality of $\\mathbf{y}^{*(r)}$, the condition $\\mathbf{f}_j \\in \\mathbf{K}_j(\\mathbf{\\tilde{a}})$ holds true due to Proposition \\ref{lemma:statical_admissibility}. Based on Proposition \\ref{th:feasibility_cond}, the upper bound solution can thus be constructed.\n\t\n\tConsequently, there exists $\\delta$ such that both types of constraints are satisfied simultaneously.\n\\end{proof}","units":[{"raw_tex":"\\begin{proposition}[Mass admissibility]\\label{lemma:dynamic_admissibility}\n\tLet $$\\mathbf{\\hat{a}}\\in\\left\\{\\mathbf{a} \\; \\vert \\; \\mathbf{K}_j(\\mathbf{a}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\mathbf{a}) \\succeq 0, \\mathbf{a}\\ge \\mathbf{0}\\right\\}.$$ Then, $\\mathrm{Im}\\left(\\mathbf{M}_j(\\hat{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\hat{\\mathbf{a}})\\right)$.\n\\end{proposition}","start":33537,"end":33935},{"raw_tex":"\\begin{proof}\n\tThe matrix inequality \\eqref{eq:opt_fv} is equivalent to\n\t%\n\t\\begin{equation}\\label{eq:inimage}\n\t\t\\mathbf{v}_j^\\mathrm{T}\\mathbf{K}_j(\\mathbf{a})\\mathbf{v}_j - \\underline{\\lambda}_j \\mathbf{v}_j^\\mathrm{T}\\mathbf{M}_j(\\mathbf{a})\\mathbf{v}_j\\ge 0\\quad \\forall \\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}.\n\t\\end{equation}\n\t%\n\tLet us now proceed by contradiction and assume that $\\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right) \\nsubseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\mathbf{a})\\right)$. Then, there must exist $\\hat{\\mathbf{v}}_j\\in\\left\\{\\mathbf{v}_j\\;\\vert\\; \\mathbf{v}_j\\in \\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right), \\mathbf{v}_j\\notin \\mathrm{Im}\\left(\\mathbf{K}_j(\\mathbf{a})\\right)\\right\\}$, for which the left-hand side of \\eqref{eq:inimage} evaluates as\n\t%\n\t\\begin{equation}\n\t\t-\\underline{\\lambda}_j \\hat{\\mathbf{v}}_j^\\mathrm{T}\\mathbf{M}_j(\\mathbf{a})\\hat{\\mathbf{v}}_j < 0,\n\t\\end{equation}\n\t%\n\twhich is strictly negative because $\\underline{\\lambda}_j>0$, $\\mathbf{M}_j(\\mathbf{a}) \\succeq 0$, and $\\hat{\\mathbf{v}}_j\\in \\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right)$, contradicting~\\eqref{eq:inimage}.\n\\end{proof}","start":33936,"end":35093},{"raw_tex":"\\begin{proposition}[{Positive finite eigenvalue}]\\label{prop:exists_gamma_fv}\n\tLet $\\mathrm{Im}(\\mathbf{M}_j(\\mathbf{a})) \\subseteq \\mathrm{Im}(\\mathbf{K}_j(\\mathbf{a}))$. Then, there exists $\\hat{\\lambda}>0$ such that\n\t%\n\t$\\mathbf{K}_j(\\mathbf{a}) - \\hat{\\lambda} \\mathbf{M}_j(\\mathbf{a})\\succeq 0$.\n\\end{proposition}","start":36743,"end":37061},{"raw_tex":"\\begin{proof}\nSince $\\mathbf{K}_j(\\mathbf{a}) \\in \\mathbb{S}^{{n_\\mathrm{dof},j}}_{\\succeq 0}$ and $\\mathbf{M}_j(\\mathbf{a}) \\in \\mathbb{S}^{{n_\\mathrm{dof},j}}_{\\succeq 0}$, the nonzero eigenvalues of the generalized eigenvalue problem $\\mathbf{K}_j(\\mathbf{a})\\mathbf{v}_j - \\lambda \\mathbf{M}_j(\\mathbf{a}) \\mathbf{v}_j = \\mathbf{0}$, with $\\mathbf{v}_j \\in \\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right)$, must be real and positive. Thus, setting $0<\\hat{\\lambda}= \\lambda_1$ with $\\lambda_1$ being the lowest nonzero eigenvalue, the eigenvalue constraint is satisfied.\n\\end{proof}","start":37062,"end":37649},{"raw_tex":"\\begin{proposition}\\label{prop:monotonic}\nLet $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ be as defined in \\eqref{eq:RQs} and $j \\in \\mathcal{L}_\\mathrm{fv}$. Then, $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ is a non-decreasing function.\n\\begin{proof}\nLet $\\Delta > 0$; we need to show that $\\lambda_{\\min,j}((\\delta+\\Delta) \\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})\\ge 0$. Using the definition \\eqref{eq:RQs}, we have %\\textcolor{purple}{ We don't really need to write equations (3.11), it is clear enough to directly proceed to (3.12). }\n\\begin{multline}\n\t\\lambda_{\\min,j}((\\delta + \\Delta)\\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})=\\\\\n\t%\n\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + (\\delta+\\Delta) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + (\\delta+\\Delta)^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta }\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\\\\n\t%\n\t- \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}.\n\\end{multline}\n%\nSplitting the first infimum into two parts using the triangle inequality, we obtain \n%\n\\begin{multline}\\label{eq:lamdiff}\n\t\\lambda_{\\min,j}((\\delta + \\Delta)\\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})\\ge\\\\\n\t%\n\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\\\\n\t%\n\t+ \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\Delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + (2\\delta \\Delta + \\Delta^2) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}\\\\\n\t%\n\t- \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}.\n\\end{multline}\n%\nBecause $\\forall \\delta,\\Delta >0: \\frac{1}{\\delta} \\ge \\frac{1}{\\delta + \\Delta}$, it holds for all $\\mathbf{v}_j$, including the minimizing ones, that\n\\begin{equation}\n\t\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j \\ge \\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j.\n\\end{equation}\n%\nConsequently,\n% \n\\begin{multline}\n\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\\\\n\t- \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\ge 0,\n\\end{multline}\n%\nwhich allows us to simplify \\eqref{eq:lamdiff} to\n%\n\\begin{equation}\\label{eq:monotoneous}\n\\begin{multlined}\n\\lambda_{\\min,j}((\\delta + \\Delta)\\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) \\ge\\\\\n \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\\frac{\\Delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + (2\\delta \\Delta + \\Delta^2) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\ge 0.\n\\end{multlined}\n\\end{equation}\n%\nIn \\eqref{eq:monotoneous}, the nonnegativity follows from $\\Delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j \\ge 0$ and $\\left(2\\delta \\Delta + \\Delta^2\\right) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j\\ge 0$, which is because of $\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}, \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\succeq 0$ and $\\delta\\ge 0, \\Delta > 0$.\n\\end{proof}\n\\end{proposition}","start":44422,"end":51187},{"raw_tex":"\\begin{proposition}\\label{prop:supremum}\nLet $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ be as defined in \\eqref{eq:RQs}, and let $\\mathbf{v}_j := \\mathbf{t}_j + \\mathbf{w}_j$ as defined above. Then, \n\\begin{equation}\\label{eq:supremum}\n\t\\sup_{\\delta} \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) = \\inf_{\\mathbf{w}_j \\in \\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{w}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{w}_j}{\\mathbf{w}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{w}_j}.\n\\end{equation}\n\\begin{proof}\nWe prove the statement by splitting $\\mathbf{v}_j$ into $\\mathbf{t}_j$ and $\\mathbf{w}_j$ and showing that it must hold that $\\mathbf{t}_j=\\mathbf{0}$.  For this splitting, the supremum is expressed as\n\\begin{equation}\\label{eq:splitting}\n\t\\begin{multlined}\n\t\t\\sup_\\delta \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})= \\\\\n\t\t\\sup_\\delta \\inf\\limits_{\\mathbf{t}_j, \\mathbf{w}_j}\n\t\t \\frac{\\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\left(\\mathbf{t}_j+\\mathbf{w}_j\\right) + \\delta\\mathbf{t}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\mathbf{t}_j + \\delta^2\\mathbf{t}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\mathbf{t}_j }{\\frac{1}{\\delta}\\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\left(\\mathbf{t}_j+\\mathbf{w}_j\\right) + \\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)}.\n\t\\end{multlined}\n\\end{equation}\n%\nBecause of the monotonicity (recall Proposition~\\ref{prop:monotonic}), $\\sup_\\delta \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) = \\lim\\limits_{\\delta\\rightarrow \\infty}\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$. Further, suppose by {contradiction} that $\\mathbf{t}_j \\neq \\mathbf{0}$ holds at the infimum. Then, $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) \\rightarrow \\infty$ as $\\delta \\rightarrow \\infty$, due to $\\mathbf{w}_j \\in \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$ and because $\\mathbf{t}_j$ cannot be in $\\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right)$. On the other hand, for $\\mathbf{t}_j = \\mathbf{0}$, $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ is finite for all $\\delta$ due to $\\mathbf{w}_j \\in \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$. Consequently, $\\mathbf{t}_j^*=\\mathbf{0}$ must hold at the infimum. After inserting $\\mathbf{t}_j^*=\\mathbf{0}$ and $\\delta^*\\rightarrow \\infty$ into \\eqref{eq:splitting}, we receive \\eqref{eq:supremum}, which completes the proof.\n\\end{proof}\n\\end{proposition}","start":52353,"end":55259},{"raw_tex":"\\begin{proposition}\\label{prop:scaling}\n\tThere exists $\\delta$ such that $\\mathbf{K}_j(\\delta \\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\delta \\tilde{\\mathbf{a}}) \\succeq 0$ if and only if $\\tilde{\\mathbf{a}} \\in \\left\\{\\mathbf{a} \\;\\vert\\; \\mathbf{a}\\ge \\mathbf{0}, \\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\mathbf{a})\\right),\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0\\right\\}$, where $\\mathbf{W}_j$ denotes an orthonormal basis of $\\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$.\n\\end{proposition}","start":55919,"end":56765},{"raw_tex":"\\begin{proof}\n\\fbox{$\\Rightarrow$} Let $\\mathbf{K}_j(\\delta \\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\delta \\tilde{\\mathbf{a}}) \\succeq 0$ hold. Then, $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ based on Proposition \\ref{lemma:dynamic_admissibility}. Next, we project the matrix inequality using $\\mathbf{W}_j$, yielding\n%\n\\begin{equation}\n\t\\delta \\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} \\mathbf{W}_j- \\underline{\\lambda}_j \\delta \\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{W}_j - \\underline{\\lambda}_j\\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{W}_j \\succeq 0.\n\\end{equation}\n%\nFor $\\delta>0$, this is further equivalent to\n%\n\\begin{equation}\n\t\\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} \\mathbf{W}_j- \\underline{\\lambda}_j \\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{W}_j \\succeq \\frac{1}{\\delta}\\underline{\\lambda}_j\\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{W}_j \\succeq 0.\n\\end{equation}\n%\n\\fbox{$\\Leftarrow$} Let $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ and $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$ hold. Because of the first assumption and Proposition \\ref{prop:exists_gamma_fv}, it holds that\n%\n\\begin{equation}\n\\forall \\mathbf{v}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right) \\exists \\gamma>0: \\mathbf{v}_j^\\mathrm{T}\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\mathbf{v}_j - \\gamma\\mathbf{v}_j^\\mathrm{T}\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\mathbf{v}_j \\ge 0,\n\\end{equation}\n%\nand thus also $\\mathbf{K}_j(\\tilde{\\mathbf{a}}) - \\gamma\\mathbf{M}_j(\\tilde{\\mathbf{a}}) \\succeq 0$ for $\\gamma > 0$. Consequently, it remains to show that $\\gamma\\ge \\underline{\\lambda}_j$.\n\nBecause of $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$, we also have\n%\n\\begin{equation}\\label{eq:inequality}\n\t\\forall \\mathbf{w}_j \\in \\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right):  \\mathbf{w}_j^\\mathrm{T} \\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right] \\mathbf{w}_j \\ge 0.\n\\end{equation}\n%\nIn particular, the above holds for $w_j$ minimizing the left-hand side of \\eqref{eq:inequality}, yielding\n%\n\\begin{equation}\n\t\\underline{\\lambda}_j \\le \\inf_{\\mathbf{w}_j \\in \\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{w}_j^\\mathrm{T} \\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} \\mathbf{w}_j}{\\mathbf{w}_j^\\mathrm{T} \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\mathbf{w}_j} = \\sup_{\\delta} \\lambda_{\\min,j}(\\mathbf{a}(\\delta))\n\\end{equation}\n%\nbased on Proposition \\ref{prop:supremum}.\n\\end{proof}","start":56766,"end":60141},{"raw_tex":"\\begin{proposition}\\label{th:feasibility_cond}\n    The optimization problem \\eqref{eq:bilevel-lower} has a non-empty interior if and only if $\\tilde{\\mathbf{a}} \\in \\mathbb{R}^{n_{\\mathrm{e}}}$ satisfies {the following feasibility conditions:}\n    {\\begin{itemize}\n    \\item $\\tilde{\\mathbf{a}} \\ge \\mathbf{0}$,\n    \\item $\\mathbf{f}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j (\\tilde{\\mathbf{a}})\\right)$ for all $j \\in \\mathcal{L}_\\mathrm{s}$,\n    \\item $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ for all $j \\in \\mathcal{L}_\\mathrm{fv}$,\n    \\item $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$ for all $j \\in \\mathcal{L}_\\mathrm{fv}$,\n    \\end{itemize}}%\n    %\n    \\noindent where $\\mathbf{W}_j$ is an orthonormal basis of $\\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$.\n\\end{proposition}","start":60363,"end":61513},{"raw_tex":"\\begin{proof}\n\tBased on \\cite[Proposition 5]{Tyburec2023}, for $\\mathbf{f}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j (\\tilde{\\mathbf{a}})\\right)$ there exists a finite $\\delta_{\\mathrm{c},j}^*>0$ such that the static matrix inequality in \\eqref{eq:bilevel-lower} is satisfied. Conversely, for $\\mathbf{f}_j \\notin \\mathrm{Im}\\left(\\mathbf{K}_j (\\tilde{\\mathbf{a}})\\right)$, the optimization problem \\eqref{eq:bilevel-lower} is infeasible.\n\t\n\tUsing Proposition \\ref{prop:scaling}, there exists $\\delta_{\\mathrm{fv},j}^*>0$ such that the free-vibration constraint is satisfied if and only if $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ and $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$.\n    Finally, there exists $\\delta^* = \\max\\{\\max_{j{\\in \\mathcal{L}_\\mathrm{s}}} \\delta^*_{\\mathrm{c},j}, \\max_{j{\\in \\mathcal{L}_\\mathrm{fv}}} \\delta^*_{\\mathrm{fv},j} \\}>0$ such that the free-vibration and compliance constraints are satisfied if and only if the {right-hand-side} conditions of the proposition are satisfied. This is (i) because we have upper bounds for the compliance $\\overline{c}_j$ and compliance is a nonincreasing function in $\\delta$ \\cite[Proposition 1]{Tyburec2023}, and (ii) because we have lower bounds for the lowest free-vibration eigenvalues $\\underline{\\lambda}_j$ and the lowest free-vibration eigenvalue function is a nondecreasing function in $\\delta$ (Proposition \\ref{prop:monotonic}).\n\\end{proof}","start":61514,"end":63129},{"raw_tex":"\\begin{proposition}\\label{prop:inertia}\n\tLet $\\mathbf{y}^{*(r)}$ be the optimal moments associated with the cross-section areas $\\mathbf{a}$ obtained by solving the $r$-th degree relaxation. If $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)}) > 0$ or $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)}) > 0$, then $a_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}^{*(r)}) > 0$.\n\t%\n\t\\begin{proof}\n        By contradiction, let us assume that $a_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}^{*(r)}) = 0$, i.e., $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$. Then, we need to show that $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)}) = 0$ and $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)}) = 0$.\n\t\t\n\t\tConsider first the case of $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)})$. The moment matrix {(recall the definition in \\eqref{eq:moment})} then contains the principal submatrix\n\t\t%\n\t\t\\begin{equation}\n\t\t\t\\begin{pmatrix}\n\t\t\t\t1 & y_{a_{\\mathrm{s},e}^1}^{*(r)}\\\\\n\t\t\t\ty_{a_{\\mathrm{s},e}^1}^{*(r)} & y_{a_{\\mathrm{s},e}^2}^{*(r)}\n\t\t\t\\end{pmatrix} \\succeq 0.\n\t\t\\end{equation}\n\t\t%\n\t\tThe determinant of this matrix must be nonnegative, so that we obtain the inequality\n\t\t%\n\t\t\\begin{equation}\n\t\t\ty_{a_{\\mathrm{s},e}^2}^{*(r)} - \\left[y_{a_{\\mathrm{s},e}^1}^{*(r)}\\right]^2 \\ge 0.\n\t\t\\end{equation}\n\t\t%\n\t\tAfter inserting $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$, we obtain $y_{a_{\\mathrm{s},e}^2}^{*(r)} \\ge 1$. Due to the bound constraints \\eqref{eq:opt_nonneg_po}, $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$. Using $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$ with $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$, \\eqref{eq:rel_a_2} is then evaluated as $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)})=0$.\n\t\t\n\t\tFor the case of, $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)})$, the moment matrix contains the principal submatrix\n\t\t%\n\t\t\\begin{equation}\n\t\t\t\\begin{pmatrix}\n\t\t\t\t1 & y_{a_{\\mathrm{s},e}^1}^{*(r)} & y_{a_{\\mathrm{s},e}^2}^{*(r)}\\\\\n\t\t\t\ty_{a_{\\mathrm{s},e}^1}^{*(r)} & y_{a_{\\mathrm{s},e}^2}^{*(r)} & y_{a_{\\mathrm{s},e}^3}^{*(r)}\\\\\n\t\t\t\ty_{a_{\\mathrm{s},e}^2}^{*(r)} & y_{a_{\\mathrm{s},e}^3}^{*(r)} & y_{a_{\\mathrm{s},e}^4}^{*(r)}\n\t\t\t\\end{pmatrix} \\succeq 0.\n\t\t\\end{equation}\n\t\t%\n\t\tAfter inserting $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$ and $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$, the determinant evaluates as\n\t\t%\n\t\t\\begin{equation}\n\t\t\t- \\left(y_{a_{\\mathrm{s},e}^3}^{*(r)} + 1\\right)^2 \\ge 0,\n\t\t\\end{equation}\n\t\t%\n\t\tso that $y_{a_{\\mathrm{s},e}^3}^{*(r)}=-1$ is the only feasible solution. Inserting $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$, $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$, and $y_{a_{\\mathrm{s},e}^3}^{*(r)} = -1$ into \\eqref{eq:rel_a_3}, we again receive $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)})=0$, which finishes the proof.\n\t\\end{proof}\n\\end{proposition}","start":73554,"end":76356},{"raw_tex":"\\begin{proposition}\\label{th:ub}\n\tLet $\\mathbf{y}^{*(r)}$ be the optimal moments obtained by solving the $r$-th degree relaxation and set $\\tilde{\\mathbf{a}} := \\mathbf{a}_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}^{*(r)})$. Then, there exists $\\delta$ such that\n\t%\n\t\\begin{equation}\\label{eq:scalar_fvineq}\n\t\t\\mathbf{K}_j(\\delta\\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\delta\\tilde{\\mathbf{a}}) \\succeq 0\\,.\n\t\\end{equation}\n\\end{proposition}","start":76589,"end":77051},{"raw_tex":"\\begin{proof}\n\tFrom the hierarchy construction {\\eqref{eq:general_moment}}, the {localizing matrix \\eqref{eq:localizing} will always contain $L_\\mathbf{y}(1 \\otimes \\left[\\mathbf{K}_j (\\mathbf{a}) - \\underline{\\lambda} \\mathbf{M}_j(\\mathbf{a})\\right])$ as principal submatrix due to} \\eqref{eq:opt_fv}{, which} can be written in terms of the lifted variables as\n\t%\n\t\\begin{multline}\\label{eq:fvrelaxed}\n\t\t\\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)})\\right]^{\\circ\\frac{1}{2}}\\right) + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)})\\right]^{\\circ\\frac{1}{3}}\\right) \\\\\n\t\t- \\underline{\\lambda}_j\\left( \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) \\right)\\succeq 0,\n\t\\end{multline}\n\t%\n\twith $\\bullet^{\\circ q}$ raising each element in $\\bullet$ to the power of $q$. We note here that the power factors in \\eqref{eq:fvrelaxed} eliminate the powers introduced in the notation \\eqref{eq:hatnotation}, so that \\eqref{eq:fvrelaxed} is indeed linear in $\\mathbf{a}_\\mathrm{rel}^{(i)}$ for all $i$.\n \n    After projecting via an orthonormal basis of $\\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$ denoted with $\\mathbf{W}_j$, we receive\n\t%\n\t\\begin{equation}\\label{eq:rel_cond1}\n\t\t\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) \\right]\\mathbf{W}_j \\succeq \\underline{\\lambda}_j \\mathbf{W}_j^\\mathrm{T} \\hat{\\mathbf{M}}_{j}^{\\langle 0\\rangle}\\mathbf{W}_j \\succeq 0.\n\t\\end{equation}\n\t%\n\tFurther, because of Proposition \\ref{prop:inertia}, {$a_{\\mathrm{rel},e}^{\\langle 1 \\rangle} \\mathbf{K}_{j,e}^{\\langle 1\\rangle}\\neq \\mathbf{0}$} whenever {$a_{\\mathrm{rel},e}^{\\langle2\\rangle} \\mathbf{K}_{j,e}^{\\langle 2\\rangle} + a_{\\mathrm{rel},e}^{\\langle3 \\rangle} \\mathbf{K}_{j,e}^{\\langle 3\\rangle} \\neq \\mathbf{0}$}. Consequently, \n\t\\begin{multline}\\label{eq:spans}\n\t\t\\mathrm{Span}\\left(\\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} (\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 2\\rangle}\\right]^{\\circ\\frac{1}{2}}\\right) + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 3\\rangle}\\right]^{\\circ\\frac{1}{3}}\\right)\\right)\\\\ = \\mathrm{Span}\\left(\\hat{\\mathbf{K}}_j^{\\langle 1 \\rangle} (\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}(\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{{\\langle 3\\rangle}}(\\tilde{\\mathbf{a}}) \\right).\n\t\\end{multline}\n\t%\n\tThus, because \\eqref{eq:fvrelaxed} is feasible due to $\\mathbf{y}^{*(r)}$ being optimal, it holds based on Proposition \\ref{lemma:dynamic_admissibility} and based on \\eqref{eq:spans} that \n\t%\n\t\\begin{equation}\\label{eq:rel_cond2}\n\t\t\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right).\n\t\\end{equation}\n\t%\n\tUsing \\eqref{eq:rel_cond1} and \\eqref{eq:rel_cond2}, Proposition \\ref{prop:scaling} certifies existence of $\\delta$ such that the matrix inequality \\eqref{eq:scalar_fvineq} holds.\n\\end{proof}","start":77052,"end":80485},{"raw_tex":"\\begin{proof}\n\tFor free-vibration constraints, the proof follows from Proposition \\ref{th:ub}. For static constraints, based on Proposition \\ref{prop:inertia}, equation~\\eqref{eq:spans} holds. Thus, since\n\t%\n\t\\begin{equation}\n\t\t\\begin{pmatrix}\n\t\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t\t-\\mathbf{f}_j & \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 2\\rangle}\\right]^{\\circ\\frac{1}{2}}\\right) + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 3\\rangle}\\right]^{\\circ\\frac{1}{3}}\\right)\n\t\t\\end{pmatrix} \\succeq 0\n\t\\end{equation}\n\t%\n\tis satisfied by optimality of $\\mathbf{y}^{*(r)}$, the condition $\\mathbf{f}_j \\in \\mathbf{K}_j(\\mathbf{\\tilde{a}})$ holds true due to Proposition \\ref{lemma:statical_admissibility}. Based on Proposition \\ref{th:feasibility_cond}, the upper bound solution can thus be constructed.\n\t\n\tConsequently, there exists $\\delta$ such that both types of constraints are satisfied simultaneously.\n\\end{proof}","start":81159,"end":82228}]},"nodes":[{"id":"S001","semantic_type":"notation","kind":"notation","labels":["strandard_basis"],"declared_external":false,"source_only":false,"title":"标准单项式基","source_units":[{"raw_tex":"Consider a polynomial $f(\\mathbf{x})$ of a degree $d$, which can be written as $f(\\mathbf{x}) = \\mathbf{p}^\\mathrm{T}\\mathbf{b}_d(\\mathbf{x})$, where\n%\n\\begin{equation}\n\\begin{aligned}\n\\mathbf{b}_d(\\mathbf{x})=&\\Big( 1 \\quad x_1 \\quad x_2 \\quad \\cdots \\quad x_1^2 \\quad x_1x_2 \\quad  \\cdots  \\quad x_1x_n \\\\ &  x_2^2 \\quad x_2x_3 \\quad  \\cdots \\quad x_n^2 \\quad \\cdots \\quad x_1^d \\quad \\cdots \\quad x_n^d \\Big)^\\mathrm{T}\n\\end{aligned}\n\\label{strandard_basis}\n\\end{equation}\n%\ndenotes the canonical monomial basis of degree up to $d$ and $\\mathbf{p} \\in \\mathbb{R}^{|\\mathbf{b}_d(\\mathbf{x})|}$ is the corresponding coefficient vector. To certify polynomial nonnegativity, we rely on the Sum-Of-Squares (SOS) representation\n","start":18904,"end":19629}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"矩变量与里斯泛函","source_units":[{"raw_tex":"{To construct these truncations, we introduce moment variables $\\mathbf{y} \\in \\mathbb{R}^{|\\mathbf{b}_{2r}(\\mathbf{x})|}$ indexed by the monomial basis $\\mathbf{b}_{2r}(\\mathbf{x})$. Further, let us define} the Riesz functional $L_\\mathbf{y} (\\mathbf{Z}(\\mathbf{x}))$ that linearizes {any} polynomial matrix $\\mathbf{Z}(\\mathbf{x})$ using $\\mathbf{y}$. Specifically, the operator is applied entry-wise and for the $(i,j)$-th component of $\\mathbf{Z}$ we write\n%\n\\begin{equation}\n    Z_{i,j}(\\mathbf{x}) = \\mathbf{p}_{{{i,j}}}^\\mathrm{T} \\mathbf{b}_{2r}(\\mathbf{x})\\quad \\Longleftrightarrow \\quad L_\\mathbf{y}(Z_{i,j}) = \\mathbf{p}_{{{i,j}}}^\\mathrm{T} \\mathbf{y}\\; \\text{ with }\\mathbf{y}\\text{ indexed in }\\mathbf{b}_{2r}(\\mathbf{x})\n\\end{equation}\n%\nwith a coefficient vector $\\mathbf{p}_{{{i,j}}}$. In this manuscript, we further adopt the notation $y_{x_i^d}$ to express the component of $\\mathbf{y}$ belonging to $x_i^d \\in \\mathbf{b}_{2r}(\\mathbf{x})$.\n\n","start":21814,"end":22775}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":["eq:localizing","eq:moment"],"declared_external":false,"source_only":false,"title":"矩与局部化矩阵","source_units":[{"raw_tex":"{\nUnder Assumption \\ref{as:archimedean} (Archimedean assumption), polynomial optimization can be solved via the moment-sum-of-squares hierarchy:\n%\n\\begin{subequations}\\label{eq:general_moment}\n\\begin{align}\n    \\min_{\\mathbf{y}} \\; & L_\\mathbf{y}(f(\\mathbf{x}))\\\\\n    \\mathrm{s.t.}\\; & L_{\\mathbf{y}}\\bigl(\\mathbf{b}_{r-d_\\mathrm{G}}(\\mathbf{x}) \\mathbf{b}_{r-d_\\mathrm{G}}(\\mathbf{x})^\\mathrm{T} \\otimes \\mathbf{G}(\\mathbf{x})\\bigr) \\succeq 0, & \\text{(localizing matrix)}\\label{eq:localizing}\\\\\n    & L_{\\mathbf{y}}\\bigl(\\mathbf{b}_r(\\mathbf{x}) \\mathbf{b}_r(\\mathbf{x})^\\mathrm{T}\\bigr) \\succeq 0, & \\text{(moment matrix)}\\label{eq:moment}\n\\end{align}\n\\end{subequations}\n%\nwhere $d_\\mathrm{G}= \\lceil \\mathrm{deg}(\\mathbf{G}(\\mathbf{x}))/2 \\rceil$ (with $\\lceil \\bullet \\rceil$ denoting the ceiling function), and $\\otimes$ denotes the Kronecker product. The constraints define moment matrices $\\mathbf{M}_r(\\mathbf{y})$ and localizing matrices $\\mathbf{M}_{r-d_\\mathrm{G}} (\\mathbf{G}\\mathbf{y})$ that correspond to the SOS conditions in the dual formulation.}\n\n","start":22775,"end":23841}]},{"id":"S004","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"松弛阶数与收敛背景","source_units":[{"raw_tex":"{Although \\eqref{eq:general_moment} is convex in the moments $\\mathbf{y}$, it is computationally intractable for $r\\rightarrow \\infty$. Instead, we solve a hierarchy of finite-dimensional truncations, starting with relaxation order $r = \\max\\left\\{d_\\mathrm{G}; \\left\\lceil\\mathrm{deg}(f(\\mathbf{x}))/2\\right\\rceil\\right\\}$ and increasing until convergence occurs. Since constraints of relaxation $r$ are included in relaxation $r+1$, we obtain a monotonic sequence of lower bounds: $f^{(r)} \\le f^{(r+1)} \\le f^* = f^{(\\infty)}$. Remarkably, the hierarchy also exhibits generic finite convergence \\cite{Nie2013}, though the convergence degree is not known \\textit{a priori}.}\n","start":23841,"end":24518}]},{"id":"S005","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:opt_orig_compliance","eq:opt_orig_fv","eq:opt_orig_nonneg","eq:opt_orig_objective","eq:opt_orig_static"],"declared_external":false,"source_only":false,"title":"多工况最小重量问题","source_units":[{"raw_tex":"{Let $\\mathcal{L}_\\mathrm{s}$ and $\\mathcal{L}_\\mathrm{fv}$ denote the sets of indices corresponding to static and free-vibration load cases, respectively.} We investigate the global solution to topology optimization of least-weight frame structures \\eqref{eq:opt_orig_objective}, while accounting for upper bounds $\\overline{c}_j \\in \\mathbb{R}_{>0}$, $j \\in {\\mathcal{L}_\\mathrm{s}}$, on structural compliances \\eqref{eq:opt_orig_compliance} of {$\\lvert \\mathcal{L}_\\mathrm{s} \\rvert$ static} load cases and lower bounds $\\underline{\\lambda}_j \\in \\mathbb{R}_{>0}${, $j \\in \\mathcal{L}_\\mathrm{fv}$,} on the lowest nonzero free-vibration eigenvalues {of the $\\lvert \\mathcal{L}_\\mathrm{fv} \\rvert$ free-vibration load cases} expressed using the Rayleigh quotient \\eqref{eq:opt_orig_fv}. Such optimization problems are naturally formalized in terms of cross-section areas $\\mathbf{a}$, displacements $\\mathcal{U} = {\\{\\mathbf{u}_j : j \\in \\mathcal{L}_\\mathrm{s}\\}}$ with $\\mathbf{u}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}$, and normalized eigenmodes $\\mathcal{V} = {\\{\\mathbf{v}_j : j \\in \\mathcal{L}_\\mathrm{fv}\\}}$ with $\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}$ as\n%\n\\begin{subequations}\\label{eq:opt_orig}\n\t\\begin{align}\n\t\t\\min_{\\mathbf{a}\\in \\mathbb{R}^{n_{\\mathrm{e}}}\\!,\\;\n        \\mathcal{U},\\, \n        \\mathcal{V}} \n        & \\sum_{e=1}^{n_\\mathrm{e} } \\rho_e \\ell_e a_e\\label{eq:opt_orig_objective}\\\\\n\t\t\\text{s.t.}\\; & \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}} \\setminus \\mathrm{Ker}\\left( \\mathbf{M}(\\mathbf{a}) \\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\mathbf{K}_j(\\mathbf{a}) \\mathbf{v}_j}{\\mathbf{v}_j^\\mathrm{T} \\mathbf{M}_j(\\mathbf{a}) \\mathbf{v}_j} \\ge \\underline{\\lambda}_j,\\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\label{eq:opt_orig_fv}\\\\\n\t\t& \\mathbf{K}_j(\\mathbf{a}) \\mathbf{u}_j = \\mathbf{f}_j,\\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\label{eq:opt_orig_static}\\\\\n\t\t& \\mathbf{f}_j^\\mathrm{T} \\mathbf{u}_j \\le \\overline{c}_j,\\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\label{eq:opt_orig_compliance}\\\\\n\t\t& \\mathbf{a} \\ge \\mathbf{0},\\label{eq:opt_orig_nonneg}\n\t\\end{align}\n\\end{subequations}\n%\nwhere $\\mathbf{K}_j(\\mathbf{a}) \\in \\mathbb{S}^{n_{\\mathrm{dof},j}}_{\\succeq 0}$ and $\\mathbf{M}_j(\\mathbf{a}) \\in \\mathbb{S}^{n_\\mathrm{dof},j}_{\\succeq 0}$ stand for the stiffness and mass matrices, $n_{\\mathrm{dof},j}$ is the number of degrees of freedom in the $j$-th load case, and $\\mathbf{f}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}$ stands for the generalized force vector of the $j$-th load case, which includes the nodal forces and moments. In addition, $\\bm{\\ell} \\in \\mathbb{R}^{n_\\mathrm{e}}_{>0}$ and $\\bm{\\rho} \\in \\mathbb{R}^{n_\\mathrm{e}}_{>0}$ are vectors of lengths and densities of the $n_\\mathrm{e}$ elements. {Because the stiffness and mass matrices may vary across the load cases, the setting of \\eqref{eq:opt_orig} covers varying kinematic boundary conditions.}     {Notice that \\eqref{eq:opt_orig} is a non-smooth problem, as the free-vibration eigenvalue constraints \\eqref{eq:opt_orig_fv} are nondifferentiable for repeated eigenvalues.}\n\n","start":26611,"end":29690}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","labels":["eq:stiffness"],"declared_external":false,"source_only":false,"title":"半正定多项式刚度","source_units":[{"raw_tex":"In bending-resistant structures such as frames, the stiffness matrix is usually a {low-degree} polynomial function of the design variables {(see Appendix \\ref{app:polydep} for more details)}, and follows from the assembly\n%\n\\begin{equation}\\label{eq:stiffness}\n\t\\mathbf{K}_j (\\mathbf{a}) = \\sum_{e=1}^{n_\\mathrm{e}} \\left[a_e \\mathbf{K}_{j,e}^{\\langle 1\\rangle} + a_e^2 \\mathbf{K}_{j,e}^{\\langle 2\\rangle} + a_e^3 \\mathbf{K}_{j,e}^{\\langle 3\\rangle}\\right],\n\\end{equation}\n%\nwhere $\\mathbf{K}_{j,e}^{\\langle i\\rangle} \\succeq 0$, $e \\in \\{1, \\dots, n_\\mathrm{e}\\}$, $i \\in \\{1,2,3\\}$, are auxiliary element stiffness matrices. In \\eqref{eq:stiffness}, the terms $ a_e \\mathbf{K}_{j,e}^{\\langle 1\\rangle}$ {typically} cover the membrane behavior, whereas the higher-order terms $a_e^2 \\mathbf{K}_{j,e}^{\\langle 2\\rangle}$ and $a_e^3 \\mathbf{K}_{j,e}^{\\langle 3\\rangle}$ capture the bending effects.\n\n","start":29690,"end":30589}]},{"id":"S007","semantic_type":"assumption","kind":"assumption","labels":["eq:mass"],"declared_external":false,"source_only":false,"title":"仿射质量矩阵","source_units":[{"raw_tex":"For the mass matrix $\\mathbf{M}_j(\\mathbf{a})$, we assume the form {(again, see Appendix \\ref{app:polydep})}\n%\n\\begin{equation}\\label{eq:mass}\n\t\\mathbf{M}_j (\\mathbf{a}) = \\mathbf{M}_j^{\\langle 0\\rangle} + \\sum_{e=1}^{n_\\mathrm{e}} a_e\\mathbf{M}_{j,e}^{\\langle 1\\rangle},\n\\end{equation}\n%\nin which $a_e \\mathbf{M}_{j,e}^{\\langle 1\\rangle} \\succeq 0$, $e \\in \\{1, \\dots, n_\\mathrm{e}\\}$, are the element mass matrices and $\\mathbf{M}_j^{\\langle 0\\rangle} \\succeq 0$ constitutes a non-structural mass. In this paper, we adopt stiffness-consistent mass matrices, but any other form can be used assuming it maintains a linear dependence on the cross-section areas. An extension to nonlinear terms in $\\mathbf{M}_j(\\mathbf{a})$ might be possible but is not covered in this manuscript.\n","start":30589,"end":31369}]},{"id":"S008","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:opt_fv","eq:opt_nonneg","eq:opt_objective","eq:opt_static"],"declared_external":false,"source_only":false,"title":"设计变量半定模型","source_units":[{"raw_tex":"One of the difficulties of the optimization problem \\eqref{eq:opt_orig} lies in the fact that \\eqref{eq:opt_orig_fv} and \\eqref{eq:opt_orig_static} include products of state variables $\\mathbf{u}_j$ and $\\mathbf{v}_j$ with design variables $\\mathbf{a}$. In \\cite[Proposition 2.3]{Achtziger2008}, it has been shown that the state variables $\\mathbf{u}_j$ and $\\mathbf{v}_j$ can be eliminated from the formulation, leading to an equivalent semidefinite programming reformulation in the design variables only:\n%\n\\begin{subequations}\\label{eq:opt}\n\\begin{align}\n\t\\min_{\\mathbf{a}\\in \\mathbb{R}^{n_{\\mathrm{e}}}}\\; & \\sum_{e=1}^{n_\\mathrm{e} } \\rho_e \\ell_e a_e\\label{eq:opt_objective}\\\\\n\t\\text{s.t.}\\; & \\mathbf{K}_j(\\mathbf{a}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\mathbf{a}) \\succeq 0,\\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\label{eq:opt_fv}\\\\\n\t& \\begin{pmatrix}\n\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t-\\mathbf{f}_j & \\mathbf{K}_j(\\mathbf{a})\n\t\\end{pmatrix} \\succeq 0,\\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\label{eq:opt_static}\\\\\n\t& \\mathbf{a} \\ge \\mathbf{0},\\label{eq:opt_nonneg}\n\\end{align}\n\\end{subequations}\n%\nwhere the constraints \\eqref{eq:opt_fv} and \\eqref{eq:opt_static} are matrix inequalities. These inequalities are generally nonconvex due to their polynomial dependence on $\\mathbf{a}$, and thus obtaining a (global) solution to \\eqref{eq:opt} remains challenging. {However, the non-differentiability of the free-vibration eigenvalue constraints is avoided, as the Rayleigh quotient \\eqref{eq:opt_orig_fv} is replaced by a polynomial matrix inequality \\eqref{eq:opt_fv}.}\n\nFrom the formulation \\eqref{eq:opt}, we can state the following basic properties of the feasible designs $\\hat{\\mathbf{a}}$:\n","start":31370,"end":33085}]},{"id":"S009","semantic_type":"proposition","kind":"proposition","labels":["lemma:statical_admissibility"],"declared_external":true,"declaration_text":["原始前提清单明确许可 B0018；作者引用 Tyburec2021, Proposition 1。"],"source_only":false,"title":"静力可容许性","source_units":[{"raw_tex":"\\begin{proposition}[Static admissibility {\\cite[Proposition 1]{Tyburec2021}}]\\label{lemma:statical_admissibility}\n\tLet $$\\mathbf{\\hat{a}} \\in \\left\\{\\mathbf{a} \\; \\vert \\; \\begin{pmatrix}\n\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t-\\mathbf{f}_j & \\mathbf{K}_j(\\mathbf{a})\n\t\\end{pmatrix} \\succeq 0, \\mathbf{a}\\ge \\mathbf{0}\\right\\}$$ hold true. Then, $\\mathbf{f}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j(\\hat{\\mathbf{a}})\\right)$.\n\\end{proposition}\n","start":33087,"end":33535}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"固定比例整体缩放","source_units":[{"raw_tex":"In this section, we introduce a bilevel programming reformulation of the problem \\eqref{eq:opt_orig} and provide conditions for its solvability. This reformulation will further enable us to construct feasible solutions to \\eqref{eq:opt_orig}.\n\nSetting $\\mathbf{a}(\\delta) := \\delta \\tilde{\\mathbf{a}}$ with $\\tilde{\\mathbf{a}}\\ge \\mathbf{0}$, the optimization problem \\eqref{eq:opt} becomes equivalent to a bilevel optimization problem, with the upper level seeking \\textit{admissible} ratios of the cross-section areas $\\tilde{\\mathbf{a}}$ and the lower level searching for the optimal scaling factor $\\delta$ of the fixed cross-section ratios:\n","start":37715,"end":38361}]},{"id":"S011","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:bilevel-atilde","eq:bilevel-lower","eq:bilevel-membraine","eq:bilevel_mass","eq:bilevel_static"],"declared_external":false,"source_only":false,"title":"双层模型及下层约束","source_units":[{"raw_tex":"%\n\\begin{subequations}\\label{eq:bilevel}\n\t\\begin{align}\n\t\t\\min_{\\tilde{\\mathbf{a}}\\in \\mathbb{R}^{n_{\\mathrm{e}}}\\!,\\; \\delta^*\\in\\mathbb{R}}\\; & \\delta^* \\sum_{e=1}^{n_\\mathrm{e}} \\rho_e \\ell_e \\tilde{a}_e\\\\\n\t\t\\mathrm{s.t.}\\ & \\tilde{\\mathbf{a}} \\ge \\mathbf{0},\\label{eq:bilevel-atilde}\\\\\n\t\t& \\mathbf{f}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right), \\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\label{eq:bilevel_static}\\\\\n\t\t& \\mathrm{Im}\\left(\\mathbf{M}_{j}(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_{j}(\\tilde{\\mathbf{a}})\\right),\\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\label{eq:bilevel_mass}\\\\\n\t\t& \\mathbf{W}_j^\\mathrm{T}\\left[ \\sum_{e=1}^{n_\\mathrm{e}}\\tilde{a}_e\\mathbf{K}_{j,e}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\sum_{e=1}^{n_\\mathrm{e}} \\tilde{a}_e \\mathbf{M}_{j,e}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0, \\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\label{eq:bilevel-membraine}\\\\\n\t\t& \\begin{aligned}\\label{eq:bilevel-lower}\n\t\t\t\\delta^* \\in \\argmin_{\\delta\\in\\mathbb{R}}\\; & \\delta\\\\\n\t\t\t\\mathrm{s.t.}\\; & \\begin{pmatrix}\n\t\t\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t\t\t-\\mathbf{f}_j & \\mathbf{K}_j (\\delta \\tilde{\\mathbf{a}})\n\t\t\t\\end{pmatrix} \\succeq 0, \\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\\\\n\t\t\t& \\mathbf{K}_j(\\delta \\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\delta \\tilde{\\mathbf{a}}) \\succeq 0, \\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\\\\n\t\t\t& \\delta \\ge 0,\n\t\t\\end{aligned}\n\t\\end{align}\n\\end{subequations}\n%\nwhere $\\mathbf{W}_j$ is an orthonormal basis of $\\mathrm{Ker}\\left(\\sum_{e=1}^{n_\\mathrm{e}}\\left[\\mathbf{K}_{j,e}^{\\langle 2\\rangle} + \\mathbf{K}_{j,e}^{\\langle 3\\rangle}\\right]\\right)$ and the constraints \\eqref{eq:bilevel-atilde}-\\eqref{eq:bilevel-membraine} ensure that the lower-level problem \\eqref{eq:bilevel-lower} {has a non-empty interior}. In particular, these {constraints} include nonnegative components of the cross-sectional ratios \\eqref{eq:bilevel-atilde}, static \\eqref{eq:bilevel_static} and mass \\eqref{eq:bilevel_mass} admissibility of the ratios, and the pure membrane eigenvalues to be at least $\\underline{\\lambda}_j$ \\eqref{eq:bilevel-membraine}, which we will prove in Proposition \\ref{th:feasibility_cond} (that uses the notation set up in \\eqref{eq:hatnotation}). Moreover, the lower-level problem \\eqref{eq:bilevel-lower} is quasi-convex in $\\delta$, allowing one to obtain a globally optimal scaling $\\delta^*$ for fixed ratios $\\tilde{\\mathbf{a}}$ using bisection. Proving the quasi-convexity property and showing that the conditions \\eqref{eq:bilevel-atilde}--\\eqref{eq:bilevel-membraine} are sufficient and necessary for solvability of the lower-level problem \\eqref{eq:bilevel-lower} constitutes the main goal of this section.\n\n","start":38361,"end":41099}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["eq:RQ"],"declared_external":false,"source_only":false,"title":"缩放特征值的瑞利商","source_units":[{"raw_tex":"Let us first focus on the lower-level problem \\eqref{eq:bilevel-lower}. We start by expressing the lowest nonzero\n%non-singular \nfree-vibration eigenvalue function $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ using the Raleigh quotient \\cite[Definition 2.2 and Appendix A.1]{Achtziger2008} as\n%\n\\begin{equation}\\label{eq:RQ}\n\t\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) = \\min_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\left[\\delta\\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} + \\delta^2\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\delta^3\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\right]  \\mathbf{v}_j}{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\delta\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j},\n\\end{equation}\n%\n","start":41099,"end":42016}]},{"id":"S013","semantic_type":"notation","kind":"notation","labels":["eq:hatnotation"],"declared_external":false,"source_only":false,"title":"按次数聚合的矩阵","source_units":[{"raw_tex":"in which we adopt the simplified notation\n%\n\\begin{equation}\\label{eq:hatnotation}\n\t\\hat{\\mathbf{K}}_j^{\\langle i\\rangle}(\\tilde{\\mathbf{a}}) = \\sum_{e=1}^{n_\\mathrm{e}}\\tilde{a}_e^i\\mathbf{K}_{j,e}^{\\langle i\\rangle}, \\qquad \n\t\\hat{\\mathbf{M}}_j^{\\langle i\\rangle}(\\tilde{\\mathbf{a}}) = \\sum_{e=1}^{n_\\mathrm{e}}\\tilde{a}_e^i \\mathbf{M}_{j,e}^{\\langle i\\rangle}.\n\\end{equation}\n%\nand omit the argument of $\\hat{\\mathbf{K}}_j^{\\langle i\\rangle}$ and $\\hat{\\mathbf{M}}_j^{\\langle i\\rangle}$ \nwhen it is clear from the context, to shorten the equations.\n\n","start":42016,"end":42569}]},{"id":"S014","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"质量可容许下的正性","source_units":[{"raw_tex":"Because $\\forall j \\in \\mathcal{L}_{\\mathrm{fv}}: \\mathrm{Im}\\left(\\mathbf{M}_{j}(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_{j}(\\tilde{\\mathbf{a}})\\right)$ (see Proposition~\\ref{prop:exists_gamma_fv}), \\eqref{eq:RQ} renders that $\\lambda_{\\min,j}>0$. Thus, to prove feasibility of the eigenvalue constraint in \\eqref{eq:bilevel-lower} with respect to $\\delta$, it remains to find the condition under which $\\sup_{\\delta} \\lambda_{\\min,j} \\ge \\underline{\\lambda}_j$. \n\n","start":42569,"end":43060}]},{"id":"S015","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:RQs"],"declared_external":false,"source_only":false,"title":"除去缩放因子的商","source_units":[{"raw_tex":"Because of $\\delta > 0$, equation~\\eqref{eq:RQ} can further be rewritten to\n%\n\\begin{equation}\\label{eq:RQs}\n\t\\begin{multlined}\n\t\t\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) =\\\\\n\t\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}.\n\t\\end{multlined}\n\\end{equation}\n","start":43060,"end":43857}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"弯曲与膜方向分解","source_units":[{"raw_tex":"With this result, we have shown that by increasing $\\delta$ we do not decrease $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$. {Notice that this holds up to $\\delta\\rightarrow \\infty$, as we can choose $\\Delta\\rightarrow \\infty$ in the above proof.} For the solvability of the lower-level problem \\eqref{eq:bilevel-lower}, it is crucial to guarantee that there exits a $\\delta$ such that $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) \\ge \\underline{\\lambda}_j$. In other words, we require that $\\sup_{\\delta} \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) \\ge \\underline{\\lambda}_j$. To show this, let $\\mathbf{v}_j := \\mathbf{t}_j + \\mathbf{w}_j$, where $\\mathbf{t}_j \\in \\mathrm{Im}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) + \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right) $ and $\\mathbf{w}_j \\in \\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$.\nThen, we can write the expression for the supremum.\n","start":51189,"end":52352}]},{"id":"S017","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"可行重量给出设计上界","source_units":[{"raw_tex":"{Let us start with the first modification.} Since the objective function \\eqref{eq:opt_objective}, representing the weight of the structure, is a conic combination of nonnegative variables, any feasible point to the optimization problem \\eqref{eq:opt} provides an upper-bound on  the objective function value $\\overline{w}$ that also bounds the cross-section areas from above. This upper bound can be obtained by solving the convex problem \\eqref{eq:linprob}, providing the cross-sectional ratios $\\tilde{\\mathbf{a}}$, and{, subsequently,} by finding the minimal scaling factor $\\delta^*$ by solving the lower-level problem \\eqref{eq:bilevel-lower} globally using bisection. Consequently, the optimal cross-section areas are bounded as\n%\n\\begin{equation}\n\t0 \\le a_e \\le \\frac{\\overline{w}}{\\rho_e\\ell_e}, \\forall e \\in \\{1,\\dots,n_\\mathrm{e}\\}.\n\\end{equation}\n%\n","start":66732,"end":67594}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"区间缩放与重量限制","source_units":[{"raw_tex":"Furthermore, using the substitution $a_e = \\frac{\\overline{w}}{2\\rho_e\\ell_e} (a_{\\mathrm{s},e} + 1)$, these bounds can be written as $a_{\\mathrm{s},e}^2 \\le 1$. %Similarly, we add a weight constraint to strengthen the relaxations as\n","start":67594,"end":67828},{"raw_tex":"Similarly, we add a weight constraint $\\sum_{e=1}^{n_\\mathrm{e}} \\rho_e \\ell_e a_e \\le \\overline{w}$ to strengthen the relaxations.\nThe influence of this constraint on the feasible set of relaxations will be illustrated in Section~\\ref{sec:illustrative}. Using the above substitution of $a_e$, we obtain the formulation\n","start":68026,"end":68346}]},{"id":"S019","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"停用的重量公式草稿","source_units":[{"raw_tex":"%\n%\\begin{equation}\n%    \\sum_{e=1}^{n_\\mathrm{e}} \\rho_e \\ell_e a_e \\le \\overline{w}\\quad\\longleftrightarrow\\quad 2-n_\\mathrm{e} - \\sum_{e=1}^{n_\\mathrm{e}} a_{\\mathrm{s},e} \\ge 0.\n%\\end{equation}\n","start":67828,"end":68026}]},{"id":"S020","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:opt_fv_po","eq:opt_nonneg_po","eq:opt_objective_po","eq:opt_static_po","eq:opt_weight_po"],"declared_external":false,"source_only":false,"title":"紧化后的多项式模型","source_units":[{"raw_tex":"%\n\\begin{subequations}\\label{eq:opt_po}\n\t\\begin{align}\n\t\t\\min_{\\mathbf{a}_{{\\mathrm{s}}}\\in\\mathbb{R}^{n_\\mathrm{e}}}\\; & {\\frac{1}{2}} \\overline{w} (n_\\mathrm{e} + \\mathbf{1}^\\mathrm{T} \\mathbf{a}_\\mathrm{s})\\label{eq:opt_objective_po}\\\\\n\t\t\\text{s.t.}\\; & \\mathbf{K}_j(\\mathbf{a}_\\mathrm{s}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\mathbf{a}_\\mathrm{s}) \\succeq 0, \\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\label{eq:opt_fv_po}\\\\\n\t\t& \\begin{pmatrix}\n\t\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t\t-\\mathbf{f}_j & \\mathbf{K}_j(\\mathbf{a}_\\mathrm{s})\n\t\t\\end{pmatrix} \\succeq 0,\\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\label{eq:opt_static_po}\\\\\n            & 2-n_\\mathrm{e} - \\sum_{e=1}^{n_\\mathrm{e}} a_{\\mathrm{s},e} \\ge 0,\\label{eq:opt_weight_po}\\\\\n\t\t& 1 - a_\\mathrm{s,e}^2 \\ge 0, \\quad e \\in \\{1,\\dots,n_\\mathrm{e}\\}\\label{eq:opt_nonneg_po}\n\t\\end{align}\n\\end{subequations}\n%\n{in the scaled variables $\\mathbf{a}_{\\mathrm{s}} \\in \\mathbb{R}^{n_\\mathrm{e}}$. Here,} we slightly abuse the notation and express the stiffness and mass matrices in terms of the scaled variables $\\mathbf{a}_\\mathrm{s}$ as $\\mathbf{K}_j(\\mathbf{a}_\\mathrm{s})$ and $\\mathbf{M}_j(\\mathbf{a}_\\mathrm{s})$, respectively. {Although this compactification means that \\eqref{eq:opt} and \\eqref{eq:opt_po} do not have the same feasible set (after rescaling), they maintain the same set of optimal solutions. Thus, solution to \\eqref{eq:opt_po} also solves \\eqref{eq:opt}.}\n","start":68346,"end":69781}]},{"id":"S021","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"一阶矩可行化的目标","source_units":[{"raw_tex":"\\subsubsection{Feasible upper bounds from first-order moments}\nIn what follows, we show that the first-order moments obtained by solving any relaxation of \\eqref{eq:opt_po} satisfy the {feasibility conditions} set in Proposition \\ref{th:feasibility_cond}, which allows us to construct a feasible upper-bound solution to \\eqref{eq:opt}.\n\nTo show this, let us introduce the notation\n","start":71353,"end":71734}]},{"id":"S022","semantic_type":"definition","kind":"definition","labels":["eq:rel_a_1","eq:rel_a_2","eq:rel_a_3"],"declared_external":false,"source_only":false,"title":"未缩放单项式的提升量","source_units":[{"raw_tex":"%\n\\begin{subequations}\n\t\\begin{align}\n\t\ta_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}) &= 0.5\\,{\\frac{\\overline{w}}{\\rho_e \\ell_e}} (y_{a_{\\mathrm{s},e}^1} + 1),\\label{eq:rel_a_1}\\\\\n\t\ta_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}) &= 0.25\\, {\\frac{\\overline{w}^2}{\\rho_e^2 \\ell_e^2}} \\left( y_{a_{\\mathrm{s},e}^2} + 2 y_{a_{\\mathrm{s},e}^1} + 1 \\right),\\label{eq:rel_a_2}\\\\\n\t\ta_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}) &= 0.125\\, {\\frac{\\overline{w}^3}{\\rho_e^3 \\ell_e^3}} \\left( y_{a_{\\mathrm{s},e}^3} + 3 y_{a_{\\mathrm{s},e}^2} + 3 y_{a_{\\mathrm{s},e}^1} + 1 \\right)\\label{eq:rel_a_3}\n\t\\end{align}\n\\end{subequations}\n%\nfor evaluating {expected values of the} monomials of the (unscaled) cross-section areas based on the (scaled) moments $\\mathbf{y}$. In this notation, the superscript $\\bullet^{\\langle i\\rangle}$ indicates that it corresponds to the $i$-th degree monomial of $\\bullet$. We wish to emphasize here that, generally, $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}) \\neq \\left[a_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y})\\right]^2$ and $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}) \\neq \\left[a_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y})\\right]^3$ due to the relaxation procedure.\n\n","start":71734,"end":72962}]},{"id":"S023","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"刚度支撑保持的策略","source_units":[{"raw_tex":"As the first step in the proof, we need to show that replacing higher-order moments with the powers of the first-order moments does not change the rank of the stiffness matrix. Since the stiffness matrix is a conic combination of positive semidefinite matrices with the coefficients being monomials of the cross-section areas, recall \\eqref{eq:stiffness}, it suffices to show that strict positivity of $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y})$ or $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y})$ implies strict positivity of $a_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y})$.\n","start":72962,"end":73553}]},{"id":"S024","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单声明的标准半正定矩阵事实；谱性质部分为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"半正定矩阵基本性质","source_units":[]},{"id":"S025","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单声明的半正定矩阵关于和、核与值域的标准事实。"],"source_only":false,"title":"半正定和的值域性质","source_units":[]},{"id":"S026","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单及第376行引用：Achtziger2008, Definition 2.2 and Appendix A.1。"],"source_only":false,"title":"广义瑞利商表征","source_units":[]},{"id":"S027","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单及第542行引用：Tyburec2023, Proposition 5。"],"source_only":false,"title":"柔度约束的有限缩放","source_units":[]},{"id":"S028","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单及第545行引用：Tyburec2023, Proposition 1。"],"source_only":false,"title":"柔度随缩放不增","source_units":[]},{"id":"S029","semantic_type":"proposition","kind":"proposition","labels":["lemma:dynamic_admissibility"],"declared_external":false,"source_only":false,"title":"质量可容许性","source_units":[{"raw_tex":"\\begin{proposition}[Mass admissibility]\\label{lemma:dynamic_admissibility}\n\tLet $$\\mathbf{\\hat{a}}\\in\\left\\{\\mathbf{a} \\; \\vert \\; \\mathbf{K}_j(\\mathbf{a}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\mathbf{a}) \\succeq 0, \\mathbf{a}\\ge \\mathbf{0}\\right\\}.$$ Then, $\\mathrm{Im}\\left(\\mathbf{M}_j(\\hat{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\hat{\\mathbf{a}})\\right)$.\n\\end{proposition}","start":33537,"end":33935}]},{"id":"S030","semantic_type":"proof","kind":"proof","labels":["eq:inimage"],"declared_external":false,"source_only":false,"title":"质量可容许性的作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\tThe matrix inequality \\eqref{eq:opt_fv} is equivalent to\n\t%\n\t\\begin{equation}\\label{eq:inimage}\n\t\t\\mathbf{v}_j^\\mathrm{T}\\mathbf{K}_j(\\mathbf{a})\\mathbf{v}_j - \\underline{\\lambda}_j \\mathbf{v}_j^\\mathrm{T}\\mathbf{M}_j(\\mathbf{a})\\mathbf{v}_j\\ge 0\\quad \\forall \\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}.\n\t\\end{equation}\n\t%\n\tLet us now proceed by contradiction and assume that $\\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right) \\nsubseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\mathbf{a})\\right)$. Then, there must exist $\\hat{\\mathbf{v}}_j\\in\\left\\{\\mathbf{v}_j\\;\\vert\\; \\mathbf{v}_j\\in \\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right), \\mathbf{v}_j\\notin \\mathrm{Im}\\left(\\mathbf{K}_j(\\mathbf{a})\\right)\\right\\}$, for which the left-hand side of \\eqref{eq:inimage} evaluates as\n\t%\n\t\\begin{equation}\n\t\t-\\underline{\\lambda}_j \\hat{\\mathbf{v}}_j^\\mathrm{T}\\mathbf{M}_j(\\mathbf{a})\\hat{\\mathbf{v}}_j < 0,\n\t\\end{equation}\n\t%\n\twhich is strictly negative because $\\underline{\\lambda}_j>0$, $\\mathbf{M}_j(\\mathbf{a}) \\succeq 0$, and $\\hat{\\mathbf{v}}_j\\in \\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right)$, contradicting~\\eqref{eq:inimage}.\n\\end{proof}","start":33936,"end":35093}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":["eq:inimage"],"declared_external":false,"source_only":false,"title":"改写振动二次型条件","source_units":[{"raw_tex":"\tThe matrix inequality \\eqref{eq:opt_fv} is equivalent to\n\t%\n\t\\begin{equation}\\label{eq:inimage}\n\t\t\\mathbf{v}_j^\\mathrm{T}\\mathbf{K}_j(\\mathbf{a})\\mathbf{v}_j - \\underline{\\lambda}_j \\mathbf{v}_j^\\mathrm{T}\\mathbf{M}_j(\\mathbf{a})\\mathbf{v}_j\\ge 0\\quad \\forall \\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}.\n\t\\end{equation}\n","start":33950,"end":34277}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"以像空间不包含反证","source_units":[{"raw_tex":"\tLet us now proceed by contradiction and assume that $\\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right) \\nsubseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\mathbf{a})\\right)$. Then, there must exist $\\hat{\\mathbf{v}}_j\\in\\left\\{\\mathbf{v}_j\\;\\vert\\; \\mathbf{v}_j\\in \\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right), \\mathbf{v}_j\\notin \\mathrm{Im}\\left(\\mathbf{K}_j(\\mathbf{a})\\right)\\right\\}$, for which the left-hand side of \\eqref{eq:inimage} evaluates as\n\t%\n\t\\begin{equation}\n\t\t-\\underline{\\lambda}_j \\hat{\\mathbf{v}}_j^\\mathrm{T}\\mathbf{M}_j(\\mathbf{a})\\hat{\\mathbf{v}}_j < 0,\n\t\\end{equation}\n\t%\n\twhich is strictly negative because $\\underline{\\lambda}_j>0$, $\\mathbf{M}_j(\\mathbf{a}) \\succeq 0$, and $\\hat{\\mathbf{v}}_j\\in \\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right)$, contradicting~\\eqref{eq:inimage}.\n","start":34280,"end":35082}]},{"id":"S033","semantic_type":"proposition","kind":"proposition","labels":["prop:exists_gamma_fv"],"declared_external":false,"source_only":false,"title":"正特征值阈值存在","source_units":[{"raw_tex":"\\begin{proposition}[{Positive finite eigenvalue}]\\label{prop:exists_gamma_fv}\n\tLet $\\mathrm{Im}(\\mathbf{M}_j(\\mathbf{a})) \\subseteq \\mathrm{Im}(\\mathbf{K}_j(\\mathbf{a}))$. Then, there exists $\\hat{\\lambda}>0$ such that\n\t%\n\t$\\mathbf{K}_j(\\mathbf{a}) - \\hat{\\lambda} \\mathbf{M}_j(\\mathbf{a})\\succeq 0$.\n\\end{proposition}","start":36743,"end":37061}]},{"id":"S034","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"正阈值存在的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nSince $\\mathbf{K}_j(\\mathbf{a}) \\in \\mathbb{S}^{{n_\\mathrm{dof},j}}_{\\succeq 0}$ and $\\mathbf{M}_j(\\mathbf{a}) \\in \\mathbb{S}^{{n_\\mathrm{dof},j}}_{\\succeq 0}$, the nonzero eigenvalues of the generalized eigenvalue problem $\\mathbf{K}_j(\\mathbf{a})\\mathbf{v}_j - \\lambda \\mathbf{M}_j(\\mathbf{a}) \\mathbf{v}_j = \\mathbf{0}$, with $\\mathbf{v}_j \\in \\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right)$, must be real and positive. Thus, setting $0<\\hat{\\lambda}= \\lambda_1$ with $\\lambda_1$ being the lowest nonzero eigenvalue, the eigenvalue constraint is satisfied.\n\\end{proof}","start":37062,"end":37649}]},{"id":"S035","semantic_type":"proposition","kind":"proposition","labels":["prop:monotonic"],"declared_external":false,"source_only":false,"title":"缩放特征值单调不减","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:monotonic}\nLet $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ be as defined in \\eqref{eq:RQs} and $j \\in \\mathcal{L}_\\mathrm{fv}$. Then, $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ is a non-decreasing function.\n","start":44422,"end":44665},{"raw_tex":"\\end{proposition}","start":51170,"end":51187}]},{"id":"S036","semantic_type":"proof","kind":"proof","labels":["eq:lamdiff","eq:monotoneous","prop:monotonic"],"declared_external":false,"source_only":false,"title":"特征值单调性的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $\\Delta > 0$; we need to show that $\\lambda_{\\min,j}((\\delta+\\Delta) \\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})\\ge 0$. Using the definition \\eqref{eq:RQs}, we have %\\textcolor{purple}{ We don't really need to write equations (3.11), it is clear enough to directly proceed to (3.12). }\n\\begin{multline}\n\t\\lambda_{\\min,j}((\\delta + \\Delta)\\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})=\\\\\n\t%\n\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + (\\delta+\\Delta) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + (\\delta+\\Delta)^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta }\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\\\\n\t%\n\t- \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}.\n\\end{multline}\n%\nSplitting the first infimum into two parts using the triangle inequality, we obtain \n%\n\\begin{multline}\\label{eq:lamdiff}\n\t\\lambda_{\\min,j}((\\delta + \\Delta)\\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})\\ge\\\\\n\t%\n\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\\\\n\t%\n\t+ \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\Delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + (2\\delta \\Delta + \\Delta^2) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}\\\\\n\t%\n\t- \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}.\n\\end{multline}\n%\nBecause $\\forall \\delta,\\Delta >0: \\frac{1}{\\delta} \\ge \\frac{1}{\\delta + \\Delta}$, it holds for all $\\mathbf{v}_j$, including the minimizing ones, that\n\\begin{equation}\n\t\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j \\ge \\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j.\n\\end{equation}\n%\nConsequently,\n% \n\\begin{multline}\n\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\\\\n\t- \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\ge 0,\n\\end{multline}\n%\nwhich allows us to simplify \\eqref{eq:lamdiff} to\n%\n\\begin{equation}\\label{eq:monotoneous}\n\\begin{multlined}\n\\lambda_{\\min,j}((\\delta + \\Delta)\\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) \\ge\\\\\n \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\\frac{\\Delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + (2\\delta \\Delta + \\Delta^2) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\ge 0.\n\\end{multlined}\n\\end{equation}\n%\nIn \\eqref{eq:monotoneous}, the nonnegativity follows from $\\Delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j \\ge 0$ and $\\left(2\\delta \\Delta + \\Delta^2\\right) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j\\ge 0$, which is because of $\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}, \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\succeq 0$ and $\\delta\\ge 0, \\Delta > 0$.\n\\end{proof}\n","start":44665,"end":51170}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":["eq:lamdiff","prop:monotonic"],"declared_external":false,"source_only":false,"title":"拆分特征值增量","source_units":[{"raw_tex":"Let $\\Delta > 0$; we need to show that $\\lambda_{\\min,j}((\\delta+\\Delta) \\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})\\ge 0$. Using the definition \\eqref{eq:RQs}, we have %\\textcolor{purple}{ We don't really need to write equations (3.11), it is clear enough to directly proceed to (3.12). }\n\\begin{multline}\n\t\\lambda_{\\min,j}((\\delta + \\Delta)\\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})=\\\\\n\t%\n\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + (\\delta+\\Delta) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + (\\delta+\\Delta)^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta }\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\\\\n\t%\n\t- \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}.\n\\end{multline}\n%\nSplitting the first infimum into two parts using the triangle inequality, we obtain \n%\n\\begin{multline}\\label{eq:lamdiff}\n\t\\lambda_{\\min,j}((\\delta + \\Delta)\\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})\\ge\\\\\n\t%\n\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\\\\n\t%\n\t+ \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\Delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + (2\\delta \\Delta + \\Delta^2) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}\\\\\n\t%\n\t- \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}.\n\\end{multline}\n","start":44679,"end":48323}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":["prop:monotonic"],"declared_external":false,"source_only":false,"title":"比较非结构质量分母","source_units":[{"raw_tex":"Because $\\forall \\delta,\\Delta >0: \\frac{1}{\\delta} \\ge \\frac{1}{\\delta + \\Delta}$, it holds for all $\\mathbf{v}_j$, including the minimizing ones, that\n\\begin{equation}\n\t\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j \\ge \\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j.\n\\end{equation}\n%\nConsequently,\n% \n\\begin{multline}\n\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\\\\n\t- \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\ge 0,\n\\end{multline}\n","start":48325,"end":49951}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":["eq:monotoneous","prop:monotonic"],"declared_external":false,"source_only":false,"title":"以弯曲项非负收尾","source_units":[{"raw_tex":"which allows us to simplify \\eqref{eq:lamdiff} to\n%\n\\begin{equation}\\label{eq:monotoneous}\n\\begin{multlined}\n\\lambda_{\\min,j}((\\delta + \\Delta)\\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) \\ge\\\\\n \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\\frac{\\Delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + (2\\delta \\Delta + \\Delta^2) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\ge 0.\n\\end{multlined}\n\\end{equation}\n%\nIn \\eqref{eq:monotoneous}, the nonnegativity follows from $\\Delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j \\ge 0$ and $\\left(2\\delta \\Delta + \\Delta^2\\right) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j\\ge 0$, which is because of $\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}, \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\succeq 0$ and $\\delta\\ge 0, \\Delta > 0$.\n","start":49953,"end":51158}]},{"id":"S040","semantic_type":"proposition","kind":"proposition","labels":["eq:supremum","prop:supremum"],"declared_external":false,"source_only":false,"title":"特征值上确界的膜表达","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:supremum}\nLet $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ be as defined in \\eqref{eq:RQs}, and let $\\mathbf{v}_j := \\mathbf{t}_j + \\mathbf{w}_j$ as defined above. Then, \n\\begin{equation}\\label{eq:supremum}\n\t\\sup_{\\delta} \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) = \\inf_{\\mathbf{w}_j \\in \\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{w}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{w}_j}{\\mathbf{w}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{w}_j}.\n\\end{equation}\n","start":52353,"end":53060},{"raw_tex":"\\end{proposition}","start":55242,"end":55259}]},{"id":"S041","semantic_type":"proof","kind":"proof","labels":["eq:splitting","prop:supremum"],"declared_external":false,"source_only":false,"title":"膜上确界公式的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nWe prove the statement by splitting $\\mathbf{v}_j$ into $\\mathbf{t}_j$ and $\\mathbf{w}_j$ and showing that it must hold that $\\mathbf{t}_j=\\mathbf{0}$.  For this splitting, the supremum is expressed as\n\\begin{equation}\\label{eq:splitting}\n\t\\begin{multlined}\n\t\t\\sup_\\delta \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})= \\\\\n\t\t\\sup_\\delta \\inf\\limits_{\\mathbf{t}_j, \\mathbf{w}_j}\n\t\t \\frac{\\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\left(\\mathbf{t}_j+\\mathbf{w}_j\\right) + \\delta\\mathbf{t}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\mathbf{t}_j + \\delta^2\\mathbf{t}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\mathbf{t}_j }{\\frac{1}{\\delta}\\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\left(\\mathbf{t}_j+\\mathbf{w}_j\\right) + \\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)}.\n\t\\end{multlined}\n\\end{equation}\n%\nBecause of the monotonicity (recall Proposition~\\ref{prop:monotonic}), $\\sup_\\delta \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) = \\lim\\limits_{\\delta\\rightarrow \\infty}\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$. Further, suppose by {contradiction} that $\\mathbf{t}_j \\neq \\mathbf{0}$ holds at the infimum. Then, $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) \\rightarrow \\infty$ as $\\delta \\rightarrow \\infty$, due to $\\mathbf{w}_j \\in \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$ and because $\\mathbf{t}_j$ cannot be in $\\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right)$. On the other hand, for $\\mathbf{t}_j = \\mathbf{0}$, $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ is finite for all $\\delta$ due to $\\mathbf{w}_j \\in \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$. Consequently, $\\mathbf{t}_j^*=\\mathbf{0}$ must hold at the infimum. After inserting $\\mathbf{t}_j^*=\\mathbf{0}$ and $\\delta^*\\rightarrow \\infty$ into \\eqref{eq:splitting}, we receive \\eqref{eq:supremum}, which completes the proof.\n\\end{proof}\n","start":53060,"end":55242}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":["eq:splitting","prop:supremum"],"declared_external":false,"source_only":false,"title":"展开分解后的广义商","source_units":[{"raw_tex":"We prove the statement by splitting $\\mathbf{v}_j$ into $\\mathbf{t}_j$ and $\\mathbf{w}_j$ and showing that it must hold that $\\mathbf{t}_j=\\mathbf{0}$.  For this splitting, the supremum is expressed as\n\\begin{equation}\\label{eq:splitting}\n\t\\begin{multlined}\n\t\t\\sup_\\delta \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})= \\\\\n\t\t\\sup_\\delta \\inf\\limits_{\\mathbf{t}_j, \\mathbf{w}_j}\n\t\t \\frac{\\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\left(\\mathbf{t}_j+\\mathbf{w}_j\\right) + \\delta\\mathbf{t}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\mathbf{t}_j + \\delta^2\\mathbf{t}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\mathbf{t}_j }{\\frac{1}{\\delta}\\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\left(\\mathbf{t}_j+\\mathbf{w}_j\\right) + \\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)}.\n\t\\end{multlined}\n\\end{equation}\n","start":53074,"end":54058}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":["prop:supremum"],"declared_external":false,"source_only":false,"title":"取无穷缩放并消去弯曲方向","source_units":[{"raw_tex":"Because of the monotonicity (recall Proposition~\\ref{prop:monotonic}), $\\sup_\\delta \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) = \\lim\\limits_{\\delta\\rightarrow \\infty}\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$. Further, suppose by {contradiction} that $\\mathbf{t}_j \\neq \\mathbf{0}$ holds at the infimum. Then, $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) \\rightarrow \\infty$ as $\\delta \\rightarrow \\infty$, due to $\\mathbf{w}_j \\in \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$ and because $\\mathbf{t}_j$ cannot be in $\\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right)$. On the other hand, for $\\mathbf{t}_j = \\mathbf{0}$, $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ is finite for all $\\delta$ due to $\\mathbf{w}_j \\in \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$. Consequently, $\\mathbf{t}_j^*=\\mathbf{0}$ must hold at the infimum. After inserting $\\mathbf{t}_j^*=\\mathbf{0}$ and $\\delta^*\\rightarrow \\infty$ into \\eqref{eq:splitting}, we receive \\eqref{eq:supremum}, which completes the proof.\n","start":54060,"end":55230}]},{"id":"S044","semantic_type":"proposition","kind":"proposition","labels":["prop:scaling"],"declared_external":false,"source_only":false,"title":"振动约束的缩放条件","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:scaling}\n\tThere exists $\\delta$ such that $\\mathbf{K}_j(\\delta \\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\delta \\tilde{\\mathbf{a}}) \\succeq 0$ if and only if $\\tilde{\\mathbf{a}} \\in \\left\\{\\mathbf{a} \\;\\vert\\; \\mathbf{a}\\ge \\mathbf{0}, \\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\mathbf{a})\\right),\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0\\right\\}$, where $\\mathbf{W}_j$ denotes an orthonormal basis of $\\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$.\n\\end{proposition}","start":55919,"end":56765}]},{"id":"S045","semantic_type":"proof","kind":"proof","labels":["eq:inequality"],"declared_external":false,"source_only":false,"title":"振动缩放条件的作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\\fbox{$\\Rightarrow$} Let $\\mathbf{K}_j(\\delta \\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\delta \\tilde{\\mathbf{a}}) \\succeq 0$ hold. Then, $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ based on Proposition \\ref{lemma:dynamic_admissibility}. Next, we project the matrix inequality using $\\mathbf{W}_j$, yielding\n%\n\\begin{equation}\n\t\\delta \\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} \\mathbf{W}_j- \\underline{\\lambda}_j \\delta \\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{W}_j - \\underline{\\lambda}_j\\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{W}_j \\succeq 0.\n\\end{equation}\n%\nFor $\\delta>0$, this is further equivalent to\n%\n\\begin{equation}\n\t\\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} \\mathbf{W}_j- \\underline{\\lambda}_j \\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{W}_j \\succeq \\frac{1}{\\delta}\\underline{\\lambda}_j\\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{W}_j \\succeq 0.\n\\end{equation}\n%\n\\fbox{$\\Leftarrow$} Let $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ and $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$ hold. Because of the first assumption and Proposition \\ref{prop:exists_gamma_fv}, it holds that\n%\n\\begin{equation}\n\\forall \\mathbf{v}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right) \\exists \\gamma>0: \\mathbf{v}_j^\\mathrm{T}\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\mathbf{v}_j - \\gamma\\mathbf{v}_j^\\mathrm{T}\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\mathbf{v}_j \\ge 0,\n\\end{equation}\n%\nand thus also $\\mathbf{K}_j(\\tilde{\\mathbf{a}}) - \\gamma\\mathbf{M}_j(\\tilde{\\mathbf{a}}) \\succeq 0$ for $\\gamma > 0$. Consequently, it remains to show that $\\gamma\\ge \\underline{\\lambda}_j$.\n\nBecause of $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$, we also have\n%\n\\begin{equation}\\label{eq:inequality}\n\t\\forall \\mathbf{w}_j \\in \\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right):  \\mathbf{w}_j^\\mathrm{T} \\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right] \\mathbf{w}_j \\ge 0.\n\\end{equation}\n%\nIn particular, the above holds for $w_j$ minimizing the left-hand side of \\eqref{eq:inequality}, yielding\n%\n\\begin{equation}\n\t\\underline{\\lambda}_j \\le \\inf_{\\mathbf{w}_j \\in \\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{w}_j^\\mathrm{T} \\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} \\mathbf{w}_j}{\\mathbf{w}_j^\\mathrm{T} \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\mathbf{w}_j} = \\sup_{\\delta} \\lambda_{\\min,j}(\\mathbf{a}(\\delta))\n\\end{equation}\n%\nbased on Proposition \\ref{prop:supremum}.\n\\end{proof}","start":56766,"end":60141}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"由可行缩放推出必要条件","source_units":[{"raw_tex":"\\fbox{$\\Rightarrow$} Let $\\mathbf{K}_j(\\delta \\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\delta \\tilde{\\mathbf{a}}) \\succeq 0$ hold. Then, $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ based on Proposition \\ref{lemma:dynamic_admissibility}. Next, we project the matrix inequality using $\\mathbf{W}_j$, yielding\n%\n\\begin{equation}\n\t\\delta \\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} \\mathbf{W}_j- \\underline{\\lambda}_j \\delta \\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{W}_j - \\underline{\\lambda}_j\\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{W}_j \\succeq 0.\n\\end{equation}\n%\nFor $\\delta>0$, this is further equivalent to\n%\n\\begin{equation}\n\t\\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} \\mathbf{W}_j- \\underline{\\lambda}_j \\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{W}_j \\succeq \\frac{1}{\\delta}\\underline{\\lambda}_j\\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{W}_j \\succeq 0.\n\\end{equation}\n","start":56780,"end":57927}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"先取得某个正阈值","source_units":[{"raw_tex":"\\fbox{$\\Leftarrow$} Let $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ and $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$ hold. Because of the first assumption and Proposition \\ref{prop:exists_gamma_fv}, it holds that\n%\n\\begin{equation}\n\\forall \\mathbf{v}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right) \\exists \\gamma>0: \\mathbf{v}_j^\\mathrm{T}\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\mathbf{v}_j - \\gamma\\mathbf{v}_j^\\mathrm{T}\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\mathbf{v}_j \\ge 0,\n\\end{equation}\n%\nand thus also $\\mathbf{K}_j(\\tilde{\\mathbf{a}}) - \\gamma\\mathbf{M}_j(\\tilde{\\mathbf{a}}) \\succeq 0$ for $\\gamma > 0$. Consequently, it remains to show that $\\gamma\\ge \\underline{\\lambda}_j$.\n","start":57929,"end":58822}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":["eq:inequality"],"declared_external":false,"source_only":false,"title":"膜条件给出上确界下界","source_units":[{"raw_tex":"Because of $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$, we also have\n%\n\\begin{equation}\\label{eq:inequality}\n\t\\forall \\mathbf{w}_j \\in \\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right):  \\mathbf{w}_j^\\mathrm{T} \\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right] \\mathbf{w}_j \\ge 0.\n\\end{equation}\n%\nIn particular, the above holds for $w_j$ minimizing the left-hand side of \\eqref{eq:inequality}, yielding\n%\n\\begin{equation}\n\t\\underline{\\lambda}_j \\le \\inf_{\\mathbf{w}_j \\in \\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{w}_j^\\mathrm{T} \\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} \\mathbf{w}_j}{\\mathbf{w}_j^\\mathrm{T} \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\mathbf{w}_j} = \\sup_{\\delta} \\lambda_{\\min,j}(\\mathbf{a}(\\delta))\n\\end{equation}\n%\nbased on Proposition \\ref{prop:supremum}.\n","start":58823,"end":60130}]},{"id":"S049","semantic_type":"proposition","kind":"proposition","labels":["th:feasibility_cond"],"declared_external":false,"source_only":false,"title":"下层问题的可行性条件","source_units":[{"raw_tex":"\\begin{proposition}\\label{th:feasibility_cond}\n    The optimization problem \\eqref{eq:bilevel-lower} has a non-empty interior if and only if $\\tilde{\\mathbf{a}} \\in \\mathbb{R}^{n_{\\mathrm{e}}}$ satisfies {the following feasibility conditions:}\n    {\\begin{itemize}\n    \\item $\\tilde{\\mathbf{a}} \\ge \\mathbf{0}$,\n    \\item $\\mathbf{f}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j (\\tilde{\\mathbf{a}})\\right)$ for all $j \\in \\mathcal{L}_\\mathrm{s}$,\n    \\item $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ for all $j \\in \\mathcal{L}_\\mathrm{fv}$,\n    \\item $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$ for all $j \\in \\mathcal{L}_\\mathrm{fv}$,\n    \\end{itemize}}%\n    %\n    \\noindent where $\\mathbf{W}_j$ is an orthonormal basis of $\\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$.\n\\end{proposition}","start":60363,"end":61513}]},{"id":"S050","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"联合可行性的作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\tBased on \\cite[Proposition 5]{Tyburec2023}, for $\\mathbf{f}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j (\\tilde{\\mathbf{a}})\\right)$ there exists a finite $\\delta_{\\mathrm{c},j}^*>0$ such that the static matrix inequality in \\eqref{eq:bilevel-lower} is satisfied. Conversely, for $\\mathbf{f}_j \\notin \\mathrm{Im}\\left(\\mathbf{K}_j (\\tilde{\\mathbf{a}})\\right)$, the optimization problem \\eqref{eq:bilevel-lower} is infeasible.\n\t\n\tUsing Proposition \\ref{prop:scaling}, there exists $\\delta_{\\mathrm{fv},j}^*>0$ such that the free-vibration constraint is satisfied if and only if $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ and $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$.\n    Finally, there exists $\\delta^* = \\max\\{\\max_{j{\\in \\mathcal{L}_\\mathrm{s}}} \\delta^*_{\\mathrm{c},j}, \\max_{j{\\in \\mathcal{L}_\\mathrm{fv}}} \\delta^*_{\\mathrm{fv},j} \\}>0$ such that the free-vibration and compliance constraints are satisfied if and only if the {right-hand-side} conditions of the proposition are satisfied. This is (i) because we have upper bounds for the compliance $\\overline{c}_j$ and compliance is a nonincreasing function in $\\delta$ \\cite[Proposition 1]{Tyburec2023}, and (ii) because we have lower bounds for the lowest free-vibration eigenvalues $\\underline{\\lambda}_j$ and the lowest free-vibration eigenvalue function is a nondecreasing function in $\\delta$ (Proposition \\ref{prop:monotonic}).\n\\end{proof}","start":61514,"end":63129}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"取得各静力工况阈值","source_units":[{"raw_tex":"\tBased on \\cite[Proposition 5]{Tyburec2023}, for $\\mathbf{f}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j (\\tilde{\\mathbf{a}})\\right)$ there exists a finite $\\delta_{\\mathrm{c},j}^*>0$ such that the static matrix inequality in \\eqref{eq:bilevel-lower} is satisfied. Conversely, for $\\mathbf{f}_j \\notin \\mathrm{Im}\\left(\\mathbf{K}_j (\\tilde{\\mathbf{a}})\\right)$, the optimization problem \\eqref{eq:bilevel-lower} is infeasible.\n","start":61528,"end":61947}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"取得各振动工况阈值","source_units":[{"raw_tex":"\tUsing Proposition \\ref{prop:scaling}, there exists $\\delta_{\\mathrm{fv},j}^*>0$ such that the free-vibration constraint is satisfied if and only if $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ and $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$.\n","start":61949,"end":62394}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"取最大阈值统一工况","source_units":[{"raw_tex":"    Finally, there exists $\\delta^* = \\max\\{\\max_{j{\\in \\mathcal{L}_\\mathrm{s}}} \\delta^*_{\\mathrm{c},j}, \\max_{j{\\in \\mathcal{L}_\\mathrm{fv}}} \\delta^*_{\\mathrm{fv},j} \\}>0$ such that the free-vibration and compliance constraints are satisfied if and only if the {right-hand-side} conditions of the proposition are satisfied. This is (i) because we have upper bounds for the compliance $\\overline{c}_j$ and compliance is a nonincreasing function in $\\delta$ \\cite[Proposition 1]{Tyburec2023}, and (ii) because we have lower bounds for the lowest free-vibration eigenvalues $\\underline{\\lambda}_j$ and the lowest free-vibration eigenvalue function is a nondecreasing function in $\\delta$ (Proposition \\ref{prop:monotonic}).\n","start":62394,"end":63118}]},{"id":"S054","semantic_type":"proposition","kind":"proposition","labels":["prop:inertia"],"declared_external":false,"source_only":false,"title":"高次正矩迫使一阶为正","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:inertia}\n\tLet $\\mathbf{y}^{*(r)}$ be the optimal moments associated with the cross-section areas $\\mathbf{a}$ obtained by solving the $r$-th degree relaxation. If $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)}) > 0$ or $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)}) > 0$, then $a_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}^{*(r)}) > 0$.\n\t%\n","start":73554,"end":73950},{"raw_tex":"\\end{proposition}","start":76339,"end":76356}]},{"id":"S055","semantic_type":"proof","kind":"proof","labels":["prop:inertia"],"declared_external":false,"source_only":false,"title":"矩零模式的作者证明","source_units":[{"raw_tex":"\t\\begin{proof}\n        By contradiction, let us assume that $a_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}^{*(r)}) = 0$, i.e., $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$. Then, we need to show that $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)}) = 0$ and $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)}) = 0$.\n\t\t\n\t\tConsider first the case of $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)})$. The moment matrix {(recall the definition in \\eqref{eq:moment})} then contains the principal submatrix\n\t\t%\n\t\t\\begin{equation}\n\t\t\t\\begin{pmatrix}\n\t\t\t\t1 & y_{a_{\\mathrm{s},e}^1}^{*(r)}\\\\\n\t\t\t\ty_{a_{\\mathrm{s},e}^1}^{*(r)} & y_{a_{\\mathrm{s},e}^2}^{*(r)}\n\t\t\t\\end{pmatrix} \\succeq 0.\n\t\t\\end{equation}\n\t\t%\n\t\tThe determinant of this matrix must be nonnegative, so that we obtain the inequality\n\t\t%\n\t\t\\begin{equation}\n\t\t\ty_{a_{\\mathrm{s},e}^2}^{*(r)} - \\left[y_{a_{\\mathrm{s},e}^1}^{*(r)}\\right]^2 \\ge 0.\n\t\t\\end{equation}\n\t\t%\n\t\tAfter inserting $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$, we obtain $y_{a_{\\mathrm{s},e}^2}^{*(r)} \\ge 1$. Due to the bound constraints \\eqref{eq:opt_nonneg_po}, $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$. Using $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$ with $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$, \\eqref{eq:rel_a_2} is then evaluated as $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)})=0$.\n\t\t\n\t\tFor the case of, $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)})$, the moment matrix contains the principal submatrix\n\t\t%\n\t\t\\begin{equation}\n\t\t\t\\begin{pmatrix}\n\t\t\t\t1 & y_{a_{\\mathrm{s},e}^1}^{*(r)} & y_{a_{\\mathrm{s},e}^2}^{*(r)}\\\\\n\t\t\t\ty_{a_{\\mathrm{s},e}^1}^{*(r)} & y_{a_{\\mathrm{s},e}^2}^{*(r)} & y_{a_{\\mathrm{s},e}^3}^{*(r)}\\\\\n\t\t\t\ty_{a_{\\mathrm{s},e}^2}^{*(r)} & y_{a_{\\mathrm{s},e}^3}^{*(r)} & y_{a_{\\mathrm{s},e}^4}^{*(r)}\n\t\t\t\\end{pmatrix} \\succeq 0.\n\t\t\\end{equation}\n\t\t%\n\t\tAfter inserting $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$ and $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$, the determinant evaluates as\n\t\t%\n\t\t\\begin{equation}\n\t\t\t- \\left(y_{a_{\\mathrm{s},e}^3}^{*(r)} + 1\\right)^2 \\ge 0,\n\t\t\\end{equation}\n\t\t%\n\t\tso that $y_{a_{\\mathrm{s},e}^3}^{*(r)}=-1$ is the only feasible solution. Inserting $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$, $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$, and $y_{a_{\\mathrm{s},e}^3}^{*(r)} = -1$ into \\eqref{eq:rel_a_3}, we again receive $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)})=0$, which finishes the proof.\n\t\\end{proof}\n","start":73950,"end":76339}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":["prop:inertia"],"declared_external":false,"source_only":false,"title":"设一阶提升量为零","source_units":[{"raw_tex":"        By contradiction, let us assume that $a_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}^{*(r)}) = 0$, i.e., $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$. Then, we need to show that $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)}) = 0$ and $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)}) = 0$.\n","start":73965,"end":74276}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":["prop:inertia"],"declared_external":false,"source_only":false,"title":"二阶主子式锁定二次矩","source_units":[{"raw_tex":"\t\tConsider first the case of $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)})$. The moment matrix {(recall the definition in \\eqref{eq:moment})} then contains the principal submatrix\n\t\t%\n\t\t\\begin{equation}\n\t\t\t\\begin{pmatrix}\n\t\t\t\t1 & y_{a_{\\mathrm{s},e}^1}^{*(r)}\\\\\n\t\t\t\ty_{a_{\\mathrm{s},e}^1}^{*(r)} & y_{a_{\\mathrm{s},e}^2}^{*(r)}\n\t\t\t\\end{pmatrix} \\succeq 0.\n\t\t\\end{equation}\n\t\t%\n\t\tThe determinant of this matrix must be nonnegative, so that we obtain the inequality\n\t\t%\n\t\t\\begin{equation}\n\t\t\ty_{a_{\\mathrm{s},e}^2}^{*(r)} - \\left[y_{a_{\\mathrm{s},e}^1}^{*(r)}\\right]^2 \\ge 0.\n\t\t\\end{equation}\n\t\t%\n\t\tAfter inserting $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$, we obtain $y_{a_{\\mathrm{s},e}^2}^{*(r)} \\ge 1$. Due to the bound constraints \\eqref{eq:opt_nonneg_po}, $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$. Using $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$ with $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$, \\eqref{eq:rel_a_2} is then evaluated as $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)})=0$.\n","start":74279,"end":75270}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":["prop:inertia"],"declared_external":false,"source_only":false,"title":"三阶主子式锁定三次矩","source_units":[{"raw_tex":"\t\tFor the case of, $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)})$, the moment matrix contains the principal submatrix\n\t\t%\n\t\t\\begin{equation}\n\t\t\t\\begin{pmatrix}\n\t\t\t\t1 & y_{a_{\\mathrm{s},e}^1}^{*(r)} & y_{a_{\\mathrm{s},e}^2}^{*(r)}\\\\\n\t\t\t\ty_{a_{\\mathrm{s},e}^1}^{*(r)} & y_{a_{\\mathrm{s},e}^2}^{*(r)} & y_{a_{\\mathrm{s},e}^3}^{*(r)}\\\\\n\t\t\t\ty_{a_{\\mathrm{s},e}^2}^{*(r)} & y_{a_{\\mathrm{s},e}^3}^{*(r)} & y_{a_{\\mathrm{s},e}^4}^{*(r)}\n\t\t\t\\end{pmatrix} \\succeq 0.\n\t\t\\end{equation}\n\t\t%\n\t\tAfter inserting $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$ and $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$, the determinant evaluates as\n\t\t%\n\t\t\\begin{equation}\n\t\t\t- \\left(y_{a_{\\mathrm{s},e}^3}^{*(r)} + 1\\right)^2 \\ge 0,\n\t\t\\end{equation}\n\t\t%\n\t\tso that $y_{a_{\\mathrm{s},e}^3}^{*(r)}=-1$ is the only feasible solution. Inserting $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$, $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$, and $y_{a_{\\mathrm{s},e}^3}^{*(r)} = -1$ into \\eqref{eq:rel_a_3}, we again receive $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)})=0$, which finishes the proof.\n","start":75273,"end":76326}]},{"id":"S059","semantic_type":"proposition","kind":"proposition","labels":["eq:scalar_fvineq","th:ub"],"declared_external":false,"source_only":false,"title":"一阶矩比例的振动可行缩放","source_units":[{"raw_tex":"\\begin{proposition}\\label{th:ub}\n\tLet $\\mathbf{y}^{*(r)}$ be the optimal moments obtained by solving the $r$-th degree relaxation and set $\\tilde{\\mathbf{a}} := \\mathbf{a}_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}^{*(r)})$. Then, there exists $\\delta$ such that\n\t%\n\t\\begin{equation}\\label{eq:scalar_fvineq}\n\t\t\\mathbf{K}_j(\\delta\\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\delta\\tilde{\\mathbf{a}}) \\succeq 0\\,.\n\t\\end{equation}\n\\end{proposition}","start":76589,"end":77051}]},{"id":"S060","semantic_type":"proof","kind":"proof","labels":["eq:fvrelaxed","eq:rel_cond1","eq:rel_cond2","eq:spans"],"declared_external":false,"source_only":false,"title":"振动上界的作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\tFrom the hierarchy construction {\\eqref{eq:general_moment}}, the {localizing matrix \\eqref{eq:localizing} will always contain $L_\\mathbf{y}(1 \\otimes \\left[\\mathbf{K}_j (\\mathbf{a}) - \\underline{\\lambda} \\mathbf{M}_j(\\mathbf{a})\\right])$ as principal submatrix due to} \\eqref{eq:opt_fv}{, which} can be written in terms of the lifted variables as\n\t%\n\t\\begin{multline}\\label{eq:fvrelaxed}\n\t\t\\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)})\\right]^{\\circ\\frac{1}{2}}\\right) + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)})\\right]^{\\circ\\frac{1}{3}}\\right) \\\\\n\t\t- \\underline{\\lambda}_j\\left( \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) \\right)\\succeq 0,\n\t\\end{multline}\n\t%\n\twith $\\bullet^{\\circ q}$ raising each element in $\\bullet$ to the power of $q$. We note here that the power factors in \\eqref{eq:fvrelaxed} eliminate the powers introduced in the notation \\eqref{eq:hatnotation}, so that \\eqref{eq:fvrelaxed} is indeed linear in $\\mathbf{a}_\\mathrm{rel}^{(i)}$ for all $i$.\n \n    After projecting via an orthonormal basis of $\\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$ denoted with $\\mathbf{W}_j$, we receive\n\t%\n\t\\begin{equation}\\label{eq:rel_cond1}\n\t\t\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) \\right]\\mathbf{W}_j \\succeq \\underline{\\lambda}_j \\mathbf{W}_j^\\mathrm{T} \\hat{\\mathbf{M}}_{j}^{\\langle 0\\rangle}\\mathbf{W}_j \\succeq 0.\n\t\\end{equation}\n\t%\n\tFurther, because of Proposition \\ref{prop:inertia}, {$a_{\\mathrm{rel},e}^{\\langle 1 \\rangle} \\mathbf{K}_{j,e}^{\\langle 1\\rangle}\\neq \\mathbf{0}$} whenever {$a_{\\mathrm{rel},e}^{\\langle2\\rangle} \\mathbf{K}_{j,e}^{\\langle 2\\rangle} + a_{\\mathrm{rel},e}^{\\langle3 \\rangle} \\mathbf{K}_{j,e}^{\\langle 3\\rangle} \\neq \\mathbf{0}$}. Consequently, \n\t\\begin{multline}\\label{eq:spans}\n\t\t\\mathrm{Span}\\left(\\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} (\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 2\\rangle}\\right]^{\\circ\\frac{1}{2}}\\right) + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 3\\rangle}\\right]^{\\circ\\frac{1}{3}}\\right)\\right)\\\\ = \\mathrm{Span}\\left(\\hat{\\mathbf{K}}_j^{\\langle 1 \\rangle} (\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}(\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{{\\langle 3\\rangle}}(\\tilde{\\mathbf{a}}) \\right).\n\t\\end{multline}\n\t%\n\tThus, because \\eqref{eq:fvrelaxed} is feasible due to $\\mathbf{y}^{*(r)}$ being optimal, it holds based on Proposition \\ref{lemma:dynamic_admissibility} and based on \\eqref{eq:spans} that \n\t%\n\t\\begin{equation}\\label{eq:rel_cond2}\n\t\t\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right).\n\t\\end{equation}\n\t%\n\tUsing \\eqref{eq:rel_cond1} and \\eqref{eq:rel_cond2}, Proposition \\ref{prop:scaling} certifies existence of $\\delta$ such that the matrix inequality \\eqref{eq:scalar_fvineq} holds.\n\\end{proof}","start":77052,"end":80485}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":["eq:fvrelaxed"],"declared_external":false,"source_only":false,"title":"抽取提升振动矩阵","source_units":[{"raw_tex":"\tFrom the hierarchy construction {\\eqref{eq:general_moment}}, the {localizing matrix \\eqref{eq:localizing} will always contain $L_\\mathbf{y}(1 \\otimes \\left[\\mathbf{K}_j (\\mathbf{a}) - \\underline{\\lambda} \\mathbf{M}_j(\\mathbf{a})\\right])$ as principal submatrix due to} \\eqref{eq:opt_fv}{, which} can be written in terms of the lifted variables as\n\t%\n\t\\begin{multline}\\label{eq:fvrelaxed}\n\t\t\\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)})\\right]^{\\circ\\frac{1}{2}}\\right) + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)})\\right]^{\\circ\\frac{1}{3}}\\right) \\\\\n\t\t- \\underline{\\lambda}_j\\left( \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) \\right)\\succeq 0,\n\t\\end{multline}\n\t%\n\twith $\\bullet^{\\circ q}$ raising each element in $\\bullet$ to the power of $q$. We note here that the power factors in \\eqref{eq:fvrelaxed} eliminate the powers introduced in the notation \\eqref{eq:hatnotation}, so that \\eqref{eq:fvrelaxed} is indeed linear in $\\mathbf{a}_\\mathrm{rel}^{(i)}$ for all $i$.\n","start":77066,"end":78288}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":["eq:rel_cond1"],"declared_external":false,"source_only":false,"title":"投影得到膜可行条件","source_units":[{"raw_tex":"    After projecting via an orthonormal basis of $\\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$ denoted with $\\mathbf{W}_j$, we receive\n\t%\n\t\\begin{equation}\\label{eq:rel_cond1}\n\t\t\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) \\right]\\mathbf{W}_j \\succeq \\underline{\\lambda}_j \\mathbf{W}_j^\\mathrm{T} \\hat{\\mathbf{M}}_{j}^{\\langle 0\\rangle}\\mathbf{W}_j \\succeq 0.\n\t\\end{equation}\n","start":78290,"end":78960}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":["eq:spans"],"declared_external":false,"source_only":false,"title":"提升刚度与实际刚度同像","source_units":[{"raw_tex":"\tFurther, because of Proposition \\ref{prop:inertia}, {$a_{\\mathrm{rel},e}^{\\langle 1 \\rangle} \\mathbf{K}_{j,e}^{\\langle 1\\rangle}\\neq \\mathbf{0}$} whenever {$a_{\\mathrm{rel},e}^{\\langle2\\rangle} \\mathbf{K}_{j,e}^{\\langle 2\\rangle} + a_{\\mathrm{rel},e}^{\\langle3 \\rangle} \\mathbf{K}_{j,e}^{\\langle 3\\rangle} \\neq \\mathbf{0}$}. Consequently, \n\t\\begin{multline}\\label{eq:spans}\n\t\t\\mathrm{Span}\\left(\\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} (\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 2\\rangle}\\right]^{\\circ\\frac{1}{2}}\\right) + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 3\\rangle}\\right]^{\\circ\\frac{1}{3}}\\right)\\right)\\\\ = \\mathrm{Span}\\left(\\hat{\\mathbf{K}}_j^{\\langle 1 \\rangle} (\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}(\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{{\\langle 3\\rangle}}(\\tilde{\\mathbf{a}}) \\right).\n\t\\end{multline}\n","start":78963,"end":79913}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":["eq:rel_cond2"],"declared_external":false,"source_only":false,"title":"转移质量可容许性","source_units":[{"raw_tex":"\tThus, because \\eqref{eq:fvrelaxed} is feasible due to $\\mathbf{y}^{*(r)}$ being optimal, it holds based on Proposition \\ref{lemma:dynamic_admissibility} and based on \\eqref{eq:spans} that \n\t%\n\t\\begin{equation}\\label{eq:rel_cond2}\n\t\t\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right).\n\t\\end{equation}\n","start":79916,"end":80290}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"应用振动缩放命题","source_units":[{"raw_tex":"\tUsing \\eqref{eq:rel_cond1} and \\eqref{eq:rel_cond2}, Proposition \\ref{prop:scaling} certifies existence of $\\delta$ such that the matrix inequality \\eqref{eq:scalar_fvineq} holds.\n","start":80293,"end":80474}]},{"id":"S066","semantic_type":"proposition","kind":"proposition","labels":["prop:feasibleub"],"declared_external":false,"source_only":false,"title":"一阶矩比例的联合可行缩放","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:feasibleub}\n\tLet $\\mathbf{y}^{*(r)}$ be the optimal moments obtained by solving the $r$-th degree relaxation and set $\\tilde{\\mathbf{a}} := \\mathbf{a}_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}^{*(r)})$. Then, there exists a $\\delta$ such that $\\mathbf{a}(\\delta):=\\delta\\tilde{\\mathbf{a}}$ is feasible to the lower-level problem \\eqref{eq:bilevel-lower}.\n\\end{proposition}","start":80750,"end":81158}]},{"id":"S067","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"联合可行缩放的作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\tFor free-vibration constraints, the proof follows from Proposition \\ref{th:ub}. For static constraints, based on Proposition \\ref{prop:inertia}, equation~\\eqref{eq:spans} holds. Thus, since\n\t%\n\t\\begin{equation}\n\t\t\\begin{pmatrix}\n\t\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t\t-\\mathbf{f}_j & \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 2\\rangle}\\right]^{\\circ\\frac{1}{2}}\\right) + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 3\\rangle}\\right]^{\\circ\\frac{1}{3}}\\right)\n\t\t\\end{pmatrix} \\succeq 0\n\t\\end{equation}\n\t%\n\tis satisfied by optimality of $\\mathbf{y}^{*(r)}$, the condition $\\mathbf{f}_j \\in \\mathbf{K}_j(\\mathbf{\\tilde{a}})$ holds true due to Proposition \\ref{lemma:statical_admissibility}. Based on Proposition \\ref{th:feasibility_cond}, the upper bound solution can thus be constructed.\n\t\n\tConsequently, there exists $\\delta$ such that both types of constraints are satisfied simultaneously.\n\\end{proof}","start":81159,"end":82228}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"沿用振动可行缩放","source_units":[{"raw_tex":"\tFor free-vibration constraints, the proof follows from Proposition \\ref{th:ub}. For static constraints, based on Proposition \\ref{prop:inertia}, equation~\\eqref{eq:spans} holds. Thus, since\n","start":81173,"end":81364}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"使用提升柔度分块约束","source_units":[{"raw_tex":"\t\\begin{equation}\n\t\t\\begin{pmatrix}\n\t\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t\t-\\mathbf{f}_j & \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 2\\rangle}\\right]^{\\circ\\frac{1}{2}}\\right) + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 3\\rangle}\\right]^{\\circ\\frac{1}{3}}\\right)\n\t\t\\end{pmatrix} \\succeq 0\n\t\\end{equation}\n","start":81367,"end":81827}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"得到实际比例静力可容许","source_units":[{"raw_tex":"\tFor free-vibration constraints, the proof follows from Proposition \\ref{th:ub}. For static constraints, based on Proposition \\ref{prop:inertia}, equation~\\eqref{eq:spans} holds. Thus, since\n","start":81173,"end":81364},{"raw_tex":"\tis satisfied by optimality of $\\mathbf{y}^{*(r)}$, the condition $\\mathbf{f}_j \\in \\mathbf{K}_j(\\mathbf{\\tilde{a}})$ holds true due to Proposition \\ref{lemma:statical_admissibility}. Based on Proposition \\ref{th:feasibility_cond}, the upper bound solution can thus be constructed.\n","start":81830,"end":82112}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"合并得到共同缩放","source_units":[{"raw_tex":"\tis satisfied by optimality of $\\mathbf{y}^{*(r)}$, the condition $\\mathbf{f}_j \\in \\mathbf{K}_j(\\mathbf{\\tilde{a}})$ holds true due to Proposition \\ref{lemma:statical_admissibility}. Based on Proposition \\ref{th:feasibility_cond}, the upper bound solution can thus be constructed.\n","start":81830,"end":82112},{"raw_tex":"\tConsequently, there exists $\\delta$ such that both types of constraints are satisfied simultaneously.\n","start":82114,"end":82217}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S066","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S001","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S005","target_id":"S008","kind":"supports"},{"source_id":"S006","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S006","target_id":"S013","kind":"supports"},{"source_id":"S007","target_id":"S013","kind":"supports"},{"source_id":"S010","target_id":"S013","kind":"supports"},{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S026","target_id":"S012","kind":"supports"},{"source_id":"S033","target_id":"S014","kind":"supports"},{"source_id":"S012","target_id":"S014","kind":"supports"},{"source_id":"S012","target_id":"S015","kind":"supports"},{"source_id":"S010","target_id":"S015","kind":"supports"},{"source_id":"S013","target_id":"S016","kind":"supports"},{"source_id":"S007","target_id":"S016","kind":"supports"},{"source_id":"S005","target_id":"S017","kind":"supports"},{"source_id":"S008","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S008","target_id":"S020","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S002","target_id":"S022","kind":"supports"},{"source_id":"S018","target_id":"S022","kind":"supports"},{"source_id":"S003","target_id":"S066","kind":"supports"},{"source_id":"S020","target_id":"S066","kind":"supports"},{"source_id":"S022","target_id":"S066","kind":"supports"},{"source_id":"S008","target_id":"S031","kind":"supports"},{"source_id":"S024","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S007","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S006","target_id":"S034","kind":"supports"},{"source_id":"S007","target_id":"S034","kind":"supports"},{"source_id":"S024","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S015","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S007","target_id":"S038","kind":"supports"},{"source_id":"S024","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S006","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S039","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S016","target_id":"S042","kind":"supports"},{"source_id":"S015","target_id":"S042","kind":"supports"},{"source_id":"S035","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S043","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S029","target_id":"S046","kind":"supports"},{"source_id":"S006","target_id":"S046","kind":"supports"},{"source_id":"S007","target_id":"S046","kind":"supports"},{"source_id":"S013","target_id":"S046","kind":"supports"},{"source_id":"S024","target_id":"S046","kind":"supports"},{"source_id":"S025","target_id":"S046","kind":"supports"},{"source_id":"S033","target_id":"S047","kind":"supports"},{"source_id":"S006","target_id":"S047","kind":"supports"},{"source_id":"S007","target_id":"S047","kind":"supports"},{"source_id":"S040","target_id":"S048","kind":"supports"},{"source_id":"S013","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S048","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S027","target_id":"S051","kind":"supports"},{"source_id":"S011","target_id":"S051","kind":"supports"},{"source_id":"S044","target_id":"S052","kind":"supports"},{"source_id":"S051","target_id":"S053","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S028","target_id":"S053","kind":"supports"},{"source_id":"S035","target_id":"S053","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S022","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S003","target_id":"S057","kind":"supports"},{"source_id":"S020","target_id":"S057","kind":"supports"},{"source_id":"S022","target_id":"S057","kind":"supports"},{"source_id":"S024","target_id":"S057","kind":"supports"},{"source_id":"S056","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S003","target_id":"S058","kind":"supports"},{"source_id":"S022","target_id":"S058","kind":"supports"},{"source_id":"S024","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S055","kind":"supports"},{"source_id":"S057","target_id":"S055","kind":"supports"},{"source_id":"S058","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S054","kind":"supports"},{"source_id":"S003","target_id":"S061","kind":"supports"},{"source_id":"S008","target_id":"S061","kind":"supports"},{"source_id":"S020","target_id":"S061","kind":"supports"},{"source_id":"S022","target_id":"S061","kind":"supports"},{"source_id":"S013","target_id":"S061","kind":"supports"},{"source_id":"S024","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S054","target_id":"S062","kind":"supports"},{"source_id":"S006","target_id":"S062","kind":"supports"},{"source_id":"S024","target_id":"S062","kind":"supports"},{"source_id":"S025","target_id":"S062","kind":"supports"},{"source_id":"S054","target_id":"S063","kind":"supports"},{"source_id":"S006","target_id":"S063","kind":"supports"},{"source_id":"S025","target_id":"S063","kind":"supports"},{"source_id":"S061","target_id":"S064","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S029","target_id":"S064","kind":"supports"},{"source_id":"S062","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S044","target_id":"S065","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S060","kind":"supports"},{"source_id":"S063","target_id":"S060","kind":"supports"},{"source_id":"S064","target_id":"S060","kind":"supports"},{"source_id":"S065","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S068","kind":"supports"},{"source_id":"S003","target_id":"S069","kind":"supports"},{"source_id":"S020","target_id":"S069","kind":"supports"},{"source_id":"S022","target_id":"S069","kind":"supports"},{"source_id":"S013","target_id":"S069","kind":"supports"},{"source_id":"S069","target_id":"S070","kind":"supports"},{"source_id":"S009","target_id":"S070","kind":"supports"},{"source_id":"S054","target_id":"S070","kind":"supports"},{"source_id":"S063","target_id":"S070","kind":"supports"},{"source_id":"S068","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S049","target_id":"S071","kind":"supports"},{"source_id":"S068","target_id":"S067","kind":"supports"},{"source_id":"S069","target_id":"S067","kind":"supports"},{"source_id":"S070","target_id":"S067","kind":"supports"},{"source_id":"S071","target_id":"S067","kind":"supports"},{"source_id":"S067","target_id":"S066","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S066","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S001","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S005","target_id":"S008","kind":"supports"},{"source_id":"S006","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S006","target_id":"S013","kind":"supports"},{"source_id":"S007","target_id":"S013","kind":"supports"},{"source_id":"S010","target_id":"S013","kind":"supports"},{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S026","target_id":"S012","kind":"supports"},{"source_id":"S033","target_id":"S014","kind":"supports"},{"source_id":"S012","target_id":"S014","kind":"supports"},{"source_id":"S012","target_id":"S015","kind":"supports"},{"source_id":"S010","target_id":"S015","kind":"supports"},{"source_id":"S013","target_id":"S016","kind":"supports"},{"source_id":"S007","target_id":"S016","kind":"supports"},{"source_id":"S005","target_id":"S017","kind":"supports"},{"source_id":"S008","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S008","target_id":"S020","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S002","target_id":"S022","kind":"supports"},{"source_id":"S018","target_id":"S022","kind":"supports"},{"source_id":"S003","target_id":"S066","kind":"supports"},{"source_id":"S020","target_id":"S066","kind":"supports"},{"source_id":"S022","target_id":"S066","kind":"supports"},{"source_id":"S008","target_id":"S029","kind":"supports"},{"source_id":"S024","target_id":"S029","kind":"supports"},{"source_id":"S007","target_id":"S029","kind":"supports"},{"source_id":"S006","target_id":"S033","kind":"supports"},{"source_id":"S007","target_id":"S033","kind":"supports"},{"source_id":"S024","target_id":"S033","kind":"supports"},{"source_id":"S015","target_id":"S035","kind":"supports"},{"source_id":"S007","target_id":"S035","kind":"supports"},{"source_id":"S024","target_id":"S035","kind":"supports"},{"source_id":"S006","target_id":"S035","kind":"supports"},{"source_id":"S016","target_id":"S040","kind":"supports"},{"source_id":"S015","target_id":"S040","kind":"supports"},{"source_id":"S035","target_id":"S040","kind":"supports"},{"source_id":"S029","target_id":"S044","kind":"supports"},{"source_id":"S006","target_id":"S044","kind":"supports"},{"source_id":"S007","target_id":"S044","kind":"supports"},{"source_id":"S013","target_id":"S044","kind":"supports"},{"source_id":"S024","target_id":"S044","kind":"supports"},{"source_id":"S025","target_id":"S044","kind":"supports"},{"source_id":"S033","target_id":"S044","kind":"supports"},{"source_id":"S040","target_id":"S044","kind":"supports"},{"source_id":"S027","target_id":"S049","kind":"supports"},{"source_id":"S011","target_id":"S049","kind":"supports"},{"source_id":"S044","target_id":"S049","kind":"supports"},{"source_id":"S028","target_id":"S049","kind":"supports"},{"source_id":"S035","target_id":"S049","kind":"supports"},{"source_id":"S022","target_id":"S054","kind":"supports"},{"source_id":"S003","target_id":"S054","kind":"supports"},{"source_id":"S020","target_id":"S054","kind":"supports"},{"source_id":"S024","target_id":"S054","kind":"supports"},{"source_id":"S003","target_id":"S059","kind":"supports"},{"source_id":"S008","target_id":"S059","kind":"supports"},{"source_id":"S020","target_id":"S059","kind":"supports"},{"source_id":"S022","target_id":"S059","kind":"supports"},{"source_id":"S013","target_id":"S059","kind":"supports"},{"source_id":"S024","target_id":"S059","kind":"supports"},{"source_id":"S054","target_id":"S059","kind":"supports"},{"source_id":"S006","target_id":"S059","kind":"supports"},{"source_id":"S025","target_id":"S059","kind":"supports"},{"source_id":"S054","target_id":"S063","kind":"supports"},{"source_id":"S006","target_id":"S063","kind":"supports"},{"source_id":"S025","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S059","kind":"supports"},{"source_id":"S029","target_id":"S059","kind":"supports"},{"source_id":"S044","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S068","kind":"supports"},{"source_id":"S003","target_id":"S069","kind":"supports"},{"source_id":"S020","target_id":"S069","kind":"supports"},{"source_id":"S022","target_id":"S069","kind":"supports"},{"source_id":"S013","target_id":"S069","kind":"supports"},{"source_id":"S069","target_id":"S070","kind":"supports"},{"source_id":"S009","target_id":"S070","kind":"supports"},{"source_id":"S054","target_id":"S070","kind":"supports"},{"source_id":"S063","target_id":"S070","kind":"supports"},{"source_id":"S068","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S049","target_id":"S071","kind":"supports"},{"source_id":"S068","target_id":"S067","kind":"supports"},{"source_id":"S069","target_id":"S067","kind":"supports"},{"source_id":"S070","target_id":"S067","kind":"supports"},{"source_id":"S071","target_id":"S067","kind":"supports"},{"source_id":"S067","target_id":"S066","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"notation","kind":"notation","labels":["strandard_basis"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"标准单项式基","source_units":[{"raw_tex":"Consider a polynomial $f(\\mathbf{x})$ of a degree $d$, which can be written as $f(\\mathbf{x}) = \\mathbf{p}^\\mathrm{T}\\mathbf{b}_d(\\mathbf{x})$, where\n%\n\\begin{equation}\n\\begin{aligned}\n\\mathbf{b}_d(\\mathbf{x})=&\\Big( 1 \\quad x_1 \\quad x_2 \\quad \\cdots \\quad x_1^2 \\quad x_1x_2 \\quad  \\cdots  \\quad x_1x_n \\\\ &  x_2^2 \\quad x_2x_3 \\quad  \\cdots \\quad x_n^2 \\quad \\cdots \\quad x_1^d \\quad \\cdots \\quad x_n^d \\Big)^\\mathrm{T}\n\\end{aligned}\n\\label{strandard_basis}\n\\end{equation}\n%\ndenotes the canonical monomial basis of degree up to $d$ and $\\mathbf{p} \\in \\mathbb{R}^{|\\mathbf{b}_d(\\mathbf{x})|}$ is the corresponding coefficient vector. To certify polynomial nonnegativity, we rely on the Sum-Of-Squares (SOS) representation\n","start":18904,"end":19629}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"矩变量与里斯泛函","source_units":[{"raw_tex":"{To construct these truncations, we introduce moment variables $\\mathbf{y} \\in \\mathbb{R}^{|\\mathbf{b}_{2r}(\\mathbf{x})|}$ indexed by the monomial basis $\\mathbf{b}_{2r}(\\mathbf{x})$. Further, let us define} the Riesz functional $L_\\mathbf{y} (\\mathbf{Z}(\\mathbf{x}))$ that linearizes {any} polynomial matrix $\\mathbf{Z}(\\mathbf{x})$ using $\\mathbf{y}$. Specifically, the operator is applied entry-wise and for the $(i,j)$-th component of $\\mathbf{Z}$ we write\n%\n\\begin{equation}\n    Z_{i,j}(\\mathbf{x}) = \\mathbf{p}_{{{i,j}}}^\\mathrm{T} \\mathbf{b}_{2r}(\\mathbf{x})\\quad \\Longleftrightarrow \\quad L_\\mathbf{y}(Z_{i,j}) = \\mathbf{p}_{{{i,j}}}^\\mathrm{T} \\mathbf{y}\\; \\text{ with }\\mathbf{y}\\text{ indexed in }\\mathbf{b}_{2r}(\\mathbf{x})\n\\end{equation}\n%\nwith a coefficient vector $\\mathbf{p}_{{{i,j}}}$. In this manuscript, we further adopt the notation $y_{x_i^d}$ to express the component of $\\mathbf{y}$ belonging to $x_i^d \\in \\mathbf{b}_{2r}(\\mathbf{x})$.\n\n","start":21814,"end":22775}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":["eq:localizing","eq:moment"],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"矩与局部化矩阵","source_units":[{"raw_tex":"{\nUnder Assumption \\ref{as:archimedean} (Archimedean assumption), polynomial optimization can be solved via the moment-sum-of-squares hierarchy:\n%\n\\begin{subequations}\\label{eq:general_moment}\n\\begin{align}\n    \\min_{\\mathbf{y}} \\; & L_\\mathbf{y}(f(\\mathbf{x}))\\\\\n    \\mathrm{s.t.}\\; & L_{\\mathbf{y}}\\bigl(\\mathbf{b}_{r-d_\\mathrm{G}}(\\mathbf{x}) \\mathbf{b}_{r-d_\\mathrm{G}}(\\mathbf{x})^\\mathrm{T} \\otimes \\mathbf{G}(\\mathbf{x})\\bigr) \\succeq 0, & \\text{(localizing matrix)}\\label{eq:localizing}\\\\\n    & L_{\\mathbf{y}}\\bigl(\\mathbf{b}_r(\\mathbf{x}) \\mathbf{b}_r(\\mathbf{x})^\\mathrm{T}\\bigr) \\succeq 0, & \\text{(moment matrix)}\\label{eq:moment}\n\\end{align}\n\\end{subequations}\n%\nwhere $d_\\mathrm{G}= \\lceil \\mathrm{deg}(\\mathbf{G}(\\mathbf{x}))/2 \\rceil$ (with $\\lceil \\bullet \\rceil$ denoting the ceiling function), and $\\otimes$ denotes the Kronecker product. The constraints define moment matrices $\\mathbf{M}_r(\\mathbf{y})$ and localizing matrices $\\mathbf{M}_{r-d_\\mathrm{G}} (\\mathbf{G}\\mathbf{y})$ that correspond to the SOS conditions in the dual formulation.}\n\n","start":22775,"end":23841}]},{"id":"S005","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:opt_orig_compliance","eq:opt_orig_fv","eq:opt_orig_nonneg","eq:opt_orig_objective","eq:opt_orig_static"],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"多工况最小重量问题","source_units":[{"raw_tex":"{Let $\\mathcal{L}_\\mathrm{s}$ and $\\mathcal{L}_\\mathrm{fv}$ denote the sets of indices corresponding to static and free-vibration load cases, respectively.} We investigate the global solution to topology optimization of least-weight frame structures \\eqref{eq:opt_orig_objective}, while accounting for upper bounds $\\overline{c}_j \\in \\mathbb{R}_{>0}$, $j \\in {\\mathcal{L}_\\mathrm{s}}$, on structural compliances \\eqref{eq:opt_orig_compliance} of {$\\lvert \\mathcal{L}_\\mathrm{s} \\rvert$ static} load cases and lower bounds $\\underline{\\lambda}_j \\in \\mathbb{R}_{>0}${, $j \\in \\mathcal{L}_\\mathrm{fv}$,} on the lowest nonzero free-vibration eigenvalues {of the $\\lvert \\mathcal{L}_\\mathrm{fv} \\rvert$ free-vibration load cases} expressed using the Rayleigh quotient \\eqref{eq:opt_orig_fv}. Such optimization problems are naturally formalized in terms of cross-section areas $\\mathbf{a}$, displacements $\\mathcal{U} = {\\{\\mathbf{u}_j : j \\in \\mathcal{L}_\\mathrm{s}\\}}$ with $\\mathbf{u}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}$, and normalized eigenmodes $\\mathcal{V} = {\\{\\mathbf{v}_j : j \\in \\mathcal{L}_\\mathrm{fv}\\}}$ with $\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}$ as\n%\n\\begin{subequations}\\label{eq:opt_orig}\n\t\\begin{align}\n\t\t\\min_{\\mathbf{a}\\in \\mathbb{R}^{n_{\\mathrm{e}}}\\!,\\;\n        \\mathcal{U},\\, \n        \\mathcal{V}} \n        & \\sum_{e=1}^{n_\\mathrm{e} } \\rho_e \\ell_e a_e\\label{eq:opt_orig_objective}\\\\\n\t\t\\text{s.t.}\\; & \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}} \\setminus \\mathrm{Ker}\\left( \\mathbf{M}(\\mathbf{a}) \\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\mathbf{K}_j(\\mathbf{a}) \\mathbf{v}_j}{\\mathbf{v}_j^\\mathrm{T} \\mathbf{M}_j(\\mathbf{a}) \\mathbf{v}_j} \\ge \\underline{\\lambda}_j,\\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\label{eq:opt_orig_fv}\\\\\n\t\t& \\mathbf{K}_j(\\mathbf{a}) \\mathbf{u}_j = \\mathbf{f}_j,\\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\label{eq:opt_orig_static}\\\\\n\t\t& \\mathbf{f}_j^\\mathrm{T} \\mathbf{u}_j \\le \\overline{c}_j,\\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\label{eq:opt_orig_compliance}\\\\\n\t\t& \\mathbf{a} \\ge \\mathbf{0},\\label{eq:opt_orig_nonneg}\n\t\\end{align}\n\\end{subequations}\n%\nwhere $\\mathbf{K}_j(\\mathbf{a}) \\in \\mathbb{S}^{n_{\\mathrm{dof},j}}_{\\succeq 0}$ and $\\mathbf{M}_j(\\mathbf{a}) \\in \\mathbb{S}^{n_\\mathrm{dof},j}_{\\succeq 0}$ stand for the stiffness and mass matrices, $n_{\\mathrm{dof},j}$ is the number of degrees of freedom in the $j$-th load case, and $\\mathbf{f}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}$ stands for the generalized force vector of the $j$-th load case, which includes the nodal forces and moments. In addition, $\\bm{\\ell} \\in \\mathbb{R}^{n_\\mathrm{e}}_{>0}$ and $\\bm{\\rho} \\in \\mathbb{R}^{n_\\mathrm{e}}_{>0}$ are vectors of lengths and densities of the $n_\\mathrm{e}$ elements. {Because the stiffness and mass matrices may vary across the load cases, the setting of \\eqref{eq:opt_orig} covers varying kinematic boundary conditions.}     {Notice that \\eqref{eq:opt_orig} is a non-smooth problem, as the free-vibration eigenvalue constraints \\eqref{eq:opt_orig_fv} are nondifferentiable for repeated eigenvalues.}\n\n","start":26611,"end":29690}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","labels":["eq:stiffness"],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"半正定多项式刚度","source_units":[{"raw_tex":"In bending-resistant structures such as frames, the stiffness matrix is usually a {low-degree} polynomial function of the design variables {(see Appendix \\ref{app:polydep} for more details)}, and follows from the assembly\n%\n\\begin{equation}\\label{eq:stiffness}\n\t\\mathbf{K}_j (\\mathbf{a}) = \\sum_{e=1}^{n_\\mathrm{e}} \\left[a_e \\mathbf{K}_{j,e}^{\\langle 1\\rangle} + a_e^2 \\mathbf{K}_{j,e}^{\\langle 2\\rangle} + a_e^3 \\mathbf{K}_{j,e}^{\\langle 3\\rangle}\\right],\n\\end{equation}\n%\nwhere $\\mathbf{K}_{j,e}^{\\langle i\\rangle} \\succeq 0$, $e \\in \\{1, \\dots, n_\\mathrm{e}\\}$, $i \\in \\{1,2,3\\}$, are auxiliary element stiffness matrices. In \\eqref{eq:stiffness}, the terms $ a_e \\mathbf{K}_{j,e}^{\\langle 1\\rangle}$ {typically} cover the membrane behavior, whereas the higher-order terms $a_e^2 \\mathbf{K}_{j,e}^{\\langle 2\\rangle}$ and $a_e^3 \\mathbf{K}_{j,e}^{\\langle 3\\rangle}$ capture the bending effects.\n\n","start":29690,"end":30589}]},{"id":"S007","semantic_type":"assumption","kind":"assumption","labels":["eq:mass"],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"仿射质量矩阵","source_units":[{"raw_tex":"For the mass matrix $\\mathbf{M}_j(\\mathbf{a})$, we assume the form {(again, see Appendix \\ref{app:polydep})}\n%\n\\begin{equation}\\label{eq:mass}\n\t\\mathbf{M}_j (\\mathbf{a}) = \\mathbf{M}_j^{\\langle 0\\rangle} + \\sum_{e=1}^{n_\\mathrm{e}} a_e\\mathbf{M}_{j,e}^{\\langle 1\\rangle},\n\\end{equation}\n%\nin which $a_e \\mathbf{M}_{j,e}^{\\langle 1\\rangle} \\succeq 0$, $e \\in \\{1, \\dots, n_\\mathrm{e}\\}$, are the element mass matrices and $\\mathbf{M}_j^{\\langle 0\\rangle} \\succeq 0$ constitutes a non-structural mass. In this paper, we adopt stiffness-consistent mass matrices, but any other form can be used assuming it maintains a linear dependence on the cross-section areas. An extension to nonlinear terms in $\\mathbf{M}_j(\\mathbf{a})$ might be possible but is not covered in this manuscript.\n","start":30589,"end":31369}]},{"id":"S008","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:opt_fv","eq:opt_nonneg","eq:opt_objective","eq:opt_static"],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"设计变量半定模型","source_units":[{"raw_tex":"One of the difficulties of the optimization problem \\eqref{eq:opt_orig} lies in the fact that \\eqref{eq:opt_orig_fv} and \\eqref{eq:opt_orig_static} include products of state variables $\\mathbf{u}_j$ and $\\mathbf{v}_j$ with design variables $\\mathbf{a}$. In \\cite[Proposition 2.3]{Achtziger2008}, it has been shown that the state variables $\\mathbf{u}_j$ and $\\mathbf{v}_j$ can be eliminated from the formulation, leading to an equivalent semidefinite programming reformulation in the design variables only:\n%\n\\begin{subequations}\\label{eq:opt}\n\\begin{align}\n\t\\min_{\\mathbf{a}\\in \\mathbb{R}^{n_{\\mathrm{e}}}}\\; & \\sum_{e=1}^{n_\\mathrm{e} } \\rho_e \\ell_e a_e\\label{eq:opt_objective}\\\\\n\t\\text{s.t.}\\; & \\mathbf{K}_j(\\mathbf{a}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\mathbf{a}) \\succeq 0,\\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\label{eq:opt_fv}\\\\\n\t& \\begin{pmatrix}\n\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t-\\mathbf{f}_j & \\mathbf{K}_j(\\mathbf{a})\n\t\\end{pmatrix} \\succeq 0,\\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\label{eq:opt_static}\\\\\n\t& \\mathbf{a} \\ge \\mathbf{0},\\label{eq:opt_nonneg}\n\\end{align}\n\\end{subequations}\n%\nwhere the constraints \\eqref{eq:opt_fv} and \\eqref{eq:opt_static} are matrix inequalities. These inequalities are generally nonconvex due to their polynomial dependence on $\\mathbf{a}$, and thus obtaining a (global) solution to \\eqref{eq:opt} remains challenging. {However, the non-differentiability of the free-vibration eigenvalue constraints is avoided, as the Rayleigh quotient \\eqref{eq:opt_orig_fv} is replaced by a polynomial matrix inequality \\eqref{eq:opt_fv}.}\n\nFrom the formulation \\eqref{eq:opt}, we can state the following basic properties of the feasible designs $\\hat{\\mathbf{a}}$:\n","start":31370,"end":33085}]},{"id":"S009","semantic_type":"proposition","kind":"proposition","labels":["lemma:statical_admissibility"],"member_ids":["S009"],"declared_external":true,"declaration_text":["原始前提清单明确许可 B0018；作者引用 Tyburec2021, Proposition 1。"],"source_only":false,"title":"静力可容许性","source_units":[{"raw_tex":"\\begin{proposition}[Static admissibility {\\cite[Proposition 1]{Tyburec2021}}]\\label{lemma:statical_admissibility}\n\tLet $$\\mathbf{\\hat{a}} \\in \\left\\{\\mathbf{a} \\; \\vert \\; \\begin{pmatrix}\n\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t-\\mathbf{f}_j & \\mathbf{K}_j(\\mathbf{a})\n\t\\end{pmatrix} \\succeq 0, \\mathbf{a}\\ge \\mathbf{0}\\right\\}$$ hold true. Then, $\\mathbf{f}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j(\\hat{\\mathbf{a}})\\right)$.\n\\end{proposition}\n","start":33087,"end":33535}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"固定比例整体缩放","source_units":[{"raw_tex":"In this section, we introduce a bilevel programming reformulation of the problem \\eqref{eq:opt_orig} and provide conditions for its solvability. This reformulation will further enable us to construct feasible solutions to \\eqref{eq:opt_orig}.\n\nSetting $\\mathbf{a}(\\delta) := \\delta \\tilde{\\mathbf{a}}$ with $\\tilde{\\mathbf{a}}\\ge \\mathbf{0}$, the optimization problem \\eqref{eq:opt} becomes equivalent to a bilevel optimization problem, with the upper level seeking \\textit{admissible} ratios of the cross-section areas $\\tilde{\\mathbf{a}}$ and the lower level searching for the optimal scaling factor $\\delta$ of the fixed cross-section ratios:\n","start":37715,"end":38361}]},{"id":"S011","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:bilevel-atilde","eq:bilevel-lower","eq:bilevel-membraine","eq:bilevel_mass","eq:bilevel_static"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"双层模型及下层约束","source_units":[{"raw_tex":"%\n\\begin{subequations}\\label{eq:bilevel}\n\t\\begin{align}\n\t\t\\min_{\\tilde{\\mathbf{a}}\\in \\mathbb{R}^{n_{\\mathrm{e}}}\\!,\\; \\delta^*\\in\\mathbb{R}}\\; & \\delta^* \\sum_{e=1}^{n_\\mathrm{e}} \\rho_e \\ell_e \\tilde{a}_e\\\\\n\t\t\\mathrm{s.t.}\\ & \\tilde{\\mathbf{a}} \\ge \\mathbf{0},\\label{eq:bilevel-atilde}\\\\\n\t\t& \\mathbf{f}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right), \\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\label{eq:bilevel_static}\\\\\n\t\t& \\mathrm{Im}\\left(\\mathbf{M}_{j}(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_{j}(\\tilde{\\mathbf{a}})\\right),\\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\label{eq:bilevel_mass}\\\\\n\t\t& \\mathbf{W}_j^\\mathrm{T}\\left[ \\sum_{e=1}^{n_\\mathrm{e}}\\tilde{a}_e\\mathbf{K}_{j,e}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\sum_{e=1}^{n_\\mathrm{e}} \\tilde{a}_e \\mathbf{M}_{j,e}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0, \\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\label{eq:bilevel-membraine}\\\\\n\t\t& \\begin{aligned}\\label{eq:bilevel-lower}\n\t\t\t\\delta^* \\in \\argmin_{\\delta\\in\\mathbb{R}}\\; & \\delta\\\\\n\t\t\t\\mathrm{s.t.}\\; & \\begin{pmatrix}\n\t\t\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t\t\t-\\mathbf{f}_j & \\mathbf{K}_j (\\delta \\tilde{\\mathbf{a}})\n\t\t\t\\end{pmatrix} \\succeq 0, \\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\\\\n\t\t\t& \\mathbf{K}_j(\\delta \\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\delta \\tilde{\\mathbf{a}}) \\succeq 0, \\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\\\\n\t\t\t& \\delta \\ge 0,\n\t\t\\end{aligned}\n\t\\end{align}\n\\end{subequations}\n%\nwhere $\\mathbf{W}_j$ is an orthonormal basis of $\\mathrm{Ker}\\left(\\sum_{e=1}^{n_\\mathrm{e}}\\left[\\mathbf{K}_{j,e}^{\\langle 2\\rangle} + \\mathbf{K}_{j,e}^{\\langle 3\\rangle}\\right]\\right)$ and the constraints \\eqref{eq:bilevel-atilde}-\\eqref{eq:bilevel-membraine} ensure that the lower-level problem \\eqref{eq:bilevel-lower} {has a non-empty interior}. In particular, these {constraints} include nonnegative components of the cross-sectional ratios \\eqref{eq:bilevel-atilde}, static \\eqref{eq:bilevel_static} and mass \\eqref{eq:bilevel_mass} admissibility of the ratios, and the pure membrane eigenvalues to be at least $\\underline{\\lambda}_j$ \\eqref{eq:bilevel-membraine}, which we will prove in Proposition \\ref{th:feasibility_cond} (that uses the notation set up in \\eqref{eq:hatnotation}). Moreover, the lower-level problem \\eqref{eq:bilevel-lower} is quasi-convex in $\\delta$, allowing one to obtain a globally optimal scaling $\\delta^*$ for fixed ratios $\\tilde{\\mathbf{a}}$ using bisection. Proving the quasi-convexity property and showing that the conditions \\eqref{eq:bilevel-atilde}--\\eqref{eq:bilevel-membraine} are sufficient and necessary for solvability of the lower-level problem \\eqref{eq:bilevel-lower} constitutes the main goal of this section.\n\n","start":38361,"end":41099}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["eq:RQ"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"缩放特征值的瑞利商","source_units":[{"raw_tex":"Let us first focus on the lower-level problem \\eqref{eq:bilevel-lower}. We start by expressing the lowest nonzero\n%non-singular \nfree-vibration eigenvalue function $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ using the Raleigh quotient \\cite[Definition 2.2 and Appendix A.1]{Achtziger2008} as\n%\n\\begin{equation}\\label{eq:RQ}\n\t\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) = \\min_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\left[\\delta\\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} + \\delta^2\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\delta^3\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\right]  \\mathbf{v}_j}{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\delta\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j},\n\\end{equation}\n%\n","start":41099,"end":42016}]},{"id":"S013","semantic_type":"notation","kind":"notation","labels":["eq:hatnotation"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"按次数聚合的矩阵","source_units":[{"raw_tex":"in which we adopt the simplified notation\n%\n\\begin{equation}\\label{eq:hatnotation}\n\t\\hat{\\mathbf{K}}_j^{\\langle i\\rangle}(\\tilde{\\mathbf{a}}) = \\sum_{e=1}^{n_\\mathrm{e}}\\tilde{a}_e^i\\mathbf{K}_{j,e}^{\\langle i\\rangle}, \\qquad \n\t\\hat{\\mathbf{M}}_j^{\\langle i\\rangle}(\\tilde{\\mathbf{a}}) = \\sum_{e=1}^{n_\\mathrm{e}}\\tilde{a}_e^i \\mathbf{M}_{j,e}^{\\langle i\\rangle}.\n\\end{equation}\n%\nand omit the argument of $\\hat{\\mathbf{K}}_j^{\\langle i\\rangle}$ and $\\hat{\\mathbf{M}}_j^{\\langle i\\rangle}$ \nwhen it is clear from the context, to shorten the equations.\n\n","start":42016,"end":42569}]},{"id":"S014","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"质量可容许下的正性","source_units":[{"raw_tex":"Because $\\forall j \\in \\mathcal{L}_{\\mathrm{fv}}: \\mathrm{Im}\\left(\\mathbf{M}_{j}(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_{j}(\\tilde{\\mathbf{a}})\\right)$ (see Proposition~\\ref{prop:exists_gamma_fv}), \\eqref{eq:RQ} renders that $\\lambda_{\\min,j}>0$. Thus, to prove feasibility of the eigenvalue constraint in \\eqref{eq:bilevel-lower} with respect to $\\delta$, it remains to find the condition under which $\\sup_{\\delta} \\lambda_{\\min,j} \\ge \\underline{\\lambda}_j$. \n\n","start":42569,"end":43060}]},{"id":"S015","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:RQs"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"除去缩放因子的商","source_units":[{"raw_tex":"Because of $\\delta > 0$, equation~\\eqref{eq:RQ} can further be rewritten to\n%\n\\begin{equation}\\label{eq:RQs}\n\t\\begin{multlined}\n\t\t\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) =\\\\\n\t\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}.\n\t\\end{multlined}\n\\end{equation}\n","start":43060,"end":43857}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"弯曲与膜方向分解","source_units":[{"raw_tex":"With this result, we have shown that by increasing $\\delta$ we do not decrease $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$. {Notice that this holds up to $\\delta\\rightarrow \\infty$, as we can choose $\\Delta\\rightarrow \\infty$ in the above proof.} For the solvability of the lower-level problem \\eqref{eq:bilevel-lower}, it is crucial to guarantee that there exits a $\\delta$ such that $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) \\ge \\underline{\\lambda}_j$. In other words, we require that $\\sup_{\\delta} \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) \\ge \\underline{\\lambda}_j$. To show this, let $\\mathbf{v}_j := \\mathbf{t}_j + \\mathbf{w}_j$, where $\\mathbf{t}_j \\in \\mathrm{Im}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) + \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right) $ and $\\mathbf{w}_j \\in \\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$.\nThen, we can write the expression for the supremum.\n","start":51189,"end":52352}]},{"id":"S017","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"可行重量给出设计上界","source_units":[{"raw_tex":"{Let us start with the first modification.} Since the objective function \\eqref{eq:opt_objective}, representing the weight of the structure, is a conic combination of nonnegative variables, any feasible point to the optimization problem \\eqref{eq:opt} provides an upper-bound on  the objective function value $\\overline{w}$ that also bounds the cross-section areas from above. This upper bound can be obtained by solving the convex problem \\eqref{eq:linprob}, providing the cross-sectional ratios $\\tilde{\\mathbf{a}}$, and{, subsequently,} by finding the minimal scaling factor $\\delta^*$ by solving the lower-level problem \\eqref{eq:bilevel-lower} globally using bisection. Consequently, the optimal cross-section areas are bounded as\n%\n\\begin{equation}\n\t0 \\le a_e \\le \\frac{\\overline{w}}{\\rho_e\\ell_e}, \\forall e \\in \\{1,\\dots,n_\\mathrm{e}\\}.\n\\end{equation}\n%\n","start":66732,"end":67594}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"区间缩放与重量限制","source_units":[{"raw_tex":"Furthermore, using the substitution $a_e = \\frac{\\overline{w}}{2\\rho_e\\ell_e} (a_{\\mathrm{s},e} + 1)$, these bounds can be written as $a_{\\mathrm{s},e}^2 \\le 1$. %Similarly, we add a weight constraint to strengthen the relaxations as\n","start":67594,"end":67828},{"raw_tex":"Similarly, we add a weight constraint $\\sum_{e=1}^{n_\\mathrm{e}} \\rho_e \\ell_e a_e \\le \\overline{w}$ to strengthen the relaxations.\nThe influence of this constraint on the feasible set of relaxations will be illustrated in Section~\\ref{sec:illustrative}. Using the above substitution of $a_e$, we obtain the formulation\n","start":68026,"end":68346}]},{"id":"S020","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:opt_fv_po","eq:opt_nonneg_po","eq:opt_objective_po","eq:opt_static_po","eq:opt_weight_po"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"紧化后的多项式模型","source_units":[{"raw_tex":"%\n\\begin{subequations}\\label{eq:opt_po}\n\t\\begin{align}\n\t\t\\min_{\\mathbf{a}_{{\\mathrm{s}}}\\in\\mathbb{R}^{n_\\mathrm{e}}}\\; & {\\frac{1}{2}} \\overline{w} (n_\\mathrm{e} + \\mathbf{1}^\\mathrm{T} \\mathbf{a}_\\mathrm{s})\\label{eq:opt_objective_po}\\\\\n\t\t\\text{s.t.}\\; & \\mathbf{K}_j(\\mathbf{a}_\\mathrm{s}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\mathbf{a}_\\mathrm{s}) \\succeq 0, \\quad j \\in {\\mathcal{L}_\\mathrm{fv}}\\label{eq:opt_fv_po}\\\\\n\t\t& \\begin{pmatrix}\n\t\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t\t-\\mathbf{f}_j & \\mathbf{K}_j(\\mathbf{a}_\\mathrm{s})\n\t\t\\end{pmatrix} \\succeq 0,\\quad j \\in {\\mathcal{L}_\\mathrm{s}}\\label{eq:opt_static_po}\\\\\n            & 2-n_\\mathrm{e} - \\sum_{e=1}^{n_\\mathrm{e}} a_{\\mathrm{s},e} \\ge 0,\\label{eq:opt_weight_po}\\\\\n\t\t& 1 - a_\\mathrm{s,e}^2 \\ge 0, \\quad e \\in \\{1,\\dots,n_\\mathrm{e}\\}\\label{eq:opt_nonneg_po}\n\t\\end{align}\n\\end{subequations}\n%\n{in the scaled variables $\\mathbf{a}_{\\mathrm{s}} \\in \\mathbb{R}^{n_\\mathrm{e}}$. Here,} we slightly abuse the notation and express the stiffness and mass matrices in terms of the scaled variables $\\mathbf{a}_\\mathrm{s}$ as $\\mathbf{K}_j(\\mathbf{a}_\\mathrm{s})$ and $\\mathbf{M}_j(\\mathbf{a}_\\mathrm{s})$, respectively. {Although this compactification means that \\eqref{eq:opt} and \\eqref{eq:opt_po} do not have the same feasible set (after rescaling), they maintain the same set of optimal solutions. Thus, solution to \\eqref{eq:opt_po} also solves \\eqref{eq:opt}.}\n","start":68346,"end":69781}]},{"id":"S022","semantic_type":"definition","kind":"definition","labels":["eq:rel_a_1","eq:rel_a_2","eq:rel_a_3"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"未缩放单项式的提升量","source_units":[{"raw_tex":"%\n\\begin{subequations}\n\t\\begin{align}\n\t\ta_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}) &= 0.5\\,{\\frac{\\overline{w}}{\\rho_e \\ell_e}} (y_{a_{\\mathrm{s},e}^1} + 1),\\label{eq:rel_a_1}\\\\\n\t\ta_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}) &= 0.25\\, {\\frac{\\overline{w}^2}{\\rho_e^2 \\ell_e^2}} \\left( y_{a_{\\mathrm{s},e}^2} + 2 y_{a_{\\mathrm{s},e}^1} + 1 \\right),\\label{eq:rel_a_2}\\\\\n\t\ta_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}) &= 0.125\\, {\\frac{\\overline{w}^3}{\\rho_e^3 \\ell_e^3}} \\left( y_{a_{\\mathrm{s},e}^3} + 3 y_{a_{\\mathrm{s},e}^2} + 3 y_{a_{\\mathrm{s},e}^1} + 1 \\right)\\label{eq:rel_a_3}\n\t\\end{align}\n\\end{subequations}\n%\nfor evaluating {expected values of the} monomials of the (unscaled) cross-section areas based on the (scaled) moments $\\mathbf{y}$. In this notation, the superscript $\\bullet^{\\langle i\\rangle}$ indicates that it corresponds to the $i$-th degree monomial of $\\bullet$. We wish to emphasize here that, generally, $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}) \\neq \\left[a_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y})\\right]^2$ and $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}) \\neq \\left[a_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y})\\right]^3$ due to the relaxation procedure.\n\n","start":71734,"end":72962}]},{"id":"S024","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S024"],"declared_external":true,"declaration_text":["原始前提清单声明的标准半正定矩阵事实；谱性质部分为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"半正定矩阵基本性质","source_units":[]},{"id":"S025","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S025"],"declared_external":true,"declaration_text":["原始前提清单声明的半正定矩阵关于和、核与值域的标准事实。"],"source_only":false,"title":"半正定和的值域性质","source_units":[]},{"id":"S026","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S026"],"declared_external":true,"declaration_text":["原始前提清单及第376行引用：Achtziger2008, Definition 2.2 and Appendix A.1。"],"source_only":false,"title":"广义瑞利商表征","source_units":[]},{"id":"S027","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S027"],"declared_external":true,"declaration_text":["原始前提清单及第542行引用：Tyburec2023, Proposition 5。"],"source_only":false,"title":"柔度约束的有限缩放","source_units":[]},{"id":"S028","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S028"],"declared_external":true,"declaration_text":["原始前提清单及第545行引用：Tyburec2023, Proposition 1。"],"source_only":false,"title":"柔度随缩放不增","source_units":[]},{"id":"S029","semantic_type":"proposition","kind":"proposition","labels":["lemma:dynamic_admissibility"],"member_ids":["S029","S030","S031","S032"],"declared_external":false,"source_only":false,"title":"质量可容许性","source_units":[{"raw_tex":"\\begin{proposition}[Mass admissibility]\\label{lemma:dynamic_admissibility}\n\tLet $$\\mathbf{\\hat{a}}\\in\\left\\{\\mathbf{a} \\; \\vert \\; \\mathbf{K}_j(\\mathbf{a}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\mathbf{a}) \\succeq 0, \\mathbf{a}\\ge \\mathbf{0}\\right\\}.$$ Then, $\\mathrm{Im}\\left(\\mathbf{M}_j(\\hat{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\hat{\\mathbf{a}})\\right)$.\n\\end{proposition}","start":33537,"end":33935},{"raw_tex":"\\begin{proof}\n\tThe matrix inequality \\eqref{eq:opt_fv} is equivalent to\n\t%\n\t\\begin{equation}\\label{eq:inimage}\n\t\t\\mathbf{v}_j^\\mathrm{T}\\mathbf{K}_j(\\mathbf{a})\\mathbf{v}_j - \\underline{\\lambda}_j \\mathbf{v}_j^\\mathrm{T}\\mathbf{M}_j(\\mathbf{a})\\mathbf{v}_j\\ge 0\\quad \\forall \\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}.\n\t\\end{equation}\n\t%\n\tLet us now proceed by contradiction and assume that $\\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right) \\nsubseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\mathbf{a})\\right)$. Then, there must exist $\\hat{\\mathbf{v}}_j\\in\\left\\{\\mathbf{v}_j\\;\\vert\\; \\mathbf{v}_j\\in \\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right), \\mathbf{v}_j\\notin \\mathrm{Im}\\left(\\mathbf{K}_j(\\mathbf{a})\\right)\\right\\}$, for which the left-hand side of \\eqref{eq:inimage} evaluates as\n\t%\n\t\\begin{equation}\n\t\t-\\underline{\\lambda}_j \\hat{\\mathbf{v}}_j^\\mathrm{T}\\mathbf{M}_j(\\mathbf{a})\\hat{\\mathbf{v}}_j < 0,\n\t\\end{equation}\n\t%\n\twhich is strictly negative because $\\underline{\\lambda}_j>0$, $\\mathbf{M}_j(\\mathbf{a}) \\succeq 0$, and $\\hat{\\mathbf{v}}_j\\in \\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right)$, contradicting~\\eqref{eq:inimage}.\n\\end{proof}","start":33936,"end":35093},{"raw_tex":"\tThe matrix inequality \\eqref{eq:opt_fv} is equivalent to\n\t%\n\t\\begin{equation}\\label{eq:inimage}\n\t\t\\mathbf{v}_j^\\mathrm{T}\\mathbf{K}_j(\\mathbf{a})\\mathbf{v}_j - \\underline{\\lambda}_j \\mathbf{v}_j^\\mathrm{T}\\mathbf{M}_j(\\mathbf{a})\\mathbf{v}_j\\ge 0\\quad \\forall \\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}.\n\t\\end{equation}\n","start":33950,"end":34277},{"raw_tex":"\tLet us now proceed by contradiction and assume that $\\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right) \\nsubseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\mathbf{a})\\right)$. Then, there must exist $\\hat{\\mathbf{v}}_j\\in\\left\\{\\mathbf{v}_j\\;\\vert\\; \\mathbf{v}_j\\in \\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right), \\mathbf{v}_j\\notin \\mathrm{Im}\\left(\\mathbf{K}_j(\\mathbf{a})\\right)\\right\\}$, for which the left-hand side of \\eqref{eq:inimage} evaluates as\n\t%\n\t\\begin{equation}\n\t\t-\\underline{\\lambda}_j \\hat{\\mathbf{v}}_j^\\mathrm{T}\\mathbf{M}_j(\\mathbf{a})\\hat{\\mathbf{v}}_j < 0,\n\t\\end{equation}\n\t%\n\twhich is strictly negative because $\\underline{\\lambda}_j>0$, $\\mathbf{M}_j(\\mathbf{a}) \\succeq 0$, and $\\hat{\\mathbf{v}}_j\\in \\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right)$, contradicting~\\eqref{eq:inimage}.\n","start":34280,"end":35082}]},{"id":"S033","semantic_type":"proposition","kind":"proposition","labels":["prop:exists_gamma_fv"],"member_ids":["S033","S034"],"declared_external":false,"source_only":false,"title":"正特征值阈值存在","source_units":[{"raw_tex":"\\begin{proposition}[{Positive finite eigenvalue}]\\label{prop:exists_gamma_fv}\n\tLet $\\mathrm{Im}(\\mathbf{M}_j(\\mathbf{a})) \\subseteq \\mathrm{Im}(\\mathbf{K}_j(\\mathbf{a}))$. Then, there exists $\\hat{\\lambda}>0$ such that\n\t%\n\t$\\mathbf{K}_j(\\mathbf{a}) - \\hat{\\lambda} \\mathbf{M}_j(\\mathbf{a})\\succeq 0$.\n\\end{proposition}","start":36743,"end":37061},{"raw_tex":"\\begin{proof}\nSince $\\mathbf{K}_j(\\mathbf{a}) \\in \\mathbb{S}^{{n_\\mathrm{dof},j}}_{\\succeq 0}$ and $\\mathbf{M}_j(\\mathbf{a}) \\in \\mathbb{S}^{{n_\\mathrm{dof},j}}_{\\succeq 0}$, the nonzero eigenvalues of the generalized eigenvalue problem $\\mathbf{K}_j(\\mathbf{a})\\mathbf{v}_j - \\lambda \\mathbf{M}_j(\\mathbf{a}) \\mathbf{v}_j = \\mathbf{0}$, with $\\mathbf{v}_j \\in \\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right)$, must be real and positive. Thus, setting $0<\\hat{\\lambda}= \\lambda_1$ with $\\lambda_1$ being the lowest nonzero eigenvalue, the eigenvalue constraint is satisfied.\n\\end{proof}","start":37062,"end":37649}]},{"id":"S035","semantic_type":"proposition","kind":"proposition","labels":["prop:monotonic"],"member_ids":["S035","S036","S037","S038","S039"],"declared_external":false,"source_only":false,"title":"缩放特征值单调不减","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:monotonic}\nLet $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ be as defined in \\eqref{eq:RQs} and $j \\in \\mathcal{L}_\\mathrm{fv}$. Then, $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ is a non-decreasing function.\n","start":44422,"end":44665},{"raw_tex":"\\begin{proof}\nLet $\\Delta > 0$; we need to show that $\\lambda_{\\min,j}((\\delta+\\Delta) \\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})\\ge 0$. Using the definition \\eqref{eq:RQs}, we have %\\textcolor{purple}{ We don't really need to write equations (3.11), it is clear enough to directly proceed to (3.12). }\n\\begin{multline}\n\t\\lambda_{\\min,j}((\\delta + \\Delta)\\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})=\\\\\n\t%\n\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + (\\delta+\\Delta) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + (\\delta+\\Delta)^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta }\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\\\\n\t%\n\t- \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}.\n\\end{multline}\n%\nSplitting the first infimum into two parts using the triangle inequality, we obtain \n%\n\\begin{multline}\\label{eq:lamdiff}\n\t\\lambda_{\\min,j}((\\delta + \\Delta)\\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})\\ge\\\\\n\t%\n\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\\\\n\t%\n\t+ \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\Delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + (2\\delta \\Delta + \\Delta^2) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}\\\\\n\t%\n\t- \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}.\n\\end{multline}\n%\nBecause $\\forall \\delta,\\Delta >0: \\frac{1}{\\delta} \\ge \\frac{1}{\\delta + \\Delta}$, it holds for all $\\mathbf{v}_j$, including the minimizing ones, that\n\\begin{equation}\n\t\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j \\ge \\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j.\n\\end{equation}\n%\nConsequently,\n% \n\\begin{multline}\n\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\\\\n\t- \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\ge 0,\n\\end{multline}\n%\nwhich allows us to simplify \\eqref{eq:lamdiff} to\n%\n\\begin{equation}\\label{eq:monotoneous}\n\\begin{multlined}\n\\lambda_{\\min,j}((\\delta + \\Delta)\\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) \\ge\\\\\n \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\\frac{\\Delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + (2\\delta \\Delta + \\Delta^2) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\ge 0.\n\\end{multlined}\n\\end{equation}\n%\nIn \\eqref{eq:monotoneous}, the nonnegativity follows from $\\Delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j \\ge 0$ and $\\left(2\\delta \\Delta + \\Delta^2\\right) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j\\ge 0$, which is because of $\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}, \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\succeq 0$ and $\\delta\\ge 0, \\Delta > 0$.\n\\end{proof}\n","start":44665,"end":51170},{"raw_tex":"Let $\\Delta > 0$; we need to show that $\\lambda_{\\min,j}((\\delta+\\Delta) \\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})\\ge 0$. Using the definition \\eqref{eq:RQs}, we have %\\textcolor{purple}{ We don't really need to write equations (3.11), it is clear enough to directly proceed to (3.12). }\n\\begin{multline}\n\t\\lambda_{\\min,j}((\\delta + \\Delta)\\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})=\\\\\n\t%\n\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + (\\delta+\\Delta) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + (\\delta+\\Delta)^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta }\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\\\\n\t%\n\t- \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}.\n\\end{multline}\n%\nSplitting the first infimum into two parts using the triangle inequality, we obtain \n%\n\\begin{multline}\\label{eq:lamdiff}\n\t\\lambda_{\\min,j}((\\delta + \\Delta)\\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})\\ge\\\\\n\t%\n\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\\\\n\t%\n\t+ \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\Delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + (2\\delta \\Delta + \\Delta^2) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}\\\\\n\t%\n\t- \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j}.\n\\end{multline}\n","start":44679,"end":48323},{"raw_tex":"Because $\\forall \\delta,\\Delta >0: \\frac{1}{\\delta} \\ge \\frac{1}{\\delta + \\Delta}$, it holds for all $\\mathbf{v}_j$, including the minimizing ones, that\n\\begin{equation}\n\t\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j \\ge \\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j.\n\\end{equation}\n%\nConsequently,\n% \n\\begin{multline}\n\t\\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\t\\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\\\\n\t- \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{v}_j + \\delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + \\delta^2 \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\ge 0,\n\\end{multline}\n","start":48325,"end":49951},{"raw_tex":"which allows us to simplify \\eqref{eq:lamdiff} to\n%\n\\begin{equation}\\label{eq:monotoneous}\n\\begin{multlined}\n\\lambda_{\\min,j}((\\delta + \\Delta)\\tilde{\\mathbf{a}}) - \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) \\ge\\\\\n \\inf_{\\mathbf{v}_j \\in \\mathbb{R}^{n_{\\mathrm{dof},j}}\\setminus \\mathrm{Ker}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle}+\\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \n\\frac{\\Delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j + (2\\delta \\Delta + \\Delta^2) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j}{\\frac{1}{\\delta + \\Delta}\\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{v}_j + \\mathbf{v}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{v}_j} \\ge 0.\n\\end{multlined}\n\\end{equation}\n%\nIn \\eqref{eq:monotoneous}, the nonnegativity follows from $\\Delta \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} \\mathbf{v}_j \\ge 0$ and $\\left(2\\delta \\Delta + \\Delta^2\\right) \\mathbf{v}_j^\\mathrm{T}\\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\mathbf{v}_j\\ge 0$, which is because of $\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}, \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle} \\succeq 0$ and $\\delta\\ge 0, \\Delta > 0$.\n","start":49953,"end":51158},{"raw_tex":"\\end{proposition}","start":51170,"end":51187}]},{"id":"S040","semantic_type":"proposition","kind":"proposition","labels":["eq:supremum","prop:supremum"],"member_ids":["S040","S041","S042","S043"],"declared_external":false,"source_only":false,"title":"特征值上确界的膜表达","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:supremum}\nLet $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ be as defined in \\eqref{eq:RQs}, and let $\\mathbf{v}_j := \\mathbf{t}_j + \\mathbf{w}_j$ as defined above. Then, \n\\begin{equation}\\label{eq:supremum}\n\t\\sup_{\\delta} \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) = \\inf_{\\mathbf{w}_j \\in \\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{w}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\mathbf{w}_j}{\\mathbf{w}_j^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{w}_j}.\n\\end{equation}\n","start":52353,"end":53060},{"raw_tex":"\\begin{proof}\nWe prove the statement by splitting $\\mathbf{v}_j$ into $\\mathbf{t}_j$ and $\\mathbf{w}_j$ and showing that it must hold that $\\mathbf{t}_j=\\mathbf{0}$.  For this splitting, the supremum is expressed as\n\\begin{equation}\\label{eq:splitting}\n\t\\begin{multlined}\n\t\t\\sup_\\delta \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})= \\\\\n\t\t\\sup_\\delta \\inf\\limits_{\\mathbf{t}_j, \\mathbf{w}_j}\n\t\t \\frac{\\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\left(\\mathbf{t}_j+\\mathbf{w}_j\\right) + \\delta\\mathbf{t}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\mathbf{t}_j + \\delta^2\\mathbf{t}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\mathbf{t}_j }{\\frac{1}{\\delta}\\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\left(\\mathbf{t}_j+\\mathbf{w}_j\\right) + \\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)}.\n\t\\end{multlined}\n\\end{equation}\n%\nBecause of the monotonicity (recall Proposition~\\ref{prop:monotonic}), $\\sup_\\delta \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) = \\lim\\limits_{\\delta\\rightarrow \\infty}\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$. Further, suppose by {contradiction} that $\\mathbf{t}_j \\neq \\mathbf{0}$ holds at the infimum. Then, $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) \\rightarrow \\infty$ as $\\delta \\rightarrow \\infty$, due to $\\mathbf{w}_j \\in \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$ and because $\\mathbf{t}_j$ cannot be in $\\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right)$. On the other hand, for $\\mathbf{t}_j = \\mathbf{0}$, $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ is finite for all $\\delta$ due to $\\mathbf{w}_j \\in \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$. Consequently, $\\mathbf{t}_j^*=\\mathbf{0}$ must hold at the infimum. After inserting $\\mathbf{t}_j^*=\\mathbf{0}$ and $\\delta^*\\rightarrow \\infty$ into \\eqref{eq:splitting}, we receive \\eqref{eq:supremum}, which completes the proof.\n\\end{proof}\n","start":53060,"end":55242},{"raw_tex":"We prove the statement by splitting $\\mathbf{v}_j$ into $\\mathbf{t}_j$ and $\\mathbf{w}_j$ and showing that it must hold that $\\mathbf{t}_j=\\mathbf{0}$.  For this splitting, the supremum is expressed as\n\\begin{equation}\\label{eq:splitting}\n\t\\begin{multlined}\n\t\t\\sup_\\delta \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})= \\\\\n\t\t\\sup_\\delta \\inf\\limits_{\\mathbf{t}_j, \\mathbf{w}_j}\n\t\t \\frac{\\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}\\left(\\mathbf{t}_j+\\mathbf{w}_j\\right) + \\delta\\mathbf{t}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\mathbf{t}_j + \\delta^2\\mathbf{t}_j^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\mathbf{t}_j }{\\frac{1}{\\delta}\\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\left(\\mathbf{t}_j+\\mathbf{w}_j\\right) + \\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\left(\\mathbf{t}_j+\\mathbf{w}_j\\right)}.\n\t\\end{multlined}\n\\end{equation}\n","start":53074,"end":54058},{"raw_tex":"Because of the monotonicity (recall Proposition~\\ref{prop:monotonic}), $\\sup_\\delta \\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) = \\lim\\limits_{\\delta\\rightarrow \\infty}\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$. Further, suppose by {contradiction} that $\\mathbf{t}_j \\neq \\mathbf{0}$ holds at the infimum. Then, $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}}) \\rightarrow \\infty$ as $\\delta \\rightarrow \\infty$, due to $\\mathbf{w}_j \\in \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$ and because $\\mathbf{t}_j$ cannot be in $\\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right)$. On the other hand, for $\\mathbf{t}_j = \\mathbf{0}$, $\\lambda_{\\min,j}(\\delta \\tilde{\\mathbf{a}})$ is finite for all $\\delta$ due to $\\mathbf{w}_j \\in \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$. Consequently, $\\mathbf{t}_j^*=\\mathbf{0}$ must hold at the infimum. After inserting $\\mathbf{t}_j^*=\\mathbf{0}$ and $\\delta^*\\rightarrow \\infty$ into \\eqref{eq:splitting}, we receive \\eqref{eq:supremum}, which completes the proof.\n","start":54060,"end":55230},{"raw_tex":"\\end{proposition}","start":55242,"end":55259}]},{"id":"S044","semantic_type":"proposition","kind":"proposition","labels":["prop:scaling"],"member_ids":["S044","S045","S046","S047","S048"],"declared_external":false,"source_only":false,"title":"振动约束的缩放条件","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:scaling}\n\tThere exists $\\delta$ such that $\\mathbf{K}_j(\\delta \\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\delta \\tilde{\\mathbf{a}}) \\succeq 0$ if and only if $\\tilde{\\mathbf{a}} \\in \\left\\{\\mathbf{a} \\;\\vert\\; \\mathbf{a}\\ge \\mathbf{0}, \\mathrm{Im}\\left(\\mathbf{M}_j(\\mathbf{a})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\mathbf{a})\\right),\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0\\right\\}$, where $\\mathbf{W}_j$ denotes an orthonormal basis of $\\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$.\n\\end{proposition}","start":55919,"end":56765},{"raw_tex":"\\begin{proof}\n\\fbox{$\\Rightarrow$} Let $\\mathbf{K}_j(\\delta \\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\delta \\tilde{\\mathbf{a}}) \\succeq 0$ hold. Then, $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ based on Proposition \\ref{lemma:dynamic_admissibility}. Next, we project the matrix inequality using $\\mathbf{W}_j$, yielding\n%\n\\begin{equation}\n\t\\delta \\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} \\mathbf{W}_j- \\underline{\\lambda}_j \\delta \\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{W}_j - \\underline{\\lambda}_j\\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{W}_j \\succeq 0.\n\\end{equation}\n%\nFor $\\delta>0$, this is further equivalent to\n%\n\\begin{equation}\n\t\\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} \\mathbf{W}_j- \\underline{\\lambda}_j \\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{W}_j \\succeq \\frac{1}{\\delta}\\underline{\\lambda}_j\\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{W}_j \\succeq 0.\n\\end{equation}\n%\n\\fbox{$\\Leftarrow$} Let $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ and $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$ hold. Because of the first assumption and Proposition \\ref{prop:exists_gamma_fv}, it holds that\n%\n\\begin{equation}\n\\forall \\mathbf{v}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right) \\exists \\gamma>0: \\mathbf{v}_j^\\mathrm{T}\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\mathbf{v}_j - \\gamma\\mathbf{v}_j^\\mathrm{T}\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\mathbf{v}_j \\ge 0,\n\\end{equation}\n%\nand thus also $\\mathbf{K}_j(\\tilde{\\mathbf{a}}) - \\gamma\\mathbf{M}_j(\\tilde{\\mathbf{a}}) \\succeq 0$ for $\\gamma > 0$. Consequently, it remains to show that $\\gamma\\ge \\underline{\\lambda}_j$.\n\nBecause of $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$, we also have\n%\n\\begin{equation}\\label{eq:inequality}\n\t\\forall \\mathbf{w}_j \\in \\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right):  \\mathbf{w}_j^\\mathrm{T} \\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right] \\mathbf{w}_j \\ge 0.\n\\end{equation}\n%\nIn particular, the above holds for $w_j$ minimizing the left-hand side of \\eqref{eq:inequality}, yielding\n%\n\\begin{equation}\n\t\\underline{\\lambda}_j \\le \\inf_{\\mathbf{w}_j \\in \\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{w}_j^\\mathrm{T} \\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} \\mathbf{w}_j}{\\mathbf{w}_j^\\mathrm{T} \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\mathbf{w}_j} = \\sup_{\\delta} \\lambda_{\\min,j}(\\mathbf{a}(\\delta))\n\\end{equation}\n%\nbased on Proposition \\ref{prop:supremum}.\n\\end{proof}","start":56766,"end":60141},{"raw_tex":"\\fbox{$\\Rightarrow$} Let $\\mathbf{K}_j(\\delta \\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\delta \\tilde{\\mathbf{a}}) \\succeq 0$ hold. Then, $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ based on Proposition \\ref{lemma:dynamic_admissibility}. Next, we project the matrix inequality using $\\mathbf{W}_j$, yielding\n%\n\\begin{equation}\n\t\\delta \\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} \\mathbf{W}_j- \\underline{\\lambda}_j \\delta \\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{W}_j - \\underline{\\lambda}_j\\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{W}_j \\succeq 0.\n\\end{equation}\n%\nFor $\\delta>0$, this is further equivalent to\n%\n\\begin{equation}\n\t\\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} \\mathbf{W}_j- \\underline{\\lambda}_j \\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle} \\mathbf{W}_j \\succeq \\frac{1}{\\delta}\\underline{\\lambda}_j\\mathbf{W}_{{j}}^\\mathrm{T} \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} \\mathbf{W}_j \\succeq 0.\n\\end{equation}\n","start":56780,"end":57927},{"raw_tex":"\\fbox{$\\Leftarrow$} Let $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ and $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$ hold. Because of the first assumption and Proposition \\ref{prop:exists_gamma_fv}, it holds that\n%\n\\begin{equation}\n\\forall \\mathbf{v}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right) \\exists \\gamma>0: \\mathbf{v}_j^\\mathrm{T}\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\mathbf{v}_j - \\gamma\\mathbf{v}_j^\\mathrm{T}\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\mathbf{v}_j \\ge 0,\n\\end{equation}\n%\nand thus also $\\mathbf{K}_j(\\tilde{\\mathbf{a}}) - \\gamma\\mathbf{M}_j(\\tilde{\\mathbf{a}}) \\succeq 0$ for $\\gamma > 0$. Consequently, it remains to show that $\\gamma\\ge \\underline{\\lambda}_j$.\n","start":57929,"end":58822},{"raw_tex":"Because of $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$, we also have\n%\n\\begin{equation}\\label{eq:inequality}\n\t\\forall \\mathbf{w}_j \\in \\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right):  \\mathbf{w}_j^\\mathrm{T} \\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right] \\mathbf{w}_j \\ge 0.\n\\end{equation}\n%\nIn particular, the above holds for $w_j$ minimizing the left-hand side of \\eqref{eq:inequality}, yielding\n%\n\\begin{equation}\n\t\\underline{\\lambda}_j \\le \\inf_{\\mathbf{w}_j \\in \\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)} \\frac{\\mathbf{w}_j^\\mathrm{T} \\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} \\mathbf{w}_j}{\\mathbf{w}_j^\\mathrm{T} \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\mathbf{w}_j} = \\sup_{\\delta} \\lambda_{\\min,j}(\\mathbf{a}(\\delta))\n\\end{equation}\n%\nbased on Proposition \\ref{prop:supremum}.\n","start":58823,"end":60130}]},{"id":"S049","semantic_type":"proposition","kind":"proposition","labels":["th:feasibility_cond"],"member_ids":["S049","S050","S051","S052","S053"],"declared_external":false,"source_only":false,"title":"下层问题的可行性条件","source_units":[{"raw_tex":"\\begin{proposition}\\label{th:feasibility_cond}\n    The optimization problem \\eqref{eq:bilevel-lower} has a non-empty interior if and only if $\\tilde{\\mathbf{a}} \\in \\mathbb{R}^{n_{\\mathrm{e}}}$ satisfies {the following feasibility conditions:}\n    {\\begin{itemize}\n    \\item $\\tilde{\\mathbf{a}} \\ge \\mathbf{0}$,\n    \\item $\\mathbf{f}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j (\\tilde{\\mathbf{a}})\\right)$ for all $j \\in \\mathcal{L}_\\mathrm{s}$,\n    \\item $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ for all $j \\in \\mathcal{L}_\\mathrm{fv}$,\n    \\item $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$ for all $j \\in \\mathcal{L}_\\mathrm{fv}$,\n    \\end{itemize}}%\n    %\n    \\noindent where $\\mathbf{W}_j$ is an orthonormal basis of $\\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$.\n\\end{proposition}","start":60363,"end":61513},{"raw_tex":"\\begin{proof}\n\tBased on \\cite[Proposition 5]{Tyburec2023}, for $\\mathbf{f}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j (\\tilde{\\mathbf{a}})\\right)$ there exists a finite $\\delta_{\\mathrm{c},j}^*>0$ such that the static matrix inequality in \\eqref{eq:bilevel-lower} is satisfied. Conversely, for $\\mathbf{f}_j \\notin \\mathrm{Im}\\left(\\mathbf{K}_j (\\tilde{\\mathbf{a}})\\right)$, the optimization problem \\eqref{eq:bilevel-lower} is infeasible.\n\t\n\tUsing Proposition \\ref{prop:scaling}, there exists $\\delta_{\\mathrm{fv},j}^*>0$ such that the free-vibration constraint is satisfied if and only if $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ and $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$.\n    Finally, there exists $\\delta^* = \\max\\{\\max_{j{\\in \\mathcal{L}_\\mathrm{s}}} \\delta^*_{\\mathrm{c},j}, \\max_{j{\\in \\mathcal{L}_\\mathrm{fv}}} \\delta^*_{\\mathrm{fv},j} \\}>0$ such that the free-vibration and compliance constraints are satisfied if and only if the {right-hand-side} conditions of the proposition are satisfied. This is (i) because we have upper bounds for the compliance $\\overline{c}_j$ and compliance is a nonincreasing function in $\\delta$ \\cite[Proposition 1]{Tyburec2023}, and (ii) because we have lower bounds for the lowest free-vibration eigenvalues $\\underline{\\lambda}_j$ and the lowest free-vibration eigenvalue function is a nondecreasing function in $\\delta$ (Proposition \\ref{prop:monotonic}).\n\\end{proof}","start":61514,"end":63129},{"raw_tex":"\tBased on \\cite[Proposition 5]{Tyburec2023}, for $\\mathbf{f}_j \\in \\mathrm{Im}\\left(\\mathbf{K}_j (\\tilde{\\mathbf{a}})\\right)$ there exists a finite $\\delta_{\\mathrm{c},j}^*>0$ such that the static matrix inequality in \\eqref{eq:bilevel-lower} is satisfied. Conversely, for $\\mathbf{f}_j \\notin \\mathrm{Im}\\left(\\mathbf{K}_j (\\tilde{\\mathbf{a}})\\right)$, the optimization problem \\eqref{eq:bilevel-lower} is infeasible.\n","start":61528,"end":61947},{"raw_tex":"\tUsing Proposition \\ref{prop:scaling}, there exists $\\delta_{\\mathrm{fv},j}^*>0$ such that the free-vibration constraint is satisfied if and only if $\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right)$ and $\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle} - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle} \\right]\\mathbf{W}_j \\succeq 0$.\n","start":61949,"end":62394},{"raw_tex":"    Finally, there exists $\\delta^* = \\max\\{\\max_{j{\\in \\mathcal{L}_\\mathrm{s}}} \\delta^*_{\\mathrm{c},j}, \\max_{j{\\in \\mathcal{L}_\\mathrm{fv}}} \\delta^*_{\\mathrm{fv},j} \\}>0$ such that the free-vibration and compliance constraints are satisfied if and only if the {right-hand-side} conditions of the proposition are satisfied. This is (i) because we have upper bounds for the compliance $\\overline{c}_j$ and compliance is a nonincreasing function in $\\delta$ \\cite[Proposition 1]{Tyburec2023}, and (ii) because we have lower bounds for the lowest free-vibration eigenvalues $\\underline{\\lambda}_j$ and the lowest free-vibration eigenvalue function is a nondecreasing function in $\\delta$ (Proposition \\ref{prop:monotonic}).\n","start":62394,"end":63118}]},{"id":"S054","semantic_type":"proposition","kind":"proposition","labels":["prop:inertia"],"member_ids":["S054","S055","S056","S057","S058"],"declared_external":false,"source_only":false,"title":"高次正矩迫使一阶为正","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:inertia}\n\tLet $\\mathbf{y}^{*(r)}$ be the optimal moments associated with the cross-section areas $\\mathbf{a}$ obtained by solving the $r$-th degree relaxation. If $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)}) > 0$ or $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)}) > 0$, then $a_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}^{*(r)}) > 0$.\n\t%\n","start":73554,"end":73950},{"raw_tex":"\t\\begin{proof}\n        By contradiction, let us assume that $a_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}^{*(r)}) = 0$, i.e., $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$. Then, we need to show that $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)}) = 0$ and $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)}) = 0$.\n\t\t\n\t\tConsider first the case of $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)})$. The moment matrix {(recall the definition in \\eqref{eq:moment})} then contains the principal submatrix\n\t\t%\n\t\t\\begin{equation}\n\t\t\t\\begin{pmatrix}\n\t\t\t\t1 & y_{a_{\\mathrm{s},e}^1}^{*(r)}\\\\\n\t\t\t\ty_{a_{\\mathrm{s},e}^1}^{*(r)} & y_{a_{\\mathrm{s},e}^2}^{*(r)}\n\t\t\t\\end{pmatrix} \\succeq 0.\n\t\t\\end{equation}\n\t\t%\n\t\tThe determinant of this matrix must be nonnegative, so that we obtain the inequality\n\t\t%\n\t\t\\begin{equation}\n\t\t\ty_{a_{\\mathrm{s},e}^2}^{*(r)} - \\left[y_{a_{\\mathrm{s},e}^1}^{*(r)}\\right]^2 \\ge 0.\n\t\t\\end{equation}\n\t\t%\n\t\tAfter inserting $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$, we obtain $y_{a_{\\mathrm{s},e}^2}^{*(r)} \\ge 1$. Due to the bound constraints \\eqref{eq:opt_nonneg_po}, $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$. Using $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$ with $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$, \\eqref{eq:rel_a_2} is then evaluated as $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)})=0$.\n\t\t\n\t\tFor the case of, $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)})$, the moment matrix contains the principal submatrix\n\t\t%\n\t\t\\begin{equation}\n\t\t\t\\begin{pmatrix}\n\t\t\t\t1 & y_{a_{\\mathrm{s},e}^1}^{*(r)} & y_{a_{\\mathrm{s},e}^2}^{*(r)}\\\\\n\t\t\t\ty_{a_{\\mathrm{s},e}^1}^{*(r)} & y_{a_{\\mathrm{s},e}^2}^{*(r)} & y_{a_{\\mathrm{s},e}^3}^{*(r)}\\\\\n\t\t\t\ty_{a_{\\mathrm{s},e}^2}^{*(r)} & y_{a_{\\mathrm{s},e}^3}^{*(r)} & y_{a_{\\mathrm{s},e}^4}^{*(r)}\n\t\t\t\\end{pmatrix} \\succeq 0.\n\t\t\\end{equation}\n\t\t%\n\t\tAfter inserting $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$ and $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$, the determinant evaluates as\n\t\t%\n\t\t\\begin{equation}\n\t\t\t- \\left(y_{a_{\\mathrm{s},e}^3}^{*(r)} + 1\\right)^2 \\ge 0,\n\t\t\\end{equation}\n\t\t%\n\t\tso that $y_{a_{\\mathrm{s},e}^3}^{*(r)}=-1$ is the only feasible solution. Inserting $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$, $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$, and $y_{a_{\\mathrm{s},e}^3}^{*(r)} = -1$ into \\eqref{eq:rel_a_3}, we again receive $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)})=0$, which finishes the proof.\n\t\\end{proof}\n","start":73950,"end":76339},{"raw_tex":"        By contradiction, let us assume that $a_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}^{*(r)}) = 0$, i.e., $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$. Then, we need to show that $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)}) = 0$ and $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)}) = 0$.\n","start":73965,"end":74276},{"raw_tex":"\t\tConsider first the case of $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)})$. The moment matrix {(recall the definition in \\eqref{eq:moment})} then contains the principal submatrix\n\t\t%\n\t\t\\begin{equation}\n\t\t\t\\begin{pmatrix}\n\t\t\t\t1 & y_{a_{\\mathrm{s},e}^1}^{*(r)}\\\\\n\t\t\t\ty_{a_{\\mathrm{s},e}^1}^{*(r)} & y_{a_{\\mathrm{s},e}^2}^{*(r)}\n\t\t\t\\end{pmatrix} \\succeq 0.\n\t\t\\end{equation}\n\t\t%\n\t\tThe determinant of this matrix must be nonnegative, so that we obtain the inequality\n\t\t%\n\t\t\\begin{equation}\n\t\t\ty_{a_{\\mathrm{s},e}^2}^{*(r)} - \\left[y_{a_{\\mathrm{s},e}^1}^{*(r)}\\right]^2 \\ge 0.\n\t\t\\end{equation}\n\t\t%\n\t\tAfter inserting $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$, we obtain $y_{a_{\\mathrm{s},e}^2}^{*(r)} \\ge 1$. Due to the bound constraints \\eqref{eq:opt_nonneg_po}, $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$. Using $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$ with $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$, \\eqref{eq:rel_a_2} is then evaluated as $a_{\\mathrm{rel},e}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)})=0$.\n","start":74279,"end":75270},{"raw_tex":"\t\tFor the case of, $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)})$, the moment matrix contains the principal submatrix\n\t\t%\n\t\t\\begin{equation}\n\t\t\t\\begin{pmatrix}\n\t\t\t\t1 & y_{a_{\\mathrm{s},e}^1}^{*(r)} & y_{a_{\\mathrm{s},e}^2}^{*(r)}\\\\\n\t\t\t\ty_{a_{\\mathrm{s},e}^1}^{*(r)} & y_{a_{\\mathrm{s},e}^2}^{*(r)} & y_{a_{\\mathrm{s},e}^3}^{*(r)}\\\\\n\t\t\t\ty_{a_{\\mathrm{s},e}^2}^{*(r)} & y_{a_{\\mathrm{s},e}^3}^{*(r)} & y_{a_{\\mathrm{s},e}^4}^{*(r)}\n\t\t\t\\end{pmatrix} \\succeq 0.\n\t\t\\end{equation}\n\t\t%\n\t\tAfter inserting $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$ and $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$, the determinant evaluates as\n\t\t%\n\t\t\\begin{equation}\n\t\t\t- \\left(y_{a_{\\mathrm{s},e}^3}^{*(r)} + 1\\right)^2 \\ge 0,\n\t\t\\end{equation}\n\t\t%\n\t\tso that $y_{a_{\\mathrm{s},e}^3}^{*(r)}=-1$ is the only feasible solution. Inserting $y_{a_{\\mathrm{s},e}^1}^{*(r)} = -1$, $y_{a_{\\mathrm{s},e}^2}^{*(r)} = 1$, and $y_{a_{\\mathrm{s},e}^3}^{*(r)} = -1$ into \\eqref{eq:rel_a_3}, we again receive $a_{\\mathrm{rel},e}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)})=0$, which finishes the proof.\n","start":75273,"end":76326},{"raw_tex":"\\end{proposition}","start":76339,"end":76356}]},{"id":"S059","semantic_type":"proposition","kind":"proposition","labels":["eq:scalar_fvineq","th:ub"],"member_ids":["S059","S060","S061","S062","S064","S065"],"declared_external":false,"source_only":false,"title":"一阶矩比例的振动可行缩放","source_units":[{"raw_tex":"\\begin{proposition}\\label{th:ub}\n\tLet $\\mathbf{y}^{*(r)}$ be the optimal moments obtained by solving the $r$-th degree relaxation and set $\\tilde{\\mathbf{a}} := \\mathbf{a}_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}^{*(r)})$. Then, there exists $\\delta$ such that\n\t%\n\t\\begin{equation}\\label{eq:scalar_fvineq}\n\t\t\\mathbf{K}_j(\\delta\\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\mathbf{M}_j(\\delta\\tilde{\\mathbf{a}}) \\succeq 0\\,.\n\t\\end{equation}\n\\end{proposition}","start":76589,"end":77051},{"raw_tex":"\\begin{proof}\n\tFrom the hierarchy construction {\\eqref{eq:general_moment}}, the {localizing matrix \\eqref{eq:localizing} will always contain $L_\\mathbf{y}(1 \\otimes \\left[\\mathbf{K}_j (\\mathbf{a}) - \\underline{\\lambda} \\mathbf{M}_j(\\mathbf{a})\\right])$ as principal submatrix due to} \\eqref{eq:opt_fv}{, which} can be written in terms of the lifted variables as\n\t%\n\t\\begin{multline}\\label{eq:fvrelaxed}\n\t\t\\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)})\\right]^{\\circ\\frac{1}{2}}\\right) + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)})\\right]^{\\circ\\frac{1}{3}}\\right) \\\\\n\t\t- \\underline{\\lambda}_j\\left( \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) \\right)\\succeq 0,\n\t\\end{multline}\n\t%\n\twith $\\bullet^{\\circ q}$ raising each element in $\\bullet$ to the power of $q$. We note here that the power factors in \\eqref{eq:fvrelaxed} eliminate the powers introduced in the notation \\eqref{eq:hatnotation}, so that \\eqref{eq:fvrelaxed} is indeed linear in $\\mathbf{a}_\\mathrm{rel}^{(i)}$ for all $i$.\n \n    After projecting via an orthonormal basis of $\\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$ denoted with $\\mathbf{W}_j$, we receive\n\t%\n\t\\begin{equation}\\label{eq:rel_cond1}\n\t\t\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) \\right]\\mathbf{W}_j \\succeq \\underline{\\lambda}_j \\mathbf{W}_j^\\mathrm{T} \\hat{\\mathbf{M}}_{j}^{\\langle 0\\rangle}\\mathbf{W}_j \\succeq 0.\n\t\\end{equation}\n\t%\n\tFurther, because of Proposition \\ref{prop:inertia}, {$a_{\\mathrm{rel},e}^{\\langle 1 \\rangle} \\mathbf{K}_{j,e}^{\\langle 1\\rangle}\\neq \\mathbf{0}$} whenever {$a_{\\mathrm{rel},e}^{\\langle2\\rangle} \\mathbf{K}_{j,e}^{\\langle 2\\rangle} + a_{\\mathrm{rel},e}^{\\langle3 \\rangle} \\mathbf{K}_{j,e}^{\\langle 3\\rangle} \\neq \\mathbf{0}$}. Consequently, \n\t\\begin{multline}\\label{eq:spans}\n\t\t\\mathrm{Span}\\left(\\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} (\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 2\\rangle}\\right]^{\\circ\\frac{1}{2}}\\right) + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 3\\rangle}\\right]^{\\circ\\frac{1}{3}}\\right)\\right)\\\\ = \\mathrm{Span}\\left(\\hat{\\mathbf{K}}_j^{\\langle 1 \\rangle} (\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}(\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{{\\langle 3\\rangle}}(\\tilde{\\mathbf{a}}) \\right).\n\t\\end{multline}\n\t%\n\tThus, because \\eqref{eq:fvrelaxed} is feasible due to $\\mathbf{y}^{*(r)}$ being optimal, it holds based on Proposition \\ref{lemma:dynamic_admissibility} and based on \\eqref{eq:spans} that \n\t%\n\t\\begin{equation}\\label{eq:rel_cond2}\n\t\t\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right).\n\t\\end{equation}\n\t%\n\tUsing \\eqref{eq:rel_cond1} and \\eqref{eq:rel_cond2}, Proposition \\ref{prop:scaling} certifies existence of $\\delta$ such that the matrix inequality \\eqref{eq:scalar_fvineq} holds.\n\\end{proof}","start":77052,"end":80485},{"raw_tex":"\tFrom the hierarchy construction {\\eqref{eq:general_moment}}, the {localizing matrix \\eqref{eq:localizing} will always contain $L_\\mathbf{y}(1 \\otimes \\left[\\mathbf{K}_j (\\mathbf{a}) - \\underline{\\lambda} \\mathbf{M}_j(\\mathbf{a})\\right])$ as principal submatrix due to} \\eqref{eq:opt_fv}{, which} can be written in terms of the lifted variables as\n\t%\n\t\\begin{multline}\\label{eq:fvrelaxed}\n\t\t\\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 2\\rangle}(\\mathbf{y}^{*(r)})\\right]^{\\circ\\frac{1}{2}}\\right) + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 3\\rangle}(\\mathbf{y}^{*(r)})\\right]^{\\circ\\frac{1}{3}}\\right) \\\\\n\t\t- \\underline{\\lambda}_j\\left( \\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) \\right)\\succeq 0,\n\t\\end{multline}\n\t%\n\twith $\\bullet^{\\circ q}$ raising each element in $\\bullet$ to the power of $q$. We note here that the power factors in \\eqref{eq:fvrelaxed} eliminate the powers introduced in the notation \\eqref{eq:hatnotation}, so that \\eqref{eq:fvrelaxed} is indeed linear in $\\mathbf{a}_\\mathrm{rel}^{(i)}$ for all $i$.\n","start":77066,"end":78288},{"raw_tex":"    After projecting via an orthonormal basis of $\\mathrm{Ker}\\left(\\hat{\\mathbf{K}}_j^{\\langle 2\\rangle} + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\right) \\cap \\mathrm{Im}\\left(\\hat{\\mathbf{M}}_j^{\\langle 0\\rangle} + \\hat{\\mathbf{M}}_j^{\\langle 1\\rangle}\\right)$ denoted with $\\mathbf{W}_j$, we receive\n\t%\n\t\\begin{equation}\\label{eq:rel_cond1}\n\t\t\\mathbf{W}_j^\\mathrm{T}\\left[\\hat{\\mathbf{K}}_{j}^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) - \\underline{\\lambda}_j \\hat{\\mathbf{M}}_{j}^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) \\right]\\mathbf{W}_j \\succeq \\underline{\\lambda}_j \\mathbf{W}_j^\\mathrm{T} \\hat{\\mathbf{M}}_{j}^{\\langle 0\\rangle}\\mathbf{W}_j \\succeq 0.\n\t\\end{equation}\n","start":78290,"end":78960},{"raw_tex":"\tThus, because \\eqref{eq:fvrelaxed} is feasible due to $\\mathbf{y}^{*(r)}$ being optimal, it holds based on Proposition \\ref{lemma:dynamic_admissibility} and based on \\eqref{eq:spans} that \n\t%\n\t\\begin{equation}\\label{eq:rel_cond2}\n\t\t\\mathrm{Im}\\left(\\mathbf{M}_j(\\tilde{\\mathbf{a}})\\right) \\subseteq \\mathrm{Im}\\left(\\mathbf{K}_j(\\tilde{\\mathbf{a}})\\right).\n\t\\end{equation}\n","start":79916,"end":80290},{"raw_tex":"\tUsing \\eqref{eq:rel_cond1} and \\eqref{eq:rel_cond2}, Proposition \\ref{prop:scaling} certifies existence of $\\delta$ such that the matrix inequality \\eqref{eq:scalar_fvineq} holds.\n","start":80293,"end":80474}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":["eq:spans"],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"提升刚度与实际刚度同像","source_units":[{"raw_tex":"\tFurther, because of Proposition \\ref{prop:inertia}, {$a_{\\mathrm{rel},e}^{\\langle 1 \\rangle} \\mathbf{K}_{j,e}^{\\langle 1\\rangle}\\neq \\mathbf{0}$} whenever {$a_{\\mathrm{rel},e}^{\\langle2\\rangle} \\mathbf{K}_{j,e}^{\\langle 2\\rangle} + a_{\\mathrm{rel},e}^{\\langle3 \\rangle} \\mathbf{K}_{j,e}^{\\langle 3\\rangle} \\neq \\mathbf{0}$}. Consequently, \n\t\\begin{multline}\\label{eq:spans}\n\t\t\\mathrm{Span}\\left(\\hat{\\mathbf{K}}_j^{\\langle 1\\rangle} (\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 2\\rangle}\\right]^{\\circ\\frac{1}{2}}\\right) + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 3\\rangle}\\right]^{\\circ\\frac{1}{3}}\\right)\\right)\\\\ = \\mathrm{Span}\\left(\\hat{\\mathbf{K}}_j^{\\langle 1 \\rangle} (\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}(\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{{\\langle 3\\rangle}}(\\tilde{\\mathbf{a}}) \\right).\n\t\\end{multline}\n","start":78963,"end":79913}]},{"id":"S066","semantic_type":"proposition","kind":"proposition","labels":["prop:feasibleub"],"member_ids":["S066"],"declared_external":false,"source_only":false,"title":"一阶矩比例的联合可行缩放","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:feasibleub}\n\tLet $\\mathbf{y}^{*(r)}$ be the optimal moments obtained by solving the $r$-th degree relaxation and set $\\tilde{\\mathbf{a}} := \\mathbf{a}_{\\mathrm{rel},e}^{\\langle 1\\rangle}(\\mathbf{y}^{*(r)})$. Then, there exists a $\\delta$ such that $\\mathbf{a}(\\delta):=\\delta\\tilde{\\mathbf{a}}$ is feasible to the lower-level problem \\eqref{eq:bilevel-lower}.\n\\end{proposition}","start":80750,"end":81158}]},{"id":"S067","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S067"],"declared_external":false,"source_only":false,"title":"联合可行缩放的作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\tFor free-vibration constraints, the proof follows from Proposition \\ref{th:ub}. For static constraints, based on Proposition \\ref{prop:inertia}, equation~\\eqref{eq:spans} holds. Thus, since\n\t%\n\t\\begin{equation}\n\t\t\\begin{pmatrix}\n\t\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t\t-\\mathbf{f}_j & \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 2\\rangle}\\right]^{\\circ\\frac{1}{2}}\\right) + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 3\\rangle}\\right]^{\\circ\\frac{1}{3}}\\right)\n\t\t\\end{pmatrix} \\succeq 0\n\t\\end{equation}\n\t%\n\tis satisfied by optimality of $\\mathbf{y}^{*(r)}$, the condition $\\mathbf{f}_j \\in \\mathbf{K}_j(\\mathbf{\\tilde{a}})$ holds true due to Proposition \\ref{lemma:statical_admissibility}. Based on Proposition \\ref{th:feasibility_cond}, the upper bound solution can thus be constructed.\n\t\n\tConsequently, there exists $\\delta$ such that both types of constraints are satisfied simultaneously.\n\\end{proof}","start":81159,"end":82228}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S068"],"declared_external":false,"source_only":false,"title":"沿用振动可行缩放","source_units":[{"raw_tex":"\tFor free-vibration constraints, the proof follows from Proposition \\ref{th:ub}. For static constraints, based on Proposition \\ref{prop:inertia}, equation~\\eqref{eq:spans} holds. Thus, since\n","start":81173,"end":81364}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S069"],"declared_external":false,"source_only":false,"title":"使用提升柔度分块约束","source_units":[{"raw_tex":"\t\\begin{equation}\n\t\t\\begin{pmatrix}\n\t\t\t\\overline{c}_j & -\\mathbf{f}_j^\\mathrm{T}\\\\\n\t\t\t-\\mathbf{f}_j & \\hat{\\mathbf{K}}_j^{\\langle 1\\rangle}(\\tilde{\\mathbf{a}}) + \\hat{\\mathbf{K}}_j^{\\langle 2\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 2\\rangle}\\right]^{\\circ\\frac{1}{2}}\\right) + \\hat{\\mathbf{K}}_j^{\\langle 3\\rangle}\\left(\\left[\\mathbf{a}_{\\mathrm{rel}}^{\\langle 3\\rangle}\\right]^{\\circ\\frac{1}{3}}\\right)\n\t\t\\end{pmatrix} \\succeq 0\n\t\\end{equation}\n","start":81367,"end":81827}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S070"],"declared_external":false,"source_only":false,"title":"得到实际比例静力可容许","source_units":[{"raw_tex":"\tFor free-vibration constraints, the proof follows from Proposition \\ref{th:ub}. For static constraints, based on Proposition \\ref{prop:inertia}, equation~\\eqref{eq:spans} holds. Thus, since\n","start":81173,"end":81364},{"raw_tex":"\tis satisfied by optimality of $\\mathbf{y}^{*(r)}$, the condition $\\mathbf{f}_j \\in \\mathbf{K}_j(\\mathbf{\\tilde{a}})$ holds true due to Proposition \\ref{lemma:statical_admissibility}. Based on Proposition \\ref{th:feasibility_cond}, the upper bound solution can thus be constructed.\n","start":81830,"end":82112}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S071"],"declared_external":false,"source_only":false,"title":"合并得到共同缩放","source_units":[{"raw_tex":"\tis satisfied by optimality of $\\mathbf{y}^{*(r)}$, the condition $\\mathbf{f}_j \\in \\mathbf{K}_j(\\mathbf{\\tilde{a}})$ holds true due to Proposition \\ref{lemma:statical_admissibility}. Based on Proposition \\ref{th:feasibility_cond}, the upper bound solution can thus be constructed.\n","start":81830,"end":82112},{"raw_tex":"\tConsequently, there exists $\\delta$ such that both types of constraints are satisfied simultaneously.\n","start":82114,"end":82217}]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S020":"S020","S022":"S022","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S029","S031":"S029","S032":"S029","S033":"S033","S034":"S033","S035":"S035","S036":"S035","S037":"S035","S038":"S035","S039":"S035","S040":"S040","S041":"S040","S042":"S040","S043":"S040","S044":"S044","S045":"S044","S046":"S044","S047":"S044","S048":"S044","S049":"S049","S050":"S049","S051":"S049","S052":"S049","S053":"S049","S054":"S054","S055":"S054","S056":"S054","S057":"S054","S058":"S054","S059":"S059","S060":"S059","S061":"S059","S062":"S059","S063":"S063","S064":"S059","S065":"S059","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071"}}}}
{"record_id":"2011.09782v4","paper":{"title":"Inertial Proximal Block Coordinate Method for a Class of Nonsmooth Sum-of-Ratios Optimization Problems","authors":["Boţ, Radu Ioan","Dao, Minh N.","Li, Guoyin"],"arxiv_id":"2011.09782","version":"v4","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/2011.09782v4","pdf_url":"https://arxiv.org/pdf/2011.09782v4","doi":"10.1137/22M1472000"},"context":{"raw_tex":"In this section, we recall some basic notations and preliminary results which will be used in this paper. We assume throughout that $\\mathcal{H}$, $\\mathcal{H}_1$, \\dots, $\\mathcal{H}_m$ are finite-dimensional real Hilbert spaces\nwith inner product $\\scal{\\cdot}{\\cdot}$ and induced norm $\\|\\cdot\\|$. The product space $\\mathcal{H}_1 \\times \\dots \\times \\mathcal{H}_m$ is also a real Hilbert space endowed with the inner product given by\n    $\\langle (x_1,\\dots,x_m),(y_1,\\dots,y_m) \\rangle = \\sum_{i=1}^m \\langle x_i,y_i \\rangle.$\nThe set of nonnegative integers is denoted by $\\mathbb{N}$, the set of real numbers by $\\mathbb{R}$, the set of nonnegative real numbers by $\\RP$, and the set of the positive real numbers by $\\RPP$.\n\nThe \\emph{indicator function} of a set $C$ is defined by $\\delta_C(\\x) :=0$ if $\\x \\in C$, and $\\delta_C(\\x) :=+\\infty$ otherwise. Given an extended-real-valued function $f\\colon \\mathcal{H}\\to \\left[-\\infty,+\\infty\\right]$, its \\emph{domain} is defined by $\\dom f :=\\{\\x \\in \\mathcal{H}: f(\\x)<+\\infty\\}$. The function $f$ is \\emph{proper} if $\\dom f\\neq \\varnothing$ and it never equals $-\\infty$. We say that $f$ is \\emph{lower semicontinuous} if, for all $\\x\\in  \\mathcal{H}$, $f(\\x)\\leq \\liminf_{\\y\\to \\x} f(\\y)$, and \\emph{convex} if its epigraph $\\{(\\x,\\rho)\\in \\mathcal{H}\\times \\mathbb{R}: f(\\x)\\leq \\rho\\}$ is a convex subset of $\\mathcal{H}\\times \\mathbb{R}$. The function $f$ is said to be \\emph{weakly convex (on $\\mathcal{H}$)} if there exists $\\alpha \\geq 0$ such that $f+\\frac{\\alpha}{2}\\|\\cdot\\|^2$ is a convex function. The smallest constant $\\alpha$ such that $f+\\frac{\\alpha}{2}\\|\\cdot\\|^2$ is convex is called the \\emph{modulus} of weak convexity. More generally, $f$ is said to be \\emph{weakly convex on $S \\subseteq \\mathcal{H}$ with modulus $\\alpha$} if $f+\\delta_S$ is weakly convex with modulus $\\alpha$. Weakly convex functions form a broad class of functions which covers quadratic functions, convex functions, differentiable functions whose gradient is Lipschitz continuous, and the composition of a convex and Lipschitz continuous function with a $C^1$-smooth mapping whose Jacobian is Lipschitz continuous (see \\cite[Lemma~4.2]{DP19}).\n\n\nLet $f\\colon\\mathcal{H} \\to \\left[-\\infty,+\\infty\\right]$ and $\\x\\in \\mathcal{H}$ with $|f(\\x)| <+\\infty$. The \\emph{Fr\\'echet subdifferential} of $f$ at $\\x$ is given by\n\\begin{equation*}\n\\widehat{\\partial} f(\\x) :=\\left\\{\\bu\\in \\mathcal{H}:\\; \\liminf_{\\z\\to \\x}\\frac{f(\\z)-f(\\x)-\\langle \\bu,\\z-\\x\\rangle}{\\|\\z-\\x\\|}\\geq 0\\right\\},\n\\end{equation*}\nthe \\emph{limiting subdifferential} of $f$ at $\\x$ is given by\n\\begin{equation*}\n\\partial_L f(\\x) :=\\left\\{\\bu\\in \\mathcal{H}:\\; \\exists \\x_n\\st{f}{\\to}\\x,\\ \\bu_n\\to \\bu\\text{~~with~~} \\bu_n\\in \\widehat{\\partial} f(\\x_n)\\right\\},\n\\end{equation*}\nand the \\emph{horizon subdifferential} of $f$ at $\\x$ is given by\n\\begin{equation*}\n\\partial^{\\infty}f(\\x) :=\\left\\{\\bu\\in \\mathcal{H}:\\; \\exists \\x_n\\st{f}{\\to}\\x,\\ \\lambda_n\\downarrow 0,\\ \\lambda_n\\bu_n\\to \\bu\\text{~~with~~} \\bu_n\\in \\widehat{\\partial} f(\\x_n)\\right\\}.\n\\end{equation*}\nHere, the notation $\\z\\st{f}{\\to}\\x$ means $\\z\\to \\x$ with $f(\\z)\\to f(\\x)$. It follows from the above definition that the limiting subdifferential has the following \\emph{robustness property}\n\\begin{equation*}\n\\partial_L f(\\x) =\\left\\{\\bu\\in \\mathcal{H}:\\; \\exists \\x_n \\st{f}{\\to}\\x,\\ \\bu_n\\to \\bu \\text{~~with~~} \\bu_n\\in \\partial_L f(\\x_n)\\right\\}.\n\\end{equation*}\nThe \\emph{domain} of $\\partial_L f$ is $\\dom\\partial_L f :=\\{\\x\\in \\mathcal{H}: \\partial_L f(\\x)\\neq \\varnothing\\}$. If $f$ is Lipschitz continuous around $\\x$, then $\\partial_L f(\\x)$ is bounded and $\\partial^{\\infty} f(\\x) =\\{0\\}$ (see \\cite[Corollary~1.81]{Mor06}). If $f$ is strictly differentiable\\footnote{A function $f$ is strictly differentiable at $\\x$ if there exists $\\bu \\in \\mathcal{H}$ such that $\\lim_{\\y,\\z \\to \\x}\\frac{f(\\y)-f(\\z)-\\langle \\bu,\\y-\\z\\rangle}{\\|\\y-\\z\\|} =0$. Clearly, if $f$ is continuously differentiable at $\\x$, then it is strictly differentiable at $\\x$.} at $\\x$, then $\\widehat{\\partial} f$ and $\\partial_L f$ reduce to the derivative of $f$, denoted by $\\nabla f$ (see \\cite[Corollary~1.82]{Mor06}). If $f$ is convex, then both Fr\\'echet and limiting subdifferentials at $\\x$ reduce to the classical subdifferential in convex analysis (see \\cite[Theorem~1.93]{Mor06})\n\\begin{equation*}\n\\partial f(\\x) :=\\left\\{\\bu\\in \\mathcal{H}:\\; \\forall \\z\\in \\mathcal{H},\\; \\langle \\bu,\\z-\\x\\rangle\\leq f(\\z)-f(\\x)\\right\\}.\n\\end{equation*}\nWe say that $f$ is \\emph{regular\\footnote{This is also referred as \\emph{lower regular} in \\cite{Mor06,MNY06}.} at $\\x\\in \\mathcal{H}$} if $\\widehat{\\partial} f(\\x) =\\partial_L f(\\x)$, and that $f$ is \\emph{regular on $C\\subseteq \\mathcal{H}$} if it is regular at any $\\x\\in C$. For a  proper lower semicontinuous function $f$, it is clear that if $f$ is convex around $\\x$ or strictly differentiable at $\\x$, then it is regular at $\\x$. A nonempty set $S$ in $\\mathcal{H}$ is \\emph{regular} at $\\x\\in S$ if $\\delta_S$ is regular at $\\x$. We say that $S$ is regular if it is regular at all of its points. From the definition, it can be verified that $C$ is regular if $C$ is a closed and convex set or $C$ is a smooth manifold given by $C =\\{\\x\\in \\mathcal{H}: g_i(\\x)=0,\\, i=1,\\dots, m\\}$, where $g_i$ are smooth functions satisfying the so-called linear independent constraint qualification (that is, $\\{\\nabla g_i(\\x): i=1,\\dots,m\\}$ are linearly independent for all $\\x \\in C$).\n\n\n\\begin{lemma}[calculus rules]\n\\label{l:calrules}\nLet $f,g\\colon \\mathcal{H}\\to \\left(-\\infty,+\\infty\\right]$ be proper lower semicontinuous functions and let $\\x \\in \\dom f$. Then the following statements hold:\n\\begin{enumerate}\n\\item\\label{l:calrules_separable}\n(Separable sum rule) If $f(\\x) =\\sum_{i=1}^m f_i (x_i)$ with $\\x =(x_1,\\dots,x_m)$, then $\\partial_L f(\\x) =\\partial_L f_1(x_1) \\times \\dots \\times \\partial_L f_m(x_m)$ and $f$ is regular at $\\x$ when each $f_i$ regular at $x_i$.\n\n\\item\\label{l:calrules_sum}\n(Sum rule)\nIf $\\partial^{\\infty} f(\\x) \\cap (-\\partial^{\\infty} g(\\x)) =\\{0\\}$, then $\\partial_L(f+g)(\\x) \\subseteq \\partial_L f(\\x) + \\partial_L g(\\x)$, where the equality holds when both $f$ and $g$ are regular at $\\x$, in which case $f+g$ is also regular at $\\x$. Moreover, if $g$ is strictly differentiable at $\\x$, then $\\partial_L (f+g)(\\x) =\\partial_L f(\\x) +\\nabla g(\\x)$.\n\n\\item\\label{l:calrules_sign}\n(Sign rule)\nIf $f$ is Lipschitz continuous around $\\x$ and $\\widehat{\\partial} f$ is nonempty-valued around $\\x$, then $\\partial_L (-f)(\\x)\\subseteq -\\partial_L f(\\x)$. \n\n\\item\\label{l:calrules_quotient}\n(Quotient rule) \nSuppose that $f$ and $g$ are Lipschitz continuous around $\\x$, and $g(\\x)\\neq 0$. If $\\widehat{\\partial} f$ is nonempty-valued around $\\x$, then\n\\begin{equation*}\n\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) \\subseteq \\frac{-g(\\x)\\partial_L f(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}.\n\\end{equation*}\nIf $f$ is strictly differentiable at $\\x$, then\n\\begin{equation*}\n\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{-g(\\x)\\nabla f(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}\n\\end{equation*}\nand, consequently, $-f/g$ is regular at $\\x$ if and only if $f(\\x)g$ is regular at $\\x$.\n\n\\item\\label{l:calrules_sqrt}\n(Chain rule and square root rule)\nIf $f$ is Lipschitz continuous around $\\x$ and $\\theta\\colon \\mathbb{R}\\to \\mathbb{R}$ is continuously differentiable around $f(\\x)$,  \nthen $\\partial_L(\\theta \\circ f)(\\x) =\\partial_L (\\theta'(f(\\x))f)(\\x)$. In particular, if $f$ is Lipschitz continuous around $\\x$ and $f(\\x)>0$, then\n\\begin{equation*}\n\\partial_L\\left(\\sqrt{f}\\right)(\\x) =\\frac{\\partial_L f(\\x)}{2 \\sqrt{f(\\x)}} \\text{~~and~~} \\partial_L\\left(-\\sqrt{f}\\right)(\\x) =\\frac{\\partial_L (-f)(\\x)}{2 \\sqrt{f(\\x)}}.\n\\end{equation*}\n\\end{enumerate}\n\\end{lemma}\n\n\\begin{lemma}\n\\label{l:subdiff}\nLet $\\Lambda :=\\{\\x\\in \\mathbb{R}^d: \\|\\x\\|=1\\}$ and $C_r :=\\{\\x\\in \\mathbb{R}^d: \\|\\x\\|_0 \\leq r\\}$. Given $\\x =(x_1,\\dots,x_d)\\in \\mathbb{R}^d$, set $\\supp(\\x): =\\{j: x_j \\neq 0\\}$. Then the following statements hold:\n\\begin{enumerate}\n\\item\\label{l:subdiff_l0}\n$\\forall \\x\\in \\mathbb{R}^d$, $\\widehat{\\partial} (\\|\\cdot\\|_0)(\\x) =\\partial_L (\\|\\cdot\\|_0)(\\x) =\\partial_L^{\\infty} (\\|\\cdot\\|_0)(\\x) =\\{\\bv: v_j=0 \\text{~if~} j \\in \\supp(\\x)\\}$.\n\n\\item\\label{l:subdiff_sphere}\n$\\forall \\x\\in \\Lambda$, $\\widehat{\\partial} \\delta_{\\Lambda}(\\x) =\\partial_L \\delta_{\\Lambda}(\\x) =\\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{ t\\x: t \\in \\mathbb{R}\\}$.\n\n\\item\\label{l:subdiff_ball}\nIf $\\|\\x\\|_0=r$, then $\\partial_L^{\\infty} \\delta_{C_r}(\\x) =\\partial_L \\delta_{C_r}(\\x) =\\{\\bv: v_j=0 \\text{~if~} j \\in \\supp(\\x)\\}$. If $\\|\\x\\|_0<r$, then\n\\begin{multline*}\n\\partial_L^{\\infty} \\delta_{C_r}(\\x) =\\partial_L \\delta_{C_r}(\\x) =\\{\\bv: \\exists \\widehat{J} \\subseteq \\{1,\\dots,d\\} \\smallsetminus \\supp(\\x) \\text{~with~} |\\widehat{J}|=r-\\|\\x\\|_0,\\\\ v_j=0 \\text{~if~} j \\in (\\supp(\\x) \\cup \\widehat{J})\\}.    \n\\end{multline*}\n\n\\item\\label{l:subdiff_sum}\n$\\forall \\x\\in \\Lambda$, $\\partial_L (\\|\\cdot\\|_0+\\delta_{\\Lambda})(\\x) =\\partial_L (\\|\\cdot\\|_0)(\\x) +\\partial_L \\delta_{\\Lambda}(\\x)$.\n\n\\item\\label{l:subdiff_sum'}\n$\\forall \\x\\in \\Lambda \\cap C_r$, $\\partial_L (\\delta_{C_r}+\\delta_{\\Lambda})(\\x)\\subseteq \\partial_L\\delta_{C_r}(\\x) +\\partial_L \\delta_{\\Lambda}(\\x)$.\n\\end{enumerate}\n\\end{lemma}\n\nWe now consider the global convergence of the full sequence generated by Algorithm~\\ref{algo:epasa}. \nRecall that a proper lower semicontinuous function $f\\colon \\mathcal{H}\\to \\left(-\\infty, +\\infty\\right]$ is said to satisfy the \\emph{KL property} \\cite{Kur98,Loj63} at $\\overline{\\x}\\in \\dom\\partial_L f$ if there exist a neighborhood $U$ of $\\overline{\\x}$, $\\eta \\in \\left(0,+\\infty\\right]$, and a continuous concave function $\\varphi\\colon \\left[0, \\eta\\right)\\to \\mathbb{R}_+$ such that $\\varphi(0) =0$, $\\varphi$ is continuously differentiable with $\\varphi' >0$ on $\\left(0, \\eta\\right)$, and, for all $\\x\\in U$ with $f(\\overline{\\x}) <f(\\x) <f(\\overline{\\x})+\\eta$, \n\\begin{equation*}\n\\varphi'(f(\\x)-f(\\overline{\\x}))\\dist(0,\\partial_L f(\\x))\\geq 1.\n\\end{equation*}\nIf $f$ satisfies the KL property at any $\\overline{\\x}\\in \\dom\\partial_L f$, then it is called a \\emph{KL function}. We say that $f$ has the \\emph{KL property at $\\overline{\\x}\\in \\dom\\partial_L f$ with exponent $\\alpha$} if it satisfies the KL property at $\\overline{\\x}\\in \\dom\\partial_L f$ and the corresponding function $\\varphi$ (often referred as desingularization function) can be chosen as $\\varphi(s) =\\gamma s^{1-\\alpha}$ for some $\\gamma\\in \\RPP$ and $\\alpha\\in \\left[0,1\\right)$. If $f$ is a KL function and has the same exponent $\\alpha$ at any $\\overline{\\x}\\in \\dom\\partial_L f$, then it is called a \\emph{KL function with exponent $\\alpha$}.\n\n\n\\begin{lemma}[cf. {\\cite[Theorem 3.6]{LP18}}]\n\\label{l:LP}\nLet $f$ be a proper lower semicontinuous function. Suppose that $f$ satisfies the KL property at $\\overline{\\x} \\in \\dom\\partial_L f$ with exponent $\\alpha \\in [\\frac{1}{2},1)$. Then, for all $\\rho \\geq 0$, $\\widetilde{f}(\\x,\\bu)=f(\\x)+\\rho \\|\\x-\\bu\\|^2$ satisfies  the KL property with exponent $\\alpha$ at $(\\overline{\\x},\\overline{\\x})$.\n\\end{lemma}\n\n\nConsider the generalized eigenvalue problem with cardinality regularization\n\\begin{equation}\n\\max_{\\x \\in \\mathbb{R}^d} \\frac{\\x^\\top A_1\\x}{\\x^\\top B_1\\x}- \\lambda \\|\\x\\|_0 \n\\quad\\text{s.t.}\\quad \\|\\x\\|=1,\n\\tag{GEP}\n\\end{equation}\nwhere $A_1,B_1$ are symmetric matrices such that $A_1$ is positive semidefinite and $B_1$ is positive definite, and $\\lambda>0.$ For this generalized eigenvalue problem with cardinality regularization the corresponding merit function for the proposed Algorithm~\\ref{algo:epasa} takes the form\n\\begin{equation*}\n\\widehat{\\Phi}_{GEP}(\\x,\\bu) =\\frac{\\x^\\top A \\x}{\\x^\\top B \\x} +\\lambda \\|\\x\\|_0 +\\delta_{\\Lambda}(\\x) +\\rho \\|\\x-\\bu\\|^2,\n\\end{equation*}\nwith $A=-A_1$ a symmetric matrix,  $B=B_1$ is a positive definite matrix, $\\Lambda =\\{\\x\\in \\mathbb{R}^d: \\|\\x\\|=1\\}$, and $\\rho \\geq 0$. Below, we derive the KL exponent of the merit function $\\widehat{\\Phi}_{GEP}$. To this end, we will use the following lemma from \\cite{LP18}. Here we provide an alternative short proof for it.\n\n\nNext we prove that the KL exponent of the merit function $\\widehat{\\Phi}_{GEP}$ is $\\frac{1}{2}$. To do this, for an index set $J=\\{j_1,\\dots,j_k\\} \\subseteq \\{1,\\dots,d\\}$ with $k \\leq d$, we denote $\\x_J:=(x_{j_1},\\dots,x_{j_k})$. Moreover, for two index sets $I,J$, we denote\n$A_{IJ}=(A_{ij})_{i \\in I, j \\in J}.$\n","units":[{"raw_tex":"In this section, we recall some basic notations and preliminary results which will be used in this paper. We assume throughout that $\\mathcal{H}$, $\\mathcal{H}_1$, \\dots, $\\mathcal{H}_m$ are finite-dimensional real Hilbert spaces\nwith inner product $\\scal{\\cdot}{\\cdot}$ and induced norm $\\|\\cdot\\|$. The product space $\\mathcal{H}_1 \\times \\dots \\times \\mathcal{H}_m$ is also a real Hilbert space endowed with the inner product given by\n    $\\langle (x_1,\\dots,x_m),(y_1,\\dots,y_m) \\rangle = \\sum_{i=1}^m \\langle x_i,y_i \\rangle.$\nThe set of nonnegative integers is denoted by $\\mathbb{N}$, the set of real numbers by $\\mathbb{R}$, the set of nonnegative real numbers by $\\RP$, and the set of the positive real numbers by $\\RPP$.\n\nThe \\emph{indicator function} of a set $C$ is defined by $\\delta_C(\\x) :=0$ if $\\x \\in C$, and $\\delta_C(\\x) :=+\\infty$ otherwise. Given an extended-real-valued function $f\\colon \\mathcal{H}\\to \\left[-\\infty,+\\infty\\right]$, its \\emph{domain} is defined by $\\dom f :=\\{\\x \\in \\mathcal{H}: f(\\x)<+\\infty\\}$. The function $f$ is \\emph{proper} if $\\dom f\\neq \\varnothing$ and it never equals $-\\infty$. We say that $f$ is \\emph{lower semicontinuous} if, for all $\\x\\in  \\mathcal{H}$, $f(\\x)\\leq \\liminf_{\\y\\to \\x} f(\\y)$, and \\emph{convex} if its epigraph $\\{(\\x,\\rho)\\in \\mathcal{H}\\times \\mathbb{R}: f(\\x)\\leq \\rho\\}$ is a convex subset of $\\mathcal{H}\\times \\mathbb{R}$. The function $f$ is said to be \\emph{weakly convex (on $\\mathcal{H}$)} if there exists $\\alpha \\geq 0$ such that $f+\\frac{\\alpha}{2}\\|\\cdot\\|^2$ is a convex function. The smallest constant $\\alpha$ such that $f+\\frac{\\alpha}{2}\\|\\cdot\\|^2$ is convex is called the \\emph{modulus} of weak convexity. More generally, $f$ is said to be \\emph{weakly convex on $S \\subseteq \\mathcal{H}$ with modulus $\\alpha$} if $f+\\delta_S$ is weakly convex with modulus $\\alpha$. Weakly convex functions form a broad class of functions which covers quadratic functions, convex functions, differentiable functions whose gradient is Lipschitz continuous, and the composition of a convex and Lipschitz continuous function with a $C^1$-smooth mapping whose Jacobian is Lipschitz continuous (see \\cite[Lemma~4.2]{DP19}).\n\n\nLet $f\\colon\\mathcal{H} \\to \\left[-\\infty,+\\infty\\right]$ and $\\x\\in \\mathcal{H}$ with $|f(\\x)| <+\\infty$. The \\emph{Fr\\'echet subdifferential} of $f$ at $\\x$ is given by\n\\begin{equation*}\n\\widehat{\\partial} f(\\x) :=\\left\\{\\bu\\in \\mathcal{H}:\\; \\liminf_{\\z\\to \\x}\\frac{f(\\z)-f(\\x)-\\langle \\bu,\\z-\\x\\rangle}{\\|\\z-\\x\\|}\\geq 0\\right\\},\n\\end{equation*}\nthe \\emph{limiting subdifferential} of $f$ at $\\x$ is given by\n\\begin{equation*}\n\\partial_L f(\\x) :=\\left\\{\\bu\\in \\mathcal{H}:\\; \\exists \\x_n\\st{f}{\\to}\\x,\\ \\bu_n\\to \\bu\\text{~~with~~} \\bu_n\\in \\widehat{\\partial} f(\\x_n)\\right\\},\n\\end{equation*}\nand the \\emph{horizon subdifferential} of $f$ at $\\x$ is given by\n\\begin{equation*}\n\\partial^{\\infty}f(\\x) :=\\left\\{\\bu\\in \\mathcal{H}:\\; \\exists \\x_n\\st{f}{\\to}\\x,\\ \\lambda_n\\downarrow 0,\\ \\lambda_n\\bu_n\\to \\bu\\text{~~with~~} \\bu_n\\in \\widehat{\\partial} f(\\x_n)\\right\\}.\n\\end{equation*}\nHere, the notation $\\z\\st{f}{\\to}\\x$ means $\\z\\to \\x$ with $f(\\z)\\to f(\\x)$. It follows from the above definition that the limiting subdifferential has the following \\emph{robustness property}\n\\begin{equation*}\n\\partial_L f(\\x) =\\left\\{\\bu\\in \\mathcal{H}:\\; \\exists \\x_n \\st{f}{\\to}\\x,\\ \\bu_n\\to \\bu \\text{~~with~~} \\bu_n\\in \\partial_L f(\\x_n)\\right\\}.\n\\end{equation*}\nThe \\emph{domain} of $\\partial_L f$ is $\\dom\\partial_L f :=\\{\\x\\in \\mathcal{H}: \\partial_L f(\\x)\\neq \\varnothing\\}$. If $f$ is Lipschitz continuous around $\\x$, then $\\partial_L f(\\x)$ is bounded and $\\partial^{\\infty} f(\\x) =\\{0\\}$ (see \\cite[Corollary~1.81]{Mor06}). If $f$ is strictly differentiable\\footnote{A function $f$ is strictly differentiable at $\\x$ if there exists $\\bu \\in \\mathcal{H}$ such that $\\lim_{\\y,\\z \\to \\x}\\frac{f(\\y)-f(\\z)-\\langle \\bu,\\y-\\z\\rangle}{\\|\\y-\\z\\|} =0$. Clearly, if $f$ is continuously differentiable at $\\x$, then it is strictly differentiable at $\\x$.} at $\\x$, then $\\widehat{\\partial} f$ and $\\partial_L f$ reduce to the derivative of $f$, denoted by $\\nabla f$ (see \\cite[Corollary~1.82]{Mor06}). If $f$ is convex, then both Fr\\'echet and limiting subdifferentials at $\\x$ reduce to the classical subdifferential in convex analysis (see \\cite[Theorem~1.93]{Mor06})\n\\begin{equation*}\n\\partial f(\\x) :=\\left\\{\\bu\\in \\mathcal{H}:\\; \\forall \\z\\in \\mathcal{H},\\; \\langle \\bu,\\z-\\x\\rangle\\leq f(\\z)-f(\\x)\\right\\}.\n\\end{equation*}\nWe say that $f$ is \\emph{regular\\footnote{This is also referred as \\emph{lower regular} in \\cite{Mor06,MNY06}.} at $\\x\\in \\mathcal{H}$} if $\\widehat{\\partial} f(\\x) =\\partial_L f(\\x)$, and that $f$ is \\emph{regular on $C\\subseteq \\mathcal{H}$} if it is regular at any $\\x\\in C$. For a  proper lower semicontinuous function $f$, it is clear that if $f$ is convex around $\\x$ or strictly differentiable at $\\x$, then it is regular at $\\x$. A nonempty set $S$ in $\\mathcal{H}$ is \\emph{regular} at $\\x\\in S$ if $\\delta_S$ is regular at $\\x$. We say that $S$ is regular if it is regular at all of its points. From the definition, it can be verified that $C$ is regular if $C$ is a closed and convex set or $C$ is a smooth manifold given by $C =\\{\\x\\in \\mathcal{H}: g_i(\\x)=0,\\, i=1,\\dots, m\\}$, where $g_i$ are smooth functions satisfying the so-called linear independent constraint qualification (that is, $\\{\\nabla g_i(\\x): i=1,\\dots,m\\}$ are linearly independent for all $\\x \\in C$).\n","start":15915,"end":21416},{"raw_tex":"\\begin{lemma}[calculus rules]\n\\label{l:calrules}\nLet $f,g\\colon \\mathcal{H}\\to \\left(-\\infty,+\\infty\\right]$ be proper lower semicontinuous functions and let $\\x \\in \\dom f$. Then the following statements hold:\n\\begin{enumerate}\n\\item\\label{l:calrules_separable}\n(Separable sum rule) If $f(\\x) =\\sum_{i=1}^m f_i (x_i)$ with $\\x =(x_1,\\dots,x_m)$, then $\\partial_L f(\\x) =\\partial_L f_1(x_1) \\times \\dots \\times \\partial_L f_m(x_m)$ and $f$ is regular at $\\x$ when each $f_i$ regular at $x_i$.\n\n\\item\\label{l:calrules_sum}\n(Sum rule)\nIf $\\partial^{\\infty} f(\\x) \\cap (-\\partial^{\\infty} g(\\x)) =\\{0\\}$, then $\\partial_L(f+g)(\\x) \\subseteq \\partial_L f(\\x) + \\partial_L g(\\x)$, where the equality holds when both $f$ and $g$ are regular at $\\x$, in which case $f+g$ is also regular at $\\x$. Moreover, if $g$ is strictly differentiable at $\\x$, then $\\partial_L (f+g)(\\x) =\\partial_L f(\\x) +\\nabla g(\\x)$.\n\n\\item\\label{l:calrules_sign}\n(Sign rule)\nIf $f$ is Lipschitz continuous around $\\x$ and $\\widehat{\\partial} f$ is nonempty-valued around $\\x$, then $\\partial_L (-f)(\\x)\\subseteq -\\partial_L f(\\x)$. \n\n\\item\\label{l:calrules_quotient}\n(Quotient rule) \nSuppose that $f$ and $g$ are Lipschitz continuous around $\\x$, and $g(\\x)\\neq 0$. If $\\widehat{\\partial} f$ is nonempty-valued around $\\x$, then\n\\begin{equation*}\n\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) \\subseteq \\frac{-g(\\x)\\partial_L f(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}.\n\\end{equation*}\nIf $f$ is strictly differentiable at $\\x$, then\n\\begin{equation*}\n\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{-g(\\x)\\nabla f(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}\n\\end{equation*}\nand, consequently, $-f/g$ is regular at $\\x$ if and only if $f(\\x)g$ is regular at $\\x$.\n\n\\item\\label{l:calrules_sqrt}\n(Chain rule and square root rule)\nIf $f$ is Lipschitz continuous around $\\x$ and $\\theta\\colon \\mathbb{R}\\to \\mathbb{R}$ is continuously differentiable around $f(\\x)$,  \nthen $\\partial_L(\\theta \\circ f)(\\x) =\\partial_L (\\theta'(f(\\x))f)(\\x)$. In particular, if $f$ is Lipschitz continuous around $\\x$ and $f(\\x)>0$, then\n\\begin{equation*}\n\\partial_L\\left(\\sqrt{f}\\right)(\\x) =\\frac{\\partial_L f(\\x)}{2 \\sqrt{f(\\x)}} \\text{~~and~~} \\partial_L\\left(-\\sqrt{f}\\right)(\\x) =\\frac{\\partial_L (-f)(\\x)}{2 \\sqrt{f(\\x)}}.\n\\end{equation*}\n\\end{enumerate}\n\\end{lemma}","start":21574,"end":23878},{"raw_tex":"\\begin{lemma}\n\\label{l:subdiff}\nLet $\\Lambda :=\\{\\x\\in \\mathbb{R}^d: \\|\\x\\|=1\\}$ and $C_r :=\\{\\x\\in \\mathbb{R}^d: \\|\\x\\|_0 \\leq r\\}$. Given $\\x =(x_1,\\dots,x_d)\\in \\mathbb{R}^d$, set $\\supp(\\x): =\\{j: x_j \\neq 0\\}$. Then the following statements hold:\n\\begin{enumerate}\n\\item\\label{l:subdiff_l0}\n$\\forall \\x\\in \\mathbb{R}^d$, $\\widehat{\\partial} (\\|\\cdot\\|_0)(\\x) =\\partial_L (\\|\\cdot\\|_0)(\\x) =\\partial_L^{\\infty} (\\|\\cdot\\|_0)(\\x) =\\{\\bv: v_j=0 \\text{~if~} j \\in \\supp(\\x)\\}$.\n\n\\item\\label{l:subdiff_sphere}\n$\\forall \\x\\in \\Lambda$, $\\widehat{\\partial} \\delta_{\\Lambda}(\\x) =\\partial_L \\delta_{\\Lambda}(\\x) =\\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{ t\\x: t \\in \\mathbb{R}\\}$.\n\n\\item\\label{l:subdiff_ball}\nIf $\\|\\x\\|_0=r$, then $\\partial_L^{\\infty} \\delta_{C_r}(\\x) =\\partial_L \\delta_{C_r}(\\x) =\\{\\bv: v_j=0 \\text{~if~} j \\in \\supp(\\x)\\}$. If $\\|\\x\\|_0<r$, then\n\\begin{multline*}\n\\partial_L^{\\infty} \\delta_{C_r}(\\x) =\\partial_L \\delta_{C_r}(\\x) =\\{\\bv: \\exists \\widehat{J} \\subseteq \\{1,\\dots,d\\} \\smallsetminus \\supp(\\x) \\text{~with~} |\\widehat{J}|=r-\\|\\x\\|_0,\\\\ v_j=0 \\text{~if~} j \\in (\\supp(\\x) \\cup \\widehat{J})\\}.    \n\\end{multline*}\n\n\\item\\label{l:subdiff_sum}\n$\\forall \\x\\in \\Lambda$, $\\partial_L (\\|\\cdot\\|_0+\\delta_{\\Lambda})(\\x) =\\partial_L (\\|\\cdot\\|_0)(\\x) +\\partial_L \\delta_{\\Lambda}(\\x)$.\n\n\\item\\label{l:subdiff_sum'}\n$\\forall \\x\\in \\Lambda \\cap C_r$, $\\partial_L (\\delta_{C_r}+\\delta_{\\Lambda})(\\x)\\subseteq \\partial_L\\delta_{C_r}(\\x) +\\partial_L \\delta_{\\Lambda}(\\x)$.\n\\end{enumerate}\n\\end{lemma}","start":23881,"end":25401},{"raw_tex":"We now consider the global convergence of the full sequence generated by Algorithm~\\ref{algo:epasa}. \nRecall that a proper lower semicontinuous function $f\\colon \\mathcal{H}\\to \\left(-\\infty, +\\infty\\right]$ is said to satisfy the \\emph{KL property} \\cite{Kur98,Loj63} at $\\overline{\\x}\\in \\dom\\partial_L f$ if there exist a neighborhood $U$ of $\\overline{\\x}$, $\\eta \\in \\left(0,+\\infty\\right]$, and a continuous concave function $\\varphi\\colon \\left[0, \\eta\\right)\\to \\mathbb{R}_+$ such that $\\varphi(0) =0$, $\\varphi$ is continuously differentiable with $\\varphi' >0$ on $\\left(0, \\eta\\right)$, and, for all $\\x\\in U$ with $f(\\overline{\\x}) <f(\\x) <f(\\overline{\\x})+\\eta$, \n\\begin{equation*}\n\\varphi'(f(\\x)-f(\\overline{\\x}))\\dist(0,\\partial_L f(\\x))\\geq 1.\n\\end{equation*}\nIf $f$ satisfies the KL property at any $\\overline{\\x}\\in \\dom\\partial_L f$, then it is called a \\emph{KL function}. We say that $f$ has the \\emph{KL property at $\\overline{\\x}\\in \\dom\\partial_L f$ with exponent $\\alpha$} if it satisfies the KL property at $\\overline{\\x}\\in \\dom\\partial_L f$ and the corresponding function $\\varphi$ (often referred as desingularization function) can be chosen as $\\varphi(s) =\\gamma s^{1-\\alpha}$ for some $\\gamma\\in \\RPP$ and $\\alpha\\in \\left[0,1\\right)$. If $f$ is a KL function and has the same exponent $\\alpha$ at any $\\overline{\\x}\\in \\dom\\partial_L f$, then it is called a \\emph{KL function with exponent $\\alpha$}.\n","start":57716,"end":59150},{"raw_tex":"\\begin{lemma}[cf. {\\cite[Theorem 3.6]{LP18}}]\n\\label{l:LP}\nLet $f$ be a proper lower semicontinuous function. Suppose that $f$ satisfies the KL property at $\\overline{\\x} \\in \\dom\\partial_L f$ with exponent $\\alpha \\in [\\frac{1}{2},1)$. Then, for all $\\rho \\geq 0$, $\\widetilde{f}(\\x,\\bu)=f(\\x)+\\rho \\|\\x-\\bu\\|^2$ satisfies  the KL property with exponent $\\alpha$ at $(\\overline{\\x},\\overline{\\x})$.\n\\end{lemma}\n","start":68549,"end":68961},{"raw_tex":"Consider the generalized eigenvalue problem with cardinality regularization\n\\begin{equation}\n\\max_{\\x \\in \\mathbb{R}^d} \\frac{\\x^\\top A_1\\x}{\\x^\\top B_1\\x}- \\lambda \\|\\x\\|_0 \n\\quad\\text{s.t.}\\quad \\|\\x\\|=1,\n\\tag{GEP}\n\\end{equation}\nwhere $A_1,B_1$ are symmetric matrices such that $A_1$ is positive semidefinite and $B_1$ is positive definite, and $\\lambda>0.$ For this generalized eigenvalue problem with cardinality regularization the corresponding merit function for the proposed Algorithm~\\ref{algo:epasa} takes the form\n\\begin{equation*}\n\\widehat{\\Phi}_{GEP}(\\x,\\bu) =\\frac{\\x^\\top A \\x}{\\x^\\top B \\x} +\\lambda \\|\\x\\|_0 +\\delta_{\\Lambda}(\\x) +\\rho \\|\\x-\\bu\\|^2,\n\\end{equation*}\nwith $A=-A_1$ a symmetric matrix,  $B=B_1$ is a positive definite matrix, $\\Lambda =\\{\\x\\in \\mathbb{R}^d: \\|\\x\\|=1\\}$, and $\\rho \\geq 0$. Below, we derive the KL exponent of the merit function $\\widehat{\\Phi}_{GEP}$. To this end, we will use the following lemma from \\cite{LP18}. Here we provide an alternative short proof for it.\n","start":83185,"end":84199},{"raw_tex":"Next we prove that the KL exponent of the merit function $\\widehat{\\Phi}_{GEP}$ is $\\frac{1}{2}$. To do this, for an index set $J=\\{j_1,\\dots,j_k\\} \\subseteq \\{1,\\dots,d\\}$ with $k \\leq d$, we denote $\\x_J:=(x_{j_1},\\dots,x_{j_k})$. Moreover, for two index sets $I,J$, we denote\n$A_{IJ}=(A_{ij})_{i \\in I, j \\in J}.$\n","start":85243,"end":85560}]},"statement":{"raw_tex":"\\begin{theorem}\n\\label{t:KL_0norm_reg}\nConsider the function\n$\\Phi(\\x)=\\frac{\\x^\\top A\\x}{\\x^\\top B\\x}+ \\lambda \\|\\x\\|_0 + \\delta_{\\Lambda}(\\x)$,\nwhere $\\Lambda=\\{\\x: \\|\\x\\|=1\\}$, $A,B$ are symmetric matrices with $B$ positive definite, and $\\lambda>0$. Then $\\Phi$ is a KL function with exponent $\\frac{1}{2}$. In particular, for all $\\rho \\geq 0$,\n\\begin{equation*}\n\\widehat{\\Phi}_{GEP}(\\x,\\bu)= \\frac{\\x^\\top A\\x}{\\x^\\top B\\x}+ \\lambda \\|\\x\\|_0 + \\delta_{\\Lambda}(\\x)+ \\rho \\|\\x-\\bu\\|^2\n\\end{equation*}\nsatisfies the KL property with exponent $\\frac{1}{2}$ at $(\\overline{\\x},\\overline{\\x})$ for all $\\overline{\\x} \\in \\dom\\partial_L \\Phi$ .\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\n\\label{t:KL_0norm_reg}\nConsider the function\n$\\Phi(\\x)=\\frac{\\x^\\top A\\x}{\\x^\\top B\\x}+ \\lambda \\|\\x\\|_0 + \\delta_{\\Lambda}(\\x)$,\nwhere $\\Lambda=\\{\\x: \\|\\x\\|=1\\}$, $A,B$ are symmetric matrices with $B$ positive definite, and $\\lambda>0$. Then $\\Phi$ is a KL function with exponent $\\frac{1}{2}$. In particular, for all $\\rho \\geq 0$,\n\\begin{equation*}\n\\widehat{\\Phi}_{GEP}(\\x,\\bu)= \\frac{\\x^\\top A\\x}{\\x^\\top B\\x}+ \\lambda \\|\\x\\|_0 + \\delta_{\\Lambda}(\\x)+ \\rho \\|\\x-\\bu\\|^2\n\\end{equation*}\nsatisfies the KL property with exponent $\\frac{1}{2}$ at $(\\overline{\\x},\\overline{\\x})$ for all $\\overline{\\x} \\in \\dom\\partial_L \\Phi$ .\n\\end{theorem}","start":85561,"end":86219}]},"proof":{"raw_tex":"\\begin{lemma}\n\\label{l:bound}\nLet $Q$ be a symmetric $d \\times d$ matrix. Then there exists $c>0$ such that, for all $\\x \\in \\mathbb{R}^d$, $\\|Q\\x\\|^2 \\geq c\\, (\\x^\\top Q\\x)$.\n\\end{lemma}\n\n\\begin{proof}\nLet $Q=U^\\top \\Sigma U$ where $U$ is an orthonormal matrix and $\\Sigma={\\rm diag}(\\lambda_1,\\dots,\\lambda_n)$ is a diagonal matrix whose diagonal elements are the eigenvalues of $Q$ with $\\lambda_1 \\leq \\lambda_2 \\leq \\dots \\leq \\lambda_n$. Let $\\x \\in \\mathbb{R}^d$, $\\y:=U\\x$, and \n$I_0=\\{j:\\lambda_j \\neq 0\\}$. Then $\\x^\\top Q\\x= \\sum_{j=1}^N \\lambda_j y_j^2 =\\sum_{j \\in I_0} \\lambda_j y_j^2$ and  \n\\begin{equation*}\n\\|Q\\x\\|^2=\\x^\\top (Q^\\top Q)\\x= (U\\x)^\\top \\Sigma^2 (U\\x)= \\sum_{j=1}^N \\lambda_j^2 y_j^2=\\sum_{j \\in I_0} \\lambda_j^2 y_j^2.\n\\end{equation*}\nSetting $c :=\\min\\{|\\lambda_j|: j \\in I_0\\}$, we see that\n\\begin{equation*}\nc\\, (\\x^\\top Q\\x)= \\sum_{j \\in I_0} c\\lambda_j y_j^2 \\leq \\sum_{j \\in I_0} c|\\lambda_j| y_j^2 \\leq \\sum_{j \\in I_0} |\\lambda_j|^2 y_j^2=\\|Q\\x\\|^2,\n\\end{equation*}\nwhich completes the proof.\n\\end{proof}\n\n\\begin{proof}\nTake any $\\overline{\\x} \\in \\dom\\partial_L \\Phi$. Then $\\overline{\\x}  \\in \\Lambda$.\nLet $J=\\supp(\\overline{\\x})$ and use $|J|$ to denote the cardinality of $J$.  Choose $\\eta \\in (0,1)$ such that, for all $\\|\\x-\\overline{\\x}\\| < \\eta$,\n\\begin{equation*}\n\\left|\\, \\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}-\\frac{\\x^\\top A\\x}{\\x^\\top B\\x}\\, \\right| \\leq \\frac{\\lambda}{4} \\text{ and } \\eta< \\frac{\\lambda}{4}.\n\\end{equation*} Let $\\x$ with $\\|\\x-\\overline{\\x}\\|<\\eta$ and  $\\Phi(\\overline{\\x})<\\Phi(\\x)<\\Phi(\\overline{\\x})+\\eta$.\nWe first see that, by shrinking $\\eta$ if necessary, one can assume that\n\\begin{equation*}\nJ=\\supp(\\overline{\\x})=\\supp(\\x).\n\\end{equation*}\nIndeed, by continuity and by shrinking $\\eta$ if necessary, one has $\\supp(\\overline{\\x})\\subseteq \\supp(\\x)$. Suppose that $\\supp(\\overline{\\x}) \\subsetneq \\supp(\\x)$. Then $\\|\\x\\|_0>\\|\\overline{\\x}\\|_0$, and so,\n$\\|\\x\\|_0 \\geq \\|\\overline{\\x}\\|_0+1$.\nFrom our choice of $\\x$, one has $\\x \\in \\Lambda$ and\n\\begin{equation*}\n\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}+ \\lambda \\|\\overline{\\x}\\|_0< \\frac{\\x^\\top A\\x}{\\x^\\top B\\x}+ \\lambda \\|\\x\\|_0 <\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}+ \\lambda \\|\\overline{\\x}\\|_0 +\\eta. \\end{equation*}\nThis shows that\n$\\|\\x\\|_0<\\frac{1}{2}+ \\|\\overline{\\x}\\|_0$,\nwhich is impossible.\n\nUsing Lemma~\\ref{l:subdiff} and noting that $J =\\supp(\\x)$, we derive that\n\\begin{multline*}\n\\partial_L \\Phi(\\x) \\subseteq \\Bigg\\{\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2} +\\lambda \\bv +t \\x:  t \\in \\mathbb{R}, \\\\\nv_j=0 \\text{~if~} j \\in J, \\text{~and~} v_j \\in \\mathbb{R} \\text{~if~} j \\notin J\\Bigg\\}.\n\\end{multline*}\nDenoting $[a]_J=(a_j)_{j \\in J} \\in \\mathbb{R}^{|J|}$, this implies that\n\\begin{equation*}\n\\dist(0, \\partial_L \\Phi(\\x)) \\geq \\inf_{t \\in \\mathbb{R}} \\left\\|\\left[\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right]_J +t \\x_J\\right\\|.\n\\end{equation*}\nA direct verification shows that\n\\begin{equation*}\n\\x^\\top  \\left( \\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right) =0,\n\\end{equation*}\nwhich, together with $J=\\supp(\\x)$, implies that\n\\begin{equation*}\n(\\x_J)^\\top \\left( \\left[\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right]_J\\right) =0.\n\\end{equation*}\nTherefore,\n\\begin{equation*}\n\\dist(0, \\partial_L \\Phi(\\x)) \\geq \\left\\|\\left[\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right]_J\\right\\|\n=\\frac{2}{\\x^\\top B\\x} \\left\\| [A\\x]_J -\\frac{\\x^\\top A\\x}{\\x^\\top B\\x} [B\\x]_J\\right\\|.\n\\end{equation*}\nUsing $J=\\supp(\\x)$ again, we have that\n\\begin{equation*}\n[A\\x]_J =A_{JJ}\\x_J,\\; \\x^\\top A\\x =(\\x_J)^\\top A_{JJ}\\x_J,\\; [B\\x]_J= B_{JJ}\\x_J, \\text{~and~} \\x^\\top B\\x=(\\x_J)^\\top B_{JJ}\\x_J,\n\\end{equation*}\nand hence $\\dist(0, \\partial_L \\Phi(\\x))\n\\geq \\frac{2}{\\x^\\top B\\x} \\left\\| A_{JJ}\\x_J -\\frac{\\x^\\top A\\x}{\\x^\\top B\\x} \nB_{JJ}\\x_J\\right\\|$.\n\nNow, let $q(\\z) =\\z^\\top A_{JJ}\\z -\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}\\, \\z^\\top B_{JJ}\\z$ for $\\z \\in \\mathbb{R}^{|J|}$.\nThen\n\\begin{align*}\n\\dist(0, \\partial_L \\Phi(\\x))\n& \\geq \\frac{2}{\\x^\\top B\\x} \\left\\| A_{JJ}\\x_J -  \\frac{\\x^\\top A\\x}{\\x^\\top B\\x} \nB_{JJ}\\x_J\\right\\| \\\\\n& \\geq \\frac{2}{\\x^\\top B\\x} \\left(\\left\\| A_{JJ}\\x_J -  \\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}} \nB_{JJ}\\x_J\\right\\| - \\left|\\frac{\\x^\\top A\\x}{\\x^\\top B\\x}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}\n\\right| \\, \\|B_{JJ}\\x_J\\|\\right) \\\\\n& = \\frac{1}{\\x^\\top B\\x} \\|\\nabla q (\\x_J)\\| -\\frac{2}{\\x^\\top B\\x}|\\Phi(\\x)-\\Phi(\\overline{\\x})|\\, \\|B_{JJ}\\x_J\\|,\n\\end{align*}\nwhere the second inequality follows from the triangle inequality and the last equality holds as $\\x, \\overline{\\x} \\in \\Lambda$ and $J=\\supp(\\x)=\\supp(\\overline{\\x})$ (and so, $\\|\\x\\|_0=\\|\\overline{\\x}\\|_0$).\n\nFrom Lemma \\ref{l:bound}, there exists $c>0$ such that, for all $\\z$,\n$\\|\\nabla q(\\z)\\|^2 \\geq c\\, q(\\z)$.\nIndeed, one can set $c :=\\min_{1 \\leq j \\leq |J|} \\left \\{4|\\lambda_j(A_{JJ}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}B_{JJ})|: \\lambda_j(A_{JJ}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}B_{JJ}) \\neq 0 \\right\\} $, where $\\lambda_j(Q)$ are the eigenvalues of a matrix $Q$. Noting that\n\\begin{align*}\n\\frac{q(\\x_J)}{\\x^\\top B\\x} &= \\left[(\\x_J)^\\top A_{JJ}(\\x_J)-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}(\\x_J)^\\top B_{JJ}(\\x_J)\\right] \\frac{1}{\\x^\\top B\\x} \\notag \\\\\n& = \\left[\\x^\\top A\\x - \\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}} \\x^\\top B\\x \\right] \\frac{1}{\\x^\\top B\\x} \\notag \\\\\n& = \\frac{\\x^\\top A\\x}{\\x^\\top B\\x}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}} =\\Phi(\\x)-\\Phi(\\overline{\\x})>0,\n\\end{align*}\none has\n\\begin{align*}\n\\dist(0, \\partial_L \\Phi(\\x))\n&\\geq \\frac{\\sqrt{c} q(\\x_J)^{1/2}}{\\x^\\top B\\x} -2|\\Phi(\\x)-\\Phi(\\overline{\\x})|\\frac{\\|B_{JJ}\\x_J\\|}{\\x^\\top B\\x} \\\\\n& = [\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2} \\left(\\frac{\\sqrt{c}}{\\sqrt{\\x^\\top B\\x}} -2[\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2}\\frac{\\|B_{JJ}\\x_J\\|}{\\x^\\top B\\x}\\right).\n\\end{align*}\nLet \n$c_1 :=\\min\\{\\sqrt{\\x^\\top B\\x}: \\x \\in \\Lambda\\}$ and $c_2 :=\\max\\{\\sqrt{\\x^\\top B\\x}: \\x \\in \\Lambda\\}$. By shrinking $\\eta$ if necessary, we can assume that $\\eta \\in (0,1)$ and\n\\begin{equation*}\n2[\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2}\\frac{\\|B_{JJ}\\x_J\\|}{\\x^\\top B\\x}\\leq 2\\eta^{1/2}\\frac{\\|B_{JJ}\\x_J\\|}{c_1}\\leq \\frac{\\sqrt{c}}{2c_2},\n\\end{equation*}\nwhere the first inequality follows by the fact $\\Phi(\\overline{\\x})<\\Phi(\\x)<\\Phi(\\overline{\\x})+\\eta$. Then, we see that\n\\begin{equation*}\n\\dist(0, \\partial_L \\Phi(\\x))\n\\geq [\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2} \\left(\\frac{\\sqrt{c}}{c_2} -\\frac{\\sqrt{c}}{2c_2}\\right) =\\frac{\\sqrt{c}}{2c_2}[\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2}.\n\\end{equation*}\nThus, $\\Phi$ satisfies the KL property with exponent $\\frac{1}{2}$. This shows that, according to Lemma~\\ref{l:LP},\n$\\widehat{\\Phi}_{GEP}$ satisfies the KL property with exponent $\\frac{1}{2}$ at $\\overline{\\x}$ for all $\\overline{\\x} \\in \\dom\\partial_L \\Phi$.\n\\end{proof}","units":[{"raw_tex":"\\begin{lemma}\n\\label{l:bound}\nLet $Q$ be a symmetric $d \\times d$ matrix. Then there exists $c>0$ such that, for all $\\x \\in \\mathbb{R}^d$, $\\|Q\\x\\|^2 \\geq c\\, (\\x^\\top Q\\x)$.\n\\end{lemma}","start":84199,"end":84386},{"raw_tex":"\\begin{proof}\nLet $Q=U^\\top \\Sigma U$ where $U$ is an orthonormal matrix and $\\Sigma={\\rm diag}(\\lambda_1,\\dots,\\lambda_n)$ is a diagonal matrix whose diagonal elements are the eigenvalues of $Q$ with $\\lambda_1 \\leq \\lambda_2 \\leq \\dots \\leq \\lambda_n$. Let $\\x \\in \\mathbb{R}^d$, $\\y:=U\\x$, and \n$I_0=\\{j:\\lambda_j \\neq 0\\}$. Then $\\x^\\top Q\\x= \\sum_{j=1}^N \\lambda_j y_j^2 =\\sum_{j \\in I_0} \\lambda_j y_j^2$ and  \n\\begin{equation*}\n\\|Q\\x\\|^2=\\x^\\top (Q^\\top Q)\\x= (U\\x)^\\top \\Sigma^2 (U\\x)= \\sum_{j=1}^N \\lambda_j^2 y_j^2=\\sum_{j \\in I_0} \\lambda_j^2 y_j^2.\n\\end{equation*}\nSetting $c :=\\min\\{|\\lambda_j|: j \\in I_0\\}$, we see that\n\\begin{equation*}\nc\\, (\\x^\\top Q\\x)= \\sum_{j \\in I_0} c\\lambda_j y_j^2 \\leq \\sum_{j \\in I_0} c|\\lambda_j| y_j^2 \\leq \\sum_{j \\in I_0} |\\lambda_j|^2 y_j^2=\\|Q\\x\\|^2,\n\\end{equation*}\nwhich completes the proof.\n\\end{proof}","start":84387,"end":85241},{"raw_tex":"\\begin{proof}\nTake any $\\overline{\\x} \\in \\dom\\partial_L \\Phi$. Then $\\overline{\\x}  \\in \\Lambda$.\nLet $J=\\supp(\\overline{\\x})$ and use $|J|$ to denote the cardinality of $J$.  Choose $\\eta \\in (0,1)$ such that, for all $\\|\\x-\\overline{\\x}\\| < \\eta$,\n\\begin{equation*}\n\\left|\\, \\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}-\\frac{\\x^\\top A\\x}{\\x^\\top B\\x}\\, \\right| \\leq \\frac{\\lambda}{4} \\text{ and } \\eta< \\frac{\\lambda}{4}.\n\\end{equation*} Let $\\x$ with $\\|\\x-\\overline{\\x}\\|<\\eta$ and  $\\Phi(\\overline{\\x})<\\Phi(\\x)<\\Phi(\\overline{\\x})+\\eta$.\nWe first see that, by shrinking $\\eta$ if necessary, one can assume that\n\\begin{equation*}\nJ=\\supp(\\overline{\\x})=\\supp(\\x).\n\\end{equation*}\nIndeed, by continuity and by shrinking $\\eta$ if necessary, one has $\\supp(\\overline{\\x})\\subseteq \\supp(\\x)$. Suppose that $\\supp(\\overline{\\x}) \\subsetneq \\supp(\\x)$. Then $\\|\\x\\|_0>\\|\\overline{\\x}\\|_0$, and so,\n$\\|\\x\\|_0 \\geq \\|\\overline{\\x}\\|_0+1$.\nFrom our choice of $\\x$, one has $\\x \\in \\Lambda$ and\n\\begin{equation*}\n\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}+ \\lambda \\|\\overline{\\x}\\|_0< \\frac{\\x^\\top A\\x}{\\x^\\top B\\x}+ \\lambda \\|\\x\\|_0 <\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}+ \\lambda \\|\\overline{\\x}\\|_0 +\\eta. \\end{equation*}\nThis shows that\n$\\|\\x\\|_0<\\frac{1}{2}+ \\|\\overline{\\x}\\|_0$,\nwhich is impossible.\n\nUsing Lemma~\\ref{l:subdiff} and noting that $J =\\supp(\\x)$, we derive that\n\\begin{multline*}\n\\partial_L \\Phi(\\x) \\subseteq \\Bigg\\{\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2} +\\lambda \\bv +t \\x:  t \\in \\mathbb{R}, \\\\\nv_j=0 \\text{~if~} j \\in J, \\text{~and~} v_j \\in \\mathbb{R} \\text{~if~} j \\notin J\\Bigg\\}.\n\\end{multline*}\nDenoting $[a]_J=(a_j)_{j \\in J} \\in \\mathbb{R}^{|J|}$, this implies that\n\\begin{equation*}\n\\dist(0, \\partial_L \\Phi(\\x)) \\geq \\inf_{t \\in \\mathbb{R}} \\left\\|\\left[\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right]_J +t \\x_J\\right\\|.\n\\end{equation*}\nA direct verification shows that\n\\begin{equation*}\n\\x^\\top  \\left( \\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right) =0,\n\\end{equation*}\nwhich, together with $J=\\supp(\\x)$, implies that\n\\begin{equation*}\n(\\x_J)^\\top \\left( \\left[\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right]_J\\right) =0.\n\\end{equation*}\nTherefore,\n\\begin{equation*}\n\\dist(0, \\partial_L \\Phi(\\x)) \\geq \\left\\|\\left[\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right]_J\\right\\|\n=\\frac{2}{\\x^\\top B\\x} \\left\\| [A\\x]_J -\\frac{\\x^\\top A\\x}{\\x^\\top B\\x} [B\\x]_J\\right\\|.\n\\end{equation*}\nUsing $J=\\supp(\\x)$ again, we have that\n\\begin{equation*}\n[A\\x]_J =A_{JJ}\\x_J,\\; \\x^\\top A\\x =(\\x_J)^\\top A_{JJ}\\x_J,\\; [B\\x]_J= B_{JJ}\\x_J, \\text{~and~} \\x^\\top B\\x=(\\x_J)^\\top B_{JJ}\\x_J,\n\\end{equation*}\nand hence $\\dist(0, \\partial_L \\Phi(\\x))\n\\geq \\frac{2}{\\x^\\top B\\x} \\left\\| A_{JJ}\\x_J -\\frac{\\x^\\top A\\x}{\\x^\\top B\\x} \nB_{JJ}\\x_J\\right\\|$.\n\nNow, let $q(\\z) =\\z^\\top A_{JJ}\\z -\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}\\, \\z^\\top B_{JJ}\\z$ for $\\z \\in \\mathbb{R}^{|J|}$.\nThen\n\\begin{align*}\n\\dist(0, \\partial_L \\Phi(\\x))\n& \\geq \\frac{2}{\\x^\\top B\\x} \\left\\| A_{JJ}\\x_J -  \\frac{\\x^\\top A\\x}{\\x^\\top B\\x} \nB_{JJ}\\x_J\\right\\| \\\\\n& \\geq \\frac{2}{\\x^\\top B\\x} \\left(\\left\\| A_{JJ}\\x_J -  \\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}} \nB_{JJ}\\x_J\\right\\| - \\left|\\frac{\\x^\\top A\\x}{\\x^\\top B\\x}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}\n\\right| \\, \\|B_{JJ}\\x_J\\|\\right) \\\\\n& = \\frac{1}{\\x^\\top B\\x} \\|\\nabla q (\\x_J)\\| -\\frac{2}{\\x^\\top B\\x}|\\Phi(\\x)-\\Phi(\\overline{\\x})|\\, \\|B_{JJ}\\x_J\\|,\n\\end{align*}\nwhere the second inequality follows from the triangle inequality and the last equality holds as $\\x, \\overline{\\x} \\in \\Lambda$ and $J=\\supp(\\x)=\\supp(\\overline{\\x})$ (and so, $\\|\\x\\|_0=\\|\\overline{\\x}\\|_0$).\n\nFrom Lemma \\ref{l:bound}, there exists $c>0$ such that, for all $\\z$,\n$\\|\\nabla q(\\z)\\|^2 \\geq c\\, q(\\z)$.\nIndeed, one can set $c :=\\min_{1 \\leq j \\leq |J|} \\left \\{4|\\lambda_j(A_{JJ}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}B_{JJ})|: \\lambda_j(A_{JJ}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}B_{JJ}) \\neq 0 \\right\\} $, where $\\lambda_j(Q)$ are the eigenvalues of a matrix $Q$. Noting that\n\\begin{align*}\n\\frac{q(\\x_J)}{\\x^\\top B\\x} &= \\left[(\\x_J)^\\top A_{JJ}(\\x_J)-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}(\\x_J)^\\top B_{JJ}(\\x_J)\\right] \\frac{1}{\\x^\\top B\\x} \\notag \\\\\n& = \\left[\\x^\\top A\\x - \\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}} \\x^\\top B\\x \\right] \\frac{1}{\\x^\\top B\\x} \\notag \\\\\n& = \\frac{\\x^\\top A\\x}{\\x^\\top B\\x}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}} =\\Phi(\\x)-\\Phi(\\overline{\\x})>0,\n\\end{align*}\none has\n\\begin{align*}\n\\dist(0, \\partial_L \\Phi(\\x))\n&\\geq \\frac{\\sqrt{c} q(\\x_J)^{1/2}}{\\x^\\top B\\x} -2|\\Phi(\\x)-\\Phi(\\overline{\\x})|\\frac{\\|B_{JJ}\\x_J\\|}{\\x^\\top B\\x} \\\\\n& = [\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2} \\left(\\frac{\\sqrt{c}}{\\sqrt{\\x^\\top B\\x}} -2[\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2}\\frac{\\|B_{JJ}\\x_J\\|}{\\x^\\top B\\x}\\right).\n\\end{align*}\nLet \n$c_1 :=\\min\\{\\sqrt{\\x^\\top B\\x}: \\x \\in \\Lambda\\}$ and $c_2 :=\\max\\{\\sqrt{\\x^\\top B\\x}: \\x \\in \\Lambda\\}$. By shrinking $\\eta$ if necessary, we can assume that $\\eta \\in (0,1)$ and\n\\begin{equation*}\n2[\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2}\\frac{\\|B_{JJ}\\x_J\\|}{\\x^\\top B\\x}\\leq 2\\eta^{1/2}\\frac{\\|B_{JJ}\\x_J\\|}{c_1}\\leq \\frac{\\sqrt{c}}{2c_2},\n\\end{equation*}\nwhere the first inequality follows by the fact $\\Phi(\\overline{\\x})<\\Phi(\\x)<\\Phi(\\overline{\\x})+\\eta$. Then, we see that\n\\begin{equation*}\n\\dist(0, \\partial_L \\Phi(\\x))\n\\geq [\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2} \\left(\\frac{\\sqrt{c}}{c_2} -\\frac{\\sqrt{c}}{2c_2}\\right) =\\frac{\\sqrt{c}}{2c_2}[\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2}.\n\\end{equation*}\nThus, $\\Phi$ satisfies the KL property with exponent $\\frac{1}{2}$. This shows that, according to Lemma~\\ref{l:LP},\n$\\widehat{\\Phi}_{GEP}$ satisfies the KL property with exponent $\\frac{1}{2}$ at $\\overline{\\x}$ for all $\\overline{\\x} \\in \\dom\\partial_L \\Phi$.\n\\end{proof}","start":86220,"end":92458}]},"nodes":[{"id":"S001","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"有限维空间与基本记号","source_units":[{"raw_tex":"In this section, we recall some basic notations and preliminary results which will be used in this paper. We assume throughout that $\\mathcal{H}$, $\\mathcal{H}_1$, \\dots, $\\mathcal{H}_m$ are finite-dimensional real Hilbert spaces\nwith inner product $\\scal{\\cdot}{\\cdot}$ and induced norm $\\|\\cdot\\|$. The product space $\\mathcal{H}_1 \\times \\dots \\times \\mathcal{H}_m$ is also a real Hilbert space endowed with the inner product given by\n    $\\langle (x_1,\\dots,x_m),(y_1,\\dots,y_m) \\rangle = \\sum_{i=1}^m \\langle x_i,y_i \\rangle.$\nThe set of nonnegative integers is denoted by $\\mathbb{N}$, the set of real numbers by $\\mathbb{R}$, the set of nonnegative real numbers by $\\RP$, and the set of the positive real numbers by $\\RPP$.\n\n","start":15915,"end":16647}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"指标函数与函数基本性质","source_units":[{"raw_tex":"The \\emph{indicator function} of a set $C$ is defined by $\\delta_C(\\x) :=0$ if $\\x \\in C$, and $\\delta_C(\\x) :=+\\infty$ otherwise. Given an extended-real-valued function $f\\colon \\mathcal{H}\\to \\left[-\\infty,+\\infty\\right]$, its \\emph{domain} is defined by $\\dom f :=\\{\\x \\in \\mathcal{H}: f(\\x)<+\\infty\\}$. The function $f$ is \\emph{proper} if $\\dom f\\neq \\varnothing$ and it never equals $-\\infty$. We say that $f$ is \\emph{lower semicontinuous} if, for all $\\x\\in  \\mathcal{H}$, $f(\\x)\\leq \\liminf_{\\y\\to \\x} f(\\y)$, and \\emph{convex} if its epigraph $\\{(\\x,\\rho)\\in \\mathcal{H}\\times \\mathbb{R}: f(\\x)\\leq \\rho\\}$ is a convex subset of $\\mathcal{H}\\times \\mathbb{R}$. The function $f$ is said to be \\emph{weakly convex (on $\\mathcal{H}$)} if there exists $\\alpha \\geq 0$ such that $f+\\frac{\\alpha}{2}\\|\\cdot\\|^2$ is a convex function. The smallest constant $\\alpha$ such that $f+\\frac{\\alpha}{2}\\|\\cdot\\|^2$ is convex is called the \\emph{modulus} of weak convexity. More generally, $f$ is said to be \\emph{weakly convex on $S \\subseteq \\mathcal{H}$ with modulus $\\alpha$} if $f+\\delta_S$ is weakly convex with modulus $\\alpha$. Weakly convex functions form a broad class of functions which covers quadratic functions, convex functions, differentiable functions whose gradient is Lipschitz continuous, and the composition of a convex and Lipschitz continuous function with a $C^1$-smooth mapping whose Jacobian is Lipschitz continuous (see \\cite[Lemma~4.2]{DP19}).\n\n\n","start":16647,"end":18116}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"Fréchet与极限次微分","source_units":[{"raw_tex":"Let $f\\colon\\mathcal{H} \\to \\left[-\\infty,+\\infty\\right]$ and $\\x\\in \\mathcal{H}$ with $|f(\\x)| <+\\infty$. The \\emph{Fr\\'echet subdifferential} of $f$ at $\\x$ is given by\n\\begin{equation*}\n\\widehat{\\partial} f(\\x) :=\\left\\{\\bu\\in \\mathcal{H}:\\; \\liminf_{\\z\\to \\x}\\frac{f(\\z)-f(\\x)-\\langle \\bu,\\z-\\x\\rangle}{\\|\\z-\\x\\|}\\geq 0\\right\\},\n\\end{equation*}\nthe \\emph{limiting subdifferential} of $f$ at $\\x$ is given by\n\\begin{equation*}\n\\partial_L f(\\x) :=\\left\\{\\bu\\in \\mathcal{H}:\\; \\exists \\x_n\\st{f}{\\to}\\x,\\ \\bu_n\\to \\bu\\text{~~with~~} \\bu_n\\in \\widehat{\\partial} f(\\x_n)\\right\\},\n\\end{equation*}\n","start":18116,"end":18711}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"水平次微分","source_units":[{"raw_tex":"and the \\emph{horizon subdifferential} of $f$ at $\\x$ is given by\n\\begin{equation*}\n\\partial^{\\infty}f(\\x) :=\\left\\{\\bu\\in \\mathcal{H}:\\; \\exists \\x_n\\st{f}{\\to}\\x,\\ \\lambda_n\\downarrow 0,\\ \\lambda_n\\bu_n\\to \\bu\\text{~~with~~} \\bu_n\\in \\widehat{\\partial} f(\\x_n)\\right\\}.\n\\end{equation*}\n","start":18711,"end":18999}]},{"id":"S005","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"极限次微分稳健性","source_units":[{"raw_tex":"Here, the notation $\\z\\st{f}{\\to}\\x$ means $\\z\\to \\x$ with $f(\\z)\\to f(\\x)$. It follows from the above definition that the limiting subdifferential has the following \\emph{robustness property}\n\\begin{equation*}\n\\partial_L f(\\x) =\\left\\{\\bu\\in \\mathcal{H}:\\; \\exists \\x_n \\st{f}{\\to}\\x,\\ \\bu_n\\to \\bu \\text{~~with~~} \\bu_n\\in \\partial_L f(\\x_n)\\right\\}.\n\\end{equation*}\n","start":18999,"end":19368}]},{"id":"S006","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"光滑函数与次微分","source_units":[{"raw_tex":"The \\emph{domain} of $\\partial_L f$ is $\\dom\\partial_L f :=\\{\\x\\in \\mathcal{H}: \\partial_L f(\\x)\\neq \\varnothing\\}$. If $f$ is Lipschitz continuous around $\\x$, then $\\partial_L f(\\x)$ is bounded and $\\partial^{\\infty} f(\\x) =\\{0\\}$ (see \\cite[Corollary~1.81]{Mor06}). If $f$ is strictly differentiable\\footnote{A function $f$ is strictly differentiable at $\\x$ if there exists $\\bu \\in \\mathcal{H}$ such that $\\lim_{\\y,\\z \\to \\x}\\frac{f(\\y)-f(\\z)-\\langle \\bu,\\y-\\z\\rangle}{\\|\\y-\\z\\|} =0$. Clearly, if $f$ is continuously differentiable at $\\x$, then it is strictly differentiable at $\\x$.} at $\\x$, then $\\widehat{\\partial} f$ and $\\partial_L f$ reduce to the derivative of $f$, denoted by $\\nabla f$ (see \\cite[Corollary~1.82]{Mor06}). If $f$ is convex, then both Fr\\'echet and limiting subdifferentials at $\\x$ reduce to the classical subdifferential in convex analysis (see \\cite[Theorem~1.93]{Mor06})\n\\begin{equation*}\n\\partial f(\\x) :=\\left\\{\\bu\\in \\mathcal{H}:\\; \\forall \\z\\in \\mathcal{H},\\; \\langle \\bu,\\z-\\x\\rangle\\leq f(\\z)-f(\\x)\\right\\}.\n\\end{equation*}\n","start":19368,"end":20433}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"函数与集合的正则性","source_units":[{"raw_tex":"We say that $f$ is \\emph{regular\\footnote{This is also referred as \\emph{lower regular} in \\cite{Mor06,MNY06}.} at $\\x\\in \\mathcal{H}$} if $\\widehat{\\partial} f(\\x) =\\partial_L f(\\x)$, and that $f$ is \\emph{regular on $C\\subseteq \\mathcal{H}$} if it is regular at any $\\x\\in C$. For a  proper lower semicontinuous function $f$, it is clear that if $f$ is convex around $\\x$ or strictly differentiable at $\\x$, then it is regular at $\\x$. A nonempty set $S$ in $\\mathcal{H}$ is \\emph{regular} at $\\x\\in S$ if $\\delta_S$ is regular at $\\x$. We say that $S$ is regular if it is regular at all of its points. From the definition, it can be verified that $C$ is regular if $C$ is a closed and convex set or $C$ is a smooth manifold given by $C =\\{\\x\\in \\mathcal{H}: g_i(\\x)=0,\\, i=1,\\dots, m\\}$, where $g_i$ are smooth functions satisfying the so-called linear independent constraint qualification (that is, $\\{\\nabla g_i(\\x): i=1,\\dots,m\\}$ are linearly independent for all $\\x \\in C$).\n","start":20433,"end":21416}]},{"id":"S008","semantic_type":"lemma","kind":"lemma","labels":["l:calrules","l:calrules_quotient","l:calrules_separable","l:calrules_sign","l:calrules_sqrt","l:calrules_sum"],"declared_external":false,"source_only":false,"title":"次微分演算法则总述","source_units":[{"raw_tex":"\\begin{lemma}[calculus rules]\n\\label{l:calrules}\nLet $f,g\\colon \\mathcal{H}\\to \\left(-\\infty,+\\infty\\right]$ be proper lower semicontinuous functions and let $\\x \\in \\dom f$. Then the following statements hold:\n\\begin{enumerate}\n\\item\\label{l:calrules_separable}\n(Separable sum rule) If $f(\\x) =\\sum_{i=1}^m f_i (x_i)$ with $\\x =(x_1,\\dots,x_m)$, then $\\partial_L f(\\x) =\\partial_L f_1(x_1) \\times \\dots \\times \\partial_L f_m(x_m)$ and $f$ is regular at $\\x$ when each $f_i$ regular at $x_i$.\n\n\\item\\label{l:calrules_sum}\n(Sum rule)\nIf $\\partial^{\\infty} f(\\x) \\cap (-\\partial^{\\infty} g(\\x)) =\\{0\\}$, then $\\partial_L(f+g)(\\x) \\subseteq \\partial_L f(\\x) + \\partial_L g(\\x)$, where the equality holds when both $f$ and $g$ are regular at $\\x$, in which case $f+g$ is also regular at $\\x$. Moreover, if $g$ is strictly differentiable at $\\x$, then $\\partial_L (f+g)(\\x) =\\partial_L f(\\x) +\\nabla g(\\x)$.\n\n\\item\\label{l:calrules_sign}\n(Sign rule)\nIf $f$ is Lipschitz continuous around $\\x$ and $\\widehat{\\partial} f$ is nonempty-valued around $\\x$, then $\\partial_L (-f)(\\x)\\subseteq -\\partial_L f(\\x)$. \n\n\\item\\label{l:calrules_quotient}\n(Quotient rule) \nSuppose that $f$ and $g$ are Lipschitz continuous around $\\x$, and $g(\\x)\\neq 0$. If $\\widehat{\\partial} f$ is nonempty-valued around $\\x$, then\n\\begin{equation*}\n\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) \\subseteq \\frac{-g(\\x)\\partial_L f(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}.\n\\end{equation*}\nIf $f$ is strictly differentiable at $\\x$, then\n\\begin{equation*}\n\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{-g(\\x)\\nabla f(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}\n\\end{equation*}\nand, consequently, $-f/g$ is regular at $\\x$ if and only if $f(\\x)g$ is regular at $\\x$.\n\n\\item\\label{l:calrules_sqrt}\n(Chain rule and square root rule)\nIf $f$ is Lipschitz continuous around $\\x$ and $\\theta\\colon \\mathbb{R}\\to \\mathbb{R}$ is continuously differentiable around $f(\\x)$,  \nthen $\\partial_L(\\theta \\circ f)(\\x) =\\partial_L (\\theta'(f(\\x))f)(\\x)$. In particular, if $f$ is Lipschitz continuous around $\\x$ and $f(\\x)>0$, then\n\\begin{equation*}\n\\partial_L\\left(\\sqrt{f}\\right)(\\x) =\\frac{\\partial_L f(\\x)}{2 \\sqrt{f(\\x)}} \\text{~~and~~} \\partial_L\\left(-\\sqrt{f}\\right)(\\x) =\\frac{\\partial_L (-f)(\\x)}{2 \\sqrt{f(\\x)}}.\n\\end{equation*}\n\\end{enumerate}\n\\end{lemma}","start":21574,"end":23878}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","labels":["l:calrules","l:calrules_quotient","l:calrules_separable","l:calrules_sign","l:calrules_sqrt","l:calrules_sum"],"declared_external":false,"source_only":false,"title":"演算法则的共同条件","source_units":[{"raw_tex":"\\begin{lemma}[calculus rules]\n\\label{l:calrules}\nLet $f,g\\colon \\mathcal{H}\\to \\left(-\\infty,+\\infty\\right]$ be proper lower semicontinuous functions and let $\\x \\in \\dom f$. Then the following statements hold:\n\\begin{enumerate}\n","start":21574,"end":21803},{"raw_tex":"\\end{enumerate}\n\\end{lemma}","start":23851,"end":23878}]},{"id":"S010","semantic_type":"lemma","kind":"lemma","labels":["l:calrules","l:calrules_quotient","l:calrules_separable","l:calrules_sign","l:calrules_sqrt","l:calrules_sum"],"declared_external":false,"source_only":false,"title":"可分和法则","source_units":[{"raw_tex":"\\item\\label{l:calrules_separable}\n(Separable sum rule) If $f(\\x) =\\sum_{i=1}^m f_i (x_i)$ with $\\x =(x_1,\\dots,x_m)$, then $\\partial_L f(\\x) =\\partial_L f_1(x_1) \\times \\dots \\times \\partial_L f_m(x_m)$ and $f$ is regular at $\\x$ when each $f_i$ regular at $x_i$.\n\n","start":21803,"end":22068}]},{"id":"S011","semantic_type":"lemma","kind":"lemma","labels":["l:calrules","l:calrules_quotient","l:calrules_separable","l:calrules_sign","l:calrules_sqrt","l:calrules_sum"],"declared_external":false,"source_only":false,"title":"一般和与光滑加项法则","source_units":[{"raw_tex":"\\item\\label{l:calrules_sum}\n(Sum rule)\nIf $\\partial^{\\infty} f(\\x) \\cap (-\\partial^{\\infty} g(\\x)) =\\{0\\}$, then $\\partial_L(f+g)(\\x) \\subseteq \\partial_L f(\\x) + \\partial_L g(\\x)$, where the equality holds when both $f$ and $g$ are regular at $\\x$, in which case $f+g$ is also regular at $\\x$. Moreover, if $g$ is strictly differentiable at $\\x$, then $\\partial_L (f+g)(\\x) =\\partial_L f(\\x) +\\nabla g(\\x)$.\n\n","start":22068,"end":22478}]},{"id":"S012","semantic_type":"lemma","kind":"lemma","labels":["l:calrules","l:calrules_quotient","l:calrules_separable","l:calrules_sign","l:calrules_sqrt","l:calrules_sum"],"declared_external":false,"source_only":false,"title":"变号包含法则","source_units":[{"raw_tex":"\\item\\label{l:calrules_sign}\n(Sign rule)\nIf $f$ is Lipschitz continuous around $\\x$ and $\\widehat{\\partial} f$ is nonempty-valued around $\\x$, then $\\partial_L (-f)(\\x)\\subseteq -\\partial_L f(\\x)$. \n\n","start":22478,"end":22678}]},{"id":"S013","semantic_type":"lemma","kind":"lemma","labels":["l:calrules","l:calrules_quotient","l:calrules_separable","l:calrules_sign","l:calrules_sqrt","l:calrules_sum"],"declared_external":false,"source_only":false,"title":"商法则与正则性","source_units":[{"raw_tex":"\\item\\label{l:calrules_quotient}\n(Quotient rule) \nSuppose that $f$ and $g$ are Lipschitz continuous around $\\x$, and $g(\\x)\\neq 0$. If $\\widehat{\\partial} f$ is nonempty-valued around $\\x$, then\n\\begin{equation*}\n\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) \\subseteq \\frac{-g(\\x)\\partial_L f(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}.\n\\end{equation*}\nIf $f$ is strictly differentiable at $\\x$, then\n\\begin{equation*}\n\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{-g(\\x)\\nabla f(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}\n\\end{equation*}\nand, consequently, $-f/g$ is regular at $\\x$ if and only if $f(\\x)g$ is regular at $\\x$.\n\n","start":22678,"end":23293}]},{"id":"S014","semantic_type":"lemma","kind":"lemma","labels":["l:calrules","l:calrules_quotient","l:calrules_separable","l:calrules_sign","l:calrules_sqrt","l:calrules_sum"],"declared_external":false,"source_only":false,"title":"复合与平方根法则","source_units":[{"raw_tex":"\\item\\label{l:calrules_sqrt}\n(Chain rule and square root rule)\nIf $f$ is Lipschitz continuous around $\\x$ and $\\theta\\colon \\mathbb{R}\\to \\mathbb{R}$ is continuously differentiable around $f(\\x)$,  \nthen $\\partial_L(\\theta \\circ f)(\\x) =\\partial_L (\\theta'(f(\\x))f)(\\x)$. In particular, if $f$ is Lipschitz continuous around $\\x$ and $f(\\x)>0$, then\n\\begin{equation*}\n\\partial_L\\left(\\sqrt{f}\\right)(\\x) =\\frac{\\partial_L f(\\x)}{2 \\sqrt{f(\\x)}} \\text{~~and~~} \\partial_L\\left(-\\sqrt{f}\\right)(\\x) =\\frac{\\partial_L (-f)(\\x)}{2 \\sqrt{f(\\x)}}.\n\\end{equation*}\n","start":23293,"end":23851}]},{"id":"S015","semantic_type":"lemma","kind":"lemma","labels":["l:subdiff","l:subdiff_ball","l:subdiff_l0","l:subdiff_sphere","l:subdiff_sum","l:subdiff_sum'"],"declared_external":false,"source_only":false,"title":"稀疏项与球面次微分总述","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{l:subdiff}\nLet $\\Lambda :=\\{\\x\\in \\mathbb{R}^d: \\|\\x\\|=1\\}$ and $C_r :=\\{\\x\\in \\mathbb{R}^d: \\|\\x\\|_0 \\leq r\\}$. Given $\\x =(x_1,\\dots,x_d)\\in \\mathbb{R}^d$, set $\\supp(\\x): =\\{j: x_j \\neq 0\\}$. Then the following statements hold:\n\\begin{enumerate}\n\\item\\label{l:subdiff_l0}\n$\\forall \\x\\in \\mathbb{R}^d$, $\\widehat{\\partial} (\\|\\cdot\\|_0)(\\x) =\\partial_L (\\|\\cdot\\|_0)(\\x) =\\partial_L^{\\infty} (\\|\\cdot\\|_0)(\\x) =\\{\\bv: v_j=0 \\text{~if~} j \\in \\supp(\\x)\\}$.\n\n\\item\\label{l:subdiff_sphere}\n$\\forall \\x\\in \\Lambda$, $\\widehat{\\partial} \\delta_{\\Lambda}(\\x) =\\partial_L \\delta_{\\Lambda}(\\x) =\\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{ t\\x: t \\in \\mathbb{R}\\}$.\n\n\\item\\label{l:subdiff_ball}\nIf $\\|\\x\\|_0=r$, then $\\partial_L^{\\infty} \\delta_{C_r}(\\x) =\\partial_L \\delta_{C_r}(\\x) =\\{\\bv: v_j=0 \\text{~if~} j \\in \\supp(\\x)\\}$. If $\\|\\x\\|_0<r$, then\n\\begin{multline*}\n\\partial_L^{\\infty} \\delta_{C_r}(\\x) =\\partial_L \\delta_{C_r}(\\x) =\\{\\bv: \\exists \\widehat{J} \\subseteq \\{1,\\dots,d\\} \\smallsetminus \\supp(\\x) \\text{~with~} |\\widehat{J}|=r-\\|\\x\\|_0,\\\\ v_j=0 \\text{~if~} j \\in (\\supp(\\x) \\cup \\widehat{J})\\}.    \n\\end{multline*}\n\n\\item\\label{l:subdiff_sum}\n$\\forall \\x\\in \\Lambda$, $\\partial_L (\\|\\cdot\\|_0+\\delta_{\\Lambda})(\\x) =\\partial_L (\\|\\cdot\\|_0)(\\x) +\\partial_L \\delta_{\\Lambda}(\\x)$.\n\n\\item\\label{l:subdiff_sum'}\n$\\forall \\x\\in \\Lambda \\cap C_r$, $\\partial_L (\\delta_{C_r}+\\delta_{\\Lambda})(\\x)\\subseteq \\partial_L\\delta_{C_r}(\\x) +\\partial_L \\delta_{\\Lambda}(\\x)$.\n\\end{enumerate}\n\\end{lemma}","start":23881,"end":25401}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":["l:subdiff","l:subdiff_ball","l:subdiff_l0","l:subdiff_sphere","l:subdiff_sum","l:subdiff_sum'"],"declared_external":false,"source_only":false,"title":"球面、稀疏集与支撑","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{l:subdiff}\nLet $\\Lambda :=\\{\\x\\in \\mathbb{R}^d: \\|\\x\\|=1\\}$ and $C_r :=\\{\\x\\in \\mathbb{R}^d: \\|\\x\\|_0 \\leq r\\}$. Given $\\x =(x_1,\\dots,x_d)\\in \\mathbb{R}^d$, set $\\supp(\\x): =\\{j: x_j \\neq 0\\}$. Then the following statements hold:\n\\begin{enumerate}\n","start":23881,"end":24151},{"raw_tex":"\\end{enumerate}\n\\end{lemma}","start":25374,"end":25401}]},{"id":"S017","semantic_type":"lemma","kind":"lemma","labels":["l:subdiff","l:subdiff_ball","l:subdiff_l0","l:subdiff_sphere","l:subdiff_sum","l:subdiff_sum'"],"declared_external":false,"source_only":false,"title":"基数函数次微分","source_units":[{"raw_tex":"\\item\\label{l:subdiff_l0}\n$\\forall \\x\\in \\mathbb{R}^d$, $\\widehat{\\partial} (\\|\\cdot\\|_0)(\\x) =\\partial_L (\\|\\cdot\\|_0)(\\x) =\\partial_L^{\\infty} (\\|\\cdot\\|_0)(\\x) =\\{\\bv: v_j=0 \\text{~if~} j \\in \\supp(\\x)\\}$.\n\n","start":24151,"end":24361}]},{"id":"S018","semantic_type":"lemma","kind":"lemma","labels":["l:subdiff","l:subdiff_ball","l:subdiff_l0","l:subdiff_sphere","l:subdiff_sum","l:subdiff_sum'"],"declared_external":false,"source_only":false,"title":"球面指标次微分","source_units":[{"raw_tex":"\\item\\label{l:subdiff_sphere}\n$\\forall \\x\\in \\Lambda$, $\\widehat{\\partial} \\delta_{\\Lambda}(\\x) =\\partial_L \\delta_{\\Lambda}(\\x) =\\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{ t\\x: t \\in \\mathbb{R}\\}$.\n\n","start":24361,"end":24563}]},{"id":"S019","semantic_type":"lemma","kind":"lemma","labels":["l:subdiff","l:subdiff_ball","l:subdiff_l0","l:subdiff_sphere","l:subdiff_sum","l:subdiff_sum'"],"declared_external":false,"source_only":false,"title":"稀疏集合的法向公式","source_units":[{"raw_tex":"\\item\\label{l:subdiff_ball}\nIf $\\|\\x\\|_0=r$, then $\\partial_L^{\\infty} \\delta_{C_r}(\\x) =\\partial_L \\delta_{C_r}(\\x) =\\{\\bv: v_j=0 \\text{~if~} j \\in \\supp(\\x)\\}$. If $\\|\\x\\|_0<r$, then\n\\begin{multline*}\n\\partial_L^{\\infty} \\delta_{C_r}(\\x) =\\partial_L \\delta_{C_r}(\\x) =\\{\\bv: \\exists \\widehat{J} \\subseteq \\{1,\\dots,d\\} \\smallsetminus \\supp(\\x) \\text{~with~} |\\widehat{J}|=r-\\|\\x\\|_0,\\\\ v_j=0 \\text{~if~} j \\in (\\supp(\\x) \\cup \\widehat{J})\\}.    \n\\end{multline*}\n\n","start":24563,"end":25028}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","labels":["l:subdiff","l:subdiff_ball","l:subdiff_l0","l:subdiff_sphere","l:subdiff_sum","l:subdiff_sum'"],"declared_external":false,"source_only":false,"title":"基数项与球面指标求和","source_units":[{"raw_tex":"\\item\\label{l:subdiff_sum}\n$\\forall \\x\\in \\Lambda$, $\\partial_L (\\|\\cdot\\|_0+\\delta_{\\Lambda})(\\x) =\\partial_L (\\|\\cdot\\|_0)(\\x) +\\partial_L \\delta_{\\Lambda}(\\x)$.\n\n","start":25028,"end":25193}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","labels":["l:subdiff","l:subdiff_ball","l:subdiff_l0","l:subdiff_sphere","l:subdiff_sum","l:subdiff_sum'"],"declared_external":false,"source_only":false,"title":"稀疏指标与球面指标求和","source_units":[{"raw_tex":"\\item\\label{l:subdiff_sum'}\n$\\forall \\x\\in \\Lambda \\cap C_r$, $\\partial_L (\\delta_{C_r}+\\delta_{\\Lambda})(\\x)\\subseteq \\partial_L\\delta_{C_r}(\\x) +\\partial_L \\delta_{\\Lambda}(\\x)$.\n","start":25193,"end":25374}]},{"id":"S022","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"KL性质定义","source_units":[{"raw_tex":"We now consider the global convergence of the full sequence generated by Algorithm~\\ref{algo:epasa}. \nRecall that a proper lower semicontinuous function $f\\colon \\mathcal{H}\\to \\left(-\\infty, +\\infty\\right]$ is said to satisfy the \\emph{KL property} \\cite{Kur98,Loj63} at $\\overline{\\x}\\in \\dom\\partial_L f$ if there exist a neighborhood $U$ of $\\overline{\\x}$, $\\eta \\in \\left(0,+\\infty\\right]$, and a continuous concave function $\\varphi\\colon \\left[0, \\eta\\right)\\to \\mathbb{R}_+$ such that $\\varphi(0) =0$, $\\varphi$ is continuously differentiable with $\\varphi' >0$ on $\\left(0, \\eta\\right)$, and, for all $\\x\\in U$ with $f(\\overline{\\x}) <f(\\x) <f(\\overline{\\x})+\\eta$, \n\\begin{equation*}\n\\varphi'(f(\\x)-f(\\overline{\\x}))\\dist(0,\\partial_L f(\\x))\\geq 1.\n\\end{equation*}\n","start":57716,"end":58492}]},{"id":"S023","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"KL指数与KL函数","source_units":[{"raw_tex":"If $f$ satisfies the KL property at any $\\overline{\\x}\\in \\dom\\partial_L f$, then it is called a \\emph{KL function}. We say that $f$ has the \\emph{KL property at $\\overline{\\x}\\in \\dom\\partial_L f$ with exponent $\\alpha$} if it satisfies the KL property at $\\overline{\\x}\\in \\dom\\partial_L f$ and the corresponding function $\\varphi$ (often referred as desingularization function) can be chosen as $\\varphi(s) =\\gamma s^{1-\\alpha}$ for some $\\gamma\\in \\RPP$ and $\\alpha\\in \\left[0,1\\right)$. If $f$ is a KL function and has the same exponent $\\alpha$ at any $\\overline{\\x}\\in \\dom\\partial_L f$, then it is called a \\emph{KL function with exponent $\\alpha$}.\n","start":58492,"end":59150}]},{"id":"S024","semantic_type":"external_fact","kind":"external_fact","labels":["l:LP"],"declared_external":true,"declaration_text":["原先决条件：LP18, Theorem 3.6；原文在B0091明确转述。"],"source_only":false,"title":"二次提升保留KL指数","source_units":[{"raw_tex":"\\begin{lemma}[cf. {\\cite[Theorem 3.6]{LP18}}]\n\\label{l:LP}\nLet $f$ be a proper lower semicontinuous function. Suppose that $f$ satisfies the KL property at $\\overline{\\x} \\in \\dom\\partial_L f$ with exponent $\\alpha \\in [\\frac{1}{2},1)$. Then, for all $\\rho \\geq 0$, $\\widetilde{f}(\\x,\\bu)=f(\\x)+\\rho \\|\\x-\\bu\\|^2$ satisfies  the KL property with exponent $\\alpha$ at $(\\overline{\\x},\\overline{\\x})$.\n\\end{lemma}\n","start":68549,"end":68961}]},{"id":"S025","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"基数正则广义特征值问题","source_units":[{"raw_tex":"Consider the generalized eigenvalue problem with cardinality regularization\n\\begin{equation}\n\\max_{\\x \\in \\mathbb{R}^d} \\frac{\\x^\\top A_1\\x}{\\x^\\top B_1\\x}- \\lambda \\|\\x\\|_0 \n\\quad\\text{s.t.}\\quad \\|\\x\\|=1,\n\\tag{GEP}\n\\end{equation}\nwhere $A_1,B_1$ are symmetric matrices such that $A_1$ is positive semidefinite and $B_1$ is positive definite, and $\\lambda>0.$ For this generalized eigenvalue problem with cardinality regularization the corresponding merit function for the proposed Algorithm~\\ref{algo:epasa} takes the form\n","start":83185,"end":83710}]},{"id":"S026","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"应用中的提升评价函数","source_units":[{"raw_tex":"\\begin{equation*}\n\\widehat{\\Phi}_{GEP}(\\x,\\bu) =\\frac{\\x^\\top A \\x}{\\x^\\top B \\x} +\\lambda \\|\\x\\|_0 +\\delta_{\\Lambda}(\\x) +\\rho \\|\\x-\\bu\\|^2,\n\\end{equation*}\nwith $A=-A_1$ a symmetric matrix,  $B=B_1$ is a positive definite matrix, $\\Lambda =\\{\\x\\in \\mathbb{R}^d: \\|\\x\\|=1\\}$, and $\\rho \\geq 0$. Below, we derive the KL exponent of the merit function $\\widehat{\\Phi}_{GEP}$. To this end, we will use the following lemma from \\cite{LP18}. Here we provide an alternative short proof for it.\n","start":83710,"end":84199}]},{"id":"S027","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"坐标限制与矩阵子块","source_units":[{"raw_tex":"Next we prove that the KL exponent of the merit function $\\widehat{\\Phi}_{GEP}$ is $\\frac{1}{2}$. To do this, for an index set $J=\\{j_1,\\dots,j_k\\} \\subseteq \\{1,\\dots,d\\}$ with $k \\leq d$, we denote $\\x_J:=(x_{j_1},\\dots,x_{j_k})$. Moreover, for two index sets $I,J$, we denote\n$A_{IJ}=(A_{ij})_{i \\in I, j \\in J}.$\n","start":85243,"end":85560}]},{"id":"S028","semantic_type":"assumption","kind":"assumption","labels":["t:KL_0norm_reg"],"declared_external":false,"source_only":false,"title":"主定理的结构条件","source_units":[{"raw_tex":"where $\\Lambda=\\{\\x: \\|\\x\\|=1\\}$, $A,B$ are symmetric matrices with $B$ positive definite, and $\\lambda>0$. Then $\\Phi$ is a KL function with exponent $\\frac{1}{2}$. In particular, for all $\\rho \\geq 0$,\n","start":85707,"end":85911}]},{"id":"S029","semantic_type":"theorem","kind":"theorem","labels":["t:KL_0norm_reg"],"declared_external":false,"source_only":false,"title":"基数正则模型的半阶KL指数","source_units":[{"raw_tex":"\\begin{theorem}\n\\label{t:KL_0norm_reg}\nConsider the function\n$\\Phi(\\x)=\\frac{\\x^\\top A\\x}{\\x^\\top B\\x}+ \\lambda \\|\\x\\|_0 + \\delta_{\\Lambda}(\\x)$,\nwhere $\\Lambda=\\{\\x: \\|\\x\\|=1\\}$, $A,B$ are symmetric matrices with $B$ positive definite, and $\\lambda>0$. Then $\\Phi$ is a KL function with exponent $\\frac{1}{2}$. In particular, for all $\\rho \\geq 0$,\n\\begin{equation*}\n\\widehat{\\Phi}_{GEP}(\\x,\\bu)= \\frac{\\x^\\top A\\x}{\\x^\\top B\\x}+ \\lambda \\|\\x\\|_0 + \\delta_{\\Lambda}(\\x)+ \\rho \\|\\x-\\bu\\|^2\n\\end{equation*}\nsatisfies the KL property with exponent $\\frac{1}{2}$ at $(\\overline{\\x},\\overline{\\x})$ for all $\\overline{\\x} \\in \\dom\\partial_L \\Phi$ .\n\\end{theorem}","start":85561,"end":86219}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","labels":["l:bound"],"declared_external":false,"source_only":false,"title":"对称矩阵的谱下界","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{l:bound}\nLet $Q$ be a symmetric $d \\times d$ matrix. Then there exists $c>0$ such that, for all $\\x \\in \\mathbb{R}^d$, $\\|Q\\x\\|^2 \\geq c\\, (\\x^\\top Q\\x)$.\n\\end{lemma}","start":84199,"end":84386}]},{"id":"S031","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"谱下界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $Q=U^\\top \\Sigma U$ where $U$ is an orthonormal matrix and $\\Sigma={\\rm diag}(\\lambda_1,\\dots,\\lambda_n)$ is a diagonal matrix whose diagonal elements are the eigenvalues of $Q$ with $\\lambda_1 \\leq \\lambda_2 \\leq \\dots \\leq \\lambda_n$. Let $\\x \\in \\mathbb{R}^d$, $\\y:=U\\x$, and \n$I_0=\\{j:\\lambda_j \\neq 0\\}$. Then $\\x^\\top Q\\x= \\sum_{j=1}^N \\lambda_j y_j^2 =\\sum_{j \\in I_0} \\lambda_j y_j^2$ and  \n\\begin{equation*}\n\\|Q\\x\\|^2=\\x^\\top (Q^\\top Q)\\x= (U\\x)^\\top \\Sigma^2 (U\\x)= \\sum_{j=1}^N \\lambda_j^2 y_j^2=\\sum_{j \\in I_0} \\lambda_j^2 y_j^2.\n\\end{equation*}\nSetting $c :=\\min\\{|\\lambda_j|: j \\in I_0\\}$, we see that\n\\begin{equation*}\nc\\, (\\x^\\top Q\\x)= \\sum_{j \\in I_0} c\\lambda_j y_j^2 \\leq \\sum_{j \\in I_0} c|\\lambda_j| y_j^2 \\leq \\sum_{j \\in I_0} |\\lambda_j|^2 y_j^2=\\|Q\\x\\|^2,\n\\end{equation*}\nwhich completes the proof.\n\\end{proof}","start":84387,"end":85241}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"谱分解与坐标表达","source_units":[{"raw_tex":"\\begin{proof}\nLet $Q=U^\\top \\Sigma U$ where $U$ is an orthonormal matrix and $\\Sigma={\\rm diag}(\\lambda_1,\\dots,\\lambda_n)$ is a diagonal matrix whose diagonal elements are the eigenvalues of $Q$ with $\\lambda_1 \\leq \\lambda_2 \\leq \\dots \\leq \\lambda_n$. Let $\\x \\in \\mathbb{R}^d$, $\\y:=U\\x$, and \n$I_0=\\{j:\\lambda_j \\neq 0\\}$. Then $\\x^\\top Q\\x= \\sum_{j=1}^N \\lambda_j y_j^2 =\\sum_{j \\in I_0} \\lambda_j y_j^2$ and  \n\\begin{equation*}\n\\|Q\\x\\|^2=\\x^\\top (Q^\\top Q)\\x= (U\\x)^\\top \\Sigma^2 (U\\x)= \\sum_{j=1}^N \\lambda_j^2 y_j^2=\\sum_{j \\in I_0} \\lambda_j^2 y_j^2.\n\\end{equation*}\n","start":84387,"end":84964}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"选取非零谱常数","source_units":[{"raw_tex":"Setting $c :=\\min\\{|\\lambda_j|: j \\in I_0\\}$, we see that\n","start":84964,"end":85022}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"逐项比较完成谱界","source_units":[{"raw_tex":"\\begin{equation*}\nc\\, (\\x^\\top Q\\x)= \\sum_{j \\in I_0} c\\lambda_j y_j^2 \\leq \\sum_{j \\in I_0} c|\\lambda_j| y_j^2 \\leq \\sum_{j \\in I_0} |\\lambda_j|^2 y_j^2=\\|Q\\x\\|^2,\n\\end{equation*}\nwhich completes the proof.\n\\end{proof}","start":85022,"end":85241}]},{"id":"S035","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nTake any $\\overline{\\x} \\in \\dom\\partial_L \\Phi$. Then $\\overline{\\x}  \\in \\Lambda$.\nLet $J=\\supp(\\overline{\\x})$ and use $|J|$ to denote the cardinality of $J$.  Choose $\\eta \\in (0,1)$ such that, for all $\\|\\x-\\overline{\\x}\\| < \\eta$,\n\\begin{equation*}\n\\left|\\, \\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}-\\frac{\\x^\\top A\\x}{\\x^\\top B\\x}\\, \\right| \\leq \\frac{\\lambda}{4} \\text{ and } \\eta< \\frac{\\lambda}{4}.\n\\end{equation*} Let $\\x$ with $\\|\\x-\\overline{\\x}\\|<\\eta$ and  $\\Phi(\\overline{\\x})<\\Phi(\\x)<\\Phi(\\overline{\\x})+\\eta$.\nWe first see that, by shrinking $\\eta$ if necessary, one can assume that\n\\begin{equation*}\nJ=\\supp(\\overline{\\x})=\\supp(\\x).\n\\end{equation*}\nIndeed, by continuity and by shrinking $\\eta$ if necessary, one has $\\supp(\\overline{\\x})\\subseteq \\supp(\\x)$. Suppose that $\\supp(\\overline{\\x}) \\subsetneq \\supp(\\x)$. Then $\\|\\x\\|_0>\\|\\overline{\\x}\\|_0$, and so,\n$\\|\\x\\|_0 \\geq \\|\\overline{\\x}\\|_0+1$.\nFrom our choice of $\\x$, one has $\\x \\in \\Lambda$ and\n\\begin{equation*}\n\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}+ \\lambda \\|\\overline{\\x}\\|_0< \\frac{\\x^\\top A\\x}{\\x^\\top B\\x}+ \\lambda \\|\\x\\|_0 <\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}+ \\lambda \\|\\overline{\\x}\\|_0 +\\eta. \\end{equation*}\nThis shows that\n$\\|\\x\\|_0<\\frac{1}{2}+ \\|\\overline{\\x}\\|_0$,\nwhich is impossible.\n\nUsing Lemma~\\ref{l:subdiff} and noting that $J =\\supp(\\x)$, we derive that\n\\begin{multline*}\n\\partial_L \\Phi(\\x) \\subseteq \\Bigg\\{\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2} +\\lambda \\bv +t \\x:  t \\in \\mathbb{R}, \\\\\nv_j=0 \\text{~if~} j \\in J, \\text{~and~} v_j \\in \\mathbb{R} \\text{~if~} j \\notin J\\Bigg\\}.\n\\end{multline*}\nDenoting $[a]_J=(a_j)_{j \\in J} \\in \\mathbb{R}^{|J|}$, this implies that\n\\begin{equation*}\n\\dist(0, \\partial_L \\Phi(\\x)) \\geq \\inf_{t \\in \\mathbb{R}} \\left\\|\\left[\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right]_J +t \\x_J\\right\\|.\n\\end{equation*}\nA direct verification shows that\n\\begin{equation*}\n\\x^\\top  \\left( \\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right) =0,\n\\end{equation*}\nwhich, together with $J=\\supp(\\x)$, implies that\n\\begin{equation*}\n(\\x_J)^\\top \\left( \\left[\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right]_J\\right) =0.\n\\end{equation*}\nTherefore,\n\\begin{equation*}\n\\dist(0, \\partial_L \\Phi(\\x)) \\geq \\left\\|\\left[\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right]_J\\right\\|\n=\\frac{2}{\\x^\\top B\\x} \\left\\| [A\\x]_J -\\frac{\\x^\\top A\\x}{\\x^\\top B\\x} [B\\x]_J\\right\\|.\n\\end{equation*}\nUsing $J=\\supp(\\x)$ again, we have that\n\\begin{equation*}\n[A\\x]_J =A_{JJ}\\x_J,\\; \\x^\\top A\\x =(\\x_J)^\\top A_{JJ}\\x_J,\\; [B\\x]_J= B_{JJ}\\x_J, \\text{~and~} \\x^\\top B\\x=(\\x_J)^\\top B_{JJ}\\x_J,\n\\end{equation*}\nand hence $\\dist(0, \\partial_L \\Phi(\\x))\n\\geq \\frac{2}{\\x^\\top B\\x} \\left\\| A_{JJ}\\x_J -\\frac{\\x^\\top A\\x}{\\x^\\top B\\x} \nB_{JJ}\\x_J\\right\\|$.\n\nNow, let $q(\\z) =\\z^\\top A_{JJ}\\z -\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}\\, \\z^\\top B_{JJ}\\z$ for $\\z \\in \\mathbb{R}^{|J|}$.\nThen\n\\begin{align*}\n\\dist(0, \\partial_L \\Phi(\\x))\n& \\geq \\frac{2}{\\x^\\top B\\x} \\left\\| A_{JJ}\\x_J -  \\frac{\\x^\\top A\\x}{\\x^\\top B\\x} \nB_{JJ}\\x_J\\right\\| \\\\\n& \\geq \\frac{2}{\\x^\\top B\\x} \\left(\\left\\| A_{JJ}\\x_J -  \\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}} \nB_{JJ}\\x_J\\right\\| - \\left|\\frac{\\x^\\top A\\x}{\\x^\\top B\\x}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}\n\\right| \\, \\|B_{JJ}\\x_J\\|\\right) \\\\\n& = \\frac{1}{\\x^\\top B\\x} \\|\\nabla q (\\x_J)\\| -\\frac{2}{\\x^\\top B\\x}|\\Phi(\\x)-\\Phi(\\overline{\\x})|\\, \\|B_{JJ}\\x_J\\|,\n\\end{align*}\nwhere the second inequality follows from the triangle inequality and the last equality holds as $\\x, \\overline{\\x} \\in \\Lambda$ and $J=\\supp(\\x)=\\supp(\\overline{\\x})$ (and so, $\\|\\x\\|_0=\\|\\overline{\\x}\\|_0$).\n\nFrom Lemma \\ref{l:bound}, there exists $c>0$ such that, for all $\\z$,\n$\\|\\nabla q(\\z)\\|^2 \\geq c\\, q(\\z)$.\nIndeed, one can set $c :=\\min_{1 \\leq j \\leq |J|} \\left \\{4|\\lambda_j(A_{JJ}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}B_{JJ})|: \\lambda_j(A_{JJ}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}B_{JJ}) \\neq 0 \\right\\} $, where $\\lambda_j(Q)$ are the eigenvalues of a matrix $Q$. Noting that\n\\begin{align*}\n\\frac{q(\\x_J)}{\\x^\\top B\\x} &= \\left[(\\x_J)^\\top A_{JJ}(\\x_J)-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}(\\x_J)^\\top B_{JJ}(\\x_J)\\right] \\frac{1}{\\x^\\top B\\x} \\notag \\\\\n& = \\left[\\x^\\top A\\x - \\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}} \\x^\\top B\\x \\right] \\frac{1}{\\x^\\top B\\x} \\notag \\\\\n& = \\frac{\\x^\\top A\\x}{\\x^\\top B\\x}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}} =\\Phi(\\x)-\\Phi(\\overline{\\x})>0,\n\\end{align*}\none has\n\\begin{align*}\n\\dist(0, \\partial_L \\Phi(\\x))\n&\\geq \\frac{\\sqrt{c} q(\\x_J)^{1/2}}{\\x^\\top B\\x} -2|\\Phi(\\x)-\\Phi(\\overline{\\x})|\\frac{\\|B_{JJ}\\x_J\\|}{\\x^\\top B\\x} \\\\\n& = [\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2} \\left(\\frac{\\sqrt{c}}{\\sqrt{\\x^\\top B\\x}} -2[\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2}\\frac{\\|B_{JJ}\\x_J\\|}{\\x^\\top B\\x}\\right).\n\\end{align*}\nLet \n$c_1 :=\\min\\{\\sqrt{\\x^\\top B\\x}: \\x \\in \\Lambda\\}$ and $c_2 :=\\max\\{\\sqrt{\\x^\\top B\\x}: \\x \\in \\Lambda\\}$. By shrinking $\\eta$ if necessary, we can assume that $\\eta \\in (0,1)$ and\n\\begin{equation*}\n2[\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2}\\frac{\\|B_{JJ}\\x_J\\|}{\\x^\\top B\\x}\\leq 2\\eta^{1/2}\\frac{\\|B_{JJ}\\x_J\\|}{c_1}\\leq \\frac{\\sqrt{c}}{2c_2},\n\\end{equation*}\nwhere the first inequality follows by the fact $\\Phi(\\overline{\\x})<\\Phi(\\x)<\\Phi(\\overline{\\x})+\\eta$. Then, we see that\n\\begin{equation*}\n\\dist(0, \\partial_L \\Phi(\\x))\n\\geq [\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2} \\left(\\frac{\\sqrt{c}}{c_2} -\\frac{\\sqrt{c}}{2c_2}\\right) =\\frac{\\sqrt{c}}{2c_2}[\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2}.\n\\end{equation*}\nThus, $\\Phi$ satisfies the KL property with exponent $\\frac{1}{2}$. This shows that, according to Lemma~\\ref{l:LP},\n$\\widehat{\\Phi}_{GEP}$ satisfies the KL property with exponent $\\frac{1}{2}$ at $\\overline{\\x}$ for all $\\overline{\\x} \\in \\dom\\partial_L \\Phi$.\n\\end{proof}","start":86220,"end":92458}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"任意基点与局部窗口","source_units":[{"raw_tex":"\\begin{proof}\nTake any $\\overline{\\x} \\in \\dom\\partial_L \\Phi$. Then $\\overline{\\x}  \\in \\Lambda$.\nLet $J=\\supp(\\overline{\\x})$ and use $|J|$ to denote the cardinality of $J$.  Choose $\\eta \\in (0,1)$ such that, for all $\\|\\x-\\overline{\\x}\\| < \\eta$,\n\\begin{equation*}\n\\left|\\, \\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}-\\frac{\\x^\\top A\\x}{\\x^\\top B\\x}\\, \\right| \\leq \\frac{\\lambda}{4} \\text{ and } \\eta< \\frac{\\lambda}{4}.\n\\end{equation*} Let $\\x$ with $\\|\\x-\\overline{\\x}\\|<\\eta$ and  $\\Phi(\\overline{\\x})<\\Phi(\\x)<\\Phi(\\overline{\\x})+\\eta$.\n","start":86220,"end":86797}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"局部支撑包含","source_units":[{"raw_tex":"We first see that, by shrinking $\\eta$ if necessary, one can assume that\n\\begin{equation*}\nJ=\\supp(\\overline{\\x})=\\supp(\\x).\n\\end{equation*}\nIndeed, by continuity and by shrinking $\\eta$ if necessary, one has $\\supp(\\overline{\\x})\\subseteq \\supp(\\x)$. Suppose that $\\supp(\\overline{\\x}) \\subsetneq \\supp(\\x)$. Then $\\|\\x\\|_0>\\|\\overline{\\x}\\|_0$, and so,\n$\\|\\x\\|_0 \\geq \\|\\overline{\\x}\\|_0+1$.\n","start":86797,"end":87191}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"排除支撑增大","source_units":[{"raw_tex":"From our choice of $\\x$, one has $\\x \\in \\Lambda$ and\n\\begin{equation*}\n\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}+ \\lambda \\|\\overline{\\x}\\|_0< \\frac{\\x^\\top A\\x}{\\x^\\top B\\x}+ \\lambda \\|\\x\\|_0 <\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}+ \\lambda \\|\\overline{\\x}\\|_0 +\\eta. \\end{equation*}\nThis shows that\n$\\|\\x\\|_0<\\frac{1}{2}+ \\|\\overline{\\x}\\|_0$,\nwhich is impossible.\n\n","start":87191,"end":87631}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"目标次微分的结构包含","source_units":[{"raw_tex":"Using Lemma~\\ref{l:subdiff} and noting that $J =\\supp(\\x)$, we derive that\n\\begin{multline*}\n\\partial_L \\Phi(\\x) \\subseteq \\Bigg\\{\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2} +\\lambda \\bv +t \\x:  t \\in \\mathbb{R}, \\\\\nv_j=0 \\text{~if~} j \\in J, \\text{~and~} v_j \\in \\mathbb{R} \\text{~if~} j \\notin J\\Bigg\\}.\n\\end{multline*}\n","start":87631,"end":87969}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"限制到活跃坐标","source_units":[{"raw_tex":"Denoting $[a]_J=(a_j)_{j \\in J} \\in \\mathbb{R}^{|J|}$, this implies that\n\\begin{equation*}\n\\dist(0, \\partial_L \\Phi(\\x)) \\geq \\inf_{t \\in \\mathbb{R}} \\left\\|\\left[\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right]_J +t \\x_J\\right\\|.\n\\end{equation*}\n","start":87969,"end":88234}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"商梯度与径向正交","source_units":[{"raw_tex":"A direct verification shows that\n\\begin{equation*}\n\\x^\\top  \\left( \\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right) =0,\n\\end{equation*}\nwhich, together with $J=\\supp(\\x)$, implies that\n\\begin{equation*}\n(\\x_J)^\\top \\left( \\left[\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right]_J\\right) =0.\n\\end{equation*}\n","start":88234,"end":88576}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"消去球面径向乘子","source_units":[{"raw_tex":"Therefore,\n\\begin{equation*}\n\\dist(0, \\partial_L \\Phi(\\x)) \\geq \\left\\|\\left[\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right]_J\\right\\|\n=\\frac{2}{\\x^\\top B\\x} \\left\\| [A\\x]_J -\\frac{\\x^\\top A\\x}{\\x^\\top B\\x} [B\\x]_J\\right\\|.\n\\end{equation*}\n","start":88576,"end":88835}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"改写为主子块残差","source_units":[{"raw_tex":"Using $J=\\supp(\\x)$ again, we have that\n\\begin{equation*}\n[A\\x]_J =A_{JJ}\\x_J,\\; \\x^\\top A\\x =(\\x_J)^\\top A_{JJ}\\x_J,\\; [B\\x]_J= B_{JJ}\\x_J, \\text{~and~} \\x^\\top B\\x=(\\x_J)^\\top B_{JJ}\\x_J,\n\\end{equation*}\nand hence $\\dist(0, \\partial_L \\Phi(\\x))\n\\geq \\frac{2}{\\x^\\top B\\x} \\left\\| A_{JJ}\\x_J -\\frac{\\x^\\top A\\x}{\\x^\\top B\\x} \nB_{JJ}\\x_J\\right\\|$.\n\n","start":88835,"end":89184}]},{"id":"S044","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"固定基点商值的二次型","source_units":[{"raw_tex":"Now, let $q(\\z) =\\z^\\top A_{JJ}\\z -\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}\\, \\z^\\top B_{JJ}\\z$ for $\\z \\in \\mathbb{R}^{|J|}$.\nThen\n","start":89184,"end":89351}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"梯度主项与函数差误差","source_units":[{"raw_tex":"\\begin{align*}\n\\dist(0, \\partial_L \\Phi(\\x))\n& \\geq \\frac{2}{\\x^\\top B\\x} \\left\\| A_{JJ}\\x_J -  \\frac{\\x^\\top A\\x}{\\x^\\top B\\x} \nB_{JJ}\\x_J\\right\\| \\\\\n& \\geq \\frac{2}{\\x^\\top B\\x} \\left(\\left\\| A_{JJ}\\x_J -  \\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}} \nB_{JJ}\\x_J\\right\\| - \\left|\\frac{\\x^\\top A\\x}{\\x^\\top B\\x}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}\n\\right| \\, \\|B_{JJ}\\x_J\\|\\right) \\\\\n& = \\frac{1}{\\x^\\top B\\x} \\|\\nabla q (\\x_J)\\| -\\frac{2}{\\x^\\top B\\x}|\\Phi(\\x)-\\Phi(\\overline{\\x})|\\, \\|B_{JJ}\\x_J\\|,\n\\end{align*}\nwhere the second inequality follows from the triangle inequality and the last equality holds as $\\x, \\overline{\\x} \\in \\Lambda$ and $J=\\supp(\\x)=\\supp(\\overline{\\x})$ (and so, $\\|\\x\\|_0=\\|\\overline{\\x}\\|_0$).\n","start":89351,"end":90146}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"谱界控制二次型梯度","source_units":[{"raw_tex":"\nFrom Lemma \\ref{l:bound}, there exists $c>0$ such that, for all $\\z$,\n$\\|\\nabla q(\\z)\\|^2 \\geq c\\, q(\\z)$.\nIndeed, one can set $c :=\\min_{1 \\leq j \\leq |J|} \\left \\{4|\\lambda_j(A_{JJ}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}B_{JJ})|: \\lambda_j(A_{JJ}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}B_{JJ}) \\neq 0 \\right\\} $, where $\\lambda_j(Q)$ are the eigenvalues of a matrix $Q$. Noting that\n","start":90146,"end":90605}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"二次型等于正函数值差","source_units":[{"raw_tex":"\\begin{align*}\n\\frac{q(\\x_J)}{\\x^\\top B\\x} &= \\left[(\\x_J)^\\top A_{JJ}(\\x_J)-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}(\\x_J)^\\top B_{JJ}(\\x_J)\\right] \\frac{1}{\\x^\\top B\\x} \\notag \\\\\n& = \\left[\\x^\\top A\\x - \\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}} \\x^\\top B\\x \\right] \\frac{1}{\\x^\\top B\\x} \\notag \\\\\n& = \\frac{\\x^\\top A\\x}{\\x^\\top B\\x}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}} =\\Phi(\\x)-\\Phi(\\overline{\\x})>0,\n\\end{align*}\n","start":90605,"end":91131}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"提取半阶函数值因子","source_units":[{"raw_tex":"one has\n\\begin{align*}\n\\dist(0, \\partial_L \\Phi(\\x))\n&\\geq \\frac{\\sqrt{c} q(\\x_J)^{1/2}}{\\x^\\top B\\x} -2|\\Phi(\\x)-\\Phi(\\overline{\\x})|\\frac{\\|B_{JJ}\\x_J\\|}{\\x^\\top B\\x} \\\\\n& = [\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2} \\left(\\frac{\\sqrt{c}}{\\sqrt{\\x^\\top B\\x}} -2[\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2}\\frac{\\|B_{JJ}\\x_J\\|}{\\x^\\top B\\x}\\right).\n\\end{align*}\n","start":91131,"end":91480}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"球面分母界与误差吸收","source_units":[{"raw_tex":"Let \n$c_1 :=\\min\\{\\sqrt{\\x^\\top B\\x}: \\x \\in \\Lambda\\}$ and $c_2 :=\\max\\{\\sqrt{\\x^\\top B\\x}: \\x \\in \\Lambda\\}$. By shrinking $\\eta$ if necessary, we can assume that $\\eta \\in (0,1)$ and\n\\begin{equation*}\n2[\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2}\\frac{\\|B_{JJ}\\x_J\\|}{\\x^\\top B\\x}\\leq 2\\eta^{1/2}\\frac{\\|B_{JJ}\\x_J\\|}{c_1}\\leq \\frac{\\sqrt{c}}{2c_2},\n\\end{equation*}\n","start":91480,"end":91841}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"取得局部半阶KL不等式","source_units":[{"raw_tex":"where the first inequality follows by the fact $\\Phi(\\overline{\\x})<\\Phi(\\x)<\\Phi(\\overline{\\x})+\\eta$. Then, we see that\n\\begin{equation*}\n\\dist(0, \\partial_L \\Phi(\\x))\n\\geq [\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2} \\left(\\frac{\\sqrt{c}}{c_2} -\\frac{\\sqrt{c}}{2c_2}\\right) =\\frac{\\sqrt{c}}{2c_2}[\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2}.\n\\end{equation*}\nThus, $\\Phi$ satisfies the KL property with exponent $\\frac{1}{2}$. This shows that, according to Lemma~\\ref{l:LP},\n","start":91841,"end":92302}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"应用二次提升引理","source_units":[{"raw_tex":"Thus, $\\Phi$ satisfies the KL property with exponent $\\frac{1}{2}$. This shows that, according to Lemma~\\ref{l:LP},\n$\\widehat{\\Phi}_{GEP}$ satisfies the KL property with exponent $\\frac{1}{2}$ at $\\overline{\\x}$ for all $\\overline{\\x} \\in \\dom\\partial_L \\Phi$.\n\\end{proof}","start":92186,"end":92458}]},{"id":"S052","semantic_type":"proof","kind":"proof","labels":["eq:quotient"],"declared_external":false,"source_only":true,"title":"演算法则的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of Lemma~\\ref{l:calrules}]\n\\ref{l:calrules_separable}: This is given in \\cite[Proposition 10.5]{RW98}.\n\n\\ref{l:calrules_sum}: This follows from \\cite[Corollary~10.9]{RW98}.\n\n\\ref{l:calrules_sign}: This is an application of \\cite[Corollary~3.4]{MNY06} with $\\varphi_1 \\equiv 0$ and $\\varphi_2 =f$. \n\n\\ref{l:calrules_quotient}: We first have from \\cite[Proposition~1.111(ii)]{Mor06} and \\ref{l:calrules_sum} that \n\\begin{equation}\\label{eq:quotient}\n\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{\\partial_L(-g(\\x)f +f(\\x)g)(\\x)}{g(\\x)^2} \\subseteq \\frac{\\partial_L(-g(\\x)f)(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}.\n\\end{equation}\nAssume that $\\widehat{\\partial} f$ is nonempty-valued around $\\x$. Then, if $g(\\x) >0$, $\\partial_L(-g(\\x)f)(\\x) =g(\\x)\\partial_L(-f)(\\x) \\subseteq -g(\\x)\\partial_L f(\\x)$ due to \\ref{l:calrules_sign}. If $g(\\x) <0$, then $-g(\\x) >0$ and $\\partial_L(-g(\\x)f)(\\x) =-g(\\x)\\partial_L f(\\x)$. Thus, we obtain the desired inclusion.\n\nNow, assume that $f$ is strictly differentiable at $\\x$. Then, by combining \\eqref{eq:quotient} with the last assertion in \\ref{l:calrules_sum}, $\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{-g(\\x)\\nabla f(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}$.\nOn the other hand, we have from \\cite[Corollaries 1.12.2 and 1.14.2]{Kru03} that $\\widehat{\\partial}\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{-g(\\x)\\nabla f(\\x) +\\widehat{\\partial}(f(\\x)g)(\\x)}{g(\\x)^2}$.\nThe remaining conclusion follows from these two equalities.\n\n\\ref{l:calrules_sqrt}: The chain rule is given in \\cite[Theorem~1.110(ii)]{Mor06}. The two square root rules follow by letting $\\theta(t) =\\sqrt{t}$ and $\\theta(t) =-\\sqrt{t}$, respectively.\n\\end{proof}\n","start":112891,"end":114579}]},{"id":"S053","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"可分和法则的引用证明","source_units":[{"raw_tex":"\\ref{l:calrules_separable}: This is given in \\cite[Proposition 10.5]{RW98}.\n\n","start":112938,"end":113015}]},{"id":"S054","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"一般和法则的引用证明","source_units":[{"raw_tex":"\\ref{l:calrules_sum}: This follows from \\cite[Corollary~10.9]{RW98}.\n\n","start":113015,"end":113085}]},{"id":"S055","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"变号法则的引用证明","source_units":[{"raw_tex":"\\ref{l:calrules_sign}: This is an application of \\cite[Corollary~3.4]{MNY06} with $\\varphi_1 \\equiv 0$ and $\\varphi_2 =f$. \n\n","start":113085,"end":113210}]},{"id":"S056","semantic_type":"proof","kind":"proof","labels":["eq:quotient"],"declared_external":false,"source_only":true,"title":"商法则的完整分项证明","source_units":[{"raw_tex":"\\ref{l:calrules_quotient}: We first have from \\cite[Proposition~1.111(ii)]{Mor06} and \\ref{l:calrules_sum} that \n\\begin{equation}\\label{eq:quotient}\n\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{\\partial_L(-g(\\x)f +f(\\x)g)(\\x)}{g(\\x)^2} \\subseteq \\frac{\\partial_L(-g(\\x)f)(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}.\n\\end{equation}\nAssume that $\\widehat{\\partial} f$ is nonempty-valued around $\\x$. Then, if $g(\\x) >0$, $\\partial_L(-g(\\x)f)(\\x) =g(\\x)\\partial_L(-f)(\\x) \\subseteq -g(\\x)\\partial_L f(\\x)$ due to \\ref{l:calrules_sign}. If $g(\\x) <0$, then $-g(\\x) >0$ and $\\partial_L(-g(\\x)f)(\\x) =-g(\\x)\\partial_L f(\\x)$. Thus, we obtain the desired inclusion.\n\nNow, assume that $f$ is strictly differentiable at $\\x$. Then, by combining \\eqref{eq:quotient} with the last assertion in \\ref{l:calrules_sum}, $\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{-g(\\x)\\nabla f(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}$.\nOn the other hand, we have from \\cite[Corollaries 1.12.2 and 1.14.2]{Kru03} that $\\widehat{\\partial}\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{-g(\\x)\\nabla f(\\x) +\\widehat{\\partial}(f(\\x)g)(\\x)}{g(\\x)^2}$.\nThe remaining conclusion follows from these two equalities.\n\n","start":113210,"end":114376}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":["eq:quotient"],"declared_external":false,"source_only":true,"title":"商次微分的固定系数改写","source_units":[{"raw_tex":"\\ref{l:calrules_quotient}: We first have from \\cite[Proposition~1.111(ii)]{Mor06} and \\ref{l:calrules_sum} that \n\\begin{equation}\\label{eq:quotient}\n\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{\\partial_L(-g(\\x)f +f(\\x)g)(\\x)}{g(\\x)^2} \\subseteq \\frac{\\partial_L(-g(\\x)f)(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}.\n\\end{equation}\n","start":113210,"end":113538}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"按分母符号处理分子","source_units":[{"raw_tex":"Assume that $\\widehat{\\partial} f$ is nonempty-valued around $\\x$. Then, if $g(\\x) >0$, $\\partial_L(-g(\\x)f)(\\x) =g(\\x)\\partial_L(-f)(\\x) \\subseteq -g(\\x)\\partial_L f(\\x)$ due to \\ref{l:calrules_sign}. If $g(\\x) <0$, then $-g(\\x) >0$ and $\\partial_L(-g(\\x)f)(\\x) =-g(\\x)\\partial_L f(\\x)$. Thus, we obtain the desired inclusion.\n\n","start":113538,"end":113867}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"严格可微时获得等式","source_units":[{"raw_tex":"Now, assume that $f$ is strictly differentiable at $\\x$. Then, by combining \\eqref{eq:quotient} with the last assertion in \\ref{l:calrules_sum}, $\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{-g(\\x)\\nabla f(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}$.\n","start":113867,"end":114115}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"比较Fréchet公式得到正则性","source_units":[{"raw_tex":"On the other hand, we have from \\cite[Corollaries 1.12.2 and 1.14.2]{Kru03} that $\\widehat{\\partial}\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{-g(\\x)\\nabla f(\\x) +\\widehat{\\partial}(f(\\x)g)(\\x)}{g(\\x)^2}$.\nThe remaining conclusion follows from these two equalities.\n\n","start":114115,"end":114376}]},{"id":"S061","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"复合规则及平方根特化证明","source_units":[{"raw_tex":"\\ref{l:calrules_sqrt}: The chain rule is given in \\cite[Theorem~1.110(ii)]{Mor06}. The two square root rules follow by letting $\\theta(t) =\\sqrt{t}$ and $\\theta(t) =-\\sqrt{t}$, respectively.\n\\end{proof}\n","start":114376,"end":114579}]},{"id":"S062","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"稀疏与球面公式的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of Lemma~\\ref{l:subdiff}]\n\\ref{l:subdiff_l0}: The formula for Fr\\'echet and limiting subdifferentials of $\\|\\cdot\\|_0$ can be found in \\cite[Section~3]{Le13}. The formula for the horizon subdifferential can be verified directly.\n\n\\ref{l:subdiff_sphere}: This follows by a direct verification.\n\n\\ref{l:subdiff_ball}: The limiting subdifferential formula for $\\delta_{C_r}$ can be found in \\cite[Theorem~3.9]{BLPW14}. The formula for the horizon subdifferential can be verified directly.\n\n\\ref{l:subdiff_sum}\\&\\ref{l:subdiff_sum'}: We deduce from \\ref{l:subdiff_l0}, \\ref{l:subdiff_sphere}, and \\ref{l:subdiff_ball} that, for all $\\x\\in \\Lambda$, $\\|\\cdot\\|_0$ and $\\delta_{\\Lambda}$ are regular at $\\x$ and $\\left(-\\partial_L^{\\infty} (\\|\\cdot\\|_0)(\\x)\\right) \\cap \\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{0\\}$, and that, for all $\\x\\in \\Lambda \\cap C_r$, $\\left(-\\partial_L^{\\infty} \\delta_{C_r}(\\x)\\right) \\cap \\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{0\\}$. The conclusions then follow from Lemma~\\ref{l:calrules}\\ref{l:calrules_sum}.\n\\end{proof}\n","start":114581,"end":115655}]},{"id":"S063","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"基数函数公式的作者证明","source_units":[{"raw_tex":"\\ref{l:subdiff_l0}: The formula for Fr\\'echet and limiting subdifferentials of $\\|\\cdot\\|_0$ can be found in \\cite[Section~3]{Le13}. The formula for the horizon subdifferential can be verified directly.\n\n","start":114627,"end":114831}]},{"id":"S064","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"球面公式的作者证明","source_units":[{"raw_tex":"\\ref{l:subdiff_sphere}: This follows by a direct verification.\n\n","start":114831,"end":114895}]},{"id":"S065","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"稀疏集合公式的作者证明","source_units":[{"raw_tex":"\\ref{l:subdiff_ball}: The limiting subdifferential formula for $\\delta_{C_r}$ can be found in \\cite[Theorem~3.9]{BLPW14}. The formula for the horizon subdifferential can be verified directly.\n\n","start":114895,"end":115088}]},{"id":"S066","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"两个求和公式的作者证明","source_units":[{"raw_tex":"\\ref{l:subdiff_sum}\\&\\ref{l:subdiff_sum'}: We deduce from \\ref{l:subdiff_l0}, \\ref{l:subdiff_sphere}, and \\ref{l:subdiff_ball} that, for all $\\x\\in \\Lambda$, $\\|\\cdot\\|_0$ and $\\delta_{\\Lambda}$ are regular at $\\x$ and $\\left(-\\partial_L^{\\infty} (\\|\\cdot\\|_0)(\\x)\\right) \\cap \\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{0\\}$, and that, for all $\\x\\in \\Lambda \\cap C_r$, $\\left(-\\partial_L^{\\infty} \\delta_{C_r}(\\x)\\right) \\cap \\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{0\\}$. The conclusions then follow from Lemma~\\ref{l:calrules}\\ref{l:calrules_sum}.\n","start":115088,"end":115643}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"检验正则性与水平资格","source_units":[{"raw_tex":"\\ref{l:subdiff_sum}\\&\\ref{l:subdiff_sum'}: We deduce from \\ref{l:subdiff_l0}, \\ref{l:subdiff_sphere}, and \\ref{l:subdiff_ball} that, for all $\\x\\in \\Lambda$, $\\|\\cdot\\|_0$ and $\\delta_{\\Lambda}$ are regular at $\\x$ and $\\left(-\\partial_L^{\\infty} (\\|\\cdot\\|_0)(\\x)\\right) \\cap \\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{0\\}$, and that, for all $\\x\\in \\Lambda \\cap C_r$, $\\left(-\\partial_L^{\\infty} \\delta_{C_r}(\\x)\\right) \\cap \\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{0\\}$. The conclusions then follow from Lemma~\\ref{l:calrules}\\ref{l:calrules_sum}.\n","start":115088,"end":115643}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"和法则给出等式与包含","source_units":[{"raw_tex":"\\ref{l:subdiff_sum}\\&\\ref{l:subdiff_sum'}: We deduce from \\ref{l:subdiff_l0}, \\ref{l:subdiff_sphere}, and \\ref{l:subdiff_ball} that, for all $\\x\\in \\Lambda$, $\\|\\cdot\\|_0$ and $\\delta_{\\Lambda}$ are regular at $\\x$ and $\\left(-\\partial_L^{\\infty} (\\|\\cdot\\|_0)(\\x)\\right) \\cap \\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{0\\}$, and that, for all $\\x\\in \\Lambda \\cap C_r$, $\\left(-\\partial_L^{\\infty} \\delta_{C_r}(\\x)\\right) \\cap \\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{0\\}$. The conclusions then follow from Lemma~\\ref{l:calrules}\\ref{l:calrules_sum}.\n","start":115088,"end":115643}]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件：Standard finite-dimensional spectral decomposition。"],"source_only":false,"title":"有限维对称谱分解","source_units":[]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件中的有限维紧性、连续性与基数记号；正定性、商光滑性及约束扩展的proper下半连续性为implicit standard fact made explicit by annotator。"],"source_only":false,"title":"正定性、连续性与紧性背景","source_units":[]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；原先决条件没有单独列出正缩放规则。"],"source_only":false,"title":"正缩放的次微分演算","source_units":[]},{"id":"S072","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件：RW98, Proposition 10.5；原文L1362明确引用。"],"source_only":false,"title":"可分和的外部演算结果","source_units":[]},{"id":"S073","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件：RW98, Corollary 10.9；原文L1364明确引用。"],"source_only":false,"title":"一般和的外部演算结果","source_units":[]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件：MNY06, Corollary 3.4；原文L1366引用。"],"source_only":false,"title":"变号法则的外部依据","source_units":[]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件：Mor06, Proposition 1.111(ii)；原文L1368引用。"],"source_only":false,"title":"商改写的外部依据","source_units":[]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件：Kru03, Corollaries 1.12.2 and 1.14.2；原文L1375引用。"],"source_only":false,"title":"Fréchet商公式的外部依据","source_units":[]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件：Mor06, Theorem 1.110(ii)；原文L1378引用。"],"source_only":false,"title":"复合规则的外部依据","source_units":[]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件：Le13, Section 3；原文L1383引用。"],"source_only":false,"title":"基数函数次微分的外部依据","source_units":[]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件：BLPW14, Theorem 3.9；原文L1387引用。"],"source_only":false,"title":"稀疏集合次微分的外部依据","source_units":[]},{"id":"S080","semantic_type":"assumption","kind":"assumption","labels":["l:calrules","l:calrules_quotient","l:calrules_separable","l:calrules_sign","l:calrules_sqrt","l:calrules_sum"],"declared_external":false,"source_only":false,"title":"商法则的局部适用条件","source_units":[{"raw_tex":"Suppose that $f$ and $g$ are Lipschitz continuous around $\\x$, and $g(\\x)\\neq 0$. If $\\widehat{\\partial} f$ is nonempty-valued around $\\x$, then\n","start":22728,"end":22873}]},{"id":"S081","semantic_type":"assumption","kind":"assumption","labels":["l:calrules","l:calrules_quotient","l:calrules_separable","l:calrules_sign","l:calrules_sqrt","l:calrules_sum"],"declared_external":false,"source_only":false,"title":"商包含分支的非空条件","source_units":[{"raw_tex":"Suppose that $f$ and $g$ are Lipschitz continuous around $\\x$, and $g(\\x)\\neq 0$. If $\\widehat{\\partial} f$ is nonempty-valued around $\\x$, then\n","start":22728,"end":22873}]},{"id":"S082","semantic_type":"assumption","kind":"assumption","labels":["l:calrules","l:calrules_quotient","l:calrules_separable","l:calrules_sign","l:calrules_sqrt","l:calrules_sum"],"declared_external":false,"source_only":false,"title":"商等式分支的严格可微性","source_units":[{"raw_tex":"If $f$ is strictly differentiable at $\\x$, then\n","start":23021,"end":23069}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S029","edges":[{"source_id":"S003","target_id":"S008","kind":"supports"},{"source_id":"S004","target_id":"S008","kind":"supports"},{"source_id":"S006","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S003","target_id":"S015","kind":"supports"},{"source_id":"S004","target_id":"S015","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S003","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S002","target_id":"S029","kind":"supports"},{"source_id":"S023","target_id":"S029","kind":"supports"},{"source_id":"S072","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S010","kind":"supports"},{"source_id":"S073","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S011","kind":"supports"},{"source_id":"S074","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S012","kind":"supports"},{"source_id":"S075","target_id":"S057","kind":"supports"},{"source_id":"S011","target_id":"S057","kind":"supports"},{"source_id":"S009","target_id":"S057","kind":"supports"},{"source_id":"S080","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S012","target_id":"S058","kind":"supports"},{"source_id":"S071","target_id":"S058","kind":"supports"},{"source_id":"S081","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S059","kind":"supports"},{"source_id":"S011","target_id":"S059","kind":"supports"},{"source_id":"S006","target_id":"S059","kind":"supports"},{"source_id":"S082","target_id":"S059","kind":"supports"},{"source_id":"S076","target_id":"S060","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S007","target_id":"S060","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S058","target_id":"S056","kind":"supports"},{"source_id":"S059","target_id":"S056","kind":"supports"},{"source_id":"S060","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S013","kind":"supports"},{"source_id":"S077","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S014","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S055","target_id":"S052","kind":"supports"},{"source_id":"S056","target_id":"S052","kind":"supports"},{"source_id":"S061","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S008","kind":"supports"},{"source_id":"S078","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S017","kind":"supports"},{"source_id":"S064","target_id":"S018","kind":"supports"},{"source_id":"S079","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S019","kind":"supports"},{"source_id":"S017","target_id":"S067","kind":"supports"},{"source_id":"S018","target_id":"S067","kind":"supports"},{"source_id":"S019","target_id":"S067","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S011","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S066","kind":"supports"},{"source_id":"S068","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S020","kind":"supports"},{"source_id":"S066","target_id":"S021","kind":"supports"},{"source_id":"S063","target_id":"S062","kind":"supports"},{"source_id":"S064","target_id":"S062","kind":"supports"},{"source_id":"S065","target_id":"S062","kind":"supports"},{"source_id":"S066","target_id":"S062","kind":"supports"},{"source_id":"S062","target_id":"S015","kind":"supports"},{"source_id":"S069","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S034","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S028","target_id":"S036","kind":"supports"},{"source_id":"S016","target_id":"S036","kind":"supports"},{"source_id":"S070","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S070","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S017","target_id":"S039","kind":"supports"},{"source_id":"S018","target_id":"S039","kind":"supports"},{"source_id":"S020","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S011","target_id":"S039","kind":"supports"},{"source_id":"S071","target_id":"S039","kind":"supports"},{"source_id":"S070","target_id":"S039","kind":"supports"},{"source_id":"S028","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S027","target_id":"S040","kind":"supports"},{"source_id":"S038","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S038","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S027","target_id":"S043","kind":"supports"},{"source_id":"S027","target_id":"S044","kind":"supports"},{"source_id":"S036","target_id":"S044","kind":"supports"},{"source_id":"S028","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S038","target_id":"S045","kind":"supports"},{"source_id":"S030","target_id":"S046","kind":"supports"},{"source_id":"S044","target_id":"S046","kind":"supports"},{"source_id":"S028","target_id":"S046","kind":"supports"},{"source_id":"S044","target_id":"S047","kind":"supports"},{"source_id":"S038","target_id":"S047","kind":"supports"},{"source_id":"S036","target_id":"S047","kind":"supports"},{"source_id":"S045","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S036","target_id":"S049","kind":"supports"},{"source_id":"S046","target_id":"S049","kind":"supports"},{"source_id":"S028","target_id":"S049","kind":"supports"},{"source_id":"S070","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S023","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S024","target_id":"S051","kind":"supports"},{"source_id":"S070","target_id":"S051","kind":"supports"},{"source_id":"S028","target_id":"S051","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S035","kind":"supports"},{"source_id":"S039","target_id":"S035","kind":"supports"},{"source_id":"S040","target_id":"S035","kind":"supports"},{"source_id":"S041","target_id":"S035","kind":"supports"},{"source_id":"S042","target_id":"S035","kind":"supports"},{"source_id":"S043","target_id":"S035","kind":"supports"},{"source_id":"S044","target_id":"S035","kind":"supports"},{"source_id":"S045","target_id":"S035","kind":"supports"},{"source_id":"S046","target_id":"S035","kind":"supports"},{"source_id":"S047","target_id":"S035","kind":"supports"},{"source_id":"S048","target_id":"S035","kind":"supports"},{"source_id":"S049","target_id":"S035","kind":"supports"},{"source_id":"S050","target_id":"S035","kind":"supports"},{"source_id":"S051","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S029","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S029","edges":[{"source_id":"S003","target_id":"S008","kind":"supports"},{"source_id":"S004","target_id":"S008","kind":"supports"},{"source_id":"S006","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S003","target_id":"S015","kind":"supports"},{"source_id":"S004","target_id":"S015","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S003","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S002","target_id":"S029","kind":"supports"},{"source_id":"S023","target_id":"S029","kind":"supports"},{"source_id":"S072","target_id":"S010","kind":"supports"},{"source_id":"S073","target_id":"S011","kind":"supports"},{"source_id":"S074","target_id":"S012","kind":"supports"},{"source_id":"S075","target_id":"S013","kind":"supports"},{"source_id":"S011","target_id":"S013","kind":"supports"},{"source_id":"S009","target_id":"S013","kind":"supports"},{"source_id":"S080","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S071","target_id":"S013","kind":"supports"},{"source_id":"S081","target_id":"S013","kind":"supports"},{"source_id":"S006","target_id":"S013","kind":"supports"},{"source_id":"S082","target_id":"S013","kind":"supports"},{"source_id":"S076","target_id":"S013","kind":"supports"},{"source_id":"S007","target_id":"S013","kind":"supports"},{"source_id":"S077","target_id":"S014","kind":"supports"},{"source_id":"S010","target_id":"S008","kind":"supports"},{"source_id":"S011","target_id":"S008","kind":"supports"},{"source_id":"S012","target_id":"S008","kind":"supports"},{"source_id":"S013","target_id":"S008","kind":"supports"},{"source_id":"S014","target_id":"S008","kind":"supports"},{"source_id":"S078","target_id":"S017","kind":"supports"},{"source_id":"S079","target_id":"S019","kind":"supports"},{"source_id":"S017","target_id":"S067","kind":"supports"},{"source_id":"S018","target_id":"S067","kind":"supports"},{"source_id":"S019","target_id":"S067","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S011","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S066","kind":"supports"},{"source_id":"S068","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S020","kind":"supports"},{"source_id":"S066","target_id":"S021","kind":"supports"},{"source_id":"S017","target_id":"S015","kind":"supports"},{"source_id":"S018","target_id":"S015","kind":"supports"},{"source_id":"S019","target_id":"S015","kind":"supports"},{"source_id":"S066","target_id":"S015","kind":"supports"},{"source_id":"S069","target_id":"S030","kind":"supports"},{"source_id":"S028","target_id":"S036","kind":"supports"},{"source_id":"S016","target_id":"S036","kind":"supports"},{"source_id":"S070","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S070","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S017","target_id":"S039","kind":"supports"},{"source_id":"S018","target_id":"S039","kind":"supports"},{"source_id":"S020","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S011","target_id":"S039","kind":"supports"},{"source_id":"S071","target_id":"S039","kind":"supports"},{"source_id":"S070","target_id":"S039","kind":"supports"},{"source_id":"S028","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S027","target_id":"S040","kind":"supports"},{"source_id":"S038","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S038","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S027","target_id":"S043","kind":"supports"},{"source_id":"S027","target_id":"S044","kind":"supports"},{"source_id":"S036","target_id":"S044","kind":"supports"},{"source_id":"S028","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S038","target_id":"S045","kind":"supports"},{"source_id":"S030","target_id":"S046","kind":"supports"},{"source_id":"S044","target_id":"S046","kind":"supports"},{"source_id":"S028","target_id":"S046","kind":"supports"},{"source_id":"S044","target_id":"S047","kind":"supports"},{"source_id":"S038","target_id":"S047","kind":"supports"},{"source_id":"S036","target_id":"S047","kind":"supports"},{"source_id":"S045","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S036","target_id":"S049","kind":"supports"},{"source_id":"S046","target_id":"S049","kind":"supports"},{"source_id":"S028","target_id":"S049","kind":"supports"},{"source_id":"S070","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S023","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S024","target_id":"S051","kind":"supports"},{"source_id":"S070","target_id":"S051","kind":"supports"},{"source_id":"S028","target_id":"S051","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S035","kind":"supports"},{"source_id":"S039","target_id":"S035","kind":"supports"},{"source_id":"S040","target_id":"S035","kind":"supports"},{"source_id":"S041","target_id":"S035","kind":"supports"},{"source_id":"S042","target_id":"S035","kind":"supports"},{"source_id":"S043","target_id":"S035","kind":"supports"},{"source_id":"S044","target_id":"S035","kind":"supports"},{"source_id":"S045","target_id":"S035","kind":"supports"},{"source_id":"S046","target_id":"S035","kind":"supports"},{"source_id":"S047","target_id":"S035","kind":"supports"},{"source_id":"S048","target_id":"S035","kind":"supports"},{"source_id":"S049","target_id":"S035","kind":"supports"},{"source_id":"S050","target_id":"S035","kind":"supports"},{"source_id":"S051","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S029","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"有限维空间与基本记号","source_units":[{"raw_tex":"In this section, we recall some basic notations and preliminary results which will be used in this paper. We assume throughout that $\\mathcal{H}$, $\\mathcal{H}_1$, \\dots, $\\mathcal{H}_m$ are finite-dimensional real Hilbert spaces\nwith inner product $\\scal{\\cdot}{\\cdot}$ and induced norm $\\|\\cdot\\|$. The product space $\\mathcal{H}_1 \\times \\dots \\times \\mathcal{H}_m$ is also a real Hilbert space endowed with the inner product given by\n    $\\langle (x_1,\\dots,x_m),(y_1,\\dots,y_m) \\rangle = \\sum_{i=1}^m \\langle x_i,y_i \\rangle.$\nThe set of nonnegative integers is denoted by $\\mathbb{N}$, the set of real numbers by $\\mathbb{R}$, the set of nonnegative real numbers by $\\RP$, and the set of the positive real numbers by $\\RPP$.\n\n","start":15915,"end":16647}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"指标函数与函数基本性质","source_units":[{"raw_tex":"The \\emph{indicator function} of a set $C$ is defined by $\\delta_C(\\x) :=0$ if $\\x \\in C$, and $\\delta_C(\\x) :=+\\infty$ otherwise. Given an extended-real-valued function $f\\colon \\mathcal{H}\\to \\left[-\\infty,+\\infty\\right]$, its \\emph{domain} is defined by $\\dom f :=\\{\\x \\in \\mathcal{H}: f(\\x)<+\\infty\\}$. The function $f$ is \\emph{proper} if $\\dom f\\neq \\varnothing$ and it never equals $-\\infty$. We say that $f$ is \\emph{lower semicontinuous} if, for all $\\x\\in  \\mathcal{H}$, $f(\\x)\\leq \\liminf_{\\y\\to \\x} f(\\y)$, and \\emph{convex} if its epigraph $\\{(\\x,\\rho)\\in \\mathcal{H}\\times \\mathbb{R}: f(\\x)\\leq \\rho\\}$ is a convex subset of $\\mathcal{H}\\times \\mathbb{R}$. The function $f$ is said to be \\emph{weakly convex (on $\\mathcal{H}$)} if there exists $\\alpha \\geq 0$ such that $f+\\frac{\\alpha}{2}\\|\\cdot\\|^2$ is a convex function. The smallest constant $\\alpha$ such that $f+\\frac{\\alpha}{2}\\|\\cdot\\|^2$ is convex is called the \\emph{modulus} of weak convexity. More generally, $f$ is said to be \\emph{weakly convex on $S \\subseteq \\mathcal{H}$ with modulus $\\alpha$} if $f+\\delta_S$ is weakly convex with modulus $\\alpha$. Weakly convex functions form a broad class of functions which covers quadratic functions, convex functions, differentiable functions whose gradient is Lipschitz continuous, and the composition of a convex and Lipschitz continuous function with a $C^1$-smooth mapping whose Jacobian is Lipschitz continuous (see \\cite[Lemma~4.2]{DP19}).\n\n\n","start":16647,"end":18116}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"Fréchet与极限次微分","source_units":[{"raw_tex":"Let $f\\colon\\mathcal{H} \\to \\left[-\\infty,+\\infty\\right]$ and $\\x\\in \\mathcal{H}$ with $|f(\\x)| <+\\infty$. The \\emph{Fr\\'echet subdifferential} of $f$ at $\\x$ is given by\n\\begin{equation*}\n\\widehat{\\partial} f(\\x) :=\\left\\{\\bu\\in \\mathcal{H}:\\; \\liminf_{\\z\\to \\x}\\frac{f(\\z)-f(\\x)-\\langle \\bu,\\z-\\x\\rangle}{\\|\\z-\\x\\|}\\geq 0\\right\\},\n\\end{equation*}\nthe \\emph{limiting subdifferential} of $f$ at $\\x$ is given by\n\\begin{equation*}\n\\partial_L f(\\x) :=\\left\\{\\bu\\in \\mathcal{H}:\\; \\exists \\x_n\\st{f}{\\to}\\x,\\ \\bu_n\\to \\bu\\text{~~with~~} \\bu_n\\in \\widehat{\\partial} f(\\x_n)\\right\\},\n\\end{equation*}\n","start":18116,"end":18711}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"水平次微分","source_units":[{"raw_tex":"and the \\emph{horizon subdifferential} of $f$ at $\\x$ is given by\n\\begin{equation*}\n\\partial^{\\infty}f(\\x) :=\\left\\{\\bu\\in \\mathcal{H}:\\; \\exists \\x_n\\st{f}{\\to}\\x,\\ \\lambda_n\\downarrow 0,\\ \\lambda_n\\bu_n\\to \\bu\\text{~~with~~} \\bu_n\\in \\widehat{\\partial} f(\\x_n)\\right\\}.\n\\end{equation*}\n","start":18711,"end":18999}]},{"id":"S005","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"极限次微分稳健性","source_units":[{"raw_tex":"Here, the notation $\\z\\st{f}{\\to}\\x$ means $\\z\\to \\x$ with $f(\\z)\\to f(\\x)$. It follows from the above definition that the limiting subdifferential has the following \\emph{robustness property}\n\\begin{equation*}\n\\partial_L f(\\x) =\\left\\{\\bu\\in \\mathcal{H}:\\; \\exists \\x_n \\st{f}{\\to}\\x,\\ \\bu_n\\to \\bu \\text{~~with~~} \\bu_n\\in \\partial_L f(\\x_n)\\right\\}.\n\\end{equation*}\n","start":18999,"end":19368}]},{"id":"S006","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"光滑函数与次微分","source_units":[{"raw_tex":"The \\emph{domain} of $\\partial_L f$ is $\\dom\\partial_L f :=\\{\\x\\in \\mathcal{H}: \\partial_L f(\\x)\\neq \\varnothing\\}$. If $f$ is Lipschitz continuous around $\\x$, then $\\partial_L f(\\x)$ is bounded and $\\partial^{\\infty} f(\\x) =\\{0\\}$ (see \\cite[Corollary~1.81]{Mor06}). If $f$ is strictly differentiable\\footnote{A function $f$ is strictly differentiable at $\\x$ if there exists $\\bu \\in \\mathcal{H}$ such that $\\lim_{\\y,\\z \\to \\x}\\frac{f(\\y)-f(\\z)-\\langle \\bu,\\y-\\z\\rangle}{\\|\\y-\\z\\|} =0$. Clearly, if $f$ is continuously differentiable at $\\x$, then it is strictly differentiable at $\\x$.} at $\\x$, then $\\widehat{\\partial} f$ and $\\partial_L f$ reduce to the derivative of $f$, denoted by $\\nabla f$ (see \\cite[Corollary~1.82]{Mor06}). If $f$ is convex, then both Fr\\'echet and limiting subdifferentials at $\\x$ reduce to the classical subdifferential in convex analysis (see \\cite[Theorem~1.93]{Mor06})\n\\begin{equation*}\n\\partial f(\\x) :=\\left\\{\\bu\\in \\mathcal{H}:\\; \\forall \\z\\in \\mathcal{H},\\; \\langle \\bu,\\z-\\x\\rangle\\leq f(\\z)-f(\\x)\\right\\}.\n\\end{equation*}\n","start":19368,"end":20433}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"函数与集合的正则性","source_units":[{"raw_tex":"We say that $f$ is \\emph{regular\\footnote{This is also referred as \\emph{lower regular} in \\cite{Mor06,MNY06}.} at $\\x\\in \\mathcal{H}$} if $\\widehat{\\partial} f(\\x) =\\partial_L f(\\x)$, and that $f$ is \\emph{regular on $C\\subseteq \\mathcal{H}$} if it is regular at any $\\x\\in C$. For a  proper lower semicontinuous function $f$, it is clear that if $f$ is convex around $\\x$ or strictly differentiable at $\\x$, then it is regular at $\\x$. A nonempty set $S$ in $\\mathcal{H}$ is \\emph{regular} at $\\x\\in S$ if $\\delta_S$ is regular at $\\x$. We say that $S$ is regular if it is regular at all of its points. From the definition, it can be verified that $C$ is regular if $C$ is a closed and convex set or $C$ is a smooth manifold given by $C =\\{\\x\\in \\mathcal{H}: g_i(\\x)=0,\\, i=1,\\dots, m\\}$, where $g_i$ are smooth functions satisfying the so-called linear independent constraint qualification (that is, $\\{\\nabla g_i(\\x): i=1,\\dots,m\\}$ are linearly independent for all $\\x \\in C$).\n","start":20433,"end":21416}]},{"id":"S008","semantic_type":"lemma","kind":"lemma","labels":["l:calrules","l:calrules_quotient","l:calrules_separable","l:calrules_sign","l:calrules_sqrt","l:calrules_sum"],"member_ids":["S008","S052"],"declared_external":false,"source_only":false,"title":"次微分演算法则总述","source_units":[{"raw_tex":"\\begin{lemma}[calculus rules]\n\\label{l:calrules}\nLet $f,g\\colon \\mathcal{H}\\to \\left(-\\infty,+\\infty\\right]$ be proper lower semicontinuous functions and let $\\x \\in \\dom f$. Then the following statements hold:\n\\begin{enumerate}\n\\item\\label{l:calrules_separable}\n(Separable sum rule) If $f(\\x) =\\sum_{i=1}^m f_i (x_i)$ with $\\x =(x_1,\\dots,x_m)$, then $\\partial_L f(\\x) =\\partial_L f_1(x_1) \\times \\dots \\times \\partial_L f_m(x_m)$ and $f$ is regular at $\\x$ when each $f_i$ regular at $x_i$.\n\n\\item\\label{l:calrules_sum}\n(Sum rule)\nIf $\\partial^{\\infty} f(\\x) \\cap (-\\partial^{\\infty} g(\\x)) =\\{0\\}$, then $\\partial_L(f+g)(\\x) \\subseteq \\partial_L f(\\x) + \\partial_L g(\\x)$, where the equality holds when both $f$ and $g$ are regular at $\\x$, in which case $f+g$ is also regular at $\\x$. Moreover, if $g$ is strictly differentiable at $\\x$, then $\\partial_L (f+g)(\\x) =\\partial_L f(\\x) +\\nabla g(\\x)$.\n\n\\item\\label{l:calrules_sign}\n(Sign rule)\nIf $f$ is Lipschitz continuous around $\\x$ and $\\widehat{\\partial} f$ is nonempty-valued around $\\x$, then $\\partial_L (-f)(\\x)\\subseteq -\\partial_L f(\\x)$. \n\n\\item\\label{l:calrules_quotient}\n(Quotient rule) \nSuppose that $f$ and $g$ are Lipschitz continuous around $\\x$, and $g(\\x)\\neq 0$. If $\\widehat{\\partial} f$ is nonempty-valued around $\\x$, then\n\\begin{equation*}\n\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) \\subseteq \\frac{-g(\\x)\\partial_L f(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}.\n\\end{equation*}\nIf $f$ is strictly differentiable at $\\x$, then\n\\begin{equation*}\n\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{-g(\\x)\\nabla f(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}\n\\end{equation*}\nand, consequently, $-f/g$ is regular at $\\x$ if and only if $f(\\x)g$ is regular at $\\x$.\n\n\\item\\label{l:calrules_sqrt}\n(Chain rule and square root rule)\nIf $f$ is Lipschitz continuous around $\\x$ and $\\theta\\colon \\mathbb{R}\\to \\mathbb{R}$ is continuously differentiable around $f(\\x)$,  \nthen $\\partial_L(\\theta \\circ f)(\\x) =\\partial_L (\\theta'(f(\\x))f)(\\x)$. In particular, if $f$ is Lipschitz continuous around $\\x$ and $f(\\x)>0$, then\n\\begin{equation*}\n\\partial_L\\left(\\sqrt{f}\\right)(\\x) =\\frac{\\partial_L f(\\x)}{2 \\sqrt{f(\\x)}} \\text{~~and~~} \\partial_L\\left(-\\sqrt{f}\\right)(\\x) =\\frac{\\partial_L (-f)(\\x)}{2 \\sqrt{f(\\x)}}.\n\\end{equation*}\n\\end{enumerate}\n\\end{lemma}","start":21574,"end":23878},{"raw_tex":"\\begin{proof}[Proof of Lemma~\\ref{l:calrules}]\n\\ref{l:calrules_separable}: This is given in \\cite[Proposition 10.5]{RW98}.\n\n\\ref{l:calrules_sum}: This follows from \\cite[Corollary~10.9]{RW98}.\n\n\\ref{l:calrules_sign}: This is an application of \\cite[Corollary~3.4]{MNY06} with $\\varphi_1 \\equiv 0$ and $\\varphi_2 =f$. \n\n\\ref{l:calrules_quotient}: We first have from \\cite[Proposition~1.111(ii)]{Mor06} and \\ref{l:calrules_sum} that \n\\begin{equation}\\label{eq:quotient}\n\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{\\partial_L(-g(\\x)f +f(\\x)g)(\\x)}{g(\\x)^2} \\subseteq \\frac{\\partial_L(-g(\\x)f)(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}.\n\\end{equation}\nAssume that $\\widehat{\\partial} f$ is nonempty-valued around $\\x$. Then, if $g(\\x) >0$, $\\partial_L(-g(\\x)f)(\\x) =g(\\x)\\partial_L(-f)(\\x) \\subseteq -g(\\x)\\partial_L f(\\x)$ due to \\ref{l:calrules_sign}. If $g(\\x) <0$, then $-g(\\x) >0$ and $\\partial_L(-g(\\x)f)(\\x) =-g(\\x)\\partial_L f(\\x)$. Thus, we obtain the desired inclusion.\n\nNow, assume that $f$ is strictly differentiable at $\\x$. Then, by combining \\eqref{eq:quotient} with the last assertion in \\ref{l:calrules_sum}, $\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{-g(\\x)\\nabla f(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}$.\nOn the other hand, we have from \\cite[Corollaries 1.12.2 and 1.14.2]{Kru03} that $\\widehat{\\partial}\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{-g(\\x)\\nabla f(\\x) +\\widehat{\\partial}(f(\\x)g)(\\x)}{g(\\x)^2}$.\nThe remaining conclusion follows from these two equalities.\n\n\\ref{l:calrules_sqrt}: The chain rule is given in \\cite[Theorem~1.110(ii)]{Mor06}. The two square root rules follow by letting $\\theta(t) =\\sqrt{t}$ and $\\theta(t) =-\\sqrt{t}$, respectively.\n\\end{proof}\n","start":112891,"end":114579}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","labels":["l:calrules","l:calrules_quotient","l:calrules_separable","l:calrules_sign","l:calrules_sqrt","l:calrules_sum"],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"演算法则的共同条件","source_units":[{"raw_tex":"\\begin{lemma}[calculus rules]\n\\label{l:calrules}\nLet $f,g\\colon \\mathcal{H}\\to \\left(-\\infty,+\\infty\\right]$ be proper lower semicontinuous functions and let $\\x \\in \\dom f$. Then the following statements hold:\n\\begin{enumerate}\n","start":21574,"end":21803},{"raw_tex":"\\end{enumerate}\n\\end{lemma}","start":23851,"end":23878}]},{"id":"S010","semantic_type":"lemma","kind":"lemma","labels":["l:calrules","l:calrules_quotient","l:calrules_separable","l:calrules_sign","l:calrules_sqrt","l:calrules_sum"],"member_ids":["S010","S053"],"declared_external":false,"source_only":false,"title":"可分和法则","source_units":[{"raw_tex":"\\item\\label{l:calrules_separable}\n(Separable sum rule) If $f(\\x) =\\sum_{i=1}^m f_i (x_i)$ with $\\x =(x_1,\\dots,x_m)$, then $\\partial_L f(\\x) =\\partial_L f_1(x_1) \\times \\dots \\times \\partial_L f_m(x_m)$ and $f$ is regular at $\\x$ when each $f_i$ regular at $x_i$.\n\n","start":21803,"end":22068},{"raw_tex":"\\ref{l:calrules_separable}: This is given in \\cite[Proposition 10.5]{RW98}.\n\n","start":112938,"end":113015}]},{"id":"S011","semantic_type":"lemma","kind":"lemma","labels":["l:calrules","l:calrules_quotient","l:calrules_separable","l:calrules_sign","l:calrules_sqrt","l:calrules_sum"],"member_ids":["S011","S054"],"declared_external":false,"source_only":false,"title":"一般和与光滑加项法则","source_units":[{"raw_tex":"\\item\\label{l:calrules_sum}\n(Sum rule)\nIf $\\partial^{\\infty} f(\\x) \\cap (-\\partial^{\\infty} g(\\x)) =\\{0\\}$, then $\\partial_L(f+g)(\\x) \\subseteq \\partial_L f(\\x) + \\partial_L g(\\x)$, where the equality holds when both $f$ and $g$ are regular at $\\x$, in which case $f+g$ is also regular at $\\x$. Moreover, if $g$ is strictly differentiable at $\\x$, then $\\partial_L (f+g)(\\x) =\\partial_L f(\\x) +\\nabla g(\\x)$.\n\n","start":22068,"end":22478},{"raw_tex":"\\ref{l:calrules_sum}: This follows from \\cite[Corollary~10.9]{RW98}.\n\n","start":113015,"end":113085}]},{"id":"S012","semantic_type":"lemma","kind":"lemma","labels":["l:calrules","l:calrules_quotient","l:calrules_separable","l:calrules_sign","l:calrules_sqrt","l:calrules_sum"],"member_ids":["S012","S055"],"declared_external":false,"source_only":false,"title":"变号包含法则","source_units":[{"raw_tex":"\\item\\label{l:calrules_sign}\n(Sign rule)\nIf $f$ is Lipschitz continuous around $\\x$ and $\\widehat{\\partial} f$ is nonempty-valued around $\\x$, then $\\partial_L (-f)(\\x)\\subseteq -\\partial_L f(\\x)$. \n\n","start":22478,"end":22678},{"raw_tex":"\\ref{l:calrules_sign}: This is an application of \\cite[Corollary~3.4]{MNY06} with $\\varphi_1 \\equiv 0$ and $\\varphi_2 =f$. \n\n","start":113085,"end":113210}]},{"id":"S013","semantic_type":"lemma","kind":"lemma","labels":["l:calrules","l:calrules_quotient","l:calrules_separable","l:calrules_sign","l:calrules_sqrt","l:calrules_sum"],"member_ids":["S013","S056","S057","S058","S059","S060"],"declared_external":false,"source_only":false,"title":"商法则与正则性","source_units":[{"raw_tex":"\\item\\label{l:calrules_quotient}\n(Quotient rule) \nSuppose that $f$ and $g$ are Lipschitz continuous around $\\x$, and $g(\\x)\\neq 0$. If $\\widehat{\\partial} f$ is nonempty-valued around $\\x$, then\n\\begin{equation*}\n\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) \\subseteq \\frac{-g(\\x)\\partial_L f(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}.\n\\end{equation*}\nIf $f$ is strictly differentiable at $\\x$, then\n\\begin{equation*}\n\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{-g(\\x)\\nabla f(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}\n\\end{equation*}\nand, consequently, $-f/g$ is regular at $\\x$ if and only if $f(\\x)g$ is regular at $\\x$.\n\n","start":22678,"end":23293},{"raw_tex":"\\ref{l:calrules_quotient}: We first have from \\cite[Proposition~1.111(ii)]{Mor06} and \\ref{l:calrules_sum} that \n\\begin{equation}\\label{eq:quotient}\n\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{\\partial_L(-g(\\x)f +f(\\x)g)(\\x)}{g(\\x)^2} \\subseteq \\frac{\\partial_L(-g(\\x)f)(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}.\n\\end{equation}\n","start":113210,"end":113538},{"raw_tex":"\\ref{l:calrules_quotient}: We first have from \\cite[Proposition~1.111(ii)]{Mor06} and \\ref{l:calrules_sum} that \n\\begin{equation}\\label{eq:quotient}\n\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{\\partial_L(-g(\\x)f +f(\\x)g)(\\x)}{g(\\x)^2} \\subseteq \\frac{\\partial_L(-g(\\x)f)(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}.\n\\end{equation}\nAssume that $\\widehat{\\partial} f$ is nonempty-valued around $\\x$. Then, if $g(\\x) >0$, $\\partial_L(-g(\\x)f)(\\x) =g(\\x)\\partial_L(-f)(\\x) \\subseteq -g(\\x)\\partial_L f(\\x)$ due to \\ref{l:calrules_sign}. If $g(\\x) <0$, then $-g(\\x) >0$ and $\\partial_L(-g(\\x)f)(\\x) =-g(\\x)\\partial_L f(\\x)$. Thus, we obtain the desired inclusion.\n\nNow, assume that $f$ is strictly differentiable at $\\x$. Then, by combining \\eqref{eq:quotient} with the last assertion in \\ref{l:calrules_sum}, $\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{-g(\\x)\\nabla f(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}$.\nOn the other hand, we have from \\cite[Corollaries 1.12.2 and 1.14.2]{Kru03} that $\\widehat{\\partial}\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{-g(\\x)\\nabla f(\\x) +\\widehat{\\partial}(f(\\x)g)(\\x)}{g(\\x)^2}$.\nThe remaining conclusion follows from these two equalities.\n\n","start":113210,"end":114376},{"raw_tex":"Assume that $\\widehat{\\partial} f$ is nonempty-valued around $\\x$. Then, if $g(\\x) >0$, $\\partial_L(-g(\\x)f)(\\x) =g(\\x)\\partial_L(-f)(\\x) \\subseteq -g(\\x)\\partial_L f(\\x)$ due to \\ref{l:calrules_sign}. If $g(\\x) <0$, then $-g(\\x) >0$ and $\\partial_L(-g(\\x)f)(\\x) =-g(\\x)\\partial_L f(\\x)$. Thus, we obtain the desired inclusion.\n\n","start":113538,"end":113867},{"raw_tex":"Now, assume that $f$ is strictly differentiable at $\\x$. Then, by combining \\eqref{eq:quotient} with the last assertion in \\ref{l:calrules_sum}, $\\partial_L\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{-g(\\x)\\nabla f(\\x) +\\partial_L(f(\\x)g)(\\x)}{g(\\x)^2}$.\n","start":113867,"end":114115},{"raw_tex":"On the other hand, we have from \\cite[Corollaries 1.12.2 and 1.14.2]{Kru03} that $\\widehat{\\partial}\\left(\\frac{-f}{g}\\right)(\\x) =\\frac{-g(\\x)\\nabla f(\\x) +\\widehat{\\partial}(f(\\x)g)(\\x)}{g(\\x)^2}$.\nThe remaining conclusion follows from these two equalities.\n\n","start":114115,"end":114376}]},{"id":"S014","semantic_type":"lemma","kind":"lemma","labels":["l:calrules","l:calrules_quotient","l:calrules_separable","l:calrules_sign","l:calrules_sqrt","l:calrules_sum"],"member_ids":["S014","S061"],"declared_external":false,"source_only":false,"title":"复合与平方根法则","source_units":[{"raw_tex":"\\item\\label{l:calrules_sqrt}\n(Chain rule and square root rule)\nIf $f$ is Lipschitz continuous around $\\x$ and $\\theta\\colon \\mathbb{R}\\to \\mathbb{R}$ is continuously differentiable around $f(\\x)$,  \nthen $\\partial_L(\\theta \\circ f)(\\x) =\\partial_L (\\theta'(f(\\x))f)(\\x)$. In particular, if $f$ is Lipschitz continuous around $\\x$ and $f(\\x)>0$, then\n\\begin{equation*}\n\\partial_L\\left(\\sqrt{f}\\right)(\\x) =\\frac{\\partial_L f(\\x)}{2 \\sqrt{f(\\x)}} \\text{~~and~~} \\partial_L\\left(-\\sqrt{f}\\right)(\\x) =\\frac{\\partial_L (-f)(\\x)}{2 \\sqrt{f(\\x)}}.\n\\end{equation*}\n","start":23293,"end":23851},{"raw_tex":"\\ref{l:calrules_sqrt}: The chain rule is given in \\cite[Theorem~1.110(ii)]{Mor06}. The two square root rules follow by letting $\\theta(t) =\\sqrt{t}$ and $\\theta(t) =-\\sqrt{t}$, respectively.\n\\end{proof}\n","start":114376,"end":114579}]},{"id":"S015","semantic_type":"lemma","kind":"lemma","labels":["l:subdiff","l:subdiff_ball","l:subdiff_l0","l:subdiff_sphere","l:subdiff_sum","l:subdiff_sum'"],"member_ids":["S015","S062"],"declared_external":false,"source_only":false,"title":"稀疏项与球面次微分总述","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{l:subdiff}\nLet $\\Lambda :=\\{\\x\\in \\mathbb{R}^d: \\|\\x\\|=1\\}$ and $C_r :=\\{\\x\\in \\mathbb{R}^d: \\|\\x\\|_0 \\leq r\\}$. Given $\\x =(x_1,\\dots,x_d)\\in \\mathbb{R}^d$, set $\\supp(\\x): =\\{j: x_j \\neq 0\\}$. Then the following statements hold:\n\\begin{enumerate}\n\\item\\label{l:subdiff_l0}\n$\\forall \\x\\in \\mathbb{R}^d$, $\\widehat{\\partial} (\\|\\cdot\\|_0)(\\x) =\\partial_L (\\|\\cdot\\|_0)(\\x) =\\partial_L^{\\infty} (\\|\\cdot\\|_0)(\\x) =\\{\\bv: v_j=0 \\text{~if~} j \\in \\supp(\\x)\\}$.\n\n\\item\\label{l:subdiff_sphere}\n$\\forall \\x\\in \\Lambda$, $\\widehat{\\partial} \\delta_{\\Lambda}(\\x) =\\partial_L \\delta_{\\Lambda}(\\x) =\\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{ t\\x: t \\in \\mathbb{R}\\}$.\n\n\\item\\label{l:subdiff_ball}\nIf $\\|\\x\\|_0=r$, then $\\partial_L^{\\infty} \\delta_{C_r}(\\x) =\\partial_L \\delta_{C_r}(\\x) =\\{\\bv: v_j=0 \\text{~if~} j \\in \\supp(\\x)\\}$. If $\\|\\x\\|_0<r$, then\n\\begin{multline*}\n\\partial_L^{\\infty} \\delta_{C_r}(\\x) =\\partial_L \\delta_{C_r}(\\x) =\\{\\bv: \\exists \\widehat{J} \\subseteq \\{1,\\dots,d\\} \\smallsetminus \\supp(\\x) \\text{~with~} |\\widehat{J}|=r-\\|\\x\\|_0,\\\\ v_j=0 \\text{~if~} j \\in (\\supp(\\x) \\cup \\widehat{J})\\}.    \n\\end{multline*}\n\n\\item\\label{l:subdiff_sum}\n$\\forall \\x\\in \\Lambda$, $\\partial_L (\\|\\cdot\\|_0+\\delta_{\\Lambda})(\\x) =\\partial_L (\\|\\cdot\\|_0)(\\x) +\\partial_L \\delta_{\\Lambda}(\\x)$.\n\n\\item\\label{l:subdiff_sum'}\n$\\forall \\x\\in \\Lambda \\cap C_r$, $\\partial_L (\\delta_{C_r}+\\delta_{\\Lambda})(\\x)\\subseteq \\partial_L\\delta_{C_r}(\\x) +\\partial_L \\delta_{\\Lambda}(\\x)$.\n\\end{enumerate}\n\\end{lemma}","start":23881,"end":25401},{"raw_tex":"\\begin{proof}[Proof of Lemma~\\ref{l:subdiff}]\n\\ref{l:subdiff_l0}: The formula for Fr\\'echet and limiting subdifferentials of $\\|\\cdot\\|_0$ can be found in \\cite[Section~3]{Le13}. The formula for the horizon subdifferential can be verified directly.\n\n\\ref{l:subdiff_sphere}: This follows by a direct verification.\n\n\\ref{l:subdiff_ball}: The limiting subdifferential formula for $\\delta_{C_r}$ can be found in \\cite[Theorem~3.9]{BLPW14}. The formula for the horizon subdifferential can be verified directly.\n\n\\ref{l:subdiff_sum}\\&\\ref{l:subdiff_sum'}: We deduce from \\ref{l:subdiff_l0}, \\ref{l:subdiff_sphere}, and \\ref{l:subdiff_ball} that, for all $\\x\\in \\Lambda$, $\\|\\cdot\\|_0$ and $\\delta_{\\Lambda}$ are regular at $\\x$ and $\\left(-\\partial_L^{\\infty} (\\|\\cdot\\|_0)(\\x)\\right) \\cap \\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{0\\}$, and that, for all $\\x\\in \\Lambda \\cap C_r$, $\\left(-\\partial_L^{\\infty} \\delta_{C_r}(\\x)\\right) \\cap \\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{0\\}$. The conclusions then follow from Lemma~\\ref{l:calrules}\\ref{l:calrules_sum}.\n\\end{proof}\n","start":114581,"end":115655}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":["l:subdiff","l:subdiff_ball","l:subdiff_l0","l:subdiff_sphere","l:subdiff_sum","l:subdiff_sum'"],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"球面、稀疏集与支撑","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{l:subdiff}\nLet $\\Lambda :=\\{\\x\\in \\mathbb{R}^d: \\|\\x\\|=1\\}$ and $C_r :=\\{\\x\\in \\mathbb{R}^d: \\|\\x\\|_0 \\leq r\\}$. Given $\\x =(x_1,\\dots,x_d)\\in \\mathbb{R}^d$, set $\\supp(\\x): =\\{j: x_j \\neq 0\\}$. Then the following statements hold:\n\\begin{enumerate}\n","start":23881,"end":24151},{"raw_tex":"\\end{enumerate}\n\\end{lemma}","start":25374,"end":25401}]},{"id":"S017","semantic_type":"lemma","kind":"lemma","labels":["l:subdiff","l:subdiff_ball","l:subdiff_l0","l:subdiff_sphere","l:subdiff_sum","l:subdiff_sum'"],"member_ids":["S017","S063"],"declared_external":false,"source_only":false,"title":"基数函数次微分","source_units":[{"raw_tex":"\\item\\label{l:subdiff_l0}\n$\\forall \\x\\in \\mathbb{R}^d$, $\\widehat{\\partial} (\\|\\cdot\\|_0)(\\x) =\\partial_L (\\|\\cdot\\|_0)(\\x) =\\partial_L^{\\infty} (\\|\\cdot\\|_0)(\\x) =\\{\\bv: v_j=0 \\text{~if~} j \\in \\supp(\\x)\\}$.\n\n","start":24151,"end":24361},{"raw_tex":"\\ref{l:subdiff_l0}: The formula for Fr\\'echet and limiting subdifferentials of $\\|\\cdot\\|_0$ can be found in \\cite[Section~3]{Le13}. The formula for the horizon subdifferential can be verified directly.\n\n","start":114627,"end":114831}]},{"id":"S018","semantic_type":"lemma","kind":"lemma","labels":["l:subdiff","l:subdiff_ball","l:subdiff_l0","l:subdiff_sphere","l:subdiff_sum","l:subdiff_sum'"],"member_ids":["S018","S064"],"declared_external":false,"source_only":false,"title":"球面指标次微分","source_units":[{"raw_tex":"\\item\\label{l:subdiff_sphere}\n$\\forall \\x\\in \\Lambda$, $\\widehat{\\partial} \\delta_{\\Lambda}(\\x) =\\partial_L \\delta_{\\Lambda}(\\x) =\\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{ t\\x: t \\in \\mathbb{R}\\}$.\n\n","start":24361,"end":24563},{"raw_tex":"\\ref{l:subdiff_sphere}: This follows by a direct verification.\n\n","start":114831,"end":114895}]},{"id":"S019","semantic_type":"lemma","kind":"lemma","labels":["l:subdiff","l:subdiff_ball","l:subdiff_l0","l:subdiff_sphere","l:subdiff_sum","l:subdiff_sum'"],"member_ids":["S019","S065"],"declared_external":false,"source_only":false,"title":"稀疏集合的法向公式","source_units":[{"raw_tex":"\\item\\label{l:subdiff_ball}\nIf $\\|\\x\\|_0=r$, then $\\partial_L^{\\infty} \\delta_{C_r}(\\x) =\\partial_L \\delta_{C_r}(\\x) =\\{\\bv: v_j=0 \\text{~if~} j \\in \\supp(\\x)\\}$. If $\\|\\x\\|_0<r$, then\n\\begin{multline*}\n\\partial_L^{\\infty} \\delta_{C_r}(\\x) =\\partial_L \\delta_{C_r}(\\x) =\\{\\bv: \\exists \\widehat{J} \\subseteq \\{1,\\dots,d\\} \\smallsetminus \\supp(\\x) \\text{~with~} |\\widehat{J}|=r-\\|\\x\\|_0,\\\\ v_j=0 \\text{~if~} j \\in (\\supp(\\x) \\cup \\widehat{J})\\}.    \n\\end{multline*}\n\n","start":24563,"end":25028},{"raw_tex":"\\ref{l:subdiff_ball}: The limiting subdifferential formula for $\\delta_{C_r}$ can be found in \\cite[Theorem~3.9]{BLPW14}. The formula for the horizon subdifferential can be verified directly.\n\n","start":114895,"end":115088}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","labels":["l:subdiff","l:subdiff_ball","l:subdiff_l0","l:subdiff_sphere","l:subdiff_sum","l:subdiff_sum'"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"基数项与球面指标求和","source_units":[{"raw_tex":"\\item\\label{l:subdiff_sum}\n$\\forall \\x\\in \\Lambda$, $\\partial_L (\\|\\cdot\\|_0+\\delta_{\\Lambda})(\\x) =\\partial_L (\\|\\cdot\\|_0)(\\x) +\\partial_L \\delta_{\\Lambda}(\\x)$.\n\n","start":25028,"end":25193}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","labels":["l:subdiff","l:subdiff_ball","l:subdiff_l0","l:subdiff_sphere","l:subdiff_sum","l:subdiff_sum'"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"稀疏指标与球面指标求和","source_units":[{"raw_tex":"\\item\\label{l:subdiff_sum'}\n$\\forall \\x\\in \\Lambda \\cap C_r$, $\\partial_L (\\delta_{C_r}+\\delta_{\\Lambda})(\\x)\\subseteq \\partial_L\\delta_{C_r}(\\x) +\\partial_L \\delta_{\\Lambda}(\\x)$.\n","start":25193,"end":25374}]},{"id":"S022","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"KL性质定义","source_units":[{"raw_tex":"We now consider the global convergence of the full sequence generated by Algorithm~\\ref{algo:epasa}. \nRecall that a proper lower semicontinuous function $f\\colon \\mathcal{H}\\to \\left(-\\infty, +\\infty\\right]$ is said to satisfy the \\emph{KL property} \\cite{Kur98,Loj63} at $\\overline{\\x}\\in \\dom\\partial_L f$ if there exist a neighborhood $U$ of $\\overline{\\x}$, $\\eta \\in \\left(0,+\\infty\\right]$, and a continuous concave function $\\varphi\\colon \\left[0, \\eta\\right)\\to \\mathbb{R}_+$ such that $\\varphi(0) =0$, $\\varphi$ is continuously differentiable with $\\varphi' >0$ on $\\left(0, \\eta\\right)$, and, for all $\\x\\in U$ with $f(\\overline{\\x}) <f(\\x) <f(\\overline{\\x})+\\eta$, \n\\begin{equation*}\n\\varphi'(f(\\x)-f(\\overline{\\x}))\\dist(0,\\partial_L f(\\x))\\geq 1.\n\\end{equation*}\n","start":57716,"end":58492}]},{"id":"S023","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"KL指数与KL函数","source_units":[{"raw_tex":"If $f$ satisfies the KL property at any $\\overline{\\x}\\in \\dom\\partial_L f$, then it is called a \\emph{KL function}. We say that $f$ has the \\emph{KL property at $\\overline{\\x}\\in \\dom\\partial_L f$ with exponent $\\alpha$} if it satisfies the KL property at $\\overline{\\x}\\in \\dom\\partial_L f$ and the corresponding function $\\varphi$ (often referred as desingularization function) can be chosen as $\\varphi(s) =\\gamma s^{1-\\alpha}$ for some $\\gamma\\in \\RPP$ and $\\alpha\\in \\left[0,1\\right)$. If $f$ is a KL function and has the same exponent $\\alpha$ at any $\\overline{\\x}\\in \\dom\\partial_L f$, then it is called a \\emph{KL function with exponent $\\alpha$}.\n","start":58492,"end":59150}]},{"id":"S024","semantic_type":"external_fact","kind":"external_fact","labels":["l:LP"],"member_ids":["S024"],"declared_external":true,"declaration_text":["原先决条件：LP18, Theorem 3.6；原文在B0091明确转述。"],"source_only":false,"title":"二次提升保留KL指数","source_units":[{"raw_tex":"\\begin{lemma}[cf. {\\cite[Theorem 3.6]{LP18}}]\n\\label{l:LP}\nLet $f$ be a proper lower semicontinuous function. Suppose that $f$ satisfies the KL property at $\\overline{\\x} \\in \\dom\\partial_L f$ with exponent $\\alpha \\in [\\frac{1}{2},1)$. Then, for all $\\rho \\geq 0$, $\\widetilde{f}(\\x,\\bu)=f(\\x)+\\rho \\|\\x-\\bu\\|^2$ satisfies  the KL property with exponent $\\alpha$ at $(\\overline{\\x},\\overline{\\x})$.\n\\end{lemma}\n","start":68549,"end":68961}]},{"id":"S025","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"基数正则广义特征值问题","source_units":[{"raw_tex":"Consider the generalized eigenvalue problem with cardinality regularization\n\\begin{equation}\n\\max_{\\x \\in \\mathbb{R}^d} \\frac{\\x^\\top A_1\\x}{\\x^\\top B_1\\x}- \\lambda \\|\\x\\|_0 \n\\quad\\text{s.t.}\\quad \\|\\x\\|=1,\n\\tag{GEP}\n\\end{equation}\nwhere $A_1,B_1$ are symmetric matrices such that $A_1$ is positive semidefinite and $B_1$ is positive definite, and $\\lambda>0.$ For this generalized eigenvalue problem with cardinality regularization the corresponding merit function for the proposed Algorithm~\\ref{algo:epasa} takes the form\n","start":83185,"end":83710}]},{"id":"S026","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"应用中的提升评价函数","source_units":[{"raw_tex":"\\begin{equation*}\n\\widehat{\\Phi}_{GEP}(\\x,\\bu) =\\frac{\\x^\\top A \\x}{\\x^\\top B \\x} +\\lambda \\|\\x\\|_0 +\\delta_{\\Lambda}(\\x) +\\rho \\|\\x-\\bu\\|^2,\n\\end{equation*}\nwith $A=-A_1$ a symmetric matrix,  $B=B_1$ is a positive definite matrix, $\\Lambda =\\{\\x\\in \\mathbb{R}^d: \\|\\x\\|=1\\}$, and $\\rho \\geq 0$. Below, we derive the KL exponent of the merit function $\\widehat{\\Phi}_{GEP}$. To this end, we will use the following lemma from \\cite{LP18}. Here we provide an alternative short proof for it.\n","start":83710,"end":84199}]},{"id":"S027","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"坐标限制与矩阵子块","source_units":[{"raw_tex":"Next we prove that the KL exponent of the merit function $\\widehat{\\Phi}_{GEP}$ is $\\frac{1}{2}$. To do this, for an index set $J=\\{j_1,\\dots,j_k\\} \\subseteq \\{1,\\dots,d\\}$ with $k \\leq d$, we denote $\\x_J:=(x_{j_1},\\dots,x_{j_k})$. Moreover, for two index sets $I,J$, we denote\n$A_{IJ}=(A_{ij})_{i \\in I, j \\in J}.$\n","start":85243,"end":85560}]},{"id":"S028","semantic_type":"assumption","kind":"assumption","labels":["t:KL_0norm_reg"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"主定理的结构条件","source_units":[{"raw_tex":"where $\\Lambda=\\{\\x: \\|\\x\\|=1\\}$, $A,B$ are symmetric matrices with $B$ positive definite, and $\\lambda>0$. Then $\\Phi$ is a KL function with exponent $\\frac{1}{2}$. In particular, for all $\\rho \\geq 0$,\n","start":85707,"end":85911}]},{"id":"S029","semantic_type":"theorem","kind":"theorem","labels":["t:KL_0norm_reg"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"基数正则模型的半阶KL指数","source_units":[{"raw_tex":"\\begin{theorem}\n\\label{t:KL_0norm_reg}\nConsider the function\n$\\Phi(\\x)=\\frac{\\x^\\top A\\x}{\\x^\\top B\\x}+ \\lambda \\|\\x\\|_0 + \\delta_{\\Lambda}(\\x)$,\nwhere $\\Lambda=\\{\\x: \\|\\x\\|=1\\}$, $A,B$ are symmetric matrices with $B$ positive definite, and $\\lambda>0$. Then $\\Phi$ is a KL function with exponent $\\frac{1}{2}$. In particular, for all $\\rho \\geq 0$,\n\\begin{equation*}\n\\widehat{\\Phi}_{GEP}(\\x,\\bu)= \\frac{\\x^\\top A\\x}{\\x^\\top B\\x}+ \\lambda \\|\\x\\|_0 + \\delta_{\\Lambda}(\\x)+ \\rho \\|\\x-\\bu\\|^2\n\\end{equation*}\nsatisfies the KL property with exponent $\\frac{1}{2}$ at $(\\overline{\\x},\\overline{\\x})$ for all $\\overline{\\x} \\in \\dom\\partial_L \\Phi$ .\n\\end{theorem}","start":85561,"end":86219}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","labels":["l:bound"],"member_ids":["S030","S031","S032","S033","S034"],"declared_external":false,"source_only":false,"title":"对称矩阵的谱下界","source_units":[{"raw_tex":"\\begin{lemma}\n\\label{l:bound}\nLet $Q$ be a symmetric $d \\times d$ matrix. Then there exists $c>0$ such that, for all $\\x \\in \\mathbb{R}^d$, $\\|Q\\x\\|^2 \\geq c\\, (\\x^\\top Q\\x)$.\n\\end{lemma}","start":84199,"end":84386},{"raw_tex":"\\begin{proof}\nLet $Q=U^\\top \\Sigma U$ where $U$ is an orthonormal matrix and $\\Sigma={\\rm diag}(\\lambda_1,\\dots,\\lambda_n)$ is a diagonal matrix whose diagonal elements are the eigenvalues of $Q$ with $\\lambda_1 \\leq \\lambda_2 \\leq \\dots \\leq \\lambda_n$. Let $\\x \\in \\mathbb{R}^d$, $\\y:=U\\x$, and \n$I_0=\\{j:\\lambda_j \\neq 0\\}$. Then $\\x^\\top Q\\x= \\sum_{j=1}^N \\lambda_j y_j^2 =\\sum_{j \\in I_0} \\lambda_j y_j^2$ and  \n\\begin{equation*}\n\\|Q\\x\\|^2=\\x^\\top (Q^\\top Q)\\x= (U\\x)^\\top \\Sigma^2 (U\\x)= \\sum_{j=1}^N \\lambda_j^2 y_j^2=\\sum_{j \\in I_0} \\lambda_j^2 y_j^2.\n\\end{equation*}\n","start":84387,"end":84964},{"raw_tex":"\\begin{proof}\nLet $Q=U^\\top \\Sigma U$ where $U$ is an orthonormal matrix and $\\Sigma={\\rm diag}(\\lambda_1,\\dots,\\lambda_n)$ is a diagonal matrix whose diagonal elements are the eigenvalues of $Q$ with $\\lambda_1 \\leq \\lambda_2 \\leq \\dots \\leq \\lambda_n$. Let $\\x \\in \\mathbb{R}^d$, $\\y:=U\\x$, and \n$I_0=\\{j:\\lambda_j \\neq 0\\}$. Then $\\x^\\top Q\\x= \\sum_{j=1}^N \\lambda_j y_j^2 =\\sum_{j \\in I_0} \\lambda_j y_j^2$ and  \n\\begin{equation*}\n\\|Q\\x\\|^2=\\x^\\top (Q^\\top Q)\\x= (U\\x)^\\top \\Sigma^2 (U\\x)= \\sum_{j=1}^N \\lambda_j^2 y_j^2=\\sum_{j \\in I_0} \\lambda_j^2 y_j^2.\n\\end{equation*}\nSetting $c :=\\min\\{|\\lambda_j|: j \\in I_0\\}$, we see that\n\\begin{equation*}\nc\\, (\\x^\\top Q\\x)= \\sum_{j \\in I_0} c\\lambda_j y_j^2 \\leq \\sum_{j \\in I_0} c|\\lambda_j| y_j^2 \\leq \\sum_{j \\in I_0} |\\lambda_j|^2 y_j^2=\\|Q\\x\\|^2,\n\\end{equation*}\nwhich completes the proof.\n\\end{proof}","start":84387,"end":85241},{"raw_tex":"Setting $c :=\\min\\{|\\lambda_j|: j \\in I_0\\}$, we see that\n","start":84964,"end":85022},{"raw_tex":"\\begin{equation*}\nc\\, (\\x^\\top Q\\x)= \\sum_{j \\in I_0} c\\lambda_j y_j^2 \\leq \\sum_{j \\in I_0} c|\\lambda_j| y_j^2 \\leq \\sum_{j \\in I_0} |\\lambda_j|^2 y_j^2=\\|Q\\x\\|^2,\n\\end{equation*}\nwhich completes the proof.\n\\end{proof}","start":85022,"end":85241}]},{"id":"S035","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nTake any $\\overline{\\x} \\in \\dom\\partial_L \\Phi$. Then $\\overline{\\x}  \\in \\Lambda$.\nLet $J=\\supp(\\overline{\\x})$ and use $|J|$ to denote the cardinality of $J$.  Choose $\\eta \\in (0,1)$ such that, for all $\\|\\x-\\overline{\\x}\\| < \\eta$,\n\\begin{equation*}\n\\left|\\, \\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}-\\frac{\\x^\\top A\\x}{\\x^\\top B\\x}\\, \\right| \\leq \\frac{\\lambda}{4} \\text{ and } \\eta< \\frac{\\lambda}{4}.\n\\end{equation*} Let $\\x$ with $\\|\\x-\\overline{\\x}\\|<\\eta$ and  $\\Phi(\\overline{\\x})<\\Phi(\\x)<\\Phi(\\overline{\\x})+\\eta$.\nWe first see that, by shrinking $\\eta$ if necessary, one can assume that\n\\begin{equation*}\nJ=\\supp(\\overline{\\x})=\\supp(\\x).\n\\end{equation*}\nIndeed, by continuity and by shrinking $\\eta$ if necessary, one has $\\supp(\\overline{\\x})\\subseteq \\supp(\\x)$. Suppose that $\\supp(\\overline{\\x}) \\subsetneq \\supp(\\x)$. Then $\\|\\x\\|_0>\\|\\overline{\\x}\\|_0$, and so,\n$\\|\\x\\|_0 \\geq \\|\\overline{\\x}\\|_0+1$.\nFrom our choice of $\\x$, one has $\\x \\in \\Lambda$ and\n\\begin{equation*}\n\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}+ \\lambda \\|\\overline{\\x}\\|_0< \\frac{\\x^\\top A\\x}{\\x^\\top B\\x}+ \\lambda \\|\\x\\|_0 <\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}+ \\lambda \\|\\overline{\\x}\\|_0 +\\eta. \\end{equation*}\nThis shows that\n$\\|\\x\\|_0<\\frac{1}{2}+ \\|\\overline{\\x}\\|_0$,\nwhich is impossible.\n\nUsing Lemma~\\ref{l:subdiff} and noting that $J =\\supp(\\x)$, we derive that\n\\begin{multline*}\n\\partial_L \\Phi(\\x) \\subseteq \\Bigg\\{\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2} +\\lambda \\bv +t \\x:  t \\in \\mathbb{R}, \\\\\nv_j=0 \\text{~if~} j \\in J, \\text{~and~} v_j \\in \\mathbb{R} \\text{~if~} j \\notin J\\Bigg\\}.\n\\end{multline*}\nDenoting $[a]_J=(a_j)_{j \\in J} \\in \\mathbb{R}^{|J|}$, this implies that\n\\begin{equation*}\n\\dist(0, \\partial_L \\Phi(\\x)) \\geq \\inf_{t \\in \\mathbb{R}} \\left\\|\\left[\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right]_J +t \\x_J\\right\\|.\n\\end{equation*}\nA direct verification shows that\n\\begin{equation*}\n\\x^\\top  \\left( \\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right) =0,\n\\end{equation*}\nwhich, together with $J=\\supp(\\x)$, implies that\n\\begin{equation*}\n(\\x_J)^\\top \\left( \\left[\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right]_J\\right) =0.\n\\end{equation*}\nTherefore,\n\\begin{equation*}\n\\dist(0, \\partial_L \\Phi(\\x)) \\geq \\left\\|\\left[\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right]_J\\right\\|\n=\\frac{2}{\\x^\\top B\\x} \\left\\| [A\\x]_J -\\frac{\\x^\\top A\\x}{\\x^\\top B\\x} [B\\x]_J\\right\\|.\n\\end{equation*}\nUsing $J=\\supp(\\x)$ again, we have that\n\\begin{equation*}\n[A\\x]_J =A_{JJ}\\x_J,\\; \\x^\\top A\\x =(\\x_J)^\\top A_{JJ}\\x_J,\\; [B\\x]_J= B_{JJ}\\x_J, \\text{~and~} \\x^\\top B\\x=(\\x_J)^\\top B_{JJ}\\x_J,\n\\end{equation*}\nand hence $\\dist(0, \\partial_L \\Phi(\\x))\n\\geq \\frac{2}{\\x^\\top B\\x} \\left\\| A_{JJ}\\x_J -\\frac{\\x^\\top A\\x}{\\x^\\top B\\x} \nB_{JJ}\\x_J\\right\\|$.\n\nNow, let $q(\\z) =\\z^\\top A_{JJ}\\z -\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}\\, \\z^\\top B_{JJ}\\z$ for $\\z \\in \\mathbb{R}^{|J|}$.\nThen\n\\begin{align*}\n\\dist(0, \\partial_L \\Phi(\\x))\n& \\geq \\frac{2}{\\x^\\top B\\x} \\left\\| A_{JJ}\\x_J -  \\frac{\\x^\\top A\\x}{\\x^\\top B\\x} \nB_{JJ}\\x_J\\right\\| \\\\\n& \\geq \\frac{2}{\\x^\\top B\\x} \\left(\\left\\| A_{JJ}\\x_J -  \\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}} \nB_{JJ}\\x_J\\right\\| - \\left|\\frac{\\x^\\top A\\x}{\\x^\\top B\\x}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}\n\\right| \\, \\|B_{JJ}\\x_J\\|\\right) \\\\\n& = \\frac{1}{\\x^\\top B\\x} \\|\\nabla q (\\x_J)\\| -\\frac{2}{\\x^\\top B\\x}|\\Phi(\\x)-\\Phi(\\overline{\\x})|\\, \\|B_{JJ}\\x_J\\|,\n\\end{align*}\nwhere the second inequality follows from the triangle inequality and the last equality holds as $\\x, \\overline{\\x} \\in \\Lambda$ and $J=\\supp(\\x)=\\supp(\\overline{\\x})$ (and so, $\\|\\x\\|_0=\\|\\overline{\\x}\\|_0$).\n\nFrom Lemma \\ref{l:bound}, there exists $c>0$ such that, for all $\\z$,\n$\\|\\nabla q(\\z)\\|^2 \\geq c\\, q(\\z)$.\nIndeed, one can set $c :=\\min_{1 \\leq j \\leq |J|} \\left \\{4|\\lambda_j(A_{JJ}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}B_{JJ})|: \\lambda_j(A_{JJ}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}B_{JJ}) \\neq 0 \\right\\} $, where $\\lambda_j(Q)$ are the eigenvalues of a matrix $Q$. Noting that\n\\begin{align*}\n\\frac{q(\\x_J)}{\\x^\\top B\\x} &= \\left[(\\x_J)^\\top A_{JJ}(\\x_J)-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}(\\x_J)^\\top B_{JJ}(\\x_J)\\right] \\frac{1}{\\x^\\top B\\x} \\notag \\\\\n& = \\left[\\x^\\top A\\x - \\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}} \\x^\\top B\\x \\right] \\frac{1}{\\x^\\top B\\x} \\notag \\\\\n& = \\frac{\\x^\\top A\\x}{\\x^\\top B\\x}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}} =\\Phi(\\x)-\\Phi(\\overline{\\x})>0,\n\\end{align*}\none has\n\\begin{align*}\n\\dist(0, \\partial_L \\Phi(\\x))\n&\\geq \\frac{\\sqrt{c} q(\\x_J)^{1/2}}{\\x^\\top B\\x} -2|\\Phi(\\x)-\\Phi(\\overline{\\x})|\\frac{\\|B_{JJ}\\x_J\\|}{\\x^\\top B\\x} \\\\\n& = [\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2} \\left(\\frac{\\sqrt{c}}{\\sqrt{\\x^\\top B\\x}} -2[\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2}\\frac{\\|B_{JJ}\\x_J\\|}{\\x^\\top B\\x}\\right).\n\\end{align*}\nLet \n$c_1 :=\\min\\{\\sqrt{\\x^\\top B\\x}: \\x \\in \\Lambda\\}$ and $c_2 :=\\max\\{\\sqrt{\\x^\\top B\\x}: \\x \\in \\Lambda\\}$. By shrinking $\\eta$ if necessary, we can assume that $\\eta \\in (0,1)$ and\n\\begin{equation*}\n2[\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2}\\frac{\\|B_{JJ}\\x_J\\|}{\\x^\\top B\\x}\\leq 2\\eta^{1/2}\\frac{\\|B_{JJ}\\x_J\\|}{c_1}\\leq \\frac{\\sqrt{c}}{2c_2},\n\\end{equation*}\nwhere the first inequality follows by the fact $\\Phi(\\overline{\\x})<\\Phi(\\x)<\\Phi(\\overline{\\x})+\\eta$. Then, we see that\n\\begin{equation*}\n\\dist(0, \\partial_L \\Phi(\\x))\n\\geq [\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2} \\left(\\frac{\\sqrt{c}}{c_2} -\\frac{\\sqrt{c}}{2c_2}\\right) =\\frac{\\sqrt{c}}{2c_2}[\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2}.\n\\end{equation*}\nThus, $\\Phi$ satisfies the KL property with exponent $\\frac{1}{2}$. This shows that, according to Lemma~\\ref{l:LP},\n$\\widehat{\\Phi}_{GEP}$ satisfies the KL property with exponent $\\frac{1}{2}$ at $\\overline{\\x}$ for all $\\overline{\\x} \\in \\dom\\partial_L \\Phi$.\n\\end{proof}","start":86220,"end":92458}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"任意基点与局部窗口","source_units":[{"raw_tex":"\\begin{proof}\nTake any $\\overline{\\x} \\in \\dom\\partial_L \\Phi$. Then $\\overline{\\x}  \\in \\Lambda$.\nLet $J=\\supp(\\overline{\\x})$ and use $|J|$ to denote the cardinality of $J$.  Choose $\\eta \\in (0,1)$ such that, for all $\\|\\x-\\overline{\\x}\\| < \\eta$,\n\\begin{equation*}\n\\left|\\, \\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}-\\frac{\\x^\\top A\\x}{\\x^\\top B\\x}\\, \\right| \\leq \\frac{\\lambda}{4} \\text{ and } \\eta< \\frac{\\lambda}{4}.\n\\end{equation*} Let $\\x$ with $\\|\\x-\\overline{\\x}\\|<\\eta$ and  $\\Phi(\\overline{\\x})<\\Phi(\\x)<\\Phi(\\overline{\\x})+\\eta$.\n","start":86220,"end":86797}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"局部支撑包含","source_units":[{"raw_tex":"We first see that, by shrinking $\\eta$ if necessary, one can assume that\n\\begin{equation*}\nJ=\\supp(\\overline{\\x})=\\supp(\\x).\n\\end{equation*}\nIndeed, by continuity and by shrinking $\\eta$ if necessary, one has $\\supp(\\overline{\\x})\\subseteq \\supp(\\x)$. Suppose that $\\supp(\\overline{\\x}) \\subsetneq \\supp(\\x)$. Then $\\|\\x\\|_0>\\|\\overline{\\x}\\|_0$, and so,\n$\\|\\x\\|_0 \\geq \\|\\overline{\\x}\\|_0+1$.\n","start":86797,"end":87191}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"排除支撑增大","source_units":[{"raw_tex":"From our choice of $\\x$, one has $\\x \\in \\Lambda$ and\n\\begin{equation*}\n\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}+ \\lambda \\|\\overline{\\x}\\|_0< \\frac{\\x^\\top A\\x}{\\x^\\top B\\x}+ \\lambda \\|\\x\\|_0 <\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}+ \\lambda \\|\\overline{\\x}\\|_0 +\\eta. \\end{equation*}\nThis shows that\n$\\|\\x\\|_0<\\frac{1}{2}+ \\|\\overline{\\x}\\|_0$,\nwhich is impossible.\n\n","start":87191,"end":87631}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"目标次微分的结构包含","source_units":[{"raw_tex":"Using Lemma~\\ref{l:subdiff} and noting that $J =\\supp(\\x)$, we derive that\n\\begin{multline*}\n\\partial_L \\Phi(\\x) \\subseteq \\Bigg\\{\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2} +\\lambda \\bv +t \\x:  t \\in \\mathbb{R}, \\\\\nv_j=0 \\text{~if~} j \\in J, \\text{~and~} v_j \\in \\mathbb{R} \\text{~if~} j \\notin J\\Bigg\\}.\n\\end{multline*}\n","start":87631,"end":87969}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"限制到活跃坐标","source_units":[{"raw_tex":"Denoting $[a]_J=(a_j)_{j \\in J} \\in \\mathbb{R}^{|J|}$, this implies that\n\\begin{equation*}\n\\dist(0, \\partial_L \\Phi(\\x)) \\geq \\inf_{t \\in \\mathbb{R}} \\left\\|\\left[\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right]_J +t \\x_J\\right\\|.\n\\end{equation*}\n","start":87969,"end":88234}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"商梯度与径向正交","source_units":[{"raw_tex":"A direct verification shows that\n\\begin{equation*}\n\\x^\\top  \\left( \\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right) =0,\n\\end{equation*}\nwhich, together with $J=\\supp(\\x)$, implies that\n\\begin{equation*}\n(\\x_J)^\\top \\left( \\left[\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right]_J\\right) =0.\n\\end{equation*}\n","start":88234,"end":88576}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"消去球面径向乘子","source_units":[{"raw_tex":"Therefore,\n\\begin{equation*}\n\\dist(0, \\partial_L \\Phi(\\x)) \\geq \\left\\|\\left[\\frac{2A\\x(\\x^\\top B\\x)-2B\\x(\\x^\\top A\\x)}{(\\x^\\top B\\x)^2}\\right]_J\\right\\|\n=\\frac{2}{\\x^\\top B\\x} \\left\\| [A\\x]_J -\\frac{\\x^\\top A\\x}{\\x^\\top B\\x} [B\\x]_J\\right\\|.\n\\end{equation*}\n","start":88576,"end":88835}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"改写为主子块残差","source_units":[{"raw_tex":"Using $J=\\supp(\\x)$ again, we have that\n\\begin{equation*}\n[A\\x]_J =A_{JJ}\\x_J,\\; \\x^\\top A\\x =(\\x_J)^\\top A_{JJ}\\x_J,\\; [B\\x]_J= B_{JJ}\\x_J, \\text{~and~} \\x^\\top B\\x=(\\x_J)^\\top B_{JJ}\\x_J,\n\\end{equation*}\nand hence $\\dist(0, \\partial_L \\Phi(\\x))\n\\geq \\frac{2}{\\x^\\top B\\x} \\left\\| A_{JJ}\\x_J -\\frac{\\x^\\top A\\x}{\\x^\\top B\\x} \nB_{JJ}\\x_J\\right\\|$.\n\n","start":88835,"end":89184}]},{"id":"S044","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"固定基点商值的二次型","source_units":[{"raw_tex":"Now, let $q(\\z) =\\z^\\top A_{JJ}\\z -\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}\\, \\z^\\top B_{JJ}\\z$ for $\\z \\in \\mathbb{R}^{|J|}$.\nThen\n","start":89184,"end":89351}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"梯度主项与函数差误差","source_units":[{"raw_tex":"\\begin{align*}\n\\dist(0, \\partial_L \\Phi(\\x))\n& \\geq \\frac{2}{\\x^\\top B\\x} \\left\\| A_{JJ}\\x_J -  \\frac{\\x^\\top A\\x}{\\x^\\top B\\x} \nB_{JJ}\\x_J\\right\\| \\\\\n& \\geq \\frac{2}{\\x^\\top B\\x} \\left(\\left\\| A_{JJ}\\x_J -  \\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}} \nB_{JJ}\\x_J\\right\\| - \\left|\\frac{\\x^\\top A\\x}{\\x^\\top B\\x}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}\n\\right| \\, \\|B_{JJ}\\x_J\\|\\right) \\\\\n& = \\frac{1}{\\x^\\top B\\x} \\|\\nabla q (\\x_J)\\| -\\frac{2}{\\x^\\top B\\x}|\\Phi(\\x)-\\Phi(\\overline{\\x})|\\, \\|B_{JJ}\\x_J\\|,\n\\end{align*}\nwhere the second inequality follows from the triangle inequality and the last equality holds as $\\x, \\overline{\\x} \\in \\Lambda$ and $J=\\supp(\\x)=\\supp(\\overline{\\x})$ (and so, $\\|\\x\\|_0=\\|\\overline{\\x}\\|_0$).\n","start":89351,"end":90146}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"谱界控制二次型梯度","source_units":[{"raw_tex":"\nFrom Lemma \\ref{l:bound}, there exists $c>0$ such that, for all $\\z$,\n$\\|\\nabla q(\\z)\\|^2 \\geq c\\, q(\\z)$.\nIndeed, one can set $c :=\\min_{1 \\leq j \\leq |J|} \\left \\{4|\\lambda_j(A_{JJ}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}B_{JJ})|: \\lambda_j(A_{JJ}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}B_{JJ}) \\neq 0 \\right\\} $, where $\\lambda_j(Q)$ are the eigenvalues of a matrix $Q$. Noting that\n","start":90146,"end":90605}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"二次型等于正函数值差","source_units":[{"raw_tex":"\\begin{align*}\n\\frac{q(\\x_J)}{\\x^\\top B\\x} &= \\left[(\\x_J)^\\top A_{JJ}(\\x_J)-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}}(\\x_J)^\\top B_{JJ}(\\x_J)\\right] \\frac{1}{\\x^\\top B\\x} \\notag \\\\\n& = \\left[\\x^\\top A\\x - \\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}} \\x^\\top B\\x \\right] \\frac{1}{\\x^\\top B\\x} \\notag \\\\\n& = \\frac{\\x^\\top A\\x}{\\x^\\top B\\x}-\\frac{\\overline{\\x}^\\top A\\overline{\\x}}{\\overline{\\x}^\\top B\\overline{\\x}} =\\Phi(\\x)-\\Phi(\\overline{\\x})>0,\n\\end{align*}\n","start":90605,"end":91131}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"提取半阶函数值因子","source_units":[{"raw_tex":"one has\n\\begin{align*}\n\\dist(0, \\partial_L \\Phi(\\x))\n&\\geq \\frac{\\sqrt{c} q(\\x_J)^{1/2}}{\\x^\\top B\\x} -2|\\Phi(\\x)-\\Phi(\\overline{\\x})|\\frac{\\|B_{JJ}\\x_J\\|}{\\x^\\top B\\x} \\\\\n& = [\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2} \\left(\\frac{\\sqrt{c}}{\\sqrt{\\x^\\top B\\x}} -2[\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2}\\frac{\\|B_{JJ}\\x_J\\|}{\\x^\\top B\\x}\\right).\n\\end{align*}\n","start":91131,"end":91480}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"球面分母界与误差吸收","source_units":[{"raw_tex":"Let \n$c_1 :=\\min\\{\\sqrt{\\x^\\top B\\x}: \\x \\in \\Lambda\\}$ and $c_2 :=\\max\\{\\sqrt{\\x^\\top B\\x}: \\x \\in \\Lambda\\}$. By shrinking $\\eta$ if necessary, we can assume that $\\eta \\in (0,1)$ and\n\\begin{equation*}\n2[\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2}\\frac{\\|B_{JJ}\\x_J\\|}{\\x^\\top B\\x}\\leq 2\\eta^{1/2}\\frac{\\|B_{JJ}\\x_J\\|}{c_1}\\leq \\frac{\\sqrt{c}}{2c_2},\n\\end{equation*}\n","start":91480,"end":91841}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"取得局部半阶KL不等式","source_units":[{"raw_tex":"where the first inequality follows by the fact $\\Phi(\\overline{\\x})<\\Phi(\\x)<\\Phi(\\overline{\\x})+\\eta$. Then, we see that\n\\begin{equation*}\n\\dist(0, \\partial_L \\Phi(\\x))\n\\geq [\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2} \\left(\\frac{\\sqrt{c}}{c_2} -\\frac{\\sqrt{c}}{2c_2}\\right) =\\frac{\\sqrt{c}}{2c_2}[\\Phi(\\x)-\\Phi(\\overline{\\x})]^{1/2}.\n\\end{equation*}\nThus, $\\Phi$ satisfies the KL property with exponent $\\frac{1}{2}$. This shows that, according to Lemma~\\ref{l:LP},\n","start":91841,"end":92302}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"应用二次提升引理","source_units":[{"raw_tex":"Thus, $\\Phi$ satisfies the KL property with exponent $\\frac{1}{2}$. This shows that, according to Lemma~\\ref{l:LP},\n$\\widehat{\\Phi}_{GEP}$ satisfies the KL property with exponent $\\frac{1}{2}$ at $\\overline{\\x}$ for all $\\overline{\\x} \\in \\dom\\partial_L \\Phi$.\n\\end{proof}","start":92186,"end":92458}]},{"id":"S066","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S066"],"declared_external":false,"source_only":true,"title":"两个求和公式的作者证明","source_units":[{"raw_tex":"\\ref{l:subdiff_sum}\\&\\ref{l:subdiff_sum'}: We deduce from \\ref{l:subdiff_l0}, \\ref{l:subdiff_sphere}, and \\ref{l:subdiff_ball} that, for all $\\x\\in \\Lambda$, $\\|\\cdot\\|_0$ and $\\delta_{\\Lambda}$ are regular at $\\x$ and $\\left(-\\partial_L^{\\infty} (\\|\\cdot\\|_0)(\\x)\\right) \\cap \\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{0\\}$, and that, for all $\\x\\in \\Lambda \\cap C_r$, $\\left(-\\partial_L^{\\infty} \\delta_{C_r}(\\x)\\right) \\cap \\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{0\\}$. The conclusions then follow from Lemma~\\ref{l:calrules}\\ref{l:calrules_sum}.\n","start":115088,"end":115643}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S067"],"declared_external":false,"source_only":true,"title":"检验正则性与水平资格","source_units":[{"raw_tex":"\\ref{l:subdiff_sum}\\&\\ref{l:subdiff_sum'}: We deduce from \\ref{l:subdiff_l0}, \\ref{l:subdiff_sphere}, and \\ref{l:subdiff_ball} that, for all $\\x\\in \\Lambda$, $\\|\\cdot\\|_0$ and $\\delta_{\\Lambda}$ are regular at $\\x$ and $\\left(-\\partial_L^{\\infty} (\\|\\cdot\\|_0)(\\x)\\right) \\cap \\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{0\\}$, and that, for all $\\x\\in \\Lambda \\cap C_r$, $\\left(-\\partial_L^{\\infty} \\delta_{C_r}(\\x)\\right) \\cap \\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{0\\}$. The conclusions then follow from Lemma~\\ref{l:calrules}\\ref{l:calrules_sum}.\n","start":115088,"end":115643}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S068"],"declared_external":false,"source_only":true,"title":"和法则给出等式与包含","source_units":[{"raw_tex":"\\ref{l:subdiff_sum}\\&\\ref{l:subdiff_sum'}: We deduce from \\ref{l:subdiff_l0}, \\ref{l:subdiff_sphere}, and \\ref{l:subdiff_ball} that, for all $\\x\\in \\Lambda$, $\\|\\cdot\\|_0$ and $\\delta_{\\Lambda}$ are regular at $\\x$ and $\\left(-\\partial_L^{\\infty} (\\|\\cdot\\|_0)(\\x)\\right) \\cap \\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{0\\}$, and that, for all $\\x\\in \\Lambda \\cap C_r$, $\\left(-\\partial_L^{\\infty} \\delta_{C_r}(\\x)\\right) \\cap \\partial_L^{\\infty} \\delta_{\\Lambda}(\\x) =\\{0\\}$. The conclusions then follow from Lemma~\\ref{l:calrules}\\ref{l:calrules_sum}.\n","start":115088,"end":115643}]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S069"],"declared_external":true,"declaration_text":["原先决条件：Standard finite-dimensional spectral decomposition。"],"source_only":false,"title":"有限维对称谱分解","source_units":[]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S070"],"declared_external":true,"declaration_text":["原先决条件中的有限维紧性、连续性与基数记号；正定性、商光滑性及约束扩展的proper下半连续性为implicit standard fact made explicit by annotator。"],"source_only":false,"title":"正定性、连续性与紧性背景","source_units":[]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S071"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；原先决条件没有单独列出正缩放规则。"],"source_only":false,"title":"正缩放的次微分演算","source_units":[]},{"id":"S072","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S072"],"declared_external":true,"declaration_text":["原先决条件：RW98, Proposition 10.5；原文L1362明确引用。"],"source_only":false,"title":"可分和的外部演算结果","source_units":[]},{"id":"S073","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S073"],"declared_external":true,"declaration_text":["原先决条件：RW98, Corollary 10.9；原文L1364明确引用。"],"source_only":false,"title":"一般和的外部演算结果","source_units":[]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S074"],"declared_external":true,"declaration_text":["原先决条件：MNY06, Corollary 3.4；原文L1366引用。"],"source_only":false,"title":"变号法则的外部依据","source_units":[]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S075"],"declared_external":true,"declaration_text":["原先决条件：Mor06, Proposition 1.111(ii)；原文L1368引用。"],"source_only":false,"title":"商改写的外部依据","source_units":[]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S076"],"declared_external":true,"declaration_text":["原先决条件：Kru03, Corollaries 1.12.2 and 1.14.2；原文L1375引用。"],"source_only":false,"title":"Fréchet商公式的外部依据","source_units":[]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S077"],"declared_external":true,"declaration_text":["原先决条件：Mor06, Theorem 1.110(ii)；原文L1378引用。"],"source_only":false,"title":"复合规则的外部依据","source_units":[]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S078"],"declared_external":true,"declaration_text":["原先决条件：Le13, Section 3；原文L1383引用。"],"source_only":false,"title":"基数函数次微分的外部依据","source_units":[]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S079"],"declared_external":true,"declaration_text":["原先决条件：BLPW14, Theorem 3.9；原文L1387引用。"],"source_only":false,"title":"稀疏集合次微分的外部依据","source_units":[]},{"id":"S080","semantic_type":"assumption","kind":"assumption","labels":["l:calrules","l:calrules_quotient","l:calrules_separable","l:calrules_sign","l:calrules_sqrt","l:calrules_sum"],"member_ids":["S080"],"declared_external":false,"source_only":false,"title":"商法则的局部适用条件","source_units":[{"raw_tex":"Suppose that $f$ and $g$ are Lipschitz continuous around $\\x$, and $g(\\x)\\neq 0$. If $\\widehat{\\partial} f$ is nonempty-valued around $\\x$, then\n","start":22728,"end":22873}]},{"id":"S081","semantic_type":"assumption","kind":"assumption","labels":["l:calrules","l:calrules_quotient","l:calrules_separable","l:calrules_sign","l:calrules_sqrt","l:calrules_sum"],"member_ids":["S081"],"declared_external":false,"source_only":false,"title":"商包含分支的非空条件","source_units":[{"raw_tex":"Suppose that $f$ and $g$ are Lipschitz continuous around $\\x$, and $g(\\x)\\neq 0$. If $\\widehat{\\partial} f$ is nonempty-valued around $\\x$, then\n","start":22728,"end":22873}]},{"id":"S082","semantic_type":"assumption","kind":"assumption","labels":["l:calrules","l:calrules_quotient","l:calrules_separable","l:calrules_sign","l:calrules_sqrt","l:calrules_sum"],"member_ids":["S082"],"declared_external":false,"source_only":false,"title":"商等式分支的严格可微性","source_units":[{"raw_tex":"If $f$ is strictly differentiable at $\\x$, then\n","start":23021,"end":23069}]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S030","S032":"S030","S033":"S030","S034":"S030","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S008","S053":"S010","S054":"S011","S055":"S012","S056":"S013","S057":"S013","S058":"S013","S059":"S013","S060":"S013","S061":"S014","S062":"S015","S063":"S017","S064":"S018","S065":"S019","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076","S077":"S077","S078":"S078","S079":"S079","S080":"S080","S081":"S081","S082":"S082"}}}}
{"record_id":"1809.00761v3","paper":{"title":"Adaptive Douglas-Rachford splitting algorithm for the sum of two operators","authors":["Dao, Minh N.","Phan, Hung M."],"arxiv_id":"1809.00761","version":"v3","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/1809.00761v3","pdf_url":"https://arxiv.org/pdf/1809.00761v3","doi":"10.1137/18M121160X"},"context":{"raw_tex":"Throughout this work, $X$ is a real Hilbert space with inner product $\\scal{\\cdot}{\\cdot}$ and induced norm $\\|\\cdot\\|$. \nThe set of nonnegative integers is denoted by $\\NN$, the set of real numbers by $\\RR$, the set of nonnegative real numbers by $\\RP := \\menge{x \\in \\RR}{x \\geq 0}$, and the set of the positive real numbers by $\\RPP := \\menge{x \\in \\RR}{x >0}$.\nWe use the notation $A\\colon X\\rightrightarrows X$ to indicate that $A$ is a set-valued operator on $X$ and the notation $A\\colon X\\to X$ to indicate that $A$ is a single-valued operator on $X$. \n\nLet $A$ be an operator on $X$. The \\emph{domain} of $A$ is $\\dom A :=\\menge{x\\in X}{Ax\\neq \\varnothing}$, the \\emph{graph} of $A$ is $\\gra A :=\\menge{(x,u)\\in X\\times X}{u\\in Ax}$, and the set of \\emph{fixed points} of $A$ is $\\Fix A :=\\menge{x\\in X}{x\\in Ax}$. The \\emph{inverse} of $A$, denoted by $A^{-1}$, is the operator with graph $\\gra A^{-1} :=\\menge{(u,x)\\in X\\times X}{u\\in Ax}$.\nWe say that $A$ is \\emph{Lipschitz continuous} with constant $\\ell\\in \\RP$ if it is single-valued and\n\\begin{equation}\n\\forall x, y\\in \\dom A,\\quad \\|Ax-Ay\\|\\leq \\ell\\|x-y\\|.\n\\end{equation}\nThe operator $A$ is \\emph{nonexpansive} if it is Lipschitz continuous with constant $1$, i.e.,\n\\begin{equation}\n\\forall x, y\\in \\dom A,\\quad \\|Ax-Ay\\|\\leq \\|x-y\\|.\n\\end{equation}\nAn operator $A\\colon X\\rightrightarrows X$ is said to be \\emph{monotone} if\n\\begin{equation}\n\\forall (x, u), (y, v)\\in \\gra A,\\quad \\scal{x-y}{u-v}\\geq 0,\n\\end{equation}\nand said to be \\emph{maximally monotone} if it is monotone and there exists no monotone operator $B\\colon X\\rightrightarrows X$ such that $\\gra B$ properly contains $\\gra A$. The \\emph{resolvent} of $A\\colon X\\rightrightarrows X$ is defined by\n\\begin{equation}\nJ_A:= (\\Id+ A)^{-1},\n\\end{equation}\nwhere $\\Id$ is the identity operator. The \\emph{relaxed resolvent} of $A$ with parameter $\\lambda\\in\\RP$ is defined by\n\\begin{equation}\nJ_A^\\lambda:= (1-\\lambda)\\Id+ \\lambda J_A.\n\\end{equation}\n\n\n\\begin{definition}[$\\alpha$-monotonicity]\n\\label{d:alpha_mono}\nAn operator $A\\colon X\\rightrightarrows X$ is said to be \\emph{$\\alpha$-monotone} ($\\alpha\\in \\RR$) if\n\\begin{equation}\n\\label{e:alpha-mono}\n\\forall (x,u), (y,v)\\in \\gra A,\\quad \\scal{x-y}{u-v}\\geq \\alpha\\|x-y\\|^2.\n\\end{equation}\nThe constant $\\alpha$ is referred to as the \\emph{monotonicity constant}.\nWe also say that $A$ is \\emph{maximally $\\alpha$-monotone} if it is $\\alpha$-monotone and there is no $\\alpha$-monotone operator whose graph strictly contains $\\gra A$.  \n\\end{definition}\n\nWe note that $0$-monotonicity simply means monotonicity, that if $\\alpha>0$, then $\\alpha$-monotonicity is precisely the notion of \\emph{strong monotonicity} \\cite[Definition~22.1(iv)]{BC17}, and that if $\\alpha<0$, then $\\alpha$-monotonicity can be referred to as \\emph{weak monotonicity}.\n\n\n\\begin{remark}[Lipschitz $\\alpha$-monotone operators]\n\\label{r:Lip_mono}\nSuppose that $A$ is Lipschitz continuous with constant $\\ell$. Then $A$ is single-valued and \n\\begin{equation}\n\\forall x,y\\in\\dom A,\\quad\n|\\scal{x-y}{Ax-Ay}|\\leq \\|x-y\\|\\cdot\\|Ax-Ay\\|\\leq \\ell\\|x-y\\|^2,\n\\end{equation}\nwhich yields\n\\begin{equation}\n\\label{e:L-inequality}\n\\forall x,y\\in\\dom A,\\quad -\\ell\\|x-y\\|^2\\leq \\scal{x-y}{Ax-Ay}\\leq \\ell\\|x-y\\|^2.\n\\end{equation}\nWe immediately deduce that $A$ is $(-\\ell)$-monotone. Now suppose, in addition, that $A$ is $\\alpha$-monotone. On the one hand, we can always assume without loss of generality that $\\alpha\\geq-\\ell$. On the other hand, it follows from the $\\alpha$-monotonicity and \\eqref{e:L-inequality} that $\\alpha\\leq \\ell$ as soon as $\\dom A$ has more than one element. Therefore, unless otherwise stated, whenever $A$ is both $\\alpha$-monotone and Lipschitz continuous with constant $\\ell$, we assume that $|\\alpha|\\leq\\ell$.\n\\end{remark}\n\nThroughout this section, $A,B\\colon X\\rightrightarrows X$, $(\\gamma,\\delta,\\lambda,\\mu)\\in \\RPP^4$, and $\\kappa\\in \\left]0,1\\right[$. We define\n\\begin{subequations}\\label{e:JRT}\n\\begin{align}\nJ_1&:=J_{\\gamma A}=(\\Id+\\gamma A)^{-1}, \n&R_1&:=J_{\\gamma A}^\\lambda=(1-\\lambda)\\Id+\\lambda J_1,\\\\\nJ_2&:=J_{\\delta B}=(\\Id+\\delta B)^{-1},\n&R_2&:=J_{\\delta B}^\\mu=(1-\\mu)\\Id+\\mu J_2\n\\end{align}\nand consider the \\emph{adaptive DR operator} defined by\n\\begin{align}\\label{e:aDR}\nT:= (1-\\kappa)\\Id+ \\kappa R_2R_1.\n\\end{align}\n\\end{subequations}  \n\n\nThe adaptive DR operator is indeed motivated by the problem of finding a zero of the sum of two operators, that is,\n\\begin{equation}\\label{e:sumprob}\n\\text{find $x\\in X$ such that}\\quad 0\\in Ax+Bx.\n\\end{equation}\nWe also denote by\n\\begin{equation}\n\\zer(A+B) :=(A+B)^{-1}(0) =\\menge{x\\in X}{0\\in Ax+Bx}\n\\end{equation}\nthe set of solutions of problem \\eqref{e:sumprob}. Given a starting point $x_0\\in X$, the \\emph{adaptive DR algorithm} generates a sequence $(x_n)_\\nnn$, also called a \\emph{DR sequence}, by \n\\begin{equation}\\label{e:dr_seq}\n\\forall\\nnn,\\quad x_{n+1}\\in Tx_n.\n\\end{equation}\n\n\nRecall that a sequence $(x_n)_\\nnn$ converges to $\\overline{x}$ with \\emph{$Q$-linear} (or simply \\emph{linear}) rate $\\rho\\in\\left[0,1\\right[$ if\n\\begin{equation}\n\\forall\\nnn,\\quad \\|x_{n+1}-\\overline{x}\\|\\leq \\rho\\|x_n-\\overline{x}\\|.\n\\end{equation}\n","units":[{"raw_tex":"Throughout this work, $X$ is a real Hilbert space with inner product $\\scal{\\cdot}{\\cdot}$ and induced norm $\\|\\cdot\\|$. \nThe set of nonnegative integers is denoted by $\\NN$, the set of real numbers by $\\RR$, the set of nonnegative real numbers by $\\RP := \\menge{x \\in \\RR}{x \\geq 0}$, and the set of the positive real numbers by $\\RPP := \\menge{x \\in \\RR}{x >0}$.\nWe use the notation $A\\colon X\\rightrightarrows X$ to indicate that $A$ is a set-valued operator on $X$ and the notation $A\\colon X\\to X$ to indicate that $A$ is a single-valued operator on $X$. \n\nLet $A$ be an operator on $X$. The \\emph{domain} of $A$ is $\\dom A :=\\menge{x\\in X}{Ax\\neq \\varnothing}$, the \\emph{graph} of $A$ is $\\gra A :=\\menge{(x,u)\\in X\\times X}{u\\in Ax}$, and the set of \\emph{fixed points} of $A$ is $\\Fix A :=\\menge{x\\in X}{x\\in Ax}$. The \\emph{inverse} of $A$, denoted by $A^{-1}$, is the operator with graph $\\gra A^{-1} :=\\menge{(u,x)\\in X\\times X}{u\\in Ax}$.\nWe say that $A$ is \\emph{Lipschitz continuous} with constant $\\ell\\in \\RP$ if it is single-valued and\n\\begin{equation}\n\\forall x, y\\in \\dom A,\\quad \\|Ax-Ay\\|\\leq \\ell\\|x-y\\|.\n\\end{equation}\nThe operator $A$ is \\emph{nonexpansive} if it is Lipschitz continuous with constant $1$, i.e.,\n\\begin{equation}\n\\forall x, y\\in \\dom A,\\quad \\|Ax-Ay\\|\\leq \\|x-y\\|.\n\\end{equation}\nAn operator $A\\colon X\\rightrightarrows X$ is said to be \\emph{monotone} if\n\\begin{equation}\n\\forall (x, u), (y, v)\\in \\gra A,\\quad \\scal{x-y}{u-v}\\geq 0,\n\\end{equation}\nand said to be \\emph{maximally monotone} if it is monotone and there exists no monotone operator $B\\colon X\\rightrightarrows X$ such that $\\gra B$ properly contains $\\gra A$. The \\emph{resolvent} of $A\\colon X\\rightrightarrows X$ is defined by\n\\begin{equation}\nJ_A:= (\\Id+ A)^{-1},\n\\end{equation}\nwhere $\\Id$ is the identity operator. The \\emph{relaxed resolvent} of $A$ with parameter $\\lambda\\in\\RP$ is defined by\n\\begin{equation}\nJ_A^\\lambda:= (1-\\lambda)\\Id+ \\lambda J_A.\n\\end{equation}\n","start":12234,"end":14216},{"raw_tex":"\\begin{definition}[$\\alpha$-monotonicity]\n\\label{d:alpha_mono}\nAn operator $A\\colon X\\rightrightarrows X$ is said to be \\emph{$\\alpha$-monotone} ($\\alpha\\in \\RR$) if\n\\begin{equation}\n\\label{e:alpha-mono}\n\\forall (x,u), (y,v)\\in \\gra A,\\quad \\scal{x-y}{u-v}\\geq \\alpha\\|x-y\\|^2.\n\\end{equation}\nThe constant $\\alpha$ is referred to as the \\emph{monotonicity constant}.\nWe also say that $A$ is \\emph{maximally $\\alpha$-monotone} if it is $\\alpha$-monotone and there is no $\\alpha$-monotone operator whose graph strictly contains $\\gra A$.  \n\\end{definition}\n\nWe note that $0$-monotonicity simply means monotonicity, that if $\\alpha>0$, then $\\alpha$-monotonicity is precisely the notion of \\emph{strong monotonicity} \\cite[Definition~22.1(iv)]{BC17}, and that if $\\alpha<0$, then $\\alpha$-monotonicity can be referred to as \\emph{weak monotonicity}.\n","start":15504,"end":16351},{"raw_tex":"\\begin{remark}[Lipschitz $\\alpha$-monotone operators]\n\\label{r:Lip_mono}\nSuppose that $A$ is Lipschitz continuous with constant $\\ell$. Then $A$ is single-valued and \n\\begin{equation}\n\\forall x,y\\in\\dom A,\\quad\n|\\scal{x-y}{Ax-Ay}|\\leq \\|x-y\\|\\cdot\\|Ax-Ay\\|\\leq \\ell\\|x-y\\|^2,\n\\end{equation}\nwhich yields\n\\begin{equation}\n\\label{e:L-inequality}\n\\forall x,y\\in\\dom A,\\quad -\\ell\\|x-y\\|^2\\leq \\scal{x-y}{Ax-Ay}\\leq \\ell\\|x-y\\|^2.\n\\end{equation}\nWe immediately deduce that $A$ is $(-\\ell)$-monotone. Now suppose, in addition, that $A$ is $\\alpha$-monotone. On the one hand, we can always assume without loss of generality that $\\alpha\\geq-\\ell$. On the other hand, it follows from the $\\alpha$-monotonicity and \\eqref{e:L-inequality} that $\\alpha\\leq \\ell$ as soon as $\\dom A$ has more than one element. Therefore, unless otherwise stated, whenever $A$ is both $\\alpha$-monotone and Lipschitz continuous with constant $\\ell$, we assume that $|\\alpha|\\leq\\ell$.\n\\end{remark}","start":22888,"end":23857},{"raw_tex":"Throughout this section, $A,B\\colon X\\rightrightarrows X$, $(\\gamma,\\delta,\\lambda,\\mu)\\in \\RPP^4$, and $\\kappa\\in \\left]0,1\\right[$. We define\n\\begin{subequations}\\label{e:JRT}\n\\begin{align}\nJ_1&:=J_{\\gamma A}=(\\Id+\\gamma A)^{-1}, \n&R_1&:=J_{\\gamma A}^\\lambda=(1-\\lambda)\\Id+\\lambda J_1,\\\\\nJ_2&:=J_{\\delta B}=(\\Id+\\delta B)^{-1},\n&R_2&:=J_{\\delta B}^\\mu=(1-\\mu)\\Id+\\mu J_2\n\\end{align}\nand consider the \\emph{adaptive DR operator} defined by\n\\begin{align}\\label{e:aDR}\nT:= (1-\\kappa)\\Id+ \\kappa R_2R_1.\n\\end{align}\n\\end{subequations}  \n","start":39838,"end":40374},{"raw_tex":"The adaptive DR operator is indeed motivated by the problem of finding a zero of the sum of two operators, that is,\n\\begin{equation}\\label{e:sumprob}\n\\text{find $x\\in X$ such that}\\quad 0\\in Ax+Bx.\n\\end{equation}\nWe also denote by\n\\begin{equation}\n\\zer(A+B) :=(A+B)^{-1}(0) =\\menge{x\\in X}{0\\in Ax+Bx}\n\\end{equation}\nthe set of solutions of problem \\eqref{e:sumprob}. Given a starting point $x_0\\in X$, the \\emph{adaptive DR algorithm} generates a sequence $(x_n)_\\nnn$, also called a \\emph{DR sequence}, by \n\\begin{equation}\\label{e:dr_seq}\n\\forall\\nnn,\\quad x_{n+1}\\in Tx_n.\n\\end{equation}\n","start":41114,"end":41706},{"raw_tex":"Recall that a sequence $(x_n)_\\nnn$ converges to $\\overline{x}$ with \\emph{$Q$-linear} (or simply \\emph{linear}) rate $\\rho\\in\\left[0,1\\right[$ if\n\\begin{equation}\n\\forall\\nnn,\\quad \\|x_{n+1}-\\overline{x}\\|\\leq \\rho\\|x_n-\\overline{x}\\|.\n\\end{equation}\n","start":62303,"end":62555}]},"statement":{"raw_tex":"\\begin{theorem}[linear convergence when $A$ is Lipschitz]\n\\label{t:linear}\nSuppose that either\n\\begin{enumerate}\n\\item\\label{t:linear_full} \n$A$ is $\\alpha$-monotone and Lipschitz continuous with constant $\\ell$, $B$ is maximally $\\beta$-monotone, and $\\alpha+\\beta >0$; or\n\\item\\label{t:linear_reduce} \n$A$ is Lipschitz continuous with constant $\\ell$, $B$ is maximally $\\beta$-monotone with $\\beta >\\ell$, and $\\alpha :=-\\ell$.\n\\end{enumerate}\nSuppose also that $\\zer(A+B)\\neq \\varnothing$ and that $(\\gamma,\\delta,\\lambda,\\mu)\\in\\RPP^2\\times \\left]1,+\\infty\\right[^2$ satisfies \\eqref{e:parameters}.\nThen $T$ is Lipschitz continuous with constant\n\\begin{equation}\n\\rho :=(1-\\kappa)\n\\sqrt{\\big(1+\\varepsilon_1(\\lambda-1)\\big)^2-\\varphi\\alpha_J} +\\kappa(1-\\varepsilon(\\lambda-1))\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}} <1,\n\\end{equation} \nwhere\n\\begin{subequations}\n\\begin{align}\n&\\varepsilon :=\\frac{\\mu-(2-2\\gamma\\beta)}{2(1+\\delta\\beta)},\\ \\varepsilon_1:=\\frac{\\kappa\\varepsilon}{1-\\kappa},\\\\\n&\\varphi :=\\varepsilon_1\\lambda[2(1+\\gamma\\alpha)+\\varepsilon_1\\big(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\big)],\\\\\n&\\text{$\\alpha_J$ as in \\eqref{e:alpha_J}}.\n\\end{align}\n\\end{subequations}\nConsequently, every DR sequence $(x_n)_\\nnn$ generated by $T$ converges strongly to the unique fixed point $\\overline{x}$ of $T$ with linear rate $\\rho$.\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}[linear convergence when $A$ is Lipschitz]\n\\label{t:linear}\nSuppose that either\n\\begin{enumerate}\n\\item\\label{t:linear_full} \n$A$ is $\\alpha$-monotone and Lipschitz continuous with constant $\\ell$, $B$ is maximally $\\beta$-monotone, and $\\alpha+\\beta >0$; or\n\\item\\label{t:linear_reduce} \n$A$ is Lipschitz continuous with constant $\\ell$, $B$ is maximally $\\beta$-monotone with $\\beta >\\ell$, and $\\alpha :=-\\ell$.\n\\end{enumerate}\nSuppose also that $\\zer(A+B)\\neq \\varnothing$ and that $(\\gamma,\\delta,\\lambda,\\mu)\\in\\RPP^2\\times \\left]1,+\\infty\\right[^2$ satisfies \\eqref{e:parameters}.\nThen $T$ is Lipschitz continuous with constant\n\\begin{equation}\n\\rho :=(1-\\kappa)\n\\sqrt{\\big(1+\\varepsilon_1(\\lambda-1)\\big)^2-\\varphi\\alpha_J} +\\kappa(1-\\varepsilon(\\lambda-1))\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}} <1,\n\\end{equation} \nwhere\n\\begin{subequations}\n\\begin{align}\n&\\varepsilon :=\\frac{\\mu-(2-2\\gamma\\beta)}{2(1+\\delta\\beta)},\\ \\varepsilon_1:=\\frac{\\kappa\\varepsilon}{1-\\kappa},\\\\\n&\\varphi :=\\varepsilon_1\\lambda[2(1+\\gamma\\alpha)+\\varepsilon_1\\big(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\big)],\\\\\n&\\text{$\\alpha_J$ as in \\eqref{e:alpha_J}}.\n\\end{align}\n\\end{subequations}\nConsequently, every DR sequence $(x_n)_\\nnn$ generated by $T$ converges strongly to the unique fixed point $\\overline{x}$ of $T$ with linear rate $\\rho$.\n\\end{theorem}","start":62557,"end":63949}]},"proof":{"raw_tex":"\\begin{fact}\n\\label{f:Minty}\nLet $A\\colon X\\rightrightarrows X$ be monotone and let $\\gamma\\in \\RPP$. Then $\\dom J_{\\gamma A} =X$ if and only if $A$ is maximally monotone.\n\\end{fact}\n\n\\begin{proof}\nBy definition, $\\dom J_{\\gamma A} =\\ran(\\Id+\\gamma A) :=(\\Id+\\gamma A)(X)$. Since $\\gamma\\in \\RPP$, it holds that $\\gamma A$ is monotone. According to Minty's theorem (see, e.g., \\cite[Theorem~21.1]{BC17}), $\\dom J_{\\gamma A} =\\ran(\\Id+\\gamma A) =X$ if and only if $\\gamma A$ is maximally monotone. By \\cite[Proposition~20.22]{BC17}, the latter occurs if and only if $A$ is maximally monotone. \n\\end{proof}\n\n\\begin{lemma}[monotonicity versus $\\alpha$-monotonicity]\n\\label{l:transfer}\nLet $A\\colon X\\rightrightarrows X$ and let $\\alpha, \\beta\\in \\RR$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{l:transfer_mono} \n$A$ is $\\alpha$-monotone if and only if $A-\\beta\\Id$ is $(\\alpha-\\beta)$-monotone.\n\\item\\label{l:transfer_maxmono}\n$A$ is maximally $\\alpha$-monotone if and only if $A-\\beta\\Id$ is maximally $(\\alpha-\\beta)$-monotone. \n\\end{enumerate}\nConsequently, $A$ is (resp., maximally) $\\alpha$-monotone if and only if $A-\\alpha\\Id$ is (resp., maximally) monotone.\n\\end{lemma}\n\n\\begin{proof}\n\\ref{l:transfer_mono}: We first have the equivalences\n\\begin{subequations}\n\\begin{align}\n(x,u)\\in \\gra A &\\iff (x,u-\\beta x)\\in \\gra(A-\\beta\\Id), \\\\\n(y,v)\\in \\gra A &\\iff (y,v-\\beta y)\\in \\gra(A-\\beta\\Id),\n\\end{align}\n\\end{subequations}\nand\n\\begin{equation}\n\\scal{x-y}{u-v}\\geq \\alpha\\|x-y\\|^2 \\iff \\scal{x-y}{(u-\\beta x)-(v-\\beta y)}\\geq (\\alpha-\\beta)\\|x-y\\|^2,\n\\end{equation}\nfrom which the conclusion follows.\n\n\\ref{l:transfer_maxmono}: Assume that $A$ is maximally $\\alpha$-monotone. By \\ref{l:transfer_mono}, $A-\\beta\\Id$ is $(\\alpha-\\beta)$-monotone. Now, suppose that $A-\\beta\\Id$ is not maximally $(\\alpha-\\beta)$-monotone. Then there must exist $B'\\colon X\\rightrightarrows X$ such that $B'$ is $(\\alpha-\\beta)$-monotone and $\\gra(A-\\beta\\Id)\\subsetneq \\gra B'$. It follows that $B :=B'+\\beta\\Id$ is $\\alpha$-monotone due to \\ref{l:transfer_mono} and that $\\gra A\\subsetneq \\gra B$, which contradict the maximal $\\alpha$-monotonicity of $A$. We deduce that if $A$ is maximally $\\alpha$-monotone, then $A-\\beta\\Id$ is maximally $(\\alpha-\\beta)$-monotone. This also implies that if $A-\\beta\\Id$ is maximally $(\\alpha-\\beta)$-monotone, then $A =(A-\\beta\\Id)+\\beta\\Id$ is maximally $\\alpha$-monotone, and we are done. \n\\end{proof}\n\n\\begin{lemma}[resolvents of $\\alpha$-monotone operators]\n\\label{l:resol}\nLet $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{l:resol_gen}\nFor all $(x,a), (y,b)\\in \\gra J_{\\gamma A}$,\n\\begin{subequations}\n\\begin{align}\n\\label{e:resol_gen}\n\\scal{x-y}{a-b} &\\geq (1+\\gamma\\alpha)\\|a-b\\|^2 \\quad\\text{and}\\\\\n\\|x-y\\| &\\geq (1+\\gamma\\alpha)\\|a-b\\|.\n\\end{align} \n\\end{subequations}\n\\item\\label{l:resol_single}\nIf $J_{\\gamma A}$ is single-valued, then, for all $x,y\\in \\dom J_{\\gamma A}$,\n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq (1+\\gamma\\alpha)\\|J_{\\gamma A}x-J_{\\gamma A}y\\|^2,\n\\end{equation}\ni.e., $J_{\\gamma A}$ is $(1+\\gamma\\alpha)$-cocoercive.\n\\end{enumerate}\n\\end{lemma}\n\n\\begin{proof}\n\\ref{l:resol_gen}: Let $(x,a), (y,b)\\in \\gra J_{\\gamma A}$. Then $x\\in (\\Id+\\gamma A)a$, $y\\in (\\Id+\\gamma A)b$, and so $x =a+\\gamma u$, $y =b+\\gamma v$ for some $u\\in Aa$, $v\\in Ab$. We derive from the $\\alpha$-monotonicity of $A$ that \n\\begin{subequations}\n\\begin{align}\n\\scal{x-y}{a-b}\n&=\\scal{(a+\\gamma u)-(b+\\gamma v)}{a-b}\\\\\n&=\\|a-b\\|^2+\\gamma\\scal{a-b}{u-v}\\\\\n&\\geq \\|a-b\\|^2+\\gamma\\alpha\\|a-b\\|^2\\\\\n&=(1+\\gamma\\alpha)\\|a-b\\|^2.\n\\end{align}\n\\end{subequations}\nNow, by the Cauchy--Schwarz inequality,\n\\begin{equation}\n\\|x-y\\|\\|a-b\\|\\geq \\scal{x-y}{a-b}\\geq (1+\\gamma\\alpha)\\|a-b\\|^2.\n\\end{equation}\nwhich gives $\\|x-y\\|\\geq (1+\\gamma\\alpha)\\|a-b\\|$ while noting that this is trivial when $a =b$. \n\n\\ref{l:resol_single}: This is a direct consequence of \\ref{l:resol_gen}.\n\\end{proof}\n\n\\begin{proposition}[single-valuedness and full domain]\n\\label{p:resol}\nLet $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$ such that $1+\\gamma\\alpha >0$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{p:resol_single} \n$J_{\\gamma A}$ is single-valued.\n\\item\\label{p:resol_dom} \n$\\dom J_{\\gamma A} =X$ if and only if $A$ is maximally $\\alpha$-monotone.\n\\end{enumerate}\n\\end{proposition}\n\n\\begin{proof}\n\\ref{p:resol_single}: This follows from Lemma~\\ref{l:resol}\\ref{l:resol_gen}.\n\n\\ref{p:resol_dom}: By Lemma~\\ref{l:transfer}\\ref{l:transfer_mono}, $A':=A-\\alpha\\Id$ is monotone.\nNoting that $(\\beta T)^{-1} =T^{-1}\\circ \\frac{1}{\\beta}\\Id$ for any operator $T$ and any $\\beta\\in \\RR\\smallsetminus\\{0\\}$, we have\n\\begin{subequations}\n\\begin{align}\nJ_{\\gamma A}&=(\\Id+\\gamma A)^{-1}=\\big((1+\\gamma\\alpha)\\Id+\\gamma(A-\\alpha\\Id)\\big)^{-1}\\\\\n&=\\left(\\Id+\\frac{\\gamma}{1+\\gamma\\alpha} A'\\right)^{-1} \\circ \\left(\\frac{1}{1+\\gamma\\alpha}\\Id\\right)\\\\\n&=J_{\\frac{\\gamma}{1+\\gamma\\alpha}A'}\\circ \\left(\\frac{1}{1+\\gamma\\alpha}\\Id\\right).\n\\end{align}\n\\end{subequations}\nIt follows that \n\\begin{subequations}\n\\begin{align}\n\\dom J_{\\gamma A} =X & \\iff \\dom J_{\\frac{\\gamma}{1+\\gamma\\alpha}A'} =X &&\\\\\n& \\iff \\text{$A'$ is maximally monotone} &&\\hspace*{-.8in}\\text{(by Fact~\\ref{f:Minty})} \\\\\n& \\iff \\text{$A$ is maximally $\\alpha$-monotone} &&\\hspace*{-.8in}\\text{(by Lemma~\\ref{l:transfer}\\ref{l:transfer_maxmono})}.\n\\end{align}\n\\end{subequations}\nThe proof is complete.\n\\end{proof}\n\n\\begin{proposition}[maximal $\\alpha$-monotonicity]\n\\label{p:maxalpha}\nThe following statements hold: \n\\begin{enumerate}\n\\item\\label{p:maxalpha_mono} \nLet $A\\colon X\\rightrightarrows X$ and $\\alpha\\in \\RP$. Then $A$ is maximally $\\alpha$-monotone if and only if $A$ is $\\alpha$-monotone and maximally monotone.\n\\item\\label{p:maxalpha_cont} \nLet $A\\colon X\\to X$ and $\\alpha\\in \\RR$. Then $A$ is maximally $\\alpha$-monotone if $A$ is $\\alpha$-monotone and continuous with full domain.\n\\end{enumerate}\n\\end{proposition}\n\n\\begin{proof}\n\\ref{p:maxalpha_mono}: Since $\\alpha\\geq 0$, it follows from Fact~\\ref{f:Minty} that $A$ is $\\alpha$-monotone and maximally monotone if and only if $A$ is $\\alpha$-monotone and $\\dom J_A =X$, which, by Proposition~\\ref{p:resol}\\ref{p:resol_dom}, happen if and only if $A$ is maximally $\\alpha$-monotone.\n\n\\ref{p:maxalpha_cont}: Set $A' :=A-\\alpha\\Id$. Then $A'$ is monotone (due to Lemma~\\ref{l:transfer}\\ref{l:transfer_mono}) and continuous with full domain. By \\cite[Corollary~20.28]{BC17}, $A'$ is maximally monotone, and by Lemma~\\ref{l:transfer}\\ref{l:transfer_maxmono}, $A$ is maximally $\\alpha$-monotone. \n\\end{proof}\n\n\\begin{lemma}[resolvents of Lipschitz $\\alpha$-monotone operators]\n\\label{l:L-resol}\nLet $A\\colon X\\to X$ be Lipschitz continuous with constant $\\ell$ and let $\\gamma\\in \\RPP$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{l:L-resol_only} \nFor all $(x,a), (y,b)\\in \\gra J_{\\gamma A}$,\n\\begin{subequations}\n\\begin{align}\n\\label{e:expan}\n\\|a-b\\|&\\geq \\frac{1}{1+\\gamma\\ell}\\|x-y\\|,\\\\\n\\label{e:L-resol_only}\n\\scal{x-y}{a-b}&\\geq \\frac{1}{2}\\|x-y\\|^2 +\\frac{1}{2}(1-\\gamma^2\\ell^2)\\|a-b\\|^2,\n\\end{align}\n\\end{subequations}\nand if $\\gamma\\ell\\leq 1$, then\n\\begin{equation}\\label{e:L-resol_only1}\n\\scal{x-y}{a-b}\\geq \\frac{1}{1+\\gamma\\ell}\\|x-y\\|^2.\n\\end{equation}\n\\item\\label{l:L-resol_mono}\nIf $A$ is $\\alpha$-monotone with $1+\\gamma\\alpha >0$, then, for all $x, y\\in \\dom J_{\\gamma A}$, \n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq (1+\\gamma\\alpha)\\alpha_J\\|x-y\\|^2,\n\\end{equation}\nwhere \n\\begin{subequations}\n\\label{e:alpha_J}\n\\begin{equation}\n\\alpha_J \n:=\\begin{cases}\n\\dfrac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2} &\\text{if~} \\gamma\\ell\\geq 1 \n,\\\\\n\\dfrac{1}{(1+\\gamma\\alpha)(1+\\gamma\\ell)} &\\text{if~} \\gamma\\ell\\leq 1;\n\\end{cases}\n\\end{equation}\nand if additionally $A$ satisfies \\eqref{e:alpha-mono} with equality, then \n\\begin{equation}\n\\alpha_J :=\\frac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}. \n\\end{equation}\n\\end{subequations}\n\\end{enumerate}\n\\end{lemma}\n\n\\begin{proof}\n\\ref{l:L-resol_only}: Let $(x,a), (y,b)\\in \\gra J_{\\gamma A}$. Then $x =a+\\gamma Aa$ and $y =b+\\gamma Ab$. By Lipschitz continuity, $\\|Aa-Ab\\|\\leq \\ell\\|a-b\\|$. It follows that\n\\begin{equation}\n\\|x-y\\|\\leq \\|a-b\\| +\\gamma\\|Aa-Ab\\|\\leq (1+\\gamma\\ell)\\|a-b\\|\n\\end{equation}\nand that\n\\begin{subequations}\n\\label{e:L-resol}\n\\begin{align}\n2\\scal{x-y}{a-b} &=\\|x-y\\|^2 +\\|a-b\\|^2 -\\|(x-a)-(y-b)\\|^2\\\\\n&=\\|x-y\\|^2 +\\|a-b\\|^2 -\\gamma^2\\|Aa-Ab\\|^2\\\\\n&\\geq \\|x-y\\|^2 +(1-\\gamma^2\\ell^2)\\|a-b\\|^2.\n\\end{align}\n\\end{subequations}\nIf $\\gamma\\ell\\leq 1$, then combining the above inequalities yields\n\\begin{equation}\n2\\scal{x-y}{a-b}\\geq \\|x-y\\|^2 +\\frac{1-\\gamma^2\\ell^2}{(1+\\gamma\\ell)^2}\\|x-y\\|^2 =\\frac{2}{1+\\gamma\\ell}\\|x-y\\|^2,\n\\end{equation}\nand we get the claim.\n\n\\ref{l:L-resol_mono}: We first note that $J_{\\gamma A}$ is single-valued due to Proposition~\\ref{p:resol}\\ref{p:resol_single}. Then \\eqref{e:L-resol} reads as\n\\begin{equation}\n\\label{e:L-resol'}\n2\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\|x-y\\|^2 +(1-\\gamma^2\\ell^2)\\|J_{\\gamma A}x-J_{\\gamma A}y\\|^2.\n\\end{equation}\nWe claim that if $\\gamma\\ell\\geq 1$ or $A$ satisfies \\eqref{e:alpha-mono} with equality, then \n\\begin{equation}\n\\label{e:L-resol''}\n2\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\|x-y\\|^2 +\\frac{1-\\gamma^2\\ell^2}{1+\\gamma\\alpha}\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}.\n\\end{equation} \nIndeed, the former case implies $1-\\gamma^2\\ell^2\\leq 0$ and, by combining \\eqref{e:L-resol'} with Lemma~\\ref{l:resol}\\ref{l:resol_single} and noting that $1+\\gamma\\alpha >0$, we get \\eqref{e:L-resol''}.  \nIn the latter case, Lemma~\\ref{l:resol}\\ref{l:resol_single} reduces to\n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y} =(1+\\gamma\\alpha)\\|J_{\\gamma A}x-J_{\\gamma A}y\\|^2.\n\\end{equation} \nSubstituting this into \\eqref{e:L-resol'}, we also obtain \\eqref{e:L-resol''}. \n\nNow, in view of Remark~\\ref{r:Lip_mono}, $1+2\\gamma\\alpha+\\gamma^2\\ell^2\\geq 1+2\\gamma\\alpha+\\gamma^2\\alpha^2 =(1+\\gamma\\alpha)^2 >0$. It thus follows from \\eqref{e:L-resol''} that \n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\frac{1+\\gamma\\alpha}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}\\|x-y\\|^2.\n\\end{equation}\nFinally, if $\\gamma\\ell\\leq 1$, then, by \\ref{l:L-resol_only}, \n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\frac{1}{1+\\gamma\\ell}\\|x-y\\|^2,\n\\end{equation}\nand the conclusion follows.\n\\end{proof}\n\n\\begin{proposition}[linear combinations of resolvents and the identity]\n\\label{p:cresol}\nLet $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$. Set $J:=J_{\\gamma A}$ and define $Q :=\\nu\\Id+\\lambda J$ with $\\nu, \\lambda\\in \\RR$. \n\\begin{enumerate}\n\\item\\label{p:cresol_mono} \nSuppose that $J$ is single-valued and $\\nu\\lambda\\leq 0$. Then, for all $x,y\\in \\dom J$, \n\\begin{equation}\n\\|Qx-Qy\\|^2\\leq \\nu^2\\|x-y\\|^2 +\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\|Jx-Jy\\|^2.\n\\end{equation}\n\\item\\label{p:cresol_Lip}\nSuppose that $A$ is Lipschitz continuous with constant $\\ell$, $1+\\gamma\\alpha >0$, and $\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\leq 0$. Then $Q$ is Lipschitz continuous with constant    \n\\begin{equation}\n\\label{e:LipQ}\n\\rho :=\\sqrt{\\nu^2+\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\alpha_J},\n\\end{equation}\nwhere $\\alpha_J$ is defined as \\eqref{e:alpha_J}.\nIf additionally $\\lambda\\left(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\right)\\geq 0$ whenever $\\gamma\\ell <1$, then the Lipschitz constant \\eqref{e:LipQ} can be improved to\n\\begin{equation}\n\\overline{\\rho} :=\\sqrt{\\nu^2+\\frac{\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}}\\leq \\rho.\n\\end{equation}\n\\end{enumerate}\n\\end{proposition}\n\n\\begin{proof}\nLet $x,y\\in \\dom J$. By the definition of $Q$,\n\\begin{subequations}\n\\label{e:identityQ}\n\\begin{align}\n\\|Qx-Qy\\|^2\n&=\\|\\nu(x-y)+\\lambda(Jx-Jy)\\|^2\\\\\n&=\\nu^2\\|x-y\\|^2 +2\\nu\\lambda\\scal{x-y}{Jx-Jy} +\\lambda^2\\|Jx-Jy\\|^2.\n\\end{align}\n\\end{subequations}\n\n\\ref{p:cresol_mono}: Since $\\nu\\lambda\\leq 0$, combining \\eqref{e:identityQ} with Lemma~\\ref{l:resol}\\ref{l:resol_single} yields\n\\begin{subequations}\n\\begin{align}\n\\|Qx-Qy\\|^2\n&\\leq \\nu^2\\|x-y\\|^2 +2\\nu\\lambda(1+\\gamma\\alpha)\\|Jx-Jy\\|^2 +\\lambda^2\\|Jx-Jy\\|^2\\\\\n&=\\nu^2\\|x-y\\|^2 +\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\|Jx-Jy\\|^2.\n\\end{align}\n\\end{subequations}\n\n\\ref{p:cresol_Lip}: First, according to Proposition~\\ref{p:resol}\\ref{p:resol_single}, $J$ is single-valued, and so is $Q$. Next, using \\eqref{e:identityQ}, Lemma~\\ref{l:resol}\\ref{l:resol_single}, and Lemma~\\ref{l:L-resol}\\ref{l:L-resol_mono} and noting that $\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\leq 0$, we have\n\\begin{subequations}\n\\label{e:cresol_Lip}\n\\begin{align}\n\\|Qx-Qy\\|^2 \n&\\leq \\nu^2\\|x-y\\|^2 +2\\nu\\lambda\\scal{x-y}{Jx-Jy} +\\frac{\\lambda^2}{1+\\gamma\\alpha}\\scal{x-y}{Jx-Jy}\\\\ \n&= \\nu^2\\|x-y\\|^2 +\\frac{\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)}{1+\\gamma\\alpha}\\scal{x-y}{Jx-Jy}\\\\  \n&\\leq \\nu^2\\|x-y\\|^2 +\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\alpha_J\\|x-y\\|^2 =\\rho^2\\|x-y\\|^2, \n\\end{align}\n\\end{subequations}\nwhich implies that $Q$ is Lipschitz continuous with constant $\\rho$.\n\nFor the last statement, we show that $\\alpha_J$ in formula \\eqref{e:LipQ} can be replaced by \n\\begin{equation}\n\\overline{\\alpha}_J :=\\frac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}.\n\\end{equation}\nIf $\\scal{x-y}{Jx-Jy}\\geq (1+\\gamma\\alpha)\\overline{\\alpha}_J\\|x-y\\|^2$, then \\eqref{e:cresol_Lip} also holds with $\\alpha_J$ replaced by $\\overline{\\alpha}_J$. Now, assume that $\\scal{x-y}{Jx-Jy} <(1+\\gamma\\alpha)\\overline{\\alpha}_J\\|x-y\\|^2$. By Lemma~\\ref{l:L-resol}\\ref{l:L-resol_mono}, we must have $\\gamma\\ell <1$, and then, by assumption, $\\lambda\\left(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\right)\\geq 0$. It now follows from \\eqref{e:identityQ} and \\eqref{e:L-resol_only} that\n\\begin{subequations}\n\\begin{align}\n\\|Qx-Qy\\|^2\n&\\leq \\nu^2\\|x-y\\|^2 +2\\nu\\lambda\\scal{x-y}{Jx-Jy} \\notag\\\\ \n&\\qquad +\\frac{\\lambda^2}{1-\\gamma^2\\ell^2}(2\\scal{x-y}{Jx-Jy} -\\|x-y\\|^2)\\\\\n&=\\Big(\\nu^2-\\frac{\\lambda^2}{1-\\gamma^2\\ell^2}\\Big)\\|x-y\\|^2 +2\\lambda\\Big(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\Big)\\scal{x-y}{Jx-Jy}\\\\\n&\\leq \\Big(\\nu^2-\\frac{\\lambda^2}{1-\\gamma^2\\ell^2}\\Big)\\|x-y\\|^2 +2\\lambda\\Big(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\Big)(1+\\gamma\\alpha)\\overline{\\alpha}_J\\|x-y\\|^2\\\\\n&=\\Big(\\nu^2+\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\overline{\\alpha}_J\\Big)\\|x-y\\|^2 =\\overline{\\rho}^2\\|x-y\\|^2.\n\\end{align}\n\\end{subequations}\nFinally, we will prove $\\overline{\\alpha}_J\\geq \\alpha_J$, which implies $\\overline{\\rho}\\leq \\rho$, i.e., the Lipschitz constant is indeed improved. From the definition of $\\alpha_J$, it suffices to consider the case in which $\\alpha_J =1/((1+\\gamma\\alpha)(1+\\gamma\\ell))$. Then $\\gamma\\ell\\leq 1$. \nSince $|\\alpha|\\leq \\ell$ (see Remark~\\ref{r:Lip_mono}), it holds that\n\\begin{subequations}\n\\begin{align}\n0 <(1+\\gamma\\alpha)^2\\leq 1+2\\gamma\\alpha+\\gamma^2\\ell^2\n&\\leq \n1+2\\gamma\\alpha+\\gamma^2\\ell^2\n+(\\gamma\\ell-\\gamma\\alpha)-\\gamma\\ell(\\gamma\\ell-\\gamma\\alpha)\\\\\n&=1+\\gamma\\alpha+\\gamma\\ell+(\\gamma\\alpha)(\\gamma\\ell)\\\\\n&=(1+\\gamma\\alpha)(1+\\gamma\\ell),\n\\end{align}\n\\end{subequations}\nand so \n\\begin{equation}\n\\overline{\\alpha}_J =\\frac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}\\geq \\alpha_J =\\frac{1}{(1+\\gamma\\alpha)(1+\\gamma\\ell)}.\n\\end{equation}\nThe proof is complete.\n\\end{proof}\n\n\\begin{corollary}[relaxed resolvents of $\\alpha$-monotone operators]\n\\label{c:rresol}\nLet $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$. Suppose that $J:=J_{\\gamma A}$ is single-valued and define $R:=(1-\\lambda)\\Id +\\lambda J$ with $\\lambda\\in \\RP$, and $Q:=R+\\varepsilon\\Id$ with $\\varepsilon\\in \\RR$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{c:rresol_relax} \nIf $\\lambda\\geq 1$, then, for all $x, y\\in \\dom J$, \n\\begin{equation}\n\\|Rx-Ry\\|^2\\leq (\\lambda-1)^2\\|x-y\\|^2 -\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda\\big)\\|Jx-Jy\\|^2.\n\\end{equation}\n\\item\\label{c:rresol_perturb} \nIf $\\varepsilon\\leq \\lambda-1$, then, for all $x, y\\in \\dom J$, \n\\begin{align}\n\\|Qx-Qy\\|^2 &\\leq (\\lambda-1-\\varepsilon)^2\\|x-y\\|^2 \\notag\\\\ \n&\\quad -\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda-2\\varepsilon(1+\\gamma\\alpha)\\big)\\|Jx-Jy\\|^2.\n\\end{align}\nConsequently, if additionally $(\\lambda-1)(2+2\\gamma\\alpha)-\\lambda-2\\varepsilon(1+\\gamma\\alpha)\\geq 0$, then $Q$ is Lipschitz continuous with constant $(\\lambda-1-\\varepsilon)$.\n\\end{enumerate} \n\\end{corollary}\n\n\\begin{proof}\nBecause \\ref{c:rresol_relax} is a consequence of \\ref{c:rresol_perturb} with $\\varepsilon =0$, it suffices to prove only the latter. To this end, noting that $Q =R+\\varepsilon\\Id =(1-\\lambda+\\varepsilon)\\Id+\\lambda J$ and using Proposition~\\ref{p:cresol}\\ref{p:cresol_mono} with $\\nu =1-\\lambda+\\varepsilon \\leq 0$, we have that\n\\begin{subequations}\n\\begin{align}\n\\|Qx-Qy\\|^2&\\leq (1-\\lambda+\\varepsilon)^2\\|x-y\\|^2 +\\lambda\\left(2(1-\\lambda+\\varepsilon)(1+\\gamma\\alpha)+\\lambda\\right)\\|Jx-Jy\\|^2\\\\\n&=(\\lambda-1-\\varepsilon)^2\\|x-y\\|^2 -\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda-2\\varepsilon(1+\\gamma\\alpha)\\big)\\|Jx-Jy\\|^2\n\\end{align}\n\\end{subequations}\nwhich proves \\ref{c:rresol_perturb}. \n\\end{proof}\n\n\\begin{corollary}[relaxed resolvents of Lipschitz $\\alpha$-monotone operators]\n\\label{c:L-rresol}\nLet $A\\colon X\\to X$ be $\\alpha$-monotone and Lipschitz continuous with constant $\\ell$. Also let $\\gamma\\in\\RPP$ and $\\lambda\\in\\RPP$ be such that\n\\begin{equation}\\label{e:cL-rresol-1}\n1+\\gamma\\alpha>0\n\\ \\text{ and }\\ \n\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\geq 0.\n\\end{equation}\nDefine $J:=J_{\\gamma A}$, $R:=(1-\\lambda)\\Id +\\lambda J$, and $Q :=\\Id-\\varepsilon R$ with $\\varepsilon\\in \\RP$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{c:L-rresol_relax} \n$R$ is Lipschitz continuous with constant \n\\begin{equation}\n\\sqrt{(\\lambda-1)^2-\\frac{\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda\\big)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}}.\n\\end{equation} \n\\item\\label{c:L-rresol_perturb} \n$Q$ is Lipschitz continuous with constant \n\\begin{equation}\n\\sqrt{(1+\\varepsilon(\\lambda-1))^2 -\\varepsilon\\lambda\\Big[2(1+\\gamma\\alpha) +\\varepsilon\\big(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\big)\\Big]\\alpha_J},\n\\end{equation}\nwhere $\\alpha_J$ is defined as \\eqref{e:alpha_J}.\n\\end{enumerate}\n\\end{corollary}\n\n\\begin{proof}\n\\ref{c:L-rresol_relax}: We observe that $R=\\nu\\Id+\\lambda J$ with $\\nu :=1-\\lambda$, that\n\\begin{equation}\n\\lambda(2\\nu(1+\\gamma\\alpha)+\\lambda) =-\\lambda\\Big(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\Big)\\leq 0,\n\\end{equation} \nand that, whenever $\\gamma\\ell <1$, \n\\begin{equation}\n\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}=\n(1-\\lambda)+\\frac{\\lambda}{1-\\gamma^2\\ell^2} >1-\\lambda+\\lambda =1 >0.\n\\end{equation}\nApplying Proposition~\\ref{p:cresol}\\ref{p:cresol_Lip} to $R=\\nu\\Id+\\lambda\\Id$ implies that $R$ is Lipschitz continuous with constant\n\\begin{equation}\n\\sqrt{(1-\\lambda)^2+\\frac{\\lambda\\big(2(1-\\lambda)(1+\\gamma\\alpha)+\\lambda\\big)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}},\n\\end{equation} \nwhich gives the claim.\n\n\\ref{c:L-rresol_perturb}: Using the first part of Proposition~\\ref{p:cresol}\\ref{p:cresol_Lip} and writing $Q =\\widetilde{\\nu}\\Id+\\widetilde{\\lambda}J$ with $\\widetilde{\\nu} :=1+\\varepsilon(\\lambda-1)$ and $\\widetilde{\\lambda} :=-\\varepsilon\\lambda$, it suffices to check that\n\\begin{equation}\\label{e:cL-rresol-2}\n\\widetilde{\\lambda}(2\\widetilde{\\nu}(1+\\gamma\\alpha)+\\widetilde{\\lambda})=(-\\varepsilon\\lambda)\\big( 2(1+\\varepsilon(\\lambda-1))(1+\\gamma\\alpha)+(-\\varepsilon\\lambda) \\big)\\leq 0.\n\\end{equation}\nIndeed, we have that $\\varepsilon\\geq 0$, $\\lambda >0$, and  \n\\begin{subequations}\n\\begin{align}\n2(1+\\varepsilon(\\lambda-1))(1+\\gamma\\alpha) -\\varepsilon\\lambda \n&=2(1+\\gamma\\alpha) +\\varepsilon\\big(2(\\lambda-1)(1+\\gamma\\alpha)-\\lambda\\big)\\\\\n&=2(1+\\gamma\\alpha) +\\varepsilon(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)) >0\n\\end{align}\n\\end{subequations}\nby \\eqref{e:cL-rresol-1}. So \\eqref{e:cL-rresol-2} holds and the conclusion follows.\n\\end{proof}\n\n\\begin{lemma}[single-valuedness and full domain of adaptive DR operator]\n\\label{l:domT}\nSuppose that $A$ and $B$ are maximally $\\alpha$- and $\\beta$-monotone with $\\alpha+\\beta\\geq 0$. \nThen there exists $(\\gamma,\\delta,\\lambda,\\mu)\\in \\RPP^2\\times \\left]1,+\\infty\\right[^2$ such that\n\\begin{subequations}\n\\label{e:parameters}\n\\begin{align}\n&1+2\\gamma\\alpha >0,\n\\label{e:parameters_a}\\\\\n&\\mu\\in \\left[2-2\\gamma\\beta, 2+2\\gamma\\alpha\\right],\n\\label{e:parameters_b} \\\\\n&(\\lambda-1)(\\mu-1) =1, \\text{~and~} \\delta =(\\lambda-1)\\gamma. \\label{e:parameters_c}\n\\end{align}\n\\end{subequations}\nMoreover, \\eqref{e:parameters} implies that $\\min\\{1+\\gamma\\alpha, 1+\\delta\\beta\\} >0$ and that $J_1$, $J_2$, and $T$ are single-valued and have full domain. \n\\end{lemma}\n\n\\begin{proof}\nTo show the existence, we first take $\\gamma>0$ such that $1/\\gamma >-2\\alpha$. Then $1+2\\gamma\\alpha >0$ and $2+2\\gamma\\alpha =1+(1+2\\gamma\\alpha) >1$.\nUsing $\\alpha+\\beta\\geq 0$, we derive that\n\\begin{equation}\n2+2\\gamma\\alpha =2\\gamma(\\alpha+\\beta) +(2-2\\gamma\\beta)\\geq 2-2\\gamma\\beta.\n\\end{equation}\nHence, we can always choose $\\mu >1$ satisfying \\eqref{e:parameters_b}. Next, with such $\\mu$, we define $\\lambda :=\\mu/(\\mu-1) =1+1/(\\mu-1) >1$ and $\\delta :=(\\lambda-1)\\gamma$. Then \\eqref{e:parameters_c} is clearly satisfied. \n\nNow, take any $(\\gamma,\\delta,\\lambda,\\mu)$ satisfying \\eqref{e:parameters}. We have\n\\begin{equation}\n1+\\delta\\beta =(\\lambda-1)(\\mu-1)+(\\lambda-1)\\gamma\\beta =\\frac{1}{2}(\\lambda-1)(\\mu + \\mu - (2-2\\gamma\\beta)) >0.\n\\end{equation}\nThus, $\\min\\{1+\\gamma\\alpha, 1+\\delta\\beta\\} >0$. The remaining conclusion follows from Proposition~\\ref{p:resol}. \n\\end{proof}\n\n\\begin{proof}\nIn view of Remark~\\ref{r:Lip_mono}, assumption \\ref{t:linear_reduce} implies assumption \\ref{t:linear_full} because if $A$ is Lipschitz continuous with constant $\\ell$, then $A$ is also $\\alpha$-monotone with $\\alpha :=-\\ell$. It thus suffices to assume \\ref{t:linear_full}.  \nFirst, Proposition~\\ref{p:maxalpha}\\ref{p:maxalpha_cont} implies that $A$ is maximally $\\alpha$-monotone. Next, we learn from Lemma~\\ref{l:domT} that \n\\begin{equation}\n\\label{e:domT'}\n\\min\\{1+\\gamma\\alpha,1+\\delta\\beta\\} >0\n\\end{equation}\nand that all operators $J_1,J_2$, and $T$ are single-valued and have full domain. \n\nBy the choice of $\\mu$, it holds that $0 <\\mu-1 \\leq 1+2\\gamma\\alpha$, and so\n\\begin{equation}\n\\lambda =1+\\frac{1}{\\mu-1}\\geq 1+\\frac{1}{1+2\\gamma\\alpha} =\\frac{2(1+\\gamma\\alpha)}{1+2\\gamma\\alpha},\n\\end{equation} \nwhich yields\n\\begin{equation}\n\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\geq 0.\n\\end{equation}\nFrom $\\mu\\geq 2-2\\gamma\\beta$, we have that $\\varepsilon\\geq 0$ and $\\varepsilon_1\\geq 0$. It follows that $\\varphi\\geq 0$ and that\n\\begin{equation}\\label{e:180823a}\n\\varphi =0 \\iff \\varepsilon_1 =0 \\iff \\varepsilon =0 \\iff \\mu =2-2\\gamma\\beta.\n\\end{equation}\nDefine $Q_1 :=\\Id -\\varepsilon_1 R_1$. Using Corollary~\\ref{c:L-rresol} and noting that $\\lambda =\\mu(\\lambda-1)$, we derive that $R_1$ is Lipschitz continuous with constant  \n\\begin{equation}\n\\sqrt{(\\lambda-1)^2-\\frac{\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda\\big)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}} =(\\lambda-1)\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}},\n\\end{equation} \nand that $Q_1$ is Lipschitz continuous with constant \n\\begin{equation}\n\\label{e:rho1}\n\\rho_1 :=\\sqrt{(1+\\varepsilon_1(\\lambda-1))^2 -\\varphi\\alpha_J}\\leq 1+\\varepsilon_1(\\lambda-1),\n\\end{equation}\nwhere $\\alpha_J$ is defined as in \\eqref{e:alpha_J}. It follows from \\eqref{e:180823a} that the inequality is strict whenever $\\mu >2-2\\gamma\\beta$.\n\nNext, define $Q_2 :=R_2+\\varepsilon\\Id$. \nSince $\\gamma =(\\mu-1)\\delta$, we note that\n\\begin{equation}\n\\varepsilon =\\frac{\\mu-(2-2\\gamma\\beta)}{2(1+\\delta\\beta)} =\\frac{(\\mu-1)(2+2\\delta\\beta)-\\mu}{2(1+\\delta\\beta)} <\\mu-1,\n\\end{equation}\nwhich also gives\n\\begin{equation}\n(\\mu-1)(2+2\\delta\\beta)-\\mu-2\\varepsilon(1+\\delta\\beta) =0.\n\\end{equation}\nBy Corollary~\\ref{c:rresol}\\ref{c:rresol_perturb}, $Q_2$ is Lipschitz continuous with constant $(\\mu-1-\\varepsilon)$. Combining with the Lipschitz continuity of $R_1$ and noting that $(\\mu-1)(\\lambda-1) =1$, we have that $Q_2R_1$ is Lipschitz continuous with constant\n\\begin{subequations}\n\\label{e:rho2}\n\\begin{align}\n\\rho_2\n&:=(\\mu-1-\\varepsilon)(\\lambda-1)\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}}\\\\\n&\\phantom{:}=(1-\\varepsilon(\\lambda-1))\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}}\\\\\n&\\phantom{:}\\leq 1-\\varepsilon(\\lambda-1),\n\\end{align}\n\\end{subequations} \nwhere the inequality is strict whenever $\\mu <2+2\\gamma\\alpha$.  \n\nNow, we express\n\\begin{subequations}\n\\begin{align}\nT&=(1-\\kappa)\\Id -(1-\\kappa)\\varepsilon_1R_1 + \\kappa R_2R_1+\\kappa\\varepsilon R_1\\\\\n&=(1-\\kappa)(\\Id-\\varepsilon_1R_1)+\\kappa(R_2+\\varepsilon\\Id)R_1\\\\\n&=(1-\\kappa)Q_1+\\kappa Q_2R_1.\n\\end{align}\n\\end{subequations}\nWe note from $\\alpha+\\beta >0$ that $2+2\\gamma\\alpha >2-2\\gamma\\beta$, so at least one of two inequalities in \\eqref{e:rho1} and \\eqref{e:rho2} is strict.\nTherefore, $T$ is Lipschitz continuous with constant\n\\begin{equation}\n\\rho :=(1-\\kappa)\\rho_1+\\kappa \\rho_2 <(1-\\kappa)(1+\\varepsilon_1(\\lambda-1))+\\kappa(1-\\varepsilon(\\lambda-1))=1,\n\\end{equation}\nwhich completes the proof.\n\\end{proof}","units":[{"raw_tex":"\\begin{fact}\n\\label{f:Minty}\nLet $A\\colon X\\rightrightarrows X$ be monotone and let $\\gamma\\in \\RPP$. Then $\\dom J_{\\gamma A} =X$ if and only if $A$ is maximally monotone.\n\\end{fact}","start":14295,"end":14477},{"raw_tex":"\\begin{proof}\nBy definition, $\\dom J_{\\gamma A} =\\ran(\\Id+\\gamma A) :=(\\Id+\\gamma A)(X)$. Since $\\gamma\\in \\RPP$, it holds that $\\gamma A$ is monotone. According to Minty's theorem (see, e.g., \\cite[Theorem~21.1]{BC17}), $\\dom J_{\\gamma A} =\\ran(\\Id+\\gamma A) =X$ if and only if $\\gamma A$ is maximally monotone. By \\cite[Proposition~20.22]{BC17}, the latter occurs if and only if $A$ is maximally monotone. \n\\end{proof}","start":14478,"end":14898},{"raw_tex":"\\begin{lemma}[monotonicity versus $\\alpha$-monotonicity]\n\\label{l:transfer}\nLet $A\\colon X\\rightrightarrows X$ and let $\\alpha, \\beta\\in \\RR$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{l:transfer_mono} \n$A$ is $\\alpha$-monotone if and only if $A-\\beta\\Id$ is $(\\alpha-\\beta)$-monotone.\n\\item\\label{l:transfer_maxmono}\n$A$ is maximally $\\alpha$-monotone if and only if $A-\\beta\\Id$ is maximally $(\\alpha-\\beta)$-monotone. \n\\end{enumerate}\nConsequently, $A$ is (resp., maximally) $\\alpha$-monotone if and only if $A-\\alpha\\Id$ is (resp., maximally) monotone.\n\\end{lemma}","start":16520,"end":17101},{"raw_tex":"\\begin{proof}\n\\ref{l:transfer_mono}: We first have the equivalences\n\\begin{subequations}\n\\begin{align}\n(x,u)\\in \\gra A &\\iff (x,u-\\beta x)\\in \\gra(A-\\beta\\Id), \\\\\n(y,v)\\in \\gra A &\\iff (y,v-\\beta y)\\in \\gra(A-\\beta\\Id),\n\\end{align}\n\\end{subequations}\nand\n\\begin{equation}\n\\scal{x-y}{u-v}\\geq \\alpha\\|x-y\\|^2 \\iff \\scal{x-y}{(u-\\beta x)-(v-\\beta y)}\\geq (\\alpha-\\beta)\\|x-y\\|^2,\n\\end{equation}\nfrom which the conclusion follows.\n\n\\ref{l:transfer_maxmono}: Assume that $A$ is maximally $\\alpha$-monotone. By \\ref{l:transfer_mono}, $A-\\beta\\Id$ is $(\\alpha-\\beta)$-monotone. Now, suppose that $A-\\beta\\Id$ is not maximally $(\\alpha-\\beta)$-monotone. Then there must exist $B'\\colon X\\rightrightarrows X$ such that $B'$ is $(\\alpha-\\beta)$-monotone and $\\gra(A-\\beta\\Id)\\subsetneq \\gra B'$. It follows that $B :=B'+\\beta\\Id$ is $\\alpha$-monotone due to \\ref{l:transfer_mono} and that $\\gra A\\subsetneq \\gra B$, which contradict the maximal $\\alpha$-monotonicity of $A$. We deduce that if $A$ is maximally $\\alpha$-monotone, then $A-\\beta\\Id$ is maximally $(\\alpha-\\beta)$-monotone. This also implies that if $A-\\beta\\Id$ is maximally $(\\alpha-\\beta)$-monotone, then $A =(A-\\beta\\Id)+\\beta\\Id$ is maximally $\\alpha$-monotone, and we are done. \n\\end{proof}","start":17102,"end":18352},{"raw_tex":"\\begin{lemma}[resolvents of $\\alpha$-monotone operators]\n\\label{l:resol}\nLet $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{l:resol_gen}\nFor all $(x,a), (y,b)\\in \\gra J_{\\gamma A}$,\n\\begin{subequations}\n\\begin{align}\n\\label{e:resol_gen}\n\\scal{x-y}{a-b} &\\geq (1+\\gamma\\alpha)\\|a-b\\|^2 \\quad\\text{and}\\\\\n\\|x-y\\| &\\geq (1+\\gamma\\alpha)\\|a-b\\|.\n\\end{align} \n\\end{subequations}\n\\item\\label{l:resol_single}\nIf $J_{\\gamma A}$ is single-valued, then, for all $x,y\\in \\dom J_{\\gamma A}$,\n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq (1+\\gamma\\alpha)\\|J_{\\gamma A}x-J_{\\gamma A}y\\|^2,\n\\end{equation}\ni.e., $J_{\\gamma A}$ is $(1+\\gamma\\alpha)$-cocoercive.\n\\end{enumerate}\n\\end{lemma}","start":18354,"end":19129},{"raw_tex":"\\begin{proof}\n\\ref{l:resol_gen}: Let $(x,a), (y,b)\\in \\gra J_{\\gamma A}$. Then $x\\in (\\Id+\\gamma A)a$, $y\\in (\\Id+\\gamma A)b$, and so $x =a+\\gamma u$, $y =b+\\gamma v$ for some $u\\in Aa$, $v\\in Ab$. We derive from the $\\alpha$-monotonicity of $A$ that \n\\begin{subequations}\n\\begin{align}\n\\scal{x-y}{a-b}\n&=\\scal{(a+\\gamma u)-(b+\\gamma v)}{a-b}\\\\\n&=\\|a-b\\|^2+\\gamma\\scal{a-b}{u-v}\\\\\n&\\geq \\|a-b\\|^2+\\gamma\\alpha\\|a-b\\|^2\\\\\n&=(1+\\gamma\\alpha)\\|a-b\\|^2.\n\\end{align}\n\\end{subequations}\nNow, by the Cauchy--Schwarz inequality,\n\\begin{equation}\n\\|x-y\\|\\|a-b\\|\\geq \\scal{x-y}{a-b}\\geq (1+\\gamma\\alpha)\\|a-b\\|^2.\n\\end{equation}\nwhich gives $\\|x-y\\|\\geq (1+\\gamma\\alpha)\\|a-b\\|$ while noting that this is trivial when $a =b$. \n\n\\ref{l:resol_single}: This is a direct consequence of \\ref{l:resol_gen}.\n\\end{proof}","start":19130,"end":19932},{"raw_tex":"\\begin{proposition}[single-valuedness and full domain]\n\\label{p:resol}\nLet $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$ such that $1+\\gamma\\alpha >0$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{p:resol_single} \n$J_{\\gamma A}$ is single-valued.\n\\item\\label{p:resol_dom} \n$\\dom J_{\\gamma A} =X$ if and only if $A$ is maximally $\\alpha$-monotone.\n\\end{enumerate}\n\\end{proposition}","start":19935,"end":20356},{"raw_tex":"\\begin{proof}\n\\ref{p:resol_single}: This follows from Lemma~\\ref{l:resol}\\ref{l:resol_gen}.\n\n\\ref{p:resol_dom}: By Lemma~\\ref{l:transfer}\\ref{l:transfer_mono}, $A':=A-\\alpha\\Id$ is monotone.\nNoting that $(\\beta T)^{-1} =T^{-1}\\circ \\frac{1}{\\beta}\\Id$ for any operator $T$ and any $\\beta\\in \\RR\\smallsetminus\\{0\\}$, we have\n\\begin{subequations}\n\\begin{align}\nJ_{\\gamma A}&=(\\Id+\\gamma A)^{-1}=\\big((1+\\gamma\\alpha)\\Id+\\gamma(A-\\alpha\\Id)\\big)^{-1}\\\\\n&=\\left(\\Id+\\frac{\\gamma}{1+\\gamma\\alpha} A'\\right)^{-1} \\circ \\left(\\frac{1}{1+\\gamma\\alpha}\\Id\\right)\\\\\n&=J_{\\frac{\\gamma}{1+\\gamma\\alpha}A'}\\circ \\left(\\frac{1}{1+\\gamma\\alpha}\\Id\\right).\n\\end{align}\n\\end{subequations}\nIt follows that \n\\begin{subequations}\n\\begin{align}\n\\dom J_{\\gamma A} =X & \\iff \\dom J_{\\frac{\\gamma}{1+\\gamma\\alpha}A'} =X &&\\\\\n& \\iff \\text{$A'$ is maximally monotone} &&\\hspace*{-.8in}\\text{(by Fact~\\ref{f:Minty})} \\\\\n& \\iff \\text{$A$ is maximally $\\alpha$-monotone} &&\\hspace*{-.8in}\\text{(by Lemma~\\ref{l:transfer}\\ref{l:transfer_maxmono})}.\n\\end{align}\n\\end{subequations}\nThe proof is complete.\n\\end{proof}","start":20357,"end":21441},{"raw_tex":"\\begin{proposition}[maximal $\\alpha$-monotonicity]\n\\label{p:maxalpha}\nThe following statements hold: \n\\begin{enumerate}\n\\item\\label{p:maxalpha_mono} \nLet $A\\colon X\\rightrightarrows X$ and $\\alpha\\in \\RP$. Then $A$ is maximally $\\alpha$-monotone if and only if $A$ is $\\alpha$-monotone and maximally monotone.\n\\item\\label{p:maxalpha_cont} \nLet $A\\colon X\\to X$ and $\\alpha\\in \\RR$. Then $A$ is maximally $\\alpha$-monotone if $A$ is $\\alpha$-monotone and continuous with full domain.\n\\end{enumerate}\n\\end{proposition}","start":21510,"end":22026},{"raw_tex":"\\begin{proof}\n\\ref{p:maxalpha_mono}: Since $\\alpha\\geq 0$, it follows from Fact~\\ref{f:Minty} that $A$ is $\\alpha$-monotone and maximally monotone if and only if $A$ is $\\alpha$-monotone and $\\dom J_A =X$, which, by Proposition~\\ref{p:resol}\\ref{p:resol_dom}, happen if and only if $A$ is maximally $\\alpha$-monotone.\n\n\\ref{p:maxalpha_cont}: Set $A' :=A-\\alpha\\Id$. Then $A'$ is monotone (due to Lemma~\\ref{l:transfer}\\ref{l:transfer_mono}) and continuous with full domain. By \\cite[Corollary~20.28]{BC17}, $A'$ is maximally monotone, and by Lemma~\\ref{l:transfer}\\ref{l:transfer_maxmono}, $A$ is maximally $\\alpha$-monotone. \n\\end{proof}","start":22027,"end":22665},{"raw_tex":"\\begin{lemma}[resolvents of Lipschitz $\\alpha$-monotone operators]\n\\label{l:L-resol}\nLet $A\\colon X\\to X$ be Lipschitz continuous with constant $\\ell$ and let $\\gamma\\in \\RPP$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{l:L-resol_only} \nFor all $(x,a), (y,b)\\in \\gra J_{\\gamma A}$,\n\\begin{subequations}\n\\begin{align}\n\\label{e:expan}\n\\|a-b\\|&\\geq \\frac{1}{1+\\gamma\\ell}\\|x-y\\|,\\\\\n\\label{e:L-resol_only}\n\\scal{x-y}{a-b}&\\geq \\frac{1}{2}\\|x-y\\|^2 +\\frac{1}{2}(1-\\gamma^2\\ell^2)\\|a-b\\|^2,\n\\end{align}\n\\end{subequations}\nand if $\\gamma\\ell\\leq 1$, then\n\\begin{equation}\\label{e:L-resol_only1}\n\\scal{x-y}{a-b}\\geq \\frac{1}{1+\\gamma\\ell}\\|x-y\\|^2.\n\\end{equation}\n\\item\\label{l:L-resol_mono}\nIf $A$ is $\\alpha$-monotone with $1+\\gamma\\alpha >0$, then, for all $x, y\\in \\dom J_{\\gamma A}$, \n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq (1+\\gamma\\alpha)\\alpha_J\\|x-y\\|^2,\n\\end{equation}\nwhere \n\\begin{subequations}\n\\label{e:alpha_J}\n\\begin{equation}\n\\alpha_J \n:=\\begin{cases}\n\\dfrac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2} &\\text{if~} \\gamma\\ell\\geq 1 \n,\\\\\n\\dfrac{1}{(1+\\gamma\\alpha)(1+\\gamma\\ell)} &\\text{if~} \\gamma\\ell\\leq 1;\n\\end{cases}\n\\end{equation}\nand if additionally $A$ satisfies \\eqref{e:alpha-mono} with equality, then \n\\begin{equation}\n\\alpha_J :=\\frac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}. \n\\end{equation}\n\\end{subequations}\n\\end{enumerate}\n\\end{lemma}","start":24136,"end":25513},{"raw_tex":"\\begin{proof}\n\\ref{l:L-resol_only}: Let $(x,a), (y,b)\\in \\gra J_{\\gamma A}$. Then $x =a+\\gamma Aa$ and $y =b+\\gamma Ab$. By Lipschitz continuity, $\\|Aa-Ab\\|\\leq \\ell\\|a-b\\|$. It follows that\n\\begin{equation}\n\\|x-y\\|\\leq \\|a-b\\| +\\gamma\\|Aa-Ab\\|\\leq (1+\\gamma\\ell)\\|a-b\\|\n\\end{equation}\nand that\n\\begin{subequations}\n\\label{e:L-resol}\n\\begin{align}\n2\\scal{x-y}{a-b} &=\\|x-y\\|^2 +\\|a-b\\|^2 -\\|(x-a)-(y-b)\\|^2\\\\\n&=\\|x-y\\|^2 +\\|a-b\\|^2 -\\gamma^2\\|Aa-Ab\\|^2\\\\\n&\\geq \\|x-y\\|^2 +(1-\\gamma^2\\ell^2)\\|a-b\\|^2.\n\\end{align}\n\\end{subequations}\nIf $\\gamma\\ell\\leq 1$, then combining the above inequalities yields\n\\begin{equation}\n2\\scal{x-y}{a-b}\\geq \\|x-y\\|^2 +\\frac{1-\\gamma^2\\ell^2}{(1+\\gamma\\ell)^2}\\|x-y\\|^2 =\\frac{2}{1+\\gamma\\ell}\\|x-y\\|^2,\n\\end{equation}\nand we get the claim.\n\n\\ref{l:L-resol_mono}: We first note that $J_{\\gamma A}$ is single-valued due to Proposition~\\ref{p:resol}\\ref{p:resol_single}. Then \\eqref{e:L-resol} reads as\n\\begin{equation}\n\\label{e:L-resol'}\n2\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\|x-y\\|^2 +(1-\\gamma^2\\ell^2)\\|J_{\\gamma A}x-J_{\\gamma A}y\\|^2.\n\\end{equation}\nWe claim that if $\\gamma\\ell\\geq 1$ or $A$ satisfies \\eqref{e:alpha-mono} with equality, then \n\\begin{equation}\n\\label{e:L-resol''}\n2\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\|x-y\\|^2 +\\frac{1-\\gamma^2\\ell^2}{1+\\gamma\\alpha}\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}.\n\\end{equation} \nIndeed, the former case implies $1-\\gamma^2\\ell^2\\leq 0$ and, by combining \\eqref{e:L-resol'} with Lemma~\\ref{l:resol}\\ref{l:resol_single} and noting that $1+\\gamma\\alpha >0$, we get \\eqref{e:L-resol''}.  \nIn the latter case, Lemma~\\ref{l:resol}\\ref{l:resol_single} reduces to\n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y} =(1+\\gamma\\alpha)\\|J_{\\gamma A}x-J_{\\gamma A}y\\|^2.\n\\end{equation} \nSubstituting this into \\eqref{e:L-resol'}, we also obtain \\eqref{e:L-resol''}. \n\nNow, in view of Remark~\\ref{r:Lip_mono}, $1+2\\gamma\\alpha+\\gamma^2\\ell^2\\geq 1+2\\gamma\\alpha+\\gamma^2\\alpha^2 =(1+\\gamma\\alpha)^2 >0$. It thus follows from \\eqref{e:L-resol''} that \n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\frac{1+\\gamma\\alpha}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}\\|x-y\\|^2.\n\\end{equation}\nFinally, if $\\gamma\\ell\\leq 1$, then, by \\ref{l:L-resol_only}, \n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\frac{1}{1+\\gamma\\ell}\\|x-y\\|^2,\n\\end{equation}\nand the conclusion follows.\n\\end{proof}","start":25514,"end":27906},{"raw_tex":"\\begin{proposition}[linear combinations of resolvents and the identity]\n\\label{p:cresol}\nLet $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$. Set $J:=J_{\\gamma A}$ and define $Q :=\\nu\\Id+\\lambda J$ with $\\nu, \\lambda\\in \\RR$. \n\\begin{enumerate}\n\\item\\label{p:cresol_mono} \nSuppose that $J$ is single-valued and $\\nu\\lambda\\leq 0$. Then, for all $x,y\\in \\dom J$, \n\\begin{equation}\n\\|Qx-Qy\\|^2\\leq \\nu^2\\|x-y\\|^2 +\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\|Jx-Jy\\|^2.\n\\end{equation}\n\\item\\label{p:cresol_Lip}\nSuppose that $A$ is Lipschitz continuous with constant $\\ell$, $1+\\gamma\\alpha >0$, and $\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\leq 0$. Then $Q$ is Lipschitz continuous with constant    \n\\begin{equation}\n\\label{e:LipQ}\n\\rho :=\\sqrt{\\nu^2+\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\alpha_J},\n\\end{equation}\nwhere $\\alpha_J$ is defined as \\eqref{e:alpha_J}.\nIf additionally $\\lambda\\left(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\right)\\geq 0$ whenever $\\gamma\\ell <1$, then the Lipschitz constant \\eqref{e:LipQ} can be improved to\n\\begin{equation}\n\\overline{\\rho} :=\\sqrt{\\nu^2+\\frac{\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}}\\leq \\rho.\n\\end{equation}\n\\end{enumerate}\n\\end{proposition}","start":28997,"end":30263},{"raw_tex":"\\begin{proof}\nLet $x,y\\in \\dom J$. By the definition of $Q$,\n\\begin{subequations}\n\\label{e:identityQ}\n\\begin{align}\n\\|Qx-Qy\\|^2\n&=\\|\\nu(x-y)+\\lambda(Jx-Jy)\\|^2\\\\\n&=\\nu^2\\|x-y\\|^2 +2\\nu\\lambda\\scal{x-y}{Jx-Jy} +\\lambda^2\\|Jx-Jy\\|^2.\n\\end{align}\n\\end{subequations}\n\n\\ref{p:cresol_mono}: Since $\\nu\\lambda\\leq 0$, combining \\eqref{e:identityQ} with Lemma~\\ref{l:resol}\\ref{l:resol_single} yields\n\\begin{subequations}\n\\begin{align}\n\\|Qx-Qy\\|^2\n&\\leq \\nu^2\\|x-y\\|^2 +2\\nu\\lambda(1+\\gamma\\alpha)\\|Jx-Jy\\|^2 +\\lambda^2\\|Jx-Jy\\|^2\\\\\n&=\\nu^2\\|x-y\\|^2 +\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\|Jx-Jy\\|^2.\n\\end{align}\n\\end{subequations}\n\n\\ref{p:cresol_Lip}: First, according to Proposition~\\ref{p:resol}\\ref{p:resol_single}, $J$ is single-valued, and so is $Q$. Next, using \\eqref{e:identityQ}, Lemma~\\ref{l:resol}\\ref{l:resol_single}, and Lemma~\\ref{l:L-resol}\\ref{l:L-resol_mono} and noting that $\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\leq 0$, we have\n\\begin{subequations}\n\\label{e:cresol_Lip}\n\\begin{align}\n\\|Qx-Qy\\|^2 \n&\\leq \\nu^2\\|x-y\\|^2 +2\\nu\\lambda\\scal{x-y}{Jx-Jy} +\\frac{\\lambda^2}{1+\\gamma\\alpha}\\scal{x-y}{Jx-Jy}\\\\ \n&= \\nu^2\\|x-y\\|^2 +\\frac{\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)}{1+\\gamma\\alpha}\\scal{x-y}{Jx-Jy}\\\\  \n&\\leq \\nu^2\\|x-y\\|^2 +\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\alpha_J\\|x-y\\|^2 =\\rho^2\\|x-y\\|^2, \n\\end{align}\n\\end{subequations}\nwhich implies that $Q$ is Lipschitz continuous with constant $\\rho$.\n\nFor the last statement, we show that $\\alpha_J$ in formula \\eqref{e:LipQ} can be replaced by \n\\begin{equation}\n\\overline{\\alpha}_J :=\\frac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}.\n\\end{equation}\nIf $\\scal{x-y}{Jx-Jy}\\geq (1+\\gamma\\alpha)\\overline{\\alpha}_J\\|x-y\\|^2$, then \\eqref{e:cresol_Lip} also holds with $\\alpha_J$ replaced by $\\overline{\\alpha}_J$. Now, assume that $\\scal{x-y}{Jx-Jy} <(1+\\gamma\\alpha)\\overline{\\alpha}_J\\|x-y\\|^2$. By Lemma~\\ref{l:L-resol}\\ref{l:L-resol_mono}, we must have $\\gamma\\ell <1$, and then, by assumption, $\\lambda\\left(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\right)\\geq 0$. It now follows from \\eqref{e:identityQ} and \\eqref{e:L-resol_only} that\n\\begin{subequations}\n\\begin{align}\n\\|Qx-Qy\\|^2\n&\\leq \\nu^2\\|x-y\\|^2 +2\\nu\\lambda\\scal{x-y}{Jx-Jy} \\notag\\\\ \n&\\qquad +\\frac{\\lambda^2}{1-\\gamma^2\\ell^2}(2\\scal{x-y}{Jx-Jy} -\\|x-y\\|^2)\\\\\n&=\\Big(\\nu^2-\\frac{\\lambda^2}{1-\\gamma^2\\ell^2}\\Big)\\|x-y\\|^2 +2\\lambda\\Big(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\Big)\\scal{x-y}{Jx-Jy}\\\\\n&\\leq \\Big(\\nu^2-\\frac{\\lambda^2}{1-\\gamma^2\\ell^2}\\Big)\\|x-y\\|^2 +2\\lambda\\Big(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\Big)(1+\\gamma\\alpha)\\overline{\\alpha}_J\\|x-y\\|^2\\\\\n&=\\Big(\\nu^2+\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\overline{\\alpha}_J\\Big)\\|x-y\\|^2 =\\overline{\\rho}^2\\|x-y\\|^2.\n\\end{align}\n\\end{subequations}\nFinally, we will prove $\\overline{\\alpha}_J\\geq \\alpha_J$, which implies $\\overline{\\rho}\\leq \\rho$, i.e., the Lipschitz constant is indeed improved. From the definition of $\\alpha_J$, it suffices to consider the case in which $\\alpha_J =1/((1+\\gamma\\alpha)(1+\\gamma\\ell))$. Then $\\gamma\\ell\\leq 1$. \nSince $|\\alpha|\\leq \\ell$ (see Remark~\\ref{r:Lip_mono}), it holds that\n\\begin{subequations}\n\\begin{align}\n0 <(1+\\gamma\\alpha)^2\\leq 1+2\\gamma\\alpha+\\gamma^2\\ell^2\n&\\leq \n1+2\\gamma\\alpha+\\gamma^2\\ell^2\n+(\\gamma\\ell-\\gamma\\alpha)-\\gamma\\ell(\\gamma\\ell-\\gamma\\alpha)\\\\\n&=1+\\gamma\\alpha+\\gamma\\ell+(\\gamma\\alpha)(\\gamma\\ell)\\\\\n&=(1+\\gamma\\alpha)(1+\\gamma\\ell),\n\\end{align}\n\\end{subequations}\nand so \n\\begin{equation}\n\\overline{\\alpha}_J =\\frac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}\\geq \\alpha_J =\\frac{1}{(1+\\gamma\\alpha)(1+\\gamma\\ell)}.\n\\end{equation}\nThe proof is complete.\n\\end{proof}","start":30264,"end":33905},{"raw_tex":"\\begin{corollary}[relaxed resolvents of $\\alpha$-monotone operators]\n\\label{c:rresol}\nLet $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$. Suppose that $J:=J_{\\gamma A}$ is single-valued and define $R:=(1-\\lambda)\\Id +\\lambda J$ with $\\lambda\\in \\RP$, and $Q:=R+\\varepsilon\\Id$ with $\\varepsilon\\in \\RR$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{c:rresol_relax} \nIf $\\lambda\\geq 1$, then, for all $x, y\\in \\dom J$, \n\\begin{equation}\n\\|Rx-Ry\\|^2\\leq (\\lambda-1)^2\\|x-y\\|^2 -\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda\\big)\\|Jx-Jy\\|^2.\n\\end{equation}\n\\item\\label{c:rresol_perturb} \nIf $\\varepsilon\\leq \\lambda-1$, then, for all $x, y\\in \\dom J$, \n\\begin{align}\n\\|Qx-Qy\\|^2 &\\leq (\\lambda-1-\\varepsilon)^2\\|x-y\\|^2 \\notag\\\\ \n&\\quad -\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda-2\\varepsilon(1+\\gamma\\alpha)\\big)\\|Jx-Jy\\|^2.\n\\end{align}\nConsequently, if additionally $(\\lambda-1)(2+2\\gamma\\alpha)-\\lambda-2\\varepsilon(1+\\gamma\\alpha)\\geq 0$, then $Q$ is Lipschitz continuous with constant $(\\lambda-1-\\varepsilon)$.\n\\end{enumerate} \n\\end{corollary}","start":35151,"end":36247},{"raw_tex":"\\begin{proof}\nBecause \\ref{c:rresol_relax} is a consequence of \\ref{c:rresol_perturb} with $\\varepsilon =0$, it suffices to prove only the latter. To this end, noting that $Q =R+\\varepsilon\\Id =(1-\\lambda+\\varepsilon)\\Id+\\lambda J$ and using Proposition~\\ref{p:cresol}\\ref{p:cresol_mono} with $\\nu =1-\\lambda+\\varepsilon \\leq 0$, we have that\n\\begin{subequations}\n\\begin{align}\n\\|Qx-Qy\\|^2&\\leq (1-\\lambda+\\varepsilon)^2\\|x-y\\|^2 +\\lambda\\left(2(1-\\lambda+\\varepsilon)(1+\\gamma\\alpha)+\\lambda\\right)\\|Jx-Jy\\|^2\\\\\n&=(\\lambda-1-\\varepsilon)^2\\|x-y\\|^2 -\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda-2\\varepsilon(1+\\gamma\\alpha)\\big)\\|Jx-Jy\\|^2\n\\end{align}\n\\end{subequations}\nwhich proves \\ref{c:rresol_perturb}. \n\\end{proof}","start":36248,"end":36973},{"raw_tex":"\\begin{corollary}[relaxed resolvents of Lipschitz $\\alpha$-monotone operators]\n\\label{c:L-rresol}\nLet $A\\colon X\\to X$ be $\\alpha$-monotone and Lipschitz continuous with constant $\\ell$. Also let $\\gamma\\in\\RPP$ and $\\lambda\\in\\RPP$ be such that\n\\begin{equation}\\label{e:cL-rresol-1}\n1+\\gamma\\alpha>0\n\\ \\text{ and }\\ \n\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\geq 0.\n\\end{equation}\nDefine $J:=J_{\\gamma A}$, $R:=(1-\\lambda)\\Id +\\lambda J$, and $Q :=\\Id-\\varepsilon R$ with $\\varepsilon\\in \\RP$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{c:L-rresol_relax} \n$R$ is Lipschitz continuous with constant \n\\begin{equation}\n\\sqrt{(\\lambda-1)^2-\\frac{\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda\\big)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}}.\n\\end{equation} \n\\item\\label{c:L-rresol_perturb} \n$Q$ is Lipschitz continuous with constant \n\\begin{equation}\n\\sqrt{(1+\\varepsilon(\\lambda-1))^2 -\\varepsilon\\lambda\\Big[2(1+\\gamma\\alpha) +\\varepsilon\\big(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\big)\\Big]\\alpha_J},\n\\end{equation}\nwhere $\\alpha_J$ is defined as \\eqref{e:alpha_J}.\n\\end{enumerate}\n\\end{corollary}","start":36977,"end":38084},{"raw_tex":"\\begin{proof}\n\\ref{c:L-rresol_relax}: We observe that $R=\\nu\\Id+\\lambda J$ with $\\nu :=1-\\lambda$, that\n\\begin{equation}\n\\lambda(2\\nu(1+\\gamma\\alpha)+\\lambda) =-\\lambda\\Big(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\Big)\\leq 0,\n\\end{equation} \nand that, whenever $\\gamma\\ell <1$, \n\\begin{equation}\n\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}=\n(1-\\lambda)+\\frac{\\lambda}{1-\\gamma^2\\ell^2} >1-\\lambda+\\lambda =1 >0.\n\\end{equation}\nApplying Proposition~\\ref{p:cresol}\\ref{p:cresol_Lip} to $R=\\nu\\Id+\\lambda\\Id$ implies that $R$ is Lipschitz continuous with constant\n\\begin{equation}\n\\sqrt{(1-\\lambda)^2+\\frac{\\lambda\\big(2(1-\\lambda)(1+\\gamma\\alpha)+\\lambda\\big)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}},\n\\end{equation} \nwhich gives the claim.\n\n\\ref{c:L-rresol_perturb}: Using the first part of Proposition~\\ref{p:cresol}\\ref{p:cresol_Lip} and writing $Q =\\widetilde{\\nu}\\Id+\\widetilde{\\lambda}J$ with $\\widetilde{\\nu} :=1+\\varepsilon(\\lambda-1)$ and $\\widetilde{\\lambda} :=-\\varepsilon\\lambda$, it suffices to check that\n\\begin{equation}\\label{e:cL-rresol-2}\n\\widetilde{\\lambda}(2\\widetilde{\\nu}(1+\\gamma\\alpha)+\\widetilde{\\lambda})=(-\\varepsilon\\lambda)\\big( 2(1+\\varepsilon(\\lambda-1))(1+\\gamma\\alpha)+(-\\varepsilon\\lambda) \\big)\\leq 0.\n\\end{equation}\nIndeed, we have that $\\varepsilon\\geq 0$, $\\lambda >0$, and  \n\\begin{subequations}\n\\begin{align}\n2(1+\\varepsilon(\\lambda-1))(1+\\gamma\\alpha) -\\varepsilon\\lambda \n&=2(1+\\gamma\\alpha) +\\varepsilon\\big(2(\\lambda-1)(1+\\gamma\\alpha)-\\lambda\\big)\\\\\n&=2(1+\\gamma\\alpha) +\\varepsilon(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)) >0\n\\end{align}\n\\end{subequations}\nby \\eqref{e:cL-rresol-1}. So \\eqref{e:cL-rresol-2} holds and the conclusion follows.\n\\end{proof}","start":38085,"end":39773},{"raw_tex":"\\begin{lemma}[single-valuedness and full domain of adaptive DR operator]\n\\label{l:domT}\nSuppose that $A$ and $B$ are maximally $\\alpha$- and $\\beta$-monotone with $\\alpha+\\beta\\geq 0$. \nThen there exists $(\\gamma,\\delta,\\lambda,\\mu)\\in \\RPP^2\\times \\left]1,+\\infty\\right[^2$ such that\n\\begin{subequations}\n\\label{e:parameters}\n\\begin{align}\n&1+2\\gamma\\alpha >0,\n\\label{e:parameters_a}\\\\\n&\\mu\\in \\left[2-2\\gamma\\beta, 2+2\\gamma\\alpha\\right],\n\\label{e:parameters_b} \\\\\n&(\\lambda-1)(\\mu-1) =1, \\text{~and~} \\delta =(\\lambda-1)\\gamma. \\label{e:parameters_c}\n\\end{align}\n\\end{subequations}\nMoreover, \\eqref{e:parameters} implies that $\\min\\{1+\\gamma\\alpha, 1+\\delta\\beta\\} >0$ and that $J_1$, $J_2$, and $T$ are single-valued and have full domain. \n\\end{lemma}","start":53912,"end":54667},{"raw_tex":"\\begin{proof}\nTo show the existence, we first take $\\gamma>0$ such that $1/\\gamma >-2\\alpha$. Then $1+2\\gamma\\alpha >0$ and $2+2\\gamma\\alpha =1+(1+2\\gamma\\alpha) >1$.\nUsing $\\alpha+\\beta\\geq 0$, we derive that\n\\begin{equation}\n2+2\\gamma\\alpha =2\\gamma(\\alpha+\\beta) +(2-2\\gamma\\beta)\\geq 2-2\\gamma\\beta.\n\\end{equation}\nHence, we can always choose $\\mu >1$ satisfying \\eqref{e:parameters_b}. Next, with such $\\mu$, we define $\\lambda :=\\mu/(\\mu-1) =1+1/(\\mu-1) >1$ and $\\delta :=(\\lambda-1)\\gamma$. Then \\eqref{e:parameters_c} is clearly satisfied. \n\nNow, take any $(\\gamma,\\delta,\\lambda,\\mu)$ satisfying \\eqref{e:parameters}. We have\n\\begin{equation}\n1+\\delta\\beta =(\\lambda-1)(\\mu-1)+(\\lambda-1)\\gamma\\beta =\\frac{1}{2}(\\lambda-1)(\\mu + \\mu - (2-2\\gamma\\beta)) >0.\n\\end{equation}\nThus, $\\min\\{1+\\gamma\\alpha, 1+\\delta\\beta\\} >0$. The remaining conclusion follows from Proposition~\\ref{p:resol}. \n\\end{proof}","start":54668,"end":55577},{"raw_tex":"\\begin{proof}\nIn view of Remark~\\ref{r:Lip_mono}, assumption \\ref{t:linear_reduce} implies assumption \\ref{t:linear_full} because if $A$ is Lipschitz continuous with constant $\\ell$, then $A$ is also $\\alpha$-monotone with $\\alpha :=-\\ell$. It thus suffices to assume \\ref{t:linear_full}.  \nFirst, Proposition~\\ref{p:maxalpha}\\ref{p:maxalpha_cont} implies that $A$ is maximally $\\alpha$-monotone. Next, we learn from Lemma~\\ref{l:domT} that \n\\begin{equation}\n\\label{e:domT'}\n\\min\\{1+\\gamma\\alpha,1+\\delta\\beta\\} >0\n\\end{equation}\nand that all operators $J_1,J_2$, and $T$ are single-valued and have full domain. \n\nBy the choice of $\\mu$, it holds that $0 <\\mu-1 \\leq 1+2\\gamma\\alpha$, and so\n\\begin{equation}\n\\lambda =1+\\frac{1}{\\mu-1}\\geq 1+\\frac{1}{1+2\\gamma\\alpha} =\\frac{2(1+\\gamma\\alpha)}{1+2\\gamma\\alpha},\n\\end{equation} \nwhich yields\n\\begin{equation}\n\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\geq 0.\n\\end{equation}\nFrom $\\mu\\geq 2-2\\gamma\\beta$, we have that $\\varepsilon\\geq 0$ and $\\varepsilon_1\\geq 0$. It follows that $\\varphi\\geq 0$ and that\n\\begin{equation}\\label{e:180823a}\n\\varphi =0 \\iff \\varepsilon_1 =0 \\iff \\varepsilon =0 \\iff \\mu =2-2\\gamma\\beta.\n\\end{equation}\nDefine $Q_1 :=\\Id -\\varepsilon_1 R_1$. Using Corollary~\\ref{c:L-rresol} and noting that $\\lambda =\\mu(\\lambda-1)$, we derive that $R_1$ is Lipschitz continuous with constant  \n\\begin{equation}\n\\sqrt{(\\lambda-1)^2-\\frac{\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda\\big)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}} =(\\lambda-1)\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}},\n\\end{equation} \nand that $Q_1$ is Lipschitz continuous with constant \n\\begin{equation}\n\\label{e:rho1}\n\\rho_1 :=\\sqrt{(1+\\varepsilon_1(\\lambda-1))^2 -\\varphi\\alpha_J}\\leq 1+\\varepsilon_1(\\lambda-1),\n\\end{equation}\nwhere $\\alpha_J$ is defined as in \\eqref{e:alpha_J}. It follows from \\eqref{e:180823a} that the inequality is strict whenever $\\mu >2-2\\gamma\\beta$.\n\nNext, define $Q_2 :=R_2+\\varepsilon\\Id$. \nSince $\\gamma =(\\mu-1)\\delta$, we note that\n\\begin{equation}\n\\varepsilon =\\frac{\\mu-(2-2\\gamma\\beta)}{2(1+\\delta\\beta)} =\\frac{(\\mu-1)(2+2\\delta\\beta)-\\mu}{2(1+\\delta\\beta)} <\\mu-1,\n\\end{equation}\nwhich also gives\n\\begin{equation}\n(\\mu-1)(2+2\\delta\\beta)-\\mu-2\\varepsilon(1+\\delta\\beta) =0.\n\\end{equation}\nBy Corollary~\\ref{c:rresol}\\ref{c:rresol_perturb}, $Q_2$ is Lipschitz continuous with constant $(\\mu-1-\\varepsilon)$. Combining with the Lipschitz continuity of $R_1$ and noting that $(\\mu-1)(\\lambda-1) =1$, we have that $Q_2R_1$ is Lipschitz continuous with constant\n\\begin{subequations}\n\\label{e:rho2}\n\\begin{align}\n\\rho_2\n&:=(\\mu-1-\\varepsilon)(\\lambda-1)\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}}\\\\\n&\\phantom{:}=(1-\\varepsilon(\\lambda-1))\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}}\\\\\n&\\phantom{:}\\leq 1-\\varepsilon(\\lambda-1),\n\\end{align}\n\\end{subequations} \nwhere the inequality is strict whenever $\\mu <2+2\\gamma\\alpha$.  \n\nNow, we express\n\\begin{subequations}\n\\begin{align}\nT&=(1-\\kappa)\\Id -(1-\\kappa)\\varepsilon_1R_1 + \\kappa R_2R_1+\\kappa\\varepsilon R_1\\\\\n&=(1-\\kappa)(\\Id-\\varepsilon_1R_1)+\\kappa(R_2+\\varepsilon\\Id)R_1\\\\\n&=(1-\\kappa)Q_1+\\kappa Q_2R_1.\n\\end{align}\n\\end{subequations}\nWe note from $\\alpha+\\beta >0$ that $2+2\\gamma\\alpha >2-2\\gamma\\beta$, so at least one of two inequalities in \\eqref{e:rho1} and \\eqref{e:rho2} is strict.\nTherefore, $T$ is Lipschitz continuous with constant\n\\begin{equation}\n\\rho :=(1-\\kappa)\\rho_1+\\kappa \\rho_2 <(1-\\kappa)(1+\\varepsilon_1(\\lambda-1))+\\kappa(1-\\varepsilon(\\lambda-1))=1,\n\\end{equation}\nwhich completes the proof.\n\\end{proof}","start":63950,"end":67583}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"实希尔伯特空间","source_units":[{"raw_tex":"Throughout this work, $X$ is a real Hilbert space with inner product $\\scal{\\cdot}{\\cdot}$ and induced norm $\\|\\cdot\\|$. \nThe set of nonnegative integers is denoted by $\\NN$, the set of real numbers by $\\RR$, the set of nonnegative real numbers by $\\RP := \\menge{x \\in \\RR}{x \\geq 0}$, and the set of the positive real numbers by $\\RPP := \\menge{x \\in \\RR}{x >0}$.\nWe use the notation $A\\colon X\\rightrightarrows X$ to indicate that $A$ is a set-valued operator on $X$ and the notation $A\\colon X\\to X$ to indicate that $A$ is a single-valued operator on $X$. \n\n","start":12234,"end":12796}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"算子集合记号","source_units":[{"raw_tex":"Let $A$ be an operator on $X$. The \\emph{domain} of $A$ is $\\dom A :=\\menge{x\\in X}{Ax\\neq \\varnothing}$, the \\emph{graph} of $A$ is $\\gra A :=\\menge{(x,u)\\in X\\times X}{u\\in Ax}$, and the set of \\emph{fixed points} of $A$ is $\\Fix A :=\\menge{x\\in X}{x\\in Ax}$. The \\emph{inverse} of $A$, denoted by $A^{-1}$, is the operator with graph $\\gra A^{-1} :=\\menge{(u,x)\\in X\\times X}{u\\in Ax}$.\n","start":12796,"end":13186}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"利普希茨连续性","source_units":[{"raw_tex":"We say that $A$ is \\emph{Lipschitz continuous} with constant $\\ell\\in \\RP$ if it is single-valued and\n\\begin{equation}\n\\forall x, y\\in \\dom A,\\quad \\|Ax-Ay\\|\\leq \\ell\\|x-y\\|.\n\\end{equation}\nThe operator $A$ is \\emph{nonexpansive} if it is Lipschitz continuous with constant $1$, i.e.,\n\\begin{equation}\n\\forall x, y\\in \\dom A,\\quad \\|Ax-Ay\\|\\leq \\|x-y\\|.\n\\end{equation}\n","start":13186,"end":13555}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"单调与极大单调","source_units":[{"raw_tex":"An operator $A\\colon X\\rightrightarrows X$ is said to be \\emph{monotone} if\n\\begin{equation}\n\\forall (x, u), (y, v)\\in \\gra A,\\quad \\scal{x-y}{u-v}\\geq 0,\n\\end{equation}\nand said to be \\emph{maximally monotone} if it is monotone and there exists no monotone operator $B\\colon X\\rightrightarrows X$ such that $\\gra B$ properly contains $\\gra A$. The \\emph{resolvent} of $A\\colon X\\rightrightarrows X$ is defined by\n","start":13555,"end":13969}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"预解式与松弛预解式","source_units":[{"raw_tex":"and said to be \\emph{maximally monotone} if it is monotone and there exists no monotone operator $B\\colon X\\rightrightarrows X$ such that $\\gra B$ properly contains $\\gra A$. The \\emph{resolvent} of $A\\colon X\\rightrightarrows X$ is defined by\n\\begin{equation}\nJ_A:= (\\Id+ A)^{-1},\n\\end{equation}\nwhere $\\Id$ is the identity operator. The \\emph{relaxed resolvent} of $A$ with parameter $\\lambda\\in\\RP$ is defined by\n\\begin{equation}\nJ_A^\\lambda:= (1-\\lambda)\\Id+ \\lambda J_A.\n\\end{equation}\n","start":13725,"end":14216}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["d:alpha_mono","e:alpha-mono"],"declared_external":false,"source_only":false,"title":"广义单调性","source_units":[{"raw_tex":"\\begin{definition}[$\\alpha$-monotonicity]\n\\label{d:alpha_mono}\nAn operator $A\\colon X\\rightrightarrows X$ is said to be \\emph{$\\alpha$-monotone} ($\\alpha\\in \\RR$) if\n\\begin{equation}\n\\label{e:alpha-mono}\n\\forall (x,u), (y,v)\\in \\gra A,\\quad \\scal{x-y}{u-v}\\geq \\alpha\\|x-y\\|^2.\n\\end{equation}\nThe constant $\\alpha$ is referred to as the \\emph{monotonicity constant}.\nWe also say that $A$ is \\emph{maximally $\\alpha$-monotone} if it is $\\alpha$-monotone and there is no $\\alpha$-monotone operator whose graph strictly contains $\\gra A$.  \n\\end{definition}\n\n","start":15504,"end":16060}]},{"id":"S007","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"强弱单调的统一","source_units":[{"raw_tex":"We note that $0$-monotonicity simply means monotonicity, that if $\\alpha>0$, then $\\alpha$-monotonicity is precisely the notion of \\emph{strong monotonicity} \\cite[Definition~22.1(iv)]{BC17}, and that if $\\alpha<0$, then $\\alpha$-monotonicity can be referred to as \\emph{weak monotonicity}.\n","start":16060,"end":16351}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","labels":["e:L-inequality","r:Lip_mono"],"declared_external":false,"source_only":false,"title":"内积的双侧界","source_units":[{"raw_tex":"\\begin{remark}[Lipschitz $\\alpha$-monotone operators]\n\\label{r:Lip_mono}\nSuppose that $A$ is Lipschitz continuous with constant $\\ell$. Then $A$ is single-valued and \n\\begin{equation}\n\\forall x,y\\in\\dom A,\\quad\n|\\scal{x-y}{Ax-Ay}|\\leq \\|x-y\\|\\cdot\\|Ax-Ay\\|\\leq \\ell\\|x-y\\|^2,\n\\end{equation}\nwhich yields\n\\begin{equation}\n\\label{e:L-inequality}\n\\forall x,y\\in\\dom A,\\quad -\\ell\\|x-y\\|^2\\leq \\scal{x-y}{Ax-Ay}\\leq \\ell\\|x-y\\|^2.\n\\end{equation}\n","start":22888,"end":23330}]},{"id":"S009","semantic_type":"proof","kind":"proof","labels":["e:L-inequality","r:Lip_mono"],"declared_external":false,"source_only":false,"title":"内积双侧界的行内证明","source_units":[{"raw_tex":"Suppose that $A$ is Lipschitz continuous with constant $\\ell$. Then $A$ is single-valued and \n\\begin{equation}\n\\forall x,y\\in\\dom A,\\quad\n|\\scal{x-y}{Ax-Ay}|\\leq \\|x-y\\|\\cdot\\|Ax-Ay\\|\\leq \\ell\\|x-y\\|^2,\n\\end{equation}\nwhich yields\n\\begin{equation}\n\\label{e:L-inequality}\n\\forall x,y\\in\\dom A,\\quad -\\ell\\|x-y\\|^2\\leq \\scal{x-y}{Ax-Ay}\\leq \\ell\\|x-y\\|^2.\n\\end{equation}\n","start":22961,"end":23330}]},{"id":"S010","semantic_type":"derived_fact","kind":"derived_fact","labels":["r:Lip_mono"],"declared_external":false,"source_only":false,"title":"自动弱单调性","source_units":[{"raw_tex":"We immediately deduce that $A$ is $(-\\ell)$-monotone. Now suppose, in addition, that $A$ is $\\alpha$-monotone. On the one hand, we can always assume without loss of generality that $\\alpha\\geq-\\ell$. On the other hand, it follows from the $\\alpha$-monotonicity and \\eqref{e:L-inequality} that $\\alpha\\leq \\ell$ as soon as $\\dom A$ has more than one element. Therefore, unless otherwise stated, whenever $A$ is both $\\alpha$-monotone and Lipschitz continuous with constant $\\ell$, we assume that $|\\alpha|\\leq\\ell$.\n","start":23330,"end":23845}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","labels":["r:Lip_mono"],"declared_external":false,"source_only":false,"title":"单调常数的默认约定","source_units":[{"raw_tex":"We immediately deduce that $A$ is $(-\\ell)$-monotone. Now suppose, in addition, that $A$ is $\\alpha$-monotone. On the one hand, we can always assume without loss of generality that $\\alpha\\geq-\\ell$. On the other hand, it follows from the $\\alpha$-monotonicity and \\eqref{e:L-inequality} that $\\alpha\\leq \\ell$ as soon as $\\dom A$ has more than one element. Therefore, unless otherwise stated, whenever $A$ is both $\\alpha$-monotone and Lipschitz continuous with constant $\\ell$, we assume that $|\\alpha|\\leq\\ell$.\n\\end{remark}","start":23330,"end":23857}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"全节算子与参数","source_units":[{"raw_tex":"Throughout this section, $A,B\\colon X\\rightrightarrows X$, $(\\gamma,\\delta,\\lambda,\\mu)\\in \\RPP^4$, and $\\kappa\\in \\left]0,1\\right[$. We define\n\\begin{subequations}\\label{e:JRT}\n","start":39838,"end":40016}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["e:aDR"],"declared_external":false,"source_only":false,"title":"自适应 DR 算子","source_units":[{"raw_tex":"\\begin{align}\nJ_1&:=J_{\\gamma A}=(\\Id+\\gamma A)^{-1}, \n&R_1&:=J_{\\gamma A}^\\lambda=(1-\\lambda)\\Id+\\lambda J_1,\\\\\nJ_2&:=J_{\\delta B}=(\\Id+\\delta B)^{-1},\n&R_2&:=J_{\\delta B}^\\mu=(1-\\mu)\\Id+\\mu J_2\n\\end{align}\nand consider the \\emph{adaptive DR operator} defined by\n\\begin{align}\\label{e:aDR}\nT:= (1-\\kappa)\\Id+ \\kappa R_2R_1.\n\\end{align}\n\\end{subequations}  \n","start":40016,"end":40374}]},{"id":"S014","semantic_type":"problem_setting","kind":"problem_setting","labels":["e:sumprob"],"declared_external":false,"source_only":false,"title":"算子和的零点问题","source_units":[{"raw_tex":"The adaptive DR operator is indeed motivated by the problem of finding a zero of the sum of two operators, that is,\n\\begin{equation}\\label{e:sumprob}\n\\text{find $x\\in X$ such that}\\quad 0\\in Ax+Bx.\n\\end{equation}\nWe also denote by\n\\begin{equation}\n\\zer(A+B) :=(A+B)^{-1}(0) =\\menge{x\\in X}{0\\in Ax+Bx}\n\\end{equation}\n","start":41114,"end":41431}]},{"id":"S015","semantic_type":"algorithm","kind":"algorithm","labels":["e:dr_seq"],"declared_external":false,"source_only":false,"title":"DR 迭代序列","source_units":[{"raw_tex":"the set of solutions of problem \\eqref{e:sumprob}. Given a starting point $x_0\\in X$, the \\emph{adaptive DR algorithm} generates a sequence $(x_n)_\\nnn$, also called a \\emph{DR sequence}, by \n\\begin{equation}\\label{e:dr_seq}\n\\forall\\nnn,\\quad x_{n+1}\\in Tx_n.\n\\end{equation}\n","start":41431,"end":41706}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"逐步线性收敛率","source_units":[{"raw_tex":"Recall that a sequence $(x_n)_\\nnn$ converges to $\\overline{x}$ with \\emph{$Q$-linear} (or simply \\emph{linear}) rate $\\rho\\in\\left[0,1\\right[$ if\n\\begin{equation}\n\\forall\\nnn,\\quad \\|x_{n+1}-\\overline{x}\\|\\leq \\rho\\|x_n-\\overline{x}\\|.\n\\end{equation}\n","start":62303,"end":62555}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","labels":["t:linear","t:linear_full","t:linear_reduce"],"declared_external":false,"source_only":false,"title":"主定理的两种算子条件","source_units":[{"raw_tex":"Suppose that either\n\\begin{enumerate}\n\\item\\label{t:linear_full} \n$A$ is $\\alpha$-monotone and Lipschitz continuous with constant $\\ell$, $B$ is maximally $\\beta$-monotone, and $\\alpha+\\beta >0$; or\n\\item\\label{t:linear_reduce} \n$A$ is Lipschitz continuous with constant $\\ell$, $B$ is maximally $\\beta$-monotone with $\\beta >\\ell$, and $\\alpha :=-\\ell$.\n\\end{enumerate}\n","start":62632,"end":63003}]},{"id":"S018","semantic_type":"assumption","kind":"assumption","labels":["t:linear","t:linear_full","t:linear_reduce"],"declared_external":false,"source_only":false,"title":"主定理的可行参数","source_units":[{"raw_tex":"Suppose also that $\\zer(A+B)\\neq \\varnothing$ and that $(\\gamma,\\delta,\\lambda,\\mu)\\in\\RPP^2\\times \\left]1,+\\infty\\right[^2$ satisfies \\eqref{e:parameters}.\n","start":63003,"end":63160}]},{"id":"S019","semantic_type":"assumption","kind":"assumption","labels":["t:linear","t:linear_full","t:linear_reduce"],"declared_external":false,"source_only":false,"title":"零点非空假设","source_units":[{"raw_tex":"Suppose also that $\\zer(A+B)\\neq \\varnothing$ and that $(\\gamma,\\delta,\\lambda,\\mu)\\in\\RPP^2\\times \\left]1,+\\infty\\right[^2$ satisfies \\eqref{e:parameters}.\n","start":63003,"end":63160}]},{"id":"S020","semantic_type":"notation","kind":"notation","labels":["t:linear","t:linear_full","t:linear_reduce"],"declared_external":false,"source_only":false,"title":"分解中的扰动常数","source_units":[{"raw_tex":"where\n\\begin{subequations}\n\\begin{align}\n&\\varepsilon :=\\frac{\\mu-(2-2\\gamma\\beta)}{2(1+\\delta\\beta)},\\ \\varepsilon_1:=\\frac{\\kappa\\varepsilon}{1-\\kappa},\\\\\n&\\varphi :=\\varepsilon_1\\lambda[2(1+\\gamma\\alpha)+\\varepsilon_1\\big(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\big)],\\\\\n&\\text{$\\alpha_J$ as in \\eqref{e:alpha_J}}.\n\\end{align}\n\\end{subequations}\n","start":63430,"end":63782}]},{"id":"S021","semantic_type":"theorem","kind":"theorem","labels":["t:linear","t:linear_full","t:linear_reduce"],"declared_external":false,"source_only":false,"title":"A 为利普希茨时的线性收敛","source_units":[{"raw_tex":"\\begin{theorem}[linear convergence when $A$ is Lipschitz]\n\\label{t:linear}\nSuppose that either\n\\begin{enumerate}\n\\item\\label{t:linear_full} \n$A$ is $\\alpha$-monotone and Lipschitz continuous with constant $\\ell$, $B$ is maximally $\\beta$-monotone, and $\\alpha+\\beta >0$; or\n\\item\\label{t:linear_reduce} \n$A$ is Lipschitz continuous with constant $\\ell$, $B$ is maximally $\\beta$-monotone with $\\beta >\\ell$, and $\\alpha :=-\\ell$.\n\\end{enumerate}\nSuppose also that $\\zer(A+B)\\neq \\varnothing$ and that $(\\gamma,\\delta,\\lambda,\\mu)\\in\\RPP^2\\times \\left]1,+\\infty\\right[^2$ satisfies \\eqref{e:parameters}.\nThen $T$ is Lipschitz continuous with constant\n\\begin{equation}\n\\rho :=(1-\\kappa)\n\\sqrt{\\big(1+\\varepsilon_1(\\lambda-1)\\big)^2-\\varphi\\alpha_J} +\\kappa(1-\\varepsilon(\\lambda-1))\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}} <1,\n\\end{equation} \nwhere\n\\begin{subequations}\n\\begin{align}\n&\\varepsilon :=\\frac{\\mu-(2-2\\gamma\\beta)}{2(1+\\delta\\beta)},\\ \\varepsilon_1:=\\frac{\\kappa\\varepsilon}{1-\\kappa},\\\\\n&\\varphi :=\\varepsilon_1\\lambda[2(1+\\gamma\\alpha)+\\varepsilon_1\\big(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\big)],\\\\\n&\\text{$\\alpha_J$ as in \\eqref{e:alpha_J}}.\n\\end{align}\n\\end{subequations}\nConsequently, every DR sequence $(x_n)_\\nnn$ generated by $T$ converges strongly to the unique fixed point $\\overline{x}$ of $T$ with linear rate $\\rho$.\n\\end{theorem}","start":62557,"end":63949}]},{"id":"S022","semantic_type":"lemma","kind":"lemma","labels":["f:Minty"],"declared_external":false,"source_only":false,"title":"Minty 的满域刻画","source_units":[{"raw_tex":"\\begin{fact}\n\\label{f:Minty}\nLet $A\\colon X\\rightrightarrows X$ be monotone and let $\\gamma\\in \\RPP$. Then $\\dom J_{\\gamma A} =X$ if and only if $A$ is maximally monotone.\n\\end{fact}","start":14295,"end":14477}]},{"id":"S023","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"满域刻画的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nBy definition, $\\dom J_{\\gamma A} =\\ran(\\Id+\\gamma A) :=(\\Id+\\gamma A)(X)$. Since $\\gamma\\in \\RPP$, it holds that $\\gamma A$ is monotone. According to Minty's theorem (see, e.g., \\cite[Theorem~21.1]{BC17}), $\\dom J_{\\gamma A} =\\ran(\\Id+\\gamma A) =X$ if and only if $\\gamma A$ is maximally monotone. By \\cite[Proposition~20.22]{BC17}, the latter occurs if and only if $A$ is maximally monotone. \n\\end{proof}","start":14478,"end":14898}]},{"id":"S024","semantic_type":"lemma","kind":"lemma","labels":["l:transfer","l:transfer_maxmono","l:transfer_mono"],"declared_external":false,"source_only":false,"title":"平移单调常数","source_units":[{"raw_tex":"\\begin{lemma}[monotonicity versus $\\alpha$-monotonicity]\n\\label{l:transfer}\nLet $A\\colon X\\rightrightarrows X$ and let $\\alpha, \\beta\\in \\RR$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{l:transfer_mono} \n$A$ is $\\alpha$-monotone if and only if $A-\\beta\\Id$ is $(\\alpha-\\beta)$-monotone.\n","start":16520,"end":16819}]},{"id":"S025","semantic_type":"lemma","kind":"lemma","labels":["l:transfer","l:transfer_maxmono","l:transfer_mono"],"declared_external":false,"source_only":false,"title":"平移保持极大性","source_units":[{"raw_tex":"Let $A\\colon X\\rightrightarrows X$ and let $\\alpha, \\beta\\in \\RR$. Then the following hold:\n","start":16596,"end":16688},{"raw_tex":"\\item\\label{l:transfer_maxmono}\n$A$ is maximally $\\alpha$-monotone if and only if $A-\\beta\\Id$ is maximally $(\\alpha-\\beta)$-monotone. \n\\end{enumerate}\nConsequently, $A$ is (resp., maximally) $\\alpha$-monotone if and only if $A-\\alpha\\Id$ is (resp., maximally) monotone.\n\\end{lemma}","start":16819,"end":17101}]},{"id":"S026","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"平移引理的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{l:transfer_mono}: We first have the equivalences\n\\begin{subequations}\n\\begin{align}\n(x,u)\\in \\gra A &\\iff (x,u-\\beta x)\\in \\gra(A-\\beta\\Id), \\\\\n(y,v)\\in \\gra A &\\iff (y,v-\\beta y)\\in \\gra(A-\\beta\\Id),\n\\end{align}\n\\end{subequations}\nand\n\\begin{equation}\n\\scal{x-y}{u-v}\\geq \\alpha\\|x-y\\|^2 \\iff \\scal{x-y}{(u-\\beta x)-(v-\\beta y)}\\geq (\\alpha-\\beta)\\|x-y\\|^2,\n\\end{equation}\nfrom which the conclusion follows.\n\n\\ref{l:transfer_maxmono}: Assume that $A$ is maximally $\\alpha$-monotone. By \\ref{l:transfer_mono}, $A-\\beta\\Id$ is $(\\alpha-\\beta)$-monotone. Now, suppose that $A-\\beta\\Id$ is not maximally $(\\alpha-\\beta)$-monotone. Then there must exist $B'\\colon X\\rightrightarrows X$ such that $B'$ is $(\\alpha-\\beta)$-monotone and $\\gra(A-\\beta\\Id)\\subsetneq \\gra B'$. It follows that $B :=B'+\\beta\\Id$ is $\\alpha$-monotone due to \\ref{l:transfer_mono} and that $\\gra A\\subsetneq \\gra B$, which contradict the maximal $\\alpha$-monotonicity of $A$. We deduce that if $A$ is maximally $\\alpha$-monotone, then $A-\\beta\\Id$ is maximally $(\\alpha-\\beta)$-monotone. This also implies that if $A-\\beta\\Id$ is maximally $(\\alpha-\\beta)$-monotone, then $A =(A-\\beta\\Id)+\\beta\\Id$ is maximally $\\alpha$-monotone, and we are done. \n\\end{proof}","start":17102,"end":18352}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"图变换与内积等价","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{l:transfer_mono}: We first have the equivalences\n\\begin{subequations}\n\\begin{align}\n(x,u)\\in \\gra A &\\iff (x,u-\\beta x)\\in \\gra(A-\\beta\\Id), \\\\\n(y,v)\\in \\gra A &\\iff (y,v-\\beta y)\\in \\gra(A-\\beta\\Id),\n\\end{align}\n\\end{subequations}\nand\n\\begin{equation}\n\\scal{x-y}{u-v}\\geq \\alpha\\|x-y\\|^2 \\iff \\scal{x-y}{(u-\\beta x)-(v-\\beta y)}\\geq (\\alpha-\\beta)\\|x-y\\|^2,\n\\end{equation}\nfrom which the conclusion follows.\n","start":17102,"end":17530}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"严格图扩张的反证","source_units":[{"raw_tex":"\\ref{l:transfer_maxmono}: Assume that $A$ is maximally $\\alpha$-monotone. By \\ref{l:transfer_mono}, $A-\\beta\\Id$ is $(\\alpha-\\beta)$-monotone. Now, suppose that $A-\\beta\\Id$ is not maximally $(\\alpha-\\beta)$-monotone. Then there must exist $B'\\colon X\\rightrightarrows X$ such that $B'$ is $(\\alpha-\\beta)$-monotone and $\\gra(A-\\beta\\Id)\\subsetneq \\gra B'$. It follows that $B :=B'+\\beta\\Id$ is $\\alpha$-monotone due to \\ref{l:transfer_mono} and that $\\gra A\\subsetneq \\gra B$, which contradict the maximal $\\alpha$-monotonicity of $A$. We deduce that if $A$ is maximally $\\alpha$-monotone, then $A-\\beta\\Id$ is maximally $(\\alpha-\\beta)$-monotone. This also implies that if $A-\\beta\\Id$ is maximally $(\\alpha-\\beta)$-monotone, then $A =(A-\\beta\\Id)+\\beta\\Id$ is maximally $\\alpha$-monotone, and we are done. \n\\end{proof}","start":17531,"end":18352}]},{"id":"S029","semantic_type":"lemma","kind":"lemma","labels":["e:resol_gen","l:resol","l:resol_gen","l:resol_single"],"declared_external":false,"source_only":false,"title":"预解式的图上估计","source_units":[{"raw_tex":"\\begin{lemma}[resolvents of $\\alpha$-monotone operators]\n\\label{l:resol}\nLet $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{l:resol_gen}\nFor all $(x,a), (y,b)\\in \\gra J_{\\gamma A}$,\n\\begin{subequations}\n\\begin{align}\n\\label{e:resol_gen}\n\\scal{x-y}{a-b} &\\geq (1+\\gamma\\alpha)\\|a-b\\|^2 \\quad\\text{and}\\\\\n\\|x-y\\| &\\geq (1+\\gamma\\alpha)\\|a-b\\|.\n\\end{align} \n\\end{subequations}\n","start":18354,"end":18814}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","labels":["l:resol","l:resol_gen","l:resol_single"],"declared_external":false,"source_only":false,"title":"单值预解式的余强制界","source_units":[{"raw_tex":"Let $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$. Then the following hold:\n","start":18427,"end":18534},{"raw_tex":"\\item\\label{l:resol_single}\nIf $J_{\\gamma A}$ is single-valued, then, for all $x,y\\in \\dom J_{\\gamma A}$,\n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq (1+\\gamma\\alpha)\\|J_{\\gamma A}x-J_{\\gamma A}y\\|^2,\n\\end{equation}\ni.e., $J_{\\gamma A}$ is $(1+\\gamma\\alpha)$-cocoercive.\n\\end{enumerate}\n\\end{lemma}","start":18814,"end":19129}]},{"id":"S031","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"预解式基本估计的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{l:resol_gen}: Let $(x,a), (y,b)\\in \\gra J_{\\gamma A}$. Then $x\\in (\\Id+\\gamma A)a$, $y\\in (\\Id+\\gamma A)b$, and so $x =a+\\gamma u$, $y =b+\\gamma v$ for some $u\\in Aa$, $v\\in Ab$. We derive from the $\\alpha$-monotonicity of $A$ that \n\\begin{subequations}\n\\begin{align}\n\\scal{x-y}{a-b}\n&=\\scal{(a+\\gamma u)-(b+\\gamma v)}{a-b}\\\\\n&=\\|a-b\\|^2+\\gamma\\scal{a-b}{u-v}\\\\\n&\\geq \\|a-b\\|^2+\\gamma\\alpha\\|a-b\\|^2\\\\\n&=(1+\\gamma\\alpha)\\|a-b\\|^2.\n\\end{align}\n\\end{subequations}\nNow, by the Cauchy--Schwarz inequality,\n\\begin{equation}\n\\|x-y\\|\\|a-b\\|\\geq \\scal{x-y}{a-b}\\geq (1+\\gamma\\alpha)\\|a-b\\|^2.\n\\end{equation}\nwhich gives $\\|x-y\\|\\geq (1+\\gamma\\alpha)\\|a-b\\|$ while noting that this is trivial when $a =b$. \n\n\\ref{l:resol_single}: This is a direct consequence of \\ref{l:resol_gen}.\n\\end{proof}","start":19130,"end":19932}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"图上展开及距离界","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{l:resol_gen}: Let $(x,a), (y,b)\\in \\gra J_{\\gamma A}$. Then $x\\in (\\Id+\\gamma A)a$, $y\\in (\\Id+\\gamma A)b$, and so $x =a+\\gamma u$, $y =b+\\gamma v$ for some $u\\in Aa$, $v\\in Ab$. We derive from the $\\alpha$-monotonicity of $A$ that \n\\begin{subequations}\n\\begin{align}\n\\scal{x-y}{a-b}\n&=\\scal{(a+\\gamma u)-(b+\\gamma v)}{a-b}\\\\\n&=\\|a-b\\|^2+\\gamma\\scal{a-b}{u-v}\\\\\n&\\geq \\|a-b\\|^2+\\gamma\\alpha\\|a-b\\|^2\\\\\n&=(1+\\gamma\\alpha)\\|a-b\\|^2.\n\\end{align}\n\\end{subequations}\nNow, by the Cauchy--Schwarz inequality,\n\\begin{equation}\n\\|x-y\\|\\|a-b\\|\\geq \\scal{x-y}{a-b}\\geq (1+\\gamma\\alpha)\\|a-b\\|^2.\n\\end{equation}\nwhich gives $\\|x-y\\|\\geq (1+\\gamma\\alpha)\\|a-b\\|$ while noting that this is trivial when $a =b$. \n","start":19130,"end":19847}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"专门化到单值预解式","source_units":[{"raw_tex":"\\ref{l:resol_single}: This is a direct consequence of \\ref{l:resol_gen}.\n\\end{proof}","start":19848,"end":19932}]},{"id":"S034","semantic_type":"proposition","kind":"proposition","labels":["p:resol","p:resol_dom","p:resol_single"],"declared_external":false,"source_only":false,"title":"预解式的单值性","source_units":[{"raw_tex":"\\begin{proposition}[single-valuedness and full domain]\n\\label{p:resol}\nLet $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$ such that $1+\\gamma\\alpha >0$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{p:resol_single} \n$J_{\\gamma A}$ is single-valued.\n","start":19935,"end":20223}]},{"id":"S035","semantic_type":"proposition","kind":"proposition","labels":["p:resol","p:resol_dom","p:resol_single"],"declared_external":false,"source_only":false,"title":"广义单调预解式的满域刻画","source_units":[{"raw_tex":"Let $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$ such that $1+\\gamma\\alpha >0$. Then the following hold:\n","start":20006,"end":20143},{"raw_tex":"\\item\\label{p:resol_dom} \n$\\dom J_{\\gamma A} =X$ if and only if $A$ is maximally $\\alpha$-monotone.\n\\end{enumerate}\n\\end{proposition}","start":20223,"end":20356}]},{"id":"S036","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"单值与满域的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{p:resol_single}: This follows from Lemma~\\ref{l:resol}\\ref{l:resol_gen}.\n\n\\ref{p:resol_dom}: By Lemma~\\ref{l:transfer}\\ref{l:transfer_mono}, $A':=A-\\alpha\\Id$ is monotone.\nNoting that $(\\beta T)^{-1} =T^{-1}\\circ \\frac{1}{\\beta}\\Id$ for any operator $T$ and any $\\beta\\in \\RR\\smallsetminus\\{0\\}$, we have\n\\begin{subequations}\n\\begin{align}\nJ_{\\gamma A}&=(\\Id+\\gamma A)^{-1}=\\big((1+\\gamma\\alpha)\\Id+\\gamma(A-\\alpha\\Id)\\big)^{-1}\\\\\n&=\\left(\\Id+\\frac{\\gamma}{1+\\gamma\\alpha} A'\\right)^{-1} \\circ \\left(\\frac{1}{1+\\gamma\\alpha}\\Id\\right)\\\\\n&=J_{\\frac{\\gamma}{1+\\gamma\\alpha}A'}\\circ \\left(\\frac{1}{1+\\gamma\\alpha}\\Id\\right).\n\\end{align}\n\\end{subequations}\nIt follows that \n\\begin{subequations}\n\\begin{align}\n\\dom J_{\\gamma A} =X & \\iff \\dom J_{\\frac{\\gamma}{1+\\gamma\\alpha}A'} =X &&\\\\\n& \\iff \\text{$A'$ is maximally monotone} &&\\hspace*{-.8in}\\text{(by Fact~\\ref{f:Minty})} \\\\\n& \\iff \\text{$A$ is maximally $\\alpha$-monotone} &&\\hspace*{-.8in}\\text{(by Lemma~\\ref{l:transfer}\\ref{l:transfer_maxmono})}.\n\\end{align}\n\\end{subequations}\nThe proof is complete.\n\\end{proof}","start":20357,"end":21441}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"正系数迫使单值","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{p:resol_single}: This follows from Lemma~\\ref{l:resol}\\ref{l:resol_gen}.\n","start":20357,"end":20449}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"平移缩放后应用 Minty","source_units":[{"raw_tex":"\n\\ref{p:resol_dom}: By Lemma~\\ref{l:transfer}\\ref{l:transfer_mono}, $A':=A-\\alpha\\Id$ is monotone.\nNoting that $(\\beta T)^{-1} =T^{-1}\\circ \\frac{1}{\\beta}\\Id$ for any operator $T$ and any $\\beta\\in \\RR\\smallsetminus\\{0\\}$, we have\n\\begin{subequations}\n\\begin{align}\nJ_{\\gamma A}&=(\\Id+\\gamma A)^{-1}=\\big((1+\\gamma\\alpha)\\Id+\\gamma(A-\\alpha\\Id)\\big)^{-1}\\\\\n&=\\left(\\Id+\\frac{\\gamma}{1+\\gamma\\alpha} A'\\right)^{-1} \\circ \\left(\\frac{1}{1+\\gamma\\alpha}\\Id\\right)\\\\\n&=J_{\\frac{\\gamma}{1+\\gamma\\alpha}A'}\\circ \\left(\\frac{1}{1+\\gamma\\alpha}\\Id\\right).\n\\end{align}\n\\end{subequations}\nIt follows that \n\\begin{subequations}\n\\begin{align}\n\\dom J_{\\gamma A} =X & \\iff \\dom J_{\\frac{\\gamma}{1+\\gamma\\alpha}A'} =X &&\\\\\n& \\iff \\text{$A'$ is maximally monotone} &&\\hspace*{-.8in}\\text{(by Fact~\\ref{f:Minty})} \\\\\n& \\iff \\text{$A$ is maximally $\\alpha$-monotone} &&\\hspace*{-.8in}\\text{(by Lemma~\\ref{l:transfer}\\ref{l:transfer_maxmono})}.\n\\end{align}\n\\end{subequations}\nThe proof is complete.\n\\end{proof}","start":20449,"end":21441}]},{"id":"S039","semantic_type":"proposition","kind":"proposition","labels":["p:maxalpha","p:maxalpha_cont","p:maxalpha_mono"],"declared_external":false,"source_only":false,"title":"非负常数下的极大性","source_units":[{"raw_tex":"\\begin{proposition}[maximal $\\alpha$-monotonicity]\n\\label{p:maxalpha}\nThe following statements hold: \n\\begin{enumerate}\n\\item\\label{p:maxalpha_mono} \nLet $A\\colon X\\rightrightarrows X$ and $\\alpha\\in \\RP$. Then $A$ is maximally $\\alpha$-monotone if and only if $A$ is $\\alpha$-monotone and maximally monotone.\n","start":21510,"end":21820}]},{"id":"S040","semantic_type":"proposition","kind":"proposition","labels":["p:maxalpha","p:maxalpha_cont","p:maxalpha_mono"],"declared_external":false,"source_only":false,"title":"连续满域推出极大性","source_units":[{"raw_tex":"The following statements hold: \n","start":21580,"end":21612},{"raw_tex":"\\item\\label{p:maxalpha_cont} \nLet $A\\colon X\\to X$ and $\\alpha\\in \\RR$. Then $A$ is maximally $\\alpha$-monotone if $A$ is $\\alpha$-monotone and continuous with full domain.\n\\end{enumerate}\n\\end{proposition}","start":21820,"end":22026}]},{"id":"S041","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"两种极大性判据的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{p:maxalpha_mono}: Since $\\alpha\\geq 0$, it follows from Fact~\\ref{f:Minty} that $A$ is $\\alpha$-monotone and maximally monotone if and only if $A$ is $\\alpha$-monotone and $\\dom J_A =X$, which, by Proposition~\\ref{p:resol}\\ref{p:resol_dom}, happen if and only if $A$ is maximally $\\alpha$-monotone.\n\n\\ref{p:maxalpha_cont}: Set $A' :=A-\\alpha\\Id$. Then $A'$ is monotone (due to Lemma~\\ref{l:transfer}\\ref{l:transfer_mono}) and continuous with full domain. By \\cite[Corollary~20.28]{BC17}, $A'$ is maximally monotone, and by Lemma~\\ref{l:transfer}\\ref{l:transfer_maxmono}, $A$ is maximally $\\alpha$-monotone. \n\\end{proof}","start":22027,"end":22665}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"非负常数的满域等价","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{p:maxalpha_mono}: Since $\\alpha\\geq 0$, it follows from Fact~\\ref{f:Minty} that $A$ is $\\alpha$-monotone and maximally monotone if and only if $A$ is $\\alpha$-monotone and $\\dom J_A =X$, which, by Proposition~\\ref{p:resol}\\ref{p:resol_dom}, happen if and only if $A$ is maximally $\\alpha$-monotone.\n","start":22027,"end":22345}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"连续满域算子的平移","source_units":[{"raw_tex":"\n\\ref{p:maxalpha_cont}: Set $A' :=A-\\alpha\\Id$. Then $A'$ is monotone (due to Lemma~\\ref{l:transfer}\\ref{l:transfer_mono}) and continuous with full domain. By \\cite[Corollary~20.28]{BC17}, $A'$ is maximally monotone, and by Lemma~\\ref{l:transfer}\\ref{l:transfer_maxmono}, $A$ is maximally $\\alpha$-monotone. \n\\end{proof}","start":22345,"end":22665}]},{"id":"S044","semantic_type":"lemma","kind":"lemma","labels":["e:L-resol_only","e:L-resol_only1","e:alpha_J","e:expan","l:L-resol","l:L-resol_mono","l:L-resol_only"],"declared_external":false,"source_only":false,"title":"利普希茨预解式的基础下界","source_units":[{"raw_tex":"\\begin{lemma}[resolvents of Lipschitz $\\alpha$-monotone operators]\n\\label{l:L-resol}\nLet $A\\colon X\\to X$ be Lipschitz continuous with constant $\\ell$ and let $\\gamma\\in \\RPP$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{l:L-resol_only} \nFor all $(x,a), (y,b)\\in \\gra J_{\\gamma A}$,\n\\begin{subequations}\n\\begin{align}\n\\label{e:expan}\n\\|a-b\\|&\\geq \\frac{1}{1+\\gamma\\ell}\\|x-y\\|,\\\\\n\\label{e:L-resol_only}\n\\scal{x-y}{a-b}&\\geq \\frac{1}{2}\\|x-y\\|^2 +\\frac{1}{2}(1-\\gamma^2\\ell^2)\\|a-b\\|^2,\n\\end{align}\n\\end{subequations}\nand if $\\gamma\\ell\\leq 1$, then\n\\begin{equation}\\label{e:L-resol_only1}\n\\scal{x-y}{a-b}\\geq \\frac{1}{1+\\gamma\\ell}\\|x-y\\|^2.\n\\end{equation}\n","start":24136,"end":24804}]},{"id":"S045","semantic_type":"lemma","kind":"lemma","labels":["e:alpha_J","l:L-resol","l:L-resol_mono","l:L-resol_only"],"declared_external":false,"source_only":false,"title":"单调利普希茨预解式的增强界","source_units":[{"raw_tex":"Let $A\\colon X\\to X$ be Lipschitz continuous with constant $\\ell$ and let $\\gamma\\in \\RPP$. Then the following hold:\n","start":24221,"end":24338},{"raw_tex":"\\item\\label{l:L-resol_mono}\nIf $A$ is $\\alpha$-monotone with $1+\\gamma\\alpha >0$, then, for all $x, y\\in \\dom J_{\\gamma A}$, \n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq (1+\\gamma\\alpha)\\alpha_J\\|x-y\\|^2,\n\\end{equation}\nwhere \n\\begin{subequations}\n\\label{e:alpha_J}\n\\begin{equation}\n\\alpha_J \n:=\\begin{cases}\n\\dfrac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2} &\\text{if~} \\gamma\\ell\\geq 1 \n,\\\\\n\\dfrac{1}{(1+\\gamma\\alpha)(1+\\gamma\\ell)} &\\text{if~} \\gamma\\ell\\leq 1;\n\\end{cases}\n\\end{equation}\nand if additionally $A$ satisfies \\eqref{e:alpha-mono} with equality, then \n\\begin{equation}\n\\alpha_J :=\\frac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}. \n\\end{equation}\n\\end{subequations}\n\\end{enumerate}\n\\end{lemma}","start":24804,"end":25513}]},{"id":"S046","semantic_type":"definition","kind":"definition","labels":["e:alpha_J","l:L-resol","l:L-resol_mono","l:L-resol_only"],"declared_external":false,"source_only":false,"title":"增强界中的分段常数","source_units":[{"raw_tex":"where \n\\begin{subequations}\n\\label{e:alpha_J}\n\\begin{equation}\n\\alpha_J \n:=\\begin{cases}\n\\dfrac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2} &\\text{if~} \\gamma\\ell\\geq 1 \n,\\\\\n\\dfrac{1}{(1+\\gamma\\alpha)(1+\\gamma\\ell)} &\\text{if~} \\gamma\\ell\\leq 1;\n\\end{cases}\n\\end{equation}\nand if additionally $A$ satisfies \\eqref{e:alpha-mono} with equality, then \n\\begin{equation}\n\\alpha_J :=\\frac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}. \n\\end{equation}\n\\end{subequations}\n","start":25041,"end":25486}]},{"id":"S047","semantic_type":"proof","kind":"proof","labels":["e:L-resol","e:L-resol'","e:L-resol''"],"declared_external":false,"source_only":false,"title":"增强预解式估计的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{l:L-resol_only}: Let $(x,a), (y,b)\\in \\gra J_{\\gamma A}$. Then $x =a+\\gamma Aa$ and $y =b+\\gamma Ab$. By Lipschitz continuity, $\\|Aa-Ab\\|\\leq \\ell\\|a-b\\|$. It follows that\n\\begin{equation}\n\\|x-y\\|\\leq \\|a-b\\| +\\gamma\\|Aa-Ab\\|\\leq (1+\\gamma\\ell)\\|a-b\\|\n\\end{equation}\nand that\n\\begin{subequations}\n\\label{e:L-resol}\n\\begin{align}\n2\\scal{x-y}{a-b} &=\\|x-y\\|^2 +\\|a-b\\|^2 -\\|(x-a)-(y-b)\\|^2\\\\\n&=\\|x-y\\|^2 +\\|a-b\\|^2 -\\gamma^2\\|Aa-Ab\\|^2\\\\\n&\\geq \\|x-y\\|^2 +(1-\\gamma^2\\ell^2)\\|a-b\\|^2.\n\\end{align}\n\\end{subequations}\nIf $\\gamma\\ell\\leq 1$, then combining the above inequalities yields\n\\begin{equation}\n2\\scal{x-y}{a-b}\\geq \\|x-y\\|^2 +\\frac{1-\\gamma^2\\ell^2}{(1+\\gamma\\ell)^2}\\|x-y\\|^2 =\\frac{2}{1+\\gamma\\ell}\\|x-y\\|^2,\n\\end{equation}\nand we get the claim.\n\n\\ref{l:L-resol_mono}: We first note that $J_{\\gamma A}$ is single-valued due to Proposition~\\ref{p:resol}\\ref{p:resol_single}. Then \\eqref{e:L-resol} reads as\n\\begin{equation}\n\\label{e:L-resol'}\n2\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\|x-y\\|^2 +(1-\\gamma^2\\ell^2)\\|J_{\\gamma A}x-J_{\\gamma A}y\\|^2.\n\\end{equation}\nWe claim that if $\\gamma\\ell\\geq 1$ or $A$ satisfies \\eqref{e:alpha-mono} with equality, then \n\\begin{equation}\n\\label{e:L-resol''}\n2\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\|x-y\\|^2 +\\frac{1-\\gamma^2\\ell^2}{1+\\gamma\\alpha}\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}.\n\\end{equation} \nIndeed, the former case implies $1-\\gamma^2\\ell^2\\leq 0$ and, by combining \\eqref{e:L-resol'} with Lemma~\\ref{l:resol}\\ref{l:resol_single} and noting that $1+\\gamma\\alpha >0$, we get \\eqref{e:L-resol''}.  \nIn the latter case, Lemma~\\ref{l:resol}\\ref{l:resol_single} reduces to\n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y} =(1+\\gamma\\alpha)\\|J_{\\gamma A}x-J_{\\gamma A}y\\|^2.\n\\end{equation} \nSubstituting this into \\eqref{e:L-resol'}, we also obtain \\eqref{e:L-resol''}. \n\nNow, in view of Remark~\\ref{r:Lip_mono}, $1+2\\gamma\\alpha+\\gamma^2\\ell^2\\geq 1+2\\gamma\\alpha+\\gamma^2\\alpha^2 =(1+\\gamma\\alpha)^2 >0$. It thus follows from \\eqref{e:L-resol''} that \n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\frac{1+\\gamma\\alpha}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}\\|x-y\\|^2.\n\\end{equation}\nFinally, if $\\gamma\\ell\\leq 1$, then, by \\ref{l:L-resol_only}, \n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\frac{1}{1+\\gamma\\ell}\\|x-y\\|^2,\n\\end{equation}\nand the conclusion follows.\n\\end{proof}","start":25514,"end":27906}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":["e:L-resol"],"declared_external":false,"source_only":false,"title":"三角界与极化展开","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{l:L-resol_only}: Let $(x,a), (y,b)\\in \\gra J_{\\gamma A}$. Then $x =a+\\gamma Aa$ and $y =b+\\gamma Ab$. By Lipschitz continuity, $\\|Aa-Ab\\|\\leq \\ell\\|a-b\\|$. It follows that\n\\begin{equation}\n\\|x-y\\|\\leq \\|a-b\\| +\\gamma\\|Aa-Ab\\|\\leq (1+\\gamma\\ell)\\|a-b\\|\n\\end{equation}\nand that\n\\begin{subequations}\n\\label{e:L-resol}\n\\begin{align}\n2\\scal{x-y}{a-b} &=\\|x-y\\|^2 +\\|a-b\\|^2 -\\|(x-a)-(y-b)\\|^2\\\\\n&=\\|x-y\\|^2 +\\|a-b\\|^2 -\\gamma^2\\|Aa-Ab\\|^2\\\\\n&\\geq \\|x-y\\|^2 +(1-\\gamma^2\\ell^2)\\|a-b\\|^2.\n\\end{align}\n\\end{subequations}\nIf $\\gamma\\ell\\leq 1$, then combining the above inequalities yields\n\\begin{equation}\n2\\scal{x-y}{a-b}\\geq \\|x-y\\|^2 +\\frac{1-\\gamma^2\\ell^2}{(1+\\gamma\\ell)^2}\\|x-y\\|^2 =\\frac{2}{1+\\gamma\\ell}\\|x-y\\|^2,\n\\end{equation}\nand we get the claim.\n","start":25514,"end":26285}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["e:L-resol","e:L-resol'","e:L-resol''"],"declared_external":false,"source_only":false,"title":"按系数符号替换范数项","source_units":[{"raw_tex":"\n\\ref{l:L-resol_mono}: We first note that $J_{\\gamma A}$ is single-valued due to Proposition~\\ref{p:resol}\\ref{p:resol_single}. Then \\eqref{e:L-resol} reads as\n\\begin{equation}\n\\label{e:L-resol'}\n2\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\|x-y\\|^2 +(1-\\gamma^2\\ell^2)\\|J_{\\gamma A}x-J_{\\gamma A}y\\|^2.\n\\end{equation}\nWe claim that if $\\gamma\\ell\\geq 1$ or $A$ satisfies \\eqref{e:alpha-mono} with equality, then \n\\begin{equation}\n\\label{e:L-resol''}\n2\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\|x-y\\|^2 +\\frac{1-\\gamma^2\\ell^2}{1+\\gamma\\alpha}\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}.\n\\end{equation} \nIndeed, the former case implies $1-\\gamma^2\\ell^2\\leq 0$ and, by combining \\eqref{e:L-resol'} with Lemma~\\ref{l:resol}\\ref{l:resol_single} and noting that $1+\\gamma\\alpha >0$, we get \\eqref{e:L-resol''}.  \nIn the latter case, Lemma~\\ref{l:resol}\\ref{l:resol_single} reduces to\n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y} =(1+\\gamma\\alpha)\\|J_{\\gamma A}x-J_{\\gamma A}y\\|^2.\n\\end{equation} \nSubstituting this into \\eqref{e:L-resol'}, we also obtain \\eqref{e:L-resol''}. \n","start":26285,"end":27371}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":["e:L-resol"],"declared_external":false,"source_only":false,"title":"分母正性与分段结论","source_units":[{"raw_tex":"\nNow, in view of Remark~\\ref{r:Lip_mono}, $1+2\\gamma\\alpha+\\gamma^2\\ell^2\\geq 1+2\\gamma\\alpha+\\gamma^2\\alpha^2 =(1+\\gamma\\alpha)^2 >0$. It thus follows from \\eqref{e:L-resol''} that \n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\frac{1+\\gamma\\alpha}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}\\|x-y\\|^2.\n\\end{equation}\nFinally, if $\\gamma\\ell\\leq 1$, then, by \\ref{l:L-resol_only}, \n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\frac{1}{1+\\gamma\\ell}\\|x-y\\|^2,\n\\end{equation}\nand the conclusion follows.\n\\end{proof}","start":27371,"end":27906}]},{"id":"S051","semantic_type":"proposition","kind":"proposition","labels":["p:cresol","p:cresol_Lip","p:cresol_mono"],"declared_external":false,"source_only":false,"title":"恒等与预解式的度量界","source_units":[{"raw_tex":"\\begin{proposition}[linear combinations of resolvents and the identity]\n\\label{p:cresol}\nLet $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$. Set $J:=J_{\\gamma A}$ and define $Q :=\\nu\\Id+\\lambda J$ with $\\nu, \\lambda\\in \\RR$. \n\\begin{enumerate}\n\\item\\label{p:cresol_mono} \nSuppose that $J$ is single-valued and $\\nu\\lambda\\leq 0$. Then, for all $x,y\\in \\dom J$, \n\\begin{equation}\n\\|Qx-Qy\\|^2\\leq \\nu^2\\|x-y\\|^2 +\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\|Jx-Jy\\|^2.\n\\end{equation}\n","start":28997,"end":29511}]},{"id":"S052","semantic_type":"proposition","kind":"proposition","labels":["e:LipQ","p:cresol","p:cresol_Lip","p:cresol_mono"],"declared_external":false,"source_only":false,"title":"线性组合的利普希茨常数","source_units":[{"raw_tex":"Let $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$. Set $J:=J_{\\gamma A}$ and define $Q :=\\nu\\Id+\\lambda J$ with $\\nu, \\lambda\\in \\RR$. \n","start":29086,"end":29253},{"raw_tex":"\\item\\label{p:cresol_Lip}\nSuppose that $A$ is Lipschitz continuous with constant $\\ell$, $1+\\gamma\\alpha >0$, and $\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\leq 0$. Then $Q$ is Lipschitz continuous with constant    \n\\begin{equation}\n\\label{e:LipQ}\n\\rho :=\\sqrt{\\nu^2+\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\alpha_J},\n\\end{equation}\nwhere $\\alpha_J$ is defined as \\eqref{e:alpha_J}.\nIf additionally $\\lambda\\left(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\right)\\geq 0$ whenever $\\gamma\\ell <1$, then the Lipschitz constant \\eqref{e:LipQ} can be improved to\n\\begin{equation}\n\\overline{\\rho} :=\\sqrt{\\nu^2+\\frac{\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}}\\leq \\rho.\n\\end{equation}\n\\end{enumerate}\n\\end{proposition}","start":29511,"end":30263}]},{"id":"S053","semantic_type":"proof","kind":"proof","labels":["e:cresol_Lip","e:identityQ"],"declared_external":false,"source_only":false,"title":"线性组合估计的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $x,y\\in \\dom J$. By the definition of $Q$,\n\\begin{subequations}\n\\label{e:identityQ}\n\\begin{align}\n\\|Qx-Qy\\|^2\n&=\\|\\nu(x-y)+\\lambda(Jx-Jy)\\|^2\\\\\n&=\\nu^2\\|x-y\\|^2 +2\\nu\\lambda\\scal{x-y}{Jx-Jy} +\\lambda^2\\|Jx-Jy\\|^2.\n\\end{align}\n\\end{subequations}\n\n\\ref{p:cresol_mono}: Since $\\nu\\lambda\\leq 0$, combining \\eqref{e:identityQ} with Lemma~\\ref{l:resol}\\ref{l:resol_single} yields\n\\begin{subequations}\n\\begin{align}\n\\|Qx-Qy\\|^2\n&\\leq \\nu^2\\|x-y\\|^2 +2\\nu\\lambda(1+\\gamma\\alpha)\\|Jx-Jy\\|^2 +\\lambda^2\\|Jx-Jy\\|^2\\\\\n&=\\nu^2\\|x-y\\|^2 +\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\|Jx-Jy\\|^2.\n\\end{align}\n\\end{subequations}\n\n\\ref{p:cresol_Lip}: First, according to Proposition~\\ref{p:resol}\\ref{p:resol_single}, $J$ is single-valued, and so is $Q$. Next, using \\eqref{e:identityQ}, Lemma~\\ref{l:resol}\\ref{l:resol_single}, and Lemma~\\ref{l:L-resol}\\ref{l:L-resol_mono} and noting that $\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\leq 0$, we have\n\\begin{subequations}\n\\label{e:cresol_Lip}\n\\begin{align}\n\\|Qx-Qy\\|^2 \n&\\leq \\nu^2\\|x-y\\|^2 +2\\nu\\lambda\\scal{x-y}{Jx-Jy} +\\frac{\\lambda^2}{1+\\gamma\\alpha}\\scal{x-y}{Jx-Jy}\\\\ \n&= \\nu^2\\|x-y\\|^2 +\\frac{\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)}{1+\\gamma\\alpha}\\scal{x-y}{Jx-Jy}\\\\  \n&\\leq \\nu^2\\|x-y\\|^2 +\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\alpha_J\\|x-y\\|^2 =\\rho^2\\|x-y\\|^2, \n\\end{align}\n\\end{subequations}\nwhich implies that $Q$ is Lipschitz continuous with constant $\\rho$.\n\nFor the last statement, we show that $\\alpha_J$ in formula \\eqref{e:LipQ} can be replaced by \n\\begin{equation}\n\\overline{\\alpha}_J :=\\frac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}.\n\\end{equation}\nIf $\\scal{x-y}{Jx-Jy}\\geq (1+\\gamma\\alpha)\\overline{\\alpha}_J\\|x-y\\|^2$, then \\eqref{e:cresol_Lip} also holds with $\\alpha_J$ replaced by $\\overline{\\alpha}_J$. Now, assume that $\\scal{x-y}{Jx-Jy} <(1+\\gamma\\alpha)\\overline{\\alpha}_J\\|x-y\\|^2$. By Lemma~\\ref{l:L-resol}\\ref{l:L-resol_mono}, we must have $\\gamma\\ell <1$, and then, by assumption, $\\lambda\\left(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\right)\\geq 0$. It now follows from \\eqref{e:identityQ} and \\eqref{e:L-resol_only} that\n\\begin{subequations}\n\\begin{align}\n\\|Qx-Qy\\|^2\n&\\leq \\nu^2\\|x-y\\|^2 +2\\nu\\lambda\\scal{x-y}{Jx-Jy} \\notag\\\\ \n&\\qquad +\\frac{\\lambda^2}{1-\\gamma^2\\ell^2}(2\\scal{x-y}{Jx-Jy} -\\|x-y\\|^2)\\\\\n&=\\Big(\\nu^2-\\frac{\\lambda^2}{1-\\gamma^2\\ell^2}\\Big)\\|x-y\\|^2 +2\\lambda\\Big(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\Big)\\scal{x-y}{Jx-Jy}\\\\\n&\\leq \\Big(\\nu^2-\\frac{\\lambda^2}{1-\\gamma^2\\ell^2}\\Big)\\|x-y\\|^2 +2\\lambda\\Big(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\Big)(1+\\gamma\\alpha)\\overline{\\alpha}_J\\|x-y\\|^2\\\\\n&=\\Big(\\nu^2+\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\overline{\\alpha}_J\\Big)\\|x-y\\|^2 =\\overline{\\rho}^2\\|x-y\\|^2.\n\\end{align}\n\\end{subequations}\nFinally, we will prove $\\overline{\\alpha}_J\\geq \\alpha_J$, which implies $\\overline{\\rho}\\leq \\rho$, i.e., the Lipschitz constant is indeed improved. From the definition of $\\alpha_J$, it suffices to consider the case in which $\\alpha_J =1/((1+\\gamma\\alpha)(1+\\gamma\\ell))$. Then $\\gamma\\ell\\leq 1$. \nSince $|\\alpha|\\leq \\ell$ (see Remark~\\ref{r:Lip_mono}), it holds that\n\\begin{subequations}\n\\begin{align}\n0 <(1+\\gamma\\alpha)^2\\leq 1+2\\gamma\\alpha+\\gamma^2\\ell^2\n&\\leq \n1+2\\gamma\\alpha+\\gamma^2\\ell^2\n+(\\gamma\\ell-\\gamma\\alpha)-\\gamma\\ell(\\gamma\\ell-\\gamma\\alpha)\\\\\n&=1+\\gamma\\alpha+\\gamma\\ell+(\\gamma\\alpha)(\\gamma\\ell)\\\\\n&=(1+\\gamma\\alpha)(1+\\gamma\\ell),\n\\end{align}\n\\end{subequations}\nand so \n\\begin{equation}\n\\overline{\\alpha}_J =\\frac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}\\geq \\alpha_J =\\frac{1}{(1+\\gamma\\alpha)(1+\\gamma\\ell)}.\n\\end{equation}\nThe proof is complete.\n\\end{proof}","start":30264,"end":33905}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":["e:cresol_Lip","e:identityQ"],"declared_external":false,"source_only":false,"title":"线性组合的平方展开","source_units":[{"raw_tex":"\\begin{proof}\nLet $x,y\\in \\dom J$. By the definition of $Q$,\n\\begin{subequations}\n\\label{e:identityQ}\n\\begin{align}\n\\|Qx-Qy\\|^2\n&=\\|\\nu(x-y)+\\lambda(Jx-Jy)\\|^2\\\\\n&=\\nu^2\\|x-y\\|^2 +2\\nu\\lambda\\scal{x-y}{Jx-Jy} +\\lambda^2\\|Jx-Jy\\|^2.\n\\end{align}\n\\end{subequations}\n\n","start":30264,"end":30528}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["e:cresol_Lip","e:identityQ"],"declared_external":false,"source_only":false,"title":"负交叉系数的单调估计","source_units":[{"raw_tex":"\\ref{p:cresol_mono}: Since $\\nu\\lambda\\leq 0$, combining \\eqref{e:identityQ} with Lemma~\\ref{l:resol}\\ref{l:resol_single} yields\n\\begin{subequations}\n\\begin{align}\n\\|Qx-Qy\\|^2\n&\\leq \\nu^2\\|x-y\\|^2 +2\\nu\\lambda(1+\\gamma\\alpha)\\|Jx-Jy\\|^2 +\\lambda^2\\|Jx-Jy\\|^2\\\\\n&=\\nu^2\\|x-y\\|^2 +\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\|Jx-Jy\\|^2.\n\\end{align}\n\\end{subequations}\n","start":30528,"end":30896}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":["e:cresol_Lip","e:identityQ"],"declared_external":false,"source_only":false,"title":"两级内积界导出常数","source_units":[{"raw_tex":"\n\\ref{p:cresol_Lip}: First, according to Proposition~\\ref{p:resol}\\ref{p:resol_single}, $J$ is single-valued, and so is $Q$. Next, using \\eqref{e:identityQ}, Lemma~\\ref{l:resol}\\ref{l:resol_single}, and Lemma~\\ref{l:L-resol}\\ref{l:L-resol_mono} and noting that $\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\leq 0$, we have\n\\begin{subequations}\n\\label{e:cresol_Lip}\n\\begin{align}\n\\|Qx-Qy\\|^2 \n&\\leq \\nu^2\\|x-y\\|^2 +2\\nu\\lambda\\scal{x-y}{Jx-Jy} +\\frac{\\lambda^2}{1+\\gamma\\alpha}\\scal{x-y}{Jx-Jy}\\\\ \n&= \\nu^2\\|x-y\\|^2 +\\frac{\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)}{1+\\gamma\\alpha}\\scal{x-y}{Jx-Jy}\\\\  \n&\\leq \\nu^2\\|x-y\\|^2 +\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\alpha_J\\|x-y\\|^2 =\\rho^2\\|x-y\\|^2, \n\\end{align}\n\\end{subequations}\nwhich implies that $Q$ is Lipschitz continuous with constant $\\rho$.\n\n","start":30896,"end":31708}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":["e:cresol_Lip","e:identityQ"],"declared_external":false,"source_only":false,"title":"按内积大小改善常数","source_units":[{"raw_tex":"For the last statement, we show that $\\alpha_J$ in formula \\eqref{e:LipQ} can be replaced by \n\\begin{equation}\n\\overline{\\alpha}_J :=\\frac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}.\n\\end{equation}\nIf $\\scal{x-y}{Jx-Jy}\\geq (1+\\gamma\\alpha)\\overline{\\alpha}_J\\|x-y\\|^2$, then \\eqref{e:cresol_Lip} also holds with $\\alpha_J$ replaced by $\\overline{\\alpha}_J$. Now, assume that $\\scal{x-y}{Jx-Jy} <(1+\\gamma\\alpha)\\overline{\\alpha}_J\\|x-y\\|^2$. By Lemma~\\ref{l:L-resol}\\ref{l:L-resol_mono}, we must have $\\gamma\\ell <1$, and then, by assumption, $\\lambda\\left(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\right)\\geq 0$. It now follows from \\eqref{e:identityQ} and \\eqref{e:L-resol_only} that\n\\begin{subequations}\n\\begin{align}\n\\|Qx-Qy\\|^2\n&\\leq \\nu^2\\|x-y\\|^2 +2\\nu\\lambda\\scal{x-y}{Jx-Jy} \\notag\\\\ \n&\\qquad +\\frac{\\lambda^2}{1-\\gamma^2\\ell^2}(2\\scal{x-y}{Jx-Jy} -\\|x-y\\|^2)\\\\\n&=\\Big(\\nu^2-\\frac{\\lambda^2}{1-\\gamma^2\\ell^2}\\Big)\\|x-y\\|^2 +2\\lambda\\Big(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\Big)\\scal{x-y}{Jx-Jy}\\\\\n&\\leq \\Big(\\nu^2-\\frac{\\lambda^2}{1-\\gamma^2\\ell^2}\\Big)\\|x-y\\|^2 +2\\lambda\\Big(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\Big)(1+\\gamma\\alpha)\\overline{\\alpha}_J\\|x-y\\|^2\\\\\n&=\\Big(\\nu^2+\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\overline{\\alpha}_J\\Big)\\|x-y\\|^2 =\\overline{\\rho}^2\\|x-y\\|^2.\n\\end{align}\n\\end{subequations}\n","start":31708,"end":33024}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":["e:cresol_Lip","e:identityQ"],"declared_external":false,"source_only":false,"title":"比较两种预解式常数","source_units":[{"raw_tex":"Finally, we will prove $\\overline{\\alpha}_J\\geq \\alpha_J$, which implies $\\overline{\\rho}\\leq \\rho$, i.e., the Lipschitz constant is indeed improved. From the definition of $\\alpha_J$, it suffices to consider the case in which $\\alpha_J =1/((1+\\gamma\\alpha)(1+\\gamma\\ell))$. Then $\\gamma\\ell\\leq 1$. \nSince $|\\alpha|\\leq \\ell$ (see Remark~\\ref{r:Lip_mono}), it holds that\n\\begin{subequations}\n\\begin{align}\n0 <(1+\\gamma\\alpha)^2\\leq 1+2\\gamma\\alpha+\\gamma^2\\ell^2\n&\\leq \n1+2\\gamma\\alpha+\\gamma^2\\ell^2\n+(\\gamma\\ell-\\gamma\\alpha)-\\gamma\\ell(\\gamma\\ell-\\gamma\\alpha)\\\\\n&=1+\\gamma\\alpha+\\gamma\\ell+(\\gamma\\alpha)(\\gamma\\ell)\\\\\n&=(1+\\gamma\\alpha)(1+\\gamma\\ell),\n\\end{align}\n\\end{subequations}\nand so \n\\begin{equation}\n\\overline{\\alpha}_J =\\frac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}\\geq \\alpha_J =\\frac{1}{(1+\\gamma\\alpha)(1+\\gamma\\ell)}.\n\\end{equation}\nThe proof is complete.\n\\end{proof}","start":33024,"end":33905}]},{"id":"S059","semantic_type":"corollary","kind":"corollary","labels":["c:rresol","c:rresol_perturb","c:rresol_relax"],"declared_external":false,"source_only":false,"title":"松弛预解式的度量界","source_units":[{"raw_tex":"\\begin{corollary}[relaxed resolvents of $\\alpha$-monotone operators]\n\\label{c:rresol}\nLet $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$. Suppose that $J:=J_{\\gamma A}$ is single-valued and define $R:=(1-\\lambda)\\Id +\\lambda J$ with $\\lambda\\in \\RP$, and $Q:=R+\\varepsilon\\Id$ with $\\varepsilon\\in \\RR$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{c:rresol_relax} \nIf $\\lambda\\geq 1$, then, for all $x, y\\in \\dom J$, \n\\begin{equation}\n\\|Rx-Ry\\|^2\\leq (\\lambda-1)^2\\|x-y\\|^2 -\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda\\big)\\|Jx-Jy\\|^2.\n\\end{equation}\n","start":35151,"end":35748}]},{"id":"S060","semantic_type":"corollary","kind":"corollary","labels":["c:rresol","c:rresol_perturb","c:rresol_relax"],"declared_external":false,"source_only":false,"title":"加恒等扰动后的上界","source_units":[{"raw_tex":"Let $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$. Suppose that $J:=J_{\\gamma A}$ is single-valued and define $R:=(1-\\lambda)\\Id +\\lambda J$ with $\\lambda\\in \\RP$, and $Q:=R+\\varepsilon\\Id$ with $\\varepsilon\\in \\RR$. Then the following hold:\n","start":35237,"end":35510},{"raw_tex":"\\item\\label{c:rresol_perturb} \nIf $\\varepsilon\\leq \\lambda-1$, then, for all $x, y\\in \\dom J$, \n\\begin{align}\n\\|Qx-Qy\\|^2 &\\leq (\\lambda-1-\\varepsilon)^2\\|x-y\\|^2 \\notag\\\\ \n&\\quad -\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda-2\\varepsilon(1+\\gamma\\alpha)\\big)\\|Jx-Jy\\|^2.\n\\end{align}\nConsequently, if additionally $(\\lambda-1)(2+2\\gamma\\alpha)-\\lambda-2\\varepsilon(1+\\gamma\\alpha)\\geq 0$, then $Q$ is Lipschitz continuous with constant $(\\lambda-1-\\varepsilon)$.\n\\end{enumerate} \n\\end{corollary}","start":35748,"end":36247}]},{"id":"S061","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"一般松弛估计的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nBecause \\ref{c:rresol_relax} is a consequence of \\ref{c:rresol_perturb} with $\\varepsilon =0$, it suffices to prove only the latter. To this end, noting that $Q =R+\\varepsilon\\Id =(1-\\lambda+\\varepsilon)\\Id+\\lambda J$ and using Proposition~\\ref{p:cresol}\\ref{p:cresol_mono} with $\\nu =1-\\lambda+\\varepsilon \\leq 0$, we have that\n\\begin{subequations}\n\\begin{align}\n\\|Qx-Qy\\|^2&\\leq (1-\\lambda+\\varepsilon)^2\\|x-y\\|^2 +\\lambda\\left(2(1-\\lambda+\\varepsilon)(1+\\gamma\\alpha)+\\lambda\\right)\\|Jx-Jy\\|^2\\\\\n&=(\\lambda-1-\\varepsilon)^2\\|x-y\\|^2 -\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda-2\\varepsilon(1+\\gamma\\alpha)\\big)\\|Jx-Jy\\|^2\n\\end{align}\n\\end{subequations}\nwhich proves \\ref{c:rresol_perturb}. \n\\end{proof}","start":36248,"end":36973}]},{"id":"S062","semantic_type":"corollary","kind":"corollary","labels":["c:L-rresol","c:L-rresol_perturb","c:L-rresol_relax","e:cL-rresol-1"],"declared_external":false,"source_only":false,"title":"利普希茨松弛预解式常数","source_units":[{"raw_tex":"\\begin{corollary}[relaxed resolvents of Lipschitz $\\alpha$-monotone operators]\n\\label{c:L-rresol}\nLet $A\\colon X\\to X$ be $\\alpha$-monotone and Lipschitz continuous with constant $\\ell$. Also let $\\gamma\\in\\RPP$ and $\\lambda\\in\\RPP$ be such that\n\\begin{equation}\\label{e:cL-rresol-1}\n1+\\gamma\\alpha>0\n\\ \\text{ and }\\ \n\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\geq 0.\n\\end{equation}\nDefine $J:=J_{\\gamma A}$, $R:=(1-\\lambda)\\Id +\\lambda J$, and $Q :=\\Id-\\varepsilon R$ with $\\varepsilon\\in \\RP$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{c:L-rresol_relax} \n$R$ is Lipschitz continuous with constant \n\\begin{equation}\n\\sqrt{(\\lambda-1)^2-\\frac{\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda\\big)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}}.\n\\end{equation} \n","start":36977,"end":37738}]},{"id":"S063","semantic_type":"corollary","kind":"corollary","labels":["c:L-rresol","c:L-rresol_perturb","c:L-rresol_relax","e:cL-rresol-1"],"declared_external":false,"source_only":false,"title":"减去松弛预解式的常数","source_units":[{"raw_tex":"Let $A\\colon X\\to X$ be $\\alpha$-monotone and Lipschitz continuous with constant $\\ell$. Also let $\\gamma\\in\\RPP$ and $\\lambda\\in\\RPP$ be such that\n\\begin{equation}\\label{e:cL-rresol-1}\n1+\\gamma\\alpha>0\n\\ \\text{ and }\\ \n\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\geq 0.\n\\end{equation}\nDefine $J:=J_{\\gamma A}$, $R:=(1-\\lambda)\\Id +\\lambda J$, and $Q :=\\Id-\\varepsilon R$ with $\\varepsilon\\in \\RP$. Then the following hold:\n","start":37075,"end":37498},{"raw_tex":"\\item\\label{c:L-rresol_perturb} \n$Q$ is Lipschitz continuous with constant \n\\begin{equation}\n\\sqrt{(1+\\varepsilon(\\lambda-1))^2 -\\varepsilon\\lambda\\Big[2(1+\\gamma\\alpha) +\\varepsilon\\big(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\big)\\Big]\\alpha_J},\n\\end{equation}\nwhere $\\alpha_J$ is defined as \\eqref{e:alpha_J}.\n\\end{enumerate}\n\\end{corollary}","start":37738,"end":38084}]},{"id":"S064","semantic_type":"proof","kind":"proof","labels":["e:cL-rresol-2"],"declared_external":false,"source_only":false,"title":"利普希茨松弛推论的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{c:L-rresol_relax}: We observe that $R=\\nu\\Id+\\lambda J$ with $\\nu :=1-\\lambda$, that\n\\begin{equation}\n\\lambda(2\\nu(1+\\gamma\\alpha)+\\lambda) =-\\lambda\\Big(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\Big)\\leq 0,\n\\end{equation} \nand that, whenever $\\gamma\\ell <1$, \n\\begin{equation}\n\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}=\n(1-\\lambda)+\\frac{\\lambda}{1-\\gamma^2\\ell^2} >1-\\lambda+\\lambda =1 >0.\n\\end{equation}\nApplying Proposition~\\ref{p:cresol}\\ref{p:cresol_Lip} to $R=\\nu\\Id+\\lambda\\Id$ implies that $R$ is Lipschitz continuous with constant\n\\begin{equation}\n\\sqrt{(1-\\lambda)^2+\\frac{\\lambda\\big(2(1-\\lambda)(1+\\gamma\\alpha)+\\lambda\\big)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}},\n\\end{equation} \nwhich gives the claim.\n\n\\ref{c:L-rresol_perturb}: Using the first part of Proposition~\\ref{p:cresol}\\ref{p:cresol_Lip} and writing $Q =\\widetilde{\\nu}\\Id+\\widetilde{\\lambda}J$ with $\\widetilde{\\nu} :=1+\\varepsilon(\\lambda-1)$ and $\\widetilde{\\lambda} :=-\\varepsilon\\lambda$, it suffices to check that\n\\begin{equation}\\label{e:cL-rresol-2}\n\\widetilde{\\lambda}(2\\widetilde{\\nu}(1+\\gamma\\alpha)+\\widetilde{\\lambda})=(-\\varepsilon\\lambda)\\big( 2(1+\\varepsilon(\\lambda-1))(1+\\gamma\\alpha)+(-\\varepsilon\\lambda) \\big)\\leq 0.\n\\end{equation}\nIndeed, we have that $\\varepsilon\\geq 0$, $\\lambda >0$, and  \n\\begin{subequations}\n\\begin{align}\n2(1+\\varepsilon(\\lambda-1))(1+\\gamma\\alpha) -\\varepsilon\\lambda \n&=2(1+\\gamma\\alpha) +\\varepsilon\\big(2(\\lambda-1)(1+\\gamma\\alpha)-\\lambda\\big)\\\\\n&=2(1+\\gamma\\alpha) +\\varepsilon(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)) >0\n\\end{align}\n\\end{subequations}\nby \\eqref{e:cL-rresol-1}. So \\eqref{e:cL-rresol-2} holds and the conclusion follows.\n\\end{proof}","start":38085,"end":39773}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"验证松弛算子的改善条件","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{c:L-rresol_relax}: We observe that $R=\\nu\\Id+\\lambda J$ with $\\nu :=1-\\lambda$, that\n\\begin{equation}\n\\lambda(2\\nu(1+\\gamma\\alpha)+\\lambda) =-\\lambda\\Big(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\Big)\\leq 0,\n\\end{equation} \nand that, whenever $\\gamma\\ell <1$, \n\\begin{equation}\n\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}=\n(1-\\lambda)+\\frac{\\lambda}{1-\\gamma^2\\ell^2} >1-\\lambda+\\lambda =1 >0.\n\\end{equation}\nApplying Proposition~\\ref{p:cresol}\\ref{p:cresol_Lip} to $R=\\nu\\Id+\\lambda\\Id$ implies that $R$ is Lipschitz continuous with constant\n\\begin{equation}\n\\sqrt{(1-\\lambda)^2+\\frac{\\lambda\\big(2(1-\\lambda)(1+\\gamma\\alpha)+\\lambda\\big)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}},\n\\end{equation} \nwhich gives the claim.\n","start":38085,"end":38812}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":["e:cL-rresol-2"],"declared_external":false,"source_only":false,"title":"验证扰动组合的系数符号","source_units":[{"raw_tex":"\n\\ref{c:L-rresol_perturb}: Using the first part of Proposition~\\ref{p:cresol}\\ref{p:cresol_Lip} and writing $Q =\\widetilde{\\nu}\\Id+\\widetilde{\\lambda}J$ with $\\widetilde{\\nu} :=1+\\varepsilon(\\lambda-1)$ and $\\widetilde{\\lambda} :=-\\varepsilon\\lambda$, it suffices to check that\n\\begin{equation}\\label{e:cL-rresol-2}\n\\widetilde{\\lambda}(2\\widetilde{\\nu}(1+\\gamma\\alpha)+\\widetilde{\\lambda})=(-\\varepsilon\\lambda)\\big( 2(1+\\varepsilon(\\lambda-1))(1+\\gamma\\alpha)+(-\\varepsilon\\lambda) \\big)\\leq 0.\n\\end{equation}\nIndeed, we have that $\\varepsilon\\geq 0$, $\\lambda >0$, and  \n\\begin{subequations}\n\\begin{align}\n2(1+\\varepsilon(\\lambda-1))(1+\\gamma\\alpha) -\\varepsilon\\lambda \n&=2(1+\\gamma\\alpha) +\\varepsilon\\big(2(\\lambda-1)(1+\\gamma\\alpha)-\\lambda\\big)\\\\\n&=2(1+\\gamma\\alpha) +\\varepsilon(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)) >0\n\\end{align}\n\\end{subequations}\nby \\eqref{e:cL-rresol-1}. So \\eqref{e:cL-rresol-2} holds and the conclusion follows.\n\\end{proof}","start":38812,"end":39773}]},{"id":"S067","semantic_type":"assumption","kind":"assumption","labels":["e:parameters","e:parameters_a","e:parameters_b","e:parameters_c","l:domT"],"declared_external":false,"source_only":false,"title":"自适应参数约束","source_units":[{"raw_tex":"Then there exists $(\\gamma,\\delta,\\lambda,\\mu)\\in \\RPP^2\\times \\left]1,+\\infty\\right[^2$ such that\n\\begin{subequations}\n\\label{e:parameters}\n\\begin{align}\n&1+2\\gamma\\alpha >0,\n\\label{e:parameters_a}\\\\\n&\\mu\\in \\left[2-2\\gamma\\beta, 2+2\\gamma\\alpha\\right],\n\\label{e:parameters_b} \\\\\n&(\\lambda-1)(\\mu-1) =1, \\text{~and~} \\delta =(\\lambda-1)\\gamma. \\label{e:parameters_c}\n\\end{align}\n\\end{subequations}\n","start":54098,"end":54497}]},{"id":"S068","semantic_type":"lemma","kind":"lemma","labels":["e:parameters","e:parameters_a","e:parameters_b","e:parameters_c","l:domT"],"declared_external":false,"source_only":false,"title":"参数存在与 DR 满域性","source_units":[{"raw_tex":"\\begin{lemma}[single-valuedness and full domain of adaptive DR operator]\n\\label{l:domT}\nSuppose that $A$ and $B$ are maximally $\\alpha$- and $\\beta$-monotone with $\\alpha+\\beta\\geq 0$. \nThen there exists $(\\gamma,\\delta,\\lambda,\\mu)\\in \\RPP^2\\times \\left]1,+\\infty\\right[^2$ such that\n\\begin{subequations}\n\\label{e:parameters}\n\\begin{align}\n&1+2\\gamma\\alpha >0,\n\\label{e:parameters_a}\\\\\n&\\mu\\in \\left[2-2\\gamma\\beta, 2+2\\gamma\\alpha\\right],\n\\label{e:parameters_b} \\\\\n&(\\lambda-1)(\\mu-1) =1, \\text{~and~} \\delta =(\\lambda-1)\\gamma. \\label{e:parameters_c}\n\\end{align}\n\\end{subequations}\nMoreover, \\eqref{e:parameters} implies that $\\min\\{1+\\gamma\\alpha, 1+\\delta\\beta\\} >0$ and that $J_1$, $J_2$, and $T$ are single-valued and have full domain. \n\\end{lemma}","start":53912,"end":54667}]},{"id":"S069","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"参数与满域引理的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nTo show the existence, we first take $\\gamma>0$ such that $1/\\gamma >-2\\alpha$. Then $1+2\\gamma\\alpha >0$ and $2+2\\gamma\\alpha =1+(1+2\\gamma\\alpha) >1$.\nUsing $\\alpha+\\beta\\geq 0$, we derive that\n\\begin{equation}\n2+2\\gamma\\alpha =2\\gamma(\\alpha+\\beta) +(2-2\\gamma\\beta)\\geq 2-2\\gamma\\beta.\n\\end{equation}\nHence, we can always choose $\\mu >1$ satisfying \\eqref{e:parameters_b}. Next, with such $\\mu$, we define $\\lambda :=\\mu/(\\mu-1) =1+1/(\\mu-1) >1$ and $\\delta :=(\\lambda-1)\\gamma$. Then \\eqref{e:parameters_c} is clearly satisfied. \n\nNow, take any $(\\gamma,\\delta,\\lambda,\\mu)$ satisfying \\eqref{e:parameters}. We have\n\\begin{equation}\n1+\\delta\\beta =(\\lambda-1)(\\mu-1)+(\\lambda-1)\\gamma\\beta =\\frac{1}{2}(\\lambda-1)(\\mu + \\mu - (2-2\\gamma\\beta)) >0.\n\\end{equation}\nThus, $\\min\\{1+\\gamma\\alpha, 1+\\delta\\beta\\} >0$. The remaining conclusion follows from Proposition~\\ref{p:resol}. \n\\end{proof}","start":54668,"end":55577}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"构造可行参数","source_units":[{"raw_tex":"\\begin{proof}\nTo show the existence, we first take $\\gamma>0$ such that $1/\\gamma >-2\\alpha$. Then $1+2\\gamma\\alpha >0$ and $2+2\\gamma\\alpha =1+(1+2\\gamma\\alpha) >1$.\nUsing $\\alpha+\\beta\\geq 0$, we derive that\n\\begin{equation}\n2+2\\gamma\\alpha =2\\gamma(\\alpha+\\beta) +(2-2\\gamma\\beta)\\geq 2-2\\gamma\\beta.\n\\end{equation}\nHence, we can always choose $\\mu >1$ satisfying \\eqref{e:parameters_b}. Next, with such $\\mu$, we define $\\lambda :=\\mu/(\\mu-1) =1+1/(\\mu-1) >1$ and $\\delta :=(\\lambda-1)\\gamma$. Then \\eqref{e:parameters_c} is clearly satisfied. \n","start":54668,"end":55217}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"正系数保证单值满域","source_units":[{"raw_tex":"\nNow, take any $(\\gamma,\\delta,\\lambda,\\mu)$ satisfying \\eqref{e:parameters}. We have\n\\begin{equation}\n1+\\delta\\beta =(\\lambda-1)(\\mu-1)+(\\lambda-1)\\gamma\\beta =\\frac{1}{2}(\\lambda-1)(\\mu + \\mu - (2-2\\gamma\\beta)) >0.\n\\end{equation}\nThus, $\\min\\{1+\\gamma\\alpha, 1+\\delta\\beta\\} >0$. The remaining conclusion follows from Proposition~\\ref{p:resol}. \n\\end{proof}","start":55217,"end":55577}]},{"id":"S072","semantic_type":"proof","kind":"proof","labels":["e:180823a","e:domT'","e:rho1","e:rho2"],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nIn view of Remark~\\ref{r:Lip_mono}, assumption \\ref{t:linear_reduce} implies assumption \\ref{t:linear_full} because if $A$ is Lipschitz continuous with constant $\\ell$, then $A$ is also $\\alpha$-monotone with $\\alpha :=-\\ell$. It thus suffices to assume \\ref{t:linear_full}.  \nFirst, Proposition~\\ref{p:maxalpha}\\ref{p:maxalpha_cont} implies that $A$ is maximally $\\alpha$-monotone. Next, we learn from Lemma~\\ref{l:domT} that \n\\begin{equation}\n\\label{e:domT'}\n\\min\\{1+\\gamma\\alpha,1+\\delta\\beta\\} >0\n\\end{equation}\nand that all operators $J_1,J_2$, and $T$ are single-valued and have full domain. \n\nBy the choice of $\\mu$, it holds that $0 <\\mu-1 \\leq 1+2\\gamma\\alpha$, and so\n\\begin{equation}\n\\lambda =1+\\frac{1}{\\mu-1}\\geq 1+\\frac{1}{1+2\\gamma\\alpha} =\\frac{2(1+\\gamma\\alpha)}{1+2\\gamma\\alpha},\n\\end{equation} \nwhich yields\n\\begin{equation}\n\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\geq 0.\n\\end{equation}\nFrom $\\mu\\geq 2-2\\gamma\\beta$, we have that $\\varepsilon\\geq 0$ and $\\varepsilon_1\\geq 0$. It follows that $\\varphi\\geq 0$ and that\n\\begin{equation}\\label{e:180823a}\n\\varphi =0 \\iff \\varepsilon_1 =0 \\iff \\varepsilon =0 \\iff \\mu =2-2\\gamma\\beta.\n\\end{equation}\nDefine $Q_1 :=\\Id -\\varepsilon_1 R_1$. Using Corollary~\\ref{c:L-rresol} and noting that $\\lambda =\\mu(\\lambda-1)$, we derive that $R_1$ is Lipschitz continuous with constant  \n\\begin{equation}\n\\sqrt{(\\lambda-1)^2-\\frac{\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda\\big)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}} =(\\lambda-1)\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}},\n\\end{equation} \nand that $Q_1$ is Lipschitz continuous with constant \n\\begin{equation}\n\\label{e:rho1}\n\\rho_1 :=\\sqrt{(1+\\varepsilon_1(\\lambda-1))^2 -\\varphi\\alpha_J}\\leq 1+\\varepsilon_1(\\lambda-1),\n\\end{equation}\nwhere $\\alpha_J$ is defined as in \\eqref{e:alpha_J}. It follows from \\eqref{e:180823a} that the inequality is strict whenever $\\mu >2-2\\gamma\\beta$.\n\nNext, define $Q_2 :=R_2+\\varepsilon\\Id$. \nSince $\\gamma =(\\mu-1)\\delta$, we note that\n\\begin{equation}\n\\varepsilon =\\frac{\\mu-(2-2\\gamma\\beta)}{2(1+\\delta\\beta)} =\\frac{(\\mu-1)(2+2\\delta\\beta)-\\mu}{2(1+\\delta\\beta)} <\\mu-1,\n\\end{equation}\nwhich also gives\n\\begin{equation}\n(\\mu-1)(2+2\\delta\\beta)-\\mu-2\\varepsilon(1+\\delta\\beta) =0.\n\\end{equation}\nBy Corollary~\\ref{c:rresol}\\ref{c:rresol_perturb}, $Q_2$ is Lipschitz continuous with constant $(\\mu-1-\\varepsilon)$. Combining with the Lipschitz continuity of $R_1$ and noting that $(\\mu-1)(\\lambda-1) =1$, we have that $Q_2R_1$ is Lipschitz continuous with constant\n\\begin{subequations}\n\\label{e:rho2}\n\\begin{align}\n\\rho_2\n&:=(\\mu-1-\\varepsilon)(\\lambda-1)\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}}\\\\\n&\\phantom{:}=(1-\\varepsilon(\\lambda-1))\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}}\\\\\n&\\phantom{:}\\leq 1-\\varepsilon(\\lambda-1),\n\\end{align}\n\\end{subequations} \nwhere the inequality is strict whenever $\\mu <2+2\\gamma\\alpha$.  \n\nNow, we express\n\\begin{subequations}\n\\begin{align}\nT&=(1-\\kappa)\\Id -(1-\\kappa)\\varepsilon_1R_1 + \\kappa R_2R_1+\\kappa\\varepsilon R_1\\\\\n&=(1-\\kappa)(\\Id-\\varepsilon_1R_1)+\\kappa(R_2+\\varepsilon\\Id)R_1\\\\\n&=(1-\\kappa)Q_1+\\kappa Q_2R_1.\n\\end{align}\n\\end{subequations}\nWe note from $\\alpha+\\beta >0$ that $2+2\\gamma\\alpha >2-2\\gamma\\beta$, so at least one of two inequalities in \\eqref{e:rho1} and \\eqref{e:rho2} is strict.\nTherefore, $T$ is Lipschitz continuous with constant\n\\begin{equation}\n\\rho :=(1-\\kappa)\\rho_1+\\kappa \\rho_2 <(1-\\kappa)(1+\\varepsilon_1(\\lambda-1))+\\kappa(1-\\varepsilon(\\lambda-1))=1,\n\\end{equation}\nwhich completes the proof.\n\\end{proof}","start":63950,"end":67583}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":["e:rho2"],"declared_external":false,"source_only":false,"title":"归约无单调假设的分支","source_units":[{"raw_tex":"\\begin{proof}\nIn view of Remark~\\ref{r:Lip_mono}, assumption \\ref{t:linear_reduce} implies assumption \\ref{t:linear_full} because if $A$ is Lipschitz continuous with constant $\\ell$, then $A$ is also $\\alpha$-monotone with $\\alpha :=-\\ell$. It thus suffices to assume \\ref{t:linear_full}.  \n","start":63950,"end":64241}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":["e:domT'","e:rho2"],"declared_external":false,"source_only":false,"title":"极大性与全定义域","source_units":[{"raw_tex":"First, Proposition~\\ref{p:maxalpha}\\ref{p:maxalpha_cont} implies that $A$ is maximally $\\alpha$-monotone. Next, we learn from Lemma~\\ref{l:domT} that \n\\begin{equation}\n\\label{e:domT'}\n\\min\\{1+\\gamma\\alpha,1+\\delta\\beta\\} >0\n\\end{equation}\nand that all operators $J_1,J_2$, and $T$ are single-valued and have full domain. \n\n","start":64241,"end":64564}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":["e:rho2"],"declared_external":false,"source_only":false,"title":"推出松弛参数的符号条件","source_units":[{"raw_tex":"By the choice of $\\mu$, it holds that $0 <\\mu-1 \\leq 1+2\\gamma\\alpha$, and so\n\\begin{equation}\n\\lambda =1+\\frac{1}{\\mu-1}\\geq 1+\\frac{1}{1+2\\gamma\\alpha} =\\frac{2(1+\\gamma\\alpha)}{1+2\\gamma\\alpha},\n\\end{equation} \nwhich yields\n\\begin{equation}\n\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\geq 0.\n\\end{equation}\n","start":64564,"end":64873}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","labels":["e:180823a","e:rho2"],"declared_external":false,"source_only":false,"title":"扰动常数的非负与零点","source_units":[{"raw_tex":"From $\\mu\\geq 2-2\\gamma\\beta$, we have that $\\varepsilon\\geq 0$ and $\\varepsilon_1\\geq 0$. It follows that $\\varphi\\geq 0$ and that\n\\begin{equation}\\label{e:180823a}\n\\varphi =0 \\iff \\varepsilon_1 =0 \\iff \\varepsilon =0 \\iff \\mu =2-2\\gamma\\beta.\n\\end{equation}\n","start":64873,"end":65133}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","labels":["e:rho2"],"declared_external":false,"source_only":false,"title":"第一松弛算子的估计","source_units":[{"raw_tex":"Define $Q_1 :=\\Id -\\varepsilon_1 R_1$. Using Corollary~\\ref{c:L-rresol} and noting that $\\lambda =\\mu(\\lambda-1)$, we derive that $R_1$ is Lipschitz continuous with constant  \n\\begin{equation}\n\\sqrt{(\\lambda-1)^2-\\frac{\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda\\big)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}} =(\\lambda-1)\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}},\n\\end{equation} \n","start":65133,"end":65542}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","labels":["e:rho1","e:rho2"],"declared_external":false,"source_only":false,"title":"第一扰动算子的严格界","source_units":[{"raw_tex":"and that $Q_1$ is Lipschitz continuous with constant \n\\begin{equation}\n\\label{e:rho1}\n\\rho_1 :=\\sqrt{(1+\\varepsilon_1(\\lambda-1))^2 -\\varphi\\alpha_J}\\leq 1+\\varepsilon_1(\\lambda-1),\n\\end{equation}\nwhere $\\alpha_J$ is defined as in \\eqref{e:alpha_J}. It follows from \\eqref{e:180823a} that the inequality is strict whenever $\\mu >2-2\\gamma\\beta$.\n\n","start":65542,"end":65889}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","labels":["e:rho2"],"declared_external":false,"source_only":false,"title":"第二扰动算子的消项","source_units":[{"raw_tex":"Next, define $Q_2 :=R_2+\\varepsilon\\Id$. \nSince $\\gamma =(\\mu-1)\\delta$, we note that\n\\begin{equation}\n\\varepsilon =\\frac{\\mu-(2-2\\gamma\\beta)}{2(1+\\delta\\beta)} =\\frac{(\\mu-1)(2+2\\delta\\beta)-\\mu}{2(1+\\delta\\beta)} <\\mu-1,\n\\end{equation}\nwhich also gives\n\\begin{equation}\n(\\mu-1)(2+2\\delta\\beta)-\\mu-2\\varepsilon(1+\\delta\\beta) =0.\n\\end{equation}\nBy Corollary~\\ref{c:rresol}\\ref{c:rresol_perturb}, $Q_2$ is Lipschitz continuous with constant $(\\mu-1-\\varepsilon)$. Combining with the Lipschitz continuity of $R_1$ and noting that $(\\mu-1)(\\lambda-1) =1$, we have that $Q_2R_1$ is Lipschitz continuous with constant\n","start":65889,"end":66505}]},{"id":"S080","semantic_type":"proof_step","kind":"proof_step","labels":["e:rho2"],"declared_external":false,"source_only":false,"title":"复合算子的严格界","source_units":[{"raw_tex":"By Corollary~\\ref{c:rresol}\\ref{c:rresol_perturb}, $Q_2$ is Lipschitz continuous with constant $(\\mu-1-\\varepsilon)$. Combining with the Lipschitz continuity of $R_1$ and noting that $(\\mu-1)(\\lambda-1) =1$, we have that $Q_2R_1$ is Lipschitz continuous with constant\n\\begin{subequations}\n\\label{e:rho2}\n\\begin{align}\n\\rho_2\n&:=(\\mu-1-\\varepsilon)(\\lambda-1)\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}}\\\\\n&\\phantom{:}=(1-\\varepsilon(\\lambda-1))\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}}\\\\\n&\\phantom{:}\\leq 1-\\varepsilon(\\lambda-1),\n\\end{align}\n\\end{subequations} \nwhere the inequality is strict whenever $\\mu <2+2\\gamma\\alpha$.  \n\n","start":66237,"end":66926}]},{"id":"S081","semantic_type":"proof_step","kind":"proof_step","labels":["e:rho2"],"declared_external":false,"source_only":false,"title":"DR 算子的扰动分解","source_units":[{"raw_tex":"Now, we express\n\\begin{subequations}\n\\begin{align}\nT&=(1-\\kappa)\\Id -(1-\\kappa)\\varepsilon_1R_1 + \\kappa R_2R_1+\\kappa\\varepsilon R_1\\\\\n&=(1-\\kappa)(\\Id-\\varepsilon_1R_1)+\\kappa(R_2+\\varepsilon\\Id)R_1\\\\\n&=(1-\\kappa)Q_1+\\kappa Q_2R_1.\n\\end{align}\n\\end{subequations}\n","start":66926,"end":67191}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","labels":["e:rho2"],"declared_external":false,"source_only":false,"title":"加权常数严格小于一","source_units":[{"raw_tex":"We note from $\\alpha+\\beta >0$ that $2+2\\gamma\\alpha >2-2\\gamma\\beta$, so at least one of two inequalities in \\eqref{e:rho1} and \\eqref{e:rho2} is strict.\nTherefore, $T$ is Lipschitz continuous with constant\n\\begin{equation}\n\\rho :=(1-\\kappa)\\rho_1+\\kappa \\rho_2 <(1-\\kappa)(1+\\varepsilon_1(\\lambda-1))+\\kappa(1-\\varepsilon(\\lambda-1))=1,\n\\end{equation}\nwhich completes the proof.\n\\end{proof}","start":67191,"end":67583}]},{"id":"S083","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisite 列表：BC17, Theorem 21.1；原文第 237 行明确引用。"],"source_only":false,"title":"Minty 定理","source_units":[]},{"id":"S084","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisite 列表：BC17, Proposition 20.22；原文第 237 行明确引用。"],"source_only":false,"title":"正缩放保持极大单调性","source_units":[]},{"id":"S085","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisite 列表：BC17, Corollary 20.28；原文第 391 行明确引用。"],"source_only":false,"title":"连续满域单调算子的极大性","source_units":[]},{"id":"S086","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisite 列表中的标准 Hilbert 范数恒等式、三角不等式及 Cauchy–Schwarz；复合与加权和估计作为这些标准不等式的隐含应用，由标注者显式列出。"],"source_only":false,"title":"希尔伯特空间基本不等式","source_units":[]},{"id":"S087","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 prerequisite 列表明确列出的 Banach contraction principle on a complete Hilbert space；原文未具名引用。"],"source_only":false,"title":"Banach 收缩原理","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S021","edges":[{"source_id":"S001","target_id":"S005","kind":"supports"},{"source_id":"S002","target_id":"S005","kind":"supports"},{"source_id":"S002","target_id":"S006","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S003","target_id":"S009","kind":"supports"},{"source_id":"S086","target_id":"S009","kind":"supports"},{"source_id":"S009","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S010","kind":"supports"},{"source_id":"S006","target_id":"S010","kind":"supports"},{"source_id":"S008","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S006","target_id":"S011","kind":"supports"},{"source_id":"S005","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S019","kind":"supports"},{"source_id":"S067","target_id":"S018","kind":"supports"},{"source_id":"S012","target_id":"S018","kind":"supports"},{"source_id":"S046","target_id":"S020","kind":"supports"},{"source_id":"S005","target_id":"S023","kind":"supports"},{"source_id":"S083","target_id":"S023","kind":"supports"},{"source_id":"S084","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S006","target_id":"S027","kind":"supports"},{"source_id":"S002","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S006","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S005","target_id":"S032","kind":"supports"},{"source_id":"S006","target_id":"S032","kind":"supports"},{"source_id":"S086","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S037","kind":"supports"},{"source_id":"S024","target_id":"S038","kind":"supports"},{"source_id":"S025","target_id":"S038","kind":"supports"},{"source_id":"S022","target_id":"S038","kind":"supports"},{"source_id":"S005","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S022","target_id":"S042","kind":"supports"},{"source_id":"S035","target_id":"S042","kind":"supports"},{"source_id":"S024","target_id":"S043","kind":"supports"},{"source_id":"S025","target_id":"S043","kind":"supports"},{"source_id":"S085","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S043","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S039","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S003","target_id":"S048","kind":"supports"},{"source_id":"S005","target_id":"S048","kind":"supports"},{"source_id":"S086","target_id":"S048","kind":"supports"},{"source_id":"S034","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S030","target_id":"S049","kind":"supports"},{"source_id":"S006","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S011","target_id":"S050","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S049","target_id":"S047","kind":"supports"},{"source_id":"S050","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S046","target_id":"S052","kind":"supports"},{"source_id":"S086","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S030","target_id":"S055","kind":"supports"},{"source_id":"S034","target_id":"S056","kind":"supports"},{"source_id":"S054","target_id":"S056","kind":"supports"},{"source_id":"S030","target_id":"S056","kind":"supports"},{"source_id":"S045","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S054","target_id":"S057","kind":"supports"},{"source_id":"S044","target_id":"S057","kind":"supports"},{"source_id":"S045","target_id":"S057","kind":"supports"},{"source_id":"S046","target_id":"S058","kind":"supports"},{"source_id":"S011","target_id":"S058","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S055","target_id":"S053","kind":"supports"},{"source_id":"S056","target_id":"S053","kind":"supports"},{"source_id":"S057","target_id":"S053","kind":"supports"},{"source_id":"S058","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S051","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S059","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S052","target_id":"S065","kind":"supports"},{"source_id":"S052","target_id":"S066","kind":"supports"},{"source_id":"S046","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S062","kind":"supports"},{"source_id":"S064","target_id":"S063","kind":"supports"},{"source_id":"S067","target_id":"S070","kind":"supports"},{"source_id":"S067","target_id":"S071","kind":"supports"},{"source_id":"S034","target_id":"S071","kind":"supports"},{"source_id":"S035","target_id":"S071","kind":"supports"},{"source_id":"S013","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S069","kind":"supports"},{"source_id":"S071","target_id":"S069","kind":"supports"},{"source_id":"S069","target_id":"S068","kind":"supports"},{"source_id":"S017","target_id":"S073","kind":"supports"},{"source_id":"S010","target_id":"S073","kind":"supports"},{"source_id":"S017","target_id":"S074","kind":"supports"},{"source_id":"S018","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S040","target_id":"S074","kind":"supports"},{"source_id":"S068","target_id":"S074","kind":"supports"},{"source_id":"S018","target_id":"S075","kind":"supports"},{"source_id":"S067","target_id":"S075","kind":"supports"},{"source_id":"S020","target_id":"S076","kind":"supports"},{"source_id":"S012","target_id":"S076","kind":"supports"},{"source_id":"S067","target_id":"S076","kind":"supports"},{"source_id":"S018","target_id":"S076","kind":"supports"},{"source_id":"S074","target_id":"S076","kind":"supports"},{"source_id":"S075","target_id":"S076","kind":"supports"},{"source_id":"S062","target_id":"S077","kind":"supports"},{"source_id":"S073","target_id":"S077","kind":"supports"},{"source_id":"S017","target_id":"S077","kind":"supports"},{"source_id":"S074","target_id":"S077","kind":"supports"},{"source_id":"S075","target_id":"S077","kind":"supports"},{"source_id":"S018","target_id":"S077","kind":"supports"},{"source_id":"S067","target_id":"S077","kind":"supports"},{"source_id":"S063","target_id":"S078","kind":"supports"},{"source_id":"S020","target_id":"S078","kind":"supports"},{"source_id":"S046","target_id":"S078","kind":"supports"},{"source_id":"S074","target_id":"S078","kind":"supports"},{"source_id":"S075","target_id":"S078","kind":"supports"},{"source_id":"S076","target_id":"S078","kind":"supports"},{"source_id":"S060","target_id":"S079","kind":"supports"},{"source_id":"S020","target_id":"S079","kind":"supports"},{"source_id":"S074","target_id":"S079","kind":"supports"},{"source_id":"S018","target_id":"S079","kind":"supports"},{"source_id":"S067","target_id":"S079","kind":"supports"},{"source_id":"S079","target_id":"S080","kind":"supports"},{"source_id":"S077","target_id":"S080","kind":"supports"},{"source_id":"S018","target_id":"S080","kind":"supports"},{"source_id":"S067","target_id":"S080","kind":"supports"},{"source_id":"S086","target_id":"S080","kind":"supports"},{"source_id":"S013","target_id":"S081","kind":"supports"},{"source_id":"S020","target_id":"S081","kind":"supports"},{"source_id":"S078","target_id":"S082","kind":"supports"},{"source_id":"S080","target_id":"S082","kind":"supports"},{"source_id":"S081","target_id":"S082","kind":"supports"},{"source_id":"S073","target_id":"S082","kind":"supports"},{"source_id":"S017","target_id":"S082","kind":"supports"},{"source_id":"S012","target_id":"S082","kind":"supports"},{"source_id":"S020","target_id":"S082","kind":"supports"},{"source_id":"S086","target_id":"S082","kind":"supports"},{"source_id":"S073","target_id":"S072","kind":"supports"},{"source_id":"S074","target_id":"S072","kind":"supports"},{"source_id":"S075","target_id":"S072","kind":"supports"},{"source_id":"S076","target_id":"S072","kind":"supports"},{"source_id":"S077","target_id":"S072","kind":"supports"},{"source_id":"S078","target_id":"S072","kind":"supports"},{"source_id":"S079","target_id":"S072","kind":"supports"},{"source_id":"S080","target_id":"S072","kind":"supports"},{"source_id":"S081","target_id":"S072","kind":"supports"},{"source_id":"S082","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S021","kind":"supports"},{"source_id":"S001","target_id":"S021","kind":"supports"},{"source_id":"S074","target_id":"S021","kind":"supports"},{"source_id":"S082","target_id":"S021","kind":"supports"},{"source_id":"S015","target_id":"S021","kind":"supports"},{"source_id":"S016","target_id":"S021","kind":"supports"},{"source_id":"S087","target_id":"S021","kind":"supports"},{"source_id":"S013","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S021","edges":[{"source_id":"S001","target_id":"S005","kind":"supports"},{"source_id":"S002","target_id":"S005","kind":"supports"},{"source_id":"S002","target_id":"S006","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S003","target_id":"S009","kind":"supports"},{"source_id":"S086","target_id":"S009","kind":"supports"},{"source_id":"S009","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S010","kind":"supports"},{"source_id":"S006","target_id":"S010","kind":"supports"},{"source_id":"S008","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S006","target_id":"S011","kind":"supports"},{"source_id":"S005","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S019","kind":"supports"},{"source_id":"S067","target_id":"S018","kind":"supports"},{"source_id":"S012","target_id":"S018","kind":"supports"},{"source_id":"S046","target_id":"S020","kind":"supports"},{"source_id":"S005","target_id":"S022","kind":"supports"},{"source_id":"S083","target_id":"S022","kind":"supports"},{"source_id":"S084","target_id":"S022","kind":"supports"},{"source_id":"S006","target_id":"S027","kind":"supports"},{"source_id":"S002","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S006","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S005","target_id":"S032","kind":"supports"},{"source_id":"S006","target_id":"S032","kind":"supports"},{"source_id":"S086","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S037","kind":"supports"},{"source_id":"S024","target_id":"S038","kind":"supports"},{"source_id":"S025","target_id":"S038","kind":"supports"},{"source_id":"S022","target_id":"S038","kind":"supports"},{"source_id":"S005","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S022","target_id":"S042","kind":"supports"},{"source_id":"S035","target_id":"S042","kind":"supports"},{"source_id":"S024","target_id":"S043","kind":"supports"},{"source_id":"S025","target_id":"S043","kind":"supports"},{"source_id":"S085","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S043","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S039","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S003","target_id":"S048","kind":"supports"},{"source_id":"S005","target_id":"S048","kind":"supports"},{"source_id":"S086","target_id":"S048","kind":"supports"},{"source_id":"S034","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S030","target_id":"S049","kind":"supports"},{"source_id":"S006","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S011","target_id":"S050","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S049","target_id":"S047","kind":"supports"},{"source_id":"S050","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S046","target_id":"S052","kind":"supports"},{"source_id":"S086","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S030","target_id":"S055","kind":"supports"},{"source_id":"S034","target_id":"S056","kind":"supports"},{"source_id":"S054","target_id":"S056","kind":"supports"},{"source_id":"S030","target_id":"S056","kind":"supports"},{"source_id":"S045","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S054","target_id":"S057","kind":"supports"},{"source_id":"S044","target_id":"S057","kind":"supports"},{"source_id":"S045","target_id":"S057","kind":"supports"},{"source_id":"S046","target_id":"S058","kind":"supports"},{"source_id":"S011","target_id":"S058","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S055","target_id":"S053","kind":"supports"},{"source_id":"S056","target_id":"S053","kind":"supports"},{"source_id":"S057","target_id":"S053","kind":"supports"},{"source_id":"S058","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S051","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S059","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S052","target_id":"S065","kind":"supports"},{"source_id":"S052","target_id":"S066","kind":"supports"},{"source_id":"S046","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S062","kind":"supports"},{"source_id":"S064","target_id":"S063","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S034","target_id":"S068","kind":"supports"},{"source_id":"S035","target_id":"S068","kind":"supports"},{"source_id":"S013","target_id":"S068","kind":"supports"},{"source_id":"S017","target_id":"S073","kind":"supports"},{"source_id":"S010","target_id":"S073","kind":"supports"},{"source_id":"S017","target_id":"S074","kind":"supports"},{"source_id":"S018","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S040","target_id":"S074","kind":"supports"},{"source_id":"S068","target_id":"S074","kind":"supports"},{"source_id":"S018","target_id":"S075","kind":"supports"},{"source_id":"S067","target_id":"S075","kind":"supports"},{"source_id":"S020","target_id":"S076","kind":"supports"},{"source_id":"S012","target_id":"S076","kind":"supports"},{"source_id":"S067","target_id":"S076","kind":"supports"},{"source_id":"S018","target_id":"S076","kind":"supports"},{"source_id":"S074","target_id":"S076","kind":"supports"},{"source_id":"S075","target_id":"S076","kind":"supports"},{"source_id":"S062","target_id":"S077","kind":"supports"},{"source_id":"S073","target_id":"S077","kind":"supports"},{"source_id":"S017","target_id":"S077","kind":"supports"},{"source_id":"S074","target_id":"S077","kind":"supports"},{"source_id":"S075","target_id":"S077","kind":"supports"},{"source_id":"S018","target_id":"S077","kind":"supports"},{"source_id":"S067","target_id":"S077","kind":"supports"},{"source_id":"S063","target_id":"S078","kind":"supports"},{"source_id":"S020","target_id":"S078","kind":"supports"},{"source_id":"S046","target_id":"S078","kind":"supports"},{"source_id":"S074","target_id":"S078","kind":"supports"},{"source_id":"S075","target_id":"S078","kind":"supports"},{"source_id":"S076","target_id":"S078","kind":"supports"},{"source_id":"S060","target_id":"S079","kind":"supports"},{"source_id":"S020","target_id":"S079","kind":"supports"},{"source_id":"S074","target_id":"S079","kind":"supports"},{"source_id":"S018","target_id":"S079","kind":"supports"},{"source_id":"S067","target_id":"S079","kind":"supports"},{"source_id":"S079","target_id":"S080","kind":"supports"},{"source_id":"S077","target_id":"S080","kind":"supports"},{"source_id":"S018","target_id":"S080","kind":"supports"},{"source_id":"S067","target_id":"S080","kind":"supports"},{"source_id":"S086","target_id":"S080","kind":"supports"},{"source_id":"S013","target_id":"S081","kind":"supports"},{"source_id":"S020","target_id":"S081","kind":"supports"},{"source_id":"S078","target_id":"S082","kind":"supports"},{"source_id":"S080","target_id":"S082","kind":"supports"},{"source_id":"S081","target_id":"S082","kind":"supports"},{"source_id":"S073","target_id":"S082","kind":"supports"},{"source_id":"S017","target_id":"S082","kind":"supports"},{"source_id":"S012","target_id":"S082","kind":"supports"},{"source_id":"S020","target_id":"S082","kind":"supports"},{"source_id":"S086","target_id":"S082","kind":"supports"},{"source_id":"S073","target_id":"S072","kind":"supports"},{"source_id":"S074","target_id":"S072","kind":"supports"},{"source_id":"S075","target_id":"S072","kind":"supports"},{"source_id":"S076","target_id":"S072","kind":"supports"},{"source_id":"S077","target_id":"S072","kind":"supports"},{"source_id":"S078","target_id":"S072","kind":"supports"},{"source_id":"S079","target_id":"S072","kind":"supports"},{"source_id":"S080","target_id":"S072","kind":"supports"},{"source_id":"S081","target_id":"S072","kind":"supports"},{"source_id":"S082","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S021","kind":"supports"},{"source_id":"S001","target_id":"S021","kind":"supports"},{"source_id":"S074","target_id":"S021","kind":"supports"},{"source_id":"S082","target_id":"S021","kind":"supports"},{"source_id":"S015","target_id":"S021","kind":"supports"},{"source_id":"S016","target_id":"S021","kind":"supports"},{"source_id":"S087","target_id":"S021","kind":"supports"},{"source_id":"S013","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"实希尔伯特空间","source_units":[{"raw_tex":"Throughout this work, $X$ is a real Hilbert space with inner product $\\scal{\\cdot}{\\cdot}$ and induced norm $\\|\\cdot\\|$. \nThe set of nonnegative integers is denoted by $\\NN$, the set of real numbers by $\\RR$, the set of nonnegative real numbers by $\\RP := \\menge{x \\in \\RR}{x \\geq 0}$, and the set of the positive real numbers by $\\RPP := \\menge{x \\in \\RR}{x >0}$.\nWe use the notation $A\\colon X\\rightrightarrows X$ to indicate that $A$ is a set-valued operator on $X$ and the notation $A\\colon X\\to X$ to indicate that $A$ is a single-valued operator on $X$. \n\n","start":12234,"end":12796}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"算子集合记号","source_units":[{"raw_tex":"Let $A$ be an operator on $X$. The \\emph{domain} of $A$ is $\\dom A :=\\menge{x\\in X}{Ax\\neq \\varnothing}$, the \\emph{graph} of $A$ is $\\gra A :=\\menge{(x,u)\\in X\\times X}{u\\in Ax}$, and the set of \\emph{fixed points} of $A$ is $\\Fix A :=\\menge{x\\in X}{x\\in Ax}$. The \\emph{inverse} of $A$, denoted by $A^{-1}$, is the operator with graph $\\gra A^{-1} :=\\menge{(u,x)\\in X\\times X}{u\\in Ax}$.\n","start":12796,"end":13186}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"利普希茨连续性","source_units":[{"raw_tex":"We say that $A$ is \\emph{Lipschitz continuous} with constant $\\ell\\in \\RP$ if it is single-valued and\n\\begin{equation}\n\\forall x, y\\in \\dom A,\\quad \\|Ax-Ay\\|\\leq \\ell\\|x-y\\|.\n\\end{equation}\nThe operator $A$ is \\emph{nonexpansive} if it is Lipschitz continuous with constant $1$, i.e.,\n\\begin{equation}\n\\forall x, y\\in \\dom A,\\quad \\|Ax-Ay\\|\\leq \\|x-y\\|.\n\\end{equation}\n","start":13186,"end":13555}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"单调与极大单调","source_units":[{"raw_tex":"An operator $A\\colon X\\rightrightarrows X$ is said to be \\emph{monotone} if\n\\begin{equation}\n\\forall (x, u), (y, v)\\in \\gra A,\\quad \\scal{x-y}{u-v}\\geq 0,\n\\end{equation}\nand said to be \\emph{maximally monotone} if it is monotone and there exists no monotone operator $B\\colon X\\rightrightarrows X$ such that $\\gra B$ properly contains $\\gra A$. The \\emph{resolvent} of $A\\colon X\\rightrightarrows X$ is defined by\n","start":13555,"end":13969}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"预解式与松弛预解式","source_units":[{"raw_tex":"and said to be \\emph{maximally monotone} if it is monotone and there exists no monotone operator $B\\colon X\\rightrightarrows X$ such that $\\gra B$ properly contains $\\gra A$. The \\emph{resolvent} of $A\\colon X\\rightrightarrows X$ is defined by\n\\begin{equation}\nJ_A:= (\\Id+ A)^{-1},\n\\end{equation}\nwhere $\\Id$ is the identity operator. The \\emph{relaxed resolvent} of $A$ with parameter $\\lambda\\in\\RP$ is defined by\n\\begin{equation}\nJ_A^\\lambda:= (1-\\lambda)\\Id+ \\lambda J_A.\n\\end{equation}\n","start":13725,"end":14216}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["d:alpha_mono","e:alpha-mono"],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"广义单调性","source_units":[{"raw_tex":"\\begin{definition}[$\\alpha$-monotonicity]\n\\label{d:alpha_mono}\nAn operator $A\\colon X\\rightrightarrows X$ is said to be \\emph{$\\alpha$-monotone} ($\\alpha\\in \\RR$) if\n\\begin{equation}\n\\label{e:alpha-mono}\n\\forall (x,u), (y,v)\\in \\gra A,\\quad \\scal{x-y}{u-v}\\geq \\alpha\\|x-y\\|^2.\n\\end{equation}\nThe constant $\\alpha$ is referred to as the \\emph{monotonicity constant}.\nWe also say that $A$ is \\emph{maximally $\\alpha$-monotone} if it is $\\alpha$-monotone and there is no $\\alpha$-monotone operator whose graph strictly contains $\\gra A$.  \n\\end{definition}\n\n","start":15504,"end":16060}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","labels":["e:L-inequality","r:Lip_mono"],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"内积的双侧界","source_units":[{"raw_tex":"\\begin{remark}[Lipschitz $\\alpha$-monotone operators]\n\\label{r:Lip_mono}\nSuppose that $A$ is Lipschitz continuous with constant $\\ell$. Then $A$ is single-valued and \n\\begin{equation}\n\\forall x,y\\in\\dom A,\\quad\n|\\scal{x-y}{Ax-Ay}|\\leq \\|x-y\\|\\cdot\\|Ax-Ay\\|\\leq \\ell\\|x-y\\|^2,\n\\end{equation}\nwhich yields\n\\begin{equation}\n\\label{e:L-inequality}\n\\forall x,y\\in\\dom A,\\quad -\\ell\\|x-y\\|^2\\leq \\scal{x-y}{Ax-Ay}\\leq \\ell\\|x-y\\|^2.\n\\end{equation}\n","start":22888,"end":23330}]},{"id":"S009","semantic_type":"proof","kind":"proof","labels":["e:L-inequality","r:Lip_mono"],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"内积双侧界的行内证明","source_units":[{"raw_tex":"Suppose that $A$ is Lipschitz continuous with constant $\\ell$. Then $A$ is single-valued and \n\\begin{equation}\n\\forall x,y\\in\\dom A,\\quad\n|\\scal{x-y}{Ax-Ay}|\\leq \\|x-y\\|\\cdot\\|Ax-Ay\\|\\leq \\ell\\|x-y\\|^2,\n\\end{equation}\nwhich yields\n\\begin{equation}\n\\label{e:L-inequality}\n\\forall x,y\\in\\dom A,\\quad -\\ell\\|x-y\\|^2\\leq \\scal{x-y}{Ax-Ay}\\leq \\ell\\|x-y\\|^2.\n\\end{equation}\n","start":22961,"end":23330}]},{"id":"S010","semantic_type":"derived_fact","kind":"derived_fact","labels":["r:Lip_mono"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"自动弱单调性","source_units":[{"raw_tex":"We immediately deduce that $A$ is $(-\\ell)$-monotone. Now suppose, in addition, that $A$ is $\\alpha$-monotone. On the one hand, we can always assume without loss of generality that $\\alpha\\geq-\\ell$. On the other hand, it follows from the $\\alpha$-monotonicity and \\eqref{e:L-inequality} that $\\alpha\\leq \\ell$ as soon as $\\dom A$ has more than one element. Therefore, unless otherwise stated, whenever $A$ is both $\\alpha$-monotone and Lipschitz continuous with constant $\\ell$, we assume that $|\\alpha|\\leq\\ell$.\n","start":23330,"end":23845}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","labels":["r:Lip_mono"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"单调常数的默认约定","source_units":[{"raw_tex":"We immediately deduce that $A$ is $(-\\ell)$-monotone. Now suppose, in addition, that $A$ is $\\alpha$-monotone. On the one hand, we can always assume without loss of generality that $\\alpha\\geq-\\ell$. On the other hand, it follows from the $\\alpha$-monotonicity and \\eqref{e:L-inequality} that $\\alpha\\leq \\ell$ as soon as $\\dom A$ has more than one element. Therefore, unless otherwise stated, whenever $A$ is both $\\alpha$-monotone and Lipschitz continuous with constant $\\ell$, we assume that $|\\alpha|\\leq\\ell$.\n\\end{remark}","start":23330,"end":23857}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"全节算子与参数","source_units":[{"raw_tex":"Throughout this section, $A,B\\colon X\\rightrightarrows X$, $(\\gamma,\\delta,\\lambda,\\mu)\\in \\RPP^4$, and $\\kappa\\in \\left]0,1\\right[$. We define\n\\begin{subequations}\\label{e:JRT}\n","start":39838,"end":40016}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["e:aDR"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"自适应 DR 算子","source_units":[{"raw_tex":"\\begin{align}\nJ_1&:=J_{\\gamma A}=(\\Id+\\gamma A)^{-1}, \n&R_1&:=J_{\\gamma A}^\\lambda=(1-\\lambda)\\Id+\\lambda J_1,\\\\\nJ_2&:=J_{\\delta B}=(\\Id+\\delta B)^{-1},\n&R_2&:=J_{\\delta B}^\\mu=(1-\\mu)\\Id+\\mu J_2\n\\end{align}\nand consider the \\emph{adaptive DR operator} defined by\n\\begin{align}\\label{e:aDR}\nT:= (1-\\kappa)\\Id+ \\kappa R_2R_1.\n\\end{align}\n\\end{subequations}  \n","start":40016,"end":40374}]},{"id":"S014","semantic_type":"problem_setting","kind":"problem_setting","labels":["e:sumprob"],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"算子和的零点问题","source_units":[{"raw_tex":"The adaptive DR operator is indeed motivated by the problem of finding a zero of the sum of two operators, that is,\n\\begin{equation}\\label{e:sumprob}\n\\text{find $x\\in X$ such that}\\quad 0\\in Ax+Bx.\n\\end{equation}\nWe also denote by\n\\begin{equation}\n\\zer(A+B) :=(A+B)^{-1}(0) =\\menge{x\\in X}{0\\in Ax+Bx}\n\\end{equation}\n","start":41114,"end":41431}]},{"id":"S015","semantic_type":"algorithm","kind":"algorithm","labels":["e:dr_seq"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"DR 迭代序列","source_units":[{"raw_tex":"the set of solutions of problem \\eqref{e:sumprob}. Given a starting point $x_0\\in X$, the \\emph{adaptive DR algorithm} generates a sequence $(x_n)_\\nnn$, also called a \\emph{DR sequence}, by \n\\begin{equation}\\label{e:dr_seq}\n\\forall\\nnn,\\quad x_{n+1}\\in Tx_n.\n\\end{equation}\n","start":41431,"end":41706}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"逐步线性收敛率","source_units":[{"raw_tex":"Recall that a sequence $(x_n)_\\nnn$ converges to $\\overline{x}$ with \\emph{$Q$-linear} (or simply \\emph{linear}) rate $\\rho\\in\\left[0,1\\right[$ if\n\\begin{equation}\n\\forall\\nnn,\\quad \\|x_{n+1}-\\overline{x}\\|\\leq \\rho\\|x_n-\\overline{x}\\|.\n\\end{equation}\n","start":62303,"end":62555}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","labels":["t:linear","t:linear_full","t:linear_reduce"],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"主定理的两种算子条件","source_units":[{"raw_tex":"Suppose that either\n\\begin{enumerate}\n\\item\\label{t:linear_full} \n$A$ is $\\alpha$-monotone and Lipschitz continuous with constant $\\ell$, $B$ is maximally $\\beta$-monotone, and $\\alpha+\\beta >0$; or\n\\item\\label{t:linear_reduce} \n$A$ is Lipschitz continuous with constant $\\ell$, $B$ is maximally $\\beta$-monotone with $\\beta >\\ell$, and $\\alpha :=-\\ell$.\n\\end{enumerate}\n","start":62632,"end":63003}]},{"id":"S018","semantic_type":"assumption","kind":"assumption","labels":["t:linear","t:linear_full","t:linear_reduce"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"主定理的可行参数","source_units":[{"raw_tex":"Suppose also that $\\zer(A+B)\\neq \\varnothing$ and that $(\\gamma,\\delta,\\lambda,\\mu)\\in\\RPP^2\\times \\left]1,+\\infty\\right[^2$ satisfies \\eqref{e:parameters}.\n","start":63003,"end":63160}]},{"id":"S019","semantic_type":"assumption","kind":"assumption","labels":["t:linear","t:linear_full","t:linear_reduce"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"零点非空假设","source_units":[{"raw_tex":"Suppose also that $\\zer(A+B)\\neq \\varnothing$ and that $(\\gamma,\\delta,\\lambda,\\mu)\\in\\RPP^2\\times \\left]1,+\\infty\\right[^2$ satisfies \\eqref{e:parameters}.\n","start":63003,"end":63160}]},{"id":"S020","semantic_type":"notation","kind":"notation","labels":["t:linear","t:linear_full","t:linear_reduce"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"分解中的扰动常数","source_units":[{"raw_tex":"where\n\\begin{subequations}\n\\begin{align}\n&\\varepsilon :=\\frac{\\mu-(2-2\\gamma\\beta)}{2(1+\\delta\\beta)},\\ \\varepsilon_1:=\\frac{\\kappa\\varepsilon}{1-\\kappa},\\\\\n&\\varphi :=\\varepsilon_1\\lambda[2(1+\\gamma\\alpha)+\\varepsilon_1\\big(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\big)],\\\\\n&\\text{$\\alpha_J$ as in \\eqref{e:alpha_J}}.\n\\end{align}\n\\end{subequations}\n","start":63430,"end":63782}]},{"id":"S021","semantic_type":"theorem","kind":"theorem","labels":["t:linear","t:linear_full","t:linear_reduce"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"A 为利普希茨时的线性收敛","source_units":[{"raw_tex":"\\begin{theorem}[linear convergence when $A$ is Lipschitz]\n\\label{t:linear}\nSuppose that either\n\\begin{enumerate}\n\\item\\label{t:linear_full} \n$A$ is $\\alpha$-monotone and Lipschitz continuous with constant $\\ell$, $B$ is maximally $\\beta$-monotone, and $\\alpha+\\beta >0$; or\n\\item\\label{t:linear_reduce} \n$A$ is Lipschitz continuous with constant $\\ell$, $B$ is maximally $\\beta$-monotone with $\\beta >\\ell$, and $\\alpha :=-\\ell$.\n\\end{enumerate}\nSuppose also that $\\zer(A+B)\\neq \\varnothing$ and that $(\\gamma,\\delta,\\lambda,\\mu)\\in\\RPP^2\\times \\left]1,+\\infty\\right[^2$ satisfies \\eqref{e:parameters}.\nThen $T$ is Lipschitz continuous with constant\n\\begin{equation}\n\\rho :=(1-\\kappa)\n\\sqrt{\\big(1+\\varepsilon_1(\\lambda-1)\\big)^2-\\varphi\\alpha_J} +\\kappa(1-\\varepsilon(\\lambda-1))\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}} <1,\n\\end{equation} \nwhere\n\\begin{subequations}\n\\begin{align}\n&\\varepsilon :=\\frac{\\mu-(2-2\\gamma\\beta)}{2(1+\\delta\\beta)},\\ \\varepsilon_1:=\\frac{\\kappa\\varepsilon}{1-\\kappa},\\\\\n&\\varphi :=\\varepsilon_1\\lambda[2(1+\\gamma\\alpha)+\\varepsilon_1\\big(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\big)],\\\\\n&\\text{$\\alpha_J$ as in \\eqref{e:alpha_J}}.\n\\end{align}\n\\end{subequations}\nConsequently, every DR sequence $(x_n)_\\nnn$ generated by $T$ converges strongly to the unique fixed point $\\overline{x}$ of $T$ with linear rate $\\rho$.\n\\end{theorem}","start":62557,"end":63949}]},{"id":"S022","semantic_type":"lemma","kind":"lemma","labels":["f:Minty"],"member_ids":["S022","S023"],"declared_external":false,"source_only":false,"title":"Minty 的满域刻画","source_units":[{"raw_tex":"\\begin{fact}\n\\label{f:Minty}\nLet $A\\colon X\\rightrightarrows X$ be monotone and let $\\gamma\\in \\RPP$. Then $\\dom J_{\\gamma A} =X$ if and only if $A$ is maximally monotone.\n\\end{fact}","start":14295,"end":14477},{"raw_tex":"\\begin{proof}\nBy definition, $\\dom J_{\\gamma A} =\\ran(\\Id+\\gamma A) :=(\\Id+\\gamma A)(X)$. Since $\\gamma\\in \\RPP$, it holds that $\\gamma A$ is monotone. According to Minty's theorem (see, e.g., \\cite[Theorem~21.1]{BC17}), $\\dom J_{\\gamma A} =\\ran(\\Id+\\gamma A) =X$ if and only if $\\gamma A$ is maximally monotone. By \\cite[Proposition~20.22]{BC17}, the latter occurs if and only if $A$ is maximally monotone. \n\\end{proof}","start":14478,"end":14898}]},{"id":"S024","semantic_type":"lemma","kind":"lemma","labels":["l:transfer","l:transfer_maxmono","l:transfer_mono"],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"平移单调常数","source_units":[{"raw_tex":"\\begin{lemma}[monotonicity versus $\\alpha$-monotonicity]\n\\label{l:transfer}\nLet $A\\colon X\\rightrightarrows X$ and let $\\alpha, \\beta\\in \\RR$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{l:transfer_mono} \n$A$ is $\\alpha$-monotone if and only if $A-\\beta\\Id$ is $(\\alpha-\\beta)$-monotone.\n","start":16520,"end":16819}]},{"id":"S025","semantic_type":"lemma","kind":"lemma","labels":["l:transfer","l:transfer_maxmono","l:transfer_mono"],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"平移保持极大性","source_units":[{"raw_tex":"Let $A\\colon X\\rightrightarrows X$ and let $\\alpha, \\beta\\in \\RR$. Then the following hold:\n","start":16596,"end":16688},{"raw_tex":"\\item\\label{l:transfer_maxmono}\n$A$ is maximally $\\alpha$-monotone if and only if $A-\\beta\\Id$ is maximally $(\\alpha-\\beta)$-monotone. \n\\end{enumerate}\nConsequently, $A$ is (resp., maximally) $\\alpha$-monotone if and only if $A-\\alpha\\Id$ is (resp., maximally) monotone.\n\\end{lemma}","start":16819,"end":17101}]},{"id":"S026","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"平移引理的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{l:transfer_mono}: We first have the equivalences\n\\begin{subequations}\n\\begin{align}\n(x,u)\\in \\gra A &\\iff (x,u-\\beta x)\\in \\gra(A-\\beta\\Id), \\\\\n(y,v)\\in \\gra A &\\iff (y,v-\\beta y)\\in \\gra(A-\\beta\\Id),\n\\end{align}\n\\end{subequations}\nand\n\\begin{equation}\n\\scal{x-y}{u-v}\\geq \\alpha\\|x-y\\|^2 \\iff \\scal{x-y}{(u-\\beta x)-(v-\\beta y)}\\geq (\\alpha-\\beta)\\|x-y\\|^2,\n\\end{equation}\nfrom which the conclusion follows.\n\n\\ref{l:transfer_maxmono}: Assume that $A$ is maximally $\\alpha$-monotone. By \\ref{l:transfer_mono}, $A-\\beta\\Id$ is $(\\alpha-\\beta)$-monotone. Now, suppose that $A-\\beta\\Id$ is not maximally $(\\alpha-\\beta)$-monotone. Then there must exist $B'\\colon X\\rightrightarrows X$ such that $B'$ is $(\\alpha-\\beta)$-monotone and $\\gra(A-\\beta\\Id)\\subsetneq \\gra B'$. It follows that $B :=B'+\\beta\\Id$ is $\\alpha$-monotone due to \\ref{l:transfer_mono} and that $\\gra A\\subsetneq \\gra B$, which contradict the maximal $\\alpha$-monotonicity of $A$. We deduce that if $A$ is maximally $\\alpha$-monotone, then $A-\\beta\\Id$ is maximally $(\\alpha-\\beta)$-monotone. This also implies that if $A-\\beta\\Id$ is maximally $(\\alpha-\\beta)$-monotone, then $A =(A-\\beta\\Id)+\\beta\\Id$ is maximally $\\alpha$-monotone, and we are done. \n\\end{proof}","start":17102,"end":18352}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"图变换与内积等价","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{l:transfer_mono}: We first have the equivalences\n\\begin{subequations}\n\\begin{align}\n(x,u)\\in \\gra A &\\iff (x,u-\\beta x)\\in \\gra(A-\\beta\\Id), \\\\\n(y,v)\\in \\gra A &\\iff (y,v-\\beta y)\\in \\gra(A-\\beta\\Id),\n\\end{align}\n\\end{subequations}\nand\n\\begin{equation}\n\\scal{x-y}{u-v}\\geq \\alpha\\|x-y\\|^2 \\iff \\scal{x-y}{(u-\\beta x)-(v-\\beta y)}\\geq (\\alpha-\\beta)\\|x-y\\|^2,\n\\end{equation}\nfrom which the conclusion follows.\n","start":17102,"end":17530}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"严格图扩张的反证","source_units":[{"raw_tex":"\\ref{l:transfer_maxmono}: Assume that $A$ is maximally $\\alpha$-monotone. By \\ref{l:transfer_mono}, $A-\\beta\\Id$ is $(\\alpha-\\beta)$-monotone. Now, suppose that $A-\\beta\\Id$ is not maximally $(\\alpha-\\beta)$-monotone. Then there must exist $B'\\colon X\\rightrightarrows X$ such that $B'$ is $(\\alpha-\\beta)$-monotone and $\\gra(A-\\beta\\Id)\\subsetneq \\gra B'$. It follows that $B :=B'+\\beta\\Id$ is $\\alpha$-monotone due to \\ref{l:transfer_mono} and that $\\gra A\\subsetneq \\gra B$, which contradict the maximal $\\alpha$-monotonicity of $A$. We deduce that if $A$ is maximally $\\alpha$-monotone, then $A-\\beta\\Id$ is maximally $(\\alpha-\\beta)$-monotone. This also implies that if $A-\\beta\\Id$ is maximally $(\\alpha-\\beta)$-monotone, then $A =(A-\\beta\\Id)+\\beta\\Id$ is maximally $\\alpha$-monotone, and we are done. \n\\end{proof}","start":17531,"end":18352}]},{"id":"S029","semantic_type":"lemma","kind":"lemma","labels":["e:resol_gen","l:resol","l:resol_gen","l:resol_single"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"预解式的图上估计","source_units":[{"raw_tex":"\\begin{lemma}[resolvents of $\\alpha$-monotone operators]\n\\label{l:resol}\nLet $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{l:resol_gen}\nFor all $(x,a), (y,b)\\in \\gra J_{\\gamma A}$,\n\\begin{subequations}\n\\begin{align}\n\\label{e:resol_gen}\n\\scal{x-y}{a-b} &\\geq (1+\\gamma\\alpha)\\|a-b\\|^2 \\quad\\text{and}\\\\\n\\|x-y\\| &\\geq (1+\\gamma\\alpha)\\|a-b\\|.\n\\end{align} \n\\end{subequations}\n","start":18354,"end":18814}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","labels":["l:resol","l:resol_gen","l:resol_single"],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"单值预解式的余强制界","source_units":[{"raw_tex":"Let $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$. Then the following hold:\n","start":18427,"end":18534},{"raw_tex":"\\item\\label{l:resol_single}\nIf $J_{\\gamma A}$ is single-valued, then, for all $x,y\\in \\dom J_{\\gamma A}$,\n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq (1+\\gamma\\alpha)\\|J_{\\gamma A}x-J_{\\gamma A}y\\|^2,\n\\end{equation}\ni.e., $J_{\\gamma A}$ is $(1+\\gamma\\alpha)$-cocoercive.\n\\end{enumerate}\n\\end{lemma}","start":18814,"end":19129}]},{"id":"S031","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"预解式基本估计的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{l:resol_gen}: Let $(x,a), (y,b)\\in \\gra J_{\\gamma A}$. Then $x\\in (\\Id+\\gamma A)a$, $y\\in (\\Id+\\gamma A)b$, and so $x =a+\\gamma u$, $y =b+\\gamma v$ for some $u\\in Aa$, $v\\in Ab$. We derive from the $\\alpha$-monotonicity of $A$ that \n\\begin{subequations}\n\\begin{align}\n\\scal{x-y}{a-b}\n&=\\scal{(a+\\gamma u)-(b+\\gamma v)}{a-b}\\\\\n&=\\|a-b\\|^2+\\gamma\\scal{a-b}{u-v}\\\\\n&\\geq \\|a-b\\|^2+\\gamma\\alpha\\|a-b\\|^2\\\\\n&=(1+\\gamma\\alpha)\\|a-b\\|^2.\n\\end{align}\n\\end{subequations}\nNow, by the Cauchy--Schwarz inequality,\n\\begin{equation}\n\\|x-y\\|\\|a-b\\|\\geq \\scal{x-y}{a-b}\\geq (1+\\gamma\\alpha)\\|a-b\\|^2.\n\\end{equation}\nwhich gives $\\|x-y\\|\\geq (1+\\gamma\\alpha)\\|a-b\\|$ while noting that this is trivial when $a =b$. \n\n\\ref{l:resol_single}: This is a direct consequence of \\ref{l:resol_gen}.\n\\end{proof}","start":19130,"end":19932}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"图上展开及距离界","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{l:resol_gen}: Let $(x,a), (y,b)\\in \\gra J_{\\gamma A}$. Then $x\\in (\\Id+\\gamma A)a$, $y\\in (\\Id+\\gamma A)b$, and so $x =a+\\gamma u$, $y =b+\\gamma v$ for some $u\\in Aa$, $v\\in Ab$. We derive from the $\\alpha$-monotonicity of $A$ that \n\\begin{subequations}\n\\begin{align}\n\\scal{x-y}{a-b}\n&=\\scal{(a+\\gamma u)-(b+\\gamma v)}{a-b}\\\\\n&=\\|a-b\\|^2+\\gamma\\scal{a-b}{u-v}\\\\\n&\\geq \\|a-b\\|^2+\\gamma\\alpha\\|a-b\\|^2\\\\\n&=(1+\\gamma\\alpha)\\|a-b\\|^2.\n\\end{align}\n\\end{subequations}\nNow, by the Cauchy--Schwarz inequality,\n\\begin{equation}\n\\|x-y\\|\\|a-b\\|\\geq \\scal{x-y}{a-b}\\geq (1+\\gamma\\alpha)\\|a-b\\|^2.\n\\end{equation}\nwhich gives $\\|x-y\\|\\geq (1+\\gamma\\alpha)\\|a-b\\|$ while noting that this is trivial when $a =b$. \n","start":19130,"end":19847}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"专门化到单值预解式","source_units":[{"raw_tex":"\\ref{l:resol_single}: This is a direct consequence of \\ref{l:resol_gen}.\n\\end{proof}","start":19848,"end":19932}]},{"id":"S034","semantic_type":"proposition","kind":"proposition","labels":["p:resol","p:resol_dom","p:resol_single"],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"预解式的单值性","source_units":[{"raw_tex":"\\begin{proposition}[single-valuedness and full domain]\n\\label{p:resol}\nLet $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$ such that $1+\\gamma\\alpha >0$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{p:resol_single} \n$J_{\\gamma A}$ is single-valued.\n","start":19935,"end":20223}]},{"id":"S035","semantic_type":"proposition","kind":"proposition","labels":["p:resol","p:resol_dom","p:resol_single"],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"广义单调预解式的满域刻画","source_units":[{"raw_tex":"Let $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$ such that $1+\\gamma\\alpha >0$. Then the following hold:\n","start":20006,"end":20143},{"raw_tex":"\\item\\label{p:resol_dom} \n$\\dom J_{\\gamma A} =X$ if and only if $A$ is maximally $\\alpha$-monotone.\n\\end{enumerate}\n\\end{proposition}","start":20223,"end":20356}]},{"id":"S036","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"单值与满域的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{p:resol_single}: This follows from Lemma~\\ref{l:resol}\\ref{l:resol_gen}.\n\n\\ref{p:resol_dom}: By Lemma~\\ref{l:transfer}\\ref{l:transfer_mono}, $A':=A-\\alpha\\Id$ is monotone.\nNoting that $(\\beta T)^{-1} =T^{-1}\\circ \\frac{1}{\\beta}\\Id$ for any operator $T$ and any $\\beta\\in \\RR\\smallsetminus\\{0\\}$, we have\n\\begin{subequations}\n\\begin{align}\nJ_{\\gamma A}&=(\\Id+\\gamma A)^{-1}=\\big((1+\\gamma\\alpha)\\Id+\\gamma(A-\\alpha\\Id)\\big)^{-1}\\\\\n&=\\left(\\Id+\\frac{\\gamma}{1+\\gamma\\alpha} A'\\right)^{-1} \\circ \\left(\\frac{1}{1+\\gamma\\alpha}\\Id\\right)\\\\\n&=J_{\\frac{\\gamma}{1+\\gamma\\alpha}A'}\\circ \\left(\\frac{1}{1+\\gamma\\alpha}\\Id\\right).\n\\end{align}\n\\end{subequations}\nIt follows that \n\\begin{subequations}\n\\begin{align}\n\\dom J_{\\gamma A} =X & \\iff \\dom J_{\\frac{\\gamma}{1+\\gamma\\alpha}A'} =X &&\\\\\n& \\iff \\text{$A'$ is maximally monotone} &&\\hspace*{-.8in}\\text{(by Fact~\\ref{f:Minty})} \\\\\n& \\iff \\text{$A$ is maximally $\\alpha$-monotone} &&\\hspace*{-.8in}\\text{(by Lemma~\\ref{l:transfer}\\ref{l:transfer_maxmono})}.\n\\end{align}\n\\end{subequations}\nThe proof is complete.\n\\end{proof}","start":20357,"end":21441}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"正系数迫使单值","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{p:resol_single}: This follows from Lemma~\\ref{l:resol}\\ref{l:resol_gen}.\n","start":20357,"end":20449}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"平移缩放后应用 Minty","source_units":[{"raw_tex":"\n\\ref{p:resol_dom}: By Lemma~\\ref{l:transfer}\\ref{l:transfer_mono}, $A':=A-\\alpha\\Id$ is monotone.\nNoting that $(\\beta T)^{-1} =T^{-1}\\circ \\frac{1}{\\beta}\\Id$ for any operator $T$ and any $\\beta\\in \\RR\\smallsetminus\\{0\\}$, we have\n\\begin{subequations}\n\\begin{align}\nJ_{\\gamma A}&=(\\Id+\\gamma A)^{-1}=\\big((1+\\gamma\\alpha)\\Id+\\gamma(A-\\alpha\\Id)\\big)^{-1}\\\\\n&=\\left(\\Id+\\frac{\\gamma}{1+\\gamma\\alpha} A'\\right)^{-1} \\circ \\left(\\frac{1}{1+\\gamma\\alpha}\\Id\\right)\\\\\n&=J_{\\frac{\\gamma}{1+\\gamma\\alpha}A'}\\circ \\left(\\frac{1}{1+\\gamma\\alpha}\\Id\\right).\n\\end{align}\n\\end{subequations}\nIt follows that \n\\begin{subequations}\n\\begin{align}\n\\dom J_{\\gamma A} =X & \\iff \\dom J_{\\frac{\\gamma}{1+\\gamma\\alpha}A'} =X &&\\\\\n& \\iff \\text{$A'$ is maximally monotone} &&\\hspace*{-.8in}\\text{(by Fact~\\ref{f:Minty})} \\\\\n& \\iff \\text{$A$ is maximally $\\alpha$-monotone} &&\\hspace*{-.8in}\\text{(by Lemma~\\ref{l:transfer}\\ref{l:transfer_maxmono})}.\n\\end{align}\n\\end{subequations}\nThe proof is complete.\n\\end{proof}","start":20449,"end":21441}]},{"id":"S039","semantic_type":"proposition","kind":"proposition","labels":["p:maxalpha","p:maxalpha_cont","p:maxalpha_mono"],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"非负常数下的极大性","source_units":[{"raw_tex":"\\begin{proposition}[maximal $\\alpha$-monotonicity]\n\\label{p:maxalpha}\nThe following statements hold: \n\\begin{enumerate}\n\\item\\label{p:maxalpha_mono} \nLet $A\\colon X\\rightrightarrows X$ and $\\alpha\\in \\RP$. Then $A$ is maximally $\\alpha$-monotone if and only if $A$ is $\\alpha$-monotone and maximally monotone.\n","start":21510,"end":21820}]},{"id":"S040","semantic_type":"proposition","kind":"proposition","labels":["p:maxalpha","p:maxalpha_cont","p:maxalpha_mono"],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"连续满域推出极大性","source_units":[{"raw_tex":"The following statements hold: \n","start":21580,"end":21612},{"raw_tex":"\\item\\label{p:maxalpha_cont} \nLet $A\\colon X\\to X$ and $\\alpha\\in \\RR$. Then $A$ is maximally $\\alpha$-monotone if $A$ is $\\alpha$-monotone and continuous with full domain.\n\\end{enumerate}\n\\end{proposition}","start":21820,"end":22026}]},{"id":"S041","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"两种极大性判据的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{p:maxalpha_mono}: Since $\\alpha\\geq 0$, it follows from Fact~\\ref{f:Minty} that $A$ is $\\alpha$-monotone and maximally monotone if and only if $A$ is $\\alpha$-monotone and $\\dom J_A =X$, which, by Proposition~\\ref{p:resol}\\ref{p:resol_dom}, happen if and only if $A$ is maximally $\\alpha$-monotone.\n\n\\ref{p:maxalpha_cont}: Set $A' :=A-\\alpha\\Id$. Then $A'$ is monotone (due to Lemma~\\ref{l:transfer}\\ref{l:transfer_mono}) and continuous with full domain. By \\cite[Corollary~20.28]{BC17}, $A'$ is maximally monotone, and by Lemma~\\ref{l:transfer}\\ref{l:transfer_maxmono}, $A$ is maximally $\\alpha$-monotone. \n\\end{proof}","start":22027,"end":22665}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"非负常数的满域等价","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{p:maxalpha_mono}: Since $\\alpha\\geq 0$, it follows from Fact~\\ref{f:Minty} that $A$ is $\\alpha$-monotone and maximally monotone if and only if $A$ is $\\alpha$-monotone and $\\dom J_A =X$, which, by Proposition~\\ref{p:resol}\\ref{p:resol_dom}, happen if and only if $A$ is maximally $\\alpha$-monotone.\n","start":22027,"end":22345}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"连续满域算子的平移","source_units":[{"raw_tex":"\n\\ref{p:maxalpha_cont}: Set $A' :=A-\\alpha\\Id$. Then $A'$ is monotone (due to Lemma~\\ref{l:transfer}\\ref{l:transfer_mono}) and continuous with full domain. By \\cite[Corollary~20.28]{BC17}, $A'$ is maximally monotone, and by Lemma~\\ref{l:transfer}\\ref{l:transfer_maxmono}, $A$ is maximally $\\alpha$-monotone. \n\\end{proof}","start":22345,"end":22665}]},{"id":"S044","semantic_type":"lemma","kind":"lemma","labels":["e:L-resol_only","e:L-resol_only1","e:alpha_J","e:expan","l:L-resol","l:L-resol_mono","l:L-resol_only"],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"利普希茨预解式的基础下界","source_units":[{"raw_tex":"\\begin{lemma}[resolvents of Lipschitz $\\alpha$-monotone operators]\n\\label{l:L-resol}\nLet $A\\colon X\\to X$ be Lipschitz continuous with constant $\\ell$ and let $\\gamma\\in \\RPP$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{l:L-resol_only} \nFor all $(x,a), (y,b)\\in \\gra J_{\\gamma A}$,\n\\begin{subequations}\n\\begin{align}\n\\label{e:expan}\n\\|a-b\\|&\\geq \\frac{1}{1+\\gamma\\ell}\\|x-y\\|,\\\\\n\\label{e:L-resol_only}\n\\scal{x-y}{a-b}&\\geq \\frac{1}{2}\\|x-y\\|^2 +\\frac{1}{2}(1-\\gamma^2\\ell^2)\\|a-b\\|^2,\n\\end{align}\n\\end{subequations}\nand if $\\gamma\\ell\\leq 1$, then\n\\begin{equation}\\label{e:L-resol_only1}\n\\scal{x-y}{a-b}\\geq \\frac{1}{1+\\gamma\\ell}\\|x-y\\|^2.\n\\end{equation}\n","start":24136,"end":24804}]},{"id":"S045","semantic_type":"lemma","kind":"lemma","labels":["e:alpha_J","l:L-resol","l:L-resol_mono","l:L-resol_only"],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"单调利普希茨预解式的增强界","source_units":[{"raw_tex":"Let $A\\colon X\\to X$ be Lipschitz continuous with constant $\\ell$ and let $\\gamma\\in \\RPP$. Then the following hold:\n","start":24221,"end":24338},{"raw_tex":"\\item\\label{l:L-resol_mono}\nIf $A$ is $\\alpha$-monotone with $1+\\gamma\\alpha >0$, then, for all $x, y\\in \\dom J_{\\gamma A}$, \n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq (1+\\gamma\\alpha)\\alpha_J\\|x-y\\|^2,\n\\end{equation}\nwhere \n\\begin{subequations}\n\\label{e:alpha_J}\n\\begin{equation}\n\\alpha_J \n:=\\begin{cases}\n\\dfrac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2} &\\text{if~} \\gamma\\ell\\geq 1 \n,\\\\\n\\dfrac{1}{(1+\\gamma\\alpha)(1+\\gamma\\ell)} &\\text{if~} \\gamma\\ell\\leq 1;\n\\end{cases}\n\\end{equation}\nand if additionally $A$ satisfies \\eqref{e:alpha-mono} with equality, then \n\\begin{equation}\n\\alpha_J :=\\frac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}. \n\\end{equation}\n\\end{subequations}\n\\end{enumerate}\n\\end{lemma}","start":24804,"end":25513}]},{"id":"S046","semantic_type":"definition","kind":"definition","labels":["e:alpha_J","l:L-resol","l:L-resol_mono","l:L-resol_only"],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"增强界中的分段常数","source_units":[{"raw_tex":"where \n\\begin{subequations}\n\\label{e:alpha_J}\n\\begin{equation}\n\\alpha_J \n:=\\begin{cases}\n\\dfrac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2} &\\text{if~} \\gamma\\ell\\geq 1 \n,\\\\\n\\dfrac{1}{(1+\\gamma\\alpha)(1+\\gamma\\ell)} &\\text{if~} \\gamma\\ell\\leq 1;\n\\end{cases}\n\\end{equation}\nand if additionally $A$ satisfies \\eqref{e:alpha-mono} with equality, then \n\\begin{equation}\n\\alpha_J :=\\frac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}. \n\\end{equation}\n\\end{subequations}\n","start":25041,"end":25486}]},{"id":"S047","semantic_type":"proof","kind":"proof","labels":["e:L-resol","e:L-resol'","e:L-resol''"],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"增强预解式估计的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{l:L-resol_only}: Let $(x,a), (y,b)\\in \\gra J_{\\gamma A}$. Then $x =a+\\gamma Aa$ and $y =b+\\gamma Ab$. By Lipschitz continuity, $\\|Aa-Ab\\|\\leq \\ell\\|a-b\\|$. It follows that\n\\begin{equation}\n\\|x-y\\|\\leq \\|a-b\\| +\\gamma\\|Aa-Ab\\|\\leq (1+\\gamma\\ell)\\|a-b\\|\n\\end{equation}\nand that\n\\begin{subequations}\n\\label{e:L-resol}\n\\begin{align}\n2\\scal{x-y}{a-b} &=\\|x-y\\|^2 +\\|a-b\\|^2 -\\|(x-a)-(y-b)\\|^2\\\\\n&=\\|x-y\\|^2 +\\|a-b\\|^2 -\\gamma^2\\|Aa-Ab\\|^2\\\\\n&\\geq \\|x-y\\|^2 +(1-\\gamma^2\\ell^2)\\|a-b\\|^2.\n\\end{align}\n\\end{subequations}\nIf $\\gamma\\ell\\leq 1$, then combining the above inequalities yields\n\\begin{equation}\n2\\scal{x-y}{a-b}\\geq \\|x-y\\|^2 +\\frac{1-\\gamma^2\\ell^2}{(1+\\gamma\\ell)^2}\\|x-y\\|^2 =\\frac{2}{1+\\gamma\\ell}\\|x-y\\|^2,\n\\end{equation}\nand we get the claim.\n\n\\ref{l:L-resol_mono}: We first note that $J_{\\gamma A}$ is single-valued due to Proposition~\\ref{p:resol}\\ref{p:resol_single}. Then \\eqref{e:L-resol} reads as\n\\begin{equation}\n\\label{e:L-resol'}\n2\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\|x-y\\|^2 +(1-\\gamma^2\\ell^2)\\|J_{\\gamma A}x-J_{\\gamma A}y\\|^2.\n\\end{equation}\nWe claim that if $\\gamma\\ell\\geq 1$ or $A$ satisfies \\eqref{e:alpha-mono} with equality, then \n\\begin{equation}\n\\label{e:L-resol''}\n2\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\|x-y\\|^2 +\\frac{1-\\gamma^2\\ell^2}{1+\\gamma\\alpha}\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}.\n\\end{equation} \nIndeed, the former case implies $1-\\gamma^2\\ell^2\\leq 0$ and, by combining \\eqref{e:L-resol'} with Lemma~\\ref{l:resol}\\ref{l:resol_single} and noting that $1+\\gamma\\alpha >0$, we get \\eqref{e:L-resol''}.  \nIn the latter case, Lemma~\\ref{l:resol}\\ref{l:resol_single} reduces to\n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y} =(1+\\gamma\\alpha)\\|J_{\\gamma A}x-J_{\\gamma A}y\\|^2.\n\\end{equation} \nSubstituting this into \\eqref{e:L-resol'}, we also obtain \\eqref{e:L-resol''}. \n\nNow, in view of Remark~\\ref{r:Lip_mono}, $1+2\\gamma\\alpha+\\gamma^2\\ell^2\\geq 1+2\\gamma\\alpha+\\gamma^2\\alpha^2 =(1+\\gamma\\alpha)^2 >0$. It thus follows from \\eqref{e:L-resol''} that \n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\frac{1+\\gamma\\alpha}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}\\|x-y\\|^2.\n\\end{equation}\nFinally, if $\\gamma\\ell\\leq 1$, then, by \\ref{l:L-resol_only}, \n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\frac{1}{1+\\gamma\\ell}\\|x-y\\|^2,\n\\end{equation}\nand the conclusion follows.\n\\end{proof}","start":25514,"end":27906}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":["e:L-resol"],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"三角界与极化展开","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{l:L-resol_only}: Let $(x,a), (y,b)\\in \\gra J_{\\gamma A}$. Then $x =a+\\gamma Aa$ and $y =b+\\gamma Ab$. By Lipschitz continuity, $\\|Aa-Ab\\|\\leq \\ell\\|a-b\\|$. It follows that\n\\begin{equation}\n\\|x-y\\|\\leq \\|a-b\\| +\\gamma\\|Aa-Ab\\|\\leq (1+\\gamma\\ell)\\|a-b\\|\n\\end{equation}\nand that\n\\begin{subequations}\n\\label{e:L-resol}\n\\begin{align}\n2\\scal{x-y}{a-b} &=\\|x-y\\|^2 +\\|a-b\\|^2 -\\|(x-a)-(y-b)\\|^2\\\\\n&=\\|x-y\\|^2 +\\|a-b\\|^2 -\\gamma^2\\|Aa-Ab\\|^2\\\\\n&\\geq \\|x-y\\|^2 +(1-\\gamma^2\\ell^2)\\|a-b\\|^2.\n\\end{align}\n\\end{subequations}\nIf $\\gamma\\ell\\leq 1$, then combining the above inequalities yields\n\\begin{equation}\n2\\scal{x-y}{a-b}\\geq \\|x-y\\|^2 +\\frac{1-\\gamma^2\\ell^2}{(1+\\gamma\\ell)^2}\\|x-y\\|^2 =\\frac{2}{1+\\gamma\\ell}\\|x-y\\|^2,\n\\end{equation}\nand we get the claim.\n","start":25514,"end":26285}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["e:L-resol","e:L-resol'","e:L-resol''"],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"按系数符号替换范数项","source_units":[{"raw_tex":"\n\\ref{l:L-resol_mono}: We first note that $J_{\\gamma A}$ is single-valued due to Proposition~\\ref{p:resol}\\ref{p:resol_single}. Then \\eqref{e:L-resol} reads as\n\\begin{equation}\n\\label{e:L-resol'}\n2\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\|x-y\\|^2 +(1-\\gamma^2\\ell^2)\\|J_{\\gamma A}x-J_{\\gamma A}y\\|^2.\n\\end{equation}\nWe claim that if $\\gamma\\ell\\geq 1$ or $A$ satisfies \\eqref{e:alpha-mono} with equality, then \n\\begin{equation}\n\\label{e:L-resol''}\n2\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\|x-y\\|^2 +\\frac{1-\\gamma^2\\ell^2}{1+\\gamma\\alpha}\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}.\n\\end{equation} \nIndeed, the former case implies $1-\\gamma^2\\ell^2\\leq 0$ and, by combining \\eqref{e:L-resol'} with Lemma~\\ref{l:resol}\\ref{l:resol_single} and noting that $1+\\gamma\\alpha >0$, we get \\eqref{e:L-resol''}.  \nIn the latter case, Lemma~\\ref{l:resol}\\ref{l:resol_single} reduces to\n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y} =(1+\\gamma\\alpha)\\|J_{\\gamma A}x-J_{\\gamma A}y\\|^2.\n\\end{equation} \nSubstituting this into \\eqref{e:L-resol'}, we also obtain \\eqref{e:L-resol''}. \n","start":26285,"end":27371}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":["e:L-resol"],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"分母正性与分段结论","source_units":[{"raw_tex":"\nNow, in view of Remark~\\ref{r:Lip_mono}, $1+2\\gamma\\alpha+\\gamma^2\\ell^2\\geq 1+2\\gamma\\alpha+\\gamma^2\\alpha^2 =(1+\\gamma\\alpha)^2 >0$. It thus follows from \\eqref{e:L-resol''} that \n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\frac{1+\\gamma\\alpha}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}\\|x-y\\|^2.\n\\end{equation}\nFinally, if $\\gamma\\ell\\leq 1$, then, by \\ref{l:L-resol_only}, \n\\begin{equation}\n\\scal{x-y}{J_{\\gamma A}x-J_{\\gamma A}y}\\geq \\frac{1}{1+\\gamma\\ell}\\|x-y\\|^2,\n\\end{equation}\nand the conclusion follows.\n\\end{proof}","start":27371,"end":27906}]},{"id":"S051","semantic_type":"proposition","kind":"proposition","labels":["p:cresol","p:cresol_Lip","p:cresol_mono"],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"恒等与预解式的度量界","source_units":[{"raw_tex":"\\begin{proposition}[linear combinations of resolvents and the identity]\n\\label{p:cresol}\nLet $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$. Set $J:=J_{\\gamma A}$ and define $Q :=\\nu\\Id+\\lambda J$ with $\\nu, \\lambda\\in \\RR$. \n\\begin{enumerate}\n\\item\\label{p:cresol_mono} \nSuppose that $J$ is single-valued and $\\nu\\lambda\\leq 0$. Then, for all $x,y\\in \\dom J$, \n\\begin{equation}\n\\|Qx-Qy\\|^2\\leq \\nu^2\\|x-y\\|^2 +\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\|Jx-Jy\\|^2.\n\\end{equation}\n","start":28997,"end":29511}]},{"id":"S052","semantic_type":"proposition","kind":"proposition","labels":["e:LipQ","p:cresol","p:cresol_Lip","p:cresol_mono"],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"线性组合的利普希茨常数","source_units":[{"raw_tex":"Let $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$. Set $J:=J_{\\gamma A}$ and define $Q :=\\nu\\Id+\\lambda J$ with $\\nu, \\lambda\\in \\RR$. \n","start":29086,"end":29253},{"raw_tex":"\\item\\label{p:cresol_Lip}\nSuppose that $A$ is Lipschitz continuous with constant $\\ell$, $1+\\gamma\\alpha >0$, and $\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\leq 0$. Then $Q$ is Lipschitz continuous with constant    \n\\begin{equation}\n\\label{e:LipQ}\n\\rho :=\\sqrt{\\nu^2+\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\alpha_J},\n\\end{equation}\nwhere $\\alpha_J$ is defined as \\eqref{e:alpha_J}.\nIf additionally $\\lambda\\left(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\right)\\geq 0$ whenever $\\gamma\\ell <1$, then the Lipschitz constant \\eqref{e:LipQ} can be improved to\n\\begin{equation}\n\\overline{\\rho} :=\\sqrt{\\nu^2+\\frac{\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}}\\leq \\rho.\n\\end{equation}\n\\end{enumerate}\n\\end{proposition}","start":29511,"end":30263}]},{"id":"S053","semantic_type":"proof","kind":"proof","labels":["e:cresol_Lip","e:identityQ"],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"线性组合估计的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $x,y\\in \\dom J$. By the definition of $Q$,\n\\begin{subequations}\n\\label{e:identityQ}\n\\begin{align}\n\\|Qx-Qy\\|^2\n&=\\|\\nu(x-y)+\\lambda(Jx-Jy)\\|^2\\\\\n&=\\nu^2\\|x-y\\|^2 +2\\nu\\lambda\\scal{x-y}{Jx-Jy} +\\lambda^2\\|Jx-Jy\\|^2.\n\\end{align}\n\\end{subequations}\n\n\\ref{p:cresol_mono}: Since $\\nu\\lambda\\leq 0$, combining \\eqref{e:identityQ} with Lemma~\\ref{l:resol}\\ref{l:resol_single} yields\n\\begin{subequations}\n\\begin{align}\n\\|Qx-Qy\\|^2\n&\\leq \\nu^2\\|x-y\\|^2 +2\\nu\\lambda(1+\\gamma\\alpha)\\|Jx-Jy\\|^2 +\\lambda^2\\|Jx-Jy\\|^2\\\\\n&=\\nu^2\\|x-y\\|^2 +\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\|Jx-Jy\\|^2.\n\\end{align}\n\\end{subequations}\n\n\\ref{p:cresol_Lip}: First, according to Proposition~\\ref{p:resol}\\ref{p:resol_single}, $J$ is single-valued, and so is $Q$. Next, using \\eqref{e:identityQ}, Lemma~\\ref{l:resol}\\ref{l:resol_single}, and Lemma~\\ref{l:L-resol}\\ref{l:L-resol_mono} and noting that $\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\leq 0$, we have\n\\begin{subequations}\n\\label{e:cresol_Lip}\n\\begin{align}\n\\|Qx-Qy\\|^2 \n&\\leq \\nu^2\\|x-y\\|^2 +2\\nu\\lambda\\scal{x-y}{Jx-Jy} +\\frac{\\lambda^2}{1+\\gamma\\alpha}\\scal{x-y}{Jx-Jy}\\\\ \n&= \\nu^2\\|x-y\\|^2 +\\frac{\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)}{1+\\gamma\\alpha}\\scal{x-y}{Jx-Jy}\\\\  \n&\\leq \\nu^2\\|x-y\\|^2 +\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\alpha_J\\|x-y\\|^2 =\\rho^2\\|x-y\\|^2, \n\\end{align}\n\\end{subequations}\nwhich implies that $Q$ is Lipschitz continuous with constant $\\rho$.\n\nFor the last statement, we show that $\\alpha_J$ in formula \\eqref{e:LipQ} can be replaced by \n\\begin{equation}\n\\overline{\\alpha}_J :=\\frac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}.\n\\end{equation}\nIf $\\scal{x-y}{Jx-Jy}\\geq (1+\\gamma\\alpha)\\overline{\\alpha}_J\\|x-y\\|^2$, then \\eqref{e:cresol_Lip} also holds with $\\alpha_J$ replaced by $\\overline{\\alpha}_J$. Now, assume that $\\scal{x-y}{Jx-Jy} <(1+\\gamma\\alpha)\\overline{\\alpha}_J\\|x-y\\|^2$. By Lemma~\\ref{l:L-resol}\\ref{l:L-resol_mono}, we must have $\\gamma\\ell <1$, and then, by assumption, $\\lambda\\left(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\right)\\geq 0$. It now follows from \\eqref{e:identityQ} and \\eqref{e:L-resol_only} that\n\\begin{subequations}\n\\begin{align}\n\\|Qx-Qy\\|^2\n&\\leq \\nu^2\\|x-y\\|^2 +2\\nu\\lambda\\scal{x-y}{Jx-Jy} \\notag\\\\ \n&\\qquad +\\frac{\\lambda^2}{1-\\gamma^2\\ell^2}(2\\scal{x-y}{Jx-Jy} -\\|x-y\\|^2)\\\\\n&=\\Big(\\nu^2-\\frac{\\lambda^2}{1-\\gamma^2\\ell^2}\\Big)\\|x-y\\|^2 +2\\lambda\\Big(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\Big)\\scal{x-y}{Jx-Jy}\\\\\n&\\leq \\Big(\\nu^2-\\frac{\\lambda^2}{1-\\gamma^2\\ell^2}\\Big)\\|x-y\\|^2 +2\\lambda\\Big(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\Big)(1+\\gamma\\alpha)\\overline{\\alpha}_J\\|x-y\\|^2\\\\\n&=\\Big(\\nu^2+\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\overline{\\alpha}_J\\Big)\\|x-y\\|^2 =\\overline{\\rho}^2\\|x-y\\|^2.\n\\end{align}\n\\end{subequations}\nFinally, we will prove $\\overline{\\alpha}_J\\geq \\alpha_J$, which implies $\\overline{\\rho}\\leq \\rho$, i.e., the Lipschitz constant is indeed improved. From the definition of $\\alpha_J$, it suffices to consider the case in which $\\alpha_J =1/((1+\\gamma\\alpha)(1+\\gamma\\ell))$. Then $\\gamma\\ell\\leq 1$. \nSince $|\\alpha|\\leq \\ell$ (see Remark~\\ref{r:Lip_mono}), it holds that\n\\begin{subequations}\n\\begin{align}\n0 <(1+\\gamma\\alpha)^2\\leq 1+2\\gamma\\alpha+\\gamma^2\\ell^2\n&\\leq \n1+2\\gamma\\alpha+\\gamma^2\\ell^2\n+(\\gamma\\ell-\\gamma\\alpha)-\\gamma\\ell(\\gamma\\ell-\\gamma\\alpha)\\\\\n&=1+\\gamma\\alpha+\\gamma\\ell+(\\gamma\\alpha)(\\gamma\\ell)\\\\\n&=(1+\\gamma\\alpha)(1+\\gamma\\ell),\n\\end{align}\n\\end{subequations}\nand so \n\\begin{equation}\n\\overline{\\alpha}_J =\\frac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}\\geq \\alpha_J =\\frac{1}{(1+\\gamma\\alpha)(1+\\gamma\\ell)}.\n\\end{equation}\nThe proof is complete.\n\\end{proof}","start":30264,"end":33905}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":["e:cresol_Lip","e:identityQ"],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"线性组合的平方展开","source_units":[{"raw_tex":"\\begin{proof}\nLet $x,y\\in \\dom J$. By the definition of $Q$,\n\\begin{subequations}\n\\label{e:identityQ}\n\\begin{align}\n\\|Qx-Qy\\|^2\n&=\\|\\nu(x-y)+\\lambda(Jx-Jy)\\|^2\\\\\n&=\\nu^2\\|x-y\\|^2 +2\\nu\\lambda\\scal{x-y}{Jx-Jy} +\\lambda^2\\|Jx-Jy\\|^2.\n\\end{align}\n\\end{subequations}\n\n","start":30264,"end":30528}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["e:cresol_Lip","e:identityQ"],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"负交叉系数的单调估计","source_units":[{"raw_tex":"\\ref{p:cresol_mono}: Since $\\nu\\lambda\\leq 0$, combining \\eqref{e:identityQ} with Lemma~\\ref{l:resol}\\ref{l:resol_single} yields\n\\begin{subequations}\n\\begin{align}\n\\|Qx-Qy\\|^2\n&\\leq \\nu^2\\|x-y\\|^2 +2\\nu\\lambda(1+\\gamma\\alpha)\\|Jx-Jy\\|^2 +\\lambda^2\\|Jx-Jy\\|^2\\\\\n&=\\nu^2\\|x-y\\|^2 +\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\|Jx-Jy\\|^2.\n\\end{align}\n\\end{subequations}\n","start":30528,"end":30896}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":["e:cresol_Lip","e:identityQ"],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"两级内积界导出常数","source_units":[{"raw_tex":"\n\\ref{p:cresol_Lip}: First, according to Proposition~\\ref{p:resol}\\ref{p:resol_single}, $J$ is single-valued, and so is $Q$. Next, using \\eqref{e:identityQ}, Lemma~\\ref{l:resol}\\ref{l:resol_single}, and Lemma~\\ref{l:L-resol}\\ref{l:L-resol_mono} and noting that $\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\leq 0$, we have\n\\begin{subequations}\n\\label{e:cresol_Lip}\n\\begin{align}\n\\|Qx-Qy\\|^2 \n&\\leq \\nu^2\\|x-y\\|^2 +2\\nu\\lambda\\scal{x-y}{Jx-Jy} +\\frac{\\lambda^2}{1+\\gamma\\alpha}\\scal{x-y}{Jx-Jy}\\\\ \n&= \\nu^2\\|x-y\\|^2 +\\frac{\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)}{1+\\gamma\\alpha}\\scal{x-y}{Jx-Jy}\\\\  \n&\\leq \\nu^2\\|x-y\\|^2 +\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\alpha_J\\|x-y\\|^2 =\\rho^2\\|x-y\\|^2, \n\\end{align}\n\\end{subequations}\nwhich implies that $Q$ is Lipschitz continuous with constant $\\rho$.\n\n","start":30896,"end":31708}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":["e:cresol_Lip","e:identityQ"],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"按内积大小改善常数","source_units":[{"raw_tex":"For the last statement, we show that $\\alpha_J$ in formula \\eqref{e:LipQ} can be replaced by \n\\begin{equation}\n\\overline{\\alpha}_J :=\\frac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}.\n\\end{equation}\nIf $\\scal{x-y}{Jx-Jy}\\geq (1+\\gamma\\alpha)\\overline{\\alpha}_J\\|x-y\\|^2$, then \\eqref{e:cresol_Lip} also holds with $\\alpha_J$ replaced by $\\overline{\\alpha}_J$. Now, assume that $\\scal{x-y}{Jx-Jy} <(1+\\gamma\\alpha)\\overline{\\alpha}_J\\|x-y\\|^2$. By Lemma~\\ref{l:L-resol}\\ref{l:L-resol_mono}, we must have $\\gamma\\ell <1$, and then, by assumption, $\\lambda\\left(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\right)\\geq 0$. It now follows from \\eqref{e:identityQ} and \\eqref{e:L-resol_only} that\n\\begin{subequations}\n\\begin{align}\n\\|Qx-Qy\\|^2\n&\\leq \\nu^2\\|x-y\\|^2 +2\\nu\\lambda\\scal{x-y}{Jx-Jy} \\notag\\\\ \n&\\qquad +\\frac{\\lambda^2}{1-\\gamma^2\\ell^2}(2\\scal{x-y}{Jx-Jy} -\\|x-y\\|^2)\\\\\n&=\\Big(\\nu^2-\\frac{\\lambda^2}{1-\\gamma^2\\ell^2}\\Big)\\|x-y\\|^2 +2\\lambda\\Big(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\Big)\\scal{x-y}{Jx-Jy}\\\\\n&\\leq \\Big(\\nu^2-\\frac{\\lambda^2}{1-\\gamma^2\\ell^2}\\Big)\\|x-y\\|^2 +2\\lambda\\Big(\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}\\Big)(1+\\gamma\\alpha)\\overline{\\alpha}_J\\|x-y\\|^2\\\\\n&=\\Big(\\nu^2+\\lambda\\big(2\\nu(1+\\gamma\\alpha)+\\lambda\\big)\\overline{\\alpha}_J\\Big)\\|x-y\\|^2 =\\overline{\\rho}^2\\|x-y\\|^2.\n\\end{align}\n\\end{subequations}\n","start":31708,"end":33024}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":["e:cresol_Lip","e:identityQ"],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"比较两种预解式常数","source_units":[{"raw_tex":"Finally, we will prove $\\overline{\\alpha}_J\\geq \\alpha_J$, which implies $\\overline{\\rho}\\leq \\rho$, i.e., the Lipschitz constant is indeed improved. From the definition of $\\alpha_J$, it suffices to consider the case in which $\\alpha_J =1/((1+\\gamma\\alpha)(1+\\gamma\\ell))$. Then $\\gamma\\ell\\leq 1$. \nSince $|\\alpha|\\leq \\ell$ (see Remark~\\ref{r:Lip_mono}), it holds that\n\\begin{subequations}\n\\begin{align}\n0 <(1+\\gamma\\alpha)^2\\leq 1+2\\gamma\\alpha+\\gamma^2\\ell^2\n&\\leq \n1+2\\gamma\\alpha+\\gamma^2\\ell^2\n+(\\gamma\\ell-\\gamma\\alpha)-\\gamma\\ell(\\gamma\\ell-\\gamma\\alpha)\\\\\n&=1+\\gamma\\alpha+\\gamma\\ell+(\\gamma\\alpha)(\\gamma\\ell)\\\\\n&=(1+\\gamma\\alpha)(1+\\gamma\\ell),\n\\end{align}\n\\end{subequations}\nand so \n\\begin{equation}\n\\overline{\\alpha}_J =\\frac{1}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}\\geq \\alpha_J =\\frac{1}{(1+\\gamma\\alpha)(1+\\gamma\\ell)}.\n\\end{equation}\nThe proof is complete.\n\\end{proof}","start":33024,"end":33905}]},{"id":"S059","semantic_type":"corollary","kind":"corollary","labels":["c:rresol","c:rresol_perturb","c:rresol_relax"],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"松弛预解式的度量界","source_units":[{"raw_tex":"\\begin{corollary}[relaxed resolvents of $\\alpha$-monotone operators]\n\\label{c:rresol}\nLet $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$. Suppose that $J:=J_{\\gamma A}$ is single-valued and define $R:=(1-\\lambda)\\Id +\\lambda J$ with $\\lambda\\in \\RP$, and $Q:=R+\\varepsilon\\Id$ with $\\varepsilon\\in \\RR$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{c:rresol_relax} \nIf $\\lambda\\geq 1$, then, for all $x, y\\in \\dom J$, \n\\begin{equation}\n\\|Rx-Ry\\|^2\\leq (\\lambda-1)^2\\|x-y\\|^2 -\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda\\big)\\|Jx-Jy\\|^2.\n\\end{equation}\n","start":35151,"end":35748}]},{"id":"S060","semantic_type":"corollary","kind":"corollary","labels":["c:rresol","c:rresol_perturb","c:rresol_relax"],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"加恒等扰动后的上界","source_units":[{"raw_tex":"Let $A\\colon X\\rightrightarrows X$ be $\\alpha$-monotone and let $\\gamma\\in \\RPP$. Suppose that $J:=J_{\\gamma A}$ is single-valued and define $R:=(1-\\lambda)\\Id +\\lambda J$ with $\\lambda\\in \\RP$, and $Q:=R+\\varepsilon\\Id$ with $\\varepsilon\\in \\RR$. Then the following hold:\n","start":35237,"end":35510},{"raw_tex":"\\item\\label{c:rresol_perturb} \nIf $\\varepsilon\\leq \\lambda-1$, then, for all $x, y\\in \\dom J$, \n\\begin{align}\n\\|Qx-Qy\\|^2 &\\leq (\\lambda-1-\\varepsilon)^2\\|x-y\\|^2 \\notag\\\\ \n&\\quad -\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda-2\\varepsilon(1+\\gamma\\alpha)\\big)\\|Jx-Jy\\|^2.\n\\end{align}\nConsequently, if additionally $(\\lambda-1)(2+2\\gamma\\alpha)-\\lambda-2\\varepsilon(1+\\gamma\\alpha)\\geq 0$, then $Q$ is Lipschitz continuous with constant $(\\lambda-1-\\varepsilon)$.\n\\end{enumerate} \n\\end{corollary}","start":35748,"end":36247}]},{"id":"S061","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"一般松弛估计的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nBecause \\ref{c:rresol_relax} is a consequence of \\ref{c:rresol_perturb} with $\\varepsilon =0$, it suffices to prove only the latter. To this end, noting that $Q =R+\\varepsilon\\Id =(1-\\lambda+\\varepsilon)\\Id+\\lambda J$ and using Proposition~\\ref{p:cresol}\\ref{p:cresol_mono} with $\\nu =1-\\lambda+\\varepsilon \\leq 0$, we have that\n\\begin{subequations}\n\\begin{align}\n\\|Qx-Qy\\|^2&\\leq (1-\\lambda+\\varepsilon)^2\\|x-y\\|^2 +\\lambda\\left(2(1-\\lambda+\\varepsilon)(1+\\gamma\\alpha)+\\lambda\\right)\\|Jx-Jy\\|^2\\\\\n&=(\\lambda-1-\\varepsilon)^2\\|x-y\\|^2 -\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda-2\\varepsilon(1+\\gamma\\alpha)\\big)\\|Jx-Jy\\|^2\n\\end{align}\n\\end{subequations}\nwhich proves \\ref{c:rresol_perturb}. \n\\end{proof}","start":36248,"end":36973}]},{"id":"S062","semantic_type":"corollary","kind":"corollary","labels":["c:L-rresol","c:L-rresol_perturb","c:L-rresol_relax","e:cL-rresol-1"],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"利普希茨松弛预解式常数","source_units":[{"raw_tex":"\\begin{corollary}[relaxed resolvents of Lipschitz $\\alpha$-monotone operators]\n\\label{c:L-rresol}\nLet $A\\colon X\\to X$ be $\\alpha$-monotone and Lipschitz continuous with constant $\\ell$. Also let $\\gamma\\in\\RPP$ and $\\lambda\\in\\RPP$ be such that\n\\begin{equation}\\label{e:cL-rresol-1}\n1+\\gamma\\alpha>0\n\\ \\text{ and }\\ \n\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\geq 0.\n\\end{equation}\nDefine $J:=J_{\\gamma A}$, $R:=(1-\\lambda)\\Id +\\lambda J$, and $Q :=\\Id-\\varepsilon R$ with $\\varepsilon\\in \\RP$. Then the following hold:\n\\begin{enumerate}\n\\item\\label{c:L-rresol_relax} \n$R$ is Lipschitz continuous with constant \n\\begin{equation}\n\\sqrt{(\\lambda-1)^2-\\frac{\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda\\big)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}}.\n\\end{equation} \n","start":36977,"end":37738}]},{"id":"S063","semantic_type":"corollary","kind":"corollary","labels":["c:L-rresol","c:L-rresol_perturb","c:L-rresol_relax","e:cL-rresol-1"],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"减去松弛预解式的常数","source_units":[{"raw_tex":"Let $A\\colon X\\to X$ be $\\alpha$-monotone and Lipschitz continuous with constant $\\ell$. Also let $\\gamma\\in\\RPP$ and $\\lambda\\in\\RPP$ be such that\n\\begin{equation}\\label{e:cL-rresol-1}\n1+\\gamma\\alpha>0\n\\ \\text{ and }\\ \n\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\geq 0.\n\\end{equation}\nDefine $J:=J_{\\gamma A}$, $R:=(1-\\lambda)\\Id +\\lambda J$, and $Q :=\\Id-\\varepsilon R$ with $\\varepsilon\\in \\RP$. Then the following hold:\n","start":37075,"end":37498},{"raw_tex":"\\item\\label{c:L-rresol_perturb} \n$Q$ is Lipschitz continuous with constant \n\\begin{equation}\n\\sqrt{(1+\\varepsilon(\\lambda-1))^2 -\\varepsilon\\lambda\\Big[2(1+\\gamma\\alpha) +\\varepsilon\\big(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\big)\\Big]\\alpha_J},\n\\end{equation}\nwhere $\\alpha_J$ is defined as \\eqref{e:alpha_J}.\n\\end{enumerate}\n\\end{corollary}","start":37738,"end":38084}]},{"id":"S064","semantic_type":"proof","kind":"proof","labels":["e:cL-rresol-2"],"member_ids":["S064"],"declared_external":false,"source_only":false,"title":"利普希茨松弛推论的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{c:L-rresol_relax}: We observe that $R=\\nu\\Id+\\lambda J$ with $\\nu :=1-\\lambda$, that\n\\begin{equation}\n\\lambda(2\\nu(1+\\gamma\\alpha)+\\lambda) =-\\lambda\\Big(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\Big)\\leq 0,\n\\end{equation} \nand that, whenever $\\gamma\\ell <1$, \n\\begin{equation}\n\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}=\n(1-\\lambda)+\\frac{\\lambda}{1-\\gamma^2\\ell^2} >1-\\lambda+\\lambda =1 >0.\n\\end{equation}\nApplying Proposition~\\ref{p:cresol}\\ref{p:cresol_Lip} to $R=\\nu\\Id+\\lambda\\Id$ implies that $R$ is Lipschitz continuous with constant\n\\begin{equation}\n\\sqrt{(1-\\lambda)^2+\\frac{\\lambda\\big(2(1-\\lambda)(1+\\gamma\\alpha)+\\lambda\\big)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}},\n\\end{equation} \nwhich gives the claim.\n\n\\ref{c:L-rresol_perturb}: Using the first part of Proposition~\\ref{p:cresol}\\ref{p:cresol_Lip} and writing $Q =\\widetilde{\\nu}\\Id+\\widetilde{\\lambda}J$ with $\\widetilde{\\nu} :=1+\\varepsilon(\\lambda-1)$ and $\\widetilde{\\lambda} :=-\\varepsilon\\lambda$, it suffices to check that\n\\begin{equation}\\label{e:cL-rresol-2}\n\\widetilde{\\lambda}(2\\widetilde{\\nu}(1+\\gamma\\alpha)+\\widetilde{\\lambda})=(-\\varepsilon\\lambda)\\big( 2(1+\\varepsilon(\\lambda-1))(1+\\gamma\\alpha)+(-\\varepsilon\\lambda) \\big)\\leq 0.\n\\end{equation}\nIndeed, we have that $\\varepsilon\\geq 0$, $\\lambda >0$, and  \n\\begin{subequations}\n\\begin{align}\n2(1+\\varepsilon(\\lambda-1))(1+\\gamma\\alpha) -\\varepsilon\\lambda \n&=2(1+\\gamma\\alpha) +\\varepsilon\\big(2(\\lambda-1)(1+\\gamma\\alpha)-\\lambda\\big)\\\\\n&=2(1+\\gamma\\alpha) +\\varepsilon(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)) >0\n\\end{align}\n\\end{subequations}\nby \\eqref{e:cL-rresol-1}. So \\eqref{e:cL-rresol-2} holds and the conclusion follows.\n\\end{proof}","start":38085,"end":39773}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S065"],"declared_external":false,"source_only":false,"title":"验证松弛算子的改善条件","source_units":[{"raw_tex":"\\begin{proof}\n\\ref{c:L-rresol_relax}: We observe that $R=\\nu\\Id+\\lambda J$ with $\\nu :=1-\\lambda$, that\n\\begin{equation}\n\\lambda(2\\nu(1+\\gamma\\alpha)+\\lambda) =-\\lambda\\Big(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\Big)\\leq 0,\n\\end{equation} \nand that, whenever $\\gamma\\ell <1$, \n\\begin{equation}\n\\nu+\\frac{\\lambda}{1-\\gamma^2\\ell^2}=\n(1-\\lambda)+\\frac{\\lambda}{1-\\gamma^2\\ell^2} >1-\\lambda+\\lambda =1 >0.\n\\end{equation}\nApplying Proposition~\\ref{p:cresol}\\ref{p:cresol_Lip} to $R=\\nu\\Id+\\lambda\\Id$ implies that $R$ is Lipschitz continuous with constant\n\\begin{equation}\n\\sqrt{(1-\\lambda)^2+\\frac{\\lambda\\big(2(1-\\lambda)(1+\\gamma\\alpha)+\\lambda\\big)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}},\n\\end{equation} \nwhich gives the claim.\n","start":38085,"end":38812}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":["e:cL-rresol-2"],"member_ids":["S066"],"declared_external":false,"source_only":false,"title":"验证扰动组合的系数符号","source_units":[{"raw_tex":"\n\\ref{c:L-rresol_perturb}: Using the first part of Proposition~\\ref{p:cresol}\\ref{p:cresol_Lip} and writing $Q =\\widetilde{\\nu}\\Id+\\widetilde{\\lambda}J$ with $\\widetilde{\\nu} :=1+\\varepsilon(\\lambda-1)$ and $\\widetilde{\\lambda} :=-\\varepsilon\\lambda$, it suffices to check that\n\\begin{equation}\\label{e:cL-rresol-2}\n\\widetilde{\\lambda}(2\\widetilde{\\nu}(1+\\gamma\\alpha)+\\widetilde{\\lambda})=(-\\varepsilon\\lambda)\\big( 2(1+\\varepsilon(\\lambda-1))(1+\\gamma\\alpha)+(-\\varepsilon\\lambda) \\big)\\leq 0.\n\\end{equation}\nIndeed, we have that $\\varepsilon\\geq 0$, $\\lambda >0$, and  \n\\begin{subequations}\n\\begin{align}\n2(1+\\varepsilon(\\lambda-1))(1+\\gamma\\alpha) -\\varepsilon\\lambda \n&=2(1+\\gamma\\alpha) +\\varepsilon\\big(2(\\lambda-1)(1+\\gamma\\alpha)-\\lambda\\big)\\\\\n&=2(1+\\gamma\\alpha) +\\varepsilon(\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)) >0\n\\end{align}\n\\end{subequations}\nby \\eqref{e:cL-rresol-1}. So \\eqref{e:cL-rresol-2} holds and the conclusion follows.\n\\end{proof}","start":38812,"end":39773}]},{"id":"S067","semantic_type":"assumption","kind":"assumption","labels":["e:parameters","e:parameters_a","e:parameters_b","e:parameters_c","l:domT"],"member_ids":["S067"],"declared_external":false,"source_only":false,"title":"自适应参数约束","source_units":[{"raw_tex":"Then there exists $(\\gamma,\\delta,\\lambda,\\mu)\\in \\RPP^2\\times \\left]1,+\\infty\\right[^2$ such that\n\\begin{subequations}\n\\label{e:parameters}\n\\begin{align}\n&1+2\\gamma\\alpha >0,\n\\label{e:parameters_a}\\\\\n&\\mu\\in \\left[2-2\\gamma\\beta, 2+2\\gamma\\alpha\\right],\n\\label{e:parameters_b} \\\\\n&(\\lambda-1)(\\mu-1) =1, \\text{~and~} \\delta =(\\lambda-1)\\gamma. \\label{e:parameters_c}\n\\end{align}\n\\end{subequations}\n","start":54098,"end":54497}]},{"id":"S068","semantic_type":"lemma","kind":"lemma","labels":["e:parameters","e:parameters_a","e:parameters_b","e:parameters_c","l:domT"],"member_ids":["S068","S069","S070","S071"],"declared_external":false,"source_only":false,"title":"参数存在与 DR 满域性","source_units":[{"raw_tex":"\\begin{lemma}[single-valuedness and full domain of adaptive DR operator]\n\\label{l:domT}\nSuppose that $A$ and $B$ are maximally $\\alpha$- and $\\beta$-monotone with $\\alpha+\\beta\\geq 0$. \nThen there exists $(\\gamma,\\delta,\\lambda,\\mu)\\in \\RPP^2\\times \\left]1,+\\infty\\right[^2$ such that\n\\begin{subequations}\n\\label{e:parameters}\n\\begin{align}\n&1+2\\gamma\\alpha >0,\n\\label{e:parameters_a}\\\\\n&\\mu\\in \\left[2-2\\gamma\\beta, 2+2\\gamma\\alpha\\right],\n\\label{e:parameters_b} \\\\\n&(\\lambda-1)(\\mu-1) =1, \\text{~and~} \\delta =(\\lambda-1)\\gamma. \\label{e:parameters_c}\n\\end{align}\n\\end{subequations}\nMoreover, \\eqref{e:parameters} implies that $\\min\\{1+\\gamma\\alpha, 1+\\delta\\beta\\} >0$ and that $J_1$, $J_2$, and $T$ are single-valued and have full domain. \n\\end{lemma}","start":53912,"end":54667},{"raw_tex":"\\begin{proof}\nTo show the existence, we first take $\\gamma>0$ such that $1/\\gamma >-2\\alpha$. Then $1+2\\gamma\\alpha >0$ and $2+2\\gamma\\alpha =1+(1+2\\gamma\\alpha) >1$.\nUsing $\\alpha+\\beta\\geq 0$, we derive that\n\\begin{equation}\n2+2\\gamma\\alpha =2\\gamma(\\alpha+\\beta) +(2-2\\gamma\\beta)\\geq 2-2\\gamma\\beta.\n\\end{equation}\nHence, we can always choose $\\mu >1$ satisfying \\eqref{e:parameters_b}. Next, with such $\\mu$, we define $\\lambda :=\\mu/(\\mu-1) =1+1/(\\mu-1) >1$ and $\\delta :=(\\lambda-1)\\gamma$. Then \\eqref{e:parameters_c} is clearly satisfied. \n","start":54668,"end":55217},{"raw_tex":"\\begin{proof}\nTo show the existence, we first take $\\gamma>0$ such that $1/\\gamma >-2\\alpha$. Then $1+2\\gamma\\alpha >0$ and $2+2\\gamma\\alpha =1+(1+2\\gamma\\alpha) >1$.\nUsing $\\alpha+\\beta\\geq 0$, we derive that\n\\begin{equation}\n2+2\\gamma\\alpha =2\\gamma(\\alpha+\\beta) +(2-2\\gamma\\beta)\\geq 2-2\\gamma\\beta.\n\\end{equation}\nHence, we can always choose $\\mu >1$ satisfying \\eqref{e:parameters_b}. Next, with such $\\mu$, we define $\\lambda :=\\mu/(\\mu-1) =1+1/(\\mu-1) >1$ and $\\delta :=(\\lambda-1)\\gamma$. Then \\eqref{e:parameters_c} is clearly satisfied. \n\nNow, take any $(\\gamma,\\delta,\\lambda,\\mu)$ satisfying \\eqref{e:parameters}. We have\n\\begin{equation}\n1+\\delta\\beta =(\\lambda-1)(\\mu-1)+(\\lambda-1)\\gamma\\beta =\\frac{1}{2}(\\lambda-1)(\\mu + \\mu - (2-2\\gamma\\beta)) >0.\n\\end{equation}\nThus, $\\min\\{1+\\gamma\\alpha, 1+\\delta\\beta\\} >0$. The remaining conclusion follows from Proposition~\\ref{p:resol}. \n\\end{proof}","start":54668,"end":55577},{"raw_tex":"\nNow, take any $(\\gamma,\\delta,\\lambda,\\mu)$ satisfying \\eqref{e:parameters}. We have\n\\begin{equation}\n1+\\delta\\beta =(\\lambda-1)(\\mu-1)+(\\lambda-1)\\gamma\\beta =\\frac{1}{2}(\\lambda-1)(\\mu + \\mu - (2-2\\gamma\\beta)) >0.\n\\end{equation}\nThus, $\\min\\{1+\\gamma\\alpha, 1+\\delta\\beta\\} >0$. The remaining conclusion follows from Proposition~\\ref{p:resol}. \n\\end{proof}","start":55217,"end":55577}]},{"id":"S072","semantic_type":"proof","kind":"proof","labels":["e:180823a","e:domT'","e:rho1","e:rho2"],"member_ids":["S072"],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nIn view of Remark~\\ref{r:Lip_mono}, assumption \\ref{t:linear_reduce} implies assumption \\ref{t:linear_full} because if $A$ is Lipschitz continuous with constant $\\ell$, then $A$ is also $\\alpha$-monotone with $\\alpha :=-\\ell$. It thus suffices to assume \\ref{t:linear_full}.  \nFirst, Proposition~\\ref{p:maxalpha}\\ref{p:maxalpha_cont} implies that $A$ is maximally $\\alpha$-monotone. Next, we learn from Lemma~\\ref{l:domT} that \n\\begin{equation}\n\\label{e:domT'}\n\\min\\{1+\\gamma\\alpha,1+\\delta\\beta\\} >0\n\\end{equation}\nand that all operators $J_1,J_2$, and $T$ are single-valued and have full domain. \n\nBy the choice of $\\mu$, it holds that $0 <\\mu-1 \\leq 1+2\\gamma\\alpha$, and so\n\\begin{equation}\n\\lambda =1+\\frac{1}{\\mu-1}\\geq 1+\\frac{1}{1+2\\gamma\\alpha} =\\frac{2(1+\\gamma\\alpha)}{1+2\\gamma\\alpha},\n\\end{equation} \nwhich yields\n\\begin{equation}\n\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\geq 0.\n\\end{equation}\nFrom $\\mu\\geq 2-2\\gamma\\beta$, we have that $\\varepsilon\\geq 0$ and $\\varepsilon_1\\geq 0$. It follows that $\\varphi\\geq 0$ and that\n\\begin{equation}\\label{e:180823a}\n\\varphi =0 \\iff \\varepsilon_1 =0 \\iff \\varepsilon =0 \\iff \\mu =2-2\\gamma\\beta.\n\\end{equation}\nDefine $Q_1 :=\\Id -\\varepsilon_1 R_1$. Using Corollary~\\ref{c:L-rresol} and noting that $\\lambda =\\mu(\\lambda-1)$, we derive that $R_1$ is Lipschitz continuous with constant  \n\\begin{equation}\n\\sqrt{(\\lambda-1)^2-\\frac{\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda\\big)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}} =(\\lambda-1)\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}},\n\\end{equation} \nand that $Q_1$ is Lipschitz continuous with constant \n\\begin{equation}\n\\label{e:rho1}\n\\rho_1 :=\\sqrt{(1+\\varepsilon_1(\\lambda-1))^2 -\\varphi\\alpha_J}\\leq 1+\\varepsilon_1(\\lambda-1),\n\\end{equation}\nwhere $\\alpha_J$ is defined as in \\eqref{e:alpha_J}. It follows from \\eqref{e:180823a} that the inequality is strict whenever $\\mu >2-2\\gamma\\beta$.\n\nNext, define $Q_2 :=R_2+\\varepsilon\\Id$. \nSince $\\gamma =(\\mu-1)\\delta$, we note that\n\\begin{equation}\n\\varepsilon =\\frac{\\mu-(2-2\\gamma\\beta)}{2(1+\\delta\\beta)} =\\frac{(\\mu-1)(2+2\\delta\\beta)-\\mu}{2(1+\\delta\\beta)} <\\mu-1,\n\\end{equation}\nwhich also gives\n\\begin{equation}\n(\\mu-1)(2+2\\delta\\beta)-\\mu-2\\varepsilon(1+\\delta\\beta) =0.\n\\end{equation}\nBy Corollary~\\ref{c:rresol}\\ref{c:rresol_perturb}, $Q_2$ is Lipschitz continuous with constant $(\\mu-1-\\varepsilon)$. Combining with the Lipschitz continuity of $R_1$ and noting that $(\\mu-1)(\\lambda-1) =1$, we have that $Q_2R_1$ is Lipschitz continuous with constant\n\\begin{subequations}\n\\label{e:rho2}\n\\begin{align}\n\\rho_2\n&:=(\\mu-1-\\varepsilon)(\\lambda-1)\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}}\\\\\n&\\phantom{:}=(1-\\varepsilon(\\lambda-1))\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}}\\\\\n&\\phantom{:}\\leq 1-\\varepsilon(\\lambda-1),\n\\end{align}\n\\end{subequations} \nwhere the inequality is strict whenever $\\mu <2+2\\gamma\\alpha$.  \n\nNow, we express\n\\begin{subequations}\n\\begin{align}\nT&=(1-\\kappa)\\Id -(1-\\kappa)\\varepsilon_1R_1 + \\kappa R_2R_1+\\kappa\\varepsilon R_1\\\\\n&=(1-\\kappa)(\\Id-\\varepsilon_1R_1)+\\kappa(R_2+\\varepsilon\\Id)R_1\\\\\n&=(1-\\kappa)Q_1+\\kappa Q_2R_1.\n\\end{align}\n\\end{subequations}\nWe note from $\\alpha+\\beta >0$ that $2+2\\gamma\\alpha >2-2\\gamma\\beta$, so at least one of two inequalities in \\eqref{e:rho1} and \\eqref{e:rho2} is strict.\nTherefore, $T$ is Lipschitz continuous with constant\n\\begin{equation}\n\\rho :=(1-\\kappa)\\rho_1+\\kappa \\rho_2 <(1-\\kappa)(1+\\varepsilon_1(\\lambda-1))+\\kappa(1-\\varepsilon(\\lambda-1))=1,\n\\end{equation}\nwhich completes the proof.\n\\end{proof}","start":63950,"end":67583}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":["e:rho2"],"member_ids":["S073"],"declared_external":false,"source_only":false,"title":"归约无单调假设的分支","source_units":[{"raw_tex":"\\begin{proof}\nIn view of Remark~\\ref{r:Lip_mono}, assumption \\ref{t:linear_reduce} implies assumption \\ref{t:linear_full} because if $A$ is Lipschitz continuous with constant $\\ell$, then $A$ is also $\\alpha$-monotone with $\\alpha :=-\\ell$. It thus suffices to assume \\ref{t:linear_full}.  \n","start":63950,"end":64241}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":["e:domT'","e:rho2"],"member_ids":["S074"],"declared_external":false,"source_only":false,"title":"极大性与全定义域","source_units":[{"raw_tex":"First, Proposition~\\ref{p:maxalpha}\\ref{p:maxalpha_cont} implies that $A$ is maximally $\\alpha$-monotone. Next, we learn from Lemma~\\ref{l:domT} that \n\\begin{equation}\n\\label{e:domT'}\n\\min\\{1+\\gamma\\alpha,1+\\delta\\beta\\} >0\n\\end{equation}\nand that all operators $J_1,J_2$, and $T$ are single-valued and have full domain. \n\n","start":64241,"end":64564}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":["e:rho2"],"member_ids":["S075"],"declared_external":false,"source_only":false,"title":"推出松弛参数的符号条件","source_units":[{"raw_tex":"By the choice of $\\mu$, it holds that $0 <\\mu-1 \\leq 1+2\\gamma\\alpha$, and so\n\\begin{equation}\n\\lambda =1+\\frac{1}{\\mu-1}\\geq 1+\\frac{1}{1+2\\gamma\\alpha} =\\frac{2(1+\\gamma\\alpha)}{1+2\\gamma\\alpha},\n\\end{equation} \nwhich yields\n\\begin{equation}\n\\lambda(1+2\\gamma\\alpha)-2(1+\\gamma\\alpha)\\geq 0.\n\\end{equation}\n","start":64564,"end":64873}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","labels":["e:180823a","e:rho2"],"member_ids":["S076"],"declared_external":false,"source_only":false,"title":"扰动常数的非负与零点","source_units":[{"raw_tex":"From $\\mu\\geq 2-2\\gamma\\beta$, we have that $\\varepsilon\\geq 0$ and $\\varepsilon_1\\geq 0$. It follows that $\\varphi\\geq 0$ and that\n\\begin{equation}\\label{e:180823a}\n\\varphi =0 \\iff \\varepsilon_1 =0 \\iff \\varepsilon =0 \\iff \\mu =2-2\\gamma\\beta.\n\\end{equation}\n","start":64873,"end":65133}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","labels":["e:rho2"],"member_ids":["S077"],"declared_external":false,"source_only":false,"title":"第一松弛算子的估计","source_units":[{"raw_tex":"Define $Q_1 :=\\Id -\\varepsilon_1 R_1$. Using Corollary~\\ref{c:L-rresol} and noting that $\\lambda =\\mu(\\lambda-1)$, we derive that $R_1$ is Lipschitz continuous with constant  \n\\begin{equation}\n\\sqrt{(\\lambda-1)^2-\\frac{\\lambda\\big((\\lambda-1)(2+2\\gamma\\alpha)-\\lambda\\big)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}} =(\\lambda-1)\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}},\n\\end{equation} \n","start":65133,"end":65542}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","labels":["e:rho1","e:rho2"],"member_ids":["S078"],"declared_external":false,"source_only":false,"title":"第一扰动算子的严格界","source_units":[{"raw_tex":"and that $Q_1$ is Lipschitz continuous with constant \n\\begin{equation}\n\\label{e:rho1}\n\\rho_1 :=\\sqrt{(1+\\varepsilon_1(\\lambda-1))^2 -\\varphi\\alpha_J}\\leq 1+\\varepsilon_1(\\lambda-1),\n\\end{equation}\nwhere $\\alpha_J$ is defined as in \\eqref{e:alpha_J}. It follows from \\eqref{e:180823a} that the inequality is strict whenever $\\mu >2-2\\gamma\\beta$.\n\n","start":65542,"end":65889}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","labels":["e:rho2"],"member_ids":["S079"],"declared_external":false,"source_only":false,"title":"第二扰动算子的消项","source_units":[{"raw_tex":"Next, define $Q_2 :=R_2+\\varepsilon\\Id$. \nSince $\\gamma =(\\mu-1)\\delta$, we note that\n\\begin{equation}\n\\varepsilon =\\frac{\\mu-(2-2\\gamma\\beta)}{2(1+\\delta\\beta)} =\\frac{(\\mu-1)(2+2\\delta\\beta)-\\mu}{2(1+\\delta\\beta)} <\\mu-1,\n\\end{equation}\nwhich also gives\n\\begin{equation}\n(\\mu-1)(2+2\\delta\\beta)-\\mu-2\\varepsilon(1+\\delta\\beta) =0.\n\\end{equation}\nBy Corollary~\\ref{c:rresol}\\ref{c:rresol_perturb}, $Q_2$ is Lipschitz continuous with constant $(\\mu-1-\\varepsilon)$. Combining with the Lipschitz continuity of $R_1$ and noting that $(\\mu-1)(\\lambda-1) =1$, we have that $Q_2R_1$ is Lipschitz continuous with constant\n","start":65889,"end":66505}]},{"id":"S080","semantic_type":"proof_step","kind":"proof_step","labels":["e:rho2"],"member_ids":["S080"],"declared_external":false,"source_only":false,"title":"复合算子的严格界","source_units":[{"raw_tex":"By Corollary~\\ref{c:rresol}\\ref{c:rresol_perturb}, $Q_2$ is Lipschitz continuous with constant $(\\mu-1-\\varepsilon)$. Combining with the Lipschitz continuity of $R_1$ and noting that $(\\mu-1)(\\lambda-1) =1$, we have that $Q_2R_1$ is Lipschitz continuous with constant\n\\begin{subequations}\n\\label{e:rho2}\n\\begin{align}\n\\rho_2\n&:=(\\mu-1-\\varepsilon)(\\lambda-1)\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}}\\\\\n&\\phantom{:}=(1-\\varepsilon(\\lambda-1))\\sqrt{1-\\frac{\\mu(2+2\\gamma\\alpha-\\mu)}{1+2\\gamma\\alpha+\\gamma^2\\ell^2}}\\\\\n&\\phantom{:}\\leq 1-\\varepsilon(\\lambda-1),\n\\end{align}\n\\end{subequations} \nwhere the inequality is strict whenever $\\mu <2+2\\gamma\\alpha$.  \n\n","start":66237,"end":66926}]},{"id":"S081","semantic_type":"proof_step","kind":"proof_step","labels":["e:rho2"],"member_ids":["S081"],"declared_external":false,"source_only":false,"title":"DR 算子的扰动分解","source_units":[{"raw_tex":"Now, we express\n\\begin{subequations}\n\\begin{align}\nT&=(1-\\kappa)\\Id -(1-\\kappa)\\varepsilon_1R_1 + \\kappa R_2R_1+\\kappa\\varepsilon R_1\\\\\n&=(1-\\kappa)(\\Id-\\varepsilon_1R_1)+\\kappa(R_2+\\varepsilon\\Id)R_1\\\\\n&=(1-\\kappa)Q_1+\\kappa Q_2R_1.\n\\end{align}\n\\end{subequations}\n","start":66926,"end":67191}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","labels":["e:rho2"],"member_ids":["S082"],"declared_external":false,"source_only":false,"title":"加权常数严格小于一","source_units":[{"raw_tex":"We note from $\\alpha+\\beta >0$ that $2+2\\gamma\\alpha >2-2\\gamma\\beta$, so at least one of two inequalities in \\eqref{e:rho1} and \\eqref{e:rho2} is strict.\nTherefore, $T$ is Lipschitz continuous with constant\n\\begin{equation}\n\\rho :=(1-\\kappa)\\rho_1+\\kappa \\rho_2 <(1-\\kappa)(1+\\varepsilon_1(\\lambda-1))+\\kappa(1-\\varepsilon(\\lambda-1))=1,\n\\end{equation}\nwhich completes the proof.\n\\end{proof}","start":67191,"end":67583}]},{"id":"S083","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S083"],"declared_external":true,"declaration_text":["原始 prerequisite 列表：BC17, Theorem 21.1；原文第 237 行明确引用。"],"source_only":false,"title":"Minty 定理","source_units":[]},{"id":"S084","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S084"],"declared_external":true,"declaration_text":["原始 prerequisite 列表：BC17, Proposition 20.22；原文第 237 行明确引用。"],"source_only":false,"title":"正缩放保持极大单调性","source_units":[]},{"id":"S085","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S085"],"declared_external":true,"declaration_text":["原始 prerequisite 列表：BC17, Corollary 20.28；原文第 391 行明确引用。"],"source_only":false,"title":"连续满域单调算子的极大性","source_units":[]},{"id":"S086","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S086"],"declared_external":true,"declaration_text":["原始 prerequisite 列表中的标准 Hilbert 范数恒等式、三角不等式及 Cauchy–Schwarz；复合与加权和估计作为这些标准不等式的隐含应用，由标注者显式列出。"],"source_only":false,"title":"希尔伯特空间基本不等式","source_units":[]},{"id":"S087","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S087"],"declared_external":true,"declaration_text":["原始 prerequisite 列表明确列出的 Banach contraction principle on a complete Hilbert space；原文未具名引用。"],"source_only":false,"title":"Banach 收缩原理","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S022","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S068","S070":"S068","S071":"S068","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076","S077":"S077","S078":"S078","S079":"S079","S080":"S080","S081":"S081","S082":"S082","S083":"S083","S084":"S084","S085":"S085","S086":"S086","S087":"S087"}}}}
{"record_id":"1512.02549v4","paper":{"title":"Facial Reduction and Partial Polyhedrality","authors":["Lourenço, Bruno F.","Muramatsu, Masakazu","Tsuchiya, Takashi"],"arxiv_id":"1512.02549","version":"v4","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/1512.02549v4","pdf_url":"https://arxiv.org/pdf/1512.02549v4","doi":"10.1137/15M1051634"},"context":{"raw_tex":"Consider the following pair of primal and dual conic linear programs (CLPs):\r\n\\vspace*{-3\\baselineskip} \r\n\\begin{multicols}{2}\r\n\\begin{align}\r\n\\underset{x}{\\inf} & \\quad \\inProd{c}{x} \\label{eq:primal}\\tag{P}\\\\ \r\n\\mbox{subject to} & \\quad \\stdMap x = b \\nonumber \\\\ \r\n&\\quad x \\in \\stdCone ^* \\nonumber \r\n\\end{align}\r\n\t\\break\r\n\t\\begin{align}\r\n\t\\underset{y}{\\sup} & \\quad \\inProd{b}{y} \\label{eq:dual} \\tag{D} \\\\ \r\n\t\\mbox{subject to} & \\quad c - \\stdMap ^\\T y \\in \\stdCone \\nonumber,\\\\ \\nonumber\r\n\t\\end{align}\t\r\n\\end{multicols}\r\n\\noindent where $\\stdCone \\subseteq \\Re^n$ is a closed convex cone and $\\stdCone^*$ is the dual cone $\\{s \\in \\Re^n \\mid \\inProd{s}{x} \\geq 0, \\forall x \\in \\stdCone \\}$. We have \r\nthat $\\stdMap: \\Re^n \\to \\Re^m$ is a linear map, $b \\in \\Re^m$, $c \\in \\Re^n$ and \r\n$\\stdMap^\\T$ denotes the adjoint map. We also have $\\stdMap^\\T y = \\sum _{i=1}^m \\stdMap _i y_i$, for certain \r\nelements $\\stdMap _i \\in \\Re^n$. The  inner product is denoted by $\\inProd{\\cdot}{\\cdot}$.\r\nWe will use $\\pOpt$ and $\\dOpt$ to denote the primal and  dual optimal value, respectively.\r\nIt is understood that $\\pOpt = +\\infty$ if \\eqref{eq:primal} is infeasible and \r\n$\\dOpt = -\\infty$ if \\eqref{eq:dual} is infeasible.\r\n\n\nIn the absence of either a primal relative interior feasible solution or a dual \r\nrelative interior slack, it is possible that $\\pOpt \\neq \\dOpt$. A possible \r\nway of  correcting that is to let $\\minFaceD$ be \r\nthe minimal face of $\\stdCone$ which contains the feasible slacks $\\feasS = \\{c- \\stdMap ^\\T y \\in \\stdCone \\mid y \\in \\Re^m  \\}$, then \r\nwe substitute $\\stdCone$ by $\\minFaceD$ and $\\stdCone^*$ by $(\\minFaceD)^*$. With that, \r\nthe new primal optimal value $\\pOpt ' $ will satisfy $\\pOpt ' = \\dOpt$. This is precisely what \r\nfacial reduction \\cite{Borwein1981495, pataki_strong_2013, article_waki_muramatsu} approaches do.\r\n\n\nFor $C$ a closed convex set, we will denote by $\\reInt C$ and \r\n$\\closure C$ the relative interior and the closure of $C$, respectively.\r\nIf $\\mathcal{U}$ is an \r\narbitrary set, we denote by $\\mathcal{U}^\\perp$ the subspace which contains \r\nthe elements orthogonal to it. We will denote by $\\PSDcone{n}$ the cone of \r\n$n\\times n$ symmetric positive semidefinite matrices and by $\\SOC{n}$ the \r\nLorentz cone $\\{(x_0,\\overline{x}) \\in \\Re\\times \\Re^{n-1} \\mid x_0 \\geq \\norm{\\overline{x}}_2 \\}$, \r\nwhere $\\norm{.}_2$ is the usual Euclidean norm.\r\nThe nonnegative orthant will be denoted by $\\Re^n_+$. \r\n\r\n\r\n\r\nIf $\\stdCone$ is a closed convex cone, we write $\\stdCone^*$ for its dual \r\ncone. We write $\\spanVec \\stdCone$ for its linear span and $\\lineality \\stdCone$ for \r\nits lineality space, which is $\\stdCone \\cap -\\stdCone$. $\\stdCone$ is said to be \\emph{pointed} if \r\n$\\lineality \\stdCone  = \\{0\\}$. We have $\\lineality (\\stdCone^*) = \\stdCone^\\perp$, see Theorem 14.6 in \\cite{rockafellar}. Also, if we select $\\stdInt \\in \\reInt \\stdCone$ and $x \\in \\stdCone^*$, then $x \\in \\stdCone^\\perp $ if and only if $\\inProd{\\stdInt}{x} = 0$\\footnote{Suppose $\\inProd{\\stdInt}{x} = 0$. Then, given $y \\in \\stdCone$ we have $\\alpha \\stdInt +(1-\\alpha)y \\in \\stdCone$ for some $\\alpha > 1$, due to Theorem 6.4 in \\cite{rockafellar}. Taking the inner product with $x$, we see that $\\inProd{y}{x}$ must be zero.}.\r\n\r\n\r\n\r\nThe conic linear program \\eqref{eq:dual} can be in four different feasibility \r\nstatuses: $i)$ strongly feasible if $\\feasS \\cap \\reInt \\stdCone \\neq \\emptyset$; $ii)$ weakly feasible if \r\n$\\feasS \\neq \\emptyset$ but $\\feasS \\cap \\reInt \\stdCone  = \\emptyset$; $iii)$ weakly infeasible if \r\n$\\feasS = \\emptyset$ but $\\text{dist}(c + \\matRange \\stdMap^\\T ,\\stdCone) = 0$; $iv)$ strongly \r\ninfeasible if $\\text{dist}(c + \\matRange \\stdMap^\\T ,\\stdCone) > 0$.  Note that \r\n\\eqref{eq:primal} admits analogous definitions. \r\n\r\nThe \r\nstrong duality theorem states that if \\eqref{eq:dual} is strongly feasible and \r\n$\\dOpt < + \\infty$, then $\\pOpt = \\dOpt$ and $\\pOpt$ is attained. \r\nOn the other hand, if \\eqref{eq:primal} is strongly feasible and \r\n$\\pOpt > -\\infty$,  then $\\pOpt = \\dOpt$ and $\\dOpt$ is attained. \r\nWe  will also need a special version of the strong duality theorem. \r\nFirst, we need the following definition.\r\n\r\n\\begin{definition}[Partial Polyhedral Slater's condition]\\label{def:pps}\r\n\tLet $\\stdCone = \\stdCone ^1\\times \\stdCone ^2$, where  $\\stdCone ^1\\subseteq \\Re^{n_1},\\stdCone ^2 \\subseteq \\Re^{n_2}$ are closed convex cones such that $\\stdCone ^2$ is polyhedral.\r\n\tWe say that \\eqref{eq:dual} satisfies the Partial Polyhedral Slater's (\\PPS) condition if there is \r\n\ta slack $(s_1,s_2) = c-\\stdMap^\\T y$, such that $s_1 \\in \\reInt \\stdCone^1$ and $s_2 \\in \\stdCone^2$.\r\n\tSimilarly, we say that \\eqref{eq:primal} satisfies the {\\PPS} condition, if there is a primal \r\n\tfeasible solution $x = (x_1,x_2)$ for which $x_1 \\in \\reInt (\\stdCone^1)^*$.\r\n\\end{definition}\r\nThe following is a strong duality theorem based on the {\\PPS} condition.\r\nAs we could not find a precise reference for it, we give a proof in the Appendix \\ref{app:partial}.  \r\n\n\nHere, we will suppose that our main interest is in the dual problem \\eqref{eq:dual}.\r\n Facial Reduction hinges on the fact that \r\nstrong feasibility fails  if and only if there \r\nis $d \\in \\stdCone ^*$ such that $\\stdMap d = 0$  and\r\none of the two alternatives holds: $(i)$  $\\inProd{c}{d} = 0$ and $d \\not \\in \\stdCone ^\\perp$; or \r\n$(ii)$ $\\inProd{d}{c} < 0$, see Lemma 3.2 in \\cite{article_waki_muramatsu}. If \r\nalternative $(i)$ holds, $\\stdFace = \\stdCone \\cap \\{d\\}^\\perp$ is a proper face of $\\stdCone$ \r\ncontaining $\\feasS$.  We then substitute  $\\stdCone$ by $\\stdFace $ and repeat.\r\nIf  $(ii)$ holds, \\eqref{eq:dual} is infeasible. \r\n We write below a generic \r\nfacial reduction algorithm similar to the one described in \\cite{article_waki_muramatsu}.\r\n\r\n\\noindent\r\n[{\\bf Generic Facial Reduction}]\\label{alg:gen_fra}\r\n\r\n{\\bf Input:} \\eqref{eq:dual}\r\n\r\n{\\bf Output:} A set of reducing directions $\\{d_1, \\ldots, d_\\ell\\}$ and $\\minFaceD$.\r\n\\begin{enumerate}\r\n\\item $\\stdFace _1 \\leftarrow \\stdCone, i \\leftarrow 1 $\r\n\\item Let $d_i$ be an element in $\\stdFace _i ^* \\cap \\ker \\stdMap$ such that either: \r\n$i)$ $d_i \\not \\in \\stdFace _i^\\perp$ and $\\inProd{c}{d_i} = 0$; or $ii)$ $\\inProd{c}{d_i} < 0$.\r\nIf no such $d_i$ exists, let $\\minFaceD \\leftarrow \\stdFace _i$ and stop. \r\n\\item If $\\inProd{c}{d_i} < 0$, let $\\minFaceD \\leftarrow  \\emptyset$ and stop.\r\n\\item If $\\inProd{c}{d_i} = 0$, let $\\stdFace _{i+1}  \\leftarrow \\stdFace _i  \\cap \\{d_i\\}^\\perp, i\\leftarrow i+1$ and return to 2).\r\n\\end{enumerate}\r\n\r\n\r\nWe will refer to the directions satisfying $d_i \\in \\stdFace _i^* \\cap \\ker \\stdMap$ and \r\n$\\inProd{c}{d_i} \\leq 0$ as \\emph{reducing directions}, so that the $d_i$ in Step 2. are indeed reducing directions. An important issue when doing facial reduction is how to model the \r\nsearch for the reducing directions. It is sometimes said that doing facial \r\nreduction can be as hard as solving the original problem. However, an important \r\ndifference is that the search for the $d _i$ can be cast as a pair of primal and \r\ndual problems which are always strongly feasible.\r\nThis was shown in the work by Cheung, Schurr and Wolkowicz \\cite{csw13} and \r\nin our previous work \\cite{lourenco_muramatsu_tsuchiya3}. \r\n\n\nIn this article, we will search for reducing directions by\r\nconsidering the pair \\eqref{eq:red} and \\eqref{eq:red_dual} introduced in \\cite{lourenco_muramatsu_tsuchiya3}, which \r\nare parametrized by $\\stdMap$, $c$, $\\stdCone, \\stdInt, \\stdInt^*$. In Phase 1 of FRA-Poly, we will always \r\nselect $\\stdInt$ and $\\stdInt^*$ according to Lemma \\ref{lemma:red2}. Different choices will be discussed/used \r\nin Phase 2 of FRA-Poly and on Sections \\ref{sec:compl} and \\ref{sec:tight}.\r\n\\begin{align}\r\n\\underset{x,t,w}{\\mbox{minimize}} & \\quad t & \\tag{$P_\\stdCone $}\\label{eq:red}  \\\\ \r\n\\mbox{subject to} & \\quad -\\inProd{c}{x  -t\\stdInt^*} + t - w &= 0 \\label{eq:red:1}\\\\\r\n&\\quad \\inProd{\\stdInt}{x} + w &= 1 \\label{eq:red:2}\\\\\r\n& \\quad\\stdMap x  -t\\stdMap \\stdInt^* & = 0 \\label{eq:red:3}\\\\\r\n&\\quad (x,t,w) \\in \\stdCone ^* \\times \\Re_+ \\times \\Re_+ \\nonumber\\\\\r\n\\underset{y_1,y_2,y_3}{\\mbox{maximize}} &\\quad  y_2 & \\tag{$D_\\stdCone $}\\label{eq:red_dual}  \\\\ \r\n\\mbox{subject to} & \\quad cy_1 -\\stdInt y_2 -\\stdMap^\\T y_3 \\in \\stdCone \\label{eq:red_dual:1} \\\\\r\n&\\quad 1-y_1(1+\\inProd{c}{\\stdInt^*}) + \\inProd{\\stdInt^* }{\\stdMap ^\\T y_3}\\geq 0 \\label{eq:red_dual:2} \\\\\r\n& \\quad y_1-y_2 \\geq 0 \\label{eq:red_dual:3} \\\\\r\n&\\quad (y_1,y_2,y_3) \\in \\Re \\times \\Re \\times \\Re^m. \\nonumber\r\n\\end{align}\r\n\r\nRecall that our goal is to find a point $x \\in \\ker \\stdMap \\cap \\stdCone$ satisfying  $\\inProd{c}{x} \\leq 0$. The idea behind  \\eqref{eq:red} is to shift the problem by $-t\\stdInt^*$ (Equations \\eqref{eq:red:1} and \\eqref{eq:red:3}) and add constraints to ensure \r\nthe $x$ stays in a bounded region (Equation \\eqref{eq:red:2}). These changes ensure that \\eqref{eq:red} and \\eqref{eq:red_dual} \r\nsatisfy the PPS condition when the parameters $\\stdInt, \\stdInt^*$ are chosen appropriately as in\r\nthe next section.\r\n\n\nWe now present a disclaimer of sorts.\r\nThe theoretical results presented in this section and the next stand whether FRA-Poly is doable or \r\nnot for a given $\\stdCone$. \r\nIf we wish to do facial reduction concretely (even if it is by hand!), we need to \r\nmake a few assumptions on our computational capabilities \r\nand on our knowledge on the lattice of faces of $\\stdCone$. \r\nFirst of all, we must be able to solve problems over faces of $\\stdCone$ such that both the primal and the dual \r\nsatisfy the {\\PPS} condition and we must also be able to do \r\nbasic linear algebraic operations.  Also, for \r\neach face $\\stdFace$ of $\\stdCone$ we must know:\r\n\\begin{enumerate}\r\n\t\\item $\\spanVec \\stdFace$, \r\n\t\\item at least one point $\\stdInt \\in \\reInt \\stdFace$,\r\n\t\\item at least one point $\\stdInt^* \\in \\reInt \\stdFace^*$,\r\n\t\\item whether $\\stdFace$ is polyhedral or not.\r\n\\end{enumerate}\r\nWe remark that \r\napart from  knowledge about the polyhedral faces, our assumptions are not very different from what it is \r\nusually assumed \\emph{implicitly} in the FRA literature. \r\nFor symmetric cones, which \r\ninclude direct products of $\\PSDcone{n}$, $\\SOC{n}$ and $\\Re^n_+$, they are reasonable since their \r\nlattice of faces is well-understood and every face is again a symmetric cone. \r\nSo, for instance, $\\stdInt$ can be taken as the identity element for \r\nthe corresponding Jordan algebra. On the other hand, if $\\stdCone$ is, say, the copositive \r\ncone $\\mathcal{C}^n$, we might have some trouble fulfilling the requirements, inasmuch  as our knowledge of \r\nthe faces of $\\mathcal{C}^n$ is still lacking.\r\n\r\n\\subsection{Distance to Polyhedrality}\r\nHere we introduce the notion of \\emph{distance to polyhedrality}. In what follows, \r\nif we have a chain of faces $\\stdFace _1 \\subsetneq \\ldots \\subsetneq \\stdFace _\\ell$, the \r\nlength of the chain is defined to be $\\ell$.\r\n\\begin{definition}\r\n\tLet $\\stdCone$ be a nonempty closed convex cone. The \\emph{distance to polyhedrality} $\\distP(\\stdCone)$ is the \r\n\tlength \\emph{minus one} of the longest strictly ascending chain of nonempty faces $\\stdFace _1 \\subsetneq \\ldots \\subsetneq \\stdFace _{\\ell} $ which satisfies:\r\n\t\\begin{enumerate}\r\n\t\t\\item  $\\stdFace _1$ is polyhedral;  \r\n\t\t\\item $\\stdFace _j$ is not polyhedral for $j > 1$.\r\n\t\\end{enumerate}\r\n\\end{definition}\r\n\r\nThe distance to polyhedrality is a well-defined concept, because the lineality space\r\nof $\\stdCone$ is always a polyhedral face of $\\stdCone$.\r\nMoreover, $\\distP(\\stdCone)$ counts the maximum number of facial reduction steps that \r\ncan be taken before we reach a polyhedral face.\r\n\n\nFor general $\\stdCone$, these equivalencies may not hold and we might need \r\nto distinguish between primal and dual strict complementarity, see \r\nfor instance, Definition 3.4 and Remark 3.6 in the chapter by Pataki \\cite{pataki_handbook} and Equation (2.6) in Section 2 of the work by Tun\\c{c}el and Wolkowicz \\cite{TW12}. \r\nBased on those references, we will say \r\nthat \\eqref{eq:red} and \\eqref{eq:red_dual}  satisfies  \r\n\\emph{(dual) strict complementarity} if $\\opt{\\text{\\ref{eq:red}}} = \\opt{\\text{\\ref{eq:red_dual}}}$\r\nand there are optimal solutions $(x^*, t^*,w^*), (y_1^*,y_2^*,y_3^*)$ such that\r\n \\begin{align}\r\n cy_1^* -\\stdInt y_2^* -\\stdMap^\\T y_3^* &\\in \\reInt (\\stdCone \\cap\\{ x^*\\}^\\perp)  \\label{eq:strict1}\\\\\r\n t^* +  1-y_1^*(1+\\inProd{c}{\\stdInt^*}) + \\inProd{\\stdInt^* }{\\stdMap ^\\T y_3^*}&> 0  \\label{eq:strict2}\\\\\r\n w^* + y_1^*-y_2^* & > 0. \\label{eq:strict3} \r\n \\end{align}\r\n\n\nHenceforth, we will assume that $\\stdCone$ is the product of $r$ cones and we will \r\nwrite $\\stdCone = \\stdCone ^1 \\times \\ldots \\times \\stdCone  ^r $. Then, recall that \r\nif $\\stdFace$ is face of $\\stdCone$, we can write $\\stdFace = \\stdFace ^1 \\times \\ldots \\times \\stdFace  ^r$, where $\\stdFace ^i$ is a face of $\\stdCone ^i$ for every $i$.\r\n\r\nConsider the following \r\nFRA variant, which we call FRA-Poly.\r\n\r\n\\noindent\r\n[{\\bf Facial Reduction Poly - Phase 1}]\r\n\r\n{\\bf Input:} \\eqref{eq:dual}\r\n\r\n{\\bf Output:} A set of reducing directions $\\{d_1, \\ldots, d_\\ell\\}$. If \r\n\\eqref{eq:dual} is feasible, it outputs  some face $\\stdFace \\subseteq \\stdCone$ \r\nfor which the PPS condition holds, together \r\nwith a dual slack $s'$ for which $s_j' \\in \\reInt \\stdFace ^j $ for every $j$ such that $\\stdFace ^j$  is nonpolyhedral.\r\nIf \\eqref{eq:dual} is infeasible, the directions form a certificate of infeasibility.\r\n\\begin{enumerate}\r\n\\item $\\stdFace _1 \\leftarrow \\stdCone, i \\leftarrow 1 $\r\n\\item Let  $(x^*,t^*,w^*)$ and $(y_1^*,y_2^*,y_3^*)$ be any pair of optimal solutions to \r\n\\eqref{eq:red} and \\eqref{eq:red_dual} with\r\n\\begin{itemize}\r\n\t\\item $\\stdFace _i$ in place of $\\stdCone $, where $\\stdFace_i = \\stdFace_i ^1 \\times \\ldots \\times \\stdFace_i  ^r$,\r\n\t\\item any $\\stdInt^* \\in \\reInt \\stdFace _i^*$,\r\n\t\\item any $\\stdInt$  such that $\\stdInt _j = 0$ if $\\stdFace _i^j$ is polyhedral and \t$\\stdInt _j \\in \\reInt \\stdFace _i^j$, otherwise.\r\n\\end{itemize}\r\n\r\n\\item If $t^* = 0$ and  $\\inProd{c}{x^*} < 0$, let $\\minFaceD \\leftarrow \\emptyset$ and stop. \\eqref{eq:dual}  is infeasible.\r\n\\item If $t^* = 0$ and $\\inProd{c}{x^*} = 0$, let $d_i \\leftarrow x^*, \\stdFace _{i+1}  \\leftarrow \\stdFace _i  \\cap \\{d_i\\}^\\perp, i\\leftarrow i+1$ and return to 2).\r\n\\item If $t^* > 0$, $s' \\leftarrow  c  -\\stdMap^\\T  \\frac{y_3^*}{y_1^*}$, $\\stdFace \\leftarrow \\stdFace _i$ and stop. {\\PPS} condition is satisfied.\r\n\\end{enumerate}\r\n\n\n[{\\bf Facial Reduction Poly - Phase 2}]\r\n\r\n{\\bf Input:} \\eqref{eq:dual}, the output of Phase 1: $\\stdFace$ and $s'$, with $\\stdFace \\neq \\emptyset$.\r\n\r\n{\\bf Output:} $\\minFaceD$, a dual feasible slack $\\hat s \\in \\reInt \\minFaceD$ and, perhaps, \r\nan extra reducing direction $d$.\r\n\\begin{enumerate}\r\n\\item Let $\\hat \\stdCone = {\\hat \\stdCone} ^1\\times \\ldots \\times {\\hat \\stdCone} ^r$ such that ${\\hat \\stdCone} ^j = \\stdFace ^j$ if \r\n$\\stdFace ^j$ is polyhedral and $\\hat \\stdCone ^j = \\spanVec \\stdFace^j$ otherwise. \r\n\\item Let $(x^*,t^*,w^*)$ and $(y_1^*,y_2^*,y_3^*)$ be\r\nany pair of \\emph{strictly complementary} optimal solutions to \\eqref{eq:red} and \\eqref{eq:red_dual} with\r\n\\begin{itemize}\r\n\t\\item $\\hat \\stdCone$ in place of $\\stdCone$,\r\n\t\\item any $\\stdInt \\in \\reInt \\hat \\stdCone$,\r\n\t\\item any $\\stdInt^* \\in \\reInt {\\hat \\stdCone}^* $.\r\n\\end{itemize}\r\n\\item If $t^* = 0$, let $d \\leftarrow x^*$, $\\minFaceD \\leftarrow \\stdFace \\cap \\{x^*\\}^\\perp$.\r\nLet $\\tilde s$ be $c  -\\stdMap^\\T  \\frac{y_3^*}{y_1^*} $. Then, we let \r\n$\\hat s$ be a convex combination of $\\tilde s$ and $s'$ such that $\\hat s \\in \\reInt \\minFaceD$ and stop.\\footnote{\\label{foot}In Proposition \\ref{prop:min_face} it is shown that if $z_\\beta = \\beta s' + (1-\\beta)\\tilde s $ is such that $\\beta \\in (0,1)$ and $\\beta$ is sufficiently close to $1$ then $z_\\beta \\in  \\reInt \\minFaceD$. Therefore, after determining $\\minFaceD$, $\\hat s$ can be found through a simple search procedure, e.g., the bisection method.} %by starting with $\\beta = 0.5$ and letting $\\beta$ be $0.75,0.875,0.9375, \\ldots$ and so on, until $z_\\beta \\in \\reInt \\minFaceD$.  }\r\n\\item If $t^* > 0$,  $\\minFaceD \\leftarrow \\stdFace$.\r\nLet $\\tilde s$ be $c  -\\stdMap^\\T  \\frac{y_3^*}{y_1^*} $. Then, we let \r\n$\\hat s$ be a convex combination of $\\tilde s$ and $s'$ such that $\\hat s \\in \\reInt \\minFaceD$ and stop.\\footref{foot}%\\footnote{See footnote \\ref{foot}.}\r\n\r\n\\end{enumerate}\r\n\r\n\r\nNote that in Phase 2, the cone $\\hat{\\stdCone}$ is polyhedral, therefore, \r\nboth \\eqref{eq:red_dual} and \\eqref{eq:red} are \r\npolyhedral problems. Therefore,  strictly complementary solutions are ensured to exist, which is a \r\nconsequence of Goldman-Tucker Theorem and also follows from the results of \r\nMcLinden \\cite{McLinden82} and Akg\\\"ul \\cite{Akgul84}. \r\nWe also remark that a strictly complementary solution of a polyhedral problem can be found by solving \r\na single linear program, see, for instance, the article by Freund, Roundy and Todd \\cite{FRT85} and the related \r\nwork by Mehrotra and Ye \\cite{MY93}. \r\n\n\nLet $f:\\Re^n \\to \\Re\\cup \\{-\\infty,+\\infty \\}$ be a convex function.\r\nWe denote the domain of $f$ by $\\fDom f = \\{x \\in \\Re^n \\mid f(x) < \\infty \\}$. If $\\fDom f \\neq \\emptyset $ and $f$ is never $-\\infty$, then $f$ is said to be proper.\r\nIts conjugate will be denoted by $f^*$ and it satisfies $f^*(s) = \\sup _{x} \\inProd{s}{x} - f(x)$.\r\nIf the epigraph of $f$ is a polyhedral set, then $f$ is said to be a polyhedral function. \r\nWe recall Theorem 20.1 from \\cite{rockafellar}.\r\n\r\n\\begin{theorem}[Rockafellar]\\label{theo:conv}\r\nLet $f_1, \\ldots, f_m$ be proper convex functions and let $f_{k+1}, \\ldots ,$ $f_m$ be \r\npolyhedral functions. Suppose also that\r\n$$\r\n\\reInt (\\fDom f_1) \\cap \\cdots \\cap \\reInt (\\fDom f_k) \\cap \\fDom f_{k+1} \\cap \\cdots \\cap \\fDom f_m \\neq \\emptyset.\r\n$$\r\nThen the following holds:\r\n\\begin{equation*}\r\n(f_1 + \\cdots + f_m)^*(s) = \\inf \\{ f_1^*(s_1) + \\cdots + f_m^*(s_m) \\mid s_1 + \\cdots + s_m = s \\},\r\n\\end{equation*}\r\nwhere for each $s$ the infimum is attained whenever it is finite.\r\n\\end{theorem}\r\n","units":[{"raw_tex":"Consider the following pair of primal and dual conic linear programs (CLPs):\r\n\\vspace*{-3\\baselineskip} \r\n\\begin{multicols}{2}\r\n\\begin{align}\r\n\\underset{x}{\\inf} & \\quad \\inProd{c}{x} \\label{eq:primal}\\tag{P}\\\\ \r\n\\mbox{subject to} & \\quad \\stdMap x = b \\nonumber \\\\ \r\n&\\quad x \\in \\stdCone ^* \\nonumber \r\n\\end{align}\r\n\t\\break\r\n\t\\begin{align}\r\n\t\\underset{y}{\\sup} & \\quad \\inProd{b}{y} \\label{eq:dual} \\tag{D} \\\\ \r\n\t\\mbox{subject to} & \\quad c - \\stdMap ^\\T y \\in \\stdCone \\nonumber,\\\\ \\nonumber\r\n\t\\end{align}\t\r\n\\end{multicols}\r\n\\noindent where $\\stdCone \\subseteq \\Re^n$ is a closed convex cone and $\\stdCone^*$ is the dual cone $\\{s \\in \\Re^n \\mid \\inProd{s}{x} \\geq 0, \\forall x \\in \\stdCone \\}$. We have \r\nthat $\\stdMap: \\Re^n \\to \\Re^m$ is a linear map, $b \\in \\Re^m$, $c \\in \\Re^n$ and \r\n$\\stdMap^\\T$ denotes the adjoint map. We also have $\\stdMap^\\T y = \\sum _{i=1}^m \\stdMap _i y_i$, for certain \r\nelements $\\stdMap _i \\in \\Re^n$. The  inner product is denoted by $\\inProd{\\cdot}{\\cdot}$.\r\nWe will use $\\pOpt$ and $\\dOpt$ to denote the primal and  dual optimal value, respectively.\r\nIt is understood that $\\pOpt = +\\infty$ if \\eqref{eq:primal} is infeasible and \r\n$\\dOpt = -\\infty$ if \\eqref{eq:dual} is infeasible.\r\n","start":6343,"end":7567},{"raw_tex":"In the absence of either a primal relative interior feasible solution or a dual \r\nrelative interior slack, it is possible that $\\pOpt \\neq \\dOpt$. A possible \r\nway of  correcting that is to let $\\minFaceD$ be \r\nthe minimal face of $\\stdCone$ which contains the feasible slacks $\\feasS = \\{c- \\stdMap ^\\T y \\in \\stdCone \\mid y \\in \\Re^m  \\}$, then \r\nwe substitute $\\stdCone$ by $\\minFaceD$ and $\\stdCone^*$ by $(\\minFaceD)^*$. With that, \r\nthe new primal optimal value $\\pOpt ' $ will satisfy $\\pOpt ' = \\dOpt$. This is precisely what \r\nfacial reduction \\cite{Borwein1981495, pataki_strong_2013, article_waki_muramatsu} approaches do.\r\n","start":7573,"end":8208},{"raw_tex":"For $C$ a closed convex set, we will denote by $\\reInt C$ and \r\n$\\closure C$ the relative interior and the closure of $C$, respectively.\r\nIf $\\mathcal{U}$ is an \r\narbitrary set, we denote by $\\mathcal{U}^\\perp$ the subspace which contains \r\nthe elements orthogonal to it. We will denote by $\\PSDcone{n}$ the cone of \r\n$n\\times n$ symmetric positive semidefinite matrices and by $\\SOC{n}$ the \r\nLorentz cone $\\{(x_0,\\overline{x}) \\in \\Re\\times \\Re^{n-1} \\mid x_0 \\geq \\norm{\\overline{x}}_2 \\}$, \r\nwhere $\\norm{.}_2$ is the usual Euclidean norm.\r\nThe nonnegative orthant will be denoted by $\\Re^n_+$. \r\n\r\n\r\n\r\nIf $\\stdCone$ is a closed convex cone, we write $\\stdCone^*$ for its dual \r\ncone. We write $\\spanVec \\stdCone$ for its linear span and $\\lineality \\stdCone$ for \r\nits lineality space, which is $\\stdCone \\cap -\\stdCone$. $\\stdCone$ is said to be \\emph{pointed} if \r\n$\\lineality \\stdCone  = \\{0\\}$. We have $\\lineality (\\stdCone^*) = \\stdCone^\\perp$, see Theorem 14.6 in \\cite{rockafellar}. Also, if we select $\\stdInt \\in \\reInt \\stdCone$ and $x \\in \\stdCone^*$, then $x \\in \\stdCone^\\perp $ if and only if $\\inProd{\\stdInt}{x} = 0$\\footnote{Suppose $\\inProd{\\stdInt}{x} = 0$. Then, given $y \\in \\stdCone$ we have $\\alpha \\stdInt +(1-\\alpha)y \\in \\stdCone$ for some $\\alpha > 1$, due to Theorem 6.4 in \\cite{rockafellar}. Taking the inner product with $x$, we see that $\\inProd{y}{x}$ must be zero.}.\r\n\r\n\r\n\r\nThe conic linear program \\eqref{eq:dual} can be in four different feasibility \r\nstatuses: $i)$ strongly feasible if $\\feasS \\cap \\reInt \\stdCone \\neq \\emptyset$; $ii)$ weakly feasible if \r\n$\\feasS \\neq \\emptyset$ but $\\feasS \\cap \\reInt \\stdCone  = \\emptyset$; $iii)$ weakly infeasible if \r\n$\\feasS = \\emptyset$ but $\\text{dist}(c + \\matRange \\stdMap^\\T ,\\stdCone) = 0$; $iv)$ strongly \r\ninfeasible if $\\text{dist}(c + \\matRange \\stdMap^\\T ,\\stdCone) > 0$.  Note that \r\n\\eqref{eq:primal} admits analogous definitions. \r\n\r\nThe \r\nstrong duality theorem states that if \\eqref{eq:dual} is strongly feasible and \r\n$\\dOpt < + \\infty$, then $\\pOpt = \\dOpt$ and $\\pOpt$ is attained. \r\nOn the other hand, if \\eqref{eq:primal} is strongly feasible and \r\n$\\pOpt > -\\infty$,  then $\\pOpt = \\dOpt$ and $\\dOpt$ is attained. \r\nWe  will also need a special version of the strong duality theorem. \r\nFirst, we need the following definition.\r\n\r\n\\begin{definition}[Partial Polyhedral Slater's condition]\\label{def:pps}\r\n\tLet $\\stdCone = \\stdCone ^1\\times \\stdCone ^2$, where  $\\stdCone ^1\\subseteq \\Re^{n_1},\\stdCone ^2 \\subseteq \\Re^{n_2}$ are closed convex cones such that $\\stdCone ^2$ is polyhedral.\r\n\tWe say that \\eqref{eq:dual} satisfies the Partial Polyhedral Slater's (\\PPS) condition if there is \r\n\ta slack $(s_1,s_2) = c-\\stdMap^\\T y$, such that $s_1 \\in \\reInt \\stdCone^1$ and $s_2 \\in \\stdCone^2$.\r\n\tSimilarly, we say that \\eqref{eq:primal} satisfies the {\\PPS} condition, if there is a primal \r\n\tfeasible solution $x = (x_1,x_2)$ for which $x_1 \\in \\reInt (\\stdCone^1)^*$.\r\n\\end{definition}\r\nThe following is a strong duality theorem based on the {\\PPS} condition.\r\nAs we could not find a precise reference for it, we give a proof in the Appendix \\ref{app:partial}.  \r\n","start":16275,"end":19451},{"raw_tex":"Here, we will suppose that our main interest is in the dual problem \\eqref{eq:dual}.\r\n Facial Reduction hinges on the fact that \r\nstrong feasibility fails  if and only if there \r\nis $d \\in \\stdCone ^*$ such that $\\stdMap d = 0$  and\r\none of the two alternatives holds: $(i)$  $\\inProd{c}{d} = 0$ and $d \\not \\in \\stdCone ^\\perp$; or \r\n$(ii)$ $\\inProd{d}{c} < 0$, see Lemma 3.2 in \\cite{article_waki_muramatsu}. If \r\nalternative $(i)$ holds, $\\stdFace = \\stdCone \\cap \\{d\\}^\\perp$ is a proper face of $\\stdCone$ \r\ncontaining $\\feasS$.  We then substitute  $\\stdCone$ by $\\stdFace $ and repeat.\r\nIf  $(ii)$ holds, \\eqref{eq:dual} is infeasible. \r\n We write below a generic \r\nfacial reduction algorithm similar to the one described in \\cite{article_waki_muramatsu}.\r\n\r\n\\noindent\r\n[{\\bf Generic Facial Reduction}]\\label{alg:gen_fra}\r\n\r\n{\\bf Input:} \\eqref{eq:dual}\r\n\r\n{\\bf Output:} A set of reducing directions $\\{d_1, \\ldots, d_\\ell\\}$ and $\\minFaceD$.\r\n\\begin{enumerate}\r\n\\item $\\stdFace _1 \\leftarrow \\stdCone, i \\leftarrow 1 $\r\n\\item Let $d_i$ be an element in $\\stdFace _i ^* \\cap \\ker \\stdMap$ such that either: \r\n$i)$ $d_i \\not \\in \\stdFace _i^\\perp$ and $\\inProd{c}{d_i} = 0$; or $ii)$ $\\inProd{c}{d_i} < 0$.\r\nIf no such $d_i$ exists, let $\\minFaceD \\leftarrow \\stdFace _i$ and stop. \r\n\\item If $\\inProd{c}{d_i} < 0$, let $\\minFaceD \\leftarrow  \\emptyset$ and stop.\r\n\\item If $\\inProd{c}{d_i} = 0$, let $\\stdFace _{i+1}  \\leftarrow \\stdFace _i  \\cap \\{d_i\\}^\\perp, i\\leftarrow i+1$ and return to 2).\r\n\\end{enumerate}\r\n\r\n\r\nWe will refer to the directions satisfying $d_i \\in \\stdFace _i^* \\cap \\ker \\stdMap$ and \r\n$\\inProd{c}{d_i} \\leq 0$ as \\emph{reducing directions}, so that the $d_i$ in Step 2. are indeed reducing directions. An important issue when doing facial reduction is how to model the \r\nsearch for the reducing directions. It is sometimes said that doing facial \r\nreduction can be as hard as solving the original problem. However, an important \r\ndifference is that the search for the $d _i$ can be cast as a pair of primal and \r\ndual problems which are always strongly feasible.\r\nThis was shown in the work by Cheung, Schurr and Wolkowicz \\cite{csw13} and \r\nin our previous work \\cite{lourenco_muramatsu_tsuchiya3}. \r\n","start":20447,"end":22681},{"raw_tex":"In this article, we will search for reducing directions by\r\nconsidering the pair \\eqref{eq:red} and \\eqref{eq:red_dual} introduced in \\cite{lourenco_muramatsu_tsuchiya3}, which \r\nare parametrized by $\\stdMap$, $c$, $\\stdCone, \\stdInt, \\stdInt^*$. In Phase 1 of FRA-Poly, we will always \r\nselect $\\stdInt$ and $\\stdInt^*$ according to Lemma \\ref{lemma:red2}. Different choices will be discussed/used \r\nin Phase 2 of FRA-Poly and on Sections \\ref{sec:compl} and \\ref{sec:tight}.\r\n\\begin{align}\r\n\\underset{x,t,w}{\\mbox{minimize}} & \\quad t & \\tag{$P_\\stdCone $}\\label{eq:red}  \\\\ \r\n\\mbox{subject to} & \\quad -\\inProd{c}{x  -t\\stdInt^*} + t - w &= 0 \\label{eq:red:1}\\\\\r\n&\\quad \\inProd{\\stdInt}{x} + w &= 1 \\label{eq:red:2}\\\\\r\n& \\quad\\stdMap x  -t\\stdMap \\stdInt^* & = 0 \\label{eq:red:3}\\\\\r\n&\\quad (x,t,w) \\in \\stdCone ^* \\times \\Re_+ \\times \\Re_+ \\nonumber\\\\\r\n\\underset{y_1,y_2,y_3}{\\mbox{maximize}} &\\quad  y_2 & \\tag{$D_\\stdCone $}\\label{eq:red_dual}  \\\\ \r\n\\mbox{subject to} & \\quad cy_1 -\\stdInt y_2 -\\stdMap^\\T y_3 \\in \\stdCone \\label{eq:red_dual:1} \\\\\r\n&\\quad 1-y_1(1+\\inProd{c}{\\stdInt^*}) + \\inProd{\\stdInt^* }{\\stdMap ^\\T y_3}\\geq 0 \\label{eq:red_dual:2} \\\\\r\n& \\quad y_1-y_2 \\geq 0 \\label{eq:red_dual:3} \\\\\r\n&\\quad (y_1,y_2,y_3) \\in \\Re \\times \\Re \\times \\Re^m. \\nonumber\r\n\\end{align}\r\n\r\nRecall that our goal is to find a point $x \\in \\ker \\stdMap \\cap \\stdCone$ satisfying  $\\inProd{c}{x} \\leq 0$. The idea behind  \\eqref{eq:red} is to shift the problem by $-t\\stdInt^*$ (Equations \\eqref{eq:red:1} and \\eqref{eq:red:3}) and add constraints to ensure \r\nthe $x$ stays in a bounded region (Equation \\eqref{eq:red:2}). These changes ensure that \\eqref{eq:red} and \\eqref{eq:red_dual} \r\nsatisfy the PPS condition when the parameters $\\stdInt, \\stdInt^*$ are chosen appropriately as in\r\nthe next section.\r\n","start":23456,"end":25262},{"raw_tex":"We now present a disclaimer of sorts.\r\nThe theoretical results presented in this section and the next stand whether FRA-Poly is doable or \r\nnot for a given $\\stdCone$. \r\nIf we wish to do facial reduction concretely (even if it is by hand!), we need to \r\nmake a few assumptions on our computational capabilities \r\nand on our knowledge on the lattice of faces of $\\stdCone$. \r\nFirst of all, we must be able to solve problems over faces of $\\stdCone$ such that both the primal and the dual \r\nsatisfy the {\\PPS} condition and we must also be able to do \r\nbasic linear algebraic operations.  Also, for \r\neach face $\\stdFace$ of $\\stdCone$ we must know:\r\n\\begin{enumerate}\r\n\t\\item $\\spanVec \\stdFace$, \r\n\t\\item at least one point $\\stdInt \\in \\reInt \\stdFace$,\r\n\t\\item at least one point $\\stdInt^* \\in \\reInt \\stdFace^*$,\r\n\t\\item whether $\\stdFace$ is polyhedral or not.\r\n\\end{enumerate}\r\nWe remark that \r\napart from  knowledge about the polyhedral faces, our assumptions are not very different from what it is \r\nusually assumed \\emph{implicitly} in the FRA literature. \r\nFor symmetric cones, which \r\ninclude direct products of $\\PSDcone{n}$, $\\SOC{n}$ and $\\Re^n_+$, they are reasonable since their \r\nlattice of faces is well-understood and every face is again a symmetric cone. \r\nSo, for instance, $\\stdInt$ can be taken as the identity element for \r\nthe corresponding Jordan algebra. On the other hand, if $\\stdCone$ is, say, the copositive \r\ncone $\\mathcal{C}^n$, we might have some trouble fulfilling the requirements, inasmuch  as our knowledge of \r\nthe faces of $\\mathcal{C}^n$ is still lacking.\r\n\r\n\\subsection{Distance to Polyhedrality}\r\nHere we introduce the notion of \\emph{distance to polyhedrality}. In what follows, \r\nif we have a chain of faces $\\stdFace _1 \\subsetneq \\ldots \\subsetneq \\stdFace _\\ell$, the \r\nlength of the chain is defined to be $\\ell$.\r\n\\begin{definition}\r\n\tLet $\\stdCone$ be a nonempty closed convex cone. The \\emph{distance to polyhedrality} $\\distP(\\stdCone)$ is the \r\n\tlength \\emph{minus one} of the longest strictly ascending chain of nonempty faces $\\stdFace _1 \\subsetneq \\ldots \\subsetneq \\stdFace _{\\ell} $ which satisfies:\r\n\t\\begin{enumerate}\r\n\t\t\\item  $\\stdFace _1$ is polyhedral;  \r\n\t\t\\item $\\stdFace _j$ is not polyhedral for $j > 1$.\r\n\t\\end{enumerate}\r\n\\end{definition}\r\n\r\nThe distance to polyhedrality is a well-defined concept, because the lineality space\r\nof $\\stdCone$ is always a polyhedral face of $\\stdCone$.\r\nMoreover, $\\distP(\\stdCone)$ counts the maximum number of facial reduction steps that \r\ncan be taken before we reach a polyhedral face.\r\n","start":41456,"end":44052},{"raw_tex":"For general $\\stdCone$, these equivalencies may not hold and we might need \r\nto distinguish between primal and dual strict complementarity, see \r\nfor instance, Definition 3.4 and Remark 3.6 in the chapter by Pataki \\cite{pataki_handbook} and Equation (2.6) in Section 2 of the work by Tun\\c{c}el and Wolkowicz \\cite{TW12}. \r\nBased on those references, we will say \r\nthat \\eqref{eq:red} and \\eqref{eq:red_dual}  satisfies  \r\n\\emph{(dual) strict complementarity} if $\\opt{\\text{\\ref{eq:red}}} = \\opt{\\text{\\ref{eq:red_dual}}}$\r\nand there are optimal solutions $(x^*, t^*,w^*), (y_1^*,y_2^*,y_3^*)$ such that\r\n \\begin{align}\r\n cy_1^* -\\stdInt y_2^* -\\stdMap^\\T y_3^* &\\in \\reInt (\\stdCone \\cap\\{ x^*\\}^\\perp)  \\label{eq:strict1}\\\\\r\n t^* +  1-y_1^*(1+\\inProd{c}{\\stdInt^*}) + \\inProd{\\stdInt^* }{\\stdMap ^\\T y_3^*}&> 0  \\label{eq:strict2}\\\\\r\n w^* + y_1^*-y_2^* & > 0. \\label{eq:strict3} \r\n \\end{align}\r\n","start":45905,"end":46804},{"raw_tex":"Henceforth, we will assume that $\\stdCone$ is the product of $r$ cones and we will \r\nwrite $\\stdCone = \\stdCone ^1 \\times \\ldots \\times \\stdCone  ^r $. Then, recall that \r\nif $\\stdFace$ is face of $\\stdCone$, we can write $\\stdFace = \\stdFace ^1 \\times \\ldots \\times \\stdFace  ^r$, where $\\stdFace ^i$ is a face of $\\stdCone ^i$ for every $i$.\r\n\r\nConsider the following \r\nFRA variant, which we call FRA-Poly.\r\n\r\n\\noindent\r\n[{\\bf Facial Reduction Poly - Phase 1}]\r\n\r\n{\\bf Input:} \\eqref{eq:dual}\r\n\r\n{\\bf Output:} A set of reducing directions $\\{d_1, \\ldots, d_\\ell\\}$. If \r\n\\eqref{eq:dual} is feasible, it outputs  some face $\\stdFace \\subseteq \\stdCone$ \r\nfor which the PPS condition holds, together \r\nwith a dual slack $s'$ for which $s_j' \\in \\reInt \\stdFace ^j $ for every $j$ such that $\\stdFace ^j$  is nonpolyhedral.\r\nIf \\eqref{eq:dual} is infeasible, the directions form a certificate of infeasibility.\r\n\\begin{enumerate}\r\n\\item $\\stdFace _1 \\leftarrow \\stdCone, i \\leftarrow 1 $\r\n\\item Let  $(x^*,t^*,w^*)$ and $(y_1^*,y_2^*,y_3^*)$ be any pair of optimal solutions to \r\n\\eqref{eq:red} and \\eqref{eq:red_dual} with\r\n\\begin{itemize}\r\n\t\\item $\\stdFace _i$ in place of $\\stdCone $, where $\\stdFace_i = \\stdFace_i ^1 \\times \\ldots \\times \\stdFace_i  ^r$,\r\n\t\\item any $\\stdInt^* \\in \\reInt \\stdFace _i^*$,\r\n\t\\item any $\\stdInt$  such that $\\stdInt _j = 0$ if $\\stdFace _i^j$ is polyhedral and \t$\\stdInt _j \\in \\reInt \\stdFace _i^j$, otherwise.\r\n\\end{itemize}\r\n\r\n\\item If $t^* = 0$ and  $\\inProd{c}{x^*} < 0$, let $\\minFaceD \\leftarrow \\emptyset$ and stop. \\eqref{eq:dual}  is infeasible.\r\n\\item If $t^* = 0$ and $\\inProd{c}{x^*} = 0$, let $d_i \\leftarrow x^*, \\stdFace _{i+1}  \\leftarrow \\stdFace _i  \\cap \\{d_i\\}^\\perp, i\\leftarrow i+1$ and return to 2).\r\n\\item If $t^* > 0$, $s' \\leftarrow  c  -\\stdMap^\\T  \\frac{y_3^*}{y_1^*}$, $\\stdFace \\leftarrow \\stdFace _i$ and stop. {\\PPS} condition is satisfied.\r\n\\end{enumerate}\r\n","start":49239,"end":51166},{"raw_tex":"[{\\bf Facial Reduction Poly - Phase 2}]\r\n\r\n{\\bf Input:} \\eqref{eq:dual}, the output of Phase 1: $\\stdFace$ and $s'$, with $\\stdFace \\neq \\emptyset$.\r\n\r\n{\\bf Output:} $\\minFaceD$, a dual feasible slack $\\hat s \\in \\reInt \\minFaceD$ and, perhaps, \r\nan extra reducing direction $d$.\r\n\\begin{enumerate}\r\n\\item Let $\\hat \\stdCone = {\\hat \\stdCone} ^1\\times \\ldots \\times {\\hat \\stdCone} ^r$ such that ${\\hat \\stdCone} ^j = \\stdFace ^j$ if \r\n$\\stdFace ^j$ is polyhedral and $\\hat \\stdCone ^j = \\spanVec \\stdFace^j$ otherwise. \r\n\\item Let $(x^*,t^*,w^*)$ and $(y_1^*,y_2^*,y_3^*)$ be\r\nany pair of \\emph{strictly complementary} optimal solutions to \\eqref{eq:red} and \\eqref{eq:red_dual} with\r\n\\begin{itemize}\r\n\t\\item $\\hat \\stdCone$ in place of $\\stdCone$,\r\n\t\\item any $\\stdInt \\in \\reInt \\hat \\stdCone$,\r\n\t\\item any $\\stdInt^* \\in \\reInt {\\hat \\stdCone}^* $.\r\n\\end{itemize}\r\n\\item If $t^* = 0$, let $d \\leftarrow x^*$, $\\minFaceD \\leftarrow \\stdFace \\cap \\{x^*\\}^\\perp$.\r\nLet $\\tilde s$ be $c  -\\stdMap^\\T  \\frac{y_3^*}{y_1^*} $. Then, we let \r\n$\\hat s$ be a convex combination of $\\tilde s$ and $s'$ such that $\\hat s \\in \\reInt \\minFaceD$ and stop.\\footnote{\\label{foot}In Proposition \\ref{prop:min_face} it is shown that if $z_\\beta = \\beta s' + (1-\\beta)\\tilde s $ is such that $\\beta \\in (0,1)$ and $\\beta$ is sufficiently close to $1$ then $z_\\beta \\in  \\reInt \\minFaceD$. Therefore, after determining $\\minFaceD$, $\\hat s$ can be found through a simple search procedure, e.g., the bisection method.} %by starting with $\\beta = 0.5$ and letting $\\beta$ be $0.75,0.875,0.9375, \\ldots$ and so on, until $z_\\beta \\in \\reInt \\minFaceD$.  }\r\n\\item If $t^* > 0$,  $\\minFaceD \\leftarrow \\stdFace$.\r\nLet $\\tilde s$ be $c  -\\stdMap^\\T  \\frac{y_3^*}{y_1^*} $. Then, we let \r\n$\\hat s$ be a convex combination of $\\tilde s$ and $s'$ such that $\\hat s \\in \\reInt \\minFaceD$ and stop.\\footref{foot}%\\footnote{See footnote \\ref{foot}.}\r\n\r\n\\end{enumerate}\r\n\r\n\r\nNote that in Phase 2, the cone $\\hat{\\stdCone}$ is polyhedral, therefore, \r\nboth \\eqref{eq:red_dual} and \\eqref{eq:red} are \r\npolyhedral problems. Therefore,  strictly complementary solutions are ensured to exist, which is a \r\nconsequence of Goldman-Tucker Theorem and also follows from the results of \r\nMcLinden \\cite{McLinden82} and Akg\\\"ul \\cite{Akgul84}. \r\nWe also remark that a strictly complementary solution of a polyhedral problem can be found by solving \r\na single linear program, see, for instance, the article by Freund, Roundy and Todd \\cite{FRT85} and the related \r\nwork by Mehrotra and Ye \\cite{MY93}. \r\n","start":59177,"end":61740},{"raw_tex":"Let $f:\\Re^n \\to \\Re\\cup \\{-\\infty,+\\infty \\}$ be a convex function.\r\nWe denote the domain of $f$ by $\\fDom f = \\{x \\in \\Re^n \\mid f(x) < \\infty \\}$. If $\\fDom f \\neq \\emptyset $ and $f$ is never $-\\infty$, then $f$ is said to be proper.\r\nIts conjugate will be denoted by $f^*$ and it satisfies $f^*(s) = \\sup _{x} \\inProd{s}{x} - f(x)$.\r\nIf the epigraph of $f$ is a polyhedral set, then $f$ is said to be a polyhedral function. \r\nWe recall Theorem 20.1 from \\cite{rockafellar}.\r\n\r\n\\begin{theorem}[Rockafellar]\\label{theo:conv}\r\nLet $f_1, \\ldots, f_m$ be proper convex functions and let $f_{k+1}, \\ldots ,$ $f_m$ be \r\npolyhedral functions. Suppose also that\r\n$$\r\n\\reInt (\\fDom f_1) \\cap \\cdots \\cap \\reInt (\\fDom f_k) \\cap \\fDom f_{k+1} \\cap \\cdots \\cap \\fDom f_m \\neq \\emptyset.\r\n$$\r\nThen the following holds:\r\n\\begin{equation*}\r\n(f_1 + \\cdots + f_m)^*(s) = \\inf \\{ f_1^*(s_1) + \\cdots + f_m^*(s_m) \\mid s_1 + \\cdots + s_m = s \\},\r\n\\end{equation*}\r\nwhere for each $s$ the infimum is attained whenever it is finite.\r\n\\end{theorem}\r\n","start":115268,"end":116300}]},"statement":{"raw_tex":"\\begin{theorem}\\label{theo:fra_bound}\r\nLet $\\stdCone = \\stdCone ^1\\times \\ldots \\times \\stdCone^r$. The minimum face $\\minFaceD$ \r\nthat contains the feasible region of \\eqref{eq:dual} can be found in no more than \r\n$1+ \\sum _{i=1}^{r} \\distP(\\stdCone ^i) $ facial reduction steps.\r\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\\label{theo:fra_bound}\r\nLet $\\stdCone = \\stdCone ^1\\times \\ldots \\times \\stdCone^r$. The minimum face $\\minFaceD$ \r\nthat contains the feasible region of \\eqref{eq:dual} can be found in no more than \r\n$1+ \\sum _{i=1}^{r} \\distP(\\stdCone ^i) $ facial reduction steps.\r\n\\end{theorem}","start":66021,"end":66316}]},"proof":{"raw_tex":"\\begin{proposition}[{\\PPS}-Strong Duality]\\label{prop:slater2}\r\n\tLet $\\stdCone = \\stdCone ^1\\times \\stdCone ^2$, where  $\\stdCone ^1\\subseteq \\Re^{n_1},\\stdCone ^2 \\subseteq \\Re^{n_2}$ are closed convex cones such that $\\stdCone ^2$ is polyhedral.\r\n\t\\begin{enumerate}[label=(\\roman*)]\r\n\t\t\\item If $\\pOpt$ is finite and \\eqref{eq:primal} satisfies the {\\PPS} condition, then $\\pOpt = \\dOpt$ and the dual optimal value is attained.\r\n\t\t\\item If $\\dOpt$ is finite and \\eqref{eq:dual} satisfies the {\\PPS} condition, then $\\pOpt = \\dOpt$ and the primal \r\n\t\toptimal value is attained.\r\n\t\\end{enumerate}\r\n\\end{proposition}\n\n\\begin{lemma}\\label{lemma:red2}\r\nLet $\\stdCone = \\stdCone ^1 \\times \\stdCone ^2$ be a \r\nclosed convex cone, such that $\\stdCone ^2$ is polyhedral.\r\nConsider the pair \\eqref{eq:red} and \\eqref{eq:red_dual} with \r\n$\\stdInt$ and $\\stdInt^*$ such that  $\\stdInt = (\\stdInt_1,0) \\in (\\reInt \\stdCone ^1)\\times \\{0\\}$ and \r\n$\\stdInt^* \\in \\reInt \\stdCone^*$.  The following properties hold.\r\n\\begin{enumerate}[label=(\\roman*)]\r\n\\item The following are solutions to \\eqref{eq:red} and \\eqref{eq:red_dual} that satisfy the {\\PPS} condition: \r\n\\begin{align*}\r\n(x,t,w)& = \\left(\\frac{\\stdInt^*}{\\inProd{\\stdInt}{\\stdInt^*}+1},\\frac{1}{\\inProd{\\stdInt}{\\stdInt^*}+1}, \\frac{1}{\\inProd{\\stdInt}{\\stdInt^*}+1}\\right) \\\\\r\n (y_1,y_2,y_3) & = (0,-1,0).\r\n \\end{align*} \r\nIn particular, \r\n$\\opt{\\text{\\ref{eq:red}}} =  \\opt{\\text{\\ref{eq:red_dual}}}$.\r\n\\end{enumerate}\r\nLet $(x^*,t^*,w^*)$ be any primal optimal solution and \r\n$(y_1^*,y_2^*,y_3^*)$ be any dual optimal solution.\r\n \\begin{enumerate}[label=(\\roman*)]\r\n \\setcounter{enumi}{1} \r\n\\item $\\opt{\\text{\\ref{eq:red}}} =  \\opt{\\text{\\ref{eq:red_dual}}} = 0$ if and only if \r\none of the two alternatives below holds:\r\n\\begin{enumerate}\r\n\\item $\\inProd{c}{x^*} < 0 $ and $\\feasS = \\minFaceD = \\emptyset$, or\r\n\\item $\\inProd{c}{x^*} = 0 $, $\\feasS \\subseteq  \\stdCone \\cap \\{x^*\\}^\\perp \\subsetneq \\stdCone$ and \r\n$x^*_1 \\not \\in (\\stdCone^1)^\\perp = \\lineality( (\\stdCone^1 )^*)$.\r\n\\end{enumerate}\r\n\r\n\\item $\\opt{\\text{\\ref{eq:red}}} =  \\opt{\\text{\\ref{eq:red_dual}}} >  0$ if and only if the {\\PPS} condition is satisfied for \\eqref{eq:dual}. In this case, we have\r\n$\r\nc  -\\stdMap^\\T \\frac{y_3^*}{y_1^*} \\in (\\reInt \\stdCone ^1)\\times \\{0\\}. \r\n$ (In particular, it is feasible for \\eqref{eq:dual})\r\n\r\n\\end{enumerate}\r\n\\end{lemma}\n\n\\begin{proof}\r\n\r\n\\begin{enumerate}[label=({\\it{\\roman*}})]\r\n\\item  Due to choice of $\\stdInt$ and $\\stdInt^*$, it is clear that the solutions \r\nmeet the PPS condition.\r\n\r\n\r\n\\item \\fbox{$\\opt{\\text{\\ref{eq:red}}} = 0 \\Rightarrow$  (a) or (b) holds.}\r\nSuppose that $\\opt{\\text{\\ref{eq:red}}} = 0$ and let $(x^*,0,w^*)$ be an optimal solution for \\eqref{eq:red}.\r\nDue to the constraints in \\eqref{eq:red}, we  have\r\n$$\\stdMap x^* = 0, \\qquad -\\inProd{c}{x^*} = w^* \\geq 0, \\qquad x^* \\in \\stdCone ^*.$$ \r\nNote that if $\\inProd{c}{x^*} < 0$, then $\\minFaceD = \\feasS = \\emptyset$. This is \r\nalternative (a).\r\n\r\nIf $\\inProd{c}{x^*} = 0$, then (\\eqref{eq:red:1}) implies \r\n$w^* = 0$. Since $\\stdMap x^* = 0$, we obtain  \r\n$$\\feasS \\subseteq  \\stdCone \\cap \\{x^*\\}^\\perp. $$\r\nBy \\eqref{eq:red:2}, we have $\\inProd{\\stdInt}{x^*} = 1$, so that $x^* = (x_1^*,x_2^*)$ satisfies $x_1^* \\not \\in (\\stdCone^1)^\\perp$, due to the choice of $\\stdInt$. \r\nTherefore, the inclusion $\\stdCone \\cap \\{x^*\\}^\\perp \\subseteq \\stdCone$ is strict, \r\nthat is,  $\\stdCone \\cap \\{x^*\\}^\\perp \\subsetneq \\stdCone$. This is alternative (b).\r\n\r\n\\fbox{(a) or (b) holds $\\Rightarrow \\opt{\\text{\\ref{eq:red}}} = 0$.} \r\nFirst recall that  $t$ is constrained to be nonnegative, therefore  $\\opt{\\text{\\ref{eq:red}}} \\geq 0$.\r\nNext, we will prove the following inequality\r\n\\begin{equation}\\label{eq:lemma:aux}\r\n\t{\\inProd{\\stdInt}{x^*} - \\inProd{c}{x^*}} > 0.\r\n\\end{equation}\r\nSince $\\stdInt \\in \\stdCone,x^* \\in \\stdCone^*$, we have \r\n$\\inProd{\\stdInt}{x^*} \\geq 0$. If $(a)$ holds, then $- \\inProd{c}{x^*}> 0$.\r\nIn particular, \\eqref{eq:lemma:aux} holds.\r\nNow, we consider the case where (b) holds. \r\nIn this case, we have $\\inProd{c}{x^*} = 0$. \r\nIf $\\inProd{\\stdInt}{x^*}$ is also zero, then $\\inProd{\\stdInt_1}{x_1^*}$ is zero and since $\\stdInt_1 \\in \\reInt \\stdCone^1$, $ x^*_1$ must \r\nbelong to $(\\stdCone^1)^\\perp$. However, this is not allowed \r\nsince we assumed that $x_1^* \\not \\in (\\stdCone^1)^\\perp$.\r\nTherefore, we have $\\inProd{\\stdInt}{x^*} > 0$ and, in particular,  \\eqref{eq:lemma:aux} also holds.\r\n\r\nLet  $$\r\n\\alpha = \\frac{1}{\\inProd{\\stdInt}{x^*} - \\inProd{c}{x^*}}.\r\n$$\r\nThen $\\alpha > 0$ and \r\n$(\\alpha x^*,0,-\\alpha \\inProd{c}{x^*})$ is a solution to \\eqref{eq:red} with value $0$, \r\nso that $\\opt{\\text{\\ref{eq:red}}} = 0$.\r\n\r\n\\item \r\n\\fbox{ $\\opt{\\text{\\ref{eq:red_dual}}} > 0 \\Rightarrow $ PPS holds for \\eqref{eq:dual}}\r\nFrom \\eqref{eq:red_dual:1}, we have\r\n$$\r\nc y_1^* - \\stdInt y_2^* - {\\stdMap}^\\T y_3^* \\in \\stdCone.\r\n$$\r\nSince $y_2^*=\\opt{\\text{\\ref{eq:red_dual}}}\r\n>0$, we have, due to \\eqref{eq:red_dual:3}, $y_1^* > 0$.\r\n Hence, \r\n $$c - \\stdInt \\frac{y_2^*}{y_1^*} - {\\stdMap}^\\T \\frac{y_3^*}{y_1^*} \\in \\stdCone.$$ \r\nWe conclude that \r\n$c - {\\stdMap}^\\T y_3^*/y_1^* \\in (\\stdCone^1 +y_2^*/y_1^* \\stdInt_1)\\times  \\{0\\} \\subseteq (\\reInt \\stdCone ^1)\\times \\{0\\}$, due to the choice of  $\\stdInt$ and the fact that $y_2^*/y_1^* > 0$. \r\n\r\n\\fbox{ $\\opt{\\text{\\ref{eq:red_dual}}} = 0 \\Rightarrow $ PPS does not hold for \\eqref{eq:dual}  } \r\nIf $t^* = 0$ and $(x^*,0,w^*)$ is an optimal solution \r\nfor \\eqref{eq:red}, then either $(a)$ or $(b)$ of item $(ii)$ is satisfied. \r\nIf $(a)$ is satisfied, then \\eqref{eq:dual} is infeasible and thus the PPS condition\r\ncannot hold. \r\nIf $(b)$ is satisfied, then $\\inProd{c}{x^*} = 0$,\r\n$\\stdMap x^* = 0$ and $x_1^* \\not \\in (\\stdCone ^1)^\\perp$. If \r\n$(s_1,s_2)$ is a feasible slack for \\eqref{eq:dual}, we have $\\inProd{s_1}{x_1^*} + \\inProd{s_2}{x_2^*} = 0$, so that \r\n$\\inProd{s_1}{x_1^*} = 0$.  \r\nAs $x_1^* \\not \\in (\\stdCone ^1)^\\perp$, we have that $s_1 \\not \\in \\reInt \\stdCone^1$. This means\r\nthat there is no feasible solution for \\eqref{eq:dual} satisfying the PPS\r\ncondition.\r\n\\end{enumerate}\r\n\\end{proof}\n\n\\begin{proposition}\\label{prop:strict}\r\nSuppose $\\opt{\\text{\\ref{eq:red}}} = \\opt{\\text{\\ref{eq:red_dual}}} = 0$  and that  we have  optimal solutions to \\eqref{eq:red} and \r\n\\eqref{eq:red_dual} satisfying dual strict complementarity. \r\nIf $w^* = 0$, then $\\minFaceD = \\stdCone \\cap\\{ x^*\\}^\\perp$.\r\n%Reciprocally, suppose that $\\minFace{D} \\neq \\stdCone $ and that there is a reducing direction $d$ for which $\\minFaceD = \\stdCone \\cap\\{ d\\}^\\perp$. Then, \r\n%\\eqref{eq:red} and \\eqref{eq:red_dual} satisfy dual strict complementarity if $\\stdInt$ is chosen as to satisfy $\\inProd{\\stdInt}{d} \\neq 0$ (in particular, if \r\n%$\\stdInt \\in \\reInt \\stdCone$).\r\n\\end{proposition}\n\n\\begin{proof}\r\nSince $w^* = 0$, \\eqref{eq:strict3} implies that $y_1^* > y_2^*$. Then, $\\opt{\\text{\\ref{eq:red}}} = \\opt{\\text{\\ref{eq:red_dual}}} $ and \r\n$\\opt{\\text{\\ref{eq:red}}} \\geq 0 $ implies that $y_2^* \\geq 0$, so that $y_1^* > 0$ as well.\r\nTherefore, from \\eqref{eq:strict1} we obtain $c -\\stdMap^\\T \\frac{y_3^*}{y_1^*} \\in \\reInt (\\stdCone \\cap\\{ x^*\\}^\\perp).$\r\n%To prove converse, let $y$ be such that $c - \\stdMap^\\T y \\in \\reInt (\\stdCone \\cap\\{ d\\}^\\perp)$. Then, for $\\alpha > 0$ sufficiently small, we have \r\n%that $(d/\\inProd{e}{d},0,0)$, $(\\alpha,0,\\alpha y)$ are optimal solutions satisfying strict complementarity.\r\n\\end{proof}\n\n\\begin{proposition}\\label{prop:phase1}\r\nThe following hold.\r\n\\begin{enumerate}[label=(\\roman*)]\r\n\t\\item if \\eqref{eq:dual} is feasible, then the output face $\\stdFace$ satisfies \r\n\t$\\minFaceD \\subseteq \\stdFace$. Moreover, $s'$ is a dual \r\nfeasible slack such that $s_j' \\in \\reInt \\stdFace ^j $  for every $j$ such that $\\stdFace ^j$  is nonpolyhedral, i.e., the PPS condition is satisfied for $\\stdFace$.\r\n\r\nIn this case, Phase 1  stops after finding at most $\\sum _{i=1}^{r} \\distP(\\stdCone ^i) $ directions. \r\n\t\\item \\eqref{eq:dual} is infeasible if and only if Step 3 is reached. In this case, Phase 1 stops after finding \r\n\tat most $1+ \\sum _{i=1}^{r} \\distP(\\stdCone ^i) $ directions.\r\n\\end{enumerate}\r\n\\end{proposition}\n\n\\begin{proof}\r\nWe will focus on the statements about the bounds, since the other statements \r\nare direct consequences of \r\nLemma \\ref{lemma:red2}. Note that whenever Step 4 is reached, we have \r\n$\\stdFace _{i+1} \\subsetneq \\stdFace _i$, since \r\n$x^*_j \\not \\in (\\stdFace _i^j)^\\perp$ for at least one nonpolyhedral cone $\\stdFace _i^j$, due to item $(ii)$-$(b)$ of Lemma \\ref{lemma:red2}. \r\nTherefore, whenever a new (proper) face is found, it is because \r\nwe are making progress towards a polyhedral face for at least one nonpolyhedral cone. \t\r\n\t\r\nBy definition, after finding $\\hat \\ell = \\sum _{i=1}^{r} (\\distP(\\stdCone ^i)) $ directions, \r\n$\\stdFace _{\\hat \\ell + 1}$ is polyhedral. We now consider what happens if \r\nthe algorithm has not stopped after all these directions were found.\r\nIn this case, when it is time to build \\eqref{eq:red} and \\eqref{eq:red_dual} with $\\stdFace_{l+1}$ in\r\nplace of $\\stdCone$ at Step 2, Phase 1 selects $e = 0$ and $\\stdInt^* \\in (\\stdFace _{\\hat \\ell + 1})^*$.\r\n\r\nFirst, suppose that \\eqref{eq:dual} is feasible and  let \r\n$y$ be such that $c-\\stdMap^\\T y \\in \\feasS$. \r\nSince $\\stdInt = 0$, $(\\alpha, \\alpha, \\alpha y)$\r\nis a feasible solution for \\eqref{eq:red_dual}, when $\\alpha>0$ is sufficiently small. It follows \r\nthat $\\opt{\\text{\\ref{eq:red_dual}}} > 0$ and that Phase 1 eventually reaches Step 5. This gives item $(i)$.\r\n\r\n\r\n\r\nFinally, suppose that \\eqref{eq:dual} is infeasible. By Lemma \\ref{lemma:red2},  $\\opt{\\text{\\ref{eq:red_dual}}} = 0$. Since $\\stdInt = 0$,  \\eqref{eq:red:2} implies that every optimal solution of \r\n\\eqref{eq:red} will be a triple of the form\r\n$(x^*,0,1)$, which implies that Step 3 will be reached and a single new direction will be \r\nadded. This gives  item $(ii)$.\r\n\\end{proof}\n\n\\begin{proposition}\\label{prop:min_face}\r\nThe output face of Phase 2 is $\\minFaceD$ and there exists $\\hat s$ as in Steps 3. and 4.\r\n\\end{proposition}\n\n\\begin{proof}\r\nSuppose that the output face $\\stdFace$ of Phase 1 satisfies $\\stdFace \\neq \\minFaceD$. \r\nBy Lemma 3.2 in \\cite{article_waki_muramatsu}, there is \r\na reducing direction $x$ such that $x \\in \\stdFace ^* \\cap \\ker \\stdMap \\cap \\{c\\}^\\perp$ \r\nand $x \\not \\in \\stdFace ^\\perp$.\r\nDue to Proposition \\ref{prop:phase1}, any such reducing direction $x$ satisfies $\\inProd{x}{s'} = 0$, which implies that \r\n $x_j \\in ({\\stdFace ^j})^\\perp = ({\\spanVec \\stdFace ^j})^\\perp$ for every $j$ such that $\\stdFace ^j$ is not polyhedral, \r\n since $s_j' \\in \\reInt \\stdFace ^j $ for those $j$.\r\nTherefore, the possible reducing directions are confined to the polyhedral cone $\\hat \\stdCone^*$,\r\nwhere $\\hat \\stdCone$ is the cone in Step 1. of Phase 2.\r\n\r\nSince $\\hat \\stdCone$ is polyhedral, the problems \\eqref{eq:red} and \\eqref{eq:red_dual} \r\n are polyhedral and they admit strictly complementary optimal \r\nsolutions $(x^*,t^*,w^*)$, $(y_1^*,y_2^*,y_3^*)$.\r\nThe fact that $x \\not \\in \\stdFace^\\perp$ implies that $\\inProd{\\stdInt}{x}\\neq 0$ so that $(x/\\inProd{\\stdInt}{x},0,0) $ is an optimal \r\nsolution to \\eqref{eq:red}. Therefore, $t^* = y_2^* = 0$. Moreover, since \\eqref{eq:dual} is \r\nfeasible, we have $w^* = 0$.\r\nBy Proposition \\ref{prop:strict}, we have \r\n$c -\\stdMap^\\T  \\frac{y_3^*}{y_1^*} \\in \\reInt (\\hat \\stdCone \\cap \\{x^*\\}^\\perp).$\r\n\r\nLet $\\tilde s = c -\\stdMap^\\T  \\frac{y_3^*}{y_1^*}$. Note that $\\stdFace \\cap \\{x^*\\}^\\perp$ is a face of \r\n$\\stdFace$ containing $\\feasS$, since we argued that $x^*$ must be a reducing direction. We will prove that $\\minFaceD = \\stdFace \\cap \\{x^*\\}^\\perp$ by \r\n showing that some convex combination of $s'$ and $\\tilde s$ lies in $\\reInt (\\stdFace \\cap \\{x^*\\}^\\perp)$. \r\n% Intuitively, $s'$ and $\\tilde s$ are  relative interior points \r\n%of the nonpolyhedral and polyhedral parts of $\\stdFace \\cap \\{x^*\\}^\\perp$, respectively. So some convex \r\n%combination of them will belong to $\\feasS$ and be a relative interior point of $\\stdFace \\cap \\{x^*\\}^\\perp$.\r\n\r\nLet $z_\\beta = \\beta s' + (1-\\beta)\\tilde s$. For \r\nall $\\beta \\in (0,1)$ and all $j$ such that $\\stdFace ^j$ is polyhedral, we \r\nhave $(z_\\beta)_j \\in \\reInt ( \\stdFace ^j  \\cap \\{x_j^*\\}^\\perp)$, because $\\tilde s _j \\in  \\reInt (\\stdFace ^j  \\cap \\{x_j\\}^\\perp)$ and \r\n$s'$ is feasible.\r\nIf $\\stdFace ^j$ is not polyhedral, then $\\stdFace ^j  \\cap \\{x_j^*\\}^\\perp = \\stdFace ^j$, \r\nsince $x_j \\in (\\stdFace ^j)^\\perp$. Because $\\tilde s _j \\in \\spanVec{\\stdFace ^j}$ and \r\n$s_j' \\in \\reInt \\stdFace^j$, for all $\\beta $ sufficiently close to $1$ we have \r\n$(z_\\beta)_j \\in \\reInt \\stdFace^j$. Therefore, it is possible to select $\\beta \\in (0,1)$ \r\nsuch that $(z_\\beta)_j \\in \\reInt (\\stdFace ^j  \\cap \\{x_j\\}^\\perp) $ for all $j$. This \r\nshows that $\\minFaceD = \\stdFace \\cap \\{x^*\\}^\\perp $.\r\n\r\nIf $\\stdFace$ was already the minimal face to begin with, then \r\n$t^* > 0$. We can then  proceed in a \r\nsimilar fashion. The only difference is that due to \\eqref{eq:red_dual:1}, we \r\nwill have that $\\tilde s = c -\\stdMap^\\T  \\frac{y_3^*}{y_1^*} $ satisfies \r\n$\\tilde s _j \\in  \\reInt (\\stdFace ^j)$ for every $j$ such that $\\stdFace ^j$ is polyhedral.\r\nAnd as before, we can select a convex combination of $s'$ and $\\tilde s$ belonging to \r\nthe relative interior of $\\minFaceD$.\r\n\\end{proof}\n\n\\begin{proof}\r\nIf \\eqref{eq:dual} is infeasible, then $\\minFaceD = \\emptyset$ and the result \r\nfollows from Proposition \\ref{prop:phase1}. So suppose now that \r\n\\eqref{eq:dual} is feasible. Then Phase 1 ends after finding at most \r\n$\\sum _{i=1}^{r} \\distP(\\stdCone ^i)$ directions. Due to Proposition \\ref{prop:min_face}, at most one \r\nextra direction is needed to jump to the minimal face.\r\n\\end{proof}\n\n\\begin{proof}\r\nWe will prove $(i)$ first. Let \r\n$f_1$ be such that $f_1(x) = \\inProd{c}{x}$ if $Ax = b$ and $+\\infty$ otherwise.\r\nLet  $f_2$ be the indicator function of $\\Re^{n_1}\\times (\\stdCone ^2)^*$ and $f_3$ be the indicator \r\nfunction of $(\\stdCone ^1)^*\\times \\Re^{n_2}$. Since there is  a primal feasible solution $x = (x_1,x_2)$ such \r\nthat $x_1 \\in \\reInt (\\stdCone ^1)^*$, we have that $\\fDom f_1 \\cap \\fDom f_2 \\cap \\reInt (\\fDom f_3)$ is \r\nnonempty. In addition, $f_1$ and $f_2$ are polyhedral functions. Let us now observe that:\r\n\\begin{align*}\r\nf_1^*(s) = \\left\\{\r\n  \\begin{array}{l l}\r\n    \\inProd{b}{y}  & \\quad \\text{if there is $y$ with $s-c = \\stdMap^\\T y$}\\\\\r\n    +\\infty & \\quad \\text{otherwise}\r\n  \\end{array}\\right. \r\n\\end{align*}\r\nNote that, due to feasibility, for fixed $s$, $\\inProd{b}{y}$ does not depend on the choice \r\nof $y$, as long as $c+\\stdMap ^\\T y = s$. This is because since there is $x$ such that $Ax = b$, we have $\\inProd{b}{y} = \\inProd{x}{s-c}$.\r\nThe conjugate $f_2^*$ is the indicator function of $- \\{0\\}\\times \\stdCone ^2$ and \r\n$f_3^*$ is the indicator function of $- \\stdCone ^1\\times \\{0\\}$. \r\nApplying Theorem \\ref{theo:conv} with $s = 0$, we have:\r\n\\begin{align*}\r\n(f_1+f_2+f_3)^*(0) & = \\inf \\left\\{ \\inProd{b}{y} \\mid  c+ \\stdMap ^\\T y = s_1, s_1 - (s_3,s_2) = 0,  s_2 \\in  \\stdCone ^2, s_3 \\in \\stdCone ^1      \\right\\} \\\\\r\n & = \\inf \\left\\{ \\inProd{b}{y} \\mid  c+ \\stdMap ^\\T y = s_1, s_1 \\in  \\stdCone ^1\\times \\stdCone ^2      \\right\\} \\\\\r\n& = - \\sup \\left\\{ \\inProd{b}{y} \\mid  c -\\stdMap ^\\T y = s_1,  s_1 \\in  \\stdCone ^1 \\times \\stdCone ^2      \\right\\},\r\n\\end{align*}\r\nwhere the $\\sup$ in the last equation is attained. So, there is some dual feasible $y$ such that \r\n$(f_1+f_2+f_3)^*(0) =  \\inProd{b}{y}$. However, using the definition of conjugate, we also have:\r\n\\begin{align*}\r\n(f_1+f_2+f_3)^*(0) = - \\inf \\{ \\inProd{c}{x} \\mid Ax = b, x \\in \\stdCone ^1\\times \\stdCone ^2\\} = -\\theta _P.\r\n\\end{align*}\r\nIt follows that $\\pOpt = \\dOpt$ and the dual is attained at $y$. \r\nTo prove $(ii)$, let $g_1 = f_1^*$, and let $g_2$ and $g_3$ be the indicator \r\nfunctions  of $\\Re^{n_1}\\times \\stdCone ^2$ and  $\\stdCone ^1 \\times \\Re^{n_2}$, respectively. Again, \r\nit is enough to compute $(g_1+g_2+g_3)^*(0)$ using both the definition of conjugate function and \r\nusing Theorem \\ref{theo:conv}. \r\n\\end{proof}","units":[{"raw_tex":"\\begin{proposition}[{\\PPS}-Strong Duality]\\label{prop:slater2}\r\n\tLet $\\stdCone = \\stdCone ^1\\times \\stdCone ^2$, where  $\\stdCone ^1\\subseteq \\Re^{n_1},\\stdCone ^2 \\subseteq \\Re^{n_2}$ are closed convex cones such that $\\stdCone ^2$ is polyhedral.\r\n\t\\begin{enumerate}[label=(\\roman*)]\r\n\t\t\\item If $\\pOpt$ is finite and \\eqref{eq:primal} satisfies the {\\PPS} condition, then $\\pOpt = \\dOpt$ and the dual optimal value is attained.\r\n\t\t\\item If $\\dOpt$ is finite and \\eqref{eq:dual} satisfies the {\\PPS} condition, then $\\pOpt = \\dOpt$ and the primal \r\n\t\toptimal value is attained.\r\n\t\\end{enumerate}\r\n\\end{proposition}","start":19451,"end":20066},{"raw_tex":"\\begin{lemma}\\label{lemma:red2}\r\nLet $\\stdCone = \\stdCone ^1 \\times \\stdCone ^2$ be a \r\nclosed convex cone, such that $\\stdCone ^2$ is polyhedral.\r\nConsider the pair \\eqref{eq:red} and \\eqref{eq:red_dual} with \r\n$\\stdInt$ and $\\stdInt^*$ such that  $\\stdInt = (\\stdInt_1,0) \\in (\\reInt \\stdCone ^1)\\times \\{0\\}$ and \r\n$\\stdInt^* \\in \\reInt \\stdCone^*$.  The following properties hold.\r\n\\begin{enumerate}[label=(\\roman*)]\r\n\\item The following are solutions to \\eqref{eq:red} and \\eqref{eq:red_dual} that satisfy the {\\PPS} condition: \r\n\\begin{align*}\r\n(x,t,w)& = \\left(\\frac{\\stdInt^*}{\\inProd{\\stdInt}{\\stdInt^*}+1},\\frac{1}{\\inProd{\\stdInt}{\\stdInt^*}+1}, \\frac{1}{\\inProd{\\stdInt}{\\stdInt^*}+1}\\right) \\\\\r\n (y_1,y_2,y_3) & = (0,-1,0).\r\n \\end{align*} \r\nIn particular, \r\n$\\opt{\\text{\\ref{eq:red}}} =  \\opt{\\text{\\ref{eq:red_dual}}}$.\r\n\\end{enumerate}\r\nLet $(x^*,t^*,w^*)$ be any primal optimal solution and \r\n$(y_1^*,y_2^*,y_3^*)$ be any dual optimal solution.\r\n \\begin{enumerate}[label=(\\roman*)]\r\n \\setcounter{enumi}{1} \r\n\\item $\\opt{\\text{\\ref{eq:red}}} =  \\opt{\\text{\\ref{eq:red_dual}}} = 0$ if and only if \r\none of the two alternatives below holds:\r\n\\begin{enumerate}\r\n\\item $\\inProd{c}{x^*} < 0 $ and $\\feasS = \\minFaceD = \\emptyset$, or\r\n\\item $\\inProd{c}{x^*} = 0 $, $\\feasS \\subseteq  \\stdCone \\cap \\{x^*\\}^\\perp \\subsetneq \\stdCone$ and \r\n$x^*_1 \\not \\in (\\stdCone^1)^\\perp = \\lineality( (\\stdCone^1 )^*)$.\r\n\\end{enumerate}\r\n\r\n\\item $\\opt{\\text{\\ref{eq:red}}} =  \\opt{\\text{\\ref{eq:red_dual}}} >  0$ if and only if the {\\PPS} condition is satisfied for \\eqref{eq:dual}. In this case, we have\r\n$\r\nc  -\\stdMap^\\T \\frac{y_3^*}{y_1^*} \\in (\\reInt \\stdCone ^1)\\times \\{0\\}. \r\n$ (In particular, it is feasible for \\eqref{eq:dual})\r\n\r\n\\end{enumerate}\r\n\\end{lemma}","start":28060,"end":29826},{"raw_tex":"\\begin{proof}\r\n\r\n\\begin{enumerate}[label=({\\it{\\roman*}})]\r\n\\item  Due to choice of $\\stdInt$ and $\\stdInt^*$, it is clear that the solutions \r\nmeet the PPS condition.\r\n\r\n\r\n\\item \\fbox{$\\opt{\\text{\\ref{eq:red}}} = 0 \\Rightarrow$  (a) or (b) holds.}\r\nSuppose that $\\opt{\\text{\\ref{eq:red}}} = 0$ and let $(x^*,0,w^*)$ be an optimal solution for \\eqref{eq:red}.\r\nDue to the constraints in \\eqref{eq:red}, we  have\r\n$$\\stdMap x^* = 0, \\qquad -\\inProd{c}{x^*} = w^* \\geq 0, \\qquad x^* \\in \\stdCone ^*.$$ \r\nNote that if $\\inProd{c}{x^*} < 0$, then $\\minFaceD = \\feasS = \\emptyset$. This is \r\nalternative (a).\r\n\r\nIf $\\inProd{c}{x^*} = 0$, then (\\eqref{eq:red:1}) implies \r\n$w^* = 0$. Since $\\stdMap x^* = 0$, we obtain  \r\n$$\\feasS \\subseteq  \\stdCone \\cap \\{x^*\\}^\\perp. $$\r\nBy \\eqref{eq:red:2}, we have $\\inProd{\\stdInt}{x^*} = 1$, so that $x^* = (x_1^*,x_2^*)$ satisfies $x_1^* \\not \\in (\\stdCone^1)^\\perp$, due to the choice of $\\stdInt$. \r\nTherefore, the inclusion $\\stdCone \\cap \\{x^*\\}^\\perp \\subseteq \\stdCone$ is strict, \r\nthat is,  $\\stdCone \\cap \\{x^*\\}^\\perp \\subsetneq \\stdCone$. This is alternative (b).\r\n\r\n\\fbox{(a) or (b) holds $\\Rightarrow \\opt{\\text{\\ref{eq:red}}} = 0$.} \r\nFirst recall that  $t$ is constrained to be nonnegative, therefore  $\\opt{\\text{\\ref{eq:red}}} \\geq 0$.\r\nNext, we will prove the following inequality\r\n\\begin{equation}\\label{eq:lemma:aux}\r\n\t{\\inProd{\\stdInt}{x^*} - \\inProd{c}{x^*}} > 0.\r\n\\end{equation}\r\nSince $\\stdInt \\in \\stdCone,x^* \\in \\stdCone^*$, we have \r\n$\\inProd{\\stdInt}{x^*} \\geq 0$. If $(a)$ holds, then $- \\inProd{c}{x^*}> 0$.\r\nIn particular, \\eqref{eq:lemma:aux} holds.\r\nNow, we consider the case where (b) holds. \r\nIn this case, we have $\\inProd{c}{x^*} = 0$. \r\nIf $\\inProd{\\stdInt}{x^*}$ is also zero, then $\\inProd{\\stdInt_1}{x_1^*}$ is zero and since $\\stdInt_1 \\in \\reInt \\stdCone^1$, $ x^*_1$ must \r\nbelong to $(\\stdCone^1)^\\perp$. However, this is not allowed \r\nsince we assumed that $x_1^* \\not \\in (\\stdCone^1)^\\perp$.\r\nTherefore, we have $\\inProd{\\stdInt}{x^*} > 0$ and, in particular,  \\eqref{eq:lemma:aux} also holds.\r\n\r\nLet  $$\r\n\\alpha = \\frac{1}{\\inProd{\\stdInt}{x^*} - \\inProd{c}{x^*}}.\r\n$$\r\nThen $\\alpha > 0$ and \r\n$(\\alpha x^*,0,-\\alpha \\inProd{c}{x^*})$ is a solution to \\eqref{eq:red} with value $0$, \r\nso that $\\opt{\\text{\\ref{eq:red}}} = 0$.\r\n\r\n\\item \r\n\\fbox{ $\\opt{\\text{\\ref{eq:red_dual}}} > 0 \\Rightarrow $ PPS holds for \\eqref{eq:dual}}\r\nFrom \\eqref{eq:red_dual:1}, we have\r\n$$\r\nc y_1^* - \\stdInt y_2^* - {\\stdMap}^\\T y_3^* \\in \\stdCone.\r\n$$\r\nSince $y_2^*=\\opt{\\text{\\ref{eq:red_dual}}}\r\n>0$, we have, due to \\eqref{eq:red_dual:3}, $y_1^* > 0$.\r\n Hence, \r\n $$c - \\stdInt \\frac{y_2^*}{y_1^*} - {\\stdMap}^\\T \\frac{y_3^*}{y_1^*} \\in \\stdCone.$$ \r\nWe conclude that \r\n$c - {\\stdMap}^\\T y_3^*/y_1^* \\in (\\stdCone^1 +y_2^*/y_1^* \\stdInt_1)\\times  \\{0\\} \\subseteq (\\reInt \\stdCone ^1)\\times \\{0\\}$, due to the choice of  $\\stdInt$ and the fact that $y_2^*/y_1^* > 0$. \r\n\r\n\\fbox{ $\\opt{\\text{\\ref{eq:red_dual}}} = 0 \\Rightarrow $ PPS does not hold for \\eqref{eq:dual}  } \r\nIf $t^* = 0$ and $(x^*,0,w^*)$ is an optimal solution \r\nfor \\eqref{eq:red}, then either $(a)$ or $(b)$ of item $(ii)$ is satisfied. \r\nIf $(a)$ is satisfied, then \\eqref{eq:dual} is infeasible and thus the PPS condition\r\ncannot hold. \r\nIf $(b)$ is satisfied, then $\\inProd{c}{x^*} = 0$,\r\n$\\stdMap x^* = 0$ and $x_1^* \\not \\in (\\stdCone ^1)^\\perp$. If \r\n$(s_1,s_2)$ is a feasible slack for \\eqref{eq:dual}, we have $\\inProd{s_1}{x_1^*} + \\inProd{s_2}{x_2^*} = 0$, so that \r\n$\\inProd{s_1}{x_1^*} = 0$.  \r\nAs $x_1^* \\not \\in (\\stdCone ^1)^\\perp$, we have that $s_1 \\not \\in \\reInt \\stdCone^1$. This means\r\nthat there is no feasible solution for \\eqref{eq:dual} satisfying the PPS\r\ncondition.\r\n\\end{enumerate}\r\n\\end{proof}","start":31151,"end":34906},{"raw_tex":"\\begin{proposition}\\label{prop:strict}\r\nSuppose $\\opt{\\text{\\ref{eq:red}}} = \\opt{\\text{\\ref{eq:red_dual}}} = 0$  and that  we have  optimal solutions to \\eqref{eq:red} and \r\n\\eqref{eq:red_dual} satisfying dual strict complementarity. \r\nIf $w^* = 0$, then $\\minFaceD = \\stdCone \\cap\\{ x^*\\}^\\perp$.\r\n%Reciprocally, suppose that $\\minFace{D} \\neq \\stdCone $ and that there is a reducing direction $d$ for which $\\minFaceD = \\stdCone \\cap\\{ d\\}^\\perp$. Then, \r\n%\\eqref{eq:red} and \\eqref{eq:red_dual} satisfy dual strict complementarity if $\\stdInt$ is chosen as to satisfy $\\inProd{\\stdInt}{d} \\neq 0$ (in particular, if \r\n%$\\stdInt \\in \\reInt \\stdCone$).\r\n\\end{proposition}","start":46831,"end":47504},{"raw_tex":"\\begin{proof}\r\nSince $w^* = 0$, \\eqref{eq:strict3} implies that $y_1^* > y_2^*$. Then, $\\opt{\\text{\\ref{eq:red}}} = \\opt{\\text{\\ref{eq:red_dual}}} $ and \r\n$\\opt{\\text{\\ref{eq:red}}} \\geq 0 $ implies that $y_2^* \\geq 0$, so that $y_1^* > 0$ as well.\r\nTherefore, from \\eqref{eq:strict1} we obtain $c -\\stdMap^\\T \\frac{y_3^*}{y_1^*} \\in \\reInt (\\stdCone \\cap\\{ x^*\\}^\\perp).$\r\n%To prove converse, let $y$ be such that $c - \\stdMap^\\T y \\in \\reInt (\\stdCone \\cap\\{ d\\}^\\perp)$. Then, for $\\alpha > 0$ sufficiently small, we have \r\n%that $(d/\\inProd{e}{d},0,0)$, $(\\alpha,0,\\alpha y)$ are optimal solutions satisfying strict complementarity.\r\n\\end{proof}","start":47506,"end":48155},{"raw_tex":"\\begin{proposition}\\label{prop:phase1}\r\nThe following hold.\r\n\\begin{enumerate}[label=(\\roman*)]\r\n\t\\item if \\eqref{eq:dual} is feasible, then the output face $\\stdFace$ satisfies \r\n\t$\\minFaceD \\subseteq \\stdFace$. Moreover, $s'$ is a dual \r\nfeasible slack such that $s_j' \\in \\reInt \\stdFace ^j $  for every $j$ such that $\\stdFace ^j$  is nonpolyhedral, i.e., the PPS condition is satisfied for $\\stdFace$.\r\n\r\nIn this case, Phase 1  stops after finding at most $\\sum _{i=1}^{r} \\distP(\\stdCone ^i) $ directions. \r\n\t\\item \\eqref{eq:dual} is infeasible if and only if Step 3 is reached. In this case, Phase 1 stops after finding \r\n\tat most $1+ \\sum _{i=1}^{r} \\distP(\\stdCone ^i) $ directions.\r\n\\end{enumerate}\r\n\\end{proposition}","start":51430,"end":52157},{"raw_tex":"\\begin{proof}\r\nWe will focus on the statements about the bounds, since the other statements \r\nare direct consequences of \r\nLemma \\ref{lemma:red2}. Note that whenever Step 4 is reached, we have \r\n$\\stdFace _{i+1} \\subsetneq \\stdFace _i$, since \r\n$x^*_j \\not \\in (\\stdFace _i^j)^\\perp$ for at least one nonpolyhedral cone $\\stdFace _i^j$, due to item $(ii)$-$(b)$ of Lemma \\ref{lemma:red2}. \r\nTherefore, whenever a new (proper) face is found, it is because \r\nwe are making progress towards a polyhedral face for at least one nonpolyhedral cone. \t\r\n\t\r\nBy definition, after finding $\\hat \\ell = \\sum _{i=1}^{r} (\\distP(\\stdCone ^i)) $ directions, \r\n$\\stdFace _{\\hat \\ell + 1}$ is polyhedral. We now consider what happens if \r\nthe algorithm has not stopped after all these directions were found.\r\nIn this case, when it is time to build \\eqref{eq:red} and \\eqref{eq:red_dual} with $\\stdFace_{l+1}$ in\r\nplace of $\\stdCone$ at Step 2, Phase 1 selects $e = 0$ and $\\stdInt^* \\in (\\stdFace _{\\hat \\ell + 1})^*$.\r\n\r\nFirst, suppose that \\eqref{eq:dual} is feasible and  let \r\n$y$ be such that $c-\\stdMap^\\T y \\in \\feasS$. \r\nSince $\\stdInt = 0$, $(\\alpha, \\alpha, \\alpha y)$\r\nis a feasible solution for \\eqref{eq:red_dual}, when $\\alpha>0$ is sufficiently small. It follows \r\nthat $\\opt{\\text{\\ref{eq:red_dual}}} > 0$ and that Phase 1 eventually reaches Step 5. This gives item $(i)$.\r\n\r\n\r\n\r\nFinally, suppose that \\eqref{eq:dual} is infeasible. By Lemma \\ref{lemma:red2},  $\\opt{\\text{\\ref{eq:red_dual}}} = 0$. Since $\\stdInt = 0$,  \\eqref{eq:red:2} implies that every optimal solution of \r\n\\eqref{eq:red} will be a triple of the form\r\n$(x^*,0,1)$, which implies that Step 3 will be reached and a single new direction will be \r\nadded. This gives  item $(ii)$.\r\n\\end{proof}","start":52159,"end":53918},{"raw_tex":"\\begin{proposition}\\label{prop:min_face}\r\nThe output face of Phase 2 is $\\minFaceD$ and there exists $\\hat s$ as in Steps 3. and 4.\r\n\\end{proposition}","start":62497,"end":62647},{"raw_tex":"\\begin{proof}\r\nSuppose that the output face $\\stdFace$ of Phase 1 satisfies $\\stdFace \\neq \\minFaceD$. \r\nBy Lemma 3.2 in \\cite{article_waki_muramatsu}, there is \r\na reducing direction $x$ such that $x \\in \\stdFace ^* \\cap \\ker \\stdMap \\cap \\{c\\}^\\perp$ \r\nand $x \\not \\in \\stdFace ^\\perp$.\r\nDue to Proposition \\ref{prop:phase1}, any such reducing direction $x$ satisfies $\\inProd{x}{s'} = 0$, which implies that \r\n $x_j \\in ({\\stdFace ^j})^\\perp = ({\\spanVec \\stdFace ^j})^\\perp$ for every $j$ such that $\\stdFace ^j$ is not polyhedral, \r\n since $s_j' \\in \\reInt \\stdFace ^j $ for those $j$.\r\nTherefore, the possible reducing directions are confined to the polyhedral cone $\\hat \\stdCone^*$,\r\nwhere $\\hat \\stdCone$ is the cone in Step 1. of Phase 2.\r\n\r\nSince $\\hat \\stdCone$ is polyhedral, the problems \\eqref{eq:red} and \\eqref{eq:red_dual} \r\n are polyhedral and they admit strictly complementary optimal \r\nsolutions $(x^*,t^*,w^*)$, $(y_1^*,y_2^*,y_3^*)$.\r\nThe fact that $x \\not \\in \\stdFace^\\perp$ implies that $\\inProd{\\stdInt}{x}\\neq 0$ so that $(x/\\inProd{\\stdInt}{x},0,0) $ is an optimal \r\nsolution to \\eqref{eq:red}. Therefore, $t^* = y_2^* = 0$. Moreover, since \\eqref{eq:dual} is \r\nfeasible, we have $w^* = 0$.\r\nBy Proposition \\ref{prop:strict}, we have \r\n$c -\\stdMap^\\T  \\frac{y_3^*}{y_1^*} \\in \\reInt (\\hat \\stdCone \\cap \\{x^*\\}^\\perp).$\r\n\r\nLet $\\tilde s = c -\\stdMap^\\T  \\frac{y_3^*}{y_1^*}$. Note that $\\stdFace \\cap \\{x^*\\}^\\perp$ is a face of \r\n$\\stdFace$ containing $\\feasS$, since we argued that $x^*$ must be a reducing direction. We will prove that $\\minFaceD = \\stdFace \\cap \\{x^*\\}^\\perp$ by \r\n showing that some convex combination of $s'$ and $\\tilde s$ lies in $\\reInt (\\stdFace \\cap \\{x^*\\}^\\perp)$. \r\n% Intuitively, $s'$ and $\\tilde s$ are  relative interior points \r\n%of the nonpolyhedral and polyhedral parts of $\\stdFace \\cap \\{x^*\\}^\\perp$, respectively. So some convex \r\n%combination of them will belong to $\\feasS$ and be a relative interior point of $\\stdFace \\cap \\{x^*\\}^\\perp$.\r\n\r\nLet $z_\\beta = \\beta s' + (1-\\beta)\\tilde s$. For \r\nall $\\beta \\in (0,1)$ and all $j$ such that $\\stdFace ^j$ is polyhedral, we \r\nhave $(z_\\beta)_j \\in \\reInt ( \\stdFace ^j  \\cap \\{x_j^*\\}^\\perp)$, because $\\tilde s _j \\in  \\reInt (\\stdFace ^j  \\cap \\{x_j\\}^\\perp)$ and \r\n$s'$ is feasible.\r\nIf $\\stdFace ^j$ is not polyhedral, then $\\stdFace ^j  \\cap \\{x_j^*\\}^\\perp = \\stdFace ^j$, \r\nsince $x_j \\in (\\stdFace ^j)^\\perp$. Because $\\tilde s _j \\in \\spanVec{\\stdFace ^j}$ and \r\n$s_j' \\in \\reInt \\stdFace^j$, for all $\\beta $ sufficiently close to $1$ we have \r\n$(z_\\beta)_j \\in \\reInt \\stdFace^j$. Therefore, it is possible to select $\\beta \\in (0,1)$ \r\nsuch that $(z_\\beta)_j \\in \\reInt (\\stdFace ^j  \\cap \\{x_j\\}^\\perp) $ for all $j$. This \r\nshows that $\\minFaceD = \\stdFace \\cap \\{x^*\\}^\\perp $.\r\n\r\nIf $\\stdFace$ was already the minimal face to begin with, then \r\n$t^* > 0$. We can then  proceed in a \r\nsimilar fashion. The only difference is that due to \\eqref{eq:red_dual:1}, we \r\nwill have that $\\tilde s = c -\\stdMap^\\T  \\frac{y_3^*}{y_1^*} $ satisfies \r\n$\\tilde s _j \\in  \\reInt (\\stdFace ^j)$ for every $j$ such that $\\stdFace ^j$ is polyhedral.\r\nAnd as before, we can select a convex combination of $s'$ and $\\tilde s$ belonging to \r\nthe relative interior of $\\minFaceD$.\r\n\\end{proof}","start":62649,"end":65957},{"raw_tex":"\\begin{proof}\r\nIf \\eqref{eq:dual} is infeasible, then $\\minFaceD = \\emptyset$ and the result \r\nfollows from Proposition \\ref{prop:phase1}. So suppose now that \r\n\\eqref{eq:dual} is feasible. Then Phase 1 ends after finding at most \r\n$\\sum _{i=1}^{r} \\distP(\\stdCone ^i)$ directions. Due to Proposition \\ref{prop:min_face}, at most one \r\nextra direction is needed to jump to the minimal face.\r\n\\end{proof}","start":66318,"end":66721},{"raw_tex":"\\begin{proof}\r\nWe will prove $(i)$ first. Let \r\n$f_1$ be such that $f_1(x) = \\inProd{c}{x}$ if $Ax = b$ and $+\\infty$ otherwise.\r\nLet  $f_2$ be the indicator function of $\\Re^{n_1}\\times (\\stdCone ^2)^*$ and $f_3$ be the indicator \r\nfunction of $(\\stdCone ^1)^*\\times \\Re^{n_2}$. Since there is  a primal feasible solution $x = (x_1,x_2)$ such \r\nthat $x_1 \\in \\reInt (\\stdCone ^1)^*$, we have that $\\fDom f_1 \\cap \\fDom f_2 \\cap \\reInt (\\fDom f_3)$ is \r\nnonempty. In addition, $f_1$ and $f_2$ are polyhedral functions. Let us now observe that:\r\n\\begin{align*}\r\nf_1^*(s) = \\left\\{\r\n  \\begin{array}{l l}\r\n    \\inProd{b}{y}  & \\quad \\text{if there is $y$ with $s-c = \\stdMap^\\T y$}\\\\\r\n    +\\infty & \\quad \\text{otherwise}\r\n  \\end{array}\\right. \r\n\\end{align*}\r\nNote that, due to feasibility, for fixed $s$, $\\inProd{b}{y}$ does not depend on the choice \r\nof $y$, as long as $c+\\stdMap ^\\T y = s$. This is because since there is $x$ such that $Ax = b$, we have $\\inProd{b}{y} = \\inProd{x}{s-c}$.\r\nThe conjugate $f_2^*$ is the indicator function of $- \\{0\\}\\times \\stdCone ^2$ and \r\n$f_3^*$ is the indicator function of $- \\stdCone ^1\\times \\{0\\}$. \r\nApplying Theorem \\ref{theo:conv} with $s = 0$, we have:\r\n\\begin{align*}\r\n(f_1+f_2+f_3)^*(0) & = \\inf \\left\\{ \\inProd{b}{y} \\mid  c+ \\stdMap ^\\T y = s_1, s_1 - (s_3,s_2) = 0,  s_2 \\in  \\stdCone ^2, s_3 \\in \\stdCone ^1      \\right\\} \\\\\r\n & = \\inf \\left\\{ \\inProd{b}{y} \\mid  c+ \\stdMap ^\\T y = s_1, s_1 \\in  \\stdCone ^1\\times \\stdCone ^2      \\right\\} \\\\\r\n& = - \\sup \\left\\{ \\inProd{b}{y} \\mid  c -\\stdMap ^\\T y = s_1,  s_1 \\in  \\stdCone ^1 \\times \\stdCone ^2      \\right\\},\r\n\\end{align*}\r\nwhere the $\\sup$ in the last equation is attained. So, there is some dual feasible $y$ such that \r\n$(f_1+f_2+f_3)^*(0) =  \\inProd{b}{y}$. However, using the definition of conjugate, we also have:\r\n\\begin{align*}\r\n(f_1+f_2+f_3)^*(0) = - \\inf \\{ \\inProd{c}{x} \\mid Ax = b, x \\in \\stdCone ^1\\times \\stdCone ^2\\} = -\\theta _P.\r\n\\end{align*}\r\nIt follows that $\\pOpt = \\dOpt$ and the dual is attained at $y$. \r\nTo prove $(ii)$, let $g_1 = f_1^*$, and let $g_2$ and $g_3$ be the indicator \r\nfunctions  of $\\Re^{n_1}\\times \\stdCone ^2$ and  $\\stdCone ^1 \\times \\Re^{n_2}$, respectively. Again, \r\nit is enough to compute $(g_1+g_2+g_3)^*(0)$ using both the definition of conjugate function and \r\nusing Theorem \\ref{theo:conv}. \r\n\\end{proof}","start":116866,"end":119230}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:dual","eq:primal"],"declared_external":false,"source_only":false,"title":"原始与对偶锥规划","source_units":[{"raw_tex":"Consider the following pair of primal and dual conic linear programs (CLPs):\r\n\\vspace*{-3\\baselineskip} \r\n\\begin{multicols}{2}\r\n\\begin{align}\r\n\\underset{x}{\\inf} & \\quad \\inProd{c}{x} \\label{eq:primal}\\tag{P}\\\\ \r\n\\mbox{subject to} & \\quad \\stdMap x = b \\nonumber \\\\ \r\n&\\quad x \\in \\stdCone ^* \\nonumber \r\n\\end{align}\r\n\t\\break\r\n\t\\begin{align}\r\n\t\\underset{y}{\\sup} & \\quad \\inProd{b}{y} \\label{eq:dual} \\tag{D} \\\\ \r\n\t\\mbox{subject to} & \\quad c - \\stdMap ^\\T y \\in \\stdCone \\nonumber,\\\\ \\nonumber\r\n\t\\end{align}\t\r\n\\end{multicols}\r\n\\noindent where $\\stdCone \\subseteq \\Re^n$ is a closed convex cone and $\\stdCone^*$ is the dual cone $\\{s \\in \\Re^n \\mid \\inProd{s}{x} \\geq 0, \\forall x \\in \\stdCone \\}$. We have \r\nthat $\\stdMap: \\Re^n \\to \\Re^m$ is a linear map, $b \\in \\Re^m$, $c \\in \\Re^n$ and \r\n$\\stdMap^\\T$ denotes the adjoint map. We also have $\\stdMap^\\T y = \\sum _{i=1}^m \\stdMap _i y_i$, for certain \r\nelements $\\stdMap _i \\in \\Re^n$. The  inner product is denoted by $\\inProd{\\cdot}{\\cdot}$.\r\nWe will use $\\pOpt$ and $\\dOpt$ to denote the primal and  dual optimal value, respectively.\r\nIt is understood that $\\pOpt = +\\infty$ if \\eqref{eq:primal} is infeasible and \r\n$\\dOpt = -\\infty$ if \\eqref{eq:dual} is infeasible.\r\n","start":6343,"end":7567}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"可行松弛与最小面","source_units":[{"raw_tex":"In the absence of either a primal relative interior feasible solution or a dual \r\nrelative interior slack, it is possible that $\\pOpt \\neq \\dOpt$. A possible \r\nway of  correcting that is to let $\\minFaceD$ be \r\nthe minimal face of $\\stdCone$ which contains the feasible slacks $\\feasS = \\{c- \\stdMap ^\\T y \\in \\stdCone \\mid y \\in \\Re^m  \\}$, then \r\n","start":7573,"end":7922}]},{"id":"S003","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"面约化的对偶动机","source_units":[{"raw_tex":"we substitute $\\stdCone$ by $\\minFaceD$ and $\\stdCone^*$ by $(\\minFaceD)^*$. With that, \r\nthe new primal optimal value $\\pOpt ' $ will satisfy $\\pOpt ' = \\dOpt$. This is precisely what \r\nfacial reduction \\cite{Borwein1981495, pataki_strong_2013, article_waki_muramatsu} approaches do.\r\n","start":7922,"end":8208}]},{"id":"S004","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"相对内部及常见锥记号","source_units":[{"raw_tex":"For $C$ a closed convex set, we will denote by $\\reInt C$ and \r\n$\\closure C$ the relative interior and the closure of $C$, respectively.\r\nIf $\\mathcal{U}$ is an \r\narbitrary set, we denote by $\\mathcal{U}^\\perp$ the subspace which contains \r\nthe elements orthogonal to it. We will denote by $\\PSDcone{n}$ the cone of \r\n$n\\times n$ symmetric positive semidefinite matrices and by $\\SOC{n}$ the \r\nLorentz cone $\\{(x_0,\\overline{x}) \\in \\Re\\times \\Re^{n-1} \\mid x_0 \\geq \\norm{\\overline{x}}_2 \\}$, \r\nwhere $\\norm{.}_2$ is the usual Euclidean norm.\r\nThe nonnegative orthant will be denoted by $\\Re^n_+$. \r\n\r\n\r\n\r\n","start":16275,"end":16882}]},{"id":"S005","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":true,"declaration_text":["原文第322行引用 Rockafellar, Convex Analysis, Theorem 14.6；属于声明允许的标准有限维凸分析背景。"],"source_only":false,"title":"线性部分与正交补","source_units":[{"raw_tex":"If $\\stdCone$ is a closed convex cone, we write $\\stdCone^*$ for its dual \r\ncone. We write $\\spanVec \\stdCone$ for its linear span and $\\lineality \\stdCone$ for \r\nits lineality space, which is $\\stdCone \\cap -\\stdCone$. $\\stdCone$ is said to be \\emph{pointed} if \r\n$\\lineality \\stdCone  = \\{0\\}$. We have $\\lineality (\\stdCone^*) = \\stdCone^\\perp$, see Theorem 14.6 in \\cite{rockafellar}. Also, if we select $\\stdInt \\in \\reInt \\stdCone$ and $x \\in \\stdCone^*$, then $x \\in \\stdCone^\\perp $ if and only if $\\inProd{\\stdInt}{x} = 0$\\footnote{Suppose $\\inProd{\\stdInt}{x} = 0$. Then, given $y \\in \\stdCone$ we have $\\alpha \\stdInt +(1-\\alpha)y \\in \\stdCone$ for some $\\alpha > 1$, due to Theorem 6.4 in \\cite{rockafellar}. Taking the inner product with $x$, we see that $\\inProd{y}{x}$ must be zero.}.\r\n","start":16882,"end":17683}]},{"id":"S006","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"内部点检测正交性","source_units":[{"raw_tex":"$\\lineality \\stdCone  = \\{0\\}$. We have $\\lineality (\\stdCone^*) = \\stdCone^\\perp$, see Theorem 14.6 in \\cite{rockafellar}. Also, if we select $\\stdInt \\in \\reInt \\stdCone$ and $x \\in \\stdCone^*$, then $x \\in \\stdCone^\\perp $ if and only if $\\inProd{\\stdInt}{x} = 0$\\footnote{Suppose $\\inProd{\\stdInt}{x} = 0$. Then, given $y \\in \\stdCone$ we have $\\alpha \\stdInt +(1-\\alpha)y \\in \\stdCone$ for some $\\alpha > 1$, due to Theorem 6.4 in \\cite{rockafellar}. Taking the inner product with $x$, we see that $\\inProd{y}{x}$ must be zero.}.\r\n","start":17147,"end":17683}]},{"id":"S007","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"正交判据的脚注证明","source_units":[{"raw_tex":"$\\lineality \\stdCone  = \\{0\\}$. We have $\\lineality (\\stdCone^*) = \\stdCone^\\perp$, see Theorem 14.6 in \\cite{rockafellar}. Also, if we select $\\stdInt \\in \\reInt \\stdCone$ and $x \\in \\stdCone^*$, then $x \\in \\stdCone^\\perp $ if and only if $\\inProd{\\stdInt}{x} = 0$\\footnote{Suppose $\\inProd{\\stdInt}{x} = 0$. Then, given $y \\in \\stdCone$ we have $\\alpha \\stdInt +(1-\\alpha)y \\in \\stdCone$ for some $\\alpha > 1$, due to Theorem 6.4 in \\cite{rockafellar}. Taking the inner product with $x$, we see that $\\inProd{y}{x}$ must be zero.}.\r\n","start":17147,"end":17683}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"四种可行性状态","source_units":[{"raw_tex":"\r\n\r\n\r\nThe conic linear program \\eqref{eq:dual} can be in four different feasibility \r\nstatuses: $i)$ strongly feasible if $\\feasS \\cap \\reInt \\stdCone \\neq \\emptyset$; $ii)$ weakly feasible if \r\n$\\feasS \\neq \\emptyset$ but $\\feasS \\cap \\reInt \\stdCone  = \\emptyset$; $iii)$ weakly infeasible if \r\n$\\feasS = \\emptyset$ but $\\text{dist}(c + \\matRange \\stdMap^\\T ,\\stdCone) = 0$; $iv)$ strongly \r\ninfeasible if $\\text{dist}(c + \\matRange \\stdMap^\\T ,\\stdCone) > 0$.  Note that \r\n\\eqref{eq:primal} admits analogous definitions. \r\n\r\n","start":17683,"end":18211}]},{"id":"S009","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原文回顾的标准有限维锥规划强对偶定理；属于声明允许的标准凸分析背景，未在此给出专门出处。"],"source_only":false,"title":"通常的强对偶定理","source_units":[{"raw_tex":"The \r\nstrong duality theorem states that if \\eqref{eq:dual} is strongly feasible and \r\n$\\dOpt < + \\infty$, then $\\pOpt = \\dOpt$ and $\\pOpt$ is attained. \r\nOn the other hand, if \\eqref{eq:primal} is strongly feasible and \r\n$\\pOpt > -\\infty$,  then $\\pOpt = \\dOpt$ and $\\dOpt$ is attained. \r\nWe  will also need a special version of the strong duality theorem. \r\nFirst, we need the following definition.\r\n\r\n","start":18211,"end":18615}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":["def:pps"],"declared_external":false,"source_only":false,"title":"部分多面体 Slater 条件","source_units":[{"raw_tex":"\\begin{definition}[Partial Polyhedral Slater's condition]\\label{def:pps}\r\n\tLet $\\stdCone = \\stdCone ^1\\times \\stdCone ^2$, where  $\\stdCone ^1\\subseteq \\Re^{n_1},\\stdCone ^2 \\subseteq \\Re^{n_2}$ are closed convex cones such that $\\stdCone ^2$ is polyhedral.\r\n\tWe say that \\eqref{eq:dual} satisfies the Partial Polyhedral Slater's (\\PPS) condition if there is \r\n\ta slack $(s_1,s_2) = c-\\stdMap^\\T y$, such that $s_1 \\in \\reInt \\stdCone^1$ and $s_2 \\in \\stdCone^2$.\r\n\tSimilarly, we say that \\eqref{eq:primal} satisfies the {\\PPS} condition, if there is a primal \r\n\tfeasible solution $x = (x_1,x_2)$ for which $x_1 \\in \\reInt (\\stdCone^1)^*$.\r\n\\end{definition}\r\n","start":18615,"end":19274}]},{"id":"S011","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"附录证明的指引","source_units":[{"raw_tex":"The following is a strong duality theorem based on the {\\PPS} condition.\r\nAs we could not find a precise reference for it, we give a proof in the Appendix \\ref{app:partial}.  \r\n","start":19274,"end":19451}]},{"id":"S012","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明允许的 Waki and Muramatsu, Lemma 3.2；原文第372行及1050行调用。"],"source_only":false,"title":"强可行失败的约化判据","source_units":[{"raw_tex":"Here, we will suppose that our main interest is in the dual problem \\eqref{eq:dual}.\r\n Facial Reduction hinges on the fact that \r\nstrong feasibility fails  if and only if there \r\nis $d \\in \\stdCone ^*$ such that $\\stdMap d = 0$  and\r\none of the two alternatives holds: $(i)$  $\\inProd{c}{d} = 0$ and $d \\not \\in \\stdCone ^\\perp$; or \r\n$(ii)$ $\\inProd{d}{c} < 0$, see Lemma 3.2 in \\cite{article_waki_muramatsu}. If \r\nalternative $(i)$ holds, $\\stdFace = \\stdCone \\cap \\{d\\}^\\perp$ is a proper face of $\\stdCone$ \r\ncontaining $\\feasS$.  We then substitute  $\\stdCone$ by $\\stdFace $ and repeat.\r\nIf  $(ii)$ holds, \\eqref{eq:dual} is infeasible. \r\n","start":20447,"end":21092}]},{"id":"S013","semantic_type":"algorithm","kind":"algorithm","labels":[],"declared_external":false,"source_only":false,"title":"通用面约化算法","source_units":[{"raw_tex":" We write below a generic \r\nfacial reduction algorithm similar to the one described in \\cite{article_waki_muramatsu}.\r\n\r\n\\noindent\r\n[{\\bf Generic Facial Reduction}]\\label{alg:gen_fra}\r\n\r\n{\\bf Input:} \\eqref{eq:dual}\r\n\r\n{\\bf Output:} A set of reducing directions $\\{d_1, \\ldots, d_\\ell\\}$ and $\\minFaceD$.\r\n\\begin{enumerate}\r\n\\item $\\stdFace _1 \\leftarrow \\stdCone, i \\leftarrow 1 $\r\n\\item Let $d_i$ be an element in $\\stdFace _i ^* \\cap \\ker \\stdMap$ such that either: \r\n$i)$ $d_i \\not \\in \\stdFace _i^\\perp$ and $\\inProd{c}{d_i} = 0$; or $ii)$ $\\inProd{c}{d_i} < 0$.\r\nIf no such $d_i$ exists, let $\\minFaceD \\leftarrow \\stdFace _i$ and stop. \r\n\\item If $\\inProd{c}{d_i} < 0$, let $\\minFaceD \\leftarrow  \\emptyset$ and stop.\r\n\\item If $\\inProd{c}{d_i} = 0$, let $\\stdFace _{i+1}  \\leftarrow \\stdFace _i  \\cap \\{d_i\\}^\\perp, i\\leftarrow i+1$ and return to 2).\r\n\\end{enumerate}\r\n","start":21092,"end":21969}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"约化方向的定义","source_units":[{"raw_tex":"\r\n\r\nWe will refer to the directions satisfying $d_i \\in \\stdFace _i^* \\cap \\ker \\stdMap$ and \r\n$\\inProd{c}{d_i} \\leq 0$ as \\emph{reducing directions}, so that the $d_i$ in Step 2. are indeed reducing directions. An important issue when doing facial reduction is how to model the \r\n","start":21969,"end":22250}]},{"id":"S015","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"方向搜索的建模背景","source_units":[{"raw_tex":"search for the reducing directions. It is sometimes said that doing facial \r\nreduction can be as hard as solving the original problem. However, an important \r\ndifference is that the search for the $d _i$ can be cast as a pair of primal and \r\ndual problems which are always strongly feasible.\r\nThis was shown in the work by Cheung, Schurr and Wolkowicz \\cite{csw13} and \r\nin our previous work \\cite{lourenco_muramatsu_tsuchiya3}. \r\n","start":22250,"end":22681}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":["eq:red","eq:red:1","eq:red:2","eq:red:3","eq:red_dual","eq:red_dual:1","eq:red_dual:2","eq:red_dual:3"],"declared_external":false,"source_only":false,"title":"约化辅助问题对","source_units":[{"raw_tex":"In this article, we will search for reducing directions by\r\nconsidering the pair \\eqref{eq:red} and \\eqref{eq:red_dual} introduced in \\cite{lourenco_muramatsu_tsuchiya3}, which \r\nare parametrized by $\\stdMap$, $c$, $\\stdCone, \\stdInt, \\stdInt^*$. In Phase 1 of FRA-Poly, we will always \r\nselect $\\stdInt$ and $\\stdInt^*$ according to Lemma \\ref{lemma:red2}. Different choices will be discussed/used \r\nin Phase 2 of FRA-Poly and on Sections \\ref{sec:compl} and \\ref{sec:tight}.\r\n\\begin{align}\r\n\\underset{x,t,w}{\\mbox{minimize}} & \\quad t & \\tag{$P_\\stdCone $}\\label{eq:red}  \\\\ \r\n\\mbox{subject to} & \\quad -\\inProd{c}{x  -t\\stdInt^*} + t - w &= 0 \\label{eq:red:1}\\\\\r\n&\\quad \\inProd{\\stdInt}{x} + w &= 1 \\label{eq:red:2}\\\\\r\n& \\quad\\stdMap x  -t\\stdMap \\stdInt^* & = 0 \\label{eq:red:3}\\\\\r\n&\\quad (x,t,w) \\in \\stdCone ^* \\times \\Re_+ \\times \\Re_+ \\nonumber\\\\\r\n\\underset{y_1,y_2,y_3}{\\mbox{maximize}} &\\quad  y_2 & \\tag{$D_\\stdCone $}\\label{eq:red_dual}  \\\\ \r\n\\mbox{subject to} & \\quad cy_1 -\\stdInt y_2 -\\stdMap^\\T y_3 \\in \\stdCone \\label{eq:red_dual:1} \\\\\r\n&\\quad 1-y_1(1+\\inProd{c}{\\stdInt^*}) + \\inProd{\\stdInt^* }{\\stdMap ^\\T y_3}\\geq 0 \\label{eq:red_dual:2} \\\\\r\n& \\quad y_1-y_2 \\geq 0 \\label{eq:red_dual:3} \\\\\r\n&\\quad (y_1,y_2,y_3) \\in \\Re \\times \\Re \\times \\Re^m. \\nonumber\r\n\\end{align}\r\n\r\nRecall that our goal is to find a point $x \\in \\ker \\stdMap \\cap \\stdCone$ satisfying  $\\inProd{c}{x} \\leq 0$. The idea behind  \\eqref{eq:red} is to shift the problem by $-t\\stdInt^*$ (Equations \\eqref{eq:red:1} and \\eqref{eq:red:3}) and add constraints to ensure \r\nthe $x$ stays in a bounded region (Equation \\eqref{eq:red:2}). These changes ensure that \\eqref{eq:red} and \\eqref{eq:red_dual} \r\nsatisfy the PPS condition when the parameters $\\stdInt, \\stdInt^*$ are chosen appropriately as in\r\nthe next section.\r\n","start":23456,"end":25262}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"算法实施能力假设","source_units":[{"raw_tex":"We now present a disclaimer of sorts.\r\nThe theoretical results presented in this section and the next stand whether FRA-Poly is doable or \r\nnot for a given $\\stdCone$. \r\nIf we wish to do facial reduction concretely (even if it is by hand!), we need to \r\nmake a few assumptions on our computational capabilities \r\nand on our knowledge on the lattice of faces of $\\stdCone$. \r\nFirst of all, we must be able to solve problems over faces of $\\stdCone$ such that both the primal and the dual \r\nsatisfy the {\\PPS} condition and we must also be able to do \r\nbasic linear algebraic operations.  Also, for \r\neach face $\\stdFace$ of $\\stdCone$ we must know:\r\n\\begin{enumerate}\r\n\t\\item $\\spanVec \\stdFace$, \r\n\t\\item at least one point $\\stdInt \\in \\reInt \\stdFace$,\r\n\t\\item at least one point $\\stdInt^* \\in \\reInt \\stdFace^*$,\r\n\t\\item whether $\\stdFace$ is polyhedral or not.\r\n\\end{enumerate}\r\n","start":41456,"end":42340}]},{"id":"S018","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"计算假设的适用讨论","source_units":[{"raw_tex":"We remark that \r\napart from  knowledge about the polyhedral faces, our assumptions are not very different from what it is \r\nusually assumed \\emph{implicitly} in the FRA literature. \r\nFor symmetric cones, which \r\ninclude direct products of $\\PSDcone{n}$, $\\SOC{n}$ and $\\Re^n_+$, they are reasonable since their \r\nlattice of faces is well-understood and every face is again a symmetric cone. \r\nSo, for instance, $\\stdInt$ can be taken as the identity element for \r\nthe corresponding Jordan algebra. On the other hand, if $\\stdCone$ is, say, the copositive \r\ncone $\\mathcal{C}^n$, we might have some trouble fulfilling the requirements, inasmuch  as our knowledge of \r\nthe faces of $\\mathcal{C}^n$ is still lacking.\r\n\r\n","start":42340,"end":43057}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"到多面体性的距离","source_units":[{"raw_tex":"\\subsection{Distance to Polyhedrality}\r\nHere we introduce the notion of \\emph{distance to polyhedrality}. In what follows, \r\nif we have a chain of faces $\\stdFace _1 \\subsetneq \\ldots \\subsetneq \\stdFace _\\ell$, the \r\nlength of the chain is defined to be $\\ell$.\r\n\\begin{definition}\r\n\tLet $\\stdCone$ be a nonempty closed convex cone. The \\emph{distance to polyhedrality} $\\distP(\\stdCone)$ is the \r\n\tlength \\emph{minus one} of the longest strictly ascending chain of nonempty faces $\\stdFace _1 \\subsetneq \\ldots \\subsetneq \\stdFace _{\\ell} $ which satisfies:\r\n\t\\begin{enumerate}\r\n\t\t\\item  $\\stdFace _1$ is polyhedral;  \r\n\t\t\\item $\\stdFace _j$ is not polyhedral for $j > 1$.\r\n\t\\end{enumerate}\r\n\\end{definition}\r\n","start":43057,"end":43769}]},{"id":"S020","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"距离参数的约化含义","source_units":[{"raw_tex":"\r\nThe distance to polyhedrality is a well-defined concept, because the lineality space\r\nof $\\stdCone$ is always a polyhedral face of $\\stdCone$.\r\nMoreover, $\\distP(\\stdCone)$ counts the maximum number of facial reduction steps that \r\ncan be taken before we reach a polyhedral face.\r\n","start":43769,"end":44052}]},{"id":"S021","semantic_type":"definition","kind":"definition","labels":["eq:strict1","eq:strict2","eq:strict3"],"declared_external":false,"source_only":false,"title":"对偶严格互补条件","source_units":[{"raw_tex":"For general $\\stdCone$, these equivalencies may not hold and we might need \r\nto distinguish between primal and dual strict complementarity, see \r\nfor instance, Definition 3.4 and Remark 3.6 in the chapter by Pataki \\cite{pataki_handbook} and Equation (2.6) in Section 2 of the work by Tun\\c{c}el and Wolkowicz \\cite{TW12}. \r\nBased on those references, we will say \r\nthat \\eqref{eq:red} and \\eqref{eq:red_dual}  satisfies  \r\n\\emph{(dual) strict complementarity} if $\\opt{\\text{\\ref{eq:red}}} = \\opt{\\text{\\ref{eq:red_dual}}}$\r\nand there are optimal solutions $(x^*, t^*,w^*), (y_1^*,y_2^*,y_3^*)$ such that\r\n \\begin{align}\r\n cy_1^* -\\stdInt y_2^* -\\stdMap^\\T y_3^* &\\in \\reInt (\\stdCone \\cap\\{ x^*\\}^\\perp)  \\label{eq:strict1}\\\\\r\n t^* +  1-y_1^*(1+\\inProd{c}{\\stdInt^*}) + \\inProd{\\stdInt^* }{\\stdMap ^\\T y_3^*}&> 0  \\label{eq:strict2}\\\\\r\n w^* + y_1^*-y_2^* & > 0. \\label{eq:strict3} \r\n \\end{align}\r\n","start":45905,"end":46804}]},{"id":"S022","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"乘积锥及其分块面","source_units":[{"raw_tex":"Henceforth, we will assume that $\\stdCone$ is the product of $r$ cones and we will \r\nwrite $\\stdCone = \\stdCone ^1 \\times \\ldots \\times \\stdCone  ^r $. Then, recall that \r\nif $\\stdFace$ is face of $\\stdCone$, we can write $\\stdFace = \\stdFace ^1 \\times \\ldots \\times \\stdFace  ^r$, where $\\stdFace ^i$ is a face of $\\stdCone ^i$ for every $i$.\r\n\r\nConsider the following \r\nFRA variant, which we call FRA-Poly.\r\n","start":49239,"end":49649}]},{"id":"S023","semantic_type":"algorithm","kind":"algorithm","labels":[],"declared_external":false,"source_only":false,"title":"FRA-Poly 第一阶段","source_units":[{"raw_tex":"\r\n\\noindent\r\n[{\\bf Facial Reduction Poly - Phase 1}]\r\n\r\n{\\bf Input:} \\eqref{eq:dual}\r\n\r\n{\\bf Output:} A set of reducing directions $\\{d_1, \\ldots, d_\\ell\\}$. If \r\n\\eqref{eq:dual} is feasible, it outputs  some face $\\stdFace \\subseteq \\stdCone$ \r\nfor which the PPS condition holds, together \r\nwith a dual slack $s'$ for which $s_j' \\in \\reInt \\stdFace ^j $ for every $j$ such that $\\stdFace ^j$  is nonpolyhedral.\r\nIf \\eqref{eq:dual} is infeasible, the directions form a certificate of infeasibility.\r\n\\begin{enumerate}\r\n\\item $\\stdFace _1 \\leftarrow \\stdCone, i \\leftarrow 1 $\r\n\\item Let  $(x^*,t^*,w^*)$ and $(y_1^*,y_2^*,y_3^*)$ be any pair of optimal solutions to \r\n\\eqref{eq:red} and \\eqref{eq:red_dual} with\r\n\\begin{itemize}\r\n\t\\item $\\stdFace _i$ in place of $\\stdCone $, where $\\stdFace_i = \\stdFace_i ^1 \\times \\ldots \\times \\stdFace_i  ^r$,\r\n\t\\item any $\\stdInt^* \\in \\reInt \\stdFace _i^*$,\r\n\t\\item any $\\stdInt$  such that $\\stdInt _j = 0$ if $\\stdFace _i^j$ is polyhedral and \t$\\stdInt _j \\in \\reInt \\stdFace _i^j$, otherwise.\r\n\\end{itemize}\r\n\r\n\\item If $t^* = 0$ and  $\\inProd{c}{x^*} < 0$, let $\\minFaceD \\leftarrow \\emptyset$ and stop. \\eqref{eq:dual}  is infeasible.\r\n\\item If $t^* = 0$ and $\\inProd{c}{x^*} = 0$, let $d_i \\leftarrow x^*, \\stdFace _{i+1}  \\leftarrow \\stdFace _i  \\cap \\{d_i\\}^\\perp, i\\leftarrow i+1$ and return to 2).\r\n\\item If $t^* > 0$, $s' \\leftarrow  c  -\\stdMap^\\T  \\frac{y_3^*}{y_1^*}$, $\\stdFace \\leftarrow \\stdFace _i$ and stop. {\\PPS} condition is satisfied.\r\n\\end{enumerate}\r\n","start":49649,"end":51166}]},{"id":"S024","semantic_type":"algorithm","kind":"algorithm","labels":[],"declared_external":false,"source_only":false,"title":"FRA-Poly 第二阶段","source_units":[{"raw_tex":"[{\\bf Facial Reduction Poly - Phase 2}]\r\n\r\n{\\bf Input:} \\eqref{eq:dual}, the output of Phase 1: $\\stdFace$ and $s'$, with $\\stdFace \\neq \\emptyset$.\r\n\r\n{\\bf Output:} $\\minFaceD$, a dual feasible slack $\\hat s \\in \\reInt \\minFaceD$ and, perhaps, \r\nan extra reducing direction $d$.\r\n\\begin{enumerate}\r\n\\item Let $\\hat \\stdCone = {\\hat \\stdCone} ^1\\times \\ldots \\times {\\hat \\stdCone} ^r$ such that ${\\hat \\stdCone} ^j = \\stdFace ^j$ if \r\n$\\stdFace ^j$ is polyhedral and $\\hat \\stdCone ^j = \\spanVec \\stdFace^j$ otherwise. \r\n\\item Let $(x^*,t^*,w^*)$ and $(y_1^*,y_2^*,y_3^*)$ be\r\nany pair of \\emph{strictly complementary} optimal solutions to \\eqref{eq:red} and \\eqref{eq:red_dual} with\r\n\\begin{itemize}\r\n\t\\item $\\hat \\stdCone$ in place of $\\stdCone$,\r\n\t\\item any $\\stdInt \\in \\reInt \\hat \\stdCone$,\r\n\t\\item any $\\stdInt^* \\in \\reInt {\\hat \\stdCone}^* $.\r\n\\end{itemize}\r\n\\item If $t^* = 0$, let $d \\leftarrow x^*$, $\\minFaceD \\leftarrow \\stdFace \\cap \\{x^*\\}^\\perp$.\r\nLet $\\tilde s$ be $c  -\\stdMap^\\T  \\frac{y_3^*}{y_1^*} $. Then, we let \r\n$\\hat s$ be a convex combination of $\\tilde s$ and $s'$ such that $\\hat s \\in \\reInt \\minFaceD$ and stop.\\footnote{\\label{foot}In Proposition \\ref{prop:min_face} it is shown that if $z_\\beta = \\beta s' + (1-\\beta)\\tilde s $ is such that $\\beta \\in (0,1)$ and $\\beta$ is sufficiently close to $1$ then $z_\\beta \\in  \\reInt \\minFaceD$. Therefore, after determining $\\minFaceD$, $\\hat s$ can be found through a simple search procedure, e.g., the bisection method.} %by starting with $\\beta = 0.5$ and letting $\\beta$ be $0.75,0.875,0.9375, \\ldots$ and so on, until $z_\\beta \\in \\reInt \\minFaceD$.  }\r\n\\item If $t^* > 0$,  $\\minFaceD \\leftarrow \\stdFace$.\r\nLet $\\tilde s$ be $c  -\\stdMap^\\T  \\frac{y_3^*}{y_1^*} $. Then, we let \r\n$\\hat s$ be a convex combination of $\\tilde s$ and $s'$ such that $\\hat s \\in \\reInt \\minFaceD$ and stop.\\footref{foot}%\\footnote{See footnote \\ref{foot}.}\r\n\r\n\\end{enumerate}\r\n","start":59177,"end":61118}]},{"id":"S025","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明允许的 Goldman–Tucker 严格互补定理；原文同时引用 McLinden82、Akgul84，计算讨论引用 FRT85、MY93。"],"source_only":false,"title":"多面体问题的严格互补","source_units":[{"raw_tex":"\r\n\r\nNote that in Phase 2, the cone $\\hat{\\stdCone}$ is polyhedral, therefore, \r\nboth \\eqref{eq:red_dual} and \\eqref{eq:red} are \r\npolyhedral problems. Therefore,  strictly complementary solutions are ensured to exist, which is a \r\nconsequence of Goldman-Tucker Theorem and also follows from the results of \r\nMcLinden \\cite{McLinden82} and Akg\\\"ul \\cite{Akgul84}. \r\nWe also remark that a strictly complementary solution of a polyhedral problem can be found by solving \r\na single linear program, see, for instance, the article by Freund, Roundy and Todd \\cite{FRT85} and the related \r\nwork by Mehrotra and Ye \\cite{MY93}. \r\n","start":61118,"end":61740}]},{"id":"S026","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"凸共轭及多面体函数","source_units":[{"raw_tex":"Let $f:\\Re^n \\to \\Re\\cup \\{-\\infty,+\\infty \\}$ be a convex function.\r\nWe denote the domain of $f$ by $\\fDom f = \\{x \\in \\Re^n \\mid f(x) < \\infty \\}$. If $\\fDom f \\neq \\emptyset $ and $f$ is never $-\\infty$, then $f$ is said to be proper.\r\nIts conjugate will be denoted by $f^*$ and it satisfies $f^*(s) = \\sup _{x} \\inProd{s}{x} - f(x)$.\r\nIf the epigraph of $f$ is a polyhedral set, then $f$ is said to be a polyhedral function. \r\nWe recall Theorem 20.1 from \\cite{rockafellar}.\r\n\r\n","start":115268,"end":115750}]},{"id":"S027","semantic_type":"external_fact","kind":"external_fact","labels":["theo:conv"],"declared_external":true,"declaration_text":["明确许可的 Rockafellar, Convex Analysis, Theorem 20.1，原文重述为 B0093。"],"source_only":false,"title":"部分多面体共轭定理","source_units":[{"raw_tex":"\\begin{theorem}[Rockafellar]\\label{theo:conv}\r\nLet $f_1, \\ldots, f_m$ be proper convex functions and let $f_{k+1}, \\ldots ,$ $f_m$ be \r\npolyhedral functions. Suppose also that\r\n$$\r\n\\reInt (\\fDom f_1) \\cap \\cdots \\cap \\reInt (\\fDom f_k) \\cap \\fDom f_{k+1} \\cap \\cdots \\cap \\fDom f_m \\neq \\emptyset.\r\n$$\r\nThen the following holds:\r\n\\begin{equation*}\r\n(f_1 + \\cdots + f_m)^*(s) = \\inf \\{ f_1^*(s_1) + \\cdots + f_m^*(s_m) \\mid s_1 + \\cdots + s_m = s \\},\r\n\\end{equation*}\r\nwhere for each $s$ the infimum is attained whenever it is finite.\r\n\\end{theorem}\r\n","start":115750,"end":116300}]},{"id":"S028","semantic_type":"proposition","kind":"proposition","labels":["prop:slater2"],"declared_external":false,"source_only":false,"title":"PPS 强对偶命题","source_units":[{"raw_tex":"\\begin{proposition}[{\\PPS}-Strong Duality]\\label{prop:slater2}\r\n\tLet $\\stdCone = \\stdCone ^1\\times \\stdCone ^2$, where  $\\stdCone ^1\\subseteq \\Re^{n_1},\\stdCone ^2 \\subseteq \\Re^{n_2}$ are closed convex cones such that $\\stdCone ^2$ is polyhedral.\r\n\t\\begin{enumerate}[label=(\\roman*)]\r\n\t\t\\item If $\\pOpt$ is finite and \\eqref{eq:primal} satisfies the {\\PPS} condition, then $\\pOpt = \\dOpt$ and the dual optimal value is attained.\r\n\t\t\\item If $\\dOpt$ is finite and \\eqref{eq:dual} satisfies the {\\PPS} condition, then $\\pOpt = \\dOpt$ and the primal \r\n\t\toptimal value is attained.\r\n\t\\end{enumerate}\r\n\\end{proposition}","start":19451,"end":20066}]},{"id":"S029","semantic_type":"lemma","kind":"lemma","labels":["lemma:red2"],"declared_external":false,"source_only":false,"title":"部分多面体辅助问题引理","source_units":[{"raw_tex":"\\begin{lemma}\\label{lemma:red2}\r\nLet $\\stdCone = \\stdCone ^1 \\times \\stdCone ^2$ be a \r\nclosed convex cone, such that $\\stdCone ^2$ is polyhedral.\r\nConsider the pair \\eqref{eq:red} and \\eqref{eq:red_dual} with \r\n$\\stdInt$ and $\\stdInt^*$ such that  $\\stdInt = (\\stdInt_1,0) \\in (\\reInt \\stdCone ^1)\\times \\{0\\}$ and \r\n$\\stdInt^* \\in \\reInt \\stdCone^*$.  The following properties hold.\r\n\\begin{enumerate}[label=(\\roman*)]\r\n\\item The following are solutions to \\eqref{eq:red} and \\eqref{eq:red_dual} that satisfy the {\\PPS} condition: \r\n\\begin{align*}\r\n(x,t,w)& = \\left(\\frac{\\stdInt^*}{\\inProd{\\stdInt}{\\stdInt^*}+1},\\frac{1}{\\inProd{\\stdInt}{\\stdInt^*}+1}, \\frac{1}{\\inProd{\\stdInt}{\\stdInt^*}+1}\\right) \\\\\r\n (y_1,y_2,y_3) & = (0,-1,0).\r\n \\end{align*} \r\nIn particular, \r\n$\\opt{\\text{\\ref{eq:red}}} =  \\opt{\\text{\\ref{eq:red_dual}}}$.\r\n\\end{enumerate}\r\nLet $(x^*,t^*,w^*)$ be any primal optimal solution and \r\n$(y_1^*,y_2^*,y_3^*)$ be any dual optimal solution.\r\n \\begin{enumerate}[label=(\\roman*)]\r\n \\setcounter{enumi}{1} \r\n\\item $\\opt{\\text{\\ref{eq:red}}} =  \\opt{\\text{\\ref{eq:red_dual}}} = 0$ if and only if \r\none of the two alternatives below holds:\r\n\\begin{enumerate}\r\n\\item $\\inProd{c}{x^*} < 0 $ and $\\feasS = \\minFaceD = \\emptyset$, or\r\n\\item $\\inProd{c}{x^*} = 0 $, $\\feasS \\subseteq  \\stdCone \\cap \\{x^*\\}^\\perp \\subsetneq \\stdCone$ and \r\n$x^*_1 \\not \\in (\\stdCone^1)^\\perp = \\lineality( (\\stdCone^1 )^*)$.\r\n\\end{enumerate}\r\n\r\n\\item $\\opt{\\text{\\ref{eq:red}}} =  \\opt{\\text{\\ref{eq:red_dual}}} >  0$ if and only if the {\\PPS} condition is satisfied for \\eqref{eq:dual}. In this case, we have\r\n$\r\nc  -\\stdMap^\\T \\frac{y_3^*}{y_1^*} \\in (\\reInt \\stdCone ^1)\\times \\{0\\}. \r\n$ (In particular, it is feasible for \\eqref{eq:dual})\r\n\r\n\\end{enumerate}\r\n\\end{lemma}","start":28060,"end":29826}]},{"id":"S030","semantic_type":"assumption","kind":"assumption","labels":["lemma:red2"],"declared_external":false,"source_only":false,"title":"辅助引理的参数条件","source_units":[{"raw_tex":"\\begin{lemma}\\label{lemma:red2}\r\nLet $\\stdCone = \\stdCone ^1 \\times \\stdCone ^2$ be a \r\nclosed convex cone, such that $\\stdCone ^2$ is polyhedral.\r\nConsider the pair \\eqref{eq:red} and \\eqref{eq:red_dual} with \r\n$\\stdInt$ and $\\stdInt^*$ such that  $\\stdInt = (\\stdInt_1,0) \\in (\\reInt \\stdCone ^1)\\times \\{0\\}$ and \r\n$\\stdInt^* \\in \\reInt \\stdCone^*$.  The following properties hold.\r\n","start":28060,"end":28446}]},{"id":"S031","semantic_type":"lemma","kind":"lemma","labels":["lemma:red2"],"declared_external":false,"source_only":false,"title":"辅助问题的 PPS 可行点","source_units":[{"raw_tex":"\\begin{enumerate}[label=(\\roman*)]\r\n\\item The following are solutions to \\eqref{eq:red} and \\eqref{eq:red_dual} that satisfy the {\\PPS} condition: \r\n\\begin{align*}\r\n(x,t,w)& = \\left(\\frac{\\stdInt^*}{\\inProd{\\stdInt}{\\stdInt^*}+1},\\frac{1}{\\inProd{\\stdInt}{\\stdInt^*}+1}, \\frac{1}{\\inProd{\\stdInt}{\\stdInt^*}+1}\\right) \\\\\r\n (y_1,y_2,y_3) & = (0,-1,0).\r\n \\end{align*} \r\nIn particular, \r\n$\\opt{\\text{\\ref{eq:red}}} =  \\opt{\\text{\\ref{eq:red_dual}}}$.\r\n\\end{enumerate}\r\n","start":28446,"end":28912}]},{"id":"S032","semantic_type":"lemma","kind":"lemma","labels":["lemma:red2"],"declared_external":false,"source_only":false,"title":"零值与有效约化分支","source_units":[{"raw_tex":"Let $(x^*,t^*,w^*)$ be any primal optimal solution and \r\n$(y_1^*,y_2^*,y_3^*)$ be any dual optimal solution.\r\n \\begin{enumerate}[label=(\\roman*)]\r\n \\setcounter{enumi}{1} \r\n\\item $\\opt{\\text{\\ref{eq:red}}} =  \\opt{\\text{\\ref{eq:red_dual}}} = 0$ if and only if \r\none of the two alternatives below holds:\r\n\\begin{enumerate}\r\n\\item $\\inProd{c}{x^*} < 0 $ and $\\feasS = \\minFaceD = \\emptyset$, or\r\n\\item $\\inProd{c}{x^*} = 0 $, $\\feasS \\subseteq  \\stdCone \\cap \\{x^*\\}^\\perp \\subsetneq \\stdCone$ and \r\n$x^*_1 \\not \\in (\\stdCone^1)^\\perp = \\lineality( (\\stdCone^1 )^*)$.\r\n\\end{enumerate}\r\n\r\n","start":28912,"end":29497}]},{"id":"S033","semantic_type":"lemma","kind":"lemma","labels":["lemma:red2"],"declared_external":false,"source_only":false,"title":"正值等价于 PPS","source_units":[{"raw_tex":"\\item $\\opt{\\text{\\ref{eq:red}}} =  \\opt{\\text{\\ref{eq:red_dual}}} >  0$ if and only if the {\\PPS} condition is satisfied for \\eqref{eq:dual}. In this case, we have\r\n","start":29497,"end":29663}]},{"id":"S034","semantic_type":"lemma","kind":"lemma","labels":["lemma:red2"],"declared_external":false,"source_only":false,"title":"输出松弛的过强零块结论","source_units":[{"raw_tex":"$\r\nc  -\\stdMap^\\T \\frac{y_3^*}{y_1^*} \\in (\\reInt \\stdCone ^1)\\times \\{0\\}. \r\n$ (In particular, it is feasible for \\eqref{eq:dual})\r\n\r\n\\end{enumerate}\r\n","start":29663,"end":29815}]},{"id":"S035","semantic_type":"proof","kind":"proof","labels":["eq:lemma:aux"],"declared_external":false,"source_only":false,"title":"辅助引理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\r\n\r\n\\begin{enumerate}[label=({\\it{\\roman*}})]\r\n\\item  Due to choice of $\\stdInt$ and $\\stdInt^*$, it is clear that the solutions \r\nmeet the PPS condition.\r\n\r\n\r\n\\item \\fbox{$\\opt{\\text{\\ref{eq:red}}} = 0 \\Rightarrow$  (a) or (b) holds.}\r\nSuppose that $\\opt{\\text{\\ref{eq:red}}} = 0$ and let $(x^*,0,w^*)$ be an optimal solution for \\eqref{eq:red}.\r\nDue to the constraints in \\eqref{eq:red}, we  have\r\n$$\\stdMap x^* = 0, \\qquad -\\inProd{c}{x^*} = w^* \\geq 0, \\qquad x^* \\in \\stdCone ^*.$$ \r\nNote that if $\\inProd{c}{x^*} < 0$, then $\\minFaceD = \\feasS = \\emptyset$. This is \r\nalternative (a).\r\n\r\nIf $\\inProd{c}{x^*} = 0$, then (\\eqref{eq:red:1}) implies \r\n$w^* = 0$. Since $\\stdMap x^* = 0$, we obtain  \r\n$$\\feasS \\subseteq  \\stdCone \\cap \\{x^*\\}^\\perp. $$\r\nBy \\eqref{eq:red:2}, we have $\\inProd{\\stdInt}{x^*} = 1$, so that $x^* = (x_1^*,x_2^*)$ satisfies $x_1^* \\not \\in (\\stdCone^1)^\\perp$, due to the choice of $\\stdInt$. \r\nTherefore, the inclusion $\\stdCone \\cap \\{x^*\\}^\\perp \\subseteq \\stdCone$ is strict, \r\nthat is,  $\\stdCone \\cap \\{x^*\\}^\\perp \\subsetneq \\stdCone$. This is alternative (b).\r\n\r\n\\fbox{(a) or (b) holds $\\Rightarrow \\opt{\\text{\\ref{eq:red}}} = 0$.} \r\nFirst recall that  $t$ is constrained to be nonnegative, therefore  $\\opt{\\text{\\ref{eq:red}}} \\geq 0$.\r\nNext, we will prove the following inequality\r\n\\begin{equation}\\label{eq:lemma:aux}\r\n\t{\\inProd{\\stdInt}{x^*} - \\inProd{c}{x^*}} > 0.\r\n\\end{equation}\r\nSince $\\stdInt \\in \\stdCone,x^* \\in \\stdCone^*$, we have \r\n$\\inProd{\\stdInt}{x^*} \\geq 0$. If $(a)$ holds, then $- \\inProd{c}{x^*}> 0$.\r\nIn particular, \\eqref{eq:lemma:aux} holds.\r\nNow, we consider the case where (b) holds. \r\nIn this case, we have $\\inProd{c}{x^*} = 0$. \r\nIf $\\inProd{\\stdInt}{x^*}$ is also zero, then $\\inProd{\\stdInt_1}{x_1^*}$ is zero and since $\\stdInt_1 \\in \\reInt \\stdCone^1$, $ x^*_1$ must \r\nbelong to $(\\stdCone^1)^\\perp$. However, this is not allowed \r\nsince we assumed that $x_1^* \\not \\in (\\stdCone^1)^\\perp$.\r\nTherefore, we have $\\inProd{\\stdInt}{x^*} > 0$ and, in particular,  \\eqref{eq:lemma:aux} also holds.\r\n\r\nLet  $$\r\n\\alpha = \\frac{1}{\\inProd{\\stdInt}{x^*} - \\inProd{c}{x^*}}.\r\n$$\r\nThen $\\alpha > 0$ and \r\n$(\\alpha x^*,0,-\\alpha \\inProd{c}{x^*})$ is a solution to \\eqref{eq:red} with value $0$, \r\nso that $\\opt{\\text{\\ref{eq:red}}} = 0$.\r\n\r\n\\item \r\n\\fbox{ $\\opt{\\text{\\ref{eq:red_dual}}} > 0 \\Rightarrow $ PPS holds for \\eqref{eq:dual}}\r\nFrom \\eqref{eq:red_dual:1}, we have\r\n$$\r\nc y_1^* - \\stdInt y_2^* - {\\stdMap}^\\T y_3^* \\in \\stdCone.\r\n$$\r\nSince $y_2^*=\\opt{\\text{\\ref{eq:red_dual}}}\r\n>0$, we have, due to \\eqref{eq:red_dual:3}, $y_1^* > 0$.\r\n Hence, \r\n $$c - \\stdInt \\frac{y_2^*}{y_1^*} - {\\stdMap}^\\T \\frac{y_3^*}{y_1^*} \\in \\stdCone.$$ \r\nWe conclude that \r\n$c - {\\stdMap}^\\T y_3^*/y_1^* \\in (\\stdCone^1 +y_2^*/y_1^* \\stdInt_1)\\times  \\{0\\} \\subseteq (\\reInt \\stdCone ^1)\\times \\{0\\}$, due to the choice of  $\\stdInt$ and the fact that $y_2^*/y_1^* > 0$. \r\n\r\n\\fbox{ $\\opt{\\text{\\ref{eq:red_dual}}} = 0 \\Rightarrow $ PPS does not hold for \\eqref{eq:dual}  } \r\nIf $t^* = 0$ and $(x^*,0,w^*)$ is an optimal solution \r\nfor \\eqref{eq:red}, then either $(a)$ or $(b)$ of item $(ii)$ is satisfied. \r\nIf $(a)$ is satisfied, then \\eqref{eq:dual} is infeasible and thus the PPS condition\r\ncannot hold. \r\nIf $(b)$ is satisfied, then $\\inProd{c}{x^*} = 0$,\r\n$\\stdMap x^* = 0$ and $x_1^* \\not \\in (\\stdCone ^1)^\\perp$. If \r\n$(s_1,s_2)$ is a feasible slack for \\eqref{eq:dual}, we have $\\inProd{s_1}{x_1^*} + \\inProd{s_2}{x_2^*} = 0$, so that \r\n$\\inProd{s_1}{x_1^*} = 0$.  \r\nAs $x_1^* \\not \\in (\\stdCone ^1)^\\perp$, we have that $s_1 \\not \\in \\reInt \\stdCone^1$. This means\r\nthat there is no feasible solution for \\eqref{eq:dual} satisfying the PPS\r\ncondition.\r\n\\end{enumerate}\r\n\\end{proof}","start":31151,"end":34906}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"验证两侧 PPS 可行性","source_units":[{"raw_tex":"\\begin{proof}\r\n\r\n\\begin{enumerate}[label=({\\it{\\roman*}})]\r\n\\item  Due to choice of $\\stdInt$ and $\\stdInt^*$, it is clear that the solutions \r\nmeet the PPS condition.\r\n\r\n\r\n","start":31151,"end":31324}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"零值产生不可行证书","source_units":[{"raw_tex":"\\item \\fbox{$\\opt{\\text{\\ref{eq:red}}} = 0 \\Rightarrow$  (a) or (b) holds.}\r\nSuppose that $\\opt{\\text{\\ref{eq:red}}} = 0$ and let $(x^*,0,w^*)$ be an optimal solution for \\eqref{eq:red}.\r\nDue to the constraints in \\eqref{eq:red}, we  have\r\n$$\\stdMap x^* = 0, \\qquad -\\inProd{c}{x^*} = w^* \\geq 0, \\qquad x^* \\in \\stdCone ^*.$$ \r\nNote that if $\\inProd{c}{x^*} < 0$, then $\\minFaceD = \\feasS = \\emptyset$. This is \r\nalternative (a).\r\n\r\n","start":31324,"end":31758}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"零配对约化非多面体部分","source_units":[{"raw_tex":"If $\\inProd{c}{x^*} = 0$, then (\\eqref{eq:red:1}) implies \r\n$w^* = 0$. Since $\\stdMap x^* = 0$, we obtain  \r\n$$\\feasS \\subseteq  \\stdCone \\cap \\{x^*\\}^\\perp. $$\r\nBy \\eqref{eq:red:2}, we have $\\inProd{\\stdInt}{x^*} = 1$, so that $x^* = (x_1^*,x_2^*)$ satisfies $x_1^* \\not \\in (\\stdCone^1)^\\perp$, due to the choice of $\\stdInt$. \r\nTherefore, the inclusion $\\stdCone \\cap \\{x^*\\}^\\perp \\subseteq \\stdCone$ is strict, \r\nthat is,  $\\stdCone \\cap \\{x^*\\}^\\perp \\subsetneq \\stdCone$. This is alternative (b).\r\n\r\n","start":31758,"end":32265}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":["eq:lemma:aux"],"declared_external":false,"source_only":false,"title":"建立归一化分母正性","source_units":[{"raw_tex":"\\fbox{(a) or (b) holds $\\Rightarrow \\opt{\\text{\\ref{eq:red}}} = 0$.} \r\nFirst recall that  $t$ is constrained to be nonnegative, therefore  $\\opt{\\text{\\ref{eq:red}}} \\geq 0$.\r\nNext, we will prove the following inequality\r\n\\begin{equation}\\label{eq:lemma:aux}\r\n\t{\\inProd{\\stdInt}{x^*} - \\inProd{c}{x^*}} > 0.\r\n\\end{equation}\r\nSince $\\stdInt \\in \\stdCone,x^* \\in \\stdCone^*$, we have \r\n$\\inProd{\\stdInt}{x^*} \\geq 0$. If $(a)$ holds, then $- \\inProd{c}{x^*}> 0$.\r\nIn particular, \\eqref{eq:lemma:aux} holds.\r\nNow, we consider the case where (b) holds. \r\nIn this case, we have $\\inProd{c}{x^*} = 0$. \r\nIf $\\inProd{\\stdInt}{x^*}$ is also zero, then $\\inProd{\\stdInt_1}{x_1^*}$ is zero and since $\\stdInt_1 \\in \\reInt \\stdCone^1$, $ x^*_1$ must \r\nbelong to $(\\stdCone^1)^\\perp$. However, this is not allowed \r\nsince we assumed that $x_1^* \\not \\in (\\stdCone^1)^\\perp$.\r\nTherefore, we have $\\inProd{\\stdInt}{x^*} > 0$ and, in particular,  \\eqref{eq:lemma:aux} also holds.\r\n","start":32265,"end":33231}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"缩放得到零值可行解","source_units":[{"raw_tex":"\r\nLet  $$\r\n\\alpha = \\frac{1}{\\inProd{\\stdInt}{x^*} - \\inProd{c}{x^*}}.\r\n$$\r\nThen $\\alpha > 0$ and \r\n$(\\alpha x^*,0,-\\alpha \\inProd{c}{x^*})$ is a solution to \\eqref{eq:red} with value $0$, \r\nso that $\\opt{\\text{\\ref{eq:red}}} = 0$.\r\n\r\n","start":33231,"end":33466}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"正目标值构造 PPS 松弛","source_units":[{"raw_tex":"\\item \r\n\\fbox{ $\\opt{\\text{\\ref{eq:red_dual}}} > 0 \\Rightarrow $ PPS holds for \\eqref{eq:dual}}\r\nFrom \\eqref{eq:red_dual:1}, we have\r\n$$\r\nc y_1^* - \\stdInt y_2^* - {\\stdMap}^\\T y_3^* \\in \\stdCone.\r\n$$\r\nSince $y_2^*=\\opt{\\text{\\ref{eq:red_dual}}}\r\n>0$, we have, due to \\eqref{eq:red_dual:3}, $y_1^* > 0$.\r\n Hence, \r\n $$c - \\stdInt \\frac{y_2^*}{y_1^*} - {\\stdMap}^\\T \\frac{y_3^*}{y_1^*} \\in \\stdCone.$$ \r\nWe conclude that \r\n$c - {\\stdMap}^\\T y_3^*/y_1^* \\in (\\stdCone^1 +y_2^*/y_1^* \\stdInt_1)\\times  \\{0\\} \\subseteq (\\reInt \\stdCone ^1)\\times \\{0\\}$, due to the choice of  $\\stdInt$ and the fact that $y_2^*/y_1^* > 0$. \r\n\r\n","start":33466,"end":34089}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"零目标值排除 PPS","source_units":[{"raw_tex":"\\fbox{ $\\opt{\\text{\\ref{eq:red_dual}}} = 0 \\Rightarrow $ PPS does not hold for \\eqref{eq:dual}  } \r\nIf $t^* = 0$ and $(x^*,0,w^*)$ is an optimal solution \r\nfor \\eqref{eq:red}, then either $(a)$ or $(b)$ of item $(ii)$ is satisfied. \r\nIf $(a)$ is satisfied, then \\eqref{eq:dual} is infeasible and thus the PPS condition\r\ncannot hold. \r\nIf $(b)$ is satisfied, then $\\inProd{c}{x^*} = 0$,\r\n$\\stdMap x^* = 0$ and $x_1^* \\not \\in (\\stdCone ^1)^\\perp$. If \r\n$(s_1,s_2)$ is a feasible slack for \\eqref{eq:dual}, we have $\\inProd{s_1}{x_1^*} + \\inProd{s_2}{x_2^*} = 0$, so that \r\n$\\inProd{s_1}{x_1^*} = 0$.  \r\nAs $x_1^* \\not \\in (\\stdCone ^1)^\\perp$, we have that $s_1 \\not \\in \\reInt \\stdCone^1$. This means\r\nthat there is no feasible solution for \\eqref{eq:dual} satisfying the PPS\r\ncondition.\r\n\\end{enumerate}\r\n\\end{proof}","start":34089,"end":34906}]},{"id":"S043","semantic_type":"proposition","kind":"proposition","labels":["prop:strict"],"declared_external":false,"source_only":false,"title":"严格互补实现一步到面","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:strict}\r\nSuppose $\\opt{\\text{\\ref{eq:red}}} = \\opt{\\text{\\ref{eq:red_dual}}} = 0$  and that  we have  optimal solutions to \\eqref{eq:red} and \r\n\\eqref{eq:red_dual} satisfying dual strict complementarity. \r\nIf $w^* = 0$, then $\\minFaceD = \\stdCone \\cap\\{ x^*\\}^\\perp$.\r\n%Reciprocally, suppose that $\\minFace{D} \\neq \\stdCone $ and that there is a reducing direction $d$ for which $\\minFaceD = \\stdCone \\cap\\{ d\\}^\\perp$. Then, \r\n%\\eqref{eq:red} and \\eqref{eq:red_dual} satisfy dual strict complementarity if $\\stdInt$ is chosen as to satisfy $\\inProd{\\stdInt}{d} \\neq 0$ (in particular, if \r\n%$\\stdInt \\in \\reInt \\stdCone$).\r\n\\end{proposition}","start":46831,"end":47504}]},{"id":"S044","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"一步到面命题的作者证明","source_units":[{"raw_tex":"\\begin{proof}\r\nSince $w^* = 0$, \\eqref{eq:strict3} implies that $y_1^* > y_2^*$. Then, $\\opt{\\text{\\ref{eq:red}}} = \\opt{\\text{\\ref{eq:red_dual}}} $ and \r\n$\\opt{\\text{\\ref{eq:red}}} \\geq 0 $ implies that $y_2^* \\geq 0$, so that $y_1^* > 0$ as well.\r\nTherefore, from \\eqref{eq:strict1} we obtain $c -\\stdMap^\\T \\frac{y_3^*}{y_1^*} \\in \\reInt (\\stdCone \\cap\\{ x^*\\}^\\perp).$\r\n%To prove converse, let $y$ be such that $c - \\stdMap^\\T y \\in \\reInt (\\stdCone \\cap\\{ d\\}^\\perp)$. Then, for $\\alpha > 0$ sufficiently small, we have \r\n%that $(d/\\inProd{e}{d},0,0)$, $(\\alpha,0,\\alpha y)$ are optimal solutions satisfying strict complementarity.\r\n\\end{proof}","start":47506,"end":48155}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"严格互补保证除数为正","source_units":[{"raw_tex":"\\begin{proof}\r\nSince $w^* = 0$, \\eqref{eq:strict3} implies that $y_1^* > y_2^*$. Then, $\\opt{\\text{\\ref{eq:red}}} = \\opt{\\text{\\ref{eq:red_dual}}} $ and \r\n$\\opt{\\text{\\ref{eq:red}}} \\geq 0 $ implies that $y_2^* \\geq 0$, so that $y_1^* > 0$ as well.\r\n","start":47506,"end":47756}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"构造截面内部的可行松弛","source_units":[{"raw_tex":"Therefore, from \\eqref{eq:strict1} we obtain $c -\\stdMap^\\T \\frac{y_3^*}{y_1^*} \\in \\reInt (\\stdCone \\cap\\{ x^*\\}^\\perp).$\r\n","start":47756,"end":47880}]},{"id":"S047","semantic_type":"proposition","kind":"proposition","labels":["prop:phase1"],"declared_external":false,"source_only":false,"title":"第一阶段正确性与计数","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:phase1}\r\nThe following hold.\r\n\\begin{enumerate}[label=(\\roman*)]\r\n\t\\item if \\eqref{eq:dual} is feasible, then the output face $\\stdFace$ satisfies \r\n\t$\\minFaceD \\subseteq \\stdFace$. Moreover, $s'$ is a dual \r\nfeasible slack such that $s_j' \\in \\reInt \\stdFace ^j $  for every $j$ such that $\\stdFace ^j$  is nonpolyhedral, i.e., the PPS condition is satisfied for $\\stdFace$.\r\n\r\nIn this case, Phase 1  stops after finding at most $\\sum _{i=1}^{r} \\distP(\\stdCone ^i) $ directions. \r\n\t\\item \\eqref{eq:dual} is infeasible if and only if Step 3 is reached. In this case, Phase 1 stops after finding \r\n\tat most $1+ \\sum _{i=1}^{r} \\distP(\\stdCone ^i) $ directions.\r\n\\end{enumerate}\r\n\\end{proposition}","start":51430,"end":52157}]},{"id":"S048","semantic_type":"proposition","kind":"proposition","labels":["prop:phase1"],"declared_external":false,"source_only":false,"title":"第一阶段的逐块内部松弛","source_units":[{"raw_tex":"\t\\item if \\eqref{eq:dual} is feasible, then the output face $\\stdFace$ satisfies \r\n\t$\\minFaceD \\subseteq \\stdFace$. Moreover, $s'$ is a dual \r\nfeasible slack such that $s_j' \\in \\reInt \\stdFace ^j $  for every $j$ such that $\\stdFace ^j$  is nonpolyhedral, i.e., the PPS condition is satisfied for $\\stdFace$.\r\n","start":51527,"end":51838}]},{"id":"S049","semantic_type":"proposition","kind":"proposition","labels":["prop:phase1"],"declared_external":false,"source_only":false,"title":"可行时第一阶段方向数","source_units":[{"raw_tex":"\r\nIn this case, Phase 1  stops after finding at most $\\sum _{i=1}^{r} \\distP(\\stdCone ^i) $ directions. \r\n","start":51838,"end":51944}]},{"id":"S050","semantic_type":"proposition","kind":"proposition","labels":["prop:phase1"],"declared_external":false,"source_only":false,"title":"不可行检测及方向数","source_units":[{"raw_tex":"\t\\item \\eqref{eq:dual} is infeasible if and only if Step 3 is reached. In this case, Phase 1 stops after finding \r\n\tat most $1+ \\sum _{i=1}^{r} \\distP(\\stdCone ^i) $ directions.\r\n","start":51944,"end":52123}]},{"id":"S051","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"第一阶段命题的完整证明","source_units":[{"raw_tex":"\\begin{proof}\r\nWe will focus on the statements about the bounds, since the other statements \r\nare direct consequences of \r\nLemma \\ref{lemma:red2}. Note that whenever Step 4 is reached, we have \r\n$\\stdFace _{i+1} \\subsetneq \\stdFace _i$, since \r\n$x^*_j \\not \\in (\\stdFace _i^j)^\\perp$ for at least one nonpolyhedral cone $\\stdFace _i^j$, due to item $(ii)$-$(b)$ of Lemma \\ref{lemma:red2}. \r\nTherefore, whenever a new (proper) face is found, it is because \r\nwe are making progress towards a polyhedral face for at least one nonpolyhedral cone. \t\r\n\t\r\nBy definition, after finding $\\hat \\ell = \\sum _{i=1}^{r} (\\distP(\\stdCone ^i)) $ directions, \r\n$\\stdFace _{\\hat \\ell + 1}$ is polyhedral. We now consider what happens if \r\nthe algorithm has not stopped after all these directions were found.\r\nIn this case, when it is time to build \\eqref{eq:red} and \\eqref{eq:red_dual} with $\\stdFace_{l+1}$ in\r\nplace of $\\stdCone$ at Step 2, Phase 1 selects $e = 0$ and $\\stdInt^* \\in (\\stdFace _{\\hat \\ell + 1})^*$.\r\n\r\nFirst, suppose that \\eqref{eq:dual} is feasible and  let \r\n$y$ be such that $c-\\stdMap^\\T y \\in \\feasS$. \r\nSince $\\stdInt = 0$, $(\\alpha, \\alpha, \\alpha y)$\r\nis a feasible solution for \\eqref{eq:red_dual}, when $\\alpha>0$ is sufficiently small. It follows \r\nthat $\\opt{\\text{\\ref{eq:red_dual}}} > 0$ and that Phase 1 eventually reaches Step 5. This gives item $(i)$.\r\n\r\n\r\n\r\nFinally, suppose that \\eqref{eq:dual} is infeasible. By Lemma \\ref{lemma:red2},  $\\opt{\\text{\\ref{eq:red_dual}}} = 0$. Since $\\stdInt = 0$,  \\eqref{eq:red:2} implies that every optimal solution of \r\n\\eqref{eq:red} will be a triple of the form\r\n$(x^*,0,1)$, which implies that Step 3 will be reached and a single new direction will be \r\nadded. This gives  item $(ii)$.\r\n\\end{proof}","start":52159,"end":53918}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"调用辅助引理保证正确性","source_units":[{"raw_tex":"\\begin{proof}\r\nWe will focus on the statements about the bounds, since the other statements \r\nare direct consequences of \r\nLemma \\ref{lemma:red2}. Note that whenever Step 4 is reached, we have \r\n","start":52159,"end":52354}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"每次截面推进非多面体块","source_units":[{"raw_tex":"$\\stdFace _{i+1} \\subsetneq \\stdFace _i$, since \r\n$x^*_j \\not \\in (\\stdFace _i^j)^\\perp$ for at least one nonpolyhedral cone $\\stdFace _i^j$, due to item $(ii)$-$(b)$ of Lemma \\ref{lemma:red2}. \r\nTherefore, whenever a new (proper) face is found, it is because \r\nwe are making progress towards a polyhedral face for at least one nonpolyhedral cone. \t\r\n\t\r\n","start":52354,"end":52708}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"距离之和控制多面体终点","source_units":[{"raw_tex":"By definition, after finding $\\hat \\ell = \\sum _{i=1}^{r} (\\distP(\\stdCone ^i)) $ directions, \r\n$\\stdFace _{\\hat \\ell + 1}$ is polyhedral. We now consider what happens if \r\nthe algorithm has not stopped after all these directions were found.\r\nIn this case, when it is time to build \\eqref{eq:red} and \\eqref{eq:red_dual} with $\\stdFace_{l+1}$ in\r\nplace of $\\stdCone$ at Step 2, Phase 1 selects $e = 0$ and $\\stdInt^* \\in (\\stdFace _{\\hat \\ell + 1})^*$.\r\n\r\n","start":52708,"end":53164}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"多面体可行终点直接停止","source_units":[{"raw_tex":"First, suppose that \\eqref{eq:dual} is feasible and  let \r\n$y$ be such that $c-\\stdMap^\\T y \\in \\feasS$. \r\nSince $\\stdInt = 0$, $(\\alpha, \\alpha, \\alpha y)$\r\nis a feasible solution for \\eqref{eq:red_dual}, when $\\alpha>0$ is sufficiently small. It follows \r\nthat $\\opt{\\text{\\ref{eq:red_dual}}} > 0$ and that Phase 1 eventually reaches Step 5. This gives item $(i)$.\r\n\r\n\r\n\r\n","start":53164,"end":53538}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"多面体不可行终点增加证书","source_units":[{"raw_tex":"Finally, suppose that \\eqref{eq:dual} is infeasible. By Lemma \\ref{lemma:red2},  $\\opt{\\text{\\ref{eq:red_dual}}} = 0$. Since $\\stdInt = 0$,  \\eqref{eq:red:2} implies that every optimal solution of \r\n\\eqref{eq:red} will be a triple of the form\r\n$(x^*,0,1)$, which implies that Step 3 will be reached and a single new direction will be \r\nadded. This gives  item $(ii)$.\r\n\\end{proof}","start":53538,"end":53918}]},{"id":"S057","semantic_type":"proposition","kind":"proposition","labels":["prop:min_face"],"declared_external":false,"source_only":false,"title":"第二阶段得到最小面","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:min_face}\r\nThe output face of Phase 2 is $\\minFaceD$ and there exists $\\hat s$ as in Steps 3. and 4.\r\n\\end{proposition}","start":62497,"end":62647}]},{"id":"S058","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"第二阶段命题的完整证明","source_units":[{"raw_tex":"\\begin{proof}\r\nSuppose that the output face $\\stdFace$ of Phase 1 satisfies $\\stdFace \\neq \\minFaceD$. \r\nBy Lemma 3.2 in \\cite{article_waki_muramatsu}, there is \r\na reducing direction $x$ such that $x \\in \\stdFace ^* \\cap \\ker \\stdMap \\cap \\{c\\}^\\perp$ \r\nand $x \\not \\in \\stdFace ^\\perp$.\r\nDue to Proposition \\ref{prop:phase1}, any such reducing direction $x$ satisfies $\\inProd{x}{s'} = 0$, which implies that \r\n $x_j \\in ({\\stdFace ^j})^\\perp = ({\\spanVec \\stdFace ^j})^\\perp$ for every $j$ such that $\\stdFace ^j$ is not polyhedral, \r\n since $s_j' \\in \\reInt \\stdFace ^j $ for those $j$.\r\nTherefore, the possible reducing directions are confined to the polyhedral cone $\\hat \\stdCone^*$,\r\nwhere $\\hat \\stdCone$ is the cone in Step 1. of Phase 2.\r\n\r\nSince $\\hat \\stdCone$ is polyhedral, the problems \\eqref{eq:red} and \\eqref{eq:red_dual} \r\n are polyhedral and they admit strictly complementary optimal \r\nsolutions $(x^*,t^*,w^*)$, $(y_1^*,y_2^*,y_3^*)$.\r\nThe fact that $x \\not \\in \\stdFace^\\perp$ implies that $\\inProd{\\stdInt}{x}\\neq 0$ so that $(x/\\inProd{\\stdInt}{x},0,0) $ is an optimal \r\nsolution to \\eqref{eq:red}. Therefore, $t^* = y_2^* = 0$. Moreover, since \\eqref{eq:dual} is \r\nfeasible, we have $w^* = 0$.\r\nBy Proposition \\ref{prop:strict}, we have \r\n$c -\\stdMap^\\T  \\frac{y_3^*}{y_1^*} \\in \\reInt (\\hat \\stdCone \\cap \\{x^*\\}^\\perp).$\r\n\r\nLet $\\tilde s = c -\\stdMap^\\T  \\frac{y_3^*}{y_1^*}$. Note that $\\stdFace \\cap \\{x^*\\}^\\perp$ is a face of \r\n$\\stdFace$ containing $\\feasS$, since we argued that $x^*$ must be a reducing direction. We will prove that $\\minFaceD = \\stdFace \\cap \\{x^*\\}^\\perp$ by \r\n showing that some convex combination of $s'$ and $\\tilde s$ lies in $\\reInt (\\stdFace \\cap \\{x^*\\}^\\perp)$. \r\n% Intuitively, $s'$ and $\\tilde s$ are  relative interior points \r\n%of the nonpolyhedral and polyhedral parts of $\\stdFace \\cap \\{x^*\\}^\\perp$, respectively. So some convex \r\n%combination of them will belong to $\\feasS$ and be a relative interior point of $\\stdFace \\cap \\{x^*\\}^\\perp$.\r\n\r\nLet $z_\\beta = \\beta s' + (1-\\beta)\\tilde s$. For \r\nall $\\beta \\in (0,1)$ and all $j$ such that $\\stdFace ^j$ is polyhedral, we \r\nhave $(z_\\beta)_j \\in \\reInt ( \\stdFace ^j  \\cap \\{x_j^*\\}^\\perp)$, because $\\tilde s _j \\in  \\reInt (\\stdFace ^j  \\cap \\{x_j\\}^\\perp)$ and \r\n$s'$ is feasible.\r\nIf $\\stdFace ^j$ is not polyhedral, then $\\stdFace ^j  \\cap \\{x_j^*\\}^\\perp = \\stdFace ^j$, \r\nsince $x_j \\in (\\stdFace ^j)^\\perp$. Because $\\tilde s _j \\in \\spanVec{\\stdFace ^j}$ and \r\n$s_j' \\in \\reInt \\stdFace^j$, for all $\\beta $ sufficiently close to $1$ we have \r\n$(z_\\beta)_j \\in \\reInt \\stdFace^j$. Therefore, it is possible to select $\\beta \\in (0,1)$ \r\nsuch that $(z_\\beta)_j \\in \\reInt (\\stdFace ^j  \\cap \\{x_j\\}^\\perp) $ for all $j$. This \r\nshows that $\\minFaceD = \\stdFace \\cap \\{x^*\\}^\\perp $.\r\n\r\nIf $\\stdFace$ was already the minimal face to begin with, then \r\n$t^* > 0$. We can then  proceed in a \r\nsimilar fashion. The only difference is that due to \\eqref{eq:red_dual:1}, we \r\nwill have that $\\tilde s = c -\\stdMap^\\T  \\frac{y_3^*}{y_1^*} $ satisfies \r\n$\\tilde s _j \\in  \\reInt (\\stdFace ^j)$ for every $j$ such that $\\stdFace ^j$ is polyhedral.\r\nAnd as before, we can select a convex combination of $s'$ and $\\tilde s$ belonging to \r\nthe relative interior of $\\minFaceD$.\r\n\\end{proof}","start":62649,"end":65957}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"未达最小面则存在有效方向","source_units":[{"raw_tex":"\\begin{proof}\r\nSuppose that the output face $\\stdFace$ of Phase 1 satisfies $\\stdFace \\neq \\minFaceD$. \r\nBy Lemma 3.2 in \\cite{article_waki_muramatsu}, there is \r\na reducing direction $x$ such that $x \\in \\stdFace ^* \\cap \\ker \\stdMap \\cap \\{c\\}^\\perp$ \r\nand $x \\not \\in \\stdFace ^\\perp$.\r\n","start":62649,"end":62939}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"约化方向限于放松锥对偶","source_units":[{"raw_tex":"Due to Proposition \\ref{prop:phase1}, any such reducing direction $x$ satisfies $\\inProd{x}{s'} = 0$, which implies that \r\n $x_j \\in ({\\stdFace ^j})^\\perp = ({\\spanVec \\stdFace ^j})^\\perp$ for every $j$ such that $\\stdFace ^j$ is not polyhedral, \r\n since $s_j' \\in \\reInt \\stdFace ^j $ for those $j$.\r\nTherefore, the possible reducing directions are confined to the polyhedral cone $\\hat \\stdCone^*$,\r\nwhere $\\hat \\stdCone$ is the cone in Step 1. of Phase 2.\r\n\r\n","start":62939,"end":63401}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"多面体辅助问题达到零值","source_units":[{"raw_tex":"Since $\\hat \\stdCone$ is polyhedral, the problems \\eqref{eq:red} and \\eqref{eq:red_dual} \r\n are polyhedral and they admit strictly complementary optimal \r\nsolutions $(x^*,t^*,w^*)$, $(y_1^*,y_2^*,y_3^*)$.\r\nThe fact that $x \\not \\in \\stdFace^\\perp$ implies that $\\inProd{\\stdInt}{x}\\neq 0$ so that $(x/\\inProd{\\stdInt}{x},0,0) $ is an optimal \r\nsolution to \\eqref{eq:red}. Therefore, $t^* = y_2^* = 0$. Moreover, since \\eqref{eq:dual} is \r\nfeasible, we have $w^* = 0$.\r\n","start":63401,"end":63870}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"获得放松截面的内部松弛","source_units":[{"raw_tex":"By Proposition \\ref{prop:strict}, we have \r\n$c -\\stdMap^\\T  \\frac{y_3^*}{y_1^*} \\in \\reInt (\\hat \\stdCone \\cap \\{x^*\\}^\\perp).$\r\n\r\n","start":63870,"end":64001}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"以内部可行点识别目标截面","source_units":[{"raw_tex":"Let $\\tilde s = c -\\stdMap^\\T  \\frac{y_3^*}{y_1^*}$. Note that $\\stdFace \\cap \\{x^*\\}^\\perp$ is a face of \r\n$\\stdFace$ containing $\\feasS$, since we argued that $x^*$ must be a reducing direction. We will prove that $\\minFaceD = \\stdFace \\cap \\{x^*\\}^\\perp$ by \r\n showing that some convex combination of $s'$ and $\\tilde s$ lies in $\\reInt (\\stdFace \\cap \\{x^*\\}^\\perp)$. \r\n% Intuitively, $s'$ and $\\tilde s$ are  relative interior points \r\n%of the nonpolyhedral and polyhedral parts of $\\stdFace \\cap \\{x^*\\}^\\perp$, respectively. So some convex \r\n%combination of them will belong to $\\feasS$ and be a relative interior point of $\\stdFace \\cap \\{x^*\\}^\\perp$.\r\n\r\n","start":64001,"end":64665}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"凸组合保持多面体块内部性","source_units":[{"raw_tex":"Let $z_\\beta = \\beta s' + (1-\\beta)\\tilde s$. For \r\nall $\\beta \\in (0,1)$ and all $j$ such that $\\stdFace ^j$ is polyhedral, we \r\nhave $(z_\\beta)_j \\in \\reInt ( \\stdFace ^j  \\cap \\{x_j^*\\}^\\perp)$, because $\\tilde s _j \\in  \\reInt (\\stdFace ^j  \\cap \\{x_j\\}^\\perp)$ and \r\n$s'$ is feasible.\r\n","start":64665,"end":64956}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"靠近原松弛恢复非线性块","source_units":[{"raw_tex":"If $\\stdFace ^j$ is not polyhedral, then $\\stdFace ^j  \\cap \\{x_j^*\\}^\\perp = \\stdFace ^j$, \r\nsince $x_j \\in (\\stdFace ^j)^\\perp$. Because $\\tilde s _j \\in \\spanVec{\\stdFace ^j}$ and \r\n$s_j' \\in \\reInt \\stdFace^j$, for all $\\beta $ sufficiently close to $1$ we have \r\n$(z_\\beta)_j \\in \\reInt \\stdFace^j$. Therefore, it is possible to select $\\beta \\in (0,1)$ \r\nsuch that $(z_\\beta)_j \\in \\reInt (\\stdFace ^j  \\cap \\{x_j\\}^\\perp) $ for all $j$. This \r\nshows that $\\minFaceD = \\stdFace \\cap \\{x^*\\}^\\perp $.\r\n","start":64956,"end":65463}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"已在最小面时的处理","source_units":[{"raw_tex":"\r\nIf $\\stdFace$ was already the minimal face to begin with, then \r\n$t^* > 0$. We can then  proceed in a \r\nsimilar fashion. The only difference is that due to \\eqref{eq:red_dual:1}, we \r\nwill have that $\\tilde s = c -\\stdMap^\\T  \\frac{y_3^*}{y_1^*} $ satisfies \r\n$\\tilde s _j \\in  \\reInt (\\stdFace ^j)$ for every $j$ such that $\\stdFace ^j$ is polyhedral.\r\nAnd as before, we can select a convex combination of $s'$ and $\\tilde s$ belonging to \r\nthe relative interior of $\\minFaceD$.\r\n\\end{proof}","start":65463,"end":65957}]},{"id":"S067","semantic_type":"theorem","kind":"theorem","labels":["theo:fra_bound"],"declared_external":false,"source_only":false,"title":"FRA-Poly 面约化步数界","source_units":[{"raw_tex":"\\begin{theorem}\\label{theo:fra_bound}\r\nLet $\\stdCone = \\stdCone ^1\\times \\ldots \\times \\stdCone^r$. The minimum face $\\minFaceD$ \r\nthat contains the feasible region of \\eqref{eq:dual} can be found in no more than \r\n$1+ \\sum _{i=1}^{r} \\distP(\\stdCone ^i) $ facial reduction steps.\r\n\\end{theorem}","start":66021,"end":66316}]},{"id":"S068","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\r\nIf \\eqref{eq:dual} is infeasible, then $\\minFaceD = \\emptyset$ and the result \r\nfollows from Proposition \\ref{prop:phase1}. So suppose now that \r\n\\eqref{eq:dual} is feasible. Then Phase 1 ends after finding at most \r\n$\\sum _{i=1}^{r} \\distP(\\stdCone ^i)$ directions. Due to Proposition \\ref{prop:min_face}, at most one \r\nextra direction is needed to jump to the minimal face.\r\n\\end{proof}","start":66318,"end":66721}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"主证明的不可行分支","source_units":[{"raw_tex":"\\begin{proof}\r\nIf \\eqref{eq:dual} is infeasible, then $\\minFaceD = \\emptyset$ and the result \r\nfollows from Proposition \\ref{prop:phase1}. So suppose now that \r\n","start":66318,"end":66479}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"主证明的可行分支","source_units":[{"raw_tex":"\\eqref{eq:dual} is feasible. Then Phase 1 ends after finding at most \r\n$\\sum _{i=1}^{r} \\distP(\\stdCone ^i)$ directions. Due to Proposition \\ref{prop:min_face}, at most one \r\nextra direction is needed to jump to the minimal face.\r\n\\end{proof}","start":66479,"end":66721}]},{"id":"S071","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"PPS 强对偶的附录证明","source_units":[{"raw_tex":"\\begin{proof}\r\nWe will prove $(i)$ first. Let \r\n$f_1$ be such that $f_1(x) = \\inProd{c}{x}$ if $Ax = b$ and $+\\infty$ otherwise.\r\nLet  $f_2$ be the indicator function of $\\Re^{n_1}\\times (\\stdCone ^2)^*$ and $f_3$ be the indicator \r\nfunction of $(\\stdCone ^1)^*\\times \\Re^{n_2}$. Since there is  a primal feasible solution $x = (x_1,x_2)$ such \r\nthat $x_1 \\in \\reInt (\\stdCone ^1)^*$, we have that $\\fDom f_1 \\cap \\fDom f_2 \\cap \\reInt (\\fDom f_3)$ is \r\nnonempty. In addition, $f_1$ and $f_2$ are polyhedral functions. Let us now observe that:\r\n\\begin{align*}\r\nf_1^*(s) = \\left\\{\r\n  \\begin{array}{l l}\r\n    \\inProd{b}{y}  & \\quad \\text{if there is $y$ with $s-c = \\stdMap^\\T y$}\\\\\r\n    +\\infty & \\quad \\text{otherwise}\r\n  \\end{array}\\right. \r\n\\end{align*}\r\nNote that, due to feasibility, for fixed $s$, $\\inProd{b}{y}$ does not depend on the choice \r\nof $y$, as long as $c+\\stdMap ^\\T y = s$. This is because since there is $x$ such that $Ax = b$, we have $\\inProd{b}{y} = \\inProd{x}{s-c}$.\r\nThe conjugate $f_2^*$ is the indicator function of $- \\{0\\}\\times \\stdCone ^2$ and \r\n$f_3^*$ is the indicator function of $- \\stdCone ^1\\times \\{0\\}$. \r\nApplying Theorem \\ref{theo:conv} with $s = 0$, we have:\r\n\\begin{align*}\r\n(f_1+f_2+f_3)^*(0) & = \\inf \\left\\{ \\inProd{b}{y} \\mid  c+ \\stdMap ^\\T y = s_1, s_1 - (s_3,s_2) = 0,  s_2 \\in  \\stdCone ^2, s_3 \\in \\stdCone ^1      \\right\\} \\\\\r\n & = \\inf \\left\\{ \\inProd{b}{y} \\mid  c+ \\stdMap ^\\T y = s_1, s_1 \\in  \\stdCone ^1\\times \\stdCone ^2      \\right\\} \\\\\r\n& = - \\sup \\left\\{ \\inProd{b}{y} \\mid  c -\\stdMap ^\\T y = s_1,  s_1 \\in  \\stdCone ^1 \\times \\stdCone ^2      \\right\\},\r\n\\end{align*}\r\nwhere the $\\sup$ in the last equation is attained. So, there is some dual feasible $y$ such that \r\n$(f_1+f_2+f_3)^*(0) =  \\inProd{b}{y}$. However, using the definition of conjugate, we also have:\r\n\\begin{align*}\r\n(f_1+f_2+f_3)^*(0) = - \\inf \\{ \\inProd{c}{x} \\mid Ax = b, x \\in \\stdCone ^1\\times \\stdCone ^2\\} = -\\theta _P.\r\n\\end{align*}\r\nIt follows that $\\pOpt = \\dOpt$ and the dual is attained at $y$. \r\nTo prove $(ii)$, let $g_1 = f_1^*$, and let $g_2$ and $g_3$ be the indicator \r\nfunctions  of $\\Re^{n_1}\\times \\stdCone ^2$ and  $\\stdCone ^1 \\times \\Re^{n_2}$, respectively. Again, \r\nit is enough to compute $(g_1+g_2+g_3)^*(0)$ using both the definition of conjugate function and \r\nusing Theorem \\ref{theo:conv}. \r\n\\end{proof}","start":116866,"end":119230}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"构造函数并核对交点条件","source_units":[{"raw_tex":"\\begin{proof}\r\nWe will prove $(i)$ first. Let \r\n$f_1$ be such that $f_1(x) = \\inProd{c}{x}$ if $Ax = b$ and $+\\infty$ otherwise.\r\nLet  $f_2$ be the indicator function of $\\Re^{n_1}\\times (\\stdCone ^2)^*$ and $f_3$ be the indicator \r\nfunction of $(\\stdCone ^1)^*\\times \\Re^{n_2}$. Since there is  a primal feasible solution $x = (x_1,x_2)$ such \r\nthat $x_1 \\in \\reInt (\\stdCone ^1)^*$, we have that $\\fDom f_1 \\cap \\fDom f_2 \\cap \\reInt (\\fDom f_3)$ is \r\nnonempty. In addition, $f_1$ and $f_2$ are polyhedral functions. Let us now observe that:\r\n","start":116866,"end":117411}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"计算分解函数的共轭","source_units":[{"raw_tex":"\\begin{align*}\r\nf_1^*(s) = \\left\\{\r\n  \\begin{array}{l l}\r\n    \\inProd{b}{y}  & \\quad \\text{if there is $y$ with $s-c = \\stdMap^\\T y$}\\\\\r\n    +\\infty & \\quad \\text{otherwise}\r\n  \\end{array}\\right. \r\n\\end{align*}\r\nNote that, due to feasibility, for fixed $s$, $\\inProd{b}{y}$ does not depend on the choice \r\nof $y$, as long as $c+\\stdMap ^\\T y = s$. This is because since there is $x$ such that $Ax = b$, we have $\\inProd{b}{y} = \\inProd{x}{s-c}$.\r\nThe conjugate $f_2^*$ is the indicator function of $- \\{0\\}\\times \\stdCone ^2$ and \r\n$f_3^*$ is the indicator function of $- \\stdCone ^1\\times \\{0\\}$. \r\n","start":117411,"end":118011}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"应用共轭定理得到对偶表示","source_units":[{"raw_tex":"Applying Theorem \\ref{theo:conv} with $s = 0$, we have:\r\n\\begin{align*}\r\n(f_1+f_2+f_3)^*(0) & = \\inf \\left\\{ \\inProd{b}{y} \\mid  c+ \\stdMap ^\\T y = s_1, s_1 - (s_3,s_2) = 0,  s_2 \\in  \\stdCone ^2, s_3 \\in \\stdCone ^1      \\right\\} \\\\\r\n & = \\inf \\left\\{ \\inProd{b}{y} \\mid  c+ \\stdMap ^\\T y = s_1, s_1 \\in  \\stdCone ^1\\times \\stdCone ^2      \\right\\} \\\\\r\n& = - \\sup \\left\\{ \\inProd{b}{y} \\mid  c -\\stdMap ^\\T y = s_1,  s_1 \\in  \\stdCone ^1 \\times \\stdCone ^2      \\right\\},\r\n\\end{align*}\r\nwhere the $\\sup$ in the last equation is attained. So, there is some dual feasible $y$ such that \r\n$(f_1+f_2+f_3)^*(0) =  \\inProd{b}{y}$. However, using the definition of conjugate, we also have:\r\n","start":118011,"end":118696}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"按定义识别原始最优值","source_units":[{"raw_tex":"\\begin{align*}\r\n(f_1+f_2+f_3)^*(0) = - \\inf \\{ \\inProd{c}{x} \\mid Ax = b, x \\in \\stdCone ^1\\times \\stdCone ^2\\} = -\\theta _P.\r\n\\end{align*}\r\nIt follows that $\\pOpt = \\dOpt$ and the dual is attained at $y$. \r\n","start":118696,"end":118904}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"对偶 PPS 情形的略证","source_units":[{"raw_tex":"To prove $(ii)$, let $g_1 = f_1^*$, and let $g_2$ and $g_3$ be the indicator \r\nfunctions  of $\\Re^{n_1}\\times \\stdCone ^2$ and  $\\stdCone ^1 \\times \\Re^{n_2}$, respectively. Again, \r\nit is enough to compute $(g_1+g_2+g_3)^*(0)$ using both the definition of conjugate function and \r\nusing Theorem \\ref{theo:conv}. \r\n\\end{proof}","start":118904,"end":119230}]},{"id":"S077","semantic_type":"proposition","kind":"proposition","labels":[],"declared_external":false,"source_only":true,"title":"附录重复的强对偶命题","source_units":[{"raw_tex":"\\begin{proposition}\r\nLet $\\stdCone = \\stdCone ^1\\times \\stdCone ^2$, where  $\\stdCone ^1\\subseteq \\Re^{n_1},\\stdCone ^2 \\subseteq \\Re^{n_2}$ are closed convex cones such that $\\stdCone ^2$ is polyhedral.\r\n\\begin{enumerate}[label=({\\it\\roman*})]\r\n\\item If $\\pOpt$ is finite and \\eqref{eq:primal} satisfies the PPS condition, then $\\pOpt = \\dOpt$ and the dual optimal value is attained.\r\n\\item If $\\dOpt$ is finite and \\eqref{eq:dual} satisfies the PPS condition, then $\\pOpt = \\dOpt$ and the primal \r\noptimal value is attained.\r\n\\end{enumerate}\r\n\\end{proposition}\r\n","start":116302,"end":116866}]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明允许的标准有限维凸分析之相对内部运算；外推性质在原文第322行明确引用 Rockafellar Theorem 6.4，其余为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"相对内部的基本运算","source_units":[]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明允许的标准有限维凸分析：对偶锥恒等式与乘积面结构；此处为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"乘积锥及暴露截面恒等式","source_units":[]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明允许的标准有限维凸分析：最小面刻画；implicit standard fact made explicit by annotator。"],"source_only":false,"title":"最小面的内部点判据","source_units":[]},{"id":"S081","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明允许的弱锥对偶与初等线性代数；implicit standard fact made explicit by annotator。"],"source_only":false,"title":"弱对偶与配对计算","source_units":[]},{"id":"S082","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原许可清单中的标准有限维凸分析、对偶锥恒等式与初等线性代数；具体共轭公式为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"指标函数的共轭公式","source_units":[]},{"id":"S083","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"第二阶段的输入数据","source_units":[{"raw_tex":"{\\bf Input:} \\eqref{eq:dual}, the output of Phase 1: $\\stdFace$ and $s'$, with $\\stdFace \\neq \\emptyset$.\r\n","start":59220,"end":59327}]},{"id":"S084","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"当前面尚非最小面的分支","source_units":[{"raw_tex":"Suppose that the output face $\\stdFace$ of Phase 1 satisfies $\\stdFace \\neq \\minFaceD$. \r\n","start":62664,"end":62754}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S067","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S078","target_id":"S007","kind":"supports"},{"source_id":"S007","target_id":"S006","kind":"supports"},{"source_id":"S001","target_id":"S028","kind":"supports"},{"source_id":"S010","target_id":"S028","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S001","target_id":"S016","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S079","target_id":"S022","kind":"supports"},{"source_id":"S016","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S014","target_id":"S023","kind":"supports"},{"source_id":"S010","target_id":"S023","kind":"supports"},{"source_id":"S016","target_id":"S024","kind":"supports"},{"source_id":"S021","target_id":"S024","kind":"supports"},{"source_id":"S022","target_id":"S024","kind":"supports"},{"source_id":"S057","target_id":"S024","kind":"supports"},{"source_id":"S017","target_id":"S023","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S036","kind":"supports"},{"source_id":"S016","target_id":"S036","kind":"supports"},{"source_id":"S010","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S031","kind":"supports"},{"source_id":"S028","target_id":"S031","kind":"supports"},{"source_id":"S081","target_id":"S031","kind":"supports"},{"source_id":"S016","target_id":"S037","kind":"supports"},{"source_id":"S081","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S016","target_id":"S038","kind":"supports"},{"source_id":"S030","target_id":"S038","kind":"supports"},{"source_id":"S030","target_id":"S039","kind":"supports"},{"source_id":"S006","target_id":"S039","kind":"supports"},{"source_id":"S081","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S016","target_id":"S040","kind":"supports"},{"source_id":"S016","target_id":"S041","kind":"supports"},{"source_id":"S030","target_id":"S041","kind":"supports"},{"source_id":"S078","target_id":"S041","kind":"supports"},{"source_id":"S037","target_id":"S042","kind":"supports"},{"source_id":"S038","target_id":"S042","kind":"supports"},{"source_id":"S006","target_id":"S042","kind":"supports"},{"source_id":"S081","target_id":"S042","kind":"supports"},{"source_id":"S037","target_id":"S032","kind":"supports"},{"source_id":"S038","target_id":"S032","kind":"supports"},{"source_id":"S039","target_id":"S032","kind":"supports"},{"source_id":"S040","target_id":"S032","kind":"supports"},{"source_id":"S041","target_id":"S033","kind":"supports"},{"source_id":"S042","target_id":"S033","kind":"supports"},{"source_id":"S031","target_id":"S033","kind":"supports"},{"source_id":"S041","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S035","kind":"supports"},{"source_id":"S039","target_id":"S035","kind":"supports"},{"source_id":"S040","target_id":"S035","kind":"supports"},{"source_id":"S041","target_id":"S035","kind":"supports"},{"source_id":"S042","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S029","kind":"supports"},{"source_id":"S021","target_id":"S045","kind":"supports"},{"source_id":"S016","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S021","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S080","target_id":"S044","kind":"supports"},{"source_id":"S081","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S032","target_id":"S052","kind":"supports"},{"source_id":"S033","target_id":"S052","kind":"supports"},{"source_id":"S032","target_id":"S053","kind":"supports"},{"source_id":"S022","target_id":"S053","kind":"supports"},{"source_id":"S023","target_id":"S053","kind":"supports"},{"source_id":"S079","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S019","target_id":"S054","kind":"supports"},{"source_id":"S020","target_id":"S054","kind":"supports"},{"source_id":"S022","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S016","target_id":"S055","kind":"supports"},{"source_id":"S081","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S056","kind":"supports"},{"source_id":"S033","target_id":"S056","kind":"supports"},{"source_id":"S016","target_id":"S056","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S051","kind":"supports"},{"source_id":"S055","target_id":"S051","kind":"supports"},{"source_id":"S056","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S047","kind":"supports"},{"source_id":"S052","target_id":"S048","kind":"supports"},{"source_id":"S053","target_id":"S049","kind":"supports"},{"source_id":"S054","target_id":"S049","kind":"supports"},{"source_id":"S055","target_id":"S049","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S050","kind":"supports"},{"source_id":"S054","target_id":"S050","kind":"supports"},{"source_id":"S056","target_id":"S050","kind":"supports"},{"source_id":"S012","target_id":"S059","kind":"supports"},{"source_id":"S083","target_id":"S059","kind":"supports"},{"source_id":"S080","target_id":"S059","kind":"supports"},{"source_id":"S084","target_id":"S059","kind":"supports"},{"source_id":"S048","target_id":"S060","kind":"supports"},{"source_id":"S006","target_id":"S060","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S079","target_id":"S060","kind":"supports"},{"source_id":"S081","target_id":"S060","kind":"supports"},{"source_id":"S059","target_id":"S061","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S006","target_id":"S061","kind":"supports"},{"source_id":"S016","target_id":"S061","kind":"supports"},{"source_id":"S025","target_id":"S061","kind":"supports"},{"source_id":"S081","target_id":"S061","kind":"supports"},{"source_id":"S043","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S063","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S080","target_id":"S063","kind":"supports"},{"source_id":"S081","target_id":"S063","kind":"supports"},{"source_id":"S062","target_id":"S064","kind":"supports"},{"source_id":"S048","target_id":"S064","kind":"supports"},{"source_id":"S078","target_id":"S064","kind":"supports"},{"source_id":"S079","target_id":"S064","kind":"supports"},{"source_id":"S060","target_id":"S065","kind":"supports"},{"source_id":"S062","target_id":"S065","kind":"supports"},{"source_id":"S048","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S078","target_id":"S065","kind":"supports"},{"source_id":"S079","target_id":"S065","kind":"supports"},{"source_id":"S080","target_id":"S065","kind":"supports"},{"source_id":"S048","target_id":"S066","kind":"supports"},{"source_id":"S016","target_id":"S066","kind":"supports"},{"source_id":"S033","target_id":"S066","kind":"supports"},{"source_id":"S078","target_id":"S066","kind":"supports"},{"source_id":"S080","target_id":"S066","kind":"supports"},{"source_id":"S059","target_id":"S058","kind":"supports"},{"source_id":"S060","target_id":"S058","kind":"supports"},{"source_id":"S061","target_id":"S058","kind":"supports"},{"source_id":"S062","target_id":"S058","kind":"supports"},{"source_id":"S063","target_id":"S058","kind":"supports"},{"source_id":"S064","target_id":"S058","kind":"supports"},{"source_id":"S065","target_id":"S058","kind":"supports"},{"source_id":"S066","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S050","target_id":"S069","kind":"supports"},{"source_id":"S049","target_id":"S070","kind":"supports"},{"source_id":"S057","target_id":"S070","kind":"supports"},{"source_id":"S024","target_id":"S070","kind":"supports"},{"source_id":"S069","target_id":"S068","kind":"supports"},{"source_id":"S070","target_id":"S068","kind":"supports"},{"source_id":"S068","target_id":"S067","kind":"supports"},{"source_id":"S001","target_id":"S067","kind":"supports"},{"source_id":"S022","target_id":"S067","kind":"supports"},{"source_id":"S019","target_id":"S067","kind":"supports"},{"source_id":"S002","target_id":"S067","kind":"supports"},{"source_id":"S010","target_id":"S072","kind":"supports"},{"source_id":"S026","target_id":"S072","kind":"supports"},{"source_id":"S001","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S082","target_id":"S073","kind":"supports"},{"source_id":"S081","target_id":"S073","kind":"supports"},{"source_id":"S027","target_id":"S074","kind":"supports"},{"source_id":"S072","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S072","target_id":"S075","kind":"supports"},{"source_id":"S026","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S027","target_id":"S076","kind":"supports"},{"source_id":"S073","target_id":"S076","kind":"supports"},{"source_id":"S026","target_id":"S076","kind":"supports"},{"source_id":"S010","target_id":"S076","kind":"supports"},{"source_id":"S072","target_id":"S071","kind":"supports"},{"source_id":"S073","target_id":"S071","kind":"supports"},{"source_id":"S074","target_id":"S071","kind":"supports"},{"source_id":"S075","target_id":"S071","kind":"supports"},{"source_id":"S076","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S077","kind":"supports"},{"source_id":"S071","target_id":"S028","kind":"supports"},{"source_id":"S048","target_id":"S083","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S067","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S078","target_id":"S007","kind":"supports"},{"source_id":"S007","target_id":"S006","kind":"supports"},{"source_id":"S001","target_id":"S028","kind":"supports"},{"source_id":"S010","target_id":"S028","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S001","target_id":"S016","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S079","target_id":"S022","kind":"supports"},{"source_id":"S016","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S014","target_id":"S023","kind":"supports"},{"source_id":"S010","target_id":"S023","kind":"supports"},{"source_id":"S016","target_id":"S024","kind":"supports"},{"source_id":"S021","target_id":"S024","kind":"supports"},{"source_id":"S022","target_id":"S024","kind":"supports"},{"source_id":"S057","target_id":"S024","kind":"supports"},{"source_id":"S017","target_id":"S023","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S036","kind":"supports"},{"source_id":"S016","target_id":"S036","kind":"supports"},{"source_id":"S010","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S031","kind":"supports"},{"source_id":"S028","target_id":"S031","kind":"supports"},{"source_id":"S081","target_id":"S031","kind":"supports"},{"source_id":"S016","target_id":"S037","kind":"supports"},{"source_id":"S081","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S016","target_id":"S038","kind":"supports"},{"source_id":"S030","target_id":"S038","kind":"supports"},{"source_id":"S030","target_id":"S039","kind":"supports"},{"source_id":"S006","target_id":"S039","kind":"supports"},{"source_id":"S081","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S016","target_id":"S040","kind":"supports"},{"source_id":"S016","target_id":"S041","kind":"supports"},{"source_id":"S030","target_id":"S041","kind":"supports"},{"source_id":"S078","target_id":"S041","kind":"supports"},{"source_id":"S037","target_id":"S042","kind":"supports"},{"source_id":"S038","target_id":"S042","kind":"supports"},{"source_id":"S006","target_id":"S042","kind":"supports"},{"source_id":"S081","target_id":"S042","kind":"supports"},{"source_id":"S037","target_id":"S032","kind":"supports"},{"source_id":"S038","target_id":"S032","kind":"supports"},{"source_id":"S039","target_id":"S032","kind":"supports"},{"source_id":"S040","target_id":"S032","kind":"supports"},{"source_id":"S041","target_id":"S033","kind":"supports"},{"source_id":"S042","target_id":"S033","kind":"supports"},{"source_id":"S031","target_id":"S033","kind":"supports"},{"source_id":"S041","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S029","kind":"supports"},{"source_id":"S037","target_id":"S029","kind":"supports"},{"source_id":"S038","target_id":"S029","kind":"supports"},{"source_id":"S039","target_id":"S029","kind":"supports"},{"source_id":"S040","target_id":"S029","kind":"supports"},{"source_id":"S041","target_id":"S029","kind":"supports"},{"source_id":"S042","target_id":"S029","kind":"supports"},{"source_id":"S021","target_id":"S043","kind":"supports"},{"source_id":"S016","target_id":"S043","kind":"supports"},{"source_id":"S080","target_id":"S043","kind":"supports"},{"source_id":"S081","target_id":"S043","kind":"supports"},{"source_id":"S032","target_id":"S052","kind":"supports"},{"source_id":"S033","target_id":"S052","kind":"supports"},{"source_id":"S032","target_id":"S053","kind":"supports"},{"source_id":"S022","target_id":"S053","kind":"supports"},{"source_id":"S023","target_id":"S053","kind":"supports"},{"source_id":"S079","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S019","target_id":"S054","kind":"supports"},{"source_id":"S020","target_id":"S054","kind":"supports"},{"source_id":"S022","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S016","target_id":"S055","kind":"supports"},{"source_id":"S081","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S056","kind":"supports"},{"source_id":"S033","target_id":"S056","kind":"supports"},{"source_id":"S016","target_id":"S056","kind":"supports"},{"source_id":"S052","target_id":"S047","kind":"supports"},{"source_id":"S053","target_id":"S047","kind":"supports"},{"source_id":"S054","target_id":"S047","kind":"supports"},{"source_id":"S055","target_id":"S047","kind":"supports"},{"source_id":"S056","target_id":"S047","kind":"supports"},{"source_id":"S052","target_id":"S048","kind":"supports"},{"source_id":"S053","target_id":"S049","kind":"supports"},{"source_id":"S054","target_id":"S049","kind":"supports"},{"source_id":"S055","target_id":"S049","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S050","kind":"supports"},{"source_id":"S054","target_id":"S050","kind":"supports"},{"source_id":"S056","target_id":"S050","kind":"supports"},{"source_id":"S012","target_id":"S057","kind":"supports"},{"source_id":"S083","target_id":"S057","kind":"supports"},{"source_id":"S080","target_id":"S057","kind":"supports"},{"source_id":"S084","target_id":"S057","kind":"supports"},{"source_id":"S048","target_id":"S057","kind":"supports"},{"source_id":"S006","target_id":"S057","kind":"supports"},{"source_id":"S079","target_id":"S057","kind":"supports"},{"source_id":"S081","target_id":"S057","kind":"supports"},{"source_id":"S016","target_id":"S057","kind":"supports"},{"source_id":"S025","target_id":"S057","kind":"supports"},{"source_id":"S043","target_id":"S057","kind":"supports"},{"source_id":"S078","target_id":"S057","kind":"supports"},{"source_id":"S033","target_id":"S057","kind":"supports"},{"source_id":"S050","target_id":"S069","kind":"supports"},{"source_id":"S049","target_id":"S070","kind":"supports"},{"source_id":"S057","target_id":"S070","kind":"supports"},{"source_id":"S024","target_id":"S070","kind":"supports"},{"source_id":"S069","target_id":"S068","kind":"supports"},{"source_id":"S070","target_id":"S068","kind":"supports"},{"source_id":"S068","target_id":"S067","kind":"supports"},{"source_id":"S001","target_id":"S067","kind":"supports"},{"source_id":"S022","target_id":"S067","kind":"supports"},{"source_id":"S019","target_id":"S067","kind":"supports"},{"source_id":"S002","target_id":"S067","kind":"supports"},{"source_id":"S010","target_id":"S072","kind":"supports"},{"source_id":"S026","target_id":"S072","kind":"supports"},{"source_id":"S001","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S082","target_id":"S073","kind":"supports"},{"source_id":"S081","target_id":"S073","kind":"supports"},{"source_id":"S027","target_id":"S074","kind":"supports"},{"source_id":"S072","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S072","target_id":"S075","kind":"supports"},{"source_id":"S026","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S027","target_id":"S076","kind":"supports"},{"source_id":"S073","target_id":"S076","kind":"supports"},{"source_id":"S026","target_id":"S076","kind":"supports"},{"source_id":"S010","target_id":"S076","kind":"supports"},{"source_id":"S072","target_id":"S071","kind":"supports"},{"source_id":"S073","target_id":"S071","kind":"supports"},{"source_id":"S074","target_id":"S071","kind":"supports"},{"source_id":"S075","target_id":"S071","kind":"supports"},{"source_id":"S076","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S077","kind":"supports"},{"source_id":"S071","target_id":"S028","kind":"supports"},{"source_id":"S048","target_id":"S083","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:dual","eq:primal"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"原始与对偶锥规划","source_units":[{"raw_tex":"Consider the following pair of primal and dual conic linear programs (CLPs):\r\n\\vspace*{-3\\baselineskip} \r\n\\begin{multicols}{2}\r\n\\begin{align}\r\n\\underset{x}{\\inf} & \\quad \\inProd{c}{x} \\label{eq:primal}\\tag{P}\\\\ \r\n\\mbox{subject to} & \\quad \\stdMap x = b \\nonumber \\\\ \r\n&\\quad x \\in \\stdCone ^* \\nonumber \r\n\\end{align}\r\n\t\\break\r\n\t\\begin{align}\r\n\t\\underset{y}{\\sup} & \\quad \\inProd{b}{y} \\label{eq:dual} \\tag{D} \\\\ \r\n\t\\mbox{subject to} & \\quad c - \\stdMap ^\\T y \\in \\stdCone \\nonumber,\\\\ \\nonumber\r\n\t\\end{align}\t\r\n\\end{multicols}\r\n\\noindent where $\\stdCone \\subseteq \\Re^n$ is a closed convex cone and $\\stdCone^*$ is the dual cone $\\{s \\in \\Re^n \\mid \\inProd{s}{x} \\geq 0, \\forall x \\in \\stdCone \\}$. We have \r\nthat $\\stdMap: \\Re^n \\to \\Re^m$ is a linear map, $b \\in \\Re^m$, $c \\in \\Re^n$ and \r\n$\\stdMap^\\T$ denotes the adjoint map. We also have $\\stdMap^\\T y = \\sum _{i=1}^m \\stdMap _i y_i$, for certain \r\nelements $\\stdMap _i \\in \\Re^n$. The  inner product is denoted by $\\inProd{\\cdot}{\\cdot}$.\r\nWe will use $\\pOpt$ and $\\dOpt$ to denote the primal and  dual optimal value, respectively.\r\nIt is understood that $\\pOpt = +\\infty$ if \\eqref{eq:primal} is infeasible and \r\n$\\dOpt = -\\infty$ if \\eqref{eq:dual} is infeasible.\r\n","start":6343,"end":7567}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"可行松弛与最小面","source_units":[{"raw_tex":"In the absence of either a primal relative interior feasible solution or a dual \r\nrelative interior slack, it is possible that $\\pOpt \\neq \\dOpt$. A possible \r\nway of  correcting that is to let $\\minFaceD$ be \r\nthe minimal face of $\\stdCone$ which contains the feasible slacks $\\feasS = \\{c- \\stdMap ^\\T y \\in \\stdCone \\mid y \\in \\Re^m  \\}$, then \r\n","start":7573,"end":7922}]},{"id":"S004","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"相对内部及常见锥记号","source_units":[{"raw_tex":"For $C$ a closed convex set, we will denote by $\\reInt C$ and \r\n$\\closure C$ the relative interior and the closure of $C$, respectively.\r\nIf $\\mathcal{U}$ is an \r\narbitrary set, we denote by $\\mathcal{U}^\\perp$ the subspace which contains \r\nthe elements orthogonal to it. We will denote by $\\PSDcone{n}$ the cone of \r\n$n\\times n$ symmetric positive semidefinite matrices and by $\\SOC{n}$ the \r\nLorentz cone $\\{(x_0,\\overline{x}) \\in \\Re\\times \\Re^{n-1} \\mid x_0 \\geq \\norm{\\overline{x}}_2 \\}$, \r\nwhere $\\norm{.}_2$ is the usual Euclidean norm.\r\nThe nonnegative orthant will be denoted by $\\Re^n_+$. \r\n\r\n\r\n\r\n","start":16275,"end":16882}]},{"id":"S005","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S005"],"declared_external":true,"declaration_text":["原文第322行引用 Rockafellar, Convex Analysis, Theorem 14.6；属于声明允许的标准有限维凸分析背景。"],"source_only":false,"title":"线性部分与正交补","source_units":[{"raw_tex":"If $\\stdCone$ is a closed convex cone, we write $\\stdCone^*$ for its dual \r\ncone. We write $\\spanVec \\stdCone$ for its linear span and $\\lineality \\stdCone$ for \r\nits lineality space, which is $\\stdCone \\cap -\\stdCone$. $\\stdCone$ is said to be \\emph{pointed} if \r\n$\\lineality \\stdCone  = \\{0\\}$. We have $\\lineality (\\stdCone^*) = \\stdCone^\\perp$, see Theorem 14.6 in \\cite{rockafellar}. Also, if we select $\\stdInt \\in \\reInt \\stdCone$ and $x \\in \\stdCone^*$, then $x \\in \\stdCone^\\perp $ if and only if $\\inProd{\\stdInt}{x} = 0$\\footnote{Suppose $\\inProd{\\stdInt}{x} = 0$. Then, given $y \\in \\stdCone$ we have $\\alpha \\stdInt +(1-\\alpha)y \\in \\stdCone$ for some $\\alpha > 1$, due to Theorem 6.4 in \\cite{rockafellar}. Taking the inner product with $x$, we see that $\\inProd{y}{x}$ must be zero.}.\r\n","start":16882,"end":17683}]},{"id":"S006","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"内部点检测正交性","source_units":[{"raw_tex":"$\\lineality \\stdCone  = \\{0\\}$. We have $\\lineality (\\stdCone^*) = \\stdCone^\\perp$, see Theorem 14.6 in \\cite{rockafellar}. Also, if we select $\\stdInt \\in \\reInt \\stdCone$ and $x \\in \\stdCone^*$, then $x \\in \\stdCone^\\perp $ if and only if $\\inProd{\\stdInt}{x} = 0$\\footnote{Suppose $\\inProd{\\stdInt}{x} = 0$. Then, given $y \\in \\stdCone$ we have $\\alpha \\stdInt +(1-\\alpha)y \\in \\stdCone$ for some $\\alpha > 1$, due to Theorem 6.4 in \\cite{rockafellar}. Taking the inner product with $x$, we see that $\\inProd{y}{x}$ must be zero.}.\r\n","start":17147,"end":17683}]},{"id":"S007","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"正交判据的脚注证明","source_units":[{"raw_tex":"$\\lineality \\stdCone  = \\{0\\}$. We have $\\lineality (\\stdCone^*) = \\stdCone^\\perp$, see Theorem 14.6 in \\cite{rockafellar}. Also, if we select $\\stdInt \\in \\reInt \\stdCone$ and $x \\in \\stdCone^*$, then $x \\in \\stdCone^\\perp $ if and only if $\\inProd{\\stdInt}{x} = 0$\\footnote{Suppose $\\inProd{\\stdInt}{x} = 0$. Then, given $y \\in \\stdCone$ we have $\\alpha \\stdInt +(1-\\alpha)y \\in \\stdCone$ for some $\\alpha > 1$, due to Theorem 6.4 in \\cite{rockafellar}. Taking the inner product with $x$, we see that $\\inProd{y}{x}$ must be zero.}.\r\n","start":17147,"end":17683}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"四种可行性状态","source_units":[{"raw_tex":"\r\n\r\n\r\nThe conic linear program \\eqref{eq:dual} can be in four different feasibility \r\nstatuses: $i)$ strongly feasible if $\\feasS \\cap \\reInt \\stdCone \\neq \\emptyset$; $ii)$ weakly feasible if \r\n$\\feasS \\neq \\emptyset$ but $\\feasS \\cap \\reInt \\stdCone  = \\emptyset$; $iii)$ weakly infeasible if \r\n$\\feasS = \\emptyset$ but $\\text{dist}(c + \\matRange \\stdMap^\\T ,\\stdCone) = 0$; $iv)$ strongly \r\ninfeasible if $\\text{dist}(c + \\matRange \\stdMap^\\T ,\\stdCone) > 0$.  Note that \r\n\\eqref{eq:primal} admits analogous definitions. \r\n\r\n","start":17683,"end":18211}]},{"id":"S009","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S009"],"declared_external":true,"declaration_text":["原文回顾的标准有限维锥规划强对偶定理；属于声明允许的标准凸分析背景，未在此给出专门出处。"],"source_only":false,"title":"通常的强对偶定理","source_units":[{"raw_tex":"The \r\nstrong duality theorem states that if \\eqref{eq:dual} is strongly feasible and \r\n$\\dOpt < + \\infty$, then $\\pOpt = \\dOpt$ and $\\pOpt$ is attained. \r\nOn the other hand, if \\eqref{eq:primal} is strongly feasible and \r\n$\\pOpt > -\\infty$,  then $\\pOpt = \\dOpt$ and $\\dOpt$ is attained. \r\nWe  will also need a special version of the strong duality theorem. \r\nFirst, we need the following definition.\r\n\r\n","start":18211,"end":18615}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":["def:pps"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"部分多面体 Slater 条件","source_units":[{"raw_tex":"\\begin{definition}[Partial Polyhedral Slater's condition]\\label{def:pps}\r\n\tLet $\\stdCone = \\stdCone ^1\\times \\stdCone ^2$, where  $\\stdCone ^1\\subseteq \\Re^{n_1},\\stdCone ^2 \\subseteq \\Re^{n_2}$ are closed convex cones such that $\\stdCone ^2$ is polyhedral.\r\n\tWe say that \\eqref{eq:dual} satisfies the Partial Polyhedral Slater's (\\PPS) condition if there is \r\n\ta slack $(s_1,s_2) = c-\\stdMap^\\T y$, such that $s_1 \\in \\reInt \\stdCone^1$ and $s_2 \\in \\stdCone^2$.\r\n\tSimilarly, we say that \\eqref{eq:primal} satisfies the {\\PPS} condition, if there is a primal \r\n\tfeasible solution $x = (x_1,x_2)$ for which $x_1 \\in \\reInt (\\stdCone^1)^*$.\r\n\\end{definition}\r\n","start":18615,"end":19274}]},{"id":"S012","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S012"],"declared_external":true,"declaration_text":["声明允许的 Waki and Muramatsu, Lemma 3.2；原文第372行及1050行调用。"],"source_only":false,"title":"强可行失败的约化判据","source_units":[{"raw_tex":"Here, we will suppose that our main interest is in the dual problem \\eqref{eq:dual}.\r\n Facial Reduction hinges on the fact that \r\nstrong feasibility fails  if and only if there \r\nis $d \\in \\stdCone ^*$ such that $\\stdMap d = 0$  and\r\none of the two alternatives holds: $(i)$  $\\inProd{c}{d} = 0$ and $d \\not \\in \\stdCone ^\\perp$; or \r\n$(ii)$ $\\inProd{d}{c} < 0$, see Lemma 3.2 in \\cite{article_waki_muramatsu}. If \r\nalternative $(i)$ holds, $\\stdFace = \\stdCone \\cap \\{d\\}^\\perp$ is a proper face of $\\stdCone$ \r\ncontaining $\\feasS$.  We then substitute  $\\stdCone$ by $\\stdFace $ and repeat.\r\nIf  $(ii)$ holds, \\eqref{eq:dual} is infeasible. \r\n","start":20447,"end":21092}]},{"id":"S013","semantic_type":"algorithm","kind":"algorithm","labels":[],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"通用面约化算法","source_units":[{"raw_tex":" We write below a generic \r\nfacial reduction algorithm similar to the one described in \\cite{article_waki_muramatsu}.\r\n\r\n\\noindent\r\n[{\\bf Generic Facial Reduction}]\\label{alg:gen_fra}\r\n\r\n{\\bf Input:} \\eqref{eq:dual}\r\n\r\n{\\bf Output:} A set of reducing directions $\\{d_1, \\ldots, d_\\ell\\}$ and $\\minFaceD$.\r\n\\begin{enumerate}\r\n\\item $\\stdFace _1 \\leftarrow \\stdCone, i \\leftarrow 1 $\r\n\\item Let $d_i$ be an element in $\\stdFace _i ^* \\cap \\ker \\stdMap$ such that either: \r\n$i)$ $d_i \\not \\in \\stdFace _i^\\perp$ and $\\inProd{c}{d_i} = 0$; or $ii)$ $\\inProd{c}{d_i} < 0$.\r\nIf no such $d_i$ exists, let $\\minFaceD \\leftarrow \\stdFace _i$ and stop. \r\n\\item If $\\inProd{c}{d_i} < 0$, let $\\minFaceD \\leftarrow  \\emptyset$ and stop.\r\n\\item If $\\inProd{c}{d_i} = 0$, let $\\stdFace _{i+1}  \\leftarrow \\stdFace _i  \\cap \\{d_i\\}^\\perp, i\\leftarrow i+1$ and return to 2).\r\n\\end{enumerate}\r\n","start":21092,"end":21969}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"约化方向的定义","source_units":[{"raw_tex":"\r\n\r\nWe will refer to the directions satisfying $d_i \\in \\stdFace _i^* \\cap \\ker \\stdMap$ and \r\n$\\inProd{c}{d_i} \\leq 0$ as \\emph{reducing directions}, so that the $d_i$ in Step 2. are indeed reducing directions. An important issue when doing facial reduction is how to model the \r\n","start":21969,"end":22250}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":["eq:red","eq:red:1","eq:red:2","eq:red:3","eq:red_dual","eq:red_dual:1","eq:red_dual:2","eq:red_dual:3"],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"约化辅助问题对","source_units":[{"raw_tex":"In this article, we will search for reducing directions by\r\nconsidering the pair \\eqref{eq:red} and \\eqref{eq:red_dual} introduced in \\cite{lourenco_muramatsu_tsuchiya3}, which \r\nare parametrized by $\\stdMap$, $c$, $\\stdCone, \\stdInt, \\stdInt^*$. In Phase 1 of FRA-Poly, we will always \r\nselect $\\stdInt$ and $\\stdInt^*$ according to Lemma \\ref{lemma:red2}. Different choices will be discussed/used \r\nin Phase 2 of FRA-Poly and on Sections \\ref{sec:compl} and \\ref{sec:tight}.\r\n\\begin{align}\r\n\\underset{x,t,w}{\\mbox{minimize}} & \\quad t & \\tag{$P_\\stdCone $}\\label{eq:red}  \\\\ \r\n\\mbox{subject to} & \\quad -\\inProd{c}{x  -t\\stdInt^*} + t - w &= 0 \\label{eq:red:1}\\\\\r\n&\\quad \\inProd{\\stdInt}{x} + w &= 1 \\label{eq:red:2}\\\\\r\n& \\quad\\stdMap x  -t\\stdMap \\stdInt^* & = 0 \\label{eq:red:3}\\\\\r\n&\\quad (x,t,w) \\in \\stdCone ^* \\times \\Re_+ \\times \\Re_+ \\nonumber\\\\\r\n\\underset{y_1,y_2,y_3}{\\mbox{maximize}} &\\quad  y_2 & \\tag{$D_\\stdCone $}\\label{eq:red_dual}  \\\\ \r\n\\mbox{subject to} & \\quad cy_1 -\\stdInt y_2 -\\stdMap^\\T y_3 \\in \\stdCone \\label{eq:red_dual:1} \\\\\r\n&\\quad 1-y_1(1+\\inProd{c}{\\stdInt^*}) + \\inProd{\\stdInt^* }{\\stdMap ^\\T y_3}\\geq 0 \\label{eq:red_dual:2} \\\\\r\n& \\quad y_1-y_2 \\geq 0 \\label{eq:red_dual:3} \\\\\r\n&\\quad (y_1,y_2,y_3) \\in \\Re \\times \\Re \\times \\Re^m. \\nonumber\r\n\\end{align}\r\n\r\nRecall that our goal is to find a point $x \\in \\ker \\stdMap \\cap \\stdCone$ satisfying  $\\inProd{c}{x} \\leq 0$. The idea behind  \\eqref{eq:red} is to shift the problem by $-t\\stdInt^*$ (Equations \\eqref{eq:red:1} and \\eqref{eq:red:3}) and add constraints to ensure \r\nthe $x$ stays in a bounded region (Equation \\eqref{eq:red:2}). These changes ensure that \\eqref{eq:red} and \\eqref{eq:red_dual} \r\nsatisfy the PPS condition when the parameters $\\stdInt, \\stdInt^*$ are chosen appropriately as in\r\nthe next section.\r\n","start":23456,"end":25262}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"算法实施能力假设","source_units":[{"raw_tex":"We now present a disclaimer of sorts.\r\nThe theoretical results presented in this section and the next stand whether FRA-Poly is doable or \r\nnot for a given $\\stdCone$. \r\nIf we wish to do facial reduction concretely (even if it is by hand!), we need to \r\nmake a few assumptions on our computational capabilities \r\nand on our knowledge on the lattice of faces of $\\stdCone$. \r\nFirst of all, we must be able to solve problems over faces of $\\stdCone$ such that both the primal and the dual \r\nsatisfy the {\\PPS} condition and we must also be able to do \r\nbasic linear algebraic operations.  Also, for \r\neach face $\\stdFace$ of $\\stdCone$ we must know:\r\n\\begin{enumerate}\r\n\t\\item $\\spanVec \\stdFace$, \r\n\t\\item at least one point $\\stdInt \\in \\reInt \\stdFace$,\r\n\t\\item at least one point $\\stdInt^* \\in \\reInt \\stdFace^*$,\r\n\t\\item whether $\\stdFace$ is polyhedral or not.\r\n\\end{enumerate}\r\n","start":41456,"end":42340}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"到多面体性的距离","source_units":[{"raw_tex":"\\subsection{Distance to Polyhedrality}\r\nHere we introduce the notion of \\emph{distance to polyhedrality}. In what follows, \r\nif we have a chain of faces $\\stdFace _1 \\subsetneq \\ldots \\subsetneq \\stdFace _\\ell$, the \r\nlength of the chain is defined to be $\\ell$.\r\n\\begin{definition}\r\n\tLet $\\stdCone$ be a nonempty closed convex cone. The \\emph{distance to polyhedrality} $\\distP(\\stdCone)$ is the \r\n\tlength \\emph{minus one} of the longest strictly ascending chain of nonempty faces $\\stdFace _1 \\subsetneq \\ldots \\subsetneq \\stdFace _{\\ell} $ which satisfies:\r\n\t\\begin{enumerate}\r\n\t\t\\item  $\\stdFace _1$ is polyhedral;  \r\n\t\t\\item $\\stdFace _j$ is not polyhedral for $j > 1$.\r\n\t\\end{enumerate}\r\n\\end{definition}\r\n","start":43057,"end":43769}]},{"id":"S020","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"距离参数的约化含义","source_units":[{"raw_tex":"\r\nThe distance to polyhedrality is a well-defined concept, because the lineality space\r\nof $\\stdCone$ is always a polyhedral face of $\\stdCone$.\r\nMoreover, $\\distP(\\stdCone)$ counts the maximum number of facial reduction steps that \r\ncan be taken before we reach a polyhedral face.\r\n","start":43769,"end":44052}]},{"id":"S021","semantic_type":"definition","kind":"definition","labels":["eq:strict1","eq:strict2","eq:strict3"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"对偶严格互补条件","source_units":[{"raw_tex":"For general $\\stdCone$, these equivalencies may not hold and we might need \r\nto distinguish between primal and dual strict complementarity, see \r\nfor instance, Definition 3.4 and Remark 3.6 in the chapter by Pataki \\cite{pataki_handbook} and Equation (2.6) in Section 2 of the work by Tun\\c{c}el and Wolkowicz \\cite{TW12}. \r\nBased on those references, we will say \r\nthat \\eqref{eq:red} and \\eqref{eq:red_dual}  satisfies  \r\n\\emph{(dual) strict complementarity} if $\\opt{\\text{\\ref{eq:red}}} = \\opt{\\text{\\ref{eq:red_dual}}}$\r\nand there are optimal solutions $(x^*, t^*,w^*), (y_1^*,y_2^*,y_3^*)$ such that\r\n \\begin{align}\r\n cy_1^* -\\stdInt y_2^* -\\stdMap^\\T y_3^* &\\in \\reInt (\\stdCone \\cap\\{ x^*\\}^\\perp)  \\label{eq:strict1}\\\\\r\n t^* +  1-y_1^*(1+\\inProd{c}{\\stdInt^*}) + \\inProd{\\stdInt^* }{\\stdMap ^\\T y_3^*}&> 0  \\label{eq:strict2}\\\\\r\n w^* + y_1^*-y_2^* & > 0. \\label{eq:strict3} \r\n \\end{align}\r\n","start":45905,"end":46804}]},{"id":"S022","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"乘积锥及其分块面","source_units":[{"raw_tex":"Henceforth, we will assume that $\\stdCone$ is the product of $r$ cones and we will \r\nwrite $\\stdCone = \\stdCone ^1 \\times \\ldots \\times \\stdCone  ^r $. Then, recall that \r\nif $\\stdFace$ is face of $\\stdCone$, we can write $\\stdFace = \\stdFace ^1 \\times \\ldots \\times \\stdFace  ^r$, where $\\stdFace ^i$ is a face of $\\stdCone ^i$ for every $i$.\r\n\r\nConsider the following \r\nFRA variant, which we call FRA-Poly.\r\n","start":49239,"end":49649}]},{"id":"S023","semantic_type":"algorithm","kind":"algorithm","labels":[],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"FRA-Poly 第一阶段","source_units":[{"raw_tex":"\r\n\\noindent\r\n[{\\bf Facial Reduction Poly - Phase 1}]\r\n\r\n{\\bf Input:} \\eqref{eq:dual}\r\n\r\n{\\bf Output:} A set of reducing directions $\\{d_1, \\ldots, d_\\ell\\}$. If \r\n\\eqref{eq:dual} is feasible, it outputs  some face $\\stdFace \\subseteq \\stdCone$ \r\nfor which the PPS condition holds, together \r\nwith a dual slack $s'$ for which $s_j' \\in \\reInt \\stdFace ^j $ for every $j$ such that $\\stdFace ^j$  is nonpolyhedral.\r\nIf \\eqref{eq:dual} is infeasible, the directions form a certificate of infeasibility.\r\n\\begin{enumerate}\r\n\\item $\\stdFace _1 \\leftarrow \\stdCone, i \\leftarrow 1 $\r\n\\item Let  $(x^*,t^*,w^*)$ and $(y_1^*,y_2^*,y_3^*)$ be any pair of optimal solutions to \r\n\\eqref{eq:red} and \\eqref{eq:red_dual} with\r\n\\begin{itemize}\r\n\t\\item $\\stdFace _i$ in place of $\\stdCone $, where $\\stdFace_i = \\stdFace_i ^1 \\times \\ldots \\times \\stdFace_i  ^r$,\r\n\t\\item any $\\stdInt^* \\in \\reInt \\stdFace _i^*$,\r\n\t\\item any $\\stdInt$  such that $\\stdInt _j = 0$ if $\\stdFace _i^j$ is polyhedral and \t$\\stdInt _j \\in \\reInt \\stdFace _i^j$, otherwise.\r\n\\end{itemize}\r\n\r\n\\item If $t^* = 0$ and  $\\inProd{c}{x^*} < 0$, let $\\minFaceD \\leftarrow \\emptyset$ and stop. \\eqref{eq:dual}  is infeasible.\r\n\\item If $t^* = 0$ and $\\inProd{c}{x^*} = 0$, let $d_i \\leftarrow x^*, \\stdFace _{i+1}  \\leftarrow \\stdFace _i  \\cap \\{d_i\\}^\\perp, i\\leftarrow i+1$ and return to 2).\r\n\\item If $t^* > 0$, $s' \\leftarrow  c  -\\stdMap^\\T  \\frac{y_3^*}{y_1^*}$, $\\stdFace \\leftarrow \\stdFace _i$ and stop. {\\PPS} condition is satisfied.\r\n\\end{enumerate}\r\n","start":49649,"end":51166}]},{"id":"S024","semantic_type":"algorithm","kind":"algorithm","labels":[],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"FRA-Poly 第二阶段","source_units":[{"raw_tex":"[{\\bf Facial Reduction Poly - Phase 2}]\r\n\r\n{\\bf Input:} \\eqref{eq:dual}, the output of Phase 1: $\\stdFace$ and $s'$, with $\\stdFace \\neq \\emptyset$.\r\n\r\n{\\bf Output:} $\\minFaceD$, a dual feasible slack $\\hat s \\in \\reInt \\minFaceD$ and, perhaps, \r\nan extra reducing direction $d$.\r\n\\begin{enumerate}\r\n\\item Let $\\hat \\stdCone = {\\hat \\stdCone} ^1\\times \\ldots \\times {\\hat \\stdCone} ^r$ such that ${\\hat \\stdCone} ^j = \\stdFace ^j$ if \r\n$\\stdFace ^j$ is polyhedral and $\\hat \\stdCone ^j = \\spanVec \\stdFace^j$ otherwise. \r\n\\item Let $(x^*,t^*,w^*)$ and $(y_1^*,y_2^*,y_3^*)$ be\r\nany pair of \\emph{strictly complementary} optimal solutions to \\eqref{eq:red} and \\eqref{eq:red_dual} with\r\n\\begin{itemize}\r\n\t\\item $\\hat \\stdCone$ in place of $\\stdCone$,\r\n\t\\item any $\\stdInt \\in \\reInt \\hat \\stdCone$,\r\n\t\\item any $\\stdInt^* \\in \\reInt {\\hat \\stdCone}^* $.\r\n\\end{itemize}\r\n\\item If $t^* = 0$, let $d \\leftarrow x^*$, $\\minFaceD \\leftarrow \\stdFace \\cap \\{x^*\\}^\\perp$.\r\nLet $\\tilde s$ be $c  -\\stdMap^\\T  \\frac{y_3^*}{y_1^*} $. Then, we let \r\n$\\hat s$ be a convex combination of $\\tilde s$ and $s'$ such that $\\hat s \\in \\reInt \\minFaceD$ and stop.\\footnote{\\label{foot}In Proposition \\ref{prop:min_face} it is shown that if $z_\\beta = \\beta s' + (1-\\beta)\\tilde s $ is such that $\\beta \\in (0,1)$ and $\\beta$ is sufficiently close to $1$ then $z_\\beta \\in  \\reInt \\minFaceD$. Therefore, after determining $\\minFaceD$, $\\hat s$ can be found through a simple search procedure, e.g., the bisection method.} %by starting with $\\beta = 0.5$ and letting $\\beta$ be $0.75,0.875,0.9375, \\ldots$ and so on, until $z_\\beta \\in \\reInt \\minFaceD$.  }\r\n\\item If $t^* > 0$,  $\\minFaceD \\leftarrow \\stdFace$.\r\nLet $\\tilde s$ be $c  -\\stdMap^\\T  \\frac{y_3^*}{y_1^*} $. Then, we let \r\n$\\hat s$ be a convex combination of $\\tilde s$ and $s'$ such that $\\hat s \\in \\reInt \\minFaceD$ and stop.\\footref{foot}%\\footnote{See footnote \\ref{foot}.}\r\n\r\n\\end{enumerate}\r\n","start":59177,"end":61118}]},{"id":"S025","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S025"],"declared_external":true,"declaration_text":["声明允许的 Goldman–Tucker 严格互补定理；原文同时引用 McLinden82、Akgul84，计算讨论引用 FRT85、MY93。"],"source_only":false,"title":"多面体问题的严格互补","source_units":[{"raw_tex":"\r\n\r\nNote that in Phase 2, the cone $\\hat{\\stdCone}$ is polyhedral, therefore, \r\nboth \\eqref{eq:red_dual} and \\eqref{eq:red} are \r\npolyhedral problems. Therefore,  strictly complementary solutions are ensured to exist, which is a \r\nconsequence of Goldman-Tucker Theorem and also follows from the results of \r\nMcLinden \\cite{McLinden82} and Akg\\\"ul \\cite{Akgul84}. \r\nWe also remark that a strictly complementary solution of a polyhedral problem can be found by solving \r\na single linear program, see, for instance, the article by Freund, Roundy and Todd \\cite{FRT85} and the related \r\nwork by Mehrotra and Ye \\cite{MY93}. \r\n","start":61118,"end":61740}]},{"id":"S026","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"凸共轭及多面体函数","source_units":[{"raw_tex":"Let $f:\\Re^n \\to \\Re\\cup \\{-\\infty,+\\infty \\}$ be a convex function.\r\nWe denote the domain of $f$ by $\\fDom f = \\{x \\in \\Re^n \\mid f(x) < \\infty \\}$. If $\\fDom f \\neq \\emptyset $ and $f$ is never $-\\infty$, then $f$ is said to be proper.\r\nIts conjugate will be denoted by $f^*$ and it satisfies $f^*(s) = \\sup _{x} \\inProd{s}{x} - f(x)$.\r\nIf the epigraph of $f$ is a polyhedral set, then $f$ is said to be a polyhedral function. \r\nWe recall Theorem 20.1 from \\cite{rockafellar}.\r\n\r\n","start":115268,"end":115750}]},{"id":"S027","semantic_type":"external_fact","kind":"external_fact","labels":["theo:conv"],"member_ids":["S027"],"declared_external":true,"declaration_text":["明确许可的 Rockafellar, Convex Analysis, Theorem 20.1，原文重述为 B0093。"],"source_only":false,"title":"部分多面体共轭定理","source_units":[{"raw_tex":"\\begin{theorem}[Rockafellar]\\label{theo:conv}\r\nLet $f_1, \\ldots, f_m$ be proper convex functions and let $f_{k+1}, \\ldots ,$ $f_m$ be \r\npolyhedral functions. Suppose also that\r\n$$\r\n\\reInt (\\fDom f_1) \\cap \\cdots \\cap \\reInt (\\fDom f_k) \\cap \\fDom f_{k+1} \\cap \\cdots \\cap \\fDom f_m \\neq \\emptyset.\r\n$$\r\nThen the following holds:\r\n\\begin{equation*}\r\n(f_1 + \\cdots + f_m)^*(s) = \\inf \\{ f_1^*(s_1) + \\cdots + f_m^*(s_m) \\mid s_1 + \\cdots + s_m = s \\},\r\n\\end{equation*}\r\nwhere for each $s$ the infimum is attained whenever it is finite.\r\n\\end{theorem}\r\n","start":115750,"end":116300}]},{"id":"S028","semantic_type":"proposition","kind":"proposition","labels":["prop:slater2"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"PPS 强对偶命题","source_units":[{"raw_tex":"\\begin{proposition}[{\\PPS}-Strong Duality]\\label{prop:slater2}\r\n\tLet $\\stdCone = \\stdCone ^1\\times \\stdCone ^2$, where  $\\stdCone ^1\\subseteq \\Re^{n_1},\\stdCone ^2 \\subseteq \\Re^{n_2}$ are closed convex cones such that $\\stdCone ^2$ is polyhedral.\r\n\t\\begin{enumerate}[label=(\\roman*)]\r\n\t\t\\item If $\\pOpt$ is finite and \\eqref{eq:primal} satisfies the {\\PPS} condition, then $\\pOpt = \\dOpt$ and the dual optimal value is attained.\r\n\t\t\\item If $\\dOpt$ is finite and \\eqref{eq:dual} satisfies the {\\PPS} condition, then $\\pOpt = \\dOpt$ and the primal \r\n\t\toptimal value is attained.\r\n\t\\end{enumerate}\r\n\\end{proposition}","start":19451,"end":20066}]},{"id":"S029","semantic_type":"lemma","kind":"lemma","labels":["lemma:red2"],"member_ids":["S029","S035"],"declared_external":false,"source_only":false,"title":"部分多面体辅助问题引理","source_units":[{"raw_tex":"\\begin{lemma}\\label{lemma:red2}\r\nLet $\\stdCone = \\stdCone ^1 \\times \\stdCone ^2$ be a \r\nclosed convex cone, such that $\\stdCone ^2$ is polyhedral.\r\nConsider the pair \\eqref{eq:red} and \\eqref{eq:red_dual} with \r\n$\\stdInt$ and $\\stdInt^*$ such that  $\\stdInt = (\\stdInt_1,0) \\in (\\reInt \\stdCone ^1)\\times \\{0\\}$ and \r\n$\\stdInt^* \\in \\reInt \\stdCone^*$.  The following properties hold.\r\n\\begin{enumerate}[label=(\\roman*)]\r\n\\item The following are solutions to \\eqref{eq:red} and \\eqref{eq:red_dual} that satisfy the {\\PPS} condition: \r\n\\begin{align*}\r\n(x,t,w)& = \\left(\\frac{\\stdInt^*}{\\inProd{\\stdInt}{\\stdInt^*}+1},\\frac{1}{\\inProd{\\stdInt}{\\stdInt^*}+1}, \\frac{1}{\\inProd{\\stdInt}{\\stdInt^*}+1}\\right) \\\\\r\n (y_1,y_2,y_3) & = (0,-1,0).\r\n \\end{align*} \r\nIn particular, \r\n$\\opt{\\text{\\ref{eq:red}}} =  \\opt{\\text{\\ref{eq:red_dual}}}$.\r\n\\end{enumerate}\r\nLet $(x^*,t^*,w^*)$ be any primal optimal solution and \r\n$(y_1^*,y_2^*,y_3^*)$ be any dual optimal solution.\r\n \\begin{enumerate}[label=(\\roman*)]\r\n \\setcounter{enumi}{1} \r\n\\item $\\opt{\\text{\\ref{eq:red}}} =  \\opt{\\text{\\ref{eq:red_dual}}} = 0$ if and only if \r\none of the two alternatives below holds:\r\n\\begin{enumerate}\r\n\\item $\\inProd{c}{x^*} < 0 $ and $\\feasS = \\minFaceD = \\emptyset$, or\r\n\\item $\\inProd{c}{x^*} = 0 $, $\\feasS \\subseteq  \\stdCone \\cap \\{x^*\\}^\\perp \\subsetneq \\stdCone$ and \r\n$x^*_1 \\not \\in (\\stdCone^1)^\\perp = \\lineality( (\\stdCone^1 )^*)$.\r\n\\end{enumerate}\r\n\r\n\\item $\\opt{\\text{\\ref{eq:red}}} =  \\opt{\\text{\\ref{eq:red_dual}}} >  0$ if and only if the {\\PPS} condition is satisfied for \\eqref{eq:dual}. In this case, we have\r\n$\r\nc  -\\stdMap^\\T \\frac{y_3^*}{y_1^*} \\in (\\reInt \\stdCone ^1)\\times \\{0\\}. \r\n$ (In particular, it is feasible for \\eqref{eq:dual})\r\n\r\n\\end{enumerate}\r\n\\end{lemma}","start":28060,"end":29826},{"raw_tex":"\\begin{proof}\r\n\r\n\\begin{enumerate}[label=({\\it{\\roman*}})]\r\n\\item  Due to choice of $\\stdInt$ and $\\stdInt^*$, it is clear that the solutions \r\nmeet the PPS condition.\r\n\r\n\r\n\\item \\fbox{$\\opt{\\text{\\ref{eq:red}}} = 0 \\Rightarrow$  (a) or (b) holds.}\r\nSuppose that $\\opt{\\text{\\ref{eq:red}}} = 0$ and let $(x^*,0,w^*)$ be an optimal solution for \\eqref{eq:red}.\r\nDue to the constraints in \\eqref{eq:red}, we  have\r\n$$\\stdMap x^* = 0, \\qquad -\\inProd{c}{x^*} = w^* \\geq 0, \\qquad x^* \\in \\stdCone ^*.$$ \r\nNote that if $\\inProd{c}{x^*} < 0$, then $\\minFaceD = \\feasS = \\emptyset$. This is \r\nalternative (a).\r\n\r\nIf $\\inProd{c}{x^*} = 0$, then (\\eqref{eq:red:1}) implies \r\n$w^* = 0$. Since $\\stdMap x^* = 0$, we obtain  \r\n$$\\feasS \\subseteq  \\stdCone \\cap \\{x^*\\}^\\perp. $$\r\nBy \\eqref{eq:red:2}, we have $\\inProd{\\stdInt}{x^*} = 1$, so that $x^* = (x_1^*,x_2^*)$ satisfies $x_1^* \\not \\in (\\stdCone^1)^\\perp$, due to the choice of $\\stdInt$. \r\nTherefore, the inclusion $\\stdCone \\cap \\{x^*\\}^\\perp \\subseteq \\stdCone$ is strict, \r\nthat is,  $\\stdCone \\cap \\{x^*\\}^\\perp \\subsetneq \\stdCone$. This is alternative (b).\r\n\r\n\\fbox{(a) or (b) holds $\\Rightarrow \\opt{\\text{\\ref{eq:red}}} = 0$.} \r\nFirst recall that  $t$ is constrained to be nonnegative, therefore  $\\opt{\\text{\\ref{eq:red}}} \\geq 0$.\r\nNext, we will prove the following inequality\r\n\\begin{equation}\\label{eq:lemma:aux}\r\n\t{\\inProd{\\stdInt}{x^*} - \\inProd{c}{x^*}} > 0.\r\n\\end{equation}\r\nSince $\\stdInt \\in \\stdCone,x^* \\in \\stdCone^*$, we have \r\n$\\inProd{\\stdInt}{x^*} \\geq 0$. If $(a)$ holds, then $- \\inProd{c}{x^*}> 0$.\r\nIn particular, \\eqref{eq:lemma:aux} holds.\r\nNow, we consider the case where (b) holds. \r\nIn this case, we have $\\inProd{c}{x^*} = 0$. \r\nIf $\\inProd{\\stdInt}{x^*}$ is also zero, then $\\inProd{\\stdInt_1}{x_1^*}$ is zero and since $\\stdInt_1 \\in \\reInt \\stdCone^1$, $ x^*_1$ must \r\nbelong to $(\\stdCone^1)^\\perp$. However, this is not allowed \r\nsince we assumed that $x_1^* \\not \\in (\\stdCone^1)^\\perp$.\r\nTherefore, we have $\\inProd{\\stdInt}{x^*} > 0$ and, in particular,  \\eqref{eq:lemma:aux} also holds.\r\n\r\nLet  $$\r\n\\alpha = \\frac{1}{\\inProd{\\stdInt}{x^*} - \\inProd{c}{x^*}}.\r\n$$\r\nThen $\\alpha > 0$ and \r\n$(\\alpha x^*,0,-\\alpha \\inProd{c}{x^*})$ is a solution to \\eqref{eq:red} with value $0$, \r\nso that $\\opt{\\text{\\ref{eq:red}}} = 0$.\r\n\r\n\\item \r\n\\fbox{ $\\opt{\\text{\\ref{eq:red_dual}}} > 0 \\Rightarrow $ PPS holds for \\eqref{eq:dual}}\r\nFrom \\eqref{eq:red_dual:1}, we have\r\n$$\r\nc y_1^* - \\stdInt y_2^* - {\\stdMap}^\\T y_3^* \\in \\stdCone.\r\n$$\r\nSince $y_2^*=\\opt{\\text{\\ref{eq:red_dual}}}\r\n>0$, we have, due to \\eqref{eq:red_dual:3}, $y_1^* > 0$.\r\n Hence, \r\n $$c - \\stdInt \\frac{y_2^*}{y_1^*} - {\\stdMap}^\\T \\frac{y_3^*}{y_1^*} \\in \\stdCone.$$ \r\nWe conclude that \r\n$c - {\\stdMap}^\\T y_3^*/y_1^* \\in (\\stdCone^1 +y_2^*/y_1^* \\stdInt_1)\\times  \\{0\\} \\subseteq (\\reInt \\stdCone ^1)\\times \\{0\\}$, due to the choice of  $\\stdInt$ and the fact that $y_2^*/y_1^* > 0$. \r\n\r\n\\fbox{ $\\opt{\\text{\\ref{eq:red_dual}}} = 0 \\Rightarrow $ PPS does not hold for \\eqref{eq:dual}  } \r\nIf $t^* = 0$ and $(x^*,0,w^*)$ is an optimal solution \r\nfor \\eqref{eq:red}, then either $(a)$ or $(b)$ of item $(ii)$ is satisfied. \r\nIf $(a)$ is satisfied, then \\eqref{eq:dual} is infeasible and thus the PPS condition\r\ncannot hold. \r\nIf $(b)$ is satisfied, then $\\inProd{c}{x^*} = 0$,\r\n$\\stdMap x^* = 0$ and $x_1^* \\not \\in (\\stdCone ^1)^\\perp$. If \r\n$(s_1,s_2)$ is a feasible slack for \\eqref{eq:dual}, we have $\\inProd{s_1}{x_1^*} + \\inProd{s_2}{x_2^*} = 0$, so that \r\n$\\inProd{s_1}{x_1^*} = 0$.  \r\nAs $x_1^* \\not \\in (\\stdCone ^1)^\\perp$, we have that $s_1 \\not \\in \\reInt \\stdCone^1$. This means\r\nthat there is no feasible solution for \\eqref{eq:dual} satisfying the PPS\r\ncondition.\r\n\\end{enumerate}\r\n\\end{proof}","start":31151,"end":34906}]},{"id":"S030","semantic_type":"assumption","kind":"assumption","labels":["lemma:red2"],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"辅助引理的参数条件","source_units":[{"raw_tex":"\\begin{lemma}\\label{lemma:red2}\r\nLet $\\stdCone = \\stdCone ^1 \\times \\stdCone ^2$ be a \r\nclosed convex cone, such that $\\stdCone ^2$ is polyhedral.\r\nConsider the pair \\eqref{eq:red} and \\eqref{eq:red_dual} with \r\n$\\stdInt$ and $\\stdInt^*$ such that  $\\stdInt = (\\stdInt_1,0) \\in (\\reInt \\stdCone ^1)\\times \\{0\\}$ and \r\n$\\stdInt^* \\in \\reInt \\stdCone^*$.  The following properties hold.\r\n","start":28060,"end":28446}]},{"id":"S031","semantic_type":"lemma","kind":"lemma","labels":["lemma:red2"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"辅助问题的 PPS 可行点","source_units":[{"raw_tex":"\\begin{enumerate}[label=(\\roman*)]\r\n\\item The following are solutions to \\eqref{eq:red} and \\eqref{eq:red_dual} that satisfy the {\\PPS} condition: \r\n\\begin{align*}\r\n(x,t,w)& = \\left(\\frac{\\stdInt^*}{\\inProd{\\stdInt}{\\stdInt^*}+1},\\frac{1}{\\inProd{\\stdInt}{\\stdInt^*}+1}, \\frac{1}{\\inProd{\\stdInt}{\\stdInt^*}+1}\\right) \\\\\r\n (y_1,y_2,y_3) & = (0,-1,0).\r\n \\end{align*} \r\nIn particular, \r\n$\\opt{\\text{\\ref{eq:red}}} =  \\opt{\\text{\\ref{eq:red_dual}}}$.\r\n\\end{enumerate}\r\n","start":28446,"end":28912}]},{"id":"S032","semantic_type":"lemma","kind":"lemma","labels":["lemma:red2"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"零值与有效约化分支","source_units":[{"raw_tex":"Let $(x^*,t^*,w^*)$ be any primal optimal solution and \r\n$(y_1^*,y_2^*,y_3^*)$ be any dual optimal solution.\r\n \\begin{enumerate}[label=(\\roman*)]\r\n \\setcounter{enumi}{1} \r\n\\item $\\opt{\\text{\\ref{eq:red}}} =  \\opt{\\text{\\ref{eq:red_dual}}} = 0$ if and only if \r\none of the two alternatives below holds:\r\n\\begin{enumerate}\r\n\\item $\\inProd{c}{x^*} < 0 $ and $\\feasS = \\minFaceD = \\emptyset$, or\r\n\\item $\\inProd{c}{x^*} = 0 $, $\\feasS \\subseteq  \\stdCone \\cap \\{x^*\\}^\\perp \\subsetneq \\stdCone$ and \r\n$x^*_1 \\not \\in (\\stdCone^1)^\\perp = \\lineality( (\\stdCone^1 )^*)$.\r\n\\end{enumerate}\r\n\r\n","start":28912,"end":29497}]},{"id":"S033","semantic_type":"lemma","kind":"lemma","labels":["lemma:red2"],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"正值等价于 PPS","source_units":[{"raw_tex":"\\item $\\opt{\\text{\\ref{eq:red}}} =  \\opt{\\text{\\ref{eq:red_dual}}} >  0$ if and only if the {\\PPS} condition is satisfied for \\eqref{eq:dual}. In this case, we have\r\n","start":29497,"end":29663}]},{"id":"S034","semantic_type":"lemma","kind":"lemma","labels":["lemma:red2"],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"输出松弛的过强零块结论","source_units":[{"raw_tex":"$\r\nc  -\\stdMap^\\T \\frac{y_3^*}{y_1^*} \\in (\\reInt \\stdCone ^1)\\times \\{0\\}. \r\n$ (In particular, it is feasible for \\eqref{eq:dual})\r\n\r\n\\end{enumerate}\r\n","start":29663,"end":29815}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"验证两侧 PPS 可行性","source_units":[{"raw_tex":"\\begin{proof}\r\n\r\n\\begin{enumerate}[label=({\\it{\\roman*}})]\r\n\\item  Due to choice of $\\stdInt$ and $\\stdInt^*$, it is clear that the solutions \r\nmeet the PPS condition.\r\n\r\n\r\n","start":31151,"end":31324}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"零值产生不可行证书","source_units":[{"raw_tex":"\\item \\fbox{$\\opt{\\text{\\ref{eq:red}}} = 0 \\Rightarrow$  (a) or (b) holds.}\r\nSuppose that $\\opt{\\text{\\ref{eq:red}}} = 0$ and let $(x^*,0,w^*)$ be an optimal solution for \\eqref{eq:red}.\r\nDue to the constraints in \\eqref{eq:red}, we  have\r\n$$\\stdMap x^* = 0, \\qquad -\\inProd{c}{x^*} = w^* \\geq 0, \\qquad x^* \\in \\stdCone ^*.$$ \r\nNote that if $\\inProd{c}{x^*} < 0$, then $\\minFaceD = \\feasS = \\emptyset$. This is \r\nalternative (a).\r\n\r\n","start":31324,"end":31758}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"零配对约化非多面体部分","source_units":[{"raw_tex":"If $\\inProd{c}{x^*} = 0$, then (\\eqref{eq:red:1}) implies \r\n$w^* = 0$. Since $\\stdMap x^* = 0$, we obtain  \r\n$$\\feasS \\subseteq  \\stdCone \\cap \\{x^*\\}^\\perp. $$\r\nBy \\eqref{eq:red:2}, we have $\\inProd{\\stdInt}{x^*} = 1$, so that $x^* = (x_1^*,x_2^*)$ satisfies $x_1^* \\not \\in (\\stdCone^1)^\\perp$, due to the choice of $\\stdInt$. \r\nTherefore, the inclusion $\\stdCone \\cap \\{x^*\\}^\\perp \\subseteq \\stdCone$ is strict, \r\nthat is,  $\\stdCone \\cap \\{x^*\\}^\\perp \\subsetneq \\stdCone$. This is alternative (b).\r\n\r\n","start":31758,"end":32265}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":["eq:lemma:aux"],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"建立归一化分母正性","source_units":[{"raw_tex":"\\fbox{(a) or (b) holds $\\Rightarrow \\opt{\\text{\\ref{eq:red}}} = 0$.} \r\nFirst recall that  $t$ is constrained to be nonnegative, therefore  $\\opt{\\text{\\ref{eq:red}}} \\geq 0$.\r\nNext, we will prove the following inequality\r\n\\begin{equation}\\label{eq:lemma:aux}\r\n\t{\\inProd{\\stdInt}{x^*} - \\inProd{c}{x^*}} > 0.\r\n\\end{equation}\r\nSince $\\stdInt \\in \\stdCone,x^* \\in \\stdCone^*$, we have \r\n$\\inProd{\\stdInt}{x^*} \\geq 0$. If $(a)$ holds, then $- \\inProd{c}{x^*}> 0$.\r\nIn particular, \\eqref{eq:lemma:aux} holds.\r\nNow, we consider the case where (b) holds. \r\nIn this case, we have $\\inProd{c}{x^*} = 0$. \r\nIf $\\inProd{\\stdInt}{x^*}$ is also zero, then $\\inProd{\\stdInt_1}{x_1^*}$ is zero and since $\\stdInt_1 \\in \\reInt \\stdCone^1$, $ x^*_1$ must \r\nbelong to $(\\stdCone^1)^\\perp$. However, this is not allowed \r\nsince we assumed that $x_1^* \\not \\in (\\stdCone^1)^\\perp$.\r\nTherefore, we have $\\inProd{\\stdInt}{x^*} > 0$ and, in particular,  \\eqref{eq:lemma:aux} also holds.\r\n","start":32265,"end":33231}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"缩放得到零值可行解","source_units":[{"raw_tex":"\r\nLet  $$\r\n\\alpha = \\frac{1}{\\inProd{\\stdInt}{x^*} - \\inProd{c}{x^*}}.\r\n$$\r\nThen $\\alpha > 0$ and \r\n$(\\alpha x^*,0,-\\alpha \\inProd{c}{x^*})$ is a solution to \\eqref{eq:red} with value $0$, \r\nso that $\\opt{\\text{\\ref{eq:red}}} = 0$.\r\n\r\n","start":33231,"end":33466}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"正目标值构造 PPS 松弛","source_units":[{"raw_tex":"\\item \r\n\\fbox{ $\\opt{\\text{\\ref{eq:red_dual}}} > 0 \\Rightarrow $ PPS holds for \\eqref{eq:dual}}\r\nFrom \\eqref{eq:red_dual:1}, we have\r\n$$\r\nc y_1^* - \\stdInt y_2^* - {\\stdMap}^\\T y_3^* \\in \\stdCone.\r\n$$\r\nSince $y_2^*=\\opt{\\text{\\ref{eq:red_dual}}}\r\n>0$, we have, due to \\eqref{eq:red_dual:3}, $y_1^* > 0$.\r\n Hence, \r\n $$c - \\stdInt \\frac{y_2^*}{y_1^*} - {\\stdMap}^\\T \\frac{y_3^*}{y_1^*} \\in \\stdCone.$$ \r\nWe conclude that \r\n$c - {\\stdMap}^\\T y_3^*/y_1^* \\in (\\stdCone^1 +y_2^*/y_1^* \\stdInt_1)\\times  \\{0\\} \\subseteq (\\reInt \\stdCone ^1)\\times \\{0\\}$, due to the choice of  $\\stdInt$ and the fact that $y_2^*/y_1^* > 0$. \r\n\r\n","start":33466,"end":34089}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"零目标值排除 PPS","source_units":[{"raw_tex":"\\fbox{ $\\opt{\\text{\\ref{eq:red_dual}}} = 0 \\Rightarrow $ PPS does not hold for \\eqref{eq:dual}  } \r\nIf $t^* = 0$ and $(x^*,0,w^*)$ is an optimal solution \r\nfor \\eqref{eq:red}, then either $(a)$ or $(b)$ of item $(ii)$ is satisfied. \r\nIf $(a)$ is satisfied, then \\eqref{eq:dual} is infeasible and thus the PPS condition\r\ncannot hold. \r\nIf $(b)$ is satisfied, then $\\inProd{c}{x^*} = 0$,\r\n$\\stdMap x^* = 0$ and $x_1^* \\not \\in (\\stdCone ^1)^\\perp$. If \r\n$(s_1,s_2)$ is a feasible slack for \\eqref{eq:dual}, we have $\\inProd{s_1}{x_1^*} + \\inProd{s_2}{x_2^*} = 0$, so that \r\n$\\inProd{s_1}{x_1^*} = 0$.  \r\nAs $x_1^* \\not \\in (\\stdCone ^1)^\\perp$, we have that $s_1 \\not \\in \\reInt \\stdCone^1$. This means\r\nthat there is no feasible solution for \\eqref{eq:dual} satisfying the PPS\r\ncondition.\r\n\\end{enumerate}\r\n\\end{proof}","start":34089,"end":34906}]},{"id":"S043","semantic_type":"proposition","kind":"proposition","labels":["prop:strict"],"member_ids":["S043","S044","S045","S046"],"declared_external":false,"source_only":false,"title":"严格互补实现一步到面","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:strict}\r\nSuppose $\\opt{\\text{\\ref{eq:red}}} = \\opt{\\text{\\ref{eq:red_dual}}} = 0$  and that  we have  optimal solutions to \\eqref{eq:red} and \r\n\\eqref{eq:red_dual} satisfying dual strict complementarity. \r\nIf $w^* = 0$, then $\\minFaceD = \\stdCone \\cap\\{ x^*\\}^\\perp$.\r\n%Reciprocally, suppose that $\\minFace{D} \\neq \\stdCone $ and that there is a reducing direction $d$ for which $\\minFaceD = \\stdCone \\cap\\{ d\\}^\\perp$. Then, \r\n%\\eqref{eq:red} and \\eqref{eq:red_dual} satisfy dual strict complementarity if $\\stdInt$ is chosen as to satisfy $\\inProd{\\stdInt}{d} \\neq 0$ (in particular, if \r\n%$\\stdInt \\in \\reInt \\stdCone$).\r\n\\end{proposition}","start":46831,"end":47504},{"raw_tex":"\\begin{proof}\r\nSince $w^* = 0$, \\eqref{eq:strict3} implies that $y_1^* > y_2^*$. Then, $\\opt{\\text{\\ref{eq:red}}} = \\opt{\\text{\\ref{eq:red_dual}}} $ and \r\n$\\opt{\\text{\\ref{eq:red}}} \\geq 0 $ implies that $y_2^* \\geq 0$, so that $y_1^* > 0$ as well.\r\n","start":47506,"end":47756},{"raw_tex":"\\begin{proof}\r\nSince $w^* = 0$, \\eqref{eq:strict3} implies that $y_1^* > y_2^*$. Then, $\\opt{\\text{\\ref{eq:red}}} = \\opt{\\text{\\ref{eq:red_dual}}} $ and \r\n$\\opt{\\text{\\ref{eq:red}}} \\geq 0 $ implies that $y_2^* \\geq 0$, so that $y_1^* > 0$ as well.\r\nTherefore, from \\eqref{eq:strict1} we obtain $c -\\stdMap^\\T \\frac{y_3^*}{y_1^*} \\in \\reInt (\\stdCone \\cap\\{ x^*\\}^\\perp).$\r\n%To prove converse, let $y$ be such that $c - \\stdMap^\\T y \\in \\reInt (\\stdCone \\cap\\{ d\\}^\\perp)$. Then, for $\\alpha > 0$ sufficiently small, we have \r\n%that $(d/\\inProd{e}{d},0,0)$, $(\\alpha,0,\\alpha y)$ are optimal solutions satisfying strict complementarity.\r\n\\end{proof}","start":47506,"end":48155},{"raw_tex":"Therefore, from \\eqref{eq:strict1} we obtain $c -\\stdMap^\\T \\frac{y_3^*}{y_1^*} \\in \\reInt (\\stdCone \\cap\\{ x^*\\}^\\perp).$\r\n","start":47756,"end":47880}]},{"id":"S047","semantic_type":"proposition","kind":"proposition","labels":["prop:phase1"],"member_ids":["S047","S051"],"declared_external":false,"source_only":false,"title":"第一阶段正确性与计数","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:phase1}\r\nThe following hold.\r\n\\begin{enumerate}[label=(\\roman*)]\r\n\t\\item if \\eqref{eq:dual} is feasible, then the output face $\\stdFace$ satisfies \r\n\t$\\minFaceD \\subseteq \\stdFace$. Moreover, $s'$ is a dual \r\nfeasible slack such that $s_j' \\in \\reInt \\stdFace ^j $  for every $j$ such that $\\stdFace ^j$  is nonpolyhedral, i.e., the PPS condition is satisfied for $\\stdFace$.\r\n\r\nIn this case, Phase 1  stops after finding at most $\\sum _{i=1}^{r} \\distP(\\stdCone ^i) $ directions. \r\n\t\\item \\eqref{eq:dual} is infeasible if and only if Step 3 is reached. In this case, Phase 1 stops after finding \r\n\tat most $1+ \\sum _{i=1}^{r} \\distP(\\stdCone ^i) $ directions.\r\n\\end{enumerate}\r\n\\end{proposition}","start":51430,"end":52157},{"raw_tex":"\\begin{proof}\r\nWe will focus on the statements about the bounds, since the other statements \r\nare direct consequences of \r\nLemma \\ref{lemma:red2}. Note that whenever Step 4 is reached, we have \r\n$\\stdFace _{i+1} \\subsetneq \\stdFace _i$, since \r\n$x^*_j \\not \\in (\\stdFace _i^j)^\\perp$ for at least one nonpolyhedral cone $\\stdFace _i^j$, due to item $(ii)$-$(b)$ of Lemma \\ref{lemma:red2}. \r\nTherefore, whenever a new (proper) face is found, it is because \r\nwe are making progress towards a polyhedral face for at least one nonpolyhedral cone. \t\r\n\t\r\nBy definition, after finding $\\hat \\ell = \\sum _{i=1}^{r} (\\distP(\\stdCone ^i)) $ directions, \r\n$\\stdFace _{\\hat \\ell + 1}$ is polyhedral. We now consider what happens if \r\nthe algorithm has not stopped after all these directions were found.\r\nIn this case, when it is time to build \\eqref{eq:red} and \\eqref{eq:red_dual} with $\\stdFace_{l+1}$ in\r\nplace of $\\stdCone$ at Step 2, Phase 1 selects $e = 0$ and $\\stdInt^* \\in (\\stdFace _{\\hat \\ell + 1})^*$.\r\n\r\nFirst, suppose that \\eqref{eq:dual} is feasible and  let \r\n$y$ be such that $c-\\stdMap^\\T y \\in \\feasS$. \r\nSince $\\stdInt = 0$, $(\\alpha, \\alpha, \\alpha y)$\r\nis a feasible solution for \\eqref{eq:red_dual}, when $\\alpha>0$ is sufficiently small. It follows \r\nthat $\\opt{\\text{\\ref{eq:red_dual}}} > 0$ and that Phase 1 eventually reaches Step 5. This gives item $(i)$.\r\n\r\n\r\n\r\nFinally, suppose that \\eqref{eq:dual} is infeasible. By Lemma \\ref{lemma:red2},  $\\opt{\\text{\\ref{eq:red_dual}}} = 0$. Since $\\stdInt = 0$,  \\eqref{eq:red:2} implies that every optimal solution of \r\n\\eqref{eq:red} will be a triple of the form\r\n$(x^*,0,1)$, which implies that Step 3 will be reached and a single new direction will be \r\nadded. This gives  item $(ii)$.\r\n\\end{proof}","start":52159,"end":53918}]},{"id":"S048","semantic_type":"proposition","kind":"proposition","labels":["prop:phase1"],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"第一阶段的逐块内部松弛","source_units":[{"raw_tex":"\t\\item if \\eqref{eq:dual} is feasible, then the output face $\\stdFace$ satisfies \r\n\t$\\minFaceD \\subseteq \\stdFace$. Moreover, $s'$ is a dual \r\nfeasible slack such that $s_j' \\in \\reInt \\stdFace ^j $  for every $j$ such that $\\stdFace ^j$  is nonpolyhedral, i.e., the PPS condition is satisfied for $\\stdFace$.\r\n","start":51527,"end":51838}]},{"id":"S049","semantic_type":"proposition","kind":"proposition","labels":["prop:phase1"],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"可行时第一阶段方向数","source_units":[{"raw_tex":"\r\nIn this case, Phase 1  stops after finding at most $\\sum _{i=1}^{r} \\distP(\\stdCone ^i) $ directions. \r\n","start":51838,"end":51944}]},{"id":"S050","semantic_type":"proposition","kind":"proposition","labels":["prop:phase1"],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"不可行检测及方向数","source_units":[{"raw_tex":"\t\\item \\eqref{eq:dual} is infeasible if and only if Step 3 is reached. In this case, Phase 1 stops after finding \r\n\tat most $1+ \\sum _{i=1}^{r} \\distP(\\stdCone ^i) $ directions.\r\n","start":51944,"end":52123}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"调用辅助引理保证正确性","source_units":[{"raw_tex":"\\begin{proof}\r\nWe will focus on the statements about the bounds, since the other statements \r\nare direct consequences of \r\nLemma \\ref{lemma:red2}. Note that whenever Step 4 is reached, we have \r\n","start":52159,"end":52354}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"每次截面推进非多面体块","source_units":[{"raw_tex":"$\\stdFace _{i+1} \\subsetneq \\stdFace _i$, since \r\n$x^*_j \\not \\in (\\stdFace _i^j)^\\perp$ for at least one nonpolyhedral cone $\\stdFace _i^j$, due to item $(ii)$-$(b)$ of Lemma \\ref{lemma:red2}. \r\nTherefore, whenever a new (proper) face is found, it is because \r\nwe are making progress towards a polyhedral face for at least one nonpolyhedral cone. \t\r\n\t\r\n","start":52354,"end":52708}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"距离之和控制多面体终点","source_units":[{"raw_tex":"By definition, after finding $\\hat \\ell = \\sum _{i=1}^{r} (\\distP(\\stdCone ^i)) $ directions, \r\n$\\stdFace _{\\hat \\ell + 1}$ is polyhedral. We now consider what happens if \r\nthe algorithm has not stopped after all these directions were found.\r\nIn this case, when it is time to build \\eqref{eq:red} and \\eqref{eq:red_dual} with $\\stdFace_{l+1}$ in\r\nplace of $\\stdCone$ at Step 2, Phase 1 selects $e = 0$ and $\\stdInt^* \\in (\\stdFace _{\\hat \\ell + 1})^*$.\r\n\r\n","start":52708,"end":53164}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"多面体可行终点直接停止","source_units":[{"raw_tex":"First, suppose that \\eqref{eq:dual} is feasible and  let \r\n$y$ be such that $c-\\stdMap^\\T y \\in \\feasS$. \r\nSince $\\stdInt = 0$, $(\\alpha, \\alpha, \\alpha y)$\r\nis a feasible solution for \\eqref{eq:red_dual}, when $\\alpha>0$ is sufficiently small. It follows \r\nthat $\\opt{\\text{\\ref{eq:red_dual}}} > 0$ and that Phase 1 eventually reaches Step 5. This gives item $(i)$.\r\n\r\n\r\n\r\n","start":53164,"end":53538}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"多面体不可行终点增加证书","source_units":[{"raw_tex":"Finally, suppose that \\eqref{eq:dual} is infeasible. By Lemma \\ref{lemma:red2},  $\\opt{\\text{\\ref{eq:red_dual}}} = 0$. Since $\\stdInt = 0$,  \\eqref{eq:red:2} implies that every optimal solution of \r\n\\eqref{eq:red} will be a triple of the form\r\n$(x^*,0,1)$, which implies that Step 3 will be reached and a single new direction will be \r\nadded. This gives  item $(ii)$.\r\n\\end{proof}","start":53538,"end":53918}]},{"id":"S057","semantic_type":"proposition","kind":"proposition","labels":["prop:min_face"],"member_ids":["S057","S058","S059","S060","S061","S062","S063","S064","S065","S066"],"declared_external":false,"source_only":false,"title":"第二阶段得到最小面","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:min_face}\r\nThe output face of Phase 2 is $\\minFaceD$ and there exists $\\hat s$ as in Steps 3. and 4.\r\n\\end{proposition}","start":62497,"end":62647},{"raw_tex":"\\begin{proof}\r\nSuppose that the output face $\\stdFace$ of Phase 1 satisfies $\\stdFace \\neq \\minFaceD$. \r\nBy Lemma 3.2 in \\cite{article_waki_muramatsu}, there is \r\na reducing direction $x$ such that $x \\in \\stdFace ^* \\cap \\ker \\stdMap \\cap \\{c\\}^\\perp$ \r\nand $x \\not \\in \\stdFace ^\\perp$.\r\n","start":62649,"end":62939},{"raw_tex":"\\begin{proof}\r\nSuppose that the output face $\\stdFace$ of Phase 1 satisfies $\\stdFace \\neq \\minFaceD$. \r\nBy Lemma 3.2 in \\cite{article_waki_muramatsu}, there is \r\na reducing direction $x$ such that $x \\in \\stdFace ^* \\cap \\ker \\stdMap \\cap \\{c\\}^\\perp$ \r\nand $x \\not \\in \\stdFace ^\\perp$.\r\nDue to Proposition \\ref{prop:phase1}, any such reducing direction $x$ satisfies $\\inProd{x}{s'} = 0$, which implies that \r\n $x_j \\in ({\\stdFace ^j})^\\perp = ({\\spanVec \\stdFace ^j})^\\perp$ for every $j$ such that $\\stdFace ^j$ is not polyhedral, \r\n since $s_j' \\in \\reInt \\stdFace ^j $ for those $j$.\r\nTherefore, the possible reducing directions are confined to the polyhedral cone $\\hat \\stdCone^*$,\r\nwhere $\\hat \\stdCone$ is the cone in Step 1. of Phase 2.\r\n\r\nSince $\\hat \\stdCone$ is polyhedral, the problems \\eqref{eq:red} and \\eqref{eq:red_dual} \r\n are polyhedral and they admit strictly complementary optimal \r\nsolutions $(x^*,t^*,w^*)$, $(y_1^*,y_2^*,y_3^*)$.\r\nThe fact that $x \\not \\in \\stdFace^\\perp$ implies that $\\inProd{\\stdInt}{x}\\neq 0$ so that $(x/\\inProd{\\stdInt}{x},0,0) $ is an optimal \r\nsolution to \\eqref{eq:red}. Therefore, $t^* = y_2^* = 0$. Moreover, since \\eqref{eq:dual} is \r\nfeasible, we have $w^* = 0$.\r\nBy Proposition \\ref{prop:strict}, we have \r\n$c -\\stdMap^\\T  \\frac{y_3^*}{y_1^*} \\in \\reInt (\\hat \\stdCone \\cap \\{x^*\\}^\\perp).$\r\n\r\nLet $\\tilde s = c -\\stdMap^\\T  \\frac{y_3^*}{y_1^*}$. Note that $\\stdFace \\cap \\{x^*\\}^\\perp$ is a face of \r\n$\\stdFace$ containing $\\feasS$, since we argued that $x^*$ must be a reducing direction. We will prove that $\\minFaceD = \\stdFace \\cap \\{x^*\\}^\\perp$ by \r\n showing that some convex combination of $s'$ and $\\tilde s$ lies in $\\reInt (\\stdFace \\cap \\{x^*\\}^\\perp)$. \r\n% Intuitively, $s'$ and $\\tilde s$ are  relative interior points \r\n%of the nonpolyhedral and polyhedral parts of $\\stdFace \\cap \\{x^*\\}^\\perp$, respectively. So some convex \r\n%combination of them will belong to $\\feasS$ and be a relative interior point of $\\stdFace \\cap \\{x^*\\}^\\perp$.\r\n\r\nLet $z_\\beta = \\beta s' + (1-\\beta)\\tilde s$. For \r\nall $\\beta \\in (0,1)$ and all $j$ such that $\\stdFace ^j$ is polyhedral, we \r\nhave $(z_\\beta)_j \\in \\reInt ( \\stdFace ^j  \\cap \\{x_j^*\\}^\\perp)$, because $\\tilde s _j \\in  \\reInt (\\stdFace ^j  \\cap \\{x_j\\}^\\perp)$ and \r\n$s'$ is feasible.\r\nIf $\\stdFace ^j$ is not polyhedral, then $\\stdFace ^j  \\cap \\{x_j^*\\}^\\perp = \\stdFace ^j$, \r\nsince $x_j \\in (\\stdFace ^j)^\\perp$. Because $\\tilde s _j \\in \\spanVec{\\stdFace ^j}$ and \r\n$s_j' \\in \\reInt \\stdFace^j$, for all $\\beta $ sufficiently close to $1$ we have \r\n$(z_\\beta)_j \\in \\reInt \\stdFace^j$. Therefore, it is possible to select $\\beta \\in (0,1)$ \r\nsuch that $(z_\\beta)_j \\in \\reInt (\\stdFace ^j  \\cap \\{x_j\\}^\\perp) $ for all $j$. This \r\nshows that $\\minFaceD = \\stdFace \\cap \\{x^*\\}^\\perp $.\r\n\r\nIf $\\stdFace$ was already the minimal face to begin with, then \r\n$t^* > 0$. We can then  proceed in a \r\nsimilar fashion. The only difference is that due to \\eqref{eq:red_dual:1}, we \r\nwill have that $\\tilde s = c -\\stdMap^\\T  \\frac{y_3^*}{y_1^*} $ satisfies \r\n$\\tilde s _j \\in  \\reInt (\\stdFace ^j)$ for every $j$ such that $\\stdFace ^j$ is polyhedral.\r\nAnd as before, we can select a convex combination of $s'$ and $\\tilde s$ belonging to \r\nthe relative interior of $\\minFaceD$.\r\n\\end{proof}","start":62649,"end":65957},{"raw_tex":"Due to Proposition \\ref{prop:phase1}, any such reducing direction $x$ satisfies $\\inProd{x}{s'} = 0$, which implies that \r\n $x_j \\in ({\\stdFace ^j})^\\perp = ({\\spanVec \\stdFace ^j})^\\perp$ for every $j$ such that $\\stdFace ^j$ is not polyhedral, \r\n since $s_j' \\in \\reInt \\stdFace ^j $ for those $j$.\r\nTherefore, the possible reducing directions are confined to the polyhedral cone $\\hat \\stdCone^*$,\r\nwhere $\\hat \\stdCone$ is the cone in Step 1. of Phase 2.\r\n\r\n","start":62939,"end":63401},{"raw_tex":"Since $\\hat \\stdCone$ is polyhedral, the problems \\eqref{eq:red} and \\eqref{eq:red_dual} \r\n are polyhedral and they admit strictly complementary optimal \r\nsolutions $(x^*,t^*,w^*)$, $(y_1^*,y_2^*,y_3^*)$.\r\nThe fact that $x \\not \\in \\stdFace^\\perp$ implies that $\\inProd{\\stdInt}{x}\\neq 0$ so that $(x/\\inProd{\\stdInt}{x},0,0) $ is an optimal \r\nsolution to \\eqref{eq:red}. Therefore, $t^* = y_2^* = 0$. Moreover, since \\eqref{eq:dual} is \r\nfeasible, we have $w^* = 0$.\r\n","start":63401,"end":63870},{"raw_tex":"By Proposition \\ref{prop:strict}, we have \r\n$c -\\stdMap^\\T  \\frac{y_3^*}{y_1^*} \\in \\reInt (\\hat \\stdCone \\cap \\{x^*\\}^\\perp).$\r\n\r\n","start":63870,"end":64001},{"raw_tex":"Let $\\tilde s = c -\\stdMap^\\T  \\frac{y_3^*}{y_1^*}$. Note that $\\stdFace \\cap \\{x^*\\}^\\perp$ is a face of \r\n$\\stdFace$ containing $\\feasS$, since we argued that $x^*$ must be a reducing direction. We will prove that $\\minFaceD = \\stdFace \\cap \\{x^*\\}^\\perp$ by \r\n showing that some convex combination of $s'$ and $\\tilde s$ lies in $\\reInt (\\stdFace \\cap \\{x^*\\}^\\perp)$. \r\n% Intuitively, $s'$ and $\\tilde s$ are  relative interior points \r\n%of the nonpolyhedral and polyhedral parts of $\\stdFace \\cap \\{x^*\\}^\\perp$, respectively. So some convex \r\n%combination of them will belong to $\\feasS$ and be a relative interior point of $\\stdFace \\cap \\{x^*\\}^\\perp$.\r\n\r\n","start":64001,"end":64665},{"raw_tex":"Let $z_\\beta = \\beta s' + (1-\\beta)\\tilde s$. For \r\nall $\\beta \\in (0,1)$ and all $j$ such that $\\stdFace ^j$ is polyhedral, we \r\nhave $(z_\\beta)_j \\in \\reInt ( \\stdFace ^j  \\cap \\{x_j^*\\}^\\perp)$, because $\\tilde s _j \\in  \\reInt (\\stdFace ^j  \\cap \\{x_j\\}^\\perp)$ and \r\n$s'$ is feasible.\r\n","start":64665,"end":64956},{"raw_tex":"If $\\stdFace ^j$ is not polyhedral, then $\\stdFace ^j  \\cap \\{x_j^*\\}^\\perp = \\stdFace ^j$, \r\nsince $x_j \\in (\\stdFace ^j)^\\perp$. Because $\\tilde s _j \\in \\spanVec{\\stdFace ^j}$ and \r\n$s_j' \\in \\reInt \\stdFace^j$, for all $\\beta $ sufficiently close to $1$ we have \r\n$(z_\\beta)_j \\in \\reInt \\stdFace^j$. Therefore, it is possible to select $\\beta \\in (0,1)$ \r\nsuch that $(z_\\beta)_j \\in \\reInt (\\stdFace ^j  \\cap \\{x_j\\}^\\perp) $ for all $j$. This \r\nshows that $\\minFaceD = \\stdFace \\cap \\{x^*\\}^\\perp $.\r\n","start":64956,"end":65463},{"raw_tex":"\r\nIf $\\stdFace$ was already the minimal face to begin with, then \r\n$t^* > 0$. We can then  proceed in a \r\nsimilar fashion. The only difference is that due to \\eqref{eq:red_dual:1}, we \r\nwill have that $\\tilde s = c -\\stdMap^\\T  \\frac{y_3^*}{y_1^*} $ satisfies \r\n$\\tilde s _j \\in  \\reInt (\\stdFace ^j)$ for every $j$ such that $\\stdFace ^j$ is polyhedral.\r\nAnd as before, we can select a convex combination of $s'$ and $\\tilde s$ belonging to \r\nthe relative interior of $\\minFaceD$.\r\n\\end{proof}","start":65463,"end":65957}]},{"id":"S067","semantic_type":"theorem","kind":"theorem","labels":["theo:fra_bound"],"member_ids":["S067"],"declared_external":false,"source_only":false,"title":"FRA-Poly 面约化步数界","source_units":[{"raw_tex":"\\begin{theorem}\\label{theo:fra_bound}\r\nLet $\\stdCone = \\stdCone ^1\\times \\ldots \\times \\stdCone^r$. The minimum face $\\minFaceD$ \r\nthat contains the feasible region of \\eqref{eq:dual} can be found in no more than \r\n$1+ \\sum _{i=1}^{r} \\distP(\\stdCone ^i) $ facial reduction steps.\r\n\\end{theorem}","start":66021,"end":66316}]},{"id":"S068","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S068"],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\r\nIf \\eqref{eq:dual} is infeasible, then $\\minFaceD = \\emptyset$ and the result \r\nfollows from Proposition \\ref{prop:phase1}. So suppose now that \r\n\\eqref{eq:dual} is feasible. Then Phase 1 ends after finding at most \r\n$\\sum _{i=1}^{r} \\distP(\\stdCone ^i)$ directions. Due to Proposition \\ref{prop:min_face}, at most one \r\nextra direction is needed to jump to the minimal face.\r\n\\end{proof}","start":66318,"end":66721}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S069"],"declared_external":false,"source_only":false,"title":"主证明的不可行分支","source_units":[{"raw_tex":"\\begin{proof}\r\nIf \\eqref{eq:dual} is infeasible, then $\\minFaceD = \\emptyset$ and the result \r\nfollows from Proposition \\ref{prop:phase1}. So suppose now that \r\n","start":66318,"end":66479}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S070"],"declared_external":false,"source_only":false,"title":"主证明的可行分支","source_units":[{"raw_tex":"\\eqref{eq:dual} is feasible. Then Phase 1 ends after finding at most \r\n$\\sum _{i=1}^{r} \\distP(\\stdCone ^i)$ directions. Due to Proposition \\ref{prop:min_face}, at most one \r\nextra direction is needed to jump to the minimal face.\r\n\\end{proof}","start":66479,"end":66721}]},{"id":"S071","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S071"],"declared_external":false,"source_only":false,"title":"PPS 强对偶的附录证明","source_units":[{"raw_tex":"\\begin{proof}\r\nWe will prove $(i)$ first. Let \r\n$f_1$ be such that $f_1(x) = \\inProd{c}{x}$ if $Ax = b$ and $+\\infty$ otherwise.\r\nLet  $f_2$ be the indicator function of $\\Re^{n_1}\\times (\\stdCone ^2)^*$ and $f_3$ be the indicator \r\nfunction of $(\\stdCone ^1)^*\\times \\Re^{n_2}$. Since there is  a primal feasible solution $x = (x_1,x_2)$ such \r\nthat $x_1 \\in \\reInt (\\stdCone ^1)^*$, we have that $\\fDom f_1 \\cap \\fDom f_2 \\cap \\reInt (\\fDom f_3)$ is \r\nnonempty. In addition, $f_1$ and $f_2$ are polyhedral functions. Let us now observe that:\r\n\\begin{align*}\r\nf_1^*(s) = \\left\\{\r\n  \\begin{array}{l l}\r\n    \\inProd{b}{y}  & \\quad \\text{if there is $y$ with $s-c = \\stdMap^\\T y$}\\\\\r\n    +\\infty & \\quad \\text{otherwise}\r\n  \\end{array}\\right. \r\n\\end{align*}\r\nNote that, due to feasibility, for fixed $s$, $\\inProd{b}{y}$ does not depend on the choice \r\nof $y$, as long as $c+\\stdMap ^\\T y = s$. This is because since there is $x$ such that $Ax = b$, we have $\\inProd{b}{y} = \\inProd{x}{s-c}$.\r\nThe conjugate $f_2^*$ is the indicator function of $- \\{0\\}\\times \\stdCone ^2$ and \r\n$f_3^*$ is the indicator function of $- \\stdCone ^1\\times \\{0\\}$. \r\nApplying Theorem \\ref{theo:conv} with $s = 0$, we have:\r\n\\begin{align*}\r\n(f_1+f_2+f_3)^*(0) & = \\inf \\left\\{ \\inProd{b}{y} \\mid  c+ \\stdMap ^\\T y = s_1, s_1 - (s_3,s_2) = 0,  s_2 \\in  \\stdCone ^2, s_3 \\in \\stdCone ^1      \\right\\} \\\\\r\n & = \\inf \\left\\{ \\inProd{b}{y} \\mid  c+ \\stdMap ^\\T y = s_1, s_1 \\in  \\stdCone ^1\\times \\stdCone ^2      \\right\\} \\\\\r\n& = - \\sup \\left\\{ \\inProd{b}{y} \\mid  c -\\stdMap ^\\T y = s_1,  s_1 \\in  \\stdCone ^1 \\times \\stdCone ^2      \\right\\},\r\n\\end{align*}\r\nwhere the $\\sup$ in the last equation is attained. So, there is some dual feasible $y$ such that \r\n$(f_1+f_2+f_3)^*(0) =  \\inProd{b}{y}$. However, using the definition of conjugate, we also have:\r\n\\begin{align*}\r\n(f_1+f_2+f_3)^*(0) = - \\inf \\{ \\inProd{c}{x} \\mid Ax = b, x \\in \\stdCone ^1\\times \\stdCone ^2\\} = -\\theta _P.\r\n\\end{align*}\r\nIt follows that $\\pOpt = \\dOpt$ and the dual is attained at $y$. \r\nTo prove $(ii)$, let $g_1 = f_1^*$, and let $g_2$ and $g_3$ be the indicator \r\nfunctions  of $\\Re^{n_1}\\times \\stdCone ^2$ and  $\\stdCone ^1 \\times \\Re^{n_2}$, respectively. Again, \r\nit is enough to compute $(g_1+g_2+g_3)^*(0)$ using both the definition of conjugate function and \r\nusing Theorem \\ref{theo:conv}. \r\n\\end{proof}","start":116866,"end":119230}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S072"],"declared_external":false,"source_only":false,"title":"构造函数并核对交点条件","source_units":[{"raw_tex":"\\begin{proof}\r\nWe will prove $(i)$ first. Let \r\n$f_1$ be such that $f_1(x) = \\inProd{c}{x}$ if $Ax = b$ and $+\\infty$ otherwise.\r\nLet  $f_2$ be the indicator function of $\\Re^{n_1}\\times (\\stdCone ^2)^*$ and $f_3$ be the indicator \r\nfunction of $(\\stdCone ^1)^*\\times \\Re^{n_2}$. Since there is  a primal feasible solution $x = (x_1,x_2)$ such \r\nthat $x_1 \\in \\reInt (\\stdCone ^1)^*$, we have that $\\fDom f_1 \\cap \\fDom f_2 \\cap \\reInt (\\fDom f_3)$ is \r\nnonempty. In addition, $f_1$ and $f_2$ are polyhedral functions. Let us now observe that:\r\n","start":116866,"end":117411}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S073"],"declared_external":false,"source_only":false,"title":"计算分解函数的共轭","source_units":[{"raw_tex":"\\begin{align*}\r\nf_1^*(s) = \\left\\{\r\n  \\begin{array}{l l}\r\n    \\inProd{b}{y}  & \\quad \\text{if there is $y$ with $s-c = \\stdMap^\\T y$}\\\\\r\n    +\\infty & \\quad \\text{otherwise}\r\n  \\end{array}\\right. \r\n\\end{align*}\r\nNote that, due to feasibility, for fixed $s$, $\\inProd{b}{y}$ does not depend on the choice \r\nof $y$, as long as $c+\\stdMap ^\\T y = s$. This is because since there is $x$ such that $Ax = b$, we have $\\inProd{b}{y} = \\inProd{x}{s-c}$.\r\nThe conjugate $f_2^*$ is the indicator function of $- \\{0\\}\\times \\stdCone ^2$ and \r\n$f_3^*$ is the indicator function of $- \\stdCone ^1\\times \\{0\\}$. \r\n","start":117411,"end":118011}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S074"],"declared_external":false,"source_only":false,"title":"应用共轭定理得到对偶表示","source_units":[{"raw_tex":"Applying Theorem \\ref{theo:conv} with $s = 0$, we have:\r\n\\begin{align*}\r\n(f_1+f_2+f_3)^*(0) & = \\inf \\left\\{ \\inProd{b}{y} \\mid  c+ \\stdMap ^\\T y = s_1, s_1 - (s_3,s_2) = 0,  s_2 \\in  \\stdCone ^2, s_3 \\in \\stdCone ^1      \\right\\} \\\\\r\n & = \\inf \\left\\{ \\inProd{b}{y} \\mid  c+ \\stdMap ^\\T y = s_1, s_1 \\in  \\stdCone ^1\\times \\stdCone ^2      \\right\\} \\\\\r\n& = - \\sup \\left\\{ \\inProd{b}{y} \\mid  c -\\stdMap ^\\T y = s_1,  s_1 \\in  \\stdCone ^1 \\times \\stdCone ^2      \\right\\},\r\n\\end{align*}\r\nwhere the $\\sup$ in the last equation is attained. So, there is some dual feasible $y$ such that \r\n$(f_1+f_2+f_3)^*(0) =  \\inProd{b}{y}$. However, using the definition of conjugate, we also have:\r\n","start":118011,"end":118696}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S075"],"declared_external":false,"source_only":false,"title":"按定义识别原始最优值","source_units":[{"raw_tex":"\\begin{align*}\r\n(f_1+f_2+f_3)^*(0) = - \\inf \\{ \\inProd{c}{x} \\mid Ax = b, x \\in \\stdCone ^1\\times \\stdCone ^2\\} = -\\theta _P.\r\n\\end{align*}\r\nIt follows that $\\pOpt = \\dOpt$ and the dual is attained at $y$. \r\n","start":118696,"end":118904}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S076"],"declared_external":false,"source_only":false,"title":"对偶 PPS 情形的略证","source_units":[{"raw_tex":"To prove $(ii)$, let $g_1 = f_1^*$, and let $g_2$ and $g_3$ be the indicator \r\nfunctions  of $\\Re^{n_1}\\times \\stdCone ^2$ and  $\\stdCone ^1 \\times \\Re^{n_2}$, respectively. Again, \r\nit is enough to compute $(g_1+g_2+g_3)^*(0)$ using both the definition of conjugate function and \r\nusing Theorem \\ref{theo:conv}. \r\n\\end{proof}","start":118904,"end":119230}]},{"id":"S077","semantic_type":"proposition","kind":"proposition","labels":[],"member_ids":["S077"],"declared_external":false,"source_only":true,"title":"附录重复的强对偶命题","source_units":[{"raw_tex":"\\begin{proposition}\r\nLet $\\stdCone = \\stdCone ^1\\times \\stdCone ^2$, where  $\\stdCone ^1\\subseteq \\Re^{n_1},\\stdCone ^2 \\subseteq \\Re^{n_2}$ are closed convex cones such that $\\stdCone ^2$ is polyhedral.\r\n\\begin{enumerate}[label=({\\it\\roman*})]\r\n\\item If $\\pOpt$ is finite and \\eqref{eq:primal} satisfies the PPS condition, then $\\pOpt = \\dOpt$ and the dual optimal value is attained.\r\n\\item If $\\dOpt$ is finite and \\eqref{eq:dual} satisfies the PPS condition, then $\\pOpt = \\dOpt$ and the primal \r\noptimal value is attained.\r\n\\end{enumerate}\r\n\\end{proposition}\r\n","start":116302,"end":116866}]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S078"],"declared_external":true,"declaration_text":["声明允许的标准有限维凸分析之相对内部运算；外推性质在原文第322行明确引用 Rockafellar Theorem 6.4，其余为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"相对内部的基本运算","source_units":[]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S079"],"declared_external":true,"declaration_text":["声明允许的标准有限维凸分析：对偶锥恒等式与乘积面结构；此处为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"乘积锥及暴露截面恒等式","source_units":[]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S080"],"declared_external":true,"declaration_text":["声明允许的标准有限维凸分析：最小面刻画；implicit standard fact made explicit by annotator。"],"source_only":false,"title":"最小面的内部点判据","source_units":[]},{"id":"S081","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S081"],"declared_external":true,"declaration_text":["声明允许的弱锥对偶与初等线性代数；implicit standard fact made explicit by annotator。"],"source_only":false,"title":"弱对偶与配对计算","source_units":[]},{"id":"S082","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S082"],"declared_external":true,"declaration_text":["原许可清单中的标准有限维凸分析、对偶锥恒等式与初等线性代数；具体共轭公式为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"指标函数的共轭公式","source_units":[]},{"id":"S083","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S083"],"declared_external":false,"source_only":false,"title":"第二阶段的输入数据","source_units":[{"raw_tex":"{\\bf Input:} \\eqref{eq:dual}, the output of Phase 1: $\\stdFace$ and $s'$, with $\\stdFace \\neq \\emptyset$.\r\n","start":59220,"end":59327}]},{"id":"S084","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S084"],"declared_external":false,"source_only":false,"title":"当前面尚非最小面的分支","source_units":[{"raw_tex":"Suppose that the output face $\\stdFace$ of Phase 1 satisfies $\\stdFace \\neq \\minFaceD$. \r\n","start":62664,"end":62754}]}],"collapse_map":{"S001":"S001","S002":"S002","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S012":"S012","S013":"S013","S014":"S014","S016":"S016","S017":"S017","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S029","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S043","S045":"S043","S046":"S043","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S047","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S057","S059":"S057","S060":"S057","S061":"S057","S062":"S057","S063":"S057","S064":"S057","S065":"S057","S066":"S057","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076","S077":"S077","S078":"S078","S079":"S079","S080":"S080","S081":"S081","S082":"S082","S083":"S083","S084":"S084"}}}}
{"record_id":"1810.05633v2","paper":{"title":"Stochastic (Approximate) Proximal Point Methods: Convergence, Optimality, and Adaptivity","authors":["Asi, Hilal","Duchi, John C."],"arxiv_id":"1810.05633","version":"v2","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/1810.05633v2","pdf_url":"https://arxiv.org/pdf/1810.05633v2","doi":"10.1137/18M1230323"},"context":{"raw_tex":"In this paper, we develop and analyze a family of model-based methods,\nmoving beyond naive stochastic gradient methods, for solving the stochastic\nconvex optimization problem\n\\begin{equation}\n  \\label{eqn:objective}\n  \\begin{split}\n    \\minimize ~ & F(x) = \\E_P[f(x;\\statrv)]\n    = \\int_\\statdomain f(x; \\statval) dP(\\statval) \\\\\n    \\subjectto ~ & x \\in \\mc{X}.\n  \\end{split}\n\\end{equation}\nIn problem~\\eqref{eqn:objective}, the set $\\statdomain$ is a sample space,\nand for each $\\statval \\in \\statdomain$, the function $f(\\cdot; \\statval) :\n\\R^n \\to \\R$ is a closed convex function, subdifferentiable\non the closed convex set $\\mc{X} \\subset \\R^n$.\n\n\nWe show how to extend this modeling perspective to stochastic convex\noptimization problems, leveraging it to build a new family of algorithms for\nsolving problem~\\eqref{eqn:objective}, which, in homage to the stochastic\nproximal point iteration~\\eqref{eqn:sppm}, we call the \\aProx (approximate\nproximal point) algorithms, with substantially better theoretical guarantees\nand empirical performance than naive stochastic subgradient methods. The\n\\aProx algorithms iterate as follows: for $k = 1, 2, \\ldots$, we draw a\nrandom $\\statrv_k \\simiid P$, then update the iterate $x_k$ by minimizing a\nregularized approximation to $f(\\cdot; \\statrv_k)$, setting\n\\begin{equation}\n  \\label{eqn:model-iteration}\n  x_{k+1} \\defeq \\argmin_{x \\in \\mc{X}}\n  \\left\\{ f_{x_k}(x ; \\statrv_k) + \\frac{1}{2 \\stepsize_k}\n  \\ltwo{x - x_k}^2 \\right\\}.\n\\end{equation}\nThe function $f_x(\\cdot ; \\statval)$ is a \\emph{model} of $f(\\cdot;\n\\statval)$ at the point $x$, meaning that $f_x$ satisfies the\nfollowing conditions on its structure and local approximation\nproperties for $f$:\n\\begin{enumerate}[label=(C.\\roman*),leftmargin=*]\n\\item \\label{cond:convex-model}\n  The function $y \\mapsto f_x(y; \\statval)$ is convex\n  and subdifferentiable on $\\mc{X}$.\n\\item \\label{cond:lower-model}\n  The model $f_x$ satisfies the equality\n  $f_x(x; \\statval) = f(x; \\statval)$ and\n  \\begin{equation*}\n    f_x(y; \\statval) \\le f(y; \\statval) ~~ \\mbox{for~all~} y.\n  \\end{equation*}\n%\\item \\label{cond:subgrad-model}\n%  At $y = x$, we have the containment\n%  \\begin{equation*}\n%    \\left.\\partial_y f_x(y; \\statval)\\right|_{y = x}\n%    \\subset \\partial_x f(x; \\statval).\n%  \\end{equation*}\n\\end{enumerate}\n\\noindent\nBy the first-order conditions for convexity, for any $g \\in \\partial_y\nf_x(y; \\statval)|_{y=x}$ Conditions~\\ref{cond:convex-model}\nand~\\ref{cond:lower-model} imply $f(y; \\statval) \\ge f_x(y; \\statval) \\ge\nf_x(x; \\statval) + \\<g, y - x\\> = f(x; \\statval) + \\<g, y - x\\>$, yielding\nthe containment\n\\begin{equation}\n  \\label{eqn:subgrad-containment}\n  \\left.\\partial_y f_x(y; \\statval)\\right|_{y = x}\n  \\subset \\partial_x f(x; \\statval).\n\\end{equation}\n\n\n\\begin{assumption}\n  \\label{assumption:convex}\n  The set $\\mc{X}\\opt \\defeq \\argmin_{x \\in \\mc{X}} \\{F(x)\\}$ is non-empty,\n  and there exists $\\sigma^2 < \\infty$\n  such that $\\E[\\ltwo{f'(x\\opt;\\statrv)}^2] \\leq \\sigma^2$\n  for $x\\opt \\in \\mc{X}\\opt$ \n  and all measurable selections $f'(x\\opt; \\statval) \\in \\partial\n  f(x\\opt; \\statval)$.\n\\end{assumption}\n\n\\begin{assumption}\n  \\label{assumption:very-weak-moment}\n  There exists a non-decreasing function $\\growfunc : \\R_+ \\to\n  \\openright{0}{\\infty}$ such that for all $x \\in\n  \\mc{X}$ and measurable selections $f'(x; \\statval) \\in \\partial\n  f(x; \\statval)$,\n  $\\E[\\norm{f'(x; \\statrv)}^2] \\le \\growfunc(\\dist(x, \\mc{X}\\opt))$.\n\\end{assumption}\n\n\n\\paragraph{Notation}\nFor a convex function $f$, $\\partial f(x)$ denotes its subgradient set at\n$x$, and $f'(x) \\in \\partial f(x)$ denotes an arbitrary element of\nthe subdifferential.  Throughout, $x\\opt$ denotes a minimizer of\nproblem~\\eqref{eqn:objective} and $\\mc{X}\\opt = \\argmin_{x \\in \\mc{X}} F(x)$\nits optimal set.  We let $\\mc{F}_k \\defeq \\sigma(\\statrv_1, \\ldots,\n\\statrv_k)$ be the $\\sigma$-field generated by the first $k$ random\nvariables $\\statrv_i$, so $x_k \\in\n\\mc{F}_{k-1}$ for all $k$ under iteration~\\eqref{eqn:model-iteration}.\n\n\n\\begin{lemma}[\\cite{RobbinsSi71}]\n  \\label{lemma:robbins-siegmund}\n  Let $A_k, B_k, C_k, D_k \\ge 0$ be non-negative random variables adapted to\n  the filtration $\\mc{F}_k$ and satisfying $\\E[A_{k + 1} \\mid \\mc{F}_k] \\le\n  (1 + B_k) A_k + C_k - D_k$. Then on the event $\\{\\sum_k B_k < \\infty,\n  \\sum_k C_k < \\infty\\}$, there is a random $A_\\infty < \\infty$\n  such that $A_k \\cas A_\\infty$ and $\\sum_k D_k < \\infty$.\n\\end{lemma}\n\n\nThe key consequence of Theorem~\\ref{theorem:prox-point-bounded} is that the\niterates $x_k$ are stable in probability~\\eqref{eqn:bounded-distance},\nremaining bounded with probability 1. Iterate boundedness of algorithms does\nnot guarantee convergence in general; however, in this section, we show that\nas a consequence of this boundedness, any algorithm satisfying\nConditions~\\ref{cond:convex-model}--\\ref{cond:lower-model} is convergent.\nAssumptions~\\ref{assumption:convex}\nand~\\ref{assumption:very-weak-moment} are insufficient to guarantee\nconvergence of subgradient methods, which---as our examples show---may\ndiverge without uniform boundedness conditions on the subgradients.  In\ncontrast, any \\aProx method that is stable in\nprobability~\\eqref{eqn:bounded-distance} guarantees convergence with\nprobability 1.  Throughout this section and\nSection~\\ref{sec:asymptotic-normality}, we without comment make the\nassumption that the stepsizes $\\stepsize_k$ satisfy the standard summability\nconditions\n\\begin{equation*}\n  \\stepsize_k > 0 ~ \\mbox{for~all~} k,\n  ~~~\n  \\sum_{k=1}^\\infty \\stepsize_k = \\infty,\n  ~~~ \\mbox{and} ~~~\n  \\sum_{k=1}^\\infty \\stepsize_k^2 < \\infty.\n\\end{equation*}\n","units":[{"raw_tex":"In this paper, we develop and analyze a family of model-based methods,\nmoving beyond naive stochastic gradient methods, for solving the stochastic\nconvex optimization problem\n\\begin{equation}\n  \\label{eqn:objective}\n  \\begin{split}\n    \\minimize ~ & F(x) = \\E_P[f(x;\\statrv)]\n    = \\int_\\statdomain f(x; \\statval) dP(\\statval) \\\\\n    \\subjectto ~ & x \\in \\mc{X}.\n  \\end{split}\n\\end{equation}\nIn problem~\\eqref{eqn:objective}, the set $\\statdomain$ is a sample space,\nand for each $\\statval \\in \\statdomain$, the function $f(\\cdot; \\statval) :\n\\R^n \\to \\R$ is a closed convex function, subdifferentiable\non the closed convex set $\\mc{X} \\subset \\R^n$.\n","start":3159,"end":3810},{"raw_tex":"We show how to extend this modeling perspective to stochastic convex\noptimization problems, leveraging it to build a new family of algorithms for\nsolving problem~\\eqref{eqn:objective}, which, in homage to the stochastic\nproximal point iteration~\\eqref{eqn:sppm}, we call the \\aProx (approximate\nproximal point) algorithms, with substantially better theoretical guarantees\nand empirical performance than naive stochastic subgradient methods. The\n\\aProx algorithms iterate as follows: for $k = 1, 2, \\ldots$, we draw a\nrandom $\\statrv_k \\simiid P$, then update the iterate $x_k$ by minimizing a\nregularized approximation to $f(\\cdot; \\statrv_k)$, setting\n\\begin{equation}\n  \\label{eqn:model-iteration}\n  x_{k+1} \\defeq \\argmin_{x \\in \\mc{X}}\n  \\left\\{ f_{x_k}(x ; \\statrv_k) + \\frac{1}{2 \\stepsize_k}\n  \\ltwo{x - x_k}^2 \\right\\}.\n\\end{equation}\nThe function $f_x(\\cdot ; \\statval)$ is a \\emph{model} of $f(\\cdot;\n\\statval)$ at the point $x$, meaning that $f_x$ satisfies the\nfollowing conditions on its structure and local approximation\nproperties for $f$:\n\\begin{enumerate}[label=(C.\\roman*),leftmargin=*]\n\\item \\label{cond:convex-model}\n  The function $y \\mapsto f_x(y; \\statval)$ is convex\n  and subdifferentiable on $\\mc{X}$.\n\\item \\label{cond:lower-model}\n  The model $f_x$ satisfies the equality\n  $f_x(x; \\statval) = f(x; \\statval)$ and\n  \\begin{equation*}\n    f_x(y; \\statval) \\le f(y; \\statval) ~~ \\mbox{for~all~} y.\n  \\end{equation*}\n%\\item \\label{cond:subgrad-model}\n%  At $y = x$, we have the containment\n%  \\begin{equation*}\n%    \\left.\\partial_y f_x(y; \\statval)\\right|_{y = x}\n%    \\subset \\partial_x f(x; \\statval).\n%  \\end{equation*}\n\\end{enumerate}\n\\noindent\nBy the first-order conditions for convexity, for any $g \\in \\partial_y\nf_x(y; \\statval)|_{y=x}$ Conditions~\\ref{cond:convex-model}\nand~\\ref{cond:lower-model} imply $f(y; \\statval) \\ge f_x(y; \\statval) \\ge\nf_x(x; \\statval) + \\<g, y - x\\> = f(x; \\statval) + \\<g, y - x\\>$, yielding\nthe containment\n\\begin{equation}\n  \\label{eqn:subgrad-containment}\n  \\left.\\partial_y f_x(y; \\statval)\\right|_{y = x}\n  \\subset \\partial_x f(x; \\statval).\n\\end{equation}\n","start":7143,"end":9268},{"raw_tex":"\\begin{assumption}\n  \\label{assumption:convex}\n  The set $\\mc{X}\\opt \\defeq \\argmin_{x \\in \\mc{X}} \\{F(x)\\}$ is non-empty,\n  and there exists $\\sigma^2 < \\infty$\n  such that $\\E[\\ltwo{f'(x\\opt;\\statrv)}^2] \\leq \\sigma^2$\n  for $x\\opt \\in \\mc{X}\\opt$ \n  and all measurable selections $f'(x\\opt; \\statval) \\in \\partial\n  f(x\\opt; \\statval)$.\n\\end{assumption}\n\n\\begin{assumption}\n  \\label{assumption:very-weak-moment}\n  There exists a non-decreasing function $\\growfunc : \\R_+ \\to\n  \\openright{0}{\\infty}$ such that for all $x \\in\n  \\mc{X}$ and measurable selections $f'(x; \\statval) \\in \\partial\n  f(x; \\statval)$,\n  $\\E[\\norm{f'(x; \\statrv)}^2] \\le \\growfunc(\\dist(x, \\mc{X}\\opt))$.\n\\end{assumption}\n","start":10349,"end":11048},{"raw_tex":"\\paragraph{Notation}\nFor a convex function $f$, $\\partial f(x)$ denotes its subgradient set at\n$x$, and $f'(x) \\in \\partial f(x)$ denotes an arbitrary element of\nthe subdifferential.  Throughout, $x\\opt$ denotes a minimizer of\nproblem~\\eqref{eqn:objective} and $\\mc{X}\\opt = \\argmin_{x \\in \\mc{X}} F(x)$\nits optimal set.  We let $\\mc{F}_k \\defeq \\sigma(\\statrv_1, \\ldots,\n\\statrv_k)$ be the $\\sigma$-field generated by the first $k$ random\nvariables $\\statrv_i$, so $x_k \\in\n\\mc{F}_{k-1}$ for all $k$ under iteration~\\eqref{eqn:model-iteration}.\n","start":18567,"end":19113},{"raw_tex":"\\begin{lemma}[\\cite{RobbinsSi71}]\n  \\label{lemma:robbins-siegmund}\n  Let $A_k, B_k, C_k, D_k \\ge 0$ be non-negative random variables adapted to\n  the filtration $\\mc{F}_k$ and satisfying $\\E[A_{k + 1} \\mid \\mc{F}_k] \\le\n  (1 + B_k) A_k + C_k - D_k$. Then on the event $\\{\\sum_k B_k < \\infty,\n  \\sum_k C_k < \\infty\\}$, there is a random $A_\\infty < \\infty$\n  such that $A_k \\cas A_\\infty$ and $\\sum_k D_k < \\infty$.\n\\end{lemma}\n","start":38604,"end":39031},{"raw_tex":"The key consequence of Theorem~\\ref{theorem:prox-point-bounded} is that the\niterates $x_k$ are stable in probability~\\eqref{eqn:bounded-distance},\nremaining bounded with probability 1. Iterate boundedness of algorithms does\nnot guarantee convergence in general; however, in this section, we show that\nas a consequence of this boundedness, any algorithm satisfying\nConditions~\\ref{cond:convex-model}--\\ref{cond:lower-model} is convergent.\nAssumptions~\\ref{assumption:convex}\nand~\\ref{assumption:very-weak-moment} are insufficient to guarantee\nconvergence of subgradient methods, which---as our examples show---may\ndiverge without uniform boundedness conditions on the subgradients.  In\ncontrast, any \\aProx method that is stable in\nprobability~\\eqref{eqn:bounded-distance} guarantees convergence with\nprobability 1.  Throughout this section and\nSection~\\ref{sec:asymptotic-normality}, we without comment make the\nassumption that the stepsizes $\\stepsize_k$ satisfy the standard summability\nconditions\n\\begin{equation*}\n  \\stepsize_k > 0 ~ \\mbox{for~all~} k,\n  ~~~\n  \\sum_{k=1}^\\infty \\stepsize_k = \\infty,\n  ~~~ \\mbox{and} ~~~\n  \\sum_{k=1}^\\infty \\stepsize_k^2 < \\infty.\n\\end{equation*}\n","start":44997,"end":46183}]},"statement":{"raw_tex":"\\begin{proposition}\n  \\label{proposition:convergence-from-boundedness}\n  Let Assumptions~\\ref{assumption:convex} and\n  \\ref{assumption:very-weak-moment} hold. Let the iterates $x_k$ be\n  generated by any method satisfying\n  Conditions~\\ref{cond:convex-model}--\\ref{cond:lower-model}\n  and $F\\opt = \\inf_{x \\in \\mc{X}} F(x)$.\n  On the event that\n  $\\sup_k \\dist(x_k, \\mc{X}\\opt) < \\infty$, with probability\n  one both $\\sum_k \\stepsize_k (F(x_k) - F\\opt) < \\infty$,\n  and the iterates $x_k$ are convergent: there exists\n  $x\\opt \\in \\mc{X}\\opt$ such that\n  $\\norm{x_k - x\\opt} \\cas 0$ and\n  $F(x_k) \\cas F(x\\opt)$.\n\\end{proposition}","units":[{"raw_tex":"\\begin{proposition}\n  \\label{proposition:convergence-from-boundedness}\n  Let Assumptions~\\ref{assumption:convex} and\n  \\ref{assumption:very-weak-moment} hold. Let the iterates $x_k$ be\n  generated by any method satisfying\n  Conditions~\\ref{cond:convex-model}--\\ref{cond:lower-model}\n  and $F\\opt = \\inf_{x \\in \\mc{X}} F(x)$.\n  On the event that\n  $\\sup_k \\dist(x_k, \\mc{X}\\opt) < \\infty$, with probability\n  one both $\\sum_k \\stepsize_k (F(x_k) - F\\opt) < \\infty$,\n  and the iterates $x_k$ are convergent: there exists\n  $x\\opt \\in \\mc{X}\\opt$ such that\n  $\\norm{x_k - x\\opt} \\cas 0$ and\n  $F(x_k) \\cas F(x\\opt)$.\n\\end{proposition}","start":46183,"end":46814}]},"proof":{"raw_tex":"\\begin{lemma}\n  \\label{lemma:single-step-progress}\n  Let Condition~\\ref{cond:convex-model} hold.\n  In each step of the method~\\eqref{eqn:model-iteration},\n  for any $x \\in \\mc{X}$,\n  \\begin{equation*}\n    \\half \\ltwo{x_{k + 1} - x}^2\n    \\le \\half \\ltwo{x_k - x}^2\n    - \\stepsize_k\\left[f_{x_k}(x_{k + 1}; \\statrv_k)\n      - f_{x_k}(x; \\statrv_k)\\right]\n    - \\half \\ltwo{x_k - x_{k+1}}^2.\n  \\end{equation*}\n\\end{lemma}\n\n\\begin{proof}\n  By the first-order conditions for convex optimization,\n  for some $g_k \\in \\partial f_{x_k}(x_{k+1}; \\statrv_k)$ we have that\n  $\\<\\stepsize_k g_k + (x_{k+1} - x_k), y - x_{k+1}\\> \\ge 0$ for\n  all $y \\in \\mc{X}$.\n  Setting $y = x$, we obtain\n  \\begin{equation*}\n    \\stepsize_k \\<g_k, x_{k+1} - x\\>\n    \\le \\<x_{k + 1} - x_k, x - x_{k + 1}\\>\n    = \\half \\left[\\ltwo{x_k - x}^2\n      - \\ltwo{x_{k + 1} - x}^2\n      - \\ltwo{x_{k + 1} - x_k}^2 \\right].\n  \\end{equation*}\n  As $f_{x_k}(x; \\statrv_k) \\ge f_{x_k}(x_{k+1}; \\statrv_k) + \\<g_k,\n  x - x_{k+1}\\>$ by Condition~\\ref{cond:convex-model}, this gives the\n  result.\n\\end{proof}\n\n\\begin{lemma}\n  \\label{lemma:single-recentered-progress}\n  Let Conditions~\\ref{cond:convex-model}--\\ref{cond:lower-model} hold and\n  let $x_k$ be generated by the updates~\\eqref{eqn:model-iteration}.\n  Then for any $x \\in \\mc{X}$,\n  \\begin{equation*}\n    \\half \\ltwo{x_{k + 1} - x}^2\n    \\le \\half \\ltwo{x_k - x}^2\n    - \\stepsize_k [f(x_k; \\statrv_k) - f(x; \\statrv_k)]\n    + \\frac{\\stepsize_k^2}{2} \\ltwo{f'(x_k; \\statrv_k)}^2.\n  \\end{equation*}\n\\end{lemma}\n\n\\begin{proof}\n  Using Lemma~\\ref{lemma:single-step-progress}, it suffices to\n  show that for any $\\stepsize > 0$ and $x_0, x_1, x \\in \\mc{X}$\n  \\begin{equation*}\n    -\\stepsize [f_{x_0}(x_1; \\statval) - f_{x_0}(x; \\statval)]\n    - \\half \\ltwo{x_1 - x_0}^2\n    \\le -\\stepsize [f(x_0; \\statval) - f(x; \\statval)]\n    + \\frac{\\stepsize^2}{2} \\ltwo{f'(x_0; \\statval)}^2.\n  \\end{equation*}\n  To see this, recall that\n  Conditions~\\ref{cond:convex-model}--\\ref{cond:lower-model} imply the\n  containment~\\eqref{eqn:subgrad-containment}, which in turn implies\n  \\begin{align*}\n    -f_{x_0}(x_1; \\statval) + f_{x_0}(x; \\statval)\n    & = -[f_{x_0}(x_0; \\statval) - f_{x_0}(x; \\statval)]\n    + f_{x_0}(x_0; \\statval) - f_{x_0}(x_1; \\statval) \\\\\n    %& \\stackrel{\\ref{cond:subgrad-model}}{\\le}\n    & \\le\n    -[f_{x_0}(x_0; \\statval) - f_{x_0}(x; \\statval)]\n    + \\<f'(x_0;\\statval), x_0 - x_1\\> \\\\\n    & \\stackrel{\\ref{cond:lower-model}}{\\le}\n    -[f(x_0;\\statval) - f(x; \\statval)]\n    + \\<f'(x_0;\\statval), x_0 - x_1\\>.\n  \\end{align*}\n  Then we use that for any vector $v$,\n  $\\stepsize \\<v, \\Delta\\> - \\half \\ltwo{\\Delta}^2 \\le\n  \\frac{\\stepsize^2}{2} \\ltwo{v}^2$, which\n  gives the result.\n\\end{proof}\n\n\\begin{proof-of-proposition}[\\ref{proposition:convergence-from-boundedness}]\n  By Assumption~\\ref{assumption:very-weak-moment}, $\\E[\\norm{f'(x;\n      \\statrv)}^2] \\le \\growfunc(r)$ for all $x$ such that $\\dist(x,\n  \\mc{X}\\opt) \\le r$.  Lemma~\\ref{lemma:single-recentered-progress} implies\n  that for any $x\\opt \\in\\mc{X}\\opt$,\n  \\begin{align}\n    \\label{eqn:single-dingle}\n    \\E[\\norms{x_{k+1} - x\\opt}^2 \\mid \\mc{F}_{k-1}]\n    %% & \\le \\dist(x_k, \\mc{X}\\opt)^2 - 2 \\stepsize_k (F(x_k) - F\\opt)\n    %% + \\stepsize_k^2\n    %% \\sup_{\\norm{x - x\\opt} \\le \\dist(x_k, \\mc{X}\\opt)}\n    %% \\E[\\norms{f'(x; \\statrv)}^2] \\\\\n    & \\le \\norms{x_k - x\\opt}^2 - 2 \\stepsize_k (F(x_k) - F\\opt)\n    + \\stepsize_k^2\n    \\growfunc(\\dist(x_k, \\mc{X}\\opt)).\n  \\end{align}\n  On the event that $\\sup_k \\dist(x_k, \\mc{X}\\opt) < \\infty$, we have\n  $\\sum_k \\stepsize_k^2 \\growfunc(\\dist(x_k, \\mc{X}\\opt)) < \\infty$, and so\n  the Robbins-Siegmund Lemma~\\ref{lemma:robbins-siegmund} implies\n  for any $x\\opt \\in \\mc{X}\\opt$,\n  there is some (random) $V(x\\opt) < \\infty$ such that\n  $\\norms{x_k - x\\opt} \\cas V(x\\opt)$, and\n  $\\sum_k \\stepsize_k (F(x_k) - F(x\\opt)) < \\infty$.\n  \n  We now show that $\\dist(x_k, \\mc{X}\\opt) \\cas 0$.\n  Letting $x\\opt$ be the\n  projection of $x_k$ onto $\\mc{X}\\opt$,\n  inequality~\\eqref{eqn:single-dingle} gives $\\E[\\dist(x_{k+1},\n    \\mc{X}\\opt)^2 \\mid \\mc{F}_{k-1}] \\le \\dist(x_k, \\mc{X}\\opt)^2 - 2\n  \\stepsize_k(F(x_k) - F\\opt) + \\stepsize_k^2 \\growfunc(\\dist(x_k,\n  \\mc{X}\\opt))$. Thus using Lemma~\\ref{lemma:robbins-siegmund}, there exists\n  a random $D_\\infty < \\infty$ such that $\\dist(x_k, \\mc{X}\\opt) \\cas\n  D_\\infty$.\n  For $\\epsilon_i \\in \\R_+$, define the gap function\n  \\begin{equation*}\n    \\gapfunc(x\\opt, \\epsilon_1, \\epsilon_2)\n    \\defeq \\inf_{x \\in \\mc{X}} \\left\\{F(x) - F\\opt\n    \\mid \\epsilon_1 \\le \\norm{x - x\\opt} \\le 4 \\epsilon_1,\n    \\dist(x, \\mc{X}\\opt) \\ge \\epsilon_2\n    \\right\\}.\n  \\end{equation*}\n  The set $\\{x \\in \\mc{X} \\mid \\norm{x - x\\opt} \\in\n  [\\epsilon_1, 4\\epsilon_1], \\dist(x, \\mc{X}\\opt) \\ge \\epsilon_2\\}$ is\n  compact, so the infimum in the definition of the gap $\\gapfunc$ is\n  attained and $\\gapfunc(x\\opt, \\epsilon_1, \\epsilon_2) > 0$ for\n  $\\epsilon_i > 0$.  When the a.s.\\ limits\n  $V(x\\opt)$ and $D_\\infty$ are positive,\n  there exists a (random) $K <\\infty$ such that\n  $\\norm{x_k - x\\opt} \\in [V(x\\opt)/2, 2 V(x\\opt)]$ and\n  $\\dist(x_k, \\mc{X}\\opt) \\ge D_\\infty / 2$ for all\n  $k \\ge K$. Thus with probability 1,\n  \\begin{equation*}\n    \\infty > \\sum_k \\stepsize_k (F(x_k) - F\\opt)\n    %% \\ge \\sum_{k \\ge K}\n    %% \\stepsize_k \\inf_{x \\in \\mc{X}_\\epsilon} (F(x) - F\\opt)\n    \\ge \\sum_{k \\ge K} \\stepsize_k \\gapfunc(x\\opt, V(x\\opt) / 2, D_\\infty / 2).\n  \\end{equation*}\n  As $\\sum_k \\stepsize_k = \\infty$ and\n  $\\gapfunc(x\\opt, \\epsilon_1, \\epsilon_2) > 0$ for\n  $\\epsilon_i > 0$, we have $D_\\infty = 0$\n  with probability 1.\n\n  Finally, we show the sequence $x_k$ converges.\n  We begin by noting\n  that $V(x\\opt) = \\lim_k \\norm{x_k - x\\opt}$ is $1$-Lipschitz.\n  Indeed, let $x_1\\opt, x_2\\opt \\in \\mc{X}\\opt$. By\n  the a.s.\\ limit definition of $V$,\n  \\begin{equation*}\n    \\left|V(x_1\\opt) - V(x_2\\opt)\\right|\n    \\le\n    % \\limsup_k \\left|\\norm{x_k - x_1\\opt} - V(x_1\\opt)\\right|\n    % + \\limsup_k \\left|\\norm{x_k - x_2\\opt} - V(x_2\\opt)\\right|\n    \\limsup_k \\left|\\norm{x_k - x_1\\opt} - \\norm{x_k - x_2\\opt}\\right|\n    \\le \\norm{x_1\\opt - x_2\\opt}.\n  \\end{equation*}\n  Let us now show that for some $x\\opt \\in \\mc{X}\\opt$, we have $V(x\\opt) =\n  0$.  Let $\\ball = \\{x \\in \\R^n \\mid \\norm{x} \\le 1\\}$ be the ball.  As\n  $V(x\\opt) < \\infty$, there exists a (random) $r < \\infty$ such that the\n  $x_k \\in r \\ball$ for all $k$. As $\\dist(x_k, \\mc{X}\\opt) \\cas 0$, the\n  compactness of $\\ball$ implies that $r \\ball \\cap \\mc{X}\\opt \\neq\n  \\emptyset$.  Let $x\\opt \\in r\\ball \\cap \\mc{X}\\opt$.  For any $x \\in\n  r\\ball$, the projection $\\pi(x)$ of $x$ onto $\\mc{X}\\opt$ satisfies\n  $\\norm{\\pi(x) - x} \\le \\norm{x\\opt - x} \\le 2r$, and so $\\norm{\\pi(x)} \\le\n  3r$, and defining the set $C = 3r \\ball$,\n  we have $\\dist(x_k, \\mc{X}\\opt) = \\dist(x_k, \\mc{X}\\opt \\cap C)$ for all $k$.\n  Now, fix $\\epsilon > 0$, and let $\\{x_i\\opt\\}_{i = 1}^N$ be\n  an $\\epsilon$-net of $C \\cap \\mc{X}\\opt$, where $N < \\infty$.\n  As $x_k \\in C$ for all $k$,\n  \\begin{equation*}\n    \\min_{i \\in [N]}\n    \\norm{x_k - x_i\\opt} - \\epsilon\n    \\le \\dist(x_k, \\mc{X}\\opt \\cap C)\n    = \\dist(x_k, \\mc{X}\\opt) \\cas 0,\n  \\end{equation*}\n  and $\\min_{i \\in [N]} \\norm{x_k - x_i\\opt} \\cas \\min_{i \\in [N]}\n  V(x_i\\opt)$. Thus for any $\\epsilon > 0$,\n  there exists $x_\\epsilon \\in \\mc{X}\\opt \\cap C$ such that\n  $V(x_\\epsilon) \\le \\epsilon$, so that\n  $\\inf_{x \\in C \\cap \\mc{X}\\opt} V(x) = 0$. By the continuity of $V$,\n  the infimum is attained and there is\n  $x\\opt$ such that $V(x\\opt) = 0$. The sequence $x_k$ thus converges to\n  this $x\\opt \\in \\mc{X}\\opt$,\n  and $F(x_k) \\cas F(x\\opt) = F\\opt$.\n\\end{proof-of-proposition}","units":[{"raw_tex":"\\begin{lemma}\n  \\label{lemma:single-step-progress}\n  Let Condition~\\ref{cond:convex-model} hold.\n  In each step of the method~\\eqref{eqn:model-iteration},\n  for any $x \\in \\mc{X}$,\n  \\begin{equation*}\n    \\half \\ltwo{x_{k + 1} - x}^2\n    \\le \\half \\ltwo{x_k - x}^2\n    - \\stepsize_k\\left[f_{x_k}(x_{k + 1}; \\statrv_k)\n      - f_{x_k}(x; \\statrv_k)\\right]\n    - \\half \\ltwo{x_k - x_{k+1}}^2.\n  \\end{equation*}\n\\end{lemma}","start":41493,"end":41913},{"raw_tex":"\\begin{proof}\n  By the first-order conditions for convex optimization,\n  for some $g_k \\in \\partial f_{x_k}(x_{k+1}; \\statrv_k)$ we have that\n  $\\<\\stepsize_k g_k + (x_{k+1} - x_k), y - x_{k+1}\\> \\ge 0$ for\n  all $y \\in \\mc{X}$.\n  Setting $y = x$, we obtain\n  \\begin{equation*}\n    \\stepsize_k \\<g_k, x_{k+1} - x\\>\n    \\le \\<x_{k + 1} - x_k, x - x_{k + 1}\\>\n    = \\half \\left[\\ltwo{x_k - x}^2\n      - \\ltwo{x_{k + 1} - x}^2\n      - \\ltwo{x_{k + 1} - x_k}^2 \\right].\n  \\end{equation*}\n  As $f_{x_k}(x; \\statrv_k) \\ge f_{x_k}(x_{k+1}; \\statrv_k) + \\<g_k,\n  x - x_{k+1}\\>$ by Condition~\\ref{cond:convex-model}, this gives the\n  result.\n\\end{proof}","start":41914,"end":42558},{"raw_tex":"\\begin{lemma}\n  \\label{lemma:single-recentered-progress}\n  Let Conditions~\\ref{cond:convex-model}--\\ref{cond:lower-model} hold and\n  let $x_k$ be generated by the updates~\\eqref{eqn:model-iteration}.\n  Then for any $x \\in \\mc{X}$,\n  \\begin{equation*}\n    \\half \\ltwo{x_{k + 1} - x}^2\n    \\le \\half \\ltwo{x_k - x}^2\n    - \\stepsize_k [f(x_k; \\statrv_k) - f(x; \\statrv_k)]\n    + \\frac{\\stepsize_k^2}{2} \\ltwo{f'(x_k; \\statrv_k)}^2.\n  \\end{equation*}\n\\end{lemma}","start":48263,"end":48722},{"raw_tex":"\\begin{proof}\n  Using Lemma~\\ref{lemma:single-step-progress}, it suffices to\n  show that for any $\\stepsize > 0$ and $x_0, x_1, x \\in \\mc{X}$\n  \\begin{equation*}\n    -\\stepsize [f_{x_0}(x_1; \\statval) - f_{x_0}(x; \\statval)]\n    - \\half \\ltwo{x_1 - x_0}^2\n    \\le -\\stepsize [f(x_0; \\statval) - f(x; \\statval)]\n    + \\frac{\\stepsize^2}{2} \\ltwo{f'(x_0; \\statval)}^2.\n  \\end{equation*}\n  To see this, recall that\n  Conditions~\\ref{cond:convex-model}--\\ref{cond:lower-model} imply the\n  containment~\\eqref{eqn:subgrad-containment}, which in turn implies\n  \\begin{align*}\n    -f_{x_0}(x_1; \\statval) + f_{x_0}(x; \\statval)\n    & = -[f_{x_0}(x_0; \\statval) - f_{x_0}(x; \\statval)]\n    + f_{x_0}(x_0; \\statval) - f_{x_0}(x_1; \\statval) \\\\\n    %& \\stackrel{\\ref{cond:subgrad-model}}{\\le}\n    & \\le\n    -[f_{x_0}(x_0; \\statval) - f_{x_0}(x; \\statval)]\n    + \\<f'(x_0;\\statval), x_0 - x_1\\> \\\\\n    & \\stackrel{\\ref{cond:lower-model}}{\\le}\n    -[f(x_0;\\statval) - f(x; \\statval)]\n    + \\<f'(x_0;\\statval), x_0 - x_1\\>.\n  \\end{align*}\n  Then we use that for any vector $v$,\n  $\\stepsize \\<v, \\Delta\\> - \\half \\ltwo{\\Delta}^2 \\le\n  \\frac{\\stepsize^2}{2} \\ltwo{v}^2$, which\n  gives the result.\n\\end{proof}","start":48723,"end":49916},{"raw_tex":"\\begin{proof-of-proposition}[\\ref{proposition:convergence-from-boundedness}]\n  By Assumption~\\ref{assumption:very-weak-moment}, $\\E[\\norm{f'(x;\n      \\statrv)}^2] \\le \\growfunc(r)$ for all $x$ such that $\\dist(x,\n  \\mc{X}\\opt) \\le r$.  Lemma~\\ref{lemma:single-recentered-progress} implies\n  that for any $x\\opt \\in\\mc{X}\\opt$,\n  \\begin{align}\n    \\label{eqn:single-dingle}\n    \\E[\\norms{x_{k+1} - x\\opt}^2 \\mid \\mc{F}_{k-1}]\n    %% & \\le \\dist(x_k, \\mc{X}\\opt)^2 - 2 \\stepsize_k (F(x_k) - F\\opt)\n    %% + \\stepsize_k^2\n    %% \\sup_{\\norm{x - x\\opt} \\le \\dist(x_k, \\mc{X}\\opt)}\n    %% \\E[\\norms{f'(x; \\statrv)}^2] \\\\\n    & \\le \\norms{x_k - x\\opt}^2 - 2 \\stepsize_k (F(x_k) - F\\opt)\n    + \\stepsize_k^2\n    \\growfunc(\\dist(x_k, \\mc{X}\\opt)).\n  \\end{align}\n  On the event that $\\sup_k \\dist(x_k, \\mc{X}\\opt) < \\infty$, we have\n  $\\sum_k \\stepsize_k^2 \\growfunc(\\dist(x_k, \\mc{X}\\opt)) < \\infty$, and so\n  the Robbins-Siegmund Lemma~\\ref{lemma:robbins-siegmund} implies\n  for any $x\\opt \\in \\mc{X}\\opt$,\n  there is some (random) $V(x\\opt) < \\infty$ such that\n  $\\norms{x_k - x\\opt} \\cas V(x\\opt)$, and\n  $\\sum_k \\stepsize_k (F(x_k) - F(x\\opt)) < \\infty$.\n  \n  We now show that $\\dist(x_k, \\mc{X}\\opt) \\cas 0$.\n  Letting $x\\opt$ be the\n  projection of $x_k$ onto $\\mc{X}\\opt$,\n  inequality~\\eqref{eqn:single-dingle} gives $\\E[\\dist(x_{k+1},\n    \\mc{X}\\opt)^2 \\mid \\mc{F}_{k-1}] \\le \\dist(x_k, \\mc{X}\\opt)^2 - 2\n  \\stepsize_k(F(x_k) - F\\opt) + \\stepsize_k^2 \\growfunc(\\dist(x_k,\n  \\mc{X}\\opt))$. Thus using Lemma~\\ref{lemma:robbins-siegmund}, there exists\n  a random $D_\\infty < \\infty$ such that $\\dist(x_k, \\mc{X}\\opt) \\cas\n  D_\\infty$.\n  For $\\epsilon_i \\in \\R_+$, define the gap function\n  \\begin{equation*}\n    \\gapfunc(x\\opt, \\epsilon_1, \\epsilon_2)\n    \\defeq \\inf_{x \\in \\mc{X}} \\left\\{F(x) - F\\opt\n    \\mid \\epsilon_1 \\le \\norm{x - x\\opt} \\le 4 \\epsilon_1,\n    \\dist(x, \\mc{X}\\opt) \\ge \\epsilon_2\n    \\right\\}.\n  \\end{equation*}\n  The set $\\{x \\in \\mc{X} \\mid \\norm{x - x\\opt} \\in\n  [\\epsilon_1, 4\\epsilon_1], \\dist(x, \\mc{X}\\opt) \\ge \\epsilon_2\\}$ is\n  compact, so the infimum in the definition of the gap $\\gapfunc$ is\n  attained and $\\gapfunc(x\\opt, \\epsilon_1, \\epsilon_2) > 0$ for\n  $\\epsilon_i > 0$.  When the a.s.\\ limits\n  $V(x\\opt)$ and $D_\\infty$ are positive,\n  there exists a (random) $K <\\infty$ such that\n  $\\norm{x_k - x\\opt} \\in [V(x\\opt)/2, 2 V(x\\opt)]$ and\n  $\\dist(x_k, \\mc{X}\\opt) \\ge D_\\infty / 2$ for all\n  $k \\ge K$. Thus with probability 1,\n  \\begin{equation*}\n    \\infty > \\sum_k \\stepsize_k (F(x_k) - F\\opt)\n    %% \\ge \\sum_{k \\ge K}\n    %% \\stepsize_k \\inf_{x \\in \\mc{X}_\\epsilon} (F(x) - F\\opt)\n    \\ge \\sum_{k \\ge K} \\stepsize_k \\gapfunc(x\\opt, V(x\\opt) / 2, D_\\infty / 2).\n  \\end{equation*}\n  As $\\sum_k \\stepsize_k = \\infty$ and\n  $\\gapfunc(x\\opt, \\epsilon_1, \\epsilon_2) > 0$ for\n  $\\epsilon_i > 0$, we have $D_\\infty = 0$\n  with probability 1.\n\n  Finally, we show the sequence $x_k$ converges.\n  We begin by noting\n  that $V(x\\opt) = \\lim_k \\norm{x_k - x\\opt}$ is $1$-Lipschitz.\n  Indeed, let $x_1\\opt, x_2\\opt \\in \\mc{X}\\opt$. By\n  the a.s.\\ limit definition of $V$,\n  \\begin{equation*}\n    \\left|V(x_1\\opt) - V(x_2\\opt)\\right|\n    \\le\n    % \\limsup_k \\left|\\norm{x_k - x_1\\opt} - V(x_1\\opt)\\right|\n    % + \\limsup_k \\left|\\norm{x_k - x_2\\opt} - V(x_2\\opt)\\right|\n    \\limsup_k \\left|\\norm{x_k - x_1\\opt} - \\norm{x_k - x_2\\opt}\\right|\n    \\le \\norm{x_1\\opt - x_2\\opt}.\n  \\end{equation*}\n  Let us now show that for some $x\\opt \\in \\mc{X}\\opt$, we have $V(x\\opt) =\n  0$.  Let $\\ball = \\{x \\in \\R^n \\mid \\norm{x} \\le 1\\}$ be the ball.  As\n  $V(x\\opt) < \\infty$, there exists a (random) $r < \\infty$ such that the\n  $x_k \\in r \\ball$ for all $k$. As $\\dist(x_k, \\mc{X}\\opt) \\cas 0$, the\n  compactness of $\\ball$ implies that $r \\ball \\cap \\mc{X}\\opt \\neq\n  \\emptyset$.  Let $x\\opt \\in r\\ball \\cap \\mc{X}\\opt$.  For any $x \\in\n  r\\ball$, the projection $\\pi(x)$ of $x$ onto $\\mc{X}\\opt$ satisfies\n  $\\norm{\\pi(x) - x} \\le \\norm{x\\opt - x} \\le 2r$, and so $\\norm{\\pi(x)} \\le\n  3r$, and defining the set $C = 3r \\ball$,\n  we have $\\dist(x_k, \\mc{X}\\opt) = \\dist(x_k, \\mc{X}\\opt \\cap C)$ for all $k$.\n  Now, fix $\\epsilon > 0$, and let $\\{x_i\\opt\\}_{i = 1}^N$ be\n  an $\\epsilon$-net of $C \\cap \\mc{X}\\opt$, where $N < \\infty$.\n  As $x_k \\in C$ for all $k$,\n  \\begin{equation*}\n    \\min_{i \\in [N]}\n    \\norm{x_k - x_i\\opt} - \\epsilon\n    \\le \\dist(x_k, \\mc{X}\\opt \\cap C)\n    = \\dist(x_k, \\mc{X}\\opt) \\cas 0,\n  \\end{equation*}\n  and $\\min_{i \\in [N]} \\norm{x_k - x_i\\opt} \\cas \\min_{i \\in [N]}\n  V(x_i\\opt)$. Thus for any $\\epsilon > 0$,\n  there exists $x_\\epsilon \\in \\mc{X}\\opt \\cap C$ such that\n  $V(x_\\epsilon) \\le \\epsilon$, so that\n  $\\inf_{x \\in C \\cap \\mc{X}\\opt} V(x) = 0$. By the continuity of $V$,\n  the infimum is attained and there is\n  $x\\opt$ such that $V(x\\opt) = 0$. The sequence $x_k$ thus converges to\n  this $x\\opt \\in \\mc{X}\\opt$,\n  and $F(x_k) \\cas F(x\\opt) = F\\opt$.\n\\end{proof-of-proposition}","start":49918,"end":54863}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eqn:objective"],"declared_external":false,"source_only":false,"title":"随机凸优化问题","source_units":[{"raw_tex":"In this paper, we develop and analyze a family of model-based methods,\nmoving beyond naive stochastic gradient methods, for solving the stochastic\nconvex optimization problem\n\\begin{equation}\n  \\label{eqn:objective}\n  \\begin{split}\n    \\minimize ~ & F(x) = \\E_P[f(x;\\statrv)]\n    = \\int_\\statdomain f(x; \\statval) dP(\\statval) \\\\\n    \\subjectto ~ & x \\in \\mc{X}.\n  \\end{split}\n\\end{equation}\n","start":3159,"end":3551}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"有限维闭凸设定","source_units":[{"raw_tex":"In problem~\\eqref{eqn:objective}, the set $\\statdomain$ is a sample space,\nand for each $\\statval \\in \\statdomain$, the function $f(\\cdot; \\statval) :\n\\R^n \\to \\R$ is a closed convex function, subdifferentiable\non the closed convex set $\\mc{X} \\subset \\R^n$.\n","start":3551,"end":3810}]},{"id":"S003","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"模型方法的引入","source_units":[{"raw_tex":"We show how to extend this modeling perspective to stochastic convex\noptimization problems, leveraging it to build a new family of algorithms for\nsolving problem~\\eqref{eqn:objective}, which, in homage to the stochastic\nproximal point iteration~\\eqref{eqn:sppm}, we call the \\aProx (approximate\nproximal point) algorithms, with substantially better theoretical guarantees\nand empirical performance than naive stochastic subgradient methods. The\n","start":7143,"end":7588}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"独立同分布采样","source_units":[{"raw_tex":"\\aProx algorithms iterate as follows: for $k = 1, 2, \\ldots$, we draw a\nrandom $\\statrv_k \\simiid P$, then update the iterate $x_k$ by minimizing a\nregularized approximation to $f(\\cdot; \\statrv_k)$, setting\n","start":7588,"end":7796}]},{"id":"S005","semantic_type":"algorithm","kind":"algorithm","labels":["eqn:model-iteration"],"declared_external":false,"source_only":false,"title":"正则化模型更新","source_units":[{"raw_tex":"\\begin{equation}\n  \\label{eqn:model-iteration}\n  x_{k+1} \\defeq \\argmin_{x \\in \\mc{X}}\n  \\left\\{ f_{x_k}(x ; \\statrv_k) + \\frac{1}{2 \\stepsize_k}\n  \\ltwo{x - x_k}^2 \\right\\}.\n\\end{equation}\n","start":7796,"end":7986}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"中心模型记号","source_units":[{"raw_tex":"The function $f_x(\\cdot ; \\statval)$ is a \\emph{model} of $f(\\cdot;\n\\statval)$ at the point $x$, meaning that $f_x$ satisfies the\nfollowing conditions on its structure and local approximation\nproperties for $f$:\n\\begin{enumerate}[label=(C.\\roman*),leftmargin=*]\n","start":7986,"end":8248}]},{"id":"S007","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"模型凸性条件","source_units":[{"raw_tex":"\\item \\label{cond:convex-model}\n  The function $y \\mapsto f_x(y; \\statval)$ is convex\n  and subdifferentiable on $\\mc{X}$.\n","start":8248,"end":8371}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"相切下模型条件","source_units":[{"raw_tex":"\\item \\label{cond:lower-model}\n  The model $f_x$ satisfies the equality\n  $f_x(x; \\statval) = f(x; \\statval)$ and\n  \\begin{equation*}\n    f_x(y; \\statval) \\le f(y; \\statval) ~~ \\mbox{for~all~} y.\n  \\end{equation*}\n","start":8371,"end":8585}]},{"id":"S009","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"已注释的旧条件","source_units":[{"raw_tex":"%\\item \\label{cond:subgrad-model}\n%  At $y = x$, we have the containment\n%  \\begin{equation*}\n%    \\left.\\partial_y f_x(y; \\statval)\\right|_{y = x}\n%    \\subset \\partial_x f(x; \\statval).\n%  \\end{equation*}\n\\end{enumerate}\n","start":8585,"end":8808}]},{"id":"S010","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:subgrad-containment"],"declared_external":false,"source_only":false,"title":"中心次梯度包含","source_units":[{"raw_tex":"\\noindent\nBy the first-order conditions for convexity, for any $g \\in \\partial_y\nf_x(y; \\statval)|_{y=x}$ Conditions~\\ref{cond:convex-model}\nand~\\ref{cond:lower-model} imply $f(y; \\statval) \\ge f_x(y; \\statval) \\ge\nf_x(x; \\statval) + \\<g, y - x\\> = f(x; \\statval) + \\<g, y - x\\>$, yielding\nthe containment\n\\begin{equation}\n  \\label{eqn:subgrad-containment}\n  \\left.\\partial_y f_x(y; \\statval)\\right|_{y = x}\n  \\subset \\partial_x f(x; \\statval).\n\\end{equation}\n","start":8808,"end":9268}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","labels":["assumption:convex"],"declared_external":false,"source_only":false,"title":"最优解与最优点矩条件","source_units":[{"raw_tex":"\\begin{assumption}\n  \\label{assumption:convex}\n  The set $\\mc{X}\\opt \\defeq \\argmin_{x \\in \\mc{X}} \\{F(x)\\}$ is non-empty,\n  and there exists $\\sigma^2 < \\infty$\n  such that $\\E[\\ltwo{f'(x\\opt;\\statrv)}^2] \\leq \\sigma^2$\n  for $x\\opt \\in \\mc{X}\\opt$ \n  and all measurable selections $f'(x\\opt; \\statval) \\in \\partial\n  f(x\\opt; \\statval)$.\n\\end{assumption}\n\n","start":10349,"end":10707}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["assumption:convex"],"declared_external":false,"source_only":false,"title":"非空最优集","source_units":[{"raw_tex":"  The set $\\mc{X}\\opt \\defeq \\argmin_{x \\in \\mc{X}} \\{F(x)\\}$ is non-empty,\n","start":10396,"end":10472}]},{"id":"S013","semantic_type":"assumption","kind":"assumption","labels":["assumption:very-weak-moment"],"declared_external":false,"source_only":false,"title":"距离控制的弱矩条件","source_units":[{"raw_tex":"\\begin{assumption}\n  \\label{assumption:very-weak-moment}\n  There exists a non-decreasing function $\\growfunc : \\R_+ \\to\n  \\openright{0}{\\infty}$ such that for all $x \\in\n  \\mc{X}$ and measurable selections $f'(x; \\statval) \\in \\partial\n  f(x; \\statval)$,\n  $\\E[\\norm{f'(x; \\statrv)}^2] \\le \\growfunc(\\dist(x, \\mc{X}\\opt))$.\n\\end{assumption}\n","start":10707,"end":11048}]},{"id":"S014","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"次梯度与最优点记号","source_units":[{"raw_tex":"\\paragraph{Notation}\nFor a convex function $f$, $\\partial f(x)$ denotes its subgradient set at\n$x$, and $f'(x) \\in \\partial f(x)$ denotes an arbitrary element of\nthe subdifferential.  Throughout, $x\\opt$ denotes a minimizer of\nproblem~\\eqref{eqn:objective} and $\\mc{X}\\opt = \\argmin_{x \\in \\mc{X}} F(x)$\n","start":18567,"end":18871}]},{"id":"S015","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"滤过与可预测迭代","source_units":[{"raw_tex":"its optimal set.  We let $\\mc{F}_k \\defeq \\sigma(\\statrv_1, \\ldots,\n\\statrv_k)$ be the $\\sigma$-field generated by the first $k$ random\nvariables $\\statrv_i$, so $x_k \\in\n\\mc{F}_{k-1}$ for all $k$ under iteration~\\eqref{eqn:model-iteration}.\n","start":18871,"end":19113}]},{"id":"S016","semantic_type":"external_fact","kind":"external_fact","labels":["lemma:robbins-siegmund"],"declared_external":true,"declaration_text":["原始许可前提：Robbins–Siegmund almost-supermartingale convergence theorem，RobbinsSi71，按 B0039 所述版本使用。"],"source_only":false,"title":"几乎超鞅收敛引理","source_units":[{"raw_tex":"\\begin{lemma}[\\cite{RobbinsSi71}]\n  \\label{lemma:robbins-siegmund}\n  Let $A_k, B_k, C_k, D_k \\ge 0$ be non-negative random variables adapted to\n  the filtration $\\mc{F}_k$ and satisfying $\\E[A_{k + 1} \\mid \\mc{F}_k] \\le\n  (1 + B_k) A_k + C_k - D_k$. Then on the event $\\{\\sum_k B_k < \\infty,\n  \\sum_k C_k < \\infty\\}$, there is a random $A_\\infty < \\infty$\n  such that $A_k \\cas A_\\infty$ and $\\sum_k D_k < \\infty$.\n\\end{lemma}\n","start":38604,"end":39031}]},{"id":"S017","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"稳定性带来的收敛意义","source_units":[{"raw_tex":"The key consequence of Theorem~\\ref{theorem:prox-point-bounded} is that the\niterates $x_k$ are stable in probability~\\eqref{eqn:bounded-distance},\nremaining bounded with probability 1. Iterate boundedness of algorithms does\nnot guarantee convergence in general; however, in this section, we show that\nas a consequence of this boundedness, any algorithm satisfying\nConditions~\\ref{cond:convex-model}--\\ref{cond:lower-model} is convergent.\nAssumptions~\\ref{assumption:convex}\nand~\\ref{assumption:very-weak-moment} are insufficient to guarantee\nconvergence of subgradient methods, which---as our examples show---may\ndiverge without uniform boundedness conditions on the subgradients.  In\ncontrast, any \\aProx method that is stable in\nprobability~\\eqref{eqn:bounded-distance} guarantees convergence with\nprobability 1.  Throughout this section and\n","start":44997,"end":45841}]},{"id":"S018","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"步长求和条件","source_units":[{"raw_tex":"Section~\\ref{sec:asymptotic-normality}, we without comment make the\nassumption that the stepsizes $\\stepsize_k$ satisfy the standard summability\nconditions\n\\begin{equation*}\n  \\stepsize_k > 0 ~ \\mbox{for~all~} k,\n  ~~~\n  \\sum_{k=1}^\\infty \\stepsize_k = \\infty,\n  ~~~ \\mbox{and} ~~~\n  \\sum_{k=1}^\\infty \\stepsize_k^2 < \\infty.\n\\end{equation*}\n","start":45841,"end":46183}]},{"id":"S019","semantic_type":"assumption","kind":"assumption","labels":["proposition:convergence-from-boundedness"],"declared_external":false,"source_only":false,"title":"距离有界事件","source_units":[{"raw_tex":"  On the event that\n  $\\sup_k \\dist(x_k, \\mc{X}\\opt) < \\infty$, with probability\n","start":46508,"end":46589}]},{"id":"S020","semantic_type":"proposition","kind":"proposition","labels":["proposition:convergence-from-boundedness"],"declared_external":false,"source_only":false,"title":"由有界性推出几乎必然收敛","source_units":[{"raw_tex":"\\begin{proposition}\n  \\label{proposition:convergence-from-boundedness}\n  Let Assumptions~\\ref{assumption:convex} and\n  \\ref{assumption:very-weak-moment} hold. Let the iterates $x_k$ be\n  generated by any method satisfying\n  Conditions~\\ref{cond:convex-model}--\\ref{cond:lower-model}\n  and $F\\opt = \\inf_{x \\in \\mc{X}} F(x)$.\n  On the event that\n  $\\sup_k \\dist(x_k, \\mc{X}\\opt) < \\infty$, with probability\n  one both $\\sum_k \\stepsize_k (F(x_k) - F\\opt) < \\infty$,\n  and the iterates $x_k$ are convergent: there exists\n  $x\\opt \\in \\mc{X}\\opt$ such that\n  $\\norm{x_k - x\\opt} \\cas 0$ and\n  $F(x_k) \\cas F(x\\opt)$.\n\\end{proposition}","start":46183,"end":46814}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","labels":["lemma:single-step-progress"],"declared_external":false,"source_only":false,"title":"模型单步进展引理","source_units":[{"raw_tex":"\\begin{lemma}\n  \\label{lemma:single-step-progress}\n  Let Condition~\\ref{cond:convex-model} hold.\n  In each step of the method~\\eqref{eqn:model-iteration},\n  for any $x \\in \\mc{X}$,\n  \\begin{equation*}\n    \\half \\ltwo{x_{k + 1} - x}^2\n    \\le \\half \\ltwo{x_k - x}^2\n    - \\stepsize_k\\left[f_{x_k}(x_{k + 1}; \\statrv_k)\n      - f_{x_k}(x; \\statrv_k)\\right]\n    - \\half \\ltwo{x_k - x_{k+1}}^2.\n  \\end{equation*}\n\\end{lemma}","start":41493,"end":41913}]},{"id":"S022","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"模型单步进展完整证明","source_units":[{"raw_tex":"\\begin{proof}\n  By the first-order conditions for convex optimization,\n  for some $g_k \\in \\partial f_{x_k}(x_{k+1}; \\statrv_k)$ we have that\n  $\\<\\stepsize_k g_k + (x_{k+1} - x_k), y - x_{k+1}\\> \\ge 0$ for\n  all $y \\in \\mc{X}$.\n  Setting $y = x$, we obtain\n  \\begin{equation*}\n    \\stepsize_k \\<g_k, x_{k+1} - x\\>\n    \\le \\<x_{k + 1} - x_k, x - x_{k + 1}\\>\n    = \\half \\left[\\ltwo{x_k - x}^2\n      - \\ltwo{x_{k + 1} - x}^2\n      - \\ltwo{x_{k + 1} - x_k}^2 \\right].\n  \\end{equation*}\n  As $f_{x_k}(x; \\statrv_k) \\ge f_{x_k}(x_{k+1}; \\statrv_k) + \\<g_k,\n  x - x_{k+1}\\>$ by Condition~\\ref{cond:convex-model}, this gives the\n  result.\n\\end{proof}","start":41914,"end":42558}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"更新点的一阶最优性","source_units":[{"raw_tex":"\\begin{proof}\n  By the first-order conditions for convex optimization,\n  for some $g_k \\in \\partial f_{x_k}(x_{k+1}; \\statrv_k)$ we have that\n  $\\<\\stepsize_k g_k + (x_{k+1} - x_k), y - x_{k+1}\\> \\ge 0$ for\n  all $y \\in \\mc{X}$.\n","start":41914,"end":42143}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"展开三点距离","source_units":[{"raw_tex":"  Setting $y = x$, we obtain\n  \\begin{equation*}\n    \\stepsize_k \\<g_k, x_{k+1} - x\\>\n    \\le \\<x_{k + 1} - x_k, x - x_{k + 1}\\>\n    = \\half \\left[\\ltwo{x_k - x}^2\n      - \\ltwo{x_{k + 1} - x}^2\n      - \\ltwo{x_{k + 1} - x_k}^2 \\right].\n  \\end{equation*}\n","start":42143,"end":42398}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"以模型值差替换内积","source_units":[{"raw_tex":"  As $f_{x_k}(x; \\statrv_k) \\ge f_{x_k}(x_{k+1}; \\statrv_k) + \\<g_k,\n  x - x_{k+1}\\>$ by Condition~\\ref{cond:convex-model}, this gives the\n  result.\n\\end{proof}","start":42398,"end":42558}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","labels":["lemma:single-recentered-progress"],"declared_external":false,"source_only":false,"title":"中心化单步进展引理","source_units":[{"raw_tex":"\\begin{lemma}\n  \\label{lemma:single-recentered-progress}\n  Let Conditions~\\ref{cond:convex-model}--\\ref{cond:lower-model} hold and\n  let $x_k$ be generated by the updates~\\eqref{eqn:model-iteration}.\n  Then for any $x \\in \\mc{X}$,\n  \\begin{equation*}\n    \\half \\ltwo{x_{k + 1} - x}^2\n    \\le \\half \\ltwo{x_k - x}^2\n    - \\stepsize_k [f(x_k; \\statrv_k) - f(x; \\statrv_k)]\n    + \\frac{\\stepsize_k^2}{2} \\ltwo{f'(x_k; \\statrv_k)}^2.\n  \\end{equation*}\n\\end{lemma}","start":48263,"end":48722}]},{"id":"S027","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"中心化进展完整证明","source_units":[{"raw_tex":"\\begin{proof}\n  Using Lemma~\\ref{lemma:single-step-progress}, it suffices to\n  show that for any $\\stepsize > 0$ and $x_0, x_1, x \\in \\mc{X}$\n  \\begin{equation*}\n    -\\stepsize [f_{x_0}(x_1; \\statval) - f_{x_0}(x; \\statval)]\n    - \\half \\ltwo{x_1 - x_0}^2\n    \\le -\\stepsize [f(x_0; \\statval) - f(x; \\statval)]\n    + \\frac{\\stepsize^2}{2} \\ltwo{f'(x_0; \\statval)}^2.\n  \\end{equation*}\n  To see this, recall that\n  Conditions~\\ref{cond:convex-model}--\\ref{cond:lower-model} imply the\n  containment~\\eqref{eqn:subgrad-containment}, which in turn implies\n  \\begin{align*}\n    -f_{x_0}(x_1; \\statval) + f_{x_0}(x; \\statval)\n    & = -[f_{x_0}(x_0; \\statval) - f_{x_0}(x; \\statval)]\n    + f_{x_0}(x_0; \\statval) - f_{x_0}(x_1; \\statval) \\\\\n    %& \\stackrel{\\ref{cond:subgrad-model}}{\\le}\n    & \\le\n    -[f_{x_0}(x_0; \\statval) - f_{x_0}(x; \\statval)]\n    + \\<f'(x_0;\\statval), x_0 - x_1\\> \\\\\n    & \\stackrel{\\ref{cond:lower-model}}{\\le}\n    -[f(x_0;\\statval) - f(x; \\statval)]\n    + \\<f'(x_0;\\statval), x_0 - x_1\\>.\n  \\end{align*}\n  Then we use that for any vector $v$,\n  $\\stepsize \\<v, \\Delta\\> - \\half \\ltwo{\\Delta}^2 \\le\n  \\frac{\\stepsize^2}{2} \\ltwo{v}^2$, which\n  gives the result.\n\\end{proof}","start":48723,"end":49916}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"归约为模型误差估计","source_units":[{"raw_tex":"\\begin{proof}\n  Using Lemma~\\ref{lemma:single-step-progress}, it suffices to\n  show that for any $\\stepsize > 0$ and $x_0, x_1, x \\in \\mc{X}$\n  \\begin{equation*}\n    -\\stepsize [f_{x_0}(x_1; \\statval) - f_{x_0}(x; \\statval)]\n    - \\half \\ltwo{x_1 - x_0}^2\n    \\le -\\stepsize [f(x_0; \\statval) - f(x; \\statval)]\n    + \\frac{\\stepsize^2}{2} \\ltwo{f'(x_0; \\statval)}^2.\n  \\end{equation*}\n","start":48723,"end":49108}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"在模型中心重新展开","source_units":[{"raw_tex":"  To see this, recall that\n  Conditions~\\ref{cond:convex-model}--\\ref{cond:lower-model} imply the\n  containment~\\eqref{eqn:subgrad-containment}, which in turn implies\n  \\begin{align*}\n    -f_{x_0}(x_1; \\statval) + f_{x_0}(x; \\statval)\n    & = -[f_{x_0}(x_0; \\statval) - f_{x_0}(x; \\statval)]\n    + f_{x_0}(x_0; \\statval) - f_{x_0}(x_1; \\statval) \\\\\n    %& \\stackrel{\\ref{cond:subgrad-model}}{\\le}\n    & \\le\n    -[f_{x_0}(x_0; \\statval) - f_{x_0}(x; \\statval)]\n    + \\<f'(x_0;\\statval), x_0 - x_1\\> \\\\\n    & \\stackrel{\\ref{cond:lower-model}}{\\le}\n    -[f(x_0;\\statval) - f(x; \\statval)]\n    + \\<f'(x_0;\\statval), x_0 - x_1\\>.\n  \\end{align*}\n","start":49108,"end":49748}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"吸收二次步长项","source_units":[{"raw_tex":"  Then we use that for any vector $v$,\n  $\\stepsize \\<v, \\Delta\\> - \\half \\ltwo{\\Delta}^2 \\le\n  \\frac{\\stepsize^2}{2} \\ltwo{v}^2$, which\n  gives the result.\n\\end{proof}","start":49748,"end":49916}]},{"id":"S031","semantic_type":"proof","kind":"proof","labels":["eqn:single-dingle"],"declared_external":false,"source_only":false,"title":"有界性收敛命题完整证明","source_units":[{"raw_tex":"\\begin{proof-of-proposition}[\\ref{proposition:convergence-from-boundedness}]\n  By Assumption~\\ref{assumption:very-weak-moment}, $\\E[\\norm{f'(x;\n      \\statrv)}^2] \\le \\growfunc(r)$ for all $x$ such that $\\dist(x,\n  \\mc{X}\\opt) \\le r$.  Lemma~\\ref{lemma:single-recentered-progress} implies\n  that for any $x\\opt \\in\\mc{X}\\opt$,\n  \\begin{align}\n    \\label{eqn:single-dingle}\n    \\E[\\norms{x_{k+1} - x\\opt}^2 \\mid \\mc{F}_{k-1}]\n    %% & \\le \\dist(x_k, \\mc{X}\\opt)^2 - 2 \\stepsize_k (F(x_k) - F\\opt)\n    %% + \\stepsize_k^2\n    %% \\sup_{\\norm{x - x\\opt} \\le \\dist(x_k, \\mc{X}\\opt)}\n    %% \\E[\\norms{f'(x; \\statrv)}^2] \\\\\n    & \\le \\norms{x_k - x\\opt}^2 - 2 \\stepsize_k (F(x_k) - F\\opt)\n    + \\stepsize_k^2\n    \\growfunc(\\dist(x_k, \\mc{X}\\opt)).\n  \\end{align}\n  On the event that $\\sup_k \\dist(x_k, \\mc{X}\\opt) < \\infty$, we have\n  $\\sum_k \\stepsize_k^2 \\growfunc(\\dist(x_k, \\mc{X}\\opt)) < \\infty$, and so\n  the Robbins-Siegmund Lemma~\\ref{lemma:robbins-siegmund} implies\n  for any $x\\opt \\in \\mc{X}\\opt$,\n  there is some (random) $V(x\\opt) < \\infty$ such that\n  $\\norms{x_k - x\\opt} \\cas V(x\\opt)$, and\n  $\\sum_k \\stepsize_k (F(x_k) - F(x\\opt)) < \\infty$.\n  \n  We now show that $\\dist(x_k, \\mc{X}\\opt) \\cas 0$.\n  Letting $x\\opt$ be the\n  projection of $x_k$ onto $\\mc{X}\\opt$,\n  inequality~\\eqref{eqn:single-dingle} gives $\\E[\\dist(x_{k+1},\n    \\mc{X}\\opt)^2 \\mid \\mc{F}_{k-1}] \\le \\dist(x_k, \\mc{X}\\opt)^2 - 2\n  \\stepsize_k(F(x_k) - F\\opt) + \\stepsize_k^2 \\growfunc(\\dist(x_k,\n  \\mc{X}\\opt))$. Thus using Lemma~\\ref{lemma:robbins-siegmund}, there exists\n  a random $D_\\infty < \\infty$ such that $\\dist(x_k, \\mc{X}\\opt) \\cas\n  D_\\infty$.\n  For $\\epsilon_i \\in \\R_+$, define the gap function\n  \\begin{equation*}\n    \\gapfunc(x\\opt, \\epsilon_1, \\epsilon_2)\n    \\defeq \\inf_{x \\in \\mc{X}} \\left\\{F(x) - F\\opt\n    \\mid \\epsilon_1 \\le \\norm{x - x\\opt} \\le 4 \\epsilon_1,\n    \\dist(x, \\mc{X}\\opt) \\ge \\epsilon_2\n    \\right\\}.\n  \\end{equation*}\n  The set $\\{x \\in \\mc{X} \\mid \\norm{x - x\\opt} \\in\n  [\\epsilon_1, 4\\epsilon_1], \\dist(x, \\mc{X}\\opt) \\ge \\epsilon_2\\}$ is\n  compact, so the infimum in the definition of the gap $\\gapfunc$ is\n  attained and $\\gapfunc(x\\opt, \\epsilon_1, \\epsilon_2) > 0$ for\n  $\\epsilon_i > 0$.  When the a.s.\\ limits\n  $V(x\\opt)$ and $D_\\infty$ are positive,\n  there exists a (random) $K <\\infty$ such that\n  $\\norm{x_k - x\\opt} \\in [V(x\\opt)/2, 2 V(x\\opt)]$ and\n  $\\dist(x_k, \\mc{X}\\opt) \\ge D_\\infty / 2$ for all\n  $k \\ge K$. Thus with probability 1,\n  \\begin{equation*}\n    \\infty > \\sum_k \\stepsize_k (F(x_k) - F\\opt)\n    %% \\ge \\sum_{k \\ge K}\n    %% \\stepsize_k \\inf_{x \\in \\mc{X}_\\epsilon} (F(x) - F\\opt)\n    \\ge \\sum_{k \\ge K} \\stepsize_k \\gapfunc(x\\opt, V(x\\opt) / 2, D_\\infty / 2).\n  \\end{equation*}\n  As $\\sum_k \\stepsize_k = \\infty$ and\n  $\\gapfunc(x\\opt, \\epsilon_1, \\epsilon_2) > 0$ for\n  $\\epsilon_i > 0$, we have $D_\\infty = 0$\n  with probability 1.\n\n  Finally, we show the sequence $x_k$ converges.\n  We begin by noting\n  that $V(x\\opt) = \\lim_k \\norm{x_k - x\\opt}$ is $1$-Lipschitz.\n  Indeed, let $x_1\\opt, x_2\\opt \\in \\mc{X}\\opt$. By\n  the a.s.\\ limit definition of $V$,\n  \\begin{equation*}\n    \\left|V(x_1\\opt) - V(x_2\\opt)\\right|\n    \\le\n    % \\limsup_k \\left|\\norm{x_k - x_1\\opt} - V(x_1\\opt)\\right|\n    % + \\limsup_k \\left|\\norm{x_k - x_2\\opt} - V(x_2\\opt)\\right|\n    \\limsup_k \\left|\\norm{x_k - x_1\\opt} - \\norm{x_k - x_2\\opt}\\right|\n    \\le \\norm{x_1\\opt - x_2\\opt}.\n  \\end{equation*}\n  Let us now show that for some $x\\opt \\in \\mc{X}\\opt$, we have $V(x\\opt) =\n  0$.  Let $\\ball = \\{x \\in \\R^n \\mid \\norm{x} \\le 1\\}$ be the ball.  As\n  $V(x\\opt) < \\infty$, there exists a (random) $r < \\infty$ such that the\n  $x_k \\in r \\ball$ for all $k$. As $\\dist(x_k, \\mc{X}\\opt) \\cas 0$, the\n  compactness of $\\ball$ implies that $r \\ball \\cap \\mc{X}\\opt \\neq\n  \\emptyset$.  Let $x\\opt \\in r\\ball \\cap \\mc{X}\\opt$.  For any $x \\in\n  r\\ball$, the projection $\\pi(x)$ of $x$ onto $\\mc{X}\\opt$ satisfies\n  $\\norm{\\pi(x) - x} \\le \\norm{x\\opt - x} \\le 2r$, and so $\\norm{\\pi(x)} \\le\n  3r$, and defining the set $C = 3r \\ball$,\n  we have $\\dist(x_k, \\mc{X}\\opt) = \\dist(x_k, \\mc{X}\\opt \\cap C)$ for all $k$.\n  Now, fix $\\epsilon > 0$, and let $\\{x_i\\opt\\}_{i = 1}^N$ be\n  an $\\epsilon$-net of $C \\cap \\mc{X}\\opt$, where $N < \\infty$.\n  As $x_k \\in C$ for all $k$,\n  \\begin{equation*}\n    \\min_{i \\in [N]}\n    \\norm{x_k - x_i\\opt} - \\epsilon\n    \\le \\dist(x_k, \\mc{X}\\opt \\cap C)\n    = \\dist(x_k, \\mc{X}\\opt) \\cas 0,\n  \\end{equation*}\n  and $\\min_{i \\in [N]} \\norm{x_k - x_i\\opt} \\cas \\min_{i \\in [N]}\n  V(x_i\\opt)$. Thus for any $\\epsilon > 0$,\n  there exists $x_\\epsilon \\in \\mc{X}\\opt \\cap C$ such that\n  $V(x_\\epsilon) \\le \\epsilon$, so that\n  $\\inf_{x \\in C \\cap \\mc{X}\\opt} V(x) = 0$. By the continuity of $V$,\n  the infimum is attained and there is\n  $x\\opt$ such that $V(x\\opt) = 0$. The sequence $x_k$ thus converges to\n  this $x\\opt \\in \\mc{X}\\opt$,\n  and $F(x_k) \\cas F(x\\opt) = F\\opt$.\n\\end{proof-of-proposition}","start":49918,"end":54863}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:single-dingle"],"declared_external":false,"source_only":false,"title":"条件期望距离递推","source_units":[{"raw_tex":"\\begin{proof-of-proposition}[\\ref{proposition:convergence-from-boundedness}]\n  By Assumption~\\ref{assumption:very-weak-moment}, $\\E[\\norm{f'(x;\n      \\statrv)}^2] \\le \\growfunc(r)$ for all $x$ such that $\\dist(x,\n  \\mc{X}\\opt) \\le r$.  Lemma~\\ref{lemma:single-recentered-progress} implies\n  that for any $x\\opt \\in\\mc{X}\\opt$,\n  \\begin{align}\n    \\label{eqn:single-dingle}\n    \\E[\\norms{x_{k+1} - x\\opt}^2 \\mid \\mc{F}_{k-1}]\n    %% & \\le \\dist(x_k, \\mc{X}\\opt)^2 - 2 \\stepsize_k (F(x_k) - F\\opt)\n    %% + \\stepsize_k^2\n    %% \\sup_{\\norm{x - x\\opt} \\le \\dist(x_k, \\mc{X}\\opt)}\n    %% \\E[\\norms{f'(x; \\statrv)}^2] \\\\\n    & \\le \\norms{x_k - x\\opt}^2 - 2 \\stepsize_k (F(x_k) - F\\opt)\n    + \\stepsize_k^2\n    \\growfunc(\\dist(x_k, \\mc{X}\\opt)).\n  \\end{align}\n","start":49918,"end":50672}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"有界事件上的误差可和","source_units":[{"raw_tex":"  On the event that $\\sup_k \\dist(x_k, \\mc{X}\\opt) < \\infty$, we have\n  $\\sum_k \\stepsize_k^2 \\growfunc(\\dist(x_k, \\mc{X}\\opt)) < \\infty$, and so\n","start":50672,"end":50818}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"固定点距离极限与间隙可和","source_units":[{"raw_tex":"  the Robbins-Siegmund Lemma~\\ref{lemma:robbins-siegmund} implies\n  for any $x\\opt \\in \\mc{X}\\opt$,\n  there is some (random) $V(x\\opt) < \\infty$ such that\n  $\\norms{x_k - x\\opt} \\cas V(x\\opt)$, and\n  $\\sum_k \\stepsize_k (F(x_k) - F(x\\opt)) < \\infty$.\n  \n","start":50818,"end":51072}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"投影导出最优集距离递推","source_units":[{"raw_tex":"  We now show that $\\dist(x_k, \\mc{X}\\opt) \\cas 0$.\n  Letting $x\\opt$ be the\n  projection of $x_k$ onto $\\mc{X}\\opt$,\n  inequality~\\eqref{eqn:single-dingle} gives $\\E[\\dist(x_{k+1},\n    \\mc{X}\\opt)^2 \\mid \\mc{F}_{k-1}] \\le \\dist(x_k, \\mc{X}\\opt)^2 - 2\n  \\stepsize_k(F(x_k) - F\\opt) + \\stepsize_k^2 \\growfunc(\\dist(x_k,\n  \\mc{X}\\opt))$. Thus using Lemma~\\ref{lemma:robbins-siegmund}, there exists\n","start":51072,"end":51468}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"最优集距离存在极限","source_units":[{"raw_tex":"  \\mc{X}\\opt))$. Thus using Lemma~\\ref{lemma:robbins-siegmund}, there exists\n  a random $D_\\infty < \\infty$ such that $\\dist(x_k, \\mc{X}\\opt) \\cas\n  D_\\infty$.\n","start":51391,"end":51551}]},{"id":"S037","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"局部环带间隙函数","source_units":[{"raw_tex":"  For $\\epsilon_i \\in \\R_+$, define the gap function\n  \\begin{equation*}\n    \\gapfunc(x\\opt, \\epsilon_1, \\epsilon_2)\n    \\defeq \\inf_{x \\in \\mc{X}} \\left\\{F(x) - F\\opt\n    \\mid \\epsilon_1 \\le \\norm{x - x\\opt} \\le 4 \\epsilon_1,\n    \\dist(x, \\mc{X}\\opt) \\ge \\epsilon_2\n    \\right\\}.\n  \\end{equation*}\n","start":51551,"end":51850}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"紧环带上的正间隙","source_units":[{"raw_tex":"  The set $\\{x \\in \\mc{X} \\mid \\norm{x - x\\opt} \\in\n  [\\epsilon_1, 4\\epsilon_1], \\dist(x, \\mc{X}\\opt) \\ge \\epsilon_2\\}$ is\n  compact, so the infimum in the definition of the gap $\\gapfunc$ is\n  attained and $\\gapfunc(x\\opt, \\epsilon_1, \\epsilon_2) > 0$ for\n  $\\epsilon_i > 0$.  When the a.s.\\ limits\n","start":51850,"end":52150}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"正极限迫使尾部进入环带","source_units":[{"raw_tex":"  $\\epsilon_i > 0$.  When the a.s.\\ limits\n  $V(x\\opt)$ and $D_\\infty$ are positive,\n  there exists a (random) $K <\\infty$ such that\n  $\\norm{x_k - x\\opt} \\in [V(x\\opt)/2, 2 V(x\\opt)]$ and\n  $\\dist(x_k, \\mc{X}\\opt) \\ge D_\\infty / 2$ for all\n  $k \\ge K$. Thus with probability 1,\n","start":52107,"end":52386}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"发散步长排除正距离极限","source_units":[{"raw_tex":"  \\begin{equation*}\n    \\infty > \\sum_k \\stepsize_k (F(x_k) - F\\opt)\n    %% \\ge \\sum_{k \\ge K}\n    %% \\stepsize_k \\inf_{x \\in \\mc{X}_\\epsilon} (F(x) - F\\opt)\n    \\ge \\sum_{k \\ge K} \\stepsize_k \\gapfunc(x\\opt, V(x\\opt) / 2, D_\\infty / 2).\n  \\end{equation*}\n  As $\\sum_k \\stepsize_k = \\infty$ and\n  $\\gapfunc(x\\opt, \\epsilon_1, \\epsilon_2) > 0$ for\n  $\\epsilon_i > 0$, we have $D_\\infty = 0$\n  with probability 1.\n","start":52386,"end":52798}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"极限距离函数为利普希茨","source_units":[{"raw_tex":"\n  Finally, we show the sequence $x_k$ converges.\n  We begin by noting\n  that $V(x\\opt) = \\lim_k \\norm{x_k - x\\opt}$ is $1$-Lipschitz.\n  Indeed, let $x_1\\opt, x_2\\opt \\in \\mc{X}\\opt$. By\n  the a.s.\\ limit definition of $V$,\n  \\begin{equation*}\n    \\left|V(x_1\\opt) - V(x_2\\opt)\\right|\n    \\le\n    % \\limsup_k \\left|\\norm{x_k - x_1\\opt} - V(x_1\\opt)\\right|\n    % + \\limsup_k \\left|\\norm{x_k - x_2\\opt} - V(x_2\\opt)\\right|\n    \\limsup_k \\left|\\norm{x_k - x_1\\opt} - \\norm{x_k - x_2\\opt}\\right|\n    \\le \\norm{x_1\\opt - x_2\\opt}.\n  \\end{equation*}\n","start":52798,"end":53342}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"固定点距离极限推出点列有界","source_units":[{"raw_tex":"  Let us now show that for some $x\\opt \\in \\mc{X}\\opt$, we have $V(x\\opt) =\n  0$.  Let $\\ball = \\{x \\in \\R^n \\mid \\norm{x} \\le 1\\}$ be the ball.  As\n  $V(x\\opt) < \\infty$, there exists a (random) $r < \\infty$ such that the\n  $x_k \\in r \\ball$ for all $k$. As $\\dist(x_k, \\mc{X}\\opt) \\cas 0$, the\n","start":53342,"end":53638}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"有界球内存在最优点","source_units":[{"raw_tex":"  $x_k \\in r \\ball$ for all $k$. As $\\dist(x_k, \\mc{X}\\opt) \\cas 0$, the\n  compactness of $\\ball$ implies that $r \\ball \\cap \\mc{X}\\opt \\neq\n  \\emptyset$.  Let $x\\opt \\in r\\ball \\cap \\mc{X}\\opt$.  For any $x \\in\n","start":53565,"end":53777}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"投影落在固定紧球","source_units":[{"raw_tex":"  \\emptyset$.  Let $x\\opt \\in r\\ball \\cap \\mc{X}\\opt$.  For any $x \\in\n  r\\ball$, the projection $\\pi(x)$ of $x$ onto $\\mc{X}\\opt$ satisfies\n  $\\norm{\\pi(x) - x} \\le \\norm{x\\opt - x} \\le 2r$, and so $\\norm{\\pi(x)} \\le\n  3r$, and defining the set $C = 3r \\ball$,\n  we have $\\dist(x_k, \\mc{X}\\opt) = \\dist(x_k, \\mc{X}\\opt \\cap C)$ for all $k$.\n","start":53706,"end":54048}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"有限网近似最优集距离","source_units":[{"raw_tex":"  Now, fix $\\epsilon > 0$, and let $\\{x_i\\opt\\}_{i = 1}^N$ be\n  an $\\epsilon$-net of $C \\cap \\mc{X}\\opt$, where $N < \\infty$.\n  As $x_k \\in C$ for all $k$,\n  \\begin{equation*}\n    \\min_{i \\in [N]}\n    \\norm{x_k - x_i\\opt} - \\epsilon\n    \\le \\dist(x_k, \\mc{X}\\opt \\cap C)\n    = \\dist(x_k, \\mc{X}\\opt) \\cas 0,\n  \\end{equation*}\n","start":54048,"end":54374}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"有限网极限迫使下确界为零","source_units":[{"raw_tex":"  and $\\min_{i \\in [N]} \\norm{x_k - x_i\\opt} \\cas \\min_{i \\in [N]}\n  V(x_i\\opt)$. Thus for any $\\epsilon > 0$,\n  there exists $x_\\epsilon \\in \\mc{X}\\opt \\cap C$ such that\n  $V(x_\\epsilon) \\le \\epsilon$, so that\n  $\\inf_{x \\in C \\cap \\mc{X}\\opt} V(x) = 0$. By the continuity of $V$,\n","start":54374,"end":54656}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"零极限距离给出点列收敛","source_units":[{"raw_tex":"  $\\inf_{x \\in C \\cap \\mc{X}\\opt} V(x) = 0$. By the continuity of $V$,\n  the infimum is attained and there is\n  $x\\opt$ such that $V(x\\opt) = 0$. The sequence $x_k$ thus converges to\n  this $x\\opt \\in \\mc{X}\\opt$,\n","start":54585,"end":54799}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"目标函数值收敛","source_units":[{"raw_tex":"  and $F(x_k) \\cas F(x\\opt) = F\\opt$.\n\\end{proof-of-proposition}","start":54799,"end":54863}]},{"id":"S049","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":true,"title":"可测次梯度选择背景","source_units":[{"raw_tex":"list two assumptions that we frequently make. As $f(\\cdot; \\statval)$ is\nsubdifferentiable on $\\mc{X}$, there exist measurable selections $f'(x;\n\\statval) \\in \\partial f(x; \\statval)$, and $\\E[\\partial f(x; \\statrv)] =\n\\partial F(x)$ for $x \\in \\mc{X}$ (cf.~\\cite[Sec.~2]{Bertsekas73}).\n","start":10061,"end":10348}]},{"id":"S050","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始许可前提中的 standard convex optimality and subgradient inequalities；三点距离恒等式为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"凸最优性与次梯度工具","source_units":[]},{"id":"S051","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始许可前提中的 Young's inequality。"],"source_only":false,"title":"杨氏二次内积界","source_units":[]},{"id":"S052","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始许可前提中的 conditional expectation under independent sampling。"],"source_only":false,"title":"独立采样的条件期望","source_units":[]},{"id":"S053","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始许可前提中的 standard finite-dimensional projection arguments。"],"source_only":false,"title":"闭凸投影与距离性质","source_units":[]},{"id":"S054","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始许可前提中的 standard finite-dimensional compactness and continuity arguments。"],"source_only":false,"title":"有限维紧性与有限网","source_units":[]},{"id":"S055","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始许可前提中的 standard finite-dimensional separability arguments；作者未展开其应用，作为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"可分性与共同概率一事件","source_units":[]},{"id":"S056","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始许可前提中的 standard convex subgradient inequalities and finite-dimensional continuity arguments；具体期望目标正则性为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"目标连续性的适用背景","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S020","edges":[{"source_id":"S007","target_id":"S010","kind":"supports"},{"source_id":"S008","target_id":"S010","kind":"supports"},{"source_id":"S050","target_id":"S010","kind":"supports"},{"source_id":"S006","target_id":"S005","kind":"supports"},{"source_id":"S001","target_id":"S020","kind":"supports"},{"source_id":"S002","target_id":"S020","kind":"supports"},{"source_id":"S005","target_id":"S020","kind":"supports"},{"source_id":"S007","target_id":"S020","kind":"supports"},{"source_id":"S008","target_id":"S020","kind":"supports"},{"source_id":"S011","target_id":"S020","kind":"supports"},{"source_id":"S013","target_id":"S020","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S005","target_id":"S023","kind":"supports"},{"source_id":"S007","target_id":"S023","kind":"supports"},{"source_id":"S002","target_id":"S023","kind":"supports"},{"source_id":"S050","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S050","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S007","target_id":"S025","kind":"supports"},{"source_id":"S050","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S005","target_id":"S021","kind":"supports"},{"source_id":"S007","target_id":"S021","kind":"supports"},{"source_id":"S005","target_id":"S026","kind":"supports"},{"source_id":"S007","target_id":"S026","kind":"supports"},{"source_id":"S008","target_id":"S026","kind":"supports"},{"source_id":"S021","target_id":"S028","kind":"supports"},{"source_id":"S010","target_id":"S029","kind":"supports"},{"source_id":"S007","target_id":"S029","kind":"supports"},{"source_id":"S008","target_id":"S029","kind":"supports"},{"source_id":"S050","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S051","target_id":"S030","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S032","kind":"supports"},{"source_id":"S013","target_id":"S032","kind":"supports"},{"source_id":"S001","target_id":"S032","kind":"supports"},{"source_id":"S004","target_id":"S032","kind":"supports"},{"source_id":"S012","target_id":"S032","kind":"supports"},{"source_id":"S015","target_id":"S032","kind":"supports"},{"source_id":"S049","target_id":"S032","kind":"supports"},{"source_id":"S052","target_id":"S032","kind":"supports"},{"source_id":"S013","target_id":"S033","kind":"supports"},{"source_id":"S018","target_id":"S033","kind":"supports"},{"source_id":"S019","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S016","target_id":"S034","kind":"supports"},{"source_id":"S012","target_id":"S034","kind":"supports"},{"source_id":"S018","target_id":"S034","kind":"supports"},{"source_id":"S032","target_id":"S035","kind":"supports"},{"source_id":"S012","target_id":"S035","kind":"supports"},{"source_id":"S015","target_id":"S035","kind":"supports"},{"source_id":"S053","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S033","target_id":"S036","kind":"supports"},{"source_id":"S016","target_id":"S036","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S002","target_id":"S038","kind":"supports"},{"source_id":"S012","target_id":"S038","kind":"supports"},{"source_id":"S053","target_id":"S038","kind":"supports"},{"source_id":"S054","target_id":"S038","kind":"supports"},{"source_id":"S056","target_id":"S038","kind":"supports"},{"source_id":"S034","target_id":"S039","kind":"supports"},{"source_id":"S036","target_id":"S039","kind":"supports"},{"source_id":"S034","target_id":"S040","kind":"supports"},{"source_id":"S036","target_id":"S040","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S018","target_id":"S040","kind":"supports"},{"source_id":"S053","target_id":"S040","kind":"supports"},{"source_id":"S034","target_id":"S041","kind":"supports"},{"source_id":"S055","target_id":"S041","kind":"supports"},{"source_id":"S034","target_id":"S042","kind":"supports"},{"source_id":"S012","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S040","target_id":"S043","kind":"supports"},{"source_id":"S054","target_id":"S043","kind":"supports"},{"source_id":"S056","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S053","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S040","target_id":"S045","kind":"supports"},{"source_id":"S054","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S034","target_id":"S046","kind":"supports"},{"source_id":"S041","target_id":"S046","kind":"supports"},{"source_id":"S055","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S041","target_id":"S047","kind":"supports"},{"source_id":"S044","target_id":"S047","kind":"supports"},{"source_id":"S054","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S056","target_id":"S048","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S034","target_id":"S031","kind":"supports"},{"source_id":"S035","target_id":"S031","kind":"supports"},{"source_id":"S036","target_id":"S031","kind":"supports"},{"source_id":"S037","target_id":"S031","kind":"supports"},{"source_id":"S038","target_id":"S031","kind":"supports"},{"source_id":"S039","target_id":"S031","kind":"supports"},{"source_id":"S040","target_id":"S031","kind":"supports"},{"source_id":"S041","target_id":"S031","kind":"supports"},{"source_id":"S042","target_id":"S031","kind":"supports"},{"source_id":"S043","target_id":"S031","kind":"supports"},{"source_id":"S044","target_id":"S031","kind":"supports"},{"source_id":"S045","target_id":"S031","kind":"supports"},{"source_id":"S046","target_id":"S031","kind":"supports"},{"source_id":"S047","target_id":"S031","kind":"supports"},{"source_id":"S048","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S017","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S020","edges":[{"source_id":"S007","target_id":"S010","kind":"supports"},{"source_id":"S008","target_id":"S010","kind":"supports"},{"source_id":"S050","target_id":"S010","kind":"supports"},{"source_id":"S006","target_id":"S005","kind":"supports"},{"source_id":"S001","target_id":"S020","kind":"supports"},{"source_id":"S002","target_id":"S020","kind":"supports"},{"source_id":"S005","target_id":"S020","kind":"supports"},{"source_id":"S007","target_id":"S020","kind":"supports"},{"source_id":"S008","target_id":"S020","kind":"supports"},{"source_id":"S011","target_id":"S020","kind":"supports"},{"source_id":"S013","target_id":"S020","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S005","target_id":"S021","kind":"supports"},{"source_id":"S007","target_id":"S021","kind":"supports"},{"source_id":"S002","target_id":"S021","kind":"supports"},{"source_id":"S050","target_id":"S021","kind":"supports"},{"source_id":"S005","target_id":"S026","kind":"supports"},{"source_id":"S007","target_id":"S026","kind":"supports"},{"source_id":"S008","target_id":"S026","kind":"supports"},{"source_id":"S021","target_id":"S026","kind":"supports"},{"source_id":"S010","target_id":"S026","kind":"supports"},{"source_id":"S050","target_id":"S026","kind":"supports"},{"source_id":"S051","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S032","kind":"supports"},{"source_id":"S013","target_id":"S032","kind":"supports"},{"source_id":"S001","target_id":"S032","kind":"supports"},{"source_id":"S004","target_id":"S032","kind":"supports"},{"source_id":"S012","target_id":"S032","kind":"supports"},{"source_id":"S015","target_id":"S032","kind":"supports"},{"source_id":"S049","target_id":"S032","kind":"supports"},{"source_id":"S052","target_id":"S032","kind":"supports"},{"source_id":"S013","target_id":"S033","kind":"supports"},{"source_id":"S018","target_id":"S033","kind":"supports"},{"source_id":"S019","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S016","target_id":"S034","kind":"supports"},{"source_id":"S012","target_id":"S034","kind":"supports"},{"source_id":"S018","target_id":"S034","kind":"supports"},{"source_id":"S032","target_id":"S035","kind":"supports"},{"source_id":"S012","target_id":"S035","kind":"supports"},{"source_id":"S015","target_id":"S035","kind":"supports"},{"source_id":"S053","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S033","target_id":"S036","kind":"supports"},{"source_id":"S016","target_id":"S036","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S002","target_id":"S038","kind":"supports"},{"source_id":"S012","target_id":"S038","kind":"supports"},{"source_id":"S053","target_id":"S038","kind":"supports"},{"source_id":"S054","target_id":"S038","kind":"supports"},{"source_id":"S056","target_id":"S038","kind":"supports"},{"source_id":"S034","target_id":"S039","kind":"supports"},{"source_id":"S036","target_id":"S039","kind":"supports"},{"source_id":"S034","target_id":"S040","kind":"supports"},{"source_id":"S036","target_id":"S040","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S018","target_id":"S040","kind":"supports"},{"source_id":"S053","target_id":"S040","kind":"supports"},{"source_id":"S034","target_id":"S041","kind":"supports"},{"source_id":"S055","target_id":"S041","kind":"supports"},{"source_id":"S034","target_id":"S042","kind":"supports"},{"source_id":"S012","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S040","target_id":"S043","kind":"supports"},{"source_id":"S054","target_id":"S043","kind":"supports"},{"source_id":"S056","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S053","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S040","target_id":"S045","kind":"supports"},{"source_id":"S054","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S034","target_id":"S046","kind":"supports"},{"source_id":"S041","target_id":"S046","kind":"supports"},{"source_id":"S055","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S041","target_id":"S047","kind":"supports"},{"source_id":"S044","target_id":"S047","kind":"supports"},{"source_id":"S054","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S056","target_id":"S048","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S034","target_id":"S031","kind":"supports"},{"source_id":"S035","target_id":"S031","kind":"supports"},{"source_id":"S036","target_id":"S031","kind":"supports"},{"source_id":"S037","target_id":"S031","kind":"supports"},{"source_id":"S038","target_id":"S031","kind":"supports"},{"source_id":"S039","target_id":"S031","kind":"supports"},{"source_id":"S040","target_id":"S031","kind":"supports"},{"source_id":"S041","target_id":"S031","kind":"supports"},{"source_id":"S042","target_id":"S031","kind":"supports"},{"source_id":"S043","target_id":"S031","kind":"supports"},{"source_id":"S044","target_id":"S031","kind":"supports"},{"source_id":"S045","target_id":"S031","kind":"supports"},{"source_id":"S046","target_id":"S031","kind":"supports"},{"source_id":"S047","target_id":"S031","kind":"supports"},{"source_id":"S048","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S020","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eqn:objective"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"随机凸优化问题","source_units":[{"raw_tex":"In this paper, we develop and analyze a family of model-based methods,\nmoving beyond naive stochastic gradient methods, for solving the stochastic\nconvex optimization problem\n\\begin{equation}\n  \\label{eqn:objective}\n  \\begin{split}\n    \\minimize ~ & F(x) = \\E_P[f(x;\\statrv)]\n    = \\int_\\statdomain f(x; \\statval) dP(\\statval) \\\\\n    \\subjectto ~ & x \\in \\mc{X}.\n  \\end{split}\n\\end{equation}\n","start":3159,"end":3551}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"有限维闭凸设定","source_units":[{"raw_tex":"In problem~\\eqref{eqn:objective}, the set $\\statdomain$ is a sample space,\nand for each $\\statval \\in \\statdomain$, the function $f(\\cdot; \\statval) :\n\\R^n \\to \\R$ is a closed convex function, subdifferentiable\non the closed convex set $\\mc{X} \\subset \\R^n$.\n","start":3551,"end":3810}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"独立同分布采样","source_units":[{"raw_tex":"\\aProx algorithms iterate as follows: for $k = 1, 2, \\ldots$, we draw a\nrandom $\\statrv_k \\simiid P$, then update the iterate $x_k$ by minimizing a\nregularized approximation to $f(\\cdot; \\statrv_k)$, setting\n","start":7588,"end":7796}]},{"id":"S005","semantic_type":"algorithm","kind":"algorithm","labels":["eqn:model-iteration"],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"正则化模型更新","source_units":[{"raw_tex":"\\begin{equation}\n  \\label{eqn:model-iteration}\n  x_{k+1} \\defeq \\argmin_{x \\in \\mc{X}}\n  \\left\\{ f_{x_k}(x ; \\statrv_k) + \\frac{1}{2 \\stepsize_k}\n  \\ltwo{x - x_k}^2 \\right\\}.\n\\end{equation}\n","start":7796,"end":7986}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"中心模型记号","source_units":[{"raw_tex":"The function $f_x(\\cdot ; \\statval)$ is a \\emph{model} of $f(\\cdot;\n\\statval)$ at the point $x$, meaning that $f_x$ satisfies the\nfollowing conditions on its structure and local approximation\nproperties for $f$:\n\\begin{enumerate}[label=(C.\\roman*),leftmargin=*]\n","start":7986,"end":8248}]},{"id":"S007","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"模型凸性条件","source_units":[{"raw_tex":"\\item \\label{cond:convex-model}\n  The function $y \\mapsto f_x(y; \\statval)$ is convex\n  and subdifferentiable on $\\mc{X}$.\n","start":8248,"end":8371}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"相切下模型条件","source_units":[{"raw_tex":"\\item \\label{cond:lower-model}\n  The model $f_x$ satisfies the equality\n  $f_x(x; \\statval) = f(x; \\statval)$ and\n  \\begin{equation*}\n    f_x(y; \\statval) \\le f(y; \\statval) ~~ \\mbox{for~all~} y.\n  \\end{equation*}\n","start":8371,"end":8585}]},{"id":"S010","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:subgrad-containment"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"中心次梯度包含","source_units":[{"raw_tex":"\\noindent\nBy the first-order conditions for convexity, for any $g \\in \\partial_y\nf_x(y; \\statval)|_{y=x}$ Conditions~\\ref{cond:convex-model}\nand~\\ref{cond:lower-model} imply $f(y; \\statval) \\ge f_x(y; \\statval) \\ge\nf_x(x; \\statval) + \\<g, y - x\\> = f(x; \\statval) + \\<g, y - x\\>$, yielding\nthe containment\n\\begin{equation}\n  \\label{eqn:subgrad-containment}\n  \\left.\\partial_y f_x(y; \\statval)\\right|_{y = x}\n  \\subset \\partial_x f(x; \\statval).\n\\end{equation}\n","start":8808,"end":9268}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","labels":["assumption:convex"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"最优解与最优点矩条件","source_units":[{"raw_tex":"\\begin{assumption}\n  \\label{assumption:convex}\n  The set $\\mc{X}\\opt \\defeq \\argmin_{x \\in \\mc{X}} \\{F(x)\\}$ is non-empty,\n  and there exists $\\sigma^2 < \\infty$\n  such that $\\E[\\ltwo{f'(x\\opt;\\statrv)}^2] \\leq \\sigma^2$\n  for $x\\opt \\in \\mc{X}\\opt$ \n  and all measurable selections $f'(x\\opt; \\statval) \\in \\partial\n  f(x\\opt; \\statval)$.\n\\end{assumption}\n\n","start":10349,"end":10707}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["assumption:convex"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"非空最优集","source_units":[{"raw_tex":"  The set $\\mc{X}\\opt \\defeq \\argmin_{x \\in \\mc{X}} \\{F(x)\\}$ is non-empty,\n","start":10396,"end":10472}]},{"id":"S013","semantic_type":"assumption","kind":"assumption","labels":["assumption:very-weak-moment"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"距离控制的弱矩条件","source_units":[{"raw_tex":"\\begin{assumption}\n  \\label{assumption:very-weak-moment}\n  There exists a non-decreasing function $\\growfunc : \\R_+ \\to\n  \\openright{0}{\\infty}$ such that for all $x \\in\n  \\mc{X}$ and measurable selections $f'(x; \\statval) \\in \\partial\n  f(x; \\statval)$,\n  $\\E[\\norm{f'(x; \\statrv)}^2] \\le \\growfunc(\\dist(x, \\mc{X}\\opt))$.\n\\end{assumption}\n","start":10707,"end":11048}]},{"id":"S014","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"次梯度与最优点记号","source_units":[{"raw_tex":"\\paragraph{Notation}\nFor a convex function $f$, $\\partial f(x)$ denotes its subgradient set at\n$x$, and $f'(x) \\in \\partial f(x)$ denotes an arbitrary element of\nthe subdifferential.  Throughout, $x\\opt$ denotes a minimizer of\nproblem~\\eqref{eqn:objective} and $\\mc{X}\\opt = \\argmin_{x \\in \\mc{X}} F(x)$\n","start":18567,"end":18871}]},{"id":"S015","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"滤过与可预测迭代","source_units":[{"raw_tex":"its optimal set.  We let $\\mc{F}_k \\defeq \\sigma(\\statrv_1, \\ldots,\n\\statrv_k)$ be the $\\sigma$-field generated by the first $k$ random\nvariables $\\statrv_i$, so $x_k \\in\n\\mc{F}_{k-1}$ for all $k$ under iteration~\\eqref{eqn:model-iteration}.\n","start":18871,"end":19113}]},{"id":"S016","semantic_type":"external_fact","kind":"external_fact","labels":["lemma:robbins-siegmund"],"member_ids":["S016"],"declared_external":true,"declaration_text":["原始许可前提：Robbins–Siegmund almost-supermartingale convergence theorem，RobbinsSi71，按 B0039 所述版本使用。"],"source_only":false,"title":"几乎超鞅收敛引理","source_units":[{"raw_tex":"\\begin{lemma}[\\cite{RobbinsSi71}]\n  \\label{lemma:robbins-siegmund}\n  Let $A_k, B_k, C_k, D_k \\ge 0$ be non-negative random variables adapted to\n  the filtration $\\mc{F}_k$ and satisfying $\\E[A_{k + 1} \\mid \\mc{F}_k] \\le\n  (1 + B_k) A_k + C_k - D_k$. Then on the event $\\{\\sum_k B_k < \\infty,\n  \\sum_k C_k < \\infty\\}$, there is a random $A_\\infty < \\infty$\n  such that $A_k \\cas A_\\infty$ and $\\sum_k D_k < \\infty$.\n\\end{lemma}\n","start":38604,"end":39031}]},{"id":"S018","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"步长求和条件","source_units":[{"raw_tex":"Section~\\ref{sec:asymptotic-normality}, we without comment make the\nassumption that the stepsizes $\\stepsize_k$ satisfy the standard summability\nconditions\n\\begin{equation*}\n  \\stepsize_k > 0 ~ \\mbox{for~all~} k,\n  ~~~\n  \\sum_{k=1}^\\infty \\stepsize_k = \\infty,\n  ~~~ \\mbox{and} ~~~\n  \\sum_{k=1}^\\infty \\stepsize_k^2 < \\infty.\n\\end{equation*}\n","start":45841,"end":46183}]},{"id":"S019","semantic_type":"assumption","kind":"assumption","labels":["proposition:convergence-from-boundedness"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"距离有界事件","source_units":[{"raw_tex":"  On the event that\n  $\\sup_k \\dist(x_k, \\mc{X}\\opt) < \\infty$, with probability\n","start":46508,"end":46589}]},{"id":"S020","semantic_type":"proposition","kind":"proposition","labels":["proposition:convergence-from-boundedness"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"由有界性推出几乎必然收敛","source_units":[{"raw_tex":"\\begin{proposition}\n  \\label{proposition:convergence-from-boundedness}\n  Let Assumptions~\\ref{assumption:convex} and\n  \\ref{assumption:very-weak-moment} hold. Let the iterates $x_k$ be\n  generated by any method satisfying\n  Conditions~\\ref{cond:convex-model}--\\ref{cond:lower-model}\n  and $F\\opt = \\inf_{x \\in \\mc{X}} F(x)$.\n  On the event that\n  $\\sup_k \\dist(x_k, \\mc{X}\\opt) < \\infty$, with probability\n  one both $\\sum_k \\stepsize_k (F(x_k) - F\\opt) < \\infty$,\n  and the iterates $x_k$ are convergent: there exists\n  $x\\opt \\in \\mc{X}\\opt$ such that\n  $\\norm{x_k - x\\opt} \\cas 0$ and\n  $F(x_k) \\cas F(x\\opt)$.\n\\end{proposition}","start":46183,"end":46814}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","labels":["lemma:single-step-progress"],"member_ids":["S021","S022","S023","S024","S025"],"declared_external":false,"source_only":false,"title":"模型单步进展引理","source_units":[{"raw_tex":"\\begin{lemma}\n  \\label{lemma:single-step-progress}\n  Let Condition~\\ref{cond:convex-model} hold.\n  In each step of the method~\\eqref{eqn:model-iteration},\n  for any $x \\in \\mc{X}$,\n  \\begin{equation*}\n    \\half \\ltwo{x_{k + 1} - x}^2\n    \\le \\half \\ltwo{x_k - x}^2\n    - \\stepsize_k\\left[f_{x_k}(x_{k + 1}; \\statrv_k)\n      - f_{x_k}(x; \\statrv_k)\\right]\n    - \\half \\ltwo{x_k - x_{k+1}}^2.\n  \\end{equation*}\n\\end{lemma}","start":41493,"end":41913},{"raw_tex":"\\begin{proof}\n  By the first-order conditions for convex optimization,\n  for some $g_k \\in \\partial f_{x_k}(x_{k+1}; \\statrv_k)$ we have that\n  $\\<\\stepsize_k g_k + (x_{k+1} - x_k), y - x_{k+1}\\> \\ge 0$ for\n  all $y \\in \\mc{X}$.\n","start":41914,"end":42143},{"raw_tex":"\\begin{proof}\n  By the first-order conditions for convex optimization,\n  for some $g_k \\in \\partial f_{x_k}(x_{k+1}; \\statrv_k)$ we have that\n  $\\<\\stepsize_k g_k + (x_{k+1} - x_k), y - x_{k+1}\\> \\ge 0$ for\n  all $y \\in \\mc{X}$.\n  Setting $y = x$, we obtain\n  \\begin{equation*}\n    \\stepsize_k \\<g_k, x_{k+1} - x\\>\n    \\le \\<x_{k + 1} - x_k, x - x_{k + 1}\\>\n    = \\half \\left[\\ltwo{x_k - x}^2\n      - \\ltwo{x_{k + 1} - x}^2\n      - \\ltwo{x_{k + 1} - x_k}^2 \\right].\n  \\end{equation*}\n  As $f_{x_k}(x; \\statrv_k) \\ge f_{x_k}(x_{k+1}; \\statrv_k) + \\<g_k,\n  x - x_{k+1}\\>$ by Condition~\\ref{cond:convex-model}, this gives the\n  result.\n\\end{proof}","start":41914,"end":42558},{"raw_tex":"  Setting $y = x$, we obtain\n  \\begin{equation*}\n    \\stepsize_k \\<g_k, x_{k+1} - x\\>\n    \\le \\<x_{k + 1} - x_k, x - x_{k + 1}\\>\n    = \\half \\left[\\ltwo{x_k - x}^2\n      - \\ltwo{x_{k + 1} - x}^2\n      - \\ltwo{x_{k + 1} - x_k}^2 \\right].\n  \\end{equation*}\n","start":42143,"end":42398},{"raw_tex":"  As $f_{x_k}(x; \\statrv_k) \\ge f_{x_k}(x_{k+1}; \\statrv_k) + \\<g_k,\n  x - x_{k+1}\\>$ by Condition~\\ref{cond:convex-model}, this gives the\n  result.\n\\end{proof}","start":42398,"end":42558}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","labels":["lemma:single-recentered-progress"],"member_ids":["S026","S027","S028","S029","S030"],"declared_external":false,"source_only":false,"title":"中心化单步进展引理","source_units":[{"raw_tex":"\\begin{lemma}\n  \\label{lemma:single-recentered-progress}\n  Let Conditions~\\ref{cond:convex-model}--\\ref{cond:lower-model} hold and\n  let $x_k$ be generated by the updates~\\eqref{eqn:model-iteration}.\n  Then for any $x \\in \\mc{X}$,\n  \\begin{equation*}\n    \\half \\ltwo{x_{k + 1} - x}^2\n    \\le \\half \\ltwo{x_k - x}^2\n    - \\stepsize_k [f(x_k; \\statrv_k) - f(x; \\statrv_k)]\n    + \\frac{\\stepsize_k^2}{2} \\ltwo{f'(x_k; \\statrv_k)}^2.\n  \\end{equation*}\n\\end{lemma}","start":48263,"end":48722},{"raw_tex":"\\begin{proof}\n  Using Lemma~\\ref{lemma:single-step-progress}, it suffices to\n  show that for any $\\stepsize > 0$ and $x_0, x_1, x \\in \\mc{X}$\n  \\begin{equation*}\n    -\\stepsize [f_{x_0}(x_1; \\statval) - f_{x_0}(x; \\statval)]\n    - \\half \\ltwo{x_1 - x_0}^2\n    \\le -\\stepsize [f(x_0; \\statval) - f(x; \\statval)]\n    + \\frac{\\stepsize^2}{2} \\ltwo{f'(x_0; \\statval)}^2.\n  \\end{equation*}\n","start":48723,"end":49108},{"raw_tex":"\\begin{proof}\n  Using Lemma~\\ref{lemma:single-step-progress}, it suffices to\n  show that for any $\\stepsize > 0$ and $x_0, x_1, x \\in \\mc{X}$\n  \\begin{equation*}\n    -\\stepsize [f_{x_0}(x_1; \\statval) - f_{x_0}(x; \\statval)]\n    - \\half \\ltwo{x_1 - x_0}^2\n    \\le -\\stepsize [f(x_0; \\statval) - f(x; \\statval)]\n    + \\frac{\\stepsize^2}{2} \\ltwo{f'(x_0; \\statval)}^2.\n  \\end{equation*}\n  To see this, recall that\n  Conditions~\\ref{cond:convex-model}--\\ref{cond:lower-model} imply the\n  containment~\\eqref{eqn:subgrad-containment}, which in turn implies\n  \\begin{align*}\n    -f_{x_0}(x_1; \\statval) + f_{x_0}(x; \\statval)\n    & = -[f_{x_0}(x_0; \\statval) - f_{x_0}(x; \\statval)]\n    + f_{x_0}(x_0; \\statval) - f_{x_0}(x_1; \\statval) \\\\\n    %& \\stackrel{\\ref{cond:subgrad-model}}{\\le}\n    & \\le\n    -[f_{x_0}(x_0; \\statval) - f_{x_0}(x; \\statval)]\n    + \\<f'(x_0;\\statval), x_0 - x_1\\> \\\\\n    & \\stackrel{\\ref{cond:lower-model}}{\\le}\n    -[f(x_0;\\statval) - f(x; \\statval)]\n    + \\<f'(x_0;\\statval), x_0 - x_1\\>.\n  \\end{align*}\n  Then we use that for any vector $v$,\n  $\\stepsize \\<v, \\Delta\\> - \\half \\ltwo{\\Delta}^2 \\le\n  \\frac{\\stepsize^2}{2} \\ltwo{v}^2$, which\n  gives the result.\n\\end{proof}","start":48723,"end":49916},{"raw_tex":"  To see this, recall that\n  Conditions~\\ref{cond:convex-model}--\\ref{cond:lower-model} imply the\n  containment~\\eqref{eqn:subgrad-containment}, which in turn implies\n  \\begin{align*}\n    -f_{x_0}(x_1; \\statval) + f_{x_0}(x; \\statval)\n    & = -[f_{x_0}(x_0; \\statval) - f_{x_0}(x; \\statval)]\n    + f_{x_0}(x_0; \\statval) - f_{x_0}(x_1; \\statval) \\\\\n    %& \\stackrel{\\ref{cond:subgrad-model}}{\\le}\n    & \\le\n    -[f_{x_0}(x_0; \\statval) - f_{x_0}(x; \\statval)]\n    + \\<f'(x_0;\\statval), x_0 - x_1\\> \\\\\n    & \\stackrel{\\ref{cond:lower-model}}{\\le}\n    -[f(x_0;\\statval) - f(x; \\statval)]\n    + \\<f'(x_0;\\statval), x_0 - x_1\\>.\n  \\end{align*}\n","start":49108,"end":49748},{"raw_tex":"  Then we use that for any vector $v$,\n  $\\stepsize \\<v, \\Delta\\> - \\half \\ltwo{\\Delta}^2 \\le\n  \\frac{\\stepsize^2}{2} \\ltwo{v}^2$, which\n  gives the result.\n\\end{proof}","start":49748,"end":49916}]},{"id":"S031","semantic_type":"proof","kind":"proof","labels":["eqn:single-dingle"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"有界性收敛命题完整证明","source_units":[{"raw_tex":"\\begin{proof-of-proposition}[\\ref{proposition:convergence-from-boundedness}]\n  By Assumption~\\ref{assumption:very-weak-moment}, $\\E[\\norm{f'(x;\n      \\statrv)}^2] \\le \\growfunc(r)$ for all $x$ such that $\\dist(x,\n  \\mc{X}\\opt) \\le r$.  Lemma~\\ref{lemma:single-recentered-progress} implies\n  that for any $x\\opt \\in\\mc{X}\\opt$,\n  \\begin{align}\n    \\label{eqn:single-dingle}\n    \\E[\\norms{x_{k+1} - x\\opt}^2 \\mid \\mc{F}_{k-1}]\n    %% & \\le \\dist(x_k, \\mc{X}\\opt)^2 - 2 \\stepsize_k (F(x_k) - F\\opt)\n    %% + \\stepsize_k^2\n    %% \\sup_{\\norm{x - x\\opt} \\le \\dist(x_k, \\mc{X}\\opt)}\n    %% \\E[\\norms{f'(x; \\statrv)}^2] \\\\\n    & \\le \\norms{x_k - x\\opt}^2 - 2 \\stepsize_k (F(x_k) - F\\opt)\n    + \\stepsize_k^2\n    \\growfunc(\\dist(x_k, \\mc{X}\\opt)).\n  \\end{align}\n  On the event that $\\sup_k \\dist(x_k, \\mc{X}\\opt) < \\infty$, we have\n  $\\sum_k \\stepsize_k^2 \\growfunc(\\dist(x_k, \\mc{X}\\opt)) < \\infty$, and so\n  the Robbins-Siegmund Lemma~\\ref{lemma:robbins-siegmund} implies\n  for any $x\\opt \\in \\mc{X}\\opt$,\n  there is some (random) $V(x\\opt) < \\infty$ such that\n  $\\norms{x_k - x\\opt} \\cas V(x\\opt)$, and\n  $\\sum_k \\stepsize_k (F(x_k) - F(x\\opt)) < \\infty$.\n  \n  We now show that $\\dist(x_k, \\mc{X}\\opt) \\cas 0$.\n  Letting $x\\opt$ be the\n  projection of $x_k$ onto $\\mc{X}\\opt$,\n  inequality~\\eqref{eqn:single-dingle} gives $\\E[\\dist(x_{k+1},\n    \\mc{X}\\opt)^2 \\mid \\mc{F}_{k-1}] \\le \\dist(x_k, \\mc{X}\\opt)^2 - 2\n  \\stepsize_k(F(x_k) - F\\opt) + \\stepsize_k^2 \\growfunc(\\dist(x_k,\n  \\mc{X}\\opt))$. Thus using Lemma~\\ref{lemma:robbins-siegmund}, there exists\n  a random $D_\\infty < \\infty$ such that $\\dist(x_k, \\mc{X}\\opt) \\cas\n  D_\\infty$.\n  For $\\epsilon_i \\in \\R_+$, define the gap function\n  \\begin{equation*}\n    \\gapfunc(x\\opt, \\epsilon_1, \\epsilon_2)\n    \\defeq \\inf_{x \\in \\mc{X}} \\left\\{F(x) - F\\opt\n    \\mid \\epsilon_1 \\le \\norm{x - x\\opt} \\le 4 \\epsilon_1,\n    \\dist(x, \\mc{X}\\opt) \\ge \\epsilon_2\n    \\right\\}.\n  \\end{equation*}\n  The set $\\{x \\in \\mc{X} \\mid \\norm{x - x\\opt} \\in\n  [\\epsilon_1, 4\\epsilon_1], \\dist(x, \\mc{X}\\opt) \\ge \\epsilon_2\\}$ is\n  compact, so the infimum in the definition of the gap $\\gapfunc$ is\n  attained and $\\gapfunc(x\\opt, \\epsilon_1, \\epsilon_2) > 0$ for\n  $\\epsilon_i > 0$.  When the a.s.\\ limits\n  $V(x\\opt)$ and $D_\\infty$ are positive,\n  there exists a (random) $K <\\infty$ such that\n  $\\norm{x_k - x\\opt} \\in [V(x\\opt)/2, 2 V(x\\opt)]$ and\n  $\\dist(x_k, \\mc{X}\\opt) \\ge D_\\infty / 2$ for all\n  $k \\ge K$. Thus with probability 1,\n  \\begin{equation*}\n    \\infty > \\sum_k \\stepsize_k (F(x_k) - F\\opt)\n    %% \\ge \\sum_{k \\ge K}\n    %% \\stepsize_k \\inf_{x \\in \\mc{X}_\\epsilon} (F(x) - F\\opt)\n    \\ge \\sum_{k \\ge K} \\stepsize_k \\gapfunc(x\\opt, V(x\\opt) / 2, D_\\infty / 2).\n  \\end{equation*}\n  As $\\sum_k \\stepsize_k = \\infty$ and\n  $\\gapfunc(x\\opt, \\epsilon_1, \\epsilon_2) > 0$ for\n  $\\epsilon_i > 0$, we have $D_\\infty = 0$\n  with probability 1.\n\n  Finally, we show the sequence $x_k$ converges.\n  We begin by noting\n  that $V(x\\opt) = \\lim_k \\norm{x_k - x\\opt}$ is $1$-Lipschitz.\n  Indeed, let $x_1\\opt, x_2\\opt \\in \\mc{X}\\opt$. By\n  the a.s.\\ limit definition of $V$,\n  \\begin{equation*}\n    \\left|V(x_1\\opt) - V(x_2\\opt)\\right|\n    \\le\n    % \\limsup_k \\left|\\norm{x_k - x_1\\opt} - V(x_1\\opt)\\right|\n    % + \\limsup_k \\left|\\norm{x_k - x_2\\opt} - V(x_2\\opt)\\right|\n    \\limsup_k \\left|\\norm{x_k - x_1\\opt} - \\norm{x_k - x_2\\opt}\\right|\n    \\le \\norm{x_1\\opt - x_2\\opt}.\n  \\end{equation*}\n  Let us now show that for some $x\\opt \\in \\mc{X}\\opt$, we have $V(x\\opt) =\n  0$.  Let $\\ball = \\{x \\in \\R^n \\mid \\norm{x} \\le 1\\}$ be the ball.  As\n  $V(x\\opt) < \\infty$, there exists a (random) $r < \\infty$ such that the\n  $x_k \\in r \\ball$ for all $k$. As $\\dist(x_k, \\mc{X}\\opt) \\cas 0$, the\n  compactness of $\\ball$ implies that $r \\ball \\cap \\mc{X}\\opt \\neq\n  \\emptyset$.  Let $x\\opt \\in r\\ball \\cap \\mc{X}\\opt$.  For any $x \\in\n  r\\ball$, the projection $\\pi(x)$ of $x$ onto $\\mc{X}\\opt$ satisfies\n  $\\norm{\\pi(x) - x} \\le \\norm{x\\opt - x} \\le 2r$, and so $\\norm{\\pi(x)} \\le\n  3r$, and defining the set $C = 3r \\ball$,\n  we have $\\dist(x_k, \\mc{X}\\opt) = \\dist(x_k, \\mc{X}\\opt \\cap C)$ for all $k$.\n  Now, fix $\\epsilon > 0$, and let $\\{x_i\\opt\\}_{i = 1}^N$ be\n  an $\\epsilon$-net of $C \\cap \\mc{X}\\opt$, where $N < \\infty$.\n  As $x_k \\in C$ for all $k$,\n  \\begin{equation*}\n    \\min_{i \\in [N]}\n    \\norm{x_k - x_i\\opt} - \\epsilon\n    \\le \\dist(x_k, \\mc{X}\\opt \\cap C)\n    = \\dist(x_k, \\mc{X}\\opt) \\cas 0,\n  \\end{equation*}\n  and $\\min_{i \\in [N]} \\norm{x_k - x_i\\opt} \\cas \\min_{i \\in [N]}\n  V(x_i\\opt)$. Thus for any $\\epsilon > 0$,\n  there exists $x_\\epsilon \\in \\mc{X}\\opt \\cap C$ such that\n  $V(x_\\epsilon) \\le \\epsilon$, so that\n  $\\inf_{x \\in C \\cap \\mc{X}\\opt} V(x) = 0$. By the continuity of $V$,\n  the infimum is attained and there is\n  $x\\opt$ such that $V(x\\opt) = 0$. The sequence $x_k$ thus converges to\n  this $x\\opt \\in \\mc{X}\\opt$,\n  and $F(x_k) \\cas F(x\\opt) = F\\opt$.\n\\end{proof-of-proposition}","start":49918,"end":54863}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:single-dingle"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"条件期望距离递推","source_units":[{"raw_tex":"\\begin{proof-of-proposition}[\\ref{proposition:convergence-from-boundedness}]\n  By Assumption~\\ref{assumption:very-weak-moment}, $\\E[\\norm{f'(x;\n      \\statrv)}^2] \\le \\growfunc(r)$ for all $x$ such that $\\dist(x,\n  \\mc{X}\\opt) \\le r$.  Lemma~\\ref{lemma:single-recentered-progress} implies\n  that for any $x\\opt \\in\\mc{X}\\opt$,\n  \\begin{align}\n    \\label{eqn:single-dingle}\n    \\E[\\norms{x_{k+1} - x\\opt}^2 \\mid \\mc{F}_{k-1}]\n    %% & \\le \\dist(x_k, \\mc{X}\\opt)^2 - 2 \\stepsize_k (F(x_k) - F\\opt)\n    %% + \\stepsize_k^2\n    %% \\sup_{\\norm{x - x\\opt} \\le \\dist(x_k, \\mc{X}\\opt)}\n    %% \\E[\\norms{f'(x; \\statrv)}^2] \\\\\n    & \\le \\norms{x_k - x\\opt}^2 - 2 \\stepsize_k (F(x_k) - F\\opt)\n    + \\stepsize_k^2\n    \\growfunc(\\dist(x_k, \\mc{X}\\opt)).\n  \\end{align}\n","start":49918,"end":50672}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"有界事件上的误差可和","source_units":[{"raw_tex":"  On the event that $\\sup_k \\dist(x_k, \\mc{X}\\opt) < \\infty$, we have\n  $\\sum_k \\stepsize_k^2 \\growfunc(\\dist(x_k, \\mc{X}\\opt)) < \\infty$, and so\n","start":50672,"end":50818}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"固定点距离极限与间隙可和","source_units":[{"raw_tex":"  the Robbins-Siegmund Lemma~\\ref{lemma:robbins-siegmund} implies\n  for any $x\\opt \\in \\mc{X}\\opt$,\n  there is some (random) $V(x\\opt) < \\infty$ such that\n  $\\norms{x_k - x\\opt} \\cas V(x\\opt)$, and\n  $\\sum_k \\stepsize_k (F(x_k) - F(x\\opt)) < \\infty$.\n  \n","start":50818,"end":51072}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"投影导出最优集距离递推","source_units":[{"raw_tex":"  We now show that $\\dist(x_k, \\mc{X}\\opt) \\cas 0$.\n  Letting $x\\opt$ be the\n  projection of $x_k$ onto $\\mc{X}\\opt$,\n  inequality~\\eqref{eqn:single-dingle} gives $\\E[\\dist(x_{k+1},\n    \\mc{X}\\opt)^2 \\mid \\mc{F}_{k-1}] \\le \\dist(x_k, \\mc{X}\\opt)^2 - 2\n  \\stepsize_k(F(x_k) - F\\opt) + \\stepsize_k^2 \\growfunc(\\dist(x_k,\n  \\mc{X}\\opt))$. Thus using Lemma~\\ref{lemma:robbins-siegmund}, there exists\n","start":51072,"end":51468}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"最优集距离存在极限","source_units":[{"raw_tex":"  \\mc{X}\\opt))$. Thus using Lemma~\\ref{lemma:robbins-siegmund}, there exists\n  a random $D_\\infty < \\infty$ such that $\\dist(x_k, \\mc{X}\\opt) \\cas\n  D_\\infty$.\n","start":51391,"end":51551}]},{"id":"S037","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"局部环带间隙函数","source_units":[{"raw_tex":"  For $\\epsilon_i \\in \\R_+$, define the gap function\n  \\begin{equation*}\n    \\gapfunc(x\\opt, \\epsilon_1, \\epsilon_2)\n    \\defeq \\inf_{x \\in \\mc{X}} \\left\\{F(x) - F\\opt\n    \\mid \\epsilon_1 \\le \\norm{x - x\\opt} \\le 4 \\epsilon_1,\n    \\dist(x, \\mc{X}\\opt) \\ge \\epsilon_2\n    \\right\\}.\n  \\end{equation*}\n","start":51551,"end":51850}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"紧环带上的正间隙","source_units":[{"raw_tex":"  The set $\\{x \\in \\mc{X} \\mid \\norm{x - x\\opt} \\in\n  [\\epsilon_1, 4\\epsilon_1], \\dist(x, \\mc{X}\\opt) \\ge \\epsilon_2\\}$ is\n  compact, so the infimum in the definition of the gap $\\gapfunc$ is\n  attained and $\\gapfunc(x\\opt, \\epsilon_1, \\epsilon_2) > 0$ for\n  $\\epsilon_i > 0$.  When the a.s.\\ limits\n","start":51850,"end":52150}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"正极限迫使尾部进入环带","source_units":[{"raw_tex":"  $\\epsilon_i > 0$.  When the a.s.\\ limits\n  $V(x\\opt)$ and $D_\\infty$ are positive,\n  there exists a (random) $K <\\infty$ such that\n  $\\norm{x_k - x\\opt} \\in [V(x\\opt)/2, 2 V(x\\opt)]$ and\n  $\\dist(x_k, \\mc{X}\\opt) \\ge D_\\infty / 2$ for all\n  $k \\ge K$. Thus with probability 1,\n","start":52107,"end":52386}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"发散步长排除正距离极限","source_units":[{"raw_tex":"  \\begin{equation*}\n    \\infty > \\sum_k \\stepsize_k (F(x_k) - F\\opt)\n    %% \\ge \\sum_{k \\ge K}\n    %% \\stepsize_k \\inf_{x \\in \\mc{X}_\\epsilon} (F(x) - F\\opt)\n    \\ge \\sum_{k \\ge K} \\stepsize_k \\gapfunc(x\\opt, V(x\\opt) / 2, D_\\infty / 2).\n  \\end{equation*}\n  As $\\sum_k \\stepsize_k = \\infty$ and\n  $\\gapfunc(x\\opt, \\epsilon_1, \\epsilon_2) > 0$ for\n  $\\epsilon_i > 0$, we have $D_\\infty = 0$\n  with probability 1.\n","start":52386,"end":52798}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"极限距离函数为利普希茨","source_units":[{"raw_tex":"\n  Finally, we show the sequence $x_k$ converges.\n  We begin by noting\n  that $V(x\\opt) = \\lim_k \\norm{x_k - x\\opt}$ is $1$-Lipschitz.\n  Indeed, let $x_1\\opt, x_2\\opt \\in \\mc{X}\\opt$. By\n  the a.s.\\ limit definition of $V$,\n  \\begin{equation*}\n    \\left|V(x_1\\opt) - V(x_2\\opt)\\right|\n    \\le\n    % \\limsup_k \\left|\\norm{x_k - x_1\\opt} - V(x_1\\opt)\\right|\n    % + \\limsup_k \\left|\\norm{x_k - x_2\\opt} - V(x_2\\opt)\\right|\n    \\limsup_k \\left|\\norm{x_k - x_1\\opt} - \\norm{x_k - x_2\\opt}\\right|\n    \\le \\norm{x_1\\opt - x_2\\opt}.\n  \\end{equation*}\n","start":52798,"end":53342}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"固定点距离极限推出点列有界","source_units":[{"raw_tex":"  Let us now show that for some $x\\opt \\in \\mc{X}\\opt$, we have $V(x\\opt) =\n  0$.  Let $\\ball = \\{x \\in \\R^n \\mid \\norm{x} \\le 1\\}$ be the ball.  As\n  $V(x\\opt) < \\infty$, there exists a (random) $r < \\infty$ such that the\n  $x_k \\in r \\ball$ for all $k$. As $\\dist(x_k, \\mc{X}\\opt) \\cas 0$, the\n","start":53342,"end":53638}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"有界球内存在最优点","source_units":[{"raw_tex":"  $x_k \\in r \\ball$ for all $k$. As $\\dist(x_k, \\mc{X}\\opt) \\cas 0$, the\n  compactness of $\\ball$ implies that $r \\ball \\cap \\mc{X}\\opt \\neq\n  \\emptyset$.  Let $x\\opt \\in r\\ball \\cap \\mc{X}\\opt$.  For any $x \\in\n","start":53565,"end":53777}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"投影落在固定紧球","source_units":[{"raw_tex":"  \\emptyset$.  Let $x\\opt \\in r\\ball \\cap \\mc{X}\\opt$.  For any $x \\in\n  r\\ball$, the projection $\\pi(x)$ of $x$ onto $\\mc{X}\\opt$ satisfies\n  $\\norm{\\pi(x) - x} \\le \\norm{x\\opt - x} \\le 2r$, and so $\\norm{\\pi(x)} \\le\n  3r$, and defining the set $C = 3r \\ball$,\n  we have $\\dist(x_k, \\mc{X}\\opt) = \\dist(x_k, \\mc{X}\\opt \\cap C)$ for all $k$.\n","start":53706,"end":54048}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"有限网近似最优集距离","source_units":[{"raw_tex":"  Now, fix $\\epsilon > 0$, and let $\\{x_i\\opt\\}_{i = 1}^N$ be\n  an $\\epsilon$-net of $C \\cap \\mc{X}\\opt$, where $N < \\infty$.\n  As $x_k \\in C$ for all $k$,\n  \\begin{equation*}\n    \\min_{i \\in [N]}\n    \\norm{x_k - x_i\\opt} - \\epsilon\n    \\le \\dist(x_k, \\mc{X}\\opt \\cap C)\n    = \\dist(x_k, \\mc{X}\\opt) \\cas 0,\n  \\end{equation*}\n","start":54048,"end":54374}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"有限网极限迫使下确界为零","source_units":[{"raw_tex":"  and $\\min_{i \\in [N]} \\norm{x_k - x_i\\opt} \\cas \\min_{i \\in [N]}\n  V(x_i\\opt)$. Thus for any $\\epsilon > 0$,\n  there exists $x_\\epsilon \\in \\mc{X}\\opt \\cap C$ such that\n  $V(x_\\epsilon) \\le \\epsilon$, so that\n  $\\inf_{x \\in C \\cap \\mc{X}\\opt} V(x) = 0$. By the continuity of $V$,\n","start":54374,"end":54656}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"零极限距离给出点列收敛","source_units":[{"raw_tex":"  $\\inf_{x \\in C \\cap \\mc{X}\\opt} V(x) = 0$. By the continuity of $V$,\n  the infimum is attained and there is\n  $x\\opt$ such that $V(x\\opt) = 0$. The sequence $x_k$ thus converges to\n  this $x\\opt \\in \\mc{X}\\opt$,\n","start":54585,"end":54799}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"目标函数值收敛","source_units":[{"raw_tex":"  and $F(x_k) \\cas F(x\\opt) = F\\opt$.\n\\end{proof-of-proposition}","start":54799,"end":54863}]},{"id":"S049","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S049"],"declared_external":false,"source_only":true,"title":"可测次梯度选择背景","source_units":[{"raw_tex":"list two assumptions that we frequently make. As $f(\\cdot; \\statval)$ is\nsubdifferentiable on $\\mc{X}$, there exist measurable selections $f'(x;\n\\statval) \\in \\partial f(x; \\statval)$, and $\\E[\\partial f(x; \\statrv)] =\n\\partial F(x)$ for $x \\in \\mc{X}$ (cf.~\\cite[Sec.~2]{Bertsekas73}).\n","start":10061,"end":10348}]},{"id":"S050","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S050"],"declared_external":true,"declaration_text":["原始许可前提中的 standard convex optimality and subgradient inequalities；三点距离恒等式为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"凸最优性与次梯度工具","source_units":[]},{"id":"S051","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S051"],"declared_external":true,"declaration_text":["原始许可前提中的 Young's inequality。"],"source_only":false,"title":"杨氏二次内积界","source_units":[]},{"id":"S052","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S052"],"declared_external":true,"declaration_text":["原始许可前提中的 conditional expectation under independent sampling。"],"source_only":false,"title":"独立采样的条件期望","source_units":[]},{"id":"S053","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S053"],"declared_external":true,"declaration_text":["原始许可前提中的 standard finite-dimensional projection arguments。"],"source_only":false,"title":"闭凸投影与距离性质","source_units":[]},{"id":"S054","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S054"],"declared_external":true,"declaration_text":["原始许可前提中的 standard finite-dimensional compactness and continuity arguments。"],"source_only":false,"title":"有限维紧性与有限网","source_units":[]},{"id":"S055","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S055"],"declared_external":true,"declaration_text":["原始许可前提中的 standard finite-dimensional separability arguments；作者未展开其应用，作为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"可分性与共同概率一事件","source_units":[]},{"id":"S056","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S056"],"declared_external":true,"declaration_text":["原始许可前提中的 standard convex subgradient inequalities and finite-dimensional continuity arguments；具体期望目标正则性为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"目标连续性的适用背景","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S021","S023":"S021","S024":"S021","S025":"S021","S026":"S026","S027":"S026","S028":"S026","S029":"S026","S030":"S026","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056"}}}}
{"record_id":"2412.18929v5","paper":{"title":"Alternating Gradient-Type Algorithm for Bilevel Optimization with Inexact Lower-Level Solutions via Moreau Envelope-based Reformulation","authors":["Bai, Xiaoning","Zeng, Shangzhi","Zhang, Jin","Zhang, Lezhi"],"arxiv_id":"2412.18929","version":"v5","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/2412.18929v5","pdf_url":"https://arxiv.org/pdf/2412.18929v5","doi":"10.1137/24M1721049"},"context":{"raw_tex":"Bilevel optimization problems are a class of hierarchical optimization problems where the feasible region is implicitly defined by the solution set of a lower-level optimization problem. In this work, we focus on a bilevel optimization problem which has a convex composite lower-level problem,\n\\begin{equation}\\label{eq1}\n\t\\begin{aligned}\n\t\t\\min\\limits_{x \\in X, y \\in Y} &\\ F(x,y)\\\\\n\t\t{\\rm s.t.}\\quad &\\ y \\in S(x),\n\t\\end{aligned}\n\\end{equation}\nwhere $S(x)$ denotes the set of optimal solutions to the lower-level problem:\n\\begin{equation}\\label{LL}\n\t\\min\\limits_{y \\in Y}\\ \\varphi(x,y) := f(x,y) + g(x,y).\n\\end{equation}\nHere, $X \\subset \\Re^n$ and $Y \\subset \\Re^m$ are closed convex sets, $f(x, y) : \\Re^n \\times \\Re^m \\rightarrow \\Re$ is smooth and convex with respect to $y$ on $Y$ for any $x \\in X$, and $g(x, y) : \\Re^n \\times \\Re^m \\rightarrow \\Re$ is convex but potentially nonsmooth with respect to $y$ on $Y$ for any $x \\in X$. Additionally, we assume that $g(x, \\cdot) : \\Re^m \\rightarrow \\Re$ is proximal-friendly, enabling efficient evaluation of its proximal operator.\n\n\nTo overcome this challenge, in this paper, we consider the following Moreau envelope-based reformulation of the bilevel optimization problem \\eqref{eq1}, first introduced in \\cite{gao2024},\n\\begin{equation}\\label{reformualtion_Moreau}\n\t({\\rm VP})_\\gamma \\quad\\quad\n\t\\min\\limits_{x\\in X,y\\in Y}\\  F(x,y) \\quad\t\t{\\rm s.t.}\\quad\\ \\varphi (x,y)-v_{\\gamma} (x,y)\\le 0,\n\\end{equation}\nwhere $v_{\\gamma}(x, y)$ is the Moreau envelope associated with the lower-level problem,\n\\begin{equation}\\label{def_vg}\n\tv_{\\gamma}(x,y):=\\inf\\limits_{\\theta\\in Y}\\varphi(x,\\theta)+\\frac{1}{2\\gamma}\\|\\theta-y\\|^2,\n\\end{equation}\nwhere $\\gamma > 0$. When the lower-level problem is convex with respect to $y$,  this reformulation is equivalent to the original bilevel optimization problem \\eqref{eq1} (see \\cite[Theorem 1]{gao2024}).  Directional optimality conditions for $({\\rm VP})_\\gamma$ are discussed in \\cite{bai2025optimality}. However, as shown in \\cite{gao2024} and \\cite[Proposition 7]{ye2023}, classical constraint qualification (CQ) conditions do not hold at any feasible point of $({\\rm VP})_\\gamma$, motivating us to study the following relaxed approximation problem,\n\\begin{equation}\\label{reformualtion_Moreau_relax}\n\t({\\rm VP})_\\gamma^{\\epsilon} \\quad\\quad\n\t\\min\\limits_{x\\in X,y\\in Y}\\  F(x,y) \\quad\t\t{\\rm s.t.}\\quad\\ \\varphi (x,y)-v_{\\gamma} (x,y)\\le \\epsilon,\n\\end{equation}\n\n\nLet $\\mathbb{R}^n$ denote the $n$-dimensional Euclidean space, and let $\\mathbb{B}$ denote the closed unit ball centered at the origin. The nonnegative and positive orthants in $\\mathbb{R}^n$ are denoted by $\\mathbb{R}^n_+$ and $\\mathbb{R}^n_{++}$, respectively. The standard inner product and Euclidean norm are denoted by $\\langle \\cdot, \\cdot \\rangle$ and $\\|\\cdot\\|$, respectively.\nFor a vector \\( x \\in \\mathbb{R}^n \\) and a closed convex set \\( S \\subset \\mathbb{R}^n \\), the distance from \\( x \\) to \\( S \\) is defined as \\(\\mathrm{dist}(x, S) = \\min_{y \\in S} \\|x - y\\|\\). The indicator function \\(\\delta_S: \\mathbb{R}^n \\to \\mathbb{R}\\cup\\{+\\infty\\}\\) is defined by \\(\\delta_S(x) = 0\\) if \\(x\\in S\\) and \\(\\delta_S(x) = +\\infty\\) otherwise. Moreover, \\(\\mathcal{N}_S(x)\\) denotes the normal cone to \\(S\\) at \\(x\\). The Euclidean projection operator onto \\( S \\) is denoted by \\(\\mathrm{Proj}_S\\). The Cartesian product of two sets \\( X \\) and \\( Y \\) is denoted by \\( X \\times Y \\).\nLet \\( h: \\mathbb{R}^n \\to \\mathbb{R} \\cup \\{\\infty\\} \\) be a proper closed function. The proximal mapping of \\( h\\) is defined as $\\operatorname{Prox}_h(y):=\\arg\\min_{\\theta \\in \\Re^m}\\left\\{h(\\theta)+\\|\\theta-y\\|^2 / 2\\right\\}$. The Fr\\'{e}chet (regular) subdifferential \\(\\hat{\\partial} h\\) and the Mordukhovich (limiting) subdifferential \\(\\partial h\\) at a point \\( \\bar{z} \\in \\mathrm{dom}\\, h := \\{z \\mid h(z) < \\infty\\} \\) are defined as:\n\\[\n\\begin{aligned}\n\t\\hat{\\partial} h(\\bar{z}) &= \\left\\{ v \\in \\mathbb{R}^n \\;\\middle|\\; h(z) \\geq h(\\bar{z}) + \\langle v, z - \\bar{z} \\rangle + o(\\|z - \\bar{z}\\|), \\; \\forall z \\in \\mathbb{R}^n \\right\\}, \\\\\n\t\\partial h(\\bar{z}) &= \\left\\{ v \\in \\mathbb{R}^n \\;\\middle|\\; \\exists z_k \\xrightarrow{h} \\bar{z}, \\; \\exists v_k \\in \\hat{\\partial} h(z_k), \\; v_k \\to v \\right\\},\n\\end{aligned}\n\\]\nwhere \\( o(z) \\) denotes a function satisfying \\( o(z)/z\\to 0 \\) as \\( z \\downarrow 0 \\), and \\( z_k \\xrightarrow{h} \\bar{z} \\) indicates \\( z_k \\to \\bar{z} \\) and \\( h(z_k) \\to h(\\bar{z}) \\) as \\( k \\to \\infty \\). If \\( \\bar{z} \\notin \\mathrm{dom}\\, h \\), we define \\(\\partial h(\\bar{z}) = \\emptyset\\).\n\n\nThis part outlines the assumptions on the problem data for the bilevel optimization problem studied in this work. First, we assume that the upper-level objective $F$ is smooth and bounded below.\n\n\\begin{assumption}\\label{asup1}\n\tThe function \\(F : \\mathbb{R}^n \\times \\mathbb{R}^m \\to \\mathbb{R}\\) is bounded below on \\(X \\times Y\\), i.e., \\(\\underline{F} := \\inf_{(x, y) \\in X \\times Y} F(x, y) > -\\infty\\). \\(F\\) is continuously differentiable, and its gradients \\(\\nabla_x F\\) and \\(\\nabla_y F\\) are \\(L_{F_x}\\)- and \\(L_{F_y}\\)-Lipschitz continuous on \\(X \\times Y\\), respectively.\n\\end{assumption}\n\nThroughout this paper, we focus on cases where the lower-level problem is a convex optimization problem.\n\n\\begin{assumption}\\label{asup2}\n\tFor any \\(x \\in X\\), the functions \\(f(x, \\cdot)\\) and \\(g(x, \\cdot)\\) are convex and defined on the closed convex set \\(Y\\). \n\\end{assumption} \n\nSmoothness and weak convexity conditions are required for the lower-level objective functions. We recall that a function \\(h : \\mathbb{R}^n \\times \\mathbb{R}^m \\to \\mathbb{R}\\) is \\((\\rho_1, \\rho_2)\\)-weakly convex on a convex set \\(X \\times Y\\) if \\(h(x, y) + \\frac{\\rho_1}{2} \\|x\\|^2 + \\frac{\\rho_2}{2} \\|y\\|^2\\) is convex on \\(X \\times Y\\). This notion generalizes convexity (\\(\\rho_1 = \\rho_2 = 0\\)). Weakly convex functions \\cite{nurminskii1973quasigradient}, extends the class of convex functions to include many nonconvex functions. For weakly convex functions, the Fréchet (regular) subdifferential coincides with both the limiting subdifferential and the Clarke subdifferential (see, e.g., \\cite[Proposition 3.1 and Theorem 3.6]{ngai2000approximate}).\n\n\\begin{assumption}\\label{asup3}\n\tThe lower-level objective functions \\(f(x, y)\\) and \\(g(x, y)\\) satisfy:\n\t\\begin{itemize}\n\t\t\\item[(1)] \\(f(x, y)\\) is continuously differentiable on \\(X \\times Y\\), and its gradients \\(\\nabla_x f\\) and \\(\\nabla_y f\\) are \\(L_{f_x}\\)- and \\(L_{f_y}\\)-Lipschitz continuous on \\(X \\times Y\\), respectively;\n\t\t\\item[(2)] \\(g(x, y)\\) is continuous on \\(X \\times Y\\), \\(\\nabla_x g(x, y)\\) exists, and \\(\\nabla_x g(\\cdot, y)\\) is \\(L_{g_1}\\)-Lipschitz continuous on \\(X\\) for any \\(y \\in Y\\), while \\(\\nabla_x g(x, \\cdot)\\) is \\(L_{g_2}\\)-Lipschitz continuous on \\(Y\\) for any \\(x \\in X\\);\n\t\t\\item[(3)] \\(f(x, y)\\) is \\((\\rho_{f_1}, \\rho_{f_2})\\)-weakly convex on \\(X \\times Y\\), and \\(g(x, y)\\) is \\((\\rho_{g_1}, \\rho_{g_2})\\)-weakly convex on \\(X \\times Y\\).\n\t\\end{itemize}\n\\end{assumption}\n\n\nUnder Assumption \\ref{asup2}, the solution mapping\n\\begin{equation}\\label{proximal_LL}\n\tS_{\\gamma}(x, y) := \\mathrm{argmin}_{\\theta \\in Y} \\left\\{ f(x, \\theta) + g(x, \\theta) + \\frac{1}{2\\gamma}\\|\\theta - y\\|^2 \\right\\}\n\\end{equation}\nis well-defined and single-valued  on $X \\times \\mathbb{R}^m$. We denote its unique solution by $\\theta_\\gamma^*(x, y)$.\nBelow, we summarize sensitivity results for $v_\\gamma(x, y)$ from Theorems 2, 3, and 5 of \\cite{gao2024}.\n\n\n\t\\begin{equation}\\label{valuefinc}\n\t\t\\nabla v_\\gamma(x, y) = \\Big( \\nabla_x f(x, \\theta_\\gamma^*(x, y)) + \\nabla_x g(x, \\theta_\\gamma^*(x, y)), \\left({y - \\theta_\\gamma^*(x, y)}\\right)/{\\gamma} \\Big).\n\t\\end{equation}\n\n\n\\begin{proposition}\\label{partial_grad_v}\n\tSuppose Assumptions \\ref{asup2} and \\ref{asup3} hold. For $\\gamma \\in (0, 1/(\\rho_{f_2} + \\rho_{g_2}))$, the function $v_\\gamma(x, y)$ is $\\rho_{v_1}$-weakly convex with respect to $x$ on $X$ for any $y \\in Y$ with $\\rho_{v_1} \\geq \\rho_{f_1} + \\rho_{g_1}$. It is also convex with respect to $y$ on $\\mathbb{R}^m$ for any $x \\in X$. Moreover, $v_\\gamma(x, y)$ is differentiable with gradient given by \\eqref{valuefinc} on $X \\times \\mathbb{R}^m$, and $\\theta_\\gamma^*(x, y)$ and $\\nabla v_\\gamma(x, y)$ are continuous on $X \\times \\mathbb{R}^m$.\n\\end{proposition}\n\n\\begin{definition}\n\tLet $(\\bar{x},\\bar{y})$ be a feasible solution of problem $({\\rm VP})_{\\gamma}^{\\epsilon}$ with $\\epsilon > 0$. We say that $(\\bar{x},\\bar{y})$ is a stationary/KKT point of problem $({\\rm VP})_{\\gamma}^{\\epsilon}$ if there exists a multiplier $\\lambda \\ge 0$ such that\n\t\\begin{equation*}\n\t\t\\begin{cases}\n\t\t\t0\\in \\nabla F(\\bar{x},\\bar{y}) + \\lambda \\left(\\nabla f(\\bar{x},\\bar{y}) + \\partial g(\\bar{x},\\bar{y}) - \\nabla v_{\\gamma}(\\bar{x},\\bar{y})\\right) + \\mathcal{N}_{X\\times Y}(\\bar{x},\\bar{y}), \\\\\n\t\t\tf(\\bar{x},\\bar{y}) + g(\\bar{x},\\bar{y}) - v_{\\gamma}(\\bar{x},\\bar{y}) \\leq \\epsilon, \\qquad \\lambda \\left(f(\\bar{x},\\bar{y}) + g(\\bar{x},\\bar{y}) - v_{\\gamma}(\\bar{x},\\bar{y})-\\epsilon\\right) = 0.\n\t\t\\end{cases}\n\t\\end{equation*}\n\\end{definition}\n\n\nIn this section, we propose a new gradient-type algorithm, Alternating Gradient-type algorithm with Inexact Lower-level Solutions (AGILS), to solve ${\\rm (VP)}_\\gamma^\\epsilon$. Given that $f(x, y) + g(x,y) - v_{\\gamma}(x, y)$ is always nonnegative, a natural approach is to consider the following penalized problem,\n\\begin{equation}\\label{penprob}\n\t\\min_{x \\in X, y \\in Y}\\psi_{p_k}\t(x, y) := \\frac{1}{p_k}F(x, y) + f(x, y) + g(x,y) - v_{\\gamma}(x, y),\n\\end{equation}\nwhere $p_k$ is the penalty parameter at iteration $k$. Due to the nonsmoothness of $g(x,y)$ with respect to $y$ and the availability of its proximal operator, it is natural to adopt a proximal alternating linearization method \\cite{bolte2014proximal} to construct the algorithm. Given the current iterate $(x^k, y^k)$, linearizing the smooth components of $\\psi_{p_k}\t(x, y)$ requires evaluating gradients, including $\\nabla v_{\\gamma}(x, y)$ as provided in \\cref{grad_v}. Calculating the exact gradient  $\\nabla v_{\\gamma}(x^k, y^k)$ requires solving the proximal lower-level problem in \\eqref{proximal_LL} to obtain $\\theta_\\gamma^*(x^k, y^k)$. However, this solution typically does not have a closed-form expression, and solving it requires an iterative optimization solver, which can be computationally expensive. To address this, we propose an inexact gradient approximation by substituting $\\theta_{\\gamma}^*$ with an inexact estimate $\\theta^k$.\n\nThe proposed algorithm alternates between updating $y$ and $x$ while refining the inexact approximation of $\\theta_\\gamma^*(x^k, y^k)$. At each iteration $k=0,1,\\ldots$, given the current iterate $(x^k, y^k)$, along with the feasibility-corrected iterate $\\tilde{y}^k$ and corresponding $\\tilde{\\theta}^k$ (constructed from $y^k$ and  $\\theta^{k}$ via the procedure detailed below), the next iterate $(x^{k+1}, y^{k+1})$ and the refined approximation $\\theta^{k+1}$ are updated as follows.\n\n{\\bf Update $y$ (Fixing $x$).}\nWe compute the update direction for $y$,\n\\begin{equation}\\label{dy}\n\td_y^k:= \\frac{1}{p_k} \\nabla_y F\\left(x^{k}, \\tilde{y}^k\\right)+\\nabla_y f\\left(x^{k}, \\tilde{y}^k\\right)-\\frac{\\tilde{y}^k-\\tilde{\\theta}^{k}}{\\gamma},\n\\end{equation}\nwhere $(\\tilde{y}^k-\\tilde{\\theta}^{k})/{\\gamma}$ is an inexact approximation of $\\nabla_y v_{\\gamma}(x^{k}, \\tilde{y}^k) =( {\\tilde{y}^k} -  \\theta_{\\gamma}^*(x^{k}, \\tilde{y}^k))/ \\gamma $ (see (\\ref{valuefinc})). Using $\td_y^k$, the variable $y$ is updated as\n\\begin{equation}\\label{update_y}\n\ty^{k+1} = \\underset{y \\in Y}{\\mathrm{argmin}} ~ \\left\\langle d_y^k, y - \\tilde{y}^{k} \\right\\rangle + g(x^k, y) + \\frac{1}{2\\beta_k} \\left\\| y - \\tilde{y}^k \\right\\|^2, \n\\end{equation}\nwhere $\\beta_k>0$ is the stepsize. \nThis can also be expressed using the proximal operator\n\\begin{equation}\\label{update_y2}\n\ty^{k+1} = \\operatorname{Prox}_{\\beta_k\\tilde{g}(x^{k},\\cdot)}\\left(\\tilde{y}^k-\\beta_k d^k_y\\right),\n\\end{equation}\nwhere $\\tilde{g}(x,y) := g(x,y) + \\delta_Y(y)$. \n\n{\\bf Find inexact approximation $\\theta^{k+1/2}$.}\nAfter updating $y^{k+1}$, we compute an inexact approximation $\\theta^{k+1/2}$ for $\\theta_{\\gamma}^*(x^k, y^{k+1})$ by solving\n\\begin{equation}\\label{theta_y}\n\t\\theta^{k+1/2} \\approx \t\\underset{\\theta \\in Y}{\\mathrm{argmin}} \\left\\{f(x^{k},\\theta) + g(x^{k}, \\theta) + \\frac{1}{2\\gamma}\\left\\|\\theta-y^{k+1}\\right\\|^2\\right\\}, \n\\end{equation}\nsubject to either an absolute or a relative inexactness criterion\n\\begin{equation}\\label{inexacty1}\n\t\\begin{aligned}\n\t\t\\mathcal{G}(\\theta^{k+1/2}, x^k,y^{k+1}) \\le s_k, \\quad \\text{or} \\quad \\mathcal{G}(\\theta^{k+1/2}, x^k,y^{k+1}) \\le \\tau_k \\mathcal{G}(\\theta^{k-1}, x^{k-1},y^{k-1}),\n\t\\end{aligned}\n\\end{equation}\nwhich can be compactly expressed as\n\\[\n\\mathcal{G}(\\theta^{k+1/2}, x^{k}, y^{k+1})\n\\le\n\\max\\left\\{ s_k,\\ \\tau_k\\,\\mathcal{G}(\\theta^{k-1}, x^{k-1}, y^{k-1}) \\right\\}.\n\\]\nThe parameters $s_k$ and $\\tau_k$ control the solution's inexactness, and are required to satisfy $\\sum_{k=0}^{\\infty}s_k^2<\\infty$ and $\\tau_k:=\\tau_0/(k+1)^{p_\\tau}$ with constants $\\tau_0 > 0$ and $p_\\tau >0$. The prox-gradient residual $\\mathcal{G}(\\theta, x,y)$ quantifies the inexactness and is defined as\n\\[\n\\mathcal{G}(\\theta, x,y):= \t\\left\\| \\theta -\\operatorname{Prox}_{\\eta \\tilde{g}\\left(x, \\cdot\\right)}\\left(\\theta-\\eta\\left(\\nabla_y f\\left(x, \\theta\\right) + (\\theta -y)/ {\\gamma}\\right)\\right)\\right\\|,\n\\]\nwhere $\\eta > 0$. Note that $\\mathcal{G}(\\theta, x,y) = 0$ if and only if $\\theta =\\theta_{\\gamma}^*(x,y)$. \n\n\n{\\bf Update $x$ (Fixing $y$).}\nWe compute the update direction for $x$,\n\\begin{equation}\\label{dx}\n\td_x^k:= \\frac{1}{p_k} \\nabla_xF\\left(x^k, y^{k+1}\\right)+\\nabla_x \\varphi \\left(x^k, y^{k+1}\\right)-\\nabla_x f\\left(x^k, \\theta^{k+1/2}\\right)-\\nabla_x g\\left(x^k, \\theta^{k+1/2}\\right),\n\\end{equation}\nwhere $\\nabla_x f\\left(x^k, \\theta^{k+1/2}\\right)+\\nabla_x g\\left(x^k, \\theta^{k+1/2}\\right)$ is an approximation to \\( \\nabla_x v_{\\gamma}(x^k, y^{k+1}) = \\nabla_x f(x^k, \\theta^*_{\\gamma}(x^k, y^{k+1}))+\\nabla_x g(x^k, \\theta^*_{\\gamma}(x^k, y^{k+1}))\\) (see \\eqref{valuefinc}).\nUsing the direction $d_x^k$, we update $x$ with stepsize $\\alpha_k>0$ as\n\\begin{equation}\\label{update_x}\n\tx^{k+1}=\\operatorname{Proj}_X\\left(x^k-\\alpha_k d_x^k\\right).\n\\end{equation}\n\n{\\bf Find inexact approximation $\\theta^{k+1}$.}\nAfter updating $x^{k+1}$, a new inexact approximation $\\theta^{k+1}$ of $\\theta^*_{\\gamma}(x^{k+1}, y^{k+1})$  is computed by approximately solving\n\\[\n\\theta^{k+1} \\approx \t\\underset{\\theta \\in Y}{\\mathrm{argmin}} \\left\\{f(x^{k+1},\\theta) + g(x^{k+1}, \\theta) + \\frac{1}{2\\gamma}\\left\\|\\theta-y^{k+1}\\right\\|^2\\right\\}, \n\\]\nsuch that $\\theta^{k+1}$ satisfies either an absolute or a relative inexactness criterion\n\\begin{equation}\\label{inexacty2}\n\t\\begin{aligned}\n\t\t\\mathcal{G}(\\theta^{k+1}, x^{k+1},y^{k+1}) \\le s_{k+1}, \\quad \\text{or} \\quad  \\mathcal{G}(\\theta^{k+1}, x^{k+1},y^{k+1}) \\le \\tau_{k+1}\\mathcal{G}(\\theta^{k}, x^{k},y^{k}).\n\t\\end{aligned}\n\\end{equation}\n\n\n{\\bf Penalty parameter update and feasibility correction.}\nTo encourage feasibility of the constraint $\\varphi(x, y) - v_{\\gamma}(x, y) \\le \\epsilon$, the penalty parameter $p_k$ is updated at each iteration.\nEvaluating the constraint violation requires computing $ v_{\\gamma}(x, y)$, which involves solving the proximal lower-level problem in \\eqref{proximal_LL}. Instead, we use $\\theta^{k+1}$ to estimate the constraint violation as\n\\begin{equation}\\label{def_t}\n\tt^{k+1}=\\max\\left\\{\\varphi(x^{k+1},y^{k+1})-\\varphi(x^{k+1},\\theta^{k+1})-\\frac{1}{2\\gamma}\\| \\theta^{k+1} - y^{k+1}\\|^2-\\epsilon,0\\right\\}.\n\\end{equation}\nThe penalty parameter \\(p_k\\) is increased to $p_{k+1}=\tp_k + \\varrho_p$ (for some $\\varrho_p > 0$) only when the change in the iterates, $\\Delta_{k+1} := \\left\\| (x^{k+1}, y^{k+1}) - (x^k, \\tilde{y}^k) \\right\\|$, is small relative to the penalty parameter (i.e., $\\Delta_{k+1} < c_p/p_k$ for some $c_p > 0$), while the estimated constraint violation $t^{k+1}$ is large relative to this change (i.e., $t^{k+1} > \\Delta_{k+1}/c_p$).\nThese two conditions can be compactly written as \\begin{equation}\\label{update_p_1}\n\t\\Delta_{k+1} < c_p \\min\\left\\{1/p_k,{t^{k+1}}\\right\\}.\n\\end{equation}\n\nHowever,  simply increasing $p_{k}$ does not always guarantee feasibility. It can sometimes cause the algorithm to stagnate near an infeasible stationary point of the constraint function $\\varphi(x, y) - v_{\\gamma}(x, y)$. \nTo mitigate this, we use the bilevel optimization structure, that the feasible region is the solution set of the convex lower-level problem \\eqref{LL}. We introduce a {\\it feasibility correction procedure} designed to help the iterates escape such stationary points.\n\nThe feasibility correction procedure aims to generate a corrected iterate $\\tilde{y}^{k+1} $ from $y^{k+1}$. It is based on the observation that if $(x^{k+1},y^{k+1})$ is feasible, then $y^{k+1}$ should be a reasonably accurate solution to the lower-level problem \\eqref{LL}  for $x= x^{k+1}$, implying that $\\| \\nabla_y v_{\\gamma}(x^{k+1},y^{k+1})\\| $ should be small. We use the magnitude of $( y^{k+1} - \\theta^{k+1})/\\gamma$ to approximate $\\| \\nabla_y v_{\\gamma}(x^{k+1},y^{k+1})\\| $. The feasibility correction  is triggered if:\n\\begin{equation}\\label{theta_gap_condition}\n\t\\left\\| y^{k+1} - \\theta^{k+1} \\right\\| \\ge \\frac{c_y \\gamma}{p_k},\n\\end{equation}\nwhere $c_y > 0$. Condition \\eqref{theta_gap_condition} indicates that the iterate $(x^{k+1},y^{k+1})$ might be approaching an undesirable stationary point of the constraint function.\n\nIf condition \\eqref{theta_gap_condition} holds, we attempt to correct $y^{k+1}$ by finding a candidate iterate $\\tilde{y}^{k+1} $  as an inexact solution to the lower-level problem \\eqref{LL} with $x= x^{k+1}$ satisfying\n\\begin{equation}\\label{inexacty_y}\n\t\\left\\| \\tilde{y}^{k+1} - \\operatorname{Prox}_{ \\tilde{g}(x^{k+1}, \\cdot)} \\left( \\tilde{y}^{k+1} -  \\nabla_y f(x^{k+1}, \\tilde{y}^{k+1}) \\right) \\right\\| \\le \\frac{c_{\\tilde{y}} }{p_k} \\Delta_{k+1}, \n\\end{equation}\nwhere $c_{\\tilde{y}} > 0$. However, unconditionally accepting such a $\\tilde{y}^{k+1} $ could adversely affect the algorithm's convergence. Therefore, to preserve convergence, $\\tilde{y}^{k+1} $ is accepted only if it satisfies the following descent condition\n\\begin{equation}\\label{decrease_new_y}\n\t\\begin{aligned}\n\t\t&\\frac{1}{p_k}F(x^{k+1}, \\tilde{y}^{k+1}) + \\varphi(x^{k+1}, \\tilde{y}^{k+1}) - \\left( \\varphi(x^{k+1},  \\tilde{\\theta}^{k+1} ) + \\frac{1}{2\\gamma} \\| \\tilde{\\theta}^{k+1} - \\tilde{y}^{k+1} \\|^2 \\right)\\\\\n\t\t\\le\\; &\n\t\t\\frac{1}{p_k}F(x^{k+1}, y^{k+1}) + \\varphi(x^{k+1}, y^{k+1}) - \\left( \\varphi(x^{k+1}, \\theta^{k+1}) + \\frac{1}{2\\gamma} \\| \\theta^{k+1} - y^{k+1} \\|^2 \\right),\n\t\\end{aligned}\n\\end{equation}\nwhere, $\\tilde{\\theta}^{k+1}$ is a new approximation of $\\theta^*_{\\gamma}(x^{k+1}, \\tilde{y}^{k+1})$. We construct $\\tilde{\\theta}^{k+1}$ by first finding a point $\\hat{\\theta}^{k+1}$ that satisfies the inexactness criterion\n\\begin{equation}\\label{inexacty3}\n\t\\mathcal{G}(\\hat{\\theta}^{k+1}, x^{k+1},\\tilde{y}^{k+1}) \\le s_{k+1},\n\t\\quad \\text{or} \\quad\n\t\\mathcal{G}(\\hat{\\theta}^{k+1}, x^{k+1},\\tilde{y}^{k+1})\n\t\\le \\tau_{k+1}\\mathcal{G}(\\theta^{k}, x^{k},y^{k}).\n\\end{equation}\nand subsequently applying a single proximal-gradient step to $\\hat{\\theta}^{k+1}$\n\\[\n\\tilde{\\theta}^{k+1}\n=\n\\operatorname{Prox}_{\\eta \\tilde g(x^{k+1},\\cdot)}\n\\left(\n\\hat{\\theta}^{k+1}\n-\\eta\\left(\n\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n+\\frac{\\hat{\\theta}^{k+1}-\\tilde y^{k+1}}{\\gamma}\n\\right)\n\\right).\n\\]\n\nIf the descent condition \\eqref{decrease_new_y} holds, the algorithm adopts the candidate  $\\tilde{y}^{k+1} $ along with its corresponding $\\tilde{\\theta}^{k+1}$, and keep the penalty parameter unchanged, i.e., $p_{k+1}=\tp_k$. Otherwise, the correction is discarded, meaning $\\tilde{y}^{k+1} = y^{k+1}$ and $\\tilde{\\theta}^{k+1} = \\theta^{k+1}$ and the penalty parameter is increased, i.e., $p_{k+1}=\tp_k + \\varrho_p$. The algorithm then proceeds with the selected pair $(\\tilde{y}^{k+1}, \\tilde{\\theta}^{k+1})$. A detailed description of this feasibility correction procedure is presented in Algorithm \\ref{alg2}.\n\n\n\\begin{algorithm}[htp!]\n\\caption{Alternating Gradient-type algorithm with Inexact Lower-level Solutions (AGILS)}\n\\label{alg1}\n\\begin{algorithmic}[1]\n\t\\STATE{{\\bf Input}: Initial iterates $x^0,\\theta^{-1}, y^0, \\tilde{\\theta}^0, \\tilde{y}^0$, set $(x^{-1}, y^{-1}, \\theta^{-1}) = (x^0, y^0, \\tilde{\\theta}^0)$, stepsizes $\\alpha_k, \\beta_k$, relaxation parameter $\\epsilon$, proximal parameter $\\gamma$, penalty parameters $p_{0}$,  $\\varrho_p$, $c_p$, $c_y$, inexact parameters $\\eta$, $s_{k}$, $\\tau_k$, tolerance $tol$.}\n\t\\FOR{$k=0,\\ 1,\\cdots$}\n\t\\STATE{Construct the direction $d_y^k$ according to \\eqref{dy}, and update $y^{k+1}$ as\n\t\t$$\n\t\ty^{k+1} = \\operatorname{Prox}_{\\beta_k\\tilde{g}(x^{k,\\cdot})}\\left(\\tilde{y}^k-\\beta_k d^k_y\\right).\n\t\t$$}\n\t\n\t\\STATE{Find an inexact solution $\\theta^{k+1/2}$ that satisfies the inexactness criterion \\eqref{inexacty1}.}\n\t\n\t\\STATE{Construct the direction $d_x^k$ according to \\eqref{dx}, and update $x^{k+1}$ as \n\t\t$$ x^{k+1}=\\operatorname{Proj}_X\\left(x^k-\\alpha_k d_x^k\\right). \n\t\t$$}\n\t\n\t\\STATE{Find an inexact solution $\\theta^{k+1}$ that satisfies the inexactness criterion \\eqref{inexacty2}.}\n\t\n\t\\STATE{Calculate $\\Delta_{k+1} := \\left\\| (x^{k+1}, y^{k+1}) - (x^k, \\tilde{y}^k) \\right\\|$ and $t^{k+1}$ according to \\eqref{def_t}. \\\\\n\t\tStop when $k \\ge 1$ and \\(\\max\\{\\Delta_{k+1}, s_{k}, t^{k+1} \\} \\le tol.\\)\n\t}\n\t\\STATE{Update penalty parameter and apply feasibility correction:}\n\t\n\t\\STATE{\\hspace*{10pt}\\textbf{Case 1:}\n\t\t$\\Delta_{k+1} \\geq c_p \\min\\big\\{1/p_k,\\, t^{k+1} \\big\\}$ \\\\\n\t\t\\vspace*{1pt}\n\t\t\\qquad\\qquad\\quad Set \\( p_{k+1} := p_k \\), \\( \\tilde{y}^{k+1} := y^{k+1} \\), \\(\\tilde{\\theta}^{k+1} :=\\theta^{k+1}\\).}\n\t\\vspace*{2pt}\n\t\\STATE{\\hspace*{10pt}\\textbf{Case 2:} \n\t\t$\\Delta_{k+1} < c_p \\min\\big\\{1/p_k,\\, t^{k+1} \\big\\},\\,\\| y^{k+1} - \\theta^{k+1} \\| \\le c_y \\gamma/p_k$\\\\  \n\t\t\\vspace*{1pt}\n\t\t\\qquad\\qquad\\quad Set \\( p_{k+1} := p_k + \\varrho_p \\), \\( \\tilde{y}^{k+1} := y^{k+1} \\), \\(\\tilde{\\theta}^{k+1} :=\\theta^{k+1}\\).}\n\t\\vspace*{2pt}\n\t\n\t\\STATE{\\hspace*{10pt}\\textbf{Case 3:} \n\t\t$\\Delta_{k+1} < c_p \\min\\big\\{1/p_k,\\, t^{k+1} \\big\\},\\,\\| y^{k+1} - \\theta^{k+1} \\| > c_y \\gamma/p_k$\\\\\n\t\t\\vspace*{1pt}\n\t\t\\qquad\\qquad\\quad Apply the feasibility correction procedure (Algorithm~\\ref{alg2}).}\n\\ENDFOR\n\\RETURN $(x^{k+1},\\tilde{y}^{k+1})$\n\\end{algorithmic}\n\\end{algorithm}\n\n\\begin{algorithm}[htp!]\n\\caption{Feasibility Correction Procedure}\n\\label{alg2}\n\\begin{algorithmic}[1]\n\n\\STATE{{\\bf Input}: Current iterates $x^{k+1}, y^{k+1}, \\theta^{k+1}$, $\\Delta_{k+1}$, proximal parameter $\\gamma$, penalty parameters $p_{k}$, $\\varrho_p$, inexact parameters $\\eta$, $s_{k}$, $\\tau_k$, constant $c_{\\tilde{y}}$.}\n\n\\STATE{Compute a candidate $\\tilde{y}^{k+1}$ by inexactly solving the lower-level problem \\eqref{LL} with $x=x^{k+1}$  to satisfy \\eqref{inexacty_y}.\n}\n\n\\STATE{Find an intermediate point $\\hat{\\theta}^{k+1}$ that satisfies the inexactness criterion \\eqref{inexacty3}, and compute $\\tilde{\\theta}^{k+1}=\n\\operatorname{Prox}_{\\eta \\tilde g(x^{k+1},\\cdot)}\n(\n\\hat{\\theta}^{k+1}\n-\\eta(\n\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n+(\\hat{\\theta}^{k+1}-\\tilde y^{k+1})/\\gamma\n)\n)$.}\n\n\\IF{the descent condition \\eqref{decrease_new_y} holds}\n\\STATE{Accept the candidate: Set $p_{k+1} := p_k$.}\n\\ELSE\n\\STATE{Reject the candidate: Set $\\tilde{y}^{k+1} := y^{k+1}$, $\\tilde{\\theta}^{k+1} := \\theta^{k+1}$ and $p_{k+1} := p_k + \\varrho_p$.}\n\\ENDIF\n\\RETURN $(\\tilde{y}^{k+1},\\tilde{\\theta}^{k+1},p_{k+1})$\n\\end{algorithmic}\n\\end{algorithm}\n\n\nWe begin by considering the following lemma, which provides an estimate for the distance between the approximation $\\theta$ and the exact solution $\\theta^*_{\\gamma}(x,y)$ of the proximal lower-level problem, based on the prox-gradient residual $\\mathcal{G}(\\theta, x,y)$ in the inexact criteria \\eqref{inexacty1} and \\eqref{inexacty2}.  This estimate is derived from the $1/\\gamma$-strong convexity and $(1/\\gamma + L_f)$-smoothness of the component $f(x, \\theta) + \\|\\theta - y\\|^2/(2\\gamma)$ with respect to $\\theta$ in the objective of the proximal lower level problem, see, e.g., \\cite[Theorem 3.4 and 3.5]{drusvyatskiy2018error} or \\cite[Proposition 2.2]{mordukhovich2023globally}.\n\\begin{lemma}\\label{error_bound}\n\tSuppose Assumptions \\ref{asup2} and \\ref{asup3} hold, and let $\\gamma > 0$, $\\eta > 0$. Then, there exists $C_\\eta > 0$ such that for any $(x,y) \\in X \\times \\Re^m$, and $\\theta \\in \\Re^m$, it holds that\n\t\\[\n\t\\|\\theta - \\theta^*_{\\gamma}(x,y) \\| \\le C_\\eta  \\left\\|\\theta-\\operatorname{Prox}_{\\eta \\tilde{g}\\left(x, \\cdot\\right)}\\left(\\theta-\\eta\\left(\\nabla_y f\\left(x, \\theta\\right)+ (\\theta-y)/{\\gamma}\\right)\\right)\\right\\|,\n\t\\]\n\twhere $\\theta^*_{\\gamma}(x,y) := \\arg\\min_{\\theta \\in Y} \\{ \\varphi (x,\\theta)+\\frac{1}{2\\gamma}\\|\\theta-y\\|^2 \\}$.\n\\end{lemma}\n\n\nWe now proceed to establish a decreasing property for the proposed method based on the following merit function:\n\\begin{equation*}\n\t\\tilde{\\psi}_{p_{k}}(x,y) :=\\frac{1}{p_k}(F(x,y)-\\underline{F})+ f(x,y) + g(x,y)-v_{\\gamma}(x,y).\n\\end{equation*}\n\nBefore establishing the main decreasing property, we first demonstrate that the descent condition \\eqref{decrease_new_y}, imposed by the feasibility correction procedure (Algorithm \\ref{alg2}), ensures a decrease in $\\tilde{\\psi}_{p_{k}}$ when moving from $y^{k+1}$ to the corrected iterate $\\tilde{y}^{k+1} $.\n\\begin{lemma}\\label{lem0}\n\tSuppose Assumptions \\ref{asup2} and \\ref{asup3} hold. Then, there exists $C_\\gamma>0$,  such that the sequence $\\{(x^k,y^k,\\tilde{y}^k)\\}$ generated by AGILS (\\cref{alg1}) satisfies\n\t\\begin{equation}\\label{lem1_eq12}\n\t\t\\tilde{\\psi}_{p_k}(x^{k+1}, \\tilde{y}^{k+1}) \\le  \\tilde{\\psi}_{p_k}(x^{k+1}, y^{k+1}) + C_{\\gamma} \\zeta_{k+1}^2,\n\t\\end{equation}\n\twhere  $\\zeta_{k} := \\max\\{ s_k,  ( \\prod_{i=0}^{k} \\tau_i ) \\mathcal{G}(\\theta^{0}, x^{0}, y^{0})\\}$.\n\\end{lemma}\n\nThroughout this subsection and the remainder of this section, we assume that Assumptions \\ref{asup1}, \\ref{asup2}, and \\ref{asup3} hold, and $\\gamma \\in (0, 1/(\\rho_{f_2} +\\rho_{g_2}) )$. The sequences $\\{(x^k, y^k, \\tilde{y}^k)\\}$ and $\\{p_k\\}$ \nare generated by AGILS (\\cref{alg1}), where the step sizes satisfy $\\alpha_k \\in [\\underline{\\alpha}, 2/(L_{\\psi_{x,k}} + c_\\alpha)]$ and $\\beta_k \\in [\\underline{\\beta}, 2/(L_{\\psi_{y,k}} + c_\\beta)]$, where $L_{\\psi_{x,k}} = L_{F_x}/p_k+L_{f_x}+L_{g_1}+\\rho_{f_1} + \\rho_{g_1}$, $L_{\\psi_{y,k}} = L_{F_y}/p_k+L_{f_y}$ and $\\underline{\\alpha}, \\underline{\\beta}, c_\\alpha, c_\\beta$ are positive constants. The inexactness parameters satisfy $\\sum_{k=0}^{\\infty}s_k^2 < \\infty$ and \n$\\tau_k := \\tau_0/(k+1)^{p_\\tau}$ with $\\tau_0 > 0$ and $p_\\tau >0$.\n","units":[{"raw_tex":"Bilevel optimization problems are a class of hierarchical optimization problems where the feasible region is implicitly defined by the solution set of a lower-level optimization problem. In this work, we focus on a bilevel optimization problem which has a convex composite lower-level problem,\n\\begin{equation}\\label{eq1}\n\t\\begin{aligned}\n\t\t\\min\\limits_{x \\in X, y \\in Y} &\\ F(x,y)\\\\\n\t\t{\\rm s.t.}\\quad &\\ y \\in S(x),\n\t\\end{aligned}\n\\end{equation}\nwhere $S(x)$ denotes the set of optimal solutions to the lower-level problem:\n\\begin{equation}\\label{LL}\n\t\\min\\limits_{y \\in Y}\\ \\varphi(x,y) := f(x,y) + g(x,y).\n\\end{equation}\nHere, $X \\subset \\Re^n$ and $Y \\subset \\Re^m$ are closed convex sets, $f(x, y) : \\Re^n \\times \\Re^m \\rightarrow \\Re$ is smooth and convex with respect to $y$ on $Y$ for any $x \\in X$, and $g(x, y) : \\Re^n \\times \\Re^m \\rightarrow \\Re$ is convex but potentially nonsmooth with respect to $y$ on $Y$ for any $x \\in X$. Additionally, we assume that $g(x, \\cdot) : \\Re^m \\rightarrow \\Re$ is proximal-friendly, enabling efficient evaluation of its proximal operator.\n","start":3423,"end":4509},{"raw_tex":"To overcome this challenge, in this paper, we consider the following Moreau envelope-based reformulation of the bilevel optimization problem \\eqref{eq1}, first introduced in \\cite{gao2024},\n\\begin{equation}\\label{reformualtion_Moreau}\n\t({\\rm VP})_\\gamma \\quad\\quad\n\t\\min\\limits_{x\\in X,y\\in Y}\\  F(x,y) \\quad\t\t{\\rm s.t.}\\quad\\ \\varphi (x,y)-v_{\\gamma} (x,y)\\le 0,\n\\end{equation}\nwhere $v_{\\gamma}(x, y)$ is the Moreau envelope associated with the lower-level problem,\n\\begin{equation}\\label{def_vg}\n\tv_{\\gamma}(x,y):=\\inf\\limits_{\\theta\\in Y}\\varphi(x,\\theta)+\\frac{1}{2\\gamma}\\|\\theta-y\\|^2,\n\\end{equation}\nwhere $\\gamma > 0$. When the lower-level problem is convex with respect to $y$,  this reformulation is equivalent to the original bilevel optimization problem \\eqref{eq1} (see \\cite[Theorem 1]{gao2024}).  Directional optimality conditions for $({\\rm VP})_\\gamma$ are discussed in \\cite{bai2025optimality}. However, as shown in \\cite{gao2024} and \\cite[Proposition 7]{ye2023}, classical constraint qualification (CQ) conditions do not hold at any feasible point of $({\\rm VP})_\\gamma$, motivating us to study the following relaxed approximation problem,\n\\begin{equation}\\label{reformualtion_Moreau_relax}\n\t({\\rm VP})_\\gamma^{\\epsilon} \\quad\\quad\n\t\\min\\limits_{x\\in X,y\\in Y}\\  F(x,y) \\quad\t\t{\\rm s.t.}\\quad\\ \\varphi (x,y)-v_{\\gamma} (x,y)\\le \\epsilon,\n\\end{equation}\n","start":11073,"end":12447},{"raw_tex":"Let $\\mathbb{R}^n$ denote the $n$-dimensional Euclidean space, and let $\\mathbb{B}$ denote the closed unit ball centered at the origin. The nonnegative and positive orthants in $\\mathbb{R}^n$ are denoted by $\\mathbb{R}^n_+$ and $\\mathbb{R}^n_{++}$, respectively. The standard inner product and Euclidean norm are denoted by $\\langle \\cdot, \\cdot \\rangle$ and $\\|\\cdot\\|$, respectively.\nFor a vector \\( x \\in \\mathbb{R}^n \\) and a closed convex set \\( S \\subset \\mathbb{R}^n \\), the distance from \\( x \\) to \\( S \\) is defined as \\(\\mathrm{dist}(x, S) = \\min_{y \\in S} \\|x - y\\|\\). The indicator function \\(\\delta_S: \\mathbb{R}^n \\to \\mathbb{R}\\cup\\{+\\infty\\}\\) is defined by \\(\\delta_S(x) = 0\\) if \\(x\\in S\\) and \\(\\delta_S(x) = +\\infty\\) otherwise. Moreover, \\(\\mathcal{N}_S(x)\\) denotes the normal cone to \\(S\\) at \\(x\\). The Euclidean projection operator onto \\( S \\) is denoted by \\(\\mathrm{Proj}_S\\). The Cartesian product of two sets \\( X \\) and \\( Y \\) is denoted by \\( X \\times Y \\).\nLet \\( h: \\mathbb{R}^n \\to \\mathbb{R} \\cup \\{\\infty\\} \\) be a proper closed function. The proximal mapping of \\( h\\) is defined as $\\operatorname{Prox}_h(y):=\\arg\\min_{\\theta \\in \\Re^m}\\left\\{h(\\theta)+\\|\\theta-y\\|^2 / 2\\right\\}$. The Fr\\'{e}chet (regular) subdifferential \\(\\hat{\\partial} h\\) and the Mordukhovich (limiting) subdifferential \\(\\partial h\\) at a point \\( \\bar{z} \\in \\mathrm{dom}\\, h := \\{z \\mid h(z) < \\infty\\} \\) are defined as:\n\\[\n\\begin{aligned}\n\t\\hat{\\partial} h(\\bar{z}) &= \\left\\{ v \\in \\mathbb{R}^n \\;\\middle|\\; h(z) \\geq h(\\bar{z}) + \\langle v, z - \\bar{z} \\rangle + o(\\|z - \\bar{z}\\|), \\; \\forall z \\in \\mathbb{R}^n \\right\\}, \\\\\n\t\\partial h(\\bar{z}) &= \\left\\{ v \\in \\mathbb{R}^n \\;\\middle|\\; \\exists z_k \\xrightarrow{h} \\bar{z}, \\; \\exists v_k \\in \\hat{\\partial} h(z_k), \\; v_k \\to v \\right\\},\n\\end{aligned}\n\\]\nwhere \\( o(z) \\) denotes a function satisfying \\( o(z)/z\\to 0 \\) as \\( z \\downarrow 0 \\), and \\( z_k \\xrightarrow{h} \\bar{z} \\) indicates \\( z_k \\to \\bar{z} \\) and \\( h(z_k) \\to h(\\bar{z}) \\) as \\( k \\to \\infty \\). If \\( \\bar{z} \\notin \\mathrm{dom}\\, h \\), we define \\(\\partial h(\\bar{z}) = \\emptyset\\).\n","start":16383,"end":18517},{"raw_tex":"This part outlines the assumptions on the problem data for the bilevel optimization problem studied in this work. First, we assume that the upper-level objective $F$ is smooth and bounded below.\n\n\\begin{assumption}\\label{asup1}\n\tThe function \\(F : \\mathbb{R}^n \\times \\mathbb{R}^m \\to \\mathbb{R}\\) is bounded below on \\(X \\times Y\\), i.e., \\(\\underline{F} := \\inf_{(x, y) \\in X \\times Y} F(x, y) > -\\infty\\). \\(F\\) is continuously differentiable, and its gradients \\(\\nabla_x F\\) and \\(\\nabla_y F\\) are \\(L_{F_x}\\)- and \\(L_{F_y}\\)-Lipschitz continuous on \\(X \\times Y\\), respectively.\n\\end{assumption}\n\nThroughout this paper, we focus on cases where the lower-level problem is a convex optimization problem.\n\n\\begin{assumption}\\label{asup2}\n\tFor any \\(x \\in X\\), the functions \\(f(x, \\cdot)\\) and \\(g(x, \\cdot)\\) are convex and defined on the closed convex set \\(Y\\). \n\\end{assumption} \n\nSmoothness and weak convexity conditions are required for the lower-level objective functions. We recall that a function \\(h : \\mathbb{R}^n \\times \\mathbb{R}^m \\to \\mathbb{R}\\) is \\((\\rho_1, \\rho_2)\\)-weakly convex on a convex set \\(X \\times Y\\) if \\(h(x, y) + \\frac{\\rho_1}{2} \\|x\\|^2 + \\frac{\\rho_2}{2} \\|y\\|^2\\) is convex on \\(X \\times Y\\). This notion generalizes convexity (\\(\\rho_1 = \\rho_2 = 0\\)). Weakly convex functions \\cite{nurminskii1973quasigradient}, extends the class of convex functions to include many nonconvex functions. For weakly convex functions, the Fréchet (regular) subdifferential coincides with both the limiting subdifferential and the Clarke subdifferential (see, e.g., \\cite[Proposition 3.1 and Theorem 3.6]{ngai2000approximate}).\n\n\\begin{assumption}\\label{asup3}\n\tThe lower-level objective functions \\(f(x, y)\\) and \\(g(x, y)\\) satisfy:\n\t\\begin{itemize}\n\t\t\\item[(1)] \\(f(x, y)\\) is continuously differentiable on \\(X \\times Y\\), and its gradients \\(\\nabla_x f\\) and \\(\\nabla_y f\\) are \\(L_{f_x}\\)- and \\(L_{f_y}\\)-Lipschitz continuous on \\(X \\times Y\\), respectively;\n\t\t\\item[(2)] \\(g(x, y)\\) is continuous on \\(X \\times Y\\), \\(\\nabla_x g(x, y)\\) exists, and \\(\\nabla_x g(\\cdot, y)\\) is \\(L_{g_1}\\)-Lipschitz continuous on \\(X\\) for any \\(y \\in Y\\), while \\(\\nabla_x g(x, \\cdot)\\) is \\(L_{g_2}\\)-Lipschitz continuous on \\(Y\\) for any \\(x \\in X\\);\n\t\t\\item[(3)] \\(f(x, y)\\) is \\((\\rho_{f_1}, \\rho_{f_2})\\)-weakly convex on \\(X \\times Y\\), and \\(g(x, y)\\) is \\((\\rho_{g_1}, \\rho_{g_2})\\)-weakly convex on \\(X \\times Y\\).\n\t\\end{itemize}\n\\end{assumption}\n","start":18552,"end":21022},{"raw_tex":"Under Assumption \\ref{asup2}, the solution mapping\n\\begin{equation}\\label{proximal_LL}\n\tS_{\\gamma}(x, y) := \\mathrm{argmin}_{\\theta \\in Y} \\left\\{ f(x, \\theta) + g(x, \\theta) + \\frac{1}{2\\gamma}\\|\\theta - y\\|^2 \\right\\}\n\\end{equation}\nis well-defined and single-valued  on $X \\times \\mathbb{R}^m$. We denote its unique solution by $\\theta_\\gamma^*(x, y)$.\nBelow, we summarize sensitivity results for $v_\\gamma(x, y)$ from Theorems 2, 3, and 5 of \\cite{gao2024}.\n","start":24483,"end":24945},{"raw_tex":"\t\\begin{equation}\\label{valuefinc}\n\t\t\\nabla v_\\gamma(x, y) = \\Big( \\nabla_x f(x, \\theta_\\gamma^*(x, y)) + \\nabla_x g(x, \\theta_\\gamma^*(x, y)), \\left({y - \\theta_\\gamma^*(x, y)}\\right)/{\\gamma} \\Big).\n\t\\end{equation}\n","start":25358,"end":25575},{"raw_tex":"\\begin{proposition}\\label{partial_grad_v}\n\tSuppose Assumptions \\ref{asup2} and \\ref{asup3} hold. For $\\gamma \\in (0, 1/(\\rho_{f_2} + \\rho_{g_2}))$, the function $v_\\gamma(x, y)$ is $\\rho_{v_1}$-weakly convex with respect to $x$ on $X$ for any $y \\in Y$ with $\\rho_{v_1} \\geq \\rho_{f_1} + \\rho_{g_1}$. It is also convex with respect to $y$ on $\\mathbb{R}^m$ for any $x \\in X$. Moreover, $v_\\gamma(x, y)$ is differentiable with gradient given by \\eqref{valuefinc} on $X \\times \\mathbb{R}^m$, and $\\theta_\\gamma^*(x, y)$ and $\\nabla v_\\gamma(x, y)$ are continuous on $X \\times \\mathbb{R}^m$.\n\\end{proposition}","start":26030,"end":26636},{"raw_tex":"\\begin{definition}\n\tLet $(\\bar{x},\\bar{y})$ be a feasible solution of problem $({\\rm VP})_{\\gamma}^{\\epsilon}$ with $\\epsilon > 0$. We say that $(\\bar{x},\\bar{y})$ is a stationary/KKT point of problem $({\\rm VP})_{\\gamma}^{\\epsilon}$ if there exists a multiplier $\\lambda \\ge 0$ such that\n\t\\begin{equation*}\n\t\t\\begin{cases}\n\t\t\t0\\in \\nabla F(\\bar{x},\\bar{y}) + \\lambda \\left(\\nabla f(\\bar{x},\\bar{y}) + \\partial g(\\bar{x},\\bar{y}) - \\nabla v_{\\gamma}(\\bar{x},\\bar{y})\\right) + \\mathcal{N}_{X\\times Y}(\\bar{x},\\bar{y}), \\\\\n\t\t\tf(\\bar{x},\\bar{y}) + g(\\bar{x},\\bar{y}) - v_{\\gamma}(\\bar{x},\\bar{y}) \\leq \\epsilon, \\qquad \\lambda \\left(f(\\bar{x},\\bar{y}) + g(\\bar{x},\\bar{y}) - v_{\\gamma}(\\bar{x},\\bar{y})-\\epsilon\\right) = 0.\n\t\t\\end{cases}\n\t\\end{equation*}\n\\end{definition}\n","start":34074,"end":34843},{"raw_tex":"In this section, we propose a new gradient-type algorithm, Alternating Gradient-type algorithm with Inexact Lower-level Solutions (AGILS), to solve ${\\rm (VP)}_\\gamma^\\epsilon$. Given that $f(x, y) + g(x,y) - v_{\\gamma}(x, y)$ is always nonnegative, a natural approach is to consider the following penalized problem,\n\\begin{equation}\\label{penprob}\n\t\\min_{x \\in X, y \\in Y}\\psi_{p_k}\t(x, y) := \\frac{1}{p_k}F(x, y) + f(x, y) + g(x,y) - v_{\\gamma}(x, y),\n\\end{equation}\nwhere $p_k$ is the penalty parameter at iteration $k$. Due to the nonsmoothness of $g(x,y)$ with respect to $y$ and the availability of its proximal operator, it is natural to adopt a proximal alternating linearization method \\cite{bolte2014proximal} to construct the algorithm. Given the current iterate $(x^k, y^k)$, linearizing the smooth components of $\\psi_{p_k}\t(x, y)$ requires evaluating gradients, including $\\nabla v_{\\gamma}(x, y)$ as provided in \\cref{grad_v}. Calculating the exact gradient  $\\nabla v_{\\gamma}(x^k, y^k)$ requires solving the proximal lower-level problem in \\eqref{proximal_LL} to obtain $\\theta_\\gamma^*(x^k, y^k)$. However, this solution typically does not have a closed-form expression, and solving it requires an iterative optimization solver, which can be computationally expensive. To address this, we propose an inexact gradient approximation by substituting $\\theta_{\\gamma}^*$ with an inexact estimate $\\theta^k$.\n\nThe proposed algorithm alternates between updating $y$ and $x$ while refining the inexact approximation of $\\theta_\\gamma^*(x^k, y^k)$. At each iteration $k=0,1,\\ldots$, given the current iterate $(x^k, y^k)$, along with the feasibility-corrected iterate $\\tilde{y}^k$ and corresponding $\\tilde{\\theta}^k$ (constructed from $y^k$ and  $\\theta^{k}$ via the procedure detailed below), the next iterate $(x^{k+1}, y^{k+1})$ and the refined approximation $\\theta^{k+1}$ are updated as follows.\n\n{\\bf Update $y$ (Fixing $x$).}\nWe compute the update direction for $y$,\n\\begin{equation}\\label{dy}\n\td_y^k:= \\frac{1}{p_k} \\nabla_y F\\left(x^{k}, \\tilde{y}^k\\right)+\\nabla_y f\\left(x^{k}, \\tilde{y}^k\\right)-\\frac{\\tilde{y}^k-\\tilde{\\theta}^{k}}{\\gamma},\n\\end{equation}\nwhere $(\\tilde{y}^k-\\tilde{\\theta}^{k})/{\\gamma}$ is an inexact approximation of $\\nabla_y v_{\\gamma}(x^{k}, \\tilde{y}^k) =( {\\tilde{y}^k} -  \\theta_{\\gamma}^*(x^{k}, \\tilde{y}^k))/ \\gamma $ (see (\\ref{valuefinc})). Using $\td_y^k$, the variable $y$ is updated as\n\\begin{equation}\\label{update_y}\n\ty^{k+1} = \\underset{y \\in Y}{\\mathrm{argmin}} ~ \\left\\langle d_y^k, y - \\tilde{y}^{k} \\right\\rangle + g(x^k, y) + \\frac{1}{2\\beta_k} \\left\\| y - \\tilde{y}^k \\right\\|^2, \n\\end{equation}\nwhere $\\beta_k>0$ is the stepsize. \nThis can also be expressed using the proximal operator\n\\begin{equation}\\label{update_y2}\n\ty^{k+1} = \\operatorname{Prox}_{\\beta_k\\tilde{g}(x^{k},\\cdot)}\\left(\\tilde{y}^k-\\beta_k d^k_y\\right),\n\\end{equation}\nwhere $\\tilde{g}(x,y) := g(x,y) + \\delta_Y(y)$. \n\n{\\bf Find inexact approximation $\\theta^{k+1/2}$.}\nAfter updating $y^{k+1}$, we compute an inexact approximation $\\theta^{k+1/2}$ for $\\theta_{\\gamma}^*(x^k, y^{k+1})$ by solving\n\\begin{equation}\\label{theta_y}\n\t\\theta^{k+1/2} \\approx \t\\underset{\\theta \\in Y}{\\mathrm{argmin}} \\left\\{f(x^{k},\\theta) + g(x^{k}, \\theta) + \\frac{1}{2\\gamma}\\left\\|\\theta-y^{k+1}\\right\\|^2\\right\\}, \n\\end{equation}\nsubject to either an absolute or a relative inexactness criterion\n\\begin{equation}\\label{inexacty1}\n\t\\begin{aligned}\n\t\t\\mathcal{G}(\\theta^{k+1/2}, x^k,y^{k+1}) \\le s_k, \\quad \\text{or} \\quad \\mathcal{G}(\\theta^{k+1/2}, x^k,y^{k+1}) \\le \\tau_k \\mathcal{G}(\\theta^{k-1}, x^{k-1},y^{k-1}),\n\t\\end{aligned}\n\\end{equation}\nwhich can be compactly expressed as\n\\[\n\\mathcal{G}(\\theta^{k+1/2}, x^{k}, y^{k+1})\n\\le\n\\max\\left\\{ s_k,\\ \\tau_k\\,\\mathcal{G}(\\theta^{k-1}, x^{k-1}, y^{k-1}) \\right\\}.\n\\]\nThe parameters $s_k$ and $\\tau_k$ control the solution's inexactness, and are required to satisfy $\\sum_{k=0}^{\\infty}s_k^2<\\infty$ and $\\tau_k:=\\tau_0/(k+1)^{p_\\tau}$ with constants $\\tau_0 > 0$ and $p_\\tau >0$. The prox-gradient residual $\\mathcal{G}(\\theta, x,y)$ quantifies the inexactness and is defined as\n\\[\n\\mathcal{G}(\\theta, x,y):= \t\\left\\| \\theta -\\operatorname{Prox}_{\\eta \\tilde{g}\\left(x, \\cdot\\right)}\\left(\\theta-\\eta\\left(\\nabla_y f\\left(x, \\theta\\right) + (\\theta -y)/ {\\gamma}\\right)\\right)\\right\\|,\n\\]\nwhere $\\eta > 0$. Note that $\\mathcal{G}(\\theta, x,y) = 0$ if and only if $\\theta =\\theta_{\\gamma}^*(x,y)$. \n","start":34884,"end":39353},{"raw_tex":"{\\bf Update $x$ (Fixing $y$).}\nWe compute the update direction for $x$,\n\\begin{equation}\\label{dx}\n\td_x^k:= \\frac{1}{p_k} \\nabla_xF\\left(x^k, y^{k+1}\\right)+\\nabla_x \\varphi \\left(x^k, y^{k+1}\\right)-\\nabla_x f\\left(x^k, \\theta^{k+1/2}\\right)-\\nabla_x g\\left(x^k, \\theta^{k+1/2}\\right),\n\\end{equation}\nwhere $\\nabla_x f\\left(x^k, \\theta^{k+1/2}\\right)+\\nabla_x g\\left(x^k, \\theta^{k+1/2}\\right)$ is an approximation to \\( \\nabla_x v_{\\gamma}(x^k, y^{k+1}) = \\nabla_x f(x^k, \\theta^*_{\\gamma}(x^k, y^{k+1}))+\\nabla_x g(x^k, \\theta^*_{\\gamma}(x^k, y^{k+1}))\\) (see \\eqref{valuefinc}).\nUsing the direction $d_x^k$, we update $x$ with stepsize $\\alpha_k>0$ as\n\\begin{equation}\\label{update_x}\n\tx^{k+1}=\\operatorname{Proj}_X\\left(x^k-\\alpha_k d_x^k\\right).\n\\end{equation}\n\n{\\bf Find inexact approximation $\\theta^{k+1}$.}\nAfter updating $x^{k+1}$, a new inexact approximation $\\theta^{k+1}$ of $\\theta^*_{\\gamma}(x^{k+1}, y^{k+1})$  is computed by approximately solving\n\\[\n\\theta^{k+1} \\approx \t\\underset{\\theta \\in Y}{\\mathrm{argmin}} \\left\\{f(x^{k+1},\\theta) + g(x^{k+1}, \\theta) + \\frac{1}{2\\gamma}\\left\\|\\theta-y^{k+1}\\right\\|^2\\right\\}, \n\\]\nsuch that $\\theta^{k+1}$ satisfies either an absolute or a relative inexactness criterion\n\\begin{equation}\\label{inexacty2}\n\t\\begin{aligned}\n\t\t\\mathcal{G}(\\theta^{k+1}, x^{k+1},y^{k+1}) \\le s_{k+1}, \\quad \\text{or} \\quad  \\mathcal{G}(\\theta^{k+1}, x^{k+1},y^{k+1}) \\le \\tau_{k+1}\\mathcal{G}(\\theta^{k}, x^{k},y^{k}).\n\t\\end{aligned}\n\\end{equation}\n","start":39807,"end":41295},{"raw_tex":"{\\bf Penalty parameter update and feasibility correction.}\nTo encourage feasibility of the constraint $\\varphi(x, y) - v_{\\gamma}(x, y) \\le \\epsilon$, the penalty parameter $p_k$ is updated at each iteration.\nEvaluating the constraint violation requires computing $ v_{\\gamma}(x, y)$, which involves solving the proximal lower-level problem in \\eqref{proximal_LL}. Instead, we use $\\theta^{k+1}$ to estimate the constraint violation as\n\\begin{equation}\\label{def_t}\n\tt^{k+1}=\\max\\left\\{\\varphi(x^{k+1},y^{k+1})-\\varphi(x^{k+1},\\theta^{k+1})-\\frac{1}{2\\gamma}\\| \\theta^{k+1} - y^{k+1}\\|^2-\\epsilon,0\\right\\}.\n\\end{equation}\nThe penalty parameter \\(p_k\\) is increased to $p_{k+1}=\tp_k + \\varrho_p$ (for some $\\varrho_p > 0$) only when the change in the iterates, $\\Delta_{k+1} := \\left\\| (x^{k+1}, y^{k+1}) - (x^k, \\tilde{y}^k) \\right\\|$, is small relative to the penalty parameter (i.e., $\\Delta_{k+1} < c_p/p_k$ for some $c_p > 0$), while the estimated constraint violation $t^{k+1}$ is large relative to this change (i.e., $t^{k+1} > \\Delta_{k+1}/c_p$).\nThese two conditions can be compactly written as \\begin{equation}\\label{update_p_1}\n\t\\Delta_{k+1} < c_p \\min\\left\\{1/p_k,{t^{k+1}}\\right\\}.\n\\end{equation}\n\nHowever,  simply increasing $p_{k}$ does not always guarantee feasibility. It can sometimes cause the algorithm to stagnate near an infeasible stationary point of the constraint function $\\varphi(x, y) - v_{\\gamma}(x, y)$. \nTo mitigate this, we use the bilevel optimization structure, that the feasible region is the solution set of the convex lower-level problem \\eqref{LL}. We introduce a {\\it feasibility correction procedure} designed to help the iterates escape such stationary points.\n\nThe feasibility correction procedure aims to generate a corrected iterate $\\tilde{y}^{k+1} $ from $y^{k+1}$. It is based on the observation that if $(x^{k+1},y^{k+1})$ is feasible, then $y^{k+1}$ should be a reasonably accurate solution to the lower-level problem \\eqref{LL}  for $x= x^{k+1}$, implying that $\\| \\nabla_y v_{\\gamma}(x^{k+1},y^{k+1})\\| $ should be small. We use the magnitude of $( y^{k+1} - \\theta^{k+1})/\\gamma$ to approximate $\\| \\nabla_y v_{\\gamma}(x^{k+1},y^{k+1})\\| $. The feasibility correction  is triggered if:\n\\begin{equation}\\label{theta_gap_condition}\n\t\\left\\| y^{k+1} - \\theta^{k+1} \\right\\| \\ge \\frac{c_y \\gamma}{p_k},\n\\end{equation}\nwhere $c_y > 0$. Condition \\eqref{theta_gap_condition} indicates that the iterate $(x^{k+1},y^{k+1})$ might be approaching an undesirable stationary point of the constraint function.\n\nIf condition \\eqref{theta_gap_condition} holds, we attempt to correct $y^{k+1}$ by finding a candidate iterate $\\tilde{y}^{k+1} $  as an inexact solution to the lower-level problem \\eqref{LL} with $x= x^{k+1}$ satisfying\n\\begin{equation}\\label{inexacty_y}\n\t\\left\\| \\tilde{y}^{k+1} - \\operatorname{Prox}_{ \\tilde{g}(x^{k+1}, \\cdot)} \\left( \\tilde{y}^{k+1} -  \\nabla_y f(x^{k+1}, \\tilde{y}^{k+1}) \\right) \\right\\| \\le \\frac{c_{\\tilde{y}} }{p_k} \\Delta_{k+1}, \n\\end{equation}\nwhere $c_{\\tilde{y}} > 0$. However, unconditionally accepting such a $\\tilde{y}^{k+1} $ could adversely affect the algorithm's convergence. Therefore, to preserve convergence, $\\tilde{y}^{k+1} $ is accepted only if it satisfies the following descent condition\n\\begin{equation}\\label{decrease_new_y}\n\t\\begin{aligned}\n\t\t&\\frac{1}{p_k}F(x^{k+1}, \\tilde{y}^{k+1}) + \\varphi(x^{k+1}, \\tilde{y}^{k+1}) - \\left( \\varphi(x^{k+1},  \\tilde{\\theta}^{k+1} ) + \\frac{1}{2\\gamma} \\| \\tilde{\\theta}^{k+1} - \\tilde{y}^{k+1} \\|^2 \\right)\\\\\n\t\t\\le\\; &\n\t\t\\frac{1}{p_k}F(x^{k+1}, y^{k+1}) + \\varphi(x^{k+1}, y^{k+1}) - \\left( \\varphi(x^{k+1}, \\theta^{k+1}) + \\frac{1}{2\\gamma} \\| \\theta^{k+1} - y^{k+1} \\|^2 \\right),\n\t\\end{aligned}\n\\end{equation}\nwhere, $\\tilde{\\theta}^{k+1}$ is a new approximation of $\\theta^*_{\\gamma}(x^{k+1}, \\tilde{y}^{k+1})$. We construct $\\tilde{\\theta}^{k+1}$ by first finding a point $\\hat{\\theta}^{k+1}$ that satisfies the inexactness criterion\n\\begin{equation}\\label{inexacty3}\n\t\\mathcal{G}(\\hat{\\theta}^{k+1}, x^{k+1},\\tilde{y}^{k+1}) \\le s_{k+1},\n\t\\quad \\text{or} \\quad\n\t\\mathcal{G}(\\hat{\\theta}^{k+1}, x^{k+1},\\tilde{y}^{k+1})\n\t\\le \\tau_{k+1}\\mathcal{G}(\\theta^{k}, x^{k},y^{k}).\n\\end{equation}\nand subsequently applying a single proximal-gradient step to $\\hat{\\theta}^{k+1}$\n\\[\n\\tilde{\\theta}^{k+1}\n=\n\\operatorname{Prox}_{\\eta \\tilde g(x^{k+1},\\cdot)}\n\\left(\n\\hat{\\theta}^{k+1}\n-\\eta\\left(\n\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n+\\frac{\\hat{\\theta}^{k+1}-\\tilde y^{k+1}}{\\gamma}\n\\right)\n\\right).\n\\]\n\nIf the descent condition \\eqref{decrease_new_y} holds, the algorithm adopts the candidate  $\\tilde{y}^{k+1} $ along with its corresponding $\\tilde{\\theta}^{k+1}$, and keep the penalty parameter unchanged, i.e., $p_{k+1}=\tp_k$. Otherwise, the correction is discarded, meaning $\\tilde{y}^{k+1} = y^{k+1}$ and $\\tilde{\\theta}^{k+1} = \\theta^{k+1}$ and the penalty parameter is increased, i.e., $p_{k+1}=\tp_k + \\varrho_p$. The algorithm then proceeds with the selected pair $(\\tilde{y}^{k+1}, \\tilde{\\theta}^{k+1})$. A detailed description of this feasibility correction procedure is presented in Algorithm \\ref{alg2}.\n","start":41581,"end":46732},{"raw_tex":"\\begin{algorithm}[htp!]\n\\caption{Alternating Gradient-type algorithm with Inexact Lower-level Solutions (AGILS)}\n\\label{alg1}\n\\begin{algorithmic}[1]\n\t\\STATE{{\\bf Input}: Initial iterates $x^0,\\theta^{-1}, y^0, \\tilde{\\theta}^0, \\tilde{y}^0$, set $(x^{-1}, y^{-1}, \\theta^{-1}) = (x^0, y^0, \\tilde{\\theta}^0)$, stepsizes $\\alpha_k, \\beta_k$, relaxation parameter $\\epsilon$, proximal parameter $\\gamma$, penalty parameters $p_{0}$,  $\\varrho_p$, $c_p$, $c_y$, inexact parameters $\\eta$, $s_{k}$, $\\tau_k$, tolerance $tol$.}\n\t\\FOR{$k=0,\\ 1,\\cdots$}\n\t\\STATE{Construct the direction $d_y^k$ according to \\eqref{dy}, and update $y^{k+1}$ as\n\t\t$$\n\t\ty^{k+1} = \\operatorname{Prox}_{\\beta_k\\tilde{g}(x^{k,\\cdot})}\\left(\\tilde{y}^k-\\beta_k d^k_y\\right).\n\t\t$$}\n\t\n\t\\STATE{Find an inexact solution $\\theta^{k+1/2}$ that satisfies the inexactness criterion \\eqref{inexacty1}.}\n\t\n\t\\STATE{Construct the direction $d_x^k$ according to \\eqref{dx}, and update $x^{k+1}$ as \n\t\t$$ x^{k+1}=\\operatorname{Proj}_X\\left(x^k-\\alpha_k d_x^k\\right). \n\t\t$$}\n\t\n\t\\STATE{Find an inexact solution $\\theta^{k+1}$ that satisfies the inexactness criterion \\eqref{inexacty2}.}\n\t\n\t\\STATE{Calculate $\\Delta_{k+1} := \\left\\| (x^{k+1}, y^{k+1}) - (x^k, \\tilde{y}^k) \\right\\|$ and $t^{k+1}$ according to \\eqref{def_t}. \\\\\n\t\tStop when $k \\ge 1$ and \\(\\max\\{\\Delta_{k+1}, s_{k}, t^{k+1} \\} \\le tol.\\)\n\t}\n\t\\STATE{Update penalty parameter and apply feasibility correction:}\n\t\n\t\\STATE{\\hspace*{10pt}\\textbf{Case 1:}\n\t\t$\\Delta_{k+1} \\geq c_p \\min\\big\\{1/p_k,\\, t^{k+1} \\big\\}$ \\\\\n\t\t\\vspace*{1pt}\n\t\t\\qquad\\qquad\\quad Set \\( p_{k+1} := p_k \\), \\( \\tilde{y}^{k+1} := y^{k+1} \\), \\(\\tilde{\\theta}^{k+1} :=\\theta^{k+1}\\).}\n\t\\vspace*{2pt}\n\t\\STATE{\\hspace*{10pt}\\textbf{Case 2:} \n\t\t$\\Delta_{k+1} < c_p \\min\\big\\{1/p_k,\\, t^{k+1} \\big\\},\\,\\| y^{k+1} - \\theta^{k+1} \\| \\le c_y \\gamma/p_k$\\\\  \n\t\t\\vspace*{1pt}\n\t\t\\qquad\\qquad\\quad Set \\( p_{k+1} := p_k + \\varrho_p \\), \\( \\tilde{y}^{k+1} := y^{k+1} \\), \\(\\tilde{\\theta}^{k+1} :=\\theta^{k+1}\\).}\n\t\\vspace*{2pt}\n\t\n\t\\STATE{\\hspace*{10pt}\\textbf{Case 3:} \n\t\t$\\Delta_{k+1} < c_p \\min\\big\\{1/p_k,\\, t^{k+1} \\big\\},\\,\\| y^{k+1} - \\theta^{k+1} \\| > c_y \\gamma/p_k$\\\\\n\t\t\\vspace*{1pt}\n\t\t\\qquad\\qquad\\quad Apply the feasibility correction procedure (Algorithm~\\ref{alg2}).}\n\\ENDFOR\n\\RETURN $(x^{k+1},\\tilde{y}^{k+1})$\n\\end{algorithmic}\n\\end{algorithm}\n\n\\begin{algorithm}[htp!]\n\\caption{Feasibility Correction Procedure}\n\\label{alg2}\n\\begin{algorithmic}[1]\n\n\\STATE{{\\bf Input}: Current iterates $x^{k+1}, y^{k+1}, \\theta^{k+1}$, $\\Delta_{k+1}$, proximal parameter $\\gamma$, penalty parameters $p_{k}$, $\\varrho_p$, inexact parameters $\\eta$, $s_{k}$, $\\tau_k$, constant $c_{\\tilde{y}}$.}\n\n\\STATE{Compute a candidate $\\tilde{y}^{k+1}$ by inexactly solving the lower-level problem \\eqref{LL} with $x=x^{k+1}$  to satisfy \\eqref{inexacty_y}.\n}\n\n\\STATE{Find an intermediate point $\\hat{\\theta}^{k+1}$ that satisfies the inexactness criterion \\eqref{inexacty3}, and compute $\\tilde{\\theta}^{k+1}=\n\\operatorname{Prox}_{\\eta \\tilde g(x^{k+1},\\cdot)}\n(\n\\hat{\\theta}^{k+1}\n-\\eta(\n\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n+(\\hat{\\theta}^{k+1}-\\tilde y^{k+1})/\\gamma\n)\n)$.}\n\n\\IF{the descent condition \\eqref{decrease_new_y} holds}\n\\STATE{Accept the candidate: Set $p_{k+1} := p_k$.}\n\\ELSE\n\\STATE{Reject the candidate: Set $\\tilde{y}^{k+1} := y^{k+1}$, $\\tilde{\\theta}^{k+1} := \\theta^{k+1}$ and $p_{k+1} := p_k + \\varrho_p$.}\n\\ENDIF\n\\RETURN $(\\tilde{y}^{k+1},\\tilde{\\theta}^{k+1},p_{k+1})$\n\\end{algorithmic}\n\\end{algorithm}\n","start":47378,"end":50868},{"raw_tex":"We begin by considering the following lemma, which provides an estimate for the distance between the approximation $\\theta$ and the exact solution $\\theta^*_{\\gamma}(x,y)$ of the proximal lower-level problem, based on the prox-gradient residual $\\mathcal{G}(\\theta, x,y)$ in the inexact criteria \\eqref{inexacty1} and \\eqref{inexacty2}.  This estimate is derived from the $1/\\gamma$-strong convexity and $(1/\\gamma + L_f)$-smoothness of the component $f(x, \\theta) + \\|\\theta - y\\|^2/(2\\gamma)$ with respect to $\\theta$ in the objective of the proximal lower level problem, see, e.g., \\cite[Theorem 3.4 and 3.5]{drusvyatskiy2018error} or \\cite[Proposition 2.2]{mordukhovich2023globally}.\n\\begin{lemma}\\label{error_bound}\n\tSuppose Assumptions \\ref{asup2} and \\ref{asup3} hold, and let $\\gamma > 0$, $\\eta > 0$. Then, there exists $C_\\eta > 0$ such that for any $(x,y) \\in X \\times \\Re^m$, and $\\theta \\in \\Re^m$, it holds that\n\t\\[\n\t\\|\\theta - \\theta^*_{\\gamma}(x,y) \\| \\le C_\\eta  \\left\\|\\theta-\\operatorname{Prox}_{\\eta \\tilde{g}\\left(x, \\cdot\\right)}\\left(\\theta-\\eta\\left(\\nabla_y f\\left(x, \\theta\\right)+ (\\theta-y)/{\\gamma}\\right)\\right)\\right\\|,\n\t\\]\n\twhere $\\theta^*_{\\gamma}(x,y) := \\arg\\min_{\\theta \\in Y} \\{ \\varphi (x,\\theta)+\\frac{1}{2\\gamma}\\|\\theta-y\\|^2 \\}$.\n\\end{lemma}\n","start":51250,"end":52534},{"raw_tex":"We now proceed to establish a decreasing property for the proposed method based on the following merit function:\n\\begin{equation*}\n\t\\tilde{\\psi}_{p_{k}}(x,y) :=\\frac{1}{p_k}(F(x,y)-\\underline{F})+ f(x,y) + g(x,y)-v_{\\gamma}(x,y).\n\\end{equation*}\n\nBefore establishing the main decreasing property, we first demonstrate that the descent condition \\eqref{decrease_new_y}, imposed by the feasibility correction procedure (Algorithm \\ref{alg2}), ensures a decrease in $\\tilde{\\psi}_{p_{k}}$ when moving from $y^{k+1}$ to the corrected iterate $\\tilde{y}^{k+1} $.\n\\begin{lemma}\\label{lem0}\n\tSuppose Assumptions \\ref{asup2} and \\ref{asup3} hold. Then, there exists $C_\\gamma>0$,  such that the sequence $\\{(x^k,y^k,\\tilde{y}^k)\\}$ generated by AGILS (\\cref{alg1}) satisfies\n\t\\begin{equation}\\label{lem1_eq12}\n\t\t\\tilde{\\psi}_{p_k}(x^{k+1}, \\tilde{y}^{k+1}) \\le  \\tilde{\\psi}_{p_k}(x^{k+1}, y^{k+1}) + C_{\\gamma} \\zeta_{k+1}^2,\n\t\\end{equation}\n\twhere  $\\zeta_{k} := \\max\\{ s_k,  ( \\prod_{i=0}^{k} \\tau_i ) \\mathcal{G}(\\theta^{0}, x^{0}, y^{0})\\}$.\n\\end{lemma}","start":53837,"end":54887},{"raw_tex":"Throughout this subsection and the remainder of this section, we assume that Assumptions \\ref{asup1}, \\ref{asup2}, and \\ref{asup3} hold, and $\\gamma \\in (0, 1/(\\rho_{f_2} +\\rho_{g_2}) )$. The sequences $\\{(x^k, y^k, \\tilde{y}^k)\\}$ and $\\{p_k\\}$ \nare generated by AGILS (\\cref{alg1}), where the step sizes satisfy $\\alpha_k \\in [\\underline{\\alpha}, 2/(L_{\\psi_{x,k}} + c_\\alpha)]$ and $\\beta_k \\in [\\underline{\\beta}, 2/(L_{\\psi_{y,k}} + c_\\beta)]$, where $L_{\\psi_{x,k}} = L_{F_x}/p_k+L_{f_x}+L_{g_1}+\\rho_{f_1} + \\rho_{g_1}$, $L_{\\psi_{y,k}} = L_{F_y}/p_k+L_{f_y}$ and $\\underline{\\alpha}, \\underline{\\beta}, c_\\alpha, c_\\beta$ are positive constants. The inexactness parameters satisfy $\\sum_{k=0}^{\\infty}s_k^2 < \\infty$ and \n$\\tau_k := \\tau_0/(k+1)^{p_\\tau}$ with $\\tau_0 > 0$ and $p_\\tau >0$.\n","start":68918,"end":69717}]},"statement":{"raw_tex":"\\begin{theorem}\\label{th1}\n\tLet $\\epsilon > 0$. If the sequence $\\{p_k\\}$ is bounded, then any accumulation point $(\\bar{x}, \\bar{y})$ of $\\{(x^k,y^k)\\}$ is a KKT point of $({\\rm VP})_\\gamma^{\\bar{\\epsilon}}$ for some  $\\bar{\\epsilon}\\le\\epsilon$. \n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\\label{th1}\n\tLet $\\epsilon > 0$. If the sequence $\\{p_k\\}$ is bounded, then any accumulation point $(\\bar{x}, \\bar{y})$ of $\\{(x^k,y^k)\\}$ is a KKT point of $({\\rm VP})_\\gamma^{\\bar{\\epsilon}}$ for some  $\\bar{\\epsilon}\\le\\epsilon$. \n\\end{theorem}","start":92183,"end":92445}]},"proof":{"raw_tex":"\\begin{lemma}\\label{lem9}\n\tSuppose Assumptions \\ref{asup2} and \\ref{asup3} hold. Let $\\gamma \\in (0, 1/(\\rho_{f_2} +\\rho_{g_2}) )$ and  $(\\bar{x},\\bar{y})\\in X\\times \\Re^m$. Then, for any $(x,y) \\in X \\times \\Re^m$, the following inequalities hold,\n\t\\begin{align}\n\t\t-v_{\\gamma}(x,\\bar{y}) &\\le -v_{\\gamma}(\\bar{x},\\bar{y})-\\langle \\nabla_x v_{\\gamma}(\\bar{x},\\bar{y}),x-\\bar{x}\\rangle +\\frac{\\rho_{f_1} + \\rho_{g_1}}{2}\\|\\bar{x}-x\\|^2,  \\label{lem9_eq1} \\\\\n\t\t-v_{\\gamma}(\\bar{x},y) &\\le -v_{\\gamma}(\\bar{x},\\bar{y})-\\langle \\nabla_y v_{\\gamma}(\\bar{x},\\bar{y}),y-\\bar{y}\\rangle. \\label{lem9_eq2}\n\t\\end{align}\n\\end{lemma}\n\n\\begin{proof}\n\tAccording to Proposition \\ref{partial_grad_v}, for any $\\bar{y} \\in \\Re^m$, $v_{\\gamma}(x, \\bar{y})$ is a $(\\rho_{f_1} + \\rho_{g_1})$-weakly convex function with respect to $x$ on $X$, that is, $v_{\\gamma}(x,\\bar{y})+\\frac{\\rho_{f_1} + \\rho_{g_1}}{2}\\|x\\|^2$ is convex on $X$. Then we can obtain \\eqref{lem9_eq1} immediately. The second inequality \\eqref{lem9_eq2} follows from Proposition \\ref{partial_grad_v} that for any $\\bar{x} \\in X$, $v_{\\gamma}(\\bar{x}, y)$ is convex with respect to $y$.\n\\end{proof}\n\n\\begin{lemma}\\label{lem1}\n\tSuppose Assumptions \\ref{asup1}, \\ref{asup2} and \\ref{asup3} hold. Let $\\gamma \\in (0, 1/(\\rho_{f_2} +\\rho_{g_2}) )$, $\\alpha_k \\in [0, 2/(L_{\\psi_{x,k}} + c_\\alpha)]$ and $\\beta_k \\in [0, 2/(L_{\\psi_{y,k}} + c_\\beta)]$, where $L_{\\psi_{x,k}} = L_{F_x}/p_k+L_{f_x}+L_{g_1}+\\rho_{f_1} + \\rho_{g_1}$, $L_{\\psi_{y,k}} = L_{F_y}/p_k+L_{f_y}$ and $c_\\alpha, c_\\beta$ are positive constants. Then, there exists $C_\\eta , C_\\gamma > 0$ such that the sequence $\\{(x^k,y^k,\\tilde{y}^k)\\}$ generated by AGILS {\\rm (}\\cref{alg1}{\\rm )} satisfies the following inequality,\n\t\\begin{equation}\\label{lem1_eq}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_{k+1}}\\left(x^{k+1}, \\tilde{y}^{k+1}\\right)\\leq\\,& \\tilde{\\psi}_{p_k}\\left(x^k, \\tilde{y}^k\\right) +  \\left( \\frac{C_\\eta^2}{c_\\alpha}(L_{f_x} + L_{g_2})^2 + \\frac{C_\\eta^2}{c_\\beta\\gamma^2 } \\right) \\zeta_{k}^2 + C_{\\gamma} \\zeta_{k+1}^2\\\\\n\t\t\t&- \\frac{c_\\alpha}{4} \\left\\|x^{k+1}-x^k\\right\\|^2  -\\frac{c_\\beta}{4} \\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2,\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere  $\\zeta_{k} := \\max\\{ s_k,  ( \\prod_{i=0}^{k} \\tau_i ) \\mathcal{G}(\\theta^{0}, x^{0}, y^{0})\\}$ and $\\{ \\zeta_{k} \\}$ is square summable.\n\\end{lemma}\n\n\\begin{proof}\n\t\n\tFirst, the update rule for \\( y^{k+1} \\) given in \\eqref{update_y} defines it as the minimizer of the function $ g(x^k, y) + \\langle d_y^k, y - \\tilde{y}^{k} \\rangle + \\frac{1}{2\\beta_k} \\left\\| y - \\tilde{y}^k \\right\\|^2$. This function is $1/\\beta_k$-strongly convex in $y$. By applying \\cite[Theorem 6.39]{beck2017first}, we obtain\n\t\\begin{equation}\\label{lem1_eq5}\n\t\tg(x^{k}, y^{k+1}) + \\langle d_y^k, y^{k+1}-\\tilde{y}^k\\rangle+\\frac{1}{\\beta_k}\\|y^{k+1}-\\tilde{y}^k\\|^2 \\le g(x^{k}, \\tilde{y}^{k}).\n\t\\end{equation}\n\t\n\t\n\tAccording to the assumptions, with $x^{k} \\in X$, we have that $\\nabla_y F(x^{k},y)$ and $\\nabla_y f(x^{k},y)$ are $L_{F_y}$- and  $L_{f_y}$-Lipschitz continuous with respect to variable $y$ on $Y$, respectively, and combine this with \\eqref{lem9_eq2}, we have\n\t\\begin{equation}\\label{lem1_eq6}\n\t\t\\begin{aligned}\n\t\t\t&(\\tilde{\\psi}_{p_k}-g)(x^{k},y^{k+1})  \\\\\n\t\t\t\\le \\, & (\\tilde{\\psi}_{p_k}-g)\t(x^{k},\\tilde{y}^k) + \\langle \\nabla_y (\\tilde{\\psi}_{p_k}-g)(x^{k},\\tilde{y}^k),y^{k+1}-\\tilde{y}^k\\rangle +\\frac{L_{\\psi_{y,k}}}{2}\\|y^{k+1}-\\tilde{y}^k\\|^2,\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere $L_{\\psi_{y,k}}=L_{F_y}/p_k+L_{f_y}$. \n\tCombining \\eqref{lem1_eq5} and \\eqref{lem1_eq6} yields\n\t\\begin{equation}\\label{lem1_eq7}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k}, y^{k+1}\\right) \\leq \\,& \t\\tilde{\\psi}_{p_k}\\left(x^{k}, \\tilde{y}^k\\right) + \\langle \\nabla_y (\t\\tilde{\\psi}_{p_k}-g)(x^{k},\\tilde{y}^k) - d_y^k, y^{k+1}-\\tilde{y}^k\\rangle  \\\\\n\t\t\t&-\\left(\\frac{1}{\\beta_k}-\\frac{L_{\\psi_{y,k}}}{2}\\right)\\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2.\n\t\t\\end{aligned}\n\t\\end{equation}\t\n\tUsing the formula of \\( \\nabla_y v_\\gamma \\) in \\eqref{valuefinc}, and the construction of $d_y^k$ in \\eqref{dy}, we have\n\t\\begin{equation}\\label{lem1_eq11}\n\t\t\\begin{aligned}\n\t\t\t\\left\\| \\nabla_y (\t\\tilde{\\psi}_{p_k}-g)(x^{k},\\tilde{y}^k) - d_y^k \\right\\| & = \\| \\nabla_y v_\\gamma(x^k, \\tilde{y}^k) + (\\tilde{y}^k-\\theta^{k})/\\gamma\\| \\\\\n\t\t\t& = \\frac{1}{\\gamma}\\left\\| \\theta^*_{\\gamma}(x^{k},\\tilde{y}^k) - \\theta^{k} \\right\\| \\le \\frac{C_\\eta}{\\gamma} \\mathcal{G}({\\theta}^{k}, x^{k}, \\tilde{y}^{k}) \\le \\frac{C_\\eta}{\\gamma} \\zeta_{k},\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere the first inequality follows from Lemma \\ref{error_bound} and the second inequality is a consequence of the inexactness condition \\eqref{inexacty2}. \n\t\n\tCombining the above inequality with \\eqref{lem1_eq7} gives us\n\t\\begin{equation}\\label{lem1_eq8}\n\t\t\\tilde{\\psi}_{p_k}\\left(x^{k}, y^{k+1}\\right) \\leq \\,  \t\\tilde{\\psi}_{p_k}\\left(x^{k}, \\tilde{y}^k\\right)  - \\left(\\frac{1}{\\beta_k}-\\frac{L_{\\psi_{y,k}}}{2}- \\frac{c_\\beta}{4}\\right)\\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2 + \\frac{C_\\eta^2}{c_\\beta\\gamma^2} \\zeta_{k}^2.\n\t\\end{equation}\n\tNext, considering the update rule for $x^{k+1}$ given in \\eqref{update_x}, we have\n\t\\begin{equation}\\label{lem1_eq1}\n\t\t\\langle d_x^k, x^{k+1}-x^k\\rangle+\\frac{1}{\\alpha_k}\\|x^{k+1}-x^k\\|^2 \\le 0.\n\t\\end{equation}\t\n\tBy the assumptions, with $y^{k+1} \\in Y$, we have that $\\nabla_x F(x, y^{k+1})$, $\\nabla_x f(x, y^{k+1})$ and $\\nabla_{x} g(x, y^{k+1})$ are $L_{F_x}$-, $L_{f_x}$-, and $L_{g_1}$-Lipschitz continuous with respect to variable $x$ on $X$, respectively, and combine this with \\eqref{lem9_eq1}, we have\n\t\\begin{equation}\\label{lem1_eq2}\n\t\t\\tilde{\\psi}_{p_k}(x^{k+1}, y^{k+1})\\le  \\tilde{\\psi}_{p_k}\t(x^{k}, y^{k+1})+\\langle \\nabla_x \t\\tilde{\\psi}_{p_k}(x^{k}, y^{k+1}),x^{k+1}-x^k\\rangle+\\frac{L_{\\psi_{x,k}}}{2}\\|x^{k+1}-x^k\\|^2,\n\t\\end{equation}\n\twhere $L_{\\psi_{x,k}}=L_{F_x}/p_k+L_{f_x}+L_{g_1}+ \\rho_{f_1} + \\rho_{g_1} $. Combining \\eqref{lem1_eq1} and \\eqref{lem1_eq2} yields\n\t\\begin{equation}\\label{lem1_eq3}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k+1}, y^{k+1}\\right) \\leq \\,& \\tilde{\\psi}_{p_k}\\left(x^k, y^{k+1}\\right) + \\langle \\nabla_x \\tilde{\\psi}_{p_k}(x^{k},y^{k+1}) - d_x^k,x^{k+1}-x^k\\rangle  \\\\\n\t\t\t&-\\left(\\frac{1}{\\alpha_k}-\\frac{L_{\\psi_{x,k}}}{2}\\right)\\left\\|x^{k+1}-x^k\\right\\|^2.\n\t\t\\end{aligned}\n\t\\end{equation}\n\tUsing the formula of the gradient of $v_\\gamma$ given in \\eqref{valuefinc} and the construction of $d_x^k$ in \\eqref{dx}, and the $L_{f_x}$- and $L_{g_2}$-Lipschitz continuity of $\\nabla_x f(x^k,y)$ and $\\nabla_{x} g(x^k,y)$ with respect to variable $y$ on $Y$, we have\n\t\\begin{equation}\\label{lem1_eq10}\n\t\t\\begin{aligned}\n\t\t\t\\left\\| \\nabla_x  \\tilde{\\psi}_{p_k}(x^{k},y^{k+1}) - d_x^k \\right\\| \n\t\t\t&= \\left\\| \\nabla_{x} \\varphi(x^k, \\theta^*_{\\gamma}(x^k,y^{k+1})) - \\nabla_{x} \\varphi(x^k, \\theta^{k+1/2}) \\right\\| \\\\\n\t\t\t& \\le (L_{f_x} + L_{g_2})\\left\\| \\theta^*_{\\gamma}(x^k,y^{k+1}) - \\theta^{k+1/2} \\right\\| \\le (L_{f_x} + L_{g_2})C_\\eta \\zeta_{k},\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere the last inequality follows from \\cref{error_bound} and the inexact criterion \\eqref{inexacty1}. Combining the above inequality with \\eqref{lem1_eq3} gives us\n\t\\begin{equation}\\label{lem1_eq4}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k+1}, y^{k+1}\\right) \\leq \\,& \\tilde{\\psi}_{p_k}\\left(x^k, y^{k+1}\\right) - \\left(\\frac{1}{\\alpha_k}-\\frac{L_{\\psi_{x,k}}}{2}-\\frac{c_\\alpha}{4}\\right)\\left\\|x^{k+1}-x^k\\right\\|^2 \\\\ & + \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2C_\\eta^2 \\zeta_{k}^2.\n\t\t\\end{aligned}\n\t\\end{equation}\n\tCombining  \\eqref{lem1_eq8}, \\eqref{lem1_eq4}, and then  \\eqref{lem1_eq12} established in  Lemma \\ref{lem0}, we obtain\n\t\\begin{equation}\\label{lem1_eq9}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k+1}, \\tilde{y}^{k+1}\\right)\\leq\\,& \t\\tilde{\\psi}_{p_k}\\left(x^k, \\tilde{y}^k\\right)+  \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2C_\\eta^2 \\zeta_{k}^2 + \\frac{C_\\eta^2}{c_\\beta\\gamma^2} \\zeta_{k}^2 + C_{\\gamma} \\zeta_{k+1}^2\\\\\n\t\t\t&\\hspace{-50pt}-\\left(\\frac{1}{\\alpha_k}-\\frac{L_{\\psi_{x,k}}}{2}- \\frac{c_\\alpha}{4} \\right)\\left\\|x^{k+1}-x^k\\right\\|^2 -\\left(\\frac{1}{\\beta_k}-\\frac{L_{\\psi_{y,k}}}{2}- \\frac{c_\\beta}{4} \\right)\\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2.\n\t\t\\end{aligned}\n\t\\end{equation}\n\tBecause  $\\alpha_k \\in [0, 2/(L_{\\psi_{x,k}} + c_\\alpha)]$ and $\\beta_k \\in [0, 2/(L_{\\psi_{y,k}} + c_\\beta)]$, and $\\tilde{\\psi}_{p_{k+1}}(x^{k+1}, \\tilde{y}^{k+1}) \\leq \t\\tilde{\\psi}_{p_k}(x^{k+1}, \\tilde{y}^{k+1})$ following from the non-decreasing property of the penalty parameter $p_k$ and the non-negative property of $F-\\underline{F}$, we can obtain \\eqref{lem1_eq} from \\eqref{lem1_eq9}.\n\t\n\tFinally, we show that \\( \\{\\zeta_k\\} \\) is square summable. Let $ \\mathcal{G}_0 = \\mathcal{G}(\\theta^{0}, x^{0}, y^{0})$ and $r_k :=  ( \\prod_{i=0}^{k} \\tau_i )\\mathcal{G}_0 $. Since $\\zeta_{k} := \\max\\{ s_k, r_k\\}$, we have $\\sum_{k=0}^\\infty \\zeta_k^2 \\le \\sum_{k=0}^\\infty s_k^2 + \\sum_{k=0}^\\infty r_k^2$. By assumption, \\( \\sum_{k=0}^\\infty s_k^2 < \\infty \\). Thus, we only need to show that  \\( \\{r_k\\} \\) is square summable. Recall that \\( \\tau_k = \\tau_0/(k+1)^{p_\\tau} \\) with \\( p_\\tau > 0 \\), then\n\t\\[\n\tr_k = \\left( \\prod_{i=0}^{k} \\tau_i \\right) \\mathcal{G}_0  = \\tau_0^{k+1} \\left( \\prod_{i=1}^{k+1} \\frac{1}{i^{p_\\tau}} \\right) \\cdot \\mathcal{G}_0  = \\frac{\\tau_0^{k+1}}{((k+1)!)^{p_\\tau}}\\mathcal{G}_0 .\n\t\\]\n\tFor the ratio test on  $\\sum_{k=0}^\\infty r_k^2$, we consider \n\t\\[\n\t\\lim_{k \\rightarrow \\infty}\\frac{r_{k+1}^2}{r_k^2} =  \\lim_{k \\rightarrow \\infty} \\frac{\\tau_0^2}{(k+2)^{2p_\\tau}}  = 0.\n\t\\]\n\tTherefore, by the ratio test, $\\sum_{k=0}^\\infty r_k^2 < \\infty$, implying that \\( \\{\\zeta_k\\} \\) is square summable.\n\\end{proof}\n\n\\begin{lemma}\\label{lem3}\n\tThe sequence $\\left\\{(x^k,y^k,\\tilde{y}^k)\\right\\}$ satisfies\n\t\\begin{equation*}\n\t\t\\sum\\limits_{k=0}^{{\\infty}}\\left\\|(x^{k+1},y^{k+1})-(x^k,\\tilde{y}^k)\\right\\|^2<\\infty, \\quad \\text{and} \\quad \\lim\\limits_{k\\to\\infty}\\left\\|(x^{k+1},y^{k+1})-(x^k,\\tilde{y}^k)\\right\\|=0.\n\t\\end{equation*}\n\\end{lemma}\n\n\\begin{proof}\n\tSumming  \\cref{lem1_eq} from Lemma \\ref{lem1} over $k=0,1,\\dots,K-1$ yields,\n\t\\begin{equation}\\label{lem3_eq1}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_{K}}\\left(x^{K}, \\tilde{y}^{K}\\right)- \\tilde{\\psi}_{p_{0}}\\left(x^{0}, \\tilde{y}^{0}\\right) \\leq & -\\sum\\limits_{k=0}^{K-1} \\left(  \\frac{c_\\alpha}{4} \\left\\|x^{k+1}-x^k\\right\\|^2+ \\frac{c_\\beta}{4} \\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2\\right)\\\\& +\\sum\\limits_{k=0}^{K-1}\\left( \\frac{C_\\eta^2 }{c_\\alpha}(L_{f_x} + L_{g_2})^2+ \\frac{C_\\eta^2}{c_\\beta\\gamma^2} \\right) \\zeta_{k}^2+ \\sum\\limits_{k=0}^{K-1}C_{\\gamma} \\zeta_{k+1}^2.\\\\\n\t\t\\end{aligned}\n\t\\end{equation}\n\tSince \\( \\{\\zeta_k\\} \\) is square summable from Lemma \\ref{lem1}, it follows that\n\t\\[\n\t\\sum\\limits_{k=0}^{\\infty} \\left( \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2 + \\frac{1}{c_\\beta\\gamma^2} \\right) C_\\eta^2\\zeta_{k}^2 + \\sum\\limits_{k=0}^{\\infty}  C_{\\gamma} \\zeta_{k+1}^2 < \\infty.\n\t\\]\n\tAdditionally, because $\\tilde{\\psi}_{p_{K}}\\left(x^{K}, \\tilde{y}^{K}\\right) \\ge 0$, by taking $K \\rightarrow \\infty$ in \\eqref{lem3_eq1}, we have\n\t\\begin{equation}\\label{lem3_eq2}\n\t\t\\begin{aligned}\n\t\t\t&\\quad\\,\\sum\\limits_{k=0}^{\\infty} \\left(  \\frac{c_\\alpha}{4} \\left\\|x^{k+1}-x^k\\right\\|^2+ \\frac{c_\\beta}{4} \\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2\\right) \\\\\n\t\t\t&\\le \\tilde{\\psi}_{p_{0}}\\left(x^{0}, \\tilde{y}^{0}\\right) + \\sum\\limits_{k=0}^{\\infty} \\left( \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2 + \\frac{1}{c_\\beta\\gamma^2} \\right) C_\\eta^2\\zeta_{k}^2 + \\sum\\limits_{k=0}^{\\infty}  C_{\\gamma} \\zeta_{k+1}^2  < \\infty,\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhich proves the result.\n\\end{proof}\n\n\\begin{lemma}\\label{lem4}\n\tLet $\\{ (x^{k_j},\\tilde{y}^{k_j}) \\} $ be any convergent subsequence of iterates $\\{(x^k,\\tilde{y}^k)\\}$ such that $(x^{k_j},\\tilde{y}^{k_j})$ satisfies condition \\eqref{inexacty_y} with $k = k_j - 1$. \n\tIf $(x^{k_j},\\tilde{y}^{k_j}) \\rightarrow (\\bar{x},\\bar{y}) $ as $j \\rightarrow \\infty$ for some $(\\bar{x},\\bar{y}) $, then $\\bar{y} \\in S(\\bar{x})$. \n\\end{lemma}\n\n\\begin{proof}\n\tDefine the residual term $\\tilde{r}^k := \\tilde{y}^{k} - \\operatorname{Prox}_{ \\tilde{g}(x^{k}, \\cdot)} \\left( \\tilde{y}^{k} -  \\nabla_y f(x^{k}, \\tilde{y}^{k}) \\right) $. By the first-order optimality condition for the proximal operator, $\\tilde{r}^k$ satisfies\n\t\\begin{equation}\\label{lem4_eq1}\n\t\t\\tilde{r}^k \\in \\nabla_y f(x^{k}, \\tilde{y}^{k}) + \\partial_{y}\\tilde{g}(x^{k}, \\tilde{y}^{k} - \\tilde{r}^k).\n\t\\end{equation}\n\tThe sequence  $(x^{k_j},\\tilde{y}^{k_j})$ satisfies condition \\eqref{inexacty_y} with $k = k_j - 1$, which implies \n\t\\[\n\t\\left\\| \\tilde{r}^{k_j} \\right\\| \\le \\frac{c_{\\tilde{y}} }{p_{k_j}} \\left\\|(x^{k_j},y^{k_j})-(x^{k_j-1},\\tilde{y}^{k_j-1})\\right\\|.\n\t\\]\n\tFrom Lemma \\ref{lem3}, we have $\\lim_{k_j \\to\\infty} \\|(x^{k_j},y^{k_j})-(x^{k_j-1},\\tilde{y}^{k_j-1}) \\|=0$. As $p_{k_j} \\ge p_0$, the above inequality ensures that $\\lim_{k_j \\to\\infty} \\|\\tilde{r}^{k_j} \\|=0$.\n\t\n\tTaking the limit as  $k_j \\rightarrow \\infty$ in \\eqref{lem4_eq1} with $k = k_j$, by the continuity of $ \\nabla_y f$ and the outer semicontinuity of $\\partial_{y}\\tilde{g}$, it follows that\n\t\\[\n\t0  \\in \\nabla_y f(\\bar{x},\\bar{y}) + \\partial_{y}\\tilde{g}(\\bar{x},\\bar{y}).\n\t\\] \n\tSince the lower-level problem is convex in $y$, this condition\n\timplies $\\bar{y} \\in S(\\bar{x})$.\n\\end{proof}\n\n\\begin{proposition}\\label{prop0}\n\tLet $\\epsilon > 0$. If the sequence $\\{p_k\\}$ is bounded, then any accumulation point $(\\bar{x}, \\bar{y})$ of the iterate sequence $\\{(x^k, {y}^k)\\}$ satisfies\n\t\\[\n\t\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon.\n\t\\]\n\\end{proposition}\n\n\\begin{proof}\n\t\n\tLet $(\\bar{x}, \\bar{y})$ be an accumulation point of $\\{(x^k, {y}^k)\\}$, and let \\(\\{k_j\\}_{j \\in \\mathbb{N}} \\subseteq \\mathbb{N} \\) be an index subsequence such that $(x^{k_j+1},y^{k_j+1})\\to (\\bar{x},\\bar{y})$ as $j \\rightarrow \\infty$. By Lemma \\ref{lem3}, we have  $\\lim_{k_j \\to\\infty} \\|(x^{k_j+1},y^{k_j+1})-(x^{k_j},\\tilde{y}^{k_j}) \\|=0$, which implies that $(x^{k_j},\\tilde{y}^{k_j})$ also converges to $ (\\bar{x},\\bar{y})$ .\n\t\n\tSince the sequence $\\{p_k\\}$ is bounded and non-decreasing by construction in AGILS (\\cref{alg1}), there exists $k_0 >0$ such that $p_k = p_{k_0}$ for all $k \\ge k_0$. By passing to a subsequence if necessary, we can assume that $k_j - 1\\ge k_0$ for all $j \\in \\mathbb{N}$.\t\n\t\n\tAccording to the update rule of $p_k$ in AGILS (\\cref{alg1}), the condition $p_{k_j - 1} = p_{k_j}$ implies that one of the following two cases must occur for each \\(  k = k_j -1 \\):\n\t\n\tCase I: $\\left\\|(x^{k_j},y^{k_j})-(x^{k_j-1}, \\tilde{y}^{k_j-1})\\right\\| \\ge c_p \\min\\left\\{1/p_{k_0},{t^{k_j}}\\right\\}$ and $\\tilde{y}^{k_j} = y^{k_j}$.\n\t\n\tCase II: $\\left\\|(x^{k_j},y^{k_j})-(x^{k_j-1}, \\tilde{y}^{k_j-1})\\right\\| < c_p \\min\\left\\{1/p_{k_0},{t^{k_j}}\\right\\}$ and the feasibility correction procedure is applied to generate $\\tilde{y}^{k_j}$ satisfying condition \\eqref{inexacty_y}.\n\t\n\tAt least one of these two cases must hold for an infinite number of indices in the subsequence \t$\\{ k_j  - 1\\}$. We analyze them separately.\n\t\n\tCase 1:  Suppose Case I occurs for an infinite subsequence of indices. By passing to a further subsequence, we assume it holds for all $j \\in \\mathbb{N}$.\n\t\n\tThe condition $\\tilde{y}^{k_j} = y^{k_j}$ combined with $(x^{k_j},\\tilde{y}^{k_j})\\to (\\bar{x},\\bar{y})$ implies that $(x^{k_j}, {y}^{k_j})$\\\\\n\t$ \\to (\\bar{x},\\bar{y})$. \n\tFrom Lemma \\ref{lem3}, we know $\\lim_{k_j \\to\\infty} \\|(x^{k_j},y^{k_j})-(x^{k_j-1},\\tilde{y}^{k_j-1}) \\|=0$. For the condition of Case I to hold, we must therefore have  $t^{k_j} \\rightarrow 0$. This implies\n\t\\begin{equation*}\n\t\t\\limsup_{j \\rightarrow \\infty}\t\\left\\{ \\varphi(x^{k_j}, y^{k_j})-\\varphi(x^{k_j},\\theta^{k_j})-\\frac{1}{2\\gamma}\\left\\| \\theta^{k_j} - y^{k_j} \\right\\|^2-\\epsilon \\right\\} \\le 0.\n\t\\end{equation*}\n\tSince $\\zeta_{k} \\rightarrow 0$ and $\\mathcal{G}( \\theta^{k_j}, x^{k_j}, y^{k_j}) \\le \\zeta_{k_j}$, Lemma \\ref{error_bound} ensures that $\\| \\theta^{k_j} - \\theta^*_{\\gamma}( x^{k_j}, y^{k_j} )\\|\\rightarrow 0$.\n\tBy the continuity of $\\theta^*_{\\gamma}(x,y)$ (Proposition \\ref{partial_grad_v}), it follows that $\\theta^{k_j} \\rightarrow \\theta^*_{\\gamma}(\\bar{x}, \\bar{y}) $.\n\tThe continuity of $\\varphi$ then implies that $\\varphi(x^{k_j}, \\theta^{k_j})+\\frac{1}{2\\gamma}\\|\\theta^{k_j} - y^{k_j}\\|^2 \\rightarrow v_\\gamma(\\bar{x},\\bar{y})$. Then taking the limit in the inequality above as  $j \\rightarrow \\infty$ yields $\\varphi(\\bar{x},\\bar{y})-v_{\\gamma}(\\bar{x},\\bar{y})-\\epsilon \\le 0$. \n\t\n\tCase 2: \n\tNow, suppose Case II occurs for an infinite subsequence of indices. Again, we assume it holds for all $j \\in \\mathbb{N}$.\n\t\n\tIn this scenario, the feasibility correction procedure is applied at each iteration $k = k_j -1$ to generate $\\tilde{y}^{k_j}$ satisfying condition \\eqref{inexacty_y}. Since $(x^{k_j},\\tilde{y}^{k_j}) \\rightarrow (\\bar{x},\\bar{y}) $ and satisfies condition \\eqref{inexacty_y}, it follows from Lemma \\ref{lem4} that $\\bar{y} \\in S(\\bar{x})$. For any such point, $\\theta^*_{\\gamma}(\\bar{x}, \\bar{y}) = \\bar{y} $ (see, e.g., \\cite[Theorem 10.7]{beck2017first}), and therefore \\( \\varphi(\\bar{x}, \\bar{y}) - v_\\gamma(\\bar{x}, \\bar{y}) = 0 \\).  Since $ \\epsilon > 0$, the constraint $\t\t\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon$ is satisfied.\n\t\n\tSince any accumulation point must be the limit of a subsequence that falls into one of these two cases, we conclude that $\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon$.\n\\end{proof}\n\n\\begin{proof}\n\tThe proof builds upon techniques used in the proof of Theorem 1 in \\cite{ye2023}. \n\tLet $(\\bar{x}, \\bar{y})$ be an accumulation point of the sequence $\\{(x^k,y^k)\\}$, and let $\\{(x^{k_j+1},y^{k_j+1})\\}$ be a convergent subsequence such that $(x^{k_j+1},y^{k_j+1}) \\rightarrow (\\bar{x}, \\bar{y})$ as $k_j \\rightarrow \\infty$. \n\tBy Lemma \\ref{lem3}, we have  $\\lim_{k_j \\to\\infty} \\|(x^{k_j+1},y^{k_j+1})-(x^{k_j},\\tilde{y}^{k_j}) \\|=0$, which implies that $(x^{k_j},\\tilde{y}^{k_j}) \\to (\\bar{x},\\bar{y})$ .\n\tSince $\\{p_k\\}$ is bounded, there exists $k_0 > 0$ such that $p_k = \\bar{p} := p_{k_0}$ for any $k \\ge k_0$. \n\t\n\t\n\tFrom the update rules \\eqref{update_x} and \\eqref{update_y}, and the expressions for $d_x^k$ and $d_y^k$, we have    \n\t\\begin{equation}\\label{th1_eq2}\n\t\t\\begin{aligned}\n\t\t\t\\xi_x^k & \\in \\nabla_x F(x^{k+1},y^{k+1})+p_k \\left(\\nabla_x \\varphi(x^{k+1},y^{k+1}) - \\nabla_x v_\\gamma(x^{k+1},y^{k+1})\\right) +\\mathcal{N}_{X}(x^{k+1}),\\\\\n\t\t\t\\xi_y^k & \\in \\nabla_y F(x^{k},y^{k+1})+p_k \\left(\\partial_y \\varphi(x^{k},y^{k+1}) - \\nabla_y v_\\gamma(x^{k},y^{k+1})\\right) +\\mathcal{N}_{Y}(y^{k+1}),\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere $\\xi_x^k \\in \\Re^n$ and $\\xi_y^k \\in \\Re^m$ are given by\n\t\\begin{equation}\\label{th1_eq4}\n\t\t\\begin{aligned}\n\t\t\t\\xi_x^k &:=  p_k \\left(\\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - d_x^k \\right)  - \\frac{p_k}{\\alpha_k}\\left(x^{k+1}-x^k\\right), \\\\\n\t\t\t\\xi_y^k &:=  p_k \\left( \\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, y^{k+1}\\right) - d_y^k \\right)-\\frac{p_k}{\\beta_k}\\left(y^{k+1}-\\tilde{y}^k\\right).\n\t\t\\end{aligned} \n\t\\end{equation} \n\tWe now show that $\\xi_x^{k_j}, \\xi_y^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$. First, we have \n\t\\begin{equation}\n\t\t\\begin{aligned}\n\t\t\t& \\| \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - d_x^k\\| \\\\ \\le \\, & \\|  \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k}, y^{k+1} \\right) \\| + \\| \\nabla_x \\tilde{\\psi}_{p_{k}} \\left(x^{k}, y^{k+1} \\right) - d_x^k \\| \\\\\n\t\t\t\\le \\, & \\|  \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k}, y^{k+1} \\right) \\| + (L_{f_x} + L_{g_2})C_\\eta \\zeta_{k}, \n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere the last inequality follows from \\eqref{lem1_eq10}. Since $p_k = \\bar{p}$ for all $k \\ge k_0$, and assuming that $\\nabla_{x} F, \\nabla_{x} f$ and $\\nabla_{x} g$ are continuous on $X \\times Y$, and $\\nabla_{x} v_\\gamma$ is continuous on $X \\times Y$ (Proposition \\ref{partial_grad_v}), we conclude that $ \\nabla_x \\tilde{\\psi}_{p_{k}} =  \\nabla_x \\tilde{\\psi}_{\\bar{p}}$ is continuous on $X \\times Y$. \n\tAs $\\lim_{k_j\\to\\infty} (x^{k_j+1},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},\\tilde{y}^{k_j}) = (\\bar{x}, \\bar{y})$ and $\\zeta_{k} \\rightarrow 0$, it follows that $\\nabla_x \\tilde{\\psi}_{p_{k_j}} (x^{k_j+1}, y^{k_j+1} ) - d_x^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$. Combining this with $ \\|x^{k_j+1}-x^{k_j} \\| \\to 0$ and $\\alpha_k \\ge \\underline{\\alpha} > 0$, we obtain from \\eqref{th1_eq4} that $\\xi_x^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$.\n\t\n\tA similar argument can be applied to $\\xi_y^k$. From \\eqref{lem1_eq11}, we have \n\t\\begin{equation}\\label{th1_eq5}\n\t\t\\begin{aligned}\n\t\t\t& \\|\\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, y^{k+1}\\right) - d_y^k \\| \\\\ \\le \\, & \\| \\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, y^{k+1} \\right) - \\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, \\tilde{y}^k \\right) \\| + {C_\\eta}\\zeta_{k}/\\gamma.\n\t\t\\end{aligned}\n\t\\end{equation}\n\tSince $p_k = \\bar{p}$ for all $k \\ge k_0$, and under the assumption that $\\nabla_{y} F$ and $\\nabla_{y} f$ are continuous on $X \\times Y$, and $\\nabla_{y} v_\\gamma$ is continuous on $X \\times Y$ (Proposition \\ref{partial_grad_v}), we have that $ \\nabla_y (\\tilde{\\psi}_{p_{k}} - g) = \\nabla_y (\\tilde{\\psi}_{\\bar{p}} - g)$ is continuous on $X \\times Y$. As $\\lim_{k_j\\to\\infty} (x^{k_j},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},\\tilde{y}^{k_j}) = (\\bar{x}, \\bar{y})$ and $\\zeta_{k} \\rightarrow 0$, it follows that $\\nabla_y (\\tilde{\\psi}_{p_{k_j}} - g)(x^{k_j}, y^{k_j+1} ) - d_y^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$. Combining this with $ \\|y^{k_j+1}-\\tilde{y}^{k_j} \\| \\to 0$ and $\\beta_k \\ge \\underline{\\beta} > 0$, we conclude from \\eqref{th1_eq5} that $\\xi_y^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$.\n\t\n\tNow, taking the limit as $k = k_j \\rightarrow \\infty$ in \\eqref{th1_eq2}, since $\\xi_x^{k_j}, \\xi_y^{k_j} \\rightarrow 0$, $p_k = \\bar{p}$ for all $k \\ge k_0$, $\\lim_{k_j\\to\\infty} (x^{k_j+1},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},\\tilde{y}^{k_j}) = (\\bar{x}, \\bar{y})$, $\\nabla F$, $\\nabla v_\\gamma$, and $\\nabla_x \\varphi$ are continuous on $X \\times Y$, $\\partial_y \\varphi$ and $\\mathcal{N}_{X\\times Y}$ are outer semicontinuous on $X \\times Y$, and $\\partial \\varphi = \\{\\nabla_x \\varphi\\} \\times \\partial_y \\varphi$ on $X \\times Y$, we obtain \n\t\\begin{equation}\\label{th1_eq6}\n\t\t0\\in\\nabla F(\\bar{x}, \\bar{y})+\\bar{p}\\left(\\partial \\varphi(\\bar{x},\\bar{y})-\\nabla v_\\gamma(\\bar{x}, \\bar{y})\\right)+\\mathcal{N}_{X\\times Y}(\\bar{x}, \\bar{y}).\n\t\\end{equation}\n\t\n\tFinally, from Proposition \\ref{prop0}, we know that the limit point $(\\bar{x}, \\bar{y})$ satisfies $\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon $. Let $\\bar{\\epsilon}:= \\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\in [0, \\epsilon]$. Combining this with \\eqref{th1_eq6}, we have\n\t\\begin{equation}\n\t\t\\left\\{\n\t\t\\begin{aligned}\n\t\t\t&0\\in\\nabla F(\\bar{x}, \\bar{y}) + \\bar{p}\\left( \\partial \\varphi(\\bar{x}, \\bar{y}) - \\nabla v_\\gamma(\\bar{x}, \\bar{y})\\right)+\\mathcal{N}_{X\\times Y}(\\bar{x}, \\bar{y}),\\\\\n\t\t\t&\\varphi(\\bar{x}, \\bar{y}) - v_{\\gamma}(\\bar{x}, \\bar{y})- \\bar{\\epsilon} \\le 0, \\qquad  \\bar{p}\\left( \\varphi(\\bar{x}, \\bar{y}) - v_{\\gamma}(\\bar{x}, \\bar{y})- \\bar{\\epsilon} \\right)=0,\n\t\t\\end{aligned}\\right.\n\t\\end{equation}\n\twhich implies that $(\\bar{x},\\bar{y})$ is a KKT point of problem $({\\rm VP})_\\gamma^{\\bar{\\epsilon}}$ with $\\bar{\\epsilon}\\le \\epsilon$.\n\\end{proof}","units":[{"raw_tex":"\\begin{lemma}\\label{lem9}\n\tSuppose Assumptions \\ref{asup2} and \\ref{asup3} hold. Let $\\gamma \\in (0, 1/(\\rho_{f_2} +\\rho_{g_2}) )$ and  $(\\bar{x},\\bar{y})\\in X\\times \\Re^m$. Then, for any $(x,y) \\in X \\times \\Re^m$, the following inequalities hold,\n\t\\begin{align}\n\t\t-v_{\\gamma}(x,\\bar{y}) &\\le -v_{\\gamma}(\\bar{x},\\bar{y})-\\langle \\nabla_x v_{\\gamma}(\\bar{x},\\bar{y}),x-\\bar{x}\\rangle +\\frac{\\rho_{f_1} + \\rho_{g_1}}{2}\\|\\bar{x}-x\\|^2,  \\label{lem9_eq1} \\\\\n\t\t-v_{\\gamma}(\\bar{x},y) &\\le -v_{\\gamma}(\\bar{x},\\bar{y})-\\langle \\nabla_y v_{\\gamma}(\\bar{x},\\bar{y}),y-\\bar{y}\\rangle. \\label{lem9_eq2}\n\t\\end{align}\n\\end{lemma}","start":52691,"end":53311},{"raw_tex":"\\begin{proof}\n\tAccording to Proposition \\ref{partial_grad_v}, for any $\\bar{y} \\in \\Re^m$, $v_{\\gamma}(x, \\bar{y})$ is a $(\\rho_{f_1} + \\rho_{g_1})$-weakly convex function with respect to $x$ on $X$, that is, $v_{\\gamma}(x,\\bar{y})+\\frac{\\rho_{f_1} + \\rho_{g_1}}{2}\\|x\\|^2$ is convex on $X$. Then we can obtain \\eqref{lem9_eq1} immediately. The second inequality \\eqref{lem9_eq2} follows from Proposition \\ref{partial_grad_v} that for any $\\bar{x} \\in X$, $v_{\\gamma}(\\bar{x}, y)$ is convex with respect to $y$.\n\\end{proof}","start":53312,"end":53835},{"raw_tex":"\\begin{lemma}\\label{lem1}\n\tSuppose Assumptions \\ref{asup1}, \\ref{asup2} and \\ref{asup3} hold. Let $\\gamma \\in (0, 1/(\\rho_{f_2} +\\rho_{g_2}) )$, $\\alpha_k \\in [0, 2/(L_{\\psi_{x,k}} + c_\\alpha)]$ and $\\beta_k \\in [0, 2/(L_{\\psi_{y,k}} + c_\\beta)]$, where $L_{\\psi_{x,k}} = L_{F_x}/p_k+L_{f_x}+L_{g_1}+\\rho_{f_1} + \\rho_{g_1}$, $L_{\\psi_{y,k}} = L_{F_y}/p_k+L_{f_y}$ and $c_\\alpha, c_\\beta$ are positive constants. Then, there exists $C_\\eta , C_\\gamma > 0$ such that the sequence $\\{(x^k,y^k,\\tilde{y}^k)\\}$ generated by AGILS {\\rm (}\\cref{alg1}{\\rm )} satisfies the following inequality,\n\t\\begin{equation}\\label{lem1_eq}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_{k+1}}\\left(x^{k+1}, \\tilde{y}^{k+1}\\right)\\leq\\,& \\tilde{\\psi}_{p_k}\\left(x^k, \\tilde{y}^k\\right) +  \\left( \\frac{C_\\eta^2}{c_\\alpha}(L_{f_x} + L_{g_2})^2 + \\frac{C_\\eta^2}{c_\\beta\\gamma^2 } \\right) \\zeta_{k}^2 + C_{\\gamma} \\zeta_{k+1}^2\\\\\n\t\t\t&- \\frac{c_\\alpha}{4} \\left\\|x^{k+1}-x^k\\right\\|^2  -\\frac{c_\\beta}{4} \\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2,\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere  $\\zeta_{k} := \\max\\{ s_k,  ( \\prod_{i=0}^{k} \\tau_i ) \\mathcal{G}(\\theta^{0}, x^{0}, y^{0})\\}$ and $\\{ \\zeta_{k} \\}$ is square summable.\n\\end{lemma}","start":60296,"end":61495},{"raw_tex":"\\begin{proof}\n\t\n\tFirst, the update rule for \\( y^{k+1} \\) given in \\eqref{update_y} defines it as the minimizer of the function $ g(x^k, y) + \\langle d_y^k, y - \\tilde{y}^{k} \\rangle + \\frac{1}{2\\beta_k} \\left\\| y - \\tilde{y}^k \\right\\|^2$. This function is $1/\\beta_k$-strongly convex in $y$. By applying \\cite[Theorem 6.39]{beck2017first}, we obtain\n\t\\begin{equation}\\label{lem1_eq5}\n\t\tg(x^{k}, y^{k+1}) + \\langle d_y^k, y^{k+1}-\\tilde{y}^k\\rangle+\\frac{1}{\\beta_k}\\|y^{k+1}-\\tilde{y}^k\\|^2 \\le g(x^{k}, \\tilde{y}^{k}).\n\t\\end{equation}\n\t\n\t\n\tAccording to the assumptions, with $x^{k} \\in X$, we have that $\\nabla_y F(x^{k},y)$ and $\\nabla_y f(x^{k},y)$ are $L_{F_y}$- and  $L_{f_y}$-Lipschitz continuous with respect to variable $y$ on $Y$, respectively, and combine this with \\eqref{lem9_eq2}, we have\n\t\\begin{equation}\\label{lem1_eq6}\n\t\t\\begin{aligned}\n\t\t\t&(\\tilde{\\psi}_{p_k}-g)(x^{k},y^{k+1})  \\\\\n\t\t\t\\le \\, & (\\tilde{\\psi}_{p_k}-g)\t(x^{k},\\tilde{y}^k) + \\langle \\nabla_y (\\tilde{\\psi}_{p_k}-g)(x^{k},\\tilde{y}^k),y^{k+1}-\\tilde{y}^k\\rangle +\\frac{L_{\\psi_{y,k}}}{2}\\|y^{k+1}-\\tilde{y}^k\\|^2,\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere $L_{\\psi_{y,k}}=L_{F_y}/p_k+L_{f_y}$. \n\tCombining \\eqref{lem1_eq5} and \\eqref{lem1_eq6} yields\n\t\\begin{equation}\\label{lem1_eq7}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k}, y^{k+1}\\right) \\leq \\,& \t\\tilde{\\psi}_{p_k}\\left(x^{k}, \\tilde{y}^k\\right) + \\langle \\nabla_y (\t\\tilde{\\psi}_{p_k}-g)(x^{k},\\tilde{y}^k) - d_y^k, y^{k+1}-\\tilde{y}^k\\rangle  \\\\\n\t\t\t&-\\left(\\frac{1}{\\beta_k}-\\frac{L_{\\psi_{y,k}}}{2}\\right)\\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2.\n\t\t\\end{aligned}\n\t\\end{equation}\t\n\tUsing the formula of \\( \\nabla_y v_\\gamma \\) in \\eqref{valuefinc}, and the construction of $d_y^k$ in \\eqref{dy}, we have\n\t\\begin{equation}\\label{lem1_eq11}\n\t\t\\begin{aligned}\n\t\t\t\\left\\| \\nabla_y (\t\\tilde{\\psi}_{p_k}-g)(x^{k},\\tilde{y}^k) - d_y^k \\right\\| & = \\| \\nabla_y v_\\gamma(x^k, \\tilde{y}^k) + (\\tilde{y}^k-\\theta^{k})/\\gamma\\| \\\\\n\t\t\t& = \\frac{1}{\\gamma}\\left\\| \\theta^*_{\\gamma}(x^{k},\\tilde{y}^k) - \\theta^{k} \\right\\| \\le \\frac{C_\\eta}{\\gamma} \\mathcal{G}({\\theta}^{k}, x^{k}, \\tilde{y}^{k}) \\le \\frac{C_\\eta}{\\gamma} \\zeta_{k},\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere the first inequality follows from Lemma \\ref{error_bound} and the second inequality is a consequence of the inexactness condition \\eqref{inexacty2}. \n\t\n\tCombining the above inequality with \\eqref{lem1_eq7} gives us\n\t\\begin{equation}\\label{lem1_eq8}\n\t\t\\tilde{\\psi}_{p_k}\\left(x^{k}, y^{k+1}\\right) \\leq \\,  \t\\tilde{\\psi}_{p_k}\\left(x^{k}, \\tilde{y}^k\\right)  - \\left(\\frac{1}{\\beta_k}-\\frac{L_{\\psi_{y,k}}}{2}- \\frac{c_\\beta}{4}\\right)\\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2 + \\frac{C_\\eta^2}{c_\\beta\\gamma^2} \\zeta_{k}^2.\n\t\\end{equation}\n\tNext, considering the update rule for $x^{k+1}$ given in \\eqref{update_x}, we have\n\t\\begin{equation}\\label{lem1_eq1}\n\t\t\\langle d_x^k, x^{k+1}-x^k\\rangle+\\frac{1}{\\alpha_k}\\|x^{k+1}-x^k\\|^2 \\le 0.\n\t\\end{equation}\t\n\tBy the assumptions, with $y^{k+1} \\in Y$, we have that $\\nabla_x F(x, y^{k+1})$, $\\nabla_x f(x, y^{k+1})$ and $\\nabla_{x} g(x, y^{k+1})$ are $L_{F_x}$-, $L_{f_x}$-, and $L_{g_1}$-Lipschitz continuous with respect to variable $x$ on $X$, respectively, and combine this with \\eqref{lem9_eq1}, we have\n\t\\begin{equation}\\label{lem1_eq2}\n\t\t\\tilde{\\psi}_{p_k}(x^{k+1}, y^{k+1})\\le  \\tilde{\\psi}_{p_k}\t(x^{k}, y^{k+1})+\\langle \\nabla_x \t\\tilde{\\psi}_{p_k}(x^{k}, y^{k+1}),x^{k+1}-x^k\\rangle+\\frac{L_{\\psi_{x,k}}}{2}\\|x^{k+1}-x^k\\|^2,\n\t\\end{equation}\n\twhere $L_{\\psi_{x,k}}=L_{F_x}/p_k+L_{f_x}+L_{g_1}+ \\rho_{f_1} + \\rho_{g_1} $. Combining \\eqref{lem1_eq1} and \\eqref{lem1_eq2} yields\n\t\\begin{equation}\\label{lem1_eq3}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k+1}, y^{k+1}\\right) \\leq \\,& \\tilde{\\psi}_{p_k}\\left(x^k, y^{k+1}\\right) + \\langle \\nabla_x \\tilde{\\psi}_{p_k}(x^{k},y^{k+1}) - d_x^k,x^{k+1}-x^k\\rangle  \\\\\n\t\t\t&-\\left(\\frac{1}{\\alpha_k}-\\frac{L_{\\psi_{x,k}}}{2}\\right)\\left\\|x^{k+1}-x^k\\right\\|^2.\n\t\t\\end{aligned}\n\t\\end{equation}\n\tUsing the formula of the gradient of $v_\\gamma$ given in \\eqref{valuefinc} and the construction of $d_x^k$ in \\eqref{dx}, and the $L_{f_x}$- and $L_{g_2}$-Lipschitz continuity of $\\nabla_x f(x^k,y)$ and $\\nabla_{x} g(x^k,y)$ with respect to variable $y$ on $Y$, we have\n\t\\begin{equation}\\label{lem1_eq10}\n\t\t\\begin{aligned}\n\t\t\t\\left\\| \\nabla_x  \\tilde{\\psi}_{p_k}(x^{k},y^{k+1}) - d_x^k \\right\\| \n\t\t\t&= \\left\\| \\nabla_{x} \\varphi(x^k, \\theta^*_{\\gamma}(x^k,y^{k+1})) - \\nabla_{x} \\varphi(x^k, \\theta^{k+1/2}) \\right\\| \\\\\n\t\t\t& \\le (L_{f_x} + L_{g_2})\\left\\| \\theta^*_{\\gamma}(x^k,y^{k+1}) - \\theta^{k+1/2} \\right\\| \\le (L_{f_x} + L_{g_2})C_\\eta \\zeta_{k},\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere the last inequality follows from \\cref{error_bound} and the inexact criterion \\eqref{inexacty1}. Combining the above inequality with \\eqref{lem1_eq3} gives us\n\t\\begin{equation}\\label{lem1_eq4}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k+1}, y^{k+1}\\right) \\leq \\,& \\tilde{\\psi}_{p_k}\\left(x^k, y^{k+1}\\right) - \\left(\\frac{1}{\\alpha_k}-\\frac{L_{\\psi_{x,k}}}{2}-\\frac{c_\\alpha}{4}\\right)\\left\\|x^{k+1}-x^k\\right\\|^2 \\\\ & + \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2C_\\eta^2 \\zeta_{k}^2.\n\t\t\\end{aligned}\n\t\\end{equation}\n\tCombining  \\eqref{lem1_eq8}, \\eqref{lem1_eq4}, and then  \\eqref{lem1_eq12} established in  Lemma \\ref{lem0}, we obtain\n\t\\begin{equation}\\label{lem1_eq9}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k+1}, \\tilde{y}^{k+1}\\right)\\leq\\,& \t\\tilde{\\psi}_{p_k}\\left(x^k, \\tilde{y}^k\\right)+  \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2C_\\eta^2 \\zeta_{k}^2 + \\frac{C_\\eta^2}{c_\\beta\\gamma^2} \\zeta_{k}^2 + C_{\\gamma} \\zeta_{k+1}^2\\\\\n\t\t\t&\\hspace{-50pt}-\\left(\\frac{1}{\\alpha_k}-\\frac{L_{\\psi_{x,k}}}{2}- \\frac{c_\\alpha}{4} \\right)\\left\\|x^{k+1}-x^k\\right\\|^2 -\\left(\\frac{1}{\\beta_k}-\\frac{L_{\\psi_{y,k}}}{2}- \\frac{c_\\beta}{4} \\right)\\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2.\n\t\t\\end{aligned}\n\t\\end{equation}\n\tBecause  $\\alpha_k \\in [0, 2/(L_{\\psi_{x,k}} + c_\\alpha)]$ and $\\beta_k \\in [0, 2/(L_{\\psi_{y,k}} + c_\\beta)]$, and $\\tilde{\\psi}_{p_{k+1}}(x^{k+1}, \\tilde{y}^{k+1}) \\leq \t\\tilde{\\psi}_{p_k}(x^{k+1}, \\tilde{y}^{k+1})$ following from the non-decreasing property of the penalty parameter $p_k$ and the non-negative property of $F-\\underline{F}$, we can obtain \\eqref{lem1_eq} from \\eqref{lem1_eq9}.\n\t\n\tFinally, we show that \\( \\{\\zeta_k\\} \\) is square summable. Let $ \\mathcal{G}_0 = \\mathcal{G}(\\theta^{0}, x^{0}, y^{0})$ and $r_k :=  ( \\prod_{i=0}^{k} \\tau_i )\\mathcal{G}_0 $. Since $\\zeta_{k} := \\max\\{ s_k, r_k\\}$, we have $\\sum_{k=0}^\\infty \\zeta_k^2 \\le \\sum_{k=0}^\\infty s_k^2 + \\sum_{k=0}^\\infty r_k^2$. By assumption, \\( \\sum_{k=0}^\\infty s_k^2 < \\infty \\). Thus, we only need to show that  \\( \\{r_k\\} \\) is square summable. Recall that \\( \\tau_k = \\tau_0/(k+1)^{p_\\tau} \\) with \\( p_\\tau > 0 \\), then\n\t\\[\n\tr_k = \\left( \\prod_{i=0}^{k} \\tau_i \\right) \\mathcal{G}_0  = \\tau_0^{k+1} \\left( \\prod_{i=1}^{k+1} \\frac{1}{i^{p_\\tau}} \\right) \\cdot \\mathcal{G}_0  = \\frac{\\tau_0^{k+1}}{((k+1)!)^{p_\\tau}}\\mathcal{G}_0 .\n\t\\]\n\tFor the ratio test on  $\\sum_{k=0}^\\infty r_k^2$, we consider \n\t\\[\n\t\\lim_{k \\rightarrow \\infty}\\frac{r_{k+1}^2}{r_k^2} =  \\lim_{k \\rightarrow \\infty} \\frac{\\tau_0^2}{(k+2)^{2p_\\tau}}  = 0.\n\t\\]\n\tTherefore, by the ratio test, $\\sum_{k=0}^\\infty r_k^2 < \\infty$, implying that \\( \\{\\zeta_k\\} \\) is square summable.\n\\end{proof}","start":61497,"end":68849},{"raw_tex":"\\begin{lemma}\\label{lem3}\n\tThe sequence $\\left\\{(x^k,y^k,\\tilde{y}^k)\\right\\}$ satisfies\n\t\\begin{equation*}\n\t\t\\sum\\limits_{k=0}^{{\\infty}}\\left\\|(x^{k+1},y^{k+1})-(x^k,\\tilde{y}^k)\\right\\|^2<\\infty, \\quad \\text{and} \\quad \\lim\\limits_{k\\to\\infty}\\left\\|(x^{k+1},y^{k+1})-(x^k,\\tilde{y}^k)\\right\\|=0.\n\t\\end{equation*}\n\\end{lemma}","start":69955,"end":70283},{"raw_tex":"\\begin{proof}\n\tSumming  \\cref{lem1_eq} from Lemma \\ref{lem1} over $k=0,1,\\dots,K-1$ yields,\n\t\\begin{equation}\\label{lem3_eq1}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_{K}}\\left(x^{K}, \\tilde{y}^{K}\\right)- \\tilde{\\psi}_{p_{0}}\\left(x^{0}, \\tilde{y}^{0}\\right) \\leq & -\\sum\\limits_{k=0}^{K-1} \\left(  \\frac{c_\\alpha}{4} \\left\\|x^{k+1}-x^k\\right\\|^2+ \\frac{c_\\beta}{4} \\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2\\right)\\\\& +\\sum\\limits_{k=0}^{K-1}\\left( \\frac{C_\\eta^2 }{c_\\alpha}(L_{f_x} + L_{g_2})^2+ \\frac{C_\\eta^2}{c_\\beta\\gamma^2} \\right) \\zeta_{k}^2+ \\sum\\limits_{k=0}^{K-1}C_{\\gamma} \\zeta_{k+1}^2.\\\\\n\t\t\\end{aligned}\n\t\\end{equation}\n\tSince \\( \\{\\zeta_k\\} \\) is square summable from Lemma \\ref{lem1}, it follows that\n\t\\[\n\t\\sum\\limits_{k=0}^{\\infty} \\left( \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2 + \\frac{1}{c_\\beta\\gamma^2} \\right) C_\\eta^2\\zeta_{k}^2 + \\sum\\limits_{k=0}^{\\infty}  C_{\\gamma} \\zeta_{k+1}^2 < \\infty.\n\t\\]\n\tAdditionally, because $\\tilde{\\psi}_{p_{K}}\\left(x^{K}, \\tilde{y}^{K}\\right) \\ge 0$, by taking $K \\rightarrow \\infty$ in \\eqref{lem3_eq1}, we have\n\t\\begin{equation}\\label{lem3_eq2}\n\t\t\\begin{aligned}\n\t\t\t&\\quad\\,\\sum\\limits_{k=0}^{\\infty} \\left(  \\frac{c_\\alpha}{4} \\left\\|x^{k+1}-x^k\\right\\|^2+ \\frac{c_\\beta}{4} \\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2\\right) \\\\\n\t\t\t&\\le \\tilde{\\psi}_{p_{0}}\\left(x^{0}, \\tilde{y}^{0}\\right) + \\sum\\limits_{k=0}^{\\infty} \\left( \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2 + \\frac{1}{c_\\beta\\gamma^2} \\right) C_\\eta^2\\zeta_{k}^2 + \\sum\\limits_{k=0}^{\\infty}  C_{\\gamma} \\zeta_{k+1}^2  < \\infty,\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhich proves the result.\n\\end{proof}","start":70284,"end":71887},{"raw_tex":"\\begin{lemma}\\label{lem4}\n\tLet $\\{ (x^{k_j},\\tilde{y}^{k_j}) \\} $ be any convergent subsequence of iterates $\\{(x^k,\\tilde{y}^k)\\}$ such that $(x^{k_j},\\tilde{y}^{k_j})$ satisfies condition \\eqref{inexacty_y} with $k = k_j - 1$. \n\tIf $(x^{k_j},\\tilde{y}^{k_j}) \\rightarrow (\\bar{x},\\bar{y}) $ as $j \\rightarrow \\infty$ for some $(\\bar{x},\\bar{y}) $, then $\\bar{y} \\in S(\\bar{x})$. \n\\end{lemma}","start":72078,"end":72471},{"raw_tex":"\\begin{proof}\n\tDefine the residual term $\\tilde{r}^k := \\tilde{y}^{k} - \\operatorname{Prox}_{ \\tilde{g}(x^{k}, \\cdot)} \\left( \\tilde{y}^{k} -  \\nabla_y f(x^{k}, \\tilde{y}^{k}) \\right) $. By the first-order optimality condition for the proximal operator, $\\tilde{r}^k$ satisfies\n\t\\begin{equation}\\label{lem4_eq1}\n\t\t\\tilde{r}^k \\in \\nabla_y f(x^{k}, \\tilde{y}^{k}) + \\partial_{y}\\tilde{g}(x^{k}, \\tilde{y}^{k} - \\tilde{r}^k).\n\t\\end{equation}\n\tThe sequence  $(x^{k_j},\\tilde{y}^{k_j})$ satisfies condition \\eqref{inexacty_y} with $k = k_j - 1$, which implies \n\t\\[\n\t\\left\\| \\tilde{r}^{k_j} \\right\\| \\le \\frac{c_{\\tilde{y}} }{p_{k_j}} \\left\\|(x^{k_j},y^{k_j})-(x^{k_j-1},\\tilde{y}^{k_j-1})\\right\\|.\n\t\\]\n\tFrom Lemma \\ref{lem3}, we have $\\lim_{k_j \\to\\infty} \\|(x^{k_j},y^{k_j})-(x^{k_j-1},\\tilde{y}^{k_j-1}) \\|=0$. As $p_{k_j} \\ge p_0$, the above inequality ensures that $\\lim_{k_j \\to\\infty} \\|\\tilde{r}^{k_j} \\|=0$.\n\t\n\tTaking the limit as  $k_j \\rightarrow \\infty$ in \\eqref{lem4_eq1} with $k = k_j$, by the continuity of $ \\nabla_y f$ and the outer semicontinuity of $\\partial_{y}\\tilde{g}$, it follows that\n\t\\[\n\t0  \\in \\nabla_y f(\\bar{x},\\bar{y}) + \\partial_{y}\\tilde{g}(\\bar{x},\\bar{y}).\n\t\\] \n\tSince the lower-level problem is convex in $y$, this condition\n\timplies $\\bar{y} \\in S(\\bar{x})$.\n\\end{proof}","start":72472,"end":73774},{"raw_tex":"\\begin{proposition}\\label{prop0}\n\tLet $\\epsilon > 0$. If the sequence $\\{p_k\\}$ is bounded, then any accumulation point $(\\bar{x}, \\bar{y})$ of the iterate sequence $\\{(x^k, {y}^k)\\}$ satisfies\n\t\\[\n\t\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon.\n\t\\]\n\\end{proposition}","start":73976,"end":74265},{"raw_tex":"\\begin{proof}\n\t\n\tLet $(\\bar{x}, \\bar{y})$ be an accumulation point of $\\{(x^k, {y}^k)\\}$, and let \\(\\{k_j\\}_{j \\in \\mathbb{N}} \\subseteq \\mathbb{N} \\) be an index subsequence such that $(x^{k_j+1},y^{k_j+1})\\to (\\bar{x},\\bar{y})$ as $j \\rightarrow \\infty$. By Lemma \\ref{lem3}, we have  $\\lim_{k_j \\to\\infty} \\|(x^{k_j+1},y^{k_j+1})-(x^{k_j},\\tilde{y}^{k_j}) \\|=0$, which implies that $(x^{k_j},\\tilde{y}^{k_j})$ also converges to $ (\\bar{x},\\bar{y})$ .\n\t\n\tSince the sequence $\\{p_k\\}$ is bounded and non-decreasing by construction in AGILS (\\cref{alg1}), there exists $k_0 >0$ such that $p_k = p_{k_0}$ for all $k \\ge k_0$. By passing to a subsequence if necessary, we can assume that $k_j - 1\\ge k_0$ for all $j \\in \\mathbb{N}$.\t\n\t\n\tAccording to the update rule of $p_k$ in AGILS (\\cref{alg1}), the condition $p_{k_j - 1} = p_{k_j}$ implies that one of the following two cases must occur for each \\(  k = k_j -1 \\):\n\t\n\tCase I: $\\left\\|(x^{k_j},y^{k_j})-(x^{k_j-1}, \\tilde{y}^{k_j-1})\\right\\| \\ge c_p \\min\\left\\{1/p_{k_0},{t^{k_j}}\\right\\}$ and $\\tilde{y}^{k_j} = y^{k_j}$.\n\t\n\tCase II: $\\left\\|(x^{k_j},y^{k_j})-(x^{k_j-1}, \\tilde{y}^{k_j-1})\\right\\| < c_p \\min\\left\\{1/p_{k_0},{t^{k_j}}\\right\\}$ and the feasibility correction procedure is applied to generate $\\tilde{y}^{k_j}$ satisfying condition \\eqref{inexacty_y}.\n\t\n\tAt least one of these two cases must hold for an infinite number of indices in the subsequence \t$\\{ k_j  - 1\\}$. We analyze them separately.\n\t\n\tCase 1:  Suppose Case I occurs for an infinite subsequence of indices. By passing to a further subsequence, we assume it holds for all $j \\in \\mathbb{N}$.\n\t\n\tThe condition $\\tilde{y}^{k_j} = y^{k_j}$ combined with $(x^{k_j},\\tilde{y}^{k_j})\\to (\\bar{x},\\bar{y})$ implies that $(x^{k_j}, {y}^{k_j})$\\\\\n\t$ \\to (\\bar{x},\\bar{y})$. \n\tFrom Lemma \\ref{lem3}, we know $\\lim_{k_j \\to\\infty} \\|(x^{k_j},y^{k_j})-(x^{k_j-1},\\tilde{y}^{k_j-1}) \\|=0$. For the condition of Case I to hold, we must therefore have  $t^{k_j} \\rightarrow 0$. This implies\n\t\\begin{equation*}\n\t\t\\limsup_{j \\rightarrow \\infty}\t\\left\\{ \\varphi(x^{k_j}, y^{k_j})-\\varphi(x^{k_j},\\theta^{k_j})-\\frac{1}{2\\gamma}\\left\\| \\theta^{k_j} - y^{k_j} \\right\\|^2-\\epsilon \\right\\} \\le 0.\n\t\\end{equation*}\n\tSince $\\zeta_{k} \\rightarrow 0$ and $\\mathcal{G}( \\theta^{k_j}, x^{k_j}, y^{k_j}) \\le \\zeta_{k_j}$, Lemma \\ref{error_bound} ensures that $\\| \\theta^{k_j} - \\theta^*_{\\gamma}( x^{k_j}, y^{k_j} )\\|\\rightarrow 0$.\n\tBy the continuity of $\\theta^*_{\\gamma}(x,y)$ (Proposition \\ref{partial_grad_v}), it follows that $\\theta^{k_j} \\rightarrow \\theta^*_{\\gamma}(\\bar{x}, \\bar{y}) $.\n\tThe continuity of $\\varphi$ then implies that $\\varphi(x^{k_j}, \\theta^{k_j})+\\frac{1}{2\\gamma}\\|\\theta^{k_j} - y^{k_j}\\|^2 \\rightarrow v_\\gamma(\\bar{x},\\bar{y})$. Then taking the limit in the inequality above as  $j \\rightarrow \\infty$ yields $\\varphi(\\bar{x},\\bar{y})-v_{\\gamma}(\\bar{x},\\bar{y})-\\epsilon \\le 0$. \n\t\n\tCase 2: \n\tNow, suppose Case II occurs for an infinite subsequence of indices. Again, we assume it holds for all $j \\in \\mathbb{N}$.\n\t\n\tIn this scenario, the feasibility correction procedure is applied at each iteration $k = k_j -1$ to generate $\\tilde{y}^{k_j}$ satisfying condition \\eqref{inexacty_y}. Since $(x^{k_j},\\tilde{y}^{k_j}) \\rightarrow (\\bar{x},\\bar{y}) $ and satisfies condition \\eqref{inexacty_y}, it follows from Lemma \\ref{lem4} that $\\bar{y} \\in S(\\bar{x})$. For any such point, $\\theta^*_{\\gamma}(\\bar{x}, \\bar{y}) = \\bar{y} $ (see, e.g., \\cite[Theorem 10.7]{beck2017first}), and therefore \\( \\varphi(\\bar{x}, \\bar{y}) - v_\\gamma(\\bar{x}, \\bar{y}) = 0 \\).  Since $ \\epsilon > 0$, the constraint $\t\t\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon$ is satisfied.\n\t\n\tSince any accumulation point must be the limit of a subsequence that falls into one of these two cases, we conclude that $\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon$.\n\\end{proof}","start":74266,"end":78191},{"raw_tex":"\\begin{proof}\n\tThe proof builds upon techniques used in the proof of Theorem 1 in \\cite{ye2023}. \n\tLet $(\\bar{x}, \\bar{y})$ be an accumulation point of the sequence $\\{(x^k,y^k)\\}$, and let $\\{(x^{k_j+1},y^{k_j+1})\\}$ be a convergent subsequence such that $(x^{k_j+1},y^{k_j+1}) \\rightarrow (\\bar{x}, \\bar{y})$ as $k_j \\rightarrow \\infty$. \n\tBy Lemma \\ref{lem3}, we have  $\\lim_{k_j \\to\\infty} \\|(x^{k_j+1},y^{k_j+1})-(x^{k_j},\\tilde{y}^{k_j}) \\|=0$, which implies that $(x^{k_j},\\tilde{y}^{k_j}) \\to (\\bar{x},\\bar{y})$ .\n\tSince $\\{p_k\\}$ is bounded, there exists $k_0 > 0$ such that $p_k = \\bar{p} := p_{k_0}$ for any $k \\ge k_0$. \n\t\n\t\n\tFrom the update rules \\eqref{update_x} and \\eqref{update_y}, and the expressions for $d_x^k$ and $d_y^k$, we have    \n\t\\begin{equation}\\label{th1_eq2}\n\t\t\\begin{aligned}\n\t\t\t\\xi_x^k & \\in \\nabla_x F(x^{k+1},y^{k+1})+p_k \\left(\\nabla_x \\varphi(x^{k+1},y^{k+1}) - \\nabla_x v_\\gamma(x^{k+1},y^{k+1})\\right) +\\mathcal{N}_{X}(x^{k+1}),\\\\\n\t\t\t\\xi_y^k & \\in \\nabla_y F(x^{k},y^{k+1})+p_k \\left(\\partial_y \\varphi(x^{k},y^{k+1}) - \\nabla_y v_\\gamma(x^{k},y^{k+1})\\right) +\\mathcal{N}_{Y}(y^{k+1}),\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere $\\xi_x^k \\in \\Re^n$ and $\\xi_y^k \\in \\Re^m$ are given by\n\t\\begin{equation}\\label{th1_eq4}\n\t\t\\begin{aligned}\n\t\t\t\\xi_x^k &:=  p_k \\left(\\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - d_x^k \\right)  - \\frac{p_k}{\\alpha_k}\\left(x^{k+1}-x^k\\right), \\\\\n\t\t\t\\xi_y^k &:=  p_k \\left( \\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, y^{k+1}\\right) - d_y^k \\right)-\\frac{p_k}{\\beta_k}\\left(y^{k+1}-\\tilde{y}^k\\right).\n\t\t\\end{aligned} \n\t\\end{equation} \n\tWe now show that $\\xi_x^{k_j}, \\xi_y^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$. First, we have \n\t\\begin{equation}\n\t\t\\begin{aligned}\n\t\t\t& \\| \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - d_x^k\\| \\\\ \\le \\, & \\|  \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k}, y^{k+1} \\right) \\| + \\| \\nabla_x \\tilde{\\psi}_{p_{k}} \\left(x^{k}, y^{k+1} \\right) - d_x^k \\| \\\\\n\t\t\t\\le \\, & \\|  \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k}, y^{k+1} \\right) \\| + (L_{f_x} + L_{g_2})C_\\eta \\zeta_{k}, \n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere the last inequality follows from \\eqref{lem1_eq10}. Since $p_k = \\bar{p}$ for all $k \\ge k_0$, and assuming that $\\nabla_{x} F, \\nabla_{x} f$ and $\\nabla_{x} g$ are continuous on $X \\times Y$, and $\\nabla_{x} v_\\gamma$ is continuous on $X \\times Y$ (Proposition \\ref{partial_grad_v}), we conclude that $ \\nabla_x \\tilde{\\psi}_{p_{k}} =  \\nabla_x \\tilde{\\psi}_{\\bar{p}}$ is continuous on $X \\times Y$. \n\tAs $\\lim_{k_j\\to\\infty} (x^{k_j+1},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},\\tilde{y}^{k_j}) = (\\bar{x}, \\bar{y})$ and $\\zeta_{k} \\rightarrow 0$, it follows that $\\nabla_x \\tilde{\\psi}_{p_{k_j}} (x^{k_j+1}, y^{k_j+1} ) - d_x^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$. Combining this with $ \\|x^{k_j+1}-x^{k_j} \\| \\to 0$ and $\\alpha_k \\ge \\underline{\\alpha} > 0$, we obtain from \\eqref{th1_eq4} that $\\xi_x^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$.\n\t\n\tA similar argument can be applied to $\\xi_y^k$. From \\eqref{lem1_eq11}, we have \n\t\\begin{equation}\\label{th1_eq5}\n\t\t\\begin{aligned}\n\t\t\t& \\|\\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, y^{k+1}\\right) - d_y^k \\| \\\\ \\le \\, & \\| \\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, y^{k+1} \\right) - \\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, \\tilde{y}^k \\right) \\| + {C_\\eta}\\zeta_{k}/\\gamma.\n\t\t\\end{aligned}\n\t\\end{equation}\n\tSince $p_k = \\bar{p}$ for all $k \\ge k_0$, and under the assumption that $\\nabla_{y} F$ and $\\nabla_{y} f$ are continuous on $X \\times Y$, and $\\nabla_{y} v_\\gamma$ is continuous on $X \\times Y$ (Proposition \\ref{partial_grad_v}), we have that $ \\nabla_y (\\tilde{\\psi}_{p_{k}} - g) = \\nabla_y (\\tilde{\\psi}_{\\bar{p}} - g)$ is continuous on $X \\times Y$. As $\\lim_{k_j\\to\\infty} (x^{k_j},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},\\tilde{y}^{k_j}) = (\\bar{x}, \\bar{y})$ and $\\zeta_{k} \\rightarrow 0$, it follows that $\\nabla_y (\\tilde{\\psi}_{p_{k_j}} - g)(x^{k_j}, y^{k_j+1} ) - d_y^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$. Combining this with $ \\|y^{k_j+1}-\\tilde{y}^{k_j} \\| \\to 0$ and $\\beta_k \\ge \\underline{\\beta} > 0$, we conclude from \\eqref{th1_eq5} that $\\xi_y^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$.\n\t\n\tNow, taking the limit as $k = k_j \\rightarrow \\infty$ in \\eqref{th1_eq2}, since $\\xi_x^{k_j}, \\xi_y^{k_j} \\rightarrow 0$, $p_k = \\bar{p}$ for all $k \\ge k_0$, $\\lim_{k_j\\to\\infty} (x^{k_j+1},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},\\tilde{y}^{k_j}) = (\\bar{x}, \\bar{y})$, $\\nabla F$, $\\nabla v_\\gamma$, and $\\nabla_x \\varphi$ are continuous on $X \\times Y$, $\\partial_y \\varphi$ and $\\mathcal{N}_{X\\times Y}$ are outer semicontinuous on $X \\times Y$, and $\\partial \\varphi = \\{\\nabla_x \\varphi\\} \\times \\partial_y \\varphi$ on $X \\times Y$, we obtain \n\t\\begin{equation}\\label{th1_eq6}\n\t\t0\\in\\nabla F(\\bar{x}, \\bar{y})+\\bar{p}\\left(\\partial \\varphi(\\bar{x},\\bar{y})-\\nabla v_\\gamma(\\bar{x}, \\bar{y})\\right)+\\mathcal{N}_{X\\times Y}(\\bar{x}, \\bar{y}).\n\t\\end{equation}\n\t\n\tFinally, from Proposition \\ref{prop0}, we know that the limit point $(\\bar{x}, \\bar{y})$ satisfies $\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon $. Let $\\bar{\\epsilon}:= \\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\in [0, \\epsilon]$. Combining this with \\eqref{th1_eq6}, we have\n\t\\begin{equation}\n\t\t\\left\\{\n\t\t\\begin{aligned}\n\t\t\t&0\\in\\nabla F(\\bar{x}, \\bar{y}) + \\bar{p}\\left( \\partial \\varphi(\\bar{x}, \\bar{y}) - \\nabla v_\\gamma(\\bar{x}, \\bar{y})\\right)+\\mathcal{N}_{X\\times Y}(\\bar{x}, \\bar{y}),\\\\\n\t\t\t&\\varphi(\\bar{x}, \\bar{y}) - v_{\\gamma}(\\bar{x}, \\bar{y})- \\bar{\\epsilon} \\le 0, \\qquad  \\bar{p}\\left( \\varphi(\\bar{x}, \\bar{y}) - v_{\\gamma}(\\bar{x}, \\bar{y})- \\bar{\\epsilon} \\right)=0,\n\t\t\\end{aligned}\\right.\n\t\\end{equation}\n\twhich implies that $(\\bar{x},\\bar{y})$ is a KKT point of problem $({\\rm VP})_\\gamma^{\\bar{\\epsilon}}$ with $\\bar{\\epsilon}\\le \\epsilon$.\n\\end{proof}","start":92446,"end":98589}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["LL","eq1"],"declared_external":false,"source_only":false,"title":"双层优化与下层解集","source_units":[{"raw_tex":"Bilevel optimization problems are a class of hierarchical optimization problems where the feasible region is implicitly defined by the solution set of a lower-level optimization problem. In this work, we focus on a bilevel optimization problem which has a convex composite lower-level problem,\n\\begin{equation}\\label{eq1}\n\t\\begin{aligned}\n\t\t\\min\\limits_{x \\in X, y \\in Y} &\\ F(x,y)\\\\\n\t\t{\\rm s.t.}\\quad &\\ y \\in S(x),\n\t\\end{aligned}\n\\end{equation}\nwhere $S(x)$ denotes the set of optimal solutions to the lower-level problem:\n\\begin{equation}\\label{LL}\n\t\\min\\limits_{y \\in Y}\\ \\varphi(x,y) := f(x,y) + g(x,y).\n\\end{equation}\n","start":3423,"end":4047}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"闭凸域与复合结构","source_units":[{"raw_tex":"Here, $X \\subset \\Re^n$ and $Y \\subset \\Re^m$ are closed convex sets, $f(x, y) : \\Re^n \\times \\Re^m \\rightarrow \\Re$ is smooth and convex with respect to $y$ on $Y$ for any $x \\in X$, and $g(x, y) : \\Re^n \\times \\Re^m \\rightarrow \\Re$ is convex but potentially nonsmooth with respect to $y$ on $Y$ for any $x \\in X$. Additionally, we assume that $g(x, \\cdot) : \\Re^m \\rightarrow \\Re$ is proximal-friendly, enabling efficient evaluation of its proximal operator.\n","start":4047,"end":4509}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","labels":["reformualtion_Moreau"],"declared_external":false,"source_only":false,"title":"未松弛包络重构","source_units":[{"raw_tex":"To overcome this challenge, in this paper, we consider the following Moreau envelope-based reformulation of the bilevel optimization problem \\eqref{eq1}, first introduced in \\cite{gao2024},\n\\begin{equation}\\label{reformualtion_Moreau}\n\t({\\rm VP})_\\gamma \\quad\\quad\n\t\\min\\limits_{x\\in X,y\\in Y}\\  F(x,y) \\quad\t\t{\\rm s.t.}\\quad\\ \\varphi (x,y)-v_{\\gamma} (x,y)\\le 0,\n\\end{equation}\n","start":11073,"end":11452}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["def_vg"],"declared_external":false,"source_only":false,"title":"下层Moreau包络","source_units":[{"raw_tex":"where $v_{\\gamma}(x, y)$ is the Moreau envelope associated with the lower-level problem,\n\\begin{equation}\\label{def_vg}\n\tv_{\\gamma}(x,y):=\\inf\\limits_{\\theta\\in Y}\\varphi(x,\\theta)+\\frac{1}{2\\gamma}\\|\\theta-y\\|^2,\n\\end{equation}\n","start":11452,"end":11681}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":["reformualtion_Moreau_relax"],"declared_external":false,"source_only":false,"title":"松弛包络约束问题","source_units":[{"raw_tex":"where $\\gamma > 0$. When the lower-level problem is convex with respect to $y$,  this reformulation is equivalent to the original bilevel optimization problem \\eqref{eq1} (see \\cite[Theorem 1]{gao2024}).  Directional optimality conditions for $({\\rm VP})_\\gamma$ are discussed in \\cite{bai2025optimality}. However, as shown in \\cite{gao2024} and \\cite[Proposition 7]{ye2023}, classical constraint qualification (CQ) conditions do not hold at any feasible point of $({\\rm VP})_\\gamma$, motivating us to study the following relaxed approximation problem,\n\\begin{equation}\\label{reformualtion_Moreau_relax}\n\t({\\rm VP})_\\gamma^{\\epsilon} \\quad\\quad\n\t\\min\\limits_{x\\in X,y\\in Y}\\  F(x,y) \\quad\t\t{\\rm s.t.}\\quad\\ \\varphi (x,y)-v_{\\gamma} (x,y)\\le \\epsilon,\n\\end{equation}\n","start":11681,"end":12447}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"欧氏空间记号","source_units":[{"raw_tex":"Let $\\mathbb{R}^n$ denote the $n$-dimensional Euclidean space, and let $\\mathbb{B}$ denote the closed unit ball centered at the origin. The nonnegative and positive orthants in $\\mathbb{R}^n$ are denoted by $\\mathbb{R}^n_+$ and $\\mathbb{R}^n_{++}$, respectively. The standard inner product and Euclidean norm are denoted by $\\langle \\cdot, \\cdot \\rangle$ and $\\|\\cdot\\|$, respectively.\n","start":16383,"end":16769}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"集合与投影算子","source_units":[{"raw_tex":"For a vector \\( x \\in \\mathbb{R}^n \\) and a closed convex set \\( S \\subset \\mathbb{R}^n \\), the distance from \\( x \\) to \\( S \\) is defined as \\(\\mathrm{dist}(x, S) = \\min_{y \\in S} \\|x - y\\|\\). The indicator function \\(\\delta_S: \\mathbb{R}^n \\to \\mathbb{R}\\cup\\{+\\infty\\}\\) is defined by \\(\\delta_S(x) = 0\\) if \\(x\\in S\\) and \\(\\delta_S(x) = +\\infty\\) otherwise. Moreover, \\(\\mathcal{N}_S(x)\\) denotes the normal cone to \\(S\\) at \\(x\\). The Euclidean projection operator onto \\( S \\) is denoted by \\(\\mathrm{Proj}_S\\). The Cartesian product of two sets \\( X \\) and \\( Y \\) is denoted by \\( X \\times Y \\).\n","start":16769,"end":17375}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"近端映射与次微分","source_units":[{"raw_tex":"Let \\( h: \\mathbb{R}^n \\to \\mathbb{R} \\cup \\{\\infty\\} \\) be a proper closed function. The proximal mapping of \\( h\\) is defined as $\\operatorname{Prox}_h(y):=\\arg\\min_{\\theta \\in \\Re^m}\\left\\{h(\\theta)+\\|\\theta-y\\|^2 / 2\\right\\}$. The Fr\\'{e}chet (regular) subdifferential \\(\\hat{\\partial} h\\) and the Mordukhovich (limiting) subdifferential \\(\\partial h\\) at a point \\( \\bar{z} \\in \\mathrm{dom}\\, h := \\{z \\mid h(z) < \\infty\\} \\) are defined as:\n\\[\n\\begin{aligned}\n\t\\hat{\\partial} h(\\bar{z}) &= \\left\\{ v \\in \\mathbb{R}^n \\;\\middle|\\; h(z) \\geq h(\\bar{z}) + \\langle v, z - \\bar{z} \\rangle + o(\\|z - \\bar{z}\\|), \\; \\forall z \\in \\mathbb{R}^n \\right\\}, \\\\\n\t\\partial h(\\bar{z}) &= \\left\\{ v \\in \\mathbb{R}^n \\;\\middle|\\; \\exists z_k \\xrightarrow{h} \\bar{z}, \\; \\exists v_k \\in \\hat{\\partial} h(z_k), \\; v_k \\to v \\right\\},\n\\end{aligned}\n\\]\nwhere \\( o(z) \\) denotes a function satisfying \\( o(z)/z\\to 0 \\) as \\( z \\downarrow 0 \\), and \\( z_k \\xrightarrow{h} \\bar{z} \\) indicates \\( z_k \\to \\bar{z} \\) and \\( h(z_k) \\to h(\\bar{z}) \\) as \\( k \\to \\infty \\). If \\( \\bar{z} \\notin \\mathrm{dom}\\, h \\), we define \\(\\partial h(\\bar{z}) = \\emptyset\\).\n","start":17375,"end":18517}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","labels":["asup1"],"declared_external":false,"source_only":false,"title":"上层目标的正则性","source_units":[{"raw_tex":"This part outlines the assumptions on the problem data for the bilevel optimization problem studied in this work. First, we assume that the upper-level objective $F$ is smooth and bounded below.\n\n\\begin{assumption}\\label{asup1}\n\tThe function \\(F : \\mathbb{R}^n \\times \\mathbb{R}^m \\to \\mathbb{R}\\) is bounded below on \\(X \\times Y\\), i.e., \\(\\underline{F} := \\inf_{(x, y) \\in X \\times Y} F(x, y) > -\\infty\\). \\(F\\) is continuously differentiable, and its gradients \\(\\nabla_x F\\) and \\(\\nabla_y F\\) are \\(L_{F_x}\\)- and \\(L_{F_y}\\)-Lipschitz continuous on \\(X \\times Y\\), respectively.\n\\end{assumption}\n\n","start":18552,"end":19156}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","labels":["asup2"],"declared_external":false,"source_only":false,"title":"下层变量凸性","source_units":[{"raw_tex":"Throughout this paper, we focus on cases where the lower-level problem is a convex optimization problem.\n\n\\begin{assumption}\\label{asup2}\n\tFor any \\(x \\in X\\), the functions \\(f(x, \\cdot)\\) and \\(g(x, \\cdot)\\) are convex and defined on the closed convex set \\(Y\\). \n\\end{assumption} \n","start":19156,"end":19440}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"分块弱凸性","source_units":[{"raw_tex":"\nSmoothness and weak convexity conditions are required for the lower-level objective functions. We recall that a function \\(h : \\mathbb{R}^n \\times \\mathbb{R}^m \\to \\mathbb{R}\\) is \\((\\rho_1, \\rho_2)\\)-weakly convex on a convex set \\(X \\times Y\\) if \\(h(x, y) + \\frac{\\rho_1}{2} \\|x\\|^2 + \\frac{\\rho_2}{2} \\|y\\|^2\\) is convex on \\(X \\times Y\\). This notion generalizes convexity (\\(\\rho_1 = \\rho_2 = 0\\)). Weakly convex functions \\cite{nurminskii1973quasigradient}, extends the class of convex functions to include many nonconvex functions. For weakly convex functions, the Fréchet (regular) subdifferential coincides with both the limiting subdifferential and the Clarke subdifferential (see, e.g., \\cite[Proposition 3.1 and Theorem 3.6]{ngai2000approximate}).\n\n","start":19440,"end":20203}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","labels":["asup3"],"declared_external":false,"source_only":false,"title":"下层正则性与弱凸性","source_units":[{"raw_tex":"\\begin{assumption}\\label{asup3}\n\tThe lower-level objective functions \\(f(x, y)\\) and \\(g(x, y)\\) satisfy:\n\t\\begin{itemize}\n\t\t\\item[(1)] \\(f(x, y)\\) is continuously differentiable on \\(X \\times Y\\), and its gradients \\(\\nabla_x f\\) and \\(\\nabla_y f\\) are \\(L_{f_x}\\)- and \\(L_{f_y}\\)-Lipschitz continuous on \\(X \\times Y\\), respectively;\n\t\t\\item[(2)] \\(g(x, y)\\) is continuous on \\(X \\times Y\\), \\(\\nabla_x g(x, y)\\) exists, and \\(\\nabla_x g(\\cdot, y)\\) is \\(L_{g_1}\\)-Lipschitz continuous on \\(X\\) for any \\(y \\in Y\\), while \\(\\nabla_x g(x, \\cdot)\\) is \\(L_{g_2}\\)-Lipschitz continuous on \\(Y\\) for any \\(x \\in X\\);\n\t\t\\item[(3)] \\(f(x, y)\\) is \\((\\rho_{f_1}, \\rho_{f_2})\\)-weakly convex on \\(X \\times Y\\), and \\(g(x, y)\\) is \\((\\rho_{g_1}, \\rho_{g_2})\\)-weakly convex on \\(X \\times Y\\).\n\t\\end{itemize}\n\\end{assumption}\n","start":20203,"end":21022}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["proximal_LL"],"declared_external":false,"source_only":false,"title":"精确近端下层解","source_units":[{"raw_tex":"Under Assumption \\ref{asup2}, the solution mapping\n\\begin{equation}\\label{proximal_LL}\n\tS_{\\gamma}(x, y) := \\mathrm{argmin}_{\\theta \\in Y} \\left\\{ f(x, \\theta) + g(x, \\theta) + \\frac{1}{2\\gamma}\\|\\theta - y\\|^2 \\right\\}\n\\end{equation}\nis well-defined and single-valued  on $X \\times \\mathbb{R}^m$. We denote its unique solution by $\\theta_\\gamma^*(x, y)$.\nBelow, we summarize sensitivity results for $v_\\gamma(x, y)$ from Theorems 2, 3, and 5 of \\cite{gao2024}.\n","start":24483,"end":24945}]},{"id":"S014","semantic_type":"derived_fact","kind":"derived_fact","labels":["grad_v","valuefinc"],"declared_external":false,"source_only":false,"title":"包络梯度表达式","source_units":[{"raw_tex":"\t\\begin{equation}\\label{valuefinc}\n\t\t\\nabla v_\\gamma(x, y) = \\Big( \\nabla_x f(x, \\theta_\\gamma^*(x, y)) + \\nabla_x g(x, \\theta_\\gamma^*(x, y)), \\left({y - \\theta_\\gamma^*(x, y)}\\right)/{\\gamma} \\Big).\n\t\\end{equation}\n","start":25358,"end":25575}]},{"id":"S015","semantic_type":"proposition","kind":"proposition","labels":["partial_grad_v"],"declared_external":false,"source_only":false,"title":"包络偏弱凸性与连续性","source_units":[{"raw_tex":"\\begin{proposition}\\label{partial_grad_v}\n\tSuppose Assumptions \\ref{asup2} and \\ref{asup3} hold. For $\\gamma \\in (0, 1/(\\rho_{f_2} + \\rho_{g_2}))$, the function $v_\\gamma(x, y)$ is $\\rho_{v_1}$-weakly convex with respect to $x$ on $X$ for any $y \\in Y$ with $\\rho_{v_1} \\geq \\rho_{f_1} + \\rho_{g_1}$. It is also convex with respect to $y$ on $\\mathbb{R}^m$ for any $x \\in X$. Moreover, $v_\\gamma(x, y)$ is differentiable with gradient given by \\eqref{valuefinc} on $X \\times \\mathbb{R}^m$, and $\\theta_\\gamma^*(x, y)$ and $\\nabla v_\\gamma(x, y)$ are continuous on $X \\times \\mathbb{R}^m$.\n\\end{proposition}","start":26030,"end":26636}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"松弛问题的KKT点","source_units":[{"raw_tex":"\\begin{definition}\n\tLet $(\\bar{x},\\bar{y})$ be a feasible solution of problem $({\\rm VP})_{\\gamma}^{\\epsilon}$ with $\\epsilon > 0$. We say that $(\\bar{x},\\bar{y})$ is a stationary/KKT point of problem $({\\rm VP})_{\\gamma}^{\\epsilon}$ if there exists a multiplier $\\lambda \\ge 0$ such that\n\t\\begin{equation*}\n\t\t\\begin{cases}\n\t\t\t0\\in \\nabla F(\\bar{x},\\bar{y}) + \\lambda \\left(\\nabla f(\\bar{x},\\bar{y}) + \\partial g(\\bar{x},\\bar{y}) - \\nabla v_{\\gamma}(\\bar{x},\\bar{y})\\right) + \\mathcal{N}_{X\\times Y}(\\bar{x},\\bar{y}), \\\\\n\t\t\tf(\\bar{x},\\bar{y}) + g(\\bar{x},\\bar{y}) - v_{\\gamma}(\\bar{x},\\bar{y}) \\leq \\epsilon, \\qquad \\lambda \\left(f(\\bar{x},\\bar{y}) + g(\\bar{x},\\bar{y}) - v_{\\gamma}(\\bar{x},\\bar{y})-\\epsilon\\right) = 0.\n\t\t\\end{cases}\n\t\\end{equation*}\n\\end{definition}\n","start":34074,"end":34843}]},{"id":"S017","semantic_type":"definition","kind":"definition","labels":["penprob"],"declared_external":false,"source_only":false,"title":"缩放罚目标","source_units":[{"raw_tex":"In this section, we propose a new gradient-type algorithm, Alternating Gradient-type algorithm with Inexact Lower-level Solutions (AGILS), to solve ${\\rm (VP)}_\\gamma^\\epsilon$. Given that $f(x, y) + g(x,y) - v_{\\gamma}(x, y)$ is always nonnegative, a natural approach is to consider the following penalized problem,\n\\begin{equation}\\label{penprob}\n\t\\min_{x \\in X, y \\in Y}\\psi_{p_k}\t(x, y) := \\frac{1}{p_k}F(x, y) + f(x, y) + g(x,y) - v_{\\gamma}(x, y),\n\\end{equation}\nwhere $p_k$ is the penalty parameter at iteration $k$. Due to the nonsmoothness of $g(x,y)$ with respect to $y$ and the availability of its proximal operator, it is natural to adopt a proximal alternating linearization method \\cite{bolte2014proximal} to construct the algorithm. Given the current iterate $(x^k, y^k)$, linearizing the smooth components of $\\psi_{p_k}\t(x, y)$ requires evaluating gradients, including $\\nabla v_{\\gamma}(x, y)$ as provided in \\cref{grad_v}. Calculating the exact gradient  $\\nabla v_{\\gamma}(x^k, y^k)$ requires solving the proximal lower-level problem in \\eqref{proximal_LL} to obtain $\\theta_\\gamma^*(x^k, y^k)$. However, this solution typically does not have a closed-form expression, and solving it requires an iterative optimization solver, which can be computationally expensive. To address this, we propose an inexact gradient approximation by substituting $\\theta_{\\gamma}^*$ with an inexact estimate $\\theta^k$.\n\nThe proposed algorithm alternates between updating $y$ and $x$ while refining the inexact approximation of $\\theta_\\gamma^*(x^k, y^k)$. At each iteration $k=0,1,\\ldots$, given the current iterate $(x^k, y^k)$, along with the feasibility-corrected iterate $\\tilde{y}^k$ and corresponding $\\tilde{\\theta}^k$ (constructed from $y^k$ and  $\\theta^{k}$ via the procedure detailed below), the next iterate $(x^{k+1}, y^{k+1})$ and the refined approximation $\\theta^{k+1}$ are updated as follows.\n","start":34884,"end":36797}]},{"id":"S018","semantic_type":"algorithm","kind":"algorithm","labels":["dy","update_y","update_y2"],"declared_external":false,"source_only":false,"title":"校正起点的y更新","source_units":[{"raw_tex":"\n{\\bf Update $y$ (Fixing $x$).}\nWe compute the update direction for $y$,\n\\begin{equation}\\label{dy}\n\td_y^k:= \\frac{1}{p_k} \\nabla_y F\\left(x^{k}, \\tilde{y}^k\\right)+\\nabla_y f\\left(x^{k}, \\tilde{y}^k\\right)-\\frac{\\tilde{y}^k-\\tilde{\\theta}^{k}}{\\gamma},\n\\end{equation}\nwhere $(\\tilde{y}^k-\\tilde{\\theta}^{k})/{\\gamma}$ is an inexact approximation of $\\nabla_y v_{\\gamma}(x^{k}, \\tilde{y}^k) =( {\\tilde{y}^k} -  \\theta_{\\gamma}^*(x^{k}, \\tilde{y}^k))/ \\gamma $ (see (\\ref{valuefinc})). Using $\td_y^k$, the variable $y$ is updated as\n\\begin{equation}\\label{update_y}\n\ty^{k+1} = \\underset{y \\in Y}{\\mathrm{argmin}} ~ \\left\\langle d_y^k, y - \\tilde{y}^{k} \\right\\rangle + g(x^k, y) + \\frac{1}{2\\beta_k} \\left\\| y - \\tilde{y}^k \\right\\|^2, \n\\end{equation}\nwhere $\\beta_k>0$ is the stepsize. \nThis can also be expressed using the proximal operator\n\\begin{equation}\\label{update_y2}\n\ty^{k+1} = \\operatorname{Prox}_{\\beta_k\\tilde{g}(x^{k},\\cdot)}\\left(\\tilde{y}^k-\\beta_k d^k_y\\right),\n\\end{equation}\nwhere $\\tilde{g}(x,y) := g(x,y) + \\delta_Y(y)$. \n","start":36797,"end":37839}]},{"id":"S019","semantic_type":"algorithm","kind":"algorithm","labels":["inexacty1","theta_y"],"declared_external":false,"source_only":false,"title":"半步近端解误差准则","source_units":[{"raw_tex":"\n{\\bf Find inexact approximation $\\theta^{k+1/2}$.}\nAfter updating $y^{k+1}$, we compute an inexact approximation $\\theta^{k+1/2}$ for $\\theta_{\\gamma}^*(x^k, y^{k+1})$ by solving\n\\begin{equation}\\label{theta_y}\n\t\\theta^{k+1/2} \\approx \t\\underset{\\theta \\in Y}{\\mathrm{argmin}} \\left\\{f(x^{k},\\theta) + g(x^{k}, \\theta) + \\frac{1}{2\\gamma}\\left\\|\\theta-y^{k+1}\\right\\|^2\\right\\}, \n\\end{equation}\nsubject to either an absolute or a relative inexactness criterion\n\\begin{equation}\\label{inexacty1}\n\t\\begin{aligned}\n\t\t\\mathcal{G}(\\theta^{k+1/2}, x^k,y^{k+1}) \\le s_k, \\quad \\text{or} \\quad \\mathcal{G}(\\theta^{k+1/2}, x^k,y^{k+1}) \\le \\tau_k \\mathcal{G}(\\theta^{k-1}, x^{k-1},y^{k-1}),\n\t\\end{aligned}\n\\end{equation}\nwhich can be compactly expressed as\n\\[\n\\mathcal{G}(\\theta^{k+1/2}, x^{k}, y^{k+1})\n\\le\n\\max\\left\\{ s_k,\\ \\tau_k\\,\\mathcal{G}(\\theta^{k-1}, x^{k-1}, y^{k-1}) \\right\\}.\n\\]\n","start":37839,"end":38722}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"近端梯度残差与误差序列","source_units":[{"raw_tex":"The parameters $s_k$ and $\\tau_k$ control the solution's inexactness, and are required to satisfy $\\sum_{k=0}^{\\infty}s_k^2<\\infty$ and $\\tau_k:=\\tau_0/(k+1)^{p_\\tau}$ with constants $\\tau_0 > 0$ and $p_\\tau >0$. The prox-gradient residual $\\mathcal{G}(\\theta, x,y)$ quantifies the inexactness and is defined as\n\\[\n\\mathcal{G}(\\theta, x,y):= \t\\left\\| \\theta -\\operatorname{Prox}_{\\eta \\tilde{g}\\left(x, \\cdot\\right)}\\left(\\theta-\\eta\\left(\\nabla_y f\\left(x, \\theta\\right) + (\\theta -y)/ {\\gamma}\\right)\\right)\\right\\|,\n\\]\nwhere $\\eta > 0$. Note that $\\mathcal{G}(\\theta, x,y) = 0$ if and only if $\\theta =\\theta_{\\gamma}^*(x,y)$. \n","start":38722,"end":39353}]},{"id":"S021","semantic_type":"algorithm","kind":"algorithm","labels":["dx","update_x"],"declared_external":false,"source_only":false,"title":"x方向与投影更新","source_units":[{"raw_tex":"{\\bf Update $x$ (Fixing $y$).}\nWe compute the update direction for $x$,\n\\begin{equation}\\label{dx}\n\td_x^k:= \\frac{1}{p_k} \\nabla_xF\\left(x^k, y^{k+1}\\right)+\\nabla_x \\varphi \\left(x^k, y^{k+1}\\right)-\\nabla_x f\\left(x^k, \\theta^{k+1/2}\\right)-\\nabla_x g\\left(x^k, \\theta^{k+1/2}\\right),\n\\end{equation}\nwhere $\\nabla_x f\\left(x^k, \\theta^{k+1/2}\\right)+\\nabla_x g\\left(x^k, \\theta^{k+1/2}\\right)$ is an approximation to \\( \\nabla_x v_{\\gamma}(x^k, y^{k+1}) = \\nabla_x f(x^k, \\theta^*_{\\gamma}(x^k, y^{k+1}))+\\nabla_x g(x^k, \\theta^*_{\\gamma}(x^k, y^{k+1}))\\) (see \\eqref{valuefinc}).\nUsing the direction $d_x^k$, we update $x$ with stepsize $\\alpha_k>0$ as\n\\begin{equation}\\label{update_x}\n\tx^{k+1}=\\operatorname{Proj}_X\\left(x^k-\\alpha_k d_x^k\\right).\n\\end{equation}\n","start":39807,"end":40574}]},{"id":"S022","semantic_type":"algorithm","kind":"algorithm","labels":["inexacty2"],"declared_external":false,"source_only":false,"title":"整步近端解误差准则","source_units":[{"raw_tex":"\n{\\bf Find inexact approximation $\\theta^{k+1}$.}\nAfter updating $x^{k+1}$, a new inexact approximation $\\theta^{k+1}$ of $\\theta^*_{\\gamma}(x^{k+1}, y^{k+1})$  is computed by approximately solving\n\\[\n\\theta^{k+1} \\approx \t\\underset{\\theta \\in Y}{\\mathrm{argmin}} \\left\\{f(x^{k+1},\\theta) + g(x^{k+1}, \\theta) + \\frac{1}{2\\gamma}\\left\\|\\theta-y^{k+1}\\right\\|^2\\right\\}, \n\\]\nsuch that $\\theta^{k+1}$ satisfies either an absolute or a relative inexactness criterion\n\\begin{equation}\\label{inexacty2}\n\t\\begin{aligned}\n\t\t\\mathcal{G}(\\theta^{k+1}, x^{k+1},y^{k+1}) \\le s_{k+1}, \\quad \\text{or} \\quad  \\mathcal{G}(\\theta^{k+1}, x^{k+1},y^{k+1}) \\le \\tau_{k+1}\\mathcal{G}(\\theta^{k}, x^{k},y^{k}).\n\t\\end{aligned}\n\\end{equation}\n","start":40574,"end":41295}]},{"id":"S023","semantic_type":"algorithm","kind":"algorithm","labels":["def_t","update_p_1"],"declared_external":false,"source_only":false,"title":"违约估计与增罚判据","source_units":[{"raw_tex":"{\\bf Penalty parameter update and feasibility correction.}\nTo encourage feasibility of the constraint $\\varphi(x, y) - v_{\\gamma}(x, y) \\le \\epsilon$, the penalty parameter $p_k$ is updated at each iteration.\nEvaluating the constraint violation requires computing $ v_{\\gamma}(x, y)$, which involves solving the proximal lower-level problem in \\eqref{proximal_LL}. Instead, we use $\\theta^{k+1}$ to estimate the constraint violation as\n\\begin{equation}\\label{def_t}\n\tt^{k+1}=\\max\\left\\{\\varphi(x^{k+1},y^{k+1})-\\varphi(x^{k+1},\\theta^{k+1})-\\frac{1}{2\\gamma}\\| \\theta^{k+1} - y^{k+1}\\|^2-\\epsilon,0\\right\\}.\n\\end{equation}\nThe penalty parameter \\(p_k\\) is increased to $p_{k+1}=\tp_k + \\varrho_p$ (for some $\\varrho_p > 0$) only when the change in the iterates, $\\Delta_{k+1} := \\left\\| (x^{k+1}, y^{k+1}) - (x^k, \\tilde{y}^k) \\right\\|$, is small relative to the penalty parameter (i.e., $\\Delta_{k+1} < c_p/p_k$ for some $c_p > 0$), while the estimated constraint violation $t^{k+1}$ is large relative to this change (i.e., $t^{k+1} > \\Delta_{k+1}/c_p$).\nThese two conditions can be compactly written as \\begin{equation}\\label{update_p_1}\n\t\\Delta_{k+1} < c_p \\min\\left\\{1/p_k,{t^{k+1}}\\right\\}.\n\\end{equation}\n","start":41581,"end":42791}]},{"id":"S024","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"可行性校正的设计动机","source_units":[{"raw_tex":"\nHowever,  simply increasing $p_{k}$ does not always guarantee feasibility. It can sometimes cause the algorithm to stagnate near an infeasible stationary point of the constraint function $\\varphi(x, y) - v_{\\gamma}(x, y)$. \nTo mitigate this, we use the bilevel optimization structure, that the feasible region is the solution set of the convex lower-level problem \\eqref{LL}. We introduce a {\\it feasibility correction procedure} designed to help the iterates escape such stationary points.\n\n","start":42791,"end":43284}]},{"id":"S025","semantic_type":"algorithm","kind":"algorithm","labels":["theta_gap_condition"],"declared_external":false,"source_only":false,"title":"校正触发阈值","source_units":[{"raw_tex":"The feasibility correction procedure aims to generate a corrected iterate $\\tilde{y}^{k+1} $ from $y^{k+1}$. It is based on the observation that if $(x^{k+1},y^{k+1})$ is feasible, then $y^{k+1}$ should be a reasonably accurate solution to the lower-level problem \\eqref{LL}  for $x= x^{k+1}$, implying that $\\| \\nabla_y v_{\\gamma}(x^{k+1},y^{k+1})\\| $ should be small. We use the magnitude of $( y^{k+1} - \\theta^{k+1})/\\gamma$ to approximate $\\| \\nabla_y v_{\\gamma}(x^{k+1},y^{k+1})\\| $. The feasibility correction  is triggered if:\n\\begin{equation}\\label{theta_gap_condition}\n\t\\left\\| y^{k+1} - \\theta^{k+1} \\right\\| \\ge \\frac{c_y \\gamma}{p_k},\n\\end{equation}\nwhere $c_y > 0$. Condition \\eqref{theta_gap_condition} indicates that the iterate $(x^{k+1},y^{k+1})$ might be approaching an undesirable stationary point of the constraint function.\n","start":43284,"end":44130}]},{"id":"S026","semantic_type":"algorithm","kind":"algorithm","labels":["inexacty_y"],"declared_external":false,"source_only":false,"title":"下层校正残差要求","source_units":[{"raw_tex":"\nIf condition \\eqref{theta_gap_condition} holds, we attempt to correct $y^{k+1}$ by finding a candidate iterate $\\tilde{y}^{k+1} $  as an inexact solution to the lower-level problem \\eqref{LL} with $x= x^{k+1}$ satisfying\n\\begin{equation}\\label{inexacty_y}\n\t\\left\\| \\tilde{y}^{k+1} - \\operatorname{Prox}_{ \\tilde{g}(x^{k+1}, \\cdot)} \\left( \\tilde{y}^{k+1} -  \\nabla_y f(x^{k+1}, \\tilde{y}^{k+1}) \\right) \\right\\| \\le \\frac{c_{\\tilde{y}} }{p_k} \\Delta_{k+1}, \n\\end{equation}\n","start":44130,"end":44604}]},{"id":"S027","semantic_type":"algorithm","kind":"algorithm","labels":["decrease_new_y"],"declared_external":false,"source_only":false,"title":"校正候选的接受不等式","source_units":[{"raw_tex":"where $c_{\\tilde{y}} > 0$. However, unconditionally accepting such a $\\tilde{y}^{k+1} $ could adversely affect the algorithm's convergence. Therefore, to preserve convergence, $\\tilde{y}^{k+1} $ is accepted only if it satisfies the following descent condition\n\\begin{equation}\\label{decrease_new_y}\n\t\\begin{aligned}\n\t\t&\\frac{1}{p_k}F(x^{k+1}, \\tilde{y}^{k+1}) + \\varphi(x^{k+1}, \\tilde{y}^{k+1}) - \\left( \\varphi(x^{k+1},  \\tilde{\\theta}^{k+1} ) + \\frac{1}{2\\gamma} \\| \\tilde{\\theta}^{k+1} - \\tilde{y}^{k+1} \\|^2 \\right)\\\\\n\t\t\\le\\; &\n\t\t\\frac{1}{p_k}F(x^{k+1}, y^{k+1}) + \\varphi(x^{k+1}, y^{k+1}) - \\left( \\varphi(x^{k+1}, \\theta^{k+1}) + \\frac{1}{2\\gamma} \\| \\theta^{k+1} - y^{k+1} \\|^2 \\right),\n\t\\end{aligned}\n\\end{equation}\n","start":44604,"end":45330}]},{"id":"S028","semantic_type":"algorithm","kind":"algorithm","labels":["inexacty3"],"declared_external":false,"source_only":false,"title":"校正近端解的一步细化","source_units":[{"raw_tex":"where, $\\tilde{\\theta}^{k+1}$ is a new approximation of $\\theta^*_{\\gamma}(x^{k+1}, \\tilde{y}^{k+1})$. We construct $\\tilde{\\theta}^{k+1}$ by first finding a point $\\hat{\\theta}^{k+1}$ that satisfies the inexactness criterion\n\\begin{equation}\\label{inexacty3}\n\t\\mathcal{G}(\\hat{\\theta}^{k+1}, x^{k+1},\\tilde{y}^{k+1}) \\le s_{k+1},\n\t\\quad \\text{or} \\quad\n\t\\mathcal{G}(\\hat{\\theta}^{k+1}, x^{k+1},\\tilde{y}^{k+1})\n\t\\le \\tau_{k+1}\\mathcal{G}(\\theta^{k}, x^{k},y^{k}).\n\\end{equation}\nand subsequently applying a single proximal-gradient step to $\\hat{\\theta}^{k+1}$\n\\[\n\\tilde{\\theta}^{k+1}\n=\n\\operatorname{Prox}_{\\eta \\tilde g(x^{k+1},\\cdot)}\n\\left(\n\\hat{\\theta}^{k+1}\n-\\eta\\left(\n\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n+\\frac{\\hat{\\theta}^{k+1}-\\tilde y^{k+1}}{\\gamma}\n\\right)\n\\right).\n\\]\n","start":45330,"end":46116}]},{"id":"S029","semantic_type":"algorithm","kind":"algorithm","labels":[],"declared_external":false,"source_only":false,"title":"接受与拒绝后的状态","source_units":[{"raw_tex":"\nIf the descent condition \\eqref{decrease_new_y} holds, the algorithm adopts the candidate  $\\tilde{y}^{k+1} $ along with its corresponding $\\tilde{\\theta}^{k+1}$, and keep the penalty parameter unchanged, i.e., $p_{k+1}=\tp_k$. Otherwise, the correction is discarded, meaning $\\tilde{y}^{k+1} = y^{k+1}$ and $\\tilde{\\theta}^{k+1} = \\theta^{k+1}$ and the penalty parameter is increased, i.e., $p_{k+1}=\tp_k + \\varrho_p$. The algorithm then proceeds with the selected pair $(\\tilde{y}^{k+1}, \\tilde{\\theta}^{k+1})$. A detailed description of this feasibility correction procedure is presented in Algorithm \\ref{alg2}.\n","start":46116,"end":46732}]},{"id":"S030","semantic_type":"algorithm","kind":"algorithm","labels":["alg1"],"declared_external":false,"source_only":false,"title":"AGILS完整流程","source_units":[{"raw_tex":"\\begin{algorithm}[htp!]\n\\caption{Alternating Gradient-type algorithm with Inexact Lower-level Solutions (AGILS)}\n\\label{alg1}\n\\begin{algorithmic}[1]\n\t\\STATE{{\\bf Input}: Initial iterates $x^0,\\theta^{-1}, y^0, \\tilde{\\theta}^0, \\tilde{y}^0$, set $(x^{-1}, y^{-1}, \\theta^{-1}) = (x^0, y^0, \\tilde{\\theta}^0)$, stepsizes $\\alpha_k, \\beta_k$, relaxation parameter $\\epsilon$, proximal parameter $\\gamma$, penalty parameters $p_{0}$,  $\\varrho_p$, $c_p$, $c_y$, inexact parameters $\\eta$, $s_{k}$, $\\tau_k$, tolerance $tol$.}\n\t\\FOR{$k=0,\\ 1,\\cdots$}\n\t\\STATE{Construct the direction $d_y^k$ according to \\eqref{dy}, and update $y^{k+1}$ as\n\t\t$$\n\t\ty^{k+1} = \\operatorname{Prox}_{\\beta_k\\tilde{g}(x^{k,\\cdot})}\\left(\\tilde{y}^k-\\beta_k d^k_y\\right).\n\t\t$$}\n\t\n\t\\STATE{Find an inexact solution $\\theta^{k+1/2}$ that satisfies the inexactness criterion \\eqref{inexacty1}.}\n\t\n\t\\STATE{Construct the direction $d_x^k$ according to \\eqref{dx}, and update $x^{k+1}$ as \n\t\t$$ x^{k+1}=\\operatorname{Proj}_X\\left(x^k-\\alpha_k d_x^k\\right). \n\t\t$$}\n\t\n\t\\STATE{Find an inexact solution $\\theta^{k+1}$ that satisfies the inexactness criterion \\eqref{inexacty2}.}\n\t\n\t\\STATE{Calculate $\\Delta_{k+1} := \\left\\| (x^{k+1}, y^{k+1}) - (x^k, \\tilde{y}^k) \\right\\|$ and $t^{k+1}$ according to \\eqref{def_t}. \\\\\n\t\tStop when $k \\ge 1$ and \\(\\max\\{\\Delta_{k+1}, s_{k}, t^{k+1} \\} \\le tol.\\)\n\t}\n\t\\STATE{Update penalty parameter and apply feasibility correction:}\n\t\n\t\\STATE{\\hspace*{10pt}\\textbf{Case 1:}\n\t\t$\\Delta_{k+1} \\geq c_p \\min\\big\\{1/p_k,\\, t^{k+1} \\big\\}$ \\\\\n\t\t\\vspace*{1pt}\n\t\t\\qquad\\qquad\\quad Set \\( p_{k+1} := p_k \\), \\( \\tilde{y}^{k+1} := y^{k+1} \\), \\(\\tilde{\\theta}^{k+1} :=\\theta^{k+1}\\).}\n\t\\vspace*{2pt}\n\t\\STATE{\\hspace*{10pt}\\textbf{Case 2:} \n\t\t$\\Delta_{k+1} < c_p \\min\\big\\{1/p_k,\\, t^{k+1} \\big\\},\\,\\| y^{k+1} - \\theta^{k+1} \\| \\le c_y \\gamma/p_k$\\\\  \n\t\t\\vspace*{1pt}\n\t\t\\qquad\\qquad\\quad Set \\( p_{k+1} := p_k + \\varrho_p \\), \\( \\tilde{y}^{k+1} := y^{k+1} \\), \\(\\tilde{\\theta}^{k+1} :=\\theta^{k+1}\\).}\n\t\\vspace*{2pt}\n\t\n\t\\STATE{\\hspace*{10pt}\\textbf{Case 3:} \n\t\t$\\Delta_{k+1} < c_p \\min\\big\\{1/p_k,\\, t^{k+1} \\big\\},\\,\\| y^{k+1} - \\theta^{k+1} \\| > c_y \\gamma/p_k$\\\\\n\t\t\\vspace*{1pt}\n\t\t\\qquad\\qquad\\quad Apply the feasibility correction procedure (Algorithm~\\ref{alg2}).}\n\\ENDFOR\n\\RETURN $(x^{k+1},\\tilde{y}^{k+1})$\n\\end{algorithmic}\n\\end{algorithm}\n","start":47378,"end":49710}]},{"id":"S031","semantic_type":"algorithm","kind":"algorithm","labels":["alg2"],"declared_external":false,"source_only":false,"title":"可行性校正完整流程","source_units":[{"raw_tex":"\n\\begin{algorithm}[htp!]\n\\caption{Feasibility Correction Procedure}\n\\label{alg2}\n\\begin{algorithmic}[1]\n\n\\STATE{{\\bf Input}: Current iterates $x^{k+1}, y^{k+1}, \\theta^{k+1}$, $\\Delta_{k+1}$, proximal parameter $\\gamma$, penalty parameters $p_{k}$, $\\varrho_p$, inexact parameters $\\eta$, $s_{k}$, $\\tau_k$, constant $c_{\\tilde{y}}$.}\n\n\\STATE{Compute a candidate $\\tilde{y}^{k+1}$ by inexactly solving the lower-level problem \\eqref{LL} with $x=x^{k+1}$  to satisfy \\eqref{inexacty_y}.\n}\n\n\\STATE{Find an intermediate point $\\hat{\\theta}^{k+1}$ that satisfies the inexactness criterion \\eqref{inexacty3}, and compute $\\tilde{\\theta}^{k+1}=\n\\operatorname{Prox}_{\\eta \\tilde g(x^{k+1},\\cdot)}\n(\n\\hat{\\theta}^{k+1}\n-\\eta(\n\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n+(\\hat{\\theta}^{k+1}-\\tilde y^{k+1})/\\gamma\n)\n)$.}\n\n\\IF{the descent condition \\eqref{decrease_new_y} holds}\n\\STATE{Accept the candidate: Set $p_{k+1} := p_k$.}\n\\ELSE\n\\STATE{Reject the candidate: Set $\\tilde{y}^{k+1} := y^{k+1}$, $\\tilde{\\theta}^{k+1} := \\theta^{k+1}$ and $p_{k+1} := p_k + \\varrho_p$.}\n\\ENDIF\n\\RETURN $(\\tilde{y}^{k+1},\\tilde{\\theta}^{k+1},p_{k+1})$\n\\end{algorithmic}\n\\end{algorithm}\n","start":49710,"end":50868}]},{"id":"S032","semantic_type":"lemma","kind":"lemma","labels":["error_bound"],"declared_external":true,"declaration_text":["原前提列表允许 B0069；来源为 drusvyatskiy2018error, Theorems 3.4–3.5，或 mordukhovich2023globally, Proposition 2.2；作者在第525行明确归因，文内未给完整证明。"],"source_only":false,"title":"近端残差的距离误差界","source_units":[{"raw_tex":"We begin by considering the following lemma, which provides an estimate for the distance between the approximation $\\theta$ and the exact solution $\\theta^*_{\\gamma}(x,y)$ of the proximal lower-level problem, based on the prox-gradient residual $\\mathcal{G}(\\theta, x,y)$ in the inexact criteria \\eqref{inexacty1} and \\eqref{inexacty2}.  This estimate is derived from the $1/\\gamma$-strong convexity and $(1/\\gamma + L_f)$-smoothness of the component $f(x, \\theta) + \\|\\theta - y\\|^2/(2\\gamma)$ with respect to $\\theta$ in the objective of the proximal lower level problem, see, e.g., \\cite[Theorem 3.4 and 3.5]{drusvyatskiy2018error} or \\cite[Proposition 2.2]{mordukhovich2023globally}.\n\\begin{lemma}\\label{error_bound}\n\tSuppose Assumptions \\ref{asup2} and \\ref{asup3} hold, and let $\\gamma > 0$, $\\eta > 0$. Then, there exists $C_\\eta > 0$ such that for any $(x,y) \\in X \\times \\Re^m$, and $\\theta \\in \\Re^m$, it holds that\n\t\\[\n\t\\|\\theta - \\theta^*_{\\gamma}(x,y) \\| \\le C_\\eta  \\left\\|\\theta-\\operatorname{Prox}_{\\eta \\tilde{g}\\left(x, \\cdot\\right)}\\left(\\theta-\\eta\\left(\\nabla_y f\\left(x, \\theta\\right)+ (\\theta-y)/{\\gamma}\\right)\\right)\\right\\|,\n\t\\]\n\twhere $\\theta^*_{\\gamma}(x,y) := \\arg\\min_{\\theta \\in Y} \\{ \\varphi (x,\\theta)+\\frac{1}{2\\gamma}\\|\\theta-y\\|^2 \\}$.\n\\end{lemma}\n","start":51250,"end":52534}]},{"id":"S033","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"平移后的非负价值函数","source_units":[{"raw_tex":"We now proceed to establish a decreasing property for the proposed method based on the following merit function:\n\\begin{equation*}\n\t\\tilde{\\psi}_{p_{k}}(x,y) :=\\frac{1}{p_k}(F(x,y)-\\underline{F})+ f(x,y) + g(x,y)-v_{\\gamma}(x,y).\n\\end{equation*}\n\nBefore establishing the main decreasing property, we first demonstrate that the descent condition \\eqref{decrease_new_y}, imposed by the feasibility correction procedure (Algorithm \\ref{alg2}), ensures a decrease in $\\tilde{\\psi}_{p_{k}}$ when moving from $y^{k+1}$ to the corrected iterate $\\tilde{y}^{k+1} $.\n","start":53837,"end":54395}]},{"id":"S034","semantic_type":"lemma","kind":"lemma","labels":["lem0","lem1_eq12"],"declared_external":false,"source_only":false,"title":"校正步的近似下降","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem0}\n\tSuppose Assumptions \\ref{asup2} and \\ref{asup3} hold. Then, there exists $C_\\gamma>0$,  such that the sequence $\\{(x^k,y^k,\\tilde{y}^k)\\}$ generated by AGILS (\\cref{alg1}) satisfies\n\t\\begin{equation}\\label{lem1_eq12}\n\t\t\\tilde{\\psi}_{p_k}(x^{k+1}, \\tilde{y}^{k+1}) \\le  \\tilde{\\psi}_{p_k}(x^{k+1}, y^{k+1}) + C_{\\gamma} \\zeta_{k+1}^2,\n\t\\end{equation}\n\twhere  $\\zeta_{k} := \\max\\{ s_k,  ( \\prod_{i=0}^{k} \\tau_i ) \\mathcal{G}(\\theta^{0}, x^{0}, y^{0})\\}$.\n\\end{lemma}","start":54395,"end":54887}]},{"id":"S035","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"主分析继承的参数条件","source_units":[{"raw_tex":"Throughout this subsection and the remainder of this section, we assume that Assumptions \\ref{asup1}, \\ref{asup2}, and \\ref{asup3} hold, and $\\gamma \\in (0, 1/(\\rho_{f_2} +\\rho_{g_2}) )$. The sequences $\\{(x^k, y^k, \\tilde{y}^k)\\}$ and $\\{p_k\\}$ \nare generated by AGILS (\\cref{alg1}), where the step sizes satisfy $\\alpha_k \\in [\\underline{\\alpha}, 2/(L_{\\psi_{x,k}} + c_\\alpha)]$ and $\\beta_k \\in [\\underline{\\beta}, 2/(L_{\\psi_{y,k}} + c_\\beta)]$, where $L_{\\psi_{x,k}} = L_{F_x}/p_k+L_{f_x}+L_{g_1}+\\rho_{f_1} + \\rho_{g_1}$, $L_{\\psi_{y,k}} = L_{F_y}/p_k+L_{f_y}$ and $\\underline{\\alpha}, \\underline{\\beta}, c_\\alpha, c_\\beta$ are positive constants. The inexactness parameters satisfy $\\sum_{k=0}^{\\infty}s_k^2 < \\infty$ and \n$\\tau_k := \\tau_0/(k+1)^{p_\\tau}$ with $\\tau_0 > 0$ and $p_\\tau >0$.\n","start":68918,"end":69717}]},{"id":"S036","semantic_type":"lemma","kind":"lemma","labels":["lem9"],"declared_external":false,"source_only":false,"title":"负包络的x二次上界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem9}\n\tSuppose Assumptions \\ref{asup2} and \\ref{asup3} hold. Let $\\gamma \\in (0, 1/(\\rho_{f_2} +\\rho_{g_2}) )$ and  $(\\bar{x},\\bar{y})\\in X\\times \\Re^m$. Then, for any $(x,y) \\in X \\times \\Re^m$, the following inequalities hold,\n\t\\begin{align}\n\t\t-v_{\\gamma}(x,\\bar{y}) &\\le -v_{\\gamma}(\\bar{x},\\bar{y})-\\langle \\nabla_x v_{\\gamma}(\\bar{x},\\bar{y}),x-\\bar{x}\\rangle +\\frac{\\rho_{f_1} + \\rho_{g_1}}{2}\\|\\bar{x}-x\\|^2,  \\label{lem9_eq1} \\\\\n","start":52691,"end":53148}]},{"id":"S037","semantic_type":"lemma","kind":"lemma","labels":["lem9"],"declared_external":false,"source_only":false,"title":"负包络的y仿射上界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem9}\n\tSuppose Assumptions \\ref{asup2} and \\ref{asup3} hold. Let $\\gamma \\in (0, 1/(\\rho_{f_2} +\\rho_{g_2}) )$ and  $(\\bar{x},\\bar{y})\\in X\\times \\Re^m$. Then, for any $(x,y) \\in X \\times \\Re^m$, the following inequalities hold,\n","start":52691,"end":52940},{"raw_tex":"\t\t-v_{\\gamma}(\\bar{x},y) &\\le -v_{\\gamma}(\\bar{x},\\bar{y})-\\langle \\nabla_y v_{\\gamma}(\\bar{x},\\bar{y}),y-\\bar{y}\\rangle. \\label{lem9_eq2}\n\t\\end{align}\n\\end{lemma}","start":53148,"end":53311}]},{"id":"S038","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"两项包络上界的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\tAccording to Proposition \\ref{partial_grad_v}, for any $\\bar{y} \\in \\Re^m$, $v_{\\gamma}(x, \\bar{y})$ is a $(\\rho_{f_1} + \\rho_{g_1})$-weakly convex function with respect to $x$ on $X$, that is, $v_{\\gamma}(x,\\bar{y})+\\frac{\\rho_{f_1} + \\rho_{g_1}}{2}\\|x\\|^2$ is convex on $X$. Then we can obtain \\eqref{lem9_eq1} immediately. The second inequality \\eqref{lem9_eq2} follows from Proposition \\ref{partial_grad_v} that for any $\\bar{x} \\in X$, $v_{\\gamma}(\\bar{x}, y)$ is convex with respect to $y$.\n\\end{proof}","start":53312,"end":53835}]},{"id":"S039","semantic_type":"lemma","kind":"lemma","labels":["lem1","lem1_eq"],"declared_external":false,"source_only":false,"title":"交替更新的充分下降递推","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem1}\n\tSuppose Assumptions \\ref{asup1}, \\ref{asup2} and \\ref{asup3} hold. Let $\\gamma \\in (0, 1/(\\rho_{f_2} +\\rho_{g_2}) )$, $\\alpha_k \\in [0, 2/(L_{\\psi_{x,k}} + c_\\alpha)]$ and $\\beta_k \\in [0, 2/(L_{\\psi_{y,k}} + c_\\beta)]$, where $L_{\\psi_{x,k}} = L_{F_x}/p_k+L_{f_x}+L_{g_1}+\\rho_{f_1} + \\rho_{g_1}$, $L_{\\psi_{y,k}} = L_{F_y}/p_k+L_{f_y}$ and $c_\\alpha, c_\\beta$ are positive constants. Then, there exists $C_\\eta , C_\\gamma > 0$ such that the sequence $\\{(x^k,y^k,\\tilde{y}^k)\\}$ generated by AGILS {\\rm (}\\cref{alg1}{\\rm )} satisfies the following inequality,\n\t\\begin{equation}\\label{lem1_eq}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_{k+1}}\\left(x^{k+1}, \\tilde{y}^{k+1}\\right)\\leq\\,& \\tilde{\\psi}_{p_k}\\left(x^k, \\tilde{y}^k\\right) +  \\left( \\frac{C_\\eta^2}{c_\\alpha}(L_{f_x} + L_{g_2})^2 + \\frac{C_\\eta^2}{c_\\beta\\gamma^2 } \\right) \\zeta_{k}^2 + C_{\\gamma} \\zeta_{k+1}^2\\\\\n\t\t\t&- \\frac{c_\\alpha}{4} \\left\\|x^{k+1}-x^k\\right\\|^2  -\\frac{c_\\beta}{4} \\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2,\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere  $\\zeta_{k} := \\max\\{ s_k,  ( \\prod_{i=0}^{k} \\tau_i ) \\mathcal{G}(\\theta^{0}, x^{0}, y^{0})\\}$ and $\\{ \\zeta_{k} \\}$ is square summable.\n\\end{lemma}","start":60296,"end":61495}]},{"id":"S040","semantic_type":"derived_fact","kind":"derived_fact","labels":["lem1"],"declared_external":false,"source_only":false,"title":"作者定义的误差包络可和","source_units":[{"raw_tex":"\twhere  $\\zeta_{k} := \\max\\{ s_k,  ( \\prod_{i=0}^{k} \\tau_i ) \\mathcal{G}(\\theta^{0}, x^{0}, y^{0})\\}$ and $\\{ \\zeta_{k} \\}$ is square summable.\n","start":61339,"end":61484}]},{"id":"S041","semantic_type":"proof","kind":"proof","labels":["lem1_eq1","lem1_eq10","lem1_eq11","lem1_eq2","lem1_eq3","lem1_eq4","lem1_eq5","lem1_eq6","lem1_eq7","lem1_eq8","lem1_eq9"],"declared_external":false,"source_only":false,"title":"充分下降递推的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\t\n\tFirst, the update rule for \\( y^{k+1} \\) given in \\eqref{update_y} defines it as the minimizer of the function $ g(x^k, y) + \\langle d_y^k, y - \\tilde{y}^{k} \\rangle + \\frac{1}{2\\beta_k} \\left\\| y - \\tilde{y}^k \\right\\|^2$. This function is $1/\\beta_k$-strongly convex in $y$. By applying \\cite[Theorem 6.39]{beck2017first}, we obtain\n\t\\begin{equation}\\label{lem1_eq5}\n\t\tg(x^{k}, y^{k+1}) + \\langle d_y^k, y^{k+1}-\\tilde{y}^k\\rangle+\\frac{1}{\\beta_k}\\|y^{k+1}-\\tilde{y}^k\\|^2 \\le g(x^{k}, \\tilde{y}^{k}).\n\t\\end{equation}\n\t\n\t\n\tAccording to the assumptions, with $x^{k} \\in X$, we have that $\\nabla_y F(x^{k},y)$ and $\\nabla_y f(x^{k},y)$ are $L_{F_y}$- and  $L_{f_y}$-Lipschitz continuous with respect to variable $y$ on $Y$, respectively, and combine this with \\eqref{lem9_eq2}, we have\n\t\\begin{equation}\\label{lem1_eq6}\n\t\t\\begin{aligned}\n\t\t\t&(\\tilde{\\psi}_{p_k}-g)(x^{k},y^{k+1})  \\\\\n\t\t\t\\le \\, & (\\tilde{\\psi}_{p_k}-g)\t(x^{k},\\tilde{y}^k) + \\langle \\nabla_y (\\tilde{\\psi}_{p_k}-g)(x^{k},\\tilde{y}^k),y^{k+1}-\\tilde{y}^k\\rangle +\\frac{L_{\\psi_{y,k}}}{2}\\|y^{k+1}-\\tilde{y}^k\\|^2,\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere $L_{\\psi_{y,k}}=L_{F_y}/p_k+L_{f_y}$. \n\tCombining \\eqref{lem1_eq5} and \\eqref{lem1_eq6} yields\n\t\\begin{equation}\\label{lem1_eq7}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k}, y^{k+1}\\right) \\leq \\,& \t\\tilde{\\psi}_{p_k}\\left(x^{k}, \\tilde{y}^k\\right) + \\langle \\nabla_y (\t\\tilde{\\psi}_{p_k}-g)(x^{k},\\tilde{y}^k) - d_y^k, y^{k+1}-\\tilde{y}^k\\rangle  \\\\\n\t\t\t&-\\left(\\frac{1}{\\beta_k}-\\frac{L_{\\psi_{y,k}}}{2}\\right)\\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2.\n\t\t\\end{aligned}\n\t\\end{equation}\t\n\tUsing the formula of \\( \\nabla_y v_\\gamma \\) in \\eqref{valuefinc}, and the construction of $d_y^k$ in \\eqref{dy}, we have\n\t\\begin{equation}\\label{lem1_eq11}\n\t\t\\begin{aligned}\n\t\t\t\\left\\| \\nabla_y (\t\\tilde{\\psi}_{p_k}-g)(x^{k},\\tilde{y}^k) - d_y^k \\right\\| & = \\| \\nabla_y v_\\gamma(x^k, \\tilde{y}^k) + (\\tilde{y}^k-\\theta^{k})/\\gamma\\| \\\\\n\t\t\t& = \\frac{1}{\\gamma}\\left\\| \\theta^*_{\\gamma}(x^{k},\\tilde{y}^k) - \\theta^{k} \\right\\| \\le \\frac{C_\\eta}{\\gamma} \\mathcal{G}({\\theta}^{k}, x^{k}, \\tilde{y}^{k}) \\le \\frac{C_\\eta}{\\gamma} \\zeta_{k},\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere the first inequality follows from Lemma \\ref{error_bound} and the second inequality is a consequence of the inexactness condition \\eqref{inexacty2}. \n\t\n\tCombining the above inequality with \\eqref{lem1_eq7} gives us\n\t\\begin{equation}\\label{lem1_eq8}\n\t\t\\tilde{\\psi}_{p_k}\\left(x^{k}, y^{k+1}\\right) \\leq \\,  \t\\tilde{\\psi}_{p_k}\\left(x^{k}, \\tilde{y}^k\\right)  - \\left(\\frac{1}{\\beta_k}-\\frac{L_{\\psi_{y,k}}}{2}- \\frac{c_\\beta}{4}\\right)\\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2 + \\frac{C_\\eta^2}{c_\\beta\\gamma^2} \\zeta_{k}^2.\n\t\\end{equation}\n\tNext, considering the update rule for $x^{k+1}$ given in \\eqref{update_x}, we have\n\t\\begin{equation}\\label{lem1_eq1}\n\t\t\\langle d_x^k, x^{k+1}-x^k\\rangle+\\frac{1}{\\alpha_k}\\|x^{k+1}-x^k\\|^2 \\le 0.\n\t\\end{equation}\t\n\tBy the assumptions, with $y^{k+1} \\in Y$, we have that $\\nabla_x F(x, y^{k+1})$, $\\nabla_x f(x, y^{k+1})$ and $\\nabla_{x} g(x, y^{k+1})$ are $L_{F_x}$-, $L_{f_x}$-, and $L_{g_1}$-Lipschitz continuous with respect to variable $x$ on $X$, respectively, and combine this with \\eqref{lem9_eq1}, we have\n\t\\begin{equation}\\label{lem1_eq2}\n\t\t\\tilde{\\psi}_{p_k}(x^{k+1}, y^{k+1})\\le  \\tilde{\\psi}_{p_k}\t(x^{k}, y^{k+1})+\\langle \\nabla_x \t\\tilde{\\psi}_{p_k}(x^{k}, y^{k+1}),x^{k+1}-x^k\\rangle+\\frac{L_{\\psi_{x,k}}}{2}\\|x^{k+1}-x^k\\|^2,\n\t\\end{equation}\n\twhere $L_{\\psi_{x,k}}=L_{F_x}/p_k+L_{f_x}+L_{g_1}+ \\rho_{f_1} + \\rho_{g_1} $. Combining \\eqref{lem1_eq1} and \\eqref{lem1_eq2} yields\n\t\\begin{equation}\\label{lem1_eq3}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k+1}, y^{k+1}\\right) \\leq \\,& \\tilde{\\psi}_{p_k}\\left(x^k, y^{k+1}\\right) + \\langle \\nabla_x \\tilde{\\psi}_{p_k}(x^{k},y^{k+1}) - d_x^k,x^{k+1}-x^k\\rangle  \\\\\n\t\t\t&-\\left(\\frac{1}{\\alpha_k}-\\frac{L_{\\psi_{x,k}}}{2}\\right)\\left\\|x^{k+1}-x^k\\right\\|^2.\n\t\t\\end{aligned}\n\t\\end{equation}\n\tUsing the formula of the gradient of $v_\\gamma$ given in \\eqref{valuefinc} and the construction of $d_x^k$ in \\eqref{dx}, and the $L_{f_x}$- and $L_{g_2}$-Lipschitz continuity of $\\nabla_x f(x^k,y)$ and $\\nabla_{x} g(x^k,y)$ with respect to variable $y$ on $Y$, we have\n\t\\begin{equation}\\label{lem1_eq10}\n\t\t\\begin{aligned}\n\t\t\t\\left\\| \\nabla_x  \\tilde{\\psi}_{p_k}(x^{k},y^{k+1}) - d_x^k \\right\\| \n\t\t\t&= \\left\\| \\nabla_{x} \\varphi(x^k, \\theta^*_{\\gamma}(x^k,y^{k+1})) - \\nabla_{x} \\varphi(x^k, \\theta^{k+1/2}) \\right\\| \\\\\n\t\t\t& \\le (L_{f_x} + L_{g_2})\\left\\| \\theta^*_{\\gamma}(x^k,y^{k+1}) - \\theta^{k+1/2} \\right\\| \\le (L_{f_x} + L_{g_2})C_\\eta \\zeta_{k},\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere the last inequality follows from \\cref{error_bound} and the inexact criterion \\eqref{inexacty1}. Combining the above inequality with \\eqref{lem1_eq3} gives us\n\t\\begin{equation}\\label{lem1_eq4}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k+1}, y^{k+1}\\right) \\leq \\,& \\tilde{\\psi}_{p_k}\\left(x^k, y^{k+1}\\right) - \\left(\\frac{1}{\\alpha_k}-\\frac{L_{\\psi_{x,k}}}{2}-\\frac{c_\\alpha}{4}\\right)\\left\\|x^{k+1}-x^k\\right\\|^2 \\\\ & + \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2C_\\eta^2 \\zeta_{k}^2.\n\t\t\\end{aligned}\n\t\\end{equation}\n\tCombining  \\eqref{lem1_eq8}, \\eqref{lem1_eq4}, and then  \\eqref{lem1_eq12} established in  Lemma \\ref{lem0}, we obtain\n\t\\begin{equation}\\label{lem1_eq9}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k+1}, \\tilde{y}^{k+1}\\right)\\leq\\,& \t\\tilde{\\psi}_{p_k}\\left(x^k, \\tilde{y}^k\\right)+  \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2C_\\eta^2 \\zeta_{k}^2 + \\frac{C_\\eta^2}{c_\\beta\\gamma^2} \\zeta_{k}^2 + C_{\\gamma} \\zeta_{k+1}^2\\\\\n\t\t\t&\\hspace{-50pt}-\\left(\\frac{1}{\\alpha_k}-\\frac{L_{\\psi_{x,k}}}{2}- \\frac{c_\\alpha}{4} \\right)\\left\\|x^{k+1}-x^k\\right\\|^2 -\\left(\\frac{1}{\\beta_k}-\\frac{L_{\\psi_{y,k}}}{2}- \\frac{c_\\beta}{4} \\right)\\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2.\n\t\t\\end{aligned}\n\t\\end{equation}\n\tBecause  $\\alpha_k \\in [0, 2/(L_{\\psi_{x,k}} + c_\\alpha)]$ and $\\beta_k \\in [0, 2/(L_{\\psi_{y,k}} + c_\\beta)]$, and $\\tilde{\\psi}_{p_{k+1}}(x^{k+1}, \\tilde{y}^{k+1}) \\leq \t\\tilde{\\psi}_{p_k}(x^{k+1}, \\tilde{y}^{k+1})$ following from the non-decreasing property of the penalty parameter $p_k$ and the non-negative property of $F-\\underline{F}$, we can obtain \\eqref{lem1_eq} from \\eqref{lem1_eq9}.\n\t\n\tFinally, we show that \\( \\{\\zeta_k\\} \\) is square summable. Let $ \\mathcal{G}_0 = \\mathcal{G}(\\theta^{0}, x^{0}, y^{0})$ and $r_k :=  ( \\prod_{i=0}^{k} \\tau_i )\\mathcal{G}_0 $. Since $\\zeta_{k} := \\max\\{ s_k, r_k\\}$, we have $\\sum_{k=0}^\\infty \\zeta_k^2 \\le \\sum_{k=0}^\\infty s_k^2 + \\sum_{k=0}^\\infty r_k^2$. By assumption, \\( \\sum_{k=0}^\\infty s_k^2 < \\infty \\). Thus, we only need to show that  \\( \\{r_k\\} \\) is square summable. Recall that \\( \\tau_k = \\tau_0/(k+1)^{p_\\tau} \\) with \\( p_\\tau > 0 \\), then\n\t\\[\n\tr_k = \\left( \\prod_{i=0}^{k} \\tau_i \\right) \\mathcal{G}_0  = \\tau_0^{k+1} \\left( \\prod_{i=1}^{k+1} \\frac{1}{i^{p_\\tau}} \\right) \\cdot \\mathcal{G}_0  = \\frac{\\tau_0^{k+1}}{((k+1)!)^{p_\\tau}}\\mathcal{G}_0 .\n\t\\]\n\tFor the ratio test on  $\\sum_{k=0}^\\infty r_k^2$, we consider \n\t\\[\n\t\\lim_{k \\rightarrow \\infty}\\frac{r_{k+1}^2}{r_k^2} =  \\lim_{k \\rightarrow \\infty} \\frac{\\tau_0^2}{(k+2)^{2p_\\tau}}  = 0.\n\t\\]\n\tTherefore, by the ratio test, $\\sum_{k=0}^\\infty r_k^2 < \\infty$, implying that \\( \\{\\zeta_k\\} \\) is square summable.\n\\end{proof}","start":61497,"end":68849}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":["lem1_eq5"],"declared_external":false,"source_only":false,"title":"y近端子问题的强凸估计","source_units":[{"raw_tex":"\\begin{proof}\n\t\n\tFirst, the update rule for \\( y^{k+1} \\) given in \\eqref{update_y} defines it as the minimizer of the function $ g(x^k, y) + \\langle d_y^k, y - \\tilde{y}^{k} \\rangle + \\frac{1}{2\\beta_k} \\left\\| y - \\tilde{y}^k \\right\\|^2$. This function is $1/\\beta_k$-strongly convex in $y$. By applying \\cite[Theorem 6.39]{beck2017first}, we obtain\n\t\\begin{equation}\\label{lem1_eq5}\n\t\tg(x^{k}, y^{k+1}) + \\langle d_y^k, y^{k+1}-\\tilde{y}^k\\rangle+\\frac{1}{\\beta_k}\\|y^{k+1}-\\tilde{y}^k\\|^2 \\le g(x^{k}, \\tilde{y}^{k}).\n\t\\end{equation}\n\t\n\t\n","start":61497,"end":62039}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":["lem1_eq6"],"declared_external":false,"source_only":false,"title":"y方向光滑项的上界","source_units":[{"raw_tex":"\tAccording to the assumptions, with $x^{k} \\in X$, we have that $\\nabla_y F(x^{k},y)$ and $\\nabla_y f(x^{k},y)$ are $L_{F_y}$- and  $L_{f_y}$-Lipschitz continuous with respect to variable $y$ on $Y$, respectively, and combine this with \\eqref{lem9_eq2}, we have\n\t\\begin{equation}\\label{lem1_eq6}\n\t\t\\begin{aligned}\n\t\t\t&(\\tilde{\\psi}_{p_k}-g)(x^{k},y^{k+1})  \\\\\n\t\t\t\\le \\, & (\\tilde{\\psi}_{p_k}-g)\t(x^{k},\\tilde{y}^k) + \\langle \\nabla_y (\\tilde{\\psi}_{p_k}-g)(x^{k},\\tilde{y}^k),y^{k+1}-\\tilde{y}^k\\rangle +\\frac{L_{\\psi_{y,k}}}{2}\\|y^{k+1}-\\tilde{y}^k\\|^2,\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere $L_{\\psi_{y,k}}=L_{F_y}/p_k+L_{f_y}$. \n","start":62039,"end":62672}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":["lem1_eq7"],"declared_external":false,"source_only":false,"title":"合并y步比较式","source_units":[{"raw_tex":"\tCombining \\eqref{lem1_eq5} and \\eqref{lem1_eq6} yields\n\t\\begin{equation}\\label{lem1_eq7}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k}, y^{k+1}\\right) \\leq \\,& \t\\tilde{\\psi}_{p_k}\\left(x^{k}, \\tilde{y}^k\\right) + \\langle \\nabla_y (\t\\tilde{\\psi}_{p_k}-g)(x^{k},\\tilde{y}^k) - d_y^k, y^{k+1}-\\tilde{y}^k\\rangle  \\\\\n\t\t\t&-\\left(\\frac{1}{\\beta_k}-\\frac{L_{\\psi_{y,k}}}{2}\\right)\\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2.\n\t\t\\end{aligned}\n\t\\end{equation}\t\n","start":62672,"end":63122}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["lem1_eq11"],"declared_external":false,"source_only":false,"title":"y方向近似梯度误差","source_units":[{"raw_tex":"\tUsing the formula of \\( \\nabla_y v_\\gamma \\) in \\eqref{valuefinc}, and the construction of $d_y^k$ in \\eqref{dy}, we have\n\t\\begin{equation}\\label{lem1_eq11}\n\t\t\\begin{aligned}\n\t\t\t\\left\\| \\nabla_y (\t\\tilde{\\psi}_{p_k}-g)(x^{k},\\tilde{y}^k) - d_y^k \\right\\| & = \\| \\nabla_y v_\\gamma(x^k, \\tilde{y}^k) + (\\tilde{y}^k-\\theta^{k})/\\gamma\\| \\\\\n\t\t\t& = \\frac{1}{\\gamma}\\left\\| \\theta^*_{\\gamma}(x^{k},\\tilde{y}^k) - \\theta^{k} \\right\\| \\le \\frac{C_\\eta}{\\gamma} \\mathcal{G}({\\theta}^{k}, x^{k}, \\tilde{y}^{k}) \\le \\frac{C_\\eta}{\\gamma} \\zeta_{k},\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere the first inequality follows from Lemma \\ref{error_bound} and the second inequality is a consequence of the inexactness condition \\eqref{inexacty2}. \n","start":63122,"end":63850}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["lem1_eq8"],"declared_external":false,"source_only":false,"title":"吸收y方向误差","source_units":[{"raw_tex":"\t\n\tCombining the above inequality with \\eqref{lem1_eq7} gives us\n\t\\begin{equation}\\label{lem1_eq8}\n\t\t\\tilde{\\psi}_{p_k}\\left(x^{k}, y^{k+1}\\right) \\leq \\,  \t\\tilde{\\psi}_{p_k}\\left(x^{k}, \\tilde{y}^k\\right)  - \\left(\\frac{1}{\\beta_k}-\\frac{L_{\\psi_{y,k}}}{2}- \\frac{c_\\beta}{4}\\right)\\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2 + \\frac{C_\\eta^2}{c_\\beta\\gamma^2} \\zeta_{k}^2.\n\t\\end{equation}\n","start":63850,"end":64235}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":["lem1_eq1","lem1_eq2"],"declared_external":false,"source_only":false,"title":"x投影与目标二次上界","source_units":[{"raw_tex":"\tNext, considering the update rule for $x^{k+1}$ given in \\eqref{update_x}, we have\n\t\\begin{equation}\\label{lem1_eq1}\n\t\t\\langle d_x^k, x^{k+1}-x^k\\rangle+\\frac{1}{\\alpha_k}\\|x^{k+1}-x^k\\|^2 \\le 0.\n\t\\end{equation}\t\n\tBy the assumptions, with $y^{k+1} \\in Y$, we have that $\\nabla_x F(x, y^{k+1})$, $\\nabla_x f(x, y^{k+1})$ and $\\nabla_{x} g(x, y^{k+1})$ are $L_{F_x}$-, $L_{f_x}$-, and $L_{g_1}$-Lipschitz continuous with respect to variable $x$ on $X$, respectively, and combine this with \\eqref{lem9_eq1}, we have\n\t\\begin{equation}\\label{lem1_eq2}\n\t\t\\tilde{\\psi}_{p_k}(x^{k+1}, y^{k+1})\\le  \\tilde{\\psi}_{p_k}\t(x^{k}, y^{k+1})+\\langle \\nabla_x \t\\tilde{\\psi}_{p_k}(x^{k}, y^{k+1}),x^{k+1}-x^k\\rangle+\\frac{L_{\\psi_{x,k}}}{2}\\|x^{k+1}-x^k\\|^2,\n\t\\end{equation}\n\twhere $L_{\\psi_{x,k}}=L_{F_x}/p_k+L_{f_x}+L_{g_1}+ \\rho_{f_1} + \\rho_{g_1} $. Combining \\eqref{lem1_eq1} and \\eqref{lem1_eq2} yields\n","start":64235,"end":65127}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":["lem1_eq3"],"declared_external":false,"source_only":false,"title":"合并x步比较式","source_units":[{"raw_tex":"\t\\begin{equation}\\label{lem1_eq3}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k+1}, y^{k+1}\\right) \\leq \\,& \\tilde{\\psi}_{p_k}\\left(x^k, y^{k+1}\\right) + \\langle \\nabla_x \\tilde{\\psi}_{p_k}(x^{k},y^{k+1}) - d_x^k,x^{k+1}-x^k\\rangle  \\\\\n\t\t\t&-\\left(\\frac{1}{\\alpha_k}-\\frac{L_{\\psi_{x,k}}}{2}\\right)\\left\\|x^{k+1}-x^k\\right\\|^2.\n\t\t\\end{aligned}\n\t\\end{equation}\n","start":65127,"end":65490}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["lem1_eq10"],"declared_external":false,"source_only":false,"title":"x方向近似梯度误差","source_units":[{"raw_tex":"\tUsing the formula of the gradient of $v_\\gamma$ given in \\eqref{valuefinc} and the construction of $d_x^k$ in \\eqref{dx}, and the $L_{f_x}$- and $L_{g_2}$-Lipschitz continuity of $\\nabla_x f(x^k,y)$ and $\\nabla_{x} g(x^k,y)$ with respect to variable $y$ on $Y$, we have\n\t\\begin{equation}\\label{lem1_eq10}\n\t\t\\begin{aligned}\n\t\t\t\\left\\| \\nabla_x  \\tilde{\\psi}_{p_k}(x^{k},y^{k+1}) - d_x^k \\right\\| \n\t\t\t&= \\left\\| \\nabla_{x} \\varphi(x^k, \\theta^*_{\\gamma}(x^k,y^{k+1})) - \\nabla_{x} \\varphi(x^k, \\theta^{k+1/2}) \\right\\| \\\\\n\t\t\t& \\le (L_{f_x} + L_{g_2})\\left\\| \\theta^*_{\\gamma}(x^k,y^{k+1}) - \\theta^{k+1/2} \\right\\| \\le (L_{f_x} + L_{g_2})C_\\eta \\zeta_{k},\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere the last inequality follows from \\cref{error_bound} and the inexact criterion \\eqref{inexacty1}. Combining the above inequality with \\eqref{lem1_eq3} gives us\n","start":65490,"end":66343}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":["lem1_eq4"],"declared_external":false,"source_only":false,"title":"吸收x方向误差","source_units":[{"raw_tex":"\t\\begin{equation}\\label{lem1_eq4}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k+1}, y^{k+1}\\right) \\leq \\,& \\tilde{\\psi}_{p_k}\\left(x^k, y^{k+1}\\right) - \\left(\\frac{1}{\\alpha_k}-\\frac{L_{\\psi_{x,k}}}{2}-\\frac{c_\\alpha}{4}\\right)\\left\\|x^{k+1}-x^k\\right\\|^2 \\\\ & + \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2C_\\eta^2 \\zeta_{k}^2.\n\t\t\\end{aligned}\n\t\\end{equation}\n","start":66343,"end":66705}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":["lem1_eq9"],"declared_external":false,"source_only":false,"title":"合并校正与罚参数更新","source_units":[{"raw_tex":"\tCombining  \\eqref{lem1_eq8}, \\eqref{lem1_eq4}, and then  \\eqref{lem1_eq12} established in  Lemma \\ref{lem0}, we obtain\n\t\\begin{equation}\\label{lem1_eq9}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k+1}, \\tilde{y}^{k+1}\\right)\\leq\\,& \t\\tilde{\\psi}_{p_k}\\left(x^k, \\tilde{y}^k\\right)+  \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2C_\\eta^2 \\zeta_{k}^2 + \\frac{C_\\eta^2}{c_\\beta\\gamma^2} \\zeta_{k}^2 + C_{\\gamma} \\zeta_{k+1}^2\\\\\n\t\t\t&\\hspace{-50pt}-\\left(\\frac{1}{\\alpha_k}-\\frac{L_{\\psi_{x,k}}}{2}- \\frac{c_\\alpha}{4} \\right)\\left\\|x^{k+1}-x^k\\right\\|^2 -\\left(\\frac{1}{\\beta_k}-\\frac{L_{\\psi_{y,k}}}{2}- \\frac{c_\\beta}{4} \\right)\\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2.\n\t\t\\end{aligned}\n\t\\end{equation}\n\tBecause  $\\alpha_k \\in [0, 2/(L_{\\psi_{x,k}} + c_\\alpha)]$ and $\\beta_k \\in [0, 2/(L_{\\psi_{y,k}} + c_\\beta)]$, and $\\tilde{\\psi}_{p_{k+1}}(x^{k+1}, \\tilde{y}^{k+1}) \\leq \t\\tilde{\\psi}_{p_k}(x^{k+1}, \\tilde{y}^{k+1})$ following from the non-decreasing property of the penalty parameter $p_k$ and the non-negative property of $F-\\underline{F}$, we can obtain \\eqref{lem1_eq} from \\eqref{lem1_eq9}.\n","start":66705,"end":67799}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"阶乘衰减与平方可和","source_units":[{"raw_tex":"\t\n\tFinally, we show that \\( \\{\\zeta_k\\} \\) is square summable. Let $ \\mathcal{G}_0 = \\mathcal{G}(\\theta^{0}, x^{0}, y^{0})$ and $r_k :=  ( \\prod_{i=0}^{k} \\tau_i )\\mathcal{G}_0 $. Since $\\zeta_{k} := \\max\\{ s_k, r_k\\}$, we have $\\sum_{k=0}^\\infty \\zeta_k^2 \\le \\sum_{k=0}^\\infty s_k^2 + \\sum_{k=0}^\\infty r_k^2$. By assumption, \\( \\sum_{k=0}^\\infty s_k^2 < \\infty \\). Thus, we only need to show that  \\( \\{r_k\\} \\) is square summable. Recall that \\( \\tau_k = \\tau_0/(k+1)^{p_\\tau} \\) with \\( p_\\tau > 0 \\), then\n\t\\[\n\tr_k = \\left( \\prod_{i=0}^{k} \\tau_i \\right) \\mathcal{G}_0  = \\tau_0^{k+1} \\left( \\prod_{i=1}^{k+1} \\frac{1}{i^{p_\\tau}} \\right) \\cdot \\mathcal{G}_0  = \\frac{\\tau_0^{k+1}}{((k+1)!)^{p_\\tau}}\\mathcal{G}_0 .\n\t\\]\n\tFor the ratio test on  $\\sum_{k=0}^\\infty r_k^2$, we consider \n\t\\[\n\t\\lim_{k \\rightarrow \\infty}\\frac{r_{k+1}^2}{r_k^2} =  \\lim_{k \\rightarrow \\infty} \\frac{\\tau_0^2}{(k+2)^{2p_\\tau}}  = 0.\n\t\\]\n\tTherefore, by the ratio test, $\\sum_{k=0}^\\infty r_k^2 < \\infty$, implying that \\( \\{\\zeta_k\\} \\) is square summable.\n\\end{proof}","start":67799,"end":68849}]},{"id":"S053","semantic_type":"lemma","kind":"lemma","labels":["lem3"],"declared_external":false,"source_only":false,"title":"交替步差平方可和","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem3}\n\tThe sequence $\\left\\{(x^k,y^k,\\tilde{y}^k)\\right\\}$ satisfies\n\t\\begin{equation*}\n\t\t\\sum\\limits_{k=0}^{{\\infty}}\\left\\|(x^{k+1},y^{k+1})-(x^k,\\tilde{y}^k)\\right\\|^2<\\infty, \\quad \\text{and} \\quad \\lim\\limits_{k\\to\\infty}\\left\\|(x^{k+1},y^{k+1})-(x^k,\\tilde{y}^k)\\right\\|=0.\n\t\\end{equation*}\n\\end{lemma}","start":69955,"end":70283}]},{"id":"S054","semantic_type":"proof","kind":"proof","labels":["lem3_eq1","lem3_eq2"],"declared_external":false,"source_only":false,"title":"步差可和性的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\tSumming  \\cref{lem1_eq} from Lemma \\ref{lem1} over $k=0,1,\\dots,K-1$ yields,\n\t\\begin{equation}\\label{lem3_eq1}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_{K}}\\left(x^{K}, \\tilde{y}^{K}\\right)- \\tilde{\\psi}_{p_{0}}\\left(x^{0}, \\tilde{y}^{0}\\right) \\leq & -\\sum\\limits_{k=0}^{K-1} \\left(  \\frac{c_\\alpha}{4} \\left\\|x^{k+1}-x^k\\right\\|^2+ \\frac{c_\\beta}{4} \\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2\\right)\\\\& +\\sum\\limits_{k=0}^{K-1}\\left( \\frac{C_\\eta^2 }{c_\\alpha}(L_{f_x} + L_{g_2})^2+ \\frac{C_\\eta^2}{c_\\beta\\gamma^2} \\right) \\zeta_{k}^2+ \\sum\\limits_{k=0}^{K-1}C_{\\gamma} \\zeta_{k+1}^2.\\\\\n\t\t\\end{aligned}\n\t\\end{equation}\n\tSince \\( \\{\\zeta_k\\} \\) is square summable from Lemma \\ref{lem1}, it follows that\n\t\\[\n\t\\sum\\limits_{k=0}^{\\infty} \\left( \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2 + \\frac{1}{c_\\beta\\gamma^2} \\right) C_\\eta^2\\zeta_{k}^2 + \\sum\\limits_{k=0}^{\\infty}  C_{\\gamma} \\zeta_{k+1}^2 < \\infty.\n\t\\]\n\tAdditionally, because $\\tilde{\\psi}_{p_{K}}\\left(x^{K}, \\tilde{y}^{K}\\right) \\ge 0$, by taking $K \\rightarrow \\infty$ in \\eqref{lem3_eq1}, we have\n\t\\begin{equation}\\label{lem3_eq2}\n\t\t\\begin{aligned}\n\t\t\t&\\quad\\,\\sum\\limits_{k=0}^{\\infty} \\left(  \\frac{c_\\alpha}{4} \\left\\|x^{k+1}-x^k\\right\\|^2+ \\frac{c_\\beta}{4} \\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2\\right) \\\\\n\t\t\t&\\le \\tilde{\\psi}_{p_{0}}\\left(x^{0}, \\tilde{y}^{0}\\right) + \\sum\\limits_{k=0}^{\\infty} \\left( \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2 + \\frac{1}{c_\\beta\\gamma^2} \\right) C_\\eta^2\\zeta_{k}^2 + \\sum\\limits_{k=0}^{\\infty}  C_{\\gamma} \\zeta_{k+1}^2  < \\infty,\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhich proves the result.\n\\end{proof}","start":70284,"end":71887}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["lem3_eq1"],"declared_external":false,"source_only":false,"title":"对下降递推望远镜求和","source_units":[{"raw_tex":"\\begin{proof}\n\tSumming  \\cref{lem1_eq} from Lemma \\ref{lem1} over $k=0,1,\\dots,K-1$ yields,\n\t\\begin{equation}\\label{lem3_eq1}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_{K}}\\left(x^{K}, \\tilde{y}^{K}\\right)- \\tilde{\\psi}_{p_{0}}\\left(x^{0}, \\tilde{y}^{0}\\right) \\leq & -\\sum\\limits_{k=0}^{K-1} \\left(  \\frac{c_\\alpha}{4} \\left\\|x^{k+1}-x^k\\right\\|^2+ \\frac{c_\\beta}{4} \\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2\\right)\\\\& +\\sum\\limits_{k=0}^{K-1}\\left( \\frac{C_\\eta^2 }{c_\\alpha}(L_{f_x} + L_{g_2})^2+ \\frac{C_\\eta^2}{c_\\beta\\gamma^2} \\right) \\zeta_{k}^2+ \\sum\\limits_{k=0}^{K-1}C_{\\gamma} \\zeta_{k+1}^2.\\\\\n\t\t\\end{aligned}\n\t\\end{equation}\n","start":70284,"end":70911}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":["lem3_eq2"],"declared_external":false,"source_only":false,"title":"用非负下界控制总步差","source_units":[{"raw_tex":"\tSince \\( \\{\\zeta_k\\} \\) is square summable from Lemma \\ref{lem1}, it follows that\n\t\\[\n\t\\sum\\limits_{k=0}^{\\infty} \\left( \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2 + \\frac{1}{c_\\beta\\gamma^2} \\right) C_\\eta^2\\zeta_{k}^2 + \\sum\\limits_{k=0}^{\\infty}  C_{\\gamma} \\zeta_{k+1}^2 < \\infty.\n\t\\]\n\tAdditionally, because $\\tilde{\\psi}_{p_{K}}\\left(x^{K}, \\tilde{y}^{K}\\right) \\ge 0$, by taking $K \\rightarrow \\infty$ in \\eqref{lem3_eq1}, we have\n\t\\begin{equation}\\label{lem3_eq2}\n\t\t\\begin{aligned}\n\t\t\t&\\quad\\,\\sum\\limits_{k=0}^{\\infty} \\left(  \\frac{c_\\alpha}{4} \\left\\|x^{k+1}-x^k\\right\\|^2+ \\frac{c_\\beta}{4} \\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2\\right) \\\\\n\t\t\t&\\le \\tilde{\\psi}_{p_{0}}\\left(x^{0}, \\tilde{y}^{0}\\right) + \\sum\\limits_{k=0}^{\\infty} \\left( \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2 + \\frac{1}{c_\\beta\\gamma^2} \\right) C_\\eta^2\\zeta_{k}^2 + \\sum\\limits_{k=0}^{\\infty}  C_{\\gamma} \\zeta_{k+1}^2  < \\infty,\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhich proves the result.\n\\end{proof}","start":70911,"end":71887}]},{"id":"S057","semantic_type":"lemma","kind":"lemma","labels":["lem4"],"declared_external":false,"source_only":false,"title":"校正聚点为下层最优解","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem4}\n\tLet $\\{ (x^{k_j},\\tilde{y}^{k_j}) \\} $ be any convergent subsequence of iterates $\\{(x^k,\\tilde{y}^k)\\}$ such that $(x^{k_j},\\tilde{y}^{k_j})$ satisfies condition \\eqref{inexacty_y} with $k = k_j - 1$. \n\tIf $(x^{k_j},\\tilde{y}^{k_j}) \\rightarrow (\\bar{x},\\bar{y}) $ as $j \\rightarrow \\infty$ for some $(\\bar{x},\\bar{y}) $, then $\\bar{y} \\in S(\\bar{x})$. \n\\end{lemma}","start":72078,"end":72471}]},{"id":"S058","semantic_type":"proof","kind":"proof","labels":["lem4_eq1"],"declared_external":false,"source_only":false,"title":"校正聚点最优性的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\tDefine the residual term $\\tilde{r}^k := \\tilde{y}^{k} - \\operatorname{Prox}_{ \\tilde{g}(x^{k}, \\cdot)} \\left( \\tilde{y}^{k} -  \\nabla_y f(x^{k}, \\tilde{y}^{k}) \\right) $. By the first-order optimality condition for the proximal operator, $\\tilde{r}^k$ satisfies\n\t\\begin{equation}\\label{lem4_eq1}\n\t\t\\tilde{r}^k \\in \\nabla_y f(x^{k}, \\tilde{y}^{k}) + \\partial_{y}\\tilde{g}(x^{k}, \\tilde{y}^{k} - \\tilde{r}^k).\n\t\\end{equation}\n\tThe sequence  $(x^{k_j},\\tilde{y}^{k_j})$ satisfies condition \\eqref{inexacty_y} with $k = k_j - 1$, which implies \n\t\\[\n\t\\left\\| \\tilde{r}^{k_j} \\right\\| \\le \\frac{c_{\\tilde{y}} }{p_{k_j}} \\left\\|(x^{k_j},y^{k_j})-(x^{k_j-1},\\tilde{y}^{k_j-1})\\right\\|.\n\t\\]\n\tFrom Lemma \\ref{lem3}, we have $\\lim_{k_j \\to\\infty} \\|(x^{k_j},y^{k_j})-(x^{k_j-1},\\tilde{y}^{k_j-1}) \\|=0$. As $p_{k_j} \\ge p_0$, the above inequality ensures that $\\lim_{k_j \\to\\infty} \\|\\tilde{r}^{k_j} \\|=0$.\n\t\n\tTaking the limit as  $k_j \\rightarrow \\infty$ in \\eqref{lem4_eq1} with $k = k_j$, by the continuity of $ \\nabla_y f$ and the outer semicontinuity of $\\partial_{y}\\tilde{g}$, it follows that\n\t\\[\n\t0  \\in \\nabla_y f(\\bar{x},\\bar{y}) + \\partial_{y}\\tilde{g}(\\bar{x},\\bar{y}).\n\t\\] \n\tSince the lower-level problem is convex in $y$, this condition\n\timplies $\\bar{y} \\in S(\\bar{x})$.\n\\end{proof}","start":72472,"end":73774}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":["lem4_eq1"],"declared_external":false,"source_only":false,"title":"校正残差的近端最优性","source_units":[{"raw_tex":"\\begin{proof}\n\tDefine the residual term $\\tilde{r}^k := \\tilde{y}^{k} - \\operatorname{Prox}_{ \\tilde{g}(x^{k}, \\cdot)} \\left( \\tilde{y}^{k} -  \\nabla_y f(x^{k}, \\tilde{y}^{k}) \\right) $. By the first-order optimality condition for the proximal operator, $\\tilde{r}^k$ satisfies\n\t\\begin{equation}\\label{lem4_eq1}\n\t\t\\tilde{r}^k \\in \\nabla_y f(x^{k}, \\tilde{y}^{k}) + \\partial_{y}\\tilde{g}(x^{k}, \\tilde{y}^{k} - \\tilde{r}^k).\n\t\\end{equation}\n","start":72472,"end":72912}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"校正残差趋于零","source_units":[{"raw_tex":"\tThe sequence  $(x^{k_j},\\tilde{y}^{k_j})$ satisfies condition \\eqref{inexacty_y} with $k = k_j - 1$, which implies \n\t\\[\n\t\\left\\| \\tilde{r}^{k_j} \\right\\| \\le \\frac{c_{\\tilde{y}} }{p_{k_j}} \\left\\|(x^{k_j},y^{k_j})-(x^{k_j-1},\\tilde{y}^{k_j-1})\\right\\|.\n\t\\]\n\tFrom Lemma \\ref{lem3}, we have $\\lim_{k_j \\to\\infty} \\|(x^{k_j},y^{k_j})-(x^{k_j-1},\\tilde{y}^{k_j-1}) \\|=0$. As $p_{k_j} \\ge p_0$, the above inequality ensures that $\\lim_{k_j \\to\\infty} \\|\\tilde{r}^{k_j} \\|=0$.\n\t\n","start":72912,"end":73386}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"极限包含式与凸最优性","source_units":[{"raw_tex":"\tTaking the limit as  $k_j \\rightarrow \\infty$ in \\eqref{lem4_eq1} with $k = k_j$, by the continuity of $ \\nabla_y f$ and the outer semicontinuity of $\\partial_{y}\\tilde{g}$, it follows that\n\t\\[\n\t0  \\in \\nabla_y f(\\bar{x},\\bar{y}) + \\partial_{y}\\tilde{g}(\\bar{x},\\bar{y}).\n\t\\] \n\tSince the lower-level problem is convex in $y$, this condition\n\timplies $\\bar{y} \\in S(\\bar{x})$.\n\\end{proof}","start":73386,"end":73774}]},{"id":"S062","semantic_type":"proposition","kind":"proposition","labels":["prop0"],"declared_external":false,"source_only":false,"title":"有界罚参数下的聚点可行性","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop0}\n\tLet $\\epsilon > 0$. If the sequence $\\{p_k\\}$ is bounded, then any accumulation point $(\\bar{x}, \\bar{y})$ of the iterate sequence $\\{(x^k, {y}^k)\\}$ satisfies\n\t\\[\n\t\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon.\n\t\\]\n\\end{proposition}","start":73976,"end":74265}]},{"id":"S063","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"聚点可行性的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\t\n\tLet $(\\bar{x}, \\bar{y})$ be an accumulation point of $\\{(x^k, {y}^k)\\}$, and let \\(\\{k_j\\}_{j \\in \\mathbb{N}} \\subseteq \\mathbb{N} \\) be an index subsequence such that $(x^{k_j+1},y^{k_j+1})\\to (\\bar{x},\\bar{y})$ as $j \\rightarrow \\infty$. By Lemma \\ref{lem3}, we have  $\\lim_{k_j \\to\\infty} \\|(x^{k_j+1},y^{k_j+1})-(x^{k_j},\\tilde{y}^{k_j}) \\|=0$, which implies that $(x^{k_j},\\tilde{y}^{k_j})$ also converges to $ (\\bar{x},\\bar{y})$ .\n\t\n\tSince the sequence $\\{p_k\\}$ is bounded and non-decreasing by construction in AGILS (\\cref{alg1}), there exists $k_0 >0$ such that $p_k = p_{k_0}$ for all $k \\ge k_0$. By passing to a subsequence if necessary, we can assume that $k_j - 1\\ge k_0$ for all $j \\in \\mathbb{N}$.\t\n\t\n\tAccording to the update rule of $p_k$ in AGILS (\\cref{alg1}), the condition $p_{k_j - 1} = p_{k_j}$ implies that one of the following two cases must occur for each \\(  k = k_j -1 \\):\n\t\n\tCase I: $\\left\\|(x^{k_j},y^{k_j})-(x^{k_j-1}, \\tilde{y}^{k_j-1})\\right\\| \\ge c_p \\min\\left\\{1/p_{k_0},{t^{k_j}}\\right\\}$ and $\\tilde{y}^{k_j} = y^{k_j}$.\n\t\n\tCase II: $\\left\\|(x^{k_j},y^{k_j})-(x^{k_j-1}, \\tilde{y}^{k_j-1})\\right\\| < c_p \\min\\left\\{1/p_{k_0},{t^{k_j}}\\right\\}$ and the feasibility correction procedure is applied to generate $\\tilde{y}^{k_j}$ satisfying condition \\eqref{inexacty_y}.\n\t\n\tAt least one of these two cases must hold for an infinite number of indices in the subsequence \t$\\{ k_j  - 1\\}$. We analyze them separately.\n\t\n\tCase 1:  Suppose Case I occurs for an infinite subsequence of indices. By passing to a further subsequence, we assume it holds for all $j \\in \\mathbb{N}$.\n\t\n\tThe condition $\\tilde{y}^{k_j} = y^{k_j}$ combined with $(x^{k_j},\\tilde{y}^{k_j})\\to (\\bar{x},\\bar{y})$ implies that $(x^{k_j}, {y}^{k_j})$\\\\\n\t$ \\to (\\bar{x},\\bar{y})$. \n\tFrom Lemma \\ref{lem3}, we know $\\lim_{k_j \\to\\infty} \\|(x^{k_j},y^{k_j})-(x^{k_j-1},\\tilde{y}^{k_j-1}) \\|=0$. For the condition of Case I to hold, we must therefore have  $t^{k_j} \\rightarrow 0$. This implies\n\t\\begin{equation*}\n\t\t\\limsup_{j \\rightarrow \\infty}\t\\left\\{ \\varphi(x^{k_j}, y^{k_j})-\\varphi(x^{k_j},\\theta^{k_j})-\\frac{1}{2\\gamma}\\left\\| \\theta^{k_j} - y^{k_j} \\right\\|^2-\\epsilon \\right\\} \\le 0.\n\t\\end{equation*}\n\tSince $\\zeta_{k} \\rightarrow 0$ and $\\mathcal{G}( \\theta^{k_j}, x^{k_j}, y^{k_j}) \\le \\zeta_{k_j}$, Lemma \\ref{error_bound} ensures that $\\| \\theta^{k_j} - \\theta^*_{\\gamma}( x^{k_j}, y^{k_j} )\\|\\rightarrow 0$.\n\tBy the continuity of $\\theta^*_{\\gamma}(x,y)$ (Proposition \\ref{partial_grad_v}), it follows that $\\theta^{k_j} \\rightarrow \\theta^*_{\\gamma}(\\bar{x}, \\bar{y}) $.\n\tThe continuity of $\\varphi$ then implies that $\\varphi(x^{k_j}, \\theta^{k_j})+\\frac{1}{2\\gamma}\\|\\theta^{k_j} - y^{k_j}\\|^2 \\rightarrow v_\\gamma(\\bar{x},\\bar{y})$. Then taking the limit in the inequality above as  $j \\rightarrow \\infty$ yields $\\varphi(\\bar{x},\\bar{y})-v_{\\gamma}(\\bar{x},\\bar{y})-\\epsilon \\le 0$. \n\t\n\tCase 2: \n\tNow, suppose Case II occurs for an infinite subsequence of indices. Again, we assume it holds for all $j \\in \\mathbb{N}$.\n\t\n\tIn this scenario, the feasibility correction procedure is applied at each iteration $k = k_j -1$ to generate $\\tilde{y}^{k_j}$ satisfying condition \\eqref{inexacty_y}. Since $(x^{k_j},\\tilde{y}^{k_j}) \\rightarrow (\\bar{x},\\bar{y}) $ and satisfies condition \\eqref{inexacty_y}, it follows from Lemma \\ref{lem4} that $\\bar{y} \\in S(\\bar{x})$. For any such point, $\\theta^*_{\\gamma}(\\bar{x}, \\bar{y}) = \\bar{y} $ (see, e.g., \\cite[Theorem 10.7]{beck2017first}), and therefore \\( \\varphi(\\bar{x}, \\bar{y}) - v_\\gamma(\\bar{x}, \\bar{y}) = 0 \\).  Since $ \\epsilon > 0$, the constraint $\t\t\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon$ is satisfied.\n\t\n\tSince any accumulation point must be the limit of a subsequence that falls into one of these two cases, we conclude that $\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon$.\n\\end{proof}","start":74266,"end":78191}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"对齐聚点并固定罚参数","source_units":[{"raw_tex":"\\begin{proof}\n\t\n\tLet $(\\bar{x}, \\bar{y})$ be an accumulation point of $\\{(x^k, {y}^k)\\}$, and let \\(\\{k_j\\}_{j \\in \\mathbb{N}} \\subseteq \\mathbb{N} \\) be an index subsequence such that $(x^{k_j+1},y^{k_j+1})\\to (\\bar{x},\\bar{y})$ as $j \\rightarrow \\infty$. By Lemma \\ref{lem3}, we have  $\\lim_{k_j \\to\\infty} \\|(x^{k_j+1},y^{k_j+1})-(x^{k_j},\\tilde{y}^{k_j}) \\|=0$, which implies that $(x^{k_j},\\tilde{y}^{k_j})$ also converges to $ (\\bar{x},\\bar{y})$ .\n\t\n\tSince the sequence $\\{p_k\\}$ is bounded and non-decreasing by construction in AGILS (\\cref{alg1}), there exists $k_0 >0$ such that $p_k = p_{k_0}$ for all $k \\ge k_0$. By passing to a subsequence if necessary, we can assume that $k_j - 1\\ge k_0$ for all $j \\in \\mathbb{N}$.\t\n","start":74266,"end":74998}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"保持罚参数的两类分支","source_units":[{"raw_tex":"\t\n\tAccording to the update rule of $p_k$ in AGILS (\\cref{alg1}), the condition $p_{k_j - 1} = p_{k_j}$ implies that one of the following two cases must occur for each \\(  k = k_j -1 \\):\n\t\n\tCase I: $\\left\\|(x^{k_j},y^{k_j})-(x^{k_j-1}, \\tilde{y}^{k_j-1})\\right\\| \\ge c_p \\min\\left\\{1/p_{k_0},{t^{k_j}}\\right\\}$ and $\\tilde{y}^{k_j} = y^{k_j}$.\n\t\n\tCase II: $\\left\\|(x^{k_j},y^{k_j})-(x^{k_j-1}, \\tilde{y}^{k_j-1})\\right\\| < c_p \\min\\left\\{1/p_{k_0},{t^{k_j}}\\right\\}$ and the feasibility correction procedure is applied to generate $\\tilde{y}^{k_j}$ satisfying condition \\eqref{inexacty_y}.\n\t\n\tAt least one of these two cases must hold for an infinite number of indices in the subsequence \t$\\{ k_j  - 1\\}$. We analyze them separately.\n\t\n","start":74998,"end":75733}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"常规分支的违约量消失","source_units":[{"raw_tex":"\tCase 1:  Suppose Case I occurs for an infinite subsequence of indices. By passing to a further subsequence, we assume it holds for all $j \\in \\mathbb{N}$.\n\t\n\tThe condition $\\tilde{y}^{k_j} = y^{k_j}$ combined with $(x^{k_j},\\tilde{y}^{k_j})\\to (\\bar{x},\\bar{y})$ implies that $(x^{k_j}, {y}^{k_j})$\\\\\n\t$ \\to (\\bar{x},\\bar{y})$. \n\tFrom Lemma \\ref{lem3}, we know $\\lim_{k_j \\to\\infty} \\|(x^{k_j},y^{k_j})-(x^{k_j-1},\\tilde{y}^{k_j-1}) \\|=0$. For the condition of Case I to hold, we must therefore have  $t^{k_j} \\rightarrow 0$. This implies\n\t\\begin{equation*}\n\t\t\\limsup_{j \\rightarrow \\infty}\t\\left\\{ \\varphi(x^{k_j}, y^{k_j})-\\varphi(x^{k_j},\\theta^{k_j})-\\frac{1}{2\\gamma}\\left\\| \\theta^{k_j} - y^{k_j} \\right\\|^2-\\epsilon \\right\\} \\le 0.\n\t\\end{equation*}\n","start":75733,"end":76490}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"近端候选恢复真实包络值","source_units":[{"raw_tex":"\tSince $\\zeta_{k} \\rightarrow 0$ and $\\mathcal{G}( \\theta^{k_j}, x^{k_j}, y^{k_j}) \\le \\zeta_{k_j}$, Lemma \\ref{error_bound} ensures that $\\| \\theta^{k_j} - \\theta^*_{\\gamma}( x^{k_j}, y^{k_j} )\\|\\rightarrow 0$.\n\tBy the continuity of $\\theta^*_{\\gamma}(x,y)$ (Proposition \\ref{partial_grad_v}), it follows that $\\theta^{k_j} \\rightarrow \\theta^*_{\\gamma}(\\bar{x}, \\bar{y}) $.\n\tThe continuity of $\\varphi$ then implies that $\\varphi(x^{k_j}, \\theta^{k_j})+\\frac{1}{2\\gamma}\\|\\theta^{k_j} - y^{k_j}\\|^2 \\rightarrow v_\\gamma(\\bar{x},\\bar{y})$. Then taking the limit in the inequality above as  $j \\rightarrow \\infty$ yields $\\varphi(\\bar{x},\\bar{y})-v_{\\gamma}(\\bar{x},\\bar{y})-\\epsilon \\le 0$. \n\t\n","start":76490,"end":77185}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"校正分支的零包络间隙","source_units":[{"raw_tex":"\tCase 2: \n\tNow, suppose Case II occurs for an infinite subsequence of indices. Again, we assume it holds for all $j \\in \\mathbb{N}$.\n\t\n\tIn this scenario, the feasibility correction procedure is applied at each iteration $k = k_j -1$ to generate $\\tilde{y}^{k_j}$ satisfying condition \\eqref{inexacty_y}. Since $(x^{k_j},\\tilde{y}^{k_j}) \\rightarrow (\\bar{x},\\bar{y}) $ and satisfies condition \\eqref{inexacty_y}, it follows from Lemma \\ref{lem4} that $\\bar{y} \\in S(\\bar{x})$. For any such point, $\\theta^*_{\\gamma}(\\bar{x}, \\bar{y}) = \\bar{y} $ (see, e.g., \\cite[Theorem 10.7]{beck2017first}), and therefore \\( \\varphi(\\bar{x}, \\bar{y}) - v_\\gamma(\\bar{x}, \\bar{y}) = 0 \\).  Since $ \\epsilon > 0$, the constraint $\t\t\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon$ is satisfied.\n\t\n\tSince any accumulation point must be the limit of a subsequence that falls into one of these two cases, we conclude that $\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon$.\n\\end{proof}","start":77185,"end":78191}]},{"id":"S069","semantic_type":"theorem","kind":"theorem","labels":["th1"],"declared_external":false,"source_only":false,"title":"聚点满足更紧松弛问题KKT条件","source_units":[{"raw_tex":"\\begin{theorem}\\label{th1}\n\tLet $\\epsilon > 0$. If the sequence $\\{p_k\\}$ is bounded, then any accumulation point $(\\bar{x}, \\bar{y})$ of $\\{(x^k,y^k)\\}$ is a KKT point of $({\\rm VP})_\\gamma^{\\bar{\\epsilon}}$ for some  $\\bar{\\epsilon}\\le\\epsilon$. \n\\end{theorem}","start":92183,"end":92445}]},{"id":"S070","semantic_type":"proof","kind":"proof","labels":["th1_eq2","th1_eq4","th1_eq5","th1_eq6"],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\tThe proof builds upon techniques used in the proof of Theorem 1 in \\cite{ye2023}. \n\tLet $(\\bar{x}, \\bar{y})$ be an accumulation point of the sequence $\\{(x^k,y^k)\\}$, and let $\\{(x^{k_j+1},y^{k_j+1})\\}$ be a convergent subsequence such that $(x^{k_j+1},y^{k_j+1}) \\rightarrow (\\bar{x}, \\bar{y})$ as $k_j \\rightarrow \\infty$. \n\tBy Lemma \\ref{lem3}, we have  $\\lim_{k_j \\to\\infty} \\|(x^{k_j+1},y^{k_j+1})-(x^{k_j},\\tilde{y}^{k_j}) \\|=0$, which implies that $(x^{k_j},\\tilde{y}^{k_j}) \\to (\\bar{x},\\bar{y})$ .\n\tSince $\\{p_k\\}$ is bounded, there exists $k_0 > 0$ such that $p_k = \\bar{p} := p_{k_0}$ for any $k \\ge k_0$. \n\t\n\t\n\tFrom the update rules \\eqref{update_x} and \\eqref{update_y}, and the expressions for $d_x^k$ and $d_y^k$, we have    \n\t\\begin{equation}\\label{th1_eq2}\n\t\t\\begin{aligned}\n\t\t\t\\xi_x^k & \\in \\nabla_x F(x^{k+1},y^{k+1})+p_k \\left(\\nabla_x \\varphi(x^{k+1},y^{k+1}) - \\nabla_x v_\\gamma(x^{k+1},y^{k+1})\\right) +\\mathcal{N}_{X}(x^{k+1}),\\\\\n\t\t\t\\xi_y^k & \\in \\nabla_y F(x^{k},y^{k+1})+p_k \\left(\\partial_y \\varphi(x^{k},y^{k+1}) - \\nabla_y v_\\gamma(x^{k},y^{k+1})\\right) +\\mathcal{N}_{Y}(y^{k+1}),\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere $\\xi_x^k \\in \\Re^n$ and $\\xi_y^k \\in \\Re^m$ are given by\n\t\\begin{equation}\\label{th1_eq4}\n\t\t\\begin{aligned}\n\t\t\t\\xi_x^k &:=  p_k \\left(\\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - d_x^k \\right)  - \\frac{p_k}{\\alpha_k}\\left(x^{k+1}-x^k\\right), \\\\\n\t\t\t\\xi_y^k &:=  p_k \\left( \\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, y^{k+1}\\right) - d_y^k \\right)-\\frac{p_k}{\\beta_k}\\left(y^{k+1}-\\tilde{y}^k\\right).\n\t\t\\end{aligned} \n\t\\end{equation} \n\tWe now show that $\\xi_x^{k_j}, \\xi_y^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$. First, we have \n\t\\begin{equation}\n\t\t\\begin{aligned}\n\t\t\t& \\| \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - d_x^k\\| \\\\ \\le \\, & \\|  \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k}, y^{k+1} \\right) \\| + \\| \\nabla_x \\tilde{\\psi}_{p_{k}} \\left(x^{k}, y^{k+1} \\right) - d_x^k \\| \\\\\n\t\t\t\\le \\, & \\|  \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k}, y^{k+1} \\right) \\| + (L_{f_x} + L_{g_2})C_\\eta \\zeta_{k}, \n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere the last inequality follows from \\eqref{lem1_eq10}. Since $p_k = \\bar{p}$ for all $k \\ge k_0$, and assuming that $\\nabla_{x} F, \\nabla_{x} f$ and $\\nabla_{x} g$ are continuous on $X \\times Y$, and $\\nabla_{x} v_\\gamma$ is continuous on $X \\times Y$ (Proposition \\ref{partial_grad_v}), we conclude that $ \\nabla_x \\tilde{\\psi}_{p_{k}} =  \\nabla_x \\tilde{\\psi}_{\\bar{p}}$ is continuous on $X \\times Y$. \n\tAs $\\lim_{k_j\\to\\infty} (x^{k_j+1},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},\\tilde{y}^{k_j}) = (\\bar{x}, \\bar{y})$ and $\\zeta_{k} \\rightarrow 0$, it follows that $\\nabla_x \\tilde{\\psi}_{p_{k_j}} (x^{k_j+1}, y^{k_j+1} ) - d_x^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$. Combining this with $ \\|x^{k_j+1}-x^{k_j} \\| \\to 0$ and $\\alpha_k \\ge \\underline{\\alpha} > 0$, we obtain from \\eqref{th1_eq4} that $\\xi_x^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$.\n\t\n\tA similar argument can be applied to $\\xi_y^k$. From \\eqref{lem1_eq11}, we have \n\t\\begin{equation}\\label{th1_eq5}\n\t\t\\begin{aligned}\n\t\t\t& \\|\\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, y^{k+1}\\right) - d_y^k \\| \\\\ \\le \\, & \\| \\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, y^{k+1} \\right) - \\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, \\tilde{y}^k \\right) \\| + {C_\\eta}\\zeta_{k}/\\gamma.\n\t\t\\end{aligned}\n\t\\end{equation}\n\tSince $p_k = \\bar{p}$ for all $k \\ge k_0$, and under the assumption that $\\nabla_{y} F$ and $\\nabla_{y} f$ are continuous on $X \\times Y$, and $\\nabla_{y} v_\\gamma$ is continuous on $X \\times Y$ (Proposition \\ref{partial_grad_v}), we have that $ \\nabla_y (\\tilde{\\psi}_{p_{k}} - g) = \\nabla_y (\\tilde{\\psi}_{\\bar{p}} - g)$ is continuous on $X \\times Y$. As $\\lim_{k_j\\to\\infty} (x^{k_j},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},\\tilde{y}^{k_j}) = (\\bar{x}, \\bar{y})$ and $\\zeta_{k} \\rightarrow 0$, it follows that $\\nabla_y (\\tilde{\\psi}_{p_{k_j}} - g)(x^{k_j}, y^{k_j+1} ) - d_y^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$. Combining this with $ \\|y^{k_j+1}-\\tilde{y}^{k_j} \\| \\to 0$ and $\\beta_k \\ge \\underline{\\beta} > 0$, we conclude from \\eqref{th1_eq5} that $\\xi_y^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$.\n\t\n\tNow, taking the limit as $k = k_j \\rightarrow \\infty$ in \\eqref{th1_eq2}, since $\\xi_x^{k_j}, \\xi_y^{k_j} \\rightarrow 0$, $p_k = \\bar{p}$ for all $k \\ge k_0$, $\\lim_{k_j\\to\\infty} (x^{k_j+1},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},\\tilde{y}^{k_j}) = (\\bar{x}, \\bar{y})$, $\\nabla F$, $\\nabla v_\\gamma$, and $\\nabla_x \\varphi$ are continuous on $X \\times Y$, $\\partial_y \\varphi$ and $\\mathcal{N}_{X\\times Y}$ are outer semicontinuous on $X \\times Y$, and $\\partial \\varphi = \\{\\nabla_x \\varphi\\} \\times \\partial_y \\varphi$ on $X \\times Y$, we obtain \n\t\\begin{equation}\\label{th1_eq6}\n\t\t0\\in\\nabla F(\\bar{x}, \\bar{y})+\\bar{p}\\left(\\partial \\varphi(\\bar{x},\\bar{y})-\\nabla v_\\gamma(\\bar{x}, \\bar{y})\\right)+\\mathcal{N}_{X\\times Y}(\\bar{x}, \\bar{y}).\n\t\\end{equation}\n\t\n\tFinally, from Proposition \\ref{prop0}, we know that the limit point $(\\bar{x}, \\bar{y})$ satisfies $\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon $. Let $\\bar{\\epsilon}:= \\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\in [0, \\epsilon]$. Combining this with \\eqref{th1_eq6}, we have\n\t\\begin{equation}\n\t\t\\left\\{\n\t\t\\begin{aligned}\n\t\t\t&0\\in\\nabla F(\\bar{x}, \\bar{y}) + \\bar{p}\\left( \\partial \\varphi(\\bar{x}, \\bar{y}) - \\nabla v_\\gamma(\\bar{x}, \\bar{y})\\right)+\\mathcal{N}_{X\\times Y}(\\bar{x}, \\bar{y}),\\\\\n\t\t\t&\\varphi(\\bar{x}, \\bar{y}) - v_{\\gamma}(\\bar{x}, \\bar{y})- \\bar{\\epsilon} \\le 0, \\qquad  \\bar{p}\\left( \\varphi(\\bar{x}, \\bar{y}) - v_{\\gamma}(\\bar{x}, \\bar{y})- \\bar{\\epsilon} \\right)=0,\n\t\t\\end{aligned}\\right.\n\t\\end{equation}\n\twhich implies that $(\\bar{x},\\bar{y})$ is a KKT point of problem $({\\rm VP})_\\gamma^{\\bar{\\epsilon}}$ with $\\bar{\\epsilon}\\le \\epsilon$.\n\\end{proof}","start":92446,"end":98589}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"选择子序列与恒定罚参数","source_units":[{"raw_tex":"\\begin{proof}\n\tThe proof builds upon techniques used in the proof of Theorem 1 in \\cite{ye2023}. \n\tLet $(\\bar{x}, \\bar{y})$ be an accumulation point of the sequence $\\{(x^k,y^k)\\}$, and let $\\{(x^{k_j+1},y^{k_j+1})\\}$ be a convergent subsequence such that $(x^{k_j+1},y^{k_j+1}) \\rightarrow (\\bar{x}, \\bar{y})$ as $k_j \\rightarrow \\infty$. \n\tBy Lemma \\ref{lem3}, we have  $\\lim_{k_j \\to\\infty} \\|(x^{k_j+1},y^{k_j+1})-(x^{k_j},\\tilde{y}^{k_j}) \\|=0$, which implies that $(x^{k_j},\\tilde{y}^{k_j}) \\to (\\bar{x},\\bar{y})$ .\n\tSince $\\{p_k\\}$ is bounded, there exists $k_0 > 0$ such that $p_k = \\bar{p} := p_{k_0}$ for any $k \\ge k_0$. \n","start":92446,"end":93079}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":["th1_eq2","th1_eq4"],"declared_external":false,"source_only":false,"title":"两块最优性条件与扰动定义","source_units":[{"raw_tex":"\t\n\t\n\tFrom the update rules \\eqref{update_x} and \\eqref{update_y}, and the expressions for $d_x^k$ and $d_y^k$, we have    \n\t\\begin{equation}\\label{th1_eq2}\n\t\t\\begin{aligned}\n\t\t\t\\xi_x^k & \\in \\nabla_x F(x^{k+1},y^{k+1})+p_k \\left(\\nabla_x \\varphi(x^{k+1},y^{k+1}) - \\nabla_x v_\\gamma(x^{k+1},y^{k+1})\\right) +\\mathcal{N}_{X}(x^{k+1}),\\\\\n\t\t\t\\xi_y^k & \\in \\nabla_y F(x^{k},y^{k+1})+p_k \\left(\\partial_y \\varphi(x^{k},y^{k+1}) - \\nabla_y v_\\gamma(x^{k},y^{k+1})\\right) +\\mathcal{N}_{Y}(y^{k+1}),\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere $\\xi_x^k \\in \\Re^n$ and $\\xi_y^k \\in \\Re^m$ are given by\n\t\\begin{equation}\\label{th1_eq4}\n\t\t\\begin{aligned}\n\t\t\t\\xi_x^k &:=  p_k \\left(\\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - d_x^k \\right)  - \\frac{p_k}{\\alpha_k}\\left(x^{k+1}-x^k\\right), \\\\\n\t\t\t\\xi_y^k &:=  p_k \\left( \\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, y^{k+1}\\right) - d_y^k \\right)-\\frac{p_k}{\\beta_k}\\left(y^{k+1}-\\tilde{y}^k\\right).\n\t\t\\end{aligned} \n\t\\end{equation} \n","start":93079,"end":94071}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"x块最优性扰动消失","source_units":[{"raw_tex":"\tWe now show that $\\xi_x^{k_j}, \\xi_y^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$. First, we have \n\t\\begin{equation}\n\t\t\\begin{aligned}\n\t\t\t& \\| \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - d_x^k\\| \\\\ \\le \\, & \\|  \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k}, y^{k+1} \\right) \\| + \\| \\nabla_x \\tilde{\\psi}_{p_{k}} \\left(x^{k}, y^{k+1} \\right) - d_x^k \\| \\\\\n\t\t\t\\le \\, & \\|  \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k}, y^{k+1} \\right) \\| + (L_{f_x} + L_{g_2})C_\\eta \\zeta_{k}, \n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere the last inequality follows from \\eqref{lem1_eq10}. Since $p_k = \\bar{p}$ for all $k \\ge k_0$, and assuming that $\\nabla_{x} F, \\nabla_{x} f$ and $\\nabla_{x} g$ are continuous on $X \\times Y$, and $\\nabla_{x} v_\\gamma$ is continuous on $X \\times Y$ (Proposition \\ref{partial_grad_v}), we conclude that $ \\nabla_x \\tilde{\\psi}_{p_{k}} =  \\nabla_x \\tilde{\\psi}_{\\bar{p}}$ is continuous on $X \\times Y$. \n\tAs $\\lim_{k_j\\to\\infty} (x^{k_j+1},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},\\tilde{y}^{k_j}) = (\\bar{x}, \\bar{y})$ and $\\zeta_{k} \\rightarrow 0$, it follows that $\\nabla_x \\tilde{\\psi}_{p_{k_j}} (x^{k_j+1}, y^{k_j+1} ) - d_x^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$. Combining this with $ \\|x^{k_j+1}-x^{k_j} \\| \\to 0$ and $\\alpha_k \\ge \\underline{\\alpha} > 0$, we obtain from \\eqref{th1_eq4} that $\\xi_x^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$.\n","start":94071,"end":95591}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":["th1_eq5"],"declared_external":false,"source_only":false,"title":"y块最优性扰动消失","source_units":[{"raw_tex":"\t\n\tA similar argument can be applied to $\\xi_y^k$. From \\eqref{lem1_eq11}, we have \n\t\\begin{equation}\\label{th1_eq5}\n\t\t\\begin{aligned}\n\t\t\t& \\|\\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, y^{k+1}\\right) - d_y^k \\| \\\\ \\le \\, & \\| \\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, y^{k+1} \\right) - \\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, \\tilde{y}^k \\right) \\| + {C_\\eta}\\zeta_{k}/\\gamma.\n\t\t\\end{aligned}\n\t\\end{equation}\n\tSince $p_k = \\bar{p}$ for all $k \\ge k_0$, and under the assumption that $\\nabla_{y} F$ and $\\nabla_{y} f$ are continuous on $X \\times Y$, and $\\nabla_{y} v_\\gamma$ is continuous on $X \\times Y$ (Proposition \\ref{partial_grad_v}), we have that $ \\nabla_y (\\tilde{\\psi}_{p_{k}} - g) = \\nabla_y (\\tilde{\\psi}_{\\bar{p}} - g)$ is continuous on $X \\times Y$. As $\\lim_{k_j\\to\\infty} (x^{k_j},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},\\tilde{y}^{k_j}) = (\\bar{x}, \\bar{y})$ and $\\zeta_{k} \\rightarrow 0$, it follows that $\\nabla_y (\\tilde{\\psi}_{p_{k_j}} - g)(x^{k_j}, y^{k_j+1} ) - d_y^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$. Combining this with $ \\|y^{k_j+1}-\\tilde{y}^{k_j} \\| \\to 0$ and $\\beta_k \\ge \\underline{\\beta} > 0$, we conclude from \\eqref{th1_eq5} that $\\xi_y^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$.\n","start":95591,"end":96865}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":["th1_eq6"],"declared_external":false,"source_only":false,"title":"传递到联合驻点包含式","source_units":[{"raw_tex":"\t\n\tNow, taking the limit as $k = k_j \\rightarrow \\infty$ in \\eqref{th1_eq2}, since $\\xi_x^{k_j}, \\xi_y^{k_j} \\rightarrow 0$, $p_k = \\bar{p}$ for all $k \\ge k_0$, $\\lim_{k_j\\to\\infty} (x^{k_j+1},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},\\tilde{y}^{k_j}) = (\\bar{x}, \\bar{y})$, $\\nabla F$, $\\nabla v_\\gamma$, and $\\nabla_x \\varphi$ are continuous on $X \\times Y$, $\\partial_y \\varphi$ and $\\mathcal{N}_{X\\times Y}$ are outer semicontinuous on $X \\times Y$, and $\\partial \\varphi = \\{\\nabla_x \\varphi\\} \\times \\partial_y \\varphi$ on $X \\times Y$, we obtain \n\t\\begin{equation}\\label{th1_eq6}\n\t\t0\\in\\nabla F(\\bar{x}, \\bar{y})+\\bar{p}\\left(\\partial \\varphi(\\bar{x},\\bar{y})-\\nabla v_\\gamma(\\bar{x}, \\bar{y})\\right)+\\mathcal{N}_{X\\times Y}(\\bar{x}, \\bar{y}).\n\t\\end{equation}\n","start":96865,"end":97675}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"实际间隙定义松弛与互补","source_units":[{"raw_tex":"\t\n\tFinally, from Proposition \\ref{prop0}, we know that the limit point $(\\bar{x}, \\bar{y})$ satisfies $\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon $. Let $\\bar{\\epsilon}:= \\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\in [0, \\epsilon]$. Combining this with \\eqref{th1_eq6}, we have\n\t\\begin{equation}\n\t\t\\left\\{\n\t\t\\begin{aligned}\n\t\t\t&0\\in\\nabla F(\\bar{x}, \\bar{y}) + \\bar{p}\\left( \\partial \\varphi(\\bar{x}, \\bar{y}) - \\nabla v_\\gamma(\\bar{x}, \\bar{y})\\right)+\\mathcal{N}_{X\\times Y}(\\bar{x}, \\bar{y}),\\\\\n\t\t\t&\\varphi(\\bar{x}, \\bar{y}) - v_{\\gamma}(\\bar{x}, \\bar{y})- \\bar{\\epsilon} \\le 0, \\qquad  \\bar{p}\\left( \\varphi(\\bar{x}, \\bar{y}) - v_{\\gamma}(\\bar{x}, \\bar{y})- \\bar{\\epsilon} \\right)=0,\n\t\t\\end{aligned}\\right.\n\t\\end{equation}\n\twhich implies that $(\\bar{x},\\bar{y})$ is a KKT point of problem $({\\rm VP})_\\gamma^{\\bar{\\epsilon}}$ with $\\bar{\\epsilon}\\le \\epsilon$.\n\\end{proof}","start":97675,"end":98589}]},{"id":"S077","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"包络正则性的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\tWe begin by showing that $v_{\\gamma}(x,y) $ is continuously differentiable on $X \\times \\mathbb{R}^m$. By Assumption \\ref{asup2}, the function $f(x, \\theta) + g(x, \\theta) + \\frac{1}{2\\gamma}\\|\\theta-y\\|^2$ is level-bounded in $\\theta$ locally uniformly for any $(\\bar{x}, \\bar{y}) \\in X \\times \\mathbb{R}^m $. Specifically, for any $c \\in \\mathbb{R}$, there exist a compact set $D$ and a neighborhood $Z$ of $(\\bar{x}, \\bar{y})$ such that the level set $\\{ \\theta \\in \\mathbb{R}^m \\mid f(x, \\theta) + g(x, \\theta) + \\frac{1}{2\\gamma}\\|\\theta-y\\|^2 \\le c\\}$ is contained in $D$ for all $(x,y) \\in Z$. Using Corollary 4.3(ii) from \\cite{mordukhovich2018variational}, it follows that $v_{\\gamma}(x,y) $ is locally Lipschitz continuous on $X \\times \\mathbb{R}^m$. Furthermore, Theorem 1.22, Theorem 4.1(ii) and Theorem 4.17 in \\cite{mordukhovich2018variational} imply that $v_\\gamma({x},{y})$ is differentiable on $X \\times \\mathbb{R}^m$ with its gradient given as in \\eqref{valuefinc}. The partial weak convexity property of $v_\\gamma({x},{y})$ can be established using arguments similar to those in the proof of Theorem 2 in \\cite{gao2024}.\n\t\n\tNext, we show that $ \\nabla v_\\gamma({x},{y})$ is continuous on $X \\times \\mathbb{R}^m$ by showing the continuity of $\\theta_{\\gamma}^*(x,y)$ on $X \\times \\mathbb{R}^m$. Let $\\{(x_k, y_k)\\} \\subset X \\times \\mathbb{R}^m$ be any sequence such that $(x_k, y_k) \\rightarrow (\\bar{x}, \\bar{y})$. Since $f(x, \\theta) + g(x, \\theta) + \\frac{1}{2\\gamma}\\|\\theta-y\\|^2$ is level-bounded in $\\theta$ locally uniformly for $(\\bar{x}, \\bar{y}) \\in X \\times \\mathbb{R}^m $, the sequence $\\{\\theta^*_{\\gamma}(x_k,y_k)\\}$ is bounded. Let $\\bar{\\theta}$ be an accumulation point of $\\{\\theta^*_{\\gamma}(x_k,y_k)\\}$. For any $\\theta \\in Y$, we have $f(x_k, \\theta^*_{\\gamma}(x_k,y_k)) + g(x_k, \\theta^*_{\\gamma}(x_k,y_k)) + \\frac{1}{2\\gamma}\\|\\theta^*_{\\gamma}(x_k,y_k)-y_k\\|^2 \\le f(x_k, \\theta) + g(x_k, \\theta) + \\frac{1}{2\\gamma}\\|\\theta-y_k\\|^2$. Taking the limit as $k \\rightarrow \\infty$, and using the continuity of $f$ and $g$, we obtain that for any $\\theta \\in Y$, $f(\\bar{x}, \\bar{\\theta}) + g(\\bar{x}, \\bar{\\theta}) + \\frac{1}{2\\gamma}\\|\\bar{\\theta}-\\bar{y}\\|^2 \\le f(\\bar{x}, \\theta) + g(\\bar{x}, \\theta) + \\frac{1}{2\\gamma}\\|\\theta-\\bar{y}\\|^2$. This implies $\\bar{\\theta} = \\theta^*_{\\gamma}(\\bar{x}, \\bar{y})$. Therefore, $\\theta^*_{\\gamma}(x_k,y_k) \\rightarrow \\theta^*_{\\gamma}(\\bar{x}, \\bar{y})$, establishing the continuity of $\\theta_{\\gamma}^*(x,y)$ on $X \\times \\mathbb{R}^m$.\n\\end{proof}\n","start":26638,"end":29193}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"边际函数敏感性与偏弱凸性","source_units":[{"raw_tex":"\\begin{proof}\n\tWe begin by showing that $v_{\\gamma}(x,y) $ is continuously differentiable on $X \\times \\mathbb{R}^m$. By Assumption \\ref{asup2}, the function $f(x, \\theta) + g(x, \\theta) + \\frac{1}{2\\gamma}\\|\\theta-y\\|^2$ is level-bounded in $\\theta$ locally uniformly for any $(\\bar{x}, \\bar{y}) \\in X \\times \\mathbb{R}^m $. Specifically, for any $c \\in \\mathbb{R}$, there exist a compact set $D$ and a neighborhood $Z$ of $(\\bar{x}, \\bar{y})$ such that the level set $\\{ \\theta \\in \\mathbb{R}^m \\mid f(x, \\theta) + g(x, \\theta) + \\frac{1}{2\\gamma}\\|\\theta-y\\|^2 \\le c\\}$ is contained in $D$ for all $(x,y) \\in Z$. Using Corollary 4.3(ii) from \\cite{mordukhovich2018variational}, it follows that $v_{\\gamma}(x,y) $ is locally Lipschitz continuous on $X \\times \\mathbb{R}^m$. Furthermore, Theorem 1.22, Theorem 4.1(ii) and Theorem 4.17 in \\cite{mordukhovich2018variational} imply that $v_\\gamma({x},{y})$ is differentiable on $X \\times \\mathbb{R}^m$ with its gradient given as in \\eqref{valuefinc}. The partial weak convexity property of $v_\\gamma({x},{y})$ can be established using arguments similar to those in the proof of Theorem 2 in \\cite{gao2024}.\n","start":26638,"end":27793}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"精确近端解的连续性","source_units":[{"raw_tex":"\t\n\tNext, we show that $ \\nabla v_\\gamma({x},{y})$ is continuous on $X \\times \\mathbb{R}^m$ by showing the continuity of $\\theta_{\\gamma}^*(x,y)$ on $X \\times \\mathbb{R}^m$. Let $\\{(x_k, y_k)\\} \\subset X \\times \\mathbb{R}^m$ be any sequence such that $(x_k, y_k) \\rightarrow (\\bar{x}, \\bar{y})$. Since $f(x, \\theta) + g(x, \\theta) + \\frac{1}{2\\gamma}\\|\\theta-y\\|^2$ is level-bounded in $\\theta$ locally uniformly for $(\\bar{x}, \\bar{y}) \\in X \\times \\mathbb{R}^m $, the sequence $\\{\\theta^*_{\\gamma}(x_k,y_k)\\}$ is bounded. Let $\\bar{\\theta}$ be an accumulation point of $\\{\\theta^*_{\\gamma}(x_k,y_k)\\}$. For any $\\theta \\in Y$, we have $f(x_k, \\theta^*_{\\gamma}(x_k,y_k)) + g(x_k, \\theta^*_{\\gamma}(x_k,y_k)) + \\frac{1}{2\\gamma}\\|\\theta^*_{\\gamma}(x_k,y_k)-y_k\\|^2 \\le f(x_k, \\theta) + g(x_k, \\theta) + \\frac{1}{2\\gamma}\\|\\theta-y_k\\|^2$. Taking the limit as $k \\rightarrow \\infty$, and using the continuity of $f$ and $g$, we obtain that for any $\\theta \\in Y$, $f(\\bar{x}, \\bar{\\theta}) + g(\\bar{x}, \\bar{\\theta}) + \\frac{1}{2\\gamma}\\|\\bar{\\theta}-\\bar{y}\\|^2 \\le f(\\bar{x}, \\theta) + g(\\bar{x}, \\theta) + \\frac{1}{2\\gamma}\\|\\theta-\\bar{y}\\|^2$. This implies $\\bar{\\theta} = \\theta^*_{\\gamma}(\\bar{x}, \\bar{y})$. Therefore, $\\theta^*_{\\gamma}(x_k,y_k) \\rightarrow \\theta^*_{\\gamma}(\\bar{x}, \\bar{y})$, establishing the continuity of $\\theta_{\\gamma}^*(x,y)$ on $X \\times \\mathbb{R}^m$.\n\\end{proof}\n","start":27793,"end":29193}]},{"id":"S080","semantic_type":"proof","kind":"proof","labels":["KL"],"declared_external":false,"source_only":true,"title":"校正近似下降的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\tAt a given iteration $k$, if $\\tilde{y}^{k+1} = y^{k+1}$, the inequality holds trivially.\n\t\n\tNow, consider the case where $\\tilde{y}^{k+1} \\neq y^{k+1}$. This implies that the feasibility correction procedure (Algorithm \\ref{alg2}) is applied at iteration $k$. Accordingly, the corrected iterate  $\\tilde{y}^{k+1}$ satisfies condition \\eqref{decrease_new_y}, and its corresponding $\\tilde{\\theta}^{k+1}$ satisfies condition \\eqref{inexacty3}. Condition \\eqref{inexacty3} implies that $\\mathcal{G}(\\hat{\\theta}^{k+1}, x^{k+1}, \\tilde{y}^{k+1}) \\le \\zeta_{k+1}$.\n\n\tBy convexity of the lower-level objective in \\(\\theta\\), the function $f(x,\\theta) + \\tilde{g}(x,\\theta) + \\frac{1}{2\\gamma}\\|\\theta-y\\|^2$ is $1/\\gamma$-strongly convex with respect to $\\theta$ on $Y$. Thus, it satisfies a global quadratic growth condition and, by \\cite[Proposition 2]{ye2021variational}, a global KL property with exponent $1/2$ on $Y$. Noting that $\\tilde{g}(x,\\theta) = g(x,\\theta) + \\delta_Y(\\theta)$, we have $\\tilde{g}(x,\\theta) = g(x,\\theta)$ for all $\\theta \\in Y$. This yields \n\t\\begin{equation}\\label{KL}\n\t\\begin{aligned}\n\t\tf(x,\\theta)+g(x,\\theta)+\\frac{1}{2\\gamma}\\|\\theta-y\\|^2-v_\\gamma(x,y)\\le\n\t\t\\frac{\\gamma}{2}\\,\n\t\t\\operatorname{dist}\\Bigl(\n\t\t0,\\,\n\t\t\\nabla_y f(x,\\theta)+\\frac{1}{\\gamma}(\\theta-y)+\\partial \\tilde g(x,\\theta)\n\t\t\\Bigr)^2.\n\t\\end{aligned}\n\\end{equation}\nThe optimality condition of the proximal step defining $\\tilde{\\theta}^{k+1}$,\n\\[\n\\tilde{\\theta}^{k+1}\n=\n\\operatorname{Prox}_{\\eta \\tilde g(x^{k+1},\\cdot)}\n\\left(\n\\hat{\\theta}^{k+1}\n-\\eta\\left(\n\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n+\\frac{1}{\\gamma}(\\hat{\\theta}^{k+1}-\\tilde y^{k+1})\n\\right)\n\\right),\n\\]\nyields\n\\[\n-\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n-\\frac{1}{\\gamma}(\\hat{\\theta}^{k+1}-\\tilde y^{k+1})\n-\\frac{1}{\\eta}(\\tilde{\\theta}^{k+1}-\\hat{\\theta}^{k+1})\n\\in\n\\partial \\tilde g(x^{k+1},\\tilde{\\theta}^{k+1}).\n\\]\nAdding $\\nabla_y f(x^{k+1},\\tilde{\\theta}^{k+1})+(\\tilde{\\theta}^{k+1}-\\tilde y^{k+1})/\\gamma$ to both sides gives\n\\[\n\\begin{aligned}\n\t&\\nabla_y f(x^{k+1},\\tilde{\\theta}^{k+1})\n\t+\\frac{1}{\\gamma}(\\tilde{\\theta}^{k+1}-\\tilde y^{k+1}) -\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n\t-\\frac{1}{\\gamma}(\\hat{\\theta}^{k+1}-\\tilde y^{k+1})\n\t-\\frac{1}{\\eta}(\\tilde{\\theta}^{k+1}-\\hat{\\theta}^{k+1}) \\\\\n\t\\in \\ &\n\t\\nabla_y f(x^{k+1},\\tilde{\\theta}^{k+1})\n\t+\\frac{1}{\\gamma}(\\tilde{\\theta}^{k+1}-\\tilde y^{k+1})\n\t+\\partial \\tilde g(x^{k+1},\\tilde{\\theta}^{k+1}).\n\\end{aligned}\n\\]\n\tBy the $L_{f_y}$-Lipschitz continuity of $\\nabla_y f(x,\\cdot)$ and the definition of the gradient mapping $\\mathcal{G}$, the norm of the left-hand element is bounded by\n\t\\[\n\t\\begin{aligned}\n\t\t&\\quad\\ \n\t\t\\Bigl\\|\n\t\t\\nabla_y f(x^{k+1},\\tilde{\\theta}^{k+1})\n\t\t-\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n\t\t+\\frac{1}{\\gamma}(\\tilde{\\theta}^{k+1}-\\hat{\\theta}^{k+1})\n\t\t+\\frac{1}{\\eta}(\\hat{\\theta}^{k+1}-\\tilde{\\theta}^{k+1})\n\t\t\\Bigr\\| \\\\\n\t\t&\\le\n\t\t\\Bigl(L_{f_y}+\\frac{1}{\\gamma}+\\frac{1}{\\eta}\\Bigr)\n\t\t\\|\\tilde{\\theta}^{k+1}-\\hat{\\theta}^{k+1}\\| \\\\\n\t\t&=\n\t\t\\Bigl(L_{f_y}+\\frac{1}{\\gamma}+\\frac{1}{\\eta}\\Bigr)\n\t\t\\mathcal G(\\hat{\\theta}^{k+1},x^{k+1},\\tilde y^{k+1}),\n\t\\end{aligned}\n\t\\]\nTherefore, by the definition of the distance from the origin to a set,\n\t$$\n\t\\begin{aligned}\n\t\t&\\operatorname{dist}\\left(0, \\nabla_y f(x^{k+1},\\tilde{\\theta}^{k+1}) + \\frac{1}{\\gamma}(\\tilde{\\theta}^{k+1}-\\tilde{y}^{k+1}) + \\partial \\tilde{g}(x^{k+1},\\tilde{\\theta}^{k+1})\\right) \\\\\n\t\t&\\le \\left(L_{f_y} + \\frac{1}{\\gamma} + \\frac{1}{\\eta}\\right) \\mathcal{G}(\\hat{\\theta}^{k+1}, x^{k+1}, \\tilde{y}^{k+1}).\n\t\\end{aligned}\n\t$$\n\tApplying the KL inequality \\eqref{KL} with $(x^{k+1}, \\tilde{y}^{k+1}, \\tilde{\\theta}^{k+1})$ and substituting the above estimate gives\n\t$$\n\t\\begin{aligned}\n\t\t&f(x^{k+1},\\tilde{\\theta}^{k+1}) + g(x^{k+1},\\tilde{\\theta}^{k+1}) + \\frac{1}{2\\gamma}\\|\\tilde{\\theta}^{k+1}-\\tilde{y}^{k+1}\\|^2 - v_\\gamma(x^{k+1},\\tilde{y}^{k+1}) \\\\\n\t\t&\\le \\frac{\\gamma}{2} \\left(L_{f_y} + \\frac{1}{\\gamma} + \\frac{1}{\\eta}\\right)^2 \\mathcal{G}(\\hat{\\theta}^{k+1}, x^{k+1}, \\tilde{y}^{k+1})^2.\n\t\\end{aligned}\n\t$$\n\tLetting $C_{\\gamma} := \\frac{\\gamma}{2} (L_{f_y} + 1/\\gamma + 1/\\eta)^2$ and recalling $\\varphi(x,\\theta) = f(x,\\theta) + g(x,\\theta)$, this rearranges to:\n\t$$\n\tv_\\gamma(x^{k+1},\\tilde{y}^{k+1}) \\ge \\varphi(x^{k+1},\\tilde{\\theta}^{k+1}) + \\frac{1}{2\\gamma}\\|\\tilde{\\theta}^{k+1}-\\tilde{y}^{k+1}\\|^2 - C_{\\gamma}\\zeta_{k+1}^2.\n\t$$\n\n\tBy condition \\eqref{decrease_new_y}, which $\\tilde{y}^{k+1}$ satisfies, we have\n\t\\begin{equation*}\n\t\t\\begin{aligned}\n\t\t\t&\\quad \\tilde{\\psi}_{p_k}(x^{k+1}, \\tilde{y}^{k+1}) - \\tilde{\\psi}_{p_k}(x^{k+1}, y^{k+1}) \\\\\n\t\t\t& \\le - v_\\gamma(x^{k+1}, \\tilde{y}^{k+1}) + \\left( \\varphi(x^{k+1},  \\tilde{\\theta}^{k+1} ) + \\frac{1}{2\\gamma} \\| \\tilde{\\theta}^{k+1} - \\tilde{y}^{k+1} \\|^2 \\right)\\\\\n\t\t\t& \\quad +  v_\\gamma(x^{k+1}, {y}^{k+1}) - \\left( \\varphi(x^{k+1}, \\theta^{k+1}) + \\frac{1}{2\\gamma} \\| \\theta^{k+1} - y^{k+1} \\|^2 \\right) \\\\\n\t\t\t&\\le C_{\\gamma} \\zeta_{k+1}^2,\n\t\t\\end{aligned}\n\t\\end{equation*}\n\twhere the last inequality follows from the definition of $v_\\gamma(x^{k+1}, {y}^{k+1})$ as an infimum $ \\inf_{\\theta\\in Y} \\{ \\varphi(x^{k+1}, {\\theta}) + \\frac{1}{2\\gamma} \\| {\\theta} - y^{k+1} \\|^2 \\}$. Thus, the conclusion follows.\n\\end{proof}\n","start":54888,"end":60122}]},{"id":"S081","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"区分是否接受校正","source_units":[{"raw_tex":"\\begin{proof}\n\tAt a given iteration $k$, if $\\tilde{y}^{k+1} = y^{k+1}$, the inequality holds trivially.\n\t\n\tNow, consider the case where $\\tilde{y}^{k+1} \\neq y^{k+1}$. This implies that the feasibility correction procedure (Algorithm \\ref{alg2}) is applied at iteration $k$. Accordingly, the corrected iterate  $\\tilde{y}^{k+1}$ satisfies condition \\eqref{decrease_new_y}, and its corresponding $\\tilde{\\theta}^{k+1}$ satisfies condition \\eqref{inexacty3}. Condition \\eqref{inexacty3} implies that $\\mathcal{G}(\\hat{\\theta}^{k+1}, x^{k+1}, \\tilde{y}^{k+1}) \\le \\zeta_{k+1}$.\n","start":54888,"end":55464}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","labels":["KL"],"declared_external":false,"source_only":true,"title":"强凸目标的次梯度误差估计","source_units":[{"raw_tex":"\n\tBy convexity of the lower-level objective in \\(\\theta\\), the function $f(x,\\theta) + \\tilde{g}(x,\\theta) + \\frac{1}{2\\gamma}\\|\\theta-y\\|^2$ is $1/\\gamma$-strongly convex with respect to $\\theta$ on $Y$. Thus, it satisfies a global quadratic growth condition and, by \\cite[Proposition 2]{ye2021variational}, a global KL property with exponent $1/2$ on $Y$. Noting that $\\tilde{g}(x,\\theta) = g(x,\\theta) + \\delta_Y(\\theta)$, we have $\\tilde{g}(x,\\theta) = g(x,\\theta)$ for all $\\theta \\in Y$. This yields \n\t\\begin{equation}\\label{KL}\n\t\\begin{aligned}\n\t\tf(x,\\theta)+g(x,\\theta)+\\frac{1}{2\\gamma}\\|\\theta-y\\|^2-v_\\gamma(x,y)\\le\n\t\t\\frac{\\gamma}{2}\\,\n\t\t\\operatorname{dist}\\Bigl(\n\t\t0,\\,\n\t\t\\nabla_y f(x,\\theta)+\\frac{1}{\\gamma}(\\theta-y)+\\partial \\tilde g(x,\\theta)\n\t\t\\Bigr)^2.\n\t\\end{aligned}\n\\end{equation}\n","start":55464,"end":56267}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"一步细化控制次梯度距离","source_units":[{"raw_tex":"The optimality condition of the proximal step defining $\\tilde{\\theta}^{k+1}$,\n\\[\n\\tilde{\\theta}^{k+1}\n=\n\\operatorname{Prox}_{\\eta \\tilde g(x^{k+1},\\cdot)}\n\\left(\n\\hat{\\theta}^{k+1}\n-\\eta\\left(\n\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n+\\frac{1}{\\gamma}(\\hat{\\theta}^{k+1}-\\tilde y^{k+1})\n\\right)\n\\right),\n\\]\nyields\n\\[\n-\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n-\\frac{1}{\\gamma}(\\hat{\\theta}^{k+1}-\\tilde y^{k+1})\n-\\frac{1}{\\eta}(\\tilde{\\theta}^{k+1}-\\hat{\\theta}^{k+1})\n\\in\n\\partial \\tilde g(x^{k+1},\\tilde{\\theta}^{k+1}).\n\\]\nAdding $\\nabla_y f(x^{k+1},\\tilde{\\theta}^{k+1})+(\\tilde{\\theta}^{k+1}-\\tilde y^{k+1})/\\gamma$ to both sides gives\n\\[\n\\begin{aligned}\n\t&\\nabla_y f(x^{k+1},\\tilde{\\theta}^{k+1})\n\t+\\frac{1}{\\gamma}(\\tilde{\\theta}^{k+1}-\\tilde y^{k+1}) -\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n\t-\\frac{1}{\\gamma}(\\hat{\\theta}^{k+1}-\\tilde y^{k+1})\n\t-\\frac{1}{\\eta}(\\tilde{\\theta}^{k+1}-\\hat{\\theta}^{k+1}) \\\\\n\t\\in \\ &\n\t\\nabla_y f(x^{k+1},\\tilde{\\theta}^{k+1})\n\t+\\frac{1}{\\gamma}(\\tilde{\\theta}^{k+1}-\\tilde y^{k+1})\n\t+\\partial \\tilde g(x^{k+1},\\tilde{\\theta}^{k+1}).\n\\end{aligned}\n\\]\n\tBy the $L_{f_y}$-Lipschitz continuity of $\\nabla_y f(x,\\cdot)$ and the definition of the gradient mapping $\\mathcal{G}$, the norm of the left-hand element is bounded by\n\t\\[\n\t\\begin{aligned}\n\t\t&\\quad\\ \n\t\t\\Bigl\\|\n\t\t\\nabla_y f(x^{k+1},\\tilde{\\theta}^{k+1})\n\t\t-\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n\t\t+\\frac{1}{\\gamma}(\\tilde{\\theta}^{k+1}-\\hat{\\theta}^{k+1})\n\t\t+\\frac{1}{\\eta}(\\hat{\\theta}^{k+1}-\\tilde{\\theta}^{k+1})\n\t\t\\Bigr\\| \\\\\n\t\t&\\le\n\t\t\\Bigl(L_{f_y}+\\frac{1}{\\gamma}+\\frac{1}{\\eta}\\Bigr)\n\t\t\\|\\tilde{\\theta}^{k+1}-\\hat{\\theta}^{k+1}\\| \\\\\n\t\t&=\n\t\t\\Bigl(L_{f_y}+\\frac{1}{\\gamma}+\\frac{1}{\\eta}\\Bigr)\n\t\t\\mathcal G(\\hat{\\theta}^{k+1},x^{k+1},\\tilde y^{k+1}),\n\t\\end{aligned}\n\t\\]\nTherefore, by the definition of the distance from the origin to a set,\n\t$$\n\t\\begin{aligned}\n\t\t&\\operatorname{dist}\\left(0, \\nabla_y f(x^{k+1},\\tilde{\\theta}^{k+1}) + \\frac{1}{\\gamma}(\\tilde{\\theta}^{k+1}-\\tilde{y}^{k+1}) + \\partial \\tilde{g}(x^{k+1},\\tilde{\\theta}^{k+1})\\right) \\\\\n\t\t&\\le \\left(L_{f_y} + \\frac{1}{\\gamma} + \\frac{1}{\\eta}\\right) \\mathcal{G}(\\hat{\\theta}^{k+1}, x^{k+1}, \\tilde{y}^{k+1}).\n\t\\end{aligned}\n\t$$\n","start":56267,"end":58451}]},{"id":"S084","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"近端候选值与包络的误差","source_units":[{"raw_tex":"\tApplying the KL inequality \\eqref{KL} with $(x^{k+1}, \\tilde{y}^{k+1}, \\tilde{\\theta}^{k+1})$ and substituting the above estimate gives\n\t$$\n\t\\begin{aligned}\n\t\t&f(x^{k+1},\\tilde{\\theta}^{k+1}) + g(x^{k+1},\\tilde{\\theta}^{k+1}) + \\frac{1}{2\\gamma}\\|\\tilde{\\theta}^{k+1}-\\tilde{y}^{k+1}\\|^2 - v_\\gamma(x^{k+1},\\tilde{y}^{k+1}) \\\\\n\t\t&\\le \\frac{\\gamma}{2} \\left(L_{f_y} + \\frac{1}{\\gamma} + \\frac{1}{\\eta}\\right)^2 \\mathcal{G}(\\hat{\\theta}^{k+1}, x^{k+1}, \\tilde{y}^{k+1})^2.\n\t\\end{aligned}\n\t$$\n\tLetting $C_{\\gamma} := \\frac{\\gamma}{2} (L_{f_y} + 1/\\gamma + 1/\\eta)^2$ and recalling $\\varphi(x,\\theta) = f(x,\\theta) + g(x,\\theta)$, this rearranges to:\n\t$$\n\tv_\\gamma(x^{k+1},\\tilde{y}^{k+1}) \\ge \\varphi(x^{k+1},\\tilde{\\theta}^{k+1}) + \\frac{1}{2\\gamma}\\|\\tilde{\\theta}^{k+1}-\\tilde{y}^{k+1}\\|^2 - C_{\\gamma}\\zeta_{k+1}^2.\n\t$$\n","start":58451,"end":59273}]},{"id":"S085","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"接受条件推出校正近似下降","source_units":[{"raw_tex":"\n\tBy condition \\eqref{decrease_new_y}, which $\\tilde{y}^{k+1}$ satisfies, we have\n\t\\begin{equation*}\n\t\t\\begin{aligned}\n\t\t\t&\\quad \\tilde{\\psi}_{p_k}(x^{k+1}, \\tilde{y}^{k+1}) - \\tilde{\\psi}_{p_k}(x^{k+1}, y^{k+1}) \\\\\n\t\t\t& \\le - v_\\gamma(x^{k+1}, \\tilde{y}^{k+1}) + \\left( \\varphi(x^{k+1},  \\tilde{\\theta}^{k+1} ) + \\frac{1}{2\\gamma} \\| \\tilde{\\theta}^{k+1} - \\tilde{y}^{k+1} \\|^2 \\right)\\\\\n\t\t\t& \\quad +  v_\\gamma(x^{k+1}, {y}^{k+1}) - \\left( \\varphi(x^{k+1}, \\theta^{k+1}) + \\frac{1}{2\\gamma} \\| \\theta^{k+1} - y^{k+1} \\|^2 \\right) \\\\\n\t\t\t&\\le C_{\\gamma} \\zeta_{k+1}^2,\n\t\t\\end{aligned}\n\t\\end{equation*}\n\twhere the last inequality follows from the definition of $v_\\gamma(x^{k+1}, {y}^{k+1})$ as an infimum $ \\inf_{\\theta\\in Y} \\{ \\varphi(x^{k+1}, {\\theta}) + \\frac{1}{2\\gamma} \\| {\\theta} - y^{k+1} \\|^2 \\}$. Thus, the conclusion follows.\n\\end{proof}\n","start":59273,"end":60122}]},{"id":"S086","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原前提列表：mordukhovich2018variational, Corollary 4.3(ii), Theorems 1.22, 4.1(ii), 4.17；作者实际引用见第226行。"],"source_only":false,"title":"边际函数敏感性结果","source_units":[]},{"id":"S087","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原前提列表：gao2024, Theorem 2 的证明论证；引用见第226行。"],"source_only":false,"title":"偏弱凸性的外部证明思路","source_units":[]},{"id":"S088","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原前提列表：ye2021variational, Proposition 2；作者引用见第564行。"],"source_only":false,"title":"强凸目标的KL误差估计","source_units":[]},{"id":"S089","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原前提列表：beck2017first, Theorem 6.39；作者引用见第673行。"],"source_only":false,"title":"强凸极小点比较不等式","source_units":[]},{"id":"S090","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原前提列表：beck2017first, Theorem 10.7；作者在第853行引用。"],"source_only":false,"title":"凸最优解的近端固定点性质","source_units":[]},{"id":"S091","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原前提列表中的 Standard convex optimality, projection and proximal calculus；具体常识步骤由注释者显式标出。"],"source_only":false,"title":"凸最优性与近端投影计算","source_units":[]},{"id":"S092","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原前提列表中的 Standard descent inequalities；具体常识步骤由注释者显式标出。"],"source_only":false,"title":"下降引理与内积误差吸收","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S069","edges":[{"source_id":"S001","target_id":"S005","kind":"supports"},{"source_id":"S002","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S005","target_id":"S016","kind":"supports"},{"source_id":"S007","target_id":"S016","kind":"supports"},{"source_id":"S008","target_id":"S016","kind":"supports"},{"source_id":"S010","target_id":"S078","kind":"supports"},{"source_id":"S012","target_id":"S078","kind":"supports"},{"source_id":"S013","target_id":"S078","kind":"supports"},{"source_id":"S086","target_id":"S078","kind":"supports"},{"source_id":"S011","target_id":"S078","kind":"supports"},{"source_id":"S087","target_id":"S078","kind":"supports"},{"source_id":"S078","target_id":"S079","kind":"supports"},{"source_id":"S010","target_id":"S079","kind":"supports"},{"source_id":"S012","target_id":"S079","kind":"supports"},{"source_id":"S013","target_id":"S079","kind":"supports"},{"source_id":"S014","target_id":"S079","kind":"supports"},{"source_id":"S078","target_id":"S077","kind":"supports"},{"source_id":"S079","target_id":"S077","kind":"supports"},{"source_id":"S077","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S004","target_id":"S033","kind":"supports"},{"source_id":"S009","target_id":"S033","kind":"supports"},{"source_id":"S014","target_id":"S018","kind":"supports"},{"source_id":"S015","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S013","target_id":"S018","kind":"supports"},{"source_id":"S014","target_id":"S021","kind":"supports"},{"source_id":"S015","target_id":"S021","kind":"supports"},{"source_id":"S017","target_id":"S021","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S013","target_id":"S019","kind":"supports"},{"source_id":"S020","target_id":"S022","kind":"supports"},{"source_id":"S013","target_id":"S022","kind":"supports"},{"source_id":"S020","target_id":"S028","kind":"supports"},{"source_id":"S008","target_id":"S028","kind":"supports"},{"source_id":"S018","target_id":"S030","kind":"supports"},{"source_id":"S019","target_id":"S030","kind":"supports"},{"source_id":"S021","target_id":"S030","kind":"supports"},{"source_id":"S022","target_id":"S030","kind":"supports"},{"source_id":"S023","target_id":"S030","kind":"supports"},{"source_id":"S025","target_id":"S030","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S026","target_id":"S031","kind":"supports"},{"source_id":"S027","target_id":"S031","kind":"supports"},{"source_id":"S028","target_id":"S031","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S020","target_id":"S032","kind":"supports"},{"source_id":"S013","target_id":"S032","kind":"supports"},{"source_id":"S015","target_id":"S038","kind":"supports"},{"source_id":"S011","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S027","target_id":"S081","kind":"supports"},{"source_id":"S028","target_id":"S081","kind":"supports"},{"source_id":"S030","target_id":"S081","kind":"supports"},{"source_id":"S031","target_id":"S081","kind":"supports"},{"source_id":"S010","target_id":"S082","kind":"supports"},{"source_id":"S004","target_id":"S082","kind":"supports"},{"source_id":"S008","target_id":"S082","kind":"supports"},{"source_id":"S088","target_id":"S082","kind":"supports"},{"source_id":"S028","target_id":"S083","kind":"supports"},{"source_id":"S012","target_id":"S083","kind":"supports"},{"source_id":"S020","target_id":"S083","kind":"supports"},{"source_id":"S091","target_id":"S083","kind":"supports"},{"source_id":"S081","target_id":"S084","kind":"supports"},{"source_id":"S082","target_id":"S084","kind":"supports"},{"source_id":"S083","target_id":"S084","kind":"supports"},{"source_id":"S084","target_id":"S085","kind":"supports"},{"source_id":"S027","target_id":"S085","kind":"supports"},{"source_id":"S004","target_id":"S085","kind":"supports"},{"source_id":"S033","target_id":"S085","kind":"supports"},{"source_id":"S081","target_id":"S080","kind":"supports"},{"source_id":"S082","target_id":"S080","kind":"supports"},{"source_id":"S083","target_id":"S080","kind":"supports"},{"source_id":"S084","target_id":"S080","kind":"supports"},{"source_id":"S085","target_id":"S080","kind":"supports"},{"source_id":"S080","target_id":"S034","kind":"supports"},{"source_id":"S018","target_id":"S042","kind":"supports"},{"source_id":"S010","target_id":"S042","kind":"supports"},{"source_id":"S089","target_id":"S042","kind":"supports"},{"source_id":"S009","target_id":"S043","kind":"supports"},{"source_id":"S012","target_id":"S043","kind":"supports"},{"source_id":"S033","target_id":"S043","kind":"supports"},{"source_id":"S037","target_id":"S043","kind":"supports"},{"source_id":"S092","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S014","target_id":"S045","kind":"supports"},{"source_id":"S015","target_id":"S045","kind":"supports"},{"source_id":"S018","target_id":"S045","kind":"supports"},{"source_id":"S032","target_id":"S045","kind":"supports"},{"source_id":"S022","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S092","target_id":"S046","kind":"supports"},{"source_id":"S021","target_id":"S047","kind":"supports"},{"source_id":"S002","target_id":"S047","kind":"supports"},{"source_id":"S009","target_id":"S047","kind":"supports"},{"source_id":"S012","target_id":"S047","kind":"supports"},{"source_id":"S036","target_id":"S047","kind":"supports"},{"source_id":"S091","target_id":"S047","kind":"supports"},{"source_id":"S092","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S014","target_id":"S049","kind":"supports"},{"source_id":"S015","target_id":"S049","kind":"supports"},{"source_id":"S021","target_id":"S049","kind":"supports"},{"source_id":"S012","target_id":"S049","kind":"supports"},{"source_id":"S032","target_id":"S049","kind":"supports"},{"source_id":"S019","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S092","target_id":"S050","kind":"supports"},{"source_id":"S046","target_id":"S051","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S034","target_id":"S051","kind":"supports"},{"source_id":"S009","target_id":"S051","kind":"supports"},{"source_id":"S030","target_id":"S051","kind":"supports"},{"source_id":"S020","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S040","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S043","target_id":"S041","kind":"supports"},{"source_id":"S044","target_id":"S041","kind":"supports"},{"source_id":"S045","target_id":"S041","kind":"supports"},{"source_id":"S046","target_id":"S041","kind":"supports"},{"source_id":"S047","target_id":"S041","kind":"supports"},{"source_id":"S048","target_id":"S041","kind":"supports"},{"source_id":"S049","target_id":"S041","kind":"supports"},{"source_id":"S050","target_id":"S041","kind":"supports"},{"source_id":"S051","target_id":"S041","kind":"supports"},{"source_id":"S052","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S055","kind":"supports"},{"source_id":"S035","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S040","target_id":"S056","kind":"supports"},{"source_id":"S033","target_id":"S056","kind":"supports"},{"source_id":"S009","target_id":"S056","kind":"supports"},{"source_id":"S004","target_id":"S056","kind":"supports"},{"source_id":"S002","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S054","kind":"supports"},{"source_id":"S056","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S026","target_id":"S059","kind":"supports"},{"source_id":"S008","target_id":"S059","kind":"supports"},{"source_id":"S091","target_id":"S059","kind":"supports"},{"source_id":"S026","target_id":"S060","kind":"supports"},{"source_id":"S053","target_id":"S060","kind":"supports"},{"source_id":"S030","target_id":"S060","kind":"supports"},{"source_id":"S059","target_id":"S061","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S012","target_id":"S061","kind":"supports"},{"source_id":"S010","target_id":"S061","kind":"supports"},{"source_id":"S091","target_id":"S061","kind":"supports"},{"source_id":"S059","target_id":"S058","kind":"supports"},{"source_id":"S060","target_id":"S058","kind":"supports"},{"source_id":"S061","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S053","target_id":"S064","kind":"supports"},{"source_id":"S030","target_id":"S064","kind":"supports"},{"source_id":"S023","target_id":"S064","kind":"supports"},{"source_id":"S029","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S030","target_id":"S065","kind":"supports"},{"source_id":"S031","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S053","target_id":"S066","kind":"supports"},{"source_id":"S023","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S032","target_id":"S067","kind":"supports"},{"source_id":"S015","target_id":"S067","kind":"supports"},{"source_id":"S022","target_id":"S067","kind":"supports"},{"source_id":"S040","target_id":"S067","kind":"supports"},{"source_id":"S012","target_id":"S067","kind":"supports"},{"source_id":"S013","target_id":"S067","kind":"supports"},{"source_id":"S064","target_id":"S068","kind":"supports"},{"source_id":"S065","target_id":"S068","kind":"supports"},{"source_id":"S057","target_id":"S068","kind":"supports"},{"source_id":"S090","target_id":"S068","kind":"supports"},{"source_id":"S004","target_id":"S068","kind":"supports"},{"source_id":"S013","target_id":"S068","kind":"supports"},{"source_id":"S064","target_id":"S063","kind":"supports"},{"source_id":"S065","target_id":"S063","kind":"supports"},{"source_id":"S066","target_id":"S063","kind":"supports"},{"source_id":"S067","target_id":"S063","kind":"supports"},{"source_id":"S068","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S062","kind":"supports"},{"source_id":"S053","target_id":"S071","kind":"supports"},{"source_id":"S030","target_id":"S071","kind":"supports"},{"source_id":"S023","target_id":"S071","kind":"supports"},{"source_id":"S029","target_id":"S071","kind":"supports"},{"source_id":"S035","target_id":"S071","kind":"supports"},{"source_id":"S018","target_id":"S072","kind":"supports"},{"source_id":"S021","target_id":"S072","kind":"supports"},{"source_id":"S033","target_id":"S072","kind":"supports"},{"source_id":"S007","target_id":"S072","kind":"supports"},{"source_id":"S008","target_id":"S072","kind":"supports"},{"source_id":"S091","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S049","target_id":"S073","kind":"supports"},{"source_id":"S015","target_id":"S073","kind":"supports"},{"source_id":"S009","target_id":"S073","kind":"supports"},{"source_id":"S012","target_id":"S073","kind":"supports"},{"source_id":"S035","target_id":"S073","kind":"supports"},{"source_id":"S040","target_id":"S073","kind":"supports"},{"source_id":"S053","target_id":"S073","kind":"supports"},{"source_id":"S071","target_id":"S074","kind":"supports"},{"source_id":"S072","target_id":"S074","kind":"supports"},{"source_id":"S045","target_id":"S074","kind":"supports"},{"source_id":"S015","target_id":"S074","kind":"supports"},{"source_id":"S009","target_id":"S074","kind":"supports"},{"source_id":"S012","target_id":"S074","kind":"supports"},{"source_id":"S035","target_id":"S074","kind":"supports"},{"source_id":"S040","target_id":"S074","kind":"supports"},{"source_id":"S053","target_id":"S074","kind":"supports"},{"source_id":"S071","target_id":"S075","kind":"supports"},{"source_id":"S072","target_id":"S075","kind":"supports"},{"source_id":"S073","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S015","target_id":"S075","kind":"supports"},{"source_id":"S009","target_id":"S075","kind":"supports"},{"source_id":"S012","target_id":"S075","kind":"supports"},{"source_id":"S007","target_id":"S075","kind":"supports"},{"source_id":"S008","target_id":"S075","kind":"supports"},{"source_id":"S075","target_id":"S076","kind":"supports"},{"source_id":"S062","target_id":"S076","kind":"supports"},{"source_id":"S004","target_id":"S076","kind":"supports"},{"source_id":"S016","target_id":"S076","kind":"supports"},{"source_id":"S071","target_id":"S076","kind":"supports"},{"source_id":"S071","target_id":"S070","kind":"supports"},{"source_id":"S072","target_id":"S070","kind":"supports"},{"source_id":"S073","target_id":"S070","kind":"supports"},{"source_id":"S074","target_id":"S070","kind":"supports"},{"source_id":"S075","target_id":"S070","kind":"supports"},{"source_id":"S076","target_id":"S070","kind":"supports"},{"source_id":"S070","target_id":"S069","kind":"supports"},{"source_id":"S035","target_id":"S069","kind":"supports"},{"source_id":"S005","target_id":"S069","kind":"supports"},{"source_id":"S016","target_id":"S069","kind":"supports"},{"source_id":"S030","target_id":"S069","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S069","edges":[{"source_id":"S001","target_id":"S005","kind":"supports"},{"source_id":"S002","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S005","target_id":"S016","kind":"supports"},{"source_id":"S007","target_id":"S016","kind":"supports"},{"source_id":"S008","target_id":"S016","kind":"supports"},{"source_id":"S010","target_id":"S015","kind":"supports"},{"source_id":"S012","target_id":"S015","kind":"supports"},{"source_id":"S013","target_id":"S015","kind":"supports"},{"source_id":"S086","target_id":"S015","kind":"supports"},{"source_id":"S011","target_id":"S015","kind":"supports"},{"source_id":"S087","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S004","target_id":"S033","kind":"supports"},{"source_id":"S009","target_id":"S033","kind":"supports"},{"source_id":"S014","target_id":"S018","kind":"supports"},{"source_id":"S015","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S013","target_id":"S018","kind":"supports"},{"source_id":"S014","target_id":"S021","kind":"supports"},{"source_id":"S015","target_id":"S021","kind":"supports"},{"source_id":"S017","target_id":"S021","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S013","target_id":"S019","kind":"supports"},{"source_id":"S020","target_id":"S022","kind":"supports"},{"source_id":"S013","target_id":"S022","kind":"supports"},{"source_id":"S020","target_id":"S028","kind":"supports"},{"source_id":"S008","target_id":"S028","kind":"supports"},{"source_id":"S018","target_id":"S030","kind":"supports"},{"source_id":"S019","target_id":"S030","kind":"supports"},{"source_id":"S021","target_id":"S030","kind":"supports"},{"source_id":"S022","target_id":"S030","kind":"supports"},{"source_id":"S023","target_id":"S030","kind":"supports"},{"source_id":"S025","target_id":"S030","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S026","target_id":"S031","kind":"supports"},{"source_id":"S027","target_id":"S031","kind":"supports"},{"source_id":"S028","target_id":"S031","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S020","target_id":"S032","kind":"supports"},{"source_id":"S013","target_id":"S032","kind":"supports"},{"source_id":"S015","target_id":"S038","kind":"supports"},{"source_id":"S011","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S027","target_id":"S034","kind":"supports"},{"source_id":"S028","target_id":"S034","kind":"supports"},{"source_id":"S030","target_id":"S034","kind":"supports"},{"source_id":"S031","target_id":"S034","kind":"supports"},{"source_id":"S010","target_id":"S034","kind":"supports"},{"source_id":"S004","target_id":"S034","kind":"supports"},{"source_id":"S008","target_id":"S034","kind":"supports"},{"source_id":"S088","target_id":"S034","kind":"supports"},{"source_id":"S012","target_id":"S034","kind":"supports"},{"source_id":"S020","target_id":"S034","kind":"supports"},{"source_id":"S091","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S018","target_id":"S039","kind":"supports"},{"source_id":"S010","target_id":"S039","kind":"supports"},{"source_id":"S089","target_id":"S039","kind":"supports"},{"source_id":"S009","target_id":"S039","kind":"supports"},{"source_id":"S012","target_id":"S039","kind":"supports"},{"source_id":"S033","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S092","target_id":"S039","kind":"supports"},{"source_id":"S014","target_id":"S045","kind":"supports"},{"source_id":"S015","target_id":"S045","kind":"supports"},{"source_id":"S018","target_id":"S045","kind":"supports"},{"source_id":"S032","target_id":"S045","kind":"supports"},{"source_id":"S022","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S039","kind":"supports"},{"source_id":"S021","target_id":"S039","kind":"supports"},{"source_id":"S002","target_id":"S039","kind":"supports"},{"source_id":"S036","target_id":"S039","kind":"supports"},{"source_id":"S091","target_id":"S039","kind":"supports"},{"source_id":"S014","target_id":"S049","kind":"supports"},{"source_id":"S015","target_id":"S049","kind":"supports"},{"source_id":"S021","target_id":"S049","kind":"supports"},{"source_id":"S012","target_id":"S049","kind":"supports"},{"source_id":"S032","target_id":"S049","kind":"supports"},{"source_id":"S019","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S039","kind":"supports"},{"source_id":"S034","target_id":"S039","kind":"supports"},{"source_id":"S030","target_id":"S039","kind":"supports"},{"source_id":"S020","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S040","kind":"supports"},{"source_id":"S052","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S053","kind":"supports"},{"source_id":"S035","target_id":"S053","kind":"supports"},{"source_id":"S040","target_id":"S053","kind":"supports"},{"source_id":"S033","target_id":"S053","kind":"supports"},{"source_id":"S009","target_id":"S053","kind":"supports"},{"source_id":"S004","target_id":"S053","kind":"supports"},{"source_id":"S002","target_id":"S053","kind":"supports"},{"source_id":"S026","target_id":"S057","kind":"supports"},{"source_id":"S008","target_id":"S057","kind":"supports"},{"source_id":"S091","target_id":"S057","kind":"supports"},{"source_id":"S053","target_id":"S057","kind":"supports"},{"source_id":"S030","target_id":"S057","kind":"supports"},{"source_id":"S012","target_id":"S057","kind":"supports"},{"source_id":"S010","target_id":"S057","kind":"supports"},{"source_id":"S053","target_id":"S062","kind":"supports"},{"source_id":"S030","target_id":"S062","kind":"supports"},{"source_id":"S023","target_id":"S062","kind":"supports"},{"source_id":"S029","target_id":"S062","kind":"supports"},{"source_id":"S031","target_id":"S062","kind":"supports"},{"source_id":"S032","target_id":"S062","kind":"supports"},{"source_id":"S015","target_id":"S062","kind":"supports"},{"source_id":"S022","target_id":"S062","kind":"supports"},{"source_id":"S040","target_id":"S062","kind":"supports"},{"source_id":"S012","target_id":"S062","kind":"supports"},{"source_id":"S013","target_id":"S062","kind":"supports"},{"source_id":"S057","target_id":"S062","kind":"supports"},{"source_id":"S090","target_id":"S062","kind":"supports"},{"source_id":"S004","target_id":"S062","kind":"supports"},{"source_id":"S053","target_id":"S071","kind":"supports"},{"source_id":"S030","target_id":"S071","kind":"supports"},{"source_id":"S023","target_id":"S071","kind":"supports"},{"source_id":"S029","target_id":"S071","kind":"supports"},{"source_id":"S035","target_id":"S071","kind":"supports"},{"source_id":"S018","target_id":"S072","kind":"supports"},{"source_id":"S021","target_id":"S072","kind":"supports"},{"source_id":"S033","target_id":"S072","kind":"supports"},{"source_id":"S007","target_id":"S072","kind":"supports"},{"source_id":"S008","target_id":"S072","kind":"supports"},{"source_id":"S091","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S049","target_id":"S073","kind":"supports"},{"source_id":"S015","target_id":"S073","kind":"supports"},{"source_id":"S009","target_id":"S073","kind":"supports"},{"source_id":"S012","target_id":"S073","kind":"supports"},{"source_id":"S035","target_id":"S073","kind":"supports"},{"source_id":"S040","target_id":"S073","kind":"supports"},{"source_id":"S053","target_id":"S073","kind":"supports"},{"source_id":"S071","target_id":"S074","kind":"supports"},{"source_id":"S072","target_id":"S074","kind":"supports"},{"source_id":"S045","target_id":"S074","kind":"supports"},{"source_id":"S015","target_id":"S074","kind":"supports"},{"source_id":"S009","target_id":"S074","kind":"supports"},{"source_id":"S012","target_id":"S074","kind":"supports"},{"source_id":"S035","target_id":"S074","kind":"supports"},{"source_id":"S040","target_id":"S074","kind":"supports"},{"source_id":"S053","target_id":"S074","kind":"supports"},{"source_id":"S071","target_id":"S075","kind":"supports"},{"source_id":"S072","target_id":"S075","kind":"supports"},{"source_id":"S073","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S015","target_id":"S075","kind":"supports"},{"source_id":"S009","target_id":"S075","kind":"supports"},{"source_id":"S012","target_id":"S075","kind":"supports"},{"source_id":"S007","target_id":"S075","kind":"supports"},{"source_id":"S008","target_id":"S075","kind":"supports"},{"source_id":"S075","target_id":"S076","kind":"supports"},{"source_id":"S062","target_id":"S076","kind":"supports"},{"source_id":"S004","target_id":"S076","kind":"supports"},{"source_id":"S016","target_id":"S076","kind":"supports"},{"source_id":"S071","target_id":"S076","kind":"supports"},{"source_id":"S071","target_id":"S070","kind":"supports"},{"source_id":"S072","target_id":"S070","kind":"supports"},{"source_id":"S073","target_id":"S070","kind":"supports"},{"source_id":"S074","target_id":"S070","kind":"supports"},{"source_id":"S075","target_id":"S070","kind":"supports"},{"source_id":"S076","target_id":"S070","kind":"supports"},{"source_id":"S070","target_id":"S069","kind":"supports"},{"source_id":"S035","target_id":"S069","kind":"supports"},{"source_id":"S005","target_id":"S069","kind":"supports"},{"source_id":"S016","target_id":"S069","kind":"supports"},{"source_id":"S030","target_id":"S069","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["LL","eq1"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"双层优化与下层解集","source_units":[{"raw_tex":"Bilevel optimization problems are a class of hierarchical optimization problems where the feasible region is implicitly defined by the solution set of a lower-level optimization problem. In this work, we focus on a bilevel optimization problem which has a convex composite lower-level problem,\n\\begin{equation}\\label{eq1}\n\t\\begin{aligned}\n\t\t\\min\\limits_{x \\in X, y \\in Y} &\\ F(x,y)\\\\\n\t\t{\\rm s.t.}\\quad &\\ y \\in S(x),\n\t\\end{aligned}\n\\end{equation}\nwhere $S(x)$ denotes the set of optimal solutions to the lower-level problem:\n\\begin{equation}\\label{LL}\n\t\\min\\limits_{y \\in Y}\\ \\varphi(x,y) := f(x,y) + g(x,y).\n\\end{equation}\n","start":3423,"end":4047}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"闭凸域与复合结构","source_units":[{"raw_tex":"Here, $X \\subset \\Re^n$ and $Y \\subset \\Re^m$ are closed convex sets, $f(x, y) : \\Re^n \\times \\Re^m \\rightarrow \\Re$ is smooth and convex with respect to $y$ on $Y$ for any $x \\in X$, and $g(x, y) : \\Re^n \\times \\Re^m \\rightarrow \\Re$ is convex but potentially nonsmooth with respect to $y$ on $Y$ for any $x \\in X$. Additionally, we assume that $g(x, \\cdot) : \\Re^m \\rightarrow \\Re$ is proximal-friendly, enabling efficient evaluation of its proximal operator.\n","start":4047,"end":4509}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","labels":["reformualtion_Moreau"],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"未松弛包络重构","source_units":[{"raw_tex":"To overcome this challenge, in this paper, we consider the following Moreau envelope-based reformulation of the bilevel optimization problem \\eqref{eq1}, first introduced in \\cite{gao2024},\n\\begin{equation}\\label{reformualtion_Moreau}\n\t({\\rm VP})_\\gamma \\quad\\quad\n\t\\min\\limits_{x\\in X,y\\in Y}\\  F(x,y) \\quad\t\t{\\rm s.t.}\\quad\\ \\varphi (x,y)-v_{\\gamma} (x,y)\\le 0,\n\\end{equation}\n","start":11073,"end":11452}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["def_vg"],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"下层Moreau包络","source_units":[{"raw_tex":"where $v_{\\gamma}(x, y)$ is the Moreau envelope associated with the lower-level problem,\n\\begin{equation}\\label{def_vg}\n\tv_{\\gamma}(x,y):=\\inf\\limits_{\\theta\\in Y}\\varphi(x,\\theta)+\\frac{1}{2\\gamma}\\|\\theta-y\\|^2,\n\\end{equation}\n","start":11452,"end":11681}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":["reformualtion_Moreau_relax"],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"松弛包络约束问题","source_units":[{"raw_tex":"where $\\gamma > 0$. When the lower-level problem is convex with respect to $y$,  this reformulation is equivalent to the original bilevel optimization problem \\eqref{eq1} (see \\cite[Theorem 1]{gao2024}).  Directional optimality conditions for $({\\rm VP})_\\gamma$ are discussed in \\cite{bai2025optimality}. However, as shown in \\cite{gao2024} and \\cite[Proposition 7]{ye2023}, classical constraint qualification (CQ) conditions do not hold at any feasible point of $({\\rm VP})_\\gamma$, motivating us to study the following relaxed approximation problem,\n\\begin{equation}\\label{reformualtion_Moreau_relax}\n\t({\\rm VP})_\\gamma^{\\epsilon} \\quad\\quad\n\t\\min\\limits_{x\\in X,y\\in Y}\\  F(x,y) \\quad\t\t{\\rm s.t.}\\quad\\ \\varphi (x,y)-v_{\\gamma} (x,y)\\le \\epsilon,\n\\end{equation}\n","start":11681,"end":12447}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"欧氏空间记号","source_units":[{"raw_tex":"Let $\\mathbb{R}^n$ denote the $n$-dimensional Euclidean space, and let $\\mathbb{B}$ denote the closed unit ball centered at the origin. The nonnegative and positive orthants in $\\mathbb{R}^n$ are denoted by $\\mathbb{R}^n_+$ and $\\mathbb{R}^n_{++}$, respectively. The standard inner product and Euclidean norm are denoted by $\\langle \\cdot, \\cdot \\rangle$ and $\\|\\cdot\\|$, respectively.\n","start":16383,"end":16769}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"集合与投影算子","source_units":[{"raw_tex":"For a vector \\( x \\in \\mathbb{R}^n \\) and a closed convex set \\( S \\subset \\mathbb{R}^n \\), the distance from \\( x \\) to \\( S \\) is defined as \\(\\mathrm{dist}(x, S) = \\min_{y \\in S} \\|x - y\\|\\). The indicator function \\(\\delta_S: \\mathbb{R}^n \\to \\mathbb{R}\\cup\\{+\\infty\\}\\) is defined by \\(\\delta_S(x) = 0\\) if \\(x\\in S\\) and \\(\\delta_S(x) = +\\infty\\) otherwise. Moreover, \\(\\mathcal{N}_S(x)\\) denotes the normal cone to \\(S\\) at \\(x\\). The Euclidean projection operator onto \\( S \\) is denoted by \\(\\mathrm{Proj}_S\\). The Cartesian product of two sets \\( X \\) and \\( Y \\) is denoted by \\( X \\times Y \\).\n","start":16769,"end":17375}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"近端映射与次微分","source_units":[{"raw_tex":"Let \\( h: \\mathbb{R}^n \\to \\mathbb{R} \\cup \\{\\infty\\} \\) be a proper closed function. The proximal mapping of \\( h\\) is defined as $\\operatorname{Prox}_h(y):=\\arg\\min_{\\theta \\in \\Re^m}\\left\\{h(\\theta)+\\|\\theta-y\\|^2 / 2\\right\\}$. The Fr\\'{e}chet (regular) subdifferential \\(\\hat{\\partial} h\\) and the Mordukhovich (limiting) subdifferential \\(\\partial h\\) at a point \\( \\bar{z} \\in \\mathrm{dom}\\, h := \\{z \\mid h(z) < \\infty\\} \\) are defined as:\n\\[\n\\begin{aligned}\n\t\\hat{\\partial} h(\\bar{z}) &= \\left\\{ v \\in \\mathbb{R}^n \\;\\middle|\\; h(z) \\geq h(\\bar{z}) + \\langle v, z - \\bar{z} \\rangle + o(\\|z - \\bar{z}\\|), \\; \\forall z \\in \\mathbb{R}^n \\right\\}, \\\\\n\t\\partial h(\\bar{z}) &= \\left\\{ v \\in \\mathbb{R}^n \\;\\middle|\\; \\exists z_k \\xrightarrow{h} \\bar{z}, \\; \\exists v_k \\in \\hat{\\partial} h(z_k), \\; v_k \\to v \\right\\},\n\\end{aligned}\n\\]\nwhere \\( o(z) \\) denotes a function satisfying \\( o(z)/z\\to 0 \\) as \\( z \\downarrow 0 \\), and \\( z_k \\xrightarrow{h} \\bar{z} \\) indicates \\( z_k \\to \\bar{z} \\) and \\( h(z_k) \\to h(\\bar{z}) \\) as \\( k \\to \\infty \\). If \\( \\bar{z} \\notin \\mathrm{dom}\\, h \\), we define \\(\\partial h(\\bar{z}) = \\emptyset\\).\n","start":17375,"end":18517}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","labels":["asup1"],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"上层目标的正则性","source_units":[{"raw_tex":"This part outlines the assumptions on the problem data for the bilevel optimization problem studied in this work. First, we assume that the upper-level objective $F$ is smooth and bounded below.\n\n\\begin{assumption}\\label{asup1}\n\tThe function \\(F : \\mathbb{R}^n \\times \\mathbb{R}^m \\to \\mathbb{R}\\) is bounded below on \\(X \\times Y\\), i.e., \\(\\underline{F} := \\inf_{(x, y) \\in X \\times Y} F(x, y) > -\\infty\\). \\(F\\) is continuously differentiable, and its gradients \\(\\nabla_x F\\) and \\(\\nabla_y F\\) are \\(L_{F_x}\\)- and \\(L_{F_y}\\)-Lipschitz continuous on \\(X \\times Y\\), respectively.\n\\end{assumption}\n\n","start":18552,"end":19156}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","labels":["asup2"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"下层变量凸性","source_units":[{"raw_tex":"Throughout this paper, we focus on cases where the lower-level problem is a convex optimization problem.\n\n\\begin{assumption}\\label{asup2}\n\tFor any \\(x \\in X\\), the functions \\(f(x, \\cdot)\\) and \\(g(x, \\cdot)\\) are convex and defined on the closed convex set \\(Y\\). \n\\end{assumption} \n","start":19156,"end":19440}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"分块弱凸性","source_units":[{"raw_tex":"\nSmoothness and weak convexity conditions are required for the lower-level objective functions. We recall that a function \\(h : \\mathbb{R}^n \\times \\mathbb{R}^m \\to \\mathbb{R}\\) is \\((\\rho_1, \\rho_2)\\)-weakly convex on a convex set \\(X \\times Y\\) if \\(h(x, y) + \\frac{\\rho_1}{2} \\|x\\|^2 + \\frac{\\rho_2}{2} \\|y\\|^2\\) is convex on \\(X \\times Y\\). This notion generalizes convexity (\\(\\rho_1 = \\rho_2 = 0\\)). Weakly convex functions \\cite{nurminskii1973quasigradient}, extends the class of convex functions to include many nonconvex functions. For weakly convex functions, the Fréchet (regular) subdifferential coincides with both the limiting subdifferential and the Clarke subdifferential (see, e.g., \\cite[Proposition 3.1 and Theorem 3.6]{ngai2000approximate}).\n\n","start":19440,"end":20203}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","labels":["asup3"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"下层正则性与弱凸性","source_units":[{"raw_tex":"\\begin{assumption}\\label{asup3}\n\tThe lower-level objective functions \\(f(x, y)\\) and \\(g(x, y)\\) satisfy:\n\t\\begin{itemize}\n\t\t\\item[(1)] \\(f(x, y)\\) is continuously differentiable on \\(X \\times Y\\), and its gradients \\(\\nabla_x f\\) and \\(\\nabla_y f\\) are \\(L_{f_x}\\)- and \\(L_{f_y}\\)-Lipschitz continuous on \\(X \\times Y\\), respectively;\n\t\t\\item[(2)] \\(g(x, y)\\) is continuous on \\(X \\times Y\\), \\(\\nabla_x g(x, y)\\) exists, and \\(\\nabla_x g(\\cdot, y)\\) is \\(L_{g_1}\\)-Lipschitz continuous on \\(X\\) for any \\(y \\in Y\\), while \\(\\nabla_x g(x, \\cdot)\\) is \\(L_{g_2}\\)-Lipschitz continuous on \\(Y\\) for any \\(x \\in X\\);\n\t\t\\item[(3)] \\(f(x, y)\\) is \\((\\rho_{f_1}, \\rho_{f_2})\\)-weakly convex on \\(X \\times Y\\), and \\(g(x, y)\\) is \\((\\rho_{g_1}, \\rho_{g_2})\\)-weakly convex on \\(X \\times Y\\).\n\t\\end{itemize}\n\\end{assumption}\n","start":20203,"end":21022}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["proximal_LL"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"精确近端下层解","source_units":[{"raw_tex":"Under Assumption \\ref{asup2}, the solution mapping\n\\begin{equation}\\label{proximal_LL}\n\tS_{\\gamma}(x, y) := \\mathrm{argmin}_{\\theta \\in Y} \\left\\{ f(x, \\theta) + g(x, \\theta) + \\frac{1}{2\\gamma}\\|\\theta - y\\|^2 \\right\\}\n\\end{equation}\nis well-defined and single-valued  on $X \\times \\mathbb{R}^m$. We denote its unique solution by $\\theta_\\gamma^*(x, y)$.\nBelow, we summarize sensitivity results for $v_\\gamma(x, y)$ from Theorems 2, 3, and 5 of \\cite{gao2024}.\n","start":24483,"end":24945}]},{"id":"S014","semantic_type":"derived_fact","kind":"derived_fact","labels":["grad_v","valuefinc"],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"包络梯度表达式","source_units":[{"raw_tex":"\t\\begin{equation}\\label{valuefinc}\n\t\t\\nabla v_\\gamma(x, y) = \\Big( \\nabla_x f(x, \\theta_\\gamma^*(x, y)) + \\nabla_x g(x, \\theta_\\gamma^*(x, y)), \\left({y - \\theta_\\gamma^*(x, y)}\\right)/{\\gamma} \\Big).\n\t\\end{equation}\n","start":25358,"end":25575}]},{"id":"S015","semantic_type":"proposition","kind":"proposition","labels":["partial_grad_v"],"member_ids":["S015","S077","S078","S079"],"declared_external":false,"source_only":false,"title":"包络偏弱凸性与连续性","source_units":[{"raw_tex":"\\begin{proposition}\\label{partial_grad_v}\n\tSuppose Assumptions \\ref{asup2} and \\ref{asup3} hold. For $\\gamma \\in (0, 1/(\\rho_{f_2} + \\rho_{g_2}))$, the function $v_\\gamma(x, y)$ is $\\rho_{v_1}$-weakly convex with respect to $x$ on $X$ for any $y \\in Y$ with $\\rho_{v_1} \\geq \\rho_{f_1} + \\rho_{g_1}$. It is also convex with respect to $y$ on $\\mathbb{R}^m$ for any $x \\in X$. Moreover, $v_\\gamma(x, y)$ is differentiable with gradient given by \\eqref{valuefinc} on $X \\times \\mathbb{R}^m$, and $\\theta_\\gamma^*(x, y)$ and $\\nabla v_\\gamma(x, y)$ are continuous on $X \\times \\mathbb{R}^m$.\n\\end{proposition}","start":26030,"end":26636},{"raw_tex":"\\begin{proof}\n\tWe begin by showing that $v_{\\gamma}(x,y) $ is continuously differentiable on $X \\times \\mathbb{R}^m$. By Assumption \\ref{asup2}, the function $f(x, \\theta) + g(x, \\theta) + \\frac{1}{2\\gamma}\\|\\theta-y\\|^2$ is level-bounded in $\\theta$ locally uniformly for any $(\\bar{x}, \\bar{y}) \\in X \\times \\mathbb{R}^m $. Specifically, for any $c \\in \\mathbb{R}$, there exist a compact set $D$ and a neighborhood $Z$ of $(\\bar{x}, \\bar{y})$ such that the level set $\\{ \\theta \\in \\mathbb{R}^m \\mid f(x, \\theta) + g(x, \\theta) + \\frac{1}{2\\gamma}\\|\\theta-y\\|^2 \\le c\\}$ is contained in $D$ for all $(x,y) \\in Z$. Using Corollary 4.3(ii) from \\cite{mordukhovich2018variational}, it follows that $v_{\\gamma}(x,y) $ is locally Lipschitz continuous on $X \\times \\mathbb{R}^m$. Furthermore, Theorem 1.22, Theorem 4.1(ii) and Theorem 4.17 in \\cite{mordukhovich2018variational} imply that $v_\\gamma({x},{y})$ is differentiable on $X \\times \\mathbb{R}^m$ with its gradient given as in \\eqref{valuefinc}. The partial weak convexity property of $v_\\gamma({x},{y})$ can be established using arguments similar to those in the proof of Theorem 2 in \\cite{gao2024}.\n","start":26638,"end":27793},{"raw_tex":"\\begin{proof}\n\tWe begin by showing that $v_{\\gamma}(x,y) $ is continuously differentiable on $X \\times \\mathbb{R}^m$. By Assumption \\ref{asup2}, the function $f(x, \\theta) + g(x, \\theta) + \\frac{1}{2\\gamma}\\|\\theta-y\\|^2$ is level-bounded in $\\theta$ locally uniformly for any $(\\bar{x}, \\bar{y}) \\in X \\times \\mathbb{R}^m $. Specifically, for any $c \\in \\mathbb{R}$, there exist a compact set $D$ and a neighborhood $Z$ of $(\\bar{x}, \\bar{y})$ such that the level set $\\{ \\theta \\in \\mathbb{R}^m \\mid f(x, \\theta) + g(x, \\theta) + \\frac{1}{2\\gamma}\\|\\theta-y\\|^2 \\le c\\}$ is contained in $D$ for all $(x,y) \\in Z$. Using Corollary 4.3(ii) from \\cite{mordukhovich2018variational}, it follows that $v_{\\gamma}(x,y) $ is locally Lipschitz continuous on $X \\times \\mathbb{R}^m$. Furthermore, Theorem 1.22, Theorem 4.1(ii) and Theorem 4.17 in \\cite{mordukhovich2018variational} imply that $v_\\gamma({x},{y})$ is differentiable on $X \\times \\mathbb{R}^m$ with its gradient given as in \\eqref{valuefinc}. The partial weak convexity property of $v_\\gamma({x},{y})$ can be established using arguments similar to those in the proof of Theorem 2 in \\cite{gao2024}.\n\t\n\tNext, we show that $ \\nabla v_\\gamma({x},{y})$ is continuous on $X \\times \\mathbb{R}^m$ by showing the continuity of $\\theta_{\\gamma}^*(x,y)$ on $X \\times \\mathbb{R}^m$. Let $\\{(x_k, y_k)\\} \\subset X \\times \\mathbb{R}^m$ be any sequence such that $(x_k, y_k) \\rightarrow (\\bar{x}, \\bar{y})$. Since $f(x, \\theta) + g(x, \\theta) + \\frac{1}{2\\gamma}\\|\\theta-y\\|^2$ is level-bounded in $\\theta$ locally uniformly for $(\\bar{x}, \\bar{y}) \\in X \\times \\mathbb{R}^m $, the sequence $\\{\\theta^*_{\\gamma}(x_k,y_k)\\}$ is bounded. Let $\\bar{\\theta}$ be an accumulation point of $\\{\\theta^*_{\\gamma}(x_k,y_k)\\}$. For any $\\theta \\in Y$, we have $f(x_k, \\theta^*_{\\gamma}(x_k,y_k)) + g(x_k, \\theta^*_{\\gamma}(x_k,y_k)) + \\frac{1}{2\\gamma}\\|\\theta^*_{\\gamma}(x_k,y_k)-y_k\\|^2 \\le f(x_k, \\theta) + g(x_k, \\theta) + \\frac{1}{2\\gamma}\\|\\theta-y_k\\|^2$. Taking the limit as $k \\rightarrow \\infty$, and using the continuity of $f$ and $g$, we obtain that for any $\\theta \\in Y$, $f(\\bar{x}, \\bar{\\theta}) + g(\\bar{x}, \\bar{\\theta}) + \\frac{1}{2\\gamma}\\|\\bar{\\theta}-\\bar{y}\\|^2 \\le f(\\bar{x}, \\theta) + g(\\bar{x}, \\theta) + \\frac{1}{2\\gamma}\\|\\theta-\\bar{y}\\|^2$. This implies $\\bar{\\theta} = \\theta^*_{\\gamma}(\\bar{x}, \\bar{y})$. Therefore, $\\theta^*_{\\gamma}(x_k,y_k) \\rightarrow \\theta^*_{\\gamma}(\\bar{x}, \\bar{y})$, establishing the continuity of $\\theta_{\\gamma}^*(x,y)$ on $X \\times \\mathbb{R}^m$.\n\\end{proof}\n","start":26638,"end":29193},{"raw_tex":"\t\n\tNext, we show that $ \\nabla v_\\gamma({x},{y})$ is continuous on $X \\times \\mathbb{R}^m$ by showing the continuity of $\\theta_{\\gamma}^*(x,y)$ on $X \\times \\mathbb{R}^m$. Let $\\{(x_k, y_k)\\} \\subset X \\times \\mathbb{R}^m$ be any sequence such that $(x_k, y_k) \\rightarrow (\\bar{x}, \\bar{y})$. Since $f(x, \\theta) + g(x, \\theta) + \\frac{1}{2\\gamma}\\|\\theta-y\\|^2$ is level-bounded in $\\theta$ locally uniformly for $(\\bar{x}, \\bar{y}) \\in X \\times \\mathbb{R}^m $, the sequence $\\{\\theta^*_{\\gamma}(x_k,y_k)\\}$ is bounded. Let $\\bar{\\theta}$ be an accumulation point of $\\{\\theta^*_{\\gamma}(x_k,y_k)\\}$. For any $\\theta \\in Y$, we have $f(x_k, \\theta^*_{\\gamma}(x_k,y_k)) + g(x_k, \\theta^*_{\\gamma}(x_k,y_k)) + \\frac{1}{2\\gamma}\\|\\theta^*_{\\gamma}(x_k,y_k)-y_k\\|^2 \\le f(x_k, \\theta) + g(x_k, \\theta) + \\frac{1}{2\\gamma}\\|\\theta-y_k\\|^2$. Taking the limit as $k \\rightarrow \\infty$, and using the continuity of $f$ and $g$, we obtain that for any $\\theta \\in Y$, $f(\\bar{x}, \\bar{\\theta}) + g(\\bar{x}, \\bar{\\theta}) + \\frac{1}{2\\gamma}\\|\\bar{\\theta}-\\bar{y}\\|^2 \\le f(\\bar{x}, \\theta) + g(\\bar{x}, \\theta) + \\frac{1}{2\\gamma}\\|\\theta-\\bar{y}\\|^2$. This implies $\\bar{\\theta} = \\theta^*_{\\gamma}(\\bar{x}, \\bar{y})$. Therefore, $\\theta^*_{\\gamma}(x_k,y_k) \\rightarrow \\theta^*_{\\gamma}(\\bar{x}, \\bar{y})$, establishing the continuity of $\\theta_{\\gamma}^*(x,y)$ on $X \\times \\mathbb{R}^m$.\n\\end{proof}\n","start":27793,"end":29193}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"松弛问题的KKT点","source_units":[{"raw_tex":"\\begin{definition}\n\tLet $(\\bar{x},\\bar{y})$ be a feasible solution of problem $({\\rm VP})_{\\gamma}^{\\epsilon}$ with $\\epsilon > 0$. We say that $(\\bar{x},\\bar{y})$ is a stationary/KKT point of problem $({\\rm VP})_{\\gamma}^{\\epsilon}$ if there exists a multiplier $\\lambda \\ge 0$ such that\n\t\\begin{equation*}\n\t\t\\begin{cases}\n\t\t\t0\\in \\nabla F(\\bar{x},\\bar{y}) + \\lambda \\left(\\nabla f(\\bar{x},\\bar{y}) + \\partial g(\\bar{x},\\bar{y}) - \\nabla v_{\\gamma}(\\bar{x},\\bar{y})\\right) + \\mathcal{N}_{X\\times Y}(\\bar{x},\\bar{y}), \\\\\n\t\t\tf(\\bar{x},\\bar{y}) + g(\\bar{x},\\bar{y}) - v_{\\gamma}(\\bar{x},\\bar{y}) \\leq \\epsilon, \\qquad \\lambda \\left(f(\\bar{x},\\bar{y}) + g(\\bar{x},\\bar{y}) - v_{\\gamma}(\\bar{x},\\bar{y})-\\epsilon\\right) = 0.\n\t\t\\end{cases}\n\t\\end{equation*}\n\\end{definition}\n","start":34074,"end":34843}]},{"id":"S017","semantic_type":"definition","kind":"definition","labels":["penprob"],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"缩放罚目标","source_units":[{"raw_tex":"In this section, we propose a new gradient-type algorithm, Alternating Gradient-type algorithm with Inexact Lower-level Solutions (AGILS), to solve ${\\rm (VP)}_\\gamma^\\epsilon$. Given that $f(x, y) + g(x,y) - v_{\\gamma}(x, y)$ is always nonnegative, a natural approach is to consider the following penalized problem,\n\\begin{equation}\\label{penprob}\n\t\\min_{x \\in X, y \\in Y}\\psi_{p_k}\t(x, y) := \\frac{1}{p_k}F(x, y) + f(x, y) + g(x,y) - v_{\\gamma}(x, y),\n\\end{equation}\nwhere $p_k$ is the penalty parameter at iteration $k$. Due to the nonsmoothness of $g(x,y)$ with respect to $y$ and the availability of its proximal operator, it is natural to adopt a proximal alternating linearization method \\cite{bolte2014proximal} to construct the algorithm. Given the current iterate $(x^k, y^k)$, linearizing the smooth components of $\\psi_{p_k}\t(x, y)$ requires evaluating gradients, including $\\nabla v_{\\gamma}(x, y)$ as provided in \\cref{grad_v}. Calculating the exact gradient  $\\nabla v_{\\gamma}(x^k, y^k)$ requires solving the proximal lower-level problem in \\eqref{proximal_LL} to obtain $\\theta_\\gamma^*(x^k, y^k)$. However, this solution typically does not have a closed-form expression, and solving it requires an iterative optimization solver, which can be computationally expensive. To address this, we propose an inexact gradient approximation by substituting $\\theta_{\\gamma}^*$ with an inexact estimate $\\theta^k$.\n\nThe proposed algorithm alternates between updating $y$ and $x$ while refining the inexact approximation of $\\theta_\\gamma^*(x^k, y^k)$. At each iteration $k=0,1,\\ldots$, given the current iterate $(x^k, y^k)$, along with the feasibility-corrected iterate $\\tilde{y}^k$ and corresponding $\\tilde{\\theta}^k$ (constructed from $y^k$ and  $\\theta^{k}$ via the procedure detailed below), the next iterate $(x^{k+1}, y^{k+1})$ and the refined approximation $\\theta^{k+1}$ are updated as follows.\n","start":34884,"end":36797}]},{"id":"S018","semantic_type":"algorithm","kind":"algorithm","labels":["dy","update_y","update_y2"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"校正起点的y更新","source_units":[{"raw_tex":"\n{\\bf Update $y$ (Fixing $x$).}\nWe compute the update direction for $y$,\n\\begin{equation}\\label{dy}\n\td_y^k:= \\frac{1}{p_k} \\nabla_y F\\left(x^{k}, \\tilde{y}^k\\right)+\\nabla_y f\\left(x^{k}, \\tilde{y}^k\\right)-\\frac{\\tilde{y}^k-\\tilde{\\theta}^{k}}{\\gamma},\n\\end{equation}\nwhere $(\\tilde{y}^k-\\tilde{\\theta}^{k})/{\\gamma}$ is an inexact approximation of $\\nabla_y v_{\\gamma}(x^{k}, \\tilde{y}^k) =( {\\tilde{y}^k} -  \\theta_{\\gamma}^*(x^{k}, \\tilde{y}^k))/ \\gamma $ (see (\\ref{valuefinc})). Using $\td_y^k$, the variable $y$ is updated as\n\\begin{equation}\\label{update_y}\n\ty^{k+1} = \\underset{y \\in Y}{\\mathrm{argmin}} ~ \\left\\langle d_y^k, y - \\tilde{y}^{k} \\right\\rangle + g(x^k, y) + \\frac{1}{2\\beta_k} \\left\\| y - \\tilde{y}^k \\right\\|^2, \n\\end{equation}\nwhere $\\beta_k>0$ is the stepsize. \nThis can also be expressed using the proximal operator\n\\begin{equation}\\label{update_y2}\n\ty^{k+1} = \\operatorname{Prox}_{\\beta_k\\tilde{g}(x^{k},\\cdot)}\\left(\\tilde{y}^k-\\beta_k d^k_y\\right),\n\\end{equation}\nwhere $\\tilde{g}(x,y) := g(x,y) + \\delta_Y(y)$. \n","start":36797,"end":37839}]},{"id":"S019","semantic_type":"algorithm","kind":"algorithm","labels":["inexacty1","theta_y"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"半步近端解误差准则","source_units":[{"raw_tex":"\n{\\bf Find inexact approximation $\\theta^{k+1/2}$.}\nAfter updating $y^{k+1}$, we compute an inexact approximation $\\theta^{k+1/2}$ for $\\theta_{\\gamma}^*(x^k, y^{k+1})$ by solving\n\\begin{equation}\\label{theta_y}\n\t\\theta^{k+1/2} \\approx \t\\underset{\\theta \\in Y}{\\mathrm{argmin}} \\left\\{f(x^{k},\\theta) + g(x^{k}, \\theta) + \\frac{1}{2\\gamma}\\left\\|\\theta-y^{k+1}\\right\\|^2\\right\\}, \n\\end{equation}\nsubject to either an absolute or a relative inexactness criterion\n\\begin{equation}\\label{inexacty1}\n\t\\begin{aligned}\n\t\t\\mathcal{G}(\\theta^{k+1/2}, x^k,y^{k+1}) \\le s_k, \\quad \\text{or} \\quad \\mathcal{G}(\\theta^{k+1/2}, x^k,y^{k+1}) \\le \\tau_k \\mathcal{G}(\\theta^{k-1}, x^{k-1},y^{k-1}),\n\t\\end{aligned}\n\\end{equation}\nwhich can be compactly expressed as\n\\[\n\\mathcal{G}(\\theta^{k+1/2}, x^{k}, y^{k+1})\n\\le\n\\max\\left\\{ s_k,\\ \\tau_k\\,\\mathcal{G}(\\theta^{k-1}, x^{k-1}, y^{k-1}) \\right\\}.\n\\]\n","start":37839,"end":38722}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"近端梯度残差与误差序列","source_units":[{"raw_tex":"The parameters $s_k$ and $\\tau_k$ control the solution's inexactness, and are required to satisfy $\\sum_{k=0}^{\\infty}s_k^2<\\infty$ and $\\tau_k:=\\tau_0/(k+1)^{p_\\tau}$ with constants $\\tau_0 > 0$ and $p_\\tau >0$. The prox-gradient residual $\\mathcal{G}(\\theta, x,y)$ quantifies the inexactness and is defined as\n\\[\n\\mathcal{G}(\\theta, x,y):= \t\\left\\| \\theta -\\operatorname{Prox}_{\\eta \\tilde{g}\\left(x, \\cdot\\right)}\\left(\\theta-\\eta\\left(\\nabla_y f\\left(x, \\theta\\right) + (\\theta -y)/ {\\gamma}\\right)\\right)\\right\\|,\n\\]\nwhere $\\eta > 0$. Note that $\\mathcal{G}(\\theta, x,y) = 0$ if and only if $\\theta =\\theta_{\\gamma}^*(x,y)$. \n","start":38722,"end":39353}]},{"id":"S021","semantic_type":"algorithm","kind":"algorithm","labels":["dx","update_x"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"x方向与投影更新","source_units":[{"raw_tex":"{\\bf Update $x$ (Fixing $y$).}\nWe compute the update direction for $x$,\n\\begin{equation}\\label{dx}\n\td_x^k:= \\frac{1}{p_k} \\nabla_xF\\left(x^k, y^{k+1}\\right)+\\nabla_x \\varphi \\left(x^k, y^{k+1}\\right)-\\nabla_x f\\left(x^k, \\theta^{k+1/2}\\right)-\\nabla_x g\\left(x^k, \\theta^{k+1/2}\\right),\n\\end{equation}\nwhere $\\nabla_x f\\left(x^k, \\theta^{k+1/2}\\right)+\\nabla_x g\\left(x^k, \\theta^{k+1/2}\\right)$ is an approximation to \\( \\nabla_x v_{\\gamma}(x^k, y^{k+1}) = \\nabla_x f(x^k, \\theta^*_{\\gamma}(x^k, y^{k+1}))+\\nabla_x g(x^k, \\theta^*_{\\gamma}(x^k, y^{k+1}))\\) (see \\eqref{valuefinc}).\nUsing the direction $d_x^k$, we update $x$ with stepsize $\\alpha_k>0$ as\n\\begin{equation}\\label{update_x}\n\tx^{k+1}=\\operatorname{Proj}_X\\left(x^k-\\alpha_k d_x^k\\right).\n\\end{equation}\n","start":39807,"end":40574}]},{"id":"S022","semantic_type":"algorithm","kind":"algorithm","labels":["inexacty2"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"整步近端解误差准则","source_units":[{"raw_tex":"\n{\\bf Find inexact approximation $\\theta^{k+1}$.}\nAfter updating $x^{k+1}$, a new inexact approximation $\\theta^{k+1}$ of $\\theta^*_{\\gamma}(x^{k+1}, y^{k+1})$  is computed by approximately solving\n\\[\n\\theta^{k+1} \\approx \t\\underset{\\theta \\in Y}{\\mathrm{argmin}} \\left\\{f(x^{k+1},\\theta) + g(x^{k+1}, \\theta) + \\frac{1}{2\\gamma}\\left\\|\\theta-y^{k+1}\\right\\|^2\\right\\}, \n\\]\nsuch that $\\theta^{k+1}$ satisfies either an absolute or a relative inexactness criterion\n\\begin{equation}\\label{inexacty2}\n\t\\begin{aligned}\n\t\t\\mathcal{G}(\\theta^{k+1}, x^{k+1},y^{k+1}) \\le s_{k+1}, \\quad \\text{or} \\quad  \\mathcal{G}(\\theta^{k+1}, x^{k+1},y^{k+1}) \\le \\tau_{k+1}\\mathcal{G}(\\theta^{k}, x^{k},y^{k}).\n\t\\end{aligned}\n\\end{equation}\n","start":40574,"end":41295}]},{"id":"S023","semantic_type":"algorithm","kind":"algorithm","labels":["def_t","update_p_1"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"违约估计与增罚判据","source_units":[{"raw_tex":"{\\bf Penalty parameter update and feasibility correction.}\nTo encourage feasibility of the constraint $\\varphi(x, y) - v_{\\gamma}(x, y) \\le \\epsilon$, the penalty parameter $p_k$ is updated at each iteration.\nEvaluating the constraint violation requires computing $ v_{\\gamma}(x, y)$, which involves solving the proximal lower-level problem in \\eqref{proximal_LL}. Instead, we use $\\theta^{k+1}$ to estimate the constraint violation as\n\\begin{equation}\\label{def_t}\n\tt^{k+1}=\\max\\left\\{\\varphi(x^{k+1},y^{k+1})-\\varphi(x^{k+1},\\theta^{k+1})-\\frac{1}{2\\gamma}\\| \\theta^{k+1} - y^{k+1}\\|^2-\\epsilon,0\\right\\}.\n\\end{equation}\nThe penalty parameter \\(p_k\\) is increased to $p_{k+1}=\tp_k + \\varrho_p$ (for some $\\varrho_p > 0$) only when the change in the iterates, $\\Delta_{k+1} := \\left\\| (x^{k+1}, y^{k+1}) - (x^k, \\tilde{y}^k) \\right\\|$, is small relative to the penalty parameter (i.e., $\\Delta_{k+1} < c_p/p_k$ for some $c_p > 0$), while the estimated constraint violation $t^{k+1}$ is large relative to this change (i.e., $t^{k+1} > \\Delta_{k+1}/c_p$).\nThese two conditions can be compactly written as \\begin{equation}\\label{update_p_1}\n\t\\Delta_{k+1} < c_p \\min\\left\\{1/p_k,{t^{k+1}}\\right\\}.\n\\end{equation}\n","start":41581,"end":42791}]},{"id":"S025","semantic_type":"algorithm","kind":"algorithm","labels":["theta_gap_condition"],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"校正触发阈值","source_units":[{"raw_tex":"The feasibility correction procedure aims to generate a corrected iterate $\\tilde{y}^{k+1} $ from $y^{k+1}$. It is based on the observation that if $(x^{k+1},y^{k+1})$ is feasible, then $y^{k+1}$ should be a reasonably accurate solution to the lower-level problem \\eqref{LL}  for $x= x^{k+1}$, implying that $\\| \\nabla_y v_{\\gamma}(x^{k+1},y^{k+1})\\| $ should be small. We use the magnitude of $( y^{k+1} - \\theta^{k+1})/\\gamma$ to approximate $\\| \\nabla_y v_{\\gamma}(x^{k+1},y^{k+1})\\| $. The feasibility correction  is triggered if:\n\\begin{equation}\\label{theta_gap_condition}\n\t\\left\\| y^{k+1} - \\theta^{k+1} \\right\\| \\ge \\frac{c_y \\gamma}{p_k},\n\\end{equation}\nwhere $c_y > 0$. Condition \\eqref{theta_gap_condition} indicates that the iterate $(x^{k+1},y^{k+1})$ might be approaching an undesirable stationary point of the constraint function.\n","start":43284,"end":44130}]},{"id":"S026","semantic_type":"algorithm","kind":"algorithm","labels":["inexacty_y"],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"下层校正残差要求","source_units":[{"raw_tex":"\nIf condition \\eqref{theta_gap_condition} holds, we attempt to correct $y^{k+1}$ by finding a candidate iterate $\\tilde{y}^{k+1} $  as an inexact solution to the lower-level problem \\eqref{LL} with $x= x^{k+1}$ satisfying\n\\begin{equation}\\label{inexacty_y}\n\t\\left\\| \\tilde{y}^{k+1} - \\operatorname{Prox}_{ \\tilde{g}(x^{k+1}, \\cdot)} \\left( \\tilde{y}^{k+1} -  \\nabla_y f(x^{k+1}, \\tilde{y}^{k+1}) \\right) \\right\\| \\le \\frac{c_{\\tilde{y}} }{p_k} \\Delta_{k+1}, \n\\end{equation}\n","start":44130,"end":44604}]},{"id":"S027","semantic_type":"algorithm","kind":"algorithm","labels":["decrease_new_y"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"校正候选的接受不等式","source_units":[{"raw_tex":"where $c_{\\tilde{y}} > 0$. However, unconditionally accepting such a $\\tilde{y}^{k+1} $ could adversely affect the algorithm's convergence. Therefore, to preserve convergence, $\\tilde{y}^{k+1} $ is accepted only if it satisfies the following descent condition\n\\begin{equation}\\label{decrease_new_y}\n\t\\begin{aligned}\n\t\t&\\frac{1}{p_k}F(x^{k+1}, \\tilde{y}^{k+1}) + \\varphi(x^{k+1}, \\tilde{y}^{k+1}) - \\left( \\varphi(x^{k+1},  \\tilde{\\theta}^{k+1} ) + \\frac{1}{2\\gamma} \\| \\tilde{\\theta}^{k+1} - \\tilde{y}^{k+1} \\|^2 \\right)\\\\\n\t\t\\le\\; &\n\t\t\\frac{1}{p_k}F(x^{k+1}, y^{k+1}) + \\varphi(x^{k+1}, y^{k+1}) - \\left( \\varphi(x^{k+1}, \\theta^{k+1}) + \\frac{1}{2\\gamma} \\| \\theta^{k+1} - y^{k+1} \\|^2 \\right),\n\t\\end{aligned}\n\\end{equation}\n","start":44604,"end":45330}]},{"id":"S028","semantic_type":"algorithm","kind":"algorithm","labels":["inexacty3"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"校正近端解的一步细化","source_units":[{"raw_tex":"where, $\\tilde{\\theta}^{k+1}$ is a new approximation of $\\theta^*_{\\gamma}(x^{k+1}, \\tilde{y}^{k+1})$. We construct $\\tilde{\\theta}^{k+1}$ by first finding a point $\\hat{\\theta}^{k+1}$ that satisfies the inexactness criterion\n\\begin{equation}\\label{inexacty3}\n\t\\mathcal{G}(\\hat{\\theta}^{k+1}, x^{k+1},\\tilde{y}^{k+1}) \\le s_{k+1},\n\t\\quad \\text{or} \\quad\n\t\\mathcal{G}(\\hat{\\theta}^{k+1}, x^{k+1},\\tilde{y}^{k+1})\n\t\\le \\tau_{k+1}\\mathcal{G}(\\theta^{k}, x^{k},y^{k}).\n\\end{equation}\nand subsequently applying a single proximal-gradient step to $\\hat{\\theta}^{k+1}$\n\\[\n\\tilde{\\theta}^{k+1}\n=\n\\operatorname{Prox}_{\\eta \\tilde g(x^{k+1},\\cdot)}\n\\left(\n\\hat{\\theta}^{k+1}\n-\\eta\\left(\n\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n+\\frac{\\hat{\\theta}^{k+1}-\\tilde y^{k+1}}{\\gamma}\n\\right)\n\\right).\n\\]\n","start":45330,"end":46116}]},{"id":"S029","semantic_type":"algorithm","kind":"algorithm","labels":[],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"接受与拒绝后的状态","source_units":[{"raw_tex":"\nIf the descent condition \\eqref{decrease_new_y} holds, the algorithm adopts the candidate  $\\tilde{y}^{k+1} $ along with its corresponding $\\tilde{\\theta}^{k+1}$, and keep the penalty parameter unchanged, i.e., $p_{k+1}=\tp_k$. Otherwise, the correction is discarded, meaning $\\tilde{y}^{k+1} = y^{k+1}$ and $\\tilde{\\theta}^{k+1} = \\theta^{k+1}$ and the penalty parameter is increased, i.e., $p_{k+1}=\tp_k + \\varrho_p$. The algorithm then proceeds with the selected pair $(\\tilde{y}^{k+1}, \\tilde{\\theta}^{k+1})$. A detailed description of this feasibility correction procedure is presented in Algorithm \\ref{alg2}.\n","start":46116,"end":46732}]},{"id":"S030","semantic_type":"algorithm","kind":"algorithm","labels":["alg1"],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"AGILS完整流程","source_units":[{"raw_tex":"\\begin{algorithm}[htp!]\n\\caption{Alternating Gradient-type algorithm with Inexact Lower-level Solutions (AGILS)}\n\\label{alg1}\n\\begin{algorithmic}[1]\n\t\\STATE{{\\bf Input}: Initial iterates $x^0,\\theta^{-1}, y^0, \\tilde{\\theta}^0, \\tilde{y}^0$, set $(x^{-1}, y^{-1}, \\theta^{-1}) = (x^0, y^0, \\tilde{\\theta}^0)$, stepsizes $\\alpha_k, \\beta_k$, relaxation parameter $\\epsilon$, proximal parameter $\\gamma$, penalty parameters $p_{0}$,  $\\varrho_p$, $c_p$, $c_y$, inexact parameters $\\eta$, $s_{k}$, $\\tau_k$, tolerance $tol$.}\n\t\\FOR{$k=0,\\ 1,\\cdots$}\n\t\\STATE{Construct the direction $d_y^k$ according to \\eqref{dy}, and update $y^{k+1}$ as\n\t\t$$\n\t\ty^{k+1} = \\operatorname{Prox}_{\\beta_k\\tilde{g}(x^{k,\\cdot})}\\left(\\tilde{y}^k-\\beta_k d^k_y\\right).\n\t\t$$}\n\t\n\t\\STATE{Find an inexact solution $\\theta^{k+1/2}$ that satisfies the inexactness criterion \\eqref{inexacty1}.}\n\t\n\t\\STATE{Construct the direction $d_x^k$ according to \\eqref{dx}, and update $x^{k+1}$ as \n\t\t$$ x^{k+1}=\\operatorname{Proj}_X\\left(x^k-\\alpha_k d_x^k\\right). \n\t\t$$}\n\t\n\t\\STATE{Find an inexact solution $\\theta^{k+1}$ that satisfies the inexactness criterion \\eqref{inexacty2}.}\n\t\n\t\\STATE{Calculate $\\Delta_{k+1} := \\left\\| (x^{k+1}, y^{k+1}) - (x^k, \\tilde{y}^k) \\right\\|$ and $t^{k+1}$ according to \\eqref{def_t}. \\\\\n\t\tStop when $k \\ge 1$ and \\(\\max\\{\\Delta_{k+1}, s_{k}, t^{k+1} \\} \\le tol.\\)\n\t}\n\t\\STATE{Update penalty parameter and apply feasibility correction:}\n\t\n\t\\STATE{\\hspace*{10pt}\\textbf{Case 1:}\n\t\t$\\Delta_{k+1} \\geq c_p \\min\\big\\{1/p_k,\\, t^{k+1} \\big\\}$ \\\\\n\t\t\\vspace*{1pt}\n\t\t\\qquad\\qquad\\quad Set \\( p_{k+1} := p_k \\), \\( \\tilde{y}^{k+1} := y^{k+1} \\), \\(\\tilde{\\theta}^{k+1} :=\\theta^{k+1}\\).}\n\t\\vspace*{2pt}\n\t\\STATE{\\hspace*{10pt}\\textbf{Case 2:} \n\t\t$\\Delta_{k+1} < c_p \\min\\big\\{1/p_k,\\, t^{k+1} \\big\\},\\,\\| y^{k+1} - \\theta^{k+1} \\| \\le c_y \\gamma/p_k$\\\\  \n\t\t\\vspace*{1pt}\n\t\t\\qquad\\qquad\\quad Set \\( p_{k+1} := p_k + \\varrho_p \\), \\( \\tilde{y}^{k+1} := y^{k+1} \\), \\(\\tilde{\\theta}^{k+1} :=\\theta^{k+1}\\).}\n\t\\vspace*{2pt}\n\t\n\t\\STATE{\\hspace*{10pt}\\textbf{Case 3:} \n\t\t$\\Delta_{k+1} < c_p \\min\\big\\{1/p_k,\\, t^{k+1} \\big\\},\\,\\| y^{k+1} - \\theta^{k+1} \\| > c_y \\gamma/p_k$\\\\\n\t\t\\vspace*{1pt}\n\t\t\\qquad\\qquad\\quad Apply the feasibility correction procedure (Algorithm~\\ref{alg2}).}\n\\ENDFOR\n\\RETURN $(x^{k+1},\\tilde{y}^{k+1})$\n\\end{algorithmic}\n\\end{algorithm}\n","start":47378,"end":49710}]},{"id":"S031","semantic_type":"algorithm","kind":"algorithm","labels":["alg2"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"可行性校正完整流程","source_units":[{"raw_tex":"\n\\begin{algorithm}[htp!]\n\\caption{Feasibility Correction Procedure}\n\\label{alg2}\n\\begin{algorithmic}[1]\n\n\\STATE{{\\bf Input}: Current iterates $x^{k+1}, y^{k+1}, \\theta^{k+1}$, $\\Delta_{k+1}$, proximal parameter $\\gamma$, penalty parameters $p_{k}$, $\\varrho_p$, inexact parameters $\\eta$, $s_{k}$, $\\tau_k$, constant $c_{\\tilde{y}}$.}\n\n\\STATE{Compute a candidate $\\tilde{y}^{k+1}$ by inexactly solving the lower-level problem \\eqref{LL} with $x=x^{k+1}$  to satisfy \\eqref{inexacty_y}.\n}\n\n\\STATE{Find an intermediate point $\\hat{\\theta}^{k+1}$ that satisfies the inexactness criterion \\eqref{inexacty3}, and compute $\\tilde{\\theta}^{k+1}=\n\\operatorname{Prox}_{\\eta \\tilde g(x^{k+1},\\cdot)}\n(\n\\hat{\\theta}^{k+1}\n-\\eta(\n\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n+(\\hat{\\theta}^{k+1}-\\tilde y^{k+1})/\\gamma\n)\n)$.}\n\n\\IF{the descent condition \\eqref{decrease_new_y} holds}\n\\STATE{Accept the candidate: Set $p_{k+1} := p_k$.}\n\\ELSE\n\\STATE{Reject the candidate: Set $\\tilde{y}^{k+1} := y^{k+1}$, $\\tilde{\\theta}^{k+1} := \\theta^{k+1}$ and $p_{k+1} := p_k + \\varrho_p$.}\n\\ENDIF\n\\RETURN $(\\tilde{y}^{k+1},\\tilde{\\theta}^{k+1},p_{k+1})$\n\\end{algorithmic}\n\\end{algorithm}\n","start":49710,"end":50868}]},{"id":"S032","semantic_type":"lemma","kind":"lemma","labels":["error_bound"],"member_ids":["S032"],"declared_external":true,"declaration_text":["原前提列表允许 B0069；来源为 drusvyatskiy2018error, Theorems 3.4–3.5，或 mordukhovich2023globally, Proposition 2.2；作者在第525行明确归因，文内未给完整证明。"],"source_only":false,"title":"近端残差的距离误差界","source_units":[{"raw_tex":"We begin by considering the following lemma, which provides an estimate for the distance between the approximation $\\theta$ and the exact solution $\\theta^*_{\\gamma}(x,y)$ of the proximal lower-level problem, based on the prox-gradient residual $\\mathcal{G}(\\theta, x,y)$ in the inexact criteria \\eqref{inexacty1} and \\eqref{inexacty2}.  This estimate is derived from the $1/\\gamma$-strong convexity and $(1/\\gamma + L_f)$-smoothness of the component $f(x, \\theta) + \\|\\theta - y\\|^2/(2\\gamma)$ with respect to $\\theta$ in the objective of the proximal lower level problem, see, e.g., \\cite[Theorem 3.4 and 3.5]{drusvyatskiy2018error} or \\cite[Proposition 2.2]{mordukhovich2023globally}.\n\\begin{lemma}\\label{error_bound}\n\tSuppose Assumptions \\ref{asup2} and \\ref{asup3} hold, and let $\\gamma > 0$, $\\eta > 0$. Then, there exists $C_\\eta > 0$ such that for any $(x,y) \\in X \\times \\Re^m$, and $\\theta \\in \\Re^m$, it holds that\n\t\\[\n\t\\|\\theta - \\theta^*_{\\gamma}(x,y) \\| \\le C_\\eta  \\left\\|\\theta-\\operatorname{Prox}_{\\eta \\tilde{g}\\left(x, \\cdot\\right)}\\left(\\theta-\\eta\\left(\\nabla_y f\\left(x, \\theta\\right)+ (\\theta-y)/{\\gamma}\\right)\\right)\\right\\|,\n\t\\]\n\twhere $\\theta^*_{\\gamma}(x,y) := \\arg\\min_{\\theta \\in Y} \\{ \\varphi (x,\\theta)+\\frac{1}{2\\gamma}\\|\\theta-y\\|^2 \\}$.\n\\end{lemma}\n","start":51250,"end":52534}]},{"id":"S033","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"平移后的非负价值函数","source_units":[{"raw_tex":"We now proceed to establish a decreasing property for the proposed method based on the following merit function:\n\\begin{equation*}\n\t\\tilde{\\psi}_{p_{k}}(x,y) :=\\frac{1}{p_k}(F(x,y)-\\underline{F})+ f(x,y) + g(x,y)-v_{\\gamma}(x,y).\n\\end{equation*}\n\nBefore establishing the main decreasing property, we first demonstrate that the descent condition \\eqref{decrease_new_y}, imposed by the feasibility correction procedure (Algorithm \\ref{alg2}), ensures a decrease in $\\tilde{\\psi}_{p_{k}}$ when moving from $y^{k+1}$ to the corrected iterate $\\tilde{y}^{k+1} $.\n","start":53837,"end":54395}]},{"id":"S034","semantic_type":"lemma","kind":"lemma","labels":["lem0","lem1_eq12"],"member_ids":["S034","S080","S081","S082","S083","S084","S085"],"declared_external":false,"source_only":false,"title":"校正步的近似下降","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem0}\n\tSuppose Assumptions \\ref{asup2} and \\ref{asup3} hold. Then, there exists $C_\\gamma>0$,  such that the sequence $\\{(x^k,y^k,\\tilde{y}^k)\\}$ generated by AGILS (\\cref{alg1}) satisfies\n\t\\begin{equation}\\label{lem1_eq12}\n\t\t\\tilde{\\psi}_{p_k}(x^{k+1}, \\tilde{y}^{k+1}) \\le  \\tilde{\\psi}_{p_k}(x^{k+1}, y^{k+1}) + C_{\\gamma} \\zeta_{k+1}^2,\n\t\\end{equation}\n\twhere  $\\zeta_{k} := \\max\\{ s_k,  ( \\prod_{i=0}^{k} \\tau_i ) \\mathcal{G}(\\theta^{0}, x^{0}, y^{0})\\}$.\n\\end{lemma}","start":54395,"end":54887},{"raw_tex":"\\begin{proof}\n\tAt a given iteration $k$, if $\\tilde{y}^{k+1} = y^{k+1}$, the inequality holds trivially.\n\t\n\tNow, consider the case where $\\tilde{y}^{k+1} \\neq y^{k+1}$. This implies that the feasibility correction procedure (Algorithm \\ref{alg2}) is applied at iteration $k$. Accordingly, the corrected iterate  $\\tilde{y}^{k+1}$ satisfies condition \\eqref{decrease_new_y}, and its corresponding $\\tilde{\\theta}^{k+1}$ satisfies condition \\eqref{inexacty3}. Condition \\eqref{inexacty3} implies that $\\mathcal{G}(\\hat{\\theta}^{k+1}, x^{k+1}, \\tilde{y}^{k+1}) \\le \\zeta_{k+1}$.\n","start":54888,"end":55464},{"raw_tex":"\\begin{proof}\n\tAt a given iteration $k$, if $\\tilde{y}^{k+1} = y^{k+1}$, the inequality holds trivially.\n\t\n\tNow, consider the case where $\\tilde{y}^{k+1} \\neq y^{k+1}$. This implies that the feasibility correction procedure (Algorithm \\ref{alg2}) is applied at iteration $k$. Accordingly, the corrected iterate  $\\tilde{y}^{k+1}$ satisfies condition \\eqref{decrease_new_y}, and its corresponding $\\tilde{\\theta}^{k+1}$ satisfies condition \\eqref{inexacty3}. Condition \\eqref{inexacty3} implies that $\\mathcal{G}(\\hat{\\theta}^{k+1}, x^{k+1}, \\tilde{y}^{k+1}) \\le \\zeta_{k+1}$.\n\n\tBy convexity of the lower-level objective in \\(\\theta\\), the function $f(x,\\theta) + \\tilde{g}(x,\\theta) + \\frac{1}{2\\gamma}\\|\\theta-y\\|^2$ is $1/\\gamma$-strongly convex with respect to $\\theta$ on $Y$. Thus, it satisfies a global quadratic growth condition and, by \\cite[Proposition 2]{ye2021variational}, a global KL property with exponent $1/2$ on $Y$. Noting that $\\tilde{g}(x,\\theta) = g(x,\\theta) + \\delta_Y(\\theta)$, we have $\\tilde{g}(x,\\theta) = g(x,\\theta)$ for all $\\theta \\in Y$. This yields \n\t\\begin{equation}\\label{KL}\n\t\\begin{aligned}\n\t\tf(x,\\theta)+g(x,\\theta)+\\frac{1}{2\\gamma}\\|\\theta-y\\|^2-v_\\gamma(x,y)\\le\n\t\t\\frac{\\gamma}{2}\\,\n\t\t\\operatorname{dist}\\Bigl(\n\t\t0,\\,\n\t\t\\nabla_y f(x,\\theta)+\\frac{1}{\\gamma}(\\theta-y)+\\partial \\tilde g(x,\\theta)\n\t\t\\Bigr)^2.\n\t\\end{aligned}\n\\end{equation}\nThe optimality condition of the proximal step defining $\\tilde{\\theta}^{k+1}$,\n\\[\n\\tilde{\\theta}^{k+1}\n=\n\\operatorname{Prox}_{\\eta \\tilde g(x^{k+1},\\cdot)}\n\\left(\n\\hat{\\theta}^{k+1}\n-\\eta\\left(\n\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n+\\frac{1}{\\gamma}(\\hat{\\theta}^{k+1}-\\tilde y^{k+1})\n\\right)\n\\right),\n\\]\nyields\n\\[\n-\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n-\\frac{1}{\\gamma}(\\hat{\\theta}^{k+1}-\\tilde y^{k+1})\n-\\frac{1}{\\eta}(\\tilde{\\theta}^{k+1}-\\hat{\\theta}^{k+1})\n\\in\n\\partial \\tilde g(x^{k+1},\\tilde{\\theta}^{k+1}).\n\\]\nAdding $\\nabla_y f(x^{k+1},\\tilde{\\theta}^{k+1})+(\\tilde{\\theta}^{k+1}-\\tilde y^{k+1})/\\gamma$ to both sides gives\n\\[\n\\begin{aligned}\n\t&\\nabla_y f(x^{k+1},\\tilde{\\theta}^{k+1})\n\t+\\frac{1}{\\gamma}(\\tilde{\\theta}^{k+1}-\\tilde y^{k+1}) -\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n\t-\\frac{1}{\\gamma}(\\hat{\\theta}^{k+1}-\\tilde y^{k+1})\n\t-\\frac{1}{\\eta}(\\tilde{\\theta}^{k+1}-\\hat{\\theta}^{k+1}) \\\\\n\t\\in \\ &\n\t\\nabla_y f(x^{k+1},\\tilde{\\theta}^{k+1})\n\t+\\frac{1}{\\gamma}(\\tilde{\\theta}^{k+1}-\\tilde y^{k+1})\n\t+\\partial \\tilde g(x^{k+1},\\tilde{\\theta}^{k+1}).\n\\end{aligned}\n\\]\n\tBy the $L_{f_y}$-Lipschitz continuity of $\\nabla_y f(x,\\cdot)$ and the definition of the gradient mapping $\\mathcal{G}$, the norm of the left-hand element is bounded by\n\t\\[\n\t\\begin{aligned}\n\t\t&\\quad\\ \n\t\t\\Bigl\\|\n\t\t\\nabla_y f(x^{k+1},\\tilde{\\theta}^{k+1})\n\t\t-\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n\t\t+\\frac{1}{\\gamma}(\\tilde{\\theta}^{k+1}-\\hat{\\theta}^{k+1})\n\t\t+\\frac{1}{\\eta}(\\hat{\\theta}^{k+1}-\\tilde{\\theta}^{k+1})\n\t\t\\Bigr\\| \\\\\n\t\t&\\le\n\t\t\\Bigl(L_{f_y}+\\frac{1}{\\gamma}+\\frac{1}{\\eta}\\Bigr)\n\t\t\\|\\tilde{\\theta}^{k+1}-\\hat{\\theta}^{k+1}\\| \\\\\n\t\t&=\n\t\t\\Bigl(L_{f_y}+\\frac{1}{\\gamma}+\\frac{1}{\\eta}\\Bigr)\n\t\t\\mathcal G(\\hat{\\theta}^{k+1},x^{k+1},\\tilde y^{k+1}),\n\t\\end{aligned}\n\t\\]\nTherefore, by the definition of the distance from the origin to a set,\n\t$$\n\t\\begin{aligned}\n\t\t&\\operatorname{dist}\\left(0, \\nabla_y f(x^{k+1},\\tilde{\\theta}^{k+1}) + \\frac{1}{\\gamma}(\\tilde{\\theta}^{k+1}-\\tilde{y}^{k+1}) + \\partial \\tilde{g}(x^{k+1},\\tilde{\\theta}^{k+1})\\right) \\\\\n\t\t&\\le \\left(L_{f_y} + \\frac{1}{\\gamma} + \\frac{1}{\\eta}\\right) \\mathcal{G}(\\hat{\\theta}^{k+1}, x^{k+1}, \\tilde{y}^{k+1}).\n\t\\end{aligned}\n\t$$\n\tApplying the KL inequality \\eqref{KL} with $(x^{k+1}, \\tilde{y}^{k+1}, \\tilde{\\theta}^{k+1})$ and substituting the above estimate gives\n\t$$\n\t\\begin{aligned}\n\t\t&f(x^{k+1},\\tilde{\\theta}^{k+1}) + g(x^{k+1},\\tilde{\\theta}^{k+1}) + \\frac{1}{2\\gamma}\\|\\tilde{\\theta}^{k+1}-\\tilde{y}^{k+1}\\|^2 - v_\\gamma(x^{k+1},\\tilde{y}^{k+1}) \\\\\n\t\t&\\le \\frac{\\gamma}{2} \\left(L_{f_y} + \\frac{1}{\\gamma} + \\frac{1}{\\eta}\\right)^2 \\mathcal{G}(\\hat{\\theta}^{k+1}, x^{k+1}, \\tilde{y}^{k+1})^2.\n\t\\end{aligned}\n\t$$\n\tLetting $C_{\\gamma} := \\frac{\\gamma}{2} (L_{f_y} + 1/\\gamma + 1/\\eta)^2$ and recalling $\\varphi(x,\\theta) = f(x,\\theta) + g(x,\\theta)$, this rearranges to:\n\t$$\n\tv_\\gamma(x^{k+1},\\tilde{y}^{k+1}) \\ge \\varphi(x^{k+1},\\tilde{\\theta}^{k+1}) + \\frac{1}{2\\gamma}\\|\\tilde{\\theta}^{k+1}-\\tilde{y}^{k+1}\\|^2 - C_{\\gamma}\\zeta_{k+1}^2.\n\t$$\n\n\tBy condition \\eqref{decrease_new_y}, which $\\tilde{y}^{k+1}$ satisfies, we have\n\t\\begin{equation*}\n\t\t\\begin{aligned}\n\t\t\t&\\quad \\tilde{\\psi}_{p_k}(x^{k+1}, \\tilde{y}^{k+1}) - \\tilde{\\psi}_{p_k}(x^{k+1}, y^{k+1}) \\\\\n\t\t\t& \\le - v_\\gamma(x^{k+1}, \\tilde{y}^{k+1}) + \\left( \\varphi(x^{k+1},  \\tilde{\\theta}^{k+1} ) + \\frac{1}{2\\gamma} \\| \\tilde{\\theta}^{k+1} - \\tilde{y}^{k+1} \\|^2 \\right)\\\\\n\t\t\t& \\quad +  v_\\gamma(x^{k+1}, {y}^{k+1}) - \\left( \\varphi(x^{k+1}, \\theta^{k+1}) + \\frac{1}{2\\gamma} \\| \\theta^{k+1} - y^{k+1} \\|^2 \\right) \\\\\n\t\t\t&\\le C_{\\gamma} \\zeta_{k+1}^2,\n\t\t\\end{aligned}\n\t\\end{equation*}\n\twhere the last inequality follows from the definition of $v_\\gamma(x^{k+1}, {y}^{k+1})$ as an infimum $ \\inf_{\\theta\\in Y} \\{ \\varphi(x^{k+1}, {\\theta}) + \\frac{1}{2\\gamma} \\| {\\theta} - y^{k+1} \\|^2 \\}$. Thus, the conclusion follows.\n\\end{proof}\n","start":54888,"end":60122},{"raw_tex":"\n\tBy convexity of the lower-level objective in \\(\\theta\\), the function $f(x,\\theta) + \\tilde{g}(x,\\theta) + \\frac{1}{2\\gamma}\\|\\theta-y\\|^2$ is $1/\\gamma$-strongly convex with respect to $\\theta$ on $Y$. Thus, it satisfies a global quadratic growth condition and, by \\cite[Proposition 2]{ye2021variational}, a global KL property with exponent $1/2$ on $Y$. Noting that $\\tilde{g}(x,\\theta) = g(x,\\theta) + \\delta_Y(\\theta)$, we have $\\tilde{g}(x,\\theta) = g(x,\\theta)$ for all $\\theta \\in Y$. This yields \n\t\\begin{equation}\\label{KL}\n\t\\begin{aligned}\n\t\tf(x,\\theta)+g(x,\\theta)+\\frac{1}{2\\gamma}\\|\\theta-y\\|^2-v_\\gamma(x,y)\\le\n\t\t\\frac{\\gamma}{2}\\,\n\t\t\\operatorname{dist}\\Bigl(\n\t\t0,\\,\n\t\t\\nabla_y f(x,\\theta)+\\frac{1}{\\gamma}(\\theta-y)+\\partial \\tilde g(x,\\theta)\n\t\t\\Bigr)^2.\n\t\\end{aligned}\n\\end{equation}\n","start":55464,"end":56267},{"raw_tex":"The optimality condition of the proximal step defining $\\tilde{\\theta}^{k+1}$,\n\\[\n\\tilde{\\theta}^{k+1}\n=\n\\operatorname{Prox}_{\\eta \\tilde g(x^{k+1},\\cdot)}\n\\left(\n\\hat{\\theta}^{k+1}\n-\\eta\\left(\n\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n+\\frac{1}{\\gamma}(\\hat{\\theta}^{k+1}-\\tilde y^{k+1})\n\\right)\n\\right),\n\\]\nyields\n\\[\n-\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n-\\frac{1}{\\gamma}(\\hat{\\theta}^{k+1}-\\tilde y^{k+1})\n-\\frac{1}{\\eta}(\\tilde{\\theta}^{k+1}-\\hat{\\theta}^{k+1})\n\\in\n\\partial \\tilde g(x^{k+1},\\tilde{\\theta}^{k+1}).\n\\]\nAdding $\\nabla_y f(x^{k+1},\\tilde{\\theta}^{k+1})+(\\tilde{\\theta}^{k+1}-\\tilde y^{k+1})/\\gamma$ to both sides gives\n\\[\n\\begin{aligned}\n\t&\\nabla_y f(x^{k+1},\\tilde{\\theta}^{k+1})\n\t+\\frac{1}{\\gamma}(\\tilde{\\theta}^{k+1}-\\tilde y^{k+1}) -\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n\t-\\frac{1}{\\gamma}(\\hat{\\theta}^{k+1}-\\tilde y^{k+1})\n\t-\\frac{1}{\\eta}(\\tilde{\\theta}^{k+1}-\\hat{\\theta}^{k+1}) \\\\\n\t\\in \\ &\n\t\\nabla_y f(x^{k+1},\\tilde{\\theta}^{k+1})\n\t+\\frac{1}{\\gamma}(\\tilde{\\theta}^{k+1}-\\tilde y^{k+1})\n\t+\\partial \\tilde g(x^{k+1},\\tilde{\\theta}^{k+1}).\n\\end{aligned}\n\\]\n\tBy the $L_{f_y}$-Lipschitz continuity of $\\nabla_y f(x,\\cdot)$ and the definition of the gradient mapping $\\mathcal{G}$, the norm of the left-hand element is bounded by\n\t\\[\n\t\\begin{aligned}\n\t\t&\\quad\\ \n\t\t\\Bigl\\|\n\t\t\\nabla_y f(x^{k+1},\\tilde{\\theta}^{k+1})\n\t\t-\\nabla_y f(x^{k+1},\\hat{\\theta}^{k+1})\n\t\t+\\frac{1}{\\gamma}(\\tilde{\\theta}^{k+1}-\\hat{\\theta}^{k+1})\n\t\t+\\frac{1}{\\eta}(\\hat{\\theta}^{k+1}-\\tilde{\\theta}^{k+1})\n\t\t\\Bigr\\| \\\\\n\t\t&\\le\n\t\t\\Bigl(L_{f_y}+\\frac{1}{\\gamma}+\\frac{1}{\\eta}\\Bigr)\n\t\t\\|\\tilde{\\theta}^{k+1}-\\hat{\\theta}^{k+1}\\| \\\\\n\t\t&=\n\t\t\\Bigl(L_{f_y}+\\frac{1}{\\gamma}+\\frac{1}{\\eta}\\Bigr)\n\t\t\\mathcal G(\\hat{\\theta}^{k+1},x^{k+1},\\tilde y^{k+1}),\n\t\\end{aligned}\n\t\\]\nTherefore, by the definition of the distance from the origin to a set,\n\t$$\n\t\\begin{aligned}\n\t\t&\\operatorname{dist}\\left(0, \\nabla_y f(x^{k+1},\\tilde{\\theta}^{k+1}) + \\frac{1}{\\gamma}(\\tilde{\\theta}^{k+1}-\\tilde{y}^{k+1}) + \\partial \\tilde{g}(x^{k+1},\\tilde{\\theta}^{k+1})\\right) \\\\\n\t\t&\\le \\left(L_{f_y} + \\frac{1}{\\gamma} + \\frac{1}{\\eta}\\right) \\mathcal{G}(\\hat{\\theta}^{k+1}, x^{k+1}, \\tilde{y}^{k+1}).\n\t\\end{aligned}\n\t$$\n","start":56267,"end":58451},{"raw_tex":"\tApplying the KL inequality \\eqref{KL} with $(x^{k+1}, \\tilde{y}^{k+1}, \\tilde{\\theta}^{k+1})$ and substituting the above estimate gives\n\t$$\n\t\\begin{aligned}\n\t\t&f(x^{k+1},\\tilde{\\theta}^{k+1}) + g(x^{k+1},\\tilde{\\theta}^{k+1}) + \\frac{1}{2\\gamma}\\|\\tilde{\\theta}^{k+1}-\\tilde{y}^{k+1}\\|^2 - v_\\gamma(x^{k+1},\\tilde{y}^{k+1}) \\\\\n\t\t&\\le \\frac{\\gamma}{2} \\left(L_{f_y} + \\frac{1}{\\gamma} + \\frac{1}{\\eta}\\right)^2 \\mathcal{G}(\\hat{\\theta}^{k+1}, x^{k+1}, \\tilde{y}^{k+1})^2.\n\t\\end{aligned}\n\t$$\n\tLetting $C_{\\gamma} := \\frac{\\gamma}{2} (L_{f_y} + 1/\\gamma + 1/\\eta)^2$ and recalling $\\varphi(x,\\theta) = f(x,\\theta) + g(x,\\theta)$, this rearranges to:\n\t$$\n\tv_\\gamma(x^{k+1},\\tilde{y}^{k+1}) \\ge \\varphi(x^{k+1},\\tilde{\\theta}^{k+1}) + \\frac{1}{2\\gamma}\\|\\tilde{\\theta}^{k+1}-\\tilde{y}^{k+1}\\|^2 - C_{\\gamma}\\zeta_{k+1}^2.\n\t$$\n","start":58451,"end":59273},{"raw_tex":"\n\tBy condition \\eqref{decrease_new_y}, which $\\tilde{y}^{k+1}$ satisfies, we have\n\t\\begin{equation*}\n\t\t\\begin{aligned}\n\t\t\t&\\quad \\tilde{\\psi}_{p_k}(x^{k+1}, \\tilde{y}^{k+1}) - \\tilde{\\psi}_{p_k}(x^{k+1}, y^{k+1}) \\\\\n\t\t\t& \\le - v_\\gamma(x^{k+1}, \\tilde{y}^{k+1}) + \\left( \\varphi(x^{k+1},  \\tilde{\\theta}^{k+1} ) + \\frac{1}{2\\gamma} \\| \\tilde{\\theta}^{k+1} - \\tilde{y}^{k+1} \\|^2 \\right)\\\\\n\t\t\t& \\quad +  v_\\gamma(x^{k+1}, {y}^{k+1}) - \\left( \\varphi(x^{k+1}, \\theta^{k+1}) + \\frac{1}{2\\gamma} \\| \\theta^{k+1} - y^{k+1} \\|^2 \\right) \\\\\n\t\t\t&\\le C_{\\gamma} \\zeta_{k+1}^2,\n\t\t\\end{aligned}\n\t\\end{equation*}\n\twhere the last inequality follows from the definition of $v_\\gamma(x^{k+1}, {y}^{k+1})$ as an infimum $ \\inf_{\\theta\\in Y} \\{ \\varphi(x^{k+1}, {\\theta}) + \\frac{1}{2\\gamma} \\| {\\theta} - y^{k+1} \\|^2 \\}$. Thus, the conclusion follows.\n\\end{proof}\n","start":59273,"end":60122}]},{"id":"S035","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"主分析继承的参数条件","source_units":[{"raw_tex":"Throughout this subsection and the remainder of this section, we assume that Assumptions \\ref{asup1}, \\ref{asup2}, and \\ref{asup3} hold, and $\\gamma \\in (0, 1/(\\rho_{f_2} +\\rho_{g_2}) )$. The sequences $\\{(x^k, y^k, \\tilde{y}^k)\\}$ and $\\{p_k\\}$ \nare generated by AGILS (\\cref{alg1}), where the step sizes satisfy $\\alpha_k \\in [\\underline{\\alpha}, 2/(L_{\\psi_{x,k}} + c_\\alpha)]$ and $\\beta_k \\in [\\underline{\\beta}, 2/(L_{\\psi_{y,k}} + c_\\beta)]$, where $L_{\\psi_{x,k}} = L_{F_x}/p_k+L_{f_x}+L_{g_1}+\\rho_{f_1} + \\rho_{g_1}$, $L_{\\psi_{y,k}} = L_{F_y}/p_k+L_{f_y}$ and $\\underline{\\alpha}, \\underline{\\beta}, c_\\alpha, c_\\beta$ are positive constants. The inexactness parameters satisfy $\\sum_{k=0}^{\\infty}s_k^2 < \\infty$ and \n$\\tau_k := \\tau_0/(k+1)^{p_\\tau}$ with $\\tau_0 > 0$ and $p_\\tau >0$.\n","start":68918,"end":69717}]},{"id":"S036","semantic_type":"lemma","kind":"lemma","labels":["lem9"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"负包络的x二次上界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem9}\n\tSuppose Assumptions \\ref{asup2} and \\ref{asup3} hold. Let $\\gamma \\in (0, 1/(\\rho_{f_2} +\\rho_{g_2}) )$ and  $(\\bar{x},\\bar{y})\\in X\\times \\Re^m$. Then, for any $(x,y) \\in X \\times \\Re^m$, the following inequalities hold,\n\t\\begin{align}\n\t\t-v_{\\gamma}(x,\\bar{y}) &\\le -v_{\\gamma}(\\bar{x},\\bar{y})-\\langle \\nabla_x v_{\\gamma}(\\bar{x},\\bar{y}),x-\\bar{x}\\rangle +\\frac{\\rho_{f_1} + \\rho_{g_1}}{2}\\|\\bar{x}-x\\|^2,  \\label{lem9_eq1} \\\\\n","start":52691,"end":53148}]},{"id":"S037","semantic_type":"lemma","kind":"lemma","labels":["lem9"],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"负包络的y仿射上界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem9}\n\tSuppose Assumptions \\ref{asup2} and \\ref{asup3} hold. Let $\\gamma \\in (0, 1/(\\rho_{f_2} +\\rho_{g_2}) )$ and  $(\\bar{x},\\bar{y})\\in X\\times \\Re^m$. Then, for any $(x,y) \\in X \\times \\Re^m$, the following inequalities hold,\n","start":52691,"end":52940},{"raw_tex":"\t\t-v_{\\gamma}(\\bar{x},y) &\\le -v_{\\gamma}(\\bar{x},\\bar{y})-\\langle \\nabla_y v_{\\gamma}(\\bar{x},\\bar{y}),y-\\bar{y}\\rangle. \\label{lem9_eq2}\n\t\\end{align}\n\\end{lemma}","start":53148,"end":53311}]},{"id":"S038","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"两项包络上界的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\tAccording to Proposition \\ref{partial_grad_v}, for any $\\bar{y} \\in \\Re^m$, $v_{\\gamma}(x, \\bar{y})$ is a $(\\rho_{f_1} + \\rho_{g_1})$-weakly convex function with respect to $x$ on $X$, that is, $v_{\\gamma}(x,\\bar{y})+\\frac{\\rho_{f_1} + \\rho_{g_1}}{2}\\|x\\|^2$ is convex on $X$. Then we can obtain \\eqref{lem9_eq1} immediately. The second inequality \\eqref{lem9_eq2} follows from Proposition \\ref{partial_grad_v} that for any $\\bar{x} \\in X$, $v_{\\gamma}(\\bar{x}, y)$ is convex with respect to $y$.\n\\end{proof}","start":53312,"end":53835}]},{"id":"S039","semantic_type":"lemma","kind":"lemma","labels":["lem1","lem1_eq"],"member_ids":["S039","S041","S042","S043","S044","S046","S047","S048","S050","S051"],"declared_external":false,"source_only":false,"title":"交替更新的充分下降递推","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem1}\n\tSuppose Assumptions \\ref{asup1}, \\ref{asup2} and \\ref{asup3} hold. Let $\\gamma \\in (0, 1/(\\rho_{f_2} +\\rho_{g_2}) )$, $\\alpha_k \\in [0, 2/(L_{\\psi_{x,k}} + c_\\alpha)]$ and $\\beta_k \\in [0, 2/(L_{\\psi_{y,k}} + c_\\beta)]$, where $L_{\\psi_{x,k}} = L_{F_x}/p_k+L_{f_x}+L_{g_1}+\\rho_{f_1} + \\rho_{g_1}$, $L_{\\psi_{y,k}} = L_{F_y}/p_k+L_{f_y}$ and $c_\\alpha, c_\\beta$ are positive constants. Then, there exists $C_\\eta , C_\\gamma > 0$ such that the sequence $\\{(x^k,y^k,\\tilde{y}^k)\\}$ generated by AGILS {\\rm (}\\cref{alg1}{\\rm )} satisfies the following inequality,\n\t\\begin{equation}\\label{lem1_eq}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_{k+1}}\\left(x^{k+1}, \\tilde{y}^{k+1}\\right)\\leq\\,& \\tilde{\\psi}_{p_k}\\left(x^k, \\tilde{y}^k\\right) +  \\left( \\frac{C_\\eta^2}{c_\\alpha}(L_{f_x} + L_{g_2})^2 + \\frac{C_\\eta^2}{c_\\beta\\gamma^2 } \\right) \\zeta_{k}^2 + C_{\\gamma} \\zeta_{k+1}^2\\\\\n\t\t\t&- \\frac{c_\\alpha}{4} \\left\\|x^{k+1}-x^k\\right\\|^2  -\\frac{c_\\beta}{4} \\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2,\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere  $\\zeta_{k} := \\max\\{ s_k,  ( \\prod_{i=0}^{k} \\tau_i ) \\mathcal{G}(\\theta^{0}, x^{0}, y^{0})\\}$ and $\\{ \\zeta_{k} \\}$ is square summable.\n\\end{lemma}","start":60296,"end":61495},{"raw_tex":"\\begin{proof}\n\t\n\tFirst, the update rule for \\( y^{k+1} \\) given in \\eqref{update_y} defines it as the minimizer of the function $ g(x^k, y) + \\langle d_y^k, y - \\tilde{y}^{k} \\rangle + \\frac{1}{2\\beta_k} \\left\\| y - \\tilde{y}^k \\right\\|^2$. This function is $1/\\beta_k$-strongly convex in $y$. By applying \\cite[Theorem 6.39]{beck2017first}, we obtain\n\t\\begin{equation}\\label{lem1_eq5}\n\t\tg(x^{k}, y^{k+1}) + \\langle d_y^k, y^{k+1}-\\tilde{y}^k\\rangle+\\frac{1}{\\beta_k}\\|y^{k+1}-\\tilde{y}^k\\|^2 \\le g(x^{k}, \\tilde{y}^{k}).\n\t\\end{equation}\n\t\n\t\n","start":61497,"end":62039},{"raw_tex":"\\begin{proof}\n\t\n\tFirst, the update rule for \\( y^{k+1} \\) given in \\eqref{update_y} defines it as the minimizer of the function $ g(x^k, y) + \\langle d_y^k, y - \\tilde{y}^{k} \\rangle + \\frac{1}{2\\beta_k} \\left\\| y - \\tilde{y}^k \\right\\|^2$. This function is $1/\\beta_k$-strongly convex in $y$. By applying \\cite[Theorem 6.39]{beck2017first}, we obtain\n\t\\begin{equation}\\label{lem1_eq5}\n\t\tg(x^{k}, y^{k+1}) + \\langle d_y^k, y^{k+1}-\\tilde{y}^k\\rangle+\\frac{1}{\\beta_k}\\|y^{k+1}-\\tilde{y}^k\\|^2 \\le g(x^{k}, \\tilde{y}^{k}).\n\t\\end{equation}\n\t\n\t\n\tAccording to the assumptions, with $x^{k} \\in X$, we have that $\\nabla_y F(x^{k},y)$ and $\\nabla_y f(x^{k},y)$ are $L_{F_y}$- and  $L_{f_y}$-Lipschitz continuous with respect to variable $y$ on $Y$, respectively, and combine this with \\eqref{lem9_eq2}, we have\n\t\\begin{equation}\\label{lem1_eq6}\n\t\t\\begin{aligned}\n\t\t\t&(\\tilde{\\psi}_{p_k}-g)(x^{k},y^{k+1})  \\\\\n\t\t\t\\le \\, & (\\tilde{\\psi}_{p_k}-g)\t(x^{k},\\tilde{y}^k) + \\langle \\nabla_y (\\tilde{\\psi}_{p_k}-g)(x^{k},\\tilde{y}^k),y^{k+1}-\\tilde{y}^k\\rangle +\\frac{L_{\\psi_{y,k}}}{2}\\|y^{k+1}-\\tilde{y}^k\\|^2,\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere $L_{\\psi_{y,k}}=L_{F_y}/p_k+L_{f_y}$. \n\tCombining \\eqref{lem1_eq5} and \\eqref{lem1_eq6} yields\n\t\\begin{equation}\\label{lem1_eq7}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k}, y^{k+1}\\right) \\leq \\,& \t\\tilde{\\psi}_{p_k}\\left(x^{k}, \\tilde{y}^k\\right) + \\langle \\nabla_y (\t\\tilde{\\psi}_{p_k}-g)(x^{k},\\tilde{y}^k) - d_y^k, y^{k+1}-\\tilde{y}^k\\rangle  \\\\\n\t\t\t&-\\left(\\frac{1}{\\beta_k}-\\frac{L_{\\psi_{y,k}}}{2}\\right)\\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2.\n\t\t\\end{aligned}\n\t\\end{equation}\t\n\tUsing the formula of \\( \\nabla_y v_\\gamma \\) in \\eqref{valuefinc}, and the construction of $d_y^k$ in \\eqref{dy}, we have\n\t\\begin{equation}\\label{lem1_eq11}\n\t\t\\begin{aligned}\n\t\t\t\\left\\| \\nabla_y (\t\\tilde{\\psi}_{p_k}-g)(x^{k},\\tilde{y}^k) - d_y^k \\right\\| & = \\| \\nabla_y v_\\gamma(x^k, \\tilde{y}^k) + (\\tilde{y}^k-\\theta^{k})/\\gamma\\| \\\\\n\t\t\t& = \\frac{1}{\\gamma}\\left\\| \\theta^*_{\\gamma}(x^{k},\\tilde{y}^k) - \\theta^{k} \\right\\| \\le \\frac{C_\\eta}{\\gamma} \\mathcal{G}({\\theta}^{k}, x^{k}, \\tilde{y}^{k}) \\le \\frac{C_\\eta}{\\gamma} \\zeta_{k},\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere the first inequality follows from Lemma \\ref{error_bound} and the second inequality is a consequence of the inexactness condition \\eqref{inexacty2}. \n\t\n\tCombining the above inequality with \\eqref{lem1_eq7} gives us\n\t\\begin{equation}\\label{lem1_eq8}\n\t\t\\tilde{\\psi}_{p_k}\\left(x^{k}, y^{k+1}\\right) \\leq \\,  \t\\tilde{\\psi}_{p_k}\\left(x^{k}, \\tilde{y}^k\\right)  - \\left(\\frac{1}{\\beta_k}-\\frac{L_{\\psi_{y,k}}}{2}- \\frac{c_\\beta}{4}\\right)\\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2 + \\frac{C_\\eta^2}{c_\\beta\\gamma^2} \\zeta_{k}^2.\n\t\\end{equation}\n\tNext, considering the update rule for $x^{k+1}$ given in \\eqref{update_x}, we have\n\t\\begin{equation}\\label{lem1_eq1}\n\t\t\\langle d_x^k, x^{k+1}-x^k\\rangle+\\frac{1}{\\alpha_k}\\|x^{k+1}-x^k\\|^2 \\le 0.\n\t\\end{equation}\t\n\tBy the assumptions, with $y^{k+1} \\in Y$, we have that $\\nabla_x F(x, y^{k+1})$, $\\nabla_x f(x, y^{k+1})$ and $\\nabla_{x} g(x, y^{k+1})$ are $L_{F_x}$-, $L_{f_x}$-, and $L_{g_1}$-Lipschitz continuous with respect to variable $x$ on $X$, respectively, and combine this with \\eqref{lem9_eq1}, we have\n\t\\begin{equation}\\label{lem1_eq2}\n\t\t\\tilde{\\psi}_{p_k}(x^{k+1}, y^{k+1})\\le  \\tilde{\\psi}_{p_k}\t(x^{k}, y^{k+1})+\\langle \\nabla_x \t\\tilde{\\psi}_{p_k}(x^{k}, y^{k+1}),x^{k+1}-x^k\\rangle+\\frac{L_{\\psi_{x,k}}}{2}\\|x^{k+1}-x^k\\|^2,\n\t\\end{equation}\n\twhere $L_{\\psi_{x,k}}=L_{F_x}/p_k+L_{f_x}+L_{g_1}+ \\rho_{f_1} + \\rho_{g_1} $. Combining \\eqref{lem1_eq1} and \\eqref{lem1_eq2} yields\n\t\\begin{equation}\\label{lem1_eq3}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k+1}, y^{k+1}\\right) \\leq \\,& \\tilde{\\psi}_{p_k}\\left(x^k, y^{k+1}\\right) + \\langle \\nabla_x \\tilde{\\psi}_{p_k}(x^{k},y^{k+1}) - d_x^k,x^{k+1}-x^k\\rangle  \\\\\n\t\t\t&-\\left(\\frac{1}{\\alpha_k}-\\frac{L_{\\psi_{x,k}}}{2}\\right)\\left\\|x^{k+1}-x^k\\right\\|^2.\n\t\t\\end{aligned}\n\t\\end{equation}\n\tUsing the formula of the gradient of $v_\\gamma$ given in \\eqref{valuefinc} and the construction of $d_x^k$ in \\eqref{dx}, and the $L_{f_x}$- and $L_{g_2}$-Lipschitz continuity of $\\nabla_x f(x^k,y)$ and $\\nabla_{x} g(x^k,y)$ with respect to variable $y$ on $Y$, we have\n\t\\begin{equation}\\label{lem1_eq10}\n\t\t\\begin{aligned}\n\t\t\t\\left\\| \\nabla_x  \\tilde{\\psi}_{p_k}(x^{k},y^{k+1}) - d_x^k \\right\\| \n\t\t\t&= \\left\\| \\nabla_{x} \\varphi(x^k, \\theta^*_{\\gamma}(x^k,y^{k+1})) - \\nabla_{x} \\varphi(x^k, \\theta^{k+1/2}) \\right\\| \\\\\n\t\t\t& \\le (L_{f_x} + L_{g_2})\\left\\| \\theta^*_{\\gamma}(x^k,y^{k+1}) - \\theta^{k+1/2} \\right\\| \\le (L_{f_x} + L_{g_2})C_\\eta \\zeta_{k},\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere the last inequality follows from \\cref{error_bound} and the inexact criterion \\eqref{inexacty1}. Combining the above inequality with \\eqref{lem1_eq3} gives us\n\t\\begin{equation}\\label{lem1_eq4}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k+1}, y^{k+1}\\right) \\leq \\,& \\tilde{\\psi}_{p_k}\\left(x^k, y^{k+1}\\right) - \\left(\\frac{1}{\\alpha_k}-\\frac{L_{\\psi_{x,k}}}{2}-\\frac{c_\\alpha}{4}\\right)\\left\\|x^{k+1}-x^k\\right\\|^2 \\\\ & + \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2C_\\eta^2 \\zeta_{k}^2.\n\t\t\\end{aligned}\n\t\\end{equation}\n\tCombining  \\eqref{lem1_eq8}, \\eqref{lem1_eq4}, and then  \\eqref{lem1_eq12} established in  Lemma \\ref{lem0}, we obtain\n\t\\begin{equation}\\label{lem1_eq9}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k+1}, \\tilde{y}^{k+1}\\right)\\leq\\,& \t\\tilde{\\psi}_{p_k}\\left(x^k, \\tilde{y}^k\\right)+  \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2C_\\eta^2 \\zeta_{k}^2 + \\frac{C_\\eta^2}{c_\\beta\\gamma^2} \\zeta_{k}^2 + C_{\\gamma} \\zeta_{k+1}^2\\\\\n\t\t\t&\\hspace{-50pt}-\\left(\\frac{1}{\\alpha_k}-\\frac{L_{\\psi_{x,k}}}{2}- \\frac{c_\\alpha}{4} \\right)\\left\\|x^{k+1}-x^k\\right\\|^2 -\\left(\\frac{1}{\\beta_k}-\\frac{L_{\\psi_{y,k}}}{2}- \\frac{c_\\beta}{4} \\right)\\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2.\n\t\t\\end{aligned}\n\t\\end{equation}\n\tBecause  $\\alpha_k \\in [0, 2/(L_{\\psi_{x,k}} + c_\\alpha)]$ and $\\beta_k \\in [0, 2/(L_{\\psi_{y,k}} + c_\\beta)]$, and $\\tilde{\\psi}_{p_{k+1}}(x^{k+1}, \\tilde{y}^{k+1}) \\leq \t\\tilde{\\psi}_{p_k}(x^{k+1}, \\tilde{y}^{k+1})$ following from the non-decreasing property of the penalty parameter $p_k$ and the non-negative property of $F-\\underline{F}$, we can obtain \\eqref{lem1_eq} from \\eqref{lem1_eq9}.\n\t\n\tFinally, we show that \\( \\{\\zeta_k\\} \\) is square summable. Let $ \\mathcal{G}_0 = \\mathcal{G}(\\theta^{0}, x^{0}, y^{0})$ and $r_k :=  ( \\prod_{i=0}^{k} \\tau_i )\\mathcal{G}_0 $. Since $\\zeta_{k} := \\max\\{ s_k, r_k\\}$, we have $\\sum_{k=0}^\\infty \\zeta_k^2 \\le \\sum_{k=0}^\\infty s_k^2 + \\sum_{k=0}^\\infty r_k^2$. By assumption, \\( \\sum_{k=0}^\\infty s_k^2 < \\infty \\). Thus, we only need to show that  \\( \\{r_k\\} \\) is square summable. Recall that \\( \\tau_k = \\tau_0/(k+1)^{p_\\tau} \\) with \\( p_\\tau > 0 \\), then\n\t\\[\n\tr_k = \\left( \\prod_{i=0}^{k} \\tau_i \\right) \\mathcal{G}_0  = \\tau_0^{k+1} \\left( \\prod_{i=1}^{k+1} \\frac{1}{i^{p_\\tau}} \\right) \\cdot \\mathcal{G}_0  = \\frac{\\tau_0^{k+1}}{((k+1)!)^{p_\\tau}}\\mathcal{G}_0 .\n\t\\]\n\tFor the ratio test on  $\\sum_{k=0}^\\infty r_k^2$, we consider \n\t\\[\n\t\\lim_{k \\rightarrow \\infty}\\frac{r_{k+1}^2}{r_k^2} =  \\lim_{k \\rightarrow \\infty} \\frac{\\tau_0^2}{(k+2)^{2p_\\tau}}  = 0.\n\t\\]\n\tTherefore, by the ratio test, $\\sum_{k=0}^\\infty r_k^2 < \\infty$, implying that \\( \\{\\zeta_k\\} \\) is square summable.\n\\end{proof}","start":61497,"end":68849},{"raw_tex":"\tAccording to the assumptions, with $x^{k} \\in X$, we have that $\\nabla_y F(x^{k},y)$ and $\\nabla_y f(x^{k},y)$ are $L_{F_y}$- and  $L_{f_y}$-Lipschitz continuous with respect to variable $y$ on $Y$, respectively, and combine this with \\eqref{lem9_eq2}, we have\n\t\\begin{equation}\\label{lem1_eq6}\n\t\t\\begin{aligned}\n\t\t\t&(\\tilde{\\psi}_{p_k}-g)(x^{k},y^{k+1})  \\\\\n\t\t\t\\le \\, & (\\tilde{\\psi}_{p_k}-g)\t(x^{k},\\tilde{y}^k) + \\langle \\nabla_y (\\tilde{\\psi}_{p_k}-g)(x^{k},\\tilde{y}^k),y^{k+1}-\\tilde{y}^k\\rangle +\\frac{L_{\\psi_{y,k}}}{2}\\|y^{k+1}-\\tilde{y}^k\\|^2,\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere $L_{\\psi_{y,k}}=L_{F_y}/p_k+L_{f_y}$. \n","start":62039,"end":62672},{"raw_tex":"\tCombining \\eqref{lem1_eq5} and \\eqref{lem1_eq6} yields\n\t\\begin{equation}\\label{lem1_eq7}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k}, y^{k+1}\\right) \\leq \\,& \t\\tilde{\\psi}_{p_k}\\left(x^{k}, \\tilde{y}^k\\right) + \\langle \\nabla_y (\t\\tilde{\\psi}_{p_k}-g)(x^{k},\\tilde{y}^k) - d_y^k, y^{k+1}-\\tilde{y}^k\\rangle  \\\\\n\t\t\t&-\\left(\\frac{1}{\\beta_k}-\\frac{L_{\\psi_{y,k}}}{2}\\right)\\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2.\n\t\t\\end{aligned}\n\t\\end{equation}\t\n","start":62672,"end":63122},{"raw_tex":"\t\n\tCombining the above inequality with \\eqref{lem1_eq7} gives us\n\t\\begin{equation}\\label{lem1_eq8}\n\t\t\\tilde{\\psi}_{p_k}\\left(x^{k}, y^{k+1}\\right) \\leq \\,  \t\\tilde{\\psi}_{p_k}\\left(x^{k}, \\tilde{y}^k\\right)  - \\left(\\frac{1}{\\beta_k}-\\frac{L_{\\psi_{y,k}}}{2}- \\frac{c_\\beta}{4}\\right)\\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2 + \\frac{C_\\eta^2}{c_\\beta\\gamma^2} \\zeta_{k}^2.\n\t\\end{equation}\n","start":63850,"end":64235},{"raw_tex":"\tNext, considering the update rule for $x^{k+1}$ given in \\eqref{update_x}, we have\n\t\\begin{equation}\\label{lem1_eq1}\n\t\t\\langle d_x^k, x^{k+1}-x^k\\rangle+\\frac{1}{\\alpha_k}\\|x^{k+1}-x^k\\|^2 \\le 0.\n\t\\end{equation}\t\n\tBy the assumptions, with $y^{k+1} \\in Y$, we have that $\\nabla_x F(x, y^{k+1})$, $\\nabla_x f(x, y^{k+1})$ and $\\nabla_{x} g(x, y^{k+1})$ are $L_{F_x}$-, $L_{f_x}$-, and $L_{g_1}$-Lipschitz continuous with respect to variable $x$ on $X$, respectively, and combine this with \\eqref{lem9_eq1}, we have\n\t\\begin{equation}\\label{lem1_eq2}\n\t\t\\tilde{\\psi}_{p_k}(x^{k+1}, y^{k+1})\\le  \\tilde{\\psi}_{p_k}\t(x^{k}, y^{k+1})+\\langle \\nabla_x \t\\tilde{\\psi}_{p_k}(x^{k}, y^{k+1}),x^{k+1}-x^k\\rangle+\\frac{L_{\\psi_{x,k}}}{2}\\|x^{k+1}-x^k\\|^2,\n\t\\end{equation}\n\twhere $L_{\\psi_{x,k}}=L_{F_x}/p_k+L_{f_x}+L_{g_1}+ \\rho_{f_1} + \\rho_{g_1} $. Combining \\eqref{lem1_eq1} and \\eqref{lem1_eq2} yields\n","start":64235,"end":65127},{"raw_tex":"\t\\begin{equation}\\label{lem1_eq3}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k+1}, y^{k+1}\\right) \\leq \\,& \\tilde{\\psi}_{p_k}\\left(x^k, y^{k+1}\\right) + \\langle \\nabla_x \\tilde{\\psi}_{p_k}(x^{k},y^{k+1}) - d_x^k,x^{k+1}-x^k\\rangle  \\\\\n\t\t\t&-\\left(\\frac{1}{\\alpha_k}-\\frac{L_{\\psi_{x,k}}}{2}\\right)\\left\\|x^{k+1}-x^k\\right\\|^2.\n\t\t\\end{aligned}\n\t\\end{equation}\n","start":65127,"end":65490},{"raw_tex":"\t\\begin{equation}\\label{lem1_eq4}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k+1}, y^{k+1}\\right) \\leq \\,& \\tilde{\\psi}_{p_k}\\left(x^k, y^{k+1}\\right) - \\left(\\frac{1}{\\alpha_k}-\\frac{L_{\\psi_{x,k}}}{2}-\\frac{c_\\alpha}{4}\\right)\\left\\|x^{k+1}-x^k\\right\\|^2 \\\\ & + \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2C_\\eta^2 \\zeta_{k}^2.\n\t\t\\end{aligned}\n\t\\end{equation}\n","start":66343,"end":66705},{"raw_tex":"\tCombining  \\eqref{lem1_eq8}, \\eqref{lem1_eq4}, and then  \\eqref{lem1_eq12} established in  Lemma \\ref{lem0}, we obtain\n\t\\begin{equation}\\label{lem1_eq9}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_k}\\left(x^{k+1}, \\tilde{y}^{k+1}\\right)\\leq\\,& \t\\tilde{\\psi}_{p_k}\\left(x^k, \\tilde{y}^k\\right)+  \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2C_\\eta^2 \\zeta_{k}^2 + \\frac{C_\\eta^2}{c_\\beta\\gamma^2} \\zeta_{k}^2 + C_{\\gamma} \\zeta_{k+1}^2\\\\\n\t\t\t&\\hspace{-50pt}-\\left(\\frac{1}{\\alpha_k}-\\frac{L_{\\psi_{x,k}}}{2}- \\frac{c_\\alpha}{4} \\right)\\left\\|x^{k+1}-x^k\\right\\|^2 -\\left(\\frac{1}{\\beta_k}-\\frac{L_{\\psi_{y,k}}}{2}- \\frac{c_\\beta}{4} \\right)\\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2.\n\t\t\\end{aligned}\n\t\\end{equation}\n\tBecause  $\\alpha_k \\in [0, 2/(L_{\\psi_{x,k}} + c_\\alpha)]$ and $\\beta_k \\in [0, 2/(L_{\\psi_{y,k}} + c_\\beta)]$, and $\\tilde{\\psi}_{p_{k+1}}(x^{k+1}, \\tilde{y}^{k+1}) \\leq \t\\tilde{\\psi}_{p_k}(x^{k+1}, \\tilde{y}^{k+1})$ following from the non-decreasing property of the penalty parameter $p_k$ and the non-negative property of $F-\\underline{F}$, we can obtain \\eqref{lem1_eq} from \\eqref{lem1_eq9}.\n","start":66705,"end":67799}]},{"id":"S040","semantic_type":"derived_fact","kind":"derived_fact","labels":["lem1"],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"作者定义的误差包络可和","source_units":[{"raw_tex":"\twhere  $\\zeta_{k} := \\max\\{ s_k,  ( \\prod_{i=0}^{k} \\tau_i ) \\mathcal{G}(\\theta^{0}, x^{0}, y^{0})\\}$ and $\\{ \\zeta_{k} \\}$ is square summable.\n","start":61339,"end":61484}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["lem1_eq11"],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"y方向近似梯度误差","source_units":[{"raw_tex":"\tUsing the formula of \\( \\nabla_y v_\\gamma \\) in \\eqref{valuefinc}, and the construction of $d_y^k$ in \\eqref{dy}, we have\n\t\\begin{equation}\\label{lem1_eq11}\n\t\t\\begin{aligned}\n\t\t\t\\left\\| \\nabla_y (\t\\tilde{\\psi}_{p_k}-g)(x^{k},\\tilde{y}^k) - d_y^k \\right\\| & = \\| \\nabla_y v_\\gamma(x^k, \\tilde{y}^k) + (\\tilde{y}^k-\\theta^{k})/\\gamma\\| \\\\\n\t\t\t& = \\frac{1}{\\gamma}\\left\\| \\theta^*_{\\gamma}(x^{k},\\tilde{y}^k) - \\theta^{k} \\right\\| \\le \\frac{C_\\eta}{\\gamma} \\mathcal{G}({\\theta}^{k}, x^{k}, \\tilde{y}^{k}) \\le \\frac{C_\\eta}{\\gamma} \\zeta_{k},\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere the first inequality follows from Lemma \\ref{error_bound} and the second inequality is a consequence of the inexactness condition \\eqref{inexacty2}. \n","start":63122,"end":63850}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["lem1_eq10"],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"x方向近似梯度误差","source_units":[{"raw_tex":"\tUsing the formula of the gradient of $v_\\gamma$ given in \\eqref{valuefinc} and the construction of $d_x^k$ in \\eqref{dx}, and the $L_{f_x}$- and $L_{g_2}$-Lipschitz continuity of $\\nabla_x f(x^k,y)$ and $\\nabla_{x} g(x^k,y)$ with respect to variable $y$ on $Y$, we have\n\t\\begin{equation}\\label{lem1_eq10}\n\t\t\\begin{aligned}\n\t\t\t\\left\\| \\nabla_x  \\tilde{\\psi}_{p_k}(x^{k},y^{k+1}) - d_x^k \\right\\| \n\t\t\t&= \\left\\| \\nabla_{x} \\varphi(x^k, \\theta^*_{\\gamma}(x^k,y^{k+1})) - \\nabla_{x} \\varphi(x^k, \\theta^{k+1/2}) \\right\\| \\\\\n\t\t\t& \\le (L_{f_x} + L_{g_2})\\left\\| \\theta^*_{\\gamma}(x^k,y^{k+1}) - \\theta^{k+1/2} \\right\\| \\le (L_{f_x} + L_{g_2})C_\\eta \\zeta_{k},\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere the last inequality follows from \\cref{error_bound} and the inexact criterion \\eqref{inexacty1}. Combining the above inequality with \\eqref{lem1_eq3} gives us\n","start":65490,"end":66343}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"阶乘衰减与平方可和","source_units":[{"raw_tex":"\t\n\tFinally, we show that \\( \\{\\zeta_k\\} \\) is square summable. Let $ \\mathcal{G}_0 = \\mathcal{G}(\\theta^{0}, x^{0}, y^{0})$ and $r_k :=  ( \\prod_{i=0}^{k} \\tau_i )\\mathcal{G}_0 $. Since $\\zeta_{k} := \\max\\{ s_k, r_k\\}$, we have $\\sum_{k=0}^\\infty \\zeta_k^2 \\le \\sum_{k=0}^\\infty s_k^2 + \\sum_{k=0}^\\infty r_k^2$. By assumption, \\( \\sum_{k=0}^\\infty s_k^2 < \\infty \\). Thus, we only need to show that  \\( \\{r_k\\} \\) is square summable. Recall that \\( \\tau_k = \\tau_0/(k+1)^{p_\\tau} \\) with \\( p_\\tau > 0 \\), then\n\t\\[\n\tr_k = \\left( \\prod_{i=0}^{k} \\tau_i \\right) \\mathcal{G}_0  = \\tau_0^{k+1} \\left( \\prod_{i=1}^{k+1} \\frac{1}{i^{p_\\tau}} \\right) \\cdot \\mathcal{G}_0  = \\frac{\\tau_0^{k+1}}{((k+1)!)^{p_\\tau}}\\mathcal{G}_0 .\n\t\\]\n\tFor the ratio test on  $\\sum_{k=0}^\\infty r_k^2$, we consider \n\t\\[\n\t\\lim_{k \\rightarrow \\infty}\\frac{r_{k+1}^2}{r_k^2} =  \\lim_{k \\rightarrow \\infty} \\frac{\\tau_0^2}{(k+2)^{2p_\\tau}}  = 0.\n\t\\]\n\tTherefore, by the ratio test, $\\sum_{k=0}^\\infty r_k^2 < \\infty$, implying that \\( \\{\\zeta_k\\} \\) is square summable.\n\\end{proof}","start":67799,"end":68849}]},{"id":"S053","semantic_type":"lemma","kind":"lemma","labels":["lem3"],"member_ids":["S053","S054","S055","S056"],"declared_external":false,"source_only":false,"title":"交替步差平方可和","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem3}\n\tThe sequence $\\left\\{(x^k,y^k,\\tilde{y}^k)\\right\\}$ satisfies\n\t\\begin{equation*}\n\t\t\\sum\\limits_{k=0}^{{\\infty}}\\left\\|(x^{k+1},y^{k+1})-(x^k,\\tilde{y}^k)\\right\\|^2<\\infty, \\quad \\text{and} \\quad \\lim\\limits_{k\\to\\infty}\\left\\|(x^{k+1},y^{k+1})-(x^k,\\tilde{y}^k)\\right\\|=0.\n\t\\end{equation*}\n\\end{lemma}","start":69955,"end":70283},{"raw_tex":"\\begin{proof}\n\tSumming  \\cref{lem1_eq} from Lemma \\ref{lem1} over $k=0,1,\\dots,K-1$ yields,\n\t\\begin{equation}\\label{lem3_eq1}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_{K}}\\left(x^{K}, \\tilde{y}^{K}\\right)- \\tilde{\\psi}_{p_{0}}\\left(x^{0}, \\tilde{y}^{0}\\right) \\leq & -\\sum\\limits_{k=0}^{K-1} \\left(  \\frac{c_\\alpha}{4} \\left\\|x^{k+1}-x^k\\right\\|^2+ \\frac{c_\\beta}{4} \\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2\\right)\\\\& +\\sum\\limits_{k=0}^{K-1}\\left( \\frac{C_\\eta^2 }{c_\\alpha}(L_{f_x} + L_{g_2})^2+ \\frac{C_\\eta^2}{c_\\beta\\gamma^2} \\right) \\zeta_{k}^2+ \\sum\\limits_{k=0}^{K-1}C_{\\gamma} \\zeta_{k+1}^2.\\\\\n\t\t\\end{aligned}\n\t\\end{equation}\n","start":70284,"end":70911},{"raw_tex":"\\begin{proof}\n\tSumming  \\cref{lem1_eq} from Lemma \\ref{lem1} over $k=0,1,\\dots,K-1$ yields,\n\t\\begin{equation}\\label{lem3_eq1}\n\t\t\\begin{aligned}\n\t\t\t\\tilde{\\psi}_{p_{K}}\\left(x^{K}, \\tilde{y}^{K}\\right)- \\tilde{\\psi}_{p_{0}}\\left(x^{0}, \\tilde{y}^{0}\\right) \\leq & -\\sum\\limits_{k=0}^{K-1} \\left(  \\frac{c_\\alpha}{4} \\left\\|x^{k+1}-x^k\\right\\|^2+ \\frac{c_\\beta}{4} \\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2\\right)\\\\& +\\sum\\limits_{k=0}^{K-1}\\left( \\frac{C_\\eta^2 }{c_\\alpha}(L_{f_x} + L_{g_2})^2+ \\frac{C_\\eta^2}{c_\\beta\\gamma^2} \\right) \\zeta_{k}^2+ \\sum\\limits_{k=0}^{K-1}C_{\\gamma} \\zeta_{k+1}^2.\\\\\n\t\t\\end{aligned}\n\t\\end{equation}\n\tSince \\( \\{\\zeta_k\\} \\) is square summable from Lemma \\ref{lem1}, it follows that\n\t\\[\n\t\\sum\\limits_{k=0}^{\\infty} \\left( \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2 + \\frac{1}{c_\\beta\\gamma^2} \\right) C_\\eta^2\\zeta_{k}^2 + \\sum\\limits_{k=0}^{\\infty}  C_{\\gamma} \\zeta_{k+1}^2 < \\infty.\n\t\\]\n\tAdditionally, because $\\tilde{\\psi}_{p_{K}}\\left(x^{K}, \\tilde{y}^{K}\\right) \\ge 0$, by taking $K \\rightarrow \\infty$ in \\eqref{lem3_eq1}, we have\n\t\\begin{equation}\\label{lem3_eq2}\n\t\t\\begin{aligned}\n\t\t\t&\\quad\\,\\sum\\limits_{k=0}^{\\infty} \\left(  \\frac{c_\\alpha}{4} \\left\\|x^{k+1}-x^k\\right\\|^2+ \\frac{c_\\beta}{4} \\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2\\right) \\\\\n\t\t\t&\\le \\tilde{\\psi}_{p_{0}}\\left(x^{0}, \\tilde{y}^{0}\\right) + \\sum\\limits_{k=0}^{\\infty} \\left( \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2 + \\frac{1}{c_\\beta\\gamma^2} \\right) C_\\eta^2\\zeta_{k}^2 + \\sum\\limits_{k=0}^{\\infty}  C_{\\gamma} \\zeta_{k+1}^2  < \\infty,\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhich proves the result.\n\\end{proof}","start":70284,"end":71887},{"raw_tex":"\tSince \\( \\{\\zeta_k\\} \\) is square summable from Lemma \\ref{lem1}, it follows that\n\t\\[\n\t\\sum\\limits_{k=0}^{\\infty} \\left( \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2 + \\frac{1}{c_\\beta\\gamma^2} \\right) C_\\eta^2\\zeta_{k}^2 + \\sum\\limits_{k=0}^{\\infty}  C_{\\gamma} \\zeta_{k+1}^2 < \\infty.\n\t\\]\n\tAdditionally, because $\\tilde{\\psi}_{p_{K}}\\left(x^{K}, \\tilde{y}^{K}\\right) \\ge 0$, by taking $K \\rightarrow \\infty$ in \\eqref{lem3_eq1}, we have\n\t\\begin{equation}\\label{lem3_eq2}\n\t\t\\begin{aligned}\n\t\t\t&\\quad\\,\\sum\\limits_{k=0}^{\\infty} \\left(  \\frac{c_\\alpha}{4} \\left\\|x^{k+1}-x^k\\right\\|^2+ \\frac{c_\\beta}{4} \\left\\|y^{k+1}-\\tilde{y}^k\\right\\|^2\\right) \\\\\n\t\t\t&\\le \\tilde{\\psi}_{p_{0}}\\left(x^{0}, \\tilde{y}^{0}\\right) + \\sum\\limits_{k=0}^{\\infty} \\left( \\frac{1}{c_\\alpha}(L_{f_x} + L_{g_2})^2 + \\frac{1}{c_\\beta\\gamma^2} \\right) C_\\eta^2\\zeta_{k}^2 + \\sum\\limits_{k=0}^{\\infty}  C_{\\gamma} \\zeta_{k+1}^2  < \\infty,\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhich proves the result.\n\\end{proof}","start":70911,"end":71887}]},{"id":"S057","semantic_type":"lemma","kind":"lemma","labels":["lem4"],"member_ids":["S057","S058","S059","S060","S061"],"declared_external":false,"source_only":false,"title":"校正聚点为下层最优解","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem4}\n\tLet $\\{ (x^{k_j},\\tilde{y}^{k_j}) \\} $ be any convergent subsequence of iterates $\\{(x^k,\\tilde{y}^k)\\}$ such that $(x^{k_j},\\tilde{y}^{k_j})$ satisfies condition \\eqref{inexacty_y} with $k = k_j - 1$. \n\tIf $(x^{k_j},\\tilde{y}^{k_j}) \\rightarrow (\\bar{x},\\bar{y}) $ as $j \\rightarrow \\infty$ for some $(\\bar{x},\\bar{y}) $, then $\\bar{y} \\in S(\\bar{x})$. \n\\end{lemma}","start":72078,"end":72471},{"raw_tex":"\\begin{proof}\n\tDefine the residual term $\\tilde{r}^k := \\tilde{y}^{k} - \\operatorname{Prox}_{ \\tilde{g}(x^{k}, \\cdot)} \\left( \\tilde{y}^{k} -  \\nabla_y f(x^{k}, \\tilde{y}^{k}) \\right) $. By the first-order optimality condition for the proximal operator, $\\tilde{r}^k$ satisfies\n\t\\begin{equation}\\label{lem4_eq1}\n\t\t\\tilde{r}^k \\in \\nabla_y f(x^{k}, \\tilde{y}^{k}) + \\partial_{y}\\tilde{g}(x^{k}, \\tilde{y}^{k} - \\tilde{r}^k).\n\t\\end{equation}\n","start":72472,"end":72912},{"raw_tex":"\\begin{proof}\n\tDefine the residual term $\\tilde{r}^k := \\tilde{y}^{k} - \\operatorname{Prox}_{ \\tilde{g}(x^{k}, \\cdot)} \\left( \\tilde{y}^{k} -  \\nabla_y f(x^{k}, \\tilde{y}^{k}) \\right) $. By the first-order optimality condition for the proximal operator, $\\tilde{r}^k$ satisfies\n\t\\begin{equation}\\label{lem4_eq1}\n\t\t\\tilde{r}^k \\in \\nabla_y f(x^{k}, \\tilde{y}^{k}) + \\partial_{y}\\tilde{g}(x^{k}, \\tilde{y}^{k} - \\tilde{r}^k).\n\t\\end{equation}\n\tThe sequence  $(x^{k_j},\\tilde{y}^{k_j})$ satisfies condition \\eqref{inexacty_y} with $k = k_j - 1$, which implies \n\t\\[\n\t\\left\\| \\tilde{r}^{k_j} \\right\\| \\le \\frac{c_{\\tilde{y}} }{p_{k_j}} \\left\\|(x^{k_j},y^{k_j})-(x^{k_j-1},\\tilde{y}^{k_j-1})\\right\\|.\n\t\\]\n\tFrom Lemma \\ref{lem3}, we have $\\lim_{k_j \\to\\infty} \\|(x^{k_j},y^{k_j})-(x^{k_j-1},\\tilde{y}^{k_j-1}) \\|=0$. As $p_{k_j} \\ge p_0$, the above inequality ensures that $\\lim_{k_j \\to\\infty} \\|\\tilde{r}^{k_j} \\|=0$.\n\t\n\tTaking the limit as  $k_j \\rightarrow \\infty$ in \\eqref{lem4_eq1} with $k = k_j$, by the continuity of $ \\nabla_y f$ and the outer semicontinuity of $\\partial_{y}\\tilde{g}$, it follows that\n\t\\[\n\t0  \\in \\nabla_y f(\\bar{x},\\bar{y}) + \\partial_{y}\\tilde{g}(\\bar{x},\\bar{y}).\n\t\\] \n\tSince the lower-level problem is convex in $y$, this condition\n\timplies $\\bar{y} \\in S(\\bar{x})$.\n\\end{proof}","start":72472,"end":73774},{"raw_tex":"\tThe sequence  $(x^{k_j},\\tilde{y}^{k_j})$ satisfies condition \\eqref{inexacty_y} with $k = k_j - 1$, which implies \n\t\\[\n\t\\left\\| \\tilde{r}^{k_j} \\right\\| \\le \\frac{c_{\\tilde{y}} }{p_{k_j}} \\left\\|(x^{k_j},y^{k_j})-(x^{k_j-1},\\tilde{y}^{k_j-1})\\right\\|.\n\t\\]\n\tFrom Lemma \\ref{lem3}, we have $\\lim_{k_j \\to\\infty} \\|(x^{k_j},y^{k_j})-(x^{k_j-1},\\tilde{y}^{k_j-1}) \\|=0$. As $p_{k_j} \\ge p_0$, the above inequality ensures that $\\lim_{k_j \\to\\infty} \\|\\tilde{r}^{k_j} \\|=0$.\n\t\n","start":72912,"end":73386},{"raw_tex":"\tTaking the limit as  $k_j \\rightarrow \\infty$ in \\eqref{lem4_eq1} with $k = k_j$, by the continuity of $ \\nabla_y f$ and the outer semicontinuity of $\\partial_{y}\\tilde{g}$, it follows that\n\t\\[\n\t0  \\in \\nabla_y f(\\bar{x},\\bar{y}) + \\partial_{y}\\tilde{g}(\\bar{x},\\bar{y}).\n\t\\] \n\tSince the lower-level problem is convex in $y$, this condition\n\timplies $\\bar{y} \\in S(\\bar{x})$.\n\\end{proof}","start":73386,"end":73774}]},{"id":"S062","semantic_type":"proposition","kind":"proposition","labels":["prop0"],"member_ids":["S062","S063","S064","S065","S066","S067","S068"],"declared_external":false,"source_only":false,"title":"有界罚参数下的聚点可行性","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop0}\n\tLet $\\epsilon > 0$. If the sequence $\\{p_k\\}$ is bounded, then any accumulation point $(\\bar{x}, \\bar{y})$ of the iterate sequence $\\{(x^k, {y}^k)\\}$ satisfies\n\t\\[\n\t\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon.\n\t\\]\n\\end{proposition}","start":73976,"end":74265},{"raw_tex":"\\begin{proof}\n\t\n\tLet $(\\bar{x}, \\bar{y})$ be an accumulation point of $\\{(x^k, {y}^k)\\}$, and let \\(\\{k_j\\}_{j \\in \\mathbb{N}} \\subseteq \\mathbb{N} \\) be an index subsequence such that $(x^{k_j+1},y^{k_j+1})\\to (\\bar{x},\\bar{y})$ as $j \\rightarrow \\infty$. By Lemma \\ref{lem3}, we have  $\\lim_{k_j \\to\\infty} \\|(x^{k_j+1},y^{k_j+1})-(x^{k_j},\\tilde{y}^{k_j}) \\|=0$, which implies that $(x^{k_j},\\tilde{y}^{k_j})$ also converges to $ (\\bar{x},\\bar{y})$ .\n\t\n\tSince the sequence $\\{p_k\\}$ is bounded and non-decreasing by construction in AGILS (\\cref{alg1}), there exists $k_0 >0$ such that $p_k = p_{k_0}$ for all $k \\ge k_0$. By passing to a subsequence if necessary, we can assume that $k_j - 1\\ge k_0$ for all $j \\in \\mathbb{N}$.\t\n","start":74266,"end":74998},{"raw_tex":"\\begin{proof}\n\t\n\tLet $(\\bar{x}, \\bar{y})$ be an accumulation point of $\\{(x^k, {y}^k)\\}$, and let \\(\\{k_j\\}_{j \\in \\mathbb{N}} \\subseteq \\mathbb{N} \\) be an index subsequence such that $(x^{k_j+1},y^{k_j+1})\\to (\\bar{x},\\bar{y})$ as $j \\rightarrow \\infty$. By Lemma \\ref{lem3}, we have  $\\lim_{k_j \\to\\infty} \\|(x^{k_j+1},y^{k_j+1})-(x^{k_j},\\tilde{y}^{k_j}) \\|=0$, which implies that $(x^{k_j},\\tilde{y}^{k_j})$ also converges to $ (\\bar{x},\\bar{y})$ .\n\t\n\tSince the sequence $\\{p_k\\}$ is bounded and non-decreasing by construction in AGILS (\\cref{alg1}), there exists $k_0 >0$ such that $p_k = p_{k_0}$ for all $k \\ge k_0$. By passing to a subsequence if necessary, we can assume that $k_j - 1\\ge k_0$ for all $j \\in \\mathbb{N}$.\t\n\t\n\tAccording to the update rule of $p_k$ in AGILS (\\cref{alg1}), the condition $p_{k_j - 1} = p_{k_j}$ implies that one of the following two cases must occur for each \\(  k = k_j -1 \\):\n\t\n\tCase I: $\\left\\|(x^{k_j},y^{k_j})-(x^{k_j-1}, \\tilde{y}^{k_j-1})\\right\\| \\ge c_p \\min\\left\\{1/p_{k_0},{t^{k_j}}\\right\\}$ and $\\tilde{y}^{k_j} = y^{k_j}$.\n\t\n\tCase II: $\\left\\|(x^{k_j},y^{k_j})-(x^{k_j-1}, \\tilde{y}^{k_j-1})\\right\\| < c_p \\min\\left\\{1/p_{k_0},{t^{k_j}}\\right\\}$ and the feasibility correction procedure is applied to generate $\\tilde{y}^{k_j}$ satisfying condition \\eqref{inexacty_y}.\n\t\n\tAt least one of these two cases must hold for an infinite number of indices in the subsequence \t$\\{ k_j  - 1\\}$. We analyze them separately.\n\t\n\tCase 1:  Suppose Case I occurs for an infinite subsequence of indices. By passing to a further subsequence, we assume it holds for all $j \\in \\mathbb{N}$.\n\t\n\tThe condition $\\tilde{y}^{k_j} = y^{k_j}$ combined with $(x^{k_j},\\tilde{y}^{k_j})\\to (\\bar{x},\\bar{y})$ implies that $(x^{k_j}, {y}^{k_j})$\\\\\n\t$ \\to (\\bar{x},\\bar{y})$. \n\tFrom Lemma \\ref{lem3}, we know $\\lim_{k_j \\to\\infty} \\|(x^{k_j},y^{k_j})-(x^{k_j-1},\\tilde{y}^{k_j-1}) \\|=0$. For the condition of Case I to hold, we must therefore have  $t^{k_j} \\rightarrow 0$. This implies\n\t\\begin{equation*}\n\t\t\\limsup_{j \\rightarrow \\infty}\t\\left\\{ \\varphi(x^{k_j}, y^{k_j})-\\varphi(x^{k_j},\\theta^{k_j})-\\frac{1}{2\\gamma}\\left\\| \\theta^{k_j} - y^{k_j} \\right\\|^2-\\epsilon \\right\\} \\le 0.\n\t\\end{equation*}\n\tSince $\\zeta_{k} \\rightarrow 0$ and $\\mathcal{G}( \\theta^{k_j}, x^{k_j}, y^{k_j}) \\le \\zeta_{k_j}$, Lemma \\ref{error_bound} ensures that $\\| \\theta^{k_j} - \\theta^*_{\\gamma}( x^{k_j}, y^{k_j} )\\|\\rightarrow 0$.\n\tBy the continuity of $\\theta^*_{\\gamma}(x,y)$ (Proposition \\ref{partial_grad_v}), it follows that $\\theta^{k_j} \\rightarrow \\theta^*_{\\gamma}(\\bar{x}, \\bar{y}) $.\n\tThe continuity of $\\varphi$ then implies that $\\varphi(x^{k_j}, \\theta^{k_j})+\\frac{1}{2\\gamma}\\|\\theta^{k_j} - y^{k_j}\\|^2 \\rightarrow v_\\gamma(\\bar{x},\\bar{y})$. Then taking the limit in the inequality above as  $j \\rightarrow \\infty$ yields $\\varphi(\\bar{x},\\bar{y})-v_{\\gamma}(\\bar{x},\\bar{y})-\\epsilon \\le 0$. \n\t\n\tCase 2: \n\tNow, suppose Case II occurs for an infinite subsequence of indices. Again, we assume it holds for all $j \\in \\mathbb{N}$.\n\t\n\tIn this scenario, the feasibility correction procedure is applied at each iteration $k = k_j -1$ to generate $\\tilde{y}^{k_j}$ satisfying condition \\eqref{inexacty_y}. Since $(x^{k_j},\\tilde{y}^{k_j}) \\rightarrow (\\bar{x},\\bar{y}) $ and satisfies condition \\eqref{inexacty_y}, it follows from Lemma \\ref{lem4} that $\\bar{y} \\in S(\\bar{x})$. For any such point, $\\theta^*_{\\gamma}(\\bar{x}, \\bar{y}) = \\bar{y} $ (see, e.g., \\cite[Theorem 10.7]{beck2017first}), and therefore \\( \\varphi(\\bar{x}, \\bar{y}) - v_\\gamma(\\bar{x}, \\bar{y}) = 0 \\).  Since $ \\epsilon > 0$, the constraint $\t\t\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon$ is satisfied.\n\t\n\tSince any accumulation point must be the limit of a subsequence that falls into one of these two cases, we conclude that $\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon$.\n\\end{proof}","start":74266,"end":78191},{"raw_tex":"\t\n\tAccording to the update rule of $p_k$ in AGILS (\\cref{alg1}), the condition $p_{k_j - 1} = p_{k_j}$ implies that one of the following two cases must occur for each \\(  k = k_j -1 \\):\n\t\n\tCase I: $\\left\\|(x^{k_j},y^{k_j})-(x^{k_j-1}, \\tilde{y}^{k_j-1})\\right\\| \\ge c_p \\min\\left\\{1/p_{k_0},{t^{k_j}}\\right\\}$ and $\\tilde{y}^{k_j} = y^{k_j}$.\n\t\n\tCase II: $\\left\\|(x^{k_j},y^{k_j})-(x^{k_j-1}, \\tilde{y}^{k_j-1})\\right\\| < c_p \\min\\left\\{1/p_{k_0},{t^{k_j}}\\right\\}$ and the feasibility correction procedure is applied to generate $\\tilde{y}^{k_j}$ satisfying condition \\eqref{inexacty_y}.\n\t\n\tAt least one of these two cases must hold for an infinite number of indices in the subsequence \t$\\{ k_j  - 1\\}$. We analyze them separately.\n\t\n","start":74998,"end":75733},{"raw_tex":"\tCase 1:  Suppose Case I occurs for an infinite subsequence of indices. By passing to a further subsequence, we assume it holds for all $j \\in \\mathbb{N}$.\n\t\n\tThe condition $\\tilde{y}^{k_j} = y^{k_j}$ combined with $(x^{k_j},\\tilde{y}^{k_j})\\to (\\bar{x},\\bar{y})$ implies that $(x^{k_j}, {y}^{k_j})$\\\\\n\t$ \\to (\\bar{x},\\bar{y})$. \n\tFrom Lemma \\ref{lem3}, we know $\\lim_{k_j \\to\\infty} \\|(x^{k_j},y^{k_j})-(x^{k_j-1},\\tilde{y}^{k_j-1}) \\|=0$. For the condition of Case I to hold, we must therefore have  $t^{k_j} \\rightarrow 0$. This implies\n\t\\begin{equation*}\n\t\t\\limsup_{j \\rightarrow \\infty}\t\\left\\{ \\varphi(x^{k_j}, y^{k_j})-\\varphi(x^{k_j},\\theta^{k_j})-\\frac{1}{2\\gamma}\\left\\| \\theta^{k_j} - y^{k_j} \\right\\|^2-\\epsilon \\right\\} \\le 0.\n\t\\end{equation*}\n","start":75733,"end":76490},{"raw_tex":"\tSince $\\zeta_{k} \\rightarrow 0$ and $\\mathcal{G}( \\theta^{k_j}, x^{k_j}, y^{k_j}) \\le \\zeta_{k_j}$, Lemma \\ref{error_bound} ensures that $\\| \\theta^{k_j} - \\theta^*_{\\gamma}( x^{k_j}, y^{k_j} )\\|\\rightarrow 0$.\n\tBy the continuity of $\\theta^*_{\\gamma}(x,y)$ (Proposition \\ref{partial_grad_v}), it follows that $\\theta^{k_j} \\rightarrow \\theta^*_{\\gamma}(\\bar{x}, \\bar{y}) $.\n\tThe continuity of $\\varphi$ then implies that $\\varphi(x^{k_j}, \\theta^{k_j})+\\frac{1}{2\\gamma}\\|\\theta^{k_j} - y^{k_j}\\|^2 \\rightarrow v_\\gamma(\\bar{x},\\bar{y})$. Then taking the limit in the inequality above as  $j \\rightarrow \\infty$ yields $\\varphi(\\bar{x},\\bar{y})-v_{\\gamma}(\\bar{x},\\bar{y})-\\epsilon \\le 0$. \n\t\n","start":76490,"end":77185},{"raw_tex":"\tCase 2: \n\tNow, suppose Case II occurs for an infinite subsequence of indices. Again, we assume it holds for all $j \\in \\mathbb{N}$.\n\t\n\tIn this scenario, the feasibility correction procedure is applied at each iteration $k = k_j -1$ to generate $\\tilde{y}^{k_j}$ satisfying condition \\eqref{inexacty_y}. Since $(x^{k_j},\\tilde{y}^{k_j}) \\rightarrow (\\bar{x},\\bar{y}) $ and satisfies condition \\eqref{inexacty_y}, it follows from Lemma \\ref{lem4} that $\\bar{y} \\in S(\\bar{x})$. For any such point, $\\theta^*_{\\gamma}(\\bar{x}, \\bar{y}) = \\bar{y} $ (see, e.g., \\cite[Theorem 10.7]{beck2017first}), and therefore \\( \\varphi(\\bar{x}, \\bar{y}) - v_\\gamma(\\bar{x}, \\bar{y}) = 0 \\).  Since $ \\epsilon > 0$, the constraint $\t\t\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon$ is satisfied.\n\t\n\tSince any accumulation point must be the limit of a subsequence that falls into one of these two cases, we conclude that $\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon$.\n\\end{proof}","start":77185,"end":78191}]},{"id":"S069","semantic_type":"theorem","kind":"theorem","labels":["th1"],"member_ids":["S069"],"declared_external":false,"source_only":false,"title":"聚点满足更紧松弛问题KKT条件","source_units":[{"raw_tex":"\\begin{theorem}\\label{th1}\n\tLet $\\epsilon > 0$. If the sequence $\\{p_k\\}$ is bounded, then any accumulation point $(\\bar{x}, \\bar{y})$ of $\\{(x^k,y^k)\\}$ is a KKT point of $({\\rm VP})_\\gamma^{\\bar{\\epsilon}}$ for some  $\\bar{\\epsilon}\\le\\epsilon$. \n\\end{theorem}","start":92183,"end":92445}]},{"id":"S070","semantic_type":"proof","kind":"proof","labels":["th1_eq2","th1_eq4","th1_eq5","th1_eq6"],"member_ids":["S070"],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\tThe proof builds upon techniques used in the proof of Theorem 1 in \\cite{ye2023}. \n\tLet $(\\bar{x}, \\bar{y})$ be an accumulation point of the sequence $\\{(x^k,y^k)\\}$, and let $\\{(x^{k_j+1},y^{k_j+1})\\}$ be a convergent subsequence such that $(x^{k_j+1},y^{k_j+1}) \\rightarrow (\\bar{x}, \\bar{y})$ as $k_j \\rightarrow \\infty$. \n\tBy Lemma \\ref{lem3}, we have  $\\lim_{k_j \\to\\infty} \\|(x^{k_j+1},y^{k_j+1})-(x^{k_j},\\tilde{y}^{k_j}) \\|=0$, which implies that $(x^{k_j},\\tilde{y}^{k_j}) \\to (\\bar{x},\\bar{y})$ .\n\tSince $\\{p_k\\}$ is bounded, there exists $k_0 > 0$ such that $p_k = \\bar{p} := p_{k_0}$ for any $k \\ge k_0$. \n\t\n\t\n\tFrom the update rules \\eqref{update_x} and \\eqref{update_y}, and the expressions for $d_x^k$ and $d_y^k$, we have    \n\t\\begin{equation}\\label{th1_eq2}\n\t\t\\begin{aligned}\n\t\t\t\\xi_x^k & \\in \\nabla_x F(x^{k+1},y^{k+1})+p_k \\left(\\nabla_x \\varphi(x^{k+1},y^{k+1}) - \\nabla_x v_\\gamma(x^{k+1},y^{k+1})\\right) +\\mathcal{N}_{X}(x^{k+1}),\\\\\n\t\t\t\\xi_y^k & \\in \\nabla_y F(x^{k},y^{k+1})+p_k \\left(\\partial_y \\varphi(x^{k},y^{k+1}) - \\nabla_y v_\\gamma(x^{k},y^{k+1})\\right) +\\mathcal{N}_{Y}(y^{k+1}),\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere $\\xi_x^k \\in \\Re^n$ and $\\xi_y^k \\in \\Re^m$ are given by\n\t\\begin{equation}\\label{th1_eq4}\n\t\t\\begin{aligned}\n\t\t\t\\xi_x^k &:=  p_k \\left(\\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - d_x^k \\right)  - \\frac{p_k}{\\alpha_k}\\left(x^{k+1}-x^k\\right), \\\\\n\t\t\t\\xi_y^k &:=  p_k \\left( \\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, y^{k+1}\\right) - d_y^k \\right)-\\frac{p_k}{\\beta_k}\\left(y^{k+1}-\\tilde{y}^k\\right).\n\t\t\\end{aligned} \n\t\\end{equation} \n\tWe now show that $\\xi_x^{k_j}, \\xi_y^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$. First, we have \n\t\\begin{equation}\n\t\t\\begin{aligned}\n\t\t\t& \\| \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - d_x^k\\| \\\\ \\le \\, & \\|  \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k}, y^{k+1} \\right) \\| + \\| \\nabla_x \\tilde{\\psi}_{p_{k}} \\left(x^{k}, y^{k+1} \\right) - d_x^k \\| \\\\\n\t\t\t\\le \\, & \\|  \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k}, y^{k+1} \\right) \\| + (L_{f_x} + L_{g_2})C_\\eta \\zeta_{k}, \n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere the last inequality follows from \\eqref{lem1_eq10}. Since $p_k = \\bar{p}$ for all $k \\ge k_0$, and assuming that $\\nabla_{x} F, \\nabla_{x} f$ and $\\nabla_{x} g$ are continuous on $X \\times Y$, and $\\nabla_{x} v_\\gamma$ is continuous on $X \\times Y$ (Proposition \\ref{partial_grad_v}), we conclude that $ \\nabla_x \\tilde{\\psi}_{p_{k}} =  \\nabla_x \\tilde{\\psi}_{\\bar{p}}$ is continuous on $X \\times Y$. \n\tAs $\\lim_{k_j\\to\\infty} (x^{k_j+1},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},\\tilde{y}^{k_j}) = (\\bar{x}, \\bar{y})$ and $\\zeta_{k} \\rightarrow 0$, it follows that $\\nabla_x \\tilde{\\psi}_{p_{k_j}} (x^{k_j+1}, y^{k_j+1} ) - d_x^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$. Combining this with $ \\|x^{k_j+1}-x^{k_j} \\| \\to 0$ and $\\alpha_k \\ge \\underline{\\alpha} > 0$, we obtain from \\eqref{th1_eq4} that $\\xi_x^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$.\n\t\n\tA similar argument can be applied to $\\xi_y^k$. From \\eqref{lem1_eq11}, we have \n\t\\begin{equation}\\label{th1_eq5}\n\t\t\\begin{aligned}\n\t\t\t& \\|\\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, y^{k+1}\\right) - d_y^k \\| \\\\ \\le \\, & \\| \\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, y^{k+1} \\right) - \\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, \\tilde{y}^k \\right) \\| + {C_\\eta}\\zeta_{k}/\\gamma.\n\t\t\\end{aligned}\n\t\\end{equation}\n\tSince $p_k = \\bar{p}$ for all $k \\ge k_0$, and under the assumption that $\\nabla_{y} F$ and $\\nabla_{y} f$ are continuous on $X \\times Y$, and $\\nabla_{y} v_\\gamma$ is continuous on $X \\times Y$ (Proposition \\ref{partial_grad_v}), we have that $ \\nabla_y (\\tilde{\\psi}_{p_{k}} - g) = \\nabla_y (\\tilde{\\psi}_{\\bar{p}} - g)$ is continuous on $X \\times Y$. As $\\lim_{k_j\\to\\infty} (x^{k_j},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},\\tilde{y}^{k_j}) = (\\bar{x}, \\bar{y})$ and $\\zeta_{k} \\rightarrow 0$, it follows that $\\nabla_y (\\tilde{\\psi}_{p_{k_j}} - g)(x^{k_j}, y^{k_j+1} ) - d_y^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$. Combining this with $ \\|y^{k_j+1}-\\tilde{y}^{k_j} \\| \\to 0$ and $\\beta_k \\ge \\underline{\\beta} > 0$, we conclude from \\eqref{th1_eq5} that $\\xi_y^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$.\n\t\n\tNow, taking the limit as $k = k_j \\rightarrow \\infty$ in \\eqref{th1_eq2}, since $\\xi_x^{k_j}, \\xi_y^{k_j} \\rightarrow 0$, $p_k = \\bar{p}$ for all $k \\ge k_0$, $\\lim_{k_j\\to\\infty} (x^{k_j+1},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},\\tilde{y}^{k_j}) = (\\bar{x}, \\bar{y})$, $\\nabla F$, $\\nabla v_\\gamma$, and $\\nabla_x \\varphi$ are continuous on $X \\times Y$, $\\partial_y \\varphi$ and $\\mathcal{N}_{X\\times Y}$ are outer semicontinuous on $X \\times Y$, and $\\partial \\varphi = \\{\\nabla_x \\varphi\\} \\times \\partial_y \\varphi$ on $X \\times Y$, we obtain \n\t\\begin{equation}\\label{th1_eq6}\n\t\t0\\in\\nabla F(\\bar{x}, \\bar{y})+\\bar{p}\\left(\\partial \\varphi(\\bar{x},\\bar{y})-\\nabla v_\\gamma(\\bar{x}, \\bar{y})\\right)+\\mathcal{N}_{X\\times Y}(\\bar{x}, \\bar{y}).\n\t\\end{equation}\n\t\n\tFinally, from Proposition \\ref{prop0}, we know that the limit point $(\\bar{x}, \\bar{y})$ satisfies $\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon $. Let $\\bar{\\epsilon}:= \\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\in [0, \\epsilon]$. Combining this with \\eqref{th1_eq6}, we have\n\t\\begin{equation}\n\t\t\\left\\{\n\t\t\\begin{aligned}\n\t\t\t&0\\in\\nabla F(\\bar{x}, \\bar{y}) + \\bar{p}\\left( \\partial \\varphi(\\bar{x}, \\bar{y}) - \\nabla v_\\gamma(\\bar{x}, \\bar{y})\\right)+\\mathcal{N}_{X\\times Y}(\\bar{x}, \\bar{y}),\\\\\n\t\t\t&\\varphi(\\bar{x}, \\bar{y}) - v_{\\gamma}(\\bar{x}, \\bar{y})- \\bar{\\epsilon} \\le 0, \\qquad  \\bar{p}\\left( \\varphi(\\bar{x}, \\bar{y}) - v_{\\gamma}(\\bar{x}, \\bar{y})- \\bar{\\epsilon} \\right)=0,\n\t\t\\end{aligned}\\right.\n\t\\end{equation}\n\twhich implies that $(\\bar{x},\\bar{y})$ is a KKT point of problem $({\\rm VP})_\\gamma^{\\bar{\\epsilon}}$ with $\\bar{\\epsilon}\\le \\epsilon$.\n\\end{proof}","start":92446,"end":98589}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S071"],"declared_external":false,"source_only":false,"title":"选择子序列与恒定罚参数","source_units":[{"raw_tex":"\\begin{proof}\n\tThe proof builds upon techniques used in the proof of Theorem 1 in \\cite{ye2023}. \n\tLet $(\\bar{x}, \\bar{y})$ be an accumulation point of the sequence $\\{(x^k,y^k)\\}$, and let $\\{(x^{k_j+1},y^{k_j+1})\\}$ be a convergent subsequence such that $(x^{k_j+1},y^{k_j+1}) \\rightarrow (\\bar{x}, \\bar{y})$ as $k_j \\rightarrow \\infty$. \n\tBy Lemma \\ref{lem3}, we have  $\\lim_{k_j \\to\\infty} \\|(x^{k_j+1},y^{k_j+1})-(x^{k_j},\\tilde{y}^{k_j}) \\|=0$, which implies that $(x^{k_j},\\tilde{y}^{k_j}) \\to (\\bar{x},\\bar{y})$ .\n\tSince $\\{p_k\\}$ is bounded, there exists $k_0 > 0$ such that $p_k = \\bar{p} := p_{k_0}$ for any $k \\ge k_0$. \n","start":92446,"end":93079}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":["th1_eq2","th1_eq4"],"member_ids":["S072"],"declared_external":false,"source_only":false,"title":"两块最优性条件与扰动定义","source_units":[{"raw_tex":"\t\n\t\n\tFrom the update rules \\eqref{update_x} and \\eqref{update_y}, and the expressions for $d_x^k$ and $d_y^k$, we have    \n\t\\begin{equation}\\label{th1_eq2}\n\t\t\\begin{aligned}\n\t\t\t\\xi_x^k & \\in \\nabla_x F(x^{k+1},y^{k+1})+p_k \\left(\\nabla_x \\varphi(x^{k+1},y^{k+1}) - \\nabla_x v_\\gamma(x^{k+1},y^{k+1})\\right) +\\mathcal{N}_{X}(x^{k+1}),\\\\\n\t\t\t\\xi_y^k & \\in \\nabla_y F(x^{k},y^{k+1})+p_k \\left(\\partial_y \\varphi(x^{k},y^{k+1}) - \\nabla_y v_\\gamma(x^{k},y^{k+1})\\right) +\\mathcal{N}_{Y}(y^{k+1}),\n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere $\\xi_x^k \\in \\Re^n$ and $\\xi_y^k \\in \\Re^m$ are given by\n\t\\begin{equation}\\label{th1_eq4}\n\t\t\\begin{aligned}\n\t\t\t\\xi_x^k &:=  p_k \\left(\\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - d_x^k \\right)  - \\frac{p_k}{\\alpha_k}\\left(x^{k+1}-x^k\\right), \\\\\n\t\t\t\\xi_y^k &:=  p_k \\left( \\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, y^{k+1}\\right) - d_y^k \\right)-\\frac{p_k}{\\beta_k}\\left(y^{k+1}-\\tilde{y}^k\\right).\n\t\t\\end{aligned} \n\t\\end{equation} \n","start":93079,"end":94071}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S073"],"declared_external":false,"source_only":false,"title":"x块最优性扰动消失","source_units":[{"raw_tex":"\tWe now show that $\\xi_x^{k_j}, \\xi_y^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$. First, we have \n\t\\begin{equation}\n\t\t\\begin{aligned}\n\t\t\t& \\| \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - d_x^k\\| \\\\ \\le \\, & \\|  \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k}, y^{k+1} \\right) \\| + \\| \\nabla_x \\tilde{\\psi}_{p_{k}} \\left(x^{k}, y^{k+1} \\right) - d_x^k \\| \\\\\n\t\t\t\\le \\, & \\|  \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k+1}, y^{k+1} \\right) - \\nabla_x \\tilde{\\psi}_{p_{k}}\\left(x^{k}, y^{k+1} \\right) \\| + (L_{f_x} + L_{g_2})C_\\eta \\zeta_{k}, \n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere the last inequality follows from \\eqref{lem1_eq10}. Since $p_k = \\bar{p}$ for all $k \\ge k_0$, and assuming that $\\nabla_{x} F, \\nabla_{x} f$ and $\\nabla_{x} g$ are continuous on $X \\times Y$, and $\\nabla_{x} v_\\gamma$ is continuous on $X \\times Y$ (Proposition \\ref{partial_grad_v}), we conclude that $ \\nabla_x \\tilde{\\psi}_{p_{k}} =  \\nabla_x \\tilde{\\psi}_{\\bar{p}}$ is continuous on $X \\times Y$. \n\tAs $\\lim_{k_j\\to\\infty} (x^{k_j+1},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},\\tilde{y}^{k_j}) = (\\bar{x}, \\bar{y})$ and $\\zeta_{k} \\rightarrow 0$, it follows that $\\nabla_x \\tilde{\\psi}_{p_{k_j}} (x^{k_j+1}, y^{k_j+1} ) - d_x^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$. Combining this with $ \\|x^{k_j+1}-x^{k_j} \\| \\to 0$ and $\\alpha_k \\ge \\underline{\\alpha} > 0$, we obtain from \\eqref{th1_eq4} that $\\xi_x^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$.\n","start":94071,"end":95591}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":["th1_eq5"],"member_ids":["S074"],"declared_external":false,"source_only":false,"title":"y块最优性扰动消失","source_units":[{"raw_tex":"\t\n\tA similar argument can be applied to $\\xi_y^k$. From \\eqref{lem1_eq11}, we have \n\t\\begin{equation}\\label{th1_eq5}\n\t\t\\begin{aligned}\n\t\t\t& \\|\\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, y^{k+1}\\right) - d_y^k \\| \\\\ \\le \\, & \\| \\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, y^{k+1} \\right) - \\nabla_y\\left(\\tilde{\\psi}_{p_k}-g\\right)\\left(x^{k}, \\tilde{y}^k \\right) \\| + {C_\\eta}\\zeta_{k}/\\gamma.\n\t\t\\end{aligned}\n\t\\end{equation}\n\tSince $p_k = \\bar{p}$ for all $k \\ge k_0$, and under the assumption that $\\nabla_{y} F$ and $\\nabla_{y} f$ are continuous on $X \\times Y$, and $\\nabla_{y} v_\\gamma$ is continuous on $X \\times Y$ (Proposition \\ref{partial_grad_v}), we have that $ \\nabla_y (\\tilde{\\psi}_{p_{k}} - g) = \\nabla_y (\\tilde{\\psi}_{\\bar{p}} - g)$ is continuous on $X \\times Y$. As $\\lim_{k_j\\to\\infty} (x^{k_j},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},\\tilde{y}^{k_j}) = (\\bar{x}, \\bar{y})$ and $\\zeta_{k} \\rightarrow 0$, it follows that $\\nabla_y (\\tilde{\\psi}_{p_{k_j}} - g)(x^{k_j}, y^{k_j+1} ) - d_y^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$. Combining this with $ \\|y^{k_j+1}-\\tilde{y}^{k_j} \\| \\to 0$ and $\\beta_k \\ge \\underline{\\beta} > 0$, we conclude from \\eqref{th1_eq5} that $\\xi_y^{k_j} \\rightarrow 0$ as $k_j \\rightarrow \\infty$.\n","start":95591,"end":96865}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":["th1_eq6"],"member_ids":["S075"],"declared_external":false,"source_only":false,"title":"传递到联合驻点包含式","source_units":[{"raw_tex":"\t\n\tNow, taking the limit as $k = k_j \\rightarrow \\infty$ in \\eqref{th1_eq2}, since $\\xi_x^{k_j}, \\xi_y^{k_j} \\rightarrow 0$, $p_k = \\bar{p}$ for all $k \\ge k_0$, $\\lim_{k_j\\to\\infty} (x^{k_j+1},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},y^{k_j+1}) = \\lim_{k_j\\to\\infty} (x^{k_j},\\tilde{y}^{k_j}) = (\\bar{x}, \\bar{y})$, $\\nabla F$, $\\nabla v_\\gamma$, and $\\nabla_x \\varphi$ are continuous on $X \\times Y$, $\\partial_y \\varphi$ and $\\mathcal{N}_{X\\times Y}$ are outer semicontinuous on $X \\times Y$, and $\\partial \\varphi = \\{\\nabla_x \\varphi\\} \\times \\partial_y \\varphi$ on $X \\times Y$, we obtain \n\t\\begin{equation}\\label{th1_eq6}\n\t\t0\\in\\nabla F(\\bar{x}, \\bar{y})+\\bar{p}\\left(\\partial \\varphi(\\bar{x},\\bar{y})-\\nabla v_\\gamma(\\bar{x}, \\bar{y})\\right)+\\mathcal{N}_{X\\times Y}(\\bar{x}, \\bar{y}).\n\t\\end{equation}\n","start":96865,"end":97675}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S076"],"declared_external":false,"source_only":false,"title":"实际间隙定义松弛与互补","source_units":[{"raw_tex":"\t\n\tFinally, from Proposition \\ref{prop0}, we know that the limit point $(\\bar{x}, \\bar{y})$ satisfies $\\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\le \\epsilon $. Let $\\bar{\\epsilon}:= \\varphi(\\bar{x}, \\bar{y}) -  v_\\gamma(\\bar{x},\\bar{y}) \\in [0, \\epsilon]$. Combining this with \\eqref{th1_eq6}, we have\n\t\\begin{equation}\n\t\t\\left\\{\n\t\t\\begin{aligned}\n\t\t\t&0\\in\\nabla F(\\bar{x}, \\bar{y}) + \\bar{p}\\left( \\partial \\varphi(\\bar{x}, \\bar{y}) - \\nabla v_\\gamma(\\bar{x}, \\bar{y})\\right)+\\mathcal{N}_{X\\times Y}(\\bar{x}, \\bar{y}),\\\\\n\t\t\t&\\varphi(\\bar{x}, \\bar{y}) - v_{\\gamma}(\\bar{x}, \\bar{y})- \\bar{\\epsilon} \\le 0, \\qquad  \\bar{p}\\left( \\varphi(\\bar{x}, \\bar{y}) - v_{\\gamma}(\\bar{x}, \\bar{y})- \\bar{\\epsilon} \\right)=0,\n\t\t\\end{aligned}\\right.\n\t\\end{equation}\n\twhich implies that $(\\bar{x},\\bar{y})$ is a KKT point of problem $({\\rm VP})_\\gamma^{\\bar{\\epsilon}}$ with $\\bar{\\epsilon}\\le \\epsilon$.\n\\end{proof}","start":97675,"end":98589}]},{"id":"S086","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S086"],"declared_external":true,"declaration_text":["原前提列表：mordukhovich2018variational, Corollary 4.3(ii), Theorems 1.22, 4.1(ii), 4.17；作者实际引用见第226行。"],"source_only":false,"title":"边际函数敏感性结果","source_units":[]},{"id":"S087","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S087"],"declared_external":true,"declaration_text":["原前提列表：gao2024, Theorem 2 的证明论证；引用见第226行。"],"source_only":false,"title":"偏弱凸性的外部证明思路","source_units":[]},{"id":"S088","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S088"],"declared_external":true,"declaration_text":["原前提列表：ye2021variational, Proposition 2；作者引用见第564行。"],"source_only":false,"title":"强凸目标的KL误差估计","source_units":[]},{"id":"S089","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S089"],"declared_external":true,"declaration_text":["原前提列表：beck2017first, Theorem 6.39；作者引用见第673行。"],"source_only":false,"title":"强凸极小点比较不等式","source_units":[]},{"id":"S090","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S090"],"declared_external":true,"declaration_text":["原前提列表：beck2017first, Theorem 10.7；作者在第853行引用。"],"source_only":false,"title":"凸最优解的近端固定点性质","source_units":[]},{"id":"S091","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S091"],"declared_external":true,"declaration_text":["原前提列表中的 Standard convex optimality, projection and proximal calculus；具体常识步骤由注释者显式标出。"],"source_only":false,"title":"凸最优性与近端投影计算","source_units":[]},{"id":"S092","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S092"],"declared_external":true,"declaration_text":["原前提列表中的 Standard descent inequalities；具体常识步骤由注释者显式标出。"],"source_only":false,"title":"下降引理与内积误差吸收","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S039","S042":"S039","S043":"S039","S044":"S039","S045":"S045","S046":"S039","S047":"S039","S048":"S039","S049":"S049","S050":"S039","S051":"S039","S052":"S052","S053":"S053","S054":"S053","S055":"S053","S056":"S053","S057":"S057","S058":"S057","S059":"S057","S060":"S057","S061":"S057","S062":"S062","S063":"S062","S064":"S062","S065":"S062","S066":"S062","S067":"S062","S068":"S062","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076","S077":"S015","S078":"S015","S079":"S015","S080":"S034","S081":"S034","S082":"S034","S083":"S034","S084":"S034","S085":"S034","S086":"S086","S087":"S087","S088":"S088","S089":"S089","S090":"S090","S091":"S091","S092":"S092"}}}}
{"record_id":"2412.13191v3","paper":{"title":"Moment-sos and spectral hierarchies for polynomial optimization on the sphere and quantum de Finetti theorems","authors":["Blomenhofer, Alexander Taveira","Laurent, Monique"],"arxiv_id":"2412.13191","version":"v3","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/2412.13191v3","pdf_url":"https://arxiv.org/pdf/2412.13191v3","doi":"10.1137/24M1717750"},"context":{"raw_tex":"Throughout we consider the polynomial optimization problem, asking to minimize a polynomial $p$ over the unit sphere $\\oS^{n-1}$, i.e., to compute\n\\begin{align}\\label{eqpmin}\np_{\\min}=\\min_{x\\in \\oS^{n-1}}p(x)=\\min\\{p(x): x\\in \\R^n, \\|x\\|_2=1\\}.\n\\end{align}\nTherein, $p\\in \\R[x]=\\R[x_1,\\ldots,x_n]$ is an $n$-variate polynomial of degree $2d$. In other words,\n\n\nWe begin with some notation that is used throughout the paper. $\\N=\\{0,1,2,\\ldots\\}$ is the set of nonnegative integers. For an integer $d\\in \\N$, $\\N^n_d$ denotes the set of sequences $\\alpha\\in \\N^n$ with $|\\alpha|:=\\alpha_1+\\ldots+\\alpha_n=d$. We let $\\MFS_d$ denote the set of permutations of the $d$-element set $[d]=\\{1,\\ldots,d\\}$. We let $e_1,\\ldots,e_n$ denote the standard unit vectors in $\\C^n$, where all entries of $e_i$ are 0 except 1 at the $i$th coordinate. For a vector $u\\in \\C^n$,  $\\|u \\|_2=\\sqrt {u^*u}$ denote its Euclidean norm. The unit sphere in $\\R^n$ is  denoted $\\oS^{n-1}=\\{x\\in\\R^n: \\|x\\|_2=\\sum_{i=1}^nx_i^2=1\\}$.  We let $ \\conv(S) $ denote   the convex  hull of a set $ S $ and  $ \\cone(S) $ is its conic hull.  \n\nFor $\\K=\\R$ or $\\C$, we use the trace inner product on the space $\\K^{n\\times n}$ of matrices:  $\\langle A,B\\rangle= \\Tr(A^*B)$ for $A,B\\in \\K^{n\\times n}$. Here, $A^*= \\bar A^T$ denotes the complex conjugate transpose of $A$; $A$ is Hermitian if $A^*=A$. For a Hermitian matrix $A$, $\\lambda_{\\max}(A)$ and $\\lambda_{\\min}(A)$ denote, respectively, its largest and smallest eigenvalues.  The matrix $A$ is positive semidefinite, written as $A\\succeq 0$, if $\\lambda_{\\min}(A)\\ge 0$. For Hermitian matrices $ A, B $, we write $ A\\preceq B $ if $ B-A $ is positive semidefinite. \\tcolblue{Throughout, $I_n$ denotes the $n\\times n$ identity matrix.}\n\n\nLet $\\K=\\R$ or $\\C$ be the real or complex field. For integers $r,n\\ge 1$, $(\\K^n)^{\\ot r}$ denotes the space of $r$-tensors on $\\K^n$, with elements $v=(v_{\\ui})_{\\ui \\in [n]^r}$. Here, we use the notation $\\ui=(i_1,\\ldots,i_r)\\in [n]^r$ (or simply $i_1\\ldots i_r$) for an $r$-sequence in $[n]^r$. Given two integers $1\\le d\\le r$, it may also be convenient to denote  $r$-sequences as\nthe concatenation $\\uj \\uk $ of a $d$-sequence $\\uj\\in [n]^d$ and an $(r-d)$-sequence $\\uk\\in [n]^{r-d}$.\n\\tcolblue{For  vectors $u_i\\in \\K^{n_i}$, $i=1,2$, their tensor product is the vector $u_1\\ot u_2\\in\\K^{n_1}\\ot \\K^{n_2}$ with entries\n$(u_1\\ot u_2)_{i_1i_2}=(u_ 1)_{i_1}(u_2)_{i_2}$ for $(i_1,i_2)\\in[n_1]\\times [n_2]$. For $u\\in\\K^n$, $u^{\\ot r} = u\\ot \\cdots \\ot u$ (taking the tensor product of $r$ copies).}\n\n We now recall several notions of symmetry for tensors. \\tcolblue{In Appendix \\ref{appendix-example} we illustrate some   of these notions on examples.}\n \nPermutations $\\sigma\\in \\MFS_r$ act on $[n]^r$ and thus on $r$-tensors, in the following way.\nFor $\\ui\\in [n]^r$, set $\\ui^\\sigma=(i_{\\sigma(1)},\\ldots, i_{\\sigma(r)})$ and, for  $v\\in (\\K^n)^{\\ot r}$, define $v^\\sigma$ as the $r$-tensor with entries $(v^\\sigma)_{\\ui}= v_{\\ui^\\sigma}$ for $\\ui\\in [n]^r$.\nThen, $v$ is said to be a {\\em symmetric tensor} if $v^\\sigma=v$ for all $\\sigma\\in \\MFS_r$, and $S^r(\\K^n)$ denotes the space of symmetric $r$-tensors on $\\K^n$. We let $\\Pi_r$ denote the projection from the space $(\\K^n)^{\\ot r}$ to the symmetric subspace $S^r(\\K^n)$. Concretely, we have \n\\begin{align}\\label{eqPir}\n\\Pi_r=\\frac{1}{r!}\\sum_{\\sigma\\in \\MFS_r} P_\\sigma,\n\\end{align}\nwhere $P_\\sigma$ is the square matrix indexed by $[n]^r\\times [n]^r$, with entries \n$(P_\\sigma)_{\\ui, \\uj}=1$ if $\\uj=\\ui^\\sigma$ and 0 otherwise, for $\\ui,\\uj\\in [n]^r$.\n\nConsider a  matrix $M$ indexed by the set $[n]^r\\times [n]^r$, so that $M$ represents an endomorphism from $(\\K^n)^{\\ot r}$ to $(\\K^n)^{\\ot r}$. Then, any permutation $\\sigma\\in \\MFS_r$  induces naturally a permutation of  the columns (and rows) of $M$. \n\n\\begin{definition}[Doubly symmetric matrix]\\label{defdoublesymmetric}\nA matrix $M\\in \\K^{[n]^r\\times [n]^r}$  is called  {\\em $r$-column symmetric} (resp., {\\em $r$-row symmetric}) if $M\\Pi_r =M$ (resp., $\\Pi_rM=M$). If $M$ is complex Hermitian or real symmetric, then $M$ is $r$-column symmetric if and only if it is $r$-row symmetric, or equivalently if $\\Pi_rM\\Pi_r=M$, in which case  $M$ is called {\\em $ r $-doubly symmetric} (or just {\\em doubly symmetric} if $r$ is clear from the context).\n\\end{definition}\n\n\nHermitian doubly symmetric matrices are also known as {\\em Bose symmetric} matrices in the quantum information literature. \\MoL{The following stronger symmetry property is sometimes needed (it is considered, e.g.,   in \\cite{Doherty_Wehner_2013}).}\n\n\\begin{definition}[Maximally symmetric matrix]\\label{defmaximallysymmetric}\nA  matrix $M\\in \\K^{[n]^r\\times [n]^r}$ can be   viewed as  a $2r$-tensor $\\vvec(M)\\in (\\K^n)^{\\ot 2r}$, with entries  $\\vvec(M)_{\\ui\\uj} = M_{\\ui,\\uj}$ for $\\ui,\\uj\\in [n]^r$.\nThen, $M$ is said to be {\\em maximally symmetric} when $\\vvec(M)$ is a symmetric $2r$-tensor.\n\\end{definition}\n\n\\tcolred{Note that a Hermitian matrix that is maximally symmetric is in fact real-valued (see also Lemma \\ref{lem:ppt-sym-is-real-rank-2} below).} Let $\\PiMS$ denote the projection from the space $\\K^{[n]^r\\times [n]^r}$ onto $\\MaxSym_r(\\K^n)$, the  space of maximally symmetric matrices. Concretely, the projection $\\PiMS(M)$ is obtained by first constructing the symmetric $2r$-tensor $\\Pi_{2r}(\\vvec(M))$ and then reshaping it as a square matrix indexed by $[n]^r$ (by splitting the $2r$ indices into two classes of size $r$, which can be done   arbitrarily  as the tensor is $2r$-symmetric).\n\n\nThroughout, let $ R := \\R[x] = \\R[x_1,\\ldots,x_n] $ denote  the polynomial ring in variables $ x=(x_1,\\ldots,x_n) $. For an integer $d\\in\\N$,  $ R_d $ denotes its $ d $-th graded component, which consists of the homogeneous polynomials (aka forms) with degree $d$, and $R_{\\le d}=R_0\\oplus R_1\\oplus\\ldots\\oplus R_d$ consists of the polynomials with degree at most $d$. \nWe let $[x]_d= (x^\\alpha)_{\\alpha\\in \\N^n_d}$ denote the vector of all monomials with degree $d$.\nThroughout we use the notation  $$s=x_1^2+\\ldots +x_n^2.$$\n\n\nA polynomial $f\\in R$ is a {\\em sum of squares} (abbreviated as sos) if $ f = p_1^2  + \\ldots + p_N^2 $ for some $ N\\in \\N $ and $ p_1,\\ldots,p_N\\in R$. Then, all $p_i$ lie in $R_d$ (resp., $R_{\\le d}$) if $f\\in R_{2d}$ (resp., $f\\in R_{\\le 2d}$). The following characterization is well-known \\tcolblue{(see \\cite{Choi-Lam-Reznick-1995,Powers-Wormann-1998})}.\n\n\\begin{lemma}\\label{lem:sos-equiv-gram-matrix}\nA polynomial $f$ is a sum of squares if and only if it admits a positive semidefinite representing matrix (then, often called a {\\em Gram matrix} of $f$).\n\\end{lemma}\n\n\n\\begin{definition} \\label{defGramtensor}\n\tLet $f\\in R_{2d}$. A symmetric matrix $G$ indexed by $[n]^d$ is called a {\\em tensor-indexed representing matrix} of  $ f $ if the following polynomial identity holds:\n\t\\begin{align*}\n\t\tf = (x^{\\otimes d})^{T}G x^{\\otimes d} = \\langle G, (x x^{T})^{\\otimes d} \\rangle. \n\t\\end{align*}\n\\end{definition}\n\t\n\tThen, any polynomial  $f\\in R_{2d}$ admits  a (unique) tensor-indexed representing matrix  that is maximally symmetric, denoted by $\\MaxSym(f)$. Namely, we have $\\MaxSym(f)=\\PiMS(G)$, where $G$ is any tensor-indexed representing matrix of $f$.\nMoreover,\nin analogy with \\Cref{lem:sos-equiv-gram-matrix}, a polynomial $ f $ is a sum of squares if and only if it has a tensor-indexed positive semidefinite representing matrix. \n\n\n\\MoL{\\johnstonlovitz{} \\cite{Johnston_Lovitz_Vijayaraghavan_2023} proposed recently a hierarchy of spectral bounds $\\spec_r(p)$ for the minimum value $p_{\\min}$ of a homogeneous polynomial $p$ over the unit sphere,\\footnote{ Their hierarchy  applies more generally to optimization of  a multivariate tensor over a spherical Segre-Veronese variety. We only focus here on their hierarchy for polynomials on the sphere.} \nsee Section \\ref{secJLV-intro} below for their definition and relationship to the  moment-sos bounds.  Throughout, let $p\\in R_{2d}$, $s=x_1^2+\\cdots +x_n^2$, and let \n\\begin{align}\\label{eqQpM}\n\tQ(p) = \\maxsym(p), \\quad  M = \\maxsym(s^d) \n\\end{align}\ndenote  the maximally symmetric  matrices representing the polynomials $ p $ and $s^d$, respectively (as in Definition \\ref{defGramtensor}). They show the following convergence rate.\n}\n\n\nWe introduce the spectral hierarchy of \\johnstonlovitz{} \\cite{Johnston_Lovitz_Vijayaraghavan_2023}. \nLet $p\\in R_{2d}$ and recall the matrices $Q(p)$   and $M$ from (\\ref{eqQpM}). For an integer $ r\\ge d $, define the matrices \n\\begin{align}\\label{eqQrpM}\n\tQ_r(p) = \\Pi_{r}(Q(p) \\otimes I_n^{\\otimes (r-d)})\\Pi_{r}, \\\\\n\tM_r= \\Pi_{r}(M \\otimes I_n^{\\otimes (r-d)})\\Pi_{r}. \n\\end{align}\nSo,  $ Q_r(p) $ and $M_r$ are   $ r $-doubly symmetric matrices    representing  $ s^{r-d}p $ and $s^r$, respectively. That is, $\\langle Q_r(p), x^{\\ot r}(x^{\\ot r})^T\\rangle = s^{r-d}p$ and $\n\\langle M_r, x^{\\ot r}(x^{\\ot r})^T\\rangle= s^r$ hold.\nThen, the  spectral bound  $\\spec_r(p)$ at the $ r $-th level is defined as the smallest generalized eigenvalue of the matrix $ Q_r(p) $ with respect to $ M_r $. We gather some equivalent formulations for $\\spec_r(p)$. \n\n\\begin{definition}\n\tThe spectral bound $\\spec_r(p)$  is defined by any of the following equivalent programs:\n\t\\begin{align}\n\t\t\\spec_r(p)& =\\max_{\\nu\\in\\R} \\{ \\nu: Q_r(p)-\\nu M_r\\succeq 0\\},\\label{eqnu1} \\\\\n\t\t& = \\min_{v\\in S^r(\\R^n) } \\{v^T Q_r(p)v: v^T M_rv=1\\}, \\label{eqnu2}\\\\\n\t\t& = \\min_{X } \\{\\langle Q_r(p), X\\rangle: X \\  r\\text{-doubly symmetric},\\ X\\succeq 0,\\ \\langle M_r,X\\rangle =1\\}.\\label{eqnu3}\n\t\\end{align}\n\\end{definition}\n\n\nWe start by picking a  quartic form  in four variables that is nonnegative but not a sum of squares.\nA concrete example is given by the \\emph{Choi-Lam form} $ p_{CL} $, \n\\begin{align}\\label{eqpCL}\n\tp_{CL}(x) = x_1^2x_2^2+x_1^2x_3^2+ x_2^2x_3^2 + x_4^4 -4 x_1x_2x_3x_4.\n\\end{align}\n\n\nThe Choi-Lam form $ p_{CL}$ is famously known to be a nonnegative polynomial, which is not a sum of squares (see \\cite{Choi_Lam_1977}). Hence, {\\em no} matrix representation of $ p_{CL} $ is positive semidefinite.\nThis property is stronger than necessary. As it will turn out, our strategy for proving the lower bound $\\Omega(1/r^2)$ for  the spectral hierarchy works with any nonnegative quartic form whose  maximally symmetric representation is not positive semidefinite. \n\nWe use one small trick, which is to choose an unnatural representation of   $ p_{CL} $. While $ p_{CL} $ is intrinsically a polynomial in four variables $x=(x_1,x_2,x_3,x_4)$, one can artificially view it as a polynomial $p$ in five variables $ \\underline{x} := (x_1,x_2,x_3,x_4,u) $, where $u$ is an additional variable, by setting $p(\\underline x)=p_{CL}(x)$.\nIf, as a 4-variate polynomial,  $ p_{CL} $ is represented by a matrix $ Q^{(4)} \\in \\R^{[4]^2\\times [4]^2} $, then, as a 5-variate polynomial in   $ \\underline{x} = (x,u) $, it is represented by the matrix \n\\begin{align}\\label{eqmatQ}\n\t\\begin{pmatrix}\n\t\tQ^{(4)} & 0\\\\\n\t\t0^{T} & 0\n\t\\end{pmatrix} \\in  \\R^{[5]^2\\times [5]^2}.\n\\end{align}\nNote that the same holds   for the maximally symmetric representations of $p_{CL}$  in four variables and of $p $ as a polynomial in five variables: \n\\begin{align}\\label{eqmatMQ}\n\t\\maxsym(p)  = \\begin{pmatrix}\n\t\t\\maxsym(p_{CL}) & 0\\\\\n\t\t0^{T} & 0\n\t\\end{pmatrix}.\n\\end{align}\nThe reason is again that $ p $ is only supported on monomials, which do not depend on the fifth variable $ u $. \n","units":[{"raw_tex":"Throughout we consider the polynomial optimization problem, asking to minimize a polynomial $p$ over the unit sphere $\\oS^{n-1}$, i.e., to compute\n\\begin{align}\\label{eqpmin}\np_{\\min}=\\min_{x\\in \\oS^{n-1}}p(x)=\\min\\{p(x): x\\in \\R^n, \\|x\\|_2=1\\}.\n\\end{align}\nTherein, $p\\in \\R[x]=\\R[x_1,\\ldots,x_n]$ is an $n$-variate polynomial of degree $2d$. In other words,\n","start":7810,"end":8170},{"raw_tex":"We begin with some notation that is used throughout the paper. $\\N=\\{0,1,2,\\ldots\\}$ is the set of nonnegative integers. For an integer $d\\in \\N$, $\\N^n_d$ denotes the set of sequences $\\alpha\\in \\N^n$ with $|\\alpha|:=\\alpha_1+\\ldots+\\alpha_n=d$. We let $\\MFS_d$ denote the set of permutations of the $d$-element set $[d]=\\{1,\\ldots,d\\}$. We let $e_1,\\ldots,e_n$ denote the standard unit vectors in $\\C^n$, where all entries of $e_i$ are 0 except 1 at the $i$th coordinate. For a vector $u\\in \\C^n$,  $\\|u \\|_2=\\sqrt {u^*u}$ denote its Euclidean norm. The unit sphere in $\\R^n$ is  denoted $\\oS^{n-1}=\\{x\\in\\R^n: \\|x\\|_2=\\sum_{i=1}^nx_i^2=1\\}$.  We let $ \\conv(S) $ denote   the convex  hull of a set $ S $ and  $ \\cone(S) $ is its conic hull.  \n\nFor $\\K=\\R$ or $\\C$, we use the trace inner product on the space $\\K^{n\\times n}$ of matrices:  $\\langle A,B\\rangle= \\Tr(A^*B)$ for $A,B\\in \\K^{n\\times n}$. Here, $A^*= \\bar A^T$ denotes the complex conjugate transpose of $A$; $A$ is Hermitian if $A^*=A$. For a Hermitian matrix $A$, $\\lambda_{\\max}(A)$ and $\\lambda_{\\min}(A)$ denote, respectively, its largest and smallest eigenvalues.  The matrix $A$ is positive semidefinite, written as $A\\succeq 0$, if $\\lambda_{\\min}(A)\\ge 0$. For Hermitian matrices $ A, B $, we write $ A\\preceq B $ if $ B-A $ is positive semidefinite. \\tcolblue{Throughout, $I_n$ denotes the $n\\times n$ identity matrix.}\n","start":24256,"end":25651},{"raw_tex":"Let $\\K=\\R$ or $\\C$ be the real or complex field. For integers $r,n\\ge 1$, $(\\K^n)^{\\ot r}$ denotes the space of $r$-tensors on $\\K^n$, with elements $v=(v_{\\ui})_{\\ui \\in [n]^r}$. Here, we use the notation $\\ui=(i_1,\\ldots,i_r)\\in [n]^r$ (or simply $i_1\\ldots i_r$) for an $r$-sequence in $[n]^r$. Given two integers $1\\le d\\le r$, it may also be convenient to denote  $r$-sequences as\nthe concatenation $\\uj \\uk $ of a $d$-sequence $\\uj\\in [n]^d$ and an $(r-d)$-sequence $\\uk\\in [n]^{r-d}$.\n\\tcolblue{For  vectors $u_i\\in \\K^{n_i}$, $i=1,2$, their tensor product is the vector $u_1\\ot u_2\\in\\K^{n_1}\\ot \\K^{n_2}$ with entries\n$(u_1\\ot u_2)_{i_1i_2}=(u_ 1)_{i_1}(u_2)_{i_2}$ for $(i_1,i_2)\\in[n_1]\\times [n_2]$. For $u\\in\\K^n$, $u^{\\ot r} = u\\ot \\cdots \\ot u$ (taking the tensor product of $r$ copies).}\n\n We now recall several notions of symmetry for tensors. \\tcolblue{In Appendix \\ref{appendix-example} we illustrate some   of these notions on examples.}\n \nPermutations $\\sigma\\in \\MFS_r$ act on $[n]^r$ and thus on $r$-tensors, in the following way.\nFor $\\ui\\in [n]^r$, set $\\ui^\\sigma=(i_{\\sigma(1)},\\ldots, i_{\\sigma(r)})$ and, for  $v\\in (\\K^n)^{\\ot r}$, define $v^\\sigma$ as the $r$-tensor with entries $(v^\\sigma)_{\\ui}= v_{\\ui^\\sigma}$ for $\\ui\\in [n]^r$.\nThen, $v$ is said to be a {\\em symmetric tensor} if $v^\\sigma=v$ for all $\\sigma\\in \\MFS_r$, and $S^r(\\K^n)$ denotes the space of symmetric $r$-tensors on $\\K^n$. We let $\\Pi_r$ denote the projection from the space $(\\K^n)^{\\ot r}$ to the symmetric subspace $S^r(\\K^n)$. Concretely, we have \n\\begin{align}\\label{eqPir}\n\\Pi_r=\\frac{1}{r!}\\sum_{\\sigma\\in \\MFS_r} P_\\sigma,\n\\end{align}\nwhere $P_\\sigma$ is the square matrix indexed by $[n]^r\\times [n]^r$, with entries \n$(P_\\sigma)_{\\ui, \\uj}=1$ if $\\uj=\\ui^\\sigma$ and 0 otherwise, for $\\ui,\\uj\\in [n]^r$.\n\nConsider a  matrix $M$ indexed by the set $[n]^r\\times [n]^r$, so that $M$ represents an endomorphism from $(\\K^n)^{\\ot r}$ to $(\\K^n)^{\\ot r}$. Then, any permutation $\\sigma\\in \\MFS_r$  induces naturally a permutation of  the columns (and rows) of $M$. \n\n\\begin{definition}[Doubly symmetric matrix]\\label{defdoublesymmetric}\nA matrix $M\\in \\K^{[n]^r\\times [n]^r}$  is called  {\\em $r$-column symmetric} (resp., {\\em $r$-row symmetric}) if $M\\Pi_r =M$ (resp., $\\Pi_rM=M$). If $M$ is complex Hermitian or real symmetric, then $M$ is $r$-column symmetric if and only if it is $r$-row symmetric, or equivalently if $\\Pi_rM\\Pi_r=M$, in which case  $M$ is called {\\em $ r $-doubly symmetric} (or just {\\em doubly symmetric} if $r$ is clear from the context).\n\\end{definition}\n\n\nHermitian doubly symmetric matrices are also known as {\\em Bose symmetric} matrices in the quantum information literature. \\MoL{The following stronger symmetry property is sometimes needed (it is considered, e.g.,   in \\cite{Doherty_Wehner_2013}).}\n\n\\begin{definition}[Maximally symmetric matrix]\\label{defmaximallysymmetric}\nA  matrix $M\\in \\K^{[n]^r\\times [n]^r}$ can be   viewed as  a $2r$-tensor $\\vvec(M)\\in (\\K^n)^{\\ot 2r}$, with entries  $\\vvec(M)_{\\ui\\uj} = M_{\\ui,\\uj}$ for $\\ui,\\uj\\in [n]^r$.\nThen, $M$ is said to be {\\em maximally symmetric} when $\\vvec(M)$ is a symmetric $2r$-tensor.\n\\end{definition}\n\n\\tcolred{Note that a Hermitian matrix that is maximally symmetric is in fact real-valued (see also Lemma \\ref{lem:ppt-sym-is-real-rank-2} below).} Let $\\PiMS$ denote the projection from the space $\\K^{[n]^r\\times [n]^r}$ onto $\\MaxSym_r(\\K^n)$, the  space of maximally symmetric matrices. Concretely, the projection $\\PiMS(M)$ is obtained by first constructing the symmetric $2r$-tensor $\\Pi_{2r}(\\vvec(M))$ and then reshaping it as a square matrix indexed by $[n]^r$ (by splitting the $2r$ indices into two classes of size $r$, which can be done   arbitrarily  as the tensor is $2r$-symmetric).\n","start":27264,"end":31070},{"raw_tex":"Throughout, let $ R := \\R[x] = \\R[x_1,\\ldots,x_n] $ denote  the polynomial ring in variables $ x=(x_1,\\ldots,x_n) $. For an integer $d\\in\\N$,  $ R_d $ denotes its $ d $-th graded component, which consists of the homogeneous polynomials (aka forms) with degree $d$, and $R_{\\le d}=R_0\\oplus R_1\\oplus\\ldots\\oplus R_d$ consists of the polynomials with degree at most $d$. \nWe let $[x]_d= (x^\\alpha)_{\\alpha\\in \\N^n_d}$ denote the vector of all monomials with degree $d$.\nThroughout we use the notation  $$s=x_1^2+\\ldots +x_n^2.$$\n","start":34823,"end":35351},{"raw_tex":"A polynomial $f\\in R$ is a {\\em sum of squares} (abbreviated as sos) if $ f = p_1^2  + \\ldots + p_N^2 $ for some $ N\\in \\N $ and $ p_1,\\ldots,p_N\\in R$. Then, all $p_i$ lie in $R_d$ (resp., $R_{\\le d}$) if $f\\in R_{2d}$ (resp., $f\\in R_{\\le 2d}$). The following characterization is well-known \\tcolblue{(see \\cite{Choi-Lam-Reznick-1995,Powers-Wormann-1998})}.\n\n\\begin{lemma}\\label{lem:sos-equiv-gram-matrix}\nA polynomial $f$ is a sum of squares if and only if it admits a positive semidefinite representing matrix (then, often called a {\\em Gram matrix} of $f$).\n\\end{lemma}\n","start":35616,"end":36191},{"raw_tex":"\\begin{definition} \\label{defGramtensor}\n\tLet $f\\in R_{2d}$. A symmetric matrix $G$ indexed by $[n]^d$ is called a {\\em tensor-indexed representing matrix} of  $ f $ if the following polynomial identity holds:\n\t\\begin{align*}\n\t\tf = (x^{\\otimes d})^{T}G x^{\\otimes d} = \\langle G, (x x^{T})^{\\otimes d} \\rangle. \n\t\\end{align*}\n\\end{definition}\n\t\n\tThen, any polynomial  $f\\in R_{2d}$ admits  a (unique) tensor-indexed representing matrix  that is maximally symmetric, denoted by $\\MaxSym(f)$. Namely, we have $\\MaxSym(f)=\\PiMS(G)$, where $G$ is any tensor-indexed representing matrix of $f$.\nMoreover,\nin analogy with \\Cref{lem:sos-equiv-gram-matrix}, a polynomial $ f $ is a sum of squares if and only if it has a tensor-indexed positive semidefinite representing matrix. \n","start":39095,"end":39867},{"raw_tex":"\\MoL{\\johnstonlovitz{} \\cite{Johnston_Lovitz_Vijayaraghavan_2023} proposed recently a hierarchy of spectral bounds $\\spec_r(p)$ for the minimum value $p_{\\min}$ of a homogeneous polynomial $p$ over the unit sphere,\\footnote{ Their hierarchy  applies more generally to optimization of  a multivariate tensor over a spherical Segre-Veronese variety. We only focus here on their hierarchy for polynomials on the sphere.} \nsee Section \\ref{secJLV-intro} below for their definition and relationship to the  moment-sos bounds.  Throughout, let $p\\in R_{2d}$, $s=x_1^2+\\cdots +x_n^2$, and let \n\\begin{align}\\label{eqQpM}\n\tQ(p) = \\maxsym(p), \\quad  M = \\maxsym(s^d) \n\\end{align}\ndenote  the maximally symmetric  matrices representing the polynomials $ p $ and $s^d$, respectively (as in Definition \\ref{defGramtensor}). They show the following convergence rate.\n}\n","start":69899,"end":70755},{"raw_tex":"We introduce the spectral hierarchy of \\johnstonlovitz{} \\cite{Johnston_Lovitz_Vijayaraghavan_2023}. \nLet $p\\in R_{2d}$ and recall the matrices $Q(p)$   and $M$ from (\\ref{eqQpM}). For an integer $ r\\ge d $, define the matrices \n\\begin{align}\\label{eqQrpM}\n\tQ_r(p) = \\Pi_{r}(Q(p) \\otimes I_n^{\\otimes (r-d)})\\Pi_{r}, \\\\\n\tM_r= \\Pi_{r}(M \\otimes I_n^{\\otimes (r-d)})\\Pi_{r}. \n\\end{align}\nSo,  $ Q_r(p) $ and $M_r$ are   $ r $-doubly symmetric matrices    representing  $ s^{r-d}p $ and $s^r$, respectively. That is, $\\langle Q_r(p), x^{\\ot r}(x^{\\ot r})^T\\rangle = s^{r-d}p$ and $\n\\langle M_r, x^{\\ot r}(x^{\\ot r})^T\\rangle= s^r$ hold.\nThen, the  spectral bound  $\\spec_r(p)$ at the $ r $-th level is defined as the smallest generalized eigenvalue of the matrix $ Q_r(p) $ with respect to $ M_r $. We gather some equivalent formulations for $\\spec_r(p)$. \n\n\\begin{definition}\n\tThe spectral bound $\\spec_r(p)$  is defined by any of the following equivalent programs:\n\t\\begin{align}\n\t\t\\spec_r(p)& =\\max_{\\nu\\in\\R} \\{ \\nu: Q_r(p)-\\nu M_r\\succeq 0\\},\\label{eqnu1} \\\\\n\t\t& = \\min_{v\\in S^r(\\R^n) } \\{v^T Q_r(p)v: v^T M_rv=1\\}, \\label{eqnu2}\\\\\n\t\t& = \\min_{X } \\{\\langle Q_r(p), X\\rangle: X \\  r\\text{-doubly symmetric},\\ X\\succeq 0,\\ \\langle M_r,X\\rangle =1\\}.\\label{eqnu3}\n\t\\end{align}\n\\end{definition}\n","start":72976,"end":74271},{"raw_tex":"We start by picking a  quartic form  in four variables that is nonnegative but not a sum of squares.\nA concrete example is given by the \\emph{Choi-Lam form} $ p_{CL} $, \n\\begin{align}\\label{eqpCL}\n\tp_{CL}(x) = x_1^2x_2^2+x_1^2x_3^2+ x_2^2x_3^2 + x_4^4 -4 x_1x_2x_3x_4.\n\\end{align}\n","start":85861,"end":86142},{"raw_tex":"The Choi-Lam form $ p_{CL}$ is famously known to be a nonnegative polynomial, which is not a sum of squares (see \\cite{Choi_Lam_1977}). Hence, {\\em no} matrix representation of $ p_{CL} $ is positive semidefinite.\nThis property is stronger than necessary. As it will turn out, our strategy for proving the lower bound $\\Omega(1/r^2)$ for  the spectral hierarchy works with any nonnegative quartic form whose  maximally symmetric representation is not positive semidefinite. \n\nWe use one small trick, which is to choose an unnatural representation of   $ p_{CL} $. While $ p_{CL} $ is intrinsically a polynomial in four variables $x=(x_1,x_2,x_3,x_4)$, one can artificially view it as a polynomial $p$ in five variables $ \\underline{x} := (x_1,x_2,x_3,x_4,u) $, where $u$ is an additional variable, by setting $p(\\underline x)=p_{CL}(x)$.\nIf, as a 4-variate polynomial,  $ p_{CL} $ is represented by a matrix $ Q^{(4)} \\in \\R^{[4]^2\\times [4]^2} $, then, as a 5-variate polynomial in   $ \\underline{x} = (x,u) $, it is represented by the matrix \n\\begin{align}\\label{eqmatQ}\n\t\\begin{pmatrix}\n\t\tQ^{(4)} & 0\\\\\n\t\t0^{T} & 0\n\t\\end{pmatrix} \\in  \\R^{[5]^2\\times [5]^2}.\n\\end{align}\nNote that the same holds   for the maximally symmetric representations of $p_{CL}$  in four variables and of $p $ as a polynomial in five variables: \n\\begin{align}\\label{eqmatMQ}\n\t\\maxsym(p)  = \\begin{pmatrix}\n\t\t\\maxsym(p_{CL}) & 0\\\\\n\t\t0^{T} & 0\n\t\\end{pmatrix}.\n\\end{align}\nThe reason is again that $ p $ is only supported on monomials, which do not depend on the fifth variable $ u $. \n","start":86426,"end":87987}]},"statement":{"raw_tex":"\\begin{theorem}\\label{thm:lower-bound-spectral}\n\tLet $\\spec_r(p)$ denote the spectral bound of order $r$ for the Choi-Lam form $p \\in \\R[x_1,x_2,x_3,x_4,u]$. Then, we have\t$p_{\\min}=0$ and $\\spec_r(p)=\\Omega(1/r^2)$.\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\\label{thm:lower-bound-spectral}\n\tLet $\\spec_r(p)$ denote the spectral bound of order $r$ for the Choi-Lam form $p \\in \\R[x_1,x_2,x_3,x_4,u]$. Then, we have\t$p_{\\min}=0$ and $\\spec_r(p)=\\Omega(1/r^2)$.\n\\end{theorem}","start":87988,"end":88218}]},"proof":{"raw_tex":"\\begin{proof}\n\tLet the variables be ordered as $ \\underline{x} = (x,u)$, where $ x = (x_1,\\ldots,x_4) $ and  $ u $ is the fifth variable. Let $r\\ge 2$. Set $ s = x_1^2  + x_2^2+x_3^2 + x_4^2 + u^2 $ and \n\t\\begin{align}\\label{eqmats}\n\t\\begin{split}\n\t & Q :=   \\maxsym(p),\\quad  M := \\maxsym(s^2), \\\\\n\t &  Q_r := \\Pi_r (Q\\otimes I_5^{\\otimes (r-2)}) \\Pi_r,\\quad  M_r :=  \\Pi_r (M\\otimes I_5^{\\otimes (r-2)}) \\Pi_r.\n\t\\end{split}\n\t\\end{align} \n\tSo, $Q$ has the block-form as in (\\ref{eqmatQ}), (\\ref{eqmatMQ}), where $Q^{(4)} :=  \\MaxSym(p_{CL}) \\in \\R^{[4]^2 \\times [4]^2} $ and the remaining blocks are filled by zeros. As in (\\ref{eqPir}),  $\\Pi_2\\in \\R^{[5]^2\\times [5]^2}$ denotes the projection onto the symmetric subspace $S^2(\\R^5)$.\tThe space $\\R^{[4]^2}$ can be seen as a subspace of $\\R^{[5]^2}$ corresponding to the first four variable indices. Let us write $ \\Pi_2^{(4)} \\in \\R^{[4]^2 \\times [4]^2} $ for the restriction of $ \\Pi_2$ to the subspace $ \\R^{[4]^2} $. \n\n\tRecall  $\\spec_r=\\spec_r(p)$ for short. By the definition, we have $\\spec_r\\le p_{\\min}=0$ and \n\t\\begin{align*}\n\t\tQ_r + |\\spec_r| M_r = \\Pi_r ((Q+ |\\spec_r|M) \\otimes I_5^{\\otimes (r-2)}) \\Pi_r \\succeq 0.\n\t\\end{align*}\n\tBy using  $ M\\preceq \\lambda_{\\max}(M) I_5^{\\otimes 2} $, we obtain that \n\t\\begin{align*}\n\t\tG_r :=  \\Pi_r ((Q+ \\lambda_{\\max}(M)|\\spec_r|I_5^{\\otimes 2}) \\otimes I_5^{\\otimes (r-2)}) \\Pi_r \\succeq 0.\n\t\\end{align*}\n\tIn particular, every principal submatrix of $ G_r $ is positive semidefinite. We will now examine \\tcolblue{in detail} the following principal submatrix \n\t\\begin{align}\\label{eqmatH}\n\t\tH:= ((G_r)_{(i_1,i_2,5,\\ldots,5), (j_1,j_2,5,\\ldots,5)})_{i_1,i_2,j_1,j_2 \\in \\{1,\\ldots,4\\}} \\in \\R^{[4]^2\\times [4]^2}\n\t\\end{align}\n\tof $ G_r $, where the first two row indices $ i_1, i_2 $ and the first two column indices $ j_1,j_2 $ each range from $ 1 $ to $ 4 $, while the other $ r-2 $ row and column indices are set equal to $ 5 $.\n\t\n\n\tRecall the notation $ \\underline{i}^{\\sigma} := (i_{\\sigma(1)},\\ldots,i_{\\sigma(r)}) $ for any $\\sigma\\in \\MFS_r$. For $\\ui,\\uj\\in [5]^r$, using (\\ref{eqPir}), we may expand \n\t\\begin{align*}\n\t\t(G_r)_{\\underline{i},\\underline{j}} = \\left(\\frac{1}{r!}\\right)^2 \\sum_{\\sigma \\in \\mathfrak{S}_{r}}\\sum_{\\tau \\in \\mathfrak{S}_{r}}  (Q \\otimes I_5^{\\otimes (r-2)} + \\lambda_{\\max}(M)|\\spec_r|I_5^{\\otimes r})_{\\underline{i}^{\\sigma}, \\underline{j}^{\\tau}}.\n\t\\end{align*}\n\tIn order to compute the submatrix $ H $, we set the last $ r-2 $ indices of both $ \\underline{i} $ and $ \\underline{j} $ equal to $ 5 $ and the other two indices not equal to $ 5 $. Hence, on the right hand side, we have to count how many permutations ``survive'' when we do this, in the sense of giving a nonzero contribution to $ H $. \n\tWe may write \n\t\\begin{align}\\label{eqmatH2}\n\t H = A + \\lambda_{\\max}(M)|\\spec_r|B,\n\t \\end{align}\n\t  where $ A $ is the contribution arising from $ Q \\otimes I_5^{\\otimes (r-2)} $ and $ B $ is the contribution from $ I_5^{\\otimes r}  $. Since $ I_5^{\\otimes r}  $ is diagonal, the only nonzero entries of $ B $ are at indices of the form $ (i_1i_2,i_1i_2) $ and $ (i_1i_2,i_2i_1) $. \n\tFor later use, notice that $ (\\Pi_{2})_{i_1i_2,i_1i_2} = \\frac{1}{2} $ if $ i_1\\ne i_2 \\in [4]$, and $(\\Pi_{2})_{i_1i_1,i_1i_1} = 1 $ for $i_1\\in [4]$. \n\tWe now calculate the matrix $B$: for $i_1,i_2\\in [4]$, \n\t\\begin{align*}\n\t\tB_{i_1i_2,i_1i_2} &= B_{i_1i_2,i_2i_1} = \\frac{1}{(r!)^2} \\sum_{\\sigma \\in \\mathfrak{S}_{r}}\\sum_{\\tau \\in \\mathfrak{S}_{r}}  (I_5^{\\otimes r})_{(i_{1},i_{2},5,\\ldots,5)^{\\sigma}, (i_{1},i_{2},5,\\ldots,5)^{\\tau}}\\\\\n\t\t&= 2(\\Pi_{2})_{i_1i_2,i_1i_2} \\frac{(r-2)!}{(r!)^2} \\sum_{\\sigma \\in \\mathfrak{S}_{r}}  (I_5^{\\otimes r})_{(i_{1},i_{2},5,\\ldots,5)^{\\sigma}, (i_{1},i_{2},5,\\ldots,5)^{\\sigma}}\\\\\n\t\t&= 2(\\Pi_{2})_{i_1i_2,i_1i_2}\\frac{(r-2)!}{r!} =  \\binom{r}{2}^{-1} (\\Pi_2)_{i_1i_2,i_1i_2}. \n\t\\end{align*}\t\n\tHere, in the second line, we   used the fact that $ I_n^{\\otimes r} $ is diagonal and that the row and column indices are equal if and only if $ (i_{\\sigma(1)},i_{\\sigma(2)}) = (i_{\\tau(1)},i_{\\tau(2)}) $. \n\tGiven any permutation $ \\sigma \\in \\mathfrak{S}_r$, there are $ (r-2)! $ permutations $ \\tau \\in \\mathfrak{S}_r $ with this property if $ i_1\\ne i_2 $, and $ 2(r-2)! $ permutations $ \\tau $ with this property if $ i_1 = i_2 $. Hence, there are $ 2(r-2)!(\\Pi_{2})_{i_1i_2,i_1i_2} $ permutations $ \\tau $ with this property for any given $ \\sigma $. \n\tIn the third line, we used the fact that all diagonal entries of $ I_n^{\\otimes r} $ are equal to $ 1 $, so the sum evaluates to $ r! $.  \n\tSummarized, we conclude that $ B = \\binom{r}{2}^{-1} \\Pi_2^{(4)} $. \n\tNow, let us calculate $ A $: for $i_1,i_2,j_1,j_2\\in [4]$,\n\t\\begin{align*}\n\t\tA_{i_1i_2,j_1j_2} = \\frac{1}{(r!)^2} \\sum_{\\sigma \\in \\mathfrak{S}_{r}}\\sum_{\\tau \\in \\mathfrak{S}_{r}}  (Q\\otimes I_5^{\\otimes r-2})_{(i_{1},i_{2},5,\\ldots,5)^{\\sigma}, (j_{1},j_{2},5,\\ldots,5)^{\\tau}}.\n\t\\end{align*}\n\tRecall that $ Q_{i,5} = 0 $ if $ i\\in \\{1,\\ldots,4\\} $. Therefore, the only nonzero contributions come from permutations $ \\sigma $ and $ \\tau $, which keep all 5-indices `to the right', i.e., at the last $r-2$ positions. There are $ 2(r-2)! $ permutations $ \\sigma $ and $ 2(r-2)! $ permutations $ \\tau $ with this property. They may be written as $ \\sigma = \\sigma_2\\circ \\sigma_{r-2} $ and $ \\tau =  \\tau_2\\circ \\tau_{r-2}$, respectively, with $ \\sigma_2, \\tau_2 \\in \\mathfrak{S}_2 $ and $ \\sigma_{r-2}, \\tau_{r-2} \\in \\mathfrak{S}_{r-2} $.\n\tSo, we obtain  \n\t\\begin{align*}\n\t\tA_{i_1i_2,j_1j_2} &= \\frac{((r-2)!)^2}{(r!)^2} \\sum_{\\sigma_2, \\tau_2 \\in \\mathfrak{S}_{2}} (Q\\otimes I_n^{\\otimes r-2})_{(i_{\\sigma_2(1)},i_{\\sigma_2(2)},5,\\ldots,5), (j_{\\tau_2(1)},j_{\\tau_2(2)},5,\\ldots,5)}.\\\\\n\t\t&= \\left(\\frac{(r-2)!}{r!}\\right)^2 \\sum_{\\sigma_2, \\tau_2 \\in \\mathfrak{S}_{2}} Q_{i_{\\sigma_2(1)}i_{\\sigma_2(2)}, j_{\\tau_2(1)}j_{\\tau_2(2)}}.\\\\\n\t\t&= \\binom{r}{2}^{-2} Q_{i_1i_2,j_1j_2}.\n\t\\end{align*}\n\tIn the first line, we used the fact that the $ (r-2)!^2 $ permutations of the 5-indices all yield the same indices. In the second line, we used that $ (I_n^{\\otimes r-2})_{5\\ldots5,5\\ldots5} = 1 $. In the third line, we used that $ Q $ is maximally symmetric, hence in particular doubly symmetric. Thus, the $ 2^2 = 4 $ permutations of the non-5 indices all yield the same entry. Summarizing, we have shown  \n\t$$ A = \\binom{r}{2}^{-2}Q^{(4)},\\quad  B = \\binom{r}{2}^{-1}\\Pi_{2}^{(4)}.\n\t$$\n\t We conclude that the   matrix $ H $ from (\\ref{eqmatH})-(\\ref{eqmatH2}) can be expressed as\n\t\\begin{align*}\n\t\t\\binom{r}{2}^{2}  H = Q^{(4)} + \\lambda_{\\max}(M)|\\spec_r|\\binom{r}{2}\\Pi_{2}^{(4)}. \n\t\\end{align*}\n\tNow, we use that $H\\succeq 0$ and $\\begin{pmatrix} \\Pi_2^{(4)} & 0 \\\\ 0^{T} & 0 \\end{pmatrix} \\preceq I_5^{\\otimes 2} \\preceq \\frac{1}{\\lambda_{\\min}(M)} M $ to obtain  \n\t\\begin{align*}\n\t\t0\\preceq \\binom{r}{2}^{2} \\begin{pmatrix} H & 0 \\\\ 0^{T} & 0 \\end{pmatrix}  \\preceq Q + |\\spec_r|\\frac{\\lambda_{\\max}(M)}{\\lambda_{\\min}(M)} \\binom{r}{2} M. \n\t\\end{align*}\n\tBy definition, $ |\\spec_2| $ is the smallest value such that \n\t\\begin{align*}\n\t\tQ + |\\spec_2| M \\succeq 0. \n\t\\end{align*}\n\tHence, it follows that \n\t\\begin{align*}\n\t\t |\\spec_r|\\frac{\\lambda_{\\max}(M)}{\\lambda_{\\min}(M)} \\binom{r}{2} \\ge |\\spec_2|, \\:\\: \\text{ and thus }\\:\\: \t|\\spec_r| \\ge \\frac{\\lambda_{\\min}(M)}{\\lambda_{\\max}(M)} |\\spec_2|\\binom{r}{2}^{-1} = \\Omega\\left(\\frac{1}{r^2}\\right),\t\t \n\t\\end{align*}\n\\tcolblue{using the fact that $M\\succ 0$ \\cite{Johnston_Lovitz_Vijayaraghavan_2023} and $|\\spec_2|>0$ as $p$ is not sos.} \n\\end{proof}","units":[{"raw_tex":"\\begin{proof}\n\tLet the variables be ordered as $ \\underline{x} = (x,u)$, where $ x = (x_1,\\ldots,x_4) $ and  $ u $ is the fifth variable. Let $r\\ge 2$. Set $ s = x_1^2  + x_2^2+x_3^2 + x_4^2 + u^2 $ and \n\t\\begin{align}\\label{eqmats}\n\t\\begin{split}\n\t & Q :=   \\maxsym(p),\\quad  M := \\maxsym(s^2), \\\\\n\t &  Q_r := \\Pi_r (Q\\otimes I_5^{\\otimes (r-2)}) \\Pi_r,\\quad  M_r :=  \\Pi_r (M\\otimes I_5^{\\otimes (r-2)}) \\Pi_r.\n\t\\end{split}\n\t\\end{align} \n\tSo, $Q$ has the block-form as in (\\ref{eqmatQ}), (\\ref{eqmatMQ}), where $Q^{(4)} :=  \\MaxSym(p_{CL}) \\in \\R^{[4]^2 \\times [4]^2} $ and the remaining blocks are filled by zeros. As in (\\ref{eqPir}),  $\\Pi_2\\in \\R^{[5]^2\\times [5]^2}$ denotes the projection onto the symmetric subspace $S^2(\\R^5)$.\tThe space $\\R^{[4]^2}$ can be seen as a subspace of $\\R^{[5]^2}$ corresponding to the first four variable indices. Let us write $ \\Pi_2^{(4)} \\in \\R^{[4]^2 \\times [4]^2} $ for the restriction of $ \\Pi_2$ to the subspace $ \\R^{[4]^2} $. \n\n\tRecall  $\\spec_r=\\spec_r(p)$ for short. By the definition, we have $\\spec_r\\le p_{\\min}=0$ and \n\t\\begin{align*}\n\t\tQ_r + |\\spec_r| M_r = \\Pi_r ((Q+ |\\spec_r|M) \\otimes I_5^{\\otimes (r-2)}) \\Pi_r \\succeq 0.\n\t\\end{align*}\n\tBy using  $ M\\preceq \\lambda_{\\max}(M) I_5^{\\otimes 2} $, we obtain that \n\t\\begin{align*}\n\t\tG_r :=  \\Pi_r ((Q+ \\lambda_{\\max}(M)|\\spec_r|I_5^{\\otimes 2}) \\otimes I_5^{\\otimes (r-2)}) \\Pi_r \\succeq 0.\n\t\\end{align*}\n\tIn particular, every principal submatrix of $ G_r $ is positive semidefinite. We will now examine \\tcolblue{in detail} the following principal submatrix \n\t\\begin{align}\\label{eqmatH}\n\t\tH:= ((G_r)_{(i_1,i_2,5,\\ldots,5), (j_1,j_2,5,\\ldots,5)})_{i_1,i_2,j_1,j_2 \\in \\{1,\\ldots,4\\}} \\in \\R^{[4]^2\\times [4]^2}\n\t\\end{align}\n\tof $ G_r $, where the first two row indices $ i_1, i_2 $ and the first two column indices $ j_1,j_2 $ each range from $ 1 $ to $ 4 $, while the other $ r-2 $ row and column indices are set equal to $ 5 $.\n\t\n\n\tRecall the notation $ \\underline{i}^{\\sigma} := (i_{\\sigma(1)},\\ldots,i_{\\sigma(r)}) $ for any $\\sigma\\in \\MFS_r$. For $\\ui,\\uj\\in [5]^r$, using (\\ref{eqPir}), we may expand \n\t\\begin{align*}\n\t\t(G_r)_{\\underline{i},\\underline{j}} = \\left(\\frac{1}{r!}\\right)^2 \\sum_{\\sigma \\in \\mathfrak{S}_{r}}\\sum_{\\tau \\in \\mathfrak{S}_{r}}  (Q \\otimes I_5^{\\otimes (r-2)} + \\lambda_{\\max}(M)|\\spec_r|I_5^{\\otimes r})_{\\underline{i}^{\\sigma}, \\underline{j}^{\\tau}}.\n\t\\end{align*}\n\tIn order to compute the submatrix $ H $, we set the last $ r-2 $ indices of both $ \\underline{i} $ and $ \\underline{j} $ equal to $ 5 $ and the other two indices not equal to $ 5 $. Hence, on the right hand side, we have to count how many permutations ``survive'' when we do this, in the sense of giving a nonzero contribution to $ H $. \n\tWe may write \n\t\\begin{align}\\label{eqmatH2}\n\t H = A + \\lambda_{\\max}(M)|\\spec_r|B,\n\t \\end{align}\n\t  where $ A $ is the contribution arising from $ Q \\otimes I_5^{\\otimes (r-2)} $ and $ B $ is the contribution from $ I_5^{\\otimes r}  $. Since $ I_5^{\\otimes r}  $ is diagonal, the only nonzero entries of $ B $ are at indices of the form $ (i_1i_2,i_1i_2) $ and $ (i_1i_2,i_2i_1) $. \n\tFor later use, notice that $ (\\Pi_{2})_{i_1i_2,i_1i_2} = \\frac{1}{2} $ if $ i_1\\ne i_2 \\in [4]$, and $(\\Pi_{2})_{i_1i_1,i_1i_1} = 1 $ for $i_1\\in [4]$. \n\tWe now calculate the matrix $B$: for $i_1,i_2\\in [4]$, \n\t\\begin{align*}\n\t\tB_{i_1i_2,i_1i_2} &= B_{i_1i_2,i_2i_1} = \\frac{1}{(r!)^2} \\sum_{\\sigma \\in \\mathfrak{S}_{r}}\\sum_{\\tau \\in \\mathfrak{S}_{r}}  (I_5^{\\otimes r})_{(i_{1},i_{2},5,\\ldots,5)^{\\sigma}, (i_{1},i_{2},5,\\ldots,5)^{\\tau}}\\\\\n\t\t&= 2(\\Pi_{2})_{i_1i_2,i_1i_2} \\frac{(r-2)!}{(r!)^2} \\sum_{\\sigma \\in \\mathfrak{S}_{r}}  (I_5^{\\otimes r})_{(i_{1},i_{2},5,\\ldots,5)^{\\sigma}, (i_{1},i_{2},5,\\ldots,5)^{\\sigma}}\\\\\n\t\t&= 2(\\Pi_{2})_{i_1i_2,i_1i_2}\\frac{(r-2)!}{r!} =  \\binom{r}{2}^{-1} (\\Pi_2)_{i_1i_2,i_1i_2}. \n\t\\end{align*}\t\n\tHere, in the second line, we   used the fact that $ I_n^{\\otimes r} $ is diagonal and that the row and column indices are equal if and only if $ (i_{\\sigma(1)},i_{\\sigma(2)}) = (i_{\\tau(1)},i_{\\tau(2)}) $. \n\tGiven any permutation $ \\sigma \\in \\mathfrak{S}_r$, there are $ (r-2)! $ permutations $ \\tau \\in \\mathfrak{S}_r $ with this property if $ i_1\\ne i_2 $, and $ 2(r-2)! $ permutations $ \\tau $ with this property if $ i_1 = i_2 $. Hence, there are $ 2(r-2)!(\\Pi_{2})_{i_1i_2,i_1i_2} $ permutations $ \\tau $ with this property for any given $ \\sigma $. \n\tIn the third line, we used the fact that all diagonal entries of $ I_n^{\\otimes r} $ are equal to $ 1 $, so the sum evaluates to $ r! $.  \n\tSummarized, we conclude that $ B = \\binom{r}{2}^{-1} \\Pi_2^{(4)} $. \n\tNow, let us calculate $ A $: for $i_1,i_2,j_1,j_2\\in [4]$,\n\t\\begin{align*}\n\t\tA_{i_1i_2,j_1j_2} = \\frac{1}{(r!)^2} \\sum_{\\sigma \\in \\mathfrak{S}_{r}}\\sum_{\\tau \\in \\mathfrak{S}_{r}}  (Q\\otimes I_5^{\\otimes r-2})_{(i_{1},i_{2},5,\\ldots,5)^{\\sigma}, (j_{1},j_{2},5,\\ldots,5)^{\\tau}}.\n\t\\end{align*}\n\tRecall that $ Q_{i,5} = 0 $ if $ i\\in \\{1,\\ldots,4\\} $. Therefore, the only nonzero contributions come from permutations $ \\sigma $ and $ \\tau $, which keep all 5-indices `to the right', i.e., at the last $r-2$ positions. There are $ 2(r-2)! $ permutations $ \\sigma $ and $ 2(r-2)! $ permutations $ \\tau $ with this property. They may be written as $ \\sigma = \\sigma_2\\circ \\sigma_{r-2} $ and $ \\tau =  \\tau_2\\circ \\tau_{r-2}$, respectively, with $ \\sigma_2, \\tau_2 \\in \\mathfrak{S}_2 $ and $ \\sigma_{r-2}, \\tau_{r-2} \\in \\mathfrak{S}_{r-2} $.\n\tSo, we obtain  \n\t\\begin{align*}\n\t\tA_{i_1i_2,j_1j_2} &= \\frac{((r-2)!)^2}{(r!)^2} \\sum_{\\sigma_2, \\tau_2 \\in \\mathfrak{S}_{2}} (Q\\otimes I_n^{\\otimes r-2})_{(i_{\\sigma_2(1)},i_{\\sigma_2(2)},5,\\ldots,5), (j_{\\tau_2(1)},j_{\\tau_2(2)},5,\\ldots,5)}.\\\\\n\t\t&= \\left(\\frac{(r-2)!}{r!}\\right)^2 \\sum_{\\sigma_2, \\tau_2 \\in \\mathfrak{S}_{2}} Q_{i_{\\sigma_2(1)}i_{\\sigma_2(2)}, j_{\\tau_2(1)}j_{\\tau_2(2)}}.\\\\\n\t\t&= \\binom{r}{2}^{-2} Q_{i_1i_2,j_1j_2}.\n\t\\end{align*}\n\tIn the first line, we used the fact that the $ (r-2)!^2 $ permutations of the 5-indices all yield the same indices. In the second line, we used that $ (I_n^{\\otimes r-2})_{5\\ldots5,5\\ldots5} = 1 $. In the third line, we used that $ Q $ is maximally symmetric, hence in particular doubly symmetric. Thus, the $ 2^2 = 4 $ permutations of the non-5 indices all yield the same entry. Summarizing, we have shown  \n\t$$ A = \\binom{r}{2}^{-2}Q^{(4)},\\quad  B = \\binom{r}{2}^{-1}\\Pi_{2}^{(4)}.\n\t$$\n\t We conclude that the   matrix $ H $ from (\\ref{eqmatH})-(\\ref{eqmatH2}) can be expressed as\n\t\\begin{align*}\n\t\t\\binom{r}{2}^{2}  H = Q^{(4)} + \\lambda_{\\max}(M)|\\spec_r|\\binom{r}{2}\\Pi_{2}^{(4)}. \n\t\\end{align*}\n\tNow, we use that $H\\succeq 0$ and $\\begin{pmatrix} \\Pi_2^{(4)} & 0 \\\\ 0^{T} & 0 \\end{pmatrix} \\preceq I_5^{\\otimes 2} \\preceq \\frac{1}{\\lambda_{\\min}(M)} M $ to obtain  \n\t\\begin{align*}\n\t\t0\\preceq \\binom{r}{2}^{2} \\begin{pmatrix} H & 0 \\\\ 0^{T} & 0 \\end{pmatrix}  \\preceq Q + |\\spec_r|\\frac{\\lambda_{\\max}(M)}{\\lambda_{\\min}(M)} \\binom{r}{2} M. \n\t\\end{align*}\n\tBy definition, $ |\\spec_2| $ is the smallest value such that \n\t\\begin{align*}\n\t\tQ + |\\spec_2| M \\succeq 0. \n\t\\end{align*}\n\tHence, it follows that \n\t\\begin{align*}\n\t\t |\\spec_r|\\frac{\\lambda_{\\max}(M)}{\\lambda_{\\min}(M)} \\binom{r}{2} \\ge |\\spec_2|, \\:\\: \\text{ and thus }\\:\\: \t|\\spec_r| \\ge \\frac{\\lambda_{\\min}(M)}{\\lambda_{\\max}(M)} |\\spec_2|\\binom{r}{2}^{-1} = \\Omega\\left(\\frac{1}{r^2}\\right),\t\t \n\t\\end{align*}\n\\tcolblue{using the fact that $M\\succ 0$ \\cite{Johnston_Lovitz_Vijayaraghavan_2023} and $|\\spec_2|>0$ as $p$ is not sos.} \n\\end{proof}","start":88220,"end":95758}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eqpmin"],"declared_external":false,"source_only":false,"title":"球面多项式最小化","source_units":[{"raw_tex":"Throughout we consider the polynomial optimization problem, asking to minimize a polynomial $p$ over the unit sphere $\\oS^{n-1}$, i.e., to compute\n\\begin{align}\\label{eqpmin}\np_{\\min}=\\min_{x\\in \\oS^{n-1}}p(x)=\\min\\{p(x): x\\in \\R^n, \\|x\\|_2=1\\}.\n\\end{align}\nTherein, $p\\in \\R[x]=\\R[x_1,\\ldots,x_n]$ is an $n$-variate polynomial of degree $2d$. In other words,\n","start":7810,"end":8170}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"指标与置换记号","source_units":[{"raw_tex":"We begin with some notation that is used throughout the paper. $\\N=\\{0,1,2,\\ldots\\}$ is the set of nonnegative integers. For an integer $d\\in \\N$, $\\N^n_d$ denotes the set of sequences $\\alpha\\in \\N^n$ with $|\\alpha|:=\\alpha_1+\\ldots+\\alpha_n=d$. We let $\\MFS_d$ denote the set of permutations of the $d$-element set $[d]=\\{1,\\ldots,d\\}$. We let $e_1,\\ldots,e_n$ denote the standard unit vectors in $\\C^n$, where all entries of $e_i$ are 0 except 1 at the $i$th coordinate. For a vector $u\\in \\C^n$,  $\\|u \\|_2=\\sqrt {u^*u}$ denote its Euclidean norm. The unit sphere in $\\R^n$ is  denoted $\\oS^{n-1}=\\{x\\in\\R^n: \\|x\\|_2=\\sum_{i=1}^nx_i^2=1\\}$.  We let $ \\conv(S) $ denote   the convex  hull of a set $ S $ and  $ \\cone(S) $ is its conic hull.  \n\n","start":24256,"end":25003}]},{"id":"S003","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"矩阵序与特征值","source_units":[{"raw_tex":"For $\\K=\\R$ or $\\C$, we use the trace inner product on the space $\\K^{n\\times n}$ of matrices:  $\\langle A,B\\rangle= \\Tr(A^*B)$ for $A,B\\in \\K^{n\\times n}$. Here, $A^*= \\bar A^T$ denotes the complex conjugate transpose of $A$; $A$ is Hermitian if $A^*=A$. For a Hermitian matrix $A$, $\\lambda_{\\max}(A)$ and $\\lambda_{\\min}(A)$ denote, respectively, its largest and smallest eigenvalues.  The matrix $A$ is positive semidefinite, written as $A\\succeq 0$, if $\\lambda_{\\min}(A)\\ge 0$. For Hermitian matrices $ A, B $, we write $ A\\preceq B $ if $ B-A $ is positive semidefinite. \\tcolblue{Throughout, $I_n$ denotes the $n\\times n$ identity matrix.}\n","start":25003,"end":25651}]},{"id":"S004","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"张量与序列指标","source_units":[{"raw_tex":"Let $\\K=\\R$ or $\\C$ be the real or complex field. For integers $r,n\\ge 1$, $(\\K^n)^{\\ot r}$ denotes the space of $r$-tensors on $\\K^n$, with elements $v=(v_{\\ui})_{\\ui \\in [n]^r}$. Here, we use the notation $\\ui=(i_1,\\ldots,i_r)\\in [n]^r$ (or simply $i_1\\ldots i_r$) for an $r$-sequence in $[n]^r$. Given two integers $1\\le d\\le r$, it may also be convenient to denote  $r$-sequences as\nthe concatenation $\\uj \\uk $ of a $d$-sequence $\\uj\\in [n]^d$ and an $(r-d)$-sequence $\\uk\\in [n]^{r-d}$.\n\\tcolblue{For  vectors $u_i\\in \\K^{n_i}$, $i=1,2$, their tensor product is the vector $u_1\\ot u_2\\in\\K^{n_1}\\ot \\K^{n_2}$ with entries\n$(u_1\\ot u_2)_{i_1i_2}=(u_ 1)_{i_1}(u_2)_{i_2}$ for $(i_1,i_2)\\in[n_1]\\times [n_2]$. For $u\\in\\K^n$, $u^{\\ot r} = u\\ot \\cdots \\ot u$ (taking the tensor product of $r$ copies).}\n\n","start":27264,"end":28070}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":["eqPir"],"declared_external":false,"source_only":false,"title":"对称张量投影","source_units":[{"raw_tex":" We now recall several notions of symmetry for tensors. \\tcolblue{In Appendix \\ref{appendix-example} we illustrate some   of these notions on examples.}\n \nPermutations $\\sigma\\in \\MFS_r$ act on $[n]^r$ and thus on $r$-tensors, in the following way.\nFor $\\ui\\in [n]^r$, set $\\ui^\\sigma=(i_{\\sigma(1)},\\ldots, i_{\\sigma(r)})$ and, for  $v\\in (\\K^n)^{\\ot r}$, define $v^\\sigma$ as the $r$-tensor with entries $(v^\\sigma)_{\\ui}= v_{\\ui^\\sigma}$ for $\\ui\\in [n]^r$.\nThen, $v$ is said to be a {\\em symmetric tensor} if $v^\\sigma=v$ for all $\\sigma\\in \\MFS_r$, and $S^r(\\K^n)$ denotes the space of symmetric $r$-tensors on $\\K^n$. We let $\\Pi_r$ denote the projection from the space $(\\K^n)^{\\ot r}$ to the symmetric subspace $S^r(\\K^n)$. Concretely, we have \n\\begin{align}\\label{eqPir}\n\\Pi_r=\\frac{1}{r!}\\sum_{\\sigma\\in \\MFS_r} P_\\sigma,\n\\end{align}\nwhere $P_\\sigma$ is the square matrix indexed by $[n]^r\\times [n]^r$, with entries \n$(P_\\sigma)_{\\ui, \\uj}=1$ if $\\uj=\\ui^\\sigma$ and 0 otherwise, for $\\ui,\\uj\\in [n]^r$.\n\n","start":28070,"end":29086}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["defdoublesymmetric"],"declared_external":false,"source_only":false,"title":"矩阵双对称性","source_units":[{"raw_tex":"Consider a  matrix $M$ indexed by the set $[n]^r\\times [n]^r$, so that $M$ represents an endomorphism from $(\\K^n)^{\\ot r}$ to $(\\K^n)^{\\ot r}$. Then, any permutation $\\sigma\\in \\MFS_r$  induces naturally a permutation of  the columns (and rows) of $M$. \n\n\\begin{definition}[Doubly symmetric matrix]\\label{defdoublesymmetric}\nA matrix $M\\in \\K^{[n]^r\\times [n]^r}$  is called  {\\em $r$-column symmetric} (resp., {\\em $r$-row symmetric}) if $M\\Pi_r =M$ (resp., $\\Pi_rM=M$). If $M$ is complex Hermitian or real symmetric, then $M$ is $r$-column symmetric if and only if it is $r$-row symmetric, or equivalently if $\\Pi_rM\\Pi_r=M$, in which case  $M$ is called {\\em $ r $-doubly symmetric} (or just {\\em doubly symmetric} if $r$ is clear from the context).\n\\end{definition}\n\n\nHermitian doubly symmetric matrices are also known as {\\em Bose symmetric} matrices in the quantum information literature. \\MoL{The following stronger symmetry property is sometimes needed (it is considered, e.g.,   in \\cite{Doherty_Wehner_2013}).}\n\n","start":29086,"end":30109}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":["defmaximallysymmetric"],"declared_external":false,"source_only":false,"title":"最大对称性","source_units":[{"raw_tex":"\\begin{definition}[Maximally symmetric matrix]\\label{defmaximallysymmetric}\nA  matrix $M\\in \\K^{[n]^r\\times [n]^r}$ can be   viewed as  a $2r$-tensor $\\vvec(M)\\in (\\K^n)^{\\ot 2r}$, with entries  $\\vvec(M)_{\\ui\\uj} = M_{\\ui,\\uj}$ for $\\ui,\\uj\\in [n]^r$.\nThen, $M$ is said to be {\\em maximally symmetric} when $\\vvec(M)$ is a symmetric $2r$-tensor.\n\\end{definition}\n\n","start":30109,"end":30474}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"最大对称化投影","source_units":[{"raw_tex":"\\tcolred{Note that a Hermitian matrix that is maximally symmetric is in fact real-valued (see also Lemma \\ref{lem:ppt-sym-is-real-rank-2} below).} Let $\\PiMS$ denote the projection from the space $\\K^{[n]^r\\times [n]^r}$ onto $\\MaxSym_r(\\K^n)$, the  space of maximally symmetric matrices. Concretely, the projection $\\PiMS(M)$ is obtained by first constructing the symmetric $2r$-tensor $\\Pi_{2r}(\\vvec(M))$ and then reshaping it as a square matrix indexed by $[n]^r$ (by splitting the $2r$ indices into two classes of size $r$, which can be done   arbitrarily  as the tensor is $2r$-symmetric).\n","start":30474,"end":31070}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"最大对称矩阵的实性","source_units":[{"raw_tex":"\\tcolred{Note that a Hermitian matrix that is maximally symmetric is in fact real-valued (see also Lemma \\ref{lem:ppt-sym-is-real-rank-2} below).} Let $\\PiMS$ denote the projection from the space $\\K^{[n]^r\\times [n]^r}$ onto $\\MaxSym_r(\\K^n)$, the  space of maximally symmetric matrices. Concretely, the projection $\\PiMS(M)$ is obtained by first constructing the symmetric $2r$-tensor $\\Pi_{2r}(\\vvec(M))$ and then reshaping it as a square matrix indexed by $[n]^r$ (by splitting the $2r$ indices into two classes of size $r$, which can be done   arbitrarily  as the tensor is $2r$-symmetric).\n","start":30474,"end":31070}]},{"id":"S010","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"齐次多项式空间","source_units":[{"raw_tex":"Throughout, let $ R := \\R[x] = \\R[x_1,\\ldots,x_n] $ denote  the polynomial ring in variables $ x=(x_1,\\ldots,x_n) $. For an integer $d\\in\\N$,  $ R_d $ denotes its $ d $-th graded component, which consists of the homogeneous polynomials (aka forms) with degree $d$, and $R_{\\le d}=R_0\\oplus R_1\\oplus\\ldots\\oplus R_d$ consists of the polynomials with degree at most $d$. \nWe let $[x]_d= (x^\\alpha)_{\\alpha\\in \\N^n_d}$ denote the vector of all monomials with degree $d$.\n","start":34823,"end":35292}]},{"id":"S011","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"平方半径多项式","source_units":[{"raw_tex":"Throughout we use the notation  $$s=x_1^2+\\ldots +x_n^2.$$\n","start":35292,"end":35351}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"平方和多项式","source_units":[{"raw_tex":"A polynomial $f\\in R$ is a {\\em sum of squares} (abbreviated as sos) if $ f = p_1^2  + \\ldots + p_N^2 $ for some $ N\\in \\N $ and $ p_1,\\ldots,p_N\\in R$. Then, all $p_i$ lie in $R_d$ (resp., $R_{\\le d}$) if $f\\in R_{2d}$ (resp., $f\\in R_{\\le 2d}$). The following characterization is well-known \\tcolblue{(see \\cite{Choi-Lam-Reznick-1995,Powers-Wormann-1998})}.\n\n","start":35616,"end":35977}]},{"id":"S013","semantic_type":"lemma","kind":"lemma","labels":["lem:sos-equiv-gram-matrix"],"declared_external":true,"declaration_text":["原先决条件明确允许B0027；原文420行将该标准刻画归于Choi-Lam-Reznick-1995及Powers-Wormann-1998。"],"source_only":false,"title":"平方和的Gram刻画","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:sos-equiv-gram-matrix}\nA polynomial $f$ is a sum of squares if and only if it admits a positive semidefinite representing matrix (then, often called a {\\em Gram matrix} of $f$).\n\\end{lemma}\n","start":35977,"end":36191}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":["defGramtensor"],"declared_external":false,"source_only":false,"title":"张量指标表示矩阵","source_units":[{"raw_tex":"\\begin{definition} \\label{defGramtensor}\n\tLet $f\\in R_{2d}$. A symmetric matrix $G$ indexed by $[n]^d$ is called a {\\em tensor-indexed representing matrix} of  $ f $ if the following polynomial identity holds:\n\t\\begin{align*}\n\t\tf = (x^{\\otimes d})^{T}G x^{\\otimes d} = \\langle G, (x x^{T})^{\\otimes d} \\rangle. \n\t\\end{align*}\n\\end{definition}\n\t\n","start":39095,"end":39440}]},{"id":"S015","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"唯一最大对称表示","source_units":[{"raw_tex":"\tThen, any polynomial  $f\\in R_{2d}$ admits  a (unique) tensor-indexed representing matrix  that is maximally symmetric, denoted by $\\MaxSym(f)$. Namely, we have $\\MaxSym(f)=\\PiMS(G)$, where $G$ is any tensor-indexed representing matrix of $f$.\n","start":39440,"end":39685}]},{"id":"S016","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件明确允许469–471行的张量指标Gram刻画；471行明确援引B0027的对应结论。"],"source_only":false,"title":"张量Gram平方和刻画","source_units":[{"raw_tex":"Moreover,\nin analogy with \\Cref{lem:sos-equiv-gram-matrix}, a polynomial $ f $ is a sum of squares if and only if it has a tensor-indexed positive semidefinite representing matrix. \n","start":39685,"end":39867}]},{"id":"S017","semantic_type":"definition","kind":"definition","labels":["eqQpM"],"declared_external":false,"source_only":false,"title":"谱层级的参考矩阵","source_units":[{"raw_tex":"\\MoL{\\johnstonlovitz{} \\cite{Johnston_Lovitz_Vijayaraghavan_2023} proposed recently a hierarchy of spectral bounds $\\spec_r(p)$ for the minimum value $p_{\\min}$ of a homogeneous polynomial $p$ over the unit sphere,\\footnote{ Their hierarchy  applies more generally to optimization of  a multivariate tensor over a spherical Segre-Veronese variety. We only focus here on their hierarchy for polynomials on the sphere.} \nsee Section \\ref{secJLV-intro} below for their definition and relationship to the  moment-sos bounds.  Throughout, let $p\\in R_{2d}$, $s=x_1^2+\\cdots +x_n^2$, and let \n\\begin{align}\\label{eqQpM}\n\tQ(p) = \\maxsym(p), \\quad  M = \\maxsym(s^d) \n\\end{align}\ndenote  the maximally symmetric  matrices representing the polynomials $ p $ and $s^d$, respectively (as in Definition \\ref{defGramtensor}). They show the following convergence rate.\n}\n","start":69899,"end":70755}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":["eqQrpM"],"declared_external":false,"source_only":false,"title":"高阶提升矩阵","source_units":[{"raw_tex":"We introduce the spectral hierarchy of \\johnstonlovitz{} \\cite{Johnston_Lovitz_Vijayaraghavan_2023}. \nLet $p\\in R_{2d}$ and recall the matrices $Q(p)$   and $M$ from (\\ref{eqQpM}). For an integer $ r\\ge d $, define the matrices \n\\begin{align}\\label{eqQrpM}\n\tQ_r(p) = \\Pi_{r}(Q(p) \\otimes I_n^{\\otimes (r-d)})\\Pi_{r}, \\\\\n\tM_r= \\Pi_{r}(M \\otimes I_n^{\\otimes (r-d)})\\Pi_{r}. \n\\end{align}\n","start":72976,"end":73362}]},{"id":"S019","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"提升矩阵所表示的形式","source_units":[{"raw_tex":"So,  $ Q_r(p) $ and $M_r$ are   $ r $-doubly symmetric matrices    representing  $ s^{r-d}p $ and $s^r$, respectively. That is, $\\langle Q_r(p), x^{\\ot r}(x^{\\ot r})^T\\rangle = s^{r-d}p$ and $\n\\langle M_r, x^{\\ot r}(x^{\\ot r})^T\\rangle= s^r$ hold.\nThen, the  spectral bound  $\\spec_r(p)$ at the $ r $-th level is defined as the smallest generalized eigenvalue of the matrix $ Q_r(p) $ with respect to $ M_r $. We gather some equivalent formulations for $\\spec_r(p)$. \n\n","start":73362,"end":73831}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":["eqnu1","eqnu2","eqnu3"],"declared_external":false,"source_only":false,"title":"谱界的等价定义","source_units":[{"raw_tex":"\\begin{definition}\n\tThe spectral bound $\\spec_r(p)$  is defined by any of the following equivalent programs:\n\t\\begin{align}\n\t\t\\spec_r(p)& =\\max_{\\nu\\in\\R} \\{ \\nu: Q_r(p)-\\nu M_r\\succeq 0\\},\\label{eqnu1} \\\\\n\t\t& = \\min_{v\\in S^r(\\R^n) } \\{v^T Q_r(p)v: v^T M_rv=1\\}, \\label{eqnu2}\\\\\n\t\t& = \\min_{X } \\{\\langle Q_r(p), X\\rangle: X \\  r\\text{-doubly symmetric},\\ X\\succeq 0,\\ \\langle M_r,X\\rangle =1\\}.\\label{eqnu3}\n\t\\end{align}\n\\end{definition}\n","start":73831,"end":74271}]},{"id":"S021","semantic_type":"definition","kind":"definition","labels":["eqpCL"],"declared_external":false,"source_only":false,"title":"Choi–Lam四次形式","source_units":[{"raw_tex":"We start by picking a  quartic form  in four variables that is nonnegative but not a sum of squares.\nA concrete example is given by the \\emph{Choi-Lam form} $ p_{CL} $, \n\\begin{align}\\label{eqpCL}\n\tp_{CL}(x) = x_1^2x_2^2+x_1^2x_3^2+ x_2^2x_3^2 + x_4^4 -4 x_1x_2x_3x_4.\n\\end{align}\n","start":85861,"end":86142}]},{"id":"S022","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件Choi_Lam_1977；1047行明确引用该文的非负性结论。"],"source_only":false,"title":"Choi–Lam形式非负","source_units":[{"raw_tex":"The Choi-Lam form $ p_{CL}$ is famously known to be a nonnegative polynomial, which is not a sum of squares (see \\cite{Choi_Lam_1977}). Hence, {\\em no} matrix representation of $ p_{CL} $ is positive semidefinite.\n","start":86426,"end":86640}]},{"id":"S023","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件Choi_Lam_1977；1047行明确引用该文的非平方和结论。"],"source_only":false,"title":"Choi–Lam形式非平方和","source_units":[{"raw_tex":"The Choi-Lam form $ p_{CL}$ is famously known to be a nonnegative polynomial, which is not a sum of squares (see \\cite{Choi_Lam_1977}). Hence, {\\em no} matrix representation of $ p_{CL} $ is positive semidefinite.\n","start":86426,"end":86640}]},{"id":"S024","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"不存在半正定表示","source_units":[{"raw_tex":"The Choi-Lam form $ p_{CL}$ is famously known to be a nonnegative polynomial, which is not a sum of squares (see \\cite{Choi_Lam_1977}). Hence, {\\em no} matrix representation of $ p_{CL} $ is positive semidefinite.\n","start":86426,"end":86640}]},{"id":"S025","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"实例条件可弱化","source_units":[{"raw_tex":"This property is stronger than necessary. As it will turn out, our strategy for proving the lower bound $\\Omega(1/r^2)$ for  the spectral hierarchy works with any nonnegative quartic form whose  maximally symmetric representation is not positive semidefinite. \n\n","start":86640,"end":86902}]},{"id":"S026","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"加入第五个变量","source_units":[{"raw_tex":"We use one small trick, which is to choose an unnatural representation of   $ p_{CL} $. While $ p_{CL} $ is intrinsically a polynomial in four variables $x=(x_1,x_2,x_3,x_4)$, one can artificially view it as a polynomial $p$ in five variables $ \\underline{x} := (x_1,x_2,x_3,x_4,u) $, where $u$ is an additional variable, by setting $p(\\underline x)=p_{CL}(x)$.\n","start":86902,"end":87264}]},{"id":"S027","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqmatQ"],"declared_external":false,"source_only":false,"title":"表示矩阵的补零嵌入","source_units":[{"raw_tex":"If, as a 4-variate polynomial,  $ p_{CL} $ is represented by a matrix $ Q^{(4)} \\in \\R^{[4]^2\\times [4]^2} $, then, as a 5-variate polynomial in   $ \\underline{x} = (x,u) $, it is represented by the matrix \n\\begin{align}\\label{eqmatQ}\n\t\\begin{pmatrix}\n\t\tQ^{(4)} & 0\\\\\n\t\t0^{T} & 0\n\t\\end{pmatrix} \\in  \\R^{[5]^2\\times [5]^2}.\n\\end{align}\n","start":87264,"end":87600}]},{"id":"S028","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqmatMQ"],"declared_external":false,"source_only":false,"title":"最大对称表示的补零结构","source_units":[{"raw_tex":"Note that the same holds   for the maximally symmetric representations of $p_{CL}$  in four variables and of $p $ as a polynomial in five variables: \n\\begin{align}\\label{eqmatMQ}\n\t\\maxsym(p)  = \\begin{pmatrix}\n\t\t\\maxsym(p_{CL}) & 0\\\\\n\t\t0^{T} & 0\n\t\\end{pmatrix}.\n\\end{align}\nThe reason is again that $ p $ is only supported on monomials, which do not depend on the fifth variable $ u $. \n","start":87600,"end":87987}]},{"id":"S029","semantic_type":"theorem","kind":"theorem","labels":["thm:lower-bound-spectral"],"declared_external":false,"source_only":false,"title":"谱层级的二次阶下界","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm:lower-bound-spectral}\n\tLet $\\spec_r(p)$ denote the spectral bound of order $r$ for the Choi-Lam form $p \\in \\R[x_1,x_2,x_3,x_4,u]$. Then, we have\t$p_{\\min}=0$ and $\\spec_r(p)=\\Omega(1/r^2)$.\n\\end{theorem}","start":87988,"end":88218}]},{"id":"S030","semantic_type":"proof","kind":"proof","labels":["eqmatH","eqmatH2","eqmats"],"declared_external":false,"source_only":false,"title":"谱下界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\tLet the variables be ordered as $ \\underline{x} = (x,u)$, where $ x = (x_1,\\ldots,x_4) $ and  $ u $ is the fifth variable. Let $r\\ge 2$. Set $ s = x_1^2  + x_2^2+x_3^2 + x_4^2 + u^2 $ and \n\t\\begin{align}\\label{eqmats}\n\t\\begin{split}\n\t & Q :=   \\maxsym(p),\\quad  M := \\maxsym(s^2), \\\\\n\t &  Q_r := \\Pi_r (Q\\otimes I_5^{\\otimes (r-2)}) \\Pi_r,\\quad  M_r :=  \\Pi_r (M\\otimes I_5^{\\otimes (r-2)}) \\Pi_r.\n\t\\end{split}\n\t\\end{align} \n\tSo, $Q$ has the block-form as in (\\ref{eqmatQ}), (\\ref{eqmatMQ}), where $Q^{(4)} :=  \\MaxSym(p_{CL}) \\in \\R^{[4]^2 \\times [4]^2} $ and the remaining blocks are filled by zeros. As in (\\ref{eqPir}),  $\\Pi_2\\in \\R^{[5]^2\\times [5]^2}$ denotes the projection onto the symmetric subspace $S^2(\\R^5)$.\tThe space $\\R^{[4]^2}$ can be seen as a subspace of $\\R^{[5]^2}$ corresponding to the first four variable indices. Let us write $ \\Pi_2^{(4)} \\in \\R^{[4]^2 \\times [4]^2} $ for the restriction of $ \\Pi_2$ to the subspace $ \\R^{[4]^2} $. \n\n\tRecall  $\\spec_r=\\spec_r(p)$ for short. By the definition, we have $\\spec_r\\le p_{\\min}=0$ and \n\t\\begin{align*}\n\t\tQ_r + |\\spec_r| M_r = \\Pi_r ((Q+ |\\spec_r|M) \\otimes I_5^{\\otimes (r-2)}) \\Pi_r \\succeq 0.\n\t\\end{align*}\n\tBy using  $ M\\preceq \\lambda_{\\max}(M) I_5^{\\otimes 2} $, we obtain that \n\t\\begin{align*}\n\t\tG_r :=  \\Pi_r ((Q+ \\lambda_{\\max}(M)|\\spec_r|I_5^{\\otimes 2}) \\otimes I_5^{\\otimes (r-2)}) \\Pi_r \\succeq 0.\n\t\\end{align*}\n\tIn particular, every principal submatrix of $ G_r $ is positive semidefinite. We will now examine \\tcolblue{in detail} the following principal submatrix \n\t\\begin{align}\\label{eqmatH}\n\t\tH:= ((G_r)_{(i_1,i_2,5,\\ldots,5), (j_1,j_2,5,\\ldots,5)})_{i_1,i_2,j_1,j_2 \\in \\{1,\\ldots,4\\}} \\in \\R^{[4]^2\\times [4]^2}\n\t\\end{align}\n\tof $ G_r $, where the first two row indices $ i_1, i_2 $ and the first two column indices $ j_1,j_2 $ each range from $ 1 $ to $ 4 $, while the other $ r-2 $ row and column indices are set equal to $ 5 $.\n\t\n\n\tRecall the notation $ \\underline{i}^{\\sigma} := (i_{\\sigma(1)},\\ldots,i_{\\sigma(r)}) $ for any $\\sigma\\in \\MFS_r$. For $\\ui,\\uj\\in [5]^r$, using (\\ref{eqPir}), we may expand \n\t\\begin{align*}\n\t\t(G_r)_{\\underline{i},\\underline{j}} = \\left(\\frac{1}{r!}\\right)^2 \\sum_{\\sigma \\in \\mathfrak{S}_{r}}\\sum_{\\tau \\in \\mathfrak{S}_{r}}  (Q \\otimes I_5^{\\otimes (r-2)} + \\lambda_{\\max}(M)|\\spec_r|I_5^{\\otimes r})_{\\underline{i}^{\\sigma}, \\underline{j}^{\\tau}}.\n\t\\end{align*}\n\tIn order to compute the submatrix $ H $, we set the last $ r-2 $ indices of both $ \\underline{i} $ and $ \\underline{j} $ equal to $ 5 $ and the other two indices not equal to $ 5 $. Hence, on the right hand side, we have to count how many permutations ``survive'' when we do this, in the sense of giving a nonzero contribution to $ H $. \n\tWe may write \n\t\\begin{align}\\label{eqmatH2}\n\t H = A + \\lambda_{\\max}(M)|\\spec_r|B,\n\t \\end{align}\n\t  where $ A $ is the contribution arising from $ Q \\otimes I_5^{\\otimes (r-2)} $ and $ B $ is the contribution from $ I_5^{\\otimes r}  $. Since $ I_5^{\\otimes r}  $ is diagonal, the only nonzero entries of $ B $ are at indices of the form $ (i_1i_2,i_1i_2) $ and $ (i_1i_2,i_2i_1) $. \n\tFor later use, notice that $ (\\Pi_{2})_{i_1i_2,i_1i_2} = \\frac{1}{2} $ if $ i_1\\ne i_2 \\in [4]$, and $(\\Pi_{2})_{i_1i_1,i_1i_1} = 1 $ for $i_1\\in [4]$. \n\tWe now calculate the matrix $B$: for $i_1,i_2\\in [4]$, \n\t\\begin{align*}\n\t\tB_{i_1i_2,i_1i_2} &= B_{i_1i_2,i_2i_1} = \\frac{1}{(r!)^2} \\sum_{\\sigma \\in \\mathfrak{S}_{r}}\\sum_{\\tau \\in \\mathfrak{S}_{r}}  (I_5^{\\otimes r})_{(i_{1},i_{2},5,\\ldots,5)^{\\sigma}, (i_{1},i_{2},5,\\ldots,5)^{\\tau}}\\\\\n\t\t&= 2(\\Pi_{2})_{i_1i_2,i_1i_2} \\frac{(r-2)!}{(r!)^2} \\sum_{\\sigma \\in \\mathfrak{S}_{r}}  (I_5^{\\otimes r})_{(i_{1},i_{2},5,\\ldots,5)^{\\sigma}, (i_{1},i_{2},5,\\ldots,5)^{\\sigma}}\\\\\n\t\t&= 2(\\Pi_{2})_{i_1i_2,i_1i_2}\\frac{(r-2)!}{r!} =  \\binom{r}{2}^{-1} (\\Pi_2)_{i_1i_2,i_1i_2}. \n\t\\end{align*}\t\n\tHere, in the second line, we   used the fact that $ I_n^{\\otimes r} $ is diagonal and that the row and column indices are equal if and only if $ (i_{\\sigma(1)},i_{\\sigma(2)}) = (i_{\\tau(1)},i_{\\tau(2)}) $. \n\tGiven any permutation $ \\sigma \\in \\mathfrak{S}_r$, there are $ (r-2)! $ permutations $ \\tau \\in \\mathfrak{S}_r $ with this property if $ i_1\\ne i_2 $, and $ 2(r-2)! $ permutations $ \\tau $ with this property if $ i_1 = i_2 $. Hence, there are $ 2(r-2)!(\\Pi_{2})_{i_1i_2,i_1i_2} $ permutations $ \\tau $ with this property for any given $ \\sigma $. \n\tIn the third line, we used the fact that all diagonal entries of $ I_n^{\\otimes r} $ are equal to $ 1 $, so the sum evaluates to $ r! $.  \n\tSummarized, we conclude that $ B = \\binom{r}{2}^{-1} \\Pi_2^{(4)} $. \n\tNow, let us calculate $ A $: for $i_1,i_2,j_1,j_2\\in [4]$,\n\t\\begin{align*}\n\t\tA_{i_1i_2,j_1j_2} = \\frac{1}{(r!)^2} \\sum_{\\sigma \\in \\mathfrak{S}_{r}}\\sum_{\\tau \\in \\mathfrak{S}_{r}}  (Q\\otimes I_5^{\\otimes r-2})_{(i_{1},i_{2},5,\\ldots,5)^{\\sigma}, (j_{1},j_{2},5,\\ldots,5)^{\\tau}}.\n\t\\end{align*}\n\tRecall that $ Q_{i,5} = 0 $ if $ i\\in \\{1,\\ldots,4\\} $. Therefore, the only nonzero contributions come from permutations $ \\sigma $ and $ \\tau $, which keep all 5-indices `to the right', i.e., at the last $r-2$ positions. There are $ 2(r-2)! $ permutations $ \\sigma $ and $ 2(r-2)! $ permutations $ \\tau $ with this property. They may be written as $ \\sigma = \\sigma_2\\circ \\sigma_{r-2} $ and $ \\tau =  \\tau_2\\circ \\tau_{r-2}$, respectively, with $ \\sigma_2, \\tau_2 \\in \\mathfrak{S}_2 $ and $ \\sigma_{r-2}, \\tau_{r-2} \\in \\mathfrak{S}_{r-2} $.\n\tSo, we obtain  \n\t\\begin{align*}\n\t\tA_{i_1i_2,j_1j_2} &= \\frac{((r-2)!)^2}{(r!)^2} \\sum_{\\sigma_2, \\tau_2 \\in \\mathfrak{S}_{2}} (Q\\otimes I_n^{\\otimes r-2})_{(i_{\\sigma_2(1)},i_{\\sigma_2(2)},5,\\ldots,5), (j_{\\tau_2(1)},j_{\\tau_2(2)},5,\\ldots,5)}.\\\\\n\t\t&= \\left(\\frac{(r-2)!}{r!}\\right)^2 \\sum_{\\sigma_2, \\tau_2 \\in \\mathfrak{S}_{2}} Q_{i_{\\sigma_2(1)}i_{\\sigma_2(2)}, j_{\\tau_2(1)}j_{\\tau_2(2)}}.\\\\\n\t\t&= \\binom{r}{2}^{-2} Q_{i_1i_2,j_1j_2}.\n\t\\end{align*}\n\tIn the first line, we used the fact that the $ (r-2)!^2 $ permutations of the 5-indices all yield the same indices. In the second line, we used that $ (I_n^{\\otimes r-2})_{5\\ldots5,5\\ldots5} = 1 $. In the third line, we used that $ Q $ is maximally symmetric, hence in particular doubly symmetric. Thus, the $ 2^2 = 4 $ permutations of the non-5 indices all yield the same entry. Summarizing, we have shown  \n\t$$ A = \\binom{r}{2}^{-2}Q^{(4)},\\quad  B = \\binom{r}{2}^{-1}\\Pi_{2}^{(4)}.\n\t$$\n\t We conclude that the   matrix $ H $ from (\\ref{eqmatH})-(\\ref{eqmatH2}) can be expressed as\n\t\\begin{align*}\n\t\t\\binom{r}{2}^{2}  H = Q^{(4)} + \\lambda_{\\max}(M)|\\spec_r|\\binom{r}{2}\\Pi_{2}^{(4)}. \n\t\\end{align*}\n\tNow, we use that $H\\succeq 0$ and $\\begin{pmatrix} \\Pi_2^{(4)} & 0 \\\\ 0^{T} & 0 \\end{pmatrix} \\preceq I_5^{\\otimes 2} \\preceq \\frac{1}{\\lambda_{\\min}(M)} M $ to obtain  \n\t\\begin{align*}\n\t\t0\\preceq \\binom{r}{2}^{2} \\begin{pmatrix} H & 0 \\\\ 0^{T} & 0 \\end{pmatrix}  \\preceq Q + |\\spec_r|\\frac{\\lambda_{\\max}(M)}{\\lambda_{\\min}(M)} \\binom{r}{2} M. \n\t\\end{align*}\n\tBy definition, $ |\\spec_2| $ is the smallest value such that \n\t\\begin{align*}\n\t\tQ + |\\spec_2| M \\succeq 0. \n\t\\end{align*}\n\tHence, it follows that \n\t\\begin{align*}\n\t\t |\\spec_r|\\frac{\\lambda_{\\max}(M)}{\\lambda_{\\min}(M)} \\binom{r}{2} \\ge |\\spec_2|, \\:\\: \\text{ and thus }\\:\\: \t|\\spec_r| \\ge \\frac{\\lambda_{\\min}(M)}{\\lambda_{\\max}(M)} |\\spec_2|\\binom{r}{2}^{-1} = \\Omega\\left(\\frac{1}{r^2}\\right),\t\t \n\t\\end{align*}\n\\tcolblue{using the fact that $M\\succ 0$ \\cite{Johnston_Lovitz_Vijayaraghavan_2023} and $|\\spec_2|>0$ as $p$ is not sos.} \n\\end{proof}","start":88220,"end":95758}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":["eqmats"],"declared_external":false,"source_only":false,"title":"固定五元矩阵与参数","source_units":[{"raw_tex":"\\begin{proof}\n\tLet the variables be ordered as $ \\underline{x} = (x,u)$, where $ x = (x_1,\\ldots,x_4) $ and  $ u $ is the fifth variable. Let $r\\ge 2$. Set $ s = x_1^2  + x_2^2+x_3^2 + x_4^2 + u^2 $ and \n\t\\begin{align}\\label{eqmats}\n\t\\begin{split}\n\t & Q :=   \\maxsym(p),\\quad  M := \\maxsym(s^2), \\\\\n\t &  Q_r := \\Pi_r (Q\\otimes I_5^{\\otimes (r-2)}) \\Pi_r,\\quad  M_r :=  \\Pi_r (M\\otimes I_5^{\\otimes (r-2)}) \\Pi_r.\n\t\\end{split}\n\t\\end{align} \n\tSo, $Q$ has the block-form as in (\\ref{eqmatQ}), (\\ref{eqmatMQ}), where $Q^{(4)} :=  \\MaxSym(p_{CL}) \\in \\R^{[4]^2 \\times [4]^2} $ and the remaining blocks are filled by zeros. As in (\\ref{eqPir}),  $\\Pi_2\\in \\R^{[5]^2\\times [5]^2}$ denotes the projection onto the symmetric subspace $S^2(\\R^5)$.\tThe space $\\R^{[4]^2}$ can be seen as a subspace of $\\R^{[5]^2}$ corresponding to the first four variable indices. Let us write $ \\Pi_2^{(4)} \\in \\R^{[4]^2 \\times [4]^2} $ for the restriction of $ \\Pi_2$ to the subspace $ \\R^{[4]^2} $. \n\n","start":88220,"end":89196}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"实例最小值为零","source_units":[{"raw_tex":"\tRecall  $\\spec_r=\\spec_r(p)$ for short. By the definition, we have $\\spec_r\\le p_{\\min}=0$ and \n","start":89196,"end":89293}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"谱界为非正数","source_units":[{"raw_tex":"\tRecall  $\\spec_r=\\spec_r(p)$ for short. By the definition, we have $\\spec_r\\le p_{\\min}=0$ and \n","start":89196,"end":89293}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"最优谱矩阵铅笔半正定","source_units":[{"raw_tex":"\tRecall  $\\spec_r=\\spec_r(p)$ for short. By the definition, we have $\\spec_r\\le p_{\\min}=0$ and \n\t\\begin{align*}\n\t\tQ_r + |\\spec_r| M_r = \\Pi_r ((Q+ |\\spec_r|M) \\otimes I_5^{\\otimes (r-2)}) \\Pi_r \\succeq 0.\n\t\\end{align*}\n","start":89196,"end":89416}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"构造放大的半正定矩阵","source_units":[{"raw_tex":"\tBy using  $ M\\preceq \\lambda_{\\max}(M) I_5^{\\otimes 2} $, we obtain that \n\t\\begin{align*}\n\t\tG_r :=  \\Pi_r ((Q+ \\lambda_{\\max}(M)|\\spec_r|I_5^{\\otimes 2}) \\otimes I_5^{\\otimes (r-2)}) \\Pi_r \\succeq 0.\n\t\\end{align*}\n","start":89416,"end":89631}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":["eqmatH"],"declared_external":false,"source_only":false,"title":"选取特殊主子矩阵","source_units":[{"raw_tex":"\tIn particular, every principal submatrix of $ G_r $ is positive semidefinite. We will now examine \\tcolblue{in detail} the following principal submatrix \n\t\\begin{align}\\label{eqmatH}\n\t\tH:= ((G_r)_{(i_1,i_2,5,\\ldots,5), (j_1,j_2,5,\\ldots,5)})_{i_1,i_2,j_1,j_2 \\in \\{1,\\ldots,4\\}} \\in \\R^{[4]^2\\times [4]^2}\n\t\\end{align}\n\tof $ G_r $, where the first two row indices $ i_1, i_2 $ and the first two column indices $ j_1,j_2 $ each range from $ 1 $ to $ 4 $, while the other $ r-2 $ row and column indices are set equal to $ 5 $.\n\t\n","start":89631,"end":90159}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"展开两侧置换平均","source_units":[{"raw_tex":"\n\tRecall the notation $ \\underline{i}^{\\sigma} := (i_{\\sigma(1)},\\ldots,i_{\\sigma(r)}) $ for any $\\sigma\\in \\MFS_r$. For $\\ui,\\uj\\in [5]^r$, using (\\ref{eqPir}), we may expand \n\t\\begin{align*}\n\t\t(G_r)_{\\underline{i},\\underline{j}} = \\left(\\frac{1}{r!}\\right)^2 \\sum_{\\sigma \\in \\mathfrak{S}_{r}}\\sum_{\\tau \\in \\mathfrak{S}_{r}}  (Q \\otimes I_5^{\\otimes (r-2)} + \\lambda_{\\max}(M)|\\spec_r|I_5^{\\otimes r})_{\\underline{i}^{\\sigma}, \\underline{j}^{\\tau}}.\n\t\\end{align*}\n\tIn order to compute the submatrix $ H $, we set the last $ r-2 $ indices of both $ \\underline{i} $ and $ \\underline{j} $ equal to $ 5 $ and the other two indices not equal to $ 5 $. Hence, on the right hand side, we have to count how many permutations ``survive'' when we do this, in the sense of giving a nonzero contribution to $ H $. \n","start":90159,"end":90965}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":["eqmatH2"],"declared_external":false,"source_only":false,"title":"拆分两类矩阵贡献","source_units":[{"raw_tex":"\tWe may write \n\t\\begin{align}\\label{eqmatH2}\n\t H = A + \\lambda_{\\max}(M)|\\spec_r|B,\n\t \\end{align}\n\t  where $ A $ is the contribution arising from $ Q \\otimes I_5^{\\otimes (r-2)} $ and $ B $ is the contribution from $ I_5^{\\otimes r}  $. Since $ I_5^{\\otimes r}  $ is diagonal, the only nonzero entries of $ B $ are at indices of the form $ (i_1i_2,i_1i_2) $ and $ (i_1i_2,i_2i_1) $. \n","start":90965,"end":91349}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"二阶投影的对角元","source_units":[{"raw_tex":"\tFor later use, notice that $ (\\Pi_{2})_{i_1i_2,i_1i_2} = \\frac{1}{2} $ if $ i_1\\ne i_2 \\in [4]$, and $(\\Pi_{2})_{i_1i_1,i_1i_1} = 1 $ for $i_1\\in [4]$. \n","start":91349,"end":91503}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"计算单位阵贡献","source_units":[{"raw_tex":"\tWe now calculate the matrix $B$: for $i_1,i_2\\in [4]$, \n\t\\begin{align*}\n\t\tB_{i_1i_2,i_1i_2} &= B_{i_1i_2,i_2i_1} = \\frac{1}{(r!)^2} \\sum_{\\sigma \\in \\mathfrak{S}_{r}}\\sum_{\\tau \\in \\mathfrak{S}_{r}}  (I_5^{\\otimes r})_{(i_{1},i_{2},5,\\ldots,5)^{\\sigma}, (i_{1},i_{2},5,\\ldots,5)^{\\tau}}\\\\\n\t\t&= 2(\\Pi_{2})_{i_1i_2,i_1i_2} \\frac{(r-2)!}{(r!)^2} \\sum_{\\sigma \\in \\mathfrak{S}_{r}}  (I_5^{\\otimes r})_{(i_{1},i_{2},5,\\ldots,5)^{\\sigma}, (i_{1},i_{2},5,\\ldots,5)^{\\sigma}}\\\\\n\t\t&= 2(\\Pi_{2})_{i_1i_2,i_1i_2}\\frac{(r-2)!}{r!} =  \\binom{r}{2}^{-1} (\\Pi_2)_{i_1i_2,i_1i_2}. \n\t\\end{align*}\t\n\tHere, in the second line, we   used the fact that $ I_n^{\\otimes r} $ is diagonal and that the row and column indices are equal if and only if $ (i_{\\sigma(1)},i_{\\sigma(2)}) = (i_{\\tau(1)},i_{\\tau(2)}) $. \n\tGiven any permutation $ \\sigma \\in \\mathfrak{S}_r$, there are $ (r-2)! $ permutations $ \\tau \\in \\mathfrak{S}_r $ with this property if $ i_1\\ne i_2 $, and $ 2(r-2)! $ permutations $ \\tau $ with this property if $ i_1 = i_2 $. Hence, there are $ 2(r-2)!(\\Pi_{2})_{i_1i_2,i_1i_2} $ permutations $ \\tau $ with this property for any given $ \\sigma $. \n\tIn the third line, we used the fact that all diagonal entries of $ I_n^{\\otimes r} $ are equal to $ 1 $, so the sum evaluates to $ r! $.  \n","start":91503,"end":92783}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"得到B的投影公式","source_units":[{"raw_tex":"\tSummarized, we conclude that $ B = \\binom{r}{2}^{-1} \\Pi_2^{(4)} $. \n","start":92783,"end":92853}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"筛选Q贡献的置换","source_units":[{"raw_tex":"\tNow, let us calculate $ A $: for $i_1,i_2,j_1,j_2\\in [4]$,\n\t\\begin{align*}\n\t\tA_{i_1i_2,j_1j_2} = \\frac{1}{(r!)^2} \\sum_{\\sigma \\in \\mathfrak{S}_{r}}\\sum_{\\tau \\in \\mathfrak{S}_{r}}  (Q\\otimes I_5^{\\otimes r-2})_{(i_{1},i_{2},5,\\ldots,5)^{\\sigma}, (j_{1},j_{2},5,\\ldots,5)^{\\tau}}.\n\t\\end{align*}\n\tRecall that $ Q_{i,5} = 0 $ if $ i\\in \\{1,\\ldots,4\\} $. Therefore, the only nonzero contributions come from permutations $ \\sigma $ and $ \\tau $, which keep all 5-indices `to the right', i.e., at the last $r-2$ positions. There are $ 2(r-2)! $ permutations $ \\sigma $ and $ 2(r-2)! $ permutations $ \\tau $ with this property. They may be written as $ \\sigma = \\sigma_2\\circ \\sigma_{r-2} $ and $ \\tau =  \\tau_2\\circ \\tau_{r-2}$, respectively, with $ \\sigma_2, \\tau_2 \\in \\mathfrak{S}_2 $ and $ \\sigma_{r-2}, \\tau_{r-2} \\in \\mathfrak{S}_{r-2} $.\n","start":92853,"end":93694}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"计算Q贡献系数","source_units":[{"raw_tex":"\tSo, we obtain  \n\t\\begin{align*}\n\t\tA_{i_1i_2,j_1j_2} &= \\frac{((r-2)!)^2}{(r!)^2} \\sum_{\\sigma_2, \\tau_2 \\in \\mathfrak{S}_{2}} (Q\\otimes I_n^{\\otimes r-2})_{(i_{\\sigma_2(1)},i_{\\sigma_2(2)},5,\\ldots,5), (j_{\\tau_2(1)},j_{\\tau_2(2)},5,\\ldots,5)}.\\\\\n\t\t&= \\left(\\frac{(r-2)!}{r!}\\right)^2 \\sum_{\\sigma_2, \\tau_2 \\in \\mathfrak{S}_{2}} Q_{i_{\\sigma_2(1)}i_{\\sigma_2(2)}, j_{\\tau_2(1)}j_{\\tau_2(2)}}.\\\\\n\t\t&= \\binom{r}{2}^{-2} Q_{i_1i_2,j_1j_2}.\n\t\\end{align*}\n\tIn the first line, we used the fact that the $ (r-2)!^2 $ permutations of the 5-indices all yield the same indices. In the second line, we used that $ (I_n^{\\otimes r-2})_{5\\ldots5,5\\ldots5} = 1 $. In the third line, we used that $ Q $ is maximally symmetric, hence in particular doubly symmetric. Thus, the $ 2^2 = 4 $ permutations of the non-5 indices all yield the same entry. Summarizing, we have shown  \n","start":93694,"end":94557}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"汇总两类贡献","source_units":[{"raw_tex":"\t$$ A = \\binom{r}{2}^{-2}Q^{(4)},\\quad  B = \\binom{r}{2}^{-1}\\Pi_{2}^{(4)}.\n\t$$\n","start":94557,"end":94637}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"得到缩放主子矩阵公式","source_units":[{"raw_tex":"\t We conclude that the   matrix $ H $ from (\\ref{eqmatH})-(\\ref{eqmatH2}) can be expressed as\n\t\\begin{align*}\n\t\t\\binom{r}{2}^{2}  H = Q^{(4)} + \\lambda_{\\max}(M)|\\spec_r|\\binom{r}{2}\\Pi_{2}^{(4)}. \n\t\\end{align*}\n","start":94637,"end":94849}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"嵌入并比较参考矩阵","source_units":[{"raw_tex":"\tNow, we use that $H\\succeq 0$ and $\\begin{pmatrix} \\Pi_2^{(4)} & 0 \\\\ 0^{T} & 0 \\end{pmatrix} \\preceq I_5^{\\otimes 2} \\preceq \\frac{1}{\\lambda_{\\min}(M)} M $ to obtain  \n\t\\begin{align*}\n\t\t0\\preceq \\binom{r}{2}^{2} \\begin{pmatrix} H & 0 \\\\ 0^{T} & 0 \\end{pmatrix}  \\preceq Q + |\\spec_r|\\frac{\\lambda_{\\max}(M)}{\\lambda_{\\min}(M)} \\binom{r}{2} M. \n\t\\end{align*}\n","start":94849,"end":95210}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"调用二阶谱阈值","source_units":[{"raw_tex":"\tBy definition, $ |\\spec_2| $ is the smallest value such that \n\t\\begin{align*}\n\t\tQ + |\\spec_2| M \\succeq 0. \n\t\\end{align*}\n","start":95210,"end":95333}]},{"id":"S048","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件Johnston_Lovitz_Vijayaraghavan_2023；1147行明确引用M≻0，但未注明所用命题编号或谱空间。"],"source_only":false,"title":"参考矩阵的严格正定性","source_units":[{"raw_tex":"\\tcolblue{using the fact that $M\\succ 0$ \\cite{Johnston_Lovitz_Vijayaraghavan_2023} and $|\\spec_2|>0$ as $p$ is not sos.} \n","start":95624,"end":95747}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"二阶谱误差严格为正","source_units":[{"raw_tex":"\\tcolblue{using the fact that $M\\succ 0$ \\cite{Johnston_Lovitz_Vijayaraghavan_2023} and $|\\spec_2|>0$ as $p$ is not sos.} \n","start":95624,"end":95747}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"推出绝对值二次阶下界","source_units":[{"raw_tex":"\tHence, it follows that \n\t\\begin{align*}\n\t\t |\\spec_r|\\frac{\\lambda_{\\max}(M)}{\\lambda_{\\min}(M)} \\binom{r}{2} \\ge |\\spec_2|, \\:\\: \\text{ and thus }\\:\\: \t|\\spec_r| \\ge \\frac{\\lambda_{\\min}(M)}{\\lambda_{\\max}(M)} |\\spec_2|\\binom{r}{2}^{-1} = \\Omega\\left(\\frac{1}{r^2}\\right),\t\t \n\t\\end{align*}\n\\tcolblue{using the fact that $M\\succ 0$ \\cite{Johnston_Lovitz_Vijayaraghavan_2023} and $|\\spec_2|>0$ as $p$ is not sos.} \n\\end{proof}","start":95333,"end":95758}]},{"id":"S051","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件明确允许：Standard finite-dimensional linear algebra，包括orthogonal projections、positive semidefinite order、principal submatrices、tensor products及eigenvalue bounds。"],"source_only":false,"title":"有限维半正定矩阵规则","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S029","edges":[{"source_id":"S002","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S005","target_id":"S008","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S010","target_id":"S014","kind":"supports"},{"source_id":"S004","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S008","target_id":"S015","kind":"supports"},{"source_id":"S013","target_id":"S016","kind":"supports"},{"source_id":"S014","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S011","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S005","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S014","target_id":"S019","kind":"supports"},{"source_id":"S006","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S005","target_id":"S020","kind":"supports"},{"source_id":"S006","target_id":"S020","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S016","target_id":"S024","kind":"supports"},{"source_id":"S021","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S014","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S015","target_id":"S028","kind":"supports"},{"source_id":"S026","target_id":"S029","kind":"supports"},{"source_id":"S020","target_id":"S029","kind":"supports"},{"source_id":"S001","target_id":"S029","kind":"supports"},{"source_id":"S017","target_id":"S031","kind":"supports"},{"source_id":"S018","target_id":"S031","kind":"supports"},{"source_id":"S011","target_id":"S031","kind":"supports"},{"source_id":"S027","target_id":"S031","kind":"supports"},{"source_id":"S028","target_id":"S031","kind":"supports"},{"source_id":"S005","target_id":"S031","kind":"supports"},{"source_id":"S022","target_id":"S032","kind":"supports"},{"source_id":"S021","target_id":"S032","kind":"supports"},{"source_id":"S026","target_id":"S032","kind":"supports"},{"source_id":"S001","target_id":"S032","kind":"supports"},{"source_id":"S020","target_id":"S033","kind":"supports"},{"source_id":"S019","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S020","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S031","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S051","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S051","target_id":"S036","kind":"supports"},{"source_id":"S005","target_id":"S037","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S005","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S038","target_id":"S041","kind":"supports"},{"source_id":"S037","target_id":"S042","kind":"supports"},{"source_id":"S038","target_id":"S042","kind":"supports"},{"source_id":"S028","target_id":"S042","kind":"supports"},{"source_id":"S031","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S007","target_id":"S043","kind":"supports"},{"source_id":"S006","target_id":"S043","kind":"supports"},{"source_id":"S031","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S041","target_id":"S044","kind":"supports"},{"source_id":"S031","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S036","target_id":"S045","kind":"supports"},{"source_id":"S038","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S036","target_id":"S046","kind":"supports"},{"source_id":"S028","target_id":"S046","kind":"supports"},{"source_id":"S051","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S020","target_id":"S047","kind":"supports"},{"source_id":"S031","target_id":"S047","kind":"supports"},{"source_id":"S033","target_id":"S047","kind":"supports"},{"source_id":"S023","target_id":"S049","kind":"supports"},{"source_id":"S026","target_id":"S049","kind":"supports"},{"source_id":"S016","target_id":"S049","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S046","target_id":"S050","kind":"supports"},{"source_id":"S047","target_id":"S050","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S031","target_id":"S050","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S030","kind":"supports"},{"source_id":"S033","target_id":"S030","kind":"supports"},{"source_id":"S034","target_id":"S030","kind":"supports"},{"source_id":"S035","target_id":"S030","kind":"supports"},{"source_id":"S036","target_id":"S030","kind":"supports"},{"source_id":"S037","target_id":"S030","kind":"supports"},{"source_id":"S038","target_id":"S030","kind":"supports"},{"source_id":"S039","target_id":"S030","kind":"supports"},{"source_id":"S040","target_id":"S030","kind":"supports"},{"source_id":"S041","target_id":"S030","kind":"supports"},{"source_id":"S042","target_id":"S030","kind":"supports"},{"source_id":"S043","target_id":"S030","kind":"supports"},{"source_id":"S044","target_id":"S030","kind":"supports"},{"source_id":"S045","target_id":"S030","kind":"supports"},{"source_id":"S046","target_id":"S030","kind":"supports"},{"source_id":"S047","target_id":"S030","kind":"supports"},{"source_id":"S049","target_id":"S030","kind":"supports"},{"source_id":"S050","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S029","edges":[{"source_id":"S002","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S005","target_id":"S008","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S010","target_id":"S014","kind":"supports"},{"source_id":"S004","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S008","target_id":"S015","kind":"supports"},{"source_id":"S013","target_id":"S016","kind":"supports"},{"source_id":"S014","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S011","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S005","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S014","target_id":"S019","kind":"supports"},{"source_id":"S006","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S005","target_id":"S020","kind":"supports"},{"source_id":"S006","target_id":"S020","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S016","target_id":"S024","kind":"supports"},{"source_id":"S021","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S014","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S015","target_id":"S028","kind":"supports"},{"source_id":"S026","target_id":"S029","kind":"supports"},{"source_id":"S020","target_id":"S029","kind":"supports"},{"source_id":"S001","target_id":"S029","kind":"supports"},{"source_id":"S017","target_id":"S031","kind":"supports"},{"source_id":"S018","target_id":"S031","kind":"supports"},{"source_id":"S011","target_id":"S031","kind":"supports"},{"source_id":"S027","target_id":"S031","kind":"supports"},{"source_id":"S028","target_id":"S031","kind":"supports"},{"source_id":"S005","target_id":"S031","kind":"supports"},{"source_id":"S022","target_id":"S032","kind":"supports"},{"source_id":"S021","target_id":"S032","kind":"supports"},{"source_id":"S026","target_id":"S032","kind":"supports"},{"source_id":"S001","target_id":"S032","kind":"supports"},{"source_id":"S020","target_id":"S033","kind":"supports"},{"source_id":"S019","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S020","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S031","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S051","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S051","target_id":"S036","kind":"supports"},{"source_id":"S005","target_id":"S037","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S005","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S038","target_id":"S041","kind":"supports"},{"source_id":"S037","target_id":"S042","kind":"supports"},{"source_id":"S038","target_id":"S042","kind":"supports"},{"source_id":"S028","target_id":"S042","kind":"supports"},{"source_id":"S031","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S007","target_id":"S043","kind":"supports"},{"source_id":"S006","target_id":"S043","kind":"supports"},{"source_id":"S031","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S041","target_id":"S044","kind":"supports"},{"source_id":"S031","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S036","target_id":"S045","kind":"supports"},{"source_id":"S038","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S036","target_id":"S046","kind":"supports"},{"source_id":"S028","target_id":"S046","kind":"supports"},{"source_id":"S051","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S020","target_id":"S047","kind":"supports"},{"source_id":"S031","target_id":"S047","kind":"supports"},{"source_id":"S033","target_id":"S047","kind":"supports"},{"source_id":"S023","target_id":"S049","kind":"supports"},{"source_id":"S026","target_id":"S049","kind":"supports"},{"source_id":"S016","target_id":"S049","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S046","target_id":"S050","kind":"supports"},{"source_id":"S047","target_id":"S050","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S031","target_id":"S050","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S030","kind":"supports"},{"source_id":"S033","target_id":"S030","kind":"supports"},{"source_id":"S034","target_id":"S030","kind":"supports"},{"source_id":"S035","target_id":"S030","kind":"supports"},{"source_id":"S036","target_id":"S030","kind":"supports"},{"source_id":"S037","target_id":"S030","kind":"supports"},{"source_id":"S038","target_id":"S030","kind":"supports"},{"source_id":"S039","target_id":"S030","kind":"supports"},{"source_id":"S040","target_id":"S030","kind":"supports"},{"source_id":"S041","target_id":"S030","kind":"supports"},{"source_id":"S042","target_id":"S030","kind":"supports"},{"source_id":"S043","target_id":"S030","kind":"supports"},{"source_id":"S044","target_id":"S030","kind":"supports"},{"source_id":"S045","target_id":"S030","kind":"supports"},{"source_id":"S046","target_id":"S030","kind":"supports"},{"source_id":"S047","target_id":"S030","kind":"supports"},{"source_id":"S049","target_id":"S030","kind":"supports"},{"source_id":"S050","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eqpmin"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"球面多项式最小化","source_units":[{"raw_tex":"Throughout we consider the polynomial optimization problem, asking to minimize a polynomial $p$ over the unit sphere $\\oS^{n-1}$, i.e., to compute\n\\begin{align}\\label{eqpmin}\np_{\\min}=\\min_{x\\in \\oS^{n-1}}p(x)=\\min\\{p(x): x\\in \\R^n, \\|x\\|_2=1\\}.\n\\end{align}\nTherein, $p\\in \\R[x]=\\R[x_1,\\ldots,x_n]$ is an $n$-variate polynomial of degree $2d$. In other words,\n","start":7810,"end":8170}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"指标与置换记号","source_units":[{"raw_tex":"We begin with some notation that is used throughout the paper. $\\N=\\{0,1,2,\\ldots\\}$ is the set of nonnegative integers. For an integer $d\\in \\N$, $\\N^n_d$ denotes the set of sequences $\\alpha\\in \\N^n$ with $|\\alpha|:=\\alpha_1+\\ldots+\\alpha_n=d$. We let $\\MFS_d$ denote the set of permutations of the $d$-element set $[d]=\\{1,\\ldots,d\\}$. We let $e_1,\\ldots,e_n$ denote the standard unit vectors in $\\C^n$, where all entries of $e_i$ are 0 except 1 at the $i$th coordinate. For a vector $u\\in \\C^n$,  $\\|u \\|_2=\\sqrt {u^*u}$ denote its Euclidean norm. The unit sphere in $\\R^n$ is  denoted $\\oS^{n-1}=\\{x\\in\\R^n: \\|x\\|_2=\\sum_{i=1}^nx_i^2=1\\}$.  We let $ \\conv(S) $ denote   the convex  hull of a set $ S $ and  $ \\cone(S) $ is its conic hull.  \n\n","start":24256,"end":25003}]},{"id":"S003","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"矩阵序与特征值","source_units":[{"raw_tex":"For $\\K=\\R$ or $\\C$, we use the trace inner product on the space $\\K^{n\\times n}$ of matrices:  $\\langle A,B\\rangle= \\Tr(A^*B)$ for $A,B\\in \\K^{n\\times n}$. Here, $A^*= \\bar A^T$ denotes the complex conjugate transpose of $A$; $A$ is Hermitian if $A^*=A$. For a Hermitian matrix $A$, $\\lambda_{\\max}(A)$ and $\\lambda_{\\min}(A)$ denote, respectively, its largest and smallest eigenvalues.  The matrix $A$ is positive semidefinite, written as $A\\succeq 0$, if $\\lambda_{\\min}(A)\\ge 0$. For Hermitian matrices $ A, B $, we write $ A\\preceq B $ if $ B-A $ is positive semidefinite. \\tcolblue{Throughout, $I_n$ denotes the $n\\times n$ identity matrix.}\n","start":25003,"end":25651}]},{"id":"S004","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"张量与序列指标","source_units":[{"raw_tex":"Let $\\K=\\R$ or $\\C$ be the real or complex field. For integers $r,n\\ge 1$, $(\\K^n)^{\\ot r}$ denotes the space of $r$-tensors on $\\K^n$, with elements $v=(v_{\\ui})_{\\ui \\in [n]^r}$. Here, we use the notation $\\ui=(i_1,\\ldots,i_r)\\in [n]^r$ (or simply $i_1\\ldots i_r$) for an $r$-sequence in $[n]^r$. Given two integers $1\\le d\\le r$, it may also be convenient to denote  $r$-sequences as\nthe concatenation $\\uj \\uk $ of a $d$-sequence $\\uj\\in [n]^d$ and an $(r-d)$-sequence $\\uk\\in [n]^{r-d}$.\n\\tcolblue{For  vectors $u_i\\in \\K^{n_i}$, $i=1,2$, their tensor product is the vector $u_1\\ot u_2\\in\\K^{n_1}\\ot \\K^{n_2}$ with entries\n$(u_1\\ot u_2)_{i_1i_2}=(u_ 1)_{i_1}(u_2)_{i_2}$ for $(i_1,i_2)\\in[n_1]\\times [n_2]$. For $u\\in\\K^n$, $u^{\\ot r} = u\\ot \\cdots \\ot u$ (taking the tensor product of $r$ copies).}\n\n","start":27264,"end":28070}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":["eqPir"],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"对称张量投影","source_units":[{"raw_tex":" We now recall several notions of symmetry for tensors. \\tcolblue{In Appendix \\ref{appendix-example} we illustrate some   of these notions on examples.}\n \nPermutations $\\sigma\\in \\MFS_r$ act on $[n]^r$ and thus on $r$-tensors, in the following way.\nFor $\\ui\\in [n]^r$, set $\\ui^\\sigma=(i_{\\sigma(1)},\\ldots, i_{\\sigma(r)})$ and, for  $v\\in (\\K^n)^{\\ot r}$, define $v^\\sigma$ as the $r$-tensor with entries $(v^\\sigma)_{\\ui}= v_{\\ui^\\sigma}$ for $\\ui\\in [n]^r$.\nThen, $v$ is said to be a {\\em symmetric tensor} if $v^\\sigma=v$ for all $\\sigma\\in \\MFS_r$, and $S^r(\\K^n)$ denotes the space of symmetric $r$-tensors on $\\K^n$. We let $\\Pi_r$ denote the projection from the space $(\\K^n)^{\\ot r}$ to the symmetric subspace $S^r(\\K^n)$. Concretely, we have \n\\begin{align}\\label{eqPir}\n\\Pi_r=\\frac{1}{r!}\\sum_{\\sigma\\in \\MFS_r} P_\\sigma,\n\\end{align}\nwhere $P_\\sigma$ is the square matrix indexed by $[n]^r\\times [n]^r$, with entries \n$(P_\\sigma)_{\\ui, \\uj}=1$ if $\\uj=\\ui^\\sigma$ and 0 otherwise, for $\\ui,\\uj\\in [n]^r$.\n\n","start":28070,"end":29086}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["defdoublesymmetric"],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"矩阵双对称性","source_units":[{"raw_tex":"Consider a  matrix $M$ indexed by the set $[n]^r\\times [n]^r$, so that $M$ represents an endomorphism from $(\\K^n)^{\\ot r}$ to $(\\K^n)^{\\ot r}$. Then, any permutation $\\sigma\\in \\MFS_r$  induces naturally a permutation of  the columns (and rows) of $M$. \n\n\\begin{definition}[Doubly symmetric matrix]\\label{defdoublesymmetric}\nA matrix $M\\in \\K^{[n]^r\\times [n]^r}$  is called  {\\em $r$-column symmetric} (resp., {\\em $r$-row symmetric}) if $M\\Pi_r =M$ (resp., $\\Pi_rM=M$). If $M$ is complex Hermitian or real symmetric, then $M$ is $r$-column symmetric if and only if it is $r$-row symmetric, or equivalently if $\\Pi_rM\\Pi_r=M$, in which case  $M$ is called {\\em $ r $-doubly symmetric} (or just {\\em doubly symmetric} if $r$ is clear from the context).\n\\end{definition}\n\n\nHermitian doubly symmetric matrices are also known as {\\em Bose symmetric} matrices in the quantum information literature. \\MoL{The following stronger symmetry property is sometimes needed (it is considered, e.g.,   in \\cite{Doherty_Wehner_2013}).}\n\n","start":29086,"end":30109}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":["defmaximallysymmetric"],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"最大对称性","source_units":[{"raw_tex":"\\begin{definition}[Maximally symmetric matrix]\\label{defmaximallysymmetric}\nA  matrix $M\\in \\K^{[n]^r\\times [n]^r}$ can be   viewed as  a $2r$-tensor $\\vvec(M)\\in (\\K^n)^{\\ot 2r}$, with entries  $\\vvec(M)_{\\ui\\uj} = M_{\\ui,\\uj}$ for $\\ui,\\uj\\in [n]^r$.\nThen, $M$ is said to be {\\em maximally symmetric} when $\\vvec(M)$ is a symmetric $2r$-tensor.\n\\end{definition}\n\n","start":30109,"end":30474}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"最大对称化投影","source_units":[{"raw_tex":"\\tcolred{Note that a Hermitian matrix that is maximally symmetric is in fact real-valued (see also Lemma \\ref{lem:ppt-sym-is-real-rank-2} below).} Let $\\PiMS$ denote the projection from the space $\\K^{[n]^r\\times [n]^r}$ onto $\\MaxSym_r(\\K^n)$, the  space of maximally symmetric matrices. Concretely, the projection $\\PiMS(M)$ is obtained by first constructing the symmetric $2r$-tensor $\\Pi_{2r}(\\vvec(M))$ and then reshaping it as a square matrix indexed by $[n]^r$ (by splitting the $2r$ indices into two classes of size $r$, which can be done   arbitrarily  as the tensor is $2r$-symmetric).\n","start":30474,"end":31070}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"最大对称矩阵的实性","source_units":[{"raw_tex":"\\tcolred{Note that a Hermitian matrix that is maximally symmetric is in fact real-valued (see also Lemma \\ref{lem:ppt-sym-is-real-rank-2} below).} Let $\\PiMS$ denote the projection from the space $\\K^{[n]^r\\times [n]^r}$ onto $\\MaxSym_r(\\K^n)$, the  space of maximally symmetric matrices. Concretely, the projection $\\PiMS(M)$ is obtained by first constructing the symmetric $2r$-tensor $\\Pi_{2r}(\\vvec(M))$ and then reshaping it as a square matrix indexed by $[n]^r$ (by splitting the $2r$ indices into two classes of size $r$, which can be done   arbitrarily  as the tensor is $2r$-symmetric).\n","start":30474,"end":31070}]},{"id":"S010","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"齐次多项式空间","source_units":[{"raw_tex":"Throughout, let $ R := \\R[x] = \\R[x_1,\\ldots,x_n] $ denote  the polynomial ring in variables $ x=(x_1,\\ldots,x_n) $. For an integer $d\\in\\N$,  $ R_d $ denotes its $ d $-th graded component, which consists of the homogeneous polynomials (aka forms) with degree $d$, and $R_{\\le d}=R_0\\oplus R_1\\oplus\\ldots\\oplus R_d$ consists of the polynomials with degree at most $d$. \nWe let $[x]_d= (x^\\alpha)_{\\alpha\\in \\N^n_d}$ denote the vector of all monomials with degree $d$.\n","start":34823,"end":35292}]},{"id":"S011","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"平方半径多项式","source_units":[{"raw_tex":"Throughout we use the notation  $$s=x_1^2+\\ldots +x_n^2.$$\n","start":35292,"end":35351}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"平方和多项式","source_units":[{"raw_tex":"A polynomial $f\\in R$ is a {\\em sum of squares} (abbreviated as sos) if $ f = p_1^2  + \\ldots + p_N^2 $ for some $ N\\in \\N $ and $ p_1,\\ldots,p_N\\in R$. Then, all $p_i$ lie in $R_d$ (resp., $R_{\\le d}$) if $f\\in R_{2d}$ (resp., $f\\in R_{\\le 2d}$). The following characterization is well-known \\tcolblue{(see \\cite{Choi-Lam-Reznick-1995,Powers-Wormann-1998})}.\n\n","start":35616,"end":35977}]},{"id":"S013","semantic_type":"lemma","kind":"lemma","labels":["lem:sos-equiv-gram-matrix"],"member_ids":["S013"],"declared_external":true,"declaration_text":["原先决条件明确允许B0027；原文420行将该标准刻画归于Choi-Lam-Reznick-1995及Powers-Wormann-1998。"],"source_only":false,"title":"平方和的Gram刻画","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:sos-equiv-gram-matrix}\nA polynomial $f$ is a sum of squares if and only if it admits a positive semidefinite representing matrix (then, often called a {\\em Gram matrix} of $f$).\n\\end{lemma}\n","start":35977,"end":36191}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":["defGramtensor"],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"张量指标表示矩阵","source_units":[{"raw_tex":"\\begin{definition} \\label{defGramtensor}\n\tLet $f\\in R_{2d}$. A symmetric matrix $G$ indexed by $[n]^d$ is called a {\\em tensor-indexed representing matrix} of  $ f $ if the following polynomial identity holds:\n\t\\begin{align*}\n\t\tf = (x^{\\otimes d})^{T}G x^{\\otimes d} = \\langle G, (x x^{T})^{\\otimes d} \\rangle. \n\t\\end{align*}\n\\end{definition}\n\t\n","start":39095,"end":39440}]},{"id":"S015","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"唯一最大对称表示","source_units":[{"raw_tex":"\tThen, any polynomial  $f\\in R_{2d}$ admits  a (unique) tensor-indexed representing matrix  that is maximally symmetric, denoted by $\\MaxSym(f)$. Namely, we have $\\MaxSym(f)=\\PiMS(G)$, where $G$ is any tensor-indexed representing matrix of $f$.\n","start":39440,"end":39685}]},{"id":"S016","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S016"],"declared_external":true,"declaration_text":["原先决条件明确允许469–471行的张量指标Gram刻画；471行明确援引B0027的对应结论。"],"source_only":false,"title":"张量Gram平方和刻画","source_units":[{"raw_tex":"Moreover,\nin analogy with \\Cref{lem:sos-equiv-gram-matrix}, a polynomial $ f $ is a sum of squares if and only if it has a tensor-indexed positive semidefinite representing matrix. \n","start":39685,"end":39867}]},{"id":"S017","semantic_type":"definition","kind":"definition","labels":["eqQpM"],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"谱层级的参考矩阵","source_units":[{"raw_tex":"\\MoL{\\johnstonlovitz{} \\cite{Johnston_Lovitz_Vijayaraghavan_2023} proposed recently a hierarchy of spectral bounds $\\spec_r(p)$ for the minimum value $p_{\\min}$ of a homogeneous polynomial $p$ over the unit sphere,\\footnote{ Their hierarchy  applies more generally to optimization of  a multivariate tensor over a spherical Segre-Veronese variety. We only focus here on their hierarchy for polynomials on the sphere.} \nsee Section \\ref{secJLV-intro} below for their definition and relationship to the  moment-sos bounds.  Throughout, let $p\\in R_{2d}$, $s=x_1^2+\\cdots +x_n^2$, and let \n\\begin{align}\\label{eqQpM}\n\tQ(p) = \\maxsym(p), \\quad  M = \\maxsym(s^d) \n\\end{align}\ndenote  the maximally symmetric  matrices representing the polynomials $ p $ and $s^d$, respectively (as in Definition \\ref{defGramtensor}). They show the following convergence rate.\n}\n","start":69899,"end":70755}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":["eqQrpM"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"高阶提升矩阵","source_units":[{"raw_tex":"We introduce the spectral hierarchy of \\johnstonlovitz{} \\cite{Johnston_Lovitz_Vijayaraghavan_2023}. \nLet $p\\in R_{2d}$ and recall the matrices $Q(p)$   and $M$ from (\\ref{eqQpM}). For an integer $ r\\ge d $, define the matrices \n\\begin{align}\\label{eqQrpM}\n\tQ_r(p) = \\Pi_{r}(Q(p) \\otimes I_n^{\\otimes (r-d)})\\Pi_{r}, \\\\\n\tM_r= \\Pi_{r}(M \\otimes I_n^{\\otimes (r-d)})\\Pi_{r}. \n\\end{align}\n","start":72976,"end":73362}]},{"id":"S019","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"提升矩阵所表示的形式","source_units":[{"raw_tex":"So,  $ Q_r(p) $ and $M_r$ are   $ r $-doubly symmetric matrices    representing  $ s^{r-d}p $ and $s^r$, respectively. That is, $\\langle Q_r(p), x^{\\ot r}(x^{\\ot r})^T\\rangle = s^{r-d}p$ and $\n\\langle M_r, x^{\\ot r}(x^{\\ot r})^T\\rangle= s^r$ hold.\nThen, the  spectral bound  $\\spec_r(p)$ at the $ r $-th level is defined as the smallest generalized eigenvalue of the matrix $ Q_r(p) $ with respect to $ M_r $. We gather some equivalent formulations for $\\spec_r(p)$. \n\n","start":73362,"end":73831}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":["eqnu1","eqnu2","eqnu3"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"谱界的等价定义","source_units":[{"raw_tex":"\\begin{definition}\n\tThe spectral bound $\\spec_r(p)$  is defined by any of the following equivalent programs:\n\t\\begin{align}\n\t\t\\spec_r(p)& =\\max_{\\nu\\in\\R} \\{ \\nu: Q_r(p)-\\nu M_r\\succeq 0\\},\\label{eqnu1} \\\\\n\t\t& = \\min_{v\\in S^r(\\R^n) } \\{v^T Q_r(p)v: v^T M_rv=1\\}, \\label{eqnu2}\\\\\n\t\t& = \\min_{X } \\{\\langle Q_r(p), X\\rangle: X \\  r\\text{-doubly symmetric},\\ X\\succeq 0,\\ \\langle M_r,X\\rangle =1\\}.\\label{eqnu3}\n\t\\end{align}\n\\end{definition}\n","start":73831,"end":74271}]},{"id":"S021","semantic_type":"definition","kind":"definition","labels":["eqpCL"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"Choi–Lam四次形式","source_units":[{"raw_tex":"We start by picking a  quartic form  in four variables that is nonnegative but not a sum of squares.\nA concrete example is given by the \\emph{Choi-Lam form} $ p_{CL} $, \n\\begin{align}\\label{eqpCL}\n\tp_{CL}(x) = x_1^2x_2^2+x_1^2x_3^2+ x_2^2x_3^2 + x_4^4 -4 x_1x_2x_3x_4.\n\\end{align}\n","start":85861,"end":86142}]},{"id":"S022","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S022"],"declared_external":true,"declaration_text":["原先决条件Choi_Lam_1977；1047行明确引用该文的非负性结论。"],"source_only":false,"title":"Choi–Lam形式非负","source_units":[{"raw_tex":"The Choi-Lam form $ p_{CL}$ is famously known to be a nonnegative polynomial, which is not a sum of squares (see \\cite{Choi_Lam_1977}). Hence, {\\em no} matrix representation of $ p_{CL} $ is positive semidefinite.\n","start":86426,"end":86640}]},{"id":"S023","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S023"],"declared_external":true,"declaration_text":["原先决条件Choi_Lam_1977；1047行明确引用该文的非平方和结论。"],"source_only":false,"title":"Choi–Lam形式非平方和","source_units":[{"raw_tex":"The Choi-Lam form $ p_{CL}$ is famously known to be a nonnegative polynomial, which is not a sum of squares (see \\cite{Choi_Lam_1977}). Hence, {\\em no} matrix representation of $ p_{CL} $ is positive semidefinite.\n","start":86426,"end":86640}]},{"id":"S024","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"不存在半正定表示","source_units":[{"raw_tex":"The Choi-Lam form $ p_{CL}$ is famously known to be a nonnegative polynomial, which is not a sum of squares (see \\cite{Choi_Lam_1977}). Hence, {\\em no} matrix representation of $ p_{CL} $ is positive semidefinite.\n","start":86426,"end":86640}]},{"id":"S026","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"加入第五个变量","source_units":[{"raw_tex":"We use one small trick, which is to choose an unnatural representation of   $ p_{CL} $. While $ p_{CL} $ is intrinsically a polynomial in four variables $x=(x_1,x_2,x_3,x_4)$, one can artificially view it as a polynomial $p$ in five variables $ \\underline{x} := (x_1,x_2,x_3,x_4,u) $, where $u$ is an additional variable, by setting $p(\\underline x)=p_{CL}(x)$.\n","start":86902,"end":87264}]},{"id":"S027","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqmatQ"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"表示矩阵的补零嵌入","source_units":[{"raw_tex":"If, as a 4-variate polynomial,  $ p_{CL} $ is represented by a matrix $ Q^{(4)} \\in \\R^{[4]^2\\times [4]^2} $, then, as a 5-variate polynomial in   $ \\underline{x} = (x,u) $, it is represented by the matrix \n\\begin{align}\\label{eqmatQ}\n\t\\begin{pmatrix}\n\t\tQ^{(4)} & 0\\\\\n\t\t0^{T} & 0\n\t\\end{pmatrix} \\in  \\R^{[5]^2\\times [5]^2}.\n\\end{align}\n","start":87264,"end":87600}]},{"id":"S028","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqmatMQ"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"最大对称表示的补零结构","source_units":[{"raw_tex":"Note that the same holds   for the maximally symmetric representations of $p_{CL}$  in four variables and of $p $ as a polynomial in five variables: \n\\begin{align}\\label{eqmatMQ}\n\t\\maxsym(p)  = \\begin{pmatrix}\n\t\t\\maxsym(p_{CL}) & 0\\\\\n\t\t0^{T} & 0\n\t\\end{pmatrix}.\n\\end{align}\nThe reason is again that $ p $ is only supported on monomials, which do not depend on the fifth variable $ u $. \n","start":87600,"end":87987}]},{"id":"S029","semantic_type":"theorem","kind":"theorem","labels":["thm:lower-bound-spectral"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"谱层级的二次阶下界","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm:lower-bound-spectral}\n\tLet $\\spec_r(p)$ denote the spectral bound of order $r$ for the Choi-Lam form $p \\in \\R[x_1,x_2,x_3,x_4,u]$. Then, we have\t$p_{\\min}=0$ and $\\spec_r(p)=\\Omega(1/r^2)$.\n\\end{theorem}","start":87988,"end":88218}]},{"id":"S030","semantic_type":"proof","kind":"proof","labels":["eqmatH","eqmatH2","eqmats"],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"谱下界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\tLet the variables be ordered as $ \\underline{x} = (x,u)$, where $ x = (x_1,\\ldots,x_4) $ and  $ u $ is the fifth variable. Let $r\\ge 2$. Set $ s = x_1^2  + x_2^2+x_3^2 + x_4^2 + u^2 $ and \n\t\\begin{align}\\label{eqmats}\n\t\\begin{split}\n\t & Q :=   \\maxsym(p),\\quad  M := \\maxsym(s^2), \\\\\n\t &  Q_r := \\Pi_r (Q\\otimes I_5^{\\otimes (r-2)}) \\Pi_r,\\quad  M_r :=  \\Pi_r (M\\otimes I_5^{\\otimes (r-2)}) \\Pi_r.\n\t\\end{split}\n\t\\end{align} \n\tSo, $Q$ has the block-form as in (\\ref{eqmatQ}), (\\ref{eqmatMQ}), where $Q^{(4)} :=  \\MaxSym(p_{CL}) \\in \\R^{[4]^2 \\times [4]^2} $ and the remaining blocks are filled by zeros. As in (\\ref{eqPir}),  $\\Pi_2\\in \\R^{[5]^2\\times [5]^2}$ denotes the projection onto the symmetric subspace $S^2(\\R^5)$.\tThe space $\\R^{[4]^2}$ can be seen as a subspace of $\\R^{[5]^2}$ corresponding to the first four variable indices. Let us write $ \\Pi_2^{(4)} \\in \\R^{[4]^2 \\times [4]^2} $ for the restriction of $ \\Pi_2$ to the subspace $ \\R^{[4]^2} $. \n\n\tRecall  $\\spec_r=\\spec_r(p)$ for short. By the definition, we have $\\spec_r\\le p_{\\min}=0$ and \n\t\\begin{align*}\n\t\tQ_r + |\\spec_r| M_r = \\Pi_r ((Q+ |\\spec_r|M) \\otimes I_5^{\\otimes (r-2)}) \\Pi_r \\succeq 0.\n\t\\end{align*}\n\tBy using  $ M\\preceq \\lambda_{\\max}(M) I_5^{\\otimes 2} $, we obtain that \n\t\\begin{align*}\n\t\tG_r :=  \\Pi_r ((Q+ \\lambda_{\\max}(M)|\\spec_r|I_5^{\\otimes 2}) \\otimes I_5^{\\otimes (r-2)}) \\Pi_r \\succeq 0.\n\t\\end{align*}\n\tIn particular, every principal submatrix of $ G_r $ is positive semidefinite. We will now examine \\tcolblue{in detail} the following principal submatrix \n\t\\begin{align}\\label{eqmatH}\n\t\tH:= ((G_r)_{(i_1,i_2,5,\\ldots,5), (j_1,j_2,5,\\ldots,5)})_{i_1,i_2,j_1,j_2 \\in \\{1,\\ldots,4\\}} \\in \\R^{[4]^2\\times [4]^2}\n\t\\end{align}\n\tof $ G_r $, where the first two row indices $ i_1, i_2 $ and the first two column indices $ j_1,j_2 $ each range from $ 1 $ to $ 4 $, while the other $ r-2 $ row and column indices are set equal to $ 5 $.\n\t\n\n\tRecall the notation $ \\underline{i}^{\\sigma} := (i_{\\sigma(1)},\\ldots,i_{\\sigma(r)}) $ for any $\\sigma\\in \\MFS_r$. For $\\ui,\\uj\\in [5]^r$, using (\\ref{eqPir}), we may expand \n\t\\begin{align*}\n\t\t(G_r)_{\\underline{i},\\underline{j}} = \\left(\\frac{1}{r!}\\right)^2 \\sum_{\\sigma \\in \\mathfrak{S}_{r}}\\sum_{\\tau \\in \\mathfrak{S}_{r}}  (Q \\otimes I_5^{\\otimes (r-2)} + \\lambda_{\\max}(M)|\\spec_r|I_5^{\\otimes r})_{\\underline{i}^{\\sigma}, \\underline{j}^{\\tau}}.\n\t\\end{align*}\n\tIn order to compute the submatrix $ H $, we set the last $ r-2 $ indices of both $ \\underline{i} $ and $ \\underline{j} $ equal to $ 5 $ and the other two indices not equal to $ 5 $. Hence, on the right hand side, we have to count how many permutations ``survive'' when we do this, in the sense of giving a nonzero contribution to $ H $. \n\tWe may write \n\t\\begin{align}\\label{eqmatH2}\n\t H = A + \\lambda_{\\max}(M)|\\spec_r|B,\n\t \\end{align}\n\t  where $ A $ is the contribution arising from $ Q \\otimes I_5^{\\otimes (r-2)} $ and $ B $ is the contribution from $ I_5^{\\otimes r}  $. Since $ I_5^{\\otimes r}  $ is diagonal, the only nonzero entries of $ B $ are at indices of the form $ (i_1i_2,i_1i_2) $ and $ (i_1i_2,i_2i_1) $. \n\tFor later use, notice that $ (\\Pi_{2})_{i_1i_2,i_1i_2} = \\frac{1}{2} $ if $ i_1\\ne i_2 \\in [4]$, and $(\\Pi_{2})_{i_1i_1,i_1i_1} = 1 $ for $i_1\\in [4]$. \n\tWe now calculate the matrix $B$: for $i_1,i_2\\in [4]$, \n\t\\begin{align*}\n\t\tB_{i_1i_2,i_1i_2} &= B_{i_1i_2,i_2i_1} = \\frac{1}{(r!)^2} \\sum_{\\sigma \\in \\mathfrak{S}_{r}}\\sum_{\\tau \\in \\mathfrak{S}_{r}}  (I_5^{\\otimes r})_{(i_{1},i_{2},5,\\ldots,5)^{\\sigma}, (i_{1},i_{2},5,\\ldots,5)^{\\tau}}\\\\\n\t\t&= 2(\\Pi_{2})_{i_1i_2,i_1i_2} \\frac{(r-2)!}{(r!)^2} \\sum_{\\sigma \\in \\mathfrak{S}_{r}}  (I_5^{\\otimes r})_{(i_{1},i_{2},5,\\ldots,5)^{\\sigma}, (i_{1},i_{2},5,\\ldots,5)^{\\sigma}}\\\\\n\t\t&= 2(\\Pi_{2})_{i_1i_2,i_1i_2}\\frac{(r-2)!}{r!} =  \\binom{r}{2}^{-1} (\\Pi_2)_{i_1i_2,i_1i_2}. \n\t\\end{align*}\t\n\tHere, in the second line, we   used the fact that $ I_n^{\\otimes r} $ is diagonal and that the row and column indices are equal if and only if $ (i_{\\sigma(1)},i_{\\sigma(2)}) = (i_{\\tau(1)},i_{\\tau(2)}) $. \n\tGiven any permutation $ \\sigma \\in \\mathfrak{S}_r$, there are $ (r-2)! $ permutations $ \\tau \\in \\mathfrak{S}_r $ with this property if $ i_1\\ne i_2 $, and $ 2(r-2)! $ permutations $ \\tau $ with this property if $ i_1 = i_2 $. Hence, there are $ 2(r-2)!(\\Pi_{2})_{i_1i_2,i_1i_2} $ permutations $ \\tau $ with this property for any given $ \\sigma $. \n\tIn the third line, we used the fact that all diagonal entries of $ I_n^{\\otimes r} $ are equal to $ 1 $, so the sum evaluates to $ r! $.  \n\tSummarized, we conclude that $ B = \\binom{r}{2}^{-1} \\Pi_2^{(4)} $. \n\tNow, let us calculate $ A $: for $i_1,i_2,j_1,j_2\\in [4]$,\n\t\\begin{align*}\n\t\tA_{i_1i_2,j_1j_2} = \\frac{1}{(r!)^2} \\sum_{\\sigma \\in \\mathfrak{S}_{r}}\\sum_{\\tau \\in \\mathfrak{S}_{r}}  (Q\\otimes I_5^{\\otimes r-2})_{(i_{1},i_{2},5,\\ldots,5)^{\\sigma}, (j_{1},j_{2},5,\\ldots,5)^{\\tau}}.\n\t\\end{align*}\n\tRecall that $ Q_{i,5} = 0 $ if $ i\\in \\{1,\\ldots,4\\} $. Therefore, the only nonzero contributions come from permutations $ \\sigma $ and $ \\tau $, which keep all 5-indices `to the right', i.e., at the last $r-2$ positions. There are $ 2(r-2)! $ permutations $ \\sigma $ and $ 2(r-2)! $ permutations $ \\tau $ with this property. They may be written as $ \\sigma = \\sigma_2\\circ \\sigma_{r-2} $ and $ \\tau =  \\tau_2\\circ \\tau_{r-2}$, respectively, with $ \\sigma_2, \\tau_2 \\in \\mathfrak{S}_2 $ and $ \\sigma_{r-2}, \\tau_{r-2} \\in \\mathfrak{S}_{r-2} $.\n\tSo, we obtain  \n\t\\begin{align*}\n\t\tA_{i_1i_2,j_1j_2} &= \\frac{((r-2)!)^2}{(r!)^2} \\sum_{\\sigma_2, \\tau_2 \\in \\mathfrak{S}_{2}} (Q\\otimes I_n^{\\otimes r-2})_{(i_{\\sigma_2(1)},i_{\\sigma_2(2)},5,\\ldots,5), (j_{\\tau_2(1)},j_{\\tau_2(2)},5,\\ldots,5)}.\\\\\n\t\t&= \\left(\\frac{(r-2)!}{r!}\\right)^2 \\sum_{\\sigma_2, \\tau_2 \\in \\mathfrak{S}_{2}} Q_{i_{\\sigma_2(1)}i_{\\sigma_2(2)}, j_{\\tau_2(1)}j_{\\tau_2(2)}}.\\\\\n\t\t&= \\binom{r}{2}^{-2} Q_{i_1i_2,j_1j_2}.\n\t\\end{align*}\n\tIn the first line, we used the fact that the $ (r-2)!^2 $ permutations of the 5-indices all yield the same indices. In the second line, we used that $ (I_n^{\\otimes r-2})_{5\\ldots5,5\\ldots5} = 1 $. In the third line, we used that $ Q $ is maximally symmetric, hence in particular doubly symmetric. Thus, the $ 2^2 = 4 $ permutations of the non-5 indices all yield the same entry. Summarizing, we have shown  \n\t$$ A = \\binom{r}{2}^{-2}Q^{(4)},\\quad  B = \\binom{r}{2}^{-1}\\Pi_{2}^{(4)}.\n\t$$\n\t We conclude that the   matrix $ H $ from (\\ref{eqmatH})-(\\ref{eqmatH2}) can be expressed as\n\t\\begin{align*}\n\t\t\\binom{r}{2}^{2}  H = Q^{(4)} + \\lambda_{\\max}(M)|\\spec_r|\\binom{r}{2}\\Pi_{2}^{(4)}. \n\t\\end{align*}\n\tNow, we use that $H\\succeq 0$ and $\\begin{pmatrix} \\Pi_2^{(4)} & 0 \\\\ 0^{T} & 0 \\end{pmatrix} \\preceq I_5^{\\otimes 2} \\preceq \\frac{1}{\\lambda_{\\min}(M)} M $ to obtain  \n\t\\begin{align*}\n\t\t0\\preceq \\binom{r}{2}^{2} \\begin{pmatrix} H & 0 \\\\ 0^{T} & 0 \\end{pmatrix}  \\preceq Q + |\\spec_r|\\frac{\\lambda_{\\max}(M)}{\\lambda_{\\min}(M)} \\binom{r}{2} M. \n\t\\end{align*}\n\tBy definition, $ |\\spec_2| $ is the smallest value such that \n\t\\begin{align*}\n\t\tQ + |\\spec_2| M \\succeq 0. \n\t\\end{align*}\n\tHence, it follows that \n\t\\begin{align*}\n\t\t |\\spec_r|\\frac{\\lambda_{\\max}(M)}{\\lambda_{\\min}(M)} \\binom{r}{2} \\ge |\\spec_2|, \\:\\: \\text{ and thus }\\:\\: \t|\\spec_r| \\ge \\frac{\\lambda_{\\min}(M)}{\\lambda_{\\max}(M)} |\\spec_2|\\binom{r}{2}^{-1} = \\Omega\\left(\\frac{1}{r^2}\\right),\t\t \n\t\\end{align*}\n\\tcolblue{using the fact that $M\\succ 0$ \\cite{Johnston_Lovitz_Vijayaraghavan_2023} and $|\\spec_2|>0$ as $p$ is not sos.} \n\\end{proof}","start":88220,"end":95758}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":["eqmats"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"固定五元矩阵与参数","source_units":[{"raw_tex":"\\begin{proof}\n\tLet the variables be ordered as $ \\underline{x} = (x,u)$, where $ x = (x_1,\\ldots,x_4) $ and  $ u $ is the fifth variable. Let $r\\ge 2$. Set $ s = x_1^2  + x_2^2+x_3^2 + x_4^2 + u^2 $ and \n\t\\begin{align}\\label{eqmats}\n\t\\begin{split}\n\t & Q :=   \\maxsym(p),\\quad  M := \\maxsym(s^2), \\\\\n\t &  Q_r := \\Pi_r (Q\\otimes I_5^{\\otimes (r-2)}) \\Pi_r,\\quad  M_r :=  \\Pi_r (M\\otimes I_5^{\\otimes (r-2)}) \\Pi_r.\n\t\\end{split}\n\t\\end{align} \n\tSo, $Q$ has the block-form as in (\\ref{eqmatQ}), (\\ref{eqmatMQ}), where $Q^{(4)} :=  \\MaxSym(p_{CL}) \\in \\R^{[4]^2 \\times [4]^2} $ and the remaining blocks are filled by zeros. As in (\\ref{eqPir}),  $\\Pi_2\\in \\R^{[5]^2\\times [5]^2}$ denotes the projection onto the symmetric subspace $S^2(\\R^5)$.\tThe space $\\R^{[4]^2}$ can be seen as a subspace of $\\R^{[5]^2}$ corresponding to the first four variable indices. Let us write $ \\Pi_2^{(4)} \\in \\R^{[4]^2 \\times [4]^2} $ for the restriction of $ \\Pi_2$ to the subspace $ \\R^{[4]^2} $. \n\n","start":88220,"end":89196}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"实例最小值为零","source_units":[{"raw_tex":"\tRecall  $\\spec_r=\\spec_r(p)$ for short. By the definition, we have $\\spec_r\\le p_{\\min}=0$ and \n","start":89196,"end":89293}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"谱界为非正数","source_units":[{"raw_tex":"\tRecall  $\\spec_r=\\spec_r(p)$ for short. By the definition, we have $\\spec_r\\le p_{\\min}=0$ and \n","start":89196,"end":89293}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"最优谱矩阵铅笔半正定","source_units":[{"raw_tex":"\tRecall  $\\spec_r=\\spec_r(p)$ for short. By the definition, we have $\\spec_r\\le p_{\\min}=0$ and \n\t\\begin{align*}\n\t\tQ_r + |\\spec_r| M_r = \\Pi_r ((Q+ |\\spec_r|M) \\otimes I_5^{\\otimes (r-2)}) \\Pi_r \\succeq 0.\n\t\\end{align*}\n","start":89196,"end":89416}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"构造放大的半正定矩阵","source_units":[{"raw_tex":"\tBy using  $ M\\preceq \\lambda_{\\max}(M) I_5^{\\otimes 2} $, we obtain that \n\t\\begin{align*}\n\t\tG_r :=  \\Pi_r ((Q+ \\lambda_{\\max}(M)|\\spec_r|I_5^{\\otimes 2}) \\otimes I_5^{\\otimes (r-2)}) \\Pi_r \\succeq 0.\n\t\\end{align*}\n","start":89416,"end":89631}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":["eqmatH"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"选取特殊主子矩阵","source_units":[{"raw_tex":"\tIn particular, every principal submatrix of $ G_r $ is positive semidefinite. We will now examine \\tcolblue{in detail} the following principal submatrix \n\t\\begin{align}\\label{eqmatH}\n\t\tH:= ((G_r)_{(i_1,i_2,5,\\ldots,5), (j_1,j_2,5,\\ldots,5)})_{i_1,i_2,j_1,j_2 \\in \\{1,\\ldots,4\\}} \\in \\R^{[4]^2\\times [4]^2}\n\t\\end{align}\n\tof $ G_r $, where the first two row indices $ i_1, i_2 $ and the first two column indices $ j_1,j_2 $ each range from $ 1 $ to $ 4 $, while the other $ r-2 $ row and column indices are set equal to $ 5 $.\n\t\n","start":89631,"end":90159}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"展开两侧置换平均","source_units":[{"raw_tex":"\n\tRecall the notation $ \\underline{i}^{\\sigma} := (i_{\\sigma(1)},\\ldots,i_{\\sigma(r)}) $ for any $\\sigma\\in \\MFS_r$. For $\\ui,\\uj\\in [5]^r$, using (\\ref{eqPir}), we may expand \n\t\\begin{align*}\n\t\t(G_r)_{\\underline{i},\\underline{j}} = \\left(\\frac{1}{r!}\\right)^2 \\sum_{\\sigma \\in \\mathfrak{S}_{r}}\\sum_{\\tau \\in \\mathfrak{S}_{r}}  (Q \\otimes I_5^{\\otimes (r-2)} + \\lambda_{\\max}(M)|\\spec_r|I_5^{\\otimes r})_{\\underline{i}^{\\sigma}, \\underline{j}^{\\tau}}.\n\t\\end{align*}\n\tIn order to compute the submatrix $ H $, we set the last $ r-2 $ indices of both $ \\underline{i} $ and $ \\underline{j} $ equal to $ 5 $ and the other two indices not equal to $ 5 $. Hence, on the right hand side, we have to count how many permutations ``survive'' when we do this, in the sense of giving a nonzero contribution to $ H $. \n","start":90159,"end":90965}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":["eqmatH2"],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"拆分两类矩阵贡献","source_units":[{"raw_tex":"\tWe may write \n\t\\begin{align}\\label{eqmatH2}\n\t H = A + \\lambda_{\\max}(M)|\\spec_r|B,\n\t \\end{align}\n\t  where $ A $ is the contribution arising from $ Q \\otimes I_5^{\\otimes (r-2)} $ and $ B $ is the contribution from $ I_5^{\\otimes r}  $. Since $ I_5^{\\otimes r}  $ is diagonal, the only nonzero entries of $ B $ are at indices of the form $ (i_1i_2,i_1i_2) $ and $ (i_1i_2,i_2i_1) $. \n","start":90965,"end":91349}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"二阶投影的对角元","source_units":[{"raw_tex":"\tFor later use, notice that $ (\\Pi_{2})_{i_1i_2,i_1i_2} = \\frac{1}{2} $ if $ i_1\\ne i_2 \\in [4]$, and $(\\Pi_{2})_{i_1i_1,i_1i_1} = 1 $ for $i_1\\in [4]$. \n","start":91349,"end":91503}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"计算单位阵贡献","source_units":[{"raw_tex":"\tWe now calculate the matrix $B$: for $i_1,i_2\\in [4]$, \n\t\\begin{align*}\n\t\tB_{i_1i_2,i_1i_2} &= B_{i_1i_2,i_2i_1} = \\frac{1}{(r!)^2} \\sum_{\\sigma \\in \\mathfrak{S}_{r}}\\sum_{\\tau \\in \\mathfrak{S}_{r}}  (I_5^{\\otimes r})_{(i_{1},i_{2},5,\\ldots,5)^{\\sigma}, (i_{1},i_{2},5,\\ldots,5)^{\\tau}}\\\\\n\t\t&= 2(\\Pi_{2})_{i_1i_2,i_1i_2} \\frac{(r-2)!}{(r!)^2} \\sum_{\\sigma \\in \\mathfrak{S}_{r}}  (I_5^{\\otimes r})_{(i_{1},i_{2},5,\\ldots,5)^{\\sigma}, (i_{1},i_{2},5,\\ldots,5)^{\\sigma}}\\\\\n\t\t&= 2(\\Pi_{2})_{i_1i_2,i_1i_2}\\frac{(r-2)!}{r!} =  \\binom{r}{2}^{-1} (\\Pi_2)_{i_1i_2,i_1i_2}. \n\t\\end{align*}\t\n\tHere, in the second line, we   used the fact that $ I_n^{\\otimes r} $ is diagonal and that the row and column indices are equal if and only if $ (i_{\\sigma(1)},i_{\\sigma(2)}) = (i_{\\tau(1)},i_{\\tau(2)}) $. \n\tGiven any permutation $ \\sigma \\in \\mathfrak{S}_r$, there are $ (r-2)! $ permutations $ \\tau \\in \\mathfrak{S}_r $ with this property if $ i_1\\ne i_2 $, and $ 2(r-2)! $ permutations $ \\tau $ with this property if $ i_1 = i_2 $. Hence, there are $ 2(r-2)!(\\Pi_{2})_{i_1i_2,i_1i_2} $ permutations $ \\tau $ with this property for any given $ \\sigma $. \n\tIn the third line, we used the fact that all diagonal entries of $ I_n^{\\otimes r} $ are equal to $ 1 $, so the sum evaluates to $ r! $.  \n","start":91503,"end":92783}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"得到B的投影公式","source_units":[{"raw_tex":"\tSummarized, we conclude that $ B = \\binom{r}{2}^{-1} \\Pi_2^{(4)} $. \n","start":92783,"end":92853}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"筛选Q贡献的置换","source_units":[{"raw_tex":"\tNow, let us calculate $ A $: for $i_1,i_2,j_1,j_2\\in [4]$,\n\t\\begin{align*}\n\t\tA_{i_1i_2,j_1j_2} = \\frac{1}{(r!)^2} \\sum_{\\sigma \\in \\mathfrak{S}_{r}}\\sum_{\\tau \\in \\mathfrak{S}_{r}}  (Q\\otimes I_5^{\\otimes r-2})_{(i_{1},i_{2},5,\\ldots,5)^{\\sigma}, (j_{1},j_{2},5,\\ldots,5)^{\\tau}}.\n\t\\end{align*}\n\tRecall that $ Q_{i,5} = 0 $ if $ i\\in \\{1,\\ldots,4\\} $. Therefore, the only nonzero contributions come from permutations $ \\sigma $ and $ \\tau $, which keep all 5-indices `to the right', i.e., at the last $r-2$ positions. There are $ 2(r-2)! $ permutations $ \\sigma $ and $ 2(r-2)! $ permutations $ \\tau $ with this property. They may be written as $ \\sigma = \\sigma_2\\circ \\sigma_{r-2} $ and $ \\tau =  \\tau_2\\circ \\tau_{r-2}$, respectively, with $ \\sigma_2, \\tau_2 \\in \\mathfrak{S}_2 $ and $ \\sigma_{r-2}, \\tau_{r-2} \\in \\mathfrak{S}_{r-2} $.\n","start":92853,"end":93694}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"计算Q贡献系数","source_units":[{"raw_tex":"\tSo, we obtain  \n\t\\begin{align*}\n\t\tA_{i_1i_2,j_1j_2} &= \\frac{((r-2)!)^2}{(r!)^2} \\sum_{\\sigma_2, \\tau_2 \\in \\mathfrak{S}_{2}} (Q\\otimes I_n^{\\otimes r-2})_{(i_{\\sigma_2(1)},i_{\\sigma_2(2)},5,\\ldots,5), (j_{\\tau_2(1)},j_{\\tau_2(2)},5,\\ldots,5)}.\\\\\n\t\t&= \\left(\\frac{(r-2)!}{r!}\\right)^2 \\sum_{\\sigma_2, \\tau_2 \\in \\mathfrak{S}_{2}} Q_{i_{\\sigma_2(1)}i_{\\sigma_2(2)}, j_{\\tau_2(1)}j_{\\tau_2(2)}}.\\\\\n\t\t&= \\binom{r}{2}^{-2} Q_{i_1i_2,j_1j_2}.\n\t\\end{align*}\n\tIn the first line, we used the fact that the $ (r-2)!^2 $ permutations of the 5-indices all yield the same indices. In the second line, we used that $ (I_n^{\\otimes r-2})_{5\\ldots5,5\\ldots5} = 1 $. In the third line, we used that $ Q $ is maximally symmetric, hence in particular doubly symmetric. Thus, the $ 2^2 = 4 $ permutations of the non-5 indices all yield the same entry. Summarizing, we have shown  \n","start":93694,"end":94557}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"汇总两类贡献","source_units":[{"raw_tex":"\t$$ A = \\binom{r}{2}^{-2}Q^{(4)},\\quad  B = \\binom{r}{2}^{-1}\\Pi_{2}^{(4)}.\n\t$$\n","start":94557,"end":94637}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"得到缩放主子矩阵公式","source_units":[{"raw_tex":"\t We conclude that the   matrix $ H $ from (\\ref{eqmatH})-(\\ref{eqmatH2}) can be expressed as\n\t\\begin{align*}\n\t\t\\binom{r}{2}^{2}  H = Q^{(4)} + \\lambda_{\\max}(M)|\\spec_r|\\binom{r}{2}\\Pi_{2}^{(4)}. \n\t\\end{align*}\n","start":94637,"end":94849}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"嵌入并比较参考矩阵","source_units":[{"raw_tex":"\tNow, we use that $H\\succeq 0$ and $\\begin{pmatrix} \\Pi_2^{(4)} & 0 \\\\ 0^{T} & 0 \\end{pmatrix} \\preceq I_5^{\\otimes 2} \\preceq \\frac{1}{\\lambda_{\\min}(M)} M $ to obtain  \n\t\\begin{align*}\n\t\t0\\preceq \\binom{r}{2}^{2} \\begin{pmatrix} H & 0 \\\\ 0^{T} & 0 \\end{pmatrix}  \\preceq Q + |\\spec_r|\\frac{\\lambda_{\\max}(M)}{\\lambda_{\\min}(M)} \\binom{r}{2} M. \n\t\\end{align*}\n","start":94849,"end":95210}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"调用二阶谱阈值","source_units":[{"raw_tex":"\tBy definition, $ |\\spec_2| $ is the smallest value such that \n\t\\begin{align*}\n\t\tQ + |\\spec_2| M \\succeq 0. \n\t\\end{align*}\n","start":95210,"end":95333}]},{"id":"S048","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S048"],"declared_external":true,"declaration_text":["原先决条件Johnston_Lovitz_Vijayaraghavan_2023；1147行明确引用M≻0，但未注明所用命题编号或谱空间。"],"source_only":false,"title":"参考矩阵的严格正定性","source_units":[{"raw_tex":"\\tcolblue{using the fact that $M\\succ 0$ \\cite{Johnston_Lovitz_Vijayaraghavan_2023} and $|\\spec_2|>0$ as $p$ is not sos.} \n","start":95624,"end":95747}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"二阶谱误差严格为正","source_units":[{"raw_tex":"\\tcolblue{using the fact that $M\\succ 0$ \\cite{Johnston_Lovitz_Vijayaraghavan_2023} and $|\\spec_2|>0$ as $p$ is not sos.} \n","start":95624,"end":95747}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"推出绝对值二次阶下界","source_units":[{"raw_tex":"\tHence, it follows that \n\t\\begin{align*}\n\t\t |\\spec_r|\\frac{\\lambda_{\\max}(M)}{\\lambda_{\\min}(M)} \\binom{r}{2} \\ge |\\spec_2|, \\:\\: \\text{ and thus }\\:\\: \t|\\spec_r| \\ge \\frac{\\lambda_{\\min}(M)}{\\lambda_{\\max}(M)} |\\spec_2|\\binom{r}{2}^{-1} = \\Omega\\left(\\frac{1}{r^2}\\right),\t\t \n\t\\end{align*}\n\\tcolblue{using the fact that $M\\succ 0$ \\cite{Johnston_Lovitz_Vijayaraghavan_2023} and $|\\spec_2|>0$ as $p$ is not sos.} \n\\end{proof}","start":95333,"end":95758}]},{"id":"S051","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S051"],"declared_external":true,"declaration_text":["原先决条件明确允许：Standard finite-dimensional linear algebra，包括orthogonal projections、positive semidefinite order、principal submatrices、tensor products及eigenvalue bounds。"],"source_only":false,"title":"有限维半正定矩阵规则","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051"}}}}
{"record_id":"2403.19448v3","paper":{"title":"Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradients","authors":["Müller, Johannes","Çaycı, Semih","Montúfar, Guido"],"arxiv_id":"2403.19448","version":"v3","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/2403.19448v3","pdf_url":"https://arxiv.org/pdf/2403.19448v3","doi":"10.1137/24M1653422"},"context":{"raw_tex":"For a finite set $\\X$, we denote the \\emph{free vector space} over $\\X$ by $\\R^\\X = \\{ \\mu\\colon \\X \\to \\R \\}$. \nIts elements can be identified with vectors $(\\mu_x)_{x\\in\\X}$. \nSimilarly, we denote the vectors with non-negative entries and positive entries by $\\R_{\\ge0}^\\X$ and $\\R_{>0}^\\X$, respectively. \nFor two elements $\\mu, \\nu\\in\\R^\\X$ we denote the \\emph{Hadamard product}, i.e., the entrywise product, between $\\mu$ and $\\nu$ by $\\mu\\odot\\nu\\in\\R^\\X$, so that $\\mu\\odot\\nu(x) \\coloneqq \\mu(x)\\nu(x)$.\nThe \\emph{total variation} norm $\\lVert \\cdot \\rVert_{\\TV}\\colon\\R^\\X\\to\\R$ is given by $\\lVert \\mu \\rVert_{\\TV} \\coloneqq \\frac12\\sum_{x}\\lvert \\mu_x \\rvert$. \nFinally, with $\\mathds{1}_\\X\\in\\R^\\X$ we denote the all-one vector. \n\nA \\emph{polyhedron} is a set $P = \\{ \\mu\\in\\R^\\X : \\ell_i(\\mu) \\ge 0 \\text{ for } i=1, \\dots, k \\}\\subseteq \\R^\\X$, where $\\ell_i\\colon\\R^\\X\\to\\R$ are affine linear functions for $i=1, \\dots, k$. \nA bounded (and thus compact) polyhedron is called a \\emph{polytope}. \nA polytope can be shown to be the convex hull of finitely many extreme points, which are called \\emph{vertices} and which we denote by $\\operatorname{Vert}(P)$. \nTwo vertices $\\mu_1, \\mu_2\\in\\operatorname{Vert}(P)$ are called \\emph{neighbors} if the subspace $\\{ c\\in\\R^\\X : c^\\top \\mu_1 = c^\\top \\mu_2 = \\max_{\\mu\\in P} c^\\top \\mu \\}$ has dimension $\\lvert \\X \\rvert-1$. \nWe denote the set of all neighbors of a vertex $\\mu$ by $N(\\mu) \\subseteq\\operatorname{Vert}(P)$.  \nThe \\emph{affine space} $\\operatorname{aff\\,span}(P)$ of a polytope $P\\subseteq\\R^\\X$ is the smallest affine subspace of $\\R^\\X$ containing $P$. \nThe relative interior $\\operatorname{int}(P)$ and boundary $\\partial P$ of $P$ are the interior and boundary of $P$ in its affine hull. \nFinally, the \\emph{tangent space} $TP$ of $P$ is given by the linear part of $\\operatorname{aff\\,span}(P)$. \n\nWe call $\\Delta_\\X \\coloneqq \\left\\{ \\mu\\in\\R^\\X_{\\ge0} : \\sum_{x} \\mu_x=1 \\right\\}$ the \\emph{probability simplex}. \nWe say that $\\mu\\in\\Delta_\\X$ is absolutely continuous with respect to $\\nu\\in\\Delta_\\X$ if $\\nu(x) = 0$ implies $\\mu(x)=0$ and write $\\mu \\ll \\nu$. \n\n\nThe \\emph{Fisher-Rao metric} is a Riemannian metric on the positive orthant given by \n\\begin{equation}\n    g_{\\mu}^{\\FR}(v,w) \\coloneqq \\sum_{x\\in \\X} \\frac{v_x w_x}{\\mu_x} \\quad \\text{for all } v, w\\in\\R^{\\X}, \\mu\\in\\R^{\\X}_{>0},\n\\end{equation}\nwhere we denote the induced norm by $\\lVert v \\rVert_{g_\\mu^{\\FR}} \\coloneqq {g_{\\mu}^{\\FR}(v,v)}^{\\frac12}$. \n\n\nThe Fisher-Rao metric is closely connected to the negative {Shannon entropy} \n\\begin{equation}\n    \\phi(\\mu) = -H(\\mu) \\coloneqq \\sum_{x\\in\\X} \\mu_x \\log \\mu_x  \\quad \\text{for all } \\mu\\in\\R^{\\X}_{>0}\n\\end{equation}\nas it is induced by the Hessian of the (negative) entropy, meaning that we have $g_{\\mu}^{\\FR}(v,w) = v^\\top \\nabla^2 \\phi(\\mu) w$ for all $v, w\\in\\R^{\\X}, \\mu\\in\\R^{\\X}_{>0}$. \nAs such, the Fisher-Rao metric falls into the class of \\emph{Hessian metrics} that have been studied in convex optimization; we refer to~\\cite{alvarez2004hessian, mueller2023thesis} for general well-posedness and convergence results. \nAn important concept in the analysis of Hessian gradient flows is the \\emph{Bregman divergence} induced by $\\phi$, which in the case of the negative entropy is given by the \\emph{KL-divergence} \n\\begin{equation}\n    \\KL (\\mu, \\nu) \\coloneqq  \\phi(\\mu) - \\phi(\\nu) - \\nabla \\phi(\\nu)(\\mu-\\nu) = \\sum_{x\\in \\X} \\mu_x \\log \\frac{\\mu_x}{\\nu_x} - \\sum_{x\\in \\X} \\mu_x + \\sum_{x\\in \\X} \\nu_x \n\\end{equation}\nfor $\\mu, \\nu\\in\\R_{\\ge0}^\\X$ with $\\mu\\ll\\nu$, where we use the common convention $0\\log\\frac00\\coloneqq0$. \n\nConsider now a continuously differentiable function $f\\colon\\R_{\\ge0}^\\X\\to\\R$ that we assume to be differentiable on $\\R_{>0}^\\X$ that we want to optimize over a polytope $P= \\R_{\\ge0}^{\\X}\\cap\\mathcal L$, where $\\mathcal L$ is a linear space. \nWe denote the gradient of $f\\colon \\R_{>0}^\\X\\to\\R$ at $\\mu\\in\\R_{>0}^\\X$ with respect to the Fisher-Rao metric by $\\nabla^{\\FR}f(\\mu)$ and call it the \\emph{Fisher-Rao gradient}. \nFurther, we denote the Fisher-Rao gradient of $f\\colon\\operatorname{int}(P)\\to\\R$ by $\\nabla^{\\FR}_P f(\\mu)\\in TP$, which is uniquely determined by \n\\begin{equation}\n    g_\\mu^{\\FR}(\\nabla^{\\FR}_P f(\\mu),v) = df(\\mu)v \\quad \\text{for all } v\\in TP.\n\\end{equation}\nNote that $\\nabla^{\\FR}_P f(\\mu)$ is the projection of $\\nabla^{\\FR}f(\\mu)$ with respect to the Fisher-Rao metric onto $TP$.\nBy examining the definition of the Fisher-Rao metric we see that this is equivalent to \n\\begin{equation}\n\\label{eq:characterizationFRGradient}\n    \\langle \\nabla^2\\phi(\\mu)\\nabla^{\\FR}_P  f(\\mu), v \\rangle = \\langle \\nabla f(\\mu), v \\rangle \\quad \\text{for all } v\\in TP. \n\\end{equation}\n    \nWe say that $(\\mu_t)_{t\\in [0, T)} \\subseteq \\operatorname{int}(P)$ solves the \\emph{Fisher-Rao gradient flow} if it solves the gradient flow with respect to the Fisher-Rao metric, i.e., if \n\\begin{equation}\\label{eq:FRGF}\n    \\partial_t \\mu_t = \\nabla^{\\FR}_P  f(\\mu_t) \\quad \\text{for all } t\\in[0, T). \n\\end{equation}\nBy using the characterization~\\eqref{eq:characterizationFRGradient} of $\\nabla^{\\FR}_P f(\\mu_t)$, we see that $(\\mu_t)_{t\\in [0, T)} \\subseteq \\operatorname{int}(P)$ solves the Fisher-Rao gradient flow~\\eqref{eq:FRGF}\nif and only if we have \n\\begin{equation}\\label{eq:explicitFRGF}\n    \\langle \\nabla^2 \\phi(\\mu_t) \\partial_t \\mu_t, v \\rangle = \\langle \\nabla f(\\mu_t), v \\rangle \n    \\quad \\text{for all } v\\in TP, t\\in[0, T). \n\\end{equation}\n\n\nIn the remainder, we study linear programs and work in the following setting. \n\n\\begin{setting}\\label{setting:generalLP}\nWe consider a finite set $\\X$ and a linear program \n    \\begin{equation}\\label{eq:LP}\n        \\max c^\\top \\mu \\quad \\text{subject to } \\mu\\in P, \n    \\end{equation}\nwith cost $c\\in\\R^{\\X}$ and feasible region $P = \\R^\\X_{\\ge0} \\cap  \\mathcal L$ with $P \\cap \\R^\\X_{>0}\\ne\\emptyset$, where $\\mathcal L\\subseteq\\R^{\\X}$ is an affine space. \nBy $(\\mu_t)_{t\\in[0, T)}\\subseteq\\operatorname{int}(P)$ we denote a solution of the Fisher-Rao gradient flow~\\eqref{eq:FRGF} with initial condition $\\mu_0\\in P \\cap \\R_{>0}^\\X$ and potential $f(\\mu) = c^\\top \\mu$, where $T\\in\\R_{\\ge0}\\cup\\{+\\infty\\}$. \n\\end{setting}\n\n\n\\section{Convergence of Fisher-Rao Gradient Flows}\\label{sec:linearConvergeFRGF}\nWe have seen that Fisher-Rao gradient flows converge globally at a sublinear rate $O(t^{-1})$. \nWe now build on this analysis and show that once the gradient flow enters a vicinity of the optimizer, it converges at a quasi-linear rate $O(t^\\kappa e^{-\\Delta t})$, where $\\Delta>0$ depends on the geometry of the linear program and $\\kappa>0$ depends on the initial condition $\\mu_0$. \nNote that this yields $O(e^{-c t})$ convergence for all $c<\\Delta$ and hence we also simply talk of a linear convergence rate.\nWe consider linear programs of the following form. \n\\begin{setting}\\label{setting:LP-inside-Simplex}\nWe consider a finite set $\\X$ and a linear program \n    \\begin{equation}\\label{eq:LP-inside-simplex}\n        \\max c^\\top \\mu \\quad \\text{subject to } \\mu\\in P, \n    \\end{equation}\nwith cost $c\\in\\R^{\\X}$ and feasible region $P = \\Delta_\\X \\cap  \\mathcal L$ with $P \\cap \\R^\\X_{>0}\\ne\\emptyset$, where $\\mathcal L\\subseteq\\R^{\\X}$ is an affine space. \nBy $(\\mu_t)_{t\\ge0}\\subseteq\\operatorname{int}(P)$ we denote the solution of the Fisher-Rao gradient flow~\\eqref{eq:FRGF} with initial condition $\\mu_0\\in P \\cap \\R_{>0}^\\X$ and the potential $f(\\mu) = c^\\top \\mu$. \n\\end{setting}\n","units":[{"raw_tex":"For a finite set $\\X$, we denote the \\emph{free vector space} over $\\X$ by $\\R^\\X = \\{ \\mu\\colon \\X \\to \\R \\}$. \nIts elements can be identified with vectors $(\\mu_x)_{x\\in\\X}$. \nSimilarly, we denote the vectors with non-negative entries and positive entries by $\\R_{\\ge0}^\\X$ and $\\R_{>0}^\\X$, respectively. \nFor two elements $\\mu, \\nu\\in\\R^\\X$ we denote the \\emph{Hadamard product}, i.e., the entrywise product, between $\\mu$ and $\\nu$ by $\\mu\\odot\\nu\\in\\R^\\X$, so that $\\mu\\odot\\nu(x) \\coloneqq \\mu(x)\\nu(x)$.\nThe \\emph{total variation} norm $\\lVert \\cdot \\rVert_{\\TV}\\colon\\R^\\X\\to\\R$ is given by $\\lVert \\mu \\rVert_{\\TV} \\coloneqq \\frac12\\sum_{x}\\lvert \\mu_x \\rvert$. \nFinally, with $\\mathds{1}_\\X\\in\\R^\\X$ we denote the all-one vector. \n\nA \\emph{polyhedron} is a set $P = \\{ \\mu\\in\\R^\\X : \\ell_i(\\mu) \\ge 0 \\text{ for } i=1, \\dots, k \\}\\subseteq \\R^\\X$, where $\\ell_i\\colon\\R^\\X\\to\\R$ are affine linear functions for $i=1, \\dots, k$. \nA bounded (and thus compact) polyhedron is called a \\emph{polytope}. \nA polytope can be shown to be the convex hull of finitely many extreme points, which are called \\emph{vertices} and which we denote by $\\operatorname{Vert}(P)$. \nTwo vertices $\\mu_1, \\mu_2\\in\\operatorname{Vert}(P)$ are called \\emph{neighbors} if the subspace $\\{ c\\in\\R^\\X : c^\\top \\mu_1 = c^\\top \\mu_2 = \\max_{\\mu\\in P} c^\\top \\mu \\}$ has dimension $\\lvert \\X \\rvert-1$. \nWe denote the set of all neighbors of a vertex $\\mu$ by $N(\\mu) \\subseteq\\operatorname{Vert}(P)$.  \nThe \\emph{affine space} $\\operatorname{aff\\,span}(P)$ of a polytope $P\\subseteq\\R^\\X$ is the smallest affine subspace of $\\R^\\X$ containing $P$. \nThe relative interior $\\operatorname{int}(P)$ and boundary $\\partial P$ of $P$ are the interior and boundary of $P$ in its affine hull. \nFinally, the \\emph{tangent space} $TP$ of $P$ is given by the linear part of $\\operatorname{aff\\,span}(P)$. \n\nWe call $\\Delta_\\X \\coloneqq \\left\\{ \\mu\\in\\R^\\X_{\\ge0} : \\sum_{x} \\mu_x=1 \\right\\}$ the \\emph{probability simplex}. \nWe say that $\\mu\\in\\Delta_\\X$ is absolutely continuous with respect to $\\nu\\in\\Delta_\\X$ if $\\nu(x) = 0$ implies $\\mu(x)=0$ and write $\\mu \\ll \\nu$. \n","start":14041,"end":16185},{"raw_tex":"The \\emph{Fisher-Rao metric} is a Riemannian metric on the positive orthant given by \n\\begin{equation}\n    g_{\\mu}^{\\FR}(v,w) \\coloneqq \\sum_{x\\in \\X} \\frac{v_x w_x}{\\mu_x} \\quad \\text{for all } v, w\\in\\R^{\\X}, \\mu\\in\\R^{\\X}_{>0},\n\\end{equation}\nwhere we denote the induced norm by $\\lVert v \\rVert_{g_\\mu^{\\FR}} \\coloneqq {g_{\\mu}^{\\FR}(v,v)}^{\\frac12}$. \n","start":18406,"end":18763},{"raw_tex":"The Fisher-Rao metric is closely connected to the negative {Shannon entropy} \n\\begin{equation}\n    \\phi(\\mu) = -H(\\mu) \\coloneqq \\sum_{x\\in\\X} \\mu_x \\log \\mu_x  \\quad \\text{for all } \\mu\\in\\R^{\\X}_{>0}\n\\end{equation}\nas it is induced by the Hessian of the (negative) entropy, meaning that we have $g_{\\mu}^{\\FR}(v,w) = v^\\top \\nabla^2 \\phi(\\mu) w$ for all $v, w\\in\\R^{\\X}, \\mu\\in\\R^{\\X}_{>0}$. \nAs such, the Fisher-Rao metric falls into the class of \\emph{Hessian metrics} that have been studied in convex optimization; we refer to~\\cite{alvarez2004hessian, mueller2023thesis} for general well-posedness and convergence results. \nAn important concept in the analysis of Hessian gradient flows is the \\emph{Bregman divergence} induced by $\\phi$, which in the case of the negative entropy is given by the \\emph{KL-divergence} \n\\begin{equation}\n    \\KL (\\mu, \\nu) \\coloneqq  \\phi(\\mu) - \\phi(\\nu) - \\nabla \\phi(\\nu)(\\mu-\\nu) = \\sum_{x\\in \\X} \\mu_x \\log \\frac{\\mu_x}{\\nu_x} - \\sum_{x\\in \\X} \\mu_x + \\sum_{x\\in \\X} \\nu_x \n\\end{equation}\nfor $\\mu, \\nu\\in\\R_{\\ge0}^\\X$ with $\\mu\\ll\\nu$, where we use the common convention $0\\log\\frac00\\coloneqq0$. \n\nConsider now a continuously differentiable function $f\\colon\\R_{\\ge0}^\\X\\to\\R$ that we assume to be differentiable on $\\R_{>0}^\\X$ that we want to optimize over a polytope $P= \\R_{\\ge0}^{\\X}\\cap\\mathcal L$, where $\\mathcal L$ is a linear space. \nWe denote the gradient of $f\\colon \\R_{>0}^\\X\\to\\R$ at $\\mu\\in\\R_{>0}^\\X$ with respect to the Fisher-Rao metric by $\\nabla^{\\FR}f(\\mu)$ and call it the \\emph{Fisher-Rao gradient}. \nFurther, we denote the Fisher-Rao gradient of $f\\colon\\operatorname{int}(P)\\to\\R$ by $\\nabla^{\\FR}_P f(\\mu)\\in TP$, which is uniquely determined by \n\\begin{equation}\n    g_\\mu^{\\FR}(\\nabla^{\\FR}_P f(\\mu),v) = df(\\mu)v \\quad \\text{for all } v\\in TP.\n\\end{equation}\nNote that $\\nabla^{\\FR}_P f(\\mu)$ is the projection of $\\nabla^{\\FR}f(\\mu)$ with respect to the Fisher-Rao metric onto $TP$.\nBy examining the definition of the Fisher-Rao metric we see that this is equivalent to \n\\begin{equation}\n\\label{eq:characterizationFRGradient}\n    \\langle \\nabla^2\\phi(\\mu)\\nabla^{\\FR}_P  f(\\mu), v \\rangle = \\langle \\nabla f(\\mu), v \\rangle \\quad \\text{for all } v\\in TP. \n\\end{equation}\n    \nWe say that $(\\mu_t)_{t\\in [0, T)} \\subseteq \\operatorname{int}(P)$ solves the \\emph{Fisher-Rao gradient flow} if it solves the gradient flow with respect to the Fisher-Rao metric, i.e., if \n\\begin{equation}\\label{eq:FRGF}\n    \\partial_t \\mu_t = \\nabla^{\\FR}_P  f(\\mu_t) \\quad \\text{for all } t\\in[0, T). \n\\end{equation}\nBy using the characterization~\\eqref{eq:characterizationFRGradient} of $\\nabla^{\\FR}_P f(\\mu_t)$, we see that $(\\mu_t)_{t\\in [0, T)} \\subseteq \\operatorname{int}(P)$ solves the Fisher-Rao gradient flow~\\eqref{eq:FRGF}\nif and only if we have \n\\begin{equation}\\label{eq:explicitFRGF}\n    \\langle \\nabla^2 \\phi(\\mu_t) \\partial_t \\mu_t, v \\rangle = \\langle \\nabla f(\\mu_t), v \\rangle \n    \\quad \\text{for all } v\\in TP, t\\in[0, T). \n\\end{equation}\n","start":19970,"end":22987},{"raw_tex":"In the remainder, we study linear programs and work in the following setting. \n\n\\begin{setting}\\label{setting:generalLP}\nWe consider a finite set $\\X$ and a linear program \n    \\begin{equation}\\label{eq:LP}\n        \\max c^\\top \\mu \\quad \\text{subject to } \\mu\\in P, \n    \\end{equation}\nwith cost $c\\in\\R^{\\X}$ and feasible region $P = \\R^\\X_{\\ge0} \\cap  \\mathcal L$ with $P \\cap \\R^\\X_{>0}\\ne\\emptyset$, where $\\mathcal L\\subseteq\\R^{\\X}$ is an affine space. \nBy $(\\mu_t)_{t\\in[0, T)}\\subseteq\\operatorname{int}(P)$ we denote a solution of the Fisher-Rao gradient flow~\\eqref{eq:FRGF} with initial condition $\\mu_0\\in P \\cap \\R_{>0}^\\X$ and potential $f(\\mu) = c^\\top \\mu$, where $T\\in\\R_{\\ge0}\\cup\\{+\\infty\\}$. \n\\end{setting}\n","start":22988,"end":23715},{"raw_tex":"\\section{Convergence of Fisher-Rao Gradient Flows}\\label{sec:linearConvergeFRGF}\nWe have seen that Fisher-Rao gradient flows converge globally at a sublinear rate $O(t^{-1})$. \nWe now build on this analysis and show that once the gradient flow enters a vicinity of the optimizer, it converges at a quasi-linear rate $O(t^\\kappa e^{-\\Delta t})$, where $\\Delta>0$ depends on the geometry of the linear program and $\\kappa>0$ depends on the initial condition $\\mu_0$. \nNote that this yields $O(e^{-c t})$ convergence for all $c<\\Delta$ and hence we also simply talk of a linear convergence rate.\nWe consider linear programs of the following form. \n\\begin{setting}\\label{setting:LP-inside-Simplex}\nWe consider a finite set $\\X$ and a linear program \n    \\begin{equation}\\label{eq:LP-inside-simplex}\n        \\max c^\\top \\mu \\quad \\text{subject to } \\mu\\in P, \n    \\end{equation}\nwith cost $c\\in\\R^{\\X}$ and feasible region $P = \\Delta_\\X \\cap  \\mathcal L$ with $P \\cap \\R^\\X_{>0}\\ne\\emptyset$, where $\\mathcal L\\subseteq\\R^{\\X}$ is an affine space. \nBy $(\\mu_t)_{t\\ge0}\\subseteq\\operatorname{int}(P)$ we denote the solution of the Fisher-Rao gradient flow~\\eqref{eq:FRGF} with initial condition $\\mu_0\\in P \\cap \\R_{>0}^\\X$ and the potential $f(\\mu) = c^\\top \\mu$. \n\\end{setting}\n","start":27574,"end":28849}]},"statement":{"raw_tex":"\\begin{restatable}[Linear convergence of Fisher-Rao GFs of LPs]{theorem}{convergenceFRGF}\n\\label{thm:convergenceFRGF}\nConsider Setting~\\ref{setting:LP-inside-Simplex} and assume that the linear program~\\eqref{eq:LP-inside-simplex} admits a unique solution $\\mu^\\star\\in P$. \nLet \n\\begin{align}\\label{eq:Delta}\n    \\Delta \\coloneqq \\min \\left\\{ \\frac{c^\\top \\mu^\\star - \n    c^\\top \\mu}{\\lVert \\mu^\\star - \\mu \\rVert_{\\TV }} : \\mu\\in N(\\mu^\\star)  \\right\\},\n\\end{align}\nwhere $N(\\mu^\\star)$ denotes the set of neighboring vertices of $\\mu^\\star$ and set \n\\begin{equation}\\label{eq:definitiont_0}\n    t_0 \\coloneqq \\frac{2 \\KL (\\mu^\\star, \\mu_0)}{\\Delta \\cdot \\min\\{ \\mu^\\star_x : \\mu^\\star_x>0 \\}} . \n\\end{equation}\nThen for any $t\\ge t_0$ we have \n\\begin{align}\\label{eq:linearConvergenceKL}\n    \\KL (\\mu^\\star, \\mu_t) \\le \\KL (\\mu^\\star, \\mu_0) \\exp\\left( -\\Delta(t-t_0) +  2 t_0 \\Delta \n    \\log\\left(\\frac{t+t_0}{2t_0}\\right) \\right), \n\\end{align}\nas well as \n\\begin{align}\\label{eq:linearConvergenceValue}\n    c^\\top \\mu^\\star - c^\\top \\mu_t \\le \\Delta \\KL (\\mu^\\star, \\mu_0) \\exp\\left( -\\Delta(t-t_0) +  2 t_0 \\Delta \\log\\left(\\frac{t+t_0}{2t_0}\\right) \\right). \n\\end{align}\n\\end{restatable}","units":[{"raw_tex":"\\begin{restatable}[Linear convergence of Fisher-Rao GFs of LPs]{theorem}{convergenceFRGF}\n\\label{thm:convergenceFRGF}\nConsider Setting~\\ref{setting:LP-inside-Simplex} and assume that the linear program~\\eqref{eq:LP-inside-simplex} admits a unique solution $\\mu^\\star\\in P$. \nLet \n\\begin{align}\\label{eq:Delta}\n    \\Delta \\coloneqq \\min \\left\\{ \\frac{c^\\top \\mu^\\star - \n    c^\\top \\mu}{\\lVert \\mu^\\star - \\mu \\rVert_{\\TV }} : \\mu\\in N(\\mu^\\star)  \\right\\},\n\\end{align}\nwhere $N(\\mu^\\star)$ denotes the set of neighboring vertices of $\\mu^\\star$ and set \n\\begin{equation}\\label{eq:definitiont_0}\n    t_0 \\coloneqq \\frac{2 \\KL (\\mu^\\star, \\mu_0)}{\\Delta \\cdot \\min\\{ \\mu^\\star_x : \\mu^\\star_x>0 \\}} . \n\\end{equation}\nThen for any $t\\ge t_0$ we have \n\\begin{align}\\label{eq:linearConvergenceKL}\n    \\KL (\\mu^\\star, \\mu_t) \\le \\KL (\\mu^\\star, \\mu_0) \\exp\\left( -\\Delta(t-t_0) +  2 t_0 \\Delta \n    \\log\\left(\\frac{t+t_0}{2t_0}\\right) \\right), \n\\end{align}\nas well as \n\\begin{align}\\label{eq:linearConvergenceValue}\n    c^\\top \\mu^\\star - c^\\top \\mu_t \\le \\Delta \\KL (\\mu^\\star, \\mu_0) \\exp\\left( -\\Delta(t-t_0) +  2 t_0 \\Delta \\log\\left(\\frac{t+t_0}{2t_0}\\right) \\right). \n\\end{align}\n\\end{restatable}","start":29173,"end":30369}]},"proof":{"raw_tex":"\\begin{proposition}[Central path property,\\cite{alvarez2004hessian}]\n    Consider Setting~\\ref{setting:generalLP}. \n    Then $\\mu_t$ is uniquely characterized by \n    \\begin{equation}\\label{eq:centralPath}\n        \\mu_t = \\argmax\\left\\{ c^\\top \\mu - t^{-1} \\KL (\\mu, \\mu_0) : \\mu \\in P \\right\\} \\quad \\text{for all } t\\in(0,T). \n    \\end{equation}\n\\end{proposition}\n\n\\begin{proof}\n    Let $\\hat{\\mu}_t\\in P$ denote the unique maximizer of $g(\\mu)\\coloneqq c^\\top\\mu - t^{-1} \\KL (\\mu, \\mu_0)$ over $P$ for $t>0$, then surely $\\hat{\\mu}_t\\in \\operatorname{int}(P)$. \n    Thus, $\\hat{\\mu}_t$ is uniquely determined by $\\langle \\nabla g(\\hat{\\mu}_t), v\\rangle = 0$ for all $v\\in TP$. \n    Direct computation yields $\\nabla g({\\mu}) = c - t^{-1} (\\nabla\\phi(\\mu) - \\nabla\\phi(\\mu_0))$ and hence $\\hat{\\mu}_t$ is uniquely determined by \n    \\begin{equation*}\n        t\\langle c, v\\rangle = \\langle \\nabla\\phi(\\hat{\\mu}_t) - \\nabla\\phi(\\mu_0), v\\rangle \\quad \\text{for all } v\\in TP. \n    \\end{equation*} \n    On the other hand, for the gradient flow, we can use~\\eqref{eq:explicitFRGF} and compute for $v\\in TP$ \n    \\begin{align*}\n        \\langle \\nabla\\phi(\\mu_t) - \\nabla\\phi(\\mu_0), v \\rangle & = \\int_{0}^t \\partial_s \\langle \\nabla \\phi(\\mu_s), v \\rangle \\D s \n        = \\int_{0}^t \\langle \\nabla^2 \\phi(\\mu_s)\\partial_s \\mu_s, v \\rangle \\D s \n        \\\\ & = \\int_{0}^t \\langle \\nabla f(\\mu_s) \\mu_s, v \\rangle \\D s \n        = \\int_{0}^t \\langle c, v \\rangle \\D s = t \\langle c, v \\rangle . \n    \\end{align*}\n    This shows $\\mu_t = \\hat{\\mu}_t$ as claimed. \n\\end{proof}\n\n\\begin{corollary}[Sublinear convergence rate,\\cite{alvarez2004hessian}]\\label{prop:sublinearRate}\n    Consider Setting~\\ref{setting:generalLP} and assume that the linear program~\\eqref{eq:LP} admits a solution $\\mu^\\star\\in P$. \n    Then for $\\mu\\in P$ it holds that \n\\begin{equation}\\label{eq:sublinearConvergence}\n        c^\\top \\mu^\\star - c^\\top\\mu_t \\le \\frac{\\KL (\\mu^\\star, \\mu) - \\KL (\\mu_t, \\mu)}t \\le \\frac{\\KL (\\mu^\\star, \\mu_0)}t \\quad \\text{for all } t\\in[0, T). \n    \\end{equation}\n\\end{corollary}\n\n\\begin{proof}\n    We have $c^\\top \\mu_t - t^{-1} \\KL (\\mu_t, \\mu) \\ge c^\\top \\mu^\\star - t^{-1} \\KL (\\mu^\\star, \\mu)$ by the central path property.\n    Rearranging yields the result. \n\\end{proof}\n\n\\begin{theorem}[Well-posedness of  FR GFs,\\cite{alvarez2004hessian}]\n    Consider Setting~\\ref{setting:generalLP}. \n    Then there exists a unique global solution $(\\mu_{t})_{t\\ge0}\\subseteq\\operatorname{int}(P)$ of the Fisher-Rao gradient flow~\\eqref{eq:FRGF}. \n\\end{theorem}\n\n\\begin{proof}\n    The local existence and uniqueness follow from the Picard-Lindelöf theorem~\\cite{teschl2024ordinary}. Hence, it suffices to show that the Fisher-Rao gradient flow does not hit the boundary $\\partial P$ in finite time. \n    By the central path property, this is equivalent to the statement that the solutions of all KL-regularized problems~\\eqref{eq:centralPath} lie in the interior $\\operatorname{int}(P)$ of the polyhedron, which can be easily checked. \n\\end{proof}\n\n\\begin{lemma}\\label{prop:ratesTrajectories} \n    Consider Setting~\\ref{setting:generalLP} and assume that there is an optimizer $\\mu^\\star\\in P$ and $\\kappa_t>0$ for $t>t_0\\ge0$ such that\n\\begin{equation}\\label{eq:strongConvexity}\n    c^\\top \\mu^\\star - c^\\top \\mu_t \\ge \\kappa_t \\KL (\\mu^\\star, \\mu_t) \\quad \\text{for all } t> t_0. \n\\end{equation}\nThen we have \n\\begin{equation}\\label{eq:lem:KLConvergence}\n    \\KL (\\mu^\\star, \\mu_t) \\le \\KL (\\mu^\\star, \\mu_{0}) \\exp\\left( -\\int_{t_0}^t \\kappa_s \\D s \\right) \\quad \\text{for all } t\\ge t_0,\n\\end{equation}\nas well as \n\\begin{equation}\\label{eq:lem:convergenceValue}\n    c^\\top \\mu^\\star - c^\\top \\mu_t \\le \\kappa_t \\KL (\\mu^\\star, \\mu_{0}) \\exp\\left( -\\int_{t_0}^t \\kappa_s \\D s \\right) \\quad \\text{for all } t\\ge t_0.\n\\end{equation}\n\\end{lemma}\n\n\\begin{lemma}[\\cite{alvarez2004hessian}]\\label{lem:evolutionKL}\nConsider Setting~\\ref{setting:generalLP}, whereby we allow $f\\colon\\R_{>0}^\\X\\to\\R$ to be an arbitrary differentiable function, and fix $\\mu\\in P$. \nThen for any $t\\ge0$, it holds that \n    \\begin{equation}\\label{eq:derivativeu}\n        \\partial_t \\KL (\\mu, \\mu_t) = \\langle \\nabla f(\\mu_t), \\mu_t-\\mu\\rangle. \n    \\end{equation}\n\\end{lemma}\n\n\\begin{proof}\n    Denoting the negative Shannon entropy by $\\phi$, we compute \n    \\begin{align*}\n        \\partial_t \\KL (\\mu, \\mu_t) = - \\partial_t \\phi(\\mu_t) - \\partial_t \\langle \\nabla \\phi(\\mu_t), \\mu - \\mu_t \\rangle = \\langle \\nabla^2 \\phi(\\mu_t)\\partial_t \\mu_t, \\mu_t - \\mu \\rangle.\n    \\end{align*}\n    Now~\\eqref{eq:explicitFRGF} yields the claim. \n\\end{proof}\n\n\\begin{proof}[Proof of \\Cref{prop:ratesTrajectories}]\nUsing~\\eqref{eq:derivativeu} and~\\eqref{eq:strongConvexity} we find that for all $t\\ge T$ it holds that \n$\\partial_t \\KL (\\mu^\\star, \\mu_t) = c^\\top \\mu_t  - c^\\top \\mu^\\star \\le - \\kappa_t \\KL (\\mu^\\star, \\mu_t)$. \nNow Gronwall's inequality yields~\\eqref{eq:lem:KLConvergence}. \nBy \\Cref{prop:sublinearRate} we have for any $h>0$ that \n    \\[ c^\\top \\mu^\\star - c^\\top \\mu_t \\le \\frac{\\KL (\\mu^\\star, \\mu_{t-h})}{h} \\le  \\KL (\\mu^\\star, \\mu_{0}) \\cdot \\frac{\\exp\\left(-\\int_{t_0}^{t-h} \\kappa_s \\D s \\right)}{h}. \\] \nTaking the limit $h\\to0$ yields~\\eqref{eq:lem:convergenceValue}. \n\\end{proof}\n\n\\begin{restatable}{lemma}{TVLPEstimateWU}\n\\label{lem:TV-LP-estimate-withoutUniqueness}\nConsider a polytope $P \\subseteq \\R^{\\X}$ and denote by $F^\\star$ the face of maximizers of the linear function $\\mu\\mapsto c^\\top \\mu$ over $P$. \nDenote the set of neighboring vertices of a vertex $\\mu$ by $N(\\mu)$ and let $\\lVert \\cdot\\rVert\\colon\\R^{\\X}\\to\\R_{\\ge0}$ be an arbitrary semi-norm. \nThen either $F^\\star = P$ or with $\\frac{c}{0}\\coloneqq+\\infty$ for $c>0$, we have \n\\begin{equation}\n    \\Delta\\coloneqq \\min \\left\\{ \\frac{c^\\top\\mu^\\star - c^\\top\\mu }{\\lVert \\mu^\\star - \\mu \\rVert} : \\mu^\\star \\in\\operatorname{vert}(F^\\star), \\mu\\in N(\\mu^\\star)\\setminus F^\\star \\right\\} > 0,\n\\end{equation}\nand further \n\\begin{equation}\\label{eq:gapVSNorm-2}\n     c^\\top \\mu^\\star - c^\\top \\mu \\ge \\Delta \\cdot \\inf_{\\mu^\\star\\in F^\\star}  \\lVert \\mu^\\star - \\mu\\rVert \\quad \\text{for all } \\mu\\in P.\n\\end{equation}\n\\end{restatable}\n\n\\begin{proof}\nIf $F^\\star \\ne P$, then $c^\\top \\mu^\\star-c^\\top \\mu >0$ for some vertex $\\mu$, which implies $\\Delta>0$. \nTo simplify notation we denote the set $E\\coloneqq\\{\\mu-\\mu^\\star: \\mu\\in N(\\mu^\\star) \\setminus F^\\star, \\mu^\\star\\in \\operatorname{vert}(F^\\star)\\}$ of edges such that exactly one of the two endpoints is contained in $F^\\star$. \nThen, the polytope $P$ is contained in \n    \\[ F^\\star+C = \\left\\{ \\mu^\\star + \\sum_{e\\in E} \\alpha_e e : \\mu^\\star\\in F^\\star, \\alpha_e\\ge0 \\text{ for all } e\\in E \\right\\}, \\]\nsee \\Cref{app:lem:cone} and hence we can write $\\mu\\in P$ as $\\mu = \\mu^\\star + \\sum_{e} \\alpha_e e$ for some $\\mu^\\star\\in F^\\star$. \nUsing the triangle inequality we obtain \n    \\[ \\Delta \\lVert \\mu^\\star - \\mu \\rVert \\le \\Delta \\sum_{e\\in E}\\alpha_e \\lVert e \\rVert \\le - \\sum_{e\\in E} \\alpha_e c^\\top e = c^\\top \\mu^\\star - c^\\top\\mu. \\]\n\\end{proof}\n\n\\begin{lemma}\\label{lem:KLEstimateL1}\nConsider a finite set $\\X$ and a probability distribution $\\mu\\in \\Delta_\\X$. Let $c>1$ and set $\\delta\\coloneqq \\frac{c-1}{c+1} \n    \\cdot \\min\\{ \\mu_x : \\mu_x>0 \\}>0$. \nThen for all $\\nu\\in\\Delta_\\X$ satisfying $\\lVert \\mu-\\nu\\rVert_\\infty\\le\\delta$ it holds that \n\\begin{equation}\\label{eq:KLvsL1}\n    \\KL (\\mu, \\nu) \\le c \\cdot \\lVert \\mu - \\nu \\rVert_{\\TV }. \n\\end{equation}\n\\end{lemma}\n\n\\begin{proof}\nWe bound the individual summands in the KL-divergence\n    \\[ \\KL (\\mu, \\nu) = \\sum_{x\\in\\X} \\mu_x \\log\\left( \\frac{\\mu_x}{\\nu_x}\\right) = \\sum_{x\\in X} \\mu_x \\log\\left( \\frac{\\mu_x}{\\nu_x}\\right), \\]\nwhere $X\\coloneqq\\{x\\in\\X : \\mu_x>0\\}$. \nIf $\\mu_x,\\nu_x>0$ then \n\\begin{align}\\label{eq:klIndividual}\n    \\begin{split}\n        \\mu_x \\log \\left(\\frac{\\mu_x}{\\nu_x}\\right) & = \\mu_x \\left( \\log(\\nu_x+(\\mu_x-\\nu_x)) - \\log(\\nu_x) \\right) \\\\\n        & \\le \\mu_x \\left( \\log(\\nu_x) + \\frac{\\mu_x-\\nu_x}{\\nu_x} - \\log(\\nu_x) \\right) \n         = (\\mu_x - \\nu_x)\\cdot\\frac{\\mu_x}{\\nu_x},\n    \\end{split}\n\\end{align}\nwhere we used the convexity $\\log(t+h) \\le \\log(t) + h/t$ for $t>0, t+h>0$. \nWe set $\\varepsilon\\coloneqq\\frac{c-1}{2}\\in(0,1)$, such that \n\\[ \\delta = \\frac{\\varepsilon}{1+\\varepsilon}  \n    \\cdot \\min\\big\\{ \\mu_x : \\mu_x>0 \\big\\}. \n    \\]\nIf $\\lVert \\mu-\\nu\\rVert_\\infty\\le\\delta$ then \n    \\[ \\nu_x \\ge \\mu_x - \\delta \\ge \\mu_x \\left(1-\\frac{\\varepsilon}{1+\\varepsilon}\\right) = \\frac{\\mu_x}{1+\\varepsilon} \\]\nas well as\n    \\[ \\nu_x \\le \\mu_x + \\delta  \\le \\mu_x \\left(1 + \\frac{\\varepsilon}{1+\\varepsilon}\\right) \\le \\mu_x \\left(1 + \\frac{\\varepsilon}{1-\\varepsilon}\\right) = \\frac{\\mu_x}{1-\\varepsilon} \\]\nand therefore $1-\\varepsilon\\le \\frac{\\mu_x}{\\nu_x}\\le1+\\varepsilon$. \nIf $\\mu_x\\ge\\nu_x$ then \n    \\[ \n    (\\mu_x-\\nu_x)\\cdot \\frac{\\mu_x}{\\nu_x} \n    \\le (1+\\varepsilon)(\\mu_x-\\nu_x) \n    = \\mu_x-\\nu_x + \\varepsilon \\lvert \\mu_x - \\nu_x \\rvert , \n    \\]\nand if $\\mu_x<\\nu_x$ then \n\\begin{equation}\\label{eq:almostThere}\n    (\\mu_x-\\nu_x)\\cdot \\frac{\\mu_x}{\\nu_x} \\le (1-\\varepsilon)(\\mu_x-\\nu_x) = \\mu_x-\\nu_x + \\varepsilon \\lvert \\mu_x - \\nu_x \\rvert. \n\\end{equation}\nTogether with~\\eqref{eq:klIndividual} summing over $x$ yields \n\\begin{equation}\\label{eq:there}\n    \\KL (\\mu, \\nu) \\le \\sum_{x\\in X} (\\mu_x-\\nu_x) + \\varepsilon \\sum_{x\\in X} \\lvert \\mu_x - \\nu_x \\rvert \\le \\sum_{x\\in X} (\\mu_x-\\nu_x) + 2 \\varepsilon \\lVert \\mu-\\nu\\rVert_{\\TV } . \n\\end{equation}\nIt remains to estimate the first part. \nSetting $X^c\\coloneqq\\X\\setminus X$, we have \n\\[ \n\\sum_{x\\in X} (\\mu_x-\\nu_x) = \\sum_{x\\in\\X} (\\mu_x-\\nu_x) -  \\sum_{x\\in X^c} (\\mu_x-\\nu_x) = -  \\sum_{x\\in X^c} (\\mu_x-\\nu_x) = \\sum_{x\\in X^c} \\lvert  \\mu_x-\\nu_x \\rvert  \n\\]\nsince $\\mu_x=0$ for $x\\in X^c$. \nNow we can estimate\n\\begin{align}\\label{eq:lastStep}\n    2\\sum_{x\\in X} (\\mu_x-\\nu_x) = \\sum_{x\\in X} (\\mu_x-\\nu_x) + \\sum_{x\\in X^c} \\lvert  \\mu_x-\\nu_x \\rvert \\le \\lVert \\mu-\\nu\\rVert_1 = 2\\lVert \\mu-\\nu\\rVert_{\\TV }.\n\\end{align}\nCombining~\\eqref{eq:there} and~\\eqref{eq:lastStep} yields\n\\[ \\KL (\\mu, \\nu) \\le (1+2\\varepsilon)\\lVert \\mu - \\nu \\rVert_{\\TV } = c\\cdot\\lVert \\mu - \\nu \\rVert_{\\TV }. \\]\n\\end{proof}\n\n\\begin{corollary}[Local KL-TV estimate]\\label{cor:TV-KL}\nConsider a finite set $\\X$ and a probability distribution $\\mu\\in \\Delta_\\X$. \nThen for all $\\nu\\in\\Delta_\\X$ satisfying \n\\begin{equation}\n    \\lVert \\mu - \\nu \\rVert_\\infty < \\min\\{ \\mu_x : \\mu_x>0 \\} \n\\end{equation}\nit holds that \n\\begin{equation}\\label{eq:KLvsL1-2}\n    \\KL (\\mu, \\nu) \\le \\frac{\\min\\{ \\mu_x : \\mu_x>0 \\} +  \\lVert \\mu-\\nu\\rVert_\\infty}{\\min\\{ \\mu_x : \\mu_x>0 \\} - \\lVert \\mu-\\nu\\rVert_\\infty} \\cdot \\lVert \\mu - \\nu \\rVert_{\\TV }. \n\\end{equation}\n\\end{corollary}\n\n\\begin{proof}\n    This is a direct consequence of \\Cref{lem:KLEstimateL1}. \n    Indeed, for $\\varepsilon>0$ small enough we have $\\lVert \\mu - \\nu \\rVert_\\infty \\le \\frac{2-\\varepsilon}{2+\\varepsilon} \\cdot \\min\\{ \\mu_x : \\mu_x>0 \\}$ and thus by \\Cref{lem:KLEstimateL1} with $c=1+\\varepsilon$ we have $\\KL (\\mu, \\nu) \\le (1+\\varepsilon) \\lVert \\mu - \\nu \\rVert_{\\TV }$. \n    Note that $\\varepsilon>0$ was arbitrary. \n\\end{proof}\n\n\\begin{proof}[Proof of \\Cref{thm:convergenceFRGF}]\n    Setting $\\delta\\coloneqq \\min\\{ \\mu_x^\\star : \\mu_x^\\star>0 \\}$ and using \\Cref{lem:TV-LP-estimate-withoutUniqueness} with $\\lVert\\cdot\\rVert_{\\TV}$ and \\Cref{cor:TV-KL} we have \n    \\begin{equation*}\n        c^\\top \\mu^\\star - c^\\top \\mu_t \\ge \\Delta \\lVert \\mu^\\star - \\mu_t \\rVert_{\\TV } \\ge \\Delta\\cdot\\frac{\\delta -  \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty}{\\delta + \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty} \\cdot \\KL (\\mu^\\star, \\mu_t),\n    \\end{equation*}\n    if $\\lVert \\mu^\\star - \\mu_t \\rVert_\\infty < \\delta$.\n    By \\Cref{prop:sublinearRate} we have \n    \\begin{align*}\n        \\lVert \\mu^\\star - \\mu_t \\rVert_\\infty \\le 2 \\lVert \\mu^\\star - \\mu_t \\rVert_{\\TV } \\le \\frac{2c^\\top(\\mu^\\star-\\mu_t)}\\Delta \\le \\frac{2\\KL (\\mu^\\star,\\mu_0)}{\\Delta \\cdot t}.\n    \\end{align*}\n    Hence, for $t>t_0$ we have $\\lVert \\mu^\\star - \\mu_t \\rVert_\\infty < \\delta$. \n    In this case, we can estimate \n    \\begin{align*}\n        \\frac{\\delta -  \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty}{\\delta + \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty} & \\le \\frac{\\delta -  2 \\KL (\\mu^\\star, \\mu_0)\\Delta^{-1} t^{-1}} {\\delta + 2 \\KL (\\mu^\\star, \\mu_0)\\Delta^{-1} t^{-1}} \n        = \\frac{t - t_0}{t + t_0} \\eqqcolon \\kappa_t. \n    \\end{align*}\n    Thus for $t>t_0$ we have $c^\\top \\mu^\\star - c^\\top \\mu_t \\ge \\Delta \\kappa_t \\KL (\\mu^\\star, \\mu_t)$, and \\Cref{prop:ratesTrajectories} together with \n    \\begin{equation*}\n        \\int_{t_0}^t \\frac{s-t_0}{s+t_0} \\D s = s - 2 t_0 \\log(s+t_0)\\Big|^{s=t}_{s=t_0} = (t-t_0) - 2 t_0 \\log\\left( \\frac{t+t_0}{2t_0} \\right)\n    \\end{equation*}\n    yield the result. \n\\end{proof}\n\n\\begin{lemma}\\label{app:lem:cone}\n    Consider a polytope $P\\subseteq\\R^\\X$, a face $F\\subseteq P$ and consider the cone \n    \\begin{align*}\n        C\\coloneqq \\operatorname{cone}\\Big\\{ \\nu-\\mu : \\mu\\in\\operatorname{vert}(F), \\nu\\in N(\\mu)\\setminus F \\Big\\}, \n    \\end{align*}\n    which is generated by the edges pointing out of $F$. \n    Then we have $P\\subseteq F+C$. \n\\end{lemma}\n\n\\begin{proof}\nThis is a generalization of~\\cite[Lemma 3.6]{ziegler2012lectures}, which covers the case that $F$ consists of a single vertex. \nWe will show that \n\\begin{align*}\n    F+C \\supseteq\\tilde P \\coloneqq \\bigcap \\Big\\{ H\\subseteq \\R^\\X : H \\text{ is a halfspace }, P\\subseteq H, H\\cap F\\ne\\emptyset \\Big\\} \\supseteq P, \n\\end{align*}\nfor which we pick an element $u\\in \\tilde P$. \nConsider a hyperplane $H = \\{\\mu:a^\\top \\mu = \\alpha\\}$ separating $F$ and $\\operatorname{vert}(P)\\setminus F$ and consider the \\emph{face figure} $P/F\\coloneqq P\\cap H$, which is a polytope. \nNow, we consider a translation $\\tilde H = \\{ \\mu : a^\\top\\mu= \\beta \\}$ of $H$,  such that $u\\in \\tilde H$. \nNow we have \n\\begin{align*}\n    \\tilde P = \\operatorname{conv}\\left\\{ \\mu + \\frac{a^\\top \\mu - \\beta}{a^\\top \\mu - a^\\top \\nu} \\cdot (\\nu-\\mu) : \\mu\\in\\operatorname{vert}(F), \\nu\\in N(\\mu)\\setminus F \\right\\},\n\\end{align*}\nsee~\\cite[Proposition 2.30]{zieglerdiscrete}. \nHence, we can choose convex weights $\\lambda_i$ such that \n\\begin{align*}\n    u = \\sum_{i} \\lambda_i (\\mu_i + \\alpha_i (\\nu_i-\\mu_i)) =  \\sum_{i} \\lambda_i \\mu_i + \\sum_{i} \\alpha_i \\lambda_i (\\nu_i-\\mu_i) \\in F + C,\n\\end{align*}\nwhere $\\mu_i\\in\\operatorname{vert}(F), \\nu_i \\in N(\\mu_i)\\setminus F$.\n\\end{proof}","units":[{"raw_tex":"\\begin{proposition}[Central path property,\\cite{alvarez2004hessian}]\n    Consider Setting~\\ref{setting:generalLP}. \n    Then $\\mu_t$ is uniquely characterized by \n    \\begin{equation}\\label{eq:centralPath}\n        \\mu_t = \\argmax\\left\\{ c^\\top \\mu - t^{-1} \\KL (\\mu, \\mu_0) : \\mu \\in P \\right\\} \\quad \\text{for all } t\\in(0,T). \n    \\end{equation}\n\\end{proposition}","start":24059,"end":24424},{"raw_tex":"\\begin{proof}\n    Let $\\hat{\\mu}_t\\in P$ denote the unique maximizer of $g(\\mu)\\coloneqq c^\\top\\mu - t^{-1} \\KL (\\mu, \\mu_0)$ over $P$ for $t>0$, then surely $\\hat{\\mu}_t\\in \\operatorname{int}(P)$. \n    Thus, $\\hat{\\mu}_t$ is uniquely determined by $\\langle \\nabla g(\\hat{\\mu}_t), v\\rangle = 0$ for all $v\\in TP$. \n    Direct computation yields $\\nabla g({\\mu}) = c - t^{-1} (\\nabla\\phi(\\mu) - \\nabla\\phi(\\mu_0))$ and hence $\\hat{\\mu}_t$ is uniquely determined by \n    \\begin{equation*}\n        t\\langle c, v\\rangle = \\langle \\nabla\\phi(\\hat{\\mu}_t) - \\nabla\\phi(\\mu_0), v\\rangle \\quad \\text{for all } v\\in TP. \n    \\end{equation*} \n    On the other hand, for the gradient flow, we can use~\\eqref{eq:explicitFRGF} and compute for $v\\in TP$ \n    \\begin{align*}\n        \\langle \\nabla\\phi(\\mu_t) - \\nabla\\phi(\\mu_0), v \\rangle & = \\int_{0}^t \\partial_s \\langle \\nabla \\phi(\\mu_s), v \\rangle \\D s \n        = \\int_{0}^t \\langle \\nabla^2 \\phi(\\mu_s)\\partial_s \\mu_s, v \\rangle \\D s \n        \\\\ & = \\int_{0}^t \\langle \\nabla f(\\mu_s) \\mu_s, v \\rangle \\D s \n        = \\int_{0}^t \\langle c, v \\rangle \\D s = t \\langle c, v \\rangle . \n    \\end{align*}\n    This shows $\\mu_t = \\hat{\\mu}_t$ as claimed. \n\\end{proof}","start":24429,"end":25633},{"raw_tex":"\\begin{corollary}[Sublinear convergence rate,\\cite{alvarez2004hessian}]\\label{prop:sublinearRate}\n    Consider Setting~\\ref{setting:generalLP} and assume that the linear program~\\eqref{eq:LP} admits a solution $\\mu^\\star\\in P$. \n    Then for $\\mu\\in P$ it holds that \n\\begin{equation}\\label{eq:sublinearConvergence}\n        c^\\top \\mu^\\star - c^\\top\\mu_t \\le \\frac{\\KL (\\mu^\\star, \\mu) - \\KL (\\mu_t, \\mu)}t \\le \\frac{\\KL (\\mu^\\star, \\mu_0)}t \\quad \\text{for all } t\\in[0, T). \n    \\end{equation}\n\\end{corollary}","start":25809,"end":26320},{"raw_tex":"\\begin{proof}\n    We have $c^\\top \\mu_t - t^{-1} \\KL (\\mu_t, \\mu) \\ge c^\\top \\mu^\\star - t^{-1} \\KL (\\mu^\\star, \\mu)$ by the central path property.\n    Rearranging yields the result. \n\\end{proof}","start":26321,"end":26516},{"raw_tex":"\\begin{theorem}[Well-posedness of  FR GFs,\\cite{alvarez2004hessian}]\n    Consider Setting~\\ref{setting:generalLP}. \n    Then there exists a unique global solution $(\\mu_{t})_{t\\ge0}\\subseteq\\operatorname{int}(P)$ of the Fisher-Rao gradient flow~\\eqref{eq:FRGF}. \n\\end{theorem}","start":26811,"end":27087},{"raw_tex":"\\begin{proof}\n    The local existence and uniqueness follow from the Picard-Lindelöf theorem~\\cite{teschl2024ordinary}. Hence, it suffices to show that the Fisher-Rao gradient flow does not hit the boundary $\\partial P$ in finite time. \n    By the central path property, this is equivalent to the statement that the solutions of all KL-regularized problems~\\eqref{eq:centralPath} lie in the interior $\\operatorname{int}(P)$ of the polyhedron, which can be easily checked. \n\\end{proof}","start":27088,"end":27572},{"raw_tex":"\\begin{lemma}\\label{prop:ratesTrajectories} \n    Consider Setting~\\ref{setting:generalLP} and assume that there is an optimizer $\\mu^\\star\\in P$ and $\\kappa_t>0$ for $t>t_0\\ge0$ such that\n\\begin{equation}\\label{eq:strongConvexity}\n    c^\\top \\mu^\\star - c^\\top \\mu_t \\ge \\kappa_t \\KL (\\mu^\\star, \\mu_t) \\quad \\text{for all } t> t_0. \n\\end{equation}\nThen we have \n\\begin{equation}\\label{eq:lem:KLConvergence}\n    \\KL (\\mu^\\star, \\mu_t) \\le \\KL (\\mu^\\star, \\mu_{0}) \\exp\\left( -\\int_{t_0}^t \\kappa_s \\D s \\right) \\quad \\text{for all } t\\ge t_0,\n\\end{equation}\nas well as \n\\begin{equation}\\label{eq:lem:convergenceValue}\n    c^\\top \\mu^\\star - c^\\top \\mu_t \\le \\kappa_t \\KL (\\mu^\\star, \\mu_{0}) \\exp\\left( -\\int_{t_0}^t \\kappa_s \\D s \\right) \\quad \\text{for all } t\\ge t_0.\n\\end{equation}\n\\end{lemma}","start":40737,"end":41534},{"raw_tex":"\\begin{lemma}[\\cite{alvarez2004hessian}]\\label{lem:evolutionKL}\nConsider Setting~\\ref{setting:generalLP}, whereby we allow $f\\colon\\R_{>0}^\\X\\to\\R$ to be an arbitrary differentiable function, and fix $\\mu\\in P$. \nThen for any $t\\ge0$, it holds that \n    \\begin{equation}\\label{eq:derivativeu}\n        \\partial_t \\KL (\\mu, \\mu_t) = \\langle \\nabla f(\\mu_t), \\mu_t-\\mu\\rangle. \n    \\end{equation}\n\\end{lemma}","start":41602,"end":42007},{"raw_tex":"\\begin{proof}\n    Denoting the negative Shannon entropy by $\\phi$, we compute \n    \\begin{align*}\n        \\partial_t \\KL (\\mu, \\mu_t) = - \\partial_t \\phi(\\mu_t) - \\partial_t \\langle \\nabla \\phi(\\mu_t), \\mu - \\mu_t \\rangle = \\langle \\nabla^2 \\phi(\\mu_t)\\partial_t \\mu_t, \\mu_t - \\mu \\rangle.\n    \\end{align*}\n    Now~\\eqref{eq:explicitFRGF} yields the claim. \n\\end{proof}","start":42008,"end":42378},{"raw_tex":"\\begin{proof}[Proof of \\Cref{prop:ratesTrajectories}]\nUsing~\\eqref{eq:derivativeu} and~\\eqref{eq:strongConvexity} we find that for all $t\\ge T$ it holds that \n$\\partial_t \\KL (\\mu^\\star, \\mu_t) = c^\\top \\mu_t  - c^\\top \\mu^\\star \\le - \\kappa_t \\KL (\\mu^\\star, \\mu_t)$. \nNow Gronwall's inequality yields~\\eqref{eq:lem:KLConvergence}. \nBy \\Cref{prop:sublinearRate} we have for any $h>0$ that \n    \\[ c^\\top \\mu^\\star - c^\\top \\mu_t \\le \\frac{\\KL (\\mu^\\star, \\mu_{t-h})}{h} \\le  \\KL (\\mu^\\star, \\mu_{0}) \\cdot \\frac{\\exp\\left(-\\int_{t_0}^{t-h} \\kappa_s \\D s \\right)}{h}. \\] \nTaking the limit $h\\to0$ yields~\\eqref{eq:lem:convergenceValue}. \n\\end{proof}","start":42379,"end":43028},{"raw_tex":"\\begin{restatable}{lemma}{TVLPEstimateWU}\n\\label{lem:TV-LP-estimate-withoutUniqueness}\nConsider a polytope $P \\subseteq \\R^{\\X}$ and denote by $F^\\star$ the face of maximizers of the linear function $\\mu\\mapsto c^\\top \\mu$ over $P$. \nDenote the set of neighboring vertices of a vertex $\\mu$ by $N(\\mu)$ and let $\\lVert \\cdot\\rVert\\colon\\R^{\\X}\\to\\R_{\\ge0}$ be an arbitrary semi-norm. \nThen either $F^\\star = P$ or with $\\frac{c}{0}\\coloneqq+\\infty$ for $c>0$, we have \n\\begin{equation}\n    \\Delta\\coloneqq \\min \\left\\{ \\frac{c^\\top\\mu^\\star - c^\\top\\mu }{\\lVert \\mu^\\star - \\mu \\rVert} : \\mu^\\star \\in\\operatorname{vert}(F^\\star), \\mu\\in N(\\mu^\\star)\\setminus F^\\star \\right\\} > 0,\n\\end{equation}\nand further \n\\begin{equation}\\label{eq:gapVSNorm-2}\n     c^\\top \\mu^\\star - c^\\top \\mu \\ge \\Delta \\cdot \\inf_{\\mu^\\star\\in F^\\star}  \\lVert \\mu^\\star - \\mu\\rVert \\quad \\text{for all } \\mu\\in P.\n\\end{equation}\n\\end{restatable}","start":43603,"end":44525},{"raw_tex":"\\begin{proof}\nIf $F^\\star \\ne P$, then $c^\\top \\mu^\\star-c^\\top \\mu >0$ for some vertex $\\mu$, which implies $\\Delta>0$. \nTo simplify notation we denote the set $E\\coloneqq\\{\\mu-\\mu^\\star: \\mu\\in N(\\mu^\\star) \\setminus F^\\star, \\mu^\\star\\in \\operatorname{vert}(F^\\star)\\}$ of edges such that exactly one of the two endpoints is contained in $F^\\star$. \nThen, the polytope $P$ is contained in \n    \\[ F^\\star+C = \\left\\{ \\mu^\\star + \\sum_{e\\in E} \\alpha_e e : \\mu^\\star\\in F^\\star, \\alpha_e\\ge0 \\text{ for all } e\\in E \\right\\}, \\]\nsee \\Cref{app:lem:cone} and hence we can write $\\mu\\in P$ as $\\mu = \\mu^\\star + \\sum_{e} \\alpha_e e$ for some $\\mu^\\star\\in F^\\star$. \nUsing the triangle inequality we obtain \n    \\[ \\Delta \\lVert \\mu^\\star - \\mu \\rVert \\le \\Delta \\sum_{e\\in E}\\alpha_e \\lVert e \\rVert \\le - \\sum_{e\\in E} \\alpha_e c^\\top e = c^\\top \\mu^\\star - c^\\top\\mu. \\]\n\\end{proof}","start":44526,"end":45410},{"raw_tex":"\\begin{lemma}\\label{lem:KLEstimateL1}\nConsider a finite set $\\X$ and a probability distribution $\\mu\\in \\Delta_\\X$. Let $c>1$ and set $\\delta\\coloneqq \\frac{c-1}{c+1} \n    \\cdot \\min\\{ \\mu_x : \\mu_x>0 \\}>0$. \nThen for all $\\nu\\in\\Delta_\\X$ satisfying $\\lVert \\mu-\\nu\\rVert_\\infty\\le\\delta$ it holds that \n\\begin{equation}\\label{eq:KLvsL1}\n    \\KL (\\mu, \\nu) \\le c \\cdot \\lVert \\mu - \\nu \\rVert_{\\TV }. \n\\end{equation}\n\\end{lemma}","start":45412,"end":45841},{"raw_tex":"\\begin{proof}\nWe bound the individual summands in the KL-divergence\n    \\[ \\KL (\\mu, \\nu) = \\sum_{x\\in\\X} \\mu_x \\log\\left( \\frac{\\mu_x}{\\nu_x}\\right) = \\sum_{x\\in X} \\mu_x \\log\\left( \\frac{\\mu_x}{\\nu_x}\\right), \\]\nwhere $X\\coloneqq\\{x\\in\\X : \\mu_x>0\\}$. \nIf $\\mu_x,\\nu_x>0$ then \n\\begin{align}\\label{eq:klIndividual}\n    \\begin{split}\n        \\mu_x \\log \\left(\\frac{\\mu_x}{\\nu_x}\\right) & = \\mu_x \\left( \\log(\\nu_x+(\\mu_x-\\nu_x)) - \\log(\\nu_x) \\right) \\\\\n        & \\le \\mu_x \\left( \\log(\\nu_x) + \\frac{\\mu_x-\\nu_x}{\\nu_x} - \\log(\\nu_x) \\right) \n         = (\\mu_x - \\nu_x)\\cdot\\frac{\\mu_x}{\\nu_x},\n    \\end{split}\n\\end{align}\nwhere we used the convexity $\\log(t+h) \\le \\log(t) + h/t$ for $t>0, t+h>0$. \nWe set $\\varepsilon\\coloneqq\\frac{c-1}{2}\\in(0,1)$, such that \n\\[ \\delta = \\frac{\\varepsilon}{1+\\varepsilon}  \n    \\cdot \\min\\big\\{ \\mu_x : \\mu_x>0 \\big\\}. \n    \\]\nIf $\\lVert \\mu-\\nu\\rVert_\\infty\\le\\delta$ then \n    \\[ \\nu_x \\ge \\mu_x - \\delta \\ge \\mu_x \\left(1-\\frac{\\varepsilon}{1+\\varepsilon}\\right) = \\frac{\\mu_x}{1+\\varepsilon} \\]\nas well as\n    \\[ \\nu_x \\le \\mu_x + \\delta  \\le \\mu_x \\left(1 + \\frac{\\varepsilon}{1+\\varepsilon}\\right) \\le \\mu_x \\left(1 + \\frac{\\varepsilon}{1-\\varepsilon}\\right) = \\frac{\\mu_x}{1-\\varepsilon} \\]\nand therefore $1-\\varepsilon\\le \\frac{\\mu_x}{\\nu_x}\\le1+\\varepsilon$. \nIf $\\mu_x\\ge\\nu_x$ then \n    \\[ \n    (\\mu_x-\\nu_x)\\cdot \\frac{\\mu_x}{\\nu_x} \n    \\le (1+\\varepsilon)(\\mu_x-\\nu_x) \n    = \\mu_x-\\nu_x + \\varepsilon \\lvert \\mu_x - \\nu_x \\rvert , \n    \\]\nand if $\\mu_x<\\nu_x$ then \n\\begin{equation}\\label{eq:almostThere}\n    (\\mu_x-\\nu_x)\\cdot \\frac{\\mu_x}{\\nu_x} \\le (1-\\varepsilon)(\\mu_x-\\nu_x) = \\mu_x-\\nu_x + \\varepsilon \\lvert \\mu_x - \\nu_x \\rvert. \n\\end{equation}\nTogether with~\\eqref{eq:klIndividual} summing over $x$ yields \n\\begin{equation}\\label{eq:there}\n    \\KL (\\mu, \\nu) \\le \\sum_{x\\in X} (\\mu_x-\\nu_x) + \\varepsilon \\sum_{x\\in X} \\lvert \\mu_x - \\nu_x \\rvert \\le \\sum_{x\\in X} (\\mu_x-\\nu_x) + 2 \\varepsilon \\lVert \\mu-\\nu\\rVert_{\\TV } . \n\\end{equation}\nIt remains to estimate the first part. \nSetting $X^c\\coloneqq\\X\\setminus X$, we have \n\\[ \n\\sum_{x\\in X} (\\mu_x-\\nu_x) = \\sum_{x\\in\\X} (\\mu_x-\\nu_x) -  \\sum_{x\\in X^c} (\\mu_x-\\nu_x) = -  \\sum_{x\\in X^c} (\\mu_x-\\nu_x) = \\sum_{x\\in X^c} \\lvert  \\mu_x-\\nu_x \\rvert  \n\\]\nsince $\\mu_x=0$ for $x\\in X^c$. \nNow we can estimate\n\\begin{align}\\label{eq:lastStep}\n    2\\sum_{x\\in X} (\\mu_x-\\nu_x) = \\sum_{x\\in X} (\\mu_x-\\nu_x) + \\sum_{x\\in X^c} \\lvert  \\mu_x-\\nu_x \\rvert \\le \\lVert \\mu-\\nu\\rVert_1 = 2\\lVert \\mu-\\nu\\rVert_{\\TV }.\n\\end{align}\nCombining~\\eqref{eq:there} and~\\eqref{eq:lastStep} yields\n\\[ \\KL (\\mu, \\nu) \\le (1+2\\varepsilon)\\lVert \\mu - \\nu \\rVert_{\\TV } = c\\cdot\\lVert \\mu - \\nu \\rVert_{\\TV }. \\]\n\\end{proof}","start":45842,"end":48559},{"raw_tex":"\\begin{corollary}[Local KL-TV estimate]\\label{cor:TV-KL}\nConsider a finite set $\\X$ and a probability distribution $\\mu\\in \\Delta_\\X$. \nThen for all $\\nu\\in\\Delta_\\X$ satisfying \n\\begin{equation}\n    \\lVert \\mu - \\nu \\rVert_\\infty < \\min\\{ \\mu_x : \\mu_x>0 \\} \n\\end{equation}\nit holds that \n\\begin{equation}\\label{eq:KLvsL1-2}\n    \\KL (\\mu, \\nu) \\le \\frac{\\min\\{ \\mu_x : \\mu_x>0 \\} +  \\lVert \\mu-\\nu\\rVert_\\infty}{\\min\\{ \\mu_x : \\mu_x>0 \\} - \\lVert \\mu-\\nu\\rVert_\\infty} \\cdot \\lVert \\mu - \\nu \\rVert_{\\TV }. \n\\end{equation}\n\\end{corollary}","start":48562,"end":49101},{"raw_tex":"\\begin{proof}\n    This is a direct consequence of \\Cref{lem:KLEstimateL1}. \n    Indeed, for $\\varepsilon>0$ small enough we have $\\lVert \\mu - \\nu \\rVert_\\infty \\le \\frac{2-\\varepsilon}{2+\\varepsilon} \\cdot \\min\\{ \\mu_x : \\mu_x>0 \\}$ and thus by \\Cref{lem:KLEstimateL1} with $c=1+\\varepsilon$ we have $\\KL (\\mu, \\nu) \\le (1+\\varepsilon) \\lVert \\mu - \\nu \\rVert_{\\TV }$. \n    Note that $\\varepsilon>0$ was arbitrary. \n\\end{proof}","start":49102,"end":49530},{"raw_tex":"\\begin{proof}[Proof of \\Cref{thm:convergenceFRGF}]\n    Setting $\\delta\\coloneqq \\min\\{ \\mu_x^\\star : \\mu_x^\\star>0 \\}$ and using \\Cref{lem:TV-LP-estimate-withoutUniqueness} with $\\lVert\\cdot\\rVert_{\\TV}$ and \\Cref{cor:TV-KL} we have \n    \\begin{equation*}\n        c^\\top \\mu^\\star - c^\\top \\mu_t \\ge \\Delta \\lVert \\mu^\\star - \\mu_t \\rVert_{\\TV } \\ge \\Delta\\cdot\\frac{\\delta -  \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty}{\\delta + \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty} \\cdot \\KL (\\mu^\\star, \\mu_t),\n    \\end{equation*}\n    if $\\lVert \\mu^\\star - \\mu_t \\rVert_\\infty < \\delta$.\n    By \\Cref{prop:sublinearRate} we have \n    \\begin{align*}\n        \\lVert \\mu^\\star - \\mu_t \\rVert_\\infty \\le 2 \\lVert \\mu^\\star - \\mu_t \\rVert_{\\TV } \\le \\frac{2c^\\top(\\mu^\\star-\\mu_t)}\\Delta \\le \\frac{2\\KL (\\mu^\\star,\\mu_0)}{\\Delta \\cdot t}.\n    \\end{align*}\n    Hence, for $t>t_0$ we have $\\lVert \\mu^\\star - \\mu_t \\rVert_\\infty < \\delta$. \n    In this case, we can estimate \n    \\begin{align*}\n        \\frac{\\delta -  \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty}{\\delta + \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty} & \\le \\frac{\\delta -  2 \\KL (\\mu^\\star, \\mu_0)\\Delta^{-1} t^{-1}} {\\delta + 2 \\KL (\\mu^\\star, \\mu_0)\\Delta^{-1} t^{-1}} \n        = \\frac{t - t_0}{t + t_0} \\eqqcolon \\kappa_t. \n    \\end{align*}\n    Thus for $t>t_0$ we have $c^\\top \\mu^\\star - c^\\top \\mu_t \\ge \\Delta \\kappa_t \\KL (\\mu^\\star, \\mu_t)$, and \\Cref{prop:ratesTrajectories} together with \n    \\begin{equation*}\n        \\int_{t_0}^t \\frac{s-t_0}{s+t_0} \\D s = s - 2 t_0 \\log(s+t_0)\\Big|^{s=t}_{s=t_0} = (t-t_0) - 2 t_0 \\log\\left( \\frac{t+t_0}{2t_0} \\right)\n    \\end{equation*}\n    yield the result. \n\\end{proof}","start":49612,"end":51257},{"raw_tex":"\\begin{lemma}\\label{app:lem:cone}\n    Consider a polytope $P\\subseteq\\R^\\X$, a face $F\\subseteq P$ and consider the cone \n    \\begin{align*}\n        C\\coloneqq \\operatorname{cone}\\Big\\{ \\nu-\\mu : \\mu\\in\\operatorname{vert}(F), \\nu\\in N(\\mu)\\setminus F \\Big\\}, \n    \\end{align*}\n    which is generated by the edges pointing out of $F$. \n    Then we have $P\\subseteq F+C$. \n\\end{lemma}","start":106453,"end":106835},{"raw_tex":"\\begin{proof}\nThis is a generalization of~\\cite[Lemma 3.6]{ziegler2012lectures}, which covers the case that $F$ consists of a single vertex. \nWe will show that \n\\begin{align*}\n    F+C \\supseteq\\tilde P \\coloneqq \\bigcap \\Big\\{ H\\subseteq \\R^\\X : H \\text{ is a halfspace }, P\\subseteq H, H\\cap F\\ne\\emptyset \\Big\\} \\supseteq P, \n\\end{align*}\nfor which we pick an element $u\\in \\tilde P$. \nConsider a hyperplane $H = \\{\\mu:a^\\top \\mu = \\alpha\\}$ separating $F$ and $\\operatorname{vert}(P)\\setminus F$ and consider the \\emph{face figure} $P/F\\coloneqq P\\cap H$, which is a polytope. \nNow, we consider a translation $\\tilde H = \\{ \\mu : a^\\top\\mu= \\beta \\}$ of $H$,  such that $u\\in \\tilde H$. \nNow we have \n\\begin{align*}\n    \\tilde P = \\operatorname{conv}\\left\\{ \\mu + \\frac{a^\\top \\mu - \\beta}{a^\\top \\mu - a^\\top \\nu} \\cdot (\\nu-\\mu) : \\mu\\in\\operatorname{vert}(F), \\nu\\in N(\\mu)\\setminus F \\right\\},\n\\end{align*}\nsee~\\cite[Proposition 2.30]{zieglerdiscrete}. \nHence, we can choose convex weights $\\lambda_i$ such that \n\\begin{align*}\n    u = \\sum_{i} \\lambda_i (\\mu_i + \\alpha_i (\\nu_i-\\mu_i)) =  \\sum_{i} \\lambda_i \\mu_i + \\sum_{i} \\alpha_i \\lambda_i (\\nu_i-\\mu_i) \\in F + C,\n\\end{align*}\nwhere $\\mu_i\\in\\operatorname{vert}(F), \\nu_i \\in N(\\mu_i)\\setminus F$.\n\\end{proof}","start":106836,"end":108109}]},"nodes":[{"id":"S001","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"有限向量空间记号","source_units":[{"raw_tex":"For a finite set $\\X$, we denote the \\emph{free vector space} over $\\X$ by $\\R^\\X = \\{ \\mu\\colon \\X \\to \\R \\}$. \nIts elements can be identified with vectors $(\\mu_x)_{x\\in\\X}$. \nSimilarly, we denote the vectors with non-negative entries and positive entries by $\\R_{\\ge0}^\\X$ and $\\R_{>0}^\\X$, respectively. \nFor two elements $\\mu, \\nu\\in\\R^\\X$ we denote the \\emph{Hadamard product}, i.e., the entrywise product, between $\\mu$ and $\\nu$ by $\\mu\\odot\\nu\\in\\R^\\X$, so that $\\mu\\odot\\nu(x) \\coloneqq \\mu(x)\\nu(x)$.\n","start":14041,"end":14553}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"全变差范数","source_units":[{"raw_tex":"The \\emph{total variation} norm $\\lVert \\cdot \\rVert_{\\TV}\\colon\\R^\\X\\to\\R$ is given by $\\lVert \\mu \\rVert_{\\TV} \\coloneqq \\frac12\\sum_{x}\\lvert \\mu_x \\rvert$. \nFinally, with $\\mathds{1}_\\X\\in\\R^\\X$ we denote the all-one vector. \n\n","start":14553,"end":14784}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"多面体与顶点","source_units":[{"raw_tex":"A \\emph{polyhedron} is a set $P = \\{ \\mu\\in\\R^\\X : \\ell_i(\\mu) \\ge 0 \\text{ for } i=1, \\dots, k \\}\\subseteq \\R^\\X$, where $\\ell_i\\colon\\R^\\X\\to\\R$ are affine linear functions for $i=1, \\dots, k$. \nA bounded (and thus compact) polyhedron is called a \\emph{polytope}. \nA polytope can be shown to be the convex hull of finitely many extreme points, which are called \\emph{vertices} and which we denote by $\\operatorname{Vert}(P)$. \n","start":14784,"end":15213}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"相邻顶点","source_units":[{"raw_tex":"Two vertices $\\mu_1, \\mu_2\\in\\operatorname{Vert}(P)$ are called \\emph{neighbors} if the subspace $\\{ c\\in\\R^\\X : c^\\top \\mu_1 = c^\\top \\mu_2 = \\max_{\\mu\\in P} c^\\top \\mu \\}$ has dimension $\\lvert \\X \\rvert-1$. \nWe denote the set of all neighbors of a vertex $\\mu$ by $N(\\mu) \\subseteq\\operatorname{Vert}(P)$.  \n","start":15213,"end":15524}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"仿射与切空间","source_units":[{"raw_tex":"The \\emph{affine space} $\\operatorname{aff\\,span}(P)$ of a polytope $P\\subseteq\\R^\\X$ is the smallest affine subspace of $\\R^\\X$ containing $P$. \nThe relative interior $\\operatorname{int}(P)$ and boundary $\\partial P$ of $P$ are the interior and boundary of $P$ in its affine hull. \nFinally, the \\emph{tangent space} $TP$ of $P$ is given by the linear part of $\\operatorname{aff\\,span}(P)$. \n\n","start":15524,"end":15917}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"概率单纯形与支撑","source_units":[{"raw_tex":"We call $\\Delta_\\X \\coloneqq \\left\\{ \\mu\\in\\R^\\X_{\\ge0} : \\sum_{x} \\mu_x=1 \\right\\}$ the \\emph{probability simplex}. \nWe say that $\\mu\\in\\Delta_\\X$ is absolutely continuous with respect to $\\nu\\in\\Delta_\\X$ if $\\nu(x) = 0$ implies $\\mu(x)=0$ and write $\\mu \\ll \\nu$. \n","start":15917,"end":16185}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"Fisher-Rao 度量","source_units":[{"raw_tex":"The \\emph{Fisher-Rao metric} is a Riemannian metric on the positive orthant given by \n\\begin{equation}\n    g_{\\mu}^{\\FR}(v,w) \\coloneqq \\sum_{x\\in \\X} \\frac{v_x w_x}{\\mu_x} \\quad \\text{for all } v, w\\in\\R^{\\X}, \\mu\\in\\R^{\\X}_{>0},\n\\end{equation}\nwhere we denote the induced norm by $\\lVert v \\rVert_{g_\\mu^{\\FR}} \\coloneqq {g_{\\mu}^{\\FR}(v,v)}^{\\frac12}$. \n","start":18406,"end":18763}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"负熵的 Hessian 表示","source_units":[{"raw_tex":"The Fisher-Rao metric is closely connected to the negative {Shannon entropy} \n\\begin{equation}\n    \\phi(\\mu) = -H(\\mu) \\coloneqq \\sum_{x\\in\\X} \\mu_x \\log \\mu_x  \\quad \\text{for all } \\mu\\in\\R^{\\X}_{>0}\n\\end{equation}\nas it is induced by the Hessian of the (negative) entropy, meaning that we have $g_{\\mu}^{\\FR}(v,w) = v^\\top \\nabla^2 \\phi(\\mu) w$ for all $v, w\\in\\R^{\\X}, \\mu\\in\\R^{\\X}_{>0}$. \n","start":19970,"end":20365}]},{"id":"S009","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"Hessian 几何背景","source_units":[{"raw_tex":"As such, the Fisher-Rao metric falls into the class of \\emph{Hessian metrics} that have been studied in convex optimization; we refer to~\\cite{alvarez2004hessian, mueller2023thesis} for general well-posedness and convergence results. \n","start":20365,"end":20600}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"广义 KL 散度","source_units":[{"raw_tex":"An important concept in the analysis of Hessian gradient flows is the \\emph{Bregman divergence} induced by $\\phi$, which in the case of the negative entropy is given by the \\emph{KL-divergence} \n\\begin{equation}\n    \\KL (\\mu, \\nu) \\coloneqq  \\phi(\\mu) - \\phi(\\nu) - \\nabla \\phi(\\nu)(\\mu-\\nu) = \\sum_{x\\in \\X} \\mu_x \\log \\frac{\\mu_x}{\\nu_x} - \\sum_{x\\in \\X} \\mu_x + \\sum_{x\\in \\X} \\nu_x \n\\end{equation}\nfor $\\mu, \\nu\\in\\R_{\\ge0}^\\X$ with $\\mu\\ll\\nu$, where we use the common convention $0\\log\\frac00\\coloneqq0$. \n\n","start":20600,"end":21113}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"约束 Fisher-Rao 梯度","source_units":[{"raw_tex":"Consider now a continuously differentiable function $f\\colon\\R_{\\ge0}^\\X\\to\\R$ that we assume to be differentiable on $\\R_{>0}^\\X$ that we want to optimize over a polytope $P= \\R_{\\ge0}^{\\X}\\cap\\mathcal L$, where $\\mathcal L$ is a linear space. \nWe denote the gradient of $f\\colon \\R_{>0}^\\X\\to\\R$ at $\\mu\\in\\R_{>0}^\\X$ with respect to the Fisher-Rao metric by $\\nabla^{\\FR}f(\\mu)$ and call it the \\emph{Fisher-Rao gradient}. \nFurther, we denote the Fisher-Rao gradient of $f\\colon\\operatorname{int}(P)\\to\\R$ by $\\nabla^{\\FR}_P f(\\mu)\\in TP$, which is uniquely determined by \n\\begin{equation}\n    g_\\mu^{\\FR}(\\nabla^{\\FR}_P f(\\mu),v) = df(\\mu)v \\quad \\text{for all } v\\in TP.\n\\end{equation}\nNote that $\\nabla^{\\FR}_P f(\\mu)$ is the projection of $\\nabla^{\\FR}f(\\mu)$ with respect to the Fisher-Rao metric onto $TP$.\n","start":21113,"end":21929}]},{"id":"S012","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:characterizationFRGradient"],"declared_external":false,"source_only":false,"title":"约束梯度的 Hessian 刻画","source_units":[{"raw_tex":"By examining the definition of the Fisher-Rao metric we see that this is equivalent to \n\\begin{equation}\n\\label{eq:characterizationFRGradient}\n    \\langle \\nabla^2\\phi(\\mu)\\nabla^{\\FR}_P  f(\\mu), v \\rangle = \\langle \\nabla f(\\mu), v \\rangle \\quad \\text{for all } v\\in TP. \n\\end{equation}\n","start":21929,"end":22217}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["eq:FRGF"],"declared_external":false,"source_only":false,"title":"梯度上升流","source_units":[{"raw_tex":"    \nWe say that $(\\mu_t)_{t\\in [0, T)} \\subseteq \\operatorname{int}(P)$ solves the \\emph{Fisher-Rao gradient flow} if it solves the gradient flow with respect to the Fisher-Rao metric, i.e., if \n\\begin{equation}\\label{eq:FRGF}\n    \\partial_t \\mu_t = \\nabla^{\\FR}_P  f(\\mu_t) \\quad \\text{for all } t\\in[0, T). \n\\end{equation}\n","start":22217,"end":22543}]},{"id":"S014","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:explicitFRGF"],"declared_external":false,"source_only":false,"title":"梯度流弱形式","source_units":[{"raw_tex":"By using the characterization~\\eqref{eq:characterizationFRGradient} of $\\nabla^{\\FR}_P f(\\mu_t)$, we see that $(\\mu_t)_{t\\in [0, T)} \\subseteq \\operatorname{int}(P)$ solves the Fisher-Rao gradient flow~\\eqref{eq:FRGF}\nif and only if we have \n\\begin{equation}\\label{eq:explicitFRGF}\n    \\langle \\nabla^2 \\phi(\\mu_t) \\partial_t \\mu_t, v \\rangle = \\langle \\nabla f(\\mu_t), v \\rangle \n    \\quad \\text{for all } v\\in TP, t\\in[0, T). \n\\end{equation}\n","start":22543,"end":22987}]},{"id":"S015","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:LP"],"declared_external":false,"source_only":false,"title":"一般线性规划设定","source_units":[{"raw_tex":"In the remainder, we study linear programs and work in the following setting. \n\n\\begin{setting}\\label{setting:generalLP}\nWe consider a finite set $\\X$ and a linear program \n    \\begin{equation}\\label{eq:LP}\n        \\max c^\\top \\mu \\quad \\text{subject to } \\mu\\in P, \n    \\end{equation}\nwith cost $c\\in\\R^{\\X}$ and feasible region $P = \\R^\\X_{\\ge0} \\cap  \\mathcal L$ with $P \\cap \\R^\\X_{>0}\\ne\\emptyset$, where $\\mathcal L\\subseteq\\R^{\\X}$ is an affine space. \nBy $(\\mu_t)_{t\\in[0, T)}\\subseteq\\operatorname{int}(P)$ we denote a solution of the Fisher-Rao gradient flow~\\eqref{eq:FRGF} with initial condition $\\mu_0\\in P \\cap \\R_{>0}^\\X$ and potential $f(\\mu) = c^\\top \\mu$, where $T\\in\\R_{\\ge0}\\cup\\{+\\infty\\}$. \n\\end{setting}\n","start":22988,"end":23715}]},{"id":"S016","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"收敛速率术语","source_units":[{"raw_tex":"\\section{Convergence of Fisher-Rao Gradient Flows}\\label{sec:linearConvergeFRGF}\nWe have seen that Fisher-Rao gradient flows converge globally at a sublinear rate $O(t^{-1})$. \nWe now build on this analysis and show that once the gradient flow enters a vicinity of the optimizer, it converges at a quasi-linear rate $O(t^\\kappa e^{-\\Delta t})$, where $\\Delta>0$ depends on the geometry of the linear program and $\\kappa>0$ depends on the initial condition $\\mu_0$. \nNote that this yields $O(e^{-c t})$ convergence for all $c<\\Delta$ and hence we also simply talk of a linear convergence rate.\nWe consider linear programs of the following form. \n","start":27574,"end":28219}]},{"id":"S017","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:LP-inside-simplex"],"declared_external":false,"source_only":false,"title":"单纯形内线性规划","source_units":[{"raw_tex":"\\begin{setting}\\label{setting:LP-inside-Simplex}\nWe consider a finite set $\\X$ and a linear program \n    \\begin{equation}\\label{eq:LP-inside-simplex}\n        \\max c^\\top \\mu \\quad \\text{subject to } \\mu\\in P, \n    \\end{equation}\nwith cost $c\\in\\R^{\\X}$ and feasible region $P = \\Delta_\\X \\cap  \\mathcal L$ with $P \\cap \\R^\\X_{>0}\\ne\\emptyset$, where $\\mathcal L\\subseteq\\R^{\\X}$ is an affine space. \nBy $(\\mu_t)_{t\\ge0}\\subseteq\\operatorname{int}(P)$ we denote the solution of the Fisher-Rao gradient flow~\\eqref{eq:FRGF} with initial condition $\\mu_0\\in P \\cap \\R_{>0}^\\X$ and the potential $f(\\mu) = c^\\top \\mu$. \n\\end{setting}\n","start":28219,"end":28849}]},{"id":"S018","semantic_type":"theorem","kind":"theorem","labels":["eq:Delta","eq:definitiont_0","eq:linearConvergenceKL","eq:linearConvergenceValue","thm:convergenceFRGF"],"declared_external":false,"source_only":false,"title":"唯一最优解的显式收敛率","source_units":[{"raw_tex":"\\begin{restatable}[Linear convergence of Fisher-Rao GFs of LPs]{theorem}{convergenceFRGF}\n\\label{thm:convergenceFRGF}\nConsider Setting~\\ref{setting:LP-inside-Simplex} and assume that the linear program~\\eqref{eq:LP-inside-simplex} admits a unique solution $\\mu^\\star\\in P$. \nLet \n\\begin{align}\\label{eq:Delta}\n    \\Delta \\coloneqq \\min \\left\\{ \\frac{c^\\top \\mu^\\star - \n    c^\\top \\mu}{\\lVert \\mu^\\star - \\mu \\rVert_{\\TV }} : \\mu\\in N(\\mu^\\star)  \\right\\},\n\\end{align}\nwhere $N(\\mu^\\star)$ denotes the set of neighboring vertices of $\\mu^\\star$ and set \n\\begin{equation}\\label{eq:definitiont_0}\n    t_0 \\coloneqq \\frac{2 \\KL (\\mu^\\star, \\mu_0)}{\\Delta \\cdot \\min\\{ \\mu^\\star_x : \\mu^\\star_x>0 \\}} . \n\\end{equation}\nThen for any $t\\ge t_0$ we have \n\\begin{align}\\label{eq:linearConvergenceKL}\n    \\KL (\\mu^\\star, \\mu_t) \\le \\KL (\\mu^\\star, \\mu_0) \\exp\\left( -\\Delta(t-t_0) +  2 t_0 \\Delta \n    \\log\\left(\\frac{t+t_0}{2t_0}\\right) \\right), \n\\end{align}\nas well as \n\\begin{align}\\label{eq:linearConvergenceValue}\n    c^\\top \\mu^\\star - c^\\top \\mu_t \\le \\Delta \\KL (\\mu^\\star, \\mu_0) \\exp\\left( -\\Delta(t-t_0) +  2 t_0 \\Delta \\log\\left(\\frac{t+t_0}{2t_0}\\right) \\right). \n\\end{align}\n\\end{restatable}","start":29173,"end":30369}]},{"id":"S019","semantic_type":"proposition","kind":"proposition","labels":["eq:centralPath"],"declared_external":false,"source_only":false,"title":"中心路径性质","source_units":[{"raw_tex":"\\begin{proposition}[Central path property,\\cite{alvarez2004hessian}]\n    Consider Setting~\\ref{setting:generalLP}. \n    Then $\\mu_t$ is uniquely characterized by \n    \\begin{equation}\\label{eq:centralPath}\n        \\mu_t = \\argmax\\left\\{ c^\\top \\mu - t^{-1} \\KL (\\mu, \\mu_0) : \\mu \\in P \\right\\} \\quad \\text{for all } t\\in(0,T). \n    \\end{equation}\n\\end{proposition}","start":24059,"end":24424}]},{"id":"S020","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"中心路径的作者证明","source_units":[{"raw_tex":"\\begin{proof}\n    Let $\\hat{\\mu}_t\\in P$ denote the unique maximizer of $g(\\mu)\\coloneqq c^\\top\\mu - t^{-1} \\KL (\\mu, \\mu_0)$ over $P$ for $t>0$, then surely $\\hat{\\mu}_t\\in \\operatorname{int}(P)$. \n    Thus, $\\hat{\\mu}_t$ is uniquely determined by $\\langle \\nabla g(\\hat{\\mu}_t), v\\rangle = 0$ for all $v\\in TP$. \n    Direct computation yields $\\nabla g({\\mu}) = c - t^{-1} (\\nabla\\phi(\\mu) - \\nabla\\phi(\\mu_0))$ and hence $\\hat{\\mu}_t$ is uniquely determined by \n    \\begin{equation*}\n        t\\langle c, v\\rangle = \\langle \\nabla\\phi(\\hat{\\mu}_t) - \\nabla\\phi(\\mu_0), v\\rangle \\quad \\text{for all } v\\in TP. \n    \\end{equation*} \n    On the other hand, for the gradient flow, we can use~\\eqref{eq:explicitFRGF} and compute for $v\\in TP$ \n    \\begin{align*}\n        \\langle \\nabla\\phi(\\mu_t) - \\nabla\\phi(\\mu_0), v \\rangle & = \\int_{0}^t \\partial_s \\langle \\nabla \\phi(\\mu_s), v \\rangle \\D s \n        = \\int_{0}^t \\langle \\nabla^2 \\phi(\\mu_s)\\partial_s \\mu_s, v \\rangle \\D s \n        \\\\ & = \\int_{0}^t \\langle \\nabla f(\\mu_s) \\mu_s, v \\rangle \\D s \n        = \\int_{0}^t \\langle c, v \\rangle \\D s = t \\langle c, v \\rangle . \n    \\end{align*}\n    This shows $\\mu_t = \\hat{\\mu}_t$ as claimed. \n\\end{proof}","start":24429,"end":25633}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"正则化解的一阶条件","source_units":[{"raw_tex":"    Let $\\hat{\\mu}_t\\in P$ denote the unique maximizer of $g(\\mu)\\coloneqq c^\\top\\mu - t^{-1} \\KL (\\mu, \\mu_0)$ over $P$ for $t>0$, then surely $\\hat{\\mu}_t\\in \\operatorname{int}(P)$. \n    Thus, $\\hat{\\mu}_t$ is uniquely determined by $\\langle \\nabla g(\\hat{\\mu}_t), v\\rangle = 0$ for all $v\\in TP$. \n    Direct computation yields $\\nabla g({\\mu}) = c - t^{-1} (\\nabla\\phi(\\mu) - \\nabla\\phi(\\mu_0))$ and hence $\\hat{\\mu}_t$ is uniquely determined by \n    \\begin{equation*}\n        t\\langle c, v\\rangle = \\langle \\nabla\\phi(\\hat{\\mu}_t) - \\nabla\\phi(\\mu_0), v\\rangle \\quad \\text{for all } v\\in TP. \n    \\end{equation*} \n","start":24443,"end":25062}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"积分识别中心路径","source_units":[{"raw_tex":"    On the other hand, for the gradient flow, we can use~\\eqref{eq:explicitFRGF} and compute for $v\\in TP$ \n    \\begin{align*}\n        \\langle \\nabla\\phi(\\mu_t) - \\nabla\\phi(\\mu_0), v \\rangle & = \\int_{0}^t \\partial_s \\langle \\nabla \\phi(\\mu_s), v \\rangle \\D s \n        = \\int_{0}^t \\langle \\nabla^2 \\phi(\\mu_s)\\partial_s \\mu_s, v \\rangle \\D s \n        \\\\ & = \\int_{0}^t \\langle \\nabla f(\\mu_s) \\mu_s, v \\rangle \\D s \n        = \\int_{0}^t \\langle c, v \\rangle \\D s = t \\langle c, v \\rangle . \n    \\end{align*}\n    This shows $\\mu_t = \\hat{\\mu}_t$ as claimed. \n","start":25062,"end":25622}]},{"id":"S023","semantic_type":"corollary","kind":"corollary","labels":["eq:sublinearConvergence","prop:sublinearRate"],"declared_external":false,"source_only":false,"title":"次线性目标误差界","source_units":[{"raw_tex":"\\begin{corollary}[Sublinear convergence rate,\\cite{alvarez2004hessian}]\\label{prop:sublinearRate}\n    Consider Setting~\\ref{setting:generalLP} and assume that the linear program~\\eqref{eq:LP} admits a solution $\\mu^\\star\\in P$. \n    Then for $\\mu\\in P$ it holds that \n\\begin{equation}\\label{eq:sublinearConvergence}\n        c^\\top \\mu^\\star - c^\\top\\mu_t \\le \\frac{\\KL (\\mu^\\star, \\mu) - \\KL (\\mu_t, \\mu)}t \\le \\frac{\\KL (\\mu^\\star, \\mu_0)}t \\quad \\text{for all } t\\in[0, T). \n    \\end{equation}\n\\end{corollary}","start":25809,"end":26320}]},{"id":"S024","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"次线性界的作者证明","source_units":[{"raw_tex":"\\begin{proof}\n    We have $c^\\top \\mu_t - t^{-1} \\KL (\\mu_t, \\mu) \\ge c^\\top \\mu^\\star - t^{-1} \\KL (\\mu^\\star, \\mu)$ by the central path property.\n    Rearranging yields the result. \n\\end{proof}","start":26321,"end":26516}]},{"id":"S025","semantic_type":"theorem","kind":"theorem","labels":[],"declared_external":false,"source_only":false,"title":"梯度流全局适定性","source_units":[{"raw_tex":"\\begin{theorem}[Well-posedness of  FR GFs,\\cite{alvarez2004hessian}]\n    Consider Setting~\\ref{setting:generalLP}. \n    Then there exists a unique global solution $(\\mu_{t})_{t\\ge0}\\subseteq\\operatorname{int}(P)$ of the Fisher-Rao gradient flow~\\eqref{eq:FRGF}. \n\\end{theorem}","start":26811,"end":27087}]},{"id":"S026","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"适定性的作者证明","source_units":[{"raw_tex":"\\begin{proof}\n    The local existence and uniqueness follow from the Picard-Lindelöf theorem~\\cite{teschl2024ordinary}. Hence, it suffices to show that the Fisher-Rao gradient flow does not hit the boundary $\\partial P$ in finite time. \n    By the central path property, this is equivalent to the statement that the solutions of all KL-regularized problems~\\eqref{eq:centralPath} lie in the interior $\\operatorname{int}(P)$ of the polyhedron, which can be easily checked. \n\\end{proof}","start":27088,"end":27572}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"局部存在唯一性","source_units":[{"raw_tex":"    The local existence and uniqueness follow from the Picard-Lindelöf theorem~\\cite{teschl2024ordinary}. Hence, it suffices to show that the Fisher-Rao gradient flow does not hit the boundary $\\partial P$ in finite time. \n","start":27102,"end":27325}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"中心路径排除触边","source_units":[{"raw_tex":"    By the central path property, this is equivalent to the statement that the solutions of all KL-regularized problems~\\eqref{eq:centralPath} lie in the interior $\\operatorname{int}(P)$ of the polyhedron, which can be easily checked. \n","start":27325,"end":27561}]},{"id":"S029","semantic_type":"lemma","kind":"lemma","labels":["eq:lem:KLConvergence","eq:lem:convergenceValue","eq:strongConvexity","prop:ratesTrajectories"],"declared_external":false,"source_only":false,"title":"沿轨迹支配推出速率","source_units":[{"raw_tex":"\\begin{lemma}\\label{prop:ratesTrajectories} \n    Consider Setting~\\ref{setting:generalLP} and assume that there is an optimizer $\\mu^\\star\\in P$ and $\\kappa_t>0$ for $t>t_0\\ge0$ such that\n\\begin{equation}\\label{eq:strongConvexity}\n    c^\\top \\mu^\\star - c^\\top \\mu_t \\ge \\kappa_t \\KL (\\mu^\\star, \\mu_t) \\quad \\text{for all } t> t_0. \n\\end{equation}\nThen we have \n\\begin{equation}\\label{eq:lem:KLConvergence}\n    \\KL (\\mu^\\star, \\mu_t) \\le \\KL (\\mu^\\star, \\mu_{0}) \\exp\\left( -\\int_{t_0}^t \\kappa_s \\D s \\right) \\quad \\text{for all } t\\ge t_0,\n\\end{equation}\nas well as \n\\begin{equation}\\label{eq:lem:convergenceValue}\n    c^\\top \\mu^\\star - c^\\top \\mu_t \\le \\kappa_t \\KL (\\mu^\\star, \\mu_{0}) \\exp\\left( -\\int_{t_0}^t \\kappa_s \\D s \\right) \\quad \\text{for all } t\\ge t_0.\n\\end{equation}\n\\end{lemma}","start":40737,"end":41534}]},{"id":"S030","semantic_type":"assumption","kind":"assumption","labels":["eq:strongConvexity","prop:ratesTrajectories"],"declared_external":false,"source_only":false,"title":"沿轨迹 KL 支配条件","source_units":[{"raw_tex":"    Consider Setting~\\ref{setting:generalLP} and assume that there is an optimizer $\\mu^\\star\\in P$ and $\\kappa_t>0$ for $t>t_0\\ge0$ such that\n\\begin{equation}\\label{eq:strongConvexity}\n    c^\\top \\mu^\\star - c^\\top \\mu_t \\ge \\kappa_t \\KL (\\mu^\\star, \\mu_t) \\quad \\text{for all } t> t_0. \n\\end{equation}\n","start":40782,"end":41086}]},{"id":"S031","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:lem:KLConvergence","prop:ratesTrajectories"],"declared_external":false,"source_only":false,"title":"轨迹引理的 KL 分结论","source_units":[{"raw_tex":"Then we have \n\\begin{equation}\\label{eq:lem:KLConvergence}\n    \\KL (\\mu^\\star, \\mu_t) \\le \\KL (\\mu^\\star, \\mu_{0}) \\exp\\left( -\\int_{t_0}^t \\kappa_s \\D s \\right) \\quad \\text{for all } t\\ge t_0,\n\\end{equation}\n","start":41086,"end":41295}]},{"id":"S032","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:lem:convergenceValue","prop:ratesTrajectories"],"declared_external":false,"source_only":false,"title":"轨迹引理的目标值分结论","source_units":[{"raw_tex":"as well as \n\\begin{equation}\\label{eq:lem:convergenceValue}\n    c^\\top \\mu^\\star - c^\\top \\mu_t \\le \\kappa_t \\KL (\\mu^\\star, \\mu_{0}) \\exp\\left( -\\int_{t_0}^t \\kappa_s \\D s \\right) \\quad \\text{for all } t\\ge t_0.\n\\end{equation}\n","start":41295,"end":41523}]},{"id":"S033","semantic_type":"lemma","kind":"lemma","labels":["eq:derivativeu","lem:evolutionKL"],"declared_external":false,"source_only":false,"title":"KL 沿流演化恒等式","source_units":[{"raw_tex":"\\begin{lemma}[\\cite{alvarez2004hessian}]\\label{lem:evolutionKL}\nConsider Setting~\\ref{setting:generalLP}, whereby we allow $f\\colon\\R_{>0}^\\X\\to\\R$ to be an arbitrary differentiable function, and fix $\\mu\\in P$. \nThen for any $t\\ge0$, it holds that \n    \\begin{equation}\\label{eq:derivativeu}\n        \\partial_t \\KL (\\mu, \\mu_t) = \\langle \\nabla f(\\mu_t), \\mu_t-\\mu\\rangle. \n    \\end{equation}\n\\end{lemma}","start":41602,"end":42007}]},{"id":"S034","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"KL 演化的作者证明","source_units":[{"raw_tex":"\\begin{proof}\n    Denoting the negative Shannon entropy by $\\phi$, we compute \n    \\begin{align*}\n        \\partial_t \\KL (\\mu, \\mu_t) = - \\partial_t \\phi(\\mu_t) - \\partial_t \\langle \\nabla \\phi(\\mu_t), \\mu - \\mu_t \\rangle = \\langle \\nabla^2 \\phi(\\mu_t)\\partial_t \\mu_t, \\mu_t - \\mu \\rangle.\n    \\end{align*}\n    Now~\\eqref{eq:explicitFRGF} yields the claim. \n\\end{proof}","start":42008,"end":42378}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"对 Bregman 形式求导","source_units":[{"raw_tex":"    Denoting the negative Shannon entropy by $\\phi$, we compute \n    \\begin{align*}\n        \\partial_t \\KL (\\mu, \\mu_t) = - \\partial_t \\phi(\\mu_t) - \\partial_t \\langle \\nabla \\phi(\\mu_t), \\mu - \\mu_t \\rangle = \\langle \\nabla^2 \\phi(\\mu_t)\\partial_t \\mu_t, \\mu_t - \\mu \\rangle.\n    \\end{align*}\n","start":42022,"end":42316}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"代入流的弱形式","source_units":[{"raw_tex":"    Now~\\eqref{eq:explicitFRGF} yields the claim. \n","start":42316,"end":42367}]},{"id":"S037","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"轨迹速率引理的作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\Cref{prop:ratesTrajectories}]\nUsing~\\eqref{eq:derivativeu} and~\\eqref{eq:strongConvexity} we find that for all $t\\ge T$ it holds that \n$\\partial_t \\KL (\\mu^\\star, \\mu_t) = c^\\top \\mu_t  - c^\\top \\mu^\\star \\le - \\kappa_t \\KL (\\mu^\\star, \\mu_t)$. \nNow Gronwall's inequality yields~\\eqref{eq:lem:KLConvergence}. \nBy \\Cref{prop:sublinearRate} we have for any $h>0$ that \n    \\[ c^\\top \\mu^\\star - c^\\top \\mu_t \\le \\frac{\\KL (\\mu^\\star, \\mu_{t-h})}{h} \\le  \\KL (\\mu^\\star, \\mu_{0}) \\cdot \\frac{\\exp\\left(-\\int_{t_0}^{t-h} \\kappa_s \\D s \\right)}{h}. \\] \nTaking the limit $h\\to0$ yields~\\eqref{eq:lem:convergenceValue}. \n\\end{proof}","start":42379,"end":43028}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"微分不等式与 Gronwall","source_units":[{"raw_tex":"Using~\\eqref{eq:derivativeu} and~\\eqref{eq:strongConvexity} we find that for all $t\\ge T$ it holds that \n$\\partial_t \\KL (\\mu^\\star, \\mu_t) = c^\\top \\mu_t  - c^\\top \\mu^\\star \\le - \\kappa_t \\KL (\\mu^\\star, \\mu_t)$. \nNow Gronwall's inequality yields~\\eqref{eq:lem:KLConvergence}. \n","start":42433,"end":42713}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"目标值界的极限尝试","source_units":[{"raw_tex":"By \\Cref{prop:sublinearRate} we have for any $h>0$ that \n    \\[ c^\\top \\mu^\\star - c^\\top \\mu_t \\le \\frac{\\KL (\\mu^\\star, \\mu_{t-h})}{h} \\le  \\KL (\\mu^\\star, \\mu_{0}) \\cdot \\frac{\\exp\\left(-\\int_{t_0}^{t-h} \\kappa_s \\D s \\right)}{h}. \\] \nTaking the limit $h\\to0$ yields~\\eqref{eq:lem:convergenceValue}. \n","start":42713,"end":43017}]},{"id":"S040","semantic_type":"lemma","kind":"lemma","labels":["eq:gapVSNorm-2","lem:TV-LP-estimate-withoutUniqueness"],"declared_external":false,"source_only":false,"title":"邻边控制最优面距离","source_units":[{"raw_tex":"\\begin{restatable}{lemma}{TVLPEstimateWU}\n\\label{lem:TV-LP-estimate-withoutUniqueness}\nConsider a polytope $P \\subseteq \\R^{\\X}$ and denote by $F^\\star$ the face of maximizers of the linear function $\\mu\\mapsto c^\\top \\mu$ over $P$. \nDenote the set of neighboring vertices of a vertex $\\mu$ by $N(\\mu)$ and let $\\lVert \\cdot\\rVert\\colon\\R^{\\X}\\to\\R_{\\ge0}$ be an arbitrary semi-norm. \nThen either $F^\\star = P$ or with $\\frac{c}{0}\\coloneqq+\\infty$ for $c>0$, we have \n\\begin{equation}\n    \\Delta\\coloneqq \\min \\left\\{ \\frac{c^\\top\\mu^\\star - c^\\top\\mu }{\\lVert \\mu^\\star - \\mu \\rVert} : \\mu^\\star \\in\\operatorname{vert}(F^\\star), \\mu\\in N(\\mu^\\star)\\setminus F^\\star \\right\\} > 0,\n\\end{equation}\nand further \n\\begin{equation}\\label{eq:gapVSNorm-2}\n     c^\\top \\mu^\\star - c^\\top \\mu \\ge \\Delta \\cdot \\inf_{\\mu^\\star\\in F^\\star}  \\lVert \\mu^\\star - \\mu\\rVert \\quad \\text{for all } \\mu\\in P.\n\\end{equation}\n\\end{restatable}","start":43603,"end":44525}]},{"id":"S041","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"目标差几何界的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nIf $F^\\star \\ne P$, then $c^\\top \\mu^\\star-c^\\top \\mu >0$ for some vertex $\\mu$, which implies $\\Delta>0$. \nTo simplify notation we denote the set $E\\coloneqq\\{\\mu-\\mu^\\star: \\mu\\in N(\\mu^\\star) \\setminus F^\\star, \\mu^\\star\\in \\operatorname{vert}(F^\\star)\\}$ of edges such that exactly one of the two endpoints is contained in $F^\\star$. \nThen, the polytope $P$ is contained in \n    \\[ F^\\star+C = \\left\\{ \\mu^\\star + \\sum_{e\\in E} \\alpha_e e : \\mu^\\star\\in F^\\star, \\alpha_e\\ge0 \\text{ for all } e\\in E \\right\\}, \\]\nsee \\Cref{app:lem:cone} and hence we can write $\\mu\\in P$ as $\\mu = \\mu^\\star + \\sum_{e} \\alpha_e e$ for some $\\mu^\\star\\in F^\\star$. \nUsing the triangle inequality we obtain \n    \\[ \\Delta \\lVert \\mu^\\star - \\mu \\rVert \\le \\Delta \\sum_{e\\in E}\\alpha_e \\lVert e \\rVert \\le - \\sum_{e\\in E} \\alpha_e c^\\top e = c^\\top \\mu^\\star - c^\\top\\mu. \\]\n\\end{proof}","start":44526,"end":45410}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"正间隙与出面边","source_units":[{"raw_tex":"If $F^\\star \\ne P$, then $c^\\top \\mu^\\star-c^\\top \\mu >0$ for some vertex $\\mu$, which implies $\\Delta>0$. \nTo simplify notation we denote the set $E\\coloneqq\\{\\mu-\\mu^\\star: \\mu\\in N(\\mu^\\star) \\setminus F^\\star, \\mu^\\star\\in \\operatorname{vert}(F^\\star)\\}$ of edges such that exactly one of the two endpoints is contained in $F^\\star$. \n","start":44540,"end":44879}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"可行点的锥分解","source_units":[{"raw_tex":"Then, the polytope $P$ is contained in \n    \\[ F^\\star+C = \\left\\{ \\mu^\\star + \\sum_{e\\in E} \\alpha_e e : \\mu^\\star\\in F^\\star, \\alpha_e\\ge0 \\text{ for all } e\\in E \\right\\}, \\]\nsee \\Cref{app:lem:cone} and hence we can write $\\mu\\in P$ as $\\mu = \\mu^\\star + \\sum_{e} \\alpha_e e$ for some $\\mu^\\star\\in F^\\star$. \n","start":44879,"end":45192}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"逐边相加得到距离界","source_units":[{"raw_tex":"Using the triangle inequality we obtain \n    \\[ \\Delta \\lVert \\mu^\\star - \\mu \\rVert \\le \\Delta \\sum_{e\\in E}\\alpha_e \\lVert e \\rVert \\le - \\sum_{e\\in E} \\alpha_e c^\\top e = c^\\top \\mu^\\star - c^\\top\\mu. \\]\n","start":45192,"end":45399}]},{"id":"S045","semantic_type":"lemma","kind":"lemma","labels":["eq:KLvsL1","lem:KLEstimateL1"],"declared_external":false,"source_only":false,"title":"带参数的局部 KL-TV 界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:KLEstimateL1}\nConsider a finite set $\\X$ and a probability distribution $\\mu\\in \\Delta_\\X$. Let $c>1$ and set $\\delta\\coloneqq \\frac{c-1}{c+1} \n    \\cdot \\min\\{ \\mu_x : \\mu_x>0 \\}>0$. \nThen for all $\\nu\\in\\Delta_\\X$ satisfying $\\lVert \\mu-\\nu\\rVert_\\infty\\le\\delta$ it holds that \n\\begin{equation}\\label{eq:KLvsL1}\n    \\KL (\\mu, \\nu) \\le c \\cdot \\lVert \\mu - \\nu \\rVert_{\\TV }. \n\\end{equation}\n\\end{lemma}","start":45412,"end":45841}]},{"id":"S046","semantic_type":"proof","kind":"proof","labels":["eq:almostThere","eq:klIndividual","eq:lastStep","eq:there"],"declared_external":false,"source_only":false,"title":"局部 KL-TV 界的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nWe bound the individual summands in the KL-divergence\n    \\[ \\KL (\\mu, \\nu) = \\sum_{x\\in\\X} \\mu_x \\log\\left( \\frac{\\mu_x}{\\nu_x}\\right) = \\sum_{x\\in X} \\mu_x \\log\\left( \\frac{\\mu_x}{\\nu_x}\\right), \\]\nwhere $X\\coloneqq\\{x\\in\\X : \\mu_x>0\\}$. \nIf $\\mu_x,\\nu_x>0$ then \n\\begin{align}\\label{eq:klIndividual}\n    \\begin{split}\n        \\mu_x \\log \\left(\\frac{\\mu_x}{\\nu_x}\\right) & = \\mu_x \\left( \\log(\\nu_x+(\\mu_x-\\nu_x)) - \\log(\\nu_x) \\right) \\\\\n        & \\le \\mu_x \\left( \\log(\\nu_x) + \\frac{\\mu_x-\\nu_x}{\\nu_x} - \\log(\\nu_x) \\right) \n         = (\\mu_x - \\nu_x)\\cdot\\frac{\\mu_x}{\\nu_x},\n    \\end{split}\n\\end{align}\nwhere we used the convexity $\\log(t+h) \\le \\log(t) + h/t$ for $t>0, t+h>0$. \nWe set $\\varepsilon\\coloneqq\\frac{c-1}{2}\\in(0,1)$, such that \n\\[ \\delta = \\frac{\\varepsilon}{1+\\varepsilon}  \n    \\cdot \\min\\big\\{ \\mu_x : \\mu_x>0 \\big\\}. \n    \\]\nIf $\\lVert \\mu-\\nu\\rVert_\\infty\\le\\delta$ then \n    \\[ \\nu_x \\ge \\mu_x - \\delta \\ge \\mu_x \\left(1-\\frac{\\varepsilon}{1+\\varepsilon}\\right) = \\frac{\\mu_x}{1+\\varepsilon} \\]\nas well as\n    \\[ \\nu_x \\le \\mu_x + \\delta  \\le \\mu_x \\left(1 + \\frac{\\varepsilon}{1+\\varepsilon}\\right) \\le \\mu_x \\left(1 + \\frac{\\varepsilon}{1-\\varepsilon}\\right) = \\frac{\\mu_x}{1-\\varepsilon} \\]\nand therefore $1-\\varepsilon\\le \\frac{\\mu_x}{\\nu_x}\\le1+\\varepsilon$. \nIf $\\mu_x\\ge\\nu_x$ then \n    \\[ \n    (\\mu_x-\\nu_x)\\cdot \\frac{\\mu_x}{\\nu_x} \n    \\le (1+\\varepsilon)(\\mu_x-\\nu_x) \n    = \\mu_x-\\nu_x + \\varepsilon \\lvert \\mu_x - \\nu_x \\rvert , \n    \\]\nand if $\\mu_x<\\nu_x$ then \n\\begin{equation}\\label{eq:almostThere}\n    (\\mu_x-\\nu_x)\\cdot \\frac{\\mu_x}{\\nu_x} \\le (1-\\varepsilon)(\\mu_x-\\nu_x) = \\mu_x-\\nu_x + \\varepsilon \\lvert \\mu_x - \\nu_x \\rvert. \n\\end{equation}\nTogether with~\\eqref{eq:klIndividual} summing over $x$ yields \n\\begin{equation}\\label{eq:there}\n    \\KL (\\mu, \\nu) \\le \\sum_{x\\in X} (\\mu_x-\\nu_x) + \\varepsilon \\sum_{x\\in X} \\lvert \\mu_x - \\nu_x \\rvert \\le \\sum_{x\\in X} (\\mu_x-\\nu_x) + 2 \\varepsilon \\lVert \\mu-\\nu\\rVert_{\\TV } . \n\\end{equation}\nIt remains to estimate the first part. \nSetting $X^c\\coloneqq\\X\\setminus X$, we have \n\\[ \n\\sum_{x\\in X} (\\mu_x-\\nu_x) = \\sum_{x\\in\\X} (\\mu_x-\\nu_x) -  \\sum_{x\\in X^c} (\\mu_x-\\nu_x) = -  \\sum_{x\\in X^c} (\\mu_x-\\nu_x) = \\sum_{x\\in X^c} \\lvert  \\mu_x-\\nu_x \\rvert  \n\\]\nsince $\\mu_x=0$ for $x\\in X^c$. \nNow we can estimate\n\\begin{align}\\label{eq:lastStep}\n    2\\sum_{x\\in X} (\\mu_x-\\nu_x) = \\sum_{x\\in X} (\\mu_x-\\nu_x) + \\sum_{x\\in X^c} \\lvert  \\mu_x-\\nu_x \\rvert \\le \\lVert \\mu-\\nu\\rVert_1 = 2\\lVert \\mu-\\nu\\rVert_{\\TV }.\n\\end{align}\nCombining~\\eqref{eq:there} and~\\eqref{eq:lastStep} yields\n\\[ \\KL (\\mu, \\nu) \\le (1+2\\varepsilon)\\lVert \\mu - \\nu \\rVert_{\\TV } = c\\cdot\\lVert \\mu - \\nu \\rVert_{\\TV }. \\]\n\\end{proof}","start":45842,"end":48559}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"限制到参考分布支撑","source_units":[{"raw_tex":"We bound the individual summands in the KL-divergence\n    \\[ \\KL (\\mu, \\nu) = \\sum_{x\\in\\X} \\mu_x \\log\\left( \\frac{\\mu_x}{\\nu_x}\\right) = \\sum_{x\\in X} \\mu_x \\log\\left( \\frac{\\mu_x}{\\nu_x}\\right), \\]\nwhere $X\\coloneqq\\{x\\in\\X : \\mu_x>0\\}$. \n","start":45856,"end":46097}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":["eq:klIndividual"],"declared_external":false,"source_only":false,"title":"逐坐标对数上界","source_units":[{"raw_tex":"If $\\mu_x,\\nu_x>0$ then \n\\begin{align}\\label{eq:klIndividual}\n    \\begin{split}\n        \\mu_x \\log \\left(\\frac{\\mu_x}{\\nu_x}\\right) & = \\mu_x \\left( \\log(\\nu_x+(\\mu_x-\\nu_x)) - \\log(\\nu_x) \\right) \\\\\n        & \\le \\mu_x \\left( \\log(\\nu_x) + \\frac{\\mu_x-\\nu_x}{\\nu_x} - \\log(\\nu_x) \\right) \n         = (\\mu_x - \\nu_x)\\cdot\\frac{\\mu_x}{\\nu_x},\n    \\end{split}\n\\end{align}\nwhere we used the convexity $\\log(t+h) \\le \\log(t) + h/t$ for $t>0, t+h>0$. \n","start":46097,"end":46544}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"邻域内的坐标比值","source_units":[{"raw_tex":"We set $\\varepsilon\\coloneqq\\frac{c-1}{2}\\in(0,1)$, such that \n\\[ \\delta = \\frac{\\varepsilon}{1+\\varepsilon}  \n    \\cdot \\min\\big\\{ \\mu_x : \\mu_x>0 \\big\\}. \n    \\]\nIf $\\lVert \\mu-\\nu\\rVert_\\infty\\le\\delta$ then \n    \\[ \\nu_x \\ge \\mu_x - \\delta \\ge \\mu_x \\left(1-\\frac{\\varepsilon}{1+\\varepsilon}\\right) = \\frac{\\mu_x}{1+\\varepsilon} \\]\nas well as\n    \\[ \\nu_x \\le \\mu_x + \\delta  \\le \\mu_x \\left(1 + \\frac{\\varepsilon}{1+\\varepsilon}\\right) \\le \\mu_x \\left(1 + \\frac{\\varepsilon}{1-\\varepsilon}\\right) = \\frac{\\mu_x}{1-\\varepsilon} \\]\nand therefore $1-\\varepsilon\\le \\frac{\\mu_x}{\\nu_x}\\le1+\\varepsilon$. \n","start":46544,"end":47150}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":["eq:almostThere"],"declared_external":false,"source_only":false,"title":"按坐标差符号估计","source_units":[{"raw_tex":"If $\\mu_x\\ge\\nu_x$ then \n    \\[ \n    (\\mu_x-\\nu_x)\\cdot \\frac{\\mu_x}{\\nu_x} \n    \\le (1+\\varepsilon)(\\mu_x-\\nu_x) \n    = \\mu_x-\\nu_x + \\varepsilon \\lvert \\mu_x - \\nu_x \\rvert , \n    \\]\nand if $\\mu_x<\\nu_x$ then \n\\begin{equation}\\label{eq:almostThere}\n    (\\mu_x-\\nu_x)\\cdot \\frac{\\mu_x}{\\nu_x} \\le (1-\\varepsilon)(\\mu_x-\\nu_x) = \\mu_x-\\nu_x + \\varepsilon \\lvert \\mu_x - \\nu_x \\rvert. \n\\end{equation}\n","start":47150,"end":47550}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":["eq:there"],"declared_external":false,"source_only":false,"title":"求和得到支撑内余项","source_units":[{"raw_tex":"Together with~\\eqref{eq:klIndividual} summing over $x$ yields \n\\begin{equation}\\label{eq:there}\n    \\KL (\\mu, \\nu) \\le \\sum_{x\\in X} (\\mu_x-\\nu_x) + \\varepsilon \\sum_{x\\in X} \\lvert \\mu_x - \\nu_x \\rvert \\le \\sum_{x\\in X} (\\mu_x-\\nu_x) + 2 \\varepsilon \\lVert \\mu-\\nu\\rVert_{\\TV } . \n\\end{equation}\n","start":47550,"end":47847}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":["eq:lastStep"],"declared_external":false,"source_only":false,"title":"支撑外质量控制余项","source_units":[{"raw_tex":"It remains to estimate the first part. \nSetting $X^c\\coloneqq\\X\\setminus X$, we have \n\\[ \n\\sum_{x\\in X} (\\mu_x-\\nu_x) = \\sum_{x\\in\\X} (\\mu_x-\\nu_x) -  \\sum_{x\\in X^c} (\\mu_x-\\nu_x) = -  \\sum_{x\\in X^c} (\\mu_x-\\nu_x) = \\sum_{x\\in X^c} \\lvert  \\mu_x-\\nu_x \\rvert  \n\\]\nsince $\\mu_x=0$ for $x\\in X^c$. \nNow we can estimate\n\\begin{align}\\label{eq:lastStep}\n    2\\sum_{x\\in X} (\\mu_x-\\nu_x) = \\sum_{x\\in X} (\\mu_x-\\nu_x) + \\sum_{x\\in X^c} \\lvert  \\mu_x-\\nu_x \\rvert \\le \\lVert \\mu-\\nu\\rVert_1 = 2\\lVert \\mu-\\nu\\rVert_{\\TV }.\n\\end{align}\n","start":47847,"end":48378}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"合并局部估计","source_units":[{"raw_tex":"Combining~\\eqref{eq:there} and~\\eqref{eq:lastStep} yields\n\\[ \\KL (\\mu, \\nu) \\le (1+2\\varepsilon)\\lVert \\mu - \\nu \\rVert_{\\TV } = c\\cdot\\lVert \\mu - \\nu \\rVert_{\\TV }. \\]\n","start":48378,"end":48548}]},{"id":"S054","semantic_type":"corollary","kind":"corollary","labels":["cor:TV-KL","eq:KLvsL1-2"],"declared_external":false,"source_only":false,"title":"显式局部 KL-TV 系数","source_units":[{"raw_tex":"\\begin{corollary}[Local KL-TV estimate]\\label{cor:TV-KL}\nConsider a finite set $\\X$ and a probability distribution $\\mu\\in \\Delta_\\X$. \nThen for all $\\nu\\in\\Delta_\\X$ satisfying \n\\begin{equation}\n    \\lVert \\mu - \\nu \\rVert_\\infty < \\min\\{ \\mu_x : \\mu_x>0 \\} \n\\end{equation}\nit holds that \n\\begin{equation}\\label{eq:KLvsL1-2}\n    \\KL (\\mu, \\nu) \\le \\frac{\\min\\{ \\mu_x : \\mu_x>0 \\} +  \\lVert \\mu-\\nu\\rVert_\\infty}{\\min\\{ \\mu_x : \\mu_x>0 \\} - \\lVert \\mu-\\nu\\rVert_\\infty} \\cdot \\lVert \\mu - \\nu \\rVert_{\\TV }. \n\\end{equation}\n\\end{corollary}","start":48562,"end":49101}]},{"id":"S055","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"局部推论的作者证明","source_units":[{"raw_tex":"\\begin{proof}\n    This is a direct consequence of \\Cref{lem:KLEstimateL1}. \n    Indeed, for $\\varepsilon>0$ small enough we have $\\lVert \\mu - \\nu \\rVert_\\infty \\le \\frac{2-\\varepsilon}{2+\\varepsilon} \\cdot \\min\\{ \\mu_x : \\mu_x>0 \\}$ and thus by \\Cref{lem:KLEstimateL1} with $c=1+\\varepsilon$ we have $\\KL (\\mu, \\nu) \\le (1+\\varepsilon) \\lVert \\mu - \\nu \\rVert_{\\TV }$. \n    Note that $\\varepsilon>0$ was arbitrary. \n\\end{proof}","start":49102,"end":49530}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"推论中的参数替换","source_units":[{"raw_tex":"    This is a direct consequence of \\Cref{lem:KLEstimateL1}. \n    Indeed, for $\\varepsilon>0$ small enough we have $\\lVert \\mu - \\nu \\rVert_\\infty \\le \\frac{2-\\varepsilon}{2+\\varepsilon} \\cdot \\min\\{ \\mu_x : \\mu_x>0 \\}$ and thus by \\Cref{lem:KLEstimateL1} with $c=1+\\varepsilon$ we have $\\KL (\\mu, \\nu) \\le (1+\\varepsilon) \\lVert \\mu - \\nu \\rVert_{\\TV }$. \n    Note that $\\varepsilon>0$ was arbitrary. \n","start":49116,"end":49519}]},{"id":"S057","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\Cref{thm:convergenceFRGF}]\n    Setting $\\delta\\coloneqq \\min\\{ \\mu_x^\\star : \\mu_x^\\star>0 \\}$ and using \\Cref{lem:TV-LP-estimate-withoutUniqueness} with $\\lVert\\cdot\\rVert_{\\TV}$ and \\Cref{cor:TV-KL} we have \n    \\begin{equation*}\n        c^\\top \\mu^\\star - c^\\top \\mu_t \\ge \\Delta \\lVert \\mu^\\star - \\mu_t \\rVert_{\\TV } \\ge \\Delta\\cdot\\frac{\\delta -  \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty}{\\delta + \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty} \\cdot \\KL (\\mu^\\star, \\mu_t),\n    \\end{equation*}\n    if $\\lVert \\mu^\\star - \\mu_t \\rVert_\\infty < \\delta$.\n    By \\Cref{prop:sublinearRate} we have \n    \\begin{align*}\n        \\lVert \\mu^\\star - \\mu_t \\rVert_\\infty \\le 2 \\lVert \\mu^\\star - \\mu_t \\rVert_{\\TV } \\le \\frac{2c^\\top(\\mu^\\star-\\mu_t)}\\Delta \\le \\frac{2\\KL (\\mu^\\star,\\mu_0)}{\\Delta \\cdot t}.\n    \\end{align*}\n    Hence, for $t>t_0$ we have $\\lVert \\mu^\\star - \\mu_t \\rVert_\\infty < \\delta$. \n    In this case, we can estimate \n    \\begin{align*}\n        \\frac{\\delta -  \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty}{\\delta + \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty} & \\le \\frac{\\delta -  2 \\KL (\\mu^\\star, \\mu_0)\\Delta^{-1} t^{-1}} {\\delta + 2 \\KL (\\mu^\\star, \\mu_0)\\Delta^{-1} t^{-1}} \n        = \\frac{t - t_0}{t + t_0} \\eqqcolon \\kappa_t. \n    \\end{align*}\n    Thus for $t>t_0$ we have $c^\\top \\mu^\\star - c^\\top \\mu_t \\ge \\Delta \\kappa_t \\KL (\\mu^\\star, \\mu_t)$, and \\Cref{prop:ratesTrajectories} together with \n    \\begin{equation*}\n        \\int_{t_0}^t \\frac{s-t_0}{s+t_0} \\D s = s - 2 t_0 \\log(s+t_0)\\Big|^{s=t}_{s=t_0} = (t-t_0) - 2 t_0 \\log\\left( \\frac{t+t_0}{2t_0} \\right)\n    \\end{equation*}\n    yield the result. \n\\end{proof}","start":49612,"end":51257}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"局部目标差支配 KL","source_units":[{"raw_tex":"    Setting $\\delta\\coloneqq \\min\\{ \\mu_x^\\star : \\mu_x^\\star>0 \\}$ and using \\Cref{lem:TV-LP-estimate-withoutUniqueness} with $\\lVert\\cdot\\rVert_{\\TV}$ and \\Cref{cor:TV-KL} we have \n    \\begin{equation*}\n        c^\\top \\mu^\\star - c^\\top \\mu_t \\ge \\Delta \\lVert \\mu^\\star - \\mu_t \\rVert_{\\TV } \\ge \\Delta\\cdot\\frac{\\delta -  \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty}{\\delta + \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty} \\cdot \\KL (\\mu^\\star, \\mu_t),\n    \\end{equation*}\n    if $\\lVert \\mu^\\star - \\mu_t \\rVert_\\infty < \\delta$.\n","start":49663,"end":50180}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"次线性界给出进入时间","source_units":[{"raw_tex":"    By \\Cref{prop:sublinearRate} we have \n    \\begin{align*}\n        \\lVert \\mu^\\star - \\mu_t \\rVert_\\infty \\le 2 \\lVert \\mu^\\star - \\mu_t \\rVert_{\\TV } \\le \\frac{2c^\\top(\\mu^\\star-\\mu_t)}\\Delta \\le \\frac{2\\KL (\\mu^\\star,\\mu_0)}{\\Delta \\cdot t}.\n    \\end{align*}\n    Hence, for $t>t_0$ we have $\\lVert \\mu^\\star - \\mu_t \\rVert_\\infty < \\delta$. \n","start":50180,"end":50526}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"时间支配系数的比较","source_units":[{"raw_tex":"    In this case, we can estimate \n    \\begin{align*}\n        \\frac{\\delta -  \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty}{\\delta + \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty} & \\le \\frac{\\delta -  2 \\KL (\\mu^\\star, \\mu_0)\\Delta^{-1} t^{-1}} {\\delta + 2 \\KL (\\mu^\\star, \\mu_0)\\Delta^{-1} t^{-1}} \n        = \\frac{t - t_0}{t + t_0} \\eqqcolon \\kappa_t. \n    \\end{align*}\n","start":50526,"end":50880}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"积分时间系数","source_units":[{"raw_tex":"    \\begin{equation*}\n        \\int_{t_0}^t \\frac{s-t_0}{s+t_0} \\D s = s - 2 t_0 \\log(s+t_0)\\Big|^{s=t}_{s=t_0} = (t-t_0) - 2 t_0 \\log\\left( \\frac{t+t_0}{2t_0} \\right)\n    \\end{equation*}\n","start":51036,"end":51223}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"应用轨迹引理完成主证明","source_units":[{"raw_tex":"    Thus for $t>t_0$ we have $c^\\top \\mu^\\star - c^\\top \\mu_t \\ge \\Delta \\kappa_t \\KL (\\mu^\\star, \\mu_t)$, and \\Cref{prop:ratesTrajectories} together with \n    \\begin{equation*}\n        \\int_{t_0}^t \\frac{s-t_0}{s+t_0} \\D s = s - 2 t_0 \\log(s+t_0)\\Big|^{s=t}_{s=t_0} = (t-t_0) - 2 t_0 \\log\\left( \\frac{t+t_0}{2t_0} \\right)\n    \\end{equation*}\n    yield the result. \n","start":50880,"end":51246}]},{"id":"S063","semantic_type":"lemma","kind":"lemma","labels":["app:lem:cone"],"declared_external":false,"source_only":false,"title":"最优面出边的锥包含","source_units":[{"raw_tex":"\\begin{lemma}\\label{app:lem:cone}\n    Consider a polytope $P\\subseteq\\R^\\X$, a face $F\\subseteq P$ and consider the cone \n    \\begin{align*}\n        C\\coloneqq \\operatorname{cone}\\Big\\{ \\nu-\\mu : \\mu\\in\\operatorname{vert}(F), \\nu\\in N(\\mu)\\setminus F \\Big\\}, \n    \\end{align*}\n    which is generated by the edges pointing out of $F$. \n    Then we have $P\\subseteq F+C$. \n\\end{lemma}","start":106453,"end":106835}]},{"id":"S064","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"锥包含的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nThis is a generalization of~\\cite[Lemma 3.6]{ziegler2012lectures}, which covers the case that $F$ consists of a single vertex. \nWe will show that \n\\begin{align*}\n    F+C \\supseteq\\tilde P \\coloneqq \\bigcap \\Big\\{ H\\subseteq \\R^\\X : H \\text{ is a halfspace }, P\\subseteq H, H\\cap F\\ne\\emptyset \\Big\\} \\supseteq P, \n\\end{align*}\nfor which we pick an element $u\\in \\tilde P$. \nConsider a hyperplane $H = \\{\\mu:a^\\top \\mu = \\alpha\\}$ separating $F$ and $\\operatorname{vert}(P)\\setminus F$ and consider the \\emph{face figure} $P/F\\coloneqq P\\cap H$, which is a polytope. \nNow, we consider a translation $\\tilde H = \\{ \\mu : a^\\top\\mu= \\beta \\}$ of $H$,  such that $u\\in \\tilde H$. \nNow we have \n\\begin{align*}\n    \\tilde P = \\operatorname{conv}\\left\\{ \\mu + \\frac{a^\\top \\mu - \\beta}{a^\\top \\mu - a^\\top \\nu} \\cdot (\\nu-\\mu) : \\mu\\in\\operatorname{vert}(F), \\nu\\in N(\\mu)\\setminus F \\right\\},\n\\end{align*}\nsee~\\cite[Proposition 2.30]{zieglerdiscrete}. \nHence, we can choose convex weights $\\lambda_i$ such that \n\\begin{align*}\n    u = \\sum_{i} \\lambda_i (\\mu_i + \\alpha_i (\\nu_i-\\mu_i)) =  \\sum_{i} \\lambda_i \\mu_i + \\sum_{i} \\alpha_i \\lambda_i (\\nu_i-\\mu_i) \\in F + C,\n\\end{align*}\nwhere $\\mu_i\\in\\operatorname{vert}(F), \\nu_i \\in N(\\mu_i)\\setminus F$.\n\\end{proof}","start":106836,"end":108109}]},{"id":"S065","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"单顶点情形的文献比较","source_units":[{"raw_tex":"This is a generalization of~\\cite[Lemma 3.6]{ziegler2012lectures}, which covers the case that $F$ consists of a single vertex. \n","start":106850,"end":106978}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"半空间交的包含归约","source_units":[{"raw_tex":"We will show that \n\\begin{align*}\n    F+C \\supseteq\\tilde P \\coloneqq \\bigcap \\Big\\{ H\\subseteq \\R^\\X : H \\text{ is a halfspace }, P\\subseteq H, H\\cap F\\ne\\emptyset \\Big\\} \\supseteq P, \n\\end{align*}\nfor which we pick an element $u\\in \\tilde P$. \n","start":106978,"end":107224}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"分离切片与面图表示","source_units":[{"raw_tex":"Consider a hyperplane $H = \\{\\mu:a^\\top \\mu = \\alpha\\}$ separating $F$ and $\\operatorname{vert}(P)\\setminus F$ and consider the \\emph{face figure} $P/F\\coloneqq P\\cap H$, which is a polytope. \nNow, we consider a translation $\\tilde H = \\{ \\mu : a^\\top\\mu= \\beta \\}$ of $H$,  such that $u\\in \\tilde H$. \nNow we have \n\\begin{align*}\n    \\tilde P = \\operatorname{conv}\\left\\{ \\mu + \\frac{a^\\top \\mu - \\beta}{a^\\top \\mu - a^\\top \\nu} \\cdot (\\nu-\\mu) : \\mu\\in\\operatorname{vert}(F), \\nu\\in N(\\mu)\\setminus F \\right\\},\n\\end{align*}\nsee~\\cite[Proposition 2.30]{zieglerdiscrete}. \n","start":107224,"end":107797}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"凸组合拆成面与锥","source_units":[{"raw_tex":"Hence, we can choose convex weights $\\lambda_i$ such that \n\\begin{align*}\n    u = \\sum_{i} \\lambda_i (\\mu_i + \\alpha_i (\\nu_i-\\mu_i)) =  \\sum_{i} \\lambda_i \\mu_i + \\sum_{i} \\alpha_i \\lambda_i (\\nu_i-\\mu_i) \\in F + C,\n\\end{align*}\nwhere $\\mu_i\\in\\operatorname{vert}(F), \\nu_i \\in N(\\mu_i)\\setminus F$.\n","start":107797,"end":108098}]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites：Standard finite-dimensional convex analysis，包括 first-order optimality 与 strict convexity。"],"source_only":false,"title":"凸优化的一阶刻画","source_units":[]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites：Standard finite-dimensional convex analysis 中明确列出的 KL nonnegativity。"],"source_only":false,"title":"KL 非负性","source_units":[]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites：Standard finite-dimensional convex analysis 中的 entropy boundary behavior。"],"source_only":false,"title":"熵的边界行为","source_units":[]},{"id":"S072","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites：Gronwall's inequality，原文第528行调用。"],"source_only":false,"title":"Gronwall 不等式","source_units":[]},{"id":"S073","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites：Picard-Lindelöf local existence and uniqueness theorem；原文第340行引用 teschl2024ordinary。"],"source_only":false,"title":"Picard-Lindelöf 定理","source_units":[]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites：zieglerdiscrete, Proposition 2.30；原文第1371行引用。"],"source_only":false,"title":"面图的外部命题","source_units":[]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；属于原始先决条件所列标准有限维凸分析背景，原文第580行实际写出此不等式。"],"source_only":false,"title":"对数切线不等式","source_units":[]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites：Standard finite-dimensional convex analysis，包括 elementary polytope separation。"],"source_only":false,"title":"多胞体面的严格分离","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S018","edges":[{"source_id":"S007","target_id":"S012","kind":"supports"},{"source_id":"S008","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S008","target_id":"S010","kind":"supports"},{"source_id":"S006","target_id":"S010","kind":"supports"},{"source_id":"S005","target_id":"S011","kind":"supports"},{"source_id":"S007","target_id":"S011","kind":"supports"},{"source_id":"S015","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S019","kind":"supports"},{"source_id":"S015","target_id":"S021","kind":"supports"},{"source_id":"S008","target_id":"S021","kind":"supports"},{"source_id":"S010","target_id":"S021","kind":"supports"},{"source_id":"S069","target_id":"S021","kind":"supports"},{"source_id":"S071","target_id":"S021","kind":"supports"},{"source_id":"S014","target_id":"S022","kind":"supports"},{"source_id":"S015","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S015","target_id":"S023","kind":"supports"},{"source_id":"S010","target_id":"S023","kind":"supports"},{"source_id":"S019","target_id":"S024","kind":"supports"},{"source_id":"S070","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S073","target_id":"S027","kind":"supports"},{"source_id":"S019","target_id":"S028","kind":"supports"},{"source_id":"S071","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S004","target_id":"S018","kind":"supports"},{"source_id":"S002","target_id":"S018","kind":"supports"},{"source_id":"S010","target_id":"S018","kind":"supports"},{"source_id":"S015","target_id":"S033","kind":"supports"},{"source_id":"S013","target_id":"S033","kind":"supports"},{"source_id":"S010","target_id":"S033","kind":"supports"},{"source_id":"S008","target_id":"S035","kind":"supports"},{"source_id":"S010","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S014","target_id":"S036","kind":"supports"},{"source_id":"S005","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S038","kind":"supports"},{"source_id":"S030","target_id":"S038","kind":"supports"},{"source_id":"S072","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S031","kind":"supports"},{"source_id":"S023","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S032","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S029","kind":"supports"},{"source_id":"S003","target_id":"S040","kind":"supports"},{"source_id":"S004","target_id":"S040","kind":"supports"},{"source_id":"S003","target_id":"S042","kind":"supports"},{"source_id":"S004","target_id":"S042","kind":"supports"},{"source_id":"S063","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S043","target_id":"S041","kind":"supports"},{"source_id":"S044","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S006","target_id":"S045","kind":"supports"},{"source_id":"S002","target_id":"S045","kind":"supports"},{"source_id":"S010","target_id":"S045","kind":"supports"},{"source_id":"S006","target_id":"S047","kind":"supports"},{"source_id":"S010","target_id":"S047","kind":"supports"},{"source_id":"S075","target_id":"S048","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S047","target_id":"S051","kind":"supports"},{"source_id":"S048","target_id":"S051","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S002","target_id":"S051","kind":"supports"},{"source_id":"S047","target_id":"S052","kind":"supports"},{"source_id":"S006","target_id":"S052","kind":"supports"},{"source_id":"S002","target_id":"S052","kind":"supports"},{"source_id":"S051","target_id":"S053","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S049","target_id":"S046","kind":"supports"},{"source_id":"S050","target_id":"S046","kind":"supports"},{"source_id":"S051","target_id":"S046","kind":"supports"},{"source_id":"S052","target_id":"S046","kind":"supports"},{"source_id":"S053","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S006","target_id":"S054","kind":"supports"},{"source_id":"S002","target_id":"S054","kind":"supports"},{"source_id":"S010","target_id":"S054","kind":"supports"},{"source_id":"S045","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S054","kind":"supports"},{"source_id":"S040","target_id":"S058","kind":"supports"},{"source_id":"S054","target_id":"S058","kind":"supports"},{"source_id":"S017","target_id":"S058","kind":"supports"},{"source_id":"S002","target_id":"S059","kind":"supports"},{"source_id":"S040","target_id":"S059","kind":"supports"},{"source_id":"S023","target_id":"S059","kind":"supports"},{"source_id":"S017","target_id":"S059","kind":"supports"},{"source_id":"S025","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S058","target_id":"S062","kind":"supports"},{"source_id":"S060","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S029","target_id":"S062","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S059","target_id":"S057","kind":"supports"},{"source_id":"S060","target_id":"S057","kind":"supports"},{"source_id":"S061","target_id":"S057","kind":"supports"},{"source_id":"S062","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S018","kind":"supports"},{"source_id":"S003","target_id":"S063","kind":"supports"},{"source_id":"S004","target_id":"S063","kind":"supports"},{"source_id":"S076","target_id":"S067","kind":"supports"},{"source_id":"S074","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S066","target_id":"S064","kind":"supports"},{"source_id":"S067","target_id":"S064","kind":"supports"},{"source_id":"S068","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S063","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S018","edges":[{"source_id":"S007","target_id":"S012","kind":"supports"},{"source_id":"S008","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S008","target_id":"S010","kind":"supports"},{"source_id":"S006","target_id":"S010","kind":"supports"},{"source_id":"S005","target_id":"S011","kind":"supports"},{"source_id":"S007","target_id":"S011","kind":"supports"},{"source_id":"S015","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S019","kind":"supports"},{"source_id":"S008","target_id":"S019","kind":"supports"},{"source_id":"S069","target_id":"S019","kind":"supports"},{"source_id":"S071","target_id":"S019","kind":"supports"},{"source_id":"S014","target_id":"S019","kind":"supports"},{"source_id":"S015","target_id":"S023","kind":"supports"},{"source_id":"S010","target_id":"S023","kind":"supports"},{"source_id":"S019","target_id":"S023","kind":"supports"},{"source_id":"S070","target_id":"S023","kind":"supports"},{"source_id":"S073","target_id":"S025","kind":"supports"},{"source_id":"S019","target_id":"S025","kind":"supports"},{"source_id":"S071","target_id":"S025","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S004","target_id":"S018","kind":"supports"},{"source_id":"S002","target_id":"S018","kind":"supports"},{"source_id":"S010","target_id":"S018","kind":"supports"},{"source_id":"S015","target_id":"S033","kind":"supports"},{"source_id":"S013","target_id":"S033","kind":"supports"},{"source_id":"S010","target_id":"S033","kind":"supports"},{"source_id":"S008","target_id":"S033","kind":"supports"},{"source_id":"S014","target_id":"S033","kind":"supports"},{"source_id":"S005","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S038","kind":"supports"},{"source_id":"S030","target_id":"S038","kind":"supports"},{"source_id":"S072","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S031","kind":"supports"},{"source_id":"S023","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S032","kind":"supports"},{"source_id":"S038","target_id":"S029","kind":"supports"},{"source_id":"S039","target_id":"S029","kind":"supports"},{"source_id":"S003","target_id":"S040","kind":"supports"},{"source_id":"S004","target_id":"S040","kind":"supports"},{"source_id":"S063","target_id":"S040","kind":"supports"},{"source_id":"S006","target_id":"S045","kind":"supports"},{"source_id":"S002","target_id":"S045","kind":"supports"},{"source_id":"S010","target_id":"S045","kind":"supports"},{"source_id":"S075","target_id":"S045","kind":"supports"},{"source_id":"S006","target_id":"S054","kind":"supports"},{"source_id":"S002","target_id":"S054","kind":"supports"},{"source_id":"S010","target_id":"S054","kind":"supports"},{"source_id":"S045","target_id":"S054","kind":"supports"},{"source_id":"S040","target_id":"S058","kind":"supports"},{"source_id":"S054","target_id":"S058","kind":"supports"},{"source_id":"S017","target_id":"S058","kind":"supports"},{"source_id":"S002","target_id":"S059","kind":"supports"},{"source_id":"S040","target_id":"S059","kind":"supports"},{"source_id":"S023","target_id":"S059","kind":"supports"},{"source_id":"S017","target_id":"S059","kind":"supports"},{"source_id":"S025","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S058","target_id":"S062","kind":"supports"},{"source_id":"S060","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S029","target_id":"S062","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S059","target_id":"S057","kind":"supports"},{"source_id":"S060","target_id":"S057","kind":"supports"},{"source_id":"S061","target_id":"S057","kind":"supports"},{"source_id":"S062","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S018","kind":"supports"},{"source_id":"S003","target_id":"S063","kind":"supports"},{"source_id":"S004","target_id":"S063","kind":"supports"},{"source_id":"S076","target_id":"S063","kind":"supports"},{"source_id":"S074","target_id":"S063","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"有限向量空间记号","source_units":[{"raw_tex":"For a finite set $\\X$, we denote the \\emph{free vector space} over $\\X$ by $\\R^\\X = \\{ \\mu\\colon \\X \\to \\R \\}$. \nIts elements can be identified with vectors $(\\mu_x)_{x\\in\\X}$. \nSimilarly, we denote the vectors with non-negative entries and positive entries by $\\R_{\\ge0}^\\X$ and $\\R_{>0}^\\X$, respectively. \nFor two elements $\\mu, \\nu\\in\\R^\\X$ we denote the \\emph{Hadamard product}, i.e., the entrywise product, between $\\mu$ and $\\nu$ by $\\mu\\odot\\nu\\in\\R^\\X$, so that $\\mu\\odot\\nu(x) \\coloneqq \\mu(x)\\nu(x)$.\n","start":14041,"end":14553}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"全变差范数","source_units":[{"raw_tex":"The \\emph{total variation} norm $\\lVert \\cdot \\rVert_{\\TV}\\colon\\R^\\X\\to\\R$ is given by $\\lVert \\mu \\rVert_{\\TV} \\coloneqq \\frac12\\sum_{x}\\lvert \\mu_x \\rvert$. \nFinally, with $\\mathds{1}_\\X\\in\\R^\\X$ we denote the all-one vector. \n\n","start":14553,"end":14784}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"多面体与顶点","source_units":[{"raw_tex":"A \\emph{polyhedron} is a set $P = \\{ \\mu\\in\\R^\\X : \\ell_i(\\mu) \\ge 0 \\text{ for } i=1, \\dots, k \\}\\subseteq \\R^\\X$, where $\\ell_i\\colon\\R^\\X\\to\\R$ are affine linear functions for $i=1, \\dots, k$. \nA bounded (and thus compact) polyhedron is called a \\emph{polytope}. \nA polytope can be shown to be the convex hull of finitely many extreme points, which are called \\emph{vertices} and which we denote by $\\operatorname{Vert}(P)$. \n","start":14784,"end":15213}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"相邻顶点","source_units":[{"raw_tex":"Two vertices $\\mu_1, \\mu_2\\in\\operatorname{Vert}(P)$ are called \\emph{neighbors} if the subspace $\\{ c\\in\\R^\\X : c^\\top \\mu_1 = c^\\top \\mu_2 = \\max_{\\mu\\in P} c^\\top \\mu \\}$ has dimension $\\lvert \\X \\rvert-1$. \nWe denote the set of all neighbors of a vertex $\\mu$ by $N(\\mu) \\subseteq\\operatorname{Vert}(P)$.  \n","start":15213,"end":15524}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"仿射与切空间","source_units":[{"raw_tex":"The \\emph{affine space} $\\operatorname{aff\\,span}(P)$ of a polytope $P\\subseteq\\R^\\X$ is the smallest affine subspace of $\\R^\\X$ containing $P$. \nThe relative interior $\\operatorname{int}(P)$ and boundary $\\partial P$ of $P$ are the interior and boundary of $P$ in its affine hull. \nFinally, the \\emph{tangent space} $TP$ of $P$ is given by the linear part of $\\operatorname{aff\\,span}(P)$. \n\n","start":15524,"end":15917}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"概率单纯形与支撑","source_units":[{"raw_tex":"We call $\\Delta_\\X \\coloneqq \\left\\{ \\mu\\in\\R^\\X_{\\ge0} : \\sum_{x} \\mu_x=1 \\right\\}$ the \\emph{probability simplex}. \nWe say that $\\mu\\in\\Delta_\\X$ is absolutely continuous with respect to $\\nu\\in\\Delta_\\X$ if $\\nu(x) = 0$ implies $\\mu(x)=0$ and write $\\mu \\ll \\nu$. \n","start":15917,"end":16185}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"Fisher-Rao 度量","source_units":[{"raw_tex":"The \\emph{Fisher-Rao metric} is a Riemannian metric on the positive orthant given by \n\\begin{equation}\n    g_{\\mu}^{\\FR}(v,w) \\coloneqq \\sum_{x\\in \\X} \\frac{v_x w_x}{\\mu_x} \\quad \\text{for all } v, w\\in\\R^{\\X}, \\mu\\in\\R^{\\X}_{>0},\n\\end{equation}\nwhere we denote the induced norm by $\\lVert v \\rVert_{g_\\mu^{\\FR}} \\coloneqq {g_{\\mu}^{\\FR}(v,v)}^{\\frac12}$. \n","start":18406,"end":18763}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"负熵的 Hessian 表示","source_units":[{"raw_tex":"The Fisher-Rao metric is closely connected to the negative {Shannon entropy} \n\\begin{equation}\n    \\phi(\\mu) = -H(\\mu) \\coloneqq \\sum_{x\\in\\X} \\mu_x \\log \\mu_x  \\quad \\text{for all } \\mu\\in\\R^{\\X}_{>0}\n\\end{equation}\nas it is induced by the Hessian of the (negative) entropy, meaning that we have $g_{\\mu}^{\\FR}(v,w) = v^\\top \\nabla^2 \\phi(\\mu) w$ for all $v, w\\in\\R^{\\X}, \\mu\\in\\R^{\\X}_{>0}$. \n","start":19970,"end":20365}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"广义 KL 散度","source_units":[{"raw_tex":"An important concept in the analysis of Hessian gradient flows is the \\emph{Bregman divergence} induced by $\\phi$, which in the case of the negative entropy is given by the \\emph{KL-divergence} \n\\begin{equation}\n    \\KL (\\mu, \\nu) \\coloneqq  \\phi(\\mu) - \\phi(\\nu) - \\nabla \\phi(\\nu)(\\mu-\\nu) = \\sum_{x\\in \\X} \\mu_x \\log \\frac{\\mu_x}{\\nu_x} - \\sum_{x\\in \\X} \\mu_x + \\sum_{x\\in \\X} \\nu_x \n\\end{equation}\nfor $\\mu, \\nu\\in\\R_{\\ge0}^\\X$ with $\\mu\\ll\\nu$, where we use the common convention $0\\log\\frac00\\coloneqq0$. \n\n","start":20600,"end":21113}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"约束 Fisher-Rao 梯度","source_units":[{"raw_tex":"Consider now a continuously differentiable function $f\\colon\\R_{\\ge0}^\\X\\to\\R$ that we assume to be differentiable on $\\R_{>0}^\\X$ that we want to optimize over a polytope $P= \\R_{\\ge0}^{\\X}\\cap\\mathcal L$, where $\\mathcal L$ is a linear space. \nWe denote the gradient of $f\\colon \\R_{>0}^\\X\\to\\R$ at $\\mu\\in\\R_{>0}^\\X$ with respect to the Fisher-Rao metric by $\\nabla^{\\FR}f(\\mu)$ and call it the \\emph{Fisher-Rao gradient}. \nFurther, we denote the Fisher-Rao gradient of $f\\colon\\operatorname{int}(P)\\to\\R$ by $\\nabla^{\\FR}_P f(\\mu)\\in TP$, which is uniquely determined by \n\\begin{equation}\n    g_\\mu^{\\FR}(\\nabla^{\\FR}_P f(\\mu),v) = df(\\mu)v \\quad \\text{for all } v\\in TP.\n\\end{equation}\nNote that $\\nabla^{\\FR}_P f(\\mu)$ is the projection of $\\nabla^{\\FR}f(\\mu)$ with respect to the Fisher-Rao metric onto $TP$.\n","start":21113,"end":21929}]},{"id":"S012","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:characterizationFRGradient"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"约束梯度的 Hessian 刻画","source_units":[{"raw_tex":"By examining the definition of the Fisher-Rao metric we see that this is equivalent to \n\\begin{equation}\n\\label{eq:characterizationFRGradient}\n    \\langle \\nabla^2\\phi(\\mu)\\nabla^{\\FR}_P  f(\\mu), v \\rangle = \\langle \\nabla f(\\mu), v \\rangle \\quad \\text{for all } v\\in TP. \n\\end{equation}\n","start":21929,"end":22217}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["eq:FRGF"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"梯度上升流","source_units":[{"raw_tex":"    \nWe say that $(\\mu_t)_{t\\in [0, T)} \\subseteq \\operatorname{int}(P)$ solves the \\emph{Fisher-Rao gradient flow} if it solves the gradient flow with respect to the Fisher-Rao metric, i.e., if \n\\begin{equation}\\label{eq:FRGF}\n    \\partial_t \\mu_t = \\nabla^{\\FR}_P  f(\\mu_t) \\quad \\text{for all } t\\in[0, T). \n\\end{equation}\n","start":22217,"end":22543}]},{"id":"S014","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:explicitFRGF"],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"梯度流弱形式","source_units":[{"raw_tex":"By using the characterization~\\eqref{eq:characterizationFRGradient} of $\\nabla^{\\FR}_P f(\\mu_t)$, we see that $(\\mu_t)_{t\\in [0, T)} \\subseteq \\operatorname{int}(P)$ solves the Fisher-Rao gradient flow~\\eqref{eq:FRGF}\nif and only if we have \n\\begin{equation}\\label{eq:explicitFRGF}\n    \\langle \\nabla^2 \\phi(\\mu_t) \\partial_t \\mu_t, v \\rangle = \\langle \\nabla f(\\mu_t), v \\rangle \n    \\quad \\text{for all } v\\in TP, t\\in[0, T). \n\\end{equation}\n","start":22543,"end":22987}]},{"id":"S015","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:LP"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"一般线性规划设定","source_units":[{"raw_tex":"In the remainder, we study linear programs and work in the following setting. \n\n\\begin{setting}\\label{setting:generalLP}\nWe consider a finite set $\\X$ and a linear program \n    \\begin{equation}\\label{eq:LP}\n        \\max c^\\top \\mu \\quad \\text{subject to } \\mu\\in P, \n    \\end{equation}\nwith cost $c\\in\\R^{\\X}$ and feasible region $P = \\R^\\X_{\\ge0} \\cap  \\mathcal L$ with $P \\cap \\R^\\X_{>0}\\ne\\emptyset$, where $\\mathcal L\\subseteq\\R^{\\X}$ is an affine space. \nBy $(\\mu_t)_{t\\in[0, T)}\\subseteq\\operatorname{int}(P)$ we denote a solution of the Fisher-Rao gradient flow~\\eqref{eq:FRGF} with initial condition $\\mu_0\\in P \\cap \\R_{>0}^\\X$ and potential $f(\\mu) = c^\\top \\mu$, where $T\\in\\R_{\\ge0}\\cup\\{+\\infty\\}$. \n\\end{setting}\n","start":22988,"end":23715}]},{"id":"S017","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:LP-inside-simplex"],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"单纯形内线性规划","source_units":[{"raw_tex":"\\begin{setting}\\label{setting:LP-inside-Simplex}\nWe consider a finite set $\\X$ and a linear program \n    \\begin{equation}\\label{eq:LP-inside-simplex}\n        \\max c^\\top \\mu \\quad \\text{subject to } \\mu\\in P, \n    \\end{equation}\nwith cost $c\\in\\R^{\\X}$ and feasible region $P = \\Delta_\\X \\cap  \\mathcal L$ with $P \\cap \\R^\\X_{>0}\\ne\\emptyset$, where $\\mathcal L\\subseteq\\R^{\\X}$ is an affine space. \nBy $(\\mu_t)_{t\\ge0}\\subseteq\\operatorname{int}(P)$ we denote the solution of the Fisher-Rao gradient flow~\\eqref{eq:FRGF} with initial condition $\\mu_0\\in P \\cap \\R_{>0}^\\X$ and the potential $f(\\mu) = c^\\top \\mu$. \n\\end{setting}\n","start":28219,"end":28849}]},{"id":"S018","semantic_type":"theorem","kind":"theorem","labels":["eq:Delta","eq:definitiont_0","eq:linearConvergenceKL","eq:linearConvergenceValue","thm:convergenceFRGF"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"唯一最优解的显式收敛率","source_units":[{"raw_tex":"\\begin{restatable}[Linear convergence of Fisher-Rao GFs of LPs]{theorem}{convergenceFRGF}\n\\label{thm:convergenceFRGF}\nConsider Setting~\\ref{setting:LP-inside-Simplex} and assume that the linear program~\\eqref{eq:LP-inside-simplex} admits a unique solution $\\mu^\\star\\in P$. \nLet \n\\begin{align}\\label{eq:Delta}\n    \\Delta \\coloneqq \\min \\left\\{ \\frac{c^\\top \\mu^\\star - \n    c^\\top \\mu}{\\lVert \\mu^\\star - \\mu \\rVert_{\\TV }} : \\mu\\in N(\\mu^\\star)  \\right\\},\n\\end{align}\nwhere $N(\\mu^\\star)$ denotes the set of neighboring vertices of $\\mu^\\star$ and set \n\\begin{equation}\\label{eq:definitiont_0}\n    t_0 \\coloneqq \\frac{2 \\KL (\\mu^\\star, \\mu_0)}{\\Delta \\cdot \\min\\{ \\mu^\\star_x : \\mu^\\star_x>0 \\}} . \n\\end{equation}\nThen for any $t\\ge t_0$ we have \n\\begin{align}\\label{eq:linearConvergenceKL}\n    \\KL (\\mu^\\star, \\mu_t) \\le \\KL (\\mu^\\star, \\mu_0) \\exp\\left( -\\Delta(t-t_0) +  2 t_0 \\Delta \n    \\log\\left(\\frac{t+t_0}{2t_0}\\right) \\right), \n\\end{align}\nas well as \n\\begin{align}\\label{eq:linearConvergenceValue}\n    c^\\top \\mu^\\star - c^\\top \\mu_t \\le \\Delta \\KL (\\mu^\\star, \\mu_0) \\exp\\left( -\\Delta(t-t_0) +  2 t_0 \\Delta \\log\\left(\\frac{t+t_0}{2t_0}\\right) \\right). \n\\end{align}\n\\end{restatable}","start":29173,"end":30369}]},{"id":"S019","semantic_type":"proposition","kind":"proposition","labels":["eq:centralPath"],"member_ids":["S019","S020","S021","S022"],"declared_external":false,"source_only":false,"title":"中心路径性质","source_units":[{"raw_tex":"\\begin{proposition}[Central path property,\\cite{alvarez2004hessian}]\n    Consider Setting~\\ref{setting:generalLP}. \n    Then $\\mu_t$ is uniquely characterized by \n    \\begin{equation}\\label{eq:centralPath}\n        \\mu_t = \\argmax\\left\\{ c^\\top \\mu - t^{-1} \\KL (\\mu, \\mu_0) : \\mu \\in P \\right\\} \\quad \\text{for all } t\\in(0,T). \n    \\end{equation}\n\\end{proposition}","start":24059,"end":24424},{"raw_tex":"\\begin{proof}\n    Let $\\hat{\\mu}_t\\in P$ denote the unique maximizer of $g(\\mu)\\coloneqq c^\\top\\mu - t^{-1} \\KL (\\mu, \\mu_0)$ over $P$ for $t>0$, then surely $\\hat{\\mu}_t\\in \\operatorname{int}(P)$. \n    Thus, $\\hat{\\mu}_t$ is uniquely determined by $\\langle \\nabla g(\\hat{\\mu}_t), v\\rangle = 0$ for all $v\\in TP$. \n    Direct computation yields $\\nabla g({\\mu}) = c - t^{-1} (\\nabla\\phi(\\mu) - \\nabla\\phi(\\mu_0))$ and hence $\\hat{\\mu}_t$ is uniquely determined by \n    \\begin{equation*}\n        t\\langle c, v\\rangle = \\langle \\nabla\\phi(\\hat{\\mu}_t) - \\nabla\\phi(\\mu_0), v\\rangle \\quad \\text{for all } v\\in TP. \n    \\end{equation*} \n    On the other hand, for the gradient flow, we can use~\\eqref{eq:explicitFRGF} and compute for $v\\in TP$ \n    \\begin{align*}\n        \\langle \\nabla\\phi(\\mu_t) - \\nabla\\phi(\\mu_0), v \\rangle & = \\int_{0}^t \\partial_s \\langle \\nabla \\phi(\\mu_s), v \\rangle \\D s \n        = \\int_{0}^t \\langle \\nabla^2 \\phi(\\mu_s)\\partial_s \\mu_s, v \\rangle \\D s \n        \\\\ & = \\int_{0}^t \\langle \\nabla f(\\mu_s) \\mu_s, v \\rangle \\D s \n        = \\int_{0}^t \\langle c, v \\rangle \\D s = t \\langle c, v \\rangle . \n    \\end{align*}\n    This shows $\\mu_t = \\hat{\\mu}_t$ as claimed. \n\\end{proof}","start":24429,"end":25633},{"raw_tex":"    Let $\\hat{\\mu}_t\\in P$ denote the unique maximizer of $g(\\mu)\\coloneqq c^\\top\\mu - t^{-1} \\KL (\\mu, \\mu_0)$ over $P$ for $t>0$, then surely $\\hat{\\mu}_t\\in \\operatorname{int}(P)$. \n    Thus, $\\hat{\\mu}_t$ is uniquely determined by $\\langle \\nabla g(\\hat{\\mu}_t), v\\rangle = 0$ for all $v\\in TP$. \n    Direct computation yields $\\nabla g({\\mu}) = c - t^{-1} (\\nabla\\phi(\\mu) - \\nabla\\phi(\\mu_0))$ and hence $\\hat{\\mu}_t$ is uniquely determined by \n    \\begin{equation*}\n        t\\langle c, v\\rangle = \\langle \\nabla\\phi(\\hat{\\mu}_t) - \\nabla\\phi(\\mu_0), v\\rangle \\quad \\text{for all } v\\in TP. \n    \\end{equation*} \n","start":24443,"end":25062},{"raw_tex":"    On the other hand, for the gradient flow, we can use~\\eqref{eq:explicitFRGF} and compute for $v\\in TP$ \n    \\begin{align*}\n        \\langle \\nabla\\phi(\\mu_t) - \\nabla\\phi(\\mu_0), v \\rangle & = \\int_{0}^t \\partial_s \\langle \\nabla \\phi(\\mu_s), v \\rangle \\D s \n        = \\int_{0}^t \\langle \\nabla^2 \\phi(\\mu_s)\\partial_s \\mu_s, v \\rangle \\D s \n        \\\\ & = \\int_{0}^t \\langle \\nabla f(\\mu_s) \\mu_s, v \\rangle \\D s \n        = \\int_{0}^t \\langle c, v \\rangle \\D s = t \\langle c, v \\rangle . \n    \\end{align*}\n    This shows $\\mu_t = \\hat{\\mu}_t$ as claimed. \n","start":25062,"end":25622}]},{"id":"S023","semantic_type":"corollary","kind":"corollary","labels":["eq:sublinearConvergence","prop:sublinearRate"],"member_ids":["S023","S024"],"declared_external":false,"source_only":false,"title":"次线性目标误差界","source_units":[{"raw_tex":"\\begin{corollary}[Sublinear convergence rate,\\cite{alvarez2004hessian}]\\label{prop:sublinearRate}\n    Consider Setting~\\ref{setting:generalLP} and assume that the linear program~\\eqref{eq:LP} admits a solution $\\mu^\\star\\in P$. \n    Then for $\\mu\\in P$ it holds that \n\\begin{equation}\\label{eq:sublinearConvergence}\n        c^\\top \\mu^\\star - c^\\top\\mu_t \\le \\frac{\\KL (\\mu^\\star, \\mu) - \\KL (\\mu_t, \\mu)}t \\le \\frac{\\KL (\\mu^\\star, \\mu_0)}t \\quad \\text{for all } t\\in[0, T). \n    \\end{equation}\n\\end{corollary}","start":25809,"end":26320},{"raw_tex":"\\begin{proof}\n    We have $c^\\top \\mu_t - t^{-1} \\KL (\\mu_t, \\mu) \\ge c^\\top \\mu^\\star - t^{-1} \\KL (\\mu^\\star, \\mu)$ by the central path property.\n    Rearranging yields the result. \n\\end{proof}","start":26321,"end":26516}]},{"id":"S025","semantic_type":"theorem","kind":"theorem","labels":[],"member_ids":["S025","S026","S027","S028"],"declared_external":false,"source_only":false,"title":"梯度流全局适定性","source_units":[{"raw_tex":"\\begin{theorem}[Well-posedness of  FR GFs,\\cite{alvarez2004hessian}]\n    Consider Setting~\\ref{setting:generalLP}. \n    Then there exists a unique global solution $(\\mu_{t})_{t\\ge0}\\subseteq\\operatorname{int}(P)$ of the Fisher-Rao gradient flow~\\eqref{eq:FRGF}. \n\\end{theorem}","start":26811,"end":27087},{"raw_tex":"\\begin{proof}\n    The local existence and uniqueness follow from the Picard-Lindelöf theorem~\\cite{teschl2024ordinary}. Hence, it suffices to show that the Fisher-Rao gradient flow does not hit the boundary $\\partial P$ in finite time. \n    By the central path property, this is equivalent to the statement that the solutions of all KL-regularized problems~\\eqref{eq:centralPath} lie in the interior $\\operatorname{int}(P)$ of the polyhedron, which can be easily checked. \n\\end{proof}","start":27088,"end":27572},{"raw_tex":"    The local existence and uniqueness follow from the Picard-Lindelöf theorem~\\cite{teschl2024ordinary}. Hence, it suffices to show that the Fisher-Rao gradient flow does not hit the boundary $\\partial P$ in finite time. \n","start":27102,"end":27325},{"raw_tex":"    By the central path property, this is equivalent to the statement that the solutions of all KL-regularized problems~\\eqref{eq:centralPath} lie in the interior $\\operatorname{int}(P)$ of the polyhedron, which can be easily checked. \n","start":27325,"end":27561}]},{"id":"S029","semantic_type":"lemma","kind":"lemma","labels":["eq:lem:KLConvergence","eq:lem:convergenceValue","eq:strongConvexity","prop:ratesTrajectories"],"member_ids":["S029","S037"],"declared_external":false,"source_only":false,"title":"沿轨迹支配推出速率","source_units":[{"raw_tex":"\\begin{lemma}\\label{prop:ratesTrajectories} \n    Consider Setting~\\ref{setting:generalLP} and assume that there is an optimizer $\\mu^\\star\\in P$ and $\\kappa_t>0$ for $t>t_0\\ge0$ such that\n\\begin{equation}\\label{eq:strongConvexity}\n    c^\\top \\mu^\\star - c^\\top \\mu_t \\ge \\kappa_t \\KL (\\mu^\\star, \\mu_t) \\quad \\text{for all } t> t_0. \n\\end{equation}\nThen we have \n\\begin{equation}\\label{eq:lem:KLConvergence}\n    \\KL (\\mu^\\star, \\mu_t) \\le \\KL (\\mu^\\star, \\mu_{0}) \\exp\\left( -\\int_{t_0}^t \\kappa_s \\D s \\right) \\quad \\text{for all } t\\ge t_0,\n\\end{equation}\nas well as \n\\begin{equation}\\label{eq:lem:convergenceValue}\n    c^\\top \\mu^\\star - c^\\top \\mu_t \\le \\kappa_t \\KL (\\mu^\\star, \\mu_{0}) \\exp\\left( -\\int_{t_0}^t \\kappa_s \\D s \\right) \\quad \\text{for all } t\\ge t_0.\n\\end{equation}\n\\end{lemma}","start":40737,"end":41534},{"raw_tex":"\\begin{proof}[Proof of \\Cref{prop:ratesTrajectories}]\nUsing~\\eqref{eq:derivativeu} and~\\eqref{eq:strongConvexity} we find that for all $t\\ge T$ it holds that \n$\\partial_t \\KL (\\mu^\\star, \\mu_t) = c^\\top \\mu_t  - c^\\top \\mu^\\star \\le - \\kappa_t \\KL (\\mu^\\star, \\mu_t)$. \nNow Gronwall's inequality yields~\\eqref{eq:lem:KLConvergence}. \nBy \\Cref{prop:sublinearRate} we have for any $h>0$ that \n    \\[ c^\\top \\mu^\\star - c^\\top \\mu_t \\le \\frac{\\KL (\\mu^\\star, \\mu_{t-h})}{h} \\le  \\KL (\\mu^\\star, \\mu_{0}) \\cdot \\frac{\\exp\\left(-\\int_{t_0}^{t-h} \\kappa_s \\D s \\right)}{h}. \\] \nTaking the limit $h\\to0$ yields~\\eqref{eq:lem:convergenceValue}. \n\\end{proof}","start":42379,"end":43028}]},{"id":"S030","semantic_type":"assumption","kind":"assumption","labels":["eq:strongConvexity","prop:ratesTrajectories"],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"沿轨迹 KL 支配条件","source_units":[{"raw_tex":"    Consider Setting~\\ref{setting:generalLP} and assume that there is an optimizer $\\mu^\\star\\in P$ and $\\kappa_t>0$ for $t>t_0\\ge0$ such that\n\\begin{equation}\\label{eq:strongConvexity}\n    c^\\top \\mu^\\star - c^\\top \\mu_t \\ge \\kappa_t \\KL (\\mu^\\star, \\mu_t) \\quad \\text{for all } t> t_0. \n\\end{equation}\n","start":40782,"end":41086}]},{"id":"S031","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:lem:KLConvergence","prop:ratesTrajectories"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"轨迹引理的 KL 分结论","source_units":[{"raw_tex":"Then we have \n\\begin{equation}\\label{eq:lem:KLConvergence}\n    \\KL (\\mu^\\star, \\mu_t) \\le \\KL (\\mu^\\star, \\mu_{0}) \\exp\\left( -\\int_{t_0}^t \\kappa_s \\D s \\right) \\quad \\text{for all } t\\ge t_0,\n\\end{equation}\n","start":41086,"end":41295}]},{"id":"S032","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:lem:convergenceValue","prop:ratesTrajectories"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"轨迹引理的目标值分结论","source_units":[{"raw_tex":"as well as \n\\begin{equation}\\label{eq:lem:convergenceValue}\n    c^\\top \\mu^\\star - c^\\top \\mu_t \\le \\kappa_t \\KL (\\mu^\\star, \\mu_{0}) \\exp\\left( -\\int_{t_0}^t \\kappa_s \\D s \\right) \\quad \\text{for all } t\\ge t_0.\n\\end{equation}\n","start":41295,"end":41523}]},{"id":"S033","semantic_type":"lemma","kind":"lemma","labels":["eq:derivativeu","lem:evolutionKL"],"member_ids":["S033","S034","S035","S036"],"declared_external":false,"source_only":false,"title":"KL 沿流演化恒等式","source_units":[{"raw_tex":"\\begin{lemma}[\\cite{alvarez2004hessian}]\\label{lem:evolutionKL}\nConsider Setting~\\ref{setting:generalLP}, whereby we allow $f\\colon\\R_{>0}^\\X\\to\\R$ to be an arbitrary differentiable function, and fix $\\mu\\in P$. \nThen for any $t\\ge0$, it holds that \n    \\begin{equation}\\label{eq:derivativeu}\n        \\partial_t \\KL (\\mu, \\mu_t) = \\langle \\nabla f(\\mu_t), \\mu_t-\\mu\\rangle. \n    \\end{equation}\n\\end{lemma}","start":41602,"end":42007},{"raw_tex":"\\begin{proof}\n    Denoting the negative Shannon entropy by $\\phi$, we compute \n    \\begin{align*}\n        \\partial_t \\KL (\\mu, \\mu_t) = - \\partial_t \\phi(\\mu_t) - \\partial_t \\langle \\nabla \\phi(\\mu_t), \\mu - \\mu_t \\rangle = \\langle \\nabla^2 \\phi(\\mu_t)\\partial_t \\mu_t, \\mu_t - \\mu \\rangle.\n    \\end{align*}\n    Now~\\eqref{eq:explicitFRGF} yields the claim. \n\\end{proof}","start":42008,"end":42378},{"raw_tex":"    Denoting the negative Shannon entropy by $\\phi$, we compute \n    \\begin{align*}\n        \\partial_t \\KL (\\mu, \\mu_t) = - \\partial_t \\phi(\\mu_t) - \\partial_t \\langle \\nabla \\phi(\\mu_t), \\mu - \\mu_t \\rangle = \\langle \\nabla^2 \\phi(\\mu_t)\\partial_t \\mu_t, \\mu_t - \\mu \\rangle.\n    \\end{align*}\n","start":42022,"end":42316},{"raw_tex":"    Now~\\eqref{eq:explicitFRGF} yields the claim. \n","start":42316,"end":42367}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"微分不等式与 Gronwall","source_units":[{"raw_tex":"Using~\\eqref{eq:derivativeu} and~\\eqref{eq:strongConvexity} we find that for all $t\\ge T$ it holds that \n$\\partial_t \\KL (\\mu^\\star, \\mu_t) = c^\\top \\mu_t  - c^\\top \\mu^\\star \\le - \\kappa_t \\KL (\\mu^\\star, \\mu_t)$. \nNow Gronwall's inequality yields~\\eqref{eq:lem:KLConvergence}. \n","start":42433,"end":42713}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"目标值界的极限尝试","source_units":[{"raw_tex":"By \\Cref{prop:sublinearRate} we have for any $h>0$ that \n    \\[ c^\\top \\mu^\\star - c^\\top \\mu_t \\le \\frac{\\KL (\\mu^\\star, \\mu_{t-h})}{h} \\le  \\KL (\\mu^\\star, \\mu_{0}) \\cdot \\frac{\\exp\\left(-\\int_{t_0}^{t-h} \\kappa_s \\D s \\right)}{h}. \\] \nTaking the limit $h\\to0$ yields~\\eqref{eq:lem:convergenceValue}. \n","start":42713,"end":43017}]},{"id":"S040","semantic_type":"lemma","kind":"lemma","labels":["eq:gapVSNorm-2","lem:TV-LP-estimate-withoutUniqueness"],"member_ids":["S040","S041","S042","S043","S044"],"declared_external":false,"source_only":false,"title":"邻边控制最优面距离","source_units":[{"raw_tex":"\\begin{restatable}{lemma}{TVLPEstimateWU}\n\\label{lem:TV-LP-estimate-withoutUniqueness}\nConsider a polytope $P \\subseteq \\R^{\\X}$ and denote by $F^\\star$ the face of maximizers of the linear function $\\mu\\mapsto c^\\top \\mu$ over $P$. \nDenote the set of neighboring vertices of a vertex $\\mu$ by $N(\\mu)$ and let $\\lVert \\cdot\\rVert\\colon\\R^{\\X}\\to\\R_{\\ge0}$ be an arbitrary semi-norm. \nThen either $F^\\star = P$ or with $\\frac{c}{0}\\coloneqq+\\infty$ for $c>0$, we have \n\\begin{equation}\n    \\Delta\\coloneqq \\min \\left\\{ \\frac{c^\\top\\mu^\\star - c^\\top\\mu }{\\lVert \\mu^\\star - \\mu \\rVert} : \\mu^\\star \\in\\operatorname{vert}(F^\\star), \\mu\\in N(\\mu^\\star)\\setminus F^\\star \\right\\} > 0,\n\\end{equation}\nand further \n\\begin{equation}\\label{eq:gapVSNorm-2}\n     c^\\top \\mu^\\star - c^\\top \\mu \\ge \\Delta \\cdot \\inf_{\\mu^\\star\\in F^\\star}  \\lVert \\mu^\\star - \\mu\\rVert \\quad \\text{for all } \\mu\\in P.\n\\end{equation}\n\\end{restatable}","start":43603,"end":44525},{"raw_tex":"\\begin{proof}\nIf $F^\\star \\ne P$, then $c^\\top \\mu^\\star-c^\\top \\mu >0$ for some vertex $\\mu$, which implies $\\Delta>0$. \nTo simplify notation we denote the set $E\\coloneqq\\{\\mu-\\mu^\\star: \\mu\\in N(\\mu^\\star) \\setminus F^\\star, \\mu^\\star\\in \\operatorname{vert}(F^\\star)\\}$ of edges such that exactly one of the two endpoints is contained in $F^\\star$. \nThen, the polytope $P$ is contained in \n    \\[ F^\\star+C = \\left\\{ \\mu^\\star + \\sum_{e\\in E} \\alpha_e e : \\mu^\\star\\in F^\\star, \\alpha_e\\ge0 \\text{ for all } e\\in E \\right\\}, \\]\nsee \\Cref{app:lem:cone} and hence we can write $\\mu\\in P$ as $\\mu = \\mu^\\star + \\sum_{e} \\alpha_e e$ for some $\\mu^\\star\\in F^\\star$. \nUsing the triangle inequality we obtain \n    \\[ \\Delta \\lVert \\mu^\\star - \\mu \\rVert \\le \\Delta \\sum_{e\\in E}\\alpha_e \\lVert e \\rVert \\le - \\sum_{e\\in E} \\alpha_e c^\\top e = c^\\top \\mu^\\star - c^\\top\\mu. \\]\n\\end{proof}","start":44526,"end":45410},{"raw_tex":"If $F^\\star \\ne P$, then $c^\\top \\mu^\\star-c^\\top \\mu >0$ for some vertex $\\mu$, which implies $\\Delta>0$. \nTo simplify notation we denote the set $E\\coloneqq\\{\\mu-\\mu^\\star: \\mu\\in N(\\mu^\\star) \\setminus F^\\star, \\mu^\\star\\in \\operatorname{vert}(F^\\star)\\}$ of edges such that exactly one of the two endpoints is contained in $F^\\star$. \n","start":44540,"end":44879},{"raw_tex":"Then, the polytope $P$ is contained in \n    \\[ F^\\star+C = \\left\\{ \\mu^\\star + \\sum_{e\\in E} \\alpha_e e : \\mu^\\star\\in F^\\star, \\alpha_e\\ge0 \\text{ for all } e\\in E \\right\\}, \\]\nsee \\Cref{app:lem:cone} and hence we can write $\\mu\\in P$ as $\\mu = \\mu^\\star + \\sum_{e} \\alpha_e e$ for some $\\mu^\\star\\in F^\\star$. \n","start":44879,"end":45192},{"raw_tex":"Using the triangle inequality we obtain \n    \\[ \\Delta \\lVert \\mu^\\star - \\mu \\rVert \\le \\Delta \\sum_{e\\in E}\\alpha_e \\lVert e \\rVert \\le - \\sum_{e\\in E} \\alpha_e c^\\top e = c^\\top \\mu^\\star - c^\\top\\mu. \\]\n","start":45192,"end":45399}]},{"id":"S045","semantic_type":"lemma","kind":"lemma","labels":["eq:KLvsL1","lem:KLEstimateL1"],"member_ids":["S045","S046","S047","S048","S049","S050","S051","S052","S053"],"declared_external":false,"source_only":false,"title":"带参数的局部 KL-TV 界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:KLEstimateL1}\nConsider a finite set $\\X$ and a probability distribution $\\mu\\in \\Delta_\\X$. Let $c>1$ and set $\\delta\\coloneqq \\frac{c-1}{c+1} \n    \\cdot \\min\\{ \\mu_x : \\mu_x>0 \\}>0$. \nThen for all $\\nu\\in\\Delta_\\X$ satisfying $\\lVert \\mu-\\nu\\rVert_\\infty\\le\\delta$ it holds that \n\\begin{equation}\\label{eq:KLvsL1}\n    \\KL (\\mu, \\nu) \\le c \\cdot \\lVert \\mu - \\nu \\rVert_{\\TV }. \n\\end{equation}\n\\end{lemma}","start":45412,"end":45841},{"raw_tex":"\\begin{proof}\nWe bound the individual summands in the KL-divergence\n    \\[ \\KL (\\mu, \\nu) = \\sum_{x\\in\\X} \\mu_x \\log\\left( \\frac{\\mu_x}{\\nu_x}\\right) = \\sum_{x\\in X} \\mu_x \\log\\left( \\frac{\\mu_x}{\\nu_x}\\right), \\]\nwhere $X\\coloneqq\\{x\\in\\X : \\mu_x>0\\}$. \nIf $\\mu_x,\\nu_x>0$ then \n\\begin{align}\\label{eq:klIndividual}\n    \\begin{split}\n        \\mu_x \\log \\left(\\frac{\\mu_x}{\\nu_x}\\right) & = \\mu_x \\left( \\log(\\nu_x+(\\mu_x-\\nu_x)) - \\log(\\nu_x) \\right) \\\\\n        & \\le \\mu_x \\left( \\log(\\nu_x) + \\frac{\\mu_x-\\nu_x}{\\nu_x} - \\log(\\nu_x) \\right) \n         = (\\mu_x - \\nu_x)\\cdot\\frac{\\mu_x}{\\nu_x},\n    \\end{split}\n\\end{align}\nwhere we used the convexity $\\log(t+h) \\le \\log(t) + h/t$ for $t>0, t+h>0$. \nWe set $\\varepsilon\\coloneqq\\frac{c-1}{2}\\in(0,1)$, such that \n\\[ \\delta = \\frac{\\varepsilon}{1+\\varepsilon}  \n    \\cdot \\min\\big\\{ \\mu_x : \\mu_x>0 \\big\\}. \n    \\]\nIf $\\lVert \\mu-\\nu\\rVert_\\infty\\le\\delta$ then \n    \\[ \\nu_x \\ge \\mu_x - \\delta \\ge \\mu_x \\left(1-\\frac{\\varepsilon}{1+\\varepsilon}\\right) = \\frac{\\mu_x}{1+\\varepsilon} \\]\nas well as\n    \\[ \\nu_x \\le \\mu_x + \\delta  \\le \\mu_x \\left(1 + \\frac{\\varepsilon}{1+\\varepsilon}\\right) \\le \\mu_x \\left(1 + \\frac{\\varepsilon}{1-\\varepsilon}\\right) = \\frac{\\mu_x}{1-\\varepsilon} \\]\nand therefore $1-\\varepsilon\\le \\frac{\\mu_x}{\\nu_x}\\le1+\\varepsilon$. \nIf $\\mu_x\\ge\\nu_x$ then \n    \\[ \n    (\\mu_x-\\nu_x)\\cdot \\frac{\\mu_x}{\\nu_x} \n    \\le (1+\\varepsilon)(\\mu_x-\\nu_x) \n    = \\mu_x-\\nu_x + \\varepsilon \\lvert \\mu_x - \\nu_x \\rvert , \n    \\]\nand if $\\mu_x<\\nu_x$ then \n\\begin{equation}\\label{eq:almostThere}\n    (\\mu_x-\\nu_x)\\cdot \\frac{\\mu_x}{\\nu_x} \\le (1-\\varepsilon)(\\mu_x-\\nu_x) = \\mu_x-\\nu_x + \\varepsilon \\lvert \\mu_x - \\nu_x \\rvert. \n\\end{equation}\nTogether with~\\eqref{eq:klIndividual} summing over $x$ yields \n\\begin{equation}\\label{eq:there}\n    \\KL (\\mu, \\nu) \\le \\sum_{x\\in X} (\\mu_x-\\nu_x) + \\varepsilon \\sum_{x\\in X} \\lvert \\mu_x - \\nu_x \\rvert \\le \\sum_{x\\in X} (\\mu_x-\\nu_x) + 2 \\varepsilon \\lVert \\mu-\\nu\\rVert_{\\TV } . \n\\end{equation}\nIt remains to estimate the first part. \nSetting $X^c\\coloneqq\\X\\setminus X$, we have \n\\[ \n\\sum_{x\\in X} (\\mu_x-\\nu_x) = \\sum_{x\\in\\X} (\\mu_x-\\nu_x) -  \\sum_{x\\in X^c} (\\mu_x-\\nu_x) = -  \\sum_{x\\in X^c} (\\mu_x-\\nu_x) = \\sum_{x\\in X^c} \\lvert  \\mu_x-\\nu_x \\rvert  \n\\]\nsince $\\mu_x=0$ for $x\\in X^c$. \nNow we can estimate\n\\begin{align}\\label{eq:lastStep}\n    2\\sum_{x\\in X} (\\mu_x-\\nu_x) = \\sum_{x\\in X} (\\mu_x-\\nu_x) + \\sum_{x\\in X^c} \\lvert  \\mu_x-\\nu_x \\rvert \\le \\lVert \\mu-\\nu\\rVert_1 = 2\\lVert \\mu-\\nu\\rVert_{\\TV }.\n\\end{align}\nCombining~\\eqref{eq:there} and~\\eqref{eq:lastStep} yields\n\\[ \\KL (\\mu, \\nu) \\le (1+2\\varepsilon)\\lVert \\mu - \\nu \\rVert_{\\TV } = c\\cdot\\lVert \\mu - \\nu \\rVert_{\\TV }. \\]\n\\end{proof}","start":45842,"end":48559},{"raw_tex":"We bound the individual summands in the KL-divergence\n    \\[ \\KL (\\mu, \\nu) = \\sum_{x\\in\\X} \\mu_x \\log\\left( \\frac{\\mu_x}{\\nu_x}\\right) = \\sum_{x\\in X} \\mu_x \\log\\left( \\frac{\\mu_x}{\\nu_x}\\right), \\]\nwhere $X\\coloneqq\\{x\\in\\X : \\mu_x>0\\}$. \n","start":45856,"end":46097},{"raw_tex":"If $\\mu_x,\\nu_x>0$ then \n\\begin{align}\\label{eq:klIndividual}\n    \\begin{split}\n        \\mu_x \\log \\left(\\frac{\\mu_x}{\\nu_x}\\right) & = \\mu_x \\left( \\log(\\nu_x+(\\mu_x-\\nu_x)) - \\log(\\nu_x) \\right) \\\\\n        & \\le \\mu_x \\left( \\log(\\nu_x) + \\frac{\\mu_x-\\nu_x}{\\nu_x} - \\log(\\nu_x) \\right) \n         = (\\mu_x - \\nu_x)\\cdot\\frac{\\mu_x}{\\nu_x},\n    \\end{split}\n\\end{align}\nwhere we used the convexity $\\log(t+h) \\le \\log(t) + h/t$ for $t>0, t+h>0$. \n","start":46097,"end":46544},{"raw_tex":"We set $\\varepsilon\\coloneqq\\frac{c-1}{2}\\in(0,1)$, such that \n\\[ \\delta = \\frac{\\varepsilon}{1+\\varepsilon}  \n    \\cdot \\min\\big\\{ \\mu_x : \\mu_x>0 \\big\\}. \n    \\]\nIf $\\lVert \\mu-\\nu\\rVert_\\infty\\le\\delta$ then \n    \\[ \\nu_x \\ge \\mu_x - \\delta \\ge \\mu_x \\left(1-\\frac{\\varepsilon}{1+\\varepsilon}\\right) = \\frac{\\mu_x}{1+\\varepsilon} \\]\nas well as\n    \\[ \\nu_x \\le \\mu_x + \\delta  \\le \\mu_x \\left(1 + \\frac{\\varepsilon}{1+\\varepsilon}\\right) \\le \\mu_x \\left(1 + \\frac{\\varepsilon}{1-\\varepsilon}\\right) = \\frac{\\mu_x}{1-\\varepsilon} \\]\nand therefore $1-\\varepsilon\\le \\frac{\\mu_x}{\\nu_x}\\le1+\\varepsilon$. \n","start":46544,"end":47150},{"raw_tex":"If $\\mu_x\\ge\\nu_x$ then \n    \\[ \n    (\\mu_x-\\nu_x)\\cdot \\frac{\\mu_x}{\\nu_x} \n    \\le (1+\\varepsilon)(\\mu_x-\\nu_x) \n    = \\mu_x-\\nu_x + \\varepsilon \\lvert \\mu_x - \\nu_x \\rvert , \n    \\]\nand if $\\mu_x<\\nu_x$ then \n\\begin{equation}\\label{eq:almostThere}\n    (\\mu_x-\\nu_x)\\cdot \\frac{\\mu_x}{\\nu_x} \\le (1-\\varepsilon)(\\mu_x-\\nu_x) = \\mu_x-\\nu_x + \\varepsilon \\lvert \\mu_x - \\nu_x \\rvert. \n\\end{equation}\n","start":47150,"end":47550},{"raw_tex":"Together with~\\eqref{eq:klIndividual} summing over $x$ yields \n\\begin{equation}\\label{eq:there}\n    \\KL (\\mu, \\nu) \\le \\sum_{x\\in X} (\\mu_x-\\nu_x) + \\varepsilon \\sum_{x\\in X} \\lvert \\mu_x - \\nu_x \\rvert \\le \\sum_{x\\in X} (\\mu_x-\\nu_x) + 2 \\varepsilon \\lVert \\mu-\\nu\\rVert_{\\TV } . \n\\end{equation}\n","start":47550,"end":47847},{"raw_tex":"It remains to estimate the first part. \nSetting $X^c\\coloneqq\\X\\setminus X$, we have \n\\[ \n\\sum_{x\\in X} (\\mu_x-\\nu_x) = \\sum_{x\\in\\X} (\\mu_x-\\nu_x) -  \\sum_{x\\in X^c} (\\mu_x-\\nu_x) = -  \\sum_{x\\in X^c} (\\mu_x-\\nu_x) = \\sum_{x\\in X^c} \\lvert  \\mu_x-\\nu_x \\rvert  \n\\]\nsince $\\mu_x=0$ for $x\\in X^c$. \nNow we can estimate\n\\begin{align}\\label{eq:lastStep}\n    2\\sum_{x\\in X} (\\mu_x-\\nu_x) = \\sum_{x\\in X} (\\mu_x-\\nu_x) + \\sum_{x\\in X^c} \\lvert  \\mu_x-\\nu_x \\rvert \\le \\lVert \\mu-\\nu\\rVert_1 = 2\\lVert \\mu-\\nu\\rVert_{\\TV }.\n\\end{align}\n","start":47847,"end":48378},{"raw_tex":"Combining~\\eqref{eq:there} and~\\eqref{eq:lastStep} yields\n\\[ \\KL (\\mu, \\nu) \\le (1+2\\varepsilon)\\lVert \\mu - \\nu \\rVert_{\\TV } = c\\cdot\\lVert \\mu - \\nu \\rVert_{\\TV }. \\]\n","start":48378,"end":48548}]},{"id":"S054","semantic_type":"corollary","kind":"corollary","labels":["cor:TV-KL","eq:KLvsL1-2"],"member_ids":["S054","S055","S056"],"declared_external":false,"source_only":false,"title":"显式局部 KL-TV 系数","source_units":[{"raw_tex":"\\begin{corollary}[Local KL-TV estimate]\\label{cor:TV-KL}\nConsider a finite set $\\X$ and a probability distribution $\\mu\\in \\Delta_\\X$. \nThen for all $\\nu\\in\\Delta_\\X$ satisfying \n\\begin{equation}\n    \\lVert \\mu - \\nu \\rVert_\\infty < \\min\\{ \\mu_x : \\mu_x>0 \\} \n\\end{equation}\nit holds that \n\\begin{equation}\\label{eq:KLvsL1-2}\n    \\KL (\\mu, \\nu) \\le \\frac{\\min\\{ \\mu_x : \\mu_x>0 \\} +  \\lVert \\mu-\\nu\\rVert_\\infty}{\\min\\{ \\mu_x : \\mu_x>0 \\} - \\lVert \\mu-\\nu\\rVert_\\infty} \\cdot \\lVert \\mu - \\nu \\rVert_{\\TV }. \n\\end{equation}\n\\end{corollary}","start":48562,"end":49101},{"raw_tex":"\\begin{proof}\n    This is a direct consequence of \\Cref{lem:KLEstimateL1}. \n    Indeed, for $\\varepsilon>0$ small enough we have $\\lVert \\mu - \\nu \\rVert_\\infty \\le \\frac{2-\\varepsilon}{2+\\varepsilon} \\cdot \\min\\{ \\mu_x : \\mu_x>0 \\}$ and thus by \\Cref{lem:KLEstimateL1} with $c=1+\\varepsilon$ we have $\\KL (\\mu, \\nu) \\le (1+\\varepsilon) \\lVert \\mu - \\nu \\rVert_{\\TV }$. \n    Note that $\\varepsilon>0$ was arbitrary. \n\\end{proof}","start":49102,"end":49530},{"raw_tex":"    This is a direct consequence of \\Cref{lem:KLEstimateL1}. \n    Indeed, for $\\varepsilon>0$ small enough we have $\\lVert \\mu - \\nu \\rVert_\\infty \\le \\frac{2-\\varepsilon}{2+\\varepsilon} \\cdot \\min\\{ \\mu_x : \\mu_x>0 \\}$ and thus by \\Cref{lem:KLEstimateL1} with $c=1+\\varepsilon$ we have $\\KL (\\mu, \\nu) \\le (1+\\varepsilon) \\lVert \\mu - \\nu \\rVert_{\\TV }$. \n    Note that $\\varepsilon>0$ was arbitrary. \n","start":49116,"end":49519}]},{"id":"S057","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\Cref{thm:convergenceFRGF}]\n    Setting $\\delta\\coloneqq \\min\\{ \\mu_x^\\star : \\mu_x^\\star>0 \\}$ and using \\Cref{lem:TV-LP-estimate-withoutUniqueness} with $\\lVert\\cdot\\rVert_{\\TV}$ and \\Cref{cor:TV-KL} we have \n    \\begin{equation*}\n        c^\\top \\mu^\\star - c^\\top \\mu_t \\ge \\Delta \\lVert \\mu^\\star - \\mu_t \\rVert_{\\TV } \\ge \\Delta\\cdot\\frac{\\delta -  \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty}{\\delta + \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty} \\cdot \\KL (\\mu^\\star, \\mu_t),\n    \\end{equation*}\n    if $\\lVert \\mu^\\star - \\mu_t \\rVert_\\infty < \\delta$.\n    By \\Cref{prop:sublinearRate} we have \n    \\begin{align*}\n        \\lVert \\mu^\\star - \\mu_t \\rVert_\\infty \\le 2 \\lVert \\mu^\\star - \\mu_t \\rVert_{\\TV } \\le \\frac{2c^\\top(\\mu^\\star-\\mu_t)}\\Delta \\le \\frac{2\\KL (\\mu^\\star,\\mu_0)}{\\Delta \\cdot t}.\n    \\end{align*}\n    Hence, for $t>t_0$ we have $\\lVert \\mu^\\star - \\mu_t \\rVert_\\infty < \\delta$. \n    In this case, we can estimate \n    \\begin{align*}\n        \\frac{\\delta -  \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty}{\\delta + \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty} & \\le \\frac{\\delta -  2 \\KL (\\mu^\\star, \\mu_0)\\Delta^{-1} t^{-1}} {\\delta + 2 \\KL (\\mu^\\star, \\mu_0)\\Delta^{-1} t^{-1}} \n        = \\frac{t - t_0}{t + t_0} \\eqqcolon \\kappa_t. \n    \\end{align*}\n    Thus for $t>t_0$ we have $c^\\top \\mu^\\star - c^\\top \\mu_t \\ge \\Delta \\kappa_t \\KL (\\mu^\\star, \\mu_t)$, and \\Cref{prop:ratesTrajectories} together with \n    \\begin{equation*}\n        \\int_{t_0}^t \\frac{s-t_0}{s+t_0} \\D s = s - 2 t_0 \\log(s+t_0)\\Big|^{s=t}_{s=t_0} = (t-t_0) - 2 t_0 \\log\\left( \\frac{t+t_0}{2t_0} \\right)\n    \\end{equation*}\n    yield the result. \n\\end{proof}","start":49612,"end":51257}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"局部目标差支配 KL","source_units":[{"raw_tex":"    Setting $\\delta\\coloneqq \\min\\{ \\mu_x^\\star : \\mu_x^\\star>0 \\}$ and using \\Cref{lem:TV-LP-estimate-withoutUniqueness} with $\\lVert\\cdot\\rVert_{\\TV}$ and \\Cref{cor:TV-KL} we have \n    \\begin{equation*}\n        c^\\top \\mu^\\star - c^\\top \\mu_t \\ge \\Delta \\lVert \\mu^\\star - \\mu_t \\rVert_{\\TV } \\ge \\Delta\\cdot\\frac{\\delta -  \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty}{\\delta + \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty} \\cdot \\KL (\\mu^\\star, \\mu_t),\n    \\end{equation*}\n    if $\\lVert \\mu^\\star - \\mu_t \\rVert_\\infty < \\delta$.\n","start":49663,"end":50180}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"次线性界给出进入时间","source_units":[{"raw_tex":"    By \\Cref{prop:sublinearRate} we have \n    \\begin{align*}\n        \\lVert \\mu^\\star - \\mu_t \\rVert_\\infty \\le 2 \\lVert \\mu^\\star - \\mu_t \\rVert_{\\TV } \\le \\frac{2c^\\top(\\mu^\\star-\\mu_t)}\\Delta \\le \\frac{2\\KL (\\mu^\\star,\\mu_0)}{\\Delta \\cdot t}.\n    \\end{align*}\n    Hence, for $t>t_0$ we have $\\lVert \\mu^\\star - \\mu_t \\rVert_\\infty < \\delta$. \n","start":50180,"end":50526}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"时间支配系数的比较","source_units":[{"raw_tex":"    In this case, we can estimate \n    \\begin{align*}\n        \\frac{\\delta -  \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty}{\\delta + \\lVert \\mu^\\star-\\mu_t\\rVert_\\infty} & \\le \\frac{\\delta -  2 \\KL (\\mu^\\star, \\mu_0)\\Delta^{-1} t^{-1}} {\\delta + 2 \\KL (\\mu^\\star, \\mu_0)\\Delta^{-1} t^{-1}} \n        = \\frac{t - t_0}{t + t_0} \\eqqcolon \\kappa_t. \n    \\end{align*}\n","start":50526,"end":50880}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"积分时间系数","source_units":[{"raw_tex":"    \\begin{equation*}\n        \\int_{t_0}^t \\frac{s-t_0}{s+t_0} \\D s = s - 2 t_0 \\log(s+t_0)\\Big|^{s=t}_{s=t_0} = (t-t_0) - 2 t_0 \\log\\left( \\frac{t+t_0}{2t_0} \\right)\n    \\end{equation*}\n","start":51036,"end":51223}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"应用轨迹引理完成主证明","source_units":[{"raw_tex":"    Thus for $t>t_0$ we have $c^\\top \\mu^\\star - c^\\top \\mu_t \\ge \\Delta \\kappa_t \\KL (\\mu^\\star, \\mu_t)$, and \\Cref{prop:ratesTrajectories} together with \n    \\begin{equation*}\n        \\int_{t_0}^t \\frac{s-t_0}{s+t_0} \\D s = s - 2 t_0 \\log(s+t_0)\\Big|^{s=t}_{s=t_0} = (t-t_0) - 2 t_0 \\log\\left( \\frac{t+t_0}{2t_0} \\right)\n    \\end{equation*}\n    yield the result. \n","start":50880,"end":51246}]},{"id":"S063","semantic_type":"lemma","kind":"lemma","labels":["app:lem:cone"],"member_ids":["S063","S064","S066","S067","S068"],"declared_external":false,"source_only":false,"title":"最优面出边的锥包含","source_units":[{"raw_tex":"\\begin{lemma}\\label{app:lem:cone}\n    Consider a polytope $P\\subseteq\\R^\\X$, a face $F\\subseteq P$ and consider the cone \n    \\begin{align*}\n        C\\coloneqq \\operatorname{cone}\\Big\\{ \\nu-\\mu : \\mu\\in\\operatorname{vert}(F), \\nu\\in N(\\mu)\\setminus F \\Big\\}, \n    \\end{align*}\n    which is generated by the edges pointing out of $F$. \n    Then we have $P\\subseteq F+C$. \n\\end{lemma}","start":106453,"end":106835},{"raw_tex":"\\begin{proof}\nThis is a generalization of~\\cite[Lemma 3.6]{ziegler2012lectures}, which covers the case that $F$ consists of a single vertex. \nWe will show that \n\\begin{align*}\n    F+C \\supseteq\\tilde P \\coloneqq \\bigcap \\Big\\{ H\\subseteq \\R^\\X : H \\text{ is a halfspace }, P\\subseteq H, H\\cap F\\ne\\emptyset \\Big\\} \\supseteq P, \n\\end{align*}\nfor which we pick an element $u\\in \\tilde P$. \nConsider a hyperplane $H = \\{\\mu:a^\\top \\mu = \\alpha\\}$ separating $F$ and $\\operatorname{vert}(P)\\setminus F$ and consider the \\emph{face figure} $P/F\\coloneqq P\\cap H$, which is a polytope. \nNow, we consider a translation $\\tilde H = \\{ \\mu : a^\\top\\mu= \\beta \\}$ of $H$,  such that $u\\in \\tilde H$. \nNow we have \n\\begin{align*}\n    \\tilde P = \\operatorname{conv}\\left\\{ \\mu + \\frac{a^\\top \\mu - \\beta}{a^\\top \\mu - a^\\top \\nu} \\cdot (\\nu-\\mu) : \\mu\\in\\operatorname{vert}(F), \\nu\\in N(\\mu)\\setminus F \\right\\},\n\\end{align*}\nsee~\\cite[Proposition 2.30]{zieglerdiscrete}. \nHence, we can choose convex weights $\\lambda_i$ such that \n\\begin{align*}\n    u = \\sum_{i} \\lambda_i (\\mu_i + \\alpha_i (\\nu_i-\\mu_i)) =  \\sum_{i} \\lambda_i \\mu_i + \\sum_{i} \\alpha_i \\lambda_i (\\nu_i-\\mu_i) \\in F + C,\n\\end{align*}\nwhere $\\mu_i\\in\\operatorname{vert}(F), \\nu_i \\in N(\\mu_i)\\setminus F$.\n\\end{proof}","start":106836,"end":108109},{"raw_tex":"We will show that \n\\begin{align*}\n    F+C \\supseteq\\tilde P \\coloneqq \\bigcap \\Big\\{ H\\subseteq \\R^\\X : H \\text{ is a halfspace }, P\\subseteq H, H\\cap F\\ne\\emptyset \\Big\\} \\supseteq P, \n\\end{align*}\nfor which we pick an element $u\\in \\tilde P$. \n","start":106978,"end":107224},{"raw_tex":"Consider a hyperplane $H = \\{\\mu:a^\\top \\mu = \\alpha\\}$ separating $F$ and $\\operatorname{vert}(P)\\setminus F$ and consider the \\emph{face figure} $P/F\\coloneqq P\\cap H$, which is a polytope. \nNow, we consider a translation $\\tilde H = \\{ \\mu : a^\\top\\mu= \\beta \\}$ of $H$,  such that $u\\in \\tilde H$. \nNow we have \n\\begin{align*}\n    \\tilde P = \\operatorname{conv}\\left\\{ \\mu + \\frac{a^\\top \\mu - \\beta}{a^\\top \\mu - a^\\top \\nu} \\cdot (\\nu-\\mu) : \\mu\\in\\operatorname{vert}(F), \\nu\\in N(\\mu)\\setminus F \\right\\},\n\\end{align*}\nsee~\\cite[Proposition 2.30]{zieglerdiscrete}. \n","start":107224,"end":107797},{"raw_tex":"Hence, we can choose convex weights $\\lambda_i$ such that \n\\begin{align*}\n    u = \\sum_{i} \\lambda_i (\\mu_i + \\alpha_i (\\nu_i-\\mu_i)) =  \\sum_{i} \\lambda_i \\mu_i + \\sum_{i} \\alpha_i \\lambda_i (\\nu_i-\\mu_i) \\in F + C,\n\\end{align*}\nwhere $\\mu_i\\in\\operatorname{vert}(F), \\nu_i \\in N(\\mu_i)\\setminus F$.\n","start":107797,"end":108098}]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S069"],"declared_external":true,"declaration_text":["原始 external_prerequisites：Standard finite-dimensional convex analysis，包括 first-order optimality 与 strict convexity。"],"source_only":false,"title":"凸优化的一阶刻画","source_units":[]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S070"],"declared_external":true,"declaration_text":["原始 external_prerequisites：Standard finite-dimensional convex analysis 中明确列出的 KL nonnegativity。"],"source_only":false,"title":"KL 非负性","source_units":[]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S071"],"declared_external":true,"declaration_text":["原始 external_prerequisites：Standard finite-dimensional convex analysis 中的 entropy boundary behavior。"],"source_only":false,"title":"熵的边界行为","source_units":[]},{"id":"S072","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S072"],"declared_external":true,"declaration_text":["原始 external_prerequisites：Gronwall's inequality，原文第528行调用。"],"source_only":false,"title":"Gronwall 不等式","source_units":[]},{"id":"S073","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S073"],"declared_external":true,"declaration_text":["原始 external_prerequisites：Picard-Lindelöf local existence and uniqueness theorem；原文第340行引用 teschl2024ordinary。"],"source_only":false,"title":"Picard-Lindelöf 定理","source_units":[]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S074"],"declared_external":true,"declaration_text":["原始 external_prerequisites：zieglerdiscrete, Proposition 2.30；原文第1371行引用。"],"source_only":false,"title":"面图的外部命题","source_units":[]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S075"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；属于原始先决条件所列标准有限维凸分析背景，原文第580行实际写出此不等式。"],"source_only":false,"title":"对数切线不等式","source_units":[]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S076"],"declared_external":true,"declaration_text":["原始 external_prerequisites：Standard finite-dimensional convex analysis，包括 elementary polytope separation。"],"source_only":false,"title":"多胞体面的严格分离","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S017":"S017","S018":"S018","S019":"S019","S020":"S019","S021":"S019","S022":"S019","S023":"S023","S024":"S023","S025":"S025","S026":"S025","S027":"S025","S028":"S025","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S033","S035":"S033","S036":"S033","S037":"S029","S038":"S038","S039":"S039","S040":"S040","S041":"S040","S042":"S040","S043":"S040","S044":"S040","S045":"S045","S046":"S045","S047":"S045","S048":"S045","S049":"S045","S050":"S045","S051":"S045","S052":"S045","S053":"S045","S054":"S054","S055":"S054","S056":"S054","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S064":"S063","S066":"S063","S067":"S063","S068":"S063","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076"}}}}
{"record_id":"2403.02530v3","paper":{"title":"Projected gradient descent accumulates at Bouligand stationary points","authors":["Olikier, Guillaume","Waldspurger, Irène"],"arxiv_id":"2403.02530","version":"v3","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/2403.02530v3","pdf_url":"https://arxiv.org/pdf/2403.02530v3","doi":"10.1137/24M1692782"},"context":{"raw_tex":"Let $\\mathcal{E}$ be a Euclidean vector space, $C$ a nonempty closed subset of $\\mathcal{E}$, and $f : \\mathcal{E} \\to \\R$ a function satisfying at least the first of the following two properties:\n\\begin{enumerate}[label={(H\\arabic*)}]\n\\item $f$ is differentiable on $C$, i.e., for every $x \\in C$, there exists a (unique) vector in $\\mathcal{E}$, denoted by $\\nabla f(x)$, such that\n\\begin{equation*}\n\\lim_{\\substack{y \\to x \\\\ y \\in \\mathcal{E} \\setminus \\{x\\}}} \\frac{f(y)-f(x)-\\ip{\\nabla f(x)}{y-x}}{\\norm{y-x}} = 0,\n\\end{equation*}\nand $\\nabla f : C \\to \\mathcal{E}$ is continuous;\n\\label{it:continuously_diff}\n\\item $f$ is differentiable on $\\mathcal{E}$ and $\\nabla f : \\mathcal{E} \\to \\mathcal{E}$ is locally Lipschitz continuous.\n\\label{it:Lipschitz_diff}\n\\end{enumerate}\nThis paper considers the problem\n\\begin{equation}\n\\label{eq:MinDiffFunctionClosedSet}\n\\min_{x \\in C} f(x)\n\\end{equation}\nof minimizing $f$ on~$C$.\n\n\nA point $x \\in C$ is said to be stationary for~\\eqref{eq:MinDiffFunctionClosedSet} if $-\\nabla f(x)$ is normal to $C$ at~$x$. Several definitions of normality exist. Each defines a different notion of stationarity, which is a surrogate for local optimality in the sense that, possibly under mild regularity assumptions on~$f$, every local minimizer of $f|_C$ is stationary for~\\eqref{eq:MinDiffFunctionClosedSet}. In particular, each of the three notions of normality in \\cite[Definition~6.3 and Example~6.16]{RockafellarWets}, namely normality in the general sense, in the regular sense, and in the proximal sense, yields an important definition of stationarity. The sets of general, regular, and proximal normals to $C$ at $x \\in C$ are respectively denoted by $\\norcone{C}{x}$, $\\regnorcone{C}{x}$, and $\\proxnorcone{C}{x}$. These sets are reviewed in section~\\ref{subsec:NormalityStationarity}. Importantly, they are nested as follows: for every $x \\in C$,\n\\begin{equation}\n\\label{eq:NestedNormalCones}\n\\proxnorcone{C}{x} \\subseteq \\regnorcone{C}{x} \\subseteq \\norcone{C}{x},\n\\end{equation}\nand $C$ is said to be \\emph{Clarke regular} at $x$ if the second inclusion is an equality.\n\n\n\\begin{definition}\n\\label{def:M/B/P-Stationarity}\nFor problem~\\eqref{eq:MinDiffFunctionClosedSet}, a point $x \\in C$ is said to be:\n\\begin{itemize}\n\\item \\emph{Mordukhovich stationary (M-stationary)} if $-\\nabla f(x) \\in \\norcone{C}{x}$;\n\\item \\emph{Bouligand stationary (B-stationary)} if $-\\nabla f(x) \\in \\regnorcone{C}{x}$;\n\\item \\emph{proximally stationary (P-stationary)} if $-\\nabla f(x) \\in \\proxnorcone{C}{x}$.\n\\end{itemize}\n\\end{definition}\n\n\nMonotone $\\pgd$ accumulates at M-stationary points of~\\eqref{eq:MinDiffFunctionClosedSet} if $f$ is continuously differentiable on~$\\mathcal{E}$ and bounded from below on~$C$ \\cite[Theorem~3.1]{KanzowMehlitz}. The same result holds for nonmonotone $\\pgd$ with the ``average'' rule \\cite[Theorem~4.6]{DeMarchi}, and also for nonmonotone $\\pgd$ with the ``max'' rule if $f$ is further uniformly continuous on the sublevel set\n\\begin{equation}\n\\label{eq:InitialSublevelSet}\n\\{x \\in C \\mid f(x) \\le f(x_0)\\},\n\\end{equation}\nwhere $x_0 \\in C$ is the initial iterate given to the algorithm \\cite[Theorem~4.1]{KanzowMehlitz}.\n\n\nRecall that, throughout the paper, $\\mathcal{E}$ is a Euclidean vector space and $C \\subseteq \\mathcal{E}$ is nonempty and closed. Moreover, for every $x \\in \\mathcal{E}$ and $\\rho \\in (0, \\infty)$, $\\ball(x, \\rho) \\coloneq \\{y \\in \\mathcal{E} \\mid \\norm{x-y} < \\rho\\}$ and $\\ball[x, \\rho] \\coloneq \\{y \\in \\mathcal{E} \\mid \\norm{x-y} \\le \\rho\\}$ are respectively the open and closed balls of center $x$ and radius $\\rho$ in~$\\mathcal{E}$.\nA nonempty subset $K$ of $\\mathcal{E}$ is called a \\emph{cone} if $x \\in K$ implies $\\alpha x \\in K$ for all $\\alpha \\in [0, \\infty)$ \\cite[\\S 3B]{RockafellarWets}.\n\n\nGiven $x \\in \\mathcal{E}$, the distance from $x$ to $C$ is $\\dist(x, C) \\coloneq \\min_{y \\in C} \\norm{x-y}$ and the projection of $x$ onto $C$ is $\\proj{C}{x} \\coloneq \\argmin_{y \\in C} \\norm{x-y}$. The function $\\mathcal{E} \\to \\R : x \\mapsto \\dist(x, C)$ is continuous and, for every $x \\in \\mathcal{E}$, the set $\\proj{C}{x}$ is nonempty and compact \\cite[Example~1.20]{RockafellarWets}.\nProposition~\\ref{prop:ProjectedTranslation} is invoked frequently in the rest of the paper.\n\n\nA vector $v \\in \\mathcal{E}$ is said to be \\emph{tangent} to $C$ at $x \\in C$ if there exist sequences $(x_i)_{i \\in \\N}$ in $C$ converging to $x$ and $(t_i)_{i \\in \\N}$ in $(0, \\infty)$ such that the sequence $(\\frac{x_i-x}{t_i})_{i \\in \\N}$ converges to $v$ \\cite[Definition~6.1]{RockafellarWets}. The set of all tangent vectors to $C$ at $x \\in C$ is a closed cone \\cite[Proposition~6.2]{RockafellarWets} called the \\emph{tangent cone} to $C$ at $x$ and denoted by $\\tancone{C}{x}$.\nThe \\emph{regular normal cone} to $C$ at $x \\in C$ is\n\\begin{equation*}\n\\regnorcone{C}{x} \\coloneq \\left\\{v \\in \\mathcal{E} \\mid \\ip{v}{w} \\le 0 \\; \\forall w \\in \\tancone{C}{x}\\right\\},\n\\end{equation*}\nwhich is a closed convex cone \\cite[Definition~6.3 and Proposition~6.5]{RockafellarWets}.\nA vector $v \\in \\mathcal{E}$ is said to be \\emph{normal (in the general sense)} to $C$ at $x \\in C$ if there exist sequences $(x_i)_{i \\in \\N}$ in $C$ converging to $x$ and $(v_i)_{i \\in \\N}$ converging to $v$ such that, for all $i \\in \\N$, $v_i \\in \\regnorcone{C}{x_i}$ \\cite[Definition~6.3]{RockafellarWets}. The set of all normal vectors to $C$ at $x \\in C$ is a closed cone \\cite[Proposition~6.5]{RockafellarWets} called the \\emph{normal cone} to $C$ at $x$ and denoted by $\\norcone{C}{x}$.\nA vector $v \\in \\mathcal{E}$ is called a \\emph{proximal normal} to $C$ at $x \\in C$ if there exists $\\oshort{\\alpha} \\in (0, \\infty)$ such that $x \\in \\proj{C}{x+\\oshort{\\alpha}v}$, i.e., $\\oshort{\\alpha}\\norm{v} = \\dist(x+\\oshort{\\alpha}v, C)$, which implies that, for all $\\alpha \\in [0, \\oshort{\\alpha})$, $\\proj{C}{x+\\alpha v} = \\{x\\}$ \\cite[Example~6.16]{RockafellarWets}. The set of all proximal normals to $C$ at $x \\in C$ is a convex cone called the \\emph{proximal normal cone} to $C$ at $x$ and denoted by $\\proxnorcone{C}{x}$.\n\n\n\\begin{algorithm}[H]\n\\caption{$\\pgd$ map}\n\\label{algo:PGDmap}\n\\begin{algorithmic}[1]\n\\Require\n$(\\mathcal{E}, C, f, \\ushort{\\alpha}, \\oshort{\\alpha}, \\beta, c)$, where $\\mathcal{E}$ is a Euclidean vector space, $C$ is a nonempty closed subset of~$\\mathcal{E}$, $f : \\mathcal{E} \\to \\R$ satisfies \\ref{it:continuously_diff}, $0 < \\ushort{\\alpha} \\le \\oshort{\\alpha} < \\infty$, and $\\beta, c \\in (0, 1)$.\n\\Input\n$(x, \\mu)$ with $x \\in C$ and $\\mu \\in [f(x), \\infty)$.\n\\Output\n$y \\in \\pgd(x, \\mu; \\mathcal{E}, C, f, \\ushort{\\alpha}, \\oshort{\\alpha}, \\beta, c)$.\n\n\\State\nChoose $\\alpha \\in [\\ushort{\\alpha}, \\oshort{\\alpha}]$ and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$;\n\\label{algo:PGDmap:InitialStepSize}\n\\While\n{$f(y) > \\mu + c \\ip{\\nabla f(x)}{y-x}$}\n\\label{algo:PGDmap:NonmonotoneArmijoCondition}\n\\State\n$\\alpha \\gets \\alpha \\beta$;\n\\State\nChoose $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$;\n\\EndWhile\n\\State\nReturn $y$.\n\\end{algorithmic}\n\\end{algorithm}\n\n\n\\begin{remark}\nThe Armijo condition\n\\begin{equation*}\nf(y) \\le \\mu + c \\ip{\\nabla f(x)}{y-x}\n\\end{equation*}\nensures that the decrease $\\mu-f(y)$ is at least a fraction $c$ of the opposite of the directional derivative of $f$ at $x$ along the update vector $y-x$. By \\eqref{eq:ProjectedTranslationIP}, this condition implies that\n\\begin{equation}\\label{eq:KanzowMehlitzDecay}\nf(y) \\le \\mu - \\frac{c}{2\\alpha} \\norm{y-x}^2,\n\\end{equation}\nwhich is the condition used in \\cite[Algorithms~3.1 and 4.1]{KanzowMehlitz} and \\cite[Algorithm~3.1]{DeMarchi}. Importantly, all results from \\cite{KanzowMehlitz} hold for both conditions, as is clear from the proofs.\n% Par contre, les résultats de De Marchi ne sont pas clairement valides avec notre condition (je pense au lemme 4.4).\n\\end{remark}\n\n\nThe $\\pgd$ algorithm is defined as Algorithm~\\ref{algo:PGD}. It is said to be monotone or nonmonotone depending on whether $\\mu_i=f(x_i)$ for all $i$ (that is, $l=0$ for the ``max'' rule, or $p=1$ for the ``average'' rule) or not.\n\n\\begin{algorithm}[H]\n\\caption{$\\pgd$}\n\\label{algo:PGD}\n\\begin{algorithmic}[1]\n\\Require\n$(\\mathcal{E}, C, f, \\ushort{\\alpha}, \\oshort{\\alpha}, \\beta, c, \\text{``nonmonotonicity''})$, where $\\mathcal{E}$ is a Euclidean vector space, $C$ is a nonempty closed subset of~$\\mathcal{E}$, $f : \\mathcal{E} \\to \\R$ satisfies \\ref{it:continuously_diff}, $0 < \\ushort{\\alpha} \\le \\oshort{\\alpha} < \\infty$, $\\beta, c \\in (0, 1)$, and $\\text{``nonmonotonicity''} \\in \\{(\\text{``max''}, l), (\\text{``average''}, p)\\}$ with $l \\in \\N$ and $p \\in (0, 1]$.\n\\Input\n$x_0 \\in C$.\n\\Output\na sequence in $C$.\n\n\\State\n$i \\gets 0$;\n\\State\n$\\mu_{-1} \\gets f(x_0)$;\n\\While\n{$-\\nabla f(x_i) \\notin \\regnorcone{C}{x_i}$}\n\\label{algo:PGD:while}\n\\If\n{$\\text{``nonmonotonicity''} = (\\text{``max''}, l)$}\n\\State\n$\\mu_i \\gets \\max_{j \\in \\{\\max\\{0, i-l\\}, \\dots, i\\}} f(x_j)$;\n\\ElsIf\n{$\\text{``nonmonotonicity''} = (\\text{``average''}, p)$}\n\\State\n$\\mu_i \\gets (1-p)\\mu_{i-1} + pf(x_i)$; \n\\EndIf\n\\State\nChoose $x_{i+1} \\in \\hyperref[algo:PGDmap]{\\pgd}(x_i, \\mu_i; \\mathcal{E}, C, f, \\ushort{\\alpha}, \\oshort{\\alpha}, \\beta, c)$;\n\\label{algo:PGD:PGDmap}\n\\State\n$i \\gets i+1$;\n\\EndWhile\n\\end{algorithmic}\n\\end{algorithm}\n\n\n\\begin{proposition}\n\\label{prop:NonmonotonePGD}\nLet $(x_i)_{i \\in \\N}$ be a sequence generated by $\\pgd$ (Algorithm~\\ref{algo:PGD}) using the ``max'' rule. For every $i \\in \\N$, let $g(i) \\in \\argmax_{j \\in \\{\\max\\{0, i-l\\}, \\dots, i\\}} f(x_j)$. Then:\n\\begin{enumerate}\n\\item $(f(x_{g(i)}))_{i \\in \\N}$ is monotonically nonincreasing;\n\\item $(x_i)_{i \\in \\N}$ is contained in the sublevel set~\\eqref{eq:InitialSublevelSet};\n\\item if $x \\in C$ is an accumulation point of $(x_i)_{i \\in \\N}$, then $(f(x_{g(i)}))_{i \\in \\N}$ converges to $\\varphi \\in [f(x), f(x_0)]$;\n\\item if $f$ is bounded from below and uniformly continuous on a set that contains $(x_i)_{i \\in \\N}$, then $(f(x_i))_{i \\in \\N}$ converges to $\\varphi \\in \\R$.\n\\end{enumerate}\n\\end{proposition}","units":[{"raw_tex":"Let $\\mathcal{E}$ be a Euclidean vector space, $C$ a nonempty closed subset of $\\mathcal{E}$, and $f : \\mathcal{E} \\to \\R$ a function satisfying at least the first of the following two properties:\n\\begin{enumerate}[label={(H\\arabic*)}]\n\\item $f$ is differentiable on $C$, i.e., for every $x \\in C$, there exists a (unique) vector in $\\mathcal{E}$, denoted by $\\nabla f(x)$, such that\n\\begin{equation*}\n\\lim_{\\substack{y \\to x \\\\ y \\in \\mathcal{E} \\setminus \\{x\\}}} \\frac{f(y)-f(x)-\\ip{\\nabla f(x)}{y-x}}{\\norm{y-x}} = 0,\n\\end{equation*}\nand $\\nabla f : C \\to \\mathcal{E}$ is continuous;\n\\label{it:continuously_diff}\n\\item $f$ is differentiable on $\\mathcal{E}$ and $\\nabla f : \\mathcal{E} \\to \\mathcal{E}$ is locally Lipschitz continuous.\n\\label{it:Lipschitz_diff}\n\\end{enumerate}\nThis paper considers the problem\n\\begin{equation}\n\\label{eq:MinDiffFunctionClosedSet}\n\\min_{x \\in C} f(x)\n\\end{equation}\nof minimizing $f$ on~$C$.\n","start":5818,"end":6746},{"raw_tex":"A point $x \\in C$ is said to be stationary for~\\eqref{eq:MinDiffFunctionClosedSet} if $-\\nabla f(x)$ is normal to $C$ at~$x$. Several definitions of normality exist. Each defines a different notion of stationarity, which is a surrogate for local optimality in the sense that, possibly under mild regularity assumptions on~$f$, every local minimizer of $f|_C$ is stationary for~\\eqref{eq:MinDiffFunctionClosedSet}. In particular, each of the three notions of normality in \\cite[Definition~6.3 and Example~6.16]{RockafellarWets}, namely normality in the general sense, in the regular sense, and in the proximal sense, yields an important definition of stationarity. The sets of general, regular, and proximal normals to $C$ at $x \\in C$ are respectively denoted by $\\norcone{C}{x}$, $\\regnorcone{C}{x}$, and $\\proxnorcone{C}{x}$. These sets are reviewed in section~\\ref{subsec:NormalityStationarity}. Importantly, they are nested as follows: for every $x \\in C$,\n\\begin{equation}\n\\label{eq:NestedNormalCones}\n\\proxnorcone{C}{x} \\subseteq \\regnorcone{C}{x} \\subseteq \\norcone{C}{x},\n\\end{equation}\nand $C$ is said to be \\emph{Clarke regular} at $x$ if the second inclusion is an equality.\n","start":7214,"end":8400},{"raw_tex":"\\begin{definition}\n\\label{def:M/B/P-Stationarity}\nFor problem~\\eqref{eq:MinDiffFunctionClosedSet}, a point $x \\in C$ is said to be:\n\\begin{itemize}\n\\item \\emph{Mordukhovich stationary (M-stationary)} if $-\\nabla f(x) \\in \\norcone{C}{x}$;\n\\item \\emph{Bouligand stationary (B-stationary)} if $-\\nabla f(x) \\in \\regnorcone{C}{x}$;\n\\item \\emph{proximally stationary (P-stationary)} if $-\\nabla f(x) \\in \\proxnorcone{C}{x}$.\n\\end{itemize}\n\\end{definition}\n","start":8585,"end":9036},{"raw_tex":"Monotone $\\pgd$ accumulates at M-stationary points of~\\eqref{eq:MinDiffFunctionClosedSet} if $f$ is continuously differentiable on~$\\mathcal{E}$ and bounded from below on~$C$ \\cite[Theorem~3.1]{KanzowMehlitz}. The same result holds for nonmonotone $\\pgd$ with the ``average'' rule \\cite[Theorem~4.6]{DeMarchi}, and also for nonmonotone $\\pgd$ with the ``max'' rule if $f$ is further uniformly continuous on the sublevel set\n\\begin{equation}\n\\label{eq:InitialSublevelSet}\n\\{x \\in C \\mid f(x) \\le f(x_0)\\},\n\\end{equation}\nwhere $x_0 \\in C$ is the initial iterate given to the algorithm \\cite[Theorem~4.1]{KanzowMehlitz}.\n","start":13337,"end":13956},{"raw_tex":"Recall that, throughout the paper, $\\mathcal{E}$ is a Euclidean vector space and $C \\subseteq \\mathcal{E}$ is nonempty and closed. Moreover, for every $x \\in \\mathcal{E}$ and $\\rho \\in (0, \\infty)$, $\\ball(x, \\rho) \\coloneq \\{y \\in \\mathcal{E} \\mid \\norm{x-y} < \\rho\\}$ and $\\ball[x, \\rho] \\coloneq \\{y \\in \\mathcal{E} \\mid \\norm{x-y} \\le \\rho\\}$ are respectively the open and closed balls of center $x$ and radius $\\rho$ in~$\\mathcal{E}$.\nA nonempty subset $K$ of $\\mathcal{E}$ is called a \\emph{cone} if $x \\in K$ implies $\\alpha x \\in K$ for all $\\alpha \\in [0, \\infty)$ \\cite[\\S 3B]{RockafellarWets}.\n","start":18485,"end":19090},{"raw_tex":"Given $x \\in \\mathcal{E}$, the distance from $x$ to $C$ is $\\dist(x, C) \\coloneq \\min_{y \\in C} \\norm{x-y}$ and the projection of $x$ onto $C$ is $\\proj{C}{x} \\coloneq \\argmin_{y \\in C} \\norm{x-y}$. The function $\\mathcal{E} \\to \\R : x \\mapsto \\dist(x, C)$ is continuous and, for every $x \\in \\mathcal{E}$, the set $\\proj{C}{x}$ is nonempty and compact \\cite[Example~1.20]{RockafellarWets}.\nProposition~\\ref{prop:ProjectedTranslation} is invoked frequently in the rest of the paper.\n","start":19148,"end":19631},{"raw_tex":"A vector $v \\in \\mathcal{E}$ is said to be \\emph{tangent} to $C$ at $x \\in C$ if there exist sequences $(x_i)_{i \\in \\N}$ in $C$ converging to $x$ and $(t_i)_{i \\in \\N}$ in $(0, \\infty)$ such that the sequence $(\\frac{x_i-x}{t_i})_{i \\in \\N}$ converges to $v$ \\cite[Definition~6.1]{RockafellarWets}. The set of all tangent vectors to $C$ at $x \\in C$ is a closed cone \\cite[Proposition~6.2]{RockafellarWets} called the \\emph{tangent cone} to $C$ at $x$ and denoted by $\\tancone{C}{x}$.\nThe \\emph{regular normal cone} to $C$ at $x \\in C$ is\n\\begin{equation*}\n\\regnorcone{C}{x} \\coloneq \\left\\{v \\in \\mathcal{E} \\mid \\ip{v}{w} \\le 0 \\; \\forall w \\in \\tancone{C}{x}\\right\\},\n\\end{equation*}\nwhich is a closed convex cone \\cite[Definition~6.3 and Proposition~6.5]{RockafellarWets}.\nA vector $v \\in \\mathcal{E}$ is said to be \\emph{normal (in the general sense)} to $C$ at $x \\in C$ if there exist sequences $(x_i)_{i \\in \\N}$ in $C$ converging to $x$ and $(v_i)_{i \\in \\N}$ converging to $v$ such that, for all $i \\in \\N$, $v_i \\in \\regnorcone{C}{x_i}$ \\cite[Definition~6.3]{RockafellarWets}. The set of all normal vectors to $C$ at $x \\in C$ is a closed cone \\cite[Proposition~6.5]{RockafellarWets} called the \\emph{normal cone} to $C$ at $x$ and denoted by $\\norcone{C}{x}$.\nA vector $v \\in \\mathcal{E}$ is called a \\emph{proximal normal} to $C$ at $x \\in C$ if there exists $\\oshort{\\alpha} \\in (0, \\infty)$ such that $x \\in \\proj{C}{x+\\oshort{\\alpha}v}$, i.e., $\\oshort{\\alpha}\\norm{v} = \\dist(x+\\oshort{\\alpha}v, C)$, which implies that, for all $\\alpha \\in [0, \\oshort{\\alpha})$, $\\proj{C}{x+\\alpha v} = \\{x\\}$ \\cite[Example~6.16]{RockafellarWets}. The set of all proximal normals to $C$ at $x \\in C$ is a convex cone called the \\emph{proximal normal cone} to $C$ at $x$ and denoted by $\\proxnorcone{C}{x}$.\n","start":20713,"end":22523},{"raw_tex":"\\begin{algorithm}[H]\n\\caption{$\\pgd$ map}\n\\label{algo:PGDmap}\n\\begin{algorithmic}[1]\n\\Require\n$(\\mathcal{E}, C, f, \\ushort{\\alpha}, \\oshort{\\alpha}, \\beta, c)$, where $\\mathcal{E}$ is a Euclidean vector space, $C$ is a nonempty closed subset of~$\\mathcal{E}$, $f : \\mathcal{E} \\to \\R$ satisfies \\ref{it:continuously_diff}, $0 < \\ushort{\\alpha} \\le \\oshort{\\alpha} < \\infty$, and $\\beta, c \\in (0, 1)$.\n\\Input\n$(x, \\mu)$ with $x \\in C$ and $\\mu \\in [f(x), \\infty)$.\n\\Output\n$y \\in \\pgd(x, \\mu; \\mathcal{E}, C, f, \\ushort{\\alpha}, \\oshort{\\alpha}, \\beta, c)$.\n\n\\State\nChoose $\\alpha \\in [\\ushort{\\alpha}, \\oshort{\\alpha}]$ and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$;\n\\label{algo:PGDmap:InitialStepSize}\n\\While\n{$f(y) > \\mu + c \\ip{\\nabla f(x)}{y-x}$}\n\\label{algo:PGDmap:NonmonotoneArmijoCondition}\n\\State\n$\\alpha \\gets \\alpha \\beta$;\n\\State\nChoose $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$;\n\\EndWhile\n\\State\nReturn $y$.\n\\end{algorithmic}\n\\end{algorithm}\n","start":39371,"end":40318},{"raw_tex":"\\begin{remark}\nThe Armijo condition\n\\begin{equation*}\nf(y) \\le \\mu + c \\ip{\\nabla f(x)}{y-x}\n\\end{equation*}\nensures that the decrease $\\mu-f(y)$ is at least a fraction $c$ of the opposite of the directional derivative of $f$ at $x$ along the update vector $y-x$. By \\eqref{eq:ProjectedTranslationIP}, this condition implies that\n\\begin{equation}\\label{eq:KanzowMehlitzDecay}\nf(y) \\le \\mu - \\frac{c}{2\\alpha} \\norm{y-x}^2,\n\\end{equation}\nwhich is the condition used in \\cite[Algorithms~3.1 and 4.1]{KanzowMehlitz} and \\cite[Algorithm~3.1]{DeMarchi}. Importantly, all results from \\cite{KanzowMehlitz} hold for both conditions, as is clear from the proofs.\n% Par contre, les résultats de De Marchi ne sont pas clairement valides avec notre condition (je pense au lemme 4.4).\n\\end{remark}\n","start":40319,"end":41106},{"raw_tex":"The $\\pgd$ algorithm is defined as Algorithm~\\ref{algo:PGD}. It is said to be monotone or nonmonotone depending on whether $\\mu_i=f(x_i)$ for all $i$ (that is, $l=0$ for the ``max'' rule, or $p=1$ for the ``average'' rule) or not.\n\n\\begin{algorithm}[H]\n\\caption{$\\pgd$}\n\\label{algo:PGD}\n\\begin{algorithmic}[1]\n\\Require\n$(\\mathcal{E}, C, f, \\ushort{\\alpha}, \\oshort{\\alpha}, \\beta, c, \\text{``nonmonotonicity''})$, where $\\mathcal{E}$ is a Euclidean vector space, $C$ is a nonempty closed subset of~$\\mathcal{E}$, $f : \\mathcal{E} \\to \\R$ satisfies \\ref{it:continuously_diff}, $0 < \\ushort{\\alpha} \\le \\oshort{\\alpha} < \\infty$, $\\beta, c \\in (0, 1)$, and $\\text{``nonmonotonicity''} \\in \\{(\\text{``max''}, l), (\\text{``average''}, p)\\}$ with $l \\in \\N$ and $p \\in (0, 1]$.\n\\Input\n$x_0 \\in C$.\n\\Output\na sequence in $C$.\n\n\\State\n$i \\gets 0$;\n\\State\n$\\mu_{-1} \\gets f(x_0)$;\n\\While\n{$-\\nabla f(x_i) \\notin \\regnorcone{C}{x_i}$}\n\\label{algo:PGD:while}\n\\If\n{$\\text{``nonmonotonicity''} = (\\text{``max''}, l)$}\n\\State\n$\\mu_i \\gets \\max_{j \\in \\{\\max\\{0, i-l\\}, \\dots, i\\}} f(x_j)$;\n\\ElsIf\n{$\\text{``nonmonotonicity''} = (\\text{``average''}, p)$}\n\\State\n$\\mu_i \\gets (1-p)\\mu_{i-1} + pf(x_i)$; \n\\EndIf\n\\State\nChoose $x_{i+1} \\in \\hyperref[algo:PGDmap]{\\pgd}(x_i, \\mu_i; \\mathcal{E}, C, f, \\ushort{\\alpha}, \\oshort{\\alpha}, \\beta, c)$;\n\\label{algo:PGD:PGDmap}\n\\State\n$i \\gets i+1$;\n\\EndWhile\n\\end{algorithmic}\n\\end{algorithm}\n","start":41697,"end":43116},{"raw_tex":"\\begin{proposition}\n\\label{prop:NonmonotonePGD}\nLet $(x_i)_{i \\in \\N}$ be a sequence generated by $\\pgd$ (Algorithm~\\ref{algo:PGD}) using the ``max'' rule. For every $i \\in \\N$, let $g(i) \\in \\argmax_{j \\in \\{\\max\\{0, i-l\\}, \\dots, i\\}} f(x_j)$. Then:\n\\begin{enumerate}\n\\item $(f(x_{g(i)}))_{i \\in \\N}$ is monotonically nonincreasing;\n\\item $(x_i)_{i \\in \\N}$ is contained in the sublevel set~\\eqref{eq:InitialSublevelSet};\n\\item if $x \\in C$ is an accumulation point of $(x_i)_{i \\in \\N}$, then $(f(x_{g(i)}))_{i \\in \\N}$ converges to $\\varphi \\in [f(x), f(x_0)]$;\n\\item if $f$ is bounded from below and uniformly continuous on a set that contains $(x_i)_{i \\in \\N}$, then $(f(x_i))_{i \\in \\N}$ converges to $\\varphi \\in \\R$.\n\\end{enumerate}\n\\end{proposition}","start":45746,"end":46506}]},"statement":{"raw_tex":"\\begin{theorem}\n\\label{thm:PGDaccumulatesAtBStationaryPoints}\nLet $(x_i)_{i \\in \\N}$ be a sequence generated by $\\pgd$ (Algorithm~\\ref{algo:PGD}), under~\\ref{it:continuously_diff}. Then, all accumulation points of $(x_i)_{i \\in \\N}$ are B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}.\nIf, moreover, $(x_i)_{i \\in \\N}$ has an isolated accumulation point, then $(x_i)_{i \\in \\N}$ converges.\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\n\\label{thm:PGDaccumulatesAtBStationaryPoints}\nLet $(x_i)_{i \\in \\N}$ be a sequence generated by $\\pgd$ (Algorithm~\\ref{algo:PGD}), under~\\ref{it:continuously_diff}. Then, all accumulation points of $(x_i)_{i \\in \\N}$ are B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}.\nIf, moreover, $(x_i)_{i \\in \\N}$ has an isolated accumulation point, then $(x_i)_{i \\in \\N}$ converges.\n\\end{theorem}","start":50907,"end":51315}]},"proof":{"raw_tex":"\\begin{proposition}\n\\label{prop:ProjectedTranslation}\nFor all $x \\in C$, $v \\in \\mathcal{E}$, and $y \\in \\proj{C}{x-v}$,\n\\begin{align}\n\\label{eq:ProjectedTranslationDistance}\n\\norm{y-x} &\\le 2 \\norm{v},\\\\\n\\label{eq:ProjectedTranslationIP}\n2 \\ip{v}{y-x} &\\le - \\norm{y-x}^2,\n\\end{align}\nand the inequalities are strict if $x \\notin \\proj{C}{x-v}$.\n\\end{proposition}\n\n\\begin{proof}\nBy definition of the projection, $\\norm{y-(x-v)} \\le \\norm{x-(x-v)} = \\norm{v}$ and the inequality is strict if $x \\notin \\proj{C}{x-v}$. Thus, on the one hand,\n\\begin{equation*}\n\\norm{y-x}\n= \\norm{y-(x-v)-v}\n\\le \\norm{y-(x-v)} + \\norm{-v}\n\\le \\norm{v} + \\norm{v}\n= 2 \\norm{v},\n\\end{equation*}\nand, on the other hand, $\\norm{y-(x-v)}^2 \\le \\norm{v}^2$, which is equivalent to~\\eqref{eq:ProjectedTranslationIP}.\n\\end{proof}\n\n\\begin{proposition}\n\\label{prop:NonmonotonePGDAverage}\nLet $(x_i)_{i \\in \\N}$ be a sequence generated by $\\pgd$ (Algorithm~\\ref{algo:PGD}) using the ``average'' rule. Then:\n\\begin{enumerate}\n\\item $(x_i)_{i \\in \\N}$ is contained in the sublevel set~\\eqref{eq:InitialSublevelSet};\n\\item if  $(x_i)_{i \\in \\N}$ has an accumulation point, then $(f(x_i))_{i \\in \\N}$ and $(\\mu_i)_{i\\in\\N}$ converge, toward the same (finite) value.\n\\end{enumerate}\n\\end{proposition}\n\n\\begin{proof}\nThe sequence $(\\mu_i)_{i\\in\\N}$ is monotonically nonincreasing since, for all $i \\in \\N$, $f(x_i) \\le \\mu_{i-1}$, hence $\\mu_i=(1-p)\\mu_{i-1}+ pf(x_i) \\le \\mu_{i-1}$. Therefore, for all $i \\in \\N$,\n\\begin{equation*}\nf(x_i) \\le \\mu_{i-1} \\le \\mu_{-1} = f(x_0),\n\\end{equation*}\nmeaning that $(x_i)_{i\\in\\N}$ is contained in the sublevel set~\\eqref{eq:InitialSublevelSet}.\n\nLet us prove the second item of the proposition. Assume that $(x_i)_{i\\in\\N}$ has an accumulation point $x$. Let $(x_{i_k})_{k\\in\\N}$ be a subsequence converging to~$x$. Observe that\n\\begin{equation*}\n\\lim_{k \\to \\infty} f(x_{i_k}) = f(x),\n\\end{equation*}\nsince $f$ is differentiable, and in particular continuous, at $x$.\nAs $(\\mu_i)_{i\\in\\N}$ is monotonically nonincreasing, it has a limit $\\varphi \\in \\R \\cup \\{-\\infty\\}$. For all $k \\in \\N$,\n\\begin{equation*}\nf(x_{i_k}) \\le \\mu_{i_k-1},\n\\end{equation*}\nand letting $k$ tend to infinity yields\n\\begin{equation*}\nf(x) = \\lim_{k\\to\\infty} f(x_{i_k}) \\le \\lim_{k\\to\\infty} \\mu_{i_k-1} = \\varphi.\n\\end{equation*}\nIn particular, $\\varphi$ is finite.\n\nLet us show that $\\varphi = \\liminf_{i\\to\\infty} f(x_i)$. Let $(x_{j_k})_{k\\in\\N}$ be a subsequence such that\n\\begin{equation*}\n\\lim_{k \\to \\infty} f(x_{j_k}) = \\liminf_{i\\to\\infty} f(x_i).\n\\end{equation*}\nFor all $k \\in \\N$,\n\\begin{align*}\n\\mu_{j_k} = (1-p) \\mu_{j_k-1} + p f(x_{j_k}),\n\\end{align*}\nand letting $k$ tend to infinity yields\n\\begin{equation*}\n\\varphi = (1-p) \\varphi + p \\liminf_{i\\to\\infty} f(x_i).\n\\end{equation*}\nAs $p>0$, this implies $\\varphi = \\liminf_{i\\to\\infty} f(x_i)$ (and, in particular, $\\liminf_{i\\to\\infty} f(x_i) > -\\infty$). To conclude, we observe that, for all $k \\in \\N$,\n\\begin{equation*}\nf(x_k) \\le \\mu_{k-1},\n\\end{equation*}\nhence\n\\begin{equation*}\n\\limsup_{k\\to\\infty} f(x_k) \n\\leq \\lim_{k\\to\\infty} \\mu_{k-1} = \\varphi = \\liminf_{k\\to\\infty} f(x_k).\n\\end{equation*}\nTherefore, $(f(x_k))_{k\\in\\N}$ converges to $\\varphi$.\n\\end{proof}\n\n\\begin{proposition}\n\\label{prop:ArbitrarySmallAlpha}\nAssume that $f$ satisfies~\\ref{it:continuously_diff}. Let $\\ushort{x} \\in C$ be non-B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}, and $w \\in \\tancone{C}{\\ushort{x}}$ be such that\n\\begin{equation}\n\\label{eq:def_w}\n\\ip{w}{-\\nabla f(\\ushort{x})} > 0.\n\\end{equation}\nLet $\\beta \\in (0, 1)$. Define $\\kappa \\coloneq \\sqrt{1 - \\frac{\\beta \\ip{w}{-\\nabla f(\\ushort{x})}^2}{8 \\norm{w}^2 \\norm{\\nabla f(\\ushort{x})}^2}} \\in (0, 1)$.\nFor every $\\varepsilon \\in (0, \\infty)$, there exist $\\alpha_{\\ushort{x}} \\in (0, \\varepsilon]$ and $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\infty)$ such that, for all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$ and $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$,\n\\begin{equation*}\n\\dist(x-\\alpha\\nabla f(x), C) \\le \\kappa \\alpha \\norm{\\nabla f(x)},\n\\end{equation*}\nwhich implies, for all $y \\in\\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{equation*}\n\\ip{\\nabla f(x)}{y-x} \\le - \\sqrt{1-\\kappa^2} \\norm{\\nabla f(x)} \\norm{y-x}.\n\\end{equation*}\n\\end{proposition}\n\n\\begin{proof}\nLet $\\varepsilon \\in (0, \\infty)$ be fixed. We show that there exist $\\alpha_{\\ushort{x}} \\in (0, \\varepsilon]$ and $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\infty)$ satisfying the required property.\n\nLet $(w_i)_{i\\in\\N}$ be a sequence in $C$ converging to $\\ushort{x}$, and $(t_i)_{i\\in\\N}$ be a sequence in $(0, \\infty)$ such that\n\\begin{equation*}\n\\frac{w_i - \\ushort{x}}{t_i} \\xrightarrow{i\\to\\infty} w.\n\\end{equation*}\nFrom the definition of $w$ in~\\eqref{eq:def_w}, it holds for all $i \\in \\N$ large enough that\n\\begin{equation}\n\\label{eq:def_i_1}\n\\ip{w_i - \\ushort{x}}{-\\nabla f(\\ushort{x})} > 0.\n\\end{equation}\nAs $\\frac{1}{t_i}\\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}} \\xrightarrow{i\\to\\infty} \\frac{\\norm{w}^2}{\\ip{w}{-\\nabla f(\\ushort{x})}}$ and $t_i \\xrightarrow{i\\to\\infty} 0$, it also holds for all $i \\in \\N$ large enough that\n\\begin{equation}\n\\label{eq:def_i_2}\n\\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}} < \\varepsilon.\n\\end{equation}\nSimilarly, it holds for all $i \\in \\N$ large enough that\n\\begin{equation}\n\\label{eq:def_i_3}\n\\frac{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}^2}{\\norm{w_i-\\ushort{x}}^2}\n> \\frac{\\ip{w}{-\\nabla f(\\ushort{x})}^2}{2\\norm{w}^2 }.\n\\end{equation}\nFix $i \\in \\N$ satisfying \\eqref{eq:def_i_1}, \\eqref{eq:def_i_2}, and \\eqref{eq:def_i_3}.\nPick $\\alpha_{\\ushort{x}}$ such that\n\\begin{equation*}\n\\frac{\\alpha_{\\ushort{x}}}{2}\n< \\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}}\n< \\alpha_{\\ushort{x}}\n< \\varepsilon.\n\\end{equation*}\nSince $\\nabla f$ is continuous at $\\ushort{x}$, there exists $\\rho_0 \\in (0, \\infty)$ such that, for all $x \\in \\ball[\\ushort{x}, \\rho_0] \\cap C$,\n\\begin{subequations}\n\\begin{gather}\n\\label{eq:def_rho0_1}\n\\ip{w_i-\\ushort{x}}{-\\nabla f(x)} > 0,\\\\\n\\label{eq:def_rho0_2}\n\\frac{\\alpha_{\\ushort{x}}}{2}\n< \\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(x)}}\n< \\alpha_{\\ushort{x}},\\\\\n\\label{eq:def_rho0_3}\n\\frac{\\ip{w_i-\\ushort{x}}{-\\nabla f(x)}^2}{\\norm{w_i-\\ushort{x}}^2 \\norm{\\nabla f(x)}^2}\n> \\frac{\\ip{w}{-\\nabla f(\\ushort{x})}^2}{2\\norm{w}^2 \\norm{\\nabla f(\\ushort{x})}^2}.\n\\end{gather}\n\\end{subequations}\nWe now establish the first inequality we have to prove: for an adequate value of $\\oshort{\\rho}(\\alpha_{\\ushort{x}})$, it holds for all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$ and $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$ that\n\\begin{equation*}\n\\norm{x - \\alpha \\nabla f(x) - y} \\le \\kappa \\alpha \\norm{\\nabla f(x)}\\quad\n\\forall y \\in \\proj{C}{x-\\alpha\\nabla f(x)},\n\\end{equation*}\nwhich is equivalent to $\\dist(x-\\alpha\\nabla f(x), C) \\le \\kappa \\alpha \\norm{\\nabla f(x)}$.\n\nFor the moment, let $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\rho_0]$ be arbitrary. For all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{align*}\n\\norm{x - \\alpha \\nabla f(x) - y}^2\n&\\le \\norm{x - \\alpha \\nabla f(x) - w_i}^2 \\\\\n&= \\norm{\\ushort{x}-\\alpha \\nabla f(x) - w_i}^2 + 2 \\ip{\\ushort{x}-x}{\\alpha \\nabla f(x) + w_i-\\ushort{x}} + \\norm{\\ushort{x}-x}^2 \\\\\n&\\le \\norm{\\ushort{x}-\\alpha \\nabla f(x) - w_i}^2 \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\alpha \\norm{\\nabla f(x)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&\\le \\norm{\\ushort{x}-\\alpha \\nabla f(x) - w_i}^2 \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\alpha \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&= \\alpha^2 \\norm{\\nabla f(x)}^2 -2 \\alpha \\ip{w_i-\\ushort{x}}{-\\nabla f(x)} + \\norm{w_i-\\ushort{x}}^2 \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\alpha \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 - \\alpha \\ip{w_i-\\ushort{x}}{-\\nabla f(x)} \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\frac{\\alpha_{\\ushort{x}}}{\\beta} \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right)\n+ \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2,\n\\end{align*}\nwhere the last inequality follows from \\eqref{eq:def_rho0_2} and the fact that $\\alpha_{\\ushort{x}} \\le \\alpha \\le \\frac{\\alpha_{\\ushort{x}}}{\\beta}$.\nNow, choose $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\rho_0]$ small enough to ensure\n\\begin{align*}\n&2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\frac{\\alpha_{\\ushort{x}}}{\\beta} \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&\\le \\frac{\\alpha_{\\ushort{x}}}{2} \\min_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\ip{w_i-\\ushort{x}}{-\\nabla f(z)}.\n\\end{align*}\nNote that the right-hand side of this inequality is positive, from \\eqref{eq:def_rho0_1}.\nCombining this definition with the previous inequality, we arrive at\n\\begin{align*}\n\\norm{x - \\alpha \\nabla f(x) - y}^2\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 - \\frac{\\alpha}{2} \\ip{w_i-\\ushort{x}}{-\\nabla f(x)}\\\\\n&= \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\ip{w_i-\\ushort{x}}{-\\nabla f(x)}}{2 \\alpha \\norm{\\nabla f(x)}^2}\\right)\\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\beta \\ip{w_i-\\ushort{x}}{-\\nabla f(x)}}{2 \\alpha_{\\ushort{x}} \\norm{\\nabla f(x)}^2}\\right)\n\\text{ as $\\alpha \\le \\frac{\\alpha_{\\ushort{x}}}{\\beta}$}\\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\beta \\ip{w_i-\\ushort{x}}{-\\nabla f(x)}^2}{4 \\norm{w_i-\\ushort{x}}^2 \\norm{\\nabla f(x)}^2}\\right)\n\\text{ from~\\eqref{eq:def_rho0_2}}\\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\beta \\ip{w}{-\\nabla f(\\ushort{x})}^2}{8 \\norm{w}^2 \\norm{\\nabla f(\\ushort{x})}^2}\\right)\n\\text{ from~\\eqref{eq:def_rho0_3}}\\\\\n&= \\kappa^2 \\alpha^2 \\norm{\\nabla f(x)}^2.\n\\end{align*}\nThus, for all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{equation*}\n\\norm{x - \\alpha \\nabla f(x) - y} \\le \\kappa \\alpha \\norm{\\nabla f(x)}.\n\\end{equation*}\nTo conclude, we show that this inequality implies\n\\begin{equation}\n\\label{eq:ip_xy_nabla}\n\\ip{\\frac{y-x}{\\norm{y-x}}}{\\frac{\\nabla f(x)}{\\norm{\\nabla f(x)}}} \\le - \\sqrt{1-\\kappa^2}.\n\\end{equation}\nIndeed, defining $\\theta \\in \\R$ such that $\\ip{\\frac{y-x}{\\norm{y-x}}}{\\frac{\\nabla f(x)}{\\norm{\\nabla f(x)}}} = \\cos(\\theta)$, we have\n\\begin{equation*}\n\\norm{y-x}^2 + 2 \\alpha \\norm{\\nabla f(x)} \\norm{y-x} \\cos(\\theta) + \\alpha^2 \\norm{\\nabla f(x)}^2\n\\le \\alpha^2 \\kappa^2 \\norm{\\nabla f(x)}^2.\n\\end{equation*}\nThis already shows that $\\cos(\\theta) < 0$. In addition, minimizing the left-hand side over all possible values of $\\norm{y-x}$, we get\n\\begin{equation*}\n- \\alpha^2 \\norm{\\nabla f(x)}^2 \\cos^2(\\theta) + \\alpha^2 \\norm{\\nabla f(x)}^2\n\\le \\alpha^2 \\kappa^2 \\norm{\\nabla f(x)}^2,\n\\end{equation*}\nhence $\\cos^2(\\theta) \\ge 1 - \\kappa^2$, which establishes \\eqref{eq:ip_xy_nabla}.\n\\end{proof}\n\n\\begin{proposition}\n\\label{prop:AcceptedStep}\nLet $\\ushort{\\alpha} \\in (0, \\infty)$ and $\\beta, c \\in (0, 1)$.\nAssume that $f$ satisfies~\\ref{it:continuously_diff}. Let $\\ushort{x} \\in C$ be non-B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}.\nThere exist $\\alpha_{\\ushort{x}} \\in (0, \\ushort{\\alpha}]$ and $\\rho \\in (0, \\infty)$ such that, for all $x \\in \\ball(\\ushort{x}, \\rho) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{equation*}\nf(y) < f(x) + c \\ip{\\nabla f(x)}{y-x}.\n\\end{equation*}\n\\end{proposition}\n\n\\begin{proof}\nDefine $\\kappa$ as in Proposition~\\ref{prop:ArbitrarySmallAlpha}. Let $\\delta \\in (0, \\infty)$ be small enough to ensure\n\\begin{subequations}\n\\begin{align}\n\\label{eq:def_alpha_x_small1}\n\\sup_{y \\in \\ball\\left[\\ushort{x}, \\frac{7\\delta}{2\\beta}\\norm{\\nabla f(\\ushort{x})}\\right] \\cap C \\setminus \\{\\ushort{x}\\}}\n\\frac{|f(y)-f(\\ushort{x}) - \\ip{\\nabla f(\\ushort{x})}{y-\\ushort{x}}|}{\\norm{y-\\ushort{x}}}\n&< \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4\\left(1 + \\frac{8}{3(1-\\kappa)}\\right)},\\\\\n\\label{eq:def_alpha_x_small2}\n\\sup_{y \\in \\ball\\left[\\ushort{x}, \\frac{7\\delta}{2\\beta}\\norm{\\nabla f(\\ushort{x})}\\right] \\cap C}\n\\norm{\\nabla f(y)-\\nabla f(\\ushort{x})}\n&< \\frac{(1-c)\\sqrt{1-\\kappa^2}}{4} \\norm{\\nabla f(\\ushort{x})}.\n\\end{align}\n\\end{subequations}\nThese inequalities are satisfied by all $\\delta$ small enough, from the definition of the gradient for the first one, and because the gradient is continuous at $\\ushort{x}$ for the second one.\n\nThen, define $\\varepsilon \\coloneq \\min\\left\\{\\ushort{\\alpha}, \\delta\\right\\}$ and let $\\alpha_{\\ushort{x}} \\in (0, \\varepsilon]$ and $\\oshort{\\rho}(\\alpha_{\\ushort{x}})\\in(0,\\infty)$ be given by Proposition~\\ref{prop:ArbitrarySmallAlpha}.\nDefine\n\\begin{equation*}\n\\rho \\coloneq \\min\\left\\{\\oshort{\\rho}(\\alpha_{\\ushort{x}}), \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})}\\right\\}.\n\\end{equation*}\nNote that, for all $x \\in \\ball(\\ushort{x}, \\rho) \\cap C$,\n\\begin{equation*}\n\\norm{x-\\ushort{x}}\n< \\rho \\le \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})}\n< \\frac{7\\alpha_{\\ushort{x}}}{2\\beta} \\norm{\\nabla f(\\ushort{x})}\n\\leq \\frac{7\\delta }{2\\beta} \\norm{\\nabla f(\\ushort{x})},\n\\end{equation*}\nso that from~\\eqref{eq:def_alpha_x_small2}, $\\norm{\\nabla f(x)-\\nabla f(\\ushort{x})} < \\frac{\\norm{\\nabla f(\\ushort{x})}}{4}$, which implies\n\\begin{align}\n\\nonumber\n\\frac{3}{4} \\norm{\\nabla f(\\ushort{x})}\n& < \\norm{\\nabla f(\\ushort{x})}-\\norm{\\nabla f(x)-\\nabla f(\\ushort{x})} \\\\\n& \\le \\norm{\\nabla f(x)} \\nonumber\\\\\n& \\le \\norm{\\nabla f(\\ushort{x})}+\\norm{\\nabla f(x)-\\nabla f(\\ushort{x})} \\nonumber\\\\\n& < \\frac{5}{4} \\norm{\\nabla f(\\ushort{x})}.\n\\label{eq:nablas_ushortx_x}\n\\end{align}\nFor all $x \\in \\ball(\\ushort{x}, \\rho) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{align}\nf(y)\n&= f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x} \\nonumber\\\\\n&\\qquad + \\left(f(\\ushort{x}) - f(x) - \\ip{\\nabla f(\\ushort{x})}{\\ushort{x}-x}\\right) \\nonumber\\\\\n&\\qquad + \\left(f(y) - f(\\ushort{x}) - \\ip{\\nabla f(\\ushort{x})}{y-\\ushort{x}}\\right) \\nonumber\\\\\n&\\le f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x} +\n\\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4\\left(1 + \\frac{8}{3(1-\\kappa)}\\right)} \\left(\\norm{\\ushort{x}-x} + \\norm{y-\\ushort{x}}\\right),\n\\label{eq:UpperBoundfDiffGrad}\n\\end{align}\nwhere the inequality follows from \\eqref{eq:def_alpha_x_small1}. Observe that\n\\begin{align*}\n\\norm{y-\\ushort{x}}\n&\\le \\norm{y-x} + \\norm{x-\\ushort{x}} \\\\\n&\\le 2 \\alpha \\norm{\\nabla f(x)} + \\rho\n\\text{ from \\eqref{eq:ProjectedTranslationDistance}}\\\\\n&\\le \\frac{2 \\alpha_{\\ushort{x}}}{\\beta} \\norm{\\nabla f(x)}\n+ \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})} \\\\\n&< \\frac{5\\alpha_{\\ushort{x}}}{2\\beta} \\norm{\\nabla f(\\ushort{x})}\n+ \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})} \n\\text{ from \\eqref{eq:nablas_ushortx_x}} \\\\\n&\\le \\frac{7\\alpha_{\\ushort{x}}}{2\\beta} \\norm{\\nabla f(\\ushort{x})}.\n\\end{align*}\nWe continue from \\eqref{eq:UpperBoundfDiffGrad}:\n\\begin{align*}\nf(y)\n& \\le f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x} +\n\\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{\n4\\left(1 + \\frac{8}{3(1-\\kappa)}\\right)} \\left(2\\norm{\\ushort{x}-x} + \\norm{y-x}\\right) \\\\\n&\\overset{\\text{(a)}}{<} f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x}\n+ \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4} \\norm{y-x} \\\\\n&\\le f(x) + \\ip{\\nabla f(x)}{y-x} + \\norm{\\nabla f(\\ushort{x})-\\nabla f(x)} \\norm{y-x} \\\\\n&\\qquad + \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4} \\norm{y-x} \\\\\n&\\le f(x) + \\ip{\\nabla f(x)}{y-x} + \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{2} \\norm{y-x}\n\\text{ from~\\eqref{eq:def_alpha_x_small2}}\\\\\n&< f(x) + \\ip{\\nabla f(x)}{y-x} + (1-c)\\sqrt{1-\\kappa^2} \\norm{\\nabla f(x)}\\norm{y-x}\n\\text{ from~\\eqref{eq:nablas_ushortx_x}}\\\\\n&\\le f(x) + \\ip{\\nabla f(x)}{y-x} - (1-c) \\ip{\\nabla f(x)}{y-x}\n\\text{ from Proposition~\\ref{prop:ArbitrarySmallAlpha}}\\\\\n&= f(x) + c \\ip{\\nabla f(x)}{y-x}.\n\\end{align*}\nInequality (a) is true because\n\\begin{align*}\n\\norm{y-x}\n&\\ge \\alpha \\norm{\\nabla f(x)} - \\norm{x-\\alpha \\nabla f(x)-y}\n\\text{ by the triangle inequality}\\\\\n&= \\alpha \\norm{\\nabla f(x)} - \\dist(x-\\alpha\\nabla f(x), C) \\\\\n&\\ge (1-\\kappa) \\alpha \\norm{\\nabla f(x)}\n\\text{ from Proposition~\\ref{prop:ArbitrarySmallAlpha}}\\\\\n&\\ge \\frac{3}{4} (1-\\kappa) \\rho\n\\text{ from~\\eqref{eq:nablas_ushortx_x}, the definition of $\\rho$, and $\\alpha_{\\ushort{x}} \\le \\alpha$}\\\\\n&> \\frac{3}{4} (1-\\kappa) \\norm{x-\\ushort{x}}.\n\\end{align*}\n\\end{proof}\n\n\\begin{proposition}\n\\label{prop:convergence_ik}\nLet $(x_i)_{i \\in \\N}$ be a sequence generated by $\\pgd$ (Algorithm~\\ref{algo:PGD}), under~\\ref{it:continuously_diff}, and $x$ be an accumulation point.\nThen, for every subsequence $(x_{i_k})_{k\\in\\N}$ converging to $x$, the sequence $(x_{i_k+1})_{k\\in\\N}$ also converges to $x$.\n\\end{proposition}\n\n\\begin{proof}\nLet $(x_{i_k})_{k\\in\\N}$ be a subsequence converging to $x$. We show that $(x_{i_k+1})_{k\\in\\N}$ also converges to $x$.\n\nIf the nonmonotonicity rule is set to ``average'', this is a direct consequence of Proposition~\\ref{prop:NonmonotonePGDAverage}. Indeed, for all $i \\in \\N$, from~\\eqref{eq:KanzowMehlitzDecay},\n\\begin{equation*}\nf(x_{i+1}) \\le \\mu_i - \\frac{c}{2 \\oshort{\\alpha}} \\norm{x_{i+1}-x_i}^2 \\le \\mu_i.    \n\\end{equation*}\nFrom Proposition~\\ref{prop:NonmonotonePGDAverage}, $(f(x_{i+1}))_{i\\in\\N}$ and $(\\mu_i)_{i\\in\\N}$ converge to the same limit. Therefore,\n\\begin{equation*}\n \\left(\\mu_i - \\frac{c}{2 \\oshort{\\alpha}} \\norm{x_{i+1}-x_i}^2\\right)_{i\\in\\N}\n\\end{equation*}\nalso converges to this limit.\nThis implies that $(\\norm{x_{i+1}-x_i})_{i\\in\\N}$ converges to $0$, hence $(\\norm{x_{i_k+1}-x_{i_k}})_{k\\in\\N}$ converges to $0$, and $(x_{i_k+1})_{k\\in\\N}$ converges to the same limit as $(x_{i_k})_{k\\in\\N}$, that is, $x$.\n\nNow, let us consider the ``max'' rule case. It suffices to show that $x$ is an accumulation point of every subsequence of $(x_{i_k+1})_{k\\in\\N}$. Thus, we show the following: for every subsequence $(i_{j_k})_{k\\in\\N}$ of $(i_k)_{k\\in\\N}$, there exists a subsequence of $(x_{i_{j_k}+1})_{k\\in\\N}$ that converges to $x$.\nLet $(i_{j_k})_{k\\in\\N}$ be a subsequence of $(i_k)_{k\\in\\N}$.\nFor all $i \\in \\N$, let $g(i) \\in \\argmax_{j \\in \\{\\max\\{0, i-l\\}, \\dots, i\\}} f(x_j)$, as in Proposition~\\ref{prop:NonmonotonePGD}. By the third statement of Proposition~\\ref{prop:NonmonotonePGD}, the sequence $(f(x_{g(i)}))_{i\\in\\N}$ converges to $\\varphi \\in [f(x), f(x_0)]$.\nFor every $k \\in \\N$, letting $\\alpha_{i_{j_k}} \\in (0, \\oshort{\\alpha}]$ be the number such that $x_{i_{j_k}+1} \\in \\proj{C}{x_{i_{j_k}} - \\alpha_{i_{j_k}} \\nabla f(x_{i_{j_k}})}$, by \\eqref{eq:ProjectedTranslationDistance},\n\\begin{equation*}\n\\norm{x_{i_{j_k}+1}-x_{i_{j_k}}}\n\\le 2 \\alpha_{i_{j_k}} \\norm{\\nabla f(x_{i_{j_k}})}\n\\le 2 \\oshort{\\alpha} \\norm{\\nabla f(x_{i_{j_k}})}.\n\\end{equation*}\nThus, since $(x_{i_{j_k}})_{k\\in\\N}$ is bounded and $\\nabla f$ is locally bounded (as it is continuous), the sequence $(x_{i_{j_k}+1})_{k\\in\\N}$ is bounded. If we replace $(i_{j_k})_{k\\in\\N}$ with a subsequence, we can assume that $(x_{i_{j_k}+1})_{k\\in\\N}$ converges.\n\nIterating the reasoning, we can assume that $(x_{i_{j_k}+s})_{k\\in\\N}$ converges to some $x^{s}\\in C$ for every $s \\in \\{0, \\dots, l+1\\}$. By definition of $x$, $x^{0}=x$.\n\nObserve that, from the continuity of $f$,\n\\begin{align*}\nf(x_{g(i_{j_k}+l+1)})\n&= \\max\\{f(x_{i_{j_k}+1}), \\dots, f(x_{i_{j_k}+l+1})\\} \\\\\n&\\to \\max\\{f(x^{1}), \\dots, f(x^{l+1})\\}\n\\text{ when } k \\to \\infty.\n\\end{align*}\nIn particular, there exists $s_1 \\in \\{1, \\dots, l+1\\}$ such that\n\\begin{equation}\n\\label{eq:def_s_1}\nf(x^{s_1}) = \\varphi.\n\\end{equation}\nLet $s_1$ be the smallest such integer.\nFor all $k \\in \\N$, from the condition in line~\\ref{algo:PGDmap:NonmonotoneArmijoCondition} of Algorithm~\\ref{algo:PGDmap} and~\\eqref{eq:KanzowMehlitzDecay},\n\\begin{align*}\nf(x_{i_{j_k}+s_1})\n&\\le f(x_{g(i_{j_k}+s_1-1)}) - \\frac{c}{2\\oshort{\\alpha}} \\norm{x_{i_{j_k}+s_1}-x_{i_{j_k}+s_1-1}}^2,\n\\end{align*}\nand letting $k$ tend to infinity yields\n\\begin{equation*}\n\\varphi\n= f(x^{s_1})\n\\le \\varphi - \\frac{c}{2\\oshort{\\alpha}} \\norm{x^{s_1}-x^{s_1-1}}^2.\n\\end{equation*}\nConsequently, $x^{s_1} = x^{s_1-1}$. In particular, $f(x^{s_1-1}) =f(x^{s_1}) = \\varphi$. Therefore, $s_1=1$, otherwise it would not be the smallest integer satisfying \\eqref{eq:def_s_1}. The equality $x^{s_1} = x^{s_1-1}$ then reduces to $x^{1}=x^{0}=x$ and, when $k \\to \\infty$,\n\\begin{equation*}\nx_{i_{j_k}+1} \\to x^{1} = x.\n\\end{equation*}\n\\end{proof}\n\n\\subsection{Third part: proof of Theorem~\\ref{thm:PGDaccumulatesAtBStationaryPoints}}\n\\label{subsec:proof_accumulates}\nLet $\\ushort{x}$ be an accumulation point of $(x_i)_{i\\in\\N}$. Assume, for the sake of contradiction, that $\\ushort{x}$ is not B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}. Let $(x_{i_k})_{k\\in\\N}$ be a subsequence converging to $\\ushort{x}$.\n\nLet $\\alpha_{\\ushort{x}}$ and $\\rho$ be as in Proposition~\\ref{prop:AcceptedStep}. For all $k \\in \\N$ large enough, $x_{i_k} \\! \\in \\ball(\\ushort{x}, \\rho) \\cap C$. Thus, when Algorithm~\\ref{algo:PGDmap} is called at point $x_{i_k}$, the condition in line~\\ref{algo:PGDmap:NonmonotoneArmijoCondition} stops being fulfilled for some $\\alpha_{i_k} \\ge \\alpha_{\\ushort{x}}$, meaning that\n\\begin{equation*}\nx_{i_k+1} \\in \\proj{C}{x_{i_k}-\\alpha_{i_k} \\nabla f(x_{i_k})}\n\\text{ for some } \\alpha_{i_k} \\in [\\alpha_{\\ushort{x}}, \\oshort{\\alpha}].\n\\end{equation*}\nReplacing $(i_k)_{k\\in\\N}$ with a subsequence if necessary, we can assume that $(\\alpha_{i_k})_{k\\in\\N}$ converges to some $\\alpha_{\\lim} \\in [\\alpha_{\\ushort{x}}, \\oshort{\\alpha}]$.\n\nFor all $k \\in \\N$, we have\n\\begin{equation*}\n\\norm{x_{i_k} - \\alpha_{i_k} \\nabla f(x_{i_k}) - x_{i_k+1}}\n= \\dist(x_{i_k} - \\alpha_{i_k} \\nabla f(x_{i_k}), C),\n\\end{equation*}\nand since the distance to a nonempty closed set is a continuous function, we can take this equality to the limit. We use the fact that $x_{i_k+1} \\to \\ushort{x}$ when $k \\to \\infty$, from Proposition~\\ref{prop:convergence_ik}. This yields\n\\begin{equation*}\n\\norm{\\alpha_{\\lim} \\nabla f(\\ushort{x})}\n= \\dist(\\ushort{x} - \\alpha_{\\lim}\\nabla f(\\ushort{x}), C),\n\\end{equation*}\nwhich means that $\\ushort{x} \\in \\proj{C}{\\ushort{x} - \\alpha_{\\lim}\\nabla f(\\ushort{x})}$. In particular, $-\\nabla f(\\ushort{x}) \\in \\proxnorcone{C}{\\ushort{x}} \\subseteq \\regnorcone{C}{\\ushort{x}}$, which contradicts our assumption that $\\ushort{x}$ is not B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}. We have therefore proven that every accumulation point is B-stationary.\n\nFinally, if $(x_i)_{i\\in\\N}$ has an isolated accumulation point, then the sequence $(x_i)_{i\\in\\N}$ converges, from Proposition~\\ref{prop:convergence_ik} and \\cite[Lemma~4.10]{MoreSorensen}.\n\n","units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:ProjectedTranslation}\nFor all $x \\in C$, $v \\in \\mathcal{E}$, and $y \\in \\proj{C}{x-v}$,\n\\begin{align}\n\\label{eq:ProjectedTranslationDistance}\n\\norm{y-x} &\\le 2 \\norm{v},\\\\\n\\label{eq:ProjectedTranslationIP}\n2 \\ip{v}{y-x} &\\le - \\norm{y-x}^2,\n\\end{align}\nand the inequalities are strict if $x \\notin \\proj{C}{x-v}$.\n\\end{proposition}","start":19632,"end":19996},{"raw_tex":"\\begin{proof}\nBy definition of the projection, $\\norm{y-(x-v)} \\le \\norm{x-(x-v)} = \\norm{v}$ and the inequality is strict if $x \\notin \\proj{C}{x-v}$. Thus, on the one hand,\n\\begin{equation*}\n\\norm{y-x}\n= \\norm{y-(x-v)-v}\n\\le \\norm{y-(x-v)} + \\norm{-v}\n\\le \\norm{v} + \\norm{v}\n= 2 \\norm{v},\n\\end{equation*}\nand, on the other hand, $\\norm{y-(x-v)}^2 \\le \\norm{v}^2$, which is equivalent to~\\eqref{eq:ProjectedTranslationIP}.\n\\end{proof}","start":19998,"end":20434},{"raw_tex":"\\begin{proposition}\n\\label{prop:NonmonotonePGDAverage}\nLet $(x_i)_{i \\in \\N}$ be a sequence generated by $\\pgd$ (Algorithm~\\ref{algo:PGD}) using the ``average'' rule. Then:\n\\begin{enumerate}\n\\item $(x_i)_{i \\in \\N}$ is contained in the sublevel set~\\eqref{eq:InitialSublevelSet};\n\\item if  $(x_i)_{i \\in \\N}$ has an accumulation point, then $(f(x_i))_{i \\in \\N}$ and $(\\mu_i)_{i\\in\\N}$ converge, toward the same (finite) value.\n\\end{enumerate}\n\\end{proposition}","start":47930,"end":48391},{"raw_tex":"\\begin{proof}\nThe sequence $(\\mu_i)_{i\\in\\N}$ is monotonically nonincreasing since, for all $i \\in \\N$, $f(x_i) \\le \\mu_{i-1}$, hence $\\mu_i=(1-p)\\mu_{i-1}+ pf(x_i) \\le \\mu_{i-1}$. Therefore, for all $i \\in \\N$,\n\\begin{equation*}\nf(x_i) \\le \\mu_{i-1} \\le \\mu_{-1} = f(x_0),\n\\end{equation*}\nmeaning that $(x_i)_{i\\in\\N}$ is contained in the sublevel set~\\eqref{eq:InitialSublevelSet}.\n\nLet us prove the second item of the proposition. Assume that $(x_i)_{i\\in\\N}$ has an accumulation point $x$. Let $(x_{i_k})_{k\\in\\N}$ be a subsequence converging to~$x$. Observe that\n\\begin{equation*}\n\\lim_{k \\to \\infty} f(x_{i_k}) = f(x),\n\\end{equation*}\nsince $f$ is differentiable, and in particular continuous, at $x$.\nAs $(\\mu_i)_{i\\in\\N}$ is monotonically nonincreasing, it has a limit $\\varphi \\in \\R \\cup \\{-\\infty\\}$. For all $k \\in \\N$,\n\\begin{equation*}\nf(x_{i_k}) \\le \\mu_{i_k-1},\n\\end{equation*}\nand letting $k$ tend to infinity yields\n\\begin{equation*}\nf(x) = \\lim_{k\\to\\infty} f(x_{i_k}) \\le \\lim_{k\\to\\infty} \\mu_{i_k-1} = \\varphi.\n\\end{equation*}\nIn particular, $\\varphi$ is finite.\n\nLet us show that $\\varphi = \\liminf_{i\\to\\infty} f(x_i)$. Let $(x_{j_k})_{k\\in\\N}$ be a subsequence such that\n\\begin{equation*}\n\\lim_{k \\to \\infty} f(x_{j_k}) = \\liminf_{i\\to\\infty} f(x_i).\n\\end{equation*}\nFor all $k \\in \\N$,\n\\begin{align*}\n\\mu_{j_k} = (1-p) \\mu_{j_k-1} + p f(x_{j_k}),\n\\end{align*}\nand letting $k$ tend to infinity yields\n\\begin{equation*}\n\\varphi = (1-p) \\varphi + p \\liminf_{i\\to\\infty} f(x_i).\n\\end{equation*}\nAs $p>0$, this implies $\\varphi = \\liminf_{i\\to\\infty} f(x_i)$ (and, in particular, $\\liminf_{i\\to\\infty} f(x_i) > -\\infty$). To conclude, we observe that, for all $k \\in \\N$,\n\\begin{equation*}\nf(x_k) \\le \\mu_{k-1},\n\\end{equation*}\nhence\n\\begin{equation*}\n\\limsup_{k\\to\\infty} f(x_k) \n\\leq \\lim_{k\\to\\infty} \\mu_{k-1} = \\varphi = \\liminf_{k\\to\\infty} f(x_k).\n\\end{equation*}\nTherefore, $(f(x_k))_{k\\in\\N}$ converges to $\\varphi$.\n\\end{proof}","start":48393,"end":50351},{"raw_tex":"\\begin{proposition}\n\\label{prop:ArbitrarySmallAlpha}\nAssume that $f$ satisfies~\\ref{it:continuously_diff}. Let $\\ushort{x} \\in C$ be non-B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}, and $w \\in \\tancone{C}{\\ushort{x}}$ be such that\n\\begin{equation}\n\\label{eq:def_w}\n\\ip{w}{-\\nabla f(\\ushort{x})} > 0.\n\\end{equation}\nLet $\\beta \\in (0, 1)$. Define $\\kappa \\coloneq \\sqrt{1 - \\frac{\\beta \\ip{w}{-\\nabla f(\\ushort{x})}^2}{8 \\norm{w}^2 \\norm{\\nabla f(\\ushort{x})}^2}} \\in (0, 1)$.\nFor every $\\varepsilon \\in (0, \\infty)$, there exist $\\alpha_{\\ushort{x}} \\in (0, \\varepsilon]$ and $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\infty)$ such that, for all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$ and $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$,\n\\begin{equation*}\n\\dist(x-\\alpha\\nabla f(x), C) \\le \\kappa \\alpha \\norm{\\nabla f(x)},\n\\end{equation*}\nwhich implies, for all $y \\in\\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{equation*}\n\\ip{\\nabla f(x)}{y-x} \\le - \\sqrt{1-\\kappa^2} \\norm{\\nabla f(x)} \\norm{y-x}.\n\\end{equation*}\n\\end{proposition}","start":53427,"end":54511},{"raw_tex":"\\begin{proof}\nLet $\\varepsilon \\in (0, \\infty)$ be fixed. We show that there exist $\\alpha_{\\ushort{x}} \\in (0, \\varepsilon]$ and $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\infty)$ satisfying the required property.\n\nLet $(w_i)_{i\\in\\N}$ be a sequence in $C$ converging to $\\ushort{x}$, and $(t_i)_{i\\in\\N}$ be a sequence in $(0, \\infty)$ such that\n\\begin{equation*}\n\\frac{w_i - \\ushort{x}}{t_i} \\xrightarrow{i\\to\\infty} w.\n\\end{equation*}\nFrom the definition of $w$ in~\\eqref{eq:def_w}, it holds for all $i \\in \\N$ large enough that\n\\begin{equation}\n\\label{eq:def_i_1}\n\\ip{w_i - \\ushort{x}}{-\\nabla f(\\ushort{x})} > 0.\n\\end{equation}\nAs $\\frac{1}{t_i}\\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}} \\xrightarrow{i\\to\\infty} \\frac{\\norm{w}^2}{\\ip{w}{-\\nabla f(\\ushort{x})}}$ and $t_i \\xrightarrow{i\\to\\infty} 0$, it also holds for all $i \\in \\N$ large enough that\n\\begin{equation}\n\\label{eq:def_i_2}\n\\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}} < \\varepsilon.\n\\end{equation}\nSimilarly, it holds for all $i \\in \\N$ large enough that\n\\begin{equation}\n\\label{eq:def_i_3}\n\\frac{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}^2}{\\norm{w_i-\\ushort{x}}^2}\n> \\frac{\\ip{w}{-\\nabla f(\\ushort{x})}^2}{2\\norm{w}^2 }.\n\\end{equation}\nFix $i \\in \\N$ satisfying \\eqref{eq:def_i_1}, \\eqref{eq:def_i_2}, and \\eqref{eq:def_i_3}.\nPick $\\alpha_{\\ushort{x}}$ such that\n\\begin{equation*}\n\\frac{\\alpha_{\\ushort{x}}}{2}\n< \\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}}\n< \\alpha_{\\ushort{x}}\n< \\varepsilon.\n\\end{equation*}\nSince $\\nabla f$ is continuous at $\\ushort{x}$, there exists $\\rho_0 \\in (0, \\infty)$ such that, for all $x \\in \\ball[\\ushort{x}, \\rho_0] \\cap C$,\n\\begin{subequations}\n\\begin{gather}\n\\label{eq:def_rho0_1}\n\\ip{w_i-\\ushort{x}}{-\\nabla f(x)} > 0,\\\\\n\\label{eq:def_rho0_2}\n\\frac{\\alpha_{\\ushort{x}}}{2}\n< \\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(x)}}\n< \\alpha_{\\ushort{x}},\\\\\n\\label{eq:def_rho0_3}\n\\frac{\\ip{w_i-\\ushort{x}}{-\\nabla f(x)}^2}{\\norm{w_i-\\ushort{x}}^2 \\norm{\\nabla f(x)}^2}\n> \\frac{\\ip{w}{-\\nabla f(\\ushort{x})}^2}{2\\norm{w}^2 \\norm{\\nabla f(\\ushort{x})}^2}.\n\\end{gather}\n\\end{subequations}\nWe now establish the first inequality we have to prove: for an adequate value of $\\oshort{\\rho}(\\alpha_{\\ushort{x}})$, it holds for all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$ and $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$ that\n\\begin{equation*}\n\\norm{x - \\alpha \\nabla f(x) - y} \\le \\kappa \\alpha \\norm{\\nabla f(x)}\\quad\n\\forall y \\in \\proj{C}{x-\\alpha\\nabla f(x)},\n\\end{equation*}\nwhich is equivalent to $\\dist(x-\\alpha\\nabla f(x), C) \\le \\kappa \\alpha \\norm{\\nabla f(x)}$.\n\nFor the moment, let $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\rho_0]$ be arbitrary. For all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{align*}\n\\norm{x - \\alpha \\nabla f(x) - y}^2\n&\\le \\norm{x - \\alpha \\nabla f(x) - w_i}^2 \\\\\n&= \\norm{\\ushort{x}-\\alpha \\nabla f(x) - w_i}^2 + 2 \\ip{\\ushort{x}-x}{\\alpha \\nabla f(x) + w_i-\\ushort{x}} + \\norm{\\ushort{x}-x}^2 \\\\\n&\\le \\norm{\\ushort{x}-\\alpha \\nabla f(x) - w_i}^2 \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\alpha \\norm{\\nabla f(x)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&\\le \\norm{\\ushort{x}-\\alpha \\nabla f(x) - w_i}^2 \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\alpha \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&= \\alpha^2 \\norm{\\nabla f(x)}^2 -2 \\alpha \\ip{w_i-\\ushort{x}}{-\\nabla f(x)} + \\norm{w_i-\\ushort{x}}^2 \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\alpha \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 - \\alpha \\ip{w_i-\\ushort{x}}{-\\nabla f(x)} \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\frac{\\alpha_{\\ushort{x}}}{\\beta} \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right)\n+ \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2,\n\\end{align*}\nwhere the last inequality follows from \\eqref{eq:def_rho0_2} and the fact that $\\alpha_{\\ushort{x}} \\le \\alpha \\le \\frac{\\alpha_{\\ushort{x}}}{\\beta}$.\nNow, choose $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\rho_0]$ small enough to ensure\n\\begin{align*}\n&2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\frac{\\alpha_{\\ushort{x}}}{\\beta} \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&\\le \\frac{\\alpha_{\\ushort{x}}}{2} \\min_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\ip{w_i-\\ushort{x}}{-\\nabla f(z)}.\n\\end{align*}\nNote that the right-hand side of this inequality is positive, from \\eqref{eq:def_rho0_1}.\nCombining this definition with the previous inequality, we arrive at\n\\begin{align*}\n\\norm{x - \\alpha \\nabla f(x) - y}^2\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 - \\frac{\\alpha}{2} \\ip{w_i-\\ushort{x}}{-\\nabla f(x)}\\\\\n&= \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\ip{w_i-\\ushort{x}}{-\\nabla f(x)}}{2 \\alpha \\norm{\\nabla f(x)}^2}\\right)\\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\beta \\ip{w_i-\\ushort{x}}{-\\nabla f(x)}}{2 \\alpha_{\\ushort{x}} \\norm{\\nabla f(x)}^2}\\right)\n\\text{ as $\\alpha \\le \\frac{\\alpha_{\\ushort{x}}}{\\beta}$}\\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\beta \\ip{w_i-\\ushort{x}}{-\\nabla f(x)}^2}{4 \\norm{w_i-\\ushort{x}}^2 \\norm{\\nabla f(x)}^2}\\right)\n\\text{ from~\\eqref{eq:def_rho0_2}}\\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\beta \\ip{w}{-\\nabla f(\\ushort{x})}^2}{8 \\norm{w}^2 \\norm{\\nabla f(\\ushort{x})}^2}\\right)\n\\text{ from~\\eqref{eq:def_rho0_3}}\\\\\n&= \\kappa^2 \\alpha^2 \\norm{\\nabla f(x)}^2.\n\\end{align*}\nThus, for all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{equation*}\n\\norm{x - \\alpha \\nabla f(x) - y} \\le \\kappa \\alpha \\norm{\\nabla f(x)}.\n\\end{equation*}\nTo conclude, we show that this inequality implies\n\\begin{equation}\n\\label{eq:ip_xy_nabla}\n\\ip{\\frac{y-x}{\\norm{y-x}}}{\\frac{\\nabla f(x)}{\\norm{\\nabla f(x)}}} \\le - \\sqrt{1-\\kappa^2}.\n\\end{equation}\nIndeed, defining $\\theta \\in \\R$ such that $\\ip{\\frac{y-x}{\\norm{y-x}}}{\\frac{\\nabla f(x)}{\\norm{\\nabla f(x)}}} = \\cos(\\theta)$, we have\n\\begin{equation*}\n\\norm{y-x}^2 + 2 \\alpha \\norm{\\nabla f(x)} \\norm{y-x} \\cos(\\theta) + \\alpha^2 \\norm{\\nabla f(x)}^2\n\\le \\alpha^2 \\kappa^2 \\norm{\\nabla f(x)}^2.\n\\end{equation*}\nThis already shows that $\\cos(\\theta) < 0$. In addition, minimizing the left-hand side over all possible values of $\\norm{y-x}$, we get\n\\begin{equation*}\n- \\alpha^2 \\norm{\\nabla f(x)}^2 \\cos^2(\\theta) + \\alpha^2 \\norm{\\nabla f(x)}^2\n\\le \\alpha^2 \\kappa^2 \\norm{\\nabla f(x)}^2,\n\\end{equation*}\nhence $\\cos^2(\\theta) \\ge 1 - \\kappa^2$, which establishes \\eqref{eq:ip_xy_nabla}.\n\\end{proof}","start":54513,"end":61656},{"raw_tex":"\\begin{proposition}\n\\label{prop:AcceptedStep}\nLet $\\ushort{\\alpha} \\in (0, \\infty)$ and $\\beta, c \\in (0, 1)$.\nAssume that $f$ satisfies~\\ref{it:continuously_diff}. Let $\\ushort{x} \\in C$ be non-B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}.\nThere exist $\\alpha_{\\ushort{x}} \\in (0, \\ushort{\\alpha}]$ and $\\rho \\in (0, \\infty)$ such that, for all $x \\in \\ball(\\ushort{x}, \\rho) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{equation*}\nf(y) < f(x) + c \\ip{\\nabla f(x)}{y-x}.\n\\end{equation*}\n\\end{proposition}","start":61658,"end":62248},{"raw_tex":"\\begin{proof}\nDefine $\\kappa$ as in Proposition~\\ref{prop:ArbitrarySmallAlpha}. Let $\\delta \\in (0, \\infty)$ be small enough to ensure\n\\begin{subequations}\n\\begin{align}\n\\label{eq:def_alpha_x_small1}\n\\sup_{y \\in \\ball\\left[\\ushort{x}, \\frac{7\\delta}{2\\beta}\\norm{\\nabla f(\\ushort{x})}\\right] \\cap C \\setminus \\{\\ushort{x}\\}}\n\\frac{|f(y)-f(\\ushort{x}) - \\ip{\\nabla f(\\ushort{x})}{y-\\ushort{x}}|}{\\norm{y-\\ushort{x}}}\n&< \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4\\left(1 + \\frac{8}{3(1-\\kappa)}\\right)},\\\\\n\\label{eq:def_alpha_x_small2}\n\\sup_{y \\in \\ball\\left[\\ushort{x}, \\frac{7\\delta}{2\\beta}\\norm{\\nabla f(\\ushort{x})}\\right] \\cap C}\n\\norm{\\nabla f(y)-\\nabla f(\\ushort{x})}\n&< \\frac{(1-c)\\sqrt{1-\\kappa^2}}{4} \\norm{\\nabla f(\\ushort{x})}.\n\\end{align}\n\\end{subequations}\nThese inequalities are satisfied by all $\\delta$ small enough, from the definition of the gradient for the first one, and because the gradient is continuous at $\\ushort{x}$ for the second one.\n\nThen, define $\\varepsilon \\coloneq \\min\\left\\{\\ushort{\\alpha}, \\delta\\right\\}$ and let $\\alpha_{\\ushort{x}} \\in (0, \\varepsilon]$ and $\\oshort{\\rho}(\\alpha_{\\ushort{x}})\\in(0,\\infty)$ be given by Proposition~\\ref{prop:ArbitrarySmallAlpha}.\nDefine\n\\begin{equation*}\n\\rho \\coloneq \\min\\left\\{\\oshort{\\rho}(\\alpha_{\\ushort{x}}), \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})}\\right\\}.\n\\end{equation*}\nNote that, for all $x \\in \\ball(\\ushort{x}, \\rho) \\cap C$,\n\\begin{equation*}\n\\norm{x-\\ushort{x}}\n< \\rho \\le \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})}\n< \\frac{7\\alpha_{\\ushort{x}}}{2\\beta} \\norm{\\nabla f(\\ushort{x})}\n\\leq \\frac{7\\delta }{2\\beta} \\norm{\\nabla f(\\ushort{x})},\n\\end{equation*}\nso that from~\\eqref{eq:def_alpha_x_small2}, $\\norm{\\nabla f(x)-\\nabla f(\\ushort{x})} < \\frac{\\norm{\\nabla f(\\ushort{x})}}{4}$, which implies\n\\begin{align}\n\\nonumber\n\\frac{3}{4} \\norm{\\nabla f(\\ushort{x})}\n& < \\norm{\\nabla f(\\ushort{x})}-\\norm{\\nabla f(x)-\\nabla f(\\ushort{x})} \\\\\n& \\le \\norm{\\nabla f(x)} \\nonumber\\\\\n& \\le \\norm{\\nabla f(\\ushort{x})}+\\norm{\\nabla f(x)-\\nabla f(\\ushort{x})} \\nonumber\\\\\n& < \\frac{5}{4} \\norm{\\nabla f(\\ushort{x})}.\n\\label{eq:nablas_ushortx_x}\n\\end{align}\nFor all $x \\in \\ball(\\ushort{x}, \\rho) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{align}\nf(y)\n&= f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x} \\nonumber\\\\\n&\\qquad + \\left(f(\\ushort{x}) - f(x) - \\ip{\\nabla f(\\ushort{x})}{\\ushort{x}-x}\\right) \\nonumber\\\\\n&\\qquad + \\left(f(y) - f(\\ushort{x}) - \\ip{\\nabla f(\\ushort{x})}{y-\\ushort{x}}\\right) \\nonumber\\\\\n&\\le f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x} +\n\\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4\\left(1 + \\frac{8}{3(1-\\kappa)}\\right)} \\left(\\norm{\\ushort{x}-x} + \\norm{y-\\ushort{x}}\\right),\n\\label{eq:UpperBoundfDiffGrad}\n\\end{align}\nwhere the inequality follows from \\eqref{eq:def_alpha_x_small1}. Observe that\n\\begin{align*}\n\\norm{y-\\ushort{x}}\n&\\le \\norm{y-x} + \\norm{x-\\ushort{x}} \\\\\n&\\le 2 \\alpha \\norm{\\nabla f(x)} + \\rho\n\\text{ from \\eqref{eq:ProjectedTranslationDistance}}\\\\\n&\\le \\frac{2 \\alpha_{\\ushort{x}}}{\\beta} \\norm{\\nabla f(x)}\n+ \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})} \\\\\n&< \\frac{5\\alpha_{\\ushort{x}}}{2\\beta} \\norm{\\nabla f(\\ushort{x})}\n+ \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})} \n\\text{ from \\eqref{eq:nablas_ushortx_x}} \\\\\n&\\le \\frac{7\\alpha_{\\ushort{x}}}{2\\beta} \\norm{\\nabla f(\\ushort{x})}.\n\\end{align*}\nWe continue from \\eqref{eq:UpperBoundfDiffGrad}:\n\\begin{align*}\nf(y)\n& \\le f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x} +\n\\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{\n4\\left(1 + \\frac{8}{3(1-\\kappa)}\\right)} \\left(2\\norm{\\ushort{x}-x} + \\norm{y-x}\\right) \\\\\n&\\overset{\\text{(a)}}{<} f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x}\n+ \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4} \\norm{y-x} \\\\\n&\\le f(x) + \\ip{\\nabla f(x)}{y-x} + \\norm{\\nabla f(\\ushort{x})-\\nabla f(x)} \\norm{y-x} \\\\\n&\\qquad + \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4} \\norm{y-x} \\\\\n&\\le f(x) + \\ip{\\nabla f(x)}{y-x} + \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{2} \\norm{y-x}\n\\text{ from~\\eqref{eq:def_alpha_x_small2}}\\\\\n&< f(x) + \\ip{\\nabla f(x)}{y-x} + (1-c)\\sqrt{1-\\kappa^2} \\norm{\\nabla f(x)}\\norm{y-x}\n\\text{ from~\\eqref{eq:nablas_ushortx_x}}\\\\\n&\\le f(x) + \\ip{\\nabla f(x)}{y-x} - (1-c) \\ip{\\nabla f(x)}{y-x}\n\\text{ from Proposition~\\ref{prop:ArbitrarySmallAlpha}}\\\\\n&= f(x) + c \\ip{\\nabla f(x)}{y-x}.\n\\end{align*}\nInequality (a) is true because\n\\begin{align*}\n\\norm{y-x}\n&\\ge \\alpha \\norm{\\nabla f(x)} - \\norm{x-\\alpha \\nabla f(x)-y}\n\\text{ by the triangle inequality}\\\\\n&= \\alpha \\norm{\\nabla f(x)} - \\dist(x-\\alpha\\nabla f(x), C) \\\\\n&\\ge (1-\\kappa) \\alpha \\norm{\\nabla f(x)}\n\\text{ from Proposition~\\ref{prop:ArbitrarySmallAlpha}}\\\\\n&\\ge \\frac{3}{4} (1-\\kappa) \\rho\n\\text{ from~\\eqref{eq:nablas_ushortx_x}, the definition of $\\rho$, and $\\alpha_{\\ushort{x}} \\le \\alpha$}\\\\\n&> \\frac{3}{4} (1-\\kappa) \\norm{x-\\ushort{x}}.\n\\end{align*}\n\\end{proof}","start":62250,"end":67244},{"raw_tex":"\\begin{proposition}\n\\label{prop:convergence_ik}\nLet $(x_i)_{i \\in \\N}$ be a sequence generated by $\\pgd$ (Algorithm~\\ref{algo:PGD}), under~\\ref{it:continuously_diff}, and $x$ be an accumulation point.\nThen, for every subsequence $(x_{i_k})_{k\\in\\N}$ converging to $x$, the sequence $(x_{i_k+1})_{k\\in\\N}$ also converges to $x$.\n\\end{proposition}","start":67338,"end":67683},{"raw_tex":"\\begin{proof}\nLet $(x_{i_k})_{k\\in\\N}$ be a subsequence converging to $x$. We show that $(x_{i_k+1})_{k\\in\\N}$ also converges to $x$.\n\nIf the nonmonotonicity rule is set to ``average'', this is a direct consequence of Proposition~\\ref{prop:NonmonotonePGDAverage}. Indeed, for all $i \\in \\N$, from~\\eqref{eq:KanzowMehlitzDecay},\n\\begin{equation*}\nf(x_{i+1}) \\le \\mu_i - \\frac{c}{2 \\oshort{\\alpha}} \\norm{x_{i+1}-x_i}^2 \\le \\mu_i.    \n\\end{equation*}\nFrom Proposition~\\ref{prop:NonmonotonePGDAverage}, $(f(x_{i+1}))_{i\\in\\N}$ and $(\\mu_i)_{i\\in\\N}$ converge to the same limit. Therefore,\n\\begin{equation*}\n \\left(\\mu_i - \\frac{c}{2 \\oshort{\\alpha}} \\norm{x_{i+1}-x_i}^2\\right)_{i\\in\\N}\n\\end{equation*}\nalso converges to this limit.\nThis implies that $(\\norm{x_{i+1}-x_i})_{i\\in\\N}$ converges to $0$, hence $(\\norm{x_{i_k+1}-x_{i_k}})_{k\\in\\N}$ converges to $0$, and $(x_{i_k+1})_{k\\in\\N}$ converges to the same limit as $(x_{i_k})_{k\\in\\N}$, that is, $x$.\n\nNow, let us consider the ``max'' rule case. It suffices to show that $x$ is an accumulation point of every subsequence of $(x_{i_k+1})_{k\\in\\N}$. Thus, we show the following: for every subsequence $(i_{j_k})_{k\\in\\N}$ of $(i_k)_{k\\in\\N}$, there exists a subsequence of $(x_{i_{j_k}+1})_{k\\in\\N}$ that converges to $x$.\nLet $(i_{j_k})_{k\\in\\N}$ be a subsequence of $(i_k)_{k\\in\\N}$.\nFor all $i \\in \\N$, let $g(i) \\in \\argmax_{j \\in \\{\\max\\{0, i-l\\}, \\dots, i\\}} f(x_j)$, as in Proposition~\\ref{prop:NonmonotonePGD}. By the third statement of Proposition~\\ref{prop:NonmonotonePGD}, the sequence $(f(x_{g(i)}))_{i\\in\\N}$ converges to $\\varphi \\in [f(x), f(x_0)]$.\nFor every $k \\in \\N$, letting $\\alpha_{i_{j_k}} \\in (0, \\oshort{\\alpha}]$ be the number such that $x_{i_{j_k}+1} \\in \\proj{C}{x_{i_{j_k}} - \\alpha_{i_{j_k}} \\nabla f(x_{i_{j_k}})}$, by \\eqref{eq:ProjectedTranslationDistance},\n\\begin{equation*}\n\\norm{x_{i_{j_k}+1}-x_{i_{j_k}}}\n\\le 2 \\alpha_{i_{j_k}} \\norm{\\nabla f(x_{i_{j_k}})}\n\\le 2 \\oshort{\\alpha} \\norm{\\nabla f(x_{i_{j_k}})}.\n\\end{equation*}\nThus, since $(x_{i_{j_k}})_{k\\in\\N}$ is bounded and $\\nabla f$ is locally bounded (as it is continuous), the sequence $(x_{i_{j_k}+1})_{k\\in\\N}$ is bounded. If we replace $(i_{j_k})_{k\\in\\N}$ with a subsequence, we can assume that $(x_{i_{j_k}+1})_{k\\in\\N}$ converges.\n\nIterating the reasoning, we can assume that $(x_{i_{j_k}+s})_{k\\in\\N}$ converges to some $x^{s}\\in C$ for every $s \\in \\{0, \\dots, l+1\\}$. By definition of $x$, $x^{0}=x$.\n\nObserve that, from the continuity of $f$,\n\\begin{align*}\nf(x_{g(i_{j_k}+l+1)})\n&= \\max\\{f(x_{i_{j_k}+1}), \\dots, f(x_{i_{j_k}+l+1})\\} \\\\\n&\\to \\max\\{f(x^{1}), \\dots, f(x^{l+1})\\}\n\\text{ when } k \\to \\infty.\n\\end{align*}\nIn particular, there exists $s_1 \\in \\{1, \\dots, l+1\\}$ such that\n\\begin{equation}\n\\label{eq:def_s_1}\nf(x^{s_1}) = \\varphi.\n\\end{equation}\nLet $s_1$ be the smallest such integer.\nFor all $k \\in \\N$, from the condition in line~\\ref{algo:PGDmap:NonmonotoneArmijoCondition} of Algorithm~\\ref{algo:PGDmap} and~\\eqref{eq:KanzowMehlitzDecay},\n\\begin{align*}\nf(x_{i_{j_k}+s_1})\n&\\le f(x_{g(i_{j_k}+s_1-1)}) - \\frac{c}{2\\oshort{\\alpha}} \\norm{x_{i_{j_k}+s_1}-x_{i_{j_k}+s_1-1}}^2,\n\\end{align*}\nand letting $k$ tend to infinity yields\n\\begin{equation*}\n\\varphi\n= f(x^{s_1})\n\\le \\varphi - \\frac{c}{2\\oshort{\\alpha}} \\norm{x^{s_1}-x^{s_1-1}}^2.\n\\end{equation*}\nConsequently, $x^{s_1} = x^{s_1-1}$. In particular, $f(x^{s_1-1}) =f(x^{s_1}) = \\varphi$. Therefore, $s_1=1$, otherwise it would not be the smallest integer satisfying \\eqref{eq:def_s_1}. The equality $x^{s_1} = x^{s_1-1}$ then reduces to $x^{1}=x^{0}=x$ and, when $k \\to \\infty$,\n\\begin{equation*}\nx_{i_{j_k}+1} \\to x^{1} = x.\n\\end{equation*}\n\\end{proof}","start":67685,"end":71365},{"raw_tex":"\\subsection{Third part: proof of Theorem~\\ref{thm:PGDaccumulatesAtBStationaryPoints}}\n\\label{subsec:proof_accumulates}\nLet $\\ushort{x}$ be an accumulation point of $(x_i)_{i\\in\\N}$. Assume, for the sake of contradiction, that $\\ushort{x}$ is not B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}. Let $(x_{i_k})_{k\\in\\N}$ be a subsequence converging to $\\ushort{x}$.\n\nLet $\\alpha_{\\ushort{x}}$ and $\\rho$ be as in Proposition~\\ref{prop:AcceptedStep}. For all $k \\in \\N$ large enough, $x_{i_k} \\! \\in \\ball(\\ushort{x}, \\rho) \\cap C$. Thus, when Algorithm~\\ref{algo:PGDmap} is called at point $x_{i_k}$, the condition in line~\\ref{algo:PGDmap:NonmonotoneArmijoCondition} stops being fulfilled for some $\\alpha_{i_k} \\ge \\alpha_{\\ushort{x}}$, meaning that\n\\begin{equation*}\nx_{i_k+1} \\in \\proj{C}{x_{i_k}-\\alpha_{i_k} \\nabla f(x_{i_k})}\n\\text{ for some } \\alpha_{i_k} \\in [\\alpha_{\\ushort{x}}, \\oshort{\\alpha}].\n\\end{equation*}\nReplacing $(i_k)_{k\\in\\N}$ with a subsequence if necessary, we can assume that $(\\alpha_{i_k})_{k\\in\\N}$ converges to some $\\alpha_{\\lim} \\in [\\alpha_{\\ushort{x}}, \\oshort{\\alpha}]$.\n\nFor all $k \\in \\N$, we have\n\\begin{equation*}\n\\norm{x_{i_k} - \\alpha_{i_k} \\nabla f(x_{i_k}) - x_{i_k+1}}\n= \\dist(x_{i_k} - \\alpha_{i_k} \\nabla f(x_{i_k}), C),\n\\end{equation*}\nand since the distance to a nonempty closed set is a continuous function, we can take this equality to the limit. We use the fact that $x_{i_k+1} \\to \\ushort{x}$ when $k \\to \\infty$, from Proposition~\\ref{prop:convergence_ik}. This yields\n\\begin{equation*}\n\\norm{\\alpha_{\\lim} \\nabla f(\\ushort{x})}\n= \\dist(\\ushort{x} - \\alpha_{\\lim}\\nabla f(\\ushort{x}), C),\n\\end{equation*}\nwhich means that $\\ushort{x} \\in \\proj{C}{\\ushort{x} - \\alpha_{\\lim}\\nabla f(\\ushort{x})}$. In particular, $-\\nabla f(\\ushort{x}) \\in \\proxnorcone{C}{\\ushort{x}} \\subseteq \\regnorcone{C}{\\ushort{x}}$, which contradicts our assumption that $\\ushort{x}$ is not B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}. We have therefore proven that every accumulation point is B-stationary.\n\nFinally, if $(x_i)_{i\\in\\N}$ has an isolated accumulation point, then the sequence $(x_i)_{i\\in\\N}$ converges, from Proposition~\\ref{prop:convergence_ik} and \\cite[Lemma~4.10]{MoreSorensen}.\n\n","start":71367,"end":73608}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:MinDiffFunctionClosedSet"],"declared_external":false,"source_only":false,"title":"闭集上的光滑优化问题","source_units":[{"raw_tex":"Let $\\mathcal{E}$ be a Euclidean vector space, $C$ a nonempty closed subset of $\\mathcal{E}$, and $f : \\mathcal{E} \\to \\R$ a function satisfying at least the first of the following two properties:\n","start":5818,"end":6015},{"raw_tex":"This paper considers the problem\n\\begin{equation}\n\\label{eq:MinDiffFunctionClosedSet}\n\\min_{x \\in C} f(x)\n\\end{equation}\nof minimizing $f$ on~$C$.\n","start":6599,"end":6746}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"可微性与可行集上梯度连续","source_units":[{"raw_tex":"\\begin{enumerate}[label={(H\\arabic*)}]\n\\item $f$ is differentiable on $C$, i.e., for every $x \\in C$, there exists a (unique) vector in $\\mathcal{E}$, denoted by $\\nabla f(x)$, such that\n\\begin{equation*}\n\\lim_{\\substack{y \\to x \\\\ y \\in \\mathcal{E} \\setminus \\{x\\}}} \\frac{f(y)-f(x)-\\ip{\\nabla f(x)}{y-x}}{\\norm{y-x}} = 0,\n\\end{equation*}\nand $\\nabla f : C \\to \\mathcal{E}$ is continuous;\n\\label{it:continuously_diff}\n","start":6015,"end":6434}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"更强的局部 Lipschitz 假设","source_units":[{"raw_tex":"\\item $f$ is differentiable on $\\mathcal{E}$ and $\\nabla f : \\mathcal{E} \\to \\mathcal{E}$ is locally Lipschitz continuous.\n\\label{it:Lipschitz_diff}\n\\end{enumerate}\n","start":6434,"end":6599}]},{"id":"S004","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:NestedNormalCones"],"declared_external":false,"source_only":false,"title":"法锥记号与包含关系","source_units":[{"raw_tex":"A point $x \\in C$ is said to be stationary for~\\eqref{eq:MinDiffFunctionClosedSet} if $-\\nabla f(x)$ is normal to $C$ at~$x$. Several definitions of normality exist. Each defines a different notion of stationarity, which is a surrogate for local optimality in the sense that, possibly under mild regularity assumptions on~$f$, every local minimizer of $f|_C$ is stationary for~\\eqref{eq:MinDiffFunctionClosedSet}. In particular, each of the three notions of normality in \\cite[Definition~6.3 and Example~6.16]{RockafellarWets}, namely normality in the general sense, in the regular sense, and in the proximal sense, yields an important definition of stationarity. The sets of general, regular, and proximal normals to $C$ at $x \\in C$ are respectively denoted by $\\norcone{C}{x}$, $\\regnorcone{C}{x}$, and $\\proxnorcone{C}{x}$. These sets are reviewed in section~\\ref{subsec:NormalityStationarity}. Importantly, they are nested as follows: for every $x \\in C$,\n\\begin{equation}\n\\label{eq:NestedNormalCones}\n\\proxnorcone{C}{x} \\subseteq \\regnorcone{C}{x} \\subseteq \\norcone{C}{x},\n\\end{equation}\n","start":7214,"end":8309}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"Clarke 正则性","source_units":[{"raw_tex":"and $C$ is said to be \\emph{Clarke regular} at $x$ if the second inclusion is an equality.\n","start":8309,"end":8400}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["def:M/B/P-Stationarity"],"declared_external":false,"source_only":false,"title":"Bouligand 驻点","source_units":[{"raw_tex":"\\begin{definition}\n\\label{def:M/B/P-Stationarity}\nFor problem~\\eqref{eq:MinDiffFunctionClosedSet}, a point $x \\in C$ is said to be:\n\\begin{itemize}\n","start":8585,"end":8733},{"raw_tex":"\\item \\emph{Bouligand stationary (B-stationary)} if $-\\nabla f(x) \\in \\regnorcone{C}{x}$;\n","start":8823,"end":8913},{"raw_tex":"\\end{itemize}\n\\end{definition}\n","start":9005,"end":9036}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":["def:M/B/P-Stationarity"],"declared_external":false,"source_only":false,"title":"其他两类驻点","source_units":[{"raw_tex":"\\item \\emph{Mordukhovich stationary (M-stationary)} if $-\\nabla f(x) \\in \\norcone{C}{x}$;\n","start":8733,"end":8823},{"raw_tex":"\\item \\emph{proximally stationary (P-stationary)} if $-\\nabla f(x) \\in \\proxnorcone{C}{x}$.\n","start":8913,"end":9005}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":["eq:InitialSublevelSet"],"declared_external":false,"source_only":false,"title":"初始次水平集","source_units":[{"raw_tex":"\\begin{equation}\n\\label{eq:InitialSublevelSet}\n\\{x \\in C \\mid f(x) \\le f(x_0)\\},\n\\end{equation}\n","start":13761,"end":13857}]},{"id":"S009","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"既有 M 驻点结论的条件","source_units":[{"raw_tex":"Monotone $\\pgd$ accumulates at M-stationary points of~\\eqref{eq:MinDiffFunctionClosedSet} if $f$ is continuously differentiable on~$\\mathcal{E}$ and bounded from below on~$C$ \\cite[Theorem~3.1]{KanzowMehlitz}. The same result holds for nonmonotone $\\pgd$ with the ``average'' rule \\cite[Theorem~4.6]{DeMarchi}, and also for nonmonotone $\\pgd$ with the ``max'' rule if $f$ is further uniformly continuous on the sublevel set\n","start":13337,"end":13761},{"raw_tex":"where $x_0 \\in C$ is the initial iterate given to the algorithm \\cite[Theorem~4.1]{KanzowMehlitz}.\n","start":13857,"end":13956}]},{"id":"S010","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"开球与闭球","source_units":[{"raw_tex":"Recall that, throughout the paper, $\\mathcal{E}$ is a Euclidean vector space and $C \\subseteq \\mathcal{E}$ is nonempty and closed. Moreover, for every $x \\in \\mathcal{E}$ and $\\rho \\in (0, \\infty)$, $\\ball(x, \\rho) \\coloneq \\{y \\in \\mathcal{E} \\mid \\norm{x-y} < \\rho\\}$ and $\\ball[x, \\rho] \\coloneq \\{y \\in \\mathcal{E} \\mid \\norm{x-y} \\le \\rho\\}$ are respectively the open and closed balls of center $x$ and radius $\\rho$ in~$\\mathcal{E}$.\n","start":18485,"end":18925}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"锥的含义","source_units":[{"raw_tex":"A nonempty subset $K$ of $\\mathcal{E}$ is called a \\emph{cone} if $x \\in K$ implies $\\alpha x \\in K$ for all $\\alpha \\in [0, \\infty)$ \\cite[\\S 3B]{RockafellarWets}.\n","start":18925,"end":19090}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"距离、投影与基本性质","source_units":[{"raw_tex":"Given $x \\in \\mathcal{E}$, the distance from $x$ to $C$ is $\\dist(x, C) \\coloneq \\min_{y \\in C} \\norm{x-y}$ and the projection of $x$ onto $C$ is $\\proj{C}{x} \\coloneq \\argmin_{y \\in C} \\norm{x-y}$. The function $\\mathcal{E} \\to \\R : x \\mapsto \\dist(x, C)$ is continuous and, for every $x \\in \\mathcal{E}$, the set $\\proj{C}{x}$ is nonempty and compact \\cite[Example~1.20]{RockafellarWets}.\nProposition~\\ref{prop:ProjectedTranslation} is invoked frequently in the rest of the paper.\n","start":19148,"end":19631}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"切向量的序列定义","source_units":[{"raw_tex":"A vector $v \\in \\mathcal{E}$ is said to be \\emph{tangent} to $C$ at $x \\in C$ if there exist sequences $(x_i)_{i \\in \\N}$ in $C$ converging to $x$ and $(t_i)_{i \\in \\N}$ in $(0, \\infty)$ such that the sequence $(\\frac{x_i-x}{t_i})_{i \\in \\N}$ converges to $v$ \\cite[Definition~6.1]{RockafellarWets}. The set of all tangent vectors to $C$ at $x \\in C$ is a closed cone \\cite[Proposition~6.2]{RockafellarWets} called the \\emph{tangent cone} to $C$ at $x$ and denoted by $\\tancone{C}{x}$.\n","start":20713,"end":21199}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"正则法锥的极锥定义","source_units":[{"raw_tex":"The \\emph{regular normal cone} to $C$ at $x \\in C$ is\n\\begin{equation*}\n\\regnorcone{C}{x} \\coloneq \\left\\{v \\in \\mathcal{E} \\mid \\ip{v}{w} \\le 0 \\; \\forall w \\in \\tancone{C}{x}\\right\\},\n\\end{equation*}\nwhich is a closed convex cone \\cite[Definition~6.3 and Proposition~6.5]{RockafellarWets}.\n","start":21199,"end":21491}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"一般法锥","source_units":[{"raw_tex":"A vector $v \\in \\mathcal{E}$ is said to be \\emph{normal (in the general sense)} to $C$ at $x \\in C$ if there exist sequences $(x_i)_{i \\in \\N}$ in $C$ converging to $x$ and $(v_i)_{i \\in \\N}$ converging to $v$ such that, for all $i \\in \\N$, $v_i \\in \\regnorcone{C}{x_i}$ \\cite[Definition~6.3]{RockafellarWets}. The set of all normal vectors to $C$ at $x \\in C$ is a closed cone \\cite[Proposition~6.5]{RockafellarWets} called the \\emph{normal cone} to $C$ at $x$ and denoted by $\\norcone{C}{x}$.\n","start":21491,"end":21986}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"近端法向量的投影刻画","source_units":[{"raw_tex":"A vector $v \\in \\mathcal{E}$ is called a \\emph{proximal normal} to $C$ at $x \\in C$ if there exists $\\oshort{\\alpha} \\in (0, \\infty)$ such that $x \\in \\proj{C}{x+\\oshort{\\alpha}v}$, i.e., $\\oshort{\\alpha}\\norm{v} = \\dist(x+\\oshort{\\alpha}v, C)$, which implies that, for all $\\alpha \\in [0, \\oshort{\\alpha})$, $\\proj{C}{x+\\alpha v} = \\{x\\}$ \\cite[Example~6.16]{RockafellarWets}. The set of all proximal normals to $C$ at $x \\in C$ is a convex cone called the \\emph{proximal normal cone} to $C$ at $x$ and denoted by $\\proxnorcone{C}{x}$.\n","start":21986,"end":22523}]},{"id":"S017","semantic_type":"algorithm","kind":"algorithm","labels":["algo:PGDmap","algo:PGDmap:InitialStepSize","algo:PGDmap:NonmonotoneArmijoCondition"],"declared_external":false,"source_only":false,"title":"PGD 回溯映射","source_units":[{"raw_tex":"\\begin{algorithm}[H]\n\\caption{$\\pgd$ map}\n\\label{algo:PGDmap}\n\\begin{algorithmic}[1]\n\\Require\n$(\\mathcal{E}, C, f, \\ushort{\\alpha}, \\oshort{\\alpha}, \\beta, c)$, where $\\mathcal{E}$ is a Euclidean vector space, $C$ is a nonempty closed subset of~$\\mathcal{E}$, $f : \\mathcal{E} \\to \\R$ satisfies \\ref{it:continuously_diff}, $0 < \\ushort{\\alpha} \\le \\oshort{\\alpha} < \\infty$, and $\\beta, c \\in (0, 1)$.\n\\Input\n$(x, \\mu)$ with $x \\in C$ and $\\mu \\in [f(x), \\infty)$.\n\\Output\n$y \\in \\pgd(x, \\mu; \\mathcal{E}, C, f, \\ushort{\\alpha}, \\oshort{\\alpha}, \\beta, c)$.\n\n\\State\nChoose $\\alpha \\in [\\ushort{\\alpha}, \\oshort{\\alpha}]$ and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$;\n\\label{algo:PGDmap:InitialStepSize}\n\\While\n{$f(y) > \\mu + c \\ip{\\nabla f(x)}{y-x}$}\n\\label{algo:PGDmap:NonmonotoneArmijoCondition}\n\\State\n$\\alpha \\gets \\alpha \\beta$;\n\\State\nChoose $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$;\n\\EndWhile\n\\State\nReturn $y$.\n\\end{algorithmic}\n\\end{algorithm}\n","start":39371,"end":40318}]},{"id":"S018","semantic_type":"algorithm","kind":"algorithm","labels":["algo:PGD","algo:PGD:PGDmap","algo:PGD:while"],"declared_external":false,"source_only":false,"title":"两种非单调规则的 PGD","source_units":[{"raw_tex":"The $\\pgd$ algorithm is defined as Algorithm~\\ref{algo:PGD}. It is said to be monotone or nonmonotone depending on whether $\\mu_i=f(x_i)$ for all $i$ (that is, $l=0$ for the ``max'' rule, or $p=1$ for the ``average'' rule) or not.\n\n\\begin{algorithm}[H]\n\\caption{$\\pgd$}\n\\label{algo:PGD}\n\\begin{algorithmic}[1]\n\\Require\n$(\\mathcal{E}, C, f, \\ushort{\\alpha}, \\oshort{\\alpha}, \\beta, c, \\text{``nonmonotonicity''})$, where $\\mathcal{E}$ is a Euclidean vector space, $C$ is a nonempty closed subset of~$\\mathcal{E}$, $f : \\mathcal{E} \\to \\R$ satisfies \\ref{it:continuously_diff}, $0 < \\ushort{\\alpha} \\le \\oshort{\\alpha} < \\infty$, $\\beta, c \\in (0, 1)$, and $\\text{``nonmonotonicity''} \\in \\{(\\text{``max''}, l), (\\text{``average''}, p)\\}$ with $l \\in \\N$ and $p \\in (0, 1]$.\n\\Input\n$x_0 \\in C$.\n\\Output\na sequence in $C$.\n\n\\State\n$i \\gets 0$;\n\\State\n$\\mu_{-1} \\gets f(x_0)$;\n\\While\n{$-\\nabla f(x_i) \\notin \\regnorcone{C}{x_i}$}\n\\label{algo:PGD:while}\n\\If\n{$\\text{``nonmonotonicity''} = (\\text{``max''}, l)$}\n\\State\n$\\mu_i \\gets \\max_{j \\in \\{\\max\\{0, i-l\\}, \\dots, i\\}} f(x_j)$;\n\\ElsIf\n{$\\text{``nonmonotonicity''} = (\\text{``average''}, p)$}\n\\State\n$\\mu_i \\gets (1-p)\\mu_{i-1} + pf(x_i)$; \n\\EndIf\n\\State\nChoose $x_{i+1} \\in \\hyperref[algo:PGDmap]{\\pgd}(x_i, \\mu_i; \\mathcal{E}, C, f, \\ushort{\\alpha}, \\oshort{\\alpha}, \\beta, c)$;\n\\label{algo:PGD:PGDmap}\n\\State\n$i \\gets i+1$;\n\\EndWhile\n\\end{algorithmic}\n\\end{algorithm}\n","start":41697,"end":43116}]},{"id":"S019","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:KanzowMehlitzDecay"],"declared_external":false,"source_only":false,"title":"Armijo 条件推出平方下降界","source_units":[{"raw_tex":"\\begin{remark}\nThe Armijo condition\n\\begin{equation*}\nf(y) \\le \\mu + c \\ip{\\nabla f(x)}{y-x}\n\\end{equation*}\nensures that the decrease $\\mu-f(y)$ is at least a fraction $c$ of the opposite of the directional derivative of $f$ at $x$ along the update vector $y-x$. By \\eqref{eq:ProjectedTranslationIP}, this condition implies that\n\\begin{equation}\\label{eq:KanzowMehlitzDecay}\nf(y) \\le \\mu - \\frac{c}{2\\alpha} \\norm{y-x}^2,\n\\end{equation}\nwhich is the condition used in \\cite[Algorithms~3.1 and 4.1]{KanzowMehlitz} and \\cite[Algorithm~3.1]{DeMarchi}. Importantly, all results from \\cite{KanzowMehlitz} hold for both conditions, as is clear from the proofs.\n","start":40319,"end":40975},{"raw_tex":"\\end{remark}\n","start":41093,"end":41106}]},{"id":"S020","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"不活跃的文献适用性备注","source_units":[{"raw_tex":"% Par contre, les résultats de De Marchi ne sont pas clairement valides avec notre condition (je pense au lemme 4.4).\n","start":40975,"end":41093}]},{"id":"S021","semantic_type":"proposition","kind":"proposition","labels":["prop:NonmonotonePGD"],"declared_external":false,"source_only":false,"title":"最大窗口值单调","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:NonmonotonePGD}\nLet $(x_i)_{i \\in \\N}$ be a sequence generated by $\\pgd$ (Algorithm~\\ref{algo:PGD}) using the ``max'' rule. For every $i \\in \\N$, let $g(i) \\in \\argmax_{j \\in \\{\\max\\{0, i-l\\}, \\dots, i\\}} f(x_j)$. Then:\n\\begin{enumerate}\n\\item $(f(x_{g(i)}))_{i \\in \\N}$ is monotonically nonincreasing;\n","start":45746,"end":46081},{"raw_tex":"\\end{enumerate}\n\\end{proposition}","start":46473,"end":46506}]},{"id":"S022","semantic_type":"proposition","kind":"proposition","labels":["prop:NonmonotonePGD"],"declared_external":false,"source_only":false,"title":"最大值规则的次水平包含","source_units":[{"raw_tex":"\\item $(x_i)_{i \\in \\N}$ is contained in the sublevel set~\\eqref{eq:InitialSublevelSet};\n","start":46081,"end":46170}]},{"id":"S023","semantic_type":"proposition","kind":"proposition","labels":["prop:NonmonotonePGD"],"declared_external":false,"source_only":false,"title":"存在聚点时窗口值有有限极限","source_units":[{"raw_tex":"\\item if $x \\in C$ is an accumulation point of $(x_i)_{i \\in \\N}$, then $(f(x_{g(i)}))_{i \\in \\N}$ converges to $\\varphi \\in [f(x), f(x_0)]$;\n","start":46170,"end":46312}]},{"id":"S024","semantic_type":"proposition","kind":"proposition","labels":["prop:NonmonotonePGD"],"declared_external":false,"source_only":false,"title":"额外条件下函数值全序列收敛","source_units":[{"raw_tex":"\\item if $f$ is bounded from below and uniformly continuous on a set that contains $(x_i)_{i \\in \\N}$, then $(f(x_i))_{i \\in \\N}$ converges to $\\varphi \\in \\R$.\n","start":46312,"end":46473}]},{"id":"S025","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"最大值规则命题的完整作者论证","source_units":[{"raw_tex":"\\begin{proof}\nThe first two statements are \\cite[Lemma~4.1 and Corollary~4.1]{KanzowMehlitz}.\nFor the third one, let $(x_{i_k})_{k \\in \\N}$ be a subsequence converging to $x$. Since the sequence $(f(x_{g(i)}))_{i\\in\\N}$ is monotonically nonincreasing, it has a limit in $\\R \\cup \\{-\\infty\\}$. Thus,\n\\begin{equation*}\n\\lim_{i \\to \\infty} f(x_{g(i)})\n= \\lim_{k \\to \\infty} f(x_{g(i_k)})\n\\ge \\lim_{k \\to \\infty} f(x_{i_k})\n= f(x)\n> -\\infty.\n\\end{equation*}\nIt remains to prove the fourth statement. From the first statement and because $f$ is bounded from below, $(f(x_{g(i)}))_{i\\in\\N}$ converges to some limit $\\varphi \\in \\R$. Assume, for the sake of contradiction, that $(f(x_i))_{i \\in \\N}$ does not converge to $\\varphi$. Then, there exist $\\rho \\in (0, \\infty)$ and a subsequence $(f(x_{i_j}))_{j \\in \\N}$ contained in $\\R \\setminus [\\varphi-\\rho, \\varphi+\\rho]$. For all $j \\in \\N$, define $p_j \\coloneq g(i_j+l)-i_j \\in \\{0, \\dots, l\\}$. Then, there exist $p \\in \\{0, \\dots, l\\}$ and a subsequence $(p_{j_k})_{k \\in \\N}$ such that, for all $k \\in \\N$, $p_{j_k} = p$. By \\cite[(27)]{KanzowMehlitz} or \\cite[(A.9)]{JiaEtAl}, $(f(x_{g(i)-p}))_{i \\in \\N}$ converges to $\\varphi$. Therefore, $(f(x_{g(i+l)-p}))_{i \\in \\N}$ converges to $\\varphi$. Hence, $(f(x_{g(i_{j_k}+l)-p}))_{k \\in \\N}$ converges to $\\varphi$. This is a contradiction since, for all $k \\in \\N$, $f(x_{g(i_{j_k}+l)-p}) = f(x_{i_{j_k}})$.\n\\end{proof}\n","start":46508,"end":47929}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"引入前两项外部结论","source_units":[{"raw_tex":"\\begin{proof}\nThe first two statements are \\cite[Lemma~4.1 and Corollary~4.1]{KanzowMehlitz}.\n","start":46508,"end":46602}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"以聚点排除负无穷极限","source_units":[{"raw_tex":"For the third one, let $(x_{i_k})_{k \\in \\N}$ be a subsequence converging to $x$. Since the sequence $(f(x_{g(i)}))_{i\\in\\N}$ is monotonically nonincreasing, it has a limit in $\\R \\cup \\{-\\infty\\}$. Thus,\n\\begin{equation*}\n\\lim_{i \\to \\infty} f(x_{g(i)})\n= \\lim_{k \\to \\infty} f(x_{g(i_k)})\n\\ge \\lim_{k \\to \\infty} f(x_{i_k})\n= f(x)\n> -\\infty.\n\\end{equation*}\n","start":46602,"end":46962}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":true,"title":"有限偏移子列的反证","source_units":[{"raw_tex":"It remains to prove the fourth statement. From the first statement and because $f$ is bounded from below, $(f(x_{g(i)}))_{i\\in\\N}$ converges to some limit $\\varphi \\in \\R$. Assume, for the sake of contradiction, that $(f(x_i))_{i \\in \\N}$ does not converge to $\\varphi$. Then, there exist $\\rho \\in (0, \\infty)$ and a subsequence $(f(x_{i_j}))_{j \\in \\N}$ contained in $\\R \\setminus [\\varphi-\\rho, \\varphi+\\rho]$. For all $j \\in \\N$, define $p_j \\coloneq g(i_j+l)-i_j \\in \\{0, \\dots, l\\}$. Then, there exist $p \\in \\{0, \\dots, l\\}$ and a subsequence $(p_{j_k})_{k \\in \\N}$ such that, for all $k \\in \\N$, $p_{j_k} = p$. By \\cite[(27)]{KanzowMehlitz} or \\cite[(A.9)]{JiaEtAl}, $(f(x_{g(i)-p}))_{i \\in \\N}$ converges to $\\varphi$. Therefore, $(f(x_{g(i+l)-p}))_{i \\in \\N}$ converges to $\\varphi$. Hence, $(f(x_{g(i_{j_k}+l)-p}))_{k \\in \\N}$ converges to $\\varphi$. This is a contradiction since, for all $k \\in \\N$, $f(x_{g(i_{j_k}+l)-p}) = f(x_{i_{j_k}})$.\n\\end{proof}\n","start":46962,"end":47929}]},{"id":"S029","semantic_type":"proposition","kind":"proposition","labels":["prop:ProjectedTranslation"],"declared_external":false,"source_only":false,"title":"投影平移的位移界","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:ProjectedTranslation}\nFor all $x \\in C$, $v \\in \\mathcal{E}$, and $y \\in \\proj{C}{x-v}$,\n\\begin{align}\n\\label{eq:ProjectedTranslationDistance}\n\\norm{y-x} &\\le 2 \\norm{v},\\\\\n","start":19632,"end":19837},{"raw_tex":"and the inequalities are strict if $x \\notin \\proj{C}{x-v}$.\n\\end{proposition}","start":19918,"end":19996}]},{"id":"S030","semantic_type":"proposition","kind":"proposition","labels":["prop:ProjectedTranslation"],"declared_external":false,"source_only":false,"title":"投影平移的内积界","source_units":[{"raw_tex":"For all $x \\in C$, $v \\in \\mathcal{E}$, and $y \\in \\proj{C}{x-v}$,\n","start":19686,"end":19753},{"raw_tex":"\\label{eq:ProjectedTranslationIP}\n2 \\ip{v}{y-x} &\\le - \\norm{y-x}^2,\n\\end{align}\nand the inequalities are strict if $x \\notin \\proj{C}{x-v}$.\n","start":19837,"end":19979}]},{"id":"S031","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"投影平移命题的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nBy definition of the projection, $\\norm{y-(x-v)} \\le \\norm{x-(x-v)} = \\norm{v}$ and the inequality is strict if $x \\notin \\proj{C}{x-v}$. Thus, on the one hand,\n\\begin{equation*}\n\\norm{y-x}\n= \\norm{y-(x-v)-v}\n\\le \\norm{y-(x-v)} + \\norm{-v}\n\\le \\norm{v} + \\norm{v}\n= 2 \\norm{v},\n\\end{equation*}\nand, on the other hand, $\\norm{y-(x-v)}^2 \\le \\norm{v}^2$, which is equivalent to~\\eqref{eq:ProjectedTranslationIP}.\n\\end{proof}","start":19998,"end":20434}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"投影比较与三角不等式","source_units":[{"raw_tex":"\\begin{proof}\nBy definition of the projection, $\\norm{y-(x-v)} \\le \\norm{x-(x-v)} = \\norm{v}$ and the inequality is strict if $x \\notin \\proj{C}{x-v}$. Thus, on the one hand,\n\\begin{equation*}\n\\norm{y-x}\n= \\norm{y-(x-v)-v}\n\\le \\norm{y-(x-v)} + \\norm{-v}\n\\le \\norm{v} + \\norm{v}\n= 2 \\norm{v},\n\\end{equation*}\n","start":19998,"end":20306}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"展开投影距离的平方","source_units":[{"raw_tex":"and, on the other hand, $\\norm{y-(x-v)}^2 \\le \\norm{v}^2$, which is equivalent to~\\eqref{eq:ProjectedTranslationIP}.\n\\end{proof}","start":20306,"end":20434}]},{"id":"S034","semantic_type":"proposition","kind":"proposition","labels":["prop:NonmonotonePGDAverage"],"declared_external":false,"source_only":false,"title":"平均规则的次水平包含","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:NonmonotonePGDAverage}\nLet $(x_i)_{i \\in \\N}$ be a sequence generated by $\\pgd$ (Algorithm~\\ref{algo:PGD}) using the ``average'' rule. Then:\n\\begin{enumerate}\n\\item $(x_i)_{i \\in \\N}$ is contained in the sublevel set~\\eqref{eq:InitialSublevelSet};\n","start":47930,"end":48210},{"raw_tex":"\\end{enumerate}\n\\end{proposition}","start":48358,"end":48391}]},{"id":"S035","semantic_type":"proposition","kind":"proposition","labels":["prop:NonmonotonePGDAverage"],"declared_external":false,"source_only":false,"title":"平均参考值与函数值同极限","source_units":[{"raw_tex":"\\item if  $(x_i)_{i \\in \\N}$ has an accumulation point, then $(f(x_i))_{i \\in \\N}$ and $(\\mu_i)_{i\\in\\N}$ converge, toward the same (finite) value.\n","start":48210,"end":48358}]},{"id":"S036","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"平均规则命题的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nThe sequence $(\\mu_i)_{i\\in\\N}$ is monotonically nonincreasing since, for all $i \\in \\N$, $f(x_i) \\le \\mu_{i-1}$, hence $\\mu_i=(1-p)\\mu_{i-1}+ pf(x_i) \\le \\mu_{i-1}$. Therefore, for all $i \\in \\N$,\n\\begin{equation*}\nf(x_i) \\le \\mu_{i-1} \\le \\mu_{-1} = f(x_0),\n\\end{equation*}\nmeaning that $(x_i)_{i\\in\\N}$ is contained in the sublevel set~\\eqref{eq:InitialSublevelSet}.\n\nLet us prove the second item of the proposition. Assume that $(x_i)_{i\\in\\N}$ has an accumulation point $x$. Let $(x_{i_k})_{k\\in\\N}$ be a subsequence converging to~$x$. Observe that\n\\begin{equation*}\n\\lim_{k \\to \\infty} f(x_{i_k}) = f(x),\n\\end{equation*}\nsince $f$ is differentiable, and in particular continuous, at $x$.\nAs $(\\mu_i)_{i\\in\\N}$ is monotonically nonincreasing, it has a limit $\\varphi \\in \\R \\cup \\{-\\infty\\}$. For all $k \\in \\N$,\n\\begin{equation*}\nf(x_{i_k}) \\le \\mu_{i_k-1},\n\\end{equation*}\nand letting $k$ tend to infinity yields\n\\begin{equation*}\nf(x) = \\lim_{k\\to\\infty} f(x_{i_k}) \\le \\lim_{k\\to\\infty} \\mu_{i_k-1} = \\varphi.\n\\end{equation*}\nIn particular, $\\varphi$ is finite.\n\nLet us show that $\\varphi = \\liminf_{i\\to\\infty} f(x_i)$. Let $(x_{j_k})_{k\\in\\N}$ be a subsequence such that\n\\begin{equation*}\n\\lim_{k \\to \\infty} f(x_{j_k}) = \\liminf_{i\\to\\infty} f(x_i).\n\\end{equation*}\nFor all $k \\in \\N$,\n\\begin{align*}\n\\mu_{j_k} = (1-p) \\mu_{j_k-1} + p f(x_{j_k}),\n\\end{align*}\nand letting $k$ tend to infinity yields\n\\begin{equation*}\n\\varphi = (1-p) \\varphi + p \\liminf_{i\\to\\infty} f(x_i).\n\\end{equation*}\nAs $p>0$, this implies $\\varphi = \\liminf_{i\\to\\infty} f(x_i)$ (and, in particular, $\\liminf_{i\\to\\infty} f(x_i) > -\\infty$). To conclude, we observe that, for all $k \\in \\N$,\n\\begin{equation*}\nf(x_k) \\le \\mu_{k-1},\n\\end{equation*}\nhence\n\\begin{equation*}\n\\limsup_{k\\to\\infty} f(x_k) \n\\leq \\lim_{k\\to\\infty} \\mu_{k-1} = \\varphi = \\liminf_{k\\to\\infty} f(x_k).\n\\end{equation*}\nTherefore, $(f(x_k))_{k\\in\\N}$ converges to $\\varphi$.\n\\end{proof}","start":48393,"end":50351}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"参考值单调与次水平包含","source_units":[{"raw_tex":"\\begin{proof}\nThe sequence $(\\mu_i)_{i\\in\\N}$ is monotonically nonincreasing since, for all $i \\in \\N$, $f(x_i) \\le \\mu_{i-1}$, hence $\\mu_i=(1-p)\\mu_{i-1}+ pf(x_i) \\le \\mu_{i-1}$. Therefore, for all $i \\in \\N$,\n\\begin{equation*}\nf(x_i) \\le \\mu_{i-1} \\le \\mu_{-1} = f(x_0),\n\\end{equation*}\nmeaning that $(x_i)_{i\\in\\N}$ is contained in the sublevel set~\\eqref{eq:InitialSublevelSet}.\n","start":48393,"end":48777}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"聚点保证参考极限有限","source_units":[{"raw_tex":"Let us prove the second item of the proposition. Assume that $(x_i)_{i\\in\\N}$ has an accumulation point $x$. Let $(x_{i_k})_{k\\in\\N}$ be a subsequence converging to~$x$. Observe that\n\\begin{equation*}\n\\lim_{k \\to \\infty} f(x_{i_k}) = f(x),\n\\end{equation*}\nsince $f$ is differentiable, and in particular continuous, at $x$.\nAs $(\\mu_i)_{i\\in\\N}$ is monotonically nonincreasing, it has a limit $\\varphi \\in \\R \\cup \\{-\\infty\\}$. For all $k \\in \\N$,\n\\begin{equation*}\nf(x_{i_k}) \\le \\mu_{i_k-1},\n\\end{equation*}\nand letting $k$ tend to infinity yields\n\\begin{equation*}\nf(x) = \\lim_{k\\to\\infty} f(x_{i_k}) \\le \\lim_{k\\to\\infty} \\mu_{i_k-1} = \\varphi.\n\\end{equation*}\nIn particular, $\\varphi$ is finite.\n","start":48778,"end":49478}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"平均递推识别函数值下极限","source_units":[{"raw_tex":"Let us show that $\\varphi = \\liminf_{i\\to\\infty} f(x_i)$. Let $(x_{j_k})_{k\\in\\N}$ be a subsequence such that\n\\begin{equation*}\n\\lim_{k \\to \\infty} f(x_{j_k}) = \\liminf_{i\\to\\infty} f(x_i).\n\\end{equation*}\nFor all $k \\in \\N$,\n\\begin{align*}\n\\mu_{j_k} = (1-p) \\mu_{j_k-1} + p f(x_{j_k}),\n\\end{align*}\nand letting $k$ tend to infinity yields\n\\begin{equation*}\n\\varphi = (1-p) \\varphi + p \\liminf_{i\\to\\infty} f(x_i).\n\\end{equation*}\nAs $p>0$, this implies $\\varphi = \\liminf_{i\\to\\infty} f(x_i)$ (and, in particular, $\\liminf_{i\\to\\infty} f(x_i) > -\\infty$). To conclude, we observe that, for all $k \\in \\N$,\n","start":49479,"end":50086}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"上下极限夹逼","source_units":[{"raw_tex":"\\begin{equation*}\nf(x_k) \\le \\mu_{k-1},\n\\end{equation*}\nhence\n\\begin{equation*}\n\\limsup_{k\\to\\infty} f(x_k) \n\\leq \\lim_{k\\to\\infty} \\mu_{k-1} = \\varphi = \\liminf_{k\\to\\infty} f(x_k).\n\\end{equation*}\nTherefore, $(f(x_k))_{k\\in\\N}$ converges to $\\varphi$.\n\\end{proof}","start":50086,"end":50351}]},{"id":"S041","semantic_type":"assumption","kind":"assumption","labels":["eq:def_w","prop:ArbitrarySmallAlpha"],"declared_external":false,"source_only":false,"title":"下降切向量与收缩常数","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:ArbitrarySmallAlpha}\nAssume that $f$ satisfies~\\ref{it:continuously_diff}. Let $\\ushort{x} \\in C$ be non-B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}, and $w \\in \\tancone{C}{\\ushort{x}}$ be such that\n\\begin{equation}\n\\label{eq:def_w}\n\\ip{w}{-\\nabla f(\\ushort{x})} > 0.\n\\end{equation}\nLet $\\beta \\in (0, 1)$. Define $\\kappa \\coloneq \\sqrt{1 - \\frac{\\beta \\ip{w}{-\\nabla f(\\ushort{x})}^2}{8 \\norm{w}^2 \\norm{\\nabla f(\\ushort{x})}^2}} \\in (0, 1)$.\nFor every $\\varepsilon \\in (0, \\infty)$, there exist $\\alpha_{\\ushort{x}} \\in (0, \\varepsilon]$ and $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\infty)$ such that, for all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$ and $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$,\n","start":53427,"end":54220}]},{"id":"S042","semantic_type":"proposition","kind":"proposition","labels":["prop:ArbitrarySmallAlpha"],"declared_external":false,"source_only":false,"title":"任意小尺度的投影距离收缩","source_units":[{"raw_tex":"For every $\\varepsilon \\in (0, \\infty)$, there exist $\\alpha_{\\ushort{x}} \\in (0, \\varepsilon]$ and $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\infty)$ such that, for all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$ and $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$,\n\\begin{equation*}\n\\dist(x-\\alpha\\nabla f(x), C) \\le \\kappa \\alpha \\norm{\\nabla f(x)},\n\\end{equation*}\n","start":53912,"end":54322}]},{"id":"S043","semantic_type":"proposition","kind":"proposition","labels":["prop:ArbitrarySmallAlpha"],"declared_external":false,"source_only":false,"title":"投影更新的统一下降夹角","source_units":[{"raw_tex":"which implies, for all $y \\in\\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{equation*}\n\\ip{\\nabla f(x)}{y-x} \\le - \\sqrt{1-\\kappa^2} \\norm{\\nabla f(x)} \\norm{y-x}.\n\\end{equation*}\n\\end{proposition}","start":54322,"end":54511}]},{"id":"S044","semantic_type":"proof","kind":"proof","labels":["eq:def_i_1","eq:def_i_2","eq:def_i_3","eq:def_rho0_1","eq:def_rho0_2","eq:def_rho0_3","eq:ip_xy_nabla"],"declared_external":false,"source_only":false,"title":"小尺度投影估计的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $\\varepsilon \\in (0, \\infty)$ be fixed. We show that there exist $\\alpha_{\\ushort{x}} \\in (0, \\varepsilon]$ and $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\infty)$ satisfying the required property.\n\nLet $(w_i)_{i\\in\\N}$ be a sequence in $C$ converging to $\\ushort{x}$, and $(t_i)_{i\\in\\N}$ be a sequence in $(0, \\infty)$ such that\n\\begin{equation*}\n\\frac{w_i - \\ushort{x}}{t_i} \\xrightarrow{i\\to\\infty} w.\n\\end{equation*}\nFrom the definition of $w$ in~\\eqref{eq:def_w}, it holds for all $i \\in \\N$ large enough that\n\\begin{equation}\n\\label{eq:def_i_1}\n\\ip{w_i - \\ushort{x}}{-\\nabla f(\\ushort{x})} > 0.\n\\end{equation}\nAs $\\frac{1}{t_i}\\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}} \\xrightarrow{i\\to\\infty} \\frac{\\norm{w}^2}{\\ip{w}{-\\nabla f(\\ushort{x})}}$ and $t_i \\xrightarrow{i\\to\\infty} 0$, it also holds for all $i \\in \\N$ large enough that\n\\begin{equation}\n\\label{eq:def_i_2}\n\\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}} < \\varepsilon.\n\\end{equation}\nSimilarly, it holds for all $i \\in \\N$ large enough that\n\\begin{equation}\n\\label{eq:def_i_3}\n\\frac{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}^2}{\\norm{w_i-\\ushort{x}}^2}\n> \\frac{\\ip{w}{-\\nabla f(\\ushort{x})}^2}{2\\norm{w}^2 }.\n\\end{equation}\nFix $i \\in \\N$ satisfying \\eqref{eq:def_i_1}, \\eqref{eq:def_i_2}, and \\eqref{eq:def_i_3}.\nPick $\\alpha_{\\ushort{x}}$ such that\n\\begin{equation*}\n\\frac{\\alpha_{\\ushort{x}}}{2}\n< \\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}}\n< \\alpha_{\\ushort{x}}\n< \\varepsilon.\n\\end{equation*}\nSince $\\nabla f$ is continuous at $\\ushort{x}$, there exists $\\rho_0 \\in (0, \\infty)$ such that, for all $x \\in \\ball[\\ushort{x}, \\rho_0] \\cap C$,\n\\begin{subequations}\n\\begin{gather}\n\\label{eq:def_rho0_1}\n\\ip{w_i-\\ushort{x}}{-\\nabla f(x)} > 0,\\\\\n\\label{eq:def_rho0_2}\n\\frac{\\alpha_{\\ushort{x}}}{2}\n< \\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(x)}}\n< \\alpha_{\\ushort{x}},\\\\\n\\label{eq:def_rho0_3}\n\\frac{\\ip{w_i-\\ushort{x}}{-\\nabla f(x)}^2}{\\norm{w_i-\\ushort{x}}^2 \\norm{\\nabla f(x)}^2}\n> \\frac{\\ip{w}{-\\nabla f(\\ushort{x})}^2}{2\\norm{w}^2 \\norm{\\nabla f(\\ushort{x})}^2}.\n\\end{gather}\n\\end{subequations}\nWe now establish the first inequality we have to prove: for an adequate value of $\\oshort{\\rho}(\\alpha_{\\ushort{x}})$, it holds for all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$ and $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$ that\n\\begin{equation*}\n\\norm{x - \\alpha \\nabla f(x) - y} \\le \\kappa \\alpha \\norm{\\nabla f(x)}\\quad\n\\forall y \\in \\proj{C}{x-\\alpha\\nabla f(x)},\n\\end{equation*}\nwhich is equivalent to $\\dist(x-\\alpha\\nabla f(x), C) \\le \\kappa \\alpha \\norm{\\nabla f(x)}$.\n\nFor the moment, let $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\rho_0]$ be arbitrary. For all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{align*}\n\\norm{x - \\alpha \\nabla f(x) - y}^2\n&\\le \\norm{x - \\alpha \\nabla f(x) - w_i}^2 \\\\\n&= \\norm{\\ushort{x}-\\alpha \\nabla f(x) - w_i}^2 + 2 \\ip{\\ushort{x}-x}{\\alpha \\nabla f(x) + w_i-\\ushort{x}} + \\norm{\\ushort{x}-x}^2 \\\\\n&\\le \\norm{\\ushort{x}-\\alpha \\nabla f(x) - w_i}^2 \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\alpha \\norm{\\nabla f(x)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&\\le \\norm{\\ushort{x}-\\alpha \\nabla f(x) - w_i}^2 \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\alpha \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&= \\alpha^2 \\norm{\\nabla f(x)}^2 -2 \\alpha \\ip{w_i-\\ushort{x}}{-\\nabla f(x)} + \\norm{w_i-\\ushort{x}}^2 \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\alpha \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 - \\alpha \\ip{w_i-\\ushort{x}}{-\\nabla f(x)} \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\frac{\\alpha_{\\ushort{x}}}{\\beta} \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right)\n+ \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2,\n\\end{align*}\nwhere the last inequality follows from \\eqref{eq:def_rho0_2} and the fact that $\\alpha_{\\ushort{x}} \\le \\alpha \\le \\frac{\\alpha_{\\ushort{x}}}{\\beta}$.\nNow, choose $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\rho_0]$ small enough to ensure\n\\begin{align*}\n&2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\frac{\\alpha_{\\ushort{x}}}{\\beta} \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&\\le \\frac{\\alpha_{\\ushort{x}}}{2} \\min_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\ip{w_i-\\ushort{x}}{-\\nabla f(z)}.\n\\end{align*}\nNote that the right-hand side of this inequality is positive, from \\eqref{eq:def_rho0_1}.\nCombining this definition with the previous inequality, we arrive at\n\\begin{align*}\n\\norm{x - \\alpha \\nabla f(x) - y}^2\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 - \\frac{\\alpha}{2} \\ip{w_i-\\ushort{x}}{-\\nabla f(x)}\\\\\n&= \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\ip{w_i-\\ushort{x}}{-\\nabla f(x)}}{2 \\alpha \\norm{\\nabla f(x)}^2}\\right)\\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\beta \\ip{w_i-\\ushort{x}}{-\\nabla f(x)}}{2 \\alpha_{\\ushort{x}} \\norm{\\nabla f(x)}^2}\\right)\n\\text{ as $\\alpha \\le \\frac{\\alpha_{\\ushort{x}}}{\\beta}$}\\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\beta \\ip{w_i-\\ushort{x}}{-\\nabla f(x)}^2}{4 \\norm{w_i-\\ushort{x}}^2 \\norm{\\nabla f(x)}^2}\\right)\n\\text{ from~\\eqref{eq:def_rho0_2}}\\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\beta \\ip{w}{-\\nabla f(\\ushort{x})}^2}{8 \\norm{w}^2 \\norm{\\nabla f(\\ushort{x})}^2}\\right)\n\\text{ from~\\eqref{eq:def_rho0_3}}\\\\\n&= \\kappa^2 \\alpha^2 \\norm{\\nabla f(x)}^2.\n\\end{align*}\nThus, for all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{equation*}\n\\norm{x - \\alpha \\nabla f(x) - y} \\le \\kappa \\alpha \\norm{\\nabla f(x)}.\n\\end{equation*}\nTo conclude, we show that this inequality implies\n\\begin{equation}\n\\label{eq:ip_xy_nabla}\n\\ip{\\frac{y-x}{\\norm{y-x}}}{\\frac{\\nabla f(x)}{\\norm{\\nabla f(x)}}} \\le - \\sqrt{1-\\kappa^2}.\n\\end{equation}\nIndeed, defining $\\theta \\in \\R$ such that $\\ip{\\frac{y-x}{\\norm{y-x}}}{\\frac{\\nabla f(x)}{\\norm{\\nabla f(x)}}} = \\cos(\\theta)$, we have\n\\begin{equation*}\n\\norm{y-x}^2 + 2 \\alpha \\norm{\\nabla f(x)} \\norm{y-x} \\cos(\\theta) + \\alpha^2 \\norm{\\nabla f(x)}^2\n\\le \\alpha^2 \\kappa^2 \\norm{\\nabla f(x)}^2.\n\\end{equation*}\nThis already shows that $\\cos(\\theta) < 0$. In addition, minimizing the left-hand side over all possible values of $\\norm{y-x}$, we get\n\\begin{equation*}\n- \\alpha^2 \\norm{\\nabla f(x)}^2 \\cos^2(\\theta) + \\alpha^2 \\norm{\\nabla f(x)}^2\n\\le \\alpha^2 \\kappa^2 \\norm{\\nabla f(x)}^2,\n\\end{equation*}\nhence $\\cos^2(\\theta) \\ge 1 - \\kappa^2$, which establishes \\eqref{eq:ip_xy_nabla}.\n\\end{proof}","start":54513,"end":61656}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["eq:def_i_1","eq:def_i_2","eq:def_i_3"],"declared_external":false,"source_only":false,"title":"选取满足三项估计的切向见证","source_units":[{"raw_tex":"\\begin{proof}\nLet $\\varepsilon \\in (0, \\infty)$ be fixed. We show that there exist $\\alpha_{\\ushort{x}} \\in (0, \\varepsilon]$ and $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\infty)$ satisfying the required property.\n\nLet $(w_i)_{i\\in\\N}$ be a sequence in $C$ converging to $\\ushort{x}$, and $(t_i)_{i\\in\\N}$ be a sequence in $(0, \\infty)$ such that\n\\begin{equation*}\n\\frac{w_i - \\ushort{x}}{t_i} \\xrightarrow{i\\to\\infty} w.\n\\end{equation*}\nFrom the definition of $w$ in~\\eqref{eq:def_w}, it holds for all $i \\in \\N$ large enough that\n\\begin{equation}\n\\label{eq:def_i_1}\n\\ip{w_i - \\ushort{x}}{-\\nabla f(\\ushort{x})} > 0.\n\\end{equation}\nAs $\\frac{1}{t_i}\\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}} \\xrightarrow{i\\to\\infty} \\frac{\\norm{w}^2}{\\ip{w}{-\\nabla f(\\ushort{x})}}$ and $t_i \\xrightarrow{i\\to\\infty} 0$, it also holds for all $i \\in \\N$ large enough that\n\\begin{equation}\n\\label{eq:def_i_2}\n\\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}} < \\varepsilon.\n\\end{equation}\nSimilarly, it holds for all $i \\in \\N$ large enough that\n\\begin{equation}\n\\label{eq:def_i_3}\n\\frac{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}^2}{\\norm{w_i-\\ushort{x}}^2}\n> \\frac{\\ip{w}{-\\nabla f(\\ushort{x})}^2}{2\\norm{w}^2 }.\n\\end{equation}\nFix $i \\in \\N$ satisfying \\eqref{eq:def_i_1}, \\eqref{eq:def_i_2}, and \\eqref{eq:def_i_3}.\n","start":54513,"end":55877}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"在见证比值附近选择步长","source_units":[{"raw_tex":"Pick $\\alpha_{\\ushort{x}}$ such that\n\\begin{equation*}\n\\frac{\\alpha_{\\ushort{x}}}{2}\n< \\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}}\n< \\alpha_{\\ushort{x}}\n< \\varepsilon.\n\\end{equation*}\n","start":55877,"end":56092}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":["eq:def_rho0_1","eq:def_rho0_2","eq:def_rho0_3"],"declared_external":false,"source_only":false,"title":"连续性保持三项严格估计","source_units":[{"raw_tex":"Since $\\nabla f$ is continuous at $\\ushort{x}$, there exists $\\rho_0 \\in (0, \\infty)$ such that, for all $x \\in \\ball[\\ushort{x}, \\rho_0] \\cap C$,\n\\begin{subequations}\n\\begin{gather}\n\\label{eq:def_rho0_1}\n\\ip{w_i-\\ushort{x}}{-\\nabla f(x)} > 0,\\\\\n\\label{eq:def_rho0_2}\n\\frac{\\alpha_{\\ushort{x}}}{2}\n< \\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(x)}}\n< \\alpha_{\\ushort{x}},\\\\\n\\label{eq:def_rho0_3}\n\\frac{\\ip{w_i-\\ushort{x}}{-\\nabla f(x)}^2}{\\norm{w_i-\\ushort{x}}^2 \\norm{\\nabla f(x)}^2}\n> \\frac{\\ip{w}{-\\nabla f(\\ushort{x})}^2}{2\\norm{w}^2 \\norm{\\nabla f(\\ushort{x})}^2}.\n\\end{gather}\n\\end{subequations}\n","start":56092,"end":56711}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"以可行见证点比较投影距离","source_units":[{"raw_tex":"We now establish the first inequality we have to prove: for an adequate value of $\\oshort{\\rho}(\\alpha_{\\ushort{x}})$, it holds for all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$ and $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$ that\n\\begin{equation*}\n\\norm{x - \\alpha \\nabla f(x) - y} \\le \\kappa \\alpha \\norm{\\nabla f(x)}\\quad\n\\forall y \\in \\proj{C}{x-\\alpha\\nabla f(x)},\n\\end{equation*}\nwhich is equivalent to $\\dist(x-\\alpha\\nabla f(x), C) \\le \\kappa \\alpha \\norm{\\nabla f(x)}$.\n\nFor the moment, let $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\rho_0]$ be arbitrary. For all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{align*}\n\\norm{x - \\alpha \\nabla f(x) - y}^2\n&\\le \\norm{x - \\alpha \\nabla f(x) - w_i}^2 \\\\\n&= \\norm{\\ushort{x}-\\alpha \\nabla f(x) - w_i}^2 + 2 \\ip{\\ushort{x}-x}{\\alpha \\nabla f(x) + w_i-\\ushort{x}} + \\norm{\\ushort{x}-x}^2 \\\\\n&\\le \\norm{\\ushort{x}-\\alpha \\nabla f(x) - w_i}^2 \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\alpha \\norm{\\nabla f(x)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&\\le \\norm{\\ushort{x}-\\alpha \\nabla f(x) - w_i}^2 \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\alpha \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&= \\alpha^2 \\norm{\\nabla f(x)}^2 -2 \\alpha \\ip{w_i-\\ushort{x}}{-\\nabla f(x)} + \\norm{w_i-\\ushort{x}}^2 \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\alpha \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 - \\alpha \\ip{w_i-\\ushort{x}}{-\\nabla f(x)} \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\frac{\\alpha_{\\ushort{x}}}{\\beta} \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right)\n+ \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2,\n\\end{align*}\nwhere the last inequality follows from \\eqref{eq:def_rho0_2} and the fact that $\\alpha_{\\ushort{x}} \\le \\alpha \\le \\frac{\\alpha_{\\ushort{x}}}{\\beta}$.\n","start":56711,"end":58966}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"缩小半径吸收扰动","source_units":[{"raw_tex":"Now, choose $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\rho_0]$ small enough to ensure\n\\begin{align*}\n&2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\frac{\\alpha_{\\ushort{x}}}{\\beta} \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&\\le \\frac{\\alpha_{\\ushort{x}}}{2} \\min_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\ip{w_i-\\ushort{x}}{-\\nabla f(z)}.\n\\end{align*}\nNote that the right-hand side of this inequality is positive, from \\eqref{eq:def_rho0_1}.\n","start":58966,"end":59504}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"得到统一收缩因子","source_units":[{"raw_tex":"Combining this definition with the previous inequality, we arrive at\n\\begin{align*}\n\\norm{x - \\alpha \\nabla f(x) - y}^2\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 - \\frac{\\alpha}{2} \\ip{w_i-\\ushort{x}}{-\\nabla f(x)}\\\\\n&= \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\ip{w_i-\\ushort{x}}{-\\nabla f(x)}}{2 \\alpha \\norm{\\nabla f(x)}^2}\\right)\\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\beta \\ip{w_i-\\ushort{x}}{-\\nabla f(x)}}{2 \\alpha_{\\ushort{x}} \\norm{\\nabla f(x)}^2}\\right)\n\\text{ as $\\alpha \\le \\frac{\\alpha_{\\ushort{x}}}{\\beta}$}\\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\beta \\ip{w_i-\\ushort{x}}{-\\nabla f(x)}^2}{4 \\norm{w_i-\\ushort{x}}^2 \\norm{\\nabla f(x)}^2}\\right)\n\\text{ from~\\eqref{eq:def_rho0_2}}\\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\beta \\ip{w}{-\\nabla f(\\ushort{x})}^2}{8 \\norm{w}^2 \\norm{\\nabla f(\\ushort{x})}^2}\\right)\n\\text{ from~\\eqref{eq:def_rho0_3}}\\\\\n&= \\kappa^2 \\alpha^2 \\norm{\\nabla f(x)}^2.\n\\end{align*}\nThus, for all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{equation*}\n\\norm{x - \\alpha \\nabla f(x) - y} \\le \\kappa \\alpha \\norm{\\nabla f(x)}.\n\\end{equation*}\n","start":59504,"end":60757}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":["eq:ip_xy_nabla"],"declared_external":false,"source_only":false,"title":"从距离收缩推出夹角界","source_units":[{"raw_tex":"To conclude, we show that this inequality implies\n\\begin{equation}\n\\label{eq:ip_xy_nabla}\n\\ip{\\frac{y-x}{\\norm{y-x}}}{\\frac{\\nabla f(x)}{\\norm{\\nabla f(x)}}} \\le - \\sqrt{1-\\kappa^2}.\n\\end{equation}\nIndeed, defining $\\theta \\in \\R$ such that $\\ip{\\frac{y-x}{\\norm{y-x}}}{\\frac{\\nabla f(x)}{\\norm{\\nabla f(x)}}} = \\cos(\\theta)$, we have\n\\begin{equation*}\n\\norm{y-x}^2 + 2 \\alpha \\norm{\\nabla f(x)} \\norm{y-x} \\cos(\\theta) + \\alpha^2 \\norm{\\nabla f(x)}^2\n\\le \\alpha^2 \\kappa^2 \\norm{\\nabla f(x)}^2.\n\\end{equation*}\nThis already shows that $\\cos(\\theta) < 0$. In addition, minimizing the left-hand side over all possible values of $\\norm{y-x}$, we get\n\\begin{equation*}\n- \\alpha^2 \\norm{\\nabla f(x)}^2 \\cos^2(\\theta) + \\alpha^2 \\norm{\\nabla f(x)}^2\n\\le \\alpha^2 \\kappa^2 \\norm{\\nabla f(x)}^2,\n\\end{equation*}\nhence $\\cos^2(\\theta) \\ge 1 - \\kappa^2$, which establishes \\eqref{eq:ip_xy_nabla}.\n\\end{proof}","start":60757,"end":61656}]},{"id":"S052","semantic_type":"proposition","kind":"proposition","labels":["prop:AcceptedStep"],"declared_external":false,"source_only":false,"title":"非 B 驻点附近的可接受尺度","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:AcceptedStep}\nLet $\\ushort{\\alpha} \\in (0, \\infty)$ and $\\beta, c \\in (0, 1)$.\nAssume that $f$ satisfies~\\ref{it:continuously_diff}. Let $\\ushort{x} \\in C$ be non-B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}.\nThere exist $\\alpha_{\\ushort{x}} \\in (0, \\ushort{\\alpha}]$ and $\\rho \\in (0, \\infty)$ such that, for all $x \\in \\ball(\\ushort{x}, \\rho) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{equation*}\nf(y) < f(x) + c \\ip{\\nabla f(x)}{y-x}.\n\\end{equation*}\n\\end{proposition}","start":61658,"end":62248}]},{"id":"S053","semantic_type":"proof","kind":"proof","labels":["eq:UpperBoundfDiffGrad","eq:def_alpha_x_small1","eq:def_alpha_x_small2","eq:nablas_ushortx_x"],"declared_external":false,"source_only":false,"title":"可接受尺度命题的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nDefine $\\kappa$ as in Proposition~\\ref{prop:ArbitrarySmallAlpha}. Let $\\delta \\in (0, \\infty)$ be small enough to ensure\n\\begin{subequations}\n\\begin{align}\n\\label{eq:def_alpha_x_small1}\n\\sup_{y \\in \\ball\\left[\\ushort{x}, \\frac{7\\delta}{2\\beta}\\norm{\\nabla f(\\ushort{x})}\\right] \\cap C \\setminus \\{\\ushort{x}\\}}\n\\frac{|f(y)-f(\\ushort{x}) - \\ip{\\nabla f(\\ushort{x})}{y-\\ushort{x}}|}{\\norm{y-\\ushort{x}}}\n&< \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4\\left(1 + \\frac{8}{3(1-\\kappa)}\\right)},\\\\\n\\label{eq:def_alpha_x_small2}\n\\sup_{y \\in \\ball\\left[\\ushort{x}, \\frac{7\\delta}{2\\beta}\\norm{\\nabla f(\\ushort{x})}\\right] \\cap C}\n\\norm{\\nabla f(y)-\\nabla f(\\ushort{x})}\n&< \\frac{(1-c)\\sqrt{1-\\kappa^2}}{4} \\norm{\\nabla f(\\ushort{x})}.\n\\end{align}\n\\end{subequations}\nThese inequalities are satisfied by all $\\delta$ small enough, from the definition of the gradient for the first one, and because the gradient is continuous at $\\ushort{x}$ for the second one.\n\nThen, define $\\varepsilon \\coloneq \\min\\left\\{\\ushort{\\alpha}, \\delta\\right\\}$ and let $\\alpha_{\\ushort{x}} \\in (0, \\varepsilon]$ and $\\oshort{\\rho}(\\alpha_{\\ushort{x}})\\in(0,\\infty)$ be given by Proposition~\\ref{prop:ArbitrarySmallAlpha}.\nDefine\n\\begin{equation*}\n\\rho \\coloneq \\min\\left\\{\\oshort{\\rho}(\\alpha_{\\ushort{x}}), \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})}\\right\\}.\n\\end{equation*}\nNote that, for all $x \\in \\ball(\\ushort{x}, \\rho) \\cap C$,\n\\begin{equation*}\n\\norm{x-\\ushort{x}}\n< \\rho \\le \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})}\n< \\frac{7\\alpha_{\\ushort{x}}}{2\\beta} \\norm{\\nabla f(\\ushort{x})}\n\\leq \\frac{7\\delta }{2\\beta} \\norm{\\nabla f(\\ushort{x})},\n\\end{equation*}\nso that from~\\eqref{eq:def_alpha_x_small2}, $\\norm{\\nabla f(x)-\\nabla f(\\ushort{x})} < \\frac{\\norm{\\nabla f(\\ushort{x})}}{4}$, which implies\n\\begin{align}\n\\nonumber\n\\frac{3}{4} \\norm{\\nabla f(\\ushort{x})}\n& < \\norm{\\nabla f(\\ushort{x})}-\\norm{\\nabla f(x)-\\nabla f(\\ushort{x})} \\\\\n& \\le \\norm{\\nabla f(x)} \\nonumber\\\\\n& \\le \\norm{\\nabla f(\\ushort{x})}+\\norm{\\nabla f(x)-\\nabla f(\\ushort{x})} \\nonumber\\\\\n& < \\frac{5}{4} \\norm{\\nabla f(\\ushort{x})}.\n\\label{eq:nablas_ushortx_x}\n\\end{align}\nFor all $x \\in \\ball(\\ushort{x}, \\rho) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{align}\nf(y)\n&= f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x} \\nonumber\\\\\n&\\qquad + \\left(f(\\ushort{x}) - f(x) - \\ip{\\nabla f(\\ushort{x})}{\\ushort{x}-x}\\right) \\nonumber\\\\\n&\\qquad + \\left(f(y) - f(\\ushort{x}) - \\ip{\\nabla f(\\ushort{x})}{y-\\ushort{x}}\\right) \\nonumber\\\\\n&\\le f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x} +\n\\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4\\left(1 + \\frac{8}{3(1-\\kappa)}\\right)} \\left(\\norm{\\ushort{x}-x} + \\norm{y-\\ushort{x}}\\right),\n\\label{eq:UpperBoundfDiffGrad}\n\\end{align}\nwhere the inequality follows from \\eqref{eq:def_alpha_x_small1}. Observe that\n\\begin{align*}\n\\norm{y-\\ushort{x}}\n&\\le \\norm{y-x} + \\norm{x-\\ushort{x}} \\\\\n&\\le 2 \\alpha \\norm{\\nabla f(x)} + \\rho\n\\text{ from \\eqref{eq:ProjectedTranslationDistance}}\\\\\n&\\le \\frac{2 \\alpha_{\\ushort{x}}}{\\beta} \\norm{\\nabla f(x)}\n+ \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})} \\\\\n&< \\frac{5\\alpha_{\\ushort{x}}}{2\\beta} \\norm{\\nabla f(\\ushort{x})}\n+ \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})} \n\\text{ from \\eqref{eq:nablas_ushortx_x}} \\\\\n&\\le \\frac{7\\alpha_{\\ushort{x}}}{2\\beta} \\norm{\\nabla f(\\ushort{x})}.\n\\end{align*}\nWe continue from \\eqref{eq:UpperBoundfDiffGrad}:\n\\begin{align*}\nf(y)\n& \\le f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x} +\n\\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{\n4\\left(1 + \\frac{8}{3(1-\\kappa)}\\right)} \\left(2\\norm{\\ushort{x}-x} + \\norm{y-x}\\right) \\\\\n&\\overset{\\text{(a)}}{<} f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x}\n+ \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4} \\norm{y-x} \\\\\n&\\le f(x) + \\ip{\\nabla f(x)}{y-x} + \\norm{\\nabla f(\\ushort{x})-\\nabla f(x)} \\norm{y-x} \\\\\n&\\qquad + \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4} \\norm{y-x} \\\\\n&\\le f(x) + \\ip{\\nabla f(x)}{y-x} + \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{2} \\norm{y-x}\n\\text{ from~\\eqref{eq:def_alpha_x_small2}}\\\\\n&< f(x) + \\ip{\\nabla f(x)}{y-x} + (1-c)\\sqrt{1-\\kappa^2} \\norm{\\nabla f(x)}\\norm{y-x}\n\\text{ from~\\eqref{eq:nablas_ushortx_x}}\\\\\n&\\le f(x) + \\ip{\\nabla f(x)}{y-x} - (1-c) \\ip{\\nabla f(x)}{y-x}\n\\text{ from Proposition~\\ref{prop:ArbitrarySmallAlpha}}\\\\\n&= f(x) + c \\ip{\\nabla f(x)}{y-x}.\n\\end{align*}\nInequality (a) is true because\n\\begin{align*}\n\\norm{y-x}\n&\\ge \\alpha \\norm{\\nabla f(x)} - \\norm{x-\\alpha \\nabla f(x)-y}\n\\text{ by the triangle inequality}\\\\\n&= \\alpha \\norm{\\nabla f(x)} - \\dist(x-\\alpha\\nabla f(x), C) \\\\\n&\\ge (1-\\kappa) \\alpha \\norm{\\nabla f(x)}\n\\text{ from Proposition~\\ref{prop:ArbitrarySmallAlpha}}\\\\\n&\\ge \\frac{3}{4} (1-\\kappa) \\rho\n\\text{ from~\\eqref{eq:nablas_ushortx_x}, the definition of $\\rho$, and $\\alpha_{\\ushort{x}} \\le \\alpha$}\\\\\n&> \\frac{3}{4} (1-\\kappa) \\norm{x-\\ushort{x}}.\n\\end{align*}\n\\end{proof}","start":62250,"end":67244}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":["eq:def_alpha_x_small1","eq:def_alpha_x_small2"],"declared_external":false,"source_only":false,"title":"同时控制余项和梯度变化","source_units":[{"raw_tex":"\\begin{proof}\nDefine $\\kappa$ as in Proposition~\\ref{prop:ArbitrarySmallAlpha}. Let $\\delta \\in (0, \\infty)$ be small enough to ensure\n\\begin{subequations}\n\\begin{align}\n\\label{eq:def_alpha_x_small1}\n\\sup_{y \\in \\ball\\left[\\ushort{x}, \\frac{7\\delta}{2\\beta}\\norm{\\nabla f(\\ushort{x})}\\right] \\cap C \\setminus \\{\\ushort{x}\\}}\n\\frac{|f(y)-f(\\ushort{x}) - \\ip{\\nabla f(\\ushort{x})}{y-\\ushort{x}}|}{\\norm{y-\\ushort{x}}}\n&< \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4\\left(1 + \\frac{8}{3(1-\\kappa)}\\right)},\\\\\n\\label{eq:def_alpha_x_small2}\n\\sup_{y \\in \\ball\\left[\\ushort{x}, \\frac{7\\delta}{2\\beta}\\norm{\\nabla f(\\ushort{x})}\\right] \\cap C}\n\\norm{\\nabla f(y)-\\nabla f(\\ushort{x})}\n&< \\frac{(1-c)\\sqrt{1-\\kappa^2}}{4} \\norm{\\nabla f(\\ushort{x})}.\n\\end{align}\n\\end{subequations}\nThese inequalities are satisfied by all $\\delta$ small enough, from the definition of the gradient for the first one, and because the gradient is continuous at $\\ushort{x}$ for the second one.\n","start":62250,"end":63229}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"匹配小尺度与最终邻域","source_units":[{"raw_tex":"Then, define $\\varepsilon \\coloneq \\min\\left\\{\\ushort{\\alpha}, \\delta\\right\\}$ and let $\\alpha_{\\ushort{x}} \\in (0, \\varepsilon]$ and $\\oshort{\\rho}(\\alpha_{\\ushort{x}})\\in(0,\\infty)$ be given by Proposition~\\ref{prop:ArbitrarySmallAlpha}.\nDefine\n\\begin{equation*}\n\\rho \\coloneq \\min\\left\\{\\oshort{\\rho}(\\alpha_{\\ushort{x}}), \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})}\\right\\}.\n\\end{equation*}\n","start":63230,"end":63629}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":["eq:nablas_ushortx_x"],"declared_external":false,"source_only":false,"title":"邻域内梯度范数可比较","source_units":[{"raw_tex":"Note that, for all $x \\in \\ball(\\ushort{x}, \\rho) \\cap C$,\n\\begin{equation*}\n\\norm{x-\\ushort{x}}\n< \\rho \\le \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})}\n< \\frac{7\\alpha_{\\ushort{x}}}{2\\beta} \\norm{\\nabla f(\\ushort{x})}\n\\leq \\frac{7\\delta }{2\\beta} \\norm{\\nabla f(\\ushort{x})},\n\\end{equation*}\nso that from~\\eqref{eq:def_alpha_x_small2}, $\\norm{\\nabla f(x)-\\nabla f(\\ushort{x})} < \\frac{\\norm{\\nabla f(\\ushort{x})}}{4}$, which implies\n\\begin{align}\n\\nonumber\n\\frac{3}{4} \\norm{\\nabla f(\\ushort{x})}\n& < \\norm{\\nabla f(\\ushort{x})}-\\norm{\\nabla f(x)-\\nabla f(\\ushort{x})} \\\\\n& \\le \\norm{\\nabla f(x)} \\nonumber\\\\\n& \\le \\norm{\\nabla f(\\ushort{x})}+\\norm{\\nabla f(x)-\\nabla f(\\ushort{x})} \\nonumber\\\\\n& < \\frac{5}{4} \\norm{\\nabla f(\\ushort{x})}.\n\\label{eq:nablas_ushortx_x}\n\\end{align}\n","start":63629,"end":64413}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":["eq:UpperBoundfDiffGrad"],"declared_external":false,"source_only":false,"title":"用两个基点余项估计函数差","source_units":[{"raw_tex":"For all $x \\in \\ball(\\ushort{x}, \\rho) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{align}\nf(y)\n&= f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x} \\nonumber\\\\\n&\\qquad + \\left(f(\\ushort{x}) - f(x) - \\ip{\\nabla f(\\ushort{x})}{\\ushort{x}-x}\\right) \\nonumber\\\\\n&\\qquad + \\left(f(y) - f(\\ushort{x}) - \\ip{\\nabla f(\\ushort{x})}{y-\\ushort{x}}\\right) \\nonumber\\\\\n&\\le f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x} +\n\\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4\\left(1 + \\frac{8}{3(1-\\kappa)}\\right)} \\left(\\norm{\\ushort{x}-x} + \\norm{y-\\ushort{x}}\\right),\n\\label{eq:UpperBoundfDiffGrad}\n\\end{align}\nwhere the inequality follows from \\eqref{eq:def_alpha_x_small1}. Observe that\n","start":64413,"end":65155}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"验证投影点处于余项控制球","source_units":[{"raw_tex":"\\begin{align*}\n\\norm{y-\\ushort{x}}\n&\\le \\norm{y-x} + \\norm{x-\\ushort{x}} \\\\\n&\\le 2 \\alpha \\norm{\\nabla f(x)} + \\rho\n\\text{ from \\eqref{eq:ProjectedTranslationDistance}}\\\\\n&\\le \\frac{2 \\alpha_{\\ushort{x}}}{\\beta} \\norm{\\nabla f(x)}\n+ \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})} \\\\\n&< \\frac{5\\alpha_{\\ushort{x}}}{2\\beta} \\norm{\\nabla f(\\ushort{x})}\n+ \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})} \n\\text{ from \\eqref{eq:nablas_ushortx_x}} \\\\\n&\\le \\frac{7\\alpha_{\\ushort{x}}}{2\\beta} \\norm{\\nabla f(\\ushort{x})}.\n\\end{align*}\n","start":65155,"end":65684}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"更新长度支配基点偏移","source_units":[{"raw_tex":"Inequality (a) is true because\n\\begin{align*}\n\\norm{y-x}\n&\\ge \\alpha \\norm{\\nabla f(x)} - \\norm{x-\\alpha \\nabla f(x)-y}\n\\text{ by the triangle inequality}\\\\\n&= \\alpha \\norm{\\nabla f(x)} - \\dist(x-\\alpha\\nabla f(x), C) \\\\\n&\\ge (1-\\kappa) \\alpha \\norm{\\nabla f(x)}\n\\text{ from Proposition~\\ref{prop:ArbitrarySmallAlpha}}\\\\\n&\\ge \\frac{3}{4} (1-\\kappa) \\rho\n\\text{ from~\\eqref{eq:nablas_ushortx_x}, the definition of $\\rho$, and $\\alpha_{\\ushort{x}} \\le \\alpha$}\\\\\n&> \\frac{3}{4} (1-\\kappa) \\norm{x-\\ushort{x}}.\n\\end{align*}\n","start":66712,"end":67233}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"吸收误差得到严格 Armijo 条件","source_units":[{"raw_tex":"We continue from \\eqref{eq:UpperBoundfDiffGrad}:\n\\begin{align*}\nf(y)\n& \\le f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x} +\n\\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{\n4\\left(1 + \\frac{8}{3(1-\\kappa)}\\right)} \\left(2\\norm{\\ushort{x}-x} + \\norm{y-x}\\right) \\\\\n&\\overset{\\text{(a)}}{<} f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x}\n+ \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4} \\norm{y-x} \\\\\n&\\le f(x) + \\ip{\\nabla f(x)}{y-x} + \\norm{\\nabla f(\\ushort{x})-\\nabla f(x)} \\norm{y-x} \\\\\n&\\qquad + \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4} \\norm{y-x} \\\\\n&\\le f(x) + \\ip{\\nabla f(x)}{y-x} + \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{2} \\norm{y-x}\n\\text{ from~\\eqref{eq:def_alpha_x_small2}}\\\\\n&< f(x) + \\ip{\\nabla f(x)}{y-x} + (1-c)\\sqrt{1-\\kappa^2} \\norm{\\nabla f(x)}\\norm{y-x}\n\\text{ from~\\eqref{eq:nablas_ushortx_x}}\\\\\n&\\le f(x) + \\ip{\\nabla f(x)}{y-x} - (1-c) \\ip{\\nabla f(x)}{y-x}\n\\text{ from Proposition~\\ref{prop:ArbitrarySmallAlpha}}\\\\\n&= f(x) + c \\ip{\\nabla f(x)}{y-x}.\n\\end{align*}\n","start":65684,"end":66712}]},{"id":"S061","semantic_type":"proposition","kind":"proposition","labels":["prop:convergence_ik"],"declared_external":false,"source_only":false,"title":"收敛子列的后继保持同一极限","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:convergence_ik}\nLet $(x_i)_{i \\in \\N}$ be a sequence generated by $\\pgd$ (Algorithm~\\ref{algo:PGD}), under~\\ref{it:continuously_diff}, and $x$ be an accumulation point.\nThen, for every subsequence $(x_{i_k})_{k\\in\\N}$ converging to $x$, the sequence $(x_{i_k+1})_{k\\in\\N}$ also converges to $x$.\n\\end{proposition}","start":67338,"end":67683}]},{"id":"S062","semantic_type":"proof","kind":"proof","labels":["eq:def_s_1"],"declared_external":false,"source_only":false,"title":"后继子列性质的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $(x_{i_k})_{k\\in\\N}$ be a subsequence converging to $x$. We show that $(x_{i_k+1})_{k\\in\\N}$ also converges to $x$.\n\nIf the nonmonotonicity rule is set to ``average'', this is a direct consequence of Proposition~\\ref{prop:NonmonotonePGDAverage}. Indeed, for all $i \\in \\N$, from~\\eqref{eq:KanzowMehlitzDecay},\n\\begin{equation*}\nf(x_{i+1}) \\le \\mu_i - \\frac{c}{2 \\oshort{\\alpha}} \\norm{x_{i+1}-x_i}^2 \\le \\mu_i.    \n\\end{equation*}\nFrom Proposition~\\ref{prop:NonmonotonePGDAverage}, $(f(x_{i+1}))_{i\\in\\N}$ and $(\\mu_i)_{i\\in\\N}$ converge to the same limit. Therefore,\n\\begin{equation*}\n \\left(\\mu_i - \\frac{c}{2 \\oshort{\\alpha}} \\norm{x_{i+1}-x_i}^2\\right)_{i\\in\\N}\n\\end{equation*}\nalso converges to this limit.\nThis implies that $(\\norm{x_{i+1}-x_i})_{i\\in\\N}$ converges to $0$, hence $(\\norm{x_{i_k+1}-x_{i_k}})_{k\\in\\N}$ converges to $0$, and $(x_{i_k+1})_{k\\in\\N}$ converges to the same limit as $(x_{i_k})_{k\\in\\N}$, that is, $x$.\n\nNow, let us consider the ``max'' rule case. It suffices to show that $x$ is an accumulation point of every subsequence of $(x_{i_k+1})_{k\\in\\N}$. Thus, we show the following: for every subsequence $(i_{j_k})_{k\\in\\N}$ of $(i_k)_{k\\in\\N}$, there exists a subsequence of $(x_{i_{j_k}+1})_{k\\in\\N}$ that converges to $x$.\nLet $(i_{j_k})_{k\\in\\N}$ be a subsequence of $(i_k)_{k\\in\\N}$.\nFor all $i \\in \\N$, let $g(i) \\in \\argmax_{j \\in \\{\\max\\{0, i-l\\}, \\dots, i\\}} f(x_j)$, as in Proposition~\\ref{prop:NonmonotonePGD}. By the third statement of Proposition~\\ref{prop:NonmonotonePGD}, the sequence $(f(x_{g(i)}))_{i\\in\\N}$ converges to $\\varphi \\in [f(x), f(x_0)]$.\nFor every $k \\in \\N$, letting $\\alpha_{i_{j_k}} \\in (0, \\oshort{\\alpha}]$ be the number such that $x_{i_{j_k}+1} \\in \\proj{C}{x_{i_{j_k}} - \\alpha_{i_{j_k}} \\nabla f(x_{i_{j_k}})}$, by \\eqref{eq:ProjectedTranslationDistance},\n\\begin{equation*}\n\\norm{x_{i_{j_k}+1}-x_{i_{j_k}}}\n\\le 2 \\alpha_{i_{j_k}} \\norm{\\nabla f(x_{i_{j_k}})}\n\\le 2 \\oshort{\\alpha} \\norm{\\nabla f(x_{i_{j_k}})}.\n\\end{equation*}\nThus, since $(x_{i_{j_k}})_{k\\in\\N}$ is bounded and $\\nabla f$ is locally bounded (as it is continuous), the sequence $(x_{i_{j_k}+1})_{k\\in\\N}$ is bounded. If we replace $(i_{j_k})_{k\\in\\N}$ with a subsequence, we can assume that $(x_{i_{j_k}+1})_{k\\in\\N}$ converges.\n\nIterating the reasoning, we can assume that $(x_{i_{j_k}+s})_{k\\in\\N}$ converges to some $x^{s}\\in C$ for every $s \\in \\{0, \\dots, l+1\\}$. By definition of $x$, $x^{0}=x$.\n\nObserve that, from the continuity of $f$,\n\\begin{align*}\nf(x_{g(i_{j_k}+l+1)})\n&= \\max\\{f(x_{i_{j_k}+1}), \\dots, f(x_{i_{j_k}+l+1})\\} \\\\\n&\\to \\max\\{f(x^{1}), \\dots, f(x^{l+1})\\}\n\\text{ when } k \\to \\infty.\n\\end{align*}\nIn particular, there exists $s_1 \\in \\{1, \\dots, l+1\\}$ such that\n\\begin{equation}\n\\label{eq:def_s_1}\nf(x^{s_1}) = \\varphi.\n\\end{equation}\nLet $s_1$ be the smallest such integer.\nFor all $k \\in \\N$, from the condition in line~\\ref{algo:PGDmap:NonmonotoneArmijoCondition} of Algorithm~\\ref{algo:PGDmap} and~\\eqref{eq:KanzowMehlitzDecay},\n\\begin{align*}\nf(x_{i_{j_k}+s_1})\n&\\le f(x_{g(i_{j_k}+s_1-1)}) - \\frac{c}{2\\oshort{\\alpha}} \\norm{x_{i_{j_k}+s_1}-x_{i_{j_k}+s_1-1}}^2,\n\\end{align*}\nand letting $k$ tend to infinity yields\n\\begin{equation*}\n\\varphi\n= f(x^{s_1})\n\\le \\varphi - \\frac{c}{2\\oshort{\\alpha}} \\norm{x^{s_1}-x^{s_1-1}}^2.\n\\end{equation*}\nConsequently, $x^{s_1} = x^{s_1-1}$. In particular, $f(x^{s_1-1}) =f(x^{s_1}) = \\varphi$. Therefore, $s_1=1$, otherwise it would not be the smallest integer satisfying \\eqref{eq:def_s_1}. The equality $x^{s_1} = x^{s_1-1}$ then reduces to $x^{1}=x^{0}=x$ and, when $k \\to \\infty$,\n\\begin{equation*}\nx_{i_{j_k}+1} \\to x^{1} = x.\n\\end{equation*}\n\\end{proof}","start":67685,"end":71365}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"平均规则下相邻差趋零","source_units":[{"raw_tex":"\\begin{proof}\nLet $(x_{i_k})_{k\\in\\N}$ be a subsequence converging to $x$. We show that $(x_{i_k+1})_{k\\in\\N}$ also converges to $x$.\n\nIf the nonmonotonicity rule is set to ``average'', this is a direct consequence of Proposition~\\ref{prop:NonmonotonePGDAverage}. Indeed, for all $i \\in \\N$, from~\\eqref{eq:KanzowMehlitzDecay},\n\\begin{equation*}\nf(x_{i+1}) \\le \\mu_i - \\frac{c}{2 \\oshort{\\alpha}} \\norm{x_{i+1}-x_i}^2 \\le \\mu_i.    \n\\end{equation*}\nFrom Proposition~\\ref{prop:NonmonotonePGDAverage}, $(f(x_{i+1}))_{i\\in\\N}$ and $(\\mu_i)_{i\\in\\N}$ converge to the same limit. Therefore,\n\\begin{equation*}\n \\left(\\mu_i - \\frac{c}{2 \\oshort{\\alpha}} \\norm{x_{i+1}-x_i}^2\\right)_{i\\in\\N}\n\\end{equation*}\nalso converges to this limit.\nThis implies that $(\\norm{x_{i+1}-x_i})_{i\\in\\N}$ converges to $0$, hence $(\\norm{x_{i_k+1}-x_{i_k}})_{k\\in\\N}$ converges to $0$, and $(x_{i_k+1})_{k\\in\\N}$ converges to the same limit as $(x_{i_k})_{k\\in\\N}$, that is, $x$.\n","start":67685,"end":68639}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"最大值规则的子子列判别","source_units":[{"raw_tex":"Now, let us consider the ``max'' rule case. It suffices to show that $x$ is an accumulation point of every subsequence of $(x_{i_k+1})_{k\\in\\N}$. Thus, we show the following: for every subsequence $(i_{j_k})_{k\\in\\N}$ of $(i_k)_{k\\in\\N}$, there exists a subsequence of $(x_{i_{j_k}+1})_{k\\in\\N}$ that converges to $x$.\nLet $(i_{j_k})_{k\\in\\N}$ be a subsequence of $(i_k)_{k\\in\\N}$.\nFor all $i \\in \\N$, let $g(i) \\in \\argmax_{j \\in \\{\\max\\{0, i-l\\}, \\dots, i\\}} f(x_j)$, as in Proposition~\\ref{prop:NonmonotonePGD}. By the third statement of Proposition~\\ref{prop:NonmonotonePGD}, the sequence $(f(x_{g(i)}))_{i\\in\\N}$ converges to $\\varphi \\in [f(x), f(x_0)]$.\n","start":68640,"end":69301}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"抽取有限个连续偏移极限","source_units":[{"raw_tex":"For every $k \\in \\N$, letting $\\alpha_{i_{j_k}} \\in (0, \\oshort{\\alpha}]$ be the number such that $x_{i_{j_k}+1} \\in \\proj{C}{x_{i_{j_k}} - \\alpha_{i_{j_k}} \\nabla f(x_{i_{j_k}})}$, by \\eqref{eq:ProjectedTranslationDistance},\n\\begin{equation*}\n\\norm{x_{i_{j_k}+1}-x_{i_{j_k}}}\n\\le 2 \\alpha_{i_{j_k}} \\norm{\\nabla f(x_{i_{j_k}})}\n\\le 2 \\oshort{\\alpha} \\norm{\\nabla f(x_{i_{j_k}})}.\n\\end{equation*}\nThus, since $(x_{i_{j_k}})_{k\\in\\N}$ is bounded and $\\nabla f$ is locally bounded (as it is continuous), the sequence $(x_{i_{j_k}+1})_{k\\in\\N}$ is bounded. If we replace $(i_{j_k})_{k\\in\\N}$ with a subsequence, we can assume that $(x_{i_{j_k}+1})_{k\\in\\N}$ converges.\n\nIterating the reasoning, we can assume that $(x_{i_{j_k}+s})_{k\\in\\N}$ converges to some $x^{s}\\in C$ for every $s \\in \\{0, \\dots, l+1\\}$. By definition of $x$, $x^{0}=x$.\n","start":69301,"end":70140}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":["eq:def_s_1"],"declared_external":false,"source_only":false,"title":"选取最早达到窗口极限的偏移","source_units":[{"raw_tex":"Observe that, from the continuity of $f$,\n\\begin{align*}\nf(x_{g(i_{j_k}+l+1)})\n&= \\max\\{f(x_{i_{j_k}+1}), \\dots, f(x_{i_{j_k}+l+1})\\} \\\\\n&\\to \\max\\{f(x^{1}), \\dots, f(x^{l+1})\\}\n\\text{ when } k \\to \\infty.\n\\end{align*}\nIn particular, there exists $s_1 \\in \\{1, \\dots, l+1\\}$ such that\n\\begin{equation}\n\\label{eq:def_s_1}\nf(x^{s_1}) = \\varphi.\n\\end{equation}\nLet $s_1$ be the smallest such integer.\n","start":70141,"end":70539}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"下降界迫使第一后继回到聚点","source_units":[{"raw_tex":"For all $k \\in \\N$, from the condition in line~\\ref{algo:PGDmap:NonmonotoneArmijoCondition} of Algorithm~\\ref{algo:PGDmap} and~\\eqref{eq:KanzowMehlitzDecay},\n\\begin{align*}\nf(x_{i_{j_k}+s_1})\n&\\le f(x_{g(i_{j_k}+s_1-1)}) - \\frac{c}{2\\oshort{\\alpha}} \\norm{x_{i_{j_k}+s_1}-x_{i_{j_k}+s_1-1}}^2,\n\\end{align*}\nand letting $k$ tend to infinity yields\n\\begin{equation*}\n\\varphi\n= f(x^{s_1})\n\\le \\varphi - \\frac{c}{2\\oshort{\\alpha}} \\norm{x^{s_1}-x^{s_1-1}}^2.\n\\end{equation*}\nConsequently, $x^{s_1} = x^{s_1-1}$. In particular, $f(x^{s_1-1}) =f(x^{s_1}) = \\varphi$. Therefore, $s_1=1$, otherwise it would not be the smallest integer satisfying \\eqref{eq:def_s_1}. The equality $x^{s_1} = x^{s_1-1}$ then reduces to $x^{1}=x^{0}=x$ and, when $k \\to \\infty$,\n\\begin{equation*}\nx_{i_{j_k}+1} \\to x^{1} = x.\n\\end{equation*}\n\\end{proof}","start":70539,"end":71365}]},{"id":"S068","semantic_type":"theorem","kind":"theorem","labels":["thm:PGDaccumulatesAtBStationaryPoints"],"declared_external":false,"source_only":false,"title":"PGD 聚于 B 驻点且孤立聚点保证收敛","source_units":[{"raw_tex":"\\begin{theorem}\n\\label{thm:PGDaccumulatesAtBStationaryPoints}\nLet $(x_i)_{i \\in \\N}$ be a sequence generated by $\\pgd$ (Algorithm~\\ref{algo:PGD}), under~\\ref{it:continuously_diff}. Then, all accumulation points of $(x_i)_{i \\in \\N}$ are B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}.\nIf, moreover, $(x_i)_{i \\in \\N}$ has an isolated accumulation point, then $(x_i)_{i \\in \\N}$ converges.\n\\end{theorem}","start":50907,"end":51315}]},{"id":"S069","semantic_type":"proof","kind":"proof","labels":["subsec:proof_accumulates"],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\subsection{Third part: proof of Theorem~\\ref{thm:PGDaccumulatesAtBStationaryPoints}}\n\\label{subsec:proof_accumulates}\nLet $\\ushort{x}$ be an accumulation point of $(x_i)_{i\\in\\N}$. Assume, for the sake of contradiction, that $\\ushort{x}$ is not B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}. Let $(x_{i_k})_{k\\in\\N}$ be a subsequence converging to $\\ushort{x}$.\n\nLet $\\alpha_{\\ushort{x}}$ and $\\rho$ be as in Proposition~\\ref{prop:AcceptedStep}. For all $k \\in \\N$ large enough, $x_{i_k} \\! \\in \\ball(\\ushort{x}, \\rho) \\cap C$. Thus, when Algorithm~\\ref{algo:PGDmap} is called at point $x_{i_k}$, the condition in line~\\ref{algo:PGDmap:NonmonotoneArmijoCondition} stops being fulfilled for some $\\alpha_{i_k} \\ge \\alpha_{\\ushort{x}}$, meaning that\n\\begin{equation*}\nx_{i_k+1} \\in \\proj{C}{x_{i_k}-\\alpha_{i_k} \\nabla f(x_{i_k})}\n\\text{ for some } \\alpha_{i_k} \\in [\\alpha_{\\ushort{x}}, \\oshort{\\alpha}].\n\\end{equation*}\nReplacing $(i_k)_{k\\in\\N}$ with a subsequence if necessary, we can assume that $(\\alpha_{i_k})_{k\\in\\N}$ converges to some $\\alpha_{\\lim} \\in [\\alpha_{\\ushort{x}}, \\oshort{\\alpha}]$.\n\nFor all $k \\in \\N$, we have\n\\begin{equation*}\n\\norm{x_{i_k} - \\alpha_{i_k} \\nabla f(x_{i_k}) - x_{i_k+1}}\n= \\dist(x_{i_k} - \\alpha_{i_k} \\nabla f(x_{i_k}), C),\n\\end{equation*}\nand since the distance to a nonempty closed set is a continuous function, we can take this equality to the limit. We use the fact that $x_{i_k+1} \\to \\ushort{x}$ when $k \\to \\infty$, from Proposition~\\ref{prop:convergence_ik}. This yields\n\\begin{equation*}\n\\norm{\\alpha_{\\lim} \\nabla f(\\ushort{x})}\n= \\dist(\\ushort{x} - \\alpha_{\\lim}\\nabla f(\\ushort{x}), C),\n\\end{equation*}\nwhich means that $\\ushort{x} \\in \\proj{C}{\\ushort{x} - \\alpha_{\\lim}\\nabla f(\\ushort{x})}$. In particular, $-\\nabla f(\\ushort{x}) \\in \\proxnorcone{C}{\\ushort{x}} \\subseteq \\regnorcone{C}{\\ushort{x}}$, which contradicts our assumption that $\\ushort{x}$ is not B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}. We have therefore proven that every accumulation point is B-stationary.\n\nFinally, if $(x_i)_{i\\in\\N}$ has an isolated accumulation point, then the sequence $(x_i)_{i\\in\\N}$ converges, from Proposition~\\ref{prop:convergence_ik} and \\cite[Lemma~4.10]{MoreSorensen}.\n\n","start":71367,"end":73608}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":["subsec:proof_accumulates"],"declared_external":false,"source_only":false,"title":"假设存在非 B 驻聚点","source_units":[{"raw_tex":"\\subsection{Third part: proof of Theorem~\\ref{thm:PGDaccumulatesAtBStationaryPoints}}\n\\label{subsec:proof_accumulates}\nLet $\\ushort{x}$ be an accumulation point of $(x_i)_{i\\in\\N}$. Assume, for the sake of contradiction, that $\\ushort{x}$ is not B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}. Let $(x_{i_k})_{k\\in\\N}$ be a subsequence converging to $\\ushort{x}$.\n","start":71367,"end":71737}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":["subsec:proof_accumulates"],"declared_external":false,"source_only":false,"title":"回溯步长远离零并抽取极限","source_units":[{"raw_tex":"Let $\\alpha_{\\ushort{x}}$ and $\\rho$ be as in Proposition~\\ref{prop:AcceptedStep}. For all $k \\in \\N$ large enough, $x_{i_k} \\! \\in \\ball(\\ushort{x}, \\rho) \\cap C$. Thus, when Algorithm~\\ref{algo:PGDmap} is called at point $x_{i_k}$, the condition in line~\\ref{algo:PGDmap:NonmonotoneArmijoCondition} stops being fulfilled for some $\\alpha_{i_k} \\ge \\alpha_{\\ushort{x}}$, meaning that\n\\begin{equation*}\nx_{i_k+1} \\in \\proj{C}{x_{i_k}-\\alpha_{i_k} \\nabla f(x_{i_k})}\n\\text{ for some } \\alpha_{i_k} \\in [\\alpha_{\\ushort{x}}, \\oshort{\\alpha}].\n\\end{equation*}\nReplacing $(i_k)_{k\\in\\N}$ with a subsequence if necessary, we can assume that $(\\alpha_{i_k})_{k\\in\\N}$ converges to some $\\alpha_{\\lim} \\in [\\alpha_{\\ushort{x}}, \\oshort{\\alpha}]$.\n","start":71738,"end":72478}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":["subsec:proof_accumulates"],"declared_external":false,"source_only":false,"title":"对投影距离等式取极限","source_units":[{"raw_tex":"For all $k \\in \\N$, we have\n\\begin{equation*}\n\\norm{x_{i_k} - \\alpha_{i_k} \\nabla f(x_{i_k}) - x_{i_k+1}}\n= \\dist(x_{i_k} - \\alpha_{i_k} \\nabla f(x_{i_k}), C),\n\\end{equation*}\nand since the distance to a nonempty closed set is a continuous function, we can take this equality to the limit. We use the fact that $x_{i_k+1} \\to \\ushort{x}$ when $k \\to \\infty$, from Proposition~\\ref{prop:convergence_ik}. This yields\n\\begin{equation*}\n\\norm{\\alpha_{\\lim} \\nabla f(\\ushort{x})}\n= \\dist(\\ushort{x} - \\alpha_{\\lim}\\nabla f(\\ushort{x}), C),\n\\end{equation*}\n","start":72479,"end":73030}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":["subsec:proof_accumulates"],"declared_external":false,"source_only":false,"title":"近端法向量与非 B 驻假设矛盾","source_units":[{"raw_tex":"which means that $\\ushort{x} \\in \\proj{C}{\\ushort{x} - \\alpha_{\\lim}\\nabla f(\\ushort{x})}$. In particular, $-\\nabla f(\\ushort{x}) \\in \\proxnorcone{C}{\\ushort{x}} \\subseteq \\regnorcone{C}{\\ushort{x}}$, which contradicts our assumption that $\\ushort{x}$ is not B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}. We have therefore proven that every accumulation point is B-stationary.\n","start":73030,"end":73415}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":["subsec:proof_accumulates"],"declared_external":false,"source_only":false,"title":"孤立聚点推出全序列收敛","source_units":[{"raw_tex":"Finally, if $(x_i)_{i\\in\\N}$ has an isolated accumulation point, then the sequence $(x_i)_{i\\in\\N}$ converges, from Proposition~\\ref{prop:convergence_ik} and \\cite[Lemma~4.10]{MoreSorensen}.\n\n","start":73416,"end":73608}]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原声明先决条件：Standard finite-dimensional compactness, subsequence, continuity, and inner-product facts。"],"source_only":false,"title":"有限维紧性与极限工具","source_units":[]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原声明的标准内积事实；极锥定义否定所用的存在性步骤为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"内积估计与极锥否定的标准步骤","source_units":[]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原声明先决条件：RockafellarWets, Example 1.20 and Chapter 6；正文另标 Definition 6.1、Propositions 6.2/6.5、Definition 6.3、Example 6.16。"],"source_only":false,"title":"Rockafellar–Wets 的投影与法锥背景","source_units":[]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原声明先决条件：KanzowMehlitz, Lemma 4.1 and Corollary 4.1；明确引用见原文第 568 行。"],"source_only":false,"title":"最大值规则的文献输入","source_units":[]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原声明先决条件补充：KanzowMehlitz (27) 或 JiaEtAl (A.9)；实际引用见第 577 行。"],"source_only":false,"title":"固定偏移窗口函数值收敛","source_units":[]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原声明先决条件：MoreSorensen, Lemma 4.10；明确引用见第 983 行。"],"source_only":false,"title":"孤立聚点收敛的外部引理","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S068","edges":[{"source_id":"S001","target_id":"S068","kind":"supports"},{"source_id":"S002","target_id":"S068","kind":"supports"},{"source_id":"S018","target_id":"S068","kind":"supports"},{"source_id":"S006","target_id":"S068","kind":"supports"},{"source_id":"S014","target_id":"S006","kind":"supports"},{"source_id":"S077","target_id":"S012","kind":"supports"},{"source_id":"S077","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S077","target_id":"S014","kind":"supports"},{"source_id":"S012","target_id":"S016","kind":"supports"},{"source_id":"S077","target_id":"S016","kind":"supports"},{"source_id":"S002","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S006","target_id":"S018","kind":"supports"},{"source_id":"S012","target_id":"S032","kind":"supports"},{"source_id":"S076","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S076","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S017","target_id":"S019","kind":"supports"},{"source_id":"S030","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S021","kind":"supports"},{"source_id":"S008","target_id":"S022","kind":"supports"},{"source_id":"S078","target_id":"S026","kind":"supports"},{"source_id":"S019","target_id":"S026","kind":"supports"},{"source_id":"S021","target_id":"S027","kind":"supports"},{"source_id":"S002","target_id":"S027","kind":"supports"},{"source_id":"S075","target_id":"S027","kind":"supports"},{"source_id":"S021","target_id":"S028","kind":"supports"},{"source_id":"S079","target_id":"S028","kind":"supports"},{"source_id":"S075","target_id":"S028","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S027","target_id":"S025","kind":"supports"},{"source_id":"S028","target_id":"S025","kind":"supports"},{"source_id":"S026","target_id":"S021","kind":"supports"},{"source_id":"S026","target_id":"S022","kind":"supports"},{"source_id":"S027","target_id":"S023","kind":"supports"},{"source_id":"S028","target_id":"S024","kind":"supports"},{"source_id":"S018","target_id":"S037","kind":"supports"},{"source_id":"S019","target_id":"S037","kind":"supports"},{"source_id":"S008","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S002","target_id":"S038","kind":"supports"},{"source_id":"S001","target_id":"S038","kind":"supports"},{"source_id":"S075","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S018","target_id":"S039","kind":"supports"},{"source_id":"S075","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S075","target_id":"S040","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S039","target_id":"S036","kind":"supports"},{"source_id":"S040","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S002","target_id":"S041","kind":"supports"},{"source_id":"S006","target_id":"S041","kind":"supports"},{"source_id":"S013","target_id":"S041","kind":"supports"},{"source_id":"S014","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S043","kind":"supports"},{"source_id":"S041","target_id":"S045","kind":"supports"},{"source_id":"S013","target_id":"S045","kind":"supports"},{"source_id":"S075","target_id":"S045","kind":"supports"},{"source_id":"S076","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S002","target_id":"S047","kind":"supports"},{"source_id":"S012","target_id":"S048","kind":"supports"},{"source_id":"S045","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S041","target_id":"S048","kind":"supports"},{"source_id":"S002","target_id":"S048","kind":"supports"},{"source_id":"S075","target_id":"S048","kind":"supports"},{"source_id":"S076","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S001","target_id":"S049","kind":"supports"},{"source_id":"S002","target_id":"S049","kind":"supports"},{"source_id":"S075","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S047","target_id":"S050","kind":"supports"},{"source_id":"S041","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S047","target_id":"S051","kind":"supports"},{"source_id":"S041","target_id":"S051","kind":"supports"},{"source_id":"S076","target_id":"S051","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S044","kind":"supports"},{"source_id":"S049","target_id":"S044","kind":"supports"},{"source_id":"S050","target_id":"S044","kind":"supports"},{"source_id":"S051","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S042","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S006","target_id":"S054","kind":"supports"},{"source_id":"S014","target_id":"S054","kind":"supports"},{"source_id":"S076","target_id":"S054","kind":"supports"},{"source_id":"S002","target_id":"S054","kind":"supports"},{"source_id":"S041","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S042","target_id":"S055","kind":"supports"},{"source_id":"S043","target_id":"S055","kind":"supports"},{"source_id":"S041","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S076","target_id":"S056","kind":"supports"},{"source_id":"S029","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S058","kind":"supports"},{"source_id":"S055","target_id":"S058","kind":"supports"},{"source_id":"S054","target_id":"S057","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S042","target_id":"S059","kind":"supports"},{"source_id":"S056","target_id":"S059","kind":"supports"},{"source_id":"S055","target_id":"S059","kind":"supports"},{"source_id":"S076","target_id":"S059","kind":"supports"},{"source_id":"S057","target_id":"S060","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S054","target_id":"S060","kind":"supports"},{"source_id":"S056","target_id":"S060","kind":"supports"},{"source_id":"S043","target_id":"S060","kind":"supports"},{"source_id":"S076","target_id":"S060","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S055","target_id":"S053","kind":"supports"},{"source_id":"S056","target_id":"S053","kind":"supports"},{"source_id":"S057","target_id":"S053","kind":"supports"},{"source_id":"S058","target_id":"S053","kind":"supports"},{"source_id":"S059","target_id":"S053","kind":"supports"},{"source_id":"S060","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S018","target_id":"S063","kind":"supports"},{"source_id":"S019","target_id":"S063","kind":"supports"},{"source_id":"S035","target_id":"S063","kind":"supports"},{"source_id":"S075","target_id":"S063","kind":"supports"},{"source_id":"S023","target_id":"S064","kind":"supports"},{"source_id":"S021","target_id":"S064","kind":"supports"},{"source_id":"S075","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S029","target_id":"S065","kind":"supports"},{"source_id":"S017","target_id":"S065","kind":"supports"},{"source_id":"S002","target_id":"S065","kind":"supports"},{"source_id":"S001","target_id":"S065","kind":"supports"},{"source_id":"S075","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S002","target_id":"S066","kind":"supports"},{"source_id":"S018","target_id":"S066","kind":"supports"},{"source_id":"S075","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S065","target_id":"S067","kind":"supports"},{"source_id":"S064","target_id":"S067","kind":"supports"},{"source_id":"S019","target_id":"S067","kind":"supports"},{"source_id":"S017","target_id":"S067","kind":"supports"},{"source_id":"S075","target_id":"S067","kind":"supports"},{"source_id":"S063","target_id":"S062","kind":"supports"},{"source_id":"S064","target_id":"S062","kind":"supports"},{"source_id":"S065","target_id":"S062","kind":"supports"},{"source_id":"S066","target_id":"S062","kind":"supports"},{"source_id":"S067","target_id":"S062","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S001","target_id":"S070","kind":"supports"},{"source_id":"S006","target_id":"S070","kind":"supports"},{"source_id":"S075","target_id":"S070","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S052","target_id":"S071","kind":"supports"},{"source_id":"S017","target_id":"S071","kind":"supports"},{"source_id":"S018","target_id":"S071","kind":"supports"},{"source_id":"S075","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S061","target_id":"S072","kind":"supports"},{"source_id":"S012","target_id":"S072","kind":"supports"},{"source_id":"S002","target_id":"S072","kind":"supports"},{"source_id":"S075","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S071","target_id":"S073","kind":"supports"},{"source_id":"S016","target_id":"S073","kind":"supports"},{"source_id":"S004","target_id":"S073","kind":"supports"},{"source_id":"S006","target_id":"S073","kind":"supports"},{"source_id":"S070","target_id":"S073","kind":"supports"},{"source_id":"S061","target_id":"S074","kind":"supports"},{"source_id":"S080","target_id":"S074","kind":"supports"},{"source_id":"S070","target_id":"S069","kind":"supports"},{"source_id":"S071","target_id":"S069","kind":"supports"},{"source_id":"S072","target_id":"S069","kind":"supports"},{"source_id":"S073","target_id":"S069","kind":"supports"},{"source_id":"S074","target_id":"S069","kind":"supports"},{"source_id":"S069","target_id":"S068","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S068","edges":[{"source_id":"S001","target_id":"S068","kind":"supports"},{"source_id":"S002","target_id":"S068","kind":"supports"},{"source_id":"S018","target_id":"S068","kind":"supports"},{"source_id":"S006","target_id":"S068","kind":"supports"},{"source_id":"S014","target_id":"S006","kind":"supports"},{"source_id":"S077","target_id":"S012","kind":"supports"},{"source_id":"S077","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S077","target_id":"S014","kind":"supports"},{"source_id":"S012","target_id":"S016","kind":"supports"},{"source_id":"S077","target_id":"S016","kind":"supports"},{"source_id":"S002","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S006","target_id":"S018","kind":"supports"},{"source_id":"S012","target_id":"S032","kind":"supports"},{"source_id":"S076","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S076","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S017","target_id":"S019","kind":"supports"},{"source_id":"S030","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S021","kind":"supports"},{"source_id":"S008","target_id":"S022","kind":"supports"},{"source_id":"S078","target_id":"S026","kind":"supports"},{"source_id":"S019","target_id":"S026","kind":"supports"},{"source_id":"S021","target_id":"S027","kind":"supports"},{"source_id":"S002","target_id":"S027","kind":"supports"},{"source_id":"S075","target_id":"S027","kind":"supports"},{"source_id":"S021","target_id":"S028","kind":"supports"},{"source_id":"S079","target_id":"S028","kind":"supports"},{"source_id":"S075","target_id":"S028","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S027","target_id":"S025","kind":"supports"},{"source_id":"S028","target_id":"S025","kind":"supports"},{"source_id":"S026","target_id":"S021","kind":"supports"},{"source_id":"S026","target_id":"S022","kind":"supports"},{"source_id":"S027","target_id":"S023","kind":"supports"},{"source_id":"S028","target_id":"S024","kind":"supports"},{"source_id":"S018","target_id":"S037","kind":"supports"},{"source_id":"S019","target_id":"S037","kind":"supports"},{"source_id":"S008","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S002","target_id":"S038","kind":"supports"},{"source_id":"S001","target_id":"S038","kind":"supports"},{"source_id":"S075","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S018","target_id":"S039","kind":"supports"},{"source_id":"S075","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S075","target_id":"S040","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S039","target_id":"S036","kind":"supports"},{"source_id":"S040","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S002","target_id":"S041","kind":"supports"},{"source_id":"S006","target_id":"S041","kind":"supports"},{"source_id":"S013","target_id":"S041","kind":"supports"},{"source_id":"S014","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S043","kind":"supports"},{"source_id":"S041","target_id":"S045","kind":"supports"},{"source_id":"S013","target_id":"S045","kind":"supports"},{"source_id":"S075","target_id":"S045","kind":"supports"},{"source_id":"S076","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S002","target_id":"S047","kind":"supports"},{"source_id":"S012","target_id":"S048","kind":"supports"},{"source_id":"S045","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S041","target_id":"S048","kind":"supports"},{"source_id":"S002","target_id":"S048","kind":"supports"},{"source_id":"S075","target_id":"S048","kind":"supports"},{"source_id":"S076","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S001","target_id":"S049","kind":"supports"},{"source_id":"S002","target_id":"S049","kind":"supports"},{"source_id":"S075","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S047","target_id":"S050","kind":"supports"},{"source_id":"S041","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S047","target_id":"S051","kind":"supports"},{"source_id":"S041","target_id":"S051","kind":"supports"},{"source_id":"S076","target_id":"S051","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S044","kind":"supports"},{"source_id":"S049","target_id":"S044","kind":"supports"},{"source_id":"S050","target_id":"S044","kind":"supports"},{"source_id":"S051","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S042","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S006","target_id":"S052","kind":"supports"},{"source_id":"S014","target_id":"S052","kind":"supports"},{"source_id":"S076","target_id":"S052","kind":"supports"},{"source_id":"S002","target_id":"S052","kind":"supports"},{"source_id":"S041","target_id":"S052","kind":"supports"},{"source_id":"S042","target_id":"S052","kind":"supports"},{"source_id":"S043","target_id":"S052","kind":"supports"},{"source_id":"S029","target_id":"S052","kind":"supports"},{"source_id":"S018","target_id":"S061","kind":"supports"},{"source_id":"S019","target_id":"S061","kind":"supports"},{"source_id":"S035","target_id":"S061","kind":"supports"},{"source_id":"S075","target_id":"S061","kind":"supports"},{"source_id":"S023","target_id":"S061","kind":"supports"},{"source_id":"S021","target_id":"S061","kind":"supports"},{"source_id":"S029","target_id":"S061","kind":"supports"},{"source_id":"S017","target_id":"S061","kind":"supports"},{"source_id":"S002","target_id":"S061","kind":"supports"},{"source_id":"S001","target_id":"S061","kind":"supports"},{"source_id":"S001","target_id":"S070","kind":"supports"},{"source_id":"S006","target_id":"S070","kind":"supports"},{"source_id":"S075","target_id":"S070","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S052","target_id":"S071","kind":"supports"},{"source_id":"S017","target_id":"S071","kind":"supports"},{"source_id":"S018","target_id":"S071","kind":"supports"},{"source_id":"S075","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S061","target_id":"S072","kind":"supports"},{"source_id":"S012","target_id":"S072","kind":"supports"},{"source_id":"S002","target_id":"S072","kind":"supports"},{"source_id":"S075","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S071","target_id":"S073","kind":"supports"},{"source_id":"S016","target_id":"S073","kind":"supports"},{"source_id":"S004","target_id":"S073","kind":"supports"},{"source_id":"S006","target_id":"S073","kind":"supports"},{"source_id":"S070","target_id":"S073","kind":"supports"},{"source_id":"S061","target_id":"S074","kind":"supports"},{"source_id":"S080","target_id":"S074","kind":"supports"},{"source_id":"S070","target_id":"S069","kind":"supports"},{"source_id":"S071","target_id":"S069","kind":"supports"},{"source_id":"S072","target_id":"S069","kind":"supports"},{"source_id":"S073","target_id":"S069","kind":"supports"},{"source_id":"S074","target_id":"S069","kind":"supports"},{"source_id":"S069","target_id":"S068","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:MinDiffFunctionClosedSet"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"闭集上的光滑优化问题","source_units":[{"raw_tex":"Let $\\mathcal{E}$ be a Euclidean vector space, $C$ a nonempty closed subset of $\\mathcal{E}$, and $f : \\mathcal{E} \\to \\R$ a function satisfying at least the first of the following two properties:\n","start":5818,"end":6015},{"raw_tex":"This paper considers the problem\n\\begin{equation}\n\\label{eq:MinDiffFunctionClosedSet}\n\\min_{x \\in C} f(x)\n\\end{equation}\nof minimizing $f$ on~$C$.\n","start":6599,"end":6746}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"可微性与可行集上梯度连续","source_units":[{"raw_tex":"\\begin{enumerate}[label={(H\\arabic*)}]\n\\item $f$ is differentiable on $C$, i.e., for every $x \\in C$, there exists a (unique) vector in $\\mathcal{E}$, denoted by $\\nabla f(x)$, such that\n\\begin{equation*}\n\\lim_{\\substack{y \\to x \\\\ y \\in \\mathcal{E} \\setminus \\{x\\}}} \\frac{f(y)-f(x)-\\ip{\\nabla f(x)}{y-x}}{\\norm{y-x}} = 0,\n\\end{equation*}\nand $\\nabla f : C \\to \\mathcal{E}$ is continuous;\n\\label{it:continuously_diff}\n","start":6015,"end":6434}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"更强的局部 Lipschitz 假设","source_units":[{"raw_tex":"\\item $f$ is differentiable on $\\mathcal{E}$ and $\\nabla f : \\mathcal{E} \\to \\mathcal{E}$ is locally Lipschitz continuous.\n\\label{it:Lipschitz_diff}\n\\end{enumerate}\n","start":6434,"end":6599}]},{"id":"S004","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:NestedNormalCones"],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"法锥记号与包含关系","source_units":[{"raw_tex":"A point $x \\in C$ is said to be stationary for~\\eqref{eq:MinDiffFunctionClosedSet} if $-\\nabla f(x)$ is normal to $C$ at~$x$. Several definitions of normality exist. Each defines a different notion of stationarity, which is a surrogate for local optimality in the sense that, possibly under mild regularity assumptions on~$f$, every local minimizer of $f|_C$ is stationary for~\\eqref{eq:MinDiffFunctionClosedSet}. In particular, each of the three notions of normality in \\cite[Definition~6.3 and Example~6.16]{RockafellarWets}, namely normality in the general sense, in the regular sense, and in the proximal sense, yields an important definition of stationarity. The sets of general, regular, and proximal normals to $C$ at $x \\in C$ are respectively denoted by $\\norcone{C}{x}$, $\\regnorcone{C}{x}$, and $\\proxnorcone{C}{x}$. These sets are reviewed in section~\\ref{subsec:NormalityStationarity}. Importantly, they are nested as follows: for every $x \\in C$,\n\\begin{equation}\n\\label{eq:NestedNormalCones}\n\\proxnorcone{C}{x} \\subseteq \\regnorcone{C}{x} \\subseteq \\norcone{C}{x},\n\\end{equation}\n","start":7214,"end":8309}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"Clarke 正则性","source_units":[{"raw_tex":"and $C$ is said to be \\emph{Clarke regular} at $x$ if the second inclusion is an equality.\n","start":8309,"end":8400}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["def:M/B/P-Stationarity"],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"Bouligand 驻点","source_units":[{"raw_tex":"\\begin{definition}\n\\label{def:M/B/P-Stationarity}\nFor problem~\\eqref{eq:MinDiffFunctionClosedSet}, a point $x \\in C$ is said to be:\n\\begin{itemize}\n","start":8585,"end":8733},{"raw_tex":"\\item \\emph{Bouligand stationary (B-stationary)} if $-\\nabla f(x) \\in \\regnorcone{C}{x}$;\n","start":8823,"end":8913},{"raw_tex":"\\end{itemize}\n\\end{definition}\n","start":9005,"end":9036}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":["def:M/B/P-Stationarity"],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"其他两类驻点","source_units":[{"raw_tex":"\\item \\emph{Mordukhovich stationary (M-stationary)} if $-\\nabla f(x) \\in \\norcone{C}{x}$;\n","start":8733,"end":8823},{"raw_tex":"\\item \\emph{proximally stationary (P-stationary)} if $-\\nabla f(x) \\in \\proxnorcone{C}{x}$.\n","start":8913,"end":9005}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":["eq:InitialSublevelSet"],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"初始次水平集","source_units":[{"raw_tex":"\\begin{equation}\n\\label{eq:InitialSublevelSet}\n\\{x \\in C \\mid f(x) \\le f(x_0)\\},\n\\end{equation}\n","start":13761,"end":13857}]},{"id":"S010","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"开球与闭球","source_units":[{"raw_tex":"Recall that, throughout the paper, $\\mathcal{E}$ is a Euclidean vector space and $C \\subseteq \\mathcal{E}$ is nonempty and closed. Moreover, for every $x \\in \\mathcal{E}$ and $\\rho \\in (0, \\infty)$, $\\ball(x, \\rho) \\coloneq \\{y \\in \\mathcal{E} \\mid \\norm{x-y} < \\rho\\}$ and $\\ball[x, \\rho] \\coloneq \\{y \\in \\mathcal{E} \\mid \\norm{x-y} \\le \\rho\\}$ are respectively the open and closed balls of center $x$ and radius $\\rho$ in~$\\mathcal{E}$.\n","start":18485,"end":18925}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"锥的含义","source_units":[{"raw_tex":"A nonempty subset $K$ of $\\mathcal{E}$ is called a \\emph{cone} if $x \\in K$ implies $\\alpha x \\in K$ for all $\\alpha \\in [0, \\infty)$ \\cite[\\S 3B]{RockafellarWets}.\n","start":18925,"end":19090}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"距离、投影与基本性质","source_units":[{"raw_tex":"Given $x \\in \\mathcal{E}$, the distance from $x$ to $C$ is $\\dist(x, C) \\coloneq \\min_{y \\in C} \\norm{x-y}$ and the projection of $x$ onto $C$ is $\\proj{C}{x} \\coloneq \\argmin_{y \\in C} \\norm{x-y}$. The function $\\mathcal{E} \\to \\R : x \\mapsto \\dist(x, C)$ is continuous and, for every $x \\in \\mathcal{E}$, the set $\\proj{C}{x}$ is nonempty and compact \\cite[Example~1.20]{RockafellarWets}.\nProposition~\\ref{prop:ProjectedTranslation} is invoked frequently in the rest of the paper.\n","start":19148,"end":19631}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"切向量的序列定义","source_units":[{"raw_tex":"A vector $v \\in \\mathcal{E}$ is said to be \\emph{tangent} to $C$ at $x \\in C$ if there exist sequences $(x_i)_{i \\in \\N}$ in $C$ converging to $x$ and $(t_i)_{i \\in \\N}$ in $(0, \\infty)$ such that the sequence $(\\frac{x_i-x}{t_i})_{i \\in \\N}$ converges to $v$ \\cite[Definition~6.1]{RockafellarWets}. The set of all tangent vectors to $C$ at $x \\in C$ is a closed cone \\cite[Proposition~6.2]{RockafellarWets} called the \\emph{tangent cone} to $C$ at $x$ and denoted by $\\tancone{C}{x}$.\n","start":20713,"end":21199}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"正则法锥的极锥定义","source_units":[{"raw_tex":"The \\emph{regular normal cone} to $C$ at $x \\in C$ is\n\\begin{equation*}\n\\regnorcone{C}{x} \\coloneq \\left\\{v \\in \\mathcal{E} \\mid \\ip{v}{w} \\le 0 \\; \\forall w \\in \\tancone{C}{x}\\right\\},\n\\end{equation*}\nwhich is a closed convex cone \\cite[Definition~6.3 and Proposition~6.5]{RockafellarWets}.\n","start":21199,"end":21491}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"一般法锥","source_units":[{"raw_tex":"A vector $v \\in \\mathcal{E}$ is said to be \\emph{normal (in the general sense)} to $C$ at $x \\in C$ if there exist sequences $(x_i)_{i \\in \\N}$ in $C$ converging to $x$ and $(v_i)_{i \\in \\N}$ converging to $v$ such that, for all $i \\in \\N$, $v_i \\in \\regnorcone{C}{x_i}$ \\cite[Definition~6.3]{RockafellarWets}. The set of all normal vectors to $C$ at $x \\in C$ is a closed cone \\cite[Proposition~6.5]{RockafellarWets} called the \\emph{normal cone} to $C$ at $x$ and denoted by $\\norcone{C}{x}$.\n","start":21491,"end":21986}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"近端法向量的投影刻画","source_units":[{"raw_tex":"A vector $v \\in \\mathcal{E}$ is called a \\emph{proximal normal} to $C$ at $x \\in C$ if there exists $\\oshort{\\alpha} \\in (0, \\infty)$ such that $x \\in \\proj{C}{x+\\oshort{\\alpha}v}$, i.e., $\\oshort{\\alpha}\\norm{v} = \\dist(x+\\oshort{\\alpha}v, C)$, which implies that, for all $\\alpha \\in [0, \\oshort{\\alpha})$, $\\proj{C}{x+\\alpha v} = \\{x\\}$ \\cite[Example~6.16]{RockafellarWets}. The set of all proximal normals to $C$ at $x \\in C$ is a convex cone called the \\emph{proximal normal cone} to $C$ at $x$ and denoted by $\\proxnorcone{C}{x}$.\n","start":21986,"end":22523}]},{"id":"S017","semantic_type":"algorithm","kind":"algorithm","labels":["algo:PGDmap","algo:PGDmap:InitialStepSize","algo:PGDmap:NonmonotoneArmijoCondition"],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"PGD 回溯映射","source_units":[{"raw_tex":"\\begin{algorithm}[H]\n\\caption{$\\pgd$ map}\n\\label{algo:PGDmap}\n\\begin{algorithmic}[1]\n\\Require\n$(\\mathcal{E}, C, f, \\ushort{\\alpha}, \\oshort{\\alpha}, \\beta, c)$, where $\\mathcal{E}$ is a Euclidean vector space, $C$ is a nonempty closed subset of~$\\mathcal{E}$, $f : \\mathcal{E} \\to \\R$ satisfies \\ref{it:continuously_diff}, $0 < \\ushort{\\alpha} \\le \\oshort{\\alpha} < \\infty$, and $\\beta, c \\in (0, 1)$.\n\\Input\n$(x, \\mu)$ with $x \\in C$ and $\\mu \\in [f(x), \\infty)$.\n\\Output\n$y \\in \\pgd(x, \\mu; \\mathcal{E}, C, f, \\ushort{\\alpha}, \\oshort{\\alpha}, \\beta, c)$.\n\n\\State\nChoose $\\alpha \\in [\\ushort{\\alpha}, \\oshort{\\alpha}]$ and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$;\n\\label{algo:PGDmap:InitialStepSize}\n\\While\n{$f(y) > \\mu + c \\ip{\\nabla f(x)}{y-x}$}\n\\label{algo:PGDmap:NonmonotoneArmijoCondition}\n\\State\n$\\alpha \\gets \\alpha \\beta$;\n\\State\nChoose $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$;\n\\EndWhile\n\\State\nReturn $y$.\n\\end{algorithmic}\n\\end{algorithm}\n","start":39371,"end":40318}]},{"id":"S018","semantic_type":"algorithm","kind":"algorithm","labels":["algo:PGD","algo:PGD:PGDmap","algo:PGD:while"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"两种非单调规则的 PGD","source_units":[{"raw_tex":"The $\\pgd$ algorithm is defined as Algorithm~\\ref{algo:PGD}. It is said to be monotone or nonmonotone depending on whether $\\mu_i=f(x_i)$ for all $i$ (that is, $l=0$ for the ``max'' rule, or $p=1$ for the ``average'' rule) or not.\n\n\\begin{algorithm}[H]\n\\caption{$\\pgd$}\n\\label{algo:PGD}\n\\begin{algorithmic}[1]\n\\Require\n$(\\mathcal{E}, C, f, \\ushort{\\alpha}, \\oshort{\\alpha}, \\beta, c, \\text{``nonmonotonicity''})$, where $\\mathcal{E}$ is a Euclidean vector space, $C$ is a nonempty closed subset of~$\\mathcal{E}$, $f : \\mathcal{E} \\to \\R$ satisfies \\ref{it:continuously_diff}, $0 < \\ushort{\\alpha} \\le \\oshort{\\alpha} < \\infty$, $\\beta, c \\in (0, 1)$, and $\\text{``nonmonotonicity''} \\in \\{(\\text{``max''}, l), (\\text{``average''}, p)\\}$ with $l \\in \\N$ and $p \\in (0, 1]$.\n\\Input\n$x_0 \\in C$.\n\\Output\na sequence in $C$.\n\n\\State\n$i \\gets 0$;\n\\State\n$\\mu_{-1} \\gets f(x_0)$;\n\\While\n{$-\\nabla f(x_i) \\notin \\regnorcone{C}{x_i}$}\n\\label{algo:PGD:while}\n\\If\n{$\\text{``nonmonotonicity''} = (\\text{``max''}, l)$}\n\\State\n$\\mu_i \\gets \\max_{j \\in \\{\\max\\{0, i-l\\}, \\dots, i\\}} f(x_j)$;\n\\ElsIf\n{$\\text{``nonmonotonicity''} = (\\text{``average''}, p)$}\n\\State\n$\\mu_i \\gets (1-p)\\mu_{i-1} + pf(x_i)$; \n\\EndIf\n\\State\nChoose $x_{i+1} \\in \\hyperref[algo:PGDmap]{\\pgd}(x_i, \\mu_i; \\mathcal{E}, C, f, \\ushort{\\alpha}, \\oshort{\\alpha}, \\beta, c)$;\n\\label{algo:PGD:PGDmap}\n\\State\n$i \\gets i+1$;\n\\EndWhile\n\\end{algorithmic}\n\\end{algorithm}\n","start":41697,"end":43116}]},{"id":"S019","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:KanzowMehlitzDecay"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"Armijo 条件推出平方下降界","source_units":[{"raw_tex":"\\begin{remark}\nThe Armijo condition\n\\begin{equation*}\nf(y) \\le \\mu + c \\ip{\\nabla f(x)}{y-x}\n\\end{equation*}\nensures that the decrease $\\mu-f(y)$ is at least a fraction $c$ of the opposite of the directional derivative of $f$ at $x$ along the update vector $y-x$. By \\eqref{eq:ProjectedTranslationIP}, this condition implies that\n\\begin{equation}\\label{eq:KanzowMehlitzDecay}\nf(y) \\le \\mu - \\frac{c}{2\\alpha} \\norm{y-x}^2,\n\\end{equation}\nwhich is the condition used in \\cite[Algorithms~3.1 and 4.1]{KanzowMehlitz} and \\cite[Algorithm~3.1]{DeMarchi}. Importantly, all results from \\cite{KanzowMehlitz} hold for both conditions, as is clear from the proofs.\n","start":40319,"end":40975},{"raw_tex":"\\end{remark}\n","start":41093,"end":41106}]},{"id":"S021","semantic_type":"proposition","kind":"proposition","labels":["prop:NonmonotonePGD"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"最大窗口值单调","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:NonmonotonePGD}\nLet $(x_i)_{i \\in \\N}$ be a sequence generated by $\\pgd$ (Algorithm~\\ref{algo:PGD}) using the ``max'' rule. For every $i \\in \\N$, let $g(i) \\in \\argmax_{j \\in \\{\\max\\{0, i-l\\}, \\dots, i\\}} f(x_j)$. Then:\n\\begin{enumerate}\n\\item $(f(x_{g(i)}))_{i \\in \\N}$ is monotonically nonincreasing;\n","start":45746,"end":46081},{"raw_tex":"\\end{enumerate}\n\\end{proposition}","start":46473,"end":46506}]},{"id":"S022","semantic_type":"proposition","kind":"proposition","labels":["prop:NonmonotonePGD"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"最大值规则的次水平包含","source_units":[{"raw_tex":"\\item $(x_i)_{i \\in \\N}$ is contained in the sublevel set~\\eqref{eq:InitialSublevelSet};\n","start":46081,"end":46170}]},{"id":"S023","semantic_type":"proposition","kind":"proposition","labels":["prop:NonmonotonePGD"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"存在聚点时窗口值有有限极限","source_units":[{"raw_tex":"\\item if $x \\in C$ is an accumulation point of $(x_i)_{i \\in \\N}$, then $(f(x_{g(i)}))_{i \\in \\N}$ converges to $\\varphi \\in [f(x), f(x_0)]$;\n","start":46170,"end":46312}]},{"id":"S024","semantic_type":"proposition","kind":"proposition","labels":["prop:NonmonotonePGD"],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"额外条件下函数值全序列收敛","source_units":[{"raw_tex":"\\item if $f$ is bounded from below and uniformly continuous on a set that contains $(x_i)_{i \\in \\N}$, then $(f(x_i))_{i \\in \\N}$ converges to $\\varphi \\in \\R$.\n","start":46312,"end":46473}]},{"id":"S025","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S025"],"declared_external":false,"source_only":true,"title":"最大值规则命题的完整作者论证","source_units":[{"raw_tex":"\\begin{proof}\nThe first two statements are \\cite[Lemma~4.1 and Corollary~4.1]{KanzowMehlitz}.\nFor the third one, let $(x_{i_k})_{k \\in \\N}$ be a subsequence converging to $x$. Since the sequence $(f(x_{g(i)}))_{i\\in\\N}$ is monotonically nonincreasing, it has a limit in $\\R \\cup \\{-\\infty\\}$. Thus,\n\\begin{equation*}\n\\lim_{i \\to \\infty} f(x_{g(i)})\n= \\lim_{k \\to \\infty} f(x_{g(i_k)})\n\\ge \\lim_{k \\to \\infty} f(x_{i_k})\n= f(x)\n> -\\infty.\n\\end{equation*}\nIt remains to prove the fourth statement. From the first statement and because $f$ is bounded from below, $(f(x_{g(i)}))_{i\\in\\N}$ converges to some limit $\\varphi \\in \\R$. Assume, for the sake of contradiction, that $(f(x_i))_{i \\in \\N}$ does not converge to $\\varphi$. Then, there exist $\\rho \\in (0, \\infty)$ and a subsequence $(f(x_{i_j}))_{j \\in \\N}$ contained in $\\R \\setminus [\\varphi-\\rho, \\varphi+\\rho]$. For all $j \\in \\N$, define $p_j \\coloneq g(i_j+l)-i_j \\in \\{0, \\dots, l\\}$. Then, there exist $p \\in \\{0, \\dots, l\\}$ and a subsequence $(p_{j_k})_{k \\in \\N}$ such that, for all $k \\in \\N$, $p_{j_k} = p$. By \\cite[(27)]{KanzowMehlitz} or \\cite[(A.9)]{JiaEtAl}, $(f(x_{g(i)-p}))_{i \\in \\N}$ converges to $\\varphi$. Therefore, $(f(x_{g(i+l)-p}))_{i \\in \\N}$ converges to $\\varphi$. Hence, $(f(x_{g(i_{j_k}+l)-p}))_{k \\in \\N}$ converges to $\\varphi$. This is a contradiction since, for all $k \\in \\N$, $f(x_{g(i_{j_k}+l)-p}) = f(x_{i_{j_k}})$.\n\\end{proof}\n","start":46508,"end":47929}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S026"],"declared_external":false,"source_only":true,"title":"引入前两项外部结论","source_units":[{"raw_tex":"\\begin{proof}\nThe first two statements are \\cite[Lemma~4.1 and Corollary~4.1]{KanzowMehlitz}.\n","start":46508,"end":46602}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S027"],"declared_external":false,"source_only":true,"title":"以聚点排除负无穷极限","source_units":[{"raw_tex":"For the third one, let $(x_{i_k})_{k \\in \\N}$ be a subsequence converging to $x$. Since the sequence $(f(x_{g(i)}))_{i\\in\\N}$ is monotonically nonincreasing, it has a limit in $\\R \\cup \\{-\\infty\\}$. Thus,\n\\begin{equation*}\n\\lim_{i \\to \\infty} f(x_{g(i)})\n= \\lim_{k \\to \\infty} f(x_{g(i_k)})\n\\ge \\lim_{k \\to \\infty} f(x_{i_k})\n= f(x)\n> -\\infty.\n\\end{equation*}\n","start":46602,"end":46962}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S028"],"declared_external":false,"source_only":true,"title":"有限偏移子列的反证","source_units":[{"raw_tex":"It remains to prove the fourth statement. From the first statement and because $f$ is bounded from below, $(f(x_{g(i)}))_{i\\in\\N}$ converges to some limit $\\varphi \\in \\R$. Assume, for the sake of contradiction, that $(f(x_i))_{i \\in \\N}$ does not converge to $\\varphi$. Then, there exist $\\rho \\in (0, \\infty)$ and a subsequence $(f(x_{i_j}))_{j \\in \\N}$ contained in $\\R \\setminus [\\varphi-\\rho, \\varphi+\\rho]$. For all $j \\in \\N$, define $p_j \\coloneq g(i_j+l)-i_j \\in \\{0, \\dots, l\\}$. Then, there exist $p \\in \\{0, \\dots, l\\}$ and a subsequence $(p_{j_k})_{k \\in \\N}$ such that, for all $k \\in \\N$, $p_{j_k} = p$. By \\cite[(27)]{KanzowMehlitz} or \\cite[(A.9)]{JiaEtAl}, $(f(x_{g(i)-p}))_{i \\in \\N}$ converges to $\\varphi$. Therefore, $(f(x_{g(i+l)-p}))_{i \\in \\N}$ converges to $\\varphi$. Hence, $(f(x_{g(i_{j_k}+l)-p}))_{k \\in \\N}$ converges to $\\varphi$. This is a contradiction since, for all $k \\in \\N$, $f(x_{g(i_{j_k}+l)-p}) = f(x_{i_{j_k}})$.\n\\end{proof}\n","start":46962,"end":47929}]},{"id":"S029","semantic_type":"proposition","kind":"proposition","labels":["prop:ProjectedTranslation"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"投影平移的位移界","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:ProjectedTranslation}\nFor all $x \\in C$, $v \\in \\mathcal{E}$, and $y \\in \\proj{C}{x-v}$,\n\\begin{align}\n\\label{eq:ProjectedTranslationDistance}\n\\norm{y-x} &\\le 2 \\norm{v},\\\\\n","start":19632,"end":19837},{"raw_tex":"and the inequalities are strict if $x \\notin \\proj{C}{x-v}$.\n\\end{proposition}","start":19918,"end":19996}]},{"id":"S030","semantic_type":"proposition","kind":"proposition","labels":["prop:ProjectedTranslation"],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"投影平移的内积界","source_units":[{"raw_tex":"For all $x \\in C$, $v \\in \\mathcal{E}$, and $y \\in \\proj{C}{x-v}$,\n","start":19686,"end":19753},{"raw_tex":"\\label{eq:ProjectedTranslationIP}\n2 \\ip{v}{y-x} &\\le - \\norm{y-x}^2,\n\\end{align}\nand the inequalities are strict if $x \\notin \\proj{C}{x-v}$.\n","start":19837,"end":19979}]},{"id":"S031","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"投影平移命题的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nBy definition of the projection, $\\norm{y-(x-v)} \\le \\norm{x-(x-v)} = \\norm{v}$ and the inequality is strict if $x \\notin \\proj{C}{x-v}$. Thus, on the one hand,\n\\begin{equation*}\n\\norm{y-x}\n= \\norm{y-(x-v)-v}\n\\le \\norm{y-(x-v)} + \\norm{-v}\n\\le \\norm{v} + \\norm{v}\n= 2 \\norm{v},\n\\end{equation*}\nand, on the other hand, $\\norm{y-(x-v)}^2 \\le \\norm{v}^2$, which is equivalent to~\\eqref{eq:ProjectedTranslationIP}.\n\\end{proof}","start":19998,"end":20434}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"投影比较与三角不等式","source_units":[{"raw_tex":"\\begin{proof}\nBy definition of the projection, $\\norm{y-(x-v)} \\le \\norm{x-(x-v)} = \\norm{v}$ and the inequality is strict if $x \\notin \\proj{C}{x-v}$. Thus, on the one hand,\n\\begin{equation*}\n\\norm{y-x}\n= \\norm{y-(x-v)-v}\n\\le \\norm{y-(x-v)} + \\norm{-v}\n\\le \\norm{v} + \\norm{v}\n= 2 \\norm{v},\n\\end{equation*}\n","start":19998,"end":20306}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"展开投影距离的平方","source_units":[{"raw_tex":"and, on the other hand, $\\norm{y-(x-v)}^2 \\le \\norm{v}^2$, which is equivalent to~\\eqref{eq:ProjectedTranslationIP}.\n\\end{proof}","start":20306,"end":20434}]},{"id":"S034","semantic_type":"proposition","kind":"proposition","labels":["prop:NonmonotonePGDAverage"],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"平均规则的次水平包含","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:NonmonotonePGDAverage}\nLet $(x_i)_{i \\in \\N}$ be a sequence generated by $\\pgd$ (Algorithm~\\ref{algo:PGD}) using the ``average'' rule. Then:\n\\begin{enumerate}\n\\item $(x_i)_{i \\in \\N}$ is contained in the sublevel set~\\eqref{eq:InitialSublevelSet};\n","start":47930,"end":48210},{"raw_tex":"\\end{enumerate}\n\\end{proposition}","start":48358,"end":48391}]},{"id":"S035","semantic_type":"proposition","kind":"proposition","labels":["prop:NonmonotonePGDAverage"],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"平均参考值与函数值同极限","source_units":[{"raw_tex":"\\item if  $(x_i)_{i \\in \\N}$ has an accumulation point, then $(f(x_i))_{i \\in \\N}$ and $(\\mu_i)_{i\\in\\N}$ converge, toward the same (finite) value.\n","start":48210,"end":48358}]},{"id":"S036","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"平均规则命题的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nThe sequence $(\\mu_i)_{i\\in\\N}$ is monotonically nonincreasing since, for all $i \\in \\N$, $f(x_i) \\le \\mu_{i-1}$, hence $\\mu_i=(1-p)\\mu_{i-1}+ pf(x_i) \\le \\mu_{i-1}$. Therefore, for all $i \\in \\N$,\n\\begin{equation*}\nf(x_i) \\le \\mu_{i-1} \\le \\mu_{-1} = f(x_0),\n\\end{equation*}\nmeaning that $(x_i)_{i\\in\\N}$ is contained in the sublevel set~\\eqref{eq:InitialSublevelSet}.\n\nLet us prove the second item of the proposition. Assume that $(x_i)_{i\\in\\N}$ has an accumulation point $x$. Let $(x_{i_k})_{k\\in\\N}$ be a subsequence converging to~$x$. Observe that\n\\begin{equation*}\n\\lim_{k \\to \\infty} f(x_{i_k}) = f(x),\n\\end{equation*}\nsince $f$ is differentiable, and in particular continuous, at $x$.\nAs $(\\mu_i)_{i\\in\\N}$ is monotonically nonincreasing, it has a limit $\\varphi \\in \\R \\cup \\{-\\infty\\}$. For all $k \\in \\N$,\n\\begin{equation*}\nf(x_{i_k}) \\le \\mu_{i_k-1},\n\\end{equation*}\nand letting $k$ tend to infinity yields\n\\begin{equation*}\nf(x) = \\lim_{k\\to\\infty} f(x_{i_k}) \\le \\lim_{k\\to\\infty} \\mu_{i_k-1} = \\varphi.\n\\end{equation*}\nIn particular, $\\varphi$ is finite.\n\nLet us show that $\\varphi = \\liminf_{i\\to\\infty} f(x_i)$. Let $(x_{j_k})_{k\\in\\N}$ be a subsequence such that\n\\begin{equation*}\n\\lim_{k \\to \\infty} f(x_{j_k}) = \\liminf_{i\\to\\infty} f(x_i).\n\\end{equation*}\nFor all $k \\in \\N$,\n\\begin{align*}\n\\mu_{j_k} = (1-p) \\mu_{j_k-1} + p f(x_{j_k}),\n\\end{align*}\nand letting $k$ tend to infinity yields\n\\begin{equation*}\n\\varphi = (1-p) \\varphi + p \\liminf_{i\\to\\infty} f(x_i).\n\\end{equation*}\nAs $p>0$, this implies $\\varphi = \\liminf_{i\\to\\infty} f(x_i)$ (and, in particular, $\\liminf_{i\\to\\infty} f(x_i) > -\\infty$). To conclude, we observe that, for all $k \\in \\N$,\n\\begin{equation*}\nf(x_k) \\le \\mu_{k-1},\n\\end{equation*}\nhence\n\\begin{equation*}\n\\limsup_{k\\to\\infty} f(x_k) \n\\leq \\lim_{k\\to\\infty} \\mu_{k-1} = \\varphi = \\liminf_{k\\to\\infty} f(x_k).\n\\end{equation*}\nTherefore, $(f(x_k))_{k\\in\\N}$ converges to $\\varphi$.\n\\end{proof}","start":48393,"end":50351}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"参考值单调与次水平包含","source_units":[{"raw_tex":"\\begin{proof}\nThe sequence $(\\mu_i)_{i\\in\\N}$ is monotonically nonincreasing since, for all $i \\in \\N$, $f(x_i) \\le \\mu_{i-1}$, hence $\\mu_i=(1-p)\\mu_{i-1}+ pf(x_i) \\le \\mu_{i-1}$. Therefore, for all $i \\in \\N$,\n\\begin{equation*}\nf(x_i) \\le \\mu_{i-1} \\le \\mu_{-1} = f(x_0),\n\\end{equation*}\nmeaning that $(x_i)_{i\\in\\N}$ is contained in the sublevel set~\\eqref{eq:InitialSublevelSet}.\n","start":48393,"end":48777}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"聚点保证参考极限有限","source_units":[{"raw_tex":"Let us prove the second item of the proposition. Assume that $(x_i)_{i\\in\\N}$ has an accumulation point $x$. Let $(x_{i_k})_{k\\in\\N}$ be a subsequence converging to~$x$. Observe that\n\\begin{equation*}\n\\lim_{k \\to \\infty} f(x_{i_k}) = f(x),\n\\end{equation*}\nsince $f$ is differentiable, and in particular continuous, at $x$.\nAs $(\\mu_i)_{i\\in\\N}$ is monotonically nonincreasing, it has a limit $\\varphi \\in \\R \\cup \\{-\\infty\\}$. For all $k \\in \\N$,\n\\begin{equation*}\nf(x_{i_k}) \\le \\mu_{i_k-1},\n\\end{equation*}\nand letting $k$ tend to infinity yields\n\\begin{equation*}\nf(x) = \\lim_{k\\to\\infty} f(x_{i_k}) \\le \\lim_{k\\to\\infty} \\mu_{i_k-1} = \\varphi.\n\\end{equation*}\nIn particular, $\\varphi$ is finite.\n","start":48778,"end":49478}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"平均递推识别函数值下极限","source_units":[{"raw_tex":"Let us show that $\\varphi = \\liminf_{i\\to\\infty} f(x_i)$. Let $(x_{j_k})_{k\\in\\N}$ be a subsequence such that\n\\begin{equation*}\n\\lim_{k \\to \\infty} f(x_{j_k}) = \\liminf_{i\\to\\infty} f(x_i).\n\\end{equation*}\nFor all $k \\in \\N$,\n\\begin{align*}\n\\mu_{j_k} = (1-p) \\mu_{j_k-1} + p f(x_{j_k}),\n\\end{align*}\nand letting $k$ tend to infinity yields\n\\begin{equation*}\n\\varphi = (1-p) \\varphi + p \\liminf_{i\\to\\infty} f(x_i).\n\\end{equation*}\nAs $p>0$, this implies $\\varphi = \\liminf_{i\\to\\infty} f(x_i)$ (and, in particular, $\\liminf_{i\\to\\infty} f(x_i) > -\\infty$). To conclude, we observe that, for all $k \\in \\N$,\n","start":49479,"end":50086}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"上下极限夹逼","source_units":[{"raw_tex":"\\begin{equation*}\nf(x_k) \\le \\mu_{k-1},\n\\end{equation*}\nhence\n\\begin{equation*}\n\\limsup_{k\\to\\infty} f(x_k) \n\\leq \\lim_{k\\to\\infty} \\mu_{k-1} = \\varphi = \\liminf_{k\\to\\infty} f(x_k).\n\\end{equation*}\nTherefore, $(f(x_k))_{k\\in\\N}$ converges to $\\varphi$.\n\\end{proof}","start":50086,"end":50351}]},{"id":"S041","semantic_type":"assumption","kind":"assumption","labels":["eq:def_w","prop:ArbitrarySmallAlpha"],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"下降切向量与收缩常数","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:ArbitrarySmallAlpha}\nAssume that $f$ satisfies~\\ref{it:continuously_diff}. Let $\\ushort{x} \\in C$ be non-B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}, and $w \\in \\tancone{C}{\\ushort{x}}$ be such that\n\\begin{equation}\n\\label{eq:def_w}\n\\ip{w}{-\\nabla f(\\ushort{x})} > 0.\n\\end{equation}\nLet $\\beta \\in (0, 1)$. Define $\\kappa \\coloneq \\sqrt{1 - \\frac{\\beta \\ip{w}{-\\nabla f(\\ushort{x})}^2}{8 \\norm{w}^2 \\norm{\\nabla f(\\ushort{x})}^2}} \\in (0, 1)$.\nFor every $\\varepsilon \\in (0, \\infty)$, there exist $\\alpha_{\\ushort{x}} \\in (0, \\varepsilon]$ and $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\infty)$ such that, for all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$ and $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$,\n","start":53427,"end":54220}]},{"id":"S042","semantic_type":"proposition","kind":"proposition","labels":["prop:ArbitrarySmallAlpha"],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"任意小尺度的投影距离收缩","source_units":[{"raw_tex":"For every $\\varepsilon \\in (0, \\infty)$, there exist $\\alpha_{\\ushort{x}} \\in (0, \\varepsilon]$ and $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\infty)$ such that, for all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$ and $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$,\n\\begin{equation*}\n\\dist(x-\\alpha\\nabla f(x), C) \\le \\kappa \\alpha \\norm{\\nabla f(x)},\n\\end{equation*}\n","start":53912,"end":54322}]},{"id":"S043","semantic_type":"proposition","kind":"proposition","labels":["prop:ArbitrarySmallAlpha"],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"投影更新的统一下降夹角","source_units":[{"raw_tex":"which implies, for all $y \\in\\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{equation*}\n\\ip{\\nabla f(x)}{y-x} \\le - \\sqrt{1-\\kappa^2} \\norm{\\nabla f(x)} \\norm{y-x}.\n\\end{equation*}\n\\end{proposition}","start":54322,"end":54511}]},{"id":"S044","semantic_type":"proof","kind":"proof","labels":["eq:def_i_1","eq:def_i_2","eq:def_i_3","eq:def_rho0_1","eq:def_rho0_2","eq:def_rho0_3","eq:ip_xy_nabla"],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"小尺度投影估计的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $\\varepsilon \\in (0, \\infty)$ be fixed. We show that there exist $\\alpha_{\\ushort{x}} \\in (0, \\varepsilon]$ and $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\infty)$ satisfying the required property.\n\nLet $(w_i)_{i\\in\\N}$ be a sequence in $C$ converging to $\\ushort{x}$, and $(t_i)_{i\\in\\N}$ be a sequence in $(0, \\infty)$ such that\n\\begin{equation*}\n\\frac{w_i - \\ushort{x}}{t_i} \\xrightarrow{i\\to\\infty} w.\n\\end{equation*}\nFrom the definition of $w$ in~\\eqref{eq:def_w}, it holds for all $i \\in \\N$ large enough that\n\\begin{equation}\n\\label{eq:def_i_1}\n\\ip{w_i - \\ushort{x}}{-\\nabla f(\\ushort{x})} > 0.\n\\end{equation}\nAs $\\frac{1}{t_i}\\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}} \\xrightarrow{i\\to\\infty} \\frac{\\norm{w}^2}{\\ip{w}{-\\nabla f(\\ushort{x})}}$ and $t_i \\xrightarrow{i\\to\\infty} 0$, it also holds for all $i \\in \\N$ large enough that\n\\begin{equation}\n\\label{eq:def_i_2}\n\\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}} < \\varepsilon.\n\\end{equation}\nSimilarly, it holds for all $i \\in \\N$ large enough that\n\\begin{equation}\n\\label{eq:def_i_3}\n\\frac{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}^2}{\\norm{w_i-\\ushort{x}}^2}\n> \\frac{\\ip{w}{-\\nabla f(\\ushort{x})}^2}{2\\norm{w}^2 }.\n\\end{equation}\nFix $i \\in \\N$ satisfying \\eqref{eq:def_i_1}, \\eqref{eq:def_i_2}, and \\eqref{eq:def_i_3}.\nPick $\\alpha_{\\ushort{x}}$ such that\n\\begin{equation*}\n\\frac{\\alpha_{\\ushort{x}}}{2}\n< \\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}}\n< \\alpha_{\\ushort{x}}\n< \\varepsilon.\n\\end{equation*}\nSince $\\nabla f$ is continuous at $\\ushort{x}$, there exists $\\rho_0 \\in (0, \\infty)$ such that, for all $x \\in \\ball[\\ushort{x}, \\rho_0] \\cap C$,\n\\begin{subequations}\n\\begin{gather}\n\\label{eq:def_rho0_1}\n\\ip{w_i-\\ushort{x}}{-\\nabla f(x)} > 0,\\\\\n\\label{eq:def_rho0_2}\n\\frac{\\alpha_{\\ushort{x}}}{2}\n< \\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(x)}}\n< \\alpha_{\\ushort{x}},\\\\\n\\label{eq:def_rho0_3}\n\\frac{\\ip{w_i-\\ushort{x}}{-\\nabla f(x)}^2}{\\norm{w_i-\\ushort{x}}^2 \\norm{\\nabla f(x)}^2}\n> \\frac{\\ip{w}{-\\nabla f(\\ushort{x})}^2}{2\\norm{w}^2 \\norm{\\nabla f(\\ushort{x})}^2}.\n\\end{gather}\n\\end{subequations}\nWe now establish the first inequality we have to prove: for an adequate value of $\\oshort{\\rho}(\\alpha_{\\ushort{x}})$, it holds for all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$ and $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$ that\n\\begin{equation*}\n\\norm{x - \\alpha \\nabla f(x) - y} \\le \\kappa \\alpha \\norm{\\nabla f(x)}\\quad\n\\forall y \\in \\proj{C}{x-\\alpha\\nabla f(x)},\n\\end{equation*}\nwhich is equivalent to $\\dist(x-\\alpha\\nabla f(x), C) \\le \\kappa \\alpha \\norm{\\nabla f(x)}$.\n\nFor the moment, let $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\rho_0]$ be arbitrary. For all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{align*}\n\\norm{x - \\alpha \\nabla f(x) - y}^2\n&\\le \\norm{x - \\alpha \\nabla f(x) - w_i}^2 \\\\\n&= \\norm{\\ushort{x}-\\alpha \\nabla f(x) - w_i}^2 + 2 \\ip{\\ushort{x}-x}{\\alpha \\nabla f(x) + w_i-\\ushort{x}} + \\norm{\\ushort{x}-x}^2 \\\\\n&\\le \\norm{\\ushort{x}-\\alpha \\nabla f(x) - w_i}^2 \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\alpha \\norm{\\nabla f(x)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&\\le \\norm{\\ushort{x}-\\alpha \\nabla f(x) - w_i}^2 \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\alpha \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&= \\alpha^2 \\norm{\\nabla f(x)}^2 -2 \\alpha \\ip{w_i-\\ushort{x}}{-\\nabla f(x)} + \\norm{w_i-\\ushort{x}}^2 \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\alpha \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 - \\alpha \\ip{w_i-\\ushort{x}}{-\\nabla f(x)} \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\frac{\\alpha_{\\ushort{x}}}{\\beta} \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right)\n+ \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2,\n\\end{align*}\nwhere the last inequality follows from \\eqref{eq:def_rho0_2} and the fact that $\\alpha_{\\ushort{x}} \\le \\alpha \\le \\frac{\\alpha_{\\ushort{x}}}{\\beta}$.\nNow, choose $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\rho_0]$ small enough to ensure\n\\begin{align*}\n&2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\frac{\\alpha_{\\ushort{x}}}{\\beta} \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&\\le \\frac{\\alpha_{\\ushort{x}}}{2} \\min_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\ip{w_i-\\ushort{x}}{-\\nabla f(z)}.\n\\end{align*}\nNote that the right-hand side of this inequality is positive, from \\eqref{eq:def_rho0_1}.\nCombining this definition with the previous inequality, we arrive at\n\\begin{align*}\n\\norm{x - \\alpha \\nabla f(x) - y}^2\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 - \\frac{\\alpha}{2} \\ip{w_i-\\ushort{x}}{-\\nabla f(x)}\\\\\n&= \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\ip{w_i-\\ushort{x}}{-\\nabla f(x)}}{2 \\alpha \\norm{\\nabla f(x)}^2}\\right)\\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\beta \\ip{w_i-\\ushort{x}}{-\\nabla f(x)}}{2 \\alpha_{\\ushort{x}} \\norm{\\nabla f(x)}^2}\\right)\n\\text{ as $\\alpha \\le \\frac{\\alpha_{\\ushort{x}}}{\\beta}$}\\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\beta \\ip{w_i-\\ushort{x}}{-\\nabla f(x)}^2}{4 \\norm{w_i-\\ushort{x}}^2 \\norm{\\nabla f(x)}^2}\\right)\n\\text{ from~\\eqref{eq:def_rho0_2}}\\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\beta \\ip{w}{-\\nabla f(\\ushort{x})}^2}{8 \\norm{w}^2 \\norm{\\nabla f(\\ushort{x})}^2}\\right)\n\\text{ from~\\eqref{eq:def_rho0_3}}\\\\\n&= \\kappa^2 \\alpha^2 \\norm{\\nabla f(x)}^2.\n\\end{align*}\nThus, for all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{equation*}\n\\norm{x - \\alpha \\nabla f(x) - y} \\le \\kappa \\alpha \\norm{\\nabla f(x)}.\n\\end{equation*}\nTo conclude, we show that this inequality implies\n\\begin{equation}\n\\label{eq:ip_xy_nabla}\n\\ip{\\frac{y-x}{\\norm{y-x}}}{\\frac{\\nabla f(x)}{\\norm{\\nabla f(x)}}} \\le - \\sqrt{1-\\kappa^2}.\n\\end{equation}\nIndeed, defining $\\theta \\in \\R$ such that $\\ip{\\frac{y-x}{\\norm{y-x}}}{\\frac{\\nabla f(x)}{\\norm{\\nabla f(x)}}} = \\cos(\\theta)$, we have\n\\begin{equation*}\n\\norm{y-x}^2 + 2 \\alpha \\norm{\\nabla f(x)} \\norm{y-x} \\cos(\\theta) + \\alpha^2 \\norm{\\nabla f(x)}^2\n\\le \\alpha^2 \\kappa^2 \\norm{\\nabla f(x)}^2.\n\\end{equation*}\nThis already shows that $\\cos(\\theta) < 0$. In addition, minimizing the left-hand side over all possible values of $\\norm{y-x}$, we get\n\\begin{equation*}\n- \\alpha^2 \\norm{\\nabla f(x)}^2 \\cos^2(\\theta) + \\alpha^2 \\norm{\\nabla f(x)}^2\n\\le \\alpha^2 \\kappa^2 \\norm{\\nabla f(x)}^2,\n\\end{equation*}\nhence $\\cos^2(\\theta) \\ge 1 - \\kappa^2$, which establishes \\eqref{eq:ip_xy_nabla}.\n\\end{proof}","start":54513,"end":61656}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["eq:def_i_1","eq:def_i_2","eq:def_i_3"],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"选取满足三项估计的切向见证","source_units":[{"raw_tex":"\\begin{proof}\nLet $\\varepsilon \\in (0, \\infty)$ be fixed. We show that there exist $\\alpha_{\\ushort{x}} \\in (0, \\varepsilon]$ and $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\infty)$ satisfying the required property.\n\nLet $(w_i)_{i\\in\\N}$ be a sequence in $C$ converging to $\\ushort{x}$, and $(t_i)_{i\\in\\N}$ be a sequence in $(0, \\infty)$ such that\n\\begin{equation*}\n\\frac{w_i - \\ushort{x}}{t_i} \\xrightarrow{i\\to\\infty} w.\n\\end{equation*}\nFrom the definition of $w$ in~\\eqref{eq:def_w}, it holds for all $i \\in \\N$ large enough that\n\\begin{equation}\n\\label{eq:def_i_1}\n\\ip{w_i - \\ushort{x}}{-\\nabla f(\\ushort{x})} > 0.\n\\end{equation}\nAs $\\frac{1}{t_i}\\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}} \\xrightarrow{i\\to\\infty} \\frac{\\norm{w}^2}{\\ip{w}{-\\nabla f(\\ushort{x})}}$ and $t_i \\xrightarrow{i\\to\\infty} 0$, it also holds for all $i \\in \\N$ large enough that\n\\begin{equation}\n\\label{eq:def_i_2}\n\\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}} < \\varepsilon.\n\\end{equation}\nSimilarly, it holds for all $i \\in \\N$ large enough that\n\\begin{equation}\n\\label{eq:def_i_3}\n\\frac{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}^2}{\\norm{w_i-\\ushort{x}}^2}\n> \\frac{\\ip{w}{-\\nabla f(\\ushort{x})}^2}{2\\norm{w}^2 }.\n\\end{equation}\nFix $i \\in \\N$ satisfying \\eqref{eq:def_i_1}, \\eqref{eq:def_i_2}, and \\eqref{eq:def_i_3}.\n","start":54513,"end":55877}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"在见证比值附近选择步长","source_units":[{"raw_tex":"Pick $\\alpha_{\\ushort{x}}$ such that\n\\begin{equation*}\n\\frac{\\alpha_{\\ushort{x}}}{2}\n< \\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(\\ushort{x})}}\n< \\alpha_{\\ushort{x}}\n< \\varepsilon.\n\\end{equation*}\n","start":55877,"end":56092}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":["eq:def_rho0_1","eq:def_rho0_2","eq:def_rho0_3"],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"连续性保持三项严格估计","source_units":[{"raw_tex":"Since $\\nabla f$ is continuous at $\\ushort{x}$, there exists $\\rho_0 \\in (0, \\infty)$ such that, for all $x \\in \\ball[\\ushort{x}, \\rho_0] \\cap C$,\n\\begin{subequations}\n\\begin{gather}\n\\label{eq:def_rho0_1}\n\\ip{w_i-\\ushort{x}}{-\\nabla f(x)} > 0,\\\\\n\\label{eq:def_rho0_2}\n\\frac{\\alpha_{\\ushort{x}}}{2}\n< \\frac{\\norm{w_i-\\ushort{x}}^2}{\\ip{w_i-\\ushort{x}}{-\\nabla f(x)}}\n< \\alpha_{\\ushort{x}},\\\\\n\\label{eq:def_rho0_3}\n\\frac{\\ip{w_i-\\ushort{x}}{-\\nabla f(x)}^2}{\\norm{w_i-\\ushort{x}}^2 \\norm{\\nabla f(x)}^2}\n> \\frac{\\ip{w}{-\\nabla f(\\ushort{x})}^2}{2\\norm{w}^2 \\norm{\\nabla f(\\ushort{x})}^2}.\n\\end{gather}\n\\end{subequations}\n","start":56092,"end":56711}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"以可行见证点比较投影距离","source_units":[{"raw_tex":"We now establish the first inequality we have to prove: for an adequate value of $\\oshort{\\rho}(\\alpha_{\\ushort{x}})$, it holds for all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$ and $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$ that\n\\begin{equation*}\n\\norm{x - \\alpha \\nabla f(x) - y} \\le \\kappa \\alpha \\norm{\\nabla f(x)}\\quad\n\\forall y \\in \\proj{C}{x-\\alpha\\nabla f(x)},\n\\end{equation*}\nwhich is equivalent to $\\dist(x-\\alpha\\nabla f(x), C) \\le \\kappa \\alpha \\norm{\\nabla f(x)}$.\n\nFor the moment, let $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\rho_0]$ be arbitrary. For all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{align*}\n\\norm{x - \\alpha \\nabla f(x) - y}^2\n&\\le \\norm{x - \\alpha \\nabla f(x) - w_i}^2 \\\\\n&= \\norm{\\ushort{x}-\\alpha \\nabla f(x) - w_i}^2 + 2 \\ip{\\ushort{x}-x}{\\alpha \\nabla f(x) + w_i-\\ushort{x}} + \\norm{\\ushort{x}-x}^2 \\\\\n&\\le \\norm{\\ushort{x}-\\alpha \\nabla f(x) - w_i}^2 \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\alpha \\norm{\\nabla f(x)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&\\le \\norm{\\ushort{x}-\\alpha \\nabla f(x) - w_i}^2 \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\alpha \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&= \\alpha^2 \\norm{\\nabla f(x)}^2 -2 \\alpha \\ip{w_i-\\ushort{x}}{-\\nabla f(x)} + \\norm{w_i-\\ushort{x}}^2 \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\alpha \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 - \\alpha \\ip{w_i-\\ushort{x}}{-\\nabla f(x)} \\\\\n&\\qquad + 2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\frac{\\alpha_{\\ushort{x}}}{\\beta} \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right)\n+ \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2,\n\\end{align*}\nwhere the last inequality follows from \\eqref{eq:def_rho0_2} and the fact that $\\alpha_{\\ushort{x}} \\le \\alpha \\le \\frac{\\alpha_{\\ushort{x}}}{\\beta}$.\n","start":56711,"end":58966}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"缩小半径吸收扰动","source_units":[{"raw_tex":"Now, choose $\\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\in (0, \\rho_0]$ small enough to ensure\n\\begin{align*}\n&2 \\oshort{\\rho}(\\alpha_{\\ushort{x}}) \\left(\\frac{\\alpha_{\\ushort{x}}}{\\beta} \\max_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\norm{\\nabla f(z)} + \\norm{w_i-\\ushort{x}}\\right) + \\oshort{\\rho}(\\alpha_{\\ushort{x}})^2 \\\\\n&\\le \\frac{\\alpha_{\\ushort{x}}}{2} \\min_{z \\in \\ball[\\ushort{x}, \\rho_0] \\cap C} \\ip{w_i-\\ushort{x}}{-\\nabla f(z)}.\n\\end{align*}\nNote that the right-hand side of this inequality is positive, from \\eqref{eq:def_rho0_1}.\n","start":58966,"end":59504}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"得到统一收缩因子","source_units":[{"raw_tex":"Combining this definition with the previous inequality, we arrive at\n\\begin{align*}\n\\norm{x - \\alpha \\nabla f(x) - y}^2\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 - \\frac{\\alpha}{2} \\ip{w_i-\\ushort{x}}{-\\nabla f(x)}\\\\\n&= \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\ip{w_i-\\ushort{x}}{-\\nabla f(x)}}{2 \\alpha \\norm{\\nabla f(x)}^2}\\right)\\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\beta \\ip{w_i-\\ushort{x}}{-\\nabla f(x)}}{2 \\alpha_{\\ushort{x}} \\norm{\\nabla f(x)}^2}\\right)\n\\text{ as $\\alpha \\le \\frac{\\alpha_{\\ushort{x}}}{\\beta}$}\\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\beta \\ip{w_i-\\ushort{x}}{-\\nabla f(x)}^2}{4 \\norm{w_i-\\ushort{x}}^2 \\norm{\\nabla f(x)}^2}\\right)\n\\text{ from~\\eqref{eq:def_rho0_2}}\\\\\n&\\le \\alpha^2 \\norm{\\nabla f(x)}^2 \\left(1 - \\frac{\\beta \\ip{w}{-\\nabla f(\\ushort{x})}^2}{8 \\norm{w}^2 \\norm{\\nabla f(\\ushort{x})}^2}\\right)\n\\text{ from~\\eqref{eq:def_rho0_3}}\\\\\n&= \\kappa^2 \\alpha^2 \\norm{\\nabla f(x)}^2.\n\\end{align*}\nThus, for all $x \\in \\ball(\\ushort{x}, \\oshort{\\rho}(\\alpha_{\\ushort{x}})) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{equation*}\n\\norm{x - \\alpha \\nabla f(x) - y} \\le \\kappa \\alpha \\norm{\\nabla f(x)}.\n\\end{equation*}\n","start":59504,"end":60757}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":["eq:ip_xy_nabla"],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"从距离收缩推出夹角界","source_units":[{"raw_tex":"To conclude, we show that this inequality implies\n\\begin{equation}\n\\label{eq:ip_xy_nabla}\n\\ip{\\frac{y-x}{\\norm{y-x}}}{\\frac{\\nabla f(x)}{\\norm{\\nabla f(x)}}} \\le - \\sqrt{1-\\kappa^2}.\n\\end{equation}\nIndeed, defining $\\theta \\in \\R$ such that $\\ip{\\frac{y-x}{\\norm{y-x}}}{\\frac{\\nabla f(x)}{\\norm{\\nabla f(x)}}} = \\cos(\\theta)$, we have\n\\begin{equation*}\n\\norm{y-x}^2 + 2 \\alpha \\norm{\\nabla f(x)} \\norm{y-x} \\cos(\\theta) + \\alpha^2 \\norm{\\nabla f(x)}^2\n\\le \\alpha^2 \\kappa^2 \\norm{\\nabla f(x)}^2.\n\\end{equation*}\nThis already shows that $\\cos(\\theta) < 0$. In addition, minimizing the left-hand side over all possible values of $\\norm{y-x}$, we get\n\\begin{equation*}\n- \\alpha^2 \\norm{\\nabla f(x)}^2 \\cos^2(\\theta) + \\alpha^2 \\norm{\\nabla f(x)}^2\n\\le \\alpha^2 \\kappa^2 \\norm{\\nabla f(x)}^2,\n\\end{equation*}\nhence $\\cos^2(\\theta) \\ge 1 - \\kappa^2$, which establishes \\eqref{eq:ip_xy_nabla}.\n\\end{proof}","start":60757,"end":61656}]},{"id":"S052","semantic_type":"proposition","kind":"proposition","labels":["prop:AcceptedStep"],"member_ids":["S052","S053","S054","S055","S056","S057","S058","S059","S060"],"declared_external":false,"source_only":false,"title":"非 B 驻点附近的可接受尺度","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:AcceptedStep}\nLet $\\ushort{\\alpha} \\in (0, \\infty)$ and $\\beta, c \\in (0, 1)$.\nAssume that $f$ satisfies~\\ref{it:continuously_diff}. Let $\\ushort{x} \\in C$ be non-B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}.\nThere exist $\\alpha_{\\ushort{x}} \\in (0, \\ushort{\\alpha}]$ and $\\rho \\in (0, \\infty)$ such that, for all $x \\in \\ball(\\ushort{x}, \\rho) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{equation*}\nf(y) < f(x) + c \\ip{\\nabla f(x)}{y-x}.\n\\end{equation*}\n\\end{proposition}","start":61658,"end":62248},{"raw_tex":"\\begin{proof}\nDefine $\\kappa$ as in Proposition~\\ref{prop:ArbitrarySmallAlpha}. Let $\\delta \\in (0, \\infty)$ be small enough to ensure\n\\begin{subequations}\n\\begin{align}\n\\label{eq:def_alpha_x_small1}\n\\sup_{y \\in \\ball\\left[\\ushort{x}, \\frac{7\\delta}{2\\beta}\\norm{\\nabla f(\\ushort{x})}\\right] \\cap C \\setminus \\{\\ushort{x}\\}}\n\\frac{|f(y)-f(\\ushort{x}) - \\ip{\\nabla f(\\ushort{x})}{y-\\ushort{x}}|}{\\norm{y-\\ushort{x}}}\n&< \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4\\left(1 + \\frac{8}{3(1-\\kappa)}\\right)},\\\\\n\\label{eq:def_alpha_x_small2}\n\\sup_{y \\in \\ball\\left[\\ushort{x}, \\frac{7\\delta}{2\\beta}\\norm{\\nabla f(\\ushort{x})}\\right] \\cap C}\n\\norm{\\nabla f(y)-\\nabla f(\\ushort{x})}\n&< \\frac{(1-c)\\sqrt{1-\\kappa^2}}{4} \\norm{\\nabla f(\\ushort{x})}.\n\\end{align}\n\\end{subequations}\nThese inequalities are satisfied by all $\\delta$ small enough, from the definition of the gradient for the first one, and because the gradient is continuous at $\\ushort{x}$ for the second one.\n","start":62250,"end":63229},{"raw_tex":"\\begin{proof}\nDefine $\\kappa$ as in Proposition~\\ref{prop:ArbitrarySmallAlpha}. Let $\\delta \\in (0, \\infty)$ be small enough to ensure\n\\begin{subequations}\n\\begin{align}\n\\label{eq:def_alpha_x_small1}\n\\sup_{y \\in \\ball\\left[\\ushort{x}, \\frac{7\\delta}{2\\beta}\\norm{\\nabla f(\\ushort{x})}\\right] \\cap C \\setminus \\{\\ushort{x}\\}}\n\\frac{|f(y)-f(\\ushort{x}) - \\ip{\\nabla f(\\ushort{x})}{y-\\ushort{x}}|}{\\norm{y-\\ushort{x}}}\n&< \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4\\left(1 + \\frac{8}{3(1-\\kappa)}\\right)},\\\\\n\\label{eq:def_alpha_x_small2}\n\\sup_{y \\in \\ball\\left[\\ushort{x}, \\frac{7\\delta}{2\\beta}\\norm{\\nabla f(\\ushort{x})}\\right] \\cap C}\n\\norm{\\nabla f(y)-\\nabla f(\\ushort{x})}\n&< \\frac{(1-c)\\sqrt{1-\\kappa^2}}{4} \\norm{\\nabla f(\\ushort{x})}.\n\\end{align}\n\\end{subequations}\nThese inequalities are satisfied by all $\\delta$ small enough, from the definition of the gradient for the first one, and because the gradient is continuous at $\\ushort{x}$ for the second one.\n\nThen, define $\\varepsilon \\coloneq \\min\\left\\{\\ushort{\\alpha}, \\delta\\right\\}$ and let $\\alpha_{\\ushort{x}} \\in (0, \\varepsilon]$ and $\\oshort{\\rho}(\\alpha_{\\ushort{x}})\\in(0,\\infty)$ be given by Proposition~\\ref{prop:ArbitrarySmallAlpha}.\nDefine\n\\begin{equation*}\n\\rho \\coloneq \\min\\left\\{\\oshort{\\rho}(\\alpha_{\\ushort{x}}), \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})}\\right\\}.\n\\end{equation*}\nNote that, for all $x \\in \\ball(\\ushort{x}, \\rho) \\cap C$,\n\\begin{equation*}\n\\norm{x-\\ushort{x}}\n< \\rho \\le \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})}\n< \\frac{7\\alpha_{\\ushort{x}}}{2\\beta} \\norm{\\nabla f(\\ushort{x})}\n\\leq \\frac{7\\delta }{2\\beta} \\norm{\\nabla f(\\ushort{x})},\n\\end{equation*}\nso that from~\\eqref{eq:def_alpha_x_small2}, $\\norm{\\nabla f(x)-\\nabla f(\\ushort{x})} < \\frac{\\norm{\\nabla f(\\ushort{x})}}{4}$, which implies\n\\begin{align}\n\\nonumber\n\\frac{3}{4} \\norm{\\nabla f(\\ushort{x})}\n& < \\norm{\\nabla f(\\ushort{x})}-\\norm{\\nabla f(x)-\\nabla f(\\ushort{x})} \\\\\n& \\le \\norm{\\nabla f(x)} \\nonumber\\\\\n& \\le \\norm{\\nabla f(\\ushort{x})}+\\norm{\\nabla f(x)-\\nabla f(\\ushort{x})} \\nonumber\\\\\n& < \\frac{5}{4} \\norm{\\nabla f(\\ushort{x})}.\n\\label{eq:nablas_ushortx_x}\n\\end{align}\nFor all $x \\in \\ball(\\ushort{x}, \\rho) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{align}\nf(y)\n&= f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x} \\nonumber\\\\\n&\\qquad + \\left(f(\\ushort{x}) - f(x) - \\ip{\\nabla f(\\ushort{x})}{\\ushort{x}-x}\\right) \\nonumber\\\\\n&\\qquad + \\left(f(y) - f(\\ushort{x}) - \\ip{\\nabla f(\\ushort{x})}{y-\\ushort{x}}\\right) \\nonumber\\\\\n&\\le f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x} +\n\\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4\\left(1 + \\frac{8}{3(1-\\kappa)}\\right)} \\left(\\norm{\\ushort{x}-x} + \\norm{y-\\ushort{x}}\\right),\n\\label{eq:UpperBoundfDiffGrad}\n\\end{align}\nwhere the inequality follows from \\eqref{eq:def_alpha_x_small1}. Observe that\n\\begin{align*}\n\\norm{y-\\ushort{x}}\n&\\le \\norm{y-x} + \\norm{x-\\ushort{x}} \\\\\n&\\le 2 \\alpha \\norm{\\nabla f(x)} + \\rho\n\\text{ from \\eqref{eq:ProjectedTranslationDistance}}\\\\\n&\\le \\frac{2 \\alpha_{\\ushort{x}}}{\\beta} \\norm{\\nabla f(x)}\n+ \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})} \\\\\n&< \\frac{5\\alpha_{\\ushort{x}}}{2\\beta} \\norm{\\nabla f(\\ushort{x})}\n+ \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})} \n\\text{ from \\eqref{eq:nablas_ushortx_x}} \\\\\n&\\le \\frac{7\\alpha_{\\ushort{x}}}{2\\beta} \\norm{\\nabla f(\\ushort{x})}.\n\\end{align*}\nWe continue from \\eqref{eq:UpperBoundfDiffGrad}:\n\\begin{align*}\nf(y)\n& \\le f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x} +\n\\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{\n4\\left(1 + \\frac{8}{3(1-\\kappa)}\\right)} \\left(2\\norm{\\ushort{x}-x} + \\norm{y-x}\\right) \\\\\n&\\overset{\\text{(a)}}{<} f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x}\n+ \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4} \\norm{y-x} \\\\\n&\\le f(x) + \\ip{\\nabla f(x)}{y-x} + \\norm{\\nabla f(\\ushort{x})-\\nabla f(x)} \\norm{y-x} \\\\\n&\\qquad + \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4} \\norm{y-x} \\\\\n&\\le f(x) + \\ip{\\nabla f(x)}{y-x} + \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{2} \\norm{y-x}\n\\text{ from~\\eqref{eq:def_alpha_x_small2}}\\\\\n&< f(x) + \\ip{\\nabla f(x)}{y-x} + (1-c)\\sqrt{1-\\kappa^2} \\norm{\\nabla f(x)}\\norm{y-x}\n\\text{ from~\\eqref{eq:nablas_ushortx_x}}\\\\\n&\\le f(x) + \\ip{\\nabla f(x)}{y-x} - (1-c) \\ip{\\nabla f(x)}{y-x}\n\\text{ from Proposition~\\ref{prop:ArbitrarySmallAlpha}}\\\\\n&= f(x) + c \\ip{\\nabla f(x)}{y-x}.\n\\end{align*}\nInequality (a) is true because\n\\begin{align*}\n\\norm{y-x}\n&\\ge \\alpha \\norm{\\nabla f(x)} - \\norm{x-\\alpha \\nabla f(x)-y}\n\\text{ by the triangle inequality}\\\\\n&= \\alpha \\norm{\\nabla f(x)} - \\dist(x-\\alpha\\nabla f(x), C) \\\\\n&\\ge (1-\\kappa) \\alpha \\norm{\\nabla f(x)}\n\\text{ from Proposition~\\ref{prop:ArbitrarySmallAlpha}}\\\\\n&\\ge \\frac{3}{4} (1-\\kappa) \\rho\n\\text{ from~\\eqref{eq:nablas_ushortx_x}, the definition of $\\rho$, and $\\alpha_{\\ushort{x}} \\le \\alpha$}\\\\\n&> \\frac{3}{4} (1-\\kappa) \\norm{x-\\ushort{x}}.\n\\end{align*}\n\\end{proof}","start":62250,"end":67244},{"raw_tex":"Then, define $\\varepsilon \\coloneq \\min\\left\\{\\ushort{\\alpha}, \\delta\\right\\}$ and let $\\alpha_{\\ushort{x}} \\in (0, \\varepsilon]$ and $\\oshort{\\rho}(\\alpha_{\\ushort{x}})\\in(0,\\infty)$ be given by Proposition~\\ref{prop:ArbitrarySmallAlpha}.\nDefine\n\\begin{equation*}\n\\rho \\coloneq \\min\\left\\{\\oshort{\\rho}(\\alpha_{\\ushort{x}}), \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})}\\right\\}.\n\\end{equation*}\n","start":63230,"end":63629},{"raw_tex":"Note that, for all $x \\in \\ball(\\ushort{x}, \\rho) \\cap C$,\n\\begin{equation*}\n\\norm{x-\\ushort{x}}\n< \\rho \\le \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})}\n< \\frac{7\\alpha_{\\ushort{x}}}{2\\beta} \\norm{\\nabla f(\\ushort{x})}\n\\leq \\frac{7\\delta }{2\\beta} \\norm{\\nabla f(\\ushort{x})},\n\\end{equation*}\nso that from~\\eqref{eq:def_alpha_x_small2}, $\\norm{\\nabla f(x)-\\nabla f(\\ushort{x})} < \\frac{\\norm{\\nabla f(\\ushort{x})}}{4}$, which implies\n\\begin{align}\n\\nonumber\n\\frac{3}{4} \\norm{\\nabla f(\\ushort{x})}\n& < \\norm{\\nabla f(\\ushort{x})}-\\norm{\\nabla f(x)-\\nabla f(\\ushort{x})} \\\\\n& \\le \\norm{\\nabla f(x)} \\nonumber\\\\\n& \\le \\norm{\\nabla f(\\ushort{x})}+\\norm{\\nabla f(x)-\\nabla f(\\ushort{x})} \\nonumber\\\\\n& < \\frac{5}{4} \\norm{\\nabla f(\\ushort{x})}.\n\\label{eq:nablas_ushortx_x}\n\\end{align}\n","start":63629,"end":64413},{"raw_tex":"For all $x \\in \\ball(\\ushort{x}, \\rho) \\cap C$, $\\alpha \\in [\\alpha_{\\ushort{x}}, \\alpha_{\\ushort{x}}/\\beta]$, and $y \\in \\proj{C}{x-\\alpha\\nabla f(x)}$,\n\\begin{align}\nf(y)\n&= f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x} \\nonumber\\\\\n&\\qquad + \\left(f(\\ushort{x}) - f(x) - \\ip{\\nabla f(\\ushort{x})}{\\ushort{x}-x}\\right) \\nonumber\\\\\n&\\qquad + \\left(f(y) - f(\\ushort{x}) - \\ip{\\nabla f(\\ushort{x})}{y-\\ushort{x}}\\right) \\nonumber\\\\\n&\\le f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x} +\n\\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4\\left(1 + \\frac{8}{3(1-\\kappa)}\\right)} \\left(\\norm{\\ushort{x}-x} + \\norm{y-\\ushort{x}}\\right),\n\\label{eq:UpperBoundfDiffGrad}\n\\end{align}\nwhere the inequality follows from \\eqref{eq:def_alpha_x_small1}. Observe that\n","start":64413,"end":65155},{"raw_tex":"\\begin{align*}\n\\norm{y-\\ushort{x}}\n&\\le \\norm{y-x} + \\norm{x-\\ushort{x}} \\\\\n&\\le 2 \\alpha \\norm{\\nabla f(x)} + \\rho\n\\text{ from \\eqref{eq:ProjectedTranslationDistance}}\\\\\n&\\le \\frac{2 \\alpha_{\\ushort{x}}}{\\beta} \\norm{\\nabla f(x)}\n+ \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})} \\\\\n&< \\frac{5\\alpha_{\\ushort{x}}}{2\\beta} \\norm{\\nabla f(\\ushort{x})}\n+ \\alpha_{\\ushort{x}} \\norm{\\nabla f(\\ushort{x})} \n\\text{ from \\eqref{eq:nablas_ushortx_x}} \\\\\n&\\le \\frac{7\\alpha_{\\ushort{x}}}{2\\beta} \\norm{\\nabla f(\\ushort{x})}.\n\\end{align*}\n","start":65155,"end":65684},{"raw_tex":"We continue from \\eqref{eq:UpperBoundfDiffGrad}:\n\\begin{align*}\nf(y)\n& \\le f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x} +\n\\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{\n4\\left(1 + \\frac{8}{3(1-\\kappa)}\\right)} \\left(2\\norm{\\ushort{x}-x} + \\norm{y-x}\\right) \\\\\n&\\overset{\\text{(a)}}{<} f(x) + \\ip{\\nabla f(\\ushort{x})}{y-x}\n+ \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4} \\norm{y-x} \\\\\n&\\le f(x) + \\ip{\\nabla f(x)}{y-x} + \\norm{\\nabla f(\\ushort{x})-\\nabla f(x)} \\norm{y-x} \\\\\n&\\qquad + \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{4} \\norm{y-x} \\\\\n&\\le f(x) + \\ip{\\nabla f(x)}{y-x} + \\frac{(1-c)\\sqrt{1-\\kappa^2}\\norm{\\nabla f(\\ushort{x})}}{2} \\norm{y-x}\n\\text{ from~\\eqref{eq:def_alpha_x_small2}}\\\\\n&< f(x) + \\ip{\\nabla f(x)}{y-x} + (1-c)\\sqrt{1-\\kappa^2} \\norm{\\nabla f(x)}\\norm{y-x}\n\\text{ from~\\eqref{eq:nablas_ushortx_x}}\\\\\n&\\le f(x) + \\ip{\\nabla f(x)}{y-x} - (1-c) \\ip{\\nabla f(x)}{y-x}\n\\text{ from Proposition~\\ref{prop:ArbitrarySmallAlpha}}\\\\\n&= f(x) + c \\ip{\\nabla f(x)}{y-x}.\n\\end{align*}\n","start":65684,"end":66712},{"raw_tex":"Inequality (a) is true because\n\\begin{align*}\n\\norm{y-x}\n&\\ge \\alpha \\norm{\\nabla f(x)} - \\norm{x-\\alpha \\nabla f(x)-y}\n\\text{ by the triangle inequality}\\\\\n&= \\alpha \\norm{\\nabla f(x)} - \\dist(x-\\alpha\\nabla f(x), C) \\\\\n&\\ge (1-\\kappa) \\alpha \\norm{\\nabla f(x)}\n\\text{ from Proposition~\\ref{prop:ArbitrarySmallAlpha}}\\\\\n&\\ge \\frac{3}{4} (1-\\kappa) \\rho\n\\text{ from~\\eqref{eq:nablas_ushortx_x}, the definition of $\\rho$, and $\\alpha_{\\ushort{x}} \\le \\alpha$}\\\\\n&> \\frac{3}{4} (1-\\kappa) \\norm{x-\\ushort{x}}.\n\\end{align*}\n","start":66712,"end":67233}]},{"id":"S061","semantic_type":"proposition","kind":"proposition","labels":["prop:convergence_ik"],"member_ids":["S061","S062","S063","S064","S065","S066","S067"],"declared_external":false,"source_only":false,"title":"收敛子列的后继保持同一极限","source_units":[{"raw_tex":"\\begin{proposition}\n\\label{prop:convergence_ik}\nLet $(x_i)_{i \\in \\N}$ be a sequence generated by $\\pgd$ (Algorithm~\\ref{algo:PGD}), under~\\ref{it:continuously_diff}, and $x$ be an accumulation point.\nThen, for every subsequence $(x_{i_k})_{k\\in\\N}$ converging to $x$, the sequence $(x_{i_k+1})_{k\\in\\N}$ also converges to $x$.\n\\end{proposition}","start":67338,"end":67683},{"raw_tex":"\\begin{proof}\nLet $(x_{i_k})_{k\\in\\N}$ be a subsequence converging to $x$. We show that $(x_{i_k+1})_{k\\in\\N}$ also converges to $x$.\n\nIf the nonmonotonicity rule is set to ``average'', this is a direct consequence of Proposition~\\ref{prop:NonmonotonePGDAverage}. Indeed, for all $i \\in \\N$, from~\\eqref{eq:KanzowMehlitzDecay},\n\\begin{equation*}\nf(x_{i+1}) \\le \\mu_i - \\frac{c}{2 \\oshort{\\alpha}} \\norm{x_{i+1}-x_i}^2 \\le \\mu_i.    \n\\end{equation*}\nFrom Proposition~\\ref{prop:NonmonotonePGDAverage}, $(f(x_{i+1}))_{i\\in\\N}$ and $(\\mu_i)_{i\\in\\N}$ converge to the same limit. Therefore,\n\\begin{equation*}\n \\left(\\mu_i - \\frac{c}{2 \\oshort{\\alpha}} \\norm{x_{i+1}-x_i}^2\\right)_{i\\in\\N}\n\\end{equation*}\nalso converges to this limit.\nThis implies that $(\\norm{x_{i+1}-x_i})_{i\\in\\N}$ converges to $0$, hence $(\\norm{x_{i_k+1}-x_{i_k}})_{k\\in\\N}$ converges to $0$, and $(x_{i_k+1})_{k\\in\\N}$ converges to the same limit as $(x_{i_k})_{k\\in\\N}$, that is, $x$.\n","start":67685,"end":68639},{"raw_tex":"\\begin{proof}\nLet $(x_{i_k})_{k\\in\\N}$ be a subsequence converging to $x$. We show that $(x_{i_k+1})_{k\\in\\N}$ also converges to $x$.\n\nIf the nonmonotonicity rule is set to ``average'', this is a direct consequence of Proposition~\\ref{prop:NonmonotonePGDAverage}. Indeed, for all $i \\in \\N$, from~\\eqref{eq:KanzowMehlitzDecay},\n\\begin{equation*}\nf(x_{i+1}) \\le \\mu_i - \\frac{c}{2 \\oshort{\\alpha}} \\norm{x_{i+1}-x_i}^2 \\le \\mu_i.    \n\\end{equation*}\nFrom Proposition~\\ref{prop:NonmonotonePGDAverage}, $(f(x_{i+1}))_{i\\in\\N}$ and $(\\mu_i)_{i\\in\\N}$ converge to the same limit. Therefore,\n\\begin{equation*}\n \\left(\\mu_i - \\frac{c}{2 \\oshort{\\alpha}} \\norm{x_{i+1}-x_i}^2\\right)_{i\\in\\N}\n\\end{equation*}\nalso converges to this limit.\nThis implies that $(\\norm{x_{i+1}-x_i})_{i\\in\\N}$ converges to $0$, hence $(\\norm{x_{i_k+1}-x_{i_k}})_{k\\in\\N}$ converges to $0$, and $(x_{i_k+1})_{k\\in\\N}$ converges to the same limit as $(x_{i_k})_{k\\in\\N}$, that is, $x$.\n\nNow, let us consider the ``max'' rule case. It suffices to show that $x$ is an accumulation point of every subsequence of $(x_{i_k+1})_{k\\in\\N}$. Thus, we show the following: for every subsequence $(i_{j_k})_{k\\in\\N}$ of $(i_k)_{k\\in\\N}$, there exists a subsequence of $(x_{i_{j_k}+1})_{k\\in\\N}$ that converges to $x$.\nLet $(i_{j_k})_{k\\in\\N}$ be a subsequence of $(i_k)_{k\\in\\N}$.\nFor all $i \\in \\N$, let $g(i) \\in \\argmax_{j \\in \\{\\max\\{0, i-l\\}, \\dots, i\\}} f(x_j)$, as in Proposition~\\ref{prop:NonmonotonePGD}. By the third statement of Proposition~\\ref{prop:NonmonotonePGD}, the sequence $(f(x_{g(i)}))_{i\\in\\N}$ converges to $\\varphi \\in [f(x), f(x_0)]$.\nFor every $k \\in \\N$, letting $\\alpha_{i_{j_k}} \\in (0, \\oshort{\\alpha}]$ be the number such that $x_{i_{j_k}+1} \\in \\proj{C}{x_{i_{j_k}} - \\alpha_{i_{j_k}} \\nabla f(x_{i_{j_k}})}$, by \\eqref{eq:ProjectedTranslationDistance},\n\\begin{equation*}\n\\norm{x_{i_{j_k}+1}-x_{i_{j_k}}}\n\\le 2 \\alpha_{i_{j_k}} \\norm{\\nabla f(x_{i_{j_k}})}\n\\le 2 \\oshort{\\alpha} \\norm{\\nabla f(x_{i_{j_k}})}.\n\\end{equation*}\nThus, since $(x_{i_{j_k}})_{k\\in\\N}$ is bounded and $\\nabla f$ is locally bounded (as it is continuous), the sequence $(x_{i_{j_k}+1})_{k\\in\\N}$ is bounded. If we replace $(i_{j_k})_{k\\in\\N}$ with a subsequence, we can assume that $(x_{i_{j_k}+1})_{k\\in\\N}$ converges.\n\nIterating the reasoning, we can assume that $(x_{i_{j_k}+s})_{k\\in\\N}$ converges to some $x^{s}\\in C$ for every $s \\in \\{0, \\dots, l+1\\}$. By definition of $x$, $x^{0}=x$.\n\nObserve that, from the continuity of $f$,\n\\begin{align*}\nf(x_{g(i_{j_k}+l+1)})\n&= \\max\\{f(x_{i_{j_k}+1}), \\dots, f(x_{i_{j_k}+l+1})\\} \\\\\n&\\to \\max\\{f(x^{1}), \\dots, f(x^{l+1})\\}\n\\text{ when } k \\to \\infty.\n\\end{align*}\nIn particular, there exists $s_1 \\in \\{1, \\dots, l+1\\}$ such that\n\\begin{equation}\n\\label{eq:def_s_1}\nf(x^{s_1}) = \\varphi.\n\\end{equation}\nLet $s_1$ be the smallest such integer.\nFor all $k \\in \\N$, from the condition in line~\\ref{algo:PGDmap:NonmonotoneArmijoCondition} of Algorithm~\\ref{algo:PGDmap} and~\\eqref{eq:KanzowMehlitzDecay},\n\\begin{align*}\nf(x_{i_{j_k}+s_1})\n&\\le f(x_{g(i_{j_k}+s_1-1)}) - \\frac{c}{2\\oshort{\\alpha}} \\norm{x_{i_{j_k}+s_1}-x_{i_{j_k}+s_1-1}}^2,\n\\end{align*}\nand letting $k$ tend to infinity yields\n\\begin{equation*}\n\\varphi\n= f(x^{s_1})\n\\le \\varphi - \\frac{c}{2\\oshort{\\alpha}} \\norm{x^{s_1}-x^{s_1-1}}^2.\n\\end{equation*}\nConsequently, $x^{s_1} = x^{s_1-1}$. In particular, $f(x^{s_1-1}) =f(x^{s_1}) = \\varphi$. Therefore, $s_1=1$, otherwise it would not be the smallest integer satisfying \\eqref{eq:def_s_1}. The equality $x^{s_1} = x^{s_1-1}$ then reduces to $x^{1}=x^{0}=x$ and, when $k \\to \\infty$,\n\\begin{equation*}\nx_{i_{j_k}+1} \\to x^{1} = x.\n\\end{equation*}\n\\end{proof}","start":67685,"end":71365},{"raw_tex":"Now, let us consider the ``max'' rule case. It suffices to show that $x$ is an accumulation point of every subsequence of $(x_{i_k+1})_{k\\in\\N}$. Thus, we show the following: for every subsequence $(i_{j_k})_{k\\in\\N}$ of $(i_k)_{k\\in\\N}$, there exists a subsequence of $(x_{i_{j_k}+1})_{k\\in\\N}$ that converges to $x$.\nLet $(i_{j_k})_{k\\in\\N}$ be a subsequence of $(i_k)_{k\\in\\N}$.\nFor all $i \\in \\N$, let $g(i) \\in \\argmax_{j \\in \\{\\max\\{0, i-l\\}, \\dots, i\\}} f(x_j)$, as in Proposition~\\ref{prop:NonmonotonePGD}. By the third statement of Proposition~\\ref{prop:NonmonotonePGD}, the sequence $(f(x_{g(i)}))_{i\\in\\N}$ converges to $\\varphi \\in [f(x), f(x_0)]$.\n","start":68640,"end":69301},{"raw_tex":"For every $k \\in \\N$, letting $\\alpha_{i_{j_k}} \\in (0, \\oshort{\\alpha}]$ be the number such that $x_{i_{j_k}+1} \\in \\proj{C}{x_{i_{j_k}} - \\alpha_{i_{j_k}} \\nabla f(x_{i_{j_k}})}$, by \\eqref{eq:ProjectedTranslationDistance},\n\\begin{equation*}\n\\norm{x_{i_{j_k}+1}-x_{i_{j_k}}}\n\\le 2 \\alpha_{i_{j_k}} \\norm{\\nabla f(x_{i_{j_k}})}\n\\le 2 \\oshort{\\alpha} \\norm{\\nabla f(x_{i_{j_k}})}.\n\\end{equation*}\nThus, since $(x_{i_{j_k}})_{k\\in\\N}$ is bounded and $\\nabla f$ is locally bounded (as it is continuous), the sequence $(x_{i_{j_k}+1})_{k\\in\\N}$ is bounded. If we replace $(i_{j_k})_{k\\in\\N}$ with a subsequence, we can assume that $(x_{i_{j_k}+1})_{k\\in\\N}$ converges.\n\nIterating the reasoning, we can assume that $(x_{i_{j_k}+s})_{k\\in\\N}$ converges to some $x^{s}\\in C$ for every $s \\in \\{0, \\dots, l+1\\}$. By definition of $x$, $x^{0}=x$.\n","start":69301,"end":70140},{"raw_tex":"Observe that, from the continuity of $f$,\n\\begin{align*}\nf(x_{g(i_{j_k}+l+1)})\n&= \\max\\{f(x_{i_{j_k}+1}), \\dots, f(x_{i_{j_k}+l+1})\\} \\\\\n&\\to \\max\\{f(x^{1}), \\dots, f(x^{l+1})\\}\n\\text{ when } k \\to \\infty.\n\\end{align*}\nIn particular, there exists $s_1 \\in \\{1, \\dots, l+1\\}$ such that\n\\begin{equation}\n\\label{eq:def_s_1}\nf(x^{s_1}) = \\varphi.\n\\end{equation}\nLet $s_1$ be the smallest such integer.\n","start":70141,"end":70539},{"raw_tex":"For all $k \\in \\N$, from the condition in line~\\ref{algo:PGDmap:NonmonotoneArmijoCondition} of Algorithm~\\ref{algo:PGDmap} and~\\eqref{eq:KanzowMehlitzDecay},\n\\begin{align*}\nf(x_{i_{j_k}+s_1})\n&\\le f(x_{g(i_{j_k}+s_1-1)}) - \\frac{c}{2\\oshort{\\alpha}} \\norm{x_{i_{j_k}+s_1}-x_{i_{j_k}+s_1-1}}^2,\n\\end{align*}\nand letting $k$ tend to infinity yields\n\\begin{equation*}\n\\varphi\n= f(x^{s_1})\n\\le \\varphi - \\frac{c}{2\\oshort{\\alpha}} \\norm{x^{s_1}-x^{s_1-1}}^2.\n\\end{equation*}\nConsequently, $x^{s_1} = x^{s_1-1}$. In particular, $f(x^{s_1-1}) =f(x^{s_1}) = \\varphi$. Therefore, $s_1=1$, otherwise it would not be the smallest integer satisfying \\eqref{eq:def_s_1}. The equality $x^{s_1} = x^{s_1-1}$ then reduces to $x^{1}=x^{0}=x$ and, when $k \\to \\infty$,\n\\begin{equation*}\nx_{i_{j_k}+1} \\to x^{1} = x.\n\\end{equation*}\n\\end{proof}","start":70539,"end":71365}]},{"id":"S068","semantic_type":"theorem","kind":"theorem","labels":["thm:PGDaccumulatesAtBStationaryPoints"],"member_ids":["S068"],"declared_external":false,"source_only":false,"title":"PGD 聚于 B 驻点且孤立聚点保证收敛","source_units":[{"raw_tex":"\\begin{theorem}\n\\label{thm:PGDaccumulatesAtBStationaryPoints}\nLet $(x_i)_{i \\in \\N}$ be a sequence generated by $\\pgd$ (Algorithm~\\ref{algo:PGD}), under~\\ref{it:continuously_diff}. Then, all accumulation points of $(x_i)_{i \\in \\N}$ are B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}.\nIf, moreover, $(x_i)_{i \\in \\N}$ has an isolated accumulation point, then $(x_i)_{i \\in \\N}$ converges.\n\\end{theorem}","start":50907,"end":51315}]},{"id":"S069","semantic_type":"proof","kind":"proof","labels":["subsec:proof_accumulates"],"member_ids":["S069"],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\subsection{Third part: proof of Theorem~\\ref{thm:PGDaccumulatesAtBStationaryPoints}}\n\\label{subsec:proof_accumulates}\nLet $\\ushort{x}$ be an accumulation point of $(x_i)_{i\\in\\N}$. Assume, for the sake of contradiction, that $\\ushort{x}$ is not B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}. Let $(x_{i_k})_{k\\in\\N}$ be a subsequence converging to $\\ushort{x}$.\n\nLet $\\alpha_{\\ushort{x}}$ and $\\rho$ be as in Proposition~\\ref{prop:AcceptedStep}. For all $k \\in \\N$ large enough, $x_{i_k} \\! \\in \\ball(\\ushort{x}, \\rho) \\cap C$. Thus, when Algorithm~\\ref{algo:PGDmap} is called at point $x_{i_k}$, the condition in line~\\ref{algo:PGDmap:NonmonotoneArmijoCondition} stops being fulfilled for some $\\alpha_{i_k} \\ge \\alpha_{\\ushort{x}}$, meaning that\n\\begin{equation*}\nx_{i_k+1} \\in \\proj{C}{x_{i_k}-\\alpha_{i_k} \\nabla f(x_{i_k})}\n\\text{ for some } \\alpha_{i_k} \\in [\\alpha_{\\ushort{x}}, \\oshort{\\alpha}].\n\\end{equation*}\nReplacing $(i_k)_{k\\in\\N}$ with a subsequence if necessary, we can assume that $(\\alpha_{i_k})_{k\\in\\N}$ converges to some $\\alpha_{\\lim} \\in [\\alpha_{\\ushort{x}}, \\oshort{\\alpha}]$.\n\nFor all $k \\in \\N$, we have\n\\begin{equation*}\n\\norm{x_{i_k} - \\alpha_{i_k} \\nabla f(x_{i_k}) - x_{i_k+1}}\n= \\dist(x_{i_k} - \\alpha_{i_k} \\nabla f(x_{i_k}), C),\n\\end{equation*}\nand since the distance to a nonempty closed set is a continuous function, we can take this equality to the limit. We use the fact that $x_{i_k+1} \\to \\ushort{x}$ when $k \\to \\infty$, from Proposition~\\ref{prop:convergence_ik}. This yields\n\\begin{equation*}\n\\norm{\\alpha_{\\lim} \\nabla f(\\ushort{x})}\n= \\dist(\\ushort{x} - \\alpha_{\\lim}\\nabla f(\\ushort{x}), C),\n\\end{equation*}\nwhich means that $\\ushort{x} \\in \\proj{C}{\\ushort{x} - \\alpha_{\\lim}\\nabla f(\\ushort{x})}$. In particular, $-\\nabla f(\\ushort{x}) \\in \\proxnorcone{C}{\\ushort{x}} \\subseteq \\regnorcone{C}{\\ushort{x}}$, which contradicts our assumption that $\\ushort{x}$ is not B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}. We have therefore proven that every accumulation point is B-stationary.\n\nFinally, if $(x_i)_{i\\in\\N}$ has an isolated accumulation point, then the sequence $(x_i)_{i\\in\\N}$ converges, from Proposition~\\ref{prop:convergence_ik} and \\cite[Lemma~4.10]{MoreSorensen}.\n\n","start":71367,"end":73608}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":["subsec:proof_accumulates"],"member_ids":["S070"],"declared_external":false,"source_only":false,"title":"假设存在非 B 驻聚点","source_units":[{"raw_tex":"\\subsection{Third part: proof of Theorem~\\ref{thm:PGDaccumulatesAtBStationaryPoints}}\n\\label{subsec:proof_accumulates}\nLet $\\ushort{x}$ be an accumulation point of $(x_i)_{i\\in\\N}$. Assume, for the sake of contradiction, that $\\ushort{x}$ is not B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}. Let $(x_{i_k})_{k\\in\\N}$ be a subsequence converging to $\\ushort{x}$.\n","start":71367,"end":71737}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":["subsec:proof_accumulates"],"member_ids":["S071"],"declared_external":false,"source_only":false,"title":"回溯步长远离零并抽取极限","source_units":[{"raw_tex":"Let $\\alpha_{\\ushort{x}}$ and $\\rho$ be as in Proposition~\\ref{prop:AcceptedStep}. For all $k \\in \\N$ large enough, $x_{i_k} \\! \\in \\ball(\\ushort{x}, \\rho) \\cap C$. Thus, when Algorithm~\\ref{algo:PGDmap} is called at point $x_{i_k}$, the condition in line~\\ref{algo:PGDmap:NonmonotoneArmijoCondition} stops being fulfilled for some $\\alpha_{i_k} \\ge \\alpha_{\\ushort{x}}$, meaning that\n\\begin{equation*}\nx_{i_k+1} \\in \\proj{C}{x_{i_k}-\\alpha_{i_k} \\nabla f(x_{i_k})}\n\\text{ for some } \\alpha_{i_k} \\in [\\alpha_{\\ushort{x}}, \\oshort{\\alpha}].\n\\end{equation*}\nReplacing $(i_k)_{k\\in\\N}$ with a subsequence if necessary, we can assume that $(\\alpha_{i_k})_{k\\in\\N}$ converges to some $\\alpha_{\\lim} \\in [\\alpha_{\\ushort{x}}, \\oshort{\\alpha}]$.\n","start":71738,"end":72478}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":["subsec:proof_accumulates"],"member_ids":["S072"],"declared_external":false,"source_only":false,"title":"对投影距离等式取极限","source_units":[{"raw_tex":"For all $k \\in \\N$, we have\n\\begin{equation*}\n\\norm{x_{i_k} - \\alpha_{i_k} \\nabla f(x_{i_k}) - x_{i_k+1}}\n= \\dist(x_{i_k} - \\alpha_{i_k} \\nabla f(x_{i_k}), C),\n\\end{equation*}\nand since the distance to a nonempty closed set is a continuous function, we can take this equality to the limit. We use the fact that $x_{i_k+1} \\to \\ushort{x}$ when $k \\to \\infty$, from Proposition~\\ref{prop:convergence_ik}. This yields\n\\begin{equation*}\n\\norm{\\alpha_{\\lim} \\nabla f(\\ushort{x})}\n= \\dist(\\ushort{x} - \\alpha_{\\lim}\\nabla f(\\ushort{x}), C),\n\\end{equation*}\n","start":72479,"end":73030}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":["subsec:proof_accumulates"],"member_ids":["S073"],"declared_external":false,"source_only":false,"title":"近端法向量与非 B 驻假设矛盾","source_units":[{"raw_tex":"which means that $\\ushort{x} \\in \\proj{C}{\\ushort{x} - \\alpha_{\\lim}\\nabla f(\\ushort{x})}$. In particular, $-\\nabla f(\\ushort{x}) \\in \\proxnorcone{C}{\\ushort{x}} \\subseteq \\regnorcone{C}{\\ushort{x}}$, which contradicts our assumption that $\\ushort{x}$ is not B-stationary for~\\eqref{eq:MinDiffFunctionClosedSet}. We have therefore proven that every accumulation point is B-stationary.\n","start":73030,"end":73415}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":["subsec:proof_accumulates"],"member_ids":["S074"],"declared_external":false,"source_only":false,"title":"孤立聚点推出全序列收敛","source_units":[{"raw_tex":"Finally, if $(x_i)_{i\\in\\N}$ has an isolated accumulation point, then the sequence $(x_i)_{i\\in\\N}$ converges, from Proposition~\\ref{prop:convergence_ik} and \\cite[Lemma~4.10]{MoreSorensen}.\n\n","start":73416,"end":73608}]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S075"],"declared_external":true,"declaration_text":["原声明先决条件：Standard finite-dimensional compactness, subsequence, continuity, and inner-product facts。"],"source_only":false,"title":"有限维紧性与极限工具","source_units":[]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S076"],"declared_external":true,"declaration_text":["原声明的标准内积事实；极锥定义否定所用的存在性步骤为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"内积估计与极锥否定的标准步骤","source_units":[]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S077"],"declared_external":true,"declaration_text":["原声明先决条件：RockafellarWets, Example 1.20 and Chapter 6；正文另标 Definition 6.1、Propositions 6.2/6.5、Definition 6.3、Example 6.16。"],"source_only":false,"title":"Rockafellar–Wets 的投影与法锥背景","source_units":[]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S078"],"declared_external":true,"declaration_text":["原声明先决条件：KanzowMehlitz, Lemma 4.1 and Corollary 4.1；明确引用见原文第 568 行。"],"source_only":false,"title":"最大值规则的文献输入","source_units":[]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S079"],"declared_external":true,"declaration_text":["原声明先决条件补充：KanzowMehlitz (27) 或 JiaEtAl (A.9)；实际引用见第 577 行。"],"source_only":false,"title":"固定偏移窗口函数值收敛","source_units":[]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S080"],"declared_external":true,"declaration_text":["原声明先决条件：MoreSorensen, Lemma 4.10；明确引用见第 983 行。"],"source_only":false,"title":"孤立聚点收敛的外部引理","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S052","S054":"S052","S055":"S052","S056":"S052","S057":"S052","S058":"S052","S059":"S052","S060":"S052","S061":"S061","S062":"S061","S063":"S061","S064":"S061","S065":"S061","S066":"S061","S067":"S061","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076","S077":"S077","S078":"S078","S079":"S079","S080":"S080"}}}}
{"record_id":"2303.03536v2","paper":{"title":"Certifying the absence of spurious local minima at infinity","authors":["Josz, Cédric","Li, Xiaopeng"],"arxiv_id":"2303.03536","version":"v2","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/2303.03536v2","pdf_url":"https://arxiv.org/pdf/2303.03536v2","doi":"10.1137/22m1479531"},"context":{"raw_tex":"This section contains prerequisites for the proof of \\Cref{thm:main-intro} in the next section. Throughout this paper, unless otherwise specified, we always equip the Euclidean space $\\mathbb{R}^n$ with an inner product $\\langle \\cdot,\\cdot\\rangle$ and its induced norm $\\|\\cdot\\|:=\\sqrt{\\langle \\cdot,\\cdot\\rangle}$. \n\n\nIn this subsection, we present the formal definitions and some useful properties of setwise local minimum and local minimum at infinity mentioned in \\Cref{sec:introduction}. We first review the classical definition of local and global minima. Throughout this paper, $B(x,\\epsilon)$ denotes the open ball centered at $x\\in\\mathbb{R}^n$ with radius $\\epsilon>0$. \n\n\\begin{definition}\n\\label{def:global-local-min}\nA point $x\\in\\mathbb{R}^n$ is a \\textit{local minimum} (respectively, \\textit{global minimum}) of a function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if $f(x)\\leqslant f(y)$ for all $y\\in B(x,\\epsilon)$ for some $\\epsilon>0$ (respectively, $y\\in\\mathbb{R}^n$). A local minimum is \\textit{spurious} if it is not a global minimum.\n\\end{definition}\n\nFrom \\Cref{def:global-local-min}, one can see the definition of a local minimum only considers the landscape of a function at any finite point. To discuss the function landscape at infinity, we generalize the notion of setwise local minimum first proposed in \\cite[Definition 2.5]{joszneurips2018}. \n\n\\begin{definition}[Setwise local minimum]\\label{def:setwise-local-min}\nA nonempty closed subset $S\\subset\\mathbb{R}^n$ is a \\textit{setwise local minimum} of a continuous function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if there exists an open set $U\\subset\\mathbb{R}^n$ such that $S\\subset U$ and $f(x)\\leqslant f(y)$ for all $x\\in S$, $y\\in U\\setminus S$. \\end{definition}\n\nIt is easy to see that a local minimum is a setwise local minimum by taking $S$ to be a singleton. We also define a \\textit{strict setwise local minimum} by replacing $f(x)\\leqslant f(y)$ with $f(x)<f(y)$ in \\Cref{def:setwise-local-min}. \n\n\\begin{definition}[Setwise global minimum]\\label{def:setwise-global-min}\nA subset $S$ of $\\mathbb{R}^n$ is a \\textit{setwise global minimum} of a function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if $S$ is a setwise local minimum of $f$ and $\\inf_{S} f = \\inf_{\\mathbb{R}^n} f$. \n\\end{definition}\n\nNote that $\\inf_S f$ is a shorthand for $\\inf_{x\\in S} f(x)$, and similar for $\\sup_S f$ and $\\max_S f$. Setwise local minima include setwise global minima as a special case, and we say a setwise local minimum is \\textit{spurious} if it is not a setwise global minimum. Note that \\Cref{def:setwise-local-min} is not exactly the same as \\cite[Definition 2.5]{joszneurips2018} because we do not require a setwise local minimum to be a compact set. In other words, a setwise local minimum can be either bounded or unbounded, and we say a (spurious) setwise local minimum is a (spurious) \\textit{local minimum at infinity} if it is unbounded. For example, consider the loss function of a one-hidden layer neural network with sigmoid activation $\\sigma$ and two data points $(1,1)$ and $(-1,-3)$ in \\Cref{fig:setwise}. One can see that $S$  is a setwise local minimum (in particular, a local minimum at infinity) and $U$ is the corresponding open set in \\Cref{def:setwise-local-min}. Finally, observe that $\\mathbb{R}^n$ is a strict setwise local minimum at infinity of any function.\n\n\nNow we introduce one of the most useful properties of setwise local minima in \\Cref{lem:bdd-const}. This property is intuitive and will be used in different scenarios throughout this paper. Let $\\bar{S}$, $S^\\circ$, and $\\partial S := \\bar{S} \\setminus S^\\circ$ respectively denote the closure, interior, and boundary of a subset $S$ of $\\mathbb{R}^n$.\n\n\nFinally, we discuss the case of coercive functions. Recall that a function $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ is coercive if $f(x)\\to\\infty$ as $\\|x\\|\\to\\infty$. \n\n\n\\begin{definition}\\label{def:locally-lipschitz}\nA function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}^m$ is \\textit{locally Lipschitz} if for all $a \\in \\mathbb{R}^n$, there exist positive constants $r$ and $L$ such that \n\t\\begin{equation*}\n\t\t\\forall\\, x,y \\in B(a,r), ~~~ \\|f(x)-f(y)\\| \\leqslant L \\|x-y\\|.\n\t\\end{equation*}\n\\end{definition}\n\nNotice that for a locally Lipschitz function, by \\cite[Theorem 3.2]{evans2018measure}, the derivative exists almost everywhere. Without any assumption on convexity, in order to ensure the existence of a subdifferential, we adopt the notion of Clarke subdifferential. \n\n\\begin{definition}\n\\label{def:Clarke}\n\\cite[p. 27]{clarke1990} Let $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz function. The \\textit{Clarke subdifferential} is the set-valued mapping $\\partial f$ from $\\mathbb{R}^n$ to the subsets of $\\mathbb{R}^n$ defined for all $x \\in \\mathbb{R}^n$ by\n    \\begin{equation*}\n    \\partial f(x) := \\{ s \\in \\mathbb{R}^n ~\\vert~ f^\\circ(x,d) \\geqslant \\langle s , d \\rangle, ~ \\forall\\, d\\in \\mathbb{R}^n \\} \n\\end{equation*}\nwhere\n\\begin{equation*}\n    f^\\circ(x,d) := \\limsup_{\\tiny\\begin{array}{c} y\\rightarrow x \\\\\n    t \\searrow 0\n    \\end{array}\n    } \\frac{f(y+td)-f(y)}{t}.\n\\end{equation*}\n\\end{definition}\n\nIt is well known that for any locally Lipschitz function $f$ and any $x\\in\\mathbb{R}^n$, the Clarke subdifferential $\\partial f(x)$ is a nonempty, convex, and compact set \\cite[Proposition 2.1.2(a)]{clarke1990}. Similar to differentiable functions, a point $x\\in\\mathbb{R}^n$ is called (Clarke) \\textit{critical point} if $0\\in\\partial f(x)$. A real number $y$ is called a (Clarke) \\textit{critical value} of $f$ if there exists a (Clarke) critical point $x\\in\\mathbb{R}^n$ of $f$ such that $f(x)=y$. \n\n\n\\begin{definition}\n    \\label{def:absolute_continuity} \n    \\cite[Definition 1 p. 12]{aubin1984differential}\n    Given two real numbers $a<b$, a function $x:[a,b]\\rightarrow\\mathbb{R}^n$ is \\textit{absolutely continuous} if for all $\\epsilon>0$, there exists $\\delta>0$ such that, for any finite collection of disjoint subintervals $[a_1,b_1],\\hdots,[a_m,b_m]$ of $[a,b]$ such that $\\sum_{i=1}^m b_i-a_i \\leqslant \\delta$, we have $\\sum_{i=1}^m \\|x(b_i) - x(a_i)\\| \\leqslant \\epsilon$.\n\\end{definition}\n\nBy virtue of \\cite[Theorem 20.8]{nielsen1997introduction}, $x:[a,b]\\rightarrow\\mathbb{R}^n$ is absolutely continuous if and only if it is differentiable almost everywhere on $(a,b)$, its derivative $x'$ is Lebesgue integrable, and $x(t) - x(a) = \\int_a^t x'(\\tau)d\\tau$ for all $t\\in [a,b]$. Given a non-compact interval $I$ of $\\mathbb{R}$, $x:I\\rightarrow \\mathbb{R}^n$ is absolutely continuous if it is absolutely continuous on all compact subintervals of $I$.\n\nAn absolutely continuous function $x:[0,\\infty)\\rightarrow\\mathbb{R}^n$ is called a \\textit{subgradient trajectory} of $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ starting at $x_0\\in\\mathbb{R}^n$ if it satisfies the following differential inclusion with initial condition: \n\\begin{equation}\\label{eq:ivp-subgrad}\n    x'(t) \\in -\\partial f(x(t)),\\quad \\text{for almost every $t\\geqslant 0$},\\quad x(0)=x_0,\n\\end{equation}\nwhere ``almost every'' means all elements except for those in a set of zero measure. \n\n\n\\begin{definition}\\label{def:bdd-subgrad}\nA locally Lipschitz lower bounded function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ has \\textit{bounded subgradient trajectories} if for any $x_0\\in\\mathbb{R}^n$, there exists a subgradient trajectory $x$ of $f$ starting at $x_0$ and a constant $r>0$, such that $\\|x(t)\\|\\leqslant r$ for all $t\\geqslant 0$. \n\\end{definition}\n\n\n\\begin{definition}\\label{def:chain-rule}\nLet $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ be locally Lipschitz. We say $f$ \\textit{admits a chain rule} if for any absolutely continuous function $x:[0,\\infty)\\rightarrow \\mathbb{R}^n$, we have\n\\begin{equation*}\n\t(f\\circ x)'(t) = \\langle v, x'(t)\\rangle, \\quad \\forall\\, v\\in\\partial f(x(t)),\n\\end{equation*}\nfor almost every $t\\in [0,\\infty)$. \n\\end{definition}\n\nThus, for any locally Lipschitz function that admits a chain rule, by \\cite[Lemma 5.2]{davis2020stochastic}, the function value is always decreasing in time along the subgradient trajectory. A detailed discussion on what class of functions admits a chain rule can be found in \\cite{bolte2020conservative}. Note that general Lipschitz functions are far from admitting a chain rule since they generically have a maximal Clarke subdifferential \\cite{daniilidis2020pathological,borwein2000lipschitz,daniilidis2019linear}.\n\n\n\\begin{assumption}\\label{assm:funcClass}\nLet $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ be a function such that it  \n\\begin{enumerate}[label={(\\alph*)}]\n    \\item is bounded below, namely, $\\inf_{\\mathbb{R}^n}f > -\\infty$;\n    \\item is locally Lipschitz continuous on $\\mathbb{R}^n$; see \\Cref{def:locally-lipschitz}; \n    \\item admits a chain rule; see \\Cref{def:chain-rule}; \n    \\item has finitely many critical values; see \\Cref{subsec:clarke}; \n    \\item has bounded subgradient trajectories; see \\Cref{def:bdd-subgrad}. \n\\end{enumerate}\n\\end{assumption}\n","units":[{"raw_tex":"This section contains prerequisites for the proof of \\Cref{thm:main-intro} in the next section. Throughout this paper, unless otherwise specified, we always equip the Euclidean space $\\mathbb{R}^n$ with an inner product $\\langle \\cdot,\\cdot\\rangle$ and its induced norm $\\|\\cdot\\|:=\\sqrt{\\langle \\cdot,\\cdot\\rangle}$. \n","start":12310,"end":12629},{"raw_tex":"In this subsection, we present the formal definitions and some useful properties of setwise local minimum and local minimum at infinity mentioned in \\Cref{sec:introduction}. We first review the classical definition of local and global minima. Throughout this paper, $B(x,\\epsilon)$ denotes the open ball centered at $x\\in\\mathbb{R}^n$ with radius $\\epsilon>0$. \n\n\\begin{definition}\n\\label{def:global-local-min}\nA point $x\\in\\mathbb{R}^n$ is a \\textit{local minimum} (respectively, \\textit{global minimum}) of a function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if $f(x)\\leqslant f(y)$ for all $y\\in B(x,\\epsilon)$ for some $\\epsilon>0$ (respectively, $y\\in\\mathbb{R}^n$). A local minimum is \\textit{spurious} if it is not a global minimum.\n\\end{definition}\n\nFrom \\Cref{def:global-local-min}, one can see the definition of a local minimum only considers the landscape of a function at any finite point. To discuss the function landscape at infinity, we generalize the notion of setwise local minimum first proposed in \\cite[Definition 2.5]{joszneurips2018}. \n\n\\begin{definition}[Setwise local minimum]\\label{def:setwise-local-min}\nA nonempty closed subset $S\\subset\\mathbb{R}^n$ is a \\textit{setwise local minimum} of a continuous function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if there exists an open set $U\\subset\\mathbb{R}^n$ such that $S\\subset U$ and $f(x)\\leqslant f(y)$ for all $x\\in S$, $y\\in U\\setminus S$. \\end{definition}\n\nIt is easy to see that a local minimum is a setwise local minimum by taking $S$ to be a singleton. We also define a \\textit{strict setwise local minimum} by replacing $f(x)\\leqslant f(y)$ with $f(x)<f(y)$ in \\Cref{def:setwise-local-min}. \n\n\\begin{definition}[Setwise global minimum]\\label{def:setwise-global-min}\nA subset $S$ of $\\mathbb{R}^n$ is a \\textit{setwise global minimum} of a function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if $S$ is a setwise local minimum of $f$ and $\\inf_{S} f = \\inf_{\\mathbb{R}^n} f$. \n\\end{definition}\n\nNote that $\\inf_S f$ is a shorthand for $\\inf_{x\\in S} f(x)$, and similar for $\\sup_S f$ and $\\max_S f$. Setwise local minima include setwise global minima as a special case, and we say a setwise local minimum is \\textit{spurious} if it is not a setwise global minimum. Note that \\Cref{def:setwise-local-min} is not exactly the same as \\cite[Definition 2.5]{joszneurips2018} because we do not require a setwise local minimum to be a compact set. In other words, a setwise local minimum can be either bounded or unbounded, and we say a (spurious) setwise local minimum is a (spurious) \\textit{local minimum at infinity} if it is unbounded. For example, consider the loss function of a one-hidden layer neural network with sigmoid activation $\\sigma$ and two data points $(1,1)$ and $(-1,-3)$ in \\Cref{fig:setwise}. One can see that $S$  is a setwise local minimum (in particular, a local minimum at infinity) and $U$ is the corresponding open set in \\Cref{def:setwise-local-min}. Finally, observe that $\\mathbb{R}^n$ is a strict setwise local minimum at infinity of any function.\n","start":12688,"end":15732},{"raw_tex":"Now we introduce one of the most useful properties of setwise local minima in \\Cref{lem:bdd-const}. This property is intuitive and will be used in different scenarios throughout this paper. Let $\\bar{S}$, $S^\\circ$, and $\\partial S := \\bar{S} \\setminus S^\\circ$ respectively denote the closure, interior, and boundary of a subset $S$ of $\\mathbb{R}^n$.\n","start":15963,"end":16316},{"raw_tex":"Finally, we discuss the case of coercive functions. Recall that a function $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ is coercive if $f(x)\\to\\infty$ as $\\|x\\|\\to\\infty$. \n","start":19545,"end":19712},{"raw_tex":"\\begin{definition}\\label{def:locally-lipschitz}\nA function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}^m$ is \\textit{locally Lipschitz} if for all $a \\in \\mathbb{R}^n$, there exist positive constants $r$ and $L$ such that \n\t\\begin{equation*}\n\t\t\\forall\\, x,y \\in B(a,r), ~~~ \\|f(x)-f(y)\\| \\leqslant L \\|x-y\\|.\n\t\\end{equation*}\n\\end{definition}\n\nNotice that for a locally Lipschitz function, by \\cite[Theorem 3.2]{evans2018measure}, the derivative exists almost everywhere. Without any assumption on convexity, in order to ensure the existence of a subdifferential, we adopt the notion of Clarke subdifferential. \n\n\\begin{definition}\n\\label{def:Clarke}\n\\cite[p. 27]{clarke1990} Let $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz function. The \\textit{Clarke subdifferential} is the set-valued mapping $\\partial f$ from $\\mathbb{R}^n$ to the subsets of $\\mathbb{R}^n$ defined for all $x \\in \\mathbb{R}^n$ by\n    \\begin{equation*}\n    \\partial f(x) := \\{ s \\in \\mathbb{R}^n ~\\vert~ f^\\circ(x,d) \\geqslant \\langle s , d \\rangle, ~ \\forall\\, d\\in \\mathbb{R}^n \\} \n\\end{equation*}\nwhere\n\\begin{equation*}\n    f^\\circ(x,d) := \\limsup_{\\tiny\\begin{array}{c} y\\rightarrow x \\\\\n    t \\searrow 0\n    \\end{array}\n    } \\frac{f(y+td)-f(y)}{t}.\n\\end{equation*}\n\\end{definition}\n\nIt is well known that for any locally Lipschitz function $f$ and any $x\\in\\mathbb{R}^n$, the Clarke subdifferential $\\partial f(x)$ is a nonempty, convex, and compact set \\cite[Proposition 2.1.2(a)]{clarke1990}. Similar to differentiable functions, a point $x\\in\\mathbb{R}^n$ is called (Clarke) \\textit{critical point} if $0\\in\\partial f(x)$. A real number $y$ is called a (Clarke) \\textit{critical value} of $f$ if there exists a (Clarke) critical point $x\\in\\mathbb{R}^n$ of $f$ such that $f(x)=y$. \n","start":20562,"end":22337},{"raw_tex":"\\begin{definition}\n    \\label{def:absolute_continuity} \n    \\cite[Definition 1 p. 12]{aubin1984differential}\n    Given two real numbers $a<b$, a function $x:[a,b]\\rightarrow\\mathbb{R}^n$ is \\textit{absolutely continuous} if for all $\\epsilon>0$, there exists $\\delta>0$ such that, for any finite collection of disjoint subintervals $[a_1,b_1],\\hdots,[a_m,b_m]$ of $[a,b]$ such that $\\sum_{i=1}^m b_i-a_i \\leqslant \\delta$, we have $\\sum_{i=1}^m \\|x(b_i) - x(a_i)\\| \\leqslant \\epsilon$.\n\\end{definition}\n\nBy virtue of \\cite[Theorem 20.8]{nielsen1997introduction}, $x:[a,b]\\rightarrow\\mathbb{R}^n$ is absolutely continuous if and only if it is differentiable almost everywhere on $(a,b)$, its derivative $x'$ is Lebesgue integrable, and $x(t) - x(a) = \\int_a^t x'(\\tau)d\\tau$ for all $t\\in [a,b]$. Given a non-compact interval $I$ of $\\mathbb{R}$, $x:I\\rightarrow \\mathbb{R}^n$ is absolutely continuous if it is absolutely continuous on all compact subintervals of $I$.\n\nAn absolutely continuous function $x:[0,\\infty)\\rightarrow\\mathbb{R}^n$ is called a \\textit{subgradient trajectory} of $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ starting at $x_0\\in\\mathbb{R}^n$ if it satisfies the following differential inclusion with initial condition: \n\\begin{equation}\\label{eq:ivp-subgrad}\n    x'(t) \\in -\\partial f(x(t)),\\quad \\text{for almost every $t\\geqslant 0$},\\quad x(0)=x_0,\n\\end{equation}\nwhere ``almost every'' means all elements except for those in a set of zero measure. \n","start":22499,"end":23969},{"raw_tex":"\\begin{definition}\\label{def:bdd-subgrad}\nA locally Lipschitz lower bounded function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ has \\textit{bounded subgradient trajectories} if for any $x_0\\in\\mathbb{R}^n$, there exists a subgradient trajectory $x$ of $f$ starting at $x_0$ and a constant $r>0$, such that $\\|x(t)\\|\\leqslant r$ for all $t\\geqslant 0$. \n\\end{definition}\n","start":26391,"end":26756},{"raw_tex":"\\begin{definition}\\label{def:chain-rule}\nLet $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ be locally Lipschitz. We say $f$ \\textit{admits a chain rule} if for any absolutely continuous function $x:[0,\\infty)\\rightarrow \\mathbb{R}^n$, we have\n\\begin{equation*}\n\t(f\\circ x)'(t) = \\langle v, x'(t)\\rangle, \\quad \\forall\\, v\\in\\partial f(x(t)),\n\\end{equation*}\nfor almost every $t\\in [0,\\infty)$. \n\\end{definition}\n\nThus, for any locally Lipschitz function that admits a chain rule, by \\cite[Lemma 5.2]{davis2020stochastic}, the function value is always decreasing in time along the subgradient trajectory. A detailed discussion on what class of functions admits a chain rule can be found in \\cite{bolte2020conservative}. Note that general Lipschitz functions are far from admitting a chain rule since they generically have a maximal Clarke subdifferential \\cite{daniilidis2020pathological,borwein2000lipschitz,daniilidis2019linear}.\n","start":27670,"end":28593},{"raw_tex":"\\begin{assumption}\\label{assm:funcClass}\nLet $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ be a function such that it  \n\\begin{enumerate}[label={(\\alph*)}]\n    \\item is bounded below, namely, $\\inf_{\\mathbb{R}^n}f > -\\infty$;\n    \\item is locally Lipschitz continuous on $\\mathbb{R}^n$; see \\Cref{def:locally-lipschitz}; \n    \\item admits a chain rule; see \\Cref{def:chain-rule}; \n    \\item has finitely many critical values; see \\Cref{subsec:clarke}; \n    \\item has bounded subgradient trajectories; see \\Cref{def:bdd-subgrad}. \n\\end{enumerate}\n\\end{assumption}\n","start":28958,"end":29513}]},"statement":{"raw_tex":"\\begin{theorem}\\label{thm:main-intro}\nSuppose a locally Lipschitz function is bounded below, admits a chain rule, has finitely many critical values, and has bounded subgradient trajectories. Then it has no spurious local minima if and only if it has no spurious setwise local minima. \n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\\label{thm:main-intro}\nSuppose a locally Lipschitz function is bounded below, admits a chain rule, has finitely many critical values, and has bounded subgradient trajectories. Then it has no spurious local minima if and only if it has no spurious setwise local minima. \n\\end{theorem}","start":9110,"end":9408}]},"proof":{"raw_tex":"\\begin{lemma}\\label{lem:bdd-const}\n\\label{lemma:boundary}\nIf $S$ is a setwise local minimum of a continuous function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$, then $f(x)=\\sup_S f$ for all $x\\in\\partial S$.\n\\end{lemma}\n\n\\begin{proof}\nSee \\Cref{app:lem:bdd-const}. \n\\end{proof}\n\n\\begin{proposition}\\label{prop:exist}\nIf $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ is locally Lipschitz and bounded below, then there exists a subgradient trajectory of $f$ starting at arbitrary $x_0\\in\\mathbb{R}^n$. \n\\end{proposition}\n\n\\begin{proof}\nSee \\Cref{app:prop:exist}. \n\\end{proof}\n\n\\begin{proof}[Proof of \\Cref{thm:main-intro}]\nLet $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ be a function satisfying \\Cref{assm:funcClass} If $f$ has no spurious setwise local minima, then $f$ has no spurious local minima. We next prove the converse. Let $S\\subset\\mathbb{R}^n$ be a setwise local minimum of $f$. We seek to show that $S$ is a setwise global minimum of $f$. If $S^\\circ = \\emptyset$, then by \\Cref{lem:bdd-const} $f(x) = \\sup_S f$ for all $x \\in \\partial S = \\bar{S} \\setminus S^\\circ = S$ since $S$ is closed by \\Cref{def:setwise-local-min}. Thus $f$ is constant on $S$. By definition of setwise local minima (\\Cref{def:setwise-local-min}), there exists an open subset $U$ of $\\mathbb{R}^n$ containing $S$ such that the constant value of $f$ on $S$ is less than or equal to $f(y)$ for all $y \\in U \\setminus S$. Thus every point in $S$ is a local minimum of $f$. Since every local minimum of $f$ is a global minimum, $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum according to \\Cref{def:setwise-global-min}. The rest of the proof deals with the case when $S^\\circ \\neq \\emptyset$. Let $C$ be the set of all critical points of $f$ in $S$ and consider the following optimization problem:\n\\begin{equation}\\label{eq:optimization}\n    \\inf_{x \\in C} f(x).\n\\end{equation}\nWe claim that the set of (global) solutions to \\Cref{eq:optimization} is nonempty, and that any solution is a local minimum of $f$ belonging to the setwise local minimum $S$. We first show that the feasible set of \\Cref{eq:optimization} is nonempty.\n\nSince $S^\\circ \\neq \\emptyset$, let $x_0 \\in S^\\circ$. If $x_0 \\in C$, then the feasible set $C$ is nonempty. We thus assume that $x_0 \\notin C$. Since $f$ is locally Lipschitz and bounded below, by \\Cref{prop:exist} there exists a subgradient trajectory $x:[0,\\infty)\\rightarrow\\mathbb{R}^n$ starting at $x_0$. We next show that $x([0,\\infty)) \\subset S$. We reason by contradiction and assume that $S^c \\cap x([0,\\infty)) \\neq \\emptyset$, where $S^c$ is the complement of $S$ in $\\mathbb{R}^n$. Then $S^\\circ$ and $S^c$ are disjoint open subsets of $\\mathbb{R}^n$ such that $S^\\circ \\cap x([0,\\infty)) \\neq \\emptyset$ (the intersection contains $x_0$), $S^c \\cap x([0,\\infty)) \\neq \\emptyset$, and $x([0,\\infty)) = (x([0,\\infty)) \\cap S^\\circ) \\cup (x([0,\\infty)) \\cap S^c) \\subset \\mathbb{R}^n \\setminus \\partial S$ (since\\footnote{If there exists $t>0$ such that $f(x(t)) = f(x(0))$, then $f(x(t)) - f(x(0)) = \\int_0^{t} \\|x'(s)\\|^2ds = 0$ and $x'(s) = 0$ for almost every $s\\in (0,t)$. Since $x'(s) \\in -\\partial f(x(s))$ for almost every $s>0$, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990} we have $0 \\in \\partial f(x(0))$.} $f(x(t)) < f(x(0)) = f(x_0) \\leqslant f(x)$ for all $t > 0$ and $x \\in \\partial S$, where the last inequality follows from \\Cref{lemma:boundary}). Thus the connected set $x([0,\\infty))$ is the union of two relatively open disjoint nonempty sets, which is a contradiction. \n\nSince $f$ has bounded subgradient trajectories and $x(\\cdot)$ is an arbitrary subgradient trajectory starting at $x_0$, by \\Cref{def:bdd-subgrad} and without loss of generality there exists $r>0$ such that $\\|x(t)\\| \\leqslant r$ for all $t\\geqslant 0$. We next show that there exists a critical point of $f$ in $B(0,r) \\cap S$. Suppose that there exist two constants $T,\\epsilon>0$ for which $\\lVert x'(t)\\rVert\\geqslant\\epsilon$ for all $t\\geqslant T$ such that $x'(t)\\in-\\partial f(x(t))$. By \\cite[Lemma 5.2]{davis2020stochastic}, we have $(f \\circ x)'(t) = -\\lVert x'(t)\\rVert^2 \\leqslant -\\epsilon^2$ for almost every $t\\geqslant T$. By integrating, we get $f(x(t)) - f(x(T)) \\leqslant - \\epsilon^2t$ and thus $f(x(t))$ converges to $-\\infty$ as $t\\to\\infty$. This is impossible since $x(t) \\in B(0,r)$ and $f$ is continuous. Hence there exists a time sequence $t_k\\to\\infty$ such that $\\lVert x'(t_k)\\rVert\\to 0$ as $k\\to\\infty$ and $x'(t_k)\\in-\\partial f(x(t_k))$ for all $k\\in\\mathbb{N} := \\{0,1,2,\\hdots\\}$. By the Bolzano–Weierstrass theorem, there exists a subsequence $x(t_{k_j})$ of $x(t_k)$ such that $x(t_{k_j})\\to\\tilde{x}\\in\\mathbb{R}^n$ as $j\\to\\infty$. Since $x'(t_{k_j})\\in -\\partial f(x(t_{k_j}))$, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990} we have $0\\in -\\partial f(\\tilde{x})$. Finally, since $x([0,\\infty)) \\subset S$ and $S$ is closed, we have $\\tilde{x}\\in C$. We obtain that $C\\neq \\emptyset$ as desired.\n\nSince $f$ has finitely many critical values and $C \\neq \\emptyset$, the set of solutions to \\Cref{eq:optimization} is nonempty. Let $x^* \\in C$ be a solution, that is to say $f(x^*) = \\min_C f$. Recall that $C$ is a subset of the setwise local minimum $S$. If $x^*$ is a local minimum of $f$, then it is a global minimum of $f$ in $S$ since every local minimum of $f$ is a global minimum. Thus $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum. For the remainder of the proof, we consider the case where $x^*$ is not a local minimum and show that this leads to a contradiction. We first show that there exists $s_0\\in S^\\circ$ such that $f(s_0)<f(x^*)$. This is clearly true if $x^*\\in S^\\circ$ since one can then find a ball centered at $x^*$ inside $S^\\circ$. If $x^*\\in S \\setminus S^\\circ = \\partial S$, then we reason by contradiction and assume that $f(x) \\geqslant f(x^*)$ for all $x\\in S^\\circ$. By \\Cref{lemma:boundary}, we have $f(x) = f(x^*) = \\sup_S f \\geqslant f(y) \\geqslant f(x^*)$ for all $(x,y)\\in (S \\setminus S^\\circ) \\times S^\\circ$. Hence $f(x^*) = f(x)$ for all $x\\in S$. Since $S$ is a setwise local minimum, there exists an open set $U$ such that $f(x) \\geqslant f(x^*)$ holds for all $x\\in U\\setminus S$. Thus $f(x)\\geqslant f(x^*)$ for all $x\\in U$ and $x^*$ is a local minimum. This yields a contradiction. Hence let $s_0\\in S^\\circ$ be such that $f(s_0)<f(x^*)$. The nonempty closed set $S' := S \\cap [f \\leqslant (f(s_0)+f(x^*))/2]$ is a setwise local minimum of $f$ where $[f\\leqslant \\alpha] := \\{ x \\in \\mathbb{R}^n ~\\vert~ f(x) \\leqslant \\alpha \\}$. Indeed, for all $x \\in S'$ and $y \\in U \\setminus S' = (U \\setminus S) \\cup (U \\setminus [f \\leqslant (f(s_0)+f(x^*))/2])$, we have\\footnote{Indeed, for any sets $A,B$, and $C$ it holds that $A\\setminus (B\\cap C) = (A \\setminus B) \\cup (A\\setminus C)$.} $f(x) \\leqslant (f(s_0)+f(x^*))/2 \\leqslant f(y)$. Since $s_0 \\in S^\\circ$ and $f(s_0)<(f(s_0)+f(x^*))/2$, we have $s_0 \\in S^\\circ \\cap [f\\leqslant (f(s_0)+f(x^*))/2]^\\circ = (S \\cap [f\\leqslant (f(s_0)+f(x^*))/2]))^\\circ = (S')^\\circ$. Hence the setwise local minimum $S'$ has nonempty interior. Also, $S' \\subset S$ and $\\sup_{S'} f < f(x^*) = \\min_C f$ where we remind the reader that $C$ is the set of critical points in $S$. Thus $S'$ is devoid of critical points. However, by the previous\nparagraph, setwise local minima of $f$ with nonempty interior must contain a critical point. This yields a contradiction.\n\\end{proof}\n\n\\section{Proof of \\texorpdfstring{\\Cref{lem:bdd-const}}{}}\\label{app:lem:bdd-const}\nLet $S \\subset \\mathbb{R}^n$ be a setwise local minimum of a continuous function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. Let $U\\supset S$ be an open set such that $f(x)\\leqslant f(y)$ for all $x \\in S$ and $y\\in U\\setminus S$. Note that $S$ is closed, so its boundary is defined by $\\partial S := S\\setminus S^\\circ$. Let $z\\in\\partial S$ and consider any real number $\\epsilon>0$. Since $f(z)+(-\\epsilon,\\epsilon)$ is a neighborhood of $f(z)$, by continuity of $f$, there exists a neighborhood $N(z)$ of $z$ such that $f(N(z))\\subset f(z)+(-\\epsilon,\\epsilon)$. Since $U$ is a neighborhood of $z$, $N'(z):=U \\cap N(z)$ is also a neighborhood of $z$ with $f(N'(z))\\subset f(z)+(-\\epsilon,\\epsilon)$. The set $N'(z)\\cap S$ is nonempty because $z \\in S$ and the set $N'(z)\\setminus S$ is nonempty because $z\\in\\partial S$. For any $x \\in N'(z) \\cap S$ and $y \\in N'(z)\\setminus S$, it follows that\n\\begin{equation*}\n     \\inf_{U \\setminus S} f -\\epsilon \\leqslant f(y)-\\epsilon < f(z) < f(x)+\\epsilon \\leqslant \\sup_S f+\\epsilon \\leqslant \\inf_{U\\setminus S} f +\\epsilon. \n\\end{equation*}\nThe last inequality follows from the definition of setwise local minima. As $\\epsilon>0$ was arbitrary, we deduce that \n\\begin{equation*}\n\\label{inf_sup}\n    \\inf_{U \\setminus S} f = f(z) = \\sup_S f.\n\\end{equation*}\nThus, $f$ is a constant on the boundary of $S$ and $f$ attains its maximum over $S$ on the boundary of $S$. \n\n\n\n\n\\section{Proof of \\texorpdfstring{\\Cref{prop:exist}}{}}\\label{app:prop:exist}\nFor a fixed real number $\\tau>0$, define a sequence $x_k^\\tau$ recurrently by letting $x_0^\\tau:=x_0$ and \n\\begin{equation*}\n    x_{k+1}^\\tau \\in \\arg\\min_{x\\in\\mathbb{R}^n} \\left\\{f(x)+\\frac{\\lVert x-x_k^\\tau\\rVert^2}{2\\tau}\\right\\}, \\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\nA solution exists because $f$ is bounded below and the objective function is coercive. Any solution satisfies \n\\begin{equation*}\n    v_{k+1}^\\tau := \\frac{x_{k+1}^\\tau-x_k^\\tau}{\\tau} \\in -\\partial f(x_{k+1}^\\tau), \\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\nDefine two functions $x^\\tau,\\tilde{x}^\\tau:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ where $\\mathbb{R}_+:=[0,\\infty)$ by\n\\begin{equation*}\n    x^\\tau(t) := x_{k+1}^\\tau,\\quad \\tilde{x}^\\tau(t) := x_k^\\tau + (t-k\\tau)v_{k+1}^\\tau,\\quad \\forall t\\in(k\\tau,(k+1)\\tau]\n\\end{equation*}\nfor all $k\\in\\mathbb{N}$, with initial condition $x^\\tau(0)=\\tilde{x}^\\tau(0)=x_0$. Note that $\\tilde{x}^\\tau$ is absolutely continuous because it is piecewise affine. On the contrary, $x^\\tau$ is not continuous. Also, define $v^{\\tau}:\\mathbb{R}_+\\rightarrow\\mathbb{R}^n$ by \n\\begin{equation*}\n    v^\\tau(t) := v_{k+1}^\\tau, \\quad \\forall t\\in(k\\tau,(k+1)\\tau],\\quad \\forall k\\in\\mathbb{N}, \n\\end{equation*}\nand choose $v^\\tau(0)\\in -\\partial f(x_0)$. Since  $(\\tilde{x}^\\tau)'=v^\\tau$ on $(k\\tau,(k+1)\\tau)$ for all $k \\in \\mathbb{N}$, and $v^\\tau(t)\\in -\\partial f(x^\\tau(t))$ for all $t\\geqslant 0$, we conclude that $(\\tilde{x}^\\tau)'(t)\\in -\\partial f(x^\\tau(t))$ for almost every $t\\in\\mathbb{R}_+$. By optimality of $x_{k+1}^\\tau$, we have \n\\begin{equation*}\n    f(x_{k+1}^\\tau) + \\frac{\\lVert x_{k+1}^\\tau-x_k^\\tau\\rVert^2}{2\\tau} \\leqslant f(x_k^\\tau), \\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\nFor any $l\\in\\mathbb{N}$, we have \n\\begin{equation*}\n    \\sum_{k=0}^l\\frac{\\lVert x_{k+1}^\\tau-x_k^\\tau\\rVert^2}{2\\tau} \\leqslant f(x_0^\\tau) - f(x_{l+1}^\\tau) \\leqslant f(x_0) - \\inf_{\\mathbb{R}^n} f =: C < \\infty\n\\end{equation*}\nsince $f$ is bounded below. Observe that \n\\begin{equation*}\n    \\sum_{k=0}^l\\frac{\\lVert x_{k+1}^\\tau-x_k^\\tau\\rVert^2}{2\\tau} = \\sum_{k=0}^l\\frac{\\tau}{2}\\lVert v_{k+1}^\\tau\\rVert^2 = \\frac{1}{2}\\sum_{k=0}^l\\int_{k\\tau}^{(k+1)\\tau}\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt. \n\\end{equation*}\nFix $T \\geqslant 0$ from now on. From the above, we have\n\\begin{equation}\\label{eq:L2-bounded}\n    \\int_0^T\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt \\leqslant \\sum_{k=0}^{\\lfloor T/\\tau\\rfloor}\\int_{k\\tau}^{(k+1)\\tau}\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt \\leqslant 2C. \n\\end{equation}\nSince $\\tilde{x}^\\tau$ is absolutely continuous, for any $s,t\\in [0,T]$ we have\n\\begin{subequations}\\label{eq:1/2-holder}\n    \\begin{align}\n    \\|\\tilde{x}^\\tau(t)-\\tilde{x}^\\tau(s)\\| &= \\left\\|\\int_s^t(\\tilde{x}^\\tau)'(u)\\;du\\right\\| \\\\\n    &\\leqslant \\left(\\int_0^T\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt\\right)^{1/2}\\vert t-s\\vert^{1/2} \\leqslant \\sqrt{2C}\\vert t-s\\vert^{1/2} \\label{eq:1/2-holder_cs} \n\\end{align}\n\\end{subequations}\nwhere we use the Cauchy-Schwarz inequality. Now one can see $(\\tilde{x}^\\tau)_{\\tau>0}$ is a family of uniformly bounded and equicontinuous functions on the compact interval $[0,T]$. Therefore, by Arzel\\`a-Ascoli theorem \\cite[Theorem 7.25]{rudin1964principles}, there exists a sequence of positive reals $(\\tau_k)_{k\\in\\mathbb{N}}$ such that $\\tau_k\\to 0$ and $\\tilde{x}^{\\tau_k}\\to x^*$ uniformly on $[0,T]$ as $k\\to\\infty$. For all $k\\in \\mathbb{N}$ and $t\\in (k\\tau,(k+1)\\tau]$, we have $\\tilde{x}^\\tau((k+1)\\tau)=x_k^\\tau + \\tau v^\\tau_{k+1} = x^\\tau_{k+1} = x^\\tau(t)$. Thus $\\|\\tilde{x}^\\tau(t)-x^\\tau(t)\\| = \\|\\tilde{x}^\\tau(t)-\\tilde{x}^\\tau((k+1)\\tau)\\| \\leqslant \\sqrt{2C}\\tau^{1/2}$ for all $t\\in [0,T]$ where the inequality is due to \\Cref{eq:1/2-holder} (take $s:=(k+1)\\tau$).\nCombined with the fact that $\\tilde{x}^{\\tau_k}\\to x^*$ uniformly on $[0,T]$, one can see that $x^{\\tau_k}\\to x^*$ uniformly on $[0,T]$. Since \\cref{eq:L2-bounded} implies that $((\\tilde{x}^{\\tau_k})')_{k\\in\\mathbb{N}}$ is a bounded sequence in $L^2([0,T],\\mathbb{R}^n)$, there exists a subsequence $(\\tau_{k_j})_{j\\in\\mathbb{N}}$ such that $(\\tilde{x}^{\\tau_{k_j}})'\\to v^*$ weakly in $L^1([0,T],\\mathbb{R}^n)$ as $j\\to\\infty$ by \\cite[Corollary 14 p. 413]{Fitzpatrick2010}. Since $\\tilde{x}^{\\tau_{k_j}}$ is absolutely continuous, for all $t\\in[0,T]$, we have \n\\begin{equation*}\n    \\tilde{x}^{\\tau_{k_j}}(t) - \\tilde{x}^{\\tau_{k_j}}(0) = \\int_0^t(\\tilde{x}^{\\tau_{k_j}})'(u)\\;du. \n\\end{equation*}\nTake $j\\to\\infty$ on both sides, we have \n\\begin{equation*}\n    x^*(t) - x^*(0) = \\int_0^tv^*(u)\\;du, \n\\end{equation*}\nwhere the convergence of the integral relies on the fact that the constant functions equal to the canonical basis of $\\mathbb{R}^n$ lie in $L^\\infty([0,T],\\mathbb{R}^n)$. Thus, $x^*$ is absolutely continuous and $(x^*)'(t)=v^*(t)$ for almost every $t\\in[0,T]$. Recall that for all $k\\in \\mathbb{N}$, it holds for almost every $t\\in[0,T]$ that\n\\begin{equation*}\n    (\\tilde{x}^{\\tau_k})'(t) = v^{\\tau_k}(t) \\in -\\partial f(x^{\\tau_k}(t)). \n\\end{equation*}\nSince $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d) p. 29]{clarke1990} with nonempty compact values \\cite[2.1.2 Proposition (a) p. 27]{clarke1990}, hence proper upper hemicontinuous \\cite[Proposition 1 p. 60]{aubin1984differential}. In addition, $x^{\\tau_k}\\to x^*$ uniformly on $[0,T]$ and $(\\tilde{x}^{\\tau_k})'\\to (x^*)'$ weakly in $L^1([0,T],\\mathbb{R}^n)$. Therefore,  $(x^*)'(t)\\in-\\partial f(x^*(t))$ for almost all $t\\in[0,T]$ by \\cite[Theorem 1 p. 60]{aubin1984differential}\\footnote{In the theorem we take $F:=-\\partial f$, $X=Y:=\\mathbb{R}^n$, and $I:=[0,T]$.}. The initial condition also holds since $\\tilde{x}^{\\tau}(0)=x_0$ for all $\\tau>0$. \n\\vskip 1ex\nWe have proved that for any initial point $x_0$, there exists $x^*:[0,T]\\rightarrow\\mathbb{R}^n$ such that $(x^*)'(t)=-\\partial f(x^*(t))$ holds for almost every $t\\in [0,T]$ with any $T>0$. Since $T$ is independent of $x_0$, by setting $T=1$, there exists a sequence of absolutely continuous functions $(x_k)_{k\\in\\mathbb{N}}$ such that \n\\begin{equation*}\n    x_k'(t)\\in -\\partial f(x_k(t)), \\quad \\mathrm{for~a.e.}~t \\in [0,1], \\quad x_k(0) = x_{k-1}(1), \n\\end{equation*}\nfor all $k\\in\\mathbb{N}$ where $x_{-1}(0)=x_0$. Therefore, the desired function $x:[0,\\infty)\\rightarrow \\mathbb{R}^n$ can be defined in a piecewise fashion by \n\\begin{equation*}\n    x(t) := x_k(t-k),\\quad t\\in[k,k+1),\\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\nBy construction, $x$ is absolutely continuous on any compact interval $[a,b]\\subset[0,\\infty)$. \n\n","units":[{"raw_tex":"\\begin{lemma}\\label{lem:bdd-const}\n\\label{lemma:boundary}\nIf $S$ is a setwise local minimum of a continuous function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$, then $f(x)=\\sup_S f$ for all $x\\in\\partial S$.\n\\end{lemma}","start":16317,"end":16531},{"raw_tex":"\\begin{proof}\nSee \\Cref{app:lem:bdd-const}. \n\\end{proof}","start":16532,"end":16588},{"raw_tex":"\\begin{proposition}\\label{prop:exist}\nIf $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ is locally Lipschitz and bounded below, then there exists a subgradient trajectory of $f$ starting at arbitrary $x_0\\in\\mathbb{R}^n$. \n\\end{proposition}","start":25092,"end":25323},{"raw_tex":"\\begin{proof}\nSee \\Cref{app:prop:exist}. \n\\end{proof}","start":25324,"end":25377},{"raw_tex":"\\begin{proof}[Proof of \\Cref{thm:main-intro}]\nLet $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ be a function satisfying \\Cref{assm:funcClass} If $f$ has no spurious setwise local minima, then $f$ has no spurious local minima. We next prove the converse. Let $S\\subset\\mathbb{R}^n$ be a setwise local minimum of $f$. We seek to show that $S$ is a setwise global minimum of $f$. If $S^\\circ = \\emptyset$, then by \\Cref{lem:bdd-const} $f(x) = \\sup_S f$ for all $x \\in \\partial S = \\bar{S} \\setminus S^\\circ = S$ since $S$ is closed by \\Cref{def:setwise-local-min}. Thus $f$ is constant on $S$. By definition of setwise local minima (\\Cref{def:setwise-local-min}), there exists an open subset $U$ of $\\mathbb{R}^n$ containing $S$ such that the constant value of $f$ on $S$ is less than or equal to $f(y)$ for all $y \\in U \\setminus S$. Thus every point in $S$ is a local minimum of $f$. Since every local minimum of $f$ is a global minimum, $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum according to \\Cref{def:setwise-global-min}. The rest of the proof deals with the case when $S^\\circ \\neq \\emptyset$. Let $C$ be the set of all critical points of $f$ in $S$ and consider the following optimization problem:\n\\begin{equation}\\label{eq:optimization}\n    \\inf_{x \\in C} f(x).\n\\end{equation}\nWe claim that the set of (global) solutions to \\Cref{eq:optimization} is nonempty, and that any solution is a local minimum of $f$ belonging to the setwise local minimum $S$. We first show that the feasible set of \\Cref{eq:optimization} is nonempty.\n\nSince $S^\\circ \\neq \\emptyset$, let $x_0 \\in S^\\circ$. If $x_0 \\in C$, then the feasible set $C$ is nonempty. We thus assume that $x_0 \\notin C$. Since $f$ is locally Lipschitz and bounded below, by \\Cref{prop:exist} there exists a subgradient trajectory $x:[0,\\infty)\\rightarrow\\mathbb{R}^n$ starting at $x_0$. We next show that $x([0,\\infty)) \\subset S$. We reason by contradiction and assume that $S^c \\cap x([0,\\infty)) \\neq \\emptyset$, where $S^c$ is the complement of $S$ in $\\mathbb{R}^n$. Then $S^\\circ$ and $S^c$ are disjoint open subsets of $\\mathbb{R}^n$ such that $S^\\circ \\cap x([0,\\infty)) \\neq \\emptyset$ (the intersection contains $x_0$), $S^c \\cap x([0,\\infty)) \\neq \\emptyset$, and $x([0,\\infty)) = (x([0,\\infty)) \\cap S^\\circ) \\cup (x([0,\\infty)) \\cap S^c) \\subset \\mathbb{R}^n \\setminus \\partial S$ (since\\footnote{If there exists $t>0$ such that $f(x(t)) = f(x(0))$, then $f(x(t)) - f(x(0)) = \\int_0^{t} \\|x'(s)\\|^2ds = 0$ and $x'(s) = 0$ for almost every $s\\in (0,t)$. Since $x'(s) \\in -\\partial f(x(s))$ for almost every $s>0$, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990} we have $0 \\in \\partial f(x(0))$.} $f(x(t)) < f(x(0)) = f(x_0) \\leqslant f(x)$ for all $t > 0$ and $x \\in \\partial S$, where the last inequality follows from \\Cref{lemma:boundary}). Thus the connected set $x([0,\\infty))$ is the union of two relatively open disjoint nonempty sets, which is a contradiction. \n\nSince $f$ has bounded subgradient trajectories and $x(\\cdot)$ is an arbitrary subgradient trajectory starting at $x_0$, by \\Cref{def:bdd-subgrad} and without loss of generality there exists $r>0$ such that $\\|x(t)\\| \\leqslant r$ for all $t\\geqslant 0$. We next show that there exists a critical point of $f$ in $B(0,r) \\cap S$. Suppose that there exist two constants $T,\\epsilon>0$ for which $\\lVert x'(t)\\rVert\\geqslant\\epsilon$ for all $t\\geqslant T$ such that $x'(t)\\in-\\partial f(x(t))$. By \\cite[Lemma 5.2]{davis2020stochastic}, we have $(f \\circ x)'(t) = -\\lVert x'(t)\\rVert^2 \\leqslant -\\epsilon^2$ for almost every $t\\geqslant T$. By integrating, we get $f(x(t)) - f(x(T)) \\leqslant - \\epsilon^2t$ and thus $f(x(t))$ converges to $-\\infty$ as $t\\to\\infty$. This is impossible since $x(t) \\in B(0,r)$ and $f$ is continuous. Hence there exists a time sequence $t_k\\to\\infty$ such that $\\lVert x'(t_k)\\rVert\\to 0$ as $k\\to\\infty$ and $x'(t_k)\\in-\\partial f(x(t_k))$ for all $k\\in\\mathbb{N} := \\{0,1,2,\\hdots\\}$. By the Bolzano–Weierstrass theorem, there exists a subsequence $x(t_{k_j})$ of $x(t_k)$ such that $x(t_{k_j})\\to\\tilde{x}\\in\\mathbb{R}^n$ as $j\\to\\infty$. Since $x'(t_{k_j})\\in -\\partial f(x(t_{k_j}))$, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990} we have $0\\in -\\partial f(\\tilde{x})$. Finally, since $x([0,\\infty)) \\subset S$ and $S$ is closed, we have $\\tilde{x}\\in C$. We obtain that $C\\neq \\emptyset$ as desired.\n\nSince $f$ has finitely many critical values and $C \\neq \\emptyset$, the set of solutions to \\Cref{eq:optimization} is nonempty. Let $x^* \\in C$ be a solution, that is to say $f(x^*) = \\min_C f$. Recall that $C$ is a subset of the setwise local minimum $S$. If $x^*$ is a local minimum of $f$, then it is a global minimum of $f$ in $S$ since every local minimum of $f$ is a global minimum. Thus $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum. For the remainder of the proof, we consider the case where $x^*$ is not a local minimum and show that this leads to a contradiction. We first show that there exists $s_0\\in S^\\circ$ such that $f(s_0)<f(x^*)$. This is clearly true if $x^*\\in S^\\circ$ since one can then find a ball centered at $x^*$ inside $S^\\circ$. If $x^*\\in S \\setminus S^\\circ = \\partial S$, then we reason by contradiction and assume that $f(x) \\geqslant f(x^*)$ for all $x\\in S^\\circ$. By \\Cref{lemma:boundary}, we have $f(x) = f(x^*) = \\sup_S f \\geqslant f(y) \\geqslant f(x^*)$ for all $(x,y)\\in (S \\setminus S^\\circ) \\times S^\\circ$. Hence $f(x^*) = f(x)$ for all $x\\in S$. Since $S$ is a setwise local minimum, there exists an open set $U$ such that $f(x) \\geqslant f(x^*)$ holds for all $x\\in U\\setminus S$. Thus $f(x)\\geqslant f(x^*)$ for all $x\\in U$ and $x^*$ is a local minimum. This yields a contradiction. Hence let $s_0\\in S^\\circ$ be such that $f(s_0)<f(x^*)$. The nonempty closed set $S' := S \\cap [f \\leqslant (f(s_0)+f(x^*))/2]$ is a setwise local minimum of $f$ where $[f\\leqslant \\alpha] := \\{ x \\in \\mathbb{R}^n ~\\vert~ f(x) \\leqslant \\alpha \\}$. Indeed, for all $x \\in S'$ and $y \\in U \\setminus S' = (U \\setminus S) \\cup (U \\setminus [f \\leqslant (f(s_0)+f(x^*))/2])$, we have\\footnote{Indeed, for any sets $A,B$, and $C$ it holds that $A\\setminus (B\\cap C) = (A \\setminus B) \\cup (A\\setminus C)$.} $f(x) \\leqslant (f(s_0)+f(x^*))/2 \\leqslant f(y)$. Since $s_0 \\in S^\\circ$ and $f(s_0)<(f(s_0)+f(x^*))/2$, we have $s_0 \\in S^\\circ \\cap [f\\leqslant (f(s_0)+f(x^*))/2]^\\circ = (S \\cap [f\\leqslant (f(s_0)+f(x^*))/2]))^\\circ = (S')^\\circ$. Hence the setwise local minimum $S'$ has nonempty interior. Also, $S' \\subset S$ and $\\sup_{S'} f < f(x^*) = \\min_C f$ where we remind the reader that $C$ is the set of critical points in $S$. Thus $S'$ is devoid of critical points. However, by the previous\nparagraph, setwise local minima of $f$ with nonempty interior must contain a critical point. This yields a contradiction.\n\\end{proof}","start":29515,"end":36408},{"raw_tex":"\\section{Proof of \\texorpdfstring{\\Cref{lem:bdd-const}}{}}\\label{app:lem:bdd-const}\nLet $S \\subset \\mathbb{R}^n$ be a setwise local minimum of a continuous function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. Let $U\\supset S$ be an open set such that $f(x)\\leqslant f(y)$ for all $x \\in S$ and $y\\in U\\setminus S$. Note that $S$ is closed, so its boundary is defined by $\\partial S := S\\setminus S^\\circ$. Let $z\\in\\partial S$ and consider any real number $\\epsilon>0$. Since $f(z)+(-\\epsilon,\\epsilon)$ is a neighborhood of $f(z)$, by continuity of $f$, there exists a neighborhood $N(z)$ of $z$ such that $f(N(z))\\subset f(z)+(-\\epsilon,\\epsilon)$. Since $U$ is a neighborhood of $z$, $N'(z):=U \\cap N(z)$ is also a neighborhood of $z$ with $f(N'(z))\\subset f(z)+(-\\epsilon,\\epsilon)$. The set $N'(z)\\cap S$ is nonempty because $z \\in S$ and the set $N'(z)\\setminus S$ is nonempty because $z\\in\\partial S$. For any $x \\in N'(z) \\cap S$ and $y \\in N'(z)\\setminus S$, it follows that\n\\begin{equation*}\n     \\inf_{U \\setminus S} f -\\epsilon \\leqslant f(y)-\\epsilon < f(z) < f(x)+\\epsilon \\leqslant \\sup_S f+\\epsilon \\leqslant \\inf_{U\\setminus S} f +\\epsilon. \n\\end{equation*}\nThe last inequality follows from the definition of setwise local minima. As $\\epsilon>0$ was arbitrary, we deduce that \n\\begin{equation*}\n\\label{inf_sup}\n    \\inf_{U \\setminus S} f = f(z) = \\sup_S f.\n\\end{equation*}\nThus, $f$ is a constant on the boundary of $S$ and $f$ attains its maximum over $S$ on the boundary of $S$. \n\n\n","start":71033,"end":72530},{"raw_tex":"\\section{Proof of \\texorpdfstring{\\Cref{prop:exist}}{}}\\label{app:prop:exist}\nFor a fixed real number $\\tau>0$, define a sequence $x_k^\\tau$ recurrently by letting $x_0^\\tau:=x_0$ and \n\\begin{equation*}\n    x_{k+1}^\\tau \\in \\arg\\min_{x\\in\\mathbb{R}^n} \\left\\{f(x)+\\frac{\\lVert x-x_k^\\tau\\rVert^2}{2\\tau}\\right\\}, \\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\nA solution exists because $f$ is bounded below and the objective function is coercive. Any solution satisfies \n\\begin{equation*}\n    v_{k+1}^\\tau := \\frac{x_{k+1}^\\tau-x_k^\\tau}{\\tau} \\in -\\partial f(x_{k+1}^\\tau), \\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\nDefine two functions $x^\\tau,\\tilde{x}^\\tau:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ where $\\mathbb{R}_+:=[0,\\infty)$ by\n\\begin{equation*}\n    x^\\tau(t) := x_{k+1}^\\tau,\\quad \\tilde{x}^\\tau(t) := x_k^\\tau + (t-k\\tau)v_{k+1}^\\tau,\\quad \\forall t\\in(k\\tau,(k+1)\\tau]\n\\end{equation*}\nfor all $k\\in\\mathbb{N}$, with initial condition $x^\\tau(0)=\\tilde{x}^\\tau(0)=x_0$. Note that $\\tilde{x}^\\tau$ is absolutely continuous because it is piecewise affine. On the contrary, $x^\\tau$ is not continuous. Also, define $v^{\\tau}:\\mathbb{R}_+\\rightarrow\\mathbb{R}^n$ by \n\\begin{equation*}\n    v^\\tau(t) := v_{k+1}^\\tau, \\quad \\forall t\\in(k\\tau,(k+1)\\tau],\\quad \\forall k\\in\\mathbb{N}, \n\\end{equation*}\nand choose $v^\\tau(0)\\in -\\partial f(x_0)$. Since  $(\\tilde{x}^\\tau)'=v^\\tau$ on $(k\\tau,(k+1)\\tau)$ for all $k \\in \\mathbb{N}$, and $v^\\tau(t)\\in -\\partial f(x^\\tau(t))$ for all $t\\geqslant 0$, we conclude that $(\\tilde{x}^\\tau)'(t)\\in -\\partial f(x^\\tau(t))$ for almost every $t\\in\\mathbb{R}_+$. By optimality of $x_{k+1}^\\tau$, we have \n\\begin{equation*}\n    f(x_{k+1}^\\tau) + \\frac{\\lVert x_{k+1}^\\tau-x_k^\\tau\\rVert^2}{2\\tau} \\leqslant f(x_k^\\tau), \\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\nFor any $l\\in\\mathbb{N}$, we have \n\\begin{equation*}\n    \\sum_{k=0}^l\\frac{\\lVert x_{k+1}^\\tau-x_k^\\tau\\rVert^2}{2\\tau} \\leqslant f(x_0^\\tau) - f(x_{l+1}^\\tau) \\leqslant f(x_0) - \\inf_{\\mathbb{R}^n} f =: C < \\infty\n\\end{equation*}\nsince $f$ is bounded below. Observe that \n\\begin{equation*}\n    \\sum_{k=0}^l\\frac{\\lVert x_{k+1}^\\tau-x_k^\\tau\\rVert^2}{2\\tau} = \\sum_{k=0}^l\\frac{\\tau}{2}\\lVert v_{k+1}^\\tau\\rVert^2 = \\frac{1}{2}\\sum_{k=0}^l\\int_{k\\tau}^{(k+1)\\tau}\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt. \n\\end{equation*}\nFix $T \\geqslant 0$ from now on. From the above, we have\n\\begin{equation}\\label{eq:L2-bounded}\n    \\int_0^T\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt \\leqslant \\sum_{k=0}^{\\lfloor T/\\tau\\rfloor}\\int_{k\\tau}^{(k+1)\\tau}\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt \\leqslant 2C. \n\\end{equation}\nSince $\\tilde{x}^\\tau$ is absolutely continuous, for any $s,t\\in [0,T]$ we have\n\\begin{subequations}\\label{eq:1/2-holder}\n    \\begin{align}\n    \\|\\tilde{x}^\\tau(t)-\\tilde{x}^\\tau(s)\\| &= \\left\\|\\int_s^t(\\tilde{x}^\\tau)'(u)\\;du\\right\\| \\\\\n    &\\leqslant \\left(\\int_0^T\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt\\right)^{1/2}\\vert t-s\\vert^{1/2} \\leqslant \\sqrt{2C}\\vert t-s\\vert^{1/2} \\label{eq:1/2-holder_cs} \n\\end{align}\n\\end{subequations}\nwhere we use the Cauchy-Schwarz inequality. Now one can see $(\\tilde{x}^\\tau)_{\\tau>0}$ is a family of uniformly bounded and equicontinuous functions on the compact interval $[0,T]$. Therefore, by Arzel\\`a-Ascoli theorem \\cite[Theorem 7.25]{rudin1964principles}, there exists a sequence of positive reals $(\\tau_k)_{k\\in\\mathbb{N}}$ such that $\\tau_k\\to 0$ and $\\tilde{x}^{\\tau_k}\\to x^*$ uniformly on $[0,T]$ as $k\\to\\infty$. For all $k\\in \\mathbb{N}$ and $t\\in (k\\tau,(k+1)\\tau]$, we have $\\tilde{x}^\\tau((k+1)\\tau)=x_k^\\tau + \\tau v^\\tau_{k+1} = x^\\tau_{k+1} = x^\\tau(t)$. Thus $\\|\\tilde{x}^\\tau(t)-x^\\tau(t)\\| = \\|\\tilde{x}^\\tau(t)-\\tilde{x}^\\tau((k+1)\\tau)\\| \\leqslant \\sqrt{2C}\\tau^{1/2}$ for all $t\\in [0,T]$ where the inequality is due to \\Cref{eq:1/2-holder} (take $s:=(k+1)\\tau$).\nCombined with the fact that $\\tilde{x}^{\\tau_k}\\to x^*$ uniformly on $[0,T]$, one can see that $x^{\\tau_k}\\to x^*$ uniformly on $[0,T]$. Since \\cref{eq:L2-bounded} implies that $((\\tilde{x}^{\\tau_k})')_{k\\in\\mathbb{N}}$ is a bounded sequence in $L^2([0,T],\\mathbb{R}^n)$, there exists a subsequence $(\\tau_{k_j})_{j\\in\\mathbb{N}}$ such that $(\\tilde{x}^{\\tau_{k_j}})'\\to v^*$ weakly in $L^1([0,T],\\mathbb{R}^n)$ as $j\\to\\infty$ by \\cite[Corollary 14 p. 413]{Fitzpatrick2010}. Since $\\tilde{x}^{\\tau_{k_j}}$ is absolutely continuous, for all $t\\in[0,T]$, we have \n\\begin{equation*}\n    \\tilde{x}^{\\tau_{k_j}}(t) - \\tilde{x}^{\\tau_{k_j}}(0) = \\int_0^t(\\tilde{x}^{\\tau_{k_j}})'(u)\\;du. \n\\end{equation*}\nTake $j\\to\\infty$ on both sides, we have \n\\begin{equation*}\n    x^*(t) - x^*(0) = \\int_0^tv^*(u)\\;du, \n\\end{equation*}\nwhere the convergence of the integral relies on the fact that the constant functions equal to the canonical basis of $\\mathbb{R}^n$ lie in $L^\\infty([0,T],\\mathbb{R}^n)$. Thus, $x^*$ is absolutely continuous and $(x^*)'(t)=v^*(t)$ for almost every $t\\in[0,T]$. Recall that for all $k\\in \\mathbb{N}$, it holds for almost every $t\\in[0,T]$ that\n\\begin{equation*}\n    (\\tilde{x}^{\\tau_k})'(t) = v^{\\tau_k}(t) \\in -\\partial f(x^{\\tau_k}(t)). \n\\end{equation*}\nSince $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d) p. 29]{clarke1990} with nonempty compact values \\cite[2.1.2 Proposition (a) p. 27]{clarke1990}, hence proper upper hemicontinuous \\cite[Proposition 1 p. 60]{aubin1984differential}. In addition, $x^{\\tau_k}\\to x^*$ uniformly on $[0,T]$ and $(\\tilde{x}^{\\tau_k})'\\to (x^*)'$ weakly in $L^1([0,T],\\mathbb{R}^n)$. Therefore,  $(x^*)'(t)\\in-\\partial f(x^*(t))$ for almost all $t\\in[0,T]$ by \\cite[Theorem 1 p. 60]{aubin1984differential}\\footnote{In the theorem we take $F:=-\\partial f$, $X=Y:=\\mathbb{R}^n$, and $I:=[0,T]$.}. The initial condition also holds since $\\tilde{x}^{\\tau}(0)=x_0$ for all $\\tau>0$. \n\\vskip 1ex\nWe have proved that for any initial point $x_0$, there exists $x^*:[0,T]\\rightarrow\\mathbb{R}^n$ such that $(x^*)'(t)=-\\partial f(x^*(t))$ holds for almost every $t\\in [0,T]$ with any $T>0$. Since $T$ is independent of $x_0$, by setting $T=1$, there exists a sequence of absolutely continuous functions $(x_k)_{k\\in\\mathbb{N}}$ such that \n\\begin{equation*}\n    x_k'(t)\\in -\\partial f(x_k(t)), \\quad \\mathrm{for~a.e.}~t \\in [0,1], \\quad x_k(0) = x_{k-1}(1), \n\\end{equation*}\nfor all $k\\in\\mathbb{N}$ where $x_{-1}(0)=x_0$. Therefore, the desired function $x:[0,\\infty)\\rightarrow \\mathbb{R}^n$ can be defined in a piecewise fashion by \n\\begin{equation*}\n    x(t) := x_k(t-k),\\quad t\\in[k,k+1),\\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\nBy construction, $x$ is absolutely continuous on any compact interval $[a,b]\\subset[0,\\infty)$. \n\n","start":76318,"end":83047}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"欧氏空间与范数","source_units":[{"raw_tex":"This section contains prerequisites for the proof of \\Cref{thm:main-intro} in the next section. Throughout this paper, unless otherwise specified, we always equip the Euclidean space $\\mathbb{R}^n$ with an inner product $\\langle \\cdot,\\cdot\\rangle$ and its induced norm $\\|\\cdot\\|:=\\sqrt{\\langle \\cdot,\\cdot\\rangle}$. \n","start":12310,"end":12629}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"开球记号","source_units":[{"raw_tex":"In this subsection, we present the formal definitions and some useful properties of setwise local minimum and local minimum at infinity mentioned in \\Cref{sec:introduction}. We first review the classical definition of local and global minima. Throughout this paper, $B(x,\\epsilon)$ denotes the open ball centered at $x\\in\\mathbb{R}^n$ with radius $\\epsilon>0$. \n","start":12688,"end":13050}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":["def:global-local-min"],"declared_external":false,"source_only":false,"title":"点态极小与伪极小","source_units":[{"raw_tex":"\n\\begin{definition}\n\\label{def:global-local-min}\nA point $x\\in\\mathbb{R}^n$ is a \\textit{local minimum} (respectively, \\textit{global minimum}) of a function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if $f(x)\\leqslant f(y)$ for all $y\\in B(x,\\epsilon)$ for some $\\epsilon>0$ (respectively, $y\\in\\mathbb{R}^n$). A local minimum is \\textit{spurious} if it is not a global minimum.\n\\end{definition}\n\nFrom \\Cref{def:global-local-min}, one can see the definition of a local minimum only considers the landscape of a function at any finite point. To discuss the function landscape at infinity, we generalize the notion of setwise local minimum first proposed in \\cite[Definition 2.5]{joszneurips2018}. \n","start":13050,"end":13743}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["def:setwise-local-min"],"declared_external":false,"source_only":false,"title":"集合局部极小","source_units":[{"raw_tex":"\n\\begin{definition}[Setwise local minimum]\\label{def:setwise-local-min}\nA nonempty closed subset $S\\subset\\mathbb{R}^n$ is a \\textit{setwise local minimum} of a continuous function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if there exists an open set $U\\subset\\mathbb{R}^n$ such that $S\\subset U$ and $f(x)\\leqslant f(y)$ for all $x\\in S$, $y\\in U\\setminus S$. \\end{definition}\n","start":13743,"end":14117}]},{"id":"S005","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"点极小的单点嵌入","source_units":[{"raw_tex":"\nIt is easy to see that a local minimum is a setwise local minimum by taking $S$ to be a singleton. We also define a \\textit{strict setwise local minimum} by replacing $f(x)\\leqslant f(y)$ with $f(x)<f(y)$ in \\Cref{def:setwise-local-min}. \n","start":14117,"end":14357}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"严格集合局部极小","source_units":[{"raw_tex":"It is easy to see that a local minimum is a setwise local minimum by taking $S$ to be a singleton. We also define a \\textit{strict setwise local minimum} by replacing $f(x)\\leqslant f(y)$ with $f(x)<f(y)$ in \\Cref{def:setwise-local-min}. \n","start":14118,"end":14357}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":["def:setwise-global-min"],"declared_external":false,"source_only":false,"title":"集合全局极小","source_units":[{"raw_tex":"\n\\begin{definition}[Setwise global minimum]\\label{def:setwise-global-min}\nA subset $S$ of $\\mathbb{R}^n$ is a \\textit{setwise global minimum} of a function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if $S$ is a setwise local minimum of $f$ and $\\inf_{S} f = \\inf_{\\mathbb{R}^n} f$. \n\\end{definition}\n","start":14357,"end":14652}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"集合伪极小与无穷远极小","source_units":[{"raw_tex":"\nNote that $\\inf_S f$ is a shorthand for $\\inf_{x\\in S} f(x)$, and similar for $\\sup_S f$ and $\\max_S f$. Setwise local minima include setwise global minima as a special case, and we say a setwise local minimum is \\textit{spurious} if it is not a setwise global minimum. Note that \\Cref{def:setwise-local-min} is not exactly the same as \\cite[Definition 2.5]{joszneurips2018} because we do not require a setwise local minimum to be a compact set. In other words, a setwise local minimum can be either bounded or unbounded, and we say a (spurious) setwise local minimum is a (spurious) \\textit{local minimum at infinity} if it is unbounded. For example, consider the loss function of a one-hidden layer neural network with sigmoid activation $\\sigma$ and two data points $(1,1)$ and $(-1,-3)$ in \\Cref{fig:setwise}. One can see that $S$  is a setwise local minimum (in particular, a local minimum at infinity) and $U$ is the corresponding open set in \\Cref{def:setwise-local-min}. Finally, observe that $\\mathbb{R}^n$ is a strict setwise local minimum at infinity of any function.\n","start":14652,"end":15732}]},{"id":"S009","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"闭包内部与边界","source_units":[{"raw_tex":"Now we introduce one of the most useful properties of setwise local minima in \\Cref{lem:bdd-const}. This property is intuitive and will be used in different scenarios throughout this paper. Let $\\bar{S}$, $S^\\circ$, and $\\partial S := \\bar{S} \\setminus S^\\circ$ respectively denote the closure, interior, and boundary of a subset $S$ of $\\mathbb{R}^n$.\n","start":15963,"end":16316}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"强制性","source_units":[{"raw_tex":"Finally, we discuss the case of coercive functions. Recall that a function $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ is coercive if $f(x)\\to\\infty$ as $\\|x\\|\\to\\infty$. \n","start":19545,"end":19712}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":["def:locally-lipschitz"],"declared_external":false,"source_only":false,"title":"局部 Lipschitz 性","source_units":[{"raw_tex":"\\begin{definition}\\label{def:locally-lipschitz}\nA function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}^m$ is \\textit{locally Lipschitz} if for all $a \\in \\mathbb{R}^n$, there exist positive constants $r$ and $L$ such that \n\t\\begin{equation*}\n\t\t\\forall\\, x,y \\in B(a,r), ~~~ \\|f(x)-f(y)\\| \\leqslant L \\|x-y\\|.\n\t\\end{equation*}\n\\end{definition}\n\n","start":20562,"end":20899}]},{"id":"S012","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件：Rademacher differentiability theorem；evans2018measure, Theorem 3.2。"],"source_only":false,"title":"几乎处处可微背景","source_units":[{"raw_tex":"Notice that for a locally Lipschitz function, by \\cite[Theorem 3.2]{evans2018measure}, the derivative exists almost everywhere. Without any assumption on convexity, in order to ensure the existence of a subdifferential, we adopt the notion of Clarke subdifferential. \n\n","start":20899,"end":21168}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["def:Clarke"],"declared_external":false,"source_only":false,"title":"Clarke 次微分","source_units":[{"raw_tex":"\\begin{definition}\n\\label{def:Clarke}\n\\cite[p. 27]{clarke1990} Let $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz function. The \\textit{Clarke subdifferential} is the set-valued mapping $\\partial f$ from $\\mathbb{R}^n$ to the subsets of $\\mathbb{R}^n$ defined for all $x \\in \\mathbb{R}^n$ by\n    \\begin{equation*}\n    \\partial f(x) := \\{ s \\in \\mathbb{R}^n ~\\vert~ f^\\circ(x,d) \\geqslant \\langle s , d \\rangle, ~ \\forall\\, d\\in \\mathbb{R}^n \\} \n\\end{equation*}\nwhere\n\\begin{equation*}\n    f^\\circ(x,d) := \\limsup_{\\tiny\\begin{array}{c} y\\rightarrow x \\\\\n    t \\searrow 0\n    \\end{array}\n    } \\frac{f(y+td)-f(y)}{t}.\n\\end{equation*}\n\\end{definition}\n\n","start":21168,"end":21835}]},{"id":"S014","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件：clarke1990, Proposition 2.1.2(a)。"],"source_only":false,"title":"次微分的非空凸紧性","source_units":[{"raw_tex":"It is well known that for any locally Lipschitz function $f$ and any $x\\in\\mathbb{R}^n$, the Clarke subdifferential $\\partial f(x)$ is a nonempty, convex, and compact set \\cite[Proposition 2.1.2(a)]{clarke1990}. Similar to differentiable functions, a point $x\\in\\mathbb{R}^n$ is called (Clarke) \\textit{critical point} if $0\\in\\partial f(x)$. A real number $y$ is called a (Clarke) \\textit{critical value} of $f$ if there exists a (Clarke) critical point $x\\in\\mathbb{R}^n$ of $f$ such that $f(x)=y$. \n","start":21835,"end":22337}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"Clarke 临界点与临界值","source_units":[{"raw_tex":"It is well known that for any locally Lipschitz function $f$ and any $x\\in\\mathbb{R}^n$, the Clarke subdifferential $\\partial f(x)$ is a nonempty, convex, and compact set \\cite[Proposition 2.1.2(a)]{clarke1990}. Similar to differentiable functions, a point $x\\in\\mathbb{R}^n$ is called (Clarke) \\textit{critical point} if $0\\in\\partial f(x)$. A real number $y$ is called a (Clarke) \\textit{critical value} of $f$ if there exists a (Clarke) critical point $x\\in\\mathbb{R}^n$ of $f$ such that $f(x)=y$. \n","start":21835,"end":22337}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":["def:absolute_continuity"],"declared_external":false,"source_only":false,"title":"绝对连续曲线","source_units":[{"raw_tex":"\\begin{definition}\n    \\label{def:absolute_continuity} \n    \\cite[Definition 1 p. 12]{aubin1984differential}\n    Given two real numbers $a<b$, a function $x:[a,b]\\rightarrow\\mathbb{R}^n$ is \\textit{absolutely continuous} if for all $\\epsilon>0$, there exists $\\delta>0$ such that, for any finite collection of disjoint subintervals $[a_1,b_1],\\hdots,[a_m,b_m]$ of $[a,b]$ such that $\\sum_{i=1}^m b_i-a_i \\leqslant \\delta$, we have $\\sum_{i=1}^m \\|x(b_i) - x(a_i)\\| \\leqslant \\epsilon$.\n\\end{definition}\n\n","start":22499,"end":23003}]},{"id":"S017","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件：标准绝对连续演算及 nielsen1997introduction, Theorem 20.8。"],"source_only":false,"title":"绝对连续的积分表征","source_units":[{"raw_tex":"By virtue of \\cite[Theorem 20.8]{nielsen1997introduction}, $x:[a,b]\\rightarrow\\mathbb{R}^n$ is absolutely continuous if and only if it is differentiable almost everywhere on $(a,b)$, its derivative $x'$ is Lebesgue integrable, and $x(t) - x(a) = \\int_a^t x'(\\tau)d\\tau$ for all $t\\in [a,b]$. Given a non-compact interval $I$ of $\\mathbb{R}$, $x:I\\rightarrow \\mathbb{R}^n$ is absolutely continuous if it is absolutely continuous on all compact subintervals of $I$.\n\n","start":23003,"end":23468}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":["eq:ivp-subgrad"],"declared_external":false,"source_only":false,"title":"次梯度轨迹","source_units":[{"raw_tex":"An absolutely continuous function $x:[0,\\infty)\\rightarrow\\mathbb{R}^n$ is called a \\textit{subgradient trajectory} of $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ starting at $x_0\\in\\mathbb{R}^n$ if it satisfies the following differential inclusion with initial condition: \n\\begin{equation}\\label{eq:ivp-subgrad}\n    x'(t) \\in -\\partial f(x(t)),\\quad \\text{for almost every $t\\geqslant 0$},\\quad x(0)=x_0,\n\\end{equation}\nwhere ``almost every'' means all elements except for those in a set of zero measure. \n","start":23468,"end":23969}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":["def:bdd-subgrad"],"declared_external":false,"source_only":false,"title":"有界轨迹的存在量词","source_units":[{"raw_tex":"\\begin{definition}\\label{def:bdd-subgrad}\nA locally Lipschitz lower bounded function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ has \\textit{bounded subgradient trajectories} if for any $x_0\\in\\mathbb{R}^n$, there exists a subgradient trajectory $x$ of $f$ starting at $x_0$ and a constant $r>0$, such that $\\|x(t)\\|\\leqslant r$ for all $t\\geqslant 0$. \n\\end{definition}\n","start":26391,"end":26756}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":["def:chain-rule"],"declared_external":false,"source_only":false,"title":"链式法则","source_units":[{"raw_tex":"\\begin{definition}\\label{def:chain-rule}\nLet $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ be locally Lipschitz. We say $f$ \\textit{admits a chain rule} if for any absolutely continuous function $x:[0,\\infty)\\rightarrow \\mathbb{R}^n$, we have\n\\begin{equation*}\n\t(f\\circ x)'(t) = \\langle v, x'(t)\\rangle, \\quad \\forall\\, v\\in\\partial f(x(t)),\n\\end{equation*}\nfor almost every $t\\in [0,\\infty)$. \n\\end{definition}\n\n","start":27670,"end":28075}]},{"id":"S021","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件：davis2020stochastic, Lemma 5.2；其余本行文献仅为背景讨论。"],"source_only":false,"title":"轨迹上的下降性质","source_units":[{"raw_tex":"Thus, for any locally Lipschitz function that admits a chain rule, by \\cite[Lemma 5.2]{davis2020stochastic}, the function value is always decreasing in time along the subgradient trajectory. A detailed discussion on what class of functions admits a chain rule can be found in \\cite{bolte2020conservative}. Note that general Lipschitz functions are far from admitting a chain rule since they generically have a maximal Clarke subdifferential \\cite{daniilidis2020pathological,borwein2000lipschitz,daniilidis2019linear}.\n","start":28075,"end":28593}]},{"id":"S022","semantic_type":"problem_setting","kind":"problem_setting","labels":["assm:funcClass"],"declared_external":false,"source_only":false,"title":"主定理的函数设定","source_units":[{"raw_tex":"\\begin{assumption}\\label{assm:funcClass}\nLet $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ be a function such that it  \n\\begin{enumerate}[label={(\\alph*)}]\n","start":28958,"end":29106},{"raw_tex":"\\end{enumerate}\n\\end{assumption}\n","start":29480,"end":29513}]},{"id":"S023","semantic_type":"assumption","kind":"assumption","labels":["assm:funcClass"],"declared_external":false,"source_only":false,"title":"主函数下有界","source_units":[{"raw_tex":"    \\item is bounded below, namely, $\\inf_{\\mathbb{R}^n}f > -\\infty$;\n","start":29106,"end":29176}]},{"id":"S024","semantic_type":"assumption","kind":"assumption","labels":["assm:funcClass"],"declared_external":false,"source_only":false,"title":"主函数局部 Lipschitz","source_units":[{"raw_tex":"    \\item is locally Lipschitz continuous on $\\mathbb{R}^n$; see \\Cref{def:locally-lipschitz}; \n","start":29176,"end":29272}]},{"id":"S025","semantic_type":"assumption","kind":"assumption","labels":["assm:funcClass"],"declared_external":false,"source_only":false,"title":"主函数满足链式法则","source_units":[{"raw_tex":"    \\item admits a chain rule; see \\Cref{def:chain-rule}; \n","start":29272,"end":29331}]},{"id":"S026","semantic_type":"assumption","kind":"assumption","labels":["assm:funcClass"],"declared_external":false,"source_only":false,"title":"临界值有限","source_units":[{"raw_tex":"    \\item has finitely many critical values; see \\Cref{subsec:clarke}; \n","start":29331,"end":29403}]},{"id":"S027","semantic_type":"assumption","kind":"assumption","labels":["assm:funcClass"],"declared_external":false,"source_only":false,"title":"主函数具有有界轨迹","source_units":[{"raw_tex":"    \\item has bounded subgradient trajectories; see \\Cref{def:bdd-subgrad}. \n","start":29403,"end":29480}]},{"id":"S028","semantic_type":"theorem","kind":"theorem","labels":["thm:main-intro"],"declared_external":false,"source_only":false,"title":"点态与集合无伪极小等价","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm:main-intro}\nSuppose a locally Lipschitz function is bounded below, admits a chain rule, has finitely many critical values, and has bounded subgradient trajectories. Then it has no spurious local minima if and only if it has no spurious setwise local minima. \n\\end{theorem}","start":9110,"end":9408}]},{"id":"S029","semantic_type":"lemma","kind":"lemma","labels":["lem:bdd-const","lemma:boundary"],"declared_external":false,"source_only":false,"title":"集合极小的边界恒值","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:bdd-const}\n\\label{lemma:boundary}\nIf $S$ is a setwise local minimum of a continuous function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$, then $f(x)=\\sup_S f$ for all $x\\in\\partial S$.\n\\end{lemma}","start":16317,"end":16531}]},{"id":"S030","semantic_type":"proof","kind":"proof","labels":["app:lem:bdd-const","inf_sup"],"declared_external":false,"source_only":false,"title":"边界引理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nSee \\Cref{app:lem:bdd-const}. \n\\end{proof}","start":16532,"end":16588},{"raw_tex":"\\section{Proof of \\texorpdfstring{\\Cref{lem:bdd-const}}{}}\\label{app:lem:bdd-const}\nLet $S \\subset \\mathbb{R}^n$ be a setwise local minimum of a continuous function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. Let $U\\supset S$ be an open set such that $f(x)\\leqslant f(y)$ for all $x \\in S$ and $y\\in U\\setminus S$. Note that $S$ is closed, so its boundary is defined by $\\partial S := S\\setminus S^\\circ$. Let $z\\in\\partial S$ and consider any real number $\\epsilon>0$. Since $f(z)+(-\\epsilon,\\epsilon)$ is a neighborhood of $f(z)$, by continuity of $f$, there exists a neighborhood $N(z)$ of $z$ such that $f(N(z))\\subset f(z)+(-\\epsilon,\\epsilon)$. Since $U$ is a neighborhood of $z$, $N'(z):=U \\cap N(z)$ is also a neighborhood of $z$ with $f(N'(z))\\subset f(z)+(-\\epsilon,\\epsilon)$. The set $N'(z)\\cap S$ is nonempty because $z \\in S$ and the set $N'(z)\\setminus S$ is nonempty because $z\\in\\partial S$. For any $x \\in N'(z) \\cap S$ and $y \\in N'(z)\\setminus S$, it follows that\n\\begin{equation*}\n     \\inf_{U \\setminus S} f -\\epsilon \\leqslant f(y)-\\epsilon < f(z) < f(x)+\\epsilon \\leqslant \\sup_S f+\\epsilon \\leqslant \\inf_{U\\setminus S} f +\\epsilon. \n\\end{equation*}\nThe last inequality follows from the definition of setwise local minima. As $\\epsilon>0$ was arbitrary, we deduce that \n\\begin{equation*}\n\\label{inf_sup}\n    \\inf_{U \\setminus S} f = f(z) = \\sup_S f.\n\\end{equation*}\nThus, $f$ is a constant on the boundary of $S$ and $f$ attains its maximum over $S$ on the boundary of $S$. \n\n\n","start":71033,"end":72530}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":["app:lem:bdd-const"],"declared_external":false,"source_only":false,"title":"边界点的内外邻域","source_units":[{"raw_tex":"Let $S \\subset \\mathbb{R}^n$ be a setwise local minimum of a continuous function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. Let $U\\supset S$ be an open set such that $f(x)\\leqslant f(y)$ for all $x \\in S$ and $y\\in U\\setminus S$. Note that $S$ is closed, so its boundary is defined by $\\partial S := S\\setminus S^\\circ$. Let $z\\in\\partial S$ and consider any real number $\\epsilon>0$. Since $f(z)+(-\\epsilon,\\epsilon)$ is a neighborhood of $f(z)$, by continuity of $f$, there exists a neighborhood $N(z)$ of $z$ such that $f(N(z))\\subset f(z)+(-\\epsilon,\\epsilon)$. Since $U$ is a neighborhood of $z$, $N'(z):=U \\cap N(z)$ is also a neighborhood of $z$ with $f(N'(z))\\subset f(z)+(-\\epsilon,\\epsilon)$. The set $N'(z)\\cap S$ is nonempty because $z \\in S$ and the set $N'(z)\\setminus S$ is nonempty because $z\\in\\partial S$. For any $x \\in N'(z) \\cap S$ and $y \\in N'(z)\\setminus S$, it follows that\n","start":71117,"end":72012}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":["app:lem:bdd-const","inf_sup"],"declared_external":false,"source_only":false,"title":"内外函数值夹逼","source_units":[{"raw_tex":"\\begin{equation*}\n     \\inf_{U \\setminus S} f -\\epsilon \\leqslant f(y)-\\epsilon < f(z) < f(x)+\\epsilon \\leqslant \\sup_S f+\\epsilon \\leqslant \\inf_{U\\setminus S} f +\\epsilon. \n\\end{equation*}\nThe last inequality follows from the definition of setwise local minima. As $\\epsilon>0$ was arbitrary, we deduce that \n\\begin{equation*}\n\\label{inf_sup}\n    \\inf_{U \\setminus S} f = f(z) = \\sup_S f.\n\\end{equation*}\n","start":72012,"end":72419}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":["app:lem:bdd-const"],"declared_external":false,"source_only":false,"title":"边界结论及其作用域","source_units":[{"raw_tex":"Thus, $f$ is a constant on the boundary of $S$ and $f$ attains its maximum over $S$ on the boundary of $S$. \n\n\n","start":72419,"end":72530}]},{"id":"S034","semantic_type":"proposition","kind":"proposition","labels":["prop:exist"],"declared_external":false,"source_only":false,"title":"全时间次梯度轨迹存在","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:exist}\nIf $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ is locally Lipschitz and bounded below, then there exists a subgradient trajectory of $f$ starting at arbitrary $x_0\\in\\mathbb{R}^n$. \n\\end{proposition}","start":25092,"end":25323}]},{"id":"S035","semantic_type":"proof","kind":"proof","labels":["app:prop:exist","eq:1/2-holder","eq:1/2-holder_cs","eq:L2-bounded"],"declared_external":false,"source_only":false,"title":"轨迹存在性的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nSee \\Cref{app:prop:exist}. \n\\end{proof}","start":25324,"end":25377},{"raw_tex":"\\section{Proof of \\texorpdfstring{\\Cref{prop:exist}}{}}\\label{app:prop:exist}\nFor a fixed real number $\\tau>0$, define a sequence $x_k^\\tau$ recurrently by letting $x_0^\\tau:=x_0$ and \n\\begin{equation*}\n    x_{k+1}^\\tau \\in \\arg\\min_{x\\in\\mathbb{R}^n} \\left\\{f(x)+\\frac{\\lVert x-x_k^\\tau\\rVert^2}{2\\tau}\\right\\}, \\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\nA solution exists because $f$ is bounded below and the objective function is coercive. Any solution satisfies \n\\begin{equation*}\n    v_{k+1}^\\tau := \\frac{x_{k+1}^\\tau-x_k^\\tau}{\\tau} \\in -\\partial f(x_{k+1}^\\tau), \\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\nDefine two functions $x^\\tau,\\tilde{x}^\\tau:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ where $\\mathbb{R}_+:=[0,\\infty)$ by\n\\begin{equation*}\n    x^\\tau(t) := x_{k+1}^\\tau,\\quad \\tilde{x}^\\tau(t) := x_k^\\tau + (t-k\\tau)v_{k+1}^\\tau,\\quad \\forall t\\in(k\\tau,(k+1)\\tau]\n\\end{equation*}\nfor all $k\\in\\mathbb{N}$, with initial condition $x^\\tau(0)=\\tilde{x}^\\tau(0)=x_0$. Note that $\\tilde{x}^\\tau$ is absolutely continuous because it is piecewise affine. On the contrary, $x^\\tau$ is not continuous. Also, define $v^{\\tau}:\\mathbb{R}_+\\rightarrow\\mathbb{R}^n$ by \n\\begin{equation*}\n    v^\\tau(t) := v_{k+1}^\\tau, \\quad \\forall t\\in(k\\tau,(k+1)\\tau],\\quad \\forall k\\in\\mathbb{N}, \n\\end{equation*}\nand choose $v^\\tau(0)\\in -\\partial f(x_0)$. Since  $(\\tilde{x}^\\tau)'=v^\\tau$ on $(k\\tau,(k+1)\\tau)$ for all $k \\in \\mathbb{N}$, and $v^\\tau(t)\\in -\\partial f(x^\\tau(t))$ for all $t\\geqslant 0$, we conclude that $(\\tilde{x}^\\tau)'(t)\\in -\\partial f(x^\\tau(t))$ for almost every $t\\in\\mathbb{R}_+$. By optimality of $x_{k+1}^\\tau$, we have \n\\begin{equation*}\n    f(x_{k+1}^\\tau) + \\frac{\\lVert x_{k+1}^\\tau-x_k^\\tau\\rVert^2}{2\\tau} \\leqslant f(x_k^\\tau), \\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\nFor any $l\\in\\mathbb{N}$, we have \n\\begin{equation*}\n    \\sum_{k=0}^l\\frac{\\lVert x_{k+1}^\\tau-x_k^\\tau\\rVert^2}{2\\tau} \\leqslant f(x_0^\\tau) - f(x_{l+1}^\\tau) \\leqslant f(x_0) - \\inf_{\\mathbb{R}^n} f =: C < \\infty\n\\end{equation*}\nsince $f$ is bounded below. Observe that \n\\begin{equation*}\n    \\sum_{k=0}^l\\frac{\\lVert x_{k+1}^\\tau-x_k^\\tau\\rVert^2}{2\\tau} = \\sum_{k=0}^l\\frac{\\tau}{2}\\lVert v_{k+1}^\\tau\\rVert^2 = \\frac{1}{2}\\sum_{k=0}^l\\int_{k\\tau}^{(k+1)\\tau}\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt. \n\\end{equation*}\nFix $T \\geqslant 0$ from now on. From the above, we have\n\\begin{equation}\\label{eq:L2-bounded}\n    \\int_0^T\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt \\leqslant \\sum_{k=0}^{\\lfloor T/\\tau\\rfloor}\\int_{k\\tau}^{(k+1)\\tau}\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt \\leqslant 2C. \n\\end{equation}\nSince $\\tilde{x}^\\tau$ is absolutely continuous, for any $s,t\\in [0,T]$ we have\n\\begin{subequations}\\label{eq:1/2-holder}\n    \\begin{align}\n    \\|\\tilde{x}^\\tau(t)-\\tilde{x}^\\tau(s)\\| &= \\left\\|\\int_s^t(\\tilde{x}^\\tau)'(u)\\;du\\right\\| \\\\\n    &\\leqslant \\left(\\int_0^T\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt\\right)^{1/2}\\vert t-s\\vert^{1/2} \\leqslant \\sqrt{2C}\\vert t-s\\vert^{1/2} \\label{eq:1/2-holder_cs} \n\\end{align}\n\\end{subequations}\nwhere we use the Cauchy-Schwarz inequality. Now one can see $(\\tilde{x}^\\tau)_{\\tau>0}$ is a family of uniformly bounded and equicontinuous functions on the compact interval $[0,T]$. Therefore, by Arzel\\`a-Ascoli theorem \\cite[Theorem 7.25]{rudin1964principles}, there exists a sequence of positive reals $(\\tau_k)_{k\\in\\mathbb{N}}$ such that $\\tau_k\\to 0$ and $\\tilde{x}^{\\tau_k}\\to x^*$ uniformly on $[0,T]$ as $k\\to\\infty$. For all $k\\in \\mathbb{N}$ and $t\\in (k\\tau,(k+1)\\tau]$, we have $\\tilde{x}^\\tau((k+1)\\tau)=x_k^\\tau + \\tau v^\\tau_{k+1} = x^\\tau_{k+1} = x^\\tau(t)$. Thus $\\|\\tilde{x}^\\tau(t)-x^\\tau(t)\\| = \\|\\tilde{x}^\\tau(t)-\\tilde{x}^\\tau((k+1)\\tau)\\| \\leqslant \\sqrt{2C}\\tau^{1/2}$ for all $t\\in [0,T]$ where the inequality is due to \\Cref{eq:1/2-holder} (take $s:=(k+1)\\tau$).\nCombined with the fact that $\\tilde{x}^{\\tau_k}\\to x^*$ uniformly on $[0,T]$, one can see that $x^{\\tau_k}\\to x^*$ uniformly on $[0,T]$. Since \\cref{eq:L2-bounded} implies that $((\\tilde{x}^{\\tau_k})')_{k\\in\\mathbb{N}}$ is a bounded sequence in $L^2([0,T],\\mathbb{R}^n)$, there exists a subsequence $(\\tau_{k_j})_{j\\in\\mathbb{N}}$ such that $(\\tilde{x}^{\\tau_{k_j}})'\\to v^*$ weakly in $L^1([0,T],\\mathbb{R}^n)$ as $j\\to\\infty$ by \\cite[Corollary 14 p. 413]{Fitzpatrick2010}. Since $\\tilde{x}^{\\tau_{k_j}}$ is absolutely continuous, for all $t\\in[0,T]$, we have \n\\begin{equation*}\n    \\tilde{x}^{\\tau_{k_j}}(t) - \\tilde{x}^{\\tau_{k_j}}(0) = \\int_0^t(\\tilde{x}^{\\tau_{k_j}})'(u)\\;du. \n\\end{equation*}\nTake $j\\to\\infty$ on both sides, we have \n\\begin{equation*}\n    x^*(t) - x^*(0) = \\int_0^tv^*(u)\\;du, \n\\end{equation*}\nwhere the convergence of the integral relies on the fact that the constant functions equal to the canonical basis of $\\mathbb{R}^n$ lie in $L^\\infty([0,T],\\mathbb{R}^n)$. Thus, $x^*$ is absolutely continuous and $(x^*)'(t)=v^*(t)$ for almost every $t\\in[0,T]$. Recall that for all $k\\in \\mathbb{N}$, it holds for almost every $t\\in[0,T]$ that\n\\begin{equation*}\n    (\\tilde{x}^{\\tau_k})'(t) = v^{\\tau_k}(t) \\in -\\partial f(x^{\\tau_k}(t)). \n\\end{equation*}\nSince $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d) p. 29]{clarke1990} with nonempty compact values \\cite[2.1.2 Proposition (a) p. 27]{clarke1990}, hence proper upper hemicontinuous \\cite[Proposition 1 p. 60]{aubin1984differential}. In addition, $x^{\\tau_k}\\to x^*$ uniformly on $[0,T]$ and $(\\tilde{x}^{\\tau_k})'\\to (x^*)'$ weakly in $L^1([0,T],\\mathbb{R}^n)$. Therefore,  $(x^*)'(t)\\in-\\partial f(x^*(t))$ for almost all $t\\in[0,T]$ by \\cite[Theorem 1 p. 60]{aubin1984differential}\\footnote{In the theorem we take $F:=-\\partial f$, $X=Y:=\\mathbb{R}^n$, and $I:=[0,T]$.}. The initial condition also holds since $\\tilde{x}^{\\tau}(0)=x_0$ for all $\\tau>0$. \n\\vskip 1ex\nWe have proved that for any initial point $x_0$, there exists $x^*:[0,T]\\rightarrow\\mathbb{R}^n$ such that $(x^*)'(t)=-\\partial f(x^*(t))$ holds for almost every $t\\in [0,T]$ with any $T>0$. Since $T$ is independent of $x_0$, by setting $T=1$, there exists a sequence of absolutely continuous functions $(x_k)_{k\\in\\mathbb{N}}$ such that \n\\begin{equation*}\n    x_k'(t)\\in -\\partial f(x_k(t)), \\quad \\mathrm{for~a.e.}~t \\in [0,1], \\quad x_k(0) = x_{k-1}(1), \n\\end{equation*}\nfor all $k\\in\\mathbb{N}$ where $x_{-1}(0)=x_0$. Therefore, the desired function $x:[0,\\infty)\\rightarrow \\mathbb{R}^n$ can be defined in a piecewise fashion by \n\\begin{equation*}\n    x(t) := x_k(t-k),\\quad t\\in[k,k+1),\\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\nBy construction, $x$ is absolutely continuous on any compact interval $[a,b]\\subset[0,\\infty)$. \n\n","start":76318,"end":83047}]},{"id":"S036","semantic_type":"algorithm","kind":"algorithm","labels":["app:prop:exist","eq:1/2-holder"],"declared_external":false,"source_only":false,"title":"近端离散迭代","source_units":[{"raw_tex":"\\section{Proof of \\texorpdfstring{\\Cref{prop:exist}}{}}\\label{app:prop:exist}\nFor a fixed real number $\\tau>0$, define a sequence $x_k^\\tau$ recurrently by letting $x_0^\\tau:=x_0$ and \n\\begin{equation*}\n    x_{k+1}^\\tau \\in \\arg\\min_{x\\in\\mathbb{R}^n} \\left\\{f(x)+\\frac{\\lVert x-x_k^\\tau\\rVert^2}{2\\tau}\\right\\}, \\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\n","start":76318,"end":76678}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"declared_external":false,"source_only":false,"title":"近端子问题可解","source_units":[{"raw_tex":"A solution exists because $f$ is bounded below and the objective function is coercive. Any solution satisfies \n","start":76678,"end":76789}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"declared_external":false,"source_only":false,"title":"离散速度的次微分包含","source_units":[{"raw_tex":"\\begin{equation*}\n    v_{k+1}^\\tau := \\frac{x_{k+1}^\\tau-x_k^\\tau}{\\tau} \\in -\\partial f(x_{k+1}^\\tau), \\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\n","start":76789,"end":76940}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"declared_external":false,"source_only":false,"title":"阶梯与仿射插值","source_units":[{"raw_tex":"Define two functions $x^\\tau,\\tilde{x}^\\tau:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ where $\\mathbb{R}_+:=[0,\\infty)$ by\n\\begin{equation*}\n    x^\\tau(t) := x_{k+1}^\\tau,\\quad \\tilde{x}^\\tau(t) := x_k^\\tau + (t-k\\tau)v_{k+1}^\\tau,\\quad \\forall t\\in(k\\tau,(k+1)\\tau]\n\\end{equation*}\nfor all $k\\in\\mathbb{N}$, with initial condition $x^\\tau(0)=\\tilde{x}^\\tau(0)=x_0$. Note that $\\tilde{x}^\\tau$ is absolutely continuous because it is piecewise affine. On the contrary, $x^\\tau$ is not continuous. Also, define $v^{\\tau}:\\mathbb{R}_+\\rightarrow\\mathbb{R}^n$ by \n\\begin{equation*}\n    v^\\tau(t) := v_{k+1}^\\tau, \\quad \\forall t\\in(k\\tau,(k+1)\\tau],\\quad \\forall k\\in\\mathbb{N}, \n\\end{equation*}\nand choose $v^\\tau(0)\\in -\\partial f(x_0)$. Since  $(\\tilde{x}^\\tau)'=v^\\tau$ on $(k\\tau,(k+1)\\tau)$ for all $k \\in \\mathbb{N}$, and $v^\\tau(t)\\in -\\partial f(x^\\tau(t))$ for all $t\\geqslant 0$, we conclude that $(\\tilde{x}^\\tau)'(t)\\in -\\partial f(x^\\tau(t))$ for almost every $t\\in\\mathbb{R}_+$. By optimality of $x_{k+1}^\\tau$, we have \n","start":76940,"end":77967}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"declared_external":false,"source_only":false,"title":"单步最优性不等式","source_units":[{"raw_tex":"\\begin{equation*}\n    f(x_{k+1}^\\tau) + \\frac{\\lVert x_{k+1}^\\tau-x_k^\\tau\\rVert^2}{2\\tau} \\leqslant f(x_k^\\tau), \\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\n","start":77967,"end":78128}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"declared_external":false,"source_only":false,"title":"离散能量的总界","source_units":[{"raw_tex":"For any $l\\in\\mathbb{N}$, we have \n\\begin{equation*}\n    \\sum_{k=0}^l\\frac{\\lVert x_{k+1}^\\tau-x_k^\\tau\\rVert^2}{2\\tau} \\leqslant f(x_0^\\tau) - f(x_{l+1}^\\tau) \\leqslant f(x_0) - \\inf_{\\mathbb{R}^n} f =: C < \\infty\n\\end{equation*}\nsince $f$ is bounded below. Observe that \n\\begin{equation*}\n    \\sum_{k=0}^l\\frac{\\lVert x_{k+1}^\\tau-x_k^\\tau\\rVert^2}{2\\tau} = \\sum_{k=0}^l\\frac{\\tau}{2}\\lVert v_{k+1}^\\tau\\rVert^2 = \\frac{1}{2}\\sum_{k=0}^l\\int_{k\\tau}^{(k+1)\\tau}\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt. \n\\end{equation*}\n","start":78128,"end":78658}]},{"id":"S042","semantic_type":"derived_fact","kind":"derived_fact","labels":["app:prop:exist","eq:1/2-holder","eq:L2-bounded"],"declared_external":false,"source_only":false,"title":"统一 L² 导数界","source_units":[{"raw_tex":"Fix $T \\geqslant 0$ from now on. From the above, we have\n\\begin{equation}\\label{eq:L2-bounded}\n    \\int_0^T\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt \\leqslant \\sum_{k=0}^{\\lfloor T/\\tau\\rfloor}\\int_{k\\tau}^{(k+1)\\tau}\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt \\leqslant 2C. \n\\end{equation}\n","start":78658,"end":78961}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder","eq:1/2-holder_cs"],"declared_external":false,"source_only":false,"title":"统一半阶 Hölder 估计","source_units":[{"raw_tex":"Since $\\tilde{x}^\\tau$ is absolutely continuous, for any $s,t\\in [0,T]$ we have\n\\begin{subequations}\\label{eq:1/2-holder}\n    \\begin{align}\n    \\|\\tilde{x}^\\tau(t)-\\tilde{x}^\\tau(s)\\| &= \\left\\|\\int_s^t(\\tilde{x}^\\tau)'(u)\\;du\\right\\| \\\\\n    &\\leqslant \\left(\\int_0^T\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt\\right)^{1/2}\\vert t-s\\vert^{1/2} \\leqslant \\sqrt{2C}\\vert t-s\\vert^{1/2} \\label{eq:1/2-holder_cs} \n\\end{align}\n\\end{subequations}\nwhere we use the Cauchy-Schwarz inequality. Now one can see $(\\tilde{x}^\\tau)_{\\tau>0}$ is a family of uniformly bounded and equicontinuous functions on the compact interval $[0,T]$. Therefore, by Arzel\\`a-Ascoli theorem \\cite[Theorem 7.25]{rudin1964principles}, there exists a sequence of positive reals $(\\tau_k)_{k\\in\\mathbb{N}}$ such that $\\tau_k\\to 0$ and $\\tilde{x}^{\\tau_k}\\to x^*$ uniformly on $[0,T]$ as $k\\to\\infty$. For all $k\\in \\mathbb{N}$ and $t\\in (k\\tau,(k+1)\\tau]$, we have $\\tilde{x}^\\tau((k+1)\\tau)=x_k^\\tau + \\tau v^\\tau_{k+1} = x^\\tau_{k+1} = x^\\tau(t)$. Thus $\\|\\tilde{x}^\\tau(t)-x^\\tau(t)\\| = \\|\\tilde{x}^\\tau(t)-\\tilde{x}^\\tau((k+1)\\tau)\\| \\leqslant \\sqrt{2C}\\tau^{1/2}$ for all $t\\in [0,T]$ where the inequality is due to \\Cref{eq:1/2-holder} (take $s:=(k+1)\\tau$).\n","start":78961,"end":80198}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"declared_external":false,"source_only":false,"title":"仿射插值的一致收敛子列","source_units":[{"raw_tex":"where we use the Cauchy-Schwarz inequality. Now one can see $(\\tilde{x}^\\tau)_{\\tau>0}$ is a family of uniformly bounded and equicontinuous functions on the compact interval $[0,T]$. Therefore, by Arzel\\`a-Ascoli theorem \\cite[Theorem 7.25]{rudin1964principles}, there exists a sequence of positive reals $(\\tau_k)_{k\\in\\mathbb{N}}$ such that $\\tau_k\\to 0$ and $\\tilde{x}^{\\tau_k}\\to x^*$ uniformly on $[0,T]$ as $k\\to\\infty$. For all $k\\in \\mathbb{N}$ and $t\\in (k\\tau,(k+1)\\tau]$, we have $\\tilde{x}^\\tau((k+1)\\tau)=x_k^\\tau + \\tau v^\\tau_{k+1} = x^\\tau_{k+1} = x^\\tau(t)$. Thus $\\|\\tilde{x}^\\tau(t)-x^\\tau(t)\\| = \\|\\tilde{x}^\\tau(t)-\\tilde{x}^\\tau((k+1)\\tau)\\| \\leqslant \\sqrt{2C}\\tau^{1/2}$ for all $t\\in [0,T]$ where the inequality is due to \\Cref{eq:1/2-holder} (take $s:=(k+1)\\tau$).\n","start":79407,"end":80198}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"declared_external":false,"source_only":false,"title":"两种插值趋于同一极限","source_units":[{"raw_tex":"where we use the Cauchy-Schwarz inequality. Now one can see $(\\tilde{x}^\\tau)_{\\tau>0}$ is a family of uniformly bounded and equicontinuous functions on the compact interval $[0,T]$. Therefore, by Arzel\\`a-Ascoli theorem \\cite[Theorem 7.25]{rudin1964principles}, there exists a sequence of positive reals $(\\tau_k)_{k\\in\\mathbb{N}}$ such that $\\tau_k\\to 0$ and $\\tilde{x}^{\\tau_k}\\to x^*$ uniformly on $[0,T]$ as $k\\to\\infty$. For all $k\\in \\mathbb{N}$ and $t\\in (k\\tau,(k+1)\\tau]$, we have $\\tilde{x}^\\tau((k+1)\\tau)=x_k^\\tau + \\tau v^\\tau_{k+1} = x^\\tau_{k+1} = x^\\tau(t)$. Thus $\\|\\tilde{x}^\\tau(t)-x^\\tau(t)\\| = \\|\\tilde{x}^\\tau(t)-\\tilde{x}^\\tau((k+1)\\tau)\\| \\leqslant \\sqrt{2C}\\tau^{1/2}$ for all $t\\in [0,T]$ where the inequality is due to \\Cref{eq:1/2-holder} (take $s:=(k+1)\\tau$).\nCombined with the fact that $\\tilde{x}^{\\tau_k}\\to x^*$ uniformly on $[0,T]$, one can see that $x^{\\tau_k}\\to x^*$ uniformly on $[0,T]$. Since \\cref{eq:L2-bounded} implies that $((\\tilde{x}^{\\tau_k})')_{k\\in\\mathbb{N}}$ is a bounded sequence in $L^2([0,T],\\mathbb{R}^n)$, there exists a subsequence $(\\tau_{k_j})_{j\\in\\mathbb{N}}$ such that $(\\tilde{x}^{\\tau_{k_j}})'\\to v^*$ weakly in $L^1([0,T],\\mathbb{R}^n)$ as $j\\to\\infty$ by \\cite[Corollary 14 p. 413]{Fitzpatrick2010}. Since $\\tilde{x}^{\\tau_{k_j}}$ is absolutely continuous, for all $t\\in[0,T]$, we have \n","start":79407,"end":80761}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"declared_external":false,"source_only":false,"title":"导数的弱收敛子列","source_units":[{"raw_tex":"Combined with the fact that $\\tilde{x}^{\\tau_k}\\to x^*$ uniformly on $[0,T]$, one can see that $x^{\\tau_k}\\to x^*$ uniformly on $[0,T]$. Since \\cref{eq:L2-bounded} implies that $((\\tilde{x}^{\\tau_k})')_{k\\in\\mathbb{N}}$ is a bounded sequence in $L^2([0,T],\\mathbb{R}^n)$, there exists a subsequence $(\\tau_{k_j})_{j\\in\\mathbb{N}}$ such that $(\\tilde{x}^{\\tau_{k_j}})'\\to v^*$ weakly in $L^1([0,T],\\mathbb{R}^n)$ as $j\\to\\infty$ by \\cite[Corollary 14 p. 413]{Fitzpatrick2010}. Since $\\tilde{x}^{\\tau_{k_j}}$ is absolutely continuous, for all $t\\in[0,T]$, we have \n","start":80198,"end":80761}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"declared_external":false,"source_only":false,"title":"极限曲线绝对连续","source_units":[{"raw_tex":"Combined with the fact that $\\tilde{x}^{\\tau_k}\\to x^*$ uniformly on $[0,T]$, one can see that $x^{\\tau_k}\\to x^*$ uniformly on $[0,T]$. Since \\cref{eq:L2-bounded} implies that $((\\tilde{x}^{\\tau_k})')_{k\\in\\mathbb{N}}$ is a bounded sequence in $L^2([0,T],\\mathbb{R}^n)$, there exists a subsequence $(\\tau_{k_j})_{j\\in\\mathbb{N}}$ such that $(\\tilde{x}^{\\tau_{k_j}})'\\to v^*$ weakly in $L^1([0,T],\\mathbb{R}^n)$ as $j\\to\\infty$ by \\cite[Corollary 14 p. 413]{Fitzpatrick2010}. Since $\\tilde{x}^{\\tau_{k_j}}$ is absolutely continuous, for all $t\\in[0,T]$, we have \n\\begin{equation*}\n    \\tilde{x}^{\\tau_{k_j}}(t) - \\tilde{x}^{\\tau_{k_j}}(0) = \\int_0^t(\\tilde{x}^{\\tau_{k_j}})'(u)\\;du. \n\\end{equation*}\nTake $j\\to\\infty$ on both sides, we have \n\\begin{equation*}\n    x^*(t) - x^*(0) = \\int_0^tv^*(u)\\;du, \n\\end{equation*}\nwhere the convergence of the integral relies on the fact that the constant functions equal to the canonical basis of $\\mathbb{R}^n$ lie in $L^\\infty([0,T],\\mathbb{R}^n)$. Thus, $x^*$ is absolutely continuous and $(x^*)'(t)=v^*(t)$ for almost every $t\\in[0,T]$. Recall that for all $k\\in \\mathbb{N}$, it holds for almost every $t\\in[0,T]$ that\n","start":80198,"end":81360}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"declared_external":false,"source_only":false,"title":"极限保持微分包含","source_units":[{"raw_tex":"where the convergence of the integral relies on the fact that the constant functions equal to the canonical basis of $\\mathbb{R}^n$ lie in $L^\\infty([0,T],\\mathbb{R}^n)$. Thus, $x^*$ is absolutely continuous and $(x^*)'(t)=v^*(t)$ for almost every $t\\in[0,T]$. Recall that for all $k\\in \\mathbb{N}$, it holds for almost every $t\\in[0,T]$ that\n\\begin{equation*}\n    (\\tilde{x}^{\\tau_k})'(t) = v^{\\tau_k}(t) \\in -\\partial f(x^{\\tau_k}(t)). \n\\end{equation*}\nSince $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d) p. 29]{clarke1990} with nonempty compact values \\cite[2.1.2 Proposition (a) p. 27]{clarke1990}, hence proper upper hemicontinuous \\cite[Proposition 1 p. 60]{aubin1984differential}. In addition, $x^{\\tau_k}\\to x^*$ uniformly on $[0,T]$ and $(\\tilde{x}^{\\tau_k})'\\to (x^*)'$ weakly in $L^1([0,T],\\mathbb{R}^n)$. Therefore,  $(x^*)'(t)\\in-\\partial f(x^*(t))$ for almost all $t\\in[0,T]$ by \\cite[Theorem 1 p. 60]{aubin1984differential}\\footnote{In the theorem we take $F:=-\\partial f$, $X=Y:=\\mathbb{R}^n$, and $I:=[0,T]$.}. The initial condition also holds since $\\tilde{x}^{\\tau}(0)=x_0$ for all $\\tau>0$. \n","start":81017,"end":82199}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"declared_external":false,"source_only":false,"title":"单位时间解的递推衔接","source_units":[{"raw_tex":"\\vskip 1ex\nWe have proved that for any initial point $x_0$, there exists $x^*:[0,T]\\rightarrow\\mathbb{R}^n$ such that $(x^*)'(t)=-\\partial f(x^*(t))$ holds for almost every $t\\in [0,T]$ with any $T>0$. Since $T$ is independent of $x_0$, by setting $T=1$, there exists a sequence of absolutely continuous functions $(x_k)_{k\\in\\mathbb{N}}$ such that \n\\begin{equation*}\n    x_k'(t)\\in -\\partial f(x_k(t)), \\quad \\mathrm{for~a.e.}~t \\in [0,1], \\quad x_k(0) = x_{k-1}(1), \n\\end{equation*}\nfor all $k\\in\\mathbb{N}$ where $x_{-1}(0)=x_0$. Therefore, the desired function $x:[0,\\infty)\\rightarrow \\mathbb{R}^n$ can be defined in a piecewise fashion by \n","start":82199,"end":82845}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"declared_external":false,"source_only":false,"title":"分段定义全时间轨迹","source_units":[{"raw_tex":"\\begin{equation*}\n    x(t) := x_k(t-k),\\quad t\\in[k,k+1),\\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\nBy construction, $x$ is absolutely continuous on any compact interval $[a,b]\\subset[0,\\infty)$. \n\n","start":82845,"end":83047}]},{"id":"S051","semantic_type":"proof","kind":"proof","labels":["eq:optimization"],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\Cref{thm:main-intro}]\nLet $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ be a function satisfying \\Cref{assm:funcClass} If $f$ has no spurious setwise local minima, then $f$ has no spurious local minima. We next prove the converse. Let $S\\subset\\mathbb{R}^n$ be a setwise local minimum of $f$. We seek to show that $S$ is a setwise global minimum of $f$. If $S^\\circ = \\emptyset$, then by \\Cref{lem:bdd-const} $f(x) = \\sup_S f$ for all $x \\in \\partial S = \\bar{S} \\setminus S^\\circ = S$ since $S$ is closed by \\Cref{def:setwise-local-min}. Thus $f$ is constant on $S$. By definition of setwise local minima (\\Cref{def:setwise-local-min}), there exists an open subset $U$ of $\\mathbb{R}^n$ containing $S$ such that the constant value of $f$ on $S$ is less than or equal to $f(y)$ for all $y \\in U \\setminus S$. Thus every point in $S$ is a local minimum of $f$. Since every local minimum of $f$ is a global minimum, $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum according to \\Cref{def:setwise-global-min}. The rest of the proof deals with the case when $S^\\circ \\neq \\emptyset$. Let $C$ be the set of all critical points of $f$ in $S$ and consider the following optimization problem:\n\\begin{equation}\\label{eq:optimization}\n    \\inf_{x \\in C} f(x).\n\\end{equation}\nWe claim that the set of (global) solutions to \\Cref{eq:optimization} is nonempty, and that any solution is a local minimum of $f$ belonging to the setwise local minimum $S$. We first show that the feasible set of \\Cref{eq:optimization} is nonempty.\n\nSince $S^\\circ \\neq \\emptyset$, let $x_0 \\in S^\\circ$. If $x_0 \\in C$, then the feasible set $C$ is nonempty. We thus assume that $x_0 \\notin C$. Since $f$ is locally Lipschitz and bounded below, by \\Cref{prop:exist} there exists a subgradient trajectory $x:[0,\\infty)\\rightarrow\\mathbb{R}^n$ starting at $x_0$. We next show that $x([0,\\infty)) \\subset S$. We reason by contradiction and assume that $S^c \\cap x([0,\\infty)) \\neq \\emptyset$, where $S^c$ is the complement of $S$ in $\\mathbb{R}^n$. Then $S^\\circ$ and $S^c$ are disjoint open subsets of $\\mathbb{R}^n$ such that $S^\\circ \\cap x([0,\\infty)) \\neq \\emptyset$ (the intersection contains $x_0$), $S^c \\cap x([0,\\infty)) \\neq \\emptyset$, and $x([0,\\infty)) = (x([0,\\infty)) \\cap S^\\circ) \\cup (x([0,\\infty)) \\cap S^c) \\subset \\mathbb{R}^n \\setminus \\partial S$ (since\\footnote{If there exists $t>0$ such that $f(x(t)) = f(x(0))$, then $f(x(t)) - f(x(0)) = \\int_0^{t} \\|x'(s)\\|^2ds = 0$ and $x'(s) = 0$ for almost every $s\\in (0,t)$. Since $x'(s) \\in -\\partial f(x(s))$ for almost every $s>0$, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990} we have $0 \\in \\partial f(x(0))$.} $f(x(t)) < f(x(0)) = f(x_0) \\leqslant f(x)$ for all $t > 0$ and $x \\in \\partial S$, where the last inequality follows from \\Cref{lemma:boundary}). Thus the connected set $x([0,\\infty))$ is the union of two relatively open disjoint nonempty sets, which is a contradiction. \n\nSince $f$ has bounded subgradient trajectories and $x(\\cdot)$ is an arbitrary subgradient trajectory starting at $x_0$, by \\Cref{def:bdd-subgrad} and without loss of generality there exists $r>0$ such that $\\|x(t)\\| \\leqslant r$ for all $t\\geqslant 0$. We next show that there exists a critical point of $f$ in $B(0,r) \\cap S$. Suppose that there exist two constants $T,\\epsilon>0$ for which $\\lVert x'(t)\\rVert\\geqslant\\epsilon$ for all $t\\geqslant T$ such that $x'(t)\\in-\\partial f(x(t))$. By \\cite[Lemma 5.2]{davis2020stochastic}, we have $(f \\circ x)'(t) = -\\lVert x'(t)\\rVert^2 \\leqslant -\\epsilon^2$ for almost every $t\\geqslant T$. By integrating, we get $f(x(t)) - f(x(T)) \\leqslant - \\epsilon^2t$ and thus $f(x(t))$ converges to $-\\infty$ as $t\\to\\infty$. This is impossible since $x(t) \\in B(0,r)$ and $f$ is continuous. Hence there exists a time sequence $t_k\\to\\infty$ such that $\\lVert x'(t_k)\\rVert\\to 0$ as $k\\to\\infty$ and $x'(t_k)\\in-\\partial f(x(t_k))$ for all $k\\in\\mathbb{N} := \\{0,1,2,\\hdots\\}$. By the Bolzano–Weierstrass theorem, there exists a subsequence $x(t_{k_j})$ of $x(t_k)$ such that $x(t_{k_j})\\to\\tilde{x}\\in\\mathbb{R}^n$ as $j\\to\\infty$. Since $x'(t_{k_j})\\in -\\partial f(x(t_{k_j}))$, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990} we have $0\\in -\\partial f(\\tilde{x})$. Finally, since $x([0,\\infty)) \\subset S$ and $S$ is closed, we have $\\tilde{x}\\in C$. We obtain that $C\\neq \\emptyset$ as desired.\n\nSince $f$ has finitely many critical values and $C \\neq \\emptyset$, the set of solutions to \\Cref{eq:optimization} is nonempty. Let $x^* \\in C$ be a solution, that is to say $f(x^*) = \\min_C f$. Recall that $C$ is a subset of the setwise local minimum $S$. If $x^*$ is a local minimum of $f$, then it is a global minimum of $f$ in $S$ since every local minimum of $f$ is a global minimum. Thus $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum. For the remainder of the proof, we consider the case where $x^*$ is not a local minimum and show that this leads to a contradiction. We first show that there exists $s_0\\in S^\\circ$ such that $f(s_0)<f(x^*)$. This is clearly true if $x^*\\in S^\\circ$ since one can then find a ball centered at $x^*$ inside $S^\\circ$. If $x^*\\in S \\setminus S^\\circ = \\partial S$, then we reason by contradiction and assume that $f(x) \\geqslant f(x^*)$ for all $x\\in S^\\circ$. By \\Cref{lemma:boundary}, we have $f(x) = f(x^*) = \\sup_S f \\geqslant f(y) \\geqslant f(x^*)$ for all $(x,y)\\in (S \\setminus S^\\circ) \\times S^\\circ$. Hence $f(x^*) = f(x)$ for all $x\\in S$. Since $S$ is a setwise local minimum, there exists an open set $U$ such that $f(x) \\geqslant f(x^*)$ holds for all $x\\in U\\setminus S$. Thus $f(x)\\geqslant f(x^*)$ for all $x\\in U$ and $x^*$ is a local minimum. This yields a contradiction. Hence let $s_0\\in S^\\circ$ be such that $f(s_0)<f(x^*)$. The nonempty closed set $S' := S \\cap [f \\leqslant (f(s_0)+f(x^*))/2]$ is a setwise local minimum of $f$ where $[f\\leqslant \\alpha] := \\{ x \\in \\mathbb{R}^n ~\\vert~ f(x) \\leqslant \\alpha \\}$. Indeed, for all $x \\in S'$ and $y \\in U \\setminus S' = (U \\setminus S) \\cup (U \\setminus [f \\leqslant (f(s_0)+f(x^*))/2])$, we have\\footnote{Indeed, for any sets $A,B$, and $C$ it holds that $A\\setminus (B\\cap C) = (A \\setminus B) \\cup (A\\setminus C)$.} $f(x) \\leqslant (f(s_0)+f(x^*))/2 \\leqslant f(y)$. Since $s_0 \\in S^\\circ$ and $f(s_0)<(f(s_0)+f(x^*))/2$, we have $s_0 \\in S^\\circ \\cap [f\\leqslant (f(s_0)+f(x^*))/2]^\\circ = (S \\cap [f\\leqslant (f(s_0)+f(x^*))/2]))^\\circ = (S')^\\circ$. Hence the setwise local minimum $S'$ has nonempty interior. Also, $S' \\subset S$ and $\\sup_{S'} f < f(x^*) = \\min_C f$ where we remind the reader that $C$ is the set of critical points in $S$. Thus $S'$ is devoid of critical points. However, by the previous\nparagraph, setwise local minima of $f$ with nonempty interior must contain a critical point. This yields a contradiction.\n\\end{proof}","start":29515,"end":36408}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"等价命题的直接方向","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\Cref{thm:main-intro}]\nLet $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ be a function satisfying \\Cref{assm:funcClass} If $f$ has no spurious setwise local minima, then $f$ has no spurious local minima. We next prove the converse. Let $S\\subset\\mathbb{R}^n$ be a setwise local minimum of $f$. We seek to show that $S$ is a setwise global minimum of $f$. If $S^\\circ = \\emptyset$, then by \\Cref{lem:bdd-const} $f(x) = \\sup_S f$ for all $x \\in \\partial S = \\bar{S} \\setminus S^\\circ = S$ since $S$ is closed by \\Cref{def:setwise-local-min}. Thus $f$ is constant on $S$. By definition of setwise local minima (\\Cref{def:setwise-local-min}), there exists an open subset $U$ of $\\mathbb{R}^n$ containing $S$ such that the constant value of $f$ on $S$ is less than or equal to $f(y)$ for all $y \\in U \\setminus S$. Thus every point in $S$ is a local minimum of $f$. Since every local minimum of $f$ is a global minimum, $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum according to \\Cref{def:setwise-global-min}. The rest of the proof deals with the case when $S^\\circ \\neq \\emptyset$. Let $C$ be the set of all critical points of $f$ in $S$ and consider the following optimization problem:\n","start":29515,"end":30739}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"空内部集合的处理","source_units":[{"raw_tex":"Let $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ be a function satisfying \\Cref{assm:funcClass} If $f$ has no spurious setwise local minima, then $f$ has no spurious local minima. We next prove the converse. Let $S\\subset\\mathbb{R}^n$ be a setwise local minimum of $f$. We seek to show that $S$ is a setwise global minimum of $f$. If $S^\\circ = \\emptyset$, then by \\Cref{lem:bdd-const} $f(x) = \\sup_S f$ for all $x \\in \\partial S = \\bar{S} \\setminus S^\\circ = S$ since $S$ is closed by \\Cref{def:setwise-local-min}. Thus $f$ is constant on $S$. By definition of setwise local minima (\\Cref{def:setwise-local-min}), there exists an open subset $U$ of $\\mathbb{R}^n$ containing $S$ such that the constant value of $f$ on $S$ is less than or equal to $f(y)$ for all $y \\in U \\setminus S$. Thus every point in $S$ is a local minimum of $f$. Since every local minimum of $f$ is a global minimum, $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum according to \\Cref{def:setwise-global-min}. The rest of the proof deals with the case when $S^\\circ \\neq \\emptyset$. Let $C$ be the set of all critical points of $f$ in $S$ and consider the following optimization problem:\n","start":29561,"end":30739}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":["eq:optimization"],"declared_external":false,"source_only":false,"title":"临界点上的辅助优化","source_units":[{"raw_tex":"Let $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ be a function satisfying \\Cref{assm:funcClass} If $f$ has no spurious setwise local minima, then $f$ has no spurious local minima. We next prove the converse. Let $S\\subset\\mathbb{R}^n$ be a setwise local minimum of $f$. We seek to show that $S$ is a setwise global minimum of $f$. If $S^\\circ = \\emptyset$, then by \\Cref{lem:bdd-const} $f(x) = \\sup_S f$ for all $x \\in \\partial S = \\bar{S} \\setminus S^\\circ = S$ since $S$ is closed by \\Cref{def:setwise-local-min}. Thus $f$ is constant on $S$. By definition of setwise local minima (\\Cref{def:setwise-local-min}), there exists an open subset $U$ of $\\mathbb{R}^n$ containing $S$ such that the constant value of $f$ on $S$ is less than or equal to $f(y)$ for all $y \\in U \\setminus S$. Thus every point in $S$ is a local minimum of $f$. Since every local minimum of $f$ is a global minimum, $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum according to \\Cref{def:setwise-global-min}. The rest of the proof deals with the case when $S^\\circ \\neq \\emptyset$. Let $C$ be the set of all critical points of $f$ in $S$ and consider the following optimization problem:\n\\begin{equation}\\label{eq:optimization}\n    \\inf_{x \\in C} f(x).\n\\end{equation}\nWe claim that the set of (global) solutions to \\Cref{eq:optimization} is nonempty, and that any solution is a local minimum of $f$ belonging to the setwise local minimum $S$. We first show that the feasible set of \\Cref{eq:optimization} is nonempty.\n","start":29561,"end":31069}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"从内部非临界点启动轨迹","source_units":[{"raw_tex":"Since $S^\\circ \\neq \\emptyset$, let $x_0 \\in S^\\circ$. If $x_0 \\in C$, then the feasible set $C$ is nonempty. We thus assume that $x_0 \\notin C$. Since $f$ is locally Lipschitz and bounded below, by \\Cref{prop:exist} there exists a subgradient trajectory $x:[0,\\infty)\\rightarrow\\mathbb{R}^n$ starting at $x_0$. We next show that $x([0,\\infty)) \\subset S$. We reason by contradiction and assume that $S^c \\cap x([0,\\infty)) \\neq \\emptyset$, where $S^c$ is the complement of $S$ in $\\mathbb{R}^n$. Then $S^\\circ$ and $S^c$ are disjoint open subsets of $\\mathbb{R}^n$ such that $S^\\circ \\cap x([0,\\infty)) \\neq \\emptyset$ (the intersection contains $x_0$), $S^c \\cap x([0,\\infty)) \\neq \\emptyset$, and $x([0,\\infty)) = (x([0,\\infty)) \\cap S^\\circ) \\cup (x([0,\\infty)) \\cap S^c) \\subset \\mathbb{R}^n \\setminus \\partial S$ (since\\footnote{If there exists $t>0$ such that $f(x(t)) = f(x(0))$, then $f(x(t)) - f(x(0)) = \\int_0^{t} \\|x'(s)\\|^2ds = 0$ and $x'(s) = 0$ for almost every $s\\in (0,t)$. Since $x'(s) \\in -\\partial f(x(s))$ for almost every $s>0$, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990} we have $0 \\in \\partial f(x(0))$.} $f(x(t)) < f(x(0)) = f(x_0) \\leqslant f(x)$ for all $t > 0$ and $x \\in \\partial S$, where the last inequality follows from \\Cref{lemma:boundary}). Thus the connected set $x([0,\\infty))$ is the union of two relatively open disjoint nonempty sets, which is a contradiction. \n","start":31070,"end":32479}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"非临界初值的严格下降","source_units":[{"raw_tex":"Since $S^\\circ \\neq \\emptyset$, let $x_0 \\in S^\\circ$. If $x_0 \\in C$, then the feasible set $C$ is nonempty. We thus assume that $x_0 \\notin C$. Since $f$ is locally Lipschitz and bounded below, by \\Cref{prop:exist} there exists a subgradient trajectory $x:[0,\\infty)\\rightarrow\\mathbb{R}^n$ starting at $x_0$. We next show that $x([0,\\infty)) \\subset S$. We reason by contradiction and assume that $S^c \\cap x([0,\\infty)) \\neq \\emptyset$, where $S^c$ is the complement of $S$ in $\\mathbb{R}^n$. Then $S^\\circ$ and $S^c$ are disjoint open subsets of $\\mathbb{R}^n$ such that $S^\\circ \\cap x([0,\\infty)) \\neq \\emptyset$ (the intersection contains $x_0$), $S^c \\cap x([0,\\infty)) \\neq \\emptyset$, and $x([0,\\infty)) = (x([0,\\infty)) \\cap S^\\circ) \\cup (x([0,\\infty)) \\cap S^c) \\subset \\mathbb{R}^n \\setminus \\partial S$ (since\\footnote{If there exists $t>0$ such that $f(x(t)) = f(x(0))$, then $f(x(t)) - f(x(0)) = \\int_0^{t} \\|x'(s)\\|^2ds = 0$ and $x'(s) = 0$ for almost every $s\\in (0,t)$. Since $x'(s) \\in -\\partial f(x(s))$ for almost every $s>0$, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990} we have $0 \\in \\partial f(x(0))$.} $f(x(t)) < f(x(0)) = f(x_0) \\leqslant f(x)$ for all $t > 0$ and $x \\in \\partial S$, where the last inequality follows from \\Cref{lemma:boundary}). Thus the connected set $x([0,\\infty))$ is the union of two relatively open disjoint nonempty sets, which is a contradiction. \n","start":31070,"end":32479}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"轨迹留在集合极小内","source_units":[{"raw_tex":"Since $S^\\circ \\neq \\emptyset$, let $x_0 \\in S^\\circ$. If $x_0 \\in C$, then the feasible set $C$ is nonempty. We thus assume that $x_0 \\notin C$. Since $f$ is locally Lipschitz and bounded below, by \\Cref{prop:exist} there exists a subgradient trajectory $x:[0,\\infty)\\rightarrow\\mathbb{R}^n$ starting at $x_0$. We next show that $x([0,\\infty)) \\subset S$. We reason by contradiction and assume that $S^c \\cap x([0,\\infty)) \\neq \\emptyset$, where $S^c$ is the complement of $S$ in $\\mathbb{R}^n$. Then $S^\\circ$ and $S^c$ are disjoint open subsets of $\\mathbb{R}^n$ such that $S^\\circ \\cap x([0,\\infty)) \\neq \\emptyset$ (the intersection contains $x_0$), $S^c \\cap x([0,\\infty)) \\neq \\emptyset$, and $x([0,\\infty)) = (x([0,\\infty)) \\cap S^\\circ) \\cup (x([0,\\infty)) \\cap S^c) \\subset \\mathbb{R}^n \\setminus \\partial S$ (since\\footnote{If there exists $t>0$ such that $f(x(t)) = f(x(0))$, then $f(x(t)) - f(x(0)) = \\int_0^{t} \\|x'(s)\\|^2ds = 0$ and $x'(s) = 0$ for almost every $s\\in (0,t)$. Since $x'(s) \\in -\\partial f(x(s))$ for almost every $s>0$, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990} we have $0 \\in \\partial f(x(0))$.} $f(x(t)) < f(x(0)) = f(x_0) \\leqslant f(x)$ for all $t > 0$ and $x \\in \\partial S$, where the last inequality follows from \\Cref{lemma:boundary}). Thus the connected set $x([0,\\infty))$ is the union of two relatively open disjoint nonempty sets, which is a contradiction. \n","start":31070,"end":32479}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"有界轨迹的选择说明","source_units":[{"raw_tex":"Since $f$ has bounded subgradient trajectories and $x(\\cdot)$ is an arbitrary subgradient trajectory starting at $x_0$, by \\Cref{def:bdd-subgrad} and without loss of generality there exists $r>0$ such that $\\|x(t)\\| \\leqslant r$ for all $t\\geqslant 0$. We next show that there exists a critical point of $f$ in $B(0,r) \\cap S$. Suppose that there exist two constants $T,\\epsilon>0$ for which $\\lVert x'(t)\\rVert\\geqslant\\epsilon$ for all $t\\geqslant T$ such that $x'(t)\\in-\\partial f(x(t))$. By \\cite[Lemma 5.2]{davis2020stochastic}, we have $(f \\circ x)'(t) = -\\lVert x'(t)\\rVert^2 \\leqslant -\\epsilon^2$ for almost every $t\\geqslant T$. By integrating, we get $f(x(t)) - f(x(T)) \\leqslant - \\epsilon^2t$ and thus $f(x(t))$ converges to $-\\infty$ as $t\\to\\infty$. This is impossible since $x(t) \\in B(0,r)$ and $f$ is continuous. Hence there exists a time sequence $t_k\\to\\infty$ such that $\\lVert x'(t_k)\\rVert\\to 0$ as $k\\to\\infty$ and $x'(t_k)\\in-\\partial f(x(t_k))$ for all $k\\in\\mathbb{N} := \\{0,1,2,\\hdots\\}$. By the Bolzano–Weierstrass theorem, there exists a subsequence $x(t_{k_j})$ of $x(t_k)$ such that $x(t_{k_j})\\to\\tilde{x}\\in\\mathbb{R}^n$ as $j\\to\\infty$. Since $x'(t_{k_j})\\in -\\partial f(x(t_{k_j}))$, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990} we have $0\\in -\\partial f(\\tilde{x})$. Finally, since $x([0,\\infty)) \\subset S$ and $S$ is closed, we have $\\tilde{x}\\in C$. We obtain that $C\\neq \\emptyset$ as desired.\n","start":32480,"end":33920}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"存在速度趋零的时刻","source_units":[{"raw_tex":"Since $f$ has bounded subgradient trajectories and $x(\\cdot)$ is an arbitrary subgradient trajectory starting at $x_0$, by \\Cref{def:bdd-subgrad} and without loss of generality there exists $r>0$ such that $\\|x(t)\\| \\leqslant r$ for all $t\\geqslant 0$. We next show that there exists a critical point of $f$ in $B(0,r) \\cap S$. Suppose that there exist two constants $T,\\epsilon>0$ for which $\\lVert x'(t)\\rVert\\geqslant\\epsilon$ for all $t\\geqslant T$ such that $x'(t)\\in-\\partial f(x(t))$. By \\cite[Lemma 5.2]{davis2020stochastic}, we have $(f \\circ x)'(t) = -\\lVert x'(t)\\rVert^2 \\leqslant -\\epsilon^2$ for almost every $t\\geqslant T$. By integrating, we get $f(x(t)) - f(x(T)) \\leqslant - \\epsilon^2t$ and thus $f(x(t))$ converges to $-\\infty$ as $t\\to\\infty$. This is impossible since $x(t) \\in B(0,r)$ and $f$ is continuous. Hence there exists a time sequence $t_k\\to\\infty$ such that $\\lVert x'(t_k)\\rVert\\to 0$ as $k\\to\\infty$ and $x'(t_k)\\in-\\partial f(x(t_k))$ for all $k\\in\\mathbb{N} := \\{0,1,2,\\hdots\\}$. By the Bolzano–Weierstrass theorem, there exists a subsequence $x(t_{k_j})$ of $x(t_k)$ such that $x(t_{k_j})\\to\\tilde{x}\\in\\mathbb{R}^n$ as $j\\to\\infty$. Since $x'(t_{k_j})\\in -\\partial f(x(t_{k_j}))$, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990} we have $0\\in -\\partial f(\\tilde{x})$. Finally, since $x([0,\\infty)) \\subset S$ and $S$ is closed, we have $\\tilde{x}\\in C$. We obtain that $C\\neq \\emptyset$ as desired.\n","start":32480,"end":33920}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"非空内部必含临界点","source_units":[{"raw_tex":"Since $f$ has bounded subgradient trajectories and $x(\\cdot)$ is an arbitrary subgradient trajectory starting at $x_0$, by \\Cref{def:bdd-subgrad} and without loss of generality there exists $r>0$ such that $\\|x(t)\\| \\leqslant r$ for all $t\\geqslant 0$. We next show that there exists a critical point of $f$ in $B(0,r) \\cap S$. Suppose that there exist two constants $T,\\epsilon>0$ for which $\\lVert x'(t)\\rVert\\geqslant\\epsilon$ for all $t\\geqslant T$ such that $x'(t)\\in-\\partial f(x(t))$. By \\cite[Lemma 5.2]{davis2020stochastic}, we have $(f \\circ x)'(t) = -\\lVert x'(t)\\rVert^2 \\leqslant -\\epsilon^2$ for almost every $t\\geqslant T$. By integrating, we get $f(x(t)) - f(x(T)) \\leqslant - \\epsilon^2t$ and thus $f(x(t))$ converges to $-\\infty$ as $t\\to\\infty$. This is impossible since $x(t) \\in B(0,r)$ and $f$ is continuous. Hence there exists a time sequence $t_k\\to\\infty$ such that $\\lVert x'(t_k)\\rVert\\to 0$ as $k\\to\\infty$ and $x'(t_k)\\in-\\partial f(x(t_k))$ for all $k\\in\\mathbb{N} := \\{0,1,2,\\hdots\\}$. By the Bolzano–Weierstrass theorem, there exists a subsequence $x(t_{k_j})$ of $x(t_k)$ such that $x(t_{k_j})\\to\\tilde{x}\\in\\mathbb{R}^n$ as $j\\to\\infty$. Since $x'(t_{k_j})\\in -\\partial f(x(t_{k_j}))$, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990} we have $0\\in -\\partial f(\\tilde{x})$. Finally, since $x([0,\\infty)) \\subset S$ and $S$ is closed, we have $\\tilde{x}\\in C$. We obtain that $C\\neq \\emptyset$ as desired.\n","start":32480,"end":33920}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"最低临界值取得","source_units":[{"raw_tex":"Since $f$ has finitely many critical values and $C \\neq \\emptyset$, the set of solutions to \\Cref{eq:optimization} is nonempty. Let $x^* \\in C$ be a solution, that is to say $f(x^*) = \\min_C f$. Recall that $C$ is a subset of the setwise local minimum $S$. If $x^*$ is a local minimum of $f$, then it is a global minimum of $f$ in $S$ since every local minimum of $f$ is a global minimum. Thus $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum. For the remainder of the proof, we consider the case where $x^*$ is not a local minimum and show that this leads to a contradiction. We first show that there exists $s_0\\in S^\\circ$ such that $f(s_0)<f(x^*)$. This is clearly true if $x^*\\in S^\\circ$ since one can then find a ball centered at $x^*$ inside $S^\\circ$. If $x^*\\in S \\setminus S^\\circ = \\partial S$, then we reason by contradiction and assume that $f(x) \\geqslant f(x^*)$ for all $x\\in S^\\circ$. By \\Cref{lemma:boundary}, we have $f(x) = f(x^*) = \\sup_S f \\geqslant f(y) \\geqslant f(x^*)$ for all $(x,y)\\in (S \\setminus S^\\circ) \\times S^\\circ$. Hence $f(x^*) = f(x)$ for all $x\\in S$. Since $S$ is a setwise local minimum, there exists an open set $U$ such that $f(x) \\geqslant f(x^*)$ holds for all $x\\in U\\setminus S$. Thus $f(x)\\geqslant f(x^*)$ for all $x\\in U$ and $x^*$ is a local minimum. This yields a contradiction. Hence let $s_0\\in S^\\circ$ be such that $f(s_0)<f(x^*)$. The nonempty closed set $S' := S \\cap [f \\leqslant (f(s_0)+f(x^*))/2]$ is a setwise local minimum of $f$ where $[f\\leqslant \\alpha] := \\{ x \\in \\mathbb{R}^n ~\\vert~ f(x) \\leqslant \\alpha \\}$. Indeed, for all $x \\in S'$ and $y \\in U \\setminus S' = (U \\setminus S) \\cup (U \\setminus [f \\leqslant (f(s_0)+f(x^*))/2])$, we have\\footnote{Indeed, for any sets $A,B$, and $C$ it holds that $A\\setminus (B\\cap C) = (A \\setminus B) \\cup (A\\setminus C)$.} $f(x) \\leqslant (f(s_0)+f(x^*))/2 \\leqslant f(y)$. Since $s_0 \\in S^\\circ$ and $f(s_0)<(f(s_0)+f(x^*))/2$, we have $s_0 \\in S^\\circ \\cap [f\\leqslant (f(s_0)+f(x^*))/2]^\\circ = (S \\cap [f\\leqslant (f(s_0)+f(x^*))/2]))^\\circ = (S')^\\circ$. Hence the setwise local minimum $S'$ has nonempty interior. Also, $S' \\subset S$ and $\\sup_{S'} f < f(x^*) = \\min_C f$ where we remind the reader that $C$ is the set of critical points in $S$. Thus $S'$ is devoid of critical points. However, by the previous\n","start":33921,"end":36275}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"局部极小解导出全局性","source_units":[{"raw_tex":"Since $f$ has finitely many critical values and $C \\neq \\emptyset$, the set of solutions to \\Cref{eq:optimization} is nonempty. Let $x^* \\in C$ be a solution, that is to say $f(x^*) = \\min_C f$. Recall that $C$ is a subset of the setwise local minimum $S$. If $x^*$ is a local minimum of $f$, then it is a global minimum of $f$ in $S$ since every local minimum of $f$ is a global minimum. Thus $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum. For the remainder of the proof, we consider the case where $x^*$ is not a local minimum and show that this leads to a contradiction. We first show that there exists $s_0\\in S^\\circ$ such that $f(s_0)<f(x^*)$. This is clearly true if $x^*\\in S^\\circ$ since one can then find a ball centered at $x^*$ inside $S^\\circ$. If $x^*\\in S \\setminus S^\\circ = \\partial S$, then we reason by contradiction and assume that $f(x) \\geqslant f(x^*)$ for all $x\\in S^\\circ$. By \\Cref{lemma:boundary}, we have $f(x) = f(x^*) = \\sup_S f \\geqslant f(y) \\geqslant f(x^*)$ for all $(x,y)\\in (S \\setminus S^\\circ) \\times S^\\circ$. Hence $f(x^*) = f(x)$ for all $x\\in S$. Since $S$ is a setwise local minimum, there exists an open set $U$ such that $f(x) \\geqslant f(x^*)$ holds for all $x\\in U\\setminus S$. Thus $f(x)\\geqslant f(x^*)$ for all $x\\in U$ and $x^*$ is a local minimum. This yields a contradiction. Hence let $s_0\\in S^\\circ$ be such that $f(s_0)<f(x^*)$. The nonempty closed set $S' := S \\cap [f \\leqslant (f(s_0)+f(x^*))/2]$ is a setwise local minimum of $f$ where $[f\\leqslant \\alpha] := \\{ x \\in \\mathbb{R}^n ~\\vert~ f(x) \\leqslant \\alpha \\}$. Indeed, for all $x \\in S'$ and $y \\in U \\setminus S' = (U \\setminus S) \\cup (U \\setminus [f \\leqslant (f(s_0)+f(x^*))/2])$, we have\\footnote{Indeed, for any sets $A,B$, and $C$ it holds that $A\\setminus (B\\cap C) = (A \\setminus B) \\cup (A\\setminus C)$.} $f(x) \\leqslant (f(s_0)+f(x^*))/2 \\leqslant f(y)$. Since $s_0 \\in S^\\circ$ and $f(s_0)<(f(s_0)+f(x^*))/2$, we have $s_0 \\in S^\\circ \\cap [f\\leqslant (f(s_0)+f(x^*))/2]^\\circ = (S \\cap [f\\leqslant (f(s_0)+f(x^*))/2]))^\\circ = (S')^\\circ$. Hence the setwise local minimum $S'$ has nonempty interior. Also, $S' \\subset S$ and $\\sup_{S'} f < f(x^*) = \\min_C f$ where we remind the reader that $C$ is the set of critical points in $S$. Thus $S'$ is devoid of critical points. However, by the previous\n","start":33921,"end":36275}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"非局部极小导出更低内部点","source_units":[{"raw_tex":"Since $f$ has finitely many critical values and $C \\neq \\emptyset$, the set of solutions to \\Cref{eq:optimization} is nonempty. Let $x^* \\in C$ be a solution, that is to say $f(x^*) = \\min_C f$. Recall that $C$ is a subset of the setwise local minimum $S$. If $x^*$ is a local minimum of $f$, then it is a global minimum of $f$ in $S$ since every local minimum of $f$ is a global minimum. Thus $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum. For the remainder of the proof, we consider the case where $x^*$ is not a local minimum and show that this leads to a contradiction. We first show that there exists $s_0\\in S^\\circ$ such that $f(s_0)<f(x^*)$. This is clearly true if $x^*\\in S^\\circ$ since one can then find a ball centered at $x^*$ inside $S^\\circ$. If $x^*\\in S \\setminus S^\\circ = \\partial S$, then we reason by contradiction and assume that $f(x) \\geqslant f(x^*)$ for all $x\\in S^\\circ$. By \\Cref{lemma:boundary}, we have $f(x) = f(x^*) = \\sup_S f \\geqslant f(y) \\geqslant f(x^*)$ for all $(x,y)\\in (S \\setminus S^\\circ) \\times S^\\circ$. Hence $f(x^*) = f(x)$ for all $x\\in S$. Since $S$ is a setwise local minimum, there exists an open set $U$ such that $f(x) \\geqslant f(x^*)$ holds for all $x\\in U\\setminus S$. Thus $f(x)\\geqslant f(x^*)$ for all $x\\in U$ and $x^*$ is a local minimum. This yields a contradiction. Hence let $s_0\\in S^\\circ$ be such that $f(s_0)<f(x^*)$. The nonempty closed set $S' := S \\cap [f \\leqslant (f(s_0)+f(x^*))/2]$ is a setwise local minimum of $f$ where $[f\\leqslant \\alpha] := \\{ x \\in \\mathbb{R}^n ~\\vert~ f(x) \\leqslant \\alpha \\}$. Indeed, for all $x \\in S'$ and $y \\in U \\setminus S' = (U \\setminus S) \\cup (U \\setminus [f \\leqslant (f(s_0)+f(x^*))/2])$, we have\\footnote{Indeed, for any sets $A,B$, and $C$ it holds that $A\\setminus (B\\cap C) = (A \\setminus B) \\cup (A\\setminus C)$.} $f(x) \\leqslant (f(s_0)+f(x^*))/2 \\leqslant f(y)$. Since $s_0 \\in S^\\circ$ and $f(s_0)<(f(s_0)+f(x^*))/2$, we have $s_0 \\in S^\\circ \\cap [f\\leqslant (f(s_0)+f(x^*))/2]^\\circ = (S \\cap [f\\leqslant (f(s_0)+f(x^*))/2]))^\\circ = (S')^\\circ$. Hence the setwise local minimum $S'$ has nonempty interior. Also, $S' \\subset S$ and $\\sup_{S'} f < f(x^*) = \\min_C f$ where we remind the reader that $C$ is the set of critical points in $S$. Thus $S'$ is devoid of critical points. However, by the previous\n","start":33921,"end":36275}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"构造更低的集合极小","source_units":[{"raw_tex":"Since $f$ has finitely many critical values and $C \\neq \\emptyset$, the set of solutions to \\Cref{eq:optimization} is nonempty. Let $x^* \\in C$ be a solution, that is to say $f(x^*) = \\min_C f$. Recall that $C$ is a subset of the setwise local minimum $S$. If $x^*$ is a local minimum of $f$, then it is a global minimum of $f$ in $S$ since every local minimum of $f$ is a global minimum. Thus $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum. For the remainder of the proof, we consider the case where $x^*$ is not a local minimum and show that this leads to a contradiction. We first show that there exists $s_0\\in S^\\circ$ such that $f(s_0)<f(x^*)$. This is clearly true if $x^*\\in S^\\circ$ since one can then find a ball centered at $x^*$ inside $S^\\circ$. If $x^*\\in S \\setminus S^\\circ = \\partial S$, then we reason by contradiction and assume that $f(x) \\geqslant f(x^*)$ for all $x\\in S^\\circ$. By \\Cref{lemma:boundary}, we have $f(x) = f(x^*) = \\sup_S f \\geqslant f(y) \\geqslant f(x^*)$ for all $(x,y)\\in (S \\setminus S^\\circ) \\times S^\\circ$. Hence $f(x^*) = f(x)$ for all $x\\in S$. Since $S$ is a setwise local minimum, there exists an open set $U$ such that $f(x) \\geqslant f(x^*)$ holds for all $x\\in U\\setminus S$. Thus $f(x)\\geqslant f(x^*)$ for all $x\\in U$ and $x^*$ is a local minimum. This yields a contradiction. Hence let $s_0\\in S^\\circ$ be such that $f(s_0)<f(x^*)$. The nonempty closed set $S' := S \\cap [f \\leqslant (f(s_0)+f(x^*))/2]$ is a setwise local minimum of $f$ where $[f\\leqslant \\alpha] := \\{ x \\in \\mathbb{R}^n ~\\vert~ f(x) \\leqslant \\alpha \\}$. Indeed, for all $x \\in S'$ and $y \\in U \\setminus S' = (U \\setminus S) \\cup (U \\setminus [f \\leqslant (f(s_0)+f(x^*))/2])$, we have\\footnote{Indeed, for any sets $A,B$, and $C$ it holds that $A\\setminus (B\\cap C) = (A \\setminus B) \\cup (A\\setminus C)$.} $f(x) \\leqslant (f(s_0)+f(x^*))/2 \\leqslant f(y)$. Since $s_0 \\in S^\\circ$ and $f(s_0)<(f(s_0)+f(x^*))/2$, we have $s_0 \\in S^\\circ \\cap [f\\leqslant (f(s_0)+f(x^*))/2]^\\circ = (S \\cap [f\\leqslant (f(s_0)+f(x^*))/2]))^\\circ = (S')^\\circ$. Hence the setwise local minimum $S'$ has nonempty interior. Also, $S' \\subset S$ and $\\sup_{S'} f < f(x^*) = \\min_C f$ where we remind the reader that $C$ is the set of critical points in $S$. Thus $S'$ is devoid of critical points. However, by the previous\n","start":33921,"end":36275}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"低于全部临界值的矛盾","source_units":[{"raw_tex":"Since $f$ has finitely many critical values and $C \\neq \\emptyset$, the set of solutions to \\Cref{eq:optimization} is nonempty. Let $x^* \\in C$ be a solution, that is to say $f(x^*) = \\min_C f$. Recall that $C$ is a subset of the setwise local minimum $S$. If $x^*$ is a local minimum of $f$, then it is a global minimum of $f$ in $S$ since every local minimum of $f$ is a global minimum. Thus $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum. For the remainder of the proof, we consider the case where $x^*$ is not a local minimum and show that this leads to a contradiction. We first show that there exists $s_0\\in S^\\circ$ such that $f(s_0)<f(x^*)$. This is clearly true if $x^*\\in S^\\circ$ since one can then find a ball centered at $x^*$ inside $S^\\circ$. If $x^*\\in S \\setminus S^\\circ = \\partial S$, then we reason by contradiction and assume that $f(x) \\geqslant f(x^*)$ for all $x\\in S^\\circ$. By \\Cref{lemma:boundary}, we have $f(x) = f(x^*) = \\sup_S f \\geqslant f(y) \\geqslant f(x^*)$ for all $(x,y)\\in (S \\setminus S^\\circ) \\times S^\\circ$. Hence $f(x^*) = f(x)$ for all $x\\in S$. Since $S$ is a setwise local minimum, there exists an open set $U$ such that $f(x) \\geqslant f(x^*)$ holds for all $x\\in U\\setminus S$. Thus $f(x)\\geqslant f(x^*)$ for all $x\\in U$ and $x^*$ is a local minimum. This yields a contradiction. Hence let $s_0\\in S^\\circ$ be such that $f(s_0)<f(x^*)$. The nonempty closed set $S' := S \\cap [f \\leqslant (f(s_0)+f(x^*))/2]$ is a setwise local minimum of $f$ where $[f\\leqslant \\alpha] := \\{ x \\in \\mathbb{R}^n ~\\vert~ f(x) \\leqslant \\alpha \\}$. Indeed, for all $x \\in S'$ and $y \\in U \\setminus S' = (U \\setminus S) \\cup (U \\setminus [f \\leqslant (f(s_0)+f(x^*))/2])$, we have\\footnote{Indeed, for any sets $A,B$, and $C$ it holds that $A\\setminus (B\\cap C) = (A \\setminus B) \\cup (A\\setminus C)$.} $f(x) \\leqslant (f(s_0)+f(x^*))/2 \\leqslant f(y)$. Since $s_0 \\in S^\\circ$ and $f(s_0)<(f(s_0)+f(x^*))/2$, we have $s_0 \\in S^\\circ \\cap [f\\leqslant (f(s_0)+f(x^*))/2]^\\circ = (S \\cap [f\\leqslant (f(s_0)+f(x^*))/2]))^\\circ = (S')^\\circ$. Hence the setwise local minimum $S'$ has nonempty interior. Also, $S' \\subset S$ and $\\sup_{S'} f < f(x^*) = \\min_C f$ where we remind the reader that $C$ is the set of critical points in $S$. Thus $S'$ is devoid of critical points. However, by the previous\nparagraph, setwise local minima of $f$ with nonempty interior must contain a critical point. This yields a contradiction.\n","start":33921,"end":36397}]},{"id":"S066","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"逆向证明的无伪点极小假设","source_units":[{"raw_tex":"Let $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ be a function satisfying \\Cref{assm:funcClass} If $f$ has no spurious setwise local minima, then $f$ has no spurious local minima. We next prove the converse. Let $S\\subset\\mathbb{R}^n$ be a setwise local minimum of $f$. We seek to show that $S$ is a setwise global minimum of $f$. If $S^\\circ = \\emptyset$, then by \\Cref{lem:bdd-const} $f(x) = \\sup_S f$ for all $x \\in \\partial S = \\bar{S} \\setminus S^\\circ = S$ since $S$ is closed by \\Cref{def:setwise-local-min}. Thus $f$ is constant on $S$. By definition of setwise local minima (\\Cref{def:setwise-local-min}), there exists an open subset $U$ of $\\mathbb{R}^n$ containing $S$ such that the constant value of $f$ on $S$ is less than or equal to $f(y)$ for all $y \\in U \\setminus S$. Thus every point in $S$ is a local minimum of $f$. Since every local minimum of $f$ is a global minimum, $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum according to \\Cref{def:setwise-global-min}. The rest of the proof deals with the case when $S^\\circ \\neq \\emptyset$. Let $C$ be the set of all critical points of $f$ in $S$ and consider the following optimization problem:\n","start":29561,"end":30739}]},{"id":"S067","semantic_type":"assumption","kind":"assumption","labels":["app:prop:exist","eq:1/2-holder"],"declared_external":false,"source_only":false,"title":"存在性构造的下界假设","source_units":[{"raw_tex":"A solution exists because $f$ is bounded below and the objective function is coercive. Any solution satisfies \n","start":76678,"end":76789},{"raw_tex":"For any $l\\in\\mathbb{N}$, we have \n\\begin{equation*}\n    \\sum_{k=0}^l\\frac{\\lVert x_{k+1}^\\tau-x_k^\\tau\\rVert^2}{2\\tau} \\leqslant f(x_0^\\tau) - f(x_{l+1}^\\tau) \\leqslant f(x_0) - \\inf_{\\mathbb{R}^n} f =: C < \\infty\n\\end{equation*}\nsince $f$ is bounded below. Observe that \n","start":78128,"end":78401}]},{"id":"S068","semantic_type":"assumption","kind":"assumption","labels":["app:prop:exist","eq:1/2-holder"],"declared_external":false,"source_only":false,"title":"存在性构造的正则性假设","source_units":[{"raw_tex":"Since $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d) p. 29]{clarke1990} with nonempty compact values \\cite[2.1.2 Proposition (a) p. 27]{clarke1990}, hence proper upper hemicontinuous \\cite[Proposition 1 p. 60]{aubin1984differential}. In addition, $x^{\\tau_k}\\to x^*$ uniformly on $[0,T]$ and $(\\tilde{x}^{\\tau_k})'\\to (x^*)'$ weakly in $L^1([0,T],\\mathbb{R}^n)$. Therefore,  $(x^*)'(t)\\in-\\partial f(x^*(t))$ for almost all $t\\in[0,T]$ by \\cite[Theorem 1 p. 60]{aubin1984differential}\\footnote{In the theorem we take $F:=-\\partial f$, $X=Y:=\\mathbb{R}^n$, and $I:=[0,T]$.}. The initial condition also holds since $\\tilde{x}^{\\tau}(0)=x_0$ for all $\\tau>0$. \n","start":81472,"end":82199}]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件：Standard finite-dimensional topology and compactness，含 connectedness of continuous images of intervals 与 Bolzano–Weierstrass；紧集上连续函数有界为其中的标准背景。"],"source_only":false,"title":"有限维连通性与紧性","source_units":[]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件：attainment of minima for continuous coercive functions。"],"source_only":false,"title":"连续强制目标取得极小","source_units":[]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件：Clarke Fermat necessary condition and addition of a smooth quadratic term, used implicitly for proximal minimizers。"],"source_only":false,"title":"Clarke Fermat 与光滑求和","source_units":[]},{"id":"S072","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件及作者 L326、L328 引用：clarke1990, Proposition 2.1.5(b), p.29。"],"source_only":false,"title":"Clarke 次微分闭图","source_units":[]},{"id":"S073","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件及作者 L752 引用：clarke1990, Proposition 2.1.5(d), p.29。"],"source_only":false,"title":"Clarke 次微分上半连续","source_units":[]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件及作者 L739 引用：rudin1964principles, Theorem 7.25。"],"source_only":false,"title":"Arzelà–Ascoli 紧性","source_units":[]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件及作者 L740 引用：Fitzpatrick2010, Corollary 14, p.413。"],"source_only":false,"title":"导数序列的弱紧性","source_units":[]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件及作者 L752 引用：aubin1984differential, Proposition 1, p.60。"],"source_only":false,"title":"上半连续到上半半连续","source_units":[]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件及作者 L752 引用与脚注：aubin1984differential, Theorem 1, p.60。"],"source_only":false,"title":"微分包含的极限稳定性","source_units":[]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件：Cauchy–Schwarz inequality；作者在 L739 明说使用。"],"source_only":false,"title":"Cauchy–Schwarz 不等式","source_units":[]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件：Subgradient trajectory descent and energy identity；davis2020stochastic, Lemma 5.2，作者在 L301、L328 引用。"],"source_only":false,"title":"次梯度轨迹能量恒等式","source_units":[]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator：弱 L¹ 收敛的 L∞ 对偶测试；用于显式标注 L748 省略的区间指示函数。"],"source_only":false,"title":"弱 L¹ 收敛的积分测试","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S028","edges":[{"source_id":"S011","target_id":"S024","kind":"supports"},{"source_id":"S020","target_id":"S025","kind":"supports"},{"source_id":"S015","target_id":"S026","kind":"supports"},{"source_id":"S019","target_id":"S027","kind":"supports"},{"source_id":"S013","target_id":"S018","kind":"supports"},{"source_id":"S016","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S013","target_id":"S020","kind":"supports"},{"source_id":"S016","target_id":"S020","kind":"supports"},{"source_id":"S003","target_id":"S028","kind":"supports"},{"source_id":"S004","target_id":"S028","kind":"supports"},{"source_id":"S007","target_id":"S028","kind":"supports"},{"source_id":"S008","target_id":"S028","kind":"supports"},{"source_id":"S011","target_id":"S028","kind":"supports"},{"source_id":"S015","target_id":"S028","kind":"supports"},{"source_id":"S019","target_id":"S028","kind":"supports"},{"source_id":"S020","target_id":"S028","kind":"supports"},{"source_id":"S004","target_id":"S029","kind":"supports"},{"source_id":"S009","target_id":"S029","kind":"supports"},{"source_id":"S004","target_id":"S031","kind":"supports"},{"source_id":"S009","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S004","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S030","kind":"supports"},{"source_id":"S033","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S011","target_id":"S034","kind":"supports"},{"source_id":"S018","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S067","target_id":"S037","kind":"supports"},{"source_id":"S068","target_id":"S037","kind":"supports"},{"source_id":"S010","target_id":"S037","kind":"supports"},{"source_id":"S070","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S071","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S014","target_id":"S039","kind":"supports"},{"source_id":"S036","target_id":"S040","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S067","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S039","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S017","target_id":"S043","kind":"supports"},{"source_id":"S078","target_id":"S043","kind":"supports"},{"source_id":"S039","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S074","target_id":"S044","kind":"supports"},{"source_id":"S039","target_id":"S045","kind":"supports"},{"source_id":"S043","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S042","target_id":"S046","kind":"supports"},{"source_id":"S075","target_id":"S046","kind":"supports"},{"source_id":"S044","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S017","target_id":"S047","kind":"supports"},{"source_id":"S080","target_id":"S047","kind":"supports"},{"source_id":"S039","target_id":"S048","kind":"supports"},{"source_id":"S045","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S068","target_id":"S048","kind":"supports"},{"source_id":"S014","target_id":"S048","kind":"supports"},{"source_id":"S073","target_id":"S048","kind":"supports"},{"source_id":"S076","target_id":"S048","kind":"supports"},{"source_id":"S077","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S017","target_id":"S050","kind":"supports"},{"source_id":"S018","target_id":"S050","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S035","kind":"supports"},{"source_id":"S039","target_id":"S035","kind":"supports"},{"source_id":"S040","target_id":"S035","kind":"supports"},{"source_id":"S041","target_id":"S035","kind":"supports"},{"source_id":"S042","target_id":"S035","kind":"supports"},{"source_id":"S043","target_id":"S035","kind":"supports"},{"source_id":"S044","target_id":"S035","kind":"supports"},{"source_id":"S045","target_id":"S035","kind":"supports"},{"source_id":"S046","target_id":"S035","kind":"supports"},{"source_id":"S047","target_id":"S035","kind":"supports"},{"source_id":"S048","target_id":"S035","kind":"supports"},{"source_id":"S049","target_id":"S035","kind":"supports"},{"source_id":"S050","target_id":"S035","kind":"supports"},{"source_id":"S067","target_id":"S035","kind":"supports"},{"source_id":"S068","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S005","target_id":"S052","kind":"supports"},{"source_id":"S003","target_id":"S052","kind":"supports"},{"source_id":"S007","target_id":"S052","kind":"supports"},{"source_id":"S008","target_id":"S052","kind":"supports"},{"source_id":"S004","target_id":"S053","kind":"supports"},{"source_id":"S009","target_id":"S053","kind":"supports"},{"source_id":"S029","target_id":"S053","kind":"supports"},{"source_id":"S066","target_id":"S053","kind":"supports"},{"source_id":"S003","target_id":"S053","kind":"supports"},{"source_id":"S007","target_id":"S053","kind":"supports"},{"source_id":"S015","target_id":"S054","kind":"supports"},{"source_id":"S004","target_id":"S054","kind":"supports"},{"source_id":"S009","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S023","target_id":"S055","kind":"supports"},{"source_id":"S024","target_id":"S055","kind":"supports"},{"source_id":"S034","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S025","target_id":"S056","kind":"supports"},{"source_id":"S018","target_id":"S056","kind":"supports"},{"source_id":"S017","target_id":"S056","kind":"supports"},{"source_id":"S079","target_id":"S056","kind":"supports"},{"source_id":"S072","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S029","target_id":"S057","kind":"supports"},{"source_id":"S004","target_id":"S057","kind":"supports"},{"source_id":"S018","target_id":"S057","kind":"supports"},{"source_id":"S069","target_id":"S057","kind":"supports"},{"source_id":"S027","target_id":"S058","kind":"supports"},{"source_id":"S019","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S024","target_id":"S059","kind":"supports"},{"source_id":"S025","target_id":"S059","kind":"supports"},{"source_id":"S079","target_id":"S059","kind":"supports"},{"source_id":"S069","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S057","target_id":"S060","kind":"supports"},{"source_id":"S058","target_id":"S060","kind":"supports"},{"source_id":"S004","target_id":"S060","kind":"supports"},{"source_id":"S015","target_id":"S060","kind":"supports"},{"source_id":"S054","target_id":"S060","kind":"supports"},{"source_id":"S069","target_id":"S060","kind":"supports"},{"source_id":"S072","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S026","target_id":"S061","kind":"supports"},{"source_id":"S015","target_id":"S061","kind":"supports"},{"source_id":"S054","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S066","target_id":"S062","kind":"supports"},{"source_id":"S007","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S063","kind":"supports"},{"source_id":"S004","target_id":"S063","kind":"supports"},{"source_id":"S009","target_id":"S063","kind":"supports"},{"source_id":"S029","target_id":"S063","kind":"supports"},{"source_id":"S003","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S004","target_id":"S064","kind":"supports"},{"source_id":"S024","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S061","target_id":"S065","kind":"supports"},{"source_id":"S060","target_id":"S065","kind":"supports"},{"source_id":"S022","target_id":"S051","kind":"supports"},{"source_id":"S023","target_id":"S051","kind":"supports"},{"source_id":"S024","target_id":"S051","kind":"supports"},{"source_id":"S025","target_id":"S051","kind":"supports"},{"source_id":"S026","target_id":"S051","kind":"supports"},{"source_id":"S027","target_id":"S051","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S051","kind":"supports"},{"source_id":"S055","target_id":"S051","kind":"supports"},{"source_id":"S056","target_id":"S051","kind":"supports"},{"source_id":"S057","target_id":"S051","kind":"supports"},{"source_id":"S058","target_id":"S051","kind":"supports"},{"source_id":"S059","target_id":"S051","kind":"supports"},{"source_id":"S060","target_id":"S051","kind":"supports"},{"source_id":"S061","target_id":"S051","kind":"supports"},{"source_id":"S062","target_id":"S051","kind":"supports"},{"source_id":"S063","target_id":"S051","kind":"supports"},{"source_id":"S064","target_id":"S051","kind":"supports"},{"source_id":"S065","target_id":"S051","kind":"supports"},{"source_id":"S066","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S028","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S028","edges":[{"source_id":"S011","target_id":"S024","kind":"supports"},{"source_id":"S020","target_id":"S025","kind":"supports"},{"source_id":"S015","target_id":"S026","kind":"supports"},{"source_id":"S019","target_id":"S027","kind":"supports"},{"source_id":"S013","target_id":"S018","kind":"supports"},{"source_id":"S016","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S013","target_id":"S020","kind":"supports"},{"source_id":"S016","target_id":"S020","kind":"supports"},{"source_id":"S003","target_id":"S028","kind":"supports"},{"source_id":"S004","target_id":"S028","kind":"supports"},{"source_id":"S007","target_id":"S028","kind":"supports"},{"source_id":"S008","target_id":"S028","kind":"supports"},{"source_id":"S011","target_id":"S028","kind":"supports"},{"source_id":"S015","target_id":"S028","kind":"supports"},{"source_id":"S019","target_id":"S028","kind":"supports"},{"source_id":"S020","target_id":"S028","kind":"supports"},{"source_id":"S004","target_id":"S029","kind":"supports"},{"source_id":"S009","target_id":"S029","kind":"supports"},{"source_id":"S004","target_id":"S031","kind":"supports"},{"source_id":"S009","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S004","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S030","kind":"supports"},{"source_id":"S033","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S011","target_id":"S034","kind":"supports"},{"source_id":"S018","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S067","target_id":"S037","kind":"supports"},{"source_id":"S068","target_id":"S037","kind":"supports"},{"source_id":"S010","target_id":"S037","kind":"supports"},{"source_id":"S070","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S071","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S014","target_id":"S039","kind":"supports"},{"source_id":"S036","target_id":"S040","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S067","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S039","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S017","target_id":"S043","kind":"supports"},{"source_id":"S078","target_id":"S043","kind":"supports"},{"source_id":"S039","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S074","target_id":"S044","kind":"supports"},{"source_id":"S039","target_id":"S045","kind":"supports"},{"source_id":"S043","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S042","target_id":"S046","kind":"supports"},{"source_id":"S075","target_id":"S046","kind":"supports"},{"source_id":"S044","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S017","target_id":"S047","kind":"supports"},{"source_id":"S080","target_id":"S047","kind":"supports"},{"source_id":"S039","target_id":"S048","kind":"supports"},{"source_id":"S045","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S068","target_id":"S048","kind":"supports"},{"source_id":"S014","target_id":"S048","kind":"supports"},{"source_id":"S073","target_id":"S048","kind":"supports"},{"source_id":"S076","target_id":"S048","kind":"supports"},{"source_id":"S077","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S017","target_id":"S050","kind":"supports"},{"source_id":"S018","target_id":"S050","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S035","kind":"supports"},{"source_id":"S039","target_id":"S035","kind":"supports"},{"source_id":"S040","target_id":"S035","kind":"supports"},{"source_id":"S041","target_id":"S035","kind":"supports"},{"source_id":"S042","target_id":"S035","kind":"supports"},{"source_id":"S043","target_id":"S035","kind":"supports"},{"source_id":"S044","target_id":"S035","kind":"supports"},{"source_id":"S045","target_id":"S035","kind":"supports"},{"source_id":"S046","target_id":"S035","kind":"supports"},{"source_id":"S047","target_id":"S035","kind":"supports"},{"source_id":"S048","target_id":"S035","kind":"supports"},{"source_id":"S049","target_id":"S035","kind":"supports"},{"source_id":"S050","target_id":"S035","kind":"supports"},{"source_id":"S067","target_id":"S035","kind":"supports"},{"source_id":"S068","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S005","target_id":"S052","kind":"supports"},{"source_id":"S003","target_id":"S052","kind":"supports"},{"source_id":"S007","target_id":"S052","kind":"supports"},{"source_id":"S008","target_id":"S052","kind":"supports"},{"source_id":"S004","target_id":"S053","kind":"supports"},{"source_id":"S009","target_id":"S053","kind":"supports"},{"source_id":"S029","target_id":"S053","kind":"supports"},{"source_id":"S066","target_id":"S053","kind":"supports"},{"source_id":"S003","target_id":"S053","kind":"supports"},{"source_id":"S007","target_id":"S053","kind":"supports"},{"source_id":"S015","target_id":"S054","kind":"supports"},{"source_id":"S004","target_id":"S054","kind":"supports"},{"source_id":"S009","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S023","target_id":"S055","kind":"supports"},{"source_id":"S024","target_id":"S055","kind":"supports"},{"source_id":"S034","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S025","target_id":"S056","kind":"supports"},{"source_id":"S018","target_id":"S056","kind":"supports"},{"source_id":"S017","target_id":"S056","kind":"supports"},{"source_id":"S079","target_id":"S056","kind":"supports"},{"source_id":"S072","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S029","target_id":"S057","kind":"supports"},{"source_id":"S004","target_id":"S057","kind":"supports"},{"source_id":"S018","target_id":"S057","kind":"supports"},{"source_id":"S069","target_id":"S057","kind":"supports"},{"source_id":"S027","target_id":"S058","kind":"supports"},{"source_id":"S019","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S024","target_id":"S059","kind":"supports"},{"source_id":"S025","target_id":"S059","kind":"supports"},{"source_id":"S079","target_id":"S059","kind":"supports"},{"source_id":"S069","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S057","target_id":"S060","kind":"supports"},{"source_id":"S058","target_id":"S060","kind":"supports"},{"source_id":"S004","target_id":"S060","kind":"supports"},{"source_id":"S015","target_id":"S060","kind":"supports"},{"source_id":"S054","target_id":"S060","kind":"supports"},{"source_id":"S069","target_id":"S060","kind":"supports"},{"source_id":"S072","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S026","target_id":"S061","kind":"supports"},{"source_id":"S015","target_id":"S061","kind":"supports"},{"source_id":"S054","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S066","target_id":"S062","kind":"supports"},{"source_id":"S007","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S063","kind":"supports"},{"source_id":"S004","target_id":"S063","kind":"supports"},{"source_id":"S009","target_id":"S063","kind":"supports"},{"source_id":"S029","target_id":"S063","kind":"supports"},{"source_id":"S003","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S004","target_id":"S064","kind":"supports"},{"source_id":"S024","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S061","target_id":"S065","kind":"supports"},{"source_id":"S060","target_id":"S065","kind":"supports"},{"source_id":"S022","target_id":"S051","kind":"supports"},{"source_id":"S023","target_id":"S051","kind":"supports"},{"source_id":"S024","target_id":"S051","kind":"supports"},{"source_id":"S025","target_id":"S051","kind":"supports"},{"source_id":"S026","target_id":"S051","kind":"supports"},{"source_id":"S027","target_id":"S051","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S051","kind":"supports"},{"source_id":"S055","target_id":"S051","kind":"supports"},{"source_id":"S056","target_id":"S051","kind":"supports"},{"source_id":"S057","target_id":"S051","kind":"supports"},{"source_id":"S058","target_id":"S051","kind":"supports"},{"source_id":"S059","target_id":"S051","kind":"supports"},{"source_id":"S060","target_id":"S051","kind":"supports"},{"source_id":"S061","target_id":"S051","kind":"supports"},{"source_id":"S062","target_id":"S051","kind":"supports"},{"source_id":"S063","target_id":"S051","kind":"supports"},{"source_id":"S064","target_id":"S051","kind":"supports"},{"source_id":"S065","target_id":"S051","kind":"supports"},{"source_id":"S066","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S028","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"欧氏空间与范数","source_units":[{"raw_tex":"This section contains prerequisites for the proof of \\Cref{thm:main-intro} in the next section. Throughout this paper, unless otherwise specified, we always equip the Euclidean space $\\mathbb{R}^n$ with an inner product $\\langle \\cdot,\\cdot\\rangle$ and its induced norm $\\|\\cdot\\|:=\\sqrt{\\langle \\cdot,\\cdot\\rangle}$. \n","start":12310,"end":12629}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"开球记号","source_units":[{"raw_tex":"In this subsection, we present the formal definitions and some useful properties of setwise local minimum and local minimum at infinity mentioned in \\Cref{sec:introduction}. We first review the classical definition of local and global minima. Throughout this paper, $B(x,\\epsilon)$ denotes the open ball centered at $x\\in\\mathbb{R}^n$ with radius $\\epsilon>0$. \n","start":12688,"end":13050}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":["def:global-local-min"],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"点态极小与伪极小","source_units":[{"raw_tex":"\n\\begin{definition}\n\\label{def:global-local-min}\nA point $x\\in\\mathbb{R}^n$ is a \\textit{local minimum} (respectively, \\textit{global minimum}) of a function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if $f(x)\\leqslant f(y)$ for all $y\\in B(x,\\epsilon)$ for some $\\epsilon>0$ (respectively, $y\\in\\mathbb{R}^n$). A local minimum is \\textit{spurious} if it is not a global minimum.\n\\end{definition}\n\nFrom \\Cref{def:global-local-min}, one can see the definition of a local minimum only considers the landscape of a function at any finite point. To discuss the function landscape at infinity, we generalize the notion of setwise local minimum first proposed in \\cite[Definition 2.5]{joszneurips2018}. \n","start":13050,"end":13743}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["def:setwise-local-min"],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"集合局部极小","source_units":[{"raw_tex":"\n\\begin{definition}[Setwise local minimum]\\label{def:setwise-local-min}\nA nonempty closed subset $S\\subset\\mathbb{R}^n$ is a \\textit{setwise local minimum} of a continuous function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if there exists an open set $U\\subset\\mathbb{R}^n$ such that $S\\subset U$ and $f(x)\\leqslant f(y)$ for all $x\\in S$, $y\\in U\\setminus S$. \\end{definition}\n","start":13743,"end":14117}]},{"id":"S005","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"点极小的单点嵌入","source_units":[{"raw_tex":"\nIt is easy to see that a local minimum is a setwise local minimum by taking $S$ to be a singleton. We also define a \\textit{strict setwise local minimum} by replacing $f(x)\\leqslant f(y)$ with $f(x)<f(y)$ in \\Cref{def:setwise-local-min}. \n","start":14117,"end":14357}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"严格集合局部极小","source_units":[{"raw_tex":"It is easy to see that a local minimum is a setwise local minimum by taking $S$ to be a singleton. We also define a \\textit{strict setwise local minimum} by replacing $f(x)\\leqslant f(y)$ with $f(x)<f(y)$ in \\Cref{def:setwise-local-min}. \n","start":14118,"end":14357}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":["def:setwise-global-min"],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"集合全局极小","source_units":[{"raw_tex":"\n\\begin{definition}[Setwise global minimum]\\label{def:setwise-global-min}\nA subset $S$ of $\\mathbb{R}^n$ is a \\textit{setwise global minimum} of a function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ if $S$ is a setwise local minimum of $f$ and $\\inf_{S} f = \\inf_{\\mathbb{R}^n} f$. \n\\end{definition}\n","start":14357,"end":14652}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"集合伪极小与无穷远极小","source_units":[{"raw_tex":"\nNote that $\\inf_S f$ is a shorthand for $\\inf_{x\\in S} f(x)$, and similar for $\\sup_S f$ and $\\max_S f$. Setwise local minima include setwise global minima as a special case, and we say a setwise local minimum is \\textit{spurious} if it is not a setwise global minimum. Note that \\Cref{def:setwise-local-min} is not exactly the same as \\cite[Definition 2.5]{joszneurips2018} because we do not require a setwise local minimum to be a compact set. In other words, a setwise local minimum can be either bounded or unbounded, and we say a (spurious) setwise local minimum is a (spurious) \\textit{local minimum at infinity} if it is unbounded. For example, consider the loss function of a one-hidden layer neural network with sigmoid activation $\\sigma$ and two data points $(1,1)$ and $(-1,-3)$ in \\Cref{fig:setwise}. One can see that $S$  is a setwise local minimum (in particular, a local minimum at infinity) and $U$ is the corresponding open set in \\Cref{def:setwise-local-min}. Finally, observe that $\\mathbb{R}^n$ is a strict setwise local minimum at infinity of any function.\n","start":14652,"end":15732}]},{"id":"S009","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"闭包内部与边界","source_units":[{"raw_tex":"Now we introduce one of the most useful properties of setwise local minima in \\Cref{lem:bdd-const}. This property is intuitive and will be used in different scenarios throughout this paper. Let $\\bar{S}$, $S^\\circ$, and $\\partial S := \\bar{S} \\setminus S^\\circ$ respectively denote the closure, interior, and boundary of a subset $S$ of $\\mathbb{R}^n$.\n","start":15963,"end":16316}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"强制性","source_units":[{"raw_tex":"Finally, we discuss the case of coercive functions. Recall that a function $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ is coercive if $f(x)\\to\\infty$ as $\\|x\\|\\to\\infty$. \n","start":19545,"end":19712}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":["def:locally-lipschitz"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"局部 Lipschitz 性","source_units":[{"raw_tex":"\\begin{definition}\\label{def:locally-lipschitz}\nA function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}^m$ is \\textit{locally Lipschitz} if for all $a \\in \\mathbb{R}^n$, there exist positive constants $r$ and $L$ such that \n\t\\begin{equation*}\n\t\t\\forall\\, x,y \\in B(a,r), ~~~ \\|f(x)-f(y)\\| \\leqslant L \\|x-y\\|.\n\t\\end{equation*}\n\\end{definition}\n\n","start":20562,"end":20899}]},{"id":"S012","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S012"],"declared_external":true,"declaration_text":["原先决条件：Rademacher differentiability theorem；evans2018measure, Theorem 3.2。"],"source_only":false,"title":"几乎处处可微背景","source_units":[{"raw_tex":"Notice that for a locally Lipschitz function, by \\cite[Theorem 3.2]{evans2018measure}, the derivative exists almost everywhere. Without any assumption on convexity, in order to ensure the existence of a subdifferential, we adopt the notion of Clarke subdifferential. \n\n","start":20899,"end":21168}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["def:Clarke"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"Clarke 次微分","source_units":[{"raw_tex":"\\begin{definition}\n\\label{def:Clarke}\n\\cite[p. 27]{clarke1990} Let $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$ be a locally Lipschitz function. The \\textit{Clarke subdifferential} is the set-valued mapping $\\partial f$ from $\\mathbb{R}^n$ to the subsets of $\\mathbb{R}^n$ defined for all $x \\in \\mathbb{R}^n$ by\n    \\begin{equation*}\n    \\partial f(x) := \\{ s \\in \\mathbb{R}^n ~\\vert~ f^\\circ(x,d) \\geqslant \\langle s , d \\rangle, ~ \\forall\\, d\\in \\mathbb{R}^n \\} \n\\end{equation*}\nwhere\n\\begin{equation*}\n    f^\\circ(x,d) := \\limsup_{\\tiny\\begin{array}{c} y\\rightarrow x \\\\\n    t \\searrow 0\n    \\end{array}\n    } \\frac{f(y+td)-f(y)}{t}.\n\\end{equation*}\n\\end{definition}\n\n","start":21168,"end":21835}]},{"id":"S014","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S014"],"declared_external":true,"declaration_text":["原先决条件：clarke1990, Proposition 2.1.2(a)。"],"source_only":false,"title":"次微分的非空凸紧性","source_units":[{"raw_tex":"It is well known that for any locally Lipschitz function $f$ and any $x\\in\\mathbb{R}^n$, the Clarke subdifferential $\\partial f(x)$ is a nonempty, convex, and compact set \\cite[Proposition 2.1.2(a)]{clarke1990}. Similar to differentiable functions, a point $x\\in\\mathbb{R}^n$ is called (Clarke) \\textit{critical point} if $0\\in\\partial f(x)$. A real number $y$ is called a (Clarke) \\textit{critical value} of $f$ if there exists a (Clarke) critical point $x\\in\\mathbb{R}^n$ of $f$ such that $f(x)=y$. \n","start":21835,"end":22337}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"Clarke 临界点与临界值","source_units":[{"raw_tex":"It is well known that for any locally Lipschitz function $f$ and any $x\\in\\mathbb{R}^n$, the Clarke subdifferential $\\partial f(x)$ is a nonempty, convex, and compact set \\cite[Proposition 2.1.2(a)]{clarke1990}. Similar to differentiable functions, a point $x\\in\\mathbb{R}^n$ is called (Clarke) \\textit{critical point} if $0\\in\\partial f(x)$. A real number $y$ is called a (Clarke) \\textit{critical value} of $f$ if there exists a (Clarke) critical point $x\\in\\mathbb{R}^n$ of $f$ such that $f(x)=y$. \n","start":21835,"end":22337}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":["def:absolute_continuity"],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"绝对连续曲线","source_units":[{"raw_tex":"\\begin{definition}\n    \\label{def:absolute_continuity} \n    \\cite[Definition 1 p. 12]{aubin1984differential}\n    Given two real numbers $a<b$, a function $x:[a,b]\\rightarrow\\mathbb{R}^n$ is \\textit{absolutely continuous} if for all $\\epsilon>0$, there exists $\\delta>0$ such that, for any finite collection of disjoint subintervals $[a_1,b_1],\\hdots,[a_m,b_m]$ of $[a,b]$ such that $\\sum_{i=1}^m b_i-a_i \\leqslant \\delta$, we have $\\sum_{i=1}^m \\|x(b_i) - x(a_i)\\| \\leqslant \\epsilon$.\n\\end{definition}\n\n","start":22499,"end":23003}]},{"id":"S017","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S017"],"declared_external":true,"declaration_text":["原先决条件：标准绝对连续演算及 nielsen1997introduction, Theorem 20.8。"],"source_only":false,"title":"绝对连续的积分表征","source_units":[{"raw_tex":"By virtue of \\cite[Theorem 20.8]{nielsen1997introduction}, $x:[a,b]\\rightarrow\\mathbb{R}^n$ is absolutely continuous if and only if it is differentiable almost everywhere on $(a,b)$, its derivative $x'$ is Lebesgue integrable, and $x(t) - x(a) = \\int_a^t x'(\\tau)d\\tau$ for all $t\\in [a,b]$. Given a non-compact interval $I$ of $\\mathbb{R}$, $x:I\\rightarrow \\mathbb{R}^n$ is absolutely continuous if it is absolutely continuous on all compact subintervals of $I$.\n\n","start":23003,"end":23468}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":["eq:ivp-subgrad"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"次梯度轨迹","source_units":[{"raw_tex":"An absolutely continuous function $x:[0,\\infty)\\rightarrow\\mathbb{R}^n$ is called a \\textit{subgradient trajectory} of $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ starting at $x_0\\in\\mathbb{R}^n$ if it satisfies the following differential inclusion with initial condition: \n\\begin{equation}\\label{eq:ivp-subgrad}\n    x'(t) \\in -\\partial f(x(t)),\\quad \\text{for almost every $t\\geqslant 0$},\\quad x(0)=x_0,\n\\end{equation}\nwhere ``almost every'' means all elements except for those in a set of zero measure. \n","start":23468,"end":23969}]},{"id":"S019","semantic_type":"definition","kind":"definition","labels":["def:bdd-subgrad"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"有界轨迹的存在量词","source_units":[{"raw_tex":"\\begin{definition}\\label{def:bdd-subgrad}\nA locally Lipschitz lower bounded function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ has \\textit{bounded subgradient trajectories} if for any $x_0\\in\\mathbb{R}^n$, there exists a subgradient trajectory $x$ of $f$ starting at $x_0$ and a constant $r>0$, such that $\\|x(t)\\|\\leqslant r$ for all $t\\geqslant 0$. \n\\end{definition}\n","start":26391,"end":26756}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":["def:chain-rule"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"链式法则","source_units":[{"raw_tex":"\\begin{definition}\\label{def:chain-rule}\nLet $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ be locally Lipschitz. We say $f$ \\textit{admits a chain rule} if for any absolutely continuous function $x:[0,\\infty)\\rightarrow \\mathbb{R}^n$, we have\n\\begin{equation*}\n\t(f\\circ x)'(t) = \\langle v, x'(t)\\rangle, \\quad \\forall\\, v\\in\\partial f(x(t)),\n\\end{equation*}\nfor almost every $t\\in [0,\\infty)$. \n\\end{definition}\n\n","start":27670,"end":28075}]},{"id":"S021","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S021"],"declared_external":true,"declaration_text":["原先决条件：davis2020stochastic, Lemma 5.2；其余本行文献仅为背景讨论。"],"source_only":false,"title":"轨迹上的下降性质","source_units":[{"raw_tex":"Thus, for any locally Lipschitz function that admits a chain rule, by \\cite[Lemma 5.2]{davis2020stochastic}, the function value is always decreasing in time along the subgradient trajectory. A detailed discussion on what class of functions admits a chain rule can be found in \\cite{bolte2020conservative}. Note that general Lipschitz functions are far from admitting a chain rule since they generically have a maximal Clarke subdifferential \\cite{daniilidis2020pathological,borwein2000lipschitz,daniilidis2019linear}.\n","start":28075,"end":28593}]},{"id":"S022","semantic_type":"problem_setting","kind":"problem_setting","labels":["assm:funcClass"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"主定理的函数设定","source_units":[{"raw_tex":"\\begin{assumption}\\label{assm:funcClass}\nLet $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ be a function such that it  \n\\begin{enumerate}[label={(\\alph*)}]\n","start":28958,"end":29106},{"raw_tex":"\\end{enumerate}\n\\end{assumption}\n","start":29480,"end":29513}]},{"id":"S023","semantic_type":"assumption","kind":"assumption","labels":["assm:funcClass"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"主函数下有界","source_units":[{"raw_tex":"    \\item is bounded below, namely, $\\inf_{\\mathbb{R}^n}f > -\\infty$;\n","start":29106,"end":29176}]},{"id":"S024","semantic_type":"assumption","kind":"assumption","labels":["assm:funcClass"],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"主函数局部 Lipschitz","source_units":[{"raw_tex":"    \\item is locally Lipschitz continuous on $\\mathbb{R}^n$; see \\Cref{def:locally-lipschitz}; \n","start":29176,"end":29272}]},{"id":"S025","semantic_type":"assumption","kind":"assumption","labels":["assm:funcClass"],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"主函数满足链式法则","source_units":[{"raw_tex":"    \\item admits a chain rule; see \\Cref{def:chain-rule}; \n","start":29272,"end":29331}]},{"id":"S026","semantic_type":"assumption","kind":"assumption","labels":["assm:funcClass"],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"临界值有限","source_units":[{"raw_tex":"    \\item has finitely many critical values; see \\Cref{subsec:clarke}; \n","start":29331,"end":29403}]},{"id":"S027","semantic_type":"assumption","kind":"assumption","labels":["assm:funcClass"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"主函数具有有界轨迹","source_units":[{"raw_tex":"    \\item has bounded subgradient trajectories; see \\Cref{def:bdd-subgrad}. \n","start":29403,"end":29480}]},{"id":"S028","semantic_type":"theorem","kind":"theorem","labels":["thm:main-intro"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"点态与集合无伪极小等价","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm:main-intro}\nSuppose a locally Lipschitz function is bounded below, admits a chain rule, has finitely many critical values, and has bounded subgradient trajectories. Then it has no spurious local minima if and only if it has no spurious setwise local minima. \n\\end{theorem}","start":9110,"end":9408}]},{"id":"S029","semantic_type":"lemma","kind":"lemma","labels":["lem:bdd-const","lemma:boundary"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"集合极小的边界恒值","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:bdd-const}\n\\label{lemma:boundary}\nIf $S$ is a setwise local minimum of a continuous function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$, then $f(x)=\\sup_S f$ for all $x\\in\\partial S$.\n\\end{lemma}","start":16317,"end":16531}]},{"id":"S030","semantic_type":"proof","kind":"proof","labels":["app:lem:bdd-const","inf_sup"],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"边界引理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nSee \\Cref{app:lem:bdd-const}. \n\\end{proof}","start":16532,"end":16588},{"raw_tex":"\\section{Proof of \\texorpdfstring{\\Cref{lem:bdd-const}}{}}\\label{app:lem:bdd-const}\nLet $S \\subset \\mathbb{R}^n$ be a setwise local minimum of a continuous function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. Let $U\\supset S$ be an open set such that $f(x)\\leqslant f(y)$ for all $x \\in S$ and $y\\in U\\setminus S$. Note that $S$ is closed, so its boundary is defined by $\\partial S := S\\setminus S^\\circ$. Let $z\\in\\partial S$ and consider any real number $\\epsilon>0$. Since $f(z)+(-\\epsilon,\\epsilon)$ is a neighborhood of $f(z)$, by continuity of $f$, there exists a neighborhood $N(z)$ of $z$ such that $f(N(z))\\subset f(z)+(-\\epsilon,\\epsilon)$. Since $U$ is a neighborhood of $z$, $N'(z):=U \\cap N(z)$ is also a neighborhood of $z$ with $f(N'(z))\\subset f(z)+(-\\epsilon,\\epsilon)$. The set $N'(z)\\cap S$ is nonempty because $z \\in S$ and the set $N'(z)\\setminus S$ is nonempty because $z\\in\\partial S$. For any $x \\in N'(z) \\cap S$ and $y \\in N'(z)\\setminus S$, it follows that\n\\begin{equation*}\n     \\inf_{U \\setminus S} f -\\epsilon \\leqslant f(y)-\\epsilon < f(z) < f(x)+\\epsilon \\leqslant \\sup_S f+\\epsilon \\leqslant \\inf_{U\\setminus S} f +\\epsilon. \n\\end{equation*}\nThe last inequality follows from the definition of setwise local minima. As $\\epsilon>0$ was arbitrary, we deduce that \n\\begin{equation*}\n\\label{inf_sup}\n    \\inf_{U \\setminus S} f = f(z) = \\sup_S f.\n\\end{equation*}\nThus, $f$ is a constant on the boundary of $S$ and $f$ attains its maximum over $S$ on the boundary of $S$. \n\n\n","start":71033,"end":72530}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":["app:lem:bdd-const"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"边界点的内外邻域","source_units":[{"raw_tex":"Let $S \\subset \\mathbb{R}^n$ be a setwise local minimum of a continuous function $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$. Let $U\\supset S$ be an open set such that $f(x)\\leqslant f(y)$ for all $x \\in S$ and $y\\in U\\setminus S$. Note that $S$ is closed, so its boundary is defined by $\\partial S := S\\setminus S^\\circ$. Let $z\\in\\partial S$ and consider any real number $\\epsilon>0$. Since $f(z)+(-\\epsilon,\\epsilon)$ is a neighborhood of $f(z)$, by continuity of $f$, there exists a neighborhood $N(z)$ of $z$ such that $f(N(z))\\subset f(z)+(-\\epsilon,\\epsilon)$. Since $U$ is a neighborhood of $z$, $N'(z):=U \\cap N(z)$ is also a neighborhood of $z$ with $f(N'(z))\\subset f(z)+(-\\epsilon,\\epsilon)$. The set $N'(z)\\cap S$ is nonempty because $z \\in S$ and the set $N'(z)\\setminus S$ is nonempty because $z\\in\\partial S$. For any $x \\in N'(z) \\cap S$ and $y \\in N'(z)\\setminus S$, it follows that\n","start":71117,"end":72012}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":["app:lem:bdd-const","inf_sup"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"内外函数值夹逼","source_units":[{"raw_tex":"\\begin{equation*}\n     \\inf_{U \\setminus S} f -\\epsilon \\leqslant f(y)-\\epsilon < f(z) < f(x)+\\epsilon \\leqslant \\sup_S f+\\epsilon \\leqslant \\inf_{U\\setminus S} f +\\epsilon. \n\\end{equation*}\nThe last inequality follows from the definition of setwise local minima. As $\\epsilon>0$ was arbitrary, we deduce that \n\\begin{equation*}\n\\label{inf_sup}\n    \\inf_{U \\setminus S} f = f(z) = \\sup_S f.\n\\end{equation*}\n","start":72012,"end":72419}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":["app:lem:bdd-const"],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"边界结论及其作用域","source_units":[{"raw_tex":"Thus, $f$ is a constant on the boundary of $S$ and $f$ attains its maximum over $S$ on the boundary of $S$. \n\n\n","start":72419,"end":72530}]},{"id":"S034","semantic_type":"proposition","kind":"proposition","labels":["prop:exist"],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"全时间次梯度轨迹存在","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:exist}\nIf $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ is locally Lipschitz and bounded below, then there exists a subgradient trajectory of $f$ starting at arbitrary $x_0\\in\\mathbb{R}^n$. \n\\end{proposition}","start":25092,"end":25323}]},{"id":"S035","semantic_type":"proof","kind":"proof","labels":["app:prop:exist","eq:1/2-holder","eq:1/2-holder_cs","eq:L2-bounded"],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"轨迹存在性的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nSee \\Cref{app:prop:exist}. \n\\end{proof}","start":25324,"end":25377},{"raw_tex":"\\section{Proof of \\texorpdfstring{\\Cref{prop:exist}}{}}\\label{app:prop:exist}\nFor a fixed real number $\\tau>0$, define a sequence $x_k^\\tau$ recurrently by letting $x_0^\\tau:=x_0$ and \n\\begin{equation*}\n    x_{k+1}^\\tau \\in \\arg\\min_{x\\in\\mathbb{R}^n} \\left\\{f(x)+\\frac{\\lVert x-x_k^\\tau\\rVert^2}{2\\tau}\\right\\}, \\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\nA solution exists because $f$ is bounded below and the objective function is coercive. Any solution satisfies \n\\begin{equation*}\n    v_{k+1}^\\tau := \\frac{x_{k+1}^\\tau-x_k^\\tau}{\\tau} \\in -\\partial f(x_{k+1}^\\tau), \\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\nDefine two functions $x^\\tau,\\tilde{x}^\\tau:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ where $\\mathbb{R}_+:=[0,\\infty)$ by\n\\begin{equation*}\n    x^\\tau(t) := x_{k+1}^\\tau,\\quad \\tilde{x}^\\tau(t) := x_k^\\tau + (t-k\\tau)v_{k+1}^\\tau,\\quad \\forall t\\in(k\\tau,(k+1)\\tau]\n\\end{equation*}\nfor all $k\\in\\mathbb{N}$, with initial condition $x^\\tau(0)=\\tilde{x}^\\tau(0)=x_0$. Note that $\\tilde{x}^\\tau$ is absolutely continuous because it is piecewise affine. On the contrary, $x^\\tau$ is not continuous. Also, define $v^{\\tau}:\\mathbb{R}_+\\rightarrow\\mathbb{R}^n$ by \n\\begin{equation*}\n    v^\\tau(t) := v_{k+1}^\\tau, \\quad \\forall t\\in(k\\tau,(k+1)\\tau],\\quad \\forall k\\in\\mathbb{N}, \n\\end{equation*}\nand choose $v^\\tau(0)\\in -\\partial f(x_0)$. Since  $(\\tilde{x}^\\tau)'=v^\\tau$ on $(k\\tau,(k+1)\\tau)$ for all $k \\in \\mathbb{N}$, and $v^\\tau(t)\\in -\\partial f(x^\\tau(t))$ for all $t\\geqslant 0$, we conclude that $(\\tilde{x}^\\tau)'(t)\\in -\\partial f(x^\\tau(t))$ for almost every $t\\in\\mathbb{R}_+$. By optimality of $x_{k+1}^\\tau$, we have \n\\begin{equation*}\n    f(x_{k+1}^\\tau) + \\frac{\\lVert x_{k+1}^\\tau-x_k^\\tau\\rVert^2}{2\\tau} \\leqslant f(x_k^\\tau), \\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\nFor any $l\\in\\mathbb{N}$, we have \n\\begin{equation*}\n    \\sum_{k=0}^l\\frac{\\lVert x_{k+1}^\\tau-x_k^\\tau\\rVert^2}{2\\tau} \\leqslant f(x_0^\\tau) - f(x_{l+1}^\\tau) \\leqslant f(x_0) - \\inf_{\\mathbb{R}^n} f =: C < \\infty\n\\end{equation*}\nsince $f$ is bounded below. Observe that \n\\begin{equation*}\n    \\sum_{k=0}^l\\frac{\\lVert x_{k+1}^\\tau-x_k^\\tau\\rVert^2}{2\\tau} = \\sum_{k=0}^l\\frac{\\tau}{2}\\lVert v_{k+1}^\\tau\\rVert^2 = \\frac{1}{2}\\sum_{k=0}^l\\int_{k\\tau}^{(k+1)\\tau}\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt. \n\\end{equation*}\nFix $T \\geqslant 0$ from now on. From the above, we have\n\\begin{equation}\\label{eq:L2-bounded}\n    \\int_0^T\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt \\leqslant \\sum_{k=0}^{\\lfloor T/\\tau\\rfloor}\\int_{k\\tau}^{(k+1)\\tau}\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt \\leqslant 2C. \n\\end{equation}\nSince $\\tilde{x}^\\tau$ is absolutely continuous, for any $s,t\\in [0,T]$ we have\n\\begin{subequations}\\label{eq:1/2-holder}\n    \\begin{align}\n    \\|\\tilde{x}^\\tau(t)-\\tilde{x}^\\tau(s)\\| &= \\left\\|\\int_s^t(\\tilde{x}^\\tau)'(u)\\;du\\right\\| \\\\\n    &\\leqslant \\left(\\int_0^T\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt\\right)^{1/2}\\vert t-s\\vert^{1/2} \\leqslant \\sqrt{2C}\\vert t-s\\vert^{1/2} \\label{eq:1/2-holder_cs} \n\\end{align}\n\\end{subequations}\nwhere we use the Cauchy-Schwarz inequality. Now one can see $(\\tilde{x}^\\tau)_{\\tau>0}$ is a family of uniformly bounded and equicontinuous functions on the compact interval $[0,T]$. Therefore, by Arzel\\`a-Ascoli theorem \\cite[Theorem 7.25]{rudin1964principles}, there exists a sequence of positive reals $(\\tau_k)_{k\\in\\mathbb{N}}$ such that $\\tau_k\\to 0$ and $\\tilde{x}^{\\tau_k}\\to x^*$ uniformly on $[0,T]$ as $k\\to\\infty$. For all $k\\in \\mathbb{N}$ and $t\\in (k\\tau,(k+1)\\tau]$, we have $\\tilde{x}^\\tau((k+1)\\tau)=x_k^\\tau + \\tau v^\\tau_{k+1} = x^\\tau_{k+1} = x^\\tau(t)$. Thus $\\|\\tilde{x}^\\tau(t)-x^\\tau(t)\\| = \\|\\tilde{x}^\\tau(t)-\\tilde{x}^\\tau((k+1)\\tau)\\| \\leqslant \\sqrt{2C}\\tau^{1/2}$ for all $t\\in [0,T]$ where the inequality is due to \\Cref{eq:1/2-holder} (take $s:=(k+1)\\tau$).\nCombined with the fact that $\\tilde{x}^{\\tau_k}\\to x^*$ uniformly on $[0,T]$, one can see that $x^{\\tau_k}\\to x^*$ uniformly on $[0,T]$. Since \\cref{eq:L2-bounded} implies that $((\\tilde{x}^{\\tau_k})')_{k\\in\\mathbb{N}}$ is a bounded sequence in $L^2([0,T],\\mathbb{R}^n)$, there exists a subsequence $(\\tau_{k_j})_{j\\in\\mathbb{N}}$ such that $(\\tilde{x}^{\\tau_{k_j}})'\\to v^*$ weakly in $L^1([0,T],\\mathbb{R}^n)$ as $j\\to\\infty$ by \\cite[Corollary 14 p. 413]{Fitzpatrick2010}. Since $\\tilde{x}^{\\tau_{k_j}}$ is absolutely continuous, for all $t\\in[0,T]$, we have \n\\begin{equation*}\n    \\tilde{x}^{\\tau_{k_j}}(t) - \\tilde{x}^{\\tau_{k_j}}(0) = \\int_0^t(\\tilde{x}^{\\tau_{k_j}})'(u)\\;du. \n\\end{equation*}\nTake $j\\to\\infty$ on both sides, we have \n\\begin{equation*}\n    x^*(t) - x^*(0) = \\int_0^tv^*(u)\\;du, \n\\end{equation*}\nwhere the convergence of the integral relies on the fact that the constant functions equal to the canonical basis of $\\mathbb{R}^n$ lie in $L^\\infty([0,T],\\mathbb{R}^n)$. Thus, $x^*$ is absolutely continuous and $(x^*)'(t)=v^*(t)$ for almost every $t\\in[0,T]$. Recall that for all $k\\in \\mathbb{N}$, it holds for almost every $t\\in[0,T]$ that\n\\begin{equation*}\n    (\\tilde{x}^{\\tau_k})'(t) = v^{\\tau_k}(t) \\in -\\partial f(x^{\\tau_k}(t)). \n\\end{equation*}\nSince $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d) p. 29]{clarke1990} with nonempty compact values \\cite[2.1.2 Proposition (a) p. 27]{clarke1990}, hence proper upper hemicontinuous \\cite[Proposition 1 p. 60]{aubin1984differential}. In addition, $x^{\\tau_k}\\to x^*$ uniformly on $[0,T]$ and $(\\tilde{x}^{\\tau_k})'\\to (x^*)'$ weakly in $L^1([0,T],\\mathbb{R}^n)$. Therefore,  $(x^*)'(t)\\in-\\partial f(x^*(t))$ for almost all $t\\in[0,T]$ by \\cite[Theorem 1 p. 60]{aubin1984differential}\\footnote{In the theorem we take $F:=-\\partial f$, $X=Y:=\\mathbb{R}^n$, and $I:=[0,T]$.}. The initial condition also holds since $\\tilde{x}^{\\tau}(0)=x_0$ for all $\\tau>0$. \n\\vskip 1ex\nWe have proved that for any initial point $x_0$, there exists $x^*:[0,T]\\rightarrow\\mathbb{R}^n$ such that $(x^*)'(t)=-\\partial f(x^*(t))$ holds for almost every $t\\in [0,T]$ with any $T>0$. Since $T$ is independent of $x_0$, by setting $T=1$, there exists a sequence of absolutely continuous functions $(x_k)_{k\\in\\mathbb{N}}$ such that \n\\begin{equation*}\n    x_k'(t)\\in -\\partial f(x_k(t)), \\quad \\mathrm{for~a.e.}~t \\in [0,1], \\quad x_k(0) = x_{k-1}(1), \n\\end{equation*}\nfor all $k\\in\\mathbb{N}$ where $x_{-1}(0)=x_0$. Therefore, the desired function $x:[0,\\infty)\\rightarrow \\mathbb{R}^n$ can be defined in a piecewise fashion by \n\\begin{equation*}\n    x(t) := x_k(t-k),\\quad t\\in[k,k+1),\\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\nBy construction, $x$ is absolutely continuous on any compact interval $[a,b]\\subset[0,\\infty)$. \n\n","start":76318,"end":83047}]},{"id":"S036","semantic_type":"algorithm","kind":"algorithm","labels":["app:prop:exist","eq:1/2-holder"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"近端离散迭代","source_units":[{"raw_tex":"\\section{Proof of \\texorpdfstring{\\Cref{prop:exist}}{}}\\label{app:prop:exist}\nFor a fixed real number $\\tau>0$, define a sequence $x_k^\\tau$ recurrently by letting $x_0^\\tau:=x_0$ and \n\\begin{equation*}\n    x_{k+1}^\\tau \\in \\arg\\min_{x\\in\\mathbb{R}^n} \\left\\{f(x)+\\frac{\\lVert x-x_k^\\tau\\rVert^2}{2\\tau}\\right\\}, \\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\n","start":76318,"end":76678}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"近端子问题可解","source_units":[{"raw_tex":"A solution exists because $f$ is bounded below and the objective function is coercive. Any solution satisfies \n","start":76678,"end":76789}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"离散速度的次微分包含","source_units":[{"raw_tex":"\\begin{equation*}\n    v_{k+1}^\\tau := \\frac{x_{k+1}^\\tau-x_k^\\tau}{\\tau} \\in -\\partial f(x_{k+1}^\\tau), \\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\n","start":76789,"end":76940}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"阶梯与仿射插值","source_units":[{"raw_tex":"Define two functions $x^\\tau,\\tilde{x}^\\tau:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ where $\\mathbb{R}_+:=[0,\\infty)$ by\n\\begin{equation*}\n    x^\\tau(t) := x_{k+1}^\\tau,\\quad \\tilde{x}^\\tau(t) := x_k^\\tau + (t-k\\tau)v_{k+1}^\\tau,\\quad \\forall t\\in(k\\tau,(k+1)\\tau]\n\\end{equation*}\nfor all $k\\in\\mathbb{N}$, with initial condition $x^\\tau(0)=\\tilde{x}^\\tau(0)=x_0$. Note that $\\tilde{x}^\\tau$ is absolutely continuous because it is piecewise affine. On the contrary, $x^\\tau$ is not continuous. Also, define $v^{\\tau}:\\mathbb{R}_+\\rightarrow\\mathbb{R}^n$ by \n\\begin{equation*}\n    v^\\tau(t) := v_{k+1}^\\tau, \\quad \\forall t\\in(k\\tau,(k+1)\\tau],\\quad \\forall k\\in\\mathbb{N}, \n\\end{equation*}\nand choose $v^\\tau(0)\\in -\\partial f(x_0)$. Since  $(\\tilde{x}^\\tau)'=v^\\tau$ on $(k\\tau,(k+1)\\tau)$ for all $k \\in \\mathbb{N}$, and $v^\\tau(t)\\in -\\partial f(x^\\tau(t))$ for all $t\\geqslant 0$, we conclude that $(\\tilde{x}^\\tau)'(t)\\in -\\partial f(x^\\tau(t))$ for almost every $t\\in\\mathbb{R}_+$. By optimality of $x_{k+1}^\\tau$, we have \n","start":76940,"end":77967}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"单步最优性不等式","source_units":[{"raw_tex":"\\begin{equation*}\n    f(x_{k+1}^\\tau) + \\frac{\\lVert x_{k+1}^\\tau-x_k^\\tau\\rVert^2}{2\\tau} \\leqslant f(x_k^\\tau), \\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\n","start":77967,"end":78128}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"离散能量的总界","source_units":[{"raw_tex":"For any $l\\in\\mathbb{N}$, we have \n\\begin{equation*}\n    \\sum_{k=0}^l\\frac{\\lVert x_{k+1}^\\tau-x_k^\\tau\\rVert^2}{2\\tau} \\leqslant f(x_0^\\tau) - f(x_{l+1}^\\tau) \\leqslant f(x_0) - \\inf_{\\mathbb{R}^n} f =: C < \\infty\n\\end{equation*}\nsince $f$ is bounded below. Observe that \n\\begin{equation*}\n    \\sum_{k=0}^l\\frac{\\lVert x_{k+1}^\\tau-x_k^\\tau\\rVert^2}{2\\tau} = \\sum_{k=0}^l\\frac{\\tau}{2}\\lVert v_{k+1}^\\tau\\rVert^2 = \\frac{1}{2}\\sum_{k=0}^l\\int_{k\\tau}^{(k+1)\\tau}\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt. \n\\end{equation*}\n","start":78128,"end":78658}]},{"id":"S042","semantic_type":"derived_fact","kind":"derived_fact","labels":["app:prop:exist","eq:1/2-holder","eq:L2-bounded"],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"统一 L² 导数界","source_units":[{"raw_tex":"Fix $T \\geqslant 0$ from now on. From the above, we have\n\\begin{equation}\\label{eq:L2-bounded}\n    \\int_0^T\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt \\leqslant \\sum_{k=0}^{\\lfloor T/\\tau\\rfloor}\\int_{k\\tau}^{(k+1)\\tau}\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt \\leqslant 2C. \n\\end{equation}\n","start":78658,"end":78961}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder","eq:1/2-holder_cs"],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"统一半阶 Hölder 估计","source_units":[{"raw_tex":"Since $\\tilde{x}^\\tau$ is absolutely continuous, for any $s,t\\in [0,T]$ we have\n\\begin{subequations}\\label{eq:1/2-holder}\n    \\begin{align}\n    \\|\\tilde{x}^\\tau(t)-\\tilde{x}^\\tau(s)\\| &= \\left\\|\\int_s^t(\\tilde{x}^\\tau)'(u)\\;du\\right\\| \\\\\n    &\\leqslant \\left(\\int_0^T\\left\\lVert (\\tilde{x}^\\tau)'(t)\\right\\rVert^2dt\\right)^{1/2}\\vert t-s\\vert^{1/2} \\leqslant \\sqrt{2C}\\vert t-s\\vert^{1/2} \\label{eq:1/2-holder_cs} \n\\end{align}\n\\end{subequations}\nwhere we use the Cauchy-Schwarz inequality. Now one can see $(\\tilde{x}^\\tau)_{\\tau>0}$ is a family of uniformly bounded and equicontinuous functions on the compact interval $[0,T]$. Therefore, by Arzel\\`a-Ascoli theorem \\cite[Theorem 7.25]{rudin1964principles}, there exists a sequence of positive reals $(\\tau_k)_{k\\in\\mathbb{N}}$ such that $\\tau_k\\to 0$ and $\\tilde{x}^{\\tau_k}\\to x^*$ uniformly on $[0,T]$ as $k\\to\\infty$. For all $k\\in \\mathbb{N}$ and $t\\in (k\\tau,(k+1)\\tau]$, we have $\\tilde{x}^\\tau((k+1)\\tau)=x_k^\\tau + \\tau v^\\tau_{k+1} = x^\\tau_{k+1} = x^\\tau(t)$. Thus $\\|\\tilde{x}^\\tau(t)-x^\\tau(t)\\| = \\|\\tilde{x}^\\tau(t)-\\tilde{x}^\\tau((k+1)\\tau)\\| \\leqslant \\sqrt{2C}\\tau^{1/2}$ for all $t\\in [0,T]$ where the inequality is due to \\Cref{eq:1/2-holder} (take $s:=(k+1)\\tau$).\n","start":78961,"end":80198}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"仿射插值的一致收敛子列","source_units":[{"raw_tex":"where we use the Cauchy-Schwarz inequality. Now one can see $(\\tilde{x}^\\tau)_{\\tau>0}$ is a family of uniformly bounded and equicontinuous functions on the compact interval $[0,T]$. Therefore, by Arzel\\`a-Ascoli theorem \\cite[Theorem 7.25]{rudin1964principles}, there exists a sequence of positive reals $(\\tau_k)_{k\\in\\mathbb{N}}$ such that $\\tau_k\\to 0$ and $\\tilde{x}^{\\tau_k}\\to x^*$ uniformly on $[0,T]$ as $k\\to\\infty$. For all $k\\in \\mathbb{N}$ and $t\\in (k\\tau,(k+1)\\tau]$, we have $\\tilde{x}^\\tau((k+1)\\tau)=x_k^\\tau + \\tau v^\\tau_{k+1} = x^\\tau_{k+1} = x^\\tau(t)$. Thus $\\|\\tilde{x}^\\tau(t)-x^\\tau(t)\\| = \\|\\tilde{x}^\\tau(t)-\\tilde{x}^\\tau((k+1)\\tau)\\| \\leqslant \\sqrt{2C}\\tau^{1/2}$ for all $t\\in [0,T]$ where the inequality is due to \\Cref{eq:1/2-holder} (take $s:=(k+1)\\tau$).\n","start":79407,"end":80198}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"两种插值趋于同一极限","source_units":[{"raw_tex":"where we use the Cauchy-Schwarz inequality. Now one can see $(\\tilde{x}^\\tau)_{\\tau>0}$ is a family of uniformly bounded and equicontinuous functions on the compact interval $[0,T]$. Therefore, by Arzel\\`a-Ascoli theorem \\cite[Theorem 7.25]{rudin1964principles}, there exists a sequence of positive reals $(\\tau_k)_{k\\in\\mathbb{N}}$ such that $\\tau_k\\to 0$ and $\\tilde{x}^{\\tau_k}\\to x^*$ uniformly on $[0,T]$ as $k\\to\\infty$. For all $k\\in \\mathbb{N}$ and $t\\in (k\\tau,(k+1)\\tau]$, we have $\\tilde{x}^\\tau((k+1)\\tau)=x_k^\\tau + \\tau v^\\tau_{k+1} = x^\\tau_{k+1} = x^\\tau(t)$. Thus $\\|\\tilde{x}^\\tau(t)-x^\\tau(t)\\| = \\|\\tilde{x}^\\tau(t)-\\tilde{x}^\\tau((k+1)\\tau)\\| \\leqslant \\sqrt{2C}\\tau^{1/2}$ for all $t\\in [0,T]$ where the inequality is due to \\Cref{eq:1/2-holder} (take $s:=(k+1)\\tau$).\nCombined with the fact that $\\tilde{x}^{\\tau_k}\\to x^*$ uniformly on $[0,T]$, one can see that $x^{\\tau_k}\\to x^*$ uniformly on $[0,T]$. Since \\cref{eq:L2-bounded} implies that $((\\tilde{x}^{\\tau_k})')_{k\\in\\mathbb{N}}$ is a bounded sequence in $L^2([0,T],\\mathbb{R}^n)$, there exists a subsequence $(\\tau_{k_j})_{j\\in\\mathbb{N}}$ such that $(\\tilde{x}^{\\tau_{k_j}})'\\to v^*$ weakly in $L^1([0,T],\\mathbb{R}^n)$ as $j\\to\\infty$ by \\cite[Corollary 14 p. 413]{Fitzpatrick2010}. Since $\\tilde{x}^{\\tau_{k_j}}$ is absolutely continuous, for all $t\\in[0,T]$, we have \n","start":79407,"end":80761}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"导数的弱收敛子列","source_units":[{"raw_tex":"Combined with the fact that $\\tilde{x}^{\\tau_k}\\to x^*$ uniformly on $[0,T]$, one can see that $x^{\\tau_k}\\to x^*$ uniformly on $[0,T]$. Since \\cref{eq:L2-bounded} implies that $((\\tilde{x}^{\\tau_k})')_{k\\in\\mathbb{N}}$ is a bounded sequence in $L^2([0,T],\\mathbb{R}^n)$, there exists a subsequence $(\\tau_{k_j})_{j\\in\\mathbb{N}}$ such that $(\\tilde{x}^{\\tau_{k_j}})'\\to v^*$ weakly in $L^1([0,T],\\mathbb{R}^n)$ as $j\\to\\infty$ by \\cite[Corollary 14 p. 413]{Fitzpatrick2010}. Since $\\tilde{x}^{\\tau_{k_j}}$ is absolutely continuous, for all $t\\in[0,T]$, we have \n","start":80198,"end":80761}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"极限曲线绝对连续","source_units":[{"raw_tex":"Combined with the fact that $\\tilde{x}^{\\tau_k}\\to x^*$ uniformly on $[0,T]$, one can see that $x^{\\tau_k}\\to x^*$ uniformly on $[0,T]$. Since \\cref{eq:L2-bounded} implies that $((\\tilde{x}^{\\tau_k})')_{k\\in\\mathbb{N}}$ is a bounded sequence in $L^2([0,T],\\mathbb{R}^n)$, there exists a subsequence $(\\tau_{k_j})_{j\\in\\mathbb{N}}$ such that $(\\tilde{x}^{\\tau_{k_j}})'\\to v^*$ weakly in $L^1([0,T],\\mathbb{R}^n)$ as $j\\to\\infty$ by \\cite[Corollary 14 p. 413]{Fitzpatrick2010}. Since $\\tilde{x}^{\\tau_{k_j}}$ is absolutely continuous, for all $t\\in[0,T]$, we have \n\\begin{equation*}\n    \\tilde{x}^{\\tau_{k_j}}(t) - \\tilde{x}^{\\tau_{k_j}}(0) = \\int_0^t(\\tilde{x}^{\\tau_{k_j}})'(u)\\;du. \n\\end{equation*}\nTake $j\\to\\infty$ on both sides, we have \n\\begin{equation*}\n    x^*(t) - x^*(0) = \\int_0^tv^*(u)\\;du, \n\\end{equation*}\nwhere the convergence of the integral relies on the fact that the constant functions equal to the canonical basis of $\\mathbb{R}^n$ lie in $L^\\infty([0,T],\\mathbb{R}^n)$. Thus, $x^*$ is absolutely continuous and $(x^*)'(t)=v^*(t)$ for almost every $t\\in[0,T]$. Recall that for all $k\\in \\mathbb{N}$, it holds for almost every $t\\in[0,T]$ that\n","start":80198,"end":81360}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"极限保持微分包含","source_units":[{"raw_tex":"where the convergence of the integral relies on the fact that the constant functions equal to the canonical basis of $\\mathbb{R}^n$ lie in $L^\\infty([0,T],\\mathbb{R}^n)$. Thus, $x^*$ is absolutely continuous and $(x^*)'(t)=v^*(t)$ for almost every $t\\in[0,T]$. Recall that for all $k\\in \\mathbb{N}$, it holds for almost every $t\\in[0,T]$ that\n\\begin{equation*}\n    (\\tilde{x}^{\\tau_k})'(t) = v^{\\tau_k}(t) \\in -\\partial f(x^{\\tau_k}(t)). \n\\end{equation*}\nSince $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d) p. 29]{clarke1990} with nonempty compact values \\cite[2.1.2 Proposition (a) p. 27]{clarke1990}, hence proper upper hemicontinuous \\cite[Proposition 1 p. 60]{aubin1984differential}. In addition, $x^{\\tau_k}\\to x^*$ uniformly on $[0,T]$ and $(\\tilde{x}^{\\tau_k})'\\to (x^*)'$ weakly in $L^1([0,T],\\mathbb{R}^n)$. Therefore,  $(x^*)'(t)\\in-\\partial f(x^*(t))$ for almost all $t\\in[0,T]$ by \\cite[Theorem 1 p. 60]{aubin1984differential}\\footnote{In the theorem we take $F:=-\\partial f$, $X=Y:=\\mathbb{R}^n$, and $I:=[0,T]$.}. The initial condition also holds since $\\tilde{x}^{\\tau}(0)=x_0$ for all $\\tau>0$. \n","start":81017,"end":82199}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"单位时间解的递推衔接","source_units":[{"raw_tex":"\\vskip 1ex\nWe have proved that for any initial point $x_0$, there exists $x^*:[0,T]\\rightarrow\\mathbb{R}^n$ such that $(x^*)'(t)=-\\partial f(x^*(t))$ holds for almost every $t\\in [0,T]$ with any $T>0$. Since $T$ is independent of $x_0$, by setting $T=1$, there exists a sequence of absolutely continuous functions $(x_k)_{k\\in\\mathbb{N}}$ such that \n\\begin{equation*}\n    x_k'(t)\\in -\\partial f(x_k(t)), \\quad \\mathrm{for~a.e.}~t \\in [0,1], \\quad x_k(0) = x_{k-1}(1), \n\\end{equation*}\nfor all $k\\in\\mathbb{N}$ where $x_{-1}(0)=x_0$. Therefore, the desired function $x:[0,\\infty)\\rightarrow \\mathbb{R}^n$ can be defined in a piecewise fashion by \n","start":82199,"end":82845}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":["app:prop:exist","eq:1/2-holder"],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"分段定义全时间轨迹","source_units":[{"raw_tex":"\\begin{equation*}\n    x(t) := x_k(t-k),\\quad t\\in[k,k+1),\\quad \\forall k\\in\\mathbb{N}. \n\\end{equation*}\nBy construction, $x$ is absolutely continuous on any compact interval $[a,b]\\subset[0,\\infty)$. \n\n","start":82845,"end":83047}]},{"id":"S051","semantic_type":"proof","kind":"proof","labels":["eq:optimization"],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"主定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\Cref{thm:main-intro}]\nLet $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ be a function satisfying \\Cref{assm:funcClass} If $f$ has no spurious setwise local minima, then $f$ has no spurious local minima. We next prove the converse. Let $S\\subset\\mathbb{R}^n$ be a setwise local minimum of $f$. We seek to show that $S$ is a setwise global minimum of $f$. If $S^\\circ = \\emptyset$, then by \\Cref{lem:bdd-const} $f(x) = \\sup_S f$ for all $x \\in \\partial S = \\bar{S} \\setminus S^\\circ = S$ since $S$ is closed by \\Cref{def:setwise-local-min}. Thus $f$ is constant on $S$. By definition of setwise local minima (\\Cref{def:setwise-local-min}), there exists an open subset $U$ of $\\mathbb{R}^n$ containing $S$ such that the constant value of $f$ on $S$ is less than or equal to $f(y)$ for all $y \\in U \\setminus S$. Thus every point in $S$ is a local minimum of $f$. Since every local minimum of $f$ is a global minimum, $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum according to \\Cref{def:setwise-global-min}. The rest of the proof deals with the case when $S^\\circ \\neq \\emptyset$. Let $C$ be the set of all critical points of $f$ in $S$ and consider the following optimization problem:\n\\begin{equation}\\label{eq:optimization}\n    \\inf_{x \\in C} f(x).\n\\end{equation}\nWe claim that the set of (global) solutions to \\Cref{eq:optimization} is nonempty, and that any solution is a local minimum of $f$ belonging to the setwise local minimum $S$. We first show that the feasible set of \\Cref{eq:optimization} is nonempty.\n\nSince $S^\\circ \\neq \\emptyset$, let $x_0 \\in S^\\circ$. If $x_0 \\in C$, then the feasible set $C$ is nonempty. We thus assume that $x_0 \\notin C$. Since $f$ is locally Lipschitz and bounded below, by \\Cref{prop:exist} there exists a subgradient trajectory $x:[0,\\infty)\\rightarrow\\mathbb{R}^n$ starting at $x_0$. We next show that $x([0,\\infty)) \\subset S$. We reason by contradiction and assume that $S^c \\cap x([0,\\infty)) \\neq \\emptyset$, where $S^c$ is the complement of $S$ in $\\mathbb{R}^n$. Then $S^\\circ$ and $S^c$ are disjoint open subsets of $\\mathbb{R}^n$ such that $S^\\circ \\cap x([0,\\infty)) \\neq \\emptyset$ (the intersection contains $x_0$), $S^c \\cap x([0,\\infty)) \\neq \\emptyset$, and $x([0,\\infty)) = (x([0,\\infty)) \\cap S^\\circ) \\cup (x([0,\\infty)) \\cap S^c) \\subset \\mathbb{R}^n \\setminus \\partial S$ (since\\footnote{If there exists $t>0$ such that $f(x(t)) = f(x(0))$, then $f(x(t)) - f(x(0)) = \\int_0^{t} \\|x'(s)\\|^2ds = 0$ and $x'(s) = 0$ for almost every $s\\in (0,t)$. Since $x'(s) \\in -\\partial f(x(s))$ for almost every $s>0$, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990} we have $0 \\in \\partial f(x(0))$.} $f(x(t)) < f(x(0)) = f(x_0) \\leqslant f(x)$ for all $t > 0$ and $x \\in \\partial S$, where the last inequality follows from \\Cref{lemma:boundary}). Thus the connected set $x([0,\\infty))$ is the union of two relatively open disjoint nonempty sets, which is a contradiction. \n\nSince $f$ has bounded subgradient trajectories and $x(\\cdot)$ is an arbitrary subgradient trajectory starting at $x_0$, by \\Cref{def:bdd-subgrad} and without loss of generality there exists $r>0$ such that $\\|x(t)\\| \\leqslant r$ for all $t\\geqslant 0$. We next show that there exists a critical point of $f$ in $B(0,r) \\cap S$. Suppose that there exist two constants $T,\\epsilon>0$ for which $\\lVert x'(t)\\rVert\\geqslant\\epsilon$ for all $t\\geqslant T$ such that $x'(t)\\in-\\partial f(x(t))$. By \\cite[Lemma 5.2]{davis2020stochastic}, we have $(f \\circ x)'(t) = -\\lVert x'(t)\\rVert^2 \\leqslant -\\epsilon^2$ for almost every $t\\geqslant T$. By integrating, we get $f(x(t)) - f(x(T)) \\leqslant - \\epsilon^2t$ and thus $f(x(t))$ converges to $-\\infty$ as $t\\to\\infty$. This is impossible since $x(t) \\in B(0,r)$ and $f$ is continuous. Hence there exists a time sequence $t_k\\to\\infty$ such that $\\lVert x'(t_k)\\rVert\\to 0$ as $k\\to\\infty$ and $x'(t_k)\\in-\\partial f(x(t_k))$ for all $k\\in\\mathbb{N} := \\{0,1,2,\\hdots\\}$. By the Bolzano–Weierstrass theorem, there exists a subsequence $x(t_{k_j})$ of $x(t_k)$ such that $x(t_{k_j})\\to\\tilde{x}\\in\\mathbb{R}^n$ as $j\\to\\infty$. Since $x'(t_{k_j})\\in -\\partial f(x(t_{k_j}))$, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990} we have $0\\in -\\partial f(\\tilde{x})$. Finally, since $x([0,\\infty)) \\subset S$ and $S$ is closed, we have $\\tilde{x}\\in C$. We obtain that $C\\neq \\emptyset$ as desired.\n\nSince $f$ has finitely many critical values and $C \\neq \\emptyset$, the set of solutions to \\Cref{eq:optimization} is nonempty. Let $x^* \\in C$ be a solution, that is to say $f(x^*) = \\min_C f$. Recall that $C$ is a subset of the setwise local minimum $S$. If $x^*$ is a local minimum of $f$, then it is a global minimum of $f$ in $S$ since every local minimum of $f$ is a global minimum. Thus $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum. For the remainder of the proof, we consider the case where $x^*$ is not a local minimum and show that this leads to a contradiction. We first show that there exists $s_0\\in S^\\circ$ such that $f(s_0)<f(x^*)$. This is clearly true if $x^*\\in S^\\circ$ since one can then find a ball centered at $x^*$ inside $S^\\circ$. If $x^*\\in S \\setminus S^\\circ = \\partial S$, then we reason by contradiction and assume that $f(x) \\geqslant f(x^*)$ for all $x\\in S^\\circ$. By \\Cref{lemma:boundary}, we have $f(x) = f(x^*) = \\sup_S f \\geqslant f(y) \\geqslant f(x^*)$ for all $(x,y)\\in (S \\setminus S^\\circ) \\times S^\\circ$. Hence $f(x^*) = f(x)$ for all $x\\in S$. Since $S$ is a setwise local minimum, there exists an open set $U$ such that $f(x) \\geqslant f(x^*)$ holds for all $x\\in U\\setminus S$. Thus $f(x)\\geqslant f(x^*)$ for all $x\\in U$ and $x^*$ is a local minimum. This yields a contradiction. Hence let $s_0\\in S^\\circ$ be such that $f(s_0)<f(x^*)$. The nonempty closed set $S' := S \\cap [f \\leqslant (f(s_0)+f(x^*))/2]$ is a setwise local minimum of $f$ where $[f\\leqslant \\alpha] := \\{ x \\in \\mathbb{R}^n ~\\vert~ f(x) \\leqslant \\alpha \\}$. Indeed, for all $x \\in S'$ and $y \\in U \\setminus S' = (U \\setminus S) \\cup (U \\setminus [f \\leqslant (f(s_0)+f(x^*))/2])$, we have\\footnote{Indeed, for any sets $A,B$, and $C$ it holds that $A\\setminus (B\\cap C) = (A \\setminus B) \\cup (A\\setminus C)$.} $f(x) \\leqslant (f(s_0)+f(x^*))/2 \\leqslant f(y)$. Since $s_0 \\in S^\\circ$ and $f(s_0)<(f(s_0)+f(x^*))/2$, we have $s_0 \\in S^\\circ \\cap [f\\leqslant (f(s_0)+f(x^*))/2]^\\circ = (S \\cap [f\\leqslant (f(s_0)+f(x^*))/2]))^\\circ = (S')^\\circ$. Hence the setwise local minimum $S'$ has nonempty interior. Also, $S' \\subset S$ and $\\sup_{S'} f < f(x^*) = \\min_C f$ where we remind the reader that $C$ is the set of critical points in $S$. Thus $S'$ is devoid of critical points. However, by the previous\nparagraph, setwise local minima of $f$ with nonempty interior must contain a critical point. This yields a contradiction.\n\\end{proof}","start":29515,"end":36408}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"等价命题的直接方向","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\Cref{thm:main-intro}]\nLet $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ be a function satisfying \\Cref{assm:funcClass} If $f$ has no spurious setwise local minima, then $f$ has no spurious local minima. We next prove the converse. Let $S\\subset\\mathbb{R}^n$ be a setwise local minimum of $f$. We seek to show that $S$ is a setwise global minimum of $f$. If $S^\\circ = \\emptyset$, then by \\Cref{lem:bdd-const} $f(x) = \\sup_S f$ for all $x \\in \\partial S = \\bar{S} \\setminus S^\\circ = S$ since $S$ is closed by \\Cref{def:setwise-local-min}. Thus $f$ is constant on $S$. By definition of setwise local minima (\\Cref{def:setwise-local-min}), there exists an open subset $U$ of $\\mathbb{R}^n$ containing $S$ such that the constant value of $f$ on $S$ is less than or equal to $f(y)$ for all $y \\in U \\setminus S$. Thus every point in $S$ is a local minimum of $f$. Since every local minimum of $f$ is a global minimum, $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum according to \\Cref{def:setwise-global-min}. The rest of the proof deals with the case when $S^\\circ \\neq \\emptyset$. Let $C$ be the set of all critical points of $f$ in $S$ and consider the following optimization problem:\n","start":29515,"end":30739}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"空内部集合的处理","source_units":[{"raw_tex":"Let $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ be a function satisfying \\Cref{assm:funcClass} If $f$ has no spurious setwise local minima, then $f$ has no spurious local minima. We next prove the converse. Let $S\\subset\\mathbb{R}^n$ be a setwise local minimum of $f$. We seek to show that $S$ is a setwise global minimum of $f$. If $S^\\circ = \\emptyset$, then by \\Cref{lem:bdd-const} $f(x) = \\sup_S f$ for all $x \\in \\partial S = \\bar{S} \\setminus S^\\circ = S$ since $S$ is closed by \\Cref{def:setwise-local-min}. Thus $f$ is constant on $S$. By definition of setwise local minima (\\Cref{def:setwise-local-min}), there exists an open subset $U$ of $\\mathbb{R}^n$ containing $S$ such that the constant value of $f$ on $S$ is less than or equal to $f(y)$ for all $y \\in U \\setminus S$. Thus every point in $S$ is a local minimum of $f$. Since every local minimum of $f$ is a global minimum, $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum according to \\Cref{def:setwise-global-min}. The rest of the proof deals with the case when $S^\\circ \\neq \\emptyset$. Let $C$ be the set of all critical points of $f$ in $S$ and consider the following optimization problem:\n","start":29561,"end":30739}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":["eq:optimization"],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"临界点上的辅助优化","source_units":[{"raw_tex":"Let $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ be a function satisfying \\Cref{assm:funcClass} If $f$ has no spurious setwise local minima, then $f$ has no spurious local minima. We next prove the converse. Let $S\\subset\\mathbb{R}^n$ be a setwise local minimum of $f$. We seek to show that $S$ is a setwise global minimum of $f$. If $S^\\circ = \\emptyset$, then by \\Cref{lem:bdd-const} $f(x) = \\sup_S f$ for all $x \\in \\partial S = \\bar{S} \\setminus S^\\circ = S$ since $S$ is closed by \\Cref{def:setwise-local-min}. Thus $f$ is constant on $S$. By definition of setwise local minima (\\Cref{def:setwise-local-min}), there exists an open subset $U$ of $\\mathbb{R}^n$ containing $S$ such that the constant value of $f$ on $S$ is less than or equal to $f(y)$ for all $y \\in U \\setminus S$. Thus every point in $S$ is a local minimum of $f$. Since every local minimum of $f$ is a global minimum, $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum according to \\Cref{def:setwise-global-min}. The rest of the proof deals with the case when $S^\\circ \\neq \\emptyset$. Let $C$ be the set of all critical points of $f$ in $S$ and consider the following optimization problem:\n\\begin{equation}\\label{eq:optimization}\n    \\inf_{x \\in C} f(x).\n\\end{equation}\nWe claim that the set of (global) solutions to \\Cref{eq:optimization} is nonempty, and that any solution is a local minimum of $f$ belonging to the setwise local minimum $S$. We first show that the feasible set of \\Cref{eq:optimization} is nonempty.\n","start":29561,"end":31069}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"从内部非临界点启动轨迹","source_units":[{"raw_tex":"Since $S^\\circ \\neq \\emptyset$, let $x_0 \\in S^\\circ$. If $x_0 \\in C$, then the feasible set $C$ is nonempty. We thus assume that $x_0 \\notin C$. Since $f$ is locally Lipschitz and bounded below, by \\Cref{prop:exist} there exists a subgradient trajectory $x:[0,\\infty)\\rightarrow\\mathbb{R}^n$ starting at $x_0$. We next show that $x([0,\\infty)) \\subset S$. We reason by contradiction and assume that $S^c \\cap x([0,\\infty)) \\neq \\emptyset$, where $S^c$ is the complement of $S$ in $\\mathbb{R}^n$. Then $S^\\circ$ and $S^c$ are disjoint open subsets of $\\mathbb{R}^n$ such that $S^\\circ \\cap x([0,\\infty)) \\neq \\emptyset$ (the intersection contains $x_0$), $S^c \\cap x([0,\\infty)) \\neq \\emptyset$, and $x([0,\\infty)) = (x([0,\\infty)) \\cap S^\\circ) \\cup (x([0,\\infty)) \\cap S^c) \\subset \\mathbb{R}^n \\setminus \\partial S$ (since\\footnote{If there exists $t>0$ such that $f(x(t)) = f(x(0))$, then $f(x(t)) - f(x(0)) = \\int_0^{t} \\|x'(s)\\|^2ds = 0$ and $x'(s) = 0$ for almost every $s\\in (0,t)$. Since $x'(s) \\in -\\partial f(x(s))$ for almost every $s>0$, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990} we have $0 \\in \\partial f(x(0))$.} $f(x(t)) < f(x(0)) = f(x_0) \\leqslant f(x)$ for all $t > 0$ and $x \\in \\partial S$, where the last inequality follows from \\Cref{lemma:boundary}). Thus the connected set $x([0,\\infty))$ is the union of two relatively open disjoint nonempty sets, which is a contradiction. \n","start":31070,"end":32479}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"非临界初值的严格下降","source_units":[{"raw_tex":"Since $S^\\circ \\neq \\emptyset$, let $x_0 \\in S^\\circ$. If $x_0 \\in C$, then the feasible set $C$ is nonempty. We thus assume that $x_0 \\notin C$. Since $f$ is locally Lipschitz and bounded below, by \\Cref{prop:exist} there exists a subgradient trajectory $x:[0,\\infty)\\rightarrow\\mathbb{R}^n$ starting at $x_0$. We next show that $x([0,\\infty)) \\subset S$. We reason by contradiction and assume that $S^c \\cap x([0,\\infty)) \\neq \\emptyset$, where $S^c$ is the complement of $S$ in $\\mathbb{R}^n$. Then $S^\\circ$ and $S^c$ are disjoint open subsets of $\\mathbb{R}^n$ such that $S^\\circ \\cap x([0,\\infty)) \\neq \\emptyset$ (the intersection contains $x_0$), $S^c \\cap x([0,\\infty)) \\neq \\emptyset$, and $x([0,\\infty)) = (x([0,\\infty)) \\cap S^\\circ) \\cup (x([0,\\infty)) \\cap S^c) \\subset \\mathbb{R}^n \\setminus \\partial S$ (since\\footnote{If there exists $t>0$ such that $f(x(t)) = f(x(0))$, then $f(x(t)) - f(x(0)) = \\int_0^{t} \\|x'(s)\\|^2ds = 0$ and $x'(s) = 0$ for almost every $s\\in (0,t)$. Since $x'(s) \\in -\\partial f(x(s))$ for almost every $s>0$, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990} we have $0 \\in \\partial f(x(0))$.} $f(x(t)) < f(x(0)) = f(x_0) \\leqslant f(x)$ for all $t > 0$ and $x \\in \\partial S$, where the last inequality follows from \\Cref{lemma:boundary}). Thus the connected set $x([0,\\infty))$ is the union of two relatively open disjoint nonempty sets, which is a contradiction. \n","start":31070,"end":32479}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"轨迹留在集合极小内","source_units":[{"raw_tex":"Since $S^\\circ \\neq \\emptyset$, let $x_0 \\in S^\\circ$. If $x_0 \\in C$, then the feasible set $C$ is nonempty. We thus assume that $x_0 \\notin C$. Since $f$ is locally Lipschitz and bounded below, by \\Cref{prop:exist} there exists a subgradient trajectory $x:[0,\\infty)\\rightarrow\\mathbb{R}^n$ starting at $x_0$. We next show that $x([0,\\infty)) \\subset S$. We reason by contradiction and assume that $S^c \\cap x([0,\\infty)) \\neq \\emptyset$, where $S^c$ is the complement of $S$ in $\\mathbb{R}^n$. Then $S^\\circ$ and $S^c$ are disjoint open subsets of $\\mathbb{R}^n$ such that $S^\\circ \\cap x([0,\\infty)) \\neq \\emptyset$ (the intersection contains $x_0$), $S^c \\cap x([0,\\infty)) \\neq \\emptyset$, and $x([0,\\infty)) = (x([0,\\infty)) \\cap S^\\circ) \\cup (x([0,\\infty)) \\cap S^c) \\subset \\mathbb{R}^n \\setminus \\partial S$ (since\\footnote{If there exists $t>0$ such that $f(x(t)) = f(x(0))$, then $f(x(t)) - f(x(0)) = \\int_0^{t} \\|x'(s)\\|^2ds = 0$ and $x'(s) = 0$ for almost every $s\\in (0,t)$. Since $x'(s) \\in -\\partial f(x(s))$ for almost every $s>0$, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990} we have $0 \\in \\partial f(x(0))$.} $f(x(t)) < f(x(0)) = f(x_0) \\leqslant f(x)$ for all $t > 0$ and $x \\in \\partial S$, where the last inequality follows from \\Cref{lemma:boundary}). Thus the connected set $x([0,\\infty))$ is the union of two relatively open disjoint nonempty sets, which is a contradiction. \n","start":31070,"end":32479}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"有界轨迹的选择说明","source_units":[{"raw_tex":"Since $f$ has bounded subgradient trajectories and $x(\\cdot)$ is an arbitrary subgradient trajectory starting at $x_0$, by \\Cref{def:bdd-subgrad} and without loss of generality there exists $r>0$ such that $\\|x(t)\\| \\leqslant r$ for all $t\\geqslant 0$. We next show that there exists a critical point of $f$ in $B(0,r) \\cap S$. Suppose that there exist two constants $T,\\epsilon>0$ for which $\\lVert x'(t)\\rVert\\geqslant\\epsilon$ for all $t\\geqslant T$ such that $x'(t)\\in-\\partial f(x(t))$. By \\cite[Lemma 5.2]{davis2020stochastic}, we have $(f \\circ x)'(t) = -\\lVert x'(t)\\rVert^2 \\leqslant -\\epsilon^2$ for almost every $t\\geqslant T$. By integrating, we get $f(x(t)) - f(x(T)) \\leqslant - \\epsilon^2t$ and thus $f(x(t))$ converges to $-\\infty$ as $t\\to\\infty$. This is impossible since $x(t) \\in B(0,r)$ and $f$ is continuous. Hence there exists a time sequence $t_k\\to\\infty$ such that $\\lVert x'(t_k)\\rVert\\to 0$ as $k\\to\\infty$ and $x'(t_k)\\in-\\partial f(x(t_k))$ for all $k\\in\\mathbb{N} := \\{0,1,2,\\hdots\\}$. By the Bolzano–Weierstrass theorem, there exists a subsequence $x(t_{k_j})$ of $x(t_k)$ such that $x(t_{k_j})\\to\\tilde{x}\\in\\mathbb{R}^n$ as $j\\to\\infty$. Since $x'(t_{k_j})\\in -\\partial f(x(t_{k_j}))$, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990} we have $0\\in -\\partial f(\\tilde{x})$. Finally, since $x([0,\\infty)) \\subset S$ and $S$ is closed, we have $\\tilde{x}\\in C$. We obtain that $C\\neq \\emptyset$ as desired.\n","start":32480,"end":33920}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"存在速度趋零的时刻","source_units":[{"raw_tex":"Since $f$ has bounded subgradient trajectories and $x(\\cdot)$ is an arbitrary subgradient trajectory starting at $x_0$, by \\Cref{def:bdd-subgrad} and without loss of generality there exists $r>0$ such that $\\|x(t)\\| \\leqslant r$ for all $t\\geqslant 0$. We next show that there exists a critical point of $f$ in $B(0,r) \\cap S$. Suppose that there exist two constants $T,\\epsilon>0$ for which $\\lVert x'(t)\\rVert\\geqslant\\epsilon$ for all $t\\geqslant T$ such that $x'(t)\\in-\\partial f(x(t))$. By \\cite[Lemma 5.2]{davis2020stochastic}, we have $(f \\circ x)'(t) = -\\lVert x'(t)\\rVert^2 \\leqslant -\\epsilon^2$ for almost every $t\\geqslant T$. By integrating, we get $f(x(t)) - f(x(T)) \\leqslant - \\epsilon^2t$ and thus $f(x(t))$ converges to $-\\infty$ as $t\\to\\infty$. This is impossible since $x(t) \\in B(0,r)$ and $f$ is continuous. Hence there exists a time sequence $t_k\\to\\infty$ such that $\\lVert x'(t_k)\\rVert\\to 0$ as $k\\to\\infty$ and $x'(t_k)\\in-\\partial f(x(t_k))$ for all $k\\in\\mathbb{N} := \\{0,1,2,\\hdots\\}$. By the Bolzano–Weierstrass theorem, there exists a subsequence $x(t_{k_j})$ of $x(t_k)$ such that $x(t_{k_j})\\to\\tilde{x}\\in\\mathbb{R}^n$ as $j\\to\\infty$. Since $x'(t_{k_j})\\in -\\partial f(x(t_{k_j}))$, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990} we have $0\\in -\\partial f(\\tilde{x})$. Finally, since $x([0,\\infty)) \\subset S$ and $S$ is closed, we have $\\tilde{x}\\in C$. We obtain that $C\\neq \\emptyset$ as desired.\n","start":32480,"end":33920}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"非空内部必含临界点","source_units":[{"raw_tex":"Since $f$ has bounded subgradient trajectories and $x(\\cdot)$ is an arbitrary subgradient trajectory starting at $x_0$, by \\Cref{def:bdd-subgrad} and without loss of generality there exists $r>0$ such that $\\|x(t)\\| \\leqslant r$ for all $t\\geqslant 0$. We next show that there exists a critical point of $f$ in $B(0,r) \\cap S$. Suppose that there exist two constants $T,\\epsilon>0$ for which $\\lVert x'(t)\\rVert\\geqslant\\epsilon$ for all $t\\geqslant T$ such that $x'(t)\\in-\\partial f(x(t))$. By \\cite[Lemma 5.2]{davis2020stochastic}, we have $(f \\circ x)'(t) = -\\lVert x'(t)\\rVert^2 \\leqslant -\\epsilon^2$ for almost every $t\\geqslant T$. By integrating, we get $f(x(t)) - f(x(T)) \\leqslant - \\epsilon^2t$ and thus $f(x(t))$ converges to $-\\infty$ as $t\\to\\infty$. This is impossible since $x(t) \\in B(0,r)$ and $f$ is continuous. Hence there exists a time sequence $t_k\\to\\infty$ such that $\\lVert x'(t_k)\\rVert\\to 0$ as $k\\to\\infty$ and $x'(t_k)\\in-\\partial f(x(t_k))$ for all $k\\in\\mathbb{N} := \\{0,1,2,\\hdots\\}$. By the Bolzano–Weierstrass theorem, there exists a subsequence $x(t_{k_j})$ of $x(t_k)$ such that $x(t_{k_j})\\to\\tilde{x}\\in\\mathbb{R}^n$ as $j\\to\\infty$. Since $x'(t_{k_j})\\in -\\partial f(x(t_{k_j}))$, by \\cite[2.1.5 Proposition (b) p. 29]{clarke1990} we have $0\\in -\\partial f(\\tilde{x})$. Finally, since $x([0,\\infty)) \\subset S$ and $S$ is closed, we have $\\tilde{x}\\in C$. We obtain that $C\\neq \\emptyset$ as desired.\n","start":32480,"end":33920}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"最低临界值取得","source_units":[{"raw_tex":"Since $f$ has finitely many critical values and $C \\neq \\emptyset$, the set of solutions to \\Cref{eq:optimization} is nonempty. Let $x^* \\in C$ be a solution, that is to say $f(x^*) = \\min_C f$. Recall that $C$ is a subset of the setwise local minimum $S$. If $x^*$ is a local minimum of $f$, then it is a global minimum of $f$ in $S$ since every local minimum of $f$ is a global minimum. Thus $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum. For the remainder of the proof, we consider the case where $x^*$ is not a local minimum and show that this leads to a contradiction. We first show that there exists $s_0\\in S^\\circ$ such that $f(s_0)<f(x^*)$. This is clearly true if $x^*\\in S^\\circ$ since one can then find a ball centered at $x^*$ inside $S^\\circ$. If $x^*\\in S \\setminus S^\\circ = \\partial S$, then we reason by contradiction and assume that $f(x) \\geqslant f(x^*)$ for all $x\\in S^\\circ$. By \\Cref{lemma:boundary}, we have $f(x) = f(x^*) = \\sup_S f \\geqslant f(y) \\geqslant f(x^*)$ for all $(x,y)\\in (S \\setminus S^\\circ) \\times S^\\circ$. Hence $f(x^*) = f(x)$ for all $x\\in S$. Since $S$ is a setwise local minimum, there exists an open set $U$ such that $f(x) \\geqslant f(x^*)$ holds for all $x\\in U\\setminus S$. Thus $f(x)\\geqslant f(x^*)$ for all $x\\in U$ and $x^*$ is a local minimum. This yields a contradiction. Hence let $s_0\\in S^\\circ$ be such that $f(s_0)<f(x^*)$. The nonempty closed set $S' := S \\cap [f \\leqslant (f(s_0)+f(x^*))/2]$ is a setwise local minimum of $f$ where $[f\\leqslant \\alpha] := \\{ x \\in \\mathbb{R}^n ~\\vert~ f(x) \\leqslant \\alpha \\}$. Indeed, for all $x \\in S'$ and $y \\in U \\setminus S' = (U \\setminus S) \\cup (U \\setminus [f \\leqslant (f(s_0)+f(x^*))/2])$, we have\\footnote{Indeed, for any sets $A,B$, and $C$ it holds that $A\\setminus (B\\cap C) = (A \\setminus B) \\cup (A\\setminus C)$.} $f(x) \\leqslant (f(s_0)+f(x^*))/2 \\leqslant f(y)$. Since $s_0 \\in S^\\circ$ and $f(s_0)<(f(s_0)+f(x^*))/2$, we have $s_0 \\in S^\\circ \\cap [f\\leqslant (f(s_0)+f(x^*))/2]^\\circ = (S \\cap [f\\leqslant (f(s_0)+f(x^*))/2]))^\\circ = (S')^\\circ$. Hence the setwise local minimum $S'$ has nonempty interior. Also, $S' \\subset S$ and $\\sup_{S'} f < f(x^*) = \\min_C f$ where we remind the reader that $C$ is the set of critical points in $S$. Thus $S'$ is devoid of critical points. However, by the previous\n","start":33921,"end":36275}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"局部极小解导出全局性","source_units":[{"raw_tex":"Since $f$ has finitely many critical values and $C \\neq \\emptyset$, the set of solutions to \\Cref{eq:optimization} is nonempty. Let $x^* \\in C$ be a solution, that is to say $f(x^*) = \\min_C f$. Recall that $C$ is a subset of the setwise local minimum $S$. If $x^*$ is a local minimum of $f$, then it is a global minimum of $f$ in $S$ since every local minimum of $f$ is a global minimum. Thus $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum. For the remainder of the proof, we consider the case where $x^*$ is not a local minimum and show that this leads to a contradiction. We first show that there exists $s_0\\in S^\\circ$ such that $f(s_0)<f(x^*)$. This is clearly true if $x^*\\in S^\\circ$ since one can then find a ball centered at $x^*$ inside $S^\\circ$. If $x^*\\in S \\setminus S^\\circ = \\partial S$, then we reason by contradiction and assume that $f(x) \\geqslant f(x^*)$ for all $x\\in S^\\circ$. By \\Cref{lemma:boundary}, we have $f(x) = f(x^*) = \\sup_S f \\geqslant f(y) \\geqslant f(x^*)$ for all $(x,y)\\in (S \\setminus S^\\circ) \\times S^\\circ$. Hence $f(x^*) = f(x)$ for all $x\\in S$. Since $S$ is a setwise local minimum, there exists an open set $U$ such that $f(x) \\geqslant f(x^*)$ holds for all $x\\in U\\setminus S$. Thus $f(x)\\geqslant f(x^*)$ for all $x\\in U$ and $x^*$ is a local minimum. This yields a contradiction. Hence let $s_0\\in S^\\circ$ be such that $f(s_0)<f(x^*)$. The nonempty closed set $S' := S \\cap [f \\leqslant (f(s_0)+f(x^*))/2]$ is a setwise local minimum of $f$ where $[f\\leqslant \\alpha] := \\{ x \\in \\mathbb{R}^n ~\\vert~ f(x) \\leqslant \\alpha \\}$. Indeed, for all $x \\in S'$ and $y \\in U \\setminus S' = (U \\setminus S) \\cup (U \\setminus [f \\leqslant (f(s_0)+f(x^*))/2])$, we have\\footnote{Indeed, for any sets $A,B$, and $C$ it holds that $A\\setminus (B\\cap C) = (A \\setminus B) \\cup (A\\setminus C)$.} $f(x) \\leqslant (f(s_0)+f(x^*))/2 \\leqslant f(y)$. Since $s_0 \\in S^\\circ$ and $f(s_0)<(f(s_0)+f(x^*))/2$, we have $s_0 \\in S^\\circ \\cap [f\\leqslant (f(s_0)+f(x^*))/2]^\\circ = (S \\cap [f\\leqslant (f(s_0)+f(x^*))/2]))^\\circ = (S')^\\circ$. Hence the setwise local minimum $S'$ has nonempty interior. Also, $S' \\subset S$ and $\\sup_{S'} f < f(x^*) = \\min_C f$ where we remind the reader that $C$ is the set of critical points in $S$. Thus $S'$ is devoid of critical points. However, by the previous\n","start":33921,"end":36275}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"非局部极小导出更低内部点","source_units":[{"raw_tex":"Since $f$ has finitely many critical values and $C \\neq \\emptyset$, the set of solutions to \\Cref{eq:optimization} is nonempty. Let $x^* \\in C$ be a solution, that is to say $f(x^*) = \\min_C f$. Recall that $C$ is a subset of the setwise local minimum $S$. If $x^*$ is a local minimum of $f$, then it is a global minimum of $f$ in $S$ since every local minimum of $f$ is a global minimum. Thus $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum. For the remainder of the proof, we consider the case where $x^*$ is not a local minimum and show that this leads to a contradiction. We first show that there exists $s_0\\in S^\\circ$ such that $f(s_0)<f(x^*)$. This is clearly true if $x^*\\in S^\\circ$ since one can then find a ball centered at $x^*$ inside $S^\\circ$. If $x^*\\in S \\setminus S^\\circ = \\partial S$, then we reason by contradiction and assume that $f(x) \\geqslant f(x^*)$ for all $x\\in S^\\circ$. By \\Cref{lemma:boundary}, we have $f(x) = f(x^*) = \\sup_S f \\geqslant f(y) \\geqslant f(x^*)$ for all $(x,y)\\in (S \\setminus S^\\circ) \\times S^\\circ$. Hence $f(x^*) = f(x)$ for all $x\\in S$. Since $S$ is a setwise local minimum, there exists an open set $U$ such that $f(x) \\geqslant f(x^*)$ holds for all $x\\in U\\setminus S$. Thus $f(x)\\geqslant f(x^*)$ for all $x\\in U$ and $x^*$ is a local minimum. This yields a contradiction. Hence let $s_0\\in S^\\circ$ be such that $f(s_0)<f(x^*)$. The nonempty closed set $S' := S \\cap [f \\leqslant (f(s_0)+f(x^*))/2]$ is a setwise local minimum of $f$ where $[f\\leqslant \\alpha] := \\{ x \\in \\mathbb{R}^n ~\\vert~ f(x) \\leqslant \\alpha \\}$. Indeed, for all $x \\in S'$ and $y \\in U \\setminus S' = (U \\setminus S) \\cup (U \\setminus [f \\leqslant (f(s_0)+f(x^*))/2])$, we have\\footnote{Indeed, for any sets $A,B$, and $C$ it holds that $A\\setminus (B\\cap C) = (A \\setminus B) \\cup (A\\setminus C)$.} $f(x) \\leqslant (f(s_0)+f(x^*))/2 \\leqslant f(y)$. Since $s_0 \\in S^\\circ$ and $f(s_0)<(f(s_0)+f(x^*))/2$, we have $s_0 \\in S^\\circ \\cap [f\\leqslant (f(s_0)+f(x^*))/2]^\\circ = (S \\cap [f\\leqslant (f(s_0)+f(x^*))/2]))^\\circ = (S')^\\circ$. Hence the setwise local minimum $S'$ has nonempty interior. Also, $S' \\subset S$ and $\\sup_{S'} f < f(x^*) = \\min_C f$ where we remind the reader that $C$ is the set of critical points in $S$. Thus $S'$ is devoid of critical points. However, by the previous\n","start":33921,"end":36275}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S064"],"declared_external":false,"source_only":false,"title":"构造更低的集合极小","source_units":[{"raw_tex":"Since $f$ has finitely many critical values and $C \\neq \\emptyset$, the set of solutions to \\Cref{eq:optimization} is nonempty. Let $x^* \\in C$ be a solution, that is to say $f(x^*) = \\min_C f$. Recall that $C$ is a subset of the setwise local minimum $S$. If $x^*$ is a local minimum of $f$, then it is a global minimum of $f$ in $S$ since every local minimum of $f$ is a global minimum. Thus $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum. For the remainder of the proof, we consider the case where $x^*$ is not a local minimum and show that this leads to a contradiction. We first show that there exists $s_0\\in S^\\circ$ such that $f(s_0)<f(x^*)$. This is clearly true if $x^*\\in S^\\circ$ since one can then find a ball centered at $x^*$ inside $S^\\circ$. If $x^*\\in S \\setminus S^\\circ = \\partial S$, then we reason by contradiction and assume that $f(x) \\geqslant f(x^*)$ for all $x\\in S^\\circ$. By \\Cref{lemma:boundary}, we have $f(x) = f(x^*) = \\sup_S f \\geqslant f(y) \\geqslant f(x^*)$ for all $(x,y)\\in (S \\setminus S^\\circ) \\times S^\\circ$. Hence $f(x^*) = f(x)$ for all $x\\in S$. Since $S$ is a setwise local minimum, there exists an open set $U$ such that $f(x) \\geqslant f(x^*)$ holds for all $x\\in U\\setminus S$. Thus $f(x)\\geqslant f(x^*)$ for all $x\\in U$ and $x^*$ is a local minimum. This yields a contradiction. Hence let $s_0\\in S^\\circ$ be such that $f(s_0)<f(x^*)$. The nonempty closed set $S' := S \\cap [f \\leqslant (f(s_0)+f(x^*))/2]$ is a setwise local minimum of $f$ where $[f\\leqslant \\alpha] := \\{ x \\in \\mathbb{R}^n ~\\vert~ f(x) \\leqslant \\alpha \\}$. Indeed, for all $x \\in S'$ and $y \\in U \\setminus S' = (U \\setminus S) \\cup (U \\setminus [f \\leqslant (f(s_0)+f(x^*))/2])$, we have\\footnote{Indeed, for any sets $A,B$, and $C$ it holds that $A\\setminus (B\\cap C) = (A \\setminus B) \\cup (A\\setminus C)$.} $f(x) \\leqslant (f(s_0)+f(x^*))/2 \\leqslant f(y)$. Since $s_0 \\in S^\\circ$ and $f(s_0)<(f(s_0)+f(x^*))/2$, we have $s_0 \\in S^\\circ \\cap [f\\leqslant (f(s_0)+f(x^*))/2]^\\circ = (S \\cap [f\\leqslant (f(s_0)+f(x^*))/2]))^\\circ = (S')^\\circ$. Hence the setwise local minimum $S'$ has nonempty interior. Also, $S' \\subset S$ and $\\sup_{S'} f < f(x^*) = \\min_C f$ where we remind the reader that $C$ is the set of critical points in $S$. Thus $S'$ is devoid of critical points. However, by the previous\n","start":33921,"end":36275}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S065"],"declared_external":false,"source_only":false,"title":"低于全部临界值的矛盾","source_units":[{"raw_tex":"Since $f$ has finitely many critical values and $C \\neq \\emptyset$, the set of solutions to \\Cref{eq:optimization} is nonempty. Let $x^* \\in C$ be a solution, that is to say $f(x^*) = \\min_C f$. Recall that $C$ is a subset of the setwise local minimum $S$. If $x^*$ is a local minimum of $f$, then it is a global minimum of $f$ in $S$ since every local minimum of $f$ is a global minimum. Thus $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum. For the remainder of the proof, we consider the case where $x^*$ is not a local minimum and show that this leads to a contradiction. We first show that there exists $s_0\\in S^\\circ$ such that $f(s_0)<f(x^*)$. This is clearly true if $x^*\\in S^\\circ$ since one can then find a ball centered at $x^*$ inside $S^\\circ$. If $x^*\\in S \\setminus S^\\circ = \\partial S$, then we reason by contradiction and assume that $f(x) \\geqslant f(x^*)$ for all $x\\in S^\\circ$. By \\Cref{lemma:boundary}, we have $f(x) = f(x^*) = \\sup_S f \\geqslant f(y) \\geqslant f(x^*)$ for all $(x,y)\\in (S \\setminus S^\\circ) \\times S^\\circ$. Hence $f(x^*) = f(x)$ for all $x\\in S$. Since $S$ is a setwise local minimum, there exists an open set $U$ such that $f(x) \\geqslant f(x^*)$ holds for all $x\\in U\\setminus S$. Thus $f(x)\\geqslant f(x^*)$ for all $x\\in U$ and $x^*$ is a local minimum. This yields a contradiction. Hence let $s_0\\in S^\\circ$ be such that $f(s_0)<f(x^*)$. The nonempty closed set $S' := S \\cap [f \\leqslant (f(s_0)+f(x^*))/2]$ is a setwise local minimum of $f$ where $[f\\leqslant \\alpha] := \\{ x \\in \\mathbb{R}^n ~\\vert~ f(x) \\leqslant \\alpha \\}$. Indeed, for all $x \\in S'$ and $y \\in U \\setminus S' = (U \\setminus S) \\cup (U \\setminus [f \\leqslant (f(s_0)+f(x^*))/2])$, we have\\footnote{Indeed, for any sets $A,B$, and $C$ it holds that $A\\setminus (B\\cap C) = (A \\setminus B) \\cup (A\\setminus C)$.} $f(x) \\leqslant (f(s_0)+f(x^*))/2 \\leqslant f(y)$. Since $s_0 \\in S^\\circ$ and $f(s_0)<(f(s_0)+f(x^*))/2$, we have $s_0 \\in S^\\circ \\cap [f\\leqslant (f(s_0)+f(x^*))/2]^\\circ = (S \\cap [f\\leqslant (f(s_0)+f(x^*))/2]))^\\circ = (S')^\\circ$. Hence the setwise local minimum $S'$ has nonempty interior. Also, $S' \\subset S$ and $\\sup_{S'} f < f(x^*) = \\min_C f$ where we remind the reader that $C$ is the set of critical points in $S$. Thus $S'$ is devoid of critical points. However, by the previous\nparagraph, setwise local minima of $f$ with nonempty interior must contain a critical point. This yields a contradiction.\n","start":33921,"end":36397}]},{"id":"S066","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S066"],"declared_external":false,"source_only":false,"title":"逆向证明的无伪点极小假设","source_units":[{"raw_tex":"Let $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ be a function satisfying \\Cref{assm:funcClass} If $f$ has no spurious setwise local minima, then $f$ has no spurious local minima. We next prove the converse. Let $S\\subset\\mathbb{R}^n$ be a setwise local minimum of $f$. We seek to show that $S$ is a setwise global minimum of $f$. If $S^\\circ = \\emptyset$, then by \\Cref{lem:bdd-const} $f(x) = \\sup_S f$ for all $x \\in \\partial S = \\bar{S} \\setminus S^\\circ = S$ since $S$ is closed by \\Cref{def:setwise-local-min}. Thus $f$ is constant on $S$. By definition of setwise local minima (\\Cref{def:setwise-local-min}), there exists an open subset $U$ of $\\mathbb{R}^n$ containing $S$ such that the constant value of $f$ on $S$ is less than or equal to $f(y)$ for all $y \\in U \\setminus S$. Thus every point in $S$ is a local minimum of $f$. Since every local minimum of $f$ is a global minimum, $\\inf_S f = \\inf_{\\mathbb{R}^n} f$ and $S$ is a setwise global minimum according to \\Cref{def:setwise-global-min}. The rest of the proof deals with the case when $S^\\circ \\neq \\emptyset$. Let $C$ be the set of all critical points of $f$ in $S$ and consider the following optimization problem:\n","start":29561,"end":30739}]},{"id":"S067","semantic_type":"assumption","kind":"assumption","labels":["app:prop:exist","eq:1/2-holder"],"member_ids":["S067"],"declared_external":false,"source_only":false,"title":"存在性构造的下界假设","source_units":[{"raw_tex":"A solution exists because $f$ is bounded below and the objective function is coercive. Any solution satisfies \n","start":76678,"end":76789},{"raw_tex":"For any $l\\in\\mathbb{N}$, we have \n\\begin{equation*}\n    \\sum_{k=0}^l\\frac{\\lVert x_{k+1}^\\tau-x_k^\\tau\\rVert^2}{2\\tau} \\leqslant f(x_0^\\tau) - f(x_{l+1}^\\tau) \\leqslant f(x_0) - \\inf_{\\mathbb{R}^n} f =: C < \\infty\n\\end{equation*}\nsince $f$ is bounded below. Observe that \n","start":78128,"end":78401}]},{"id":"S068","semantic_type":"assumption","kind":"assumption","labels":["app:prop:exist","eq:1/2-holder"],"member_ids":["S068"],"declared_external":false,"source_only":false,"title":"存在性构造的正则性假设","source_units":[{"raw_tex":"Since $f$ is locally Lipschitz, the set-valued function $-\\partial f$ is upper semicontinuous \\cite[2.1.5 Proposition (d) p. 29]{clarke1990} with nonempty compact values \\cite[2.1.2 Proposition (a) p. 27]{clarke1990}, hence proper upper hemicontinuous \\cite[Proposition 1 p. 60]{aubin1984differential}. In addition, $x^{\\tau_k}\\to x^*$ uniformly on $[0,T]$ and $(\\tilde{x}^{\\tau_k})'\\to (x^*)'$ weakly in $L^1([0,T],\\mathbb{R}^n)$. Therefore,  $(x^*)'(t)\\in-\\partial f(x^*(t))$ for almost all $t\\in[0,T]$ by \\cite[Theorem 1 p. 60]{aubin1984differential}\\footnote{In the theorem we take $F:=-\\partial f$, $X=Y:=\\mathbb{R}^n$, and $I:=[0,T]$.}. The initial condition also holds since $\\tilde{x}^{\\tau}(0)=x_0$ for all $\\tau>0$. \n","start":81472,"end":82199}]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S069"],"declared_external":true,"declaration_text":["原先决条件：Standard finite-dimensional topology and compactness，含 connectedness of continuous images of intervals 与 Bolzano–Weierstrass；紧集上连续函数有界为其中的标准背景。"],"source_only":false,"title":"有限维连通性与紧性","source_units":[]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S070"],"declared_external":true,"declaration_text":["原先决条件：attainment of minima for continuous coercive functions。"],"source_only":false,"title":"连续强制目标取得极小","source_units":[]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S071"],"declared_external":true,"declaration_text":["原先决条件：Clarke Fermat necessary condition and addition of a smooth quadratic term, used implicitly for proximal minimizers。"],"source_only":false,"title":"Clarke Fermat 与光滑求和","source_units":[]},{"id":"S072","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S072"],"declared_external":true,"declaration_text":["原先决条件及作者 L326、L328 引用：clarke1990, Proposition 2.1.5(b), p.29。"],"source_only":false,"title":"Clarke 次微分闭图","source_units":[]},{"id":"S073","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S073"],"declared_external":true,"declaration_text":["原先决条件及作者 L752 引用：clarke1990, Proposition 2.1.5(d), p.29。"],"source_only":false,"title":"Clarke 次微分上半连续","source_units":[]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S074"],"declared_external":true,"declaration_text":["原先决条件及作者 L739 引用：rudin1964principles, Theorem 7.25。"],"source_only":false,"title":"Arzelà–Ascoli 紧性","source_units":[]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S075"],"declared_external":true,"declaration_text":["原先决条件及作者 L740 引用：Fitzpatrick2010, Corollary 14, p.413。"],"source_only":false,"title":"导数序列的弱紧性","source_units":[]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S076"],"declared_external":true,"declaration_text":["原先决条件及作者 L752 引用：aubin1984differential, Proposition 1, p.60。"],"source_only":false,"title":"上半连续到上半半连续","source_units":[]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S077"],"declared_external":true,"declaration_text":["原先决条件及作者 L752 引用与脚注：aubin1984differential, Theorem 1, p.60。"],"source_only":false,"title":"微分包含的极限稳定性","source_units":[]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S078"],"declared_external":true,"declaration_text":["原先决条件：Cauchy–Schwarz inequality；作者在 L739 明说使用。"],"source_only":false,"title":"Cauchy–Schwarz 不等式","source_units":[]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S079"],"declared_external":true,"declaration_text":["原先决条件：Subgradient trajectory descent and energy identity；davis2020stochastic, Lemma 5.2，作者在 L301、L328 引用。"],"source_only":false,"title":"次梯度轨迹能量恒等式","source_units":[]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S080"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator：弱 L¹ 收敛的 L∞ 对偶测试；用于显式标注 L748 省略的区间指示函数。"],"source_only":false,"title":"弱 L¹ 收敛的积分测试","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076","S077":"S077","S078":"S078","S079":"S079","S080":"S080"}}}}
{"record_id":"2302.04077v3","paper":{"title":"MGProx: A nonsmooth multigrid proximal gradient method with adaptive restriction for strongly convex optimization","authors":["Ang, Andersen","De Sterck, Hans","Vavasis, Stephen"],"arxiv_id":"2302.04077","version":"v3","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/2302.04077v3","pdf_url":"https://arxiv.org/pdf/2302.04077v3","doi":"10.1137/23M1552140"},"context":{"raw_tex":"We study the combination of two iterative algorithms: proximal gradient descent and multigrid, to solve the following class of optimization problems\n\\begin{equation}\\label{prob:minfg}\n\\argmin_x ~ F_0(x) \\coloneqq f_0(x) + g_0(x).\n\\end{equation}\n\\textbf{(Assumption)} We assume $f_0 : \\IRn \\rightarrow \\IR$ is \n% lower semi-continuous, % NO need because smoothness implies LSC\n$L_0$-smooth and $\\mu_0$-strongly convex,\nand  \n%$g_0 : \\IRn \\rightarrow \\overline{\\IR} \\coloneqq \\IR \\cup \\{+\\infty\\}$ \n$g_0 : \\IRn \\rightarrow \\IR$ \nis proper, possibly nonsmooth, convex, \n%lower semi-continuous, \nand separable.\nRegarding the non-smoothness (nondifferentiability) of $g_0$, we further assume a single point of non-differentiability for $g_0$.\nWe recall that a function $\\zeta(x) : \\IRn \\rightarrow \\IR$ is separable if $\\zeta(x) = \\sum_i \\zeta_i(x_i)$, and  is $\\mu$-strongly convex (and thus coercive) with $\\mu>0$ if $\\zeta(x) - \\frac{\\mu}{2}\\| x \\|_2^2$ is convex;\nand lastly is $L$-smooth if  $\\zeta$ is $C^{1,1}_L$ and $\\nabla \\zeta$ is $L$-Lipschitz; i.e., for all $x,y$ in $\\IRn$, $\\nabla \\zeta(x)$ exists and\n\\begin{equation}\n\\zeta(y) \\leq \\zeta(x) + \\big\\langle \\nabla \\zeta(x), y-x \\big\\rangle + \\frac{L}{2}\\|y-x\\|_2^2.\n\\label{def:Lsmooth}\n\\end{equation}\n\n\n\\paragraph{Notation}\nThe symbol $x_0$ (or $x$) is called the fine variable.\nThe symbol $x_\\ell$ in \\eqref{prob:mgopt_coarse} with $\\ell \\geq 1$ is called coarse variable.\nThe subscript $\\ell \\in \\{0,1,\\dots, L \\}$ denotes the level.\nA larger $\\ell$ means a coarser level with lower resolution (fewer variables).\n%We use $\\ell=0$ to denote the finest level with the highest resolution and $\\ell = L$ to denote the coarsest level with the lowest resolution.\nFor the remainder of the paper, $L$ without a subscript stands for the number of levels, whereas $L_{\\ell}$ denotes the smoothness parameter for the level-$\\ell$ problem.\n%\nAt a level $\\ell$, the coarse version of the vector $x_\\ell \\in \\IR^{n_\\ell}$ is $x_{\\ell+1} = \\overline{\\cR}(x_\\ell) \\coloneqq \\overline{R}x_\\ell$ where $\\overline{R} \\in \\IR^{n_{\\ell+1} \\times n_{\\ell}}$ with $n_{\\ell+1} \\leq n_{\\ell}$ is called a restriction matrix.\nSimilarly, given $x_{\\ell+1} \\in \\IR^{n_{\\ell+1}}$ and a prolongation matrix $\\overline{P} \\in \\IR^{ n_{\\ell} \\times n_{\\ell +1}}$, \nwe obtain the level-$\\ell$ version of $x_{\\ell+1}$ as $x_{\\ell} = \\overline{\\cP}(x_{\\ell+1}) \\coloneqq \\overline{P} x_{\\ell+1}$.\nWe let $\\overline{P} = c \\overline{R}^\\top$ given scaling factor $c>0$ and $\\overline{R}$.\nIn multigrid, choosing $(\\overline{R},\\overline{P})$ depends on the application.\nIn this work, for the applications we consider commonly chosen $(\\overline{R},\\overline{P})$, see Section\\,\\ref{sec:exp:subsec:mgprox_eop} for our choice of \n$(\\overline{R},\\overline{P})$ used in the experiment and see \\cite{briggs2000multigrid} for an introduction on how to select $(\\overline{R},\\overline{P})$.\n\n\nThe proximal gradient method solves problems of the form \\eqref{prob:minfg} as follows.\nStarting from an initial guess $x^1$, the method updates the variable by a gradient descent step (with a stepsize $\\alpha > 0$) followed by a proximal step associated with $g_0$:\n\\begin{subequations}\\label{eq:5}\n\\begin{align}\nx^{k+1} &\\,= \\prox_{\\alpha g_0} \n\\Big( \nx^k - \\alpha \\nabla f_0(x^k)\n\\Big), \n%~~ \\alpha > 0\n\\label{updt:proxgrad}\n\\\\\n\\prox_{\\alpha g_0}(x) \n& \\coloneqq \n\\underset{u}{\\argmin} ~ \\Big\\{\\, \\alpha g_0(u) + \\frac{1}{2} \\| u - x \\|_2^2 \\,\\Big\\}\n= \\underset{u}{\\argmin} ~ \\Big\\{\\, g_0(u) + \\frac{1}{2\\alpha} \\| u - x \\|_2^2 \\,\\Big\\}.\n\\label{def:prox_opertor}\n\\end{align}\n\\end{subequations}\n\n\n\\subsection{Functions at different levels}\\label{sec:mgprox:subsec:RP}\nFollowing \\cref{sec:intro:subsec:mgopt}, \nwe use $f_\\ell : \\IR^{n_\\ell} \\rightarrow \\IR$ to denote functions at different coarse levels.\nIn this section we will focus on $\\ell \\in \\{0,1\\}$ but we remark that all the notations and definitions are generalized to $\\ell \\in \\{0,1,\\dots,L\\}$ in Section~\\ref{sec:multilevel}.\nWe denote the restriction of the fine objective function $F_0$ in \\eqref{prob:minfg} as\n$ F_1 \\coloneqq \\cR(F_0)  = \\cR(f_0 + g_0)$, where $\\cR$ is defined below.\n\n\\begin{definition}[Restriction]\\label{def:restriction}\nAt a level $\\ell \\in \\IN$, given a function $f_\\ell : \\IR^{n_\\ell} \\rightarrow \\IR$, the restriction $\\cR$ of $f_\\ell$, denoted as $f_{\\ell+1} \\coloneqq \\cR(f_\\ell)$, is defined as \n$f_{\\ell+1}(x_{\\ell+1}) \\coloneqq f_{\\ell}(R x_{\\ell})$, \nwhere $R : \\IR^{n_\\ell} \\rightarrow \\IR^{n_{\\ell+1}}$ is a restriction matrix .\nWe also define the associated prolongation matrix \n$P : \\IR^{n_{\\ell+1}} \\rightarrow \\IR^{n_\\ell}$ \n as $P = cR^\\top$ where $c>0$ is a predefined constant.\n\\end{definition}\n\n\\paragraph{Adaptive restriction and non-adaptive restriction}\nWe recall that a contribution of this work is the introduction of the adaptive restriction, to be discussed in \\cref{sec:mgprox:subsec:adaptiveR} (see Definition\\,\\ref{eg:adaptiveR_L1}).\nTo differentiate the classical non-adaptive restriction (and the associated prolongation) from the adaptive version, we denote the non-adaptive restriction by $\\overline{\\cR}, \\overline{R},\\overline{\\cP},\\overline{P}$, and denote the adaptive one by $\\cR, R,\\cP,P$.\nWe remark that Definition~\\ref{def:restriction} can be used for both versions of restriction and prolongation.\nWe give an example of $\\overline{R},\\overline{P}$ in \\cref{sec:exp}.\n\n\n\\subsection{Review of subdifferential of nonsmooth functions}\\label{sec:mgprox:subsec:subdifferential}\nThe subdifferential \\cite{rockafellar1970convex}\nis a standard framework used in convex analysis to deal with nondifferentiable functions.\nA convex function $g(x) : \\IRn \\rightarrow \\overline{\\IR} \\coloneqq \\IR \\cup \\{+\\infty\\}$ is called nonsmooth if it is not differentiable for some $x$ in $\\IRn$.\nA point $q \\in \\IRn$ is called a subgradient of $g$ at  $x$ if for all $y \\in \\IRn$ the inequality $g(y) \\geq g(x) + \\langle q, y -x \\rangle$ holds.\nThe subdifferential of $g$ at a point $x$ is defined as the set of all subgradients of $g$ at $x$, i.e.,\n\\begin{equation}\n  \\partial g(x)\n\\coloneqq \n\\Big\\{~\nq \\in \\IRn ~~\\big|~~ \ng(y)  \\geq g(x) + \\langle q, y -x \\rangle\n~\\forall y \\in \\IRn\n~\\Big\\} ~ \\subset \\IRn,\n\\label{def:subdifferential_II}\n\\end{equation}\nso $\\partial g(x)$ is generally set-valued.\nIf $g$ is differentiable at $x$, then $\\nabla g(x)$ exists and the set $\\partial g(x)$ reduces to the singleton $\\big\\{\\nabla g(x)\\big\\}$.\n\n\n\\paragraph{Subdifferential sum rule (Moreau–Rockafellar theorem)}~~\nLet $\\oplus$ denote the Minkowski sum.\nSince subdifferentials are generally set-valued, hence for two functions $f, g$, generally $\\partial (f+g) \\neq \\partial f \\oplus \\partial g$ but $\\partial (f+g) \\supset \\partial f \\oplus \\partial g$.\nThe sum rule $\\partial (f+g) = \\partial f \\oplus \\partial g$ holds if $f,g$ satisfy a qualification condition (e.g. \\cite[Theorem 3.36]{beck2017first}): the relative interior of the domain of $f$ has a non-empty intersection with the relative interior of domain of $g$,\ni.e.,\n\\begin{equation}\n\\ri (\\dom f) \\,\\cap\\, \\ri (\\dom g) \\neq \\varnothing\n%\n\\implies\n%\n\\partial \\Big( f(x) + g(x) \\Big) =\n\\partial f(x) \\oplus \\partial g(x),\n~~~\n\\forall x \\in \\textrm{dom} f \\,\\cap\\, \\textrm{dom} g.\n\\label{fact:subgrad_sum_rule}\n\\end{equation}\nThe right-hand side (RHS) of \\eqref{fact:subgrad_sum_rule}, which is the subdifferential sum rule,\nis known as the Moreau–Rockafellar theorem \\cite{lai1988moreau}.\nWe now discuss the fact that the functions $f_\\ell, g_\\ell$ for all levels $\\ell$ in this work satisfy the Moreau–Rockafellar theorem. \n\\begin{itemize}\n\\item At level $\\ell=0$, we have the left-hand side (LHS) of \\eqref{fact:subgrad_sum_rule} for $f_0, g_0$ \n%as $\\dom f_0 \n%= \\dom g_0 \n%= \\IRn$\nby assumption.\n\\item At levels $\\ell>0$, by Definition~\\ref{def:restriction}, \nthe LHS of \\eqref{fact:subgrad_sum_rule} holds for the coarse functions.\nTo see this, take the coarse function as composition of the fine function with a linear map, and recognize the fact that the domain of a function is preserved under composition with a linear map.\nTo be explicit, we have\n\\[\n\\begin{array}{rcl}\n\\ri \\Big( \\dom f_1 \\Big)\n\\cap \n\\ri \\Big( \\dom g_1 \\Big)\n~=~\n\\ri \\Big( \\underbrace{\\dom (f_0 \\circ R)}_{\\text{whole }\\IR^{n_1}} \\Big)\n\\cap \n\\ri \\Big( \\dom (g_0 \\circ R)  \\Big)\n~\\neq~ \\varnothing.\n\\end{array}\n\\]\n\\end{itemize}\nTo sum up, in this work \\eqref{fact:subgrad_sum_rule} holds for all levels $\\ell$:\n\\begin{equation}\n\\partial F_\\ell(x_\\ell) \n\\coloneqq  \\partial \\Big( f_\\ell(x_\\ell) + g_\\ell(x_\\ell) \\Big)\n= \\partial f_\\ell(x_\\ell) \\oplus \\partial g_\\ell(x_\\ell)\n= \\nabla f_\\ell(x_\\ell) + \\partial g_\\ell (x_\\ell)\n,\n\\label{ass:F_sum_rule}\n\\end{equation}\nwhere $+$ is used instead of $\\oplus$ in $\\nabla f_\\ell(x_\\ell) + \\partial g_\\ell (x_\\ell)$ because $\\nabla f_\\ell(x_\\ell)$ is a singleton.\n\n\n\\subsection{Convexity and subdifferential of coarse function}\nFrom Definition~\\ref{def:restriction}, the coarse function can be written as $F_{\\ell+1} \\coloneqq F_\\ell \\circ R$, meaning that we can see the restriction process as the fine function taking composition with the linear map $R$.\nSuch composition view point gives us a series of useful properties for this work.\nFirst, we have a closed-form expression for the subdifferential of the coarse function in terms of the the subdifferential of the fine function.\nI.e., by \\cite[Theorem 3.43]{beck2017first}, we have that\n\\begin{equation}\\label{subdiff_coarse_linearmap}\n\\partial F_{\\ell+1}(x) = \\partial (F_\\ell \\circ R)(x) = R^\\top \\partial F_\\ell (Rx). \n\\end{equation}\nThen, by the fact that convexity is preserved under linear map, we have that ``restriction preserves convexity''.\nIn other words, $f_{\\ell+1}$ is convex if $f_{\\ell}$ is convex.\nFurthermore, if $R$ is full-rank (which is the case in this paper), restriction is submultiplicative on the modulus of convexity, as illustrated in the following lemma.\n\n\n\\paragraph{Notation for sets}\nFrom now on, when we encounter an expression containing both set-valued vector(s) and singleton vector(s), we underline the set-valued term(s) for visual clarity.\n\n\n\\subsection{Adaptive restriction and the \\texorpdfstring{$\\tau$}{} vector}\\label{sec:mgprox:subsec:adaptiveR}\nSince subdifferentials are set-valued, we define $\\tau$ in \\MGProx as an element of a set.\nAt a level $\\ell$, we define a set $\\underline{\\tau_{\\ell \\rightarrow \\ell+1}} ~\\coloneqq~\n\\underline{\\partial F_{\\ell+1}(x_{\\ell+1})} \\oplus (- R) \\underline{\\partial F_\\ell (x_\\ell)} $, where $\\ell \\rightarrow \\ell+1$ specifies that $\\tau$ connects level $\\ell$ to level $\\ell+1$, and the matrix $R$ here is an adaptive restriction operator that we will define soon.\nIn \\MGProx we choose an element of $\\underline{\\tau_{\\ell \\rightarrow \\ell+1}}$ as the tau vector.\nThat is, at level $\\ell=0$,\n\\begin{subequations}\n\\begin{align}\n\\tau_{0 \\rightarrow 1}\n~\\, \\in \\,~ \n\\underline{\\tau_{0 \\rightarrow 1}}\n& \n~\\coloneqq~\n\\underline{\\partial F_1(x_1)} \\oplus (- R) \\underline{\\partial F_0(x_0)}\n\\label{def:tau_mgprox1}\n\\\\\n&\\overset{\\eqref{ass:F_sum_rule}}{=} \\nabla f_1(x_1) - R \\nabla f_0(x_0) + \\underline{\\partial g_1(x_1)}\n\\oplus \n(- R) \\underline{\\partial g_0(x_0)}.\n\\label{def:tau_mgprox2}\n\\end{align}\n\\end{subequations}\nNote that $\\underline{\\tau_{0 \\rightarrow 1}}$ is a function of two points at two different levels.\nIn \\eqref{def:tau_mgprox1} $\\underline{\\tau_{0 \\rightarrow 1}}$ is the Minkowski sum of two subdifferentials $\\partial F_1(x_1) $ and $-R\\partial F_0(x_0)$ which are generally set-valued.\nTo obtain a tractable coarse-grid optimization problem (corresponding to line (iv) in Algorithm~\\ref{algo:mgopt}) we need to avoid complications coming from the Minkowski sum, and we do this by modifying the standard restriction (and prolongation) \nby zeroing out columns in $\\overline{R}$ to form $R$ such that the second subdifferential $R \\partial g_0(x_0)$ in \\eqref{def:tau_mgprox2}\nis a singleton vector.\nSimilarly, we  zero out the corresponding rows in $\\overline{P}$ to form $P$ for the coarse correction step, such that non-differentiable fine points are not corrected by the coarse grid.\nThis zeroing out process is adapted to the current point $x_0$, so we call this $R$ an adaptive restriction operator. \nIn other words, the purpose of the adaptive restriction is to reduce a generally set-valued subdifferential $R\\partial g_0(x_0)$ to a singleton.\nWe denote the adaptive operator $R$ corresponding to a point $x$ as $R(x)$ and thereby the adaptive restriction of $x$ is denoted as $R(x)x$.\nSometimes we just write $Rx$ if the meaning is clear from the context.\nBased on the above discussion, we now formally define adaptive restriction,\nand we give an example in \\cref{sec:exp}.\n\n\n\\begin{definition}[Adaptive restriction operator for separable $g$]\\label{eg:adaptiveR_L1}\nFor a possibly nonsmooth function $g : \\IRn \\rightarrow \\IR$ that is separable, i.e., with $x = [x_1,x_2,\\dots,x_n]$, $g(x) = \\sum g_i(x_i)$ where $g_i$ is a function only of component $x_i$, \ngiven a full restriction operator $\\overline{R}$ and a vector $x$, the adaptive restriction operator $R$ with respect to a function $g$ at $x$ is defined by zeroing out the columns of $\\overline{R}$ corresponding to the elements in $\\partial g$ that are set-valued.\n\\label{def:adaptiveR}\n\\end{definition}\n\n\n\\subsection{A 2-level \\texorpdfstring{\\MGProx}{}algorithm}\\label{sec:mgprox:subsec:2lv}\nSimilar to the 2-level \\MGOPT method for solving Problem \\eqref{prob:mgopt}, we propose a 2-level \\MGProx method (Algorithm \\ref{aglo:2levelMG_exact}) that solves Problem \\eqref{prob:minfg} by utilizing a coarse problem defined as \n\\begin{equation}\\label{prob:minfg_coarse}\n\\argmin_{\\xi \\in \\IR^{n_1}}\\,\n\\bigg\\{\\,\nF^{\\tau}_1 (\\xi) \n~\\coloneqq~  F_1(\\xi) - \\langle \\tau_{0 \\rightarrow 1}^{k+1}, \\xi \\rangle\n~=~ f_1 (\\xi) + g_1(\\xi) - \\langle \\tau_{0 \\rightarrow 1}^{k+1}, \\xi \\rangle\n\\,\\bigg\\}.\n\\end{equation}\n\n\n\\begin{algorithm}\n\\caption{2-level \\MGProx for an approximate solution of \\eqref{prob:minfg}}\n\\label{aglo:2levelMG_exact}\n\\begin{algorithmic}\n\\STATE{Initialize $x^1_0$, $R$ and $P$}\n\\FOR{$k = 1,2,\\dots$}\n%\n\\STATE{(i)~~~~ $y^{k+1}_0 = \\prox_{\\frac{1}{L_0}g_0} \\Big (x^k_0 - \\frac{1}{L_0} \\nabla f(x^k_0) \\Big)$\n\\hfill level-0 proximal gradient step \\vspace{0.25mm}\n}\n\\STATE{(ii)~~~ $y^{k+1}_1  = R(y^{k+1}_0) y^{k+1}_0 $   \n\\hfill construct the level-1 coarse variable \\vspace{0.25mm}\n}\n\\STATE{(iii)~~ $\\tau_{0 \\rightarrow 1}^{k+1} \\hspace{-1mm} \\in\\,\n\\underline{\\partial F_1(y^{k+1}_1 )}  - R(y^{k+1}_0) \\, \\underline{\\partial  F_0(y^{k+1}_0)}$\n\\hfill construct the tau vector \\vspace{0.25mm}\n}\n\\STATE{(iv)~~ $\nx^{k+1}_1\n= \\underset{\\xi}{\\argmin} \\, \\Big\\{ F_1^{\\tau}(\\xi) \\coloneqq F_1(\\xi) - \\langle \\tau_{0 \\rightarrow 1}^{k+1}, \\xi \\rangle \\Big\\}\n$\n\\hfill solve the level-1 coarse problem\n}\n\\STATE{(v)\\,~~~ $z^{k+1}_0 = y^{k+1}_0  + \\alpha P\\big(  x^{k+1}_1 - y^{k+1}_1 \\big)$\n\\hfill coarse correction \\vspace{0.25mm}\n}\n\\STATE{(vi)~~\\, $x^{k+1}_0 = \\prox_{\\frac{1}{L_0}g_0} \\Big (z^{k+1}_0 - \\frac{1}{L_0} \\nabla f(z^{k+1}_0) \\Big)$\n\\hfill level-0 proximal gradient step~~\n}\n%\n\\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}\n\nHere are remarks for the steps in Algorithm~\\ref{aglo:2levelMG_exact}.\n\\begin{itemize}\n    \\item (i): we perform one or more proximal gradient iterations on the fine variable with a constant stepsize $1/L_0$, where $L_0$ is the Lipschitz constant of $\\nabla f_0$.\n    \\item (iii): we pick a value within the set to define $\\tau$;\n    as we are now using adaptive $R$, we use $+$ instead of $\\oplus$ in the expression of $\\tau$.\n    \\item (v): $\\alpha > 0$ is a stepsize; for its selection see \\cref{sec:mgprox:subsec:alpha}.\n    \\item The restriction for variable $y_0^{k+1}$ and the restriction for the subdifferential $ \\underline{\\partial  F_0(y^{k+1}_0)}$ can be slightly different.\n    On $y_0^{k+1}$ the restriction is the full restriction, on $ \\underline{\\partial  F_0(y^{k+1}_0)}$ is the adaptive one. \n    The explanation is as follows.   \n    For the particular cases of $g_0$ such as $\\ell_1$, $\\max\\{ \\cdot, 0\\}$ (element-wise maximum) and $\\iota_{[0,\\infty)}$ (indicator of nonnegative orthant), when we zero out column $i$ of $R$, the corresponding entry of $y_0^{k+1}$ is already $0$.    \n    Note that this conclusion does not hold in general for other nonsmooth functions or when the non-differentiability occur at another point (say at $x=1$). \n    Generally in those cases we will need to specify which restriction matrix (the full $R$ or the adaptive $R$) to be used to define $P$.\n        However, to simplify the presentation we always assume the non-differentiability occurs at $x=0$,\n         by performing a translation to shift the non-differentiability to occur at $x=0$.\n%        In fact, this is what exact we do in the experiment.\n\\end{itemize}\n\n\nBefore we prove the theorem we emphasize that \\eqref{iq:descent_dir} applies for any subgradient in the set $\\underline{\\partial F_0(y^{k+1}_0)} $.\nFurthermore,\n\\[\n\\eqref{iq:descent_dir} \n%\n\\iff\n%\n\\big\\langle \nP^\\top \\underline{\\partial F_0(y^{k+1}_0)} , x^{k+1}_1 - y^{k+1}_1 \n\\big\\rangle\n< 0\n%\n\\overset{P^\\top = cR}{\\iff}\n%\nc \\big\\langle \nR \\underline{\\partial F_0(y^{k+1}_0)} , x^{k+1}_1 - y^{k+1}_1 \n\\big\\rangle\n< 0\n.\n\\]\nAs $c>0$, showing \\eqref{iq:descent_dir} is equivalent to showing\n\\begin{equation}\n\\big\\langle R \\underline{\\partial F_0(y^{k+1}_0)},  ~ x^{k+1}_1 - y^{k+1}_1 \\big\\rangle < 0,\n\\label{pf:descent_to_prove}\n\\end{equation}\nwhere $R \\underline{\\partial F_0(y^{k+1}_0)} $ is a singleton vector for all subgradients in $\\underline{\\partial F_0(y^{k+1}_0)}$ due to the adaptive $R$.\n\n\n\\subsubsection{Existence of coarse correction stepsize \\texorpdfstring{$\\alpha_k$}{}}\n\\label{sec:mgprox:subsec:alpha}\nBased on Theorem~\\ref{thm:coarse_dir_descent}, we now show that there exists a stepsize $\\alpha_k > 0$ such that \n\\begin{equation}\nF_0(z^{k+1}_0) \n~\\coloneqq~\nF_0\\big(\ny^{k+1}_0 + \\alpha_k P(x^{k+1}_1 - y^{k+1}_1) \n\\big)\n~<~\nF_0(y^{k+1}_0).\n\\label{iq:coarse_descent_on_F}\n\\end{equation}\n\n\nTo prove the lemma, we make use \nof a fact of the subdifferentials of finite convex functions \\cite[Def.1.1.4, p.165]{hiriart2004fundamentals}: $\\partial g(x)$ is a nonempty compact convex set $\\cS \\subset \\IRn$ whose support function $\\displaystyle \\sup_{s}\\{ \\langle s, x\\rangle \\,|\\, s \\in \\cS  \\}$ \nis the directional derivative of $g$ at $x$.\nBy this, the subdifferential\n $\\underline{\\partial F_0(y_0^{k+1})}$ is a compact convex set whose support function is the directional derivative of $F_0$ at $y_0^{k+1}$.\nWe emphasize that here \n%$F_0 : \\IR^{n_0} \\rightarrow  \\overline{\\IR}$ at $y_0^{k+1}$ is \n$F_0(y_0^{k+1})$ is \nfinite, \n% i.e., $F_0 : \\IR^{n_0} \\rightarrow  \\overline{\\IR}$ will never reach $+\\infty$ at $y_0^{k+1}$ since $y_0^{k+1}$ is obtained by the proximal gradient step (in other words, $y_0^{k+1}$ is a ``feasible point'' if $g_0$ is an indicator function of a constraint set),\nso we can make use of the result on directional derivative in \\cite[Def. 1.1.4, p.165]{hiriart2004fundamentals}, which only applies for finite convex functions, associated with the subdifferential.\n\n\n\\begin{lemma}\n(Sufficient descent property of proximal gradient \\cite[Lemma 10.4]{beck2017first})\n\\label{lemma:suff_descent_proxgrad} \nLet $L_0$ be the Lipschitz constant of $\\nabla f_0$.\nFor step (i) in Algorithm~\\ref{aglo:2levelMG_exact},  we have\n\\begin{equation}\nF_0( y^{k+1}_0)\n\\leq \nF_0( x^k_0) - \\| G_0(x_0^k) \\|_2^2 / (2L_0),\n~~\nG_0( x_0^k) \n=\nL_0 \\big[\nx_0^k - \\prox_{ \\frac{1}{L} g_0 }\\big( x_0^k - \\nabla f_0(x_0^k)/L_0\n\\big)\n\\big].\n\\label{eqn:suffdescentPGD}\n\\end{equation}\n\\end{lemma}\n$G_0( x_0^k)$ is called the proximal gradient map of $F_0$ at $x_0^k$.\nThe inequality also holds for step (vi).\n\n\n\\begin{algorithm}\n\\caption{Naive line search}\\label{aglo:naivelinesearch}\n\\begin{algorithmic}\n\\STATE{Set $\\alpha > 0$ and select a tolerance $\\epsilon > 0$  (e.g. $10^{-15}$)}\n\\WHILE{true}\n\\STATE{\\textbf{If} $F_0\\big( y^{k+1}_0 + \\alpha P (x^{k+1}_1 - y^{k+1}_1) \\big) \\leq F_0( y^{k+1}_0 )$ \\textbf{ then }\nreturn $z_0^{k+1}=y^{k+1}_0 + \\alpha P(x^{k+1}_1 - y^{k+1}_1)$.}\n\\\\\n\\STATE{\\textbf{else if} $\\alpha > \\epsilon $ \\textbf{then} $\\alpha = \\alpha/2 $.}\n\\\\\n\\STATE{~~~~~~~\\textbf{else} return $z_0^{k+1}=y^{k+1}_0$.}\n\\ENDWHILE\n\\end{algorithmic}\n\\end{algorithm}\n\n\n\\subsubsection{Linear convergence rate}\nAll the functions and variables here are at level 0 so we omit the subscripts.\nNow we show that $\\big\\{F( x^k) \\big\\}_{k \\in \\IN}$ converges to $F^*$ with a linear rate using the \\emph{Proximal Polyak-{\\L}ojasiewicz inequality} \\cite[Section 4]{karimi2016linear}. \nThe function $F$ in Problem \\eqref{prob:minfg} is called ProxPL, if there exists $\\mu > 0$, called the ProxPL constant, such that \n\\begin{equation}\n\\cD_g(x,L)\n~\\geq~\n2 \\mu \\big( F(x) - F^* \\big)  \\qquad  \\forall x,\n\\tag{ProxPL}\n\\label{def:ProxPL}\n\\end{equation}\n\\begin{equation}\n\\cD_g(x,\\alpha) \n\\coloneqq\n-2\\alpha \\min_z \\,\n\\Big\\{\n{\\textstyle\\frac{\\alpha}{2}} \\| z - x \\|_2^2 \n+ \\big\\langle z - x, \\nabla f(x) \\big\\rangle\n+ g(z) - g(x)\n\\Big\\}\n.\n\\label{def:dg}\n\\end{equation}\nIntuitively, $\\cD_g$ is defined based on the proximal gradient operator:\n\\[\n\\prox_{\\frac{1}{L}g} \\Big( x - \\frac{\\nabla f(x)}{L} \\Big)\n=\n\\underset{z}{\\argmin} \\, \\frac{L}{2} \\| z -  x \\|_2^2\n+ \\big\\langle z - x, \\nabla f(x) \\big\\rangle\n+ g(z) - g(x).\n\\]\nIt has been shown in \\cite{karimi2016linear} that if $f$ in \\eqref{prob:minfg} is $\\mu$-strongly convex, then $F$ is $\\mu$-ProxPL.\nNow we prove the linear convergence rate of Algorithm~\\ref{aglo:2levelMG_exact}.\nNote that a standard result such as \\cite[Theorem 10.29]{beck2017first} on convergence of proximal gradient for strongly convex functions is not directly applicable because, as mentioned above, we interleave proximal gradient steps with coarse correction steps.\n","units":[{"raw_tex":"We study the combination of two iterative algorithms: proximal gradient descent and multigrid, to solve the following class of optimization problems\n\\begin{equation}\\label{prob:minfg}\n\\argmin_x ~ F_0(x) \\coloneqq f_0(x) + g_0(x).\n\\end{equation}\n\\textbf{(Assumption)} We assume $f_0 : \\IRn \\rightarrow \\IR$ is \n% lower semi-continuous, % NO need because smoothness implies LSC\n$L_0$-smooth and $\\mu_0$-strongly convex,\nand  \n%$g_0 : \\IRn \\rightarrow \\overline{\\IR} \\coloneqq \\IR \\cup \\{+\\infty\\}$ \n$g_0 : \\IRn \\rightarrow \\IR$ \nis proper, possibly nonsmooth, convex, \n%lower semi-continuous, \nand separable.\nRegarding the non-smoothness (nondifferentiability) of $g_0$, we further assume a single point of non-differentiability for $g_0$.\nWe recall that a function $\\zeta(x) : \\IRn \\rightarrow \\IR$ is separable if $\\zeta(x) = \\sum_i \\zeta_i(x_i)$, and  is $\\mu$-strongly convex (and thus coercive) with $\\mu>0$ if $\\zeta(x) - \\frac{\\mu}{2}\\| x \\|_2^2$ is convex;\nand lastly is $L$-smooth if  $\\zeta$ is $C^{1,1}_L$ and $\\nabla \\zeta$ is $L$-Lipschitz; i.e., for all $x,y$ in $\\IRn$, $\\nabla \\zeta(x)$ exists and\n\\begin{equation}\n\\zeta(y) \\leq \\zeta(x) + \\big\\langle \\nabla \\zeta(x), y-x \\big\\rangle + \\frac{L}{2}\\|y-x\\|_2^2.\n\\label{def:Lsmooth}\n\\end{equation}\n","start":6637,"end":7897},{"raw_tex":"\\paragraph{Notation}\nThe symbol $x_0$ (or $x$) is called the fine variable.\nThe symbol $x_\\ell$ in \\eqref{prob:mgopt_coarse} with $\\ell \\geq 1$ is called coarse variable.\nThe subscript $\\ell \\in \\{0,1,\\dots, L \\}$ denotes the level.\nA larger $\\ell$ means a coarser level with lower resolution (fewer variables).\n%We use $\\ell=0$ to denote the finest level with the highest resolution and $\\ell = L$ to denote the coarsest level with the lowest resolution.\nFor the remainder of the paper, $L$ without a subscript stands for the number of levels, whereas $L_{\\ell}$ denotes the smoothness parameter for the level-$\\ell$ problem.\n%\nAt a level $\\ell$, the coarse version of the vector $x_\\ell \\in \\IR^{n_\\ell}$ is $x_{\\ell+1} = \\overline{\\cR}(x_\\ell) \\coloneqq \\overline{R}x_\\ell$ where $\\overline{R} \\in \\IR^{n_{\\ell+1} \\times n_{\\ell}}$ with $n_{\\ell+1} \\leq n_{\\ell}$ is called a restriction matrix.\nSimilarly, given $x_{\\ell+1} \\in \\IR^{n_{\\ell+1}}$ and a prolongation matrix $\\overline{P} \\in \\IR^{ n_{\\ell} \\times n_{\\ell +1}}$, \nwe obtain the level-$\\ell$ version of $x_{\\ell+1}$ as $x_{\\ell} = \\overline{\\cP}(x_{\\ell+1}) \\coloneqq \\overline{P} x_{\\ell+1}$.\nWe let $\\overline{P} = c \\overline{R}^\\top$ given scaling factor $c>0$ and $\\overline{R}$.\nIn multigrid, choosing $(\\overline{R},\\overline{P})$ depends on the application.\nIn this work, for the applications we consider commonly chosen $(\\overline{R},\\overline{P})$, see Section\\,\\ref{sec:exp:subsec:mgprox_eop} for our choice of \n$(\\overline{R},\\overline{P})$ used in the experiment and see \\cite{briggs2000multigrid} for an introduction on how to select $(\\overline{R},\\overline{P})$.\n","start":11252,"end":12899},{"raw_tex":"The proximal gradient method solves problems of the form \\eqref{prob:minfg} as follows.\nStarting from an initial guess $x^1$, the method updates the variable by a gradient descent step (with a stepsize $\\alpha > 0$) followed by a proximal step associated with $g_0$:\n\\begin{subequations}\\label{eq:5}\n\\begin{align}\nx^{k+1} &\\,= \\prox_{\\alpha g_0} \n\\Big( \nx^k - \\alpha \\nabla f_0(x^k)\n\\Big), \n%~~ \\alpha > 0\n\\label{updt:proxgrad}\n\\\\\n\\prox_{\\alpha g_0}(x) \n& \\coloneqq \n\\underset{u}{\\argmin} ~ \\Big\\{\\, \\alpha g_0(u) + \\frac{1}{2} \\| u - x \\|_2^2 \\,\\Big\\}\n= \\underset{u}{\\argmin} ~ \\Big\\{\\, g_0(u) + \\frac{1}{2\\alpha} \\| u - x \\|_2^2 \\,\\Big\\}.\n\\label{def:prox_opertor}\n\\end{align}\n\\end{subequations}\n","start":18761,"end":19458},{"raw_tex":"\\subsection{Functions at different levels}\\label{sec:mgprox:subsec:RP}\nFollowing \\cref{sec:intro:subsec:mgopt}, \nwe use $f_\\ell : \\IR^{n_\\ell} \\rightarrow \\IR$ to denote functions at different coarse levels.\nIn this section we will focus on $\\ell \\in \\{0,1\\}$ but we remark that all the notations and definitions are generalized to $\\ell \\in \\{0,1,\\dots,L\\}$ in Section~\\ref{sec:multilevel}.\nWe denote the restriction of the fine objective function $F_0$ in \\eqref{prob:minfg} as\n$ F_1 \\coloneqq \\cR(F_0)  = \\cR(f_0 + g_0)$, where $\\cR$ is defined below.\n\n\\begin{definition}[Restriction]\\label{def:restriction}\nAt a level $\\ell \\in \\IN$, given a function $f_\\ell : \\IR^{n_\\ell} \\rightarrow \\IR$, the restriction $\\cR$ of $f_\\ell$, denoted as $f_{\\ell+1} \\coloneqq \\cR(f_\\ell)$, is defined as \n$f_{\\ell+1}(x_{\\ell+1}) \\coloneqq f_{\\ell}(R x_{\\ell})$, \nwhere $R : \\IR^{n_\\ell} \\rightarrow \\IR^{n_{\\ell+1}}$ is a restriction matrix .\nWe also define the associated prolongation matrix \n$P : \\IR^{n_{\\ell+1}} \\rightarrow \\IR^{n_\\ell}$ \n as $P = cR^\\top$ where $c>0$ is a predefined constant.\n\\end{definition}\n\n\\paragraph{Adaptive restriction and non-adaptive restriction}\nWe recall that a contribution of this work is the introduction of the adaptive restriction, to be discussed in \\cref{sec:mgprox:subsec:adaptiveR} (see Definition\\,\\ref{eg:adaptiveR_L1}).\nTo differentiate the classical non-adaptive restriction (and the associated prolongation) from the adaptive version, we denote the non-adaptive restriction by $\\overline{\\cR}, \\overline{R},\\overline{\\cP},\\overline{P}$, and denote the adaptive one by $\\cR, R,\\cP,P$.\nWe remark that Definition~\\ref{def:restriction} can be used for both versions of restriction and prolongation.\nWe give an example of $\\overline{R},\\overline{P}$ in \\cref{sec:exp}.\n","start":30220,"end":32020},{"raw_tex":"\\subsection{Review of subdifferential of nonsmooth functions}\\label{sec:mgprox:subsec:subdifferential}\nThe subdifferential \\cite{rockafellar1970convex}\nis a standard framework used in convex analysis to deal with nondifferentiable functions.\nA convex function $g(x) : \\IRn \\rightarrow \\overline{\\IR} \\coloneqq \\IR \\cup \\{+\\infty\\}$ is called nonsmooth if it is not differentiable for some $x$ in $\\IRn$.\nA point $q \\in \\IRn$ is called a subgradient of $g$ at  $x$ if for all $y \\in \\IRn$ the inequality $g(y) \\geq g(x) + \\langle q, y -x \\rangle$ holds.\nThe subdifferential of $g$ at a point $x$ is defined as the set of all subgradients of $g$ at $x$, i.e.,\n\\begin{equation}\n  \\partial g(x)\n\\coloneqq \n\\Big\\{~\nq \\in \\IRn ~~\\big|~~ \ng(y)  \\geq g(x) + \\langle q, y -x \\rangle\n~\\forall y \\in \\IRn\n~\\Big\\} ~ \\subset \\IRn,\n\\label{def:subdifferential_II}\n\\end{equation}\nso $\\partial g(x)$ is generally set-valued.\nIf $g$ is differentiable at $x$, then $\\nabla g(x)$ exists and the set $\\partial g(x)$ reduces to the singleton $\\big\\{\\nabla g(x)\\big\\}$.\n","start":32067,"end":33114},{"raw_tex":"\\paragraph{Subdifferential sum rule (Moreau–Rockafellar theorem)}~~\nLet $\\oplus$ denote the Minkowski sum.\nSince subdifferentials are generally set-valued, hence for two functions $f, g$, generally $\\partial (f+g) \\neq \\partial f \\oplus \\partial g$ but $\\partial (f+g) \\supset \\partial f \\oplus \\partial g$.\nThe sum rule $\\partial (f+g) = \\partial f \\oplus \\partial g$ holds if $f,g$ satisfy a qualification condition (e.g. \\cite[Theorem 3.36]{beck2017first}): the relative interior of the domain of $f$ has a non-empty intersection with the relative interior of domain of $g$,\ni.e.,\n\\begin{equation}\n\\ri (\\dom f) \\,\\cap\\, \\ri (\\dom g) \\neq \\varnothing\n%\n\\implies\n%\n\\partial \\Big( f(x) + g(x) \\Big) =\n\\partial f(x) \\oplus \\partial g(x),\n~~~\n\\forall x \\in \\textrm{dom} f \\,\\cap\\, \\textrm{dom} g.\n\\label{fact:subgrad_sum_rule}\n\\end{equation}\nThe right-hand side (RHS) of \\eqref{fact:subgrad_sum_rule}, which is the subdifferential sum rule,\nis known as the Moreau–Rockafellar theorem \\cite{lai1988moreau}.\nWe now discuss the fact that the functions $f_\\ell, g_\\ell$ for all levels $\\ell$ in this work satisfy the Moreau–Rockafellar theorem. \n\\begin{itemize}\n\\item At level $\\ell=0$, we have the left-hand side (LHS) of \\eqref{fact:subgrad_sum_rule} for $f_0, g_0$ \n%as $\\dom f_0 \n%= \\dom g_0 \n%= \\IRn$\nby assumption.\n\\item At levels $\\ell>0$, by Definition~\\ref{def:restriction}, \nthe LHS of \\eqref{fact:subgrad_sum_rule} holds for the coarse functions.\nTo see this, take the coarse function as composition of the fine function with a linear map, and recognize the fact that the domain of a function is preserved under composition with a linear map.\nTo be explicit, we have\n\\[\n\\begin{array}{rcl}\n\\ri \\Big( \\dom f_1 \\Big)\n\\cap \n\\ri \\Big( \\dom g_1 \\Big)\n~=~\n\\ri \\Big( \\underbrace{\\dom (f_0 \\circ R)}_{\\text{whole }\\IR^{n_1}} \\Big)\n\\cap \n\\ri \\Big( \\dom (g_0 \\circ R)  \\Big)\n~\\neq~ \\varnothing.\n\\end{array}\n\\]\n\\end{itemize}\nTo sum up, in this work \\eqref{fact:subgrad_sum_rule} holds for all levels $\\ell$:\n\\begin{equation}\n\\partial F_\\ell(x_\\ell) \n\\coloneqq  \\partial \\Big( f_\\ell(x_\\ell) + g_\\ell(x_\\ell) \\Big)\n= \\partial f_\\ell(x_\\ell) \\oplus \\partial g_\\ell(x_\\ell)\n= \\nabla f_\\ell(x_\\ell) + \\partial g_\\ell (x_\\ell)\n,\n\\label{ass:F_sum_rule}\n\\end{equation}\nwhere $+$ is used instead of $\\oplus$ in $\\nabla f_\\ell(x_\\ell) + \\partial g_\\ell (x_\\ell)$ because $\\nabla f_\\ell(x_\\ell)$ is a singleton.\n","start":33120,"end":35515},{"raw_tex":"\\subsection{Convexity and subdifferential of coarse function}\nFrom Definition~\\ref{def:restriction}, the coarse function can be written as $F_{\\ell+1} \\coloneqq F_\\ell \\circ R$, meaning that we can see the restriction process as the fine function taking composition with the linear map $R$.\nSuch composition view point gives us a series of useful properties for this work.\nFirst, we have a closed-form expression for the subdifferential of the coarse function in terms of the the subdifferential of the fine function.\nI.e., by \\cite[Theorem 3.43]{beck2017first}, we have that\n\\begin{equation}\\label{subdiff_coarse_linearmap}\n\\partial F_{\\ell+1}(x) = \\partial (F_\\ell \\circ R)(x) = R^\\top \\partial F_\\ell (Rx). \n\\end{equation}\nThen, by the fact that convexity is preserved under linear map, we have that ``restriction preserves convexity''.\nIn other words, $f_{\\ell+1}$ is convex if $f_{\\ell}$ is convex.\nFurthermore, if $R$ is full-rank (which is the case in this paper), restriction is submultiplicative on the modulus of convexity, as illustrated in the following lemma.\n","start":35573,"end":36646},{"raw_tex":"\\paragraph{Notation for sets}\nFrom now on, when we encounter an expression containing both set-valued vector(s) and singleton vector(s), we underline the set-valued term(s) for visual clarity.\n","start":38081,"end":38274},{"raw_tex":"\\subsection{Adaptive restriction and the \\texorpdfstring{$\\tau$}{} vector}\\label{sec:mgprox:subsec:adaptiveR}\nSince subdifferentials are set-valued, we define $\\tau$ in \\MGProx as an element of a set.\nAt a level $\\ell$, we define a set $\\underline{\\tau_{\\ell \\rightarrow \\ell+1}} ~\\coloneqq~\n\\underline{\\partial F_{\\ell+1}(x_{\\ell+1})} \\oplus (- R) \\underline{\\partial F_\\ell (x_\\ell)} $, where $\\ell \\rightarrow \\ell+1$ specifies that $\\tau$ connects level $\\ell$ to level $\\ell+1$, and the matrix $R$ here is an adaptive restriction operator that we will define soon.\nIn \\MGProx we choose an element of $\\underline{\\tau_{\\ell \\rightarrow \\ell+1}}$ as the tau vector.\nThat is, at level $\\ell=0$,\n\\begin{subequations}\n\\begin{align}\n\\tau_{0 \\rightarrow 1}\n~\\, \\in \\,~ \n\\underline{\\tau_{0 \\rightarrow 1}}\n& \n~\\coloneqq~\n\\underline{\\partial F_1(x_1)} \\oplus (- R) \\underline{\\partial F_0(x_0)}\n\\label{def:tau_mgprox1}\n\\\\\n&\\overset{\\eqref{ass:F_sum_rule}}{=} \\nabla f_1(x_1) - R \\nabla f_0(x_0) + \\underline{\\partial g_1(x_1)}\n\\oplus \n(- R) \\underline{\\partial g_0(x_0)}.\n\\label{def:tau_mgprox2}\n\\end{align}\n\\end{subequations}\nNote that $\\underline{\\tau_{0 \\rightarrow 1}}$ is a function of two points at two different levels.\nIn \\eqref{def:tau_mgprox1} $\\underline{\\tau_{0 \\rightarrow 1}}$ is the Minkowski sum of two subdifferentials $\\partial F_1(x_1) $ and $-R\\partial F_0(x_0)$ which are generally set-valued.\nTo obtain a tractable coarse-grid optimization problem (corresponding to line (iv) in Algorithm~\\ref{algo:mgopt}) we need to avoid complications coming from the Minkowski sum, and we do this by modifying the standard restriction (and prolongation) \nby zeroing out columns in $\\overline{R}$ to form $R$ such that the second subdifferential $R \\partial g_0(x_0)$ in \\eqref{def:tau_mgprox2}\nis a singleton vector.\nSimilarly, we  zero out the corresponding rows in $\\overline{P}$ to form $P$ for the coarse correction step, such that non-differentiable fine points are not corrected by the coarse grid.\nThis zeroing out process is adapted to the current point $x_0$, so we call this $R$ an adaptive restriction operator. \nIn other words, the purpose of the adaptive restriction is to reduce a generally set-valued subdifferential $R\\partial g_0(x_0)$ to a singleton.\nWe denote the adaptive operator $R$ corresponding to a point $x$ as $R(x)$ and thereby the adaptive restriction of $x$ is denoted as $R(x)x$.\nSometimes we just write $Rx$ if the meaning is clear from the context.\nBased on the above discussion, we now formally define adaptive restriction,\nand we give an example in \\cref{sec:exp}.\n\n\n\\begin{definition}[Adaptive restriction operator for separable $g$]\\label{eg:adaptiveR_L1}\nFor a possibly nonsmooth function $g : \\IRn \\rightarrow \\IR$ that is separable, i.e., with $x = [x_1,x_2,\\dots,x_n]$, $g(x) = \\sum g_i(x_i)$ where $g_i$ is a function only of component $x_i$, \ngiven a full restriction operator $\\overline{R}$ and a vector $x$, the adaptive restriction operator $R$ with respect to a function $g$ at $x$ is defined by zeroing out the columns of $\\overline{R}$ corresponding to the elements in $\\partial g$ that are set-valued.\n\\label{def:adaptiveR}\n\\end{definition}\n","start":38319,"end":41515},{"raw_tex":"\\subsection{A 2-level \\texorpdfstring{\\MGProx}{}algorithm}\\label{sec:mgprox:subsec:2lv}\nSimilar to the 2-level \\MGOPT method for solving Problem \\eqref{prob:mgopt}, we propose a 2-level \\MGProx method (Algorithm \\ref{aglo:2levelMG_exact}) that solves Problem \\eqref{prob:minfg} by utilizing a coarse problem defined as \n\\begin{equation}\\label{prob:minfg_coarse}\n\\argmin_{\\xi \\in \\IR^{n_1}}\\,\n\\bigg\\{\\,\nF^{\\tau}_1 (\\xi) \n~\\coloneqq~  F_1(\\xi) - \\langle \\tau_{0 \\rightarrow 1}^{k+1}, \\xi \\rangle\n~=~ f_1 (\\xi) + g_1(\\xi) - \\langle \\tau_{0 \\rightarrow 1}^{k+1}, \\xi \\rangle\n\\,\\bigg\\}.\n\\end{equation}\n\n\n\\begin{algorithm}\n\\caption{2-level \\MGProx for an approximate solution of \\eqref{prob:minfg}}\n\\label{aglo:2levelMG_exact}\n\\begin{algorithmic}\n\\STATE{Initialize $x^1_0$, $R$ and $P$}\n\\FOR{$k = 1,2,\\dots$}\n%\n\\STATE{(i)~~~~ $y^{k+1}_0 = \\prox_{\\frac{1}{L_0}g_0} \\Big (x^k_0 - \\frac{1}{L_0} \\nabla f(x^k_0) \\Big)$\n\\hfill level-0 proximal gradient step \\vspace{0.25mm}\n}\n\\STATE{(ii)~~~ $y^{k+1}_1  = R(y^{k+1}_0) y^{k+1}_0 $   \n\\hfill construct the level-1 coarse variable \\vspace{0.25mm}\n}\n\\STATE{(iii)~~ $\\tau_{0 \\rightarrow 1}^{k+1} \\hspace{-1mm} \\in\\,\n\\underline{\\partial F_1(y^{k+1}_1 )}  - R(y^{k+1}_0) \\, \\underline{\\partial  F_0(y^{k+1}_0)}$\n\\hfill construct the tau vector \\vspace{0.25mm}\n}\n\\STATE{(iv)~~ $\nx^{k+1}_1\n= \\underset{\\xi}{\\argmin} \\, \\Big\\{ F_1^{\\tau}(\\xi) \\coloneqq F_1(\\xi) - \\langle \\tau_{0 \\rightarrow 1}^{k+1}, \\xi \\rangle \\Big\\}\n$\n\\hfill solve the level-1 coarse problem\n}\n\\STATE{(v)\\,~~~ $z^{k+1}_0 = y^{k+1}_0  + \\alpha P\\big(  x^{k+1}_1 - y^{k+1}_1 \\big)$\n\\hfill coarse correction \\vspace{0.25mm}\n}\n\\STATE{(vi)~~\\, $x^{k+1}_0 = \\prox_{\\frac{1}{L_0}g_0} \\Big (z^{k+1}_0 - \\frac{1}{L_0} \\nabla f(z^{k+1}_0) \\Big)$\n\\hfill level-0 proximal gradient step~~\n}\n%\n\\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}\n\nHere are remarks for the steps in Algorithm~\\ref{aglo:2levelMG_exact}.\n\\begin{itemize}\n    \\item (i): we perform one or more proximal gradient iterations on the fine variable with a constant stepsize $1/L_0$, where $L_0$ is the Lipschitz constant of $\\nabla f_0$.\n    \\item (iii): we pick a value within the set to define $\\tau$;\n    as we are now using adaptive $R$, we use $+$ instead of $\\oplus$ in the expression of $\\tau$.\n    \\item (v): $\\alpha > 0$ is a stepsize; for its selection see \\cref{sec:mgprox:subsec:alpha}.\n    \\item The restriction for variable $y_0^{k+1}$ and the restriction for the subdifferential $ \\underline{\\partial  F_0(y^{k+1}_0)}$ can be slightly different.\n    On $y_0^{k+1}$ the restriction is the full restriction, on $ \\underline{\\partial  F_0(y^{k+1}_0)}$ is the adaptive one. \n    The explanation is as follows.   \n    For the particular cases of $g_0$ such as $\\ell_1$, $\\max\\{ \\cdot, 0\\}$ (element-wise maximum) and $\\iota_{[0,\\infty)}$ (indicator of nonnegative orthant), when we zero out column $i$ of $R$, the corresponding entry of $y_0^{k+1}$ is already $0$.    \n    Note that this conclusion does not hold in general for other nonsmooth functions or when the non-differentiability occur at another point (say at $x=1$). \n    Generally in those cases we will need to specify which restriction matrix (the full $R$ or the adaptive $R$) to be used to define $P$.\n        However, to simplify the presentation we always assume the non-differentiability occurs at $x=0$,\n         by performing a translation to shift the non-differentiability to occur at $x=0$.\n%        In fact, this is what exact we do in the experiment.\n\\end{itemize}\n","start":43823,"end":47322},{"raw_tex":"Before we prove the theorem we emphasize that \\eqref{iq:descent_dir} applies for any subgradient in the set $\\underline{\\partial F_0(y^{k+1}_0)} $.\nFurthermore,\n\\[\n\\eqref{iq:descent_dir} \n%\n\\iff\n%\n\\big\\langle \nP^\\top \\underline{\\partial F_0(y^{k+1}_0)} , x^{k+1}_1 - y^{k+1}_1 \n\\big\\rangle\n< 0\n%\n\\overset{P^\\top = cR}{\\iff}\n%\nc \\big\\langle \nR \\underline{\\partial F_0(y^{k+1}_0)} , x^{k+1}_1 - y^{k+1}_1 \n\\big\\rangle\n< 0\n.\n\\]\nAs $c>0$, showing \\eqref{iq:descent_dir} is equivalent to showing\n\\begin{equation}\n\\big\\langle R \\underline{\\partial F_0(y^{k+1}_0)},  ~ x^{k+1}_1 - y^{k+1}_1 \\big\\rangle < 0,\n\\label{pf:descent_to_prove}\n\\end{equation}\nwhere $R \\underline{\\partial F_0(y^{k+1}_0)} $ is a singleton vector for all subgradients in $\\underline{\\partial F_0(y^{k+1}_0)}$ due to the adaptive $R$.\n","start":51315,"end":52115},{"raw_tex":"\\subsubsection{Existence of coarse correction stepsize \\texorpdfstring{$\\alpha_k$}{}}\n\\label{sec:mgprox:subsec:alpha}\nBased on Theorem~\\ref{thm:coarse_dir_descent}, we now show that there exists a stepsize $\\alpha_k > 0$ such that \n\\begin{equation}\nF_0(z^{k+1}_0) \n~\\coloneqq~\nF_0\\big(\ny^{k+1}_0 + \\alpha_k P(x^{k+1}_1 - y^{k+1}_1) \n\\big)\n~<~\nF_0(y^{k+1}_0).\n\\label{iq:coarse_descent_on_F}\n\\end{equation}\n","start":53436,"end":53841},{"raw_tex":"To prove the lemma, we make use \nof a fact of the subdifferentials of finite convex functions \\cite[Def.1.1.4, p.165]{hiriart2004fundamentals}: $\\partial g(x)$ is a nonempty compact convex set $\\cS \\subset \\IRn$ whose support function $\\displaystyle \\sup_{s}\\{ \\langle s, x\\rangle \\,|\\, s \\in \\cS  \\}$ \nis the directional derivative of $g$ at $x$.\nBy this, the subdifferential\n $\\underline{\\partial F_0(y_0^{k+1})}$ is a compact convex set whose support function is the directional derivative of $F_0$ at $y_0^{k+1}$.\nWe emphasize that here \n%$F_0 : \\IR^{n_0} \\rightarrow  \\overline{\\IR}$ at $y_0^{k+1}$ is \n$F_0(y_0^{k+1})$ is \nfinite, \n% i.e., $F_0 : \\IR^{n_0} \\rightarrow  \\overline{\\IR}$ will never reach $+\\infty$ at $y_0^{k+1}$ since $y_0^{k+1}$ is obtained by the proximal gradient step (in other words, $y_0^{k+1}$ is a ``feasible point'' if $g_0$ is an indicator function of a constraint set),\nso we can make use of the result on directional derivative in \\cite[Def. 1.1.4, p.165]{hiriart2004fundamentals}, which only applies for finite convex functions, associated with the subdifferential.\n","start":54038,"end":55139},{"raw_tex":"\\begin{lemma}\n(Sufficient descent property of proximal gradient \\cite[Lemma 10.4]{beck2017first})\n\\label{lemma:suff_descent_proxgrad} \nLet $L_0$ be the Lipschitz constant of $\\nabla f_0$.\nFor step (i) in Algorithm~\\ref{aglo:2levelMG_exact},  we have\n\\begin{equation}\nF_0( y^{k+1}_0)\n\\leq \nF_0( x^k_0) - \\| G_0(x_0^k) \\|_2^2 / (2L_0),\n~~\nG_0( x_0^k) \n=\nL_0 \\big[\nx_0^k - \\prox_{ \\frac{1}{L} g_0 }\\big( x_0^k - \\nabla f_0(x_0^k)/L_0\n\\big)\n\\big].\n\\label{eqn:suffdescentPGD}\n\\end{equation}\n\\end{lemma}\n$G_0( x_0^k)$ is called the proximal gradient map of $F_0$ at $x_0^k$.\nThe inequality also holds for step (vi).\n","start":58297,"end":58907},{"raw_tex":"\\begin{algorithm}\n\\caption{Naive line search}\\label{aglo:naivelinesearch}\n\\begin{algorithmic}\n\\STATE{Set $\\alpha > 0$ and select a tolerance $\\epsilon > 0$  (e.g. $10^{-15}$)}\n\\WHILE{true}\n\\STATE{\\textbf{If} $F_0\\big( y^{k+1}_0 + \\alpha P (x^{k+1}_1 - y^{k+1}_1) \\big) \\leq F_0( y^{k+1}_0 )$ \\textbf{ then }\nreturn $z_0^{k+1}=y^{k+1}_0 + \\alpha P(x^{k+1}_1 - y^{k+1}_1)$.}\n\\\\\n\\STATE{\\textbf{else if} $\\alpha > \\epsilon $ \\textbf{then} $\\alpha = \\alpha/2 $.}\n\\\\\n\\STATE{~~~~~~~\\textbf{else} return $z_0^{k+1}=y^{k+1}_0$.}\n\\ENDWHILE\n\\end{algorithmic}\n\\end{algorithm}\n","start":61746,"end":62310},{"raw_tex":"\\subsubsection{Linear convergence rate}\nAll the functions and variables here are at level 0 so we omit the subscripts.\nNow we show that $\\big\\{F( x^k) \\big\\}_{k \\in \\IN}$ converges to $F^*$ with a linear rate using the \\emph{Proximal Polyak-{\\L}ojasiewicz inequality} \\cite[Section 4]{karimi2016linear}. \nThe function $F$ in Problem \\eqref{prob:minfg} is called ProxPL, if there exists $\\mu > 0$, called the ProxPL constant, such that \n\\begin{equation}\n\\cD_g(x,L)\n~\\geq~\n2 \\mu \\big( F(x) - F^* \\big)  \\qquad  \\forall x,\n\\tag{ProxPL}\n\\label{def:ProxPL}\n\\end{equation}\n\\begin{equation}\n\\cD_g(x,\\alpha) \n\\coloneqq\n-2\\alpha \\min_z \\,\n\\Big\\{\n{\\textstyle\\frac{\\alpha}{2}} \\| z - x \\|_2^2 \n+ \\big\\langle z - x, \\nabla f(x) \\big\\rangle\n+ g(z) - g(x)\n\\Big\\}\n.\n\\label{def:dg}\n\\end{equation}\nIntuitively, $\\cD_g$ is defined based on the proximal gradient operator:\n\\[\n\\prox_{\\frac{1}{L}g} \\Big( x - \\frac{\\nabla f(x)}{L} \\Big)\n=\n\\underset{z}{\\argmin} \\, \\frac{L}{2} \\| z -  x \\|_2^2\n+ \\big\\langle z - x, \\nabla f(x) \\big\\rangle\n+ g(z) - g(x).\n\\]\nIt has been shown in \\cite{karimi2016linear} that if $f$ in \\eqref{prob:minfg} is $\\mu$-strongly convex, then $F$ is $\\mu$-ProxPL.\nNow we prove the linear convergence rate of Algorithm~\\ref{aglo:2levelMG_exact}.\nNote that a standard result such as \\cite[Theorem 10.29]{beck2017first} on convergence of proximal gradient for strongly convex functions is not directly applicable because, as mentioned above, we interleave proximal gradient steps with coarse correction steps.\n","start":76605,"end":78114}]},"statement":{"raw_tex":"\\begin{theorem}\\label{thm:mgprox_converge_proxPL}\nLet $x^1_0$ be the initial guess of the algorithm,\n$x_0^* = \\argmin \\, F_0(x)$ and $F^*_0 \\coloneqq F_0(x_0^*)$.\nThe sequence $\\{x_0^k\\}_{k \\in \\IN}$ generated by \\MGProx (Algorithm~\\ref{aglo:2levelMG_exact}) for solving Problem \\eqref{prob:minfg} satisfies\n$\nF_0(x_0^{k+1}) - F_0^* \n\\leq \n\\Big( 1-\\frac{\\mu_0}{L_0} \\Big)^k \\big( F_0(x_0^1) - F_0^* \\big).\n$\n\n\n\n\n\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\\label{thm:mgprox_converge_proxPL}\nLet $x^1_0$ be the initial guess of the algorithm,\n$x_0^* = \\argmin \\, F_0(x)$ and $F^*_0 \\coloneqq F_0(x_0^*)$.\nThe sequence $\\{x_0^k\\}_{k \\in \\IN}$ generated by \\MGProx (Algorithm~\\ref{aglo:2levelMG_exact}) for solving Problem \\eqref{prob:minfg} satisfies\n$\nF_0(x_0^{k+1}) - F_0^* \n\\leq \n\\Big( 1-\\frac{\\mu_0}{L_0} \\Big)^k \\big( F_0(x_0^1) - F_0^* \\big).\n$\n\n\n","start":78147,"end":78557},{"raw_tex":"\n\\end{theorem}","start":79949,"end":79963}]},"proof":{"raw_tex":"\\begin{lemma}[Composition with full-rank matrix preserves  convexity]\n\\label{lem:strcvx_linearmap}\nGiven a function $F : \\IRn \\rightarrow \\overline{\\IR}$ that is $\\mu$-strongly convex and a rank-$n$ matrix $R \\in \\IR^{m \\times n}$, the function $F \\circ R$ is $\\mu \\sigma^2_n$-strongly convex, where $\\sigma_n$ is the $n$th singular value of $R$.\n\\begin{proof}\n$F$ is strongly convex so $\\partial F$ is strongly monotone \\cite{bauschke2011convex}: for all $x,y \\in \\dom F$, \n\\begin{equation}\\label{lem:strcvx_linearmap:fineF}\n\\big\\langle x - y, ~ \\partial F(x) - \\partial F(y ) \\big\\rangle \n~\\geq~\n\\mu \\| x - y\\|_2^2.\n\\end{equation}\nNow we show $\\partial (F \\circ R)$ is also strongly monotone.\nFor all $x,y $ in $\\dom (F \\circ R)$, we have\n\\[\n\\begin{array}{rcl}\n\\langle \nx - y, \n~ \\partial (F \\circ R)(x) - \\partial (F \\circ R)(y )\n\\rangle \n&\\overset{\\eqref{subdiff_coarse_linearmap}}{=}&\n\\langle \nx - y, \n~ R^\\top  \\partial F(Rx) - R^\\top  \\partial F (Ry )\n\\rangle \n\\\\\n&=&\n\\langle \nRx - Ry, \n~ \\partial F(Rx) -  \\partial F (Ry )\n\\rangle \n\\\\\n&\\overset{\\eqref{lem:strcvx_linearmap:fineF}}{\\geq}&\n\\mu \\| Rx-Ry\\|_2^2\n~\\geq~ \\mu \\sigma_n^2 \\| x-y\\|_2^2,\n\\end{array}\n\\]\nwhere $\\sigma_n > 0 $ is the $n$th singular value of the full-rank matrix $R$.\nThus the subdifferential $\\partial (F \\circ R)$ is  is $\\mu \\sigma_n^2$-strongly monotone, and thus the function $F \\circ R$ is $\\mu \\sigma_n^2$-strongly convex.\n\\end{proof}\n\\end{lemma}\n\n\\begin{theorem}[Angle condition of coarse correction]\\label{thm:coarse_dir_descent}\nIf $P(x^{k+1}_1 - y^{k+1}_1) \\neq 0$, then\n\\begin{equation}\\label{iq:descent_dir}\n\\big\\langle \n\\underline{\\partial F_0(y^{k+1}_0)} , P(x^{k+1}_1 - y^{k+1}_1) \n\\big\\rangle \n%\\leq \n<\n0. \n\\end{equation}\n%I.e., $P(x^{k+1}_1 - y^{k+1}_1) $ and any subgradient in $\\partial F_0(y^{k+1}_0)$ forms a strict obtuse angle.\n\\end{theorem}\n\n\\begin{proof}\nBy definition \n$\\tau_{0 \\rightarrow 1}^{k+1} \\overset{\\eqref{def:tau_mgprox1}}{\\in} \\underline{\\partial F_1(y_1^{k+1})}  - R \\underline{\\partial F_0(y_0^{k+1})}$ \nand the fact that $R \\underline{\\partial F_0(y^{k+1}_0)} $ is a singleton, we have\n$\nR \\underline{\\partial F_0(y_0^{k+1})} \n\\in\n\\underline{\\partial F_1(y_1^{k+1})}  - \\tau_{0 \\rightarrow 1}^{k+1}\n\\overset{\\eqref{prob:minfg_coarse}}{=} \\underline{\\partial F_1^\\tau(y_1^{k+1})}$,\nshowing that $R \\underline{\\partial F_0(y_0^{k+1})} $ is a subgradient of $F_1^{\\tau}$ at $y_1^{k+1}$.\nFor any subgradient in the subdifferential $\\underline{\\partial F_1^\\tau(y_1^{k+1})} $, we have the following which implies \\eqref{pf:descent_to_prove}:\n$\n\\big\\langle \\underline{\\partial F_1^{\\tau}(y^{k+1}_1)}, x^{k+1}_1 - y^{k+1}_1 \\big\\rangle\n< F_1^{\\tau}(x^{k+1}_1) - F_1^{\\tau}(y^{k+1}_1) \n< 0,\n$\nwhere the first strict inequality is due to $F_1^\\tau$ being a strongly convex function (which implies strict convexity);\nthe second inequality is by $x^{k+1}_1 \\coloneqq \\underset{\\xi}{\\argmin}\\,\nF_1^{\\tau}(\\xi)$ and the assumption that $x_1^{k+1} \\neq y_1^{k+1}$.\n\\end{proof}\n\n\\begin{lemma}[Existence of stepsize]\\label{lemma:alphaexists}\nThere exists $\\alpha_k > 0$ such that \\eqref{iq:coarse_descent_on_F} is satisfied for $P(x^{k+1}_1 - y^{k+1}_1) \\neq 0$.\n\\end{lemma}\n\n\\begin{proof}\nWe prove the lemma in 3 steps.\n\\begin{enumerate}\n\n\\item (Half-space)\nThe strict inequality in Theorem~\\ref{thm:coarse_dir_descent} means that $\\underline{\\partial F_0(y_0^{k+1})}$ is strictly inside a half-space with normal vector $p = P(x^{k+1}_1 - y^{k+1}_1)$.  \n\n\\item (Strict separation)\nBeing a compact convex set, $\\underline{\\partial F_0(y_0^{k+1}0)}$ lying strictly on one side of the hyperplane must be a positive distance (say $\\alpha_k > 0$) from that hyperplane.\n\n\\item (Support and directional derivative)\nEvaluating the support function of $\\underline{\\partial F_0(y_0^{k+1})}$, i.e., the directional derivative of $F_0$ at $y_0^{k+1}$ in the direction $p$, we have \\eqref{iq:coarse_descent_on_F}.\n\\end{enumerate}\n\\end{proof}\n\n\\begin{proof}\nFirst, by assumption $F$ is strongly convex, so $F$ is $\\mu$-ProxP\\L\\, with $\\mu > 0$ and\n\\[\n\\renewcommand*{\\arraystretch}{1.33}\n\\begin{array}{rcl}\nF(x^{k+1})\n&\\overset{\\eqref{eqn:suffdescentPGD}}{\\leq}& \\hspace{-2mm}\nF(z^{k+1})\n~\\overset{\\eqref{iq:coarse_descent_on_F}}{\\leq}~ \nF(y^{k+1})  \n~=~\nf(y^{k+1}) + g(y^{k+1}) + g(x^k) - g(x^k)\n\\\\\n&\\overset{\\eqref{def:Lsmooth}}{\\leq}&  \\hspace{-2mm}\nf(x^k) +  \\langle  \\nabla f(x^k), y^{k+1} - x^k \\rangle + \\frac{L}{2} \\|  y^{k+1} - x^k \\|_2^2 + g(x^k) +  g(y^{k+1})  -  g(x^k) \t\n\\\\\n&=& \\hspace{-2mm}\nF(x^k) +  \\langle  \\nabla f(x^k), y^{k+1} - x^k \\rangle + \\frac{L}{2} \\|  y^{k+1} - x^k \\|_2^2 +  g(y^{k+1})  -  g(x^k) \n\\\\\n&=& \\hspace{-2mm}\nF(x^k) - \\frac{1}{2L} \\underbrace{(-2L)\\Big( \n\\langle  \\nabla f(x^k), y^{k+1} - x^k \\rangle \n+ {\\textstyle \\frac{L}{2}} \\|  y^{k+1} - x^k \\|_2^2 +  g(y^{k+1})  -  g(x^k) \n\\Big)}_{\\overset{\\eqref{def:dg}}{ = } \\, \\cD_g(x^k, L), \\, \\text{ since }y^{k+1} \\,\\coloneqq\\, \\underset{z}{\\argmin} \\, \\frac{L}{2} \\| z -  x^k \\|_2^2\n+ \\langle z - x^k, \\nabla f(x^k)\\rangle\n+ g(z) - g(x^k).}\n\\\\\n&\\overset{\\eqref{def:ProxPL}}{\\leq}& \\hspace{-2mm}\nF(x^k) - \\frac{\\mu}{L} \\big(F(x^k) - F^*\\big).\n\\end{array}\n\\]\nAdding $-F^*$ on both sides of the inequality gives\n$ \nF(x^{k+1}) - F^* \n\\leq \\big( 1 - \\mu/L\\big) \\big(F(x^k) - F^*\\big)\n$.\nApplying this inequality recursively completes the proof.\n\\end{proof}","units":[{"raw_tex":"\\begin{lemma}[Composition with full-rank matrix preserves  convexity]\n\\label{lem:strcvx_linearmap}\nGiven a function $F : \\IRn \\rightarrow \\overline{\\IR}$ that is $\\mu$-strongly convex and a rank-$n$ matrix $R \\in \\IR^{m \\times n}$, the function $F \\circ R$ is $\\mu \\sigma^2_n$-strongly convex, where $\\sigma_n$ is the $n$th singular value of $R$.\n\\begin{proof}\n$F$ is strongly convex so $\\partial F$ is strongly monotone \\cite{bauschke2011convex}: for all $x,y \\in \\dom F$, \n\\begin{equation}\\label{lem:strcvx_linearmap:fineF}\n\\big\\langle x - y, ~ \\partial F(x) - \\partial F(y ) \\big\\rangle \n~\\geq~\n\\mu \\| x - y\\|_2^2.\n\\end{equation}\nNow we show $\\partial (F \\circ R)$ is also strongly monotone.\nFor all $x,y $ in $\\dom (F \\circ R)$, we have\n\\[\n\\begin{array}{rcl}\n\\langle \nx - y, \n~ \\partial (F \\circ R)(x) - \\partial (F \\circ R)(y )\n\\rangle \n&\\overset{\\eqref{subdiff_coarse_linearmap}}{=}&\n\\langle \nx - y, \n~ R^\\top  \\partial F(Rx) - R^\\top  \\partial F (Ry )\n\\rangle \n\\\\\n&=&\n\\langle \nRx - Ry, \n~ \\partial F(Rx) -  \\partial F (Ry )\n\\rangle \n\\\\\n&\\overset{\\eqref{lem:strcvx_linearmap:fineF}}{\\geq}&\n\\mu \\| Rx-Ry\\|_2^2\n~\\geq~ \\mu \\sigma_n^2 \\| x-y\\|_2^2,\n\\end{array}\n\\]\nwhere $\\sigma_n > 0 $ is the $n$th singular value of the full-rank matrix $R$.\nThus the subdifferential $\\partial (F \\circ R)$ is  is $\\mu \\sigma_n^2$-strongly monotone, and thus the function $F \\circ R$ is $\\mu \\sigma_n^2$-strongly convex.\n\\end{proof}\n\\end{lemma}","start":36648,"end":38078},{"raw_tex":"\\begin{theorem}[Angle condition of coarse correction]\\label{thm:coarse_dir_descent}\nIf $P(x^{k+1}_1 - y^{k+1}_1) \\neq 0$, then\n\\begin{equation}\\label{iq:descent_dir}\n\\big\\langle \n\\underline{\\partial F_0(y^{k+1}_0)} , P(x^{k+1}_1 - y^{k+1}_1) \n\\big\\rangle \n%\\leq \n<\n0. \n\\end{equation}\n%I.e., $P(x^{k+1}_1 - y^{k+1}_1) $ and any subgradient in $\\partial F_0(y^{k+1}_0)$ forms a strict obtuse angle.\n\\end{theorem}","start":50904,"end":51314},{"raw_tex":"\\begin{proof}\nBy definition \n$\\tau_{0 \\rightarrow 1}^{k+1} \\overset{\\eqref{def:tau_mgprox1}}{\\in} \\underline{\\partial F_1(y_1^{k+1})}  - R \\underline{\\partial F_0(y_0^{k+1})}$ \nand the fact that $R \\underline{\\partial F_0(y^{k+1}_0)} $ is a singleton, we have\n$\nR \\underline{\\partial F_0(y_0^{k+1})} \n\\in\n\\underline{\\partial F_1(y_1^{k+1})}  - \\tau_{0 \\rightarrow 1}^{k+1}\n\\overset{\\eqref{prob:minfg_coarse}}{=} \\underline{\\partial F_1^\\tau(y_1^{k+1})}$,\nshowing that $R \\underline{\\partial F_0(y_0^{k+1})} $ is a subgradient of $F_1^{\\tau}$ at $y_1^{k+1}$.\nFor any subgradient in the subdifferential $\\underline{\\partial F_1^\\tau(y_1^{k+1})} $, we have the following which implies \\eqref{pf:descent_to_prove}:\n$\n\\big\\langle \\underline{\\partial F_1^{\\tau}(y^{k+1}_1)}, x^{k+1}_1 - y^{k+1}_1 \\big\\rangle\n< F_1^{\\tau}(x^{k+1}_1) - F_1^{\\tau}(y^{k+1}_1) \n< 0,\n$\nwhere the first strict inequality is due to $F_1^\\tau$ being a strongly convex function (which implies strict convexity);\nthe second inequality is by $x^{k+1}_1 \\coloneqq \\underset{\\xi}{\\argmin}\\,\nF_1^{\\tau}(\\xi)$ and the assumption that $x_1^{k+1} \\neq y_1^{k+1}$.\n\\end{proof}","start":52115,"end":53251},{"raw_tex":"\\begin{lemma}[Existence of stepsize]\\label{lemma:alphaexists}\nThere exists $\\alpha_k > 0$ such that \\eqref{iq:coarse_descent_on_F} is satisfied for $P(x^{k+1}_1 - y^{k+1}_1) \\neq 0$.\n\\end{lemma}","start":53842,"end":54036},{"raw_tex":"\\begin{proof}\nWe prove the lemma in 3 steps.\n\\begin{enumerate}\n\n\\item (Half-space)\nThe strict inequality in Theorem~\\ref{thm:coarse_dir_descent} means that $\\underline{\\partial F_0(y_0^{k+1})}$ is strictly inside a half-space with normal vector $p = P(x^{k+1}_1 - y^{k+1}_1)$.  \n\n\\item (Strict separation)\nBeing a compact convex set, $\\underline{\\partial F_0(y_0^{k+1}0)}$ lying strictly on one side of the hyperplane must be a positive distance (say $\\alpha_k > 0$) from that hyperplane.\n\n\\item (Support and directional derivative)\nEvaluating the support function of $\\underline{\\partial F_0(y_0^{k+1})}$, i.e., the directional derivative of $F_0$ at $y_0^{k+1}$ in the direction $p$, we have \\eqref{iq:coarse_descent_on_F}.\n\\end{enumerate}\n\\end{proof}","start":55140,"end":55893},{"raw_tex":"\\begin{proof}\nFirst, by assumption $F$ is strongly convex, so $F$ is $\\mu$-ProxP\\L\\, with $\\mu > 0$ and\n\\[\n\\renewcommand*{\\arraystretch}{1.33}\n\\begin{array}{rcl}\nF(x^{k+1})\n&\\overset{\\eqref{eqn:suffdescentPGD}}{\\leq}& \\hspace{-2mm}\nF(z^{k+1})\n~\\overset{\\eqref{iq:coarse_descent_on_F}}{\\leq}~ \nF(y^{k+1})  \n~=~\nf(y^{k+1}) + g(y^{k+1}) + g(x^k) - g(x^k)\n\\\\\n&\\overset{\\eqref{def:Lsmooth}}{\\leq}&  \\hspace{-2mm}\nf(x^k) +  \\langle  \\nabla f(x^k), y^{k+1} - x^k \\rangle + \\frac{L}{2} \\|  y^{k+1} - x^k \\|_2^2 + g(x^k) +  g(y^{k+1})  -  g(x^k) \t\n\\\\\n&=& \\hspace{-2mm}\nF(x^k) +  \\langle  \\nabla f(x^k), y^{k+1} - x^k \\rangle + \\frac{L}{2} \\|  y^{k+1} - x^k \\|_2^2 +  g(y^{k+1})  -  g(x^k) \n\\\\\n&=& \\hspace{-2mm}\nF(x^k) - \\frac{1}{2L} \\underbrace{(-2L)\\Big( \n\\langle  \\nabla f(x^k), y^{k+1} - x^k \\rangle \n+ {\\textstyle \\frac{L}{2}} \\|  y^{k+1} - x^k \\|_2^2 +  g(y^{k+1})  -  g(x^k) \n\\Big)}_{\\overset{\\eqref{def:dg}}{ = } \\, \\cD_g(x^k, L), \\, \\text{ since }y^{k+1} \\,\\coloneqq\\, \\underset{z}{\\argmin} \\, \\frac{L}{2} \\| z -  x^k \\|_2^2\n+ \\langle z - x^k, \\nabla f(x^k)\\rangle\n+ g(z) - g(x^k).}\n\\\\\n&\\overset{\\eqref{def:ProxPL}}{\\leq}& \\hspace{-2mm}\nF(x^k) - \\frac{\\mu}{L} \\big(F(x^k) - F^*\\big).\n\\end{array}\n\\]\nAdding $-F^*$ on both sides of the inequality gives\n$ \nF(x^{k+1}) - F^* \n\\leq \\big( 1 - \\mu/L\\big) \\big(F(x^k) - F^*\\big)\n$.\nApplying this inequality recursively completes the proof.\n\\end{proof}","start":78557,"end":79949}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["prob:minfg"],"declared_external":false,"source_only":false,"title":"细层复合优化问题","source_units":[{"raw_tex":"We study the combination of two iterative algorithms: proximal gradient descent and multigrid, to solve the following class of optimization problems\n\\begin{equation}\\label{prob:minfg}\n\\argmin_x ~ F_0(x) \\coloneqq f_0(x) + g_0(x).\n\\end{equation}\n","start":6637,"end":6882}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"光滑项的强凸假设","source_units":[{"raw_tex":"\\textbf{(Assumption)} We assume $f_0 : \\IRn \\rightarrow \\IR$ is \n% lower semi-continuous, % NO need because smoothness implies LSC\n$L_0$-smooth and $\\mu_0$-strongly convex,\nand  \n","start":6882,"end":7061}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"非光滑项的有限性与可分性","source_units":[{"raw_tex":"%$g_0 : \\IRn \\rightarrow \\overline{\\IR} \\coloneqq \\IR \\cup \\{+\\infty\\}$ \n$g_0 : \\IRn \\rightarrow \\IR$ \nis proper, possibly nonsmooth, convex, \n%lower semi-continuous, \nand separable.\nRegarding the non-smoothness (nondifferentiability) of $g_0$, we further assume a single point of non-differentiability for $g_0$.\n","start":7061,"end":7375}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"可分性与强凸模量","source_units":[{"raw_tex":"We recall that a function $\\zeta(x) : \\IRn \\rightarrow \\IR$ is separable if $\\zeta(x) = \\sum_i \\zeta_i(x_i)$, and  is $\\mu$-strongly convex (and thus coercive) with $\\mu>0$ if $\\zeta(x) - \\frac{\\mu}{2}\\| x \\|_2^2$ is convex;\n","start":7375,"end":7600}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":["def:Lsmooth"],"declared_external":false,"source_only":false,"title":"光滑性与二次上界","source_units":[{"raw_tex":"and lastly is $L$-smooth if  $\\zeta$ is $C^{1,1}_L$ and $\\nabla \\zeta$ is $L$-Lipschitz; i.e., for all $x,y$ in $\\IRn$, $\\nabla \\zeta(x)$ exists and\n\\begin{equation}\n\\zeta(y) \\leq \\zeta(x) + \\big\\langle \\nabla \\zeta(x), y-x \\big\\rangle + \\frac{L}{2}\\|y-x\\|_2^2.\n\\label{def:Lsmooth}\n\\end{equation}\n","start":7600,"end":7897}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"层号与光滑常数记号","source_units":[{"raw_tex":"\\paragraph{Notation}\nThe symbol $x_0$ (or $x$) is called the fine variable.\nThe symbol $x_\\ell$ in \\eqref{prob:mgopt_coarse} with $\\ell \\geq 1$ is called coarse variable.\nThe subscript $\\ell \\in \\{0,1,\\dots, L \\}$ denotes the level.\nA larger $\\ell$ means a coarser level with lower resolution (fewer variables).\n%We use $\\ell=0$ to denote the finest level with the highest resolution and $\\ell = L$ to denote the coarsest level with the lowest resolution.\nFor the remainder of the paper, $L$ without a subscript stands for the number of levels, whereas $L_{\\ell}$ denotes the smoothness parameter for the level-$\\ell$ problem.\n%\n","start":11252,"end":11881}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"限制与延拓矩阵","source_units":[{"raw_tex":"At a level $\\ell$, the coarse version of the vector $x_\\ell \\in \\IR^{n_\\ell}$ is $x_{\\ell+1} = \\overline{\\cR}(x_\\ell) \\coloneqq \\overline{R}x_\\ell$ where $\\overline{R} \\in \\IR^{n_{\\ell+1} \\times n_{\\ell}}$ with $n_{\\ell+1} \\leq n_{\\ell}$ is called a restriction matrix.\nSimilarly, given $x_{\\ell+1} \\in \\IR^{n_{\\ell+1}}$ and a prolongation matrix $\\overline{P} \\in \\IR^{ n_{\\ell} \\times n_{\\ell +1}}$, \nwe obtain the level-$\\ell$ version of $x_{\\ell+1}$ as $x_{\\ell} = \\overline{\\cP}(x_{\\ell+1}) \\coloneqq \\overline{P} x_{\\ell+1}$.\nWe let $\\overline{P} = c \\overline{R}^\\top$ given scaling factor $c>0$ and $\\overline{R}$.\n","start":11881,"end":12504}]},{"id":"S008","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"跨层矩阵的应用选择","source_units":[{"raw_tex":"In multigrid, choosing $(\\overline{R},\\overline{P})$ depends on the application.\nIn this work, for the applications we consider commonly chosen $(\\overline{R},\\overline{P})$, see Section\\,\\ref{sec:exp:subsec:mgprox_eop} for our choice of \n$(\\overline{R},\\overline{P})$ used in the experiment and see \\cite{briggs2000multigrid} for an introduction on how to select $(\\overline{R},\\overline{P})$.\n","start":12504,"end":12899}]},{"id":"S009","semantic_type":"algorithm","kind":"algorithm","labels":[],"declared_external":false,"source_only":false,"title":"近端梯度更新","source_units":[{"raw_tex":"The proximal gradient method solves problems of the form \\eqref{prob:minfg} as follows.\nStarting from an initial guess $x^1$, the method updates the variable by a gradient descent step (with a stepsize $\\alpha > 0$) followed by a proximal step associated with $g_0$:\n\\begin{subequations}\\label{eq:5}\n\\begin{align}\nx^{k+1} &\\,= \\prox_{\\alpha g_0} \n\\Big( \nx^k - \\alpha \\nabla f_0(x^k)\n\\Big), \n%~~ \\alpha > 0\n\\label{updt:proxgrad}\n\\\\\n","start":18761,"end":19192}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"近端算子定义","source_units":[{"raw_tex":"\\prox_{\\alpha g_0}(x) \n& \\coloneqq \n\\underset{u}{\\argmin} ~ \\Big\\{\\, \\alpha g_0(u) + \\frac{1}{2} \\| u - x \\|_2^2 \\,\\Big\\}\n= \\underset{u}{\\argmin} ~ \\Big\\{\\, g_0(u) + \\frac{1}{2\\alpha} \\| u - x \\|_2^2 \\,\\Big\\}.\n\\label{def:prox_opertor}\n\\end{align}\n\\end{subequations}\n","start":19192,"end":19458}]},{"id":"S011","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"粗层函数记号","source_units":[{"raw_tex":"\\subsection{Functions at different levels}\\label{sec:mgprox:subsec:RP}\nFollowing \\cref{sec:intro:subsec:mgopt}, \nwe use $f_\\ell : \\IR^{n_\\ell} \\rightarrow \\IR$ to denote functions at different coarse levels.\nIn this section we will focus on $\\ell \\in \\{0,1\\}$ but we remark that all the notations and definitions are generalized to $\\ell \\in \\{0,1,\\dots,L\\}$ in Section~\\ref{sec:multilevel}.\nWe denote the restriction of the fine objective function $F_0$ in \\eqref{prob:minfg} as\n$ F_1 \\coloneqq \\cR(F_0)  = \\cR(f_0 + g_0)$, where $\\cR$ is defined below.\n\n","start":30220,"end":30776}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["def:restriction"],"declared_external":false,"source_only":false,"title":"函数限制的作者定义","source_units":[{"raw_tex":"\\begin{definition}[Restriction]\\label{def:restriction}\nAt a level $\\ell \\in \\IN$, given a function $f_\\ell : \\IR^{n_\\ell} \\rightarrow \\IR$, the restriction $\\cR$ of $f_\\ell$, denoted as $f_{\\ell+1} \\coloneqq \\cR(f_\\ell)$, is defined as \n$f_{\\ell+1}(x_{\\ell+1}) \\coloneqq f_{\\ell}(R x_{\\ell})$, \nwhere $R : \\IR^{n_\\ell} \\rightarrow \\IR^{n_{\\ell+1}}$ is a restriction matrix .\nWe also define the associated prolongation matrix \n$P : \\IR^{n_{\\ell+1}} \\rightarrow \\IR^{n_\\ell}$ \n as $P = cR^\\top$ where $c>0$ is a predefined constant.\n\\end{definition}\n","start":30776,"end":31324}]},{"id":"S013","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"自适应与固定算子的区分","source_units":[{"raw_tex":"\n\\paragraph{Adaptive restriction and non-adaptive restriction}\nWe recall that a contribution of this work is the introduction of the adaptive restriction, to be discussed in \\cref{sec:mgprox:subsec:adaptiveR} (see Definition\\,\\ref{eg:adaptiveR_L1}).\nTo differentiate the classical non-adaptive restriction (and the associated prolongation) from the adaptive version, we denote the non-adaptive restriction by $\\overline{\\cR}, \\overline{R},\\overline{\\cP},\\overline{P}$, and denote the adaptive one by $\\cR, R,\\cP,P$.\nWe remark that Definition~\\ref{def:restriction} can be used for both versions of restriction and prolongation.\nWe give an example of $\\overline{R},\\overline{P}$ in \\cref{sec:exp}.\n","start":31324,"end":32020}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":["def:subdifferential_II"],"declared_external":false,"source_only":false,"title":"次梯度与次微分","source_units":[{"raw_tex":"\\subsection{Review of subdifferential of nonsmooth functions}\\label{sec:mgprox:subsec:subdifferential}\nThe subdifferential \\cite{rockafellar1970convex}\nis a standard framework used in convex analysis to deal with nondifferentiable functions.\nA convex function $g(x) : \\IRn \\rightarrow \\overline{\\IR} \\coloneqq \\IR \\cup \\{+\\infty\\}$ is called nonsmooth if it is not differentiable for some $x$ in $\\IRn$.\nA point $q \\in \\IRn$ is called a subgradient of $g$ at  $x$ if for all $y \\in \\IRn$ the inequality $g(y) \\geq g(x) + \\langle q, y -x \\rangle$ holds.\nThe subdifferential of $g$ at a point $x$ is defined as the set of all subgradients of $g$ at $x$, i.e.,\n\\begin{equation}\n  \\partial g(x)\n\\coloneqq \n\\Big\\{~\nq \\in \\IRn ~~\\big|~~ \ng(y)  \\geq g(x) + \\langle q, y -x \\rangle\n~\\forall y \\in \\IRn\n~\\Big\\} ~ \\subset \\IRn,\n\\label{def:subdifferential_II}\n\\end{equation}\nso $\\partial g(x)$ is generally set-valued.\nIf $g$ is differentiable at $x$, then $\\nabla g(x)$ exists and the set $\\partial g(x)$ reduces to the singleton $\\big\\{\\nabla g(x)\\big\\}$.\n","start":32067,"end":33114}]},{"id":"S015","semantic_type":"external_fact","kind":"external_fact","labels":["fact:subgrad_sum_rule"],"declared_external":true,"declaration_text":["原始前提清单中的 Moreau–Rockafellar 和规则；原文第716行引用 Beck (2017), Theorem 3.36。"],"source_only":false,"title":"次微分和规则","source_units":[{"raw_tex":"\\paragraph{Subdifferential sum rule (Moreau–Rockafellar theorem)}~~\nLet $\\oplus$ denote the Minkowski sum.\nSince subdifferentials are generally set-valued, hence for two functions $f, g$, generally $\\partial (f+g) \\neq \\partial f \\oplus \\partial g$ but $\\partial (f+g) \\supset \\partial f \\oplus \\partial g$.\nThe sum rule $\\partial (f+g) = \\partial f \\oplus \\partial g$ holds if $f,g$ satisfy a qualification condition (e.g. \\cite[Theorem 3.36]{beck2017first}): the relative interior of the domain of $f$ has a non-empty intersection with the relative interior of domain of $g$,\ni.e.,\n\\begin{equation}\n\\ri (\\dom f) \\,\\cap\\, \\ri (\\dom g) \\neq \\varnothing\n%\n\\implies\n%\n\\partial \\Big( f(x) + g(x) \\Big) =\n\\partial f(x) \\oplus \\partial g(x),\n~~~\n\\forall x \\in \\textrm{dom} f \\,\\cap\\, \\textrm{dom} g.\n\\label{fact:subgrad_sum_rule}\n\\end{equation}\nThe right-hand side (RHS) of \\eqref{fact:subgrad_sum_rule}, which is the subdifferential sum rule,\nis known as the Moreau–Rockafellar theorem \\cite{lai1988moreau}.\n","start":33120,"end":34124}]},{"id":"S016","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"各层和规则资格条件","source_units":[{"raw_tex":"We now discuss the fact that the functions $f_\\ell, g_\\ell$ for all levels $\\ell$ in this work satisfy the Moreau–Rockafellar theorem. \n\\begin{itemize}\n\\item At level $\\ell=0$, we have the left-hand side (LHS) of \\eqref{fact:subgrad_sum_rule} for $f_0, g_0$ \n%as $\\dom f_0 \n%= \\dom g_0 \n%= \\IRn$\nby assumption.\n\\item At levels $\\ell>0$, by Definition~\\ref{def:restriction}, \nthe LHS of \\eqref{fact:subgrad_sum_rule} holds for the coarse functions.\nTo see this, take the coarse function as composition of the fine function with a linear map, and recognize the fact that the domain of a function is preserved under composition with a linear map.\nTo be explicit, we have\n\\[\n\\begin{array}{rcl}\n\\ri \\Big( \\dom f_1 \\Big)\n\\cap \n\\ri \\Big( \\dom g_1 \\Big)\n~=~\n\\ri \\Big( \\underbrace{\\dom (f_0 \\circ R)}_{\\text{whole }\\IR^{n_1}} \\Big)\n\\cap \n\\ri \\Big( \\dom (g_0 \\circ R)  \\Big)\n~\\neq~ \\varnothing.\n\\end{array}\n\\]\n\\end{itemize}\n","start":34124,"end":35038}]},{"id":"S017","semantic_type":"derived_fact","kind":"derived_fact","labels":["ass:F_sum_rule"],"declared_external":false,"source_only":false,"title":"各层次微分分解","source_units":[{"raw_tex":"To sum up, in this work \\eqref{fact:subgrad_sum_rule} holds for all levels $\\ell$:\n\\begin{equation}\n\\partial F_\\ell(x_\\ell) \n\\coloneqq  \\partial \\Big( f_\\ell(x_\\ell) + g_\\ell(x_\\ell) \\Big)\n= \\partial f_\\ell(x_\\ell) \\oplus \\partial g_\\ell(x_\\ell)\n= \\nabla f_\\ell(x_\\ell) + \\partial g_\\ell (x_\\ell)\n,\n\\label{ass:F_sum_rule}\n\\end{equation}\nwhere $+$ is used instead of $\\oplus$ in $\\nabla f_\\ell(x_\\ell) + \\partial g_\\ell (x_\\ell)$ because $\\nabla f_\\ell(x_\\ell)$ is a singleton.\n","start":35038,"end":35515}]},{"id":"S018","semantic_type":"external_fact","kind":"external_fact","labels":["subdiff_coarse_linearmap"],"declared_external":true,"declaration_text":["原始前提清单中的线性复合链式规则；原文第779行引用 Beck (2017), Theorem 3.43。"],"source_only":false,"title":"线性复合的次微分链式规则","source_units":[{"raw_tex":"\\subsection{Convexity and subdifferential of coarse function}\nFrom Definition~\\ref{def:restriction}, the coarse function can be written as $F_{\\ell+1} \\coloneqq F_\\ell \\circ R$, meaning that we can see the restriction process as the fine function taking composition with the linear map $R$.\nSuch composition view point gives us a series of useful properties for this work.\nFirst, we have a closed-form expression for the subdifferential of the coarse function in terms of the the subdifferential of the fine function.\nI.e., by \\cite[Theorem 3.43]{beck2017first}, we have that\n\\begin{equation}\\label{subdiff_coarse_linearmap}\n\\partial F_{\\ell+1}(x) = \\partial (F_\\ell \\circ R)(x) = R^\\top \\partial F_\\ell (Rx). \n\\end{equation}\n","start":35573,"end":36299}]},{"id":"S019","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"线性复合保持凸性","source_units":[{"raw_tex":"Then, by the fact that convexity is preserved under linear map, we have that ``restriction preserves convexity''.\nIn other words, $f_{\\ell+1}$ is convex if $f_{\\ell}$ is convex.\n","start":36299,"end":36477}]},{"id":"S020","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"限制矩阵满秩的断言","source_units":[{"raw_tex":"Furthermore, if $R$ is full-rank (which is the case in this paper), restriction is submultiplicative on the modulus of convexity, as illustrated in the following lemma.\n","start":36477,"end":36646}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","labels":["lem:strcvx_linearmap"],"declared_external":false,"source_only":false,"title":"满秩复合的强凸模量","source_units":[{"raw_tex":"\\begin{lemma}[Composition with full-rank matrix preserves  convexity]\n\\label{lem:strcvx_linearmap}\nGiven a function $F : \\IRn \\rightarrow \\overline{\\IR}$ that is $\\mu$-strongly convex and a rank-$n$ matrix $R \\in \\IR^{m \\times n}$, the function $F \\circ R$ is $\\mu \\sigma^2_n$-strongly convex, where $\\sigma_n$ is the $n$th singular value of $R$.\n","start":36648,"end":36995},{"raw_tex":"\\end{lemma}","start":38067,"end":38078}]},{"id":"S022","semantic_type":"proof","kind":"proof","labels":["lem:strcvx_linearmap","lem:strcvx_linearmap:fineF"],"declared_external":false,"source_only":false,"title":"强凸复合引理的完整证明","source_units":[{"raw_tex":"\\begin{proof}\n$F$ is strongly convex so $\\partial F$ is strongly monotone \\cite{bauschke2011convex}: for all $x,y \\in \\dom F$, \n\\begin{equation}\\label{lem:strcvx_linearmap:fineF}\n\\big\\langle x - y, ~ \\partial F(x) - \\partial F(y ) \\big\\rangle \n~\\geq~\n\\mu \\| x - y\\|_2^2.\n\\end{equation}\nNow we show $\\partial (F \\circ R)$ is also strongly monotone.\nFor all $x,y $ in $\\dom (F \\circ R)$, we have\n\\[\n\\begin{array}{rcl}\n\\langle \nx - y, \n~ \\partial (F \\circ R)(x) - \\partial (F \\circ R)(y )\n\\rangle \n&\\overset{\\eqref{subdiff_coarse_linearmap}}{=}&\n\\langle \nx - y, \n~ R^\\top  \\partial F(Rx) - R^\\top  \\partial F (Ry )\n\\rangle \n\\\\\n&=&\n\\langle \nRx - Ry, \n~ \\partial F(Rx) -  \\partial F (Ry )\n\\rangle \n\\\\\n&\\overset{\\eqref{lem:strcvx_linearmap:fineF}}{\\geq}&\n\\mu \\| Rx-Ry\\|_2^2\n~\\geq~ \\mu \\sigma_n^2 \\| x-y\\|_2^2,\n\\end{array}\n\\]\nwhere $\\sigma_n > 0 $ is the $n$th singular value of the full-rank matrix $R$.\nThus the subdifferential $\\partial (F \\circ R)$ is  is $\\mu \\sigma_n^2$-strongly monotone, and thus the function $F \\circ R$ is $\\mu \\sigma_n^2$-strongly convex.\n\\end{proof}\n","start":36995,"end":38067}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","labels":["lem:strcvx_linearmap","lem:strcvx_linearmap:fineF"],"declared_external":false,"source_only":false,"title":"将强凸性转为强单调性","source_units":[{"raw_tex":"\\begin{proof}\n$F$ is strongly convex so $\\partial F$ is strongly monotone \\cite{bauschke2011convex}: for all $x,y \\in \\dom F$, \n\\begin{equation}\\label{lem:strcvx_linearmap:fineF}\n\\big\\langle x - y, ~ \\partial F(x) - \\partial F(y ) \\big\\rangle \n~\\geq~\n\\mu \\| x - y\\|_2^2.\n\\end{equation}\n","start":36995,"end":37281}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","labels":["lem:strcvx_linearmap"],"declared_external":false,"source_only":false,"title":"链式规则与伴随转移","source_units":[{"raw_tex":"Now we show $\\partial (F \\circ R)$ is also strongly monotone.\nFor all $x,y $ in $\\dom (F \\circ R)$, we have\n\\[\n\\begin{array}{rcl}\n\\langle \nx - y, \n~ \\partial (F \\circ R)(x) - \\partial (F \\circ R)(y )\n\\rangle \n&\\overset{\\eqref{subdiff_coarse_linearmap}}{=}&\n\\langle \nx - y, \n~ R^\\top  \\partial F(Rx) - R^\\top  \\partial F (Ry )\n\\rangle \n\\\\\n&=&\n\\langle \nRx - Ry, \n~ \\partial F(Rx) -  \\partial F (Ry )\n\\rangle \n\\\\\n","start":37281,"end":37691}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":["lem:strcvx_linearmap"],"declared_external":false,"source_only":false,"title":"用奇异值控制复合范数","source_units":[{"raw_tex":"&\\overset{\\eqref{lem:strcvx_linearmap:fineF}}{\\geq}&\n\\mu \\| Rx-Ry\\|_2^2\n~\\geq~ \\mu \\sigma_n^2 \\| x-y\\|_2^2,\n\\end{array}\n\\]\nwhere $\\sigma_n > 0 $ is the $n$th singular value of the full-rank matrix $R$.\n","start":37691,"end":37893}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","labels":["lem:strcvx_linearmap"],"declared_external":false,"source_only":false,"title":"由强单调性返回强凸性","source_units":[{"raw_tex":"Thus the subdifferential $\\partial (F \\circ R)$ is  is $\\mu \\sigma_n^2$-strongly monotone, and thus the function $F \\circ R$ is $\\mu \\sigma_n^2$-strongly convex.\n\\end{proof}\n","start":37893,"end":38067}]},{"id":"S027","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"集合值项的下划线","source_units":[{"raw_tex":"\\paragraph{Notation for sets}\nFrom now on, when we encounter an expression containing both set-valued vector(s) and singleton vector(s), we underline the set-valued term(s) for visual clarity.\n","start":38081,"end":38274}]},{"id":"S028","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"跨层校正向量集合","source_units":[{"raw_tex":"\\subsection{Adaptive restriction and the \\texorpdfstring{$\\tau$}{} vector}\\label{sec:mgprox:subsec:adaptiveR}\nSince subdifferentials are set-valued, we define $\\tau$ in \\MGProx as an element of a set.\nAt a level $\\ell$, we define a set $\\underline{\\tau_{\\ell \\rightarrow \\ell+1}} ~\\coloneqq~\n\\underline{\\partial F_{\\ell+1}(x_{\\ell+1})} \\oplus (- R) \\underline{\\partial F_\\ell (x_\\ell)} $, where $\\ell \\rightarrow \\ell+1$ specifies that $\\tau$ connects level $\\ell$ to level $\\ell+1$, and the matrix $R$ here is an adaptive restriction operator that we will define soon.\nIn \\MGProx we choose an element of $\\underline{\\tau_{\\ell \\rightarrow \\ell+1}}$ as the tau vector.\nThat is, at level $\\ell=0$,\n\\begin{subequations}\n\\begin{align}\n\\tau_{0 \\rightarrow 1}\n~\\, \\in \\,~ \n\\underline{\\tau_{0 \\rightarrow 1}}\n& \n~\\coloneqq~\n\\underline{\\partial F_1(x_1)} \\oplus (- R) \\underline{\\partial F_0(x_0)}\n\\label{def:tau_mgprox1}\n\\\\\n","start":38319,"end":39237}]},{"id":"S029","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"校正集合的次微分展开","source_units":[{"raw_tex":"&\\overset{\\eqref{ass:F_sum_rule}}{=} \\nabla f_1(x_1) - R \\nabla f_0(x_0) + \\underline{\\partial g_1(x_1)}\n\\oplus \n(- R) \\underline{\\partial g_0(x_0)}.\n\\label{def:tau_mgprox2}\n\\end{align}\n\\end{subequations}\nNote that $\\underline{\\tau_{0 \\rightarrow 1}}$ is a function of two points at two different levels.\nIn \\eqref{def:tau_mgprox1} $\\underline{\\tau_{0 \\rightarrow 1}}$ is the Minkowski sum of two subdifferentials $\\partial F_1(x_1) $ and $-R\\partial F_0(x_0)$ which are generally set-valued.\n","start":39237,"end":39730}]},{"id":"S030","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"置零使限制次微分单值化","source_units":[{"raw_tex":"To obtain a tractable coarse-grid optimization problem (corresponding to line (iv) in Algorithm~\\ref{algo:mgopt}) we need to avoid complications coming from the Minkowski sum, and we do this by modifying the standard restriction (and prolongation) \nby zeroing out columns in $\\overline{R}$ to form $R$ such that the second subdifferential $R \\partial g_0(x_0)$ in \\eqref{def:tau_mgprox2}\nis a singleton vector.\nSimilarly, we  zero out the corresponding rows in $\\overline{P}$ to form $P$ for the coarse correction step, such that non-differentiable fine points are not corrected by the coarse grid.\nThis zeroing out process is adapted to the current point $x_0$, so we call this $R$ an adaptive restriction operator. \nIn other words, the purpose of the adaptive restriction is to reduce a generally set-valued subdifferential $R\\partial g_0(x_0)$ to a singleton.\nWe denote the adaptive operator $R$ corresponding to a point $x$ as $R(x)$ and thereby the adaptive restriction of $x$ is denoted as $R(x)x$.\nSometimes we just write $Rx$ if the meaning is clear from the context.\nBased on the above discussion, we now formally define adaptive restriction,\nand we give an example in \\cref{sec:exp}.\n","start":39730,"end":40924}]},{"id":"S031","semantic_type":"definition","kind":"definition","labels":["def:adaptiveR","eg:adaptiveR_L1"],"declared_external":false,"source_only":false,"title":"自适应限制算子","source_units":[{"raw_tex":"\n\n\\begin{definition}[Adaptive restriction operator for separable $g$]\\label{eg:adaptiveR_L1}\nFor a possibly nonsmooth function $g : \\IRn \\rightarrow \\IR$ that is separable, i.e., with $x = [x_1,x_2,\\dots,x_n]$, $g(x) = \\sum g_i(x_i)$ where $g_i$ is a function only of component $x_i$, \ngiven a full restriction operator $\\overline{R}$ and a vector $x$, the adaptive restriction operator $R$ with respect to a function $g$ at $x$ is defined by zeroing out the columns of $\\overline{R}$ corresponding to the elements in $\\partial g$ that are set-valued.\n\\label{def:adaptiveR}\n\\end{definition}\n","start":40924,"end":41515}]},{"id":"S032","semantic_type":"definition","kind":"definition","labels":["prob:minfg_coarse"],"declared_external":false,"source_only":false,"title":"带校正项的粗目标","source_units":[{"raw_tex":"\\subsection{A 2-level \\texorpdfstring{\\MGProx}{}algorithm}\\label{sec:mgprox:subsec:2lv}\nSimilar to the 2-level \\MGOPT method for solving Problem \\eqref{prob:mgopt}, we propose a 2-level \\MGProx method (Algorithm \\ref{aglo:2levelMG_exact}) that solves Problem \\eqref{prob:minfg} by utilizing a coarse problem defined as \n\\begin{equation}\\label{prob:minfg_coarse}\n\\argmin_{\\xi \\in \\IR^{n_1}}\\,\n\\bigg\\{\\,\nF^{\\tau}_1 (\\xi) \n~\\coloneqq~  F_1(\\xi) - \\langle \\tau_{0 \\rightarrow 1}^{k+1}, \\xi \\rangle\n~=~ f_1 (\\xi) + g_1(\\xi) - \\langle \\tau_{0 \\rightarrow 1}^{k+1}, \\xi \\rangle\n\\,\\bigg\\}.\n\\end{equation}\n\n\n","start":43823,"end":44422}]},{"id":"S033","semantic_type":"algorithm","kind":"algorithm","labels":["aglo:2levelMG_exact"],"declared_external":false,"source_only":false,"title":"完整两层 MGProx 算法","source_units":[{"raw_tex":"\\begin{algorithm}\n\\caption{2-level \\MGProx for an approximate solution of \\eqref{prob:minfg}}\n\\label{aglo:2levelMG_exact}\n\\begin{algorithmic}\n\\STATE{Initialize $x^1_0$, $R$ and $P$}\n\\FOR{$k = 1,2,\\dots$}\n%\n\\STATE{(i)~~~~ $y^{k+1}_0 = \\prox_{\\frac{1}{L_0}g_0} \\Big (x^k_0 - \\frac{1}{L_0} \\nabla f(x^k_0) \\Big)$\n\\hfill level-0 proximal gradient step \\vspace{0.25mm}\n}\n\\STATE{(ii)~~~ $y^{k+1}_1  = R(y^{k+1}_0) y^{k+1}_0 $   \n\\hfill construct the level-1 coarse variable \\vspace{0.25mm}\n}\n\\STATE{(iii)~~ $\\tau_{0 \\rightarrow 1}^{k+1} \\hspace{-1mm} \\in\\,\n\\underline{\\partial F_1(y^{k+1}_1 )}  - R(y^{k+1}_0) \\, \\underline{\\partial  F_0(y^{k+1}_0)}$\n\\hfill construct the tau vector \\vspace{0.25mm}\n}\n\\STATE{(iv)~~ $\nx^{k+1}_1\n= \\underset{\\xi}{\\argmin} \\, \\Big\\{ F_1^{\\tau}(\\xi) \\coloneqq F_1(\\xi) - \\langle \\tau_{0 \\rightarrow 1}^{k+1}, \\xi \\rangle \\Big\\}\n$\n\\hfill solve the level-1 coarse problem\n}\n\\STATE{(v)\\,~~~ $z^{k+1}_0 = y^{k+1}_0  + \\alpha P\\big(  x^{k+1}_1 - y^{k+1}_1 \\big)$\n\\hfill coarse correction \\vspace{0.25mm}\n}\n\\STATE{(vi)~~\\, $x^{k+1}_0 = \\prox_{\\frac{1}{L_0}g_0} \\Big (z^{k+1}_0 - \\frac{1}{L_0} \\nabla f(z^{k+1}_0) \\Big)$\n\\hfill level-0 proximal gradient step~~\n}\n%\n\\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}\n","start":44422,"end":45645}]},{"id":"S034","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"平滑步长与校正选择","source_units":[{"raw_tex":"\nHere are remarks for the steps in Algorithm~\\ref{aglo:2levelMG_exact}.\n\\begin{itemize}\n    \\item (i): we perform one or more proximal gradient iterations on the fine variable with a constant stepsize $1/L_0$, where $L_0$ is the Lipschitz constant of $\\nabla f_0$.\n    \\item (iii): we pick a value within the set to define $\\tau$;\n    as we are now using adaptive $R$, we use $+$ instead of $\\oplus$ in the expression of $\\tau$.\n    \\item (v): $\\alpha > 0$ is a stepsize; for its selection see \\cref{sec:mgprox:subsec:alpha}.\n","start":45645,"end":46171}]},{"id":"S035","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"非可微位置平移至零","source_units":[{"raw_tex":"    \\item The restriction for variable $y_0^{k+1}$ and the restriction for the subdifferential $ \\underline{\\partial  F_0(y^{k+1}_0)}$ can be slightly different.\n    On $y_0^{k+1}$ the restriction is the full restriction, on $ \\underline{\\partial  F_0(y^{k+1}_0)}$ is the adaptive one. \n    The explanation is as follows.   \n    For the particular cases of $g_0$ such as $\\ell_1$, $\\max\\{ \\cdot, 0\\}$ (element-wise maximum) and $\\iota_{[0,\\infty)}$ (indicator of nonnegative orthant), when we zero out column $i$ of $R$, the corresponding entry of $y_0^{k+1}$ is already $0$.    \n    Note that this conclusion does not hold in general for other nonsmooth functions or when the non-differentiability occur at another point (say at $x=1$). \n    Generally in those cases we will need to specify which restriction matrix (the full $R$ or the adaptive $R$) to be used to define $P$.\n        However, to simplify the presentation we always assume the non-differentiability occurs at $x=0$,\n         by performing a translation to shift the non-differentiability to occur at $x=0$.\n%        In fact, this is what exact we do in the experiment.\n\\end{itemize}\n","start":46171,"end":47322}]},{"id":"S036","semantic_type":"theorem","kind":"theorem","labels":["iq:descent_dir","thm:coarse_dir_descent"],"declared_external":false,"source_only":false,"title":"粗校正的严格钝角条件","source_units":[{"raw_tex":"\\begin{theorem}[Angle condition of coarse correction]\\label{thm:coarse_dir_descent}\nIf $P(x^{k+1}_1 - y^{k+1}_1) \\neq 0$, then\n\\begin{equation}\\label{iq:descent_dir}\n\\big\\langle \n\\underline{\\partial F_0(y^{k+1}_0)} , P(x^{k+1}_1 - y^{k+1}_1) \n\\big\\rangle \n%\\leq \n<\n0. \n\\end{equation}\n%I.e., $P(x^{k+1}_1 - y^{k+1}_1) $ and any subgradient in $\\partial F_0(y^{k+1}_0)$ forms a strict obtuse angle.\n\\end{theorem}","start":50904,"end":51314}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":["pf:descent_to_prove"],"declared_external":false,"source_only":false,"title":"将角条件化为粗空间内积","source_units":[{"raw_tex":"Before we prove the theorem we emphasize that \\eqref{iq:descent_dir} applies for any subgradient in the set $\\underline{\\partial F_0(y^{k+1}_0)} $.\nFurthermore,\n\\[\n\\eqref{iq:descent_dir} \n%\n\\iff\n%\n\\big\\langle \nP^\\top \\underline{\\partial F_0(y^{k+1}_0)} , x^{k+1}_1 - y^{k+1}_1 \n\\big\\rangle\n< 0\n%\n\\overset{P^\\top = cR}{\\iff}\n%\nc \\big\\langle \nR \\underline{\\partial F_0(y^{k+1}_0)} , x^{k+1}_1 - y^{k+1}_1 \n\\big\\rangle\n< 0\n.\n\\]\nAs $c>0$, showing \\eqref{iq:descent_dir} is equivalent to showing\n\\begin{equation}\n\\big\\langle R \\underline{\\partial F_0(y^{k+1}_0)},  ~ x^{k+1}_1 - y^{k+1}_1 \\big\\rangle < 0,\n\\label{pf:descent_to_prove}\n\\end{equation}\nwhere $R \\underline{\\partial F_0(y^{k+1}_0)} $ is a singleton vector for all subgradients in $\\underline{\\partial F_0(y^{k+1}_0)}$ due to the adaptive $R$.\n","start":51315,"end":52115}]},{"id":"S038","semantic_type":"proof","kind":"proof","labels":["pf:descent_to_prove"],"declared_external":false,"source_only":false,"title":"粗校正角条件的完整证明","source_units":[{"raw_tex":"Before we prove the theorem we emphasize that \\eqref{iq:descent_dir} applies for any subgradient in the set $\\underline{\\partial F_0(y^{k+1}_0)} $.\nFurthermore,\n\\[\n\\eqref{iq:descent_dir} \n%\n\\iff\n%\n\\big\\langle \nP^\\top \\underline{\\partial F_0(y^{k+1}_0)} , x^{k+1}_1 - y^{k+1}_1 \n\\big\\rangle\n< 0\n%\n\\overset{P^\\top = cR}{\\iff}\n%\nc \\big\\langle \nR \\underline{\\partial F_0(y^{k+1}_0)} , x^{k+1}_1 - y^{k+1}_1 \n\\big\\rangle\n< 0\n.\n\\]\nAs $c>0$, showing \\eqref{iq:descent_dir} is equivalent to showing\n\\begin{equation}\n\\big\\langle R \\underline{\\partial F_0(y^{k+1}_0)},  ~ x^{k+1}_1 - y^{k+1}_1 \\big\\rangle < 0,\n\\label{pf:descent_to_prove}\n\\end{equation}\nwhere $R \\underline{\\partial F_0(y^{k+1}_0)} $ is a singleton vector for all subgradients in $\\underline{\\partial F_0(y^{k+1}_0)}$ due to the adaptive $R$.\n","start":51315,"end":52115},{"raw_tex":"\\begin{proof}\nBy definition \n$\\tau_{0 \\rightarrow 1}^{k+1} \\overset{\\eqref{def:tau_mgprox1}}{\\in} \\underline{\\partial F_1(y_1^{k+1})}  - R \\underline{\\partial F_0(y_0^{k+1})}$ \nand the fact that $R \\underline{\\partial F_0(y^{k+1}_0)} $ is a singleton, we have\n$\nR \\underline{\\partial F_0(y_0^{k+1})} \n\\in\n\\underline{\\partial F_1(y_1^{k+1})}  - \\tau_{0 \\rightarrow 1}^{k+1}\n\\overset{\\eqref{prob:minfg_coarse}}{=} \\underline{\\partial F_1^\\tau(y_1^{k+1})}$,\nshowing that $R \\underline{\\partial F_0(y_0^{k+1})} $ is a subgradient of $F_1^{\\tau}$ at $y_1^{k+1}$.\nFor any subgradient in the subdifferential $\\underline{\\partial F_1^\\tau(y_1^{k+1})} $, we have the following which implies \\eqref{pf:descent_to_prove}:\n$\n\\big\\langle \\underline{\\partial F_1^{\\tau}(y^{k+1}_1)}, x^{k+1}_1 - y^{k+1}_1 \\big\\rangle\n< F_1^{\\tau}(x^{k+1}_1) - F_1^{\\tau}(y^{k+1}_1) \n< 0,\n$\nwhere the first strict inequality is due to $F_1^\\tau$ being a strongly convex function (which implies strict convexity);\nthe second inequality is by $x^{k+1}_1 \\coloneqq \\underset{\\xi}{\\argmin}\\,\nF_1^{\\tau}(\\xi)$ and the assumption that $x_1^{k+1} \\neq y_1^{k+1}$.\n\\end{proof}","start":52115,"end":53251}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"识别粗目标的一个次梯度","source_units":[{"raw_tex":"\\begin{proof}\nBy definition \n$\\tau_{0 \\rightarrow 1}^{k+1} \\overset{\\eqref{def:tau_mgprox1}}{\\in} \\underline{\\partial F_1(y_1^{k+1})}  - R \\underline{\\partial F_0(y_0^{k+1})}$ \nand the fact that $R \\underline{\\partial F_0(y^{k+1}_0)} $ is a singleton, we have\n$\nR \\underline{\\partial F_0(y_0^{k+1})} \n\\in\n\\underline{\\partial F_1(y_1^{k+1})}  - \\tau_{0 \\rightarrow 1}^{k+1}\n\\overset{\\eqref{prob:minfg_coarse}}{=} \\underline{\\partial F_1^\\tau(y_1^{k+1})}$,\nshowing that $R \\underline{\\partial F_0(y_0^{k+1})} $ is a subgradient of $F_1^{\\tau}$ at $y_1^{k+1}$.\n","start":52115,"end":52673}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"强凸粗目标给出严格负内积","source_units":[{"raw_tex":"For any subgradient in the subdifferential $\\underline{\\partial F_1^\\tau(y_1^{k+1})} $, we have the following which implies \\eqref{pf:descent_to_prove}:\n$\n\\big\\langle \\underline{\\partial F_1^{\\tau}(y^{k+1}_1)}, x^{k+1}_1 - y^{k+1}_1 \\big\\rangle\n< F_1^{\\tau}(x^{k+1}_1) - F_1^{\\tau}(y^{k+1}_1) \n< 0,\n$\nwhere the first strict inequality is due to $F_1^\\tau$ being a strongly convex function (which implies strict convexity);\nthe second inequality is by $x^{k+1}_1 \\coloneqq \\underset{\\xi}{\\argmin}\\,\nF_1^{\\tau}(\\xi)$ and the assumption that $x_1^{k+1} \\neq y_1^{k+1}$.\n\\end{proof}","start":52673,"end":53251}]},{"id":"S041","semantic_type":"definition","kind":"definition","labels":["iq:coarse_descent_on_F"],"declared_external":false,"source_only":false,"title":"粗校正的目标下降条件","source_units":[{"raw_tex":"\\subsubsection{Existence of coarse correction stepsize \\texorpdfstring{$\\alpha_k$}{}}\n\\label{sec:mgprox:subsec:alpha}\nBased on Theorem~\\ref{thm:coarse_dir_descent}, we now show that there exists a stepsize $\\alpha_k > 0$ such that \n\\begin{equation}\nF_0(z^{k+1}_0) \n~\\coloneqq~\nF_0\\big(\ny^{k+1}_0 + \\alpha_k P(x^{k+1}_1 - y^{k+1}_1) \n\\big)\n~<~\nF_0(y^{k+1}_0).\n\\label{iq:coarse_descent_on_F}\n\\end{equation}\n","start":53436,"end":53841}]},{"id":"S042","semantic_type":"lemma","kind":"lemma","labels":["lemma:alphaexists"],"declared_external":false,"source_only":false,"title":"正粗校正步长存在","source_units":[{"raw_tex":"\\begin{lemma}[Existence of stepsize]\\label{lemma:alphaexists}\nThere exists $\\alpha_k > 0$ such that \\eqref{iq:coarse_descent_on_F} is satisfied for $P(x^{k+1}_1 - y^{k+1}_1) \\neq 0$.\n\\end{lemma}","start":53842,"end":54036}]},{"id":"S043","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单中的有限凸函数次微分紧性及方向导数支撑函数表征；原文引用 Hiriart-Urruty–Lemarechal (2004), Def. 1.1.4, p.165。"],"source_only":false,"title":"紧次微分与方向导数","source_units":[{"raw_tex":"To prove the lemma, we make use \nof a fact of the subdifferentials of finite convex functions \\cite[Def.1.1.4, p.165]{hiriart2004fundamentals}: $\\partial g(x)$ is a nonempty compact convex set $\\cS \\subset \\IRn$ whose support function $\\displaystyle \\sup_{s}\\{ \\langle s, x\\rangle \\,|\\, s \\in \\cS  \\}$ \nis the directional derivative of $g$ at $x$.\n","start":54038,"end":54386}]},{"id":"S044","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"细目标次微分的紧性应用","source_units":[{"raw_tex":"By this, the subdifferential\n $\\underline{\\partial F_0(y_0^{k+1})}$ is a compact convex set whose support function is the directional derivative of $F_0$ at $y_0^{k+1}$.\nWe emphasize that here \n%$F_0 : \\IR^{n_0} \\rightarrow  \\overline{\\IR}$ at $y_0^{k+1}$ is \n$F_0(y_0^{k+1})$ is \nfinite, \n% i.e., $F_0 : \\IR^{n_0} \\rightarrow  \\overline{\\IR}$ will never reach $+\\infty$ at $y_0^{k+1}$ since $y_0^{k+1}$ is obtained by the proximal gradient step (in other words, $y_0^{k+1}$ is a ``feasible point'' if $g_0$ is an indicator function of a constraint set),\nso we can make use of the result on directional derivative in \\cite[Def. 1.1.4, p.165]{hiriart2004fundamentals}, which only applies for finite convex functions, associated with the subdifferential.\n","start":54386,"end":55139}]},{"id":"S045","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"步长存在引理的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nWe prove the lemma in 3 steps.\n\\begin{enumerate}\n\n\\item (Half-space)\nThe strict inequality in Theorem~\\ref{thm:coarse_dir_descent} means that $\\underline{\\partial F_0(y_0^{k+1})}$ is strictly inside a half-space with normal vector $p = P(x^{k+1}_1 - y^{k+1}_1)$.  \n\n\\item (Strict separation)\nBeing a compact convex set, $\\underline{\\partial F_0(y_0^{k+1}0)}$ lying strictly on one side of the hyperplane must be a positive distance (say $\\alpha_k > 0$) from that hyperplane.\n\n\\item (Support and directional derivative)\nEvaluating the support function of $\\underline{\\partial F_0(y_0^{k+1})}$, i.e., the directional derivative of $F_0$ at $y_0^{k+1}$ in the direction $p$, we have \\eqref{iq:coarse_descent_on_F}.\n\\end{enumerate}\n\\end{proof}","start":55140,"end":55893}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"全部次梯度落在严格半空间","source_units":[{"raw_tex":"\\begin{proof}\nWe prove the lemma in 3 steps.\n\\begin{enumerate}\n\n\\item (Half-space)\nThe strict inequality in Theorem~\\ref{thm:coarse_dir_descent} means that $\\underline{\\partial F_0(y_0^{k+1})}$ is strictly inside a half-space with normal vector $p = P(x^{k+1}_1 - y^{k+1}_1)$.  \n\n","start":55140,"end":55420}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"紧性给出正分离距离","source_units":[{"raw_tex":"\\item (Strict separation)\nBeing a compact convex set, $\\underline{\\partial F_0(y_0^{k+1}0)}$ lying strictly on one side of the hyperplane must be a positive distance (say $\\alpha_k > 0$) from that hyperplane.\n\n","start":55420,"end":55630}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"负方向导数产生下降步长","source_units":[{"raw_tex":"\\item (Support and directional derivative)\nEvaluating the support function of $\\underline{\\partial F_0(y_0^{k+1})}$, i.e., the directional derivative of $F_0$ at $y_0^{k+1}$ in the direction $p$, we have \\eqref{iq:coarse_descent_on_F}.\n\\end{enumerate}\n\\end{proof}","start":55630,"end":55893}]},{"id":"S049","semantic_type":"lemma","kind":"lemma","labels":["eqn:suffdescentPGD","lemma:suff_descent_proxgrad"],"declared_external":true,"declaration_text":["原始前提清单明确许可 B0071，来源为 Beck (2017), Lemma 10.4；本文没有提供其证明。"],"source_only":false,"title":"近端梯度的充分下降前提","source_units":[{"raw_tex":"\\begin{lemma}\n(Sufficient descent property of proximal gradient \\cite[Lemma 10.4]{beck2017first})\n\\label{lemma:suff_descent_proxgrad} \nLet $L_0$ be the Lipschitz constant of $\\nabla f_0$.\nFor step (i) in Algorithm~\\ref{aglo:2levelMG_exact},  we have\n\\begin{equation}\nF_0( y^{k+1}_0)\n\\leq \nF_0( x^k_0) - \\| G_0(x_0^k) \\|_2^2 / (2L_0),\n~~\nG_0( x_0^k) \n=\nL_0 \\big[\nx_0^k - \\prox_{ \\frac{1}{L} g_0 }\\big( x_0^k - \\nabla f_0(x_0^k)/L_0\n\\big)\n\\big].\n\\label{eqn:suffdescentPGD}\n\\end{equation}\n\\end{lemma}\n$G_0( x_0^k)$ is called the proximal gradient map of $F_0$ at $x_0^k$.\nThe inequality also holds for step (vi).\n","start":58297,"end":58907}]},{"id":"S050","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"近端梯度映射","source_units":[{"raw_tex":"G_0( x_0^k) \n=\nL_0 \\big[\nx_0^k - \\prox_{ \\frac{1}{L} g_0 }\\big( x_0^k - \\nabla f_0(x_0^k)/L_0\n\\big)\n\\big].\n\\label{eqn:suffdescentPGD}\n\\end{equation}\n\\end{lemma}\n$G_0( x_0^k)$ is called the proximal gradient map of $F_0$ at $x_0^k$.\n","start":58634,"end":58866}]},{"id":"S051","semantic_type":"algorithm","kind":"algorithm","labels":["aglo:naivelinesearch"],"declared_external":false,"source_only":false,"title":"完整朴素回溯线搜索","source_units":[{"raw_tex":"\\begin{algorithm}\n\\caption{Naive line search}\\label{aglo:naivelinesearch}\n\\begin{algorithmic}\n\\STATE{Set $\\alpha > 0$ and select a tolerance $\\epsilon > 0$  (e.g. $10^{-15}$)}\n\\WHILE{true}\n\\STATE{\\textbf{If} $F_0\\big( y^{k+1}_0 + \\alpha P (x^{k+1}_1 - y^{k+1}_1) \\big) \\leq F_0( y^{k+1}_0 )$ \\textbf{ then }\nreturn $z_0^{k+1}=y^{k+1}_0 + \\alpha P(x^{k+1}_1 - y^{k+1}_1)$.}\n\\\\\n\\STATE{\\textbf{else if} $\\alpha > \\epsilon $ \\textbf{then} $\\alpha = \\alpha/2 $.}\n\\\\\n\\STATE{~~~~~~~\\textbf{else} return $z_0^{k+1}=y^{k+1}_0$.}\n\\ENDWHILE\n\\end{algorithmic}\n\\end{algorithm}\n","start":61746,"end":62310}]},{"id":"S052","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"线性收敛段的细层简写","source_units":[{"raw_tex":"\\subsubsection{Linear convergence rate}\nAll the functions and variables here are at level 0 so we omit the subscripts.\nNow we show that $\\big\\{F( x^k) \\big\\}_{k \\in \\IN}$ converges to $F^*$ with a linear rate using the \\emph{Proximal Polyak-{\\L}ojasiewicz inequality} \\cite[Section 4]{karimi2016linear}. \n","start":76605,"end":76910}]},{"id":"S053","semantic_type":"definition","kind":"definition","labels":["def:ProxPL"],"declared_external":false,"source_only":false,"title":"近端 PL 不等式","source_units":[{"raw_tex":"The function $F$ in Problem \\eqref{prob:minfg} is called ProxPL, if there exists $\\mu > 0$, called the ProxPL constant, such that \n\\begin{equation}\n\\cD_g(x,L)\n~\\geq~\n2 \\mu \\big( F(x) - F^* \\big)  \\qquad  \\forall x,\n\\tag{ProxPL}\n\\label{def:ProxPL}\n\\end{equation}\n","start":76910,"end":77172}]},{"id":"S054","semantic_type":"definition","kind":"definition","labels":["def:dg"],"declared_external":false,"source_only":false,"title":"近端 PL 辅助量","source_units":[{"raw_tex":"\\begin{equation}\n\\cD_g(x,\\alpha) \n\\coloneqq\n-2\\alpha \\min_z \\,\n\\Big\\{\n{\\textstyle\\frac{\\alpha}{2}} \\| z - x \\|_2^2 \n+ \\big\\langle z - x, \\nabla f(x) \\big\\rangle\n+ g(z) - g(x)\n\\Big\\}\n.\n\\label{def:dg}\n\\end{equation}\n","start":77172,"end":77386}]},{"id":"S055","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"辅助量与近端子问题的对应","source_units":[{"raw_tex":"Intuitively, $\\cD_g$ is defined based on the proximal gradient operator:\n\\[\n\\prox_{\\frac{1}{L}g} \\Big( x - \\frac{\\nabla f(x)}{L} \\Big)\n=\n\\underset{z}{\\argmin} \\, \\frac{L}{2} \\| z -  x \\|_2^2\n+ \\big\\langle z - x, \\nabla f(x) \\big\\rangle\n+ g(z) - g(x).\n\\]\n","start":77386,"end":77640}]},{"id":"S056","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单中的强凸光滑分量蕴含同模量 ProxPL；来源为 Karimi et al. (2016), Section 4，第1822行指定章节、第1851行明确调用。"],"source_only":false,"title":"光滑项强凸蕴含近端 PL","source_units":[{"raw_tex":"It has been shown in \\cite{karimi2016linear} that if $f$ in \\eqref{prob:minfg} is $\\mu$-strongly convex, then $F$ is $\\mu$-ProxPL.\n","start":77640,"end":77771}]},{"id":"S057","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"交错更新需单独分析","source_units":[{"raw_tex":"Now we prove the linear convergence rate of Algorithm~\\ref{aglo:2levelMG_exact}.\nNote that a standard result such as \\cite[Theorem 10.29]{beck2017first} on convergence of proximal gradient for strongly convex functions is not directly applicable because, as mentioned above, we interleave proximal gradient steps with coarse correction steps.\n","start":77771,"end":78114}]},{"id":"S058","semantic_type":"theorem","kind":"theorem","labels":["thm:mgprox_converge_proxPL"],"declared_external":false,"source_only":false,"title":"两层 MGProx 的线性收敛界","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm:mgprox_converge_proxPL}\nLet $x^1_0$ be the initial guess of the algorithm,\n$x_0^* = \\argmin \\, F_0(x)$ and $F^*_0 \\coloneqq F_0(x_0^*)$.\nThe sequence $\\{x_0^k\\}_{k \\in \\IN}$ generated by \\MGProx (Algorithm~\\ref{aglo:2levelMG_exact}) for solving Problem \\eqref{prob:minfg} satisfies\n$\nF_0(x_0^{k+1}) - F_0^* \n\\leq \n\\Big( 1-\\frac{\\mu_0}{L_0} \\Big)^k \\big( F_0(x_0^1) - F_0^* \\big).\n$\n\n\n","start":78147,"end":78557},{"raw_tex":"\n\\end{theorem}","start":79949,"end":79963}]},{"id":"S059","semantic_type":"proof","kind":"proof","labels":["thm:mgprox_converge_proxPL"],"declared_external":false,"source_only":false,"title":"线性收敛定理的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nFirst, by assumption $F$ is strongly convex, so $F$ is $\\mu$-ProxP\\L\\, with $\\mu > 0$ and\n\\[\n\\renewcommand*{\\arraystretch}{1.33}\n\\begin{array}{rcl}\nF(x^{k+1})\n&\\overset{\\eqref{eqn:suffdescentPGD}}{\\leq}& \\hspace{-2mm}\nF(z^{k+1})\n~\\overset{\\eqref{iq:coarse_descent_on_F}}{\\leq}~ \nF(y^{k+1})  \n~=~\nf(y^{k+1}) + g(y^{k+1}) + g(x^k) - g(x^k)\n\\\\\n&\\overset{\\eqref{def:Lsmooth}}{\\leq}&  \\hspace{-2mm}\nf(x^k) +  \\langle  \\nabla f(x^k), y^{k+1} - x^k \\rangle + \\frac{L}{2} \\|  y^{k+1} - x^k \\|_2^2 + g(x^k) +  g(y^{k+1})  -  g(x^k) \t\n\\\\\n&=& \\hspace{-2mm}\nF(x^k) +  \\langle  \\nabla f(x^k), y^{k+1} - x^k \\rangle + \\frac{L}{2} \\|  y^{k+1} - x^k \\|_2^2 +  g(y^{k+1})  -  g(x^k) \n\\\\\n&=& \\hspace{-2mm}\nF(x^k) - \\frac{1}{2L} \\underbrace{(-2L)\\Big( \n\\langle  \\nabla f(x^k), y^{k+1} - x^k \\rangle \n+ {\\textstyle \\frac{L}{2}} \\|  y^{k+1} - x^k \\|_2^2 +  g(y^{k+1})  -  g(x^k) \n\\Big)}_{\\overset{\\eqref{def:dg}}{ = } \\, \\cD_g(x^k, L), \\, \\text{ since }y^{k+1} \\,\\coloneqq\\, \\underset{z}{\\argmin} \\, \\frac{L}{2} \\| z -  x^k \\|_2^2\n+ \\langle z - x^k, \\nabla f(x^k)\\rangle\n+ g(z) - g(x^k).}\n\\\\\n&\\overset{\\eqref{def:ProxPL}}{\\leq}& \\hspace{-2mm}\nF(x^k) - \\frac{\\mu}{L} \\big(F(x^k) - F^*\\big).\n\\end{array}\n\\]\nAdding $-F^*$ on both sides of the inequality gives\n$ \nF(x^{k+1}) - F^* \n\\leq \\big( 1 - \\mu/L\\big) \\big(F(x^k) - F^*\\big)\n$.\nApplying this inequality recursively completes the proof.\n\\end{proof}","start":78557,"end":79949}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":["thm:mgprox_converge_proxPL"],"declared_external":false,"source_only":false,"title":"引入细层近端 PL 性质","source_units":[{"raw_tex":"\\begin{proof}\nFirst, by assumption $F$ is strongly convex, so $F$ is $\\mu$-ProxP\\L\\, with $\\mu > 0$ and\n","start":78557,"end":78661}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":["thm:mgprox_converge_proxPL"],"declared_external":false,"source_only":false,"title":"组合后平滑与粗校正非增","source_units":[{"raw_tex":"\\[\n\\renewcommand*{\\arraystretch}{1.33}\n\\begin{array}{rcl}\nF(x^{k+1})\n&\\overset{\\eqref{eqn:suffdescentPGD}}{\\leq}& \\hspace{-2mm}\nF(z^{k+1})\n~\\overset{\\eqref{iq:coarse_descent_on_F}}{\\leq}~ \nF(y^{k+1})  \n~=~\nf(y^{k+1}) + g(y^{k+1}) + g(x^k) - g(x^k)\n","start":78661,"end":78909}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":["thm:mgprox_converge_proxPL"],"declared_external":false,"source_only":false,"title":"用光滑性建立模型上界","source_units":[{"raw_tex":"\\\\\n&\\overset{\\eqref{def:Lsmooth}}{\\leq}&  \\hspace{-2mm}\nf(x^k) +  \\langle  \\nabla f(x^k), y^{k+1} - x^k \\rangle + \\frac{L}{2} \\|  y^{k+1} - x^k \\|_2^2 + g(x^k) +  g(y^{k+1})  -  g(x^k) \t\n\\\\\n&=& \\hspace{-2mm}\nF(x^k) +  \\langle  \\nabla f(x^k), y^{k+1} - x^k \\rangle + \\frac{L}{2} \\|  y^{k+1} - x^k \\|_2^2 +  g(y^{k+1})  -  g(x^k) \n\\\\\n","start":78909,"end":79241}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":["thm:mgprox_converge_proxPL"],"declared_external":false,"source_only":false,"title":"识别最小模型值为辅助量","source_units":[{"raw_tex":"&=& \\hspace{-2mm}\nF(x^k) - \\frac{1}{2L} \\underbrace{(-2L)\\Big( \n\\langle  \\nabla f(x^k), y^{k+1} - x^k \\rangle \n+ {\\textstyle \\frac{L}{2}} \\|  y^{k+1} - x^k \\|_2^2 +  g(y^{k+1})  -  g(x^k) \n\\Big)}_{\\overset{\\eqref{def:dg}}{ = } \\, \\cD_g(x^k, L), \\, \\text{ since }y^{k+1} \\,\\coloneqq\\, \\underset{z}{\\argmin} \\, \\frac{L}{2} \\| z -  x^k \\|_2^2\n+ \\langle z - x^k, \\nabla f(x^k)\\rangle\n+ g(z) - g(x^k).}\n\\\\\n","start":79241,"end":79642}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":["thm:mgprox_converge_proxPL"],"declared_external":false,"source_only":false,"title":"应用近端 PL 控制误差","source_units":[{"raw_tex":"&\\overset{\\eqref{def:ProxPL}}{\\leq}& \\hspace{-2mm}\nF(x^k) - \\frac{\\mu}{L} \\big(F(x^k) - F^*\\big).\n\\end{array}\n\\]\n","start":79642,"end":79755}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":["thm:mgprox_converge_proxPL"],"declared_external":false,"source_only":false,"title":"得到单步误差递推","source_units":[{"raw_tex":"Adding $-F^*$ on both sides of the inequality gives\n$ \nF(x^{k+1}) - F^* \n\\leq \\big( 1 - \\mu/L\\big) \\big(F(x^k) - F^*\\big)\n$.\n","start":79755,"end":79880}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":["thm:mgprox_converge_proxPL"],"declared_external":false,"source_only":false,"title":"递推得到几何误差界","source_units":[{"raw_tex":"Applying this inequality recursively completes the proof.\n\\end{proof}","start":79880,"end":79949}]},{"id":"S067","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":true,"title":"收缩因子的参数范围","source_units":[{"raw_tex":"\\item Convergence rate: for a $\\mu_0$-strongly convex and $L_0$-smooth $f_0$, we have $0 < \\mu_0 \\leq L_0$ and $0 \\leq 1 - \\mu_0/L_0 < 1$.\n","start":80042,"end":80181}]},{"id":"S068","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单中的强凸性与次微分强单调性等价；原文第792行引用 Bauschke–Combettes。"],"source_only":false,"title":"强凸性与次微分强单调性等价","source_units":[]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单中的凸次梯度最优性条件和强凸次梯度不等式。"],"source_only":false,"title":"强凸次梯度不等式与最优性","source_units":[]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；原文第820—823行使用但未另列外部来源。"],"source_only":false,"title":"满列秩的奇异值下界","source_units":[]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；对应第1869行对 F 强凸性的假设性调用。"],"source_only":false,"title":"强凸项加凸项保持模量","source_units":[]},{"id":"S072","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；原文第1175行直接调用 F_1^τ 的强凸性。"],"source_only":false,"title":"线性扰动保持强凸模量","source_units":[]},{"id":"S073","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；对应第1254行从支撑函数与方向导数直接跳至下降式的标准过渡。"],"source_only":false,"title":"负方向导数蕴含局部下降","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S058","edges":[{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S003","kind":"supports"},{"source_id":"S010","target_id":"S009","kind":"supports"},{"source_id":"S001","target_id":"S011","kind":"supports"},{"source_id":"S012","target_id":"S011","kind":"supports"},{"source_id":"S002","target_id":"S016","kind":"supports"},{"source_id":"S003","target_id":"S016","kind":"supports"},{"source_id":"S012","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S018","kind":"supports"},{"source_id":"S012","target_id":"S019","kind":"supports"},{"source_id":"S068","target_id":"S023","kind":"supports"},{"source_id":"S018","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S070","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S068","target_id":"S026","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S022","kind":"supports"},{"source_id":"S026","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S014","target_id":"S028","kind":"supports"},{"source_id":"S015","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S017","target_id":"S029","kind":"supports"},{"source_id":"S003","target_id":"S030","kind":"supports"},{"source_id":"S014","target_id":"S030","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S011","target_id":"S032","kind":"supports"},{"source_id":"S028","target_id":"S032","kind":"supports"},{"source_id":"S009","target_id":"S033","kind":"supports"},{"source_id":"S010","target_id":"S033","kind":"supports"},{"source_id":"S028","target_id":"S033","kind":"supports"},{"source_id":"S031","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S012","target_id":"S033","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S035","target_id":"S033","kind":"supports"},{"source_id":"S012","target_id":"S037","kind":"supports"},{"source_id":"S030","target_id":"S037","kind":"supports"},{"source_id":"S017","target_id":"S037","kind":"supports"},{"source_id":"S028","target_id":"S039","kind":"supports"},{"source_id":"S030","target_id":"S039","kind":"supports"},{"source_id":"S017","target_id":"S039","kind":"supports"},{"source_id":"S032","target_id":"S039","kind":"supports"},{"source_id":"S021","target_id":"S040","kind":"supports"},{"source_id":"S002","target_id":"S040","kind":"supports"},{"source_id":"S003","target_id":"S040","kind":"supports"},{"source_id":"S011","target_id":"S040","kind":"supports"},{"source_id":"S012","target_id":"S040","kind":"supports"},{"source_id":"S020","target_id":"S040","kind":"supports"},{"source_id":"S032","target_id":"S040","kind":"supports"},{"source_id":"S071","target_id":"S040","kind":"supports"},{"source_id":"S072","target_id":"S040","kind":"supports"},{"source_id":"S033","target_id":"S040","kind":"supports"},{"source_id":"S069","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S033","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S002","target_id":"S044","kind":"supports"},{"source_id":"S003","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S036","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S044","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S043","target_id":"S048","kind":"supports"},{"source_id":"S044","target_id":"S048","kind":"supports"},{"source_id":"S073","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S048","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S042","kind":"supports"},{"source_id":"S010","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S033","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S010","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S056","kind":"supports"},{"source_id":"S001","target_id":"S058","kind":"supports"},{"source_id":"S033","target_id":"S058","kind":"supports"},{"source_id":"S052","target_id":"S060","kind":"supports"},{"source_id":"S002","target_id":"S060","kind":"supports"},{"source_id":"S003","target_id":"S060","kind":"supports"},{"source_id":"S071","target_id":"S060","kind":"supports"},{"source_id":"S056","target_id":"S060","kind":"supports"},{"source_id":"S049","target_id":"S061","kind":"supports"},{"source_id":"S033","target_id":"S061","kind":"supports"},{"source_id":"S042","target_id":"S061","kind":"supports"},{"source_id":"S041","target_id":"S061","kind":"supports"},{"source_id":"S051","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S002","target_id":"S062","kind":"supports"},{"source_id":"S005","target_id":"S062","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S033","target_id":"S063","kind":"supports"},{"source_id":"S054","target_id":"S063","kind":"supports"},{"source_id":"S055","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S060","target_id":"S064","kind":"supports"},{"source_id":"S053","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S002","target_id":"S067","kind":"supports"},{"source_id":"S005","target_id":"S067","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S067","target_id":"S066","kind":"supports"},{"source_id":"S052","target_id":"S066","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S061","target_id":"S059","kind":"supports"},{"source_id":"S062","target_id":"S059","kind":"supports"},{"source_id":"S063","target_id":"S059","kind":"supports"},{"source_id":"S064","target_id":"S059","kind":"supports"},{"source_id":"S065","target_id":"S059","kind":"supports"},{"source_id":"S066","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S058","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S058","edges":[{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S003","kind":"supports"},{"source_id":"S010","target_id":"S009","kind":"supports"},{"source_id":"S001","target_id":"S011","kind":"supports"},{"source_id":"S012","target_id":"S011","kind":"supports"},{"source_id":"S002","target_id":"S016","kind":"supports"},{"source_id":"S003","target_id":"S016","kind":"supports"},{"source_id":"S012","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S018","kind":"supports"},{"source_id":"S012","target_id":"S019","kind":"supports"},{"source_id":"S068","target_id":"S021","kind":"supports"},{"source_id":"S018","target_id":"S021","kind":"supports"},{"source_id":"S070","target_id":"S021","kind":"supports"},{"source_id":"S014","target_id":"S028","kind":"supports"},{"source_id":"S015","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S017","target_id":"S029","kind":"supports"},{"source_id":"S003","target_id":"S030","kind":"supports"},{"source_id":"S014","target_id":"S030","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S011","target_id":"S032","kind":"supports"},{"source_id":"S028","target_id":"S032","kind":"supports"},{"source_id":"S009","target_id":"S033","kind":"supports"},{"source_id":"S010","target_id":"S033","kind":"supports"},{"source_id":"S028","target_id":"S033","kind":"supports"},{"source_id":"S031","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S012","target_id":"S033","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S035","target_id":"S033","kind":"supports"},{"source_id":"S012","target_id":"S036","kind":"supports"},{"source_id":"S030","target_id":"S036","kind":"supports"},{"source_id":"S017","target_id":"S036","kind":"supports"},{"source_id":"S028","target_id":"S036","kind":"supports"},{"source_id":"S032","target_id":"S036","kind":"supports"},{"source_id":"S021","target_id":"S036","kind":"supports"},{"source_id":"S002","target_id":"S036","kind":"supports"},{"source_id":"S003","target_id":"S036","kind":"supports"},{"source_id":"S011","target_id":"S036","kind":"supports"},{"source_id":"S020","target_id":"S036","kind":"supports"},{"source_id":"S071","target_id":"S036","kind":"supports"},{"source_id":"S072","target_id":"S036","kind":"supports"},{"source_id":"S033","target_id":"S036","kind":"supports"},{"source_id":"S069","target_id":"S036","kind":"supports"},{"source_id":"S033","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S002","target_id":"S044","kind":"supports"},{"source_id":"S003","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S036","target_id":"S042","kind":"supports"},{"source_id":"S044","target_id":"S042","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S073","target_id":"S042","kind":"supports"},{"source_id":"S010","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S033","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S010","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S056","kind":"supports"},{"source_id":"S001","target_id":"S058","kind":"supports"},{"source_id":"S033","target_id":"S058","kind":"supports"},{"source_id":"S052","target_id":"S060","kind":"supports"},{"source_id":"S002","target_id":"S060","kind":"supports"},{"source_id":"S003","target_id":"S060","kind":"supports"},{"source_id":"S071","target_id":"S060","kind":"supports"},{"source_id":"S056","target_id":"S060","kind":"supports"},{"source_id":"S049","target_id":"S061","kind":"supports"},{"source_id":"S033","target_id":"S061","kind":"supports"},{"source_id":"S042","target_id":"S061","kind":"supports"},{"source_id":"S041","target_id":"S061","kind":"supports"},{"source_id":"S051","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S002","target_id":"S062","kind":"supports"},{"source_id":"S005","target_id":"S062","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S033","target_id":"S063","kind":"supports"},{"source_id":"S054","target_id":"S063","kind":"supports"},{"source_id":"S055","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S060","target_id":"S064","kind":"supports"},{"source_id":"S053","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S002","target_id":"S067","kind":"supports"},{"source_id":"S005","target_id":"S067","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S067","target_id":"S066","kind":"supports"},{"source_id":"S052","target_id":"S066","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S061","target_id":"S059","kind":"supports"},{"source_id":"S062","target_id":"S059","kind":"supports"},{"source_id":"S063","target_id":"S059","kind":"supports"},{"source_id":"S064","target_id":"S059","kind":"supports"},{"source_id":"S065","target_id":"S059","kind":"supports"},{"source_id":"S066","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S058","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["prob:minfg"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"细层复合优化问题","source_units":[{"raw_tex":"We study the combination of two iterative algorithms: proximal gradient descent and multigrid, to solve the following class of optimization problems\n\\begin{equation}\\label{prob:minfg}\n\\argmin_x ~ F_0(x) \\coloneqq f_0(x) + g_0(x).\n\\end{equation}\n","start":6637,"end":6882}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"光滑项的强凸假设","source_units":[{"raw_tex":"\\textbf{(Assumption)} We assume $f_0 : \\IRn \\rightarrow \\IR$ is \n% lower semi-continuous, % NO need because smoothness implies LSC\n$L_0$-smooth and $\\mu_0$-strongly convex,\nand  \n","start":6882,"end":7061}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"非光滑项的有限性与可分性","source_units":[{"raw_tex":"%$g_0 : \\IRn \\rightarrow \\overline{\\IR} \\coloneqq \\IR \\cup \\{+\\infty\\}$ \n$g_0 : \\IRn \\rightarrow \\IR$ \nis proper, possibly nonsmooth, convex, \n%lower semi-continuous, \nand separable.\nRegarding the non-smoothness (nondifferentiability) of $g_0$, we further assume a single point of non-differentiability for $g_0$.\n","start":7061,"end":7375}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"可分性与强凸模量","source_units":[{"raw_tex":"We recall that a function $\\zeta(x) : \\IRn \\rightarrow \\IR$ is separable if $\\zeta(x) = \\sum_i \\zeta_i(x_i)$, and  is $\\mu$-strongly convex (and thus coercive) with $\\mu>0$ if $\\zeta(x) - \\frac{\\mu}{2}\\| x \\|_2^2$ is convex;\n","start":7375,"end":7600}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":["def:Lsmooth"],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"光滑性与二次上界","source_units":[{"raw_tex":"and lastly is $L$-smooth if  $\\zeta$ is $C^{1,1}_L$ and $\\nabla \\zeta$ is $L$-Lipschitz; i.e., for all $x,y$ in $\\IRn$, $\\nabla \\zeta(x)$ exists and\n\\begin{equation}\n\\zeta(y) \\leq \\zeta(x) + \\big\\langle \\nabla \\zeta(x), y-x \\big\\rangle + \\frac{L}{2}\\|y-x\\|_2^2.\n\\label{def:Lsmooth}\n\\end{equation}\n","start":7600,"end":7897}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"层号与光滑常数记号","source_units":[{"raw_tex":"\\paragraph{Notation}\nThe symbol $x_0$ (or $x$) is called the fine variable.\nThe symbol $x_\\ell$ in \\eqref{prob:mgopt_coarse} with $\\ell \\geq 1$ is called coarse variable.\nThe subscript $\\ell \\in \\{0,1,\\dots, L \\}$ denotes the level.\nA larger $\\ell$ means a coarser level with lower resolution (fewer variables).\n%We use $\\ell=0$ to denote the finest level with the highest resolution and $\\ell = L$ to denote the coarsest level with the lowest resolution.\nFor the remainder of the paper, $L$ without a subscript stands for the number of levels, whereas $L_{\\ell}$ denotes the smoothness parameter for the level-$\\ell$ problem.\n%\n","start":11252,"end":11881}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"限制与延拓矩阵","source_units":[{"raw_tex":"At a level $\\ell$, the coarse version of the vector $x_\\ell \\in \\IR^{n_\\ell}$ is $x_{\\ell+1} = \\overline{\\cR}(x_\\ell) \\coloneqq \\overline{R}x_\\ell$ where $\\overline{R} \\in \\IR^{n_{\\ell+1} \\times n_{\\ell}}$ with $n_{\\ell+1} \\leq n_{\\ell}$ is called a restriction matrix.\nSimilarly, given $x_{\\ell+1} \\in \\IR^{n_{\\ell+1}}$ and a prolongation matrix $\\overline{P} \\in \\IR^{ n_{\\ell} \\times n_{\\ell +1}}$, \nwe obtain the level-$\\ell$ version of $x_{\\ell+1}$ as $x_{\\ell} = \\overline{\\cP}(x_{\\ell+1}) \\coloneqq \\overline{P} x_{\\ell+1}$.\nWe let $\\overline{P} = c \\overline{R}^\\top$ given scaling factor $c>0$ and $\\overline{R}$.\n","start":11881,"end":12504}]},{"id":"S009","semantic_type":"algorithm","kind":"algorithm","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"近端梯度更新","source_units":[{"raw_tex":"The proximal gradient method solves problems of the form \\eqref{prob:minfg} as follows.\nStarting from an initial guess $x^1$, the method updates the variable by a gradient descent step (with a stepsize $\\alpha > 0$) followed by a proximal step associated with $g_0$:\n\\begin{subequations}\\label{eq:5}\n\\begin{align}\nx^{k+1} &\\,= \\prox_{\\alpha g_0} \n\\Big( \nx^k - \\alpha \\nabla f_0(x^k)\n\\Big), \n%~~ \\alpha > 0\n\\label{updt:proxgrad}\n\\\\\n","start":18761,"end":19192}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"近端算子定义","source_units":[{"raw_tex":"\\prox_{\\alpha g_0}(x) \n& \\coloneqq \n\\underset{u}{\\argmin} ~ \\Big\\{\\, \\alpha g_0(u) + \\frac{1}{2} \\| u - x \\|_2^2 \\,\\Big\\}\n= \\underset{u}{\\argmin} ~ \\Big\\{\\, g_0(u) + \\frac{1}{2\\alpha} \\| u - x \\|_2^2 \\,\\Big\\}.\n\\label{def:prox_opertor}\n\\end{align}\n\\end{subequations}\n","start":19192,"end":19458}]},{"id":"S011","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"粗层函数记号","source_units":[{"raw_tex":"\\subsection{Functions at different levels}\\label{sec:mgprox:subsec:RP}\nFollowing \\cref{sec:intro:subsec:mgopt}, \nwe use $f_\\ell : \\IR^{n_\\ell} \\rightarrow \\IR$ to denote functions at different coarse levels.\nIn this section we will focus on $\\ell \\in \\{0,1\\}$ but we remark that all the notations and definitions are generalized to $\\ell \\in \\{0,1,\\dots,L\\}$ in Section~\\ref{sec:multilevel}.\nWe denote the restriction of the fine objective function $F_0$ in \\eqref{prob:minfg} as\n$ F_1 \\coloneqq \\cR(F_0)  = \\cR(f_0 + g_0)$, where $\\cR$ is defined below.\n\n","start":30220,"end":30776}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["def:restriction"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"函数限制的作者定义","source_units":[{"raw_tex":"\\begin{definition}[Restriction]\\label{def:restriction}\nAt a level $\\ell \\in \\IN$, given a function $f_\\ell : \\IR^{n_\\ell} \\rightarrow \\IR$, the restriction $\\cR$ of $f_\\ell$, denoted as $f_{\\ell+1} \\coloneqq \\cR(f_\\ell)$, is defined as \n$f_{\\ell+1}(x_{\\ell+1}) \\coloneqq f_{\\ell}(R x_{\\ell})$, \nwhere $R : \\IR^{n_\\ell} \\rightarrow \\IR^{n_{\\ell+1}}$ is a restriction matrix .\nWe also define the associated prolongation matrix \n$P : \\IR^{n_{\\ell+1}} \\rightarrow \\IR^{n_\\ell}$ \n as $P = cR^\\top$ where $c>0$ is a predefined constant.\n\\end{definition}\n","start":30776,"end":31324}]},{"id":"S013","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"自适应与固定算子的区分","source_units":[{"raw_tex":"\n\\paragraph{Adaptive restriction and non-adaptive restriction}\nWe recall that a contribution of this work is the introduction of the adaptive restriction, to be discussed in \\cref{sec:mgprox:subsec:adaptiveR} (see Definition\\,\\ref{eg:adaptiveR_L1}).\nTo differentiate the classical non-adaptive restriction (and the associated prolongation) from the adaptive version, we denote the non-adaptive restriction by $\\overline{\\cR}, \\overline{R},\\overline{\\cP},\\overline{P}$, and denote the adaptive one by $\\cR, R,\\cP,P$.\nWe remark that Definition~\\ref{def:restriction} can be used for both versions of restriction and prolongation.\nWe give an example of $\\overline{R},\\overline{P}$ in \\cref{sec:exp}.\n","start":31324,"end":32020}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":["def:subdifferential_II"],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"次梯度与次微分","source_units":[{"raw_tex":"\\subsection{Review of subdifferential of nonsmooth functions}\\label{sec:mgprox:subsec:subdifferential}\nThe subdifferential \\cite{rockafellar1970convex}\nis a standard framework used in convex analysis to deal with nondifferentiable functions.\nA convex function $g(x) : \\IRn \\rightarrow \\overline{\\IR} \\coloneqq \\IR \\cup \\{+\\infty\\}$ is called nonsmooth if it is not differentiable for some $x$ in $\\IRn$.\nA point $q \\in \\IRn$ is called a subgradient of $g$ at  $x$ if for all $y \\in \\IRn$ the inequality $g(y) \\geq g(x) + \\langle q, y -x \\rangle$ holds.\nThe subdifferential of $g$ at a point $x$ is defined as the set of all subgradients of $g$ at $x$, i.e.,\n\\begin{equation}\n  \\partial g(x)\n\\coloneqq \n\\Big\\{~\nq \\in \\IRn ~~\\big|~~ \ng(y)  \\geq g(x) + \\langle q, y -x \\rangle\n~\\forall y \\in \\IRn\n~\\Big\\} ~ \\subset \\IRn,\n\\label{def:subdifferential_II}\n\\end{equation}\nso $\\partial g(x)$ is generally set-valued.\nIf $g$ is differentiable at $x$, then $\\nabla g(x)$ exists and the set $\\partial g(x)$ reduces to the singleton $\\big\\{\\nabla g(x)\\big\\}$.\n","start":32067,"end":33114}]},{"id":"S015","semantic_type":"external_fact","kind":"external_fact","labels":["fact:subgrad_sum_rule"],"member_ids":["S015"],"declared_external":true,"declaration_text":["原始前提清单中的 Moreau–Rockafellar 和规则；原文第716行引用 Beck (2017), Theorem 3.36。"],"source_only":false,"title":"次微分和规则","source_units":[{"raw_tex":"\\paragraph{Subdifferential sum rule (Moreau–Rockafellar theorem)}~~\nLet $\\oplus$ denote the Minkowski sum.\nSince subdifferentials are generally set-valued, hence for two functions $f, g$, generally $\\partial (f+g) \\neq \\partial f \\oplus \\partial g$ but $\\partial (f+g) \\supset \\partial f \\oplus \\partial g$.\nThe sum rule $\\partial (f+g) = \\partial f \\oplus \\partial g$ holds if $f,g$ satisfy a qualification condition (e.g. \\cite[Theorem 3.36]{beck2017first}): the relative interior of the domain of $f$ has a non-empty intersection with the relative interior of domain of $g$,\ni.e.,\n\\begin{equation}\n\\ri (\\dom f) \\,\\cap\\, \\ri (\\dom g) \\neq \\varnothing\n%\n\\implies\n%\n\\partial \\Big( f(x) + g(x) \\Big) =\n\\partial f(x) \\oplus \\partial g(x),\n~~~\n\\forall x \\in \\textrm{dom} f \\,\\cap\\, \\textrm{dom} g.\n\\label{fact:subgrad_sum_rule}\n\\end{equation}\nThe right-hand side (RHS) of \\eqref{fact:subgrad_sum_rule}, which is the subdifferential sum rule,\nis known as the Moreau–Rockafellar theorem \\cite{lai1988moreau}.\n","start":33120,"end":34124}]},{"id":"S016","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"各层和规则资格条件","source_units":[{"raw_tex":"We now discuss the fact that the functions $f_\\ell, g_\\ell$ for all levels $\\ell$ in this work satisfy the Moreau–Rockafellar theorem. \n\\begin{itemize}\n\\item At level $\\ell=0$, we have the left-hand side (LHS) of \\eqref{fact:subgrad_sum_rule} for $f_0, g_0$ \n%as $\\dom f_0 \n%= \\dom g_0 \n%= \\IRn$\nby assumption.\n\\item At levels $\\ell>0$, by Definition~\\ref{def:restriction}, \nthe LHS of \\eqref{fact:subgrad_sum_rule} holds for the coarse functions.\nTo see this, take the coarse function as composition of the fine function with a linear map, and recognize the fact that the domain of a function is preserved under composition with a linear map.\nTo be explicit, we have\n\\[\n\\begin{array}{rcl}\n\\ri \\Big( \\dom f_1 \\Big)\n\\cap \n\\ri \\Big( \\dom g_1 \\Big)\n~=~\n\\ri \\Big( \\underbrace{\\dom (f_0 \\circ R)}_{\\text{whole }\\IR^{n_1}} \\Big)\n\\cap \n\\ri \\Big( \\dom (g_0 \\circ R)  \\Big)\n~\\neq~ \\varnothing.\n\\end{array}\n\\]\n\\end{itemize}\n","start":34124,"end":35038}]},{"id":"S017","semantic_type":"derived_fact","kind":"derived_fact","labels":["ass:F_sum_rule"],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"各层次微分分解","source_units":[{"raw_tex":"To sum up, in this work \\eqref{fact:subgrad_sum_rule} holds for all levels $\\ell$:\n\\begin{equation}\n\\partial F_\\ell(x_\\ell) \n\\coloneqq  \\partial \\Big( f_\\ell(x_\\ell) + g_\\ell(x_\\ell) \\Big)\n= \\partial f_\\ell(x_\\ell) \\oplus \\partial g_\\ell(x_\\ell)\n= \\nabla f_\\ell(x_\\ell) + \\partial g_\\ell (x_\\ell)\n,\n\\label{ass:F_sum_rule}\n\\end{equation}\nwhere $+$ is used instead of $\\oplus$ in $\\nabla f_\\ell(x_\\ell) + \\partial g_\\ell (x_\\ell)$ because $\\nabla f_\\ell(x_\\ell)$ is a singleton.\n","start":35038,"end":35515}]},{"id":"S018","semantic_type":"external_fact","kind":"external_fact","labels":["subdiff_coarse_linearmap"],"member_ids":["S018"],"declared_external":true,"declaration_text":["原始前提清单中的线性复合链式规则；原文第779行引用 Beck (2017), Theorem 3.43。"],"source_only":false,"title":"线性复合的次微分链式规则","source_units":[{"raw_tex":"\\subsection{Convexity and subdifferential of coarse function}\nFrom Definition~\\ref{def:restriction}, the coarse function can be written as $F_{\\ell+1} \\coloneqq F_\\ell \\circ R$, meaning that we can see the restriction process as the fine function taking composition with the linear map $R$.\nSuch composition view point gives us a series of useful properties for this work.\nFirst, we have a closed-form expression for the subdifferential of the coarse function in terms of the the subdifferential of the fine function.\nI.e., by \\cite[Theorem 3.43]{beck2017first}, we have that\n\\begin{equation}\\label{subdiff_coarse_linearmap}\n\\partial F_{\\ell+1}(x) = \\partial (F_\\ell \\circ R)(x) = R^\\top \\partial F_\\ell (Rx). \n\\end{equation}\n","start":35573,"end":36299}]},{"id":"S019","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"线性复合保持凸性","source_units":[{"raw_tex":"Then, by the fact that convexity is preserved under linear map, we have that ``restriction preserves convexity''.\nIn other words, $f_{\\ell+1}$ is convex if $f_{\\ell}$ is convex.\n","start":36299,"end":36477}]},{"id":"S020","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"限制矩阵满秩的断言","source_units":[{"raw_tex":"Furthermore, if $R$ is full-rank (which is the case in this paper), restriction is submultiplicative on the modulus of convexity, as illustrated in the following lemma.\n","start":36477,"end":36646}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","labels":["lem:strcvx_linearmap"],"member_ids":["S021","S022","S023","S024","S025","S026"],"declared_external":false,"source_only":false,"title":"满秩复合的强凸模量","source_units":[{"raw_tex":"\\begin{lemma}[Composition with full-rank matrix preserves  convexity]\n\\label{lem:strcvx_linearmap}\nGiven a function $F : \\IRn \\rightarrow \\overline{\\IR}$ that is $\\mu$-strongly convex and a rank-$n$ matrix $R \\in \\IR^{m \\times n}$, the function $F \\circ R$ is $\\mu \\sigma^2_n$-strongly convex, where $\\sigma_n$ is the $n$th singular value of $R$.\n","start":36648,"end":36995},{"raw_tex":"\\begin{proof}\n$F$ is strongly convex so $\\partial F$ is strongly monotone \\cite{bauschke2011convex}: for all $x,y \\in \\dom F$, \n\\begin{equation}\\label{lem:strcvx_linearmap:fineF}\n\\big\\langle x - y, ~ \\partial F(x) - \\partial F(y ) \\big\\rangle \n~\\geq~\n\\mu \\| x - y\\|_2^2.\n\\end{equation}\n","start":36995,"end":37281},{"raw_tex":"\\begin{proof}\n$F$ is strongly convex so $\\partial F$ is strongly monotone \\cite{bauschke2011convex}: for all $x,y \\in \\dom F$, \n\\begin{equation}\\label{lem:strcvx_linearmap:fineF}\n\\big\\langle x - y, ~ \\partial F(x) - \\partial F(y ) \\big\\rangle \n~\\geq~\n\\mu \\| x - y\\|_2^2.\n\\end{equation}\nNow we show $\\partial (F \\circ R)$ is also strongly monotone.\nFor all $x,y $ in $\\dom (F \\circ R)$, we have\n\\[\n\\begin{array}{rcl}\n\\langle \nx - y, \n~ \\partial (F \\circ R)(x) - \\partial (F \\circ R)(y )\n\\rangle \n&\\overset{\\eqref{subdiff_coarse_linearmap}}{=}&\n\\langle \nx - y, \n~ R^\\top  \\partial F(Rx) - R^\\top  \\partial F (Ry )\n\\rangle \n\\\\\n&=&\n\\langle \nRx - Ry, \n~ \\partial F(Rx) -  \\partial F (Ry )\n\\rangle \n\\\\\n&\\overset{\\eqref{lem:strcvx_linearmap:fineF}}{\\geq}&\n\\mu \\| Rx-Ry\\|_2^2\n~\\geq~ \\mu \\sigma_n^2 \\| x-y\\|_2^2,\n\\end{array}\n\\]\nwhere $\\sigma_n > 0 $ is the $n$th singular value of the full-rank matrix $R$.\nThus the subdifferential $\\partial (F \\circ R)$ is  is $\\mu \\sigma_n^2$-strongly monotone, and thus the function $F \\circ R$ is $\\mu \\sigma_n^2$-strongly convex.\n\\end{proof}\n","start":36995,"end":38067},{"raw_tex":"Now we show $\\partial (F \\circ R)$ is also strongly monotone.\nFor all $x,y $ in $\\dom (F \\circ R)$, we have\n\\[\n\\begin{array}{rcl}\n\\langle \nx - y, \n~ \\partial (F \\circ R)(x) - \\partial (F \\circ R)(y )\n\\rangle \n&\\overset{\\eqref{subdiff_coarse_linearmap}}{=}&\n\\langle \nx - y, \n~ R^\\top  \\partial F(Rx) - R^\\top  \\partial F (Ry )\n\\rangle \n\\\\\n&=&\n\\langle \nRx - Ry, \n~ \\partial F(Rx) -  \\partial F (Ry )\n\\rangle \n\\\\\n","start":37281,"end":37691},{"raw_tex":"&\\overset{\\eqref{lem:strcvx_linearmap:fineF}}{\\geq}&\n\\mu \\| Rx-Ry\\|_2^2\n~\\geq~ \\mu \\sigma_n^2 \\| x-y\\|_2^2,\n\\end{array}\n\\]\nwhere $\\sigma_n > 0 $ is the $n$th singular value of the full-rank matrix $R$.\n","start":37691,"end":37893},{"raw_tex":"Thus the subdifferential $\\partial (F \\circ R)$ is  is $\\mu \\sigma_n^2$-strongly monotone, and thus the function $F \\circ R$ is $\\mu \\sigma_n^2$-strongly convex.\n\\end{proof}\n","start":37893,"end":38067},{"raw_tex":"\\end{lemma}","start":38067,"end":38078}]},{"id":"S027","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"集合值项的下划线","source_units":[{"raw_tex":"\\paragraph{Notation for sets}\nFrom now on, when we encounter an expression containing both set-valued vector(s) and singleton vector(s), we underline the set-valued term(s) for visual clarity.\n","start":38081,"end":38274}]},{"id":"S028","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"跨层校正向量集合","source_units":[{"raw_tex":"\\subsection{Adaptive restriction and the \\texorpdfstring{$\\tau$}{} vector}\\label{sec:mgprox:subsec:adaptiveR}\nSince subdifferentials are set-valued, we define $\\tau$ in \\MGProx as an element of a set.\nAt a level $\\ell$, we define a set $\\underline{\\tau_{\\ell \\rightarrow \\ell+1}} ~\\coloneqq~\n\\underline{\\partial F_{\\ell+1}(x_{\\ell+1})} \\oplus (- R) \\underline{\\partial F_\\ell (x_\\ell)} $, where $\\ell \\rightarrow \\ell+1$ specifies that $\\tau$ connects level $\\ell$ to level $\\ell+1$, and the matrix $R$ here is an adaptive restriction operator that we will define soon.\nIn \\MGProx we choose an element of $\\underline{\\tau_{\\ell \\rightarrow \\ell+1}}$ as the tau vector.\nThat is, at level $\\ell=0$,\n\\begin{subequations}\n\\begin{align}\n\\tau_{0 \\rightarrow 1}\n~\\, \\in \\,~ \n\\underline{\\tau_{0 \\rightarrow 1}}\n& \n~\\coloneqq~\n\\underline{\\partial F_1(x_1)} \\oplus (- R) \\underline{\\partial F_0(x_0)}\n\\label{def:tau_mgprox1}\n\\\\\n","start":38319,"end":39237}]},{"id":"S029","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"校正集合的次微分展开","source_units":[{"raw_tex":"&\\overset{\\eqref{ass:F_sum_rule}}{=} \\nabla f_1(x_1) - R \\nabla f_0(x_0) + \\underline{\\partial g_1(x_1)}\n\\oplus \n(- R) \\underline{\\partial g_0(x_0)}.\n\\label{def:tau_mgprox2}\n\\end{align}\n\\end{subequations}\nNote that $\\underline{\\tau_{0 \\rightarrow 1}}$ is a function of two points at two different levels.\nIn \\eqref{def:tau_mgprox1} $\\underline{\\tau_{0 \\rightarrow 1}}$ is the Minkowski sum of two subdifferentials $\\partial F_1(x_1) $ and $-R\\partial F_0(x_0)$ which are generally set-valued.\n","start":39237,"end":39730}]},{"id":"S030","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"置零使限制次微分单值化","source_units":[{"raw_tex":"To obtain a tractable coarse-grid optimization problem (corresponding to line (iv) in Algorithm~\\ref{algo:mgopt}) we need to avoid complications coming from the Minkowski sum, and we do this by modifying the standard restriction (and prolongation) \nby zeroing out columns in $\\overline{R}$ to form $R$ such that the second subdifferential $R \\partial g_0(x_0)$ in \\eqref{def:tau_mgprox2}\nis a singleton vector.\nSimilarly, we  zero out the corresponding rows in $\\overline{P}$ to form $P$ for the coarse correction step, such that non-differentiable fine points are not corrected by the coarse grid.\nThis zeroing out process is adapted to the current point $x_0$, so we call this $R$ an adaptive restriction operator. \nIn other words, the purpose of the adaptive restriction is to reduce a generally set-valued subdifferential $R\\partial g_0(x_0)$ to a singleton.\nWe denote the adaptive operator $R$ corresponding to a point $x$ as $R(x)$ and thereby the adaptive restriction of $x$ is denoted as $R(x)x$.\nSometimes we just write $Rx$ if the meaning is clear from the context.\nBased on the above discussion, we now formally define adaptive restriction,\nand we give an example in \\cref{sec:exp}.\n","start":39730,"end":40924}]},{"id":"S031","semantic_type":"definition","kind":"definition","labels":["def:adaptiveR","eg:adaptiveR_L1"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"自适应限制算子","source_units":[{"raw_tex":"\n\n\\begin{definition}[Adaptive restriction operator for separable $g$]\\label{eg:adaptiveR_L1}\nFor a possibly nonsmooth function $g : \\IRn \\rightarrow \\IR$ that is separable, i.e., with $x = [x_1,x_2,\\dots,x_n]$, $g(x) = \\sum g_i(x_i)$ where $g_i$ is a function only of component $x_i$, \ngiven a full restriction operator $\\overline{R}$ and a vector $x$, the adaptive restriction operator $R$ with respect to a function $g$ at $x$ is defined by zeroing out the columns of $\\overline{R}$ corresponding to the elements in $\\partial g$ that are set-valued.\n\\label{def:adaptiveR}\n\\end{definition}\n","start":40924,"end":41515}]},{"id":"S032","semantic_type":"definition","kind":"definition","labels":["prob:minfg_coarse"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"带校正项的粗目标","source_units":[{"raw_tex":"\\subsection{A 2-level \\texorpdfstring{\\MGProx}{}algorithm}\\label{sec:mgprox:subsec:2lv}\nSimilar to the 2-level \\MGOPT method for solving Problem \\eqref{prob:mgopt}, we propose a 2-level \\MGProx method (Algorithm \\ref{aglo:2levelMG_exact}) that solves Problem \\eqref{prob:minfg} by utilizing a coarse problem defined as \n\\begin{equation}\\label{prob:minfg_coarse}\n\\argmin_{\\xi \\in \\IR^{n_1}}\\,\n\\bigg\\{\\,\nF^{\\tau}_1 (\\xi) \n~\\coloneqq~  F_1(\\xi) - \\langle \\tau_{0 \\rightarrow 1}^{k+1}, \\xi \\rangle\n~=~ f_1 (\\xi) + g_1(\\xi) - \\langle \\tau_{0 \\rightarrow 1}^{k+1}, \\xi \\rangle\n\\,\\bigg\\}.\n\\end{equation}\n\n\n","start":43823,"end":44422}]},{"id":"S033","semantic_type":"algorithm","kind":"algorithm","labels":["aglo:2levelMG_exact"],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"完整两层 MGProx 算法","source_units":[{"raw_tex":"\\begin{algorithm}\n\\caption{2-level \\MGProx for an approximate solution of \\eqref{prob:minfg}}\n\\label{aglo:2levelMG_exact}\n\\begin{algorithmic}\n\\STATE{Initialize $x^1_0$, $R$ and $P$}\n\\FOR{$k = 1,2,\\dots$}\n%\n\\STATE{(i)~~~~ $y^{k+1}_0 = \\prox_{\\frac{1}{L_0}g_0} \\Big (x^k_0 - \\frac{1}{L_0} \\nabla f(x^k_0) \\Big)$\n\\hfill level-0 proximal gradient step \\vspace{0.25mm}\n}\n\\STATE{(ii)~~~ $y^{k+1}_1  = R(y^{k+1}_0) y^{k+1}_0 $   \n\\hfill construct the level-1 coarse variable \\vspace{0.25mm}\n}\n\\STATE{(iii)~~ $\\tau_{0 \\rightarrow 1}^{k+1} \\hspace{-1mm} \\in\\,\n\\underline{\\partial F_1(y^{k+1}_1 )}  - R(y^{k+1}_0) \\, \\underline{\\partial  F_0(y^{k+1}_0)}$\n\\hfill construct the tau vector \\vspace{0.25mm}\n}\n\\STATE{(iv)~~ $\nx^{k+1}_1\n= \\underset{\\xi}{\\argmin} \\, \\Big\\{ F_1^{\\tau}(\\xi) \\coloneqq F_1(\\xi) - \\langle \\tau_{0 \\rightarrow 1}^{k+1}, \\xi \\rangle \\Big\\}\n$\n\\hfill solve the level-1 coarse problem\n}\n\\STATE{(v)\\,~~~ $z^{k+1}_0 = y^{k+1}_0  + \\alpha P\\big(  x^{k+1}_1 - y^{k+1}_1 \\big)$\n\\hfill coarse correction \\vspace{0.25mm}\n}\n\\STATE{(vi)~~\\, $x^{k+1}_0 = \\prox_{\\frac{1}{L_0}g_0} \\Big (z^{k+1}_0 - \\frac{1}{L_0} \\nabla f(z^{k+1}_0) \\Big)$\n\\hfill level-0 proximal gradient step~~\n}\n%\n\\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}\n","start":44422,"end":45645}]},{"id":"S034","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"平滑步长与校正选择","source_units":[{"raw_tex":"\nHere are remarks for the steps in Algorithm~\\ref{aglo:2levelMG_exact}.\n\\begin{itemize}\n    \\item (i): we perform one or more proximal gradient iterations on the fine variable with a constant stepsize $1/L_0$, where $L_0$ is the Lipschitz constant of $\\nabla f_0$.\n    \\item (iii): we pick a value within the set to define $\\tau$;\n    as we are now using adaptive $R$, we use $+$ instead of $\\oplus$ in the expression of $\\tau$.\n    \\item (v): $\\alpha > 0$ is a stepsize; for its selection see \\cref{sec:mgprox:subsec:alpha}.\n","start":45645,"end":46171}]},{"id":"S035","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"非可微位置平移至零","source_units":[{"raw_tex":"    \\item The restriction for variable $y_0^{k+1}$ and the restriction for the subdifferential $ \\underline{\\partial  F_0(y^{k+1}_0)}$ can be slightly different.\n    On $y_0^{k+1}$ the restriction is the full restriction, on $ \\underline{\\partial  F_0(y^{k+1}_0)}$ is the adaptive one. \n    The explanation is as follows.   \n    For the particular cases of $g_0$ such as $\\ell_1$, $\\max\\{ \\cdot, 0\\}$ (element-wise maximum) and $\\iota_{[0,\\infty)}$ (indicator of nonnegative orthant), when we zero out column $i$ of $R$, the corresponding entry of $y_0^{k+1}$ is already $0$.    \n    Note that this conclusion does not hold in general for other nonsmooth functions or when the non-differentiability occur at another point (say at $x=1$). \n    Generally in those cases we will need to specify which restriction matrix (the full $R$ or the adaptive $R$) to be used to define $P$.\n        However, to simplify the presentation we always assume the non-differentiability occurs at $x=0$,\n         by performing a translation to shift the non-differentiability to occur at $x=0$.\n%        In fact, this is what exact we do in the experiment.\n\\end{itemize}\n","start":46171,"end":47322}]},{"id":"S036","semantic_type":"theorem","kind":"theorem","labels":["iq:descent_dir","thm:coarse_dir_descent"],"member_ids":["S036","S037","S038","S039","S040"],"declared_external":false,"source_only":false,"title":"粗校正的严格钝角条件","source_units":[{"raw_tex":"\\begin{theorem}[Angle condition of coarse correction]\\label{thm:coarse_dir_descent}\nIf $P(x^{k+1}_1 - y^{k+1}_1) \\neq 0$, then\n\\begin{equation}\\label{iq:descent_dir}\n\\big\\langle \n\\underline{\\partial F_0(y^{k+1}_0)} , P(x^{k+1}_1 - y^{k+1}_1) \n\\big\\rangle \n%\\leq \n<\n0. \n\\end{equation}\n%I.e., $P(x^{k+1}_1 - y^{k+1}_1) $ and any subgradient in $\\partial F_0(y^{k+1}_0)$ forms a strict obtuse angle.\n\\end{theorem}","start":50904,"end":51314},{"raw_tex":"Before we prove the theorem we emphasize that \\eqref{iq:descent_dir} applies for any subgradient in the set $\\underline{\\partial F_0(y^{k+1}_0)} $.\nFurthermore,\n\\[\n\\eqref{iq:descent_dir} \n%\n\\iff\n%\n\\big\\langle \nP^\\top \\underline{\\partial F_0(y^{k+1}_0)} , x^{k+1}_1 - y^{k+1}_1 \n\\big\\rangle\n< 0\n%\n\\overset{P^\\top = cR}{\\iff}\n%\nc \\big\\langle \nR \\underline{\\partial F_0(y^{k+1}_0)} , x^{k+1}_1 - y^{k+1}_1 \n\\big\\rangle\n< 0\n.\n\\]\nAs $c>0$, showing \\eqref{iq:descent_dir} is equivalent to showing\n\\begin{equation}\n\\big\\langle R \\underline{\\partial F_0(y^{k+1}_0)},  ~ x^{k+1}_1 - y^{k+1}_1 \\big\\rangle < 0,\n\\label{pf:descent_to_prove}\n\\end{equation}\nwhere $R \\underline{\\partial F_0(y^{k+1}_0)} $ is a singleton vector for all subgradients in $\\underline{\\partial F_0(y^{k+1}_0)}$ due to the adaptive $R$.\n","start":51315,"end":52115},{"raw_tex":"\\begin{proof}\nBy definition \n$\\tau_{0 \\rightarrow 1}^{k+1} \\overset{\\eqref{def:tau_mgprox1}}{\\in} \\underline{\\partial F_1(y_1^{k+1})}  - R \\underline{\\partial F_0(y_0^{k+1})}$ \nand the fact that $R \\underline{\\partial F_0(y^{k+1}_0)} $ is a singleton, we have\n$\nR \\underline{\\partial F_0(y_0^{k+1})} \n\\in\n\\underline{\\partial F_1(y_1^{k+1})}  - \\tau_{0 \\rightarrow 1}^{k+1}\n\\overset{\\eqref{prob:minfg_coarse}}{=} \\underline{\\partial F_1^\\tau(y_1^{k+1})}$,\nshowing that $R \\underline{\\partial F_0(y_0^{k+1})} $ is a subgradient of $F_1^{\\tau}$ at $y_1^{k+1}$.\n","start":52115,"end":52673},{"raw_tex":"\\begin{proof}\nBy definition \n$\\tau_{0 \\rightarrow 1}^{k+1} \\overset{\\eqref{def:tau_mgprox1}}{\\in} \\underline{\\partial F_1(y_1^{k+1})}  - R \\underline{\\partial F_0(y_0^{k+1})}$ \nand the fact that $R \\underline{\\partial F_0(y^{k+1}_0)} $ is a singleton, we have\n$\nR \\underline{\\partial F_0(y_0^{k+1})} \n\\in\n\\underline{\\partial F_1(y_1^{k+1})}  - \\tau_{0 \\rightarrow 1}^{k+1}\n\\overset{\\eqref{prob:minfg_coarse}}{=} \\underline{\\partial F_1^\\tau(y_1^{k+1})}$,\nshowing that $R \\underline{\\partial F_0(y_0^{k+1})} $ is a subgradient of $F_1^{\\tau}$ at $y_1^{k+1}$.\nFor any subgradient in the subdifferential $\\underline{\\partial F_1^\\tau(y_1^{k+1})} $, we have the following which implies \\eqref{pf:descent_to_prove}:\n$\n\\big\\langle \\underline{\\partial F_1^{\\tau}(y^{k+1}_1)}, x^{k+1}_1 - y^{k+1}_1 \\big\\rangle\n< F_1^{\\tau}(x^{k+1}_1) - F_1^{\\tau}(y^{k+1}_1) \n< 0,\n$\nwhere the first strict inequality is due to $F_1^\\tau$ being a strongly convex function (which implies strict convexity);\nthe second inequality is by $x^{k+1}_1 \\coloneqq \\underset{\\xi}{\\argmin}\\,\nF_1^{\\tau}(\\xi)$ and the assumption that $x_1^{k+1} \\neq y_1^{k+1}$.\n\\end{proof}","start":52115,"end":53251},{"raw_tex":"For any subgradient in the subdifferential $\\underline{\\partial F_1^\\tau(y_1^{k+1})} $, we have the following which implies \\eqref{pf:descent_to_prove}:\n$\n\\big\\langle \\underline{\\partial F_1^{\\tau}(y^{k+1}_1)}, x^{k+1}_1 - y^{k+1}_1 \\big\\rangle\n< F_1^{\\tau}(x^{k+1}_1) - F_1^{\\tau}(y^{k+1}_1) \n< 0,\n$\nwhere the first strict inequality is due to $F_1^\\tau$ being a strongly convex function (which implies strict convexity);\nthe second inequality is by $x^{k+1}_1 \\coloneqq \\underset{\\xi}{\\argmin}\\,\nF_1^{\\tau}(\\xi)$ and the assumption that $x_1^{k+1} \\neq y_1^{k+1}$.\n\\end{proof}","start":52673,"end":53251}]},{"id":"S041","semantic_type":"definition","kind":"definition","labels":["iq:coarse_descent_on_F"],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"粗校正的目标下降条件","source_units":[{"raw_tex":"\\subsubsection{Existence of coarse correction stepsize \\texorpdfstring{$\\alpha_k$}{}}\n\\label{sec:mgprox:subsec:alpha}\nBased on Theorem~\\ref{thm:coarse_dir_descent}, we now show that there exists a stepsize $\\alpha_k > 0$ such that \n\\begin{equation}\nF_0(z^{k+1}_0) \n~\\coloneqq~\nF_0\\big(\ny^{k+1}_0 + \\alpha_k P(x^{k+1}_1 - y^{k+1}_1) \n\\big)\n~<~\nF_0(y^{k+1}_0).\n\\label{iq:coarse_descent_on_F}\n\\end{equation}\n","start":53436,"end":53841}]},{"id":"S042","semantic_type":"lemma","kind":"lemma","labels":["lemma:alphaexists"],"member_ids":["S042","S045","S046","S047","S048"],"declared_external":false,"source_only":false,"title":"正粗校正步长存在","source_units":[{"raw_tex":"\\begin{lemma}[Existence of stepsize]\\label{lemma:alphaexists}\nThere exists $\\alpha_k > 0$ such that \\eqref{iq:coarse_descent_on_F} is satisfied for $P(x^{k+1}_1 - y^{k+1}_1) \\neq 0$.\n\\end{lemma}","start":53842,"end":54036},{"raw_tex":"\\begin{proof}\nWe prove the lemma in 3 steps.\n\\begin{enumerate}\n\n\\item (Half-space)\nThe strict inequality in Theorem~\\ref{thm:coarse_dir_descent} means that $\\underline{\\partial F_0(y_0^{k+1})}$ is strictly inside a half-space with normal vector $p = P(x^{k+1}_1 - y^{k+1}_1)$.  \n\n","start":55140,"end":55420},{"raw_tex":"\\begin{proof}\nWe prove the lemma in 3 steps.\n\\begin{enumerate}\n\n\\item (Half-space)\nThe strict inequality in Theorem~\\ref{thm:coarse_dir_descent} means that $\\underline{\\partial F_0(y_0^{k+1})}$ is strictly inside a half-space with normal vector $p = P(x^{k+1}_1 - y^{k+1}_1)$.  \n\n\\item (Strict separation)\nBeing a compact convex set, $\\underline{\\partial F_0(y_0^{k+1}0)}$ lying strictly on one side of the hyperplane must be a positive distance (say $\\alpha_k > 0$) from that hyperplane.\n\n\\item (Support and directional derivative)\nEvaluating the support function of $\\underline{\\partial F_0(y_0^{k+1})}$, i.e., the directional derivative of $F_0$ at $y_0^{k+1}$ in the direction $p$, we have \\eqref{iq:coarse_descent_on_F}.\n\\end{enumerate}\n\\end{proof}","start":55140,"end":55893},{"raw_tex":"\\item (Strict separation)\nBeing a compact convex set, $\\underline{\\partial F_0(y_0^{k+1}0)}$ lying strictly on one side of the hyperplane must be a positive distance (say $\\alpha_k > 0$) from that hyperplane.\n\n","start":55420,"end":55630},{"raw_tex":"\\item (Support and directional derivative)\nEvaluating the support function of $\\underline{\\partial F_0(y_0^{k+1})}$, i.e., the directional derivative of $F_0$ at $y_0^{k+1}$ in the direction $p$, we have \\eqref{iq:coarse_descent_on_F}.\n\\end{enumerate}\n\\end{proof}","start":55630,"end":55893}]},{"id":"S043","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S043"],"declared_external":true,"declaration_text":["原始前提清单中的有限凸函数次微分紧性及方向导数支撑函数表征；原文引用 Hiriart-Urruty–Lemarechal (2004), Def. 1.1.4, p.165。"],"source_only":false,"title":"紧次微分与方向导数","source_units":[{"raw_tex":"To prove the lemma, we make use \nof a fact of the subdifferentials of finite convex functions \\cite[Def.1.1.4, p.165]{hiriart2004fundamentals}: $\\partial g(x)$ is a nonempty compact convex set $\\cS \\subset \\IRn$ whose support function $\\displaystyle \\sup_{s}\\{ \\langle s, x\\rangle \\,|\\, s \\in \\cS  \\}$ \nis the directional derivative of $g$ at $x$.\n","start":54038,"end":54386}]},{"id":"S044","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"细目标次微分的紧性应用","source_units":[{"raw_tex":"By this, the subdifferential\n $\\underline{\\partial F_0(y_0^{k+1})}$ is a compact convex set whose support function is the directional derivative of $F_0$ at $y_0^{k+1}$.\nWe emphasize that here \n%$F_0 : \\IR^{n_0} \\rightarrow  \\overline{\\IR}$ at $y_0^{k+1}$ is \n$F_0(y_0^{k+1})$ is \nfinite, \n% i.e., $F_0 : \\IR^{n_0} \\rightarrow  \\overline{\\IR}$ will never reach $+\\infty$ at $y_0^{k+1}$ since $y_0^{k+1}$ is obtained by the proximal gradient step (in other words, $y_0^{k+1}$ is a ``feasible point'' if $g_0$ is an indicator function of a constraint set),\nso we can make use of the result on directional derivative in \\cite[Def. 1.1.4, p.165]{hiriart2004fundamentals}, which only applies for finite convex functions, associated with the subdifferential.\n","start":54386,"end":55139}]},{"id":"S049","semantic_type":"lemma","kind":"lemma","labels":["eqn:suffdescentPGD","lemma:suff_descent_proxgrad"],"member_ids":["S049"],"declared_external":true,"declaration_text":["原始前提清单明确许可 B0071，来源为 Beck (2017), Lemma 10.4；本文没有提供其证明。"],"source_only":false,"title":"近端梯度的充分下降前提","source_units":[{"raw_tex":"\\begin{lemma}\n(Sufficient descent property of proximal gradient \\cite[Lemma 10.4]{beck2017first})\n\\label{lemma:suff_descent_proxgrad} \nLet $L_0$ be the Lipschitz constant of $\\nabla f_0$.\nFor step (i) in Algorithm~\\ref{aglo:2levelMG_exact},  we have\n\\begin{equation}\nF_0( y^{k+1}_0)\n\\leq \nF_0( x^k_0) - \\| G_0(x_0^k) \\|_2^2 / (2L_0),\n~~\nG_0( x_0^k) \n=\nL_0 \\big[\nx_0^k - \\prox_{ \\frac{1}{L} g_0 }\\big( x_0^k - \\nabla f_0(x_0^k)/L_0\n\\big)\n\\big].\n\\label{eqn:suffdescentPGD}\n\\end{equation}\n\\end{lemma}\n$G_0( x_0^k)$ is called the proximal gradient map of $F_0$ at $x_0^k$.\nThe inequality also holds for step (vi).\n","start":58297,"end":58907}]},{"id":"S050","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"近端梯度映射","source_units":[{"raw_tex":"G_0( x_0^k) \n=\nL_0 \\big[\nx_0^k - \\prox_{ \\frac{1}{L} g_0 }\\big( x_0^k - \\nabla f_0(x_0^k)/L_0\n\\big)\n\\big].\n\\label{eqn:suffdescentPGD}\n\\end{equation}\n\\end{lemma}\n$G_0( x_0^k)$ is called the proximal gradient map of $F_0$ at $x_0^k$.\n","start":58634,"end":58866}]},{"id":"S051","semantic_type":"algorithm","kind":"algorithm","labels":["aglo:naivelinesearch"],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"完整朴素回溯线搜索","source_units":[{"raw_tex":"\\begin{algorithm}\n\\caption{Naive line search}\\label{aglo:naivelinesearch}\n\\begin{algorithmic}\n\\STATE{Set $\\alpha > 0$ and select a tolerance $\\epsilon > 0$  (e.g. $10^{-15}$)}\n\\WHILE{true}\n\\STATE{\\textbf{If} $F_0\\big( y^{k+1}_0 + \\alpha P (x^{k+1}_1 - y^{k+1}_1) \\big) \\leq F_0( y^{k+1}_0 )$ \\textbf{ then }\nreturn $z_0^{k+1}=y^{k+1}_0 + \\alpha P(x^{k+1}_1 - y^{k+1}_1)$.}\n\\\\\n\\STATE{\\textbf{else if} $\\alpha > \\epsilon $ \\textbf{then} $\\alpha = \\alpha/2 $.}\n\\\\\n\\STATE{~~~~~~~\\textbf{else} return $z_0^{k+1}=y^{k+1}_0$.}\n\\ENDWHILE\n\\end{algorithmic}\n\\end{algorithm}\n","start":61746,"end":62310}]},{"id":"S052","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"线性收敛段的细层简写","source_units":[{"raw_tex":"\\subsubsection{Linear convergence rate}\nAll the functions and variables here are at level 0 so we omit the subscripts.\nNow we show that $\\big\\{F( x^k) \\big\\}_{k \\in \\IN}$ converges to $F^*$ with a linear rate using the \\emph{Proximal Polyak-{\\L}ojasiewicz inequality} \\cite[Section 4]{karimi2016linear}. \n","start":76605,"end":76910}]},{"id":"S053","semantic_type":"definition","kind":"definition","labels":["def:ProxPL"],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"近端 PL 不等式","source_units":[{"raw_tex":"The function $F$ in Problem \\eqref{prob:minfg} is called ProxPL, if there exists $\\mu > 0$, called the ProxPL constant, such that \n\\begin{equation}\n\\cD_g(x,L)\n~\\geq~\n2 \\mu \\big( F(x) - F^* \\big)  \\qquad  \\forall x,\n\\tag{ProxPL}\n\\label{def:ProxPL}\n\\end{equation}\n","start":76910,"end":77172}]},{"id":"S054","semantic_type":"definition","kind":"definition","labels":["def:dg"],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"近端 PL 辅助量","source_units":[{"raw_tex":"\\begin{equation}\n\\cD_g(x,\\alpha) \n\\coloneqq\n-2\\alpha \\min_z \\,\n\\Big\\{\n{\\textstyle\\frac{\\alpha}{2}} \\| z - x \\|_2^2 \n+ \\big\\langle z - x, \\nabla f(x) \\big\\rangle\n+ g(z) - g(x)\n\\Big\\}\n.\n\\label{def:dg}\n\\end{equation}\n","start":77172,"end":77386}]},{"id":"S055","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"辅助量与近端子问题的对应","source_units":[{"raw_tex":"Intuitively, $\\cD_g$ is defined based on the proximal gradient operator:\n\\[\n\\prox_{\\frac{1}{L}g} \\Big( x - \\frac{\\nabla f(x)}{L} \\Big)\n=\n\\underset{z}{\\argmin} \\, \\frac{L}{2} \\| z -  x \\|_2^2\n+ \\big\\langle z - x, \\nabla f(x) \\big\\rangle\n+ g(z) - g(x).\n\\]\n","start":77386,"end":77640}]},{"id":"S056","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S056"],"declared_external":true,"declaration_text":["原始前提清单中的强凸光滑分量蕴含同模量 ProxPL；来源为 Karimi et al. (2016), Section 4，第1822行指定章节、第1851行明确调用。"],"source_only":false,"title":"光滑项强凸蕴含近端 PL","source_units":[{"raw_tex":"It has been shown in \\cite{karimi2016linear} that if $f$ in \\eqref{prob:minfg} is $\\mu$-strongly convex, then $F$ is $\\mu$-ProxPL.\n","start":77640,"end":77771}]},{"id":"S058","semantic_type":"theorem","kind":"theorem","labels":["thm:mgprox_converge_proxPL"],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"两层 MGProx 的线性收敛界","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm:mgprox_converge_proxPL}\nLet $x^1_0$ be the initial guess of the algorithm,\n$x_0^* = \\argmin \\, F_0(x)$ and $F^*_0 \\coloneqq F_0(x_0^*)$.\nThe sequence $\\{x_0^k\\}_{k \\in \\IN}$ generated by \\MGProx (Algorithm~\\ref{aglo:2levelMG_exact}) for solving Problem \\eqref{prob:minfg} satisfies\n$\nF_0(x_0^{k+1}) - F_0^* \n\\leq \n\\Big( 1-\\frac{\\mu_0}{L_0} \\Big)^k \\big( F_0(x_0^1) - F_0^* \\big).\n$\n\n\n","start":78147,"end":78557},{"raw_tex":"\n\\end{theorem}","start":79949,"end":79963}]},{"id":"S059","semantic_type":"proof","kind":"proof","labels":["thm:mgprox_converge_proxPL"],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"线性收敛定理的完整证明","source_units":[{"raw_tex":"\\begin{proof}\nFirst, by assumption $F$ is strongly convex, so $F$ is $\\mu$-ProxP\\L\\, with $\\mu > 0$ and\n\\[\n\\renewcommand*{\\arraystretch}{1.33}\n\\begin{array}{rcl}\nF(x^{k+1})\n&\\overset{\\eqref{eqn:suffdescentPGD}}{\\leq}& \\hspace{-2mm}\nF(z^{k+1})\n~\\overset{\\eqref{iq:coarse_descent_on_F}}{\\leq}~ \nF(y^{k+1})  \n~=~\nf(y^{k+1}) + g(y^{k+1}) + g(x^k) - g(x^k)\n\\\\\n&\\overset{\\eqref{def:Lsmooth}}{\\leq}&  \\hspace{-2mm}\nf(x^k) +  \\langle  \\nabla f(x^k), y^{k+1} - x^k \\rangle + \\frac{L}{2} \\|  y^{k+1} - x^k \\|_2^2 + g(x^k) +  g(y^{k+1})  -  g(x^k) \t\n\\\\\n&=& \\hspace{-2mm}\nF(x^k) +  \\langle  \\nabla f(x^k), y^{k+1} - x^k \\rangle + \\frac{L}{2} \\|  y^{k+1} - x^k \\|_2^2 +  g(y^{k+1})  -  g(x^k) \n\\\\\n&=& \\hspace{-2mm}\nF(x^k) - \\frac{1}{2L} \\underbrace{(-2L)\\Big( \n\\langle  \\nabla f(x^k), y^{k+1} - x^k \\rangle \n+ {\\textstyle \\frac{L}{2}} \\|  y^{k+1} - x^k \\|_2^2 +  g(y^{k+1})  -  g(x^k) \n\\Big)}_{\\overset{\\eqref{def:dg}}{ = } \\, \\cD_g(x^k, L), \\, \\text{ since }y^{k+1} \\,\\coloneqq\\, \\underset{z}{\\argmin} \\, \\frac{L}{2} \\| z -  x^k \\|_2^2\n+ \\langle z - x^k, \\nabla f(x^k)\\rangle\n+ g(z) - g(x^k).}\n\\\\\n&\\overset{\\eqref{def:ProxPL}}{\\leq}& \\hspace{-2mm}\nF(x^k) - \\frac{\\mu}{L} \\big(F(x^k) - F^*\\big).\n\\end{array}\n\\]\nAdding $-F^*$ on both sides of the inequality gives\n$ \nF(x^{k+1}) - F^* \n\\leq \\big( 1 - \\mu/L\\big) \\big(F(x^k) - F^*\\big)\n$.\nApplying this inequality recursively completes the proof.\n\\end{proof}","start":78557,"end":79949}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":["thm:mgprox_converge_proxPL"],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"引入细层近端 PL 性质","source_units":[{"raw_tex":"\\begin{proof}\nFirst, by assumption $F$ is strongly convex, so $F$ is $\\mu$-ProxP\\L\\, with $\\mu > 0$ and\n","start":78557,"end":78661}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":["thm:mgprox_converge_proxPL"],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"组合后平滑与粗校正非增","source_units":[{"raw_tex":"\\[\n\\renewcommand*{\\arraystretch}{1.33}\n\\begin{array}{rcl}\nF(x^{k+1})\n&\\overset{\\eqref{eqn:suffdescentPGD}}{\\leq}& \\hspace{-2mm}\nF(z^{k+1})\n~\\overset{\\eqref{iq:coarse_descent_on_F}}{\\leq}~ \nF(y^{k+1})  \n~=~\nf(y^{k+1}) + g(y^{k+1}) + g(x^k) - g(x^k)\n","start":78661,"end":78909}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":["thm:mgprox_converge_proxPL"],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"用光滑性建立模型上界","source_units":[{"raw_tex":"\\\\\n&\\overset{\\eqref{def:Lsmooth}}{\\leq}&  \\hspace{-2mm}\nf(x^k) +  \\langle  \\nabla f(x^k), y^{k+1} - x^k \\rangle + \\frac{L}{2} \\|  y^{k+1} - x^k \\|_2^2 + g(x^k) +  g(y^{k+1})  -  g(x^k) \t\n\\\\\n&=& \\hspace{-2mm}\nF(x^k) +  \\langle  \\nabla f(x^k), y^{k+1} - x^k \\rangle + \\frac{L}{2} \\|  y^{k+1} - x^k \\|_2^2 +  g(y^{k+1})  -  g(x^k) \n\\\\\n","start":78909,"end":79241}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":["thm:mgprox_converge_proxPL"],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"识别最小模型值为辅助量","source_units":[{"raw_tex":"&=& \\hspace{-2mm}\nF(x^k) - \\frac{1}{2L} \\underbrace{(-2L)\\Big( \n\\langle  \\nabla f(x^k), y^{k+1} - x^k \\rangle \n+ {\\textstyle \\frac{L}{2}} \\|  y^{k+1} - x^k \\|_2^2 +  g(y^{k+1})  -  g(x^k) \n\\Big)}_{\\overset{\\eqref{def:dg}}{ = } \\, \\cD_g(x^k, L), \\, \\text{ since }y^{k+1} \\,\\coloneqq\\, \\underset{z}{\\argmin} \\, \\frac{L}{2} \\| z -  x^k \\|_2^2\n+ \\langle z - x^k, \\nabla f(x^k)\\rangle\n+ g(z) - g(x^k).}\n\\\\\n","start":79241,"end":79642}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":["thm:mgprox_converge_proxPL"],"member_ids":["S064"],"declared_external":false,"source_only":false,"title":"应用近端 PL 控制误差","source_units":[{"raw_tex":"&\\overset{\\eqref{def:ProxPL}}{\\leq}& \\hspace{-2mm}\nF(x^k) - \\frac{\\mu}{L} \\big(F(x^k) - F^*\\big).\n\\end{array}\n\\]\n","start":79642,"end":79755}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":["thm:mgprox_converge_proxPL"],"member_ids":["S065"],"declared_external":false,"source_only":false,"title":"得到单步误差递推","source_units":[{"raw_tex":"Adding $-F^*$ on both sides of the inequality gives\n$ \nF(x^{k+1}) - F^* \n\\leq \\big( 1 - \\mu/L\\big) \\big(F(x^k) - F^*\\big)\n$.\n","start":79755,"end":79880}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":["thm:mgprox_converge_proxPL"],"member_ids":["S066"],"declared_external":false,"source_only":false,"title":"递推得到几何误差界","source_units":[{"raw_tex":"Applying this inequality recursively completes the proof.\n\\end{proof}","start":79880,"end":79949}]},{"id":"S067","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S067"],"declared_external":false,"source_only":true,"title":"收缩因子的参数范围","source_units":[{"raw_tex":"\\item Convergence rate: for a $\\mu_0$-strongly convex and $L_0$-smooth $f_0$, we have $0 < \\mu_0 \\leq L_0$ and $0 \\leq 1 - \\mu_0/L_0 < 1$.\n","start":80042,"end":80181}]},{"id":"S068","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S068"],"declared_external":true,"declaration_text":["原始前提清单中的强凸性与次微分强单调性等价；原文第792行引用 Bauschke–Combettes。"],"source_only":false,"title":"强凸性与次微分强单调性等价","source_units":[]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S069"],"declared_external":true,"declaration_text":["原始前提清单中的凸次梯度最优性条件和强凸次梯度不等式。"],"source_only":false,"title":"强凸次梯度不等式与最优性","source_units":[]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S070"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；原文第820—823行使用但未另列外部来源。"],"source_only":false,"title":"满列秩的奇异值下界","source_units":[]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S071"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；对应第1869行对 F 强凸性的假设性调用。"],"source_only":false,"title":"强凸项加凸项保持模量","source_units":[]},{"id":"S072","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S072"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；原文第1175行直接调用 F_1^τ 的强凸性。"],"source_only":false,"title":"线性扰动保持强凸模量","source_units":[]},{"id":"S073","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S073"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；对应第1254行从支撑函数与方向导数直接跳至下降式的标准过渡。"],"source_only":false,"title":"负方向导数蕴含局部下降","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S021","S023":"S021","S024":"S021","S025":"S021","S026":"S021","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S036","S038":"S036","S039":"S036","S040":"S036","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S042","S046":"S042","S047":"S042","S048":"S042","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073"}}}}
{"record_id":"2301.11678v2","paper":{"title":"Approximating Higher-Order Derivative Tensors Using Secant Updates","authors":["Welzel, Karl","Hauser, Raphael A."],"arxiv_id":"2301.11678","version":"v2","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/2301.11678v2","pdf_url":"https://arxiv.org/pdf/2301.11678v2","doi":"10.1137/23M1549687"},"context":{"raw_tex":"The incredible success of quasi-Newton methods is largely based on the fact that the rules for updating the Hessian approximations are able to extract crucial second-order information from gradient evaluations.\nUnlike finite difference methods they do so without direct control over the evaluation points.\nRather, quasi-Newton rules are designed to handle evaluations of the first derivative at points that are generated by an extraneous process and produce best-effort approximations of the second derivative.\nWe propose generalizations of these rules that mimic the quasi-Newton approach but approximate \\(p\\)th derivatives from given evaluations of \\((p-1)\\)st derivatives for any \\(p \\geq 2\\).\n\n\nA \\emph{\\(p\\)-tensor} \\(\\tns{T}\\) of dimensions \\(n_1 \\times \\dots \\times n_p\\) is a multilinear map \\(\\R^{n_1} \\times \\dots \\times \\R^{n_p}\\to \\R\\), so that its evaluation\n\\begin{equation}\n    \\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p], \\quad \\vek{s}_i \\in \\R^{n_i}\n\\end{equation}\nis linear in each component.\nWe refer to these \\(p\\) components as the \\emph{modes} of the tensor and denote the space of such \\(p\\)-tensors by \\(\\R^{n_1 \\otimes \\dots \\otimes n_p}\\).\nIf \\(n_1 = \\dots = n_p = n\\) the space is denoted \\(\\R^{\\otimes^p n}\\) and \\(\\tns{T}[\\vek{s}, \\dots, \\vek{s}]\\) is abbreviated as \\(\\tns{T}[\\vek{s}]^p\\).\nThe notation also allows to apply the tensor to \\(q < p\\) vectors, which then results in a \\((p-q)\\)-tensor.\nMoreover, we define the application of matrices \\(\\mat{W}_1, \\dots, \\mat{W}_p\\) of appropriate dimensions to a tensor by\n\\begin{equation}\n    \\big(\\tns{T}[\\mat{W}_1, \\dots, \\mat{W}_p]\\big)[\\vek{s}_1, \\dots, \\vek{s}_p] = \\tns{T}[\\mat{W}_1 \\vek{s}_1, \\dots, \\mat{W}_p \\vek{s}_p].\n\\end{equation}\n\nThe outer product of a \\(p_1\\)-tensor \\(\\tns{T}_1\\) with a \\(p_2\\)-tensor \\(\\tns{T}_2\\) is defined as\n\\begin{equation}\n    (\\tns{T}_1 \\otimes \\tns{T}_2)[\\vek{s}_1, \\dots, \\vek{s}_{p_1 + p_2}] = \\tns{T}_1[\\vek{s}_1, \\dots, \\vek{s}_{p_1}] \\cdot \\tns{T}_2[\\vek{s}_{p_1 + 1}, \\dots, \\vek{s}_{p_1 + p_2}].\n\\end{equation}\nIn particular, tensors of the form \\(\\tns{T} = \\vek{v}_1 \\otimes \\dots \\otimes \\vek{v}_p\\) for vectors \\(\\vek{v}_1, \\dots, \\vek{v}_p \\in \\R^n\\), i.e. those where \\(\\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p] = \\prod_{i = 1}^n \\vek{v}_i^T \\vek{s}_i\\), are called \\emph{elementary} or \\emph{rank-one} tensors.\nIf all vectors are the same (\\(\\vek{v}_1 = \\dots = \\vek{v}_p = \\vek{v}\\)), we abbreviate the notation above to \\(\\otimes^p \\vek{v}\\).\n(Note that this notation is slightly inconsistent since 1-tensors should be row vectors, but are represented by standard column vectors to simplify the notation.\nThis is why \\(\\vek{v}[\\mat{W}] = \\mat{W}^T \\vek{v}\\).)\n\nFor any \\(\\tns{T} \\in \\R^{\\otimes^p n}\\) and any permutation \\(\\sigma \\in S_p\\), let \\(\\sigma(\\tns{T}) \\in \\R^{\\otimes^p n}\\) be defined by\n\\begin{equation}\n    \\sigma(\\tns{T})[\\vek{s}_1, \\dots, \\vek{s}_p] = \\tns{T}[\\vek{s}_{\\sigma(1)}, \\dots, \\vek{s}_{\\sigma(p)}].\n\\end{equation}\nIf \\(\\sigma(\\tns{T}) = \\tns{T}\\) for all \\(\\sigma \\in S_p\\), then \\(\\tns{T}\\) is called \\emph{symmetric}.\nThe space of all symmetric \\(p\\)-tensors is denoted \\(\\R^{\\otimes^p n}_{\\sym}\\).\nThe projection of \\(\\R^{\\otimes^p n}\\) onto \\(\\R^{\\otimes^p n}_{\\sym}\\) is given by\n\\begin{equation}\n    P_{\\sym}(\\tns{T}) = \\frac{1}{p!} \\sum_{\\sigma \\in S_p} \\sigma(\\tns{T}),\n\\end{equation}\nsee \\cite[Proposition 3.76]{hackbusch_tensor_2012}.\n\nJust like matrices, tensors are fully characterized by their actions on basis vectors.\nThis can be used to represent a \\(p\\)-tensor \\(\\tns{T} \\in \\R^{\\otimes^p n}\\) as a \\(p\\)-dimensional array \\((t_{i_1, \\dots, i_p})_{1 \\leq i_j \\leq n, 1 \\leq j \\leq p}\\) where\n\\begin{equation}\n    t_{i_1, \\dots, i_p} = \\tns{T}[\\vek{e}_{i_1}, \\dots, \\vek{e}_{i_p}] \\quad \\text{and} \\quad \\tns{T} = \\sum_{i_1, \\dots, i_p = 1}^n t_{i_1, \\dots, i_p} \\vek{e}_{i_1} \\otimes \\dots \\otimes \\vek{e}_{i_p}.\n\\end{equation}\nThere is a Frobenius inner product and a corresponding norm on these \\(p\\)-dimensional arrays and by extension on \\(\\R^{\\otimes^p n}\\), which we will denote by \\(\\innerprod{\\tns{T}_1}{\\tns{T}_2}_F\\) and \\(\\norm{\\tns{T}}_F\\).\nNote that \\(\\innerprod{\\tns{T}}{\\vek{s}_1 \\otimes \\dots \\otimes \\vek{s}_p}_F = \\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p]\\).\nAnother norm is the one induced by the 2-norm on \\(\\R^n\\) (Hackbusch \\cite{hackbusch_tensor_2012} calls it the injective norm) which is defined by\n\\begin{equation}\n    \\norm{\\tns{T}}_2 = \\max_{\\norm{\\vek{s}_i}_2 = 1,\\, 1 \\leq i \\leq p} \\abs{\\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p]}.\n\\end{equation}\nBoth norms are invariant under orthogonal transformations, so that if \\(\\mat{Q}_1, \\dots, \\mat{Q}_p \\in \\R^{n \\times n}\\) are orthogonal matrices, then \\(\\tns{T}[\\mat{Q}_1, \\dots, \\mat{Q}_p]\\) has the same Frobenius- and 2-norm as \\(\\tns{T}\\).\nAs for matrices, the 2-norm is bounded by the Frobenius norm:\n\\begin{equation}\n    \\norm{\\tns{T}}_2\n    % = \\max_{\\norm{\\vek{s}_i}_2 = 1,\\, 1 \\leq i \\leq p} \\abs{\\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p]}\n    = \\max_{\\norm{\\vek{s}_i}_2 = 1,\\, 1 \\leq i \\leq p} \\abs{\\innerprod{\\tns{T}}{\\vek{s}_1 \\otimes \\dots \\otimes \\vek{s}_p}_F}\n    \\leq \\norm{\\tns{T}}_F.\n\\end{equation}\nIn the last inequality we used Cauchy-Schwarz and the fact that \\(\\norm{\\vek{s}_1 \\otimes \\dots \\otimes \\vek{s}_p}_F = 1\\) if all \\(\\vek{s}_i\\) have unit 2-norm.\n\n\nWe now need to assume that \\(f \\colon \\R^n \\to \\R\\) is \\(p\\) times continuously differentiable and that as before a sequence of points \\(\\vek{x}_k\\) with a corresponding sequence of steps \\(\\vek{s}_k = \\vek{x}_{k+1} - \\vek{x}_k\\) is given.\nWe will denote the approximations to \\(D^p f(\\vek{x}_k)\\) by \\(\\tns{C}_k \\in \\R^{\\otimes^p n}_{\\sym}\\) and the true \\(p\\)th derivative averaged over all points on the line from \\(\\vek{x}_k\\) to \\(\\vek{x}_{k+1}\\) by\n\\begin{equation}\\label{eqn:C-tilde-definition}\n    \\widetilde{\\tns{C}}_k = \\int_0^1 D^p f(\\vek{x}_k + t \\vek{s}_k) \\diff t  \\in \\R^{\\otimes^p n}_{\\sym}.\n\\end{equation}\nThis means that, in particular, \\(\\widetilde{\\tns{C}}_k[\\vek{s}_k] = D^{p-1} f(\\vek{x}_{k+1}) - D^{p-1} f(\\vek{x}_k)\\).\nThe \\(p\\)th-order analogue of the secant equation \\cref{eqn:short-secant-equation} is given by\n\\begin{equation}\\label{eqn:generalized-secant-equation}\n    \\tns{C}_{k+1} [\\vek{s}_k] = D^{p-1} f(\\vek{x}_{k+1}) - D^{p-1} f(\\vek{x}_k) = \\widetilde{\\tns{C}}_k [\\vek{s}_k]\n\\end{equation}\nand the generalized update formula for \\(\\tns{C}_k\\) reads\n\\begin{equation}\\label{eqn:update-definition}\n    \\tns{C}_{k+1} = \\argmin_{\\tns{C} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{(\\tns{C} - \\tns{C}_k)[\\mat{W}_k]^p}_F \\text{ s.t. } \\tns{C}[\\vek{s}_k] = \\widetilde{\\tns{C}}_k[\\vek{s}_k]. \\tag{HOSU}\n\\end{equation}\nEven though this update is derived from the updates used in quasi-Newton methods, it is not itself associated with any optimization method and any optimization algorithm using approximate third (or higher) derivatives is also clearly different from Newton's method.\nTo highlight this distinction we will call the update rule the \\emph{higher-order secant update} \\cref{eqn:update-definition} because of its connection with the (generalized) secant equation \\cref{eqn:generalized-secant-equation}.\nIt provides a sequence of approximations of the \\(p\\)th derivative of \\(f\\) based solely on evaluations of the \\((p-1)\\)st derivative at the iterates \\(\\vek{x}_k\\), given some initial approximation \\(\\tns{C}_0\\).\n\n\nThe following theorem provides a full characterization of one step of the update in \\cref{eqn:update-definition}.\nNote that despite the intentional notational similarity the quantities in the statement are independent of the definitions in the previous section.\n","units":[{"raw_tex":"The incredible success of quasi-Newton methods is largely based on the fact that the rules for updating the Hessian approximations are able to extract crucial second-order information from gradient evaluations.\nUnlike finite difference methods they do so without direct control over the evaluation points.\nRather, quasi-Newton rules are designed to handle evaluations of the first derivative at points that are generated by an extraneous process and produce best-effort approximations of the second derivative.\nWe propose generalizations of these rules that mimic the quasi-Newton approach but approximate \\(p\\)th derivatives from given evaluations of \\((p-1)\\)st derivatives for any \\(p \\geq 2\\).\n","start":7114,"end":7812},{"raw_tex":"A \\emph{\\(p\\)-tensor} \\(\\tns{T}\\) of dimensions \\(n_1 \\times \\dots \\times n_p\\) is a multilinear map \\(\\R^{n_1} \\times \\dots \\times \\R^{n_p}\\to \\R\\), so that its evaluation\n\\begin{equation}\n    \\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p], \\quad \\vek{s}_i \\in \\R^{n_i}\n\\end{equation}\nis linear in each component.\nWe refer to these \\(p\\) components as the \\emph{modes} of the tensor and denote the space of such \\(p\\)-tensors by \\(\\R^{n_1 \\otimes \\dots \\otimes n_p}\\).\nIf \\(n_1 = \\dots = n_p = n\\) the space is denoted \\(\\R^{\\otimes^p n}\\) and \\(\\tns{T}[\\vek{s}, \\dots, \\vek{s}]\\) is abbreviated as \\(\\tns{T}[\\vek{s}]^p\\).\nThe notation also allows to apply the tensor to \\(q < p\\) vectors, which then results in a \\((p-q)\\)-tensor.\nMoreover, we define the application of matrices \\(\\mat{W}_1, \\dots, \\mat{W}_p\\) of appropriate dimensions to a tensor by\n\\begin{equation}\n    \\big(\\tns{T}[\\mat{W}_1, \\dots, \\mat{W}_p]\\big)[\\vek{s}_1, \\dots, \\vek{s}_p] = \\tns{T}[\\mat{W}_1 \\vek{s}_1, \\dots, \\mat{W}_p \\vek{s}_p].\n\\end{equation}\n\nThe outer product of a \\(p_1\\)-tensor \\(\\tns{T}_1\\) with a \\(p_2\\)-tensor \\(\\tns{T}_2\\) is defined as\n\\begin{equation}\n    (\\tns{T}_1 \\otimes \\tns{T}_2)[\\vek{s}_1, \\dots, \\vek{s}_{p_1 + p_2}] = \\tns{T}_1[\\vek{s}_1, \\dots, \\vek{s}_{p_1}] \\cdot \\tns{T}_2[\\vek{s}_{p_1 + 1}, \\dots, \\vek{s}_{p_1 + p_2}].\n\\end{equation}\nIn particular, tensors of the form \\(\\tns{T} = \\vek{v}_1 \\otimes \\dots \\otimes \\vek{v}_p\\) for vectors \\(\\vek{v}_1, \\dots, \\vek{v}_p \\in \\R^n\\), i.e. those where \\(\\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p] = \\prod_{i = 1}^n \\vek{v}_i^T \\vek{s}_i\\), are called \\emph{elementary} or \\emph{rank-one} tensors.\nIf all vectors are the same (\\(\\vek{v}_1 = \\dots = \\vek{v}_p = \\vek{v}\\)), we abbreviate the notation above to \\(\\otimes^p \\vek{v}\\).\n(Note that this notation is slightly inconsistent since 1-tensors should be row vectors, but are represented by standard column vectors to simplify the notation.\nThis is why \\(\\vek{v}[\\mat{W}] = \\mat{W}^T \\vek{v}\\).)\n\nFor any \\(\\tns{T} \\in \\R^{\\otimes^p n}\\) and any permutation \\(\\sigma \\in S_p\\), let \\(\\sigma(\\tns{T}) \\in \\R^{\\otimes^p n}\\) be defined by\n\\begin{equation}\n    \\sigma(\\tns{T})[\\vek{s}_1, \\dots, \\vek{s}_p] = \\tns{T}[\\vek{s}_{\\sigma(1)}, \\dots, \\vek{s}_{\\sigma(p)}].\n\\end{equation}\nIf \\(\\sigma(\\tns{T}) = \\tns{T}\\) for all \\(\\sigma \\in S_p\\), then \\(\\tns{T}\\) is called \\emph{symmetric}.\nThe space of all symmetric \\(p\\)-tensors is denoted \\(\\R^{\\otimes^p n}_{\\sym}\\).\nThe projection of \\(\\R^{\\otimes^p n}\\) onto \\(\\R^{\\otimes^p n}_{\\sym}\\) is given by\n\\begin{equation}\n    P_{\\sym}(\\tns{T}) = \\frac{1}{p!} \\sum_{\\sigma \\in S_p} \\sigma(\\tns{T}),\n\\end{equation}\nsee \\cite[Proposition 3.76]{hackbusch_tensor_2012}.\n\nJust like matrices, tensors are fully characterized by their actions on basis vectors.\nThis can be used to represent a \\(p\\)-tensor \\(\\tns{T} \\in \\R^{\\otimes^p n}\\) as a \\(p\\)-dimensional array \\((t_{i_1, \\dots, i_p})_{1 \\leq i_j \\leq n, 1 \\leq j \\leq p}\\) where\n\\begin{equation}\n    t_{i_1, \\dots, i_p} = \\tns{T}[\\vek{e}_{i_1}, \\dots, \\vek{e}_{i_p}] \\quad \\text{and} \\quad \\tns{T} = \\sum_{i_1, \\dots, i_p = 1}^n t_{i_1, \\dots, i_p} \\vek{e}_{i_1} \\otimes \\dots \\otimes \\vek{e}_{i_p}.\n\\end{equation}\nThere is a Frobenius inner product and a corresponding norm on these \\(p\\)-dimensional arrays and by extension on \\(\\R^{\\otimes^p n}\\), which we will denote by \\(\\innerprod{\\tns{T}_1}{\\tns{T}_2}_F\\) and \\(\\norm{\\tns{T}}_F\\).\nNote that \\(\\innerprod{\\tns{T}}{\\vek{s}_1 \\otimes \\dots \\otimes \\vek{s}_p}_F = \\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p]\\).\nAnother norm is the one induced by the 2-norm on \\(\\R^n\\) (Hackbusch \\cite{hackbusch_tensor_2012} calls it the injective norm) which is defined by\n\\begin{equation}\n    \\norm{\\tns{T}}_2 = \\max_{\\norm{\\vek{s}_i}_2 = 1,\\, 1 \\leq i \\leq p} \\abs{\\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p]}.\n\\end{equation}\nBoth norms are invariant under orthogonal transformations, so that if \\(\\mat{Q}_1, \\dots, \\mat{Q}_p \\in \\R^{n \\times n}\\) are orthogonal matrices, then \\(\\tns{T}[\\mat{Q}_1, \\dots, \\mat{Q}_p]\\) has the same Frobenius- and 2-norm as \\(\\tns{T}\\).\nAs for matrices, the 2-norm is bounded by the Frobenius norm:\n\\begin{equation}\n    \\norm{\\tns{T}}_2\n    % = \\max_{\\norm{\\vek{s}_i}_2 = 1,\\, 1 \\leq i \\leq p} \\abs{\\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p]}\n    = \\max_{\\norm{\\vek{s}_i}_2 = 1,\\, 1 \\leq i \\leq p} \\abs{\\innerprod{\\tns{T}}{\\vek{s}_1 \\otimes \\dots \\otimes \\vek{s}_p}_F}\n    \\leq \\norm{\\tns{T}}_F.\n\\end{equation}\nIn the last inequality we used Cauchy-Schwarz and the fact that \\(\\norm{\\vek{s}_1 \\otimes \\dots \\otimes \\vek{s}_p}_F = 1\\) if all \\(\\vek{s}_i\\) have unit 2-norm.\n","start":10535,"end":15146},{"raw_tex":"We now need to assume that \\(f \\colon \\R^n \\to \\R\\) is \\(p\\) times continuously differentiable and that as before a sequence of points \\(\\vek{x}_k\\) with a corresponding sequence of steps \\(\\vek{s}_k = \\vek{x}_{k+1} - \\vek{x}_k\\) is given.\nWe will denote the approximations to \\(D^p f(\\vek{x}_k)\\) by \\(\\tns{C}_k \\in \\R^{\\otimes^p n}_{\\sym}\\) and the true \\(p\\)th derivative averaged over all points on the line from \\(\\vek{x}_k\\) to \\(\\vek{x}_{k+1}\\) by\n\\begin{equation}\\label{eqn:C-tilde-definition}\n    \\widetilde{\\tns{C}}_k = \\int_0^1 D^p f(\\vek{x}_k + t \\vek{s}_k) \\diff t  \\in \\R^{\\otimes^p n}_{\\sym}.\n\\end{equation}\nThis means that, in particular, \\(\\widetilde{\\tns{C}}_k[\\vek{s}_k] = D^{p-1} f(\\vek{x}_{k+1}) - D^{p-1} f(\\vek{x}_k)\\).\nThe \\(p\\)th-order analogue of the secant equation \\cref{eqn:short-secant-equation} is given by\n\\begin{equation}\\label{eqn:generalized-secant-equation}\n    \\tns{C}_{k+1} [\\vek{s}_k] = D^{p-1} f(\\vek{x}_{k+1}) - D^{p-1} f(\\vek{x}_k) = \\widetilde{\\tns{C}}_k [\\vek{s}_k]\n\\end{equation}\nand the generalized update formula for \\(\\tns{C}_k\\) reads\n\\begin{equation}\\label{eqn:update-definition}\n    \\tns{C}_{k+1} = \\argmin_{\\tns{C} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{(\\tns{C} - \\tns{C}_k)[\\mat{W}_k]^p}_F \\text{ s.t. } \\tns{C}[\\vek{s}_k] = \\widetilde{\\tns{C}}_k[\\vek{s}_k]. \\tag{HOSU}\n\\end{equation}\nEven though this update is derived from the updates used in quasi-Newton methods, it is not itself associated with any optimization method and any optimization algorithm using approximate third (or higher) derivatives is also clearly different from Newton's method.\nTo highlight this distinction we will call the update rule the \\emph{higher-order secant update} \\cref{eqn:update-definition} because of its connection with the (generalized) secant equation \\cref{eqn:generalized-secant-equation}.\nIt provides a sequence of approximations of the \\(p\\)th derivative of \\(f\\) based solely on evaluations of the \\((p-1)\\)st derivative at the iterates \\(\\vek{x}_k\\), given some initial approximation \\(\\tns{C}_0\\).\n","start":22334,"end":24379},{"raw_tex":"The following theorem provides a full characterization of one step of the update in \\cref{eqn:update-definition}.\nNote that despite the intentional notational similarity the quantities in the statement are independent of the definitions in the previous section.\n","start":24878,"end":25140}]},"statement":{"raw_tex":"\\begin{theorem}\\label{thm:characterization}\n    Let \\(\\tns{C}_{\\bullet} \\in \\R^{\\otimes^p n}_{\\sym}\\) (the current approximation), \\(\\widetilde{\\tns{C}} \\in \\R^{\\otimes^p n}_{\\sym}\\) (the integrated true derivative), a nonsingular matrix \\(\\mat{W} \\in \\R^{n \\times n}\\) (the weight matrix) and a nonzero \\(\\vek{s} \\in \\R^n\\) (the step) be given.\n    The following equations all have the same unique solution \\(\\tns{C}_+ \\in \\R^{\\otimes^p n}_{\\sym}\\) (the new approximation):\n    \\begin{enumerate}[(a)]\n        \\item\\label{item:characterization-min-frobenius} \\(\\tns{C}_+ = \\argmin_{\\tns{C} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{(\\tns{C} - \\tns{C}_{\\bullet})[\\mat{W}]^p}_F \\text{ s.t. } \\tns{C}[\\vek{s}] = \\widetilde{\\tns{C}}[\\vek{s}]\\)\n        \\item\\label{item:characterization-explicit} \\(\\tns{C}_+ = \\tns{C}_{\\bullet} + \\sum_{j=1}^p (-1)^{j+1} \\binom{p}{j} (\\vek{v}^T \\vek{s})^{-j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{v}} \\otimes (\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]^{j}}\\)\n        \\item\\label{item:characterization-low-rank} \\(\\tns{C}_+ = \\tns{C}_{\\bullet} + P_{\\sym}(\\tns{A} \\otimes \\vek{v})\\) for the unique \\((p-1)\\)-tensor \\(\\tns{A} \\in \\R^{\\otimes^{p-1} n}_{\\sym}\\) which satisfies \\(P_{\\sym}(\\tns{A} \\otimes \\vek{v})[\\vek{s}] = (\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]\\)\n        \\item\\label{item:characterization-recursive} \\((\\tns{C}_+ - \\widetilde{\\tns{C}})[\\mat{W}]^p = (\\tns{C}_{\\bullet} - \\widetilde{\\tns{C}})[\\mat{W}]^p \\Brack{\\mat{I} - \\frac{\\mat{W}^{-1}\\vek{s}\\vek{s}^T \\mat{W}^{-T}}{\\vek{s}^T \\mat{W}^{-T} \\mat{W}^{-1} \\vek{s}}}^p\\)\n    \\end{enumerate}\n    where \\(\\vek{v} = \\mat{W}^{-T}\\mat{W}^{-1}\\vek{s}\\).\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\\label{thm:characterization}\n    Let \\(\\tns{C}_{\\bullet} \\in \\R^{\\otimes^p n}_{\\sym}\\) (the current approximation), \\(\\widetilde{\\tns{C}} \\in \\R^{\\otimes^p n}_{\\sym}\\) (the integrated true derivative), a nonsingular matrix \\(\\mat{W} \\in \\R^{n \\times n}\\) (the weight matrix) and a nonzero \\(\\vek{s} \\in \\R^n\\) (the step) be given.\n    The following equations all have the same unique solution \\(\\tns{C}_+ \\in \\R^{\\otimes^p n}_{\\sym}\\) (the new approximation):\n    \\begin{enumerate}[(a)]\n        \\item\\label{item:characterization-min-frobenius} \\(\\tns{C}_+ = \\argmin_{\\tns{C} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{(\\tns{C} - \\tns{C}_{\\bullet})[\\mat{W}]^p}_F \\text{ s.t. } \\tns{C}[\\vek{s}] = \\widetilde{\\tns{C}}[\\vek{s}]\\)\n        \\item\\label{item:characterization-explicit} \\(\\tns{C}_+ = \\tns{C}_{\\bullet} + \\sum_{j=1}^p (-1)^{j+1} \\binom{p}{j} (\\vek{v}^T \\vek{s})^{-j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{v}} \\otimes (\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]^{j}}\\)\n        \\item\\label{item:characterization-low-rank} \\(\\tns{C}_+ = \\tns{C}_{\\bullet} + P_{\\sym}(\\tns{A} \\otimes \\vek{v})\\) for the unique \\((p-1)\\)-tensor \\(\\tns{A} \\in \\R^{\\otimes^{p-1} n}_{\\sym}\\) which satisfies \\(P_{\\sym}(\\tns{A} \\otimes \\vek{v})[\\vek{s}] = (\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]\\)\n        \\item\\label{item:characterization-recursive} \\((\\tns{C}_+ - \\widetilde{\\tns{C}})[\\mat{W}]^p = (\\tns{C}_{\\bullet} - \\widetilde{\\tns{C}})[\\mat{W}]^p \\Brack{\\mat{I} - \\frac{\\mat{W}^{-1}\\vek{s}\\vek{s}^T \\mat{W}^{-T}}{\\vek{s}^T \\mat{W}^{-T} \\mat{W}^{-1} \\vek{s}}}^p\\)\n    \\end{enumerate}\n    where \\(\\vek{v} = \\mat{W}^{-T}\\mat{W}^{-1}\\vek{s}\\).\n\\end{theorem}","start":25141,"end":26805}]},"proof":{"raw_tex":"\\begin{proof}\n    We will first prove the result for \\(\\mat{W} = \\mat{I}\\) and then see how that implies the full result for any nonsingular weight matrix \\(\\mat{W}\\).\n\n    The first characterization can be rewritten as\n    \\begin{equation}\\label{eqn:general-psb-update}\n        \\tns{C}_+ = \\tns{C}_{\\bullet} + \\argmin_{\\tns{U} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{\\tns{U}}_F \\text{ s.t. } \\tns{U}[\\vek{s}] = (\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}].\n    \\end{equation}\n    Let \\(\\mat{Q} \\in \\R^{n \\times n}\\) be an orthogonal matrix that maps \\(\\vek{e}_1\\) to some scalar multiple of \\(\\vek{s}\\).\n    This means \\(\\tns{U}[\\mat{Q}]^p\\) has the same Frobenius norm as \\(\\tns{U}\\) and \\(\\tns{U}[\\mat{Q}]^p[\\vek{e}_{i_1}, \\dots, \\vek{e}_{i_p}]\\) is fully determined by the equality constraint in \\cref{eqn:general-psb-update} if \\(1 \\in \\{i_1, \\dots, i_p\\}\\) because of symmetry.\n    On the other hand, if \\(1 \\notin \\{i_1, \\dots, i_p\\}\\) there are no constraints on the values of \\(\\tns{U}[\\mat{Q}]^p[\\vek{e}_{i_1}, \\dots, \\vek{e}_{i_p}]\\), so the unique choice that minimizes \\(\\norm{\\tns{U}[\\mat{Q}]^p}_F = \\norm{\\tns{U}}_F\\) is clearly to set all of these values to zero.\n    Therefore, there is a unique solution to the minimization problem in \\labelcref{item:characterization-min-frobenius}, and it is fully characterized by the fact that the update tensor \\(\\tns{U} = \\tns{C}_+ - \\tns{C}_{\\bullet}\\) is symmetric and satisfies the following two properties:\n    \\begin{subequations}\n        \\begin{align}\n            \\tns{U}[\\vek{s}]                     & = \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}] \\label{eqn:update-secant-equation}        \\\\\n            \\tns{U}[\\vek{u}_1, \\dots, \\vek{u}_p] & = 0 \\text{ if all } \\vek{u}_i \\text{ are orthogonal to } \\vek{s} \\label{eqn:update-minimum-property}\n        \\end{align}\n    \\end{subequations}\n    We will use this as the basis to show the equivalence with all other characterizations.\n\n    In \\labelcref{item:characterization-explicit} we claim that the update \\(\\tns{U}\\) has the form\n    \\begin{equation}\n        \\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}\n    \\end{equation}\n    This tensor is clearly symmetric since it is a sum of symmetric tensors.\n    To show that it satisfies property \\cref{eqn:update-secant-equation} consider\n    \\begin{subequations}\\label{eqn:update-explicit-cancellation}\n        \\begin{align}\n             & \\phantom{={}}\\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}[\\vek{s}]                                                                                                                                                \\\\\n            \\begin{split}\n                {} = \\sum_{j=1}^p (-1)^{j+1} \\norm{\\vek{s}}_2^{-2j} \\bigg( \\tbinom{p-1}{j-1} P_{\\sym} \\Paren{\\Paren{\\otimes^{j-1} \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}} \\norm{\\vek{s}}_2^2 \\phantom{\\bigg)}\\\\\n                {} + \\tbinom{p-1}{j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j+1}} \\bigg) \\phantom{\\norm{\\vek{s}_2^2}}\n            \\end{split} \\\\\n             & = P_{\\sym} \\Paren{ \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}] } = \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}].\n        \\end{align}\n    \\end{subequations}\n    The first equality uses the fact that each summand is the symmetric projection of an outer product of two symmetric tensors, a \\(j\\)-tensor and a \\((p-j)\\)-tensor.\n    Therefore, there are \\(\\binom{p}{j}\\) distinct ways to orient this outer product and for \\(\\binom{p-1}{j-1}\\) of them the vector \\(\\vek{s}\\) is applied to the \\(j\\)-tensor and for \\(\\binom{p-1}{j}\\) to the \\((p-j)\\)-tensor.\n    A close examination of the expression on the second line shows that it is a telescoping sum where all terms except the ones with coefficients \\(\\binom{p-1}{0}\\) and \\(\\binom{p-1}{p}\\) cancel out.\n    Because \\(\\binom{p-1}{p} = 0\\) the only remaining term is the symmetric projection of \\((\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]\\).\n    Property \\cref{eqn:update-minimum-property} follows from a similar consideration to the one above.\n    If \\(\\vek{u}_1, \\dots, \\vek{u}_p\\) are all orthogonal to \\(\\vek{s}\\), then\n    \\begin{equation}\n        P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}[\\vek{u}_1, \\dots, \\vek{u}_p] = 0\n    \\end{equation}\n    for \\(j \\geq 1\\) because no matter how the outer product is oriented there is always a factor of \\(\\vek{s}^T \\vek{u}_i = 0\\) in the result.\n\n    For \\labelcref{item:characterization-low-rank} we need to show that we can always write the update in the form \\(P_{\\sym}(\\tns{A} \\otimes \\vek{s})\\) for some symmetric \\((p-1)\\)-tensor \\(\\tns{A}\\) and that \\(\\tns{A}\\) is unique such that the corresponding update satisfies the secant equation \\cref{eqn:update-secant-equation}.\n    Note that the expression for the update in \\labelcref{item:characterization-explicit} is already of the form \\(P_{\\sym}(\\tns{A} \\otimes \\vek{s})\\):\n    \\begin{subequations}\n        \\begin{align}\n             & \\phantom{={}}\\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}                               \\\\\n             & = P_{\\sym} \\Paren{ P_{\\sym} \\Paren{ \\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} \\Paren{\\otimes^{j-1} \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j} } \\otimes \\vek{s} }\n        \\end{align}\n    \\end{subequations}\n    It remains to show that among all updates of the form \\(P_{\\sym}(\\tns{A} \\otimes \\vek{s})\\) there is only one choice of \\(\\tns{A} \\in \\R^{\\otimes^p n}_{\\sym}\\) such that \\cref{eqn:update-secant-equation} holds.\n    Consider the linear map\n    \\begin{align*}\n        \\phi \\colon \\R^{\\otimes^{p-1} n}_{\\sym} & \\to \\R^{\\otimes^{p-1} n}_{\\sym}                    \\\\\n        \\tns{A}                                 & \\mapsto P_{\\sym}(\\tns{A} \\otimes \\vek{s})[\\vek{s}]\n    \\end{align*}\n    which maps a finite-dimensional vector space to itself.\n    Combining what we already showed for \\labelcref{item:characterization-explicit} with the observation that the update in \\labelcref{item:characterization-explicit} is of the desired form, this map is surjective (we can prescribe any \\((\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]\\)) and so it must be bijective, which shows uniqueness of \\(\\tns{A}\\).\n\n    Lastly, \\labelcref{item:characterization-recursive} claims that the update can be written as\n    \\begin{equation}\n        \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}} - \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}} \\Brack{\\mat{I} - \\frac{\\vek{s} \\vek{s}^T}{\\norm{\\vek{s}}_2^2}}^p.\n    \\end{equation}\n    Clearly, property \\cref{eqn:update-secant-equation} is satisfied because applying this tensor to \\(\\vek{s}\\) makes the second term vanish, leaving only the desired result.\n    Moreover, for any vector \\(\\vek{u}\\) that is orthogonal to \\(\\vek{s}\\) the matrix \\(\\mat{I} - \\frac{\\vek{s} \\vek{s}^T}{\\norm{\\vek{s}}_2^2}\\) maps \\(\\vek{u}\\) to itself.\n    This means applying the tensor above to \\(\\vek{u}_1, \\dots, \\vek{u}_p\\), all of which are orthogonal to \\(\\vek{s}\\), will give zero because both terms cancel out.\n    This is property \\cref{eqn:update-minimum-property}.\n\n    Now that the equivalence has been established for \\(\\mat{W} = \\mat{I}\\), we consider the general case where \\(\\mat{W} \\in \\R^{n \\times n}\\) is any nonsingular matrix.\n    The minimization in \\labelcref{item:characterization-min-frobenius}\n    \\begin{equation}\n        \\tns{C}_+ = \\argmin_{\\tns{C} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{(\\tns{C} - \\tns{C}_{\\bullet})[\\mat{W}]^p}_F \\text{ s.t. } \\tns{C}[\\vek{s}] = \\widetilde{\\tns{C}}[\\vek{s}]\n    \\end{equation}\n    can be rewritten using \\(\\tns{D}_{\\bullet} = \\tns{C}_{\\bullet}[\\mat{W}]^p\\), \\(\\widetilde{\\tns{D}} = \\widetilde{\\tns{C}}[\\mat{W}]^p\\), \\(\\tns{D}_+ = \\tns{C}_+[\\mat{W}]^p\\) and \\(\\vek{r} = \\mat{W}^{-1}\\vek{s}\\) as\n    \\begin{equation}\n        \\tns{D}_+ = \\argmin_{\\tns{D} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{\\tns{D} - \\tns{D}_{\\bullet}}_F \\text{ s.t. } \\tns{D}[\\vek{r}] = \\widetilde{\\tns{D}}[\\vek{r}].\n    \\end{equation}\n    Applying the existing characterizations and the fact that \\(\\tns{C}_+ = \\tns{D}_+[\\mat{W}^{-1}]^p\\) we get the claim after some algebraic manipulations.\n\\end{proof}","units":[{"raw_tex":"\\begin{proof}\n    We will first prove the result for \\(\\mat{W} = \\mat{I}\\) and then see how that implies the full result for any nonsingular weight matrix \\(\\mat{W}\\).\n\n    The first characterization can be rewritten as\n    \\begin{equation}\\label{eqn:general-psb-update}\n        \\tns{C}_+ = \\tns{C}_{\\bullet} + \\argmin_{\\tns{U} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{\\tns{U}}_F \\text{ s.t. } \\tns{U}[\\vek{s}] = (\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}].\n    \\end{equation}\n    Let \\(\\mat{Q} \\in \\R^{n \\times n}\\) be an orthogonal matrix that maps \\(\\vek{e}_1\\) to some scalar multiple of \\(\\vek{s}\\).\n    This means \\(\\tns{U}[\\mat{Q}]^p\\) has the same Frobenius norm as \\(\\tns{U}\\) and \\(\\tns{U}[\\mat{Q}]^p[\\vek{e}_{i_1}, \\dots, \\vek{e}_{i_p}]\\) is fully determined by the equality constraint in \\cref{eqn:general-psb-update} if \\(1 \\in \\{i_1, \\dots, i_p\\}\\) because of symmetry.\n    On the other hand, if \\(1 \\notin \\{i_1, \\dots, i_p\\}\\) there are no constraints on the values of \\(\\tns{U}[\\mat{Q}]^p[\\vek{e}_{i_1}, \\dots, \\vek{e}_{i_p}]\\), so the unique choice that minimizes \\(\\norm{\\tns{U}[\\mat{Q}]^p}_F = \\norm{\\tns{U}}_F\\) is clearly to set all of these values to zero.\n    Therefore, there is a unique solution to the minimization problem in \\labelcref{item:characterization-min-frobenius}, and it is fully characterized by the fact that the update tensor \\(\\tns{U} = \\tns{C}_+ - \\tns{C}_{\\bullet}\\) is symmetric and satisfies the following two properties:\n    \\begin{subequations}\n        \\begin{align}\n            \\tns{U}[\\vek{s}]                     & = \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}] \\label{eqn:update-secant-equation}        \\\\\n            \\tns{U}[\\vek{u}_1, \\dots, \\vek{u}_p] & = 0 \\text{ if all } \\vek{u}_i \\text{ are orthogonal to } \\vek{s} \\label{eqn:update-minimum-property}\n        \\end{align}\n    \\end{subequations}\n    We will use this as the basis to show the equivalence with all other characterizations.\n\n    In \\labelcref{item:characterization-explicit} we claim that the update \\(\\tns{U}\\) has the form\n    \\begin{equation}\n        \\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}\n    \\end{equation}\n    This tensor is clearly symmetric since it is a sum of symmetric tensors.\n    To show that it satisfies property \\cref{eqn:update-secant-equation} consider\n    \\begin{subequations}\\label{eqn:update-explicit-cancellation}\n        \\begin{align}\n             & \\phantom{={}}\\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}[\\vek{s}]                                                                                                                                                \\\\\n            \\begin{split}\n                {} = \\sum_{j=1}^p (-1)^{j+1} \\norm{\\vek{s}}_2^{-2j} \\bigg( \\tbinom{p-1}{j-1} P_{\\sym} \\Paren{\\Paren{\\otimes^{j-1} \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}} \\norm{\\vek{s}}_2^2 \\phantom{\\bigg)}\\\\\n                {} + \\tbinom{p-1}{j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j+1}} \\bigg) \\phantom{\\norm{\\vek{s}_2^2}}\n            \\end{split} \\\\\n             & = P_{\\sym} \\Paren{ \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}] } = \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}].\n        \\end{align}\n    \\end{subequations}\n    The first equality uses the fact that each summand is the symmetric projection of an outer product of two symmetric tensors, a \\(j\\)-tensor and a \\((p-j)\\)-tensor.\n    Therefore, there are \\(\\binom{p}{j}\\) distinct ways to orient this outer product and for \\(\\binom{p-1}{j-1}\\) of them the vector \\(\\vek{s}\\) is applied to the \\(j\\)-tensor and for \\(\\binom{p-1}{j}\\) to the \\((p-j)\\)-tensor.\n    A close examination of the expression on the second line shows that it is a telescoping sum where all terms except the ones with coefficients \\(\\binom{p-1}{0}\\) and \\(\\binom{p-1}{p}\\) cancel out.\n    Because \\(\\binom{p-1}{p} = 0\\) the only remaining term is the symmetric projection of \\((\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]\\).\n    Property \\cref{eqn:update-minimum-property} follows from a similar consideration to the one above.\n    If \\(\\vek{u}_1, \\dots, \\vek{u}_p\\) are all orthogonal to \\(\\vek{s}\\), then\n    \\begin{equation}\n        P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}[\\vek{u}_1, \\dots, \\vek{u}_p] = 0\n    \\end{equation}\n    for \\(j \\geq 1\\) because no matter how the outer product is oriented there is always a factor of \\(\\vek{s}^T \\vek{u}_i = 0\\) in the result.\n\n    For \\labelcref{item:characterization-low-rank} we need to show that we can always write the update in the form \\(P_{\\sym}(\\tns{A} \\otimes \\vek{s})\\) for some symmetric \\((p-1)\\)-tensor \\(\\tns{A}\\) and that \\(\\tns{A}\\) is unique such that the corresponding update satisfies the secant equation \\cref{eqn:update-secant-equation}.\n    Note that the expression for the update in \\labelcref{item:characterization-explicit} is already of the form \\(P_{\\sym}(\\tns{A} \\otimes \\vek{s})\\):\n    \\begin{subequations}\n        \\begin{align}\n             & \\phantom{={}}\\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}                               \\\\\n             & = P_{\\sym} \\Paren{ P_{\\sym} \\Paren{ \\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} \\Paren{\\otimes^{j-1} \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j} } \\otimes \\vek{s} }\n        \\end{align}\n    \\end{subequations}\n    It remains to show that among all updates of the form \\(P_{\\sym}(\\tns{A} \\otimes \\vek{s})\\) there is only one choice of \\(\\tns{A} \\in \\R^{\\otimes^p n}_{\\sym}\\) such that \\cref{eqn:update-secant-equation} holds.\n    Consider the linear map\n    \\begin{align*}\n        \\phi \\colon \\R^{\\otimes^{p-1} n}_{\\sym} & \\to \\R^{\\otimes^{p-1} n}_{\\sym}                    \\\\\n        \\tns{A}                                 & \\mapsto P_{\\sym}(\\tns{A} \\otimes \\vek{s})[\\vek{s}]\n    \\end{align*}\n    which maps a finite-dimensional vector space to itself.\n    Combining what we already showed for \\labelcref{item:characterization-explicit} with the observation that the update in \\labelcref{item:characterization-explicit} is of the desired form, this map is surjective (we can prescribe any \\((\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]\\)) and so it must be bijective, which shows uniqueness of \\(\\tns{A}\\).\n\n    Lastly, \\labelcref{item:characterization-recursive} claims that the update can be written as\n    \\begin{equation}\n        \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}} - \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}} \\Brack{\\mat{I} - \\frac{\\vek{s} \\vek{s}^T}{\\norm{\\vek{s}}_2^2}}^p.\n    \\end{equation}\n    Clearly, property \\cref{eqn:update-secant-equation} is satisfied because applying this tensor to \\(\\vek{s}\\) makes the second term vanish, leaving only the desired result.\n    Moreover, for any vector \\(\\vek{u}\\) that is orthogonal to \\(\\vek{s}\\) the matrix \\(\\mat{I} - \\frac{\\vek{s} \\vek{s}^T}{\\norm{\\vek{s}}_2^2}\\) maps \\(\\vek{u}\\) to itself.\n    This means applying the tensor above to \\(\\vek{u}_1, \\dots, \\vek{u}_p\\), all of which are orthogonal to \\(\\vek{s}\\), will give zero because both terms cancel out.\n    This is property \\cref{eqn:update-minimum-property}.\n\n    Now that the equivalence has been established for \\(\\mat{W} = \\mat{I}\\), we consider the general case where \\(\\mat{W} \\in \\R^{n \\times n}\\) is any nonsingular matrix.\n    The minimization in \\labelcref{item:characterization-min-frobenius}\n    \\begin{equation}\n        \\tns{C}_+ = \\argmin_{\\tns{C} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{(\\tns{C} - \\tns{C}_{\\bullet})[\\mat{W}]^p}_F \\text{ s.t. } \\tns{C}[\\vek{s}] = \\widetilde{\\tns{C}}[\\vek{s}]\n    \\end{equation}\n    can be rewritten using \\(\\tns{D}_{\\bullet} = \\tns{C}_{\\bullet}[\\mat{W}]^p\\), \\(\\widetilde{\\tns{D}} = \\widetilde{\\tns{C}}[\\mat{W}]^p\\), \\(\\tns{D}_+ = \\tns{C}_+[\\mat{W}]^p\\) and \\(\\vek{r} = \\mat{W}^{-1}\\vek{s}\\) as\n    \\begin{equation}\n        \\tns{D}_+ = \\argmin_{\\tns{D} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{\\tns{D} - \\tns{D}_{\\bullet}}_F \\text{ s.t. } \\tns{D}[\\vek{r}] = \\widetilde{\\tns{D}}[\\vek{r}].\n    \\end{equation}\n    Applying the existing characterizations and the fact that \\(\\tns{C}_+ = \\tns{D}_+[\\mat{W}^{-1}]^p\\) we get the claim after some algebraic manipulations.\n\\end{proof}","start":26807,"end":35513}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"拟牛顿更新的动机","source_units":[{"raw_tex":"The incredible success of quasi-Newton methods is largely based on the fact that the rules for updating the Hessian approximations are able to extract crucial second-order information from gradient evaluations.\nUnlike finite difference methods they do so without direct control over the evaluation points.\nRather, quasi-Newton rules are designed to handle evaluations of the first derivative at points that are generated by an extraneous process and produce best-effort approximations of the second derivative.\n","start":7114,"end":7625}]},{"id":"S002","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"高阶近似的阶数范围","source_units":[{"raw_tex":"We propose generalizations of these rules that mimic the quasi-Newton approach but approximate \\(p\\)th derivatives from given evaluations of \\((p-1)\\)st derivatives for any \\(p \\geq 2\\).\n","start":7625,"end":7812}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"张量空间与向量收缩","source_units":[{"raw_tex":"A \\emph{\\(p\\)-tensor} \\(\\tns{T}\\) of dimensions \\(n_1 \\times \\dots \\times n_p\\) is a multilinear map \\(\\R^{n_1} \\times \\dots \\times \\R^{n_p}\\to \\R\\), so that its evaluation\n\\begin{equation}\n    \\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p], \\quad \\vek{s}_i \\in \\R^{n_i}\n\\end{equation}\nis linear in each component.\nWe refer to these \\(p\\) components as the \\emph{modes} of the tensor and denote the space of such \\(p\\)-tensors by \\(\\R^{n_1 \\otimes \\dots \\otimes n_p}\\).\nIf \\(n_1 = \\dots = n_p = n\\) the space is denoted \\(\\R^{\\otimes^p n}\\) and \\(\\tns{T}[\\vek{s}, \\dots, \\vek{s}]\\) is abbreviated as \\(\\tns{T}[\\vek{s}]^p\\).\nThe notation also allows to apply the tensor to \\(q < p\\) vectors, which then results in a \\((p-q)\\)-tensor.\n","start":10535,"end":11258}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"张量的矩阵作用","source_units":[{"raw_tex":"Moreover, we define the application of matrices \\(\\mat{W}_1, \\dots, \\mat{W}_p\\) of appropriate dimensions to a tensor by\n\\begin{equation}\n    \\big(\\tns{T}[\\mat{W}_1, \\dots, \\mat{W}_p]\\big)[\\vek{s}_1, \\dots, \\vek{s}_p] = \\tns{T}[\\mat{W}_1 \\vek{s}_1, \\dots, \\mat{W}_p \\vek{s}_p].\n\\end{equation}\n\n","start":11258,"end":11552}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"外积与向量表示约定","source_units":[{"raw_tex":"The outer product of a \\(p_1\\)-tensor \\(\\tns{T}_1\\) with a \\(p_2\\)-tensor \\(\\tns{T}_2\\) is defined as\n\\begin{equation}\n    (\\tns{T}_1 \\otimes \\tns{T}_2)[\\vek{s}_1, \\dots, \\vek{s}_{p_1 + p_2}] = \\tns{T}_1[\\vek{s}_1, \\dots, \\vek{s}_{p_1}] \\cdot \\tns{T}_2[\\vek{s}_{p_1 + 1}, \\dots, \\vek{s}_{p_1 + p_2}].\n\\end{equation}\nIn particular, tensors of the form \\(\\tns{T} = \\vek{v}_1 \\otimes \\dots \\otimes \\vek{v}_p\\) for vectors \\(\\vek{v}_1, \\dots, \\vek{v}_p \\in \\R^n\\), i.e. those where \\(\\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p] = \\prod_{i = 1}^n \\vek{v}_i^T \\vek{s}_i\\), are called \\emph{elementary} or \\emph{rank-one} tensors.\nIf all vectors are the same (\\(\\vek{v}_1 = \\dots = \\vek{v}_p = \\vek{v}\\)), we abbreviate the notation above to \\(\\otimes^p \\vek{v}\\).\n(Note that this notation is slightly inconsistent since 1-tensors should be row vectors, but are represented by standard column vectors to simplify the notation.\nThis is why \\(\\vek{v}[\\mat{W}] = \\mat{W}^T \\vek{v}\\).)\n","start":11552,"end":12520}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"置换与对称张量","source_units":[{"raw_tex":"\nFor any \\(\\tns{T} \\in \\R^{\\otimes^p n}\\) and any permutation \\(\\sigma \\in S_p\\), let \\(\\sigma(\\tns{T}) \\in \\R^{\\otimes^p n}\\) be defined by\n\\begin{equation}\n    \\sigma(\\tns{T})[\\vek{s}_1, \\dots, \\vek{s}_p] = \\tns{T}[\\vek{s}_{\\sigma(1)}, \\dots, \\vek{s}_{\\sigma(p)}].\n\\end{equation}\nIf \\(\\sigma(\\tns{T}) = \\tns{T}\\) for all \\(\\sigma \\in S_p\\), then \\(\\tns{T}\\) is called \\emph{symmetric}.\nThe space of all symmetric \\(p\\)-tensors is denoted \\(\\R^{\\otimes^p n}_{\\sym}\\).\n","start":12520,"end":12989}]},{"id":"S007","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先修列表第一项；原文 L244 引用 Hackbusch, Tensor Spaces and Numerical Tensor Calculus, Proposition 3.76，cite key: hackbusch_tensor_2012。"],"source_only":false,"title":"对称化投影公式","source_units":[{"raw_tex":"The projection of \\(\\R^{\\otimes^p n}\\) onto \\(\\R^{\\otimes^p n}_{\\sym}\\) is given by\n\\begin{equation}\n    P_{\\sym}(\\tns{T}) = \\frac{1}{p!} \\sum_{\\sigma \\in S_p} \\sigma(\\tns{T}),\n\\end{equation}\nsee \\cite[Proposition 3.76]{hackbusch_tensor_2012}.\n","start":12989,"end":13233}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"基评价与坐标表示","source_units":[{"raw_tex":"\nJust like matrices, tensors are fully characterized by their actions on basis vectors.\nThis can be used to represent a \\(p\\)-tensor \\(\\tns{T} \\in \\R^{\\otimes^p n}\\) as a \\(p\\)-dimensional array \\((t_{i_1, \\dots, i_p})_{1 \\leq i_j \\leq n, 1 \\leq j \\leq p}\\) where\n\\begin{equation}\n    t_{i_1, \\dots, i_p} = \\tns{T}[\\vek{e}_{i_1}, \\dots, \\vek{e}_{i_p}] \\quad \\text{and} \\quad \\tns{T} = \\sum_{i_1, \\dots, i_p = 1}^n t_{i_1, \\dots, i_p} \\vek{e}_{i_1} \\otimes \\dots \\otimes \\vek{e}_{i_p}.\n\\end{equation}\n","start":13233,"end":13733}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"Frobenius 内积与范数","source_units":[{"raw_tex":"There is a Frobenius inner product and a corresponding norm on these \\(p\\)-dimensional arrays and by extension on \\(\\R^{\\otimes^p n}\\), which we will denote by \\(\\innerprod{\\tns{T}_1}{\\tns{T}_2}_F\\) and \\(\\norm{\\tns{T}}_F\\).\nNote that \\(\\innerprod{\\tns{T}}{\\vek{s}_1 \\otimes \\dots \\otimes \\vek{s}_p}_F = \\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p]\\).\n","start":13733,"end":14077}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"张量诱导二范数","source_units":[{"raw_tex":"Another norm is the one induced by the 2-norm on \\(\\R^n\\) (Hackbusch \\cite{hackbusch_tensor_2012} calls it the injective norm) which is defined by\n\\begin{equation}\n    \\norm{\\tns{T}}_2 = \\max_{\\norm{\\vek{s}_i}_2 = 1,\\, 1 \\leq i \\leq p} \\abs{\\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p]}.\n\\end{equation}\n","start":14077,"end":14372}]},{"id":"S011","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"Frobenius 范数的正交不变性","source_units":[{"raw_tex":"Both norms are invariant under orthogonal transformations, so that if \\(\\mat{Q}_1, \\dots, \\mat{Q}_p \\in \\R^{n \\times n}\\) are orthogonal matrices, then \\(\\tns{T}[\\mat{Q}_1, \\dots, \\mat{Q}_p]\\) has the same Frobenius- and 2-norm as \\(\\tns{T}\\).\n","start":14372,"end":14616}]},{"id":"S012","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"二范数被 Frobenius 范数控制","source_units":[{"raw_tex":"As for matrices, the 2-norm is bounded by the Frobenius norm:\n\\begin{equation}\n    \\norm{\\tns{T}}_2\n    % = \\max_{\\norm{\\vek{s}_i}_2 = 1,\\, 1 \\leq i \\leq p} \\abs{\\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p]}\n    = \\max_{\\norm{\\vek{s}_i}_2 = 1,\\, 1 \\leq i \\leq p} \\abs{\\innerprod{\\tns{T}}{\\vek{s}_1 \\otimes \\dots \\otimes \\vek{s}_p}_F}\n    \\leq \\norm{\\tns{T}}_F.\n\\end{equation}\nIn the last inequality we used Cauchy-Schwarz and the fact that \\(\\norm{\\vek{s}_1 \\otimes \\dots \\otimes \\vek{s}_p}_F = 1\\) if all \\(\\vek{s}_i\\) have unit 2-norm.\n","start":14616,"end":15146}]},{"id":"S013","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"范数比较的作者证明","source_units":[{"raw_tex":"As for matrices, the 2-norm is bounded by the Frobenius norm:\n\\begin{equation}\n    \\norm{\\tns{T}}_2\n    % = \\max_{\\norm{\\vek{s}_i}_2 = 1,\\, 1 \\leq i \\leq p} \\abs{\\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p]}\n    = \\max_{\\norm{\\vek{s}_i}_2 = 1,\\, 1 \\leq i \\leq p} \\abs{\\innerprod{\\tns{T}}{\\vek{s}_1 \\otimes \\dots \\otimes \\vek{s}_p}_F}\n    \\leq \\norm{\\tns{T}}_F.\n\\end{equation}\nIn the last inequality we used Cauchy-Schwarz and the fact that \\(\\norm{\\vek{s}_1 \\otimes \\dots \\otimes \\vek{s}_p}_F = 1\\) if all \\(\\vek{s}_i\\) have unit 2-norm.\n","start":14616,"end":15146}]},{"id":"S014","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"导数近似的应用设定","source_units":[{"raw_tex":"We now need to assume that \\(f \\colon \\R^n \\to \\R\\) is \\(p\\) times continuously differentiable and that as before a sequence of points \\(\\vek{x}_k\\) with a corresponding sequence of steps \\(\\vek{s}_k = \\vek{x}_{k+1} - \\vek{x}_k\\) is given.\n","start":22334,"end":22574}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":["eqn:C-tilde-definition"],"declared_external":false,"source_only":false,"title":"当前近似与平均导数","source_units":[{"raw_tex":"We will denote the approximations to \\(D^p f(\\vek{x}_k)\\) by \\(\\tns{C}_k \\in \\R^{\\otimes^p n}_{\\sym}\\) and the true \\(p\\)th derivative averaged over all points on the line from \\(\\vek{x}_k\\) to \\(\\vek{x}_{k+1}\\) by\n\\begin{equation}\\label{eqn:C-tilde-definition}\n    \\widetilde{\\tns{C}}_k = \\int_0^1 D^p f(\\vek{x}_k + t \\vek{s}_k) \\diff t  \\in \\R^{\\otimes^p n}_{\\sym}.\n\\end{equation}\n","start":22574,"end":22957}]},{"id":"S016","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"平均导数收缩给出导数差","source_units":[{"raw_tex":"This means that, in particular, \\(\\widetilde{\\tns{C}}_k[\\vek{s}_k] = D^{p-1} f(\\vek{x}_{k+1}) - D^{p-1} f(\\vek{x}_k)\\).\n","start":22957,"end":23077}]},{"id":"S017","semantic_type":"definition","kind":"definition","labels":["eqn:generalized-secant-equation"],"declared_external":false,"source_only":false,"title":"广义割线方程","source_units":[{"raw_tex":"The \\(p\\)th-order analogue of the secant equation \\cref{eqn:short-secant-equation} is given by\n\\begin{equation}\\label{eqn:generalized-secant-equation}\n    \\tns{C}_{k+1} [\\vek{s}_k] = D^{p-1} f(\\vek{x}_{k+1}) - D^{p-1} f(\\vek{x}_k) = \\widetilde{\\tns{C}}_k [\\vek{s}_k]\n\\end{equation}\n","start":23077,"end":23359}]},{"id":"S018","semantic_type":"algorithm","kind":"algorithm","labels":["eqn:update-definition"],"declared_external":false,"source_only":false,"title":"高阶割线更新规则","source_units":[{"raw_tex":"and the generalized update formula for \\(\\tns{C}_k\\) reads\n\\begin{equation}\\label{eqn:update-definition}\n    \\tns{C}_{k+1} = \\argmin_{\\tns{C} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{(\\tns{C} - \\tns{C}_k)[\\mat{W}_k]^p}_F \\text{ s.t. } \\tns{C}[\\vek{s}_k] = \\widetilde{\\tns{C}}_k[\\vek{s}_k]. \\tag{HOSU}\n\\end{equation}\n","start":23359,"end":23669}]},{"id":"S019","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"更新规则的命名与用途","source_units":[{"raw_tex":"Even though this update is derived from the updates used in quasi-Newton methods, it is not itself associated with any optimization method and any optimization algorithm using approximate third (or higher) derivatives is also clearly different from Newton's method.\nTo highlight this distinction we will call the update rule the \\emph{higher-order secant update} \\cref{eqn:update-definition} because of its connection with the (generalized) secant equation \\cref{eqn:generalized-secant-equation}.\nIt provides a sequence of approximations of the \\(p\\)th derivative of \\(f\\) based solely on evaluations of the \\((p-1)\\)st derivative at the iterates \\(\\vek{x}_k\\), given some initial approximation \\(\\tns{C}_0\\).\n","start":23669,"end":24379}]},{"id":"S020","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"单步刻画的抽象边界","source_units":[{"raw_tex":"The following theorem provides a full characterization of one step of the update in \\cref{eqn:update-definition}.\nNote that despite the intentional notational similarity the quantities in the statement are independent of the definitions in the previous section.\n","start":24878,"end":25140}]},{"id":"S021","semantic_type":"assumption","kind":"assumption","labels":["item:characterization-explicit","item:characterization-low-rank","item:characterization-min-frobenius","item:characterization-recursive","thm:characterization"],"declared_external":false,"source_only":false,"title":"抽象输入与非退化条件","source_units":[{"raw_tex":"    Let \\(\\tns{C}_{\\bullet} \\in \\R^{\\otimes^p n}_{\\sym}\\) (the current approximation), \\(\\widetilde{\\tns{C}} \\in \\R^{\\otimes^p n}_{\\sym}\\) (the integrated true derivative), a nonsingular matrix \\(\\mat{W} \\in \\R^{n \\times n}\\) (the weight matrix) and a nonzero \\(\\vek{s} \\in \\R^n\\) (the step) be given.\n","start":25185,"end":25487}]},{"id":"S022","semantic_type":"notation","kind":"notation","labels":["item:characterization-explicit","item:characterization-low-rank","item:characterization-min-frobenius","item:characterization-recursive","thm:characterization"],"declared_external":false,"source_only":false,"title":"权矩阵对应的向量","source_units":[{"raw_tex":"    where \\(\\vek{v} = \\mat{W}^{-T}\\mat{W}^{-1}\\vek{s}\\).\n","start":26735,"end":26792}]},{"id":"S023","semantic_type":"theorem","kind":"theorem","labels":["item:characterization-explicit","item:characterization-low-rank","item:characterization-min-frobenius","item:characterization-recursive","thm:characterization"],"declared_external":false,"source_only":false,"title":"高阶割线更新的完整刻画","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm:characterization}\n    Let \\(\\tns{C}_{\\bullet} \\in \\R^{\\otimes^p n}_{\\sym}\\) (the current approximation), \\(\\widetilde{\\tns{C}} \\in \\R^{\\otimes^p n}_{\\sym}\\) (the integrated true derivative), a nonsingular matrix \\(\\mat{W} \\in \\R^{n \\times n}\\) (the weight matrix) and a nonzero \\(\\vek{s} \\in \\R^n\\) (the step) be given.\n    The following equations all have the same unique solution \\(\\tns{C}_+ \\in \\R^{\\otimes^p n}_{\\sym}\\) (the new approximation):\n    \\begin{enumerate}[(a)]\n        \\item\\label{item:characterization-min-frobenius} \\(\\tns{C}_+ = \\argmin_{\\tns{C} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{(\\tns{C} - \\tns{C}_{\\bullet})[\\mat{W}]^p}_F \\text{ s.t. } \\tns{C}[\\vek{s}] = \\widetilde{\\tns{C}}[\\vek{s}]\\)\n        \\item\\label{item:characterization-explicit} \\(\\tns{C}_+ = \\tns{C}_{\\bullet} + \\sum_{j=1}^p (-1)^{j+1} \\binom{p}{j} (\\vek{v}^T \\vek{s})^{-j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{v}} \\otimes (\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]^{j}}\\)\n        \\item\\label{item:characterization-low-rank} \\(\\tns{C}_+ = \\tns{C}_{\\bullet} + P_{\\sym}(\\tns{A} \\otimes \\vek{v})\\) for the unique \\((p-1)\\)-tensor \\(\\tns{A} \\in \\R^{\\otimes^{p-1} n}_{\\sym}\\) which satisfies \\(P_{\\sym}(\\tns{A} \\otimes \\vek{v})[\\vek{s}] = (\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]\\)\n        \\item\\label{item:characterization-recursive} \\((\\tns{C}_+ - \\widetilde{\\tns{C}})[\\mat{W}]^p = (\\tns{C}_{\\bullet} - \\widetilde{\\tns{C}})[\\mat{W}]^p \\Brack{\\mat{I} - \\frac{\\mat{W}^{-1}\\vek{s}\\vek{s}^T \\mat{W}^{-T}}{\\vek{s}^T \\mat{W}^{-T} \\mat{W}^{-1} \\vek{s}}}^p\\)\n    \\end{enumerate}\n    where \\(\\vek{v} = \\mat{W}^{-T}\\mat{W}^{-1}\\vek{s}\\).\n\\end{theorem}","start":25141,"end":26805}]},{"id":"S024","semantic_type":"proof","kind":"proof","labels":["eqn:general-psb-update","eqn:update-explicit-cancellation","eqn:update-minimum-property","eqn:update-secant-equation"],"declared_external":false,"source_only":false,"title":"完整刻画定理的作者证明","source_units":[{"raw_tex":"\\begin{proof}\n    We will first prove the result for \\(\\mat{W} = \\mat{I}\\) and then see how that implies the full result for any nonsingular weight matrix \\(\\mat{W}\\).\n\n    The first characterization can be rewritten as\n    \\begin{equation}\\label{eqn:general-psb-update}\n        \\tns{C}_+ = \\tns{C}_{\\bullet} + \\argmin_{\\tns{U} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{\\tns{U}}_F \\text{ s.t. } \\tns{U}[\\vek{s}] = (\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}].\n    \\end{equation}\n    Let \\(\\mat{Q} \\in \\R^{n \\times n}\\) be an orthogonal matrix that maps \\(\\vek{e}_1\\) to some scalar multiple of \\(\\vek{s}\\).\n    This means \\(\\tns{U}[\\mat{Q}]^p\\) has the same Frobenius norm as \\(\\tns{U}\\) and \\(\\tns{U}[\\mat{Q}]^p[\\vek{e}_{i_1}, \\dots, \\vek{e}_{i_p}]\\) is fully determined by the equality constraint in \\cref{eqn:general-psb-update} if \\(1 \\in \\{i_1, \\dots, i_p\\}\\) because of symmetry.\n    On the other hand, if \\(1 \\notin \\{i_1, \\dots, i_p\\}\\) there are no constraints on the values of \\(\\tns{U}[\\mat{Q}]^p[\\vek{e}_{i_1}, \\dots, \\vek{e}_{i_p}]\\), so the unique choice that minimizes \\(\\norm{\\tns{U}[\\mat{Q}]^p}_F = \\norm{\\tns{U}}_F\\) is clearly to set all of these values to zero.\n    Therefore, there is a unique solution to the minimization problem in \\labelcref{item:characterization-min-frobenius}, and it is fully characterized by the fact that the update tensor \\(\\tns{U} = \\tns{C}_+ - \\tns{C}_{\\bullet}\\) is symmetric and satisfies the following two properties:\n    \\begin{subequations}\n        \\begin{align}\n            \\tns{U}[\\vek{s}]                     & = \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}] \\label{eqn:update-secant-equation}        \\\\\n            \\tns{U}[\\vek{u}_1, \\dots, \\vek{u}_p] & = 0 \\text{ if all } \\vek{u}_i \\text{ are orthogonal to } \\vek{s} \\label{eqn:update-minimum-property}\n        \\end{align}\n    \\end{subequations}\n    We will use this as the basis to show the equivalence with all other characterizations.\n\n    In \\labelcref{item:characterization-explicit} we claim that the update \\(\\tns{U}\\) has the form\n    \\begin{equation}\n        \\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}\n    \\end{equation}\n    This tensor is clearly symmetric since it is a sum of symmetric tensors.\n    To show that it satisfies property \\cref{eqn:update-secant-equation} consider\n    \\begin{subequations}\\label{eqn:update-explicit-cancellation}\n        \\begin{align}\n             & \\phantom{={}}\\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}[\\vek{s}]                                                                                                                                                \\\\\n            \\begin{split}\n                {} = \\sum_{j=1}^p (-1)^{j+1} \\norm{\\vek{s}}_2^{-2j} \\bigg( \\tbinom{p-1}{j-1} P_{\\sym} \\Paren{\\Paren{\\otimes^{j-1} \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}} \\norm{\\vek{s}}_2^2 \\phantom{\\bigg)}\\\\\n                {} + \\tbinom{p-1}{j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j+1}} \\bigg) \\phantom{\\norm{\\vek{s}_2^2}}\n            \\end{split} \\\\\n             & = P_{\\sym} \\Paren{ \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}] } = \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}].\n        \\end{align}\n    \\end{subequations}\n    The first equality uses the fact that each summand is the symmetric projection of an outer product of two symmetric tensors, a \\(j\\)-tensor and a \\((p-j)\\)-tensor.\n    Therefore, there are \\(\\binom{p}{j}\\) distinct ways to orient this outer product and for \\(\\binom{p-1}{j-1}\\) of them the vector \\(\\vek{s}\\) is applied to the \\(j\\)-tensor and for \\(\\binom{p-1}{j}\\) to the \\((p-j)\\)-tensor.\n    A close examination of the expression on the second line shows that it is a telescoping sum where all terms except the ones with coefficients \\(\\binom{p-1}{0}\\) and \\(\\binom{p-1}{p}\\) cancel out.\n    Because \\(\\binom{p-1}{p} = 0\\) the only remaining term is the symmetric projection of \\((\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]\\).\n    Property \\cref{eqn:update-minimum-property} follows from a similar consideration to the one above.\n    If \\(\\vek{u}_1, \\dots, \\vek{u}_p\\) are all orthogonal to \\(\\vek{s}\\), then\n    \\begin{equation}\n        P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}[\\vek{u}_1, \\dots, \\vek{u}_p] = 0\n    \\end{equation}\n    for \\(j \\geq 1\\) because no matter how the outer product is oriented there is always a factor of \\(\\vek{s}^T \\vek{u}_i = 0\\) in the result.\n\n    For \\labelcref{item:characterization-low-rank} we need to show that we can always write the update in the form \\(P_{\\sym}(\\tns{A} \\otimes \\vek{s})\\) for some symmetric \\((p-1)\\)-tensor \\(\\tns{A}\\) and that \\(\\tns{A}\\) is unique such that the corresponding update satisfies the secant equation \\cref{eqn:update-secant-equation}.\n    Note that the expression for the update in \\labelcref{item:characterization-explicit} is already of the form \\(P_{\\sym}(\\tns{A} \\otimes \\vek{s})\\):\n    \\begin{subequations}\n        \\begin{align}\n             & \\phantom{={}}\\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}                               \\\\\n             & = P_{\\sym} \\Paren{ P_{\\sym} \\Paren{ \\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} \\Paren{\\otimes^{j-1} \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j} } \\otimes \\vek{s} }\n        \\end{align}\n    \\end{subequations}\n    It remains to show that among all updates of the form \\(P_{\\sym}(\\tns{A} \\otimes \\vek{s})\\) there is only one choice of \\(\\tns{A} \\in \\R^{\\otimes^p n}_{\\sym}\\) such that \\cref{eqn:update-secant-equation} holds.\n    Consider the linear map\n    \\begin{align*}\n        \\phi \\colon \\R^{\\otimes^{p-1} n}_{\\sym} & \\to \\R^{\\otimes^{p-1} n}_{\\sym}                    \\\\\n        \\tns{A}                                 & \\mapsto P_{\\sym}(\\tns{A} \\otimes \\vek{s})[\\vek{s}]\n    \\end{align*}\n    which maps a finite-dimensional vector space to itself.\n    Combining what we already showed for \\labelcref{item:characterization-explicit} with the observation that the update in \\labelcref{item:characterization-explicit} is of the desired form, this map is surjective (we can prescribe any \\((\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]\\)) and so it must be bijective, which shows uniqueness of \\(\\tns{A}\\).\n\n    Lastly, \\labelcref{item:characterization-recursive} claims that the update can be written as\n    \\begin{equation}\n        \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}} - \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}} \\Brack{\\mat{I} - \\frac{\\vek{s} \\vek{s}^T}{\\norm{\\vek{s}}_2^2}}^p.\n    \\end{equation}\n    Clearly, property \\cref{eqn:update-secant-equation} is satisfied because applying this tensor to \\(\\vek{s}\\) makes the second term vanish, leaving only the desired result.\n    Moreover, for any vector \\(\\vek{u}\\) that is orthogonal to \\(\\vek{s}\\) the matrix \\(\\mat{I} - \\frac{\\vek{s} \\vek{s}^T}{\\norm{\\vek{s}}_2^2}\\) maps \\(\\vek{u}\\) to itself.\n    This means applying the tensor above to \\(\\vek{u}_1, \\dots, \\vek{u}_p\\), all of which are orthogonal to \\(\\vek{s}\\), will give zero because both terms cancel out.\n    This is property \\cref{eqn:update-minimum-property}.\n\n    Now that the equivalence has been established for \\(\\mat{W} = \\mat{I}\\), we consider the general case where \\(\\mat{W} \\in \\R^{n \\times n}\\) is any nonsingular matrix.\n    The minimization in \\labelcref{item:characterization-min-frobenius}\n    \\begin{equation}\n        \\tns{C}_+ = \\argmin_{\\tns{C} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{(\\tns{C} - \\tns{C}_{\\bullet})[\\mat{W}]^p}_F \\text{ s.t. } \\tns{C}[\\vek{s}] = \\widetilde{\\tns{C}}[\\vek{s}]\n    \\end{equation}\n    can be rewritten using \\(\\tns{D}_{\\bullet} = \\tns{C}_{\\bullet}[\\mat{W}]^p\\), \\(\\widetilde{\\tns{D}} = \\widetilde{\\tns{C}}[\\mat{W}]^p\\), \\(\\tns{D}_+ = \\tns{C}_+[\\mat{W}]^p\\) and \\(\\vek{r} = \\mat{W}^{-1}\\vek{s}\\) as\n    \\begin{equation}\n        \\tns{D}_+ = \\argmin_{\\tns{D} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{\\tns{D} - \\tns{D}_{\\bullet}}_F \\text{ s.t. } \\tns{D}[\\vek{r}] = \\widetilde{\\tns{D}}[\\vek{r}].\n    \\end{equation}\n    Applying the existing characterizations and the fact that \\(\\tns{C}_+ = \\tns{D}_+[\\mat{W}^{-1}]^p\\) we get the claim after some algebraic manipulations.\n\\end{proof}","start":26807,"end":35513}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"declared_external":false,"source_only":false,"title":"先证无权再作变换","source_units":[{"raw_tex":"    We will first prove the result for \\(\\mat{W} = \\mat{I}\\) and then see how that implies the full result for any nonsingular weight matrix \\(\\mat{W}\\).\n\n","start":26821,"end":26976}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:general-psb-update","eqn:update-explicit-cancellation"],"declared_external":false,"source_only":false,"title":"改写为最小范数增量","source_units":[{"raw_tex":"    The first characterization can be rewritten as\n    \\begin{equation}\\label{eqn:general-psb-update}\n        \\tns{C}_+ = \\tns{C}_{\\bullet} + \\argmin_{\\tns{U} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{\\tns{U}}_F \\text{ s.t. } \\tns{U}[\\vek{s}] = (\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}].\n    \\end{equation}\n","start":26976,"end":27285}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"declared_external":false,"source_only":false,"title":"沿步长选择正交坐标","source_units":[{"raw_tex":"    Let \\(\\mat{Q} \\in \\R^{n \\times n}\\) be an orthogonal matrix that maps \\(\\vek{e}_1\\) to some scalar multiple of \\(\\vek{s}\\).\n    This means \\(\\tns{U}[\\mat{Q}]^p\\) has the same Frobenius norm as \\(\\tns{U}\\) and \\(\\tns{U}[\\mat{Q}]^p[\\vek{e}_{i_1}, \\dots, \\vek{e}_{i_p}]\\) is fully determined by the equality constraint in \\cref{eqn:general-psb-update} if \\(1 \\in \\{i_1, \\dots, i_p\\}\\) because of symmetry.\n","start":27285,"end":27692}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"declared_external":false,"source_only":false,"title":"自由坐标唯一取零","source_units":[{"raw_tex":"    On the other hand, if \\(1 \\notin \\{i_1, \\dots, i_p\\}\\) there are no constraints on the values of \\(\\tns{U}[\\mat{Q}]^p[\\vek{e}_{i_1}, \\dots, \\vek{e}_{i_p}]\\), so the unique choice that minimizes \\(\\norm{\\tns{U}[\\mat{Q}]^p}_F = \\norm{\\tns{U}}_F\\) is clearly to set all of these values to zero.\n","start":27692,"end":27988}]},{"id":"S029","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:update-explicit-cancellation","eqn:update-minimum-property","eqn:update-secant-equation"],"declared_external":false,"source_only":false,"title":"唯一极小更新的双条件","source_units":[{"raw_tex":"    Therefore, there is a unique solution to the minimization problem in \\labelcref{item:characterization-min-frobenius}, and it is fully characterized by the fact that the update tensor \\(\\tns{U} = \\tns{C}_+ - \\tns{C}_{\\bullet}\\) is symmetric and satisfies the following two properties:\n    \\begin{subequations}\n        \\begin{align}\n            \\tns{U}[\\vek{s}]                     & = \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}] \\label{eqn:update-secant-equation}        \\\\\n            \\tns{U}[\\vek{u}_1, \\dots, \\vek{u}_p] & = 0 \\text{ if all } \\vek{u}_i \\text{ are orthogonal to } \\vek{s} \\label{eqn:update-minimum-property}\n        \\end{align}\n    \\end{subequations}\n    We will use this as the basis to show the equivalence with all other characterizations.\n","start":27988,"end":28765}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"declared_external":false,"source_only":false,"title":"提出对称显式增量","source_units":[{"raw_tex":"    In \\labelcref{item:characterization-explicit} we claim that the update \\(\\tns{U}\\) has the form\n    \\begin{equation}\n        \\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}\n    \\end{equation}\n    This tensor is clearly symmetric since it is a sum of symmetric tensors.\n","start":28766,"end":29164}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"declared_external":false,"source_only":false,"title":"按收缩落点统计排列","source_units":[{"raw_tex":"    To show that it satisfies property \\cref{eqn:update-secant-equation} consider\n    \\begin{subequations}\\label{eqn:update-explicit-cancellation}\n        \\begin{align}\n             & \\phantom{={}}\\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}[\\vek{s}]                                                                                                                                                \\\\\n            \\begin{split}\n                {} = \\sum_{j=1}^p (-1)^{j+1} \\norm{\\vek{s}}_2^{-2j} \\bigg( \\tbinom{p-1}{j-1} P_{\\sym} \\Paren{\\Paren{\\otimes^{j-1} \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}} \\norm{\\vek{s}}_2^2 \\phantom{\\bigg)}\\\\\n                {} + \\tbinom{p-1}{j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j+1}} \\bigg) \\phantom{\\norm{\\vek{s}_2^2}}\n            \\end{split} \\\\\n             & = P_{\\sym} \\Paren{ \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}] } = \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}].\n        \\end{align}\n    \\end{subequations}\n    The first equality uses the fact that each summand is the symmetric projection of an outer product of two symmetric tensors, a \\(j\\)-tensor and a \\((p-j)\\)-tensor.\n    Therefore, there are \\(\\binom{p}{j}\\) distinct ways to orient this outer product and for \\(\\binom{p-1}{j-1}\\) of them the vector \\(\\vek{s}\\) is applied to the \\(j\\)-tensor and for \\(\\binom{p-1}{j}\\) to the \\((p-j)\\)-tensor.\n","start":29164,"end":30768}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"declared_external":false,"source_only":false,"title":"交错和望远镜消去","source_units":[{"raw_tex":"             & = P_{\\sym} \\Paren{ \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}] } = \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}].\n","start":30176,"end":30329},{"raw_tex":"    A close examination of the expression on the second line shows that it is a telescoping sum where all terms except the ones with coefficients \\(\\binom{p-1}{0}\\) and \\(\\binom{p-1}{p}\\) cancel out.\n    Because \\(\\binom{p-1}{p} = 0\\) the only remaining term is the symmetric projection of \\((\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]\\).\n","start":30768,"end":31114}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"declared_external":false,"source_only":false,"title":"显式式在正交方向消失","source_units":[{"raw_tex":"    Property \\cref{eqn:update-minimum-property} follows from a similar consideration to the one above.\n    If \\(\\vek{u}_1, \\dots, \\vek{u}_p\\) are all orthogonal to \\(\\vek{s}\\), then\n    \\begin{equation}\n        P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}[\\vek{u}_1, \\dots, \\vek{u}_p] = 0\n    \\end{equation}\n    for \\(j \\geq 1\\) because no matter how the outer product is oriented there is always a factor of \\(\\vek{s}^T \\vek{u}_i = 0\\) in the result.\n","start":31114,"end":31633}]},{"id":"S034","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:update-explicit-cancellation"],"declared_external":false,"source_only":false,"title":"无权显式式确定唯一更新","source_units":[{"raw_tex":"    In \\labelcref{item:characterization-explicit} we claim that the update \\(\\tns{U}\\) has the form\n    \\begin{equation}\n        \\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}\n    \\end{equation}\n    This tensor is clearly symmetric since it is a sum of symmetric tensors.\n    To show that it satisfies property \\cref{eqn:update-secant-equation} consider\n    \\begin{subequations}\\label{eqn:update-explicit-cancellation}\n        \\begin{align}\n             & \\phantom{={}}\\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}[\\vek{s}]                                                                                                                                                \\\\\n            \\begin{split}\n                {} = \\sum_{j=1}^p (-1)^{j+1} \\norm{\\vek{s}}_2^{-2j} \\bigg( \\tbinom{p-1}{j-1} P_{\\sym} \\Paren{\\Paren{\\otimes^{j-1} \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}} \\norm{\\vek{s}}_2^2 \\phantom{\\bigg)}\\\\\n                {} + \\tbinom{p-1}{j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j+1}} \\bigg) \\phantom{\\norm{\\vek{s}_2^2}}\n            \\end{split} \\\\\n             & = P_{\\sym} \\Paren{ \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}] } = \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}].\n        \\end{align}\n    \\end{subequations}\n    The first equality uses the fact that each summand is the symmetric projection of an outer product of two symmetric tensors, a \\(j\\)-tensor and a \\((p-j)\\)-tensor.\n    Therefore, there are \\(\\binom{p}{j}\\) distinct ways to orient this outer product and for \\(\\binom{p-1}{j-1}\\) of them the vector \\(\\vek{s}\\) is applied to the \\(j\\)-tensor and for \\(\\binom{p-1}{j}\\) to the \\((p-j)\\)-tensor.\n    A close examination of the expression on the second line shows that it is a telescoping sum where all terms except the ones with coefficients \\(\\binom{p-1}{0}\\) and \\(\\binom{p-1}{p}\\) cancel out.\n    Because \\(\\binom{p-1}{p} = 0\\) the only remaining term is the symmetric projection of \\((\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]\\).\n    Property \\cref{eqn:update-minimum-property} follows from a similar consideration to the one above.\n    If \\(\\vek{u}_1, \\dots, \\vek{u}_p\\) are all orthogonal to \\(\\vek{s}\\), then\n    \\begin{equation}\n        P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}[\\vek{u}_1, \\dots, \\vek{u}_p] = 0\n    \\end{equation}\n    for \\(j \\geq 1\\) because no matter how the outer product is oriented there is always a factor of \\(\\vek{s}^T \\vek{u}_i = 0\\) in the result.\n","start":28766,"end":31633}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"declared_external":false,"source_only":false,"title":"从显式式提出外积因子","source_units":[{"raw_tex":"    For \\labelcref{item:characterization-low-rank} we need to show that we can always write the update in the form \\(P_{\\sym}(\\tns{A} \\otimes \\vek{s})\\) for some symmetric \\((p-1)\\)-tensor \\(\\tns{A}\\) and that \\(\\tns{A}\\) is unique such that the corresponding update satisfies the secant equation \\cref{eqn:update-secant-equation}.\n    Note that the expression for the update in \\labelcref{item:characterization-explicit} is already of the form \\(P_{\\sym}(\\tns{A} \\otimes \\vek{s})\\):\n    \\begin{subequations}\n        \\begin{align}\n             & \\phantom{={}}\\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}                               \\\\\n             & = P_{\\sym} \\Paren{ P_{\\sym} \\Paren{ \\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} \\Paren{\\otimes^{j-1} \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j} } \\otimes \\vek{s} }\n        \\end{align}\n    \\end{subequations}\n","start":31634,"end":32673}]},{"id":"S036","semantic_type":"notation","kind":"notation","labels":["eqn:update-explicit-cancellation"],"declared_external":false,"source_only":false,"title":"因子阶数的原文冲突","source_units":[{"raw_tex":"    It remains to show that among all updates of the form \\(P_{\\sym}(\\tns{A} \\otimes \\vek{s})\\) there is only one choice of \\(\\tns{A} \\in \\R^{\\otimes^p n}_{\\sym}\\) such that \\cref{eqn:update-secant-equation} holds.\n","start":32673,"end":32888}]},{"id":"S037","semantic_type":"definition","kind":"definition","labels":["eqn:update-explicit-cancellation"],"declared_external":false,"source_only":false,"title":"因子到割线数据的线性映射","source_units":[{"raw_tex":"    Consider the linear map\n    \\begin{align*}\n        \\phi \\colon \\R^{\\otimes^{p-1} n}_{\\sym} & \\to \\R^{\\otimes^{p-1} n}_{\\sym}                    \\\\\n        \\tns{A}                                 & \\mapsto P_{\\sym}(\\tns{A} \\otimes \\vek{s})[\\vek{s}]\n    \\end{align*}\n    which maps a finite-dimensional vector space to itself.\n","start":32888,"end":33217}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"declared_external":false,"source_only":false,"title":"作者断言因子映射满射","source_units":[{"raw_tex":"    Combining what we already showed for \\labelcref{item:characterization-explicit} with the observation that the update in \\labelcref{item:characterization-explicit} is of the desired form, this map is surjective (we can prescribe any \\((\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]\\)) and so it must be bijective, which shows uniqueness of \\(\\tns{A}\\).\n","start":33217,"end":33577}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"declared_external":false,"source_only":false,"title":"有限维满射推出唯一因子","source_units":[{"raw_tex":"    Combining what we already showed for \\labelcref{item:characterization-explicit} with the observation that the update in \\labelcref{item:characterization-explicit} is of the desired form, this map is surjective (we can prescribe any \\((\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]\\)) and so it must be bijective, which shows uniqueness of \\(\\tns{A}\\).\n\n","start":33217,"end":33578}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"declared_external":false,"source_only":false,"title":"提出投影差形式","source_units":[{"raw_tex":"    Lastly, \\labelcref{item:characterization-recursive} claims that the update can be written as\n    \\begin{equation}\n        \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}} - \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}} \\Brack{\\mat{I} - \\frac{\\vek{s} \\vek{s}^T}{\\norm{\\vek{s}}_2^2}}^p.\n    \\end{equation}\n","start":33578,"end":33887}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"declared_external":false,"source_only":false,"title":"投影消去步长方向","source_units":[{"raw_tex":"    Clearly, property \\cref{eqn:update-secant-equation} is satisfied because applying this tensor to \\(\\vek{s}\\) makes the second term vanish, leaving only the desired result.\n","start":33887,"end":34063}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"declared_external":false,"source_only":false,"title":"投影固定正交方向","source_units":[{"raw_tex":"    Moreover, for any vector \\(\\vek{u}\\) that is orthogonal to \\(\\vek{s}\\) the matrix \\(\\mat{I} - \\frac{\\vek{s} \\vek{s}^T}{\\norm{\\vek{s}}_2^2}\\) maps \\(\\vek{u}\\) to itself.\n    This means applying the tensor above to \\(\\vek{u}_1, \\dots, \\vek{u}_p\\), all of which are orthogonal to \\(\\vek{s}\\), will give zero because both terms cancel out.\n    This is property \\cref{eqn:update-minimum-property}.\n","start":34063,"end":34460}]},{"id":"S043","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:update-explicit-cancellation"],"declared_external":false,"source_only":false,"title":"无权四种刻画已建立","source_units":[{"raw_tex":"    Now that the equivalence has been established for \\(\\mat{W} = \\mat{I}\\), we consider the general case where \\(\\mat{W} \\in \\R^{n \\times n}\\) is any nonsingular matrix.\n","start":34461,"end":34632}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"declared_external":false,"source_only":false,"title":"回到一般加权极小问题","source_units":[{"raw_tex":"    The minimization in \\labelcref{item:characterization-min-frobenius}\n    \\begin{equation}\n        \\tns{C}_+ = \\argmin_{\\tns{C} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{(\\tns{C} - \\tns{C}_{\\bullet})[\\mat{W}]^p}_F \\text{ s.t. } \\tns{C}[\\vek{s}] = \\widetilde{\\tns{C}}[\\vek{s}]\n    \\end{equation}\n","start":34632,"end":34922}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"declared_external":false,"source_only":false,"title":"加权问题变为无权问题","source_units":[{"raw_tex":"    can be rewritten using \\(\\tns{D}_{\\bullet} = \\tns{C}_{\\bullet}[\\mat{W}]^p\\), \\(\\widetilde{\\tns{D}} = \\widetilde{\\tns{C}}[\\mat{W}]^p\\), \\(\\tns{D}_+ = \\tns{C}_+[\\mat{W}]^p\\) and \\(\\vek{r} = \\mat{W}^{-1}\\vek{s}\\) as\n    \\begin{equation}\n        \\tns{D}_+ = \\argmin_{\\tns{D} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{\\tns{D} - \\tns{D}_{\\bullet}}_F \\text{ s.t. } \\tns{D}[\\vek{r}] = \\widetilde{\\tns{D}}[\\vek{r}].\n    \\end{equation}\n","start":34922,"end":35345}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"declared_external":false,"source_only":false,"title":"逆变换恢复全部刻画","source_units":[{"raw_tex":"    Applying the existing characterizations and the fact that \\(\\tns{C}_+ = \\tns{D}_+[\\mat{W}^{-1}]^p\\) we get the claim after some algebraic manipulations.\n","start":35345,"end":35502}]},{"id":"S047","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先修列表第二项：标准有限维线性代数，包括正交基变换；此处由标注者显式化。"],"source_only":false,"title":"沿非零向量的正交基选择","source_units":[]},{"id":"S048","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先修列表第二项：标准有限维线性代数，包括自映射满射与双射的等价。"],"source_only":false,"title":"有限维自映射的满射与双射","source_units":[]},{"id":"S049","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始先修列表第三项：标准二项式恒等式与排列计数。"],"source_only":false,"title":"二项式计数与端点恒等式","source_units":[]},{"id":"S050","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；原始先修列表未单列，使用位置为 L339。"],"source_only":false,"title":"沿线段的微积分基本定理","source_units":[]},{"id":"S051","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；原始先修列表未单列，作者在 L265 明确点名 Cauchy–Schwarz。"],"source_only":false,"title":"Cauchy–Schwarz 不等式","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S023","edges":[{"source_id":"S003","target_id":"S023","kind":"supports"},{"source_id":"S004","target_id":"S023","kind":"supports"},{"source_id":"S005","target_id":"S023","kind":"supports"},{"source_id":"S006","target_id":"S023","kind":"supports"},{"source_id":"S007","target_id":"S023","kind":"supports"},{"source_id":"S009","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S002","target_id":"S023","kind":"supports"},{"source_id":"S020","target_id":"S023","kind":"supports"},{"source_id":"S003","target_id":"S011","kind":"supports"},{"source_id":"S004","target_id":"S011","kind":"supports"},{"source_id":"S005","target_id":"S013","kind":"supports"},{"source_id":"S009","target_id":"S013","kind":"supports"},{"source_id":"S010","target_id":"S013","kind":"supports"},{"source_id":"S051","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S014","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S050","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S004","target_id":"S018","kind":"supports"},{"source_id":"S006","target_id":"S018","kind":"supports"},{"source_id":"S009","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S026","kind":"supports"},{"source_id":"S003","target_id":"S026","kind":"supports"},{"source_id":"S009","target_id":"S026","kind":"supports"},{"source_id":"S021","target_id":"S027","kind":"supports"},{"source_id":"S047","target_id":"S027","kind":"supports"},{"source_id":"S004","target_id":"S027","kind":"supports"},{"source_id":"S006","target_id":"S027","kind":"supports"},{"source_id":"S008","target_id":"S027","kind":"supports"},{"source_id":"S011","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S008","target_id":"S028","kind":"supports"},{"source_id":"S009","target_id":"S028","kind":"supports"},{"source_id":"S026","target_id":"S029","kind":"supports"},{"source_id":"S027","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S003","target_id":"S030","kind":"supports"},{"source_id":"S005","target_id":"S030","kind":"supports"},{"source_id":"S007","target_id":"S030","kind":"supports"},{"source_id":"S021","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S003","target_id":"S031","kind":"supports"},{"source_id":"S005","target_id":"S031","kind":"supports"},{"source_id":"S006","target_id":"S031","kind":"supports"},{"source_id":"S007","target_id":"S031","kind":"supports"},{"source_id":"S049","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S049","target_id":"S032","kind":"supports"},{"source_id":"S006","target_id":"S032","kind":"supports"},{"source_id":"S007","target_id":"S032","kind":"supports"},{"source_id":"S030","target_id":"S033","kind":"supports"},{"source_id":"S005","target_id":"S033","kind":"supports"},{"source_id":"S007","target_id":"S033","kind":"supports"},{"source_id":"S029","target_id":"S034","kind":"supports"},{"source_id":"S030","target_id":"S034","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S005","target_id":"S035","kind":"supports"},{"source_id":"S007","target_id":"S035","kind":"supports"},{"source_id":"S003","target_id":"S037","kind":"supports"},{"source_id":"S005","target_id":"S037","kind":"supports"},{"source_id":"S006","target_id":"S037","kind":"supports"},{"source_id":"S007","target_id":"S037","kind":"supports"},{"source_id":"S034","target_id":"S038","kind":"supports"},{"source_id":"S035","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S048","target_id":"S039","kind":"supports"},{"source_id":"S003","target_id":"S040","kind":"supports"},{"source_id":"S004","target_id":"S040","kind":"supports"},{"source_id":"S021","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S004","target_id":"S041","kind":"supports"},{"source_id":"S021","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S004","target_id":"S042","kind":"supports"},{"source_id":"S029","target_id":"S043","kind":"supports"},{"source_id":"S034","target_id":"S043","kind":"supports"},{"source_id":"S035","target_id":"S043","kind":"supports"},{"source_id":"S039","target_id":"S043","kind":"supports"},{"source_id":"S040","target_id":"S043","kind":"supports"},{"source_id":"S041","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S021","target_id":"S045","kind":"supports"},{"source_id":"S003","target_id":"S045","kind":"supports"},{"source_id":"S004","target_id":"S045","kind":"supports"},{"source_id":"S006","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S043","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S004","target_id":"S046","kind":"supports"},{"source_id":"S005","target_id":"S046","kind":"supports"},{"source_id":"S007","target_id":"S046","kind":"supports"},{"source_id":"S021","target_id":"S046","kind":"supports"},{"source_id":"S022","target_id":"S046","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S027","target_id":"S024","kind":"supports"},{"source_id":"S028","target_id":"S024","kind":"supports"},{"source_id":"S029","target_id":"S024","kind":"supports"},{"source_id":"S030","target_id":"S024","kind":"supports"},{"source_id":"S031","target_id":"S024","kind":"supports"},{"source_id":"S032","target_id":"S024","kind":"supports"},{"source_id":"S033","target_id":"S024","kind":"supports"},{"source_id":"S035","target_id":"S024","kind":"supports"},{"source_id":"S036","target_id":"S024","kind":"supports"},{"source_id":"S037","target_id":"S024","kind":"supports"},{"source_id":"S038","target_id":"S024","kind":"supports"},{"source_id":"S039","target_id":"S024","kind":"supports"},{"source_id":"S040","target_id":"S024","kind":"supports"},{"source_id":"S041","target_id":"S024","kind":"supports"},{"source_id":"S042","target_id":"S024","kind":"supports"},{"source_id":"S043","target_id":"S024","kind":"supports"},{"source_id":"S044","target_id":"S024","kind":"supports"},{"source_id":"S045","target_id":"S024","kind":"supports"},{"source_id":"S046","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S023","edges":[{"source_id":"S003","target_id":"S023","kind":"supports"},{"source_id":"S004","target_id":"S023","kind":"supports"},{"source_id":"S005","target_id":"S023","kind":"supports"},{"source_id":"S006","target_id":"S023","kind":"supports"},{"source_id":"S007","target_id":"S023","kind":"supports"},{"source_id":"S009","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S002","target_id":"S023","kind":"supports"},{"source_id":"S020","target_id":"S023","kind":"supports"},{"source_id":"S003","target_id":"S011","kind":"supports"},{"source_id":"S004","target_id":"S011","kind":"supports"},{"source_id":"S005","target_id":"S013","kind":"supports"},{"source_id":"S009","target_id":"S013","kind":"supports"},{"source_id":"S010","target_id":"S013","kind":"supports"},{"source_id":"S051","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S014","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S050","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S004","target_id":"S018","kind":"supports"},{"source_id":"S006","target_id":"S018","kind":"supports"},{"source_id":"S009","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S021","target_id":"S026","kind":"supports"},{"source_id":"S003","target_id":"S026","kind":"supports"},{"source_id":"S009","target_id":"S026","kind":"supports"},{"source_id":"S021","target_id":"S027","kind":"supports"},{"source_id":"S047","target_id":"S027","kind":"supports"},{"source_id":"S004","target_id":"S027","kind":"supports"},{"source_id":"S006","target_id":"S027","kind":"supports"},{"source_id":"S008","target_id":"S027","kind":"supports"},{"source_id":"S011","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S008","target_id":"S028","kind":"supports"},{"source_id":"S009","target_id":"S028","kind":"supports"},{"source_id":"S026","target_id":"S029","kind":"supports"},{"source_id":"S027","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S003","target_id":"S030","kind":"supports"},{"source_id":"S005","target_id":"S030","kind":"supports"},{"source_id":"S007","target_id":"S030","kind":"supports"},{"source_id":"S021","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S003","target_id":"S031","kind":"supports"},{"source_id":"S005","target_id":"S031","kind":"supports"},{"source_id":"S006","target_id":"S031","kind":"supports"},{"source_id":"S007","target_id":"S031","kind":"supports"},{"source_id":"S049","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S049","target_id":"S032","kind":"supports"},{"source_id":"S006","target_id":"S032","kind":"supports"},{"source_id":"S007","target_id":"S032","kind":"supports"},{"source_id":"S030","target_id":"S033","kind":"supports"},{"source_id":"S005","target_id":"S033","kind":"supports"},{"source_id":"S007","target_id":"S033","kind":"supports"},{"source_id":"S029","target_id":"S034","kind":"supports"},{"source_id":"S030","target_id":"S034","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S005","target_id":"S035","kind":"supports"},{"source_id":"S007","target_id":"S035","kind":"supports"},{"source_id":"S003","target_id":"S037","kind":"supports"},{"source_id":"S005","target_id":"S037","kind":"supports"},{"source_id":"S006","target_id":"S037","kind":"supports"},{"source_id":"S007","target_id":"S037","kind":"supports"},{"source_id":"S034","target_id":"S038","kind":"supports"},{"source_id":"S035","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S048","target_id":"S039","kind":"supports"},{"source_id":"S003","target_id":"S040","kind":"supports"},{"source_id":"S004","target_id":"S040","kind":"supports"},{"source_id":"S021","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S004","target_id":"S041","kind":"supports"},{"source_id":"S021","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S004","target_id":"S042","kind":"supports"},{"source_id":"S029","target_id":"S043","kind":"supports"},{"source_id":"S034","target_id":"S043","kind":"supports"},{"source_id":"S035","target_id":"S043","kind":"supports"},{"source_id":"S039","target_id":"S043","kind":"supports"},{"source_id":"S040","target_id":"S043","kind":"supports"},{"source_id":"S041","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S021","target_id":"S045","kind":"supports"},{"source_id":"S003","target_id":"S045","kind":"supports"},{"source_id":"S004","target_id":"S045","kind":"supports"},{"source_id":"S006","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S043","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S004","target_id":"S046","kind":"supports"},{"source_id":"S005","target_id":"S046","kind":"supports"},{"source_id":"S007","target_id":"S046","kind":"supports"},{"source_id":"S021","target_id":"S046","kind":"supports"},{"source_id":"S022","target_id":"S046","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S027","target_id":"S024","kind":"supports"},{"source_id":"S028","target_id":"S024","kind":"supports"},{"source_id":"S029","target_id":"S024","kind":"supports"},{"source_id":"S030","target_id":"S024","kind":"supports"},{"source_id":"S031","target_id":"S024","kind":"supports"},{"source_id":"S032","target_id":"S024","kind":"supports"},{"source_id":"S033","target_id":"S024","kind":"supports"},{"source_id":"S035","target_id":"S024","kind":"supports"},{"source_id":"S036","target_id":"S024","kind":"supports"},{"source_id":"S037","target_id":"S024","kind":"supports"},{"source_id":"S038","target_id":"S024","kind":"supports"},{"source_id":"S039","target_id":"S024","kind":"supports"},{"source_id":"S040","target_id":"S024","kind":"supports"},{"source_id":"S041","target_id":"S024","kind":"supports"},{"source_id":"S042","target_id":"S024","kind":"supports"},{"source_id":"S043","target_id":"S024","kind":"supports"},{"source_id":"S044","target_id":"S024","kind":"supports"},{"source_id":"S045","target_id":"S024","kind":"supports"},{"source_id":"S046","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"拟牛顿更新的动机","source_units":[{"raw_tex":"The incredible success of quasi-Newton methods is largely based on the fact that the rules for updating the Hessian approximations are able to extract crucial second-order information from gradient evaluations.\nUnlike finite difference methods they do so without direct control over the evaluation points.\nRather, quasi-Newton rules are designed to handle evaluations of the first derivative at points that are generated by an extraneous process and produce best-effort approximations of the second derivative.\n","start":7114,"end":7625}]},{"id":"S002","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"高阶近似的阶数范围","source_units":[{"raw_tex":"We propose generalizations of these rules that mimic the quasi-Newton approach but approximate \\(p\\)th derivatives from given evaluations of \\((p-1)\\)st derivatives for any \\(p \\geq 2\\).\n","start":7625,"end":7812}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"张量空间与向量收缩","source_units":[{"raw_tex":"A \\emph{\\(p\\)-tensor} \\(\\tns{T}\\) of dimensions \\(n_1 \\times \\dots \\times n_p\\) is a multilinear map \\(\\R^{n_1} \\times \\dots \\times \\R^{n_p}\\to \\R\\), so that its evaluation\n\\begin{equation}\n    \\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p], \\quad \\vek{s}_i \\in \\R^{n_i}\n\\end{equation}\nis linear in each component.\nWe refer to these \\(p\\) components as the \\emph{modes} of the tensor and denote the space of such \\(p\\)-tensors by \\(\\R^{n_1 \\otimes \\dots \\otimes n_p}\\).\nIf \\(n_1 = \\dots = n_p = n\\) the space is denoted \\(\\R^{\\otimes^p n}\\) and \\(\\tns{T}[\\vek{s}, \\dots, \\vek{s}]\\) is abbreviated as \\(\\tns{T}[\\vek{s}]^p\\).\nThe notation also allows to apply the tensor to \\(q < p\\) vectors, which then results in a \\((p-q)\\)-tensor.\n","start":10535,"end":11258}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"张量的矩阵作用","source_units":[{"raw_tex":"Moreover, we define the application of matrices \\(\\mat{W}_1, \\dots, \\mat{W}_p\\) of appropriate dimensions to a tensor by\n\\begin{equation}\n    \\big(\\tns{T}[\\mat{W}_1, \\dots, \\mat{W}_p]\\big)[\\vek{s}_1, \\dots, \\vek{s}_p] = \\tns{T}[\\mat{W}_1 \\vek{s}_1, \\dots, \\mat{W}_p \\vek{s}_p].\n\\end{equation}\n\n","start":11258,"end":11552}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"外积与向量表示约定","source_units":[{"raw_tex":"The outer product of a \\(p_1\\)-tensor \\(\\tns{T}_1\\) with a \\(p_2\\)-tensor \\(\\tns{T}_2\\) is defined as\n\\begin{equation}\n    (\\tns{T}_1 \\otimes \\tns{T}_2)[\\vek{s}_1, \\dots, \\vek{s}_{p_1 + p_2}] = \\tns{T}_1[\\vek{s}_1, \\dots, \\vek{s}_{p_1}] \\cdot \\tns{T}_2[\\vek{s}_{p_1 + 1}, \\dots, \\vek{s}_{p_1 + p_2}].\n\\end{equation}\nIn particular, tensors of the form \\(\\tns{T} = \\vek{v}_1 \\otimes \\dots \\otimes \\vek{v}_p\\) for vectors \\(\\vek{v}_1, \\dots, \\vek{v}_p \\in \\R^n\\), i.e. those where \\(\\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p] = \\prod_{i = 1}^n \\vek{v}_i^T \\vek{s}_i\\), are called \\emph{elementary} or \\emph{rank-one} tensors.\nIf all vectors are the same (\\(\\vek{v}_1 = \\dots = \\vek{v}_p = \\vek{v}\\)), we abbreviate the notation above to \\(\\otimes^p \\vek{v}\\).\n(Note that this notation is slightly inconsistent since 1-tensors should be row vectors, but are represented by standard column vectors to simplify the notation.\nThis is why \\(\\vek{v}[\\mat{W}] = \\mat{W}^T \\vek{v}\\).)\n","start":11552,"end":12520}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"置换与对称张量","source_units":[{"raw_tex":"\nFor any \\(\\tns{T} \\in \\R^{\\otimes^p n}\\) and any permutation \\(\\sigma \\in S_p\\), let \\(\\sigma(\\tns{T}) \\in \\R^{\\otimes^p n}\\) be defined by\n\\begin{equation}\n    \\sigma(\\tns{T})[\\vek{s}_1, \\dots, \\vek{s}_p] = \\tns{T}[\\vek{s}_{\\sigma(1)}, \\dots, \\vek{s}_{\\sigma(p)}].\n\\end{equation}\nIf \\(\\sigma(\\tns{T}) = \\tns{T}\\) for all \\(\\sigma \\in S_p\\), then \\(\\tns{T}\\) is called \\emph{symmetric}.\nThe space of all symmetric \\(p\\)-tensors is denoted \\(\\R^{\\otimes^p n}_{\\sym}\\).\n","start":12520,"end":12989}]},{"id":"S007","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S007"],"declared_external":true,"declaration_text":["原始先修列表第一项；原文 L244 引用 Hackbusch, Tensor Spaces and Numerical Tensor Calculus, Proposition 3.76，cite key: hackbusch_tensor_2012。"],"source_only":false,"title":"对称化投影公式","source_units":[{"raw_tex":"The projection of \\(\\R^{\\otimes^p n}\\) onto \\(\\R^{\\otimes^p n}_{\\sym}\\) is given by\n\\begin{equation}\n    P_{\\sym}(\\tns{T}) = \\frac{1}{p!} \\sum_{\\sigma \\in S_p} \\sigma(\\tns{T}),\n\\end{equation}\nsee \\cite[Proposition 3.76]{hackbusch_tensor_2012}.\n","start":12989,"end":13233}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"基评价与坐标表示","source_units":[{"raw_tex":"\nJust like matrices, tensors are fully characterized by their actions on basis vectors.\nThis can be used to represent a \\(p\\)-tensor \\(\\tns{T} \\in \\R^{\\otimes^p n}\\) as a \\(p\\)-dimensional array \\((t_{i_1, \\dots, i_p})_{1 \\leq i_j \\leq n, 1 \\leq j \\leq p}\\) where\n\\begin{equation}\n    t_{i_1, \\dots, i_p} = \\tns{T}[\\vek{e}_{i_1}, \\dots, \\vek{e}_{i_p}] \\quad \\text{and} \\quad \\tns{T} = \\sum_{i_1, \\dots, i_p = 1}^n t_{i_1, \\dots, i_p} \\vek{e}_{i_1} \\otimes \\dots \\otimes \\vek{e}_{i_p}.\n\\end{equation}\n","start":13233,"end":13733}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"Frobenius 内积与范数","source_units":[{"raw_tex":"There is a Frobenius inner product and a corresponding norm on these \\(p\\)-dimensional arrays and by extension on \\(\\R^{\\otimes^p n}\\), which we will denote by \\(\\innerprod{\\tns{T}_1}{\\tns{T}_2}_F\\) and \\(\\norm{\\tns{T}}_F\\).\nNote that \\(\\innerprod{\\tns{T}}{\\vek{s}_1 \\otimes \\dots \\otimes \\vek{s}_p}_F = \\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p]\\).\n","start":13733,"end":14077}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"张量诱导二范数","source_units":[{"raw_tex":"Another norm is the one induced by the 2-norm on \\(\\R^n\\) (Hackbusch \\cite{hackbusch_tensor_2012} calls it the injective norm) which is defined by\n\\begin{equation}\n    \\norm{\\tns{T}}_2 = \\max_{\\norm{\\vek{s}_i}_2 = 1,\\, 1 \\leq i \\leq p} \\abs{\\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p]}.\n\\end{equation}\n","start":14077,"end":14372}]},{"id":"S011","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"Frobenius 范数的正交不变性","source_units":[{"raw_tex":"Both norms are invariant under orthogonal transformations, so that if \\(\\mat{Q}_1, \\dots, \\mat{Q}_p \\in \\R^{n \\times n}\\) are orthogonal matrices, then \\(\\tns{T}[\\mat{Q}_1, \\dots, \\mat{Q}_p]\\) has the same Frobenius- and 2-norm as \\(\\tns{T}\\).\n","start":14372,"end":14616}]},{"id":"S012","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"二范数被 Frobenius 范数控制","source_units":[{"raw_tex":"As for matrices, the 2-norm is bounded by the Frobenius norm:\n\\begin{equation}\n    \\norm{\\tns{T}}_2\n    % = \\max_{\\norm{\\vek{s}_i}_2 = 1,\\, 1 \\leq i \\leq p} \\abs{\\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p]}\n    = \\max_{\\norm{\\vek{s}_i}_2 = 1,\\, 1 \\leq i \\leq p} \\abs{\\innerprod{\\tns{T}}{\\vek{s}_1 \\otimes \\dots \\otimes \\vek{s}_p}_F}\n    \\leq \\norm{\\tns{T}}_F.\n\\end{equation}\nIn the last inequality we used Cauchy-Schwarz and the fact that \\(\\norm{\\vek{s}_1 \\otimes \\dots \\otimes \\vek{s}_p}_F = 1\\) if all \\(\\vek{s}_i\\) have unit 2-norm.\n","start":14616,"end":15146}]},{"id":"S013","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"范数比较的作者证明","source_units":[{"raw_tex":"As for matrices, the 2-norm is bounded by the Frobenius norm:\n\\begin{equation}\n    \\norm{\\tns{T}}_2\n    % = \\max_{\\norm{\\vek{s}_i}_2 = 1,\\, 1 \\leq i \\leq p} \\abs{\\tns{T}[\\vek{s}_1, \\dots, \\vek{s}_p]}\n    = \\max_{\\norm{\\vek{s}_i}_2 = 1,\\, 1 \\leq i \\leq p} \\abs{\\innerprod{\\tns{T}}{\\vek{s}_1 \\otimes \\dots \\otimes \\vek{s}_p}_F}\n    \\leq \\norm{\\tns{T}}_F.\n\\end{equation}\nIn the last inequality we used Cauchy-Schwarz and the fact that \\(\\norm{\\vek{s}_1 \\otimes \\dots \\otimes \\vek{s}_p}_F = 1\\) if all \\(\\vek{s}_i\\) have unit 2-norm.\n","start":14616,"end":15146}]},{"id":"S014","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"导数近似的应用设定","source_units":[{"raw_tex":"We now need to assume that \\(f \\colon \\R^n \\to \\R\\) is \\(p\\) times continuously differentiable and that as before a sequence of points \\(\\vek{x}_k\\) with a corresponding sequence of steps \\(\\vek{s}_k = \\vek{x}_{k+1} - \\vek{x}_k\\) is given.\n","start":22334,"end":22574}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":["eqn:C-tilde-definition"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"当前近似与平均导数","source_units":[{"raw_tex":"We will denote the approximations to \\(D^p f(\\vek{x}_k)\\) by \\(\\tns{C}_k \\in \\R^{\\otimes^p n}_{\\sym}\\) and the true \\(p\\)th derivative averaged over all points on the line from \\(\\vek{x}_k\\) to \\(\\vek{x}_{k+1}\\) by\n\\begin{equation}\\label{eqn:C-tilde-definition}\n    \\widetilde{\\tns{C}}_k = \\int_0^1 D^p f(\\vek{x}_k + t \\vek{s}_k) \\diff t  \\in \\R^{\\otimes^p n}_{\\sym}.\n\\end{equation}\n","start":22574,"end":22957}]},{"id":"S016","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"平均导数收缩给出导数差","source_units":[{"raw_tex":"This means that, in particular, \\(\\widetilde{\\tns{C}}_k[\\vek{s}_k] = D^{p-1} f(\\vek{x}_{k+1}) - D^{p-1} f(\\vek{x}_k)\\).\n","start":22957,"end":23077}]},{"id":"S017","semantic_type":"definition","kind":"definition","labels":["eqn:generalized-secant-equation"],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"广义割线方程","source_units":[{"raw_tex":"The \\(p\\)th-order analogue of the secant equation \\cref{eqn:short-secant-equation} is given by\n\\begin{equation}\\label{eqn:generalized-secant-equation}\n    \\tns{C}_{k+1} [\\vek{s}_k] = D^{p-1} f(\\vek{x}_{k+1}) - D^{p-1} f(\\vek{x}_k) = \\widetilde{\\tns{C}}_k [\\vek{s}_k]\n\\end{equation}\n","start":23077,"end":23359}]},{"id":"S018","semantic_type":"algorithm","kind":"algorithm","labels":["eqn:update-definition"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"高阶割线更新规则","source_units":[{"raw_tex":"and the generalized update formula for \\(\\tns{C}_k\\) reads\n\\begin{equation}\\label{eqn:update-definition}\n    \\tns{C}_{k+1} = \\argmin_{\\tns{C} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{(\\tns{C} - \\tns{C}_k)[\\mat{W}_k]^p}_F \\text{ s.t. } \\tns{C}[\\vek{s}_k] = \\widetilde{\\tns{C}}_k[\\vek{s}_k]. \\tag{HOSU}\n\\end{equation}\n","start":23359,"end":23669}]},{"id":"S020","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"单步刻画的抽象边界","source_units":[{"raw_tex":"The following theorem provides a full characterization of one step of the update in \\cref{eqn:update-definition}.\nNote that despite the intentional notational similarity the quantities in the statement are independent of the definitions in the previous section.\n","start":24878,"end":25140}]},{"id":"S021","semantic_type":"assumption","kind":"assumption","labels":["item:characterization-explicit","item:characterization-low-rank","item:characterization-min-frobenius","item:characterization-recursive","thm:characterization"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"抽象输入与非退化条件","source_units":[{"raw_tex":"    Let \\(\\tns{C}_{\\bullet} \\in \\R^{\\otimes^p n}_{\\sym}\\) (the current approximation), \\(\\widetilde{\\tns{C}} \\in \\R^{\\otimes^p n}_{\\sym}\\) (the integrated true derivative), a nonsingular matrix \\(\\mat{W} \\in \\R^{n \\times n}\\) (the weight matrix) and a nonzero \\(\\vek{s} \\in \\R^n\\) (the step) be given.\n","start":25185,"end":25487}]},{"id":"S022","semantic_type":"notation","kind":"notation","labels":["item:characterization-explicit","item:characterization-low-rank","item:characterization-min-frobenius","item:characterization-recursive","thm:characterization"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"权矩阵对应的向量","source_units":[{"raw_tex":"    where \\(\\vek{v} = \\mat{W}^{-T}\\mat{W}^{-1}\\vek{s}\\).\n","start":26735,"end":26792}]},{"id":"S023","semantic_type":"theorem","kind":"theorem","labels":["item:characterization-explicit","item:characterization-low-rank","item:characterization-min-frobenius","item:characterization-recursive","thm:characterization"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"高阶割线更新的完整刻画","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm:characterization}\n    Let \\(\\tns{C}_{\\bullet} \\in \\R^{\\otimes^p n}_{\\sym}\\) (the current approximation), \\(\\widetilde{\\tns{C}} \\in \\R^{\\otimes^p n}_{\\sym}\\) (the integrated true derivative), a nonsingular matrix \\(\\mat{W} \\in \\R^{n \\times n}\\) (the weight matrix) and a nonzero \\(\\vek{s} \\in \\R^n\\) (the step) be given.\n    The following equations all have the same unique solution \\(\\tns{C}_+ \\in \\R^{\\otimes^p n}_{\\sym}\\) (the new approximation):\n    \\begin{enumerate}[(a)]\n        \\item\\label{item:characterization-min-frobenius} \\(\\tns{C}_+ = \\argmin_{\\tns{C} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{(\\tns{C} - \\tns{C}_{\\bullet})[\\mat{W}]^p}_F \\text{ s.t. } \\tns{C}[\\vek{s}] = \\widetilde{\\tns{C}}[\\vek{s}]\\)\n        \\item\\label{item:characterization-explicit} \\(\\tns{C}_+ = \\tns{C}_{\\bullet} + \\sum_{j=1}^p (-1)^{j+1} \\binom{p}{j} (\\vek{v}^T \\vek{s})^{-j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{v}} \\otimes (\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]^{j}}\\)\n        \\item\\label{item:characterization-low-rank} \\(\\tns{C}_+ = \\tns{C}_{\\bullet} + P_{\\sym}(\\tns{A} \\otimes \\vek{v})\\) for the unique \\((p-1)\\)-tensor \\(\\tns{A} \\in \\R^{\\otimes^{p-1} n}_{\\sym}\\) which satisfies \\(P_{\\sym}(\\tns{A} \\otimes \\vek{v})[\\vek{s}] = (\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]\\)\n        \\item\\label{item:characterization-recursive} \\((\\tns{C}_+ - \\widetilde{\\tns{C}})[\\mat{W}]^p = (\\tns{C}_{\\bullet} - \\widetilde{\\tns{C}})[\\mat{W}]^p \\Brack{\\mat{I} - \\frac{\\mat{W}^{-1}\\vek{s}\\vek{s}^T \\mat{W}^{-T}}{\\vek{s}^T \\mat{W}^{-T} \\mat{W}^{-1} \\vek{s}}}^p\\)\n    \\end{enumerate}\n    where \\(\\vek{v} = \\mat{W}^{-T}\\mat{W}^{-1}\\vek{s}\\).\n\\end{theorem}","start":25141,"end":26805}]},{"id":"S024","semantic_type":"proof","kind":"proof","labels":["eqn:general-psb-update","eqn:update-explicit-cancellation","eqn:update-minimum-property","eqn:update-secant-equation"],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"完整刻画定理的作者证明","source_units":[{"raw_tex":"\\begin{proof}\n    We will first prove the result for \\(\\mat{W} = \\mat{I}\\) and then see how that implies the full result for any nonsingular weight matrix \\(\\mat{W}\\).\n\n    The first characterization can be rewritten as\n    \\begin{equation}\\label{eqn:general-psb-update}\n        \\tns{C}_+ = \\tns{C}_{\\bullet} + \\argmin_{\\tns{U} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{\\tns{U}}_F \\text{ s.t. } \\tns{U}[\\vek{s}] = (\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}].\n    \\end{equation}\n    Let \\(\\mat{Q} \\in \\R^{n \\times n}\\) be an orthogonal matrix that maps \\(\\vek{e}_1\\) to some scalar multiple of \\(\\vek{s}\\).\n    This means \\(\\tns{U}[\\mat{Q}]^p\\) has the same Frobenius norm as \\(\\tns{U}\\) and \\(\\tns{U}[\\mat{Q}]^p[\\vek{e}_{i_1}, \\dots, \\vek{e}_{i_p}]\\) is fully determined by the equality constraint in \\cref{eqn:general-psb-update} if \\(1 \\in \\{i_1, \\dots, i_p\\}\\) because of symmetry.\n    On the other hand, if \\(1 \\notin \\{i_1, \\dots, i_p\\}\\) there are no constraints on the values of \\(\\tns{U}[\\mat{Q}]^p[\\vek{e}_{i_1}, \\dots, \\vek{e}_{i_p}]\\), so the unique choice that minimizes \\(\\norm{\\tns{U}[\\mat{Q}]^p}_F = \\norm{\\tns{U}}_F\\) is clearly to set all of these values to zero.\n    Therefore, there is a unique solution to the minimization problem in \\labelcref{item:characterization-min-frobenius}, and it is fully characterized by the fact that the update tensor \\(\\tns{U} = \\tns{C}_+ - \\tns{C}_{\\bullet}\\) is symmetric and satisfies the following two properties:\n    \\begin{subequations}\n        \\begin{align}\n            \\tns{U}[\\vek{s}]                     & = \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}] \\label{eqn:update-secant-equation}        \\\\\n            \\tns{U}[\\vek{u}_1, \\dots, \\vek{u}_p] & = 0 \\text{ if all } \\vek{u}_i \\text{ are orthogonal to } \\vek{s} \\label{eqn:update-minimum-property}\n        \\end{align}\n    \\end{subequations}\n    We will use this as the basis to show the equivalence with all other characterizations.\n\n    In \\labelcref{item:characterization-explicit} we claim that the update \\(\\tns{U}\\) has the form\n    \\begin{equation}\n        \\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}\n    \\end{equation}\n    This tensor is clearly symmetric since it is a sum of symmetric tensors.\n    To show that it satisfies property \\cref{eqn:update-secant-equation} consider\n    \\begin{subequations}\\label{eqn:update-explicit-cancellation}\n        \\begin{align}\n             & \\phantom{={}}\\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}[\\vek{s}]                                                                                                                                                \\\\\n            \\begin{split}\n                {} = \\sum_{j=1}^p (-1)^{j+1} \\norm{\\vek{s}}_2^{-2j} \\bigg( \\tbinom{p-1}{j-1} P_{\\sym} \\Paren{\\Paren{\\otimes^{j-1} \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}} \\norm{\\vek{s}}_2^2 \\phantom{\\bigg)}\\\\\n                {} + \\tbinom{p-1}{j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j+1}} \\bigg) \\phantom{\\norm{\\vek{s}_2^2}}\n            \\end{split} \\\\\n             & = P_{\\sym} \\Paren{ \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}] } = \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}].\n        \\end{align}\n    \\end{subequations}\n    The first equality uses the fact that each summand is the symmetric projection of an outer product of two symmetric tensors, a \\(j\\)-tensor and a \\((p-j)\\)-tensor.\n    Therefore, there are \\(\\binom{p}{j}\\) distinct ways to orient this outer product and for \\(\\binom{p-1}{j-1}\\) of them the vector \\(\\vek{s}\\) is applied to the \\(j\\)-tensor and for \\(\\binom{p-1}{j}\\) to the \\((p-j)\\)-tensor.\n    A close examination of the expression on the second line shows that it is a telescoping sum where all terms except the ones with coefficients \\(\\binom{p-1}{0}\\) and \\(\\binom{p-1}{p}\\) cancel out.\n    Because \\(\\binom{p-1}{p} = 0\\) the only remaining term is the symmetric projection of \\((\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]\\).\n    Property \\cref{eqn:update-minimum-property} follows from a similar consideration to the one above.\n    If \\(\\vek{u}_1, \\dots, \\vek{u}_p\\) are all orthogonal to \\(\\vek{s}\\), then\n    \\begin{equation}\n        P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}[\\vek{u}_1, \\dots, \\vek{u}_p] = 0\n    \\end{equation}\n    for \\(j \\geq 1\\) because no matter how the outer product is oriented there is always a factor of \\(\\vek{s}^T \\vek{u}_i = 0\\) in the result.\n\n    For \\labelcref{item:characterization-low-rank} we need to show that we can always write the update in the form \\(P_{\\sym}(\\tns{A} \\otimes \\vek{s})\\) for some symmetric \\((p-1)\\)-tensor \\(\\tns{A}\\) and that \\(\\tns{A}\\) is unique such that the corresponding update satisfies the secant equation \\cref{eqn:update-secant-equation}.\n    Note that the expression for the update in \\labelcref{item:characterization-explicit} is already of the form \\(P_{\\sym}(\\tns{A} \\otimes \\vek{s})\\):\n    \\begin{subequations}\n        \\begin{align}\n             & \\phantom{={}}\\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}                               \\\\\n             & = P_{\\sym} \\Paren{ P_{\\sym} \\Paren{ \\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} \\Paren{\\otimes^{j-1} \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j} } \\otimes \\vek{s} }\n        \\end{align}\n    \\end{subequations}\n    It remains to show that among all updates of the form \\(P_{\\sym}(\\tns{A} \\otimes \\vek{s})\\) there is only one choice of \\(\\tns{A} \\in \\R^{\\otimes^p n}_{\\sym}\\) such that \\cref{eqn:update-secant-equation} holds.\n    Consider the linear map\n    \\begin{align*}\n        \\phi \\colon \\R^{\\otimes^{p-1} n}_{\\sym} & \\to \\R^{\\otimes^{p-1} n}_{\\sym}                    \\\\\n        \\tns{A}                                 & \\mapsto P_{\\sym}(\\tns{A} \\otimes \\vek{s})[\\vek{s}]\n    \\end{align*}\n    which maps a finite-dimensional vector space to itself.\n    Combining what we already showed for \\labelcref{item:characterization-explicit} with the observation that the update in \\labelcref{item:characterization-explicit} is of the desired form, this map is surjective (we can prescribe any \\((\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]\\)) and so it must be bijective, which shows uniqueness of \\(\\tns{A}\\).\n\n    Lastly, \\labelcref{item:characterization-recursive} claims that the update can be written as\n    \\begin{equation}\n        \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}} - \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}} \\Brack{\\mat{I} - \\frac{\\vek{s} \\vek{s}^T}{\\norm{\\vek{s}}_2^2}}^p.\n    \\end{equation}\n    Clearly, property \\cref{eqn:update-secant-equation} is satisfied because applying this tensor to \\(\\vek{s}\\) makes the second term vanish, leaving only the desired result.\n    Moreover, for any vector \\(\\vek{u}\\) that is orthogonal to \\(\\vek{s}\\) the matrix \\(\\mat{I} - \\frac{\\vek{s} \\vek{s}^T}{\\norm{\\vek{s}}_2^2}\\) maps \\(\\vek{u}\\) to itself.\n    This means applying the tensor above to \\(\\vek{u}_1, \\dots, \\vek{u}_p\\), all of which are orthogonal to \\(\\vek{s}\\), will give zero because both terms cancel out.\n    This is property \\cref{eqn:update-minimum-property}.\n\n    Now that the equivalence has been established for \\(\\mat{W} = \\mat{I}\\), we consider the general case where \\(\\mat{W} \\in \\R^{n \\times n}\\) is any nonsingular matrix.\n    The minimization in \\labelcref{item:characterization-min-frobenius}\n    \\begin{equation}\n        \\tns{C}_+ = \\argmin_{\\tns{C} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{(\\tns{C} - \\tns{C}_{\\bullet})[\\mat{W}]^p}_F \\text{ s.t. } \\tns{C}[\\vek{s}] = \\widetilde{\\tns{C}}[\\vek{s}]\n    \\end{equation}\n    can be rewritten using \\(\\tns{D}_{\\bullet} = \\tns{C}_{\\bullet}[\\mat{W}]^p\\), \\(\\widetilde{\\tns{D}} = \\widetilde{\\tns{C}}[\\mat{W}]^p\\), \\(\\tns{D}_+ = \\tns{C}_+[\\mat{W}]^p\\) and \\(\\vek{r} = \\mat{W}^{-1}\\vek{s}\\) as\n    \\begin{equation}\n        \\tns{D}_+ = \\argmin_{\\tns{D} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{\\tns{D} - \\tns{D}_{\\bullet}}_F \\text{ s.t. } \\tns{D}[\\vek{r}] = \\widetilde{\\tns{D}}[\\vek{r}].\n    \\end{equation}\n    Applying the existing characterizations and the fact that \\(\\tns{C}_+ = \\tns{D}_+[\\mat{W}^{-1}]^p\\) we get the claim after some algebraic manipulations.\n\\end{proof}","start":26807,"end":35513}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"先证无权再作变换","source_units":[{"raw_tex":"    We will first prove the result for \\(\\mat{W} = \\mat{I}\\) and then see how that implies the full result for any nonsingular weight matrix \\(\\mat{W}\\).\n\n","start":26821,"end":26976}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:general-psb-update","eqn:update-explicit-cancellation"],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"改写为最小范数增量","source_units":[{"raw_tex":"    The first characterization can be rewritten as\n    \\begin{equation}\\label{eqn:general-psb-update}\n        \\tns{C}_+ = \\tns{C}_{\\bullet} + \\argmin_{\\tns{U} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{\\tns{U}}_F \\text{ s.t. } \\tns{U}[\\vek{s}] = (\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}].\n    \\end{equation}\n","start":26976,"end":27285}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"沿步长选择正交坐标","source_units":[{"raw_tex":"    Let \\(\\mat{Q} \\in \\R^{n \\times n}\\) be an orthogonal matrix that maps \\(\\vek{e}_1\\) to some scalar multiple of \\(\\vek{s}\\).\n    This means \\(\\tns{U}[\\mat{Q}]^p\\) has the same Frobenius norm as \\(\\tns{U}\\) and \\(\\tns{U}[\\mat{Q}]^p[\\vek{e}_{i_1}, \\dots, \\vek{e}_{i_p}]\\) is fully determined by the equality constraint in \\cref{eqn:general-psb-update} if \\(1 \\in \\{i_1, \\dots, i_p\\}\\) because of symmetry.\n","start":27285,"end":27692}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"自由坐标唯一取零","source_units":[{"raw_tex":"    On the other hand, if \\(1 \\notin \\{i_1, \\dots, i_p\\}\\) there are no constraints on the values of \\(\\tns{U}[\\mat{Q}]^p[\\vek{e}_{i_1}, \\dots, \\vek{e}_{i_p}]\\), so the unique choice that minimizes \\(\\norm{\\tns{U}[\\mat{Q}]^p}_F = \\norm{\\tns{U}}_F\\) is clearly to set all of these values to zero.\n","start":27692,"end":27988}]},{"id":"S029","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:update-explicit-cancellation","eqn:update-minimum-property","eqn:update-secant-equation"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"唯一极小更新的双条件","source_units":[{"raw_tex":"    Therefore, there is a unique solution to the minimization problem in \\labelcref{item:characterization-min-frobenius}, and it is fully characterized by the fact that the update tensor \\(\\tns{U} = \\tns{C}_+ - \\tns{C}_{\\bullet}\\) is symmetric and satisfies the following two properties:\n    \\begin{subequations}\n        \\begin{align}\n            \\tns{U}[\\vek{s}]                     & = \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}] \\label{eqn:update-secant-equation}        \\\\\n            \\tns{U}[\\vek{u}_1, \\dots, \\vek{u}_p] & = 0 \\text{ if all } \\vek{u}_i \\text{ are orthogonal to } \\vek{s} \\label{eqn:update-minimum-property}\n        \\end{align}\n    \\end{subequations}\n    We will use this as the basis to show the equivalence with all other characterizations.\n","start":27988,"end":28765}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"提出对称显式增量","source_units":[{"raw_tex":"    In \\labelcref{item:characterization-explicit} we claim that the update \\(\\tns{U}\\) has the form\n    \\begin{equation}\n        \\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}\n    \\end{equation}\n    This tensor is clearly symmetric since it is a sum of symmetric tensors.\n","start":28766,"end":29164}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"按收缩落点统计排列","source_units":[{"raw_tex":"    To show that it satisfies property \\cref{eqn:update-secant-equation} consider\n    \\begin{subequations}\\label{eqn:update-explicit-cancellation}\n        \\begin{align}\n             & \\phantom{={}}\\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}[\\vek{s}]                                                                                                                                                \\\\\n            \\begin{split}\n                {} = \\sum_{j=1}^p (-1)^{j+1} \\norm{\\vek{s}}_2^{-2j} \\bigg( \\tbinom{p-1}{j-1} P_{\\sym} \\Paren{\\Paren{\\otimes^{j-1} \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}} \\norm{\\vek{s}}_2^2 \\phantom{\\bigg)}\\\\\n                {} + \\tbinom{p-1}{j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j+1}} \\bigg) \\phantom{\\norm{\\vek{s}_2^2}}\n            \\end{split} \\\\\n             & = P_{\\sym} \\Paren{ \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}] } = \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}].\n        \\end{align}\n    \\end{subequations}\n    The first equality uses the fact that each summand is the symmetric projection of an outer product of two symmetric tensors, a \\(j\\)-tensor and a \\((p-j)\\)-tensor.\n    Therefore, there are \\(\\binom{p}{j}\\) distinct ways to orient this outer product and for \\(\\binom{p-1}{j-1}\\) of them the vector \\(\\vek{s}\\) is applied to the \\(j\\)-tensor and for \\(\\binom{p-1}{j}\\) to the \\((p-j)\\)-tensor.\n","start":29164,"end":30768}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"交错和望远镜消去","source_units":[{"raw_tex":"             & = P_{\\sym} \\Paren{ \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}] } = \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}].\n","start":30176,"end":30329},{"raw_tex":"    A close examination of the expression on the second line shows that it is a telescoping sum where all terms except the ones with coefficients \\(\\binom{p-1}{0}\\) and \\(\\binom{p-1}{p}\\) cancel out.\n    Because \\(\\binom{p-1}{p} = 0\\) the only remaining term is the symmetric projection of \\((\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]\\).\n","start":30768,"end":31114}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"显式式在正交方向消失","source_units":[{"raw_tex":"    Property \\cref{eqn:update-minimum-property} follows from a similar consideration to the one above.\n    If \\(\\vek{u}_1, \\dots, \\vek{u}_p\\) are all orthogonal to \\(\\vek{s}\\), then\n    \\begin{equation}\n        P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}[\\vek{u}_1, \\dots, \\vek{u}_p] = 0\n    \\end{equation}\n    for \\(j \\geq 1\\) because no matter how the outer product is oriented there is always a factor of \\(\\vek{s}^T \\vek{u}_i = 0\\) in the result.\n","start":31114,"end":31633}]},{"id":"S034","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:update-explicit-cancellation"],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"无权显式式确定唯一更新","source_units":[{"raw_tex":"    In \\labelcref{item:characterization-explicit} we claim that the update \\(\\tns{U}\\) has the form\n    \\begin{equation}\n        \\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}\n    \\end{equation}\n    This tensor is clearly symmetric since it is a sum of symmetric tensors.\n    To show that it satisfies property \\cref{eqn:update-secant-equation} consider\n    \\begin{subequations}\\label{eqn:update-explicit-cancellation}\n        \\begin{align}\n             & \\phantom{={}}\\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}[\\vek{s}]                                                                                                                                                \\\\\n            \\begin{split}\n                {} = \\sum_{j=1}^p (-1)^{j+1} \\norm{\\vek{s}}_2^{-2j} \\bigg( \\tbinom{p-1}{j-1} P_{\\sym} \\Paren{\\Paren{\\otimes^{j-1} \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}} \\norm{\\vek{s}}_2^2 \\phantom{\\bigg)}\\\\\n                {} + \\tbinom{p-1}{j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j+1}} \\bigg) \\phantom{\\norm{\\vek{s}_2^2}}\n            \\end{split} \\\\\n             & = P_{\\sym} \\Paren{ \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}] } = \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}].\n        \\end{align}\n    \\end{subequations}\n    The first equality uses the fact that each summand is the symmetric projection of an outer product of two symmetric tensors, a \\(j\\)-tensor and a \\((p-j)\\)-tensor.\n    Therefore, there are \\(\\binom{p}{j}\\) distinct ways to orient this outer product and for \\(\\binom{p-1}{j-1}\\) of them the vector \\(\\vek{s}\\) is applied to the \\(j\\)-tensor and for \\(\\binom{p-1}{j}\\) to the \\((p-j)\\)-tensor.\n    A close examination of the expression on the second line shows that it is a telescoping sum where all terms except the ones with coefficients \\(\\binom{p-1}{0}\\) and \\(\\binom{p-1}{p}\\) cancel out.\n    Because \\(\\binom{p-1}{p} = 0\\) the only remaining term is the symmetric projection of \\((\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]\\).\n    Property \\cref{eqn:update-minimum-property} follows from a similar consideration to the one above.\n    If \\(\\vek{u}_1, \\dots, \\vek{u}_p\\) are all orthogonal to \\(\\vek{s}\\), then\n    \\begin{equation}\n        P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}[\\vek{u}_1, \\dots, \\vek{u}_p] = 0\n    \\end{equation}\n    for \\(j \\geq 1\\) because no matter how the outer product is oriented there is always a factor of \\(\\vek{s}^T \\vek{u}_i = 0\\) in the result.\n","start":28766,"end":31633}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"从显式式提出外积因子","source_units":[{"raw_tex":"    For \\labelcref{item:characterization-low-rank} we need to show that we can always write the update in the form \\(P_{\\sym}(\\tns{A} \\otimes \\vek{s})\\) for some symmetric \\((p-1)\\)-tensor \\(\\tns{A}\\) and that \\(\\tns{A}\\) is unique such that the corresponding update satisfies the secant equation \\cref{eqn:update-secant-equation}.\n    Note that the expression for the update in \\labelcref{item:characterization-explicit} is already of the form \\(P_{\\sym}(\\tns{A} \\otimes \\vek{s})\\):\n    \\begin{subequations}\n        \\begin{align}\n             & \\phantom{={}}\\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} P_{\\sym} \\Paren{\\Paren{\\otimes^j \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j}}                               \\\\\n             & = P_{\\sym} \\Paren{ P_{\\sym} \\Paren{ \\sum_{j=1}^p (-1)^{j+1} \\tbinom{p}{j} \\norm{\\vek{s}}_2^{-2j} \\Paren{\\otimes^{j-1} \\vek{s}} \\otimes \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}}[\\vek{s}]^{j} } \\otimes \\vek{s} }\n        \\end{align}\n    \\end{subequations}\n","start":31634,"end":32673}]},{"id":"S036","semantic_type":"notation","kind":"notation","labels":["eqn:update-explicit-cancellation"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"因子阶数的原文冲突","source_units":[{"raw_tex":"    It remains to show that among all updates of the form \\(P_{\\sym}(\\tns{A} \\otimes \\vek{s})\\) there is only one choice of \\(\\tns{A} \\in \\R^{\\otimes^p n}_{\\sym}\\) such that \\cref{eqn:update-secant-equation} holds.\n","start":32673,"end":32888}]},{"id":"S037","semantic_type":"definition","kind":"definition","labels":["eqn:update-explicit-cancellation"],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"因子到割线数据的线性映射","source_units":[{"raw_tex":"    Consider the linear map\n    \\begin{align*}\n        \\phi \\colon \\R^{\\otimes^{p-1} n}_{\\sym} & \\to \\R^{\\otimes^{p-1} n}_{\\sym}                    \\\\\n        \\tns{A}                                 & \\mapsto P_{\\sym}(\\tns{A} \\otimes \\vek{s})[\\vek{s}]\n    \\end{align*}\n    which maps a finite-dimensional vector space to itself.\n","start":32888,"end":33217}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"作者断言因子映射满射","source_units":[{"raw_tex":"    Combining what we already showed for \\labelcref{item:characterization-explicit} with the observation that the update in \\labelcref{item:characterization-explicit} is of the desired form, this map is surjective (we can prescribe any \\((\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]\\)) and so it must be bijective, which shows uniqueness of \\(\\tns{A}\\).\n","start":33217,"end":33577}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"有限维满射推出唯一因子","source_units":[{"raw_tex":"    Combining what we already showed for \\labelcref{item:characterization-explicit} with the observation that the update in \\labelcref{item:characterization-explicit} is of the desired form, this map is surjective (we can prescribe any \\((\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet})[\\vek{s}]\\)) and so it must be bijective, which shows uniqueness of \\(\\tns{A}\\).\n\n","start":33217,"end":33578}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"提出投影差形式","source_units":[{"raw_tex":"    Lastly, \\labelcref{item:characterization-recursive} claims that the update can be written as\n    \\begin{equation}\n        \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}} - \\paren{\\widetilde{\\tns{C}} - \\tns{C}_{\\bullet}} \\Brack{\\mat{I} - \\frac{\\vek{s} \\vek{s}^T}{\\norm{\\vek{s}}_2^2}}^p.\n    \\end{equation}\n","start":33578,"end":33887}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"投影消去步长方向","source_units":[{"raw_tex":"    Clearly, property \\cref{eqn:update-secant-equation} is satisfied because applying this tensor to \\(\\vek{s}\\) makes the second term vanish, leaving only the desired result.\n","start":33887,"end":34063}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"投影固定正交方向","source_units":[{"raw_tex":"    Moreover, for any vector \\(\\vek{u}\\) that is orthogonal to \\(\\vek{s}\\) the matrix \\(\\mat{I} - \\frac{\\vek{s} \\vek{s}^T}{\\norm{\\vek{s}}_2^2}\\) maps \\(\\vek{u}\\) to itself.\n    This means applying the tensor above to \\(\\vek{u}_1, \\dots, \\vek{u}_p\\), all of which are orthogonal to \\(\\vek{s}\\), will give zero because both terms cancel out.\n    This is property \\cref{eqn:update-minimum-property}.\n","start":34063,"end":34460}]},{"id":"S043","semantic_type":"derived_fact","kind":"derived_fact","labels":["eqn:update-explicit-cancellation"],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"无权四种刻画已建立","source_units":[{"raw_tex":"    Now that the equivalence has been established for \\(\\mat{W} = \\mat{I}\\), we consider the general case where \\(\\mat{W} \\in \\R^{n \\times n}\\) is any nonsingular matrix.\n","start":34461,"end":34632}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"回到一般加权极小问题","source_units":[{"raw_tex":"    The minimization in \\labelcref{item:characterization-min-frobenius}\n    \\begin{equation}\n        \\tns{C}_+ = \\argmin_{\\tns{C} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{(\\tns{C} - \\tns{C}_{\\bullet})[\\mat{W}]^p}_F \\text{ s.t. } \\tns{C}[\\vek{s}] = \\widetilde{\\tns{C}}[\\vek{s}]\n    \\end{equation}\n","start":34632,"end":34922}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"加权问题变为无权问题","source_units":[{"raw_tex":"    can be rewritten using \\(\\tns{D}_{\\bullet} = \\tns{C}_{\\bullet}[\\mat{W}]^p\\), \\(\\widetilde{\\tns{D}} = \\widetilde{\\tns{C}}[\\mat{W}]^p\\), \\(\\tns{D}_+ = \\tns{C}_+[\\mat{W}]^p\\) and \\(\\vek{r} = \\mat{W}^{-1}\\vek{s}\\) as\n    \\begin{equation}\n        \\tns{D}_+ = \\argmin_{\\tns{D} \\in \\R^{\\otimes^p n}_{\\sym}} \\norm{\\tns{D} - \\tns{D}_{\\bullet}}_F \\text{ s.t. } \\tns{D}[\\vek{r}] = \\widetilde{\\tns{D}}[\\vek{r}].\n    \\end{equation}\n","start":34922,"end":35345}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["eqn:update-explicit-cancellation"],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"逆变换恢复全部刻画","source_units":[{"raw_tex":"    Applying the existing characterizations and the fact that \\(\\tns{C}_+ = \\tns{D}_+[\\mat{W}^{-1}]^p\\) we get the claim after some algebraic manipulations.\n","start":35345,"end":35502}]},{"id":"S047","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S047"],"declared_external":true,"declaration_text":["原始先修列表第二项：标准有限维线性代数，包括正交基变换；此处由标注者显式化。"],"source_only":false,"title":"沿非零向量的正交基选择","source_units":[]},{"id":"S048","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S048"],"declared_external":true,"declaration_text":["原始先修列表第二项：标准有限维线性代数，包括自映射满射与双射的等价。"],"source_only":false,"title":"有限维自映射的满射与双射","source_units":[]},{"id":"S049","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S049"],"declared_external":true,"declaration_text":["原始先修列表第三项：标准二项式恒等式与排列计数。"],"source_only":false,"title":"二项式计数与端点恒等式","source_units":[]},{"id":"S050","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S050"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；原始先修列表未单列，使用位置为 L339。"],"source_only":false,"title":"沿线段的微积分基本定理","source_units":[]},{"id":"S051","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S051"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；原始先修列表未单列，作者在 L265 明确点名 Cauchy–Schwarz。"],"source_only":false,"title":"Cauchy–Schwarz 不等式","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051"}}}}
{"record_id":"2301.08726v2","paper":{"title":"Continuous Newton-like Methods featuring Inertia and Variable Mass","authors":["Castera, Camille","Attouch, Hedy","Fadili, Jalal","Ochs, Peter"],"arxiv_id":"2301.08726","version":"v2","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/2301.08726v2","pdf_url":"https://arxiv.org/pdf/2301.08726v2","doi":"10.1137/23M1549675"},"context":{"raw_tex":"\tBoth steps are highly challenging, our work focuses on the ODE matter. We study the following second-order dynamical system in a general setting:\n\t\\begin{equation}\\tag{VM-DIN-AVD}\\label{eq::VMDINAVD}\n\t\t\\varepsilon(t) \\ddot{x}(t) + \\alpha(t) \\dot x(t) + \\beta \\Hf(x(t))\\dot{x}(t)+\\gf(x(t))=0,\n\t\t\\quad t\\geq 0,\n\t\\end{equation}\n\twhere $f\\colon \\R^n\\to\\R$ is a smooth function, with gradient $\\gf$ and Hessian $\\Hf$ defined on $\\R^n$ equipped with scalar product $\\langle\\cdot,\\cdot\\rangle$, and induced norm $\\norm{\\cdot}$.\n\tThe system is controlled by two functions $\\eps,\\alpha\\colon\\R_+\\to\\R_+$ (where $\\R_+=[0,+\\infty[$) and a parameter $\\beta> 0$ that define the type of dynamics that drives the trajectory (or solution) $x\\colon\\R_+\\to\\R^n$, whose first- and second-order derivatives are denoted $\\dx$ and $\\ddx$ respectively.\n\n\n\tThis paper emphasizes the relation between \\eqref{eq::VMDINAVD} and special cases.\n\tIndeed, taking $\\eps=\\alpha=0$, one obtains\\footnote{\\ref{eq::CN} is usually considered with $\\beta=1$, we put $\\beta$ in the system to ease the discussions below.} the Continuous Newton (CN) method \\citep{gavurin1958nonlinear}\n\t\\begin{equation}\\tag{CN}\\label{eq::CN}\n\t\t\\beta \\Hf(x_N(t))\\dot{x}_N(t)+\\gf(x_N(t))=0, \\quad t\\geq 0,\n\t\\end{equation}\n\tknown notably for being invariant to affine transformations and yielding fast vanishing of the gradient (see Section~\\ref{sec::Control}).\n\tIn fact, this observation shows that \\eqref{eq::VMDINAVD} is a singular perturbation of \\eqref{eq::CN}, which also justifies the terminology ``Newton-like''  in DIN.\n\tWhen $\\alpha\\neq 0$ but $\\eps=0$, we recover the Levenberg--Marquardt (LM) method,\n\t\\begin{equation}\\tag{LM}\\label{eq::LM}\n\t\t\\alpha(t) \\dot x_{LM}(t) + \\beta \\Hf(x_{LM}(t))\\dot{x}_{LM}(t)+\\gf(x_{LM}(t))=0, \\quad t\\geq 0,\n\t\\end{equation}\n\talso known as regularized Newton method since it stabilizes \\eqref{eq::CN}.\n\tIn the rest of the paper, the solutions of \\eqref{eq::CN} and \\eqref{eq::LM} will always be denoted by $x_N$ and $x_{LM}$ respectively. Since the introduction of DIN, it is known (see \\cite{alvarez2002second}) that for $\\alpha=0$, $\\beta=1$, and $\\eps$ constant and small, \\eqref{eq::VMDINAVD} is a ``perturbed'' Newton method since the distance between the solutions of \\eqref{eq::VMDINAVD} and \\eqref{eq::CN}  is at most proportional to $\\sqrt{\\eps}$ at all times.\n\n\n\tWe always assume the following on the system \\eqref{eq::VMDINAVD} and the functions $f,\\eps,\\alpha$ defined in the introduction.\n\t\t\\begin{assumption}\\label{ass:general}\n\t\t\t\\textbullet\\ $f: \\R^n \\to \\R$ is convex, twice continuously differentiable, coercive, and strongly convex on bounded subsets of $\\R^n$.\n\t\t\tWe fix $x_0$, $\\dot{x}_0\\in\\R^n$, such that, unless stated otherwise, \\eqref{eq::VMDINAVD} has initial condition $(x(0),\\dx(0))=(x_0,\\dx_0)$, and \\eqref{eq::CN} and \\eqref{eq::LM} have initial conditions $x_{N}(0)=x_{LM}(0)=x_0$.\n\\\\\n\t\t\t\\textbullet\\ $\\alpha: \\R_+ \\to \\R$ is non-increasing, non-negative and differentiable. $\\eps: \\R_+ \\to \\R$ is non-increasing, positive, and twice differentiable with bounded second derivative.\n\t\t\tWe fix initial values: $\\eps(0)=\\eps_0>0$, $\\eps'(0)=\\eps'_0\\leq0$ and $\\alpha(0)=\\alpha_0\\geq 0$.\n\t\t\\end{assumption}\n\n\n\t\\section{Approximate Solutions and Asymptotic Analysis on Quadratics}\\label{sec::Quads}\n\tIn addition to Assumption~\\ref{ass:general}, we consider the case where $f$ is a strongly convex quadratic function in order to study the asymptotic behavior of \\eqref{eq::VMDINAVD} w.r.t.\\ \\eqref{eq::CN} and \\eqref{eq::LM}. Quadratic functions are the prototypical example of strongly convex functions. In particular, any strongly convex function can be locally approximated by a quadratic one around its minimizer, making the latter a good model for understanding the local behavior of dynamics.\n\tIn this section, $f$ is quadratic: $f(y)=\\frac{1}{2}\\Vert Ay-b\\Vert_2^2$ for all $y\\in \\R^n$, where $A\\in\\R^{n\\times n}$ is symmetric positive definite and $b\\in\\R^n$. Without loss of generality, we take $b=0$, so that the unique minimum is $x^\\star=0$.\n\t\\subsection{Setting: the Special Case of Quadratic Functions}\n\tQuadratic functions are particularly interesting in our setting since DIN-like ODEs take a simpler form (as observed in \\cite{attouch2020first,shi2021understanding}). Indeed, $\\forall y\\in\\R^n$, $\\gf(y) = A^TAy$ and $\\Hf(y)=A^T A$. Since $\\Hf(y)$ is independent of $y$ we can rewrite \\eqref{eq::VMDINAVD} in an eigenspace\\footnote{This can be generalized to the case where $A^TA$ is only semi-definite by considering orthogonal projections on an eigenspace spanned by the positive eigenvalues of $A^TA$.} of  $A^TA$. That is, we can study the ODE coordinate-wise by looking at one-dimensional problems of the form\n\t\\begin{equation}\\label{eq::QuadeDINAVD}\\tag{Q1-VM-DIN-AVD}\n\t\t\\eps(t)\\ddx(t)+(\\alpha(t)+\\beta\\lambda)\\dx(t)+\\lambda x(t)=0, \\quad t\\geq 0.\n\t\\end{equation}\n\tHere (and throughout what follows) $\\lambda>0$ denotes any eigenvalue of $A^TA$ and $x\\colon\\R_+\\to\\R$ now denotes the corresponding  coordinate (function) of the solution of \\eqref{eq::VMDINAVD} in an eigenspace of $A^TA$. The dynamics \\eqref{eq::QuadeDINAVD} is a \\emph{linear} second-order ODE in $x$ with non-constant coefficients. Similarly, \\eqref{eq::LM} can be rewritten coordinate-wise as\n\t\\begin{equation}\\label{eq::quadLM}\\tag{Q1-LM}\n\t\t(\\alpha(t)+\\beta\\lambda)\\dx_{LM}(t)+\\lambda x_{LM}(t)=0, \\quad t\\geq 0,\n\t\\end{equation}\n\twhere $\\xlm\\colon\\R_+\\to\\R$, and \\eqref{eq::CN} becomes\n\t\\begin{equation}\\label{eq::quadCN}\\tag{Q1-CN}\n\t\t\\beta\\dx_N(t)+ x_N(t)=0, \\quad t\\geq 0,\n\t\\end{equation}\n\twhere again, $x_N\\colon\\R_+\\to\\R$ is one-dimensional.\n\tObserve in particular that \\eqref{eq::CN} and \\eqref{eq::LM} are now first-order \\emph{linear} ODEs, whose solutions have closed forms: $\\forall t\\geq 0$,\n\t\\begin{equation}\\label{eq::closeformLMN}\n\t\tx_N(t) = x_0e^{-\\frac{t}{\\beta}}\n\t\t\\quad \\text{and}\\quad\n\t\tx_{LM}(t) = x_0\\exp\\left(-\\int_0^t \\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}\\diff s \\right).\n\t\\end{equation}\n\n\n\tThe advantage of this approach is the possibility to estimate the error made using \\eqref{eq::LG} w.r.t.\\ the true solutions of \\eqref{eq::GenericODE}. This is expressed in the following theorem which gathers results from \\cite{blumenthal1899uber,olver1961error,taylor1982improved}.\n\t\\begin{theorem}[\\cite{olver1997asymptotics}]\\label{thm::olver}\n\t\tLet $r\\colon \\R_{+}\\to \\R$ be a real, positive, twice continuously differentiable function, and define $\\varphi(t) = \\frac{4r(t)r''(t)-5r'(t)^2}{16r(t)^{5/2}}$ for all $t\\geq 0$. Then for any $T>0$, the differential equation,\n\t\t\\begin{equation}\\label{eq::GenericODEthm}\n\t\t\t\\ddot{z}(t)-r(t)z(t)=0,\\quad t\\in [0,T],\n\t\t\\end{equation}\n\t\thas two real and twice continuously differentiable solutions defined $\\forall t\\in[0,T]$ by,\n\t\t\\begin{equation*}\n\t\t\tz_1(t) = \\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t \\sqrt{r(s)}\\diff s\\right) \\ \\ \\text{and} \\ \\,\n\t\t\tz_2(t) = \\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\exp\\left(-\\int_0^t \\sqrt{r(s)}\\diff s\\right),\n\t\t\\end{equation*}\n\t\twhere $\\displaystyle\\vert\\delta_1(t)\\vert\\leq \\exp\\left(\\frac{1}{2} \\int_0^t \\vert \\varphi(s)\\vert\\diff s\\right)-1$ and $\\displaystyle\\vert\\delta_2(t)\\vert\\leq \\exp\\left(-\\frac{1}{2} \\int_t^{T} \\vert \\varphi(s)\\vert\\diff s\\right)-1$.\n\t\tIf in addition $\\displaystyle\\int_0^{+\\infty} \\vert\\varphi(s)\\vert\\diff s <+\\infty$, then the results above also hold for $T=+\\infty$.\n\t\\end{theorem}\n\t\\begin{remark}\\label{rem::olver} We make the following remarks regarding the above result.\\\\\n\t\t--\\ Note that $z_1$ and $z_2$ in Theorem~\\ref{thm::olver} are \\emph{exact} solutions to \\eqref{eq::GenericODEthm}. The LG approximations $\\hat z_1$ and $\\hat z_2$ are obtained by neglecting the unknown functions $\\delta_1$ and $\\delta_2$ in $z_1$ and $z_2$. The theorem gives a \\emph{non-asymptotic} bound for the errors $\\vert z_1(t)-\\hat{z}_1(t)\\vert$ and $\\vert z_2(t)-\\hat{z}_2(t)\\vert$, $t\\geq 0$.\n\t\t\\\\\n\t\t--\\ Since we assumed $r$ to be twice continuously differentiable and positive, $\\varphi$ is continuous, so it is integrable except maybe for $t\\to+\\infty$.\n\t\t\\\\\n\t\t--\\ For the sake of simplicity, the formulation of Theorem~\\ref{thm::olver} slightly differs from that in \\cite{olver1997asymptotics}, the original formulation can be recovered by a change of variable.\n\t\\end{remark}\n\n\n\tWe now proceed to make use of the LG method for approximating the solutions of \\eqref{eq::QuadeDINAVD}. The reader only interested in the result can jump directly to the  Section~\\ref{sec::mainasymptoticres}. We first make the following assumption.\n\t\\begin{assumption}\\label{ass:smooth}\n\t\tThe functions $\\alpha$ and $\\eps$ are three times continuously differentiable, and $\\eps_0$ is such that $\\forall t\\geq 0$, $\\eps_0<\\frac{(\\beta\\lambda)^2}{2\\vert\\alpha'(t)\\vert+4\\lambda}$.\n\t\\end{assumption}\n\n\n\tLemma~\\ref{lem::canonForm} gives a reformulation of \\eqref{eq::QuadeDINAVD} suited to apply Theorem~\\ref{thm::olver}.\n\tTo use the theorem for all $t\\geq 0$, we need to ensure that $\\varphi(t)=\\frac{4r(t)r''(t)-5r'(t)^2}{16r(t)^{5/2}}$ is integrable. To this aim we make the following assumption.\n\t\\begin{assumption}\\label{ass:subexp}\n\t\tThe functions $\\eps$ and $\\alpha$ have first, second and third-order derivatives that are integrable on $[0,+\\infty[$. In addition, $\\lim\\limits_{t\\to\\infty}{\\eps(t)}=0$ and $\\eps'(t)^2/\\eps(t)$ is integrable on $[0,+\\infty[$.\n\t\\end{assumption}\n\n\n\tWe now have an expression for $x$ which is almost explicit: we do not know $\\delta_1$ and $\\delta_2$ in closed form, but they are uniformly bounded (by Lemma~\\ref{lem::phiintegrable}). We will now  compare the asymptotic behavior of \\eqref{eq::formofx} with those of the solutions of \\eqref{eq::quadLM} and \\eqref{eq::quadCN} that we denoted $x_{LM}$ and $x_N$ respectively.\n\tOur main result of Section~\\ref{sec::Quads} is the following, where $\\sim_{+\\infty}$ denotes the asymptotic equivalence\\footnote{Two real-valued functions $g_1$ and $g_2$ are asymptotically equivalent in $+\\infty$ if and only if $\\lim_{t\\to\\infty}\\frac{g_1(t)}{g_2(t)}=1$.} between two functions as $t\\to\\infty$.\n","units":[{"raw_tex":"\tBoth steps are highly challenging, our work focuses on the ODE matter. We study the following second-order dynamical system in a general setting:\n\t\\begin{equation}\\tag{VM-DIN-AVD}\\label{eq::VMDINAVD}\n\t\t\\varepsilon(t) \\ddot{x}(t) + \\alpha(t) \\dot x(t) + \\beta \\Hf(x(t))\\dot{x}(t)+\\gf(x(t))=0,\n\t\t\\quad t\\geq 0,\n\t\\end{equation}\n\twhere $f\\colon \\R^n\\to\\R$ is a smooth function, with gradient $\\gf$ and Hessian $\\Hf$ defined on $\\R^n$ equipped with scalar product $\\langle\\cdot,\\cdot\\rangle$, and induced norm $\\norm{\\cdot}$.\n\tThe system is controlled by two functions $\\eps,\\alpha\\colon\\R_+\\to\\R_+$ (where $\\R_+=[0,+\\infty[$) and a parameter $\\beta> 0$ that define the type of dynamics that drives the trajectory (or solution) $x\\colon\\R_+\\to\\R^n$, whose first- and second-order derivatives are denoted $\\dx$ and $\\ddx$ respectively.\n","start":6442,"end":7273},{"raw_tex":"\tThis paper emphasizes the relation between \\eqref{eq::VMDINAVD} and special cases.\n\tIndeed, taking $\\eps=\\alpha=0$, one obtains\\footnote{\\ref{eq::CN} is usually considered with $\\beta=1$, we put $\\beta$ in the system to ease the discussions below.} the Continuous Newton (CN) method \\citep{gavurin1958nonlinear}\n\t\\begin{equation}\\tag{CN}\\label{eq::CN}\n\t\t\\beta \\Hf(x_N(t))\\dot{x}_N(t)+\\gf(x_N(t))=0, \\quad t\\geq 0,\n\t\\end{equation}\n\tknown notably for being invariant to affine transformations and yielding fast vanishing of the gradient (see Section~\\ref{sec::Control}).\n\tIn fact, this observation shows that \\eqref{eq::VMDINAVD} is a singular perturbation of \\eqref{eq::CN}, which also justifies the terminology ``Newton-like''  in DIN.\n\tWhen $\\alpha\\neq 0$ but $\\eps=0$, we recover the Levenberg--Marquardt (LM) method,\n\t\\begin{equation}\\tag{LM}\\label{eq::LM}\n\t\t\\alpha(t) \\dot x_{LM}(t) + \\beta \\Hf(x_{LM}(t))\\dot{x}_{LM}(t)+\\gf(x_{LM}(t))=0, \\quad t\\geq 0,\n\t\\end{equation}\n\talso known as regularized Newton method since it stabilizes \\eqref{eq::CN}.\n\tIn the rest of the paper, the solutions of \\eqref{eq::CN} and \\eqref{eq::LM} will always be denoted by $x_N$ and $x_{LM}$ respectively. Since the introduction of DIN, it is known (see \\cite{alvarez2002second}) that for $\\alpha=0$, $\\beta=1$, and $\\eps$ constant and small, \\eqref{eq::VMDINAVD} is a ``perturbed'' Newton method since the distance between the solutions of \\eqref{eq::VMDINAVD} and \\eqref{eq::CN}  is at most proportional to $\\sqrt{\\eps}$ at all times.\n","start":8490,"end":10010},{"raw_tex":"\tWe always assume the following on the system \\eqref{eq::VMDINAVD} and the functions $f,\\eps,\\alpha$ defined in the introduction.\n\t\t\\begin{assumption}\\label{ass:general}\n\t\t\t\\textbullet\\ $f: \\R^n \\to \\R$ is convex, twice continuously differentiable, coercive, and strongly convex on bounded subsets of $\\R^n$.\n\t\t\tWe fix $x_0$, $\\dot{x}_0\\in\\R^n$, such that, unless stated otherwise, \\eqref{eq::VMDINAVD} has initial condition $(x(0),\\dx(0))=(x_0,\\dx_0)$, and \\eqref{eq::CN} and \\eqref{eq::LM} have initial conditions $x_{N}(0)=x_{LM}(0)=x_0$.\n\\\\\n\t\t\t\\textbullet\\ $\\alpha: \\R_+ \\to \\R$ is non-increasing, non-negative and differentiable. $\\eps: \\R_+ \\to \\R$ is non-increasing, positive, and twice differentiable with bounded second derivative.\n\t\t\tWe fix initial values: $\\eps(0)=\\eps_0>0$, $\\eps'(0)=\\eps'_0\\leq0$ and $\\alpha(0)=\\alpha_0\\geq 0$.\n\t\t\\end{assumption}\n","start":18693,"end":19555},{"raw_tex":"\t\\section{Approximate Solutions and Asymptotic Analysis on Quadratics}\\label{sec::Quads}\n\tIn addition to Assumption~\\ref{ass:general}, we consider the case where $f$ is a strongly convex quadratic function in order to study the asymptotic behavior of \\eqref{eq::VMDINAVD} w.r.t.\\ \\eqref{eq::CN} and \\eqref{eq::LM}. Quadratic functions are the prototypical example of strongly convex functions. In particular, any strongly convex function can be locally approximated by a quadratic one around its minimizer, making the latter a good model for understanding the local behavior of dynamics.\n\tIn this section, $f$ is quadratic: $f(y)=\\frac{1}{2}\\Vert Ay-b\\Vert_2^2$ for all $y\\in \\R^n$, where $A\\in\\R^{n\\times n}$ is symmetric positive definite and $b\\in\\R^n$. Without loss of generality, we take $b=0$, so that the unique minimum is $x^\\star=0$.\n\t\\subsection{Setting: the Special Case of Quadratic Functions}\n\tQuadratic functions are particularly interesting in our setting since DIN-like ODEs take a simpler form (as observed in \\cite{attouch2020first,shi2021understanding}). Indeed, $\\forall y\\in\\R^n$, $\\gf(y) = A^TAy$ and $\\Hf(y)=A^T A$. Since $\\Hf(y)$ is independent of $y$ we can rewrite \\eqref{eq::VMDINAVD} in an eigenspace\\footnote{This can be generalized to the case where $A^TA$ is only semi-definite by considering orthogonal projections on an eigenspace spanned by the positive eigenvalues of $A^TA$.} of  $A^TA$. That is, we can study the ODE coordinate-wise by looking at one-dimensional problems of the form\n\t\\begin{equation}\\label{eq::QuadeDINAVD}\\tag{Q1-VM-DIN-AVD}\n\t\t\\eps(t)\\ddx(t)+(\\alpha(t)+\\beta\\lambda)\\dx(t)+\\lambda x(t)=0, \\quad t\\geq 0.\n\t\\end{equation}\n\tHere (and throughout what follows) $\\lambda>0$ denotes any eigenvalue of $A^TA$ and $x\\colon\\R_+\\to\\R$ now denotes the corresponding  coordinate (function) of the solution of \\eqref{eq::VMDINAVD} in an eigenspace of $A^TA$. The dynamics \\eqref{eq::QuadeDINAVD} is a \\emph{linear} second-order ODE in $x$ with non-constant coefficients. Similarly, \\eqref{eq::LM} can be rewritten coordinate-wise as\n\t\\begin{equation}\\label{eq::quadLM}\\tag{Q1-LM}\n\t\t(\\alpha(t)+\\beta\\lambda)\\dx_{LM}(t)+\\lambda x_{LM}(t)=0, \\quad t\\geq 0,\n\t\\end{equation}\n\twhere $\\xlm\\colon\\R_+\\to\\R$, and \\eqref{eq::CN} becomes\n\t\\begin{equation}\\label{eq::quadCN}\\tag{Q1-CN}\n\t\t\\beta\\dx_N(t)+ x_N(t)=0, \\quad t\\geq 0,\n\t\\end{equation}\n\twhere again, $x_N\\colon\\R_+\\to\\R$ is one-dimensional.\n\tObserve in particular that \\eqref{eq::CN} and \\eqref{eq::LM} are now first-order \\emph{linear} ODEs, whose solutions have closed forms: $\\forall t\\geq 0$,\n\t\\begin{equation}\\label{eq::closeformLMN}\n\t\tx_N(t) = x_0e^{-\\frac{t}{\\beta}}\n\t\t\\quad \\text{and}\\quad\n\t\tx_{LM}(t) = x_0\\exp\\left(-\\int_0^t \\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}\\diff s \\right).\n\t\\end{equation}\n","start":40821,"end":43615},{"raw_tex":"\tThe advantage of this approach is the possibility to estimate the error made using \\eqref{eq::LG} w.r.t.\\ the true solutions of \\eqref{eq::GenericODE}. This is expressed in the following theorem which gathers results from \\cite{blumenthal1899uber,olver1961error,taylor1982improved}.\n\t\\begin{theorem}[\\cite{olver1997asymptotics}]\\label{thm::olver}\n\t\tLet $r\\colon \\R_{+}\\to \\R$ be a real, positive, twice continuously differentiable function, and define $\\varphi(t) = \\frac{4r(t)r''(t)-5r'(t)^2}{16r(t)^{5/2}}$ for all $t\\geq 0$. Then for any $T>0$, the differential equation,\n\t\t\\begin{equation}\\label{eq::GenericODEthm}\n\t\t\t\\ddot{z}(t)-r(t)z(t)=0,\\quad t\\in [0,T],\n\t\t\\end{equation}\n\t\thas two real and twice continuously differentiable solutions defined $\\forall t\\in[0,T]$ by,\n\t\t\\begin{equation*}\n\t\t\tz_1(t) = \\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t \\sqrt{r(s)}\\diff s\\right) \\ \\ \\text{and} \\ \\,\n\t\t\tz_2(t) = \\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\exp\\left(-\\int_0^t \\sqrt{r(s)}\\diff s\\right),\n\t\t\\end{equation*}\n\t\twhere $\\displaystyle\\vert\\delta_1(t)\\vert\\leq \\exp\\left(\\frac{1}{2} \\int_0^t \\vert \\varphi(s)\\vert\\diff s\\right)-1$ and $\\displaystyle\\vert\\delta_2(t)\\vert\\leq \\exp\\left(-\\frac{1}{2} \\int_t^{T} \\vert \\varphi(s)\\vert\\diff s\\right)-1$.\n\t\tIf in addition $\\displaystyle\\int_0^{+\\infty} \\vert\\varphi(s)\\vert\\diff s <+\\infty$, then the results above also hold for $T=+\\infty$.\n\t\\end{theorem}\n\t\\begin{remark}\\label{rem::olver} We make the following remarks regarding the above result.\\\\\n\t\t--\\ Note that $z_1$ and $z_2$ in Theorem~\\ref{thm::olver} are \\emph{exact} solutions to \\eqref{eq::GenericODEthm}. The LG approximations $\\hat z_1$ and $\\hat z_2$ are obtained by neglecting the unknown functions $\\delta_1$ and $\\delta_2$ in $z_1$ and $z_2$. The theorem gives a \\emph{non-asymptotic} bound for the errors $\\vert z_1(t)-\\hat{z}_1(t)\\vert$ and $\\vert z_2(t)-\\hat{z}_2(t)\\vert$, $t\\geq 0$.\n\t\t\\\\\n\t\t--\\ Since we assumed $r$ to be twice continuously differentiable and positive, $\\varphi$ is continuous, so it is integrable except maybe for $t\\to+\\infty$.\n\t\t\\\\\n\t\t--\\ For the sake of simplicity, the formulation of Theorem~\\ref{thm::olver} slightly differs from that in \\cite{olver1997asymptotics}, the original formulation can be recovered by a change of variable.\n\t\\end{remark}\n","start":46426,"end":48713},{"raw_tex":"\tWe now proceed to make use of the LG method for approximating the solutions of \\eqref{eq::QuadeDINAVD}. The reader only interested in the result can jump directly to the  Section~\\ref{sec::mainasymptoticres}. We first make the following assumption.\n\t\\begin{assumption}\\label{ass:smooth}\n\t\tThe functions $\\alpha$ and $\\eps$ are three times continuously differentiable, and $\\eps_0$ is such that $\\forall t\\geq 0$, $\\eps_0<\\frac{(\\beta\\lambda)^2}{2\\vert\\alpha'(t)\\vert+4\\lambda}$.\n\t\\end{assumption}\n","start":48815,"end":49313},{"raw_tex":"\tLemma~\\ref{lem::canonForm} gives a reformulation of \\eqref{eq::QuadeDINAVD} suited to apply Theorem~\\ref{thm::olver}.\n\tTo use the theorem for all $t\\geq 0$, we need to ensure that $\\varphi(t)=\\frac{4r(t)r''(t)-5r'(t)^2}{16r(t)^{5/2}}$ is integrable. To this aim we make the following assumption.\n\t\\begin{assumption}\\label{ass:subexp}\n\t\tThe functions $\\eps$ and $\\alpha$ have first, second and third-order derivatives that are integrable on $[0,+\\infty[$. In addition, $\\lim\\limits_{t\\to\\infty}{\\eps(t)}=0$ and $\\eps'(t)^2/\\eps(t)$ is integrable on $[0,+\\infty[$.\n\t\\end{assumption}\n","start":52183,"end":52765},{"raw_tex":"\tWe now have an expression for $x$ which is almost explicit: we do not know $\\delta_1$ and $\\delta_2$ in closed form, but they are uniformly bounded (by Lemma~\\ref{lem::phiintegrable}). We will now  compare the asymptotic behavior of \\eqref{eq::formofx} with those of the solutions of \\eqref{eq::quadLM} and \\eqref{eq::quadCN} that we denoted $x_{LM}$ and $x_N$ respectively.\n\tOur main result of Section~\\ref{sec::Quads} is the following, where $\\sim_{+\\infty}$ denotes the asymptotic equivalence\\footnote{Two real-valued functions $g_1$ and $g_2$ are asymptotically equivalent in $+\\infty$ if and only if $\\lim_{t\\to\\infty}\\frac{g_1(t)}{g_2(t)}=1$.} between two functions as $t\\to\\infty$.\n","start":59181,"end":59871}]},"statement":{"raw_tex":"\\begin{theorem}\\label{thm::MainResAsymptotic}\n\t\tLet $x$ be the solution of \\eqref{eq::QuadeDINAVD}, given in \\eqref{eq::formofx}, and $x_{LM}$ and $x_N$ whose closed forms are stated in \\eqref{eq::closeformLMN}. Under Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp}, there exists $C>0$ such that the following asymptotic equivalences hold:\n\t\t\\begin{align}\\label{eq::equivx}\n\t\t\t\\begin{split}\n\t\t\t\tx(t) &\\sim_{+\\infty}\n\t\t\t\tx_{LM}(t)C\\exp\\left(\\int_0^t - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right),\\quad \\text{and}\n\t\t\t\t\\\\x(t)&\\sim_{+\\infty}\n\t\t\t\tx_{N}(t)C\\exp\\left(\\int_0^t \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}- \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right).\n\t\t\t\\end{split}\n\t\t\\end{align}\n\t\tAs a consequence, the convergence of $x$ to $x^\\star$ is:\n\t\t\\begin{enumerate}[label=(\\roman*)]\n\t\t\t\\item \\emph{Faster} than that of $x_{LM}$\n\t\t\tif $\\eps$ is non-integrable and as fast otherwise.\n\t\t\t\\item Slower than that of $x_{N}$ if $\\alpha$ is non-integrable and as fast if $\\alpha$ is integrable, in the case where $\\forall t\\geq 0$, $\\alpha(t)>\\eps(t)$.\n\t\t\t\\item \\emph{Faster} than that of $x_{N}$ if $\\eps$ is non-integrable and as fast if $\\eps$ is integrable, in the case where $\\forall t\\geq 0$, $\\alpha(t)< \\eps(t)$.\n\t\t\\end{enumerate}\n\t\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\\label{thm::MainResAsymptotic}\n\t\tLet $x$ be the solution of \\eqref{eq::QuadeDINAVD}, given in \\eqref{eq::formofx}, and $x_{LM}$ and $x_N$ whose closed forms are stated in \\eqref{eq::closeformLMN}. Under Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp}, there exists $C>0$ such that the following asymptotic equivalences hold:\n\t\t\\begin{align}\\label{eq::equivx}\n\t\t\t\\begin{split}\n\t\t\t\tx(t) &\\sim_{+\\infty}\n\t\t\t\tx_{LM}(t)C\\exp\\left(\\int_0^t - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right),\\quad \\text{and}\n\t\t\t\t\\\\x(t)&\\sim_{+\\infty}\n\t\t\t\tx_{N}(t)C\\exp\\left(\\int_0^t \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}- \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right).\n\t\t\t\\end{split}\n\t\t\\end{align}\n\t\tAs a consequence, the convergence of $x$ to $x^\\star$ is:\n\t\t\\begin{enumerate}[label=(\\roman*)]\n\t\t\t\\item \\emph{Faster} than that of $x_{LM}$\n\t\t\tif $\\eps$ is non-integrable and as fast otherwise.\n\t\t\t\\item Slower than that of $x_{N}$ if $\\alpha$ is non-integrable and as fast if $\\alpha$ is integrable, in the case where $\\forall t\\geq 0$, $\\alpha(t)>\\eps(t)$.\n\t\t\t\\item \\emph{Faster} than that of $x_{N}$ if $\\eps$ is non-integrable and as fast if $\\eps$ is integrable, in the case where $\\forall t\\geq 0$, $\\alpha(t)< \\eps(t)$.\n\t\t\\end{enumerate}\n\t\\end{theorem}","start":59872,"end":61197}]},"proof":{"raw_tex":"\\begin{lemma}\\label{lem::canonForm}\n\t\tSuppose that Assumption~\\ref{ass:smooth} holds, and let $x$ be the solution of \\ref{eq::QuadeDINAVD}. For all $t\\geq 0$, define\n\t\t\\begin{equation}\\label{eq::pr}\n\t\t\tp(t) = \\frac{\\alpha(t)+\\beta\\lambda}{\\eps(t)} \\quad \\text{and}  \\quad r(t) = \\frac{p(t)^2}{4} + \\frac{p'(t)}{2} - \\frac{\\lambda}{\\eps(t)}.\n\t\t\\end{equation}\n\t\tThen, $p$ and $r$ are twice continuously differentiable, $r$ is positive and the function $y$ defined for all $t\\geq 0$ by $y(t) = x(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)$ is a solution to\n\t\t\\begin{equation}\\label{eq::ODErewritten}\n\t\t\t\\ddot y(t) - r(t)y(t) = 0,\\quad t\\geq 0,%\\\\\n\t\t\\end{equation}\n\t\twith initial condition $(y(0),\\dot{y}(0) )= (x_0,\\dx_0 + \\frac{p(0)}{2}x_0)$.\n\t\\end{lemma}\n\n\\begin{proof}\n\t\tWe first check that for all $t\\geq 0$, $r(t)$ is positive. Let $t>0$,\n\t\t\\begin{equation}\\label{eq::rpositive}\n\t\t\tr(t)> 0 \\iff \\frac{(\\alpha(t)+\\beta\\lambda)^2}{4\\eps(t)^2} + \\frac{\\alpha'(t)}{2\\eps(t)} -  \\frac{(\\alpha(t)+\\beta\\lambda)\\eps'(t)}{\\eps(t)^2} - \\frac{\\lambda}{\\eps(t)} >0.\n\t\t\\end{equation}\n\t\tSince $\\eps'(t)\\leq 0$ and $\\alpha'(t)\\leq 0$, one can check that a sufficient condition for \\eqref{eq::rpositive} to hold is,\n\t\t\\begin{equation*}\n\t\t\tr(t)> 0 \\impliedby \\frac{(\\alpha(t)+\\beta\\lambda)^2}{4} > \\left(\\frac{\\vert\\alpha'(t)\\vert}{2}+\\lambda\\right)\\eps(t) \\impliedby \\frac{(\\beta\\lambda)^2}{2\\vert\\alpha'(t)\\vert+4\\lambda}>\\eps_0.\n\t\t\\end{equation*}\n\t\tSo under Assumption~\\ref{ass:smooth}, for all $t\\geq 0$, $r(t)>0$. We then check that $y$ is indeed solution to \\eqref{eq::ODErewritten}. Let $t>0$,\n\t\t\\begin{equation*}\n\t\t\t\\dot{y}(t) = \\frac{p(t)}{2}x(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right) + \\dx(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right),\\quad \\text{and,}\n\t\t\\end{equation*}\n\t\t\\begin{equation*}\n\t\t\t\\ddot{y}(t) = \\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)\\left[\\left(\\frac{p(t)^2}{4}+\\frac{p'(t)}{2}\\right)x(t) + p(t)\\dx(t) + \\ddx(t)\\right].\n\t\t\\end{equation*}\n\t\tSince $x$ solves \\eqref{eq::QuadeDINAVD}, it holds that $\\ddx(t) = -p(t)\\dx(t) - \\frac{\\lambda}{\\eps(t)}x(t)$, so,\n\t\t\\begin{multline*}\n\t\t\t\\ddot{y}(t) = \\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)\\left(\\frac{p(t)^2}{4}+\\frac{p'(t)}{2}- \\frac{\\lambda}{\\eps(t)}\\right)x(t) \\\\= \\left(\\frac{p(t)^2}{4}+\\frac{p'(t)}{2}- \\frac{\\lambda}{\\eps(t)}\\right)y(t)\n\t\t\t= r(t)y(t).\n\t\t\\end{multline*}\n\t\\end{proof}\n\n\\begin{lemma}\\label{lem::phiintegrable}\n\t\tUnder Assumption~\\ref{ass:smooth} and~\\ref{ass:subexp}, $\\int_0^{+\\infty} \\vert \\varphi(s)\\vert\\diff s<+\\infty$.\n\t\\end{lemma}\n\n\\begin{theorem}\\label{thm::MainResLG}\n\t\tSuppose that Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp} hold. There exists $A,B\\in\\R$ such that $x(0)=x_0$, $\\dx(0)=\\dx_0$ and for all $t\\geq 0$, the solution of \\eqref{eq::QuadeDINAVD} is\n\t\t\\begin{align}\\label{eq::formofx}\n\t\t\t\\begin{split}\n\t\t\t\tx(t) =&\n\t\t\t\tA\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\alpha(t)+\\beta\\lambda}}{\\sqrt{\\eps(t)}}\n\t\t\t\t\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\n\t\t\t\t\\\\\n\t\t\t\t+B&\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\n\t\t\t\t\\frac{\\sqrt{\\eps(t)}}{\\sqrt{\\alpha(t)+\\beta\\lambda}}\n\t\t\t\t\\\\\n\t\t\t\t&\\exp\\left(\\int_0^t -\\frac{\\alpha(s)+\\beta\\lambda}{\\eps(s)}+\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}+\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right),\n\t\t\t\\end{split}\n\t\t\\end{align}\n\t\twhere for all  $t\\geq 0$,\n\t\t\\begin{equation}\\label{eq::deltabounds}\n\t\t\t\\vert\\delta_1(t)\\vert \\leq \\exp\\left(\\frac{1}{2} \\int_0^t \\vert \\varphi(s)\\vert\\diff s\\right)-1\n\t\t\t\\quad \\text{and}\\quad\n\t\t\t\\vert\\delta_2(t)\\vert \\leq \\exp\\left(-\\frac{1}{2} \\int_t^{+\\infty} \\vert \\varphi(s)\\vert\\diff s\\right)-1.\n\t\t\\end{equation}\n\t\\end{theorem}\n\n\\begin{proof}[Proof of Theorem~\\ref{thm::MainResLG}]\n\t\tLet $x$ be the solution of \\eqref{eq::QuadeDINAVD} define $p,r$ as in \\eqref{eq::pr}. Let us also define $y(t)\\stackrel{\\textrm{def}}{=}x(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)$. According to Lemma~\\ref{lem::canonForm}, $r$ is positive and $y$ is solution to \\eqref{eq::ODErewritten}. Then, from Lemma~\\ref{lem::phiintegrable}, $\\int_t^{T} \\vert \\varphi(s)\\vert\\diff s<+\\infty$, so we can apply Theorem~\\ref{thm::olver} to $y$ on $[0,+\\infty[$. Therefore, there exists $A,B\\in\\R$, such that $\\forall t\\geq 0$,\n\t\t$$\n\t\ty(t)= A\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t \\sqrt{r(s)}\\diff s\\right) + B\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t -\\sqrt{r(s)}\\diff s\\right),\n\t\t$$\n\t\twhere $A$ and $B$ are determined by the initial conditions, and $\\delta_1$, $\\delta_2$ are such that \\eqref{eq::deltabounds} holds.\n\t\tGoing back to $x(t) = y(t)\\exp\\left(\\int_0^t -\\frac{p(s)}{2}\\diff s\\right)$, we obtain \tthat for all $t\\geq 0$,\n\t\t\\begin{multline}\\label{eq::formofx0}\n\t\t\tx(t) = A\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t -\\frac{p(s)}{2} +\\sqrt{r(s)}\\diff s\\right) \\\\ + B\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t -\\frac{p(s)}{2} -\\sqrt{r(s)}\\diff s\\right).\n\t\t\\end{multline}\n\t\tIt now remains to expand the terms in the two exponentials in \\eqref{eq::formofx0} in order to obtain \\eqref{eq::formofx}. To this aim, we approximate $\\sqrt{r(s)}$, let $s\\geq 0$,\n\t\t\\begin{align}\\label{eq::sqrtr}\n\t\t\t\\begin{split}\n\t\t\t\t\\sqrt{r(s)}&=\\frac{p(s)}{2}\\sqrt{1+ \\frac{2p'(s)}{p(s)^2} - \\frac{4\\lambda}{\\eps(s)p(s)^2}}\n\t\t\t\t\\\\&= \\frac{p(s)}{2}\\left(1 + \\frac{p'(s)}{p(s)^2} - \\frac{2\\lambda}{\\eps(s)p(s)^2} - \\frac{1}{8}\\left(\\frac{2p'(s)}{p(s)^2} - \\frac{4\\lambda}{\\eps(s)p(s)^2}\\right)^2 + o(\\eps(s)^2) \\right)\n\t\t\t\t\\\\ & = \\frac{p(s)}{2} +\\frac{ p'(s)}{2p(s)} - \\frac{\\lambda}{\\eps(s)p(s)} - \\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda}{\\eps(s)p(s)^{3/2}}\\right)^2 + o(\\eps(s))\n\t\t\t\t\\\\  = \\frac{p(s)}{2} &+ \\frac{p'(s)\\eps(s)}{2(\\alpha(s)+\\beta\\lambda)} - \\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2 + o(\\eps(s))\n\t\t\t\t\\\\  = \\frac{p(s)}{2} &+ \\frac{\\alpha'(s)/2-\\lambda}{\\alpha(s)+\\beta\\lambda} -\\frac{\\eps'(s)}{2\\eps(s)}  - \\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2 + o(\\eps(s))\n\t\t\t\\end{split}\n\t\t\\end{align}\n\t\tTo ease the readability, we denote $h(t)=\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2$. Focusing on the first exponential term in \\eqref{eq::formofx0}, we deduce from \\eqref{eq::sqrtr} that for all $t\\geq 0$,\n\t\t\\begin{align*}%\\label{eq::Aexpterm}\n\t\t\t\\begin{split}\n\t\t\t\t&\\exp\\left(\\int_0^t -\\frac{p(s)}{2} +\\sqrt{r(s)}\\diff s\\right)\n\t\t\t\t\\\\&= \\exp\\left(\\int_0^t \\frac{\\alpha'(s)/2}{\\alpha(s)+\\beta\\lambda} -\\frac{\\eps'(s)}{2\\eps(s)} -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{1}{16}h(t) + o(\\eps(s))\\diff s\\right)\n\t\t\t\t\\\\&=  \\frac{\\sqrt{\\alpha(t)+\\beta\\lambda})}{\\sqrt{\\alpha_0+\\beta\\lambda}} \\frac{\\sqrt{\\eps_0}}{\\sqrt{\\eps(t)}} \\exp\\left(\n\t\t\t\t\\int_0^t \\frac{-\\lambda}{\\alpha(s)+\\beta\\lambda} -\n\t\t\t\t\\frac{1}{16}h(t) + o(\\eps(s))\\diff s\\right)\n\t\t\t\t\\\\&=  \\frac{\\sqrt{\\alpha(t)+\\beta\\lambda})}{\\sqrt{\\alpha_0+\\beta\\lambda}} \\frac{\\sqrt{\\eps_0}}{\\sqrt{\\eps(t)}} \\exp\\left(\n\t\t\t\t\\int_0^t \\frac{-\\lambda}{\\alpha(s)+\\beta\\lambda} -\n\t\t\t\t\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}} + o(\\eps(s))\\diff s\\right),\n\t\t\t\\end{split}\n\t\t\\end{align*}\n\t\twhere the last line relies on further computations postponed to Lemma~\\ref{lem::remainderterm} in Appendix~\\ref{app::integrability}. Performing the exact same type of computations on $\\exp\\left(\\int_0^t -\\frac{p(s)}{2} -\\sqrt{r(s)}\\diff s\\right)$, and up to redefining $A$ and $B$ so as to encompass all the constants, we obtain \\eqref{eq::formofx} and the result is proved.\n\t\\end{proof}\n\n\\begin{proof}[Proof of Theorem~\\ref{thm::MainResAsymptotic}]\n\t\tThanks to Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp}, Theorem~\\ref{thm::MainResLG} tells us that $x$ has the form \\eqref{eq::formofx}. We now analyze the two terms in \\eqref{eq::formofx}.\n\n\t\tFirst, we know from Theorem~\\ref{thm::MainResLG} that $\\delta_1(0)=0$ and $\\lim_{t\\to+\\infty}\\delta_2(t)=0$. In addition, by Lemma~\\ref{lem::phiintegrable}, $\\delta_1$ and $\\delta_2$ are uniformly bounded by some positive constant.\n\t\tThen $r(t)^{-1/4}$ decays asymptotically like $\\sqrt{\\eps(t)}$ and $\\alpha$ is bounded. So\n\t\t$A\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\alpha(t)+\\beta\\lambda}}{\\sqrt{\\eps(t)}}$ is asymptotically equivalent to some constant $c_1\\in\\R$ as $t\\to +\\infty$. Similarly, the factor $B\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\eps(t)}}{\\sqrt{\\alpha(t)+\\beta\\lambda}}$ is equivalent to $c_2\\eps(t)$, with $c_2\\in\\R$.\n\n\t\tWe now analyze the ``exponential factors'' in \\eqref{eq::formofx}.\n\t\tOn the one hand, $\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}+\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))$ converges to $\\frac{1}{\\beta}$ as $s\\to \\infty$, while $\\frac{\\alpha(s)+\\beta\\lambda}{\\eps(s)}$ diverges to $+\\infty$. Therefore, we deduce that,\n\t\t\\begin{multline*}\n\t\t\t\\exp\\left(\\int_0^t -\\frac{\\alpha(s)+\\beta\\lambda}{\\eps(s)}+\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}+\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\n\t\t\t\\\\= o\\left(\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\\right).\n\t\t\\end{multline*}\n\t\tAs a consequence, the second term in \\eqref{eq::formofx} will decrease to $0$ faster than the first-one (let alone the additional $\\eps(t)$ decay that we have just discussed). The asymptotic behavior of $x$ will thus be governed by the first term in \\eqref{eq::formofx}.\n\n\t\tLet us now focus on the first term in \\eqref{eq::formofx}. Observe that\n\t\t$\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}\\diff s\\right)$ is exactly the decay of $x_{LM}$ in \\eqref{eq::closeformLMN}. Thus, we have proved that there exists $C>0$, such that the following asymptotic equivalence holds,\n\t\t\\begin{multline*}%\\label{eq::equivA}\n\t\t\tA\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\alpha(t)+\\beta\\lambda}}{\\sqrt{\\eps(t)}}\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\n\t\t\t\\\\ \\sim_{+\\infty}\n\t\t\tx_{LM}(t)C\\exp\\left(\\int_0^t - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right),\n\t\t\\end{multline*}\n\t\twhich proves the first part of \\eqref{eq::equivx}. The second equivalence in \\eqref{eq::equivx} is obtained using the following identity,\n\t\t\\begin{equation}\\label{eq::rewritteLM}\n\t\t\t\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}\\diff s = \\int_0^t -\\frac{1}{\\beta} + \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}\\diff s = -\\frac{t}{\\beta} + \\int_0^t  \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}\\diff s\n\t\t\\end{equation}\n\t\tand $e^{-t/\\beta}$ is precisely the rate at which $x_N$ decreases. So \\eqref{eq::equivx} holds.\n\n\t\tIt finally remains to deduce the conclusions of the theorem from \\eqref{eq::equivx}.\n\t\t\\\\--\\ Regarding the comparison with $x_{LM}$, the integral $\\int_0^t - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s$ converges if and only if $\\eps$ is integrable on $[0,+\\infty[$, and diverges to $-\\infty$ when $\\eps$ is not. So $x$ converges to $0$ at least as fast as $x_{LM}$ and faster when $\\eps$ is not integrable.\n\t\t\\\\-- As for the comparison with $x_N$, if $\\alpha(s)>\\eps(s)\\geq 0$ for all $s\\geq 0$, then the integral $\\int_0^t \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}- \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s$ is convergent in $+\\infty$ if and only if $\\alpha$ is integrable and diverges to $+\\infty$ when $\\alpha$ is non-integrable. So when $\\alpha$ is integrable, the speed of convergence of $x$ is the same as that of $x_N$. When $\\alpha$ is not integrable, the convergence to $0$ is slower but still holds. Indeed, for all $s\\geq 0$\n\t\t$\\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\frac{1}{\\beta})}< \\frac{\\alpha(s)}{\\beta\\alpha(s)}= \\frac{1}{\\beta}$. Thus for all $t>0$, $-\\frac{t}{\\beta} + \\int_0^t \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}\\diff s<0$.\n\t\t\\\\-- Finally, the comparison with $x_N$ in the case $\\eps(s)> \\alpha(s)$ is exactly the same as the comparison with $x_{LM}$ using \\eqref{eq::rewritteLM}.\n\t\\end{proof}\n\n\\begin{proof}\n\t\tWe suppose that Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp} hold. As stated in Remark~\\ref{rem::olver}, since $\\varphi$ is continuous, we only need to check its integrability when $t$ tends to $+\\infty$. Let $t>0$, we first establish some useful identities, we omit the dependence on $t$ for the sake of readability.\n\t\t\\begin{align*}%\\label{eq::pprime}\n\t\t\tp' = \\frac{\\alpha'\\eps-(\\alpha+\\beta\\lambda)\\eps'}{\\eps^2},\n\t\t\t\\quad \\text{and}\\quad\n\t\t\tp'' = \\frac{\\alpha''\\eps^2-2\\alpha'\\eps'\\eps-(\\alpha+\\beta\\lambda)\\eps''\\eps + 2(\\alpha+\\beta\\lambda)(\\eps')^2}{\\eps^3}.\n\t\t\\end{align*}\n\t\tThen,\n\t\t\\begin{align}\\label{eq::factorR}\n\t\t\t\\begin{split}\n\t\t\t\tr  &= \\frac{p^2}{4}\\left(1+ \\frac{2p'}{p^2} - \\frac{4\\lambda}{\\eps p^2}\\right) = \\frac{(\\alpha+\\beta\\lambda)^2}{4\\eps^2}\\left(1+\\frac{2p' \\eps ^2}{(\\alpha +\\beta\\lambda)^2} - \\frac{4\\lambda\\eps }{(\\alpha +\\beta\\lambda)^2}\\right)\n\t\t\t\t\\\\\n\t\t\t\t&=\\frac{(\\alpha +\\beta\\lambda)^2}{4\\eps ^2}\\left(1+\\frac{2\\alpha' \\eps }{(\\alpha +\\beta\\lambda)^2} - \\frac{2\\eps' }{(\\alpha +\\beta\\lambda)} - \\frac{4\\lambda\\eps }{(\\alpha +\\beta\\lambda)^2}\\right).\n\t\t\t\\end{split}\n\t\t\\end{align}\n\t\tAn important consequence of Assumption~\\ref{ass:subexp} is that $\\vert\\eps'(t)\\vert = o(\\eps(t))$, $\\vert\\eps''(t)\\vert = o(\\eps'(t))$ (and the same holds for $\\alpha$ w.r.t.\\ to its derivatives). Therefore, we deduce from \\eqref{eq::factorR} that\n\t\t\\begin{equation*}\n\t\t\tr(t)\\sim_{+\\infty}  \\frac{(\\alpha(t)+\\beta\\lambda)^2}{4\\eps(t)^2},\n\t\t\\end{equation*}\n\t\tand we note that $1/r$ decays at the same speed as $\\eps^2$, which will be useful later. In order to study $\\varphi$, we now differentiate $r$,\n\t\t\\begin{align}\\label{eq::rprime}\n\t\t\t\\begin{split}\n\t\t\t\tr' &= \\frac{p'p}{2}\\left(1+ \\frac{2p'}{p^2} - \\frac{4\\lambda}{\\eps p^2}\\right) + \\frac{1}{4}\\left( 2p'' - \\frac{4(p')^2}{p} + \\frac{8\\lambda p'}{\\eps p}+ \\frac{4\\lambda\\eps'}{\\eps^2} \\right)\n\t\t\t\t\\\\&= \\frac{2p'}{p}r + \\frac{1}{4}\\left( 2p'' - \\frac{4(p')^2}{p} +\\frac{8\\lambda p'}{\\eps p} + \\frac{4\\lambda\\eps'}{\\eps^2}  \\right),\\quad \\text{and,}\n\t\t\t\\end{split}\n\t\t\\end{align}\n\n\t\t\\begin{multline}\\label{eq::rsec}\n\t\t\tr'' = 2\\frac{p''p-(p')^2}{p^2}r  + \\frac{2p'}{p}r'\n\t\t\t\\\\\n\t\t\t+ \\frac{1}{4}\\left( 2p''' + 4\\frac{(p')^3-2p''p'p}{p^2}+8 \\lambda\\frac{p''p\\eps - (p')^2\\eps - p'p\\eps'}{\\eps^2 p^2} + \\frac{4\\lambda\\eps''}{\\eps^2} - \\frac{8\\lambda(\\eps')^2}{\\eps^3}  \\right).\n\t\t\\end{multline}\n\t\tThen, to justify that $\\varphi$ is integrable, we prove that $\\frac{r''}{r^{3/2}}$ and $\\frac{(r')^2}{r^{5/2}}$ are integrable. Since we know that $1/r$ decays at the same speed as $\\eps^2$, we can equivalently show that $\\eps^3r''$ and $\\eps^5(r')^2$ are integrable.\n\t\tTo this aim we fully expand all the terms in \\eqref{eq::rprime} and \\eqref{eq::rsec}.\n\t\tWe first deal with \\eqref{eq::rprime}, it holds that:\n\t\t\\begin{align*}\n\t\t\tr'^2&\\eps^{5} =\n\t\t\t\\left[- \\frac{ (\\alpha+\\beta\\lambda)^{2} \\left(- \\frac{4 \\lambda \\eps }{ (\\alpha+\\beta\\lambda)^{2}} + 1 + \\frac{\\left(- {2  (\\alpha+\\beta\\lambda) \\eps' } + {2 \\alpha'\\eps }\\right) }{ (\\alpha+\\beta\\lambda)^{2}}\\right) \\eps' }{2 \\sqrt{\\eps}}\\right.\n\t\t\t\\\\ &\\left.\n\t\t\t+ \\frac{ (\\alpha+\\beta\\lambda)^{2}\\sqrt{\\eps}}{4} \\left(- \\frac{4 \\lambda \\eps' }{ (\\alpha+\\beta\\lambda)^{2}} + \\frac{8 \\lambda \\alpha'  \\eps }{ (\\alpha+\\beta\\lambda)^{3}} + \\frac{2 \\left(- \\frac{2  (\\alpha+\\beta\\lambda) \\eps' }{\\eps } + {2 \\alpha'\\eps }\\right) \\eps'   }{ (\\alpha+\\beta\\lambda)^{2}}\n\t\t\t\\right.\\right.\n\t\t\t\\\\&\\left.\\left.\n\t\t\t+ \\frac{\\left(\\frac{4  (\\alpha+\\beta\\lambda) \\eps'^{2} }{\\eps } - {2  (\\alpha+\\beta\\lambda) \\eps'' } - {4 \\alpha'  \\eps' } + {2 \\alpha''\\eps }\\right) }{ (\\alpha+\\beta\\lambda)^{2}} - \\frac{2 \\left(-{2  (\\alpha+\\beta\\lambda) \\eps' } + {2 \\alpha'\\eps }\\right) \\alpha'   }{ (\\alpha+\\beta\\lambda)^{3}}\\right)\n\t\t\t\\right.\\\\\n\t\t\t&\\left. + \\frac{ (\\alpha+\\beta\\lambda)}{2} { \\left(- \\frac{4 \\lambda \\eps }{ (\\alpha+\\beta\\lambda)^{2}} + 1 + \\frac{\\left(- {2  (\\alpha+\\beta\\lambda) \\eps' } + {2 \\alpha'\\eps }\\right) }{ (\\alpha+\\beta\\lambda)^{2}}\\right)} \\alpha' \\sqrt{\\eps}\n\t\t\t\\phantom{\\frac{\\frac{\\beta}{\\beta}}{\\beta}}\n\t\t\t\\hspace{-1.7cm}\\phantom{\\frac{\\left( \\frac{\\frac{a'}{a'}}{a'}\\right)}{a^2}{\\sqrt{.}}\n\t\t\t}\\right]^{2}.\n\t\t\\end{align*}\n\t\tThe computations for $\\eps^2r''$ are omitted since they are longer but very similar.\n\n\t\tWe analyze the integrability of each of the terms above (and those of $\\eps^2r''$). By Assumption~\\ref{ass:subexp}, $\\eps'$, $\\eps''$ and $\\eps'''$ are integrable, as well as $\\alpha'$, $\\alpha''$ and $\\alpha'''$, this justifies the integrability of most of the terms. We also need $\\frac{(\\eps')^2}{\\eps}$ and $\\frac{(\\eps')^3}{\\eps}$ to be integrable, which holds by Assumption~\\ref{ass:subexp}. Overall, $\\varphi$ is integrable on $\\R_{+}$.\n\t\\end{proof}\n\n\\begin{lemma}\\label{lem::remainderterm}\n\t\tUnder Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp}, for all $s\\geq 0$, $$\\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2 = \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s)).$$\n\t\\end{lemma}\n\n\\begin{proof}\n\t\tWe omit the time dependence on $s\\geq 0$ for the sake of readability. Using Assumption~\\ref{ass:smooth} we can define and expand the following quantity,\n\t\t\\begin{align*}\n\t\t\t&\\frac{1}{16}\\left(\\frac{2p'}{p^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps}}{(\\alpha+\\beta\\lambda)^{3/2}}\\right)^2\n\t\t\t=\\frac{\\lambda^2\\eps}{(\\alpha+\\beta\\lambda)^{3}}  - \\frac{p'\\lambda\\sqrt{\\eps}}{p^{3/2}(\\alpha+\\beta\\lambda)^{3/2}} + \\frac{(p')^2}{4p^{3}}\n\t\t\t\\\\&= \\frac{\\lambda^2\\eps}{(\\alpha+\\beta\\lambda)^{3}}\n\t\t\t\t- \\frac{\\lambda(\\alpha'\\eps - (\\alpha+\\beta\\lambda)\\eps')}{(\\alpha+\\beta\\lambda)^{3}}\n\t\t\t\t+ \\frac{(\\alpha')^2\\eps + \\frac{(\\eps')^2}{\\eps}(\\alpha+\\beta\\lambda)^2 - 2 \\alpha'\\eps'(\\alpha+\\beta\\lambda)}{4(\\alpha+\\beta\\lambda)^{3}}.\n\t\t\\end{align*}\n\t\tAssumption~\\ref{ass:subexp}, implies in particular that $\\vert\\eps'(t)\\vert=o(\\eps(t))$ and that $\\alpha'(t)\\to 0$, which we use in the equality above to obtain the desired conclusion.\n\t\\end{proof}","units":[{"raw_tex":"\\begin{lemma}\\label{lem::canonForm}\n\t\tSuppose that Assumption~\\ref{ass:smooth} holds, and let $x$ be the solution of \\ref{eq::QuadeDINAVD}. For all $t\\geq 0$, define\n\t\t\\begin{equation}\\label{eq::pr}\n\t\t\tp(t) = \\frac{\\alpha(t)+\\beta\\lambda}{\\eps(t)} \\quad \\text{and}  \\quad r(t) = \\frac{p(t)^2}{4} + \\frac{p'(t)}{2} - \\frac{\\lambda}{\\eps(t)}.\n\t\t\\end{equation}\n\t\tThen, $p$ and $r$ are twice continuously differentiable, $r$ is positive and the function $y$ defined for all $t\\geq 0$ by $y(t) = x(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)$ is a solution to\n\t\t\\begin{equation}\\label{eq::ODErewritten}\n\t\t\t\\ddot y(t) - r(t)y(t) = 0,\\quad t\\geq 0,%\\\\\n\t\t\\end{equation}\n\t\twith initial condition $(y(0),\\dot{y}(0) )= (x_0,\\dx_0 + \\frac{p(0)}{2}x_0)$.\n\t\\end{lemma}","start":49805,"end":50565},{"raw_tex":"\\begin{proof}\n\t\tWe first check that for all $t\\geq 0$, $r(t)$ is positive. Let $t>0$,\n\t\t\\begin{equation}\\label{eq::rpositive}\n\t\t\tr(t)> 0 \\iff \\frac{(\\alpha(t)+\\beta\\lambda)^2}{4\\eps(t)^2} + \\frac{\\alpha'(t)}{2\\eps(t)} -  \\frac{(\\alpha(t)+\\beta\\lambda)\\eps'(t)}{\\eps(t)^2} - \\frac{\\lambda}{\\eps(t)} >0.\n\t\t\\end{equation}\n\t\tSince $\\eps'(t)\\leq 0$ and $\\alpha'(t)\\leq 0$, one can check that a sufficient condition for \\eqref{eq::rpositive} to hold is,\n\t\t\\begin{equation*}\n\t\t\tr(t)> 0 \\impliedby \\frac{(\\alpha(t)+\\beta\\lambda)^2}{4} > \\left(\\frac{\\vert\\alpha'(t)\\vert}{2}+\\lambda\\right)\\eps(t) \\impliedby \\frac{(\\beta\\lambda)^2}{2\\vert\\alpha'(t)\\vert+4\\lambda}>\\eps_0.\n\t\t\\end{equation*}\n\t\tSo under Assumption~\\ref{ass:smooth}, for all $t\\geq 0$, $r(t)>0$. We then check that $y$ is indeed solution to \\eqref{eq::ODErewritten}. Let $t>0$,\n\t\t\\begin{equation*}\n\t\t\t\\dot{y}(t) = \\frac{p(t)}{2}x(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right) + \\dx(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right),\\quad \\text{and,}\n\t\t\\end{equation*}\n\t\t\\begin{equation*}\n\t\t\t\\ddot{y}(t) = \\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)\\left[\\left(\\frac{p(t)^2}{4}+\\frac{p'(t)}{2}\\right)x(t) + p(t)\\dx(t) + \\ddx(t)\\right].\n\t\t\\end{equation*}\n\t\tSince $x$ solves \\eqref{eq::QuadeDINAVD}, it holds that $\\ddx(t) = -p(t)\\dx(t) - \\frac{\\lambda}{\\eps(t)}x(t)$, so,\n\t\t\\begin{multline*}\n\t\t\t\\ddot{y}(t) = \\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)\\left(\\frac{p(t)^2}{4}+\\frac{p'(t)}{2}- \\frac{\\lambda}{\\eps(t)}\\right)x(t) \\\\= \\left(\\frac{p(t)^2}{4}+\\frac{p'(t)}{2}- \\frac{\\lambda}{\\eps(t)}\\right)y(t)\n\t\t\t= r(t)y(t).\n\t\t\\end{multline*}\n\t\\end{proof}","start":50567,"end":52182},{"raw_tex":"\\begin{lemma}\\label{lem::phiintegrable}\n\t\tUnder Assumption~\\ref{ass:smooth} and~\\ref{ass:subexp}, $\\int_0^{+\\infty} \\vert \\varphi(s)\\vert\\diff s<+\\infty$.\n\t\\end{lemma}","start":53173,"end":53340},{"raw_tex":"\\begin{theorem}\\label{thm::MainResLG}\n\t\tSuppose that Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp} hold. There exists $A,B\\in\\R$ such that $x(0)=x_0$, $\\dx(0)=\\dx_0$ and for all $t\\geq 0$, the solution of \\eqref{eq::QuadeDINAVD} is\n\t\t\\begin{align}\\label{eq::formofx}\n\t\t\t\\begin{split}\n\t\t\t\tx(t) =&\n\t\t\t\tA\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\alpha(t)+\\beta\\lambda}}{\\sqrt{\\eps(t)}}\n\t\t\t\t\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\n\t\t\t\t\\\\\n\t\t\t\t+B&\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\n\t\t\t\t\\frac{\\sqrt{\\eps(t)}}{\\sqrt{\\alpha(t)+\\beta\\lambda}}\n\t\t\t\t\\\\\n\t\t\t\t&\\exp\\left(\\int_0^t -\\frac{\\alpha(s)+\\beta\\lambda}{\\eps(s)}+\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}+\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right),\n\t\t\t\\end{split}\n\t\t\\end{align}\n\t\twhere for all  $t\\geq 0$,\n\t\t\\begin{equation}\\label{eq::deltabounds}\n\t\t\t\\vert\\delta_1(t)\\vert \\leq \\exp\\left(\\frac{1}{2} \\int_0^t \\vert \\varphi(s)\\vert\\diff s\\right)-1\n\t\t\t\\quad \\text{and}\\quad\n\t\t\t\\vert\\delta_2(t)\\vert \\leq \\exp\\left(-\\frac{1}{2} \\int_t^{+\\infty} \\vert \\varphi(s)\\vert\\diff s\\right)-1.\n\t\t\\end{equation}\n\t\\end{theorem}","start":53595,"end":54786},{"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm::MainResLG}]\n\t\tLet $x$ be the solution of \\eqref{eq::QuadeDINAVD} define $p,r$ as in \\eqref{eq::pr}. Let us also define $y(t)\\stackrel{\\textrm{def}}{=}x(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)$. According to Lemma~\\ref{lem::canonForm}, $r$ is positive and $y$ is solution to \\eqref{eq::ODErewritten}. Then, from Lemma~\\ref{lem::phiintegrable}, $\\int_t^{T} \\vert \\varphi(s)\\vert\\diff s<+\\infty$, so we can apply Theorem~\\ref{thm::olver} to $y$ on $[0,+\\infty[$. Therefore, there exists $A,B\\in\\R$, such that $\\forall t\\geq 0$,\n\t\t$$\n\t\ty(t)= A\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t \\sqrt{r(s)}\\diff s\\right) + B\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t -\\sqrt{r(s)}\\diff s\\right),\n\t\t$$\n\t\twhere $A$ and $B$ are determined by the initial conditions, and $\\delta_1$, $\\delta_2$ are such that \\eqref{eq::deltabounds} holds.\n\t\tGoing back to $x(t) = y(t)\\exp\\left(\\int_0^t -\\frac{p(s)}{2}\\diff s\\right)$, we obtain \tthat for all $t\\geq 0$,\n\t\t\\begin{multline}\\label{eq::formofx0}\n\t\t\tx(t) = A\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t -\\frac{p(s)}{2} +\\sqrt{r(s)}\\diff s\\right) \\\\ + B\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t -\\frac{p(s)}{2} -\\sqrt{r(s)}\\diff s\\right).\n\t\t\\end{multline}\n\t\tIt now remains to expand the terms in the two exponentials in \\eqref{eq::formofx0} in order to obtain \\eqref{eq::formofx}. To this aim, we approximate $\\sqrt{r(s)}$, let $s\\geq 0$,\n\t\t\\begin{align}\\label{eq::sqrtr}\n\t\t\t\\begin{split}\n\t\t\t\t\\sqrt{r(s)}&=\\frac{p(s)}{2}\\sqrt{1+ \\frac{2p'(s)}{p(s)^2} - \\frac{4\\lambda}{\\eps(s)p(s)^2}}\n\t\t\t\t\\\\&= \\frac{p(s)}{2}\\left(1 + \\frac{p'(s)}{p(s)^2} - \\frac{2\\lambda}{\\eps(s)p(s)^2} - \\frac{1}{8}\\left(\\frac{2p'(s)}{p(s)^2} - \\frac{4\\lambda}{\\eps(s)p(s)^2}\\right)^2 + o(\\eps(s)^2) \\right)\n\t\t\t\t\\\\ & = \\frac{p(s)}{2} +\\frac{ p'(s)}{2p(s)} - \\frac{\\lambda}{\\eps(s)p(s)} - \\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda}{\\eps(s)p(s)^{3/2}}\\right)^2 + o(\\eps(s))\n\t\t\t\t\\\\  = \\frac{p(s)}{2} &+ \\frac{p'(s)\\eps(s)}{2(\\alpha(s)+\\beta\\lambda)} - \\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2 + o(\\eps(s))\n\t\t\t\t\\\\  = \\frac{p(s)}{2} &+ \\frac{\\alpha'(s)/2-\\lambda}{\\alpha(s)+\\beta\\lambda} -\\frac{\\eps'(s)}{2\\eps(s)}  - \\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2 + o(\\eps(s))\n\t\t\t\\end{split}\n\t\t\\end{align}\n\t\tTo ease the readability, we denote $h(t)=\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2$. Focusing on the first exponential term in \\eqref{eq::formofx0}, we deduce from \\eqref{eq::sqrtr} that for all $t\\geq 0$,\n\t\t\\begin{align*}%\\label{eq::Aexpterm}\n\t\t\t\\begin{split}\n\t\t\t\t&\\exp\\left(\\int_0^t -\\frac{p(s)}{2} +\\sqrt{r(s)}\\diff s\\right)\n\t\t\t\t\\\\&= \\exp\\left(\\int_0^t \\frac{\\alpha'(s)/2}{\\alpha(s)+\\beta\\lambda} -\\frac{\\eps'(s)}{2\\eps(s)} -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{1}{16}h(t) + o(\\eps(s))\\diff s\\right)\n\t\t\t\t\\\\&=  \\frac{\\sqrt{\\alpha(t)+\\beta\\lambda})}{\\sqrt{\\alpha_0+\\beta\\lambda}} \\frac{\\sqrt{\\eps_0}}{\\sqrt{\\eps(t)}} \\exp\\left(\n\t\t\t\t\\int_0^t \\frac{-\\lambda}{\\alpha(s)+\\beta\\lambda} -\n\t\t\t\t\\frac{1}{16}h(t) + o(\\eps(s))\\diff s\\right)\n\t\t\t\t\\\\&=  \\frac{\\sqrt{\\alpha(t)+\\beta\\lambda})}{\\sqrt{\\alpha_0+\\beta\\lambda}} \\frac{\\sqrt{\\eps_0}}{\\sqrt{\\eps(t)}} \\exp\\left(\n\t\t\t\t\\int_0^t \\frac{-\\lambda}{\\alpha(s)+\\beta\\lambda} -\n\t\t\t\t\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}} + o(\\eps(s))\\diff s\\right),\n\t\t\t\\end{split}\n\t\t\\end{align*}\n\t\twhere the last line relies on further computations postponed to Lemma~\\ref{lem::remainderterm} in Appendix~\\ref{app::integrability}. Performing the exact same type of computations on $\\exp\\left(\\int_0^t -\\frac{p(s)}{2} -\\sqrt{r(s)}\\diff s\\right)$, and up to redefining $A$ and $B$ so as to encompass all the constants, we obtain \\eqref{eq::formofx} and the result is proved.\n\t\\end{proof}","start":55049,"end":59028},{"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm::MainResAsymptotic}]\n\t\tThanks to Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp}, Theorem~\\ref{thm::MainResLG} tells us that $x$ has the form \\eqref{eq::formofx}. We now analyze the two terms in \\eqref{eq::formofx}.\n\n\t\tFirst, we know from Theorem~\\ref{thm::MainResLG} that $\\delta_1(0)=0$ and $\\lim_{t\\to+\\infty}\\delta_2(t)=0$. In addition, by Lemma~\\ref{lem::phiintegrable}, $\\delta_1$ and $\\delta_2$ are uniformly bounded by some positive constant.\n\t\tThen $r(t)^{-1/4}$ decays asymptotically like $\\sqrt{\\eps(t)}$ and $\\alpha$ is bounded. So\n\t\t$A\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\alpha(t)+\\beta\\lambda}}{\\sqrt{\\eps(t)}}$ is asymptotically equivalent to some constant $c_1\\in\\R$ as $t\\to +\\infty$. Similarly, the factor $B\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\eps(t)}}{\\sqrt{\\alpha(t)+\\beta\\lambda}}$ is equivalent to $c_2\\eps(t)$, with $c_2\\in\\R$.\n\n\t\tWe now analyze the ``exponential factors'' in \\eqref{eq::formofx}.\n\t\tOn the one hand, $\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}+\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))$ converges to $\\frac{1}{\\beta}$ as $s\\to \\infty$, while $\\frac{\\alpha(s)+\\beta\\lambda}{\\eps(s)}$ diverges to $+\\infty$. Therefore, we deduce that,\n\t\t\\begin{multline*}\n\t\t\t\\exp\\left(\\int_0^t -\\frac{\\alpha(s)+\\beta\\lambda}{\\eps(s)}+\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}+\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\n\t\t\t\\\\= o\\left(\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\\right).\n\t\t\\end{multline*}\n\t\tAs a consequence, the second term in \\eqref{eq::formofx} will decrease to $0$ faster than the first-one (let alone the additional $\\eps(t)$ decay that we have just discussed). The asymptotic behavior of $x$ will thus be governed by the first term in \\eqref{eq::formofx}.\n\n\t\tLet us now focus on the first term in \\eqref{eq::formofx}. Observe that\n\t\t$\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}\\diff s\\right)$ is exactly the decay of $x_{LM}$ in \\eqref{eq::closeformLMN}. Thus, we have proved that there exists $C>0$, such that the following asymptotic equivalence holds,\n\t\t\\begin{multline*}%\\label{eq::equivA}\n\t\t\tA\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\alpha(t)+\\beta\\lambda}}{\\sqrt{\\eps(t)}}\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\n\t\t\t\\\\ \\sim_{+\\infty}\n\t\t\tx_{LM}(t)C\\exp\\left(\\int_0^t - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right),\n\t\t\\end{multline*}\n\t\twhich proves the first part of \\eqref{eq::equivx}. The second equivalence in \\eqref{eq::equivx} is obtained using the following identity,\n\t\t\\begin{equation}\\label{eq::rewritteLM}\n\t\t\t\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}\\diff s = \\int_0^t -\\frac{1}{\\beta} + \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}\\diff s = -\\frac{t}{\\beta} + \\int_0^t  \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}\\diff s\n\t\t\\end{equation}\n\t\tand $e^{-t/\\beta}$ is precisely the rate at which $x_N$ decreases. So \\eqref{eq::equivx} holds.\n\n\t\tIt finally remains to deduce the conclusions of the theorem from \\eqref{eq::equivx}.\n\t\t\\\\--\\ Regarding the comparison with $x_{LM}$, the integral $\\int_0^t - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s$ converges if and only if $\\eps$ is integrable on $[0,+\\infty[$, and diverges to $-\\infty$ when $\\eps$ is not. So $x$ converges to $0$ at least as fast as $x_{LM}$ and faster when $\\eps$ is not integrable.\n\t\t\\\\-- As for the comparison with $x_N$, if $\\alpha(s)>\\eps(s)\\geq 0$ for all $s\\geq 0$, then the integral $\\int_0^t \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}- \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s$ is convergent in $+\\infty$ if and only if $\\alpha$ is integrable and diverges to $+\\infty$ when $\\alpha$ is non-integrable. So when $\\alpha$ is integrable, the speed of convergence of $x$ is the same as that of $x_N$. When $\\alpha$ is not integrable, the convergence to $0$ is slower but still holds. Indeed, for all $s\\geq 0$\n\t\t$\\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\frac{1}{\\beta})}< \\frac{\\alpha(s)}{\\beta\\alpha(s)}= \\frac{1}{\\beta}$. Thus for all $t>0$, $-\\frac{t}{\\beta} + \\int_0^t \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}\\diff s<0$.\n\t\t\\\\-- Finally, the comparison with $x_N$ in the case $\\eps(s)> \\alpha(s)$ is exactly the same as the comparison with $x_{LM}$ using \\eqref{eq::rewritteLM}.\n\t\\end{proof}","start":62511,"end":67064},{"raw_tex":"\\begin{proof}\n\t\tWe suppose that Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp} hold. As stated in Remark~\\ref{rem::olver}, since $\\varphi$ is continuous, we only need to check its integrability when $t$ tends to $+\\infty$. Let $t>0$, we first establish some useful identities, we omit the dependence on $t$ for the sake of readability.\n\t\t\\begin{align*}%\\label{eq::pprime}\n\t\t\tp' = \\frac{\\alpha'\\eps-(\\alpha+\\beta\\lambda)\\eps'}{\\eps^2},\n\t\t\t\\quad \\text{and}\\quad\n\t\t\tp'' = \\frac{\\alpha''\\eps^2-2\\alpha'\\eps'\\eps-(\\alpha+\\beta\\lambda)\\eps''\\eps + 2(\\alpha+\\beta\\lambda)(\\eps')^2}{\\eps^3}.\n\t\t\\end{align*}\n\t\tThen,\n\t\t\\begin{align}\\label{eq::factorR}\n\t\t\t\\begin{split}\n\t\t\t\tr  &= \\frac{p^2}{4}\\left(1+ \\frac{2p'}{p^2} - \\frac{4\\lambda}{\\eps p^2}\\right) = \\frac{(\\alpha+\\beta\\lambda)^2}{4\\eps^2}\\left(1+\\frac{2p' \\eps ^2}{(\\alpha +\\beta\\lambda)^2} - \\frac{4\\lambda\\eps }{(\\alpha +\\beta\\lambda)^2}\\right)\n\t\t\t\t\\\\\n\t\t\t\t&=\\frac{(\\alpha +\\beta\\lambda)^2}{4\\eps ^2}\\left(1+\\frac{2\\alpha' \\eps }{(\\alpha +\\beta\\lambda)^2} - \\frac{2\\eps' }{(\\alpha +\\beta\\lambda)} - \\frac{4\\lambda\\eps }{(\\alpha +\\beta\\lambda)^2}\\right).\n\t\t\t\\end{split}\n\t\t\\end{align}\n\t\tAn important consequence of Assumption~\\ref{ass:subexp} is that $\\vert\\eps'(t)\\vert = o(\\eps(t))$, $\\vert\\eps''(t)\\vert = o(\\eps'(t))$ (and the same holds for $\\alpha$ w.r.t.\\ to its derivatives). Therefore, we deduce from \\eqref{eq::factorR} that\n\t\t\\begin{equation*}\n\t\t\tr(t)\\sim_{+\\infty}  \\frac{(\\alpha(t)+\\beta\\lambda)^2}{4\\eps(t)^2},\n\t\t\\end{equation*}\n\t\tand we note that $1/r$ decays at the same speed as $\\eps^2$, which will be useful later. In order to study $\\varphi$, we now differentiate $r$,\n\t\t\\begin{align}\\label{eq::rprime}\n\t\t\t\\begin{split}\n\t\t\t\tr' &= \\frac{p'p}{2}\\left(1+ \\frac{2p'}{p^2} - \\frac{4\\lambda}{\\eps p^2}\\right) + \\frac{1}{4}\\left( 2p'' - \\frac{4(p')^2}{p} + \\frac{8\\lambda p'}{\\eps p}+ \\frac{4\\lambda\\eps'}{\\eps^2} \\right)\n\t\t\t\t\\\\&= \\frac{2p'}{p}r + \\frac{1}{4}\\left( 2p'' - \\frac{4(p')^2}{p} +\\frac{8\\lambda p'}{\\eps p} + \\frac{4\\lambda\\eps'}{\\eps^2}  \\right),\\quad \\text{and,}\n\t\t\t\\end{split}\n\t\t\\end{align}\n\n\t\t\\begin{multline}\\label{eq::rsec}\n\t\t\tr'' = 2\\frac{p''p-(p')^2}{p^2}r  + \\frac{2p'}{p}r'\n\t\t\t\\\\\n\t\t\t+ \\frac{1}{4}\\left( 2p''' + 4\\frac{(p')^3-2p''p'p}{p^2}+8 \\lambda\\frac{p''p\\eps - (p')^2\\eps - p'p\\eps'}{\\eps^2 p^2} + \\frac{4\\lambda\\eps''}{\\eps^2} - \\frac{8\\lambda(\\eps')^2}{\\eps^3}  \\right).\n\t\t\\end{multline}\n\t\tThen, to justify that $\\varphi$ is integrable, we prove that $\\frac{r''}{r^{3/2}}$ and $\\frac{(r')^2}{r^{5/2}}$ are integrable. Since we know that $1/r$ decays at the same speed as $\\eps^2$, we can equivalently show that $\\eps^3r''$ and $\\eps^5(r')^2$ are integrable.\n\t\tTo this aim we fully expand all the terms in \\eqref{eq::rprime} and \\eqref{eq::rsec}.\n\t\tWe first deal with \\eqref{eq::rprime}, it holds that:\n\t\t\\begin{align*}\n\t\t\tr'^2&\\eps^{5} =\n\t\t\t\\left[- \\frac{ (\\alpha+\\beta\\lambda)^{2} \\left(- \\frac{4 \\lambda \\eps }{ (\\alpha+\\beta\\lambda)^{2}} + 1 + \\frac{\\left(- {2  (\\alpha+\\beta\\lambda) \\eps' } + {2 \\alpha'\\eps }\\right) }{ (\\alpha+\\beta\\lambda)^{2}}\\right) \\eps' }{2 \\sqrt{\\eps}}\\right.\n\t\t\t\\\\ &\\left.\n\t\t\t+ \\frac{ (\\alpha+\\beta\\lambda)^{2}\\sqrt{\\eps}}{4} \\left(- \\frac{4 \\lambda \\eps' }{ (\\alpha+\\beta\\lambda)^{2}} + \\frac{8 \\lambda \\alpha'  \\eps }{ (\\alpha+\\beta\\lambda)^{3}} + \\frac{2 \\left(- \\frac{2  (\\alpha+\\beta\\lambda) \\eps' }{\\eps } + {2 \\alpha'\\eps }\\right) \\eps'   }{ (\\alpha+\\beta\\lambda)^{2}}\n\t\t\t\\right.\\right.\n\t\t\t\\\\&\\left.\\left.\n\t\t\t+ \\frac{\\left(\\frac{4  (\\alpha+\\beta\\lambda) \\eps'^{2} }{\\eps } - {2  (\\alpha+\\beta\\lambda) \\eps'' } - {4 \\alpha'  \\eps' } + {2 \\alpha''\\eps }\\right) }{ (\\alpha+\\beta\\lambda)^{2}} - \\frac{2 \\left(-{2  (\\alpha+\\beta\\lambda) \\eps' } + {2 \\alpha'\\eps }\\right) \\alpha'   }{ (\\alpha+\\beta\\lambda)^{3}}\\right)\n\t\t\t\\right.\\\\\n\t\t\t&\\left. + \\frac{ (\\alpha+\\beta\\lambda)}{2} { \\left(- \\frac{4 \\lambda \\eps }{ (\\alpha+\\beta\\lambda)^{2}} + 1 + \\frac{\\left(- {2  (\\alpha+\\beta\\lambda) \\eps' } + {2 \\alpha'\\eps }\\right) }{ (\\alpha+\\beta\\lambda)^{2}}\\right)} \\alpha' \\sqrt{\\eps}\n\t\t\t\\phantom{\\frac{\\frac{\\beta}{\\beta}}{\\beta}}\n\t\t\t\\hspace{-1.7cm}\\phantom{\\frac{\\left( \\frac{\\frac{a'}{a'}}{a'}\\right)}{a^2}{\\sqrt{.}}\n\t\t\t}\\right]^{2}.\n\t\t\\end{align*}\n\t\tThe computations for $\\eps^2r''$ are omitted since they are longer but very similar.\n\n\t\tWe analyze the integrability of each of the terms above (and those of $\\eps^2r''$). By Assumption~\\ref{ass:subexp}, $\\eps'$, $\\eps''$ and $\\eps'''$ are integrable, as well as $\\alpha'$, $\\alpha''$ and $\\alpha'''$, this justifies the integrability of most of the terms. We also need $\\frac{(\\eps')^2}{\\eps}$ and $\\frac{(\\eps')^3}{\\eps}$ to be integrable, which holds by Assumption~\\ref{ass:subexp}. Overall, $\\varphi$ is integrable on $\\R_{+}$.\n\t\\end{proof}","start":91200,"end":95908},{"raw_tex":"\\begin{lemma}\\label{lem::remainderterm}\n\t\tUnder Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp}, for all $s\\geq 0$, $$\\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2 = \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s)).$$\n\t\\end{lemma}","start":96010,"end":96329},{"raw_tex":"\\begin{proof}\n\t\tWe omit the time dependence on $s\\geq 0$ for the sake of readability. Using Assumption~\\ref{ass:smooth} we can define and expand the following quantity,\n\t\t\\begin{align*}\n\t\t\t&\\frac{1}{16}\\left(\\frac{2p'}{p^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps}}{(\\alpha+\\beta\\lambda)^{3/2}}\\right)^2\n\t\t\t=\\frac{\\lambda^2\\eps}{(\\alpha+\\beta\\lambda)^{3}}  - \\frac{p'\\lambda\\sqrt{\\eps}}{p^{3/2}(\\alpha+\\beta\\lambda)^{3/2}} + \\frac{(p')^2}{4p^{3}}\n\t\t\t\\\\&= \\frac{\\lambda^2\\eps}{(\\alpha+\\beta\\lambda)^{3}}\n\t\t\t\t- \\frac{\\lambda(\\alpha'\\eps - (\\alpha+\\beta\\lambda)\\eps')}{(\\alpha+\\beta\\lambda)^{3}}\n\t\t\t\t+ \\frac{(\\alpha')^2\\eps + \\frac{(\\eps')^2}{\\eps}(\\alpha+\\beta\\lambda)^2 - 2 \\alpha'\\eps'(\\alpha+\\beta\\lambda)}{4(\\alpha+\\beta\\lambda)^{3}}.\n\t\t\\end{align*}\n\t\tAssumption~\\ref{ass:subexp}, implies in particular that $\\vert\\eps'(t)\\vert=o(\\eps(t))$ and that $\\alpha'(t)\\to 0$, which we use in the equality above to obtain the desired conclusion.\n\t\\end{proof}","start":96331,"end":97273}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq::VMDINAVD"],"declared_external":false,"source_only":false,"title":"变质量牛顿型动力系统","source_units":[{"raw_tex":"\tBoth steps are highly challenging, our work focuses on the ODE matter. We study the following second-order dynamical system in a general setting:\n\t\\begin{equation}\\tag{VM-DIN-AVD}\\label{eq::VMDINAVD}\n\t\t\\varepsilon(t) \\ddot{x}(t) + \\alpha(t) \\dot x(t) + \\beta \\Hf(x(t))\\dot{x}(t)+\\gf(x(t))=0,\n\t\t\\quad t\\geq 0,\n\t\\end{equation}\n\twhere $f\\colon \\R^n\\to\\R$ is a smooth function, with gradient $\\gf$ and Hessian $\\Hf$ defined on $\\R^n$ equipped with scalar product $\\langle\\cdot,\\cdot\\rangle$, and induced norm $\\norm{\\cdot}$.\n\tThe system is controlled by two functions $\\eps,\\alpha\\colon\\R_+\\to\\R_+$ (where $\\R_+=[0,+\\infty[$) and a parameter $\\beta> 0$ that define the type of dynamics that drives the trajectory (or solution) $x\\colon\\R_+\\to\\R^n$, whose first- and second-order derivatives are denoted $\\dx$ and $\\ddx$ respectively.\n","start":6442,"end":7273}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":["eq::CN"],"declared_external":false,"source_only":false,"title":"连续牛顿比较系统","source_units":[{"raw_tex":"\tThis paper emphasizes the relation between \\eqref{eq::VMDINAVD} and special cases.\n\tIndeed, taking $\\eps=\\alpha=0$, one obtains\\footnote{\\ref{eq::CN} is usually considered with $\\beta=1$, we put $\\beta$ in the system to ease the discussions below.} the Continuous Newton (CN) method \\citep{gavurin1958nonlinear}\n\t\\begin{equation}\\tag{CN}\\label{eq::CN}\n\t\t\\beta \\Hf(x_N(t))\\dot{x}_N(t)+\\gf(x_N(t))=0, \\quad t\\geq 0,\n\t\\end{equation}\n\tknown notably for being invariant to affine transformations and yielding fast vanishing of the gradient (see Section~\\ref{sec::Control}).\n\tIn fact, this observation shows that \\eqref{eq::VMDINAVD} is a singular perturbation of \\eqref{eq::CN}, which also justifies the terminology ``Newton-like''  in DIN.\n","start":8490,"end":9227}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":["eq::LM"],"declared_external":false,"source_only":false,"title":"正则化牛顿比较系统","source_units":[{"raw_tex":"\tWhen $\\alpha\\neq 0$ but $\\eps=0$, we recover the Levenberg--Marquardt (LM) method,\n\t\\begin{equation}\\tag{LM}\\label{eq::LM}\n\t\t\\alpha(t) \\dot x_{LM}(t) + \\beta \\Hf(x_{LM}(t))\\dot{x}_{LM}(t)+\\gf(x_{LM}(t))=0, \\quad t\\geq 0,\n\t\\end{equation}\n\talso known as regularized Newton method since it stabilizes \\eqref{eq::CN}.\n\tIn the rest of the paper, the solutions of \\eqref{eq::CN} and \\eqref{eq::LM} will always be denoted by $x_N$ and $x_{LM}$ respectively. Since the introduction of DIN, it is known (see \\cite{alvarez2002second}) that for $\\alpha=0$, $\\beta=1$, and $\\eps$ constant and small, \\eqref{eq::VMDINAVD} is a ``perturbed'' Newton method since the distance between the solutions of \\eqref{eq::VMDINAVD} and \\eqref{eq::CN}  is at most proportional to $\\sqrt{\\eps}$ at all times.\n","start":9227,"end":10010}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"目标函数的一般假设","source_units":[{"raw_tex":"\tWe always assume the following on the system \\eqref{eq::VMDINAVD} and the functions $f,\\eps,\\alpha$ defined in the introduction.\n\t\t\\begin{assumption}\\label{ass:general}\n\t\t\t\\textbullet\\ $f: \\R^n \\to \\R$ is convex, twice continuously differentiable, coercive, and strongly convex on bounded subsets of $\\R^n$.\n","start":18693,"end":19002}]},{"id":"S005","semantic_type":"assumption","kind":"assumption","labels":["ass:general"],"declared_external":false,"source_only":false,"title":"共同位置与初始速度","source_units":[{"raw_tex":"\t\t\tWe fix $x_0$, $\\dot{x}_0\\in\\R^n$, such that, unless stated otherwise, \\eqref{eq::VMDINAVD} has initial condition $(x(0),\\dx(0))=(x_0,\\dx_0)$, and \\eqref{eq::CN} and \\eqref{eq::LM} have initial conditions $x_{N}(0)=x_{LM}(0)=x_0$.\n\\\\\n","start":19002,"end":19238}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"质量与阻尼的基本限制","source_units":[{"raw_tex":"\t\t\t\\textbullet\\ $\\alpha: \\R_+ \\to \\R$ is non-increasing, non-negative and differentiable. $\\eps: \\R_+ \\to \\R$ is non-increasing, positive, and twice differentiable with bounded second derivative.\n\t\t\tWe fix initial values: $\\eps(0)=\\eps_0>0$, $\\eps'(0)=\\eps'_0\\leq0$ and $\\alpha(0)=\\alpha_0\\geq 0$.\n\t\t\\end{assumption}\n","start":19238,"end":19555}]},{"id":"S007","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"强凸二次函数与谱坐标","source_units":[{"raw_tex":"\t\\section{Approximate Solutions and Asymptotic Analysis on Quadratics}\\label{sec::Quads}\n\tIn addition to Assumption~\\ref{ass:general}, we consider the case where $f$ is a strongly convex quadratic function in order to study the asymptotic behavior of \\eqref{eq::VMDINAVD} w.r.t.\\ \\eqref{eq::CN} and \\eqref{eq::LM}. Quadratic functions are the prototypical example of strongly convex functions. In particular, any strongly convex function can be locally approximated by a quadratic one around its minimizer, making the latter a good model for understanding the local behavior of dynamics.\n\tIn this section, $f$ is quadratic: $f(y)=\\frac{1}{2}\\Vert Ay-b\\Vert_2^2$ for all $y\\in \\R^n$, where $A\\in\\R^{n\\times n}$ is symmetric positive definite and $b\\in\\R^n$. Without loss of generality, we take $b=0$, so that the unique minimum is $x^\\star=0$.\n\t\\subsection{Setting: the Special Case of Quadratic Functions}\n\tQuadratic functions are particularly interesting in our setting since DIN-like ODEs take a simpler form (as observed in \\cite{attouch2020first,shi2021understanding}). Indeed, $\\forall y\\in\\R^n$, $\\gf(y) = A^TAy$ and $\\Hf(y)=A^T A$. Since $\\Hf(y)$ is independent of $y$ we can rewrite \\eqref{eq::VMDINAVD} in an eigenspace\\footnote{This can be generalized to the case where $A^TA$ is only semi-definite by considering orthogonal projections on an eigenspace spanned by the positive eigenvalues of $A^TA$.} of  $A^TA$. That is, we can study the ODE coordinate-wise by looking at one-dimensional problems of the form\n","start":40821,"end":42342}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq::QuadeDINAVD"],"declared_external":false,"source_only":false,"title":"单特征方向的质量方程","source_units":[{"raw_tex":"\t\\begin{equation}\\label{eq::QuadeDINAVD}\\tag{Q1-VM-DIN-AVD}\n\t\t\\eps(t)\\ddx(t)+(\\alpha(t)+\\beta\\lambda)\\dx(t)+\\lambda x(t)=0, \\quad t\\geq 0.\n\t\\end{equation}\n\tHere (and throughout what follows) $\\lambda>0$ denotes any eigenvalue of $A^TA$ and $x\\colon\\R_+\\to\\R$ now denotes the corresponding  coordinate (function) of the solution of \\eqref{eq::VMDINAVD} in an eigenspace of $A^TA$. The dynamics \\eqref{eq::QuadeDINAVD} is a \\emph{linear} second-order ODE in $x$ with non-constant coefficients. Similarly, \\eqref{eq::LM} can be rewritten coordinate-wise as\n","start":42342,"end":42896}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq::quadCN","eq::quadLM"],"declared_external":false,"source_only":false,"title":"牛顿与正则化牛顿标量式","source_units":[{"raw_tex":"\t\\begin{equation}\\label{eq::quadLM}\\tag{Q1-LM}\n\t\t(\\alpha(t)+\\beta\\lambda)\\dx_{LM}(t)+\\lambda x_{LM}(t)=0, \\quad t\\geq 0,\n\t\\end{equation}\n\twhere $\\xlm\\colon\\R_+\\to\\R$, and \\eqref{eq::CN} becomes\n\t\\begin{equation}\\label{eq::quadCN}\\tag{Q1-CN}\n\t\t\\beta\\dx_N(t)+ x_N(t)=0, \\quad t\\geq 0,\n\t\\end{equation}\n\twhere again, $x_N\\colon\\R_+\\to\\R$ is one-dimensional.\n","start":42896,"end":43250}]},{"id":"S010","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq::closeformLMN"],"declared_external":false,"source_only":false,"title":"比较轨迹的闭式表达","source_units":[{"raw_tex":"\tObserve in particular that \\eqref{eq::CN} and \\eqref{eq::LM} are now first-order \\emph{linear} ODEs, whose solutions have closed forms: $\\forall t\\geq 0$,\n\t\\begin{equation}\\label{eq::closeformLMN}\n\t\tx_N(t) = x_0e^{-\\frac{t}{\\beta}}\n\t\t\\quad \\text{and}\\quad\n\t\tx_{LM}(t) = x_0\\exp\\left(-\\int_0^t \\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}\\diff s \\right).\n\t\\end{equation}\n","start":43250,"end":43615}]},{"id":"S011","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"误差定理的历史引介","source_units":[{"raw_tex":"\tThe advantage of this approach is the possibility to estimate the error made using \\eqref{eq::LG} w.r.t.\\ the true solutions of \\eqref{eq::GenericODE}. This is expressed in the following theorem which gathers results from \\cite{blumenthal1899uber,olver1961error,taylor1982improved}.\n","start":46426,"end":46710}]},{"id":"S012","semantic_type":"external_fact","kind":"external_fact","labels":["eq::GenericODEthm","thm::olver"],"declared_external":true,"declaration_text":["原始获准前提：Olver's Liouville–Green approximation theorem，引用 olver1997asymptotics，文中重述为 B0063。"],"source_only":false,"title":"Olver 的全局近似定理","source_units":[{"raw_tex":"\t\\begin{theorem}[\\cite{olver1997asymptotics}]\\label{thm::olver}\n\t\tLet $r\\colon \\R_{+}\\to \\R$ be a real, positive, twice continuously differentiable function, and define $\\varphi(t) = \\frac{4r(t)r''(t)-5r'(t)^2}{16r(t)^{5/2}}$ for all $t\\geq 0$. Then for any $T>0$, the differential equation,\n\t\t\\begin{equation}\\label{eq::GenericODEthm}\n\t\t\t\\ddot{z}(t)-r(t)z(t)=0,\\quad t\\in [0,T],\n\t\t\\end{equation}\n\t\thas two real and twice continuously differentiable solutions defined $\\forall t\\in[0,T]$ by,\n\t\t\\begin{equation*}\n\t\t\tz_1(t) = \\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t \\sqrt{r(s)}\\diff s\\right) \\ \\ \\text{and} \\ \\,\n\t\t\tz_2(t) = \\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\exp\\left(-\\int_0^t \\sqrt{r(s)}\\diff s\\right),\n\t\t\\end{equation*}\n\t\twhere $\\displaystyle\\vert\\delta_1(t)\\vert\\leq \\exp\\left(\\frac{1}{2} \\int_0^t \\vert \\varphi(s)\\vert\\diff s\\right)-1$ and $\\displaystyle\\vert\\delta_2(t)\\vert\\leq \\exp\\left(-\\frac{1}{2} \\int_t^{T} \\vert \\varphi(s)\\vert\\diff s\\right)-1$.\n\t\tIf in addition $\\displaystyle\\int_0^{+\\infty} \\vert\\varphi(s)\\vert\\diff s <+\\infty$, then the results above also hold for $T=+\\infty$.\n\t\\end{theorem}\n","start":46710,"end":47828}]},{"id":"S013","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"精确解与近似解的区别","source_units":[{"raw_tex":"\t\\begin{remark}\\label{rem::olver} We make the following remarks regarding the above result.\\\\\n\t\t--\\ Note that $z_1$ and $z_2$ in Theorem~\\ref{thm::olver} are \\emph{exact} solutions to \\eqref{eq::GenericODEthm}. The LG approximations $\\hat z_1$ and $\\hat z_2$ are obtained by neglecting the unknown functions $\\delta_1$ and $\\delta_2$ in $z_1$ and $z_2$. The theorem gives a \\emph{non-asymptotic} bound for the errors $\\vert z_1(t)-\\hat{z}_1(t)\\vert$ and $\\vert z_2(t)-\\hat{z}_2(t)\\vert$, $t\\geq 0$.\n\t\t\\\\\n","start":47828,"end":48332}]},{"id":"S014","semantic_type":"derived_fact","kind":"derived_fact","labels":["rem::olver"],"declared_external":false,"source_only":false,"title":"误差密度的局部可积性","source_units":[{"raw_tex":"\t\t--\\ Since we assumed $r$ to be twice continuously differentiable and positive, $\\varphi$ is continuous, so it is integrable except maybe for $t\\to+\\infty$.\n","start":48332,"end":48490}]},{"id":"S015","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"外部定理的重述说明","source_units":[{"raw_tex":"\t\t\\\\\n\t\t--\\ For the sake of simplicity, the formulation of Theorem~\\ref{thm::olver} slightly differs from that in \\cite{olver1997asymptotics}, the original formulation can be recovered by a change of variable.\n\t\\end{remark}\n","start":48490,"end":48713}]},{"id":"S016","semantic_type":"assumption","kind":"assumption","labels":["ass:smooth"],"declared_external":false,"source_only":false,"title":"三阶光滑性与小初始质量","source_units":[{"raw_tex":"\tWe now proceed to make use of the LG method for approximating the solutions of \\eqref{eq::QuadeDINAVD}. The reader only interested in the result can jump directly to the  Section~\\ref{sec::mainasymptoticres}. We first make the following assumption.\n\t\\begin{assumption}\\label{ass:smooth}\n\t\tThe functions $\\alpha$ and $\\eps$ are three times continuously differentiable, and $\\eps_0$ is such that $\\forall t\\geq 0$, $\\eps_0<\\frac{(\\beta\\lambda)^2}{2\\vert\\alpha'(t)\\vert+4\\lambda}$.\n\t\\end{assumption}\n","start":48815,"end":49313}]},{"id":"S017","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"全局误差密度","source_units":[{"raw_tex":"\tLemma~\\ref{lem::canonForm} gives a reformulation of \\eqref{eq::QuadeDINAVD} suited to apply Theorem~\\ref{thm::olver}.\n\tTo use the theorem for all $t\\geq 0$, we need to ensure that $\\varphi(t)=\\frac{4r(t)r''(t)-5r'(t)^2}{16r(t)^{5/2}}$ is integrable. To this aim we make the following assumption.\n","start":52183,"end":52480}]},{"id":"S018","semantic_type":"assumption","kind":"assumption","labels":["ass:subexp"],"declared_external":false,"source_only":false,"title":"导数可积与质量消失","source_units":[{"raw_tex":"\t\\begin{assumption}\\label{ass:subexp}\n\t\tThe functions $\\eps$ and $\\alpha$ have first, second and third-order derivatives that are integrable on $[0,+\\infty[$. In addition, $\\lim\\limits_{t\\to\\infty}{\\eps(t)}=0$ and $\\eps'(t)^2/\\eps(t)$ is integrable on $[0,+\\infty[$.\n\t\\end{assumption}\n","start":52480,"end":52765}]},{"id":"S019","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"由近似式转向渐近比较","source_units":[{"raw_tex":"\tWe now have an expression for $x$ which is almost explicit: we do not know $\\delta_1$ and $\\delta_2$ in closed form, but they are uniformly bounded (by Lemma~\\ref{lem::phiintegrable}). We will now  compare the asymptotic behavior of \\eqref{eq::formofx} with those of the solutions of \\eqref{eq::quadLM} and \\eqref{eq::quadCN} that we denoted $x_{LM}$ and $x_N$ respectively.\n","start":59181,"end":59557}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"渐近等价的比值含义","source_units":[{"raw_tex":"\tOur main result of Section~\\ref{sec::Quads} is the following, where $\\sim_{+\\infty}$ denotes the asymptotic equivalence\\footnote{Two real-valued functions $g_1$ and $g_2$ are asymptotically equivalent in $+\\infty$ if and only if $\\lim_{t\\to\\infty}\\frac{g_1(t)}{g_2(t)}=1$.} between two functions as $t\\to\\infty$.\n","start":59557,"end":59871}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","labels":["eq::ODErewritten","eq::pr","lem::canonForm"],"declared_external":false,"source_only":false,"title":"消去一阶项的规范形引理","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem::canonForm}\n\t\tSuppose that Assumption~\\ref{ass:smooth} holds, and let $x$ be the solution of \\ref{eq::QuadeDINAVD}. For all $t\\geq 0$, define\n\t\t\\begin{equation}\\label{eq::pr}\n\t\t\tp(t) = \\frac{\\alpha(t)+\\beta\\lambda}{\\eps(t)} \\quad \\text{and}  \\quad r(t) = \\frac{p(t)^2}{4} + \\frac{p'(t)}{2} - \\frac{\\lambda}{\\eps(t)}.\n\t\t\\end{equation}\n\t\tThen, $p$ and $r$ are twice continuously differentiable, $r$ is positive and the function $y$ defined for all $t\\geq 0$ by $y(t) = x(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)$ is a solution to\n\t\t\\begin{equation}\\label{eq::ODErewritten}\n\t\t\t\\ddot y(t) - r(t)y(t) = 0,\\quad t\\geq 0,%\\\\\n\t\t\\end{equation}\n\t\twith initial condition $(y(0),\\dot{y}(0) )= (x_0,\\dx_0 + \\frac{p(0)}{2}x_0)$.\n\t\\end{lemma}","start":49805,"end":50565}]},{"id":"S022","semantic_type":"definition","kind":"definition","labels":["eq::pr","lem::canonForm"],"declared_external":false,"source_only":false,"title":"规范形系数 p 与 r","source_units":[{"raw_tex":"\t\tSuppose that Assumption~\\ref{ass:smooth} holds, and let $x$ be the solution of \\ref{eq::QuadeDINAVD}. For all $t\\geq 0$, define\n\t\t\\begin{equation}\\label{eq::pr}\n\t\t\tp(t) = \\frac{\\alpha(t)+\\beta\\lambda}{\\eps(t)} \\quad \\text{and}  \\quad r(t) = \\frac{p(t)^2}{4} + \\frac{p'(t)}{2} - \\frac{\\lambda}{\\eps(t)}.\n\t\t\\end{equation}\n","start":49841,"end":50163}]},{"id":"S023","semantic_type":"proof","kind":"proof","labels":["eq::rpositive"],"declared_external":false,"source_only":false,"title":"规范形引理的作者全证明","source_units":[{"raw_tex":"\\begin{proof}\n\t\tWe first check that for all $t\\geq 0$, $r(t)$ is positive. Let $t>0$,\n\t\t\\begin{equation}\\label{eq::rpositive}\n\t\t\tr(t)> 0 \\iff \\frac{(\\alpha(t)+\\beta\\lambda)^2}{4\\eps(t)^2} + \\frac{\\alpha'(t)}{2\\eps(t)} -  \\frac{(\\alpha(t)+\\beta\\lambda)\\eps'(t)}{\\eps(t)^2} - \\frac{\\lambda}{\\eps(t)} >0.\n\t\t\\end{equation}\n\t\tSince $\\eps'(t)\\leq 0$ and $\\alpha'(t)\\leq 0$, one can check that a sufficient condition for \\eqref{eq::rpositive} to hold is,\n\t\t\\begin{equation*}\n\t\t\tr(t)> 0 \\impliedby \\frac{(\\alpha(t)+\\beta\\lambda)^2}{4} > \\left(\\frac{\\vert\\alpha'(t)\\vert}{2}+\\lambda\\right)\\eps(t) \\impliedby \\frac{(\\beta\\lambda)^2}{2\\vert\\alpha'(t)\\vert+4\\lambda}>\\eps_0.\n\t\t\\end{equation*}\n\t\tSo under Assumption~\\ref{ass:smooth}, for all $t\\geq 0$, $r(t)>0$. We then check that $y$ is indeed solution to \\eqref{eq::ODErewritten}. Let $t>0$,\n\t\t\\begin{equation*}\n\t\t\t\\dot{y}(t) = \\frac{p(t)}{2}x(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right) + \\dx(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right),\\quad \\text{and,}\n\t\t\\end{equation*}\n\t\t\\begin{equation*}\n\t\t\t\\ddot{y}(t) = \\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)\\left[\\left(\\frac{p(t)^2}{4}+\\frac{p'(t)}{2}\\right)x(t) + p(t)\\dx(t) + \\ddx(t)\\right].\n\t\t\\end{equation*}\n\t\tSince $x$ solves \\eqref{eq::QuadeDINAVD}, it holds that $\\ddx(t) = -p(t)\\dx(t) - \\frac{\\lambda}{\\eps(t)}x(t)$, so,\n\t\t\\begin{multline*}\n\t\t\t\\ddot{y}(t) = \\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)\\left(\\frac{p(t)^2}{4}+\\frac{p'(t)}{2}- \\frac{\\lambda}{\\eps(t)}\\right)x(t) \\\\= \\left(\\frac{p(t)^2}{4}+\\frac{p'(t)}{2}- \\frac{\\lambda}{\\eps(t)}\\right)y(t)\n\t\t\t= r(t)y(t).\n\t\t\\end{multline*}\n\t\\end{proof}","start":50567,"end":52182}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","labels":["eq::rpositive"],"declared_external":false,"source_only":false,"title":"用质量上限保证 r 正","source_units":[{"raw_tex":"\\begin{proof}\n\t\tWe first check that for all $t\\geq 0$, $r(t)$ is positive. Let $t>0$,\n\t\t\\begin{equation}\\label{eq::rpositive}\n\t\t\tr(t)> 0 \\iff \\frac{(\\alpha(t)+\\beta\\lambda)^2}{4\\eps(t)^2} + \\frac{\\alpha'(t)}{2\\eps(t)} -  \\frac{(\\alpha(t)+\\beta\\lambda)\\eps'(t)}{\\eps(t)^2} - \\frac{\\lambda}{\\eps(t)} >0.\n\t\t\\end{equation}\n\t\tSince $\\eps'(t)\\leq 0$ and $\\alpha'(t)\\leq 0$, one can check that a sufficient condition for \\eqref{eq::rpositive} to hold is,\n\t\t\\begin{equation*}\n\t\t\tr(t)> 0 \\impliedby \\frac{(\\alpha(t)+\\beta\\lambda)^2}{4} > \\left(\\frac{\\vert\\alpha'(t)\\vert}{2}+\\lambda\\right)\\eps(t) \\impliedby \\frac{(\\beta\\lambda)^2}{2\\vert\\alpha'(t)\\vert+4\\lambda}>\\eps_0.\n\t\t\\end{equation*}\n\t\tSo under Assumption~\\ref{ass:smooth}, for all $t\\geq 0$, $r(t)>0$. We then check that $y$ is indeed solution to \\eqref{eq::ODErewritten}. Let $t>0$,\n","start":50567,"end":51399}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"微分指数变换并代回方程","source_units":[{"raw_tex":"\t\t\\begin{equation*}\n\t\t\t\\dot{y}(t) = \\frac{p(t)}{2}x(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right) + \\dx(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right),\\quad \\text{and,}\n\t\t\\end{equation*}\n\t\t\\begin{equation*}\n\t\t\t\\ddot{y}(t) = \\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)\\left[\\left(\\frac{p(t)^2}{4}+\\frac{p'(t)}{2}\\right)x(t) + p(t)\\dx(t) + \\ddx(t)\\right].\n\t\t\\end{equation*}\n\t\tSince $x$ solves \\eqref{eq::QuadeDINAVD}, it holds that $\\ddx(t) = -p(t)\\dx(t) - \\frac{\\lambda}{\\eps(t)}x(t)$, so,\n\t\t\\begin{multline*}\n\t\t\t\\ddot{y}(t) = \\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)\\left(\\frac{p(t)^2}{4}+\\frac{p'(t)}{2}- \\frac{\\lambda}{\\eps(t)}\\right)x(t) \\\\= \\left(\\frac{p(t)^2}{4}+\\frac{p'(t)}{2}- \\frac{\\lambda}{\\eps(t)}\\right)y(t)\n\t\t\t= r(t)y(t).\n\t\t\\end{multline*}\n\t\\end{proof}","start":51399,"end":52182}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","labels":["lem::phiintegrable"],"declared_external":false,"source_only":false,"title":"误差密度的半轴可积性","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem::phiintegrable}\n\t\tUnder Assumption~\\ref{ass:smooth} and~\\ref{ass:subexp}, $\\int_0^{+\\infty} \\vert \\varphi(s)\\vert\\diff s<+\\infty$.\n\t\\end{lemma}","start":53173,"end":53340}]},{"id":"S027","semantic_type":"proof","kind":"proof","labels":["eq::factorR","eq::rprime","eq::rsec"],"declared_external":false,"source_only":false,"title":"误差可积性的作者全证明","source_units":[{"raw_tex":"\\begin{proof}\n\t\tWe suppose that Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp} hold. As stated in Remark~\\ref{rem::olver}, since $\\varphi$ is continuous, we only need to check its integrability when $t$ tends to $+\\infty$. Let $t>0$, we first establish some useful identities, we omit the dependence on $t$ for the sake of readability.\n\t\t\\begin{align*}%\\label{eq::pprime}\n\t\t\tp' = \\frac{\\alpha'\\eps-(\\alpha+\\beta\\lambda)\\eps'}{\\eps^2},\n\t\t\t\\quad \\text{and}\\quad\n\t\t\tp'' = \\frac{\\alpha''\\eps^2-2\\alpha'\\eps'\\eps-(\\alpha+\\beta\\lambda)\\eps''\\eps + 2(\\alpha+\\beta\\lambda)(\\eps')^2}{\\eps^3}.\n\t\t\\end{align*}\n\t\tThen,\n\t\t\\begin{align}\\label{eq::factorR}\n\t\t\t\\begin{split}\n\t\t\t\tr  &= \\frac{p^2}{4}\\left(1+ \\frac{2p'}{p^2} - \\frac{4\\lambda}{\\eps p^2}\\right) = \\frac{(\\alpha+\\beta\\lambda)^2}{4\\eps^2}\\left(1+\\frac{2p' \\eps ^2}{(\\alpha +\\beta\\lambda)^2} - \\frac{4\\lambda\\eps }{(\\alpha +\\beta\\lambda)^2}\\right)\n\t\t\t\t\\\\\n\t\t\t\t&=\\frac{(\\alpha +\\beta\\lambda)^2}{4\\eps ^2}\\left(1+\\frac{2\\alpha' \\eps }{(\\alpha +\\beta\\lambda)^2} - \\frac{2\\eps' }{(\\alpha +\\beta\\lambda)} - \\frac{4\\lambda\\eps }{(\\alpha +\\beta\\lambda)^2}\\right).\n\t\t\t\\end{split}\n\t\t\\end{align}\n\t\tAn important consequence of Assumption~\\ref{ass:subexp} is that $\\vert\\eps'(t)\\vert = o(\\eps(t))$, $\\vert\\eps''(t)\\vert = o(\\eps'(t))$ (and the same holds for $\\alpha$ w.r.t.\\ to its derivatives). Therefore, we deduce from \\eqref{eq::factorR} that\n\t\t\\begin{equation*}\n\t\t\tr(t)\\sim_{+\\infty}  \\frac{(\\alpha(t)+\\beta\\lambda)^2}{4\\eps(t)^2},\n\t\t\\end{equation*}\n\t\tand we note that $1/r$ decays at the same speed as $\\eps^2$, which will be useful later. In order to study $\\varphi$, we now differentiate $r$,\n\t\t\\begin{align}\\label{eq::rprime}\n\t\t\t\\begin{split}\n\t\t\t\tr' &= \\frac{p'p}{2}\\left(1+ \\frac{2p'}{p^2} - \\frac{4\\lambda}{\\eps p^2}\\right) + \\frac{1}{4}\\left( 2p'' - \\frac{4(p')^2}{p} + \\frac{8\\lambda p'}{\\eps p}+ \\frac{4\\lambda\\eps'}{\\eps^2} \\right)\n\t\t\t\t\\\\&= \\frac{2p'}{p}r + \\frac{1}{4}\\left( 2p'' - \\frac{4(p')^2}{p} +\\frac{8\\lambda p'}{\\eps p} + \\frac{4\\lambda\\eps'}{\\eps^2}  \\right),\\quad \\text{and,}\n\t\t\t\\end{split}\n\t\t\\end{align}\n\n\t\t\\begin{multline}\\label{eq::rsec}\n\t\t\tr'' = 2\\frac{p''p-(p')^2}{p^2}r  + \\frac{2p'}{p}r'\n\t\t\t\\\\\n\t\t\t+ \\frac{1}{4}\\left( 2p''' + 4\\frac{(p')^3-2p''p'p}{p^2}+8 \\lambda\\frac{p''p\\eps - (p')^2\\eps - p'p\\eps'}{\\eps^2 p^2} + \\frac{4\\lambda\\eps''}{\\eps^2} - \\frac{8\\lambda(\\eps')^2}{\\eps^3}  \\right).\n\t\t\\end{multline}\n\t\tThen, to justify that $\\varphi$ is integrable, we prove that $\\frac{r''}{r^{3/2}}$ and $\\frac{(r')^2}{r^{5/2}}$ are integrable. Since we know that $1/r$ decays at the same speed as $\\eps^2$, we can equivalently show that $\\eps^3r''$ and $\\eps^5(r')^2$ are integrable.\n\t\tTo this aim we fully expand all the terms in \\eqref{eq::rprime} and \\eqref{eq::rsec}.\n\t\tWe first deal with \\eqref{eq::rprime}, it holds that:\n\t\t\\begin{align*}\n\t\t\tr'^2&\\eps^{5} =\n\t\t\t\\left[- \\frac{ (\\alpha+\\beta\\lambda)^{2} \\left(- \\frac{4 \\lambda \\eps }{ (\\alpha+\\beta\\lambda)^{2}} + 1 + \\frac{\\left(- {2  (\\alpha+\\beta\\lambda) \\eps' } + {2 \\alpha'\\eps }\\right) }{ (\\alpha+\\beta\\lambda)^{2}}\\right) \\eps' }{2 \\sqrt{\\eps}}\\right.\n\t\t\t\\\\ &\\left.\n\t\t\t+ \\frac{ (\\alpha+\\beta\\lambda)^{2}\\sqrt{\\eps}}{4} \\left(- \\frac{4 \\lambda \\eps' }{ (\\alpha+\\beta\\lambda)^{2}} + \\frac{8 \\lambda \\alpha'  \\eps }{ (\\alpha+\\beta\\lambda)^{3}} + \\frac{2 \\left(- \\frac{2  (\\alpha+\\beta\\lambda) \\eps' }{\\eps } + {2 \\alpha'\\eps }\\right) \\eps'   }{ (\\alpha+\\beta\\lambda)^{2}}\n\t\t\t\\right.\\right.\n\t\t\t\\\\&\\left.\\left.\n\t\t\t+ \\frac{\\left(\\frac{4  (\\alpha+\\beta\\lambda) \\eps'^{2} }{\\eps } - {2  (\\alpha+\\beta\\lambda) \\eps'' } - {4 \\alpha'  \\eps' } + {2 \\alpha''\\eps }\\right) }{ (\\alpha+\\beta\\lambda)^{2}} - \\frac{2 \\left(-{2  (\\alpha+\\beta\\lambda) \\eps' } + {2 \\alpha'\\eps }\\right) \\alpha'   }{ (\\alpha+\\beta\\lambda)^{3}}\\right)\n\t\t\t\\right.\\\\\n\t\t\t&\\left. + \\frac{ (\\alpha+\\beta\\lambda)}{2} { \\left(- \\frac{4 \\lambda \\eps }{ (\\alpha+\\beta\\lambda)^{2}} + 1 + \\frac{\\left(- {2  (\\alpha+\\beta\\lambda) \\eps' } + {2 \\alpha'\\eps }\\right) }{ (\\alpha+\\beta\\lambda)^{2}}\\right)} \\alpha' \\sqrt{\\eps}\n\t\t\t\\phantom{\\frac{\\frac{\\beta}{\\beta}}{\\beta}}\n\t\t\t\\hspace{-1.7cm}\\phantom{\\frac{\\left( \\frac{\\frac{a'}{a'}}{a'}\\right)}{a^2}{\\sqrt{.}}\n\t\t\t}\\right]^{2}.\n\t\t\\end{align*}\n\t\tThe computations for $\\eps^2r''$ are omitted since they are longer but very similar.\n\n\t\tWe analyze the integrability of each of the terms above (and those of $\\eps^2r''$). By Assumption~\\ref{ass:subexp}, $\\eps'$, $\\eps''$ and $\\eps'''$ are integrable, as well as $\\alpha'$, $\\alpha''$ and $\\alpha'''$, this justifies the integrability of most of the terms. We also need $\\frac{(\\eps')^2}{\\eps}$ and $\\frac{(\\eps')^3}{\\eps}$ to be integrable, which holds by Assumption~\\ref{ass:subexp}. Overall, $\\varphi$ is integrable on $\\R_{+}$.\n\t\\end{proof}","start":91200,"end":95908}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"归约无穷远并计算 p 的导数","source_units":[{"raw_tex":"\\begin{proof}\n\t\tWe suppose that Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp} hold. As stated in Remark~\\ref{rem::olver}, since $\\varphi$ is continuous, we only need to check its integrability when $t$ tends to $+\\infty$. Let $t>0$, we first establish some useful identities, we omit the dependence on $t$ for the sake of readability.\n\t\t\\begin{align*}%\\label{eq::pprime}\n\t\t\tp' = \\frac{\\alpha'\\eps-(\\alpha+\\beta\\lambda)\\eps'}{\\eps^2},\n\t\t\t\\quad \\text{and}\\quad\n\t\t\tp'' = \\frac{\\alpha''\\eps^2-2\\alpha'\\eps'\\eps-(\\alpha+\\beta\\lambda)\\eps''\\eps + 2(\\alpha+\\beta\\lambda)(\\eps')^2}{\\eps^3}.\n\t\t\\end{align*}\n","start":91200,"end":91802}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":["eq::factorR"],"declared_external":false,"source_only":false,"title":"分解 r 的主因子","source_units":[{"raw_tex":"\t\tThen,\n\t\t\\begin{align}\\label{eq::factorR}\n\t\t\t\\begin{split}\n\t\t\t\tr  &= \\frac{p^2}{4}\\left(1+ \\frac{2p'}{p^2} - \\frac{4\\lambda}{\\eps p^2}\\right) = \\frac{(\\alpha+\\beta\\lambda)^2}{4\\eps^2}\\left(1+\\frac{2p' \\eps ^2}{(\\alpha +\\beta\\lambda)^2} - \\frac{4\\lambda\\eps }{(\\alpha +\\beta\\lambda)^2}\\right)\n\t\t\t\t\\\\\n\t\t\t\t&=\\frac{(\\alpha +\\beta\\lambda)^2}{4\\eps ^2}\\left(1+\\frac{2\\alpha' \\eps }{(\\alpha +\\beta\\lambda)^2} - \\frac{2\\eps' }{(\\alpha +\\beta\\lambda)} - \\frac{4\\lambda\\eps }{(\\alpha +\\beta\\lambda)^2}\\right).\n\t\t\t\\end{split}\n\t\t\\end{align}\n","start":91802,"end":92332}]},{"id":"S030","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"作者声称的导数衰减层级","source_units":[{"raw_tex":"\t\tAn important consequence of Assumption~\\ref{ass:subexp} is that $\\vert\\eps'(t)\\vert = o(\\eps(t))$, $\\vert\\eps''(t)\\vert = o(\\eps'(t))$ (and the same holds for $\\alpha$ w.r.t.\\ to its derivatives). Therefore, we deduce from \\eqref{eq::factorR} that\n","start":92332,"end":92582}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"识别 r 与倒数的渐近量级","source_units":[{"raw_tex":"\t\t\\begin{equation*}\n\t\t\tr(t)\\sim_{+\\infty}  \\frac{(\\alpha(t)+\\beta\\lambda)^2}{4\\eps(t)^2},\n\t\t\\end{equation*}\n\t\tand we note that $1/r$ decays at the same speed as $\\eps^2$, which will be useful later. In order to study $\\varphi$, we now differentiate $r$,\n","start":92582,"end":92836}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":["eq::rprime","eq::rsec"],"declared_external":false,"source_only":false,"title":"展开 r 的一二阶导数","source_units":[{"raw_tex":"\t\t\\begin{align}\\label{eq::rprime}\n\t\t\t\\begin{split}\n\t\t\t\tr' &= \\frac{p'p}{2}\\left(1+ \\frac{2p'}{p^2} - \\frac{4\\lambda}{\\eps p^2}\\right) + \\frac{1}{4}\\left( 2p'' - \\frac{4(p')^2}{p} + \\frac{8\\lambda p'}{\\eps p}+ \\frac{4\\lambda\\eps'}{\\eps^2} \\right)\n\t\t\t\t\\\\&= \\frac{2p'}{p}r + \\frac{1}{4}\\left( 2p'' - \\frac{4(p')^2}{p} +\\frac{8\\lambda p'}{\\eps p} + \\frac{4\\lambda\\eps'}{\\eps^2}  \\right),\\quad \\text{and,}\n\t\t\t\\end{split}\n\t\t\\end{align}\n\n\t\t\\begin{multline}\\label{eq::rsec}\n\t\t\tr'' = 2\\frac{p''p-(p')^2}{p^2}r  + \\frac{2p'}{p}r'\n\t\t\t\\\\\n\t\t\t+ \\frac{1}{4}\\left( 2p''' + 4\\frac{(p')^3-2p''p'p}{p^2}+8 \\lambda\\frac{p''p\\eps - (p')^2\\eps - p'p\\eps'}{\\eps^2 p^2} + \\frac{4\\lambda\\eps''}{\\eps^2} - \\frac{8\\lambda(\\eps')^2}{\\eps^3}  \\right).\n\t\t\\end{multline}\n","start":92836,"end":93576}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"化为加权导数的可积性","source_units":[{"raw_tex":"\t\tThen, to justify that $\\varphi$ is integrable, we prove that $\\frac{r''}{r^{3/2}}$ and $\\frac{(r')^2}{r^{5/2}}$ are integrable. Since we know that $1/r$ decays at the same speed as $\\eps^2$, we can equivalently show that $\\eps^3r''$ and $\\eps^5(r')^2$ are integrable.\n\t\tTo this aim we fully expand all the terms in \\eqref{eq::rprime} and \\eqref{eq::rsec}.\n\t\tWe first deal with \\eqref{eq::rprime}, it holds that:\n","start":93576,"end":93990}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"展示加权平方导数展开","source_units":[{"raw_tex":"\t\t\\begin{align*}\n\t\t\tr'^2&\\eps^{5} =\n\t\t\t\\left[- \\frac{ (\\alpha+\\beta\\lambda)^{2} \\left(- \\frac{4 \\lambda \\eps }{ (\\alpha+\\beta\\lambda)^{2}} + 1 + \\frac{\\left(- {2  (\\alpha+\\beta\\lambda) \\eps' } + {2 \\alpha'\\eps }\\right) }{ (\\alpha+\\beta\\lambda)^{2}}\\right) \\eps' }{2 \\sqrt{\\eps}}\\right.\n\t\t\t\\\\ &\\left.\n\t\t\t+ \\frac{ (\\alpha+\\beta\\lambda)^{2}\\sqrt{\\eps}}{4} \\left(- \\frac{4 \\lambda \\eps' }{ (\\alpha+\\beta\\lambda)^{2}} + \\frac{8 \\lambda \\alpha'  \\eps }{ (\\alpha+\\beta\\lambda)^{3}} + \\frac{2 \\left(- \\frac{2  (\\alpha+\\beta\\lambda) \\eps' }{\\eps } + {2 \\alpha'\\eps }\\right) \\eps'   }{ (\\alpha+\\beta\\lambda)^{2}}\n\t\t\t\\right.\\right.\n\t\t\t\\\\&\\left.\\left.\n\t\t\t+ \\frac{\\left(\\frac{4  (\\alpha+\\beta\\lambda) \\eps'^{2} }{\\eps } - {2  (\\alpha+\\beta\\lambda) \\eps'' } - {4 \\alpha'  \\eps' } + {2 \\alpha''\\eps }\\right) }{ (\\alpha+\\beta\\lambda)^{2}} - \\frac{2 \\left(-{2  (\\alpha+\\beta\\lambda) \\eps' } + {2 \\alpha'\\eps }\\right) \\alpha'   }{ (\\alpha+\\beta\\lambda)^{3}}\\right)\n\t\t\t\\right.\\\\\n\t\t\t&\\left. + \\frac{ (\\alpha+\\beta\\lambda)}{2} { \\left(- \\frac{4 \\lambda \\eps }{ (\\alpha+\\beta\\lambda)^{2}} + 1 + \\frac{\\left(- {2  (\\alpha+\\beta\\lambda) \\eps' } + {2 \\alpha'\\eps }\\right) }{ (\\alpha+\\beta\\lambda)^{2}}\\right)} \\alpha' \\sqrt{\\eps}\n\t\t\t\\phantom{\\frac{\\frac{\\beta}{\\beta}}{\\beta}}\n\t\t\t\\hspace{-1.7cm}\\phantom{\\frac{\\left( \\frac{\\frac{a'}{a'}}{a'}\\right)}{a^2}{\\sqrt{.}}\n\t\t\t}\\right]^{2}.\n\t\t\\end{align*}\n","start":93990,"end":95362}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"逐项可积性与省略的计算","source_units":[{"raw_tex":"\t\tThe computations for $\\eps^2r''$ are omitted since they are longer but very similar.\n\n\t\tWe analyze the integrability of each of the terms above (and those of $\\eps^2r''$). By Assumption~\\ref{ass:subexp}, $\\eps'$, $\\eps''$ and $\\eps'''$ are integrable, as well as $\\alpha'$, $\\alpha''$ and $\\alpha'''$, this justifies the integrability of most of the terms. We also need $\\frac{(\\eps')^2}{\\eps}$ and $\\frac{(\\eps')^3}{\\eps}$ to be integrable, which holds by Assumption~\\ref{ass:subexp}. Overall, $\\varphi$ is integrable on $\\R_{+}$.\n\t\\end{proof}","start":95362,"end":95908}]},{"id":"S036","semantic_type":"lemma","kind":"lemma","labels":["lem::remainderterm"],"declared_external":false,"source_only":false,"title":"平方修正项的首阶余项","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem::remainderterm}\n\t\tUnder Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp}, for all $s\\geq 0$, $$\\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2 = \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s)).$$\n\t\\end{lemma}","start":96010,"end":96329}]},{"id":"S037","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"平方余项引理的作者全证明","source_units":[{"raw_tex":"\\begin{proof}\n\t\tWe omit the time dependence on $s\\geq 0$ for the sake of readability. Using Assumption~\\ref{ass:smooth} we can define and expand the following quantity,\n\t\t\\begin{align*}\n\t\t\t&\\frac{1}{16}\\left(\\frac{2p'}{p^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps}}{(\\alpha+\\beta\\lambda)^{3/2}}\\right)^2\n\t\t\t=\\frac{\\lambda^2\\eps}{(\\alpha+\\beta\\lambda)^{3}}  - \\frac{p'\\lambda\\sqrt{\\eps}}{p^{3/2}(\\alpha+\\beta\\lambda)^{3/2}} + \\frac{(p')^2}{4p^{3}}\n\t\t\t\\\\&= \\frac{\\lambda^2\\eps}{(\\alpha+\\beta\\lambda)^{3}}\n\t\t\t\t- \\frac{\\lambda(\\alpha'\\eps - (\\alpha+\\beta\\lambda)\\eps')}{(\\alpha+\\beta\\lambda)^{3}}\n\t\t\t\t+ \\frac{(\\alpha')^2\\eps + \\frac{(\\eps')^2}{\\eps}(\\alpha+\\beta\\lambda)^2 - 2 \\alpha'\\eps'(\\alpha+\\beta\\lambda)}{4(\\alpha+\\beta\\lambda)^{3}}.\n\t\t\\end{align*}\n\t\tAssumption~\\ref{ass:subexp}, implies in particular that $\\vert\\eps'(t)\\vert=o(\\eps(t))$ and that $\\alpha'(t)\\to 0$, which we use in the equality above to obtain the desired conclusion.\n\t\\end{proof}","start":96331,"end":97273}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"展开平方并消去 p 记号","source_units":[{"raw_tex":"\\begin{proof}\n\t\tWe omit the time dependence on $s\\geq 0$ for the sake of readability. Using Assumption~\\ref{ass:smooth} we can define and expand the following quantity,\n\t\t\\begin{align*}\n\t\t\t&\\frac{1}{16}\\left(\\frac{2p'}{p^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps}}{(\\alpha+\\beta\\lambda)^{3/2}}\\right)^2\n\t\t\t=\\frac{\\lambda^2\\eps}{(\\alpha+\\beta\\lambda)^{3}}  - \\frac{p'\\lambda\\sqrt{\\eps}}{p^{3/2}(\\alpha+\\beta\\lambda)^{3/2}} + \\frac{(p')^2}{4p^{3}}\n\t\t\t\\\\&= \\frac{\\lambda^2\\eps}{(\\alpha+\\beta\\lambda)^{3}}\n\t\t\t\t- \\frac{\\lambda(\\alpha'\\eps - (\\alpha+\\beta\\lambda)\\eps')}{(\\alpha+\\beta\\lambda)^{3}}\n\t\t\t\t+ \\frac{(\\alpha')^2\\eps + \\frac{(\\eps')^2}{\\eps}(\\alpha+\\beta\\lambda)^2 - 2 \\alpha'\\eps'(\\alpha+\\beta\\lambda)}{4(\\alpha+\\beta\\lambda)^{3}}.\n\t\t\\end{align*}\n","start":96331,"end":97074}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"用相对导数衰减识别小余项","source_units":[{"raw_tex":"\t\tAssumption~\\ref{ass:subexp}, implies in particular that $\\vert\\eps'(t)\\vert=o(\\eps(t))$ and that $\\alpha'(t)\\to 0$, which we use in the equality above to obtain the desired conclusion.\n\t\\end{proof}","start":97074,"end":97273}]},{"id":"S040","semantic_type":"theorem","kind":"theorem","labels":["eq::deltabounds","eq::formofx","thm::MainResLG"],"declared_external":false,"source_only":false,"title":"标量轨迹的两模态表达定理","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm::MainResLG}\n\t\tSuppose that Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp} hold. There exists $A,B\\in\\R$ such that $x(0)=x_0$, $\\dx(0)=\\dx_0$ and for all $t\\geq 0$, the solution of \\eqref{eq::QuadeDINAVD} is\n\t\t\\begin{align}\\label{eq::formofx}\n\t\t\t\\begin{split}\n\t\t\t\tx(t) =&\n\t\t\t\tA\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\alpha(t)+\\beta\\lambda}}{\\sqrt{\\eps(t)}}\n\t\t\t\t\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\n\t\t\t\t\\\\\n\t\t\t\t+B&\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\n\t\t\t\t\\frac{\\sqrt{\\eps(t)}}{\\sqrt{\\alpha(t)+\\beta\\lambda}}\n\t\t\t\t\\\\\n\t\t\t\t&\\exp\\left(\\int_0^t -\\frac{\\alpha(s)+\\beta\\lambda}{\\eps(s)}+\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}+\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right),\n\t\t\t\\end{split}\n\t\t\\end{align}\n\t\twhere for all  $t\\geq 0$,\n\t\t\\begin{equation}\\label{eq::deltabounds}\n\t\t\t\\vert\\delta_1(t)\\vert \\leq \\exp\\left(\\frac{1}{2} \\int_0^t \\vert \\varphi(s)\\vert\\diff s\\right)-1\n\t\t\t\\quad \\text{and}\\quad\n\t\t\t\\vert\\delta_2(t)\\vert \\leq \\exp\\left(-\\frac{1}{2} \\int_t^{+\\infty} \\vert \\varphi(s)\\vert\\diff s\\right)-1.\n\t\t\\end{equation}\n\t\\end{theorem}","start":53595,"end":54786}]},{"id":"S041","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq::formofx","thm::MainResLG"],"declared_external":false,"source_only":false,"title":"慢模态与快模态的具体形式","source_units":[{"raw_tex":"\t\t\\begin{align}\\label{eq::formofx}\n\t\t\t\\begin{split}\n\t\t\t\tx(t) =&\n\t\t\t\tA\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\alpha(t)+\\beta\\lambda}}{\\sqrt{\\eps(t)}}\n\t\t\t\t\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\n\t\t\t\t\\\\\n\t\t\t\t+B&\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\n\t\t\t\t\\frac{\\sqrt{\\eps(t)}}{\\sqrt{\\alpha(t)+\\beta\\lambda}}\n\t\t\t\t\\\\\n\t\t\t\t&\\exp\\left(\\int_0^t -\\frac{\\alpha(s)+\\beta\\lambda}{\\eps(s)}+\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}+\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right),\n\t\t\t\\end{split}\n\t\t\\end{align}\n","start":53831,"end":54452}]},{"id":"S042","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq::deltabounds","thm::MainResLG"],"declared_external":false,"source_only":false,"title":"两误差函数的原文上界","source_units":[{"raw_tex":"\t\twhere for all  $t\\geq 0$,\n\t\t\\begin{equation}\\label{eq::deltabounds}\n\t\t\t\\vert\\delta_1(t)\\vert \\leq \\exp\\left(\\frac{1}{2} \\int_0^t \\vert \\varphi(s)\\vert\\diff s\\right)-1\n\t\t\t\\quad \\text{and}\\quad\n\t\t\t\\vert\\delta_2(t)\\vert \\leq \\exp\\left(-\\frac{1}{2} \\int_t^{+\\infty} \\vert \\varphi(s)\\vert\\diff s\\right)-1.\n\t\t\\end{equation}\n","start":54452,"end":54772}]},{"id":"S043","semantic_type":"proof","kind":"proof","labels":["eq::formofx0","eq::sqrtr"],"declared_external":false,"source_only":false,"title":"两模态表达定理的作者全证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm::MainResLG}]\n\t\tLet $x$ be the solution of \\eqref{eq::QuadeDINAVD} define $p,r$ as in \\eqref{eq::pr}. Let us also define $y(t)\\stackrel{\\textrm{def}}{=}x(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)$. According to Lemma~\\ref{lem::canonForm}, $r$ is positive and $y$ is solution to \\eqref{eq::ODErewritten}. Then, from Lemma~\\ref{lem::phiintegrable}, $\\int_t^{T} \\vert \\varphi(s)\\vert\\diff s<+\\infty$, so we can apply Theorem~\\ref{thm::olver} to $y$ on $[0,+\\infty[$. Therefore, there exists $A,B\\in\\R$, such that $\\forall t\\geq 0$,\n\t\t$$\n\t\ty(t)= A\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t \\sqrt{r(s)}\\diff s\\right) + B\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t -\\sqrt{r(s)}\\diff s\\right),\n\t\t$$\n\t\twhere $A$ and $B$ are determined by the initial conditions, and $\\delta_1$, $\\delta_2$ are such that \\eqref{eq::deltabounds} holds.\n\t\tGoing back to $x(t) = y(t)\\exp\\left(\\int_0^t -\\frac{p(s)}{2}\\diff s\\right)$, we obtain \tthat for all $t\\geq 0$,\n\t\t\\begin{multline}\\label{eq::formofx0}\n\t\t\tx(t) = A\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t -\\frac{p(s)}{2} +\\sqrt{r(s)}\\diff s\\right) \\\\ + B\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t -\\frac{p(s)}{2} -\\sqrt{r(s)}\\diff s\\right).\n\t\t\\end{multline}\n\t\tIt now remains to expand the terms in the two exponentials in \\eqref{eq::formofx0} in order to obtain \\eqref{eq::formofx}. To this aim, we approximate $\\sqrt{r(s)}$, let $s\\geq 0$,\n\t\t\\begin{align}\\label{eq::sqrtr}\n\t\t\t\\begin{split}\n\t\t\t\t\\sqrt{r(s)}&=\\frac{p(s)}{2}\\sqrt{1+ \\frac{2p'(s)}{p(s)^2} - \\frac{4\\lambda}{\\eps(s)p(s)^2}}\n\t\t\t\t\\\\&= \\frac{p(s)}{2}\\left(1 + \\frac{p'(s)}{p(s)^2} - \\frac{2\\lambda}{\\eps(s)p(s)^2} - \\frac{1}{8}\\left(\\frac{2p'(s)}{p(s)^2} - \\frac{4\\lambda}{\\eps(s)p(s)^2}\\right)^2 + o(\\eps(s)^2) \\right)\n\t\t\t\t\\\\ & = \\frac{p(s)}{2} +\\frac{ p'(s)}{2p(s)} - \\frac{\\lambda}{\\eps(s)p(s)} - \\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda}{\\eps(s)p(s)^{3/2}}\\right)^2 + o(\\eps(s))\n\t\t\t\t\\\\  = \\frac{p(s)}{2} &+ \\frac{p'(s)\\eps(s)}{2(\\alpha(s)+\\beta\\lambda)} - \\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2 + o(\\eps(s))\n\t\t\t\t\\\\  = \\frac{p(s)}{2} &+ \\frac{\\alpha'(s)/2-\\lambda}{\\alpha(s)+\\beta\\lambda} -\\frac{\\eps'(s)}{2\\eps(s)}  - \\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2 + o(\\eps(s))\n\t\t\t\\end{split}\n\t\t\\end{align}\n\t\tTo ease the readability, we denote $h(t)=\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2$. Focusing on the first exponential term in \\eqref{eq::formofx0}, we deduce from \\eqref{eq::sqrtr} that for all $t\\geq 0$,\n\t\t\\begin{align*}%\\label{eq::Aexpterm}\n\t\t\t\\begin{split}\n\t\t\t\t&\\exp\\left(\\int_0^t -\\frac{p(s)}{2} +\\sqrt{r(s)}\\diff s\\right)\n\t\t\t\t\\\\&= \\exp\\left(\\int_0^t \\frac{\\alpha'(s)/2}{\\alpha(s)+\\beta\\lambda} -\\frac{\\eps'(s)}{2\\eps(s)} -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{1}{16}h(t) + o(\\eps(s))\\diff s\\right)\n\t\t\t\t\\\\&=  \\frac{\\sqrt{\\alpha(t)+\\beta\\lambda})}{\\sqrt{\\alpha_0+\\beta\\lambda}} \\frac{\\sqrt{\\eps_0}}{\\sqrt{\\eps(t)}} \\exp\\left(\n\t\t\t\t\\int_0^t \\frac{-\\lambda}{\\alpha(s)+\\beta\\lambda} -\n\t\t\t\t\\frac{1}{16}h(t) + o(\\eps(s))\\diff s\\right)\n\t\t\t\t\\\\&=  \\frac{\\sqrt{\\alpha(t)+\\beta\\lambda})}{\\sqrt{\\alpha_0+\\beta\\lambda}} \\frac{\\sqrt{\\eps_0}}{\\sqrt{\\eps(t)}} \\exp\\left(\n\t\t\t\t\\int_0^t \\frac{-\\lambda}{\\alpha(s)+\\beta\\lambda} -\n\t\t\t\t\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}} + o(\\eps(s))\\diff s\\right),\n\t\t\t\\end{split}\n\t\t\\end{align*}\n\t\twhere the last line relies on further computations postponed to Lemma~\\ref{lem::remainderterm} in Appendix~\\ref{app::integrability}. Performing the exact same type of computations on $\\exp\\left(\\int_0^t -\\frac{p(s)}{2} -\\sqrt{r(s)}\\diff s\\right)$, and up to redefining $A$ and $B$ so as to encompass all the constants, we obtain \\eqref{eq::formofx} and the result is proved.\n\t\\end{proof}","start":55049,"end":59028}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"对规范形应用全局 Olver 定理","source_units":[{"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm::MainResLG}]\n\t\tLet $x$ be the solution of \\eqref{eq::QuadeDINAVD} define $p,r$ as in \\eqref{eq::pr}. Let us also define $y(t)\\stackrel{\\textrm{def}}{=}x(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)$. According to Lemma~\\ref{lem::canonForm}, $r$ is positive and $y$ is solution to \\eqref{eq::ODErewritten}. Then, from Lemma~\\ref{lem::phiintegrable}, $\\int_t^{T} \\vert \\varphi(s)\\vert\\diff s<+\\infty$, so we can apply Theorem~\\ref{thm::olver} to $y$ on $[0,+\\infty[$. Therefore, there exists $A,B\\in\\R$, such that $\\forall t\\geq 0$,\n\t\t$$\n\t\ty(t)= A\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t \\sqrt{r(s)}\\diff s\\right) + B\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t -\\sqrt{r(s)}\\diff s\\right),\n\t\t$$\n\t\twhere $A$ and $B$ are determined by the initial conditions, and $\\delta_1$, $\\delta_2$ are such that \\eqref{eq::deltabounds} holds.\n","start":55049,"end":55937}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["eq::formofx0"],"declared_external":false,"source_only":false,"title":"逆变换得到未展开的双指数","source_units":[{"raw_tex":"\t\tGoing back to $x(t) = y(t)\\exp\\left(\\int_0^t -\\frac{p(s)}{2}\\diff s\\right)$, we obtain \tthat for all $t\\geq 0$,\n\t\t\\begin{multline}\\label{eq::formofx0}\n\t\t\tx(t) = A\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t -\\frac{p(s)}{2} +\\sqrt{r(s)}\\diff s\\right) \\\\ + B\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t -\\frac{p(s)}{2} -\\sqrt{r(s)}\\diff s\\right).\n\t\t\\end{multline}\n","start":55937,"end":56313}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["eq::sqrtr"],"declared_external":false,"source_only":false,"title":"对平方根系数作渐近展开","source_units":[{"raw_tex":"\t\tIt now remains to expand the terms in the two exponentials in \\eqref{eq::formofx0} in order to obtain \\eqref{eq::formofx}. To this aim, we approximate $\\sqrt{r(s)}$, let $s\\geq 0$,\n\t\t\\begin{align}\\label{eq::sqrtr}\n\t\t\t\\begin{split}\n\t\t\t\t\\sqrt{r(s)}&=\\frac{p(s)}{2}\\sqrt{1+ \\frac{2p'(s)}{p(s)^2} - \\frac{4\\lambda}{\\eps(s)p(s)^2}}\n\t\t\t\t\\\\&= \\frac{p(s)}{2}\\left(1 + \\frac{p'(s)}{p(s)^2} - \\frac{2\\lambda}{\\eps(s)p(s)^2} - \\frac{1}{8}\\left(\\frac{2p'(s)}{p(s)^2} - \\frac{4\\lambda}{\\eps(s)p(s)^2}\\right)^2 + o(\\eps(s)^2) \\right)\n\t\t\t\t\\\\ & = \\frac{p(s)}{2} +\\frac{ p'(s)}{2p(s)} - \\frac{\\lambda}{\\eps(s)p(s)} - \\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda}{\\eps(s)p(s)^{3/2}}\\right)^2 + o(\\eps(s))\n\t\t\t\t\\\\  = \\frac{p(s)}{2} &+ \\frac{p'(s)\\eps(s)}{2(\\alpha(s)+\\beta\\lambda)} - \\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2 + o(\\eps(s))\n\t\t\t\t\\\\  = \\frac{p(s)}{2} &+ \\frac{\\alpha'(s)/2-\\lambda}{\\alpha(s)+\\beta\\lambda} -\\frac{\\eps'(s)}{2\\eps(s)}  - \\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2 + o(\\eps(s))\n\t\t\t\\end{split}\n\t\t\\end{align}\n","start":56313,"end":57535}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"积分对数导数并代入平方余项","source_units":[{"raw_tex":"\t\tTo ease the readability, we denote $h(t)=\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2$. Focusing on the first exponential term in \\eqref{eq::formofx0}, we deduce from \\eqref{eq::sqrtr} that for all $t\\geq 0$,\n\t\t\\begin{align*}%\\label{eq::Aexpterm}\n\t\t\t\\begin{split}\n\t\t\t\t&\\exp\\left(\\int_0^t -\\frac{p(s)}{2} +\\sqrt{r(s)}\\diff s\\right)\n\t\t\t\t\\\\&= \\exp\\left(\\int_0^t \\frac{\\alpha'(s)/2}{\\alpha(s)+\\beta\\lambda} -\\frac{\\eps'(s)}{2\\eps(s)} -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{1}{16}h(t) + o(\\eps(s))\\diff s\\right)\n\t\t\t\t\\\\&=  \\frac{\\sqrt{\\alpha(t)+\\beta\\lambda})}{\\sqrt{\\alpha_0+\\beta\\lambda}} \\frac{\\sqrt{\\eps_0}}{\\sqrt{\\eps(t)}} \\exp\\left(\n\t\t\t\t\\int_0^t \\frac{-\\lambda}{\\alpha(s)+\\beta\\lambda} -\n\t\t\t\t\\frac{1}{16}h(t) + o(\\eps(s))\\diff s\\right)\n\t\t\t\t\\\\&=  \\frac{\\sqrt{\\alpha(t)+\\beta\\lambda})}{\\sqrt{\\alpha_0+\\beta\\lambda}} \\frac{\\sqrt{\\eps_0}}{\\sqrt{\\eps(t)}} \\exp\\left(\n\t\t\t\t\\int_0^t \\frac{-\\lambda}{\\alpha(s)+\\beta\\lambda} -\n\t\t\t\t\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}} + o(\\eps(s))\\diff s\\right),\n\t\t\t\\end{split}\n\t\t\\end{align*}\n","start":57535,"end":58639}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"处理第二模态并吸收常数","source_units":[{"raw_tex":"\t\twhere the last line relies on further computations postponed to Lemma~\\ref{lem::remainderterm} in Appendix~\\ref{app::integrability}. Performing the exact same type of computations on $\\exp\\left(\\int_0^t -\\frac{p(s)}{2} -\\sqrt{r(s)}\\diff s\\right)$, and up to redefining $A$ and $B$ so as to encompass all the constants, we obtain \\eqref{eq::formofx} and the result is proved.\n\t\\end{proof}","start":58639,"end":59028}]},{"id":"S049","semantic_type":"theorem","kind":"theorem","labels":["eq::equivx","thm::MainResAsymptotic"],"declared_external":false,"source_only":false,"title":"质量与阻尼控制的渐近比较定理","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm::MainResAsymptotic}\n\t\tLet $x$ be the solution of \\eqref{eq::QuadeDINAVD}, given in \\eqref{eq::formofx}, and $x_{LM}$ and $x_N$ whose closed forms are stated in \\eqref{eq::closeformLMN}. Under Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp}, there exists $C>0$ such that the following asymptotic equivalences hold:\n\t\t\\begin{align}\\label{eq::equivx}\n\t\t\t\\begin{split}\n\t\t\t\tx(t) &\\sim_{+\\infty}\n\t\t\t\tx_{LM}(t)C\\exp\\left(\\int_0^t - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right),\\quad \\text{and}\n\t\t\t\t\\\\x(t)&\\sim_{+\\infty}\n\t\t\t\tx_{N}(t)C\\exp\\left(\\int_0^t \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}- \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right).\n\t\t\t\\end{split}\n\t\t\\end{align}\n\t\tAs a consequence, the convergence of $x$ to $x^\\star$ is:\n\t\t\\begin{enumerate}[label=(\\roman*)]\n\t\t\t\\item \\emph{Faster} than that of $x_{LM}$\n\t\t\tif $\\eps$ is non-integrable and as fast otherwise.\n\t\t\t\\item Slower than that of $x_{N}$ if $\\alpha$ is non-integrable and as fast if $\\alpha$ is integrable, in the case where $\\forall t\\geq 0$, $\\alpha(t)>\\eps(t)$.\n\t\t\t\\item \\emph{Faster} than that of $x_{N}$ if $\\eps$ is non-integrable and as fast if $\\eps$ is integrable, in the case where $\\forall t\\geq 0$, $\\alpha(t)< \\eps(t)$.\n\t\t\\end{enumerate}\n\t\\end{theorem}","start":59872,"end":61197}]},{"id":"S050","semantic_type":"proof","kind":"proof","labels":["eq::rewritteLM"],"declared_external":false,"source_only":false,"title":"渐近比较主定理的作者全证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm::MainResAsymptotic}]\n\t\tThanks to Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp}, Theorem~\\ref{thm::MainResLG} tells us that $x$ has the form \\eqref{eq::formofx}. We now analyze the two terms in \\eqref{eq::formofx}.\n\n\t\tFirst, we know from Theorem~\\ref{thm::MainResLG} that $\\delta_1(0)=0$ and $\\lim_{t\\to+\\infty}\\delta_2(t)=0$. In addition, by Lemma~\\ref{lem::phiintegrable}, $\\delta_1$ and $\\delta_2$ are uniformly bounded by some positive constant.\n\t\tThen $r(t)^{-1/4}$ decays asymptotically like $\\sqrt{\\eps(t)}$ and $\\alpha$ is bounded. So\n\t\t$A\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\alpha(t)+\\beta\\lambda}}{\\sqrt{\\eps(t)}}$ is asymptotically equivalent to some constant $c_1\\in\\R$ as $t\\to +\\infty$. Similarly, the factor $B\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\eps(t)}}{\\sqrt{\\alpha(t)+\\beta\\lambda}}$ is equivalent to $c_2\\eps(t)$, with $c_2\\in\\R$.\n\n\t\tWe now analyze the ``exponential factors'' in \\eqref{eq::formofx}.\n\t\tOn the one hand, $\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}+\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))$ converges to $\\frac{1}{\\beta}$ as $s\\to \\infty$, while $\\frac{\\alpha(s)+\\beta\\lambda}{\\eps(s)}$ diverges to $+\\infty$. Therefore, we deduce that,\n\t\t\\begin{multline*}\n\t\t\t\\exp\\left(\\int_0^t -\\frac{\\alpha(s)+\\beta\\lambda}{\\eps(s)}+\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}+\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\n\t\t\t\\\\= o\\left(\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\\right).\n\t\t\\end{multline*}\n\t\tAs a consequence, the second term in \\eqref{eq::formofx} will decrease to $0$ faster than the first-one (let alone the additional $\\eps(t)$ decay that we have just discussed). The asymptotic behavior of $x$ will thus be governed by the first term in \\eqref{eq::formofx}.\n\n\t\tLet us now focus on the first term in \\eqref{eq::formofx}. Observe that\n\t\t$\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}\\diff s\\right)$ is exactly the decay of $x_{LM}$ in \\eqref{eq::closeformLMN}. Thus, we have proved that there exists $C>0$, such that the following asymptotic equivalence holds,\n\t\t\\begin{multline*}%\\label{eq::equivA}\n\t\t\tA\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\alpha(t)+\\beta\\lambda}}{\\sqrt{\\eps(t)}}\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\n\t\t\t\\\\ \\sim_{+\\infty}\n\t\t\tx_{LM}(t)C\\exp\\left(\\int_0^t - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right),\n\t\t\\end{multline*}\n\t\twhich proves the first part of \\eqref{eq::equivx}. The second equivalence in \\eqref{eq::equivx} is obtained using the following identity,\n\t\t\\begin{equation}\\label{eq::rewritteLM}\n\t\t\t\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}\\diff s = \\int_0^t -\\frac{1}{\\beta} + \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}\\diff s = -\\frac{t}{\\beta} + \\int_0^t  \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}\\diff s\n\t\t\\end{equation}\n\t\tand $e^{-t/\\beta}$ is precisely the rate at which $x_N$ decreases. So \\eqref{eq::equivx} holds.\n\n\t\tIt finally remains to deduce the conclusions of the theorem from \\eqref{eq::equivx}.\n\t\t\\\\--\\ Regarding the comparison with $x_{LM}$, the integral $\\int_0^t - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s$ converges if and only if $\\eps$ is integrable on $[0,+\\infty[$, and diverges to $-\\infty$ when $\\eps$ is not. So $x$ converges to $0$ at least as fast as $x_{LM}$ and faster when $\\eps$ is not integrable.\n\t\t\\\\-- As for the comparison with $x_N$, if $\\alpha(s)>\\eps(s)\\geq 0$ for all $s\\geq 0$, then the integral $\\int_0^t \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}- \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s$ is convergent in $+\\infty$ if and only if $\\alpha$ is integrable and diverges to $+\\infty$ when $\\alpha$ is non-integrable. So when $\\alpha$ is integrable, the speed of convergence of $x$ is the same as that of $x_N$. When $\\alpha$ is not integrable, the convergence to $0$ is slower but still holds. Indeed, for all $s\\geq 0$\n\t\t$\\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\frac{1}{\\beta})}< \\frac{\\alpha(s)}{\\beta\\alpha(s)}= \\frac{1}{\\beta}$. Thus for all $t>0$, $-\\frac{t}{\\beta} + \\int_0^t \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}\\diff s<0$.\n\t\t\\\\-- Finally, the comparison with $x_N$ in the case $\\eps(s)> \\alpha(s)$ is exactly the same as the comparison with $x_{LM}$ using \\eqref{eq::rewritteLM}.\n\t\\end{proof}","start":62511,"end":67064}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"调用两模态轨迹表达","source_units":[{"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm::MainResAsymptotic}]\n\t\tThanks to Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp}, Theorem~\\ref{thm::MainResLG} tells us that $x$ has the form \\eqref{eq::formofx}. We now analyze the two terms in \\eqref{eq::formofx}.\n","start":62511,"end":62769}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"误差函数与前因子的渐近判断","source_units":[{"raw_tex":"\t\tFirst, we know from Theorem~\\ref{thm::MainResLG} that $\\delta_1(0)=0$ and $\\lim_{t\\to+\\infty}\\delta_2(t)=0$. In addition, by Lemma~\\ref{lem::phiintegrable}, $\\delta_1$ and $\\delta_2$ are uniformly bounded by some positive constant.\n\t\tThen $r(t)^{-1/4}$ decays asymptotically like $\\sqrt{\\eps(t)}$ and $\\alpha$ is bounded. So\n\t\t$A\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\alpha(t)+\\beta\\lambda}}{\\sqrt{\\eps(t)}}$ is asymptotically equivalent to some constant $c_1\\in\\R$ as $t\\to +\\infty$. Similarly, the factor $B\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\eps(t)}}{\\sqrt{\\alpha(t)+\\beta\\lambda}}$ is equivalent to $c_2\\eps(t)$, with $c_2\\in\\R$.\n","start":62770,"end":63421}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"比较指数并宣布首模态主导","source_units":[{"raw_tex":"\t\tWe now analyze the ``exponential factors'' in \\eqref{eq::formofx}.\n\t\tOn the one hand, $\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}+\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))$ converges to $\\frac{1}{\\beta}$ as $s\\to \\infty$, while $\\frac{\\alpha(s)+\\beta\\lambda}{\\eps(s)}$ diverges to $+\\infty$. Therefore, we deduce that,\n\t\t\\begin{multline*}\n\t\t\t\\exp\\left(\\int_0^t -\\frac{\\alpha(s)+\\beta\\lambda}{\\eps(s)}+\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}+\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\n\t\t\t\\\\= o\\left(\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\\right).\n\t\t\\end{multline*}\n\t\tAs a consequence, the second term in \\eqref{eq::formofx} will decrease to $0$ faster than the first-one (let alone the additional $\\eps(t)$ decay that we have just discussed). The asymptotic behavior of $x$ will thus be governed by the first term in \\eqref{eq::formofx}.\n","start":63422,"end":64415}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"识别 LM 因子并提出正比较常数","source_units":[{"raw_tex":"\t\tLet us now focus on the first term in \\eqref{eq::formofx}. Observe that\n\t\t$\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}\\diff s\\right)$ is exactly the decay of $x_{LM}$ in \\eqref{eq::closeformLMN}. Thus, we have proved that there exists $C>0$, such that the following asymptotic equivalence holds,\n\t\t\\begin{multline*}%\\label{eq::equivA}\n\t\t\tA\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\alpha(t)+\\beta\\lambda}}{\\sqrt{\\eps(t)}}\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\n\t\t\t\\\\ \\sim_{+\\infty}\n\t\t\tx_{LM}(t)C\\exp\\left(\\int_0^t - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right),\n\t\t\\end{multline*}\n","start":64416,"end":65149}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["eq::rewritteLM"],"declared_external":false,"source_only":false,"title":"分离 CN 衰减指数","source_units":[{"raw_tex":"\t\twhich proves the first part of \\eqref{eq::equivx}. The second equivalence in \\eqref{eq::equivx} is obtained using the following identity,\n\t\t\\begin{equation}\\label{eq::rewritteLM}\n\t\t\t\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}\\diff s = \\int_0^t -\\frac{1}{\\beta} + \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}\\diff s = -\\frac{t}{\\beta} + \\int_0^t  \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}\\diff s\n\t\t\\end{equation}\n\t\tand $e^{-t/\\beta}$ is precisely the rate at which $x_N$ decreases. So \\eqref{eq::equivx} holds.\n","start":65149,"end":65675}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"按质量可积性比较 LM 速度","source_units":[{"raw_tex":"\t\tIt finally remains to deduce the conclusions of the theorem from \\eqref{eq::equivx}.\n\t\t\\\\--\\ Regarding the comparison with $x_{LM}$, the integral $\\int_0^t - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s$ converges if and only if $\\eps$ is integrable on $[0,+\\infty[$, and diverges to $-\\infty$ when $\\eps$ is not. So $x$ converges to $0$ at least as fast as $x_{LM}$ and faster when $\\eps$ is not integrable.\n","start":65676,"end":66114}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"阻尼较大时的 CN 分类","source_units":[{"raw_tex":"\t\t\\\\-- As for the comparison with $x_N$, if $\\alpha(s)>\\eps(s)\\geq 0$ for all $s\\geq 0$, then the integral $\\int_0^t \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}- \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s$ is convergent in $+\\infty$ if and only if $\\alpha$ is integrable and diverges to $+\\infty$ when $\\alpha$ is non-integrable. So when $\\alpha$ is integrable, the speed of convergence of $x$ is the same as that of $x_N$. When $\\alpha$ is not integrable, the convergence to $0$ is slower but still holds. Indeed, for all $s\\geq 0$\n\t\t$\\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\frac{1}{\\beta})}< \\frac{\\alpha(s)}{\\beta\\alpha(s)}= \\frac{1}{\\beta}$. Thus for all $t>0$, $-\\frac{t}{\\beta} + \\int_0^t \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}\\diff s<0$.\n","start":66114,"end":66895}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"质量较大时的 CN 分类","source_units":[{"raw_tex":"\t\t\\\\-- Finally, the comparison with $x_N$ in the case $\\eps(s)> \\alpha(s)$ is exactly the same as the comparison with $x_{LM}$ using \\eqref{eq::rewritteLM}.\n","start":66895,"end":67052}]},{"id":"S059","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表：Standard spectral decomposition for symmetric positive definite matrices。"],"source_only":false,"title":"对称正定矩阵的谱分解","source_units":[]},{"id":"S060","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表：linear ODE theory。"],"source_only":false,"title":"线性常微分方程基本理论","source_units":[]},{"id":"S061","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表中的 elementary asymptotic calculus；基本微积分规则由标注者显式列出。"],"source_only":false,"title":"微分积分与泰勒计算规则","source_units":[]},{"id":"S062","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表：elementary asymptotic calculus。"],"source_only":false,"title":"指数与积分的渐近比较规则","source_units":[]},{"id":"S063","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；用于原文 L548、L1003、L1022 的基本积分论操作。"],"source_only":false,"title":"局部积分与有界乘子的控制","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S049","edges":[{"source_id":"S001","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S059","target_id":"S008","kind":"supports"},{"source_id":"S002","target_id":"S009","kind":"supports"},{"source_id":"S003","target_id":"S009","kind":"supports"},{"source_id":"S007","target_id":"S009","kind":"supports"},{"source_id":"S059","target_id":"S009","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S005","target_id":"S010","kind":"supports"},{"source_id":"S060","target_id":"S010","kind":"supports"},{"source_id":"S004","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S014","kind":"supports"},{"source_id":"S063","target_id":"S014","kind":"supports"},{"source_id":"S008","target_id":"S021","kind":"supports"},{"source_id":"S016","target_id":"S021","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S005","target_id":"S021","kind":"supports"},{"source_id":"S022","target_id":"S024","kind":"supports"},{"source_id":"S006","target_id":"S024","kind":"supports"},{"source_id":"S016","target_id":"S024","kind":"supports"},{"source_id":"S008","target_id":"S025","kind":"supports"},{"source_id":"S022","target_id":"S025","kind":"supports"},{"source_id":"S061","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S017","kind":"supports"},{"source_id":"S022","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S026","kind":"supports"},{"source_id":"S016","target_id":"S026","kind":"supports"},{"source_id":"S018","target_id":"S026","kind":"supports"},{"source_id":"S014","target_id":"S028","kind":"supports"},{"source_id":"S022","target_id":"S028","kind":"supports"},{"source_id":"S016","target_id":"S028","kind":"supports"},{"source_id":"S018","target_id":"S028","kind":"supports"},{"source_id":"S061","target_id":"S028","kind":"supports"},{"source_id":"S022","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S018","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S006","target_id":"S031","kind":"supports"},{"source_id":"S029","target_id":"S032","kind":"supports"},{"source_id":"S028","target_id":"S032","kind":"supports"},{"source_id":"S022","target_id":"S032","kind":"supports"},{"source_id":"S061","target_id":"S032","kind":"supports"},{"source_id":"S017","target_id":"S033","kind":"supports"},{"source_id":"S031","target_id":"S033","kind":"supports"},{"source_id":"S006","target_id":"S033","kind":"supports"},{"source_id":"S063","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S028","target_id":"S034","kind":"supports"},{"source_id":"S022","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S018","target_id":"S035","kind":"supports"},{"source_id":"S006","target_id":"S035","kind":"supports"},{"source_id":"S063","target_id":"S035","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S027","kind":"supports"},{"source_id":"S031","target_id":"S027","kind":"supports"},{"source_id":"S032","target_id":"S027","kind":"supports"},{"source_id":"S033","target_id":"S027","kind":"supports"},{"source_id":"S034","target_id":"S027","kind":"supports"},{"source_id":"S035","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S022","target_id":"S036","kind":"supports"},{"source_id":"S016","target_id":"S036","kind":"supports"},{"source_id":"S018","target_id":"S036","kind":"supports"},{"source_id":"S022","target_id":"S038","kind":"supports"},{"source_id":"S028","target_id":"S038","kind":"supports"},{"source_id":"S016","target_id":"S038","kind":"supports"},{"source_id":"S061","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S018","target_id":"S039","kind":"supports"},{"source_id":"S030","target_id":"S039","kind":"supports"},{"source_id":"S006","target_id":"S039","kind":"supports"},{"source_id":"S061","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S008","target_id":"S040","kind":"supports"},{"source_id":"S016","target_id":"S040","kind":"supports"},{"source_id":"S018","target_id":"S040","kind":"supports"},{"source_id":"S005","target_id":"S040","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S042","target_id":"S040","kind":"supports"},{"source_id":"S008","target_id":"S044","kind":"supports"},{"source_id":"S022","target_id":"S044","kind":"supports"},{"source_id":"S021","target_id":"S044","kind":"supports"},{"source_id":"S026","target_id":"S044","kind":"supports"},{"source_id":"S012","target_id":"S044","kind":"supports"},{"source_id":"S060","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S022","target_id":"S045","kind":"supports"},{"source_id":"S022","target_id":"S046","kind":"supports"},{"source_id":"S018","target_id":"S046","kind":"supports"},{"source_id":"S030","target_id":"S046","kind":"supports"},{"source_id":"S061","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S022","target_id":"S047","kind":"supports"},{"source_id":"S036","target_id":"S047","kind":"supports"},{"source_id":"S061","target_id":"S047","kind":"supports"},{"source_id":"S045","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S036","target_id":"S048","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"},{"source_id":"S046","target_id":"S043","kind":"supports"},{"source_id":"S047","target_id":"S043","kind":"supports"},{"source_id":"S048","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S040","kind":"supports"},{"source_id":"S043","target_id":"S041","kind":"supports"},{"source_id":"S044","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S019","kind":"supports"},{"source_id":"S026","target_id":"S019","kind":"supports"},{"source_id":"S008","target_id":"S049","kind":"supports"},{"source_id":"S010","target_id":"S049","kind":"supports"},{"source_id":"S016","target_id":"S049","kind":"supports"},{"source_id":"S018","target_id":"S049","kind":"supports"},{"source_id":"S020","target_id":"S049","kind":"supports"},{"source_id":"S041","target_id":"S049","kind":"supports"},{"source_id":"S016","target_id":"S051","kind":"supports"},{"source_id":"S018","target_id":"S051","kind":"supports"},{"source_id":"S040","target_id":"S051","kind":"supports"},{"source_id":"S041","target_id":"S052","kind":"supports"},{"source_id":"S042","target_id":"S052","kind":"supports"},{"source_id":"S026","target_id":"S052","kind":"supports"},{"source_id":"S031","target_id":"S052","kind":"supports"},{"source_id":"S006","target_id":"S052","kind":"supports"},{"source_id":"S062","target_id":"S052","kind":"supports"},{"source_id":"S041","target_id":"S053","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S018","target_id":"S053","kind":"supports"},{"source_id":"S006","target_id":"S053","kind":"supports"},{"source_id":"S062","target_id":"S053","kind":"supports"},{"source_id":"S041","target_id":"S054","kind":"supports"},{"source_id":"S052","target_id":"S054","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S010","target_id":"S054","kind":"supports"},{"source_id":"S005","target_id":"S054","kind":"supports"},{"source_id":"S062","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S010","target_id":"S055","kind":"supports"},{"source_id":"S061","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S056","kind":"supports"},{"source_id":"S006","target_id":"S056","kind":"supports"},{"source_id":"S018","target_id":"S056","kind":"supports"},{"source_id":"S062","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S057","kind":"supports"},{"source_id":"S006","target_id":"S057","kind":"supports"},{"source_id":"S018","target_id":"S057","kind":"supports"},{"source_id":"S062","target_id":"S057","kind":"supports"},{"source_id":"S055","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S058","kind":"supports"},{"source_id":"S018","target_id":"S058","kind":"supports"},{"source_id":"S062","target_id":"S058","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S050","kind":"supports"},{"source_id":"S054","target_id":"S050","kind":"supports"},{"source_id":"S055","target_id":"S050","kind":"supports"},{"source_id":"S056","target_id":"S050","kind":"supports"},{"source_id":"S057","target_id":"S050","kind":"supports"},{"source_id":"S058","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S049","edges":[{"source_id":"S001","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S059","target_id":"S008","kind":"supports"},{"source_id":"S002","target_id":"S009","kind":"supports"},{"source_id":"S003","target_id":"S009","kind":"supports"},{"source_id":"S007","target_id":"S009","kind":"supports"},{"source_id":"S059","target_id":"S009","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S005","target_id":"S010","kind":"supports"},{"source_id":"S060","target_id":"S010","kind":"supports"},{"source_id":"S004","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S012","target_id":"S014","kind":"supports"},{"source_id":"S063","target_id":"S014","kind":"supports"},{"source_id":"S008","target_id":"S021","kind":"supports"},{"source_id":"S016","target_id":"S021","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S005","target_id":"S021","kind":"supports"},{"source_id":"S022","target_id":"S024","kind":"supports"},{"source_id":"S006","target_id":"S024","kind":"supports"},{"source_id":"S016","target_id":"S024","kind":"supports"},{"source_id":"S008","target_id":"S025","kind":"supports"},{"source_id":"S022","target_id":"S025","kind":"supports"},{"source_id":"S061","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S017","kind":"supports"},{"source_id":"S022","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S026","kind":"supports"},{"source_id":"S016","target_id":"S026","kind":"supports"},{"source_id":"S018","target_id":"S026","kind":"supports"},{"source_id":"S014","target_id":"S028","kind":"supports"},{"source_id":"S022","target_id":"S028","kind":"supports"},{"source_id":"S016","target_id":"S028","kind":"supports"},{"source_id":"S018","target_id":"S028","kind":"supports"},{"source_id":"S061","target_id":"S028","kind":"supports"},{"source_id":"S022","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S018","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S006","target_id":"S031","kind":"supports"},{"source_id":"S029","target_id":"S032","kind":"supports"},{"source_id":"S028","target_id":"S032","kind":"supports"},{"source_id":"S022","target_id":"S032","kind":"supports"},{"source_id":"S061","target_id":"S032","kind":"supports"},{"source_id":"S017","target_id":"S033","kind":"supports"},{"source_id":"S031","target_id":"S033","kind":"supports"},{"source_id":"S006","target_id":"S033","kind":"supports"},{"source_id":"S063","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S028","target_id":"S034","kind":"supports"},{"source_id":"S022","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S018","target_id":"S035","kind":"supports"},{"source_id":"S006","target_id":"S035","kind":"supports"},{"source_id":"S063","target_id":"S035","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S027","kind":"supports"},{"source_id":"S031","target_id":"S027","kind":"supports"},{"source_id":"S032","target_id":"S027","kind":"supports"},{"source_id":"S033","target_id":"S027","kind":"supports"},{"source_id":"S034","target_id":"S027","kind":"supports"},{"source_id":"S035","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S022","target_id":"S036","kind":"supports"},{"source_id":"S016","target_id":"S036","kind":"supports"},{"source_id":"S018","target_id":"S036","kind":"supports"},{"source_id":"S022","target_id":"S038","kind":"supports"},{"source_id":"S028","target_id":"S038","kind":"supports"},{"source_id":"S016","target_id":"S038","kind":"supports"},{"source_id":"S061","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S018","target_id":"S039","kind":"supports"},{"source_id":"S030","target_id":"S039","kind":"supports"},{"source_id":"S006","target_id":"S039","kind":"supports"},{"source_id":"S061","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S008","target_id":"S040","kind":"supports"},{"source_id":"S016","target_id":"S040","kind":"supports"},{"source_id":"S018","target_id":"S040","kind":"supports"},{"source_id":"S005","target_id":"S040","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S042","target_id":"S040","kind":"supports"},{"source_id":"S008","target_id":"S044","kind":"supports"},{"source_id":"S022","target_id":"S044","kind":"supports"},{"source_id":"S021","target_id":"S044","kind":"supports"},{"source_id":"S026","target_id":"S044","kind":"supports"},{"source_id":"S012","target_id":"S044","kind":"supports"},{"source_id":"S060","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S022","target_id":"S045","kind":"supports"},{"source_id":"S022","target_id":"S046","kind":"supports"},{"source_id":"S018","target_id":"S046","kind":"supports"},{"source_id":"S030","target_id":"S046","kind":"supports"},{"source_id":"S061","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S022","target_id":"S047","kind":"supports"},{"source_id":"S036","target_id":"S047","kind":"supports"},{"source_id":"S061","target_id":"S047","kind":"supports"},{"source_id":"S045","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S036","target_id":"S048","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"},{"source_id":"S046","target_id":"S043","kind":"supports"},{"source_id":"S047","target_id":"S043","kind":"supports"},{"source_id":"S048","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S040","kind":"supports"},{"source_id":"S043","target_id":"S041","kind":"supports"},{"source_id":"S044","target_id":"S042","kind":"supports"},{"source_id":"S008","target_id":"S049","kind":"supports"},{"source_id":"S010","target_id":"S049","kind":"supports"},{"source_id":"S016","target_id":"S049","kind":"supports"},{"source_id":"S018","target_id":"S049","kind":"supports"},{"source_id":"S020","target_id":"S049","kind":"supports"},{"source_id":"S041","target_id":"S049","kind":"supports"},{"source_id":"S016","target_id":"S051","kind":"supports"},{"source_id":"S018","target_id":"S051","kind":"supports"},{"source_id":"S040","target_id":"S051","kind":"supports"},{"source_id":"S041","target_id":"S052","kind":"supports"},{"source_id":"S042","target_id":"S052","kind":"supports"},{"source_id":"S026","target_id":"S052","kind":"supports"},{"source_id":"S031","target_id":"S052","kind":"supports"},{"source_id":"S006","target_id":"S052","kind":"supports"},{"source_id":"S062","target_id":"S052","kind":"supports"},{"source_id":"S041","target_id":"S053","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S018","target_id":"S053","kind":"supports"},{"source_id":"S006","target_id":"S053","kind":"supports"},{"source_id":"S062","target_id":"S053","kind":"supports"},{"source_id":"S041","target_id":"S054","kind":"supports"},{"source_id":"S052","target_id":"S054","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S010","target_id":"S054","kind":"supports"},{"source_id":"S005","target_id":"S054","kind":"supports"},{"source_id":"S062","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S010","target_id":"S055","kind":"supports"},{"source_id":"S061","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S056","kind":"supports"},{"source_id":"S006","target_id":"S056","kind":"supports"},{"source_id":"S018","target_id":"S056","kind":"supports"},{"source_id":"S062","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S057","kind":"supports"},{"source_id":"S006","target_id":"S057","kind":"supports"},{"source_id":"S018","target_id":"S057","kind":"supports"},{"source_id":"S062","target_id":"S057","kind":"supports"},{"source_id":"S055","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S058","kind":"supports"},{"source_id":"S018","target_id":"S058","kind":"supports"},{"source_id":"S062","target_id":"S058","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S050","kind":"supports"},{"source_id":"S054","target_id":"S050","kind":"supports"},{"source_id":"S055","target_id":"S050","kind":"supports"},{"source_id":"S056","target_id":"S050","kind":"supports"},{"source_id":"S057","target_id":"S050","kind":"supports"},{"source_id":"S058","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq::VMDINAVD"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"变质量牛顿型动力系统","source_units":[{"raw_tex":"\tBoth steps are highly challenging, our work focuses on the ODE matter. We study the following second-order dynamical system in a general setting:\n\t\\begin{equation}\\tag{VM-DIN-AVD}\\label{eq::VMDINAVD}\n\t\t\\varepsilon(t) \\ddot{x}(t) + \\alpha(t) \\dot x(t) + \\beta \\Hf(x(t))\\dot{x}(t)+\\gf(x(t))=0,\n\t\t\\quad t\\geq 0,\n\t\\end{equation}\n\twhere $f\\colon \\R^n\\to\\R$ is a smooth function, with gradient $\\gf$ and Hessian $\\Hf$ defined on $\\R^n$ equipped with scalar product $\\langle\\cdot,\\cdot\\rangle$, and induced norm $\\norm{\\cdot}$.\n\tThe system is controlled by two functions $\\eps,\\alpha\\colon\\R_+\\to\\R_+$ (where $\\R_+=[0,+\\infty[$) and a parameter $\\beta> 0$ that define the type of dynamics that drives the trajectory (or solution) $x\\colon\\R_+\\to\\R^n$, whose first- and second-order derivatives are denoted $\\dx$ and $\\ddx$ respectively.\n","start":6442,"end":7273}]},{"id":"S002","semantic_type":"definition","kind":"definition","labels":["eq::CN"],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"连续牛顿比较系统","source_units":[{"raw_tex":"\tThis paper emphasizes the relation between \\eqref{eq::VMDINAVD} and special cases.\n\tIndeed, taking $\\eps=\\alpha=0$, one obtains\\footnote{\\ref{eq::CN} is usually considered with $\\beta=1$, we put $\\beta$ in the system to ease the discussions below.} the Continuous Newton (CN) method \\citep{gavurin1958nonlinear}\n\t\\begin{equation}\\tag{CN}\\label{eq::CN}\n\t\t\\beta \\Hf(x_N(t))\\dot{x}_N(t)+\\gf(x_N(t))=0, \\quad t\\geq 0,\n\t\\end{equation}\n\tknown notably for being invariant to affine transformations and yielding fast vanishing of the gradient (see Section~\\ref{sec::Control}).\n\tIn fact, this observation shows that \\eqref{eq::VMDINAVD} is a singular perturbation of \\eqref{eq::CN}, which also justifies the terminology ``Newton-like''  in DIN.\n","start":8490,"end":9227}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":["eq::LM"],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"正则化牛顿比较系统","source_units":[{"raw_tex":"\tWhen $\\alpha\\neq 0$ but $\\eps=0$, we recover the Levenberg--Marquardt (LM) method,\n\t\\begin{equation}\\tag{LM}\\label{eq::LM}\n\t\t\\alpha(t) \\dot x_{LM}(t) + \\beta \\Hf(x_{LM}(t))\\dot{x}_{LM}(t)+\\gf(x_{LM}(t))=0, \\quad t\\geq 0,\n\t\\end{equation}\n\talso known as regularized Newton method since it stabilizes \\eqref{eq::CN}.\n\tIn the rest of the paper, the solutions of \\eqref{eq::CN} and \\eqref{eq::LM} will always be denoted by $x_N$ and $x_{LM}$ respectively. Since the introduction of DIN, it is known (see \\cite{alvarez2002second}) that for $\\alpha=0$, $\\beta=1$, and $\\eps$ constant and small, \\eqref{eq::VMDINAVD} is a ``perturbed'' Newton method since the distance between the solutions of \\eqref{eq::VMDINAVD} and \\eqref{eq::CN}  is at most proportional to $\\sqrt{\\eps}$ at all times.\n","start":9227,"end":10010}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"目标函数的一般假设","source_units":[{"raw_tex":"\tWe always assume the following on the system \\eqref{eq::VMDINAVD} and the functions $f,\\eps,\\alpha$ defined in the introduction.\n\t\t\\begin{assumption}\\label{ass:general}\n\t\t\t\\textbullet\\ $f: \\R^n \\to \\R$ is convex, twice continuously differentiable, coercive, and strongly convex on bounded subsets of $\\R^n$.\n","start":18693,"end":19002}]},{"id":"S005","semantic_type":"assumption","kind":"assumption","labels":["ass:general"],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"共同位置与初始速度","source_units":[{"raw_tex":"\t\t\tWe fix $x_0$, $\\dot{x}_0\\in\\R^n$, such that, unless stated otherwise, \\eqref{eq::VMDINAVD} has initial condition $(x(0),\\dx(0))=(x_0,\\dx_0)$, and \\eqref{eq::CN} and \\eqref{eq::LM} have initial conditions $x_{N}(0)=x_{LM}(0)=x_0$.\n\\\\\n","start":19002,"end":19238}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"质量与阻尼的基本限制","source_units":[{"raw_tex":"\t\t\t\\textbullet\\ $\\alpha: \\R_+ \\to \\R$ is non-increasing, non-negative and differentiable. $\\eps: \\R_+ \\to \\R$ is non-increasing, positive, and twice differentiable with bounded second derivative.\n\t\t\tWe fix initial values: $\\eps(0)=\\eps_0>0$, $\\eps'(0)=\\eps'_0\\leq0$ and $\\alpha(0)=\\alpha_0\\geq 0$.\n\t\t\\end{assumption}\n","start":19238,"end":19555}]},{"id":"S007","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"强凸二次函数与谱坐标","source_units":[{"raw_tex":"\t\\section{Approximate Solutions and Asymptotic Analysis on Quadratics}\\label{sec::Quads}\n\tIn addition to Assumption~\\ref{ass:general}, we consider the case where $f$ is a strongly convex quadratic function in order to study the asymptotic behavior of \\eqref{eq::VMDINAVD} w.r.t.\\ \\eqref{eq::CN} and \\eqref{eq::LM}. Quadratic functions are the prototypical example of strongly convex functions. In particular, any strongly convex function can be locally approximated by a quadratic one around its minimizer, making the latter a good model for understanding the local behavior of dynamics.\n\tIn this section, $f$ is quadratic: $f(y)=\\frac{1}{2}\\Vert Ay-b\\Vert_2^2$ for all $y\\in \\R^n$, where $A\\in\\R^{n\\times n}$ is symmetric positive definite and $b\\in\\R^n$. Without loss of generality, we take $b=0$, so that the unique minimum is $x^\\star=0$.\n\t\\subsection{Setting: the Special Case of Quadratic Functions}\n\tQuadratic functions are particularly interesting in our setting since DIN-like ODEs take a simpler form (as observed in \\cite{attouch2020first,shi2021understanding}). Indeed, $\\forall y\\in\\R^n$, $\\gf(y) = A^TAy$ and $\\Hf(y)=A^T A$. Since $\\Hf(y)$ is independent of $y$ we can rewrite \\eqref{eq::VMDINAVD} in an eigenspace\\footnote{This can be generalized to the case where $A^TA$ is only semi-definite by considering orthogonal projections on an eigenspace spanned by the positive eigenvalues of $A^TA$.} of  $A^TA$. That is, we can study the ODE coordinate-wise by looking at one-dimensional problems of the form\n","start":40821,"end":42342}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq::QuadeDINAVD"],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"单特征方向的质量方程","source_units":[{"raw_tex":"\t\\begin{equation}\\label{eq::QuadeDINAVD}\\tag{Q1-VM-DIN-AVD}\n\t\t\\eps(t)\\ddx(t)+(\\alpha(t)+\\beta\\lambda)\\dx(t)+\\lambda x(t)=0, \\quad t\\geq 0.\n\t\\end{equation}\n\tHere (and throughout what follows) $\\lambda>0$ denotes any eigenvalue of $A^TA$ and $x\\colon\\R_+\\to\\R$ now denotes the corresponding  coordinate (function) of the solution of \\eqref{eq::VMDINAVD} in an eigenspace of $A^TA$. The dynamics \\eqref{eq::QuadeDINAVD} is a \\emph{linear} second-order ODE in $x$ with non-constant coefficients. Similarly, \\eqref{eq::LM} can be rewritten coordinate-wise as\n","start":42342,"end":42896}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq::quadCN","eq::quadLM"],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"牛顿与正则化牛顿标量式","source_units":[{"raw_tex":"\t\\begin{equation}\\label{eq::quadLM}\\tag{Q1-LM}\n\t\t(\\alpha(t)+\\beta\\lambda)\\dx_{LM}(t)+\\lambda x_{LM}(t)=0, \\quad t\\geq 0,\n\t\\end{equation}\n\twhere $\\xlm\\colon\\R_+\\to\\R$, and \\eqref{eq::CN} becomes\n\t\\begin{equation}\\label{eq::quadCN}\\tag{Q1-CN}\n\t\t\\beta\\dx_N(t)+ x_N(t)=0, \\quad t\\geq 0,\n\t\\end{equation}\n\twhere again, $x_N\\colon\\R_+\\to\\R$ is one-dimensional.\n","start":42896,"end":43250}]},{"id":"S010","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq::closeformLMN"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"比较轨迹的闭式表达","source_units":[{"raw_tex":"\tObserve in particular that \\eqref{eq::CN} and \\eqref{eq::LM} are now first-order \\emph{linear} ODEs, whose solutions have closed forms: $\\forall t\\geq 0$,\n\t\\begin{equation}\\label{eq::closeformLMN}\n\t\tx_N(t) = x_0e^{-\\frac{t}{\\beta}}\n\t\t\\quad \\text{and}\\quad\n\t\tx_{LM}(t) = x_0\\exp\\left(-\\int_0^t \\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}\\diff s \\right).\n\t\\end{equation}\n","start":43250,"end":43615}]},{"id":"S012","semantic_type":"external_fact","kind":"external_fact","labels":["eq::GenericODEthm","thm::olver"],"member_ids":["S012"],"declared_external":true,"declaration_text":["原始获准前提：Olver's Liouville–Green approximation theorem，引用 olver1997asymptotics，文中重述为 B0063。"],"source_only":false,"title":"Olver 的全局近似定理","source_units":[{"raw_tex":"\t\\begin{theorem}[\\cite{olver1997asymptotics}]\\label{thm::olver}\n\t\tLet $r\\colon \\R_{+}\\to \\R$ be a real, positive, twice continuously differentiable function, and define $\\varphi(t) = \\frac{4r(t)r''(t)-5r'(t)^2}{16r(t)^{5/2}}$ for all $t\\geq 0$. Then for any $T>0$, the differential equation,\n\t\t\\begin{equation}\\label{eq::GenericODEthm}\n\t\t\t\\ddot{z}(t)-r(t)z(t)=0,\\quad t\\in [0,T],\n\t\t\\end{equation}\n\t\thas two real and twice continuously differentiable solutions defined $\\forall t\\in[0,T]$ by,\n\t\t\\begin{equation*}\n\t\t\tz_1(t) = \\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t \\sqrt{r(s)}\\diff s\\right) \\ \\ \\text{and} \\ \\,\n\t\t\tz_2(t) = \\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\exp\\left(-\\int_0^t \\sqrt{r(s)}\\diff s\\right),\n\t\t\\end{equation*}\n\t\twhere $\\displaystyle\\vert\\delta_1(t)\\vert\\leq \\exp\\left(\\frac{1}{2} \\int_0^t \\vert \\varphi(s)\\vert\\diff s\\right)-1$ and $\\displaystyle\\vert\\delta_2(t)\\vert\\leq \\exp\\left(-\\frac{1}{2} \\int_t^{T} \\vert \\varphi(s)\\vert\\diff s\\right)-1$.\n\t\tIf in addition $\\displaystyle\\int_0^{+\\infty} \\vert\\varphi(s)\\vert\\diff s <+\\infty$, then the results above also hold for $T=+\\infty$.\n\t\\end{theorem}\n","start":46710,"end":47828}]},{"id":"S014","semantic_type":"derived_fact","kind":"derived_fact","labels":["rem::olver"],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"误差密度的局部可积性","source_units":[{"raw_tex":"\t\t--\\ Since we assumed $r$ to be twice continuously differentiable and positive, $\\varphi$ is continuous, so it is integrable except maybe for $t\\to+\\infty$.\n","start":48332,"end":48490}]},{"id":"S016","semantic_type":"assumption","kind":"assumption","labels":["ass:smooth"],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"三阶光滑性与小初始质量","source_units":[{"raw_tex":"\tWe now proceed to make use of the LG method for approximating the solutions of \\eqref{eq::QuadeDINAVD}. The reader only interested in the result can jump directly to the  Section~\\ref{sec::mainasymptoticres}. We first make the following assumption.\n\t\\begin{assumption}\\label{ass:smooth}\n\t\tThe functions $\\alpha$ and $\\eps$ are three times continuously differentiable, and $\\eps_0$ is such that $\\forall t\\geq 0$, $\\eps_0<\\frac{(\\beta\\lambda)^2}{2\\vert\\alpha'(t)\\vert+4\\lambda}$.\n\t\\end{assumption}\n","start":48815,"end":49313}]},{"id":"S017","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"全局误差密度","source_units":[{"raw_tex":"\tLemma~\\ref{lem::canonForm} gives a reformulation of \\eqref{eq::QuadeDINAVD} suited to apply Theorem~\\ref{thm::olver}.\n\tTo use the theorem for all $t\\geq 0$, we need to ensure that $\\varphi(t)=\\frac{4r(t)r''(t)-5r'(t)^2}{16r(t)^{5/2}}$ is integrable. To this aim we make the following assumption.\n","start":52183,"end":52480}]},{"id":"S018","semantic_type":"assumption","kind":"assumption","labels":["ass:subexp"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"导数可积与质量消失","source_units":[{"raw_tex":"\t\\begin{assumption}\\label{ass:subexp}\n\t\tThe functions $\\eps$ and $\\alpha$ have first, second and third-order derivatives that are integrable on $[0,+\\infty[$. In addition, $\\lim\\limits_{t\\to\\infty}{\\eps(t)}=0$ and $\\eps'(t)^2/\\eps(t)$ is integrable on $[0,+\\infty[$.\n\t\\end{assumption}\n","start":52480,"end":52765}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"渐近等价的比值含义","source_units":[{"raw_tex":"\tOur main result of Section~\\ref{sec::Quads} is the following, where $\\sim_{+\\infty}$ denotes the asymptotic equivalence\\footnote{Two real-valued functions $g_1$ and $g_2$ are asymptotically equivalent in $+\\infty$ if and only if $\\lim_{t\\to\\infty}\\frac{g_1(t)}{g_2(t)}=1$.} between two functions as $t\\to\\infty$.\n","start":59557,"end":59871}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","labels":["eq::ODErewritten","eq::pr","lem::canonForm"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"消去一阶项的规范形引理","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem::canonForm}\n\t\tSuppose that Assumption~\\ref{ass:smooth} holds, and let $x$ be the solution of \\ref{eq::QuadeDINAVD}. For all $t\\geq 0$, define\n\t\t\\begin{equation}\\label{eq::pr}\n\t\t\tp(t) = \\frac{\\alpha(t)+\\beta\\lambda}{\\eps(t)} \\quad \\text{and}  \\quad r(t) = \\frac{p(t)^2}{4} + \\frac{p'(t)}{2} - \\frac{\\lambda}{\\eps(t)}.\n\t\t\\end{equation}\n\t\tThen, $p$ and $r$ are twice continuously differentiable, $r$ is positive and the function $y$ defined for all $t\\geq 0$ by $y(t) = x(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)$ is a solution to\n\t\t\\begin{equation}\\label{eq::ODErewritten}\n\t\t\t\\ddot y(t) - r(t)y(t) = 0,\\quad t\\geq 0,%\\\\\n\t\t\\end{equation}\n\t\twith initial condition $(y(0),\\dot{y}(0) )= (x_0,\\dx_0 + \\frac{p(0)}{2}x_0)$.\n\t\\end{lemma}","start":49805,"end":50565}]},{"id":"S022","semantic_type":"definition","kind":"definition","labels":["eq::pr","lem::canonForm"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"规范形系数 p 与 r","source_units":[{"raw_tex":"\t\tSuppose that Assumption~\\ref{ass:smooth} holds, and let $x$ be the solution of \\ref{eq::QuadeDINAVD}. For all $t\\geq 0$, define\n\t\t\\begin{equation}\\label{eq::pr}\n\t\t\tp(t) = \\frac{\\alpha(t)+\\beta\\lambda}{\\eps(t)} \\quad \\text{and}  \\quad r(t) = \\frac{p(t)^2}{4} + \\frac{p'(t)}{2} - \\frac{\\lambda}{\\eps(t)}.\n\t\t\\end{equation}\n","start":49841,"end":50163}]},{"id":"S023","semantic_type":"proof","kind":"proof","labels":["eq::rpositive"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"规范形引理的作者全证明","source_units":[{"raw_tex":"\\begin{proof}\n\t\tWe first check that for all $t\\geq 0$, $r(t)$ is positive. Let $t>0$,\n\t\t\\begin{equation}\\label{eq::rpositive}\n\t\t\tr(t)> 0 \\iff \\frac{(\\alpha(t)+\\beta\\lambda)^2}{4\\eps(t)^2} + \\frac{\\alpha'(t)}{2\\eps(t)} -  \\frac{(\\alpha(t)+\\beta\\lambda)\\eps'(t)}{\\eps(t)^2} - \\frac{\\lambda}{\\eps(t)} >0.\n\t\t\\end{equation}\n\t\tSince $\\eps'(t)\\leq 0$ and $\\alpha'(t)\\leq 0$, one can check that a sufficient condition for \\eqref{eq::rpositive} to hold is,\n\t\t\\begin{equation*}\n\t\t\tr(t)> 0 \\impliedby \\frac{(\\alpha(t)+\\beta\\lambda)^2}{4} > \\left(\\frac{\\vert\\alpha'(t)\\vert}{2}+\\lambda\\right)\\eps(t) \\impliedby \\frac{(\\beta\\lambda)^2}{2\\vert\\alpha'(t)\\vert+4\\lambda}>\\eps_0.\n\t\t\\end{equation*}\n\t\tSo under Assumption~\\ref{ass:smooth}, for all $t\\geq 0$, $r(t)>0$. We then check that $y$ is indeed solution to \\eqref{eq::ODErewritten}. Let $t>0$,\n\t\t\\begin{equation*}\n\t\t\t\\dot{y}(t) = \\frac{p(t)}{2}x(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right) + \\dx(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right),\\quad \\text{and,}\n\t\t\\end{equation*}\n\t\t\\begin{equation*}\n\t\t\t\\ddot{y}(t) = \\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)\\left[\\left(\\frac{p(t)^2}{4}+\\frac{p'(t)}{2}\\right)x(t) + p(t)\\dx(t) + \\ddx(t)\\right].\n\t\t\\end{equation*}\n\t\tSince $x$ solves \\eqref{eq::QuadeDINAVD}, it holds that $\\ddx(t) = -p(t)\\dx(t) - \\frac{\\lambda}{\\eps(t)}x(t)$, so,\n\t\t\\begin{multline*}\n\t\t\t\\ddot{y}(t) = \\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)\\left(\\frac{p(t)^2}{4}+\\frac{p'(t)}{2}- \\frac{\\lambda}{\\eps(t)}\\right)x(t) \\\\= \\left(\\frac{p(t)^2}{4}+\\frac{p'(t)}{2}- \\frac{\\lambda}{\\eps(t)}\\right)y(t)\n\t\t\t= r(t)y(t).\n\t\t\\end{multline*}\n\t\\end{proof}","start":50567,"end":52182}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","labels":["eq::rpositive"],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"用质量上限保证 r 正","source_units":[{"raw_tex":"\\begin{proof}\n\t\tWe first check that for all $t\\geq 0$, $r(t)$ is positive. Let $t>0$,\n\t\t\\begin{equation}\\label{eq::rpositive}\n\t\t\tr(t)> 0 \\iff \\frac{(\\alpha(t)+\\beta\\lambda)^2}{4\\eps(t)^2} + \\frac{\\alpha'(t)}{2\\eps(t)} -  \\frac{(\\alpha(t)+\\beta\\lambda)\\eps'(t)}{\\eps(t)^2} - \\frac{\\lambda}{\\eps(t)} >0.\n\t\t\\end{equation}\n\t\tSince $\\eps'(t)\\leq 0$ and $\\alpha'(t)\\leq 0$, one can check that a sufficient condition for \\eqref{eq::rpositive} to hold is,\n\t\t\\begin{equation*}\n\t\t\tr(t)> 0 \\impliedby \\frac{(\\alpha(t)+\\beta\\lambda)^2}{4} > \\left(\\frac{\\vert\\alpha'(t)\\vert}{2}+\\lambda\\right)\\eps(t) \\impliedby \\frac{(\\beta\\lambda)^2}{2\\vert\\alpha'(t)\\vert+4\\lambda}>\\eps_0.\n\t\t\\end{equation*}\n\t\tSo under Assumption~\\ref{ass:smooth}, for all $t\\geq 0$, $r(t)>0$. We then check that $y$ is indeed solution to \\eqref{eq::ODErewritten}. Let $t>0$,\n","start":50567,"end":51399}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"微分指数变换并代回方程","source_units":[{"raw_tex":"\t\t\\begin{equation*}\n\t\t\t\\dot{y}(t) = \\frac{p(t)}{2}x(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right) + \\dx(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right),\\quad \\text{and,}\n\t\t\\end{equation*}\n\t\t\\begin{equation*}\n\t\t\t\\ddot{y}(t) = \\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)\\left[\\left(\\frac{p(t)^2}{4}+\\frac{p'(t)}{2}\\right)x(t) + p(t)\\dx(t) + \\ddx(t)\\right].\n\t\t\\end{equation*}\n\t\tSince $x$ solves \\eqref{eq::QuadeDINAVD}, it holds that $\\ddx(t) = -p(t)\\dx(t) - \\frac{\\lambda}{\\eps(t)}x(t)$, so,\n\t\t\\begin{multline*}\n\t\t\t\\ddot{y}(t) = \\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)\\left(\\frac{p(t)^2}{4}+\\frac{p'(t)}{2}- \\frac{\\lambda}{\\eps(t)}\\right)x(t) \\\\= \\left(\\frac{p(t)^2}{4}+\\frac{p'(t)}{2}- \\frac{\\lambda}{\\eps(t)}\\right)y(t)\n\t\t\t= r(t)y(t).\n\t\t\\end{multline*}\n\t\\end{proof}","start":51399,"end":52182}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","labels":["lem::phiintegrable"],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"误差密度的半轴可积性","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem::phiintegrable}\n\t\tUnder Assumption~\\ref{ass:smooth} and~\\ref{ass:subexp}, $\\int_0^{+\\infty} \\vert \\varphi(s)\\vert\\diff s<+\\infty$.\n\t\\end{lemma}","start":53173,"end":53340}]},{"id":"S027","semantic_type":"proof","kind":"proof","labels":["eq::factorR","eq::rprime","eq::rsec"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"误差可积性的作者全证明","source_units":[{"raw_tex":"\\begin{proof}\n\t\tWe suppose that Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp} hold. As stated in Remark~\\ref{rem::olver}, since $\\varphi$ is continuous, we only need to check its integrability when $t$ tends to $+\\infty$. Let $t>0$, we first establish some useful identities, we omit the dependence on $t$ for the sake of readability.\n\t\t\\begin{align*}%\\label{eq::pprime}\n\t\t\tp' = \\frac{\\alpha'\\eps-(\\alpha+\\beta\\lambda)\\eps'}{\\eps^2},\n\t\t\t\\quad \\text{and}\\quad\n\t\t\tp'' = \\frac{\\alpha''\\eps^2-2\\alpha'\\eps'\\eps-(\\alpha+\\beta\\lambda)\\eps''\\eps + 2(\\alpha+\\beta\\lambda)(\\eps')^2}{\\eps^3}.\n\t\t\\end{align*}\n\t\tThen,\n\t\t\\begin{align}\\label{eq::factorR}\n\t\t\t\\begin{split}\n\t\t\t\tr  &= \\frac{p^2}{4}\\left(1+ \\frac{2p'}{p^2} - \\frac{4\\lambda}{\\eps p^2}\\right) = \\frac{(\\alpha+\\beta\\lambda)^2}{4\\eps^2}\\left(1+\\frac{2p' \\eps ^2}{(\\alpha +\\beta\\lambda)^2} - \\frac{4\\lambda\\eps }{(\\alpha +\\beta\\lambda)^2}\\right)\n\t\t\t\t\\\\\n\t\t\t\t&=\\frac{(\\alpha +\\beta\\lambda)^2}{4\\eps ^2}\\left(1+\\frac{2\\alpha' \\eps }{(\\alpha +\\beta\\lambda)^2} - \\frac{2\\eps' }{(\\alpha +\\beta\\lambda)} - \\frac{4\\lambda\\eps }{(\\alpha +\\beta\\lambda)^2}\\right).\n\t\t\t\\end{split}\n\t\t\\end{align}\n\t\tAn important consequence of Assumption~\\ref{ass:subexp} is that $\\vert\\eps'(t)\\vert = o(\\eps(t))$, $\\vert\\eps''(t)\\vert = o(\\eps'(t))$ (and the same holds for $\\alpha$ w.r.t.\\ to its derivatives). Therefore, we deduce from \\eqref{eq::factorR} that\n\t\t\\begin{equation*}\n\t\t\tr(t)\\sim_{+\\infty}  \\frac{(\\alpha(t)+\\beta\\lambda)^2}{4\\eps(t)^2},\n\t\t\\end{equation*}\n\t\tand we note that $1/r$ decays at the same speed as $\\eps^2$, which will be useful later. In order to study $\\varphi$, we now differentiate $r$,\n\t\t\\begin{align}\\label{eq::rprime}\n\t\t\t\\begin{split}\n\t\t\t\tr' &= \\frac{p'p}{2}\\left(1+ \\frac{2p'}{p^2} - \\frac{4\\lambda}{\\eps p^2}\\right) + \\frac{1}{4}\\left( 2p'' - \\frac{4(p')^2}{p} + \\frac{8\\lambda p'}{\\eps p}+ \\frac{4\\lambda\\eps'}{\\eps^2} \\right)\n\t\t\t\t\\\\&= \\frac{2p'}{p}r + \\frac{1}{4}\\left( 2p'' - \\frac{4(p')^2}{p} +\\frac{8\\lambda p'}{\\eps p} + \\frac{4\\lambda\\eps'}{\\eps^2}  \\right),\\quad \\text{and,}\n\t\t\t\\end{split}\n\t\t\\end{align}\n\n\t\t\\begin{multline}\\label{eq::rsec}\n\t\t\tr'' = 2\\frac{p''p-(p')^2}{p^2}r  + \\frac{2p'}{p}r'\n\t\t\t\\\\\n\t\t\t+ \\frac{1}{4}\\left( 2p''' + 4\\frac{(p')^3-2p''p'p}{p^2}+8 \\lambda\\frac{p''p\\eps - (p')^2\\eps - p'p\\eps'}{\\eps^2 p^2} + \\frac{4\\lambda\\eps''}{\\eps^2} - \\frac{8\\lambda(\\eps')^2}{\\eps^3}  \\right).\n\t\t\\end{multline}\n\t\tThen, to justify that $\\varphi$ is integrable, we prove that $\\frac{r''}{r^{3/2}}$ and $\\frac{(r')^2}{r^{5/2}}$ are integrable. Since we know that $1/r$ decays at the same speed as $\\eps^2$, we can equivalently show that $\\eps^3r''$ and $\\eps^5(r')^2$ are integrable.\n\t\tTo this aim we fully expand all the terms in \\eqref{eq::rprime} and \\eqref{eq::rsec}.\n\t\tWe first deal with \\eqref{eq::rprime}, it holds that:\n\t\t\\begin{align*}\n\t\t\tr'^2&\\eps^{5} =\n\t\t\t\\left[- \\frac{ (\\alpha+\\beta\\lambda)^{2} \\left(- \\frac{4 \\lambda \\eps }{ (\\alpha+\\beta\\lambda)^{2}} + 1 + \\frac{\\left(- {2  (\\alpha+\\beta\\lambda) \\eps' } + {2 \\alpha'\\eps }\\right) }{ (\\alpha+\\beta\\lambda)^{2}}\\right) \\eps' }{2 \\sqrt{\\eps}}\\right.\n\t\t\t\\\\ &\\left.\n\t\t\t+ \\frac{ (\\alpha+\\beta\\lambda)^{2}\\sqrt{\\eps}}{4} \\left(- \\frac{4 \\lambda \\eps' }{ (\\alpha+\\beta\\lambda)^{2}} + \\frac{8 \\lambda \\alpha'  \\eps }{ (\\alpha+\\beta\\lambda)^{3}} + \\frac{2 \\left(- \\frac{2  (\\alpha+\\beta\\lambda) \\eps' }{\\eps } + {2 \\alpha'\\eps }\\right) \\eps'   }{ (\\alpha+\\beta\\lambda)^{2}}\n\t\t\t\\right.\\right.\n\t\t\t\\\\&\\left.\\left.\n\t\t\t+ \\frac{\\left(\\frac{4  (\\alpha+\\beta\\lambda) \\eps'^{2} }{\\eps } - {2  (\\alpha+\\beta\\lambda) \\eps'' } - {4 \\alpha'  \\eps' } + {2 \\alpha''\\eps }\\right) }{ (\\alpha+\\beta\\lambda)^{2}} - \\frac{2 \\left(-{2  (\\alpha+\\beta\\lambda) \\eps' } + {2 \\alpha'\\eps }\\right) \\alpha'   }{ (\\alpha+\\beta\\lambda)^{3}}\\right)\n\t\t\t\\right.\\\\\n\t\t\t&\\left. + \\frac{ (\\alpha+\\beta\\lambda)}{2} { \\left(- \\frac{4 \\lambda \\eps }{ (\\alpha+\\beta\\lambda)^{2}} + 1 + \\frac{\\left(- {2  (\\alpha+\\beta\\lambda) \\eps' } + {2 \\alpha'\\eps }\\right) }{ (\\alpha+\\beta\\lambda)^{2}}\\right)} \\alpha' \\sqrt{\\eps}\n\t\t\t\\phantom{\\frac{\\frac{\\beta}{\\beta}}{\\beta}}\n\t\t\t\\hspace{-1.7cm}\\phantom{\\frac{\\left( \\frac{\\frac{a'}{a'}}{a'}\\right)}{a^2}{\\sqrt{.}}\n\t\t\t}\\right]^{2}.\n\t\t\\end{align*}\n\t\tThe computations for $\\eps^2r''$ are omitted since they are longer but very similar.\n\n\t\tWe analyze the integrability of each of the terms above (and those of $\\eps^2r''$). By Assumption~\\ref{ass:subexp}, $\\eps'$, $\\eps''$ and $\\eps'''$ are integrable, as well as $\\alpha'$, $\\alpha''$ and $\\alpha'''$, this justifies the integrability of most of the terms. We also need $\\frac{(\\eps')^2}{\\eps}$ and $\\frac{(\\eps')^3}{\\eps}$ to be integrable, which holds by Assumption~\\ref{ass:subexp}. Overall, $\\varphi$ is integrable on $\\R_{+}$.\n\t\\end{proof}","start":91200,"end":95908}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"归约无穷远并计算 p 的导数","source_units":[{"raw_tex":"\\begin{proof}\n\t\tWe suppose that Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp} hold. As stated in Remark~\\ref{rem::olver}, since $\\varphi$ is continuous, we only need to check its integrability when $t$ tends to $+\\infty$. Let $t>0$, we first establish some useful identities, we omit the dependence on $t$ for the sake of readability.\n\t\t\\begin{align*}%\\label{eq::pprime}\n\t\t\tp' = \\frac{\\alpha'\\eps-(\\alpha+\\beta\\lambda)\\eps'}{\\eps^2},\n\t\t\t\\quad \\text{and}\\quad\n\t\t\tp'' = \\frac{\\alpha''\\eps^2-2\\alpha'\\eps'\\eps-(\\alpha+\\beta\\lambda)\\eps''\\eps + 2(\\alpha+\\beta\\lambda)(\\eps')^2}{\\eps^3}.\n\t\t\\end{align*}\n","start":91200,"end":91802}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":["eq::factorR"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"分解 r 的主因子","source_units":[{"raw_tex":"\t\tThen,\n\t\t\\begin{align}\\label{eq::factorR}\n\t\t\t\\begin{split}\n\t\t\t\tr  &= \\frac{p^2}{4}\\left(1+ \\frac{2p'}{p^2} - \\frac{4\\lambda}{\\eps p^2}\\right) = \\frac{(\\alpha+\\beta\\lambda)^2}{4\\eps^2}\\left(1+\\frac{2p' \\eps ^2}{(\\alpha +\\beta\\lambda)^2} - \\frac{4\\lambda\\eps }{(\\alpha +\\beta\\lambda)^2}\\right)\n\t\t\t\t\\\\\n\t\t\t\t&=\\frac{(\\alpha +\\beta\\lambda)^2}{4\\eps ^2}\\left(1+\\frac{2\\alpha' \\eps }{(\\alpha +\\beta\\lambda)^2} - \\frac{2\\eps' }{(\\alpha +\\beta\\lambda)} - \\frac{4\\lambda\\eps }{(\\alpha +\\beta\\lambda)^2}\\right).\n\t\t\t\\end{split}\n\t\t\\end{align}\n","start":91802,"end":92332}]},{"id":"S030","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"作者声称的导数衰减层级","source_units":[{"raw_tex":"\t\tAn important consequence of Assumption~\\ref{ass:subexp} is that $\\vert\\eps'(t)\\vert = o(\\eps(t))$, $\\vert\\eps''(t)\\vert = o(\\eps'(t))$ (and the same holds for $\\alpha$ w.r.t.\\ to its derivatives). Therefore, we deduce from \\eqref{eq::factorR} that\n","start":92332,"end":92582}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"识别 r 与倒数的渐近量级","source_units":[{"raw_tex":"\t\t\\begin{equation*}\n\t\t\tr(t)\\sim_{+\\infty}  \\frac{(\\alpha(t)+\\beta\\lambda)^2}{4\\eps(t)^2},\n\t\t\\end{equation*}\n\t\tand we note that $1/r$ decays at the same speed as $\\eps^2$, which will be useful later. In order to study $\\varphi$, we now differentiate $r$,\n","start":92582,"end":92836}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":["eq::rprime","eq::rsec"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"展开 r 的一二阶导数","source_units":[{"raw_tex":"\t\t\\begin{align}\\label{eq::rprime}\n\t\t\t\\begin{split}\n\t\t\t\tr' &= \\frac{p'p}{2}\\left(1+ \\frac{2p'}{p^2} - \\frac{4\\lambda}{\\eps p^2}\\right) + \\frac{1}{4}\\left( 2p'' - \\frac{4(p')^2}{p} + \\frac{8\\lambda p'}{\\eps p}+ \\frac{4\\lambda\\eps'}{\\eps^2} \\right)\n\t\t\t\t\\\\&= \\frac{2p'}{p}r + \\frac{1}{4}\\left( 2p'' - \\frac{4(p')^2}{p} +\\frac{8\\lambda p'}{\\eps p} + \\frac{4\\lambda\\eps'}{\\eps^2}  \\right),\\quad \\text{and,}\n\t\t\t\\end{split}\n\t\t\\end{align}\n\n\t\t\\begin{multline}\\label{eq::rsec}\n\t\t\tr'' = 2\\frac{p''p-(p')^2}{p^2}r  + \\frac{2p'}{p}r'\n\t\t\t\\\\\n\t\t\t+ \\frac{1}{4}\\left( 2p''' + 4\\frac{(p')^3-2p''p'p}{p^2}+8 \\lambda\\frac{p''p\\eps - (p')^2\\eps - p'p\\eps'}{\\eps^2 p^2} + \\frac{4\\lambda\\eps''}{\\eps^2} - \\frac{8\\lambda(\\eps')^2}{\\eps^3}  \\right).\n\t\t\\end{multline}\n","start":92836,"end":93576}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"化为加权导数的可积性","source_units":[{"raw_tex":"\t\tThen, to justify that $\\varphi$ is integrable, we prove that $\\frac{r''}{r^{3/2}}$ and $\\frac{(r')^2}{r^{5/2}}$ are integrable. Since we know that $1/r$ decays at the same speed as $\\eps^2$, we can equivalently show that $\\eps^3r''$ and $\\eps^5(r')^2$ are integrable.\n\t\tTo this aim we fully expand all the terms in \\eqref{eq::rprime} and \\eqref{eq::rsec}.\n\t\tWe first deal with \\eqref{eq::rprime}, it holds that:\n","start":93576,"end":93990}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"展示加权平方导数展开","source_units":[{"raw_tex":"\t\t\\begin{align*}\n\t\t\tr'^2&\\eps^{5} =\n\t\t\t\\left[- \\frac{ (\\alpha+\\beta\\lambda)^{2} \\left(- \\frac{4 \\lambda \\eps }{ (\\alpha+\\beta\\lambda)^{2}} + 1 + \\frac{\\left(- {2  (\\alpha+\\beta\\lambda) \\eps' } + {2 \\alpha'\\eps }\\right) }{ (\\alpha+\\beta\\lambda)^{2}}\\right) \\eps' }{2 \\sqrt{\\eps}}\\right.\n\t\t\t\\\\ &\\left.\n\t\t\t+ \\frac{ (\\alpha+\\beta\\lambda)^{2}\\sqrt{\\eps}}{4} \\left(- \\frac{4 \\lambda \\eps' }{ (\\alpha+\\beta\\lambda)^{2}} + \\frac{8 \\lambda \\alpha'  \\eps }{ (\\alpha+\\beta\\lambda)^{3}} + \\frac{2 \\left(- \\frac{2  (\\alpha+\\beta\\lambda) \\eps' }{\\eps } + {2 \\alpha'\\eps }\\right) \\eps'   }{ (\\alpha+\\beta\\lambda)^{2}}\n\t\t\t\\right.\\right.\n\t\t\t\\\\&\\left.\\left.\n\t\t\t+ \\frac{\\left(\\frac{4  (\\alpha+\\beta\\lambda) \\eps'^{2} }{\\eps } - {2  (\\alpha+\\beta\\lambda) \\eps'' } - {4 \\alpha'  \\eps' } + {2 \\alpha''\\eps }\\right) }{ (\\alpha+\\beta\\lambda)^{2}} - \\frac{2 \\left(-{2  (\\alpha+\\beta\\lambda) \\eps' } + {2 \\alpha'\\eps }\\right) \\alpha'   }{ (\\alpha+\\beta\\lambda)^{3}}\\right)\n\t\t\t\\right.\\\\\n\t\t\t&\\left. + \\frac{ (\\alpha+\\beta\\lambda)}{2} { \\left(- \\frac{4 \\lambda \\eps }{ (\\alpha+\\beta\\lambda)^{2}} + 1 + \\frac{\\left(- {2  (\\alpha+\\beta\\lambda) \\eps' } + {2 \\alpha'\\eps }\\right) }{ (\\alpha+\\beta\\lambda)^{2}}\\right)} \\alpha' \\sqrt{\\eps}\n\t\t\t\\phantom{\\frac{\\frac{\\beta}{\\beta}}{\\beta}}\n\t\t\t\\hspace{-1.7cm}\\phantom{\\frac{\\left( \\frac{\\frac{a'}{a'}}{a'}\\right)}{a^2}{\\sqrt{.}}\n\t\t\t}\\right]^{2}.\n\t\t\\end{align*}\n","start":93990,"end":95362}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"逐项可积性与省略的计算","source_units":[{"raw_tex":"\t\tThe computations for $\\eps^2r''$ are omitted since they are longer but very similar.\n\n\t\tWe analyze the integrability of each of the terms above (and those of $\\eps^2r''$). By Assumption~\\ref{ass:subexp}, $\\eps'$, $\\eps''$ and $\\eps'''$ are integrable, as well as $\\alpha'$, $\\alpha''$ and $\\alpha'''$, this justifies the integrability of most of the terms. We also need $\\frac{(\\eps')^2}{\\eps}$ and $\\frac{(\\eps')^3}{\\eps}$ to be integrable, which holds by Assumption~\\ref{ass:subexp}. Overall, $\\varphi$ is integrable on $\\R_{+}$.\n\t\\end{proof}","start":95362,"end":95908}]},{"id":"S036","semantic_type":"lemma","kind":"lemma","labels":["lem::remainderterm"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"平方修正项的首阶余项","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem::remainderterm}\n\t\tUnder Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp}, for all $s\\geq 0$, $$\\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2 = \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s)).$$\n\t\\end{lemma}","start":96010,"end":96329}]},{"id":"S037","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"平方余项引理的作者全证明","source_units":[{"raw_tex":"\\begin{proof}\n\t\tWe omit the time dependence on $s\\geq 0$ for the sake of readability. Using Assumption~\\ref{ass:smooth} we can define and expand the following quantity,\n\t\t\\begin{align*}\n\t\t\t&\\frac{1}{16}\\left(\\frac{2p'}{p^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps}}{(\\alpha+\\beta\\lambda)^{3/2}}\\right)^2\n\t\t\t=\\frac{\\lambda^2\\eps}{(\\alpha+\\beta\\lambda)^{3}}  - \\frac{p'\\lambda\\sqrt{\\eps}}{p^{3/2}(\\alpha+\\beta\\lambda)^{3/2}} + \\frac{(p')^2}{4p^{3}}\n\t\t\t\\\\&= \\frac{\\lambda^2\\eps}{(\\alpha+\\beta\\lambda)^{3}}\n\t\t\t\t- \\frac{\\lambda(\\alpha'\\eps - (\\alpha+\\beta\\lambda)\\eps')}{(\\alpha+\\beta\\lambda)^{3}}\n\t\t\t\t+ \\frac{(\\alpha')^2\\eps + \\frac{(\\eps')^2}{\\eps}(\\alpha+\\beta\\lambda)^2 - 2 \\alpha'\\eps'(\\alpha+\\beta\\lambda)}{4(\\alpha+\\beta\\lambda)^{3}}.\n\t\t\\end{align*}\n\t\tAssumption~\\ref{ass:subexp}, implies in particular that $\\vert\\eps'(t)\\vert=o(\\eps(t))$ and that $\\alpha'(t)\\to 0$, which we use in the equality above to obtain the desired conclusion.\n\t\\end{proof}","start":96331,"end":97273}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"展开平方并消去 p 记号","source_units":[{"raw_tex":"\\begin{proof}\n\t\tWe omit the time dependence on $s\\geq 0$ for the sake of readability. Using Assumption~\\ref{ass:smooth} we can define and expand the following quantity,\n\t\t\\begin{align*}\n\t\t\t&\\frac{1}{16}\\left(\\frac{2p'}{p^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps}}{(\\alpha+\\beta\\lambda)^{3/2}}\\right)^2\n\t\t\t=\\frac{\\lambda^2\\eps}{(\\alpha+\\beta\\lambda)^{3}}  - \\frac{p'\\lambda\\sqrt{\\eps}}{p^{3/2}(\\alpha+\\beta\\lambda)^{3/2}} + \\frac{(p')^2}{4p^{3}}\n\t\t\t\\\\&= \\frac{\\lambda^2\\eps}{(\\alpha+\\beta\\lambda)^{3}}\n\t\t\t\t- \\frac{\\lambda(\\alpha'\\eps - (\\alpha+\\beta\\lambda)\\eps')}{(\\alpha+\\beta\\lambda)^{3}}\n\t\t\t\t+ \\frac{(\\alpha')^2\\eps + \\frac{(\\eps')^2}{\\eps}(\\alpha+\\beta\\lambda)^2 - 2 \\alpha'\\eps'(\\alpha+\\beta\\lambda)}{4(\\alpha+\\beta\\lambda)^{3}}.\n\t\t\\end{align*}\n","start":96331,"end":97074}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"用相对导数衰减识别小余项","source_units":[{"raw_tex":"\t\tAssumption~\\ref{ass:subexp}, implies in particular that $\\vert\\eps'(t)\\vert=o(\\eps(t))$ and that $\\alpha'(t)\\to 0$, which we use in the equality above to obtain the desired conclusion.\n\t\\end{proof}","start":97074,"end":97273}]},{"id":"S040","semantic_type":"theorem","kind":"theorem","labels":["eq::deltabounds","eq::formofx","thm::MainResLG"],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"标量轨迹的两模态表达定理","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm::MainResLG}\n\t\tSuppose that Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp} hold. There exists $A,B\\in\\R$ such that $x(0)=x_0$, $\\dx(0)=\\dx_0$ and for all $t\\geq 0$, the solution of \\eqref{eq::QuadeDINAVD} is\n\t\t\\begin{align}\\label{eq::formofx}\n\t\t\t\\begin{split}\n\t\t\t\tx(t) =&\n\t\t\t\tA\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\alpha(t)+\\beta\\lambda}}{\\sqrt{\\eps(t)}}\n\t\t\t\t\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\n\t\t\t\t\\\\\n\t\t\t\t+B&\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\n\t\t\t\t\\frac{\\sqrt{\\eps(t)}}{\\sqrt{\\alpha(t)+\\beta\\lambda}}\n\t\t\t\t\\\\\n\t\t\t\t&\\exp\\left(\\int_0^t -\\frac{\\alpha(s)+\\beta\\lambda}{\\eps(s)}+\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}+\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right),\n\t\t\t\\end{split}\n\t\t\\end{align}\n\t\twhere for all  $t\\geq 0$,\n\t\t\\begin{equation}\\label{eq::deltabounds}\n\t\t\t\\vert\\delta_1(t)\\vert \\leq \\exp\\left(\\frac{1}{2} \\int_0^t \\vert \\varphi(s)\\vert\\diff s\\right)-1\n\t\t\t\\quad \\text{and}\\quad\n\t\t\t\\vert\\delta_2(t)\\vert \\leq \\exp\\left(-\\frac{1}{2} \\int_t^{+\\infty} \\vert \\varphi(s)\\vert\\diff s\\right)-1.\n\t\t\\end{equation}\n\t\\end{theorem}","start":53595,"end":54786}]},{"id":"S041","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq::formofx","thm::MainResLG"],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"慢模态与快模态的具体形式","source_units":[{"raw_tex":"\t\t\\begin{align}\\label{eq::formofx}\n\t\t\t\\begin{split}\n\t\t\t\tx(t) =&\n\t\t\t\tA\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\alpha(t)+\\beta\\lambda}}{\\sqrt{\\eps(t)}}\n\t\t\t\t\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\n\t\t\t\t\\\\\n\t\t\t\t+B&\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\n\t\t\t\t\\frac{\\sqrt{\\eps(t)}}{\\sqrt{\\alpha(t)+\\beta\\lambda}}\n\t\t\t\t\\\\\n\t\t\t\t&\\exp\\left(\\int_0^t -\\frac{\\alpha(s)+\\beta\\lambda}{\\eps(s)}+\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}+\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right),\n\t\t\t\\end{split}\n\t\t\\end{align}\n","start":53831,"end":54452}]},{"id":"S042","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq::deltabounds","thm::MainResLG"],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"两误差函数的原文上界","source_units":[{"raw_tex":"\t\twhere for all  $t\\geq 0$,\n\t\t\\begin{equation}\\label{eq::deltabounds}\n\t\t\t\\vert\\delta_1(t)\\vert \\leq \\exp\\left(\\frac{1}{2} \\int_0^t \\vert \\varphi(s)\\vert\\diff s\\right)-1\n\t\t\t\\quad \\text{and}\\quad\n\t\t\t\\vert\\delta_2(t)\\vert \\leq \\exp\\left(-\\frac{1}{2} \\int_t^{+\\infty} \\vert \\varphi(s)\\vert\\diff s\\right)-1.\n\t\t\\end{equation}\n","start":54452,"end":54772}]},{"id":"S043","semantic_type":"proof","kind":"proof","labels":["eq::formofx0","eq::sqrtr"],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"两模态表达定理的作者全证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm::MainResLG}]\n\t\tLet $x$ be the solution of \\eqref{eq::QuadeDINAVD} define $p,r$ as in \\eqref{eq::pr}. Let us also define $y(t)\\stackrel{\\textrm{def}}{=}x(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)$. According to Lemma~\\ref{lem::canonForm}, $r$ is positive and $y$ is solution to \\eqref{eq::ODErewritten}. Then, from Lemma~\\ref{lem::phiintegrable}, $\\int_t^{T} \\vert \\varphi(s)\\vert\\diff s<+\\infty$, so we can apply Theorem~\\ref{thm::olver} to $y$ on $[0,+\\infty[$. Therefore, there exists $A,B\\in\\R$, such that $\\forall t\\geq 0$,\n\t\t$$\n\t\ty(t)= A\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t \\sqrt{r(s)}\\diff s\\right) + B\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t -\\sqrt{r(s)}\\diff s\\right),\n\t\t$$\n\t\twhere $A$ and $B$ are determined by the initial conditions, and $\\delta_1$, $\\delta_2$ are such that \\eqref{eq::deltabounds} holds.\n\t\tGoing back to $x(t) = y(t)\\exp\\left(\\int_0^t -\\frac{p(s)}{2}\\diff s\\right)$, we obtain \tthat for all $t\\geq 0$,\n\t\t\\begin{multline}\\label{eq::formofx0}\n\t\t\tx(t) = A\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t -\\frac{p(s)}{2} +\\sqrt{r(s)}\\diff s\\right) \\\\ + B\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t -\\frac{p(s)}{2} -\\sqrt{r(s)}\\diff s\\right).\n\t\t\\end{multline}\n\t\tIt now remains to expand the terms in the two exponentials in \\eqref{eq::formofx0} in order to obtain \\eqref{eq::formofx}. To this aim, we approximate $\\sqrt{r(s)}$, let $s\\geq 0$,\n\t\t\\begin{align}\\label{eq::sqrtr}\n\t\t\t\\begin{split}\n\t\t\t\t\\sqrt{r(s)}&=\\frac{p(s)}{2}\\sqrt{1+ \\frac{2p'(s)}{p(s)^2} - \\frac{4\\lambda}{\\eps(s)p(s)^2}}\n\t\t\t\t\\\\&= \\frac{p(s)}{2}\\left(1 + \\frac{p'(s)}{p(s)^2} - \\frac{2\\lambda}{\\eps(s)p(s)^2} - \\frac{1}{8}\\left(\\frac{2p'(s)}{p(s)^2} - \\frac{4\\lambda}{\\eps(s)p(s)^2}\\right)^2 + o(\\eps(s)^2) \\right)\n\t\t\t\t\\\\ & = \\frac{p(s)}{2} +\\frac{ p'(s)}{2p(s)} - \\frac{\\lambda}{\\eps(s)p(s)} - \\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda}{\\eps(s)p(s)^{3/2}}\\right)^2 + o(\\eps(s))\n\t\t\t\t\\\\  = \\frac{p(s)}{2} &+ \\frac{p'(s)\\eps(s)}{2(\\alpha(s)+\\beta\\lambda)} - \\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2 + o(\\eps(s))\n\t\t\t\t\\\\  = \\frac{p(s)}{2} &+ \\frac{\\alpha'(s)/2-\\lambda}{\\alpha(s)+\\beta\\lambda} -\\frac{\\eps'(s)}{2\\eps(s)}  - \\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2 + o(\\eps(s))\n\t\t\t\\end{split}\n\t\t\\end{align}\n\t\tTo ease the readability, we denote $h(t)=\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2$. Focusing on the first exponential term in \\eqref{eq::formofx0}, we deduce from \\eqref{eq::sqrtr} that for all $t\\geq 0$,\n\t\t\\begin{align*}%\\label{eq::Aexpterm}\n\t\t\t\\begin{split}\n\t\t\t\t&\\exp\\left(\\int_0^t -\\frac{p(s)}{2} +\\sqrt{r(s)}\\diff s\\right)\n\t\t\t\t\\\\&= \\exp\\left(\\int_0^t \\frac{\\alpha'(s)/2}{\\alpha(s)+\\beta\\lambda} -\\frac{\\eps'(s)}{2\\eps(s)} -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{1}{16}h(t) + o(\\eps(s))\\diff s\\right)\n\t\t\t\t\\\\&=  \\frac{\\sqrt{\\alpha(t)+\\beta\\lambda})}{\\sqrt{\\alpha_0+\\beta\\lambda}} \\frac{\\sqrt{\\eps_0}}{\\sqrt{\\eps(t)}} \\exp\\left(\n\t\t\t\t\\int_0^t \\frac{-\\lambda}{\\alpha(s)+\\beta\\lambda} -\n\t\t\t\t\\frac{1}{16}h(t) + o(\\eps(s))\\diff s\\right)\n\t\t\t\t\\\\&=  \\frac{\\sqrt{\\alpha(t)+\\beta\\lambda})}{\\sqrt{\\alpha_0+\\beta\\lambda}} \\frac{\\sqrt{\\eps_0}}{\\sqrt{\\eps(t)}} \\exp\\left(\n\t\t\t\t\\int_0^t \\frac{-\\lambda}{\\alpha(s)+\\beta\\lambda} -\n\t\t\t\t\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}} + o(\\eps(s))\\diff s\\right),\n\t\t\t\\end{split}\n\t\t\\end{align*}\n\t\twhere the last line relies on further computations postponed to Lemma~\\ref{lem::remainderterm} in Appendix~\\ref{app::integrability}. Performing the exact same type of computations on $\\exp\\left(\\int_0^t -\\frac{p(s)}{2} -\\sqrt{r(s)}\\diff s\\right)$, and up to redefining $A$ and $B$ so as to encompass all the constants, we obtain \\eqref{eq::formofx} and the result is proved.\n\t\\end{proof}","start":55049,"end":59028}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"对规范形应用全局 Olver 定理","source_units":[{"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm::MainResLG}]\n\t\tLet $x$ be the solution of \\eqref{eq::QuadeDINAVD} define $p,r$ as in \\eqref{eq::pr}. Let us also define $y(t)\\stackrel{\\textrm{def}}{=}x(t)\\exp\\left(\\int_0^t \\frac{p(s)}{2}\\diff s\\right)$. According to Lemma~\\ref{lem::canonForm}, $r$ is positive and $y$ is solution to \\eqref{eq::ODErewritten}. Then, from Lemma~\\ref{lem::phiintegrable}, $\\int_t^{T} \\vert \\varphi(s)\\vert\\diff s<+\\infty$, so we can apply Theorem~\\ref{thm::olver} to $y$ on $[0,+\\infty[$. Therefore, there exists $A,B\\in\\R$, such that $\\forall t\\geq 0$,\n\t\t$$\n\t\ty(t)= A\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t \\sqrt{r(s)}\\diff s\\right) + B\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t -\\sqrt{r(s)}\\diff s\\right),\n\t\t$$\n\t\twhere $A$ and $B$ are determined by the initial conditions, and $\\delta_1$, $\\delta_2$ are such that \\eqref{eq::deltabounds} holds.\n","start":55049,"end":55937}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["eq::formofx0"],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"逆变换得到未展开的双指数","source_units":[{"raw_tex":"\t\tGoing back to $x(t) = y(t)\\exp\\left(\\int_0^t -\\frac{p(s)}{2}\\diff s\\right)$, we obtain \tthat for all $t\\geq 0$,\n\t\t\\begin{multline}\\label{eq::formofx0}\n\t\t\tx(t) = A\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t -\\frac{p(s)}{2} +\\sqrt{r(s)}\\diff s\\right) \\\\ + B\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\exp\\left(\\int_0^t -\\frac{p(s)}{2} -\\sqrt{r(s)}\\diff s\\right).\n\t\t\\end{multline}\n","start":55937,"end":56313}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["eq::sqrtr"],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"对平方根系数作渐近展开","source_units":[{"raw_tex":"\t\tIt now remains to expand the terms in the two exponentials in \\eqref{eq::formofx0} in order to obtain \\eqref{eq::formofx}. To this aim, we approximate $\\sqrt{r(s)}$, let $s\\geq 0$,\n\t\t\\begin{align}\\label{eq::sqrtr}\n\t\t\t\\begin{split}\n\t\t\t\t\\sqrt{r(s)}&=\\frac{p(s)}{2}\\sqrt{1+ \\frac{2p'(s)}{p(s)^2} - \\frac{4\\lambda}{\\eps(s)p(s)^2}}\n\t\t\t\t\\\\&= \\frac{p(s)}{2}\\left(1 + \\frac{p'(s)}{p(s)^2} - \\frac{2\\lambda}{\\eps(s)p(s)^2} - \\frac{1}{8}\\left(\\frac{2p'(s)}{p(s)^2} - \\frac{4\\lambda}{\\eps(s)p(s)^2}\\right)^2 + o(\\eps(s)^2) \\right)\n\t\t\t\t\\\\ & = \\frac{p(s)}{2} +\\frac{ p'(s)}{2p(s)} - \\frac{\\lambda}{\\eps(s)p(s)} - \\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda}{\\eps(s)p(s)^{3/2}}\\right)^2 + o(\\eps(s))\n\t\t\t\t\\\\  = \\frac{p(s)}{2} &+ \\frac{p'(s)\\eps(s)}{2(\\alpha(s)+\\beta\\lambda)} - \\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2 + o(\\eps(s))\n\t\t\t\t\\\\  = \\frac{p(s)}{2} &+ \\frac{\\alpha'(s)/2-\\lambda}{\\alpha(s)+\\beta\\lambda} -\\frac{\\eps'(s)}{2\\eps(s)}  - \\frac{1}{16}\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2 + o(\\eps(s))\n\t\t\t\\end{split}\n\t\t\\end{align}\n","start":56313,"end":57535}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"积分对数导数并代入平方余项","source_units":[{"raw_tex":"\t\tTo ease the readability, we denote $h(t)=\\left(\\frac{2p'(s)}{p(s)^{3/2}} - \\frac{4\\lambda\\sqrt{\\eps(s)}}{(\\alpha(s)+\\beta\\lambda)^{3/2}}\\right)^2$. Focusing on the first exponential term in \\eqref{eq::formofx0}, we deduce from \\eqref{eq::sqrtr} that for all $t\\geq 0$,\n\t\t\\begin{align*}%\\label{eq::Aexpterm}\n\t\t\t\\begin{split}\n\t\t\t\t&\\exp\\left(\\int_0^t -\\frac{p(s)}{2} +\\sqrt{r(s)}\\diff s\\right)\n\t\t\t\t\\\\&= \\exp\\left(\\int_0^t \\frac{\\alpha'(s)/2}{\\alpha(s)+\\beta\\lambda} -\\frac{\\eps'(s)}{2\\eps(s)} -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{1}{16}h(t) + o(\\eps(s))\\diff s\\right)\n\t\t\t\t\\\\&=  \\frac{\\sqrt{\\alpha(t)+\\beta\\lambda})}{\\sqrt{\\alpha_0+\\beta\\lambda}} \\frac{\\sqrt{\\eps_0}}{\\sqrt{\\eps(t)}} \\exp\\left(\n\t\t\t\t\\int_0^t \\frac{-\\lambda}{\\alpha(s)+\\beta\\lambda} -\n\t\t\t\t\\frac{1}{16}h(t) + o(\\eps(s))\\diff s\\right)\n\t\t\t\t\\\\&=  \\frac{\\sqrt{\\alpha(t)+\\beta\\lambda})}{\\sqrt{\\alpha_0+\\beta\\lambda}} \\frac{\\sqrt{\\eps_0}}{\\sqrt{\\eps(t)}} \\exp\\left(\n\t\t\t\t\\int_0^t \\frac{-\\lambda}{\\alpha(s)+\\beta\\lambda} -\n\t\t\t\t\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}} + o(\\eps(s))\\diff s\\right),\n\t\t\t\\end{split}\n\t\t\\end{align*}\n","start":57535,"end":58639}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"处理第二模态并吸收常数","source_units":[{"raw_tex":"\t\twhere the last line relies on further computations postponed to Lemma~\\ref{lem::remainderterm} in Appendix~\\ref{app::integrability}. Performing the exact same type of computations on $\\exp\\left(\\int_0^t -\\frac{p(s)}{2} -\\sqrt{r(s)}\\diff s\\right)$, and up to redefining $A$ and $B$ so as to encompass all the constants, we obtain \\eqref{eq::formofx} and the result is proved.\n\t\\end{proof}","start":58639,"end":59028}]},{"id":"S049","semantic_type":"theorem","kind":"theorem","labels":["eq::equivx","thm::MainResAsymptotic"],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"质量与阻尼控制的渐近比较定理","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm::MainResAsymptotic}\n\t\tLet $x$ be the solution of \\eqref{eq::QuadeDINAVD}, given in \\eqref{eq::formofx}, and $x_{LM}$ and $x_N$ whose closed forms are stated in \\eqref{eq::closeformLMN}. Under Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp}, there exists $C>0$ such that the following asymptotic equivalences hold:\n\t\t\\begin{align}\\label{eq::equivx}\n\t\t\t\\begin{split}\n\t\t\t\tx(t) &\\sim_{+\\infty}\n\t\t\t\tx_{LM}(t)C\\exp\\left(\\int_0^t - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right),\\quad \\text{and}\n\t\t\t\t\\\\x(t)&\\sim_{+\\infty}\n\t\t\t\tx_{N}(t)C\\exp\\left(\\int_0^t \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}- \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right).\n\t\t\t\\end{split}\n\t\t\\end{align}\n\t\tAs a consequence, the convergence of $x$ to $x^\\star$ is:\n\t\t\\begin{enumerate}[label=(\\roman*)]\n\t\t\t\\item \\emph{Faster} than that of $x_{LM}$\n\t\t\tif $\\eps$ is non-integrable and as fast otherwise.\n\t\t\t\\item Slower than that of $x_{N}$ if $\\alpha$ is non-integrable and as fast if $\\alpha$ is integrable, in the case where $\\forall t\\geq 0$, $\\alpha(t)>\\eps(t)$.\n\t\t\t\\item \\emph{Faster} than that of $x_{N}$ if $\\eps$ is non-integrable and as fast if $\\eps$ is integrable, in the case where $\\forall t\\geq 0$, $\\alpha(t)< \\eps(t)$.\n\t\t\\end{enumerate}\n\t\\end{theorem}","start":59872,"end":61197}]},{"id":"S050","semantic_type":"proof","kind":"proof","labels":["eq::rewritteLM"],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"渐近比较主定理的作者全证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm::MainResAsymptotic}]\n\t\tThanks to Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp}, Theorem~\\ref{thm::MainResLG} tells us that $x$ has the form \\eqref{eq::formofx}. We now analyze the two terms in \\eqref{eq::formofx}.\n\n\t\tFirst, we know from Theorem~\\ref{thm::MainResLG} that $\\delta_1(0)=0$ and $\\lim_{t\\to+\\infty}\\delta_2(t)=0$. In addition, by Lemma~\\ref{lem::phiintegrable}, $\\delta_1$ and $\\delta_2$ are uniformly bounded by some positive constant.\n\t\tThen $r(t)^{-1/4}$ decays asymptotically like $\\sqrt{\\eps(t)}$ and $\\alpha$ is bounded. So\n\t\t$A\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\alpha(t)+\\beta\\lambda}}{\\sqrt{\\eps(t)}}$ is asymptotically equivalent to some constant $c_1\\in\\R$ as $t\\to +\\infty$. Similarly, the factor $B\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\eps(t)}}{\\sqrt{\\alpha(t)+\\beta\\lambda}}$ is equivalent to $c_2\\eps(t)$, with $c_2\\in\\R$.\n\n\t\tWe now analyze the ``exponential factors'' in \\eqref{eq::formofx}.\n\t\tOn the one hand, $\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}+\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))$ converges to $\\frac{1}{\\beta}$ as $s\\to \\infty$, while $\\frac{\\alpha(s)+\\beta\\lambda}{\\eps(s)}$ diverges to $+\\infty$. Therefore, we deduce that,\n\t\t\\begin{multline*}\n\t\t\t\\exp\\left(\\int_0^t -\\frac{\\alpha(s)+\\beta\\lambda}{\\eps(s)}+\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}+\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\n\t\t\t\\\\= o\\left(\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\\right).\n\t\t\\end{multline*}\n\t\tAs a consequence, the second term in \\eqref{eq::formofx} will decrease to $0$ faster than the first-one (let alone the additional $\\eps(t)$ decay that we have just discussed). The asymptotic behavior of $x$ will thus be governed by the first term in \\eqref{eq::formofx}.\n\n\t\tLet us now focus on the first term in \\eqref{eq::formofx}. Observe that\n\t\t$\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}\\diff s\\right)$ is exactly the decay of $x_{LM}$ in \\eqref{eq::closeformLMN}. Thus, we have proved that there exists $C>0$, such that the following asymptotic equivalence holds,\n\t\t\\begin{multline*}%\\label{eq::equivA}\n\t\t\tA\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\alpha(t)+\\beta\\lambda}}{\\sqrt{\\eps(t)}}\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\n\t\t\t\\\\ \\sim_{+\\infty}\n\t\t\tx_{LM}(t)C\\exp\\left(\\int_0^t - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right),\n\t\t\\end{multline*}\n\t\twhich proves the first part of \\eqref{eq::equivx}. The second equivalence in \\eqref{eq::equivx} is obtained using the following identity,\n\t\t\\begin{equation}\\label{eq::rewritteLM}\n\t\t\t\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}\\diff s = \\int_0^t -\\frac{1}{\\beta} + \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}\\diff s = -\\frac{t}{\\beta} + \\int_0^t  \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}\\diff s\n\t\t\\end{equation}\n\t\tand $e^{-t/\\beta}$ is precisely the rate at which $x_N$ decreases. So \\eqref{eq::equivx} holds.\n\n\t\tIt finally remains to deduce the conclusions of the theorem from \\eqref{eq::equivx}.\n\t\t\\\\--\\ Regarding the comparison with $x_{LM}$, the integral $\\int_0^t - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s$ converges if and only if $\\eps$ is integrable on $[0,+\\infty[$, and diverges to $-\\infty$ when $\\eps$ is not. So $x$ converges to $0$ at least as fast as $x_{LM}$ and faster when $\\eps$ is not integrable.\n\t\t\\\\-- As for the comparison with $x_N$, if $\\alpha(s)>\\eps(s)\\geq 0$ for all $s\\geq 0$, then the integral $\\int_0^t \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}- \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s$ is convergent in $+\\infty$ if and only if $\\alpha$ is integrable and diverges to $+\\infty$ when $\\alpha$ is non-integrable. So when $\\alpha$ is integrable, the speed of convergence of $x$ is the same as that of $x_N$. When $\\alpha$ is not integrable, the convergence to $0$ is slower but still holds. Indeed, for all $s\\geq 0$\n\t\t$\\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\frac{1}{\\beta})}< \\frac{\\alpha(s)}{\\beta\\alpha(s)}= \\frac{1}{\\beta}$. Thus for all $t>0$, $-\\frac{t}{\\beta} + \\int_0^t \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}\\diff s<0$.\n\t\t\\\\-- Finally, the comparison with $x_N$ in the case $\\eps(s)> \\alpha(s)$ is exactly the same as the comparison with $x_{LM}$ using \\eqref{eq::rewritteLM}.\n\t\\end{proof}","start":62511,"end":67064}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"调用两模态轨迹表达","source_units":[{"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm::MainResAsymptotic}]\n\t\tThanks to Assumptions~\\ref{ass:smooth} and~\\ref{ass:subexp}, Theorem~\\ref{thm::MainResLG} tells us that $x$ has the form \\eqref{eq::formofx}. We now analyze the two terms in \\eqref{eq::formofx}.\n","start":62511,"end":62769}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"误差函数与前因子的渐近判断","source_units":[{"raw_tex":"\t\tFirst, we know from Theorem~\\ref{thm::MainResLG} that $\\delta_1(0)=0$ and $\\lim_{t\\to+\\infty}\\delta_2(t)=0$. In addition, by Lemma~\\ref{lem::phiintegrable}, $\\delta_1$ and $\\delta_2$ are uniformly bounded by some positive constant.\n\t\tThen $r(t)^{-1/4}$ decays asymptotically like $\\sqrt{\\eps(t)}$ and $\\alpha$ is bounded. So\n\t\t$A\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\alpha(t)+\\beta\\lambda}}{\\sqrt{\\eps(t)}}$ is asymptotically equivalent to some constant $c_1\\in\\R$ as $t\\to +\\infty$. Similarly, the factor $B\\frac{1+\\delta_2(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\eps(t)}}{\\sqrt{\\alpha(t)+\\beta\\lambda}}$ is equivalent to $c_2\\eps(t)$, with $c_2\\in\\R$.\n","start":62770,"end":63421}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"比较指数并宣布首模态主导","source_units":[{"raw_tex":"\t\tWe now analyze the ``exponential factors'' in \\eqref{eq::formofx}.\n\t\tOn the one hand, $\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}+\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))$ converges to $\\frac{1}{\\beta}$ as $s\\to \\infty$, while $\\frac{\\alpha(s)+\\beta\\lambda}{\\eps(s)}$ diverges to $+\\infty$. Therefore, we deduce that,\n\t\t\\begin{multline*}\n\t\t\t\\exp\\left(\\int_0^t -\\frac{\\alpha(s)+\\beta\\lambda}{\\eps(s)}+\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}+\\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\n\t\t\t\\\\= o\\left(\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\\right).\n\t\t\\end{multline*}\n\t\tAs a consequence, the second term in \\eqref{eq::formofx} will decrease to $0$ faster than the first-one (let alone the additional $\\eps(t)$ decay that we have just discussed). The asymptotic behavior of $x$ will thus be governed by the first term in \\eqref{eq::formofx}.\n","start":63422,"end":64415}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"识别 LM 因子并提出正比较常数","source_units":[{"raw_tex":"\t\tLet us now focus on the first term in \\eqref{eq::formofx}. Observe that\n\t\t$\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}\\diff s\\right)$ is exactly the decay of $x_{LM}$ in \\eqref{eq::closeformLMN}. Thus, we have proved that there exists $C>0$, such that the following asymptotic equivalence holds,\n\t\t\\begin{multline*}%\\label{eq::equivA}\n\t\t\tA\\frac{1+\\delta_1(t)}{r(t)^{1/4}}\\frac{\\sqrt{\\alpha(t)+\\beta\\lambda}}{\\sqrt{\\eps(t)}}\\exp\\left(\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda} - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right)\n\t\t\t\\\\ \\sim_{+\\infty}\n\t\t\tx_{LM}(t)C\\exp\\left(\\int_0^t - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s\\right),\n\t\t\\end{multline*}\n","start":64416,"end":65149}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["eq::rewritteLM"],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"分离 CN 衰减指数","source_units":[{"raw_tex":"\t\twhich proves the first part of \\eqref{eq::equivx}. The second equivalence in \\eqref{eq::equivx} is obtained using the following identity,\n\t\t\\begin{equation}\\label{eq::rewritteLM}\n\t\t\t\\int_0^t -\\frac{\\lambda}{\\alpha(s)+\\beta\\lambda}\\diff s = \\int_0^t -\\frac{1}{\\beta} + \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}\\diff s = -\\frac{t}{\\beta} + \\int_0^t  \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}\\diff s\n\t\t\\end{equation}\n\t\tand $e^{-t/\\beta}$ is precisely the rate at which $x_N$ decreases. So \\eqref{eq::equivx} holds.\n","start":65149,"end":65675}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"按质量可积性比较 LM 速度","source_units":[{"raw_tex":"\t\tIt finally remains to deduce the conclusions of the theorem from \\eqref{eq::equivx}.\n\t\t\\\\--\\ Regarding the comparison with $x_{LM}$, the integral $\\int_0^t - \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s$ converges if and only if $\\eps$ is integrable on $[0,+\\infty[$, and diverges to $-\\infty$ when $\\eps$ is not. So $x$ converges to $0$ at least as fast as $x_{LM}$ and faster when $\\eps$ is not integrable.\n","start":65676,"end":66114}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"阻尼较大时的 CN 分类","source_units":[{"raw_tex":"\t\t\\\\-- As for the comparison with $x_N$, if $\\alpha(s)>\\eps(s)\\geq 0$ for all $s\\geq 0$, then the integral $\\int_0^t \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}- \\frac{\\lambda^2\\eps(s)}{(\\alpha(s)+\\beta\\lambda)^{3}}+o(\\eps(s))\\diff s$ is convergent in $+\\infty$ if and only if $\\alpha$ is integrable and diverges to $+\\infty$ when $\\alpha$ is non-integrable. So when $\\alpha$ is integrable, the speed of convergence of $x$ is the same as that of $x_N$. When $\\alpha$ is not integrable, the convergence to $0$ is slower but still holds. Indeed, for all $s\\geq 0$\n\t\t$\\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\frac{1}{\\beta})}< \\frac{\\alpha(s)}{\\beta\\alpha(s)}= \\frac{1}{\\beta}$. Thus for all $t>0$, $-\\frac{t}{\\beta} + \\int_0^t \\frac{\\alpha(s)}{\\beta(\\alpha(s)+\\beta\\lambda)}\\diff s<0$.\n","start":66114,"end":66895}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"质量较大时的 CN 分类","source_units":[{"raw_tex":"\t\t\\\\-- Finally, the comparison with $x_N$ in the case $\\eps(s)> \\alpha(s)$ is exactly the same as the comparison with $x_{LM}$ using \\eqref{eq::rewritteLM}.\n","start":66895,"end":67052}]},{"id":"S059","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S059"],"declared_external":true,"declaration_text":["原始前提列表：Standard spectral decomposition for symmetric positive definite matrices。"],"source_only":false,"title":"对称正定矩阵的谱分解","source_units":[]},{"id":"S060","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S060"],"declared_external":true,"declaration_text":["原始前提列表：linear ODE theory。"],"source_only":false,"title":"线性常微分方程基本理论","source_units":[]},{"id":"S061","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S061"],"declared_external":true,"declaration_text":["原始前提列表中的 elementary asymptotic calculus；基本微积分规则由标注者显式列出。"],"source_only":false,"title":"微分积分与泰勒计算规则","source_units":[]},{"id":"S062","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S062"],"declared_external":true,"declaration_text":["原始前提列表：elementary asymptotic calculus。"],"source_only":false,"title":"指数与积分的渐近比较规则","source_units":[]},{"id":"S063","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S063"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；用于原文 L548、L1003、L1022 的基本积分论操作。"],"source_only":false,"title":"局部积分与有界乘子的控制","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S012":"S012","S014":"S014","S016":"S016","S017":"S017","S018":"S018","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063"}}}}
{"record_id":"2212.06410v2","paper":{"title":"Parameter-free accelerated gradient descent for nonconvex minimization","authors":["Marumo, Naoki","Takeda, Akiko"],"arxiv_id":"2212.06410","version":"v2","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/2212.06410v2","pdf_url":"https://arxiv.org/pdf/2212.06410v2","doi":"10.1137/22M1540934"},"context":{"raw_tex":"This paper studies general nonconvex optimization problems:\n\\begin{align}\n  \\min_{x \\in \\R^d} \\ \n  f(x),\n\\end{align}\nwhere $f: \\R^d \\to \\R$ is bounded below (i.e., $\\inf_{x \\in \\R^d} f(x) > - \\infty$) and satisfies the following Lipschitz assumption.\n\\begin{assumption}\n  \\label{asm:gradient_hessian_lip}\n  Let $\\Ltrue, \\Mtrue > 0$ be constants.\n  \\begin{enuminasm}\n    \\item\n    \\label{asm:gradient_lip}\n    $\\norm*{\\nabla f(x) - \\nabla f(y)} \\leq \\Ltrue \\norm*{x - y}$ for all $x, y \\in \\R^d$,\n    \\item\n    \\label{asm:hessian_lip}\n    $\\norm*{\\nabla^2 f(x) - \\nabla^2 f(y)} \\leq \\Mtrue \\norm*{x - y}$ for all $x, y \\in \\R^d$.\n  \\end{enuminasm}\n\\end{assumption}\n\n\n\\paragraph{Notation}\nFor vectors $a, b \\in \\R^d$, let $\\inner{a}{b}$ denote the dot product and $\\norm{a}$ denote the Euclidean norm.\nFor a matrix $A \\in \\R^{m \\times n}$, let $\\norm{A}$ denote the operator norm, or equivalently the largest singular value.\n\n\n\\begin{lemma}\n  \\label{lem:eq:property_gradlip_obj}\n  Under \\cref{asm:gradient_lip}, the following holds for any $x, y \\in \\R^d$:\n  \\begin{align}\n    f(x) - f(y)\n    &\\leq\n    \\inner*{\\nabla f(y)}{x - y}\n    + \\frac{\\Ltrue}{2} \\norm*{x - y}^2.\n  \\end{align}\n\\end{lemma}\n\n\nThe proposed method is the classical accelerated gradient descent (AGD) \\cite{nesterov1983method,nesterov2018lectures} with two restart schemes, presented in \\cref{alg:proposed_agd}.\nIn the algorithm, the iteration counter $k$ is reset to $0$ when AGD restarts on Line \\ref{alg-line-agd:restart_unsuccessful} or \\ref{alg-line-agd:restart_successful}, whereas the total iteration counter $K$ is not.\nWe refer to the period between a reset of $k$ and the next reset as an \\emph{epoch}.\nNote that the total iteration counter $K$ in the algorithm is unnecessary to implement; it is included here only to make the statements in our analysis concise.\n\nThe algorithm uses estimates $L$ and $M_k$ for the Lipschitz constants $L_f$ and $M_f$.\nThe estimate $L$ is fixed during an epoch, while $M_k$ is updated at each iteration, having the subscript $k$.\nNow let us describe each operation of \\cref{alg:proposed_agd}.\n\n\\paragraph{Solution update (Lines \\ref{alg-line-agd:update_x} and \\ref{alg-line-agd:update_y})}\nWith an estimate $L$ of the Lipschitz constant $L_f$, AGD defines solution sequences $(x_k)$ and $(y_k)$ as follows:\n$y_0 = x_0$ and\n\\begin{align}\n  % \\left\\{\n  \\begin{aligned}\n    x_k\n    &=\n    y_{k-1} - \\frac{1}{L} \\nabla f(y_{k-1}),\\\\\n    y_k\n    &=\n    x_k + \\theta_k (x_k - x_{k-1})\n  \\end{aligned}\n  % \\right.\n  \\label{eq:update_xy}\n\\end{align}\nfor $k \\geq 1$, where $0 \\leq \\theta_k \\leq 1$ is the acceleration parameter.\nLet $x_{-1} \\coloneqq x_0$ for convenience, which makes the lower equation of \\cref{eq:update_xy} valid for $k = 0$.\nSetting a suitable value of $\\theta_k$ is crucial for deriving an improved complexity bound over that of gradient descent.\nIn particular, we propose to set\n\\begin{align}\n  \\theta_k = \\frac{k}{k+1}\n  \\label{eq:def_theta_k}\n\\end{align}\nfor $k \\geq 1$.\n\n\n\\begin{algorithm}[t]\n  \\caption{Proposed AGD with restart~\\label{alg:proposed_agd}} \n  \\begin{algorithmic}[1]\n    \\Require{%\n      \\pbox[t]{40em}{\n        $x_{\\mathrm{init}} \\in \\R^d$;\n        $\\Linit, M_0 > 0$;\n        $\\alpha > 1$;\n        $0 < \\beta \\leq 1$\\\\\n        Recommended: $(\\Linit, M_0, \\alpha, \\beta) = (10^{-3}, 10^{-16}, 2, 0.9)$\n      }\n    }\n    \\State{%\n      $(x_0, y_0) \\gets (x_{\\mathrm{init}}, x_{\\mathrm{init}})$,\n      $\\Lest \\gets \\Linit$,\n      $k \\gets 0$,\n      $K \\gets 0$\n    }\n    \\Repeat\n      \\State{%\n        $k \\gets k + 1$,\n        $K \\gets K + 1$\n      }\n      \\State $x_k \\gets y_{k-1} - \\frac{1}{\\Lest} \\nabla f(y_{k-1})$\n      \\label{alg-line-agd:update_x}\n      \\State $y_k \\gets x_k + \\frac{k}{k+1} (x_k - x_{k-1})$\n      \\label{alg-line-agd:update_y}\n\n      \\If{$f(x_k) > f(x_0) - \\frac{\\Lest S_k}{2 (k+1)}$}\n      \\Comment{$S_k \\coloneqq \\sum_{i=1}^k \\norm*{x_i - x_{i-1}}^2$}\n      \\label{alg-line-agd:check_decrease}\n        \\State{%\n          $(x_0, y_0) \\gets (x_{k-1}, x_{k-1})$,\n          $\\Lest \\gets \\alpha \\Lest$,\n          $k \\gets 0$\n        }\n        \\label{alg-line-agd:restart_unsuccessful}\n      \\ElsIf{$(k+1)^5 \\Mest_k^2 S_k > \\Lest^2$}\n      \\Comment{$\\Mest_k$ is defined by \\cref{eq:update_Mest_practical} or \\cref{eq:update_Mest}}\n      \\label{alg-line-agd:check_large_Sk}\n        \\State{%\n          $(x_0, y_0) \\gets (x_k, x_k)$,\n          $\\Lest \\gets \\beta \\Lest$,\n          $k \\gets 0$\n        }\n        \\label{alg-line-agd:restart_successful}\n      \\EndIf\n    \\Until{convergence}\n    \\State \\Return averaged solution $\\bar y_k$ defined by \\eqref{eq:def_ybark}\n  \\end{algorithmic}\n\\end{algorithm}\n\n\n\\paragraph{Restart mechanism 1 (Lines \\ref{alg-line-agd:check_decrease} and \\ref{alg-line-agd:restart_unsuccessful})}\nAfter updating the solutions, \\cref{alg:proposed_agd} checks whether the current estimate $L$ approximates the actual value $L_f$ well.\nIf\n\\begin{align}\n  f(x_k) - f(x_0)\n  \\leq\n  - \\frac{1 - \\theta_k}{2} \\Lest S_k\n  =\n  - \\frac{\\Lest S_k}{2 (k+1)},\n  \\quad\\text{where}\\quad\n  S_k\n  \\coloneqq\n  \\sum_{i=1}^k \\norm*{x_i - x_{i-1}}^2,\n  \\label{eq:descent_condition}\n\\end{align}\nthen $L$ is judged suitable, and the epoch continues; otherwise, AGD restarts from $x_{k-1}$ with a larger $L$.\nSince condition~\\cref{eq:descent_condition} holds in the previous iteration, starting the next epoch from $x_{k-1}$ ensures that this epoch does not increase the objective function value.\nCondition~\\cref{eq:descent_condition} guarantees a sufficient decrease in the objective function in the epoch and plays a crucial role in our analysis.\nAs will be shown later, this condition is always satisfied when $L$ is sufficiently large, which justifies the restart mechanism.\n\n\n\\paragraph{Restart mechanism 2 (Lines \\ref{alg-line-agd:check_large_Sk} and \\ref{alg-line-agd:restart_successful})}\nOur AGD also resets the effect of acceleration by restart when $S_k$ becomes large; the restart condition involves an estimate $M_k$ of the Hessian's Lipschitz constant $M_f$ (detailed later).\nIf\n\\begin{align}\n  \\Mest_k^2 S_k\n  \\leq\n  \\Lest^2 (1 - \\theta_k)^5\n  =\n  \\frac{\\Lest^2}{(k+1)^5},\n  \\label{eq:condition_continue}\n\\end{align}\nthe epoch continues, otherwise, AGD restarts with the computed solution $x_k$ and a (possibly) smaller $L$.\nCondition~\\cref{eq:condition_continue} is inspired by a condition in \\cite{li2022restarted}, $k M_f S_k \\leq \\epsilon$; unlike the existing one, our condition does not involve $M_f$ or $\\epsilon$.\nAlthough $L$ does not have to be decreased on Line~\\ref{alg-line-agd:restart_successful} to derive the complexity bound of $O(\\epsilon^{-7/4})$, decreasing $L$ will help the algorithm to capture the local curvature of $f$ and to converge faster empirically.\n\n\n\\paragraph{Averaged solution}\n\\label{sec:averaged_solution}\nOur AGD guarantees that the gradient norm is small at a weighted average $\\bar y_k$ of the solutions $y_0,\\dots,y_{k-1}$.\nThe averaged solution $\\bar y_k$ is defined by\n\\begin{align}\n  \\bar y_k\n  \\coloneqq\n  \\sum_{i=0}^{k-1} p_{k,i} y_i,\n  \\quad\\text{where}\\quad\n  p_{k,i}\n  \\coloneqq\n  \\frac{1}{Z_k} \\prod_{j=i+1}^{k-1} \\theta_j,\n  \\quad\n  Z_k\n  \\coloneqq\n  \\sum_{i=0}^{k-1} \\prod_{j=i+1}^{k-1} \\theta_j.\n  \\label{eq:def_ybark}\n\\end{align}\nNote that it is unnecessary to keep all $y_0,\\dots,y_{k-1}$ in memory to compute $\\bar y_k$ because $Z_k$ and $\\bar y_k$ satisfy a simple recursion:\n\\begin{align}\n  Z_{k+1}\n  &=\n  1 + \\theta_k Z_k,\\quad\n  \\bar y_{k+1}\n  =\n  \\frac{1}{Z_{k+1}} \\prn*{y_k + \\theta_k Z_k \\bar y_k},\n\\end{align}\nwhich maintains the computational efficiency of the algorithm.\nGiven \\cref{eq:def_theta_k}, there is another form:\n\\begin{align}\n  \\bar y_k\n  =\n  \\frac{2}{k (k + 1)}\n  \\sum_{i=0}^{k-1} (i + 1) y_i,\\quad\n  Z_k\n  =\n  \\frac{k + 1}{2},\n  \\label{eq:ybar_Zk_alternative}\n\\end{align}\nand this simplification of $\\bar y_k$ and $Z_k$ is one of the aims of setting $\\theta_k$ as in \\cref{eq:def_theta_k}.\n\n\n\\paragraph{Estimate of Hessian's Lipschitz constant}\nThe estimate $M_k$ for the Hessian is updated at each iteration, unlike $L$ for the gradient.\nOur complexity analysis requires three inequalities on $M_k$: $M_{k-1} \\leq M_k$ and\n\\begin{align}\n  &\n  f(y_k) - f(x_k)\n  \\leq\n  \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n  + \\frac{\\Mest_k}{12} \\norm*{y_k - x_k}^3,\n  \\label{eq:fy_fx_diff_upperbound_Mestk}\\\\\n  &\n  \\norm*{\n    \\nabla f(y_k)\n    + \\theta_k \\nabla f(x_{k-1})\n    - (1 + \\theta_k) \\nabla f (x_k)\n  }\n  \\leq\n  \\theta_k \\Mest_k \\norm*{x_k - x_{k-1}}^2.\n  \\label{eq:grad_interpolation_error_Mestk}\n\\end{align}\nFinding the smallest $M_k$ that satisfies these inequalities is desirable, and fortunately, it is straightforward,\n\\begin{align}\n  \\label{eq:update_Mest_practical}\n  \\Mest_k\n  =\n  \\max \\bigg\\{\n    \\Mest_{k-1},\\ \n    &\n    \\frac{\n      12\n      \\prn*{\n        f(y_k) - f(x_k) \n        - \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n      }\n    }{\\norm*{y_k - x_k}^3},\\\\\n    &\n    \\frac{\n      \\norm*{\n        \\nabla f(y_k)\n        + \\theta_k \\nabla f(x_{k-1})\n        - (1 + \\theta_k) \\nabla f (x_k)\n      }\n    }{\\theta_k \\norm*{x_k - x_{k-1}}^2}\n  \\bigg\\}\n\\end{align}\nbecause $x_k$, $y_k$, and $\\theta_k$ are already in hand.\nIn other words, our $\\theta_k$ in \\cref{eq:def_theta_k} does not depend on the estimate $M_k$ of $M_f$, which facilitates computation of $M_k$.\nThus, we can estimate $M_f$ without backtracking, thereby simplifying the algorithm and improving its efficiency.\n\nInstead of \\cref{eq:update_Mest_practical}, we can use\n\\begin{align}\n  \\label{eq:update_Mest}\n  \\Mest_k\n  =\n  \\max \\bigg\\{\n    \\Mest_{k-1},\\ \n    &\n    \\frac{\n      12\n      \\prn*{\n        f(y_k) - f(x_k) \n        - \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n      }\n    }{\\norm*{y_k - x_k}^3}\n    ,\\\\\n    &\n    \\frac{\n      \\norm*{\n        \\nabla f(y_k)\n        + \\theta_k \\nabla f(x_{k-1})\n        - (1 + \\theta_k) \\nabla f (x_k)\n      }\n    }{\\theta_k \\norm*{x_k - x_{k-1}}^2},\\\\\n    &\n    \\frac{\n      16\n      \\prn*{\n        Z_k^2 \\norm*{ \\nabla f (\\bar y_k) }\n        - Z_k \\Lest \\norm*{x_k - x_{k-1}}\n      }\n    }{(k-1) (k+5)^2 S_k}\n  \\bigg\\}.\n\\end{align}\nin the algorithm.\\footnote{\n  For $k = 1$, \\Cref{eq:update_Mest} takes the form $M_1 = \\max \\set{M_0, \\ast, \\ast, \\frac{0}{0}}$, which should be treated as $M_1 = \\max \\set{M_0, \\ast, \\ast}$.\n}\nThis replacement ensures that\n\\begin{align}\n  \\norm*{ \\nabla f (\\bar y_k) }\n  \\leq\n  \\frac{\\Lest}{Z_k} \\norm*{x_k - x_{k-1}}\n  + \\frac{(k-1) (k+5)^2}{16 Z_k^2} \\Mest_k S_k\n  \\label{eq:grad_ybark_upperbound_Mestk}\n\\end{align}\nand will lead to a smaller complexity bound with lessened dependence on $M_0$ and $M_f$, as will be confirmed later in \\cref{thm:complexity}.\n\\Cref{eq:update_Mest} derives a theoretically better bound than \\cref{eq:update_Mest_practical} but requires an additional gradient evaluation at $\\bar y_k$, increasing the computational cost per iteration.\nTherefore, we recommend using \\cref{eq:update_Mest_practical} in practice.\n\n\nTo obtain the descent condition~\\cref{eq:descent_condition}, we first evaluate the decrease for one iteration.\nHowever, a common difficulty arises when analyzing AGD: the objective function value does not necessarily decrease monotonically.\nTo deal with the problem, we introduce a potential function related to $f(x_k)$ and show that it is \\emph{nearly} decreasing.\nThe potential function $\\Phi_k$ is defined by\n\\begin{align}\n  \\Phi_k\n  \\coloneqq\n  f(x_k)\n  + \\frac{\\theta_k^2}{2} \\prn*{\n    \\inner*{\\nabla f(x_{k-1})}{x_k - x_{k-1}}\n    + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_{k-1})}^2\n    + \\Lest \\norm*{x_k - x_{k-1}}^2\n  },\n  \\label{eq:def_potential}\n\\end{align}\ninspired by \\cref{eq:hint_potential} used in the proof.\nThe value $\\Phi_k$ decreases when $\\norm*{x_k - x_{k-1}}$ is small enough as the following lemma shows.\n\n\nTo evaluate the terms of $\\norm*{x_k - x_{k-1}}^3$ and $\\norm*{x_k - x_{k-1}}^4$ in \\cref{lem:potential_decrease_iteration} in these calculations, we use \n\\begin{align}\n  \\Mest_{k-1}^2 S_{k-1}\n  \\leq\n  \\Lest^2 (1 - \\theta_{k-1})^5\n  =\n  \\frac{\\Lest^2}{k^5},\n  \\label{eq:condition_continue_k-1}\n\\end{align}\nwhich follows from the fact that the epoch did not end at iteration $k-1$.\n\nNow, we prove \\cref{lem:potential_decrease_iteration} and use it to prove \\cref{lem:decrease_epoch}.\nThe following proofs do not use the specific form \\cref{eq:def_theta_k} of $\\theta_k$ but rather a more general condition on $\\theta_k$,\n\\begin{align}\n  \\theta_{k+1}^2 \\leq \\theta_k \\leq \\theta_{k+1},\n  \\label{eq:theta_increase_not_too_fast}\n\\end{align}\nwhich is easily verified for all $k \\geq 0$ under $\\theta_0 \\coloneqq \\theta_1^2$ and \\cref{eq:def_theta_k} for $k \\geq 1$.\n\n\n\\subsection{Upper bound on gradient norm}\nLet $\\Lmax$ and $\\Mmax$ be the upper bounds on $L$ and $M_k$ given in \\cref{prop:L_upperbound,prop:Mk_upperbound}: $\\Lmax \\coloneqq \\max\\set{\\Linit, \\alpha \\Ltrue}$ and $\\Mmax \\coloneqq \\max\\set{M_0, \\Mtrue}$.\nThe following lemma provides upper bounds on the gradient norm.\n","units":[{"raw_tex":"This paper studies general nonconvex optimization problems:\n\\begin{align}\n  \\min_{x \\in \\R^d} \\ \n  f(x),\n\\end{align}\nwhere $f: \\R^d \\to \\R$ is bounded below (i.e., $\\inf_{x \\in \\R^d} f(x) > - \\infty$) and satisfies the following Lipschitz assumption.\n\\begin{assumption}\n  \\label{asm:gradient_hessian_lip}\n  Let $\\Ltrue, \\Mtrue > 0$ be constants.\n  \\begin{enuminasm}\n    \\item\n    \\label{asm:gradient_lip}\n    $\\norm*{\\nabla f(x) - \\nabla f(y)} \\leq \\Ltrue \\norm*{x - y}$ for all $x, y \\in \\R^d$,\n    \\item\n    \\label{asm:hessian_lip}\n    $\\norm*{\\nabla^2 f(x) - \\nabla^2 f(y)} \\leq \\Mtrue \\norm*{x - y}$ for all $x, y \\in \\R^d$.\n  \\end{enuminasm}\n\\end{assumption}\n","start":14872,"end":15536},{"raw_tex":"\\paragraph{Notation}\nFor vectors $a, b \\in \\R^d$, let $\\inner{a}{b}$ denote the dot product and $\\norm{a}$ denote the Euclidean norm.\nFor a matrix $A \\in \\R^{m \\times n}$, let $\\norm{A}$ denote the operator norm, or equivalently the largest singular value.\n","start":21361,"end":21618},{"raw_tex":"\\begin{lemma}\n  \\label{lem:eq:property_gradlip_obj}\n  Under \\cref{asm:gradient_lip}, the following holds for any $x, y \\in \\R^d$:\n  \\begin{align}\n    f(x) - f(y)\n    &\\leq\n    \\inner*{\\nabla f(y)}{x - y}\n    + \\frac{\\Ltrue}{2} \\norm*{x - y}^2.\n  \\end{align}\n\\end{lemma}\n","start":32617,"end":32887},{"raw_tex":"The proposed method is the classical accelerated gradient descent (AGD) \\cite{nesterov1983method,nesterov2018lectures} with two restart schemes, presented in \\cref{alg:proposed_agd}.\nIn the algorithm, the iteration counter $k$ is reset to $0$ when AGD restarts on Line \\ref{alg-line-agd:restart_unsuccessful} or \\ref{alg-line-agd:restart_successful}, whereas the total iteration counter $K$ is not.\nWe refer to the period between a reset of $k$ and the next reset as an \\emph{epoch}.\nNote that the total iteration counter $K$ in the algorithm is unnecessary to implement; it is included here only to make the statements in our analysis concise.\n\nThe algorithm uses estimates $L$ and $M_k$ for the Lipschitz constants $L_f$ and $M_f$.\nThe estimate $L$ is fixed during an epoch, while $M_k$ is updated at each iteration, having the subscript $k$.\nNow let us describe each operation of \\cref{alg:proposed_agd}.\n\n\\paragraph{Solution update (Lines \\ref{alg-line-agd:update_x} and \\ref{alg-line-agd:update_y})}\nWith an estimate $L$ of the Lipschitz constant $L_f$, AGD defines solution sequences $(x_k)$ and $(y_k)$ as follows:\n$y_0 = x_0$ and\n\\begin{align}\n  % \\left\\{\n  \\begin{aligned}\n    x_k\n    &=\n    y_{k-1} - \\frac{1}{L} \\nabla f(y_{k-1}),\\\\\n    y_k\n    &=\n    x_k + \\theta_k (x_k - x_{k-1})\n  \\end{aligned}\n  % \\right.\n  \\label{eq:update_xy}\n\\end{align}\nfor $k \\geq 1$, where $0 \\leq \\theta_k \\leq 1$ is the acceleration parameter.\nLet $x_{-1} \\coloneqq x_0$ for convenience, which makes the lower equation of \\cref{eq:update_xy} valid for $k = 0$.\nSetting a suitable value of $\\theta_k$ is crucial for deriving an improved complexity bound over that of gradient descent.\nIn particular, we propose to set\n\\begin{align}\n  \\theta_k = \\frac{k}{k+1}\n  \\label{eq:def_theta_k}\n\\end{align}\nfor $k \\geq 1$.\n","start":39638,"end":41440},{"raw_tex":"\\begin{algorithm}[t]\n  \\caption{Proposed AGD with restart~\\label{alg:proposed_agd}} \n  \\begin{algorithmic}[1]\n    \\Require{%\n      \\pbox[t]{40em}{\n        $x_{\\mathrm{init}} \\in \\R^d$;\n        $\\Linit, M_0 > 0$;\n        $\\alpha > 1$;\n        $0 < \\beta \\leq 1$\\\\\n        Recommended: $(\\Linit, M_0, \\alpha, \\beta) = (10^{-3}, 10^{-16}, 2, 0.9)$\n      }\n    }\n    \\State{%\n      $(x_0, y_0) \\gets (x_{\\mathrm{init}}, x_{\\mathrm{init}})$,\n      $\\Lest \\gets \\Linit$,\n      $k \\gets 0$,\n      $K \\gets 0$\n    }\n    \\Repeat\n      \\State{%\n        $k \\gets k + 1$,\n        $K \\gets K + 1$\n      }\n      \\State $x_k \\gets y_{k-1} - \\frac{1}{\\Lest} \\nabla f(y_{k-1})$\n      \\label{alg-line-agd:update_x}\n      \\State $y_k \\gets x_k + \\frac{k}{k+1} (x_k - x_{k-1})$\n      \\label{alg-line-agd:update_y}\n\n      \\If{$f(x_k) > f(x_0) - \\frac{\\Lest S_k}{2 (k+1)}$}\n      \\Comment{$S_k \\coloneqq \\sum_{i=1}^k \\norm*{x_i - x_{i-1}}^2$}\n      \\label{alg-line-agd:check_decrease}\n        \\State{%\n          $(x_0, y_0) \\gets (x_{k-1}, x_{k-1})$,\n          $\\Lest \\gets \\alpha \\Lest$,\n          $k \\gets 0$\n        }\n        \\label{alg-line-agd:restart_unsuccessful}\n      \\ElsIf{$(k+1)^5 \\Mest_k^2 S_k > \\Lest^2$}\n      \\Comment{$\\Mest_k$ is defined by \\cref{eq:update_Mest_practical} or \\cref{eq:update_Mest}}\n      \\label{alg-line-agd:check_large_Sk}\n        \\State{%\n          $(x_0, y_0) \\gets (x_k, x_k)$,\n          $\\Lest \\gets \\beta \\Lest$,\n          $k \\gets 0$\n        }\n        \\label{alg-line-agd:restart_successful}\n      \\EndIf\n    \\Until{convergence}\n    \\State \\Return averaged solution $\\bar y_k$ defined by \\eqref{eq:def_ybark}\n  \\end{algorithmic}\n\\end{algorithm}\n","start":41876,"end":43540},{"raw_tex":"\\paragraph{Restart mechanism 1 (Lines \\ref{alg-line-agd:check_decrease} and \\ref{alg-line-agd:restart_unsuccessful})}\nAfter updating the solutions, \\cref{alg:proposed_agd} checks whether the current estimate $L$ approximates the actual value $L_f$ well.\nIf\n\\begin{align}\n  f(x_k) - f(x_0)\n  \\leq\n  - \\frac{1 - \\theta_k}{2} \\Lest S_k\n  =\n  - \\frac{\\Lest S_k}{2 (k+1)},\n  \\quad\\text{where}\\quad\n  S_k\n  \\coloneqq\n  \\sum_{i=1}^k \\norm*{x_i - x_{i-1}}^2,\n  \\label{eq:descent_condition}\n\\end{align}\nthen $L$ is judged suitable, and the epoch continues; otherwise, AGD restarts from $x_{k-1}$ with a larger $L$.\nSince condition~\\cref{eq:descent_condition} holds in the previous iteration, starting the next epoch from $x_{k-1}$ ensures that this epoch does not increase the objective function value.\nCondition~\\cref{eq:descent_condition} guarantees a sufficient decrease in the objective function in the epoch and plays a crucial role in our analysis.\nAs will be shown later, this condition is always satisfied when $L$ is sufficiently large, which justifies the restart mechanism.\n","start":43542,"end":44618},{"raw_tex":"\\paragraph{Restart mechanism 2 (Lines \\ref{alg-line-agd:check_large_Sk} and \\ref{alg-line-agd:restart_successful})}\nOur AGD also resets the effect of acceleration by restart when $S_k$ becomes large; the restart condition involves an estimate $M_k$ of the Hessian's Lipschitz constant $M_f$ (detailed later).\nIf\n\\begin{align}\n  \\Mest_k^2 S_k\n  \\leq\n  \\Lest^2 (1 - \\theta_k)^5\n  =\n  \\frac{\\Lest^2}{(k+1)^5},\n  \\label{eq:condition_continue}\n\\end{align}\nthe epoch continues, otherwise, AGD restarts with the computed solution $x_k$ and a (possibly) smaller $L$.\nCondition~\\cref{eq:condition_continue} is inspired by a condition in \\cite{li2022restarted}, $k M_f S_k \\leq \\epsilon$; unlike the existing one, our condition does not involve $M_f$ or $\\epsilon$.\nAlthough $L$ does not have to be decreased on Line~\\ref{alg-line-agd:restart_successful} to derive the complexity bound of $O(\\epsilon^{-7/4})$, decreasing $L$ will help the algorithm to capture the local curvature of $f$ and to converge faster empirically.\n","start":45311,"end":46325},{"raw_tex":"\\paragraph{Averaged solution}\n\\label{sec:averaged_solution}\nOur AGD guarantees that the gradient norm is small at a weighted average $\\bar y_k$ of the solutions $y_0,\\dots,y_{k-1}$.\nThe averaged solution $\\bar y_k$ is defined by\n\\begin{align}\n  \\bar y_k\n  \\coloneqq\n  \\sum_{i=0}^{k-1} p_{k,i} y_i,\n  \\quad\\text{where}\\quad\n  p_{k,i}\n  \\coloneqq\n  \\frac{1}{Z_k} \\prod_{j=i+1}^{k-1} \\theta_j,\n  \\quad\n  Z_k\n  \\coloneqq\n  \\sum_{i=0}^{k-1} \\prod_{j=i+1}^{k-1} \\theta_j.\n  \\label{eq:def_ybark}\n\\end{align}\nNote that it is unnecessary to keep all $y_0,\\dots,y_{k-1}$ in memory to compute $\\bar y_k$ because $Z_k$ and $\\bar y_k$ satisfy a simple recursion:\n\\begin{align}\n  Z_{k+1}\n  &=\n  1 + \\theta_k Z_k,\\quad\n  \\bar y_{k+1}\n  =\n  \\frac{1}{Z_{k+1}} \\prn*{y_k + \\theta_k Z_k \\bar y_k},\n\\end{align}\nwhich maintains the computational efficiency of the algorithm.\nGiven \\cref{eq:def_theta_k}, there is another form:\n\\begin{align}\n  \\bar y_k\n  =\n  \\frac{2}{k (k + 1)}\n  \\sum_{i=0}^{k-1} (i + 1) y_i,\\quad\n  Z_k\n  =\n  \\frac{k + 1}{2},\n  \\label{eq:ybar_Zk_alternative}\n\\end{align}\nand this simplification of $\\bar y_k$ and $Z_k$ is one of the aims of setting $\\theta_k$ as in \\cref{eq:def_theta_k}.\n","start":46327,"end":47513},{"raw_tex":"\\paragraph{Estimate of Hessian's Lipschitz constant}\nThe estimate $M_k$ for the Hessian is updated at each iteration, unlike $L$ for the gradient.\nOur complexity analysis requires three inequalities on $M_k$: $M_{k-1} \\leq M_k$ and\n\\begin{align}\n  &\n  f(y_k) - f(x_k)\n  \\leq\n  \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n  + \\frac{\\Mest_k}{12} \\norm*{y_k - x_k}^3,\n  \\label{eq:fy_fx_diff_upperbound_Mestk}\\\\\n  &\n  \\norm*{\n    \\nabla f(y_k)\n    + \\theta_k \\nabla f(x_{k-1})\n    - (1 + \\theta_k) \\nabla f (x_k)\n  }\n  \\leq\n  \\theta_k \\Mest_k \\norm*{x_k - x_{k-1}}^2.\n  \\label{eq:grad_interpolation_error_Mestk}\n\\end{align}\nFinding the smallest $M_k$ that satisfies these inequalities is desirable, and fortunately, it is straightforward,\n\\begin{align}\n  \\label{eq:update_Mest_practical}\n  \\Mest_k\n  =\n  \\max \\bigg\\{\n    \\Mest_{k-1},\\ \n    &\n    \\frac{\n      12\n      \\prn*{\n        f(y_k) - f(x_k) \n        - \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n      }\n    }{\\norm*{y_k - x_k}^3},\\\\\n    &\n    \\frac{\n      \\norm*{\n        \\nabla f(y_k)\n        + \\theta_k \\nabla f(x_{k-1})\n        - (1 + \\theta_k) \\nabla f (x_k)\n      }\n    }{\\theta_k \\norm*{x_k - x_{k-1}}^2}\n  \\bigg\\}\n\\end{align}\nbecause $x_k$, $y_k$, and $\\theta_k$ are already in hand.\nIn other words, our $\\theta_k$ in \\cref{eq:def_theta_k} does not depend on the estimate $M_k$ of $M_f$, which facilitates computation of $M_k$.\nThus, we can estimate $M_f$ without backtracking, thereby simplifying the algorithm and improving its efficiency.\n\nInstead of \\cref{eq:update_Mest_practical}, we can use\n\\begin{align}\n  \\label{eq:update_Mest}\n  \\Mest_k\n  =\n  \\max \\bigg\\{\n    \\Mest_{k-1},\\ \n    &\n    \\frac{\n      12\n      \\prn*{\n        f(y_k) - f(x_k) \n        - \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n      }\n    }{\\norm*{y_k - x_k}^3}\n    ,\\\\\n    &\n    \\frac{\n      \\norm*{\n        \\nabla f(y_k)\n        + \\theta_k \\nabla f(x_{k-1})\n        - (1 + \\theta_k) \\nabla f (x_k)\n      }\n    }{\\theta_k \\norm*{x_k - x_{k-1}}^2},\\\\\n    &\n    \\frac{\n      16\n      \\prn*{\n        Z_k^2 \\norm*{ \\nabla f (\\bar y_k) }\n        - Z_k \\Lest \\norm*{x_k - x_{k-1}}\n      }\n    }{(k-1) (k+5)^2 S_k}\n  \\bigg\\}.\n\\end{align}\nin the algorithm.\\footnote{\n  For $k = 1$, \\Cref{eq:update_Mest} takes the form $M_1 = \\max \\set{M_0, \\ast, \\ast, \\frac{0}{0}}$, which should be treated as $M_1 = \\max \\set{M_0, \\ast, \\ast}$.\n}\nThis replacement ensures that\n\\begin{align}\n  \\norm*{ \\nabla f (\\bar y_k) }\n  \\leq\n  \\frac{\\Lest}{Z_k} \\norm*{x_k - x_{k-1}}\n  + \\frac{(k-1) (k+5)^2}{16 Z_k^2} \\Mest_k S_k\n  \\label{eq:grad_ybark_upperbound_Mestk}\n\\end{align}\nand will lead to a smaller complexity bound with lessened dependence on $M_0$ and $M_f$, as will be confirmed later in \\cref{thm:complexity}.\n\\Cref{eq:update_Mest} derives a theoretically better bound than \\cref{eq:update_Mest_practical} but requires an additional gradient evaluation at $\\bar y_k$, increasing the computational cost per iteration.\nTherefore, we recommend using \\cref{eq:update_Mest_practical} in practice.\n","start":47515,"end":50582},{"raw_tex":"To obtain the descent condition~\\cref{eq:descent_condition}, we first evaluate the decrease for one iteration.\nHowever, a common difficulty arises when analyzing AGD: the objective function value does not necessarily decrease monotonically.\nTo deal with the problem, we introduce a potential function related to $f(x_k)$ and show that it is \\emph{nearly} decreasing.\nThe potential function $\\Phi_k$ is defined by\n\\begin{align}\n  \\Phi_k\n  \\coloneqq\n  f(x_k)\n  + \\frac{\\theta_k^2}{2} \\prn*{\n    \\inner*{\\nabla f(x_{k-1})}{x_k - x_{k-1}}\n    + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_{k-1})}^2\n    + \\Lest \\norm*{x_k - x_{k-1}}^2\n  },\n  \\label{eq:def_potential}\n\\end{align}\ninspired by \\cref{eq:hint_potential} used in the proof.\nThe value $\\Phi_k$ decreases when $\\norm*{x_k - x_{k-1}}$ is small enough as the following lemma shows.\n","start":51624,"end":52451},{"raw_tex":"To evaluate the terms of $\\norm*{x_k - x_{k-1}}^3$ and $\\norm*{x_k - x_{k-1}}^4$ in \\cref{lem:potential_decrease_iteration} in these calculations, we use \n\\begin{align}\n  \\Mest_{k-1}^2 S_{k-1}\n  \\leq\n  \\Lest^2 (1 - \\theta_{k-1})^5\n  =\n  \\frac{\\Lest^2}{k^5},\n  \\label{eq:condition_continue_k-1}\n\\end{align}\nwhich follows from the fact that the epoch did not end at iteration $k-1$.\n\nNow, we prove \\cref{lem:potential_decrease_iteration} and use it to prove \\cref{lem:decrease_epoch}.\nThe following proofs do not use the specific form \\cref{eq:def_theta_k} of $\\theta_k$ but rather a more general condition on $\\theta_k$,\n\\begin{align}\n  \\theta_{k+1}^2 \\leq \\theta_k \\leq \\theta_{k+1},\n  \\label{eq:theta_increase_not_too_fast}\n\\end{align}\nwhich is easily verified for all $k \\geq 0$ under $\\theta_0 \\coloneqq \\theta_1^2$ and \\cref{eq:def_theta_k} for $k \\geq 1$.\n","start":53349,"end":54210},{"raw_tex":"\\subsection{Upper bound on gradient norm}\nLet $\\Lmax$ and $\\Mmax$ be the upper bounds on $L$ and $M_k$ given in \\cref{prop:L_upperbound,prop:Mk_upperbound}: $\\Lmax \\coloneqq \\max\\set{\\Linit, \\alpha \\Ltrue}$ and $\\Mmax \\coloneqq \\max\\set{M_0, \\Mtrue}$.\nThe following lemma provides upper bounds on the gradient norm.\n","start":70200,"end":70516}]},"statement":{"raw_tex":"\\begin{theorem}\n  \\label{thm:complexity}\n  Suppose that \\cref{asm:gradient_hessian_lip} holds and let $\\Lmax \\coloneqq \\max\\set{\\Linit, \\alpha \\Ltrue}$, $\\Mmax \\coloneqq \\max\\set{M_0, \\Mtrue}$, $\\Delta \\coloneqq f(x_\\mathrm{init}) - \\inf_{x \\in \\R^d} f(x)$,\n  \\begin{align}\n    c_1\n    \\coloneqq\n    \\log_\\alpha \\prn*{\\frac{1}{\\beta}},\n    \\quad\\text{and}\\quad\n    c_2\n    \\coloneqq\n    \\log_\\alpha \\prn*{\\frac{\\Lmax}{\\Linit}}.\n    \\label{eq:def_c1_c2}\n  \\end{align}\n  In \\cref{alg:proposed_agd}, when $\\bar y_k$ defined by \\cref{eq:def_ybark} satisfies $\\norm*{\\nabla f(\\bar y_k)} \\leq \\epsilon$ for the first time, the total iteration count $K$ is at most\n  \\begin{align}\n    \\prn[\\bigg]{\n      70\n      + \n      108 c_1 \\frac{\\Mmax^{1/4}}{M_0^{1/4}}\n    }\n    \\frac{\\Lmax^{1/2} \\Mmax^2 \\Delta}{\\epsilon^{7/4} M_0^{7/4}}\n    + 2 (c_2 + 1) \\frac{\\Lmax^{1/2} \\Mmax^{1/4}}{\\epsilon^{1/4} M_0^{1/2}}.\n    \\label{eq:complexity_bound}\n  \\end{align}\n  If $M_k$ is computed with \\cref{eq:update_Mest} in the algorithm and $\\beta$ is set as $\\beta = 1$, the above bound is improved to\n  \\begin{align}\n    70\n    \\frac{\\Lmax^{1/2} \\Mmax^{1/4} \\Delta}{\\epsilon^{7/4}}\n    +\n    2 (c_2 + 1) \\frac{\\Lmax^{1/2}}{M_0^{1/4} \\epsilon^{1/4}}.\n    \\label{eq:complexity_bound_tight}\n  \\end{align}\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\n  \\label{thm:complexity}\n  Suppose that \\cref{asm:gradient_hessian_lip} holds and let $\\Lmax \\coloneqq \\max\\set{\\Linit, \\alpha \\Ltrue}$, $\\Mmax \\coloneqq \\max\\set{M_0, \\Mtrue}$, $\\Delta \\coloneqq f(x_\\mathrm{init}) - \\inf_{x \\in \\R^d} f(x)$,\n  \\begin{align}\n    c_1\n    \\coloneqq\n    \\log_\\alpha \\prn*{\\frac{1}{\\beta}},\n    \\quad\\text{and}\\quad\n    c_2\n    \\coloneqq\n    \\log_\\alpha \\prn*{\\frac{\\Lmax}{\\Linit}}.\n    \\label{eq:def_c1_c2}\n  \\end{align}\n  In \\cref{alg:proposed_agd}, when $\\bar y_k$ defined by \\cref{eq:def_ybark} satisfies $\\norm*{\\nabla f(\\bar y_k)} \\leq \\epsilon$ for the first time, the total iteration count $K$ is at most\n  \\begin{align}\n    \\prn[\\bigg]{\n      70\n      + \n      108 c_1 \\frac{\\Mmax^{1/4}}{M_0^{1/4}}\n    }\n    \\frac{\\Lmax^{1/2} \\Mmax^2 \\Delta}{\\epsilon^{7/4} M_0^{7/4}}\n    + 2 (c_2 + 1) \\frac{\\Lmax^{1/2} \\Mmax^{1/4}}{\\epsilon^{1/4} M_0^{1/2}}.\n    \\label{eq:complexity_bound}\n  \\end{align}\n  If $M_k$ is computed with \\cref{eq:update_Mest} in the algorithm and $\\beta$ is set as $\\beta = 1$, the above bound is improved to\n  \\begin{align}\n    70\n    \\frac{\\Lmax^{1/2} \\Mmax^{1/4} \\Delta}{\\epsilon^{7/4}}\n    +\n    2 (c_2 + 1) \\frac{\\Lmax^{1/2}}{M_0^{1/4} \\epsilon^{1/4}}.\n    \\label{eq:complexity_bound_tight}\n  \\end{align}\n\\end{theorem}","start":75432,"end":76724}]},"proof":{"raw_tex":"\\begin{lemma}[Jensen-type inequality for gradients]\n  \\label{lem:gradient_jensen}\n  Under \\cref{asm:hessian_lip}, the following holds for any $z_1, \\dots, z_n \\in \\R^d$ and $\\lambda_1,\\dots,\\lambda_n \\geq 0$ such that $\\sum_{i=1}^n \\lambda_i = 1$:\n  \\begin{align}\n    \\norm*{\n      \\nabla f \\prn*{ \\sum_{i=1}^n \\lambda_i z_i }\n      - \\sum_{i=1}^n \\lambda_i \\nabla f \\prn*{ z_i }\n    }\n    \\leq\n    \\frac{\\Mtrue}{2} \\sum_{1 \\leq i < j \\leq n} \\lambda_i \\lambda_j \\norm*{z_i - z_j}^2.\n  \\end{align}\n\\end{lemma}\n\n\\begin{lemma}[Error bound for trapezoidal rule]\n  \\label{lem:trapezoidal_rule_error}\n  Under \\cref{asm:hessian_lip}, the following holds for any $x, y \\in \\R^d$:\n  \\begin{align}\n    f(x) - f(y)\n    &\\leq\n    \\frac{1}{2} \\inner*{\\nabla f(x) + \\nabla f(y)}{x - y}\n    + \\frac{\\Mtrue}{12} \\norm*{x - y}^3.\n  \\end{align}\n\\end{lemma}\n\n\\begin{proposition}\n  \\label{prop:L_upperbound}\n  Suppose that \\cref{asm:gradient_lip} holds.\n  Then, the following is true throughout \\cref{alg:proposed_agd}: $\\Lest \\leq \\max\\set{\\Linit, \\alpha \\Ltrue}$.\n\\end{proposition}\n\n\\begin{lemma}\n  \\label{lem:decrease_epoch}\n  Suppose that \\cref{asm:gradient_lip} holds.\n  During epochs with $\\Lest \\geq \\Ltrue$, condition~\\cref{eq:descent_condition} always holds.\n\\end{lemma}\n\n\\begin{lemma}\n  \\label{lem:potential_decrease_iteration}\n  Under \\cref{asm:gradient_lip} and $\\Lest \\geq \\Ltrue$, the following holds for all $k \\geq 0$:\n  \\begin{align}\n    \\Phi_{k+1} - \\Phi_k\n    &\\leq\n    \\frac{\\theta_{k+1}^2 + \\theta_k - 2}{4} \\Lest \\norm*{x_{k+1} - x_k}^2\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\\\\\n    &\\quad\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n    \\label{eq:potential_decrease_iteration}\n  \\end{align}\n\\end{lemma}\n\n\\begin{proof}[Proof of \\cref{lem:potential_decrease_iteration}]\n  In the proof, let\n  \\begin{align}\n    P_k\n    &\\coloneqq\n    \\inner*{\\nabla f(x_{k-1})}{x_k - x_{k-1}}\n  \\end{align}\n  to simplify the notation.\n  From \\cref{lem:eq:property_gradlip_obj,eq:update_xy}, we have\n  \\begin{align}\n    f(x_{k+1}) - f(y_k)\n    &\\leq\n    \\inner*{\\nabla f(y_k)}{x_{k+1} - y_k}\n    + \\frac{\\Lest}{2} \\norm*{x_{k+1} - y_k}^2\n    = \n    - \\frac{1}{2 \\Lest} \\norm*{\\nabla f(y_k)}^2.\n  \\end{align}\n  Summing this bound and \\cref{eq:fy_fx_diff_upperbound_Mestk} yields\n  \\begin{alignat}{2}\n    &\\mathInd\n    f(x_{k+1}) - f(x_k)\n    \\label{eq:descent_f_first}\\\\\n    &\\leq\n    \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n    + \\frac{\\Mest_k}{12} \\norm*{y_k - x_k}^3\n    - \\frac{1}{2 \\Lest} \\norm*{\\nabla f(y_k)}^2\\\\\n    &\\leq\n    \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n    + \\frac{\\theta_k^2 \\Mest_k}{12} \\norm*{x_k - x_{k-1}}^3\n    - \\frac{1}{2 \\Lest} \\norm*{\\nabla f(y_k)}^2,\n  \\end{alignat}\n  where the last inequality uses \\cref{eq:update_xy} and $0 \\leq \\theta_k \\leq 1$.\n  To evaluate the first term on the right-hand side, we decompose it into four terms:\n  \\begin{align}\n    &\\mathInd\n    \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\\\\\n    &=\n    \\underbrace{2 \\inner*{\\nabla f(y_k)}{y_k - x_k}}_{\\text{(A)}}\n    {}+{} \\underbrace{\\theta_k \\inner*{\\nabla f(x_{k-1})}{y_k - x_k}}_{\\text{(B)}}\\\\[-0.8\\baselineskip]\n    &\\quad\n    - \\underbrace{\\theta_k \\inner*{\\nabla f(x_k)}{y_k - x_k}}_{\\text{(C)}}\n    {}-{} \\underbrace{\n        \\inner*{\n        \\nabla f(y_k)\n        + \\theta_k \\nabla f(x_{k-1})\n        - (1 + \\theta_k) \\nabla f (x_k)\n      }{y_k - x_k}\n    }_{\\text{(D)}}.\n  \\end{align}\n  Each term can be evaluated as\n  \\begin{alignat}{2}\n    \\text{(A)}\n    &=\n    \\frac{1}{\\Lest} \\norm*{\\nabla f(y_k)}^2\n    + \\Lest \\norm*{y_k - x_k}^2\n    - \\Lest \\norm*{(y_k - x_k) - \\frac{1}{\\Lest} \\nabla f(y_k)}^2\\\\\n    &=\n    \\frac{1}{\\Lest} \\norm*{\\nabla f(y_k)}^2\n    + \\theta_k^2 \\Lest \\norm*{x_k - x_{k-1}}^2\n    - \\Lest \\norm*{x_{k+1} - x_k}^2\n    &\\quad&\\by{\\cref{eq:update_xy}},\\\\\n    \\text{(B)}\n    &=\n    \\theta_k^2 \\inner*{\\nabla f(x_{k-1})}{x_k - x_{k-1}}\n    =\n    \\theta_k^2 P_k\n    &\\quad&\\by{\\cref{eq:update_xy}},\\\\\n    \\text{(C)}\n    &=\n    - \\theta_k P_{k+1}\n    + \\theta_k \\inner*{\\nabla f(x_k)}{x_{k+1} - y_k}\\\\\n    &=\n    - \\theta_k P_{k+1}\n    - \\frac{\\theta_k}{\\Lest} \\inner*{\\nabla f(x_k)}{\\nabla f(y_k)}\n    &\\quad&\\by{\\cref{eq:update_xy}},\\\\\n    \\text{(D)}\n    &\\leq\n    \\theta_k \\Mest_k \\norm*{x_k - x_{k-1}}^2 \\norm*{y_k - x_k}\n    &\\quad&\\by{\\cref{eq:grad_interpolation_error_Mestk}}\\\\\n    &=\n    \\theta_k^2 \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    &\\quad&\\by{\\cref{eq:update_xy}}.\n  \\end{alignat}\n  Plugging the evaluations into \\cref{eq:descent_f_first} results in\n  \\begin{align}\n    % &\\mathInd\n    f(x_{k+1}) - f(x_k)\n    \\label{eq:descent_f_second}\n    &\\leq\n    \\frac{\\theta_k^2 \\Lest}{2} \\norm*{x_k - x_{k-1}}^2\n    - \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2\n    + \\frac{\\theta_k^2}{2} P_k\n    - \\frac{\\theta_k}{2} P_{k+1}\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    - \\frac{\\theta_k}{2 \\Lest} \\inner*{\\nabla f(x_k)}{\\nabla f(y_k)}.\n  \\end{align}\n\n  Next, to bound the last term on the right-hand side, we use the following inequality from \\cref{eq:grad_interpolation_error_Mestk}:\n  \\begin{align}\n    \\norm*{\n      \\nabla f(y_k)\n      - (1 + \\theta_k) \\nabla f (x_k)\n    }\n    &\\leq\n    \\theta_k \\norm*{ \\nabla f(x_{k-1})}\n    + \n    \\theta_k \\Mest_k\n    \\norm*{x_k - x_{k-1}}^2.\n  \\end{align}\n  We square both sides as\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{\n      \\nabla f(y_k)\n      - (1 + \\theta_k) \\nabla f (x_k)\n    }^2\\\\\n    &=\n    \\norm*{\\nabla f(y_k)}^2\n    + (1 + \\theta_k)^2 \\norm*{\\nabla f (x_k)}^2\n    - 2 (1 + \\theta_k) \\inner*{\\nabla f (x_k)}{\\nabla f(y_k)}\\\\\n    &\\geq\n    (1 + \\theta_k)^2 \\norm*{\\nabla f (x_k)}^2\n    - 2 (1 + \\theta_k) \\inner*{\\nabla f (x_k)}{\\nabla f(y_k)}\n  \\end{alignat}\n  and\n  \\begin{alignat}{2}\n    \\theta_k^2 \\prn*{\n      \\norm*{ \\nabla f(x_{k-1})}\n      + \\Mest_k \\norm*{x_k - x_{k-1}}^2\n    }^2\n    &\\leq\n    \\theta_k (1 + \\theta_k) \\norm*{ \\nabla f(x_{k-1})}^2\\\\\n    &\\quad\n    + \\theta_k^2 (1 + \\theta_k) \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4,\n  \\end{alignat}\n  where we have used $(a + b)^2 \\leq (1 + 1/\\theta) a^2 + (1 + \\theta) b^2$ for $a, b, \\theta > 0$.\n  Rearranging the terms yields\n  \\[\n  % \\begin{align}\n    - \\inner*{\\nabla f (x_k)}{\\nabla f(y_k)}\n    \\leq\n    \\frac{\\theta_k}{2} \\norm*{ \\nabla f(x_{k-1})}^2\n    - \\frac{1 + \\theta_k}{2} \\norm*{\\nabla f (x_k)}^2\n    + \\frac{\\theta_k^2}{2} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4.\n  % \\end{align}\n  \\]\n  Plugging this bound into \\cref{eq:descent_f_second} results in\n  \\begin{align}\n    \\label{eq:hint_potential}\n    &\\mathInd\n    f(x_{k+1}) - f(x_k)\\\\\n    &\\leq\n    \\frac{\\theta_k^2}{2} \\prn*{\n      P_k\n      + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_{k-1})}^2\n      + \\Lest \\norm*{x_k - x_{k-1}}^2\n    }\\\\\n    &\\quad\n    -\n    \\frac{\\theta_k}{2} \\prn*{\n      P_{k+1} + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_k)}^2\n    }\n    - \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n  \\end{align}\n  This inequality can be rewritten with the potential function $\\Phi_k$ defined in \\cref{eq:def_potential} as\n  \\begin{align}\n    \\Phi_{k+1} - \\Phi_k\n    &\\leq\n    \\frac{\\theta_{k+1}^2 - \\theta_k}{2}\n    \\prn*{\n      P_{k+1}\n      + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_k)}^2\n    }\n    + \\frac{\\theta_{k+1}^2 - 1}{2} \\Lest \\norm*{x_{k+1} - x_k}^2\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n  \\end{align}\n  Finally, using $\\theta_{k+1}^2 - \\theta_k \\leq 0$ from \\cref{eq:theta_increase_not_too_fast} and\n  \\begin{align}\n    - P_{k+1}\n    =\n    - \\inner*{\\nabla f(x_k)}{x_{k+1} - x_k}\n    \\leq\n    \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_k)}^2\n    + \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2\n  \\end{align}\n  from Young's inequality ($\\abs{\\inner*{a}{b}} \\leq \\frac{1}{2} \\norm*{a}^2 + \\frac{1}{2} \\norm*{b}^2$) yields the desired result as\n  \\begin{align}\n    \\Phi_{k+1} - \\Phi_k\n    &\\leq\n    \\frac{\\theta_{k+1}^2 - \\theta_k}{2} \\prn*{- \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2}\n    + \\frac{\\theta_{k+1}^2 - 1}{2} \\Lest \\norm*{x_{k+1} - x_k}^2\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2\\\\\n    &=\n    \\frac{\\theta_{k+1}^2 + \\theta_k - 2}{4} \\Lest \\norm*{x_{k+1} - x_k}^2\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\\\\\n    &\\quad\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n  \\end{align}\n\\end{proof}\n\n\\begin{proof}[Proof of \\cref{lem:decrease_epoch}]\n  Note that\n  \\begin{align}\n    \\sum_{i=1}^k \\norm*{x_i - x_{i-1}}^3\n    &\\leq\n    S_k^{3/2},\\quad\n    \\sum_{i=1}^k \\norm*{x_i - x_{i-1}}^4\n    \\leq\n    S_k^2\n    \\label{eq:sum_xk_diff_power34_upperbound}\n  \\end{align}\n  by the monotonicity of $\\ell_p$-norms, i.e., $\\norm{a}_p \\geq \\norm{a}_q$ for $1 \\leq p \\leq q$.\n  In addition, we have\n  \\begin{align}\n    \\Phi_k - f(x_k)\n    &=\n    \\frac{\\theta_k^2}{2} \\prn*{\n      \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_{k-1}) + L (x_k - x_{k-1})}^2\n      + \\frac{\\Lest}{2} \\norm*{x_k - x_{k-1}}^2\n    }\n    \\geq\n    0,\n    \\label{eq:potential_lowerbound}\\\\\n    \\Phi_0 - f(x_0)\n    &=\n    \\frac{\\theta_0^2}{4 \\Lest} \\norm*{\\nabla f(x_0)}^2\n    \\label{eq:potential_obj_0}\n  \\end{align}\n  since $x_{-1} = x_0$.\n\n  Summing \\cref{lem:potential_decrease_iteration} over $k$ and telescoping yields\n  \\begin{align}\n    &\\mathInd\n    \\Phi_k - \\Phi_0\n    \\label{eq:potential_decrease_epoch}\\\\\n    &\\leq\n    \\sum_{i=0}^{k-1}\n    \\bigg(\n      \\frac{\\theta_{i+1}^2 + \\theta_i - 2}{4} \\Lest \\norm*{x_{i+1} - x_i}^2\n      + \\frac{7 \\theta_i^2}{12} \\Mest_i \\norm*{x_i - x_{i-1}}^3\\\\\n      &\\qquad\\qquad\n      + \\frac{\\theta_i^3}{4 \\Lest} \\Mest_i^2 \\norm*{x_i - x_{i-1}}^4\n      - \\frac{\\theta_i^2}{4 \\Lest} \\norm*{\\nabla f(x_i)}^2\n    \\bigg)\\\\\n    &\\leq\n    \\frac{\\theta_k^2 + \\theta_{k-1} - 2}{4} \\Lest S_k\n    + \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} S_{k-1}^{3/2} \n    + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}^2\n    - \\frac{\\theta_0^2}{4 \\Lest} \\norm*{\\nabla f(x_0)}^2,\n  \\end{align}\n  where we have used \\cref{eq:theta_increase_not_too_fast}, \\cref{eq:sum_xk_diff_power34_upperbound}, and $\\Mest_i \\leq \\Mest_{k-1}$.\n  Combining \\cref{eq:potential_lowerbound,eq:potential_obj_0,eq:potential_decrease_epoch} yields\n  \\begin{align}\n    f(x_k) - f(x_0)\n    \\leq\n    \\frac{\\theta_k^2 + \\theta_{k-1} - 2}{4} \\Lest S_k\n    + \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} S_{k-1}^{3/2} \n    + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}^2.\n  \\end{align}\n  Furthermore, we bound the last two terms as\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} S_{k-1}^{3/2} \n    + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}^2\\\\\n    &\\leq\n    \\prn*{\n      \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} \\sqrt{S_{k-1}}\n      + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}\n    }\n    S_k\n    &\\quad&\\by{$S_{k-1} \\leq S_k$}\\\\\n    &\\leq\n    \\prn*{\n      \\frac{7}{12} \\theta_{k-1}^2 (1 - \\theta_{k-1})^{5/2}\n      + \\frac{1}{4} \\theta_{k-1}^3 (1 - \\theta_{k-1})^5\n    }\n    \\Lest S_k\n    &\\quad&\\by{\\cref{eq:condition_continue_k-1}}\\\\\n    &\\leq\n    \\prn*{\n      \\frac{7}{12} + \\frac{1}{4}\n    }\n    \\theta_{k-1}^2 (1 - \\theta_{k-1})^2\n    \\Lest S_k\n    &\\quad&\\by{$0 \\leq \\theta_{k-1} \\leq 1$}\\\\\n    &\\leq\n    \\frac{1}{4}\n    \\theta_{k-1} (1 - \\theta_{k-1})\n    \\Lest S_k\n    &\\quad&\\by{$0 \\leq \\theta_{k-1} \\leq 1$},\n  \\end{alignat}\n  which results in\n  \\begin{alignat}{2}\n    f(x_k) - f(x_0)\n    &\\leq\n    \\frac{\\theta_k^2 + \\theta_{k-1} - 2}{4} \\Lest S_k\n    + \\frac{1}{4} \\theta_{k-1} (1 - \\theta_{k-1}) \\Lest S_k\\\\\n    &=\n    - \\frac{1 - \\theta_k}{2} \\Lest S_k\n    + \\frac{(1 - \\theta_k)^2 - (1 - \\theta_{k-1})^2}{4} \\Lest S_k\\\\\n    &\\leq\n    - \\frac{1 - \\theta_k}{2} \\Lest S_k\n    &\\quad&\\by{\\cref{eq:theta_increase_not_too_fast}}\n  \\end{alignat}\n  and concludes the proof.\n\\end{proof}\n\n\\begin{proposition}\n  \\label{prop:Mk_upperbound}\n  Suppose that \\cref{asm:hessian_lip} holds.\n  Then, the following is true throughout \\cref{alg:proposed_agd}: \n  $\\Mest_k \\leq \\max\\set{M_0, \\Mtrue}$.\n\\end{proposition}\n\n\\begin{lemma}\n  \\label{lem:grad_interpolation_error}\n  Under \\cref{asm:hessian_lip}, the following hold for all $k \\geq 1$:\n  \\begin{align}\n    &\n    \\norm*{\n      \\nabla f(y_k)\n      + \\theta_k \\nabla f(x_{k-1})\n      - (1 + \\theta_k) \\nabla f (x_k)\n    }\n    \\leq\n    \\theta_k \\Mtrue \\norm*{x_k - x_{k-1}}^2,\n    \\label{eq:grad_interpolation_error_Mf}\\\\\n    &\n    \\norm*{ \\nabla f (\\bar y_k) }\n    \\leq\n    \\frac{\\Lest}{Z_k} \\norm*{x_k - x_{k-1}}\n    + \\frac{(k-1) (k+5)^2}{16 Z_k^2} \\Mtrue S_k.\n    \\label{eq:grad_ybark_upperbound_Mf}\n  \\end{align}\n\\end{lemma}\n\n\\begin{proof}[Proof of \\cref{eq:grad_interpolation_error_Mf}]\n  Note that $x_k = \\frac{1}{1 + \\theta_k} y_k + \\frac{\\theta_k}{1 + \\theta_k} x_{k-1}$ from \\cref{eq:update_xy}, and \\cref{lem:gradient_jensen} gives\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{\n      \\nabla f (x_k)\n      - \\frac{1}{1 + \\theta_k} \\nabla f(y_k)\n      - \\frac{\\theta_k}{1 + \\theta_k} \\nabla f(x_{k-1})\n    }\\\\\n    &\\leq\n    \\frac{\\Mtrue \\theta_k}{2 (1 + \\theta_k)^2} \\norm*{y_k - x_{k-1}}^2\\\\\n    &=\n    \\frac{\\Mtrue \\theta_k}{2} \\norm*{x_k - x_{k-1}}^2\n    \\leq\n    \\frac{\\Mtrue \\theta_k}{1 + \\theta_k} \\norm*{x_k - x_{k-1}}^2\n    &\\quad&\\by{\\cref{eq:update_xy} and $0 \\leq \\theta_k \\leq 1$}.\n  \\end{alignat}\n  Multiplying by $(1 + \\theta_k)$ concludes the proof.\n\\end{proof}\n\n\\begin{proof}[Proof of \\cref{eq:grad_ybark_upperbound_Mf}]\n  We have\n  \\begin{alignat}{2}\n    &\\mathInd\n    p_{k,i} \\nabla f(y_i)\n    =\n    p_{k,i} \\Lest (y_i - x_{i+1})\\\\\n    &=\n    p_{k,i} \\Lest \\prn*{\n      \\theta_i (x_i - x_{i-1})\n      - (x_{i+1} - x_i)\n    }\n    &\\quad&\\by{\\cref{eq:update_xy}}\\\\\n    &=\n    \\Lest \\prn*{\n      p_{k,i-1} (x_i - x_{i-1})\n      - p_{k,i} (x_{i+1} - x_i)\n    }\n    &\\quad&\\by{definition \\cref{eq:def_ybark} of $p_{k,i}$}.\n  \\end{alignat}\n  Summing this equality over $i$ yields\n  \\begin{align}\n    \\sum_{i=0}^{k-1}\n    p_{k,i} \\nabla f(y_i)\n    =\n    - \\Lest p_{k,k-1} (x_k - x_{k-1}),\n    \\label{eq:weighted_average_gradient_equation}\n  \\end{align}\n  where we have used $x_0 = x_{-1}$.\n  Now, we obtain\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{\n      \\nabla f (\\bar y_k)\n    }\n    \\label{eq:norm_grad_ybark_upperbound_pre}\\\\\n    &\\leq\n    \\norm*{\n      \\sum_{i=0}^{k-1} p_{k,i} \\nabla f \\prn*{ y_i }\n    }\n    + \\frac{\\Mtrue}{2} \\sum_{0 \\leq i < j < k} p_{k,i} p_{k,j} \\norm*{y_i - y_j}^2\n    &\\quad&\\by{\\cref{lem:gradient_jensen}}\\\\\n    &=\n    \\Lest p_{k,k-1} \\norm*{x_k - x_{k-1}}\n    + \\frac{\\Mtrue}{2} \\sum_{0 \\leq i < j < k} p_{k,i} p_{k,j} \\norm*{y_i - y_j}^2\n    &\\quad&\\by{\\cref{eq:weighted_average_gradient_equation}}\\\\\n    &\\leq\n    \\frac{\\Lest}{Z_k} \\norm*{x_k - x_{k-1}}\n    + \\frac{\\Mtrue}{2 Z_k^2} \\sum_{0 \\leq i < j < k} \\norm*{y_i - y_j}^2,\n  \\end{alignat}\n  where the last inequality follows from $p_{k,i} \\leq p_{k,k-1} = 1 / Z_k$ for all $0 \\leq i < k$.\n\n  Next, we bound the second term on the right-hand side.\n  Since for $0 \\leq i < j < k$,\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{y_i - y_j}\\\\\n    &\\leq\n    \\norm*{y_i - x_i}\n    + \\sum_{l=i+1}^{j-1} \\norm*{x_l - x_{l-1}}\n    + \\norm*{y_j - x_{j-1}}\n    &\\quad&\\by{the triangle inequality}\\\\\n    &=\n    \\norm*{x_i - x_{i-1}}\n    + \\sum_{l=i+1}^{j-1} \\norm*{x_l - x_{l-1}}\n    + 2 \\norm*{x_j - x_{j-1}}\n    &\\quad&\\by{\\cref{eq:update_xy} and $0 \\leq \\theta_k \\leq 1$}\\\\\n    &\\leq\n    \\prn*{\n      1^2 + \\sum_{l=i+1}^{j-1} 1^2 + 2^2\n    }^{1/2}\n    \\prn*{\n      \\sum_{l=i}^j \\norm*{x_l - x_{l-1}}^2\n    }^{1/2}\n    &\\quad&\\by{Cauchy--Schwarz}\\\\\n    &=\n    \\sqrt{j - i + 4}\n    \\prn*{\n      \\sum_{l=i}^j \\norm*{x_l - x_{l-1}}^2\n    }^{1/2},\n  \\end{alignat}\n  we have\n  \\begin{align}\n    \\sum_{0 \\leq i < j < k}\n    \\norm*{y_i - y_j}^2\n    &\\leq\n    \\sum_{0 \\leq i < j < k}\n    \\sum_{l=i}^j\n    (j - i + 4) \\norm*{x_l - x_{l-1}}^2\\\\\n    % &=\n    % \\sum_{0 \\leq i \\leq l \\leq j < k}\n    % (j - i + 4) \\norm*{x_l - x_{l-1}}^2\n    % - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &=\n    \\sum_{l=0}^{k-1}\n    \\prn*{\n      \\sum_{i=0}^l\n      \\sum_{j=l}^{k-1}\n      (j - i + 4) \n    }\n    \\norm*{x_l - x_{l-1}}^2\n    - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &=\n    \\frac{k + 7}{2}\n    \\sum_{l=0}^{k-1}\n    (l + 1) (k - l)\n    \\norm*{x_l - x_{l-1}}^2\n    - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &\\leq\n    \\frac{k + 7}{2}\n    \\sum_{l=0}^{k-1} \\frac{(k+1)^2}{4} \\norm*{x_l - x_{l-1}}^2\n    - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &=\n    \\frac{(k-1) (k+5)^2}{8}\n    \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\n    \\leq\n    \\frac{(k-1) (k+5)^2}{8} S_k.\n  \\end{align}\n  Plugging this bound into \\cref{eq:norm_grad_ybark_upperbound_pre} concludes the proof.\n\\end{proof}\n\n\\begin{lemma}\n  \\label{lem:grad_norm_ybar_upperbound}\n  Suppose that \\cref{asm:gradient_hessian_lip} holds.\n  In \\cref{alg:proposed_agd}, the following is true when $k \\geq 2$:\n  \\begin{align}\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    \\leq\n    \\frac{4 \\Lmax^2 \\Mmax}{M_0^2 k^4}.\n    \\label{eq:grad_norm_ybar_upperbound}\n  \\end{align}\n  If $M_k$ is computed with \\cref{eq:update_Mest} in the algorithm, the above bound is improved to\n  \\begin{align}\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\leq\n    4 \\Lest \\sqrt{\\frac{S_{k-1}}{k^3}}\n    \\leq\n    \\frac{4 \\Lmax^2}{M_0 k^4}.\n    \\label{eq:grad_norm_ybar_upperbound_tight}\n  \\end{align}\n\\end{lemma}\n\n\\begin{proof}[Proof of \\cref{eq:grad_norm_ybar_upperbound}] \n  Note that\n  \\begin{align}\n    &\\sum_{i=1}^{k-1}\n    \\frac{(i-1)(i+5)^2}{16}\n    =\n    \\frac{(k - 2) (k - 1) (k^2 + 13 k + 50)}{64}\n    \\leq\n    \\frac{k^3 (k+1)}{24},\n    \\label{eq:i_cubic_sum}\\\\\n    &\\sum_{i=1}^{k-1} Z_i^2\n    =\n    \\sum_{i=1}^{k-1} \\prn*{\\frac{i+1}{2}}^2\n    =\n    \\frac{(k-1) (2 k^2 + 5 k + 6)}{24}\n    \\geq \n    \\frac{k^2 (k+1)}{12}\n    \\label{eq:Zi_squared_sum}\n  \\end{align}\n  from \\cref{eq:ybar_Zk_alternative} and $k \\geq 2$.\n  Using \\cref{eq:grad_ybark_upperbound_Mf}, we have\n  \\begin{align}\n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\leq\n    \\sum_{i=1}^{k-1}\n    Z_i^2\n    \\norm*{ \\nabla f (\\bar y_i) }\\\\\n    &\\leq\n    \\sum_{i=1}^{k-1}\n    \\prn*{\n      \\Lest Z_i \\norm*{x_i - x_{i-1}}\n      + \\frac{(i-1) (i+5)^2}{16} \\Mtrue S_i\n    }.\n  \\end{align}\n  Each term on the right-hand side can be bounded as follows:\n  \\begin{alignat}{2}\n    &\\sum_{i=1}^{k-1} \\Lest Z_i \\norm*{x_i - x_{i-1}}\n    \\leq\n    \\Lest\n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }^{1/2}\n    \\sqrt{S_{k-1}}\n    &\\quad&\\by{Cauchy--Schwarz},\\\\\n    &\\sum_{i=1}^{k-1}\n    \\frac{(i-1)(i+5)^2}{16} \\Mtrue S_i\n    \\leq\n    \\frac{k^3 (k+1)}{24}\n    \\Mtrue S_{k-1}\n    &\\quad&\\by{\\cref{eq:i_cubic_sum}}.\n  \\end{alignat}\n  We thus have\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\\\\\n    &\\leq\n    \\Lest \n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }^{-1/2}\\!\\!\n    \\sqrt{S_{k-1}}\n    + \n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }^{-1}\n    \\frac{k^3 (k+1)}{24}\n    \\Mtrue S_{k-1}\\\\\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{k}{2}\n    \\Mtrue S_{k-1}\n    &\\quad&\\by{\\cref{eq:Zi_squared_sum}}\\\\\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{\\Lest \\Mtrue}{2 \\Mest_k} \n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{\\cref{eq:condition_continue_k-1}}\\\\\n    &\\leq\n    \\frac{\\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{\\Lest \\Mmax}{2 \\Mest_{k-1}} \n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{$\\Mest_{k-1}, \\Mtrue \\leq \\Mmax$}\\\\\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}},\n  \\end{alignat}\n  which is the first inequality of the desired result~\\cref{eq:grad_norm_ybar_upperbound}.\n  The second inequality follows from \\cref{eq:condition_continue_k-1}:\n  \\begin{align}\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    \\leq\n    \\frac{4 \\Lest^2 \\Mmax}{M_{k-1}^2 k^4}\n    \\leq\n    \\frac{4 \\Lmax^2 \\Mmax}{M_0 k^4}.\n  \\end{align}\n\\end{proof}\n\n\\begin{proof}[Proof of \\cref{eq:grad_norm_ybar_upperbound_tight}]\n  When $M_k$ is computed with \\cref{eq:update_Mest}, we have \\cref{eq:grad_ybark_upperbound_Mestk}.\n  Similarly to the proof of \\cref{eq:grad_norm_ybar_upperbound}, using \\cref{eq:grad_ybark_upperbound_Mestk} instead of \\cref{eq:grad_ybark_upperbound_Mf} yields\n  \\begin{alignat}{2}\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{k}{2} \\Mest_{k-1} S_{k-1}.\n  \\end{alignat}\n  The right-hand side is bounded as\n  \\begin{alignat}{2}\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{k}{2} \\Mest_{k-1} S_{k-1}\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{\\Lest}{2} \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{\\cref{eq:condition_continue_k-1}}\\\\\n    &\\leq\n    4 \\Lest\n    \\sqrt{\\frac{S_{k-1}}{k^3}},\n  \\end{alignat}\n  which implies the first inequality in \\cref{eq:grad_norm_ybar_upperbound_tight}.\n  The second inequality is obtained similarly to the second inequality of \\cref{eq:grad_norm_ybar_upperbound}.\n\\end{proof}\n\n\\begin{proof}[Proof of bound \\cref{eq:complexity_bound}]\n  We count the number of iterations of \\cref{alg:proposed_agd} separately for three types of epoch:\n  \\begin{itemize}\n    \\item \n    successful epoch: an epoch that does not find an $\\epsilon$-stationary point and ends at Line \\ref{alg-line-agd:restart_successful} with the descent condition~\\cref{eq:descent_condition} satisfied,\n    \\item\n    unsuccessful epoch: an epoch that does not find an $\\epsilon$-stationary point and ends at Line \\ref{alg-line-agd:restart_unsuccessful} with the descent condition~\\cref{eq:descent_condition} unsatisfied,\n    \\item\n    last epoch: the epoch that finds an $\\epsilon$-stationary point.\n  \\end{itemize}\n\n  \\paragraph{Successful epochs}\n  Let us focus on a successful epoch.\n  Let $k$ be the iteration number of the epoch.\n  It follows that \n  \\begin{align}\n    (k+1)^5\n    \\Mest_k^2 S_k\n    &>\n    \\Lest^2.\n    \\label{eq:restart_condition_k}\n  \\end{align}\n  When $k \\geq 2$, we have\n  \\begin{alignat}{2}\n    \\epsilon\n    &<\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\quad&\\text{(since this epoch does not find an $\\epsilon$-stationary point)}\\\\\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{\\cref{eq:grad_norm_ybar_upperbound}}\\\\\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_0} \\sqrt{\\frac{S_k}{k^3}}\n    &\\quad&\\by{$M_{k-1} \\geq M_0$ and $S_k \\geq S_{k-1}$},\n  \\end{alignat}\n  and hence \n  \\begin{align}\n    S_k > \\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3.\n    \\label{eq:Sk_lowerbound}\n  \\end{align}\n  When $k = 1$, we also have \\cref{eq:Sk_lowerbound}, as follows:\n  \\begin{align}\n    S_k\n    =\n    \\norm*{x_1 - x_0}^2\n    =\n    \\frac{1}{\\Lest^2} \\norm*{\\nabla f(x_0)}^2\n    =\n    \\frac{1}{\\Lest^2} \\norm*{\\nabla f(\\bar y_1)}^2\n    >\n    \\frac{\\epsilon^2}{\\Lest^2}\n    \\geq\n    \\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3.\n  \\end{align}\n  Combining \\cref{eq:Sk_lowerbound,eq:restart_condition_k} yields\n  \\begin{align}\n    S_k\n    =\n    S_k^{7/8}\n    S_k^{1/8}\n    &\\geq\n    \\prn[\\bigg]{\\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3}^{7/8}\n    \\prn*{\\frac{\\Lest^2}{\\Mest_k^2 (k+1)^5}}^{1/8}\\\\\n    &=\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{3/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^{7/4} \\Mest_k^{1/4}}\n    \\frac{k^{21/8}}{(k+1)^{5/8}}\n    \\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{3/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^2}\n    \\frac{k^{21/8}}{(k+1)^{5/8}}.\n  \\end{align}\n  Plugging this bound into condition \\cref{eq:descent_condition}, we obtain\n  \\begin{align}\n    f(x_0) - f(x_k)\n    \\geq\n    \\frac{\\Lest S_k}{2 (k+1)}\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{1/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^2}\n    \\prn*{\\frac{k}{k+1}}^{13/8}\n    \\frac{k}{2}\\\\\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lmax^{1/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^2}\n    \\prn*{\\frac{1}{2}}^{13/8}\n    \\frac{k}{2}\n    \\geq\n    \\frac{\\epsilon^{7/4} M_0^{7/4}}{\\Lmax^{1/2} \\Mmax^2}\n    \\frac{k}{70},\n  \\end{align}\n  and thus\n  \\begin{align}\n    k\n    \\leq\n    70\n    \\frac{\\Lmax^{1/2} \\Mmax^2}{\\epsilon^{7/4} M_0^{7/4}}\n    \\prn*{f(x_0) - f(x_k)}.\n  \\end{align}\n  Summing this bound over all successful epochs leads to the conclusion that the total iteration number of successful epochs is at most\n  \\begin{align}\n    70 \\frac{\\Lmax^{1/2} \\Mmax^2 \\Delta}{\\epsilon^{7/4} M_0^{7/4}}.\n    \\label{eq:Ksuc_upperbound}\n  \\end{align}\n  For later use, we will also evaluate the number of successful epochs.\n  Combining \\cref{eq:Sk_lowerbound,eq:restart_condition_k} also yields\n  \\begin{align}\n    S_k\n    =\n    S_k^{3/4}\n    S_k^{1/4}\n    &\\geq\n    \\prn[\\bigg]{ \\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3 }^{3/4}\n    \\prn*{\\frac{\\Lest^2}{\\Mest_k^2 (k+1)^5}}^{1/4}\\\\\n    &=\n    \\frac{(\\epsilon / 4)^{3/2}}{\\Lest}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^{3/2} \\Mest_k^{1/2}}\n    \\frac{k^{9/4}}{(k+1)^{5/4}}\n    \\geq\n    \\frac{(\\epsilon / 4)^{3/2}}{\\Lest}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^2}\n    \\frac{k^{9/4}}{(k+1)^{5/4}}.\n  \\end{align}\n  Plugging this bound into condition \\cref{eq:descent_condition} gives\n  \\begin{align}\n    f(x_0) - f(x_k)\n    \\geq\n    \\frac{\\Lest S_k}{2 (k+1)}\n    &\\geq\n    \\frac{(\\epsilon / 4)^{3/2}}{2}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^2}\n    \\prn*{\\frac{k}{k+1}}^{9/4}\\\\\n    &\\geq\n    \\frac{(\\epsilon / 4)^{3/2}}{2}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^2}\n    \\prn*{\\frac{1}{2}}^{9/4}\n    =\n    \\frac{\\epsilon^{3/2} \\Mest_0^{3/2}}{2^{25/4} \\Mmax^2}.\n  \\end{align}\n  Thus, we can deduce that the number $N_{\\mathrm{suc}}$ of successful epochs satisfies\n  \\begin{align}\n    N_{\\mathrm{suc}}\n    \\leq\n    2^{25/4}\n    \\frac{\\Mmax^2 \\Delta}{\\epsilon^{3/2} \\Mest_0^{3/2}}.\n    \\label{eq:Nsuc_upperbound}\n  \\end{align}\n\n  \\paragraph{Other epochs}\n  We deduce from \\cref{eq:grad_norm_ybar_upperbound} that the number of iterations for each epoch is at most\n  \\begin{align}\n    \\prn*{\n      \\frac{4 \\Lmax^2 \\Mmax}{\\epsilon M_0^2}\n    }^{1/4}.\n  \\end{align}\n  Let $N_{\\mathrm{unsuc}}$ be the number of unsuccessful epochs.\n  We then have\n  \\begin{align}\n    \\Linit\n    \\beta^{N_{\\mathrm{suc}}}\n    \\alpha^{N_{\\mathrm{unsuc}}}\n    \\leq\n    \\Lmax,\n  \\end{align}\n  and hence\n  \\begin{align}\n    N_{\\mathrm{unsuc}}\n    \\leq\n    c_1 N_{\\mathrm{suc}} + c_2,\n    \\label{eq:Nunsuc_upperbound}\n  \\end{align}\n  where $c_1$ and $c_2$ are defined by \\cref{eq:def_c1_c2}.\n  The total iteration number of unsuccessful and last epochs is at most\n  \\begin{alignat}{2}\n    &\\mathInd\n    (N_{\\mathrm{unsuc}} + 1)\n    \\prn*{ \\frac{4 \\Lmax^2 \\Mmax}{\\epsilon M_0^2} }^{1/4}\\\\\n    &\\leq\n    \\prn*{ c_1 N_{\\mathrm{suc}} + c_2 + 1 }\n    \\prn*{ \\frac{4 \\Lmax^2 \\Mmax}{\\epsilon M_0^2} }^{1/4}\n    &\\quad&\\by{\\cref{eq:Nunsuc_upperbound}}\\\\\n    &\\leq\n    2^{27/4} c_1\n    \\frac{\\Lmax^{1/2} \\Mmax^{9/4} \\Delta}{\\epsilon^{7/4} \\Mest_0^2}\n    + 2^{1/2} (c_2 + 1) \\frac{\\Lmax^{1/2} \\Mmax^{1/4}}{\\epsilon^{1/4} M_0^{1/2}}\n    &\\quad&\\by{\\cref{eq:Nsuc_upperbound}}\\\\\n    &\\leq\n    108 c_1\n    \\frac{\\Lmax^{1/2} \\Mmax^{9/4} \\Delta}{\\epsilon^{7/4} \\Mest_0^2}\n    + 2 (c_2 + 1) \\frac{\\Lmax^{1/2} \\Mmax^{1/4}}{\\epsilon^{1/4} M_0^{1/2}}.\n  \\end{alignat}\n  Putting this bound together with \\cref{eq:Ksuc_upperbound} concludes the proof.\n\\end{proof}\n\n\\begin{proof}[Proof of bound \\cref{eq:complexity_bound_tight}]\n  The proof is similar to \\cref{eq:complexity_bound} through the evaluation of the total iteration number of successful, unsuccessful, and last epochs.\n\n  \\paragraph{Successful epochs}\n  Let us focus on a successful epoch.\n  Let $k$ be the iteration number of the epoch.\n  As a counterpart of \\cref{eq:Sk_lowerbound}, we can obtain \n  \\begin{align}\n    S_k > \\prn*{\\frac{\\epsilon}{4 \\Lest}}^2 k^3\n    \\label{eq:Sk_lowerbound_tight}\n  \\end{align}\n  from \\cref{eq:grad_norm_ybar_upperbound_tight}.\n  Combining \\cref{eq:Sk_lowerbound_tight,eq:restart_condition_k} yields\n  \\begin{align}\n    S_k\n    &=\n    S_k^{7/8}\n    S_k^{1/8}\n    \\geq\n    \\prn*{\\prn*{\\frac{\\epsilon}{4 \\Lest}}^2 k^3}^{7/8}\n    \\prn*{\\frac{\\Lest^2}{\\Mest_k^2 (k+1)^5}}^{1/8}\n    =\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{3/2} \\Mest_k^{1/4}}\n    \\frac{k^{21/8}}{(k+1)^{5/8}}.\n  \\end{align}\n  Plugging this bound into condition \\cref{eq:descent_condition}, we obtain\n  \\begin{align}\n    f(x_0) - f(x_k)\n    \\geq\n    \\frac{\\Lest S_k}{2 (k+1)}\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{1/2} \\Mest_k^{1/4}}\n    \\prn*{\\frac{k}{k+1}}^{13/8}\n    \\frac{k}{2}\\\\\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lmax^{1/2} \\Mmax^{1/4}}\n    \\prn*{\\frac{1}{2}}^{13/8}\n    \\frac{k}{2}\n    \\geq\n    \\frac{\\epsilon^{7/4}}{\\Lmax^{1/2} \\Mmax^{1/4}}\n    \\frac{k}{70}.\n  \\end{align}\n  Hence, as in the proof of \\cref{eq:complexity_bound}, the total iteration number of all successful epochs is at most\n  \\begin{align}\n    70\n    \\frac{\\Lmax^{1/2} \\Mmax^{1/4} \\Delta}{\\epsilon^{7/4}}.\n    \\label{eq:Ksuc_upperbound_tight}\n  \\end{align}\n\n  \\paragraph{Other epochs}\n  We deduce from \\cref{eq:grad_norm_ybar_upperbound_tight} that the iteration number of each epoch is at most $\\prn[\\big]{ \\frac{4 \\Lmax^2}{M_0 \\epsilon} }^{1/4}$.\n  In addition, unsuccessful epochs occur only at most $c_2$ times when $\\beta = 1$, where $c_2$ is defined by \\cref{eq:def_c1_c2}.\n  Thus, the total iteration number of unsuccessful and last epochs is at most\n  \\begin{align}\n    \\prn*{\n      c_2 + 1\n    }\n    \\prn*{\n      \\frac{4 \\Lmax^2}{M_0 \\epsilon}\n    }^{1/4}\n    \\leq\n    2 (c_2 + 1) \\frac{\\Lmax^{1/2}}{M_0^{1/4} \\epsilon^{1/4}}.\n  \\end{align}\n  Putting this bound together with \\cref{eq:Ksuc_upperbound_tight} concludes the proof.\n\\end{proof}\n\n\\begin{proof}[Proof of \\cref{lem:gradient_jensen}]\n  To simplify the notation, let $\\bar z \\coloneqq \\sum_{i=1}^n \\lambda_i z_i$.\n  Taylor's theorem gives\n  \\[\n    \\nabla f(z_i) - \\nabla f(\\bar z)\n    =\n    \\nabla^2 f(\\bar z) (z_i - \\bar z)\n    + \\int_0^1 \\prn*{ \\nabla^2 f(\\bar z + t (z_i - \\bar z)) - \\nabla^2 f(\\bar z) } (z_i - \\bar z) dt,\n  \\]\n  and its weighted average gives\n  \\begin{align}\n    \\sum_{i=1}^n \\lambda_i \\nabla f(z_i) - \\nabla f(\\bar z)\n    &=\n    \\sum_{i=1}^n \\lambda_i\n    \\int_0^1 \\prn*{ \\nabla^2 f(\\bar z + t (z_i - \\bar z)) - \\nabla^2 f(\\bar z) } (z_i - \\bar z) dt.\n  \\end{align}\n  Therefore, we obtain\n  \\begin{alignat}{2}\n    \\norm*{\n      \\sum_{i=1}^n \\lambda_i \\nabla f(z_i) - \\nabla f(\\bar z)\n    }\n    &\\leq\n    \\sum_{i=1}^n \\lambda_i\n    \\int_0^1 \\norm*{ \\nabla^2 f(\\bar z + t (z_i - \\bar z)) - \\nabla^2 f(\\bar z) } \\norm*{z_i - \\bar z} dt\\\\\n    &\\leq\n    \\sum_{i=1}^n \\lambda_i\n    \\int_0^1 M_f t \\norm*{z_i - \\bar z}^2 dt\n    \\quad\\by{\\cref{asm:hessian_lip}}\\\\\n    &=\n    \\frac{M_f}{2} \\sum_{i=1}^n \\lambda_i \\norm*{z_i - \\bar z}^2.\n  \\end{alignat}\n  Furthermore, elementary algebra shows that $\\sum_{i=1}^n \\lambda_i \\norm*{z_i - \\bar z}^2 = \\sum_{1 \\leq i < j \\leq n} \\lambda_i \\lambda_j \\|z_i - z_j\\|^2$ as\n  \\begin{align}\n    \\sum_{1 \\leq i < j \\leq n} \\lambda_i \\lambda_j \\norm*{z_i - z_j}^2\n    &= \n    \\frac{1}{2} \\sum_{i,j = 1}^n \\lambda_i \\lambda_j \\norm*{z_i - z_j}^2\\\\\n    &= \n    \\frac{1}{2} \\sum_{i,j = 1}^n \\lambda_i \\lambda_j \\norm*{z_i}^2\n    + \\frac{1}{2} \\sum_{i,j = 1}^n \\lambda_i \\lambda_j \\norm*{z_j}^2\n    - \\sum_{i,j = 1}^n \\inner{\\lambda_i z_i}{\\lambda_j z_j}\\\\\n    &= \n    \\sum_{i=1}^n \\lambda_i \\norm*{z_i}^2 - \\norm*{\\bar z}^2\n    =\n    \\sum_{i=1}^n \\lambda_i \\norm*{z_i - \\bar z}^2,\n  \\end{align}\n  which completes the proof.\n\\end{proof}\n\n\\begin{proof}[Proof of \\cref{lem:trapezoidal_rule_error}]\n  We obtain the desired result as follows:\n  \\begin{alignat}{2}\n    &\\mathInd\n    % \\abs*{\n      f(x) - f(y)\n      - \\frac{1}{2} \\inner*{\\nabla f(x) + \\nabla f(y)}{x - y}\\\\\n    % }\\\\\n    &=\n    % \\abs*{\n      \\int_0^1 \\inner*{\\nabla f(t x + (1 - t) y)}{x - y} dt\n      - \\frac{1}{2} \\inner*{\\nabla f(x) + \\nabla f(y)}{x - y}\n    % }\n    &\\quad&\\by{Taylor's\\\\theorem}\\\\\n    &=\n    % \\abs*{\n      \\int_0^1 \\inner[\\Big]{\\nabla f(t x + (1 - t) y) - t \\nabla f(x) - (1 - t) \\nabla f(y)}{x - y} dt\\\\\n    % }\\\\\n    &\\leq\n    \\int_0^1 \\norm[\\Big]{\\nabla f(t x + (1 - t) y) - t \\nabla f(x) - (1 - t) \\nabla f(y)} \\norm{x - y} dt\\\\\n    &\\leq\n    \\frac{\\Mtrue}{2} \\int_0^1 t (1 - t) \\norm{x - y}^3 dt\n    =\n    \\frac{\\Mtrue}{12} \\norm{x - y}^3\n    &\\quad&\\by{\\cref{lem:gradient_jensen}}.\n  \\end{alignat}\n\\end{proof}","units":[{"raw_tex":"\\begin{lemma}[Jensen-type inequality for gradients]\n  \\label{lem:gradient_jensen}\n  Under \\cref{asm:hessian_lip}, the following holds for any $z_1, \\dots, z_n \\in \\R^d$ and $\\lambda_1,\\dots,\\lambda_n \\geq 0$ such that $\\sum_{i=1}^n \\lambda_i = 1$:\n  \\begin{align}\n    \\norm*{\n      \\nabla f \\prn*{ \\sum_{i=1}^n \\lambda_i z_i }\n      - \\sum_{i=1}^n \\lambda_i \\nabla f \\prn*{ z_i }\n    }\n    \\leq\n    \\frac{\\Mtrue}{2} \\sum_{1 \\leq i < j \\leq n} \\lambda_i \\lambda_j \\norm*{z_i - z_j}^2.\n  \\end{align}\n\\end{lemma}","start":35516,"end":36025},{"raw_tex":"\\begin{lemma}[Error bound for trapezoidal rule]\n  \\label{lem:trapezoidal_rule_error}\n  Under \\cref{asm:hessian_lip}, the following holds for any $x, y \\in \\R^d$:\n  \\begin{align}\n    f(x) - f(y)\n    &\\leq\n    \\frac{1}{2} \\inner*{\\nabla f(x) + \\nabla f(y)}{x - y}\n    + \\frac{\\Mtrue}{12} \\norm*{x - y}^3.\n  \\end{align}\n\\end{lemma}","start":36026,"end":36354},{"raw_tex":"\\begin{proposition}\n  \\label{prop:L_upperbound}\n  Suppose that \\cref{asm:gradient_lip} holds.\n  Then, the following is true throughout \\cref{alg:proposed_agd}: $\\Lest \\leq \\max\\set{\\Linit, \\alpha \\Ltrue}$.\n\\end{proposition}","start":51077,"end":51300},{"raw_tex":"\\begin{lemma}\n  \\label{lem:decrease_epoch}\n  Suppose that \\cref{asm:gradient_lip} holds.\n  During epochs with $\\Lest \\geq \\Ltrue$, condition~\\cref{eq:descent_condition} always holds.\n\\end{lemma}","start":51364,"end":51558},{"raw_tex":"\\begin{lemma}\n  \\label{lem:potential_decrease_iteration}\n  Under \\cref{asm:gradient_lip} and $\\Lest \\geq \\Ltrue$, the following holds for all $k \\geq 0$:\n  \\begin{align}\n    \\Phi_{k+1} - \\Phi_k\n    &\\leq\n    \\frac{\\theta_{k+1}^2 + \\theta_k - 2}{4} \\Lest \\norm*{x_{k+1} - x_k}^2\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\\\\\n    &\\quad\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n    \\label{eq:potential_decrease_iteration}\n  \\end{align}\n\\end{lemma}","start":52451,"end":52999},{"raw_tex":"\\begin{proof}[Proof of \\cref{lem:potential_decrease_iteration}]\n  In the proof, let\n  \\begin{align}\n    P_k\n    &\\coloneqq\n    \\inner*{\\nabla f(x_{k-1})}{x_k - x_{k-1}}\n  \\end{align}\n  to simplify the notation.\n  From \\cref{lem:eq:property_gradlip_obj,eq:update_xy}, we have\n  \\begin{align}\n    f(x_{k+1}) - f(y_k)\n    &\\leq\n    \\inner*{\\nabla f(y_k)}{x_{k+1} - y_k}\n    + \\frac{\\Lest}{2} \\norm*{x_{k+1} - y_k}^2\n    = \n    - \\frac{1}{2 \\Lest} \\norm*{\\nabla f(y_k)}^2.\n  \\end{align}\n  Summing this bound and \\cref{eq:fy_fx_diff_upperbound_Mestk} yields\n  \\begin{alignat}{2}\n    &\\mathInd\n    f(x_{k+1}) - f(x_k)\n    \\label{eq:descent_f_first}\\\\\n    &\\leq\n    \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n    + \\frac{\\Mest_k}{12} \\norm*{y_k - x_k}^3\n    - \\frac{1}{2 \\Lest} \\norm*{\\nabla f(y_k)}^2\\\\\n    &\\leq\n    \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n    + \\frac{\\theta_k^2 \\Mest_k}{12} \\norm*{x_k - x_{k-1}}^3\n    - \\frac{1}{2 \\Lest} \\norm*{\\nabla f(y_k)}^2,\n  \\end{alignat}\n  where the last inequality uses \\cref{eq:update_xy} and $0 \\leq \\theta_k \\leq 1$.\n  To evaluate the first term on the right-hand side, we decompose it into four terms:\n  \\begin{align}\n    &\\mathInd\n    \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\\\\\n    &=\n    \\underbrace{2 \\inner*{\\nabla f(y_k)}{y_k - x_k}}_{\\text{(A)}}\n    {}+{} \\underbrace{\\theta_k \\inner*{\\nabla f(x_{k-1})}{y_k - x_k}}_{\\text{(B)}}\\\\[-0.8\\baselineskip]\n    &\\quad\n    - \\underbrace{\\theta_k \\inner*{\\nabla f(x_k)}{y_k - x_k}}_{\\text{(C)}}\n    {}-{} \\underbrace{\n        \\inner*{\n        \\nabla f(y_k)\n        + \\theta_k \\nabla f(x_{k-1})\n        - (1 + \\theta_k) \\nabla f (x_k)\n      }{y_k - x_k}\n    }_{\\text{(D)}}.\n  \\end{align}\n  Each term can be evaluated as\n  \\begin{alignat}{2}\n    \\text{(A)}\n    &=\n    \\frac{1}{\\Lest} \\norm*{\\nabla f(y_k)}^2\n    + \\Lest \\norm*{y_k - x_k}^2\n    - \\Lest \\norm*{(y_k - x_k) - \\frac{1}{\\Lest} \\nabla f(y_k)}^2\\\\\n    &=\n    \\frac{1}{\\Lest} \\norm*{\\nabla f(y_k)}^2\n    + \\theta_k^2 \\Lest \\norm*{x_k - x_{k-1}}^2\n    - \\Lest \\norm*{x_{k+1} - x_k}^2\n    &\\quad&\\by{\\cref{eq:update_xy}},\\\\\n    \\text{(B)}\n    &=\n    \\theta_k^2 \\inner*{\\nabla f(x_{k-1})}{x_k - x_{k-1}}\n    =\n    \\theta_k^2 P_k\n    &\\quad&\\by{\\cref{eq:update_xy}},\\\\\n    \\text{(C)}\n    &=\n    - \\theta_k P_{k+1}\n    + \\theta_k \\inner*{\\nabla f(x_k)}{x_{k+1} - y_k}\\\\\n    &=\n    - \\theta_k P_{k+1}\n    - \\frac{\\theta_k}{\\Lest} \\inner*{\\nabla f(x_k)}{\\nabla f(y_k)}\n    &\\quad&\\by{\\cref{eq:update_xy}},\\\\\n    \\text{(D)}\n    &\\leq\n    \\theta_k \\Mest_k \\norm*{x_k - x_{k-1}}^2 \\norm*{y_k - x_k}\n    &\\quad&\\by{\\cref{eq:grad_interpolation_error_Mestk}}\\\\\n    &=\n    \\theta_k^2 \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    &\\quad&\\by{\\cref{eq:update_xy}}.\n  \\end{alignat}\n  Plugging the evaluations into \\cref{eq:descent_f_first} results in\n  \\begin{align}\n    % &\\mathInd\n    f(x_{k+1}) - f(x_k)\n    \\label{eq:descent_f_second}\n    &\\leq\n    \\frac{\\theta_k^2 \\Lest}{2} \\norm*{x_k - x_{k-1}}^2\n    - \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2\n    + \\frac{\\theta_k^2}{2} P_k\n    - \\frac{\\theta_k}{2} P_{k+1}\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    - \\frac{\\theta_k}{2 \\Lest} \\inner*{\\nabla f(x_k)}{\\nabla f(y_k)}.\n  \\end{align}\n\n  Next, to bound the last term on the right-hand side, we use the following inequality from \\cref{eq:grad_interpolation_error_Mestk}:\n  \\begin{align}\n    \\norm*{\n      \\nabla f(y_k)\n      - (1 + \\theta_k) \\nabla f (x_k)\n    }\n    &\\leq\n    \\theta_k \\norm*{ \\nabla f(x_{k-1})}\n    + \n    \\theta_k \\Mest_k\n    \\norm*{x_k - x_{k-1}}^2.\n  \\end{align}\n  We square both sides as\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{\n      \\nabla f(y_k)\n      - (1 + \\theta_k) \\nabla f (x_k)\n    }^2\\\\\n    &=\n    \\norm*{\\nabla f(y_k)}^2\n    + (1 + \\theta_k)^2 \\norm*{\\nabla f (x_k)}^2\n    - 2 (1 + \\theta_k) \\inner*{\\nabla f (x_k)}{\\nabla f(y_k)}\\\\\n    &\\geq\n    (1 + \\theta_k)^2 \\norm*{\\nabla f (x_k)}^2\n    - 2 (1 + \\theta_k) \\inner*{\\nabla f (x_k)}{\\nabla f(y_k)}\n  \\end{alignat}\n  and\n  \\begin{alignat}{2}\n    \\theta_k^2 \\prn*{\n      \\norm*{ \\nabla f(x_{k-1})}\n      + \\Mest_k \\norm*{x_k - x_{k-1}}^2\n    }^2\n    &\\leq\n    \\theta_k (1 + \\theta_k) \\norm*{ \\nabla f(x_{k-1})}^2\\\\\n    &\\quad\n    + \\theta_k^2 (1 + \\theta_k) \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4,\n  \\end{alignat}\n  where we have used $(a + b)^2 \\leq (1 + 1/\\theta) a^2 + (1 + \\theta) b^2$ for $a, b, \\theta > 0$.\n  Rearranging the terms yields\n  \\[\n  % \\begin{align}\n    - \\inner*{\\nabla f (x_k)}{\\nabla f(y_k)}\n    \\leq\n    \\frac{\\theta_k}{2} \\norm*{ \\nabla f(x_{k-1})}^2\n    - \\frac{1 + \\theta_k}{2} \\norm*{\\nabla f (x_k)}^2\n    + \\frac{\\theta_k^2}{2} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4.\n  % \\end{align}\n  \\]\n  Plugging this bound into \\cref{eq:descent_f_second} results in\n  \\begin{align}\n    \\label{eq:hint_potential}\n    &\\mathInd\n    f(x_{k+1}) - f(x_k)\\\\\n    &\\leq\n    \\frac{\\theta_k^2}{2} \\prn*{\n      P_k\n      + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_{k-1})}^2\n      + \\Lest \\norm*{x_k - x_{k-1}}^2\n    }\\\\\n    &\\quad\n    -\n    \\frac{\\theta_k}{2} \\prn*{\n      P_{k+1} + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_k)}^2\n    }\n    - \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n  \\end{align}\n  This inequality can be rewritten with the potential function $\\Phi_k$ defined in \\cref{eq:def_potential} as\n  \\begin{align}\n    \\Phi_{k+1} - \\Phi_k\n    &\\leq\n    \\frac{\\theta_{k+1}^2 - \\theta_k}{2}\n    \\prn*{\n      P_{k+1}\n      + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_k)}^2\n    }\n    + \\frac{\\theta_{k+1}^2 - 1}{2} \\Lest \\norm*{x_{k+1} - x_k}^2\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n  \\end{align}\n  Finally, using $\\theta_{k+1}^2 - \\theta_k \\leq 0$ from \\cref{eq:theta_increase_not_too_fast} and\n  \\begin{align}\n    - P_{k+1}\n    =\n    - \\inner*{\\nabla f(x_k)}{x_{k+1} - x_k}\n    \\leq\n    \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_k)}^2\n    + \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2\n  \\end{align}\n  from Young's inequality ($\\abs{\\inner*{a}{b}} \\leq \\frac{1}{2} \\norm*{a}^2 + \\frac{1}{2} \\norm*{b}^2$) yields the desired result as\n  \\begin{align}\n    \\Phi_{k+1} - \\Phi_k\n    &\\leq\n    \\frac{\\theta_{k+1}^2 - \\theta_k}{2} \\prn*{- \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2}\n    + \\frac{\\theta_{k+1}^2 - 1}{2} \\Lest \\norm*{x_{k+1} - x_k}^2\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2\\\\\n    &=\n    \\frac{\\theta_{k+1}^2 + \\theta_k - 2}{4} \\Lest \\norm*{x_{k+1} - x_k}^2\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\\\\\n    &\\quad\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n  \\end{align}\n\\end{proof}","start":54211,"end":61291},{"raw_tex":"\\begin{proof}[Proof of \\cref{lem:decrease_epoch}]\n  Note that\n  \\begin{align}\n    \\sum_{i=1}^k \\norm*{x_i - x_{i-1}}^3\n    &\\leq\n    S_k^{3/2},\\quad\n    \\sum_{i=1}^k \\norm*{x_i - x_{i-1}}^4\n    \\leq\n    S_k^2\n    \\label{eq:sum_xk_diff_power34_upperbound}\n  \\end{align}\n  by the monotonicity of $\\ell_p$-norms, i.e., $\\norm{a}_p \\geq \\norm{a}_q$ for $1 \\leq p \\leq q$.\n  In addition, we have\n  \\begin{align}\n    \\Phi_k - f(x_k)\n    &=\n    \\frac{\\theta_k^2}{2} \\prn*{\n      \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_{k-1}) + L (x_k - x_{k-1})}^2\n      + \\frac{\\Lest}{2} \\norm*{x_k - x_{k-1}}^2\n    }\n    \\geq\n    0,\n    \\label{eq:potential_lowerbound}\\\\\n    \\Phi_0 - f(x_0)\n    &=\n    \\frac{\\theta_0^2}{4 \\Lest} \\norm*{\\nabla f(x_0)}^2\n    \\label{eq:potential_obj_0}\n  \\end{align}\n  since $x_{-1} = x_0$.\n\n  Summing \\cref{lem:potential_decrease_iteration} over $k$ and telescoping yields\n  \\begin{align}\n    &\\mathInd\n    \\Phi_k - \\Phi_0\n    \\label{eq:potential_decrease_epoch}\\\\\n    &\\leq\n    \\sum_{i=0}^{k-1}\n    \\bigg(\n      \\frac{\\theta_{i+1}^2 + \\theta_i - 2}{4} \\Lest \\norm*{x_{i+1} - x_i}^2\n      + \\frac{7 \\theta_i^2}{12} \\Mest_i \\norm*{x_i - x_{i-1}}^3\\\\\n      &\\qquad\\qquad\n      + \\frac{\\theta_i^3}{4 \\Lest} \\Mest_i^2 \\norm*{x_i - x_{i-1}}^4\n      - \\frac{\\theta_i^2}{4 \\Lest} \\norm*{\\nabla f(x_i)}^2\n    \\bigg)\\\\\n    &\\leq\n    \\frac{\\theta_k^2 + \\theta_{k-1} - 2}{4} \\Lest S_k\n    + \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} S_{k-1}^{3/2} \n    + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}^2\n    - \\frac{\\theta_0^2}{4 \\Lest} \\norm*{\\nabla f(x_0)}^2,\n  \\end{align}\n  where we have used \\cref{eq:theta_increase_not_too_fast}, \\cref{eq:sum_xk_diff_power34_upperbound}, and $\\Mest_i \\leq \\Mest_{k-1}$.\n  Combining \\cref{eq:potential_lowerbound,eq:potential_obj_0,eq:potential_decrease_epoch} yields\n  \\begin{align}\n    f(x_k) - f(x_0)\n    \\leq\n    \\frac{\\theta_k^2 + \\theta_{k-1} - 2}{4} \\Lest S_k\n    + \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} S_{k-1}^{3/2} \n    + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}^2.\n  \\end{align}\n  Furthermore, we bound the last two terms as\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} S_{k-1}^{3/2} \n    + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}^2\\\\\n    &\\leq\n    \\prn*{\n      \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} \\sqrt{S_{k-1}}\n      + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}\n    }\n    S_k\n    &\\quad&\\by{$S_{k-1} \\leq S_k$}\\\\\n    &\\leq\n    \\prn*{\n      \\frac{7}{12} \\theta_{k-1}^2 (1 - \\theta_{k-1})^{5/2}\n      + \\frac{1}{4} \\theta_{k-1}^3 (1 - \\theta_{k-1})^5\n    }\n    \\Lest S_k\n    &\\quad&\\by{\\cref{eq:condition_continue_k-1}}\\\\\n    &\\leq\n    \\prn*{\n      \\frac{7}{12} + \\frac{1}{4}\n    }\n    \\theta_{k-1}^2 (1 - \\theta_{k-1})^2\n    \\Lest S_k\n    &\\quad&\\by{$0 \\leq \\theta_{k-1} \\leq 1$}\\\\\n    &\\leq\n    \\frac{1}{4}\n    \\theta_{k-1} (1 - \\theta_{k-1})\n    \\Lest S_k\n    &\\quad&\\by{$0 \\leq \\theta_{k-1} \\leq 1$},\n  \\end{alignat}\n  which results in\n  \\begin{alignat}{2}\n    f(x_k) - f(x_0)\n    &\\leq\n    \\frac{\\theta_k^2 + \\theta_{k-1} - 2}{4} \\Lest S_k\n    + \\frac{1}{4} \\theta_{k-1} (1 - \\theta_{k-1}) \\Lest S_k\\\\\n    &=\n    - \\frac{1 - \\theta_k}{2} \\Lest S_k\n    + \\frac{(1 - \\theta_k)^2 - (1 - \\theta_{k-1})^2}{4} \\Lest S_k\\\\\n    &\\leq\n    - \\frac{1 - \\theta_k}{2} \\Lest S_k\n    &\\quad&\\by{\\cref{eq:theta_increase_not_too_fast}}\n  \\end{alignat}\n  and concludes the proof.\n\\end{proof}","start":61293,"end":64695},{"raw_tex":"\\begin{proposition}\n  \\label{prop:Mk_upperbound}\n  Suppose that \\cref{asm:hessian_lip} holds.\n  Then, the following is true throughout \\cref{alg:proposed_agd}: \n  $\\Mest_k \\leq \\max\\set{M_0, \\Mtrue}$.\n\\end{proposition}","start":64788,"end":65006},{"raw_tex":"\\begin{lemma}\n  \\label{lem:grad_interpolation_error}\n  Under \\cref{asm:hessian_lip}, the following hold for all $k \\geq 1$:\n  \\begin{align}\n    &\n    \\norm*{\n      \\nabla f(y_k)\n      + \\theta_k \\nabla f(x_{k-1})\n      - (1 + \\theta_k) \\nabla f (x_k)\n    }\n    \\leq\n    \\theta_k \\Mtrue \\norm*{x_k - x_{k-1}}^2,\n    \\label{eq:grad_interpolation_error_Mf}\\\\\n    &\n    \\norm*{ \\nabla f (\\bar y_k) }\n    \\leq\n    \\frac{\\Lest}{Z_k} \\norm*{x_k - x_{k-1}}\n    + \\frac{(k-1) (k+5)^2}{16 Z_k^2} \\Mtrue S_k.\n    \\label{eq:grad_ybark_upperbound_Mf}\n  \\end{align}\n\\end{lemma}","start":65443,"end":66006},{"raw_tex":"\\begin{proof}[Proof of \\cref{eq:grad_interpolation_error_Mf}]\n  Note that $x_k = \\frac{1}{1 + \\theta_k} y_k + \\frac{\\theta_k}{1 + \\theta_k} x_{k-1}$ from \\cref{eq:update_xy}, and \\cref{lem:gradient_jensen} gives\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{\n      \\nabla f (x_k)\n      - \\frac{1}{1 + \\theta_k} \\nabla f(y_k)\n      - \\frac{\\theta_k}{1 + \\theta_k} \\nabla f(x_{k-1})\n    }\\\\\n    &\\leq\n    \\frac{\\Mtrue \\theta_k}{2 (1 + \\theta_k)^2} \\norm*{y_k - x_{k-1}}^2\\\\\n    &=\n    \\frac{\\Mtrue \\theta_k}{2} \\norm*{x_k - x_{k-1}}^2\n    \\leq\n    \\frac{\\Mtrue \\theta_k}{1 + \\theta_k} \\norm*{x_k - x_{k-1}}^2\n    &\\quad&\\by{\\cref{eq:update_xy} and $0 \\leq \\theta_k \\leq 1$}.\n  \\end{alignat}\n  Multiplying by $(1 + \\theta_k)$ concludes the proof.\n\\end{proof}","start":66135,"end":66890},{"raw_tex":"\\begin{proof}[Proof of \\cref{eq:grad_ybark_upperbound_Mf}]\n  We have\n  \\begin{alignat}{2}\n    &\\mathInd\n    p_{k,i} \\nabla f(y_i)\n    =\n    p_{k,i} \\Lest (y_i - x_{i+1})\\\\\n    &=\n    p_{k,i} \\Lest \\prn*{\n      \\theta_i (x_i - x_{i-1})\n      - (x_{i+1} - x_i)\n    }\n    &\\quad&\\by{\\cref{eq:update_xy}}\\\\\n    &=\n    \\Lest \\prn*{\n      p_{k,i-1} (x_i - x_{i-1})\n      - p_{k,i} (x_{i+1} - x_i)\n    }\n    &\\quad&\\by{definition \\cref{eq:def_ybark} of $p_{k,i}$}.\n  \\end{alignat}\n  Summing this equality over $i$ yields\n  \\begin{align}\n    \\sum_{i=0}^{k-1}\n    p_{k,i} \\nabla f(y_i)\n    =\n    - \\Lest p_{k,k-1} (x_k - x_{k-1}),\n    \\label{eq:weighted_average_gradient_equation}\n  \\end{align}\n  where we have used $x_0 = x_{-1}$.\n  Now, we obtain\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{\n      \\nabla f (\\bar y_k)\n    }\n    \\label{eq:norm_grad_ybark_upperbound_pre}\\\\\n    &\\leq\n    \\norm*{\n      \\sum_{i=0}^{k-1} p_{k,i} \\nabla f \\prn*{ y_i }\n    }\n    + \\frac{\\Mtrue}{2} \\sum_{0 \\leq i < j < k} p_{k,i} p_{k,j} \\norm*{y_i - y_j}^2\n    &\\quad&\\by{\\cref{lem:gradient_jensen}}\\\\\n    &=\n    \\Lest p_{k,k-1} \\norm*{x_k - x_{k-1}}\n    + \\frac{\\Mtrue}{2} \\sum_{0 \\leq i < j < k} p_{k,i} p_{k,j} \\norm*{y_i - y_j}^2\n    &\\quad&\\by{\\cref{eq:weighted_average_gradient_equation}}\\\\\n    &\\leq\n    \\frac{\\Lest}{Z_k} \\norm*{x_k - x_{k-1}}\n    + \\frac{\\Mtrue}{2 Z_k^2} \\sum_{0 \\leq i < j < k} \\norm*{y_i - y_j}^2,\n  \\end{alignat}\n  where the last inequality follows from $p_{k,i} \\leq p_{k,k-1} = 1 / Z_k$ for all $0 \\leq i < k$.\n\n  Next, we bound the second term on the right-hand side.\n  Since for $0 \\leq i < j < k$,\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{y_i - y_j}\\\\\n    &\\leq\n    \\norm*{y_i - x_i}\n    + \\sum_{l=i+1}^{j-1} \\norm*{x_l - x_{l-1}}\n    + \\norm*{y_j - x_{j-1}}\n    &\\quad&\\by{the triangle inequality}\\\\\n    &=\n    \\norm*{x_i - x_{i-1}}\n    + \\sum_{l=i+1}^{j-1} \\norm*{x_l - x_{l-1}}\n    + 2 \\norm*{x_j - x_{j-1}}\n    &\\quad&\\by{\\cref{eq:update_xy} and $0 \\leq \\theta_k \\leq 1$}\\\\\n    &\\leq\n    \\prn*{\n      1^2 + \\sum_{l=i+1}^{j-1} 1^2 + 2^2\n    }^{1/2}\n    \\prn*{\n      \\sum_{l=i}^j \\norm*{x_l - x_{l-1}}^2\n    }^{1/2}\n    &\\quad&\\by{Cauchy--Schwarz}\\\\\n    &=\n    \\sqrt{j - i + 4}\n    \\prn*{\n      \\sum_{l=i}^j \\norm*{x_l - x_{l-1}}^2\n    }^{1/2},\n  \\end{alignat}\n  we have\n  \\begin{align}\n    \\sum_{0 \\leq i < j < k}\n    \\norm*{y_i - y_j}^2\n    &\\leq\n    \\sum_{0 \\leq i < j < k}\n    \\sum_{l=i}^j\n    (j - i + 4) \\norm*{x_l - x_{l-1}}^2\\\\\n    % &=\n    % \\sum_{0 \\leq i \\leq l \\leq j < k}\n    % (j - i + 4) \\norm*{x_l - x_{l-1}}^2\n    % - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &=\n    \\sum_{l=0}^{k-1}\n    \\prn*{\n      \\sum_{i=0}^l\n      \\sum_{j=l}^{k-1}\n      (j - i + 4) \n    }\n    \\norm*{x_l - x_{l-1}}^2\n    - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &=\n    \\frac{k + 7}{2}\n    \\sum_{l=0}^{k-1}\n    (l + 1) (k - l)\n    \\norm*{x_l - x_{l-1}}^2\n    - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &\\leq\n    \\frac{k + 7}{2}\n    \\sum_{l=0}^{k-1} \\frac{(k+1)^2}{4} \\norm*{x_l - x_{l-1}}^2\n    - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &=\n    \\frac{(k-1) (k+5)^2}{8}\n    \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\n    \\leq\n    \\frac{(k-1) (k+5)^2}{8} S_k.\n  \\end{align}\n  Plugging this bound into \\cref{eq:norm_grad_ybark_upperbound_pre} concludes the proof.\n\\end{proof}","start":66891,"end":70197},{"raw_tex":"\\begin{lemma}\n  \\label{lem:grad_norm_ybar_upperbound}\n  Suppose that \\cref{asm:gradient_hessian_lip} holds.\n  In \\cref{alg:proposed_agd}, the following is true when $k \\geq 2$:\n  \\begin{align}\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    \\leq\n    \\frac{4 \\Lmax^2 \\Mmax}{M_0^2 k^4}.\n    \\label{eq:grad_norm_ybar_upperbound}\n  \\end{align}\n  If $M_k$ is computed with \\cref{eq:update_Mest} in the algorithm, the above bound is improved to\n  \\begin{align}\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\leq\n    4 \\Lest \\sqrt{\\frac{S_{k-1}}{k^3}}\n    \\leq\n    \\frac{4 \\Lmax^2}{M_0 k^4}.\n    \\label{eq:grad_norm_ybar_upperbound_tight}\n  \\end{align}\n\\end{lemma}","start":70703,"end":71462},{"raw_tex":"\\begin{proof}[Proof of \\cref{eq:grad_norm_ybar_upperbound}] \n  Note that\n  \\begin{align}\n    &\\sum_{i=1}^{k-1}\n    \\frac{(i-1)(i+5)^2}{16}\n    =\n    \\frac{(k - 2) (k - 1) (k^2 + 13 k + 50)}{64}\n    \\leq\n    \\frac{k^3 (k+1)}{24},\n    \\label{eq:i_cubic_sum}\\\\\n    &\\sum_{i=1}^{k-1} Z_i^2\n    =\n    \\sum_{i=1}^{k-1} \\prn*{\\frac{i+1}{2}}^2\n    =\n    \\frac{(k-1) (2 k^2 + 5 k + 6)}{24}\n    \\geq \n    \\frac{k^2 (k+1)}{12}\n    \\label{eq:Zi_squared_sum}\n  \\end{align}\n  from \\cref{eq:ybar_Zk_alternative} and $k \\geq 2$.\n  Using \\cref{eq:grad_ybark_upperbound_Mf}, we have\n  \\begin{align}\n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\leq\n    \\sum_{i=1}^{k-1}\n    Z_i^2\n    \\norm*{ \\nabla f (\\bar y_i) }\\\\\n    &\\leq\n    \\sum_{i=1}^{k-1}\n    \\prn*{\n      \\Lest Z_i \\norm*{x_i - x_{i-1}}\n      + \\frac{(i-1) (i+5)^2}{16} \\Mtrue S_i\n    }.\n  \\end{align}\n  Each term on the right-hand side can be bounded as follows:\n  \\begin{alignat}{2}\n    &\\sum_{i=1}^{k-1} \\Lest Z_i \\norm*{x_i - x_{i-1}}\n    \\leq\n    \\Lest\n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }^{1/2}\n    \\sqrt{S_{k-1}}\n    &\\quad&\\by{Cauchy--Schwarz},\\\\\n    &\\sum_{i=1}^{k-1}\n    \\frac{(i-1)(i+5)^2}{16} \\Mtrue S_i\n    \\leq\n    \\frac{k^3 (k+1)}{24}\n    \\Mtrue S_{k-1}\n    &\\quad&\\by{\\cref{eq:i_cubic_sum}}.\n  \\end{alignat}\n  We thus have\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\\\\\n    &\\leq\n    \\Lest \n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }^{-1/2}\\!\\!\n    \\sqrt{S_{k-1}}\n    + \n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }^{-1}\n    \\frac{k^3 (k+1)}{24}\n    \\Mtrue S_{k-1}\\\\\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{k}{2}\n    \\Mtrue S_{k-1}\n    &\\quad&\\by{\\cref{eq:Zi_squared_sum}}\\\\\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{\\Lest \\Mtrue}{2 \\Mest_k} \n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{\\cref{eq:condition_continue_k-1}}\\\\\n    &\\leq\n    \\frac{\\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{\\Lest \\Mmax}{2 \\Mest_{k-1}} \n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{$\\Mest_{k-1}, \\Mtrue \\leq \\Mmax$}\\\\\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}},\n  \\end{alignat}\n  which is the first inequality of the desired result~\\cref{eq:grad_norm_ybar_upperbound}.\n  The second inequality follows from \\cref{eq:condition_continue_k-1}:\n  \\begin{align}\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    \\leq\n    \\frac{4 \\Lest^2 \\Mmax}{M_{k-1}^2 k^4}\n    \\leq\n    \\frac{4 \\Lmax^2 \\Mmax}{M_0 k^4}.\n  \\end{align}\n\\end{proof}","start":71463,"end":74094},{"raw_tex":"\\begin{proof}[Proof of \\cref{eq:grad_norm_ybar_upperbound_tight}]\n  When $M_k$ is computed with \\cref{eq:update_Mest}, we have \\cref{eq:grad_ybark_upperbound_Mestk}.\n  Similarly to the proof of \\cref{eq:grad_norm_ybar_upperbound}, using \\cref{eq:grad_ybark_upperbound_Mestk} instead of \\cref{eq:grad_ybark_upperbound_Mf} yields\n  \\begin{alignat}{2}\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{k}{2} \\Mest_{k-1} S_{k-1}.\n  \\end{alignat}\n  The right-hand side is bounded as\n  \\begin{alignat}{2}\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{k}{2} \\Mest_{k-1} S_{k-1}\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{\\Lest}{2} \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{\\cref{eq:condition_continue_k-1}}\\\\\n    &\\leq\n    4 \\Lest\n    \\sqrt{\\frac{S_{k-1}}{k^3}},\n  \\end{alignat}\n  which implies the first inequality in \\cref{eq:grad_norm_ybar_upperbound_tight}.\n  The second inequality is obtained similarly to the second inequality of \\cref{eq:grad_norm_ybar_upperbound}.\n\\end{proof}","start":74095,"end":75190},{"raw_tex":"\\begin{proof}[Proof of bound \\cref{eq:complexity_bound}]\n  We count the number of iterations of \\cref{alg:proposed_agd} separately for three types of epoch:\n  \\begin{itemize}\n    \\item \n    successful epoch: an epoch that does not find an $\\epsilon$-stationary point and ends at Line \\ref{alg-line-agd:restart_successful} with the descent condition~\\cref{eq:descent_condition} satisfied,\n    \\item\n    unsuccessful epoch: an epoch that does not find an $\\epsilon$-stationary point and ends at Line \\ref{alg-line-agd:restart_unsuccessful} with the descent condition~\\cref{eq:descent_condition} unsatisfied,\n    \\item\n    last epoch: the epoch that finds an $\\epsilon$-stationary point.\n  \\end{itemize}\n\n  \\paragraph{Successful epochs}\n  Let us focus on a successful epoch.\n  Let $k$ be the iteration number of the epoch.\n  It follows that \n  \\begin{align}\n    (k+1)^5\n    \\Mest_k^2 S_k\n    &>\n    \\Lest^2.\n    \\label{eq:restart_condition_k}\n  \\end{align}\n  When $k \\geq 2$, we have\n  \\begin{alignat}{2}\n    \\epsilon\n    &<\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\quad&\\text{(since this epoch does not find an $\\epsilon$-stationary point)}\\\\\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{\\cref{eq:grad_norm_ybar_upperbound}}\\\\\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_0} \\sqrt{\\frac{S_k}{k^3}}\n    &\\quad&\\by{$M_{k-1} \\geq M_0$ and $S_k \\geq S_{k-1}$},\n  \\end{alignat}\n  and hence \n  \\begin{align}\n    S_k > \\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3.\n    \\label{eq:Sk_lowerbound}\n  \\end{align}\n  When $k = 1$, we also have \\cref{eq:Sk_lowerbound}, as follows:\n  \\begin{align}\n    S_k\n    =\n    \\norm*{x_1 - x_0}^2\n    =\n    \\frac{1}{\\Lest^2} \\norm*{\\nabla f(x_0)}^2\n    =\n    \\frac{1}{\\Lest^2} \\norm*{\\nabla f(\\bar y_1)}^2\n    >\n    \\frac{\\epsilon^2}{\\Lest^2}\n    \\geq\n    \\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3.\n  \\end{align}\n  Combining \\cref{eq:Sk_lowerbound,eq:restart_condition_k} yields\n  \\begin{align}\n    S_k\n    =\n    S_k^{7/8}\n    S_k^{1/8}\n    &\\geq\n    \\prn[\\bigg]{\\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3}^{7/8}\n    \\prn*{\\frac{\\Lest^2}{\\Mest_k^2 (k+1)^5}}^{1/8}\\\\\n    &=\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{3/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^{7/4} \\Mest_k^{1/4}}\n    \\frac{k^{21/8}}{(k+1)^{5/8}}\n    \\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{3/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^2}\n    \\frac{k^{21/8}}{(k+1)^{5/8}}.\n  \\end{align}\n  Plugging this bound into condition \\cref{eq:descent_condition}, we obtain\n  \\begin{align}\n    f(x_0) - f(x_k)\n    \\geq\n    \\frac{\\Lest S_k}{2 (k+1)}\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{1/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^2}\n    \\prn*{\\frac{k}{k+1}}^{13/8}\n    \\frac{k}{2}\\\\\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lmax^{1/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^2}\n    \\prn*{\\frac{1}{2}}^{13/8}\n    \\frac{k}{2}\n    \\geq\n    \\frac{\\epsilon^{7/4} M_0^{7/4}}{\\Lmax^{1/2} \\Mmax^2}\n    \\frac{k}{70},\n  \\end{align}\n  and thus\n  \\begin{align}\n    k\n    \\leq\n    70\n    \\frac{\\Lmax^{1/2} \\Mmax^2}{\\epsilon^{7/4} M_0^{7/4}}\n    \\prn*{f(x_0) - f(x_k)}.\n  \\end{align}\n  Summing this bound over all successful epochs leads to the conclusion that the total iteration number of successful epochs is at most\n  \\begin{align}\n    70 \\frac{\\Lmax^{1/2} \\Mmax^2 \\Delta}{\\epsilon^{7/4} M_0^{7/4}}.\n    \\label{eq:Ksuc_upperbound}\n  \\end{align}\n  For later use, we will also evaluate the number of successful epochs.\n  Combining \\cref{eq:Sk_lowerbound,eq:restart_condition_k} also yields\n  \\begin{align}\n    S_k\n    =\n    S_k^{3/4}\n    S_k^{1/4}\n    &\\geq\n    \\prn[\\bigg]{ \\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3 }^{3/4}\n    \\prn*{\\frac{\\Lest^2}{\\Mest_k^2 (k+1)^5}}^{1/4}\\\\\n    &=\n    \\frac{(\\epsilon / 4)^{3/2}}{\\Lest}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^{3/2} \\Mest_k^{1/2}}\n    \\frac{k^{9/4}}{(k+1)^{5/4}}\n    \\geq\n    \\frac{(\\epsilon / 4)^{3/2}}{\\Lest}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^2}\n    \\frac{k^{9/4}}{(k+1)^{5/4}}.\n  \\end{align}\n  Plugging this bound into condition \\cref{eq:descent_condition} gives\n  \\begin{align}\n    f(x_0) - f(x_k)\n    \\geq\n    \\frac{\\Lest S_k}{2 (k+1)}\n    &\\geq\n    \\frac{(\\epsilon / 4)^{3/2}}{2}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^2}\n    \\prn*{\\frac{k}{k+1}}^{9/4}\\\\\n    &\\geq\n    \\frac{(\\epsilon / 4)^{3/2}}{2}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^2}\n    \\prn*{\\frac{1}{2}}^{9/4}\n    =\n    \\frac{\\epsilon^{3/2} \\Mest_0^{3/2}}{2^{25/4} \\Mmax^2}.\n  \\end{align}\n  Thus, we can deduce that the number $N_{\\mathrm{suc}}$ of successful epochs satisfies\n  \\begin{align}\n    N_{\\mathrm{suc}}\n    \\leq\n    2^{25/4}\n    \\frac{\\Mmax^2 \\Delta}{\\epsilon^{3/2} \\Mest_0^{3/2}}.\n    \\label{eq:Nsuc_upperbound}\n  \\end{align}\n\n  \\paragraph{Other epochs}\n  We deduce from \\cref{eq:grad_norm_ybar_upperbound} that the number of iterations for each epoch is at most\n  \\begin{align}\n    \\prn*{\n      \\frac{4 \\Lmax^2 \\Mmax}{\\epsilon M_0^2}\n    }^{1/4}.\n  \\end{align}\n  Let $N_{\\mathrm{unsuc}}$ be the number of unsuccessful epochs.\n  We then have\n  \\begin{align}\n    \\Linit\n    \\beta^{N_{\\mathrm{suc}}}\n    \\alpha^{N_{\\mathrm{unsuc}}}\n    \\leq\n    \\Lmax,\n  \\end{align}\n  and hence\n  \\begin{align}\n    N_{\\mathrm{unsuc}}\n    \\leq\n    c_1 N_{\\mathrm{suc}} + c_2,\n    \\label{eq:Nunsuc_upperbound}\n  \\end{align}\n  where $c_1$ and $c_2$ are defined by \\cref{eq:def_c1_c2}.\n  The total iteration number of unsuccessful and last epochs is at most\n  \\begin{alignat}{2}\n    &\\mathInd\n    (N_{\\mathrm{unsuc}} + 1)\n    \\prn*{ \\frac{4 \\Lmax^2 \\Mmax}{\\epsilon M_0^2} }^{1/4}\\\\\n    &\\leq\n    \\prn*{ c_1 N_{\\mathrm{suc}} + c_2 + 1 }\n    \\prn*{ \\frac{4 \\Lmax^2 \\Mmax}{\\epsilon M_0^2} }^{1/4}\n    &\\quad&\\by{\\cref{eq:Nunsuc_upperbound}}\\\\\n    &\\leq\n    2^{27/4} c_1\n    \\frac{\\Lmax^{1/2} \\Mmax^{9/4} \\Delta}{\\epsilon^{7/4} \\Mest_0^2}\n    + 2^{1/2} (c_2 + 1) \\frac{\\Lmax^{1/2} \\Mmax^{1/4}}{\\epsilon^{1/4} M_0^{1/2}}\n    &\\quad&\\by{\\cref{eq:Nsuc_upperbound}}\\\\\n    &\\leq\n    108 c_1\n    \\frac{\\Lmax^{1/2} \\Mmax^{9/4} \\Delta}{\\epsilon^{7/4} \\Mest_0^2}\n    + 2 (c_2 + 1) \\frac{\\Lmax^{1/2} \\Mmax^{1/4}}{\\epsilon^{1/4} M_0^{1/2}}.\n  \\end{alignat}\n  Putting this bound together with \\cref{eq:Ksuc_upperbound} concludes the proof.\n\\end{proof}","start":76725,"end":82876},{"raw_tex":"\\begin{proof}[Proof of bound \\cref{eq:complexity_bound_tight}]\n  The proof is similar to \\cref{eq:complexity_bound} through the evaluation of the total iteration number of successful, unsuccessful, and last epochs.\n\n  \\paragraph{Successful epochs}\n  Let us focus on a successful epoch.\n  Let $k$ be the iteration number of the epoch.\n  As a counterpart of \\cref{eq:Sk_lowerbound}, we can obtain \n  \\begin{align}\n    S_k > \\prn*{\\frac{\\epsilon}{4 \\Lest}}^2 k^3\n    \\label{eq:Sk_lowerbound_tight}\n  \\end{align}\n  from \\cref{eq:grad_norm_ybar_upperbound_tight}.\n  Combining \\cref{eq:Sk_lowerbound_tight,eq:restart_condition_k} yields\n  \\begin{align}\n    S_k\n    &=\n    S_k^{7/8}\n    S_k^{1/8}\n    \\geq\n    \\prn*{\\prn*{\\frac{\\epsilon}{4 \\Lest}}^2 k^3}^{7/8}\n    \\prn*{\\frac{\\Lest^2}{\\Mest_k^2 (k+1)^5}}^{1/8}\n    =\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{3/2} \\Mest_k^{1/4}}\n    \\frac{k^{21/8}}{(k+1)^{5/8}}.\n  \\end{align}\n  Plugging this bound into condition \\cref{eq:descent_condition}, we obtain\n  \\begin{align}\n    f(x_0) - f(x_k)\n    \\geq\n    \\frac{\\Lest S_k}{2 (k+1)}\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{1/2} \\Mest_k^{1/4}}\n    \\prn*{\\frac{k}{k+1}}^{13/8}\n    \\frac{k}{2}\\\\\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lmax^{1/2} \\Mmax^{1/4}}\n    \\prn*{\\frac{1}{2}}^{13/8}\n    \\frac{k}{2}\n    \\geq\n    \\frac{\\epsilon^{7/4}}{\\Lmax^{1/2} \\Mmax^{1/4}}\n    \\frac{k}{70}.\n  \\end{align}\n  Hence, as in the proof of \\cref{eq:complexity_bound}, the total iteration number of all successful epochs is at most\n  \\begin{align}\n    70\n    \\frac{\\Lmax^{1/2} \\Mmax^{1/4} \\Delta}{\\epsilon^{7/4}}.\n    \\label{eq:Ksuc_upperbound_tight}\n  \\end{align}\n\n  \\paragraph{Other epochs}\n  We deduce from \\cref{eq:grad_norm_ybar_upperbound_tight} that the iteration number of each epoch is at most $\\prn[\\big]{ \\frac{4 \\Lmax^2}{M_0 \\epsilon} }^{1/4}$.\n  In addition, unsuccessful epochs occur only at most $c_2$ times when $\\beta = 1$, where $c_2$ is defined by \\cref{eq:def_c1_c2}.\n  Thus, the total iteration number of unsuccessful and last epochs is at most\n  \\begin{align}\n    \\prn*{\n      c_2 + 1\n    }\n    \\prn*{\n      \\frac{4 \\Lmax^2}{M_0 \\epsilon}\n    }^{1/4}\n    \\leq\n    2 (c_2 + 1) \\frac{\\Lmax^{1/2}}{M_0^{1/4} \\epsilon^{1/4}}.\n  \\end{align}\n  Putting this bound together with \\cref{eq:Ksuc_upperbound_tight} concludes the proof.\n\\end{proof}","start":82878,"end":85215},{"raw_tex":"\\begin{proof}[Proof of \\cref{lem:gradient_jensen}]\n  To simplify the notation, let $\\bar z \\coloneqq \\sum_{i=1}^n \\lambda_i z_i$.\n  Taylor's theorem gives\n  \\[\n    \\nabla f(z_i) - \\nabla f(\\bar z)\n    =\n    \\nabla^2 f(\\bar z) (z_i - \\bar z)\n    + \\int_0^1 \\prn*{ \\nabla^2 f(\\bar z + t (z_i - \\bar z)) - \\nabla^2 f(\\bar z) } (z_i - \\bar z) dt,\n  \\]\n  and its weighted average gives\n  \\begin{align}\n    \\sum_{i=1}^n \\lambda_i \\nabla f(z_i) - \\nabla f(\\bar z)\n    &=\n    \\sum_{i=1}^n \\lambda_i\n    \\int_0^1 \\prn*{ \\nabla^2 f(\\bar z + t (z_i - \\bar z)) - \\nabla^2 f(\\bar z) } (z_i - \\bar z) dt.\n  \\end{align}\n  Therefore, we obtain\n  \\begin{alignat}{2}\n    \\norm*{\n      \\sum_{i=1}^n \\lambda_i \\nabla f(z_i) - \\nabla f(\\bar z)\n    }\n    &\\leq\n    \\sum_{i=1}^n \\lambda_i\n    \\int_0^1 \\norm*{ \\nabla^2 f(\\bar z + t (z_i - \\bar z)) - \\nabla^2 f(\\bar z) } \\norm*{z_i - \\bar z} dt\\\\\n    &\\leq\n    \\sum_{i=1}^n \\lambda_i\n    \\int_0^1 M_f t \\norm*{z_i - \\bar z}^2 dt\n    \\quad\\by{\\cref{asm:hessian_lip}}\\\\\n    &=\n    \\frac{M_f}{2} \\sum_{i=1}^n \\lambda_i \\norm*{z_i - \\bar z}^2.\n  \\end{alignat}\n  Furthermore, elementary algebra shows that $\\sum_{i=1}^n \\lambda_i \\norm*{z_i - \\bar z}^2 = \\sum_{1 \\leq i < j \\leq n} \\lambda_i \\lambda_j \\|z_i - z_j\\|^2$ as\n  \\begin{align}\n    \\sum_{1 \\leq i < j \\leq n} \\lambda_i \\lambda_j \\norm*{z_i - z_j}^2\n    &= \n    \\frac{1}{2} \\sum_{i,j = 1}^n \\lambda_i \\lambda_j \\norm*{z_i - z_j}^2\\\\\n    &= \n    \\frac{1}{2} \\sum_{i,j = 1}^n \\lambda_i \\lambda_j \\norm*{z_i}^2\n    + \\frac{1}{2} \\sum_{i,j = 1}^n \\lambda_i \\lambda_j \\norm*{z_j}^2\n    - \\sum_{i,j = 1}^n \\inner{\\lambda_i z_i}{\\lambda_j z_j}\\\\\n    &= \n    \\sum_{i=1}^n \\lambda_i \\norm*{z_i}^2 - \\norm*{\\bar z}^2\n    =\n    \\sum_{i=1}^n \\lambda_i \\norm*{z_i - \\bar z}^2,\n  \\end{align}\n  which completes the proof.\n\\end{proof}","start":106558,"end":108357},{"raw_tex":"\\begin{proof}[Proof of \\cref{lem:trapezoidal_rule_error}]\n  We obtain the desired result as follows:\n  \\begin{alignat}{2}\n    &\\mathInd\n    % \\abs*{\n      f(x) - f(y)\n      - \\frac{1}{2} \\inner*{\\nabla f(x) + \\nabla f(y)}{x - y}\\\\\n    % }\\\\\n    &=\n    % \\abs*{\n      \\int_0^1 \\inner*{\\nabla f(t x + (1 - t) y)}{x - y} dt\n      - \\frac{1}{2} \\inner*{\\nabla f(x) + \\nabla f(y)}{x - y}\n    % }\n    &\\quad&\\by{Taylor's\\\\theorem}\\\\\n    &=\n    % \\abs*{\n      \\int_0^1 \\inner[\\Big]{\\nabla f(t x + (1 - t) y) - t \\nabla f(x) - (1 - t) \\nabla f(y)}{x - y} dt\\\\\n    % }\\\\\n    &\\leq\n    \\int_0^1 \\norm[\\Big]{\\nabla f(t x + (1 - t) y) - t \\nabla f(x) - (1 - t) \\nabla f(y)} \\norm{x - y} dt\\\\\n    &\\leq\n    \\frac{\\Mtrue}{2} \\int_0^1 t (1 - t) \\norm{x - y}^3 dt\n    =\n    \\frac{\\Mtrue}{12} \\norm{x - y}^3\n    &\\quad&\\by{\\cref{lem:gradient_jensen}}.\n  \\end{alignat}\n\\end{proof}","start":108360,"end":109222}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"有下界的非凸最小化","source_units":[{"raw_tex":"This paper studies general nonconvex optimization problems:\n\\begin{align}\n  \\min_{x \\in \\R^d} \\ \n  f(x),\n\\end{align}\nwhere $f: \\R^d \\to \\R$ is bounded below (i.e., $\\inf_{x \\in \\R^d} f(x) > - \\infty$) and satisfies the following Lipschitz assumption.\n","start":14872,"end":15123}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":["asm:gradient_hessian_lip","asm:gradient_lip","asm:hessian_lip"],"declared_external":false,"source_only":false,"title":"梯度全局 Lipschitz","source_units":[{"raw_tex":"\\begin{assumption}\n  \\label{asm:gradient_hessian_lip}\n  Let $\\Ltrue, \\Mtrue > 0$ be constants.\n  \\begin{enuminasm}\n    \\item\n    \\label{asm:gradient_lip}\n    $\\norm*{\\nabla f(x) - \\nabla f(y)} \\leq \\Ltrue \\norm*{x - y}$ for all $x, y \\in \\R^d$,\n","start":15123,"end":15368}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","labels":["asm:gradient_hessian_lip","asm:gradient_lip","asm:hessian_lip"],"declared_external":false,"source_only":false,"title":"Hessian 全局 Lipschitz","source_units":[{"raw_tex":"  Let $\\Ltrue, \\Mtrue > 0$ be constants.\n","start":15177,"end":15218},{"raw_tex":"    \\item\n    \\label{asm:hessian_lip}\n    $\\norm*{\\nabla^2 f(x) - \\nabla^2 f(y)} \\leq \\Mtrue \\norm*{x - y}$ for all $x, y \\in \\R^d$.\n  \\end{enuminasm}\n\\end{assumption}\n","start":15368,"end":15536}]},{"id":"S004","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"内积与范数约定","source_units":[{"raw_tex":"\\paragraph{Notation}\nFor vectors $a, b \\in \\R^d$, let $\\inner{a}{b}$ denote the dot product and $\\norm{a}$ denote the Euclidean norm.\nFor a matrix $A \\in \\R^{m \\times n}$, let $\\norm{A}$ denote the operator norm, or equivalently the largest singular value.\n","start":21361,"end":21618}]},{"id":"S005","semantic_type":"lemma","kind":"lemma","labels":["lem:eq:property_gradlip_obj"],"declared_external":true,"declaration_text":["原始允许前提清单明确允许 B0012 标准光滑下降引理，不要求提供文内证明。"],"source_only":false,"title":"标准光滑下降引理","source_units":[{"raw_tex":"\\begin{lemma}\n  \\label{lem:eq:property_gradlip_obj}\n  Under \\cref{asm:gradient_lip}, the following holds for any $x, y \\in \\R^d$:\n  \\begin{align}\n    f(x) - f(y)\n    &\\leq\n    \\inner*{\\nabla f(y)}{x - y}\n    + \\frac{\\Ltrue}{2} \\norm*{x - y}^2.\n  \\end{align}\n\\end{lemma}\n","start":32617,"end":32887}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"轮次与计数器","source_units":[{"raw_tex":"The proposed method is the classical accelerated gradient descent (AGD) \\cite{nesterov1983method,nesterov2018lectures} with two restart schemes, presented in \\cref{alg:proposed_agd}.\nIn the algorithm, the iteration counter $k$ is reset to $0$ when AGD restarts on Line \\ref{alg-line-agd:restart_unsuccessful} or \\ref{alg-line-agd:restart_successful}, whereas the total iteration counter $K$ is not.\nWe refer to the period between a reset of $k$ and the next reset as an \\emph{epoch}.\nNote that the total iteration counter $K$ in the algorithm is unnecessary to implement; it is included here only to make the statements in our analysis concise.\n\nThe algorithm uses estimates $L$ and $M_k$ for the Lipschitz constants $L_f$ and $M_f$.\nThe estimate $L$ is fixed during an epoch, while $M_k$ is updated at each iteration, having the subscript $k$.\nNow let us describe each operation of \\cref{alg:proposed_agd}.\n\n","start":39638,"end":40547}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":["eq:update_xy"],"declared_external":false,"source_only":false,"title":"加速梯度迭代","source_units":[{"raw_tex":"\\paragraph{Solution update (Lines \\ref{alg-line-agd:update_x} and \\ref{alg-line-agd:update_y})}\nWith an estimate $L$ of the Lipschitz constant $L_f$, AGD defines solution sequences $(x_k)$ and $(y_k)$ as follows:\n$y_0 = x_0$ and\n\\begin{align}\n  % \\left\\{\n  \\begin{aligned}\n    x_k\n    &=\n    y_{k-1} - \\frac{1}{L} \\nabla f(y_{k-1}),\\\\\n    y_k\n    &=\n    x_k + \\theta_k (x_k - x_{k-1})\n  \\end{aligned}\n  % \\right.\n  \\label{eq:update_xy}\n\\end{align}\nfor $k \\geq 1$, where $0 \\leq \\theta_k \\leq 1$ is the acceleration parameter.\nLet $x_{-1} \\coloneqq x_0$ for convenience, which makes the lower equation of \\cref{eq:update_xy} valid for $k = 0$.\nSetting a suitable value of $\\theta_k$ is crucial for deriving an improved complexity bound over that of gradient descent.\n","start":40547,"end":41313}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":["eq:def_theta_k"],"declared_external":false,"source_only":false,"title":"加速参数的具体选择","source_units":[{"raw_tex":"In particular, we propose to set\n\\begin{align}\n  \\theta_k = \\frac{k}{k+1}\n  \\label{eq:def_theta_k}\n\\end{align}\nfor $k \\geq 1$.\n","start":41313,"end":41440}]},{"id":"S009","semantic_type":"algorithm","kind":"algorithm","labels":["alg-line-agd:check_decrease","alg-line-agd:check_large_Sk","alg-line-agd:restart_successful","alg-line-agd:restart_unsuccessful","alg-line-agd:update_x","alg-line-agd:update_y","alg:proposed_agd"],"declared_external":false,"source_only":false,"title":"双重重启的无参数 AGD","source_units":[{"raw_tex":"\\begin{algorithm}[t]\n  \\caption{Proposed AGD with restart~\\label{alg:proposed_agd}} \n  \\begin{algorithmic}[1]\n    \\Require{%\n      \\pbox[t]{40em}{\n        $x_{\\mathrm{init}} \\in \\R^d$;\n        $\\Linit, M_0 > 0$;\n        $\\alpha > 1$;\n        $0 < \\beta \\leq 1$\\\\\n        Recommended: $(\\Linit, M_0, \\alpha, \\beta) = (10^{-3}, 10^{-16}, 2, 0.9)$\n      }\n    }\n    \\State{%\n      $(x_0, y_0) \\gets (x_{\\mathrm{init}}, x_{\\mathrm{init}})$,\n      $\\Lest \\gets \\Linit$,\n      $k \\gets 0$,\n      $K \\gets 0$\n    }\n    \\Repeat\n      \\State{%\n        $k \\gets k + 1$,\n        $K \\gets K + 1$\n      }\n      \\State $x_k \\gets y_{k-1} - \\frac{1}{\\Lest} \\nabla f(y_{k-1})$\n      \\label{alg-line-agd:update_x}\n      \\State $y_k \\gets x_k + \\frac{k}{k+1} (x_k - x_{k-1})$\n      \\label{alg-line-agd:update_y}\n\n      \\If{$f(x_k) > f(x_0) - \\frac{\\Lest S_k}{2 (k+1)}$}\n      \\Comment{$S_k \\coloneqq \\sum_{i=1}^k \\norm*{x_i - x_{i-1}}^2$}\n      \\label{alg-line-agd:check_decrease}\n        \\State{%\n          $(x_0, y_0) \\gets (x_{k-1}, x_{k-1})$,\n          $\\Lest \\gets \\alpha \\Lest$,\n          $k \\gets 0$\n        }\n        \\label{alg-line-agd:restart_unsuccessful}\n      \\ElsIf{$(k+1)^5 \\Mest_k^2 S_k > \\Lest^2$}\n      \\Comment{$\\Mest_k$ is defined by \\cref{eq:update_Mest_practical} or \\cref{eq:update_Mest}}\n      \\label{alg-line-agd:check_large_Sk}\n        \\State{%\n          $(x_0, y_0) \\gets (x_k, x_k)$,\n          $\\Lest \\gets \\beta \\Lest$,\n          $k \\gets 0$\n        }\n        \\label{alg-line-agd:restart_successful}\n      \\EndIf\n    \\Until{convergence}\n    \\State \\Return averaged solution $\\bar y_k$ defined by \\eqref{eq:def_ybark}\n  \\end{algorithmic}\n\\end{algorithm}\n","start":41876,"end":43540}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":["eq:descent_condition"],"declared_external":false,"source_only":false,"title":"下降检验与累计位移","source_units":[{"raw_tex":"\\paragraph{Restart mechanism 1 (Lines \\ref{alg-line-agd:check_decrease} and \\ref{alg-line-agd:restart_unsuccessful})}\nAfter updating the solutions, \\cref{alg:proposed_agd} checks whether the current estimate $L$ approximates the actual value $L_f$ well.\nIf\n\\begin{align}\n  f(x_k) - f(x_0)\n  \\leq\n  - \\frac{1 - \\theta_k}{2} \\Lest S_k\n  =\n  - \\frac{\\Lest S_k}{2 (k+1)},\n  \\quad\\text{where}\\quad\n  S_k\n  \\coloneqq\n  \\sum_{i=1}^k \\norm*{x_i - x_{i-1}}^2,\n  \\label{eq:descent_condition}\n\\end{align}\nthen $L$ is judged suitable, and the epoch continues; otherwise, AGD restarts from $x_{k-1}$ with a larger $L$.\n","start":43542,"end":44148}]},{"id":"S011","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"失败重启不增加目标值","source_units":[{"raw_tex":"Since condition~\\cref{eq:descent_condition} holds in the previous iteration, starting the next epoch from $x_{k-1}$ ensures that this epoch does not increase the objective function value.\nCondition~\\cref{eq:descent_condition} guarantees a sufficient decrease in the objective function in the epoch and plays a crucial role in our analysis.\nAs will be shown later, this condition is always satisfied when $L$ is sufficiently large, which justifies the restart mechanism.\n","start":44148,"end":44618}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["eq:condition_continue"],"declared_external":false,"source_only":false,"title":"续行检验与成功重启","source_units":[{"raw_tex":"\\paragraph{Restart mechanism 2 (Lines \\ref{alg-line-agd:check_large_Sk} and \\ref{alg-line-agd:restart_successful})}\nOur AGD also resets the effect of acceleration by restart when $S_k$ becomes large; the restart condition involves an estimate $M_k$ of the Hessian's Lipschitz constant $M_f$ (detailed later).\nIf\n\\begin{align}\n  \\Mest_k^2 S_k\n  \\leq\n  \\Lest^2 (1 - \\theta_k)^5\n  =\n  \\frac{\\Lest^2}{(k+1)^5},\n  \\label{eq:condition_continue}\n\\end{align}\nthe epoch continues, otherwise, AGD restarts with the computed solution $x_k$ and a (possibly) smaller $L$.\nCondition~\\cref{eq:condition_continue} is inspired by a condition in \\cite{li2022restarted}, $k M_f S_k \\leq \\epsilon$; unlike the existing one, our condition does not involve $M_f$ or $\\epsilon$.\nAlthough $L$ does not have to be decreased on Line~\\ref{alg-line-agd:restart_successful} to derive the complexity bound of $O(\\epsilon^{-7/4})$, decreasing $L$ will help the algorithm to capture the local curvature of $f$ and to converge faster empirically.\n","start":45311,"end":46325}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["eq:def_ybark"],"declared_external":false,"source_only":false,"title":"加权平均点与归一化因子","source_units":[{"raw_tex":"\\paragraph{Averaged solution}\n\\label{sec:averaged_solution}\nOur AGD guarantees that the gradient norm is small at a weighted average $\\bar y_k$ of the solutions $y_0,\\dots,y_{k-1}$.\nThe averaged solution $\\bar y_k$ is defined by\n\\begin{align}\n  \\bar y_k\n  \\coloneqq\n  \\sum_{i=0}^{k-1} p_{k,i} y_i,\n  \\quad\\text{where}\\quad\n  p_{k,i}\n  \\coloneqq\n  \\frac{1}{Z_k} \\prod_{j=i+1}^{k-1} \\theta_j,\n  \\quad\n  Z_k\n  \\coloneqq\n  \\sum_{i=0}^{k-1} \\prod_{j=i+1}^{k-1} \\theta_j.\n  \\label{eq:def_ybark}\n\\end{align}\n","start":46327,"end":46828}]},{"id":"S014","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"平均点的递推实现","source_units":[{"raw_tex":"Note that it is unnecessary to keep all $y_0,\\dots,y_{k-1}$ in memory to compute $\\bar y_k$ because $Z_k$ and $\\bar y_k$ satisfy a simple recursion:\n\\begin{align}\n  Z_{k+1}\n  &=\n  1 + \\theta_k Z_k,\\quad\n  \\bar y_{k+1}\n  =\n  \\frac{1}{Z_{k+1}} \\prn*{y_k + \\theta_k Z_k \\bar y_k},\n\\end{align}\nwhich maintains the computational efficiency of the algorithm.\n","start":46828,"end":47181}]},{"id":"S015","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:ybar_Zk_alternative"],"declared_external":false,"source_only":false,"title":"平均权重的显式形式","source_units":[{"raw_tex":"Given \\cref{eq:def_theta_k}, there is another form:\n\\begin{align}\n  \\bar y_k\n  =\n  \\frac{2}{k (k + 1)}\n  \\sum_{i=0}^{k-1} (i + 1) y_i,\\quad\n  Z_k\n  =\n  \\frac{k + 1}{2},\n  \\label{eq:ybar_Zk_alternative}\n\\end{align}\nand this simplification of $\\bar y_k$ and $Z_k$ is one of the aims of setting $\\theta_k$ as in \\cref{eq:def_theta_k}.\n","start":47181,"end":47513}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":["eq:fy_fx_diff_upperbound_Mestk","eq:grad_interpolation_error_Mestk"],"declared_external":false,"source_only":false,"title":"Hessian 估计须满足的条件","source_units":[{"raw_tex":"\\paragraph{Estimate of Hessian's Lipschitz constant}\nThe estimate $M_k$ for the Hessian is updated at each iteration, unlike $L$ for the gradient.\nOur complexity analysis requires three inequalities on $M_k$: $M_{k-1} \\leq M_k$ and\n\\begin{align}\n  &\n  f(y_k) - f(x_k)\n  \\leq\n  \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n  + \\frac{\\Mest_k}{12} \\norm*{y_k - x_k}^3,\n  \\label{eq:fy_fx_diff_upperbound_Mestk}\\\\\n  &\n  \\norm*{\n    \\nabla f(y_k)\n    + \\theta_k \\nabla f(x_{k-1})\n    - (1 + \\theta_k) \\nabla f (x_k)\n  }\n  \\leq\n  \\theta_k \\Mest_k \\norm*{x_k - x_{k-1}}^2.\n  \\label{eq:grad_interpolation_error_Mestk}\n\\end{align}\n","start":47515,"end":48153}]},{"id":"S017","semantic_type":"algorithm","kind":"algorithm","labels":["eq:update_Mest_practical"],"declared_external":false,"source_only":false,"title":"实用 Hessian 估计规则","source_units":[{"raw_tex":"Finding the smallest $M_k$ that satisfies these inequalities is desirable, and fortunately, it is straightforward,\n\\begin{align}\n  \\label{eq:update_Mest_practical}\n  \\Mest_k\n  =\n  \\max \\bigg\\{\n    \\Mest_{k-1},\\ \n    &\n    \\frac{\n      12\n      \\prn*{\n        f(y_k) - f(x_k) \n        - \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n      }\n    }{\\norm*{y_k - x_k}^3},\\\\\n    &\n    \\frac{\n      \\norm*{\n        \\nabla f(y_k)\n        + \\theta_k \\nabla f(x_{k-1})\n        - (1 + \\theta_k) \\nabla f (x_k)\n      }\n    }{\\theta_k \\norm*{x_k - x_{k-1}}^2}\n  \\bigg\\}\n\\end{align}\nbecause $x_k$, $y_k$, and $\\theta_k$ are already in hand.\nIn other words, our $\\theta_k$ in \\cref{eq:def_theta_k} does not depend on the estimate $M_k$ of $M_f$, which facilitates computation of $M_k$.\nThus, we can estimate $M_f$ without backtracking, thereby simplifying the algorithm and improving its efficiency.\n\n","start":48153,"end":49056}]},{"id":"S018","semantic_type":"algorithm","kind":"algorithm","labels":["eq:update_Mest"],"declared_external":false,"source_only":false,"title":"强化 Hessian 估计规则","source_units":[{"raw_tex":"Instead of \\cref{eq:update_Mest_practical}, we can use\n\\begin{align}\n  \\label{eq:update_Mest}\n  \\Mest_k\n  =\n  \\max \\bigg\\{\n    \\Mest_{k-1},\\ \n    &\n    \\frac{\n      12\n      \\prn*{\n        f(y_k) - f(x_k) \n        - \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n      }\n    }{\\norm*{y_k - x_k}^3}\n    ,\\\\\n    &\n    \\frac{\n      \\norm*{\n        \\nabla f(y_k)\n        + \\theta_k \\nabla f(x_{k-1})\n        - (1 + \\theta_k) \\nabla f (x_k)\n      }\n    }{\\theta_k \\norm*{x_k - x_{k-1}}^2},\\\\\n    &\n    \\frac{\n      16\n      \\prn*{\n        Z_k^2 \\norm*{ \\nabla f (\\bar y_k) }\n        - Z_k \\Lest \\norm*{x_k - x_{k-1}}\n      }\n    }{(k-1) (k+5)^2 S_k}\n  \\bigg\\}.\n\\end{align}\nin the algorithm.\\footnote{\n  For $k = 1$, \\Cref{eq:update_Mest} takes the form $M_1 = \\max \\set{M_0, \\ast, \\ast, \\frac{0}{0}}$, which should be treated as $M_1 = \\max \\set{M_0, \\ast, \\ast}$.\n}\n","start":49056,"end":49933}]},{"id":"S019","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:grad_ybark_upperbound_Mestk"],"declared_external":false,"source_only":false,"title":"强化规则强制的平均梯度界","source_units":[{"raw_tex":"This replacement ensures that\n\\begin{align}\n  \\norm*{ \\nabla f (\\bar y_k) }\n  \\leq\n  \\frac{\\Lest}{Z_k} \\norm*{x_k - x_{k-1}}\n  + \\frac{(k-1) (k+5)^2}{16 Z_k^2} \\Mest_k S_k\n  \\label{eq:grad_ybark_upperbound_Mestk}\n\\end{align}\nand will lead to a smaller complexity bound with lessened dependence on $M_0$ and $M_f$, as will be confirmed later in \\cref{thm:complexity}.\n","start":49933,"end":50300}]},{"id":"S020","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"两种估计规则的代价比较","source_units":[{"raw_tex":"\\Cref{eq:update_Mest} derives a theoretically better bound than \\cref{eq:update_Mest_practical} but requires an additional gradient evaluation at $\\bar y_k$, increasing the computational cost per iteration.\nTherefore, we recommend using \\cref{eq:update_Mest_practical} in practice.\n","start":50300,"end":50582}]},{"id":"S021","semantic_type":"definition","kind":"definition","labels":["eq:def_potential"],"declared_external":false,"source_only":false,"title":"含梯度校正的势函数","source_units":[{"raw_tex":"To obtain the descent condition~\\cref{eq:descent_condition}, we first evaluate the decrease for one iteration.\nHowever, a common difficulty arises when analyzing AGD: the objective function value does not necessarily decrease monotonically.\nTo deal with the problem, we introduce a potential function related to $f(x_k)$ and show that it is \\emph{nearly} decreasing.\nThe potential function $\\Phi_k$ is defined by\n\\begin{align}\n  \\Phi_k\n  \\coloneqq\n  f(x_k)\n  + \\frac{\\theta_k^2}{2} \\prn*{\n    \\inner*{\\nabla f(x_{k-1})}{x_k - x_{k-1}}\n    + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_{k-1})}^2\n    + \\Lest \\norm*{x_k - x_{k-1}}^2\n  },\n  \\label{eq:def_potential}\n\\end{align}\ninspired by \\cref{eq:hint_potential} used in the proof.\nThe value $\\Phi_k$ decreases when $\\norm*{x_k - x_{k-1}}$ is small enough as the following lemma shows.\n","start":51624,"end":52451}]},{"id":"S022","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:condition_continue_k-1"],"declared_external":false,"source_only":false,"title":"上一迭代未重启的约束","source_units":[{"raw_tex":"To evaluate the terms of $\\norm*{x_k - x_{k-1}}^3$ and $\\norm*{x_k - x_{k-1}}^4$ in \\cref{lem:potential_decrease_iteration} in these calculations, we use \n\\begin{align}\n  \\Mest_{k-1}^2 S_{k-1}\n  \\leq\n  \\Lest^2 (1 - \\theta_{k-1})^5\n  =\n  \\frac{\\Lest^2}{k^5},\n  \\label{eq:condition_continue_k-1}\n\\end{align}\nwhich follows from the fact that the epoch did not end at iteration $k-1$.\n","start":53349,"end":53730}]},{"id":"S023","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:theta_increase_not_too_fast"],"declared_external":false,"source_only":false,"title":"加速参数的增长约束","source_units":[{"raw_tex":"\nNow, we prove \\cref{lem:potential_decrease_iteration} and use it to prove \\cref{lem:decrease_epoch}.\nThe following proofs do not use the specific form \\cref{eq:def_theta_k} of $\\theta_k$ but rather a more general condition on $\\theta_k$,\n\\begin{align}\n  \\theta_{k+1}^2 \\leq \\theta_k \\leq \\theta_{k+1},\n  \\label{eq:theta_increase_not_too_fast}\n\\end{align}\nwhich is easily verified for all $k \\geq 0$ under $\\theta_0 \\coloneqq \\theta_1^2$ and \\cref{eq:def_theta_k} for $k \\geq 1$.\n","start":53730,"end":54210}]},{"id":"S024","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"统一估计上界","source_units":[{"raw_tex":"\\subsection{Upper bound on gradient norm}\nLet $\\Lmax$ and $\\Mmax$ be the upper bounds on $L$ and $M_k$ given in \\cref{prop:L_upperbound,prop:Mk_upperbound}: $\\Lmax \\coloneqq \\max\\set{\\Linit, \\alpha \\Ltrue}$ and $\\Mmax \\coloneqq \\max\\set{M_0, \\Mtrue}$.\nThe following lemma provides upper bounds on the gradient norm.\n","start":70200,"end":70516}]},{"id":"S025","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"函数与梯度评估复杂度","source_units":[{"raw_tex":"The number of function and gradient evaluations is of the same order as the iteration complexity given in \\cref{thm:complexity}, because \\cref{alg:proposed_agd} evaluates the objective function and the gradient at two or three points in each iteration.\n","start":85955,"end":86208}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","labels":["lem:gradient_jensen"],"declared_external":false,"source_only":false,"title":"梯度的 Jensen 型不等式","source_units":[{"raw_tex":"\\begin{lemma}[Jensen-type inequality for gradients]\n  \\label{lem:gradient_jensen}\n  Under \\cref{asm:hessian_lip}, the following holds for any $z_1, \\dots, z_n \\in \\R^d$ and $\\lambda_1,\\dots,\\lambda_n \\geq 0$ such that $\\sum_{i=1}^n \\lambda_i = 1$:\n  \\begin{align}\n    \\norm*{\n      \\nabla f \\prn*{ \\sum_{i=1}^n \\lambda_i z_i }\n      - \\sum_{i=1}^n \\lambda_i \\nabla f \\prn*{ z_i }\n    }\n    \\leq\n    \\frac{\\Mtrue}{2} \\sum_{1 \\leq i < j \\leq n} \\lambda_i \\lambda_j \\norm*{z_i - z_j}^2.\n  \\end{align}\n\\end{lemma}","start":35516,"end":36025}]},{"id":"S027","semantic_type":"lemma","kind":"lemma","labels":["lem:trapezoidal_rule_error"],"declared_external":false,"source_only":false,"title":"梯形公式的三次误差界","source_units":[{"raw_tex":"\\begin{lemma}[Error bound for trapezoidal rule]\n  \\label{lem:trapezoidal_rule_error}\n  Under \\cref{asm:hessian_lip}, the following holds for any $x, y \\in \\R^d$:\n  \\begin{align}\n    f(x) - f(y)\n    &\\leq\n    \\frac{1}{2} \\inner*{\\nabla f(x) + \\nabla f(y)}{x - y}\n    + \\frac{\\Mtrue}{12} \\norm*{x - y}^3.\n  \\end{align}\n\\end{lemma}","start":36026,"end":36354}]},{"id":"S028","semantic_type":"proposition","kind":"proposition","labels":["prop:L_upperbound"],"declared_external":false,"source_only":false,"title":"梯度 Lipschitz 估计的上界","source_units":[{"raw_tex":"\\begin{proposition}\n  \\label{prop:L_upperbound}\n  Suppose that \\cref{asm:gradient_lip} holds.\n  Then, the following is true throughout \\cref{alg:proposed_agd}: $\\Lest \\leq \\max\\set{\\Linit, \\alpha \\Ltrue}$.\n\\end{proposition}","start":51077,"end":51300}]},{"id":"S029","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"估计上界的作者简证","source_units":[{"raw_tex":"This proposition immediately follows from the following lemma.\n","start":51301,"end":51364}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","labels":["lem:decrease_epoch"],"declared_external":false,"source_only":false,"title":"充分大的估计保证 epoch 下降","source_units":[{"raw_tex":"\\begin{lemma}\n  \\label{lem:decrease_epoch}\n  Suppose that \\cref{asm:gradient_lip} holds.\n  During epochs with $\\Lest \\geq \\Ltrue$, condition~\\cref{eq:descent_condition} always holds.\n\\end{lemma}","start":51364,"end":51558}]},{"id":"S031","semantic_type":"lemma","kind":"lemma","labels":["eq:potential_decrease_iteration","lem:potential_decrease_iteration"],"declared_external":false,"source_only":false,"title":"势函数单步下降界","source_units":[{"raw_tex":"\\begin{lemma}\n  \\label{lem:potential_decrease_iteration}\n  Under \\cref{asm:gradient_lip} and $\\Lest \\geq \\Ltrue$, the following holds for all $k \\geq 0$:\n  \\begin{align}\n    \\Phi_{k+1} - \\Phi_k\n    &\\leq\n    \\frac{\\theta_{k+1}^2 + \\theta_k - 2}{4} \\Lest \\norm*{x_{k+1} - x_k}^2\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\\\\\n    &\\quad\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n    \\label{eq:potential_decrease_iteration}\n  \\end{align}\n\\end{lemma}","start":52451,"end":52999}]},{"id":"S032","semantic_type":"proof","kind":"proof","labels":["eq:descent_f_first","eq:descent_f_second","eq:hint_potential"],"declared_external":false,"source_only":false,"title":"势函数下降的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{lem:potential_decrease_iteration}]\n  In the proof, let\n  \\begin{align}\n    P_k\n    &\\coloneqq\n    \\inner*{\\nabla f(x_{k-1})}{x_k - x_{k-1}}\n  \\end{align}\n  to simplify the notation.\n  From \\cref{lem:eq:property_gradlip_obj,eq:update_xy}, we have\n  \\begin{align}\n    f(x_{k+1}) - f(y_k)\n    &\\leq\n    \\inner*{\\nabla f(y_k)}{x_{k+1} - y_k}\n    + \\frac{\\Lest}{2} \\norm*{x_{k+1} - y_k}^2\n    = \n    - \\frac{1}{2 \\Lest} \\norm*{\\nabla f(y_k)}^2.\n  \\end{align}\n  Summing this bound and \\cref{eq:fy_fx_diff_upperbound_Mestk} yields\n  \\begin{alignat}{2}\n    &\\mathInd\n    f(x_{k+1}) - f(x_k)\n    \\label{eq:descent_f_first}\\\\\n    &\\leq\n    \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n    + \\frac{\\Mest_k}{12} \\norm*{y_k - x_k}^3\n    - \\frac{1}{2 \\Lest} \\norm*{\\nabla f(y_k)}^2\\\\\n    &\\leq\n    \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n    + \\frac{\\theta_k^2 \\Mest_k}{12} \\norm*{x_k - x_{k-1}}^3\n    - \\frac{1}{2 \\Lest} \\norm*{\\nabla f(y_k)}^2,\n  \\end{alignat}\n  where the last inequality uses \\cref{eq:update_xy} and $0 \\leq \\theta_k \\leq 1$.\n  To evaluate the first term on the right-hand side, we decompose it into four terms:\n  \\begin{align}\n    &\\mathInd\n    \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\\\\\n    &=\n    \\underbrace{2 \\inner*{\\nabla f(y_k)}{y_k - x_k}}_{\\text{(A)}}\n    {}+{} \\underbrace{\\theta_k \\inner*{\\nabla f(x_{k-1})}{y_k - x_k}}_{\\text{(B)}}\\\\[-0.8\\baselineskip]\n    &\\quad\n    - \\underbrace{\\theta_k \\inner*{\\nabla f(x_k)}{y_k - x_k}}_{\\text{(C)}}\n    {}-{} \\underbrace{\n        \\inner*{\n        \\nabla f(y_k)\n        + \\theta_k \\nabla f(x_{k-1})\n        - (1 + \\theta_k) \\nabla f (x_k)\n      }{y_k - x_k}\n    }_{\\text{(D)}}.\n  \\end{align}\n  Each term can be evaluated as\n  \\begin{alignat}{2}\n    \\text{(A)}\n    &=\n    \\frac{1}{\\Lest} \\norm*{\\nabla f(y_k)}^2\n    + \\Lest \\norm*{y_k - x_k}^2\n    - \\Lest \\norm*{(y_k - x_k) - \\frac{1}{\\Lest} \\nabla f(y_k)}^2\\\\\n    &=\n    \\frac{1}{\\Lest} \\norm*{\\nabla f(y_k)}^2\n    + \\theta_k^2 \\Lest \\norm*{x_k - x_{k-1}}^2\n    - \\Lest \\norm*{x_{k+1} - x_k}^2\n    &\\quad&\\by{\\cref{eq:update_xy}},\\\\\n    \\text{(B)}\n    &=\n    \\theta_k^2 \\inner*{\\nabla f(x_{k-1})}{x_k - x_{k-1}}\n    =\n    \\theta_k^2 P_k\n    &\\quad&\\by{\\cref{eq:update_xy}},\\\\\n    \\text{(C)}\n    &=\n    - \\theta_k P_{k+1}\n    + \\theta_k \\inner*{\\nabla f(x_k)}{x_{k+1} - y_k}\\\\\n    &=\n    - \\theta_k P_{k+1}\n    - \\frac{\\theta_k}{\\Lest} \\inner*{\\nabla f(x_k)}{\\nabla f(y_k)}\n    &\\quad&\\by{\\cref{eq:update_xy}},\\\\\n    \\text{(D)}\n    &\\leq\n    \\theta_k \\Mest_k \\norm*{x_k - x_{k-1}}^2 \\norm*{y_k - x_k}\n    &\\quad&\\by{\\cref{eq:grad_interpolation_error_Mestk}}\\\\\n    &=\n    \\theta_k^2 \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    &\\quad&\\by{\\cref{eq:update_xy}}.\n  \\end{alignat}\n  Plugging the evaluations into \\cref{eq:descent_f_first} results in\n  \\begin{align}\n    % &\\mathInd\n    f(x_{k+1}) - f(x_k)\n    \\label{eq:descent_f_second}\n    &\\leq\n    \\frac{\\theta_k^2 \\Lest}{2} \\norm*{x_k - x_{k-1}}^2\n    - \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2\n    + \\frac{\\theta_k^2}{2} P_k\n    - \\frac{\\theta_k}{2} P_{k+1}\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    - \\frac{\\theta_k}{2 \\Lest} \\inner*{\\nabla f(x_k)}{\\nabla f(y_k)}.\n  \\end{align}\n\n  Next, to bound the last term on the right-hand side, we use the following inequality from \\cref{eq:grad_interpolation_error_Mestk}:\n  \\begin{align}\n    \\norm*{\n      \\nabla f(y_k)\n      - (1 + \\theta_k) \\nabla f (x_k)\n    }\n    &\\leq\n    \\theta_k \\norm*{ \\nabla f(x_{k-1})}\n    + \n    \\theta_k \\Mest_k\n    \\norm*{x_k - x_{k-1}}^2.\n  \\end{align}\n  We square both sides as\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{\n      \\nabla f(y_k)\n      - (1 + \\theta_k) \\nabla f (x_k)\n    }^2\\\\\n    &=\n    \\norm*{\\nabla f(y_k)}^2\n    + (1 + \\theta_k)^2 \\norm*{\\nabla f (x_k)}^2\n    - 2 (1 + \\theta_k) \\inner*{\\nabla f (x_k)}{\\nabla f(y_k)}\\\\\n    &\\geq\n    (1 + \\theta_k)^2 \\norm*{\\nabla f (x_k)}^2\n    - 2 (1 + \\theta_k) \\inner*{\\nabla f (x_k)}{\\nabla f(y_k)}\n  \\end{alignat}\n  and\n  \\begin{alignat}{2}\n    \\theta_k^2 \\prn*{\n      \\norm*{ \\nabla f(x_{k-1})}\n      + \\Mest_k \\norm*{x_k - x_{k-1}}^2\n    }^2\n    &\\leq\n    \\theta_k (1 + \\theta_k) \\norm*{ \\nabla f(x_{k-1})}^2\\\\\n    &\\quad\n    + \\theta_k^2 (1 + \\theta_k) \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4,\n  \\end{alignat}\n  where we have used $(a + b)^2 \\leq (1 + 1/\\theta) a^2 + (1 + \\theta) b^2$ for $a, b, \\theta > 0$.\n  Rearranging the terms yields\n  \\[\n  % \\begin{align}\n    - \\inner*{\\nabla f (x_k)}{\\nabla f(y_k)}\n    \\leq\n    \\frac{\\theta_k}{2} \\norm*{ \\nabla f(x_{k-1})}^2\n    - \\frac{1 + \\theta_k}{2} \\norm*{\\nabla f (x_k)}^2\n    + \\frac{\\theta_k^2}{2} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4.\n  % \\end{align}\n  \\]\n  Plugging this bound into \\cref{eq:descent_f_second} results in\n  \\begin{align}\n    \\label{eq:hint_potential}\n    &\\mathInd\n    f(x_{k+1}) - f(x_k)\\\\\n    &\\leq\n    \\frac{\\theta_k^2}{2} \\prn*{\n      P_k\n      + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_{k-1})}^2\n      + \\Lest \\norm*{x_k - x_{k-1}}^2\n    }\\\\\n    &\\quad\n    -\n    \\frac{\\theta_k}{2} \\prn*{\n      P_{k+1} + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_k)}^2\n    }\n    - \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n  \\end{align}\n  This inequality can be rewritten with the potential function $\\Phi_k$ defined in \\cref{eq:def_potential} as\n  \\begin{align}\n    \\Phi_{k+1} - \\Phi_k\n    &\\leq\n    \\frac{\\theta_{k+1}^2 - \\theta_k}{2}\n    \\prn*{\n      P_{k+1}\n      + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_k)}^2\n    }\n    + \\frac{\\theta_{k+1}^2 - 1}{2} \\Lest \\norm*{x_{k+1} - x_k}^2\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n  \\end{align}\n  Finally, using $\\theta_{k+1}^2 - \\theta_k \\leq 0$ from \\cref{eq:theta_increase_not_too_fast} and\n  \\begin{align}\n    - P_{k+1}\n    =\n    - \\inner*{\\nabla f(x_k)}{x_{k+1} - x_k}\n    \\leq\n    \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_k)}^2\n    + \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2\n  \\end{align}\n  from Young's inequality ($\\abs{\\inner*{a}{b}} \\leq \\frac{1}{2} \\norm*{a}^2 + \\frac{1}{2} \\norm*{b}^2$) yields the desired result as\n  \\begin{align}\n    \\Phi_{k+1} - \\Phi_k\n    &\\leq\n    \\frac{\\theta_{k+1}^2 - \\theta_k}{2} \\prn*{- \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2}\n    + \\frac{\\theta_{k+1}^2 - 1}{2} \\Lest \\norm*{x_{k+1} - x_k}^2\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2\\\\\n    &=\n    \\frac{\\theta_{k+1}^2 + \\theta_k - 2}{4} \\Lest \\norm*{x_{k+1} - x_k}^2\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\\\\\n    &\\quad\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n  \\end{align}\n\\end{proof}","start":54211,"end":61291}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":["eq:descent_f_first"],"declared_external":false,"source_only":false,"title":"合并梯度步与梯形误差","source_units":[{"raw_tex":"  From \\cref{lem:eq:property_gradlip_obj,eq:update_xy}, we have\n  \\begin{align}\n    f(x_{k+1}) - f(y_k)\n    &\\leq\n    \\inner*{\\nabla f(y_k)}{x_{k+1} - y_k}\n    + \\frac{\\Lest}{2} \\norm*{x_{k+1} - y_k}^2\n    = \n    - \\frac{1}{2 \\Lest} \\norm*{\\nabla f(y_k)}^2.\n  \\end{align}\n  Summing this bound and \\cref{eq:fy_fx_diff_upperbound_Mestk} yields\n  \\begin{alignat}{2}\n    &\\mathInd\n    f(x_{k+1}) - f(x_k)\n    \\label{eq:descent_f_first}\\\\\n    &\\leq\n    \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n    + \\frac{\\Mest_k}{12} \\norm*{y_k - x_k}^3\n    - \\frac{1}{2 \\Lest} \\norm*{\\nabla f(y_k)}^2\\\\\n    &\\leq\n    \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n    + \\frac{\\theta_k^2 \\Mest_k}{12} \\norm*{x_k - x_{k-1}}^3\n    - \\frac{1}{2 \\Lest} \\norm*{\\nabla f(y_k)}^2,\n  \\end{alignat}\n  where the last inequality uses \\cref{eq:update_xy} and $0 \\leq \\theta_k \\leq 1$.\n","start":54422,"end":55311}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":["eq:descent_f_second"],"declared_external":false,"source_only":false,"title":"四项分解控制交叉项","source_units":[{"raw_tex":"  To evaluate the first term on the right-hand side, we decompose it into four terms:\n  \\begin{align}\n    &\\mathInd\n    \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\\\\\n    &=\n    \\underbrace{2 \\inner*{\\nabla f(y_k)}{y_k - x_k}}_{\\text{(A)}}\n    {}+{} \\underbrace{\\theta_k \\inner*{\\nabla f(x_{k-1})}{y_k - x_k}}_{\\text{(B)}}\\\\[-0.8\\baselineskip]\n    &\\quad\n    - \\underbrace{\\theta_k \\inner*{\\nabla f(x_k)}{y_k - x_k}}_{\\text{(C)}}\n    {}-{} \\underbrace{\n        \\inner*{\n        \\nabla f(y_k)\n        + \\theta_k \\nabla f(x_{k-1})\n        - (1 + \\theta_k) \\nabla f (x_k)\n      }{y_k - x_k}\n    }_{\\text{(D)}}.\n  \\end{align}\n  Each term can be evaluated as\n  \\begin{alignat}{2}\n    \\text{(A)}\n    &=\n    \\frac{1}{\\Lest} \\norm*{\\nabla f(y_k)}^2\n    + \\Lest \\norm*{y_k - x_k}^2\n    - \\Lest \\norm*{(y_k - x_k) - \\frac{1}{\\Lest} \\nabla f(y_k)}^2\\\\\n    &=\n    \\frac{1}{\\Lest} \\norm*{\\nabla f(y_k)}^2\n    + \\theta_k^2 \\Lest \\norm*{x_k - x_{k-1}}^2\n    - \\Lest \\norm*{x_{k+1} - x_k}^2\n    &\\quad&\\by{\\cref{eq:update_xy}},\\\\\n    \\text{(B)}\n    &=\n    \\theta_k^2 \\inner*{\\nabla f(x_{k-1})}{x_k - x_{k-1}}\n    =\n    \\theta_k^2 P_k\n    &\\quad&\\by{\\cref{eq:update_xy}},\\\\\n    \\text{(C)}\n    &=\n    - \\theta_k P_{k+1}\n    + \\theta_k \\inner*{\\nabla f(x_k)}{x_{k+1} - y_k}\\\\\n    &=\n    - \\theta_k P_{k+1}\n    - \\frac{\\theta_k}{\\Lest} \\inner*{\\nabla f(x_k)}{\\nabla f(y_k)}\n    &\\quad&\\by{\\cref{eq:update_xy}},\\\\\n    \\text{(D)}\n    &\\leq\n    \\theta_k \\Mest_k \\norm*{x_k - x_{k-1}}^2 \\norm*{y_k - x_k}\n    &\\quad&\\by{\\cref{eq:grad_interpolation_error_Mestk}}\\\\\n    &=\n    \\theta_k^2 \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    &\\quad&\\by{\\cref{eq:update_xy}}.\n  \\end{alignat}\n  Plugging the evaluations into \\cref{eq:descent_f_first} results in\n  \\begin{align}\n    % &\\mathInd\n    f(x_{k+1}) - f(x_k)\n    \\label{eq:descent_f_second}\n    &\\leq\n    \\frac{\\theta_k^2 \\Lest}{2} \\norm*{x_k - x_{k-1}}^2\n    - \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2\n    + \\frac{\\theta_k^2}{2} P_k\n    - \\frac{\\theta_k}{2} P_{k+1}\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    - \\frac{\\theta_k}{2 \\Lest} \\inner*{\\nabla f(x_k)}{\\nabla f(y_k)}.\n  \\end{align}\n","start":55311,"end":57454}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"插值残差控制梯度内积","source_units":[{"raw_tex":"  Next, to bound the last term on the right-hand side, we use the following inequality from \\cref{eq:grad_interpolation_error_Mestk}:\n  \\begin{align}\n    \\norm*{\n      \\nabla f(y_k)\n      - (1 + \\theta_k) \\nabla f (x_k)\n    }\n    &\\leq\n    \\theta_k \\norm*{ \\nabla f(x_{k-1})}\n    + \n    \\theta_k \\Mest_k\n    \\norm*{x_k - x_{k-1}}^2.\n  \\end{align}\n  We square both sides as\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{\n      \\nabla f(y_k)\n      - (1 + \\theta_k) \\nabla f (x_k)\n    }^2\\\\\n    &=\n    \\norm*{\\nabla f(y_k)}^2\n    + (1 + \\theta_k)^2 \\norm*{\\nabla f (x_k)}^2\n    - 2 (1 + \\theta_k) \\inner*{\\nabla f (x_k)}{\\nabla f(y_k)}\\\\\n    &\\geq\n    (1 + \\theta_k)^2 \\norm*{\\nabla f (x_k)}^2\n    - 2 (1 + \\theta_k) \\inner*{\\nabla f (x_k)}{\\nabla f(y_k)}\n  \\end{alignat}\n  and\n  \\begin{alignat}{2}\n    \\theta_k^2 \\prn*{\n      \\norm*{ \\nabla f(x_{k-1})}\n      + \\Mest_k \\norm*{x_k - x_{k-1}}^2\n    }^2\n    &\\leq\n    \\theta_k (1 + \\theta_k) \\norm*{ \\nabla f(x_{k-1})}^2\\\\\n    &\\quad\n    + \\theta_k^2 (1 + \\theta_k) \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4,\n  \\end{alignat}\n  where we have used $(a + b)^2 \\leq (1 + 1/\\theta) a^2 + (1 + \\theta) b^2$ for $a, b, \\theta > 0$.\n  Rearranging the terms yields\n  \\[\n  % \\begin{align}\n    - \\inner*{\\nabla f (x_k)}{\\nabla f(y_k)}\n    \\leq\n    \\frac{\\theta_k}{2} \\norm*{ \\nabla f(x_{k-1})}^2\n    - \\frac{1 + \\theta_k}{2} \\norm*{\\nabla f (x_k)}^2\n    + \\frac{\\theta_k^2}{2} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4.\n  % \\end{align}\n  \\]\n","start":57455,"end":58914}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":["eq:hint_potential"],"declared_external":false,"source_only":false,"title":"整理为势函数差","source_units":[{"raw_tex":"  Plugging this bound into \\cref{eq:descent_f_second} results in\n  \\begin{align}\n    \\label{eq:hint_potential}\n    &\\mathInd\n    f(x_{k+1}) - f(x_k)\\\\\n    &\\leq\n    \\frac{\\theta_k^2}{2} \\prn*{\n      P_k\n      + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_{k-1})}^2\n      + \\Lest \\norm*{x_k - x_{k-1}}^2\n    }\\\\\n    &\\quad\n    -\n    \\frac{\\theta_k}{2} \\prn*{\n      P_{k+1} + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_k)}^2\n    }\n    - \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n  \\end{align}\n  This inequality can be rewritten with the potential function $\\Phi_k$ defined in \\cref{eq:def_potential} as\n  \\begin{align}\n    \\Phi_{k+1} - \\Phi_k\n    &\\leq\n    \\frac{\\theta_{k+1}^2 - \\theta_k}{2}\n    \\prn*{\n      P_{k+1}\n      + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_k)}^2\n    }\n    + \\frac{\\theta_{k+1}^2 - 1}{2} \\Lest \\norm*{x_{k+1} - x_k}^2\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n  \\end{align}\n","start":58914,"end":60150}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"利用参数符号完成单步界","source_units":[{"raw_tex":"  Finally, using $\\theta_{k+1}^2 - \\theta_k \\leq 0$ from \\cref{eq:theta_increase_not_too_fast} and\n  \\begin{align}\n    - P_{k+1}\n    =\n    - \\inner*{\\nabla f(x_k)}{x_{k+1} - x_k}\n    \\leq\n    \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_k)}^2\n    + \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2\n  \\end{align}\n  from Young's inequality ($\\abs{\\inner*{a}{b}} \\leq \\frac{1}{2} \\norm*{a}^2 + \\frac{1}{2} \\norm*{b}^2$) yields the desired result as\n  \\begin{align}\n    \\Phi_{k+1} - \\Phi_k\n    &\\leq\n    \\frac{\\theta_{k+1}^2 - \\theta_k}{2} \\prn*{- \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2}\n    + \\frac{\\theta_{k+1}^2 - 1}{2} \\Lest \\norm*{x_{k+1} - x_k}^2\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2\\\\\n    &=\n    \\frac{\\theta_{k+1}^2 + \\theta_k - 2}{4} \\Lest \\norm*{x_{k+1} - x_k}^2\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\\\\\n    &\\quad\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n  \\end{align}\n","start":60150,"end":61280}]},{"id":"S038","semantic_type":"proof","kind":"proof","labels":["eq:potential_decrease_epoch","eq:potential_lowerbound","eq:potential_obj_0","eq:sum_xk_diff_power34_upperbound"],"declared_external":false,"source_only":false,"title":"epoch 下降的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{lem:decrease_epoch}]\n  Note that\n  \\begin{align}\n    \\sum_{i=1}^k \\norm*{x_i - x_{i-1}}^3\n    &\\leq\n    S_k^{3/2},\\quad\n    \\sum_{i=1}^k \\norm*{x_i - x_{i-1}}^4\n    \\leq\n    S_k^2\n    \\label{eq:sum_xk_diff_power34_upperbound}\n  \\end{align}\n  by the monotonicity of $\\ell_p$-norms, i.e., $\\norm{a}_p \\geq \\norm{a}_q$ for $1 \\leq p \\leq q$.\n  In addition, we have\n  \\begin{align}\n    \\Phi_k - f(x_k)\n    &=\n    \\frac{\\theta_k^2}{2} \\prn*{\n      \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_{k-1}) + L (x_k - x_{k-1})}^2\n      + \\frac{\\Lest}{2} \\norm*{x_k - x_{k-1}}^2\n    }\n    \\geq\n    0,\n    \\label{eq:potential_lowerbound}\\\\\n    \\Phi_0 - f(x_0)\n    &=\n    \\frac{\\theta_0^2}{4 \\Lest} \\norm*{\\nabla f(x_0)}^2\n    \\label{eq:potential_obj_0}\n  \\end{align}\n  since $x_{-1} = x_0$.\n\n  Summing \\cref{lem:potential_decrease_iteration} over $k$ and telescoping yields\n  \\begin{align}\n    &\\mathInd\n    \\Phi_k - \\Phi_0\n    \\label{eq:potential_decrease_epoch}\\\\\n    &\\leq\n    \\sum_{i=0}^{k-1}\n    \\bigg(\n      \\frac{\\theta_{i+1}^2 + \\theta_i - 2}{4} \\Lest \\norm*{x_{i+1} - x_i}^2\n      + \\frac{7 \\theta_i^2}{12} \\Mest_i \\norm*{x_i - x_{i-1}}^3\\\\\n      &\\qquad\\qquad\n      + \\frac{\\theta_i^3}{4 \\Lest} \\Mest_i^2 \\norm*{x_i - x_{i-1}}^4\n      - \\frac{\\theta_i^2}{4 \\Lest} \\norm*{\\nabla f(x_i)}^2\n    \\bigg)\\\\\n    &\\leq\n    \\frac{\\theta_k^2 + \\theta_{k-1} - 2}{4} \\Lest S_k\n    + \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} S_{k-1}^{3/2} \n    + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}^2\n    - \\frac{\\theta_0^2}{4 \\Lest} \\norm*{\\nabla f(x_0)}^2,\n  \\end{align}\n  where we have used \\cref{eq:theta_increase_not_too_fast}, \\cref{eq:sum_xk_diff_power34_upperbound}, and $\\Mest_i \\leq \\Mest_{k-1}$.\n  Combining \\cref{eq:potential_lowerbound,eq:potential_obj_0,eq:potential_decrease_epoch} yields\n  \\begin{align}\n    f(x_k) - f(x_0)\n    \\leq\n    \\frac{\\theta_k^2 + \\theta_{k-1} - 2}{4} \\Lest S_k\n    + \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} S_{k-1}^{3/2} \n    + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}^2.\n  \\end{align}\n  Furthermore, we bound the last two terms as\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} S_{k-1}^{3/2} \n    + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}^2\\\\\n    &\\leq\n    \\prn*{\n      \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} \\sqrt{S_{k-1}}\n      + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}\n    }\n    S_k\n    &\\quad&\\by{$S_{k-1} \\leq S_k$}\\\\\n    &\\leq\n    \\prn*{\n      \\frac{7}{12} \\theta_{k-1}^2 (1 - \\theta_{k-1})^{5/2}\n      + \\frac{1}{4} \\theta_{k-1}^3 (1 - \\theta_{k-1})^5\n    }\n    \\Lest S_k\n    &\\quad&\\by{\\cref{eq:condition_continue_k-1}}\\\\\n    &\\leq\n    \\prn*{\n      \\frac{7}{12} + \\frac{1}{4}\n    }\n    \\theta_{k-1}^2 (1 - \\theta_{k-1})^2\n    \\Lest S_k\n    &\\quad&\\by{$0 \\leq \\theta_{k-1} \\leq 1$}\\\\\n    &\\leq\n    \\frac{1}{4}\n    \\theta_{k-1} (1 - \\theta_{k-1})\n    \\Lest S_k\n    &\\quad&\\by{$0 \\leq \\theta_{k-1} \\leq 1$},\n  \\end{alignat}\n  which results in\n  \\begin{alignat}{2}\n    f(x_k) - f(x_0)\n    &\\leq\n    \\frac{\\theta_k^2 + \\theta_{k-1} - 2}{4} \\Lest S_k\n    + \\frac{1}{4} \\theta_{k-1} (1 - \\theta_{k-1}) \\Lest S_k\\\\\n    &=\n    - \\frac{1 - \\theta_k}{2} \\Lest S_k\n    + \\frac{(1 - \\theta_k)^2 - (1 - \\theta_{k-1})^2}{4} \\Lest S_k\\\\\n    &\\leq\n    - \\frac{1 - \\theta_k}{2} \\Lest S_k\n    &\\quad&\\by{\\cref{eq:theta_increase_not_too_fast}}\n  \\end{alignat}\n  and concludes the proof.\n\\end{proof}","start":61293,"end":64695}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":["eq:potential_lowerbound","eq:potential_obj_0","eq:sum_xk_diff_power34_upperbound"],"declared_external":false,"source_only":false,"title":"高次位移和与势函数端点","source_units":[{"raw_tex":"  Note that\n  \\begin{align}\n    \\sum_{i=1}^k \\norm*{x_i - x_{i-1}}^3\n    &\\leq\n    S_k^{3/2},\\quad\n    \\sum_{i=1}^k \\norm*{x_i - x_{i-1}}^4\n    \\leq\n    S_k^2\n    \\label{eq:sum_xk_diff_power34_upperbound}\n  \\end{align}\n  by the monotonicity of $\\ell_p$-norms, i.e., $\\norm{a}_p \\geq \\norm{a}_q$ for $1 \\leq p \\leq q$.\n  In addition, we have\n  \\begin{align}\n    \\Phi_k - f(x_k)\n    &=\n    \\frac{\\theta_k^2}{2} \\prn*{\n      \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_{k-1}) + L (x_k - x_{k-1})}^2\n      + \\frac{\\Lest}{2} \\norm*{x_k - x_{k-1}}^2\n    }\n    \\geq\n    0,\n    \\label{eq:potential_lowerbound}\\\\\n    \\Phi_0 - f(x_0)\n    &=\n    \\frac{\\theta_0^2}{4 \\Lest} \\norm*{\\nabla f(x_0)}^2\n    \\label{eq:potential_obj_0}\n  \\end{align}\n  since $x_{-1} = x_0$.\n","start":61343,"end":62090}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":["eq:potential_decrease_epoch"],"declared_external":false,"source_only":false,"title":"单步势函数界的望远镜求和","source_units":[{"raw_tex":"  Summing \\cref{lem:potential_decrease_iteration} over $k$ and telescoping yields\n  \\begin{align}\n    &\\mathInd\n    \\Phi_k - \\Phi_0\n    \\label{eq:potential_decrease_epoch}\\\\\n    &\\leq\n    \\sum_{i=0}^{k-1}\n    \\bigg(\n      \\frac{\\theta_{i+1}^2 + \\theta_i - 2}{4} \\Lest \\norm*{x_{i+1} - x_i}^2\n      + \\frac{7 \\theta_i^2}{12} \\Mest_i \\norm*{x_i - x_{i-1}}^3\\\\\n      &\\qquad\\qquad\n      + \\frac{\\theta_i^3}{4 \\Lest} \\Mest_i^2 \\norm*{x_i - x_{i-1}}^4\n      - \\frac{\\theta_i^2}{4 \\Lest} \\norm*{\\nabla f(x_i)}^2\n    \\bigg)\\\\\n    &\\leq\n    \\frac{\\theta_k^2 + \\theta_{k-1} - 2}{4} \\Lest S_k\n    + \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} S_{k-1}^{3/2} \n    + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}^2\n    - \\frac{\\theta_0^2}{4 \\Lest} \\norm*{\\nabla f(x_0)}^2,\n  \\end{align}\n  where we have used \\cref{eq:theta_increase_not_too_fast}, \\cref{eq:sum_xk_diff_power34_upperbound}, and $\\Mest_i \\leq \\Mest_{k-1}$.\n  Combining \\cref{eq:potential_lowerbound,eq:potential_obj_0,eq:potential_decrease_epoch} yields\n  \\begin{align}\n    f(x_k) - f(x_0)\n    \\leq\n    \\frac{\\theta_k^2 + \\theta_{k-1} - 2}{4} \\Lest S_k\n    + \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} S_{k-1}^{3/2} \n    + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}^2.\n  \\end{align}\n","start":62091,"end":63335}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"续行条件吸收高次误差","source_units":[{"raw_tex":"  Furthermore, we bound the last two terms as\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} S_{k-1}^{3/2} \n    + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}^2\\\\\n    &\\leq\n    \\prn*{\n      \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} \\sqrt{S_{k-1}}\n      + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}\n    }\n    S_k\n    &\\quad&\\by{$S_{k-1} \\leq S_k$}\\\\\n    &\\leq\n    \\prn*{\n      \\frac{7}{12} \\theta_{k-1}^2 (1 - \\theta_{k-1})^{5/2}\n      + \\frac{1}{4} \\theta_{k-1}^3 (1 - \\theta_{k-1})^5\n    }\n    \\Lest S_k\n    &\\quad&\\by{\\cref{eq:condition_continue_k-1}}\\\\\n    &\\leq\n    \\prn*{\n      \\frac{7}{12} + \\frac{1}{4}\n    }\n    \\theta_{k-1}^2 (1 - \\theta_{k-1})^2\n    \\Lest S_k\n    &\\quad&\\by{$0 \\leq \\theta_{k-1} \\leq 1$}\\\\\n    &\\leq\n    \\frac{1}{4}\n    \\theta_{k-1} (1 - \\theta_{k-1})\n    \\Lest S_k\n    &\\quad&\\by{$0 \\leq \\theta_{k-1} \\leq 1$},\n  \\end{alignat}\n  which results in\n  \\begin{alignat}{2}\n    f(x_k) - f(x_0)\n    &\\leq\n    \\frac{\\theta_k^2 + \\theta_{k-1} - 2}{4} \\Lest S_k\n    + \\frac{1}{4} \\theta_{k-1} (1 - \\theta_{k-1}) \\Lest S_k\\\\\n    &=\n    - \\frac{1 - \\theta_k}{2} \\Lest S_k\n    + \\frac{(1 - \\theta_k)^2 - (1 - \\theta_{k-1})^2}{4} \\Lest S_k\\\\\n    &\\leq\n    - \\frac{1 - \\theta_k}{2} \\Lest S_k\n    &\\quad&\\by{\\cref{eq:theta_increase_not_too_fast}}\n  \\end{alignat}\n  and concludes the proof.\n","start":63335,"end":64684}]},{"id":"S042","semantic_type":"proposition","kind":"proposition","labels":["prop:Mk_upperbound"],"declared_external":false,"source_only":false,"title":"Hessian 估计的统一上界","source_units":[{"raw_tex":"\\begin{proposition}\n  \\label{prop:Mk_upperbound}\n  Suppose that \\cref{asm:hessian_lip} holds.\n  Then, the following is true throughout \\cref{alg:proposed_agd}: \n  $\\Mest_k \\leq \\max\\set{M_0, \\Mtrue}$.\n\\end{proposition}","start":64788,"end":65006}]},{"id":"S043","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"Hessian 估计上界的作者论证","source_units":[{"raw_tex":"Because of the update rules \\cref{eq:update_Mest_practical} or \\cref{eq:update_Mest} of $M_k$, it suffices to show the inequalities obtained by replacing $\\Mest_k$ with $M_f$ in \\cref{eq:fy_fx_diff_upperbound_Mestk,eq:grad_interpolation_error_Mestk,eq:grad_ybark_upperbound_Mestk}.\nOne of them has already been obtained as \\cref{lem:trapezoidal_rule_error}.\nBelow we prove the other two; the following lemma gives the formal statement.\n","start":65007,"end":65443}]},{"id":"S044","semantic_type":"lemma","kind":"lemma","labels":["lem:grad_interpolation_error"],"declared_external":false,"source_only":false,"title":"真实常数下的梯度插值界","source_units":[{"raw_tex":"\\begin{lemma}\n  \\label{lem:grad_interpolation_error}\n  Under \\cref{asm:hessian_lip}, the following hold for all $k \\geq 1$:\n  \\begin{align}\n    &\n    \\norm*{\n      \\nabla f(y_k)\n      + \\theta_k \\nabla f(x_{k-1})\n      - (1 + \\theta_k) \\nabla f (x_k)\n    }\n    \\leq\n    \\theta_k \\Mtrue \\norm*{x_k - x_{k-1}}^2,\n    \\label{eq:grad_interpolation_error_Mf}\\\\\n","start":65443,"end":65799}]},{"id":"S045","semantic_type":"lemma","kind":"lemma","labels":["lem:grad_interpolation_error"],"declared_external":false,"source_only":false,"title":"真实常数下的平均梯度界","source_units":[{"raw_tex":"\\begin{lemma}\n  \\label{lem:grad_interpolation_error}\n  Under \\cref{asm:hessian_lip}, the following hold for all $k \\geq 1$:\n","start":65443,"end":65567},{"raw_tex":"    &\n    \\norm*{ \\nabla f (\\bar y_k) }\n    \\leq\n    \\frac{\\Lest}{Z_k} \\norm*{x_k - x_{k-1}}\n    + \\frac{(k-1) (k+5)^2}{16 Z_k^2} \\Mtrue S_k.\n    \\label{eq:grad_ybark_upperbound_Mf}\n  \\end{align}\n\\end{lemma}","start":65799,"end":66006}]},{"id":"S046","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"梯度插值界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{eq:grad_interpolation_error_Mf}]\n  Note that $x_k = \\frac{1}{1 + \\theta_k} y_k + \\frac{\\theta_k}{1 + \\theta_k} x_{k-1}$ from \\cref{eq:update_xy}, and \\cref{lem:gradient_jensen} gives\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{\n      \\nabla f (x_k)\n      - \\frac{1}{1 + \\theta_k} \\nabla f(y_k)\n      - \\frac{\\theta_k}{1 + \\theta_k} \\nabla f(x_{k-1})\n    }\\\\\n    &\\leq\n    \\frac{\\Mtrue \\theta_k}{2 (1 + \\theta_k)^2} \\norm*{y_k - x_{k-1}}^2\\\\\n    &=\n    \\frac{\\Mtrue \\theta_k}{2} \\norm*{x_k - x_{k-1}}^2\n    \\leq\n    \\frac{\\Mtrue \\theta_k}{1 + \\theta_k} \\norm*{x_k - x_{k-1}}^2\n    &\\quad&\\by{\\cref{eq:update_xy} and $0 \\leq \\theta_k \\leq 1$}.\n  \\end{alignat}\n  Multiplying by $(1 + \\theta_k)$ concludes the proof.\n\\end{proof}","start":66135,"end":66890}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"两点 Jensen 界与缩放","source_units":[{"raw_tex":"  Note that $x_k = \\frac{1}{1 + \\theta_k} y_k + \\frac{\\theta_k}{1 + \\theta_k} x_{k-1}$ from \\cref{eq:update_xy}, and \\cref{lem:gradient_jensen} gives\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{\n      \\nabla f (x_k)\n      - \\frac{1}{1 + \\theta_k} \\nabla f(y_k)\n      - \\frac{\\theta_k}{1 + \\theta_k} \\nabla f(x_{k-1})\n    }\\\\\n    &\\leq\n    \\frac{\\Mtrue \\theta_k}{2 (1 + \\theta_k)^2} \\norm*{y_k - x_{k-1}}^2\\\\\n    &=\n    \\frac{\\Mtrue \\theta_k}{2} \\norm*{x_k - x_{k-1}}^2\n    \\leq\n    \\frac{\\Mtrue \\theta_k}{1 + \\theta_k} \\norm*{x_k - x_{k-1}}^2\n    &\\quad&\\by{\\cref{eq:update_xy} and $0 \\leq \\theta_k \\leq 1$}.\n  \\end{alignat}\n  Multiplying by $(1 + \\theta_k)$ concludes the proof.\n","start":66197,"end":66879}]},{"id":"S048","semantic_type":"proof","kind":"proof","labels":["eq:norm_grad_ybark_upperbound_pre","eq:weighted_average_gradient_equation"],"declared_external":false,"source_only":false,"title":"平均梯度界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{eq:grad_ybark_upperbound_Mf}]\n  We have\n  \\begin{alignat}{2}\n    &\\mathInd\n    p_{k,i} \\nabla f(y_i)\n    =\n    p_{k,i} \\Lest (y_i - x_{i+1})\\\\\n    &=\n    p_{k,i} \\Lest \\prn*{\n      \\theta_i (x_i - x_{i-1})\n      - (x_{i+1} - x_i)\n    }\n    &\\quad&\\by{\\cref{eq:update_xy}}\\\\\n    &=\n    \\Lest \\prn*{\n      p_{k,i-1} (x_i - x_{i-1})\n      - p_{k,i} (x_{i+1} - x_i)\n    }\n    &\\quad&\\by{definition \\cref{eq:def_ybark} of $p_{k,i}$}.\n  \\end{alignat}\n  Summing this equality over $i$ yields\n  \\begin{align}\n    \\sum_{i=0}^{k-1}\n    p_{k,i} \\nabla f(y_i)\n    =\n    - \\Lest p_{k,k-1} (x_k - x_{k-1}),\n    \\label{eq:weighted_average_gradient_equation}\n  \\end{align}\n  where we have used $x_0 = x_{-1}$.\n  Now, we obtain\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{\n      \\nabla f (\\bar y_k)\n    }\n    \\label{eq:norm_grad_ybark_upperbound_pre}\\\\\n    &\\leq\n    \\norm*{\n      \\sum_{i=0}^{k-1} p_{k,i} \\nabla f \\prn*{ y_i }\n    }\n    + \\frac{\\Mtrue}{2} \\sum_{0 \\leq i < j < k} p_{k,i} p_{k,j} \\norm*{y_i - y_j}^2\n    &\\quad&\\by{\\cref{lem:gradient_jensen}}\\\\\n    &=\n    \\Lest p_{k,k-1} \\norm*{x_k - x_{k-1}}\n    + \\frac{\\Mtrue}{2} \\sum_{0 \\leq i < j < k} p_{k,i} p_{k,j} \\norm*{y_i - y_j}^2\n    &\\quad&\\by{\\cref{eq:weighted_average_gradient_equation}}\\\\\n    &\\leq\n    \\frac{\\Lest}{Z_k} \\norm*{x_k - x_{k-1}}\n    + \\frac{\\Mtrue}{2 Z_k^2} \\sum_{0 \\leq i < j < k} \\norm*{y_i - y_j}^2,\n  \\end{alignat}\n  where the last inequality follows from $p_{k,i} \\leq p_{k,k-1} = 1 / Z_k$ for all $0 \\leq i < k$.\n\n  Next, we bound the second term on the right-hand side.\n  Since for $0 \\leq i < j < k$,\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{y_i - y_j}\\\\\n    &\\leq\n    \\norm*{y_i - x_i}\n    + \\sum_{l=i+1}^{j-1} \\norm*{x_l - x_{l-1}}\n    + \\norm*{y_j - x_{j-1}}\n    &\\quad&\\by{the triangle inequality}\\\\\n    &=\n    \\norm*{x_i - x_{i-1}}\n    + \\sum_{l=i+1}^{j-1} \\norm*{x_l - x_{l-1}}\n    + 2 \\norm*{x_j - x_{j-1}}\n    &\\quad&\\by{\\cref{eq:update_xy} and $0 \\leq \\theta_k \\leq 1$}\\\\\n    &\\leq\n    \\prn*{\n      1^2 + \\sum_{l=i+1}^{j-1} 1^2 + 2^2\n    }^{1/2}\n    \\prn*{\n      \\sum_{l=i}^j \\norm*{x_l - x_{l-1}}^2\n    }^{1/2}\n    &\\quad&\\by{Cauchy--Schwarz}\\\\\n    &=\n    \\sqrt{j - i + 4}\n    \\prn*{\n      \\sum_{l=i}^j \\norm*{x_l - x_{l-1}}^2\n    }^{1/2},\n  \\end{alignat}\n  we have\n  \\begin{align}\n    \\sum_{0 \\leq i < j < k}\n    \\norm*{y_i - y_j}^2\n    &\\leq\n    \\sum_{0 \\leq i < j < k}\n    \\sum_{l=i}^j\n    (j - i + 4) \\norm*{x_l - x_{l-1}}^2\\\\\n    % &=\n    % \\sum_{0 \\leq i \\leq l \\leq j < k}\n    % (j - i + 4) \\norm*{x_l - x_{l-1}}^2\n    % - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &=\n    \\sum_{l=0}^{k-1}\n    \\prn*{\n      \\sum_{i=0}^l\n      \\sum_{j=l}^{k-1}\n      (j - i + 4) \n    }\n    \\norm*{x_l - x_{l-1}}^2\n    - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &=\n    \\frac{k + 7}{2}\n    \\sum_{l=0}^{k-1}\n    (l + 1) (k - l)\n    \\norm*{x_l - x_{l-1}}^2\n    - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &\\leq\n    \\frac{k + 7}{2}\n    \\sum_{l=0}^{k-1} \\frac{(k+1)^2}{4} \\norm*{x_l - x_{l-1}}^2\n    - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &=\n    \\frac{(k-1) (k+5)^2}{8}\n    \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\n    \\leq\n    \\frac{(k-1) (k+5)^2}{8} S_k.\n  \\end{align}\n  Plugging this bound into \\cref{eq:norm_grad_ybark_upperbound_pre} concludes the proof.\n\\end{proof}","start":66891,"end":70197}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["eq:weighted_average_gradient_equation"],"declared_external":false,"source_only":false,"title":"加权梯度的望远镜恒等式","source_units":[{"raw_tex":"  We have\n  \\begin{alignat}{2}\n    &\\mathInd\n    p_{k,i} \\nabla f(y_i)\n    =\n    p_{k,i} \\Lest (y_i - x_{i+1})\\\\\n    &=\n    p_{k,i} \\Lest \\prn*{\n      \\theta_i (x_i - x_{i-1})\n      - (x_{i+1} - x_i)\n    }\n    &\\quad&\\by{\\cref{eq:update_xy}}\\\\\n    &=\n    \\Lest \\prn*{\n      p_{k,i-1} (x_i - x_{i-1})\n      - p_{k,i} (x_{i+1} - x_i)\n    }\n    &\\quad&\\by{definition \\cref{eq:def_ybark} of $p_{k,i}$}.\n  \\end{alignat}\n  Summing this equality over $i$ yields\n  \\begin{align}\n    \\sum_{i=0}^{k-1}\n    p_{k,i} \\nabla f(y_i)\n    =\n    - \\Lest p_{k,k-1} (x_k - x_{k-1}),\n    \\label{eq:weighted_average_gradient_equation}\n  \\end{align}\n  where we have used $x_0 = x_{-1}$.\n","start":66950,"end":67614}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":["eq:norm_grad_ybark_upperbound_pre"],"declared_external":false,"source_only":false,"title":"平均梯度与点间方差","source_units":[{"raw_tex":"  Now, we obtain\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{\n      \\nabla f (\\bar y_k)\n    }\n    \\label{eq:norm_grad_ybark_upperbound_pre}\\\\\n    &\\leq\n    \\norm*{\n      \\sum_{i=0}^{k-1} p_{k,i} \\nabla f \\prn*{ y_i }\n    }\n    + \\frac{\\Mtrue}{2} \\sum_{0 \\leq i < j < k} p_{k,i} p_{k,j} \\norm*{y_i - y_j}^2\n    &\\quad&\\by{\\cref{lem:gradient_jensen}}\\\\\n    &=\n    \\Lest p_{k,k-1} \\norm*{x_k - x_{k-1}}\n    + \\frac{\\Mtrue}{2} \\sum_{0 \\leq i < j < k} p_{k,i} p_{k,j} \\norm*{y_i - y_j}^2\n    &\\quad&\\by{\\cref{eq:weighted_average_gradient_equation}}\\\\\n    &\\leq\n    \\frac{\\Lest}{Z_k} \\norm*{x_k - x_{k-1}}\n    + \\frac{\\Mtrue}{2 Z_k^2} \\sum_{0 \\leq i < j < k} \\norm*{y_i - y_j}^2,\n  \\end{alignat}\n  where the last inequality follows from $p_{k,i} \\leq p_{k,k-1} = 1 / Z_k$ for all $0 \\leq i < k$.\n","start":67614,"end":68406}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"点间距离的位移控制","source_units":[{"raw_tex":"  Next, we bound the second term on the right-hand side.\n  Since for $0 \\leq i < j < k$,\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{y_i - y_j}\\\\\n    &\\leq\n    \\norm*{y_i - x_i}\n    + \\sum_{l=i+1}^{j-1} \\norm*{x_l - x_{l-1}}\n    + \\norm*{y_j - x_{j-1}}\n    &\\quad&\\by{the triangle inequality}\\\\\n    &=\n    \\norm*{x_i - x_{i-1}}\n    + \\sum_{l=i+1}^{j-1} \\norm*{x_l - x_{l-1}}\n    + 2 \\norm*{x_j - x_{j-1}}\n    &\\quad&\\by{\\cref{eq:update_xy} and $0 \\leq \\theta_k \\leq 1$}\\\\\n    &\\leq\n    \\prn*{\n      1^2 + \\sum_{l=i+1}^{j-1} 1^2 + 2^2\n    }^{1/2}\n    \\prn*{\n      \\sum_{l=i}^j \\norm*{x_l - x_{l-1}}^2\n    }^{1/2}\n    &\\quad&\\by{Cauchy--Schwarz}\\\\\n    &=\n    \\sqrt{j - i + 4}\n    \\prn*{\n      \\sum_{l=i}^j \\norm*{x_l - x_{l-1}}^2\n    }^{1/2},\n  \\end{alignat}\n","start":68407,"end":69166}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"距离和的组合计数","source_units":[{"raw_tex":"  we have\n  \\begin{align}\n    \\sum_{0 \\leq i < j < k}\n    \\norm*{y_i - y_j}^2\n    &\\leq\n    \\sum_{0 \\leq i < j < k}\n    \\sum_{l=i}^j\n    (j - i + 4) \\norm*{x_l - x_{l-1}}^2\\\\\n    % &=\n    % \\sum_{0 \\leq i \\leq l \\leq j < k}\n    % (j - i + 4) \\norm*{x_l - x_{l-1}}^2\n    % - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &=\n    \\sum_{l=0}^{k-1}\n    \\prn*{\n      \\sum_{i=0}^l\n      \\sum_{j=l}^{k-1}\n      (j - i + 4) \n    }\n    \\norm*{x_l - x_{l-1}}^2\n    - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &=\n    \\frac{k + 7}{2}\n    \\sum_{l=0}^{k-1}\n    (l + 1) (k - l)\n    \\norm*{x_l - x_{l-1}}^2\n    - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &\\leq\n    \\frac{k + 7}{2}\n    \\sum_{l=0}^{k-1} \\frac{(k+1)^2}{4} \\norm*{x_l - x_{l-1}}^2\n    - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &=\n    \\frac{(k-1) (k+5)^2}{8}\n    \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\n    \\leq\n    \\frac{(k-1) (k+5)^2}{8} S_k.\n  \\end{align}\n  Plugging this bound into \\cref{eq:norm_grad_ybark_upperbound_pre} concludes the proof.\n","start":69166,"end":70186}]},{"id":"S053","semantic_type":"lemma","kind":"lemma","labels":["eq:grad_norm_ybar_upperbound","lem:grad_norm_ybar_upperbound"],"declared_external":false,"source_only":false,"title":"一般规则的最小梯度界","source_units":[{"raw_tex":"\\begin{lemma}\n  \\label{lem:grad_norm_ybar_upperbound}\n  Suppose that \\cref{asm:gradient_hessian_lip} holds.\n  In \\cref{alg:proposed_agd}, the following is true when $k \\geq 2$:\n  \\begin{align}\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    \\leq\n    \\frac{4 \\Lmax^2 \\Mmax}{M_0^2 k^4}.\n    \\label{eq:grad_norm_ybar_upperbound}\n  \\end{align}\n","start":70703,"end":71132}]},{"id":"S054","semantic_type":"lemma","kind":"lemma","labels":["eq:grad_norm_ybar_upperbound_tight","lem:grad_norm_ybar_upperbound"],"declared_external":false,"source_only":false,"title":"强化规则的最小梯度界","source_units":[{"raw_tex":"\\begin{lemma}\n  \\label{lem:grad_norm_ybar_upperbound}\n  Suppose that \\cref{asm:gradient_hessian_lip} holds.\n  In \\cref{alg:proposed_agd}, the following is true when $k \\geq 2$:\n","start":70703,"end":70880},{"raw_tex":"  If $M_k$ is computed with \\cref{eq:update_Mest} in the algorithm, the above bound is improved to\n  \\begin{align}\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\leq\n    4 \\Lest \\sqrt{\\frac{S_{k-1}}{k^3}}\n    \\leq\n    \\frac{4 \\Lmax^2}{M_0 k^4}.\n    \\label{eq:grad_norm_ybar_upperbound_tight}\n  \\end{align}\n\\end{lemma}","start":71132,"end":71462}]},{"id":"S055","semantic_type":"proof","kind":"proof","labels":["eq:Zi_squared_sum","eq:i_cubic_sum"],"declared_external":false,"source_only":false,"title":"一般梯度界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{eq:grad_norm_ybar_upperbound}] \n  Note that\n  \\begin{align}\n    &\\sum_{i=1}^{k-1}\n    \\frac{(i-1)(i+5)^2}{16}\n    =\n    \\frac{(k - 2) (k - 1) (k^2 + 13 k + 50)}{64}\n    \\leq\n    \\frac{k^3 (k+1)}{24},\n    \\label{eq:i_cubic_sum}\\\\\n    &\\sum_{i=1}^{k-1} Z_i^2\n    =\n    \\sum_{i=1}^{k-1} \\prn*{\\frac{i+1}{2}}^2\n    =\n    \\frac{(k-1) (2 k^2 + 5 k + 6)}{24}\n    \\geq \n    \\frac{k^2 (k+1)}{12}\n    \\label{eq:Zi_squared_sum}\n  \\end{align}\n  from \\cref{eq:ybar_Zk_alternative} and $k \\geq 2$.\n  Using \\cref{eq:grad_ybark_upperbound_Mf}, we have\n  \\begin{align}\n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\leq\n    \\sum_{i=1}^{k-1}\n    Z_i^2\n    \\norm*{ \\nabla f (\\bar y_i) }\\\\\n    &\\leq\n    \\sum_{i=1}^{k-1}\n    \\prn*{\n      \\Lest Z_i \\norm*{x_i - x_{i-1}}\n      + \\frac{(i-1) (i+5)^2}{16} \\Mtrue S_i\n    }.\n  \\end{align}\n  Each term on the right-hand side can be bounded as follows:\n  \\begin{alignat}{2}\n    &\\sum_{i=1}^{k-1} \\Lest Z_i \\norm*{x_i - x_{i-1}}\n    \\leq\n    \\Lest\n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }^{1/2}\n    \\sqrt{S_{k-1}}\n    &\\quad&\\by{Cauchy--Schwarz},\\\\\n    &\\sum_{i=1}^{k-1}\n    \\frac{(i-1)(i+5)^2}{16} \\Mtrue S_i\n    \\leq\n    \\frac{k^3 (k+1)}{24}\n    \\Mtrue S_{k-1}\n    &\\quad&\\by{\\cref{eq:i_cubic_sum}}.\n  \\end{alignat}\n  We thus have\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\\\\\n    &\\leq\n    \\Lest \n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }^{-1/2}\\!\\!\n    \\sqrt{S_{k-1}}\n    + \n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }^{-1}\n    \\frac{k^3 (k+1)}{24}\n    \\Mtrue S_{k-1}\\\\\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{k}{2}\n    \\Mtrue S_{k-1}\n    &\\quad&\\by{\\cref{eq:Zi_squared_sum}}\\\\\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{\\Lest \\Mtrue}{2 \\Mest_k} \n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{\\cref{eq:condition_continue_k-1}}\\\\\n    &\\leq\n    \\frac{\\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{\\Lest \\Mmax}{2 \\Mest_{k-1}} \n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{$\\Mest_{k-1}, \\Mtrue \\leq \\Mmax$}\\\\\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}},\n  \\end{alignat}\n  which is the first inequality of the desired result~\\cref{eq:grad_norm_ybar_upperbound}.\n  The second inequality follows from \\cref{eq:condition_continue_k-1}:\n  \\begin{align}\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    \\leq\n    \\frac{4 \\Lest^2 \\Mmax}{M_{k-1}^2 k^4}\n    \\leq\n    \\frac{4 \\Lmax^2 \\Mmax}{M_0 k^4}.\n  \\end{align}\n\\end{proof}","start":71463,"end":74094}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":["eq:Zi_squared_sum","eq:i_cubic_sum"],"declared_external":false,"source_only":false,"title":"归一化平方和与误差系数和","source_units":[{"raw_tex":"  Note that\n  \\begin{align}\n    &\\sum_{i=1}^{k-1}\n    \\frac{(i-1)(i+5)^2}{16}\n    =\n    \\frac{(k - 2) (k - 1) (k^2 + 13 k + 50)}{64}\n    \\leq\n    \\frac{k^3 (k+1)}{24},\n    \\label{eq:i_cubic_sum}\\\\\n    &\\sum_{i=1}^{k-1} Z_i^2\n    =\n    \\sum_{i=1}^{k-1} \\prn*{\\frac{i+1}{2}}^2\n    =\n    \\frac{(k-1) (2 k^2 + 5 k + 6)}{24}\n    \\geq \n    \\frac{k^2 (k+1)}{12}\n    \\label{eq:Zi_squared_sum}\n  \\end{align}\n  from \\cref{eq:ybar_Zk_alternative} and $k \\geq 2$.\n","start":71524,"end":71976}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"累加平均梯度上界","source_units":[{"raw_tex":"  Using \\cref{eq:grad_ybark_upperbound_Mf}, we have\n  \\begin{align}\n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\leq\n    \\sum_{i=1}^{k-1}\n    Z_i^2\n    \\norm*{ \\nabla f (\\bar y_i) }\\\\\n    &\\leq\n    \\sum_{i=1}^{k-1}\n    \\prn*{\n      \\Lest Z_i \\norm*{x_i - x_{i-1}}\n      + \\frac{(i-1) (i+5)^2}{16} \\Mtrue S_i\n    }.\n  \\end{align}\n  Each term on the right-hand side can be bounded as follows:\n  \\begin{alignat}{2}\n    &\\sum_{i=1}^{k-1} \\Lest Z_i \\norm*{x_i - x_{i-1}}\n    \\leq\n    \\Lest\n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }^{1/2}\n    \\sqrt{S_{k-1}}\n    &\\quad&\\by{Cauchy--Schwarz},\\\\\n    &\\sum_{i=1}^{k-1}\n    \\frac{(i-1)(i+5)^2}{16} \\Mtrue S_i\n    \\leq\n    \\frac{k^3 (k+1)}{24}\n    \\Mtrue S_{k-1}\n    &\\quad&\\by{\\cref{eq:i_cubic_sum}}.\n  \\end{alignat}\n","start":71976,"end":72797}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"续行条件转化为最小梯度界","source_units":[{"raw_tex":"  We thus have\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\\\\\n    &\\leq\n    \\Lest \n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }^{-1/2}\\!\\!\n    \\sqrt{S_{k-1}}\n    + \n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }^{-1}\n    \\frac{k^3 (k+1)}{24}\n    \\Mtrue S_{k-1}\\\\\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{k}{2}\n    \\Mtrue S_{k-1}\n    &\\quad&\\by{\\cref{eq:Zi_squared_sum}}\\\\\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{\\Lest \\Mtrue}{2 \\Mest_k} \n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{\\cref{eq:condition_continue_k-1}}\\\\\n    &\\leq\n    \\frac{\\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{\\Lest \\Mmax}{2 \\Mest_{k-1}} \n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{$\\Mest_{k-1}, \\Mtrue \\leq \\Mmax$}\\\\\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}},\n  \\end{alignat}\n  which is the first inequality of the desired result~\\cref{eq:grad_norm_ybar_upperbound}.\n","start":72797,"end":73816}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"一般梯度界的四次衰减尾界","source_units":[{"raw_tex":"  The second inequality follows from \\cref{eq:condition_continue_k-1}:\n  \\begin{align}\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    \\leq\n    \\frac{4 \\Lest^2 \\Mmax}{M_{k-1}^2 k^4}\n    \\leq\n    \\frac{4 \\Lmax^2 \\Mmax}{M_0 k^4}.\n  \\end{align}\n","start":73816,"end":74083}]},{"id":"S060","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"强化梯度界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{eq:grad_norm_ybar_upperbound_tight}]\n  When $M_k$ is computed with \\cref{eq:update_Mest}, we have \\cref{eq:grad_ybark_upperbound_Mestk}.\n  Similarly to the proof of \\cref{eq:grad_norm_ybar_upperbound}, using \\cref{eq:grad_ybark_upperbound_Mestk} instead of \\cref{eq:grad_ybark_upperbound_Mf} yields\n  \\begin{alignat}{2}\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{k}{2} \\Mest_{k-1} S_{k-1}.\n  \\end{alignat}\n  The right-hand side is bounded as\n  \\begin{alignat}{2}\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{k}{2} \\Mest_{k-1} S_{k-1}\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{\\Lest}{2} \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{\\cref{eq:condition_continue_k-1}}\\\\\n    &\\leq\n    4 \\Lest\n    \\sqrt{\\frac{S_{k-1}}{k^3}},\n  \\end{alignat}\n  which implies the first inequality in \\cref{eq:grad_norm_ybar_upperbound_tight}.\n  The second inequality is obtained similarly to the second inequality of \\cref{eq:grad_norm_ybar_upperbound}.\n\\end{proof}","start":74095,"end":75190}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"强化规则替换真实常数","source_units":[{"raw_tex":"  When $M_k$ is computed with \\cref{eq:update_Mest}, we have \\cref{eq:grad_ybark_upperbound_Mestk}.\n  Similarly to the proof of \\cref{eq:grad_norm_ybar_upperbound}, using \\cref{eq:grad_ybark_upperbound_Mestk} instead of \\cref{eq:grad_ybark_upperbound_Mf} yields\n  \\begin{alignat}{2}\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{k}{2} \\Mest_{k-1} S_{k-1}.\n  \\end{alignat}\n","start":74161,"end":74612}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"消去 Hessian 估计因子","source_units":[{"raw_tex":"  The right-hand side is bounded as\n  \\begin{alignat}{2}\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{k}{2} \\Mest_{k-1} S_{k-1}\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{\\Lest}{2} \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{\\cref{eq:condition_continue_k-1}}\\\\\n    &\\leq\n    4 \\Lest\n    \\sqrt{\\frac{S_{k-1}}{k^3}},\n  \\end{alignat}\n  which implies the first inequality in \\cref{eq:grad_norm_ybar_upperbound_tight}.\n  The second inequality is obtained similarly to the second inequality of \\cref{eq:grad_norm_ybar_upperbound}.\n","start":74612,"end":75179}]},{"id":"S063","semantic_type":"theorem","kind":"theorem","labels":["eq:complexity_bound","eq:complexity_bound_tight","eq:def_c1_c2","thm:complexity"],"declared_external":false,"source_only":false,"title":"无参数 AGD 的两种复杂度界","source_units":[{"raw_tex":"\\begin{theorem}\n  \\label{thm:complexity}\n  Suppose that \\cref{asm:gradient_hessian_lip} holds and let $\\Lmax \\coloneqq \\max\\set{\\Linit, \\alpha \\Ltrue}$, $\\Mmax \\coloneqq \\max\\set{M_0, \\Mtrue}$, $\\Delta \\coloneqq f(x_\\mathrm{init}) - \\inf_{x \\in \\R^d} f(x)$,\n  \\begin{align}\n    c_1\n    \\coloneqq\n    \\log_\\alpha \\prn*{\\frac{1}{\\beta}},\n    \\quad\\text{and}\\quad\n    c_2\n    \\coloneqq\n    \\log_\\alpha \\prn*{\\frac{\\Lmax}{\\Linit}}.\n    \\label{eq:def_c1_c2}\n  \\end{align}\n  In \\cref{alg:proposed_agd}, when $\\bar y_k$ defined by \\cref{eq:def_ybark} satisfies $\\norm*{\\nabla f(\\bar y_k)} \\leq \\epsilon$ for the first time, the total iteration count $K$ is at most\n  \\begin{align}\n    \\prn[\\bigg]{\n      70\n      + \n      108 c_1 \\frac{\\Mmax^{1/4}}{M_0^{1/4}}\n    }\n    \\frac{\\Lmax^{1/2} \\Mmax^2 \\Delta}{\\epsilon^{7/4} M_0^{7/4}}\n    + 2 (c_2 + 1) \\frac{\\Lmax^{1/2} \\Mmax^{1/4}}{\\epsilon^{1/4} M_0^{1/2}}.\n    \\label{eq:complexity_bound}\n  \\end{align}\n  If $M_k$ is computed with \\cref{eq:update_Mest} in the algorithm and $\\beta$ is set as $\\beta = 1$, the above bound is improved to\n  \\begin{align}\n    70\n    \\frac{\\Lmax^{1/2} \\Mmax^{1/4} \\Delta}{\\epsilon^{7/4}}\n    +\n    2 (c_2 + 1) \\frac{\\Lmax^{1/2}}{M_0^{1/4} \\epsilon^{1/4}}.\n    \\label{eq:complexity_bound_tight}\n  \\end{align}\n\\end{theorem}","start":75432,"end":76724}]},{"id":"S064","semantic_type":"definition","kind":"definition","labels":["eq:def_c1_c2","thm:complexity"],"declared_external":false,"source_only":false,"title":"复杂度常数与目标差","source_units":[{"raw_tex":"  Suppose that \\cref{asm:gradient_hessian_lip} holds and let $\\Lmax \\coloneqq \\max\\set{\\Linit, \\alpha \\Ltrue}$, $\\Mmax \\coloneqq \\max\\set{M_0, \\Mtrue}$, $\\Delta \\coloneqq f(x_\\mathrm{init}) - \\inf_{x \\in \\R^d} f(x)$,\n  \\begin{align}\n    c_1\n    \\coloneqq\n    \\log_\\alpha \\prn*{\\frac{1}{\\beta}},\n    \\quad\\text{and}\\quad\n    c_2\n    \\coloneqq\n    \\log_\\alpha \\prn*{\\frac{\\Lmax}{\\Linit}}.\n    \\label{eq:def_c1_c2}\n  \\end{align}\n","start":75473,"end":75899}]},{"id":"S065","semantic_type":"proof","kind":"proof","labels":["eq:Ksuc_upperbound","eq:Nsuc_upperbound","eq:Nunsuc_upperbound","eq:Sk_lowerbound","eq:restart_condition_k"],"declared_external":false,"source_only":false,"title":"一般复杂度界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of bound \\cref{eq:complexity_bound}]\n  We count the number of iterations of \\cref{alg:proposed_agd} separately for three types of epoch:\n  \\begin{itemize}\n    \\item \n    successful epoch: an epoch that does not find an $\\epsilon$-stationary point and ends at Line \\ref{alg-line-agd:restart_successful} with the descent condition~\\cref{eq:descent_condition} satisfied,\n    \\item\n    unsuccessful epoch: an epoch that does not find an $\\epsilon$-stationary point and ends at Line \\ref{alg-line-agd:restart_unsuccessful} with the descent condition~\\cref{eq:descent_condition} unsatisfied,\n    \\item\n    last epoch: the epoch that finds an $\\epsilon$-stationary point.\n  \\end{itemize}\n\n  \\paragraph{Successful epochs}\n  Let us focus on a successful epoch.\n  Let $k$ be the iteration number of the epoch.\n  It follows that \n  \\begin{align}\n    (k+1)^5\n    \\Mest_k^2 S_k\n    &>\n    \\Lest^2.\n    \\label{eq:restart_condition_k}\n  \\end{align}\n  When $k \\geq 2$, we have\n  \\begin{alignat}{2}\n    \\epsilon\n    &<\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\quad&\\text{(since this epoch does not find an $\\epsilon$-stationary point)}\\\\\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{\\cref{eq:grad_norm_ybar_upperbound}}\\\\\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_0} \\sqrt{\\frac{S_k}{k^3}}\n    &\\quad&\\by{$M_{k-1} \\geq M_0$ and $S_k \\geq S_{k-1}$},\n  \\end{alignat}\n  and hence \n  \\begin{align}\n    S_k > \\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3.\n    \\label{eq:Sk_lowerbound}\n  \\end{align}\n  When $k = 1$, we also have \\cref{eq:Sk_lowerbound}, as follows:\n  \\begin{align}\n    S_k\n    =\n    \\norm*{x_1 - x_0}^2\n    =\n    \\frac{1}{\\Lest^2} \\norm*{\\nabla f(x_0)}^2\n    =\n    \\frac{1}{\\Lest^2} \\norm*{\\nabla f(\\bar y_1)}^2\n    >\n    \\frac{\\epsilon^2}{\\Lest^2}\n    \\geq\n    \\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3.\n  \\end{align}\n  Combining \\cref{eq:Sk_lowerbound,eq:restart_condition_k} yields\n  \\begin{align}\n    S_k\n    =\n    S_k^{7/8}\n    S_k^{1/8}\n    &\\geq\n    \\prn[\\bigg]{\\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3}^{7/8}\n    \\prn*{\\frac{\\Lest^2}{\\Mest_k^2 (k+1)^5}}^{1/8}\\\\\n    &=\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{3/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^{7/4} \\Mest_k^{1/4}}\n    \\frac{k^{21/8}}{(k+1)^{5/8}}\n    \\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{3/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^2}\n    \\frac{k^{21/8}}{(k+1)^{5/8}}.\n  \\end{align}\n  Plugging this bound into condition \\cref{eq:descent_condition}, we obtain\n  \\begin{align}\n    f(x_0) - f(x_k)\n    \\geq\n    \\frac{\\Lest S_k}{2 (k+1)}\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{1/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^2}\n    \\prn*{\\frac{k}{k+1}}^{13/8}\n    \\frac{k}{2}\\\\\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lmax^{1/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^2}\n    \\prn*{\\frac{1}{2}}^{13/8}\n    \\frac{k}{2}\n    \\geq\n    \\frac{\\epsilon^{7/4} M_0^{7/4}}{\\Lmax^{1/2} \\Mmax^2}\n    \\frac{k}{70},\n  \\end{align}\n  and thus\n  \\begin{align}\n    k\n    \\leq\n    70\n    \\frac{\\Lmax^{1/2} \\Mmax^2}{\\epsilon^{7/4} M_0^{7/4}}\n    \\prn*{f(x_0) - f(x_k)}.\n  \\end{align}\n  Summing this bound over all successful epochs leads to the conclusion that the total iteration number of successful epochs is at most\n  \\begin{align}\n    70 \\frac{\\Lmax^{1/2} \\Mmax^2 \\Delta}{\\epsilon^{7/4} M_0^{7/4}}.\n    \\label{eq:Ksuc_upperbound}\n  \\end{align}\n  For later use, we will also evaluate the number of successful epochs.\n  Combining \\cref{eq:Sk_lowerbound,eq:restart_condition_k} also yields\n  \\begin{align}\n    S_k\n    =\n    S_k^{3/4}\n    S_k^{1/4}\n    &\\geq\n    \\prn[\\bigg]{ \\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3 }^{3/4}\n    \\prn*{\\frac{\\Lest^2}{\\Mest_k^2 (k+1)^5}}^{1/4}\\\\\n    &=\n    \\frac{(\\epsilon / 4)^{3/2}}{\\Lest}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^{3/2} \\Mest_k^{1/2}}\n    \\frac{k^{9/4}}{(k+1)^{5/4}}\n    \\geq\n    \\frac{(\\epsilon / 4)^{3/2}}{\\Lest}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^2}\n    \\frac{k^{9/4}}{(k+1)^{5/4}}.\n  \\end{align}\n  Plugging this bound into condition \\cref{eq:descent_condition} gives\n  \\begin{align}\n    f(x_0) - f(x_k)\n    \\geq\n    \\frac{\\Lest S_k}{2 (k+1)}\n    &\\geq\n    \\frac{(\\epsilon / 4)^{3/2}}{2}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^2}\n    \\prn*{\\frac{k}{k+1}}^{9/4}\\\\\n    &\\geq\n    \\frac{(\\epsilon / 4)^{3/2}}{2}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^2}\n    \\prn*{\\frac{1}{2}}^{9/4}\n    =\n    \\frac{\\epsilon^{3/2} \\Mest_0^{3/2}}{2^{25/4} \\Mmax^2}.\n  \\end{align}\n  Thus, we can deduce that the number $N_{\\mathrm{suc}}$ of successful epochs satisfies\n  \\begin{align}\n    N_{\\mathrm{suc}}\n    \\leq\n    2^{25/4}\n    \\frac{\\Mmax^2 \\Delta}{\\epsilon^{3/2} \\Mest_0^{3/2}}.\n    \\label{eq:Nsuc_upperbound}\n  \\end{align}\n\n  \\paragraph{Other epochs}\n  We deduce from \\cref{eq:grad_norm_ybar_upperbound} that the number of iterations for each epoch is at most\n  \\begin{align}\n    \\prn*{\n      \\frac{4 \\Lmax^2 \\Mmax}{\\epsilon M_0^2}\n    }^{1/4}.\n  \\end{align}\n  Let $N_{\\mathrm{unsuc}}$ be the number of unsuccessful epochs.\n  We then have\n  \\begin{align}\n    \\Linit\n    \\beta^{N_{\\mathrm{suc}}}\n    \\alpha^{N_{\\mathrm{unsuc}}}\n    \\leq\n    \\Lmax,\n  \\end{align}\n  and hence\n  \\begin{align}\n    N_{\\mathrm{unsuc}}\n    \\leq\n    c_1 N_{\\mathrm{suc}} + c_2,\n    \\label{eq:Nunsuc_upperbound}\n  \\end{align}\n  where $c_1$ and $c_2$ are defined by \\cref{eq:def_c1_c2}.\n  The total iteration number of unsuccessful and last epochs is at most\n  \\begin{alignat}{2}\n    &\\mathInd\n    (N_{\\mathrm{unsuc}} + 1)\n    \\prn*{ \\frac{4 \\Lmax^2 \\Mmax}{\\epsilon M_0^2} }^{1/4}\\\\\n    &\\leq\n    \\prn*{ c_1 N_{\\mathrm{suc}} + c_2 + 1 }\n    \\prn*{ \\frac{4 \\Lmax^2 \\Mmax}{\\epsilon M_0^2} }^{1/4}\n    &\\quad&\\by{\\cref{eq:Nunsuc_upperbound}}\\\\\n    &\\leq\n    2^{27/4} c_1\n    \\frac{\\Lmax^{1/2} \\Mmax^{9/4} \\Delta}{\\epsilon^{7/4} \\Mest_0^2}\n    + 2^{1/2} (c_2 + 1) \\frac{\\Lmax^{1/2} \\Mmax^{1/4}}{\\epsilon^{1/4} M_0^{1/2}}\n    &\\quad&\\by{\\cref{eq:Nsuc_upperbound}}\\\\\n    &\\leq\n    108 c_1\n    \\frac{\\Lmax^{1/2} \\Mmax^{9/4} \\Delta}{\\epsilon^{7/4} \\Mest_0^2}\n    + 2 (c_2 + 1) \\frac{\\Lmax^{1/2} \\Mmax^{1/4}}{\\epsilon^{1/4} M_0^{1/2}}.\n  \\end{alignat}\n  Putting this bound together with \\cref{eq:Ksuc_upperbound} concludes the proof.\n\\end{proof}","start":76725,"end":82876}]},{"id":"S066","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"按终止原因划分 epoch","source_units":[{"raw_tex":"  We count the number of iterations of \\cref{alg:proposed_agd} separately for three types of epoch:\n  \\begin{itemize}\n    \\item \n    successful epoch: an epoch that does not find an $\\epsilon$-stationary point and ends at Line \\ref{alg-line-agd:restart_successful} with the descent condition~\\cref{eq:descent_condition} satisfied,\n    \\item\n    unsuccessful epoch: an epoch that does not find an $\\epsilon$-stationary point and ends at Line \\ref{alg-line-agd:restart_unsuccessful} with the descent condition~\\cref{eq:descent_condition} unsatisfied,\n    \\item\n    last epoch: the epoch that finds an $\\epsilon$-stationary point.\n  \\end{itemize}\n","start":76782,"end":77426}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":["eq:restart_condition_k"],"declared_external":false,"source_only":false,"title":"成功重启给出的位移下界","source_units":[{"raw_tex":"  \\paragraph{Successful epochs}\n  Let us focus on a successful epoch.\n  Let $k$ be the iteration number of the epoch.\n  It follows that \n  \\begin{align}\n    (k+1)^5\n    \\Mest_k^2 S_k\n    &>\n    \\Lest^2.\n    \\label{eq:restart_condition_k}\n  \\end{align}\n","start":77427,"end":77679}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":["eq:Sk_lowerbound"],"declared_external":false,"source_only":false,"title":"未达精度给出的位移下界","source_units":[{"raw_tex":"  When $k \\geq 2$, we have\n  \\begin{alignat}{2}\n    \\epsilon\n    &<\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\quad&\\text{(since this epoch does not find an $\\epsilon$-stationary point)}\\\\\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{\\cref{eq:grad_norm_ybar_upperbound}}\\\\\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_0} \\sqrt{\\frac{S_k}{k^3}}\n    &\\quad&\\by{$M_{k-1} \\geq M_0$ and $S_k \\geq S_{k-1}$},\n  \\end{alignat}\n  and hence \n  \\begin{align}\n    S_k > \\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3.\n    \\label{eq:Sk_lowerbound}\n  \\end{align}\n  When $k = 1$, we also have \\cref{eq:Sk_lowerbound}, as follows:\n  \\begin{align}\n    S_k\n    =\n    \\norm*{x_1 - x_0}^2\n    =\n    \\frac{1}{\\Lest^2} \\norm*{\\nabla f(x_0)}^2\n    =\n    \\frac{1}{\\Lest^2} \\norm*{\\nabla f(\\bar y_1)}^2\n    >\n    \\frac{\\epsilon^2}{\\Lest^2}\n    \\geq\n    \\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3.\n  \\end{align}\n","start":77679,"end":78641}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":["eq:Ksuc_upperbound"],"declared_external":false,"source_only":false,"title":"成功 epoch 的累计迭代数","source_units":[{"raw_tex":"  Combining \\cref{eq:Sk_lowerbound,eq:restart_condition_k} yields\n  \\begin{align}\n    S_k\n    =\n    S_k^{7/8}\n    S_k^{1/8}\n    &\\geq\n    \\prn[\\bigg]{\\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3}^{7/8}\n    \\prn*{\\frac{\\Lest^2}{\\Mest_k^2 (k+1)^5}}^{1/8}\\\\\n    &=\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{3/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^{7/4} \\Mest_k^{1/4}}\n    \\frac{k^{21/8}}{(k+1)^{5/8}}\n    \\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{3/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^2}\n    \\frac{k^{21/8}}{(k+1)^{5/8}}.\n  \\end{align}\n  Plugging this bound into condition \\cref{eq:descent_condition}, we obtain\n  \\begin{align}\n    f(x_0) - f(x_k)\n    \\geq\n    \\frac{\\Lest S_k}{2 (k+1)}\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{1/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^2}\n    \\prn*{\\frac{k}{k+1}}^{13/8}\n    \\frac{k}{2}\\\\\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lmax^{1/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^2}\n    \\prn*{\\frac{1}{2}}^{13/8}\n    \\frac{k}{2}\n    \\geq\n    \\frac{\\epsilon^{7/4} M_0^{7/4}}{\\Lmax^{1/2} \\Mmax^2}\n    \\frac{k}{70},\n  \\end{align}\n  and thus\n  \\begin{align}\n    k\n    \\leq\n    70\n    \\frac{\\Lmax^{1/2} \\Mmax^2}{\\epsilon^{7/4} M_0^{7/4}}\n    \\prn*{f(x_0) - f(x_k)}.\n  \\end{align}\n  Summing this bound over all successful epochs leads to the conclusion that the total iteration number of successful epochs is at most\n  \\begin{align}\n    70 \\frac{\\Lmax^{1/2} \\Mmax^2 \\Delta}{\\epsilon^{7/4} M_0^{7/4}}.\n    \\label{eq:Ksuc_upperbound}\n  \\end{align}\n","start":78641,"end":80096}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":["eq:Nsuc_upperbound"],"declared_external":false,"source_only":false,"title":"成功 epoch 的数量","source_units":[{"raw_tex":"  For later use, we will also evaluate the number of successful epochs.\n  Combining \\cref{eq:Sk_lowerbound,eq:restart_condition_k} also yields\n  \\begin{align}\n    S_k\n    =\n    S_k^{3/4}\n    S_k^{1/4}\n    &\\geq\n    \\prn[\\bigg]{ \\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3 }^{3/4}\n    \\prn*{\\frac{\\Lest^2}{\\Mest_k^2 (k+1)^5}}^{1/4}\\\\\n    &=\n    \\frac{(\\epsilon / 4)^{3/2}}{\\Lest}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^{3/2} \\Mest_k^{1/2}}\n    \\frac{k^{9/4}}{(k+1)^{5/4}}\n    \\geq\n    \\frac{(\\epsilon / 4)^{3/2}}{\\Lest}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^2}\n    \\frac{k^{9/4}}{(k+1)^{5/4}}.\n  \\end{align}\n  Plugging this bound into condition \\cref{eq:descent_condition} gives\n  \\begin{align}\n    f(x_0) - f(x_k)\n    \\geq\n    \\frac{\\Lest S_k}{2 (k+1)}\n    &\\geq\n    \\frac{(\\epsilon / 4)^{3/2}}{2}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^2}\n    \\prn*{\\frac{k}{k+1}}^{9/4}\\\\\n    &\\geq\n    \\frac{(\\epsilon / 4)^{3/2}}{2}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^2}\n    \\prn*{\\frac{1}{2}}^{9/4}\n    =\n    \\frac{\\epsilon^{3/2} \\Mest_0^{3/2}}{2^{25/4} \\Mmax^2}.\n  \\end{align}\n  Thus, we can deduce that the number $N_{\\mathrm{suc}}$ of successful epochs satisfies\n  \\begin{align}\n    N_{\\mathrm{suc}}\n    \\leq\n    2^{25/4}\n    \\frac{\\Mmax^2 \\Delta}{\\epsilon^{3/2} \\Mest_0^{3/2}}.\n    \\label{eq:Nsuc_upperbound}\n  \\end{align}\n","start":80096,"end":81390}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"一般规则下的 epoch 长度","source_units":[{"raw_tex":"  \\paragraph{Other epochs}\n  We deduce from \\cref{eq:grad_norm_ybar_upperbound} that the number of iterations for each epoch is at most\n  \\begin{align}\n    \\prn*{\n      \\frac{4 \\Lmax^2 \\Mmax}{\\epsilon M_0^2}\n    }^{1/4}.\n  \\end{align}\n","start":81391,"end":81626}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":["eq:Nunsuc_upperbound"],"declared_external":false,"source_only":false,"title":"失败 epoch 的对数计数","source_units":[{"raw_tex":"  Let $N_{\\mathrm{unsuc}}$ be the number of unsuccessful epochs.\n  We then have\n  \\begin{align}\n    \\Linit\n    \\beta^{N_{\\mathrm{suc}}}\n    \\alpha^{N_{\\mathrm{unsuc}}}\n    \\leq\n    \\Lmax,\n  \\end{align}\n  and hence\n  \\begin{align}\n    N_{\\mathrm{unsuc}}\n    \\leq\n    c_1 N_{\\mathrm{suc}} + c_2,\n    \\label{eq:Nunsuc_upperbound}\n  \\end{align}\n  where $c_1$ and $c_2$ are defined by \\cref{eq:def_c1_c2}.\n","start":81626,"end":82027}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"合并失败与最后 epoch 的代价","source_units":[{"raw_tex":"  The total iteration number of unsuccessful and last epochs is at most\n  \\begin{alignat}{2}\n    &\\mathInd\n    (N_{\\mathrm{unsuc}} + 1)\n    \\prn*{ \\frac{4 \\Lmax^2 \\Mmax}{\\epsilon M_0^2} }^{1/4}\\\\\n    &\\leq\n    \\prn*{ c_1 N_{\\mathrm{suc}} + c_2 + 1 }\n    \\prn*{ \\frac{4 \\Lmax^2 \\Mmax}{\\epsilon M_0^2} }^{1/4}\n    &\\quad&\\by{\\cref{eq:Nunsuc_upperbound}}\\\\\n    &\\leq\n    2^{27/4} c_1\n    \\frac{\\Lmax^{1/2} \\Mmax^{9/4} \\Delta}{\\epsilon^{7/4} \\Mest_0^2}\n    + 2^{1/2} (c_2 + 1) \\frac{\\Lmax^{1/2} \\Mmax^{1/4}}{\\epsilon^{1/4} M_0^{1/2}}\n    &\\quad&\\by{\\cref{eq:Nsuc_upperbound}}\\\\\n    &\\leq\n    108 c_1\n    \\frac{\\Lmax^{1/2} \\Mmax^{9/4} \\Delta}{\\epsilon^{7/4} \\Mest_0^2}\n    + 2 (c_2 + 1) \\frac{\\Lmax^{1/2} \\Mmax^{1/4}}{\\epsilon^{1/4} M_0^{1/2}}.\n  \\end{alignat}\n  Putting this bound together with \\cref{eq:Ksuc_upperbound} concludes the proof.\n","start":82027,"end":82865}]},{"id":"S074","semantic_type":"proof","kind":"proof","labels":["eq:Ksuc_upperbound_tight","eq:Sk_lowerbound_tight"],"declared_external":false,"source_only":false,"title":"强化复杂度界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of bound \\cref{eq:complexity_bound_tight}]\n  The proof is similar to \\cref{eq:complexity_bound} through the evaluation of the total iteration number of successful, unsuccessful, and last epochs.\n\n  \\paragraph{Successful epochs}\n  Let us focus on a successful epoch.\n  Let $k$ be the iteration number of the epoch.\n  As a counterpart of \\cref{eq:Sk_lowerbound}, we can obtain \n  \\begin{align}\n    S_k > \\prn*{\\frac{\\epsilon}{4 \\Lest}}^2 k^3\n    \\label{eq:Sk_lowerbound_tight}\n  \\end{align}\n  from \\cref{eq:grad_norm_ybar_upperbound_tight}.\n  Combining \\cref{eq:Sk_lowerbound_tight,eq:restart_condition_k} yields\n  \\begin{align}\n    S_k\n    &=\n    S_k^{7/8}\n    S_k^{1/8}\n    \\geq\n    \\prn*{\\prn*{\\frac{\\epsilon}{4 \\Lest}}^2 k^3}^{7/8}\n    \\prn*{\\frac{\\Lest^2}{\\Mest_k^2 (k+1)^5}}^{1/8}\n    =\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{3/2} \\Mest_k^{1/4}}\n    \\frac{k^{21/8}}{(k+1)^{5/8}}.\n  \\end{align}\n  Plugging this bound into condition \\cref{eq:descent_condition}, we obtain\n  \\begin{align}\n    f(x_0) - f(x_k)\n    \\geq\n    \\frac{\\Lest S_k}{2 (k+1)}\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{1/2} \\Mest_k^{1/4}}\n    \\prn*{\\frac{k}{k+1}}^{13/8}\n    \\frac{k}{2}\\\\\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lmax^{1/2} \\Mmax^{1/4}}\n    \\prn*{\\frac{1}{2}}^{13/8}\n    \\frac{k}{2}\n    \\geq\n    \\frac{\\epsilon^{7/4}}{\\Lmax^{1/2} \\Mmax^{1/4}}\n    \\frac{k}{70}.\n  \\end{align}\n  Hence, as in the proof of \\cref{eq:complexity_bound}, the total iteration number of all successful epochs is at most\n  \\begin{align}\n    70\n    \\frac{\\Lmax^{1/2} \\Mmax^{1/4} \\Delta}{\\epsilon^{7/4}}.\n    \\label{eq:Ksuc_upperbound_tight}\n  \\end{align}\n\n  \\paragraph{Other epochs}\n  We deduce from \\cref{eq:grad_norm_ybar_upperbound_tight} that the iteration number of each epoch is at most $\\prn[\\big]{ \\frac{4 \\Lmax^2}{M_0 \\epsilon} }^{1/4}$.\n  In addition, unsuccessful epochs occur only at most $c_2$ times when $\\beta = 1$, where $c_2$ is defined by \\cref{eq:def_c1_c2}.\n  Thus, the total iteration number of unsuccessful and last epochs is at most\n  \\begin{align}\n    \\prn*{\n      c_2 + 1\n    }\n    \\prn*{\n      \\frac{4 \\Lmax^2}{M_0 \\epsilon}\n    }^{1/4}\n    \\leq\n    2 (c_2 + 1) \\frac{\\Lmax^{1/2}}{M_0^{1/4} \\epsilon^{1/4}}.\n  \\end{align}\n  Putting this bound together with \\cref{eq:Ksuc_upperbound_tight} concludes the proof.\n\\end{proof}","start":82878,"end":85215}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":["eq:Sk_lowerbound_tight"],"declared_external":false,"source_only":false,"title":"强化规则下的位移下界","source_units":[{"raw_tex":"  The proof is similar to \\cref{eq:complexity_bound} through the evaluation of the total iteration number of successful, unsuccessful, and last epochs.\n\n  \\paragraph{Successful epochs}\n  Let us focus on a successful epoch.\n  Let $k$ be the iteration number of the epoch.\n  As a counterpart of \\cref{eq:Sk_lowerbound}, we can obtain \n  \\begin{align}\n    S_k > \\prn*{\\frac{\\epsilon}{4 \\Lest}}^2 k^3\n    \\label{eq:Sk_lowerbound_tight}\n  \\end{align}\n  from \\cref{eq:grad_norm_ybar_upperbound_tight}.\n","start":82941,"end":83437}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","labels":["eq:Ksuc_upperbound_tight"],"declared_external":false,"source_only":false,"title":"强化规则下的成功迭代总数","source_units":[{"raw_tex":"  Combining \\cref{eq:Sk_lowerbound_tight,eq:restart_condition_k} yields\n  \\begin{align}\n    S_k\n    &=\n    S_k^{7/8}\n    S_k^{1/8}\n    \\geq\n    \\prn*{\\prn*{\\frac{\\epsilon}{4 \\Lest}}^2 k^3}^{7/8}\n    \\prn*{\\frac{\\Lest^2}{\\Mest_k^2 (k+1)^5}}^{1/8}\n    =\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{3/2} \\Mest_k^{1/4}}\n    \\frac{k^{21/8}}{(k+1)^{5/8}}.\n  \\end{align}\n  Plugging this bound into condition \\cref{eq:descent_condition}, we obtain\n  \\begin{align}\n    f(x_0) - f(x_k)\n    \\geq\n    \\frac{\\Lest S_k}{2 (k+1)}\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{1/2} \\Mest_k^{1/4}}\n    \\prn*{\\frac{k}{k+1}}^{13/8}\n    \\frac{k}{2}\\\\\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lmax^{1/2} \\Mmax^{1/4}}\n    \\prn*{\\frac{1}{2}}^{13/8}\n    \\frac{k}{2}\n    \\geq\n    \\frac{\\epsilon^{7/4}}{\\Lmax^{1/2} \\Mmax^{1/4}}\n    \\frac{k}{70}.\n  \\end{align}\n  Hence, as in the proof of \\cref{eq:complexity_bound}, the total iteration number of all successful epochs is at most\n  \\begin{align}\n    70\n    \\frac{\\Lmax^{1/2} \\Mmax^{1/4} \\Delta}{\\epsilon^{7/4}}.\n    \\label{eq:Ksuc_upperbound_tight}\n  \\end{align}\n","start":83437,"end":84523}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"固定 β 时的其余 epoch 开销","source_units":[{"raw_tex":"  \\paragraph{Other epochs}\n  We deduce from \\cref{eq:grad_norm_ybar_upperbound_tight} that the iteration number of each epoch is at most $\\prn[\\big]{ \\frac{4 \\Lmax^2}{M_0 \\epsilon} }^{1/4}$.\n  In addition, unsuccessful epochs occur only at most $c_2$ times when $\\beta = 1$, where $c_2$ is defined by \\cref{eq:def_c1_c2}.\n  Thus, the total iteration number of unsuccessful and last epochs is at most\n  \\begin{align}\n    \\prn*{\n      c_2 + 1\n    }\n    \\prn*{\n      \\frac{4 \\Lmax^2}{M_0 \\epsilon}\n    }^{1/4}\n    \\leq\n    2 (c_2 + 1) \\frac{\\Lmax^{1/2}}{M_0^{1/4} \\epsilon^{1/4}}.\n  \\end{align}\n  Putting this bound together with \\cref{eq:Ksuc_upperbound_tight} concludes the proof.\n","start":84524,"end":85204}]},{"id":"S078","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"Jensen 型界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{lem:gradient_jensen}]\n  To simplify the notation, let $\\bar z \\coloneqq \\sum_{i=1}^n \\lambda_i z_i$.\n  Taylor's theorem gives\n  \\[\n    \\nabla f(z_i) - \\nabla f(\\bar z)\n    =\n    \\nabla^2 f(\\bar z) (z_i - \\bar z)\n    + \\int_0^1 \\prn*{ \\nabla^2 f(\\bar z + t (z_i - \\bar z)) - \\nabla^2 f(\\bar z) } (z_i - \\bar z) dt,\n  \\]\n  and its weighted average gives\n  \\begin{align}\n    \\sum_{i=1}^n \\lambda_i \\nabla f(z_i) - \\nabla f(\\bar z)\n    &=\n    \\sum_{i=1}^n \\lambda_i\n    \\int_0^1 \\prn*{ \\nabla^2 f(\\bar z + t (z_i - \\bar z)) - \\nabla^2 f(\\bar z) } (z_i - \\bar z) dt.\n  \\end{align}\n  Therefore, we obtain\n  \\begin{alignat}{2}\n    \\norm*{\n      \\sum_{i=1}^n \\lambda_i \\nabla f(z_i) - \\nabla f(\\bar z)\n    }\n    &\\leq\n    \\sum_{i=1}^n \\lambda_i\n    \\int_0^1 \\norm*{ \\nabla^2 f(\\bar z + t (z_i - \\bar z)) - \\nabla^2 f(\\bar z) } \\norm*{z_i - \\bar z} dt\\\\\n    &\\leq\n    \\sum_{i=1}^n \\lambda_i\n    \\int_0^1 M_f t \\norm*{z_i - \\bar z}^2 dt\n    \\quad\\by{\\cref{asm:hessian_lip}}\\\\\n    &=\n    \\frac{M_f}{2} \\sum_{i=1}^n \\lambda_i \\norm*{z_i - \\bar z}^2.\n  \\end{alignat}\n  Furthermore, elementary algebra shows that $\\sum_{i=1}^n \\lambda_i \\norm*{z_i - \\bar z}^2 = \\sum_{1 \\leq i < j \\leq n} \\lambda_i \\lambda_j \\|z_i - z_j\\|^2$ as\n  \\begin{align}\n    \\sum_{1 \\leq i < j \\leq n} \\lambda_i \\lambda_j \\norm*{z_i - z_j}^2\n    &= \n    \\frac{1}{2} \\sum_{i,j = 1}^n \\lambda_i \\lambda_j \\norm*{z_i - z_j}^2\\\\\n    &= \n    \\frac{1}{2} \\sum_{i,j = 1}^n \\lambda_i \\lambda_j \\norm*{z_i}^2\n    + \\frac{1}{2} \\sum_{i,j = 1}^n \\lambda_i \\lambda_j \\norm*{z_j}^2\n    - \\sum_{i,j = 1}^n \\inner{\\lambda_i z_i}{\\lambda_j z_j}\\\\\n    &= \n    \\sum_{i=1}^n \\lambda_i \\norm*{z_i}^2 - \\norm*{\\bar z}^2\n    =\n    \\sum_{i=1}^n \\lambda_i \\norm*{z_i - \\bar z}^2,\n  \\end{align}\n  which completes the proof.\n\\end{proof}","start":106558,"end":108357}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"均值处展开并抵消线性项","source_units":[{"raw_tex":"  To simplify the notation, let $\\bar z \\coloneqq \\sum_{i=1}^n \\lambda_i z_i$.\n  Taylor's theorem gives\n  \\[\n    \\nabla f(z_i) - \\nabla f(\\bar z)\n    =\n    \\nabla^2 f(\\bar z) (z_i - \\bar z)\n    + \\int_0^1 \\prn*{ \\nabla^2 f(\\bar z + t (z_i - \\bar z)) - \\nabla^2 f(\\bar z) } (z_i - \\bar z) dt,\n  \\]\n  and its weighted average gives\n  \\begin{align}\n    \\sum_{i=1}^n \\lambda_i \\nabla f(z_i) - \\nabla f(\\bar z)\n    &=\n    \\sum_{i=1}^n \\lambda_i\n    \\int_0^1 \\prn*{ \\nabla^2 f(\\bar z + t (z_i - \\bar z)) - \\nabla^2 f(\\bar z) } (z_i - \\bar z) dt.\n  \\end{align}\n","start":106609,"end":107163}]},{"id":"S080","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"积分余项的 Lipschitz 控制","source_units":[{"raw_tex":"  Therefore, we obtain\n  \\begin{alignat}{2}\n    \\norm*{\n      \\sum_{i=1}^n \\lambda_i \\nabla f(z_i) - \\nabla f(\\bar z)\n    }\n    &\\leq\n    \\sum_{i=1}^n \\lambda_i\n    \\int_0^1 \\norm*{ \\nabla^2 f(\\bar z + t (z_i - \\bar z)) - \\nabla^2 f(\\bar z) } \\norm*{z_i - \\bar z} dt\\\\\n    &\\leq\n    \\sum_{i=1}^n \\lambda_i\n    \\int_0^1 M_f t \\norm*{z_i - \\bar z}^2 dt\n    \\quad\\by{\\cref{asm:hessian_lip}}\\\\\n    &=\n    \\frac{M_f}{2} \\sum_{i=1}^n \\lambda_i \\norm*{z_i - \\bar z}^2.\n  \\end{alignat}\n","start":107163,"end":107641}]},{"id":"S081","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"加权方差与两两距离恒等式","source_units":[{"raw_tex":"  Furthermore, elementary algebra shows that $\\sum_{i=1}^n \\lambda_i \\norm*{z_i - \\bar z}^2 = \\sum_{1 \\leq i < j \\leq n} \\lambda_i \\lambda_j \\|z_i - z_j\\|^2$ as\n  \\begin{align}\n    \\sum_{1 \\leq i < j \\leq n} \\lambda_i \\lambda_j \\norm*{z_i - z_j}^2\n    &= \n    \\frac{1}{2} \\sum_{i,j = 1}^n \\lambda_i \\lambda_j \\norm*{z_i - z_j}^2\\\\\n    &= \n    \\frac{1}{2} \\sum_{i,j = 1}^n \\lambda_i \\lambda_j \\norm*{z_i}^2\n    + \\frac{1}{2} \\sum_{i,j = 1}^n \\lambda_i \\lambda_j \\norm*{z_j}^2\n    - \\sum_{i,j = 1}^n \\inner{\\lambda_i z_i}{\\lambda_j z_j}\\\\\n    &= \n    \\sum_{i=1}^n \\lambda_i \\norm*{z_i}^2 - \\norm*{\\bar z}^2\n    =\n    \\sum_{i=1}^n \\lambda_i \\norm*{z_i - \\bar z}^2,\n  \\end{align}\n  which completes the proof.\n","start":107641,"end":108346}]},{"id":"S082","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"梯形误差界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{lem:trapezoidal_rule_error}]\n  We obtain the desired result as follows:\n  \\begin{alignat}{2}\n    &\\mathInd\n    % \\abs*{\n      f(x) - f(y)\n      - \\frac{1}{2} \\inner*{\\nabla f(x) + \\nabla f(y)}{x - y}\\\\\n    % }\\\\\n    &=\n    % \\abs*{\n      \\int_0^1 \\inner*{\\nabla f(t x + (1 - t) y)}{x - y} dt\n      - \\frac{1}{2} \\inner*{\\nabla f(x) + \\nabla f(y)}{x - y}\n    % }\n    &\\quad&\\by{Taylor's\\\\theorem}\\\\\n    &=\n    % \\abs*{\n      \\int_0^1 \\inner[\\Big]{\\nabla f(t x + (1 - t) y) - t \\nabla f(x) - (1 - t) \\nabla f(y)}{x - y} dt\\\\\n    % }\\\\\n    &\\leq\n    \\int_0^1 \\norm[\\Big]{\\nabla f(t x + (1 - t) y) - t \\nabla f(x) - (1 - t) \\nabla f(y)} \\norm{x - y} dt\\\\\n    &\\leq\n    \\frac{\\Mtrue}{2} \\int_0^1 t (1 - t) \\norm{x - y}^3 dt\n    =\n    \\frac{\\Mtrue}{12} \\norm{x - y}^3\n    &\\quad&\\by{\\cref{lem:gradient_jensen}}.\n  \\end{alignat}\n\\end{proof}","start":108360,"end":109222}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"梯形误差的线段积分表示","source_units":[{"raw_tex":"  We obtain the desired result as follows:\n  \\begin{alignat}{2}\n    &\\mathInd\n    % \\abs*{\n      f(x) - f(y)\n      - \\frac{1}{2} \\inner*{\\nabla f(x) + \\nabla f(y)}{x - y}\\\\\n    % }\\\\\n    &=\n    % \\abs*{\n      \\int_0^1 \\inner*{\\nabla f(t x + (1 - t) y)}{x - y} dt\n      - \\frac{1}{2} \\inner*{\\nabla f(x) + \\nabla f(y)}{x - y}\n    % }\n    &\\quad&\\by{Taylor's\\\\theorem}\\\\\n    &=\n    % \\abs*{\n      \\int_0^1 \\inner[\\Big]{\\nabla f(t x + (1 - t) y) - t \\nabla f(x) - (1 - t) \\nabla f(y)}{x - y} dt\\\\\n    % }\\\\\n","start":108418,"end":108922}]},{"id":"S084","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"积分 Jensen 误差得到三次界","source_units":[{"raw_tex":"    &\\leq\n    \\int_0^1 \\norm[\\Big]{\\nabla f(t x + (1 - t) y) - t \\nabla f(x) - (1 - t) \\nabla f(y)} \\norm{x - y} dt\\\\\n    &\\leq\n    \\frac{\\Mtrue}{2} \\int_0^1 t (1 - t) \\norm{x - y}^3 dt\n    =\n    \\frac{\\Mtrue}{12} \\norm{x - y}^3\n    &\\quad&\\by{\\cref{lem:gradient_jensen}}.\n  \\end{alignat}\n","start":108922,"end":109211}]},{"id":"S085","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始外部前提清单：Taylor's theorem with integral remainder and the fundamental theorem of calculus。"],"source_only":false,"title":"Taylor 积分余项与微积分基本定理","source_units":[]},{"id":"S086","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始外部前提清单：Triangle, Cauchy–Schwarz, and Young inequalities。"],"source_only":false,"title":"三角与内积标准不等式","source_units":[]},{"id":"S087","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始外部前提清单：monotonicity of finite-dimensional sequence norms。"],"source_only":false,"title":"有限序列范数的单调性","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S063","edges":[{"source_id":"S004","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S005","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S009","target_id":"S011","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S008","target_id":"S015","kind":"supports"},{"source_id":"S013","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S013","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S019","kind":"supports"},{"source_id":"S012","target_id":"S022","kind":"supports"},{"source_id":"S006","target_id":"S022","kind":"supports"},{"source_id":"S008","target_id":"S022","kind":"supports"},{"source_id":"S008","target_id":"S023","kind":"supports"},{"source_id":"S005","target_id":"S033","kind":"supports"},{"source_id":"S007","target_id":"S033","kind":"supports"},{"source_id":"S002","target_id":"S033","kind":"supports"},{"source_id":"S016","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S007","target_id":"S034","kind":"supports"},{"source_id":"S016","target_id":"S034","kind":"supports"},{"source_id":"S086","target_id":"S034","kind":"supports"},{"source_id":"S016","target_id":"S035","kind":"supports"},{"source_id":"S086","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S021","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S023","target_id":"S037","kind":"supports"},{"source_id":"S086","target_id":"S037","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S032","kind":"supports"},{"source_id":"S035","target_id":"S032","kind":"supports"},{"source_id":"S036","target_id":"S032","kind":"supports"},{"source_id":"S037","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S021","target_id":"S039","kind":"supports"},{"source_id":"S007","target_id":"S039","kind":"supports"},{"source_id":"S010","target_id":"S039","kind":"supports"},{"source_id":"S087","target_id":"S039","kind":"supports"},{"source_id":"S031","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S023","target_id":"S040","kind":"supports"},{"source_id":"S016","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S022","target_id":"S041","kind":"supports"},{"source_id":"S023","target_id":"S041","kind":"supports"},{"source_id":"S010","target_id":"S041","kind":"supports"},{"source_id":"S007","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S041","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S009","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S007","target_id":"S047","kind":"supports"},{"source_id":"S026","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S007","target_id":"S049","kind":"supports"},{"source_id":"S013","target_id":"S049","kind":"supports"},{"source_id":"S026","target_id":"S050","kind":"supports"},{"source_id":"S013","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S007","target_id":"S050","kind":"supports"},{"source_id":"S086","target_id":"S050","kind":"supports"},{"source_id":"S007","target_id":"S051","kind":"supports"},{"source_id":"S086","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S050","target_id":"S052","kind":"supports"},{"source_id":"S010","target_id":"S052","kind":"supports"},{"source_id":"S007","target_id":"S052","kind":"supports"},{"source_id":"S049","target_id":"S048","kind":"supports"},{"source_id":"S050","target_id":"S048","kind":"supports"},{"source_id":"S051","target_id":"S048","kind":"supports"},{"source_id":"S052","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S045","kind":"supports"},{"source_id":"S027","target_id":"S043","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"},{"source_id":"S017","target_id":"S043","kind":"supports"},{"source_id":"S018","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S028","target_id":"S024","kind":"supports"},{"source_id":"S042","target_id":"S024","kind":"supports"},{"source_id":"S015","target_id":"S056","kind":"supports"},{"source_id":"S045","target_id":"S057","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S010","target_id":"S057","kind":"supports"},{"source_id":"S086","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S058","kind":"supports"},{"source_id":"S022","target_id":"S058","kind":"supports"},{"source_id":"S024","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S022","target_id":"S059","kind":"supports"},{"source_id":"S024","target_id":"S059","kind":"supports"},{"source_id":"S016","target_id":"S059","kind":"supports"},{"source_id":"S056","target_id":"S055","kind":"supports"},{"source_id":"S057","target_id":"S055","kind":"supports"},{"source_id":"S058","target_id":"S055","kind":"supports"},{"source_id":"S059","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S053","kind":"supports"},{"source_id":"S019","target_id":"S061","kind":"supports"},{"source_id":"S016","target_id":"S061","kind":"supports"},{"source_id":"S056","target_id":"S061","kind":"supports"},{"source_id":"S057","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S022","target_id":"S062","kind":"supports"},{"source_id":"S024","target_id":"S062","kind":"supports"},{"source_id":"S016","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S054","kind":"supports"},{"source_id":"S009","target_id":"S066","kind":"supports"},{"source_id":"S010","target_id":"S066","kind":"supports"},{"source_id":"S012","target_id":"S066","kind":"supports"},{"source_id":"S006","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S012","target_id":"S067","kind":"supports"},{"source_id":"S053","target_id":"S068","kind":"supports"},{"source_id":"S066","target_id":"S068","kind":"supports"},{"source_id":"S016","target_id":"S068","kind":"supports"},{"source_id":"S010","target_id":"S068","kind":"supports"},{"source_id":"S007","target_id":"S068","kind":"supports"},{"source_id":"S013","target_id":"S068","kind":"supports"},{"source_id":"S024","target_id":"S068","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S067","target_id":"S069","kind":"supports"},{"source_id":"S010","target_id":"S069","kind":"supports"},{"source_id":"S024","target_id":"S069","kind":"supports"},{"source_id":"S011","target_id":"S069","kind":"supports"},{"source_id":"S001","target_id":"S069","kind":"supports"},{"source_id":"S064","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S070","kind":"supports"},{"source_id":"S067","target_id":"S070","kind":"supports"},{"source_id":"S010","target_id":"S070","kind":"supports"},{"source_id":"S024","target_id":"S070","kind":"supports"},{"source_id":"S011","target_id":"S070","kind":"supports"},{"source_id":"S001","target_id":"S070","kind":"supports"},{"source_id":"S064","target_id":"S070","kind":"supports"},{"source_id":"S053","target_id":"S071","kind":"supports"},{"source_id":"S066","target_id":"S071","kind":"supports"},{"source_id":"S009","target_id":"S072","kind":"supports"},{"source_id":"S066","target_id":"S072","kind":"supports"},{"source_id":"S024","target_id":"S072","kind":"supports"},{"source_id":"S064","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S070","target_id":"S073","kind":"supports"},{"source_id":"S069","target_id":"S073","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"},{"source_id":"S067","target_id":"S065","kind":"supports"},{"source_id":"S068","target_id":"S065","kind":"supports"},{"source_id":"S069","target_id":"S065","kind":"supports"},{"source_id":"S070","target_id":"S065","kind":"supports"},{"source_id":"S071","target_id":"S065","kind":"supports"},{"source_id":"S072","target_id":"S065","kind":"supports"},{"source_id":"S073","target_id":"S065","kind":"supports"},{"source_id":"S054","target_id":"S075","kind":"supports"},{"source_id":"S066","target_id":"S075","kind":"supports"},{"source_id":"S068","target_id":"S075","kind":"supports"},{"source_id":"S007","target_id":"S075","kind":"supports"},{"source_id":"S013","target_id":"S075","kind":"supports"},{"source_id":"S075","target_id":"S076","kind":"supports"},{"source_id":"S067","target_id":"S076","kind":"supports"},{"source_id":"S010","target_id":"S076","kind":"supports"},{"source_id":"S024","target_id":"S076","kind":"supports"},{"source_id":"S011","target_id":"S076","kind":"supports"},{"source_id":"S001","target_id":"S076","kind":"supports"},{"source_id":"S064","target_id":"S076","kind":"supports"},{"source_id":"S054","target_id":"S077","kind":"supports"},{"source_id":"S072","target_id":"S077","kind":"supports"},{"source_id":"S064","target_id":"S077","kind":"supports"},{"source_id":"S076","target_id":"S077","kind":"supports"},{"source_id":"S075","target_id":"S074","kind":"supports"},{"source_id":"S076","target_id":"S074","kind":"supports"},{"source_id":"S077","target_id":"S074","kind":"supports"},{"source_id":"S065","target_id":"S063","kind":"supports"},{"source_id":"S074","target_id":"S063","kind":"supports"},{"source_id":"S002","target_id":"S063","kind":"supports"},{"source_id":"S003","target_id":"S063","kind":"supports"},{"source_id":"S009","target_id":"S063","kind":"supports"},{"source_id":"S013","target_id":"S063","kind":"supports"},{"source_id":"S064","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S025","kind":"supports"},{"source_id":"S009","target_id":"S025","kind":"supports"},{"source_id":"S017","target_id":"S025","kind":"supports"},{"source_id":"S018","target_id":"S025","kind":"supports"},{"source_id":"S085","target_id":"S079","kind":"supports"},{"source_id":"S079","target_id":"S080","kind":"supports"},{"source_id":"S003","target_id":"S080","kind":"supports"},{"source_id":"S004","target_id":"S080","kind":"supports"},{"source_id":"S086","target_id":"S080","kind":"supports"},{"source_id":"S079","target_id":"S081","kind":"supports"},{"source_id":"S004","target_id":"S081","kind":"supports"},{"source_id":"S079","target_id":"S078","kind":"supports"},{"source_id":"S080","target_id":"S078","kind":"supports"},{"source_id":"S081","target_id":"S078","kind":"supports"},{"source_id":"S078","target_id":"S026","kind":"supports"},{"source_id":"S085","target_id":"S083","kind":"supports"},{"source_id":"S083","target_id":"S084","kind":"supports"},{"source_id":"S026","target_id":"S084","kind":"supports"},{"source_id":"S086","target_id":"S084","kind":"supports"},{"source_id":"S083","target_id":"S082","kind":"supports"},{"source_id":"S084","target_id":"S082","kind":"supports"},{"source_id":"S082","target_id":"S027","kind":"supports"},{"source_id":"S013","target_id":"S045","kind":"supports"},{"source_id":"S010","target_id":"S045","kind":"supports"},{"source_id":"S007","target_id":"S045","kind":"supports"},{"source_id":"S021","target_id":"S031","kind":"supports"},{"source_id":"S017","target_id":"S020","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S063","edges":[{"source_id":"S004","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S005","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S009","target_id":"S011","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S008","target_id":"S015","kind":"supports"},{"source_id":"S013","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S013","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S019","kind":"supports"},{"source_id":"S012","target_id":"S022","kind":"supports"},{"source_id":"S006","target_id":"S022","kind":"supports"},{"source_id":"S008","target_id":"S022","kind":"supports"},{"source_id":"S008","target_id":"S023","kind":"supports"},{"source_id":"S005","target_id":"S031","kind":"supports"},{"source_id":"S007","target_id":"S031","kind":"supports"},{"source_id":"S002","target_id":"S031","kind":"supports"},{"source_id":"S016","target_id":"S031","kind":"supports"},{"source_id":"S086","target_id":"S031","kind":"supports"},{"source_id":"S021","target_id":"S031","kind":"supports"},{"source_id":"S023","target_id":"S031","kind":"supports"},{"source_id":"S021","target_id":"S030","kind":"supports"},{"source_id":"S007","target_id":"S030","kind":"supports"},{"source_id":"S010","target_id":"S030","kind":"supports"},{"source_id":"S087","target_id":"S030","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S023","target_id":"S030","kind":"supports"},{"source_id":"S016","target_id":"S030","kind":"supports"},{"source_id":"S022","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S028","kind":"supports"},{"source_id":"S009","target_id":"S028","kind":"supports"},{"source_id":"S007","target_id":"S044","kind":"supports"},{"source_id":"S026","target_id":"S044","kind":"supports"},{"source_id":"S007","target_id":"S045","kind":"supports"},{"source_id":"S013","target_id":"S045","kind":"supports"},{"source_id":"S026","target_id":"S045","kind":"supports"},{"source_id":"S086","target_id":"S045","kind":"supports"},{"source_id":"S010","target_id":"S045","kind":"supports"},{"source_id":"S027","target_id":"S042","kind":"supports"},{"source_id":"S044","target_id":"S042","kind":"supports"},{"source_id":"S045","target_id":"S042","kind":"supports"},{"source_id":"S017","target_id":"S042","kind":"supports"},{"source_id":"S018","target_id":"S042","kind":"supports"},{"source_id":"S028","target_id":"S024","kind":"supports"},{"source_id":"S042","target_id":"S024","kind":"supports"},{"source_id":"S015","target_id":"S056","kind":"supports"},{"source_id":"S045","target_id":"S057","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S010","target_id":"S057","kind":"supports"},{"source_id":"S086","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S053","kind":"supports"},{"source_id":"S056","target_id":"S053","kind":"supports"},{"source_id":"S022","target_id":"S053","kind":"supports"},{"source_id":"S024","target_id":"S053","kind":"supports"},{"source_id":"S016","target_id":"S053","kind":"supports"},{"source_id":"S019","target_id":"S054","kind":"supports"},{"source_id":"S016","target_id":"S054","kind":"supports"},{"source_id":"S056","target_id":"S054","kind":"supports"},{"source_id":"S057","target_id":"S054","kind":"supports"},{"source_id":"S022","target_id":"S054","kind":"supports"},{"source_id":"S024","target_id":"S054","kind":"supports"},{"source_id":"S009","target_id":"S066","kind":"supports"},{"source_id":"S010","target_id":"S066","kind":"supports"},{"source_id":"S012","target_id":"S066","kind":"supports"},{"source_id":"S006","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S012","target_id":"S067","kind":"supports"},{"source_id":"S053","target_id":"S068","kind":"supports"},{"source_id":"S066","target_id":"S068","kind":"supports"},{"source_id":"S016","target_id":"S068","kind":"supports"},{"source_id":"S010","target_id":"S068","kind":"supports"},{"source_id":"S007","target_id":"S068","kind":"supports"},{"source_id":"S013","target_id":"S068","kind":"supports"},{"source_id":"S024","target_id":"S068","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S067","target_id":"S069","kind":"supports"},{"source_id":"S010","target_id":"S069","kind":"supports"},{"source_id":"S024","target_id":"S069","kind":"supports"},{"source_id":"S011","target_id":"S069","kind":"supports"},{"source_id":"S001","target_id":"S069","kind":"supports"},{"source_id":"S064","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S070","kind":"supports"},{"source_id":"S067","target_id":"S070","kind":"supports"},{"source_id":"S010","target_id":"S070","kind":"supports"},{"source_id":"S024","target_id":"S070","kind":"supports"},{"source_id":"S011","target_id":"S070","kind":"supports"},{"source_id":"S001","target_id":"S070","kind":"supports"},{"source_id":"S064","target_id":"S070","kind":"supports"},{"source_id":"S053","target_id":"S071","kind":"supports"},{"source_id":"S066","target_id":"S071","kind":"supports"},{"source_id":"S009","target_id":"S072","kind":"supports"},{"source_id":"S066","target_id":"S072","kind":"supports"},{"source_id":"S024","target_id":"S072","kind":"supports"},{"source_id":"S064","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S070","target_id":"S073","kind":"supports"},{"source_id":"S069","target_id":"S073","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"},{"source_id":"S067","target_id":"S065","kind":"supports"},{"source_id":"S068","target_id":"S065","kind":"supports"},{"source_id":"S069","target_id":"S065","kind":"supports"},{"source_id":"S070","target_id":"S065","kind":"supports"},{"source_id":"S071","target_id":"S065","kind":"supports"},{"source_id":"S072","target_id":"S065","kind":"supports"},{"source_id":"S073","target_id":"S065","kind":"supports"},{"source_id":"S054","target_id":"S075","kind":"supports"},{"source_id":"S066","target_id":"S075","kind":"supports"},{"source_id":"S068","target_id":"S075","kind":"supports"},{"source_id":"S007","target_id":"S075","kind":"supports"},{"source_id":"S013","target_id":"S075","kind":"supports"},{"source_id":"S075","target_id":"S076","kind":"supports"},{"source_id":"S067","target_id":"S076","kind":"supports"},{"source_id":"S010","target_id":"S076","kind":"supports"},{"source_id":"S024","target_id":"S076","kind":"supports"},{"source_id":"S011","target_id":"S076","kind":"supports"},{"source_id":"S001","target_id":"S076","kind":"supports"},{"source_id":"S064","target_id":"S076","kind":"supports"},{"source_id":"S054","target_id":"S077","kind":"supports"},{"source_id":"S072","target_id":"S077","kind":"supports"},{"source_id":"S064","target_id":"S077","kind":"supports"},{"source_id":"S076","target_id":"S077","kind":"supports"},{"source_id":"S075","target_id":"S074","kind":"supports"},{"source_id":"S076","target_id":"S074","kind":"supports"},{"source_id":"S077","target_id":"S074","kind":"supports"},{"source_id":"S065","target_id":"S063","kind":"supports"},{"source_id":"S074","target_id":"S063","kind":"supports"},{"source_id":"S002","target_id":"S063","kind":"supports"},{"source_id":"S003","target_id":"S063","kind":"supports"},{"source_id":"S009","target_id":"S063","kind":"supports"},{"source_id":"S013","target_id":"S063","kind":"supports"},{"source_id":"S064","target_id":"S063","kind":"supports"},{"source_id":"S085","target_id":"S026","kind":"supports"},{"source_id":"S003","target_id":"S026","kind":"supports"},{"source_id":"S004","target_id":"S026","kind":"supports"},{"source_id":"S086","target_id":"S026","kind":"supports"},{"source_id":"S085","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S086","target_id":"S027","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"有下界的非凸最小化","source_units":[{"raw_tex":"This paper studies general nonconvex optimization problems:\n\\begin{align}\n  \\min_{x \\in \\R^d} \\ \n  f(x),\n\\end{align}\nwhere $f: \\R^d \\to \\R$ is bounded below (i.e., $\\inf_{x \\in \\R^d} f(x) > - \\infty$) and satisfies the following Lipschitz assumption.\n","start":14872,"end":15123}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":["asm:gradient_hessian_lip","asm:gradient_lip","asm:hessian_lip"],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"梯度全局 Lipschitz","source_units":[{"raw_tex":"\\begin{assumption}\n  \\label{asm:gradient_hessian_lip}\n  Let $\\Ltrue, \\Mtrue > 0$ be constants.\n  \\begin{enuminasm}\n    \\item\n    \\label{asm:gradient_lip}\n    $\\norm*{\\nabla f(x) - \\nabla f(y)} \\leq \\Ltrue \\norm*{x - y}$ for all $x, y \\in \\R^d$,\n","start":15123,"end":15368}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","labels":["asm:gradient_hessian_lip","asm:gradient_lip","asm:hessian_lip"],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"Hessian 全局 Lipschitz","source_units":[{"raw_tex":"  Let $\\Ltrue, \\Mtrue > 0$ be constants.\n","start":15177,"end":15218},{"raw_tex":"    \\item\n    \\label{asm:hessian_lip}\n    $\\norm*{\\nabla^2 f(x) - \\nabla^2 f(y)} \\leq \\Mtrue \\norm*{x - y}$ for all $x, y \\in \\R^d$.\n  \\end{enuminasm}\n\\end{assumption}\n","start":15368,"end":15536}]},{"id":"S004","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"内积与范数约定","source_units":[{"raw_tex":"\\paragraph{Notation}\nFor vectors $a, b \\in \\R^d$, let $\\inner{a}{b}$ denote the dot product and $\\norm{a}$ denote the Euclidean norm.\nFor a matrix $A \\in \\R^{m \\times n}$, let $\\norm{A}$ denote the operator norm, or equivalently the largest singular value.\n","start":21361,"end":21618}]},{"id":"S005","semantic_type":"lemma","kind":"lemma","labels":["lem:eq:property_gradlip_obj"],"member_ids":["S005"],"declared_external":true,"declaration_text":["原始允许前提清单明确允许 B0012 标准光滑下降引理，不要求提供文内证明。"],"source_only":false,"title":"标准光滑下降引理","source_units":[{"raw_tex":"\\begin{lemma}\n  \\label{lem:eq:property_gradlip_obj}\n  Under \\cref{asm:gradient_lip}, the following holds for any $x, y \\in \\R^d$:\n  \\begin{align}\n    f(x) - f(y)\n    &\\leq\n    \\inner*{\\nabla f(y)}{x - y}\n    + \\frac{\\Ltrue}{2} \\norm*{x - y}^2.\n  \\end{align}\n\\end{lemma}\n","start":32617,"end":32887}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"轮次与计数器","source_units":[{"raw_tex":"The proposed method is the classical accelerated gradient descent (AGD) \\cite{nesterov1983method,nesterov2018lectures} with two restart schemes, presented in \\cref{alg:proposed_agd}.\nIn the algorithm, the iteration counter $k$ is reset to $0$ when AGD restarts on Line \\ref{alg-line-agd:restart_unsuccessful} or \\ref{alg-line-agd:restart_successful}, whereas the total iteration counter $K$ is not.\nWe refer to the period between a reset of $k$ and the next reset as an \\emph{epoch}.\nNote that the total iteration counter $K$ in the algorithm is unnecessary to implement; it is included here only to make the statements in our analysis concise.\n\nThe algorithm uses estimates $L$ and $M_k$ for the Lipschitz constants $L_f$ and $M_f$.\nThe estimate $L$ is fixed during an epoch, while $M_k$ is updated at each iteration, having the subscript $k$.\nNow let us describe each operation of \\cref{alg:proposed_agd}.\n\n","start":39638,"end":40547}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":["eq:update_xy"],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"加速梯度迭代","source_units":[{"raw_tex":"\\paragraph{Solution update (Lines \\ref{alg-line-agd:update_x} and \\ref{alg-line-agd:update_y})}\nWith an estimate $L$ of the Lipschitz constant $L_f$, AGD defines solution sequences $(x_k)$ and $(y_k)$ as follows:\n$y_0 = x_0$ and\n\\begin{align}\n  % \\left\\{\n  \\begin{aligned}\n    x_k\n    &=\n    y_{k-1} - \\frac{1}{L} \\nabla f(y_{k-1}),\\\\\n    y_k\n    &=\n    x_k + \\theta_k (x_k - x_{k-1})\n  \\end{aligned}\n  % \\right.\n  \\label{eq:update_xy}\n\\end{align}\nfor $k \\geq 1$, where $0 \\leq \\theta_k \\leq 1$ is the acceleration parameter.\nLet $x_{-1} \\coloneqq x_0$ for convenience, which makes the lower equation of \\cref{eq:update_xy} valid for $k = 0$.\nSetting a suitable value of $\\theta_k$ is crucial for deriving an improved complexity bound over that of gradient descent.\n","start":40547,"end":41313}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":["eq:def_theta_k"],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"加速参数的具体选择","source_units":[{"raw_tex":"In particular, we propose to set\n\\begin{align}\n  \\theta_k = \\frac{k}{k+1}\n  \\label{eq:def_theta_k}\n\\end{align}\nfor $k \\geq 1$.\n","start":41313,"end":41440}]},{"id":"S009","semantic_type":"algorithm","kind":"algorithm","labels":["alg-line-agd:check_decrease","alg-line-agd:check_large_Sk","alg-line-agd:restart_successful","alg-line-agd:restart_unsuccessful","alg-line-agd:update_x","alg-line-agd:update_y","alg:proposed_agd"],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"双重重启的无参数 AGD","source_units":[{"raw_tex":"\\begin{algorithm}[t]\n  \\caption{Proposed AGD with restart~\\label{alg:proposed_agd}} \n  \\begin{algorithmic}[1]\n    \\Require{%\n      \\pbox[t]{40em}{\n        $x_{\\mathrm{init}} \\in \\R^d$;\n        $\\Linit, M_0 > 0$;\n        $\\alpha > 1$;\n        $0 < \\beta \\leq 1$\\\\\n        Recommended: $(\\Linit, M_0, \\alpha, \\beta) = (10^{-3}, 10^{-16}, 2, 0.9)$\n      }\n    }\n    \\State{%\n      $(x_0, y_0) \\gets (x_{\\mathrm{init}}, x_{\\mathrm{init}})$,\n      $\\Lest \\gets \\Linit$,\n      $k \\gets 0$,\n      $K \\gets 0$\n    }\n    \\Repeat\n      \\State{%\n        $k \\gets k + 1$,\n        $K \\gets K + 1$\n      }\n      \\State $x_k \\gets y_{k-1} - \\frac{1}{\\Lest} \\nabla f(y_{k-1})$\n      \\label{alg-line-agd:update_x}\n      \\State $y_k \\gets x_k + \\frac{k}{k+1} (x_k - x_{k-1})$\n      \\label{alg-line-agd:update_y}\n\n      \\If{$f(x_k) > f(x_0) - \\frac{\\Lest S_k}{2 (k+1)}$}\n      \\Comment{$S_k \\coloneqq \\sum_{i=1}^k \\norm*{x_i - x_{i-1}}^2$}\n      \\label{alg-line-agd:check_decrease}\n        \\State{%\n          $(x_0, y_0) \\gets (x_{k-1}, x_{k-1})$,\n          $\\Lest \\gets \\alpha \\Lest$,\n          $k \\gets 0$\n        }\n        \\label{alg-line-agd:restart_unsuccessful}\n      \\ElsIf{$(k+1)^5 \\Mest_k^2 S_k > \\Lest^2$}\n      \\Comment{$\\Mest_k$ is defined by \\cref{eq:update_Mest_practical} or \\cref{eq:update_Mest}}\n      \\label{alg-line-agd:check_large_Sk}\n        \\State{%\n          $(x_0, y_0) \\gets (x_k, x_k)$,\n          $\\Lest \\gets \\beta \\Lest$,\n          $k \\gets 0$\n        }\n        \\label{alg-line-agd:restart_successful}\n      \\EndIf\n    \\Until{convergence}\n    \\State \\Return averaged solution $\\bar y_k$ defined by \\eqref{eq:def_ybark}\n  \\end{algorithmic}\n\\end{algorithm}\n","start":41876,"end":43540}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":["eq:descent_condition"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"下降检验与累计位移","source_units":[{"raw_tex":"\\paragraph{Restart mechanism 1 (Lines \\ref{alg-line-agd:check_decrease} and \\ref{alg-line-agd:restart_unsuccessful})}\nAfter updating the solutions, \\cref{alg:proposed_agd} checks whether the current estimate $L$ approximates the actual value $L_f$ well.\nIf\n\\begin{align}\n  f(x_k) - f(x_0)\n  \\leq\n  - \\frac{1 - \\theta_k}{2} \\Lest S_k\n  =\n  - \\frac{\\Lest S_k}{2 (k+1)},\n  \\quad\\text{where}\\quad\n  S_k\n  \\coloneqq\n  \\sum_{i=1}^k \\norm*{x_i - x_{i-1}}^2,\n  \\label{eq:descent_condition}\n\\end{align}\nthen $L$ is judged suitable, and the epoch continues; otherwise, AGD restarts from $x_{k-1}$ with a larger $L$.\n","start":43542,"end":44148}]},{"id":"S011","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"失败重启不增加目标值","source_units":[{"raw_tex":"Since condition~\\cref{eq:descent_condition} holds in the previous iteration, starting the next epoch from $x_{k-1}$ ensures that this epoch does not increase the objective function value.\nCondition~\\cref{eq:descent_condition} guarantees a sufficient decrease in the objective function in the epoch and plays a crucial role in our analysis.\nAs will be shown later, this condition is always satisfied when $L$ is sufficiently large, which justifies the restart mechanism.\n","start":44148,"end":44618}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["eq:condition_continue"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"续行检验与成功重启","source_units":[{"raw_tex":"\\paragraph{Restart mechanism 2 (Lines \\ref{alg-line-agd:check_large_Sk} and \\ref{alg-line-agd:restart_successful})}\nOur AGD also resets the effect of acceleration by restart when $S_k$ becomes large; the restart condition involves an estimate $M_k$ of the Hessian's Lipschitz constant $M_f$ (detailed later).\nIf\n\\begin{align}\n  \\Mest_k^2 S_k\n  \\leq\n  \\Lest^2 (1 - \\theta_k)^5\n  =\n  \\frac{\\Lest^2}{(k+1)^5},\n  \\label{eq:condition_continue}\n\\end{align}\nthe epoch continues, otherwise, AGD restarts with the computed solution $x_k$ and a (possibly) smaller $L$.\nCondition~\\cref{eq:condition_continue} is inspired by a condition in \\cite{li2022restarted}, $k M_f S_k \\leq \\epsilon$; unlike the existing one, our condition does not involve $M_f$ or $\\epsilon$.\nAlthough $L$ does not have to be decreased on Line~\\ref{alg-line-agd:restart_successful} to derive the complexity bound of $O(\\epsilon^{-7/4})$, decreasing $L$ will help the algorithm to capture the local curvature of $f$ and to converge faster empirically.\n","start":45311,"end":46325}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["eq:def_ybark"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"加权平均点与归一化因子","source_units":[{"raw_tex":"\\paragraph{Averaged solution}\n\\label{sec:averaged_solution}\nOur AGD guarantees that the gradient norm is small at a weighted average $\\bar y_k$ of the solutions $y_0,\\dots,y_{k-1}$.\nThe averaged solution $\\bar y_k$ is defined by\n\\begin{align}\n  \\bar y_k\n  \\coloneqq\n  \\sum_{i=0}^{k-1} p_{k,i} y_i,\n  \\quad\\text{where}\\quad\n  p_{k,i}\n  \\coloneqq\n  \\frac{1}{Z_k} \\prod_{j=i+1}^{k-1} \\theta_j,\n  \\quad\n  Z_k\n  \\coloneqq\n  \\sum_{i=0}^{k-1} \\prod_{j=i+1}^{k-1} \\theta_j.\n  \\label{eq:def_ybark}\n\\end{align}\n","start":46327,"end":46828}]},{"id":"S014","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"平均点的递推实现","source_units":[{"raw_tex":"Note that it is unnecessary to keep all $y_0,\\dots,y_{k-1}$ in memory to compute $\\bar y_k$ because $Z_k$ and $\\bar y_k$ satisfy a simple recursion:\n\\begin{align}\n  Z_{k+1}\n  &=\n  1 + \\theta_k Z_k,\\quad\n  \\bar y_{k+1}\n  =\n  \\frac{1}{Z_{k+1}} \\prn*{y_k + \\theta_k Z_k \\bar y_k},\n\\end{align}\nwhich maintains the computational efficiency of the algorithm.\n","start":46828,"end":47181}]},{"id":"S015","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:ybar_Zk_alternative"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"平均权重的显式形式","source_units":[{"raw_tex":"Given \\cref{eq:def_theta_k}, there is another form:\n\\begin{align}\n  \\bar y_k\n  =\n  \\frac{2}{k (k + 1)}\n  \\sum_{i=0}^{k-1} (i + 1) y_i,\\quad\n  Z_k\n  =\n  \\frac{k + 1}{2},\n  \\label{eq:ybar_Zk_alternative}\n\\end{align}\nand this simplification of $\\bar y_k$ and $Z_k$ is one of the aims of setting $\\theta_k$ as in \\cref{eq:def_theta_k}.\n","start":47181,"end":47513}]},{"id":"S016","semantic_type":"definition","kind":"definition","labels":["eq:fy_fx_diff_upperbound_Mestk","eq:grad_interpolation_error_Mestk"],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"Hessian 估计须满足的条件","source_units":[{"raw_tex":"\\paragraph{Estimate of Hessian's Lipschitz constant}\nThe estimate $M_k$ for the Hessian is updated at each iteration, unlike $L$ for the gradient.\nOur complexity analysis requires three inequalities on $M_k$: $M_{k-1} \\leq M_k$ and\n\\begin{align}\n  &\n  f(y_k) - f(x_k)\n  \\leq\n  \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n  + \\frac{\\Mest_k}{12} \\norm*{y_k - x_k}^3,\n  \\label{eq:fy_fx_diff_upperbound_Mestk}\\\\\n  &\n  \\norm*{\n    \\nabla f(y_k)\n    + \\theta_k \\nabla f(x_{k-1})\n    - (1 + \\theta_k) \\nabla f (x_k)\n  }\n  \\leq\n  \\theta_k \\Mest_k \\norm*{x_k - x_{k-1}}^2.\n  \\label{eq:grad_interpolation_error_Mestk}\n\\end{align}\n","start":47515,"end":48153}]},{"id":"S017","semantic_type":"algorithm","kind":"algorithm","labels":["eq:update_Mest_practical"],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"实用 Hessian 估计规则","source_units":[{"raw_tex":"Finding the smallest $M_k$ that satisfies these inequalities is desirable, and fortunately, it is straightforward,\n\\begin{align}\n  \\label{eq:update_Mest_practical}\n  \\Mest_k\n  =\n  \\max \\bigg\\{\n    \\Mest_{k-1},\\ \n    &\n    \\frac{\n      12\n      \\prn*{\n        f(y_k) - f(x_k) \n        - \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n      }\n    }{\\norm*{y_k - x_k}^3},\\\\\n    &\n    \\frac{\n      \\norm*{\n        \\nabla f(y_k)\n        + \\theta_k \\nabla f(x_{k-1})\n        - (1 + \\theta_k) \\nabla f (x_k)\n      }\n    }{\\theta_k \\norm*{x_k - x_{k-1}}^2}\n  \\bigg\\}\n\\end{align}\nbecause $x_k$, $y_k$, and $\\theta_k$ are already in hand.\nIn other words, our $\\theta_k$ in \\cref{eq:def_theta_k} does not depend on the estimate $M_k$ of $M_f$, which facilitates computation of $M_k$.\nThus, we can estimate $M_f$ without backtracking, thereby simplifying the algorithm and improving its efficiency.\n\n","start":48153,"end":49056}]},{"id":"S018","semantic_type":"algorithm","kind":"algorithm","labels":["eq:update_Mest"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"强化 Hessian 估计规则","source_units":[{"raw_tex":"Instead of \\cref{eq:update_Mest_practical}, we can use\n\\begin{align}\n  \\label{eq:update_Mest}\n  \\Mest_k\n  =\n  \\max \\bigg\\{\n    \\Mest_{k-1},\\ \n    &\n    \\frac{\n      12\n      \\prn*{\n        f(y_k) - f(x_k) \n        - \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n      }\n    }{\\norm*{y_k - x_k}^3}\n    ,\\\\\n    &\n    \\frac{\n      \\norm*{\n        \\nabla f(y_k)\n        + \\theta_k \\nabla f(x_{k-1})\n        - (1 + \\theta_k) \\nabla f (x_k)\n      }\n    }{\\theta_k \\norm*{x_k - x_{k-1}}^2},\\\\\n    &\n    \\frac{\n      16\n      \\prn*{\n        Z_k^2 \\norm*{ \\nabla f (\\bar y_k) }\n        - Z_k \\Lest \\norm*{x_k - x_{k-1}}\n      }\n    }{(k-1) (k+5)^2 S_k}\n  \\bigg\\}.\n\\end{align}\nin the algorithm.\\footnote{\n  For $k = 1$, \\Cref{eq:update_Mest} takes the form $M_1 = \\max \\set{M_0, \\ast, \\ast, \\frac{0}{0}}$, which should be treated as $M_1 = \\max \\set{M_0, \\ast, \\ast}$.\n}\n","start":49056,"end":49933}]},{"id":"S019","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:grad_ybark_upperbound_Mestk"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"强化规则强制的平均梯度界","source_units":[{"raw_tex":"This replacement ensures that\n\\begin{align}\n  \\norm*{ \\nabla f (\\bar y_k) }\n  \\leq\n  \\frac{\\Lest}{Z_k} \\norm*{x_k - x_{k-1}}\n  + \\frac{(k-1) (k+5)^2}{16 Z_k^2} \\Mest_k S_k\n  \\label{eq:grad_ybark_upperbound_Mestk}\n\\end{align}\nand will lead to a smaller complexity bound with lessened dependence on $M_0$ and $M_f$, as will be confirmed later in \\cref{thm:complexity}.\n","start":49933,"end":50300}]},{"id":"S021","semantic_type":"definition","kind":"definition","labels":["eq:def_potential"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"含梯度校正的势函数","source_units":[{"raw_tex":"To obtain the descent condition~\\cref{eq:descent_condition}, we first evaluate the decrease for one iteration.\nHowever, a common difficulty arises when analyzing AGD: the objective function value does not necessarily decrease monotonically.\nTo deal with the problem, we introduce a potential function related to $f(x_k)$ and show that it is \\emph{nearly} decreasing.\nThe potential function $\\Phi_k$ is defined by\n\\begin{align}\n  \\Phi_k\n  \\coloneqq\n  f(x_k)\n  + \\frac{\\theta_k^2}{2} \\prn*{\n    \\inner*{\\nabla f(x_{k-1})}{x_k - x_{k-1}}\n    + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_{k-1})}^2\n    + \\Lest \\norm*{x_k - x_{k-1}}^2\n  },\n  \\label{eq:def_potential}\n\\end{align}\ninspired by \\cref{eq:hint_potential} used in the proof.\nThe value $\\Phi_k$ decreases when $\\norm*{x_k - x_{k-1}}$ is small enough as the following lemma shows.\n","start":51624,"end":52451}]},{"id":"S022","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:condition_continue_k-1"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"上一迭代未重启的约束","source_units":[{"raw_tex":"To evaluate the terms of $\\norm*{x_k - x_{k-1}}^3$ and $\\norm*{x_k - x_{k-1}}^4$ in \\cref{lem:potential_decrease_iteration} in these calculations, we use \n\\begin{align}\n  \\Mest_{k-1}^2 S_{k-1}\n  \\leq\n  \\Lest^2 (1 - \\theta_{k-1})^5\n  =\n  \\frac{\\Lest^2}{k^5},\n  \\label{eq:condition_continue_k-1}\n\\end{align}\nwhich follows from the fact that the epoch did not end at iteration $k-1$.\n","start":53349,"end":53730}]},{"id":"S023","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq:theta_increase_not_too_fast"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"加速参数的增长约束","source_units":[{"raw_tex":"\nNow, we prove \\cref{lem:potential_decrease_iteration} and use it to prove \\cref{lem:decrease_epoch}.\nThe following proofs do not use the specific form \\cref{eq:def_theta_k} of $\\theta_k$ but rather a more general condition on $\\theta_k$,\n\\begin{align}\n  \\theta_{k+1}^2 \\leq \\theta_k \\leq \\theta_{k+1},\n  \\label{eq:theta_increase_not_too_fast}\n\\end{align}\nwhich is easily verified for all $k \\geq 0$ under $\\theta_0 \\coloneqq \\theta_1^2$ and \\cref{eq:def_theta_k} for $k \\geq 1$.\n","start":53730,"end":54210}]},{"id":"S024","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"统一估计上界","source_units":[{"raw_tex":"\\subsection{Upper bound on gradient norm}\nLet $\\Lmax$ and $\\Mmax$ be the upper bounds on $L$ and $M_k$ given in \\cref{prop:L_upperbound,prop:Mk_upperbound}: $\\Lmax \\coloneqq \\max\\set{\\Linit, \\alpha \\Ltrue}$ and $\\Mmax \\coloneqq \\max\\set{M_0, \\Mtrue}$.\nThe following lemma provides upper bounds on the gradient norm.\n","start":70200,"end":70516}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","labels":["lem:gradient_jensen"],"member_ids":["S026","S078","S079","S080","S081"],"declared_external":false,"source_only":false,"title":"梯度的 Jensen 型不等式","source_units":[{"raw_tex":"\\begin{lemma}[Jensen-type inequality for gradients]\n  \\label{lem:gradient_jensen}\n  Under \\cref{asm:hessian_lip}, the following holds for any $z_1, \\dots, z_n \\in \\R^d$ and $\\lambda_1,\\dots,\\lambda_n \\geq 0$ such that $\\sum_{i=1}^n \\lambda_i = 1$:\n  \\begin{align}\n    \\norm*{\n      \\nabla f \\prn*{ \\sum_{i=1}^n \\lambda_i z_i }\n      - \\sum_{i=1}^n \\lambda_i \\nabla f \\prn*{ z_i }\n    }\n    \\leq\n    \\frac{\\Mtrue}{2} \\sum_{1 \\leq i < j \\leq n} \\lambda_i \\lambda_j \\norm*{z_i - z_j}^2.\n  \\end{align}\n\\end{lemma}","start":35516,"end":36025},{"raw_tex":"\\begin{proof}[Proof of \\cref{lem:gradient_jensen}]\n  To simplify the notation, let $\\bar z \\coloneqq \\sum_{i=1}^n \\lambda_i z_i$.\n  Taylor's theorem gives\n  \\[\n    \\nabla f(z_i) - \\nabla f(\\bar z)\n    =\n    \\nabla^2 f(\\bar z) (z_i - \\bar z)\n    + \\int_0^1 \\prn*{ \\nabla^2 f(\\bar z + t (z_i - \\bar z)) - \\nabla^2 f(\\bar z) } (z_i - \\bar z) dt,\n  \\]\n  and its weighted average gives\n  \\begin{align}\n    \\sum_{i=1}^n \\lambda_i \\nabla f(z_i) - \\nabla f(\\bar z)\n    &=\n    \\sum_{i=1}^n \\lambda_i\n    \\int_0^1 \\prn*{ \\nabla^2 f(\\bar z + t (z_i - \\bar z)) - \\nabla^2 f(\\bar z) } (z_i - \\bar z) dt.\n  \\end{align}\n  Therefore, we obtain\n  \\begin{alignat}{2}\n    \\norm*{\n      \\sum_{i=1}^n \\lambda_i \\nabla f(z_i) - \\nabla f(\\bar z)\n    }\n    &\\leq\n    \\sum_{i=1}^n \\lambda_i\n    \\int_0^1 \\norm*{ \\nabla^2 f(\\bar z + t (z_i - \\bar z)) - \\nabla^2 f(\\bar z) } \\norm*{z_i - \\bar z} dt\\\\\n    &\\leq\n    \\sum_{i=1}^n \\lambda_i\n    \\int_0^1 M_f t \\norm*{z_i - \\bar z}^2 dt\n    \\quad\\by{\\cref{asm:hessian_lip}}\\\\\n    &=\n    \\frac{M_f}{2} \\sum_{i=1}^n \\lambda_i \\norm*{z_i - \\bar z}^2.\n  \\end{alignat}\n  Furthermore, elementary algebra shows that $\\sum_{i=1}^n \\lambda_i \\norm*{z_i - \\bar z}^2 = \\sum_{1 \\leq i < j \\leq n} \\lambda_i \\lambda_j \\|z_i - z_j\\|^2$ as\n  \\begin{align}\n    \\sum_{1 \\leq i < j \\leq n} \\lambda_i \\lambda_j \\norm*{z_i - z_j}^2\n    &= \n    \\frac{1}{2} \\sum_{i,j = 1}^n \\lambda_i \\lambda_j \\norm*{z_i - z_j}^2\\\\\n    &= \n    \\frac{1}{2} \\sum_{i,j = 1}^n \\lambda_i \\lambda_j \\norm*{z_i}^2\n    + \\frac{1}{2} \\sum_{i,j = 1}^n \\lambda_i \\lambda_j \\norm*{z_j}^2\n    - \\sum_{i,j = 1}^n \\inner{\\lambda_i z_i}{\\lambda_j z_j}\\\\\n    &= \n    \\sum_{i=1}^n \\lambda_i \\norm*{z_i}^2 - \\norm*{\\bar z}^2\n    =\n    \\sum_{i=1}^n \\lambda_i \\norm*{z_i - \\bar z}^2,\n  \\end{align}\n  which completes the proof.\n\\end{proof}","start":106558,"end":108357},{"raw_tex":"  To simplify the notation, let $\\bar z \\coloneqq \\sum_{i=1}^n \\lambda_i z_i$.\n  Taylor's theorem gives\n  \\[\n    \\nabla f(z_i) - \\nabla f(\\bar z)\n    =\n    \\nabla^2 f(\\bar z) (z_i - \\bar z)\n    + \\int_0^1 \\prn*{ \\nabla^2 f(\\bar z + t (z_i - \\bar z)) - \\nabla^2 f(\\bar z) } (z_i - \\bar z) dt,\n  \\]\n  and its weighted average gives\n  \\begin{align}\n    \\sum_{i=1}^n \\lambda_i \\nabla f(z_i) - \\nabla f(\\bar z)\n    &=\n    \\sum_{i=1}^n \\lambda_i\n    \\int_0^1 \\prn*{ \\nabla^2 f(\\bar z + t (z_i - \\bar z)) - \\nabla^2 f(\\bar z) } (z_i - \\bar z) dt.\n  \\end{align}\n","start":106609,"end":107163},{"raw_tex":"  Therefore, we obtain\n  \\begin{alignat}{2}\n    \\norm*{\n      \\sum_{i=1}^n \\lambda_i \\nabla f(z_i) - \\nabla f(\\bar z)\n    }\n    &\\leq\n    \\sum_{i=1}^n \\lambda_i\n    \\int_0^1 \\norm*{ \\nabla^2 f(\\bar z + t (z_i - \\bar z)) - \\nabla^2 f(\\bar z) } \\norm*{z_i - \\bar z} dt\\\\\n    &\\leq\n    \\sum_{i=1}^n \\lambda_i\n    \\int_0^1 M_f t \\norm*{z_i - \\bar z}^2 dt\n    \\quad\\by{\\cref{asm:hessian_lip}}\\\\\n    &=\n    \\frac{M_f}{2} \\sum_{i=1}^n \\lambda_i \\norm*{z_i - \\bar z}^2.\n  \\end{alignat}\n","start":107163,"end":107641},{"raw_tex":"  Furthermore, elementary algebra shows that $\\sum_{i=1}^n \\lambda_i \\norm*{z_i - \\bar z}^2 = \\sum_{1 \\leq i < j \\leq n} \\lambda_i \\lambda_j \\|z_i - z_j\\|^2$ as\n  \\begin{align}\n    \\sum_{1 \\leq i < j \\leq n} \\lambda_i \\lambda_j \\norm*{z_i - z_j}^2\n    &= \n    \\frac{1}{2} \\sum_{i,j = 1}^n \\lambda_i \\lambda_j \\norm*{z_i - z_j}^2\\\\\n    &= \n    \\frac{1}{2} \\sum_{i,j = 1}^n \\lambda_i \\lambda_j \\norm*{z_i}^2\n    + \\frac{1}{2} \\sum_{i,j = 1}^n \\lambda_i \\lambda_j \\norm*{z_j}^2\n    - \\sum_{i,j = 1}^n \\inner{\\lambda_i z_i}{\\lambda_j z_j}\\\\\n    &= \n    \\sum_{i=1}^n \\lambda_i \\norm*{z_i}^2 - \\norm*{\\bar z}^2\n    =\n    \\sum_{i=1}^n \\lambda_i \\norm*{z_i - \\bar z}^2,\n  \\end{align}\n  which completes the proof.\n","start":107641,"end":108346}]},{"id":"S027","semantic_type":"lemma","kind":"lemma","labels":["lem:trapezoidal_rule_error"],"member_ids":["S027","S082","S083","S084"],"declared_external":false,"source_only":false,"title":"梯形公式的三次误差界","source_units":[{"raw_tex":"\\begin{lemma}[Error bound for trapezoidal rule]\n  \\label{lem:trapezoidal_rule_error}\n  Under \\cref{asm:hessian_lip}, the following holds for any $x, y \\in \\R^d$:\n  \\begin{align}\n    f(x) - f(y)\n    &\\leq\n    \\frac{1}{2} \\inner*{\\nabla f(x) + \\nabla f(y)}{x - y}\n    + \\frac{\\Mtrue}{12} \\norm*{x - y}^3.\n  \\end{align}\n\\end{lemma}","start":36026,"end":36354},{"raw_tex":"\\begin{proof}[Proof of \\cref{lem:trapezoidal_rule_error}]\n  We obtain the desired result as follows:\n  \\begin{alignat}{2}\n    &\\mathInd\n    % \\abs*{\n      f(x) - f(y)\n      - \\frac{1}{2} \\inner*{\\nabla f(x) + \\nabla f(y)}{x - y}\\\\\n    % }\\\\\n    &=\n    % \\abs*{\n      \\int_0^1 \\inner*{\\nabla f(t x + (1 - t) y)}{x - y} dt\n      - \\frac{1}{2} \\inner*{\\nabla f(x) + \\nabla f(y)}{x - y}\n    % }\n    &\\quad&\\by{Taylor's\\\\theorem}\\\\\n    &=\n    % \\abs*{\n      \\int_0^1 \\inner[\\Big]{\\nabla f(t x + (1 - t) y) - t \\nabla f(x) - (1 - t) \\nabla f(y)}{x - y} dt\\\\\n    % }\\\\\n    &\\leq\n    \\int_0^1 \\norm[\\Big]{\\nabla f(t x + (1 - t) y) - t \\nabla f(x) - (1 - t) \\nabla f(y)} \\norm{x - y} dt\\\\\n    &\\leq\n    \\frac{\\Mtrue}{2} \\int_0^1 t (1 - t) \\norm{x - y}^3 dt\n    =\n    \\frac{\\Mtrue}{12} \\norm{x - y}^3\n    &\\quad&\\by{\\cref{lem:gradient_jensen}}.\n  \\end{alignat}\n\\end{proof}","start":108360,"end":109222},{"raw_tex":"  We obtain the desired result as follows:\n  \\begin{alignat}{2}\n    &\\mathInd\n    % \\abs*{\n      f(x) - f(y)\n      - \\frac{1}{2} \\inner*{\\nabla f(x) + \\nabla f(y)}{x - y}\\\\\n    % }\\\\\n    &=\n    % \\abs*{\n      \\int_0^1 \\inner*{\\nabla f(t x + (1 - t) y)}{x - y} dt\n      - \\frac{1}{2} \\inner*{\\nabla f(x) + \\nabla f(y)}{x - y}\n    % }\n    &\\quad&\\by{Taylor's\\\\theorem}\\\\\n    &=\n    % \\abs*{\n      \\int_0^1 \\inner[\\Big]{\\nabla f(t x + (1 - t) y) - t \\nabla f(x) - (1 - t) \\nabla f(y)}{x - y} dt\\\\\n    % }\\\\\n","start":108418,"end":108922},{"raw_tex":"    &\\leq\n    \\int_0^1 \\norm[\\Big]{\\nabla f(t x + (1 - t) y) - t \\nabla f(x) - (1 - t) \\nabla f(y)} \\norm{x - y} dt\\\\\n    &\\leq\n    \\frac{\\Mtrue}{2} \\int_0^1 t (1 - t) \\norm{x - y}^3 dt\n    =\n    \\frac{\\Mtrue}{12} \\norm{x - y}^3\n    &\\quad&\\by{\\cref{lem:gradient_jensen}}.\n  \\end{alignat}\n","start":108922,"end":109211}]},{"id":"S028","semantic_type":"proposition","kind":"proposition","labels":["prop:L_upperbound"],"member_ids":["S028","S029"],"declared_external":false,"source_only":false,"title":"梯度 Lipschitz 估计的上界","source_units":[{"raw_tex":"\\begin{proposition}\n  \\label{prop:L_upperbound}\n  Suppose that \\cref{asm:gradient_lip} holds.\n  Then, the following is true throughout \\cref{alg:proposed_agd}: $\\Lest \\leq \\max\\set{\\Linit, \\alpha \\Ltrue}$.\n\\end{proposition}","start":51077,"end":51300},{"raw_tex":"This proposition immediately follows from the following lemma.\n","start":51301,"end":51364}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","labels":["lem:decrease_epoch"],"member_ids":["S030","S038","S039","S040","S041"],"declared_external":false,"source_only":false,"title":"充分大的估计保证 epoch 下降","source_units":[{"raw_tex":"\\begin{lemma}\n  \\label{lem:decrease_epoch}\n  Suppose that \\cref{asm:gradient_lip} holds.\n  During epochs with $\\Lest \\geq \\Ltrue$, condition~\\cref{eq:descent_condition} always holds.\n\\end{lemma}","start":51364,"end":51558},{"raw_tex":"\\begin{proof}[Proof of \\cref{lem:decrease_epoch}]\n  Note that\n  \\begin{align}\n    \\sum_{i=1}^k \\norm*{x_i - x_{i-1}}^3\n    &\\leq\n    S_k^{3/2},\\quad\n    \\sum_{i=1}^k \\norm*{x_i - x_{i-1}}^4\n    \\leq\n    S_k^2\n    \\label{eq:sum_xk_diff_power34_upperbound}\n  \\end{align}\n  by the monotonicity of $\\ell_p$-norms, i.e., $\\norm{a}_p \\geq \\norm{a}_q$ for $1 \\leq p \\leq q$.\n  In addition, we have\n  \\begin{align}\n    \\Phi_k - f(x_k)\n    &=\n    \\frac{\\theta_k^2}{2} \\prn*{\n      \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_{k-1}) + L (x_k - x_{k-1})}^2\n      + \\frac{\\Lest}{2} \\norm*{x_k - x_{k-1}}^2\n    }\n    \\geq\n    0,\n    \\label{eq:potential_lowerbound}\\\\\n    \\Phi_0 - f(x_0)\n    &=\n    \\frac{\\theta_0^2}{4 \\Lest} \\norm*{\\nabla f(x_0)}^2\n    \\label{eq:potential_obj_0}\n  \\end{align}\n  since $x_{-1} = x_0$.\n\n  Summing \\cref{lem:potential_decrease_iteration} over $k$ and telescoping yields\n  \\begin{align}\n    &\\mathInd\n    \\Phi_k - \\Phi_0\n    \\label{eq:potential_decrease_epoch}\\\\\n    &\\leq\n    \\sum_{i=0}^{k-1}\n    \\bigg(\n      \\frac{\\theta_{i+1}^2 + \\theta_i - 2}{4} \\Lest \\norm*{x_{i+1} - x_i}^2\n      + \\frac{7 \\theta_i^2}{12} \\Mest_i \\norm*{x_i - x_{i-1}}^3\\\\\n      &\\qquad\\qquad\n      + \\frac{\\theta_i^3}{4 \\Lest} \\Mest_i^2 \\norm*{x_i - x_{i-1}}^4\n      - \\frac{\\theta_i^2}{4 \\Lest} \\norm*{\\nabla f(x_i)}^2\n    \\bigg)\\\\\n    &\\leq\n    \\frac{\\theta_k^2 + \\theta_{k-1} - 2}{4} \\Lest S_k\n    + \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} S_{k-1}^{3/2} \n    + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}^2\n    - \\frac{\\theta_0^2}{4 \\Lest} \\norm*{\\nabla f(x_0)}^2,\n  \\end{align}\n  where we have used \\cref{eq:theta_increase_not_too_fast}, \\cref{eq:sum_xk_diff_power34_upperbound}, and $\\Mest_i \\leq \\Mest_{k-1}$.\n  Combining \\cref{eq:potential_lowerbound,eq:potential_obj_0,eq:potential_decrease_epoch} yields\n  \\begin{align}\n    f(x_k) - f(x_0)\n    \\leq\n    \\frac{\\theta_k^2 + \\theta_{k-1} - 2}{4} \\Lest S_k\n    + \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} S_{k-1}^{3/2} \n    + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}^2.\n  \\end{align}\n  Furthermore, we bound the last two terms as\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} S_{k-1}^{3/2} \n    + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}^2\\\\\n    &\\leq\n    \\prn*{\n      \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} \\sqrt{S_{k-1}}\n      + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}\n    }\n    S_k\n    &\\quad&\\by{$S_{k-1} \\leq S_k$}\\\\\n    &\\leq\n    \\prn*{\n      \\frac{7}{12} \\theta_{k-1}^2 (1 - \\theta_{k-1})^{5/2}\n      + \\frac{1}{4} \\theta_{k-1}^3 (1 - \\theta_{k-1})^5\n    }\n    \\Lest S_k\n    &\\quad&\\by{\\cref{eq:condition_continue_k-1}}\\\\\n    &\\leq\n    \\prn*{\n      \\frac{7}{12} + \\frac{1}{4}\n    }\n    \\theta_{k-1}^2 (1 - \\theta_{k-1})^2\n    \\Lest S_k\n    &\\quad&\\by{$0 \\leq \\theta_{k-1} \\leq 1$}\\\\\n    &\\leq\n    \\frac{1}{4}\n    \\theta_{k-1} (1 - \\theta_{k-1})\n    \\Lest S_k\n    &\\quad&\\by{$0 \\leq \\theta_{k-1} \\leq 1$},\n  \\end{alignat}\n  which results in\n  \\begin{alignat}{2}\n    f(x_k) - f(x_0)\n    &\\leq\n    \\frac{\\theta_k^2 + \\theta_{k-1} - 2}{4} \\Lest S_k\n    + \\frac{1}{4} \\theta_{k-1} (1 - \\theta_{k-1}) \\Lest S_k\\\\\n    &=\n    - \\frac{1 - \\theta_k}{2} \\Lest S_k\n    + \\frac{(1 - \\theta_k)^2 - (1 - \\theta_{k-1})^2}{4} \\Lest S_k\\\\\n    &\\leq\n    - \\frac{1 - \\theta_k}{2} \\Lest S_k\n    &\\quad&\\by{\\cref{eq:theta_increase_not_too_fast}}\n  \\end{alignat}\n  and concludes the proof.\n\\end{proof}","start":61293,"end":64695},{"raw_tex":"  Note that\n  \\begin{align}\n    \\sum_{i=1}^k \\norm*{x_i - x_{i-1}}^3\n    &\\leq\n    S_k^{3/2},\\quad\n    \\sum_{i=1}^k \\norm*{x_i - x_{i-1}}^4\n    \\leq\n    S_k^2\n    \\label{eq:sum_xk_diff_power34_upperbound}\n  \\end{align}\n  by the monotonicity of $\\ell_p$-norms, i.e., $\\norm{a}_p \\geq \\norm{a}_q$ for $1 \\leq p \\leq q$.\n  In addition, we have\n  \\begin{align}\n    \\Phi_k - f(x_k)\n    &=\n    \\frac{\\theta_k^2}{2} \\prn*{\n      \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_{k-1}) + L (x_k - x_{k-1})}^2\n      + \\frac{\\Lest}{2} \\norm*{x_k - x_{k-1}}^2\n    }\n    \\geq\n    0,\n    \\label{eq:potential_lowerbound}\\\\\n    \\Phi_0 - f(x_0)\n    &=\n    \\frac{\\theta_0^2}{4 \\Lest} \\norm*{\\nabla f(x_0)}^2\n    \\label{eq:potential_obj_0}\n  \\end{align}\n  since $x_{-1} = x_0$.\n","start":61343,"end":62090},{"raw_tex":"  Summing \\cref{lem:potential_decrease_iteration} over $k$ and telescoping yields\n  \\begin{align}\n    &\\mathInd\n    \\Phi_k - \\Phi_0\n    \\label{eq:potential_decrease_epoch}\\\\\n    &\\leq\n    \\sum_{i=0}^{k-1}\n    \\bigg(\n      \\frac{\\theta_{i+1}^2 + \\theta_i - 2}{4} \\Lest \\norm*{x_{i+1} - x_i}^2\n      + \\frac{7 \\theta_i^2}{12} \\Mest_i \\norm*{x_i - x_{i-1}}^3\\\\\n      &\\qquad\\qquad\n      + \\frac{\\theta_i^3}{4 \\Lest} \\Mest_i^2 \\norm*{x_i - x_{i-1}}^4\n      - \\frac{\\theta_i^2}{4 \\Lest} \\norm*{\\nabla f(x_i)}^2\n    \\bigg)\\\\\n    &\\leq\n    \\frac{\\theta_k^2 + \\theta_{k-1} - 2}{4} \\Lest S_k\n    + \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} S_{k-1}^{3/2} \n    + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}^2\n    - \\frac{\\theta_0^2}{4 \\Lest} \\norm*{\\nabla f(x_0)}^2,\n  \\end{align}\n  where we have used \\cref{eq:theta_increase_not_too_fast}, \\cref{eq:sum_xk_diff_power34_upperbound}, and $\\Mest_i \\leq \\Mest_{k-1}$.\n  Combining \\cref{eq:potential_lowerbound,eq:potential_obj_0,eq:potential_decrease_epoch} yields\n  \\begin{align}\n    f(x_k) - f(x_0)\n    \\leq\n    \\frac{\\theta_k^2 + \\theta_{k-1} - 2}{4} \\Lest S_k\n    + \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} S_{k-1}^{3/2} \n    + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}^2.\n  \\end{align}\n","start":62091,"end":63335},{"raw_tex":"  Furthermore, we bound the last two terms as\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} S_{k-1}^{3/2} \n    + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}^2\\\\\n    &\\leq\n    \\prn*{\n      \\frac{7 \\theta_{k-1}^2}{12} \\Mest_{k-1} \\sqrt{S_{k-1}}\n      + \\frac{\\theta_{k-1}^3}{4 \\Lest} \\Mest_{k-1}^2 S_{k-1}\n    }\n    S_k\n    &\\quad&\\by{$S_{k-1} \\leq S_k$}\\\\\n    &\\leq\n    \\prn*{\n      \\frac{7}{12} \\theta_{k-1}^2 (1 - \\theta_{k-1})^{5/2}\n      + \\frac{1}{4} \\theta_{k-1}^3 (1 - \\theta_{k-1})^5\n    }\n    \\Lest S_k\n    &\\quad&\\by{\\cref{eq:condition_continue_k-1}}\\\\\n    &\\leq\n    \\prn*{\n      \\frac{7}{12} + \\frac{1}{4}\n    }\n    \\theta_{k-1}^2 (1 - \\theta_{k-1})^2\n    \\Lest S_k\n    &\\quad&\\by{$0 \\leq \\theta_{k-1} \\leq 1$}\\\\\n    &\\leq\n    \\frac{1}{4}\n    \\theta_{k-1} (1 - \\theta_{k-1})\n    \\Lest S_k\n    &\\quad&\\by{$0 \\leq \\theta_{k-1} \\leq 1$},\n  \\end{alignat}\n  which results in\n  \\begin{alignat}{2}\n    f(x_k) - f(x_0)\n    &\\leq\n    \\frac{\\theta_k^2 + \\theta_{k-1} - 2}{4} \\Lest S_k\n    + \\frac{1}{4} \\theta_{k-1} (1 - \\theta_{k-1}) \\Lest S_k\\\\\n    &=\n    - \\frac{1 - \\theta_k}{2} \\Lest S_k\n    + \\frac{(1 - \\theta_k)^2 - (1 - \\theta_{k-1})^2}{4} \\Lest S_k\\\\\n    &\\leq\n    - \\frac{1 - \\theta_k}{2} \\Lest S_k\n    &\\quad&\\by{\\cref{eq:theta_increase_not_too_fast}}\n  \\end{alignat}\n  and concludes the proof.\n","start":63335,"end":64684}]},{"id":"S031","semantic_type":"lemma","kind":"lemma","labels":["eq:potential_decrease_iteration","lem:potential_decrease_iteration"],"member_ids":["S031","S032","S033","S034","S035","S036","S037"],"declared_external":false,"source_only":false,"title":"势函数单步下降界","source_units":[{"raw_tex":"\\begin{lemma}\n  \\label{lem:potential_decrease_iteration}\n  Under \\cref{asm:gradient_lip} and $\\Lest \\geq \\Ltrue$, the following holds for all $k \\geq 0$:\n  \\begin{align}\n    \\Phi_{k+1} - \\Phi_k\n    &\\leq\n    \\frac{\\theta_{k+1}^2 + \\theta_k - 2}{4} \\Lest \\norm*{x_{k+1} - x_k}^2\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\\\\\n    &\\quad\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n    \\label{eq:potential_decrease_iteration}\n  \\end{align}\n\\end{lemma}","start":52451,"end":52999},{"raw_tex":"\\begin{proof}[Proof of \\cref{lem:potential_decrease_iteration}]\n  In the proof, let\n  \\begin{align}\n    P_k\n    &\\coloneqq\n    \\inner*{\\nabla f(x_{k-1})}{x_k - x_{k-1}}\n  \\end{align}\n  to simplify the notation.\n  From \\cref{lem:eq:property_gradlip_obj,eq:update_xy}, we have\n  \\begin{align}\n    f(x_{k+1}) - f(y_k)\n    &\\leq\n    \\inner*{\\nabla f(y_k)}{x_{k+1} - y_k}\n    + \\frac{\\Lest}{2} \\norm*{x_{k+1} - y_k}^2\n    = \n    - \\frac{1}{2 \\Lest} \\norm*{\\nabla f(y_k)}^2.\n  \\end{align}\n  Summing this bound and \\cref{eq:fy_fx_diff_upperbound_Mestk} yields\n  \\begin{alignat}{2}\n    &\\mathInd\n    f(x_{k+1}) - f(x_k)\n    \\label{eq:descent_f_first}\\\\\n    &\\leq\n    \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n    + \\frac{\\Mest_k}{12} \\norm*{y_k - x_k}^3\n    - \\frac{1}{2 \\Lest} \\norm*{\\nabla f(y_k)}^2\\\\\n    &\\leq\n    \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n    + \\frac{\\theta_k^2 \\Mest_k}{12} \\norm*{x_k - x_{k-1}}^3\n    - \\frac{1}{2 \\Lest} \\norm*{\\nabla f(y_k)}^2,\n  \\end{alignat}\n  where the last inequality uses \\cref{eq:update_xy} and $0 \\leq \\theta_k \\leq 1$.\n  To evaluate the first term on the right-hand side, we decompose it into four terms:\n  \\begin{align}\n    &\\mathInd\n    \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\\\\\n    &=\n    \\underbrace{2 \\inner*{\\nabla f(y_k)}{y_k - x_k}}_{\\text{(A)}}\n    {}+{} \\underbrace{\\theta_k \\inner*{\\nabla f(x_{k-1})}{y_k - x_k}}_{\\text{(B)}}\\\\[-0.8\\baselineskip]\n    &\\quad\n    - \\underbrace{\\theta_k \\inner*{\\nabla f(x_k)}{y_k - x_k}}_{\\text{(C)}}\n    {}-{} \\underbrace{\n        \\inner*{\n        \\nabla f(y_k)\n        + \\theta_k \\nabla f(x_{k-1})\n        - (1 + \\theta_k) \\nabla f (x_k)\n      }{y_k - x_k}\n    }_{\\text{(D)}}.\n  \\end{align}\n  Each term can be evaluated as\n  \\begin{alignat}{2}\n    \\text{(A)}\n    &=\n    \\frac{1}{\\Lest} \\norm*{\\nabla f(y_k)}^2\n    + \\Lest \\norm*{y_k - x_k}^2\n    - \\Lest \\norm*{(y_k - x_k) - \\frac{1}{\\Lest} \\nabla f(y_k)}^2\\\\\n    &=\n    \\frac{1}{\\Lest} \\norm*{\\nabla f(y_k)}^2\n    + \\theta_k^2 \\Lest \\norm*{x_k - x_{k-1}}^2\n    - \\Lest \\norm*{x_{k+1} - x_k}^2\n    &\\quad&\\by{\\cref{eq:update_xy}},\\\\\n    \\text{(B)}\n    &=\n    \\theta_k^2 \\inner*{\\nabla f(x_{k-1})}{x_k - x_{k-1}}\n    =\n    \\theta_k^2 P_k\n    &\\quad&\\by{\\cref{eq:update_xy}},\\\\\n    \\text{(C)}\n    &=\n    - \\theta_k P_{k+1}\n    + \\theta_k \\inner*{\\nabla f(x_k)}{x_{k+1} - y_k}\\\\\n    &=\n    - \\theta_k P_{k+1}\n    - \\frac{\\theta_k}{\\Lest} \\inner*{\\nabla f(x_k)}{\\nabla f(y_k)}\n    &\\quad&\\by{\\cref{eq:update_xy}},\\\\\n    \\text{(D)}\n    &\\leq\n    \\theta_k \\Mest_k \\norm*{x_k - x_{k-1}}^2 \\norm*{y_k - x_k}\n    &\\quad&\\by{\\cref{eq:grad_interpolation_error_Mestk}}\\\\\n    &=\n    \\theta_k^2 \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    &\\quad&\\by{\\cref{eq:update_xy}}.\n  \\end{alignat}\n  Plugging the evaluations into \\cref{eq:descent_f_first} results in\n  \\begin{align}\n    % &\\mathInd\n    f(x_{k+1}) - f(x_k)\n    \\label{eq:descent_f_second}\n    &\\leq\n    \\frac{\\theta_k^2 \\Lest}{2} \\norm*{x_k - x_{k-1}}^2\n    - \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2\n    + \\frac{\\theta_k^2}{2} P_k\n    - \\frac{\\theta_k}{2} P_{k+1}\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    - \\frac{\\theta_k}{2 \\Lest} \\inner*{\\nabla f(x_k)}{\\nabla f(y_k)}.\n  \\end{align}\n\n  Next, to bound the last term on the right-hand side, we use the following inequality from \\cref{eq:grad_interpolation_error_Mestk}:\n  \\begin{align}\n    \\norm*{\n      \\nabla f(y_k)\n      - (1 + \\theta_k) \\nabla f (x_k)\n    }\n    &\\leq\n    \\theta_k \\norm*{ \\nabla f(x_{k-1})}\n    + \n    \\theta_k \\Mest_k\n    \\norm*{x_k - x_{k-1}}^2.\n  \\end{align}\n  We square both sides as\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{\n      \\nabla f(y_k)\n      - (1 + \\theta_k) \\nabla f (x_k)\n    }^2\\\\\n    &=\n    \\norm*{\\nabla f(y_k)}^2\n    + (1 + \\theta_k)^2 \\norm*{\\nabla f (x_k)}^2\n    - 2 (1 + \\theta_k) \\inner*{\\nabla f (x_k)}{\\nabla f(y_k)}\\\\\n    &\\geq\n    (1 + \\theta_k)^2 \\norm*{\\nabla f (x_k)}^2\n    - 2 (1 + \\theta_k) \\inner*{\\nabla f (x_k)}{\\nabla f(y_k)}\n  \\end{alignat}\n  and\n  \\begin{alignat}{2}\n    \\theta_k^2 \\prn*{\n      \\norm*{ \\nabla f(x_{k-1})}\n      + \\Mest_k \\norm*{x_k - x_{k-1}}^2\n    }^2\n    &\\leq\n    \\theta_k (1 + \\theta_k) \\norm*{ \\nabla f(x_{k-1})}^2\\\\\n    &\\quad\n    + \\theta_k^2 (1 + \\theta_k) \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4,\n  \\end{alignat}\n  where we have used $(a + b)^2 \\leq (1 + 1/\\theta) a^2 + (1 + \\theta) b^2$ for $a, b, \\theta > 0$.\n  Rearranging the terms yields\n  \\[\n  % \\begin{align}\n    - \\inner*{\\nabla f (x_k)}{\\nabla f(y_k)}\n    \\leq\n    \\frac{\\theta_k}{2} \\norm*{ \\nabla f(x_{k-1})}^2\n    - \\frac{1 + \\theta_k}{2} \\norm*{\\nabla f (x_k)}^2\n    + \\frac{\\theta_k^2}{2} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4.\n  % \\end{align}\n  \\]\n  Plugging this bound into \\cref{eq:descent_f_second} results in\n  \\begin{align}\n    \\label{eq:hint_potential}\n    &\\mathInd\n    f(x_{k+1}) - f(x_k)\\\\\n    &\\leq\n    \\frac{\\theta_k^2}{2} \\prn*{\n      P_k\n      + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_{k-1})}^2\n      + \\Lest \\norm*{x_k - x_{k-1}}^2\n    }\\\\\n    &\\quad\n    -\n    \\frac{\\theta_k}{2} \\prn*{\n      P_{k+1} + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_k)}^2\n    }\n    - \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n  \\end{align}\n  This inequality can be rewritten with the potential function $\\Phi_k$ defined in \\cref{eq:def_potential} as\n  \\begin{align}\n    \\Phi_{k+1} - \\Phi_k\n    &\\leq\n    \\frac{\\theta_{k+1}^2 - \\theta_k}{2}\n    \\prn*{\n      P_{k+1}\n      + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_k)}^2\n    }\n    + \\frac{\\theta_{k+1}^2 - 1}{2} \\Lest \\norm*{x_{k+1} - x_k}^2\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n  \\end{align}\n  Finally, using $\\theta_{k+1}^2 - \\theta_k \\leq 0$ from \\cref{eq:theta_increase_not_too_fast} and\n  \\begin{align}\n    - P_{k+1}\n    =\n    - \\inner*{\\nabla f(x_k)}{x_{k+1} - x_k}\n    \\leq\n    \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_k)}^2\n    + \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2\n  \\end{align}\n  from Young's inequality ($\\abs{\\inner*{a}{b}} \\leq \\frac{1}{2} \\norm*{a}^2 + \\frac{1}{2} \\norm*{b}^2$) yields the desired result as\n  \\begin{align}\n    \\Phi_{k+1} - \\Phi_k\n    &\\leq\n    \\frac{\\theta_{k+1}^2 - \\theta_k}{2} \\prn*{- \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2}\n    + \\frac{\\theta_{k+1}^2 - 1}{2} \\Lest \\norm*{x_{k+1} - x_k}^2\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2\\\\\n    &=\n    \\frac{\\theta_{k+1}^2 + \\theta_k - 2}{4} \\Lest \\norm*{x_{k+1} - x_k}^2\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\\\\\n    &\\quad\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n  \\end{align}\n\\end{proof}","start":54211,"end":61291},{"raw_tex":"  From \\cref{lem:eq:property_gradlip_obj,eq:update_xy}, we have\n  \\begin{align}\n    f(x_{k+1}) - f(y_k)\n    &\\leq\n    \\inner*{\\nabla f(y_k)}{x_{k+1} - y_k}\n    + \\frac{\\Lest}{2} \\norm*{x_{k+1} - y_k}^2\n    = \n    - \\frac{1}{2 \\Lest} \\norm*{\\nabla f(y_k)}^2.\n  \\end{align}\n  Summing this bound and \\cref{eq:fy_fx_diff_upperbound_Mestk} yields\n  \\begin{alignat}{2}\n    &\\mathInd\n    f(x_{k+1}) - f(x_k)\n    \\label{eq:descent_f_first}\\\\\n    &\\leq\n    \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n    + \\frac{\\Mest_k}{12} \\norm*{y_k - x_k}^3\n    - \\frac{1}{2 \\Lest} \\norm*{\\nabla f(y_k)}^2\\\\\n    &\\leq\n    \\frac{1}{2} \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\n    + \\frac{\\theta_k^2 \\Mest_k}{12} \\norm*{x_k - x_{k-1}}^3\n    - \\frac{1}{2 \\Lest} \\norm*{\\nabla f(y_k)}^2,\n  \\end{alignat}\n  where the last inequality uses \\cref{eq:update_xy} and $0 \\leq \\theta_k \\leq 1$.\n","start":54422,"end":55311},{"raw_tex":"  To evaluate the first term on the right-hand side, we decompose it into four terms:\n  \\begin{align}\n    &\\mathInd\n    \\inner*{\\nabla f(y_k) + \\nabla f(x_k)}{y_k - x_k}\\\\\n    &=\n    \\underbrace{2 \\inner*{\\nabla f(y_k)}{y_k - x_k}}_{\\text{(A)}}\n    {}+{} \\underbrace{\\theta_k \\inner*{\\nabla f(x_{k-1})}{y_k - x_k}}_{\\text{(B)}}\\\\[-0.8\\baselineskip]\n    &\\quad\n    - \\underbrace{\\theta_k \\inner*{\\nabla f(x_k)}{y_k - x_k}}_{\\text{(C)}}\n    {}-{} \\underbrace{\n        \\inner*{\n        \\nabla f(y_k)\n        + \\theta_k \\nabla f(x_{k-1})\n        - (1 + \\theta_k) \\nabla f (x_k)\n      }{y_k - x_k}\n    }_{\\text{(D)}}.\n  \\end{align}\n  Each term can be evaluated as\n  \\begin{alignat}{2}\n    \\text{(A)}\n    &=\n    \\frac{1}{\\Lest} \\norm*{\\nabla f(y_k)}^2\n    + \\Lest \\norm*{y_k - x_k}^2\n    - \\Lest \\norm*{(y_k - x_k) - \\frac{1}{\\Lest} \\nabla f(y_k)}^2\\\\\n    &=\n    \\frac{1}{\\Lest} \\norm*{\\nabla f(y_k)}^2\n    + \\theta_k^2 \\Lest \\norm*{x_k - x_{k-1}}^2\n    - \\Lest \\norm*{x_{k+1} - x_k}^2\n    &\\quad&\\by{\\cref{eq:update_xy}},\\\\\n    \\text{(B)}\n    &=\n    \\theta_k^2 \\inner*{\\nabla f(x_{k-1})}{x_k - x_{k-1}}\n    =\n    \\theta_k^2 P_k\n    &\\quad&\\by{\\cref{eq:update_xy}},\\\\\n    \\text{(C)}\n    &=\n    - \\theta_k P_{k+1}\n    + \\theta_k \\inner*{\\nabla f(x_k)}{x_{k+1} - y_k}\\\\\n    &=\n    - \\theta_k P_{k+1}\n    - \\frac{\\theta_k}{\\Lest} \\inner*{\\nabla f(x_k)}{\\nabla f(y_k)}\n    &\\quad&\\by{\\cref{eq:update_xy}},\\\\\n    \\text{(D)}\n    &\\leq\n    \\theta_k \\Mest_k \\norm*{x_k - x_{k-1}}^2 \\norm*{y_k - x_k}\n    &\\quad&\\by{\\cref{eq:grad_interpolation_error_Mestk}}\\\\\n    &=\n    \\theta_k^2 \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    &\\quad&\\by{\\cref{eq:update_xy}}.\n  \\end{alignat}\n  Plugging the evaluations into \\cref{eq:descent_f_first} results in\n  \\begin{align}\n    % &\\mathInd\n    f(x_{k+1}) - f(x_k)\n    \\label{eq:descent_f_second}\n    &\\leq\n    \\frac{\\theta_k^2 \\Lest}{2} \\norm*{x_k - x_{k-1}}^2\n    - \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2\n    + \\frac{\\theta_k^2}{2} P_k\n    - \\frac{\\theta_k}{2} P_{k+1}\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    - \\frac{\\theta_k}{2 \\Lest} \\inner*{\\nabla f(x_k)}{\\nabla f(y_k)}.\n  \\end{align}\n","start":55311,"end":57454},{"raw_tex":"  Next, to bound the last term on the right-hand side, we use the following inequality from \\cref{eq:grad_interpolation_error_Mestk}:\n  \\begin{align}\n    \\norm*{\n      \\nabla f(y_k)\n      - (1 + \\theta_k) \\nabla f (x_k)\n    }\n    &\\leq\n    \\theta_k \\norm*{ \\nabla f(x_{k-1})}\n    + \n    \\theta_k \\Mest_k\n    \\norm*{x_k - x_{k-1}}^2.\n  \\end{align}\n  We square both sides as\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{\n      \\nabla f(y_k)\n      - (1 + \\theta_k) \\nabla f (x_k)\n    }^2\\\\\n    &=\n    \\norm*{\\nabla f(y_k)}^2\n    + (1 + \\theta_k)^2 \\norm*{\\nabla f (x_k)}^2\n    - 2 (1 + \\theta_k) \\inner*{\\nabla f (x_k)}{\\nabla f(y_k)}\\\\\n    &\\geq\n    (1 + \\theta_k)^2 \\norm*{\\nabla f (x_k)}^2\n    - 2 (1 + \\theta_k) \\inner*{\\nabla f (x_k)}{\\nabla f(y_k)}\n  \\end{alignat}\n  and\n  \\begin{alignat}{2}\n    \\theta_k^2 \\prn*{\n      \\norm*{ \\nabla f(x_{k-1})}\n      + \\Mest_k \\norm*{x_k - x_{k-1}}^2\n    }^2\n    &\\leq\n    \\theta_k (1 + \\theta_k) \\norm*{ \\nabla f(x_{k-1})}^2\\\\\n    &\\quad\n    + \\theta_k^2 (1 + \\theta_k) \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4,\n  \\end{alignat}\n  where we have used $(a + b)^2 \\leq (1 + 1/\\theta) a^2 + (1 + \\theta) b^2$ for $a, b, \\theta > 0$.\n  Rearranging the terms yields\n  \\[\n  % \\begin{align}\n    - \\inner*{\\nabla f (x_k)}{\\nabla f(y_k)}\n    \\leq\n    \\frac{\\theta_k}{2} \\norm*{ \\nabla f(x_{k-1})}^2\n    - \\frac{1 + \\theta_k}{2} \\norm*{\\nabla f (x_k)}^2\n    + \\frac{\\theta_k^2}{2} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4.\n  % \\end{align}\n  \\]\n","start":57455,"end":58914},{"raw_tex":"  Plugging this bound into \\cref{eq:descent_f_second} results in\n  \\begin{align}\n    \\label{eq:hint_potential}\n    &\\mathInd\n    f(x_{k+1}) - f(x_k)\\\\\n    &\\leq\n    \\frac{\\theta_k^2}{2} \\prn*{\n      P_k\n      + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_{k-1})}^2\n      + \\Lest \\norm*{x_k - x_{k-1}}^2\n    }\\\\\n    &\\quad\n    -\n    \\frac{\\theta_k}{2} \\prn*{\n      P_{k+1} + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_k)}^2\n    }\n    - \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n  \\end{align}\n  This inequality can be rewritten with the potential function $\\Phi_k$ defined in \\cref{eq:def_potential} as\n  \\begin{align}\n    \\Phi_{k+1} - \\Phi_k\n    &\\leq\n    \\frac{\\theta_{k+1}^2 - \\theta_k}{2}\n    \\prn*{\n      P_{k+1}\n      + \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_k)}^2\n    }\n    + \\frac{\\theta_{k+1}^2 - 1}{2} \\Lest \\norm*{x_{k+1} - x_k}^2\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n  \\end{align}\n","start":58914,"end":60150},{"raw_tex":"  Finally, using $\\theta_{k+1}^2 - \\theta_k \\leq 0$ from \\cref{eq:theta_increase_not_too_fast} and\n  \\begin{align}\n    - P_{k+1}\n    =\n    - \\inner*{\\nabla f(x_k)}{x_{k+1} - x_k}\n    \\leq\n    \\frac{1}{2 \\Lest} \\norm*{\\nabla f(x_k)}^2\n    + \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2\n  \\end{align}\n  from Young's inequality ($\\abs{\\inner*{a}{b}} \\leq \\frac{1}{2} \\norm*{a}^2 + \\frac{1}{2} \\norm*{b}^2$) yields the desired result as\n  \\begin{align}\n    \\Phi_{k+1} - \\Phi_k\n    &\\leq\n    \\frac{\\theta_{k+1}^2 - \\theta_k}{2} \\prn*{- \\frac{\\Lest}{2} \\norm*{x_{k+1} - x_k}^2}\n    + \\frac{\\theta_{k+1}^2 - 1}{2} \\Lest \\norm*{x_{k+1} - x_k}^2\\\\\n    &\\quad\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2\\\\\n    &=\n    \\frac{\\theta_{k+1}^2 + \\theta_k - 2}{4} \\Lest \\norm*{x_{k+1} - x_k}^2\n    + \\frac{7 \\theta_k^2}{12} \\Mest_k \\norm*{x_k - x_{k-1}}^3\\\\\n    &\\quad\n    + \\frac{\\theta_k^3}{4 \\Lest} \\Mest_k^2 \\norm*{x_k - x_{k-1}}^4\n    - \\frac{\\theta_k^2}{4 \\Lest} \\norm*{\\nabla f (x_k)}^2.\n  \\end{align}\n","start":60150,"end":61280}]},{"id":"S042","semantic_type":"proposition","kind":"proposition","labels":["prop:Mk_upperbound"],"member_ids":["S042","S043"],"declared_external":false,"source_only":false,"title":"Hessian 估计的统一上界","source_units":[{"raw_tex":"\\begin{proposition}\n  \\label{prop:Mk_upperbound}\n  Suppose that \\cref{asm:hessian_lip} holds.\n  Then, the following is true throughout \\cref{alg:proposed_agd}: \n  $\\Mest_k \\leq \\max\\set{M_0, \\Mtrue}$.\n\\end{proposition}","start":64788,"end":65006},{"raw_tex":"Because of the update rules \\cref{eq:update_Mest_practical} or \\cref{eq:update_Mest} of $M_k$, it suffices to show the inequalities obtained by replacing $\\Mest_k$ with $M_f$ in \\cref{eq:fy_fx_diff_upperbound_Mestk,eq:grad_interpolation_error_Mestk,eq:grad_ybark_upperbound_Mestk}.\nOne of them has already been obtained as \\cref{lem:trapezoidal_rule_error}.\nBelow we prove the other two; the following lemma gives the formal statement.\n","start":65007,"end":65443}]},{"id":"S044","semantic_type":"lemma","kind":"lemma","labels":["lem:grad_interpolation_error"],"member_ids":["S044","S046","S047"],"declared_external":false,"source_only":false,"title":"真实常数下的梯度插值界","source_units":[{"raw_tex":"\\begin{lemma}\n  \\label{lem:grad_interpolation_error}\n  Under \\cref{asm:hessian_lip}, the following hold for all $k \\geq 1$:\n  \\begin{align}\n    &\n    \\norm*{\n      \\nabla f(y_k)\n      + \\theta_k \\nabla f(x_{k-1})\n      - (1 + \\theta_k) \\nabla f (x_k)\n    }\n    \\leq\n    \\theta_k \\Mtrue \\norm*{x_k - x_{k-1}}^2,\n    \\label{eq:grad_interpolation_error_Mf}\\\\\n","start":65443,"end":65799},{"raw_tex":"\\begin{proof}[Proof of \\cref{eq:grad_interpolation_error_Mf}]\n  Note that $x_k = \\frac{1}{1 + \\theta_k} y_k + \\frac{\\theta_k}{1 + \\theta_k} x_{k-1}$ from \\cref{eq:update_xy}, and \\cref{lem:gradient_jensen} gives\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{\n      \\nabla f (x_k)\n      - \\frac{1}{1 + \\theta_k} \\nabla f(y_k)\n      - \\frac{\\theta_k}{1 + \\theta_k} \\nabla f(x_{k-1})\n    }\\\\\n    &\\leq\n    \\frac{\\Mtrue \\theta_k}{2 (1 + \\theta_k)^2} \\norm*{y_k - x_{k-1}}^2\\\\\n    &=\n    \\frac{\\Mtrue \\theta_k}{2} \\norm*{x_k - x_{k-1}}^2\n    \\leq\n    \\frac{\\Mtrue \\theta_k}{1 + \\theta_k} \\norm*{x_k - x_{k-1}}^2\n    &\\quad&\\by{\\cref{eq:update_xy} and $0 \\leq \\theta_k \\leq 1$}.\n  \\end{alignat}\n  Multiplying by $(1 + \\theta_k)$ concludes the proof.\n\\end{proof}","start":66135,"end":66890},{"raw_tex":"  Note that $x_k = \\frac{1}{1 + \\theta_k} y_k + \\frac{\\theta_k}{1 + \\theta_k} x_{k-1}$ from \\cref{eq:update_xy}, and \\cref{lem:gradient_jensen} gives\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{\n      \\nabla f (x_k)\n      - \\frac{1}{1 + \\theta_k} \\nabla f(y_k)\n      - \\frac{\\theta_k}{1 + \\theta_k} \\nabla f(x_{k-1})\n    }\\\\\n    &\\leq\n    \\frac{\\Mtrue \\theta_k}{2 (1 + \\theta_k)^2} \\norm*{y_k - x_{k-1}}^2\\\\\n    &=\n    \\frac{\\Mtrue \\theta_k}{2} \\norm*{x_k - x_{k-1}}^2\n    \\leq\n    \\frac{\\Mtrue \\theta_k}{1 + \\theta_k} \\norm*{x_k - x_{k-1}}^2\n    &\\quad&\\by{\\cref{eq:update_xy} and $0 \\leq \\theta_k \\leq 1$}.\n  \\end{alignat}\n  Multiplying by $(1 + \\theta_k)$ concludes the proof.\n","start":66197,"end":66879}]},{"id":"S045","semantic_type":"lemma","kind":"lemma","labels":["lem:grad_interpolation_error"],"member_ids":["S045","S048","S049","S050","S051","S052"],"declared_external":false,"source_only":false,"title":"真实常数下的平均梯度界","source_units":[{"raw_tex":"\\begin{lemma}\n  \\label{lem:grad_interpolation_error}\n  Under \\cref{asm:hessian_lip}, the following hold for all $k \\geq 1$:\n","start":65443,"end":65567},{"raw_tex":"    &\n    \\norm*{ \\nabla f (\\bar y_k) }\n    \\leq\n    \\frac{\\Lest}{Z_k} \\norm*{x_k - x_{k-1}}\n    + \\frac{(k-1) (k+5)^2}{16 Z_k^2} \\Mtrue S_k.\n    \\label{eq:grad_ybark_upperbound_Mf}\n  \\end{align}\n\\end{lemma}","start":65799,"end":66006},{"raw_tex":"\\begin{proof}[Proof of \\cref{eq:grad_ybark_upperbound_Mf}]\n  We have\n  \\begin{alignat}{2}\n    &\\mathInd\n    p_{k,i} \\nabla f(y_i)\n    =\n    p_{k,i} \\Lest (y_i - x_{i+1})\\\\\n    &=\n    p_{k,i} \\Lest \\prn*{\n      \\theta_i (x_i - x_{i-1})\n      - (x_{i+1} - x_i)\n    }\n    &\\quad&\\by{\\cref{eq:update_xy}}\\\\\n    &=\n    \\Lest \\prn*{\n      p_{k,i-1} (x_i - x_{i-1})\n      - p_{k,i} (x_{i+1} - x_i)\n    }\n    &\\quad&\\by{definition \\cref{eq:def_ybark} of $p_{k,i}$}.\n  \\end{alignat}\n  Summing this equality over $i$ yields\n  \\begin{align}\n    \\sum_{i=0}^{k-1}\n    p_{k,i} \\nabla f(y_i)\n    =\n    - \\Lest p_{k,k-1} (x_k - x_{k-1}),\n    \\label{eq:weighted_average_gradient_equation}\n  \\end{align}\n  where we have used $x_0 = x_{-1}$.\n  Now, we obtain\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{\n      \\nabla f (\\bar y_k)\n    }\n    \\label{eq:norm_grad_ybark_upperbound_pre}\\\\\n    &\\leq\n    \\norm*{\n      \\sum_{i=0}^{k-1} p_{k,i} \\nabla f \\prn*{ y_i }\n    }\n    + \\frac{\\Mtrue}{2} \\sum_{0 \\leq i < j < k} p_{k,i} p_{k,j} \\norm*{y_i - y_j}^2\n    &\\quad&\\by{\\cref{lem:gradient_jensen}}\\\\\n    &=\n    \\Lest p_{k,k-1} \\norm*{x_k - x_{k-1}}\n    + \\frac{\\Mtrue}{2} \\sum_{0 \\leq i < j < k} p_{k,i} p_{k,j} \\norm*{y_i - y_j}^2\n    &\\quad&\\by{\\cref{eq:weighted_average_gradient_equation}}\\\\\n    &\\leq\n    \\frac{\\Lest}{Z_k} \\norm*{x_k - x_{k-1}}\n    + \\frac{\\Mtrue}{2 Z_k^2} \\sum_{0 \\leq i < j < k} \\norm*{y_i - y_j}^2,\n  \\end{alignat}\n  where the last inequality follows from $p_{k,i} \\leq p_{k,k-1} = 1 / Z_k$ for all $0 \\leq i < k$.\n\n  Next, we bound the second term on the right-hand side.\n  Since for $0 \\leq i < j < k$,\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{y_i - y_j}\\\\\n    &\\leq\n    \\norm*{y_i - x_i}\n    + \\sum_{l=i+1}^{j-1} \\norm*{x_l - x_{l-1}}\n    + \\norm*{y_j - x_{j-1}}\n    &\\quad&\\by{the triangle inequality}\\\\\n    &=\n    \\norm*{x_i - x_{i-1}}\n    + \\sum_{l=i+1}^{j-1} \\norm*{x_l - x_{l-1}}\n    + 2 \\norm*{x_j - x_{j-1}}\n    &\\quad&\\by{\\cref{eq:update_xy} and $0 \\leq \\theta_k \\leq 1$}\\\\\n    &\\leq\n    \\prn*{\n      1^2 + \\sum_{l=i+1}^{j-1} 1^2 + 2^2\n    }^{1/2}\n    \\prn*{\n      \\sum_{l=i}^j \\norm*{x_l - x_{l-1}}^2\n    }^{1/2}\n    &\\quad&\\by{Cauchy--Schwarz}\\\\\n    &=\n    \\sqrt{j - i + 4}\n    \\prn*{\n      \\sum_{l=i}^j \\norm*{x_l - x_{l-1}}^2\n    }^{1/2},\n  \\end{alignat}\n  we have\n  \\begin{align}\n    \\sum_{0 \\leq i < j < k}\n    \\norm*{y_i - y_j}^2\n    &\\leq\n    \\sum_{0 \\leq i < j < k}\n    \\sum_{l=i}^j\n    (j - i + 4) \\norm*{x_l - x_{l-1}}^2\\\\\n    % &=\n    % \\sum_{0 \\leq i \\leq l \\leq j < k}\n    % (j - i + 4) \\norm*{x_l - x_{l-1}}^2\n    % - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &=\n    \\sum_{l=0}^{k-1}\n    \\prn*{\n      \\sum_{i=0}^l\n      \\sum_{j=l}^{k-1}\n      (j - i + 4) \n    }\n    \\norm*{x_l - x_{l-1}}^2\n    - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &=\n    \\frac{k + 7}{2}\n    \\sum_{l=0}^{k-1}\n    (l + 1) (k - l)\n    \\norm*{x_l - x_{l-1}}^2\n    - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &\\leq\n    \\frac{k + 7}{2}\n    \\sum_{l=0}^{k-1} \\frac{(k+1)^2}{4} \\norm*{x_l - x_{l-1}}^2\n    - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &=\n    \\frac{(k-1) (k+5)^2}{8}\n    \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\n    \\leq\n    \\frac{(k-1) (k+5)^2}{8} S_k.\n  \\end{align}\n  Plugging this bound into \\cref{eq:norm_grad_ybark_upperbound_pre} concludes the proof.\n\\end{proof}","start":66891,"end":70197},{"raw_tex":"  We have\n  \\begin{alignat}{2}\n    &\\mathInd\n    p_{k,i} \\nabla f(y_i)\n    =\n    p_{k,i} \\Lest (y_i - x_{i+1})\\\\\n    &=\n    p_{k,i} \\Lest \\prn*{\n      \\theta_i (x_i - x_{i-1})\n      - (x_{i+1} - x_i)\n    }\n    &\\quad&\\by{\\cref{eq:update_xy}}\\\\\n    &=\n    \\Lest \\prn*{\n      p_{k,i-1} (x_i - x_{i-1})\n      - p_{k,i} (x_{i+1} - x_i)\n    }\n    &\\quad&\\by{definition \\cref{eq:def_ybark} of $p_{k,i}$}.\n  \\end{alignat}\n  Summing this equality over $i$ yields\n  \\begin{align}\n    \\sum_{i=0}^{k-1}\n    p_{k,i} \\nabla f(y_i)\n    =\n    - \\Lest p_{k,k-1} (x_k - x_{k-1}),\n    \\label{eq:weighted_average_gradient_equation}\n  \\end{align}\n  where we have used $x_0 = x_{-1}$.\n","start":66950,"end":67614},{"raw_tex":"  Now, we obtain\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{\n      \\nabla f (\\bar y_k)\n    }\n    \\label{eq:norm_grad_ybark_upperbound_pre}\\\\\n    &\\leq\n    \\norm*{\n      \\sum_{i=0}^{k-1} p_{k,i} \\nabla f \\prn*{ y_i }\n    }\n    + \\frac{\\Mtrue}{2} \\sum_{0 \\leq i < j < k} p_{k,i} p_{k,j} \\norm*{y_i - y_j}^2\n    &\\quad&\\by{\\cref{lem:gradient_jensen}}\\\\\n    &=\n    \\Lest p_{k,k-1} \\norm*{x_k - x_{k-1}}\n    + \\frac{\\Mtrue}{2} \\sum_{0 \\leq i < j < k} p_{k,i} p_{k,j} \\norm*{y_i - y_j}^2\n    &\\quad&\\by{\\cref{eq:weighted_average_gradient_equation}}\\\\\n    &\\leq\n    \\frac{\\Lest}{Z_k} \\norm*{x_k - x_{k-1}}\n    + \\frac{\\Mtrue}{2 Z_k^2} \\sum_{0 \\leq i < j < k} \\norm*{y_i - y_j}^2,\n  \\end{alignat}\n  where the last inequality follows from $p_{k,i} \\leq p_{k,k-1} = 1 / Z_k$ for all $0 \\leq i < k$.\n","start":67614,"end":68406},{"raw_tex":"  Next, we bound the second term on the right-hand side.\n  Since for $0 \\leq i < j < k$,\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\norm*{y_i - y_j}\\\\\n    &\\leq\n    \\norm*{y_i - x_i}\n    + \\sum_{l=i+1}^{j-1} \\norm*{x_l - x_{l-1}}\n    + \\norm*{y_j - x_{j-1}}\n    &\\quad&\\by{the triangle inequality}\\\\\n    &=\n    \\norm*{x_i - x_{i-1}}\n    + \\sum_{l=i+1}^{j-1} \\norm*{x_l - x_{l-1}}\n    + 2 \\norm*{x_j - x_{j-1}}\n    &\\quad&\\by{\\cref{eq:update_xy} and $0 \\leq \\theta_k \\leq 1$}\\\\\n    &\\leq\n    \\prn*{\n      1^2 + \\sum_{l=i+1}^{j-1} 1^2 + 2^2\n    }^{1/2}\n    \\prn*{\n      \\sum_{l=i}^j \\norm*{x_l - x_{l-1}}^2\n    }^{1/2}\n    &\\quad&\\by{Cauchy--Schwarz}\\\\\n    &=\n    \\sqrt{j - i + 4}\n    \\prn*{\n      \\sum_{l=i}^j \\norm*{x_l - x_{l-1}}^2\n    }^{1/2},\n  \\end{alignat}\n","start":68407,"end":69166},{"raw_tex":"  we have\n  \\begin{align}\n    \\sum_{0 \\leq i < j < k}\n    \\norm*{y_i - y_j}^2\n    &\\leq\n    \\sum_{0 \\leq i < j < k}\n    \\sum_{l=i}^j\n    (j - i + 4) \\norm*{x_l - x_{l-1}}^2\\\\\n    % &=\n    % \\sum_{0 \\leq i \\leq l \\leq j < k}\n    % (j - i + 4) \\norm*{x_l - x_{l-1}}^2\n    % - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &=\n    \\sum_{l=0}^{k-1}\n    \\prn*{\n      \\sum_{i=0}^l\n      \\sum_{j=l}^{k-1}\n      (j - i + 4) \n    }\n    \\norm*{x_l - x_{l-1}}^2\n    - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &=\n    \\frac{k + 7}{2}\n    \\sum_{l=0}^{k-1}\n    (l + 1) (k - l)\n    \\norm*{x_l - x_{l-1}}^2\n    - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &\\leq\n    \\frac{k + 7}{2}\n    \\sum_{l=0}^{k-1} \\frac{(k+1)^2}{4} \\norm*{x_l - x_{l-1}}^2\n    - 4 \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\\\\\n    &=\n    \\frac{(k-1) (k+5)^2}{8}\n    \\sum_{l=0}^{k-1} \\norm*{x_l - x_{l-1}}^2\n    \\leq\n    \\frac{(k-1) (k+5)^2}{8} S_k.\n  \\end{align}\n  Plugging this bound into \\cref{eq:norm_grad_ybark_upperbound_pre} concludes the proof.\n","start":69166,"end":70186}]},{"id":"S053","semantic_type":"lemma","kind":"lemma","labels":["eq:grad_norm_ybar_upperbound","lem:grad_norm_ybar_upperbound"],"member_ids":["S053","S055","S058","S059"],"declared_external":false,"source_only":false,"title":"一般规则的最小梯度界","source_units":[{"raw_tex":"\\begin{lemma}\n  \\label{lem:grad_norm_ybar_upperbound}\n  Suppose that \\cref{asm:gradient_hessian_lip} holds.\n  In \\cref{alg:proposed_agd}, the following is true when $k \\geq 2$:\n  \\begin{align}\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    \\leq\n    \\frac{4 \\Lmax^2 \\Mmax}{M_0^2 k^4}.\n    \\label{eq:grad_norm_ybar_upperbound}\n  \\end{align}\n","start":70703,"end":71132},{"raw_tex":"\\begin{proof}[Proof of \\cref{eq:grad_norm_ybar_upperbound}] \n  Note that\n  \\begin{align}\n    &\\sum_{i=1}^{k-1}\n    \\frac{(i-1)(i+5)^2}{16}\n    =\n    \\frac{(k - 2) (k - 1) (k^2 + 13 k + 50)}{64}\n    \\leq\n    \\frac{k^3 (k+1)}{24},\n    \\label{eq:i_cubic_sum}\\\\\n    &\\sum_{i=1}^{k-1} Z_i^2\n    =\n    \\sum_{i=1}^{k-1} \\prn*{\\frac{i+1}{2}}^2\n    =\n    \\frac{(k-1) (2 k^2 + 5 k + 6)}{24}\n    \\geq \n    \\frac{k^2 (k+1)}{12}\n    \\label{eq:Zi_squared_sum}\n  \\end{align}\n  from \\cref{eq:ybar_Zk_alternative} and $k \\geq 2$.\n  Using \\cref{eq:grad_ybark_upperbound_Mf}, we have\n  \\begin{align}\n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\leq\n    \\sum_{i=1}^{k-1}\n    Z_i^2\n    \\norm*{ \\nabla f (\\bar y_i) }\\\\\n    &\\leq\n    \\sum_{i=1}^{k-1}\n    \\prn*{\n      \\Lest Z_i \\norm*{x_i - x_{i-1}}\n      + \\frac{(i-1) (i+5)^2}{16} \\Mtrue S_i\n    }.\n  \\end{align}\n  Each term on the right-hand side can be bounded as follows:\n  \\begin{alignat}{2}\n    &\\sum_{i=1}^{k-1} \\Lest Z_i \\norm*{x_i - x_{i-1}}\n    \\leq\n    \\Lest\n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }^{1/2}\n    \\sqrt{S_{k-1}}\n    &\\quad&\\by{Cauchy--Schwarz},\\\\\n    &\\sum_{i=1}^{k-1}\n    \\frac{(i-1)(i+5)^2}{16} \\Mtrue S_i\n    \\leq\n    \\frac{k^3 (k+1)}{24}\n    \\Mtrue S_{k-1}\n    &\\quad&\\by{\\cref{eq:i_cubic_sum}}.\n  \\end{alignat}\n  We thus have\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\\\\\n    &\\leq\n    \\Lest \n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }^{-1/2}\\!\\!\n    \\sqrt{S_{k-1}}\n    + \n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }^{-1}\n    \\frac{k^3 (k+1)}{24}\n    \\Mtrue S_{k-1}\\\\\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{k}{2}\n    \\Mtrue S_{k-1}\n    &\\quad&\\by{\\cref{eq:Zi_squared_sum}}\\\\\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{\\Lest \\Mtrue}{2 \\Mest_k} \n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{\\cref{eq:condition_continue_k-1}}\\\\\n    &\\leq\n    \\frac{\\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{\\Lest \\Mmax}{2 \\Mest_{k-1}} \n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{$\\Mest_{k-1}, \\Mtrue \\leq \\Mmax$}\\\\\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}},\n  \\end{alignat}\n  which is the first inequality of the desired result~\\cref{eq:grad_norm_ybar_upperbound}.\n  The second inequality follows from \\cref{eq:condition_continue_k-1}:\n  \\begin{align}\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    \\leq\n    \\frac{4 \\Lest^2 \\Mmax}{M_{k-1}^2 k^4}\n    \\leq\n    \\frac{4 \\Lmax^2 \\Mmax}{M_0 k^4}.\n  \\end{align}\n\\end{proof}","start":71463,"end":74094},{"raw_tex":"  We thus have\n  \\begin{alignat}{2}\n    &\\mathInd\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\\\\\n    &\\leq\n    \\Lest \n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }^{-1/2}\\!\\!\n    \\sqrt{S_{k-1}}\n    + \n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }^{-1}\n    \\frac{k^3 (k+1)}{24}\n    \\Mtrue S_{k-1}\\\\\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{k}{2}\n    \\Mtrue S_{k-1}\n    &\\quad&\\by{\\cref{eq:Zi_squared_sum}}\\\\\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{\\Lest \\Mtrue}{2 \\Mest_k} \n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{\\cref{eq:condition_continue_k-1}}\\\\\n    &\\leq\n    \\frac{\\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{\\Lest \\Mmax}{2 \\Mest_{k-1}} \n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{$\\Mest_{k-1}, \\Mtrue \\leq \\Mmax$}\\\\\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}},\n  \\end{alignat}\n  which is the first inequality of the desired result~\\cref{eq:grad_norm_ybar_upperbound}.\n","start":72797,"end":73816},{"raw_tex":"  The second inequality follows from \\cref{eq:condition_continue_k-1}:\n  \\begin{align}\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    \\leq\n    \\frac{4 \\Lest^2 \\Mmax}{M_{k-1}^2 k^4}\n    \\leq\n    \\frac{4 \\Lmax^2 \\Mmax}{M_0 k^4}.\n  \\end{align}\n","start":73816,"end":74083}]},{"id":"S054","semantic_type":"lemma","kind":"lemma","labels":["eq:grad_norm_ybar_upperbound_tight","lem:grad_norm_ybar_upperbound"],"member_ids":["S054","S060","S061","S062"],"declared_external":false,"source_only":false,"title":"强化规则的最小梯度界","source_units":[{"raw_tex":"\\begin{lemma}\n  \\label{lem:grad_norm_ybar_upperbound}\n  Suppose that \\cref{asm:gradient_hessian_lip} holds.\n  In \\cref{alg:proposed_agd}, the following is true when $k \\geq 2$:\n","start":70703,"end":70880},{"raw_tex":"  If $M_k$ is computed with \\cref{eq:update_Mest} in the algorithm, the above bound is improved to\n  \\begin{align}\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\leq\n    4 \\Lest \\sqrt{\\frac{S_{k-1}}{k^3}}\n    \\leq\n    \\frac{4 \\Lmax^2}{M_0 k^4}.\n    \\label{eq:grad_norm_ybar_upperbound_tight}\n  \\end{align}\n\\end{lemma}","start":71132,"end":71462},{"raw_tex":"\\begin{proof}[Proof of \\cref{eq:grad_norm_ybar_upperbound_tight}]\n  When $M_k$ is computed with \\cref{eq:update_Mest}, we have \\cref{eq:grad_ybark_upperbound_Mestk}.\n  Similarly to the proof of \\cref{eq:grad_norm_ybar_upperbound}, using \\cref{eq:grad_ybark_upperbound_Mestk} instead of \\cref{eq:grad_ybark_upperbound_Mf} yields\n  \\begin{alignat}{2}\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{k}{2} \\Mest_{k-1} S_{k-1}.\n  \\end{alignat}\n  The right-hand side is bounded as\n  \\begin{alignat}{2}\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{k}{2} \\Mest_{k-1} S_{k-1}\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{\\Lest}{2} \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{\\cref{eq:condition_continue_k-1}}\\\\\n    &\\leq\n    4 \\Lest\n    \\sqrt{\\frac{S_{k-1}}{k^3}},\n  \\end{alignat}\n  which implies the first inequality in \\cref{eq:grad_norm_ybar_upperbound_tight}.\n  The second inequality is obtained similarly to the second inequality of \\cref{eq:grad_norm_ybar_upperbound}.\n\\end{proof}","start":74095,"end":75190},{"raw_tex":"  When $M_k$ is computed with \\cref{eq:update_Mest}, we have \\cref{eq:grad_ybark_upperbound_Mestk}.\n  Similarly to the proof of \\cref{eq:grad_norm_ybar_upperbound}, using \\cref{eq:grad_ybark_upperbound_Mestk} instead of \\cref{eq:grad_ybark_upperbound_Mf} yields\n  \\begin{alignat}{2}\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{k}{2} \\Mest_{k-1} S_{k-1}.\n  \\end{alignat}\n","start":74161,"end":74612},{"raw_tex":"  The right-hand side is bounded as\n  \\begin{alignat}{2}\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{k}{2} \\Mest_{k-1} S_{k-1}\n    &\\leq\n    \\Lest \n    \\sqrt{\\frac{12 S_{k-1}}{k^3}}\n    +\n    \\frac{\\Lest}{2} \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{\\cref{eq:condition_continue_k-1}}\\\\\n    &\\leq\n    4 \\Lest\n    \\sqrt{\\frac{S_{k-1}}{k^3}},\n  \\end{alignat}\n  which implies the first inequality in \\cref{eq:grad_norm_ybar_upperbound_tight}.\n  The second inequality is obtained similarly to the second inequality of \\cref{eq:grad_norm_ybar_upperbound}.\n","start":74612,"end":75179}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":["eq:Zi_squared_sum","eq:i_cubic_sum"],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"归一化平方和与误差系数和","source_units":[{"raw_tex":"  Note that\n  \\begin{align}\n    &\\sum_{i=1}^{k-1}\n    \\frac{(i-1)(i+5)^2}{16}\n    =\n    \\frac{(k - 2) (k - 1) (k^2 + 13 k + 50)}{64}\n    \\leq\n    \\frac{k^3 (k+1)}{24},\n    \\label{eq:i_cubic_sum}\\\\\n    &\\sum_{i=1}^{k-1} Z_i^2\n    =\n    \\sum_{i=1}^{k-1} \\prn*{\\frac{i+1}{2}}^2\n    =\n    \\frac{(k-1) (2 k^2 + 5 k + 6)}{24}\n    \\geq \n    \\frac{k^2 (k+1)}{12}\n    \\label{eq:Zi_squared_sum}\n  \\end{align}\n  from \\cref{eq:ybar_Zk_alternative} and $k \\geq 2$.\n","start":71524,"end":71976}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"累加平均梯度上界","source_units":[{"raw_tex":"  Using \\cref{eq:grad_ybark_upperbound_Mf}, we have\n  \\begin{align}\n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\leq\n    \\sum_{i=1}^{k-1}\n    Z_i^2\n    \\norm*{ \\nabla f (\\bar y_i) }\\\\\n    &\\leq\n    \\sum_{i=1}^{k-1}\n    \\prn*{\n      \\Lest Z_i \\norm*{x_i - x_{i-1}}\n      + \\frac{(i-1) (i+5)^2}{16} \\Mtrue S_i\n    }.\n  \\end{align}\n  Each term on the right-hand side can be bounded as follows:\n  \\begin{alignat}{2}\n    &\\sum_{i=1}^{k-1} \\Lest Z_i \\norm*{x_i - x_{i-1}}\n    \\leq\n    \\Lest\n    \\prn*{\n      \\sum_{i=1}^{k-1} Z_i^2\n    }^{1/2}\n    \\sqrt{S_{k-1}}\n    &\\quad&\\by{Cauchy--Schwarz},\\\\\n    &\\sum_{i=1}^{k-1}\n    \\frac{(i-1)(i+5)^2}{16} \\Mtrue S_i\n    \\leq\n    \\frac{k^3 (k+1)}{24}\n    \\Mtrue S_{k-1}\n    &\\quad&\\by{\\cref{eq:i_cubic_sum}}.\n  \\end{alignat}\n","start":71976,"end":72797}]},{"id":"S063","semantic_type":"theorem","kind":"theorem","labels":["eq:complexity_bound","eq:complexity_bound_tight","eq:def_c1_c2","thm:complexity"],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"无参数 AGD 的两种复杂度界","source_units":[{"raw_tex":"\\begin{theorem}\n  \\label{thm:complexity}\n  Suppose that \\cref{asm:gradient_hessian_lip} holds and let $\\Lmax \\coloneqq \\max\\set{\\Linit, \\alpha \\Ltrue}$, $\\Mmax \\coloneqq \\max\\set{M_0, \\Mtrue}$, $\\Delta \\coloneqq f(x_\\mathrm{init}) - \\inf_{x \\in \\R^d} f(x)$,\n  \\begin{align}\n    c_1\n    \\coloneqq\n    \\log_\\alpha \\prn*{\\frac{1}{\\beta}},\n    \\quad\\text{and}\\quad\n    c_2\n    \\coloneqq\n    \\log_\\alpha \\prn*{\\frac{\\Lmax}{\\Linit}}.\n    \\label{eq:def_c1_c2}\n  \\end{align}\n  In \\cref{alg:proposed_agd}, when $\\bar y_k$ defined by \\cref{eq:def_ybark} satisfies $\\norm*{\\nabla f(\\bar y_k)} \\leq \\epsilon$ for the first time, the total iteration count $K$ is at most\n  \\begin{align}\n    \\prn[\\bigg]{\n      70\n      + \n      108 c_1 \\frac{\\Mmax^{1/4}}{M_0^{1/4}}\n    }\n    \\frac{\\Lmax^{1/2} \\Mmax^2 \\Delta}{\\epsilon^{7/4} M_0^{7/4}}\n    + 2 (c_2 + 1) \\frac{\\Lmax^{1/2} \\Mmax^{1/4}}{\\epsilon^{1/4} M_0^{1/2}}.\n    \\label{eq:complexity_bound}\n  \\end{align}\n  If $M_k$ is computed with \\cref{eq:update_Mest} in the algorithm and $\\beta$ is set as $\\beta = 1$, the above bound is improved to\n  \\begin{align}\n    70\n    \\frac{\\Lmax^{1/2} \\Mmax^{1/4} \\Delta}{\\epsilon^{7/4}}\n    +\n    2 (c_2 + 1) \\frac{\\Lmax^{1/2}}{M_0^{1/4} \\epsilon^{1/4}}.\n    \\label{eq:complexity_bound_tight}\n  \\end{align}\n\\end{theorem}","start":75432,"end":76724}]},{"id":"S064","semantic_type":"definition","kind":"definition","labels":["eq:def_c1_c2","thm:complexity"],"member_ids":["S064"],"declared_external":false,"source_only":false,"title":"复杂度常数与目标差","source_units":[{"raw_tex":"  Suppose that \\cref{asm:gradient_hessian_lip} holds and let $\\Lmax \\coloneqq \\max\\set{\\Linit, \\alpha \\Ltrue}$, $\\Mmax \\coloneqq \\max\\set{M_0, \\Mtrue}$, $\\Delta \\coloneqq f(x_\\mathrm{init}) - \\inf_{x \\in \\R^d} f(x)$,\n  \\begin{align}\n    c_1\n    \\coloneqq\n    \\log_\\alpha \\prn*{\\frac{1}{\\beta}},\n    \\quad\\text{and}\\quad\n    c_2\n    \\coloneqq\n    \\log_\\alpha \\prn*{\\frac{\\Lmax}{\\Linit}}.\n    \\label{eq:def_c1_c2}\n  \\end{align}\n","start":75473,"end":75899}]},{"id":"S065","semantic_type":"proof","kind":"proof","labels":["eq:Ksuc_upperbound","eq:Nsuc_upperbound","eq:Nunsuc_upperbound","eq:Sk_lowerbound","eq:restart_condition_k"],"member_ids":["S065"],"declared_external":false,"source_only":false,"title":"一般复杂度界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of bound \\cref{eq:complexity_bound}]\n  We count the number of iterations of \\cref{alg:proposed_agd} separately for three types of epoch:\n  \\begin{itemize}\n    \\item \n    successful epoch: an epoch that does not find an $\\epsilon$-stationary point and ends at Line \\ref{alg-line-agd:restart_successful} with the descent condition~\\cref{eq:descent_condition} satisfied,\n    \\item\n    unsuccessful epoch: an epoch that does not find an $\\epsilon$-stationary point and ends at Line \\ref{alg-line-agd:restart_unsuccessful} with the descent condition~\\cref{eq:descent_condition} unsatisfied,\n    \\item\n    last epoch: the epoch that finds an $\\epsilon$-stationary point.\n  \\end{itemize}\n\n  \\paragraph{Successful epochs}\n  Let us focus on a successful epoch.\n  Let $k$ be the iteration number of the epoch.\n  It follows that \n  \\begin{align}\n    (k+1)^5\n    \\Mest_k^2 S_k\n    &>\n    \\Lest^2.\n    \\label{eq:restart_condition_k}\n  \\end{align}\n  When $k \\geq 2$, we have\n  \\begin{alignat}{2}\n    \\epsilon\n    &<\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\quad&\\text{(since this epoch does not find an $\\epsilon$-stationary point)}\\\\\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{\\cref{eq:grad_norm_ybar_upperbound}}\\\\\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_0} \\sqrt{\\frac{S_k}{k^3}}\n    &\\quad&\\by{$M_{k-1} \\geq M_0$ and $S_k \\geq S_{k-1}$},\n  \\end{alignat}\n  and hence \n  \\begin{align}\n    S_k > \\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3.\n    \\label{eq:Sk_lowerbound}\n  \\end{align}\n  When $k = 1$, we also have \\cref{eq:Sk_lowerbound}, as follows:\n  \\begin{align}\n    S_k\n    =\n    \\norm*{x_1 - x_0}^2\n    =\n    \\frac{1}{\\Lest^2} \\norm*{\\nabla f(x_0)}^2\n    =\n    \\frac{1}{\\Lest^2} \\norm*{\\nabla f(\\bar y_1)}^2\n    >\n    \\frac{\\epsilon^2}{\\Lest^2}\n    \\geq\n    \\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3.\n  \\end{align}\n  Combining \\cref{eq:Sk_lowerbound,eq:restart_condition_k} yields\n  \\begin{align}\n    S_k\n    =\n    S_k^{7/8}\n    S_k^{1/8}\n    &\\geq\n    \\prn[\\bigg]{\\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3}^{7/8}\n    \\prn*{\\frac{\\Lest^2}{\\Mest_k^2 (k+1)^5}}^{1/8}\\\\\n    &=\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{3/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^{7/4} \\Mest_k^{1/4}}\n    \\frac{k^{21/8}}{(k+1)^{5/8}}\n    \\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{3/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^2}\n    \\frac{k^{21/8}}{(k+1)^{5/8}}.\n  \\end{align}\n  Plugging this bound into condition \\cref{eq:descent_condition}, we obtain\n  \\begin{align}\n    f(x_0) - f(x_k)\n    \\geq\n    \\frac{\\Lest S_k}{2 (k+1)}\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{1/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^2}\n    \\prn*{\\frac{k}{k+1}}^{13/8}\n    \\frac{k}{2}\\\\\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lmax^{1/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^2}\n    \\prn*{\\frac{1}{2}}^{13/8}\n    \\frac{k}{2}\n    \\geq\n    \\frac{\\epsilon^{7/4} M_0^{7/4}}{\\Lmax^{1/2} \\Mmax^2}\n    \\frac{k}{70},\n  \\end{align}\n  and thus\n  \\begin{align}\n    k\n    \\leq\n    70\n    \\frac{\\Lmax^{1/2} \\Mmax^2}{\\epsilon^{7/4} M_0^{7/4}}\n    \\prn*{f(x_0) - f(x_k)}.\n  \\end{align}\n  Summing this bound over all successful epochs leads to the conclusion that the total iteration number of successful epochs is at most\n  \\begin{align}\n    70 \\frac{\\Lmax^{1/2} \\Mmax^2 \\Delta}{\\epsilon^{7/4} M_0^{7/4}}.\n    \\label{eq:Ksuc_upperbound}\n  \\end{align}\n  For later use, we will also evaluate the number of successful epochs.\n  Combining \\cref{eq:Sk_lowerbound,eq:restart_condition_k} also yields\n  \\begin{align}\n    S_k\n    =\n    S_k^{3/4}\n    S_k^{1/4}\n    &\\geq\n    \\prn[\\bigg]{ \\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3 }^{3/4}\n    \\prn*{\\frac{\\Lest^2}{\\Mest_k^2 (k+1)^5}}^{1/4}\\\\\n    &=\n    \\frac{(\\epsilon / 4)^{3/2}}{\\Lest}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^{3/2} \\Mest_k^{1/2}}\n    \\frac{k^{9/4}}{(k+1)^{5/4}}\n    \\geq\n    \\frac{(\\epsilon / 4)^{3/2}}{\\Lest}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^2}\n    \\frac{k^{9/4}}{(k+1)^{5/4}}.\n  \\end{align}\n  Plugging this bound into condition \\cref{eq:descent_condition} gives\n  \\begin{align}\n    f(x_0) - f(x_k)\n    \\geq\n    \\frac{\\Lest S_k}{2 (k+1)}\n    &\\geq\n    \\frac{(\\epsilon / 4)^{3/2}}{2}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^2}\n    \\prn*{\\frac{k}{k+1}}^{9/4}\\\\\n    &\\geq\n    \\frac{(\\epsilon / 4)^{3/2}}{2}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^2}\n    \\prn*{\\frac{1}{2}}^{9/4}\n    =\n    \\frac{\\epsilon^{3/2} \\Mest_0^{3/2}}{2^{25/4} \\Mmax^2}.\n  \\end{align}\n  Thus, we can deduce that the number $N_{\\mathrm{suc}}$ of successful epochs satisfies\n  \\begin{align}\n    N_{\\mathrm{suc}}\n    \\leq\n    2^{25/4}\n    \\frac{\\Mmax^2 \\Delta}{\\epsilon^{3/2} \\Mest_0^{3/2}}.\n    \\label{eq:Nsuc_upperbound}\n  \\end{align}\n\n  \\paragraph{Other epochs}\n  We deduce from \\cref{eq:grad_norm_ybar_upperbound} that the number of iterations for each epoch is at most\n  \\begin{align}\n    \\prn*{\n      \\frac{4 \\Lmax^2 \\Mmax}{\\epsilon M_0^2}\n    }^{1/4}.\n  \\end{align}\n  Let $N_{\\mathrm{unsuc}}$ be the number of unsuccessful epochs.\n  We then have\n  \\begin{align}\n    \\Linit\n    \\beta^{N_{\\mathrm{suc}}}\n    \\alpha^{N_{\\mathrm{unsuc}}}\n    \\leq\n    \\Lmax,\n  \\end{align}\n  and hence\n  \\begin{align}\n    N_{\\mathrm{unsuc}}\n    \\leq\n    c_1 N_{\\mathrm{suc}} + c_2,\n    \\label{eq:Nunsuc_upperbound}\n  \\end{align}\n  where $c_1$ and $c_2$ are defined by \\cref{eq:def_c1_c2}.\n  The total iteration number of unsuccessful and last epochs is at most\n  \\begin{alignat}{2}\n    &\\mathInd\n    (N_{\\mathrm{unsuc}} + 1)\n    \\prn*{ \\frac{4 \\Lmax^2 \\Mmax}{\\epsilon M_0^2} }^{1/4}\\\\\n    &\\leq\n    \\prn*{ c_1 N_{\\mathrm{suc}} + c_2 + 1 }\n    \\prn*{ \\frac{4 \\Lmax^2 \\Mmax}{\\epsilon M_0^2} }^{1/4}\n    &\\quad&\\by{\\cref{eq:Nunsuc_upperbound}}\\\\\n    &\\leq\n    2^{27/4} c_1\n    \\frac{\\Lmax^{1/2} \\Mmax^{9/4} \\Delta}{\\epsilon^{7/4} \\Mest_0^2}\n    + 2^{1/2} (c_2 + 1) \\frac{\\Lmax^{1/2} \\Mmax^{1/4}}{\\epsilon^{1/4} M_0^{1/2}}\n    &\\quad&\\by{\\cref{eq:Nsuc_upperbound}}\\\\\n    &\\leq\n    108 c_1\n    \\frac{\\Lmax^{1/2} \\Mmax^{9/4} \\Delta}{\\epsilon^{7/4} \\Mest_0^2}\n    + 2 (c_2 + 1) \\frac{\\Lmax^{1/2} \\Mmax^{1/4}}{\\epsilon^{1/4} M_0^{1/2}}.\n  \\end{alignat}\n  Putting this bound together with \\cref{eq:Ksuc_upperbound} concludes the proof.\n\\end{proof}","start":76725,"end":82876}]},{"id":"S066","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S066"],"declared_external":false,"source_only":false,"title":"按终止原因划分 epoch","source_units":[{"raw_tex":"  We count the number of iterations of \\cref{alg:proposed_agd} separately for three types of epoch:\n  \\begin{itemize}\n    \\item \n    successful epoch: an epoch that does not find an $\\epsilon$-stationary point and ends at Line \\ref{alg-line-agd:restart_successful} with the descent condition~\\cref{eq:descent_condition} satisfied,\n    \\item\n    unsuccessful epoch: an epoch that does not find an $\\epsilon$-stationary point and ends at Line \\ref{alg-line-agd:restart_unsuccessful} with the descent condition~\\cref{eq:descent_condition} unsatisfied,\n    \\item\n    last epoch: the epoch that finds an $\\epsilon$-stationary point.\n  \\end{itemize}\n","start":76782,"end":77426}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":["eq:restart_condition_k"],"member_ids":["S067"],"declared_external":false,"source_only":false,"title":"成功重启给出的位移下界","source_units":[{"raw_tex":"  \\paragraph{Successful epochs}\n  Let us focus on a successful epoch.\n  Let $k$ be the iteration number of the epoch.\n  It follows that \n  \\begin{align}\n    (k+1)^5\n    \\Mest_k^2 S_k\n    &>\n    \\Lest^2.\n    \\label{eq:restart_condition_k}\n  \\end{align}\n","start":77427,"end":77679}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":["eq:Sk_lowerbound"],"member_ids":["S068"],"declared_external":false,"source_only":false,"title":"未达精度给出的位移下界","source_units":[{"raw_tex":"  When $k \\geq 2$, we have\n  \\begin{alignat}{2}\n    \\epsilon\n    &<\n    \\min_{1 \\leq i < k} \\norm*{ \\nabla f (\\bar y_i) }\n    &\\quad&\\text{(since this epoch does not find an $\\epsilon$-stationary point)}\\\\\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_{k-1}}\n    \\sqrt{\\frac{S_{k-1}}{k^3}}\n    &\\quad&\\by{\\cref{eq:grad_norm_ybar_upperbound}}\\\\\n    &\\leq\n    \\frac{4 \\Lest \\Mmax}{\\Mest_0} \\sqrt{\\frac{S_k}{k^3}}\n    &\\quad&\\by{$M_{k-1} \\geq M_0$ and $S_k \\geq S_{k-1}$},\n  \\end{alignat}\n  and hence \n  \\begin{align}\n    S_k > \\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3.\n    \\label{eq:Sk_lowerbound}\n  \\end{align}\n  When $k = 1$, we also have \\cref{eq:Sk_lowerbound}, as follows:\n  \\begin{align}\n    S_k\n    =\n    \\norm*{x_1 - x_0}^2\n    =\n    \\frac{1}{\\Lest^2} \\norm*{\\nabla f(x_0)}^2\n    =\n    \\frac{1}{\\Lest^2} \\norm*{\\nabla f(\\bar y_1)}^2\n    >\n    \\frac{\\epsilon^2}{\\Lest^2}\n    \\geq\n    \\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3.\n  \\end{align}\n","start":77679,"end":78641}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":["eq:Ksuc_upperbound"],"member_ids":["S069"],"declared_external":false,"source_only":false,"title":"成功 epoch 的累计迭代数","source_units":[{"raw_tex":"  Combining \\cref{eq:Sk_lowerbound,eq:restart_condition_k} yields\n  \\begin{align}\n    S_k\n    =\n    S_k^{7/8}\n    S_k^{1/8}\n    &\\geq\n    \\prn[\\bigg]{\\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3}^{7/8}\n    \\prn*{\\frac{\\Lest^2}{\\Mest_k^2 (k+1)^5}}^{1/8}\\\\\n    &=\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{3/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^{7/4} \\Mest_k^{1/4}}\n    \\frac{k^{21/8}}{(k+1)^{5/8}}\n    \\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{3/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^2}\n    \\frac{k^{21/8}}{(k+1)^{5/8}}.\n  \\end{align}\n  Plugging this bound into condition \\cref{eq:descent_condition}, we obtain\n  \\begin{align}\n    f(x_0) - f(x_k)\n    \\geq\n    \\frac{\\Lest S_k}{2 (k+1)}\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{1/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^2}\n    \\prn*{\\frac{k}{k+1}}^{13/8}\n    \\frac{k}{2}\\\\\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lmax^{1/2}}\n    \\frac{M_0^{7/4}}{\\Mmax^2}\n    \\prn*{\\frac{1}{2}}^{13/8}\n    \\frac{k}{2}\n    \\geq\n    \\frac{\\epsilon^{7/4} M_0^{7/4}}{\\Lmax^{1/2} \\Mmax^2}\n    \\frac{k}{70},\n  \\end{align}\n  and thus\n  \\begin{align}\n    k\n    \\leq\n    70\n    \\frac{\\Lmax^{1/2} \\Mmax^2}{\\epsilon^{7/4} M_0^{7/4}}\n    \\prn*{f(x_0) - f(x_k)}.\n  \\end{align}\n  Summing this bound over all successful epochs leads to the conclusion that the total iteration number of successful epochs is at most\n  \\begin{align}\n    70 \\frac{\\Lmax^{1/2} \\Mmax^2 \\Delta}{\\epsilon^{7/4} M_0^{7/4}}.\n    \\label{eq:Ksuc_upperbound}\n  \\end{align}\n","start":78641,"end":80096}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":["eq:Nsuc_upperbound"],"member_ids":["S070"],"declared_external":false,"source_only":false,"title":"成功 epoch 的数量","source_units":[{"raw_tex":"  For later use, we will also evaluate the number of successful epochs.\n  Combining \\cref{eq:Sk_lowerbound,eq:restart_condition_k} also yields\n  \\begin{align}\n    S_k\n    =\n    S_k^{3/4}\n    S_k^{1/4}\n    &\\geq\n    \\prn[\\bigg]{ \\prn*{\\frac{\\epsilon \\Mest_0}{4 \\Lest \\Mmax}}^2 k^3 }^{3/4}\n    \\prn*{\\frac{\\Lest^2}{\\Mest_k^2 (k+1)^5}}^{1/4}\\\\\n    &=\n    \\frac{(\\epsilon / 4)^{3/2}}{\\Lest}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^{3/2} \\Mest_k^{1/2}}\n    \\frac{k^{9/4}}{(k+1)^{5/4}}\n    \\geq\n    \\frac{(\\epsilon / 4)^{3/2}}{\\Lest}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^2}\n    \\frac{k^{9/4}}{(k+1)^{5/4}}.\n  \\end{align}\n  Plugging this bound into condition \\cref{eq:descent_condition} gives\n  \\begin{align}\n    f(x_0) - f(x_k)\n    \\geq\n    \\frac{\\Lest S_k}{2 (k+1)}\n    &\\geq\n    \\frac{(\\epsilon / 4)^{3/2}}{2}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^2}\n    \\prn*{\\frac{k}{k+1}}^{9/4}\\\\\n    &\\geq\n    \\frac{(\\epsilon / 4)^{3/2}}{2}\n    \\frac{\\Mest_0^{3/2}}{\\Mmax^2}\n    \\prn*{\\frac{1}{2}}^{9/4}\n    =\n    \\frac{\\epsilon^{3/2} \\Mest_0^{3/2}}{2^{25/4} \\Mmax^2}.\n  \\end{align}\n  Thus, we can deduce that the number $N_{\\mathrm{suc}}$ of successful epochs satisfies\n  \\begin{align}\n    N_{\\mathrm{suc}}\n    \\leq\n    2^{25/4}\n    \\frac{\\Mmax^2 \\Delta}{\\epsilon^{3/2} \\Mest_0^{3/2}}.\n    \\label{eq:Nsuc_upperbound}\n  \\end{align}\n","start":80096,"end":81390}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S071"],"declared_external":false,"source_only":false,"title":"一般规则下的 epoch 长度","source_units":[{"raw_tex":"  \\paragraph{Other epochs}\n  We deduce from \\cref{eq:grad_norm_ybar_upperbound} that the number of iterations for each epoch is at most\n  \\begin{align}\n    \\prn*{\n      \\frac{4 \\Lmax^2 \\Mmax}{\\epsilon M_0^2}\n    }^{1/4}.\n  \\end{align}\n","start":81391,"end":81626}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":["eq:Nunsuc_upperbound"],"member_ids":["S072"],"declared_external":false,"source_only":false,"title":"失败 epoch 的对数计数","source_units":[{"raw_tex":"  Let $N_{\\mathrm{unsuc}}$ be the number of unsuccessful epochs.\n  We then have\n  \\begin{align}\n    \\Linit\n    \\beta^{N_{\\mathrm{suc}}}\n    \\alpha^{N_{\\mathrm{unsuc}}}\n    \\leq\n    \\Lmax,\n  \\end{align}\n  and hence\n  \\begin{align}\n    N_{\\mathrm{unsuc}}\n    \\leq\n    c_1 N_{\\mathrm{suc}} + c_2,\n    \\label{eq:Nunsuc_upperbound}\n  \\end{align}\n  where $c_1$ and $c_2$ are defined by \\cref{eq:def_c1_c2}.\n","start":81626,"end":82027}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S073"],"declared_external":false,"source_only":false,"title":"合并失败与最后 epoch 的代价","source_units":[{"raw_tex":"  The total iteration number of unsuccessful and last epochs is at most\n  \\begin{alignat}{2}\n    &\\mathInd\n    (N_{\\mathrm{unsuc}} + 1)\n    \\prn*{ \\frac{4 \\Lmax^2 \\Mmax}{\\epsilon M_0^2} }^{1/4}\\\\\n    &\\leq\n    \\prn*{ c_1 N_{\\mathrm{suc}} + c_2 + 1 }\n    \\prn*{ \\frac{4 \\Lmax^2 \\Mmax}{\\epsilon M_0^2} }^{1/4}\n    &\\quad&\\by{\\cref{eq:Nunsuc_upperbound}}\\\\\n    &\\leq\n    2^{27/4} c_1\n    \\frac{\\Lmax^{1/2} \\Mmax^{9/4} \\Delta}{\\epsilon^{7/4} \\Mest_0^2}\n    + 2^{1/2} (c_2 + 1) \\frac{\\Lmax^{1/2} \\Mmax^{1/4}}{\\epsilon^{1/4} M_0^{1/2}}\n    &\\quad&\\by{\\cref{eq:Nsuc_upperbound}}\\\\\n    &\\leq\n    108 c_1\n    \\frac{\\Lmax^{1/2} \\Mmax^{9/4} \\Delta}{\\epsilon^{7/4} \\Mest_0^2}\n    + 2 (c_2 + 1) \\frac{\\Lmax^{1/2} \\Mmax^{1/4}}{\\epsilon^{1/4} M_0^{1/2}}.\n  \\end{alignat}\n  Putting this bound together with \\cref{eq:Ksuc_upperbound} concludes the proof.\n","start":82027,"end":82865}]},{"id":"S074","semantic_type":"proof","kind":"proof","labels":["eq:Ksuc_upperbound_tight","eq:Sk_lowerbound_tight"],"member_ids":["S074"],"declared_external":false,"source_only":false,"title":"强化复杂度界的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of bound \\cref{eq:complexity_bound_tight}]\n  The proof is similar to \\cref{eq:complexity_bound} through the evaluation of the total iteration number of successful, unsuccessful, and last epochs.\n\n  \\paragraph{Successful epochs}\n  Let us focus on a successful epoch.\n  Let $k$ be the iteration number of the epoch.\n  As a counterpart of \\cref{eq:Sk_lowerbound}, we can obtain \n  \\begin{align}\n    S_k > \\prn*{\\frac{\\epsilon}{4 \\Lest}}^2 k^3\n    \\label{eq:Sk_lowerbound_tight}\n  \\end{align}\n  from \\cref{eq:grad_norm_ybar_upperbound_tight}.\n  Combining \\cref{eq:Sk_lowerbound_tight,eq:restart_condition_k} yields\n  \\begin{align}\n    S_k\n    &=\n    S_k^{7/8}\n    S_k^{1/8}\n    \\geq\n    \\prn*{\\prn*{\\frac{\\epsilon}{4 \\Lest}}^2 k^3}^{7/8}\n    \\prn*{\\frac{\\Lest^2}{\\Mest_k^2 (k+1)^5}}^{1/8}\n    =\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{3/2} \\Mest_k^{1/4}}\n    \\frac{k^{21/8}}{(k+1)^{5/8}}.\n  \\end{align}\n  Plugging this bound into condition \\cref{eq:descent_condition}, we obtain\n  \\begin{align}\n    f(x_0) - f(x_k)\n    \\geq\n    \\frac{\\Lest S_k}{2 (k+1)}\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{1/2} \\Mest_k^{1/4}}\n    \\prn*{\\frac{k}{k+1}}^{13/8}\n    \\frac{k}{2}\\\\\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lmax^{1/2} \\Mmax^{1/4}}\n    \\prn*{\\frac{1}{2}}^{13/8}\n    \\frac{k}{2}\n    \\geq\n    \\frac{\\epsilon^{7/4}}{\\Lmax^{1/2} \\Mmax^{1/4}}\n    \\frac{k}{70}.\n  \\end{align}\n  Hence, as in the proof of \\cref{eq:complexity_bound}, the total iteration number of all successful epochs is at most\n  \\begin{align}\n    70\n    \\frac{\\Lmax^{1/2} \\Mmax^{1/4} \\Delta}{\\epsilon^{7/4}}.\n    \\label{eq:Ksuc_upperbound_tight}\n  \\end{align}\n\n  \\paragraph{Other epochs}\n  We deduce from \\cref{eq:grad_norm_ybar_upperbound_tight} that the iteration number of each epoch is at most $\\prn[\\big]{ \\frac{4 \\Lmax^2}{M_0 \\epsilon} }^{1/4}$.\n  In addition, unsuccessful epochs occur only at most $c_2$ times when $\\beta = 1$, where $c_2$ is defined by \\cref{eq:def_c1_c2}.\n  Thus, the total iteration number of unsuccessful and last epochs is at most\n  \\begin{align}\n    \\prn*{\n      c_2 + 1\n    }\n    \\prn*{\n      \\frac{4 \\Lmax^2}{M_0 \\epsilon}\n    }^{1/4}\n    \\leq\n    2 (c_2 + 1) \\frac{\\Lmax^{1/2}}{M_0^{1/4} \\epsilon^{1/4}}.\n  \\end{align}\n  Putting this bound together with \\cref{eq:Ksuc_upperbound_tight} concludes the proof.\n\\end{proof}","start":82878,"end":85215}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","labels":["eq:Sk_lowerbound_tight"],"member_ids":["S075"],"declared_external":false,"source_only":false,"title":"强化规则下的位移下界","source_units":[{"raw_tex":"  The proof is similar to \\cref{eq:complexity_bound} through the evaluation of the total iteration number of successful, unsuccessful, and last epochs.\n\n  \\paragraph{Successful epochs}\n  Let us focus on a successful epoch.\n  Let $k$ be the iteration number of the epoch.\n  As a counterpart of \\cref{eq:Sk_lowerbound}, we can obtain \n  \\begin{align}\n    S_k > \\prn*{\\frac{\\epsilon}{4 \\Lest}}^2 k^3\n    \\label{eq:Sk_lowerbound_tight}\n  \\end{align}\n  from \\cref{eq:grad_norm_ybar_upperbound_tight}.\n","start":82941,"end":83437}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","labels":["eq:Ksuc_upperbound_tight"],"member_ids":["S076"],"declared_external":false,"source_only":false,"title":"强化规则下的成功迭代总数","source_units":[{"raw_tex":"  Combining \\cref{eq:Sk_lowerbound_tight,eq:restart_condition_k} yields\n  \\begin{align}\n    S_k\n    &=\n    S_k^{7/8}\n    S_k^{1/8}\n    \\geq\n    \\prn*{\\prn*{\\frac{\\epsilon}{4 \\Lest}}^2 k^3}^{7/8}\n    \\prn*{\\frac{\\Lest^2}{\\Mest_k^2 (k+1)^5}}^{1/8}\n    =\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{3/2} \\Mest_k^{1/4}}\n    \\frac{k^{21/8}}{(k+1)^{5/8}}.\n  \\end{align}\n  Plugging this bound into condition \\cref{eq:descent_condition}, we obtain\n  \\begin{align}\n    f(x_0) - f(x_k)\n    \\geq\n    \\frac{\\Lest S_k}{2 (k+1)}\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lest^{1/2} \\Mest_k^{1/4}}\n    \\prn*{\\frac{k}{k+1}}^{13/8}\n    \\frac{k}{2}\\\\\n    &\\geq\n    \\frac{(\\epsilon / 4)^{7/4}}{\\Lmax^{1/2} \\Mmax^{1/4}}\n    \\prn*{\\frac{1}{2}}^{13/8}\n    \\frac{k}{2}\n    \\geq\n    \\frac{\\epsilon^{7/4}}{\\Lmax^{1/2} \\Mmax^{1/4}}\n    \\frac{k}{70}.\n  \\end{align}\n  Hence, as in the proof of \\cref{eq:complexity_bound}, the total iteration number of all successful epochs is at most\n  \\begin{align}\n    70\n    \\frac{\\Lmax^{1/2} \\Mmax^{1/4} \\Delta}{\\epsilon^{7/4}}.\n    \\label{eq:Ksuc_upperbound_tight}\n  \\end{align}\n","start":83437,"end":84523}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S077"],"declared_external":false,"source_only":false,"title":"固定 β 时的其余 epoch 开销","source_units":[{"raw_tex":"  \\paragraph{Other epochs}\n  We deduce from \\cref{eq:grad_norm_ybar_upperbound_tight} that the iteration number of each epoch is at most $\\prn[\\big]{ \\frac{4 \\Lmax^2}{M_0 \\epsilon} }^{1/4}$.\n  In addition, unsuccessful epochs occur only at most $c_2$ times when $\\beta = 1$, where $c_2$ is defined by \\cref{eq:def_c1_c2}.\n  Thus, the total iteration number of unsuccessful and last epochs is at most\n  \\begin{align}\n    \\prn*{\n      c_2 + 1\n    }\n    \\prn*{\n      \\frac{4 \\Lmax^2}{M_0 \\epsilon}\n    }^{1/4}\n    \\leq\n    2 (c_2 + 1) \\frac{\\Lmax^{1/2}}{M_0^{1/4} \\epsilon^{1/4}}.\n  \\end{align}\n  Putting this bound together with \\cref{eq:Ksuc_upperbound_tight} concludes the proof.\n","start":84524,"end":85204}]},{"id":"S085","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S085"],"declared_external":true,"declaration_text":["原始外部前提清单：Taylor's theorem with integral remainder and the fundamental theorem of calculus。"],"source_only":false,"title":"Taylor 积分余项与微积分基本定理","source_units":[]},{"id":"S086","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S086"],"declared_external":true,"declaration_text":["原始外部前提清单：Triangle, Cauchy–Schwarz, and Young inequalities。"],"source_only":false,"title":"三角与内积标准不等式","source_units":[]},{"id":"S087","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S087"],"declared_external":true,"declaration_text":["原始外部前提清单：monotonicity of finite-dimensional sequence norms。"],"source_only":false,"title":"有限序列范数的单调性","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S026":"S026","S027":"S027","S028":"S028","S029":"S028","S030":"S030","S031":"S031","S032":"S031","S033":"S031","S034":"S031","S035":"S031","S036":"S031","S037":"S031","S038":"S030","S039":"S030","S040":"S030","S041":"S030","S042":"S042","S043":"S042","S044":"S044","S045":"S045","S046":"S044","S047":"S044","S048":"S045","S049":"S045","S050":"S045","S051":"S045","S052":"S045","S053":"S053","S054":"S054","S055":"S053","S056":"S056","S057":"S057","S058":"S053","S059":"S053","S060":"S054","S061":"S054","S062":"S054","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076","S077":"S077","S078":"S026","S079":"S026","S080":"S026","S081":"S026","S082":"S027","S083":"S027","S084":"S027","S085":"S085","S086":"S086","S087":"S087"}}}}
{"record_id":"2211.16142v3","paper":{"title":"Generalized power cones: optimal error bounds and automorphisms","authors":["Lin, Ying","Lindstrom, Scott B.","Lourenço, Bruno F.","Pong, Ting Kei"],"arxiv_id":"2211.16142","version":"v3","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/2211.16142v3","pdf_url":"https://arxiv.org/pdf/2211.16142v3","doi":"10.1137/22M1542921"},"context":{"raw_tex":"\tIn a Euclidean space \\( \\EucSpace \\), consider the conic feasibility problem:\n\t\\begin{equation}\n\t\t\\label{eq:conic-feasibility-problem}\\tag{Feas}\n\t\t\\text{find } \\quad \\bm{x} \\in (\\subSpace + \\bm{a}) \\cap \\stdCone,\n\t\\end{equation}\n\twhere \\( \\subSpace \\) is a subspace, \\( \\bm{a} \\in \\EucSpace \\), and \\( \\stdCone\\) is a closed convex cone.\n\n\n\tWe will use plain letters to represent real scalars, bold lowercase letters to denote vectors, bold uppercase letters to stand for matrices,\\footnote{With an abuse of notation, we use $\\bm 0$ to denote a zero vector / matrix, whose dimension should be clear from the context.} and curly capital letters for (sub)spaces and sets. Let \\( \\EucSpace \\) be a finite dimensional Euclidean space, \\( \\R_+ \\) and \\( \\R_- \\) be the set of nonnegative and nonpositive real numbers, respectively.\n\tThe inner product of \\( \\EucSpace \\) is denoted by \\( \\inProd{ \\cdot }{ \\cdot  } \\) and the induced norm by \\( \\| \\cdot \\| \\).\n\tWith that, for \\( \\bm{x} \\in \\EucSpace \\) and a closed convex set \\( \\C \\subseteq \\EucSpace \\), we denote the projection of \\( \\bm{x} \\) onto \\( \\C \\) by \\( P_\\C(\\bm{x}) \\) so that \\( P_\\C(\\bm{x}) = \\argmin_{\\bm{y} \\in \\C} \\| \\bm{x} - \\bm{y} \\| \\) and the distance between \\( \\bm{x} \\) and \\( \\C \\) by \\( \\dist(\\bm{x}, \\C) = \\inf_{\\bm{y} \\in \\C} \\| \\bm{x} - \\bm{y} \\| = \\| \\bm{x} - P_\\C(\\bm{x}) \\| \\).\n\tFor any \\( \\bm{x} \\in \\EucSpace \\) and \\( \\eta \\geq 0 \\), we denote the ball centered at \\( \\bm{x} \\) with radius \\( \\eta  \\) by \\( \\B(\\bm{x}; \\eta) := \\{\\bm{y} \\in \\EucSpace \\,|\\,\\| \\bm{y} - \\bm{x} \\| \\leq \\eta \\} \\); we write \\( \\B(\\eta) \\) for the ball centered at \\( \\bm{0} \\) with radius \\( \\eta \\) for simplicity.\n\tA diagonal matrix with diagonal vector being \\( \\bm{x} \\) is denoted by \\( \\diag(\\bm{x}) \\).\n\tMeanwhile, we use \\( \\C^{\\perp } \\) to denote the orthogonal complement of \\( \\C \\) and \\( \\bm{I}_n \\) to represent the \\( n \\times n \\) identity matrix.\n\n\t{\n\t  We now recall the definition of {Lipschitzian and H\\\"olderian error bounds}.\n\t  Let \\( \\C_1, \\C_2 \\subseteq \\EucSpace \\) be closed convex sets with \\( \\C_1 \\cap \\C_2 \\neq \\emptyset  \\).\n\t  We say that \\( \\C_1, \\C_2 \\) satisfy a \\emph{uniform H\\\"olderian error bound} with exponent \\( \\gamma  \\in (0, 1] \\) if for every bounded set \\( \\B \\subseteq \\EucSpace \\), there exists a constant \\( \\kappa _\\B \\) such that\n\t  $\\dist(\\bm{x}, \\C_1 \\cap \\C_2) \\!\\leq\\! \\kappa _\\B \\max\\left\\{\\dist(\\bm{x}, \\C_1), \\dist(\\bm{x}, \\C_2)\\right\\}^{\\gamma }$ for all $\\bm{x} \\in \\B$.\n\t  If \\( \\gamma  = 1 \\), then the error bound is said to be \\emph{Lipschitzian}.\n\t}\n\n\tLet \\( \\stdCone \\subseteq \\EucSpace \\) be a closed convex cone and \\( \\stdCone^{*} \\) be its dual cone.\n\tWe will use ${\\rm int}\\,\\stdCone $, \\( \\ri \\stdCone \\), \\( \\bd \\stdCone \\), \\( \\Span \\stdCone \\), \\( \\dim \\stdCone \\) to denote the interior, relative interior, boundary, linear span and dimension of \\( \\stdCone \\), respectively.\n\tIf \\( \\stdCone \\cap -\\stdCone = \\{\\bm{0}\\} \\), we say that \\( \\stdCone \\) is \\textit{pointed}.\t\n\n\tA \\textit{face} of \\( \\stdCone \\) is a closed convex cone \\( \\stdFace \\) satisfying \\( \\stdFace \\subseteq \\stdCone \\) and the property that if \\( \\bm{x}, \\bm{y} \\in \\stdCone\\) and \\( \\bm{x} + \\bm{y} \\in \\stdFace \\), then \\( \\bm{x}, \\bm{y} \\in \\stdFace \\).\\footnote{By convention, we discard the empty face.}\n\tWe write \\( \\stdFace \\unlhd \\stdCone \\) if \\( \\stdFace \\) is a face of \\( \\stdCone \\) and \\( \\stdFace \\properideal \\stdCone \\) if \\( \\stdFace \\) is a \\textit{proper} face of \\( \\stdCone \\), i.e., \\( \\stdFace \\neq \\stdCone \\).\n\tA face \\( \\stdFace \\) is said to be \\textit{nontrivial} if \\( \\stdFace \\) is proper and \\( \\stdFace \\neq \\stdCone \\cap -\\stdCone \\).\n\tIf \\( \\stdFace = \\stdCone \\cap \\{\\bm{z}\\}^{\\perp } \\) for some \\( \\bm{z} \\in \\stdCone^{*} \\), \\( \\stdFace \\) is called an \\textit{exposed face} of \\( \\stdCone \\).\n\n\n\t{\n\t  The facial structure of the closed convex cone $\\stdCone$ is important for deducing \\textit{error bounds} for \\eqref{eq:conic-feasibility-problem}; see the seminal work of Sturm \\cite{St00}.\n\t  Recently, a new framework based on the \\textit{facial reduction algorithm} \\cite{BW81, Pa13, WM13} and \\textit{one-step facial residual functions ($\\mathds{1}$-FRFs)} \\cite[Definition 3.4]{LiLoPo20} was proposed for establishing error bounds for \\eqref{eq:conic-feasibility-problem} without requiring any constraint qualifications; see \\cite{L17, LiLoPo20, LiLoPo21}.\n\t  Next, we present a very brief overview of the framework, for more detailed explanations and the underlying intuition behind the techniques see \\cite{LiLoPo20,LiLoPo21}.\n\t  First, we recall the definition of\n\t  one-step facial residual functions.\n\t\n\t  \\begin{definition}[One-step facial residual function {\\cite[Definition 3.4]{LiLoPo20}}]\\label{def:one-step-facial-residual-function}\n\t  \tLet \\( \\mathcal{K} \\) be a closed convex cone and \\( \\bm{z} \\in \\mathcal{K}^{*} \\).\n\t  \tSuppose that \\( \\psi _{\\mathcal{K}, \\bm{z}}: \\R_+ \\times \\R_+ \\to \\R_+ \\) satisfies:% the following properties:\n\t  \t\\begin{enumerate}[label={(\\roman*)}]\n\t  \t\t\\item \\( \\psi _{\\mathcal{K}, \\bm{z}} \\) is nonnegative,  nondecreasing in each argument and for every \\( t \\in \\R_+ \\), \\( \\psi _{\\mathcal{K}, \\bm{z}} (0, t) = 0 \\).\n\t  \t\t\\item The following implication holds for any \\( \\bm{x} \\in \\Span \\mathcal{K} \\) and \\( \\epsilon \\geq 0 \\):\\vspace{-0.15 cm}\n\t  \t\t\\[ \\vspace{-0.15 cm}\n\t  \t\t\\dist(\\bm{x}, \\mathcal{K}) \\leq \\epsilon ,\\,\\inProd{ \\bm{x}}{ \\bm{z} } \\leq \\epsilon \\implies \\dist(\\bm{x}, \\mathcal{K} \\cap \\{\\bm{z}\\}^{\\perp }) \\leq \\psi _{\\mathcal{K}, \\bm{z}} (\\epsilon, \\| \\bm{x} \\|).\n\t  \t\t\\]\n\t  \t\\end{enumerate}\n\t  \tThen, \\( \\psi _{\\mathcal{K}, \\bm{z}} \\) is said to be a one-step facial residual function (\\( \\mathds{1} \\)-FRF) for \\( \\mathcal{K} \\) and \\( \\bm{z} \\).\n\t  \\end{definition}\n\t  The basic idea of the  aforementioned framework  is as follows. Suppose that in each step of the facial reduction algorithm we can find a suitable one-step facial residual function for the ``current'' face and ``next'' exposing vector until we reach a face $\\stdFace$ such that $\\stdFace$ and $\\subSpace + \\bm{a}$ satisfy the \\textit{partial polyhedral Slater's (PPS) condition} \\cite[Definition 3]{L17}.\\footnote{ We note that this implies a Lipschitzian error bound holds for $\\stdFace$ and $\\subSpace + \\bm{a}$, see \\cite[Corollary~3]{BBL99} and the discussion preceding \\cite[Proposition~2.3]{LiLoPo20}.}\n\t  Then, we can  construct an error bound for $\\stdCone$ and $\\subSpace + \\bm{a}$ by composing these residual functions in a specific manner.\n\t  In this regard, if \\eqref{eq:conic-feasibility-problem} is feasible, we define the \\textit{distance to the PPS condition} of \\eqref{eq:conic-feasibility-problem}, denoted by $ \\dpps(\\stdCone, \\subSpace + \\bm{a}) $, as the length \\textit{minus one} of the \\textit{shortest} chain of faces (among those chains constructed as in \\cite[Proposition~5]{L17}) such that the PPS condition holds for the final face in the chain and $\\subSpace + \\bm{a}$.\n\t}\n\n\n\t{\n\t  We end this section with the following lemma, which is useful in the analysis of one-dimensional faces. It will be used repeatedly in our subsequent discussions.\n\t  \\begin{lemma}[{\\cite[Lemma 2.5]{LiLoPo21}}]\n\t\t\\label{lemma:Lemma-2.11}\n\t\tLet \\( \\stdCone \\) be a pointed closed convex cone and let \\( \\bm{z}\\in \\partial \\stdCone^*\\setminus \\{{\\bm 0}\\} \\) be such that \\( \\stdFace := \\{\\bm{z}\\}^{\\perp} \\cap \\stdCone \\) is a one-dimensional proper face of \\( \\stdCone \\).\n\t\tLet \\( \\bm{f} \\in \\stdCone \\setminus \\{\\bm{0}\\} \\) be such that \\( \\stdFace = \\{t \\bm{f} \\, | \\, t \\geq 0\\} \\).\n\t\tLet \\( \\eta > 0 \\) and \\( \\bm{v} \\in \\partial \\stdCone \\cap B(\\eta) \\setminus \\stdFace \\), \\( \\bm{w} = P_{\\{\\bm{z}\\}^{\\perp}}(\\bm{v}) \\) and \\( \\bm{u} = P_{\\stdFace}(\\bm{w}) \\) with \\( \\bm{u} \\neq \\bm{w} \\).\n\t\tThen it holds that \\( \\left\\langle \\bm{f}, \\bm{z} \\right\\rangle = 0 \\) and we have\n\t\t\\[\n\t\t  \\| \\bm{v} - \\bm{w} \\| = \\frac{| \\left\\langle \\bm{z}, \\bm{v} \\right\\rangle |}{\\| \\bm{z} \\|}, \\qquad \\| \\bm{u} - \\bm{w} \\| =\n\t\t  \\begin{cases}\n\t\t\t\\left\\| \\bm{v} - \\frac{\\left\\langle \\bm{z}, \\bm{v} \\right\\rangle}{\\| \\bm{z} \\|^2}\\bm{z} - \\frac{\\left\\langle \\bm{f}, \\bm{v} \\right\\rangle}{\\| \\bm{f} \\|^2}\\bm{f} \\right\\| & \\text{ if } \\left\\langle \\bm{f}, \\bm{v} \\right\\rangle \\geq 0, \\\\\n\t\t\t\\,\\,\\,\\left\\| \\bm{v} - \\frac{\\left\\langle \\bm{z}, \\bm{v} \\right\\rangle}{\\| \\bm{z} \\|^2} \\bm{z} \\right\\| & \\text{ otherwise }.\n\t\t  \\end{cases}\n\t\t\\]\n\t\tMoreover, when \\( \\left\\langle \\bm{f}, \\bm{v} \\right\\rangle \\geq 0 \\) (or, equivalently, \\( \\left\\langle \\bm{f}, \\bm{w} \\right\\rangle \\geq 0 \\)), we have \\( \\bm{u} = P_{\\Span \\stdFace}(\\bm{w}) \\).\n\t\tOn the other hand, if \\( \\left\\langle \\bm{f}, \\bm{v} \\right\\rangle < 0 \\), we have \\( \\bm{u} = \\bm{0} \\).\n\t  \\end{lemma}\n\t}\n\n\n\tWe consider the generalized power cone and its dual.\n\tLet \\( m\\ge 1 \\), \\( n\\ge 2 \\) and \\( \\bm{\\alpha} = (\\alpha_1,\\ldots,\\alpha_n)\\in (0,1)^n \\) with \\( \\sum_{i=1}^n\\alpha_i = 1 \\), the generalized power cone \\( \\gK \\) and its dual \\( \\dgK \\) are given respectively by\n\t\\begin{equation}\\label{eq:gpowercones}\n\t\\begin{aligned}\n\t\\gK  & = \\left\\{\\bm{x} = (\\bbx,\\tbx)\\in \\R^{m+n}\\,\\bigg| \\,\\|\\bbx\\|\\le \\prod_{i=1}^n\\tx_i^{\\alpha_i},\\,\\bbx\\in \\R^m,\\,\\tbx\\in \\R^n_+\\right\\},                               \\\\\n\t\\dgK & = \\left\\{\\bm{z} = (\\bbz,\\tbz)\\in \\R^{m+n}\\,\\bigg| \\,\\|\\bbz\\|\\le \\prod_{i=1}^n\\left(\\frac{\\tz_i}{\\alpha_i}\\right)^{\\alpha_i},\\,\\bbz\\in \\R^m,\\,\\tbz\\in \\R^n_+\\right\\}.\n\t\\end{aligned}\n\t\\end{equation}\n\tHere, given a vector \\( \\bm{x}\\in \\R^{m+n} \\), we let \\( \\bbx\\in \\R^m \\) be the vector corresponding to its first \\( m \\) entries and \\( \\tbx\\in \\R^n \\) be the vector corresponding to its last \\( n \\) entries.\n\n\n\t{\n\t  The next lemma is obtained by applying \\cite[Lemma 4.1]{LiLoPo21} with \\( p = q = 2 \\).\n\t  \\begin{lemma}\n\t\t\\label{lemma:Lemma-4.1}\n\t\tLet \\( \\bm{\\zeta} \\in \\R^n \\) (\\( n \\geq 1 \\)) satisfy \\( \\| \\bm{\\zeta} \\| = 1 \\).\n\t\tDefine \\( \\overline{\\bm{\\zeta}} := -\\bm{\\zeta} \\). Then there exist \\( C > 0 \\) and \\( \\epsilon > 0 \\) so that\n\t\t\\begin{equation}\n\t\t  \\label{eq:inequ-p=q=2}\n\t\t  1 + \\left\\langle \\bm{\\zeta}, \\bm{w} \\right\\rangle \\geq C \\sum_{i \\in I} | w_i - \\overline{\\zeta}_i |^2 + \\frac{1}{2} \\sum_{i \\notin I} | w_i |^2  \\ \\text{ whenever } \\| \\bm{w} - \\overline{\\bm{\\zeta}} \\| \\leq \\epsilon \\ \\text{ and } \\ \\| \\bm{w} \\| = 1,\n\t\t\\end{equation}\n\t\twhere \\( I = \\{i \\,|\\, \\overline{\\zeta}_i \\neq 0\\} \\).\n\t\tFurthermore, for any \\( \\bm{w} \\) satisfying \\( \\| \\bm{w} \\| \\leq  1\\), it holds that \\( \\left\\langle \\bm{\\zeta}, \\bm{w} \\right\\rangle \\geq -1 \\), with the equality holding if and only if \\( \\bm{w} = \\overline{\\bm{\\zeta}} \\).\n\t  \\end{lemma}\n\t}\n\n\n\\begin{remark}[Optimality of the error bound in Theorem~\\ref{thm:Holderian-1/2-1st-type-face}]\\label{remark:F-r-order-1/2}\n\t\t\\!\\!\\!Let \\( \\bm{z}\\!\\in\\! \\!\\bd\\dgK\\backslash\\!\\{\\bm{0}\\} \\) with \\( \\bbz \\neq \\bm{0} \\) and let \\( \\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Then necessarily \\( \\tz_i > 0 \\) for all \\( i \\). Moreover, we also know from the definition that \\( \\alpha_i > 0 \\) for all \\( i \\). Now, consider the continuous function \\( \\bm{q}:(0,\\alpha_1)\\to \\{\\bm{z}\\}^\\perp \\) defined by $\\epsilon \\mapsto \\bm{q}_\\epsilon:=(\\bar{\\bm{q}}_\\epsilon,\\widetilde{\\bm{q}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{q}}_\\epsilon = -\\bbz/\\|\\bbz\\|^2,\\,\\,(\\widetilde{\\bm{q}}_\\epsilon)_1 = (\\alpha_1 - \\epsilon)\\tz_1^{-1},\\,\\,(\\widetilde{\\bm{q}}_\\epsilon)_2 = (\\alpha_2 + \\epsilon)\\tz_2^{-1},\\,\\mbox{and} \\,(\\widetilde{\\bm{q}}_\\epsilon)_i = \\alpha_i \\tz_i^{-1}, \\,\\forall i \\ge 3.\n\t\t\\]\n\t\tNotice that \\( \\bm{q}_\\epsilon \\) only differs from the \\( \\bm{f} \\) in \\eqref{eq:1st-type-face-ray} in two entries. One can check that \\( \\inProd{ \\bm{z}}{\\bm{q}_\\epsilon} = 0 \\) and \\( \\bm{q}_\\epsilon\\to \\bm{f}\\in \\Fr\\backslash\\{\\bm{0}\\} \\) as $\\epsilon \\downarrow 0$.\n\t\tMoreover, we have\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t&\\prod_{i=1}^n (\\widetilde{\\bm{q}}_\\epsilon)_i^{\\alpha_i}  = (\\alpha_1 - \\epsilon)^{\\alpha_1}(\\alpha_2 + \\epsilon)^{\\alpha_2}\\tz_1^{-\\alpha_1}\\tz_2^{-\\alpha_2}\\prod_{i=3}^n\\left(\\frac{\\alpha_i}{\\tz_i}\\right)^{\\alpha_i}          \\\\\n\t\t\t& =  \\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2}\\prod_{i=1}^n\\left(\\frac{\\alpha_i}{\\tz_i}\\right)^{\\alpha_i}\n\t\t\t\\overset{\\rm (a)}= \\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2} \\|\\bbz\\|^{-1}                                                                                          \\\\\n\t\t\t& = \\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2}\\|\\bar{\\bm{q}}_\\epsilon\\|,\n\t\t\\end{aligned}\n\t\t\\]\n\t\twhere (a) holds because \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz\\neq \\bm{0} \\). In view of this, if we define a continuous function \\( \\bm{p}:(0,\\alpha_1)\\to \\gK \\)  by $\\epsilon \\mapsto \\bm{p}_\\epsilon := (\\bar{\\bm{p}}_\\epsilon,\\widetilde{\\bm{p}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{p}}_\\epsilon := -\\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2}\\frac{\\bbz}{\\|\\bbz\\|^2}\\quad{\\rm and}\\quad \\widetilde{\\bm{p}}_\\epsilon := \\widetilde{\\bm{q}}_\\epsilon;\n\t\t\\]\n\t\tthen it is clear that \\( \\bm{p}_\\epsilon \\in \\gK \\), and we can compute that\n\t\t\\begin{equation}\\label{eq:distpower}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\dist(\\bm{q}_\\epsilon,\\gK) & \\le \\|\\bm{q}_\\epsilon - \\bm{p}_\\epsilon\\| = \\frac1{\\|\\bbz\\|}\\left|\\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2} - 1\\right| \\\\\n\t\t\t\t& = \\frac1{\\|\\bbz\\|}\\left|(1 - \\epsilon + O(\\epsilon^2))(1 + \\epsilon + O(\\epsilon^2)) - 1\\right| = O(\\epsilon^2).\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\tNext, we estimate \\( \\dist(\\bm{q}_\\epsilon,\\Fr) \\).\n\t\tNotice that \\( \\inProd{ \\bm{q}_\\epsilon}{\\bm{f}} > 0 \\) for all sufficiently small \\( \\epsilon \\) because \\( \\bm{q}_\\epsilon \\to \\bm{f} \\). Hence, using the definition of \\( \\Fr \\) and {Lemma \\ref{lemma:Lemma-2.11}}, we see that\n\t\t\\[\n\t\t\\dist(\\bm{q}_\\epsilon,\\Fr)^2 = \\left\\|\\bm{q}_\\epsilon - \\frac{\\inProd{ \\bm{q}_\\epsilon}{ \\bm{f}}}{\\|\\bm{f}\\|^2}\\bm{f}\\right\\|^2 = \\|\\bm{q}_\\epsilon\\|^2 - \\frac{(\\inProd{ \\bm{q}_\\epsilon}{ \\bm{f}})^2}{\\|\\bm{f}\\|^2}.\n\t\t\\]\n\t\tA direct computation then shows that\n\t\t\\begin{align}\n\t\t\t\\|\\bm{q}_\\epsilon\\|^2 & = \\frac1{\\|\\bbz\\|^2} + (\\alpha_1-\\epsilon)^2\\tz_1^{-2} + (\\alpha_2+\\epsilon)^2\\tz_2^{-2} + \\sum_{i=3}^n \\alpha^2_i \\tz_i^{-2}                       \\notag \\\\\n\t\t\t& = \\frac1{\\|\\bbz\\|^2} + \\sum_{i=1}^n \\alpha^2_i \\tz_i^{-2} + 2\\epsilon(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2}) + \\epsilon^2(\\tz_1^{-2} + \\tz_2^{-2}) \\notag\\\\\n\t\t\t& = \\|\\bm{f}\\|^2 + 2\\epsilon(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2}) + \\epsilon^2(\\tz_1^{-2} + \\tz_2^{-2}),\\notag\n\t\t\\end{align}\n\t\twhere the last equality follows from the definition of \\( \\bm{f} \\) in \\eqref{eq:1st-type-face-ray}.\n\t\tFurthermore,\n\t\t\\begin{align}\n(\\inProd{ \\bm{q}_\\epsilon}{ \\bm{f}})^2 & =\\textstyle \\left(\\frac1{\\|\\bbz\\|^2} + \\alpha_1(\\alpha_1-\\epsilon)\\tz_1^{-2} + \\alpha_2(\\alpha_2+\\epsilon)\\tz_2^{-2} + \\sum_{i=3}^n \\alpha^2_i \\tz_i^{-2}\\right)^2 \\notag\\\\\n\t\t\t& = \\left[ \\|\\bm{f}\\|^2  + \\epsilon (\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})\\right]^2                                                                    \\notag\\\\\n\t\t\t& = \\|\\bm{f}\\|^4 + 2\\epsilon\\|\\bm{f}\\|^2 (\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2}) + \\epsilon^2(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2.\\notag\n\t\t\\end{align}\n\t\tCombining the above three identities, we deduce further that\n\t\t\\begin{align}\\label{eq:lowerlowerboundpower}\n\t\t\t\\dist(\\bm{q}_\\epsilon,\\Fr)^2 & = \\epsilon^2\\left(\\tz_1^{-2} + \\tz_2^{-2} - \\frac{(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2}{\\|\\bm{f}\\|^2}\\right)\\nonumber                          \\\\\n\t\t\t& \\ge \\epsilon^2\\left(\\tz_1^{-2} + \\tz_2^{-2} - \\frac{(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2}{\\alpha^2_2\\tz_2^{-2} + \\alpha^2_1\\tz_1^{-2}}\\right),\n\t\t\\end{align}\n\t\twhere the inequality follows from the definition of \\( \\bm{f} \\). Now, notice that in \\eqref{eq:lowerlowerboundpower}, the scalar term is strictly greater than zero, because\n\t\t\\[\n\t\t(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2 < (\\alpha_2\\tz_2^{-2} + \\alpha_1\\tz_1^{-2})^2 \\le (\\tz_1^{-2} + \\tz_2^{-2})(\\alpha^2_2\\tz_2^{-2} + \\alpha^2_1\\tz_1^{-2}),\n\t\t\\]\n\t\twhere the strict inequality holds because \\( \\alpha_i\\tz_i^{-2} > 0 \\) for \\( i = 1 \\), \\( 2 \\), and the last inequality follows from the Cauchy-Schwarz inequality.\n\t\tThis together with \\eqref{eq:lowerlowerboundpower} shows that \\( \\dist(\\bm{q}_\\epsilon,\\Fr) = \\Omega(\\epsilon) \\). Combining this with \\eqref{eq:distpower}, we obtain\n\t\t$\n\t\t\\limsup_{\\epsilon\\downarrow 0}\\frac{\\dist(\\bm{q}_\\epsilon,\\gK)^\\frac12}{\\dist(\\bm{q}_\\epsilon,\\Fr)} < \\infty\n\t\t$.\n\t\tThus \\( | \\cdot | ^{\\frac{1}{2}} \\) satisfies the asymptotic optimality criterion {(cf. \\cite[Definition 3.1]{LiLoPo21}}) for \\( \\gK \\) and \\( \\bm{z} \\){, which implies that the error bound is optimal in the sense of \\cite[Theorem 3.2(b)]{LiLoPo21}.}\n\t\\end{remark}\n\n\\begin{remark}[Optimality of the error bound in Theorem~\\ref{thm:xface}]\\label{remark:F-z-order-beta}\n\t\t\\!\\!\\!Let \\( \\bm{z}\\!\\in\\!\\! \\bd\\dgK\\backslash\\!\\{\\bm{0}\\} \\) with \\( \\bbz = \\bm{0} \\) and let \\( \\Fhatz := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Let \\( \\stdIndex := \\{i\\,| \\,\\tz_i > 0\\}\\neq \\emptyset \\) and define\n\t\t\\[\n\t\t\\beta:= \\sum_{i\\in \\stdIndex}\\alpha_i\\in (0,1).\n\t\t\\]\n\t\tFix any \\( \\bm{u}\\in \\R^m \\) with \\( \\|\\bm{u}\\|=1 \\) and define the continuous function \\( \\bm{q}:(0,1)\\to \\{\\bm{z}\\}^\\perp \\) by $\\epsilon \\mapsto \\bm{q}_\\epsilon := (\\bar{\\bm{q}}_\\epsilon,\\widetilde{\\bm{q}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{q}}_\\epsilon = \\epsilon^\\beta \\bm{u},\\,\\,(\\widetilde{\\bm{q}}_\\epsilon)_i = 0\\,\\,\\forall i \\in \\stdIndex,\\,\\mbox{and} \\,(\\widetilde{\\bm{q}}_\\epsilon)_i = 1, \\,\\forall i \\notin \\stdIndex.\n\t\t\\]\n\t\tIt is clear that for all $\\epsilon$, \\( \\inProd{ \\bm{z}}{\\bm{q}_\\epsilon} = 0 \\) and \\( \\dist(\\bm{q}_\\epsilon,\\Fhatz) \\to 0 \\) as $\\epsilon \\downarrow 0$ . Now, define the function \\( \\bm{p}:(0,1)\\to \\gK \\) by $\\epsilon \\mapsto \\bm{p}_\\epsilon := (\\bar{\\bm{p}}_\\epsilon,\\widetilde{\\bm{p}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{p}}_\\epsilon = \\epsilon^\\beta \\bm{u},\\,\\,(\\widetilde{\\bm{p}}_\\epsilon)_i = \\epsilon\\,\\,\\forall i \\in \\stdIndex,\\,\\mbox{and} \\,(\\widetilde{\\bm{p}}_\\epsilon)_i = 1, \\,\\forall i \\notin \\stdIndex.\n\t\t\\]\n\t\tClearly $\\bm{p}_\\epsilon$ lies in \\( \\gK \\), and we have that\n\t\t$\n\t\t\t\\dist(\\bm{q}_\\epsilon,\\gK)\\le \\|\\bm{q}_\\epsilon - \\bm{p}_\\epsilon\\| \\le |\\stdIndex|\\cdot\\epsilon\n\t\t$.\n\t\tOn the other hand, we have in view of \\eqref{eq:2nd-type-face-d-dim} that \\( \\dist(\\bm{q}_\\epsilon,\\Fhatz) = \\epsilon^\\beta > 0 \\).\n\t\tHence,\n\t\t$\n\t\t\\limsup_{\\epsilon\\downarrow 0}\\frac{\\dist(\\bm{q}_\\epsilon,\\gK)^\\beta}{\\dist(\\bm{q}_\\epsilon,\\Fhatz)} \\le |\\stdIndex|^\\beta < \\infty\n\t\t$.\n\t\tThus \\( | \\cdot | ^{\\beta} \\) satisfies the asymptotic optimality criterion {(cf. \\cite[Definition 3.1]{LiLoPo21}}) for \\( \\gK \\) and \\( \\bm{z} \\){, which implies that the error bound is optimal in the sense of \\cite[Theorem 3.2(b)]{LiLoPo21}.}\n\t\\end{remark}\n\n\tUsing Theorems~\\ref{thm:Holderian-1/2-1st-type-face} and \\ref{thm:xface} {together with \\cite[Lemma~3.9]{LiLoPo20}}, we have the following result concerning one-step facial residual functions.\n","units":[{"raw_tex":"\tIn a Euclidean space \\( \\EucSpace \\), consider the conic feasibility problem:\n\t\\begin{equation}\n\t\t\\label{eq:conic-feasibility-problem}\\tag{Feas}\n\t\t\\text{find } \\quad \\bm{x} \\in (\\subSpace + \\bm{a}) \\cap \\stdCone,\n\t\\end{equation}\n\twhere \\( \\subSpace \\) is a subspace, \\( \\bm{a} \\in \\EucSpace \\), and \\( \\stdCone\\) is a closed convex cone.\n","start":7572,"end":7911},{"raw_tex":"\tWe will use plain letters to represent real scalars, bold lowercase letters to denote vectors, bold uppercase letters to stand for matrices,\\footnote{With an abuse of notation, we use $\\bm 0$ to denote a zero vector / matrix, whose dimension should be clear from the context.} and curly capital letters for (sub)spaces and sets. Let \\( \\EucSpace \\) be a finite dimensional Euclidean space, \\( \\R_+ \\) and \\( \\R_- \\) be the set of nonnegative and nonpositive real numbers, respectively.\n\tThe inner product of \\( \\EucSpace \\) is denoted by \\( \\inProd{ \\cdot }{ \\cdot  } \\) and the induced norm by \\( \\| \\cdot \\| \\).\n\tWith that, for \\( \\bm{x} \\in \\EucSpace \\) and a closed convex set \\( \\C \\subseteq \\EucSpace \\), we denote the projection of \\( \\bm{x} \\) onto \\( \\C \\) by \\( P_\\C(\\bm{x}) \\) so that \\( P_\\C(\\bm{x}) = \\argmin_{\\bm{y} \\in \\C} \\| \\bm{x} - \\bm{y} \\| \\) and the distance between \\( \\bm{x} \\) and \\( \\C \\) by \\( \\dist(\\bm{x}, \\C) = \\inf_{\\bm{y} \\in \\C} \\| \\bm{x} - \\bm{y} \\| = \\| \\bm{x} - P_\\C(\\bm{x}) \\| \\).\n\tFor any \\( \\bm{x} \\in \\EucSpace \\) and \\( \\eta \\geq 0 \\), we denote the ball centered at \\( \\bm{x} \\) with radius \\( \\eta  \\) by \\( \\B(\\bm{x}; \\eta) := \\{\\bm{y} \\in \\EucSpace \\,|\\,\\| \\bm{y} - \\bm{x} \\| \\leq \\eta \\} \\); we write \\( \\B(\\eta) \\) for the ball centered at \\( \\bm{0} \\) with radius \\( \\eta \\) for simplicity.\n\tA diagonal matrix with diagonal vector being \\( \\bm{x} \\) is denoted by \\( \\diag(\\bm{x}) \\).\n\tMeanwhile, we use \\( \\C^{\\perp } \\) to denote the orthogonal complement of \\( \\C \\) and \\( \\bm{I}_n \\) to represent the \\( n \\times n \\) identity matrix.\n\n\t{\n\t  We now recall the definition of {Lipschitzian and H\\\"olderian error bounds}.\n\t  Let \\( \\C_1, \\C_2 \\subseteq \\EucSpace \\) be closed convex sets with \\( \\C_1 \\cap \\C_2 \\neq \\emptyset  \\).\n\t  We say that \\( \\C_1, \\C_2 \\) satisfy a \\emph{uniform H\\\"olderian error bound} with exponent \\( \\gamma  \\in (0, 1] \\) if for every bounded set \\( \\B \\subseteq \\EucSpace \\), there exists a constant \\( \\kappa _\\B \\) such that\n\t  $\\dist(\\bm{x}, \\C_1 \\cap \\C_2) \\!\\leq\\! \\kappa _\\B \\max\\left\\{\\dist(\\bm{x}, \\C_1), \\dist(\\bm{x}, \\C_2)\\right\\}^{\\gamma }$ for all $\\bm{x} \\in \\B$.\n\t  If \\( \\gamma  = 1 \\), then the error bound is said to be \\emph{Lipschitzian}.\n\t}\n\n\tLet \\( \\stdCone \\subseteq \\EucSpace \\) be a closed convex cone and \\( \\stdCone^{*} \\) be its dual cone.\n\tWe will use ${\\rm int}\\,\\stdCone $, \\( \\ri \\stdCone \\), \\( \\bd \\stdCone \\), \\( \\Span \\stdCone \\), \\( \\dim \\stdCone \\) to denote the interior, relative interior, boundary, linear span and dimension of \\( \\stdCone \\), respectively.\n\tIf \\( \\stdCone \\cap -\\stdCone = \\{\\bm{0}\\} \\), we say that \\( \\stdCone \\) is \\textit{pointed}.\t\n\n\tA \\textit{face} of \\( \\stdCone \\) is a closed convex cone \\( \\stdFace \\) satisfying \\( \\stdFace \\subseteq \\stdCone \\) and the property that if \\( \\bm{x}, \\bm{y} \\in \\stdCone\\) and \\( \\bm{x} + \\bm{y} \\in \\stdFace \\), then \\( \\bm{x}, \\bm{y} \\in \\stdFace \\).\\footnote{By convention, we discard the empty face.}\n\tWe write \\( \\stdFace \\unlhd \\stdCone \\) if \\( \\stdFace \\) is a face of \\( \\stdCone \\) and \\( \\stdFace \\properideal \\stdCone \\) if \\( \\stdFace \\) is a \\textit{proper} face of \\( \\stdCone \\), i.e., \\( \\stdFace \\neq \\stdCone \\).\n\tA face \\( \\stdFace \\) is said to be \\textit{nontrivial} if \\( \\stdFace \\) is proper and \\( \\stdFace \\neq \\stdCone \\cap -\\stdCone \\).\n\tIf \\( \\stdFace = \\stdCone \\cap \\{\\bm{z}\\}^{\\perp } \\) for some \\( \\bm{z} \\in \\stdCone^{*} \\), \\( \\stdFace \\) is called an \\textit{exposed face} of \\( \\stdCone \\).\n","start":14665,"end":18175},{"raw_tex":"\t{\n\t  The facial structure of the closed convex cone $\\stdCone$ is important for deducing \\textit{error bounds} for \\eqref{eq:conic-feasibility-problem}; see the seminal work of Sturm \\cite{St00}.\n\t  Recently, a new framework based on the \\textit{facial reduction algorithm} \\cite{BW81, Pa13, WM13} and \\textit{one-step facial residual functions ($\\mathds{1}$-FRFs)} \\cite[Definition 3.4]{LiLoPo20} was proposed for establishing error bounds for \\eqref{eq:conic-feasibility-problem} without requiring any constraint qualifications; see \\cite{L17, LiLoPo20, LiLoPo21}.\n\t  Next, we present a very brief overview of the framework, for more detailed explanations and the underlying intuition behind the techniques see \\cite{LiLoPo20,LiLoPo21}.\n\t  First, we recall the definition of\n\t  one-step facial residual functions.\n\t\n\t  \\begin{definition}[One-step facial residual function {\\cite[Definition 3.4]{LiLoPo20}}]\\label{def:one-step-facial-residual-function}\n\t  \tLet \\( \\mathcal{K} \\) be a closed convex cone and \\( \\bm{z} \\in \\mathcal{K}^{*} \\).\n\t  \tSuppose that \\( \\psi _{\\mathcal{K}, \\bm{z}}: \\R_+ \\times \\R_+ \\to \\R_+ \\) satisfies:% the following properties:\n\t  \t\\begin{enumerate}[label={(\\roman*)}]\n\t  \t\t\\item \\( \\psi _{\\mathcal{K}, \\bm{z}} \\) is nonnegative,  nondecreasing in each argument and for every \\( t \\in \\R_+ \\), \\( \\psi _{\\mathcal{K}, \\bm{z}} (0, t) = 0 \\).\n\t  \t\t\\item The following implication holds for any \\( \\bm{x} \\in \\Span \\mathcal{K} \\) and \\( \\epsilon \\geq 0 \\):\\vspace{-0.15 cm}\n\t  \t\t\\[ \\vspace{-0.15 cm}\n\t  \t\t\\dist(\\bm{x}, \\mathcal{K}) \\leq \\epsilon ,\\,\\inProd{ \\bm{x}}{ \\bm{z} } \\leq \\epsilon \\implies \\dist(\\bm{x}, \\mathcal{K} \\cap \\{\\bm{z}\\}^{\\perp }) \\leq \\psi _{\\mathcal{K}, \\bm{z}} (\\epsilon, \\| \\bm{x} \\|).\n\t  \t\t\\]\n\t  \t\\end{enumerate}\n\t  \tThen, \\( \\psi _{\\mathcal{K}, \\bm{z}} \\) is said to be a one-step facial residual function (\\( \\mathds{1} \\)-FRF) for \\( \\mathcal{K} \\) and \\( \\bm{z} \\).\n\t  \\end{definition}\n\t  The basic idea of the  aforementioned framework  is as follows. Suppose that in each step of the facial reduction algorithm we can find a suitable one-step facial residual function for the ``current'' face and ``next'' exposing vector until we reach a face $\\stdFace$ such that $\\stdFace$ and $\\subSpace + \\bm{a}$ satisfy the \\textit{partial polyhedral Slater's (PPS) condition} \\cite[Definition 3]{L17}.\\footnote{ We note that this implies a Lipschitzian error bound holds for $\\stdFace$ and $\\subSpace + \\bm{a}$, see \\cite[Corollary~3]{BBL99} and the discussion preceding \\cite[Proposition~2.3]{LiLoPo20}.}\n\t  Then, we can  construct an error bound for $\\stdCone$ and $\\subSpace + \\bm{a}$ by composing these residual functions in a specific manner.\n\t  In this regard, if \\eqref{eq:conic-feasibility-problem} is feasible, we define the \\textit{distance to the PPS condition} of \\eqref{eq:conic-feasibility-problem}, denoted by $ \\dpps(\\stdCone, \\subSpace + \\bm{a}) $, as the length \\textit{minus one} of the \\textit{shortest} chain of faces (among those chains constructed as in \\cite[Proposition~5]{L17}) such that the PPS condition holds for the final face in the chain and $\\subSpace + \\bm{a}$.\n\t}\n","start":18177,"end":21324},{"raw_tex":"\t{\n\t  We end this section with the following lemma, which is useful in the analysis of one-dimensional faces. It will be used repeatedly in our subsequent discussions.\n\t  \\begin{lemma}[{\\cite[Lemma 2.5]{LiLoPo21}}]\n\t\t\\label{lemma:Lemma-2.11}\n\t\tLet \\( \\stdCone \\) be a pointed closed convex cone and let \\( \\bm{z}\\in \\partial \\stdCone^*\\setminus \\{{\\bm 0}\\} \\) be such that \\( \\stdFace := \\{\\bm{z}\\}^{\\perp} \\cap \\stdCone \\) is a one-dimensional proper face of \\( \\stdCone \\).\n\t\tLet \\( \\bm{f} \\in \\stdCone \\setminus \\{\\bm{0}\\} \\) be such that \\( \\stdFace = \\{t \\bm{f} \\, | \\, t \\geq 0\\} \\).\n\t\tLet \\( \\eta > 0 \\) and \\( \\bm{v} \\in \\partial \\stdCone \\cap B(\\eta) \\setminus \\stdFace \\), \\( \\bm{w} = P_{\\{\\bm{z}\\}^{\\perp}}(\\bm{v}) \\) and \\( \\bm{u} = P_{\\stdFace}(\\bm{w}) \\) with \\( \\bm{u} \\neq \\bm{w} \\).\n\t\tThen it holds that \\( \\left\\langle \\bm{f}, \\bm{z} \\right\\rangle = 0 \\) and we have\n\t\t\\[\n\t\t  \\| \\bm{v} - \\bm{w} \\| = \\frac{| \\left\\langle \\bm{z}, \\bm{v} \\right\\rangle |}{\\| \\bm{z} \\|}, \\qquad \\| \\bm{u} - \\bm{w} \\| =\n\t\t  \\begin{cases}\n\t\t\t\\left\\| \\bm{v} - \\frac{\\left\\langle \\bm{z}, \\bm{v} \\right\\rangle}{\\| \\bm{z} \\|^2}\\bm{z} - \\frac{\\left\\langle \\bm{f}, \\bm{v} \\right\\rangle}{\\| \\bm{f} \\|^2}\\bm{f} \\right\\| & \\text{ if } \\left\\langle \\bm{f}, \\bm{v} \\right\\rangle \\geq 0, \\\\\n\t\t\t\\,\\,\\,\\left\\| \\bm{v} - \\frac{\\left\\langle \\bm{z}, \\bm{v} \\right\\rangle}{\\| \\bm{z} \\|^2} \\bm{z} \\right\\| & \\text{ otherwise }.\n\t\t  \\end{cases}\n\t\t\\]\n\t\tMoreover, when \\( \\left\\langle \\bm{f}, \\bm{v} \\right\\rangle \\geq 0 \\) (or, equivalently, \\( \\left\\langle \\bm{f}, \\bm{w} \\right\\rangle \\geq 0 \\)), we have \\( \\bm{u} = P_{\\Span \\stdFace}(\\bm{w}) \\).\n\t\tOn the other hand, if \\( \\left\\langle \\bm{f}, \\bm{v} \\right\\rangle < 0 \\), we have \\( \\bm{u} = \\bm{0} \\).\n\t  \\end{lemma}\n\t}\n","start":21325,"end":23075},{"raw_tex":"\tWe consider the generalized power cone and its dual.\n\tLet \\( m\\ge 1 \\), \\( n\\ge 2 \\) and \\( \\bm{\\alpha} = (\\alpha_1,\\ldots,\\alpha_n)\\in (0,1)^n \\) with \\( \\sum_{i=1}^n\\alpha_i = 1 \\), the generalized power cone \\( \\gK \\) and its dual \\( \\dgK \\) are given respectively by\n\t\\begin{equation}\\label{eq:gpowercones}\n\t\\begin{aligned}\n\t\\gK  & = \\left\\{\\bm{x} = (\\bbx,\\tbx)\\in \\R^{m+n}\\,\\bigg| \\,\\|\\bbx\\|\\le \\prod_{i=1}^n\\tx_i^{\\alpha_i},\\,\\bbx\\in \\R^m,\\,\\tbx\\in \\R^n_+\\right\\},                               \\\\\n\t\\dgK & = \\left\\{\\bm{z} = (\\bbz,\\tbz)\\in \\R^{m+n}\\,\\bigg| \\,\\|\\bbz\\|\\le \\prod_{i=1}^n\\left(\\frac{\\tz_i}{\\alpha_i}\\right)^{\\alpha_i},\\,\\bbz\\in \\R^m,\\,\\tbz\\in \\R^n_+\\right\\}.\n\t\\end{aligned}\n\t\\end{equation}\n\tHere, given a vector \\( \\bm{x}\\in \\R^{m+n} \\), we let \\( \\bbx\\in \\R^m \\) be the vector corresponding to its first \\( m \\) entries and \\( \\tbx\\in \\R^n \\) be the vector corresponding to its last \\( n \\) entries.\n","start":23165,"end":24085},{"raw_tex":"\t{\n\t  The next lemma is obtained by applying \\cite[Lemma 4.1]{LiLoPo21} with \\( p = q = 2 \\).\n\t  \\begin{lemma}\n\t\t\\label{lemma:Lemma-4.1}\n\t\tLet \\( \\bm{\\zeta} \\in \\R^n \\) (\\( n \\geq 1 \\)) satisfy \\( \\| \\bm{\\zeta} \\| = 1 \\).\n\t\tDefine \\( \\overline{\\bm{\\zeta}} := -\\bm{\\zeta} \\). Then there exist \\( C > 0 \\) and \\( \\epsilon > 0 \\) so that\n\t\t\\begin{equation}\n\t\t  \\label{eq:inequ-p=q=2}\n\t\t  1 + \\left\\langle \\bm{\\zeta}, \\bm{w} \\right\\rangle \\geq C \\sum_{i \\in I} | w_i - \\overline{\\zeta}_i |^2 + \\frac{1}{2} \\sum_{i \\notin I} | w_i |^2  \\ \\text{ whenever } \\| \\bm{w} - \\overline{\\bm{\\zeta}} \\| \\leq \\epsilon \\ \\text{ and } \\ \\| \\bm{w} \\| = 1,\n\t\t\\end{equation}\n\t\twhere \\( I = \\{i \\,|\\, \\overline{\\zeta}_i \\neq 0\\} \\).\n\t\tFurthermore, for any \\( \\bm{w} \\) satisfying \\( \\| \\bm{w} \\| \\leq  1\\), it holds that \\( \\left\\langle \\bm{\\zeta}, \\bm{w} \\right\\rangle \\geq -1 \\), with the equality holding if and only if \\( \\bm{w} = \\overline{\\bm{\\zeta}} \\).\n\t  \\end{lemma}\n\t}\n","start":29269,"end":30226},{"raw_tex":"\\begin{remark}[Optimality of the error bound in Theorem~\\ref{thm:Holderian-1/2-1st-type-face}]\\label{remark:F-r-order-1/2}\n\t\t\\!\\!\\!Let \\( \\bm{z}\\!\\in\\! \\!\\bd\\dgK\\backslash\\!\\{\\bm{0}\\} \\) with \\( \\bbz \\neq \\bm{0} \\) and let \\( \\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Then necessarily \\( \\tz_i > 0 \\) for all \\( i \\). Moreover, we also know from the definition that \\( \\alpha_i > 0 \\) for all \\( i \\). Now, consider the continuous function \\( \\bm{q}:(0,\\alpha_1)\\to \\{\\bm{z}\\}^\\perp \\) defined by $\\epsilon \\mapsto \\bm{q}_\\epsilon:=(\\bar{\\bm{q}}_\\epsilon,\\widetilde{\\bm{q}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{q}}_\\epsilon = -\\bbz/\\|\\bbz\\|^2,\\,\\,(\\widetilde{\\bm{q}}_\\epsilon)_1 = (\\alpha_1 - \\epsilon)\\tz_1^{-1},\\,\\,(\\widetilde{\\bm{q}}_\\epsilon)_2 = (\\alpha_2 + \\epsilon)\\tz_2^{-1},\\,\\mbox{and} \\,(\\widetilde{\\bm{q}}_\\epsilon)_i = \\alpha_i \\tz_i^{-1}, \\,\\forall i \\ge 3.\n\t\t\\]\n\t\tNotice that \\( \\bm{q}_\\epsilon \\) only differs from the \\( \\bm{f} \\) in \\eqref{eq:1st-type-face-ray} in two entries. One can check that \\( \\inProd{ \\bm{z}}{\\bm{q}_\\epsilon} = 0 \\) and \\( \\bm{q}_\\epsilon\\to \\bm{f}\\in \\Fr\\backslash\\{\\bm{0}\\} \\) as $\\epsilon \\downarrow 0$.\n\t\tMoreover, we have\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t&\\prod_{i=1}^n (\\widetilde{\\bm{q}}_\\epsilon)_i^{\\alpha_i}  = (\\alpha_1 - \\epsilon)^{\\alpha_1}(\\alpha_2 + \\epsilon)^{\\alpha_2}\\tz_1^{-\\alpha_1}\\tz_2^{-\\alpha_2}\\prod_{i=3}^n\\left(\\frac{\\alpha_i}{\\tz_i}\\right)^{\\alpha_i}          \\\\\n\t\t\t& =  \\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2}\\prod_{i=1}^n\\left(\\frac{\\alpha_i}{\\tz_i}\\right)^{\\alpha_i}\n\t\t\t\\overset{\\rm (a)}= \\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2} \\|\\bbz\\|^{-1}                                                                                          \\\\\n\t\t\t& = \\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2}\\|\\bar{\\bm{q}}_\\epsilon\\|,\n\t\t\\end{aligned}\n\t\t\\]\n\t\twhere (a) holds because \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz\\neq \\bm{0} \\). In view of this, if we define a continuous function \\( \\bm{p}:(0,\\alpha_1)\\to \\gK \\)  by $\\epsilon \\mapsto \\bm{p}_\\epsilon := (\\bar{\\bm{p}}_\\epsilon,\\widetilde{\\bm{p}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{p}}_\\epsilon := -\\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2}\\frac{\\bbz}{\\|\\bbz\\|^2}\\quad{\\rm and}\\quad \\widetilde{\\bm{p}}_\\epsilon := \\widetilde{\\bm{q}}_\\epsilon;\n\t\t\\]\n\t\tthen it is clear that \\( \\bm{p}_\\epsilon \\in \\gK \\), and we can compute that\n\t\t\\begin{equation}\\label{eq:distpower}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\dist(\\bm{q}_\\epsilon,\\gK) & \\le \\|\\bm{q}_\\epsilon - \\bm{p}_\\epsilon\\| = \\frac1{\\|\\bbz\\|}\\left|\\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2} - 1\\right| \\\\\n\t\t\t\t& = \\frac1{\\|\\bbz\\|}\\left|(1 - \\epsilon + O(\\epsilon^2))(1 + \\epsilon + O(\\epsilon^2)) - 1\\right| = O(\\epsilon^2).\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\tNext, we estimate \\( \\dist(\\bm{q}_\\epsilon,\\Fr) \\).\n\t\tNotice that \\( \\inProd{ \\bm{q}_\\epsilon}{\\bm{f}} > 0 \\) for all sufficiently small \\( \\epsilon \\) because \\( \\bm{q}_\\epsilon \\to \\bm{f} \\). Hence, using the definition of \\( \\Fr \\) and {Lemma \\ref{lemma:Lemma-2.11}}, we see that\n\t\t\\[\n\t\t\\dist(\\bm{q}_\\epsilon,\\Fr)^2 = \\left\\|\\bm{q}_\\epsilon - \\frac{\\inProd{ \\bm{q}_\\epsilon}{ \\bm{f}}}{\\|\\bm{f}\\|^2}\\bm{f}\\right\\|^2 = \\|\\bm{q}_\\epsilon\\|^2 - \\frac{(\\inProd{ \\bm{q}_\\epsilon}{ \\bm{f}})^2}{\\|\\bm{f}\\|^2}.\n\t\t\\]\n\t\tA direct computation then shows that\n\t\t\\begin{align}\n\t\t\t\\|\\bm{q}_\\epsilon\\|^2 & = \\frac1{\\|\\bbz\\|^2} + (\\alpha_1-\\epsilon)^2\\tz_1^{-2} + (\\alpha_2+\\epsilon)^2\\tz_2^{-2} + \\sum_{i=3}^n \\alpha^2_i \\tz_i^{-2}                       \\notag \\\\\n\t\t\t& = \\frac1{\\|\\bbz\\|^2} + \\sum_{i=1}^n \\alpha^2_i \\tz_i^{-2} + 2\\epsilon(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2}) + \\epsilon^2(\\tz_1^{-2} + \\tz_2^{-2}) \\notag\\\\\n\t\t\t& = \\|\\bm{f}\\|^2 + 2\\epsilon(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2}) + \\epsilon^2(\\tz_1^{-2} + \\tz_2^{-2}),\\notag\n\t\t\\end{align}\n\t\twhere the last equality follows from the definition of \\( \\bm{f} \\) in \\eqref{eq:1st-type-face-ray}.\n\t\tFurthermore,\n\t\t\\begin{align}\n(\\inProd{ \\bm{q}_\\epsilon}{ \\bm{f}})^2 & =\\textstyle \\left(\\frac1{\\|\\bbz\\|^2} + \\alpha_1(\\alpha_1-\\epsilon)\\tz_1^{-2} + \\alpha_2(\\alpha_2+\\epsilon)\\tz_2^{-2} + \\sum_{i=3}^n \\alpha^2_i \\tz_i^{-2}\\right)^2 \\notag\\\\\n\t\t\t& = \\left[ \\|\\bm{f}\\|^2  + \\epsilon (\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})\\right]^2                                                                    \\notag\\\\\n\t\t\t& = \\|\\bm{f}\\|^4 + 2\\epsilon\\|\\bm{f}\\|^2 (\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2}) + \\epsilon^2(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2.\\notag\n\t\t\\end{align}\n\t\tCombining the above three identities, we deduce further that\n\t\t\\begin{align}\\label{eq:lowerlowerboundpower}\n\t\t\t\\dist(\\bm{q}_\\epsilon,\\Fr)^2 & = \\epsilon^2\\left(\\tz_1^{-2} + \\tz_2^{-2} - \\frac{(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2}{\\|\\bm{f}\\|^2}\\right)\\nonumber                          \\\\\n\t\t\t& \\ge \\epsilon^2\\left(\\tz_1^{-2} + \\tz_2^{-2} - \\frac{(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2}{\\alpha^2_2\\tz_2^{-2} + \\alpha^2_1\\tz_1^{-2}}\\right),\n\t\t\\end{align}\n\t\twhere the inequality follows from the definition of \\( \\bm{f} \\). Now, notice that in \\eqref{eq:lowerlowerboundpower}, the scalar term is strictly greater than zero, because\n\t\t\\[\n\t\t(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2 < (\\alpha_2\\tz_2^{-2} + \\alpha_1\\tz_1^{-2})^2 \\le (\\tz_1^{-2} + \\tz_2^{-2})(\\alpha^2_2\\tz_2^{-2} + \\alpha^2_1\\tz_1^{-2}),\n\t\t\\]\n\t\twhere the strict inequality holds because \\( \\alpha_i\\tz_i^{-2} > 0 \\) for \\( i = 1 \\), \\( 2 \\), and the last inequality follows from the Cauchy-Schwarz inequality.\n\t\tThis together with \\eqref{eq:lowerlowerboundpower} shows that \\( \\dist(\\bm{q}_\\epsilon,\\Fr) = \\Omega(\\epsilon) \\). Combining this with \\eqref{eq:distpower}, we obtain\n\t\t$\n\t\t\\limsup_{\\epsilon\\downarrow 0}\\frac{\\dist(\\bm{q}_\\epsilon,\\gK)^\\frac12}{\\dist(\\bm{q}_\\epsilon,\\Fr)} < \\infty\n\t\t$.\n\t\tThus \\( | \\cdot | ^{\\frac{1}{2}} \\) satisfies the asymptotic optimality criterion {(cf. \\cite[Definition 3.1]{LiLoPo21}}) for \\( \\gK \\) and \\( \\bm{z} \\){, which implies that the error bound is optimal in the sense of \\cite[Theorem 3.2(b)]{LiLoPo21}.}\n\t\\end{remark}","start":57718,"end":64010},{"raw_tex":"\\begin{remark}[Optimality of the error bound in Theorem~\\ref{thm:xface}]\\label{remark:F-z-order-beta}\n\t\t\\!\\!\\!Let \\( \\bm{z}\\!\\in\\!\\! \\bd\\dgK\\backslash\\!\\{\\bm{0}\\} \\) with \\( \\bbz = \\bm{0} \\) and let \\( \\Fhatz := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Let \\( \\stdIndex := \\{i\\,| \\,\\tz_i > 0\\}\\neq \\emptyset \\) and define\n\t\t\\[\n\t\t\\beta:= \\sum_{i\\in \\stdIndex}\\alpha_i\\in (0,1).\n\t\t\\]\n\t\tFix any \\( \\bm{u}\\in \\R^m \\) with \\( \\|\\bm{u}\\|=1 \\) and define the continuous function \\( \\bm{q}:(0,1)\\to \\{\\bm{z}\\}^\\perp \\) by $\\epsilon \\mapsto \\bm{q}_\\epsilon := (\\bar{\\bm{q}}_\\epsilon,\\widetilde{\\bm{q}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{q}}_\\epsilon = \\epsilon^\\beta \\bm{u},\\,\\,(\\widetilde{\\bm{q}}_\\epsilon)_i = 0\\,\\,\\forall i \\in \\stdIndex,\\,\\mbox{and} \\,(\\widetilde{\\bm{q}}_\\epsilon)_i = 1, \\,\\forall i \\notin \\stdIndex.\n\t\t\\]\n\t\tIt is clear that for all $\\epsilon$, \\( \\inProd{ \\bm{z}}{\\bm{q}_\\epsilon} = 0 \\) and \\( \\dist(\\bm{q}_\\epsilon,\\Fhatz) \\to 0 \\) as $\\epsilon \\downarrow 0$ . Now, define the function \\( \\bm{p}:(0,1)\\to \\gK \\) by $\\epsilon \\mapsto \\bm{p}_\\epsilon := (\\bar{\\bm{p}}_\\epsilon,\\widetilde{\\bm{p}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{p}}_\\epsilon = \\epsilon^\\beta \\bm{u},\\,\\,(\\widetilde{\\bm{p}}_\\epsilon)_i = \\epsilon\\,\\,\\forall i \\in \\stdIndex,\\,\\mbox{and} \\,(\\widetilde{\\bm{p}}_\\epsilon)_i = 1, \\,\\forall i \\notin \\stdIndex.\n\t\t\\]\n\t\tClearly $\\bm{p}_\\epsilon$ lies in \\( \\gK \\), and we have that\n\t\t$\n\t\t\t\\dist(\\bm{q}_\\epsilon,\\gK)\\le \\|\\bm{q}_\\epsilon - \\bm{p}_\\epsilon\\| \\le |\\stdIndex|\\cdot\\epsilon\n\t\t$.\n\t\tOn the other hand, we have in view of \\eqref{eq:2nd-type-face-d-dim} that \\( \\dist(\\bm{q}_\\epsilon,\\Fhatz) = \\epsilon^\\beta > 0 \\).\n\t\tHence,\n\t\t$\n\t\t\\limsup_{\\epsilon\\downarrow 0}\\frac{\\dist(\\bm{q}_\\epsilon,\\gK)^\\beta}{\\dist(\\bm{q}_\\epsilon,\\Fhatz)} \\le |\\stdIndex|^\\beta < \\infty\n\t\t$.\n\t\tThus \\( | \\cdot | ^{\\beta} \\) satisfies the asymptotic optimality criterion {(cf. \\cite[Definition 3.1]{LiLoPo21}}) for \\( \\gK \\) and \\( \\bm{z} \\){, which implies that the error bound is optimal in the sense of \\cite[Theorem 3.2(b)]{LiLoPo21}.}\n\t\\end{remark}","start":70483,"end":72529},{"raw_tex":"\tUsing Theorems~\\ref{thm:Holderian-1/2-1st-type-face} and \\ref{thm:xface} {together with \\cite[Lemma~3.9]{LiLoPo20}}, we have the following result concerning one-step facial residual functions.\n","start":72531,"end":72725}]},"statement":{"raw_tex":"\\begin{theorem}[Error bounds for the generalized power cone and their optimality]\\label{thm:error-bound-power-cone-optimality}\n\t\tConsider \\( \\gK \\) and its dual cone \\( \\dgK \\).\n\t\tLet \\( \\subSpace \\subseteq \\R^{m + n} \\) be a subspace and \\( \\bm{a} \\in \\R^{m+n} \\) be given.\n\t\tSuppose that \\( ( \\subSpace + \\bm{a} ) \\cap \\gK \\neq \\emptyset \\).\n\t\tThen the following items hold.\n\t\t\\begin{enumerate}[label=(\\roman*)]\n\t\t\t\\item\\label{item:error-bound-power-cone-1} \\( \\dpps (\\gK, \\subSpace + \\bm{a}) \\leq 1 \\).\n\t\t\t\\item\\label{item:error-bound-power-cone-2} If \\( \\dpps (\\gK, \\subSpace + \\bm{a}) = 0 \\), then a Lipschitzian error bound holds.\n\t\t\t\\item\\label{item:error-bound-power-cone-3} If \\( \\dpps (\\gK, \\subSpace + \\bm{a}) = 1 \\), consider the chain of faces \\( \\stdFace \\subsetneq \\gK \\) with length being \\( 2 \\).\n\t\t\t\\begin{enumerate}[label=(\\alph*)]\n\t\t\t\t\\item\\label{item:error-bound-power-cone-3-1} If \\( \\stdFace = \\Fr \\), then a H\\\"olderian error bound with exponent \\( 1 / 2 \\) holds.\n\t\t\t\t\\item\\label{item:error-bound-power-cone-3-2} If \\( \\stdFace = \\Fhatz \\) with \\( \\bm{z} \\in \\dgK \\cap \\subSpace^{\\perp } \\cap \\{\\bm{a}\\}^{\\perp } \\), then a H\\\"olderian error bound with exponent \\( \\beta := \\sum_{i : \\tz_i > 0} \\alpha _i \\) holds.\n\t\t\t\t\\item\\label{item:error-bound-power-cone-3-3} If \\( \\stdFace = \\{\\bm{0}\\} \\), then a Lipschitzian error bound holds.\n\t\t\t\\end{enumerate}\n\t\t\t\\item\\label{item:error-bound-power-cone-4} All these error bounds are the best in the sense {stated in \\cite[Theorem 3.2(b)]{LiLoPo21}}.\n\t\t\\end{enumerate}\n\t\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}[Error bounds for the generalized power cone and their optimality]\\label{thm:error-bound-power-cone-optimality}\n\t\tConsider \\( \\gK \\) and its dual cone \\( \\dgK \\).\n\t\tLet \\( \\subSpace \\subseteq \\R^{m + n} \\) be a subspace and \\( \\bm{a} \\in \\R^{m+n} \\) be given.\n\t\tSuppose that \\( ( \\subSpace + \\bm{a} ) \\cap \\gK \\neq \\emptyset \\).\n\t\tThen the following items hold.\n\t\t\\begin{enumerate}[label=(\\roman*)]\n\t\t\t\\item\\label{item:error-bound-power-cone-1} \\( \\dpps (\\gK, \\subSpace + \\bm{a}) \\leq 1 \\).\n\t\t\t\\item\\label{item:error-bound-power-cone-2} If \\( \\dpps (\\gK, \\subSpace + \\bm{a}) = 0 \\), then a Lipschitzian error bound holds.\n\t\t\t\\item\\label{item:error-bound-power-cone-3} If \\( \\dpps (\\gK, \\subSpace + \\bm{a}) = 1 \\), consider the chain of faces \\( \\stdFace \\subsetneq \\gK \\) with length being \\( 2 \\).\n\t\t\t\\begin{enumerate}[label=(\\alph*)]\n\t\t\t\t\\item\\label{item:error-bound-power-cone-3-1} If \\( \\stdFace = \\Fr \\), then a H\\\"olderian error bound with exponent \\( 1 / 2 \\) holds.\n\t\t\t\t\\item\\label{item:error-bound-power-cone-3-2} If \\( \\stdFace = \\Fhatz \\) with \\( \\bm{z} \\in \\dgK \\cap \\subSpace^{\\perp } \\cap \\{\\bm{a}\\}^{\\perp } \\), then a H\\\"olderian error bound with exponent \\( \\beta := \\sum_{i : \\tz_i > 0} \\alpha _i \\) holds.\n\t\t\t\t\\item\\label{item:error-bound-power-cone-3-3} If \\( \\stdFace = \\{\\bm{0}\\} \\), then a Lipschitzian error bound holds.\n\t\t\t\\end{enumerate}\n\t\t\t\\item\\label{item:error-bound-power-cone-4} All these error bounds are the best in the sense {stated in \\cite[Theorem 3.2(b)]{LiLoPo21}}.\n\t\t\\end{enumerate}\n\t\\end{theorem}","start":74378,"end":75929}]},"proof":{"raw_tex":"\\begin{lemma}\\label{lemma:inequ-inprod-power-cone}\n\t\tLet \\( n\\ge 2 \\) and \\( \\bm{\\alpha} = (\\alpha_1,\\ldots,\\alpha_n)\\in (0,1)^n \\) with \\( \\sum_{i=1}^n\\alpha_i = 1 \\). Let \\( \\bm{\\zeta}\\in {\\rm int}\\,\\R_{-}^n \\) satisfy \\( \\prod_{i=1}^n (-\\zeta_i/\\alpha_i)^{\\alpha_i} = 1 \\). Define \\( \\widetilde{\\bm{\\zeta}} := -\\bm\\alpha\\circ \\bm{\\zeta}^{-1} \\), where \\( \\circ \\) is the Hadamard product and the inverse is taken componentwise. Then there exist \\( C > 0 \\) and \\( \\epsilon > 0 \\) so that\n\t\t\\begin{equation}\\label{eq:inequ-inprod-power-cone}\n\t\t\t-1 - \\inProd{\\bm{\\zeta}}{\\bm{\\omega}} \\ge C\\|\\bm{\\omega} - \\widetilde{\\bm{\\zeta}}\\|^2 \\,\\,{whenever}\\,\\bm{\\omega}\\in {\\rm int}\\,\\R^n_+,\\,\\|\\bm{\\omega} - \\widetilde{\\bm{\\zeta}}\\|\\le \\epsilon {\\,and\\,} \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1.\n\t\t\\end{equation}\n\t\tMoreover, for any \\( \\bm{\\omega}\\in {\\rm int}\\,\\R_+^n \\) satisfying \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\), it holds that \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}\\le-1 \\); furthermore, we have \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}=-1 \\) if and only if \\( \\bm{\\omega} = \\widetilde{\\bm{\\zeta}} \\).\n\t\\end{lemma}\n\n\\begin{proof}\n\t\tFor each \\( i \\), we see from the Taylor series of \\( \\ln(\\cdot) \\) at \\( \\widetilde{\\zeta}_i > 0 \\) that\n\t\t\\begin{equation*}\n\t\t\t\\ln(\\omega_i) = \\ln(\\widetilde{\\zeta}_i) + \\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) - \\widetilde{\\zeta}_i^{-2}(\\omega_i - \\widetilde{\\zeta}_i)^2 + O(|\\omega_i - \\widetilde{\\zeta}_i|^3)\\,\\,{\\rm as}\\,\\,\\omega_i \\to \\widetilde{\\zeta}_i, \\,\\,\\omega_i > 0.\n\t\t\\end{equation*}\n\t\tThus, there exist \\( \\epsilon_i > 0 \\) and \\( c_i > 0 \\) so that\n\t\t\\[\n\t\t\\ln(\\widetilde{\\zeta}_i) \\ge \\ln(\\omega_i) - \\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) + c_i (\\omega_i - \\widetilde{\\zeta}_i)^2\\,\\,\\,\\mbox{whenever} \\,|\\omega_i - \\widetilde{\\zeta}_i|\\le \\epsilon_i \\,{\\rm and}\\,\\omega_i > 0.\n\t\t\\]\n\t\tLet \\( \\epsilon := \\min\\limits_{1\\le i\\le n}\\epsilon_i > 0 \\). Multiplying both sides of the above inequality by \\( \\alpha_i \\) and summing the resulting inequalities from \\( i = 1 \\) to \\( n \\), we see that whenever \\( \\bm{\\omega}\\in {\\rm int}\\,\\R^n_+ \\) satisfies \\( \\|\\bm{\\omega} - \\widetilde{\\bm{\\zeta}}\\|\\le \\epsilon \\) and \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\), we have\n\t\t\\begin{align}\n\t\t\t&0  \\overset{\\rm (a)}= \\sum_{i=1}^n\\alpha_i\\ln(\\widetilde{\\zeta}_i)\\ge \\sum_{i=1}^n\\alpha_i\\ln(\\omega_i) - \\sum_{i=1}^n\\alpha_i\\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) + \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2 \\notag\\\\\n\t\t\t& \\overset{\\rm (b)}= - \\sum_{i=1}^n\\alpha_i\\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) + \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2\n\t\t\t\\overset{\\rm (c)}= - \\sum_{i=1}^n\\alpha_i\\widetilde{\\zeta}_i^{-1}\\omega_i \\!+\\! 1 \\!+\\! \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2                                                                                                                          \\notag\\\\\n\t\t\t& = \\sum_{i=1}^n\\zeta_i\\omega_i + 1 + \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2,\\notag\n\t\t\\end{align}\n\t\twhere (a) and (b) hold because \\( \\prod_{i=1}^n\\widetilde{\\zeta}_i^{\\alpha_i} = \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\), (c) uses the fact that \\( \\sum_{i=1}^n\\alpha_i = 1 \\), and the last equality follows from the definition of \\( \\widetilde{\\bm{\\zeta}} \\). Rearranging the above inequality, we conclude that \\eqref{eq:inequ-inprod-power-cone} holds with \\( C = \\min\\limits_{1\\le i\\le n}\\alpha_ic_i > 0 \\).\n\n\t\tNext, let \\( \\bm{\\omega}\\in {\\rm int}\\,\\R_+^n \\) satisfy \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\). Then \\( (-1,\\bm{\\omega})\\in \\K \\). Recall from the assumption that \\( (1,-\\bm{\\zeta})\\in \\dK \\). From these we deduce \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}\\le-1 \\). If \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}=-1 \\), then\n\t\t\\[\n\t\t\\sum_{i=1}^n \\alpha_i\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)\\omega_i = \\sum_{i=1}^n (-\\zeta_i)\\omega_i = 1 = \\prod_{i=1}^n\\omega_i^{\\alpha_i} = \\prod_{i=1}^n\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)^{\\alpha_i}\\prod_{i=1}^n\\omega_i^{\\alpha_i} .\n\t\t\\]\n\t\tTaking \\( \\ln \\) on both sides of the above equality, we see that\n\t\t\\[\n\t\t\\ln\\left[\\sum_{i=1}^n \\alpha_i\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)\\omega_i\\right] = \\sum_{i=1}^n\\alpha_i \\ln\\left[\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)\\omega_i\\right].\n\t\t\\]\n\t\tSince \\( \\ln \\) is strictly concave and \\( \\alpha_i \\in (0,1) \\) for all \\( i \\), we conclude that there exists \\( c > 0 \\) so that \\( \\omega_i\\cdot(-\\zeta_i/\\alpha_i) = c \\) for all \\( i \\). This, together with the facts that\n\t\t\\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = \\prod_{i=1}^n (-\\zeta_i/\\alpha_i)^{\\alpha_i} = 1 \\) and \\( \\sum_{i=1}^n\\alpha_i = 1 \\), gives \\( c = 1 \\). It thus follows that \\( \\bm{\\omega} = \\widetilde{\\bm{\\zeta}} \\). Conversely, it is routine to check that if \\( \\bm{\\omega} = \\widetilde{\\bm{\\zeta}} \\), then \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\) and \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}=-1 \\).\n\t\\end{proof}\n\n\\begin{proposition}[Proper nontrivial exposed faces of \\( \\gK \\)]\\label{prop:facial-stru-gK}\n\t\tLet \\( \\bm{z} = (\\bbz,\\tbz)\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\).\n\t\t\\begin{enumerate}[label={(\\roman*)}]\n\t\t  \\item \\label{item:case-I-facial-structure} If \\( \\bbz \\neq \\bm{0} \\), then \\( \\bm{z} \\) exposes the following one-dimensional face: %\\( \\Fr \\), where\n\t\t\t\t\\begin{equation}\\label{eq:1st-type-face-ray}\n\t\t\t\t  \\displayindent0pt\n\t\t\t\t  \\displaywidth\\textwidth\n\t\t\t\t  \\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK =  \\{t \\bm{f}\\in \\R^{m+n}\\,| \\,t\\ge 0\\}\\,\\,\\mbox{with} \\,\\bm{f}= (-\\bbz/\\|\\bbz\\|^2,\\bm{\\alpha}\\circ \\tbz^{-1}),\n\t\t\t\t\\end{equation}\n\t\t\t\twhere the inverse is taken componentwise.\n\n\t\t  \\item \\label{item:case-II-facial-structure} If \\( \\bbz = \\bm{0} \\), then \\( \\bm{z} \\) exposes the following face of dimension \\( n - | \\stdIndex | \\):\n\t\t\t\t\\begin{equation}\\label{eq:2nd-type-face-d-dim}\n\t\t\t\t  \\displayindent0pt\n\t\t\t\t  \\displaywidth\\textwidth\n\t\t\t\t  \\Fhatz := \\{\\bm{z}\\}^\\perp \\cap \\gK = \\{\\bm{x} = (\\bbx,\\tbx)\\in \\R_+^{m+n}\\,| \\,\\bbx = \\bm{0},\\,\\tx_i = 0\\ \\forall i\\in \\stdIndex\\},\n\t\t\t\t\\end{equation}\n\t\t\t\twhere \\( \\stdIndex := \\{i \\,|\\, \\tz_i > 0\\}\\neq \\emptyset \\) and \\( | \\stdIndex | \\) denotes the cardinality of $\\stdIndex$.\n\t\t\\end{enumerate}\n\t  \\end{proposition}\n\n\\begin{proof}\n\t  \\ref{item:case-I-facial-structure}: Notice that \\( \\bm{x} = (\\bbx,\\tbx)\\in \\{\\bm{z}\\}^\\perp\\cap \\gK\\backslash \\{\\bm{0}\\} \\) if and only if \\( \\bm{x}\\in \\bd\\gK \\), \\( \\bm{x} \\neq \\bm{0} \\) and %satisfies\n\t  \\begin{equation}\\label{eq:inprod=0-case-I}\n\t\t\\inProd{\\bbz}{\\bbx} + \\inProd{ \\tbz}{\\tbx} = 0.\n\t  \\end{equation}\n\t  The above relation yields\n\t  \\begin{equation}\\label{eq:inequ-sum-prod-power-case-I}\n\t\t\\sum_{i=1}^n \\tz_i\\tx_i = -\\inProd{\\bbz}{\\bbx}\\le \\|\\bbz\\| \\|\\bbx\\| \\le \\prod_{i=1}^n\\left(\\frac{\\tx_i\\tz_i}{\\alpha_i}\\right)^{\\alpha_i},\n\t  \\end{equation}\n\t  where the last inequality follows from the definition of $\\gK$ in \\eqref{eq:gpowercones}.\n\n\t  Note that \\( \\tx_i \\) cannot be \\textbf{all} zero, for otherwise \\( \\bbx \\) will also be zero since \\( \\bm{x}\\in \\bd\\gK \\), which contradicts \\( \\bm{x}\\neq \\bm{0} \\). In addition, we must have \\( \\tz_i > 0 \\) for all \\( i \\) because \\( \\bbz \\neq \\bm{0} \\) and \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\). Using these observations, we have $\\sum_{i=1}^n \\tz_i\\tx_i >0$. Combining this with \\eqref{eq:inequ-sum-prod-power-case-I}, we deduce that \\( \\tx_i\\tz_i > 0 \\) for all \\( i \\). Now we can take \\( \\ln \\) on both sides of \\eqref{eq:inequ-sum-prod-power-case-I} to obtain\n\t  \\begin{equation}\\label{eq:log-inequ-sum-prod-power-case-I}\n\t\t\\ln\\left[\\sum_{i=1}^n \\alpha_i\\left(\\frac{\\tx_i\\tz_i}{\\alpha_i}\\right)\\right] \\le \\alpha_1\\ln\\left(\\frac{\\tx_1\\tz_1}{\\alpha_1}\\right) + \\cdots + \\alpha_n\\ln\\left(\\frac{\\tx_n\\tz_n}{\\alpha_n}\\right).\n\t  \\end{equation}\n\t  Using this together with the fact that \\( \\ln(\\cdot) \\) is strictly concave, we deduce that \\eqref{eq:log-inequ-sum-prod-power-case-I} holds as an equality. Hence, there exists a constant \\( c > 0 \\) so that\n\t  \\begin{equation}\\label{eq:form-tildex-by-tildez-case-I}\n\t\t\\tx_i = c \\alpha_i\\tz_i^{-1}\\,\\,\\,\\forall i = 1, 2, \\dots , n.\n\t  \\end{equation}\n\t  Plugging \\eqref{eq:form-tildex-by-tildez-case-I} into \\eqref{eq:inprod=0-case-I}, we obtain\n\t  \\begin{equation}\\label{eq:inprod-barz-barx=c-case-I}\n\t\t\\inProd{\\bbz}{\\bbx} = -\\inProd{ \\tbz}{\\tbx} = -c\\sum_{i=1}^n \\alpha_i = -c.\n\t  \\end{equation}\n\t  Moreover, using \\eqref{eq:form-tildex-by-tildez-case-I} and the last relation in \\eqref{eq:inequ-sum-prod-power-case-I}, we see that\n\t  \\[\n\t\t\\|\\bbz\\| \\|\\bbx\\| \\le \\prod_{i=1}^n\\left(\\frac{\\tx_i\\tz_i}{\\alpha_i}\\right)^{\\alpha_i} = c.\n\t  \\]\n\t  {The two displayed lines} above show that \\( \\|\\bbz\\| \\|\\bbx\\| = -\\inProd{\\bbz}{\\bbx} \\), which together with \\( \\bbz \\neq \\bm{0} \\) implies that there exists \\( \\kappa > 0 \\) so that\n\t  \\begin{equation}\\label{eq:form-barx-by-barz-kappa}\n\t\t\\bbx = -\\kappa\\bbz.\n\t  \\end{equation}\n\t  Plugging \\eqref{eq:form-barx-by-barz-kappa} into \\eqref{eq:inprod-barz-barx=c-case-I}, we obtain that \\( \\kappa = c/\\|\\bbz\\|^2 \\). Using this together with \\eqref{eq:form-tildex-by-tildez-case-I} and \\eqref{eq:form-barx-by-barz-kappa}, we can now conclude that\n\t  \\[\n\t\t\\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK =  \\{t \\bm{f}\\in \\R^{m+n}\\,| \\,t\\ge 0\\}\\,\\,\\mbox{with} \\,\\bm{f}= (-\\bbz/\\|\\bbz\\|^2,\\bm{\\alpha}\\circ \\tbz^{-1}),\n\t  \\]\n\t  where the inverse is taken componentwise.\n\n\t  \\ref{item:case-II-facial-structure}: In this case, \\( \\bbz = \\bm{0} \\). Then \\( \\stdIndex:= \\{i\\,| \\,\\tz_i > 0\\} \\) is nonempty because \\( \\bm{z}\\neq \\bm{0} \\). Hence, \\( \\bm{x} = (\\bbx,\\tbx)\\in \\{\\bm{z}\\}^\\perp\\cap \\gK\\backslash \\{\\bm{0}\\} \\) if and only if \\( \\bm{x}\\in \\bd\\gK\\backslash\\{\\bm{0}\\} \\) and satisfies\n\t  \\begin{equation*}\n\t\t\\sum_{i\\in \\stdIndex} \\tz_i\\tx_i = 0.\n\t  \\end{equation*}\n\t  This means that \\( \\tx_i = 0 \\) whenever \\( i\\in \\stdIndex \\) and hence \\( \\bbx= \\bm{0} \\). Thus,\n\t  \\[\n\t\t\\Fhatz := \\{\\bm{z}\\}^\\perp \\cap \\gK = \\{\\bm{x} = (\\bbx,\\tbx)\\in \\R_+^{m+n}\\,| \\,\\bbx = \\bm{0},\\,\\tx_i = 0\\ \\forall i\\in \\stdIndex\\}.\n\t  \\]\n\t\\end{proof}\n\n\\begin{theorem}\\label{thm:Holderian-1/2-1st-type-face}\n\t\tLet \\( \\bm{z} = (\\bbz,\\tbz)\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz\\neq \\bm{0} \\) and let \\( \\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Let \\( \\eta > 0 \\) and define\n\t\t\\begin{equation}\\label{eq:gamma-1/2-1st-type-face}\n\t\t\t\\gamma_{\\bm{z},\\eta}\\! :=\\! \\inf_{\\bm{v}}\\left\\{\\frac{\\|\\bm{v} - \\bm{w}\\|^\\frac12}{\\|\\bm{u} - \\bm{w}\\|}\\,\\bigg| \\,\n\\begin{array}{c}\n\\bm{v}\\in \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fr,\\,\\bm{w} = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}),\\\\\n\\bm{u} = P_{\\Fr}(\\bm{w}),\\,\\bm{u} \\neq \\bm{w}\n\\end{array}\n\\right\\}.\n\t\t\\end{equation}\n\t\tThen it holds that \\( \\gamma_{\\bm{z},\\eta}\\in (0,\\infty] \\) and that\n\t\t\\[\n\t\t\\dist(\\bm{q},\\Fr)\\le \\max\\{2\\sqrt{\\eta},2\\gamma_{\\bm{z},\\eta}^{-1}\\}\\cdot\\dist(\\bm{q},\\gK)^{\\frac{1}{2}}\\,\\,\\text{ whenever }  \\bm{q} \\in \\{\\bm{z}\\}^\\perp\\cap \\B(\\eta) .\n\t\t\\]\n\t\\end{theorem}\n\n\\begin{proof}\n\t\tSuppose for a contradiction that \\( \\gamma_{\\bm{z},\\eta} = 0 \\). Then, {in view of \\cite[Lemma 3.12]{LiLoPo20}}, there exist \\( \\widehat{\\bm{v}}\\in \\Fr \\) and a sequence \\( \\{\\bm{v}^k\\}\\subset \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fr \\) such that\n\t\t\\begin{equation}\\label{eq:forcontradiction-1st-face}\n\t\t\t\\lim_{k\\to \\infty}\\bm{v}^k = \\lim_{k\\to \\infty}\\bm{w}^k = \\widehat{\\bm{v}}\\,\\,{\\rm and}\\,\\,\\lim_{k\\to\\infty}\\frac{\\|\\bm{w}^k - \\bm{v}^k\\|^{\\frac{1}{2}}}{\\|\\bm{w}^k - \\bm{u}^k\\|} = 0,\n\t\t\\end{equation}\n\t\twhere \\( \\bm{w}^k = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}^k) \\), \\( \\bm{u}^k = P_{\\Fr}(\\bm{w}^k) \\) and \\( \\bm{u}^k\\neq \\bm{w}^k \\).\n\n\t\tDefine, for notational simplicity, \\( z_0 := \\|\\bbz\\| \\) and \\( v^k_0:=\\|\\bbv^k\\| \\).\n\t\tThen, since \\( \\{\\bm{v}^k\\}\\subset \\bd \\gK \\) and \\( \\bm{z} \\in \\bd\\dgK \\) with \\( \\bbz \\neq \\bm{0} \\), we have\\vspace{-0.15 cm}\n\t\t\\begin{equation}\\label{eq:z0-v0-notation}\\vspace{-0.15 cm}\n\t\t\tz_0 = \\|\\bbz\\| = \\prod_{i=1}^n\\left(\\frac{\\tz_i}{\\alpha_i}\\right)^{\\alpha_i}> 0 \\,\\,{\\rm and}\\,\\,v^k_0=\\|\\bbv^k\\| = \\prod_{i=1}^n(\\tv_i^k)^{\\alpha_i}\\,\\,\\forall k.\n\t\t\\end{equation}\n\n\t\tIf it holds that \\( v_0^k = 0 \\) infinitely often, by passing to a further subsequence, we may assume that \\( v_0^k = 0 \\) for all \\( k \\). Then we have in view of {Lemma \\ref{lemma:Lemma-2.11}} that\\vspace{-0.15 cm}\n\t\t\\[\\vspace{-0.15 cm}\n\t\t\\!\\|\\bm{v}^k - \\bm{w}^k\\| \\!=\\! \\frac1{\\|\\bm{z}\\|}|\\inProd{ \\tbz}{\\tbv^k}|\\!\\overset{\\rm (a)}=\\! \\frac1{\\|\\bm{z}\\|}\\sum_{i=1}^n\\tz_i\\tv^k_i\\!\\ge\\! \\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|} \\|\\tbv^k\\|_1\\!\\ge\\! \\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|} \\|\\tbv^k\\|\n\t\t\\!\\overset{\\rm (b)}=\\!\\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|} \\|\\bm{v}^k\\|,\n\t\t\\]\n\t\twhere (a) holds because \\( \\tv^k_i \\ge 0 \\) and \\( \\tz_i > 0 \\) for all \\( i \\) (see \\eqref{eq:z0-v0-notation}), and (b) holds since \\( \\|\\bbv^k\\| = 0 \\). Since\n\t\t\\( \\|\\bm{w}^k - \\bm{u}^k\\| = \\dist(\\bm{w}^k,\\Fr) \\le \\|\\bm{w}^k\\|\\le \\|\\bm{v}^k\\| \\) as a consequence of the properties of projections, we conclude from this and the above display that\n\t\t\\( \\|\\bm{v}^k - \\bm{w}^k\\| \\ge \\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|}\\|\\bm{w}^k - \\bm{u}^k\\| \\), contradicting \\eqref{eq:forcontradiction-1st-face}.\n\n\t\tThus, by considering a further subsequence if necessary, from now on, we assume\\vspace{-0.15 cm}\n\t\t\\begin{equation}\\label{eq:vk0-notation-nonzero}\\vspace{-0.15 cm}\n\t\t\tv^k_0=\\|\\bbv^k\\| = \\prod_{i=1}^n(\\tv_i^k)^{\\alpha_i} > 0 \\,\\,\\forall k.\n\t\t\\end{equation}\n\t\tUsing {Lemma \\ref{lemma:Lemma-2.11}}, we see that\\vspace{-0.15 cm}\n\t\t\\begin{equation}\\label{eq:dist-vk-wk}\\vspace{-0.15 cm}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\|\\bm{v}^k - \\bm{w}^k\\| & =\\frac1{\\|\\bm{z}\\|}|\\inProd{ \\bm{z}}{\\bm{v}^k}| = \\frac1{\\|\\bm{z}\\|}\\left|\\sum_{i=1}^{m}\\bz_i\\bv^k_i + \\sum_{i=1}^n\\tz_i\\tv^k_i\\right|             \\\\\n\t\t\t\t& = \\frac1{\\|\\bm{z}\\|}\\left|z_0v_0^k + \\sum_{i=1}^{m}\\bz_i\\bv^k_i - \\sum_{i=1}^n(-\\tz_i)\\tv^k_i - z_0v_0^k\\right|                                        \\\\\n\t\t\t\t& = \\frac{z_0v_0^k}{\\|\\bm{z}\\|}\\left| 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1\\right|  \\\\\n\t\t\t\t& = \\frac{z_0}{\\|\\bm{z}\\|}\\left( 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1\\right)v_0^k,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\twhere the last equality holds as \\( \\|z_0^{-1}\\bbz\\|= 1 \\), \\( \\|(v_0^k)^{-1}\\bbv^k\\|= 1 \\) and  \\( \\inProd{ z_0^{-1}\\tbz}{ (v_0^k)^{-1}\\tbv^k} \\ge 1 \\), thanks to \\eqref{eq:z0-v0-notation}, \\eqref{eq:vk0-notation-nonzero} and Lemma~\\ref{lemma:inequ-inprod-power-cone} applied with \\( \\bm{\\zeta} = -z_0^{-1}\\tbz \\).\n\n\t\tLet \\( \\bm{f} \\) be defined as in \\eqref{eq:1st-type-face-ray}. We consider two cases:\n\t\t\\begin{enumerate}[label={(\\Roman*)}]\n\t\t\t\\item \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}\\ge 0 \\) for all sufficiently large \\( k \\).\n\t\t\t\\item \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}< 0 \\) infinitely often.\n\t\t\\end{enumerate}\n\n\t\t(I): By passing to a further subsequence, we may assume that \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}\\ge 0 \\) for all \\( k \\). In this case, if we define\n\t\t\\[\n\t\t\\bm{Q} = \\bm{I}_{m + n} - \\frac{\\bm{z}\\bm{z}^{\\top}}{\\|\\bm{z}\\|^2} - \\frac{\\bm{f}\\bm{f}^{\\top}}{\\|\\bm{f}\\|^2},\n\t\t\\]\n\t\twhere \\( \\bm{f} \\) is as in \\eqref{eq:1st-type-face-ray}, then we see from {Lemma \\ref{lemma:Lemma-2.11}} and \\eqref{eq:z0-v0-notation} that\n\t\t\\begin{equation}\\label{eq:upbd-dist-uk-wk-two-parts}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\|\\bm{u}^k - \\bm{w}^k\\| & = \\|\\bm{Q}\\bm{v}^k\\| = v_0^k\\left\\|\\bm{Q}\\begin{bmatrix}\n\t\t\t\t\t(v_0^k)^{-1}\\bbv^k \\\\ (v_0^k)^{-1}\\tbv^k\n\t\t\t\t\\end{bmatrix}\\right\\| \\\\\n&\\overset{\\rm (a)}= v_0^k{\\Bigg\\|}\\bm{Q}\\begin{bmatrix}\n\t\t\t\t\t(v_0^k)^{-1}\\bbv^k \\\\ (v_0^k)^{-1}\\tbv^k\n\t\t\t\t\\end{bmatrix} \\!-\\! \\bm{Q}\\underbrace{\\begin{bmatrix}\n\t\t\t\t\t\t-z_0^{-1}\\bbz \\\\ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\n\t\t\t\t\\end{bmatrix}}_{z_0\\bm{f}}{\\Bigg\\|} \\\\\n\t\t\t\t& \\le v_0^k\\left[\\|(v_0^k)^{-1}\\bbv^k + z_0^{-1}\\bbz\\| + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\right],\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\twhere (a) holds because %\\( z_0\\bm{f} = \\begin{bmatrix}\n\t\t%\t\t-z_0^{-1}\\bbz^{\\top} & \\alpha\\circ(z_0\\tbz^{-1})^{\\top}\n\t\t%\t\\end{bmatrix}^{\\top}\\)  and\n\t\t\\( \\bm{Q}\\bm{f} = \\bm{0} \\) (an identity which is clear from the definitions).\n\n\t\tNext, in view of \\eqref{eq:z0-v0-notation}, we can apply {Lemma \\ref{lemma:Lemma-4.1}} to obtain \\( C_1 > 0 \\) and \\( \\epsilon_1>0 \\) so that {\\eqref{eq:inequ-p=q=2}} holds with \\( \\overline{\\bm{\\zeta}} = -z_0^{-1}\\bbz \\), i.e.,\n\t\t\\[\n\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_1\\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|^2\n\t\t\\]\n\t\twhenever \\( \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\le \\epsilon_1 \\) and \\( \\|\\bm{\\omega}\\|=1 \\). On the other hand, in view of the positivity of $1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}$ when $\\|\\bm{\\omega}\\|=1$ and $\\bm{\\omega} \\neq -z_0^{-1}\\bbz$ {(see Lemma \\ref{lemma:Lemma-4.1})}, we know that\n\\[\nC_2 := \\inf_{\\|\\bm{\\omega}\\|=1}\\{1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}} \\,|\\, \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\ge \\epsilon_1 \\} > 0.\n\\]\nThis together with the fact $\\|z_0^{-1}\\bbz\\|=1$ (see \\eqref{eq:z0-v0-notation}) implies that\n\t\t\\[\n\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_2 \\ge 0.25 C_2 \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|^2,\n\t\t\\]\n\t\twhenever \\( \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\ge \\epsilon_1 \\) and \\( \\|\\bm{\\omega}\\|=1 \\). We thus have (with $C_3 \\!:=\\! \\min\\{C_1,C_2/4\\}$)\n\\begin{equation}\\label{eq:keylemmarel}\n\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_3\\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|^2\\ \\ \\ {\\rm whenever}\\ \\ \\|\\bm{\\omega}\\|=1.\n\t\t\\end{equation}\n\n\t\tIn addition, noting \\eqref{eq:z0-v0-notation} again, we can apply Lemma~\\ref{lemma:inequ-inprod-power-cone} with \\( \\bm{\\zeta} = -z_0^{-1}\\tbz\\in {\\rm int}\\,\\R^n_- \\) to obtain \\( C_4 > 0 \\) and \\( \\epsilon > 0 \\) so that \\eqref{eq:inequ-inprod-power-cone} holds with \\( \\widetilde{\\bm{\\zeta}} = \\bm{\\alpha}\\circ (z_0\\tbz^{-1}) \\), i.e.,\n\t\t\\begin{equation}\\label{eq:keylemmare2}\n\t\t\t-1 + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}\\ge C_4\\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2\n\t\t\\end{equation}\n\t\twhenever \\( \\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\le \\epsilon \\), \\( \\bm{\\omega}\\in {\\rm int}\\,\\R^n_+ \\) and \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i}=1 \\).\n\n%Then we have in view of the positivity of $1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}$ when $\\bm{\\omega} \\neq -z_0^{-1}\\bbz$ (see Lemma \\ref{lemma:Lemma-4.1}) and the compactness of the unit sphere that there exists \\( C_3>0 \\) so that\n%\t\t\\begin{equation}\\label{eq:keylemmare3}\n%\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_3 \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\n%\t\t\\end{equation}\n%\t\twhenever \\( \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\ge \\epsilon \\) and \\( \\|\\bm{\\omega}\\|=1 \\).\n\nFurthermore, consider \\( h:\\R^n\\to \\R\\cup\\{\\infty\\} \\) defined by\n\t\t\\begin{equation}\\label{eq:def-h}\n\t\t\th(\\bm{\\omega}) = \\begin{cases}\n\t\t\t\t\\displaystyle\\frac{\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}-1}{\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|} & {\\rm if}\\,\\|\\bm{\\omega}-\\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\ge \\epsilon,\\,\\bm{\\omega}\\in \\Upsilon, \\\\\n\t\t\t\t\\infty                                                                                                    & {\\rm otherwise},\n\t\t\t\\end{cases}\n\t\t\\end{equation}\n\t\twhere \\( \\Upsilon= \\{\\bm{\\omega}\\in \\R^n_+\\,| \\,\\prod_{i=1}^n \\omega_i^{\\alpha_i}=1\\} \\). Then we have\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t& \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty}h(\\bm{\\omega}) = \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty, \\bm{\\omega}\\in \\Upsilon}\\frac{\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}-1}{\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|} \\\\\n\t\t\t& \\ge \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty,\\bm{\\omega}\\in \\R^n_+}\\frac{\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}-1}{\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|}\n\t\t\t\\overset{\\rm (a)}{\\ge} \\inf_{\\|\\bm{\\lambda}\\|=1,\\bm{\\lambda}\\in \\R^n_+}\\inProd{ z_0^{-1}\\tbz}{\\bm{\\lambda}} \\overset{\\rm (b)}\\ge \\min_{1\\le i\\le n}z_0^{-1}\\tz_i > 0.\n\t\t\\end{aligned}\n\t\t\\]\n\t\tHere (a) may be verified by multiplying both numerator and denominator of the left side by $1/\\|\\bm{\\omega}\\|$; (b) holds because \\( z_0^{-1}\\tz_i > 0 \\) for all \\( i \\) (see \\eqref{eq:z0-v0-notation}). Since \\( h \\) in \\eqref{eq:def-h} is also lower semicontinuous on any compact set and {\\bf is always positive},\\footnote{The positivity can be seen by applying Lemma~\\ref{lemma:inequ-inprod-power-cone} with $\\bm\\zeta = -z_0^{-1}\\tbz$.} it must then hold that \\( C_5:= \\inf h > 0 \\). In particular, this means that\n\t\t\\begin{equation}\\label{eq:keylemmare4}\n\t\t\t-1 + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}\\ge C_5\\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\n\t\t\\end{equation}\n\t\twhenever \\( \\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\ge \\epsilon \\), \\( \\bm{\\omega}\\in {\\rm int}\\,\\R^n_+ \\) and \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i}=1 \\).\n\n\t\tBy passing to suitable subsequences, we will end up with one of the following two cases:\n\t\t\\begin{enumerate}[label={Case \\arabic*:}]\n\t\t\t\\item \\( \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| \\le \\epsilon \\) for all \\( k \\). Then we have from \\eqref{eq:keylemmarel} (with $\\bm{\\omega}=(v_0^k)^{-1}\\bbv^k$) and \\eqref{eq:keylemmare2} (with $\\bm{\\omega}=(v_0^k)^{-1}\\tbv^k$) that for these \\( k \\)\n\t\t\t\\begin{equation*}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t& 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1          \\\\\n\t\t\t\t\t& \\ge \\min\\{C_3,C_4\\}(\\|(v_0^k)^{-1}\\bbv^k  + z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2).\n\t\t\t\t\\end{aligned}\n\t\t\t\\end{equation*}\n\t\t\tCombining this with \\eqref{eq:dist-vk-wk} and \\eqref{eq:upbd-dist-uk-wk-two-parts}, we see further that\n\t\t\t\\[\n\t\t\t\\begin{aligned}\n\t\t\t\t&\\|\\bm{v}^k - \\bm{w}^k\\|\\\\ & \\ge \\frac{z_0}{\\|\\bm{z}\\|}\\min\\{C_3,C_4\\}(\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2) v_0^k    \\\\\n\t\t\t\t& \\ge \\frac{z_0}{2\\|\\bm{z}\\|}\\min\\{C_3,C_4\\}(\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\| + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|)^2 v_0^k     \\\\\n\t\t\t\t& \\ge \\frac{z_0\\min\\{C_3,C_4\\}}{2\\|\\bm{z}\\|v_0^k}\\|\\bm{u}^k - \\bm{w}^k\\|^2\\ge \\frac{z_0\\min\\{C_3,C_4\\}}{2\\|\\bm{z}\\|\\eta}\\|\\bm{u}^k - \\bm{w}^k\\|^2,\n\t\t\t\\end{aligned}\n\t\t\t\\]\n\t\t\twhere the last inequality holds because \\( \\bm{v}^k\\in \\B(\\eta) \\). The above display contradicts \\eqref{eq:forcontradiction-1st-face} and hence Case 1 cannot happen.\n\n\t\t\t\\item \\( \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\ge \\epsilon \\) for all \\( k \\). Then we have from \\eqref{eq:keylemmarel} and \\eqref{eq:keylemmare4} that for these \\( k \\)\n\t\t\t\\begin{equation*}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t& 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1        \\\\\n\t\t\t\t\t& \\ge \\min\\{C_3,C_5\\}(\\|(v_0^k)^{-1}\\bbv^k  + z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|).\n\t\t\t\t\\end{aligned}\n\t\t\t\\end{equation*}\n\t\t\tUsing this together with \\eqref{eq:dist-vk-wk}, we deduce that for all large \\( k \\),\n\t\t\t\\[\n\t\t\t\\!\\|\\bm{v}^k - \\bm{w}^k\\| \\!\\ge\\! \\frac{z_0\\min\\{C_3,C_5\\}}{\\|\\bm{z}\\|}(\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|) v_0^k.\n\t\t\t\\]\n\t\t\tThis implies that\n\t\t\t\\begin{equation}\\label{eq:upbd-part-dist-uk-wk-case-2}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\|            & \\le M_1\\sqrt{(v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|}, \\\\\n\t\t\t\t\t\\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| & \\le M_1(v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|,\n\t\t\t\t\\end{aligned}\n\t\t\t\\end{equation}\n\t\t\twhere \\( M_1 := \\max\\left\\{\\left(\\frac{z_0}{\\|\\bm{z}\\|}\\min\\{C_3,C_5\\}\\right)^{-1},\\left(\\frac{z_0}{\\|\\bm{z}\\|}\\min\\{C_3,C_5\\}\\right)^{-1/2}\\right\\} \\). Using \\eqref{eq:upbd-part-dist-uk-wk-case-2} together with \\eqref{eq:upbd-dist-uk-wk-two-parts}, we obtain that\n\t\t\t\\begin{equation}\n\t\t\t\\begin{aligned}\\label{eqn:3Ms}\n\t\t\t\t\\|\\bm{u}^k - \\bm{w}^k\\| & \\le M_1v_0^k\\left[\\sqrt{(v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|} + (v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|\\right]   \\\\\n\t\t\t\t%& = M_1\\left(\\sqrt{v_0^k} + \\sqrt{\\|\\bm{v}^k-\\bm{w}^k\\|}\\right)\\sqrt{\\|\\bm{v}^k - \\bm{w}^k\\|}\\\\\n\t\t\t\t&\\stackrel{\\rm (a)}{\\le}M_1\\sqrt{\\eta} \\sqrt{\\|\\bm{v}^k-\\bm{w}^k\\|} + M_1\\|\\bm{v}^k - \\bm{w}^k\\| \\\\\n\t\t\t\t&\\stackrel{\\rm (b)}{\\le} 3M_1\\sqrt{\\eta}\\sqrt{\\|\\bm{v}^k - \\bm{w}^k\\|},\n\t\t\t\\end{aligned}\n\t\t\t\\end{equation}\n\t\t\twhere (a) holds since \\( \\bm{v}^k\\in \\B(\\eta) \\) wherefore $v_0^k \\leq \\eta$, and (b) holds because \\(\\|\\bm{w}^k\\| \\leq \\|\\bm{v}^k\\| \\le \\eta \\) (because the projection onto $\\stdCone$ is nonexpansive and $\\bm 0\\in \\stdCone$), wherefore\n\t\t\t$$\n\t\t\t\\|\\bm{w}^k-\\bm{v}^k\\| = \\sqrt{\\|\\bm{w}^k-\\bm{v}^k\\|}\\sqrt{\\|\\bm{w}^k-\\bm{v}^k\\|} \\leq 2\\sqrt{\\eta}\\sqrt{\\|\\bm{w}^k-\\bm{v}^k\\|}.\n\t\t\t$$\n\t\t\tAltogether, \\eqref{eqn:3Ms} contradicts \\eqref{eq:forcontradiction-1st-face} and hence Case 2 cannot happen.\n\t\t\\end{enumerate}\n\t\tSummarizing the above discussions, we see that Case (I) cannot happen.\n\n\t\t(II):  By passing to a further subsequence, we may assume that \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}< 0 \\) for all \\( k \\). This together with the definition of \\( \\bm{f} \\) gives\n\t\t\\[\n\t\t\\frac{v_0^k}{z_0}[\\inProd{ -z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})}{ (v_0^k)^{-1}\\tbv^k}] < 0.\n\t\t\\]\n\t\tSince $\\frac{v_0^k}{z_0} > 0$, we deduce that $\\inProd{ -z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})}{ (v_0^k)^{-1}\\tbv^k} < 0$ for all $k$. Then it must hold that\n\t\t\\[\n\t\t\\lim_{k\\to \\infty}\\|(v_0^k)^{-1}\\bbv^k + z_0^{-1}\\bbz\\|+\\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\neq 0;\n\t\t\\]\notherwise, we have $(v_0^k)^{-1}\\bbv^k\\to -z_0^{-1}\\bbz$ and $(v_0^k)^{-1}\\tbv^k\\to \\bm{\\alpha}\\circ(z_0\\tbz^{-1})$, which further gives $\\inProd{ -z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})}{ (v_0^k)^{-1}\\tbv^k}\\to \\|z_0^{-1}\\bbz\\|^2+\\|\\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2 = \\|z_0\\bm f\\|^2 > 0$, a contradiction.\n\n\t\tConsequently, there exists $\\epsilon > 0$ such that for all sufficiently large $k$,\n\t\t\\begin{equation}\\label{eq:sum-out-of-neighbor}\n\t\t\t\\|(v_0^k)^{-1}\\bbv^k + z_0^{-1}\\bbz\\|+\\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| \\ge \\epsilon.\n\t\t\\end{equation}\n\t\tConsider the function \\( G:\\R^{m+n}\\to \\R\\cup\\{\\infty\\} \\) defined by\n\t\t\\[\n\t\tG(\\bm{\\xi},\\bm{\\omega}) := \\begin{cases}\n\t\t\t\\displaystyle \\frac{|\\inProd{ z_0^{-1}\\bbz}{ \\bm{\\xi}} + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}|}{\\sqrt{1+\\|\\bm{\\omega}\\|^2}} & {\\rm if}\\,(\\bm{\\xi},\\bm{\\omega})\\in \\Xi,\\,\\|\\bm{\\xi}\\|=1,\\,{\\rm and}\\,\\bm{\\omega}\\in \\Upsilon, \\\\\n\t\t\t\\infty                                                                                                                       & {\\rm otherwise},\n\t\t\\end{cases}\n\t\t\\]\n\t\twhere \\( \\Upsilon = \\{\\bm{\\omega}\\in \\R^n_+\\,| \\,\\prod_{i=1}^n \\omega_i^{\\alpha_i}=1\\} \\) and \\( \\Xi= \\{(\\bm{\\xi},\\bm{\\omega})\\,| \\,\\|\\bm{\\xi} + z_0^{-1}\\bbz\\|+\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| \\ge \\epsilon\\} \\).\n\t\tSince \\( \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\xi}} + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}} = 1+\\inProd{ z_0^{-1}\\bbz}{ \\bm{\\xi}} - \\inProd{ z_0^{-1}(-\\tbz)}{ \\bm{\\omega}}-1 \\), we see from \\eqref{eq:z0-v0-notation}, \\eqref{eq:sum-out-of-neighbor}, {Lemma \\ref{lemma:Lemma-4.1}} and Lemma~\\ref{lemma:inequ-inprod-power-cone} that \\( G \\) is never zero. Moreover, it is clearly lower semicontinuous on any compact set, and\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\!\\liminf_{\\|(\\bm{\\xi},\\bm{\\omega})\\|\\to \\infty}G(\\bm{\\xi},\\bm{\\omega}) \\!=\\!\\!\\! \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty,\\bm{\\omega}\\in \\Upsilon}\\!\\!\\frac{|\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}|}{\\sqrt{1+\\|\\bm{\\omega}\\|^2}}\\!\\overset{\\rm (a)}\\ge\\!\\!\\! \\inf_{\\|\\bm{\\lambda}\\|=1,\\bm{\\lambda}\\in \\R^{n}_+}\\!\\!|\\inProd{ z_0^{-1}\\tbz}{\\bm{\\lambda}}|\\!\\overset{\\rm (b)}\\ge\\!\\! \\min_i |z_0^{-1}\\tz_i| > 0,\n\t\t\\end{aligned}\n\t\t\\]\n\t\twhere (a) may be verified by multiplying numerator and denominator by $1/\\|\\bm{\\omega}\\|$ and (b) holds since \\( z_0^{-1}\\tz_i > 0 \\) for all \\( i \\).\n\t\tThus, \\( C_6:= \\inf G > 0 \\) and we have for all large \\( k \\),\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\frac{\\|\\bm{v}^k - \\bm{w}^k\\|}{\\|\\bm{u}^k - \\bm{w}^k\\|} & \\overset{\\rm (a)}= \\frac{\\|\\bm{v}^k - \\bm{w}^k\\|}{\\|\\bm{w}^k\\|} \\overset{\\rm (b)}\\ge \\frac{\\|\\bm{v}^k - \\bm{w}^k\\|}{\\|\\bm{v}^k\\|} \\\\\n&\\overset{\\rm (c)}=\n\t\t\t\\frac{z_0}{\\|\\bm{z}\\|}\\frac{|\\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ z_0^{-1}\\tbz}{ (v_0^k)^{-1}\\tbv^k}|v_0^k}{\\sqrt{(v^k_0)^2 + \\|\\tbv^k\\|^2}}                                                                                                            \\\\\n\t\t\t& = \\frac{z_0}{\\|\\bm{z}\\|}\\frac{|\\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ z_0^{-1}\\tbz}{ (v_0^k)^{-1}\\tbv^k}|}{\\sqrt{1 + \\|(v_0^k)^{-1}\\tbv^k\\|^2}}\\overset{\\rm (d)}\\ge \\frac{C_6 z_0}{\\|\\bm{z}\\|},\n\t\t\\end{aligned}\n\t\t\\]\n\t\twhere (a) follows from {Lemma \\ref{lemma:Lemma-2.11}}, which states that \\( \\bm{u}^k = \\bm{0} \\) in Case (II), (b) holds because the projection onto the cone is nonexpansive and $\\bm 0$ is in the cone, (c) follows from \\eqref{eq:dist-vk-wk} and (d) follows from \\eqref{eq:sum-out-of-neighbor}, \\eqref{eq:vk0-notation-nonzero} and the definitions of \\( G \\) and \\( C_6 \\). The above display contradicts \\eqref{eq:forcontradiction-1st-face}. Thus, Case (II) also cannot happen.\n\n\t\tSummarizing the above, we conclude that \\eqref{eq:forcontradiction-1st-face} cannot happen. Thus, in view of {\\cite[Lemma 3.12]{LiLoPo20}}, we must indeed have \\( \\gamma_{\\bm{z},\\eta} \\in (0,\\infty] \\) and that the desired error bound follows from {\\cite[Theorem 3.10]{LiLoPo20}}.\n\t\\end{proof}\n\n\\begin{theorem}\\label{thm:xface}\n\t\tLet \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz = \\bm{0} \\) and let \\( \\Fhatz := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Let \\( \\stdIndex := \\{i\\,| \\,\\tz_i > 0\\} \\),\\footnote{Since \\( \\bbz = \\bm{0} \\) and \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\), we must have \\( \\emptyset\\neq \\stdIndex \\subsetneq \\{1,2,\\ldots,n\\} \\).} \\( \\eta > 0 \\) and define \\( \\beta:= \\sum_{i\\in \\stdIndex}\\alpha_i \\) and\n\t\t\\begin{equation}\\label{eq:gamma-beta-2nd-type-face}\n\t\t \\!\\gamma_{\\bm{z},\\eta} \\!:=\\! \\inf_{\\bm{v}}\\left\\{\\frac{\\|\\bm{v} - \\bm{w}\\|^\\beta}{\\|\\bm{u} - \\bm{w}\\|}\\,\\bigg| \\,\n\\begin{array}{c}\n\\bm{v}\\in \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fhatz,\\,\\bm{w} = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}),\\\\\n\\bm{u} = P_{\\Fhatz}(\\bm{w}),\\,\\bm{u} \\neq \\bm{w}\n\\end{array}\\right\\}.\n\t\t\\end{equation}\n\t\tThen it holds that \\( \\gamma_{\\bm{z},\\eta}\\in (0,\\infty] \\) and that\n\t\t\\[\n\t\t\\dist(\\bm{q},\\Fhatz)\\le \\max\\{2\\eta^{1-\\beta},2\\gamma_{\\bm{z},\\eta}^{-1}\\}\\cdot\\dist(\\bm{q},\\gK)^\\beta\\,\\,\\text{ whenever }  \\bm{q}\\in \\{\\bm{z}\\}^\\perp\\cap \\B(\\eta) .\n\t\t\\]\n\t\\end{theorem}\n\n\\begin{proof}\n\t\t%\tIf \\( \\gamma_{\\bm{z},\\eta} = 0 \\), in view of \\cite[Lemma 3.12]{LiLoPo20}, there exist \\( \\widehat{\\bm{v}}\\in \\Fhatz \\) and a sequence \\( \\{\\bm{v}^k\\}\\subset \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fhatz \\) such that\n\t\t%\t\\begin{equation}\\label{eq:forcontradiction-2nd-face}\n\t\t%\t\t\\lim_{k\\to \\infty}\\bm{v}^k = \\lim_{k\\to \\infty}\\bm{w}^k = \\widehat{\\bm{v}}\\,\\,{\\rm and}\\,\\,\\lim_{k\\to\\infty}\\frac{\\|\\bm{w}^k - \\bm{v}^k\\|^{\\beta }}{\\|\\bm{w}^k - u^k\\|} = 0,\n\t\t%\t\\end{equation}\n\t\t%\twhere \\( \\bm{w}^k = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}^k) \\), \\( \\bm{u}^k = P_{\\Fhatz}(\\bm{w}^k) \\) and \\( \\bm{u}^k\\neq \\bm{w}^k \\).\n\n\t\t{In view of \\cite[Theorem 3.10]{LiLoPo20}}, we need only show that $\\gamma_{\\bm{z},\\eta} > 0$.\n%\t\t$$\n%\t\t0<\\gamma_{\\bm{z},\\eta} = \\inf_{\\bm{v}} \\left\\{ \\frac{\\|\\bm{w} -\\bm{v} \\|^\\beta}{\\|\\bm{w} -\\bm{u} \\| } \\Big|\\bm{v} \\in \\bd \\stdCone \\cap \\B(\\eta) \\setminus \\stdFace,\\bm{w} = P_{\\{\\bm{z}\\}^{\\perp }}(\\bm{v}), \\bm{u} = P_{\\stdFace}(\\bm{w}), \\bm{w} \\neq\\bm{u} \\right\\}.\n%\t\t$$\n\t\tTo that end, let \\( \\bm{v}\\in \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fhatz \\), \\( \\bm{w} = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}) \\), \\( \\bm{u} = P_{\\Fhatz}(\\bm{w}) \\), and \\( \\bm{u} \\neq \\bm{w} \\). Then a direct computation shows\n\t\tthat\n\t\t\\begin{equation}\\label{eq:dist-w-v}\n\t\t\t\\|\\bm{w}- \\bm{v}\\| = \\frac1{\\|\\bm{z}\\|}|\\inProd{ \\bm{z}}{\\bm{v}}| \\overset{\\rm (a)}= \\frac1{\\|\\bm{z}\\|}\\sum_{i\\in \\stdIndex}\\tz_i\\tv_i \\overset{\\rm (b)}\\ge \\frac{\\min_{i\\in \\stdIndex}\\tz_i}{\\|\\bm{z}\\|}\\sum_{i\\in \\stdIndex}\\tv_i\n\t\t\t\\overset{\\rm (c)}\\ge \\frac{\\min_{i\\in \\stdIndex}\\tz_i}{\\|\\bm{z}\\|}\\|\\tbv_{\\stdIndex}\\|,\n\t\t\\end{equation}\n\t\twhere (a), (b) and (c) hold because \\( \\tv_i\\ge 0 \\) and \\( \\tz_i > 0 \\) for all \\( i\\in \\stdIndex \\), with \\( \\|\\tbv_{\\stdIndex}\\|:= \\sqrt{\\sum_{i\\in \\stdIndex}\\tv^2_i} \\) (note that \\( \\stdIndex \\neq \\emptyset \\), thanks to \\( \\bbz = 0 \\) and \\( \\bm{z}\\neq \\bm{0} \\)). Next, notice that \\( \\bm{w} = \\bm{v} - \\frac{\\inProd{ \\bm{z}}{\\bm{v}}}{\\|\\bm{z}\\|^2}\\bm{z} \\). Using this and the definitions of \\( \\bm{z} \\) and \\( \\stdIndex \\), we deduce that\n\t\t\\begin{equation}\\label{eq:form-w-by-v}\n\t\t\t\\bbw = \\bbv,\\,\\,\\,\\tw_i = \\tv_i - \\frac{\\tz_i}{\\|\\bm{z}\\|^2}\\bigg(\\sum_{j\\in \\stdIndex}\\tz_j\\tv_j\\bigg)\\,\\,\\,\\, \\forall i\\in \\stdIndex\\,\\,\\,\\,\\,\\,\\,\\mbox{and\\,\\,\\,\\,\\,\\,\\,} \\tw_i = \\tv_i \\ge 0\\,\\,\\,\\,\\forall i \\notin \\stdIndex.\n\t\t\\end{equation}\n\t\tIn view of this and the definition of \\( \\Fhatz \\) in \\eqref{eq:2nd-type-face-d-dim}, we see that \\( \\tu_i = \\tw_i \\) whenever \\( i\\notin \\stdIndex \\), and hence\n\t\t\\begin{equation}\\label{eq:dist-w-u}\n\t\t\t\\|\\bm{w} - \\bm{u}\\| = \\sqrt{\\|\\bbw\\|^2 + \\sum_{i\\in \\stdIndex}\\tw^2_i} \\le \\sqrt{\\|\\bbv\\|^2 + n(1+\\sqrt{n})^2\\|\\tbv_{\\stdIndex}\\|^2},\n\t\t\\end{equation}\n\t\twhere the inequality follows from \\eqref{eq:form-w-by-v} and the fact that for each \\( i\\in \\stdIndex \\),\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t|\\tw_i| & = \\bigg|\\tv_i - \\frac{\\tz_i}{\\|\\bm{z}\\|^2}\\bigg(\\sum_{j\\in \\stdIndex}\\tz_j\\tv_j\\bigg)\\bigg|\\le \\bigg(1 + \\frac{|\\tz_i|}{\\|\\bm{z}\\|^2}\\sum_{j\\in \\stdIndex}|\\tz_j|\\bigg)\\|\\tbv_{\\stdIndex}\\| \\\\\n\t\t\t& \\le \\left(1 + \\frac{\\sqrt{n}|\\tz_i|}{\\|\\bm{z}\\|}\\right)\\|\\tbv_{\\stdIndex}\\| \\le (1+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\|.\n\t\t\\end{aligned}\n\t\t\\]\n\t\tNext, note that we have\n\t\t\\begin{equation}\\label{eq:upbd-norm-bar-v}\n\t\t\t\\|\\bbv\\|=%\n\t\t\t\\prod_{i=1}^n(\\tv_i)^{\\alpha_i} %\n\t\t\t= \\prod_{i\\notin \\stdIndex}\\tv_i^{\\alpha_i} \\cdot \\prod_{i\\in \\stdIndex}\\tv_i^{\\alpha_i}%\n\t\t\t\\le \\prod_{i\\notin \\stdIndex}\\eta^{\\alpha_i} \\cdot\\prod_{i\\in \\stdIndex}\\|\\tbv_{\\stdIndex}\\|^{\\alpha_i}%\n\t\t\t= \\eta^{1-\\beta} \\|\\tbv_{\\stdIndex}\\|^\\beta,\n\t\t\\end{equation}\n\t\twhere the inequality holds because \\( \\bm{v}\\in \\B(\\eta) \\). Combining \\eqref{eq:dist-w-v}, \\eqref{eq:dist-w-u} and \\eqref{eq:upbd-norm-bar-v}, we deduce\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\|\\bm{w} - \\bm{u}\\| & \\le \\sqrt{\\|\\bbv\\|^2 + n(1+\\sqrt{n})^2\\|\\tbv_{\\stdIndex}\\|^2} \\le \\|\\bbv\\| + (n+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\|                                                                                                     \\\\\n\t\t\t& \\le \\eta^{1-\\beta} \\|\\tbv_{\\stdIndex}\\|^\\beta + (n+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\| = (\\eta^{1-\\beta}  + (n+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\|^{1-\\beta})\\|\\tbv_{\\stdIndex}\\|^\\beta                                   \\\\\n\t\t\t& \\overset{\\rm (a)}\\le \\eta^{1-\\beta}(n+1 + \\sqrt{n})\\|\\tbv_{\\stdIndex}\\|^\\beta \\overset{\\rm (b)}{\\le}\\frac{\\eta^{1-\\beta}(n+1 + \\sqrt{n})\\|\\bm{z}\\|^\\beta}{\\left(\\min_{i\\in_{\\stdIndex}}\\tz_i\\right)^\\beta}\\|\\bm{w}-\\bm{v}\\|^\\beta.\n\t\t\\end{aligned}\n\t\t\\]\n\t\tHere (a) holds since \\( \\bm{v}\\in \\B(\\eta) \\) and \\( \\beta \\in (0,1) \\); (b) is true because of \\eqref{eq:dist-w-v}.\n\t\tThus, \\( \\gamma_{\\bm{z},\\eta}\\ge \\frac{\\left(\\min_{i\\in \\stdIndex}\\tz_i\\right)^\\beta}{\\eta^{1-\\beta}(n+1 + \\sqrt{n})\\|\\bm{z}\\|^\\beta} > 0 \\), and the desired error bound {follows from \\cite[Theorem 3.10]{LiLoPo20}}.\n\t\\end{proof}\n\n\\begin{corollary}\\label{corollary:1-FRFs-power-cones}\n\t\tConsider \\( \\gK \\) and its dual cone \\( \\dgK \\).\n\t\t\\begin{enumerate}[label=(\\roman*)]\n\t\t\t\\item\\label{3.6i} Let \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz\\neq \\bm{0} \\) and let \\( \\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Let \\( \\gamma_{\\bm{z},t} \\) be defined as in \\eqref{eq:gamma-1/2-1st-type-face}. Then the function \\( \\psi_{\\gK,\\bm{z}}:\\R_+\\times \\R_+\\to \\R_+ \\) given by\\vspace{-0.15 cm}\n\t\t\t\\begin{equation}\\label{eq:1-FRF-1st-face}\\vspace{-0.15 cm}\n\t\t\t\t\\!\\!\\psi_{\\gK,\\bm{z}}(\\epsilon,t) \\!:=\\! \\max \\left\\{\\epsilon,\\epsilon/\\|\\bm{z}\\| \\right\\} + \\max\\{2\\sqrt{t},2\\gamma_{\\bm{z},t}^{-1}\\}(\\epsilon +\\max \\left\\{\\epsilon,\\epsilon/\\|\\bm{z}\\| \\right\\} )^\\frac12\n\t\t\t\\end{equation}\n\t\t\tis a one-step facial residual function for \\( \\gK \\) and \\( \\bm{z} \\).\n\t\t\t\\item Let \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz = \\bm{0} \\) and let \\( \\Fhatz := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Let \\( \\gamma_{\\bm{z},t} \\) be defined as in \\eqref{eq:gamma-beta-2nd-type-face}, where \\( \\beta:= \\sum_{i:\\tz_i > 0}\\alpha_i \\). Then the function \\( \\psi_{\\gK,\\bm{z}}:\\R_+\\times \\R_+\\to \\R_+ \\) given by\\vspace{-0.15 cm}\n\t\t\t\\begin{equation}\\label{eq:1-FRF-2nd-face}\\vspace{-0.15 cm}\n\t\t\t\t\\!\\!\\psi_{\\gK,\\bm{z}}(\\epsilon,t) \\!:= \\!\\max\\! \\left\\{\\epsilon,\\epsilon/\\|\\bm{z}\\| \\right\\} + \\max\\{2t^{1-\\beta}\\!,2\\gamma_{\\bm{z},t}^{-1}\\}(\\epsilon +\\max \\left\\{\\epsilon,\\epsilon/\\|\\bm{z}\\| \\right\\} )^\\beta\n\t\t\t\\end{equation}\n\t\t\tis a one-step facial residual function for \\( \\gK \\) and \\( \\bm{z} \\).\n\t\t\\end{enumerate}\n\t\\end{corollary}\n\n\\begin{proof}\n\t\tAs is shown in Section \\ref{subsec:facial-structure}, all the proper exposed faces of the generalized power cone are polyhedral.\n\t\tThen the process of facial reduction needs at most one step to reach the PPS condition. Hence, \\( \\dpps(\\gK, \\subSpace + \\bm{a}) \\leq 1 \\).\n\t\tThis shows item \\ref{item:error-bound-power-cone-1}.\n\n\t\tIf \\( \\dpps(\\gK, \\subSpace + \\bm{a}) = 0 \\), i.e., \\eqref{eq:conic-feasibility-problem} satisfies the PPS condition, then by \\cite[Corollary 3]{BBL99}, a Lipschitzian error bound holds.\n\t\tThis shows item \\ref{item:error-bound-power-cone-2}.\n\n\t\tNext, let \\( \\dpps(\\gK, \\subSpace + \\bm{a}) = 1 \\); i.e., we need one step to reach the PPS condition.\n\t\tIn this case, the error bound depends on the exposed face \\( \\stdFace \\) that contains the feasible region.\n\t\tIf \\(\\stdFace =  \\Fr \\), then by Corollary \\ref{corollary:1-FRFs-power-cones}\\ref{3.6i}, we conclude that a H\\\"olderian error bound with exponent \\( 1 / 2 \\) holds.\n\t\tRemark \\ref{remark:F-r-order-1/2} implies that \\( \\mathfrak{g} = | \\cdot | ^{\\frac{1}{2}}\\) satisfies the asymptotic optimality criterion for \\( \\gK \\) and \\( \\bm{z} \\) with \\( \\bbz \\neq \\bm{0} \\).\n\t\tHence, by {\\cite[Theorem 3.2]{LiLoPo21}}, the obtained H\\\"olderian error bound with exponent \\( 1 / 2 \\) is the best error bound.\n\n\t\tIf \\( \\stdFace = \\Fhatz \\) with \\( \\bm{z} \\in \\dgK \\cap \\subSpace^{\\perp } \\cap \\{\\bm{a}\\}^{\\perp } \\), then using Corollary \\ref{corollary:1-FRFs-power-cones}(ii), we conclude that a H\\\"olderian error bound with exponent \\( \\beta := \\sum _{i \\in \\stdIndex} \\alpha _i \\) holds, where \\( \\stdIndex = \\{i\\,| \\,\\tz_i > 0\\} \\).\n\t\tThe optimality of this error bound comes from Remark \\ref{remark:F-z-order-beta} and {\\cite[Theorem 3.2]{LiLoPo21}}.\nIf \\( \\stdFace = \\{\\bm{0}\\} \\), which means the feasible region is \\( \\{\\bm{0}\\} \\), then a Lipschitzian error bound holds automatically and it is naturally tight, see \\cite[Proposition~27]{L17}.\n\t\\end{proof}","units":[{"raw_tex":"\\begin{lemma}\\label{lemma:inequ-inprod-power-cone}\n\t\tLet \\( n\\ge 2 \\) and \\( \\bm{\\alpha} = (\\alpha_1,\\ldots,\\alpha_n)\\in (0,1)^n \\) with \\( \\sum_{i=1}^n\\alpha_i = 1 \\). Let \\( \\bm{\\zeta}\\in {\\rm int}\\,\\R_{-}^n \\) satisfy \\( \\prod_{i=1}^n (-\\zeta_i/\\alpha_i)^{\\alpha_i} = 1 \\). Define \\( \\widetilde{\\bm{\\zeta}} := -\\bm\\alpha\\circ \\bm{\\zeta}^{-1} \\), where \\( \\circ \\) is the Hadamard product and the inverse is taken componentwise. Then there exist \\( C > 0 \\) and \\( \\epsilon > 0 \\) so that\n\t\t\\begin{equation}\\label{eq:inequ-inprod-power-cone}\n\t\t\t-1 - \\inProd{\\bm{\\zeta}}{\\bm{\\omega}} \\ge C\\|\\bm{\\omega} - \\widetilde{\\bm{\\zeta}}\\|^2 \\,\\,{whenever}\\,\\bm{\\omega}\\in {\\rm int}\\,\\R^n_+,\\,\\|\\bm{\\omega} - \\widetilde{\\bm{\\zeta}}\\|\\le \\epsilon {\\,and\\,} \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1.\n\t\t\\end{equation}\n\t\tMoreover, for any \\( \\bm{\\omega}\\in {\\rm int}\\,\\R_+^n \\) satisfying \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\), it holds that \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}\\le-1 \\); furthermore, we have \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}=-1 \\) if and only if \\( \\bm{\\omega} = \\widetilde{\\bm{\\zeta}} \\).\n\t\\end{lemma}","start":24319,"end":25428},{"raw_tex":"\\begin{proof}\n\t\tFor each \\( i \\), we see from the Taylor series of \\( \\ln(\\cdot) \\) at \\( \\widetilde{\\zeta}_i > 0 \\) that\n\t\t\\begin{equation*}\n\t\t\t\\ln(\\omega_i) = \\ln(\\widetilde{\\zeta}_i) + \\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) - \\widetilde{\\zeta}_i^{-2}(\\omega_i - \\widetilde{\\zeta}_i)^2 + O(|\\omega_i - \\widetilde{\\zeta}_i|^3)\\,\\,{\\rm as}\\,\\,\\omega_i \\to \\widetilde{\\zeta}_i, \\,\\,\\omega_i > 0.\n\t\t\\end{equation*}\n\t\tThus, there exist \\( \\epsilon_i > 0 \\) and \\( c_i > 0 \\) so that\n\t\t\\[\n\t\t\\ln(\\widetilde{\\zeta}_i) \\ge \\ln(\\omega_i) - \\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) + c_i (\\omega_i - \\widetilde{\\zeta}_i)^2\\,\\,\\,\\mbox{whenever} \\,|\\omega_i - \\widetilde{\\zeta}_i|\\le \\epsilon_i \\,{\\rm and}\\,\\omega_i > 0.\n\t\t\\]\n\t\tLet \\( \\epsilon := \\min\\limits_{1\\le i\\le n}\\epsilon_i > 0 \\). Multiplying both sides of the above inequality by \\( \\alpha_i \\) and summing the resulting inequalities from \\( i = 1 \\) to \\( n \\), we see that whenever \\( \\bm{\\omega}\\in {\\rm int}\\,\\R^n_+ \\) satisfies \\( \\|\\bm{\\omega} - \\widetilde{\\bm{\\zeta}}\\|\\le \\epsilon \\) and \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\), we have\n\t\t\\begin{align}\n\t\t\t&0  \\overset{\\rm (a)}= \\sum_{i=1}^n\\alpha_i\\ln(\\widetilde{\\zeta}_i)\\ge \\sum_{i=1}^n\\alpha_i\\ln(\\omega_i) - \\sum_{i=1}^n\\alpha_i\\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) + \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2 \\notag\\\\\n\t\t\t& \\overset{\\rm (b)}= - \\sum_{i=1}^n\\alpha_i\\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) + \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2\n\t\t\t\\overset{\\rm (c)}= - \\sum_{i=1}^n\\alpha_i\\widetilde{\\zeta}_i^{-1}\\omega_i \\!+\\! 1 \\!+\\! \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2                                                                                                                          \\notag\\\\\n\t\t\t& = \\sum_{i=1}^n\\zeta_i\\omega_i + 1 + \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2,\\notag\n\t\t\\end{align}\n\t\twhere (a) and (b) hold because \\( \\prod_{i=1}^n\\widetilde{\\zeta}_i^{\\alpha_i} = \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\), (c) uses the fact that \\( \\sum_{i=1}^n\\alpha_i = 1 \\), and the last equality follows from the definition of \\( \\widetilde{\\bm{\\zeta}} \\). Rearranging the above inequality, we conclude that \\eqref{eq:inequ-inprod-power-cone} holds with \\( C = \\min\\limits_{1\\le i\\le n}\\alpha_ic_i > 0 \\).\n\n\t\tNext, let \\( \\bm{\\omega}\\in {\\rm int}\\,\\R_+^n \\) satisfy \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\). Then \\( (-1,\\bm{\\omega})\\in \\K \\). Recall from the assumption that \\( (1,-\\bm{\\zeta})\\in \\dK \\). From these we deduce \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}\\le-1 \\). If \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}=-1 \\), then\n\t\t\\[\n\t\t\\sum_{i=1}^n \\alpha_i\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)\\omega_i = \\sum_{i=1}^n (-\\zeta_i)\\omega_i = 1 = \\prod_{i=1}^n\\omega_i^{\\alpha_i} = \\prod_{i=1}^n\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)^{\\alpha_i}\\prod_{i=1}^n\\omega_i^{\\alpha_i} .\n\t\t\\]\n\t\tTaking \\( \\ln \\) on both sides of the above equality, we see that\n\t\t\\[\n\t\t\\ln\\left[\\sum_{i=1}^n \\alpha_i\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)\\omega_i\\right] = \\sum_{i=1}^n\\alpha_i \\ln\\left[\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)\\omega_i\\right].\n\t\t\\]\n\t\tSince \\( \\ln \\) is strictly concave and \\( \\alpha_i \\in (0,1) \\) for all \\( i \\), we conclude that there exists \\( c > 0 \\) so that \\( \\omega_i\\cdot(-\\zeta_i/\\alpha_i) = c \\) for all \\( i \\). This, together with the facts that\n\t\t\\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = \\prod_{i=1}^n (-\\zeta_i/\\alpha_i)^{\\alpha_i} = 1 \\) and \\( \\sum_{i=1}^n\\alpha_i = 1 \\), gives \\( c = 1 \\). It thus follows that \\( \\bm{\\omega} = \\widetilde{\\bm{\\zeta}} \\). Conversely, it is routine to check that if \\( \\bm{\\omega} = \\widetilde{\\bm{\\zeta}} \\), then \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\) and \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}=-1 \\).\n\t\\end{proof}","start":25430,"end":29267},{"raw_tex":"\\begin{proposition}[Proper nontrivial exposed faces of \\( \\gK \\)]\\label{prop:facial-stru-gK}\n\t\tLet \\( \\bm{z} = (\\bbz,\\tbz)\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\).\n\t\t\\begin{enumerate}[label={(\\roman*)}]\n\t\t  \\item \\label{item:case-I-facial-structure} If \\( \\bbz \\neq \\bm{0} \\), then \\( \\bm{z} \\) exposes the following one-dimensional face: %\\( \\Fr \\), where\n\t\t\t\t\\begin{equation}\\label{eq:1st-type-face-ray}\n\t\t\t\t  \\displayindent0pt\n\t\t\t\t  \\displaywidth\\textwidth\n\t\t\t\t  \\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK =  \\{t \\bm{f}\\in \\R^{m+n}\\,| \\,t\\ge 0\\}\\,\\,\\mbox{with} \\,\\bm{f}= (-\\bbz/\\|\\bbz\\|^2,\\bm{\\alpha}\\circ \\tbz^{-1}),\n\t\t\t\t\\end{equation}\n\t\t\t\twhere the inverse is taken componentwise.\n\n\t\t  \\item \\label{item:case-II-facial-structure} If \\( \\bbz = \\bm{0} \\), then \\( \\bm{z} \\) exposes the following face of dimension \\( n - | \\stdIndex | \\):\n\t\t\t\t\\begin{equation}\\label{eq:2nd-type-face-d-dim}\n\t\t\t\t  \\displayindent0pt\n\t\t\t\t  \\displaywidth\\textwidth\n\t\t\t\t  \\Fhatz := \\{\\bm{z}\\}^\\perp \\cap \\gK = \\{\\bm{x} = (\\bbx,\\tbx)\\in \\R_+^{m+n}\\,| \\,\\bbx = \\bm{0},\\,\\tx_i = 0\\ \\forall i\\in \\stdIndex\\},\n\t\t\t\t\\end{equation}\n\t\t\t\twhere \\( \\stdIndex := \\{i \\,|\\, \\tz_i > 0\\}\\neq \\emptyset \\) and \\( | \\stdIndex | \\) denotes the cardinality of $\\stdIndex$.\n\t\t\\end{enumerate}\n\t  \\end{proposition}","start":30512,"end":31767},{"raw_tex":"\\begin{proof}\n\t  \\ref{item:case-I-facial-structure}: Notice that \\( \\bm{x} = (\\bbx,\\tbx)\\in \\{\\bm{z}\\}^\\perp\\cap \\gK\\backslash \\{\\bm{0}\\} \\) if and only if \\( \\bm{x}\\in \\bd\\gK \\), \\( \\bm{x} \\neq \\bm{0} \\) and %satisfies\n\t  \\begin{equation}\\label{eq:inprod=0-case-I}\n\t\t\\inProd{\\bbz}{\\bbx} + \\inProd{ \\tbz}{\\tbx} = 0.\n\t  \\end{equation}\n\t  The above relation yields\n\t  \\begin{equation}\\label{eq:inequ-sum-prod-power-case-I}\n\t\t\\sum_{i=1}^n \\tz_i\\tx_i = -\\inProd{\\bbz}{\\bbx}\\le \\|\\bbz\\| \\|\\bbx\\| \\le \\prod_{i=1}^n\\left(\\frac{\\tx_i\\tz_i}{\\alpha_i}\\right)^{\\alpha_i},\n\t  \\end{equation}\n\t  where the last inequality follows from the definition of $\\gK$ in \\eqref{eq:gpowercones}.\n\n\t  Note that \\( \\tx_i \\) cannot be \\textbf{all} zero, for otherwise \\( \\bbx \\) will also be zero since \\( \\bm{x}\\in \\bd\\gK \\), which contradicts \\( \\bm{x}\\neq \\bm{0} \\). In addition, we must have \\( \\tz_i > 0 \\) for all \\( i \\) because \\( \\bbz \\neq \\bm{0} \\) and \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\). Using these observations, we have $\\sum_{i=1}^n \\tz_i\\tx_i >0$. Combining this with \\eqref{eq:inequ-sum-prod-power-case-I}, we deduce that \\( \\tx_i\\tz_i > 0 \\) for all \\( i \\). Now we can take \\( \\ln \\) on both sides of \\eqref{eq:inequ-sum-prod-power-case-I} to obtain\n\t  \\begin{equation}\\label{eq:log-inequ-sum-prod-power-case-I}\n\t\t\\ln\\left[\\sum_{i=1}^n \\alpha_i\\left(\\frac{\\tx_i\\tz_i}{\\alpha_i}\\right)\\right] \\le \\alpha_1\\ln\\left(\\frac{\\tx_1\\tz_1}{\\alpha_1}\\right) + \\cdots + \\alpha_n\\ln\\left(\\frac{\\tx_n\\tz_n}{\\alpha_n}\\right).\n\t  \\end{equation}\n\t  Using this together with the fact that \\( \\ln(\\cdot) \\) is strictly concave, we deduce that \\eqref{eq:log-inequ-sum-prod-power-case-I} holds as an equality. Hence, there exists a constant \\( c > 0 \\) so that\n\t  \\begin{equation}\\label{eq:form-tildex-by-tildez-case-I}\n\t\t\\tx_i = c \\alpha_i\\tz_i^{-1}\\,\\,\\,\\forall i = 1, 2, \\dots , n.\n\t  \\end{equation}\n\t  Plugging \\eqref{eq:form-tildex-by-tildez-case-I} into \\eqref{eq:inprod=0-case-I}, we obtain\n\t  \\begin{equation}\\label{eq:inprod-barz-barx=c-case-I}\n\t\t\\inProd{\\bbz}{\\bbx} = -\\inProd{ \\tbz}{\\tbx} = -c\\sum_{i=1}^n \\alpha_i = -c.\n\t  \\end{equation}\n\t  Moreover, using \\eqref{eq:form-tildex-by-tildez-case-I} and the last relation in \\eqref{eq:inequ-sum-prod-power-case-I}, we see that\n\t  \\[\n\t\t\\|\\bbz\\| \\|\\bbx\\| \\le \\prod_{i=1}^n\\left(\\frac{\\tx_i\\tz_i}{\\alpha_i}\\right)^{\\alpha_i} = c.\n\t  \\]\n\t  {The two displayed lines} above show that \\( \\|\\bbz\\| \\|\\bbx\\| = -\\inProd{\\bbz}{\\bbx} \\), which together with \\( \\bbz \\neq \\bm{0} \\) implies that there exists \\( \\kappa > 0 \\) so that\n\t  \\begin{equation}\\label{eq:form-barx-by-barz-kappa}\n\t\t\\bbx = -\\kappa\\bbz.\n\t  \\end{equation}\n\t  Plugging \\eqref{eq:form-barx-by-barz-kappa} into \\eqref{eq:inprod-barz-barx=c-case-I}, we obtain that \\( \\kappa = c/\\|\\bbz\\|^2 \\). Using this together with \\eqref{eq:form-tildex-by-tildez-case-I} and \\eqref{eq:form-barx-by-barz-kappa}, we can now conclude that\n\t  \\[\n\t\t\\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK =  \\{t \\bm{f}\\in \\R^{m+n}\\,| \\,t\\ge 0\\}\\,\\,\\mbox{with} \\,\\bm{f}= (-\\bbz/\\|\\bbz\\|^2,\\bm{\\alpha}\\circ \\tbz^{-1}),\n\t  \\]\n\t  where the inverse is taken componentwise.\n\n\t  \\ref{item:case-II-facial-structure}: In this case, \\( \\bbz = \\bm{0} \\). Then \\( \\stdIndex:= \\{i\\,| \\,\\tz_i > 0\\} \\) is nonempty because \\( \\bm{z}\\neq \\bm{0} \\). Hence, \\( \\bm{x} = (\\bbx,\\tbx)\\in \\{\\bm{z}\\}^\\perp\\cap \\gK\\backslash \\{\\bm{0}\\} \\) if and only if \\( \\bm{x}\\in \\bd\\gK\\backslash\\{\\bm{0}\\} \\) and satisfies\n\t  \\begin{equation*}\n\t\t\\sum_{i\\in \\stdIndex} \\tz_i\\tx_i = 0.\n\t  \\end{equation*}\n\t  This means that \\( \\tx_i = 0 \\) whenever \\( i\\in \\stdIndex \\) and hence \\( \\bbx= \\bm{0} \\). Thus,\n\t  \\[\n\t\t\\Fhatz := \\{\\bm{z}\\}^\\perp \\cap \\gK = \\{\\bm{x} = (\\bbx,\\tbx)\\in \\R_+^{m+n}\\,| \\,\\bbx = \\bm{0},\\,\\tx_i = 0\\ \\forall i\\in \\stdIndex\\}.\n\t  \\]\n\t\\end{proof}","start":31773,"end":35553},{"raw_tex":"\\begin{theorem}\\label{thm:Holderian-1/2-1st-type-face}\n\t\tLet \\( \\bm{z} = (\\bbz,\\tbz)\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz\\neq \\bm{0} \\) and let \\( \\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Let \\( \\eta > 0 \\) and define\n\t\t\\begin{equation}\\label{eq:gamma-1/2-1st-type-face}\n\t\t\t\\gamma_{\\bm{z},\\eta}\\! :=\\! \\inf_{\\bm{v}}\\left\\{\\frac{\\|\\bm{v} - \\bm{w}\\|^\\frac12}{\\|\\bm{u} - \\bm{w}\\|}\\,\\bigg| \\,\n\\begin{array}{c}\n\\bm{v}\\in \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fr,\\,\\bm{w} = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}),\\\\\n\\bm{u} = P_{\\Fr}(\\bm{w}),\\,\\bm{u} \\neq \\bm{w}\n\\end{array}\n\\right\\}.\n\t\t\\end{equation}\n\t\tThen it holds that \\( \\gamma_{\\bm{z},\\eta}\\in (0,\\infty] \\) and that\n\t\t\\[\n\t\t\\dist(\\bm{q},\\Fr)\\le \\max\\{2\\sqrt{\\eta},2\\gamma_{\\bm{z},\\eta}^{-1}\\}\\cdot\\dist(\\bm{q},\\gK)^{\\frac{1}{2}}\\,\\,\\text{ whenever }  \\bm{q} \\in \\{\\bm{z}\\}^\\perp\\cap \\B(\\eta) .\n\t\t\\]\n\t\\end{theorem}","start":38549,"end":39400},{"raw_tex":"\\begin{proof}\n\t\tSuppose for a contradiction that \\( \\gamma_{\\bm{z},\\eta} = 0 \\). Then, {in view of \\cite[Lemma 3.12]{LiLoPo20}}, there exist \\( \\widehat{\\bm{v}}\\in \\Fr \\) and a sequence \\( \\{\\bm{v}^k\\}\\subset \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fr \\) such that\n\t\t\\begin{equation}\\label{eq:forcontradiction-1st-face}\n\t\t\t\\lim_{k\\to \\infty}\\bm{v}^k = \\lim_{k\\to \\infty}\\bm{w}^k = \\widehat{\\bm{v}}\\,\\,{\\rm and}\\,\\,\\lim_{k\\to\\infty}\\frac{\\|\\bm{w}^k - \\bm{v}^k\\|^{\\frac{1}{2}}}{\\|\\bm{w}^k - \\bm{u}^k\\|} = 0,\n\t\t\\end{equation}\n\t\twhere \\( \\bm{w}^k = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}^k) \\), \\( \\bm{u}^k = P_{\\Fr}(\\bm{w}^k) \\) and \\( \\bm{u}^k\\neq \\bm{w}^k \\).\n\n\t\tDefine, for notational simplicity, \\( z_0 := \\|\\bbz\\| \\) and \\( v^k_0:=\\|\\bbv^k\\| \\).\n\t\tThen, since \\( \\{\\bm{v}^k\\}\\subset \\bd \\gK \\) and \\( \\bm{z} \\in \\bd\\dgK \\) with \\( \\bbz \\neq \\bm{0} \\), we have\\vspace{-0.15 cm}\n\t\t\\begin{equation}\\label{eq:z0-v0-notation}\\vspace{-0.15 cm}\n\t\t\tz_0 = \\|\\bbz\\| = \\prod_{i=1}^n\\left(\\frac{\\tz_i}{\\alpha_i}\\right)^{\\alpha_i}> 0 \\,\\,{\\rm and}\\,\\,v^k_0=\\|\\bbv^k\\| = \\prod_{i=1}^n(\\tv_i^k)^{\\alpha_i}\\,\\,\\forall k.\n\t\t\\end{equation}\n\n\t\tIf it holds that \\( v_0^k = 0 \\) infinitely often, by passing to a further subsequence, we may assume that \\( v_0^k = 0 \\) for all \\( k \\). Then we have in view of {Lemma \\ref{lemma:Lemma-2.11}} that\\vspace{-0.15 cm}\n\t\t\\[\\vspace{-0.15 cm}\n\t\t\\!\\|\\bm{v}^k - \\bm{w}^k\\| \\!=\\! \\frac1{\\|\\bm{z}\\|}|\\inProd{ \\tbz}{\\tbv^k}|\\!\\overset{\\rm (a)}=\\! \\frac1{\\|\\bm{z}\\|}\\sum_{i=1}^n\\tz_i\\tv^k_i\\!\\ge\\! \\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|} \\|\\tbv^k\\|_1\\!\\ge\\! \\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|} \\|\\tbv^k\\|\n\t\t\\!\\overset{\\rm (b)}=\\!\\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|} \\|\\bm{v}^k\\|,\n\t\t\\]\n\t\twhere (a) holds because \\( \\tv^k_i \\ge 0 \\) and \\( \\tz_i > 0 \\) for all \\( i \\) (see \\eqref{eq:z0-v0-notation}), and (b) holds since \\( \\|\\bbv^k\\| = 0 \\). Since\n\t\t\\( \\|\\bm{w}^k - \\bm{u}^k\\| = \\dist(\\bm{w}^k,\\Fr) \\le \\|\\bm{w}^k\\|\\le \\|\\bm{v}^k\\| \\) as a consequence of the properties of projections, we conclude from this and the above display that\n\t\t\\( \\|\\bm{v}^k - \\bm{w}^k\\| \\ge \\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|}\\|\\bm{w}^k - \\bm{u}^k\\| \\), contradicting \\eqref{eq:forcontradiction-1st-face}.\n\n\t\tThus, by considering a further subsequence if necessary, from now on, we assume\\vspace{-0.15 cm}\n\t\t\\begin{equation}\\label{eq:vk0-notation-nonzero}\\vspace{-0.15 cm}\n\t\t\tv^k_0=\\|\\bbv^k\\| = \\prod_{i=1}^n(\\tv_i^k)^{\\alpha_i} > 0 \\,\\,\\forall k.\n\t\t\\end{equation}\n\t\tUsing {Lemma \\ref{lemma:Lemma-2.11}}, we see that\\vspace{-0.15 cm}\n\t\t\\begin{equation}\\label{eq:dist-vk-wk}\\vspace{-0.15 cm}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\|\\bm{v}^k - \\bm{w}^k\\| & =\\frac1{\\|\\bm{z}\\|}|\\inProd{ \\bm{z}}{\\bm{v}^k}| = \\frac1{\\|\\bm{z}\\|}\\left|\\sum_{i=1}^{m}\\bz_i\\bv^k_i + \\sum_{i=1}^n\\tz_i\\tv^k_i\\right|             \\\\\n\t\t\t\t& = \\frac1{\\|\\bm{z}\\|}\\left|z_0v_0^k + \\sum_{i=1}^{m}\\bz_i\\bv^k_i - \\sum_{i=1}^n(-\\tz_i)\\tv^k_i - z_0v_0^k\\right|                                        \\\\\n\t\t\t\t& = \\frac{z_0v_0^k}{\\|\\bm{z}\\|}\\left| 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1\\right|  \\\\\n\t\t\t\t& = \\frac{z_0}{\\|\\bm{z}\\|}\\left( 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1\\right)v_0^k,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\twhere the last equality holds as \\( \\|z_0^{-1}\\bbz\\|= 1 \\), \\( \\|(v_0^k)^{-1}\\bbv^k\\|= 1 \\) and  \\( \\inProd{ z_0^{-1}\\tbz}{ (v_0^k)^{-1}\\tbv^k} \\ge 1 \\), thanks to \\eqref{eq:z0-v0-notation}, \\eqref{eq:vk0-notation-nonzero} and Lemma~\\ref{lemma:inequ-inprod-power-cone} applied with \\( \\bm{\\zeta} = -z_0^{-1}\\tbz \\).\n\n\t\tLet \\( \\bm{f} \\) be defined as in \\eqref{eq:1st-type-face-ray}. We consider two cases:\n\t\t\\begin{enumerate}[label={(\\Roman*)}]\n\t\t\t\\item \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}\\ge 0 \\) for all sufficiently large \\( k \\).\n\t\t\t\\item \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}< 0 \\) infinitely often.\n\t\t\\end{enumerate}\n\n\t\t(I): By passing to a further subsequence, we may assume that \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}\\ge 0 \\) for all \\( k \\). In this case, if we define\n\t\t\\[\n\t\t\\bm{Q} = \\bm{I}_{m + n} - \\frac{\\bm{z}\\bm{z}^{\\top}}{\\|\\bm{z}\\|^2} - \\frac{\\bm{f}\\bm{f}^{\\top}}{\\|\\bm{f}\\|^2},\n\t\t\\]\n\t\twhere \\( \\bm{f} \\) is as in \\eqref{eq:1st-type-face-ray}, then we see from {Lemma \\ref{lemma:Lemma-2.11}} and \\eqref{eq:z0-v0-notation} that\n\t\t\\begin{equation}\\label{eq:upbd-dist-uk-wk-two-parts}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\|\\bm{u}^k - \\bm{w}^k\\| & = \\|\\bm{Q}\\bm{v}^k\\| = v_0^k\\left\\|\\bm{Q}\\begin{bmatrix}\n\t\t\t\t\t(v_0^k)^{-1}\\bbv^k \\\\ (v_0^k)^{-1}\\tbv^k\n\t\t\t\t\\end{bmatrix}\\right\\| \\\\\n&\\overset{\\rm (a)}= v_0^k{\\Bigg\\|}\\bm{Q}\\begin{bmatrix}\n\t\t\t\t\t(v_0^k)^{-1}\\bbv^k \\\\ (v_0^k)^{-1}\\tbv^k\n\t\t\t\t\\end{bmatrix} \\!-\\! \\bm{Q}\\underbrace{\\begin{bmatrix}\n\t\t\t\t\t\t-z_0^{-1}\\bbz \\\\ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\n\t\t\t\t\\end{bmatrix}}_{z_0\\bm{f}}{\\Bigg\\|} \\\\\n\t\t\t\t& \\le v_0^k\\left[\\|(v_0^k)^{-1}\\bbv^k + z_0^{-1}\\bbz\\| + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\right],\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\twhere (a) holds because %\\( z_0\\bm{f} = \\begin{bmatrix}\n\t\t%\t\t-z_0^{-1}\\bbz^{\\top} & \\alpha\\circ(z_0\\tbz^{-1})^{\\top}\n\t\t%\t\\end{bmatrix}^{\\top}\\)  and\n\t\t\\( \\bm{Q}\\bm{f} = \\bm{0} \\) (an identity which is clear from the definitions).\n\n\t\tNext, in view of \\eqref{eq:z0-v0-notation}, we can apply {Lemma \\ref{lemma:Lemma-4.1}} to obtain \\( C_1 > 0 \\) and \\( \\epsilon_1>0 \\) so that {\\eqref{eq:inequ-p=q=2}} holds with \\( \\overline{\\bm{\\zeta}} = -z_0^{-1}\\bbz \\), i.e.,\n\t\t\\[\n\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_1\\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|^2\n\t\t\\]\n\t\twhenever \\( \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\le \\epsilon_1 \\) and \\( \\|\\bm{\\omega}\\|=1 \\). On the other hand, in view of the positivity of $1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}$ when $\\|\\bm{\\omega}\\|=1$ and $\\bm{\\omega} \\neq -z_0^{-1}\\bbz$ {(see Lemma \\ref{lemma:Lemma-4.1})}, we know that\n\\[\nC_2 := \\inf_{\\|\\bm{\\omega}\\|=1}\\{1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}} \\,|\\, \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\ge \\epsilon_1 \\} > 0.\n\\]\nThis together with the fact $\\|z_0^{-1}\\bbz\\|=1$ (see \\eqref{eq:z0-v0-notation}) implies that\n\t\t\\[\n\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_2 \\ge 0.25 C_2 \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|^2,\n\t\t\\]\n\t\twhenever \\( \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\ge \\epsilon_1 \\) and \\( \\|\\bm{\\omega}\\|=1 \\). We thus have (with $C_3 \\!:=\\! \\min\\{C_1,C_2/4\\}$)\n\\begin{equation}\\label{eq:keylemmarel}\n\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_3\\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|^2\\ \\ \\ {\\rm whenever}\\ \\ \\|\\bm{\\omega}\\|=1.\n\t\t\\end{equation}\n\n\t\tIn addition, noting \\eqref{eq:z0-v0-notation} again, we can apply Lemma~\\ref{lemma:inequ-inprod-power-cone} with \\( \\bm{\\zeta} = -z_0^{-1}\\tbz\\in {\\rm int}\\,\\R^n_- \\) to obtain \\( C_4 > 0 \\) and \\( \\epsilon > 0 \\) so that \\eqref{eq:inequ-inprod-power-cone} holds with \\( \\widetilde{\\bm{\\zeta}} = \\bm{\\alpha}\\circ (z_0\\tbz^{-1}) \\), i.e.,\n\t\t\\begin{equation}\\label{eq:keylemmare2}\n\t\t\t-1 + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}\\ge C_4\\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2\n\t\t\\end{equation}\n\t\twhenever \\( \\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\le \\epsilon \\), \\( \\bm{\\omega}\\in {\\rm int}\\,\\R^n_+ \\) and \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i}=1 \\).\n\n%Then we have in view of the positivity of $1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}$ when $\\bm{\\omega} \\neq -z_0^{-1}\\bbz$ (see Lemma \\ref{lemma:Lemma-4.1}) and the compactness of the unit sphere that there exists \\( C_3>0 \\) so that\n%\t\t\\begin{equation}\\label{eq:keylemmare3}\n%\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_3 \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\n%\t\t\\end{equation}\n%\t\twhenever \\( \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\ge \\epsilon \\) and \\( \\|\\bm{\\omega}\\|=1 \\).\n\nFurthermore, consider \\( h:\\R^n\\to \\R\\cup\\{\\infty\\} \\) defined by\n\t\t\\begin{equation}\\label{eq:def-h}\n\t\t\th(\\bm{\\omega}) = \\begin{cases}\n\t\t\t\t\\displaystyle\\frac{\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}-1}{\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|} & {\\rm if}\\,\\|\\bm{\\omega}-\\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\ge \\epsilon,\\,\\bm{\\omega}\\in \\Upsilon, \\\\\n\t\t\t\t\\infty                                                                                                    & {\\rm otherwise},\n\t\t\t\\end{cases}\n\t\t\\end{equation}\n\t\twhere \\( \\Upsilon= \\{\\bm{\\omega}\\in \\R^n_+\\,| \\,\\prod_{i=1}^n \\omega_i^{\\alpha_i}=1\\} \\). Then we have\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t& \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty}h(\\bm{\\omega}) = \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty, \\bm{\\omega}\\in \\Upsilon}\\frac{\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}-1}{\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|} \\\\\n\t\t\t& \\ge \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty,\\bm{\\omega}\\in \\R^n_+}\\frac{\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}-1}{\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|}\n\t\t\t\\overset{\\rm (a)}{\\ge} \\inf_{\\|\\bm{\\lambda}\\|=1,\\bm{\\lambda}\\in \\R^n_+}\\inProd{ z_0^{-1}\\tbz}{\\bm{\\lambda}} \\overset{\\rm (b)}\\ge \\min_{1\\le i\\le n}z_0^{-1}\\tz_i > 0.\n\t\t\\end{aligned}\n\t\t\\]\n\t\tHere (a) may be verified by multiplying both numerator and denominator of the left side by $1/\\|\\bm{\\omega}\\|$; (b) holds because \\( z_0^{-1}\\tz_i > 0 \\) for all \\( i \\) (see \\eqref{eq:z0-v0-notation}). Since \\( h \\) in \\eqref{eq:def-h} is also lower semicontinuous on any compact set and {\\bf is always positive},\\footnote{The positivity can be seen by applying Lemma~\\ref{lemma:inequ-inprod-power-cone} with $\\bm\\zeta = -z_0^{-1}\\tbz$.} it must then hold that \\( C_5:= \\inf h > 0 \\). In particular, this means that\n\t\t\\begin{equation}\\label{eq:keylemmare4}\n\t\t\t-1 + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}\\ge C_5\\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\n\t\t\\end{equation}\n\t\twhenever \\( \\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\ge \\epsilon \\), \\( \\bm{\\omega}\\in {\\rm int}\\,\\R^n_+ \\) and \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i}=1 \\).\n\n\t\tBy passing to suitable subsequences, we will end up with one of the following two cases:\n\t\t\\begin{enumerate}[label={Case \\arabic*:}]\n\t\t\t\\item \\( \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| \\le \\epsilon \\) for all \\( k \\). Then we have from \\eqref{eq:keylemmarel} (with $\\bm{\\omega}=(v_0^k)^{-1}\\bbv^k$) and \\eqref{eq:keylemmare2} (with $\\bm{\\omega}=(v_0^k)^{-1}\\tbv^k$) that for these \\( k \\)\n\t\t\t\\begin{equation*}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t& 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1          \\\\\n\t\t\t\t\t& \\ge \\min\\{C_3,C_4\\}(\\|(v_0^k)^{-1}\\bbv^k  + z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2).\n\t\t\t\t\\end{aligned}\n\t\t\t\\end{equation*}\n\t\t\tCombining this with \\eqref{eq:dist-vk-wk} and \\eqref{eq:upbd-dist-uk-wk-two-parts}, we see further that\n\t\t\t\\[\n\t\t\t\\begin{aligned}\n\t\t\t\t&\\|\\bm{v}^k - \\bm{w}^k\\|\\\\ & \\ge \\frac{z_0}{\\|\\bm{z}\\|}\\min\\{C_3,C_4\\}(\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2) v_0^k    \\\\\n\t\t\t\t& \\ge \\frac{z_0}{2\\|\\bm{z}\\|}\\min\\{C_3,C_4\\}(\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\| + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|)^2 v_0^k     \\\\\n\t\t\t\t& \\ge \\frac{z_0\\min\\{C_3,C_4\\}}{2\\|\\bm{z}\\|v_0^k}\\|\\bm{u}^k - \\bm{w}^k\\|^2\\ge \\frac{z_0\\min\\{C_3,C_4\\}}{2\\|\\bm{z}\\|\\eta}\\|\\bm{u}^k - \\bm{w}^k\\|^2,\n\t\t\t\\end{aligned}\n\t\t\t\\]\n\t\t\twhere the last inequality holds because \\( \\bm{v}^k\\in \\B(\\eta) \\). The above display contradicts \\eqref{eq:forcontradiction-1st-face} and hence Case 1 cannot happen.\n\n\t\t\t\\item \\( \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\ge \\epsilon \\) for all \\( k \\). Then we have from \\eqref{eq:keylemmarel} and \\eqref{eq:keylemmare4} that for these \\( k \\)\n\t\t\t\\begin{equation*}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t& 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1        \\\\\n\t\t\t\t\t& \\ge \\min\\{C_3,C_5\\}(\\|(v_0^k)^{-1}\\bbv^k  + z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|).\n\t\t\t\t\\end{aligned}\n\t\t\t\\end{equation*}\n\t\t\tUsing this together with \\eqref{eq:dist-vk-wk}, we deduce that for all large \\( k \\),\n\t\t\t\\[\n\t\t\t\\!\\|\\bm{v}^k - \\bm{w}^k\\| \\!\\ge\\! \\frac{z_0\\min\\{C_3,C_5\\}}{\\|\\bm{z}\\|}(\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|) v_0^k.\n\t\t\t\\]\n\t\t\tThis implies that\n\t\t\t\\begin{equation}\\label{eq:upbd-part-dist-uk-wk-case-2}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\|            & \\le M_1\\sqrt{(v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|}, \\\\\n\t\t\t\t\t\\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| & \\le M_1(v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|,\n\t\t\t\t\\end{aligned}\n\t\t\t\\end{equation}\n\t\t\twhere \\( M_1 := \\max\\left\\{\\left(\\frac{z_0}{\\|\\bm{z}\\|}\\min\\{C_3,C_5\\}\\right)^{-1},\\left(\\frac{z_0}{\\|\\bm{z}\\|}\\min\\{C_3,C_5\\}\\right)^{-1/2}\\right\\} \\). Using \\eqref{eq:upbd-part-dist-uk-wk-case-2} together with \\eqref{eq:upbd-dist-uk-wk-two-parts}, we obtain that\n\t\t\t\\begin{equation}\n\t\t\t\\begin{aligned}\\label{eqn:3Ms}\n\t\t\t\t\\|\\bm{u}^k - \\bm{w}^k\\| & \\le M_1v_0^k\\left[\\sqrt{(v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|} + (v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|\\right]   \\\\\n\t\t\t\t%& = M_1\\left(\\sqrt{v_0^k} + \\sqrt{\\|\\bm{v}^k-\\bm{w}^k\\|}\\right)\\sqrt{\\|\\bm{v}^k - \\bm{w}^k\\|}\\\\\n\t\t\t\t&\\stackrel{\\rm (a)}{\\le}M_1\\sqrt{\\eta} \\sqrt{\\|\\bm{v}^k-\\bm{w}^k\\|} + M_1\\|\\bm{v}^k - \\bm{w}^k\\| \\\\\n\t\t\t\t&\\stackrel{\\rm (b)}{\\le} 3M_1\\sqrt{\\eta}\\sqrt{\\|\\bm{v}^k - \\bm{w}^k\\|},\n\t\t\t\\end{aligned}\n\t\t\t\\end{equation}\n\t\t\twhere (a) holds since \\( \\bm{v}^k\\in \\B(\\eta) \\) wherefore $v_0^k \\leq \\eta$, and (b) holds because \\(\\|\\bm{w}^k\\| \\leq \\|\\bm{v}^k\\| \\le \\eta \\) (because the projection onto $\\stdCone$ is nonexpansive and $\\bm 0\\in \\stdCone$), wherefore\n\t\t\t$$\n\t\t\t\\|\\bm{w}^k-\\bm{v}^k\\| = \\sqrt{\\|\\bm{w}^k-\\bm{v}^k\\|}\\sqrt{\\|\\bm{w}^k-\\bm{v}^k\\|} \\leq 2\\sqrt{\\eta}\\sqrt{\\|\\bm{w}^k-\\bm{v}^k\\|}.\n\t\t\t$$\n\t\t\tAltogether, \\eqref{eqn:3Ms} contradicts \\eqref{eq:forcontradiction-1st-face} and hence Case 2 cannot happen.\n\t\t\\end{enumerate}\n\t\tSummarizing the above discussions, we see that Case (I) cannot happen.\n\n\t\t(II):  By passing to a further subsequence, we may assume that \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}< 0 \\) for all \\( k \\). This together with the definition of \\( \\bm{f} \\) gives\n\t\t\\[\n\t\t\\frac{v_0^k}{z_0}[\\inProd{ -z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})}{ (v_0^k)^{-1}\\tbv^k}] < 0.\n\t\t\\]\n\t\tSince $\\frac{v_0^k}{z_0} > 0$, we deduce that $\\inProd{ -z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})}{ (v_0^k)^{-1}\\tbv^k} < 0$ for all $k$. Then it must hold that\n\t\t\\[\n\t\t\\lim_{k\\to \\infty}\\|(v_0^k)^{-1}\\bbv^k + z_0^{-1}\\bbz\\|+\\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\neq 0;\n\t\t\\]\notherwise, we have $(v_0^k)^{-1}\\bbv^k\\to -z_0^{-1}\\bbz$ and $(v_0^k)^{-1}\\tbv^k\\to \\bm{\\alpha}\\circ(z_0\\tbz^{-1})$, which further gives $\\inProd{ -z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})}{ (v_0^k)^{-1}\\tbv^k}\\to \\|z_0^{-1}\\bbz\\|^2+\\|\\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2 = \\|z_0\\bm f\\|^2 > 0$, a contradiction.\n\n\t\tConsequently, there exists $\\epsilon > 0$ such that for all sufficiently large $k$,\n\t\t\\begin{equation}\\label{eq:sum-out-of-neighbor}\n\t\t\t\\|(v_0^k)^{-1}\\bbv^k + z_0^{-1}\\bbz\\|+\\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| \\ge \\epsilon.\n\t\t\\end{equation}\n\t\tConsider the function \\( G:\\R^{m+n}\\to \\R\\cup\\{\\infty\\} \\) defined by\n\t\t\\[\n\t\tG(\\bm{\\xi},\\bm{\\omega}) := \\begin{cases}\n\t\t\t\\displaystyle \\frac{|\\inProd{ z_0^{-1}\\bbz}{ \\bm{\\xi}} + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}|}{\\sqrt{1+\\|\\bm{\\omega}\\|^2}} & {\\rm if}\\,(\\bm{\\xi},\\bm{\\omega})\\in \\Xi,\\,\\|\\bm{\\xi}\\|=1,\\,{\\rm and}\\,\\bm{\\omega}\\in \\Upsilon, \\\\\n\t\t\t\\infty                                                                                                                       & {\\rm otherwise},\n\t\t\\end{cases}\n\t\t\\]\n\t\twhere \\( \\Upsilon = \\{\\bm{\\omega}\\in \\R^n_+\\,| \\,\\prod_{i=1}^n \\omega_i^{\\alpha_i}=1\\} \\) and \\( \\Xi= \\{(\\bm{\\xi},\\bm{\\omega})\\,| \\,\\|\\bm{\\xi} + z_0^{-1}\\bbz\\|+\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| \\ge \\epsilon\\} \\).\n\t\tSince \\( \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\xi}} + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}} = 1+\\inProd{ z_0^{-1}\\bbz}{ \\bm{\\xi}} - \\inProd{ z_0^{-1}(-\\tbz)}{ \\bm{\\omega}}-1 \\), we see from \\eqref{eq:z0-v0-notation}, \\eqref{eq:sum-out-of-neighbor}, {Lemma \\ref{lemma:Lemma-4.1}} and Lemma~\\ref{lemma:inequ-inprod-power-cone} that \\( G \\) is never zero. Moreover, it is clearly lower semicontinuous on any compact set, and\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\!\\liminf_{\\|(\\bm{\\xi},\\bm{\\omega})\\|\\to \\infty}G(\\bm{\\xi},\\bm{\\omega}) \\!=\\!\\!\\! \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty,\\bm{\\omega}\\in \\Upsilon}\\!\\!\\frac{|\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}|}{\\sqrt{1+\\|\\bm{\\omega}\\|^2}}\\!\\overset{\\rm (a)}\\ge\\!\\!\\! \\inf_{\\|\\bm{\\lambda}\\|=1,\\bm{\\lambda}\\in \\R^{n}_+}\\!\\!|\\inProd{ z_0^{-1}\\tbz}{\\bm{\\lambda}}|\\!\\overset{\\rm (b)}\\ge\\!\\! \\min_i |z_0^{-1}\\tz_i| > 0,\n\t\t\\end{aligned}\n\t\t\\]\n\t\twhere (a) may be verified by multiplying numerator and denominator by $1/\\|\\bm{\\omega}\\|$ and (b) holds since \\( z_0^{-1}\\tz_i > 0 \\) for all \\( i \\).\n\t\tThus, \\( C_6:= \\inf G > 0 \\) and we have for all large \\( k \\),\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\frac{\\|\\bm{v}^k - \\bm{w}^k\\|}{\\|\\bm{u}^k - \\bm{w}^k\\|} & \\overset{\\rm (a)}= \\frac{\\|\\bm{v}^k - \\bm{w}^k\\|}{\\|\\bm{w}^k\\|} \\overset{\\rm (b)}\\ge \\frac{\\|\\bm{v}^k - \\bm{w}^k\\|}{\\|\\bm{v}^k\\|} \\\\\n&\\overset{\\rm (c)}=\n\t\t\t\\frac{z_0}{\\|\\bm{z}\\|}\\frac{|\\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ z_0^{-1}\\tbz}{ (v_0^k)^{-1}\\tbv^k}|v_0^k}{\\sqrt{(v^k_0)^2 + \\|\\tbv^k\\|^2}}                                                                                                            \\\\\n\t\t\t& = \\frac{z_0}{\\|\\bm{z}\\|}\\frac{|\\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ z_0^{-1}\\tbz}{ (v_0^k)^{-1}\\tbv^k}|}{\\sqrt{1 + \\|(v_0^k)^{-1}\\tbv^k\\|^2}}\\overset{\\rm (d)}\\ge \\frac{C_6 z_0}{\\|\\bm{z}\\|},\n\t\t\\end{aligned}\n\t\t\\]\n\t\twhere (a) follows from {Lemma \\ref{lemma:Lemma-2.11}}, which states that \\( \\bm{u}^k = \\bm{0} \\) in Case (II), (b) holds because the projection onto the cone is nonexpansive and $\\bm 0$ is in the cone, (c) follows from \\eqref{eq:dist-vk-wk} and (d) follows from \\eqref{eq:sum-out-of-neighbor}, \\eqref{eq:vk0-notation-nonzero} and the definitions of \\( G \\) and \\( C_6 \\). The above display contradicts \\eqref{eq:forcontradiction-1st-face}. Thus, Case (II) also cannot happen.\n\n\t\tSummarizing the above, we conclude that \\eqref{eq:forcontradiction-1st-face} cannot happen. Thus, in view of {\\cite[Lemma 3.12]{LiLoPo20}}, we must indeed have \\( \\gamma_{\\bm{z},\\eta} \\in (0,\\infty] \\) and that the desired error bound follows from {\\cite[Theorem 3.10]{LiLoPo20}}.\n\t\\end{proof}","start":39404,"end":57713},{"raw_tex":"\\begin{theorem}\\label{thm:xface}\n\t\tLet \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz = \\bm{0} \\) and let \\( \\Fhatz := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Let \\( \\stdIndex := \\{i\\,| \\,\\tz_i > 0\\} \\),\\footnote{Since \\( \\bbz = \\bm{0} \\) and \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\), we must have \\( \\emptyset\\neq \\stdIndex \\subsetneq \\{1,2,\\ldots,n\\} \\).} \\( \\eta > 0 \\) and define \\( \\beta:= \\sum_{i\\in \\stdIndex}\\alpha_i \\) and\n\t\t\\begin{equation}\\label{eq:gamma-beta-2nd-type-face}\n\t\t \\!\\gamma_{\\bm{z},\\eta} \\!:=\\! \\inf_{\\bm{v}}\\left\\{\\frac{\\|\\bm{v} - \\bm{w}\\|^\\beta}{\\|\\bm{u} - \\bm{w}\\|}\\,\\bigg| \\,\n\\begin{array}{c}\n\\bm{v}\\in \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fhatz,\\,\\bm{w} = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}),\\\\\n\\bm{u} = P_{\\Fhatz}(\\bm{w}),\\,\\bm{u} \\neq \\bm{w}\n\\end{array}\\right\\}.\n\t\t\\end{equation}\n\t\tThen it holds that \\( \\gamma_{\\bm{z},\\eta}\\in (0,\\infty] \\) and that\n\t\t\\[\n\t\t\\dist(\\bm{q},\\Fhatz)\\le \\max\\{2\\eta^{1-\\beta},2\\gamma_{\\bm{z},\\eta}^{-1}\\}\\cdot\\dist(\\bm{q},\\gK)^\\beta\\,\\,\\text{ whenever }  \\bm{q}\\in \\{\\bm{z}\\}^\\perp\\cap \\B(\\eta) .\n\t\t\\]\n\t\\end{theorem}","start":64632,"end":65697},{"raw_tex":"\\begin{proof}\n\t\t%\tIf \\( \\gamma_{\\bm{z},\\eta} = 0 \\), in view of \\cite[Lemma 3.12]{LiLoPo20}, there exist \\( \\widehat{\\bm{v}}\\in \\Fhatz \\) and a sequence \\( \\{\\bm{v}^k\\}\\subset \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fhatz \\) such that\n\t\t%\t\\begin{equation}\\label{eq:forcontradiction-2nd-face}\n\t\t%\t\t\\lim_{k\\to \\infty}\\bm{v}^k = \\lim_{k\\to \\infty}\\bm{w}^k = \\widehat{\\bm{v}}\\,\\,{\\rm and}\\,\\,\\lim_{k\\to\\infty}\\frac{\\|\\bm{w}^k - \\bm{v}^k\\|^{\\beta }}{\\|\\bm{w}^k - u^k\\|} = 0,\n\t\t%\t\\end{equation}\n\t\t%\twhere \\( \\bm{w}^k = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}^k) \\), \\( \\bm{u}^k = P_{\\Fhatz}(\\bm{w}^k) \\) and \\( \\bm{u}^k\\neq \\bm{w}^k \\).\n\n\t\t{In view of \\cite[Theorem 3.10]{LiLoPo20}}, we need only show that $\\gamma_{\\bm{z},\\eta} > 0$.\n%\t\t$$\n%\t\t0<\\gamma_{\\bm{z},\\eta} = \\inf_{\\bm{v}} \\left\\{ \\frac{\\|\\bm{w} -\\bm{v} \\|^\\beta}{\\|\\bm{w} -\\bm{u} \\| } \\Big|\\bm{v} \\in \\bd \\stdCone \\cap \\B(\\eta) \\setminus \\stdFace,\\bm{w} = P_{\\{\\bm{z}\\}^{\\perp }}(\\bm{v}), \\bm{u} = P_{\\stdFace}(\\bm{w}), \\bm{w} \\neq\\bm{u} \\right\\}.\n%\t\t$$\n\t\tTo that end, let \\( \\bm{v}\\in \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fhatz \\), \\( \\bm{w} = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}) \\), \\( \\bm{u} = P_{\\Fhatz}(\\bm{w}) \\), and \\( \\bm{u} \\neq \\bm{w} \\). Then a direct computation shows\n\t\tthat\n\t\t\\begin{equation}\\label{eq:dist-w-v}\n\t\t\t\\|\\bm{w}- \\bm{v}\\| = \\frac1{\\|\\bm{z}\\|}|\\inProd{ \\bm{z}}{\\bm{v}}| \\overset{\\rm (a)}= \\frac1{\\|\\bm{z}\\|}\\sum_{i\\in \\stdIndex}\\tz_i\\tv_i \\overset{\\rm (b)}\\ge \\frac{\\min_{i\\in \\stdIndex}\\tz_i}{\\|\\bm{z}\\|}\\sum_{i\\in \\stdIndex}\\tv_i\n\t\t\t\\overset{\\rm (c)}\\ge \\frac{\\min_{i\\in \\stdIndex}\\tz_i}{\\|\\bm{z}\\|}\\|\\tbv_{\\stdIndex}\\|,\n\t\t\\end{equation}\n\t\twhere (a), (b) and (c) hold because \\( \\tv_i\\ge 0 \\) and \\( \\tz_i > 0 \\) for all \\( i\\in \\stdIndex \\), with \\( \\|\\tbv_{\\stdIndex}\\|:= \\sqrt{\\sum_{i\\in \\stdIndex}\\tv^2_i} \\) (note that \\( \\stdIndex \\neq \\emptyset \\), thanks to \\( \\bbz = 0 \\) and \\( \\bm{z}\\neq \\bm{0} \\)). Next, notice that \\( \\bm{w} = \\bm{v} - \\frac{\\inProd{ \\bm{z}}{\\bm{v}}}{\\|\\bm{z}\\|^2}\\bm{z} \\). Using this and the definitions of \\( \\bm{z} \\) and \\( \\stdIndex \\), we deduce that\n\t\t\\begin{equation}\\label{eq:form-w-by-v}\n\t\t\t\\bbw = \\bbv,\\,\\,\\,\\tw_i = \\tv_i - \\frac{\\tz_i}{\\|\\bm{z}\\|^2}\\bigg(\\sum_{j\\in \\stdIndex}\\tz_j\\tv_j\\bigg)\\,\\,\\,\\, \\forall i\\in \\stdIndex\\,\\,\\,\\,\\,\\,\\,\\mbox{and\\,\\,\\,\\,\\,\\,\\,} \\tw_i = \\tv_i \\ge 0\\,\\,\\,\\,\\forall i \\notin \\stdIndex.\n\t\t\\end{equation}\n\t\tIn view of this and the definition of \\( \\Fhatz \\) in \\eqref{eq:2nd-type-face-d-dim}, we see that \\( \\tu_i = \\tw_i \\) whenever \\( i\\notin \\stdIndex \\), and hence\n\t\t\\begin{equation}\\label{eq:dist-w-u}\n\t\t\t\\|\\bm{w} - \\bm{u}\\| = \\sqrt{\\|\\bbw\\|^2 + \\sum_{i\\in \\stdIndex}\\tw^2_i} \\le \\sqrt{\\|\\bbv\\|^2 + n(1+\\sqrt{n})^2\\|\\tbv_{\\stdIndex}\\|^2},\n\t\t\\end{equation}\n\t\twhere the inequality follows from \\eqref{eq:form-w-by-v} and the fact that for each \\( i\\in \\stdIndex \\),\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t|\\tw_i| & = \\bigg|\\tv_i - \\frac{\\tz_i}{\\|\\bm{z}\\|^2}\\bigg(\\sum_{j\\in \\stdIndex}\\tz_j\\tv_j\\bigg)\\bigg|\\le \\bigg(1 + \\frac{|\\tz_i|}{\\|\\bm{z}\\|^2}\\sum_{j\\in \\stdIndex}|\\tz_j|\\bigg)\\|\\tbv_{\\stdIndex}\\| \\\\\n\t\t\t& \\le \\left(1 + \\frac{\\sqrt{n}|\\tz_i|}{\\|\\bm{z}\\|}\\right)\\|\\tbv_{\\stdIndex}\\| \\le (1+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\|.\n\t\t\\end{aligned}\n\t\t\\]\n\t\tNext, note that we have\n\t\t\\begin{equation}\\label{eq:upbd-norm-bar-v}\n\t\t\t\\|\\bbv\\|=%\n\t\t\t\\prod_{i=1}^n(\\tv_i)^{\\alpha_i} %\n\t\t\t= \\prod_{i\\notin \\stdIndex}\\tv_i^{\\alpha_i} \\cdot \\prod_{i\\in \\stdIndex}\\tv_i^{\\alpha_i}%\n\t\t\t\\le \\prod_{i\\notin \\stdIndex}\\eta^{\\alpha_i} \\cdot\\prod_{i\\in \\stdIndex}\\|\\tbv_{\\stdIndex}\\|^{\\alpha_i}%\n\t\t\t= \\eta^{1-\\beta} \\|\\tbv_{\\stdIndex}\\|^\\beta,\n\t\t\\end{equation}\n\t\twhere the inequality holds because \\( \\bm{v}\\in \\B(\\eta) \\). Combining \\eqref{eq:dist-w-v}, \\eqref{eq:dist-w-u} and \\eqref{eq:upbd-norm-bar-v}, we deduce\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\|\\bm{w} - \\bm{u}\\| & \\le \\sqrt{\\|\\bbv\\|^2 + n(1+\\sqrt{n})^2\\|\\tbv_{\\stdIndex}\\|^2} \\le \\|\\bbv\\| + (n+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\|                                                                                                     \\\\\n\t\t\t& \\le \\eta^{1-\\beta} \\|\\tbv_{\\stdIndex}\\|^\\beta + (n+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\| = (\\eta^{1-\\beta}  + (n+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\|^{1-\\beta})\\|\\tbv_{\\stdIndex}\\|^\\beta                                   \\\\\n\t\t\t& \\overset{\\rm (a)}\\le \\eta^{1-\\beta}(n+1 + \\sqrt{n})\\|\\tbv_{\\stdIndex}\\|^\\beta \\overset{\\rm (b)}{\\le}\\frac{\\eta^{1-\\beta}(n+1 + \\sqrt{n})\\|\\bm{z}\\|^\\beta}{\\left(\\min_{i\\in_{\\stdIndex}}\\tz_i\\right)^\\beta}\\|\\bm{w}-\\bm{v}\\|^\\beta.\n\t\t\\end{aligned}\n\t\t\\]\n\t\tHere (a) holds since \\( \\bm{v}\\in \\B(\\eta) \\) and \\( \\beta \\in (0,1) \\); (b) is true because of \\eqref{eq:dist-w-v}.\n\t\tThus, \\( \\gamma_{\\bm{z},\\eta}\\ge \\frac{\\left(\\min_{i\\in \\stdIndex}\\tz_i\\right)^\\beta}{\\eta^{1-\\beta}(n+1 + \\sqrt{n})\\|\\bm{z}\\|^\\beta} > 0 \\), and the desired error bound {follows from \\cite[Theorem 3.10]{LiLoPo20}}.\n\t\\end{proof}","start":65699,"end":70480},{"raw_tex":"\\begin{corollary}\\label{corollary:1-FRFs-power-cones}\n\t\tConsider \\( \\gK \\) and its dual cone \\( \\dgK \\).\n\t\t\\begin{enumerate}[label=(\\roman*)]\n\t\t\t\\item\\label{3.6i} Let \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz\\neq \\bm{0} \\) and let \\( \\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Let \\( \\gamma_{\\bm{z},t} \\) be defined as in \\eqref{eq:gamma-1/2-1st-type-face}. Then the function \\( \\psi_{\\gK,\\bm{z}}:\\R_+\\times \\R_+\\to \\R_+ \\) given by\\vspace{-0.15 cm}\n\t\t\t\\begin{equation}\\label{eq:1-FRF-1st-face}\\vspace{-0.15 cm}\n\t\t\t\t\\!\\!\\psi_{\\gK,\\bm{z}}(\\epsilon,t) \\!:=\\! \\max \\left\\{\\epsilon,\\epsilon/\\|\\bm{z}\\| \\right\\} + \\max\\{2\\sqrt{t},2\\gamma_{\\bm{z},t}^{-1}\\}(\\epsilon +\\max \\left\\{\\epsilon,\\epsilon/\\|\\bm{z}\\| \\right\\} )^\\frac12\n\t\t\t\\end{equation}\n\t\t\tis a one-step facial residual function for \\( \\gK \\) and \\( \\bm{z} \\).\n\t\t\t\\item Let \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz = \\bm{0} \\) and let \\( \\Fhatz := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Let \\( \\gamma_{\\bm{z},t} \\) be defined as in \\eqref{eq:gamma-beta-2nd-type-face}, where \\( \\beta:= \\sum_{i:\\tz_i > 0}\\alpha_i \\). Then the function \\( \\psi_{\\gK,\\bm{z}}:\\R_+\\times \\R_+\\to \\R_+ \\) given by\\vspace{-0.15 cm}\n\t\t\t\\begin{equation}\\label{eq:1-FRF-2nd-face}\\vspace{-0.15 cm}\n\t\t\t\t\\!\\!\\psi_{\\gK,\\bm{z}}(\\epsilon,t) \\!:= \\!\\max\\! \\left\\{\\epsilon,\\epsilon/\\|\\bm{z}\\| \\right\\} + \\max\\{2t^{1-\\beta}\\!,2\\gamma_{\\bm{z},t}^{-1}\\}(\\epsilon +\\max \\left\\{\\epsilon,\\epsilon/\\|\\bm{z}\\| \\right\\} )^\\beta\n\t\t\t\\end{equation}\n\t\t\tis a one-step facial residual function for \\( \\gK \\) and \\( \\bm{z} \\).\n\t\t\\end{enumerate}\n\t\\end{corollary}","start":72726,"end":74303},{"raw_tex":"\\begin{proof}\n\t\tAs is shown in Section \\ref{subsec:facial-structure}, all the proper exposed faces of the generalized power cone are polyhedral.\n\t\tThen the process of facial reduction needs at most one step to reach the PPS condition. Hence, \\( \\dpps(\\gK, \\subSpace + \\bm{a}) \\leq 1 \\).\n\t\tThis shows item \\ref{item:error-bound-power-cone-1}.\n\n\t\tIf \\( \\dpps(\\gK, \\subSpace + \\bm{a}) = 0 \\), i.e., \\eqref{eq:conic-feasibility-problem} satisfies the PPS condition, then by \\cite[Corollary 3]{BBL99}, a Lipschitzian error bound holds.\n\t\tThis shows item \\ref{item:error-bound-power-cone-2}.\n\n\t\tNext, let \\( \\dpps(\\gK, \\subSpace + \\bm{a}) = 1 \\); i.e., we need one step to reach the PPS condition.\n\t\tIn this case, the error bound depends on the exposed face \\( \\stdFace \\) that contains the feasible region.\n\t\tIf \\(\\stdFace =  \\Fr \\), then by Corollary \\ref{corollary:1-FRFs-power-cones}\\ref{3.6i}, we conclude that a H\\\"olderian error bound with exponent \\( 1 / 2 \\) holds.\n\t\tRemark \\ref{remark:F-r-order-1/2} implies that \\( \\mathfrak{g} = | \\cdot | ^{\\frac{1}{2}}\\) satisfies the asymptotic optimality criterion for \\( \\gK \\) and \\( \\bm{z} \\) with \\( \\bbz \\neq \\bm{0} \\).\n\t\tHence, by {\\cite[Theorem 3.2]{LiLoPo21}}, the obtained H\\\"olderian error bound with exponent \\( 1 / 2 \\) is the best error bound.\n\n\t\tIf \\( \\stdFace = \\Fhatz \\) with \\( \\bm{z} \\in \\dgK \\cap \\subSpace^{\\perp } \\cap \\{\\bm{a}\\}^{\\perp } \\), then using Corollary \\ref{corollary:1-FRFs-power-cones}(ii), we conclude that a H\\\"olderian error bound with exponent \\( \\beta := \\sum _{i \\in \\stdIndex} \\alpha _i \\) holds, where \\( \\stdIndex = \\{i\\,| \\,\\tz_i > 0\\} \\).\n\t\tThe optimality of this error bound comes from Remark \\ref{remark:F-z-order-beta} and {\\cite[Theorem 3.2]{LiLoPo21}}.\nIf \\( \\stdFace = \\{\\bm{0}\\} \\), which means the feasible region is \\( \\{\\bm{0}\\} \\), then a Lipschitzian error bound holds automatically and it is naturally tight, see \\cite[Proposition~27]{L17}.\n\t\\end{proof}","start":75932,"end":77887}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:conic-feasibility-problem"],"declared_external":false,"source_only":false,"title":"锥可行性问题","source_units":[{"raw_tex":"\tIn a Euclidean space \\( \\EucSpace \\), consider the conic feasibility problem:\n\t\\begin{equation}\n\t\t\\label{eq:conic-feasibility-problem}\\tag{Feas}\n\t\t\\text{find } \\quad \\bm{x} \\in (\\subSpace + \\bm{a}) \\cap \\stdCone,\n\t\\end{equation}\n\twhere \\( \\subSpace \\) is a subspace, \\( \\bm{a} \\in \\EucSpace \\), and \\( \\stdCone\\) is a closed convex cone.\n","start":7572,"end":7911}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"欧氏几何记号","source_units":[{"raw_tex":"\tWe will use plain letters to represent real scalars, bold lowercase letters to denote vectors, bold uppercase letters to stand for matrices,\\footnote{With an abuse of notation, we use $\\bm 0$ to denote a zero vector / matrix, whose dimension should be clear from the context.} and curly capital letters for (sub)spaces and sets. Let \\( \\EucSpace \\) be a finite dimensional Euclidean space, \\( \\R_+ \\) and \\( \\R_- \\) be the set of nonnegative and nonpositive real numbers, respectively.\n\tThe inner product of \\( \\EucSpace \\) is denoted by \\( \\inProd{ \\cdot }{ \\cdot  } \\) and the induced norm by \\( \\| \\cdot \\| \\).\n\tWith that, for \\( \\bm{x} \\in \\EucSpace \\) and a closed convex set \\( \\C \\subseteq \\EucSpace \\), we denote the projection of \\( \\bm{x} \\) onto \\( \\C \\) by \\( P_\\C(\\bm{x}) \\) so that \\( P_\\C(\\bm{x}) = \\argmin_{\\bm{y} \\in \\C} \\| \\bm{x} - \\bm{y} \\| \\) and the distance between \\( \\bm{x} \\) and \\( \\C \\) by \\( \\dist(\\bm{x}, \\C) = \\inf_{\\bm{y} \\in \\C} \\| \\bm{x} - \\bm{y} \\| = \\| \\bm{x} - P_\\C(\\bm{x}) \\| \\).\n\tFor any \\( \\bm{x} \\in \\EucSpace \\) and \\( \\eta \\geq 0 \\), we denote the ball centered at \\( \\bm{x} \\) with radius \\( \\eta  \\) by \\( \\B(\\bm{x}; \\eta) := \\{\\bm{y} \\in \\EucSpace \\,|\\,\\| \\bm{y} - \\bm{x} \\| \\leq \\eta \\} \\); we write \\( \\B(\\eta) \\) for the ball centered at \\( \\bm{0} \\) with radius \\( \\eta \\) for simplicity.\n\tA diagonal matrix with diagonal vector being \\( \\bm{x} \\) is denoted by \\( \\diag(\\bm{x}) \\).\n\tMeanwhile, we use \\( \\C^{\\perp } \\) to denote the orthogonal complement of \\( \\C \\) and \\( \\bm{I}_n \\) to represent the \\( n \\times n \\) identity matrix.\n\n","start":14665,"end":16254}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"一致霍尔德误差界","source_units":[{"raw_tex":"\t{\n\t  We now recall the definition of {Lipschitzian and H\\\"olderian error bounds}.\n\t  Let \\( \\C_1, \\C_2 \\subseteq \\EucSpace \\) be closed convex sets with \\( \\C_1 \\cap \\C_2 \\neq \\emptyset  \\).\n\t  We say that \\( \\C_1, \\C_2 \\) satisfy a \\emph{uniform H\\\"olderian error bound} with exponent \\( \\gamma  \\in (0, 1] \\) if for every bounded set \\( \\B \\subseteq \\EucSpace \\), there exists a constant \\( \\kappa _\\B \\) such that\n\t  $\\dist(\\bm{x}, \\C_1 \\cap \\C_2) \\!\\leq\\! \\kappa _\\B \\max\\left\\{\\dist(\\bm{x}, \\C_1), \\dist(\\bm{x}, \\C_2)\\right\\}^{\\gamma }$ for all $\\bm{x} \\in \\B$.\n\t  If \\( \\gamma  = 1 \\), then the error bound is said to be \\emph{Lipschitzian}.\n\t}\n\n","start":16254,"end":16907}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"锥与暴露面","source_units":[{"raw_tex":"\tLet \\( \\stdCone \\subseteq \\EucSpace \\) be a closed convex cone and \\( \\stdCone^{*} \\) be its dual cone.\n\tWe will use ${\\rm int}\\,\\stdCone $, \\( \\ri \\stdCone \\), \\( \\bd \\stdCone \\), \\( \\Span \\stdCone \\), \\( \\dim \\stdCone \\) to denote the interior, relative interior, boundary, linear span and dimension of \\( \\stdCone \\), respectively.\n\tIf \\( \\stdCone \\cap -\\stdCone = \\{\\bm{0}\\} \\), we say that \\( \\stdCone \\) is \\textit{pointed}.\t\n\n\tA \\textit{face} of \\( \\stdCone \\) is a closed convex cone \\( \\stdFace \\) satisfying \\( \\stdFace \\subseteq \\stdCone \\) and the property that if \\( \\bm{x}, \\bm{y} \\in \\stdCone\\) and \\( \\bm{x} + \\bm{y} \\in \\stdFace \\), then \\( \\bm{x}, \\bm{y} \\in \\stdFace \\).\\footnote{By convention, we discard the empty face.}\n\tWe write \\( \\stdFace \\unlhd \\stdCone \\) if \\( \\stdFace \\) is a face of \\( \\stdCone \\) and \\( \\stdFace \\properideal \\stdCone \\) if \\( \\stdFace \\) is a \\textit{proper} face of \\( \\stdCone \\), i.e., \\( \\stdFace \\neq \\stdCone \\).\n\tA face \\( \\stdFace \\) is said to be \\textit{nontrivial} if \\( \\stdFace \\) is proper and \\( \\stdFace \\neq \\stdCone \\cap -\\stdCone \\).\n\tIf \\( \\stdFace = \\stdCone \\cap \\{\\bm{z}\\}^{\\perp } \\) for some \\( \\bm{z} \\in \\stdCone^{*} \\), \\( \\stdFace \\) is called an \\textit{exposed face} of \\( \\stdCone \\).\n","start":16907,"end":18175}]},{"id":"S005","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"面约化方法背景","source_units":[{"raw_tex":"\t{\n\t  The facial structure of the closed convex cone $\\stdCone$ is important for deducing \\textit{error bounds} for \\eqref{eq:conic-feasibility-problem}; see the seminal work of Sturm \\cite{St00}.\n\t  Recently, a new framework based on the \\textit{facial reduction algorithm} \\cite{BW81, Pa13, WM13} and \\textit{one-step facial residual functions ($\\mathds{1}$-FRFs)} \\cite[Definition 3.4]{LiLoPo20} was proposed for establishing error bounds for \\eqref{eq:conic-feasibility-problem} without requiring any constraint qualifications; see \\cite{L17, LiLoPo20, LiLoPo21}.\n\t  Next, we present a very brief overview of the framework, for more detailed explanations and the underlying intuition behind the techniques see \\cite{LiLoPo20,LiLoPo21}.\n\t  First, we recall the definition of\n\t  one-step facial residual functions.\n\t\n","start":18177,"end":18996}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["def:one-step-facial-residual-function"],"declared_external":true,"declaration_text":["原文明确引用 LiLoPo20, Definition 3.4；该定义在所选原文中完整重述。"],"source_only":false,"title":"一步面残差函数","source_units":[{"raw_tex":"\t  \\begin{definition}[One-step facial residual function {\\cite[Definition 3.4]{LiLoPo20}}]\\label{def:one-step-facial-residual-function}\n\t  \tLet \\( \\mathcal{K} \\) be a closed convex cone and \\( \\bm{z} \\in \\mathcal{K}^{*} \\).\n\t  \tSuppose that \\( \\psi _{\\mathcal{K}, \\bm{z}}: \\R_+ \\times \\R_+ \\to \\R_+ \\) satisfies:% the following properties:\n\t  \t\\begin{enumerate}[label={(\\roman*)}]\n\t  \t\t\\item \\( \\psi _{\\mathcal{K}, \\bm{z}} \\) is nonnegative,  nondecreasing in each argument and for every \\( t \\in \\R_+ \\), \\( \\psi _{\\mathcal{K}, \\bm{z}} (0, t) = 0 \\).\n\t  \t\t\\item The following implication holds for any \\( \\bm{x} \\in \\Span \\mathcal{K} \\) and \\( \\epsilon \\geq 0 \\):\\vspace{-0.15 cm}\n\t  \t\t\\[ \\vspace{-0.15 cm}\n\t  \t\t\\dist(\\bm{x}, \\mathcal{K}) \\leq \\epsilon ,\\,\\inProd{ \\bm{x}}{ \\bm{z} } \\leq \\epsilon \\implies \\dist(\\bm{x}, \\mathcal{K} \\cap \\{\\bm{z}\\}^{\\perp }) \\leq \\psi _{\\mathcal{K}, \\bm{z}} (\\epsilon, \\| \\bm{x} \\|).\n\t  \t\t\\]\n\t  \t\\end{enumerate}\n\t  \tThen, \\( \\psi _{\\mathcal{K}, \\bm{z}} \\) is said to be a one-step facial residual function (\\( \\mathds{1} \\)-FRF) for \\( \\mathcal{K} \\) and \\( \\bm{z} \\).\n\t  \\end{definition}\n","start":18996,"end":20119}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"PPS距离与复合框架","source_units":[{"raw_tex":"\t  The basic idea of the  aforementioned framework  is as follows. Suppose that in each step of the facial reduction algorithm we can find a suitable one-step facial residual function for the ``current'' face and ``next'' exposing vector until we reach a face $\\stdFace$ such that $\\stdFace$ and $\\subSpace + \\bm{a}$ satisfy the \\textit{partial polyhedral Slater's (PPS) condition} \\cite[Definition 3]{L17}.\\footnote{ We note that this implies a Lipschitzian error bound holds for $\\stdFace$ and $\\subSpace + \\bm{a}$, see \\cite[Corollary~3]{BBL99} and the discussion preceding \\cite[Proposition~2.3]{LiLoPo20}.}\n\t  Then, we can  construct an error bound for $\\stdCone$ and $\\subSpace + \\bm{a}$ by composing these residual functions in a specific manner.\n\t  In this regard, if \\eqref{eq:conic-feasibility-problem} is feasible, we define the \\textit{distance to the PPS condition} of \\eqref{eq:conic-feasibility-problem}, denoted by $ \\dpps(\\stdCone, \\subSpace + \\bm{a}) $, as the length \\textit{minus one} of the \\textit{shortest} chain of faces (among those chains constructed as in \\cite[Proposition~5]{L17}) such that the PPS condition holds for the final face in the chain and $\\subSpace + \\bm{a}$.\n\t}\n","start":20119,"end":21324}]},{"id":"S008","semantic_type":"external_fact","kind":"external_fact","labels":["lemma:Lemma-2.11"],"declared_external":true,"declaration_text":["所声明允许前提 LiLoPo21, Lemma 2.5；原文 B0007 完整重述，未附证明。"],"source_only":false,"title":"一维面的投影公式","source_units":[{"raw_tex":"\t{\n\t  We end this section with the following lemma, which is useful in the analysis of one-dimensional faces. It will be used repeatedly in our subsequent discussions.\n\t  \\begin{lemma}[{\\cite[Lemma 2.5]{LiLoPo21}}]\n\t\t\\label{lemma:Lemma-2.11}\n\t\tLet \\( \\stdCone \\) be a pointed closed convex cone and let \\( \\bm{z}\\in \\partial \\stdCone^*\\setminus \\{{\\bm 0}\\} \\) be such that \\( \\stdFace := \\{\\bm{z}\\}^{\\perp} \\cap \\stdCone \\) is a one-dimensional proper face of \\( \\stdCone \\).\n\t\tLet \\( \\bm{f} \\in \\stdCone \\setminus \\{\\bm{0}\\} \\) be such that \\( \\stdFace = \\{t \\bm{f} \\, | \\, t \\geq 0\\} \\).\n\t\tLet \\( \\eta > 0 \\) and \\( \\bm{v} \\in \\partial \\stdCone \\cap B(\\eta) \\setminus \\stdFace \\), \\( \\bm{w} = P_{\\{\\bm{z}\\}^{\\perp}}(\\bm{v}) \\) and \\( \\bm{u} = P_{\\stdFace}(\\bm{w}) \\) with \\( \\bm{u} \\neq \\bm{w} \\).\n\t\tThen it holds that \\( \\left\\langle \\bm{f}, \\bm{z} \\right\\rangle = 0 \\) and we have\n\t\t\\[\n\t\t  \\| \\bm{v} - \\bm{w} \\| = \\frac{| \\left\\langle \\bm{z}, \\bm{v} \\right\\rangle |}{\\| \\bm{z} \\|}, \\qquad \\| \\bm{u} - \\bm{w} \\| =\n\t\t  \\begin{cases}\n\t\t\t\\left\\| \\bm{v} - \\frac{\\left\\langle \\bm{z}, \\bm{v} \\right\\rangle}{\\| \\bm{z} \\|^2}\\bm{z} - \\frac{\\left\\langle \\bm{f}, \\bm{v} \\right\\rangle}{\\| \\bm{f} \\|^2}\\bm{f} \\right\\| & \\text{ if } \\left\\langle \\bm{f}, \\bm{v} \\right\\rangle \\geq 0, \\\\\n\t\t\t\\,\\,\\,\\left\\| \\bm{v} - \\frac{\\left\\langle \\bm{z}, \\bm{v} \\right\\rangle}{\\| \\bm{z} \\|^2} \\bm{z} \\right\\| & \\text{ otherwise }.\n\t\t  \\end{cases}\n\t\t\\]\n\t\tMoreover, when \\( \\left\\langle \\bm{f}, \\bm{v} \\right\\rangle \\geq 0 \\) (or, equivalently, \\( \\left\\langle \\bm{f}, \\bm{w} \\right\\rangle \\geq 0 \\)), we have \\( \\bm{u} = P_{\\Span \\stdFace}(\\bm{w}) \\).\n\t\tOn the other hand, if \\( \\left\\langle \\bm{f}, \\bm{v} \\right\\rangle < 0 \\), we have \\( \\bm{u} = \\bm{0} \\).\n\t  \\end{lemma}\n\t}\n","start":21325,"end":23075}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"广义幂锥参数","source_units":[{"raw_tex":"\tWe consider the generalized power cone and its dual.\n\tLet \\( m\\ge 1 \\), \\( n\\ge 2 \\) and \\( \\bm{\\alpha} = (\\alpha_1,\\ldots,\\alpha_n)\\in (0,1)^n \\) with \\( \\sum_{i=1}^n\\alpha_i = 1 \\), the generalized power cone \\( \\gK \\) and its dual \\( \\dgK \\) are given respectively by\n","start":23165,"end":23437}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":["eq:gpowercones"],"declared_external":false,"source_only":false,"title":"广义幂锥及对偶","source_units":[{"raw_tex":"\t\\begin{equation}\\label{eq:gpowercones}\n\t\\begin{aligned}\n\t\\gK  & = \\left\\{\\bm{x} = (\\bbx,\\tbx)\\in \\R^{m+n}\\,\\bigg| \\,\\|\\bbx\\|\\le \\prod_{i=1}^n\\tx_i^{\\alpha_i},\\,\\bbx\\in \\R^m,\\,\\tbx\\in \\R^n_+\\right\\},                               \\\\\n\t\\dgK & = \\left\\{\\bm{z} = (\\bbz,\\tbz)\\in \\R^{m+n}\\,\\bigg| \\,\\|\\bbz\\|\\le \\prod_{i=1}^n\\left(\\frac{\\tz_i}{\\alpha_i}\\right)^{\\alpha_i},\\,\\bbz\\in \\R^m,\\,\\tbz\\in \\R^n_+\\right\\}.\n\t\\end{aligned}\n\t\\end{equation}\n","start":23437,"end":23874}]},{"id":"S011","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"向量分块","source_units":[{"raw_tex":"\tHere, given a vector \\( \\bm{x}\\in \\R^{m+n} \\), we let \\( \\bbx\\in \\R^m \\) be the vector corresponding to its first \\( m \\) entries and \\( \\tbx\\in \\R^n \\) be the vector corresponding to its last \\( n \\) entries.\n","start":23874,"end":24085}]},{"id":"S012","semantic_type":"lemma","kind":"lemma","labels":["eq:inequ-inprod-power-cone","lemma:inequ-inprod-power-cone"],"declared_external":false,"source_only":false,"title":"乘积曲面的局部二次间隙","source_units":[{"raw_tex":"\\begin{lemma}\\label{lemma:inequ-inprod-power-cone}\n\t\tLet \\( n\\ge 2 \\) and \\( \\bm{\\alpha} = (\\alpha_1,\\ldots,\\alpha_n)\\in (0,1)^n \\) with \\( \\sum_{i=1}^n\\alpha_i = 1 \\). Let \\( \\bm{\\zeta}\\in {\\rm int}\\,\\R_{-}^n \\) satisfy \\( \\prod_{i=1}^n (-\\zeta_i/\\alpha_i)^{\\alpha_i} = 1 \\). Define \\( \\widetilde{\\bm{\\zeta}} := -\\bm\\alpha\\circ \\bm{\\zeta}^{-1} \\), where \\( \\circ \\) is the Hadamard product and the inverse is taken componentwise. Then there exist \\( C > 0 \\) and \\( \\epsilon > 0 \\) so that\n\t\t\\begin{equation}\\label{eq:inequ-inprod-power-cone}\n\t\t\t-1 - \\inProd{\\bm{\\zeta}}{\\bm{\\omega}} \\ge C\\|\\bm{\\omega} - \\widetilde{\\bm{\\zeta}}\\|^2 \\,\\,{whenever}\\,\\bm{\\omega}\\in {\\rm int}\\,\\R^n_+,\\,\\|\\bm{\\omega} - \\widetilde{\\bm{\\zeta}}\\|\\le \\epsilon {\\,and\\,} \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1.\n\t\t\\end{equation}\n","start":24319,"end":25121}]},{"id":"S013","semantic_type":"lemma","kind":"lemma","labels":["lemma:inequ-inprod-power-cone"],"declared_external":false,"source_only":false,"title":"乘积曲面的唯一接触点","source_units":[{"raw_tex":"\\begin{lemma}\\label{lemma:inequ-inprod-power-cone}\n\t\tLet \\( n\\ge 2 \\) and \\( \\bm{\\alpha} = (\\alpha_1,\\ldots,\\alpha_n)\\in (0,1)^n \\) with \\( \\sum_{i=1}^n\\alpha_i = 1 \\). Let \\( \\bm{\\zeta}\\in {\\rm int}\\,\\R_{-}^n \\) satisfy \\( \\prod_{i=1}^n (-\\zeta_i/\\alpha_i)^{\\alpha_i} = 1 \\). Define \\( \\widetilde{\\bm{\\zeta}} := -\\bm\\alpha\\circ \\bm{\\zeta}^{-1} \\), where \\( \\circ \\) is the Hadamard product and the inverse is taken componentwise. Then there exist \\( C > 0 \\) and \\( \\epsilon > 0 \\) so that\n","start":24319,"end":24810},{"raw_tex":"\t\tMoreover, for any \\( \\bm{\\omega}\\in {\\rm int}\\,\\R_+^n \\) satisfying \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\), it holds that \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}\\le-1 \\); furthermore, we have \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}=-1 \\) if and only if \\( \\bm{\\omega} = \\widetilde{\\bm{\\zeta}} \\).\n\t\\end{lemma}","start":25121,"end":25428}]},{"id":"S014","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"乘积曲面引理完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\t\tFor each \\( i \\), we see from the Taylor series of \\( \\ln(\\cdot) \\) at \\( \\widetilde{\\zeta}_i > 0 \\) that\n\t\t\\begin{equation*}\n\t\t\t\\ln(\\omega_i) = \\ln(\\widetilde{\\zeta}_i) + \\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) - \\widetilde{\\zeta}_i^{-2}(\\omega_i - \\widetilde{\\zeta}_i)^2 + O(|\\omega_i - \\widetilde{\\zeta}_i|^3)\\,\\,{\\rm as}\\,\\,\\omega_i \\to \\widetilde{\\zeta}_i, \\,\\,\\omega_i > 0.\n\t\t\\end{equation*}\n\t\tThus, there exist \\( \\epsilon_i > 0 \\) and \\( c_i > 0 \\) so that\n\t\t\\[\n\t\t\\ln(\\widetilde{\\zeta}_i) \\ge \\ln(\\omega_i) - \\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) + c_i (\\omega_i - \\widetilde{\\zeta}_i)^2\\,\\,\\,\\mbox{whenever} \\,|\\omega_i - \\widetilde{\\zeta}_i|\\le \\epsilon_i \\,{\\rm and}\\,\\omega_i > 0.\n\t\t\\]\n\t\tLet \\( \\epsilon := \\min\\limits_{1\\le i\\le n}\\epsilon_i > 0 \\). Multiplying both sides of the above inequality by \\( \\alpha_i \\) and summing the resulting inequalities from \\( i = 1 \\) to \\( n \\), we see that whenever \\( \\bm{\\omega}\\in {\\rm int}\\,\\R^n_+ \\) satisfies \\( \\|\\bm{\\omega} - \\widetilde{\\bm{\\zeta}}\\|\\le \\epsilon \\) and \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\), we have\n\t\t\\begin{align}\n\t\t\t&0  \\overset{\\rm (a)}= \\sum_{i=1}^n\\alpha_i\\ln(\\widetilde{\\zeta}_i)\\ge \\sum_{i=1}^n\\alpha_i\\ln(\\omega_i) - \\sum_{i=1}^n\\alpha_i\\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) + \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2 \\notag\\\\\n\t\t\t& \\overset{\\rm (b)}= - \\sum_{i=1}^n\\alpha_i\\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) + \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2\n\t\t\t\\overset{\\rm (c)}= - \\sum_{i=1}^n\\alpha_i\\widetilde{\\zeta}_i^{-1}\\omega_i \\!+\\! 1 \\!+\\! \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2                                                                                                                          \\notag\\\\\n\t\t\t& = \\sum_{i=1}^n\\zeta_i\\omega_i + 1 + \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2,\\notag\n\t\t\\end{align}\n\t\twhere (a) and (b) hold because \\( \\prod_{i=1}^n\\widetilde{\\zeta}_i^{\\alpha_i} = \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\), (c) uses the fact that \\( \\sum_{i=1}^n\\alpha_i = 1 \\), and the last equality follows from the definition of \\( \\widetilde{\\bm{\\zeta}} \\). Rearranging the above inequality, we conclude that \\eqref{eq:inequ-inprod-power-cone} holds with \\( C = \\min\\limits_{1\\le i\\le n}\\alpha_ic_i > 0 \\).\n\n\t\tNext, let \\( \\bm{\\omega}\\in {\\rm int}\\,\\R_+^n \\) satisfy \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\). Then \\( (-1,\\bm{\\omega})\\in \\K \\). Recall from the assumption that \\( (1,-\\bm{\\zeta})\\in \\dK \\). From these we deduce \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}\\le-1 \\). If \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}=-1 \\), then\n\t\t\\[\n\t\t\\sum_{i=1}^n \\alpha_i\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)\\omega_i = \\sum_{i=1}^n (-\\zeta_i)\\omega_i = 1 = \\prod_{i=1}^n\\omega_i^{\\alpha_i} = \\prod_{i=1}^n\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)^{\\alpha_i}\\prod_{i=1}^n\\omega_i^{\\alpha_i} .\n\t\t\\]\n\t\tTaking \\( \\ln \\) on both sides of the above equality, we see that\n\t\t\\[\n\t\t\\ln\\left[\\sum_{i=1}^n \\alpha_i\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)\\omega_i\\right] = \\sum_{i=1}^n\\alpha_i \\ln\\left[\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)\\omega_i\\right].\n\t\t\\]\n\t\tSince \\( \\ln \\) is strictly concave and \\( \\alpha_i \\in (0,1) \\) for all \\( i \\), we conclude that there exists \\( c > 0 \\) so that \\( \\omega_i\\cdot(-\\zeta_i/\\alpha_i) = c \\) for all \\( i \\). This, together with the facts that\n\t\t\\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = \\prod_{i=1}^n (-\\zeta_i/\\alpha_i)^{\\alpha_i} = 1 \\) and \\( \\sum_{i=1}^n\\alpha_i = 1 \\), gives \\( c = 1 \\). It thus follows that \\( \\bm{\\omega} = \\widetilde{\\bm{\\zeta}} \\). Conversely, it is routine to check that if \\( \\bm{\\omega} = \\widetilde{\\bm{\\zeta}} \\), then \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\) and \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}=-1 \\).\n\t\\end{proof}","start":25430,"end":29267}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"对数局部二次估计","source_units":[{"raw_tex":"\\begin{proof}\n\t\tFor each \\( i \\), we see from the Taylor series of \\( \\ln(\\cdot) \\) at \\( \\widetilde{\\zeta}_i > 0 \\) that\n\t\t\\begin{equation*}\n\t\t\t\\ln(\\omega_i) = \\ln(\\widetilde{\\zeta}_i) + \\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) - \\widetilde{\\zeta}_i^{-2}(\\omega_i - \\widetilde{\\zeta}_i)^2 + O(|\\omega_i - \\widetilde{\\zeta}_i|^3)\\,\\,{\\rm as}\\,\\,\\omega_i \\to \\widetilde{\\zeta}_i, \\,\\,\\omega_i > 0.\n\t\t\\end{equation*}\n\t\tThus, there exist \\( \\epsilon_i > 0 \\) and \\( c_i > 0 \\) so that\n\t\t\\[\n\t\t\\ln(\\widetilde{\\zeta}_i) \\ge \\ln(\\omega_i) - \\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) + c_i (\\omega_i - \\widetilde{\\zeta}_i)^2\\,\\,\\,\\mbox{whenever} \\,|\\omega_i - \\widetilde{\\zeta}_i|\\le \\epsilon_i \\,{\\rm and}\\,\\omega_i > 0.\n\t\t\\]\n","start":25430,"end":26180}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"加权求和取得局部间隙","source_units":[{"raw_tex":"\t\tLet \\( \\epsilon := \\min\\limits_{1\\le i\\le n}\\epsilon_i > 0 \\). Multiplying both sides of the above inequality by \\( \\alpha_i \\) and summing the resulting inequalities from \\( i = 1 \\) to \\( n \\), we see that whenever \\( \\bm{\\omega}\\in {\\rm int}\\,\\R^n_+ \\) satisfies \\( \\|\\bm{\\omega} - \\widetilde{\\bm{\\zeta}}\\|\\le \\epsilon \\) and \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\), we have\n\t\t\\begin{align}\n\t\t\t&0  \\overset{\\rm (a)}= \\sum_{i=1}^n\\alpha_i\\ln(\\widetilde{\\zeta}_i)\\ge \\sum_{i=1}^n\\alpha_i\\ln(\\omega_i) - \\sum_{i=1}^n\\alpha_i\\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) + \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2 \\notag\\\\\n\t\t\t& \\overset{\\rm (b)}= - \\sum_{i=1}^n\\alpha_i\\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) + \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2\n\t\t\t\\overset{\\rm (c)}= - \\sum_{i=1}^n\\alpha_i\\widetilde{\\zeta}_i^{-1}\\omega_i \\!+\\! 1 \\!+\\! \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2                                                                                                                          \\notag\\\\\n\t\t\t& = \\sum_{i=1}^n\\zeta_i\\omega_i + 1 + \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2,\\notag\n\t\t\\end{align}\n\t\twhere (a) and (b) hold because \\( \\prod_{i=1}^n\\widetilde{\\zeta}_i^{\\alpha_i} = \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\), (c) uses the fact that \\( \\sum_{i=1}^n\\alpha_i = 1 \\), and the last equality follows from the definition of \\( \\widetilde{\\bm{\\zeta}} \\). Rearranging the above inequality, we conclude that \\eqref{eq:inequ-inprod-power-cone} holds with \\( C = \\min\\limits_{1\\le i\\le n}\\alpha_ic_i > 0 \\).\n","start":26180,"end":27811}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"对偶性与等号条件","source_units":[{"raw_tex":"\n\t\tNext, let \\( \\bm{\\omega}\\in {\\rm int}\\,\\R_+^n \\) satisfy \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\). Then \\( (-1,\\bm{\\omega})\\in \\K \\). Recall from the assumption that \\( (1,-\\bm{\\zeta})\\in \\dK \\). From these we deduce \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}\\le-1 \\). If \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}=-1 \\), then\n\t\t\\[\n\t\t\\sum_{i=1}^n \\alpha_i\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)\\omega_i = \\sum_{i=1}^n (-\\zeta_i)\\omega_i = 1 = \\prod_{i=1}^n\\omega_i^{\\alpha_i} = \\prod_{i=1}^n\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)^{\\alpha_i}\\prod_{i=1}^n\\omega_i^{\\alpha_i} .\n\t\t\\]\n\t\tTaking \\( \\ln \\) on both sides of the above equality, we see that\n\t\t\\[\n\t\t\\ln\\left[\\sum_{i=1}^n \\alpha_i\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)\\omega_i\\right] = \\sum_{i=1}^n\\alpha_i \\ln\\left[\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)\\omega_i\\right].\n\t\t\\]\n\t\tSince \\( \\ln \\) is strictly concave and \\( \\alpha_i \\in (0,1) \\) for all \\( i \\), we conclude that there exists \\( c > 0 \\) so that \\( \\omega_i\\cdot(-\\zeta_i/\\alpha_i) = c \\) for all \\( i \\). This, together with the facts that\n\t\t\\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = \\prod_{i=1}^n (-\\zeta_i/\\alpha_i)^{\\alpha_i} = 1 \\) and \\( \\sum_{i=1}^n\\alpha_i = 1 \\), gives \\( c = 1 \\). It thus follows that \\( \\bm{\\omega} = \\widetilde{\\bm{\\zeta}} \\). Conversely, it is routine to check that if \\( \\bm{\\omega} = \\widetilde{\\bm{\\zeta}} \\), then \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\) and \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}=-1 \\).\n\t\\end{proof}","start":27811,"end":29267}]},{"id":"S018","semantic_type":"external_fact","kind":"external_fact","labels":["eq:inequ-p=q=2","lemma:Lemma-4.1"],"declared_external":true,"declaration_text":["所声明允许前提 LiLoPo21, Lemma 4.1，原文明确以 p=q=2 专门化并完整重述，未附证明。"],"source_only":false,"title":"欧氏球面的二次间隙","source_units":[{"raw_tex":"\t{\n\t  The next lemma is obtained by applying \\cite[Lemma 4.1]{LiLoPo21} with \\( p = q = 2 \\).\n\t  \\begin{lemma}\n\t\t\\label{lemma:Lemma-4.1}\n\t\tLet \\( \\bm{\\zeta} \\in \\R^n \\) (\\( n \\geq 1 \\)) satisfy \\( \\| \\bm{\\zeta} \\| = 1 \\).\n\t\tDefine \\( \\overline{\\bm{\\zeta}} := -\\bm{\\zeta} \\). Then there exist \\( C > 0 \\) and \\( \\epsilon > 0 \\) so that\n\t\t\\begin{equation}\n\t\t  \\label{eq:inequ-p=q=2}\n\t\t  1 + \\left\\langle \\bm{\\zeta}, \\bm{w} \\right\\rangle \\geq C \\sum_{i \\in I} | w_i - \\overline{\\zeta}_i |^2 + \\frac{1}{2} \\sum_{i \\notin I} | w_i |^2  \\ \\text{ whenever } \\| \\bm{w} - \\overline{\\bm{\\zeta}} \\| \\leq \\epsilon \\ \\text{ and } \\ \\| \\bm{w} \\| = 1,\n\t\t\\end{equation}\n\t\twhere \\( I = \\{i \\,|\\, \\overline{\\zeta}_i \\neq 0\\} \\).\n\t\tFurthermore, for any \\( \\bm{w} \\) satisfying \\( \\| \\bm{w} \\| \\leq  1\\), it holds that \\( \\left\\langle \\bm{\\zeta}, \\bm{w} \\right\\rangle \\geq -1 \\), with the equality holding if and only if \\( \\bm{w} = \\overline{\\bm{\\zeta}} \\).\n\t  \\end{lemma}\n\t}\n","start":29269,"end":30226}]},{"id":"S019","semantic_type":"proposition","kind":"proposition","labels":["eq:1st-type-face-ray","item:case-I-facial-structure","item:case-II-facial-structure","prop:facial-stru-gK"],"declared_external":false,"source_only":false,"title":"非零首块暴露射线","source_units":[{"raw_tex":"\\begin{proposition}[Proper nontrivial exposed faces of \\( \\gK \\)]\\label{prop:facial-stru-gK}\n\t\tLet \\( \\bm{z} = (\\bbz,\\tbz)\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\).\n\t\t\\begin{enumerate}[label={(\\roman*)}]\n\t\t  \\item \\label{item:case-I-facial-structure} If \\( \\bbz \\neq \\bm{0} \\), then \\( \\bm{z} \\) exposes the following one-dimensional face: %\\( \\Fr \\), where\n\t\t\t\t\\begin{equation}\\label{eq:1st-type-face-ray}\n\t\t\t\t  \\displayindent0pt\n\t\t\t\t  \\displaywidth\\textwidth\n\t\t\t\t  \\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK =  \\{t \\bm{f}\\in \\R^{m+n}\\,| \\,t\\ge 0\\}\\,\\,\\mbox{with} \\,\\bm{f}= (-\\bbz/\\|\\bbz\\|^2,\\bm{\\alpha}\\circ \\tbz^{-1}),\n\t\t\t\t\\end{equation}\n\t\t\t\twhere the inverse is taken componentwise.\n","start":30512,"end":31181}]},{"id":"S020","semantic_type":"proposition","kind":"proposition","labels":["eq:2nd-type-face-d-dim","item:case-I-facial-structure","item:case-II-facial-structure","prop:facial-stru-gK"],"declared_external":false,"source_only":false,"title":"零首块暴露坐标面","source_units":[{"raw_tex":"\\begin{proposition}[Proper nontrivial exposed faces of \\( \\gK \\)]\\label{prop:facial-stru-gK}\n\t\tLet \\( \\bm{z} = (\\bbz,\\tbz)\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\).\n\t\t\\begin{enumerate}[label={(\\roman*)}]\n","start":30512,"end":30709},{"raw_tex":"\n\t\t  \\item \\label{item:case-II-facial-structure} If \\( \\bbz = \\bm{0} \\), then \\( \\bm{z} \\) exposes the following face of dimension \\( n - | \\stdIndex | \\):\n\t\t\t\t\\begin{equation}\\label{eq:2nd-type-face-d-dim}\n\t\t\t\t  \\displayindent0pt\n\t\t\t\t  \\displaywidth\\textwidth\n\t\t\t\t  \\Fhatz := \\{\\bm{z}\\}^\\perp \\cap \\gK = \\{\\bm{x} = (\\bbx,\\tbx)\\in \\R_+^{m+n}\\,| \\,\\bbx = \\bm{0},\\,\\tx_i = 0\\ \\forall i\\in \\stdIndex\\},\n\t\t\t\t\\end{equation}\n\t\t\t\twhere \\( \\stdIndex := \\{i \\,|\\, \\tz_i > 0\\}\\neq \\emptyset \\) and \\( | \\stdIndex | \\) denotes the cardinality of $\\stdIndex$.\n\t\t\\end{enumerate}\n\t  \\end{proposition}","start":31181,"end":31767}]},{"id":"S021","semantic_type":"proof","kind":"proof","labels":["eq:form-barx-by-barz-kappa","eq:form-tildex-by-tildez-case-I","eq:inequ-sum-prod-power-case-I","eq:inprod-barz-barx=c-case-I","eq:inprod=0-case-I","eq:log-inequ-sum-prod-power-case-I"],"declared_external":false,"source_only":false,"title":"暴露面分类完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\t  \\ref{item:case-I-facial-structure}: Notice that \\( \\bm{x} = (\\bbx,\\tbx)\\in \\{\\bm{z}\\}^\\perp\\cap \\gK\\backslash \\{\\bm{0}\\} \\) if and only if \\( \\bm{x}\\in \\bd\\gK \\), \\( \\bm{x} \\neq \\bm{0} \\) and %satisfies\n\t  \\begin{equation}\\label{eq:inprod=0-case-I}\n\t\t\\inProd{\\bbz}{\\bbx} + \\inProd{ \\tbz}{\\tbx} = 0.\n\t  \\end{equation}\n\t  The above relation yields\n\t  \\begin{equation}\\label{eq:inequ-sum-prod-power-case-I}\n\t\t\\sum_{i=1}^n \\tz_i\\tx_i = -\\inProd{\\bbz}{\\bbx}\\le \\|\\bbz\\| \\|\\bbx\\| \\le \\prod_{i=1}^n\\left(\\frac{\\tx_i\\tz_i}{\\alpha_i}\\right)^{\\alpha_i},\n\t  \\end{equation}\n\t  where the last inequality follows from the definition of $\\gK$ in \\eqref{eq:gpowercones}.\n\n\t  Note that \\( \\tx_i \\) cannot be \\textbf{all} zero, for otherwise \\( \\bbx \\) will also be zero since \\( \\bm{x}\\in \\bd\\gK \\), which contradicts \\( \\bm{x}\\neq \\bm{0} \\). In addition, we must have \\( \\tz_i > 0 \\) for all \\( i \\) because \\( \\bbz \\neq \\bm{0} \\) and \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\). Using these observations, we have $\\sum_{i=1}^n \\tz_i\\tx_i >0$. Combining this with \\eqref{eq:inequ-sum-prod-power-case-I}, we deduce that \\( \\tx_i\\tz_i > 0 \\) for all \\( i \\). Now we can take \\( \\ln \\) on both sides of \\eqref{eq:inequ-sum-prod-power-case-I} to obtain\n\t  \\begin{equation}\\label{eq:log-inequ-sum-prod-power-case-I}\n\t\t\\ln\\left[\\sum_{i=1}^n \\alpha_i\\left(\\frac{\\tx_i\\tz_i}{\\alpha_i}\\right)\\right] \\le \\alpha_1\\ln\\left(\\frac{\\tx_1\\tz_1}{\\alpha_1}\\right) + \\cdots + \\alpha_n\\ln\\left(\\frac{\\tx_n\\tz_n}{\\alpha_n}\\right).\n\t  \\end{equation}\n\t  Using this together with the fact that \\( \\ln(\\cdot) \\) is strictly concave, we deduce that \\eqref{eq:log-inequ-sum-prod-power-case-I} holds as an equality. Hence, there exists a constant \\( c > 0 \\) so that\n\t  \\begin{equation}\\label{eq:form-tildex-by-tildez-case-I}\n\t\t\\tx_i = c \\alpha_i\\tz_i^{-1}\\,\\,\\,\\forall i = 1, 2, \\dots , n.\n\t  \\end{equation}\n\t  Plugging \\eqref{eq:form-tildex-by-tildez-case-I} into \\eqref{eq:inprod=0-case-I}, we obtain\n\t  \\begin{equation}\\label{eq:inprod-barz-barx=c-case-I}\n\t\t\\inProd{\\bbz}{\\bbx} = -\\inProd{ \\tbz}{\\tbx} = -c\\sum_{i=1}^n \\alpha_i = -c.\n\t  \\end{equation}\n\t  Moreover, using \\eqref{eq:form-tildex-by-tildez-case-I} and the last relation in \\eqref{eq:inequ-sum-prod-power-case-I}, we see that\n\t  \\[\n\t\t\\|\\bbz\\| \\|\\bbx\\| \\le \\prod_{i=1}^n\\left(\\frac{\\tx_i\\tz_i}{\\alpha_i}\\right)^{\\alpha_i} = c.\n\t  \\]\n\t  {The two displayed lines} above show that \\( \\|\\bbz\\| \\|\\bbx\\| = -\\inProd{\\bbz}{\\bbx} \\), which together with \\( \\bbz \\neq \\bm{0} \\) implies that there exists \\( \\kappa > 0 \\) so that\n\t  \\begin{equation}\\label{eq:form-barx-by-barz-kappa}\n\t\t\\bbx = -\\kappa\\bbz.\n\t  \\end{equation}\n\t  Plugging \\eqref{eq:form-barx-by-barz-kappa} into \\eqref{eq:inprod-barz-barx=c-case-I}, we obtain that \\( \\kappa = c/\\|\\bbz\\|^2 \\). Using this together with \\eqref{eq:form-tildex-by-tildez-case-I} and \\eqref{eq:form-barx-by-barz-kappa}, we can now conclude that\n\t  \\[\n\t\t\\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK =  \\{t \\bm{f}\\in \\R^{m+n}\\,| \\,t\\ge 0\\}\\,\\,\\mbox{with} \\,\\bm{f}= (-\\bbz/\\|\\bbz\\|^2,\\bm{\\alpha}\\circ \\tbz^{-1}),\n\t  \\]\n\t  where the inverse is taken componentwise.\n\n\t  \\ref{item:case-II-facial-structure}: In this case, \\( \\bbz = \\bm{0} \\). Then \\( \\stdIndex:= \\{i\\,| \\,\\tz_i > 0\\} \\) is nonempty because \\( \\bm{z}\\neq \\bm{0} \\). Hence, \\( \\bm{x} = (\\bbx,\\tbx)\\in \\{\\bm{z}\\}^\\perp\\cap \\gK\\backslash \\{\\bm{0}\\} \\) if and only if \\( \\bm{x}\\in \\bd\\gK\\backslash\\{\\bm{0}\\} \\) and satisfies\n\t  \\begin{equation*}\n\t\t\\sum_{i\\in \\stdIndex} \\tz_i\\tx_i = 0.\n\t  \\end{equation*}\n\t  This means that \\( \\tx_i = 0 \\) whenever \\( i\\in \\stdIndex \\) and hence \\( \\bbx= \\bm{0} \\). Thus,\n\t  \\[\n\t\t\\Fhatz := \\{\\bm{z}\\}^\\perp \\cap \\gK = \\{\\bm{x} = (\\bbx,\\tbx)\\in \\R_+^{m+n}\\,| \\,\\bbx = \\bm{0},\\,\\tx_i = 0\\ \\forall i\\in \\stdIndex\\}.\n\t  \\]\n\t\\end{proof}","start":31773,"end":35553}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","labels":["eq:inequ-sum-prod-power-case-I","eq:inprod=0-case-I"],"declared_external":false,"source_only":false,"title":"正交关系与坐标正性","source_units":[{"raw_tex":"\\begin{proof}\n\t  \\ref{item:case-I-facial-structure}: Notice that \\( \\bm{x} = (\\bbx,\\tbx)\\in \\{\\bm{z}\\}^\\perp\\cap \\gK\\backslash \\{\\bm{0}\\} \\) if and only if \\( \\bm{x}\\in \\bd\\gK \\), \\( \\bm{x} \\neq \\bm{0} \\) and %satisfies\n\t  \\begin{equation}\\label{eq:inprod=0-case-I}\n\t\t\\inProd{\\bbz}{\\bbx} + \\inProd{ \\tbz}{\\tbx} = 0.\n\t  \\end{equation}\n\t  The above relation yields\n\t  \\begin{equation}\\label{eq:inequ-sum-prod-power-case-I}\n\t\t\\sum_{i=1}^n \\tz_i\\tx_i = -\\inProd{\\bbz}{\\bbx}\\le \\|\\bbz\\| \\|\\bbx\\| \\le \\prod_{i=1}^n\\left(\\frac{\\tx_i\\tz_i}{\\alpha_i}\\right)^{\\alpha_i},\n\t  \\end{equation}\n\t  where the last inequality follows from the definition of $\\gK$ in \\eqref{eq:gpowercones}.\n\n\t  Note that \\( \\tx_i \\) cannot be \\textbf{all} zero, for otherwise \\( \\bbx \\) will also be zero since \\( \\bm{x}\\in \\bd\\gK \\), which contradicts \\( \\bm{x}\\neq \\bm{0} \\). In addition, we must have \\( \\tz_i > 0 \\) for all \\( i \\) because \\( \\bbz \\neq \\bm{0} \\) and \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\). Using these observations, we have $\\sum_{i=1}^n \\tz_i\\tx_i >0$. Combining this with \\eqref{eq:inequ-sum-prod-power-case-I}, we deduce that \\( \\tx_i\\tz_i > 0 \\) for all \\( i \\). Now we can take \\( \\ln \\) on both sides of \\eqref{eq:inequ-sum-prod-power-case-I} to obtain\n","start":31773,"end":33023}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","labels":["eq:form-tildex-by-tildez-case-I","eq:log-inequ-sum-prod-power-case-I"],"declared_external":false,"source_only":false,"title":"严格凹性固定尾块比例","source_units":[{"raw_tex":"\t  \\begin{equation}\\label{eq:log-inequ-sum-prod-power-case-I}\n\t\t\\ln\\left[\\sum_{i=1}^n \\alpha_i\\left(\\frac{\\tx_i\\tz_i}{\\alpha_i}\\right)\\right] \\le \\alpha_1\\ln\\left(\\frac{\\tx_1\\tz_1}{\\alpha_1}\\right) + \\cdots + \\alpha_n\\ln\\left(\\frac{\\tx_n\\tz_n}{\\alpha_n}\\right).\n\t  \\end{equation}\n\t  Using this together with the fact that \\( \\ln(\\cdot) \\) is strictly concave, we deduce that \\eqref{eq:log-inequ-sum-prod-power-case-I} holds as an equality. Hence, there exists a constant \\( c > 0 \\) so that\n\t  \\begin{equation}\\label{eq:form-tildex-by-tildez-case-I}\n\t\t\\tx_i = c \\alpha_i\\tz_i^{-1}\\,\\,\\,\\forall i = 1, 2, \\dots , n.\n\t  \\end{equation}\n","start":33023,"end":33656}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","labels":["eq:form-barx-by-barz-kappa","eq:inprod-barz-barx=c-case-I"],"declared_external":false,"source_only":false,"title":"等号条件固定射线方向","source_units":[{"raw_tex":"\t  Plugging \\eqref{eq:form-tildex-by-tildez-case-I} into \\eqref{eq:inprod=0-case-I}, we obtain\n\t  \\begin{equation}\\label{eq:inprod-barz-barx=c-case-I}\n\t\t\\inProd{\\bbz}{\\bbx} = -\\inProd{ \\tbz}{\\tbx} = -c\\sum_{i=1}^n \\alpha_i = -c.\n\t  \\end{equation}\n\t  Moreover, using \\eqref{eq:form-tildex-by-tildez-case-I} and the last relation in \\eqref{eq:inequ-sum-prod-power-case-I}, we see that\n\t  \\[\n\t\t\\|\\bbz\\| \\|\\bbx\\| \\le \\prod_{i=1}^n\\left(\\frac{\\tx_i\\tz_i}{\\alpha_i}\\right)^{\\alpha_i} = c.\n\t  \\]\n\t  {The two displayed lines} above show that \\( \\|\\bbz\\| \\|\\bbx\\| = -\\inProd{\\bbz}{\\bbx} \\), which together with \\( \\bbz \\neq \\bm{0} \\) implies that there exists \\( \\kappa > 0 \\) so that\n\t  \\begin{equation}\\label{eq:form-barx-by-barz-kappa}\n\t\t\\bbx = -\\kappa\\bbz.\n\t  \\end{equation}\n\t  Plugging \\eqref{eq:form-barx-by-barz-kappa} into \\eqref{eq:inprod-barz-barx=c-case-I}, we obtain that \\( \\kappa = c/\\|\\bbz\\|^2 \\). Using this together with \\eqref{eq:form-tildex-by-tildez-case-I} and \\eqref{eq:form-barx-by-barz-kappa}, we can now conclude that\n\t  \\[\n\t\t\\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK =  \\{t \\bm{f}\\in \\R^{m+n}\\,| \\,t\\ge 0\\}\\,\\,\\mbox{with} \\,\\bm{f}= (-\\bbz/\\|\\bbz\\|^2,\\bm{\\alpha}\\circ \\tbz^{-1}),\n\t  \\]\n\t  where the inverse is taken componentwise.\n","start":33656,"end":34893}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"非负加权和刻画坐标面","source_units":[{"raw_tex":"\n\t  \\ref{item:case-II-facial-structure}: In this case, \\( \\bbz = \\bm{0} \\). Then \\( \\stdIndex:= \\{i\\,| \\,\\tz_i > 0\\} \\) is nonempty because \\( \\bm{z}\\neq \\bm{0} \\). Hence, \\( \\bm{x} = (\\bbx,\\tbx)\\in \\{\\bm{z}\\}^\\perp\\cap \\gK\\backslash \\{\\bm{0}\\} \\) if and only if \\( \\bm{x}\\in \\bd\\gK\\backslash\\{\\bm{0}\\} \\) and satisfies\n\t  \\begin{equation*}\n\t\t\\sum_{i\\in \\stdIndex} \\tz_i\\tx_i = 0.\n\t  \\end{equation*}\n\t  This means that \\( \\tx_i = 0 \\) whenever \\( i\\in \\stdIndex \\) and hence \\( \\bbx= \\bm{0} \\). Thus,\n\t  \\[\n\t\t\\Fhatz := \\{\\bm{z}\\}^\\perp \\cap \\gK = \\{\\bm{x} = (\\bbx,\\tbx)\\in \\R_+^{m+n}\\,| \\,\\bbx = \\bm{0},\\,\\tx_i = 0\\ \\forall i\\in \\stdIndex\\}.\n\t  \\]\n\t\\end{proof}","start":34893,"end":35553}]},{"id":"S026","semantic_type":"theorem","kind":"theorem","labels":["eq:gamma-1/2-1st-type-face","thm:Holderian-1/2-1st-type-face"],"declared_external":false,"source_only":false,"title":"射线面的平方根误差界","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm:Holderian-1/2-1st-type-face}\n\t\tLet \\( \\bm{z} = (\\bbz,\\tbz)\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz\\neq \\bm{0} \\) and let \\( \\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Let \\( \\eta > 0 \\) and define\n\t\t\\begin{equation}\\label{eq:gamma-1/2-1st-type-face}\n\t\t\t\\gamma_{\\bm{z},\\eta}\\! :=\\! \\inf_{\\bm{v}}\\left\\{\\frac{\\|\\bm{v} - \\bm{w}\\|^\\frac12}{\\|\\bm{u} - \\bm{w}\\|}\\,\\bigg| \\,\n\\begin{array}{c}\n\\bm{v}\\in \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fr,\\,\\bm{w} = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}),\\\\\n\\bm{u} = P_{\\Fr}(\\bm{w}),\\,\\bm{u} \\neq \\bm{w}\n\\end{array}\n\\right\\}.\n\t\t\\end{equation}\n\t\tThen it holds that \\( \\gamma_{\\bm{z},\\eta}\\in (0,\\infty] \\) and that\n\t\t\\[\n\t\t\\dist(\\bm{q},\\Fr)\\le \\max\\{2\\sqrt{\\eta},2\\gamma_{\\bm{z},\\eta}^{-1}\\}\\cdot\\dist(\\bm{q},\\gK)^{\\frac{1}{2}}\\,\\,\\text{ whenever }  \\bm{q} \\in \\{\\bm{z}\\}^\\perp\\cap \\B(\\eta) .\n\t\t\\]\n\t\\end{theorem}","start":38549,"end":39400}]},{"id":"S027","semantic_type":"proof","kind":"proof","labels":["eq:def-h","eq:dist-vk-wk","eq:forcontradiction-1st-face","eq:keylemmare2","eq:keylemmare4","eq:keylemmarel","eq:sum-out-of-neighbor","eq:upbd-dist-uk-wk-two-parts","eq:upbd-part-dist-uk-wk-case-2","eq:vk0-notation-nonzero","eq:z0-v0-notation","eqn:3Ms"],"declared_external":false,"source_only":false,"title":"射线面估计完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\t\tSuppose for a contradiction that \\( \\gamma_{\\bm{z},\\eta} = 0 \\). Then, {in view of \\cite[Lemma 3.12]{LiLoPo20}}, there exist \\( \\widehat{\\bm{v}}\\in \\Fr \\) and a sequence \\( \\{\\bm{v}^k\\}\\subset \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fr \\) such that\n\t\t\\begin{equation}\\label{eq:forcontradiction-1st-face}\n\t\t\t\\lim_{k\\to \\infty}\\bm{v}^k = \\lim_{k\\to \\infty}\\bm{w}^k = \\widehat{\\bm{v}}\\,\\,{\\rm and}\\,\\,\\lim_{k\\to\\infty}\\frac{\\|\\bm{w}^k - \\bm{v}^k\\|^{\\frac{1}{2}}}{\\|\\bm{w}^k - \\bm{u}^k\\|} = 0,\n\t\t\\end{equation}\n\t\twhere \\( \\bm{w}^k = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}^k) \\), \\( \\bm{u}^k = P_{\\Fr}(\\bm{w}^k) \\) and \\( \\bm{u}^k\\neq \\bm{w}^k \\).\n\n\t\tDefine, for notational simplicity, \\( z_0 := \\|\\bbz\\| \\) and \\( v^k_0:=\\|\\bbv^k\\| \\).\n\t\tThen, since \\( \\{\\bm{v}^k\\}\\subset \\bd \\gK \\) and \\( \\bm{z} \\in \\bd\\dgK \\) with \\( \\bbz \\neq \\bm{0} \\), we have\\vspace{-0.15 cm}\n\t\t\\begin{equation}\\label{eq:z0-v0-notation}\\vspace{-0.15 cm}\n\t\t\tz_0 = \\|\\bbz\\| = \\prod_{i=1}^n\\left(\\frac{\\tz_i}{\\alpha_i}\\right)^{\\alpha_i}> 0 \\,\\,{\\rm and}\\,\\,v^k_0=\\|\\bbv^k\\| = \\prod_{i=1}^n(\\tv_i^k)^{\\alpha_i}\\,\\,\\forall k.\n\t\t\\end{equation}\n\n\t\tIf it holds that \\( v_0^k = 0 \\) infinitely often, by passing to a further subsequence, we may assume that \\( v_0^k = 0 \\) for all \\( k \\). Then we have in view of {Lemma \\ref{lemma:Lemma-2.11}} that\\vspace{-0.15 cm}\n\t\t\\[\\vspace{-0.15 cm}\n\t\t\\!\\|\\bm{v}^k - \\bm{w}^k\\| \\!=\\! \\frac1{\\|\\bm{z}\\|}|\\inProd{ \\tbz}{\\tbv^k}|\\!\\overset{\\rm (a)}=\\! \\frac1{\\|\\bm{z}\\|}\\sum_{i=1}^n\\tz_i\\tv^k_i\\!\\ge\\! \\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|} \\|\\tbv^k\\|_1\\!\\ge\\! \\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|} \\|\\tbv^k\\|\n\t\t\\!\\overset{\\rm (b)}=\\!\\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|} \\|\\bm{v}^k\\|,\n\t\t\\]\n\t\twhere (a) holds because \\( \\tv^k_i \\ge 0 \\) and \\( \\tz_i > 0 \\) for all \\( i \\) (see \\eqref{eq:z0-v0-notation}), and (b) holds since \\( \\|\\bbv^k\\| = 0 \\). Since\n\t\t\\( \\|\\bm{w}^k - \\bm{u}^k\\| = \\dist(\\bm{w}^k,\\Fr) \\le \\|\\bm{w}^k\\|\\le \\|\\bm{v}^k\\| \\) as a consequence of the properties of projections, we conclude from this and the above display that\n\t\t\\( \\|\\bm{v}^k - \\bm{w}^k\\| \\ge \\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|}\\|\\bm{w}^k - \\bm{u}^k\\| \\), contradicting \\eqref{eq:forcontradiction-1st-face}.\n\n\t\tThus, by considering a further subsequence if necessary, from now on, we assume\\vspace{-0.15 cm}\n\t\t\\begin{equation}\\label{eq:vk0-notation-nonzero}\\vspace{-0.15 cm}\n\t\t\tv^k_0=\\|\\bbv^k\\| = \\prod_{i=1}^n(\\tv_i^k)^{\\alpha_i} > 0 \\,\\,\\forall k.\n\t\t\\end{equation}\n\t\tUsing {Lemma \\ref{lemma:Lemma-2.11}}, we see that\\vspace{-0.15 cm}\n\t\t\\begin{equation}\\label{eq:dist-vk-wk}\\vspace{-0.15 cm}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\|\\bm{v}^k - \\bm{w}^k\\| & =\\frac1{\\|\\bm{z}\\|}|\\inProd{ \\bm{z}}{\\bm{v}^k}| = \\frac1{\\|\\bm{z}\\|}\\left|\\sum_{i=1}^{m}\\bz_i\\bv^k_i + \\sum_{i=1}^n\\tz_i\\tv^k_i\\right|             \\\\\n\t\t\t\t& = \\frac1{\\|\\bm{z}\\|}\\left|z_0v_0^k + \\sum_{i=1}^{m}\\bz_i\\bv^k_i - \\sum_{i=1}^n(-\\tz_i)\\tv^k_i - z_0v_0^k\\right|                                        \\\\\n\t\t\t\t& = \\frac{z_0v_0^k}{\\|\\bm{z}\\|}\\left| 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1\\right|  \\\\\n\t\t\t\t& = \\frac{z_0}{\\|\\bm{z}\\|}\\left( 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1\\right)v_0^k,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\twhere the last equality holds as \\( \\|z_0^{-1}\\bbz\\|= 1 \\), \\( \\|(v_0^k)^{-1}\\bbv^k\\|= 1 \\) and  \\( \\inProd{ z_0^{-1}\\tbz}{ (v_0^k)^{-1}\\tbv^k} \\ge 1 \\), thanks to \\eqref{eq:z0-v0-notation}, \\eqref{eq:vk0-notation-nonzero} and Lemma~\\ref{lemma:inequ-inprod-power-cone} applied with \\( \\bm{\\zeta} = -z_0^{-1}\\tbz \\).\n\n\t\tLet \\( \\bm{f} \\) be defined as in \\eqref{eq:1st-type-face-ray}. We consider two cases:\n\t\t\\begin{enumerate}[label={(\\Roman*)}]\n\t\t\t\\item \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}\\ge 0 \\) for all sufficiently large \\( k \\).\n\t\t\t\\item \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}< 0 \\) infinitely often.\n\t\t\\end{enumerate}\n\n\t\t(I): By passing to a further subsequence, we may assume that \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}\\ge 0 \\) for all \\( k \\). In this case, if we define\n\t\t\\[\n\t\t\\bm{Q} = \\bm{I}_{m + n} - \\frac{\\bm{z}\\bm{z}^{\\top}}{\\|\\bm{z}\\|^2} - \\frac{\\bm{f}\\bm{f}^{\\top}}{\\|\\bm{f}\\|^2},\n\t\t\\]\n\t\twhere \\( \\bm{f} \\) is as in \\eqref{eq:1st-type-face-ray}, then we see from {Lemma \\ref{lemma:Lemma-2.11}} and \\eqref{eq:z0-v0-notation} that\n\t\t\\begin{equation}\\label{eq:upbd-dist-uk-wk-two-parts}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\|\\bm{u}^k - \\bm{w}^k\\| & = \\|\\bm{Q}\\bm{v}^k\\| = v_0^k\\left\\|\\bm{Q}\\begin{bmatrix}\n\t\t\t\t\t(v_0^k)^{-1}\\bbv^k \\\\ (v_0^k)^{-1}\\tbv^k\n\t\t\t\t\\end{bmatrix}\\right\\| \\\\\n&\\overset{\\rm (a)}= v_0^k{\\Bigg\\|}\\bm{Q}\\begin{bmatrix}\n\t\t\t\t\t(v_0^k)^{-1}\\bbv^k \\\\ (v_0^k)^{-1}\\tbv^k\n\t\t\t\t\\end{bmatrix} \\!-\\! \\bm{Q}\\underbrace{\\begin{bmatrix}\n\t\t\t\t\t\t-z_0^{-1}\\bbz \\\\ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\n\t\t\t\t\\end{bmatrix}}_{z_0\\bm{f}}{\\Bigg\\|} \\\\\n\t\t\t\t& \\le v_0^k\\left[\\|(v_0^k)^{-1}\\bbv^k + z_0^{-1}\\bbz\\| + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\right],\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\twhere (a) holds because %\\( z_0\\bm{f} = \\begin{bmatrix}\n\t\t%\t\t-z_0^{-1}\\bbz^{\\top} & \\alpha\\circ(z_0\\tbz^{-1})^{\\top}\n\t\t%\t\\end{bmatrix}^{\\top}\\)  and\n\t\t\\( \\bm{Q}\\bm{f} = \\bm{0} \\) (an identity which is clear from the definitions).\n\n\t\tNext, in view of \\eqref{eq:z0-v0-notation}, we can apply {Lemma \\ref{lemma:Lemma-4.1}} to obtain \\( C_1 > 0 \\) and \\( \\epsilon_1>0 \\) so that {\\eqref{eq:inequ-p=q=2}} holds with \\( \\overline{\\bm{\\zeta}} = -z_0^{-1}\\bbz \\), i.e.,\n\t\t\\[\n\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_1\\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|^2\n\t\t\\]\n\t\twhenever \\( \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\le \\epsilon_1 \\) and \\( \\|\\bm{\\omega}\\|=1 \\). On the other hand, in view of the positivity of $1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}$ when $\\|\\bm{\\omega}\\|=1$ and $\\bm{\\omega} \\neq -z_0^{-1}\\bbz$ {(see Lemma \\ref{lemma:Lemma-4.1})}, we know that\n\\[\nC_2 := \\inf_{\\|\\bm{\\omega}\\|=1}\\{1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}} \\,|\\, \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\ge \\epsilon_1 \\} > 0.\n\\]\nThis together with the fact $\\|z_0^{-1}\\bbz\\|=1$ (see \\eqref{eq:z0-v0-notation}) implies that\n\t\t\\[\n\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_2 \\ge 0.25 C_2 \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|^2,\n\t\t\\]\n\t\twhenever \\( \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\ge \\epsilon_1 \\) and \\( \\|\\bm{\\omega}\\|=1 \\). We thus have (with $C_3 \\!:=\\! \\min\\{C_1,C_2/4\\}$)\n\\begin{equation}\\label{eq:keylemmarel}\n\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_3\\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|^2\\ \\ \\ {\\rm whenever}\\ \\ \\|\\bm{\\omega}\\|=1.\n\t\t\\end{equation}\n\n\t\tIn addition, noting \\eqref{eq:z0-v0-notation} again, we can apply Lemma~\\ref{lemma:inequ-inprod-power-cone} with \\( \\bm{\\zeta} = -z_0^{-1}\\tbz\\in {\\rm int}\\,\\R^n_- \\) to obtain \\( C_4 > 0 \\) and \\( \\epsilon > 0 \\) so that \\eqref{eq:inequ-inprod-power-cone} holds with \\( \\widetilde{\\bm{\\zeta}} = \\bm{\\alpha}\\circ (z_0\\tbz^{-1}) \\), i.e.,\n\t\t\\begin{equation}\\label{eq:keylemmare2}\n\t\t\t-1 + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}\\ge C_4\\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2\n\t\t\\end{equation}\n\t\twhenever \\( \\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\le \\epsilon \\), \\( \\bm{\\omega}\\in {\\rm int}\\,\\R^n_+ \\) and \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i}=1 \\).\n\n%Then we have in view of the positivity of $1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}$ when $\\bm{\\omega} \\neq -z_0^{-1}\\bbz$ (see Lemma \\ref{lemma:Lemma-4.1}) and the compactness of the unit sphere that there exists \\( C_3>0 \\) so that\n%\t\t\\begin{equation}\\label{eq:keylemmare3}\n%\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_3 \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\n%\t\t\\end{equation}\n%\t\twhenever \\( \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\ge \\epsilon \\) and \\( \\|\\bm{\\omega}\\|=1 \\).\n\nFurthermore, consider \\( h:\\R^n\\to \\R\\cup\\{\\infty\\} \\) defined by\n\t\t\\begin{equation}\\label{eq:def-h}\n\t\t\th(\\bm{\\omega}) = \\begin{cases}\n\t\t\t\t\\displaystyle\\frac{\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}-1}{\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|} & {\\rm if}\\,\\|\\bm{\\omega}-\\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\ge \\epsilon,\\,\\bm{\\omega}\\in \\Upsilon, \\\\\n\t\t\t\t\\infty                                                                                                    & {\\rm otherwise},\n\t\t\t\\end{cases}\n\t\t\\end{equation}\n\t\twhere \\( \\Upsilon= \\{\\bm{\\omega}\\in \\R^n_+\\,| \\,\\prod_{i=1}^n \\omega_i^{\\alpha_i}=1\\} \\). Then we have\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t& \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty}h(\\bm{\\omega}) = \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty, \\bm{\\omega}\\in \\Upsilon}\\frac{\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}-1}{\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|} \\\\\n\t\t\t& \\ge \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty,\\bm{\\omega}\\in \\R^n_+}\\frac{\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}-1}{\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|}\n\t\t\t\\overset{\\rm (a)}{\\ge} \\inf_{\\|\\bm{\\lambda}\\|=1,\\bm{\\lambda}\\in \\R^n_+}\\inProd{ z_0^{-1}\\tbz}{\\bm{\\lambda}} \\overset{\\rm (b)}\\ge \\min_{1\\le i\\le n}z_0^{-1}\\tz_i > 0.\n\t\t\\end{aligned}\n\t\t\\]\n\t\tHere (a) may be verified by multiplying both numerator and denominator of the left side by $1/\\|\\bm{\\omega}\\|$; (b) holds because \\( z_0^{-1}\\tz_i > 0 \\) for all \\( i \\) (see \\eqref{eq:z0-v0-notation}). Since \\( h \\) in \\eqref{eq:def-h} is also lower semicontinuous on any compact set and {\\bf is always positive},\\footnote{The positivity can be seen by applying Lemma~\\ref{lemma:inequ-inprod-power-cone} with $\\bm\\zeta = -z_0^{-1}\\tbz$.} it must then hold that \\( C_5:= \\inf h > 0 \\). In particular, this means that\n\t\t\\begin{equation}\\label{eq:keylemmare4}\n\t\t\t-1 + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}\\ge C_5\\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\n\t\t\\end{equation}\n\t\twhenever \\( \\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\ge \\epsilon \\), \\( \\bm{\\omega}\\in {\\rm int}\\,\\R^n_+ \\) and \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i}=1 \\).\n\n\t\tBy passing to suitable subsequences, we will end up with one of the following two cases:\n\t\t\\begin{enumerate}[label={Case \\arabic*:}]\n\t\t\t\\item \\( \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| \\le \\epsilon \\) for all \\( k \\). Then we have from \\eqref{eq:keylemmarel} (with $\\bm{\\omega}=(v_0^k)^{-1}\\bbv^k$) and \\eqref{eq:keylemmare2} (with $\\bm{\\omega}=(v_0^k)^{-1}\\tbv^k$) that for these \\( k \\)\n\t\t\t\\begin{equation*}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t& 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1          \\\\\n\t\t\t\t\t& \\ge \\min\\{C_3,C_4\\}(\\|(v_0^k)^{-1}\\bbv^k  + z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2).\n\t\t\t\t\\end{aligned}\n\t\t\t\\end{equation*}\n\t\t\tCombining this with \\eqref{eq:dist-vk-wk} and \\eqref{eq:upbd-dist-uk-wk-two-parts}, we see further that\n\t\t\t\\[\n\t\t\t\\begin{aligned}\n\t\t\t\t&\\|\\bm{v}^k - \\bm{w}^k\\|\\\\ & \\ge \\frac{z_0}{\\|\\bm{z}\\|}\\min\\{C_3,C_4\\}(\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2) v_0^k    \\\\\n\t\t\t\t& \\ge \\frac{z_0}{2\\|\\bm{z}\\|}\\min\\{C_3,C_4\\}(\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\| + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|)^2 v_0^k     \\\\\n\t\t\t\t& \\ge \\frac{z_0\\min\\{C_3,C_4\\}}{2\\|\\bm{z}\\|v_0^k}\\|\\bm{u}^k - \\bm{w}^k\\|^2\\ge \\frac{z_0\\min\\{C_3,C_4\\}}{2\\|\\bm{z}\\|\\eta}\\|\\bm{u}^k - \\bm{w}^k\\|^2,\n\t\t\t\\end{aligned}\n\t\t\t\\]\n\t\t\twhere the last inequality holds because \\( \\bm{v}^k\\in \\B(\\eta) \\). The above display contradicts \\eqref{eq:forcontradiction-1st-face} and hence Case 1 cannot happen.\n\n\t\t\t\\item \\( \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\ge \\epsilon \\) for all \\( k \\). Then we have from \\eqref{eq:keylemmarel} and \\eqref{eq:keylemmare4} that for these \\( k \\)\n\t\t\t\\begin{equation*}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t& 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1        \\\\\n\t\t\t\t\t& \\ge \\min\\{C_3,C_5\\}(\\|(v_0^k)^{-1}\\bbv^k  + z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|).\n\t\t\t\t\\end{aligned}\n\t\t\t\\end{equation*}\n\t\t\tUsing this together with \\eqref{eq:dist-vk-wk}, we deduce that for all large \\( k \\),\n\t\t\t\\[\n\t\t\t\\!\\|\\bm{v}^k - \\bm{w}^k\\| \\!\\ge\\! \\frac{z_0\\min\\{C_3,C_5\\}}{\\|\\bm{z}\\|}(\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|) v_0^k.\n\t\t\t\\]\n\t\t\tThis implies that\n\t\t\t\\begin{equation}\\label{eq:upbd-part-dist-uk-wk-case-2}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\|            & \\le M_1\\sqrt{(v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|}, \\\\\n\t\t\t\t\t\\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| & \\le M_1(v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|,\n\t\t\t\t\\end{aligned}\n\t\t\t\\end{equation}\n\t\t\twhere \\( M_1 := \\max\\left\\{\\left(\\frac{z_0}{\\|\\bm{z}\\|}\\min\\{C_3,C_5\\}\\right)^{-1},\\left(\\frac{z_0}{\\|\\bm{z}\\|}\\min\\{C_3,C_5\\}\\right)^{-1/2}\\right\\} \\). Using \\eqref{eq:upbd-part-dist-uk-wk-case-2} together with \\eqref{eq:upbd-dist-uk-wk-two-parts}, we obtain that\n\t\t\t\\begin{equation}\n\t\t\t\\begin{aligned}\\label{eqn:3Ms}\n\t\t\t\t\\|\\bm{u}^k - \\bm{w}^k\\| & \\le M_1v_0^k\\left[\\sqrt{(v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|} + (v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|\\right]   \\\\\n\t\t\t\t%& = M_1\\left(\\sqrt{v_0^k} + \\sqrt{\\|\\bm{v}^k-\\bm{w}^k\\|}\\right)\\sqrt{\\|\\bm{v}^k - \\bm{w}^k\\|}\\\\\n\t\t\t\t&\\stackrel{\\rm (a)}{\\le}M_1\\sqrt{\\eta} \\sqrt{\\|\\bm{v}^k-\\bm{w}^k\\|} + M_1\\|\\bm{v}^k - \\bm{w}^k\\| \\\\\n\t\t\t\t&\\stackrel{\\rm (b)}{\\le} 3M_1\\sqrt{\\eta}\\sqrt{\\|\\bm{v}^k - \\bm{w}^k\\|},\n\t\t\t\\end{aligned}\n\t\t\t\\end{equation}\n\t\t\twhere (a) holds since \\( \\bm{v}^k\\in \\B(\\eta) \\) wherefore $v_0^k \\leq \\eta$, and (b) holds because \\(\\|\\bm{w}^k\\| \\leq \\|\\bm{v}^k\\| \\le \\eta \\) (because the projection onto $\\stdCone$ is nonexpansive and $\\bm 0\\in \\stdCone$), wherefore\n\t\t\t$$\n\t\t\t\\|\\bm{w}^k-\\bm{v}^k\\| = \\sqrt{\\|\\bm{w}^k-\\bm{v}^k\\|}\\sqrt{\\|\\bm{w}^k-\\bm{v}^k\\|} \\leq 2\\sqrt{\\eta}\\sqrt{\\|\\bm{w}^k-\\bm{v}^k\\|}.\n\t\t\t$$\n\t\t\tAltogether, \\eqref{eqn:3Ms} contradicts \\eqref{eq:forcontradiction-1st-face} and hence Case 2 cannot happen.\n\t\t\\end{enumerate}\n\t\tSummarizing the above discussions, we see that Case (I) cannot happen.\n\n\t\t(II):  By passing to a further subsequence, we may assume that \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}< 0 \\) for all \\( k \\). This together with the definition of \\( \\bm{f} \\) gives\n\t\t\\[\n\t\t\\frac{v_0^k}{z_0}[\\inProd{ -z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})}{ (v_0^k)^{-1}\\tbv^k}] < 0.\n\t\t\\]\n\t\tSince $\\frac{v_0^k}{z_0} > 0$, we deduce that $\\inProd{ -z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})}{ (v_0^k)^{-1}\\tbv^k} < 0$ for all $k$. Then it must hold that\n\t\t\\[\n\t\t\\lim_{k\\to \\infty}\\|(v_0^k)^{-1}\\bbv^k + z_0^{-1}\\bbz\\|+\\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\neq 0;\n\t\t\\]\notherwise, we have $(v_0^k)^{-1}\\bbv^k\\to -z_0^{-1}\\bbz$ and $(v_0^k)^{-1}\\tbv^k\\to \\bm{\\alpha}\\circ(z_0\\tbz^{-1})$, which further gives $\\inProd{ -z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})}{ (v_0^k)^{-1}\\tbv^k}\\to \\|z_0^{-1}\\bbz\\|^2+\\|\\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2 = \\|z_0\\bm f\\|^2 > 0$, a contradiction.\n\n\t\tConsequently, there exists $\\epsilon > 0$ such that for all sufficiently large $k$,\n\t\t\\begin{equation}\\label{eq:sum-out-of-neighbor}\n\t\t\t\\|(v_0^k)^{-1}\\bbv^k + z_0^{-1}\\bbz\\|+\\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| \\ge \\epsilon.\n\t\t\\end{equation}\n\t\tConsider the function \\( G:\\R^{m+n}\\to \\R\\cup\\{\\infty\\} \\) defined by\n\t\t\\[\n\t\tG(\\bm{\\xi},\\bm{\\omega}) := \\begin{cases}\n\t\t\t\\displaystyle \\frac{|\\inProd{ z_0^{-1}\\bbz}{ \\bm{\\xi}} + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}|}{\\sqrt{1+\\|\\bm{\\omega}\\|^2}} & {\\rm if}\\,(\\bm{\\xi},\\bm{\\omega})\\in \\Xi,\\,\\|\\bm{\\xi}\\|=1,\\,{\\rm and}\\,\\bm{\\omega}\\in \\Upsilon, \\\\\n\t\t\t\\infty                                                                                                                       & {\\rm otherwise},\n\t\t\\end{cases}\n\t\t\\]\n\t\twhere \\( \\Upsilon = \\{\\bm{\\omega}\\in \\R^n_+\\,| \\,\\prod_{i=1}^n \\omega_i^{\\alpha_i}=1\\} \\) and \\( \\Xi= \\{(\\bm{\\xi},\\bm{\\omega})\\,| \\,\\|\\bm{\\xi} + z_0^{-1}\\bbz\\|+\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| \\ge \\epsilon\\} \\).\n\t\tSince \\( \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\xi}} + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}} = 1+\\inProd{ z_0^{-1}\\bbz}{ \\bm{\\xi}} - \\inProd{ z_0^{-1}(-\\tbz)}{ \\bm{\\omega}}-1 \\), we see from \\eqref{eq:z0-v0-notation}, \\eqref{eq:sum-out-of-neighbor}, {Lemma \\ref{lemma:Lemma-4.1}} and Lemma~\\ref{lemma:inequ-inprod-power-cone} that \\( G \\) is never zero. Moreover, it is clearly lower semicontinuous on any compact set, and\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\!\\liminf_{\\|(\\bm{\\xi},\\bm{\\omega})\\|\\to \\infty}G(\\bm{\\xi},\\bm{\\omega}) \\!=\\!\\!\\! \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty,\\bm{\\omega}\\in \\Upsilon}\\!\\!\\frac{|\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}|}{\\sqrt{1+\\|\\bm{\\omega}\\|^2}}\\!\\overset{\\rm (a)}\\ge\\!\\!\\! \\inf_{\\|\\bm{\\lambda}\\|=1,\\bm{\\lambda}\\in \\R^{n}_+}\\!\\!|\\inProd{ z_0^{-1}\\tbz}{\\bm{\\lambda}}|\\!\\overset{\\rm (b)}\\ge\\!\\! \\min_i |z_0^{-1}\\tz_i| > 0,\n\t\t\\end{aligned}\n\t\t\\]\n\t\twhere (a) may be verified by multiplying numerator and denominator by $1/\\|\\bm{\\omega}\\|$ and (b) holds since \\( z_0^{-1}\\tz_i > 0 \\) for all \\( i \\).\n\t\tThus, \\( C_6:= \\inf G > 0 \\) and we have for all large \\( k \\),\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\frac{\\|\\bm{v}^k - \\bm{w}^k\\|}{\\|\\bm{u}^k - \\bm{w}^k\\|} & \\overset{\\rm (a)}= \\frac{\\|\\bm{v}^k - \\bm{w}^k\\|}{\\|\\bm{w}^k\\|} \\overset{\\rm (b)}\\ge \\frac{\\|\\bm{v}^k - \\bm{w}^k\\|}{\\|\\bm{v}^k\\|} \\\\\n&\\overset{\\rm (c)}=\n\t\t\t\\frac{z_0}{\\|\\bm{z}\\|}\\frac{|\\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ z_0^{-1}\\tbz}{ (v_0^k)^{-1}\\tbv^k}|v_0^k}{\\sqrt{(v^k_0)^2 + \\|\\tbv^k\\|^2}}                                                                                                            \\\\\n\t\t\t& = \\frac{z_0}{\\|\\bm{z}\\|}\\frac{|\\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ z_0^{-1}\\tbz}{ (v_0^k)^{-1}\\tbv^k}|}{\\sqrt{1 + \\|(v_0^k)^{-1}\\tbv^k\\|^2}}\\overset{\\rm (d)}\\ge \\frac{C_6 z_0}{\\|\\bm{z}\\|},\n\t\t\\end{aligned}\n\t\t\\]\n\t\twhere (a) follows from {Lemma \\ref{lemma:Lemma-2.11}}, which states that \\( \\bm{u}^k = \\bm{0} \\) in Case (II), (b) holds because the projection onto the cone is nonexpansive and $\\bm 0$ is in the cone, (c) follows from \\eqref{eq:dist-vk-wk} and (d) follows from \\eqref{eq:sum-out-of-neighbor}, \\eqref{eq:vk0-notation-nonzero} and the definitions of \\( G \\) and \\( C_6 \\). The above display contradicts \\eqref{eq:forcontradiction-1st-face}. Thus, Case (II) also cannot happen.\n\n\t\tSummarizing the above, we conclude that \\eqref{eq:forcontradiction-1st-face} cannot happen. Thus, in view of {\\cite[Lemma 3.12]{LiLoPo20}}, we must indeed have \\( \\gamma_{\\bm{z},\\eta} \\in (0,\\infty] \\) and that the desired error bound follows from {\\cite[Theorem 3.10]{LiLoPo20}}.\n\t\\end{proof}","start":39404,"end":57713}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":["eq:forcontradiction-1st-face"],"declared_external":false,"source_only":false,"title":"构造退化投影序列","source_units":[{"raw_tex":"\\begin{proof}\n\t\tSuppose for a contradiction that \\( \\gamma_{\\bm{z},\\eta} = 0 \\). Then, {in view of \\cite[Lemma 3.12]{LiLoPo20}}, there exist \\( \\widehat{\\bm{v}}\\in \\Fr \\) and a sequence \\( \\{\\bm{v}^k\\}\\subset \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fr \\) such that\n\t\t\\begin{equation}\\label{eq:forcontradiction-1st-face}\n\t\t\t\\lim_{k\\to \\infty}\\bm{v}^k = \\lim_{k\\to \\infty}\\bm{w}^k = \\widehat{\\bm{v}}\\,\\,{\\rm and}\\,\\,\\lim_{k\\to\\infty}\\frac{\\|\\bm{w}^k - \\bm{v}^k\\|^{\\frac{1}{2}}}{\\|\\bm{w}^k - \\bm{u}^k\\|} = 0,\n\t\t\\end{equation}\n\t\twhere \\( \\bm{w}^k = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}^k) \\), \\( \\bm{u}^k = P_{\\Fr}(\\bm{w}^k) \\) and \\( \\bm{u}^k\\neq \\bm{w}^k \\).\n","start":39404,"end":40044}]},{"id":"S029","semantic_type":"notation","kind":"notation","labels":["eq:z0-v0-notation"],"declared_external":false,"source_only":false,"title":"边界归一化尺度","source_units":[{"raw_tex":"\n\t\tDefine, for notational simplicity, \\( z_0 := \\|\\bbz\\| \\) and \\( v^k_0:=\\|\\bbv^k\\| \\).\n\t\tThen, since \\( \\{\\bm{v}^k\\}\\subset \\bd \\gK \\) and \\( \\bm{z} \\in \\bd\\dgK \\) with \\( \\bbz \\neq \\bm{0} \\), we have\\vspace{-0.15 cm}\n\t\t\\begin{equation}\\label{eq:z0-v0-notation}\\vspace{-0.15 cm}\n\t\t\tz_0 = \\|\\bbz\\| = \\prod_{i=1}^n\\left(\\frac{\\tz_i}{\\alpha_i}\\right)^{\\alpha_i}> 0 \\,\\,{\\rm and}\\,\\,v^k_0=\\|\\bbv^k\\| = \\prod_{i=1}^n(\\tv_i^k)^{\\alpha_i}\\,\\,\\forall k.\n\t\t\\end{equation}\n","start":40044,"end":40509}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"排除零首块子序列","source_units":[{"raw_tex":"\n\t\tIf it holds that \\( v_0^k = 0 \\) infinitely often, by passing to a further subsequence, we may assume that \\( v_0^k = 0 \\) for all \\( k \\). Then we have in view of {Lemma \\ref{lemma:Lemma-2.11}} that\\vspace{-0.15 cm}\n\t\t\\[\\vspace{-0.15 cm}\n\t\t\\!\\|\\bm{v}^k - \\bm{w}^k\\| \\!=\\! \\frac1{\\|\\bm{z}\\|}|\\inProd{ \\tbz}{\\tbv^k}|\\!\\overset{\\rm (a)}=\\! \\frac1{\\|\\bm{z}\\|}\\sum_{i=1}^n\\tz_i\\tv^k_i\\!\\ge\\! \\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|} \\|\\tbv^k\\|_1\\!\\ge\\! \\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|} \\|\\tbv^k\\|\n\t\t\\!\\overset{\\rm (b)}=\\!\\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|} \\|\\bm{v}^k\\|,\n\t\t\\]\n\t\twhere (a) holds because \\( \\tv^k_i \\ge 0 \\) and \\( \\tz_i > 0 \\) for all \\( i \\) (see \\eqref{eq:z0-v0-notation}), and (b) holds since \\( \\|\\bbv^k\\| = 0 \\). Since\n\t\t\\( \\|\\bm{w}^k - \\bm{u}^k\\| = \\dist(\\bm{w}^k,\\Fr) \\le \\|\\bm{w}^k\\|\\le \\|\\bm{v}^k\\| \\) as a consequence of the properties of projections, we conclude from this and the above display that\n\t\t\\( \\|\\bm{v}^k - \\bm{w}^k\\| \\ge \\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|}\\|\\bm{w}^k - \\bm{u}^k\\| \\), contradicting \\eqref{eq:forcontradiction-1st-face}.\n\n","start":40509,"end":41565}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":["eq:dist-vk-wk","eq:vk0-notation-nonzero"],"declared_external":false,"source_only":false,"title":"残差拆为两类间隙","source_units":[{"raw_tex":"\t\tThus, by considering a further subsequence if necessary, from now on, we assume\\vspace{-0.15 cm}\n\t\t\\begin{equation}\\label{eq:vk0-notation-nonzero}\\vspace{-0.15 cm}\n\t\t\tv^k_0=\\|\\bbv^k\\| = \\prod_{i=1}^n(\\tv_i^k)^{\\alpha_i} > 0 \\,\\,\\forall k.\n\t\t\\end{equation}\n\t\tUsing {Lemma \\ref{lemma:Lemma-2.11}}, we see that\\vspace{-0.15 cm}\n\t\t\\begin{equation}\\label{eq:dist-vk-wk}\\vspace{-0.15 cm}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\|\\bm{v}^k - \\bm{w}^k\\| & =\\frac1{\\|\\bm{z}\\|}|\\inProd{ \\bm{z}}{\\bm{v}^k}| = \\frac1{\\|\\bm{z}\\|}\\left|\\sum_{i=1}^{m}\\bz_i\\bv^k_i + \\sum_{i=1}^n\\tz_i\\tv^k_i\\right|             \\\\\n\t\t\t\t& = \\frac1{\\|\\bm{z}\\|}\\left|z_0v_0^k + \\sum_{i=1}^{m}\\bz_i\\bv^k_i - \\sum_{i=1}^n(-\\tz_i)\\tv^k_i - z_0v_0^k\\right|                                        \\\\\n\t\t\t\t& = \\frac{z_0v_0^k}{\\|\\bm{z}\\|}\\left| 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1\\right|  \\\\\n\t\t\t\t& = \\frac{z_0}{\\|\\bm{z}\\|}\\left( 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1\\right)v_0^k,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\twhere the last equality holds as \\( \\|z_0^{-1}\\bbz\\|= 1 \\), \\( \\|(v_0^k)^{-1}\\bbv^k\\|= 1 \\) and  \\( \\inProd{ z_0^{-1}\\tbz}{ (v_0^k)^{-1}\\tbv^k} \\ge 1 \\), thanks to \\eqref{eq:z0-v0-notation}, \\eqref{eq:vk0-notation-nonzero} and Lemma~\\ref{lemma:inequ-inprod-power-cone} applied with \\( \\bm{\\zeta} = -z_0^{-1}\\tbz \\).\n\n\t\tLet \\( \\bm{f} \\) be defined as in \\eqref{eq:1st-type-face-ray}. We consider two cases:\n\t\t\\begin{enumerate}[label={(\\Roman*)}]\n\t\t\t\\item \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}\\ge 0 \\) for all sufficiently large \\( k \\).\n\t\t\t\\item \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}< 0 \\) infinitely often.\n\t\t\\end{enumerate}\n","start":41565,"end":43260}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":["eq:upbd-dist-uk-wk-two-parts"],"declared_external":false,"source_only":false,"title":"非负内积时的投影上界","source_units":[{"raw_tex":"\n\t\t(I): By passing to a further subsequence, we may assume that \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}\\ge 0 \\) for all \\( k \\). In this case, if we define\n\t\t\\[\n\t\t\\bm{Q} = \\bm{I}_{m + n} - \\frac{\\bm{z}\\bm{z}^{\\top}}{\\|\\bm{z}\\|^2} - \\frac{\\bm{f}\\bm{f}^{\\top}}{\\|\\bm{f}\\|^2},\n\t\t\\]\n\t\twhere \\( \\bm{f} \\) is as in \\eqref{eq:1st-type-face-ray}, then we see from {Lemma \\ref{lemma:Lemma-2.11}} and \\eqref{eq:z0-v0-notation} that\n\t\t\\begin{equation}\\label{eq:upbd-dist-uk-wk-two-parts}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\|\\bm{u}^k - \\bm{w}^k\\| & = \\|\\bm{Q}\\bm{v}^k\\| = v_0^k\\left\\|\\bm{Q}\\begin{bmatrix}\n\t\t\t\t\t(v_0^k)^{-1}\\bbv^k \\\\ (v_0^k)^{-1}\\tbv^k\n\t\t\t\t\\end{bmatrix}\\right\\| \\\\\n&\\overset{\\rm (a)}= v_0^k{\\Bigg\\|}\\bm{Q}\\begin{bmatrix}\n\t\t\t\t\t(v_0^k)^{-1}\\bbv^k \\\\ (v_0^k)^{-1}\\tbv^k\n\t\t\t\t\\end{bmatrix} \\!-\\! \\bm{Q}\\underbrace{\\begin{bmatrix}\n\t\t\t\t\t\t-z_0^{-1}\\bbz \\\\ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\n\t\t\t\t\\end{bmatrix}}_{z_0\\bm{f}}{\\Bigg\\|} \\\\\n\t\t\t\t& \\le v_0^k\\left[\\|(v_0^k)^{-1}\\bbv^k + z_0^{-1}\\bbz\\| + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\right],\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\twhere (a) holds because %\\( z_0\\bm{f} = \\begin{bmatrix}\n\t\t%\t\t-z_0^{-1}\\bbz^{\\top} & \\alpha\\circ(z_0\\tbz^{-1})^{\\top}\n\t\t%\t\\end{bmatrix}^{\\top}\\)  and\n\t\t\\( \\bm{Q}\\bm{f} = \\bm{0} \\) (an identity which is clear from the definitions).\n","start":43260,"end":44557}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":["eq:keylemmarel"],"declared_external":false,"source_only":false,"title":"球面间隙的全局二次界","source_units":[{"raw_tex":"\n\t\tNext, in view of \\eqref{eq:z0-v0-notation}, we can apply {Lemma \\ref{lemma:Lemma-4.1}} to obtain \\( C_1 > 0 \\) and \\( \\epsilon_1>0 \\) so that {\\eqref{eq:inequ-p=q=2}} holds with \\( \\overline{\\bm{\\zeta}} = -z_0^{-1}\\bbz \\), i.e.,\n\t\t\\[\n\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_1\\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|^2\n\t\t\\]\n\t\twhenever \\( \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\le \\epsilon_1 \\) and \\( \\|\\bm{\\omega}\\|=1 \\). On the other hand, in view of the positivity of $1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}$ when $\\|\\bm{\\omega}\\|=1$ and $\\bm{\\omega} \\neq -z_0^{-1}\\bbz$ {(see Lemma \\ref{lemma:Lemma-4.1})}, we know that\n\\[\nC_2 := \\inf_{\\|\\bm{\\omega}\\|=1}\\{1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}} \\,|\\, \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\ge \\epsilon_1 \\} > 0.\n\\]\nThis together with the fact $\\|z_0^{-1}\\bbz\\|=1$ (see \\eqref{eq:z0-v0-notation}) implies that\n\t\t\\[\n\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_2 \\ge 0.25 C_2 \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|^2,\n\t\t\\]\n\t\twhenever \\( \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\ge \\epsilon_1 \\) and \\( \\|\\bm{\\omega}\\|=1 \\). We thus have (with $C_3 \\!:=\\! \\min\\{C_1,C_2/4\\}$)\n\\begin{equation}\\label{eq:keylemmarel}\n\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_3\\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|^2\\ \\ \\ {\\rm whenever}\\ \\ \\|\\bm{\\omega}\\|=1.\n\t\t\\end{equation}\n","start":44557,"end":45846}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":["eq:keylemmare2"],"declared_external":false,"source_only":false,"title":"乘积间隙的局部二次界","source_units":[{"raw_tex":"\n\t\tIn addition, noting \\eqref{eq:z0-v0-notation} again, we can apply Lemma~\\ref{lemma:inequ-inprod-power-cone} with \\( \\bm{\\zeta} = -z_0^{-1}\\tbz\\in {\\rm int}\\,\\R^n_- \\) to obtain \\( C_4 > 0 \\) and \\( \\epsilon > 0 \\) so that \\eqref{eq:inequ-inprod-power-cone} holds with \\( \\widetilde{\\bm{\\zeta}} = \\bm{\\alpha}\\circ (z_0\\tbz^{-1}) \\), i.e.,\n\t\t\\begin{equation}\\label{eq:keylemmare2}\n\t\t\t-1 + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}\\ge C_4\\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2\n\t\t\\end{equation}\n\t\twhenever \\( \\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\le \\epsilon \\), \\( \\bm{\\omega}\\in {\\rm int}\\,\\R^n_+ \\) and \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i}=1 \\).\n","start":45846,"end":46513}]},{"id":"S035","semantic_type":"interpretation","kind":"interpretation","labels":["eqn:3Ms"],"declared_external":false,"source_only":false,"title":"保留的非活动草稿","source_units":[{"raw_tex":"\n%Then we have in view of the positivity of $1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}$ when $\\bm{\\omega} \\neq -z_0^{-1}\\bbz$ (see Lemma \\ref{lemma:Lemma-4.1}) and the compactness of the unit sphere that there exists \\( C_3>0 \\) so that\n%\t\t\\begin{equation}\\label{eq:keylemmare3}\n%\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_3 \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\n%\t\t\\end{equation}\n%\t\twhenever \\( \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\ge \\epsilon \\) and \\( \\|\\bm{\\omega}\\|=1 \\).\n\n","start":46513,"end":46985},{"raw_tex":"\t\t\t\t%& = M_1\\left(\\sqrt{v_0^k} + \\sqrt{\\|\\bm{v}^k-\\bm{w}^k\\|}\\right)\\sqrt{\\|\\bm{v}^k - \\bm{w}^k\\|}\\\\\n","start":52216,"end":52317}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":["eq:def-h","eq:keylemmare4"],"declared_external":false,"source_only":false,"title":"乘积间隙的远区线性界","source_units":[{"raw_tex":"Furthermore, consider \\( h:\\R^n\\to \\R\\cup\\{\\infty\\} \\) defined by\n\t\t\\begin{equation}\\label{eq:def-h}\n\t\t\th(\\bm{\\omega}) = \\begin{cases}\n\t\t\t\t\\displaystyle\\frac{\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}-1}{\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|} & {\\rm if}\\,\\|\\bm{\\omega}-\\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\ge \\epsilon,\\,\\bm{\\omega}\\in \\Upsilon, \\\\\n\t\t\t\t\\infty                                                                                                    & {\\rm otherwise},\n\t\t\t\\end{cases}\n\t\t\\end{equation}\n\t\twhere \\( \\Upsilon= \\{\\bm{\\omega}\\in \\R^n_+\\,| \\,\\prod_{i=1}^n \\omega_i^{\\alpha_i}=1\\} \\). Then we have\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t& \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty}h(\\bm{\\omega}) = \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty, \\bm{\\omega}\\in \\Upsilon}\\frac{\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}-1}{\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|} \\\\\n\t\t\t& \\ge \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty,\\bm{\\omega}\\in \\R^n_+}\\frac{\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}-1}{\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|}\n\t\t\t\\overset{\\rm (a)}{\\ge} \\inf_{\\|\\bm{\\lambda}\\|=1,\\bm{\\lambda}\\in \\R^n_+}\\inProd{ z_0^{-1}\\tbz}{\\bm{\\lambda}} \\overset{\\rm (b)}\\ge \\min_{1\\le i\\le n}z_0^{-1}\\tz_i > 0.\n\t\t\\end{aligned}\n\t\t\\]\n\t\tHere (a) may be verified by multiplying both numerator and denominator of the left side by $1/\\|\\bm{\\omega}\\|$; (b) holds because \\( z_0^{-1}\\tz_i > 0 \\) for all \\( i \\) (see \\eqref{eq:z0-v0-notation}). Since \\( h \\) in \\eqref{eq:def-h} is also lower semicontinuous on any compact set and {\\bf is always positive},\\footnote{The positivity can be seen by applying Lemma~\\ref{lemma:inequ-inprod-power-cone} with $\\bm\\zeta = -z_0^{-1}\\tbz$.} it must then hold that \\( C_5:= \\inf h > 0 \\). In particular, this means that\n\t\t\\begin{equation}\\label{eq:keylemmare4}\n\t\t\t-1 + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}\\ge C_5\\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\n\t\t\\end{equation}\n\t\twhenever \\( \\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\ge \\epsilon \\), \\( \\bm{\\omega}\\in {\\rm int}\\,\\R^n_+ \\) and \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i}=1 \\).\n","start":46985,"end":49034}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"近区平方根比值矛盾","source_units":[{"raw_tex":"\n\t\tBy passing to suitable subsequences, we will end up with one of the following two cases:\n\t\t\\begin{enumerate}[label={Case \\arabic*:}]\n\t\t\t\\item \\( \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| \\le \\epsilon \\) for all \\( k \\). Then we have from \\eqref{eq:keylemmarel} (with $\\bm{\\omega}=(v_0^k)^{-1}\\bbv^k$) and \\eqref{eq:keylemmare2} (with $\\bm{\\omega}=(v_0^k)^{-1}\\tbv^k$) that for these \\( k \\)\n\t\t\t\\begin{equation*}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t& 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1          \\\\\n\t\t\t\t\t& \\ge \\min\\{C_3,C_4\\}(\\|(v_0^k)^{-1}\\bbv^k  + z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2).\n\t\t\t\t\\end{aligned}\n\t\t\t\\end{equation*}\n\t\t\tCombining this with \\eqref{eq:dist-vk-wk} and \\eqref{eq:upbd-dist-uk-wk-two-parts}, we see further that\n\t\t\t\\[\n\t\t\t\\begin{aligned}\n\t\t\t\t&\\|\\bm{v}^k - \\bm{w}^k\\|\\\\ & \\ge \\frac{z_0}{\\|\\bm{z}\\|}\\min\\{C_3,C_4\\}(\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2) v_0^k    \\\\\n\t\t\t\t& \\ge \\frac{z_0}{2\\|\\bm{z}\\|}\\min\\{C_3,C_4\\}(\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\| + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|)^2 v_0^k     \\\\\n\t\t\t\t& \\ge \\frac{z_0\\min\\{C_3,C_4\\}}{2\\|\\bm{z}\\|v_0^k}\\|\\bm{u}^k - \\bm{w}^k\\|^2\\ge \\frac{z_0\\min\\{C_3,C_4\\}}{2\\|\\bm{z}\\|\\eta}\\|\\bm{u}^k - \\bm{w}^k\\|^2,\n\t\t\t\\end{aligned}\n\t\t\t\\]\n\t\t\twhere the last inequality holds because \\( \\bm{v}^k\\in \\B(\\eta) \\). The above display contradicts \\eqref{eq:forcontradiction-1st-face} and hence Case 1 cannot happen.\n","start":49034,"end":50599}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":["eq:upbd-part-dist-uk-wk-case-2","eqn:3Ms"],"declared_external":false,"source_only":false,"title":"远区平方根比值矛盾","source_units":[{"raw_tex":"\n\t\t\t\\item \\( \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\ge \\epsilon \\) for all \\( k \\). Then we have from \\eqref{eq:keylemmarel} and \\eqref{eq:keylemmare4} that for these \\( k \\)\n\t\t\t\\begin{equation*}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t& 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1        \\\\\n\t\t\t\t\t& \\ge \\min\\{C_3,C_5\\}(\\|(v_0^k)^{-1}\\bbv^k  + z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|).\n\t\t\t\t\\end{aligned}\n\t\t\t\\end{equation*}\n\t\t\tUsing this together with \\eqref{eq:dist-vk-wk}, we deduce that for all large \\( k \\),\n\t\t\t\\[\n\t\t\t\\!\\|\\bm{v}^k - \\bm{w}^k\\| \\!\\ge\\! \\frac{z_0\\min\\{C_3,C_5\\}}{\\|\\bm{z}\\|}(\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|) v_0^k.\n\t\t\t\\]\n\t\t\tThis implies that\n\t\t\t\\begin{equation}\\label{eq:upbd-part-dist-uk-wk-case-2}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\|            & \\le M_1\\sqrt{(v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|}, \\\\\n\t\t\t\t\t\\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| & \\le M_1(v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|,\n\t\t\t\t\\end{aligned}\n\t\t\t\\end{equation}\n\t\t\twhere \\( M_1 := \\max\\left\\{\\left(\\frac{z_0}{\\|\\bm{z}\\|}\\min\\{C_3,C_5\\}\\right)^{-1},\\left(\\frac{z_0}{\\|\\bm{z}\\|}\\min\\{C_3,C_5\\}\\right)^{-1/2}\\right\\} \\). Using \\eqref{eq:upbd-part-dist-uk-wk-case-2} together with \\eqref{eq:upbd-dist-uk-wk-two-parts}, we obtain that\n\t\t\t\\begin{equation}\n\t\t\t\\begin{aligned}\\label{eqn:3Ms}\n\t\t\t\t\\|\\bm{u}^k - \\bm{w}^k\\| & \\le M_1v_0^k\\left[\\sqrt{(v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|} + (v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|\\right]   \\\\\n\t\t\t\t%& = M_1\\left(\\sqrt{v_0^k} + \\sqrt{\\|\\bm{v}^k-\\bm{w}^k\\|}\\right)\\sqrt{\\|\\bm{v}^k - \\bm{w}^k\\|}\\\\\n\t\t\t\t&\\stackrel{\\rm (a)}{\\le}M_1\\sqrt{\\eta} \\sqrt{\\|\\bm{v}^k-\\bm{w}^k\\|} + M_1\\|\\bm{v}^k - \\bm{w}^k\\| \\\\\n\t\t\t\t&\\stackrel{\\rm (b)}{\\le} 3M_1\\sqrt{\\eta}\\sqrt{\\|\\bm{v}^k - \\bm{w}^k\\|},\n\t\t\t\\end{aligned}\n\t\t\t\\end{equation}\n\t\t\twhere (a) holds since \\( \\bm{v}^k\\in \\B(\\eta) \\) wherefore $v_0^k \\leq \\eta$, and (b) holds because \\(\\|\\bm{w}^k\\| \\leq \\|\\bm{v}^k\\| \\le \\eta \\) (because the projection onto $\\stdCone$ is nonexpansive and $\\bm 0\\in \\stdCone$), wherefore\n\t\t\t$$\n\t\t\t\\|\\bm{w}^k-\\bm{v}^k\\| = \\sqrt{\\|\\bm{w}^k-\\bm{v}^k\\|}\\sqrt{\\|\\bm{w}^k-\\bm{v}^k\\|} \\leq 2\\sqrt{\\eta}\\sqrt{\\|\\bm{w}^k-\\bm{v}^k\\|}.\n\t\t\t$$\n\t\t\tAltogether, \\eqref{eqn:3Ms} contradicts \\eqref{eq:forcontradiction-1st-face} and hence Case 2 cannot happen.\n\t\t\\end{enumerate}\n\t\tSummarizing the above discussions, we see that Case (I) cannot happen.\n","start":50599,"end":53118}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sum-out-of-neighbor"],"declared_external":false,"source_only":false,"title":"负内积迫使离开接触点","source_units":[{"raw_tex":"\n\t\t(II):  By passing to a further subsequence, we may assume that \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}< 0 \\) for all \\( k \\). This together with the definition of \\( \\bm{f} \\) gives\n\t\t\\[\n\t\t\\frac{v_0^k}{z_0}[\\inProd{ -z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})}{ (v_0^k)^{-1}\\tbv^k}] < 0.\n\t\t\\]\n\t\tSince $\\frac{v_0^k}{z_0} > 0$, we deduce that $\\inProd{ -z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})}{ (v_0^k)^{-1}\\tbv^k} < 0$ for all $k$. Then it must hold that\n\t\t\\[\n\t\t\\lim_{k\\to \\infty}\\|(v_0^k)^{-1}\\bbv^k + z_0^{-1}\\bbz\\|+\\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\neq 0;\n\t\t\\]\notherwise, we have $(v_0^k)^{-1}\\bbv^k\\to -z_0^{-1}\\bbz$ and $(v_0^k)^{-1}\\tbv^k\\to \\bm{\\alpha}\\circ(z_0\\tbz^{-1})$, which further gives $\\inProd{ -z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})}{ (v_0^k)^{-1}\\tbv^k}\\to \\|z_0^{-1}\\bbz\\|^2+\\|\\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2 = \\|z_0\\bm f\\|^2 > 0$, a contradiction.\n\n\t\tConsequently, there exists $\\epsilon > 0$ such that for all sufficiently large $k$,\n\t\t\\begin{equation}\\label{eq:sum-out-of-neighbor}\n\t\t\t\\|(v_0^k)^{-1}\\bbv^k + z_0^{-1}\\bbz\\|+\\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| \\ge \\epsilon.\n\t\t\\end{equation}\n","start":53118,"end":54379}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"远离接触点的归一化正下界","source_units":[{"raw_tex":"\t\tConsider the function \\( G:\\R^{m+n}\\to \\R\\cup\\{\\infty\\} \\) defined by\n\t\t\\[\n\t\tG(\\bm{\\xi},\\bm{\\omega}) := \\begin{cases}\n\t\t\t\\displaystyle \\frac{|\\inProd{ z_0^{-1}\\bbz}{ \\bm{\\xi}} + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}|}{\\sqrt{1+\\|\\bm{\\omega}\\|^2}} & {\\rm if}\\,(\\bm{\\xi},\\bm{\\omega})\\in \\Xi,\\,\\|\\bm{\\xi}\\|=1,\\,{\\rm and}\\,\\bm{\\omega}\\in \\Upsilon, \\\\\n\t\t\t\\infty                                                                                                                       & {\\rm otherwise},\n\t\t\\end{cases}\n\t\t\\]\n\t\twhere \\( \\Upsilon = \\{\\bm{\\omega}\\in \\R^n_+\\,| \\,\\prod_{i=1}^n \\omega_i^{\\alpha_i}=1\\} \\) and \\( \\Xi= \\{(\\bm{\\xi},\\bm{\\omega})\\,| \\,\\|\\bm{\\xi} + z_0^{-1}\\bbz\\|+\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| \\ge \\epsilon\\} \\).\n\t\tSince \\( \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\xi}} + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}} = 1+\\inProd{ z_0^{-1}\\bbz}{ \\bm{\\xi}} - \\inProd{ z_0^{-1}(-\\tbz)}{ \\bm{\\omega}}-1 \\), we see from \\eqref{eq:z0-v0-notation}, \\eqref{eq:sum-out-of-neighbor}, {Lemma \\ref{lemma:Lemma-4.1}} and Lemma~\\ref{lemma:inequ-inprod-power-cone} that \\( G \\) is never zero. Moreover, it is clearly lower semicontinuous on any compact set, and\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\!\\liminf_{\\|(\\bm{\\xi},\\bm{\\omega})\\|\\to \\infty}G(\\bm{\\xi},\\bm{\\omega}) \\!=\\!\\!\\! \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty,\\bm{\\omega}\\in \\Upsilon}\\!\\!\\frac{|\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}|}{\\sqrt{1+\\|\\bm{\\omega}\\|^2}}\\!\\overset{\\rm (a)}\\ge\\!\\!\\! \\inf_{\\|\\bm{\\lambda}\\|=1,\\bm{\\lambda}\\in \\R^{n}_+}\\!\\!|\\inProd{ z_0^{-1}\\tbz}{\\bm{\\lambda}}|\\!\\overset{\\rm (b)}\\ge\\!\\! \\min_i |z_0^{-1}\\tz_i| > 0,\n\t\t\\end{aligned}\n\t\t\\]\n\t\twhere (a) may be verified by multiplying numerator and denominator by $1/\\|\\bm{\\omega}\\|$ and (b) holds since \\( z_0^{-1}\\tz_i > 0 \\) for all \\( i \\).\n","start":54379,"end":56130}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"负内积分支的比值矛盾","source_units":[{"raw_tex":"\t\tThus, \\( C_6:= \\inf G > 0 \\) and we have for all large \\( k \\),\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\frac{\\|\\bm{v}^k - \\bm{w}^k\\|}{\\|\\bm{u}^k - \\bm{w}^k\\|} & \\overset{\\rm (a)}= \\frac{\\|\\bm{v}^k - \\bm{w}^k\\|}{\\|\\bm{w}^k\\|} \\overset{\\rm (b)}\\ge \\frac{\\|\\bm{v}^k - \\bm{w}^k\\|}{\\|\\bm{v}^k\\|} \\\\\n&\\overset{\\rm (c)}=\n\t\t\t\\frac{z_0}{\\|\\bm{z}\\|}\\frac{|\\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ z_0^{-1}\\tbz}{ (v_0^k)^{-1}\\tbv^k}|v_0^k}{\\sqrt{(v^k_0)^2 + \\|\\tbv^k\\|^2}}                                                                                                            \\\\\n\t\t\t& = \\frac{z_0}{\\|\\bm{z}\\|}\\frac{|\\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ z_0^{-1}\\tbz}{ (v_0^k)^{-1}\\tbv^k}|}{\\sqrt{1 + \\|(v_0^k)^{-1}\\tbv^k\\|^2}}\\overset{\\rm (d)}\\ge \\frac{C_6 z_0}{\\|\\bm{z}\\|},\n\t\t\\end{aligned}\n\t\t\\]\n\t\twhere (a) follows from {Lemma \\ref{lemma:Lemma-2.11}}, which states that \\( \\bm{u}^k = \\bm{0} \\) in Case (II), (b) holds because the projection onto the cone is nonexpansive and $\\bm 0$ is in the cone, (c) follows from \\eqref{eq:dist-vk-wk} and (d) follows from \\eqref{eq:sum-out-of-neighbor}, \\eqref{eq:vk0-notation-nonzero} and the definitions of \\( G \\) and \\( C_6 \\). The above display contradicts \\eqref{eq:forcontradiction-1st-face}. Thus, Case (II) also cannot happen.\n","start":56130,"end":57417}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"正面常数转为误差界","source_units":[{"raw_tex":"\n\t\tSummarizing the above, we conclude that \\eqref{eq:forcontradiction-1st-face} cannot happen. Thus, in view of {\\cite[Lemma 3.12]{LiLoPo20}}, we must indeed have \\( \\gamma_{\\bm{z},\\eta} \\in (0,\\infty] \\) and that the desired error bound follows from {\\cite[Theorem 3.10]{LiLoPo20}}.\n\t\\end{proof}","start":57417,"end":57713}]},{"id":"S043","semantic_type":"lemma","kind":"lemma","labels":["remark:F-r-order-1/2"],"declared_external":false,"source_only":false,"title":"射线指数的渐近最优性","source_units":[{"raw_tex":"\\begin{remark}[Optimality of the error bound in Theorem~\\ref{thm:Holderian-1/2-1st-type-face}]\\label{remark:F-r-order-1/2}\n\t\t\\!\\!\\!Let \\( \\bm{z}\\!\\in\\! \\!\\bd\\dgK\\backslash\\!\\{\\bm{0}\\} \\) with \\( \\bbz \\neq \\bm{0} \\) and let \\( \\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Then necessarily \\( \\tz_i > 0 \\) for all \\( i \\). Moreover, we also know from the definition that \\( \\alpha_i > 0 \\) for all \\( i \\). Now, consider the continuous function \\( \\bm{q}:(0,\\alpha_1)\\to \\{\\bm{z}\\}^\\perp \\) defined by $\\epsilon \\mapsto \\bm{q}_\\epsilon:=(\\bar{\\bm{q}}_\\epsilon,\\widetilde{\\bm{q}}_\\epsilon)$ where\n","start":57718,"end":58303},{"raw_tex":"\t\tThus \\( | \\cdot | ^{\\frac{1}{2}} \\) satisfies the asymptotic optimality criterion {(cf. \\cite[Definition 3.1]{LiLoPo21}}) for \\( \\gK \\) and \\( \\bm{z} \\){, which implies that the error bound is optimal in the sense of \\cite[Theorem 3.2(b)]{LiLoPo21}.}\n\t\\end{remark}","start":63744,"end":64010}]},{"id":"S044","semantic_type":"proof","kind":"proof","labels":["eq:distpower","eq:lowerlowerboundpower","remark:F-r-order-1/2"],"declared_external":false,"source_only":false,"title":"射线最优性完整构造论证","source_units":[{"raw_tex":"\t\t\\!\\!\\!Let \\( \\bm{z}\\!\\in\\! \\!\\bd\\dgK\\backslash\\!\\{\\bm{0}\\} \\) with \\( \\bbz \\neq \\bm{0} \\) and let \\( \\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Then necessarily \\( \\tz_i > 0 \\) for all \\( i \\). Moreover, we also know from the definition that \\( \\alpha_i > 0 \\) for all \\( i \\). Now, consider the continuous function \\( \\bm{q}:(0,\\alpha_1)\\to \\{\\bm{z}\\}^\\perp \\) defined by $\\epsilon \\mapsto \\bm{q}_\\epsilon:=(\\bar{\\bm{q}}_\\epsilon,\\widetilde{\\bm{q}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{q}}_\\epsilon = -\\bbz/\\|\\bbz\\|^2,\\,\\,(\\widetilde{\\bm{q}}_\\epsilon)_1 = (\\alpha_1 - \\epsilon)\\tz_1^{-1},\\,\\,(\\widetilde{\\bm{q}}_\\epsilon)_2 = (\\alpha_2 + \\epsilon)\\tz_2^{-1},\\,\\mbox{and} \\,(\\widetilde{\\bm{q}}_\\epsilon)_i = \\alpha_i \\tz_i^{-1}, \\,\\forall i \\ge 3.\n\t\t\\]\n\t\tNotice that \\( \\bm{q}_\\epsilon \\) only differs from the \\( \\bm{f} \\) in \\eqref{eq:1st-type-face-ray} in two entries. One can check that \\( \\inProd{ \\bm{z}}{\\bm{q}_\\epsilon} = 0 \\) and \\( \\bm{q}_\\epsilon\\to \\bm{f}\\in \\Fr\\backslash\\{\\bm{0}\\} \\) as $\\epsilon \\downarrow 0$.\n\t\tMoreover, we have\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t&\\prod_{i=1}^n (\\widetilde{\\bm{q}}_\\epsilon)_i^{\\alpha_i}  = (\\alpha_1 - \\epsilon)^{\\alpha_1}(\\alpha_2 + \\epsilon)^{\\alpha_2}\\tz_1^{-\\alpha_1}\\tz_2^{-\\alpha_2}\\prod_{i=3}^n\\left(\\frac{\\alpha_i}{\\tz_i}\\right)^{\\alpha_i}          \\\\\n\t\t\t& =  \\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2}\\prod_{i=1}^n\\left(\\frac{\\alpha_i}{\\tz_i}\\right)^{\\alpha_i}\n\t\t\t\\overset{\\rm (a)}= \\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2} \\|\\bbz\\|^{-1}                                                                                          \\\\\n\t\t\t& = \\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2}\\|\\bar{\\bm{q}}_\\epsilon\\|,\n\t\t\\end{aligned}\n\t\t\\]\n\t\twhere (a) holds because \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz\\neq \\bm{0} \\). In view of this, if we define a continuous function \\( \\bm{p}:(0,\\alpha_1)\\to \\gK \\)  by $\\epsilon \\mapsto \\bm{p}_\\epsilon := (\\bar{\\bm{p}}_\\epsilon,\\widetilde{\\bm{p}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{p}}_\\epsilon := -\\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2}\\frac{\\bbz}{\\|\\bbz\\|^2}\\quad{\\rm and}\\quad \\widetilde{\\bm{p}}_\\epsilon := \\widetilde{\\bm{q}}_\\epsilon;\n\t\t\\]\n\t\tthen it is clear that \\( \\bm{p}_\\epsilon \\in \\gK \\), and we can compute that\n\t\t\\begin{equation}\\label{eq:distpower}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\dist(\\bm{q}_\\epsilon,\\gK) & \\le \\|\\bm{q}_\\epsilon - \\bm{p}_\\epsilon\\| = \\frac1{\\|\\bbz\\|}\\left|\\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2} - 1\\right| \\\\\n\t\t\t\t& = \\frac1{\\|\\bbz\\|}\\left|(1 - \\epsilon + O(\\epsilon^2))(1 + \\epsilon + O(\\epsilon^2)) - 1\\right| = O(\\epsilon^2).\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\tNext, we estimate \\( \\dist(\\bm{q}_\\epsilon,\\Fr) \\).\n\t\tNotice that \\( \\inProd{ \\bm{q}_\\epsilon}{\\bm{f}} > 0 \\) for all sufficiently small \\( \\epsilon \\) because \\( \\bm{q}_\\epsilon \\to \\bm{f} \\). Hence, using the definition of \\( \\Fr \\) and {Lemma \\ref{lemma:Lemma-2.11}}, we see that\n\t\t\\[\n\t\t\\dist(\\bm{q}_\\epsilon,\\Fr)^2 = \\left\\|\\bm{q}_\\epsilon - \\frac{\\inProd{ \\bm{q}_\\epsilon}{ \\bm{f}}}{\\|\\bm{f}\\|^2}\\bm{f}\\right\\|^2 = \\|\\bm{q}_\\epsilon\\|^2 - \\frac{(\\inProd{ \\bm{q}_\\epsilon}{ \\bm{f}})^2}{\\|\\bm{f}\\|^2}.\n\t\t\\]\n\t\tA direct computation then shows that\n\t\t\\begin{align}\n\t\t\t\\|\\bm{q}_\\epsilon\\|^2 & = \\frac1{\\|\\bbz\\|^2} + (\\alpha_1-\\epsilon)^2\\tz_1^{-2} + (\\alpha_2+\\epsilon)^2\\tz_2^{-2} + \\sum_{i=3}^n \\alpha^2_i \\tz_i^{-2}                       \\notag \\\\\n\t\t\t& = \\frac1{\\|\\bbz\\|^2} + \\sum_{i=1}^n \\alpha^2_i \\tz_i^{-2} + 2\\epsilon(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2}) + \\epsilon^2(\\tz_1^{-2} + \\tz_2^{-2}) \\notag\\\\\n\t\t\t& = \\|\\bm{f}\\|^2 + 2\\epsilon(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2}) + \\epsilon^2(\\tz_1^{-2} + \\tz_2^{-2}),\\notag\n\t\t\\end{align}\n\t\twhere the last equality follows from the definition of \\( \\bm{f} \\) in \\eqref{eq:1st-type-face-ray}.\n\t\tFurthermore,\n\t\t\\begin{align}\n(\\inProd{ \\bm{q}_\\epsilon}{ \\bm{f}})^2 & =\\textstyle \\left(\\frac1{\\|\\bbz\\|^2} + \\alpha_1(\\alpha_1-\\epsilon)\\tz_1^{-2} + \\alpha_2(\\alpha_2+\\epsilon)\\tz_2^{-2} + \\sum_{i=3}^n \\alpha^2_i \\tz_i^{-2}\\right)^2 \\notag\\\\\n\t\t\t& = \\left[ \\|\\bm{f}\\|^2  + \\epsilon (\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})\\right]^2                                                                    \\notag\\\\\n\t\t\t& = \\|\\bm{f}\\|^4 + 2\\epsilon\\|\\bm{f}\\|^2 (\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2}) + \\epsilon^2(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2.\\notag\n\t\t\\end{align}\n\t\tCombining the above three identities, we deduce further that\n\t\t\\begin{align}\\label{eq:lowerlowerboundpower}\n\t\t\t\\dist(\\bm{q}_\\epsilon,\\Fr)^2 & = \\epsilon^2\\left(\\tz_1^{-2} + \\tz_2^{-2} - \\frac{(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2}{\\|\\bm{f}\\|^2}\\right)\\nonumber                          \\\\\n\t\t\t& \\ge \\epsilon^2\\left(\\tz_1^{-2} + \\tz_2^{-2} - \\frac{(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2}{\\alpha^2_2\\tz_2^{-2} + \\alpha^2_1\\tz_1^{-2}}\\right),\n\t\t\\end{align}\n\t\twhere the inequality follows from the definition of \\( \\bm{f} \\). Now, notice that in \\eqref{eq:lowerlowerboundpower}, the scalar term is strictly greater than zero, because\n\t\t\\[\n\t\t(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2 < (\\alpha_2\\tz_2^{-2} + \\alpha_1\\tz_1^{-2})^2 \\le (\\tz_1^{-2} + \\tz_2^{-2})(\\alpha^2_2\\tz_2^{-2} + \\alpha^2_1\\tz_1^{-2}),\n\t\t\\]\n\t\twhere the strict inequality holds because \\( \\alpha_i\\tz_i^{-2} > 0 \\) for \\( i = 1 \\), \\( 2 \\), and the last inequality follows from the Cauchy-Schwarz inequality.\n\t\tThis together with \\eqref{eq:lowerlowerboundpower} shows that \\( \\dist(\\bm{q}_\\epsilon,\\Fr) = \\Omega(\\epsilon) \\). Combining this with \\eqref{eq:distpower}, we obtain\n\t\t$\n\t\t\\limsup_{\\epsilon\\downarrow 0}\\frac{\\dist(\\bm{q}_\\epsilon,\\gK)^\\frac12}{\\dist(\\bm{q}_\\epsilon,\\Fr)} < \\infty\n\t\t$.\n","start":57841,"end":63744}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["remark:F-r-order-1/2"],"declared_external":false,"source_only":false,"title":"保持超平面的两坐标扰动","source_units":[{"raw_tex":"\t\t\\!\\!\\!Let \\( \\bm{z}\\!\\in\\! \\!\\bd\\dgK\\backslash\\!\\{\\bm{0}\\} \\) with \\( \\bbz \\neq \\bm{0} \\) and let \\( \\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Then necessarily \\( \\tz_i > 0 \\) for all \\( i \\). Moreover, we also know from the definition that \\( \\alpha_i > 0 \\) for all \\( i \\). Now, consider the continuous function \\( \\bm{q}:(0,\\alpha_1)\\to \\{\\bm{z}\\}^\\perp \\) defined by $\\epsilon \\mapsto \\bm{q}_\\epsilon:=(\\bar{\\bm{q}}_\\epsilon,\\widetilde{\\bm{q}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{q}}_\\epsilon = -\\bbz/\\|\\bbz\\|^2,\\,\\,(\\widetilde{\\bm{q}}_\\epsilon)_1 = (\\alpha_1 - \\epsilon)\\tz_1^{-1},\\,\\,(\\widetilde{\\bm{q}}_\\epsilon)_2 = (\\alpha_2 + \\epsilon)\\tz_2^{-1},\\,\\mbox{and} \\,(\\widetilde{\\bm{q}}_\\epsilon)_i = \\alpha_i \\tz_i^{-1}, \\,\\forall i \\ge 3.\n\t\t\\]\n\t\tNotice that \\( \\bm{q}_\\epsilon \\) only differs from the \\( \\bm{f} \\) in \\eqref{eq:1st-type-face-ray} in two entries. One can check that \\( \\inProd{ \\bm{z}}{\\bm{q}_\\epsilon} = 0 \\) and \\( \\bm{q}_\\epsilon\\to \\bm{f}\\in \\Fr\\backslash\\{\\bm{0}\\} \\) as $\\epsilon \\downarrow 0$.\n","start":57841,"end":58858}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["eq:distpower","remark:F-r-order-1/2"],"declared_external":false,"source_only":false,"title":"锥距离的二阶上界","source_units":[{"raw_tex":"\t\tMoreover, we have\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t&\\prod_{i=1}^n (\\widetilde{\\bm{q}}_\\epsilon)_i^{\\alpha_i}  = (\\alpha_1 - \\epsilon)^{\\alpha_1}(\\alpha_2 + \\epsilon)^{\\alpha_2}\\tz_1^{-\\alpha_1}\\tz_2^{-\\alpha_2}\\prod_{i=3}^n\\left(\\frac{\\alpha_i}{\\tz_i}\\right)^{\\alpha_i}          \\\\\n\t\t\t& =  \\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2}\\prod_{i=1}^n\\left(\\frac{\\alpha_i}{\\tz_i}\\right)^{\\alpha_i}\n\t\t\t\\overset{\\rm (a)}= \\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2} \\|\\bbz\\|^{-1}                                                                                          \\\\\n\t\t\t& = \\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2}\\|\\bar{\\bm{q}}_\\epsilon\\|,\n\t\t\\end{aligned}\n\t\t\\]\n\t\twhere (a) holds because \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz\\neq \\bm{0} \\). In view of this, if we define a continuous function \\( \\bm{p}:(0,\\alpha_1)\\to \\gK \\)  by $\\epsilon \\mapsto \\bm{p}_\\epsilon := (\\bar{\\bm{p}}_\\epsilon,\\widetilde{\\bm{p}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{p}}_\\epsilon := -\\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2}\\frac{\\bbz}{\\|\\bbz\\|^2}\\quad{\\rm and}\\quad \\widetilde{\\bm{p}}_\\epsilon := \\widetilde{\\bm{q}}_\\epsilon;\n\t\t\\]\n\t\tthen it is clear that \\( \\bm{p}_\\epsilon \\in \\gK \\), and we can compute that\n\t\t\\begin{equation}\\label{eq:distpower}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\dist(\\bm{q}_\\epsilon,\\gK) & \\le \\|\\bm{q}_\\epsilon - \\bm{p}_\\epsilon\\| = \\frac1{\\|\\bbz\\|}\\left|\\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2} - 1\\right| \\\\\n\t\t\t\t& = \\frac1{\\|\\bbz\\|}\\left|(1 - \\epsilon + O(\\epsilon^2))(1 + \\epsilon + O(\\epsilon^2)) - 1\\right| = O(\\epsilon^2).\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\n","start":58858,"end":60745}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":["eq:lowerlowerboundpower","remark:F-r-order-1/2"],"declared_external":false,"source_only":false,"title":"射线距离的一阶下界","source_units":[{"raw_tex":"\t\tNext, we estimate \\( \\dist(\\bm{q}_\\epsilon,\\Fr) \\).\n\t\tNotice that \\( \\inProd{ \\bm{q}_\\epsilon}{\\bm{f}} > 0 \\) for all sufficiently small \\( \\epsilon \\) because \\( \\bm{q}_\\epsilon \\to \\bm{f} \\). Hence, using the definition of \\( \\Fr \\) and {Lemma \\ref{lemma:Lemma-2.11}}, we see that\n\t\t\\[\n\t\t\\dist(\\bm{q}_\\epsilon,\\Fr)^2 = \\left\\|\\bm{q}_\\epsilon - \\frac{\\inProd{ \\bm{q}_\\epsilon}{ \\bm{f}}}{\\|\\bm{f}\\|^2}\\bm{f}\\right\\|^2 = \\|\\bm{q}_\\epsilon\\|^2 - \\frac{(\\inProd{ \\bm{q}_\\epsilon}{ \\bm{f}})^2}{\\|\\bm{f}\\|^2}.\n\t\t\\]\n\t\tA direct computation then shows that\n\t\t\\begin{align}\n\t\t\t\\|\\bm{q}_\\epsilon\\|^2 & = \\frac1{\\|\\bbz\\|^2} + (\\alpha_1-\\epsilon)^2\\tz_1^{-2} + (\\alpha_2+\\epsilon)^2\\tz_2^{-2} + \\sum_{i=3}^n \\alpha^2_i \\tz_i^{-2}                       \\notag \\\\\n\t\t\t& = \\frac1{\\|\\bbz\\|^2} + \\sum_{i=1}^n \\alpha^2_i \\tz_i^{-2} + 2\\epsilon(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2}) + \\epsilon^2(\\tz_1^{-2} + \\tz_2^{-2}) \\notag\\\\\n\t\t\t& = \\|\\bm{f}\\|^2 + 2\\epsilon(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2}) + \\epsilon^2(\\tz_1^{-2} + \\tz_2^{-2}),\\notag\n\t\t\\end{align}\n\t\twhere the last equality follows from the definition of \\( \\bm{f} \\) in \\eqref{eq:1st-type-face-ray}.\n\t\tFurthermore,\n\t\t\\begin{align}\n(\\inProd{ \\bm{q}_\\epsilon}{ \\bm{f}})^2 & =\\textstyle \\left(\\frac1{\\|\\bbz\\|^2} + \\alpha_1(\\alpha_1-\\epsilon)\\tz_1^{-2} + \\alpha_2(\\alpha_2+\\epsilon)\\tz_2^{-2} + \\sum_{i=3}^n \\alpha^2_i \\tz_i^{-2}\\right)^2 \\notag\\\\\n\t\t\t& = \\left[ \\|\\bm{f}\\|^2  + \\epsilon (\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})\\right]^2                                                                    \\notag\\\\\n\t\t\t& = \\|\\bm{f}\\|^4 + 2\\epsilon\\|\\bm{f}\\|^2 (\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2}) + \\epsilon^2(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2.\\notag\n\t\t\\end{align}\n\t\tCombining the above three identities, we deduce further that\n\t\t\\begin{align}\\label{eq:lowerlowerboundpower}\n\t\t\t\\dist(\\bm{q}_\\epsilon,\\Fr)^2 & = \\epsilon^2\\left(\\tz_1^{-2} + \\tz_2^{-2} - \\frac{(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2}{\\|\\bm{f}\\|^2}\\right)\\nonumber                          \\\\\n\t\t\t& \\ge \\epsilon^2\\left(\\tz_1^{-2} + \\tz_2^{-2} - \\frac{(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2}{\\alpha^2_2\\tz_2^{-2} + \\alpha^2_1\\tz_1^{-2}}\\right),\n\t\t\\end{align}\n\t\twhere the inequality follows from the definition of \\( \\bm{f} \\). Now, notice that in \\eqref{eq:lowerlowerboundpower}, the scalar term is strictly greater than zero, because\n\t\t\\[\n\t\t(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2 < (\\alpha_2\\tz_2^{-2} + \\alpha_1\\tz_1^{-2})^2 \\le (\\tz_1^{-2} + \\tz_2^{-2})(\\alpha^2_2\\tz_2^{-2} + \\alpha^2_1\\tz_1^{-2}),\n\t\t\\]\n\t\twhere the strict inequality holds because \\( \\alpha_i\\tz_i^{-2} > 0 \\) for \\( i = 1 \\), \\( 2 \\), and the last inequality follows from the Cauchy-Schwarz inequality.\n","start":60745,"end":63455}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":["remark:F-r-order-1/2"],"declared_external":false,"source_only":false,"title":"验证平方根最优性判据","source_units":[{"raw_tex":"\t\tThis together with \\eqref{eq:lowerlowerboundpower} shows that \\( \\dist(\\bm{q}_\\epsilon,\\Fr) = \\Omega(\\epsilon) \\). Combining this with \\eqref{eq:distpower}, we obtain\n\t\t$\n\t\t\\limsup_{\\epsilon\\downarrow 0}\\frac{\\dist(\\bm{q}_\\epsilon,\\gK)^\\frac12}{\\dist(\\bm{q}_\\epsilon,\\Fr)} < \\infty\n\t\t$.\n\t\tThus \\( | \\cdot | ^{\\frac{1}{2}} \\) satisfies the asymptotic optimality criterion {(cf. \\cite[Definition 3.1]{LiLoPo21}}) for \\( \\gK \\) and \\( \\bm{z} \\){, which implies that the error bound is optimal in the sense of \\cite[Theorem 3.2(b)]{LiLoPo21}.}\n","start":63455,"end":63997}]},{"id":"S049","semantic_type":"theorem","kind":"theorem","labels":["eq:gamma-beta-2nd-type-face","thm:xface"],"declared_external":false,"source_only":false,"title":"坐标面的幂次误差界","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm:xface}\n\t\tLet \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz = \\bm{0} \\) and let \\( \\Fhatz := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Let \\( \\stdIndex := \\{i\\,| \\,\\tz_i > 0\\} \\),\\footnote{Since \\( \\bbz = \\bm{0} \\) and \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\), we must have \\( \\emptyset\\neq \\stdIndex \\subsetneq \\{1,2,\\ldots,n\\} \\).} \\( \\eta > 0 \\) and define \\( \\beta:= \\sum_{i\\in \\stdIndex}\\alpha_i \\) and\n\t\t\\begin{equation}\\label{eq:gamma-beta-2nd-type-face}\n\t\t \\!\\gamma_{\\bm{z},\\eta} \\!:=\\! \\inf_{\\bm{v}}\\left\\{\\frac{\\|\\bm{v} - \\bm{w}\\|^\\beta}{\\|\\bm{u} - \\bm{w}\\|}\\,\\bigg| \\,\n\\begin{array}{c}\n\\bm{v}\\in \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fhatz,\\,\\bm{w} = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}),\\\\\n\\bm{u} = P_{\\Fhatz}(\\bm{w}),\\,\\bm{u} \\neq \\bm{w}\n\\end{array}\\right\\}.\n\t\t\\end{equation}\n\t\tThen it holds that \\( \\gamma_{\\bm{z},\\eta}\\in (0,\\infty] \\) and that\n\t\t\\[\n\t\t\\dist(\\bm{q},\\Fhatz)\\le \\max\\{2\\eta^{1-\\beta},2\\gamma_{\\bm{z},\\eta}^{-1}\\}\\cdot\\dist(\\bm{q},\\gK)^\\beta\\,\\,\\text{ whenever }  \\bm{q}\\in \\{\\bm{z}\\}^\\perp\\cap \\B(\\eta) .\n\t\t\\]\n\t\\end{theorem}","start":64632,"end":65697}]},{"id":"S050","semantic_type":"proof","kind":"proof","labels":["eq:dist-w-u","eq:dist-w-v","eq:form-w-by-v","eq:upbd-norm-bar-v"],"declared_external":false,"source_only":false,"title":"坐标面估计完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\t\t%\tIf \\( \\gamma_{\\bm{z},\\eta} = 0 \\), in view of \\cite[Lemma 3.12]{LiLoPo20}, there exist \\( \\widehat{\\bm{v}}\\in \\Fhatz \\) and a sequence \\( \\{\\bm{v}^k\\}\\subset \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fhatz \\) such that\n\t\t%\t\\begin{equation}\\label{eq:forcontradiction-2nd-face}\n\t\t%\t\t\\lim_{k\\to \\infty}\\bm{v}^k = \\lim_{k\\to \\infty}\\bm{w}^k = \\widehat{\\bm{v}}\\,\\,{\\rm and}\\,\\,\\lim_{k\\to\\infty}\\frac{\\|\\bm{w}^k - \\bm{v}^k\\|^{\\beta }}{\\|\\bm{w}^k - u^k\\|} = 0,\n\t\t%\t\\end{equation}\n\t\t%\twhere \\( \\bm{w}^k = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}^k) \\), \\( \\bm{u}^k = P_{\\Fhatz}(\\bm{w}^k) \\) and \\( \\bm{u}^k\\neq \\bm{w}^k \\).\n\n\t\t{In view of \\cite[Theorem 3.10]{LiLoPo20}}, we need only show that $\\gamma_{\\bm{z},\\eta} > 0$.\n%\t\t$$\n%\t\t0<\\gamma_{\\bm{z},\\eta} = \\inf_{\\bm{v}} \\left\\{ \\frac{\\|\\bm{w} -\\bm{v} \\|^\\beta}{\\|\\bm{w} -\\bm{u} \\| } \\Big|\\bm{v} \\in \\bd \\stdCone \\cap \\B(\\eta) \\setminus \\stdFace,\\bm{w} = P_{\\{\\bm{z}\\}^{\\perp }}(\\bm{v}), \\bm{u} = P_{\\stdFace}(\\bm{w}), \\bm{w} \\neq\\bm{u} \\right\\}.\n%\t\t$$\n\t\tTo that end, let \\( \\bm{v}\\in \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fhatz \\), \\( \\bm{w} = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}) \\), \\( \\bm{u} = P_{\\Fhatz}(\\bm{w}) \\), and \\( \\bm{u} \\neq \\bm{w} \\). Then a direct computation shows\n\t\tthat\n\t\t\\begin{equation}\\label{eq:dist-w-v}\n\t\t\t\\|\\bm{w}- \\bm{v}\\| = \\frac1{\\|\\bm{z}\\|}|\\inProd{ \\bm{z}}{\\bm{v}}| \\overset{\\rm (a)}= \\frac1{\\|\\bm{z}\\|}\\sum_{i\\in \\stdIndex}\\tz_i\\tv_i \\overset{\\rm (b)}\\ge \\frac{\\min_{i\\in \\stdIndex}\\tz_i}{\\|\\bm{z}\\|}\\sum_{i\\in \\stdIndex}\\tv_i\n\t\t\t\\overset{\\rm (c)}\\ge \\frac{\\min_{i\\in \\stdIndex}\\tz_i}{\\|\\bm{z}\\|}\\|\\tbv_{\\stdIndex}\\|,\n\t\t\\end{equation}\n\t\twhere (a), (b) and (c) hold because \\( \\tv_i\\ge 0 \\) and \\( \\tz_i > 0 \\) for all \\( i\\in \\stdIndex \\), with \\( \\|\\tbv_{\\stdIndex}\\|:= \\sqrt{\\sum_{i\\in \\stdIndex}\\tv^2_i} \\) (note that \\( \\stdIndex \\neq \\emptyset \\), thanks to \\( \\bbz = 0 \\) and \\( \\bm{z}\\neq \\bm{0} \\)). Next, notice that \\( \\bm{w} = \\bm{v} - \\frac{\\inProd{ \\bm{z}}{\\bm{v}}}{\\|\\bm{z}\\|^2}\\bm{z} \\). Using this and the definitions of \\( \\bm{z} \\) and \\( \\stdIndex \\), we deduce that\n\t\t\\begin{equation}\\label{eq:form-w-by-v}\n\t\t\t\\bbw = \\bbv,\\,\\,\\,\\tw_i = \\tv_i - \\frac{\\tz_i}{\\|\\bm{z}\\|^2}\\bigg(\\sum_{j\\in \\stdIndex}\\tz_j\\tv_j\\bigg)\\,\\,\\,\\, \\forall i\\in \\stdIndex\\,\\,\\,\\,\\,\\,\\,\\mbox{and\\,\\,\\,\\,\\,\\,\\,} \\tw_i = \\tv_i \\ge 0\\,\\,\\,\\,\\forall i \\notin \\stdIndex.\n\t\t\\end{equation}\n\t\tIn view of this and the definition of \\( \\Fhatz \\) in \\eqref{eq:2nd-type-face-d-dim}, we see that \\( \\tu_i = \\tw_i \\) whenever \\( i\\notin \\stdIndex \\), and hence\n\t\t\\begin{equation}\\label{eq:dist-w-u}\n\t\t\t\\|\\bm{w} - \\bm{u}\\| = \\sqrt{\\|\\bbw\\|^2 + \\sum_{i\\in \\stdIndex}\\tw^2_i} \\le \\sqrt{\\|\\bbv\\|^2 + n(1+\\sqrt{n})^2\\|\\tbv_{\\stdIndex}\\|^2},\n\t\t\\end{equation}\n\t\twhere the inequality follows from \\eqref{eq:form-w-by-v} and the fact that for each \\( i\\in \\stdIndex \\),\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t|\\tw_i| & = \\bigg|\\tv_i - \\frac{\\tz_i}{\\|\\bm{z}\\|^2}\\bigg(\\sum_{j\\in \\stdIndex}\\tz_j\\tv_j\\bigg)\\bigg|\\le \\bigg(1 + \\frac{|\\tz_i|}{\\|\\bm{z}\\|^2}\\sum_{j\\in \\stdIndex}|\\tz_j|\\bigg)\\|\\tbv_{\\stdIndex}\\| \\\\\n\t\t\t& \\le \\left(1 + \\frac{\\sqrt{n}|\\tz_i|}{\\|\\bm{z}\\|}\\right)\\|\\tbv_{\\stdIndex}\\| \\le (1+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\|.\n\t\t\\end{aligned}\n\t\t\\]\n\t\tNext, note that we have\n\t\t\\begin{equation}\\label{eq:upbd-norm-bar-v}\n\t\t\t\\|\\bbv\\|=%\n\t\t\t\\prod_{i=1}^n(\\tv_i)^{\\alpha_i} %\n\t\t\t= \\prod_{i\\notin \\stdIndex}\\tv_i^{\\alpha_i} \\cdot \\prod_{i\\in \\stdIndex}\\tv_i^{\\alpha_i}%\n\t\t\t\\le \\prod_{i\\notin \\stdIndex}\\eta^{\\alpha_i} \\cdot\\prod_{i\\in \\stdIndex}\\|\\tbv_{\\stdIndex}\\|^{\\alpha_i}%\n\t\t\t= \\eta^{1-\\beta} \\|\\tbv_{\\stdIndex}\\|^\\beta,\n\t\t\\end{equation}\n\t\twhere the inequality holds because \\( \\bm{v}\\in \\B(\\eta) \\). Combining \\eqref{eq:dist-w-v}, \\eqref{eq:dist-w-u} and \\eqref{eq:upbd-norm-bar-v}, we deduce\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\|\\bm{w} - \\bm{u}\\| & \\le \\sqrt{\\|\\bbv\\|^2 + n(1+\\sqrt{n})^2\\|\\tbv_{\\stdIndex}\\|^2} \\le \\|\\bbv\\| + (n+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\|                                                                                                     \\\\\n\t\t\t& \\le \\eta^{1-\\beta} \\|\\tbv_{\\stdIndex}\\|^\\beta + (n+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\| = (\\eta^{1-\\beta}  + (n+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\|^{1-\\beta})\\|\\tbv_{\\stdIndex}\\|^\\beta                                   \\\\\n\t\t\t& \\overset{\\rm (a)}\\le \\eta^{1-\\beta}(n+1 + \\sqrt{n})\\|\\tbv_{\\stdIndex}\\|^\\beta \\overset{\\rm (b)}{\\le}\\frac{\\eta^{1-\\beta}(n+1 + \\sqrt{n})\\|\\bm{z}\\|^\\beta}{\\left(\\min_{i\\in_{\\stdIndex}}\\tz_i\\right)^\\beta}\\|\\bm{w}-\\bm{v}\\|^\\beta.\n\t\t\\end{aligned}\n\t\t\\]\n\t\tHere (a) holds since \\( \\bm{v}\\in \\B(\\eta) \\) and \\( \\beta \\in (0,1) \\); (b) is true because of \\eqref{eq:dist-w-v}.\n\t\tThus, \\( \\gamma_{\\bm{z},\\eta}\\ge \\frac{\\left(\\min_{i\\in \\stdIndex}\\tz_i\\right)^\\beta}{\\eta^{1-\\beta}(n+1 + \\sqrt{n})\\|\\bm{z}\\|^\\beta} > 0 \\), and the desired error bound {follows from \\cite[Theorem 3.10]{LiLoPo20}}.\n\t\\end{proof}","start":65699,"end":70480}]},{"id":"S051","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"坐标面证明的废弃反证稿","source_units":[{"raw_tex":"\t\t%\tIf \\( \\gamma_{\\bm{z},\\eta} = 0 \\), in view of \\cite[Lemma 3.12]{LiLoPo20}, there exist \\( \\widehat{\\bm{v}}\\in \\Fhatz \\) and a sequence \\( \\{\\bm{v}^k\\}\\subset \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fhatz \\) such that\n\t\t%\t\\begin{equation}\\label{eq:forcontradiction-2nd-face}\n\t\t%\t\t\\lim_{k\\to \\infty}\\bm{v}^k = \\lim_{k\\to \\infty}\\bm{w}^k = \\widehat{\\bm{v}}\\,\\,{\\rm and}\\,\\,\\lim_{k\\to\\infty}\\frac{\\|\\bm{w}^k - \\bm{v}^k\\|^{\\beta }}{\\|\\bm{w}^k - u^k\\|} = 0,\n\t\t%\t\\end{equation}\n\t\t%\twhere \\( \\bm{w}^k = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}^k) \\), \\( \\bm{u}^k = P_{\\Fhatz}(\\bm{w}^k) \\) and \\( \\bm{u}^k\\neq \\bm{w}^k \\).\n","start":65713,"end":66310},{"raw_tex":"%\t\t$$\n%\t\t0<\\gamma_{\\bm{z},\\eta} = \\inf_{\\bm{v}} \\left\\{ \\frac{\\|\\bm{w} -\\bm{v} \\|^\\beta}{\\|\\bm{w} -\\bm{u} \\| } \\Big|\\bm{v} \\in \\bd \\stdCone \\cap \\B(\\eta) \\setminus \\stdFace,\\bm{w} = P_{\\{\\bm{z}\\}^{\\perp }}(\\bm{v}), \\bm{u} = P_{\\stdFace}(\\bm{w}), \\bm{w} \\neq\\bm{u} \\right\\}.\n%\t\t$$\n","start":66408,"end":66688}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":["eq:dist-w-v"],"declared_external":false,"source_only":false,"title":"超平面残差控制受限坐标","source_units":[{"raw_tex":"\\begin{proof}\n","start":65699,"end":65713},{"raw_tex":"\n\t\t{In view of \\cite[Theorem 3.10]{LiLoPo20}}, we need only show that $\\gamma_{\\bm{z},\\eta} > 0$.\n","start":66310,"end":66408},{"raw_tex":"\t\tTo that end, let \\( \\bm{v}\\in \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fhatz \\), \\( \\bm{w} = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}) \\), \\( \\bm{u} = P_{\\Fhatz}(\\bm{w}) \\), and \\( \\bm{u} \\neq \\bm{w} \\). Then a direct computation shows\n\t\tthat\n\t\t\\begin{equation}\\label{eq:dist-w-v}\n\t\t\t\\|\\bm{w}- \\bm{v}\\| = \\frac1{\\|\\bm{z}\\|}|\\inProd{ \\bm{z}}{\\bm{v}}| \\overset{\\rm (a)}= \\frac1{\\|\\bm{z}\\|}\\sum_{i\\in \\stdIndex}\\tz_i\\tv_i \\overset{\\rm (b)}\\ge \\frac{\\min_{i\\in \\stdIndex}\\tz_i}{\\|\\bm{z}\\|}\\sum_{i\\in \\stdIndex}\\tv_i\n\t\t\t\\overset{\\rm (c)}\\ge \\frac{\\min_{i\\in \\stdIndex}\\tz_i}{\\|\\bm{z}\\|}\\|\\tbv_{\\stdIndex}\\|,\n\t\t\\end{equation}\n\t\twhere (a), (b) and (c) hold because \\( \\tv_i\\ge 0 \\) and \\( \\tz_i > 0 \\) for all \\( i\\in \\stdIndex \\), with \\( \\|\\tbv_{\\stdIndex}\\|:= \\sqrt{\\sum_{i\\in \\stdIndex}\\tv^2_i} \\) (note that \\( \\stdIndex \\neq \\emptyset \\), thanks to \\( \\bbz = 0 \\) and \\( \\bm{z}\\neq \\bm{0} \\)). Next, notice that \\( \\bm{w} = \\bm{v} - \\frac{\\inProd{ \\bm{z}}{\\bm{v}}}{\\|\\bm{z}\\|^2}\\bm{z} \\). Using this and the definitions of \\( \\bm{z} \\) and \\( \\stdIndex \\), we deduce that\n","start":66688,"end":67739}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":["eq:dist-w-u","eq:form-w-by-v"],"declared_external":false,"source_only":false,"title":"显式投影控制面距离","source_units":[{"raw_tex":"\t\t\\begin{equation}\\label{eq:form-w-by-v}\n\t\t\t\\bbw = \\bbv,\\,\\,\\,\\tw_i = \\tv_i - \\frac{\\tz_i}{\\|\\bm{z}\\|^2}\\bigg(\\sum_{j\\in \\stdIndex}\\tz_j\\tv_j\\bigg)\\,\\,\\,\\, \\forall i\\in \\stdIndex\\,\\,\\,\\,\\,\\,\\,\\mbox{and\\,\\,\\,\\,\\,\\,\\,} \\tw_i = \\tv_i \\ge 0\\,\\,\\,\\,\\forall i \\notin \\stdIndex.\n\t\t\\end{equation}\n\t\tIn view of this and the definition of \\( \\Fhatz \\) in \\eqref{eq:2nd-type-face-d-dim}, we see that \\( \\tu_i = \\tw_i \\) whenever \\( i\\notin \\stdIndex \\), and hence\n\t\t\\begin{equation}\\label{eq:dist-w-u}\n\t\t\t\\|\\bm{w} - \\bm{u}\\| = \\sqrt{\\|\\bbw\\|^2 + \\sum_{i\\in \\stdIndex}\\tw^2_i} \\le \\sqrt{\\|\\bbv\\|^2 + n(1+\\sqrt{n})^2\\|\\tbv_{\\stdIndex}\\|^2},\n\t\t\\end{equation}\n\t\twhere the inequality follows from \\eqref{eq:form-w-by-v} and the fact that for each \\( i\\in \\stdIndex \\),\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t|\\tw_i| & = \\bigg|\\tv_i - \\frac{\\tz_i}{\\|\\bm{z}\\|^2}\\bigg(\\sum_{j\\in \\stdIndex}\\tz_j\\tv_j\\bigg)\\bigg|\\le \\bigg(1 + \\frac{|\\tz_i|}{\\|\\bm{z}\\|^2}\\sum_{j\\in \\stdIndex}|\\tz_j|\\bigg)\\|\\tbv_{\\stdIndex}\\| \\\\\n\t\t\t& \\le \\left(1 + \\frac{\\sqrt{n}|\\tz_i|}{\\|\\bm{z}\\|}\\right)\\|\\tbv_{\\stdIndex}\\| \\le (1+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\|.\n\t\t\\end{aligned}\n\t\t\\]\n","start":67739,"end":68859}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":["eq:upbd-norm-bar-v"],"declared_external":false,"source_only":false,"title":"边界乘积产生指数","source_units":[{"raw_tex":"\t\tNext, note that we have\n\t\t\\begin{equation}\\label{eq:upbd-norm-bar-v}\n\t\t\t\\|\\bbv\\|=%\n\t\t\t\\prod_{i=1}^n(\\tv_i)^{\\alpha_i} %\n\t\t\t= \\prod_{i\\notin \\stdIndex}\\tv_i^{\\alpha_i} \\cdot \\prod_{i\\in \\stdIndex}\\tv_i^{\\alpha_i}%\n\t\t\t\\le \\prod_{i\\notin \\stdIndex}\\eta^{\\alpha_i} \\cdot\\prod_{i\\in \\stdIndex}\\|\\tbv_{\\stdIndex}\\|^{\\alpha_i}%\n\t\t\t= \\eta^{1-\\beta} \\|\\tbv_{\\stdIndex}\\|^\\beta,\n\t\t\\end{equation}\n","start":68859,"end":69247}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"取得显式正面常数","source_units":[{"raw_tex":"\t\twhere the inequality holds because \\( \\bm{v}\\in \\B(\\eta) \\). Combining \\eqref{eq:dist-w-v}, \\eqref{eq:dist-w-u} and \\eqref{eq:upbd-norm-bar-v}, we deduce\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\|\\bm{w} - \\bm{u}\\| & \\le \\sqrt{\\|\\bbv\\|^2 + n(1+\\sqrt{n})^2\\|\\tbv_{\\stdIndex}\\|^2} \\le \\|\\bbv\\| + (n+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\|                                                                                                     \\\\\n\t\t\t& \\le \\eta^{1-\\beta} \\|\\tbv_{\\stdIndex}\\|^\\beta + (n+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\| = (\\eta^{1-\\beta}  + (n+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\|^{1-\\beta})\\|\\tbv_{\\stdIndex}\\|^\\beta                                   \\\\\n\t\t\t& \\overset{\\rm (a)}\\le \\eta^{1-\\beta}(n+1 + \\sqrt{n})\\|\\tbv_{\\stdIndex}\\|^\\beta \\overset{\\rm (b)}{\\le}\\frac{\\eta^{1-\\beta}(n+1 + \\sqrt{n})\\|\\bm{z}\\|^\\beta}{\\left(\\min_{i\\in_{\\stdIndex}}\\tz_i\\right)^\\beta}\\|\\bm{w}-\\bm{v}\\|^\\beta.\n\t\t\\end{aligned}\n\t\t\\]\n\t\tHere (a) holds since \\( \\bm{v}\\in \\B(\\eta) \\) and \\( \\beta \\in (0,1) \\); (b) is true because of \\eqref{eq:dist-w-v}.\n\t\tThus, \\( \\gamma_{\\bm{z},\\eta}\\ge \\frac{\\left(\\min_{i\\in \\stdIndex}\\tz_i\\right)^\\beta}{\\eta^{1-\\beta}(n+1 + \\sqrt{n})\\|\\bm{z}\\|^\\beta} > 0 \\), and the desired error bound {follows from \\cite[Theorem 3.10]{LiLoPo20}}.\n\t\\end{proof}","start":69247,"end":70480}]},{"id":"S056","semantic_type":"lemma","kind":"lemma","labels":["remark:F-z-order-beta"],"declared_external":false,"source_only":false,"title":"坐标面指数的渐近最优性","source_units":[{"raw_tex":"\\begin{remark}[Optimality of the error bound in Theorem~\\ref{thm:xface}]\\label{remark:F-z-order-beta}\n\t\t\\!\\!\\!Let \\( \\bm{z}\\!\\in\\!\\! \\bd\\dgK\\backslash\\!\\{\\bm{0}\\} \\) with \\( \\bbz = \\bm{0} \\) and let \\( \\Fhatz := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Let \\( \\stdIndex := \\{i\\,| \\,\\tz_i > 0\\}\\neq \\emptyset \\) and define\n\t\t\\[\n\t\t\\beta:= \\sum_{i\\in \\stdIndex}\\alpha_i\\in (0,1).\n\t\t\\]\n","start":70483,"end":70853},{"raw_tex":"\t\tThus \\( | \\cdot | ^{\\beta} \\) satisfies the asymptotic optimality criterion {(cf. \\cite[Definition 3.1]{LiLoPo21}}) for \\( \\gK \\) and \\( \\bm{z} \\){, which implies that the error bound is optimal in the sense of \\cite[Theorem 3.2(b)]{LiLoPo21}.}\n\t\\end{remark}","start":72269,"end":72529}]},{"id":"S057","semantic_type":"proof","kind":"proof","labels":["remark:F-z-order-beta"],"declared_external":false,"source_only":false,"title":"坐标面最优性完整构造论证","source_units":[{"raw_tex":"\t\tFix any \\( \\bm{u}\\in \\R^m \\) with \\( \\|\\bm{u}\\|=1 \\) and define the continuous function \\( \\bm{q}:(0,1)\\to \\{\\bm{z}\\}^\\perp \\) by $\\epsilon \\mapsto \\bm{q}_\\epsilon := (\\bar{\\bm{q}}_\\epsilon,\\widetilde{\\bm{q}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{q}}_\\epsilon = \\epsilon^\\beta \\bm{u},\\,\\,(\\widetilde{\\bm{q}}_\\epsilon)_i = 0\\,\\,\\forall i \\in \\stdIndex,\\,\\mbox{and} \\,(\\widetilde{\\bm{q}}_\\epsilon)_i = 1, \\,\\forall i \\notin \\stdIndex.\n\t\t\\]\n\t\tIt is clear that for all $\\epsilon$, \\( \\inProd{ \\bm{z}}{\\bm{q}_\\epsilon} = 0 \\) and \\( \\dist(\\bm{q}_\\epsilon,\\Fhatz) \\to 0 \\) as $\\epsilon \\downarrow 0$ . Now, define the function \\( \\bm{p}:(0,1)\\to \\gK \\) by $\\epsilon \\mapsto \\bm{p}_\\epsilon := (\\bar{\\bm{p}}_\\epsilon,\\widetilde{\\bm{p}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{p}}_\\epsilon = \\epsilon^\\beta \\bm{u},\\,\\,(\\widetilde{\\bm{p}}_\\epsilon)_i = \\epsilon\\,\\,\\forall i \\in \\stdIndex,\\,\\mbox{and} \\,(\\widetilde{\\bm{p}}_\\epsilon)_i = 1, \\,\\forall i \\notin \\stdIndex.\n\t\t\\]\n\t\tClearly $\\bm{p}_\\epsilon$ lies in \\( \\gK \\), and we have that\n\t\t$\n\t\t\t\\dist(\\bm{q}_\\epsilon,\\gK)\\le \\|\\bm{q}_\\epsilon - \\bm{p}_\\epsilon\\| \\le |\\stdIndex|\\cdot\\epsilon\n\t\t$.\n\t\tOn the other hand, we have in view of \\eqref{eq:2nd-type-face-d-dim} that \\( \\dist(\\bm{q}_\\epsilon,\\Fhatz) = \\epsilon^\\beta > 0 \\).\n\t\tHence,\n\t\t$\n\t\t\\limsup_{\\epsilon\\downarrow 0}\\frac{\\dist(\\bm{q}_\\epsilon,\\gK)^\\beta}{\\dist(\\bm{q}_\\epsilon,\\Fhatz)} \\le |\\stdIndex|^\\beta < \\infty\n\t\t$.\n","start":70853,"end":72269}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":["remark:F-z-order-beta"],"declared_external":false,"source_only":false,"title":"坐标面的横向扰动路径","source_units":[{"raw_tex":"\t\tFix any \\( \\bm{u}\\in \\R^m \\) with \\( \\|\\bm{u}\\|=1 \\) and define the continuous function \\( \\bm{q}:(0,1)\\to \\{\\bm{z}\\}^\\perp \\) by $\\epsilon \\mapsto \\bm{q}_\\epsilon := (\\bar{\\bm{q}}_\\epsilon,\\widetilde{\\bm{q}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{q}}_\\epsilon = \\epsilon^\\beta \\bm{u},\\,\\,(\\widetilde{\\bm{q}}_\\epsilon)_i = 0\\,\\,\\forall i \\in \\stdIndex,\\,\\mbox{and} \\,(\\widetilde{\\bm{q}}_\\epsilon)_i = 1, \\,\\forall i \\notin \\stdIndex.\n\t\t\\]\n\t\tIt is clear that for all $\\epsilon$, \\( \\inProd{ \\bm{z}}{\\bm{q}_\\epsilon} = 0 \\) and \\( \\dist(\\bm{q}_\\epsilon,\\Fhatz) \\to 0 \\) as $\\epsilon \\downarrow 0$ . Now, define the function \\( \\bm{p}:(0,1)\\to \\gK \\) by $\\epsilon \\mapsto \\bm{p}_\\epsilon := (\\bar{\\bm{p}}_\\epsilon,\\widetilde{\\bm{p}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{p}}_\\epsilon = \\epsilon^\\beta \\bm{u},\\,\\,(\\widetilde{\\bm{p}}_\\epsilon)_i = \\epsilon\\,\\,\\forall i \\in \\stdIndex,\\,\\mbox{and} \\,(\\widetilde{\\bm{p}}_\\epsilon)_i = 1, \\,\\forall i \\notin \\stdIndex.\n\t\t\\]\n","start":70853,"end":71810}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":["remark:F-z-order-beta"],"declared_external":false,"source_only":false,"title":"坐标面最优性距离比值","source_units":[{"raw_tex":"\t\tClearly $\\bm{p}_\\epsilon$ lies in \\( \\gK \\), and we have that\n\t\t$\n\t\t\t\\dist(\\bm{q}_\\epsilon,\\gK)\\le \\|\\bm{q}_\\epsilon - \\bm{p}_\\epsilon\\| \\le |\\stdIndex|\\cdot\\epsilon\n\t\t$.\n\t\tOn the other hand, we have in view of \\eqref{eq:2nd-type-face-d-dim} that \\( \\dist(\\bm{q}_\\epsilon,\\Fhatz) = \\epsilon^\\beta > 0 \\).\n\t\tHence,\n\t\t$\n\t\t\\limsup_{\\epsilon\\downarrow 0}\\frac{\\dist(\\bm{q}_\\epsilon,\\gK)^\\beta}{\\dist(\\bm{q}_\\epsilon,\\Fhatz)} \\le |\\stdIndex|^\\beta < \\infty\n\t\t$.\n","start":71810,"end":72269}]},{"id":"S060","semantic_type":"corollary","kind":"corollary","labels":["3.6i","corollary:1-FRFs-power-cones","eq:1-FRF-1st-face"],"declared_external":false,"source_only":false,"title":"射线面的一步残差函数","source_units":[{"raw_tex":"\\begin{corollary}\\label{corollary:1-FRFs-power-cones}\n\t\tConsider \\( \\gK \\) and its dual cone \\( \\dgK \\).\n\t\t\\begin{enumerate}[label=(\\roman*)]\n\t\t\t\\item\\label{3.6i} Let \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz\\neq \\bm{0} \\) and let \\( \\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Let \\( \\gamma_{\\bm{z},t} \\) be defined as in \\eqref{eq:gamma-1/2-1st-type-face}. Then the function \\( \\psi_{\\gK,\\bm{z}}:\\R_+\\times \\R_+\\to \\R_+ \\) given by\\vspace{-0.15 cm}\n\t\t\t\\begin{equation}\\label{eq:1-FRF-1st-face}\\vspace{-0.15 cm}\n\t\t\t\t\\!\\!\\psi_{\\gK,\\bm{z}}(\\epsilon,t) \\!:=\\! \\max \\left\\{\\epsilon,\\epsilon/\\|\\bm{z}\\| \\right\\} + \\max\\{2\\sqrt{t},2\\gamma_{\\bm{z},t}^{-1}\\}(\\epsilon +\\max \\left\\{\\epsilon,\\epsilon/\\|\\bm{z}\\| \\right\\} )^\\frac12\n\t\t\t\\end{equation}\n\t\t\tis a one-step facial residual function for \\( \\gK \\) and \\( \\bm{z} \\).\n","start":72726,"end":73547}]},{"id":"S061","semantic_type":"corollary","kind":"corollary","labels":["3.6i","corollary:1-FRFs-power-cones","eq:1-FRF-2nd-face"],"declared_external":false,"source_only":false,"title":"坐标面的一步残差函数","source_units":[{"raw_tex":"\\begin{corollary}\\label{corollary:1-FRFs-power-cones}\n\t\tConsider \\( \\gK \\) and its dual cone \\( \\dgK \\).\n\t\t\\begin{enumerate}[label=(\\roman*)]\n","start":72726,"end":72868},{"raw_tex":"\t\t\t\\item Let \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz = \\bm{0} \\) and let \\( \\Fhatz := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Let \\( \\gamma_{\\bm{z},t} \\) be defined as in \\eqref{eq:gamma-beta-2nd-type-face}, where \\( \\beta:= \\sum_{i:\\tz_i > 0}\\alpha_i \\). Then the function \\( \\psi_{\\gK,\\bm{z}}:\\R_+\\times \\R_+\\to \\R_+ \\) given by\\vspace{-0.15 cm}\n\t\t\t\\begin{equation}\\label{eq:1-FRF-2nd-face}\\vspace{-0.15 cm}\n\t\t\t\t\\!\\!\\psi_{\\gK,\\bm{z}}(\\epsilon,t) \\!:= \\!\\max\\! \\left\\{\\epsilon,\\epsilon/\\|\\bm{z}\\| \\right\\} + \\max\\{2t^{1-\\beta}\\!,2\\gamma_{\\bm{z},t}^{-1}\\}(\\epsilon +\\max \\left\\{\\epsilon,\\epsilon/\\|\\bm{z}\\| \\right\\} )^\\beta\n\t\t\t\\end{equation}\n\t\t\tis a one-step facial residual function for \\( \\gK \\) and \\( \\bm{z} \\).\n\t\t\\end{enumerate}\n\t\\end{corollary}","start":73547,"end":74303}]},{"id":"S062","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"两类面估计转为FRF","source_units":[{"raw_tex":"\tUsing Theorems~\\ref{thm:Holderian-1/2-1st-type-face} and \\ref{thm:xface} {together with \\cite[Lemma~3.9]{LiLoPo20}}, we have the following result concerning one-step facial residual functions.\n","start":72531,"end":72725}]},{"id":"S063","semantic_type":"theorem","kind":"theorem","labels":["item:error-bound-power-cone-1","item:error-bound-power-cone-2","item:error-bound-power-cone-3","item:error-bound-power-cone-3-1","item:error-bound-power-cone-3-2","item:error-bound-power-cone-3-3","item:error-bound-power-cone-4","thm:error-bound-power-cone-optimality"],"declared_external":false,"source_only":false,"title":"广义幂锥误差界及最优性","source_units":[{"raw_tex":"\\begin{theorem}[Error bounds for the generalized power cone and their optimality]\\label{thm:error-bound-power-cone-optimality}\n\t\tConsider \\( \\gK \\) and its dual cone \\( \\dgK \\).\n\t\tLet \\( \\subSpace \\subseteq \\R^{m + n} \\) be a subspace and \\( \\bm{a} \\in \\R^{m+n} \\) be given.\n\t\tSuppose that \\( ( \\subSpace + \\bm{a} ) \\cap \\gK \\neq \\emptyset \\).\n\t\tThen the following items hold.\n\t\t\\begin{enumerate}[label=(\\roman*)]\n\t\t\t\\item\\label{item:error-bound-power-cone-1} \\( \\dpps (\\gK, \\subSpace + \\bm{a}) \\leq 1 \\).\n\t\t\t\\item\\label{item:error-bound-power-cone-2} If \\( \\dpps (\\gK, \\subSpace + \\bm{a}) = 0 \\), then a Lipschitzian error bound holds.\n\t\t\t\\item\\label{item:error-bound-power-cone-3} If \\( \\dpps (\\gK, \\subSpace + \\bm{a}) = 1 \\), consider the chain of faces \\( \\stdFace \\subsetneq \\gK \\) with length being \\( 2 \\).\n\t\t\t\\begin{enumerate}[label=(\\alph*)]\n\t\t\t\t\\item\\label{item:error-bound-power-cone-3-1} If \\( \\stdFace = \\Fr \\), then a H\\\"olderian error bound with exponent \\( 1 / 2 \\) holds.\n\t\t\t\t\\item\\label{item:error-bound-power-cone-3-2} If \\( \\stdFace = \\Fhatz \\) with \\( \\bm{z} \\in \\dgK \\cap \\subSpace^{\\perp } \\cap \\{\\bm{a}\\}^{\\perp } \\), then a H\\\"olderian error bound with exponent \\( \\beta := \\sum_{i : \\tz_i > 0} \\alpha _i \\) holds.\n\t\t\t\t\\item\\label{item:error-bound-power-cone-3-3} If \\( \\stdFace = \\{\\bm{0}\\} \\), then a Lipschitzian error bound holds.\n\t\t\t\\end{enumerate}\n\t\t\t\\item\\label{item:error-bound-power-cone-4} All these error bounds are the best in the sense {stated in \\cite[Theorem 3.2(b)]{LiLoPo21}}.\n\t\t\\end{enumerate}\n\t\\end{theorem}","start":74378,"end":75929}]},{"id":"S064","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"主定理完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\t\tAs is shown in Section \\ref{subsec:facial-structure}, all the proper exposed faces of the generalized power cone are polyhedral.\n\t\tThen the process of facial reduction needs at most one step to reach the PPS condition. Hence, \\( \\dpps(\\gK, \\subSpace + \\bm{a}) \\leq 1 \\).\n\t\tThis shows item \\ref{item:error-bound-power-cone-1}.\n\n\t\tIf \\( \\dpps(\\gK, \\subSpace + \\bm{a}) = 0 \\), i.e., \\eqref{eq:conic-feasibility-problem} satisfies the PPS condition, then by \\cite[Corollary 3]{BBL99}, a Lipschitzian error bound holds.\n\t\tThis shows item \\ref{item:error-bound-power-cone-2}.\n\n\t\tNext, let \\( \\dpps(\\gK, \\subSpace + \\bm{a}) = 1 \\); i.e., we need one step to reach the PPS condition.\n\t\tIn this case, the error bound depends on the exposed face \\( \\stdFace \\) that contains the feasible region.\n\t\tIf \\(\\stdFace =  \\Fr \\), then by Corollary \\ref{corollary:1-FRFs-power-cones}\\ref{3.6i}, we conclude that a H\\\"olderian error bound with exponent \\( 1 / 2 \\) holds.\n\t\tRemark \\ref{remark:F-r-order-1/2} implies that \\( \\mathfrak{g} = | \\cdot | ^{\\frac{1}{2}}\\) satisfies the asymptotic optimality criterion for \\( \\gK \\) and \\( \\bm{z} \\) with \\( \\bbz \\neq \\bm{0} \\).\n\t\tHence, by {\\cite[Theorem 3.2]{LiLoPo21}}, the obtained H\\\"olderian error bound with exponent \\( 1 / 2 \\) is the best error bound.\n\n\t\tIf \\( \\stdFace = \\Fhatz \\) with \\( \\bm{z} \\in \\dgK \\cap \\subSpace^{\\perp } \\cap \\{\\bm{a}\\}^{\\perp } \\), then using Corollary \\ref{corollary:1-FRFs-power-cones}(ii), we conclude that a H\\\"olderian error bound with exponent \\( \\beta := \\sum _{i \\in \\stdIndex} \\alpha _i \\) holds, where \\( \\stdIndex = \\{i\\,| \\,\\tz_i > 0\\} \\).\n\t\tThe optimality of this error bound comes from Remark \\ref{remark:F-z-order-beta} and {\\cite[Theorem 3.2]{LiLoPo21}}.\nIf \\( \\stdFace = \\{\\bm{0}\\} \\), which means the feasible region is \\( \\{\\bm{0}\\} \\), then a Lipschitzian error bound holds automatically and it is naturally tight, see \\cite[Proposition~27]{L17}.\n\t\\end{proof}","start":75932,"end":77887}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"多面体面使约化至多一步","source_units":[{"raw_tex":"\t\tAs is shown in Section \\ref{subsec:facial-structure}, all the proper exposed faces of the generalized power cone are polyhedral.\n\t\tThen the process of facial reduction needs at most one step to reach the PPS condition. Hence, \\( \\dpps(\\gK, \\subSpace + \\bm{a}) \\leq 1 \\).\n\t\tThis shows item \\ref{item:error-bound-power-cone-1}.\n","start":75946,"end":76274}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"零PPS距离给出线性界","source_units":[{"raw_tex":"\n\t\tIf \\( \\dpps(\\gK, \\subSpace + \\bm{a}) = 0 \\), i.e., \\eqref{eq:conic-feasibility-problem} satisfies the PPS condition, then by \\cite[Corollary 3]{BBL99}, a Lipschitzian error bound holds.\n\t\tThis shows item \\ref{item:error-bound-power-cone-2}.\n","start":76274,"end":76518}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"一步约化后的分类设置","source_units":[{"raw_tex":"\n\t\tNext, let \\( \\dpps(\\gK, \\subSpace + \\bm{a}) = 1 \\); i.e., we need one step to reach the PPS condition.\n\t\tIn this case, the error bound depends on the exposed face \\( \\stdFace \\) that contains the feasible region.\n","start":76518,"end":76734}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"射线分支的仿射误差界","source_units":[{"raw_tex":"\t\tIf \\(\\stdFace =  \\Fr \\), then by Corollary \\ref{corollary:1-FRFs-power-cones}\\ref{3.6i}, we conclude that a H\\\"olderian error bound with exponent \\( 1 / 2 \\) holds.\n","start":76734,"end":76901}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"射线分支的最优性","source_units":[{"raw_tex":"\t\tRemark \\ref{remark:F-r-order-1/2} implies that \\( \\mathfrak{g} = | \\cdot | ^{\\frac{1}{2}}\\) satisfies the asymptotic optimality criterion for \\( \\gK \\) and \\( \\bm{z} \\) with \\( \\bbz \\neq \\bm{0} \\).\n\t\tHence, by {\\cite[Theorem 3.2]{LiLoPo21}}, the obtained H\\\"olderian error bound with exponent \\( 1 / 2 \\) is the best error bound.\n","start":76901,"end":77233}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"坐标面分支的仿射误差界","source_units":[{"raw_tex":"\n\t\tIf \\( \\stdFace = \\Fhatz \\) with \\( \\bm{z} \\in \\dgK \\cap \\subSpace^{\\perp } \\cap \\{\\bm{a}\\}^{\\perp } \\), then using Corollary \\ref{corollary:1-FRFs-power-cones}(ii), we conclude that a H\\\"olderian error bound with exponent \\( \\beta := \\sum _{i \\in \\stdIndex} \\alpha _i \\) holds, where \\( \\stdIndex = \\{i\\,| \\,\\tz_i > 0\\} \\).\n","start":77233,"end":77560}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"坐标面分支的最优性","source_units":[{"raw_tex":"\t\tThe optimality of this error bound comes from Remark \\ref{remark:F-z-order-beta} and {\\cite[Theorem 3.2]{LiLoPo21}}.\n","start":77560,"end":77679}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"零面分支的线性界","source_units":[{"raw_tex":"If \\( \\stdFace = \\{\\bm{0}\\} \\), which means the feasible region is \\( \\{\\bm{0}\\} \\), then a Lipschitzian error bound holds automatically and it is naturally tight, see \\cite[Proposition~27]{L17}.\n\t\\end{proof}","start":77679,"end":77887}]},{"id":"S073","semantic_type":"assumption","kind":"assumption","labels":["item:error-bound-power-cone-1","item:error-bound-power-cone-2","item:error-bound-power-cone-3","item:error-bound-power-cone-3-1","item:error-bound-power-cone-3-2","item:error-bound-power-cone-3-3","item:error-bound-power-cone-4","thm:error-bound-power-cone-optimality"],"declared_external":false,"source_only":false,"title":"主问题的仿射可行性","source_units":[{"raw_tex":"\t\tConsider \\( \\gK \\) and its dual cone \\( \\dgK \\).\n\t\tLet \\( \\subSpace \\subseteq \\R^{m + n} \\) be a subspace and \\( \\bm{a} \\in \\R^{m+n} \\) be given.\n\t\tSuppose that \\( ( \\subSpace + \\bm{a} ) \\cap \\gK \\neq \\emptyset \\).\n","start":74505,"end":74722}]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明前提 LiLoPo20, Lemma 3.12；活动引用见 L532、L741。"],"source_only":false,"title":"零面常数的极限序列准则","source_units":[]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明前提 LiLoPo20, Theorem 3.10；活动引用见 L741、L834、L876。"],"source_only":false,"title":"正面常数的误差界转换","source_units":[]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明前提 LiLoPo20, Lemma 3.9；活动引用见 L904。"],"source_only":false,"title":"超平面误差界转为FRF","source_units":[]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明前提 L17, Definition 3、Proposition 5，以及 L327–329 指向的框架；完整原文未提供。"],"source_only":false,"title":"PPS条件与面约化链","source_units":[]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明前提 BBL99, Corollary 3；活动引用见 L327 的脚注和 L945。"],"source_only":false,"title":"PPS下的线性误差界","source_units":[]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明前提 LiLoPo21, Definition 3.1、Theorem 3.2，特别是 (b)；活动引用见 L804、L901、L936、L952、L955。"],"source_only":false,"title":"渐近最优性判据及推论","source_units":[]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["声明前提 L17, Proposition 27；活动引用见 L956。"],"source_only":false,"title":"原点单点可行域误差界","source_units":[]},{"id":"S081","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原声明标准前提中的 Taylor estimates；implicit standard fact made explicit by annotator。"],"source_only":false,"title":"局部Taylor估计","source_units":[]},{"id":"S082","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原声明标准前提中的 strict concavity of the logarithm；原文 L398、L458 明确使用。"],"source_only":false,"title":"对数严格凹性及等号条件","source_units":[]},{"id":"S083","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原声明标准前提中的 compactness；implicit standard fact made explicit by annotator。"],"source_only":false,"title":"紧性与正下界论证","source_units":[]},{"id":"S084","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原声明标准前提中的 Cauchy–Schwarz；L799 明确命名，其他使用由标注者显式指出。"],"source_only":false,"title":"柯西不等式及等号条件","source_units":[]},{"id":"S085","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原声明标准前提中的 finite-dimensional convex projection properties；implicit standard fact made explicit by annotator。"],"source_only":false,"title":"闭凸投影基本性质","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S063","edges":[{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S011","target_id":"S010","kind":"supports"},{"source_id":"S002","target_id":"S006","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S077","target_id":"S007","kind":"supports"},{"source_id":"S078","target_id":"S007","kind":"supports"},{"source_id":"S001","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S081","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S010","target_id":"S017","kind":"supports"},{"source_id":"S082","target_id":"S017","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S016","target_id":"S014","kind":"supports"},{"source_id":"S017","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S012","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S010","target_id":"S022","kind":"supports"},{"source_id":"S084","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S082","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S084","target_id":"S024","kind":"supports"},{"source_id":"S010","target_id":"S025","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S023","target_id":"S021","kind":"supports"},{"source_id":"S024","target_id":"S021","kind":"supports"},{"source_id":"S025","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S074","target_id":"S028","kind":"supports"},{"source_id":"S010","target_id":"S029","kind":"supports"},{"source_id":"S011","target_id":"S029","kind":"supports"},{"source_id":"S008","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S028","target_id":"S030","kind":"supports"},{"source_id":"S085","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S008","target_id":"S031","kind":"supports"},{"source_id":"S013","target_id":"S031","kind":"supports"},{"source_id":"S019","target_id":"S031","kind":"supports"},{"source_id":"S008","target_id":"S032","kind":"supports"},{"source_id":"S019","target_id":"S032","kind":"supports"},{"source_id":"S029","target_id":"S032","kind":"supports"},{"source_id":"S085","target_id":"S032","kind":"supports"},{"source_id":"S018","target_id":"S033","kind":"supports"},{"source_id":"S029","target_id":"S033","kind":"supports"},{"source_id":"S083","target_id":"S033","kind":"supports"},{"source_id":"S012","target_id":"S034","kind":"supports"},{"source_id":"S029","target_id":"S034","kind":"supports"},{"source_id":"S013","target_id":"S036","kind":"supports"},{"source_id":"S029","target_id":"S036","kind":"supports"},{"source_id":"S083","target_id":"S036","kind":"supports"},{"source_id":"S031","target_id":"S037","kind":"supports"},{"source_id":"S032","target_id":"S037","kind":"supports"},{"source_id":"S033","target_id":"S037","kind":"supports"},{"source_id":"S034","target_id":"S037","kind":"supports"},{"source_id":"S028","target_id":"S037","kind":"supports"},{"source_id":"S031","target_id":"S038","kind":"supports"},{"source_id":"S032","target_id":"S038","kind":"supports"},{"source_id":"S033","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S028","target_id":"S038","kind":"supports"},{"source_id":"S085","target_id":"S038","kind":"supports"},{"source_id":"S019","target_id":"S039","kind":"supports"},{"source_id":"S029","target_id":"S039","kind":"supports"},{"source_id":"S031","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S029","target_id":"S040","kind":"supports"},{"source_id":"S018","target_id":"S040","kind":"supports"},{"source_id":"S013","target_id":"S040","kind":"supports"},{"source_id":"S083","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S008","target_id":"S041","kind":"supports"},{"source_id":"S031","target_id":"S041","kind":"supports"},{"source_id":"S028","target_id":"S041","kind":"supports"},{"source_id":"S085","target_id":"S041","kind":"supports"},{"source_id":"S030","target_id":"S042","kind":"supports"},{"source_id":"S037","target_id":"S042","kind":"supports"},{"source_id":"S038","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S074","target_id":"S042","kind":"supports"},{"source_id":"S075","target_id":"S042","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S027","kind":"supports"},{"source_id":"S031","target_id":"S027","kind":"supports"},{"source_id":"S032","target_id":"S027","kind":"supports"},{"source_id":"S033","target_id":"S027","kind":"supports"},{"source_id":"S034","target_id":"S027","kind":"supports"},{"source_id":"S036","target_id":"S027","kind":"supports"},{"source_id":"S037","target_id":"S027","kind":"supports"},{"source_id":"S038","target_id":"S027","kind":"supports"},{"source_id":"S039","target_id":"S027","kind":"supports"},{"source_id":"S040","target_id":"S027","kind":"supports"},{"source_id":"S041","target_id":"S027","kind":"supports"},{"source_id":"S042","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S019","target_id":"S045","kind":"supports"},{"source_id":"S009","target_id":"S045","kind":"supports"},{"source_id":"S010","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S010","target_id":"S046","kind":"supports"},{"source_id":"S081","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S047","kind":"supports"},{"source_id":"S019","target_id":"S047","kind":"supports"},{"source_id":"S008","target_id":"S047","kind":"supports"},{"source_id":"S084","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S079","target_id":"S048","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S048","target_id":"S043","kind":"supports"},{"source_id":"S010","target_id":"S052","kind":"supports"},{"source_id":"S085","target_id":"S052","kind":"supports"},{"source_id":"S075","target_id":"S052","kind":"supports"},{"source_id":"S020","target_id":"S053","kind":"supports"},{"source_id":"S085","target_id":"S053","kind":"supports"},{"source_id":"S084","target_id":"S053","kind":"supports"},{"source_id":"S010","target_id":"S054","kind":"supports"},{"source_id":"S009","target_id":"S054","kind":"supports"},{"source_id":"S052","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S075","target_id":"S055","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S050","kind":"supports"},{"source_id":"S054","target_id":"S050","kind":"supports"},{"source_id":"S055","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S020","target_id":"S058","kind":"supports"},{"source_id":"S009","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S010","target_id":"S059","kind":"supports"},{"source_id":"S020","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S059","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S079","target_id":"S056","kind":"supports"},{"source_id":"S026","target_id":"S062","kind":"supports"},{"source_id":"S049","target_id":"S062","kind":"supports"},{"source_id":"S076","target_id":"S062","kind":"supports"},{"source_id":"S062","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S006","target_id":"S060","kind":"supports"},{"source_id":"S026","target_id":"S060","kind":"supports"},{"source_id":"S006","target_id":"S061","kind":"supports"},{"source_id":"S049","target_id":"S061","kind":"supports"},{"source_id":"S001","target_id":"S073","kind":"supports"},{"source_id":"S010","target_id":"S073","kind":"supports"},{"source_id":"S019","target_id":"S065","kind":"supports"},{"source_id":"S020","target_id":"S065","kind":"supports"},{"source_id":"S007","target_id":"S065","kind":"supports"},{"source_id":"S077","target_id":"S065","kind":"supports"},{"source_id":"S073","target_id":"S065","kind":"supports"},{"source_id":"S007","target_id":"S066","kind":"supports"},{"source_id":"S078","target_id":"S066","kind":"supports"},{"source_id":"S073","target_id":"S066","kind":"supports"},{"source_id":"S007","target_id":"S067","kind":"supports"},{"source_id":"S077","target_id":"S067","kind":"supports"},{"source_id":"S073","target_id":"S067","kind":"supports"},{"source_id":"S060","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S007","target_id":"S068","kind":"supports"},{"source_id":"S043","target_id":"S069","kind":"supports"},{"source_id":"S079","target_id":"S069","kind":"supports"},{"source_id":"S061","target_id":"S070","kind":"supports"},{"source_id":"S067","target_id":"S070","kind":"supports"},{"source_id":"S007","target_id":"S070","kind":"supports"},{"source_id":"S056","target_id":"S071","kind":"supports"},{"source_id":"S079","target_id":"S071","kind":"supports"},{"source_id":"S067","target_id":"S072","kind":"supports"},{"source_id":"S073","target_id":"S072","kind":"supports"},{"source_id":"S080","target_id":"S072","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S064","kind":"supports"},{"source_id":"S067","target_id":"S064","kind":"supports"},{"source_id":"S068","target_id":"S064","kind":"supports"},{"source_id":"S069","target_id":"S064","kind":"supports"},{"source_id":"S070","target_id":"S064","kind":"supports"},{"source_id":"S071","target_id":"S064","kind":"supports"},{"source_id":"S072","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S063","kind":"supports"},{"source_id":"S003","target_id":"S063","kind":"supports"},{"source_id":"S007","target_id":"S063","kind":"supports"},{"source_id":"S019","target_id":"S063","kind":"supports"},{"source_id":"S020","target_id":"S063","kind":"supports"},{"source_id":"S009","target_id":"S063","kind":"supports"},{"source_id":"S010","target_id":"S063","kind":"supports"},{"source_id":"S079","target_id":"S063","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S063","edges":[{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S011","target_id":"S010","kind":"supports"},{"source_id":"S002","target_id":"S006","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S077","target_id":"S007","kind":"supports"},{"source_id":"S078","target_id":"S007","kind":"supports"},{"source_id":"S001","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S081","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S010","target_id":"S017","kind":"supports"},{"source_id":"S082","target_id":"S017","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S016","target_id":"S014","kind":"supports"},{"source_id":"S017","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S012","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S010","target_id":"S022","kind":"supports"},{"source_id":"S084","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S082","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S084","target_id":"S024","kind":"supports"},{"source_id":"S010","target_id":"S025","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S023","target_id":"S021","kind":"supports"},{"source_id":"S024","target_id":"S021","kind":"supports"},{"source_id":"S025","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S074","target_id":"S028","kind":"supports"},{"source_id":"S010","target_id":"S029","kind":"supports"},{"source_id":"S011","target_id":"S029","kind":"supports"},{"source_id":"S008","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S028","target_id":"S030","kind":"supports"},{"source_id":"S085","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S008","target_id":"S031","kind":"supports"},{"source_id":"S013","target_id":"S031","kind":"supports"},{"source_id":"S019","target_id":"S031","kind":"supports"},{"source_id":"S008","target_id":"S032","kind":"supports"},{"source_id":"S019","target_id":"S032","kind":"supports"},{"source_id":"S029","target_id":"S032","kind":"supports"},{"source_id":"S085","target_id":"S032","kind":"supports"},{"source_id":"S018","target_id":"S033","kind":"supports"},{"source_id":"S029","target_id":"S033","kind":"supports"},{"source_id":"S083","target_id":"S033","kind":"supports"},{"source_id":"S012","target_id":"S034","kind":"supports"},{"source_id":"S029","target_id":"S034","kind":"supports"},{"source_id":"S013","target_id":"S036","kind":"supports"},{"source_id":"S029","target_id":"S036","kind":"supports"},{"source_id":"S083","target_id":"S036","kind":"supports"},{"source_id":"S031","target_id":"S037","kind":"supports"},{"source_id":"S032","target_id":"S037","kind":"supports"},{"source_id":"S033","target_id":"S037","kind":"supports"},{"source_id":"S034","target_id":"S037","kind":"supports"},{"source_id":"S028","target_id":"S037","kind":"supports"},{"source_id":"S031","target_id":"S038","kind":"supports"},{"source_id":"S032","target_id":"S038","kind":"supports"},{"source_id":"S033","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S028","target_id":"S038","kind":"supports"},{"source_id":"S085","target_id":"S038","kind":"supports"},{"source_id":"S019","target_id":"S039","kind":"supports"},{"source_id":"S029","target_id":"S039","kind":"supports"},{"source_id":"S031","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S029","target_id":"S040","kind":"supports"},{"source_id":"S018","target_id":"S040","kind":"supports"},{"source_id":"S013","target_id":"S040","kind":"supports"},{"source_id":"S083","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S008","target_id":"S041","kind":"supports"},{"source_id":"S031","target_id":"S041","kind":"supports"},{"source_id":"S028","target_id":"S041","kind":"supports"},{"source_id":"S085","target_id":"S041","kind":"supports"},{"source_id":"S030","target_id":"S042","kind":"supports"},{"source_id":"S037","target_id":"S042","kind":"supports"},{"source_id":"S038","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S074","target_id":"S042","kind":"supports"},{"source_id":"S075","target_id":"S042","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S027","kind":"supports"},{"source_id":"S031","target_id":"S027","kind":"supports"},{"source_id":"S032","target_id":"S027","kind":"supports"},{"source_id":"S033","target_id":"S027","kind":"supports"},{"source_id":"S034","target_id":"S027","kind":"supports"},{"source_id":"S036","target_id":"S027","kind":"supports"},{"source_id":"S037","target_id":"S027","kind":"supports"},{"source_id":"S038","target_id":"S027","kind":"supports"},{"source_id":"S039","target_id":"S027","kind":"supports"},{"source_id":"S040","target_id":"S027","kind":"supports"},{"source_id":"S041","target_id":"S027","kind":"supports"},{"source_id":"S042","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S019","target_id":"S045","kind":"supports"},{"source_id":"S009","target_id":"S045","kind":"supports"},{"source_id":"S010","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S010","target_id":"S046","kind":"supports"},{"source_id":"S081","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S047","kind":"supports"},{"source_id":"S019","target_id":"S047","kind":"supports"},{"source_id":"S008","target_id":"S047","kind":"supports"},{"source_id":"S084","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S079","target_id":"S048","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S048","target_id":"S043","kind":"supports"},{"source_id":"S010","target_id":"S052","kind":"supports"},{"source_id":"S085","target_id":"S052","kind":"supports"},{"source_id":"S075","target_id":"S052","kind":"supports"},{"source_id":"S020","target_id":"S053","kind":"supports"},{"source_id":"S085","target_id":"S053","kind":"supports"},{"source_id":"S084","target_id":"S053","kind":"supports"},{"source_id":"S010","target_id":"S054","kind":"supports"},{"source_id":"S009","target_id":"S054","kind":"supports"},{"source_id":"S052","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S075","target_id":"S055","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S050","kind":"supports"},{"source_id":"S054","target_id":"S050","kind":"supports"},{"source_id":"S055","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S020","target_id":"S058","kind":"supports"},{"source_id":"S009","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S010","target_id":"S059","kind":"supports"},{"source_id":"S020","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S059","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S079","target_id":"S056","kind":"supports"},{"source_id":"S026","target_id":"S062","kind":"supports"},{"source_id":"S049","target_id":"S062","kind":"supports"},{"source_id":"S076","target_id":"S062","kind":"supports"},{"source_id":"S062","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S006","target_id":"S060","kind":"supports"},{"source_id":"S026","target_id":"S060","kind":"supports"},{"source_id":"S006","target_id":"S061","kind":"supports"},{"source_id":"S049","target_id":"S061","kind":"supports"},{"source_id":"S001","target_id":"S073","kind":"supports"},{"source_id":"S010","target_id":"S073","kind":"supports"},{"source_id":"S019","target_id":"S065","kind":"supports"},{"source_id":"S020","target_id":"S065","kind":"supports"},{"source_id":"S007","target_id":"S065","kind":"supports"},{"source_id":"S077","target_id":"S065","kind":"supports"},{"source_id":"S073","target_id":"S065","kind":"supports"},{"source_id":"S007","target_id":"S066","kind":"supports"},{"source_id":"S078","target_id":"S066","kind":"supports"},{"source_id":"S073","target_id":"S066","kind":"supports"},{"source_id":"S007","target_id":"S067","kind":"supports"},{"source_id":"S077","target_id":"S067","kind":"supports"},{"source_id":"S073","target_id":"S067","kind":"supports"},{"source_id":"S060","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S007","target_id":"S068","kind":"supports"},{"source_id":"S043","target_id":"S069","kind":"supports"},{"source_id":"S079","target_id":"S069","kind":"supports"},{"source_id":"S061","target_id":"S070","kind":"supports"},{"source_id":"S067","target_id":"S070","kind":"supports"},{"source_id":"S007","target_id":"S070","kind":"supports"},{"source_id":"S056","target_id":"S071","kind":"supports"},{"source_id":"S079","target_id":"S071","kind":"supports"},{"source_id":"S067","target_id":"S072","kind":"supports"},{"source_id":"S073","target_id":"S072","kind":"supports"},{"source_id":"S080","target_id":"S072","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S064","kind":"supports"},{"source_id":"S067","target_id":"S064","kind":"supports"},{"source_id":"S068","target_id":"S064","kind":"supports"},{"source_id":"S069","target_id":"S064","kind":"supports"},{"source_id":"S070","target_id":"S064","kind":"supports"},{"source_id":"S071","target_id":"S064","kind":"supports"},{"source_id":"S072","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S063","kind":"supports"},{"source_id":"S003","target_id":"S063","kind":"supports"},{"source_id":"S007","target_id":"S063","kind":"supports"},{"source_id":"S019","target_id":"S063","kind":"supports"},{"source_id":"S020","target_id":"S063","kind":"supports"},{"source_id":"S009","target_id":"S063","kind":"supports"},{"source_id":"S010","target_id":"S063","kind":"supports"},{"source_id":"S079","target_id":"S063","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:conic-feasibility-problem"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"锥可行性问题","source_units":[{"raw_tex":"\tIn a Euclidean space \\( \\EucSpace \\), consider the conic feasibility problem:\n\t\\begin{equation}\n\t\t\\label{eq:conic-feasibility-problem}\\tag{Feas}\n\t\t\\text{find } \\quad \\bm{x} \\in (\\subSpace + \\bm{a}) \\cap \\stdCone,\n\t\\end{equation}\n\twhere \\( \\subSpace \\) is a subspace, \\( \\bm{a} \\in \\EucSpace \\), and \\( \\stdCone\\) is a closed convex cone.\n","start":7572,"end":7911}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"欧氏几何记号","source_units":[{"raw_tex":"\tWe will use plain letters to represent real scalars, bold lowercase letters to denote vectors, bold uppercase letters to stand for matrices,\\footnote{With an abuse of notation, we use $\\bm 0$ to denote a zero vector / matrix, whose dimension should be clear from the context.} and curly capital letters for (sub)spaces and sets. Let \\( \\EucSpace \\) be a finite dimensional Euclidean space, \\( \\R_+ \\) and \\( \\R_- \\) be the set of nonnegative and nonpositive real numbers, respectively.\n\tThe inner product of \\( \\EucSpace \\) is denoted by \\( \\inProd{ \\cdot }{ \\cdot  } \\) and the induced norm by \\( \\| \\cdot \\| \\).\n\tWith that, for \\( \\bm{x} \\in \\EucSpace \\) and a closed convex set \\( \\C \\subseteq \\EucSpace \\), we denote the projection of \\( \\bm{x} \\) onto \\( \\C \\) by \\( P_\\C(\\bm{x}) \\) so that \\( P_\\C(\\bm{x}) = \\argmin_{\\bm{y} \\in \\C} \\| \\bm{x} - \\bm{y} \\| \\) and the distance between \\( \\bm{x} \\) and \\( \\C \\) by \\( \\dist(\\bm{x}, \\C) = \\inf_{\\bm{y} \\in \\C} \\| \\bm{x} - \\bm{y} \\| = \\| \\bm{x} - P_\\C(\\bm{x}) \\| \\).\n\tFor any \\( \\bm{x} \\in \\EucSpace \\) and \\( \\eta \\geq 0 \\), we denote the ball centered at \\( \\bm{x} \\) with radius \\( \\eta  \\) by \\( \\B(\\bm{x}; \\eta) := \\{\\bm{y} \\in \\EucSpace \\,|\\,\\| \\bm{y} - \\bm{x} \\| \\leq \\eta \\} \\); we write \\( \\B(\\eta) \\) for the ball centered at \\( \\bm{0} \\) with radius \\( \\eta \\) for simplicity.\n\tA diagonal matrix with diagonal vector being \\( \\bm{x} \\) is denoted by \\( \\diag(\\bm{x}) \\).\n\tMeanwhile, we use \\( \\C^{\\perp } \\) to denote the orthogonal complement of \\( \\C \\) and \\( \\bm{I}_n \\) to represent the \\( n \\times n \\) identity matrix.\n\n","start":14665,"end":16254}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"一致霍尔德误差界","source_units":[{"raw_tex":"\t{\n\t  We now recall the definition of {Lipschitzian and H\\\"olderian error bounds}.\n\t  Let \\( \\C_1, \\C_2 \\subseteq \\EucSpace \\) be closed convex sets with \\( \\C_1 \\cap \\C_2 \\neq \\emptyset  \\).\n\t  We say that \\( \\C_1, \\C_2 \\) satisfy a \\emph{uniform H\\\"olderian error bound} with exponent \\( \\gamma  \\in (0, 1] \\) if for every bounded set \\( \\B \\subseteq \\EucSpace \\), there exists a constant \\( \\kappa _\\B \\) such that\n\t  $\\dist(\\bm{x}, \\C_1 \\cap \\C_2) \\!\\leq\\! \\kappa _\\B \\max\\left\\{\\dist(\\bm{x}, \\C_1), \\dist(\\bm{x}, \\C_2)\\right\\}^{\\gamma }$ for all $\\bm{x} \\in \\B$.\n\t  If \\( \\gamma  = 1 \\), then the error bound is said to be \\emph{Lipschitzian}.\n\t}\n\n","start":16254,"end":16907}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"锥与暴露面","source_units":[{"raw_tex":"\tLet \\( \\stdCone \\subseteq \\EucSpace \\) be a closed convex cone and \\( \\stdCone^{*} \\) be its dual cone.\n\tWe will use ${\\rm int}\\,\\stdCone $, \\( \\ri \\stdCone \\), \\( \\bd \\stdCone \\), \\( \\Span \\stdCone \\), \\( \\dim \\stdCone \\) to denote the interior, relative interior, boundary, linear span and dimension of \\( \\stdCone \\), respectively.\n\tIf \\( \\stdCone \\cap -\\stdCone = \\{\\bm{0}\\} \\), we say that \\( \\stdCone \\) is \\textit{pointed}.\t\n\n\tA \\textit{face} of \\( \\stdCone \\) is a closed convex cone \\( \\stdFace \\) satisfying \\( \\stdFace \\subseteq \\stdCone \\) and the property that if \\( \\bm{x}, \\bm{y} \\in \\stdCone\\) and \\( \\bm{x} + \\bm{y} \\in \\stdFace \\), then \\( \\bm{x}, \\bm{y} \\in \\stdFace \\).\\footnote{By convention, we discard the empty face.}\n\tWe write \\( \\stdFace \\unlhd \\stdCone \\) if \\( \\stdFace \\) is a face of \\( \\stdCone \\) and \\( \\stdFace \\properideal \\stdCone \\) if \\( \\stdFace \\) is a \\textit{proper} face of \\( \\stdCone \\), i.e., \\( \\stdFace \\neq \\stdCone \\).\n\tA face \\( \\stdFace \\) is said to be \\textit{nontrivial} if \\( \\stdFace \\) is proper and \\( \\stdFace \\neq \\stdCone \\cap -\\stdCone \\).\n\tIf \\( \\stdFace = \\stdCone \\cap \\{\\bm{z}\\}^{\\perp } \\) for some \\( \\bm{z} \\in \\stdCone^{*} \\), \\( \\stdFace \\) is called an \\textit{exposed face} of \\( \\stdCone \\).\n","start":16907,"end":18175}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":["def:one-step-facial-residual-function"],"member_ids":["S006"],"declared_external":true,"declaration_text":["原文明确引用 LiLoPo20, Definition 3.4；该定义在所选原文中完整重述。"],"source_only":false,"title":"一步面残差函数","source_units":[{"raw_tex":"\t  \\begin{definition}[One-step facial residual function {\\cite[Definition 3.4]{LiLoPo20}}]\\label{def:one-step-facial-residual-function}\n\t  \tLet \\( \\mathcal{K} \\) be a closed convex cone and \\( \\bm{z} \\in \\mathcal{K}^{*} \\).\n\t  \tSuppose that \\( \\psi _{\\mathcal{K}, \\bm{z}}: \\R_+ \\times \\R_+ \\to \\R_+ \\) satisfies:% the following properties:\n\t  \t\\begin{enumerate}[label={(\\roman*)}]\n\t  \t\t\\item \\( \\psi _{\\mathcal{K}, \\bm{z}} \\) is nonnegative,  nondecreasing in each argument and for every \\( t \\in \\R_+ \\), \\( \\psi _{\\mathcal{K}, \\bm{z}} (0, t) = 0 \\).\n\t  \t\t\\item The following implication holds for any \\( \\bm{x} \\in \\Span \\mathcal{K} \\) and \\( \\epsilon \\geq 0 \\):\\vspace{-0.15 cm}\n\t  \t\t\\[ \\vspace{-0.15 cm}\n\t  \t\t\\dist(\\bm{x}, \\mathcal{K}) \\leq \\epsilon ,\\,\\inProd{ \\bm{x}}{ \\bm{z} } \\leq \\epsilon \\implies \\dist(\\bm{x}, \\mathcal{K} \\cap \\{\\bm{z}\\}^{\\perp }) \\leq \\psi _{\\mathcal{K}, \\bm{z}} (\\epsilon, \\| \\bm{x} \\|).\n\t  \t\t\\]\n\t  \t\\end{enumerate}\n\t  \tThen, \\( \\psi _{\\mathcal{K}, \\bm{z}} \\) is said to be a one-step facial residual function (\\( \\mathds{1} \\)-FRF) for \\( \\mathcal{K} \\) and \\( \\bm{z} \\).\n\t  \\end{definition}\n","start":18996,"end":20119}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"PPS距离与复合框架","source_units":[{"raw_tex":"\t  The basic idea of the  aforementioned framework  is as follows. Suppose that in each step of the facial reduction algorithm we can find a suitable one-step facial residual function for the ``current'' face and ``next'' exposing vector until we reach a face $\\stdFace$ such that $\\stdFace$ and $\\subSpace + \\bm{a}$ satisfy the \\textit{partial polyhedral Slater's (PPS) condition} \\cite[Definition 3]{L17}.\\footnote{ We note that this implies a Lipschitzian error bound holds for $\\stdFace$ and $\\subSpace + \\bm{a}$, see \\cite[Corollary~3]{BBL99} and the discussion preceding \\cite[Proposition~2.3]{LiLoPo20}.}\n\t  Then, we can  construct an error bound for $\\stdCone$ and $\\subSpace + \\bm{a}$ by composing these residual functions in a specific manner.\n\t  In this regard, if \\eqref{eq:conic-feasibility-problem} is feasible, we define the \\textit{distance to the PPS condition} of \\eqref{eq:conic-feasibility-problem}, denoted by $ \\dpps(\\stdCone, \\subSpace + \\bm{a}) $, as the length \\textit{minus one} of the \\textit{shortest} chain of faces (among those chains constructed as in \\cite[Proposition~5]{L17}) such that the PPS condition holds for the final face in the chain and $\\subSpace + \\bm{a}$.\n\t}\n","start":20119,"end":21324}]},{"id":"S008","semantic_type":"external_fact","kind":"external_fact","labels":["lemma:Lemma-2.11"],"member_ids":["S008"],"declared_external":true,"declaration_text":["所声明允许前提 LiLoPo21, Lemma 2.5；原文 B0007 完整重述，未附证明。"],"source_only":false,"title":"一维面的投影公式","source_units":[{"raw_tex":"\t{\n\t  We end this section with the following lemma, which is useful in the analysis of one-dimensional faces. It will be used repeatedly in our subsequent discussions.\n\t  \\begin{lemma}[{\\cite[Lemma 2.5]{LiLoPo21}}]\n\t\t\\label{lemma:Lemma-2.11}\n\t\tLet \\( \\stdCone \\) be a pointed closed convex cone and let \\( \\bm{z}\\in \\partial \\stdCone^*\\setminus \\{{\\bm 0}\\} \\) be such that \\( \\stdFace := \\{\\bm{z}\\}^{\\perp} \\cap \\stdCone \\) is a one-dimensional proper face of \\( \\stdCone \\).\n\t\tLet \\( \\bm{f} \\in \\stdCone \\setminus \\{\\bm{0}\\} \\) be such that \\( \\stdFace = \\{t \\bm{f} \\, | \\, t \\geq 0\\} \\).\n\t\tLet \\( \\eta > 0 \\) and \\( \\bm{v} \\in \\partial \\stdCone \\cap B(\\eta) \\setminus \\stdFace \\), \\( \\bm{w} = P_{\\{\\bm{z}\\}^{\\perp}}(\\bm{v}) \\) and \\( \\bm{u} = P_{\\stdFace}(\\bm{w}) \\) with \\( \\bm{u} \\neq \\bm{w} \\).\n\t\tThen it holds that \\( \\left\\langle \\bm{f}, \\bm{z} \\right\\rangle = 0 \\) and we have\n\t\t\\[\n\t\t  \\| \\bm{v} - \\bm{w} \\| = \\frac{| \\left\\langle \\bm{z}, \\bm{v} \\right\\rangle |}{\\| \\bm{z} \\|}, \\qquad \\| \\bm{u} - \\bm{w} \\| =\n\t\t  \\begin{cases}\n\t\t\t\\left\\| \\bm{v} - \\frac{\\left\\langle \\bm{z}, \\bm{v} \\right\\rangle}{\\| \\bm{z} \\|^2}\\bm{z} - \\frac{\\left\\langle \\bm{f}, \\bm{v} \\right\\rangle}{\\| \\bm{f} \\|^2}\\bm{f} \\right\\| & \\text{ if } \\left\\langle \\bm{f}, \\bm{v} \\right\\rangle \\geq 0, \\\\\n\t\t\t\\,\\,\\,\\left\\| \\bm{v} - \\frac{\\left\\langle \\bm{z}, \\bm{v} \\right\\rangle}{\\| \\bm{z} \\|^2} \\bm{z} \\right\\| & \\text{ otherwise }.\n\t\t  \\end{cases}\n\t\t\\]\n\t\tMoreover, when \\( \\left\\langle \\bm{f}, \\bm{v} \\right\\rangle \\geq 0 \\) (or, equivalently, \\( \\left\\langle \\bm{f}, \\bm{w} \\right\\rangle \\geq 0 \\)), we have \\( \\bm{u} = P_{\\Span \\stdFace}(\\bm{w}) \\).\n\t\tOn the other hand, if \\( \\left\\langle \\bm{f}, \\bm{v} \\right\\rangle < 0 \\), we have \\( \\bm{u} = \\bm{0} \\).\n\t  \\end{lemma}\n\t}\n","start":21325,"end":23075}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"广义幂锥参数","source_units":[{"raw_tex":"\tWe consider the generalized power cone and its dual.\n\tLet \\( m\\ge 1 \\), \\( n\\ge 2 \\) and \\( \\bm{\\alpha} = (\\alpha_1,\\ldots,\\alpha_n)\\in (0,1)^n \\) with \\( \\sum_{i=1}^n\\alpha_i = 1 \\), the generalized power cone \\( \\gK \\) and its dual \\( \\dgK \\) are given respectively by\n","start":23165,"end":23437}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":["eq:gpowercones"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"广义幂锥及对偶","source_units":[{"raw_tex":"\t\\begin{equation}\\label{eq:gpowercones}\n\t\\begin{aligned}\n\t\\gK  & = \\left\\{\\bm{x} = (\\bbx,\\tbx)\\in \\R^{m+n}\\,\\bigg| \\,\\|\\bbx\\|\\le \\prod_{i=1}^n\\tx_i^{\\alpha_i},\\,\\bbx\\in \\R^m,\\,\\tbx\\in \\R^n_+\\right\\},                               \\\\\n\t\\dgK & = \\left\\{\\bm{z} = (\\bbz,\\tbz)\\in \\R^{m+n}\\,\\bigg| \\,\\|\\bbz\\|\\le \\prod_{i=1}^n\\left(\\frac{\\tz_i}{\\alpha_i}\\right)^{\\alpha_i},\\,\\bbz\\in \\R^m,\\,\\tbz\\in \\R^n_+\\right\\}.\n\t\\end{aligned}\n\t\\end{equation}\n","start":23437,"end":23874}]},{"id":"S011","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"向量分块","source_units":[{"raw_tex":"\tHere, given a vector \\( \\bm{x}\\in \\R^{m+n} \\), we let \\( \\bbx\\in \\R^m \\) be the vector corresponding to its first \\( m \\) entries and \\( \\tbx\\in \\R^n \\) be the vector corresponding to its last \\( n \\) entries.\n","start":23874,"end":24085}]},{"id":"S012","semantic_type":"lemma","kind":"lemma","labels":["eq:inequ-inprod-power-cone","lemma:inequ-inprod-power-cone"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"乘积曲面的局部二次间隙","source_units":[{"raw_tex":"\\begin{lemma}\\label{lemma:inequ-inprod-power-cone}\n\t\tLet \\( n\\ge 2 \\) and \\( \\bm{\\alpha} = (\\alpha_1,\\ldots,\\alpha_n)\\in (0,1)^n \\) with \\( \\sum_{i=1}^n\\alpha_i = 1 \\). Let \\( \\bm{\\zeta}\\in {\\rm int}\\,\\R_{-}^n \\) satisfy \\( \\prod_{i=1}^n (-\\zeta_i/\\alpha_i)^{\\alpha_i} = 1 \\). Define \\( \\widetilde{\\bm{\\zeta}} := -\\bm\\alpha\\circ \\bm{\\zeta}^{-1} \\), where \\( \\circ \\) is the Hadamard product and the inverse is taken componentwise. Then there exist \\( C > 0 \\) and \\( \\epsilon > 0 \\) so that\n\t\t\\begin{equation}\\label{eq:inequ-inprod-power-cone}\n\t\t\t-1 - \\inProd{\\bm{\\zeta}}{\\bm{\\omega}} \\ge C\\|\\bm{\\omega} - \\widetilde{\\bm{\\zeta}}\\|^2 \\,\\,{whenever}\\,\\bm{\\omega}\\in {\\rm int}\\,\\R^n_+,\\,\\|\\bm{\\omega} - \\widetilde{\\bm{\\zeta}}\\|\\le \\epsilon {\\,and\\,} \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1.\n\t\t\\end{equation}\n","start":24319,"end":25121}]},{"id":"S013","semantic_type":"lemma","kind":"lemma","labels":["lemma:inequ-inprod-power-cone"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"乘积曲面的唯一接触点","source_units":[{"raw_tex":"\\begin{lemma}\\label{lemma:inequ-inprod-power-cone}\n\t\tLet \\( n\\ge 2 \\) and \\( \\bm{\\alpha} = (\\alpha_1,\\ldots,\\alpha_n)\\in (0,1)^n \\) with \\( \\sum_{i=1}^n\\alpha_i = 1 \\). Let \\( \\bm{\\zeta}\\in {\\rm int}\\,\\R_{-}^n \\) satisfy \\( \\prod_{i=1}^n (-\\zeta_i/\\alpha_i)^{\\alpha_i} = 1 \\). Define \\( \\widetilde{\\bm{\\zeta}} := -\\bm\\alpha\\circ \\bm{\\zeta}^{-1} \\), where \\( \\circ \\) is the Hadamard product and the inverse is taken componentwise. Then there exist \\( C > 0 \\) and \\( \\epsilon > 0 \\) so that\n","start":24319,"end":24810},{"raw_tex":"\t\tMoreover, for any \\( \\bm{\\omega}\\in {\\rm int}\\,\\R_+^n \\) satisfying \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\), it holds that \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}\\le-1 \\); furthermore, we have \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}=-1 \\) if and only if \\( \\bm{\\omega} = \\widetilde{\\bm{\\zeta}} \\).\n\t\\end{lemma}","start":25121,"end":25428}]},{"id":"S014","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"乘积曲面引理完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\t\tFor each \\( i \\), we see from the Taylor series of \\( \\ln(\\cdot) \\) at \\( \\widetilde{\\zeta}_i > 0 \\) that\n\t\t\\begin{equation*}\n\t\t\t\\ln(\\omega_i) = \\ln(\\widetilde{\\zeta}_i) + \\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) - \\widetilde{\\zeta}_i^{-2}(\\omega_i - \\widetilde{\\zeta}_i)^2 + O(|\\omega_i - \\widetilde{\\zeta}_i|^3)\\,\\,{\\rm as}\\,\\,\\omega_i \\to \\widetilde{\\zeta}_i, \\,\\,\\omega_i > 0.\n\t\t\\end{equation*}\n\t\tThus, there exist \\( \\epsilon_i > 0 \\) and \\( c_i > 0 \\) so that\n\t\t\\[\n\t\t\\ln(\\widetilde{\\zeta}_i) \\ge \\ln(\\omega_i) - \\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) + c_i (\\omega_i - \\widetilde{\\zeta}_i)^2\\,\\,\\,\\mbox{whenever} \\,|\\omega_i - \\widetilde{\\zeta}_i|\\le \\epsilon_i \\,{\\rm and}\\,\\omega_i > 0.\n\t\t\\]\n\t\tLet \\( \\epsilon := \\min\\limits_{1\\le i\\le n}\\epsilon_i > 0 \\). Multiplying both sides of the above inequality by \\( \\alpha_i \\) and summing the resulting inequalities from \\( i = 1 \\) to \\( n \\), we see that whenever \\( \\bm{\\omega}\\in {\\rm int}\\,\\R^n_+ \\) satisfies \\( \\|\\bm{\\omega} - \\widetilde{\\bm{\\zeta}}\\|\\le \\epsilon \\) and \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\), we have\n\t\t\\begin{align}\n\t\t\t&0  \\overset{\\rm (a)}= \\sum_{i=1}^n\\alpha_i\\ln(\\widetilde{\\zeta}_i)\\ge \\sum_{i=1}^n\\alpha_i\\ln(\\omega_i) - \\sum_{i=1}^n\\alpha_i\\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) + \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2 \\notag\\\\\n\t\t\t& \\overset{\\rm (b)}= - \\sum_{i=1}^n\\alpha_i\\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) + \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2\n\t\t\t\\overset{\\rm (c)}= - \\sum_{i=1}^n\\alpha_i\\widetilde{\\zeta}_i^{-1}\\omega_i \\!+\\! 1 \\!+\\! \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2                                                                                                                          \\notag\\\\\n\t\t\t& = \\sum_{i=1}^n\\zeta_i\\omega_i + 1 + \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2,\\notag\n\t\t\\end{align}\n\t\twhere (a) and (b) hold because \\( \\prod_{i=1}^n\\widetilde{\\zeta}_i^{\\alpha_i} = \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\), (c) uses the fact that \\( \\sum_{i=1}^n\\alpha_i = 1 \\), and the last equality follows from the definition of \\( \\widetilde{\\bm{\\zeta}} \\). Rearranging the above inequality, we conclude that \\eqref{eq:inequ-inprod-power-cone} holds with \\( C = \\min\\limits_{1\\le i\\le n}\\alpha_ic_i > 0 \\).\n\n\t\tNext, let \\( \\bm{\\omega}\\in {\\rm int}\\,\\R_+^n \\) satisfy \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\). Then \\( (-1,\\bm{\\omega})\\in \\K \\). Recall from the assumption that \\( (1,-\\bm{\\zeta})\\in \\dK \\). From these we deduce \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}\\le-1 \\). If \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}=-1 \\), then\n\t\t\\[\n\t\t\\sum_{i=1}^n \\alpha_i\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)\\omega_i = \\sum_{i=1}^n (-\\zeta_i)\\omega_i = 1 = \\prod_{i=1}^n\\omega_i^{\\alpha_i} = \\prod_{i=1}^n\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)^{\\alpha_i}\\prod_{i=1}^n\\omega_i^{\\alpha_i} .\n\t\t\\]\n\t\tTaking \\( \\ln \\) on both sides of the above equality, we see that\n\t\t\\[\n\t\t\\ln\\left[\\sum_{i=1}^n \\alpha_i\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)\\omega_i\\right] = \\sum_{i=1}^n\\alpha_i \\ln\\left[\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)\\omega_i\\right].\n\t\t\\]\n\t\tSince \\( \\ln \\) is strictly concave and \\( \\alpha_i \\in (0,1) \\) for all \\( i \\), we conclude that there exists \\( c > 0 \\) so that \\( \\omega_i\\cdot(-\\zeta_i/\\alpha_i) = c \\) for all \\( i \\). This, together with the facts that\n\t\t\\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = \\prod_{i=1}^n (-\\zeta_i/\\alpha_i)^{\\alpha_i} = 1 \\) and \\( \\sum_{i=1}^n\\alpha_i = 1 \\), gives \\( c = 1 \\). It thus follows that \\( \\bm{\\omega} = \\widetilde{\\bm{\\zeta}} \\). Conversely, it is routine to check that if \\( \\bm{\\omega} = \\widetilde{\\bm{\\zeta}} \\), then \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\) and \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}=-1 \\).\n\t\\end{proof}","start":25430,"end":29267}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"对数局部二次估计","source_units":[{"raw_tex":"\\begin{proof}\n\t\tFor each \\( i \\), we see from the Taylor series of \\( \\ln(\\cdot) \\) at \\( \\widetilde{\\zeta}_i > 0 \\) that\n\t\t\\begin{equation*}\n\t\t\t\\ln(\\omega_i) = \\ln(\\widetilde{\\zeta}_i) + \\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) - \\widetilde{\\zeta}_i^{-2}(\\omega_i - \\widetilde{\\zeta}_i)^2 + O(|\\omega_i - \\widetilde{\\zeta}_i|^3)\\,\\,{\\rm as}\\,\\,\\omega_i \\to \\widetilde{\\zeta}_i, \\,\\,\\omega_i > 0.\n\t\t\\end{equation*}\n\t\tThus, there exist \\( \\epsilon_i > 0 \\) and \\( c_i > 0 \\) so that\n\t\t\\[\n\t\t\\ln(\\widetilde{\\zeta}_i) \\ge \\ln(\\omega_i) - \\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) + c_i (\\omega_i - \\widetilde{\\zeta}_i)^2\\,\\,\\,\\mbox{whenever} \\,|\\omega_i - \\widetilde{\\zeta}_i|\\le \\epsilon_i \\,{\\rm and}\\,\\omega_i > 0.\n\t\t\\]\n","start":25430,"end":26180}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"加权求和取得局部间隙","source_units":[{"raw_tex":"\t\tLet \\( \\epsilon := \\min\\limits_{1\\le i\\le n}\\epsilon_i > 0 \\). Multiplying both sides of the above inequality by \\( \\alpha_i \\) and summing the resulting inequalities from \\( i = 1 \\) to \\( n \\), we see that whenever \\( \\bm{\\omega}\\in {\\rm int}\\,\\R^n_+ \\) satisfies \\( \\|\\bm{\\omega} - \\widetilde{\\bm{\\zeta}}\\|\\le \\epsilon \\) and \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\), we have\n\t\t\\begin{align}\n\t\t\t&0  \\overset{\\rm (a)}= \\sum_{i=1}^n\\alpha_i\\ln(\\widetilde{\\zeta}_i)\\ge \\sum_{i=1}^n\\alpha_i\\ln(\\omega_i) - \\sum_{i=1}^n\\alpha_i\\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) + \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2 \\notag\\\\\n\t\t\t& \\overset{\\rm (b)}= - \\sum_{i=1}^n\\alpha_i\\widetilde{\\zeta}_i^{-1}(\\omega_i - \\widetilde{\\zeta}_i) + \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2\n\t\t\t\\overset{\\rm (c)}= - \\sum_{i=1}^n\\alpha_i\\widetilde{\\zeta}_i^{-1}\\omega_i \\!+\\! 1 \\!+\\! \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2                                                                                                                          \\notag\\\\\n\t\t\t& = \\sum_{i=1}^n\\zeta_i\\omega_i + 1 + \\sum_{i=1}^n\\alpha_ic_i (\\omega_i - \\widetilde{\\zeta}_i)^2,\\notag\n\t\t\\end{align}\n\t\twhere (a) and (b) hold because \\( \\prod_{i=1}^n\\widetilde{\\zeta}_i^{\\alpha_i} = \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\), (c) uses the fact that \\( \\sum_{i=1}^n\\alpha_i = 1 \\), and the last equality follows from the definition of \\( \\widetilde{\\bm{\\zeta}} \\). Rearranging the above inequality, we conclude that \\eqref{eq:inequ-inprod-power-cone} holds with \\( C = \\min\\limits_{1\\le i\\le n}\\alpha_ic_i > 0 \\).\n","start":26180,"end":27811}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"对偶性与等号条件","source_units":[{"raw_tex":"\n\t\tNext, let \\( \\bm{\\omega}\\in {\\rm int}\\,\\R_+^n \\) satisfy \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\). Then \\( (-1,\\bm{\\omega})\\in \\K \\). Recall from the assumption that \\( (1,-\\bm{\\zeta})\\in \\dK \\). From these we deduce \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}\\le-1 \\). If \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}=-1 \\), then\n\t\t\\[\n\t\t\\sum_{i=1}^n \\alpha_i\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)\\omega_i = \\sum_{i=1}^n (-\\zeta_i)\\omega_i = 1 = \\prod_{i=1}^n\\omega_i^{\\alpha_i} = \\prod_{i=1}^n\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)^{\\alpha_i}\\prod_{i=1}^n\\omega_i^{\\alpha_i} .\n\t\t\\]\n\t\tTaking \\( \\ln \\) on both sides of the above equality, we see that\n\t\t\\[\n\t\t\\ln\\left[\\sum_{i=1}^n \\alpha_i\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)\\omega_i\\right] = \\sum_{i=1}^n\\alpha_i \\ln\\left[\\left(\\frac{-\\zeta_i}{\\alpha_i}\\right)\\omega_i\\right].\n\t\t\\]\n\t\tSince \\( \\ln \\) is strictly concave and \\( \\alpha_i \\in (0,1) \\) for all \\( i \\), we conclude that there exists \\( c > 0 \\) so that \\( \\omega_i\\cdot(-\\zeta_i/\\alpha_i) = c \\) for all \\( i \\). This, together with the facts that\n\t\t\\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = \\prod_{i=1}^n (-\\zeta_i/\\alpha_i)^{\\alpha_i} = 1 \\) and \\( \\sum_{i=1}^n\\alpha_i = 1 \\), gives \\( c = 1 \\). It thus follows that \\( \\bm{\\omega} = \\widetilde{\\bm{\\zeta}} \\). Conversely, it is routine to check that if \\( \\bm{\\omega} = \\widetilde{\\bm{\\zeta}} \\), then \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i} = 1 \\) and \\( \\inProd{\\bm{\\zeta}}{\\bm{\\omega}}=-1 \\).\n\t\\end{proof}","start":27811,"end":29267}]},{"id":"S018","semantic_type":"external_fact","kind":"external_fact","labels":["eq:inequ-p=q=2","lemma:Lemma-4.1"],"member_ids":["S018"],"declared_external":true,"declaration_text":["所声明允许前提 LiLoPo21, Lemma 4.1，原文明确以 p=q=2 专门化并完整重述，未附证明。"],"source_only":false,"title":"欧氏球面的二次间隙","source_units":[{"raw_tex":"\t{\n\t  The next lemma is obtained by applying \\cite[Lemma 4.1]{LiLoPo21} with \\( p = q = 2 \\).\n\t  \\begin{lemma}\n\t\t\\label{lemma:Lemma-4.1}\n\t\tLet \\( \\bm{\\zeta} \\in \\R^n \\) (\\( n \\geq 1 \\)) satisfy \\( \\| \\bm{\\zeta} \\| = 1 \\).\n\t\tDefine \\( \\overline{\\bm{\\zeta}} := -\\bm{\\zeta} \\). Then there exist \\( C > 0 \\) and \\( \\epsilon > 0 \\) so that\n\t\t\\begin{equation}\n\t\t  \\label{eq:inequ-p=q=2}\n\t\t  1 + \\left\\langle \\bm{\\zeta}, \\bm{w} \\right\\rangle \\geq C \\sum_{i \\in I} | w_i - \\overline{\\zeta}_i |^2 + \\frac{1}{2} \\sum_{i \\notin I} | w_i |^2  \\ \\text{ whenever } \\| \\bm{w} - \\overline{\\bm{\\zeta}} \\| \\leq \\epsilon \\ \\text{ and } \\ \\| \\bm{w} \\| = 1,\n\t\t\\end{equation}\n\t\twhere \\( I = \\{i \\,|\\, \\overline{\\zeta}_i \\neq 0\\} \\).\n\t\tFurthermore, for any \\( \\bm{w} \\) satisfying \\( \\| \\bm{w} \\| \\leq  1\\), it holds that \\( \\left\\langle \\bm{\\zeta}, \\bm{w} \\right\\rangle \\geq -1 \\), with the equality holding if and only if \\( \\bm{w} = \\overline{\\bm{\\zeta}} \\).\n\t  \\end{lemma}\n\t}\n","start":29269,"end":30226}]},{"id":"S019","semantic_type":"proposition","kind":"proposition","labels":["eq:1st-type-face-ray","item:case-I-facial-structure","item:case-II-facial-structure","prop:facial-stru-gK"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"非零首块暴露射线","source_units":[{"raw_tex":"\\begin{proposition}[Proper nontrivial exposed faces of \\( \\gK \\)]\\label{prop:facial-stru-gK}\n\t\tLet \\( \\bm{z} = (\\bbz,\\tbz)\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\).\n\t\t\\begin{enumerate}[label={(\\roman*)}]\n\t\t  \\item \\label{item:case-I-facial-structure} If \\( \\bbz \\neq \\bm{0} \\), then \\( \\bm{z} \\) exposes the following one-dimensional face: %\\( \\Fr \\), where\n\t\t\t\t\\begin{equation}\\label{eq:1st-type-face-ray}\n\t\t\t\t  \\displayindent0pt\n\t\t\t\t  \\displaywidth\\textwidth\n\t\t\t\t  \\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK =  \\{t \\bm{f}\\in \\R^{m+n}\\,| \\,t\\ge 0\\}\\,\\,\\mbox{with} \\,\\bm{f}= (-\\bbz/\\|\\bbz\\|^2,\\bm{\\alpha}\\circ \\tbz^{-1}),\n\t\t\t\t\\end{equation}\n\t\t\t\twhere the inverse is taken componentwise.\n","start":30512,"end":31181}]},{"id":"S020","semantic_type":"proposition","kind":"proposition","labels":["eq:2nd-type-face-d-dim","item:case-I-facial-structure","item:case-II-facial-structure","prop:facial-stru-gK"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"零首块暴露坐标面","source_units":[{"raw_tex":"\\begin{proposition}[Proper nontrivial exposed faces of \\( \\gK \\)]\\label{prop:facial-stru-gK}\n\t\tLet \\( \\bm{z} = (\\bbz,\\tbz)\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\).\n\t\t\\begin{enumerate}[label={(\\roman*)}]\n","start":30512,"end":30709},{"raw_tex":"\n\t\t  \\item \\label{item:case-II-facial-structure} If \\( \\bbz = \\bm{0} \\), then \\( \\bm{z} \\) exposes the following face of dimension \\( n - | \\stdIndex | \\):\n\t\t\t\t\\begin{equation}\\label{eq:2nd-type-face-d-dim}\n\t\t\t\t  \\displayindent0pt\n\t\t\t\t  \\displaywidth\\textwidth\n\t\t\t\t  \\Fhatz := \\{\\bm{z}\\}^\\perp \\cap \\gK = \\{\\bm{x} = (\\bbx,\\tbx)\\in \\R_+^{m+n}\\,| \\,\\bbx = \\bm{0},\\,\\tx_i = 0\\ \\forall i\\in \\stdIndex\\},\n\t\t\t\t\\end{equation}\n\t\t\t\twhere \\( \\stdIndex := \\{i \\,|\\, \\tz_i > 0\\}\\neq \\emptyset \\) and \\( | \\stdIndex | \\) denotes the cardinality of $\\stdIndex$.\n\t\t\\end{enumerate}\n\t  \\end{proposition}","start":31181,"end":31767}]},{"id":"S021","semantic_type":"proof","kind":"proof","labels":["eq:form-barx-by-barz-kappa","eq:form-tildex-by-tildez-case-I","eq:inequ-sum-prod-power-case-I","eq:inprod-barz-barx=c-case-I","eq:inprod=0-case-I","eq:log-inequ-sum-prod-power-case-I"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"暴露面分类完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\t  \\ref{item:case-I-facial-structure}: Notice that \\( \\bm{x} = (\\bbx,\\tbx)\\in \\{\\bm{z}\\}^\\perp\\cap \\gK\\backslash \\{\\bm{0}\\} \\) if and only if \\( \\bm{x}\\in \\bd\\gK \\), \\( \\bm{x} \\neq \\bm{0} \\) and %satisfies\n\t  \\begin{equation}\\label{eq:inprod=0-case-I}\n\t\t\\inProd{\\bbz}{\\bbx} + \\inProd{ \\tbz}{\\tbx} = 0.\n\t  \\end{equation}\n\t  The above relation yields\n\t  \\begin{equation}\\label{eq:inequ-sum-prod-power-case-I}\n\t\t\\sum_{i=1}^n \\tz_i\\tx_i = -\\inProd{\\bbz}{\\bbx}\\le \\|\\bbz\\| \\|\\bbx\\| \\le \\prod_{i=1}^n\\left(\\frac{\\tx_i\\tz_i}{\\alpha_i}\\right)^{\\alpha_i},\n\t  \\end{equation}\n\t  where the last inequality follows from the definition of $\\gK$ in \\eqref{eq:gpowercones}.\n\n\t  Note that \\( \\tx_i \\) cannot be \\textbf{all} zero, for otherwise \\( \\bbx \\) will also be zero since \\( \\bm{x}\\in \\bd\\gK \\), which contradicts \\( \\bm{x}\\neq \\bm{0} \\). In addition, we must have \\( \\tz_i > 0 \\) for all \\( i \\) because \\( \\bbz \\neq \\bm{0} \\) and \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\). Using these observations, we have $\\sum_{i=1}^n \\tz_i\\tx_i >0$. Combining this with \\eqref{eq:inequ-sum-prod-power-case-I}, we deduce that \\( \\tx_i\\tz_i > 0 \\) for all \\( i \\). Now we can take \\( \\ln \\) on both sides of \\eqref{eq:inequ-sum-prod-power-case-I} to obtain\n\t  \\begin{equation}\\label{eq:log-inequ-sum-prod-power-case-I}\n\t\t\\ln\\left[\\sum_{i=1}^n \\alpha_i\\left(\\frac{\\tx_i\\tz_i}{\\alpha_i}\\right)\\right] \\le \\alpha_1\\ln\\left(\\frac{\\tx_1\\tz_1}{\\alpha_1}\\right) + \\cdots + \\alpha_n\\ln\\left(\\frac{\\tx_n\\tz_n}{\\alpha_n}\\right).\n\t  \\end{equation}\n\t  Using this together with the fact that \\( \\ln(\\cdot) \\) is strictly concave, we deduce that \\eqref{eq:log-inequ-sum-prod-power-case-I} holds as an equality. Hence, there exists a constant \\( c > 0 \\) so that\n\t  \\begin{equation}\\label{eq:form-tildex-by-tildez-case-I}\n\t\t\\tx_i = c \\alpha_i\\tz_i^{-1}\\,\\,\\,\\forall i = 1, 2, \\dots , n.\n\t  \\end{equation}\n\t  Plugging \\eqref{eq:form-tildex-by-tildez-case-I} into \\eqref{eq:inprod=0-case-I}, we obtain\n\t  \\begin{equation}\\label{eq:inprod-barz-barx=c-case-I}\n\t\t\\inProd{\\bbz}{\\bbx} = -\\inProd{ \\tbz}{\\tbx} = -c\\sum_{i=1}^n \\alpha_i = -c.\n\t  \\end{equation}\n\t  Moreover, using \\eqref{eq:form-tildex-by-tildez-case-I} and the last relation in \\eqref{eq:inequ-sum-prod-power-case-I}, we see that\n\t  \\[\n\t\t\\|\\bbz\\| \\|\\bbx\\| \\le \\prod_{i=1}^n\\left(\\frac{\\tx_i\\tz_i}{\\alpha_i}\\right)^{\\alpha_i} = c.\n\t  \\]\n\t  {The two displayed lines} above show that \\( \\|\\bbz\\| \\|\\bbx\\| = -\\inProd{\\bbz}{\\bbx} \\), which together with \\( \\bbz \\neq \\bm{0} \\) implies that there exists \\( \\kappa > 0 \\) so that\n\t  \\begin{equation}\\label{eq:form-barx-by-barz-kappa}\n\t\t\\bbx = -\\kappa\\bbz.\n\t  \\end{equation}\n\t  Plugging \\eqref{eq:form-barx-by-barz-kappa} into \\eqref{eq:inprod-barz-barx=c-case-I}, we obtain that \\( \\kappa = c/\\|\\bbz\\|^2 \\). Using this together with \\eqref{eq:form-tildex-by-tildez-case-I} and \\eqref{eq:form-barx-by-barz-kappa}, we can now conclude that\n\t  \\[\n\t\t\\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK =  \\{t \\bm{f}\\in \\R^{m+n}\\,| \\,t\\ge 0\\}\\,\\,\\mbox{with} \\,\\bm{f}= (-\\bbz/\\|\\bbz\\|^2,\\bm{\\alpha}\\circ \\tbz^{-1}),\n\t  \\]\n\t  where the inverse is taken componentwise.\n\n\t  \\ref{item:case-II-facial-structure}: In this case, \\( \\bbz = \\bm{0} \\). Then \\( \\stdIndex:= \\{i\\,| \\,\\tz_i > 0\\} \\) is nonempty because \\( \\bm{z}\\neq \\bm{0} \\). Hence, \\( \\bm{x} = (\\bbx,\\tbx)\\in \\{\\bm{z}\\}^\\perp\\cap \\gK\\backslash \\{\\bm{0}\\} \\) if and only if \\( \\bm{x}\\in \\bd\\gK\\backslash\\{\\bm{0}\\} \\) and satisfies\n\t  \\begin{equation*}\n\t\t\\sum_{i\\in \\stdIndex} \\tz_i\\tx_i = 0.\n\t  \\end{equation*}\n\t  This means that \\( \\tx_i = 0 \\) whenever \\( i\\in \\stdIndex \\) and hence \\( \\bbx= \\bm{0} \\). Thus,\n\t  \\[\n\t\t\\Fhatz := \\{\\bm{z}\\}^\\perp \\cap \\gK = \\{\\bm{x} = (\\bbx,\\tbx)\\in \\R_+^{m+n}\\,| \\,\\bbx = \\bm{0},\\,\\tx_i = 0\\ \\forall i\\in \\stdIndex\\}.\n\t  \\]\n\t\\end{proof}","start":31773,"end":35553}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","labels":["eq:inequ-sum-prod-power-case-I","eq:inprod=0-case-I"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"正交关系与坐标正性","source_units":[{"raw_tex":"\\begin{proof}\n\t  \\ref{item:case-I-facial-structure}: Notice that \\( \\bm{x} = (\\bbx,\\tbx)\\in \\{\\bm{z}\\}^\\perp\\cap \\gK\\backslash \\{\\bm{0}\\} \\) if and only if \\( \\bm{x}\\in \\bd\\gK \\), \\( \\bm{x} \\neq \\bm{0} \\) and %satisfies\n\t  \\begin{equation}\\label{eq:inprod=0-case-I}\n\t\t\\inProd{\\bbz}{\\bbx} + \\inProd{ \\tbz}{\\tbx} = 0.\n\t  \\end{equation}\n\t  The above relation yields\n\t  \\begin{equation}\\label{eq:inequ-sum-prod-power-case-I}\n\t\t\\sum_{i=1}^n \\tz_i\\tx_i = -\\inProd{\\bbz}{\\bbx}\\le \\|\\bbz\\| \\|\\bbx\\| \\le \\prod_{i=1}^n\\left(\\frac{\\tx_i\\tz_i}{\\alpha_i}\\right)^{\\alpha_i},\n\t  \\end{equation}\n\t  where the last inequality follows from the definition of $\\gK$ in \\eqref{eq:gpowercones}.\n\n\t  Note that \\( \\tx_i \\) cannot be \\textbf{all} zero, for otherwise \\( \\bbx \\) will also be zero since \\( \\bm{x}\\in \\bd\\gK \\), which contradicts \\( \\bm{x}\\neq \\bm{0} \\). In addition, we must have \\( \\tz_i > 0 \\) for all \\( i \\) because \\( \\bbz \\neq \\bm{0} \\) and \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\). Using these observations, we have $\\sum_{i=1}^n \\tz_i\\tx_i >0$. Combining this with \\eqref{eq:inequ-sum-prod-power-case-I}, we deduce that \\( \\tx_i\\tz_i > 0 \\) for all \\( i \\). Now we can take \\( \\ln \\) on both sides of \\eqref{eq:inequ-sum-prod-power-case-I} to obtain\n","start":31773,"end":33023}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","labels":["eq:form-tildex-by-tildez-case-I","eq:log-inequ-sum-prod-power-case-I"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"严格凹性固定尾块比例","source_units":[{"raw_tex":"\t  \\begin{equation}\\label{eq:log-inequ-sum-prod-power-case-I}\n\t\t\\ln\\left[\\sum_{i=1}^n \\alpha_i\\left(\\frac{\\tx_i\\tz_i}{\\alpha_i}\\right)\\right] \\le \\alpha_1\\ln\\left(\\frac{\\tx_1\\tz_1}{\\alpha_1}\\right) + \\cdots + \\alpha_n\\ln\\left(\\frac{\\tx_n\\tz_n}{\\alpha_n}\\right).\n\t  \\end{equation}\n\t  Using this together with the fact that \\( \\ln(\\cdot) \\) is strictly concave, we deduce that \\eqref{eq:log-inequ-sum-prod-power-case-I} holds as an equality. Hence, there exists a constant \\( c > 0 \\) so that\n\t  \\begin{equation}\\label{eq:form-tildex-by-tildez-case-I}\n\t\t\\tx_i = c \\alpha_i\\tz_i^{-1}\\,\\,\\,\\forall i = 1, 2, \\dots , n.\n\t  \\end{equation}\n","start":33023,"end":33656}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","labels":["eq:form-barx-by-barz-kappa","eq:inprod-barz-barx=c-case-I"],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"等号条件固定射线方向","source_units":[{"raw_tex":"\t  Plugging \\eqref{eq:form-tildex-by-tildez-case-I} into \\eqref{eq:inprod=0-case-I}, we obtain\n\t  \\begin{equation}\\label{eq:inprod-barz-barx=c-case-I}\n\t\t\\inProd{\\bbz}{\\bbx} = -\\inProd{ \\tbz}{\\tbx} = -c\\sum_{i=1}^n \\alpha_i = -c.\n\t  \\end{equation}\n\t  Moreover, using \\eqref{eq:form-tildex-by-tildez-case-I} and the last relation in \\eqref{eq:inequ-sum-prod-power-case-I}, we see that\n\t  \\[\n\t\t\\|\\bbz\\| \\|\\bbx\\| \\le \\prod_{i=1}^n\\left(\\frac{\\tx_i\\tz_i}{\\alpha_i}\\right)^{\\alpha_i} = c.\n\t  \\]\n\t  {The two displayed lines} above show that \\( \\|\\bbz\\| \\|\\bbx\\| = -\\inProd{\\bbz}{\\bbx} \\), which together with \\( \\bbz \\neq \\bm{0} \\) implies that there exists \\( \\kappa > 0 \\) so that\n\t  \\begin{equation}\\label{eq:form-barx-by-barz-kappa}\n\t\t\\bbx = -\\kappa\\bbz.\n\t  \\end{equation}\n\t  Plugging \\eqref{eq:form-barx-by-barz-kappa} into \\eqref{eq:inprod-barz-barx=c-case-I}, we obtain that \\( \\kappa = c/\\|\\bbz\\|^2 \\). Using this together with \\eqref{eq:form-tildex-by-tildez-case-I} and \\eqref{eq:form-barx-by-barz-kappa}, we can now conclude that\n\t  \\[\n\t\t\\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK =  \\{t \\bm{f}\\in \\R^{m+n}\\,| \\,t\\ge 0\\}\\,\\,\\mbox{with} \\,\\bm{f}= (-\\bbz/\\|\\bbz\\|^2,\\bm{\\alpha}\\circ \\tbz^{-1}),\n\t  \\]\n\t  where the inverse is taken componentwise.\n","start":33656,"end":34893}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"非负加权和刻画坐标面","source_units":[{"raw_tex":"\n\t  \\ref{item:case-II-facial-structure}: In this case, \\( \\bbz = \\bm{0} \\). Then \\( \\stdIndex:= \\{i\\,| \\,\\tz_i > 0\\} \\) is nonempty because \\( \\bm{z}\\neq \\bm{0} \\). Hence, \\( \\bm{x} = (\\bbx,\\tbx)\\in \\{\\bm{z}\\}^\\perp\\cap \\gK\\backslash \\{\\bm{0}\\} \\) if and only if \\( \\bm{x}\\in \\bd\\gK\\backslash\\{\\bm{0}\\} \\) and satisfies\n\t  \\begin{equation*}\n\t\t\\sum_{i\\in \\stdIndex} \\tz_i\\tx_i = 0.\n\t  \\end{equation*}\n\t  This means that \\( \\tx_i = 0 \\) whenever \\( i\\in \\stdIndex \\) and hence \\( \\bbx= \\bm{0} \\). Thus,\n\t  \\[\n\t\t\\Fhatz := \\{\\bm{z}\\}^\\perp \\cap \\gK = \\{\\bm{x} = (\\bbx,\\tbx)\\in \\R_+^{m+n}\\,| \\,\\bbx = \\bm{0},\\,\\tx_i = 0\\ \\forall i\\in \\stdIndex\\}.\n\t  \\]\n\t\\end{proof}","start":34893,"end":35553}]},{"id":"S026","semantic_type":"theorem","kind":"theorem","labels":["eq:gamma-1/2-1st-type-face","thm:Holderian-1/2-1st-type-face"],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"射线面的平方根误差界","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm:Holderian-1/2-1st-type-face}\n\t\tLet \\( \\bm{z} = (\\bbz,\\tbz)\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz\\neq \\bm{0} \\) and let \\( \\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Let \\( \\eta > 0 \\) and define\n\t\t\\begin{equation}\\label{eq:gamma-1/2-1st-type-face}\n\t\t\t\\gamma_{\\bm{z},\\eta}\\! :=\\! \\inf_{\\bm{v}}\\left\\{\\frac{\\|\\bm{v} - \\bm{w}\\|^\\frac12}{\\|\\bm{u} - \\bm{w}\\|}\\,\\bigg| \\,\n\\begin{array}{c}\n\\bm{v}\\in \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fr,\\,\\bm{w} = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}),\\\\\n\\bm{u} = P_{\\Fr}(\\bm{w}),\\,\\bm{u} \\neq \\bm{w}\n\\end{array}\n\\right\\}.\n\t\t\\end{equation}\n\t\tThen it holds that \\( \\gamma_{\\bm{z},\\eta}\\in (0,\\infty] \\) and that\n\t\t\\[\n\t\t\\dist(\\bm{q},\\Fr)\\le \\max\\{2\\sqrt{\\eta},2\\gamma_{\\bm{z},\\eta}^{-1}\\}\\cdot\\dist(\\bm{q},\\gK)^{\\frac{1}{2}}\\,\\,\\text{ whenever }  \\bm{q} \\in \\{\\bm{z}\\}^\\perp\\cap \\B(\\eta) .\n\t\t\\]\n\t\\end{theorem}","start":38549,"end":39400}]},{"id":"S027","semantic_type":"proof","kind":"proof","labels":["eq:def-h","eq:dist-vk-wk","eq:forcontradiction-1st-face","eq:keylemmare2","eq:keylemmare4","eq:keylemmarel","eq:sum-out-of-neighbor","eq:upbd-dist-uk-wk-two-parts","eq:upbd-part-dist-uk-wk-case-2","eq:vk0-notation-nonzero","eq:z0-v0-notation","eqn:3Ms"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"射线面估计完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\t\tSuppose for a contradiction that \\( \\gamma_{\\bm{z},\\eta} = 0 \\). Then, {in view of \\cite[Lemma 3.12]{LiLoPo20}}, there exist \\( \\widehat{\\bm{v}}\\in \\Fr \\) and a sequence \\( \\{\\bm{v}^k\\}\\subset \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fr \\) such that\n\t\t\\begin{equation}\\label{eq:forcontradiction-1st-face}\n\t\t\t\\lim_{k\\to \\infty}\\bm{v}^k = \\lim_{k\\to \\infty}\\bm{w}^k = \\widehat{\\bm{v}}\\,\\,{\\rm and}\\,\\,\\lim_{k\\to\\infty}\\frac{\\|\\bm{w}^k - \\bm{v}^k\\|^{\\frac{1}{2}}}{\\|\\bm{w}^k - \\bm{u}^k\\|} = 0,\n\t\t\\end{equation}\n\t\twhere \\( \\bm{w}^k = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}^k) \\), \\( \\bm{u}^k = P_{\\Fr}(\\bm{w}^k) \\) and \\( \\bm{u}^k\\neq \\bm{w}^k \\).\n\n\t\tDefine, for notational simplicity, \\( z_0 := \\|\\bbz\\| \\) and \\( v^k_0:=\\|\\bbv^k\\| \\).\n\t\tThen, since \\( \\{\\bm{v}^k\\}\\subset \\bd \\gK \\) and \\( \\bm{z} \\in \\bd\\dgK \\) with \\( \\bbz \\neq \\bm{0} \\), we have\\vspace{-0.15 cm}\n\t\t\\begin{equation}\\label{eq:z0-v0-notation}\\vspace{-0.15 cm}\n\t\t\tz_0 = \\|\\bbz\\| = \\prod_{i=1}^n\\left(\\frac{\\tz_i}{\\alpha_i}\\right)^{\\alpha_i}> 0 \\,\\,{\\rm and}\\,\\,v^k_0=\\|\\bbv^k\\| = \\prod_{i=1}^n(\\tv_i^k)^{\\alpha_i}\\,\\,\\forall k.\n\t\t\\end{equation}\n\n\t\tIf it holds that \\( v_0^k = 0 \\) infinitely often, by passing to a further subsequence, we may assume that \\( v_0^k = 0 \\) for all \\( k \\). Then we have in view of {Lemma \\ref{lemma:Lemma-2.11}} that\\vspace{-0.15 cm}\n\t\t\\[\\vspace{-0.15 cm}\n\t\t\\!\\|\\bm{v}^k - \\bm{w}^k\\| \\!=\\! \\frac1{\\|\\bm{z}\\|}|\\inProd{ \\tbz}{\\tbv^k}|\\!\\overset{\\rm (a)}=\\! \\frac1{\\|\\bm{z}\\|}\\sum_{i=1}^n\\tz_i\\tv^k_i\\!\\ge\\! \\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|} \\|\\tbv^k\\|_1\\!\\ge\\! \\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|} \\|\\tbv^k\\|\n\t\t\\!\\overset{\\rm (b)}=\\!\\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|} \\|\\bm{v}^k\\|,\n\t\t\\]\n\t\twhere (a) holds because \\( \\tv^k_i \\ge 0 \\) and \\( \\tz_i > 0 \\) for all \\( i \\) (see \\eqref{eq:z0-v0-notation}), and (b) holds since \\( \\|\\bbv^k\\| = 0 \\). Since\n\t\t\\( \\|\\bm{w}^k - \\bm{u}^k\\| = \\dist(\\bm{w}^k,\\Fr) \\le \\|\\bm{w}^k\\|\\le \\|\\bm{v}^k\\| \\) as a consequence of the properties of projections, we conclude from this and the above display that\n\t\t\\( \\|\\bm{v}^k - \\bm{w}^k\\| \\ge \\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|}\\|\\bm{w}^k - \\bm{u}^k\\| \\), contradicting \\eqref{eq:forcontradiction-1st-face}.\n\n\t\tThus, by considering a further subsequence if necessary, from now on, we assume\\vspace{-0.15 cm}\n\t\t\\begin{equation}\\label{eq:vk0-notation-nonzero}\\vspace{-0.15 cm}\n\t\t\tv^k_0=\\|\\bbv^k\\| = \\prod_{i=1}^n(\\tv_i^k)^{\\alpha_i} > 0 \\,\\,\\forall k.\n\t\t\\end{equation}\n\t\tUsing {Lemma \\ref{lemma:Lemma-2.11}}, we see that\\vspace{-0.15 cm}\n\t\t\\begin{equation}\\label{eq:dist-vk-wk}\\vspace{-0.15 cm}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\|\\bm{v}^k - \\bm{w}^k\\| & =\\frac1{\\|\\bm{z}\\|}|\\inProd{ \\bm{z}}{\\bm{v}^k}| = \\frac1{\\|\\bm{z}\\|}\\left|\\sum_{i=1}^{m}\\bz_i\\bv^k_i + \\sum_{i=1}^n\\tz_i\\tv^k_i\\right|             \\\\\n\t\t\t\t& = \\frac1{\\|\\bm{z}\\|}\\left|z_0v_0^k + \\sum_{i=1}^{m}\\bz_i\\bv^k_i - \\sum_{i=1}^n(-\\tz_i)\\tv^k_i - z_0v_0^k\\right|                                        \\\\\n\t\t\t\t& = \\frac{z_0v_0^k}{\\|\\bm{z}\\|}\\left| 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1\\right|  \\\\\n\t\t\t\t& = \\frac{z_0}{\\|\\bm{z}\\|}\\left( 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1\\right)v_0^k,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\twhere the last equality holds as \\( \\|z_0^{-1}\\bbz\\|= 1 \\), \\( \\|(v_0^k)^{-1}\\bbv^k\\|= 1 \\) and  \\( \\inProd{ z_0^{-1}\\tbz}{ (v_0^k)^{-1}\\tbv^k} \\ge 1 \\), thanks to \\eqref{eq:z0-v0-notation}, \\eqref{eq:vk0-notation-nonzero} and Lemma~\\ref{lemma:inequ-inprod-power-cone} applied with \\( \\bm{\\zeta} = -z_0^{-1}\\tbz \\).\n\n\t\tLet \\( \\bm{f} \\) be defined as in \\eqref{eq:1st-type-face-ray}. We consider two cases:\n\t\t\\begin{enumerate}[label={(\\Roman*)}]\n\t\t\t\\item \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}\\ge 0 \\) for all sufficiently large \\( k \\).\n\t\t\t\\item \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}< 0 \\) infinitely often.\n\t\t\\end{enumerate}\n\n\t\t(I): By passing to a further subsequence, we may assume that \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}\\ge 0 \\) for all \\( k \\). In this case, if we define\n\t\t\\[\n\t\t\\bm{Q} = \\bm{I}_{m + n} - \\frac{\\bm{z}\\bm{z}^{\\top}}{\\|\\bm{z}\\|^2} - \\frac{\\bm{f}\\bm{f}^{\\top}}{\\|\\bm{f}\\|^2},\n\t\t\\]\n\t\twhere \\( \\bm{f} \\) is as in \\eqref{eq:1st-type-face-ray}, then we see from {Lemma \\ref{lemma:Lemma-2.11}} and \\eqref{eq:z0-v0-notation} that\n\t\t\\begin{equation}\\label{eq:upbd-dist-uk-wk-two-parts}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\|\\bm{u}^k - \\bm{w}^k\\| & = \\|\\bm{Q}\\bm{v}^k\\| = v_0^k\\left\\|\\bm{Q}\\begin{bmatrix}\n\t\t\t\t\t(v_0^k)^{-1}\\bbv^k \\\\ (v_0^k)^{-1}\\tbv^k\n\t\t\t\t\\end{bmatrix}\\right\\| \\\\\n&\\overset{\\rm (a)}= v_0^k{\\Bigg\\|}\\bm{Q}\\begin{bmatrix}\n\t\t\t\t\t(v_0^k)^{-1}\\bbv^k \\\\ (v_0^k)^{-1}\\tbv^k\n\t\t\t\t\\end{bmatrix} \\!-\\! \\bm{Q}\\underbrace{\\begin{bmatrix}\n\t\t\t\t\t\t-z_0^{-1}\\bbz \\\\ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\n\t\t\t\t\\end{bmatrix}}_{z_0\\bm{f}}{\\Bigg\\|} \\\\\n\t\t\t\t& \\le v_0^k\\left[\\|(v_0^k)^{-1}\\bbv^k + z_0^{-1}\\bbz\\| + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\right],\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\twhere (a) holds because %\\( z_0\\bm{f} = \\begin{bmatrix}\n\t\t%\t\t-z_0^{-1}\\bbz^{\\top} & \\alpha\\circ(z_0\\tbz^{-1})^{\\top}\n\t\t%\t\\end{bmatrix}^{\\top}\\)  and\n\t\t\\( \\bm{Q}\\bm{f} = \\bm{0} \\) (an identity which is clear from the definitions).\n\n\t\tNext, in view of \\eqref{eq:z0-v0-notation}, we can apply {Lemma \\ref{lemma:Lemma-4.1}} to obtain \\( C_1 > 0 \\) and \\( \\epsilon_1>0 \\) so that {\\eqref{eq:inequ-p=q=2}} holds with \\( \\overline{\\bm{\\zeta}} = -z_0^{-1}\\bbz \\), i.e.,\n\t\t\\[\n\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_1\\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|^2\n\t\t\\]\n\t\twhenever \\( \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\le \\epsilon_1 \\) and \\( \\|\\bm{\\omega}\\|=1 \\). On the other hand, in view of the positivity of $1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}$ when $\\|\\bm{\\omega}\\|=1$ and $\\bm{\\omega} \\neq -z_0^{-1}\\bbz$ {(see Lemma \\ref{lemma:Lemma-4.1})}, we know that\n\\[\nC_2 := \\inf_{\\|\\bm{\\omega}\\|=1}\\{1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}} \\,|\\, \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\ge \\epsilon_1 \\} > 0.\n\\]\nThis together with the fact $\\|z_0^{-1}\\bbz\\|=1$ (see \\eqref{eq:z0-v0-notation}) implies that\n\t\t\\[\n\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_2 \\ge 0.25 C_2 \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|^2,\n\t\t\\]\n\t\twhenever \\( \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\ge \\epsilon_1 \\) and \\( \\|\\bm{\\omega}\\|=1 \\). We thus have (with $C_3 \\!:=\\! \\min\\{C_1,C_2/4\\}$)\n\\begin{equation}\\label{eq:keylemmarel}\n\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_3\\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|^2\\ \\ \\ {\\rm whenever}\\ \\ \\|\\bm{\\omega}\\|=1.\n\t\t\\end{equation}\n\n\t\tIn addition, noting \\eqref{eq:z0-v0-notation} again, we can apply Lemma~\\ref{lemma:inequ-inprod-power-cone} with \\( \\bm{\\zeta} = -z_0^{-1}\\tbz\\in {\\rm int}\\,\\R^n_- \\) to obtain \\( C_4 > 0 \\) and \\( \\epsilon > 0 \\) so that \\eqref{eq:inequ-inprod-power-cone} holds with \\( \\widetilde{\\bm{\\zeta}} = \\bm{\\alpha}\\circ (z_0\\tbz^{-1}) \\), i.e.,\n\t\t\\begin{equation}\\label{eq:keylemmare2}\n\t\t\t-1 + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}\\ge C_4\\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2\n\t\t\\end{equation}\n\t\twhenever \\( \\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\le \\epsilon \\), \\( \\bm{\\omega}\\in {\\rm int}\\,\\R^n_+ \\) and \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i}=1 \\).\n\n%Then we have in view of the positivity of $1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}$ when $\\bm{\\omega} \\neq -z_0^{-1}\\bbz$ (see Lemma \\ref{lemma:Lemma-4.1}) and the compactness of the unit sphere that there exists \\( C_3>0 \\) so that\n%\t\t\\begin{equation}\\label{eq:keylemmare3}\n%\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_3 \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\n%\t\t\\end{equation}\n%\t\twhenever \\( \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\ge \\epsilon \\) and \\( \\|\\bm{\\omega}\\|=1 \\).\n\nFurthermore, consider \\( h:\\R^n\\to \\R\\cup\\{\\infty\\} \\) defined by\n\t\t\\begin{equation}\\label{eq:def-h}\n\t\t\th(\\bm{\\omega}) = \\begin{cases}\n\t\t\t\t\\displaystyle\\frac{\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}-1}{\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|} & {\\rm if}\\,\\|\\bm{\\omega}-\\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\ge \\epsilon,\\,\\bm{\\omega}\\in \\Upsilon, \\\\\n\t\t\t\t\\infty                                                                                                    & {\\rm otherwise},\n\t\t\t\\end{cases}\n\t\t\\end{equation}\n\t\twhere \\( \\Upsilon= \\{\\bm{\\omega}\\in \\R^n_+\\,| \\,\\prod_{i=1}^n \\omega_i^{\\alpha_i}=1\\} \\). Then we have\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t& \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty}h(\\bm{\\omega}) = \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty, \\bm{\\omega}\\in \\Upsilon}\\frac{\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}-1}{\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|} \\\\\n\t\t\t& \\ge \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty,\\bm{\\omega}\\in \\R^n_+}\\frac{\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}-1}{\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|}\n\t\t\t\\overset{\\rm (a)}{\\ge} \\inf_{\\|\\bm{\\lambda}\\|=1,\\bm{\\lambda}\\in \\R^n_+}\\inProd{ z_0^{-1}\\tbz}{\\bm{\\lambda}} \\overset{\\rm (b)}\\ge \\min_{1\\le i\\le n}z_0^{-1}\\tz_i > 0.\n\t\t\\end{aligned}\n\t\t\\]\n\t\tHere (a) may be verified by multiplying both numerator and denominator of the left side by $1/\\|\\bm{\\omega}\\|$; (b) holds because \\( z_0^{-1}\\tz_i > 0 \\) for all \\( i \\) (see \\eqref{eq:z0-v0-notation}). Since \\( h \\) in \\eqref{eq:def-h} is also lower semicontinuous on any compact set and {\\bf is always positive},\\footnote{The positivity can be seen by applying Lemma~\\ref{lemma:inequ-inprod-power-cone} with $\\bm\\zeta = -z_0^{-1}\\tbz$.} it must then hold that \\( C_5:= \\inf h > 0 \\). In particular, this means that\n\t\t\\begin{equation}\\label{eq:keylemmare4}\n\t\t\t-1 + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}\\ge C_5\\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\n\t\t\\end{equation}\n\t\twhenever \\( \\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\ge \\epsilon \\), \\( \\bm{\\omega}\\in {\\rm int}\\,\\R^n_+ \\) and \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i}=1 \\).\n\n\t\tBy passing to suitable subsequences, we will end up with one of the following two cases:\n\t\t\\begin{enumerate}[label={Case \\arabic*:}]\n\t\t\t\\item \\( \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| \\le \\epsilon \\) for all \\( k \\). Then we have from \\eqref{eq:keylemmarel} (with $\\bm{\\omega}=(v_0^k)^{-1}\\bbv^k$) and \\eqref{eq:keylemmare2} (with $\\bm{\\omega}=(v_0^k)^{-1}\\tbv^k$) that for these \\( k \\)\n\t\t\t\\begin{equation*}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t& 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1          \\\\\n\t\t\t\t\t& \\ge \\min\\{C_3,C_4\\}(\\|(v_0^k)^{-1}\\bbv^k  + z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2).\n\t\t\t\t\\end{aligned}\n\t\t\t\\end{equation*}\n\t\t\tCombining this with \\eqref{eq:dist-vk-wk} and \\eqref{eq:upbd-dist-uk-wk-two-parts}, we see further that\n\t\t\t\\[\n\t\t\t\\begin{aligned}\n\t\t\t\t&\\|\\bm{v}^k - \\bm{w}^k\\|\\\\ & \\ge \\frac{z_0}{\\|\\bm{z}\\|}\\min\\{C_3,C_4\\}(\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2) v_0^k    \\\\\n\t\t\t\t& \\ge \\frac{z_0}{2\\|\\bm{z}\\|}\\min\\{C_3,C_4\\}(\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\| + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|)^2 v_0^k     \\\\\n\t\t\t\t& \\ge \\frac{z_0\\min\\{C_3,C_4\\}}{2\\|\\bm{z}\\|v_0^k}\\|\\bm{u}^k - \\bm{w}^k\\|^2\\ge \\frac{z_0\\min\\{C_3,C_4\\}}{2\\|\\bm{z}\\|\\eta}\\|\\bm{u}^k - \\bm{w}^k\\|^2,\n\t\t\t\\end{aligned}\n\t\t\t\\]\n\t\t\twhere the last inequality holds because \\( \\bm{v}^k\\in \\B(\\eta) \\). The above display contradicts \\eqref{eq:forcontradiction-1st-face} and hence Case 1 cannot happen.\n\n\t\t\t\\item \\( \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\ge \\epsilon \\) for all \\( k \\). Then we have from \\eqref{eq:keylemmarel} and \\eqref{eq:keylemmare4} that for these \\( k \\)\n\t\t\t\\begin{equation*}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t& 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1        \\\\\n\t\t\t\t\t& \\ge \\min\\{C_3,C_5\\}(\\|(v_0^k)^{-1}\\bbv^k  + z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|).\n\t\t\t\t\\end{aligned}\n\t\t\t\\end{equation*}\n\t\t\tUsing this together with \\eqref{eq:dist-vk-wk}, we deduce that for all large \\( k \\),\n\t\t\t\\[\n\t\t\t\\!\\|\\bm{v}^k - \\bm{w}^k\\| \\!\\ge\\! \\frac{z_0\\min\\{C_3,C_5\\}}{\\|\\bm{z}\\|}(\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|) v_0^k.\n\t\t\t\\]\n\t\t\tThis implies that\n\t\t\t\\begin{equation}\\label{eq:upbd-part-dist-uk-wk-case-2}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\|            & \\le M_1\\sqrt{(v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|}, \\\\\n\t\t\t\t\t\\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| & \\le M_1(v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|,\n\t\t\t\t\\end{aligned}\n\t\t\t\\end{equation}\n\t\t\twhere \\( M_1 := \\max\\left\\{\\left(\\frac{z_0}{\\|\\bm{z}\\|}\\min\\{C_3,C_5\\}\\right)^{-1},\\left(\\frac{z_0}{\\|\\bm{z}\\|}\\min\\{C_3,C_5\\}\\right)^{-1/2}\\right\\} \\). Using \\eqref{eq:upbd-part-dist-uk-wk-case-2} together with \\eqref{eq:upbd-dist-uk-wk-two-parts}, we obtain that\n\t\t\t\\begin{equation}\n\t\t\t\\begin{aligned}\\label{eqn:3Ms}\n\t\t\t\t\\|\\bm{u}^k - \\bm{w}^k\\| & \\le M_1v_0^k\\left[\\sqrt{(v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|} + (v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|\\right]   \\\\\n\t\t\t\t%& = M_1\\left(\\sqrt{v_0^k} + \\sqrt{\\|\\bm{v}^k-\\bm{w}^k\\|}\\right)\\sqrt{\\|\\bm{v}^k - \\bm{w}^k\\|}\\\\\n\t\t\t\t&\\stackrel{\\rm (a)}{\\le}M_1\\sqrt{\\eta} \\sqrt{\\|\\bm{v}^k-\\bm{w}^k\\|} + M_1\\|\\bm{v}^k - \\bm{w}^k\\| \\\\\n\t\t\t\t&\\stackrel{\\rm (b)}{\\le} 3M_1\\sqrt{\\eta}\\sqrt{\\|\\bm{v}^k - \\bm{w}^k\\|},\n\t\t\t\\end{aligned}\n\t\t\t\\end{equation}\n\t\t\twhere (a) holds since \\( \\bm{v}^k\\in \\B(\\eta) \\) wherefore $v_0^k \\leq \\eta$, and (b) holds because \\(\\|\\bm{w}^k\\| \\leq \\|\\bm{v}^k\\| \\le \\eta \\) (because the projection onto $\\stdCone$ is nonexpansive and $\\bm 0\\in \\stdCone$), wherefore\n\t\t\t$$\n\t\t\t\\|\\bm{w}^k-\\bm{v}^k\\| = \\sqrt{\\|\\bm{w}^k-\\bm{v}^k\\|}\\sqrt{\\|\\bm{w}^k-\\bm{v}^k\\|} \\leq 2\\sqrt{\\eta}\\sqrt{\\|\\bm{w}^k-\\bm{v}^k\\|}.\n\t\t\t$$\n\t\t\tAltogether, \\eqref{eqn:3Ms} contradicts \\eqref{eq:forcontradiction-1st-face} and hence Case 2 cannot happen.\n\t\t\\end{enumerate}\n\t\tSummarizing the above discussions, we see that Case (I) cannot happen.\n\n\t\t(II):  By passing to a further subsequence, we may assume that \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}< 0 \\) for all \\( k \\). This together with the definition of \\( \\bm{f} \\) gives\n\t\t\\[\n\t\t\\frac{v_0^k}{z_0}[\\inProd{ -z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})}{ (v_0^k)^{-1}\\tbv^k}] < 0.\n\t\t\\]\n\t\tSince $\\frac{v_0^k}{z_0} > 0$, we deduce that $\\inProd{ -z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})}{ (v_0^k)^{-1}\\tbv^k} < 0$ for all $k$. Then it must hold that\n\t\t\\[\n\t\t\\lim_{k\\to \\infty}\\|(v_0^k)^{-1}\\bbv^k + z_0^{-1}\\bbz\\|+\\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\neq 0;\n\t\t\\]\notherwise, we have $(v_0^k)^{-1}\\bbv^k\\to -z_0^{-1}\\bbz$ and $(v_0^k)^{-1}\\tbv^k\\to \\bm{\\alpha}\\circ(z_0\\tbz^{-1})$, which further gives $\\inProd{ -z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})}{ (v_0^k)^{-1}\\tbv^k}\\to \\|z_0^{-1}\\bbz\\|^2+\\|\\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2 = \\|z_0\\bm f\\|^2 > 0$, a contradiction.\n\n\t\tConsequently, there exists $\\epsilon > 0$ such that for all sufficiently large $k$,\n\t\t\\begin{equation}\\label{eq:sum-out-of-neighbor}\n\t\t\t\\|(v_0^k)^{-1}\\bbv^k + z_0^{-1}\\bbz\\|+\\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| \\ge \\epsilon.\n\t\t\\end{equation}\n\t\tConsider the function \\( G:\\R^{m+n}\\to \\R\\cup\\{\\infty\\} \\) defined by\n\t\t\\[\n\t\tG(\\bm{\\xi},\\bm{\\omega}) := \\begin{cases}\n\t\t\t\\displaystyle \\frac{|\\inProd{ z_0^{-1}\\bbz}{ \\bm{\\xi}} + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}|}{\\sqrt{1+\\|\\bm{\\omega}\\|^2}} & {\\rm if}\\,(\\bm{\\xi},\\bm{\\omega})\\in \\Xi,\\,\\|\\bm{\\xi}\\|=1,\\,{\\rm and}\\,\\bm{\\omega}\\in \\Upsilon, \\\\\n\t\t\t\\infty                                                                                                                       & {\\rm otherwise},\n\t\t\\end{cases}\n\t\t\\]\n\t\twhere \\( \\Upsilon = \\{\\bm{\\omega}\\in \\R^n_+\\,| \\,\\prod_{i=1}^n \\omega_i^{\\alpha_i}=1\\} \\) and \\( \\Xi= \\{(\\bm{\\xi},\\bm{\\omega})\\,| \\,\\|\\bm{\\xi} + z_0^{-1}\\bbz\\|+\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| \\ge \\epsilon\\} \\).\n\t\tSince \\( \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\xi}} + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}} = 1+\\inProd{ z_0^{-1}\\bbz}{ \\bm{\\xi}} - \\inProd{ z_0^{-1}(-\\tbz)}{ \\bm{\\omega}}-1 \\), we see from \\eqref{eq:z0-v0-notation}, \\eqref{eq:sum-out-of-neighbor}, {Lemma \\ref{lemma:Lemma-4.1}} and Lemma~\\ref{lemma:inequ-inprod-power-cone} that \\( G \\) is never zero. Moreover, it is clearly lower semicontinuous on any compact set, and\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\!\\liminf_{\\|(\\bm{\\xi},\\bm{\\omega})\\|\\to \\infty}G(\\bm{\\xi},\\bm{\\omega}) \\!=\\!\\!\\! \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty,\\bm{\\omega}\\in \\Upsilon}\\!\\!\\frac{|\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}|}{\\sqrt{1+\\|\\bm{\\omega}\\|^2}}\\!\\overset{\\rm (a)}\\ge\\!\\!\\! \\inf_{\\|\\bm{\\lambda}\\|=1,\\bm{\\lambda}\\in \\R^{n}_+}\\!\\!|\\inProd{ z_0^{-1}\\tbz}{\\bm{\\lambda}}|\\!\\overset{\\rm (b)}\\ge\\!\\! \\min_i |z_0^{-1}\\tz_i| > 0,\n\t\t\\end{aligned}\n\t\t\\]\n\t\twhere (a) may be verified by multiplying numerator and denominator by $1/\\|\\bm{\\omega}\\|$ and (b) holds since \\( z_0^{-1}\\tz_i > 0 \\) for all \\( i \\).\n\t\tThus, \\( C_6:= \\inf G > 0 \\) and we have for all large \\( k \\),\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\frac{\\|\\bm{v}^k - \\bm{w}^k\\|}{\\|\\bm{u}^k - \\bm{w}^k\\|} & \\overset{\\rm (a)}= \\frac{\\|\\bm{v}^k - \\bm{w}^k\\|}{\\|\\bm{w}^k\\|} \\overset{\\rm (b)}\\ge \\frac{\\|\\bm{v}^k - \\bm{w}^k\\|}{\\|\\bm{v}^k\\|} \\\\\n&\\overset{\\rm (c)}=\n\t\t\t\\frac{z_0}{\\|\\bm{z}\\|}\\frac{|\\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ z_0^{-1}\\tbz}{ (v_0^k)^{-1}\\tbv^k}|v_0^k}{\\sqrt{(v^k_0)^2 + \\|\\tbv^k\\|^2}}                                                                                                            \\\\\n\t\t\t& = \\frac{z_0}{\\|\\bm{z}\\|}\\frac{|\\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ z_0^{-1}\\tbz}{ (v_0^k)^{-1}\\tbv^k}|}{\\sqrt{1 + \\|(v_0^k)^{-1}\\tbv^k\\|^2}}\\overset{\\rm (d)}\\ge \\frac{C_6 z_0}{\\|\\bm{z}\\|},\n\t\t\\end{aligned}\n\t\t\\]\n\t\twhere (a) follows from {Lemma \\ref{lemma:Lemma-2.11}}, which states that \\( \\bm{u}^k = \\bm{0} \\) in Case (II), (b) holds because the projection onto the cone is nonexpansive and $\\bm 0$ is in the cone, (c) follows from \\eqref{eq:dist-vk-wk} and (d) follows from \\eqref{eq:sum-out-of-neighbor}, \\eqref{eq:vk0-notation-nonzero} and the definitions of \\( G \\) and \\( C_6 \\). The above display contradicts \\eqref{eq:forcontradiction-1st-face}. Thus, Case (II) also cannot happen.\n\n\t\tSummarizing the above, we conclude that \\eqref{eq:forcontradiction-1st-face} cannot happen. Thus, in view of {\\cite[Lemma 3.12]{LiLoPo20}}, we must indeed have \\( \\gamma_{\\bm{z},\\eta} \\in (0,\\infty] \\) and that the desired error bound follows from {\\cite[Theorem 3.10]{LiLoPo20}}.\n\t\\end{proof}","start":39404,"end":57713}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":["eq:forcontradiction-1st-face"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"构造退化投影序列","source_units":[{"raw_tex":"\\begin{proof}\n\t\tSuppose for a contradiction that \\( \\gamma_{\\bm{z},\\eta} = 0 \\). Then, {in view of \\cite[Lemma 3.12]{LiLoPo20}}, there exist \\( \\widehat{\\bm{v}}\\in \\Fr \\) and a sequence \\( \\{\\bm{v}^k\\}\\subset \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fr \\) such that\n\t\t\\begin{equation}\\label{eq:forcontradiction-1st-face}\n\t\t\t\\lim_{k\\to \\infty}\\bm{v}^k = \\lim_{k\\to \\infty}\\bm{w}^k = \\widehat{\\bm{v}}\\,\\,{\\rm and}\\,\\,\\lim_{k\\to\\infty}\\frac{\\|\\bm{w}^k - \\bm{v}^k\\|^{\\frac{1}{2}}}{\\|\\bm{w}^k - \\bm{u}^k\\|} = 0,\n\t\t\\end{equation}\n\t\twhere \\( \\bm{w}^k = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}^k) \\), \\( \\bm{u}^k = P_{\\Fr}(\\bm{w}^k) \\) and \\( \\bm{u}^k\\neq \\bm{w}^k \\).\n","start":39404,"end":40044}]},{"id":"S029","semantic_type":"notation","kind":"notation","labels":["eq:z0-v0-notation"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"边界归一化尺度","source_units":[{"raw_tex":"\n\t\tDefine, for notational simplicity, \\( z_0 := \\|\\bbz\\| \\) and \\( v^k_0:=\\|\\bbv^k\\| \\).\n\t\tThen, since \\( \\{\\bm{v}^k\\}\\subset \\bd \\gK \\) and \\( \\bm{z} \\in \\bd\\dgK \\) with \\( \\bbz \\neq \\bm{0} \\), we have\\vspace{-0.15 cm}\n\t\t\\begin{equation}\\label{eq:z0-v0-notation}\\vspace{-0.15 cm}\n\t\t\tz_0 = \\|\\bbz\\| = \\prod_{i=1}^n\\left(\\frac{\\tz_i}{\\alpha_i}\\right)^{\\alpha_i}> 0 \\,\\,{\\rm and}\\,\\,v^k_0=\\|\\bbv^k\\| = \\prod_{i=1}^n(\\tv_i^k)^{\\alpha_i}\\,\\,\\forall k.\n\t\t\\end{equation}\n","start":40044,"end":40509}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"排除零首块子序列","source_units":[{"raw_tex":"\n\t\tIf it holds that \\( v_0^k = 0 \\) infinitely often, by passing to a further subsequence, we may assume that \\( v_0^k = 0 \\) for all \\( k \\). Then we have in view of {Lemma \\ref{lemma:Lemma-2.11}} that\\vspace{-0.15 cm}\n\t\t\\[\\vspace{-0.15 cm}\n\t\t\\!\\|\\bm{v}^k - \\bm{w}^k\\| \\!=\\! \\frac1{\\|\\bm{z}\\|}|\\inProd{ \\tbz}{\\tbv^k}|\\!\\overset{\\rm (a)}=\\! \\frac1{\\|\\bm{z}\\|}\\sum_{i=1}^n\\tz_i\\tv^k_i\\!\\ge\\! \\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|} \\|\\tbv^k\\|_1\\!\\ge\\! \\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|} \\|\\tbv^k\\|\n\t\t\\!\\overset{\\rm (b)}=\\!\\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|} \\|\\bm{v}^k\\|,\n\t\t\\]\n\t\twhere (a) holds because \\( \\tv^k_i \\ge 0 \\) and \\( \\tz_i > 0 \\) for all \\( i \\) (see \\eqref{eq:z0-v0-notation}), and (b) holds since \\( \\|\\bbv^k\\| = 0 \\). Since\n\t\t\\( \\|\\bm{w}^k - \\bm{u}^k\\| = \\dist(\\bm{w}^k,\\Fr) \\le \\|\\bm{w}^k\\|\\le \\|\\bm{v}^k\\| \\) as a consequence of the properties of projections, we conclude from this and the above display that\n\t\t\\( \\|\\bm{v}^k - \\bm{w}^k\\| \\ge \\frac{\\min_i \\tz_i}{\\|\\bm{z}\\|}\\|\\bm{w}^k - \\bm{u}^k\\| \\), contradicting \\eqref{eq:forcontradiction-1st-face}.\n\n","start":40509,"end":41565}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","labels":["eq:dist-vk-wk","eq:vk0-notation-nonzero"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"残差拆为两类间隙","source_units":[{"raw_tex":"\t\tThus, by considering a further subsequence if necessary, from now on, we assume\\vspace{-0.15 cm}\n\t\t\\begin{equation}\\label{eq:vk0-notation-nonzero}\\vspace{-0.15 cm}\n\t\t\tv^k_0=\\|\\bbv^k\\| = \\prod_{i=1}^n(\\tv_i^k)^{\\alpha_i} > 0 \\,\\,\\forall k.\n\t\t\\end{equation}\n\t\tUsing {Lemma \\ref{lemma:Lemma-2.11}}, we see that\\vspace{-0.15 cm}\n\t\t\\begin{equation}\\label{eq:dist-vk-wk}\\vspace{-0.15 cm}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\|\\bm{v}^k - \\bm{w}^k\\| & =\\frac1{\\|\\bm{z}\\|}|\\inProd{ \\bm{z}}{\\bm{v}^k}| = \\frac1{\\|\\bm{z}\\|}\\left|\\sum_{i=1}^{m}\\bz_i\\bv^k_i + \\sum_{i=1}^n\\tz_i\\tv^k_i\\right|             \\\\\n\t\t\t\t& = \\frac1{\\|\\bm{z}\\|}\\left|z_0v_0^k + \\sum_{i=1}^{m}\\bz_i\\bv^k_i - \\sum_{i=1}^n(-\\tz_i)\\tv^k_i - z_0v_0^k\\right|                                        \\\\\n\t\t\t\t& = \\frac{z_0v_0^k}{\\|\\bm{z}\\|}\\left| 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1\\right|  \\\\\n\t\t\t\t& = \\frac{z_0}{\\|\\bm{z}\\|}\\left( 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1\\right)v_0^k,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\twhere the last equality holds as \\( \\|z_0^{-1}\\bbz\\|= 1 \\), \\( \\|(v_0^k)^{-1}\\bbv^k\\|= 1 \\) and  \\( \\inProd{ z_0^{-1}\\tbz}{ (v_0^k)^{-1}\\tbv^k} \\ge 1 \\), thanks to \\eqref{eq:z0-v0-notation}, \\eqref{eq:vk0-notation-nonzero} and Lemma~\\ref{lemma:inequ-inprod-power-cone} applied with \\( \\bm{\\zeta} = -z_0^{-1}\\tbz \\).\n\n\t\tLet \\( \\bm{f} \\) be defined as in \\eqref{eq:1st-type-face-ray}. We consider two cases:\n\t\t\\begin{enumerate}[label={(\\Roman*)}]\n\t\t\t\\item \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}\\ge 0 \\) for all sufficiently large \\( k \\).\n\t\t\t\\item \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}< 0 \\) infinitely often.\n\t\t\\end{enumerate}\n","start":41565,"end":43260}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","labels":["eq:upbd-dist-uk-wk-two-parts"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"非负内积时的投影上界","source_units":[{"raw_tex":"\n\t\t(I): By passing to a further subsequence, we may assume that \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}\\ge 0 \\) for all \\( k \\). In this case, if we define\n\t\t\\[\n\t\t\\bm{Q} = \\bm{I}_{m + n} - \\frac{\\bm{z}\\bm{z}^{\\top}}{\\|\\bm{z}\\|^2} - \\frac{\\bm{f}\\bm{f}^{\\top}}{\\|\\bm{f}\\|^2},\n\t\t\\]\n\t\twhere \\( \\bm{f} \\) is as in \\eqref{eq:1st-type-face-ray}, then we see from {Lemma \\ref{lemma:Lemma-2.11}} and \\eqref{eq:z0-v0-notation} that\n\t\t\\begin{equation}\\label{eq:upbd-dist-uk-wk-two-parts}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\|\\bm{u}^k - \\bm{w}^k\\| & = \\|\\bm{Q}\\bm{v}^k\\| = v_0^k\\left\\|\\bm{Q}\\begin{bmatrix}\n\t\t\t\t\t(v_0^k)^{-1}\\bbv^k \\\\ (v_0^k)^{-1}\\tbv^k\n\t\t\t\t\\end{bmatrix}\\right\\| \\\\\n&\\overset{\\rm (a)}= v_0^k{\\Bigg\\|}\\bm{Q}\\begin{bmatrix}\n\t\t\t\t\t(v_0^k)^{-1}\\bbv^k \\\\ (v_0^k)^{-1}\\tbv^k\n\t\t\t\t\\end{bmatrix} \\!-\\! \\bm{Q}\\underbrace{\\begin{bmatrix}\n\t\t\t\t\t\t-z_0^{-1}\\bbz \\\\ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\n\t\t\t\t\\end{bmatrix}}_{z_0\\bm{f}}{\\Bigg\\|} \\\\\n\t\t\t\t& \\le v_0^k\\left[\\|(v_0^k)^{-1}\\bbv^k + z_0^{-1}\\bbz\\| + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\right],\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\twhere (a) holds because %\\( z_0\\bm{f} = \\begin{bmatrix}\n\t\t%\t\t-z_0^{-1}\\bbz^{\\top} & \\alpha\\circ(z_0\\tbz^{-1})^{\\top}\n\t\t%\t\\end{bmatrix}^{\\top}\\)  and\n\t\t\\( \\bm{Q}\\bm{f} = \\bm{0} \\) (an identity which is clear from the definitions).\n","start":43260,"end":44557}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","labels":["eq:keylemmarel"],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"球面间隙的全局二次界","source_units":[{"raw_tex":"\n\t\tNext, in view of \\eqref{eq:z0-v0-notation}, we can apply {Lemma \\ref{lemma:Lemma-4.1}} to obtain \\( C_1 > 0 \\) and \\( \\epsilon_1>0 \\) so that {\\eqref{eq:inequ-p=q=2}} holds with \\( \\overline{\\bm{\\zeta}} = -z_0^{-1}\\bbz \\), i.e.,\n\t\t\\[\n\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_1\\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|^2\n\t\t\\]\n\t\twhenever \\( \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\le \\epsilon_1 \\) and \\( \\|\\bm{\\omega}\\|=1 \\). On the other hand, in view of the positivity of $1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}$ when $\\|\\bm{\\omega}\\|=1$ and $\\bm{\\omega} \\neq -z_0^{-1}\\bbz$ {(see Lemma \\ref{lemma:Lemma-4.1})}, we know that\n\\[\nC_2 := \\inf_{\\|\\bm{\\omega}\\|=1}\\{1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}} \\,|\\, \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\ge \\epsilon_1 \\} > 0.\n\\]\nThis together with the fact $\\|z_0^{-1}\\bbz\\|=1$ (see \\eqref{eq:z0-v0-notation}) implies that\n\t\t\\[\n\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_2 \\ge 0.25 C_2 \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|^2,\n\t\t\\]\n\t\twhenever \\( \\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|\\ge \\epsilon_1 \\) and \\( \\|\\bm{\\omega}\\|=1 \\). We thus have (with $C_3 \\!:=\\! \\min\\{C_1,C_2/4\\}$)\n\\begin{equation}\\label{eq:keylemmarel}\n\t\t\t1 + \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\omega}}\\ge C_3\\|\\bm{\\omega}  + z_0^{-1}\\bbz\\|^2\\ \\ \\ {\\rm whenever}\\ \\ \\|\\bm{\\omega}\\|=1.\n\t\t\\end{equation}\n","start":44557,"end":45846}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","labels":["eq:keylemmare2"],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"乘积间隙的局部二次界","source_units":[{"raw_tex":"\n\t\tIn addition, noting \\eqref{eq:z0-v0-notation} again, we can apply Lemma~\\ref{lemma:inequ-inprod-power-cone} with \\( \\bm{\\zeta} = -z_0^{-1}\\tbz\\in {\\rm int}\\,\\R^n_- \\) to obtain \\( C_4 > 0 \\) and \\( \\epsilon > 0 \\) so that \\eqref{eq:inequ-inprod-power-cone} holds with \\( \\widetilde{\\bm{\\zeta}} = \\bm{\\alpha}\\circ (z_0\\tbz^{-1}) \\), i.e.,\n\t\t\\begin{equation}\\label{eq:keylemmare2}\n\t\t\t-1 + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}\\ge C_4\\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2\n\t\t\\end{equation}\n\t\twhenever \\( \\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\le \\epsilon \\), \\( \\bm{\\omega}\\in {\\rm int}\\,\\R^n_+ \\) and \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i}=1 \\).\n","start":45846,"end":46513}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":["eq:def-h","eq:keylemmare4"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"乘积间隙的远区线性界","source_units":[{"raw_tex":"Furthermore, consider \\( h:\\R^n\\to \\R\\cup\\{\\infty\\} \\) defined by\n\t\t\\begin{equation}\\label{eq:def-h}\n\t\t\th(\\bm{\\omega}) = \\begin{cases}\n\t\t\t\t\\displaystyle\\frac{\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}-1}{\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|} & {\\rm if}\\,\\|\\bm{\\omega}-\\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\ge \\epsilon,\\,\\bm{\\omega}\\in \\Upsilon, \\\\\n\t\t\t\t\\infty                                                                                                    & {\\rm otherwise},\n\t\t\t\\end{cases}\n\t\t\\end{equation}\n\t\twhere \\( \\Upsilon= \\{\\bm{\\omega}\\in \\R^n_+\\,| \\,\\prod_{i=1}^n \\omega_i^{\\alpha_i}=1\\} \\). Then we have\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t& \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty}h(\\bm{\\omega}) = \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty, \\bm{\\omega}\\in \\Upsilon}\\frac{\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}-1}{\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|} \\\\\n\t\t\t& \\ge \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty,\\bm{\\omega}\\in \\R^n_+}\\frac{\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}-1}{\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|}\n\t\t\t\\overset{\\rm (a)}{\\ge} \\inf_{\\|\\bm{\\lambda}\\|=1,\\bm{\\lambda}\\in \\R^n_+}\\inProd{ z_0^{-1}\\tbz}{\\bm{\\lambda}} \\overset{\\rm (b)}\\ge \\min_{1\\le i\\le n}z_0^{-1}\\tz_i > 0.\n\t\t\\end{aligned}\n\t\t\\]\n\t\tHere (a) may be verified by multiplying both numerator and denominator of the left side by $1/\\|\\bm{\\omega}\\|$; (b) holds because \\( z_0^{-1}\\tz_i > 0 \\) for all \\( i \\) (see \\eqref{eq:z0-v0-notation}). Since \\( h \\) in \\eqref{eq:def-h} is also lower semicontinuous on any compact set and {\\bf is always positive},\\footnote{The positivity can be seen by applying Lemma~\\ref{lemma:inequ-inprod-power-cone} with $\\bm\\zeta = -z_0^{-1}\\tbz$.} it must then hold that \\( C_5:= \\inf h > 0 \\). In particular, this means that\n\t\t\\begin{equation}\\label{eq:keylemmare4}\n\t\t\t-1 + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}\\ge C_5\\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\n\t\t\\end{equation}\n\t\twhenever \\( \\|\\bm{\\omega}  - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\ge \\epsilon \\), \\( \\bm{\\omega}\\in {\\rm int}\\,\\R^n_+ \\) and \\( \\prod_{i=1}^n\\omega_i^{\\alpha_i}=1 \\).\n","start":46985,"end":49034}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"近区平方根比值矛盾","source_units":[{"raw_tex":"\n\t\tBy passing to suitable subsequences, we will end up with one of the following two cases:\n\t\t\\begin{enumerate}[label={Case \\arabic*:}]\n\t\t\t\\item \\( \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| \\le \\epsilon \\) for all \\( k \\). Then we have from \\eqref{eq:keylemmarel} (with $\\bm{\\omega}=(v_0^k)^{-1}\\bbv^k$) and \\eqref{eq:keylemmare2} (with $\\bm{\\omega}=(v_0^k)^{-1}\\tbv^k$) that for these \\( k \\)\n\t\t\t\\begin{equation*}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t& 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1          \\\\\n\t\t\t\t\t& \\ge \\min\\{C_3,C_4\\}(\\|(v_0^k)^{-1}\\bbv^k  + z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2).\n\t\t\t\t\\end{aligned}\n\t\t\t\\end{equation*}\n\t\t\tCombining this with \\eqref{eq:dist-vk-wk} and \\eqref{eq:upbd-dist-uk-wk-two-parts}, we see further that\n\t\t\t\\[\n\t\t\t\\begin{aligned}\n\t\t\t\t&\\|\\bm{v}^k - \\bm{w}^k\\|\\\\ & \\ge \\frac{z_0}{\\|\\bm{z}\\|}\\min\\{C_3,C_4\\}(\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2) v_0^k    \\\\\n\t\t\t\t& \\ge \\frac{z_0}{2\\|\\bm{z}\\|}\\min\\{C_3,C_4\\}(\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\| + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|)^2 v_0^k     \\\\\n\t\t\t\t& \\ge \\frac{z_0\\min\\{C_3,C_4\\}}{2\\|\\bm{z}\\|v_0^k}\\|\\bm{u}^k - \\bm{w}^k\\|^2\\ge \\frac{z_0\\min\\{C_3,C_4\\}}{2\\|\\bm{z}\\|\\eta}\\|\\bm{u}^k - \\bm{w}^k\\|^2,\n\t\t\t\\end{aligned}\n\t\t\t\\]\n\t\t\twhere the last inequality holds because \\( \\bm{v}^k\\in \\B(\\eta) \\). The above display contradicts \\eqref{eq:forcontradiction-1st-face} and hence Case 1 cannot happen.\n","start":49034,"end":50599}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":["eq:upbd-part-dist-uk-wk-case-2","eqn:3Ms"],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"远区平方根比值矛盾","source_units":[{"raw_tex":"\n\t\t\t\\item \\( \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\ge \\epsilon \\) for all \\( k \\). Then we have from \\eqref{eq:keylemmarel} and \\eqref{eq:keylemmare4} that for these \\( k \\)\n\t\t\t\\begin{equation*}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t& 1 + \\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} - \\inProd{ z_0^{-1}(-\\tbz)}{ (v_0^k)^{-1}\\tbv^k} - 1        \\\\\n\t\t\t\t\t& \\ge \\min\\{C_3,C_5\\}(\\|(v_0^k)^{-1}\\bbv^k  + z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|).\n\t\t\t\t\\end{aligned}\n\t\t\t\\end{equation*}\n\t\t\tUsing this together with \\eqref{eq:dist-vk-wk}, we deduce that for all large \\( k \\),\n\t\t\t\\[\n\t\t\t\\!\\|\\bm{v}^k - \\bm{w}^k\\| \\!\\ge\\! \\frac{z_0\\min\\{C_3,C_5\\}}{\\|\\bm{z}\\|}(\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\|^2 + \\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|) v_0^k.\n\t\t\t\\]\n\t\t\tThis implies that\n\t\t\t\\begin{equation}\\label{eq:upbd-part-dist-uk-wk-case-2}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t\\|(v_0^k)^{-1}\\bbv^k  +  z_0^{-1}\\bbz\\|            & \\le M_1\\sqrt{(v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|}, \\\\\n\t\t\t\t\t\\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| & \\le M_1(v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|,\n\t\t\t\t\\end{aligned}\n\t\t\t\\end{equation}\n\t\t\twhere \\( M_1 := \\max\\left\\{\\left(\\frac{z_0}{\\|\\bm{z}\\|}\\min\\{C_3,C_5\\}\\right)^{-1},\\left(\\frac{z_0}{\\|\\bm{z}\\|}\\min\\{C_3,C_5\\}\\right)^{-1/2}\\right\\} \\). Using \\eqref{eq:upbd-part-dist-uk-wk-case-2} together with \\eqref{eq:upbd-dist-uk-wk-two-parts}, we obtain that\n\t\t\t\\begin{equation}\n\t\t\t\\begin{aligned}\\label{eqn:3Ms}\n\t\t\t\t\\|\\bm{u}^k - \\bm{w}^k\\| & \\le M_1v_0^k\\left[\\sqrt{(v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|} + (v_0^k)^{-1}\\|\\bm{v}^k - \\bm{w}^k\\|\\right]   \\\\\n\t\t\t\t%& = M_1\\left(\\sqrt{v_0^k} + \\sqrt{\\|\\bm{v}^k-\\bm{w}^k\\|}\\right)\\sqrt{\\|\\bm{v}^k - \\bm{w}^k\\|}\\\\\n\t\t\t\t&\\stackrel{\\rm (a)}{\\le}M_1\\sqrt{\\eta} \\sqrt{\\|\\bm{v}^k-\\bm{w}^k\\|} + M_1\\|\\bm{v}^k - \\bm{w}^k\\| \\\\\n\t\t\t\t&\\stackrel{\\rm (b)}{\\le} 3M_1\\sqrt{\\eta}\\sqrt{\\|\\bm{v}^k - \\bm{w}^k\\|},\n\t\t\t\\end{aligned}\n\t\t\t\\end{equation}\n\t\t\twhere (a) holds since \\( \\bm{v}^k\\in \\B(\\eta) \\) wherefore $v_0^k \\leq \\eta$, and (b) holds because \\(\\|\\bm{w}^k\\| \\leq \\|\\bm{v}^k\\| \\le \\eta \\) (because the projection onto $\\stdCone$ is nonexpansive and $\\bm 0\\in \\stdCone$), wherefore\n\t\t\t$$\n\t\t\t\\|\\bm{w}^k-\\bm{v}^k\\| = \\sqrt{\\|\\bm{w}^k-\\bm{v}^k\\|}\\sqrt{\\|\\bm{w}^k-\\bm{v}^k\\|} \\leq 2\\sqrt{\\eta}\\sqrt{\\|\\bm{w}^k-\\bm{v}^k\\|}.\n\t\t\t$$\n\t\t\tAltogether, \\eqref{eqn:3Ms} contradicts \\eqref{eq:forcontradiction-1st-face} and hence Case 2 cannot happen.\n\t\t\\end{enumerate}\n\t\tSummarizing the above discussions, we see that Case (I) cannot happen.\n","start":50599,"end":53118}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":["eq:sum-out-of-neighbor"],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"负内积迫使离开接触点","source_units":[{"raw_tex":"\n\t\t(II):  By passing to a further subsequence, we may assume that \\( \\inProd{ \\bm{f}}{ \\bm{v}^k}< 0 \\) for all \\( k \\). This together with the definition of \\( \\bm{f} \\) gives\n\t\t\\[\n\t\t\\frac{v_0^k}{z_0}[\\inProd{ -z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})}{ (v_0^k)^{-1}\\tbv^k}] < 0.\n\t\t\\]\n\t\tSince $\\frac{v_0^k}{z_0} > 0$, we deduce that $\\inProd{ -z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})}{ (v_0^k)^{-1}\\tbv^k} < 0$ for all $k$. Then it must hold that\n\t\t\\[\n\t\t\\lim_{k\\to \\infty}\\|(v_0^k)^{-1}\\bbv^k + z_0^{-1}\\bbz\\|+\\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|\\neq 0;\n\t\t\\]\notherwise, we have $(v_0^k)^{-1}\\bbv^k\\to -z_0^{-1}\\bbz$ and $(v_0^k)^{-1}\\tbv^k\\to \\bm{\\alpha}\\circ(z_0\\tbz^{-1})$, which further gives $\\inProd{ -z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ \\bm{\\alpha}\\circ(z_0\\tbz^{-1})}{ (v_0^k)^{-1}\\tbv^k}\\to \\|z_0^{-1}\\bbz\\|^2+\\|\\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\|^2 = \\|z_0\\bm f\\|^2 > 0$, a contradiction.\n\n\t\tConsequently, there exists $\\epsilon > 0$ such that for all sufficiently large $k$,\n\t\t\\begin{equation}\\label{eq:sum-out-of-neighbor}\n\t\t\t\\|(v_0^k)^{-1}\\bbv^k + z_0^{-1}\\bbz\\|+\\|(v_0^k)^{-1}\\tbv^k - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| \\ge \\epsilon.\n\t\t\\end{equation}\n","start":53118,"end":54379}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"远离接触点的归一化正下界","source_units":[{"raw_tex":"\t\tConsider the function \\( G:\\R^{m+n}\\to \\R\\cup\\{\\infty\\} \\) defined by\n\t\t\\[\n\t\tG(\\bm{\\xi},\\bm{\\omega}) := \\begin{cases}\n\t\t\t\\displaystyle \\frac{|\\inProd{ z_0^{-1}\\bbz}{ \\bm{\\xi}} + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}|}{\\sqrt{1+\\|\\bm{\\omega}\\|^2}} & {\\rm if}\\,(\\bm{\\xi},\\bm{\\omega})\\in \\Xi,\\,\\|\\bm{\\xi}\\|=1,\\,{\\rm and}\\,\\bm{\\omega}\\in \\Upsilon, \\\\\n\t\t\t\\infty                                                                                                                       & {\\rm otherwise},\n\t\t\\end{cases}\n\t\t\\]\n\t\twhere \\( \\Upsilon = \\{\\bm{\\omega}\\in \\R^n_+\\,| \\,\\prod_{i=1}^n \\omega_i^{\\alpha_i}=1\\} \\) and \\( \\Xi= \\{(\\bm{\\xi},\\bm{\\omega})\\,| \\,\\|\\bm{\\xi} + z_0^{-1}\\bbz\\|+\\|\\bm{\\omega} - \\bm{\\alpha}\\circ(z_0\\tbz^{-1})\\| \\ge \\epsilon\\} \\).\n\t\tSince \\( \\inProd{ z_0^{-1}\\bbz}{ \\bm{\\xi}} + \\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}} = 1+\\inProd{ z_0^{-1}\\bbz}{ \\bm{\\xi}} - \\inProd{ z_0^{-1}(-\\tbz)}{ \\bm{\\omega}}-1 \\), we see from \\eqref{eq:z0-v0-notation}, \\eqref{eq:sum-out-of-neighbor}, {Lemma \\ref{lemma:Lemma-4.1}} and Lemma~\\ref{lemma:inequ-inprod-power-cone} that \\( G \\) is never zero. Moreover, it is clearly lower semicontinuous on any compact set, and\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\!\\liminf_{\\|(\\bm{\\xi},\\bm{\\omega})\\|\\to \\infty}G(\\bm{\\xi},\\bm{\\omega}) \\!=\\!\\!\\! \\liminf_{\\|\\bm{\\omega}\\|\\to \\infty,\\bm{\\omega}\\in \\Upsilon}\\!\\!\\frac{|\\inProd{ z_0^{-1}\\tbz}{ \\bm{\\omega}}|}{\\sqrt{1+\\|\\bm{\\omega}\\|^2}}\\!\\overset{\\rm (a)}\\ge\\!\\!\\! \\inf_{\\|\\bm{\\lambda}\\|=1,\\bm{\\lambda}\\in \\R^{n}_+}\\!\\!|\\inProd{ z_0^{-1}\\tbz}{\\bm{\\lambda}}|\\!\\overset{\\rm (b)}\\ge\\!\\! \\min_i |z_0^{-1}\\tz_i| > 0,\n\t\t\\end{aligned}\n\t\t\\]\n\t\twhere (a) may be verified by multiplying numerator and denominator by $1/\\|\\bm{\\omega}\\|$ and (b) holds since \\( z_0^{-1}\\tz_i > 0 \\) for all \\( i \\).\n","start":54379,"end":56130}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"负内积分支的比值矛盾","source_units":[{"raw_tex":"\t\tThus, \\( C_6:= \\inf G > 0 \\) and we have for all large \\( k \\),\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\frac{\\|\\bm{v}^k - \\bm{w}^k\\|}{\\|\\bm{u}^k - \\bm{w}^k\\|} & \\overset{\\rm (a)}= \\frac{\\|\\bm{v}^k - \\bm{w}^k\\|}{\\|\\bm{w}^k\\|} \\overset{\\rm (b)}\\ge \\frac{\\|\\bm{v}^k - \\bm{w}^k\\|}{\\|\\bm{v}^k\\|} \\\\\n&\\overset{\\rm (c)}=\n\t\t\t\\frac{z_0}{\\|\\bm{z}\\|}\\frac{|\\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ z_0^{-1}\\tbz}{ (v_0^k)^{-1}\\tbv^k}|v_0^k}{\\sqrt{(v^k_0)^2 + \\|\\tbv^k\\|^2}}                                                                                                            \\\\\n\t\t\t& = \\frac{z_0}{\\|\\bm{z}\\|}\\frac{|\\inProd{ z_0^{-1}\\bbz}{ (v_0^k)^{-1}\\bbv^k} + \\inProd{ z_0^{-1}\\tbz}{ (v_0^k)^{-1}\\tbv^k}|}{\\sqrt{1 + \\|(v_0^k)^{-1}\\tbv^k\\|^2}}\\overset{\\rm (d)}\\ge \\frac{C_6 z_0}{\\|\\bm{z}\\|},\n\t\t\\end{aligned}\n\t\t\\]\n\t\twhere (a) follows from {Lemma \\ref{lemma:Lemma-2.11}}, which states that \\( \\bm{u}^k = \\bm{0} \\) in Case (II), (b) holds because the projection onto the cone is nonexpansive and $\\bm 0$ is in the cone, (c) follows from \\eqref{eq:dist-vk-wk} and (d) follows from \\eqref{eq:sum-out-of-neighbor}, \\eqref{eq:vk0-notation-nonzero} and the definitions of \\( G \\) and \\( C_6 \\). The above display contradicts \\eqref{eq:forcontradiction-1st-face}. Thus, Case (II) also cannot happen.\n","start":56130,"end":57417}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"正面常数转为误差界","source_units":[{"raw_tex":"\n\t\tSummarizing the above, we conclude that \\eqref{eq:forcontradiction-1st-face} cannot happen. Thus, in view of {\\cite[Lemma 3.12]{LiLoPo20}}, we must indeed have \\( \\gamma_{\\bm{z},\\eta} \\in (0,\\infty] \\) and that the desired error bound follows from {\\cite[Theorem 3.10]{LiLoPo20}}.\n\t\\end{proof}","start":57417,"end":57713}]},{"id":"S043","semantic_type":"lemma","kind":"lemma","labels":["remark:F-r-order-1/2"],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"射线指数的渐近最优性","source_units":[{"raw_tex":"\\begin{remark}[Optimality of the error bound in Theorem~\\ref{thm:Holderian-1/2-1st-type-face}]\\label{remark:F-r-order-1/2}\n\t\t\\!\\!\\!Let \\( \\bm{z}\\!\\in\\! \\!\\bd\\dgK\\backslash\\!\\{\\bm{0}\\} \\) with \\( \\bbz \\neq \\bm{0} \\) and let \\( \\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Then necessarily \\( \\tz_i > 0 \\) for all \\( i \\). Moreover, we also know from the definition that \\( \\alpha_i > 0 \\) for all \\( i \\). Now, consider the continuous function \\( \\bm{q}:(0,\\alpha_1)\\to \\{\\bm{z}\\}^\\perp \\) defined by $\\epsilon \\mapsto \\bm{q}_\\epsilon:=(\\bar{\\bm{q}}_\\epsilon,\\widetilde{\\bm{q}}_\\epsilon)$ where\n","start":57718,"end":58303},{"raw_tex":"\t\tThus \\( | \\cdot | ^{\\frac{1}{2}} \\) satisfies the asymptotic optimality criterion {(cf. \\cite[Definition 3.1]{LiLoPo21}}) for \\( \\gK \\) and \\( \\bm{z} \\){, which implies that the error bound is optimal in the sense of \\cite[Theorem 3.2(b)]{LiLoPo21}.}\n\t\\end{remark}","start":63744,"end":64010}]},{"id":"S044","semantic_type":"proof","kind":"proof","labels":["eq:distpower","eq:lowerlowerboundpower","remark:F-r-order-1/2"],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"射线最优性完整构造论证","source_units":[{"raw_tex":"\t\t\\!\\!\\!Let \\( \\bm{z}\\!\\in\\! \\!\\bd\\dgK\\backslash\\!\\{\\bm{0}\\} \\) with \\( \\bbz \\neq \\bm{0} \\) and let \\( \\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Then necessarily \\( \\tz_i > 0 \\) for all \\( i \\). Moreover, we also know from the definition that \\( \\alpha_i > 0 \\) for all \\( i \\). Now, consider the continuous function \\( \\bm{q}:(0,\\alpha_1)\\to \\{\\bm{z}\\}^\\perp \\) defined by $\\epsilon \\mapsto \\bm{q}_\\epsilon:=(\\bar{\\bm{q}}_\\epsilon,\\widetilde{\\bm{q}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{q}}_\\epsilon = -\\bbz/\\|\\bbz\\|^2,\\,\\,(\\widetilde{\\bm{q}}_\\epsilon)_1 = (\\alpha_1 - \\epsilon)\\tz_1^{-1},\\,\\,(\\widetilde{\\bm{q}}_\\epsilon)_2 = (\\alpha_2 + \\epsilon)\\tz_2^{-1},\\,\\mbox{and} \\,(\\widetilde{\\bm{q}}_\\epsilon)_i = \\alpha_i \\tz_i^{-1}, \\,\\forall i \\ge 3.\n\t\t\\]\n\t\tNotice that \\( \\bm{q}_\\epsilon \\) only differs from the \\( \\bm{f} \\) in \\eqref{eq:1st-type-face-ray} in two entries. One can check that \\( \\inProd{ \\bm{z}}{\\bm{q}_\\epsilon} = 0 \\) and \\( \\bm{q}_\\epsilon\\to \\bm{f}\\in \\Fr\\backslash\\{\\bm{0}\\} \\) as $\\epsilon \\downarrow 0$.\n\t\tMoreover, we have\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t&\\prod_{i=1}^n (\\widetilde{\\bm{q}}_\\epsilon)_i^{\\alpha_i}  = (\\alpha_1 - \\epsilon)^{\\alpha_1}(\\alpha_2 + \\epsilon)^{\\alpha_2}\\tz_1^{-\\alpha_1}\\tz_2^{-\\alpha_2}\\prod_{i=3}^n\\left(\\frac{\\alpha_i}{\\tz_i}\\right)^{\\alpha_i}          \\\\\n\t\t\t& =  \\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2}\\prod_{i=1}^n\\left(\\frac{\\alpha_i}{\\tz_i}\\right)^{\\alpha_i}\n\t\t\t\\overset{\\rm (a)}= \\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2} \\|\\bbz\\|^{-1}                                                                                          \\\\\n\t\t\t& = \\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2}\\|\\bar{\\bm{q}}_\\epsilon\\|,\n\t\t\\end{aligned}\n\t\t\\]\n\t\twhere (a) holds because \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz\\neq \\bm{0} \\). In view of this, if we define a continuous function \\( \\bm{p}:(0,\\alpha_1)\\to \\gK \\)  by $\\epsilon \\mapsto \\bm{p}_\\epsilon := (\\bar{\\bm{p}}_\\epsilon,\\widetilde{\\bm{p}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{p}}_\\epsilon := -\\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2}\\frac{\\bbz}{\\|\\bbz\\|^2}\\quad{\\rm and}\\quad \\widetilde{\\bm{p}}_\\epsilon := \\widetilde{\\bm{q}}_\\epsilon;\n\t\t\\]\n\t\tthen it is clear that \\( \\bm{p}_\\epsilon \\in \\gK \\), and we can compute that\n\t\t\\begin{equation}\\label{eq:distpower}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\dist(\\bm{q}_\\epsilon,\\gK) & \\le \\|\\bm{q}_\\epsilon - \\bm{p}_\\epsilon\\| = \\frac1{\\|\\bbz\\|}\\left|\\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2} - 1\\right| \\\\\n\t\t\t\t& = \\frac1{\\|\\bbz\\|}\\left|(1 - \\epsilon + O(\\epsilon^2))(1 + \\epsilon + O(\\epsilon^2)) - 1\\right| = O(\\epsilon^2).\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\tNext, we estimate \\( \\dist(\\bm{q}_\\epsilon,\\Fr) \\).\n\t\tNotice that \\( \\inProd{ \\bm{q}_\\epsilon}{\\bm{f}} > 0 \\) for all sufficiently small \\( \\epsilon \\) because \\( \\bm{q}_\\epsilon \\to \\bm{f} \\). Hence, using the definition of \\( \\Fr \\) and {Lemma \\ref{lemma:Lemma-2.11}}, we see that\n\t\t\\[\n\t\t\\dist(\\bm{q}_\\epsilon,\\Fr)^2 = \\left\\|\\bm{q}_\\epsilon - \\frac{\\inProd{ \\bm{q}_\\epsilon}{ \\bm{f}}}{\\|\\bm{f}\\|^2}\\bm{f}\\right\\|^2 = \\|\\bm{q}_\\epsilon\\|^2 - \\frac{(\\inProd{ \\bm{q}_\\epsilon}{ \\bm{f}})^2}{\\|\\bm{f}\\|^2}.\n\t\t\\]\n\t\tA direct computation then shows that\n\t\t\\begin{align}\n\t\t\t\\|\\bm{q}_\\epsilon\\|^2 & = \\frac1{\\|\\bbz\\|^2} + (\\alpha_1-\\epsilon)^2\\tz_1^{-2} + (\\alpha_2+\\epsilon)^2\\tz_2^{-2} + \\sum_{i=3}^n \\alpha^2_i \\tz_i^{-2}                       \\notag \\\\\n\t\t\t& = \\frac1{\\|\\bbz\\|^2} + \\sum_{i=1}^n \\alpha^2_i \\tz_i^{-2} + 2\\epsilon(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2}) + \\epsilon^2(\\tz_1^{-2} + \\tz_2^{-2}) \\notag\\\\\n\t\t\t& = \\|\\bm{f}\\|^2 + 2\\epsilon(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2}) + \\epsilon^2(\\tz_1^{-2} + \\tz_2^{-2}),\\notag\n\t\t\\end{align}\n\t\twhere the last equality follows from the definition of \\( \\bm{f} \\) in \\eqref{eq:1st-type-face-ray}.\n\t\tFurthermore,\n\t\t\\begin{align}\n(\\inProd{ \\bm{q}_\\epsilon}{ \\bm{f}})^2 & =\\textstyle \\left(\\frac1{\\|\\bbz\\|^2} + \\alpha_1(\\alpha_1-\\epsilon)\\tz_1^{-2} + \\alpha_2(\\alpha_2+\\epsilon)\\tz_2^{-2} + \\sum_{i=3}^n \\alpha^2_i \\tz_i^{-2}\\right)^2 \\notag\\\\\n\t\t\t& = \\left[ \\|\\bm{f}\\|^2  + \\epsilon (\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})\\right]^2                                                                    \\notag\\\\\n\t\t\t& = \\|\\bm{f}\\|^4 + 2\\epsilon\\|\\bm{f}\\|^2 (\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2}) + \\epsilon^2(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2.\\notag\n\t\t\\end{align}\n\t\tCombining the above three identities, we deduce further that\n\t\t\\begin{align}\\label{eq:lowerlowerboundpower}\n\t\t\t\\dist(\\bm{q}_\\epsilon,\\Fr)^2 & = \\epsilon^2\\left(\\tz_1^{-2} + \\tz_2^{-2} - \\frac{(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2}{\\|\\bm{f}\\|^2}\\right)\\nonumber                          \\\\\n\t\t\t& \\ge \\epsilon^2\\left(\\tz_1^{-2} + \\tz_2^{-2} - \\frac{(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2}{\\alpha^2_2\\tz_2^{-2} + \\alpha^2_1\\tz_1^{-2}}\\right),\n\t\t\\end{align}\n\t\twhere the inequality follows from the definition of \\( \\bm{f} \\). Now, notice that in \\eqref{eq:lowerlowerboundpower}, the scalar term is strictly greater than zero, because\n\t\t\\[\n\t\t(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2 < (\\alpha_2\\tz_2^{-2} + \\alpha_1\\tz_1^{-2})^2 \\le (\\tz_1^{-2} + \\tz_2^{-2})(\\alpha^2_2\\tz_2^{-2} + \\alpha^2_1\\tz_1^{-2}),\n\t\t\\]\n\t\twhere the strict inequality holds because \\( \\alpha_i\\tz_i^{-2} > 0 \\) for \\( i = 1 \\), \\( 2 \\), and the last inequality follows from the Cauchy-Schwarz inequality.\n\t\tThis together with \\eqref{eq:lowerlowerboundpower} shows that \\( \\dist(\\bm{q}_\\epsilon,\\Fr) = \\Omega(\\epsilon) \\). Combining this with \\eqref{eq:distpower}, we obtain\n\t\t$\n\t\t\\limsup_{\\epsilon\\downarrow 0}\\frac{\\dist(\\bm{q}_\\epsilon,\\gK)^\\frac12}{\\dist(\\bm{q}_\\epsilon,\\Fr)} < \\infty\n\t\t$.\n","start":57841,"end":63744}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":["remark:F-r-order-1/2"],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"保持超平面的两坐标扰动","source_units":[{"raw_tex":"\t\t\\!\\!\\!Let \\( \\bm{z}\\!\\in\\! \\!\\bd\\dgK\\backslash\\!\\{\\bm{0}\\} \\) with \\( \\bbz \\neq \\bm{0} \\) and let \\( \\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Then necessarily \\( \\tz_i > 0 \\) for all \\( i \\). Moreover, we also know from the definition that \\( \\alpha_i > 0 \\) for all \\( i \\). Now, consider the continuous function \\( \\bm{q}:(0,\\alpha_1)\\to \\{\\bm{z}\\}^\\perp \\) defined by $\\epsilon \\mapsto \\bm{q}_\\epsilon:=(\\bar{\\bm{q}}_\\epsilon,\\widetilde{\\bm{q}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{q}}_\\epsilon = -\\bbz/\\|\\bbz\\|^2,\\,\\,(\\widetilde{\\bm{q}}_\\epsilon)_1 = (\\alpha_1 - \\epsilon)\\tz_1^{-1},\\,\\,(\\widetilde{\\bm{q}}_\\epsilon)_2 = (\\alpha_2 + \\epsilon)\\tz_2^{-1},\\,\\mbox{and} \\,(\\widetilde{\\bm{q}}_\\epsilon)_i = \\alpha_i \\tz_i^{-1}, \\,\\forall i \\ge 3.\n\t\t\\]\n\t\tNotice that \\( \\bm{q}_\\epsilon \\) only differs from the \\( \\bm{f} \\) in \\eqref{eq:1st-type-face-ray} in two entries. One can check that \\( \\inProd{ \\bm{z}}{\\bm{q}_\\epsilon} = 0 \\) and \\( \\bm{q}_\\epsilon\\to \\bm{f}\\in \\Fr\\backslash\\{\\bm{0}\\} \\) as $\\epsilon \\downarrow 0$.\n","start":57841,"end":58858}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":["eq:distpower","remark:F-r-order-1/2"],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"锥距离的二阶上界","source_units":[{"raw_tex":"\t\tMoreover, we have\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t&\\prod_{i=1}^n (\\widetilde{\\bm{q}}_\\epsilon)_i^{\\alpha_i}  = (\\alpha_1 - \\epsilon)^{\\alpha_1}(\\alpha_2 + \\epsilon)^{\\alpha_2}\\tz_1^{-\\alpha_1}\\tz_2^{-\\alpha_2}\\prod_{i=3}^n\\left(\\frac{\\alpha_i}{\\tz_i}\\right)^{\\alpha_i}          \\\\\n\t\t\t& =  \\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2}\\prod_{i=1}^n\\left(\\frac{\\alpha_i}{\\tz_i}\\right)^{\\alpha_i}\n\t\t\t\\overset{\\rm (a)}= \\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2} \\|\\bbz\\|^{-1}                                                                                          \\\\\n\t\t\t& = \\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2}\\|\\bar{\\bm{q}}_\\epsilon\\|,\n\t\t\\end{aligned}\n\t\t\\]\n\t\twhere (a) holds because \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz\\neq \\bm{0} \\). In view of this, if we define a continuous function \\( \\bm{p}:(0,\\alpha_1)\\to \\gK \\)  by $\\epsilon \\mapsto \\bm{p}_\\epsilon := (\\bar{\\bm{p}}_\\epsilon,\\widetilde{\\bm{p}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{p}}_\\epsilon := -\\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2}\\frac{\\bbz}{\\|\\bbz\\|^2}\\quad{\\rm and}\\quad \\widetilde{\\bm{p}}_\\epsilon := \\widetilde{\\bm{q}}_\\epsilon;\n\t\t\\]\n\t\tthen it is clear that \\( \\bm{p}_\\epsilon \\in \\gK \\), and we can compute that\n\t\t\\begin{equation}\\label{eq:distpower}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\dist(\\bm{q}_\\epsilon,\\gK) & \\le \\|\\bm{q}_\\epsilon - \\bm{p}_\\epsilon\\| = \\frac1{\\|\\bbz\\|}\\left|\\left(1 - \\frac{\\epsilon}{\\alpha_1}\\right)^{\\alpha_1}\\left(1 + \\frac{\\epsilon}{\\alpha_2}\\right)^{\\alpha_2} - 1\\right| \\\\\n\t\t\t\t& = \\frac1{\\|\\bbz\\|}\\left|(1 - \\epsilon + O(\\epsilon^2))(1 + \\epsilon + O(\\epsilon^2)) - 1\\right| = O(\\epsilon^2).\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\n","start":58858,"end":60745}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":["eq:lowerlowerboundpower","remark:F-r-order-1/2"],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"射线距离的一阶下界","source_units":[{"raw_tex":"\t\tNext, we estimate \\( \\dist(\\bm{q}_\\epsilon,\\Fr) \\).\n\t\tNotice that \\( \\inProd{ \\bm{q}_\\epsilon}{\\bm{f}} > 0 \\) for all sufficiently small \\( \\epsilon \\) because \\( \\bm{q}_\\epsilon \\to \\bm{f} \\). Hence, using the definition of \\( \\Fr \\) and {Lemma \\ref{lemma:Lemma-2.11}}, we see that\n\t\t\\[\n\t\t\\dist(\\bm{q}_\\epsilon,\\Fr)^2 = \\left\\|\\bm{q}_\\epsilon - \\frac{\\inProd{ \\bm{q}_\\epsilon}{ \\bm{f}}}{\\|\\bm{f}\\|^2}\\bm{f}\\right\\|^2 = \\|\\bm{q}_\\epsilon\\|^2 - \\frac{(\\inProd{ \\bm{q}_\\epsilon}{ \\bm{f}})^2}{\\|\\bm{f}\\|^2}.\n\t\t\\]\n\t\tA direct computation then shows that\n\t\t\\begin{align}\n\t\t\t\\|\\bm{q}_\\epsilon\\|^2 & = \\frac1{\\|\\bbz\\|^2} + (\\alpha_1-\\epsilon)^2\\tz_1^{-2} + (\\alpha_2+\\epsilon)^2\\tz_2^{-2} + \\sum_{i=3}^n \\alpha^2_i \\tz_i^{-2}                       \\notag \\\\\n\t\t\t& = \\frac1{\\|\\bbz\\|^2} + \\sum_{i=1}^n \\alpha^2_i \\tz_i^{-2} + 2\\epsilon(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2}) + \\epsilon^2(\\tz_1^{-2} + \\tz_2^{-2}) \\notag\\\\\n\t\t\t& = \\|\\bm{f}\\|^2 + 2\\epsilon(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2}) + \\epsilon^2(\\tz_1^{-2} + \\tz_2^{-2}),\\notag\n\t\t\\end{align}\n\t\twhere the last equality follows from the definition of \\( \\bm{f} \\) in \\eqref{eq:1st-type-face-ray}.\n\t\tFurthermore,\n\t\t\\begin{align}\n(\\inProd{ \\bm{q}_\\epsilon}{ \\bm{f}})^2 & =\\textstyle \\left(\\frac1{\\|\\bbz\\|^2} + \\alpha_1(\\alpha_1-\\epsilon)\\tz_1^{-2} + \\alpha_2(\\alpha_2+\\epsilon)\\tz_2^{-2} + \\sum_{i=3}^n \\alpha^2_i \\tz_i^{-2}\\right)^2 \\notag\\\\\n\t\t\t& = \\left[ \\|\\bm{f}\\|^2  + \\epsilon (\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})\\right]^2                                                                    \\notag\\\\\n\t\t\t& = \\|\\bm{f}\\|^4 + 2\\epsilon\\|\\bm{f}\\|^2 (\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2}) + \\epsilon^2(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2.\\notag\n\t\t\\end{align}\n\t\tCombining the above three identities, we deduce further that\n\t\t\\begin{align}\\label{eq:lowerlowerboundpower}\n\t\t\t\\dist(\\bm{q}_\\epsilon,\\Fr)^2 & = \\epsilon^2\\left(\\tz_1^{-2} + \\tz_2^{-2} - \\frac{(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2}{\\|\\bm{f}\\|^2}\\right)\\nonumber                          \\\\\n\t\t\t& \\ge \\epsilon^2\\left(\\tz_1^{-2} + \\tz_2^{-2} - \\frac{(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2}{\\alpha^2_2\\tz_2^{-2} + \\alpha^2_1\\tz_1^{-2}}\\right),\n\t\t\\end{align}\n\t\twhere the inequality follows from the definition of \\( \\bm{f} \\). Now, notice that in \\eqref{eq:lowerlowerboundpower}, the scalar term is strictly greater than zero, because\n\t\t\\[\n\t\t(\\alpha_2\\tz_2^{-2} - \\alpha_1\\tz_1^{-2})^2 < (\\alpha_2\\tz_2^{-2} + \\alpha_1\\tz_1^{-2})^2 \\le (\\tz_1^{-2} + \\tz_2^{-2})(\\alpha^2_2\\tz_2^{-2} + \\alpha^2_1\\tz_1^{-2}),\n\t\t\\]\n\t\twhere the strict inequality holds because \\( \\alpha_i\\tz_i^{-2} > 0 \\) for \\( i = 1 \\), \\( 2 \\), and the last inequality follows from the Cauchy-Schwarz inequality.\n","start":60745,"end":63455}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":["remark:F-r-order-1/2"],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"验证平方根最优性判据","source_units":[{"raw_tex":"\t\tThis together with \\eqref{eq:lowerlowerboundpower} shows that \\( \\dist(\\bm{q}_\\epsilon,\\Fr) = \\Omega(\\epsilon) \\). Combining this with \\eqref{eq:distpower}, we obtain\n\t\t$\n\t\t\\limsup_{\\epsilon\\downarrow 0}\\frac{\\dist(\\bm{q}_\\epsilon,\\gK)^\\frac12}{\\dist(\\bm{q}_\\epsilon,\\Fr)} < \\infty\n\t\t$.\n\t\tThus \\( | \\cdot | ^{\\frac{1}{2}} \\) satisfies the asymptotic optimality criterion {(cf. \\cite[Definition 3.1]{LiLoPo21}}) for \\( \\gK \\) and \\( \\bm{z} \\){, which implies that the error bound is optimal in the sense of \\cite[Theorem 3.2(b)]{LiLoPo21}.}\n","start":63455,"end":63997}]},{"id":"S049","semantic_type":"theorem","kind":"theorem","labels":["eq:gamma-beta-2nd-type-face","thm:xface"],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"坐标面的幂次误差界","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm:xface}\n\t\tLet \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz = \\bm{0} \\) and let \\( \\Fhatz := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Let \\( \\stdIndex := \\{i\\,| \\,\\tz_i > 0\\} \\),\\footnote{Since \\( \\bbz = \\bm{0} \\) and \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\), we must have \\( \\emptyset\\neq \\stdIndex \\subsetneq \\{1,2,\\ldots,n\\} \\).} \\( \\eta > 0 \\) and define \\( \\beta:= \\sum_{i\\in \\stdIndex}\\alpha_i \\) and\n\t\t\\begin{equation}\\label{eq:gamma-beta-2nd-type-face}\n\t\t \\!\\gamma_{\\bm{z},\\eta} \\!:=\\! \\inf_{\\bm{v}}\\left\\{\\frac{\\|\\bm{v} - \\bm{w}\\|^\\beta}{\\|\\bm{u} - \\bm{w}\\|}\\,\\bigg| \\,\n\\begin{array}{c}\n\\bm{v}\\in \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fhatz,\\,\\bm{w} = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}),\\\\\n\\bm{u} = P_{\\Fhatz}(\\bm{w}),\\,\\bm{u} \\neq \\bm{w}\n\\end{array}\\right\\}.\n\t\t\\end{equation}\n\t\tThen it holds that \\( \\gamma_{\\bm{z},\\eta}\\in (0,\\infty] \\) and that\n\t\t\\[\n\t\t\\dist(\\bm{q},\\Fhatz)\\le \\max\\{2\\eta^{1-\\beta},2\\gamma_{\\bm{z},\\eta}^{-1}\\}\\cdot\\dist(\\bm{q},\\gK)^\\beta\\,\\,\\text{ whenever }  \\bm{q}\\in \\{\\bm{z}\\}^\\perp\\cap \\B(\\eta) .\n\t\t\\]\n\t\\end{theorem}","start":64632,"end":65697}]},{"id":"S050","semantic_type":"proof","kind":"proof","labels":["eq:dist-w-u","eq:dist-w-v","eq:form-w-by-v","eq:upbd-norm-bar-v"],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"坐标面估计完整证明","source_units":[{"raw_tex":"\\begin{proof}\n\t\t%\tIf \\( \\gamma_{\\bm{z},\\eta} = 0 \\), in view of \\cite[Lemma 3.12]{LiLoPo20}, there exist \\( \\widehat{\\bm{v}}\\in \\Fhatz \\) and a sequence \\( \\{\\bm{v}^k\\}\\subset \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fhatz \\) such that\n\t\t%\t\\begin{equation}\\label{eq:forcontradiction-2nd-face}\n\t\t%\t\t\\lim_{k\\to \\infty}\\bm{v}^k = \\lim_{k\\to \\infty}\\bm{w}^k = \\widehat{\\bm{v}}\\,\\,{\\rm and}\\,\\,\\lim_{k\\to\\infty}\\frac{\\|\\bm{w}^k - \\bm{v}^k\\|^{\\beta }}{\\|\\bm{w}^k - u^k\\|} = 0,\n\t\t%\t\\end{equation}\n\t\t%\twhere \\( \\bm{w}^k = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}^k) \\), \\( \\bm{u}^k = P_{\\Fhatz}(\\bm{w}^k) \\) and \\( \\bm{u}^k\\neq \\bm{w}^k \\).\n\n\t\t{In view of \\cite[Theorem 3.10]{LiLoPo20}}, we need only show that $\\gamma_{\\bm{z},\\eta} > 0$.\n%\t\t$$\n%\t\t0<\\gamma_{\\bm{z},\\eta} = \\inf_{\\bm{v}} \\left\\{ \\frac{\\|\\bm{w} -\\bm{v} \\|^\\beta}{\\|\\bm{w} -\\bm{u} \\| } \\Big|\\bm{v} \\in \\bd \\stdCone \\cap \\B(\\eta) \\setminus \\stdFace,\\bm{w} = P_{\\{\\bm{z}\\}^{\\perp }}(\\bm{v}), \\bm{u} = P_{\\stdFace}(\\bm{w}), \\bm{w} \\neq\\bm{u} \\right\\}.\n%\t\t$$\n\t\tTo that end, let \\( \\bm{v}\\in \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fhatz \\), \\( \\bm{w} = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}) \\), \\( \\bm{u} = P_{\\Fhatz}(\\bm{w}) \\), and \\( \\bm{u} \\neq \\bm{w} \\). Then a direct computation shows\n\t\tthat\n\t\t\\begin{equation}\\label{eq:dist-w-v}\n\t\t\t\\|\\bm{w}- \\bm{v}\\| = \\frac1{\\|\\bm{z}\\|}|\\inProd{ \\bm{z}}{\\bm{v}}| \\overset{\\rm (a)}= \\frac1{\\|\\bm{z}\\|}\\sum_{i\\in \\stdIndex}\\tz_i\\tv_i \\overset{\\rm (b)}\\ge \\frac{\\min_{i\\in \\stdIndex}\\tz_i}{\\|\\bm{z}\\|}\\sum_{i\\in \\stdIndex}\\tv_i\n\t\t\t\\overset{\\rm (c)}\\ge \\frac{\\min_{i\\in \\stdIndex}\\tz_i}{\\|\\bm{z}\\|}\\|\\tbv_{\\stdIndex}\\|,\n\t\t\\end{equation}\n\t\twhere (a), (b) and (c) hold because \\( \\tv_i\\ge 0 \\) and \\( \\tz_i > 0 \\) for all \\( i\\in \\stdIndex \\), with \\( \\|\\tbv_{\\stdIndex}\\|:= \\sqrt{\\sum_{i\\in \\stdIndex}\\tv^2_i} \\) (note that \\( \\stdIndex \\neq \\emptyset \\), thanks to \\( \\bbz = 0 \\) and \\( \\bm{z}\\neq \\bm{0} \\)). Next, notice that \\( \\bm{w} = \\bm{v} - \\frac{\\inProd{ \\bm{z}}{\\bm{v}}}{\\|\\bm{z}\\|^2}\\bm{z} \\). Using this and the definitions of \\( \\bm{z} \\) and \\( \\stdIndex \\), we deduce that\n\t\t\\begin{equation}\\label{eq:form-w-by-v}\n\t\t\t\\bbw = \\bbv,\\,\\,\\,\\tw_i = \\tv_i - \\frac{\\tz_i}{\\|\\bm{z}\\|^2}\\bigg(\\sum_{j\\in \\stdIndex}\\tz_j\\tv_j\\bigg)\\,\\,\\,\\, \\forall i\\in \\stdIndex\\,\\,\\,\\,\\,\\,\\,\\mbox{and\\,\\,\\,\\,\\,\\,\\,} \\tw_i = \\tv_i \\ge 0\\,\\,\\,\\,\\forall i \\notin \\stdIndex.\n\t\t\\end{equation}\n\t\tIn view of this and the definition of \\( \\Fhatz \\) in \\eqref{eq:2nd-type-face-d-dim}, we see that \\( \\tu_i = \\tw_i \\) whenever \\( i\\notin \\stdIndex \\), and hence\n\t\t\\begin{equation}\\label{eq:dist-w-u}\n\t\t\t\\|\\bm{w} - \\bm{u}\\| = \\sqrt{\\|\\bbw\\|^2 + \\sum_{i\\in \\stdIndex}\\tw^2_i} \\le \\sqrt{\\|\\bbv\\|^2 + n(1+\\sqrt{n})^2\\|\\tbv_{\\stdIndex}\\|^2},\n\t\t\\end{equation}\n\t\twhere the inequality follows from \\eqref{eq:form-w-by-v} and the fact that for each \\( i\\in \\stdIndex \\),\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t|\\tw_i| & = \\bigg|\\tv_i - \\frac{\\tz_i}{\\|\\bm{z}\\|^2}\\bigg(\\sum_{j\\in \\stdIndex}\\tz_j\\tv_j\\bigg)\\bigg|\\le \\bigg(1 + \\frac{|\\tz_i|}{\\|\\bm{z}\\|^2}\\sum_{j\\in \\stdIndex}|\\tz_j|\\bigg)\\|\\tbv_{\\stdIndex}\\| \\\\\n\t\t\t& \\le \\left(1 + \\frac{\\sqrt{n}|\\tz_i|}{\\|\\bm{z}\\|}\\right)\\|\\tbv_{\\stdIndex}\\| \\le (1+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\|.\n\t\t\\end{aligned}\n\t\t\\]\n\t\tNext, note that we have\n\t\t\\begin{equation}\\label{eq:upbd-norm-bar-v}\n\t\t\t\\|\\bbv\\|=%\n\t\t\t\\prod_{i=1}^n(\\tv_i)^{\\alpha_i} %\n\t\t\t= \\prod_{i\\notin \\stdIndex}\\tv_i^{\\alpha_i} \\cdot \\prod_{i\\in \\stdIndex}\\tv_i^{\\alpha_i}%\n\t\t\t\\le \\prod_{i\\notin \\stdIndex}\\eta^{\\alpha_i} \\cdot\\prod_{i\\in \\stdIndex}\\|\\tbv_{\\stdIndex}\\|^{\\alpha_i}%\n\t\t\t= \\eta^{1-\\beta} \\|\\tbv_{\\stdIndex}\\|^\\beta,\n\t\t\\end{equation}\n\t\twhere the inequality holds because \\( \\bm{v}\\in \\B(\\eta) \\). Combining \\eqref{eq:dist-w-v}, \\eqref{eq:dist-w-u} and \\eqref{eq:upbd-norm-bar-v}, we deduce\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\|\\bm{w} - \\bm{u}\\| & \\le \\sqrt{\\|\\bbv\\|^2 + n(1+\\sqrt{n})^2\\|\\tbv_{\\stdIndex}\\|^2} \\le \\|\\bbv\\| + (n+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\|                                                                                                     \\\\\n\t\t\t& \\le \\eta^{1-\\beta} \\|\\tbv_{\\stdIndex}\\|^\\beta + (n+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\| = (\\eta^{1-\\beta}  + (n+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\|^{1-\\beta})\\|\\tbv_{\\stdIndex}\\|^\\beta                                   \\\\\n\t\t\t& \\overset{\\rm (a)}\\le \\eta^{1-\\beta}(n+1 + \\sqrt{n})\\|\\tbv_{\\stdIndex}\\|^\\beta \\overset{\\rm (b)}{\\le}\\frac{\\eta^{1-\\beta}(n+1 + \\sqrt{n})\\|\\bm{z}\\|^\\beta}{\\left(\\min_{i\\in_{\\stdIndex}}\\tz_i\\right)^\\beta}\\|\\bm{w}-\\bm{v}\\|^\\beta.\n\t\t\\end{aligned}\n\t\t\\]\n\t\tHere (a) holds since \\( \\bm{v}\\in \\B(\\eta) \\) and \\( \\beta \\in (0,1) \\); (b) is true because of \\eqref{eq:dist-w-v}.\n\t\tThus, \\( \\gamma_{\\bm{z},\\eta}\\ge \\frac{\\left(\\min_{i\\in \\stdIndex}\\tz_i\\right)^\\beta}{\\eta^{1-\\beta}(n+1 + \\sqrt{n})\\|\\bm{z}\\|^\\beta} > 0 \\), and the desired error bound {follows from \\cite[Theorem 3.10]{LiLoPo20}}.\n\t\\end{proof}","start":65699,"end":70480}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":["eq:dist-w-v"],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"超平面残差控制受限坐标","source_units":[{"raw_tex":"\\begin{proof}\n","start":65699,"end":65713},{"raw_tex":"\n\t\t{In view of \\cite[Theorem 3.10]{LiLoPo20}}, we need only show that $\\gamma_{\\bm{z},\\eta} > 0$.\n","start":66310,"end":66408},{"raw_tex":"\t\tTo that end, let \\( \\bm{v}\\in \\bd \\gK\\cap \\B(\\eta)\\backslash \\Fhatz \\), \\( \\bm{w} = P_{\\{\\bm{z}\\}^\\perp}(\\bm{v}) \\), \\( \\bm{u} = P_{\\Fhatz}(\\bm{w}) \\), and \\( \\bm{u} \\neq \\bm{w} \\). Then a direct computation shows\n\t\tthat\n\t\t\\begin{equation}\\label{eq:dist-w-v}\n\t\t\t\\|\\bm{w}- \\bm{v}\\| = \\frac1{\\|\\bm{z}\\|}|\\inProd{ \\bm{z}}{\\bm{v}}| \\overset{\\rm (a)}= \\frac1{\\|\\bm{z}\\|}\\sum_{i\\in \\stdIndex}\\tz_i\\tv_i \\overset{\\rm (b)}\\ge \\frac{\\min_{i\\in \\stdIndex}\\tz_i}{\\|\\bm{z}\\|}\\sum_{i\\in \\stdIndex}\\tv_i\n\t\t\t\\overset{\\rm (c)}\\ge \\frac{\\min_{i\\in \\stdIndex}\\tz_i}{\\|\\bm{z}\\|}\\|\\tbv_{\\stdIndex}\\|,\n\t\t\\end{equation}\n\t\twhere (a), (b) and (c) hold because \\( \\tv_i\\ge 0 \\) and \\( \\tz_i > 0 \\) for all \\( i\\in \\stdIndex \\), with \\( \\|\\tbv_{\\stdIndex}\\|:= \\sqrt{\\sum_{i\\in \\stdIndex}\\tv^2_i} \\) (note that \\( \\stdIndex \\neq \\emptyset \\), thanks to \\( \\bbz = 0 \\) and \\( \\bm{z}\\neq \\bm{0} \\)). Next, notice that \\( \\bm{w} = \\bm{v} - \\frac{\\inProd{ \\bm{z}}{\\bm{v}}}{\\|\\bm{z}\\|^2}\\bm{z} \\). Using this and the definitions of \\( \\bm{z} \\) and \\( \\stdIndex \\), we deduce that\n","start":66688,"end":67739}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":["eq:dist-w-u","eq:form-w-by-v"],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"显式投影控制面距离","source_units":[{"raw_tex":"\t\t\\begin{equation}\\label{eq:form-w-by-v}\n\t\t\t\\bbw = \\bbv,\\,\\,\\,\\tw_i = \\tv_i - \\frac{\\tz_i}{\\|\\bm{z}\\|^2}\\bigg(\\sum_{j\\in \\stdIndex}\\tz_j\\tv_j\\bigg)\\,\\,\\,\\, \\forall i\\in \\stdIndex\\,\\,\\,\\,\\,\\,\\,\\mbox{and\\,\\,\\,\\,\\,\\,\\,} \\tw_i = \\tv_i \\ge 0\\,\\,\\,\\,\\forall i \\notin \\stdIndex.\n\t\t\\end{equation}\n\t\tIn view of this and the definition of \\( \\Fhatz \\) in \\eqref{eq:2nd-type-face-d-dim}, we see that \\( \\tu_i = \\tw_i \\) whenever \\( i\\notin \\stdIndex \\), and hence\n\t\t\\begin{equation}\\label{eq:dist-w-u}\n\t\t\t\\|\\bm{w} - \\bm{u}\\| = \\sqrt{\\|\\bbw\\|^2 + \\sum_{i\\in \\stdIndex}\\tw^2_i} \\le \\sqrt{\\|\\bbv\\|^2 + n(1+\\sqrt{n})^2\\|\\tbv_{\\stdIndex}\\|^2},\n\t\t\\end{equation}\n\t\twhere the inequality follows from \\eqref{eq:form-w-by-v} and the fact that for each \\( i\\in \\stdIndex \\),\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t|\\tw_i| & = \\bigg|\\tv_i - \\frac{\\tz_i}{\\|\\bm{z}\\|^2}\\bigg(\\sum_{j\\in \\stdIndex}\\tz_j\\tv_j\\bigg)\\bigg|\\le \\bigg(1 + \\frac{|\\tz_i|}{\\|\\bm{z}\\|^2}\\sum_{j\\in \\stdIndex}|\\tz_j|\\bigg)\\|\\tbv_{\\stdIndex}\\| \\\\\n\t\t\t& \\le \\left(1 + \\frac{\\sqrt{n}|\\tz_i|}{\\|\\bm{z}\\|}\\right)\\|\\tbv_{\\stdIndex}\\| \\le (1+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\|.\n\t\t\\end{aligned}\n\t\t\\]\n","start":67739,"end":68859}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":["eq:upbd-norm-bar-v"],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"边界乘积产生指数","source_units":[{"raw_tex":"\t\tNext, note that we have\n\t\t\\begin{equation}\\label{eq:upbd-norm-bar-v}\n\t\t\t\\|\\bbv\\|=%\n\t\t\t\\prod_{i=1}^n(\\tv_i)^{\\alpha_i} %\n\t\t\t= \\prod_{i\\notin \\stdIndex}\\tv_i^{\\alpha_i} \\cdot \\prod_{i\\in \\stdIndex}\\tv_i^{\\alpha_i}%\n\t\t\t\\le \\prod_{i\\notin \\stdIndex}\\eta^{\\alpha_i} \\cdot\\prod_{i\\in \\stdIndex}\\|\\tbv_{\\stdIndex}\\|^{\\alpha_i}%\n\t\t\t= \\eta^{1-\\beta} \\|\\tbv_{\\stdIndex}\\|^\\beta,\n\t\t\\end{equation}\n","start":68859,"end":69247}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"取得显式正面常数","source_units":[{"raw_tex":"\t\twhere the inequality holds because \\( \\bm{v}\\in \\B(\\eta) \\). Combining \\eqref{eq:dist-w-v}, \\eqref{eq:dist-w-u} and \\eqref{eq:upbd-norm-bar-v}, we deduce\n\t\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\|\\bm{w} - \\bm{u}\\| & \\le \\sqrt{\\|\\bbv\\|^2 + n(1+\\sqrt{n})^2\\|\\tbv_{\\stdIndex}\\|^2} \\le \\|\\bbv\\| + (n+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\|                                                                                                     \\\\\n\t\t\t& \\le \\eta^{1-\\beta} \\|\\tbv_{\\stdIndex}\\|^\\beta + (n+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\| = (\\eta^{1-\\beta}  + (n+\\sqrt{n})\\|\\tbv_{\\stdIndex}\\|^{1-\\beta})\\|\\tbv_{\\stdIndex}\\|^\\beta                                   \\\\\n\t\t\t& \\overset{\\rm (a)}\\le \\eta^{1-\\beta}(n+1 + \\sqrt{n})\\|\\tbv_{\\stdIndex}\\|^\\beta \\overset{\\rm (b)}{\\le}\\frac{\\eta^{1-\\beta}(n+1 + \\sqrt{n})\\|\\bm{z}\\|^\\beta}{\\left(\\min_{i\\in_{\\stdIndex}}\\tz_i\\right)^\\beta}\\|\\bm{w}-\\bm{v}\\|^\\beta.\n\t\t\\end{aligned}\n\t\t\\]\n\t\tHere (a) holds since \\( \\bm{v}\\in \\B(\\eta) \\) and \\( \\beta \\in (0,1) \\); (b) is true because of \\eqref{eq:dist-w-v}.\n\t\tThus, \\( \\gamma_{\\bm{z},\\eta}\\ge \\frac{\\left(\\min_{i\\in \\stdIndex}\\tz_i\\right)^\\beta}{\\eta^{1-\\beta}(n+1 + \\sqrt{n})\\|\\bm{z}\\|^\\beta} > 0 \\), and the desired error bound {follows from \\cite[Theorem 3.10]{LiLoPo20}}.\n\t\\end{proof}","start":69247,"end":70480}]},{"id":"S056","semantic_type":"lemma","kind":"lemma","labels":["remark:F-z-order-beta"],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"坐标面指数的渐近最优性","source_units":[{"raw_tex":"\\begin{remark}[Optimality of the error bound in Theorem~\\ref{thm:xface}]\\label{remark:F-z-order-beta}\n\t\t\\!\\!\\!Let \\( \\bm{z}\\!\\in\\!\\! \\bd\\dgK\\backslash\\!\\{\\bm{0}\\} \\) with \\( \\bbz = \\bm{0} \\) and let \\( \\Fhatz := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Let \\( \\stdIndex := \\{i\\,| \\,\\tz_i > 0\\}\\neq \\emptyset \\) and define\n\t\t\\[\n\t\t\\beta:= \\sum_{i\\in \\stdIndex}\\alpha_i\\in (0,1).\n\t\t\\]\n","start":70483,"end":70853},{"raw_tex":"\t\tThus \\( | \\cdot | ^{\\beta} \\) satisfies the asymptotic optimality criterion {(cf. \\cite[Definition 3.1]{LiLoPo21}}) for \\( \\gK \\) and \\( \\bm{z} \\){, which implies that the error bound is optimal in the sense of \\cite[Theorem 3.2(b)]{LiLoPo21}.}\n\t\\end{remark}","start":72269,"end":72529}]},{"id":"S057","semantic_type":"proof","kind":"proof","labels":["remark:F-z-order-beta"],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"坐标面最优性完整构造论证","source_units":[{"raw_tex":"\t\tFix any \\( \\bm{u}\\in \\R^m \\) with \\( \\|\\bm{u}\\|=1 \\) and define the continuous function \\( \\bm{q}:(0,1)\\to \\{\\bm{z}\\}^\\perp \\) by $\\epsilon \\mapsto \\bm{q}_\\epsilon := (\\bar{\\bm{q}}_\\epsilon,\\widetilde{\\bm{q}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{q}}_\\epsilon = \\epsilon^\\beta \\bm{u},\\,\\,(\\widetilde{\\bm{q}}_\\epsilon)_i = 0\\,\\,\\forall i \\in \\stdIndex,\\,\\mbox{and} \\,(\\widetilde{\\bm{q}}_\\epsilon)_i = 1, \\,\\forall i \\notin \\stdIndex.\n\t\t\\]\n\t\tIt is clear that for all $\\epsilon$, \\( \\inProd{ \\bm{z}}{\\bm{q}_\\epsilon} = 0 \\) and \\( \\dist(\\bm{q}_\\epsilon,\\Fhatz) \\to 0 \\) as $\\epsilon \\downarrow 0$ . Now, define the function \\( \\bm{p}:(0,1)\\to \\gK \\) by $\\epsilon \\mapsto \\bm{p}_\\epsilon := (\\bar{\\bm{p}}_\\epsilon,\\widetilde{\\bm{p}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{p}}_\\epsilon = \\epsilon^\\beta \\bm{u},\\,\\,(\\widetilde{\\bm{p}}_\\epsilon)_i = \\epsilon\\,\\,\\forall i \\in \\stdIndex,\\,\\mbox{and} \\,(\\widetilde{\\bm{p}}_\\epsilon)_i = 1, \\,\\forall i \\notin \\stdIndex.\n\t\t\\]\n\t\tClearly $\\bm{p}_\\epsilon$ lies in \\( \\gK \\), and we have that\n\t\t$\n\t\t\t\\dist(\\bm{q}_\\epsilon,\\gK)\\le \\|\\bm{q}_\\epsilon - \\bm{p}_\\epsilon\\| \\le |\\stdIndex|\\cdot\\epsilon\n\t\t$.\n\t\tOn the other hand, we have in view of \\eqref{eq:2nd-type-face-d-dim} that \\( \\dist(\\bm{q}_\\epsilon,\\Fhatz) = \\epsilon^\\beta > 0 \\).\n\t\tHence,\n\t\t$\n\t\t\\limsup_{\\epsilon\\downarrow 0}\\frac{\\dist(\\bm{q}_\\epsilon,\\gK)^\\beta}{\\dist(\\bm{q}_\\epsilon,\\Fhatz)} \\le |\\stdIndex|^\\beta < \\infty\n\t\t$.\n","start":70853,"end":72269}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":["remark:F-z-order-beta"],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"坐标面的横向扰动路径","source_units":[{"raw_tex":"\t\tFix any \\( \\bm{u}\\in \\R^m \\) with \\( \\|\\bm{u}\\|=1 \\) and define the continuous function \\( \\bm{q}:(0,1)\\to \\{\\bm{z}\\}^\\perp \\) by $\\epsilon \\mapsto \\bm{q}_\\epsilon := (\\bar{\\bm{q}}_\\epsilon,\\widetilde{\\bm{q}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{q}}_\\epsilon = \\epsilon^\\beta \\bm{u},\\,\\,(\\widetilde{\\bm{q}}_\\epsilon)_i = 0\\,\\,\\forall i \\in \\stdIndex,\\,\\mbox{and} \\,(\\widetilde{\\bm{q}}_\\epsilon)_i = 1, \\,\\forall i \\notin \\stdIndex.\n\t\t\\]\n\t\tIt is clear that for all $\\epsilon$, \\( \\inProd{ \\bm{z}}{\\bm{q}_\\epsilon} = 0 \\) and \\( \\dist(\\bm{q}_\\epsilon,\\Fhatz) \\to 0 \\) as $\\epsilon \\downarrow 0$ . Now, define the function \\( \\bm{p}:(0,1)\\to \\gK \\) by $\\epsilon \\mapsto \\bm{p}_\\epsilon := (\\bar{\\bm{p}}_\\epsilon,\\widetilde{\\bm{p}}_\\epsilon)$ where\n\t\t\\[\n\t\t\\bar{\\bm{p}}_\\epsilon = \\epsilon^\\beta \\bm{u},\\,\\,(\\widetilde{\\bm{p}}_\\epsilon)_i = \\epsilon\\,\\,\\forall i \\in \\stdIndex,\\,\\mbox{and} \\,(\\widetilde{\\bm{p}}_\\epsilon)_i = 1, \\,\\forall i \\notin \\stdIndex.\n\t\t\\]\n","start":70853,"end":71810}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":["remark:F-z-order-beta"],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"坐标面最优性距离比值","source_units":[{"raw_tex":"\t\tClearly $\\bm{p}_\\epsilon$ lies in \\( \\gK \\), and we have that\n\t\t$\n\t\t\t\\dist(\\bm{q}_\\epsilon,\\gK)\\le \\|\\bm{q}_\\epsilon - \\bm{p}_\\epsilon\\| \\le |\\stdIndex|\\cdot\\epsilon\n\t\t$.\n\t\tOn the other hand, we have in view of \\eqref{eq:2nd-type-face-d-dim} that \\( \\dist(\\bm{q}_\\epsilon,\\Fhatz) = \\epsilon^\\beta > 0 \\).\n\t\tHence,\n\t\t$\n\t\t\\limsup_{\\epsilon\\downarrow 0}\\frac{\\dist(\\bm{q}_\\epsilon,\\gK)^\\beta}{\\dist(\\bm{q}_\\epsilon,\\Fhatz)} \\le |\\stdIndex|^\\beta < \\infty\n\t\t$.\n","start":71810,"end":72269}]},{"id":"S060","semantic_type":"corollary","kind":"corollary","labels":["3.6i","corollary:1-FRFs-power-cones","eq:1-FRF-1st-face"],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"射线面的一步残差函数","source_units":[{"raw_tex":"\\begin{corollary}\\label{corollary:1-FRFs-power-cones}\n\t\tConsider \\( \\gK \\) and its dual cone \\( \\dgK \\).\n\t\t\\begin{enumerate}[label=(\\roman*)]\n\t\t\t\\item\\label{3.6i} Let \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz\\neq \\bm{0} \\) and let \\( \\Fr := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Let \\( \\gamma_{\\bm{z},t} \\) be defined as in \\eqref{eq:gamma-1/2-1st-type-face}. Then the function \\( \\psi_{\\gK,\\bm{z}}:\\R_+\\times \\R_+\\to \\R_+ \\) given by\\vspace{-0.15 cm}\n\t\t\t\\begin{equation}\\label{eq:1-FRF-1st-face}\\vspace{-0.15 cm}\n\t\t\t\t\\!\\!\\psi_{\\gK,\\bm{z}}(\\epsilon,t) \\!:=\\! \\max \\left\\{\\epsilon,\\epsilon/\\|\\bm{z}\\| \\right\\} + \\max\\{2\\sqrt{t},2\\gamma_{\\bm{z},t}^{-1}\\}(\\epsilon +\\max \\left\\{\\epsilon,\\epsilon/\\|\\bm{z}\\| \\right\\} )^\\frac12\n\t\t\t\\end{equation}\n\t\t\tis a one-step facial residual function for \\( \\gK \\) and \\( \\bm{z} \\).\n","start":72726,"end":73547}]},{"id":"S061","semantic_type":"corollary","kind":"corollary","labels":["3.6i","corollary:1-FRFs-power-cones","eq:1-FRF-2nd-face"],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"坐标面的一步残差函数","source_units":[{"raw_tex":"\\begin{corollary}\\label{corollary:1-FRFs-power-cones}\n\t\tConsider \\( \\gK \\) and its dual cone \\( \\dgK \\).\n\t\t\\begin{enumerate}[label=(\\roman*)]\n","start":72726,"end":72868},{"raw_tex":"\t\t\t\\item Let \\( \\bm{z}\\in \\bd\\dgK\\backslash\\{\\bm{0}\\} \\) with \\( \\bbz = \\bm{0} \\) and let \\( \\Fhatz := \\{\\bm{z}\\}^\\perp\\cap \\gK \\). Let \\( \\gamma_{\\bm{z},t} \\) be defined as in \\eqref{eq:gamma-beta-2nd-type-face}, where \\( \\beta:= \\sum_{i:\\tz_i > 0}\\alpha_i \\). Then the function \\( \\psi_{\\gK,\\bm{z}}:\\R_+\\times \\R_+\\to \\R_+ \\) given by\\vspace{-0.15 cm}\n\t\t\t\\begin{equation}\\label{eq:1-FRF-2nd-face}\\vspace{-0.15 cm}\n\t\t\t\t\\!\\!\\psi_{\\gK,\\bm{z}}(\\epsilon,t) \\!:= \\!\\max\\! \\left\\{\\epsilon,\\epsilon/\\|\\bm{z}\\| \\right\\} + \\max\\{2t^{1-\\beta}\\!,2\\gamma_{\\bm{z},t}^{-1}\\}(\\epsilon +\\max \\left\\{\\epsilon,\\epsilon/\\|\\bm{z}\\| \\right\\} )^\\beta\n\t\t\t\\end{equation}\n\t\t\tis a one-step facial residual function for \\( \\gK \\) and \\( \\bm{z} \\).\n\t\t\\end{enumerate}\n\t\\end{corollary}","start":73547,"end":74303}]},{"id":"S062","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"两类面估计转为FRF","source_units":[{"raw_tex":"\tUsing Theorems~\\ref{thm:Holderian-1/2-1st-type-face} and \\ref{thm:xface} {together with \\cite[Lemma~3.9]{LiLoPo20}}, we have the following result concerning one-step facial residual functions.\n","start":72531,"end":72725}]},{"id":"S063","semantic_type":"theorem","kind":"theorem","labels":["item:error-bound-power-cone-1","item:error-bound-power-cone-2","item:error-bound-power-cone-3","item:error-bound-power-cone-3-1","item:error-bound-power-cone-3-2","item:error-bound-power-cone-3-3","item:error-bound-power-cone-4","thm:error-bound-power-cone-optimality"],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"广义幂锥误差界及最优性","source_units":[{"raw_tex":"\\begin{theorem}[Error bounds for the generalized power cone and their optimality]\\label{thm:error-bound-power-cone-optimality}\n\t\tConsider \\( \\gK \\) and its dual cone \\( \\dgK \\).\n\t\tLet \\( \\subSpace \\subseteq \\R^{m + n} \\) be a subspace and \\( \\bm{a} \\in \\R^{m+n} \\) be given.\n\t\tSuppose that \\( ( \\subSpace + \\bm{a} ) \\cap \\gK \\neq \\emptyset \\).\n\t\tThen the following items hold.\n\t\t\\begin{enumerate}[label=(\\roman*)]\n\t\t\t\\item\\label{item:error-bound-power-cone-1} \\( \\dpps (\\gK, \\subSpace + \\bm{a}) \\leq 1 \\).\n\t\t\t\\item\\label{item:error-bound-power-cone-2} If \\( \\dpps (\\gK, \\subSpace + \\bm{a}) = 0 \\), then a Lipschitzian error bound holds.\n\t\t\t\\item\\label{item:error-bound-power-cone-3} If \\( \\dpps (\\gK, \\subSpace + \\bm{a}) = 1 \\), consider the chain of faces \\( \\stdFace \\subsetneq \\gK \\) with length being \\( 2 \\).\n\t\t\t\\begin{enumerate}[label=(\\alph*)]\n\t\t\t\t\\item\\label{item:error-bound-power-cone-3-1} If \\( \\stdFace = \\Fr \\), then a H\\\"olderian error bound with exponent \\( 1 / 2 \\) holds.\n\t\t\t\t\\item\\label{item:error-bound-power-cone-3-2} If \\( \\stdFace = \\Fhatz \\) with \\( \\bm{z} \\in \\dgK \\cap \\subSpace^{\\perp } \\cap \\{\\bm{a}\\}^{\\perp } \\), then a H\\\"olderian error bound with exponent \\( \\beta := \\sum_{i : \\tz_i > 0} \\alpha _i \\) holds.\n\t\t\t\t\\item\\label{item:error-bound-power-cone-3-3} If \\( \\stdFace = \\{\\bm{0}\\} \\), then a Lipschitzian error bound holds.\n\t\t\t\\end{enumerate}\n\t\t\t\\item\\label{item:error-bound-power-cone-4} All these error bounds are the best in the sense {stated in \\cite[Theorem 3.2(b)]{LiLoPo21}}.\n\t\t\\end{enumerate}\n\t\\end{theorem}","start":74378,"end":75929}]},{"id":"S064","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S064"],"declared_external":false,"source_only":false,"title":"主定理完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\t\tAs is shown in Section \\ref{subsec:facial-structure}, all the proper exposed faces of the generalized power cone are polyhedral.\n\t\tThen the process of facial reduction needs at most one step to reach the PPS condition. Hence, \\( \\dpps(\\gK, \\subSpace + \\bm{a}) \\leq 1 \\).\n\t\tThis shows item \\ref{item:error-bound-power-cone-1}.\n\n\t\tIf \\( \\dpps(\\gK, \\subSpace + \\bm{a}) = 0 \\), i.e., \\eqref{eq:conic-feasibility-problem} satisfies the PPS condition, then by \\cite[Corollary 3]{BBL99}, a Lipschitzian error bound holds.\n\t\tThis shows item \\ref{item:error-bound-power-cone-2}.\n\n\t\tNext, let \\( \\dpps(\\gK, \\subSpace + \\bm{a}) = 1 \\); i.e., we need one step to reach the PPS condition.\n\t\tIn this case, the error bound depends on the exposed face \\( \\stdFace \\) that contains the feasible region.\n\t\tIf \\(\\stdFace =  \\Fr \\), then by Corollary \\ref{corollary:1-FRFs-power-cones}\\ref{3.6i}, we conclude that a H\\\"olderian error bound with exponent \\( 1 / 2 \\) holds.\n\t\tRemark \\ref{remark:F-r-order-1/2} implies that \\( \\mathfrak{g} = | \\cdot | ^{\\frac{1}{2}}\\) satisfies the asymptotic optimality criterion for \\( \\gK \\) and \\( \\bm{z} \\) with \\( \\bbz \\neq \\bm{0} \\).\n\t\tHence, by {\\cite[Theorem 3.2]{LiLoPo21}}, the obtained H\\\"olderian error bound with exponent \\( 1 / 2 \\) is the best error bound.\n\n\t\tIf \\( \\stdFace = \\Fhatz \\) with \\( \\bm{z} \\in \\dgK \\cap \\subSpace^{\\perp } \\cap \\{\\bm{a}\\}^{\\perp } \\), then using Corollary \\ref{corollary:1-FRFs-power-cones}(ii), we conclude that a H\\\"olderian error bound with exponent \\( \\beta := \\sum _{i \\in \\stdIndex} \\alpha _i \\) holds, where \\( \\stdIndex = \\{i\\,| \\,\\tz_i > 0\\} \\).\n\t\tThe optimality of this error bound comes from Remark \\ref{remark:F-z-order-beta} and {\\cite[Theorem 3.2]{LiLoPo21}}.\nIf \\( \\stdFace = \\{\\bm{0}\\} \\), which means the feasible region is \\( \\{\\bm{0}\\} \\), then a Lipschitzian error bound holds automatically and it is naturally tight, see \\cite[Proposition~27]{L17}.\n\t\\end{proof}","start":75932,"end":77887}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S065"],"declared_external":false,"source_only":false,"title":"多面体面使约化至多一步","source_units":[{"raw_tex":"\t\tAs is shown in Section \\ref{subsec:facial-structure}, all the proper exposed faces of the generalized power cone are polyhedral.\n\t\tThen the process of facial reduction needs at most one step to reach the PPS condition. Hence, \\( \\dpps(\\gK, \\subSpace + \\bm{a}) \\leq 1 \\).\n\t\tThis shows item \\ref{item:error-bound-power-cone-1}.\n","start":75946,"end":76274}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S066"],"declared_external":false,"source_only":false,"title":"零PPS距离给出线性界","source_units":[{"raw_tex":"\n\t\tIf \\( \\dpps(\\gK, \\subSpace + \\bm{a}) = 0 \\), i.e., \\eqref{eq:conic-feasibility-problem} satisfies the PPS condition, then by \\cite[Corollary 3]{BBL99}, a Lipschitzian error bound holds.\n\t\tThis shows item \\ref{item:error-bound-power-cone-2}.\n","start":76274,"end":76518}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S067"],"declared_external":false,"source_only":false,"title":"一步约化后的分类设置","source_units":[{"raw_tex":"\n\t\tNext, let \\( \\dpps(\\gK, \\subSpace + \\bm{a}) = 1 \\); i.e., we need one step to reach the PPS condition.\n\t\tIn this case, the error bound depends on the exposed face \\( \\stdFace \\) that contains the feasible region.\n","start":76518,"end":76734}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S068"],"declared_external":false,"source_only":false,"title":"射线分支的仿射误差界","source_units":[{"raw_tex":"\t\tIf \\(\\stdFace =  \\Fr \\), then by Corollary \\ref{corollary:1-FRFs-power-cones}\\ref{3.6i}, we conclude that a H\\\"olderian error bound with exponent \\( 1 / 2 \\) holds.\n","start":76734,"end":76901}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S069"],"declared_external":false,"source_only":false,"title":"射线分支的最优性","source_units":[{"raw_tex":"\t\tRemark \\ref{remark:F-r-order-1/2} implies that \\( \\mathfrak{g} = | \\cdot | ^{\\frac{1}{2}}\\) satisfies the asymptotic optimality criterion for \\( \\gK \\) and \\( \\bm{z} \\) with \\( \\bbz \\neq \\bm{0} \\).\n\t\tHence, by {\\cite[Theorem 3.2]{LiLoPo21}}, the obtained H\\\"olderian error bound with exponent \\( 1 / 2 \\) is the best error bound.\n","start":76901,"end":77233}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S070"],"declared_external":false,"source_only":false,"title":"坐标面分支的仿射误差界","source_units":[{"raw_tex":"\n\t\tIf \\( \\stdFace = \\Fhatz \\) with \\( \\bm{z} \\in \\dgK \\cap \\subSpace^{\\perp } \\cap \\{\\bm{a}\\}^{\\perp } \\), then using Corollary \\ref{corollary:1-FRFs-power-cones}(ii), we conclude that a H\\\"olderian error bound with exponent \\( \\beta := \\sum _{i \\in \\stdIndex} \\alpha _i \\) holds, where \\( \\stdIndex = \\{i\\,| \\,\\tz_i > 0\\} \\).\n","start":77233,"end":77560}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S071"],"declared_external":false,"source_only":false,"title":"坐标面分支的最优性","source_units":[{"raw_tex":"\t\tThe optimality of this error bound comes from Remark \\ref{remark:F-z-order-beta} and {\\cite[Theorem 3.2]{LiLoPo21}}.\n","start":77560,"end":77679}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S072"],"declared_external":false,"source_only":false,"title":"零面分支的线性界","source_units":[{"raw_tex":"If \\( \\stdFace = \\{\\bm{0}\\} \\), which means the feasible region is \\( \\{\\bm{0}\\} \\), then a Lipschitzian error bound holds automatically and it is naturally tight, see \\cite[Proposition~27]{L17}.\n\t\\end{proof}","start":77679,"end":77887}]},{"id":"S073","semantic_type":"assumption","kind":"assumption","labels":["item:error-bound-power-cone-1","item:error-bound-power-cone-2","item:error-bound-power-cone-3","item:error-bound-power-cone-3-1","item:error-bound-power-cone-3-2","item:error-bound-power-cone-3-3","item:error-bound-power-cone-4","thm:error-bound-power-cone-optimality"],"member_ids":["S073"],"declared_external":false,"source_only":false,"title":"主问题的仿射可行性","source_units":[{"raw_tex":"\t\tConsider \\( \\gK \\) and its dual cone \\( \\dgK \\).\n\t\tLet \\( \\subSpace \\subseteq \\R^{m + n} \\) be a subspace and \\( \\bm{a} \\in \\R^{m+n} \\) be given.\n\t\tSuppose that \\( ( \\subSpace + \\bm{a} ) \\cap \\gK \\neq \\emptyset \\).\n","start":74505,"end":74722}]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S074"],"declared_external":true,"declaration_text":["声明前提 LiLoPo20, Lemma 3.12；活动引用见 L532、L741。"],"source_only":false,"title":"零面常数的极限序列准则","source_units":[]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S075"],"declared_external":true,"declaration_text":["声明前提 LiLoPo20, Theorem 3.10；活动引用见 L741、L834、L876。"],"source_only":false,"title":"正面常数的误差界转换","source_units":[]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S076"],"declared_external":true,"declaration_text":["声明前提 LiLoPo20, Lemma 3.9；活动引用见 L904。"],"source_only":false,"title":"超平面误差界转为FRF","source_units":[]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S077"],"declared_external":true,"declaration_text":["声明前提 L17, Definition 3、Proposition 5，以及 L327–329 指向的框架；完整原文未提供。"],"source_only":false,"title":"PPS条件与面约化链","source_units":[]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S078"],"declared_external":true,"declaration_text":["声明前提 BBL99, Corollary 3；活动引用见 L327 的脚注和 L945。"],"source_only":false,"title":"PPS下的线性误差界","source_units":[]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S079"],"declared_external":true,"declaration_text":["声明前提 LiLoPo21, Definition 3.1、Theorem 3.2，特别是 (b)；活动引用见 L804、L901、L936、L952、L955。"],"source_only":false,"title":"渐近最优性判据及推论","source_units":[]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S080"],"declared_external":true,"declaration_text":["声明前提 L17, Proposition 27；活动引用见 L956。"],"source_only":false,"title":"原点单点可行域误差界","source_units":[]},{"id":"S081","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S081"],"declared_external":true,"declaration_text":["原声明标准前提中的 Taylor estimates；implicit standard fact made explicit by annotator。"],"source_only":false,"title":"局部Taylor估计","source_units":[]},{"id":"S082","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S082"],"declared_external":true,"declaration_text":["原声明标准前提中的 strict concavity of the logarithm；原文 L398、L458 明确使用。"],"source_only":false,"title":"对数严格凹性及等号条件","source_units":[]},{"id":"S083","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S083"],"declared_external":true,"declaration_text":["原声明标准前提中的 compactness；implicit standard fact made explicit by annotator。"],"source_only":false,"title":"紧性与正下界论证","source_units":[]},{"id":"S084","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S084"],"declared_external":true,"declaration_text":["原声明标准前提中的 Cauchy–Schwarz；L799 明确命名，其他使用由标注者显式指出。"],"source_only":false,"title":"柯西不等式及等号条件","source_units":[]},{"id":"S085","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S085"],"declared_external":true,"declaration_text":["原声明标准前提中的 finite-dimensional convex projection properties；implicit standard fact made explicit by annotator。"],"source_only":false,"title":"闭凸投影基本性质","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076","S077":"S077","S078":"S078","S079":"S079","S080":"S080","S081":"S081","S082":"S082","S083":"S083","S084":"S084","S085":"S085"}}}}
{"record_id":"2210.04448v4","paper":{"title":"Strong Variational Sufficiency for Nonlinear Semidefinite Programming and its Implications","authors":["Wang, Shiwei","Ding, Chao","Zhang, Yangjing","Zhao, Xinyuan"],"arxiv_id":"2210.04448","version":"v4","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/2210.04448v4","pdf_url":"https://arxiv.org/pdf/2210.04448v4","doi":"10.1137/22M1530161"},"context":{"raw_tex":" The definition of (strong) variational sufficient condition (Definition \\ref{def:svasc}) is officially given in \\cite{Roh22}  for the following general composite optimization problem\n \\begin{equation}\\label{eq:genp}\n \t\\min_{x\\in \\mathbb{X}}\\quad f(x)+\\theta(G(x)),\n \\end{equation}\n where $\\mathbb{X}$ and $\\mathbb{Y}$ are two given Euclidean spaces, $f:\\mathbb{X}\\to \\mathbb{R}$ and $G:\\mathbb{X}\\to \\mathbb{Y}$ are twice continuously differentiable, and $\\theta:\\mathbb{Y}\\to (-\\infty,\\infty]$ is a closed proper convex function. The strong variational sufficient condition is firstly introduced to deal with the local convexity of primal augmented Lagrangian function and the augmented tilt stability. It has gained more and more attention for its mathematical elegance and wide applications in the convergence analysis of multiplier methods. Several characterizations of this abstract property are also given in \\cite{Roh22}. For instance, it is equivalent to the positive definiteness of the Hessian bundle of the augmented Lagrangian function \\cite[Theorem 3]{Roh22} or the criterion \\cite[Theorem 5]{Roh22} involving quadratic bundle (Definition \\ref{def:quadratic bundle}).\n\n\n In terms of the connection with traditional optimality conditions, Rockafellar \\cite[Theorem 4]{Roh22} shows that the strong variational sufficiency is equivalent to the well-known strong second order sufficient condition (SOSC), which is expressed entirely via the program data, when the function $\\theta$ in \\eqref{eq:genp} is polyhedral convex (i.e., the epigraph ${\\rm epi}\\,\\theta$ of $\\theta$ is a polyhedral convex set). For non-polyhedral problems,  the equivalence is still valid if $\\theta$ in \\eqref{eq:genp} is the indicator function of the second order cone with $G(\\bar{x})\\neq 0$, where $\\bar{x}$ is a local optimal solution (see \\cite[Example 3]{Roh22} for details). However, an explicit and verifiable characterization of strong variational sufficient condition for general non-polyhedral problems remains unknown. In this paper, without loss of generality, we mainly focus on the characterization of strong variational sufficiency for the following nonlinear semidefinite programming (NLSDP):\n\\begin{equation}\\label{eq:NLSDPc}\n\t\\begin{array}{cl}\n\t\t\\displaystyle{\\min_{x\\in \\mathbb{X}}} & f(x) \\\\ [3pt]\n\t\t{\\rm s.t.} &{G(x)\\in \\mathbb{S}^n_+},\n\t\\end{array}\n\\end{equation}\nwhere  $\\mathbb{S}^n$ is the linear space of all $n\\times n$ real symmetric matrices equipped with the usual Frobenius inner product and its induced norm, $\\mathbb{S}_+^n$ ($\\mathbb{S}_-^n$) is the closed convex cone of all $n\\times n$ positive (negative) semidefinite matrices in $\\mathbb{S}^n$. One of our contributions lies in uncovering the equivalence between strong variational sufficient condition and strong SOSC (see Definition \\ref{def:sigt} for details) for  NLSDP problems without requiring any other constraint qualifications.\n\n\nFor NLSDP \\eqref{eq:NLSDPc}, the Lagrangian function is defined by\n\\begin{equation}\\label{eq:L-function}\n\tL(x,Y):=f(x) +\\langle Y,G(x)\\rangle, \\quad (x,Y)\\in\\mathbb{X}\\times\\mathbb{S}^n.\n\\end{equation}\nFor any $Y\\in\\mathbb{S}^n$, denote the first-order and second-order derivatives of $L(\\cdot,Y)$ at $x\\in\\mathbb{X}$ by $L_x'(x,Y)$ and $L_{xx}''(x,Y)$, respectively. Given $\\rho > 0$, the augmented Lagrangian function of \\eqref{eq:NLSDPc} takes the following form (cf. \\cite[Section 11.K]{RoWe98} and \\cite{ShapS04})\n\\begin{equation}\\label{eq:deflag}\n\t\\mathcal{L}_{\\rho}(x,Y):=f(x)+\\frac{\\rho}{2}{\\rm dist}^2(G(x)+\\frac{Y}{\\rho},\\mathbb{S}_+^n)-\\frac{\\|Y\\|^2}{2\\rho},\n\\end{equation}\nwhere ${\\rm dist}(z,\\mathbb{S}_+^n)$ is the distance from point $z$ to $\\mathbb{S}_+^n$.\n\n\nFor a given Euclidean space $\\mathbb{X}$,\nlet $C$ be any subset in $\\mathbb{X}$, ${\\rm aff}\\,{C}$ is the affine hull of $C$ \\cite[page 6]{Rockafellar1970}. The Bouligand tangent/contingent cone of $C$ at $x$ is a closed cone defined by\n\\[\n\\mathcal{T}_{C}(x):= \\left\\{d\\in\\mathbb{X}\\mid \\mbox{$\\exists\\, t^k\\downarrow 0$ and  $d^k\\to d$ with  $x+t^kd^k\\in C$ for all $k$} \\right\\}.\n\\]\nWhen $C$ is convex, the normal cone in the sense of  convex analysis \\cite{Rockafellar1970} is defined as\n$$\\mathcal{N}_{C}(x)=\\left \\{d \\in \\mathbb{X}\\mid\\langle d,x'-x\\rangle\\leq 0 \\ \\forall\\,x'\\in C \\right\\}.$$\nFor any $x\\in C$, the critical cone  associated with ${ y}\\in\\mathcal{N}_C(x)$ of $C$ is defined in \\cite[Page 98]{DoRo14}, i.e., ${\\cal C}_{C}({x},{y})=\\mathcal{T}_{C}({x})\\cap({y})^{\\perp}$.\n\nFor a set-valued mapping $\\mathcal{F}:\\mathbb{X}\\rightrightarrows\\mathbb{X}$, its $\\limsup$ means\n$$\\limsup\\limits_{{x}\\rightarrow\\bar{x}}\\mathcal{F}({x}):=\\left\\{d\\in\\mathbb{X}\\mid \\exists\\;x_k\\rightarrow\\bar{x},\\; {y}_k\\rightarrow d\\;{\\rm such\\;that}\\;{y}_k\\in\\mathcal{F}({x}_k)\\;\\forall\\,k\\right\\}.$$\nThe following definitions of proximal and (general/Mordukhovich limiting) subdifferentials of functions are adopted from \\cite[Definition 8.45 and 8.3]{RoWe98}.\n\\begin{definition}\n\tConsider a function $f:\\mathbb{X}\\rightarrow(-\\infty, +\\infty]$ and a point $\\bar{x}$ with $f(\\bar{x})$ finite. The proximal subdifferential of $f$ at $\\bar{x}$ is defined as\n\t$$\\partial^{\\pi}f(\\bar{x}):=\\{{d}\\mid \\exists\\; l>0, r>0\\;{\\rm such\\; that}\\;f(x)\\geq f(\\bar{x})+\\langle{d},{x}-\\bar{x}\\rangle-l\\|{x}-\\bar{x}\\|^2\\;\\forall\\,{x}\\in{\\cal B}_{r}(\\bar{x})\\},$$\n\twhere ${\\cal B}_{r}(\\bar{x})$ is the ball centered at $\\bar{x}$ with radius $r$.\n\tThe (general/Mordukhovich limiting) subdifferential of $f$ at $\\bar{x}$ is $\\partial f(\\bar{x}):=\\limsup\\limits_{{x}\\stackrel{f}{\\rightarrow}\\bar{x}}\\partial^{\\pi}f({x})$, where ${x}\\stackrel{f}{\\rightarrow}\\bar{x}$ signifies ${x}\\rightarrow\\bar{x}$ with $f({x})\\rightarrow f(\\bar{x})$.\n\\end{definition}\nNote that in an Euclidean space, the (general) subdifferential also can be constructed via the regular/Fr\\'{e}chet subdifferentials  of functions $\\widehat{\\partial}f$ \\cite[Definition 8.3]{RoWe98}. It is well-known that these two definitions are equivalent (see e.g., \\cite[page 345]{RoWe98} or \\cite[Theorem 1.89]{Mordukhovich06} for details). In particular, when $f$ is convex and finite at the corresponding point, the proximal and (general) subdifferentials coincide with the subdifferential in the sense of convex analysis \\cite{Rockafellar1970}.\n\n\nLet  $A \\in\\mathbb{S}^n$ be given. We use $\\lambda_{1}(A)\\ge \\lambda_2(A) \\ge \\ldots \\ge \\lambda_{n}(A)$ to denote the eigenvalues of $A$ (all real and counting multiplicity) arranging in  nonincreasing order and use $\\lambda(A)$ to denote the vector of the ordered eigenvalues of $A$. Let $\\Lambda(A):= {\\rm Diag}(\\lambda(A))$. Also, we use $v_1(A)> \\dots> v_d(A)$ to denote the different eigenvalues and\n$\\zeta_l:=\\{i\\mid \\lambda_i(A)=v_l(A)\\}$.\nConsider the eigenvalue decomposition of $A$, i.e., $A={P} \\Lambda(A){P}^{T}$, where ${P}\\in{\\cal O}^{n}(A)$ is a corresponding\northogonal matrix of the orthonormal eigenvectors.\nBy considering the index sets of positive, zero, and negative eigenvalues of $A$, we are able to write $A$ in the following form\n{\\footnotesize\n\\begin{equation}\\label{eq:eig-decomp}\n\tA= \\left[\\begin{array}{ccc}\n\t\t{P}_{\\alpha} & {P}_{\\beta} & {P}_{\\gamma}\n\t\\end{array}\\right] \\left[\\begin{array}{ccc}\n\t\t\\Lambda(A)_{\\alpha\\alpha} & 0 & 0 \\\\ [3pt]\n\t\t0 & 0 & 0\\\\ [3pt]\n\t\t0 & 0 & \\Lambda(A)_{\\gamma\\gamma}\n\t\\end{array}\\right]\\left[\\begin{array}{c}\n\t\t{P}_{\\alpha}^T \\\\ [3pt]\n\t\t{P}_{\\beta}^T \\\\\n\t\t{P}_{\\gamma}^T\n\t\\end{array}\\right],\n\\end{equation}}\nwhere $\\alpha:=\\{i: \\lambda_i(A)>0\\}$, $\\beta:=\\{i: \\lambda_i(A)=0\\}$ and $\\gamma:=\\{i: \\lambda_i(A)<0\\}$.\n\n\nThe critical cone of $\\mathbb{S}_+^n$ at ${Y}\\in{\\cal N}_{\\mathbb{S}_+^n}({X})$ with $A=X+Y$ is given by (cf. \\cite[equations (11) and (12)]{CSun08})\n\\begin{equation}\\label{eq:cricsdp}\n\t{\\cal C}_{\\mathbb{S}_+^n}({X},{Y})=\\{H\\in\\mathbb{S}^n\\mid P^T_{\\beta}HP_{\\beta}\\in\\mathbb{S}^{|\\beta|}_+,\\;P^T_{\\beta}HP_{\\gamma}=0,\\;P^T_{\\gamma}HP_{\\gamma}=0\\}\n\\end{equation}\nand the affine hull of ${\\cal C}_{\\mathbb{S}_+^n}({X},{Y})$ is\n\\begin{equation}\\label{eq:affcsdp}\n\t{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}({X},{Y})=\\{H\\in\\mathbb{S}^n\\mid P^T_{\\beta}HP_{\\gamma}=0,\\;P^T_{\\gamma}HP_{\\gamma}=0\\}.\n\\end{equation}\nThe  NLSDP \\eqref{eq:NLSDPc} can be written in the following form\n\\begin{eqnarray}\n\t%\\mbox{(SDP)}~~~~\n\t\\min\\quad && f(x)+\\delta_{\\mathbb{S}^n_+}(G(x)), \\label{eq:NLSDP}\n\\end{eqnarray}\nwhere\n$\\delta_{\\mathbb{S}^n_+} : \\mathbb{S}^n\\rightarrow (-\\infty,\\infty]$ is the indicator function of positive semidefinite cone $\\mathbb{S}^n_+$.\nDenote ${S}_{KKT}(a,b)$ the solution set of the KKT optimality condition for problem \\eqref{eq:NLSDP}, i.e.,\n\\begin{equation}\\label{eq:KKT-NLSDP-p1}\n\t{S}_{KKT}(a,b)=\\left\\{(x,Y)\\in \\mathbb{X}\\times\\mathbb{S}^n \\Bigg | \\left.\n\t\\begin{array}{l}\n\t\tL_x'(x,Y)-a=0,\\\\ [3pt]\n\t\t\\mathbb{S}_+^n\\ni (G(x)-b)\\perp Y \\in \\mathbb{S}_-^n.\n\t\\end{array}\n\t\\right.\\right \\}.\n\\end{equation}\nFor any KKT pair $(\\bar{x},\\overline{Y})$ that satisfies the KKT condition with $(a,b) = (0,0)$, we call $\\bar{x}$ a stationary point.\nSuppose $\\bar{x}$ is a stationary point. Define ${\\cal M}(\\bar{x})$ as the set of all multipliers $Y\\in\\mathbb{S}^n$ satisfying the KKT condition \\eqref{eq:KKT-NLSDP-p1}, i.e.,\n\\begin{equation}\\label{eq:defcalm}\n\t{\\cal M}(\\bar{x})=\\{Y\\in\\mathbb{S}^n \\mid (\\bar{x},Y)\\in {S}_{KKT}(0,0)\\}.\n\\end{equation}\n\n\nThe definition of strong SOSC for NLSDP, which demands the supremum of \\eqref{eq:defssosc} over $Y\\in{\\cal M}(\\bar{x})$ holds, is originally given by \\cite{Sun06} as an analogue for Robinson \\cite{Robin80} for the nonlinear programming (NLP). However, the strong SOSC mentioned here is slightly different in only requiring the validity of \\eqref{eq:defssosc} at $\\overline{Y}$. It is worth to note when ${\\cal M}(\\bar{x})$ is a singleton, both of them are the same.\n\\begin{definition}\\label{def:sigt}\\cite[Definition 3.2]{Sun06}\n\tLet $\\bar{x}$ be a stationary point of NLSDP \\eqref{eq:NLSDP} and $\\overline{Y}\\in{\\cal M}(\\bar{x})$. We say the strong second order sufficient condition (SOSC) holds at $(\\bar{x},\\overline{Y})$ if\n\t\\begin{equation}\\label{eq:defssosc}\n\t\t\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)>0\\quad\\forall\\; 0\\neq G'(\\bar{x})d\\in{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y}).\n\t\\end{equation}\n\twhere $\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)=2\\langle\\overline{Y},(G'(\\bar{x}){d})G(\\bar{x})^{\\dagger}(G'(\\bar{x}){d})\\rangle$ is the $\\sigma$-term and $G(\\bar{x})^{\\dagger}$ is the generalized inverse matrix of $G(\\bar{x})$.\n\\end{definition}\n\nSuppose $A=G(\\bar{x})+\\overline{Y}$ possesses the decomposition \\eqref{eq:eig-decomp}. From \\cite[page 386]{CSun08}, we know the $\\sigma$-term takes the explicit form of\n\\begin{equation}\\label{eq:sigtsdp}\n\t\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},H\\big)=2\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H}_{ij})^2,\n\\end{equation}\nwhere $\\widetilde{H}=P^THP$.\n\n\nNext, we introduce some essential notation for our main result. The (second order) generalized differentiability  of the augmented Lagrangian function ${\\cal L}_{\\rho}$ defined by \\eqref{eq:deflag} for NLSDP \\eqref{eq:NLSDP} has been explicitly established in \\cite{SSZhang08}.  In fact, it is well-known that ${\\cal L}_{\\rho}$ is continuously differentiable with\n\\begin{equation}\\label{eq:algrad}\n\t({\\cal L}_{\\rho})_x'(x,Y)=f'(x)+\\rho G'(x)^*\\Pi_{\\mathbb{S}_-^n}(G(x)+\\rho^{-1}Y),\n\\end{equation}\nwhere $\\Pi_{\\mathbb{S}_-^n}(\\cdot)$ is the metric projection over $\\mathbb{S}_{-}^n$ and $G'(x)^*$ denotes the adjoint of the corresponding linear mapping. Moreover, since $({\\cal L}_{\\rho})_x'$ is  Lipschitz continuous, by using Rademacher’s theorem, the B(ouligand)-subdifferential of $({\\cal L}_{\\rho})_x'$ at $(x,Y)$ is given by\n\\begin{equation}\\label{eq:partb}\n\t\\partial_B\\big(({\\cal L}_{\\rho})'_x\\big)(x,Y):=\\{\\lim_{k\\rightarrow\\infty}(({\\cal L}_{\\rho})_{x}')'(x^k,Y^k)\\mid (x^k,Y^k)\\in{\\cal U},\\ (x^k,Y^k)\\rightarrow(x,Y)\\},\n\\end{equation}\nwhere ${\\cal U}$ is the set of Fr\\'{e}chet-differentiable points of $({\\cal L}_{\\rho})_x'$.\nIt follows from \\cite[(18)]{SSZhang08} (using \\cite[Lemma 2 and 3]{SSZhang08}) that\n\\begin{align*}\n\t&\\pi_x\\partial_B\\big(({\\cal L}_{\\rho})_x'\\big)(x,Y)(\\Delta x)\\\\\n\t&=L_{xx}''(x,\\rho\\Pi_{\\mathbb{S}_-^n}(G(x)+\\rho^{-1}Y))(\\Delta x)+\\rho G'({x})^*\\partial_B\\Pi_{\\mathbb{S}_-^n}(G({x})+\\rho^{-1}Y) G'({x})(\\Delta x),\n\\end{align*}\nwhere $\\pi_x\\partial_B\\big(({\\cal L}_{\\rho})_x'\\big)(x,Y)$ is the projection of $\\partial_B\\big(({\\cal L}_{\\rho})_x'\\big)(x,Y)$ onto the space $\\mathbb{X}$ and  $\\partial_B\\Pi_{\\mathbb{S}_-^n}(G({x})+\\rho^{-1}Y)$ is the B-subdifferential of $\\Pi_{\\mathbb{S}_-^n}(\\cdot)$ at $G({x})+\\rho^{-1}Y$. Actually, the set $\\pi_x\\partial_B\\big(({\\cal L}_{\\rho})_x'\\big)(x,Y)$ is recently defined as the $x$ part of the Hessian bundle \\cite[(3.1)]{Roh22} for the augmented Lagrangian function ${\\cal L}_{\\rho}$ at $(x,Y)$ (see \\cite[(3.6)]{Roh22} for detail).\n\n\n\nLet $\\bar{x}$ be the stationary point of \\eqref{eq:NLSDP}. For each $\\overline{Y}\\in{\\cal M}(\\bar{x})$ and $W\\in\\partial_B\\Pi_{\\mathbb{S}_-^n}(G(\\bar{x})+\\rho^{-1}\\overline{Y})$, define the following mapping\n\\begin{equation}\\label{eq:algrad2}\n\t{\\cal A}_{\\rho}(\\overline{Y},W):=L_{xx}''(\\bar{x},\\overline{Y})+\\rho G'(\\bar{x})^*WG'(\\bar{x}).\n\\end{equation}\nLet $\\overline{Z}=G(\\bar{x})+\\overline{Y}$ possesses decomposition \\eqref{eq:eig-decomp} with $\\overline{P}\\in{\\cal O}^{n}(\\overline{Z})$. It follows from \\cite[Lemma 5]{SSZhang08} that $W\\in\\partial_B\\Pi_{\\mathbb{S}_-^n}(\\overline{Z})$ if and only if there exists $W_0\\in\\partial_B\\Pi_{\\mathbb{S}_-^{\\lvert\\beta\\rvert}}(0)$ such that for all $H\\in\\mathbb{S}^n$\n{\\footnotesize\n\\begin{equation*}\n\tW(H)=\\overline{P}\\left[\\begin{array}{ccc}\n\t\t0 & 0 & \\Sigma_{\\alpha\\gamma}\\circ\\overline{P}_{\\alpha}^TH\\overline{P}_{\\gamma}\\\\\n\t\t0 & W_0(\\overline{P}_{\\beta}^TH\\overline{P}_{\\beta}) & \\overline{P}_{\\beta}^TH\\overline{P}_{\\gamma} \\\\\n\t\t\\Sigma_{\\gamma\\alpha}\\circ\\overline{P}_{\\gamma}^TH\\overline{P}_{\\alpha} & \\overline{P}_{\\gamma}^TH\\overline{P}_{\\beta} & \\overline{P}_{\\gamma}^TH\\overline{P}_{\\gamma}\n\t\\end{array}\\right]\\overline{P}^T,\n\\end{equation*}}\nwhere\n\\begin{equation*}\n\\begin{cases}\t\t\\Sigma_{ij}=1-\\frac{\\max\\{\\lambda_i,0\\}+\\max\\{\\lambda_j,0\\}}{\\lvert\\lambda_i\\rvert+\\lvert\\lambda_j\\rvert}, & (i,j)\\notin\\beta\\times\\beta\\\\\n\\Sigma_{ij}\t\\in[0,1] & (i,j)\\in\\beta\\times\\beta\n\t\\end{cases}\n\\end{equation*}\nwith $\\lambda_i:=\\lambda_i(\\overline{Z})$ for short. Let ${\\cal Q}^n$ denote the set of $n$-dimensional orthogonal matrix.\nAlso, $W_0\\in\\partial_B\\Pi_{\\mathbb{S}_-^{\\lvert\\beta\\rvert}}(0)$ if and only if there exist $Q\\in{\\cal Q}^{\\lvert\\beta\\rvert}$ and $\\Omega\\in\\mathbb{S}^{\\lvert\\beta\\rvert}$ with entries $\\Omega_{ij}\\in[0,1]$ such that for all $H\\in\\mathbb{S}^{\\lvert\\beta\\rvert}$,\n$$W_0(H)=Q(\\Omega\\circ(Q^THQ))Q^T.$$\nBy \\cite[Lemma 9]{SSZhang08}, we have\n{\\small \\begin{align}\n\t\\langle d,{\\cal A}_{\\rho}(\\overline{Y},W)d\\rangle=&\\langle d,L_{xx}''(\\bar{x},\\overline{Y})d\\rangle+\\rho\\sum_{i,j\\in\\gamma}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2+2\\rho\\sum_{i\\in\\beta,j\\in\\gamma}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\label{eq:expcalA}\\\\\n\t&+2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2+\\rho\\sum_{i,j\\in\\beta}(\\Omega_{\\rho})_{ij}({P}^T(G'(\\bar{x})d){P})_{ij}^2,\\nonumber\n\\end{align}}\nwhere $\\Omega_{\\rho}\\in\\mathbb{S}^{\\lvert\\beta\\rvert}$ with entries $(\\Omega_{\\rho})_{ij}\\in[0,1]$, $P=[\\overline{P}_{\\alpha}\\;\\overline{P}_{\\beta}Q\\;\\overline{P}_{\\gamma}]$ with $Q\\in{\\cal Q}^{\\lvert\\beta\\rvert}$.\n\n\nThe definition of (strong) variational sufficient condition is given in \\cite{Roh22}  for general composite optimization problem \\eqref{eq:genp}.\nWe can recast \\eqref{eq:genp} in the form\n\\begin{equation}\\label{eq:pgpert}\n\t\\min\\;\\phi(x,u)\\;\\mbox{subject to}\\;u=0,\\quad\\mbox{where}\\;\\phi(x,u)=f(x)+\\theta(G(x)+u).\n\\end{equation}\nThe first order local optimality condition for \\eqref{eq:genp}  of $\\bar{x}$ is the existence of $\\overline{Y}$ such that\n$$L_x'(\\bar{x},\\overline{Y})=0\\quad \\mbox{with}\\quad\\overline{Y}\\in\\partial\\, \\theta(G(\\bar{x})). $$\nDefine\n\\begin{equation}\\label{eq:phir}\n\\phi_r:=\\phi(x,u)+\\frac{r}{2}|u|^2.\n\\end{equation}\nThe variational (strong) convexity, which is firstly proposed in \\cite{rock247}, refers to the existence of open convex neighborhoods ${\\cal W}$ of $(\\bar{x},0)$ and ${\\cal Z}$ of $(0,\\overline{Y})$ such that there exists a proper closed (strongly) convex function $\\psi\\leq\\phi_r2$ on ${\\cal W}$ such that\n\\begin{equation*}\n\t({\\cal W}\\times{\\cal Z})\\cap{\\rm gph}\\,\\partial\\psi=({\\cal W}\\times{\\cal Z})\\cap{\\rm gph}\\,\\partial\\phi_r\n\\end{equation*}\nand for $(x,u;v,y)$ belonging to this common set, $\\psi(x,u)=\\phi_r(x,u)$.\n\\begin{definition}\\label{def:svasc}\\cite{Roh22}\n\tThe (strong) variational sufficient condition for local optimality in \\eqref{eq:pgpert} holds with respect to $\\bar{x}$ and $\\overline{Y}$ satisfying the first order condition if there exists $r>0$ such that $\\displaystyle\\phi_r(x,u)$ is variationally (strongly) convex with respect to the pair $\\big((\\bar{x},0),(0,\\overline{Y})\\big)$ in ${\\rm gph}\\,\\partial\\phi_r$.\n\\end{definition}\n\n\nTo characterize the above property, the tools of the second subderivative and generalized quadratic form are needed.\n\\begin{definition}\\cite[Definition 13.6]{RoWe98}\n\tLet $\\bar{x}$ be a point where the function $f:\\mathbb{X}\\rightarrow[-\\infty,+\\infty]$ is finite. $f$ is twice epi-differentiable at $\\bar{x}$ for $v$ if the functions\n\t$$\\Delta^2_{t}f(\\bar{x}\\mid v)(u)=\\frac{f(\\bar{x}+tu)-f(\\bar{x})-t\\langle v,u\\rangle}{\\frac{1}{2}t^2}$$\n\tepi-converge to $d^2f(\\bar{x}\\mid v)$ as $t\\downarrow0$, where $d^2f(\\bar{x}\\mid v)$ is the second subderivative of $f$ at $\\bar{x}$ for $v$ defined as\n\t$$d^2f(\\bar{x}\\mid v)(w)=\\liminf_{t\\downarrow0,u\\rightarrow w}\\Delta^2_{t}f(\\bar{x}\\mid v)(u)$$\n\\end{definition}\nWe know from \\cite[Theorem 3.6]{mohammadi20} that $\\delta_{\\mathbb{S}_+^n}$ is twice epi-differentiable at $X$ for $Y$ with $Y\\in{\\cal N}_{\\mathbb{S}_+^n}(X)$.\n\n\\begin{definition}\\label{def:genguadf} \\cite{Roh22}\n\tA generalized linear mapping ${\\cal R}$ from $\\mathbb{X}$ to $\\mathbb{Y}$, is a set-valued mapping for which ${\\rm gph}\\,{\\cal R}$ is a subspace of $\\mathbb{X}\\times\\mathbb{Y}$. This means that ${\\rm dom}{\\cal R}$ is a subspace $\\mathbb{Z}$ of $\\mathbb{X}$, ${\\cal R}(0)$ is a subspace $\\mathbb{Z}'$ of $\\mathbb{Y}$, and there is an ordinary linear mapping ${\\cal R}_0:\\mathbb{Z}\\rightarrow\\mathbb{Y}$ such that ${\\cal R}(x)={\\cal R}_0(x)+\\mathbb{Z}'$ for $x\\in\\mathbb{Z}$.\n\tWe call a function $q: \\mathbb{X}\\rightarrow(-\\infty,+\\infty]$ is a generalized quadratic form on $\\mathbb{X}$ if $q(0)=0$ and the subgradient mapping $\\partial q:\\mathbb{X}\\rightrightarrows\\mathbb{X}$ is generalized linear. A function $g$ on $\\mathbb{X}$ will be called generalized twice differentiable at $x$ for a subgradient $y$ if it is twice epi-differentiable at $x$ for $y$ with the second-order subderivative $d^2g(x\\mid y)$ being a generalized quadratic form.\n\\end{definition}\n\n\nThe following definition of the quadratic bundle is taken from \\cite{Roh22}, which is essential in characterizing the strong variational sufficiency.\n\\begin{definition}\\label{def:quadratic bundle}\n\tFor general optimization problem \\eqref{eq:genp}, suppose $(\\bar{x},\\overline{Y})$ is a KKT pair. The quadratic bundle of $\\theta$ is defined as\n\t\\begin{equation*}\n\t\t{\\rm quad}\\,\\theta(G(\\bar{x})\\mid\\overline{Y})=\\left\\{\n\t\t\\begin{array}{l}\n\t\t\t\\mbox{the collection of generalized quadratic forms}\\; q\\; \\mbox{for which}\\\\\n\t\t\t\\exists(X^k,Y^k)\\rightarrow(G(\\bar{x}),\\overline{Y})\\;\\mbox{with}\\;\\theta\\;\\mbox{generalized twice differentiable}\\\\\n\t\t\t\\mbox{at}\\;X^k\\;\\mbox{for}\\;Y^k\\;\\mbox{and such that the generalized quadratic}\\\\\n\t\t\t\\mbox{forms}\\;q_k=\\frac{1}{2}d^2\\theta(X^k\\mid Y^k)\\;\\mbox{converge epigraphically to}\\;q.\n\t\t\\end{array}\n\t\t\\right.\n\t\\end{equation*}\n\\end{definition}\nAs mentioned in \\cite{Roh22}, the variational sufficient condition guarantees the local optimality for \\eqref{eq:pgpert}. The following result is taken from \\cite[Theorem 5]{Roh22}, which is useful for the subsequent analysis.\n\\begin{proposition}\\label{prop:thm5}\n\tFor general optimization problem \\eqref{eq:genp},\n\tstrong variational sufficient condition for local optimality with respect to $(\\bar{x},\\overline{Y})$ is equivalent to that  every $q\\in{\\rm quad}\\,\\theta(G(\\bar{x})\\mid\\overline{Y})$ has\n\t\\begin{equation}\\label{svs-equ}\n\t\t\\frac{1}{2}\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle+q(G'(\\bar{x}){d})>0\\quad{\\rm when}\\;d\\neq0,\n\t\\end{equation}\n\twhere ${\\rm quad}\\,\\theta(G(\\bar{x})\\mid\\overline{Y})$ is the quadratic bundle of $\\theta$.\n\\end{proposition}\n","units":[{"raw_tex":" The definition of (strong) variational sufficient condition (Definition \\ref{def:svasc}) is officially given in \\cite{Roh22}  for the following general composite optimization problem\n \\begin{equation}\\label{eq:genp}\n \t\\min_{x\\in \\mathbb{X}}\\quad f(x)+\\theta(G(x)),\n \\end{equation}\n where $\\mathbb{X}$ and $\\mathbb{Y}$ are two given Euclidean spaces, $f:\\mathbb{X}\\to \\mathbb{R}$ and $G:\\mathbb{X}\\to \\mathbb{Y}$ are twice continuously differentiable, and $\\theta:\\mathbb{Y}\\to (-\\infty,\\infty]$ is a closed proper convex function. The strong variational sufficient condition is firstly introduced to deal with the local convexity of primal augmented Lagrangian function and the augmented tilt stability. It has gained more and more attention for its mathematical elegance and wide applications in the convergence analysis of multiplier methods. Several characterizations of this abstract property are also given in \\cite{Roh22}. For instance, it is equivalent to the positive definiteness of the Hessian bundle of the augmented Lagrangian function \\cite[Theorem 3]{Roh22} or the criterion \\cite[Theorem 5]{Roh22} involving quadratic bundle (Definition \\ref{def:quadratic bundle}).\n","start":6647,"end":7829},{"raw_tex":" In terms of the connection with traditional optimality conditions, Rockafellar \\cite[Theorem 4]{Roh22} shows that the strong variational sufficiency is equivalent to the well-known strong second order sufficient condition (SOSC), which is expressed entirely via the program data, when the function $\\theta$ in \\eqref{eq:genp} is polyhedral convex (i.e., the epigraph ${\\rm epi}\\,\\theta$ of $\\theta$ is a polyhedral convex set). For non-polyhedral problems,  the equivalence is still valid if $\\theta$ in \\eqref{eq:genp} is the indicator function of the second order cone with $G(\\bar{x})\\neq 0$, where $\\bar{x}$ is a local optimal solution (see \\cite[Example 3]{Roh22} for details). However, an explicit and verifiable characterization of strong variational sufficient condition for general non-polyhedral problems remains unknown. In this paper, without loss of generality, we mainly focus on the characterization of strong variational sufficiency for the following nonlinear semidefinite programming (NLSDP):\n\\begin{equation}\\label{eq:NLSDPc}\n\t\\begin{array}{cl}\n\t\t\\displaystyle{\\min_{x\\in \\mathbb{X}}} & f(x) \\\\ [3pt]\n\t\t{\\rm s.t.} &{G(x)\\in \\mathbb{S}^n_+},\n\t\\end{array}\n\\end{equation}\nwhere  $\\mathbb{S}^n$ is the linear space of all $n\\times n$ real symmetric matrices equipped with the usual Frobenius inner product and its induced norm, $\\mathbb{S}_+^n$ ($\\mathbb{S}_-^n$) is the closed convex cone of all $n\\times n$ positive (negative) semidefinite matrices in $\\mathbb{S}^n$. One of our contributions lies in uncovering the equivalence between strong variational sufficient condition and strong SOSC (see Definition \\ref{def:sigt} for details) for  NLSDP problems without requiring any other constraint qualifications.\n","start":7831,"end":9560},{"raw_tex":"For NLSDP \\eqref{eq:NLSDPc}, the Lagrangian function is defined by\n\\begin{equation}\\label{eq:L-function}\n\tL(x,Y):=f(x) +\\langle Y,G(x)\\rangle, \\quad (x,Y)\\in\\mathbb{X}\\times\\mathbb{S}^n.\n\\end{equation}\nFor any $Y\\in\\mathbb{S}^n$, denote the first-order and second-order derivatives of $L(\\cdot,Y)$ at $x\\in\\mathbb{X}$ by $L_x'(x,Y)$ and $L_{xx}''(x,Y)$, respectively. Given $\\rho > 0$, the augmented Lagrangian function of \\eqref{eq:NLSDPc} takes the following form (cf. \\cite[Section 11.K]{RoWe98} and \\cite{ShapS04})\n\\begin{equation}\\label{eq:deflag}\n\t\\mathcal{L}_{\\rho}(x,Y):=f(x)+\\frac{\\rho}{2}{\\rm dist}^2(G(x)+\\frac{Y}{\\rho},\\mathbb{S}_+^n)-\\frac{\\|Y\\|^2}{2\\rho},\n\\end{equation}\nwhere ${\\rm dist}(z,\\mathbb{S}_+^n)$ is the distance from point $z$ to $\\mathbb{S}_+^n$.\n","start":11177,"end":11951},{"raw_tex":"For a given Euclidean space $\\mathbb{X}$,\nlet $C$ be any subset in $\\mathbb{X}$, ${\\rm aff}\\,{C}$ is the affine hull of $C$ \\cite[page 6]{Rockafellar1970}. The Bouligand tangent/contingent cone of $C$ at $x$ is a closed cone defined by\n\\[\n\\mathcal{T}_{C}(x):= \\left\\{d\\in\\mathbb{X}\\mid \\mbox{$\\exists\\, t^k\\downarrow 0$ and  $d^k\\to d$ with  $x+t^kd^k\\in C$ for all $k$} \\right\\}.\n\\]\nWhen $C$ is convex, the normal cone in the sense of  convex analysis \\cite{Rockafellar1970} is defined as\n$$\\mathcal{N}_{C}(x)=\\left \\{d \\in \\mathbb{X}\\mid\\langle d,x'-x\\rangle\\leq 0 \\ \\forall\\,x'\\in C \\right\\}.$$\nFor any $x\\in C$, the critical cone  associated with ${ y}\\in\\mathcal{N}_C(x)$ of $C$ is defined in \\cite[Page 98]{DoRo14}, i.e., ${\\cal C}_{C}({x},{y})=\\mathcal{T}_{C}({x})\\cap({y})^{\\perp}$.\n\nFor a set-valued mapping $\\mathcal{F}:\\mathbb{X}\\rightrightarrows\\mathbb{X}$, its $\\limsup$ means\n$$\\limsup\\limits_{{x}\\rightarrow\\bar{x}}\\mathcal{F}({x}):=\\left\\{d\\in\\mathbb{X}\\mid \\exists\\;x_k\\rightarrow\\bar{x},\\; {y}_k\\rightarrow d\\;{\\rm such\\;that}\\;{y}_k\\in\\mathcal{F}({x}_k)\\;\\forall\\,k\\right\\}.$$\nThe following definitions of proximal and (general/Mordukhovich limiting) subdifferentials of functions are adopted from \\cite[Definition 8.45 and 8.3]{RoWe98}.\n\\begin{definition}\n\tConsider a function $f:\\mathbb{X}\\rightarrow(-\\infty, +\\infty]$ and a point $\\bar{x}$ with $f(\\bar{x})$ finite. The proximal subdifferential of $f$ at $\\bar{x}$ is defined as\n\t$$\\partial^{\\pi}f(\\bar{x}):=\\{{d}\\mid \\exists\\; l>0, r>0\\;{\\rm such\\; that}\\;f(x)\\geq f(\\bar{x})+\\langle{d},{x}-\\bar{x}\\rangle-l\\|{x}-\\bar{x}\\|^2\\;\\forall\\,{x}\\in{\\cal B}_{r}(\\bar{x})\\},$$\n\twhere ${\\cal B}_{r}(\\bar{x})$ is the ball centered at $\\bar{x}$ with radius $r$.\n\tThe (general/Mordukhovich limiting) subdifferential of $f$ at $\\bar{x}$ is $\\partial f(\\bar{x}):=\\limsup\\limits_{{x}\\stackrel{f}{\\rightarrow}\\bar{x}}\\partial^{\\pi}f({x})$, where ${x}\\stackrel{f}{\\rightarrow}\\bar{x}$ signifies ${x}\\rightarrow\\bar{x}$ with $f({x})\\rightarrow f(\\bar{x})$.\n\\end{definition}\nNote that in an Euclidean space, the (general) subdifferential also can be constructed via the regular/Fr\\'{e}chet subdifferentials  of functions $\\widehat{\\partial}f$ \\cite[Definition 8.3]{RoWe98}. It is well-known that these two definitions are equivalent (see e.g., \\cite[page 345]{RoWe98} or \\cite[Theorem 1.89]{Mordukhovich06} for details). In particular, when $f$ is convex and finite at the corresponding point, the proximal and (general) subdifferentials coincide with the subdifferential in the sense of convex analysis \\cite{Rockafellar1970}.\n","start":16485,"end":19067},{"raw_tex":"Let  $A \\in\\mathbb{S}^n$ be given. We use $\\lambda_{1}(A)\\ge \\lambda_2(A) \\ge \\ldots \\ge \\lambda_{n}(A)$ to denote the eigenvalues of $A$ (all real and counting multiplicity) arranging in  nonincreasing order and use $\\lambda(A)$ to denote the vector of the ordered eigenvalues of $A$. Let $\\Lambda(A):= {\\rm Diag}(\\lambda(A))$. Also, we use $v_1(A)> \\dots> v_d(A)$ to denote the different eigenvalues and\n$\\zeta_l:=\\{i\\mid \\lambda_i(A)=v_l(A)\\}$.\nConsider the eigenvalue decomposition of $A$, i.e., $A={P} \\Lambda(A){P}^{T}$, where ${P}\\in{\\cal O}^{n}(A)$ is a corresponding\northogonal matrix of the orthonormal eigenvectors.\nBy considering the index sets of positive, zero, and negative eigenvalues of $A$, we are able to write $A$ in the following form\n{\\footnotesize\n\\begin{equation}\\label{eq:eig-decomp}\n\tA= \\left[\\begin{array}{ccc}\n\t\t{P}_{\\alpha} & {P}_{\\beta} & {P}_{\\gamma}\n\t\\end{array}\\right] \\left[\\begin{array}{ccc}\n\t\t\\Lambda(A)_{\\alpha\\alpha} & 0 & 0 \\\\ [3pt]\n\t\t0 & 0 & 0\\\\ [3pt]\n\t\t0 & 0 & \\Lambda(A)_{\\gamma\\gamma}\n\t\\end{array}\\right]\\left[\\begin{array}{c}\n\t\t{P}_{\\alpha}^T \\\\ [3pt]\n\t\t{P}_{\\beta}^T \\\\\n\t\t{P}_{\\gamma}^T\n\t\\end{array}\\right],\n\\end{equation}}\nwhere $\\alpha:=\\{i: \\lambda_i(A)>0\\}$, $\\beta:=\\{i: \\lambda_i(A)=0\\}$ and $\\gamma:=\\{i: \\lambda_i(A)<0\\}$.\n","start":19129,"end":20405},{"raw_tex":"The critical cone of $\\mathbb{S}_+^n$ at ${Y}\\in{\\cal N}_{\\mathbb{S}_+^n}({X})$ with $A=X+Y$ is given by (cf. \\cite[equations (11) and (12)]{CSun08})\n\\begin{equation}\\label{eq:cricsdp}\n\t{\\cal C}_{\\mathbb{S}_+^n}({X},{Y})=\\{H\\in\\mathbb{S}^n\\mid P^T_{\\beta}HP_{\\beta}\\in\\mathbb{S}^{|\\beta|}_+,\\;P^T_{\\beta}HP_{\\gamma}=0,\\;P^T_{\\gamma}HP_{\\gamma}=0\\}\n\\end{equation}\nand the affine hull of ${\\cal C}_{\\mathbb{S}_+^n}({X},{Y})$ is\n\\begin{equation}\\label{eq:affcsdp}\n\t{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}({X},{Y})=\\{H\\in\\mathbb{S}^n\\mid P^T_{\\beta}HP_{\\gamma}=0,\\;P^T_{\\gamma}HP_{\\gamma}=0\\}.\n\\end{equation}\nThe  NLSDP \\eqref{eq:NLSDPc} can be written in the following form\n\\begin{eqnarray}\n\t%\\mbox{(SDP)}~~~~\n\t\\min\\quad && f(x)+\\delta_{\\mathbb{S}^n_+}(G(x)), \\label{eq:NLSDP}\n\\end{eqnarray}\nwhere\n$\\delta_{\\mathbb{S}^n_+} : \\mathbb{S}^n\\rightarrow (-\\infty,\\infty]$ is the indicator function of positive semidefinite cone $\\mathbb{S}^n_+$.\nDenote ${S}_{KKT}(a,b)$ the solution set of the KKT optimality condition for problem \\eqref{eq:NLSDP}, i.e.,\n\\begin{equation}\\label{eq:KKT-NLSDP-p1}\n\t{S}_{KKT}(a,b)=\\left\\{(x,Y)\\in \\mathbb{X}\\times\\mathbb{S}^n \\Bigg | \\left.\n\t\\begin{array}{l}\n\t\tL_x'(x,Y)-a=0,\\\\ [3pt]\n\t\t\\mathbb{S}_+^n\\ni (G(x)-b)\\perp Y \\in \\mathbb{S}_-^n.\n\t\\end{array}\n\t\\right.\\right \\}.\n\\end{equation}\nFor any KKT pair $(\\bar{x},\\overline{Y})$ that satisfies the KKT condition with $(a,b) = (0,0)$, we call $\\bar{x}$ a stationary point.\nSuppose $\\bar{x}$ is a stationary point. Define ${\\cal M}(\\bar{x})$ as the set of all multipliers $Y\\in\\mathbb{S}^n$ satisfying the KKT condition \\eqref{eq:KKT-NLSDP-p1}, i.e.,\n\\begin{equation}\\label{eq:defcalm}\n\t{\\cal M}(\\bar{x})=\\{Y\\in\\mathbb{S}^n \\mid (\\bar{x},Y)\\in {S}_{KKT}(0,0)\\}.\n\\end{equation}\n\n\nThe definition of strong SOSC for NLSDP, which demands the supremum of \\eqref{eq:defssosc} over $Y\\in{\\cal M}(\\bar{x})$ holds, is originally given by \\cite{Sun06} as an analogue for Robinson \\cite{Robin80} for the nonlinear programming (NLP). However, the strong SOSC mentioned here is slightly different in only requiring the validity of \\eqref{eq:defssosc} at $\\overline{Y}$. It is worth to note when ${\\cal M}(\\bar{x})$ is a singleton, both of them are the same.\n\\begin{definition}\\label{def:sigt}\\cite[Definition 3.2]{Sun06}\n\tLet $\\bar{x}$ be a stationary point of NLSDP \\eqref{eq:NLSDP} and $\\overline{Y}\\in{\\cal M}(\\bar{x})$. We say the strong second order sufficient condition (SOSC) holds at $(\\bar{x},\\overline{Y})$ if\n\t\\begin{equation}\\label{eq:defssosc}\n\t\t\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)>0\\quad\\forall\\; 0\\neq G'(\\bar{x})d\\in{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y}).\n\t\\end{equation}\n\twhere $\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)=2\\langle\\overline{Y},(G'(\\bar{x}){d})G(\\bar{x})^{\\dagger}(G'(\\bar{x}){d})\\rangle$ is the $\\sigma$-term and $G(\\bar{x})^{\\dagger}$ is the generalized inverse matrix of $G(\\bar{x})$.\n\\end{definition}\n\nSuppose $A=G(\\bar{x})+\\overline{Y}$ possesses the decomposition \\eqref{eq:eig-decomp}. From \\cite[page 386]{CSun08}, we know the $\\sigma$-term takes the explicit form of\n\\begin{equation}\\label{eq:sigtsdp}\n\t\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},H\\big)=2\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H}_{ij})^2,\n\\end{equation}\nwhere $\\widetilde{H}=P^THP$.\n","start":20408,"end":23802},{"raw_tex":"Next, we introduce some essential notation for our main result. The (second order) generalized differentiability  of the augmented Lagrangian function ${\\cal L}_{\\rho}$ defined by \\eqref{eq:deflag} for NLSDP \\eqref{eq:NLSDP} has been explicitly established in \\cite{SSZhang08}.  In fact, it is well-known that ${\\cal L}_{\\rho}$ is continuously differentiable with\n\\begin{equation}\\label{eq:algrad}\n\t({\\cal L}_{\\rho})_x'(x,Y)=f'(x)+\\rho G'(x)^*\\Pi_{\\mathbb{S}_-^n}(G(x)+\\rho^{-1}Y),\n\\end{equation}\nwhere $\\Pi_{\\mathbb{S}_-^n}(\\cdot)$ is the metric projection over $\\mathbb{S}_{-}^n$ and $G'(x)^*$ denotes the adjoint of the corresponding linear mapping. Moreover, since $({\\cal L}_{\\rho})_x'$ is  Lipschitz continuous, by using Rademacher’s theorem, the B(ouligand)-subdifferential of $({\\cal L}_{\\rho})_x'$ at $(x,Y)$ is given by\n\\begin{equation}\\label{eq:partb}\n\t\\partial_B\\big(({\\cal L}_{\\rho})'_x\\big)(x,Y):=\\{\\lim_{k\\rightarrow\\infty}(({\\cal L}_{\\rho})_{x}')'(x^k,Y^k)\\mid (x^k,Y^k)\\in{\\cal U},\\ (x^k,Y^k)\\rightarrow(x,Y)\\},\n\\end{equation}\nwhere ${\\cal U}$ is the set of Fr\\'{e}chet-differentiable points of $({\\cal L}_{\\rho})_x'$.\nIt follows from \\cite[(18)]{SSZhang08} (using \\cite[Lemma 2 and 3]{SSZhang08}) that\n\\begin{align*}\n\t&\\pi_x\\partial_B\\big(({\\cal L}_{\\rho})_x'\\big)(x,Y)(\\Delta x)\\\\\n\t&=L_{xx}''(x,\\rho\\Pi_{\\mathbb{S}_-^n}(G(x)+\\rho^{-1}Y))(\\Delta x)+\\rho G'({x})^*\\partial_B\\Pi_{\\mathbb{S}_-^n}(G({x})+\\rho^{-1}Y) G'({x})(\\Delta x),\n\\end{align*}\nwhere $\\pi_x\\partial_B\\big(({\\cal L}_{\\rho})_x'\\big)(x,Y)$ is the projection of $\\partial_B\\big(({\\cal L}_{\\rho})_x'\\big)(x,Y)$ onto the space $\\mathbb{X}$ and  $\\partial_B\\Pi_{\\mathbb{S}_-^n}(G({x})+\\rho^{-1}Y)$ is the B-subdifferential of $\\Pi_{\\mathbb{S}_-^n}(\\cdot)$ at $G({x})+\\rho^{-1}Y$. Actually, the set $\\pi_x\\partial_B\\big(({\\cal L}_{\\rho})_x'\\big)(x,Y)$ is recently defined as the $x$ part of the Hessian bundle \\cite[(3.1)]{Roh22} for the augmented Lagrangian function ${\\cal L}_{\\rho}$ at $(x,Y)$ (see \\cite[(3.6)]{Roh22} for detail).\n\n\n\nLet $\\bar{x}$ be the stationary point of \\eqref{eq:NLSDP}. For each $\\overline{Y}\\in{\\cal M}(\\bar{x})$ and $W\\in\\partial_B\\Pi_{\\mathbb{S}_-^n}(G(\\bar{x})+\\rho^{-1}\\overline{Y})$, define the following mapping\n\\begin{equation}\\label{eq:algrad2}\n\t{\\cal A}_{\\rho}(\\overline{Y},W):=L_{xx}''(\\bar{x},\\overline{Y})+\\rho G'(\\bar{x})^*WG'(\\bar{x}).\n\\end{equation}\nLet $\\overline{Z}=G(\\bar{x})+\\overline{Y}$ possesses decomposition \\eqref{eq:eig-decomp} with $\\overline{P}\\in{\\cal O}^{n}(\\overline{Z})$. It follows from \\cite[Lemma 5]{SSZhang08} that $W\\in\\partial_B\\Pi_{\\mathbb{S}_-^n}(\\overline{Z})$ if and only if there exists $W_0\\in\\partial_B\\Pi_{\\mathbb{S}_-^{\\lvert\\beta\\rvert}}(0)$ such that for all $H\\in\\mathbb{S}^n$\n{\\footnotesize\n\\begin{equation*}\n\tW(H)=\\overline{P}\\left[\\begin{array}{ccc}\n\t\t0 & 0 & \\Sigma_{\\alpha\\gamma}\\circ\\overline{P}_{\\alpha}^TH\\overline{P}_{\\gamma}\\\\\n\t\t0 & W_0(\\overline{P}_{\\beta}^TH\\overline{P}_{\\beta}) & \\overline{P}_{\\beta}^TH\\overline{P}_{\\gamma} \\\\\n\t\t\\Sigma_{\\gamma\\alpha}\\circ\\overline{P}_{\\gamma}^TH\\overline{P}_{\\alpha} & \\overline{P}_{\\gamma}^TH\\overline{P}_{\\beta} & \\overline{P}_{\\gamma}^TH\\overline{P}_{\\gamma}\n\t\\end{array}\\right]\\overline{P}^T,\n\\end{equation*}}\nwhere\n\\begin{equation*}\n\\begin{cases}\t\t\\Sigma_{ij}=1-\\frac{\\max\\{\\lambda_i,0\\}+\\max\\{\\lambda_j,0\\}}{\\lvert\\lambda_i\\rvert+\\lvert\\lambda_j\\rvert}, & (i,j)\\notin\\beta\\times\\beta\\\\\n\\Sigma_{ij}\t\\in[0,1] & (i,j)\\in\\beta\\times\\beta\n\t\\end{cases}\n\\end{equation*}\nwith $\\lambda_i:=\\lambda_i(\\overline{Z})$ for short. Let ${\\cal Q}^n$ denote the set of $n$-dimensional orthogonal matrix.\nAlso, $W_0\\in\\partial_B\\Pi_{\\mathbb{S}_-^{\\lvert\\beta\\rvert}}(0)$ if and only if there exist $Q\\in{\\cal Q}^{\\lvert\\beta\\rvert}$ and $\\Omega\\in\\mathbb{S}^{\\lvert\\beta\\rvert}$ with entries $\\Omega_{ij}\\in[0,1]$ such that for all $H\\in\\mathbb{S}^{\\lvert\\beta\\rvert}$,\n$$W_0(H)=Q(\\Omega\\circ(Q^THQ))Q^T.$$\nBy \\cite[Lemma 9]{SSZhang08}, we have\n{\\small \\begin{align}\n\t\\langle d,{\\cal A}_{\\rho}(\\overline{Y},W)d\\rangle=&\\langle d,L_{xx}''(\\bar{x},\\overline{Y})d\\rangle+\\rho\\sum_{i,j\\in\\gamma}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2+2\\rho\\sum_{i\\in\\beta,j\\in\\gamma}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\label{eq:expcalA}\\\\\n\t&+2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2+\\rho\\sum_{i,j\\in\\beta}(\\Omega_{\\rho})_{ij}({P}^T(G'(\\bar{x})d){P})_{ij}^2,\\nonumber\n\\end{align}}\nwhere $\\Omega_{\\rho}\\in\\mathbb{S}^{\\lvert\\beta\\rvert}$ with entries $(\\Omega_{\\rho})_{ij}\\in[0,1]$, $P=[\\overline{P}_{\\alpha}\\;\\overline{P}_{\\beta}Q\\;\\overline{P}_{\\gamma}]$ with $Q\\in{\\cal Q}^{\\lvert\\beta\\rvert}$.\n","start":23803,"end":28477},{"raw_tex":"The definition of (strong) variational sufficient condition is given in \\cite{Roh22}  for general composite optimization problem \\eqref{eq:genp}.\nWe can recast \\eqref{eq:genp} in the form\n\\begin{equation}\\label{eq:pgpert}\n\t\\min\\;\\phi(x,u)\\;\\mbox{subject to}\\;u=0,\\quad\\mbox{where}\\;\\phi(x,u)=f(x)+\\theta(G(x)+u).\n\\end{equation}\nThe first order local optimality condition for \\eqref{eq:genp}  of $\\bar{x}$ is the existence of $\\overline{Y}$ such that\n$$L_x'(\\bar{x},\\overline{Y})=0\\quad \\mbox{with}\\quad\\overline{Y}\\in\\partial\\, \\theta(G(\\bar{x})). $$\nDefine\n\\begin{equation}\\label{eq:phir}\n\\phi_r:=\\phi(x,u)+\\frac{r}{2}|u|^2.\n\\end{equation}\nThe variational (strong) convexity, which is firstly proposed in \\cite{rock247}, refers to the existence of open convex neighborhoods ${\\cal W}$ of $(\\bar{x},0)$ and ${\\cal Z}$ of $(0,\\overline{Y})$ such that there exists a proper closed (strongly) convex function $\\psi\\leq\\phi_r2$ on ${\\cal W}$ such that\n\\begin{equation*}\n\t({\\cal W}\\times{\\cal Z})\\cap{\\rm gph}\\,\\partial\\psi=({\\cal W}\\times{\\cal Z})\\cap{\\rm gph}\\,\\partial\\phi_r\n\\end{equation*}\nand for $(x,u;v,y)$ belonging to this common set, $\\psi(x,u)=\\phi_r(x,u)$.\n\\begin{definition}\\label{def:svasc}\\cite{Roh22}\n\tThe (strong) variational sufficient condition for local optimality in \\eqref{eq:pgpert} holds with respect to $\\bar{x}$ and $\\overline{Y}$ satisfying the first order condition if there exists $r>0$ such that $\\displaystyle\\phi_r(x,u)$ is variationally (strongly) convex with respect to the pair $\\big((\\bar{x},0),(0,\\overline{Y})\\big)$ in ${\\rm gph}\\,\\partial\\phi_r$.\n\\end{definition}\n\n\nTo characterize the above property, the tools of the second subderivative and generalized quadratic form are needed.\n\\begin{definition}\\cite[Definition 13.6]{RoWe98}\n\tLet $\\bar{x}$ be a point where the function $f:\\mathbb{X}\\rightarrow[-\\infty,+\\infty]$ is finite. $f$ is twice epi-differentiable at $\\bar{x}$ for $v$ if the functions\n\t$$\\Delta^2_{t}f(\\bar{x}\\mid v)(u)=\\frac{f(\\bar{x}+tu)-f(\\bar{x})-t\\langle v,u\\rangle}{\\frac{1}{2}t^2}$$\n\tepi-converge to $d^2f(\\bar{x}\\mid v)$ as $t\\downarrow0$, where $d^2f(\\bar{x}\\mid v)$ is the second subderivative of $f$ at $\\bar{x}$ for $v$ defined as\n\t$$d^2f(\\bar{x}\\mid v)(w)=\\liminf_{t\\downarrow0,u\\rightarrow w}\\Delta^2_{t}f(\\bar{x}\\mid v)(u)$$\n\\end{definition}\nWe know from \\cite[Theorem 3.6]{mohammadi20} that $\\delta_{\\mathbb{S}_+^n}$ is twice epi-differentiable at $X$ for $Y$ with $Y\\in{\\cal N}_{\\mathbb{S}_+^n}(X)$.\n\n\\begin{definition}\\label{def:genguadf} \\cite{Roh22}\n\tA generalized linear mapping ${\\cal R}$ from $\\mathbb{X}$ to $\\mathbb{Y}$, is a set-valued mapping for which ${\\rm gph}\\,{\\cal R}$ is a subspace of $\\mathbb{X}\\times\\mathbb{Y}$. This means that ${\\rm dom}{\\cal R}$ is a subspace $\\mathbb{Z}$ of $\\mathbb{X}$, ${\\cal R}(0)$ is a subspace $\\mathbb{Z}'$ of $\\mathbb{Y}$, and there is an ordinary linear mapping ${\\cal R}_0:\\mathbb{Z}\\rightarrow\\mathbb{Y}$ such that ${\\cal R}(x)={\\cal R}_0(x)+\\mathbb{Z}'$ for $x\\in\\mathbb{Z}$.\n\tWe call a function $q: \\mathbb{X}\\rightarrow(-\\infty,+\\infty]$ is a generalized quadratic form on $\\mathbb{X}$ if $q(0)=0$ and the subgradient mapping $\\partial q:\\mathbb{X}\\rightrightarrows\\mathbb{X}$ is generalized linear. A function $g$ on $\\mathbb{X}$ will be called generalized twice differentiable at $x$ for a subgradient $y$ if it is twice epi-differentiable at $x$ for $y$ with the second-order subderivative $d^2g(x\\mid y)$ being a generalized quadratic form.\n\\end{definition}\n\n\nThe following definition of the quadratic bundle is taken from \\cite{Roh22}, which is essential in characterizing the strong variational sufficiency.\n\\begin{definition}\\label{def:quadratic bundle}\n\tFor general optimization problem \\eqref{eq:genp}, suppose $(\\bar{x},\\overline{Y})$ is a KKT pair. The quadratic bundle of $\\theta$ is defined as\n\t\\begin{equation*}\n\t\t{\\rm quad}\\,\\theta(G(\\bar{x})\\mid\\overline{Y})=\\left\\{\n\t\t\\begin{array}{l}\n\t\t\t\\mbox{the collection of generalized quadratic forms}\\; q\\; \\mbox{for which}\\\\\n\t\t\t\\exists(X^k,Y^k)\\rightarrow(G(\\bar{x}),\\overline{Y})\\;\\mbox{with}\\;\\theta\\;\\mbox{generalized twice differentiable}\\\\\n\t\t\t\\mbox{at}\\;X^k\\;\\mbox{for}\\;Y^k\\;\\mbox{and such that the generalized quadratic}\\\\\n\t\t\t\\mbox{forms}\\;q_k=\\frac{1}{2}d^2\\theta(X^k\\mid Y^k)\\;\\mbox{converge epigraphically to}\\;q.\n\t\t\\end{array}\n\t\t\\right.\n\t\\end{equation*}\n\\end{definition}\nAs mentioned in \\cite{Roh22}, the variational sufficient condition guarantees the local optimality for \\eqref{eq:pgpert}. The following result is taken from \\cite[Theorem 5]{Roh22}, which is useful for the subsequent analysis.\n\\begin{proposition}\\label{prop:thm5}\n\tFor general optimization problem \\eqref{eq:genp},\n\tstrong variational sufficient condition for local optimality with respect to $(\\bar{x},\\overline{Y})$ is equivalent to that  every $q\\in{\\rm quad}\\,\\theta(G(\\bar{x})\\mid\\overline{Y})$ has\n\t\\begin{equation}\\label{svs-equ}\n\t\t\\frac{1}{2}\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle+q(G'(\\bar{x}){d})>0\\quad{\\rm when}\\;d\\neq0,\n\t\\end{equation}\n\twhere ${\\rm quad}\\,\\theta(G(\\bar{x})\\mid\\overline{Y})$ is the quadratic bundle of $\\theta$.\n\\end{proposition}\n","start":28523,"end":33654}]},"statement":{"raw_tex":"\\begin{theorem}\\label{thm:varscequissosc}\n\tLet $\\bar{x}\\in\\mathbb{X}$ be a stationary point to the NLSDP \\eqref{eq:NLSDP} and $\\overline{Y}\\in {\\cal M}(\\bar{x})$. Then the following three conditions are equivalent\n\t\\begin{itemize}\n\t\t\\item[(i)] the strong variational sufficient condition with respect to $(\\bar{x},\\overline{Y})$ holds;\n\t\t\\item[(ii)] the strong second order sufficient condition (SOSC) \\eqref{eq:defssosc} holds at $(\\bar{x},\\overline{Y})$;\n\t\t\\item[(iii)] there exist $\\rho_0>0$ and $\\underline{\\eta}>0$ such that for any $\\rho\\geq \\rho_0$ and any $W\\in\\partial_B\\Pi_{\\mathbb{S}_-^n}(G(\\bar{x})+\\rho^{-1}\\overline{Y})$,\n\t\t\\begin{equation*}\\label{eq:psdofpb}\n\t\t\\langle d, {\\cal A}_{\\rho}(\\overline{Y},W)d\\rangle\\geq\\underline{\\eta}\\|d\\|^2\\quad\\forall d\\in\\mathbb{X},\n\t\t\\end{equation*}\n\t\twhere ${\\cal A}_{\\rho}(\\overline{Y},W)$ is defined by \\eqref{eq:algrad2}.\n\t\\end{itemize}\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\\label{thm:varscequissosc}\n\tLet $\\bar{x}\\in\\mathbb{X}$ be a stationary point to the NLSDP \\eqref{eq:NLSDP} and $\\overline{Y}\\in {\\cal M}(\\bar{x})$. Then the following three conditions are equivalent\n\t\\begin{itemize}\n\t\t\\item[(i)] the strong variational sufficient condition with respect to $(\\bar{x},\\overline{Y})$ holds;\n\t\t\\item[(ii)] the strong second order sufficient condition (SOSC) \\eqref{eq:defssosc} holds at $(\\bar{x},\\overline{Y})$;\n\t\t\\item[(iii)] there exist $\\rho_0>0$ and $\\underline{\\eta}>0$ such that for any $\\rho\\geq \\rho_0$ and any $W\\in\\partial_B\\Pi_{\\mathbb{S}_-^n}(G(\\bar{x})+\\rho^{-1}\\overline{Y})$,\n\t\t\\begin{equation*}\\label{eq:psdofpb}\n\t\t\\langle d, {\\cal A}_{\\rho}(\\overline{Y},W)d\\rangle\\geq\\underline{\\eta}\\|d\\|^2\\quad\\forall d\\in\\mathbb{X},\n\t\t\\end{equation*}\n\t\twhere ${\\cal A}_{\\rho}(\\overline{Y},W)$ is defined by \\eqref{eq:algrad2}.\n\t\\end{itemize}\n\\end{theorem}","start":45269,"end":46173}]},"proof":{"raw_tex":"\\begin{proposition}\\label{prop:onesidessosc}\n\tLet $\\bar{x}\\in\\mathbb{X}$ be a stationary point to NLSDP \\eqref{eq:NLSDP} and $\\overline{Y}\\in {\\cal M}(\\bar{x})$, where ${\\cal M}(\\bar{x})$ is given in \\eqref{eq:defcalm}. Let $A=G(\\bar{x})+\\overline{Y}$, which possesses the decomposition \\eqref{eq:eig-decomp}. Then, there exists $q\\in {\\rm quad}\\,\\delta_{\\mathbb{S}_+^n}(G(\\bar{x})\\mid\\overline{Y})$ such that for all $H\\in\\mathbb{S}^n$,\n\t\\begin{align}\n\t\tq(H)&=-\\frac{1}{2}\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},H\\big)+\\delta_{{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y})}(H)\\nonumber\\\\\n\t\t&=\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H}_{ij})^2+\\delta_{{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y})}(H),\\label{eq:qeqsg}\n\t\\end{align}\n\twhere $\\widetilde{H}=P^THP$.\n\\end{proposition}\n\n\\begin{proof}\n\tFor each $k$, choose\n\t\\[\n\tX^{k}={P} \\left[ \\begin{array}{ccccc} \\Lambda(A)_{\\alpha \\alpha} & 0\n\t\t& 0  \\\\   0 & (z^{k})_{\\beta} & 0 \\\\   0 & 0 &\n\t\t0 \\\\\n\t\\end{array} \\right] {P}^{T} \\quad {\\rm and} \\quad Y^{k}=\\overline{Y},\n\t\\]\n\twhere $z^k\\downarrow0$ (this notation means for each $k$, $z^k>0$ and $z^k\\rightarrow0$ as $k\\rightarrow\\infty$) with each $(z^k)_i$ non-increasing on $k$. Let $A^k=X^k+Y^k$ for each $k$. Since $\\mathbb{S}_+^n$ is ${\\cal C}^2-$cone reducible \\cite[Example 3.140]{BShapiro00}, we know from \\cite[Theorem 3.3]{mohammadi20}, \\eqref{eq:cricsdp} and \\eqref{eq:sigtsdp} that for any $H\\in\\mathbb{S}^n$,\n\t\\begin{align*}\n\t\t\\frac{1}{2}d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)=&\\sum_{i\\in\\alpha\\cup\\beta,j\\in\\gamma}\\frac{-\\lambda_j(A^k)}{\\lambda_i(A^k)}\\big(\\widetilde{H}_{ij}\\big)^2+\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)\\\\\n\t\t=&\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}\\big(\\widetilde{H}_{ij}\\big)^2+\\sum_{i\\in\\beta,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(X^k)}\\big(\\widetilde{H}_{ij}\\big)^2\\\\\n\t\t&+\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H),\n\t\\end{align*}\n\twhere ${\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)=\\{H\\in\\mathbb{S}^n\\mid\\widetilde{H}_{\\gamma\\gamma}=0\\}$. It is worth to note that $d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)$ is a generalized quadratic form by using  Definition \\ref{def:genguadf}, as $d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(0)=0$ and $\\partial d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)={\\cal R}(H)+{\\cal N}_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$\n\twith ${\\cal N}_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$ being a subspace and\n\t$${\\cal R}(H)(\\Delta H)=-4\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H})_{ij}\\cdot(P^T\\Delta HP)_{ij}-4\\sum_{i\\in\\beta,j\\in\\gamma}\\frac{\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H})_{ij}\\cdot(P^T\\Delta HP)_{ij}$$\n\tbeing linear on $H$.\n\tThis also implies that $\\delta_{\\mathbb{S}_+^n}$ is generalized twice differentiable at $X^k$ for $Y^k$.\n\t\nLet $f_1^k(H)=\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}\\big(\\widetilde{H}_{ij}\\big)^2$, $f_2^k(H)=\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$ and $f_3^k(H)=\\sum_{i\\in\\beta,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(X^k)}\\big(\\widetilde{H}_{ij}\\big)^2$. By the definition of continuous convergence mentioned in \\cite[page 250]{RoWe98}, we know that $f_1^k$ converges continuously to $f_1$ with\n$$f_1(H)=\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}\\big(\\widetilde{H}_{ij}\\big)^2.$$\n It follows from \\cite[Theorem 7.11]{RoWe98} that $f_1^k$  epi-converges to $f_1$. It can be checked easily that $f_1^k$ also  pointwise converges \\cite[page 239]{RoWe98} to $f_1$.\nSince ${\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)$ is a constant closed set, we know  that $f_2^k$ pointwise converges and epi-converges \\cite[Proposition 4.4]{RoWe98} to $f_2$ with $f_2(H)=\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$. It follows from \\cite[Theorem 7.46(a)]{RoWe98} that $f_1^k+f_2^k$ epi-convergences to $f_1+f_2$. Also, it can be checked directly that $f_1^k+f_2^k$ pointwise convergences to $f_1+f_2$. By the construction of $z^k$, the sequence of $\\{f_3^k\\}$ is nondecreasing ($f_3^k\\leq f_3^{k+1}$). We know from \\cite[Proposition 7.4]{RoWe98} that $f_3^k$ epi-converges to $\\sup_k\\{\\mbox{cl}f_3^k\\}$, where $\\mbox{cl}f_3^k$ is the closure of $f_3^k$. It is easy to see that $\\sup_k\\{\\mbox{cl}f_3^k\\}=\\sup_k \\{f_3^k\\}=\\delta_{\\cal V}$ with ${\\cal V}=\\{H\\in\\mathbb{S}^n\\mid\\widetilde{H}_{\\beta\\gamma}=0\\}$ and $f_3^k$ pointwise converges to $\\delta_{\\cal V}$. Since $f_1^k+f_2^k$ pointwise and epi-converges to $f_1+f_2$, by using \\cite[Theorem 7.46(a)]{RoWe98} again, we obtain that $f_1^k+f_2^k+f_3^k$ epi-converges to $f_1+f_2+f_3$.\n\nCombining the above discussion with \\eqref{eq:affcsdp}, the generalized quadratic form $\\frac{1}{2}d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)$ converges epigraphically to generalized quadratic form\n\t$$\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H}_{ij})^2+\\delta_{{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y})}(H). $$\n\tThus we have verified this proposition.\n\\end{proof}\n\n\\begin{proof}\n\t``$(i)\\Longrightarrow(ii)$\":  This direction can be obtained directly from Proposition \\ref{prop:onesidessosc} and Proposition \\ref{prop:thm5} as we only need to substitute \\eqref{eq:qeqsg} into \\eqref{svs-equ}.\n\t\n\t``$(ii)\\Longrightarrow(iii)$\": This proof sketch is similar to \\cite[Proposition 4]{SSZhang08} although they require the validity of nondegeneracy, which is superfluous here.\n\tIt follows from Definition \\ref{eq:defssosc} that there exists $\\eta_0>0$ such that\n\t$$\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)\\geq\\eta_0\\|d\\|^2\\quad\\forall\\; G'(\\bar{x})d\\in{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y}).$$\n\tBy using \\cite[Lemma 7]{SSZhang08}, there exist two positive numbers $\\rho_1$ and $\\underline{\\eta}\\in(0,\\eta_0/2]$ such that for any $\\rho'\\geq \\rho_1$,\n\t\\begin{align*}\n\t\t&\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)\\\\\n\t\t&+\\rho'\\|\\overline{P}^T_{\\gamma}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2+\\rho'\\|\\overline{P}^T_{\\beta}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2\\geq2\\underline{\\eta}\\|d\\|^2\\quad \\forall d\\in\\mathbb{X}.\n\t\\end{align*}\n\tSuppose a sufficient large $\\rho_0\\geq\\rho_1$. For any $\\rho\\geq\\rho_0$ and $d\\in{\\cal X}$, we have\n\t{\\small \\begin{align*}\n\t\t\t&-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)-2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\\\\n\t\t\t&=2\\sum\\limits_{i\\in\\alpha,\\,j\\in\\gamma}\\frac{-\\lambda_j}{\\lambda_i}(\\overline{P}^T(G'(\\bar{x}){d})\\overline{P})_{ij}^2-2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\\\\n\t\t\t&=2\\sum\\limits_{i\\in\\alpha,\\,j\\in\\gamma}\\frac{\\lambda_j^2}{\\lambda_i(\\rho\\lambda_i-\\lambda_j)}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\leq\\underline{\\eta}\\|d\\|^2,\n\t\\end{align*}}\n\twhere the last inequality can be obtained by sufficiently large $\\rho$.\n\tCombining the above two relations together, we have for any $\\rho\\geq\\rho_0$,  $\\rho'\\geq\\rho_0\\geq\\rho_1$,\n\t\\begin{align*}\n\t\t&\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle+2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\\\\n\t\t&+\\rho'\\|\\overline{P}^T_{\\gamma}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2+\\rho'\\|\\overline{P}^T_{\\beta}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2\\geq\\underline{\\eta}\\|d\\|^2.\n\t\\end{align*}\n\tThen it can be checked directly that for all $d\\in\\mathbb{X}$ and $\\rho\\geq\\rho_0$,  $\\langle d,{\\cal A}_{\\rho}(\\overline{Y},W)d\\rangle\\geq\\underline{\\eta}\\|d\\|^2 $.\n\t\n\t\n\t\n\t``$(iii)\\Longrightarrow(i)$\": It follows directly from \\cite[Theorem 3]{Roh22} as ${\\cal A}_{\\rho}(\\overline{Y},W)$ coincides with \\cite[(3.6)]{Roh22}.\n\\end{proof}","units":[{"raw_tex":"\\begin{proposition}\\label{prop:onesidessosc}\n\tLet $\\bar{x}\\in\\mathbb{X}$ be a stationary point to NLSDP \\eqref{eq:NLSDP} and $\\overline{Y}\\in {\\cal M}(\\bar{x})$, where ${\\cal M}(\\bar{x})$ is given in \\eqref{eq:defcalm}. Let $A=G(\\bar{x})+\\overline{Y}$, which possesses the decomposition \\eqref{eq:eig-decomp}. Then, there exists $q\\in {\\rm quad}\\,\\delta_{\\mathbb{S}_+^n}(G(\\bar{x})\\mid\\overline{Y})$ such that for all $H\\in\\mathbb{S}^n$,\n\t\\begin{align}\n\t\tq(H)&=-\\frac{1}{2}\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},H\\big)+\\delta_{{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y})}(H)\\nonumber\\\\\n\t\t&=\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H}_{ij})^2+\\delta_{{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y})}(H),\\label{eq:qeqsg}\n\t\\end{align}\n\twhere $\\widetilde{H}=P^THP$.\n\\end{proposition}","start":40074,"end":40921},{"raw_tex":"\\begin{proof}\n\tFor each $k$, choose\n\t\\[\n\tX^{k}={P} \\left[ \\begin{array}{ccccc} \\Lambda(A)_{\\alpha \\alpha} & 0\n\t\t& 0  \\\\   0 & (z^{k})_{\\beta} & 0 \\\\   0 & 0 &\n\t\t0 \\\\\n\t\\end{array} \\right] {P}^{T} \\quad {\\rm and} \\quad Y^{k}=\\overline{Y},\n\t\\]\n\twhere $z^k\\downarrow0$ (this notation means for each $k$, $z^k>0$ and $z^k\\rightarrow0$ as $k\\rightarrow\\infty$) with each $(z^k)_i$ non-increasing on $k$. Let $A^k=X^k+Y^k$ for each $k$. Since $\\mathbb{S}_+^n$ is ${\\cal C}^2-$cone reducible \\cite[Example 3.140]{BShapiro00}, we know from \\cite[Theorem 3.3]{mohammadi20}, \\eqref{eq:cricsdp} and \\eqref{eq:sigtsdp} that for any $H\\in\\mathbb{S}^n$,\n\t\\begin{align*}\n\t\t\\frac{1}{2}d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)=&\\sum_{i\\in\\alpha\\cup\\beta,j\\in\\gamma}\\frac{-\\lambda_j(A^k)}{\\lambda_i(A^k)}\\big(\\widetilde{H}_{ij}\\big)^2+\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)\\\\\n\t\t=&\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}\\big(\\widetilde{H}_{ij}\\big)^2+\\sum_{i\\in\\beta,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(X^k)}\\big(\\widetilde{H}_{ij}\\big)^2\\\\\n\t\t&+\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H),\n\t\\end{align*}\n\twhere ${\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)=\\{H\\in\\mathbb{S}^n\\mid\\widetilde{H}_{\\gamma\\gamma}=0\\}$. It is worth to note that $d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)$ is a generalized quadratic form by using  Definition \\ref{def:genguadf}, as $d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(0)=0$ and $\\partial d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)={\\cal R}(H)+{\\cal N}_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$\n\twith ${\\cal N}_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$ being a subspace and\n\t$${\\cal R}(H)(\\Delta H)=-4\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H})_{ij}\\cdot(P^T\\Delta HP)_{ij}-4\\sum_{i\\in\\beta,j\\in\\gamma}\\frac{\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H})_{ij}\\cdot(P^T\\Delta HP)_{ij}$$\n\tbeing linear on $H$.\n\tThis also implies that $\\delta_{\\mathbb{S}_+^n}$ is generalized twice differentiable at $X^k$ for $Y^k$.\n\t\nLet $f_1^k(H)=\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}\\big(\\widetilde{H}_{ij}\\big)^2$, $f_2^k(H)=\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$ and $f_3^k(H)=\\sum_{i\\in\\beta,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(X^k)}\\big(\\widetilde{H}_{ij}\\big)^2$. By the definition of continuous convergence mentioned in \\cite[page 250]{RoWe98}, we know that $f_1^k$ converges continuously to $f_1$ with\n$$f_1(H)=\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}\\big(\\widetilde{H}_{ij}\\big)^2.$$\n It follows from \\cite[Theorem 7.11]{RoWe98} that $f_1^k$  epi-converges to $f_1$. It can be checked easily that $f_1^k$ also  pointwise converges \\cite[page 239]{RoWe98} to $f_1$.\nSince ${\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)$ is a constant closed set, we know  that $f_2^k$ pointwise converges and epi-converges \\cite[Proposition 4.4]{RoWe98} to $f_2$ with $f_2(H)=\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$. It follows from \\cite[Theorem 7.46(a)]{RoWe98} that $f_1^k+f_2^k$ epi-convergences to $f_1+f_2$. Also, it can be checked directly that $f_1^k+f_2^k$ pointwise convergences to $f_1+f_2$. By the construction of $z^k$, the sequence of $\\{f_3^k\\}$ is nondecreasing ($f_3^k\\leq f_3^{k+1}$). We know from \\cite[Proposition 7.4]{RoWe98} that $f_3^k$ epi-converges to $\\sup_k\\{\\mbox{cl}f_3^k\\}$, where $\\mbox{cl}f_3^k$ is the closure of $f_3^k$. It is easy to see that $\\sup_k\\{\\mbox{cl}f_3^k\\}=\\sup_k \\{f_3^k\\}=\\delta_{\\cal V}$ with ${\\cal V}=\\{H\\in\\mathbb{S}^n\\mid\\widetilde{H}_{\\beta\\gamma}=0\\}$ and $f_3^k$ pointwise converges to $\\delta_{\\cal V}$. Since $f_1^k+f_2^k$ pointwise and epi-converges to $f_1+f_2$, by using \\cite[Theorem 7.46(a)]{RoWe98} again, we obtain that $f_1^k+f_2^k+f_3^k$ epi-converges to $f_1+f_2+f_3$.\n\nCombining the above discussion with \\eqref{eq:affcsdp}, the generalized quadratic form $\\frac{1}{2}d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)$ converges epigraphically to generalized quadratic form\n\t$$\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H}_{ij})^2+\\delta_{{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y})}(H). $$\n\tThus we have verified this proposition.\n\\end{proof}","start":40922,"end":45081},{"raw_tex":"\\begin{proof}\n\t``$(i)\\Longrightarrow(ii)$\":  This direction can be obtained directly from Proposition \\ref{prop:onesidessosc} and Proposition \\ref{prop:thm5} as we only need to substitute \\eqref{eq:qeqsg} into \\eqref{svs-equ}.\n\t\n\t``$(ii)\\Longrightarrow(iii)$\": This proof sketch is similar to \\cite[Proposition 4]{SSZhang08} although they require the validity of nondegeneracy, which is superfluous here.\n\tIt follows from Definition \\ref{eq:defssosc} that there exists $\\eta_0>0$ such that\n\t$$\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)\\geq\\eta_0\\|d\\|^2\\quad\\forall\\; G'(\\bar{x})d\\in{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y}).$$\n\tBy using \\cite[Lemma 7]{SSZhang08}, there exist two positive numbers $\\rho_1$ and $\\underline{\\eta}\\in(0,\\eta_0/2]$ such that for any $\\rho'\\geq \\rho_1$,\n\t\\begin{align*}\n\t\t&\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)\\\\\n\t\t&+\\rho'\\|\\overline{P}^T_{\\gamma}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2+\\rho'\\|\\overline{P}^T_{\\beta}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2\\geq2\\underline{\\eta}\\|d\\|^2\\quad \\forall d\\in\\mathbb{X}.\n\t\\end{align*}\n\tSuppose a sufficient large $\\rho_0\\geq\\rho_1$. For any $\\rho\\geq\\rho_0$ and $d\\in{\\cal X}$, we have\n\t{\\small \\begin{align*}\n\t\t\t&-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)-2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\\\\n\t\t\t&=2\\sum\\limits_{i\\in\\alpha,\\,j\\in\\gamma}\\frac{-\\lambda_j}{\\lambda_i}(\\overline{P}^T(G'(\\bar{x}){d})\\overline{P})_{ij}^2-2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\\\\n\t\t\t&=2\\sum\\limits_{i\\in\\alpha,\\,j\\in\\gamma}\\frac{\\lambda_j^2}{\\lambda_i(\\rho\\lambda_i-\\lambda_j)}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\leq\\underline{\\eta}\\|d\\|^2,\n\t\\end{align*}}\n\twhere the last inequality can be obtained by sufficiently large $\\rho$.\n\tCombining the above two relations together, we have for any $\\rho\\geq\\rho_0$,  $\\rho'\\geq\\rho_0\\geq\\rho_1$,\n\t\\begin{align*}\n\t\t&\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle+2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\\\\n\t\t&+\\rho'\\|\\overline{P}^T_{\\gamma}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2+\\rho'\\|\\overline{P}^T_{\\beta}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2\\geq\\underline{\\eta}\\|d\\|^2.\n\t\\end{align*}\n\tThen it can be checked directly that for all $d\\in\\mathbb{X}$ and $\\rho\\geq\\rho_0$,  $\\langle d,{\\cal A}_{\\rho}(\\overline{Y},W)d\\rangle\\geq\\underline{\\eta}\\|d\\|^2 $.\n\t\n\t\n\t\n\t``$(iii)\\Longrightarrow(i)$\": It follows directly from \\cite[Theorem 3]{Roh22} as ${\\cal A}_{\\rho}(\\overline{Y},W)$ coincides with \\cite[(3.6)]{Roh22}.\n\\end{proof}","start":46174,"end":49048}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:genp"],"declared_external":false,"source_only":false,"title":"一般复合优化模型","source_units":[{"raw_tex":" The definition of (strong) variational sufficient condition (Definition \\ref{def:svasc}) is officially given in \\cite{Roh22}  for the following general composite optimization problem\n \\begin{equation}\\label{eq:genp}\n \t\\min_{x\\in \\mathbb{X}}\\quad f(x)+\\theta(G(x)),\n \\end{equation}\n where $\\mathbb{X}$ and $\\mathbb{Y}$ are two given Euclidean spaces, $f:\\mathbb{X}\\to \\mathbb{R}$ and $G:\\mathbb{X}\\to \\mathbb{Y}$ are twice continuously differentiable, and $\\theta:\\mathbb{Y}\\to (-\\infty,\\infty]$ is a closed proper convex function. The strong variational sufficient condition is firstly introduced to deal with the local convexity of primal augmented Lagrangian function and the augmented tilt stability. It has gained more and more attention for its mathematical elegance and wide applications in the convergence analysis of multiplier methods. Several characterizations of this abstract property are also given in \\cite{Roh22}. For instance, it is equivalent to the positive definiteness of the Hessian bundle of the augmented Lagrangian function \\cite[Theorem 3]{Roh22} or the criterion \\cite[Theorem 5]{Roh22} involving quadratic bundle (Definition \\ref{def:quadratic bundle}).\n","start":6647,"end":7829}]},{"id":"S002","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"非多面体问题的研究动机","source_units":[{"raw_tex":" In terms of the connection with traditional optimality conditions, Rockafellar \\cite[Theorem 4]{Roh22} shows that the strong variational sufficiency is equivalent to the well-known strong second order sufficient condition (SOSC), which is expressed entirely via the program data, when the function $\\theta$ in \\eqref{eq:genp} is polyhedral convex (i.e., the epigraph ${\\rm epi}\\,\\theta$ of $\\theta$ is a polyhedral convex set). For non-polyhedral problems,  the equivalence is still valid if $\\theta$ in \\eqref{eq:genp} is the indicator function of the second order cone with $G(\\bar{x})\\neq 0$, where $\\bar{x}$ is a local optimal solution (see \\cite[Example 3]{Roh22} for details). However, an explicit and verifiable characterization of strong variational sufficient condition for general non-polyhedral problems remains unknown. In this paper, without loss of generality, we mainly focus on the characterization of strong variational sufficiency for the following nonlinear semidefinite programming (NLSDP):\n","start":7831,"end":8843}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:NLSDPc"],"declared_external":false,"source_only":false,"title":"半正定规划与矩阵空间","source_units":[{"raw_tex":"\\begin{equation}\\label{eq:NLSDPc}\n\t\\begin{array}{cl}\n\t\t\\displaystyle{\\min_{x\\in \\mathbb{X}}} & f(x) \\\\ [3pt]\n\t\t{\\rm s.t.} &{G(x)\\in \\mathbb{S}^n_+},\n\t\\end{array}\n\\end{equation}\nwhere  $\\mathbb{S}^n$ is the linear space of all $n\\times n$ real symmetric matrices equipped with the usual Frobenius inner product and its induced norm, $\\mathbb{S}_+^n$ ($\\mathbb{S}_-^n$) is the closed convex cone of all $n\\times n$ positive (negative) semidefinite matrices in $\\mathbb{S}^n$. One of our contributions lies in uncovering the equivalence between strong variational sufficient condition and strong SOSC (see Definition \\ref{def:sigt} for details) for  NLSDP problems without requiring any other constraint qualifications.\n","start":8843,"end":9560}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["eq:L-function"],"declared_external":false,"source_only":false,"title":"普通拉格朗日函数","source_units":[{"raw_tex":"For NLSDP \\eqref{eq:NLSDPc}, the Lagrangian function is defined by\n\\begin{equation}\\label{eq:L-function}\n\tL(x,Y):=f(x) +\\langle Y,G(x)\\rangle, \\quad (x,Y)\\in\\mathbb{X}\\times\\mathbb{S}^n.\n\\end{equation}\nFor any $Y\\in\\mathbb{S}^n$, denote the first-order and second-order derivatives of $L(\\cdot,Y)$ at $x\\in\\mathbb{X}$ by $L_x'(x,Y)$ and $L_{xx}''(x,Y)$, respectively. Given $\\rho > 0$, the augmented Lagrangian function of \\eqref{eq:NLSDPc} takes the following form (cf. \\cite[Section 11.K]{RoWe98} and \\cite{ShapS04})\n","start":11177,"end":11696}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":["eq:deflag"],"declared_external":false,"source_only":false,"title":"增广拉格朗日函数","source_units":[{"raw_tex":"\\begin{equation}\\label{eq:deflag}\n\t\\mathcal{L}_{\\rho}(x,Y):=f(x)+\\frac{\\rho}{2}{\\rm dist}^2(G(x)+\\frac{Y}{\\rho},\\mathbb{S}_+^n)-\\frac{\\|Y\\|^2}{2\\rho},\n\\end{equation}\nwhere ${\\rm dist}(z,\\mathbb{S}_+^n)$ is the distance from point $z$ to $\\mathbb{S}_+^n$.\n","start":11696,"end":11951}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"切锥法锥与临界锥","source_units":[{"raw_tex":"For a given Euclidean space $\\mathbb{X}$,\nlet $C$ be any subset in $\\mathbb{X}$, ${\\rm aff}\\,{C}$ is the affine hull of $C$ \\cite[page 6]{Rockafellar1970}. The Bouligand tangent/contingent cone of $C$ at $x$ is a closed cone defined by\n\\[\n\\mathcal{T}_{C}(x):= \\left\\{d\\in\\mathbb{X}\\mid \\mbox{$\\exists\\, t^k\\downarrow 0$ and  $d^k\\to d$ with  $x+t^kd^k\\in C$ for all $k$} \\right\\}.\n\\]\nWhen $C$ is convex, the normal cone in the sense of  convex analysis \\cite{Rockafellar1970} is defined as\n$$\\mathcal{N}_{C}(x)=\\left \\{d \\in \\mathbb{X}\\mid\\langle d,x'-x\\rangle\\leq 0 \\ \\forall\\,x'\\in C \\right\\}.$$\nFor any $x\\in C$, the critical cone  associated with ${ y}\\in\\mathcal{N}_C(x)$ of $C$ is defined in \\cite[Page 98]{DoRo14}, i.e., ${\\cal C}_{C}({x},{y})=\\mathcal{T}_{C}({x})\\cap({y})^{\\perp}$.\n","start":16485,"end":17276}]},{"id":"S007","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"集合值映射的外极限","source_units":[{"raw_tex":"\nFor a set-valued mapping $\\mathcal{F}:\\mathbb{X}\\rightrightarrows\\mathbb{X}$, its $\\limsup$ means\n$$\\limsup\\limits_{{x}\\rightarrow\\bar{x}}\\mathcal{F}({x}):=\\left\\{d\\in\\mathbb{X}\\mid \\exists\\;x_k\\rightarrow\\bar{x},\\; {y}_k\\rightarrow d\\;{\\rm such\\;that}\\;{y}_k\\in\\mathcal{F}({x}_k)\\;\\forall\\,k\\right\\}.$$\nThe following definitions of proximal and (general/Mordukhovich limiting) subdifferentials of functions are adopted from \\cite[Definition 8.45 and 8.3]{RoWe98}.\n","start":17276,"end":17742}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"近端与极限次微分","source_units":[{"raw_tex":"\\begin{definition}\n\tConsider a function $f:\\mathbb{X}\\rightarrow(-\\infty, +\\infty]$ and a point $\\bar{x}$ with $f(\\bar{x})$ finite. The proximal subdifferential of $f$ at $\\bar{x}$ is defined as\n\t$$\\partial^{\\pi}f(\\bar{x}):=\\{{d}\\mid \\exists\\; l>0, r>0\\;{\\rm such\\; that}\\;f(x)\\geq f(\\bar{x})+\\langle{d},{x}-\\bar{x}\\rangle-l\\|{x}-\\bar{x}\\|^2\\;\\forall\\,{x}\\in{\\cal B}_{r}(\\bar{x})\\},$$\n\twhere ${\\cal B}_{r}(\\bar{x})$ is the ball centered at $\\bar{x}$ with radius $r$.\n\tThe (general/Mordukhovich limiting) subdifferential of $f$ at $\\bar{x}$ is $\\partial f(\\bar{x}):=\\limsup\\limits_{{x}\\stackrel{f}{\\rightarrow}\\bar{x}}\\partial^{\\pi}f({x})$, where ${x}\\stackrel{f}{\\rightarrow}\\bar{x}$ signifies ${x}\\rightarrow\\bar{x}$ with $f({x})\\rightarrow f(\\bar{x})$.\n\\end{definition}\n","start":17742,"end":18514}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"次微分定义的兼容性","source_units":[{"raw_tex":"Note that in an Euclidean space, the (general) subdifferential also can be constructed via the regular/Fr\\'{e}chet subdifferentials  of functions $\\widehat{\\partial}f$ \\cite[Definition 8.3]{RoWe98}. It is well-known that these two definitions are equivalent (see e.g., \\cite[page 345]{RoWe98} or \\cite[Theorem 1.89]{Mordukhovich06} for details). In particular, when $f$ is convex and finite at the corresponding point, the proximal and (general) subdifferentials coincide with the subdifferential in the sense of convex analysis \\cite{Rockafellar1970}.\n","start":18514,"end":19067}]},{"id":"S010","semantic_type":"notation","kind":"notation","labels":["eq:eig-decomp"],"declared_external":false,"source_only":false,"title":"谱分解与符号分块","source_units":[{"raw_tex":"Let  $A \\in\\mathbb{S}^n$ be given. We use $\\lambda_{1}(A)\\ge \\lambda_2(A) \\ge \\ldots \\ge \\lambda_{n}(A)$ to denote the eigenvalues of $A$ (all real and counting multiplicity) arranging in  nonincreasing order and use $\\lambda(A)$ to denote the vector of the ordered eigenvalues of $A$. Let $\\Lambda(A):= {\\rm Diag}(\\lambda(A))$. Also, we use $v_1(A)> \\dots> v_d(A)$ to denote the different eigenvalues and\n$\\zeta_l:=\\{i\\mid \\lambda_i(A)=v_l(A)\\}$.\nConsider the eigenvalue decomposition of $A$, i.e., $A={P} \\Lambda(A){P}^{T}$, where ${P}\\in{\\cal O}^{n}(A)$ is a corresponding\northogonal matrix of the orthonormal eigenvectors.\nBy considering the index sets of positive, zero, and negative eigenvalues of $A$, we are able to write $A$ in the following form\n{\\footnotesize\n\\begin{equation}\\label{eq:eig-decomp}\n\tA= \\left[\\begin{array}{ccc}\n\t\t{P}_{\\alpha} & {P}_{\\beta} & {P}_{\\gamma}\n\t\\end{array}\\right] \\left[\\begin{array}{ccc}\n\t\t\\Lambda(A)_{\\alpha\\alpha} & 0 & 0 \\\\ [3pt]\n\t\t0 & 0 & 0\\\\ [3pt]\n\t\t0 & 0 & \\Lambda(A)_{\\gamma\\gamma}\n\t\\end{array}\\right]\\left[\\begin{array}{c}\n\t\t{P}_{\\alpha}^T \\\\ [3pt]\n\t\t{P}_{\\beta}^T \\\\\n\t\t{P}_{\\gamma}^T\n\t\\end{array}\\right],\n\\end{equation}}\nwhere $\\alpha:=\\{i: \\lambda_i(A)>0\\}$, $\\beta:=\\{i: \\lambda_i(A)=0\\}$ and $\\gamma:=\\{i: \\lambda_i(A)<0\\}$.\n","start":19129,"end":20405}]},{"id":"S011","semantic_type":"external_fact","kind":"external_fact","labels":["eq:cricsdp"],"declared_external":true,"declaration_text":["原始前提列表：CSun08, equations (11)–(12)；本节点保留文中重述的公式。"],"source_only":false,"title":"半正定锥临界锥公式","source_units":[{"raw_tex":"The critical cone of $\\mathbb{S}_+^n$ at ${Y}\\in{\\cal N}_{\\mathbb{S}_+^n}({X})$ with $A=X+Y$ is given by (cf. \\cite[equations (11) and (12)]{CSun08})\n\\begin{equation}\\label{eq:cricsdp}\n\t{\\cal C}_{\\mathbb{S}_+^n}({X},{Y})=\\{H\\in\\mathbb{S}^n\\mid P^T_{\\beta}HP_{\\beta}\\in\\mathbb{S}^{|\\beta|}_+,\\;P^T_{\\beta}HP_{\\gamma}=0,\\;P^T_{\\gamma}HP_{\\gamma}=0\\}\n\\end{equation}\n","start":20408,"end":20771}]},{"id":"S012","semantic_type":"external_fact","kind":"external_fact","labels":["eq:affcsdp"],"declared_external":true,"declaration_text":["原始前提列表：CSun08, equations (11)–(12)；引用见紧邻的 L263。"],"source_only":false,"title":"临界锥仿射包公式","source_units":[{"raw_tex":"and the affine hull of ${\\cal C}_{\\mathbb{S}_+^n}({X},{Y})$ is\n\\begin{equation}\\label{eq:affcsdp}\n\t{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}({X},{Y})=\\{H\\in\\mathbb{S}^n\\mid P^T_{\\beta}HP_{\\gamma}=0,\\;P^T_{\\gamma}HP_{\\gamma}=0\\}.\n\\end{equation}\n","start":20771,"end":21010}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["eq:NLSDP"],"declared_external":false,"source_only":false,"title":"指标函数形式的半正定规划","source_units":[{"raw_tex":"The  NLSDP \\eqref{eq:NLSDPc} can be written in the following form\n\\begin{eqnarray}\n\t%\\mbox{(SDP)}~~~~\n\t\\min\\quad && f(x)+\\delta_{\\mathbb{S}^n_+}(G(x)), \\label{eq:NLSDP}\n\\end{eqnarray}\nwhere\n$\\delta_{\\mathbb{S}^n_+} : \\mathbb{S}^n\\rightarrow (-\\infty,\\infty]$ is the indicator function of positive semidefinite cone $\\mathbb{S}^n_+$.\n","start":21010,"end":21343}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":["eq:KKT-NLSDP-p1"],"declared_external":false,"source_only":false,"title":"KKT 系统与驻点","source_units":[{"raw_tex":"Denote ${S}_{KKT}(a,b)$ the solution set of the KKT optimality condition for problem \\eqref{eq:NLSDP}, i.e.,\n\\begin{equation}\\label{eq:KKT-NLSDP-p1}\n\t{S}_{KKT}(a,b)=\\left\\{(x,Y)\\in \\mathbb{X}\\times\\mathbb{S}^n \\Bigg | \\left.\n\t\\begin{array}{l}\n\t\tL_x'(x,Y)-a=0,\\\\ [3pt]\n\t\t\\mathbb{S}_+^n\\ni (G(x)-b)\\perp Y \\in \\mathbb{S}_-^n.\n\t\\end{array}\n\t\\right.\\right \\}.\n\\end{equation}\nFor any KKT pair $(\\bar{x},\\overline{Y})$ that satisfies the KKT condition with $(a,b) = (0,0)$, we call $\\bar{x}$ a stationary point.\n","start":21343,"end":21849}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":["eq:defcalm"],"declared_external":false,"source_only":false,"title":"参考驻点的乘子集合","source_units":[{"raw_tex":"Suppose $\\bar{x}$ is a stationary point. Define ${\\cal M}(\\bar{x})$ as the set of all multipliers $Y\\in\\mathbb{S}^n$ satisfying the KKT condition \\eqref{eq:KKT-NLSDP-p1}, i.e.,\n\\begin{equation}\\label{eq:defcalm}\n\t{\\cal M}(\\bar{x})=\\{Y\\in\\mathbb{S}^n \\mid (\\bar{x},Y)\\in {S}_{KKT}(0,0)\\}.\n\\end{equation}\n","start":21849,"end":22152}]},{"id":"S016","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"固定乘子的强二阶条件","source_units":[{"raw_tex":"\n\nThe definition of strong SOSC for NLSDP, which demands the supremum of \\eqref{eq:defssosc} over $Y\\in{\\cal M}(\\bar{x})$ holds, is originally given by \\cite{Sun06} as an analogue for Robinson \\cite{Robin80} for the nonlinear programming (NLP). However, the strong SOSC mentioned here is slightly different in only requiring the validity of \\eqref{eq:defssosc} at $\\overline{Y}$. It is worth to note when ${\\cal M}(\\bar{x})$ is a singleton, both of them are the same.\n","start":22152,"end":22620}]},{"id":"S017","semantic_type":"definition","kind":"definition","labels":["def:sigt","eq:defssosc"],"declared_external":false,"source_only":false,"title":"强二阶充分条件与曲率项","source_units":[{"raw_tex":"\\begin{definition}\\label{def:sigt}\\cite[Definition 3.2]{Sun06}\n\tLet $\\bar{x}$ be a stationary point of NLSDP \\eqref{eq:NLSDP} and $\\overline{Y}\\in{\\cal M}(\\bar{x})$. We say the strong second order sufficient condition (SOSC) holds at $(\\bar{x},\\overline{Y})$ if\n\t\\begin{equation}\\label{eq:defssosc}\n\t\t\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)>0\\quad\\forall\\; 0\\neq G'(\\bar{x})d\\in{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y}).\n\t\\end{equation}\n\twhere $\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)=2\\langle\\overline{Y},(G'(\\bar{x}){d})G(\\bar{x})^{\\dagger}(G'(\\bar{x}){d})\\rangle$ is the $\\sigma$-term and $G(\\bar{x})^{\\dagger}$ is the generalized inverse matrix of $G(\\bar{x})$.\n\\end{definition}\n","start":22620,"end":23416}]},{"id":"S018","semantic_type":"external_fact","kind":"external_fact","labels":["eq:sigtsdp"],"declared_external":true,"declaration_text":["原始前提列表：CSun08, page 386。"],"source_only":false,"title":"曲率项的谱展开","source_units":[{"raw_tex":"\nSuppose $A=G(\\bar{x})+\\overline{Y}$ possesses the decomposition \\eqref{eq:eig-decomp}. From \\cite[page 386]{CSun08}, we know the $\\sigma$-term takes the explicit form of\n\\begin{equation}\\label{eq:sigtsdp}\n\t\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},H\\big)=2\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H}_{ij})^2,\n\\end{equation}\nwhere $\\widetilde{H}=P^THP$.\n","start":23416,"end":23802}]},{"id":"S019","semantic_type":"external_fact","kind":"external_fact","labels":["eq:algrad"],"declared_external":true,"declaration_text":["原始前提列表中的 SSZhang08 广义导数背景；文中 L309 明示引用。"],"source_only":false,"title":"增广拉格朗日梯度","source_units":[{"raw_tex":"Next, we introduce some essential notation for our main result. The (second order) generalized differentiability  of the augmented Lagrangian function ${\\cal L}_{\\rho}$ defined by \\eqref{eq:deflag} for NLSDP \\eqref{eq:NLSDP} has been explicitly established in \\cite{SSZhang08}.  In fact, it is well-known that ${\\cal L}_{\\rho}$ is continuously differentiable with\n\\begin{equation}\\label{eq:algrad}\n\t({\\cal L}_{\\rho})_x'(x,Y)=f'(x)+\\rho G'(x)^*\\Pi_{\\mathbb{S}_-^n}(G(x)+\\rho^{-1}Y),\n\\end{equation}\nwhere $\\Pi_{\\mathbb{S}_-^n}(\\cdot)$ is the metric projection over $\\mathbb{S}_{-}^n$ and $G'(x)^*$ denotes the adjoint of the corresponding linear mapping. Moreover, since $({\\cal L}_{\\rho})_x'$ is  Lipschitz continuous, by using Rademacher’s theorem, the B(ouligand)-subdifferential of $({\\cal L}_{\\rho})_x'$ at $(x,Y)$ is given by\n","start":23803,"end":24633}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":["eq:partb"],"declared_external":false,"source_only":false,"title":"梯度映射的布次微分","source_units":[{"raw_tex":"where $\\Pi_{\\mathbb{S}_-^n}(\\cdot)$ is the metric projection over $\\mathbb{S}_{-}^n$ and $G'(x)^*$ denotes the adjoint of the corresponding linear mapping. Moreover, since $({\\cal L}_{\\rho})_x'$ is  Lipschitz continuous, by using Rademacher’s theorem, the B(ouligand)-subdifferential of $({\\cal L}_{\\rho})_x'$ at $(x,Y)$ is given by\n\\begin{equation}\\label{eq:partb}\n\t\\partial_B\\big(({\\cal L}_{\\rho})'_x\\big)(x,Y):=\\{\\lim_{k\\rightarrow\\infty}(({\\cal L}_{\\rho})_{x}')'(x^k,Y^k)\\mid (x^k,Y^k)\\in{\\cal U},\\ (x^k,Y^k)\\rightarrow(x,Y)\\},\n\\end{equation}\nwhere ${\\cal U}$ is the set of Fr\\'{e}chet-differentiable points of $({\\cal L}_{\\rho})_x'$.\n","start":24300,"end":24939}]},{"id":"S021","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表：SSZhang08, equation (18), Lemmas 2–3；Roh22 的 Hessian-bundle 刻画及 equation (3.6)。所选上下文另注明 Roh22 (3.1)。"],"source_only":false,"title":"广义海森束的识别","source_units":[{"raw_tex":"It follows from \\cite[(18)]{SSZhang08} (using \\cite[Lemma 2 and 3]{SSZhang08}) that\n\\begin{align*}\n\t&\\pi_x\\partial_B\\big(({\\cal L}_{\\rho})_x'\\big)(x,Y)(\\Delta x)\\\\\n\t&=L_{xx}''(x,\\rho\\Pi_{\\mathbb{S}_-^n}(G(x)+\\rho^{-1}Y))(\\Delta x)+\\rho G'({x})^*\\partial_B\\Pi_{\\mathbb{S}_-^n}(G({x})+\\rho^{-1}Y) G'({x})(\\Delta x),\n\\end{align*}\nwhere $\\pi_x\\partial_B\\big(({\\cal L}_{\\rho})_x'\\big)(x,Y)$ is the projection of $\\partial_B\\big(({\\cal L}_{\\rho})_x'\\big)(x,Y)$ onto the space $\\mathbb{X}$ and  $\\partial_B\\Pi_{\\mathbb{S}_-^n}(G({x})+\\rho^{-1}Y)$ is the B-subdifferential of $\\Pi_{\\mathbb{S}_-^n}(\\cdot)$ at $G({x})+\\rho^{-1}Y$. Actually, the set $\\pi_x\\partial_B\\big(({\\cal L}_{\\rho})_x'\\big)(x,Y)$ is recently defined as the $x$ part of the Hessian bundle \\cite[(3.1)]{Roh22} for the augmented Lagrangian function ${\\cal L}_{\\rho}$ at $(x,Y)$ (see \\cite[(3.6)]{Roh22} for detail).\n","start":24939,"end":25815}]},{"id":"S022","semantic_type":"definition","kind":"definition","labels":["eq:algrad2"],"declared_external":false,"source_only":false,"title":"参考点的广义海森算子","source_units":[{"raw_tex":"\n\n\nLet $\\bar{x}$ be the stationary point of \\eqref{eq:NLSDP}. For each $\\overline{Y}\\in{\\cal M}(\\bar{x})$ and $W\\in\\partial_B\\Pi_{\\mathbb{S}_-^n}(G(\\bar{x})+\\rho^{-1}\\overline{Y})$, define the following mapping\n\\begin{equation}\\label{eq:algrad2}\n\t{\\cal A}_{\\rho}(\\overline{Y},W):=L_{xx}''(\\bar{x},\\overline{Y})+\\rho G'(\\bar{x})^*WG'(\\bar{x}).\n\\end{equation}\n","start":25815,"end":26173}]},{"id":"S023","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表：SSZhang08, Lemma 5。"],"source_only":false,"title":"锥投影布导数的分块形式","source_units":[{"raw_tex":"Let $\\overline{Z}=G(\\bar{x})+\\overline{Y}$ possesses decomposition \\eqref{eq:eig-decomp} with $\\overline{P}\\in{\\cal O}^{n}(\\overline{Z})$. It follows from \\cite[Lemma 5]{SSZhang08} that $W\\in\\partial_B\\Pi_{\\mathbb{S}_-^n}(\\overline{Z})$ if and only if there exists $W_0\\in\\partial_B\\Pi_{\\mathbb{S}_-^{\\lvert\\beta\\rvert}}(0)$ such that for all $H\\in\\mathbb{S}^n$\n{\\footnotesize\n\\begin{equation*}\n\tW(H)=\\overline{P}\\left[\\begin{array}{ccc}\n\t\t0 & 0 & \\Sigma_{\\alpha\\gamma}\\circ\\overline{P}_{\\alpha}^TH\\overline{P}_{\\gamma}\\\\\n\t\t0 & W_0(\\overline{P}_{\\beta}^TH\\overline{P}_{\\beta}) & \\overline{P}_{\\beta}^TH\\overline{P}_{\\gamma} \\\\\n\t\t\\Sigma_{\\gamma\\alpha}\\circ\\overline{P}_{\\gamma}^TH\\overline{P}_{\\alpha} & \\overline{P}_{\\gamma}^TH\\overline{P}_{\\beta} & \\overline{P}_{\\gamma}^TH\\overline{P}_{\\gamma}\n\t\\end{array}\\right]\\overline{P}^T,\n\\end{equation*}}\nwhere\n\\begin{equation*}\n\\begin{cases}\t\t\\Sigma_{ij}=1-\\frac{\\max\\{\\lambda_i,0\\}+\\max\\{\\lambda_j,0\\}}{\\lvert\\lambda_i\\rvert+\\lvert\\lambda_j\\rvert}, & (i,j)\\notin\\beta\\times\\beta\\\\\n\\Sigma_{ij}\t\\in[0,1] & (i,j)\\in\\beta\\times\\beta\n\t\\end{cases}\n\\end{equation*}\nwith $\\lambda_i:=\\lambda_i(\\overline{Z})$ for short. Let ${\\cal Q}^n$ denote the set of $n$-dimensional orthogonal matrix.\n","start":26173,"end":27399}]},{"id":"S024","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表：SSZhang08, Lemma 5；承接 L331 的引用。"],"source_only":false,"title":"零特征值块的表示","source_units":[{"raw_tex":"Also, $W_0\\in\\partial_B\\Pi_{\\mathbb{S}_-^{\\lvert\\beta\\rvert}}(0)$ if and only if there exist $Q\\in{\\cal Q}^{\\lvert\\beta\\rvert}$ and $\\Omega\\in\\mathbb{S}^{\\lvert\\beta\\rvert}$ with entries $\\Omega_{ij}\\in[0,1]$ such that for all $H\\in\\mathbb{S}^{\\lvert\\beta\\rvert}$,\n$$W_0(H)=Q(\\Omega\\circ(Q^THQ))Q^T.$$\n","start":27399,"end":27701}]},{"id":"S025","semantic_type":"external_fact","kind":"external_fact","labels":["eq:expcalA"],"declared_external":true,"declaration_text":["原始前提列表：SSZhang08, Lemma 9。"],"source_only":false,"title":"广义海森二次型展开","source_units":[{"raw_tex":"By \\cite[Lemma 9]{SSZhang08}, we have\n{\\small \\begin{align}\n\t\\langle d,{\\cal A}_{\\rho}(\\overline{Y},W)d\\rangle=&\\langle d,L_{xx}''(\\bar{x},\\overline{Y})d\\rangle+\\rho\\sum_{i,j\\in\\gamma}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2+2\\rho\\sum_{i\\in\\beta,j\\in\\gamma}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\label{eq:expcalA}\\\\\n\t&+2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2+\\rho\\sum_{i,j\\in\\beta}(\\Omega_{\\rho})_{ij}({P}^T(G'(\\bar{x})d){P})_{ij}^2,\\nonumber\n\\end{align}}\nwhere $\\Omega_{\\rho}\\in\\mathbb{S}^{\\lvert\\beta\\rvert}$ with entries $(\\Omega_{\\rho})_{ij}\\in[0,1]$, $P=[\\overline{P}_{\\alpha}\\;\\overline{P}_{\\beta}Q\\;\\overline{P}_{\\gamma}]$ with $Q\\in{\\cal Q}^{\\lvert\\beta\\rvert}$.\n","start":27701,"end":28477}]},{"id":"S026","semantic_type":"definition","kind":"definition","labels":["eq:pgpert"],"declared_external":false,"source_only":false,"title":"扰动模型与一阶条件","source_units":[{"raw_tex":"The definition of (strong) variational sufficient condition is given in \\cite{Roh22}  for general composite optimization problem \\eqref{eq:genp}.\nWe can recast \\eqref{eq:genp} in the form\n\\begin{equation}\\label{eq:pgpert}\n\t\\min\\;\\phi(x,u)\\;\\mbox{subject to}\\;u=0,\\quad\\mbox{where}\\;\\phi(x,u)=f(x)+\\theta(G(x)+u).\n\\end{equation}\nThe first order local optimality condition for \\eqref{eq:genp}  of $\\bar{x}$ is the existence of $\\overline{Y}$ such that\n$$L_x'(\\bar{x},\\overline{Y})=0\\quad \\mbox{with}\\quad\\overline{Y}\\in\\partial\\, \\theta(G(\\bar{x})). $$\n","start":28523,"end":29074}]},{"id":"S027","semantic_type":"definition","kind":"definition","labels":["eq:phir"],"declared_external":false,"source_only":false,"title":"变分强凸性的局部代表","source_units":[{"raw_tex":"Define\n\\begin{equation}\\label{eq:phir}\n\\phi_r:=\\phi(x,u)+\\frac{r}{2}|u|^2.\n\\end{equation}\nThe variational (strong) convexity, which is firstly proposed in \\cite{rock247}, refers to the existence of open convex neighborhoods ${\\cal W}$ of $(\\bar{x},0)$ and ${\\cal Z}$ of $(0,\\overline{Y})$ such that there exists a proper closed (strongly) convex function $\\psi\\leq\\phi_r2$ on ${\\cal W}$ such that\n\\begin{equation*}\n\t({\\cal W}\\times{\\cal Z})\\cap{\\rm gph}\\,\\partial\\psi=({\\cal W}\\times{\\cal Z})\\cap{\\rm gph}\\,\\partial\\phi_r\n\\end{equation*}\nand for $(x,u;v,y)$ belonging to this common set, $\\psi(x,u)=\\phi_r(x,u)$.\n","start":29074,"end":29687}]},{"id":"S028","semantic_type":"definition","kind":"definition","labels":["def:svasc"],"declared_external":false,"source_only":false,"title":"强变分充分性","source_units":[{"raw_tex":"\\begin{definition}\\label{def:svasc}\\cite{Roh22}\n\tThe (strong) variational sufficient condition for local optimality in \\eqref{eq:pgpert} holds with respect to $\\bar{x}$ and $\\overline{Y}$ satisfying the first order condition if there exists $r>0$ such that $\\displaystyle\\phi_r(x,u)$ is variationally (strongly) convex with respect to the pair $\\big((\\bar{x},0),(0,\\overline{Y})\\big)$ in ${\\rm gph}\\,\\partial\\phi_r$.\n\\end{definition}\n\n\n","start":29687,"end":30123}]},{"id":"S029","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"二阶次导数与上图可微性","source_units":[{"raw_tex":"To characterize the above property, the tools of the second subderivative and generalized quadratic form are needed.\n\\begin{definition}\\cite[Definition 13.6]{RoWe98}\n\tLet $\\bar{x}$ be a point where the function $f:\\mathbb{X}\\rightarrow[-\\infty,+\\infty]$ is finite. $f$ is twice epi-differentiable at $\\bar{x}$ for $v$ if the functions\n\t$$\\Delta^2_{t}f(\\bar{x}\\mid v)(u)=\\frac{f(\\bar{x}+tu)-f(\\bar{x})-t\\langle v,u\\rangle}{\\frac{1}{2}t^2}$$\n\tepi-converge to $d^2f(\\bar{x}\\mid v)$ as $t\\downarrow0$, where $d^2f(\\bar{x}\\mid v)$ is the second subderivative of $f$ at $\\bar{x}$ for $v$ defined as\n\t$$d^2f(\\bar{x}\\mid v)(w)=\\liminf_{t\\downarrow0,u\\rightarrow w}\\Delta^2_{t}f(\\bar{x}\\mid v)(u)$$\n\\end{definition}\n","start":30123,"end":30830}]},{"id":"S030","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表：mohammadi20, Theorem 3.6。"],"source_only":false,"title":"半正定锥指标的二阶可微性","source_units":[{"raw_tex":"We know from \\cite[Theorem 3.6]{mohammadi20} that $\\delta_{\\mathbb{S}_+^n}$ is twice epi-differentiable at $X$ for $Y$ with $Y\\in{\\cal N}_{\\mathbb{S}_+^n}(X)$.\n\n","start":30830,"end":30991}]},{"id":"S031","semantic_type":"definition","kind":"definition","labels":["def:genguadf"],"declared_external":false,"source_only":false,"title":"广义二次型与广义二阶可微","source_units":[{"raw_tex":"\\begin{definition}\\label{def:genguadf} \\cite{Roh22}\n\tA generalized linear mapping ${\\cal R}$ from $\\mathbb{X}$ to $\\mathbb{Y}$, is a set-valued mapping for which ${\\rm gph}\\,{\\cal R}$ is a subspace of $\\mathbb{X}\\times\\mathbb{Y}$. This means that ${\\rm dom}{\\cal R}$ is a subspace $\\mathbb{Z}$ of $\\mathbb{X}$, ${\\cal R}(0)$ is a subspace $\\mathbb{Z}'$ of $\\mathbb{Y}$, and there is an ordinary linear mapping ${\\cal R}_0:\\mathbb{Z}\\rightarrow\\mathbb{Y}$ such that ${\\cal R}(x)={\\cal R}_0(x)+\\mathbb{Z}'$ for $x\\in\\mathbb{Z}$.\n\tWe call a function $q: \\mathbb{X}\\rightarrow(-\\infty,+\\infty]$ is a generalized quadratic form on $\\mathbb{X}$ if $q(0)=0$ and the subgradient mapping $\\partial q:\\mathbb{X}\\rightrightarrows\\mathbb{X}$ is generalized linear. A function $g$ on $\\mathbb{X}$ will be called generalized twice differentiable at $x$ for a subgradient $y$ if it is twice epi-differentiable at $x$ for $y$ with the second-order subderivative $d^2g(x\\mid y)$ being a generalized quadratic form.\n\\end{definition}\n\n\n","start":30991,"end":32008}]},{"id":"S032","semantic_type":"definition","kind":"definition","labels":["def:quadratic bundle"],"declared_external":false,"source_only":false,"title":"二次束","source_units":[{"raw_tex":"The following definition of the quadratic bundle is taken from \\cite{Roh22}, which is essential in characterizing the strong variational sufficiency.\n\\begin{definition}\\label{def:quadratic bundle}\n\tFor general optimization problem \\eqref{eq:genp}, suppose $(\\bar{x},\\overline{Y})$ is a KKT pair. The quadratic bundle of $\\theta$ is defined as\n\t\\begin{equation*}\n\t\t{\\rm quad}\\,\\theta(G(\\bar{x})\\mid\\overline{Y})=\\left\\{\n\t\t\\begin{array}{l}\n\t\t\t\\mbox{the collection of generalized quadratic forms}\\; q\\; \\mbox{for which}\\\\\n\t\t\t\\exists(X^k,Y^k)\\rightarrow(G(\\bar{x}),\\overline{Y})\\;\\mbox{with}\\;\\theta\\;\\mbox{generalized twice differentiable}\\\\\n\t\t\t\\mbox{at}\\;X^k\\;\\mbox{for}\\;Y^k\\;\\mbox{and such that the generalized quadratic}\\\\\n\t\t\t\\mbox{forms}\\;q_k=\\frac{1}{2}d^2\\theta(X^k\\mid Y^k)\\;\\mbox{converge epigraphically to}\\;q.\n\t\t\\end{array}\n\t\t\\right.\n\t\\end{equation*}\n\\end{definition}\n","start":32008,"end":32884}]},{"id":"S033","semantic_type":"proposition","kind":"proposition","labels":["prop:thm5","svs-equ"],"declared_external":true,"declaration_text":["原始前提列表明确许可：Roh22, Theorem 5，文中重述为 B0046 / prop:thm5。"],"source_only":false,"title":"二次束刻画强变分充分性","source_units":[{"raw_tex":"As mentioned in \\cite{Roh22}, the variational sufficient condition guarantees the local optimality for \\eqref{eq:pgpert}. The following result is taken from \\cite[Theorem 5]{Roh22}, which is useful for the subsequent analysis.\n\\begin{proposition}\\label{prop:thm5}\n\tFor general optimization problem \\eqref{eq:genp},\n\tstrong variational sufficient condition for local optimality with respect to $(\\bar{x},\\overline{Y})$ is equivalent to that  every $q\\in{\\rm quad}\\,\\theta(G(\\bar{x})\\mid\\overline{Y})$ has\n\t\\begin{equation}\\label{svs-equ}\n\t\t\\frac{1}{2}\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle+q(G'(\\bar{x}){d})>0\\quad{\\rm when}\\;d\\neq0,\n\t\\end{equation}\n\twhere ${\\rm quad}\\,\\theta(G(\\bar{x})\\mid\\overline{Y})$ is the quadratic bundle of $\\theta$.\n\\end{proposition}\n","start":32884,"end":33654}]},{"id":"S034","semantic_type":"proposition","kind":"proposition","labels":["eq:qeqsg","prop:onesidessosc"],"declared_external":false,"source_only":false,"title":"构造仿射临界锥上的二次束元素","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:onesidessosc}\n\tLet $\\bar{x}\\in\\mathbb{X}$ be a stationary point to NLSDP \\eqref{eq:NLSDP} and $\\overline{Y}\\in {\\cal M}(\\bar{x})$, where ${\\cal M}(\\bar{x})$ is given in \\eqref{eq:defcalm}. Let $A=G(\\bar{x})+\\overline{Y}$, which possesses the decomposition \\eqref{eq:eig-decomp}. Then, there exists $q\\in {\\rm quad}\\,\\delta_{\\mathbb{S}_+^n}(G(\\bar{x})\\mid\\overline{Y})$ such that for all $H\\in\\mathbb{S}^n$,\n\t\\begin{align}\n\t\tq(H)&=-\\frac{1}{2}\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},H\\big)+\\delta_{{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y})}(H)\\nonumber\\\\\n\t\t&=\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H}_{ij})^2+\\delta_{{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y})}(H),\\label{eq:qeqsg}\n\t\\end{align}\n\twhere $\\widetilde{H}=P^THP$.\n\\end{proposition}","start":40074,"end":40921}]},{"id":"S035","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"二次束构造的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\tFor each $k$, choose\n\t\\[\n\tX^{k}={P} \\left[ \\begin{array}{ccccc} \\Lambda(A)_{\\alpha \\alpha} & 0\n\t\t& 0  \\\\   0 & (z^{k})_{\\beta} & 0 \\\\   0 & 0 &\n\t\t0 \\\\\n\t\\end{array} \\right] {P}^{T} \\quad {\\rm and} \\quad Y^{k}=\\overline{Y},\n\t\\]\n\twhere $z^k\\downarrow0$ (this notation means for each $k$, $z^k>0$ and $z^k\\rightarrow0$ as $k\\rightarrow\\infty$) with each $(z^k)_i$ non-increasing on $k$. Let $A^k=X^k+Y^k$ for each $k$. Since $\\mathbb{S}_+^n$ is ${\\cal C}^2-$cone reducible \\cite[Example 3.140]{BShapiro00}, we know from \\cite[Theorem 3.3]{mohammadi20}, \\eqref{eq:cricsdp} and \\eqref{eq:sigtsdp} that for any $H\\in\\mathbb{S}^n$,\n\t\\begin{align*}\n\t\t\\frac{1}{2}d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)=&\\sum_{i\\in\\alpha\\cup\\beta,j\\in\\gamma}\\frac{-\\lambda_j(A^k)}{\\lambda_i(A^k)}\\big(\\widetilde{H}_{ij}\\big)^2+\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)\\\\\n\t\t=&\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}\\big(\\widetilde{H}_{ij}\\big)^2+\\sum_{i\\in\\beta,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(X^k)}\\big(\\widetilde{H}_{ij}\\big)^2\\\\\n\t\t&+\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H),\n\t\\end{align*}\n\twhere ${\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)=\\{H\\in\\mathbb{S}^n\\mid\\widetilde{H}_{\\gamma\\gamma}=0\\}$. It is worth to note that $d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)$ is a generalized quadratic form by using  Definition \\ref{def:genguadf}, as $d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(0)=0$ and $\\partial d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)={\\cal R}(H)+{\\cal N}_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$\n\twith ${\\cal N}_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$ being a subspace and\n\t$${\\cal R}(H)(\\Delta H)=-4\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H})_{ij}\\cdot(P^T\\Delta HP)_{ij}-4\\sum_{i\\in\\beta,j\\in\\gamma}\\frac{\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H})_{ij}\\cdot(P^T\\Delta HP)_{ij}$$\n\tbeing linear on $H$.\n\tThis also implies that $\\delta_{\\mathbb{S}_+^n}$ is generalized twice differentiable at $X^k$ for $Y^k$.\n\t\nLet $f_1^k(H)=\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}\\big(\\widetilde{H}_{ij}\\big)^2$, $f_2^k(H)=\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$ and $f_3^k(H)=\\sum_{i\\in\\beta,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(X^k)}\\big(\\widetilde{H}_{ij}\\big)^2$. By the definition of continuous convergence mentioned in \\cite[page 250]{RoWe98}, we know that $f_1^k$ converges continuously to $f_1$ with\n$$f_1(H)=\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}\\big(\\widetilde{H}_{ij}\\big)^2.$$\n It follows from \\cite[Theorem 7.11]{RoWe98} that $f_1^k$  epi-converges to $f_1$. It can be checked easily that $f_1^k$ also  pointwise converges \\cite[page 239]{RoWe98} to $f_1$.\nSince ${\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)$ is a constant closed set, we know  that $f_2^k$ pointwise converges and epi-converges \\cite[Proposition 4.4]{RoWe98} to $f_2$ with $f_2(H)=\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$. It follows from \\cite[Theorem 7.46(a)]{RoWe98} that $f_1^k+f_2^k$ epi-convergences to $f_1+f_2$. Also, it can be checked directly that $f_1^k+f_2^k$ pointwise convergences to $f_1+f_2$. By the construction of $z^k$, the sequence of $\\{f_3^k\\}$ is nondecreasing ($f_3^k\\leq f_3^{k+1}$). We know from \\cite[Proposition 7.4]{RoWe98} that $f_3^k$ epi-converges to $\\sup_k\\{\\mbox{cl}f_3^k\\}$, where $\\mbox{cl}f_3^k$ is the closure of $f_3^k$. It is easy to see that $\\sup_k\\{\\mbox{cl}f_3^k\\}=\\sup_k \\{f_3^k\\}=\\delta_{\\cal V}$ with ${\\cal V}=\\{H\\in\\mathbb{S}^n\\mid\\widetilde{H}_{\\beta\\gamma}=0\\}$ and $f_3^k$ pointwise converges to $\\delta_{\\cal V}$. Since $f_1^k+f_2^k$ pointwise and epi-converges to $f_1+f_2$, by using \\cite[Theorem 7.46(a)]{RoWe98} again, we obtain that $f_1^k+f_2^k+f_3^k$ epi-converges to $f_1+f_2+f_3$.\n\nCombining the above discussion with \\eqref{eq:affcsdp}, the generalized quadratic form $\\frac{1}{2}d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)$ converges epigraphically to generalized quadratic form\n\t$$\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H}_{ij})^2+\\delta_{{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y})}(H). $$\n\tThus we have verified this proposition.\n\\end{proof}","start":40922,"end":45081}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"抬升零谱块的逼近构造","source_units":[{"raw_tex":"\\begin{proof}\n\tFor each $k$, choose\n\t\\[\n\tX^{k}={P} \\left[ \\begin{array}{ccccc} \\Lambda(A)_{\\alpha \\alpha} & 0\n\t\t& 0  \\\\   0 & (z^{k})_{\\beta} & 0 \\\\   0 & 0 &\n\t\t0 \\\\\n\t\\end{array} \\right] {P}^{T} \\quad {\\rm and} \\quad Y^{k}=\\overline{Y},\n\t\\]\n\twhere $z^k\\downarrow0$ (this notation means for each $k$, $z^k>0$ and $z^k\\rightarrow0$ as $k\\rightarrow\\infty$) with each $(z^k)_i$ non-increasing on $k$. Let $A^k=X^k+Y^k$ for each $k$. Since $\\mathbb{S}_+^n$ is ${\\cal C}^2-$cone reducible \\cite[Example 3.140]{BShapiro00}, we know from \\cite[Theorem 3.3]{mohammadi20}, \\eqref{eq:cricsdp} and \\eqref{eq:sigtsdp} that for any $H\\in\\mathbb{S}^n$,\n","start":40922,"end":41561}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"计算逼近点的二阶次导数","source_units":[{"raw_tex":"\twhere $z^k\\downarrow0$ (this notation means for each $k$, $z^k>0$ and $z^k\\rightarrow0$ as $k\\rightarrow\\infty$) with each $(z^k)_i$ non-increasing on $k$. Let $A^k=X^k+Y^k$ for each $k$. Since $\\mathbb{S}_+^n$ is ${\\cal C}^2-$cone reducible \\cite[Example 3.140]{BShapiro00}, we know from \\cite[Theorem 3.3]{mohammadi20}, \\eqref{eq:cricsdp} and \\eqref{eq:sigtsdp} that for any $H\\in\\mathbb{S}^n$,\n\t\\begin{align*}\n\t\t\\frac{1}{2}d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)=&\\sum_{i\\in\\alpha\\cup\\beta,j\\in\\gamma}\\frac{-\\lambda_j(A^k)}{\\lambda_i(A^k)}\\big(\\widetilde{H}_{ij}\\big)^2+\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)\\\\\n\t\t=&\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}\\big(\\widetilde{H}_{ij}\\big)^2+\\sum_{i\\in\\beta,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(X^k)}\\big(\\widetilde{H}_{ij}\\big)^2\\\\\n\t\t&+\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H),\n\t\\end{align*}\n\twhere ${\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)=\\{H\\in\\mathbb{S}^n\\mid\\widetilde{H}_{\\gamma\\gamma}=0\\}$. It is worth to note that $d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)$ is a generalized quadratic form by using  Definition \\ref{def:genguadf}, as $d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(0)=0$ and $\\partial d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)={\\cal R}(H)+{\\cal N}_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$\n","start":41163,"end":42460}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"验证邻近点的广义二阶可微性","source_units":[{"raw_tex":"\twhere ${\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)=\\{H\\in\\mathbb{S}^n\\mid\\widetilde{H}_{\\gamma\\gamma}=0\\}$. It is worth to note that $d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)$ is a generalized quadratic form by using  Definition \\ref{def:genguadf}, as $d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(0)=0$ and $\\partial d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)={\\cal R}(H)+{\\cal N}_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$\n\twith ${\\cal N}_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$ being a subspace and\n\t$${\\cal R}(H)(\\Delta H)=-4\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H})_{ij}\\cdot(P^T\\Delta HP)_{ij}-4\\sum_{i\\in\\beta,j\\in\\gamma}\\frac{\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H})_{ij}\\cdot(P^T\\Delta HP)_{ij}$$\n\tbeing linear on $H$.\n\tThis also implies that $\\delta_{\\mathbb{S}_+^n}$ is generalized twice differentiable at $X^k$ for $Y^k$.\n","start":42048,"end":42907}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"固定二次项的收敛","source_units":[{"raw_tex":"\t\nLet $f_1^k(H)=\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}\\big(\\widetilde{H}_{ij}\\big)^2$, $f_2^k(H)=\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$ and $f_3^k(H)=\\sum_{i\\in\\beta,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(X^k)}\\big(\\widetilde{H}_{ij}\\big)^2$. By the definition of continuous convergence mentioned in \\cite[page 250]{RoWe98}, we know that $f_1^k$ converges continuously to $f_1$ with\n$$f_1(H)=\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}\\big(\\widetilde{H}_{ij}\\big)^2.$$\n It follows from \\cite[Theorem 7.11]{RoWe98} that $f_1^k$  epi-converges to $f_1$. It can be checked easily that $f_1^k$ also  pointwise converges \\cite[page 239]{RoWe98} to $f_1$.\n","start":42907,"end":43612}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"固定子空间指标及前两项之和","source_units":[{"raw_tex":"Since ${\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)$ is a constant closed set, we know  that $f_2^k$ pointwise converges and epi-converges \\cite[Proposition 4.4]{RoWe98} to $f_2$ with $f_2(H)=\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$. It follows from \\cite[Theorem 7.46(a)]{RoWe98} that $f_1^k+f_2^k$ epi-convergences to $f_1+f_2$. Also, it can be checked directly that $f_1^k+f_2^k$ pointwise convergences to $f_1+f_2$. By the construction of $z^k$, the sequence of $\\{f_3^k\\}$ is nondecreasing ($f_3^k\\leq f_3^{k+1}$). We know from \\cite[Proposition 7.4]{RoWe98} that $f_3^k$ epi-converges to $\\sup_k\\{\\mbox{cl}f_3^k\\}$, where $\\mbox{cl}f_3^k$ is the closure of $f_3^k$. It is easy to see that $\\sup_k\\{\\mbox{cl}f_3^k\\}=\\sup_k \\{f_3^k\\}=\\delta_{\\cal V}$ with ${\\cal V}=\\{H\\in\\mathbb{S}^n\\mid\\widetilde{H}_{\\beta\\gamma}=0\\}$ and $f_3^k$ pointwise converges to $\\delta_{\\cal V}$. Since $f_1^k+f_2^k$ pointwise and epi-converges to $f_1+f_2$, by using \\cite[Theorem 7.46(a)]{RoWe98} again, we obtain that $f_1^k+f_2^k+f_3^k$ epi-converges to $f_1+f_2+f_3$.\n","start":43612,"end":44664}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"发散罚项产生块约束","source_units":[{"raw_tex":"Since ${\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)$ is a constant closed set, we know  that $f_2^k$ pointwise converges and epi-converges \\cite[Proposition 4.4]{RoWe98} to $f_2$ with $f_2(H)=\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$. It follows from \\cite[Theorem 7.46(a)]{RoWe98} that $f_1^k+f_2^k$ epi-convergences to $f_1+f_2$. Also, it can be checked directly that $f_1^k+f_2^k$ pointwise convergences to $f_1+f_2$. By the construction of $z^k$, the sequence of $\\{f_3^k\\}$ is nondecreasing ($f_3^k\\leq f_3^{k+1}$). We know from \\cite[Proposition 7.4]{RoWe98} that $f_3^k$ epi-converges to $\\sup_k\\{\\mbox{cl}f_3^k\\}$, where $\\mbox{cl}f_3^k$ is the closure of $f_3^k$. It is easy to see that $\\sup_k\\{\\mbox{cl}f_3^k\\}=\\sup_k \\{f_3^k\\}=\\delta_{\\cal V}$ with ${\\cal V}=\\{H\\in\\mathbb{S}^n\\mid\\widetilde{H}_{\\beta\\gamma}=0\\}$ and $f_3^k$ pointwise converges to $\\delta_{\\cal V}$. Since $f_1^k+f_2^k$ pointwise and epi-converges to $f_1+f_2$, by using \\cite[Theorem 7.46(a)]{RoWe98} again, we obtain that $f_1^k+f_2^k+f_3^k$ epi-converges to $f_1+f_2+f_3$.\n","start":43612,"end":44664}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"三项之和的上图极限","source_units":[{"raw_tex":"Since ${\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)$ is a constant closed set, we know  that $f_2^k$ pointwise converges and epi-converges \\cite[Proposition 4.4]{RoWe98} to $f_2$ with $f_2(H)=\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$. It follows from \\cite[Theorem 7.46(a)]{RoWe98} that $f_1^k+f_2^k$ epi-convergences to $f_1+f_2$. Also, it can be checked directly that $f_1^k+f_2^k$ pointwise convergences to $f_1+f_2$. By the construction of $z^k$, the sequence of $\\{f_3^k\\}$ is nondecreasing ($f_3^k\\leq f_3^{k+1}$). We know from \\cite[Proposition 7.4]{RoWe98} that $f_3^k$ epi-converges to $\\sup_k\\{\\mbox{cl}f_3^k\\}$, where $\\mbox{cl}f_3^k$ is the closure of $f_3^k$. It is easy to see that $\\sup_k\\{\\mbox{cl}f_3^k\\}=\\sup_k \\{f_3^k\\}=\\delta_{\\cal V}$ with ${\\cal V}=\\{H\\in\\mathbb{S}^n\\mid\\widetilde{H}_{\\beta\\gamma}=0\\}$ and $f_3^k$ pointwise converges to $\\delta_{\\cal V}$. Since $f_1^k+f_2^k$ pointwise and epi-converges to $f_1+f_2$, by using \\cite[Theorem 7.46(a)]{RoWe98} again, we obtain that $f_1^k+f_2^k+f_3^k$ epi-converges to $f_1+f_2+f_3$.\n","start":43612,"end":44664}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"识别目标二次束元素","source_units":[{"raw_tex":"\nCombining the above discussion with \\eqref{eq:affcsdp}, the generalized quadratic form $\\frac{1}{2}d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)$ converges epigraphically to generalized quadratic form\n\t$$\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H}_{ij})^2+\\delta_{{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y})}(H). $$\n\tThus we have verified this proposition.\n\\end{proof}","start":44664,"end":45081}]},{"id":"S044","semantic_type":"theorem","kind":"theorem","labels":["eq:psdofpb","thm:varscequissosc"],"declared_external":false,"source_only":false,"title":"强变分充分性的三重等价","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm:varscequissosc}\n\tLet $\\bar{x}\\in\\mathbb{X}$ be a stationary point to the NLSDP \\eqref{eq:NLSDP} and $\\overline{Y}\\in {\\cal M}(\\bar{x})$. Then the following three conditions are equivalent\n\t\\begin{itemize}\n\t\t\\item[(i)] the strong variational sufficient condition with respect to $(\\bar{x},\\overline{Y})$ holds;\n\t\t\\item[(ii)] the strong second order sufficient condition (SOSC) \\eqref{eq:defssosc} holds at $(\\bar{x},\\overline{Y})$;\n\t\t\\item[(iii)] there exist $\\rho_0>0$ and $\\underline{\\eta}>0$ such that for any $\\rho\\geq \\rho_0$ and any $W\\in\\partial_B\\Pi_{\\mathbb{S}_-^n}(G(\\bar{x})+\\rho^{-1}\\overline{Y})$,\n\t\t\\begin{equation*}\\label{eq:psdofpb}\n\t\t\\langle d, {\\cal A}_{\\rho}(\\overline{Y},W)d\\rangle\\geq\\underline{\\eta}\\|d\\|^2\\quad\\forall d\\in\\mathbb{X},\n\t\t\\end{equation*}\n\t\twhere ${\\cal A}_{\\rho}(\\overline{Y},W)$ is defined by \\eqref{eq:algrad2}.\n\t\\end{itemize}\n\\end{theorem}","start":45269,"end":46173}]},{"id":"S045","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"三重等价的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\t``$(i)\\Longrightarrow(ii)$\":  This direction can be obtained directly from Proposition \\ref{prop:onesidessosc} and Proposition \\ref{prop:thm5} as we only need to substitute \\eqref{eq:qeqsg} into \\eqref{svs-equ}.\n\t\n\t``$(ii)\\Longrightarrow(iii)$\": This proof sketch is similar to \\cite[Proposition 4]{SSZhang08} although they require the validity of nondegeneracy, which is superfluous here.\n\tIt follows from Definition \\ref{eq:defssosc} that there exists $\\eta_0>0$ such that\n\t$$\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)\\geq\\eta_0\\|d\\|^2\\quad\\forall\\; G'(\\bar{x})d\\in{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y}).$$\n\tBy using \\cite[Lemma 7]{SSZhang08}, there exist two positive numbers $\\rho_1$ and $\\underline{\\eta}\\in(0,\\eta_0/2]$ such that for any $\\rho'\\geq \\rho_1$,\n\t\\begin{align*}\n\t\t&\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)\\\\\n\t\t&+\\rho'\\|\\overline{P}^T_{\\gamma}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2+\\rho'\\|\\overline{P}^T_{\\beta}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2\\geq2\\underline{\\eta}\\|d\\|^2\\quad \\forall d\\in\\mathbb{X}.\n\t\\end{align*}\n\tSuppose a sufficient large $\\rho_0\\geq\\rho_1$. For any $\\rho\\geq\\rho_0$ and $d\\in{\\cal X}$, we have\n\t{\\small \\begin{align*}\n\t\t\t&-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)-2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\\\\n\t\t\t&=2\\sum\\limits_{i\\in\\alpha,\\,j\\in\\gamma}\\frac{-\\lambda_j}{\\lambda_i}(\\overline{P}^T(G'(\\bar{x}){d})\\overline{P})_{ij}^2-2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\\\\n\t\t\t&=2\\sum\\limits_{i\\in\\alpha,\\,j\\in\\gamma}\\frac{\\lambda_j^2}{\\lambda_i(\\rho\\lambda_i-\\lambda_j)}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\leq\\underline{\\eta}\\|d\\|^2,\n\t\\end{align*}}\n\twhere the last inequality can be obtained by sufficiently large $\\rho$.\n\tCombining the above two relations together, we have for any $\\rho\\geq\\rho_0$,  $\\rho'\\geq\\rho_0\\geq\\rho_1$,\n\t\\begin{align*}\n\t\t&\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle+2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\\\\n\t\t&+\\rho'\\|\\overline{P}^T_{\\gamma}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2+\\rho'\\|\\overline{P}^T_{\\beta}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2\\geq\\underline{\\eta}\\|d\\|^2.\n\t\\end{align*}\n\tThen it can be checked directly that for all $d\\in\\mathbb{X}$ and $\\rho\\geq\\rho_0$,  $\\langle d,{\\cal A}_{\\rho}(\\overline{Y},W)d\\rangle\\geq\\underline{\\eta}\\|d\\|^2 $.\n\t\n\t\n\t\n\t``$(iii)\\Longrightarrow(i)$\": It follows directly from \\cite[Theorem 3]{Roh22} as ${\\cal A}_{\\rho}(\\overline{Y},W)$ coincides with \\cite[(3.6)]{Roh22}.\n\\end{proof}","start":46174,"end":49048}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"二次束代入推出强二阶条件","source_units":[{"raw_tex":"\\begin{proof}\n\t``$(i)\\Longrightarrow(ii)$\":  This direction can be obtained directly from Proposition \\ref{prop:onesidessosc} and Proposition \\ref{prop:thm5} as we only need to substitute \\eqref{eq:qeqsg} into \\eqref{svs-equ}.\n","start":46174,"end":46401}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"声称获得统一曲率下界","source_units":[{"raw_tex":"\t\n\t``$(ii)\\Longrightarrow(iii)$\": This proof sketch is similar to \\cite[Proposition 4]{SSZhang08} although they require the validity of nondegeneracy, which is superfluous here.\n\tIt follows from Definition \\ref{eq:defssosc} that there exists $\\eta_0>0$ such that\n\t$$\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)\\geq\\eta_0\\|d\\|^2\\quad\\forall\\; G'(\\bar{x})d\\in{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y}).$$\n","start":46401,"end":46889}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"块罚项扩展为全空间下界","source_units":[{"raw_tex":"\tBy using \\cite[Lemma 7]{SSZhang08}, there exist two positive numbers $\\rho_1$ and $\\underline{\\eta}\\in(0,\\eta_0/2]$ such that for any $\\rho'\\geq \\rho_1$,\n\t\\begin{align*}\n\t\t&\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)\\\\\n\t\t&+\\rho'\\|\\overline{P}^T_{\\gamma}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2+\\rho'\\|\\overline{P}^T_{\\beta}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2\\geq2\\underline{\\eta}\\|d\\|^2\\quad \\forall d\\in\\mathbb{X}.\n\t\\end{align*}\n","start":46889,"end":47390}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"控制谱系数替换误差","source_units":[{"raw_tex":"\tSuppose a sufficient large $\\rho_0\\geq\\rho_1$. For any $\\rho\\geq\\rho_0$ and $d\\in{\\cal X}$, we have\n\t{\\small \\begin{align*}\n\t\t\t&-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)-2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\\\\n\t\t\t&=2\\sum\\limits_{i\\in\\alpha,\\,j\\in\\gamma}\\frac{-\\lambda_j}{\\lambda_i}(\\overline{P}^T(G'(\\bar{x}){d})\\overline{P})_{ij}^2-2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\\\\n\t\t\t&=2\\sum\\limits_{i\\in\\alpha,\\,j\\in\\gamma}\\frac{\\lambda_j^2}{\\lambda_i(\\rho\\lambda_i-\\lambda_j)}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\leq\\underline{\\eta}\\|d\\|^2,\n\t\\end{align*}}\n\twhere the last inequality can be obtained by sufficiently large $\\rho$.\n","start":47390,"end":48222}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"合并罚项下界与谱误差","source_units":[{"raw_tex":"\tCombining the above two relations together, we have for any $\\rho\\geq\\rho_0$,  $\\rho'\\geq\\rho_0\\geq\\rho_1$,\n\t\\begin{align*}\n\t\t&\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle+2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\\\\n\t\t&+\\rho'\\|\\overline{P}^T_{\\gamma}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2+\\rho'\\|\\overline{P}^T_{\\beta}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2\\geq\\underline{\\eta}\\|d\\|^2.\n\t\\end{align*}\n","start":48222,"end":48711}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"识别广义海森的一致正定性","source_units":[{"raw_tex":"\tThen it can be checked directly that for all $d\\in\\mathbb{X}$ and $\\rho\\geq\\rho_0$,  $\\langle d,{\\cal A}_{\\rho}(\\overline{Y},W)d\\rangle\\geq\\underline{\\eta}\\|d\\|^2 $.\n\t\n\t\n\t\n","start":48711,"end":48884}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"海森束刻画返回强变分充分性","source_units":[{"raw_tex":"\t``$(iii)\\Longrightarrow(i)$\": It follows directly from \\cite[Theorem 3]{Roh22} as ${\\cal A}_{\\rho}(\\overline{Y},W)$ coincides with \\cite[(3.6)]{Roh22}.\n\\end{proof}","start":48884,"end":49048}]},{"id":"S053","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表：BShapiro00, Example 3.140；使用位置 L508。"],"source_only":false,"title":"半正定锥的二阶锥可约性","source_units":[]},{"id":"S054","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表：mohammadi20, Theorem 3.3；使用位置 L508–L513。"],"source_only":false,"title":"可约锥指标的二阶次导数公式","source_units":[]},{"id":"S055","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表：RoWe98, Theorem 7.11；文中同时引用 pages 250、239 的收敛定义。"],"source_only":false,"title":"连续收敛蕴含上图收敛","source_units":[]},{"id":"S056","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表：RoWe98, Proposition 4.4；使用位置 L523。"],"source_only":false,"title":"闭集指标的上图收敛","source_units":[]},{"id":"S057","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表：RoWe98, Proposition 7.4；使用位置 L523。"],"source_only":false,"title":"单调函数列的上图极限","source_units":[]},{"id":"S058","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表：RoWe98, Theorem 7.46(a)；两次使用均见 L523。"],"source_only":false,"title":"函数和的上图收敛规则","source_units":[]},{"id":"S059","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表：SSZhang08, Lemma 7。"],"source_only":false,"title":"二次罚项的强制正定化","source_units":[]},{"id":"S060","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提列表：Roh22, Theorem 3 and equation (3.6)；使用位置 L582。"],"source_only":false,"title":"海森束刻画强变分充分性","source_units":[]},{"id":"S061","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；对应原始前提列表中的有限维紧性背景。"],"source_only":false,"title":"单位球面上的统一正性背景","source_units":[]},{"id":"S062","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；对应原始前提列表中的标准有限维分析。"],"source_only":false,"title":"有限谱和的统一二次估计背景","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S044","edges":[{"source_id":"S001","target_id":"S013","kind":"supports"},{"source_id":"S003","target_id":"S013","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S006","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S006","target_id":"S012","kind":"supports"},{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S004","target_id":"S014","kind":"supports"},{"source_id":"S006","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S017","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S004","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S017","kind":"supports"},{"source_id":"S010","target_id":"S018","kind":"supports"},{"source_id":"S005","target_id":"S019","kind":"supports"},{"source_id":"S003","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S004","target_id":"S022","kind":"supports"},{"source_id":"S014","target_id":"S022","kind":"supports"},{"source_id":"S015","target_id":"S022","kind":"supports"},{"source_id":"S010","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S022","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S001","target_id":"S026","kind":"supports"},{"source_id":"S004","target_id":"S026","kind":"supports"},{"source_id":"S008","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S008","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S006","target_id":"S030","kind":"supports"},{"source_id":"S013","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S008","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S029","target_id":"S032","kind":"supports"},{"source_id":"S026","target_id":"S032","kind":"supports"},{"source_id":"S001","target_id":"S033","kind":"supports"},{"source_id":"S004","target_id":"S033","kind":"supports"},{"source_id":"S028","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S014","target_id":"S034","kind":"supports"},{"source_id":"S015","target_id":"S034","kind":"supports"},{"source_id":"S010","target_id":"S034","kind":"supports"},{"source_id":"S018","target_id":"S034","kind":"supports"},{"source_id":"S012","target_id":"S034","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S013","target_id":"S044","kind":"supports"},{"source_id":"S014","target_id":"S044","kind":"supports"},{"source_id":"S015","target_id":"S044","kind":"supports"},{"source_id":"S017","target_id":"S044","kind":"supports"},{"source_id":"S028","target_id":"S044","kind":"supports"},{"source_id":"S022","target_id":"S044","kind":"supports"},{"source_id":"S010","target_id":"S036","kind":"supports"},{"source_id":"S014","target_id":"S036","kind":"supports"},{"source_id":"S015","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S053","target_id":"S037","kind":"supports"},{"source_id":"S054","target_id":"S037","kind":"supports"},{"source_id":"S011","target_id":"S037","kind":"supports"},{"source_id":"S018","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S031","target_id":"S038","kind":"supports"},{"source_id":"S030","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S055","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S056","target_id":"S040","kind":"supports"},{"source_id":"S058","target_id":"S040","kind":"supports"},{"source_id":"S036","target_id":"S041","kind":"supports"},{"source_id":"S037","target_id":"S041","kind":"supports"},{"source_id":"S057","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S058","target_id":"S042","kind":"supports"},{"source_id":"S038","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S012","target_id":"S043","kind":"supports"},{"source_id":"S018","target_id":"S043","kind":"supports"},{"source_id":"S032","target_id":"S043","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S035","kind":"supports"},{"source_id":"S039","target_id":"S035","kind":"supports"},{"source_id":"S040","target_id":"S035","kind":"supports"},{"source_id":"S041","target_id":"S035","kind":"supports"},{"source_id":"S042","target_id":"S035","kind":"supports"},{"source_id":"S043","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S046","kind":"supports"},{"source_id":"S033","target_id":"S046","kind":"supports"},{"source_id":"S017","target_id":"S046","kind":"supports"},{"source_id":"S017","target_id":"S047","kind":"supports"},{"source_id":"S012","target_id":"S047","kind":"supports"},{"source_id":"S061","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S012","target_id":"S048","kind":"supports"},{"source_id":"S059","target_id":"S048","kind":"supports"},{"source_id":"S018","target_id":"S049","kind":"supports"},{"source_id":"S010","target_id":"S049","kind":"supports"},{"source_id":"S062","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S025","target_id":"S051","kind":"supports"},{"source_id":"S024","target_id":"S051","kind":"supports"},{"source_id":"S060","target_id":"S052","kind":"supports"},{"source_id":"S021","target_id":"S052","kind":"supports"},{"source_id":"S022","target_id":"S052","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S048","target_id":"S045","kind":"supports"},{"source_id":"S049","target_id":"S045","kind":"supports"},{"source_id":"S050","target_id":"S045","kind":"supports"},{"source_id":"S051","target_id":"S045","kind":"supports"},{"source_id":"S052","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S044","edges":[{"source_id":"S001","target_id":"S013","kind":"supports"},{"source_id":"S003","target_id":"S013","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S006","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S006","target_id":"S012","kind":"supports"},{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S004","target_id":"S014","kind":"supports"},{"source_id":"S006","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S017","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S004","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S017","kind":"supports"},{"source_id":"S010","target_id":"S018","kind":"supports"},{"source_id":"S005","target_id":"S019","kind":"supports"},{"source_id":"S003","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S004","target_id":"S022","kind":"supports"},{"source_id":"S014","target_id":"S022","kind":"supports"},{"source_id":"S015","target_id":"S022","kind":"supports"},{"source_id":"S010","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S022","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S001","target_id":"S026","kind":"supports"},{"source_id":"S004","target_id":"S026","kind":"supports"},{"source_id":"S008","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S008","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S006","target_id":"S030","kind":"supports"},{"source_id":"S013","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S008","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S029","target_id":"S032","kind":"supports"},{"source_id":"S026","target_id":"S032","kind":"supports"},{"source_id":"S001","target_id":"S033","kind":"supports"},{"source_id":"S004","target_id":"S033","kind":"supports"},{"source_id":"S028","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S014","target_id":"S034","kind":"supports"},{"source_id":"S015","target_id":"S034","kind":"supports"},{"source_id":"S010","target_id":"S034","kind":"supports"},{"source_id":"S018","target_id":"S034","kind":"supports"},{"source_id":"S012","target_id":"S034","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S013","target_id":"S044","kind":"supports"},{"source_id":"S014","target_id":"S044","kind":"supports"},{"source_id":"S015","target_id":"S044","kind":"supports"},{"source_id":"S017","target_id":"S044","kind":"supports"},{"source_id":"S028","target_id":"S044","kind":"supports"},{"source_id":"S022","target_id":"S044","kind":"supports"},{"source_id":"S053","target_id":"S034","kind":"supports"},{"source_id":"S054","target_id":"S034","kind":"supports"},{"source_id":"S011","target_id":"S034","kind":"supports"},{"source_id":"S031","target_id":"S034","kind":"supports"},{"source_id":"S030","target_id":"S034","kind":"supports"},{"source_id":"S055","target_id":"S034","kind":"supports"},{"source_id":"S056","target_id":"S034","kind":"supports"},{"source_id":"S058","target_id":"S034","kind":"supports"},{"source_id":"S057","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S046","kind":"supports"},{"source_id":"S033","target_id":"S046","kind":"supports"},{"source_id":"S017","target_id":"S046","kind":"supports"},{"source_id":"S017","target_id":"S047","kind":"supports"},{"source_id":"S012","target_id":"S047","kind":"supports"},{"source_id":"S061","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S012","target_id":"S048","kind":"supports"},{"source_id":"S059","target_id":"S048","kind":"supports"},{"source_id":"S018","target_id":"S049","kind":"supports"},{"source_id":"S010","target_id":"S049","kind":"supports"},{"source_id":"S062","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S025","target_id":"S051","kind":"supports"},{"source_id":"S024","target_id":"S051","kind":"supports"},{"source_id":"S060","target_id":"S052","kind":"supports"},{"source_id":"S021","target_id":"S052","kind":"supports"},{"source_id":"S022","target_id":"S052","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S048","target_id":"S045","kind":"supports"},{"source_id":"S049","target_id":"S045","kind":"supports"},{"source_id":"S050","target_id":"S045","kind":"supports"},{"source_id":"S051","target_id":"S045","kind":"supports"},{"source_id":"S052","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:genp"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"一般复合优化模型","source_units":[{"raw_tex":" The definition of (strong) variational sufficient condition (Definition \\ref{def:svasc}) is officially given in \\cite{Roh22}  for the following general composite optimization problem\n \\begin{equation}\\label{eq:genp}\n \t\\min_{x\\in \\mathbb{X}}\\quad f(x)+\\theta(G(x)),\n \\end{equation}\n where $\\mathbb{X}$ and $\\mathbb{Y}$ are two given Euclidean spaces, $f:\\mathbb{X}\\to \\mathbb{R}$ and $G:\\mathbb{X}\\to \\mathbb{Y}$ are twice continuously differentiable, and $\\theta:\\mathbb{Y}\\to (-\\infty,\\infty]$ is a closed proper convex function. The strong variational sufficient condition is firstly introduced to deal with the local convexity of primal augmented Lagrangian function and the augmented tilt stability. It has gained more and more attention for its mathematical elegance and wide applications in the convergence analysis of multiplier methods. Several characterizations of this abstract property are also given in \\cite{Roh22}. For instance, it is equivalent to the positive definiteness of the Hessian bundle of the augmented Lagrangian function \\cite[Theorem 3]{Roh22} or the criterion \\cite[Theorem 5]{Roh22} involving quadratic bundle (Definition \\ref{def:quadratic bundle}).\n","start":6647,"end":7829}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:NLSDPc"],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"半正定规划与矩阵空间","source_units":[{"raw_tex":"\\begin{equation}\\label{eq:NLSDPc}\n\t\\begin{array}{cl}\n\t\t\\displaystyle{\\min_{x\\in \\mathbb{X}}} & f(x) \\\\ [3pt]\n\t\t{\\rm s.t.} &{G(x)\\in \\mathbb{S}^n_+},\n\t\\end{array}\n\\end{equation}\nwhere  $\\mathbb{S}^n$ is the linear space of all $n\\times n$ real symmetric matrices equipped with the usual Frobenius inner product and its induced norm, $\\mathbb{S}_+^n$ ($\\mathbb{S}_-^n$) is the closed convex cone of all $n\\times n$ positive (negative) semidefinite matrices in $\\mathbb{S}^n$. One of our contributions lies in uncovering the equivalence between strong variational sufficient condition and strong SOSC (see Definition \\ref{def:sigt} for details) for  NLSDP problems without requiring any other constraint qualifications.\n","start":8843,"end":9560}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":["eq:L-function"],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"普通拉格朗日函数","source_units":[{"raw_tex":"For NLSDP \\eqref{eq:NLSDPc}, the Lagrangian function is defined by\n\\begin{equation}\\label{eq:L-function}\n\tL(x,Y):=f(x) +\\langle Y,G(x)\\rangle, \\quad (x,Y)\\in\\mathbb{X}\\times\\mathbb{S}^n.\n\\end{equation}\nFor any $Y\\in\\mathbb{S}^n$, denote the first-order and second-order derivatives of $L(\\cdot,Y)$ at $x\\in\\mathbb{X}$ by $L_x'(x,Y)$ and $L_{xx}''(x,Y)$, respectively. Given $\\rho > 0$, the augmented Lagrangian function of \\eqref{eq:NLSDPc} takes the following form (cf. \\cite[Section 11.K]{RoWe98} and \\cite{ShapS04})\n","start":11177,"end":11696}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":["eq:deflag"],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"增广拉格朗日函数","source_units":[{"raw_tex":"\\begin{equation}\\label{eq:deflag}\n\t\\mathcal{L}_{\\rho}(x,Y):=f(x)+\\frac{\\rho}{2}{\\rm dist}^2(G(x)+\\frac{Y}{\\rho},\\mathbb{S}_+^n)-\\frac{\\|Y\\|^2}{2\\rho},\n\\end{equation}\nwhere ${\\rm dist}(z,\\mathbb{S}_+^n)$ is the distance from point $z$ to $\\mathbb{S}_+^n$.\n","start":11696,"end":11951}]},{"id":"S006","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"切锥法锥与临界锥","source_units":[{"raw_tex":"For a given Euclidean space $\\mathbb{X}$,\nlet $C$ be any subset in $\\mathbb{X}$, ${\\rm aff}\\,{C}$ is the affine hull of $C$ \\cite[page 6]{Rockafellar1970}. The Bouligand tangent/contingent cone of $C$ at $x$ is a closed cone defined by\n\\[\n\\mathcal{T}_{C}(x):= \\left\\{d\\in\\mathbb{X}\\mid \\mbox{$\\exists\\, t^k\\downarrow 0$ and  $d^k\\to d$ with  $x+t^kd^k\\in C$ for all $k$} \\right\\}.\n\\]\nWhen $C$ is convex, the normal cone in the sense of  convex analysis \\cite{Rockafellar1970} is defined as\n$$\\mathcal{N}_{C}(x)=\\left \\{d \\in \\mathbb{X}\\mid\\langle d,x'-x\\rangle\\leq 0 \\ \\forall\\,x'\\in C \\right\\}.$$\nFor any $x\\in C$, the critical cone  associated with ${ y}\\in\\mathcal{N}_C(x)$ of $C$ is defined in \\cite[Page 98]{DoRo14}, i.e., ${\\cal C}_{C}({x},{y})=\\mathcal{T}_{C}({x})\\cap({y})^{\\perp}$.\n","start":16485,"end":17276}]},{"id":"S007","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"集合值映射的外极限","source_units":[{"raw_tex":"\nFor a set-valued mapping $\\mathcal{F}:\\mathbb{X}\\rightrightarrows\\mathbb{X}$, its $\\limsup$ means\n$$\\limsup\\limits_{{x}\\rightarrow\\bar{x}}\\mathcal{F}({x}):=\\left\\{d\\in\\mathbb{X}\\mid \\exists\\;x_k\\rightarrow\\bar{x},\\; {y}_k\\rightarrow d\\;{\\rm such\\;that}\\;{y}_k\\in\\mathcal{F}({x}_k)\\;\\forall\\,k\\right\\}.$$\nThe following definitions of proximal and (general/Mordukhovich limiting) subdifferentials of functions are adopted from \\cite[Definition 8.45 and 8.3]{RoWe98}.\n","start":17276,"end":17742}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"近端与极限次微分","source_units":[{"raw_tex":"\\begin{definition}\n\tConsider a function $f:\\mathbb{X}\\rightarrow(-\\infty, +\\infty]$ and a point $\\bar{x}$ with $f(\\bar{x})$ finite. The proximal subdifferential of $f$ at $\\bar{x}$ is defined as\n\t$$\\partial^{\\pi}f(\\bar{x}):=\\{{d}\\mid \\exists\\; l>0, r>0\\;{\\rm such\\; that}\\;f(x)\\geq f(\\bar{x})+\\langle{d},{x}-\\bar{x}\\rangle-l\\|{x}-\\bar{x}\\|^2\\;\\forall\\,{x}\\in{\\cal B}_{r}(\\bar{x})\\},$$\n\twhere ${\\cal B}_{r}(\\bar{x})$ is the ball centered at $\\bar{x}$ with radius $r$.\n\tThe (general/Mordukhovich limiting) subdifferential of $f$ at $\\bar{x}$ is $\\partial f(\\bar{x}):=\\limsup\\limits_{{x}\\stackrel{f}{\\rightarrow}\\bar{x}}\\partial^{\\pi}f({x})$, where ${x}\\stackrel{f}{\\rightarrow}\\bar{x}$ signifies ${x}\\rightarrow\\bar{x}$ with $f({x})\\rightarrow f(\\bar{x})$.\n\\end{definition}\n","start":17742,"end":18514}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"次微分定义的兼容性","source_units":[{"raw_tex":"Note that in an Euclidean space, the (general) subdifferential also can be constructed via the regular/Fr\\'{e}chet subdifferentials  of functions $\\widehat{\\partial}f$ \\cite[Definition 8.3]{RoWe98}. It is well-known that these two definitions are equivalent (see e.g., \\cite[page 345]{RoWe98} or \\cite[Theorem 1.89]{Mordukhovich06} for details). In particular, when $f$ is convex and finite at the corresponding point, the proximal and (general) subdifferentials coincide with the subdifferential in the sense of convex analysis \\cite{Rockafellar1970}.\n","start":18514,"end":19067}]},{"id":"S010","semantic_type":"notation","kind":"notation","labels":["eq:eig-decomp"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"谱分解与符号分块","source_units":[{"raw_tex":"Let  $A \\in\\mathbb{S}^n$ be given. We use $\\lambda_{1}(A)\\ge \\lambda_2(A) \\ge \\ldots \\ge \\lambda_{n}(A)$ to denote the eigenvalues of $A$ (all real and counting multiplicity) arranging in  nonincreasing order and use $\\lambda(A)$ to denote the vector of the ordered eigenvalues of $A$. Let $\\Lambda(A):= {\\rm Diag}(\\lambda(A))$. Also, we use $v_1(A)> \\dots> v_d(A)$ to denote the different eigenvalues and\n$\\zeta_l:=\\{i\\mid \\lambda_i(A)=v_l(A)\\}$.\nConsider the eigenvalue decomposition of $A$, i.e., $A={P} \\Lambda(A){P}^{T}$, where ${P}\\in{\\cal O}^{n}(A)$ is a corresponding\northogonal matrix of the orthonormal eigenvectors.\nBy considering the index sets of positive, zero, and negative eigenvalues of $A$, we are able to write $A$ in the following form\n{\\footnotesize\n\\begin{equation}\\label{eq:eig-decomp}\n\tA= \\left[\\begin{array}{ccc}\n\t\t{P}_{\\alpha} & {P}_{\\beta} & {P}_{\\gamma}\n\t\\end{array}\\right] \\left[\\begin{array}{ccc}\n\t\t\\Lambda(A)_{\\alpha\\alpha} & 0 & 0 \\\\ [3pt]\n\t\t0 & 0 & 0\\\\ [3pt]\n\t\t0 & 0 & \\Lambda(A)_{\\gamma\\gamma}\n\t\\end{array}\\right]\\left[\\begin{array}{c}\n\t\t{P}_{\\alpha}^T \\\\ [3pt]\n\t\t{P}_{\\beta}^T \\\\\n\t\t{P}_{\\gamma}^T\n\t\\end{array}\\right],\n\\end{equation}}\nwhere $\\alpha:=\\{i: \\lambda_i(A)>0\\}$, $\\beta:=\\{i: \\lambda_i(A)=0\\}$ and $\\gamma:=\\{i: \\lambda_i(A)<0\\}$.\n","start":19129,"end":20405}]},{"id":"S011","semantic_type":"external_fact","kind":"external_fact","labels":["eq:cricsdp"],"member_ids":["S011"],"declared_external":true,"declaration_text":["原始前提列表：CSun08, equations (11)–(12)；本节点保留文中重述的公式。"],"source_only":false,"title":"半正定锥临界锥公式","source_units":[{"raw_tex":"The critical cone of $\\mathbb{S}_+^n$ at ${Y}\\in{\\cal N}_{\\mathbb{S}_+^n}({X})$ with $A=X+Y$ is given by (cf. \\cite[equations (11) and (12)]{CSun08})\n\\begin{equation}\\label{eq:cricsdp}\n\t{\\cal C}_{\\mathbb{S}_+^n}({X},{Y})=\\{H\\in\\mathbb{S}^n\\mid P^T_{\\beta}HP_{\\beta}\\in\\mathbb{S}^{|\\beta|}_+,\\;P^T_{\\beta}HP_{\\gamma}=0,\\;P^T_{\\gamma}HP_{\\gamma}=0\\}\n\\end{equation}\n","start":20408,"end":20771}]},{"id":"S012","semantic_type":"external_fact","kind":"external_fact","labels":["eq:affcsdp"],"member_ids":["S012"],"declared_external":true,"declaration_text":["原始前提列表：CSun08, equations (11)–(12)；引用见紧邻的 L263。"],"source_only":false,"title":"临界锥仿射包公式","source_units":[{"raw_tex":"and the affine hull of ${\\cal C}_{\\mathbb{S}_+^n}({X},{Y})$ is\n\\begin{equation}\\label{eq:affcsdp}\n\t{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}({X},{Y})=\\{H\\in\\mathbb{S}^n\\mid P^T_{\\beta}HP_{\\gamma}=0,\\;P^T_{\\gamma}HP_{\\gamma}=0\\}.\n\\end{equation}\n","start":20771,"end":21010}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["eq:NLSDP"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"指标函数形式的半正定规划","source_units":[{"raw_tex":"The  NLSDP \\eqref{eq:NLSDPc} can be written in the following form\n\\begin{eqnarray}\n\t%\\mbox{(SDP)}~~~~\n\t\\min\\quad && f(x)+\\delta_{\\mathbb{S}^n_+}(G(x)), \\label{eq:NLSDP}\n\\end{eqnarray}\nwhere\n$\\delta_{\\mathbb{S}^n_+} : \\mathbb{S}^n\\rightarrow (-\\infty,\\infty]$ is the indicator function of positive semidefinite cone $\\mathbb{S}^n_+$.\n","start":21010,"end":21343}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":["eq:KKT-NLSDP-p1"],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"KKT 系统与驻点","source_units":[{"raw_tex":"Denote ${S}_{KKT}(a,b)$ the solution set of the KKT optimality condition for problem \\eqref{eq:NLSDP}, i.e.,\n\\begin{equation}\\label{eq:KKT-NLSDP-p1}\n\t{S}_{KKT}(a,b)=\\left\\{(x,Y)\\in \\mathbb{X}\\times\\mathbb{S}^n \\Bigg | \\left.\n\t\\begin{array}{l}\n\t\tL_x'(x,Y)-a=0,\\\\ [3pt]\n\t\t\\mathbb{S}_+^n\\ni (G(x)-b)\\perp Y \\in \\mathbb{S}_-^n.\n\t\\end{array}\n\t\\right.\\right \\}.\n\\end{equation}\nFor any KKT pair $(\\bar{x},\\overline{Y})$ that satisfies the KKT condition with $(a,b) = (0,0)$, we call $\\bar{x}$ a stationary point.\n","start":21343,"end":21849}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":["eq:defcalm"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"参考驻点的乘子集合","source_units":[{"raw_tex":"Suppose $\\bar{x}$ is a stationary point. Define ${\\cal M}(\\bar{x})$ as the set of all multipliers $Y\\in\\mathbb{S}^n$ satisfying the KKT condition \\eqref{eq:KKT-NLSDP-p1}, i.e.,\n\\begin{equation}\\label{eq:defcalm}\n\t{\\cal M}(\\bar{x})=\\{Y\\in\\mathbb{S}^n \\mid (\\bar{x},Y)\\in {S}_{KKT}(0,0)\\}.\n\\end{equation}\n","start":21849,"end":22152}]},{"id":"S017","semantic_type":"definition","kind":"definition","labels":["def:sigt","eq:defssosc"],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"强二阶充分条件与曲率项","source_units":[{"raw_tex":"\\begin{definition}\\label{def:sigt}\\cite[Definition 3.2]{Sun06}\n\tLet $\\bar{x}$ be a stationary point of NLSDP \\eqref{eq:NLSDP} and $\\overline{Y}\\in{\\cal M}(\\bar{x})$. We say the strong second order sufficient condition (SOSC) holds at $(\\bar{x},\\overline{Y})$ if\n\t\\begin{equation}\\label{eq:defssosc}\n\t\t\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)>0\\quad\\forall\\; 0\\neq G'(\\bar{x})d\\in{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y}).\n\t\\end{equation}\n\twhere $\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)=2\\langle\\overline{Y},(G'(\\bar{x}){d})G(\\bar{x})^{\\dagger}(G'(\\bar{x}){d})\\rangle$ is the $\\sigma$-term and $G(\\bar{x})^{\\dagger}$ is the generalized inverse matrix of $G(\\bar{x})$.\n\\end{definition}\n","start":22620,"end":23416}]},{"id":"S018","semantic_type":"external_fact","kind":"external_fact","labels":["eq:sigtsdp"],"member_ids":["S018"],"declared_external":true,"declaration_text":["原始前提列表：CSun08, page 386。"],"source_only":false,"title":"曲率项的谱展开","source_units":[{"raw_tex":"\nSuppose $A=G(\\bar{x})+\\overline{Y}$ possesses the decomposition \\eqref{eq:eig-decomp}. From \\cite[page 386]{CSun08}, we know the $\\sigma$-term takes the explicit form of\n\\begin{equation}\\label{eq:sigtsdp}\n\t\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},H\\big)=2\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H}_{ij})^2,\n\\end{equation}\nwhere $\\widetilde{H}=P^THP$.\n","start":23416,"end":23802}]},{"id":"S019","semantic_type":"external_fact","kind":"external_fact","labels":["eq:algrad"],"member_ids":["S019"],"declared_external":true,"declaration_text":["原始前提列表中的 SSZhang08 广义导数背景；文中 L309 明示引用。"],"source_only":false,"title":"增广拉格朗日梯度","source_units":[{"raw_tex":"Next, we introduce some essential notation for our main result. The (second order) generalized differentiability  of the augmented Lagrangian function ${\\cal L}_{\\rho}$ defined by \\eqref{eq:deflag} for NLSDP \\eqref{eq:NLSDP} has been explicitly established in \\cite{SSZhang08}.  In fact, it is well-known that ${\\cal L}_{\\rho}$ is continuously differentiable with\n\\begin{equation}\\label{eq:algrad}\n\t({\\cal L}_{\\rho})_x'(x,Y)=f'(x)+\\rho G'(x)^*\\Pi_{\\mathbb{S}_-^n}(G(x)+\\rho^{-1}Y),\n\\end{equation}\nwhere $\\Pi_{\\mathbb{S}_-^n}(\\cdot)$ is the metric projection over $\\mathbb{S}_{-}^n$ and $G'(x)^*$ denotes the adjoint of the corresponding linear mapping. Moreover, since $({\\cal L}_{\\rho})_x'$ is  Lipschitz continuous, by using Rademacher’s theorem, the B(ouligand)-subdifferential of $({\\cal L}_{\\rho})_x'$ at $(x,Y)$ is given by\n","start":23803,"end":24633}]},{"id":"S020","semantic_type":"definition","kind":"definition","labels":["eq:partb"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"梯度映射的布次微分","source_units":[{"raw_tex":"where $\\Pi_{\\mathbb{S}_-^n}(\\cdot)$ is the metric projection over $\\mathbb{S}_{-}^n$ and $G'(x)^*$ denotes the adjoint of the corresponding linear mapping. Moreover, since $({\\cal L}_{\\rho})_x'$ is  Lipschitz continuous, by using Rademacher’s theorem, the B(ouligand)-subdifferential of $({\\cal L}_{\\rho})_x'$ at $(x,Y)$ is given by\n\\begin{equation}\\label{eq:partb}\n\t\\partial_B\\big(({\\cal L}_{\\rho})'_x\\big)(x,Y):=\\{\\lim_{k\\rightarrow\\infty}(({\\cal L}_{\\rho})_{x}')'(x^k,Y^k)\\mid (x^k,Y^k)\\in{\\cal U},\\ (x^k,Y^k)\\rightarrow(x,Y)\\},\n\\end{equation}\nwhere ${\\cal U}$ is the set of Fr\\'{e}chet-differentiable points of $({\\cal L}_{\\rho})_x'$.\n","start":24300,"end":24939}]},{"id":"S021","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S021"],"declared_external":true,"declaration_text":["原始前提列表：SSZhang08, equation (18), Lemmas 2–3；Roh22 的 Hessian-bundle 刻画及 equation (3.6)。所选上下文另注明 Roh22 (3.1)。"],"source_only":false,"title":"广义海森束的识别","source_units":[{"raw_tex":"It follows from \\cite[(18)]{SSZhang08} (using \\cite[Lemma 2 and 3]{SSZhang08}) that\n\\begin{align*}\n\t&\\pi_x\\partial_B\\big(({\\cal L}_{\\rho})_x'\\big)(x,Y)(\\Delta x)\\\\\n\t&=L_{xx}''(x,\\rho\\Pi_{\\mathbb{S}_-^n}(G(x)+\\rho^{-1}Y))(\\Delta x)+\\rho G'({x})^*\\partial_B\\Pi_{\\mathbb{S}_-^n}(G({x})+\\rho^{-1}Y) G'({x})(\\Delta x),\n\\end{align*}\nwhere $\\pi_x\\partial_B\\big(({\\cal L}_{\\rho})_x'\\big)(x,Y)$ is the projection of $\\partial_B\\big(({\\cal L}_{\\rho})_x'\\big)(x,Y)$ onto the space $\\mathbb{X}$ and  $\\partial_B\\Pi_{\\mathbb{S}_-^n}(G({x})+\\rho^{-1}Y)$ is the B-subdifferential of $\\Pi_{\\mathbb{S}_-^n}(\\cdot)$ at $G({x})+\\rho^{-1}Y$. Actually, the set $\\pi_x\\partial_B\\big(({\\cal L}_{\\rho})_x'\\big)(x,Y)$ is recently defined as the $x$ part of the Hessian bundle \\cite[(3.1)]{Roh22} for the augmented Lagrangian function ${\\cal L}_{\\rho}$ at $(x,Y)$ (see \\cite[(3.6)]{Roh22} for detail).\n","start":24939,"end":25815}]},{"id":"S022","semantic_type":"definition","kind":"definition","labels":["eq:algrad2"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"参考点的广义海森算子","source_units":[{"raw_tex":"\n\n\nLet $\\bar{x}$ be the stationary point of \\eqref{eq:NLSDP}. For each $\\overline{Y}\\in{\\cal M}(\\bar{x})$ and $W\\in\\partial_B\\Pi_{\\mathbb{S}_-^n}(G(\\bar{x})+\\rho^{-1}\\overline{Y})$, define the following mapping\n\\begin{equation}\\label{eq:algrad2}\n\t{\\cal A}_{\\rho}(\\overline{Y},W):=L_{xx}''(\\bar{x},\\overline{Y})+\\rho G'(\\bar{x})^*WG'(\\bar{x}).\n\\end{equation}\n","start":25815,"end":26173}]},{"id":"S023","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S023"],"declared_external":true,"declaration_text":["原始前提列表：SSZhang08, Lemma 5。"],"source_only":false,"title":"锥投影布导数的分块形式","source_units":[{"raw_tex":"Let $\\overline{Z}=G(\\bar{x})+\\overline{Y}$ possesses decomposition \\eqref{eq:eig-decomp} with $\\overline{P}\\in{\\cal O}^{n}(\\overline{Z})$. It follows from \\cite[Lemma 5]{SSZhang08} that $W\\in\\partial_B\\Pi_{\\mathbb{S}_-^n}(\\overline{Z})$ if and only if there exists $W_0\\in\\partial_B\\Pi_{\\mathbb{S}_-^{\\lvert\\beta\\rvert}}(0)$ such that for all $H\\in\\mathbb{S}^n$\n{\\footnotesize\n\\begin{equation*}\n\tW(H)=\\overline{P}\\left[\\begin{array}{ccc}\n\t\t0 & 0 & \\Sigma_{\\alpha\\gamma}\\circ\\overline{P}_{\\alpha}^TH\\overline{P}_{\\gamma}\\\\\n\t\t0 & W_0(\\overline{P}_{\\beta}^TH\\overline{P}_{\\beta}) & \\overline{P}_{\\beta}^TH\\overline{P}_{\\gamma} \\\\\n\t\t\\Sigma_{\\gamma\\alpha}\\circ\\overline{P}_{\\gamma}^TH\\overline{P}_{\\alpha} & \\overline{P}_{\\gamma}^TH\\overline{P}_{\\beta} & \\overline{P}_{\\gamma}^TH\\overline{P}_{\\gamma}\n\t\\end{array}\\right]\\overline{P}^T,\n\\end{equation*}}\nwhere\n\\begin{equation*}\n\\begin{cases}\t\t\\Sigma_{ij}=1-\\frac{\\max\\{\\lambda_i,0\\}+\\max\\{\\lambda_j,0\\}}{\\lvert\\lambda_i\\rvert+\\lvert\\lambda_j\\rvert}, & (i,j)\\notin\\beta\\times\\beta\\\\\n\\Sigma_{ij}\t\\in[0,1] & (i,j)\\in\\beta\\times\\beta\n\t\\end{cases}\n\\end{equation*}\nwith $\\lambda_i:=\\lambda_i(\\overline{Z})$ for short. Let ${\\cal Q}^n$ denote the set of $n$-dimensional orthogonal matrix.\n","start":26173,"end":27399}]},{"id":"S024","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S024"],"declared_external":true,"declaration_text":["原始前提列表：SSZhang08, Lemma 5；承接 L331 的引用。"],"source_only":false,"title":"零特征值块的表示","source_units":[{"raw_tex":"Also, $W_0\\in\\partial_B\\Pi_{\\mathbb{S}_-^{\\lvert\\beta\\rvert}}(0)$ if and only if there exist $Q\\in{\\cal Q}^{\\lvert\\beta\\rvert}$ and $\\Omega\\in\\mathbb{S}^{\\lvert\\beta\\rvert}$ with entries $\\Omega_{ij}\\in[0,1]$ such that for all $H\\in\\mathbb{S}^{\\lvert\\beta\\rvert}$,\n$$W_0(H)=Q(\\Omega\\circ(Q^THQ))Q^T.$$\n","start":27399,"end":27701}]},{"id":"S025","semantic_type":"external_fact","kind":"external_fact","labels":["eq:expcalA"],"member_ids":["S025"],"declared_external":true,"declaration_text":["原始前提列表：SSZhang08, Lemma 9。"],"source_only":false,"title":"广义海森二次型展开","source_units":[{"raw_tex":"By \\cite[Lemma 9]{SSZhang08}, we have\n{\\small \\begin{align}\n\t\\langle d,{\\cal A}_{\\rho}(\\overline{Y},W)d\\rangle=&\\langle d,L_{xx}''(\\bar{x},\\overline{Y})d\\rangle+\\rho\\sum_{i,j\\in\\gamma}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2+2\\rho\\sum_{i\\in\\beta,j\\in\\gamma}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\label{eq:expcalA}\\\\\n\t&+2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2+\\rho\\sum_{i,j\\in\\beta}(\\Omega_{\\rho})_{ij}({P}^T(G'(\\bar{x})d){P})_{ij}^2,\\nonumber\n\\end{align}}\nwhere $\\Omega_{\\rho}\\in\\mathbb{S}^{\\lvert\\beta\\rvert}$ with entries $(\\Omega_{\\rho})_{ij}\\in[0,1]$, $P=[\\overline{P}_{\\alpha}\\;\\overline{P}_{\\beta}Q\\;\\overline{P}_{\\gamma}]$ with $Q\\in{\\cal Q}^{\\lvert\\beta\\rvert}$.\n","start":27701,"end":28477}]},{"id":"S026","semantic_type":"definition","kind":"definition","labels":["eq:pgpert"],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"扰动模型与一阶条件","source_units":[{"raw_tex":"The definition of (strong) variational sufficient condition is given in \\cite{Roh22}  for general composite optimization problem \\eqref{eq:genp}.\nWe can recast \\eqref{eq:genp} in the form\n\\begin{equation}\\label{eq:pgpert}\n\t\\min\\;\\phi(x,u)\\;\\mbox{subject to}\\;u=0,\\quad\\mbox{where}\\;\\phi(x,u)=f(x)+\\theta(G(x)+u).\n\\end{equation}\nThe first order local optimality condition for \\eqref{eq:genp}  of $\\bar{x}$ is the existence of $\\overline{Y}$ such that\n$$L_x'(\\bar{x},\\overline{Y})=0\\quad \\mbox{with}\\quad\\overline{Y}\\in\\partial\\, \\theta(G(\\bar{x})). $$\n","start":28523,"end":29074}]},{"id":"S027","semantic_type":"definition","kind":"definition","labels":["eq:phir"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"变分强凸性的局部代表","source_units":[{"raw_tex":"Define\n\\begin{equation}\\label{eq:phir}\n\\phi_r:=\\phi(x,u)+\\frac{r}{2}|u|^2.\n\\end{equation}\nThe variational (strong) convexity, which is firstly proposed in \\cite{rock247}, refers to the existence of open convex neighborhoods ${\\cal W}$ of $(\\bar{x},0)$ and ${\\cal Z}$ of $(0,\\overline{Y})$ such that there exists a proper closed (strongly) convex function $\\psi\\leq\\phi_r2$ on ${\\cal W}$ such that\n\\begin{equation*}\n\t({\\cal W}\\times{\\cal Z})\\cap{\\rm gph}\\,\\partial\\psi=({\\cal W}\\times{\\cal Z})\\cap{\\rm gph}\\,\\partial\\phi_r\n\\end{equation*}\nand for $(x,u;v,y)$ belonging to this common set, $\\psi(x,u)=\\phi_r(x,u)$.\n","start":29074,"end":29687}]},{"id":"S028","semantic_type":"definition","kind":"definition","labels":["def:svasc"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"强变分充分性","source_units":[{"raw_tex":"\\begin{definition}\\label{def:svasc}\\cite{Roh22}\n\tThe (strong) variational sufficient condition for local optimality in \\eqref{eq:pgpert} holds with respect to $\\bar{x}$ and $\\overline{Y}$ satisfying the first order condition if there exists $r>0$ such that $\\displaystyle\\phi_r(x,u)$ is variationally (strongly) convex with respect to the pair $\\big((\\bar{x},0),(0,\\overline{Y})\\big)$ in ${\\rm gph}\\,\\partial\\phi_r$.\n\\end{definition}\n\n\n","start":29687,"end":30123}]},{"id":"S029","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"二阶次导数与上图可微性","source_units":[{"raw_tex":"To characterize the above property, the tools of the second subderivative and generalized quadratic form are needed.\n\\begin{definition}\\cite[Definition 13.6]{RoWe98}\n\tLet $\\bar{x}$ be a point where the function $f:\\mathbb{X}\\rightarrow[-\\infty,+\\infty]$ is finite. $f$ is twice epi-differentiable at $\\bar{x}$ for $v$ if the functions\n\t$$\\Delta^2_{t}f(\\bar{x}\\mid v)(u)=\\frac{f(\\bar{x}+tu)-f(\\bar{x})-t\\langle v,u\\rangle}{\\frac{1}{2}t^2}$$\n\tepi-converge to $d^2f(\\bar{x}\\mid v)$ as $t\\downarrow0$, where $d^2f(\\bar{x}\\mid v)$ is the second subderivative of $f$ at $\\bar{x}$ for $v$ defined as\n\t$$d^2f(\\bar{x}\\mid v)(w)=\\liminf_{t\\downarrow0,u\\rightarrow w}\\Delta^2_{t}f(\\bar{x}\\mid v)(u)$$\n\\end{definition}\n","start":30123,"end":30830}]},{"id":"S030","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S030"],"declared_external":true,"declaration_text":["原始前提列表：mohammadi20, Theorem 3.6。"],"source_only":false,"title":"半正定锥指标的二阶可微性","source_units":[{"raw_tex":"We know from \\cite[Theorem 3.6]{mohammadi20} that $\\delta_{\\mathbb{S}_+^n}$ is twice epi-differentiable at $X$ for $Y$ with $Y\\in{\\cal N}_{\\mathbb{S}_+^n}(X)$.\n\n","start":30830,"end":30991}]},{"id":"S031","semantic_type":"definition","kind":"definition","labels":["def:genguadf"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"广义二次型与广义二阶可微","source_units":[{"raw_tex":"\\begin{definition}\\label{def:genguadf} \\cite{Roh22}\n\tA generalized linear mapping ${\\cal R}$ from $\\mathbb{X}$ to $\\mathbb{Y}$, is a set-valued mapping for which ${\\rm gph}\\,{\\cal R}$ is a subspace of $\\mathbb{X}\\times\\mathbb{Y}$. This means that ${\\rm dom}{\\cal R}$ is a subspace $\\mathbb{Z}$ of $\\mathbb{X}$, ${\\cal R}(0)$ is a subspace $\\mathbb{Z}'$ of $\\mathbb{Y}$, and there is an ordinary linear mapping ${\\cal R}_0:\\mathbb{Z}\\rightarrow\\mathbb{Y}$ such that ${\\cal R}(x)={\\cal R}_0(x)+\\mathbb{Z}'$ for $x\\in\\mathbb{Z}$.\n\tWe call a function $q: \\mathbb{X}\\rightarrow(-\\infty,+\\infty]$ is a generalized quadratic form on $\\mathbb{X}$ if $q(0)=0$ and the subgradient mapping $\\partial q:\\mathbb{X}\\rightrightarrows\\mathbb{X}$ is generalized linear. A function $g$ on $\\mathbb{X}$ will be called generalized twice differentiable at $x$ for a subgradient $y$ if it is twice epi-differentiable at $x$ for $y$ with the second-order subderivative $d^2g(x\\mid y)$ being a generalized quadratic form.\n\\end{definition}\n\n\n","start":30991,"end":32008}]},{"id":"S032","semantic_type":"definition","kind":"definition","labels":["def:quadratic bundle"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"二次束","source_units":[{"raw_tex":"The following definition of the quadratic bundle is taken from \\cite{Roh22}, which is essential in characterizing the strong variational sufficiency.\n\\begin{definition}\\label{def:quadratic bundle}\n\tFor general optimization problem \\eqref{eq:genp}, suppose $(\\bar{x},\\overline{Y})$ is a KKT pair. The quadratic bundle of $\\theta$ is defined as\n\t\\begin{equation*}\n\t\t{\\rm quad}\\,\\theta(G(\\bar{x})\\mid\\overline{Y})=\\left\\{\n\t\t\\begin{array}{l}\n\t\t\t\\mbox{the collection of generalized quadratic forms}\\; q\\; \\mbox{for which}\\\\\n\t\t\t\\exists(X^k,Y^k)\\rightarrow(G(\\bar{x}),\\overline{Y})\\;\\mbox{with}\\;\\theta\\;\\mbox{generalized twice differentiable}\\\\\n\t\t\t\\mbox{at}\\;X^k\\;\\mbox{for}\\;Y^k\\;\\mbox{and such that the generalized quadratic}\\\\\n\t\t\t\\mbox{forms}\\;q_k=\\frac{1}{2}d^2\\theta(X^k\\mid Y^k)\\;\\mbox{converge epigraphically to}\\;q.\n\t\t\\end{array}\n\t\t\\right.\n\t\\end{equation*}\n\\end{definition}\n","start":32008,"end":32884}]},{"id":"S033","semantic_type":"proposition","kind":"proposition","labels":["prop:thm5","svs-equ"],"member_ids":["S033"],"declared_external":true,"declaration_text":["原始前提列表明确许可：Roh22, Theorem 5，文中重述为 B0046 / prop:thm5。"],"source_only":false,"title":"二次束刻画强变分充分性","source_units":[{"raw_tex":"As mentioned in \\cite{Roh22}, the variational sufficient condition guarantees the local optimality for \\eqref{eq:pgpert}. The following result is taken from \\cite[Theorem 5]{Roh22}, which is useful for the subsequent analysis.\n\\begin{proposition}\\label{prop:thm5}\n\tFor general optimization problem \\eqref{eq:genp},\n\tstrong variational sufficient condition for local optimality with respect to $(\\bar{x},\\overline{Y})$ is equivalent to that  every $q\\in{\\rm quad}\\,\\theta(G(\\bar{x})\\mid\\overline{Y})$ has\n\t\\begin{equation}\\label{svs-equ}\n\t\t\\frac{1}{2}\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle+q(G'(\\bar{x}){d})>0\\quad{\\rm when}\\;d\\neq0,\n\t\\end{equation}\n\twhere ${\\rm quad}\\,\\theta(G(\\bar{x})\\mid\\overline{Y})$ is the quadratic bundle of $\\theta$.\n\\end{proposition}\n","start":32884,"end":33654}]},{"id":"S034","semantic_type":"proposition","kind":"proposition","labels":["eq:qeqsg","prop:onesidessosc"],"member_ids":["S034","S035","S036","S037","S038","S039","S040","S041","S042","S043"],"declared_external":false,"source_only":false,"title":"构造仿射临界锥上的二次束元素","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:onesidessosc}\n\tLet $\\bar{x}\\in\\mathbb{X}$ be a stationary point to NLSDP \\eqref{eq:NLSDP} and $\\overline{Y}\\in {\\cal M}(\\bar{x})$, where ${\\cal M}(\\bar{x})$ is given in \\eqref{eq:defcalm}. Let $A=G(\\bar{x})+\\overline{Y}$, which possesses the decomposition \\eqref{eq:eig-decomp}. Then, there exists $q\\in {\\rm quad}\\,\\delta_{\\mathbb{S}_+^n}(G(\\bar{x})\\mid\\overline{Y})$ such that for all $H\\in\\mathbb{S}^n$,\n\t\\begin{align}\n\t\tq(H)&=-\\frac{1}{2}\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},H\\big)+\\delta_{{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y})}(H)\\nonumber\\\\\n\t\t&=\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H}_{ij})^2+\\delta_{{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y})}(H),\\label{eq:qeqsg}\n\t\\end{align}\n\twhere $\\widetilde{H}=P^THP$.\n\\end{proposition}","start":40074,"end":40921},{"raw_tex":"\\begin{proof}\n\tFor each $k$, choose\n\t\\[\n\tX^{k}={P} \\left[ \\begin{array}{ccccc} \\Lambda(A)_{\\alpha \\alpha} & 0\n\t\t& 0  \\\\   0 & (z^{k})_{\\beta} & 0 \\\\   0 & 0 &\n\t\t0 \\\\\n\t\\end{array} \\right] {P}^{T} \\quad {\\rm and} \\quad Y^{k}=\\overline{Y},\n\t\\]\n\twhere $z^k\\downarrow0$ (this notation means for each $k$, $z^k>0$ and $z^k\\rightarrow0$ as $k\\rightarrow\\infty$) with each $(z^k)_i$ non-increasing on $k$. Let $A^k=X^k+Y^k$ for each $k$. Since $\\mathbb{S}_+^n$ is ${\\cal C}^2-$cone reducible \\cite[Example 3.140]{BShapiro00}, we know from \\cite[Theorem 3.3]{mohammadi20}, \\eqref{eq:cricsdp} and \\eqref{eq:sigtsdp} that for any $H\\in\\mathbb{S}^n$,\n","start":40922,"end":41561},{"raw_tex":"\\begin{proof}\n\tFor each $k$, choose\n\t\\[\n\tX^{k}={P} \\left[ \\begin{array}{ccccc} \\Lambda(A)_{\\alpha \\alpha} & 0\n\t\t& 0  \\\\   0 & (z^{k})_{\\beta} & 0 \\\\   0 & 0 &\n\t\t0 \\\\\n\t\\end{array} \\right] {P}^{T} \\quad {\\rm and} \\quad Y^{k}=\\overline{Y},\n\t\\]\n\twhere $z^k\\downarrow0$ (this notation means for each $k$, $z^k>0$ and $z^k\\rightarrow0$ as $k\\rightarrow\\infty$) with each $(z^k)_i$ non-increasing on $k$. Let $A^k=X^k+Y^k$ for each $k$. Since $\\mathbb{S}_+^n$ is ${\\cal C}^2-$cone reducible \\cite[Example 3.140]{BShapiro00}, we know from \\cite[Theorem 3.3]{mohammadi20}, \\eqref{eq:cricsdp} and \\eqref{eq:sigtsdp} that for any $H\\in\\mathbb{S}^n$,\n\t\\begin{align*}\n\t\t\\frac{1}{2}d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)=&\\sum_{i\\in\\alpha\\cup\\beta,j\\in\\gamma}\\frac{-\\lambda_j(A^k)}{\\lambda_i(A^k)}\\big(\\widetilde{H}_{ij}\\big)^2+\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)\\\\\n\t\t=&\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}\\big(\\widetilde{H}_{ij}\\big)^2+\\sum_{i\\in\\beta,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(X^k)}\\big(\\widetilde{H}_{ij}\\big)^2\\\\\n\t\t&+\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H),\n\t\\end{align*}\n\twhere ${\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)=\\{H\\in\\mathbb{S}^n\\mid\\widetilde{H}_{\\gamma\\gamma}=0\\}$. It is worth to note that $d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)$ is a generalized quadratic form by using  Definition \\ref{def:genguadf}, as $d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(0)=0$ and $\\partial d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)={\\cal R}(H)+{\\cal N}_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$\n\twith ${\\cal N}_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$ being a subspace and\n\t$${\\cal R}(H)(\\Delta H)=-4\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H})_{ij}\\cdot(P^T\\Delta HP)_{ij}-4\\sum_{i\\in\\beta,j\\in\\gamma}\\frac{\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H})_{ij}\\cdot(P^T\\Delta HP)_{ij}$$\n\tbeing linear on $H$.\n\tThis also implies that $\\delta_{\\mathbb{S}_+^n}$ is generalized twice differentiable at $X^k$ for $Y^k$.\n\t\nLet $f_1^k(H)=\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}\\big(\\widetilde{H}_{ij}\\big)^2$, $f_2^k(H)=\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$ and $f_3^k(H)=\\sum_{i\\in\\beta,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(X^k)}\\big(\\widetilde{H}_{ij}\\big)^2$. By the definition of continuous convergence mentioned in \\cite[page 250]{RoWe98}, we know that $f_1^k$ converges continuously to $f_1$ with\n$$f_1(H)=\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}\\big(\\widetilde{H}_{ij}\\big)^2.$$\n It follows from \\cite[Theorem 7.11]{RoWe98} that $f_1^k$  epi-converges to $f_1$. It can be checked easily that $f_1^k$ also  pointwise converges \\cite[page 239]{RoWe98} to $f_1$.\nSince ${\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)$ is a constant closed set, we know  that $f_2^k$ pointwise converges and epi-converges \\cite[Proposition 4.4]{RoWe98} to $f_2$ with $f_2(H)=\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$. It follows from \\cite[Theorem 7.46(a)]{RoWe98} that $f_1^k+f_2^k$ epi-convergences to $f_1+f_2$. Also, it can be checked directly that $f_1^k+f_2^k$ pointwise convergences to $f_1+f_2$. By the construction of $z^k$, the sequence of $\\{f_3^k\\}$ is nondecreasing ($f_3^k\\leq f_3^{k+1}$). We know from \\cite[Proposition 7.4]{RoWe98} that $f_3^k$ epi-converges to $\\sup_k\\{\\mbox{cl}f_3^k\\}$, where $\\mbox{cl}f_3^k$ is the closure of $f_3^k$. It is easy to see that $\\sup_k\\{\\mbox{cl}f_3^k\\}=\\sup_k \\{f_3^k\\}=\\delta_{\\cal V}$ with ${\\cal V}=\\{H\\in\\mathbb{S}^n\\mid\\widetilde{H}_{\\beta\\gamma}=0\\}$ and $f_3^k$ pointwise converges to $\\delta_{\\cal V}$. Since $f_1^k+f_2^k$ pointwise and epi-converges to $f_1+f_2$, by using \\cite[Theorem 7.46(a)]{RoWe98} again, we obtain that $f_1^k+f_2^k+f_3^k$ epi-converges to $f_1+f_2+f_3$.\n\nCombining the above discussion with \\eqref{eq:affcsdp}, the generalized quadratic form $\\frac{1}{2}d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)$ converges epigraphically to generalized quadratic form\n\t$$\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H}_{ij})^2+\\delta_{{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y})}(H). $$\n\tThus we have verified this proposition.\n\\end{proof}","start":40922,"end":45081},{"raw_tex":"\twhere $z^k\\downarrow0$ (this notation means for each $k$, $z^k>0$ and $z^k\\rightarrow0$ as $k\\rightarrow\\infty$) with each $(z^k)_i$ non-increasing on $k$. Let $A^k=X^k+Y^k$ for each $k$. Since $\\mathbb{S}_+^n$ is ${\\cal C}^2-$cone reducible \\cite[Example 3.140]{BShapiro00}, we know from \\cite[Theorem 3.3]{mohammadi20}, \\eqref{eq:cricsdp} and \\eqref{eq:sigtsdp} that for any $H\\in\\mathbb{S}^n$,\n\t\\begin{align*}\n\t\t\\frac{1}{2}d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)=&\\sum_{i\\in\\alpha\\cup\\beta,j\\in\\gamma}\\frac{-\\lambda_j(A^k)}{\\lambda_i(A^k)}\\big(\\widetilde{H}_{ij}\\big)^2+\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)\\\\\n\t\t=&\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}\\big(\\widetilde{H}_{ij}\\big)^2+\\sum_{i\\in\\beta,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(X^k)}\\big(\\widetilde{H}_{ij}\\big)^2\\\\\n\t\t&+\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H),\n\t\\end{align*}\n\twhere ${\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)=\\{H\\in\\mathbb{S}^n\\mid\\widetilde{H}_{\\gamma\\gamma}=0\\}$. It is worth to note that $d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)$ is a generalized quadratic form by using  Definition \\ref{def:genguadf}, as $d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(0)=0$ and $\\partial d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)={\\cal R}(H)+{\\cal N}_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$\n","start":41163,"end":42460},{"raw_tex":"\twhere ${\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)=\\{H\\in\\mathbb{S}^n\\mid\\widetilde{H}_{\\gamma\\gamma}=0\\}$. It is worth to note that $d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)$ is a generalized quadratic form by using  Definition \\ref{def:genguadf}, as $d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(0)=0$ and $\\partial d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)={\\cal R}(H)+{\\cal N}_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$\n\twith ${\\cal N}_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$ being a subspace and\n\t$${\\cal R}(H)(\\Delta H)=-4\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H})_{ij}\\cdot(P^T\\Delta HP)_{ij}-4\\sum_{i\\in\\beta,j\\in\\gamma}\\frac{\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H})_{ij}\\cdot(P^T\\Delta HP)_{ij}$$\n\tbeing linear on $H$.\n\tThis also implies that $\\delta_{\\mathbb{S}_+^n}$ is generalized twice differentiable at $X^k$ for $Y^k$.\n","start":42048,"end":42907},{"raw_tex":"\t\nLet $f_1^k(H)=\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}\\big(\\widetilde{H}_{ij}\\big)^2$, $f_2^k(H)=\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$ and $f_3^k(H)=\\sum_{i\\in\\beta,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(X^k)}\\big(\\widetilde{H}_{ij}\\big)^2$. By the definition of continuous convergence mentioned in \\cite[page 250]{RoWe98}, we know that $f_1^k$ converges continuously to $f_1$ with\n$$f_1(H)=\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}\\big(\\widetilde{H}_{ij}\\big)^2.$$\n It follows from \\cite[Theorem 7.11]{RoWe98} that $f_1^k$  epi-converges to $f_1$. It can be checked easily that $f_1^k$ also  pointwise converges \\cite[page 239]{RoWe98} to $f_1$.\n","start":42907,"end":43612},{"raw_tex":"Since ${\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)$ is a constant closed set, we know  that $f_2^k$ pointwise converges and epi-converges \\cite[Proposition 4.4]{RoWe98} to $f_2$ with $f_2(H)=\\delta_{{\\cal C}_{\\mathbb{S}_+^n}(X^k,Y^k)}(H)$. It follows from \\cite[Theorem 7.46(a)]{RoWe98} that $f_1^k+f_2^k$ epi-convergences to $f_1+f_2$. Also, it can be checked directly that $f_1^k+f_2^k$ pointwise convergences to $f_1+f_2$. By the construction of $z^k$, the sequence of $\\{f_3^k\\}$ is nondecreasing ($f_3^k\\leq f_3^{k+1}$). We know from \\cite[Proposition 7.4]{RoWe98} that $f_3^k$ epi-converges to $\\sup_k\\{\\mbox{cl}f_3^k\\}$, where $\\mbox{cl}f_3^k$ is the closure of $f_3^k$. It is easy to see that $\\sup_k\\{\\mbox{cl}f_3^k\\}=\\sup_k \\{f_3^k\\}=\\delta_{\\cal V}$ with ${\\cal V}=\\{H\\in\\mathbb{S}^n\\mid\\widetilde{H}_{\\beta\\gamma}=0\\}$ and $f_3^k$ pointwise converges to $\\delta_{\\cal V}$. Since $f_1^k+f_2^k$ pointwise and epi-converges to $f_1+f_2$, by using \\cite[Theorem 7.46(a)]{RoWe98} again, we obtain that $f_1^k+f_2^k+f_3^k$ epi-converges to $f_1+f_2+f_3$.\n","start":43612,"end":44664},{"raw_tex":"\nCombining the above discussion with \\eqref{eq:affcsdp}, the generalized quadratic form $\\frac{1}{2}d^2\\delta_{\\mathbb{S}_+^n}(X^k\\mid Y^k)(H)$ converges epigraphically to generalized quadratic form\n\t$$\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j(A)}{\\lambda_i(A)}(\\widetilde{H}_{ij})^2+\\delta_{{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y})}(H). $$\n\tThus we have verified this proposition.\n\\end{proof}","start":44664,"end":45081}]},{"id":"S044","semantic_type":"theorem","kind":"theorem","labels":["eq:psdofpb","thm:varscequissosc"],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"强变分充分性的三重等价","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm:varscequissosc}\n\tLet $\\bar{x}\\in\\mathbb{X}$ be a stationary point to the NLSDP \\eqref{eq:NLSDP} and $\\overline{Y}\\in {\\cal M}(\\bar{x})$. Then the following three conditions are equivalent\n\t\\begin{itemize}\n\t\t\\item[(i)] the strong variational sufficient condition with respect to $(\\bar{x},\\overline{Y})$ holds;\n\t\t\\item[(ii)] the strong second order sufficient condition (SOSC) \\eqref{eq:defssosc} holds at $(\\bar{x},\\overline{Y})$;\n\t\t\\item[(iii)] there exist $\\rho_0>0$ and $\\underline{\\eta}>0$ such that for any $\\rho\\geq \\rho_0$ and any $W\\in\\partial_B\\Pi_{\\mathbb{S}_-^n}(G(\\bar{x})+\\rho^{-1}\\overline{Y})$,\n\t\t\\begin{equation*}\\label{eq:psdofpb}\n\t\t\\langle d, {\\cal A}_{\\rho}(\\overline{Y},W)d\\rangle\\geq\\underline{\\eta}\\|d\\|^2\\quad\\forall d\\in\\mathbb{X},\n\t\t\\end{equation*}\n\t\twhere ${\\cal A}_{\\rho}(\\overline{Y},W)$ is defined by \\eqref{eq:algrad2}.\n\t\\end{itemize}\n\\end{theorem}","start":45269,"end":46173}]},{"id":"S045","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"三重等价的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\t``$(i)\\Longrightarrow(ii)$\":  This direction can be obtained directly from Proposition \\ref{prop:onesidessosc} and Proposition \\ref{prop:thm5} as we only need to substitute \\eqref{eq:qeqsg} into \\eqref{svs-equ}.\n\t\n\t``$(ii)\\Longrightarrow(iii)$\": This proof sketch is similar to \\cite[Proposition 4]{SSZhang08} although they require the validity of nondegeneracy, which is superfluous here.\n\tIt follows from Definition \\ref{eq:defssosc} that there exists $\\eta_0>0$ such that\n\t$$\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)\\geq\\eta_0\\|d\\|^2\\quad\\forall\\; G'(\\bar{x})d\\in{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y}).$$\n\tBy using \\cite[Lemma 7]{SSZhang08}, there exist two positive numbers $\\rho_1$ and $\\underline{\\eta}\\in(0,\\eta_0/2]$ such that for any $\\rho'\\geq \\rho_1$,\n\t\\begin{align*}\n\t\t&\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)\\\\\n\t\t&+\\rho'\\|\\overline{P}^T_{\\gamma}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2+\\rho'\\|\\overline{P}^T_{\\beta}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2\\geq2\\underline{\\eta}\\|d\\|^2\\quad \\forall d\\in\\mathbb{X}.\n\t\\end{align*}\n\tSuppose a sufficient large $\\rho_0\\geq\\rho_1$. For any $\\rho\\geq\\rho_0$ and $d\\in{\\cal X}$, we have\n\t{\\small \\begin{align*}\n\t\t\t&-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)-2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\\\\n\t\t\t&=2\\sum\\limits_{i\\in\\alpha,\\,j\\in\\gamma}\\frac{-\\lambda_j}{\\lambda_i}(\\overline{P}^T(G'(\\bar{x}){d})\\overline{P})_{ij}^2-2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\\\\n\t\t\t&=2\\sum\\limits_{i\\in\\alpha,\\,j\\in\\gamma}\\frac{\\lambda_j^2}{\\lambda_i(\\rho\\lambda_i-\\lambda_j)}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\leq\\underline{\\eta}\\|d\\|^2,\n\t\\end{align*}}\n\twhere the last inequality can be obtained by sufficiently large $\\rho$.\n\tCombining the above two relations together, we have for any $\\rho\\geq\\rho_0$,  $\\rho'\\geq\\rho_0\\geq\\rho_1$,\n\t\\begin{align*}\n\t\t&\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle+2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\\\\n\t\t&+\\rho'\\|\\overline{P}^T_{\\gamma}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2+\\rho'\\|\\overline{P}^T_{\\beta}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2\\geq\\underline{\\eta}\\|d\\|^2.\n\t\\end{align*}\n\tThen it can be checked directly that for all $d\\in\\mathbb{X}$ and $\\rho\\geq\\rho_0$,  $\\langle d,{\\cal A}_{\\rho}(\\overline{Y},W)d\\rangle\\geq\\underline{\\eta}\\|d\\|^2 $.\n\t\n\t\n\t\n\t``$(iii)\\Longrightarrow(i)$\": It follows directly from \\cite[Theorem 3]{Roh22} as ${\\cal A}_{\\rho}(\\overline{Y},W)$ coincides with \\cite[(3.6)]{Roh22}.\n\\end{proof}","start":46174,"end":49048}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"二次束代入推出强二阶条件","source_units":[{"raw_tex":"\\begin{proof}\n\t``$(i)\\Longrightarrow(ii)$\":  This direction can be obtained directly from Proposition \\ref{prop:onesidessosc} and Proposition \\ref{prop:thm5} as we only need to substitute \\eqref{eq:qeqsg} into \\eqref{svs-equ}.\n","start":46174,"end":46401}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"声称获得统一曲率下界","source_units":[{"raw_tex":"\t\n\t``$(ii)\\Longrightarrow(iii)$\": This proof sketch is similar to \\cite[Proposition 4]{SSZhang08} although they require the validity of nondegeneracy, which is superfluous here.\n\tIt follows from Definition \\ref{eq:defssosc} that there exists $\\eta_0>0$ such that\n\t$$\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)\\geq\\eta_0\\|d\\|^2\\quad\\forall\\; G'(\\bar{x})d\\in{\\rm aff}\\,{\\cal C}_{\\mathbb{S}_+^n}(G(\\bar{x}),\\overline{Y}).$$\n","start":46401,"end":46889}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"块罚项扩展为全空间下界","source_units":[{"raw_tex":"\tBy using \\cite[Lemma 7]{SSZhang08}, there exist two positive numbers $\\rho_1$ and $\\underline{\\eta}\\in(0,\\eta_0/2]$ such that for any $\\rho'\\geq \\rho_1$,\n\t\\begin{align*}\n\t\t&\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)\\\\\n\t\t&+\\rho'\\|\\overline{P}^T_{\\gamma}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2+\\rho'\\|\\overline{P}^T_{\\beta}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2\\geq2\\underline{\\eta}\\|d\\|^2\\quad \\forall d\\in\\mathbb{X}.\n\t\\end{align*}\n","start":46889,"end":47390}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"控制谱系数替换误差","source_units":[{"raw_tex":"\tSuppose a sufficient large $\\rho_0\\geq\\rho_1$. For any $\\rho\\geq\\rho_0$ and $d\\in{\\cal X}$, we have\n\t{\\small \\begin{align*}\n\t\t\t&-\\varUpsilon_{G(\\bar{x})}\\big(\\overline{Y},G'(\\bar{x}){d}\\big)-2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\\\\n\t\t\t&=2\\sum\\limits_{i\\in\\alpha,\\,j\\in\\gamma}\\frac{-\\lambda_j}{\\lambda_i}(\\overline{P}^T(G'(\\bar{x}){d})\\overline{P})_{ij}^2-2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\\\\n\t\t\t&=2\\sum\\limits_{i\\in\\alpha,\\,j\\in\\gamma}\\frac{\\lambda_j^2}{\\lambda_i(\\rho\\lambda_i-\\lambda_j)}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\leq\\underline{\\eta}\\|d\\|^2,\n\t\\end{align*}}\n\twhere the last inequality can be obtained by sufficiently large $\\rho$.\n","start":47390,"end":48222}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"合并罚项下界与谱误差","source_units":[{"raw_tex":"\tCombining the above two relations together, we have for any $\\rho\\geq\\rho_0$,  $\\rho'\\geq\\rho_0\\geq\\rho_1$,\n\t\\begin{align*}\n\t\t&\\langle L_{xx}''(\\bar{x},\\overline{Y})d,d\\rangle+2\\rho\\sum_{i\\in\\alpha,j\\in\\gamma}\\frac{-\\lambda_j}{\\rho\\lambda_i-\\lambda_j}(\\overline{P}^T(G'(\\bar{x})d)\\overline{P})_{ij}^2\\\\\n\t\t&+\\rho'\\|\\overline{P}^T_{\\gamma}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2+\\rho'\\|\\overline{P}^T_{\\beta}(G'(\\bar{x})d)\\overline{P}_{\\gamma}\\|^2\\geq\\underline{\\eta}\\|d\\|^2.\n\t\\end{align*}\n","start":48222,"end":48711}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"识别广义海森的一致正定性","source_units":[{"raw_tex":"\tThen it can be checked directly that for all $d\\in\\mathbb{X}$ and $\\rho\\geq\\rho_0$,  $\\langle d,{\\cal A}_{\\rho}(\\overline{Y},W)d\\rangle\\geq\\underline{\\eta}\\|d\\|^2 $.\n\t\n\t\n\t\n","start":48711,"end":48884}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"海森束刻画返回强变分充分性","source_units":[{"raw_tex":"\t``$(iii)\\Longrightarrow(i)$\": It follows directly from \\cite[Theorem 3]{Roh22} as ${\\cal A}_{\\rho}(\\overline{Y},W)$ coincides with \\cite[(3.6)]{Roh22}.\n\\end{proof}","start":48884,"end":49048}]},{"id":"S053","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S053"],"declared_external":true,"declaration_text":["原始前提列表：BShapiro00, Example 3.140；使用位置 L508。"],"source_only":false,"title":"半正定锥的二阶锥可约性","source_units":[]},{"id":"S054","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S054"],"declared_external":true,"declaration_text":["原始前提列表：mohammadi20, Theorem 3.3；使用位置 L508–L513。"],"source_only":false,"title":"可约锥指标的二阶次导数公式","source_units":[]},{"id":"S055","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S055"],"declared_external":true,"declaration_text":["原始前提列表：RoWe98, Theorem 7.11；文中同时引用 pages 250、239 的收敛定义。"],"source_only":false,"title":"连续收敛蕴含上图收敛","source_units":[]},{"id":"S056","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S056"],"declared_external":true,"declaration_text":["原始前提列表：RoWe98, Proposition 4.4；使用位置 L523。"],"source_only":false,"title":"闭集指标的上图收敛","source_units":[]},{"id":"S057","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S057"],"declared_external":true,"declaration_text":["原始前提列表：RoWe98, Proposition 7.4；使用位置 L523。"],"source_only":false,"title":"单调函数列的上图极限","source_units":[]},{"id":"S058","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S058"],"declared_external":true,"declaration_text":["原始前提列表：RoWe98, Theorem 7.46(a)；两次使用均见 L523。"],"source_only":false,"title":"函数和的上图收敛规则","source_units":[]},{"id":"S059","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S059"],"declared_external":true,"declaration_text":["原始前提列表：SSZhang08, Lemma 7。"],"source_only":false,"title":"二次罚项的强制正定化","source_units":[]},{"id":"S060","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S060"],"declared_external":true,"declaration_text":["原始前提列表：Roh22, Theorem 3 and equation (3.6)；使用位置 L582。"],"source_only":false,"title":"海森束刻画强变分充分性","source_units":[]},{"id":"S061","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S061"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；对应原始前提列表中的有限维紧性背景。"],"source_only":false,"title":"单位球面上的统一正性背景","source_units":[]},{"id":"S062","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S062"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；对应原始前提列表中的标准有限维分析。"],"source_only":false,"title":"有限谱和的统一二次估计背景","source_units":[]}],"collapse_map":{"S001":"S001","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S034","S036":"S034","S037":"S034","S038":"S034","S039":"S034","S040":"S034","S041":"S034","S042":"S034","S043":"S034","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062"}}}}
{"record_id":"2209.13326v2","paper":{"title":"Exact augmented Lagrangian duality for mixed integer convex optimization","authors":["Bhardwaj, Avinash","Narayanan, Vishnu","Pathapati, Abhishek"],"arxiv_id":"2209.13326","version":"v2","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/2209.13326v2","pdf_url":"https://arxiv.org/pdf/2209.13326v2","doi":"10.1137/22M1526204"},"context":{"raw_tex":"Consider, to begin with, the following mixed integer convex programming (MICP) problem,\n\\begin{optprog*}\n    {$\\zIP = $ minimize}  & \\objective{\\hspace{3mm}f(\\x)}\\\\\n   \\textbf{(MICP)} \\qquad subject to & \\A\\,\\x\\,= \\b\\\\\n     & \\x \\in X \\hspace{1mm}\n\\end{optprog*}\nwhere $X:= \\set{\\x\\in \\Z^{n_1}\\times\\R^{n_2}}{\\E\\x\\leq \\f}$ is a polyhedral mixed integer set, $f:\\R^n \\mapsto \\R$ is a real valued convex function, $\\A,\\E$ and $\\b,\\f$ are full rank matrices and vectors of appropriate dimensions, respectively. Through out the remainder of this discussion we assume that the matrices $\\A, \\E$ are rational matrices and $\\b, \\f$ are rational vectors and $f(\\cdot)$ is differentiable. We further assume that both MICP and MICP's continuous relaxation are feasible and the corresponding optimal solutions exist.\\\\\n\nConsider the Lagrangian relaxation of the MICP,\n\\begin{equation*}\n    \\textbf{(LR)} \\hspace{1cm} z^{LR}(\\blambda) = \\min_{\\x \\in X} f(\\x) + {\\blambda}^{\\top}(\\b - \\A\\,\\x).\n\\end{equation*}\nThe corresponding augmented Lagrangian relaxation and augmented Lagrangian dual (\\cite{boland2015augmented,gu2020exact}) of the \\MICP\\, are defined as,\n\\begin{align*}\n    \\textbf{(ALR)} \\hspace{0.5cm} z_{\\rho}^{LR+}(\\blambda) & = \\min_{\\x \\in X} f(\\x) + {\\blambda}^{\\top}(\\b - \\A\\,\\x) + \\rho\\, \\psi(\\b - \\A\\,\\x)\\\\\n    \\textbf{(ALD)} \\hspace{1cm} z_{\\rho}^{LD+} & = \\sup_{\\blambda\\in \\R^m}\\min_{\\x \\in X} f(\\x) + {\\blambda}^{\\top}(\\b - \\A\\,\\x) + \\rho\\, \\psi(\\b - \\A\\,\\x)\n\\end{align*}\nwhere $\\psi:\\R^m \\mapsto \\R$ is a real valued function. We further designate $\\psi$ to have an \\textit{exact penalty representation} if $\\exists\\, 0 < \\rho < \\infty$, such that $z^{LD+}_{\\rho} = \\zIP$. In the case $\\psi(\\cdot) = \\norm{\\cdot}$ we augmented Lagrangian relaxation is referred to as sharp augmented Lagrangian relaxation. In particular, \n$$\n    \\textbf{(SALR)} \\hspace{1cm} z_{SALR}(\\rho) = \\min_{\\x \\in X} f(\\x) + \\rho\\, \\norm{\\b - \\A\\,\\x}\\\\\n$$\nIt follows that $z^{LR+}_{\\rho}(\\blambda) \\leq \\zIP$ and $z^{LR+}_{\\rho}(\\bz) = z_{SALR}(\\rho)$. \n\\subsection{Notation}\\label{sec:notation} The sets $\\R^n,\\Z^n,\\mathbb{Q}^n$ denote the set of real numbers, integers and rational numbers in $n-$dimensional vector space, respectively. Additionally, $\\R^n_+,\\Z^n_+,\\mathbb{Q}^n_+$ denote the non-negative counterparts of the respective sets. We use $[n]$ to denote the index set $\\{1,2 \\ldots, n\\}$. For an index set $E \\subset \\Z^+$ and $y\\in \\Z$, we define the translation of an index set as $y + E \\define \\set{x + y}{x \\in E}$. For a mixed integer set $S \\subseteq \\R^n \\times \\Z^m$, we denote by $S_R$ the continuous relaxation of $S$. Throughout the following discussion we define $I$ and $C$ as the index sets corresponding to integer and continuous variables, respectively. Analogously, for any $\\u \\in \\R^n$, let $\\u = (\\u_I,\\u_C)$ denote the partition of $\\u$ into the integer and continuous variables, respectively. Similarly for a symmetric matrix $\\Q \\in \\R^{n\\times n}$, let $Q_{UV}$ denote the submatrix of $\\Q$ formed by rows indexed by $U$ and columns indexed by $V$. For $\\delta > 0$ and $\\bar{\\x} \\in \\R^n$, we define by $\\mathcal{N}_{\\delta}(\\bar{\\x}) \\define \\set{\\x \\in \\R^n}{\\norm{\\x -\\bar{\\x}}\\leq \\delta}$, the ball with center $\\bar{\\x}$ and radius $\\delta$. The recession cone of a set $S$ is denoted by $\\rec(S)$. We also define, $\\rec(f)$, \\emph{recession cone of a convex function} $f:\\R^n \\mapsto \\R$ as the recession cone of a level set $\\set{x \\in \\R^n}{f(\\x)\\leq \\alpha}$, for some $\\alpha \\in \\R$ (\\cite{bertsekas2003convex}, Proposition 2.3.1). For given convex sets $A$ and $B$, we say that the sets $A$ and $B$ have \\emph{no common/distinct non-zero directions of recession} if $\\rec(A)\\cap \\rec(B)\\setminus \\{\\bz\\} = \\emptyset$.\\\\\n\n\nWe say that a function $g : \\R^m \\mapsto \\R$ is $L$ smooth, or alternatively, has \\textbf{Lipschitz-continuous} gradients if there exists a Lipschitz\nconstant, $L < \\infty$, such that\n$$\n    \\norm{\\nabla{g(\\x)} - \\nabla{g(y)}}\\leq\n    L\\norm{\\x-\\y}~~~~ \\forall~~ \\x,\\y \\in \\R^m.\n$$\nAdditionally, we say a function $g : \\R^m \\mapsto \\R$ is $\\mu$ \\textbf{strongly-convex} if there exists $ 0 <\\mu < \\infty$, such that\n$$\n    \\norm{\\nabla{g(\\x)} - \\nabla{g(y)}}\\geq\\mu\\norm{\\x-\\y}~~~~ \\forall~~ \\x,\\y \\in \\R^m.\n$$\n\n\nConsider the hyperplane $H_{\\u} \\define \\set{\\x\\in\\R^n}{\\A\\x = \\b + \\u}$, and let the corresponding value function associated with {\\MICP} be defined as \n\\begin{align}\n    \\phi(\\u) = \\min\\set{f(\\x)}{\\x \\in X\\cap\\Hu}.\\label{def:valuefunction}\n\\end{align}\nIn particular, $\\zIP = \\phi(\\bz) = \\min_{\\x\\in X \\cap \\Ho} f(\\x)$. Finally, we define the set $\\calU$ as the set of all possible perturbation vectors $\\u$ such that the feasible set of \\cref{def:valuefunction} is non-empty. In particular, $\\calU \\define \\set{\\u\\in \\R^m}{X \\cap \\Hu \\neq \\emptyset}$.\n\n\nConsider the partition of the feasible set $X = F_> \\cup F_{\\leq}, F_> \\cap F_{\\leq} = \\emptyset$ where $F_{\\leq} = \\set{\\x \\in X}{f(\\x) \\leq \\zIP}$ and $F_> =\\set{\\x \\in X}{f(\\x) > \\zIP}$. Observe that for $\\x \\in F_>$ any positive $\\rho > 0$ implies $f(\\x) + \\rho\\norm{\\b - \\A\\,\\x} > \\zIP$. \nThus, for MICP to have an exact penalty representation, it suffices to show that $\\exists$ $0 < \\rho < \\infty$ such that $f(\\x) + \\rho \\norm{\\b - \\A\\,\\x} \\geq \\zIP$, $\\forall \\x \\in F_{\\leq}$.\n\n\nConsider the following mixed integer convex programming problem.\n\\begin{optprog*}\n    {$\\hat{\\phi}(\\u) = $ minimize}  & \\objective{\\hspace{1.3cm}f({\\x_I},{\\x_C})}\\\\\n    $\\textup{\\MICP}_\\u$ \\qquad subject to & \\hspace{0cm} \\A_I\\,\\x_I + \\A_C\\x_C = \\b + \\u\\\\\n    & \\E_I\\x_I + \\E_C\\x_C \\leq \\f \\hspace{7mm}\\\\\n    & \\norm{\\x_I}_{\\infty}\\leq M \\hspace{0.5cm}\\\\\n    % & \\x_I\\in \\Z_{+}^{n_1}\\hspace{0.4cm}\\\\ \n    & (\\x_I, \\x_C) \\in \\Z^{n_1}\\times\\R^{n_2}\\hspace{0.3cm}\n\\end{optprog*}\nwhere $0 \\leq M < \\infty$. Denote by $F_{\\u} \\define X \\cap \\Hu \\cap S_I $ the feasible region of $\\textup{\\MICP}_\\u$, where the set $S_I = \\set{\\x_I \\in \\Z^{n_1}}{\\norm{\\x_I}_{\\infty} \\leq M}$ is finite.  As before, we let $S_I = \\{\\x^{(1)}_I, \\x_I^{(2)}, \\ldots, \\x_I^{(k)}\\}$. Using this enumeration we characterize the value functions of the continuous restrictions, parameterized in $\\x_I^{(i)},\\,i \\in [k]$ as,\n\\begin{optprog*}\n    {$\\Phi(\\u,\\x^{(i)}_I) = $ minimize}  & \\objective{\\hspace{6mm}f(\\x^{(i)}_I,\\x_C)}\\\\\n    $\\CPui$ \\qquad subject to & \\A_C\\,\\x_C = \\b + \\u - \\A_I\\x^{(i)}_{I}\\\\\n    & \\E_C\\x_C \\leq \\f - \\E_I\\x^{(i)}_{I}\\hspace{8mm}\\\\\n     & \\x_C \\in \\R^{n_2}\\hspace{1.8cm}\n\\end{optprog*}\nwhere $\\Phi(\\u,\\x^{(i)}_I)$ is the value function of $\\CPui$, a continuous convex  optimization problem. Since the constraints of $\\CPui$ are linear, Slater's condition holds and therefore the value function is lower semi-continuous.\n","units":[{"raw_tex":"Consider, to begin with, the following mixed integer convex programming (MICP) problem,\n\\begin{optprog*}\n    {$\\zIP = $ minimize}  & \\objective{\\hspace{3mm}f(\\x)}\\\\\n   \\textbf{(MICP)} \\qquad subject to & \\A\\,\\x\\,= \\b\\\\\n     & \\x \\in X \\hspace{1mm}\n\\end{optprog*}\nwhere $X:= \\set{\\x\\in \\Z^{n_1}\\times\\R^{n_2}}{\\E\\x\\leq \\f}$ is a polyhedral mixed integer set, $f:\\R^n \\mapsto \\R$ is a real valued convex function, $\\A,\\E$ and $\\b,\\f$ are full rank matrices and vectors of appropriate dimensions, respectively. Through out the remainder of this discussion we assume that the matrices $\\A, \\E$ are rational matrices and $\\b, \\f$ are rational vectors and $f(\\cdot)$ is differentiable. We further assume that both MICP and MICP's continuous relaxation are feasible and the corresponding optimal solutions exist.\\\\\n\nConsider the Lagrangian relaxation of the MICP,\n\\begin{equation*}\n    \\textbf{(LR)} \\hspace{1cm} z^{LR}(\\blambda) = \\min_{\\x \\in X} f(\\x) + {\\blambda}^{\\top}(\\b - \\A\\,\\x).\n\\end{equation*}\nThe corresponding augmented Lagrangian relaxation and augmented Lagrangian dual (\\cite{boland2015augmented,gu2020exact}) of the \\MICP\\, are defined as,\n\\begin{align*}\n    \\textbf{(ALR)} \\hspace{0.5cm} z_{\\rho}^{LR+}(\\blambda) & = \\min_{\\x \\in X} f(\\x) + {\\blambda}^{\\top}(\\b - \\A\\,\\x) + \\rho\\, \\psi(\\b - \\A\\,\\x)\\\\\n    \\textbf{(ALD)} \\hspace{1cm} z_{\\rho}^{LD+} & = \\sup_{\\blambda\\in \\R^m}\\min_{\\x \\in X} f(\\x) + {\\blambda}^{\\top}(\\b - \\A\\,\\x) + \\rho\\, \\psi(\\b - \\A\\,\\x)\n\\end{align*}\nwhere $\\psi:\\R^m \\mapsto \\R$ is a real valued function. We further designate $\\psi$ to have an \\textit{exact penalty representation} if $\\exists\\, 0 < \\rho < \\infty$, such that $z^{LD+}_{\\rho} = \\zIP$. In the case $\\psi(\\cdot) = \\norm{\\cdot}$ we augmented Lagrangian relaxation is referred to as sharp augmented Lagrangian relaxation. In particular, \n$$\n    \\textbf{(SALR)} \\hspace{1cm} z_{SALR}(\\rho) = \\min_{\\x \\in X} f(\\x) + \\rho\\, \\norm{\\b - \\A\\,\\x}\\\\\n$$\nIt follows that $z^{LR+}_{\\rho}(\\blambda) \\leq \\zIP$ and $z^{LR+}_{\\rho}(\\bz) = z_{SALR}(\\rho)$. \n\\subsection{Notation}\\label{sec:notation} The sets $\\R^n,\\Z^n,\\mathbb{Q}^n$ denote the set of real numbers, integers and rational numbers in $n-$dimensional vector space, respectively. Additionally, $\\R^n_+,\\Z^n_+,\\mathbb{Q}^n_+$ denote the non-negative counterparts of the respective sets. We use $[n]$ to denote the index set $\\{1,2 \\ldots, n\\}$. For an index set $E \\subset \\Z^+$ and $y\\in \\Z$, we define the translation of an index set as $y + E \\define \\set{x + y}{x \\in E}$. For a mixed integer set $S \\subseteq \\R^n \\times \\Z^m$, we denote by $S_R$ the continuous relaxation of $S$. Throughout the following discussion we define $I$ and $C$ as the index sets corresponding to integer and continuous variables, respectively. Analogously, for any $\\u \\in \\R^n$, let $\\u = (\\u_I,\\u_C)$ denote the partition of $\\u$ into the integer and continuous variables, respectively. Similarly for a symmetric matrix $\\Q \\in \\R^{n\\times n}$, let $Q_{UV}$ denote the submatrix of $\\Q$ formed by rows indexed by $U$ and columns indexed by $V$. For $\\delta > 0$ and $\\bar{\\x} \\in \\R^n$, we define by $\\mathcal{N}_{\\delta}(\\bar{\\x}) \\define \\set{\\x \\in \\R^n}{\\norm{\\x -\\bar{\\x}}\\leq \\delta}$, the ball with center $\\bar{\\x}$ and radius $\\delta$. The recession cone of a set $S$ is denoted by $\\rec(S)$. We also define, $\\rec(f)$, \\emph{recession cone of a convex function} $f:\\R^n \\mapsto \\R$ as the recession cone of a level set $\\set{x \\in \\R^n}{f(\\x)\\leq \\alpha}$, for some $\\alpha \\in \\R$ (\\cite{bertsekas2003convex}, Proposition 2.3.1). For given convex sets $A$ and $B$, we say that the sets $A$ and $B$ have \\emph{no common/distinct non-zero directions of recession} if $\\rec(A)\\cap \\rec(B)\\setminus \\{\\bz\\} = \\emptyset$.\\\\\n","start":10244,"end":14000},{"raw_tex":"We say that a function $g : \\R^m \\mapsto \\R$ is $L$ smooth, or alternatively, has \\textbf{Lipschitz-continuous} gradients if there exists a Lipschitz\nconstant, $L < \\infty$, such that\n$$\n    \\norm{\\nabla{g(\\x)} - \\nabla{g(y)}}\\leq\n    L\\norm{\\x-\\y}~~~~ \\forall~~ \\x,\\y \\in \\R^m.\n$$\nAdditionally, we say a function $g : \\R^m \\mapsto \\R$ is $\\mu$ \\textbf{strongly-convex} if there exists $ 0 <\\mu < \\infty$, such that\n$$\n    \\norm{\\nabla{g(\\x)} - \\nabla{g(y)}}\\geq\\mu\\norm{\\x-\\y}~~~~ \\forall~~ \\x,\\y \\in \\R^m.\n$$\n","start":14001,"end":14512},{"raw_tex":"Consider the hyperplane $H_{\\u} \\define \\set{\\x\\in\\R^n}{\\A\\x = \\b + \\u}$, and let the corresponding value function associated with {\\MICP} be defined as \n\\begin{align}\n    \\phi(\\u) = \\min\\set{f(\\x)}{\\x \\in X\\cap\\Hu}.\\label{def:valuefunction}\n\\end{align}\nIn particular, $\\zIP = \\phi(\\bz) = \\min_{\\x\\in X \\cap \\Ho} f(\\x)$. Finally, we define the set $\\calU$ as the set of all possible perturbation vectors $\\u$ such that the feasible set of \\cref{def:valuefunction} is non-empty. In particular, $\\calU \\define \\set{\\u\\in \\R^m}{X \\cap \\Hu \\neq \\emptyset}$.\n","start":14922,"end":15476},{"raw_tex":"Consider the partition of the feasible set $X = F_> \\cup F_{\\leq}, F_> \\cap F_{\\leq} = \\emptyset$ where $F_{\\leq} = \\set{\\x \\in X}{f(\\x) \\leq \\zIP}$ and $F_> =\\set{\\x \\in X}{f(\\x) > \\zIP}$. Observe that for $\\x \\in F_>$ any positive $\\rho > 0$ implies $f(\\x) + \\rho\\norm{\\b - \\A\\,\\x} > \\zIP$. \nThus, for MICP to have an exact penalty representation, it suffices to show that $\\exists$ $0 < \\rho < \\infty$ such that $f(\\x) + \\rho \\norm{\\b - \\A\\,\\x} \\geq \\zIP$, $\\forall \\x \\in F_{\\leq}$.\n","start":22903,"end":23390},{"raw_tex":"Consider the following mixed integer convex programming problem.\n\\begin{optprog*}\n    {$\\hat{\\phi}(\\u) = $ minimize}  & \\objective{\\hspace{1.3cm}f({\\x_I},{\\x_C})}\\\\\n    $\\textup{\\MICP}_\\u$ \\qquad subject to & \\hspace{0cm} \\A_I\\,\\x_I + \\A_C\\x_C = \\b + \\u\\\\\n    & \\E_I\\x_I + \\E_C\\x_C \\leq \\f \\hspace{7mm}\\\\\n    & \\norm{\\x_I}_{\\infty}\\leq M \\hspace{0.5cm}\\\\\n    % & \\x_I\\in \\Z_{+}^{n_1}\\hspace{0.4cm}\\\\ \n    & (\\x_I, \\x_C) \\in \\Z^{n_1}\\times\\R^{n_2}\\hspace{0.3cm}\n\\end{optprog*}\nwhere $0 \\leq M < \\infty$. Denote by $F_{\\u} \\define X \\cap \\Hu \\cap S_I $ the feasible region of $\\textup{\\MICP}_\\u$, where the set $S_I = \\set{\\x_I \\in \\Z^{n_1}}{\\norm{\\x_I}_{\\infty} \\leq M}$ is finite.  As before, we let $S_I = \\{\\x^{(1)}_I, \\x_I^{(2)}, \\ldots, \\x_I^{(k)}\\}$. Using this enumeration we characterize the value functions of the continuous restrictions, parameterized in $\\x_I^{(i)},\\,i \\in [k]$ as,\n\\begin{optprog*}\n    {$\\Phi(\\u,\\x^{(i)}_I) = $ minimize}  & \\objective{\\hspace{6mm}f(\\x^{(i)}_I,\\x_C)}\\\\\n    $\\CPui$ \\qquad subject to & \\A_C\\,\\x_C = \\b + \\u - \\A_I\\x^{(i)}_{I}\\\\\n    & \\E_C\\x_C \\leq \\f - \\E_I\\x^{(i)}_{I}\\hspace{8mm}\\\\\n     & \\x_C \\in \\R^{n_2}\\hspace{1.8cm}\n\\end{optprog*}\nwhere $\\Phi(\\u,\\x^{(i)}_I)$ is the value function of $\\CPui$, a continuous convex  optimization problem. Since the constraints of $\\CPui$ are linear, Slater's condition holds and therefore the value function is lower semi-continuous.\n","start":54192,"end":55608}]},"statement":{"raw_tex":"\\begin{theorem}\\label{thm:3}\nConsider the following mixed integer convex programming problem,\n\\begin{optprog*}\n    minimize  & \\objective{\\hspace{1cm}f({\\x_I},{\\x_C})}\\\\\n    \\textup{\\MICP} \\hspace{2mm} subject to & \\hspace{0cm} \\A_I\\,\\x_I + \\A_C\\x_C = \\b\\hspace{6.5mm}\\\\\n    & \\E_I\\x_I + \\E_C\\x_C \\leq \\f \\hspace{7mm}\\\\\n    % & \\norm{\\x_I}_{\\infty}\\leq M \\hspace{0.5cm}\\\\\n    % & \\x_I\\in \\Z_{+}^{n_1}\\hspace{0.4cm}\\\\ \n     & (\\x_I, \\x_C) \\in \\Z^{n_1}\\times\\R^{n_2}\\hspace{0.3cm}\n\\end{optprog*}\nLet $F = X\\cap \\Ho$ denote the feasible region of \\textup{\\MICP}.\\\\\n\\begin{enumerate}[label = (\\alph*)]\n    \\item If $\\norm{\\x_I}_{\\infty}\\leq M,$ $\\forall\\,\\x \\in F$ then there exists an exact penalty representation for \\textup{\\MICP}.\n    \\item If the recession cone of $f$ and recession cone of $F_R$, the feasible set of continuous relaxation of \\textup{\\MICP}, have no common non-zero directions of recession, then there exists an exact penalty representation for \\textup{\\MICP}.\n\\end{enumerate} \nAdditionally, if $f$ is $\\mu$-strongly convex and $L$ smooth, then the finite penalty parameter $\\rho = \\mathcal{O}\\left(\\frac{L\\beta\\gamma}{\\mu}\\right)$ where $\\displaystyle \\beta \\define \\max_{B \\in \\mathcal{B}}\\norm{B^{-1}}_F$, $\\mathcal{B}$ being the set of all possible invertible submatrices of $[\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^T_C]$ and $\\gamma$ can be explicitly computed given $f(\\bz)$, $\\norm{\\nabla{f(\\bz)}}$ and $f(\\bar{\\x})$ for any $\\bar{\\x} \\in F$.\n% $\\gamma = \\sqrt{f(\\bar{\\x})}$ for any $\\bar{\\x} \\in F$.\\\\\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\\label{thm:3}\nConsider the following mixed integer convex programming problem,\n\\begin{optprog*}\n    minimize  & \\objective{\\hspace{1cm}f({\\x_I},{\\x_C})}\\\\\n    \\textup{\\MICP} \\hspace{2mm} subject to & \\hspace{0cm} \\A_I\\,\\x_I + \\A_C\\x_C = \\b\\hspace{6.5mm}\\\\\n    & \\E_I\\x_I + \\E_C\\x_C \\leq \\f \\hspace{7mm}\\\\\n    % & \\norm{\\x_I}_{\\infty}\\leq M \\hspace{0.5cm}\\\\\n    % & \\x_I\\in \\Z_{+}^{n_1}\\hspace{0.4cm}\\\\ \n     & (\\x_I, \\x_C) \\in \\Z^{n_1}\\times\\R^{n_2}\\hspace{0.3cm}\n\\end{optprog*}\nLet $F = X\\cap \\Ho$ denote the feasible region of \\textup{\\MICP}.\\\\\n\\begin{enumerate}[label = (\\alph*)]\n    \\item If $\\norm{\\x_I}_{\\infty}\\leq M,$ $\\forall\\,\\x \\in F$ then there exists an exact penalty representation for \\textup{\\MICP}.\n    \\item If the recession cone of $f$ and recession cone of $F_R$, the feasible set of continuous relaxation of \\textup{\\MICP}, have no common non-zero directions of recession, then there exists an exact penalty representation for \\textup{\\MICP}.\n\\end{enumerate} \nAdditionally, if $f$ is $\\mu$-strongly convex and $L$ smooth, then the finite penalty parameter $\\rho = \\mathcal{O}\\left(\\frac{L\\beta\\gamma}{\\mu}\\right)$ where $\\displaystyle \\beta \\define \\max_{B \\in \\mathcal{B}}\\norm{B^{-1}}_F$, $\\mathcal{B}$ being the set of all possible invertible submatrices of $[\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^T_C]$ and $\\gamma$ can be explicitly computed given $f(\\bz)$, $\\norm{\\nabla{f(\\bz)}}$ and $f(\\bar{\\x})$ for any $\\bar{\\x} \\in F$.\n% $\\gamma = \\sqrt{f(\\bar{\\x})}$ for any $\\bar{\\x} \\in F$.\\\\\n\\end{theorem}","start":19365,"end":20905}]},"proof":{"raw_tex":"\\begin{lemma}\\label{lem:1}\n$\\displaystyle z_{SALR}(\\rho) = \\min_{\\u \\in \\calU}\\,\\, \\phi(\\u) + \\rho\\norm{\\u}$ . \n\\end{lemma}\n\n\\begin{proof}\nObserve the following inequality which follows immediately from the aforementioned definitions\n\\begin{align}\n    z_{SALR}(\\rho) = \\min_{\\x \\in X} f(\\x) + \\rho\\, \\norm{\\b - \\A\\,\\x} & \\leq \\min_{\\x\\in X \\cap \\Hu} f(\\x) + \\rho\\norm{\\u} ,\\,\\,\\,\\,\\, \\forall\\,\\u \\in \\calU \\nonumber\\\\\n                    & = \\phi(\\u) + \\rho\\norm{\\u} ,\\,\\,\\,\\,\\, \\forall\\,\\u \\in \\calU. \\label{eq:1}\n\\end{align}\nConversely, it holds that $\\forall\\,\\u \\in \\calU$\n\\begin{align*}\n    \\phi(\\u) + \\rho\\norm{\\u} & = \\min_{\\x\\in X \\cap \\Hu} f(\\x) + \\rho\\norm{\\u}\\\\\n                             & \\leq f(\\x) + \\rho\\norm{\\A\\x-\\b} ,\\,\\,\\,\\,\\, \\forall\\,\\x\\in X\\cap \\Hu.\n\\end{align*}\nThis implies that $ \\min_{\\u \\in \\calU} \\phi(\\u) + \\rho \\norm{\\u} \\leq f(\\x) + \\rho \\norm{\\A\\x - \\b} $ $\\forall~ x\\in X \\cap \\Hu,\\,\\forall~\\u \\in U$ which yields that $ \\min_{\\u \\in \\calU} \\phi(\\u) + \\rho \\norm{\\u} \\leq f(\\x) + \\rho \\norm{\\A\\x - \\b} $ $\\forall~ \\x \\in X $. This implies,\n\\begin{align}\n    \\min_{\\u\\in \\calU} \\phi(\\u) + \\rho\\norm{\\u} \\leq \\min_{\\x\\in X} f(\\x) + \\rho\\norm{\\A\\x-\\b}. \\label{eq:2}\n\\end{align}\nThe result follows from \\cref{eq:1} and \\cref{eq:2}.\n\\end{proof}\n\n\\begin{proposition}\\label{prop:2}\nConsider the continuous relaxation of MICP, and let the optimal objective value $z_R = \\min_{X_R\\cap \\Ho} f(\\x)$ be attained at $\\x_R$. \nLet $\\blambda_{\\A}$ and $\\blambda_{\\E}$ be the Lagrange multipliers for the constraints $\\A\\x = \\b$ and $\\E\\x\\leq \\f$, respectively.\\vspace{1mm}\n\\begin{enumerate}[label = \\roman*)]\n    \\item $\\phi(\\u) > -\\infty$ for all $\\u \\in \\calU$.\\vspace{1mm}\n    \\item If $\\exists ~ \\boldsymbol{\\alpha} \\in X_R$ such that $f(\\boldsymbol{\\alpha}) + \\rho\\norm{\\b-\\A\\,\\boldsymbol{\\alpha}}\\leq \\zIP$ and $\\rho > \\norm{\\blambda_\\A}$  then \n        \\[\n                \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq \\frac{\\zIP-z_R}{\\rho -\\norm{\\blambda_{\\A}}}.\n        \\]    \n\\end{enumerate}\n\\end{proposition}\n\n\\begin{proof}\nConsider the Lagrangian function for the continuous relaxation of MICP, i.e. for $\\x \\in \\R^n$\n\\begin{equation*}\n    \\mathcal{L}(\\x,\\blambda_{\\A},\\blambda_{\\E}) = f(\\x) - \\blambda_\\A^{\\top}(\\A\\,\\x-\\b)-\\blambda_\\E^{\\top}(\\f-\\E\\x)\n\\end{equation*}\nAs the relaxed problem has only affine constraints strong duality holds (Chapter 5,Section 5.2.3\\cite{boyd2004convex}).\n\n As strong duality holds, the first order necessary (KKT) conditions for $\\mathcal{L}(\\x,\\blambda_{\\A},\\blambda_{\\E})$ can be characterized as\n\\begin{align}\\label{eq:4}\n\\begin{split}\n    \\nabla{f(\\x)} = \\A^{\\top}\\blambda_\\A - \\E^{\\top}{\\blambda_\\E}\\\\\n    \\blambda_\\E^{\\top}(\\f - \\E\\x) = 0\\\\\n    \\blambda_\\A^{\\top}(\\A\\,\\x - \\b) = 0\\\\\n    \\blambda_\\E \\geq \\bz\n\\end{split}\n\\end{align}\nObserve that $\\x_R$ satisfies the system of equations \\cref{eq:4}.\\\\\n\\begin{enumerate}[label = \\roman*)]\n    \\item To see the result, observe that it suffices to show that $\\phi(\\u) > -\\infty$ for all $\\u \\in \\calU$ such that $\\phi(\\u) \\leq \\phi(\\bz) = \\zIP$. Further, consider the set $F_{\\leq}$ defined as,\n            $$\n                F_{\\leq} \\define \\set{\\x \\in X}{f(\\x) \\leq \\zIP}.\n            $$\n            We have for all $\\x \\in F_{\\leq}$, $\\zIP \\geq f(\\x) \\geq f(\\x_R) + \\nabla{f(\\x_R)}(\\x-\\x_R)= z_R + \\nabla{f(\\x_R)}(\\x-\\x_R)$. Substituting from \\cref{ineq:gradient} we obtain, for all $\\x \\in F_{\\leq}$,\n            $$\n                \\zIP \\geq f(\\x) \\geq z_R + \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b) -  \\blambda_\\E^{\\top} (\\E\\x -  \\f )\n            $$\n            As $\\blambda_{\\E} \\geq \\bz$ and $\\x \\in X_R$,\n            \\begin{align*}\n                \\zIP \\geq f(\\x) & \\geq z_R + \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b)\\\\\n                & \\geq z_R - \\norm{\\blambda_{\\A}}\\norm{(\\A\\,\\x - \\b)}\\\\\n                & = z_R - \\norm{\\blambda_{\\A}}\\norm{\\u} > -\\infty.\n            \\end{align*}\n            Since $f(\\x) > -\\infty$ for all $\\x \\in F_{\\leq}$, the result follows.\\\\\n    \\item Since we have for $\\boldsymbol{\\alpha}\\in X_R$, $f(\\boldsymbol{\\alpha}) + \\rho\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq  \\zIP$; convexity of $f$ yields,\n            \\begin{align}\n                f(\\x_R)+ \\nabla{f(\\x_R)}^{\\top}(\\boldsymbol{\\alpha}-\\x_R) +\\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} &\\leq \\zIP , \\text{ i.e.,}\\nonumber \\\\\n                z_R+ \\nabla{f(\\x_R)}^{\\top}(\\boldsymbol{\\alpha}-\\x_R) +\\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} &\\leq \\zIP. \\label{ineq:1}\n            \\end{align}\n            It follows from strong duality that\n            \\begin{align}\n            \\nabla{f(\\x_R)}^{\\top}(\\x-\\x_R) &= \\blambda_{\\A}^{\\top} \\A(\\x - \\x_R) -\\blambda_\\E^{\\top} \\E(\\x-\\x_R)\\nonumber\\\\\n             &= \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b) -  \\blambda_\\E^{\\top} (\\E\\x -  \\f ). \\label{ineq:gradient}\n            \\end{align}\n            Substituting in \\cref{ineq:1}, we obtain\n            \\begin{align*}\n            z_R+ \\blambda_{\\A}^{\\top} (\\A\\,\\boldsymbol{\\alpha} - \\b) +  \\blambda_\\E^{\\top} (\\f - \\E\\boldsymbol{\\alpha}) +\\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b}  & \\leq \\zIP .\n            \\end{align*}\n            As $\\blambda_{\\E} \\geq \\bz$ and $\\boldsymbol{\\alpha} \\in X_R$, the above inequality can be rewritten as\n            % \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b) +   \\blambda_\\E^{\\top} ( \\f - \\E\\x ) +\\rho \\norm{\\A\\,\\x - \\b} & \\leq \\zIP - z_R\\\\\n            \\begin{align}\n            \\blambda_{\\A}^{\\top}(\\A\\,\\boldsymbol{\\alpha} - \\b) + \\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} & \\leq \\zIP - z_R \\label{ineq:2}\n            \\end{align}\n            Using Cauchy Schwarz inequality in \\cref{ineq:2} yields for all $\\x \\in X_R$,\n            \\begin{align*}\n                -\\norm{\\blambda_\\A}\\,\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} + \\rho\\,\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq \\zIP - z_R\\\\\n            (\\rho - \\norm{\\blambda_\\A})\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b}\\leq \\zIP - z_R\\\\\n            \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq \\frac{\\zIP-z_R}{\\rho - \\norm{\\blambda_\\A}}.\n            \\end{align*}\n\\end{enumerate}\n\\end{proof}\n\n\\begin{corollary}\\label{cor:3}\nConsider the set $U_\\rho = \\set{\\u \\in \\calU}{\\phi(\\u) + \\rho\\norm{\\u} \\leq \\phi(\\bz)}$. If $\\rho > \\norm{\\blambda_{\\A}}$ then for all $\\u \\in U_\\rho$, $$\\norm{\\u} \\leq \\frac{\\zIP-z_R}{\\rho -\\norm{\\blambda_\\A}}.$$\n\\end{corollary}\n\n\\begin{proof}\nIf for any $~ \\u \\in U_{\\rho},~\\exists~ \\x \\in X\\cap H_{\\u}$ such that $f(\\x) + \\rho\\norm{\\A\\x - \\b} \\leq z_{IP}$, the result follows from \\Cref{prop:2}. Alternatively, if $f(\\x) + \\rho\\norm{\\A\\x - \\b} > z_{IP}\\,\\, \\forall~ \\x \\in X\\cap H_{\\u}$ for some $\\u \\in U_{\\rho}$. \n\\begin{align*}\n\\phi(\\u) + \\rho(\\norm{\\u}) =  \\inf_{X\\cap H_{\\u}} f(\\x) + \\rho\\norm{\\A\\x - \\b} \\geq z_{IP}\n\\end{align*} \nIf \\(\\inf_{X\\cap H_{\\u}} f(\\x) + \\rho\\norm{\\A\\x - \\b} > z_{IP}\\) then $\\u \\not\\in U_{\\rho}$ leading to a contradiction. If \\(\\inf_{X\\cap H_{\\u}} f(\\x) + \\rho\\norm{\\A\\x - \\b} = z_{IP}\\) and there exists an \\(\\x \\in X\\cap H_{\\u}\\) such that \\(f(\\x) + \\rho \\norm{\\A\\x - \\b} = z_{IP}\\) then the \nresult follows from \\cref{prop:2}. If there doesn't exist \\(\\x \\in X\\cap H_{\\u}\\) such that \\(f(\\x) + \\rho\\norm{\\A\\x - \\b}  = z_{IP}\\) then one can find a sequence of \\(\\x_{p}\\) such that the follows holds\n\\begin{align*}\n    f(\\x_{p}) + \\rho \\norm{\\A\\x_p - \\b} \\leq z_{IP} + \\frac{1}{p} \\text{ (From definition of infimum)}\\\\\n\\end{align*}\nLet \\(\\x_R\\) be the optimal solution of the optimization program \\(\\inf_{\\x \\in F_R} f(\\x)\\). Since we are minimizing a convex function over a rational polyhedron the KKT conditions hold as follows.\n\\begin{align*}   \n\\nabla{f(\\x_{R})} = \\A^{\\top}\\blambda_\\A - \\E^{\\top}{\\blambda_\\E}\\\\\n    \\blambda_\\E^{\\top}(\\f - \\E\\x_R) = 0\\\\\n    \\blambda_\\A^{\\top}(\\A\\,\\x_R - \\b) = 0\\\\\n    \\blambda_\\E \\geq \\bz\n\\end{align*}\n\\begin{align*} \n    f(\\x_R)+ \\nabla{f(\\x_R)}^{\\top}(\\x_{p}-\\x_R) +\\rho \\norm{\\A\\,\\x_p - \\b} &\\leq \\zIP + \\frac{1}{p}\\text{ (Convexity)}\\\\\n    z_{R} + \\blambda_{\\A}^{\\top} (\\A\\,\\x_p - \\b) +  \\blambda_\\E^{\\top} (\\f - \\E\\x_p) +\\rho \\norm{\\A\\,\\x_p - \\b} &\\leq \\zIP + \\frac{1}{p}\\\\\n    % (\\rho - \\norm{\\blambda_{\\A}}) \\norm{\\u} &\\leq \\frac{\\zIP - z_{R}}{\\rho - \\norm{\\blambda_\\A}} + \\frac{1}{p(\\rho - \\norm{\\blambda_{\\A}})}\\\\\n    \\norm{\\u} &\\leq \\frac{\\zIP - z_{R}}{\\rho - \\norm{\\blambda_{\\A}}} + \\frac{1}{p(\\rho - \\norm{\\blambda_{\\A}})}\n\\end{align*}\nwhere the last inequality follows from non-negativity of $\\blambda_\\E$ and $(\\f - \\E\\x_p).$ As \\(p\\) is arbitrary, we have as $p \\to \\infty$,\n\\begin{align*}\n    \\norm{\\u} &\\leq \\frac{\\zIP - z_{R}}{\\rho - \\norm{\\blambda_{\\A}}}\n\\end{align*}\n\\end{proof}\n\n\\begin{corollary}\\label{cor:4}\nConsider the set $\\,\\ocalU \\define \\set{\\u \\in \\calU}{\\phi(\\bz) \\geq \\phi(\\u)}$. For $\\alpha \\geq 0$, $\\phi(\\bz) \\leq \\phi(\\u) + \\alpha$ $\\forall \\,\\u \\in \\calU$ if $\\phi(\\bz) \\leq \\phi(\\u) + \\alpha$ $\\forall \\,\\u \\in \\ocalU$.\n\\end{corollary}\n\n\\begin{proof}\n    For $\\u \\in \\calU\\setminus\\ocalU$, $\\phi(\\bz) < \\phi(\\u) \\leq \\phi(\\u) + \\alpha$. The result follows.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:3}\nIf there exist $\\delta, \\kappa > 0$ such that  $\\phi(\\bz) \\leq \\phi(\\u) + \\kappa \\norm{\\u}$ for all $\\u \\in \\mathcal{N}_{\\delta}(\\bz) \\cap \\calU$,  then there exists $0 < \\rho^{*} < \\infty$ such that $z_{SALR}(\\rho) = \\zIP$ for all $\\rho > \\rho^{*}$.\n\\end{lemma}\n\n\\begin{proof}\nObserve that, for $\\rho > 0$ it suffices to consider $\\u \\in U_\\rho$. We have, from \\cref{lem:1} \n$$ z_{SALR}(\\rho) = \\min_{\\u\\in \\calU} \\phi(\\u) + \\rho \\norm{\\u}.$$\nIt follows that,\n\\begin{align*}\n    z_{SALR}(\\rho) = \\min_{\\u\\in U_\\rho}\\phi(\\u) + \\rho \\norm{\\u}\n\\end{align*}\n\\cref{prop:2} yields that for $\\rho > \\norm{\\blambda_{\\A}}$ and $\\u \\in U_{\\rho}$, $\\norm{\\u}$ is bounded. Thus, we have\n\\begin{equation*}\n    z_{SALR}(\\rho) = \\min_{\\u \\in \\mathcal{N}_{\\delta_{\\rho}}(\\bz) \\cap\\, U_\\rho}\\phi(\\u) + \\rho \\norm{\\u} \n\\end{equation*}\nwhere $\\displaystyle\\delta_{\\rho} = \\frac{\\zIP - z_R}{\\rho -\\norm{\\blambda_{\\A}}}$. From \\cref{prop:2} we see that as $\\rho \\to \\infty,\\, \\mathcal{N}_{\\delta_{\\rho}}(\\bz) \\cap U_\\rho \\to \\{\\bz\\}$. If $\\exists$ $\\delta > 0$, and $0 < \\kappa < \\infty$ such that $\\phi(\\bz) \\leq \\phi(\\u) + \\kappa \\norm{\\u}$ for all $\\u \\in \\mathcal{N}_{\\delta}(\\bz) \\cap U_\\rho$, it follows that,\n\\begin{align}\n    \\min_{\\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap\\, U_\\rho} \\phi(\\u) + \\kappa \\norm{\\u} = \\phi(\\bz) = \\zIP\\label{eq:9}\n\\end{align}\nAlternatively, we can increase $\\rho$ such that $\\forall\\, \\u \\in U_\\rho$, $\\norm{\\u} \\leq \\dfrac{\\zIP - z_R}{\\rho -\\norm{\\blambda_{\\A}}} < \\delta$.  Rearranging the terms, we obtain $\\rho > \\norm{\\blambda_\\A} + \\frac{\\zIP - z_R}{\\delta}$.  It follows from \\cref{eq:9} that for any choice of $\\displaystyle \\rho > \\max\\left\\{\\kappa, \\norm{\\blambda_\\A} + \\frac{\\zIP - z_R}{\\delta}\\right\\}$, we have  \n\\begin{align*}\n    z_{SALR}(\\rho) = \\min_{\\u\\in \\mathcal{N}_{\\delta_{\\rho}}(\\bz) \\cap\\, U_\\rho} \\phi(\\u) + \\rho \\norm{\\u} = \\zIP.\n\\end{align*}\n\\end{proof}\n\n\\begin{lemma}\\label{lem:6}\nIf the integer variables are bounded, then the value function $\\displaystyle \\hat{\\phi}(\\u) = \\min_{i \\in [k]} \\Phi(\\u,\\x^{(i)}_I)$ is lower semi-continuous at $\\bz$.\n\\end{lemma}\n\n\\begin{proof}\nObserve that for $\\u \\in \\calU, \\x^{(i)}_I \\in S_I$, $\\Phi(\\u,\\x^{(i)}_I)$ is lower semi-continuous at ${\\u}$ if and only if strong duality holds at ${\\u}$ [Proposition 6.5.2,\\cite{bertsekas2003convex}]. If for some $\\x^{(i)}_I$, $\\Phi(\\bz, \\x^{(i)}_I) < +\\infty$ then $\\Phi(\\u, \\x^{(i)}_I)$ is lower semi-continuous at $\\u = \\bz$ due to Slater Condition (in case of affine constraints only feasibility is required). On the contrary, $\\Phi(\\bz,\\x^{(i)}_I) = + \\infty$ implies that the polyhedron $P = \\set{\\x \\in \\R^{n_2}}{\\tilde{\\A}\\x = \\tilde{\\b}, \\,\\,\\x \\geq \\bz}$ is empty, where\n\\begin{align*}\n    \\tilde{\\A} = \\begin{pmatrix}\n      {\\A}_C & -{\\A}_C & \\bz\\\\\n      {\\E}_C & -{\\E}_C & \\mathbb{I}\n    \\end{pmatrix},~\n    \\x = \\begin{pmatrix}\n    \\x_C^+\\\\\n    \\x_C^-\\\\\n    \\s\n    \\end{pmatrix},~\n    \\tilde{\\b} = \\begin{pmatrix}\n     {\\b}  -{\\A}_I{\\x}^{(i)}_I\\\\\n     {\\f} - {\\E}_I{\\x}^{(i)}_I\n    \\end{pmatrix}.\n\\end{align*}\nLet $C$ be the closed convex cone spanned by the columns of $\\tilde{\\A}$. If $P = \\emptyset$ then $\\tilde{\\b} \\not\\in C$. Since $C$ is a closed, \n$\\exists\\,\\,\\delta_{i} > 0$ such that $\\mathcal{N}_{\\delta_i}(\\tilde{\\b}) \\cap C = \\emptyset$. Consider $\\u \\in \\R^m \\cap \\mathcal{N}_{\\delta_i}(\\bz)$, then $\\tilde{\\b} + \\begin{pmatrix}\\u\\\\\\bz\\end{pmatrix} \\not \\in C$.\n\nFor $J = \\set{i\\in [k]}{\\Phi({\\x}^{(i)}_I,\\bz) < \\infty}$, define $\\displaystyle\\delta_{\\min} \\define \\min_{i \\in [k]\\setminus J}\\delta_i$.\n\nConsider y such that $y  < \\hat{\\phi}(\\bz)$, it follows that \n\\begin{align*}\n    y & < \\hat{\\phi}(\\bz) < \\min_{i \\in J}\\Phi(\\bz,{\\x}^{(i)}_I) \\leq \\Phi(\\bz,{\\x}^{(i)}_I) .\n\\end{align*}\nHowever, since every $\\Phi({\\u},{\\x}^{(i)}_I), \\, i \\in J$ is lower semi-continuous at $\\bz$, there exists a $\\delta_i > 0$ such that $\\Phi({\\u},{\\x}^{(i)}_I) > y$ for every ${\\u} \\in \\mathcal{N}_{\\delta_i}(\\bz), \\, i \\in J$. Let $\\delta'_{\\min} = \\min(\\delta_i),\\, i \\in J$ and $\\delta'' = \\min(\\delta_{\\min},\\delta'_{\\min})$. This means that for every ${\\u} \\in \\mathcal{N}_{\\delta''}(\\bz)$ $\\Phi({\\u},{\\x}^{(i)}_I) > y \\implies  \\hat{\\phi}({\\u}) > y$. The result follows.\n\\end{proof}\n\nConsider $\\ocalU$ as defined in \\cref{cor:4}, it follows from \\cref{lem:6} that $\\lim_{\\u\\to \\bz} \\hat{\\phi}(\\u) = \\hat{\\phi}(\\bz)$ where $\\u \\in \\ocalU$. Now, for every $\\epsilon > 0$ there exists a $\\delta > 0$ such that\n\\begin{equation*}\n   \\abs{\\hat{\\phi}(\\u) - \\hat{\\phi}(\\bz)} < \\epsilon\\,\\, \\forall \\u\\,\\, \\in \\mathcal{N}_{\\delta}(\\bz) \\cap \\ocalU\n\\end{equation*}\n\\begin{lemma}\\label{lem:8}\nConsider the value function $\\hat{\\phi}(\\u)$ as defined earlier for $\\textup{\\MICP}_\\u$. There exists a $\\delta > 0$ such that for all $\\u \\in \\mathcal{N}_\\delta(\\bz) \\cap \\ocalU$, $\\displaystyle \\hat{\\phi}(\\u) = \\min_{\\x_I\\in S_{=}^{\\bz}} \\Phi(\\u,\\x_I)$ where $S_{=}^{\\bz} = \\set{\\x_I \\in S_I}{\\Phi(\\bz,\\x_I) = \\hat{\\phi}(\\bz)}$\n\\end{lemma}\n\n\\begin{proof}\nWe can partition the index sets corresponding to solutions of $\\CPui$ into two sets, \n$$\nS_{>}^{\\bz} = \\set{\\x_I\\in S_I}{\\Phi(\\bz,\\x_I) > \\hat{\\phi}(\\bz)} \\text{ and } S_{=}^{\\bz} = \\set{\\x_I\\in S_I}{\\Phi(\\bz,\\x_I) = \\hat{\\phi}(\\bz)}.\n$$\nFor each $\\x^{(i)}_I \\in S_I$, define $ \\overline{\\calU}_i=\\set{\\u \\in \\calU}{\\Phi({\\u},\\x^{(i)}_I)\\leq \\Phi(\\bz,\\x^{(i)}_I)} $. If there is no limiting sequence to ${\\bz}$ in $\\overline{\\calU}_i$ then there exists a neighbourhood $\\delta_i$ such that $\\mathcal{N}_{\\delta_i}(\\bz) \\cap \\overline{\\calU}_i = \\emptyset$. Alternatively, if there does exist a limiting sequence to $\\bz$ and since $\\Phi({\\u},{\\x}_I^{(i)})$ is lower semi-continuous,\n\\begin{equation*}\n    \\lim_{{\\u}\\to \\bz} \\Phi({\\u},{\\x}^{(i)}_I) = \\Phi(\\bz,{\\x}^{(i)}_I)\n\\end{equation*}\nThus, for every $\\epsilon_i > 0$ there exists a $\\delta_i > 0$ such that\n\\begin{align*}\n    \\abs{\\Phi({\\u},{\\x}^{(i)}_I) - \\Phi(\\bz,{\\x}^{(i)}_I)}< \\epsilon_i \\,\\,\\,\\,\\, \\forall \\,\\,\\,\\,{\\u}\\in \\overline{\\calU}_i\\cap \\mathcal{N}_{\\delta_i}(\\bz)\n\\end{align*}\nFor $\\x^{(i)}_I \\in S_{>}^{\\bz}$, let $\\epsilon_i = \\Phi(\\bz,{\\x}^{(i)}_I) - \\hat{\\phi}(\\bz)$. Assuming that $\\Phi(\\bz,{\\x}_I^{(i)})$ is finite, $\\forall$ $\\u \\in \\overline{\\calU}_i\\cap \\mathcal{N}_{\\delta_i}(\\bz)$\n\\begin{align*}\n    \\Phi(\\bz,{\\x}^{(i)}_I) - \\Phi({\\u},{\\x}_I^{(i)}) & <  \\Phi(\\bz,{\\x}^{(i)}_I) - \\hat{\\phi}(\\bz)\\\\\n    \\hat{\\phi}(\\bz) & < \\Phi({\\u},{\\x}^{(i)}_I)\n\\end{align*}\nThus, for all $\\x^{(i)}_I\\in S_{>}^{\\bz}$ there exists a $\\delta_i$ such that $\\Phi({\\u},{\\x}_I^{(i)}) > \\hat{\\phi}(\\bz)$. Assigning $\\delta_{\\min} = \\min_{\\x^{(i)}_I\\in S^{\\bz}_{>}} \\delta_i$ yields $\\displaystyle \\hat{\\phi}({\\u}) = \\min_{\\x_I\\in S^{\\bz}_{=}}\\Phi({\\u},{\\x}_I)$ for every $\\u \\in \\mathcal{N}_{\\delta_{\\min}}(\\bz)\\cap \\overline{\\calU}$.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:9}\nConsider the value function $\\hat{\\phi}(\\u)$ as defined earlier for $\\textup{\\MICP}_\\u$. There exists a $\\delta > 0$ and $0 < \\Gamma< \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $$\\hat{\\phi}(\\bz)\\leq \\hat{\\phi}(\\u) + \\Gamma\\norm{\\u}.$$\n\\end{lemma}\n\n\\begin{proof}\nConsider $S^{\\bz}_{=}$ as defined in \\cref{lem:8}. Analogously, define $$S_{=}^{\\u} \\define \\set{\\x_I\\in S_{=}^{\\bz}}{\\Phi(\\u,\\x_{I}) = \\hat{\\phi}(\\u)}.$$ For $\\x^{(i)}_I \\in S^{\\u}_{=}$, the first order necessary conditions for optimality (KKT conditions) for $\\CPui$ can be expressed as, %For  writing the KKT conditions\\\\\n\\begin{align}\\label{eq:32}\n\\begin{split}\n    \\nabla_{C}{f(\\x_I^{(i)},\\x_C}) & = \\A^\\top_C\\blambda^{\\u(i)}_{\\A_C} - \\E_C^\\top\\blambda^{\\u(i)}_{\\E_C}\\\\\n    \\A_C\\x_C & = \\b + \\u - \\A_I\\x^{(i)}_I\\\\\n    \\E_C\\x_C+\\E_I\\x^{(i)}_I & \\leq \\f\\\\\n    {\\blambda^{\\u(i)}_{\\E_C}}^\\top(\\f - \\E_C\\x_C - \\E_I\\x^{(i)}_I) & = \\bz\\\\\n    {\\blambda^{\\u(i)}_{\\A_C}}^\\top(\\A_C\\x_C + \\A_I\\x^{(i)}_I - \\b - \\u) & = \\bz\\\\\n    {\\blambda^{\\u(i)}_{\\E_C}} & \\geq \\bz\n\\end{split}\n\\end{align}\nDefine $H^{(j)}_{\\u} = \\set{\\x_C \\in \\R^{n_2}}{\\A_C\\x_C = \\b +\\u- \\A_I\\x^{(j)}_I}$. Let $(\\x_I^{(j)},\\x_C) \\in X \\cap H^{(j)}_{\\u}$ where $\\x_I^{(j)} \\in S_{=}^{\\u}$.\nAdditionally, let the corresponding Lagrange multipliers at $\\u = \\bz$ for $\\x^{(j)}_I$ be $\\blambda^{(j)}_{\\A_C}$ and $\\blambda^{(j)}_{\\E_C}$.\n\\begin{align*}\n    f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\nabla_C{f(\\x_I^{(j)}}\\x^{\\bz*}_C)^{\\top}(\\x^{\\u}_C -  \\x^{\\bz*}_C )\\\\ \n    & \\hspace{3cm}+\\nabla_I{f(\\x_I^{(j)}}, \\x^{\\bz*}_C)^{\\top}(\\x^{(j)}_I -  \\x^{(j)}_I)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) + ( \\blambda^{(j)^\\top}_{\\A_C}\\A_C - \\blambda^{(j)^\\top}_{\\E_C}\\E_C) (\\x^{\\u}_C -  \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}\\A_C(\\x^{\\u}_C -  \\x^{\\bz*}_C) - \\blambda^{(j)^\\top}_{\\E_C}\\E_C (\\x^{\\u}_C -  \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\A_C \\x^{\\u}_C -  \\A_C \\x^{\\bz*}_C) - \\blambda^{(j)^\\top}_{\\E_C}(\\E_C \\x^{\\u}_C -  \\E_C \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\b+\\u-\\A_I\\x^{(j)}_I - \\b + \\A_I\\x^{(j)}_I)\\\\ \n    & \\hspace{3cm}-\\blambda^{(j)^\\top}_{\\E_C}(\\E_C \\x^{\\u}_C -  \\f + \\E_I\\x^{(j)}_I)\n\\end{align*}\nwhere the equalities follow from the set of equations \\cref{eq:32}. Furthermore, observe that since $\\blambda^{(j)^\\top}_{\\E_C} \\geq \\bz$ and $\\E_C\\x^{\\u}_C + \\E\\x^{(j)}_I \\leq \\f$, it follows that \n\\begin{align*}\n   f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u), \\\\\n      \\inf_{X\\cap H^{(j)}_{\\u}}f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u), \\text{ i.e.,}\\\\\n   \\hat{\\phi}({\\u)} & \\geq \\hat{\\phi}(\\bz) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u)\n\\end{align*}\nRearranging the terms in the above inequality,     $-\\blambda^{(j)^\\top}_{\\A_C}\\u \\geq \\hat{\\phi}(\\bz) - \\hat{\\phi}({\\u)}$. As $\\hat{\\phi}(\\bz)\\geq \\hat{\\phi}(\\u)$ $\\forall\\,\\,\\, \\u \\in \\ocalU$, using Cauchy-Schwarz on the left-hand side of the inequality,\n\\begin{align*}\n    \\norm{\\blambda^{(j)}_{\\A_C}}\\cdot\\norm{\\u} \\geq \\hat{\\phi}(\\bz) - \\hat{\\phi}({\\u)} .\n\\end{align*}\nDefining $\\Gamma = \\max_{i\\in S^{\\bz}_{=} }{\\norm{\\blambda^{(j)}_{\\A_C}}}$ yields\n$$\n    \\hat{\\phi}(\\bz) - \\hat{\\phi}(\\u)\\leq \\Gamma\\norm{\\u}\\,\\,\\forall\\,\\, \\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap\\ocalU\n$$\nwhere $\\delta = \\delta_{min}$ as defined in the proof of \\cref{lem:8}. The result follows.\n\\end{proof}\n\nThis completes the proof of \\Cref{thm:3}(a). In the following section we present some cases where an equivalence can be established between (MICP) and (MICP) with bounded integer variables. In particular, we highlight that  if the objective function $f$ satisfies $\\rec(f)\\cap\\rec(F_R)\\setminus\\{\\bz\\} = \\emptyset$, then there exists $M < \\infty$ such that the following equivalence holds.\n$$ \\min\\set{f(\\x)}{\\A\\x = \\b, \\x \\in X} = \\min\\set{f(\\x)}{\\A\\x = \\b, \\norm{\\x}_{\\infty} \\leq M, \\x \\in X}.$$ Observe that if $f$ is $\\mu$-strongly convex, then $\\rec(f) = \\emptyset$, as the level sets of $f$ are bounded. \n\n\n\\begin{lemma}\\label{lem:15}\nLet  $\\delta > 0$, and let $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$. Define\n$$\n    S^\\u \\define  \\set{\\x \\in X \\cap \\Hu}{ f(\\x) \\leq \\zIP}.\n$$\nIf \\textup{\\MICP} is feasible ($F \\neq\\emptyset$) and bounded and the recession cone of $f$ and recession cone of $F_R$, the feasible set of continuous relaxation of \\textup{\\MICP}, have no common non-zero directions of recession, then  $S^\\u$ is bounded.\n\\end{lemma}\n\n\\begin{proof}\nConsider $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$. Since $\\norm{\\u}_\\infty \\leq \\norm{\\u} < \\delta$ we have $\\norm{\\b - \\A\\x}_{\\infty} < \\delta$, $\\forall\\,\\x \\in X \\cap \\Hu$. Alternatively, $\\forall\\,\\x \\in X \\cap \\Hu$, $-\\1\\delta < {\\b} - \\A\\x < \\delta\\1$. Define, for $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$\n\\begin{align*}\n  S^\\u \\define  \\set{\\x \\in X \\cap \\Hu}{ f(\\x) \\leq \\zIP}\n\\end{align*}\nAdditionally, consider\n\\begin{align*}\n   S = \\set{\\x\\in X_R}{\\begin{Bmatrix}\n                                    -{\\1}\\delta \\leq {\\b} - \\A\\x \\leq {\\1}\\delta \\\\\n                                    f(\\x) \\leq z_{IP}\n                        \\end{Bmatrix}}\n\\end{align*}\n%Here ${\\u}$ is taken over all feasible polyhedra i.e. this set $\\{\\A\\x = {\\b} + {\\u},\\,\\, {\\E}\\x\\leq {\\f}\\,\\, ,\\x\\in \\mathbb{Z}^{n_1}\\times \\mathbb{R}^{n_2}\\}$ is feasible.\nIndeed, $S^\\u \\subseteq S$. Furthermore, $\\rec(S) = \\rec(F_R) \\cap \\rec(f)$, and since $\\rec(F_R) \\cap \\rec(f) = \\{\\bz\\}$, it follows that $S$ is compact and consequently $S^\\u$ is bounded. In particular, $\\norm{\\x}_\\infty \\leq M < \\infty$, $\\forall$ $\\x \\in S^\\u$. %since MICP is feasible and optimally bounded S'' is non-empty.\\\\\n%This implies that one can prescribe a uniform bound on variables of the problem and hence $\\norm{\\x_I\\norm{\\leq M$.\n\\end{proof}\n\n\\begin{corollary}\\label{cor:16}\nIf \\textup{\\MICP} is feasible ($F \\neq\\emptyset$) and bounded and the recession cone of $f$ and recession cone of $F_R$, the feasible set of continuous relaxation of \\textup{\\MICP}, have no common non-zero directions of recession, then for ${\\u} \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$, $\\phi(\\u) = \\hat{\\phi}(\\u)$ i.e.\n\\begin{optprog*}\n    {$\\phi(\\u) = $ minimize}  & \\objective{\\hspace{1.3cm}f({\\x_I},{\\x_C})}\\\\\n    subject to & \\hspace{0cm} \\A_I\\,\\x_I + \\A_C\\x_C = \\b + \\u\\\\\n    & \\E_I\\x_I + \\E_C\\x_C \\leq \\f \\hspace{7mm}\\\\\n    & \\norm{\\x_I}_{\\infty}\\leq M \\hspace{0.5cm}\\\\\n    % & \\x_I\\in \\Z_{+}^{n_1}\\hspace{0.4cm}\\\\ \n     & (\\x_I, \\x_C) \\in \\Z^{n_1}\\times\\R^{n_2}\\hspace{0.3cm}\n\\end{optprog*}\nFurthermore, there exists a $\\delta > 0$ and $0 < \\Gamma < \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $\\phi(\\bz)\\leq \\phi(\\u) + \\Gamma\\norm{\\u}$. Additionally, if $f$ is $\\mu$-strongly convex and $L$ smooth, then $\\Gamma = \\mathcal{O}\\left(\\frac{L\\beta\\gamma}{\\mu}\\right)$ where $\\displaystyle \\beta \\define \\max_{B \\in \\mathcal{B}}\\norm{B^{-1}}_F$, $\\mathcal{B}$ being the set of all possible invertible submatrices of $[\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^T_C]$ and $\\gamma$ depends on $f(\\bz)$, $\\norm{\\nabla{f(\\bz)}}$ and $f(\\bar{\\x})$ for any $\\bar{\\x} \\in F$.\n\\end{corollary}\n\n\\begin{proof}\n\\Cref{lem:15}, along with \\cref{lem:8,lem:9} readily implies that $\\phi(\\u) = \\hat{\\phi}(\\u)$ and there exists a $\\delta > 0$ and $0 < \\Gamma < \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $\\phi(\\bz)\\leq \\phi(\\u) + \\Gamma\\norm{\\u}$.\n\nTo see the explicit bound on $\\Gamma$, recall the stationarity condition from set of equations \\cref{eq:32} at $\\u = \\bz$.\n\\begin{align}\\label{eq:stationarity}\n\\begin{split}\n    \\nabla_{C}{f(\\x_I^{(i)},\\x_C}) & = \\A^T_C(\\lambda^{(i)+}_{\\A_C} - \\lambda^{(i)-}_{\\A_C}) - {\\E}_C^T\\lambda_{{\\E}_C}\\\\\n    \\lambda_{\\E_C}^{\\top}(\\E\\x -\\f) &= 0\\\\\n    \\lambda^{+}_{\\A_C},\\lambda^{-}_{\\A_C} & \\geq \\bz\\\\\n     \\lambda_{{\\E}_C} &\\geq \\bz\n\\end{split}\n\\end{align}\n\nFor $(\\x_C,\\x^{(i)}_I)$ consider the complementary slackness conditions, that is if for some $j\\in [m]$, $\\E_{j}\\x < \\f_j$ where $\\E_{j}$ is a row of the matrix $\\E$ then the corresponding Lagrangian multiplier $\\lambda_{\\E_j} = 0$. Partition the set $[m]$ into two sets $J = \\{J_<,J_=\\}$, where $J_<=\\set{j\\in [m]}{\\E_j \\x <\\f_j}$ and $J_= =\\set{j\\in [m]}{\\E_j \\x = \\f_j}$.\\\\\n\nDefine $\\A_{aug} = [\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^{\\top}_C]$ and $\\A_{aug}^J$ as the sub-matrix which has all the columns of $\\A_{aug}$ except the columns $2n_2+J_<$.\n % $\\lambda^{(i)+}_{\\A_C},\\lambda^{(i)-}_{\\A_C}$. \n One possible solution to the system \\cref{eq:stationarity} is,\n \\begin{align}\\label{sol:lambda}\n    \\begin{pmatrix}\n     \\lambda^{+}_{\\A_C}\\\\\n      \\lambda^{-}_{\\A_C}\\\\\n      \\lambda_{{\\E}^{J_{=}}_C}\n    \\end{pmatrix} = B^{-1} \\nabla_C{f(\\x^{(i)}_I,\\x_C)}\n\\end{align}\nwhere $B$ is a basis matrix of $\\A_{aug}^J$.\n\nFurthermore, let $\\lambda_{\\A_C} = \\lambda^{+}_{\\A_C} - \\lambda^{-}_{\\A_C}$, this implies that \n\\begin{align}\n    \\norm{\\lambda_{\\A_C}} \\leq \\norm{\\lambda^{+}_{\\A_C}} + \\norm{\\lambda^{-}_{\\A_C}} \\label{ineq:lambda}    \n\\end{align}\nLet $\\x^*$ be an optimal solution to \\MICP, i.e. $f(\\x^*) = \\zIP$. Since $f$ is $\\mu$-strongly convex, we have\n\n\\begin{align*}\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 + \\nabla{f({\\bz)}}^\\top\\x^*  + f(\\bz) & \\leq z_{IP}\\\\\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 + \\nabla{f({\\bz)}}^\\top\\x^* - z_{IP} + f(\\bz)& \\leq 0\\\\\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 - \\norm{\\nabla{f({\\bz)}}}\\,\\norm{\\x^*} - \\left(z_{IP} - f(\\bz)\\right) & \\leq 0\n\\end{align*}\nIt follows that,\n$$\n    \\norm{\\x^*}\\leq \\frac{\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz})}^2 +2\\,\\mu\\cdot \\left(z_{IP} - f(\\bz)\\right)}}{\\mu}\n$$\nSince $f$ is L-smooth,\n\\begin{align*}\n    \\norm{\\nabla{f(\\x^*)} - \\nabla{f(\\bz)}} &\\leq L\\norm{\\x^*}\\\\\n    \\norm{\\nabla{f(\\x^*)}} &\\leq L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\n\\end{align*}\nFrom \\cref{sol:lambda} and \\cref{ineq:lambda} we obtain\n\\begin{align*}\n    \\norm{\\lambda_{\\A_C}} & \\leq 2\\norm{B^{-1}}_F \\norm{\\nabla{f(\\x^*)}}\\\\\n    & \\leq 2\\norm{B^{-1}}_F\\left(L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\\right)\\\\\n    & \\leq 2\\beta\\left(L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\\right)\n\\end{align*}\nwhere $\\beta \\define \\max_{B \\in \\mathcal{B}}\\norm{B^{-1}}_F$ and $\\mathcal{B}$ is the set of all possible invertible submatrices of $[\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^T_C]$.\n\n\\begin{align}\n   \\norm{\\lambda_{\\A_C}} & \\leq \\dfrac{2\\beta\\left(L\\left(\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz})}^2 +2\\mu\\,\\left(z_{IP} - f(\\bz)\\right)}\\right) + \\mu\\norm{\\nabla{f(\\bz)}}\\right)}{\\mu}\\nonumber\\\\\n    & \\leq  \\dfrac{2\\beta L}{\\mu}\\left(2\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz)}}^2 + 2\\mu\\,\\left(f(\\bar{\\x}) - f(\\bz)\\right)}\\right)\\label{ineq:thm3}\n\\end{align}\nwhere the second inequality follows as $\\zIP \\leq f(\\bar{\\x}),$ $\\forall\\,\\bar{\\x} \\in F$ and $L > \\mu$. Combining \\cref{ineq:thm3} with the construction of $\\Gamma$ in \\cref{lem:9}, yields the result.\n\\end{proof}\n\n\\cref{thm:3} follows.\n","units":[{"raw_tex":"\\begin{lemma}\\label{lem:1}\n$\\displaystyle z_{SALR}(\\rho) = \\min_{\\u \\in \\calU}\\,\\, \\phi(\\u) + \\rho\\norm{\\u}$ . \n\\end{lemma}","start":21629,"end":21752},{"raw_tex":"\\begin{proof}\nObserve the following inequality which follows immediately from the aforementioned definitions\n\\begin{align}\n    z_{SALR}(\\rho) = \\min_{\\x \\in X} f(\\x) + \\rho\\, \\norm{\\b - \\A\\,\\x} & \\leq \\min_{\\x\\in X \\cap \\Hu} f(\\x) + \\rho\\norm{\\u} ,\\,\\,\\,\\,\\, \\forall\\,\\u \\in \\calU \\nonumber\\\\\n                    & = \\phi(\\u) + \\rho\\norm{\\u} ,\\,\\,\\,\\,\\, \\forall\\,\\u \\in \\calU. \\label{eq:1}\n\\end{align}\nConversely, it holds that $\\forall\\,\\u \\in \\calU$\n\\begin{align*}\n    \\phi(\\u) + \\rho\\norm{\\u} & = \\min_{\\x\\in X \\cap \\Hu} f(\\x) + \\rho\\norm{\\u}\\\\\n                             & \\leq f(\\x) + \\rho\\norm{\\A\\x-\\b} ,\\,\\,\\,\\,\\, \\forall\\,\\x\\in X\\cap \\Hu.\n\\end{align*}\nThis implies that $ \\min_{\\u \\in \\calU} \\phi(\\u) + \\rho \\norm{\\u} \\leq f(\\x) + \\rho \\norm{\\A\\x - \\b} $ $\\forall~ x\\in X \\cap \\Hu,\\,\\forall~\\u \\in U$ which yields that $ \\min_{\\u \\in \\calU} \\phi(\\u) + \\rho \\norm{\\u} \\leq f(\\x) + \\rho \\norm{\\A\\x - \\b} $ $\\forall~ \\x \\in X $. This implies,\n\\begin{align}\n    \\min_{\\u\\in \\calU} \\phi(\\u) + \\rho\\norm{\\u} \\leq \\min_{\\x\\in X} f(\\x) + \\rho\\norm{\\A\\x-\\b}. \\label{eq:2}\n\\end{align}\nThe result follows from \\cref{eq:1} and \\cref{eq:2}.\n\\end{proof}","start":21753,"end":22902},{"raw_tex":"\\begin{proposition}\\label{prop:2}\nConsider the continuous relaxation of MICP, and let the optimal objective value $z_R = \\min_{X_R\\cap \\Ho} f(\\x)$ be attained at $\\x_R$. \nLet $\\blambda_{\\A}$ and $\\blambda_{\\E}$ be the Lagrange multipliers for the constraints $\\A\\x = \\b$ and $\\E\\x\\leq \\f$, respectively.\\vspace{1mm}\n\\begin{enumerate}[label = \\roman*)]\n    \\item $\\phi(\\u) > -\\infty$ for all $\\u \\in \\calU$.\\vspace{1mm}\n    \\item If $\\exists ~ \\boldsymbol{\\alpha} \\in X_R$ such that $f(\\boldsymbol{\\alpha}) + \\rho\\norm{\\b-\\A\\,\\boldsymbol{\\alpha}}\\leq \\zIP$ and $\\rho > \\norm{\\blambda_\\A}$  then \n        \\[\n                \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq \\frac{\\zIP-z_R}{\\rho -\\norm{\\blambda_{\\A}}}.\n        \\]    \n\\end{enumerate}\n\\end{proposition}","start":23391,"end":24147},{"raw_tex":"\\begin{proof}\nConsider the Lagrangian function for the continuous relaxation of MICP, i.e. for $\\x \\in \\R^n$\n\\begin{equation*}\n    \\mathcal{L}(\\x,\\blambda_{\\A},\\blambda_{\\E}) = f(\\x) - \\blambda_\\A^{\\top}(\\A\\,\\x-\\b)-\\blambda_\\E^{\\top}(\\f-\\E\\x)\n\\end{equation*}\nAs the relaxed problem has only affine constraints strong duality holds (Chapter 5,Section 5.2.3\\cite{boyd2004convex}).\n\n As strong duality holds, the first order necessary (KKT) conditions for $\\mathcal{L}(\\x,\\blambda_{\\A},\\blambda_{\\E})$ can be characterized as\n\\begin{align}\\label{eq:4}\n\\begin{split}\n    \\nabla{f(\\x)} = \\A^{\\top}\\blambda_\\A - \\E^{\\top}{\\blambda_\\E}\\\\\n    \\blambda_\\E^{\\top}(\\f - \\E\\x) = 0\\\\\n    \\blambda_\\A^{\\top}(\\A\\,\\x - \\b) = 0\\\\\n    \\blambda_\\E \\geq \\bz\n\\end{split}\n\\end{align}\nObserve that $\\x_R$ satisfies the system of equations \\cref{eq:4}.\\\\\n\\begin{enumerate}[label = \\roman*)]\n    \\item To see the result, observe that it suffices to show that $\\phi(\\u) > -\\infty$ for all $\\u \\in \\calU$ such that $\\phi(\\u) \\leq \\phi(\\bz) = \\zIP$. Further, consider the set $F_{\\leq}$ defined as,\n            $$\n                F_{\\leq} \\define \\set{\\x \\in X}{f(\\x) \\leq \\zIP}.\n            $$\n            We have for all $\\x \\in F_{\\leq}$, $\\zIP \\geq f(\\x) \\geq f(\\x_R) + \\nabla{f(\\x_R)}(\\x-\\x_R)= z_R + \\nabla{f(\\x_R)}(\\x-\\x_R)$. Substituting from \\cref{ineq:gradient} we obtain, for all $\\x \\in F_{\\leq}$,\n            $$\n                \\zIP \\geq f(\\x) \\geq z_R + \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b) -  \\blambda_\\E^{\\top} (\\E\\x -  \\f )\n            $$\n            As $\\blambda_{\\E} \\geq \\bz$ and $\\x \\in X_R$,\n            \\begin{align*}\n                \\zIP \\geq f(\\x) & \\geq z_R + \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b)\\\\\n                & \\geq z_R - \\norm{\\blambda_{\\A}}\\norm{(\\A\\,\\x - \\b)}\\\\\n                & = z_R - \\norm{\\blambda_{\\A}}\\norm{\\u} > -\\infty.\n            \\end{align*}\n            Since $f(\\x) > -\\infty$ for all $\\x \\in F_{\\leq}$, the result follows.\\\\\n    \\item Since we have for $\\boldsymbol{\\alpha}\\in X_R$, $f(\\boldsymbol{\\alpha}) + \\rho\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq  \\zIP$; convexity of $f$ yields,\n            \\begin{align}\n                f(\\x_R)+ \\nabla{f(\\x_R)}^{\\top}(\\boldsymbol{\\alpha}-\\x_R) +\\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} &\\leq \\zIP , \\text{ i.e.,}\\nonumber \\\\\n                z_R+ \\nabla{f(\\x_R)}^{\\top}(\\boldsymbol{\\alpha}-\\x_R) +\\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} &\\leq \\zIP. \\label{ineq:1}\n            \\end{align}\n            It follows from strong duality that\n            \\begin{align}\n            \\nabla{f(\\x_R)}^{\\top}(\\x-\\x_R) &= \\blambda_{\\A}^{\\top} \\A(\\x - \\x_R) -\\blambda_\\E^{\\top} \\E(\\x-\\x_R)\\nonumber\\\\\n             &= \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b) -  \\blambda_\\E^{\\top} (\\E\\x -  \\f ). \\label{ineq:gradient}\n            \\end{align}\n            Substituting in \\cref{ineq:1}, we obtain\n            \\begin{align*}\n            z_R+ \\blambda_{\\A}^{\\top} (\\A\\,\\boldsymbol{\\alpha} - \\b) +  \\blambda_\\E^{\\top} (\\f - \\E\\boldsymbol{\\alpha}) +\\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b}  & \\leq \\zIP .\n            \\end{align*}\n            As $\\blambda_{\\E} \\geq \\bz$ and $\\boldsymbol{\\alpha} \\in X_R$, the above inequality can be rewritten as\n            % \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b) +   \\blambda_\\E^{\\top} ( \\f - \\E\\x ) +\\rho \\norm{\\A\\,\\x - \\b} & \\leq \\zIP - z_R\\\\\n            \\begin{align}\n            \\blambda_{\\A}^{\\top}(\\A\\,\\boldsymbol{\\alpha} - \\b) + \\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} & \\leq \\zIP - z_R \\label{ineq:2}\n            \\end{align}\n            Using Cauchy Schwarz inequality in \\cref{ineq:2} yields for all $\\x \\in X_R$,\n            \\begin{align*}\n                -\\norm{\\blambda_\\A}\\,\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} + \\rho\\,\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq \\zIP - z_R\\\\\n            (\\rho - \\norm{\\blambda_\\A})\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b}\\leq \\zIP - z_R\\\\\n            \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq \\frac{\\zIP-z_R}{\\rho - \\norm{\\blambda_\\A}}.\n            \\end{align*}\n\\end{enumerate}\n\\end{proof}","start":24148,"end":28140},{"raw_tex":"\\begin{corollary}\\label{cor:3}\nConsider the set $U_\\rho = \\set{\\u \\in \\calU}{\\phi(\\u) + \\rho\\norm{\\u} \\leq \\phi(\\bz)}$. If $\\rho > \\norm{\\blambda_{\\A}}$ then for all $\\u \\in U_\\rho$, $$\\norm{\\u} \\leq \\frac{\\zIP-z_R}{\\rho -\\norm{\\blambda_\\A}}.$$\n\\end{corollary}","start":28284,"end":28544},{"raw_tex":"\\begin{proof}\nIf for any $~ \\u \\in U_{\\rho},~\\exists~ \\x \\in X\\cap H_{\\u}$ such that $f(\\x) + \\rho\\norm{\\A\\x - \\b} \\leq z_{IP}$, the result follows from \\Cref{prop:2}. Alternatively, if $f(\\x) + \\rho\\norm{\\A\\x - \\b} > z_{IP}\\,\\, \\forall~ \\x \\in X\\cap H_{\\u}$ for some $\\u \\in U_{\\rho}$. \n\\begin{align*}\n\\phi(\\u) + \\rho(\\norm{\\u}) =  \\inf_{X\\cap H_{\\u}} f(\\x) + \\rho\\norm{\\A\\x - \\b} \\geq z_{IP}\n\\end{align*} \nIf \\(\\inf_{X\\cap H_{\\u}} f(\\x) + \\rho\\norm{\\A\\x - \\b} > z_{IP}\\) then $\\u \\not\\in U_{\\rho}$ leading to a contradiction. If \\(\\inf_{X\\cap H_{\\u}} f(\\x) + \\rho\\norm{\\A\\x - \\b} = z_{IP}\\) and there exists an \\(\\x \\in X\\cap H_{\\u}\\) such that \\(f(\\x) + \\rho \\norm{\\A\\x - \\b} = z_{IP}\\) then the \nresult follows from \\cref{prop:2}. If there doesn't exist \\(\\x \\in X\\cap H_{\\u}\\) such that \\(f(\\x) + \\rho\\norm{\\A\\x - \\b}  = z_{IP}\\) then one can find a sequence of \\(\\x_{p}\\) such that the follows holds\n\\begin{align*}\n    f(\\x_{p}) + \\rho \\norm{\\A\\x_p - \\b} \\leq z_{IP} + \\frac{1}{p} \\text{ (From definition of infimum)}\\\\\n\\end{align*}\nLet \\(\\x_R\\) be the optimal solution of the optimization program \\(\\inf_{\\x \\in F_R} f(\\x)\\). Since we are minimizing a convex function over a rational polyhedron the KKT conditions hold as follows.\n\\begin{align*}   \n\\nabla{f(\\x_{R})} = \\A^{\\top}\\blambda_\\A - \\E^{\\top}{\\blambda_\\E}\\\\\n    \\blambda_\\E^{\\top}(\\f - \\E\\x_R) = 0\\\\\n    \\blambda_\\A^{\\top}(\\A\\,\\x_R - \\b) = 0\\\\\n    \\blambda_\\E \\geq \\bz\n\\end{align*}\n\\begin{align*} \n    f(\\x_R)+ \\nabla{f(\\x_R)}^{\\top}(\\x_{p}-\\x_R) +\\rho \\norm{\\A\\,\\x_p - \\b} &\\leq \\zIP + \\frac{1}{p}\\text{ (Convexity)}\\\\\n    z_{R} + \\blambda_{\\A}^{\\top} (\\A\\,\\x_p - \\b) +  \\blambda_\\E^{\\top} (\\f - \\E\\x_p) +\\rho \\norm{\\A\\,\\x_p - \\b} &\\leq \\zIP + \\frac{1}{p}\\\\\n    % (\\rho - \\norm{\\blambda_{\\A}}) \\norm{\\u} &\\leq \\frac{\\zIP - z_{R}}{\\rho - \\norm{\\blambda_\\A}} + \\frac{1}{p(\\rho - \\norm{\\blambda_{\\A}})}\\\\\n    \\norm{\\u} &\\leq \\frac{\\zIP - z_{R}}{\\rho - \\norm{\\blambda_{\\A}}} + \\frac{1}{p(\\rho - \\norm{\\blambda_{\\A}})}\n\\end{align*}\nwhere the last inequality follows from non-negativity of $\\blambda_\\E$ and $(\\f - \\E\\x_p).$ As \\(p\\) is arbitrary, we have as $p \\to \\infty$,\n\\begin{align*}\n    \\norm{\\u} &\\leq \\frac{\\zIP - z_{R}}{\\rho - \\norm{\\blambda_{\\A}}}\n\\end{align*}\n\\end{proof}","start":28545,"end":30789},{"raw_tex":"\\begin{corollary}\\label{cor:4}\nConsider the set $\\,\\ocalU \\define \\set{\\u \\in \\calU}{\\phi(\\bz) \\geq \\phi(\\u)}$. For $\\alpha \\geq 0$, $\\phi(\\bz) \\leq \\phi(\\u) + \\alpha$ $\\forall \\,\\u \\in \\calU$ if $\\phi(\\bz) \\leq \\phi(\\u) + \\alpha$ $\\forall \\,\\u \\in \\ocalU$.\n\\end{corollary}","start":30791,"end":31064},{"raw_tex":"\\begin{proof}\n    For $\\u \\in \\calU\\setminus\\ocalU$, $\\phi(\\bz) < \\phi(\\u) \\leq \\phi(\\u) + \\alpha$. The result follows.\n\\end{proof}","start":31065,"end":31196},{"raw_tex":"\\begin{lemma}\\label{lem:3}\nIf there exist $\\delta, \\kappa > 0$ such that  $\\phi(\\bz) \\leq \\phi(\\u) + \\kappa \\norm{\\u}$ for all $\\u \\in \\mathcal{N}_{\\delta}(\\bz) \\cap \\calU$,  then there exists $0 < \\rho^{*} < \\infty$ such that $z_{SALR}(\\rho) = \\zIP$ for all $\\rho > \\rho^{*}$.\n\\end{lemma}","start":31197,"end":31486},{"raw_tex":"\\begin{proof}\nObserve that, for $\\rho > 0$ it suffices to consider $\\u \\in U_\\rho$. We have, from \\cref{lem:1} \n$$ z_{SALR}(\\rho) = \\min_{\\u\\in \\calU} \\phi(\\u) + \\rho \\norm{\\u}.$$\nIt follows that,\n\\begin{align*}\n    z_{SALR}(\\rho) = \\min_{\\u\\in U_\\rho}\\phi(\\u) + \\rho \\norm{\\u}\n\\end{align*}\n\\cref{prop:2} yields that for $\\rho > \\norm{\\blambda_{\\A}}$ and $\\u \\in U_{\\rho}$, $\\norm{\\u}$ is bounded. Thus, we have\n\\begin{equation*}\n    z_{SALR}(\\rho) = \\min_{\\u \\in \\mathcal{N}_{\\delta_{\\rho}}(\\bz) \\cap\\, U_\\rho}\\phi(\\u) + \\rho \\norm{\\u} \n\\end{equation*}\nwhere $\\displaystyle\\delta_{\\rho} = \\frac{\\zIP - z_R}{\\rho -\\norm{\\blambda_{\\A}}}$. From \\cref{prop:2} we see that as $\\rho \\to \\infty,\\, \\mathcal{N}_{\\delta_{\\rho}}(\\bz) \\cap U_\\rho \\to \\{\\bz\\}$. If $\\exists$ $\\delta > 0$, and $0 < \\kappa < \\infty$ such that $\\phi(\\bz) \\leq \\phi(\\u) + \\kappa \\norm{\\u}$ for all $\\u \\in \\mathcal{N}_{\\delta}(\\bz) \\cap U_\\rho$, it follows that,\n\\begin{align}\n    \\min_{\\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap\\, U_\\rho} \\phi(\\u) + \\kappa \\norm{\\u} = \\phi(\\bz) = \\zIP\\label{eq:9}\n\\end{align}\nAlternatively, we can increase $\\rho$ such that $\\forall\\, \\u \\in U_\\rho$, $\\norm{\\u} \\leq \\dfrac{\\zIP - z_R}{\\rho -\\norm{\\blambda_{\\A}}} < \\delta$.  Rearranging the terms, we obtain $\\rho > \\norm{\\blambda_\\A} + \\frac{\\zIP - z_R}{\\delta}$.  It follows from \\cref{eq:9} that for any choice of $\\displaystyle \\rho > \\max\\left\\{\\kappa, \\norm{\\blambda_\\A} + \\frac{\\zIP - z_R}{\\delta}\\right\\}$, we have  \n\\begin{align*}\n    z_{SALR}(\\rho) = \\min_{\\u\\in \\mathcal{N}_{\\delta_{\\rho}}(\\bz) \\cap\\, U_\\rho} \\phi(\\u) + \\rho \\norm{\\u} = \\zIP.\n\\end{align*}\n\\end{proof}","start":31487,"end":33114},{"raw_tex":"\\begin{lemma}\\label{lem:6}\nIf the integer variables are bounded, then the value function $\\displaystyle \\hat{\\phi}(\\u) = \\min_{i \\in [k]} \\Phi(\\u,\\x^{(i)}_I)$ is lower semi-continuous at $\\bz$.\n\\end{lemma}","start":55608,"end":55813},{"raw_tex":"\\begin{proof}\nObserve that for $\\u \\in \\calU, \\x^{(i)}_I \\in S_I$, $\\Phi(\\u,\\x^{(i)}_I)$ is lower semi-continuous at ${\\u}$ if and only if strong duality holds at ${\\u}$ [Proposition 6.5.2,\\cite{bertsekas2003convex}]. If for some $\\x^{(i)}_I$, $\\Phi(\\bz, \\x^{(i)}_I) < +\\infty$ then $\\Phi(\\u, \\x^{(i)}_I)$ is lower semi-continuous at $\\u = \\bz$ due to Slater Condition (in case of affine constraints only feasibility is required). On the contrary, $\\Phi(\\bz,\\x^{(i)}_I) = + \\infty$ implies that the polyhedron $P = \\set{\\x \\in \\R^{n_2}}{\\tilde{\\A}\\x = \\tilde{\\b}, \\,\\,\\x \\geq \\bz}$ is empty, where\n\\begin{align*}\n    \\tilde{\\A} = \\begin{pmatrix}\n      {\\A}_C & -{\\A}_C & \\bz\\\\\n      {\\E}_C & -{\\E}_C & \\mathbb{I}\n    \\end{pmatrix},~\n    \\x = \\begin{pmatrix}\n    \\x_C^+\\\\\n    \\x_C^-\\\\\n    \\s\n    \\end{pmatrix},~\n    \\tilde{\\b} = \\begin{pmatrix}\n     {\\b}  -{\\A}_I{\\x}^{(i)}_I\\\\\n     {\\f} - {\\E}_I{\\x}^{(i)}_I\n    \\end{pmatrix}.\n\\end{align*}\nLet $C$ be the closed convex cone spanned by the columns of $\\tilde{\\A}$. If $P = \\emptyset$ then $\\tilde{\\b} \\not\\in C$. Since $C$ is a closed, \n$\\exists\\,\\,\\delta_{i} > 0$ such that $\\mathcal{N}_{\\delta_i}(\\tilde{\\b}) \\cap C = \\emptyset$. Consider $\\u \\in \\R^m \\cap \\mathcal{N}_{\\delta_i}(\\bz)$, then $\\tilde{\\b} + \\begin{pmatrix}\\u\\\\\\bz\\end{pmatrix} \\not \\in C$.\n\nFor $J = \\set{i\\in [k]}{\\Phi({\\x}^{(i)}_I,\\bz) < \\infty}$, define $\\displaystyle\\delta_{\\min} \\define \\min_{i \\in [k]\\setminus J}\\delta_i$.\n\nConsider y such that $y  < \\hat{\\phi}(\\bz)$, it follows that \n\\begin{align*}\n    y & < \\hat{\\phi}(\\bz) < \\min_{i \\in J}\\Phi(\\bz,{\\x}^{(i)}_I) \\leq \\Phi(\\bz,{\\x}^{(i)}_I) .\n\\end{align*}\nHowever, since every $\\Phi({\\u},{\\x}^{(i)}_I), \\, i \\in J$ is lower semi-continuous at $\\bz$, there exists a $\\delta_i > 0$ such that $\\Phi({\\u},{\\x}^{(i)}_I) > y$ for every ${\\u} \\in \\mathcal{N}_{\\delta_i}(\\bz), \\, i \\in J$. Let $\\delta'_{\\min} = \\min(\\delta_i),\\, i \\in J$ and $\\delta'' = \\min(\\delta_{\\min},\\delta'_{\\min})$. This means that for every ${\\u} \\in \\mathcal{N}_{\\delta''}(\\bz)$ $\\Phi({\\u},{\\x}^{(i)}_I) > y \\implies  \\hat{\\phi}({\\u}) > y$. The result follows.\n\\end{proof}","start":55814,"end":57933},{"raw_tex":"Consider $\\ocalU$ as defined in \\cref{cor:4}, it follows from \\cref{lem:6} that $\\lim_{\\u\\to \\bz} \\hat{\\phi}(\\u) = \\hat{\\phi}(\\bz)$ where $\\u \\in \\ocalU$. Now, for every $\\epsilon > 0$ there exists a $\\delta > 0$ such that\n\\begin{equation*}\n   \\abs{\\hat{\\phi}(\\u) - \\hat{\\phi}(\\bz)} < \\epsilon\\,\\, \\forall \\u\\,\\, \\in \\mathcal{N}_{\\delta}(\\bz) \\cap \\ocalU\n\\end{equation*}\n\\begin{lemma}\\label{lem:8}\nConsider the value function $\\hat{\\phi}(\\u)$ as defined earlier for $\\textup{\\MICP}_\\u$. There exists a $\\delta > 0$ such that for all $\\u \\in \\mathcal{N}_\\delta(\\bz) \\cap \\ocalU$, $\\displaystyle \\hat{\\phi}(\\u) = \\min_{\\x_I\\in S_{=}^{\\bz}} \\Phi(\\u,\\x_I)$ where $S_{=}^{\\bz} = \\set{\\x_I \\in S_I}{\\Phi(\\bz,\\x_I) = \\hat{\\phi}(\\bz)}$\n\\end{lemma}","start":57934,"end":58673},{"raw_tex":"\\begin{proof}\nWe can partition the index sets corresponding to solutions of $\\CPui$ into two sets, \n$$\nS_{>}^{\\bz} = \\set{\\x_I\\in S_I}{\\Phi(\\bz,\\x_I) > \\hat{\\phi}(\\bz)} \\text{ and } S_{=}^{\\bz} = \\set{\\x_I\\in S_I}{\\Phi(\\bz,\\x_I) = \\hat{\\phi}(\\bz)}.\n$$\nFor each $\\x^{(i)}_I \\in S_I$, define $ \\overline{\\calU}_i=\\set{\\u \\in \\calU}{\\Phi({\\u},\\x^{(i)}_I)\\leq \\Phi(\\bz,\\x^{(i)}_I)} $. If there is no limiting sequence to ${\\bz}$ in $\\overline{\\calU}_i$ then there exists a neighbourhood $\\delta_i$ such that $\\mathcal{N}_{\\delta_i}(\\bz) \\cap \\overline{\\calU}_i = \\emptyset$. Alternatively, if there does exist a limiting sequence to $\\bz$ and since $\\Phi({\\u},{\\x}_I^{(i)})$ is lower semi-continuous,\n\\begin{equation*}\n    \\lim_{{\\u}\\to \\bz} \\Phi({\\u},{\\x}^{(i)}_I) = \\Phi(\\bz,{\\x}^{(i)}_I)\n\\end{equation*}\nThus, for every $\\epsilon_i > 0$ there exists a $\\delta_i > 0$ such that\n\\begin{align*}\n    \\abs{\\Phi({\\u},{\\x}^{(i)}_I) - \\Phi(\\bz,{\\x}^{(i)}_I)}< \\epsilon_i \\,\\,\\,\\,\\, \\forall \\,\\,\\,\\,{\\u}\\in \\overline{\\calU}_i\\cap \\mathcal{N}_{\\delta_i}(\\bz)\n\\end{align*}\nFor $\\x^{(i)}_I \\in S_{>}^{\\bz}$, let $\\epsilon_i = \\Phi(\\bz,{\\x}^{(i)}_I) - \\hat{\\phi}(\\bz)$. Assuming that $\\Phi(\\bz,{\\x}_I^{(i)})$ is finite, $\\forall$ $\\u \\in \\overline{\\calU}_i\\cap \\mathcal{N}_{\\delta_i}(\\bz)$\n\\begin{align*}\n    \\Phi(\\bz,{\\x}^{(i)}_I) - \\Phi({\\u},{\\x}_I^{(i)}) & <  \\Phi(\\bz,{\\x}^{(i)}_I) - \\hat{\\phi}(\\bz)\\\\\n    \\hat{\\phi}(\\bz) & < \\Phi({\\u},{\\x}^{(i)}_I)\n\\end{align*}\nThus, for all $\\x^{(i)}_I\\in S_{>}^{\\bz}$ there exists a $\\delta_i$ such that $\\Phi({\\u},{\\x}_I^{(i)}) > \\hat{\\phi}(\\bz)$. Assigning $\\delta_{\\min} = \\min_{\\x^{(i)}_I\\in S^{\\bz}_{>}} \\delta_i$ yields $\\displaystyle \\hat{\\phi}({\\u}) = \\min_{\\x_I\\in S^{\\bz}_{=}}\\Phi({\\u},{\\x}_I)$ for every $\\u \\in \\mathcal{N}_{\\delta_{\\min}}(\\bz)\\cap \\overline{\\calU}$.\n\\end{proof}","start":58674,"end":60491},{"raw_tex":"\\begin{lemma}\\label{lem:9}\nConsider the value function $\\hat{\\phi}(\\u)$ as defined earlier for $\\textup{\\MICP}_\\u$. There exists a $\\delta > 0$ and $0 < \\Gamma< \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $$\\hat{\\phi}(\\bz)\\leq \\hat{\\phi}(\\u) + \\Gamma\\norm{\\u}.$$\n\\end{lemma}","start":60492,"end":60796},{"raw_tex":"\\begin{proof}\nConsider $S^{\\bz}_{=}$ as defined in \\cref{lem:8}. Analogously, define $$S_{=}^{\\u} \\define \\set{\\x_I\\in S_{=}^{\\bz}}{\\Phi(\\u,\\x_{I}) = \\hat{\\phi}(\\u)}.$$ For $\\x^{(i)}_I \\in S^{\\u}_{=}$, the first order necessary conditions for optimality (KKT conditions) for $\\CPui$ can be expressed as, %For  writing the KKT conditions\\\\\n\\begin{align}\\label{eq:32}\n\\begin{split}\n    \\nabla_{C}{f(\\x_I^{(i)},\\x_C}) & = \\A^\\top_C\\blambda^{\\u(i)}_{\\A_C} - \\E_C^\\top\\blambda^{\\u(i)}_{\\E_C}\\\\\n    \\A_C\\x_C & = \\b + \\u - \\A_I\\x^{(i)}_I\\\\\n    \\E_C\\x_C+\\E_I\\x^{(i)}_I & \\leq \\f\\\\\n    {\\blambda^{\\u(i)}_{\\E_C}}^\\top(\\f - \\E_C\\x_C - \\E_I\\x^{(i)}_I) & = \\bz\\\\\n    {\\blambda^{\\u(i)}_{\\A_C}}^\\top(\\A_C\\x_C + \\A_I\\x^{(i)}_I - \\b - \\u) & = \\bz\\\\\n    {\\blambda^{\\u(i)}_{\\E_C}} & \\geq \\bz\n\\end{split}\n\\end{align}\nDefine $H^{(j)}_{\\u} = \\set{\\x_C \\in \\R^{n_2}}{\\A_C\\x_C = \\b +\\u- \\A_I\\x^{(j)}_I}$. Let $(\\x_I^{(j)},\\x_C) \\in X \\cap H^{(j)}_{\\u}$ where $\\x_I^{(j)} \\in S_{=}^{\\u}$.\nAdditionally, let the corresponding Lagrange multipliers at $\\u = \\bz$ for $\\x^{(j)}_I$ be $\\blambda^{(j)}_{\\A_C}$ and $\\blambda^{(j)}_{\\E_C}$.\n\\begin{align*}\n    f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\nabla_C{f(\\x_I^{(j)}}\\x^{\\bz*}_C)^{\\top}(\\x^{\\u}_C -  \\x^{\\bz*}_C )\\\\ \n    & \\hspace{3cm}+\\nabla_I{f(\\x_I^{(j)}}, \\x^{\\bz*}_C)^{\\top}(\\x^{(j)}_I -  \\x^{(j)}_I)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) + ( \\blambda^{(j)^\\top}_{\\A_C}\\A_C - \\blambda^{(j)^\\top}_{\\E_C}\\E_C) (\\x^{\\u}_C -  \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}\\A_C(\\x^{\\u}_C -  \\x^{\\bz*}_C) - \\blambda^{(j)^\\top}_{\\E_C}\\E_C (\\x^{\\u}_C -  \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\A_C \\x^{\\u}_C -  \\A_C \\x^{\\bz*}_C) - \\blambda^{(j)^\\top}_{\\E_C}(\\E_C \\x^{\\u}_C -  \\E_C \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\b+\\u-\\A_I\\x^{(j)}_I - \\b + \\A_I\\x^{(j)}_I)\\\\ \n    & \\hspace{3cm}-\\blambda^{(j)^\\top}_{\\E_C}(\\E_C \\x^{\\u}_C -  \\f + \\E_I\\x^{(j)}_I)\n\\end{align*}\nwhere the equalities follow from the set of equations \\cref{eq:32}. Furthermore, observe that since $\\blambda^{(j)^\\top}_{\\E_C} \\geq \\bz$ and $\\E_C\\x^{\\u}_C + \\E\\x^{(j)}_I \\leq \\f$, it follows that \n\\begin{align*}\n   f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u), \\\\\n      \\inf_{X\\cap H^{(j)}_{\\u}}f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u), \\text{ i.e.,}\\\\\n   \\hat{\\phi}({\\u)} & \\geq \\hat{\\phi}(\\bz) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u)\n\\end{align*}\nRearranging the terms in the above inequality,     $-\\blambda^{(j)^\\top}_{\\A_C}\\u \\geq \\hat{\\phi}(\\bz) - \\hat{\\phi}({\\u)}$. As $\\hat{\\phi}(\\bz)\\geq \\hat{\\phi}(\\u)$ $\\forall\\,\\,\\, \\u \\in \\ocalU$, using Cauchy-Schwarz on the left-hand side of the inequality,\n\\begin{align*}\n    \\norm{\\blambda^{(j)}_{\\A_C}}\\cdot\\norm{\\u} \\geq \\hat{\\phi}(\\bz) - \\hat{\\phi}({\\u)} .\n\\end{align*}\nDefining $\\Gamma = \\max_{i\\in S^{\\bz}_{=} }{\\norm{\\blambda^{(j)}_{\\A_C}}}$ yields\n$$\n    \\hat{\\phi}(\\bz) - \\hat{\\phi}(\\u)\\leq \\Gamma\\norm{\\u}\\,\\,\\forall\\,\\, \\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap\\ocalU\n$$\nwhere $\\delta = \\delta_{min}$ as defined in the proof of \\cref{lem:8}. The result follows.\n\\end{proof}","start":60797,"end":64038},{"raw_tex":"This completes the proof of \\Cref{thm:3}(a). In the following section we present some cases where an equivalence can be established between (MICP) and (MICP) with bounded integer variables. In particular, we highlight that  if the objective function $f$ satisfies $\\rec(f)\\cap\\rec(F_R)\\setminus\\{\\bz\\} = \\emptyset$, then there exists $M < \\infty$ such that the following equivalence holds.\n$$ \\min\\set{f(\\x)}{\\A\\x = \\b, \\x \\in X} = \\min\\set{f(\\x)}{\\A\\x = \\b, \\norm{\\x}_{\\infty} \\leq M, \\x \\in X}.$$ Observe that if $f$ is $\\mu$-strongly convex, then $\\rec(f) = \\emptyset$, as the level sets of $f$ are bounded. \n","start":64039,"end":64651},{"raw_tex":"\\begin{lemma}\\label{lem:15}\nLet  $\\delta > 0$, and let $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$. Define\n$$\n    S^\\u \\define  \\set{\\x \\in X \\cap \\Hu}{ f(\\x) \\leq \\zIP}.\n$$\nIf \\textup{\\MICP} is feasible ($F \\neq\\emptyset$) and bounded and the recession cone of $f$ and recession cone of $F_R$, the feasible set of continuous relaxation of \\textup{\\MICP}, have no common non-zero directions of recession, then  $S^\\u$ is bounded.\n\\end{lemma}","start":75106,"end":75549},{"raw_tex":"\\begin{proof}\nConsider $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$. Since $\\norm{\\u}_\\infty \\leq \\norm{\\u} < \\delta$ we have $\\norm{\\b - \\A\\x}_{\\infty} < \\delta$, $\\forall\\,\\x \\in X \\cap \\Hu$. Alternatively, $\\forall\\,\\x \\in X \\cap \\Hu$, $-\\1\\delta < {\\b} - \\A\\x < \\delta\\1$. Define, for $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$\n\\begin{align*}\n  S^\\u \\define  \\set{\\x \\in X \\cap \\Hu}{ f(\\x) \\leq \\zIP}\n\\end{align*}\nAdditionally, consider\n\\begin{align*}\n   S = \\set{\\x\\in X_R}{\\begin{Bmatrix}\n                                    -{\\1}\\delta \\leq {\\b} - \\A\\x \\leq {\\1}\\delta \\\\\n                                    f(\\x) \\leq z_{IP}\n                        \\end{Bmatrix}}\n\\end{align*}\n%Here ${\\u}$ is taken over all feasible polyhedra i.e. this set $\\{\\A\\x = {\\b} + {\\u},\\,\\, {\\E}\\x\\leq {\\f}\\,\\, ,\\x\\in \\mathbb{Z}^{n_1}\\times \\mathbb{R}^{n_2}\\}$ is feasible.\nIndeed, $S^\\u \\subseteq S$. Furthermore, $\\rec(S) = \\rec(F_R) \\cap \\rec(f)$, and since $\\rec(F_R) \\cap \\rec(f) = \\{\\bz\\}$, it follows that $S$ is compact and consequently $S^\\u$ is bounded. In particular, $\\norm{\\x}_\\infty \\leq M < \\infty$, $\\forall$ $\\x \\in S^\\u$. %since MICP is feasible and optimally bounded S'' is non-empty.\\\\\n%This implies that one can prescribe a uniform bound on variables of the problem and hence $\\norm{\\x_I\\norm{\\leq M$.\n\\end{proof}","start":75550,"end":76873},{"raw_tex":"\\begin{corollary}\\label{cor:16}\nIf \\textup{\\MICP} is feasible ($F \\neq\\emptyset$) and bounded and the recession cone of $f$ and recession cone of $F_R$, the feasible set of continuous relaxation of \\textup{\\MICP}, have no common non-zero directions of recession, then for ${\\u} \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$, $\\phi(\\u) = \\hat{\\phi}(\\u)$ i.e.\n\\begin{optprog*}\n    {$\\phi(\\u) = $ minimize}  & \\objective{\\hspace{1.3cm}f({\\x_I},{\\x_C})}\\\\\n    subject to & \\hspace{0cm} \\A_I\\,\\x_I + \\A_C\\x_C = \\b + \\u\\\\\n    & \\E_I\\x_I + \\E_C\\x_C \\leq \\f \\hspace{7mm}\\\\\n    & \\norm{\\x_I}_{\\infty}\\leq M \\hspace{0.5cm}\\\\\n    % & \\x_I\\in \\Z_{+}^{n_1}\\hspace{0.4cm}\\\\ \n     & (\\x_I, \\x_C) \\in \\Z^{n_1}\\times\\R^{n_2}\\hspace{0.3cm}\n\\end{optprog*}\nFurthermore, there exists a $\\delta > 0$ and $0 < \\Gamma < \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $\\phi(\\bz)\\leq \\phi(\\u) + \\Gamma\\norm{\\u}$. Additionally, if $f$ is $\\mu$-strongly convex and $L$ smooth, then $\\Gamma = \\mathcal{O}\\left(\\frac{L\\beta\\gamma}{\\mu}\\right)$ where $\\displaystyle \\beta \\define \\max_{B \\in \\mathcal{B}}\\norm{B^{-1}}_F$, $\\mathcal{B}$ being the set of all possible invertible submatrices of $[\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^T_C]$ and $\\gamma$ depends on $f(\\bz)$, $\\norm{\\nabla{f(\\bz)}}$ and $f(\\bar{\\x})$ for any $\\bar{\\x} \\in F$.\n\\end{corollary}","start":76941,"end":78288},{"raw_tex":"\\begin{proof}\n\\Cref{lem:15}, along with \\cref{lem:8,lem:9} readily implies that $\\phi(\\u) = \\hat{\\phi}(\\u)$ and there exists a $\\delta > 0$ and $0 < \\Gamma < \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $\\phi(\\bz)\\leq \\phi(\\u) + \\Gamma\\norm{\\u}$.\n\nTo see the explicit bound on $\\Gamma$, recall the stationarity condition from set of equations \\cref{eq:32} at $\\u = \\bz$.\n\\begin{align}\\label{eq:stationarity}\n\\begin{split}\n    \\nabla_{C}{f(\\x_I^{(i)},\\x_C}) & = \\A^T_C(\\lambda^{(i)+}_{\\A_C} - \\lambda^{(i)-}_{\\A_C}) - {\\E}_C^T\\lambda_{{\\E}_C}\\\\\n    \\lambda_{\\E_C}^{\\top}(\\E\\x -\\f) &= 0\\\\\n    \\lambda^{+}_{\\A_C},\\lambda^{-}_{\\A_C} & \\geq \\bz\\\\\n     \\lambda_{{\\E}_C} &\\geq \\bz\n\\end{split}\n\\end{align}\n\nFor $(\\x_C,\\x^{(i)}_I)$ consider the complementary slackness conditions, that is if for some $j\\in [m]$, $\\E_{j}\\x < \\f_j$ where $\\E_{j}$ is a row of the matrix $\\E$ then the corresponding Lagrangian multiplier $\\lambda_{\\E_j} = 0$. Partition the set $[m]$ into two sets $J = \\{J_<,J_=\\}$, where $J_<=\\set{j\\in [m]}{\\E_j \\x <\\f_j}$ and $J_= =\\set{j\\in [m]}{\\E_j \\x = \\f_j}$.\\\\\n\nDefine $\\A_{aug} = [\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^{\\top}_C]$ and $\\A_{aug}^J$ as the sub-matrix which has all the columns of $\\A_{aug}$ except the columns $2n_2+J_<$.\n % $\\lambda^{(i)+}_{\\A_C},\\lambda^{(i)-}_{\\A_C}$. \n One possible solution to the system \\cref{eq:stationarity} is,\n \\begin{align}\\label{sol:lambda}\n    \\begin{pmatrix}\n     \\lambda^{+}_{\\A_C}\\\\\n      \\lambda^{-}_{\\A_C}\\\\\n      \\lambda_{{\\E}^{J_{=}}_C}\n    \\end{pmatrix} = B^{-1} \\nabla_C{f(\\x^{(i)}_I,\\x_C)}\n\\end{align}\nwhere $B$ is a basis matrix of $\\A_{aug}^J$.\n\nFurthermore, let $\\lambda_{\\A_C} = \\lambda^{+}_{\\A_C} - \\lambda^{-}_{\\A_C}$, this implies that \n\\begin{align}\n    \\norm{\\lambda_{\\A_C}} \\leq \\norm{\\lambda^{+}_{\\A_C}} + \\norm{\\lambda^{-}_{\\A_C}} \\label{ineq:lambda}    \n\\end{align}\nLet $\\x^*$ be an optimal solution to \\MICP, i.e. $f(\\x^*) = \\zIP$. Since $f$ is $\\mu$-strongly convex, we have\n\n\\begin{align*}\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 + \\nabla{f({\\bz)}}^\\top\\x^*  + f(\\bz) & \\leq z_{IP}\\\\\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 + \\nabla{f({\\bz)}}^\\top\\x^* - z_{IP} + f(\\bz)& \\leq 0\\\\\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 - \\norm{\\nabla{f({\\bz)}}}\\,\\norm{\\x^*} - \\left(z_{IP} - f(\\bz)\\right) & \\leq 0\n\\end{align*}\nIt follows that,\n$$\n    \\norm{\\x^*}\\leq \\frac{\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz})}^2 +2\\,\\mu\\cdot \\left(z_{IP} - f(\\bz)\\right)}}{\\mu}\n$$\nSince $f$ is L-smooth,\n\\begin{align*}\n    \\norm{\\nabla{f(\\x^*)} - \\nabla{f(\\bz)}} &\\leq L\\norm{\\x^*}\\\\\n    \\norm{\\nabla{f(\\x^*)}} &\\leq L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\n\\end{align*}\nFrom \\cref{sol:lambda} and \\cref{ineq:lambda} we obtain\n\\begin{align*}\n    \\norm{\\lambda_{\\A_C}} & \\leq 2\\norm{B^{-1}}_F \\norm{\\nabla{f(\\x^*)}}\\\\\n    & \\leq 2\\norm{B^{-1}}_F\\left(L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\\right)\\\\\n    & \\leq 2\\beta\\left(L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\\right)\n\\end{align*}\nwhere $\\beta \\define \\max_{B \\in \\mathcal{B}}\\norm{B^{-1}}_F$ and $\\mathcal{B}$ is the set of all possible invertible submatrices of $[\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^T_C]$.\n\n\\begin{align}\n   \\norm{\\lambda_{\\A_C}} & \\leq \\dfrac{2\\beta\\left(L\\left(\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz})}^2 +2\\mu\\,\\left(z_{IP} - f(\\bz)\\right)}\\right) + \\mu\\norm{\\nabla{f(\\bz)}}\\right)}{\\mu}\\nonumber\\\\\n    & \\leq  \\dfrac{2\\beta L}{\\mu}\\left(2\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz)}}^2 + 2\\mu\\,\\left(f(\\bar{\\x}) - f(\\bz)\\right)}\\right)\\label{ineq:thm3}\n\\end{align}\nwhere the second inequality follows as $\\zIP \\leq f(\\bar{\\x}),$ $\\forall\\,\\bar{\\x} \\in F$ and $L > \\mu$. Combining \\cref{ineq:thm3} with the construction of $\\Gamma$ in \\cref{lem:9}, yields the result.\n\\end{proof}","start":78289,"end":82023},{"raw_tex":"\\cref{thm:3} follows.\n","start":82024,"end":82046}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"混合整数凸规划","source_units":[{"raw_tex":"Consider, to begin with, the following mixed integer convex programming (MICP) problem,\n\\begin{optprog*}\n    {$\\zIP = $ minimize}  & \\objective{\\hspace{3mm}f(\\x)}\\\\\n   \\textbf{(MICP)} \\qquad subject to & \\A\\,\\x\\,= \\b\\\\\n     & \\x \\in X \\hspace{1mm}\n\\end{optprog*}\nwhere $X:= \\set{\\x\\in \\Z^{n_1}\\times\\R^{n_2}}{\\E\\x\\leq \\f}$ is a polyhedral mixed integer set, $f:\\R^n \\mapsto \\R$ is a real valued convex function, $\\A,\\E$ and $\\b,\\f$ are full rank matrices and vectors of appropriate dimensions, respectively. Through out the remainder of this discussion we assume that the matrices $\\A, \\E$ are rational matrices and $\\b, \\f$ are rational vectors and $f(\\cdot)$ is differentiable. We further assume that both MICP and MICP's continuous relaxation are feasible and the corresponding optimal solutions exist.\\\\\n","start":10244,"end":11052}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"有理数据与最优取到","source_units":[{"raw_tex":"where $X:= \\set{\\x\\in \\Z^{n_1}\\times\\R^{n_2}}{\\E\\x\\leq \\f}$ is a polyhedral mixed integer set, $f:\\R^n \\mapsto \\R$ is a real valued convex function, $\\A,\\E$ and $\\b,\\f$ are full rank matrices and vectors of appropriate dimensions, respectively. Through out the remainder of this discussion we assume that the matrices $\\A, \\E$ are rational matrices and $\\b, \\f$ are rational vectors and $f(\\cdot)$ is differentiable. We further assume that both MICP and MICP's continuous relaxation are feasible and the corresponding optimal solutions exist.\\\\\n","start":10507,"end":11052}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"拉格朗日增广模型","source_units":[{"raw_tex":"Consider the Lagrangian relaxation of the MICP,\n\\begin{equation*}\n    \\textbf{(LR)} \\hspace{1cm} z^{LR}(\\blambda) = \\min_{\\x \\in X} f(\\x) + {\\blambda}^{\\top}(\\b - \\A\\,\\x).\n\\end{equation*}\nThe corresponding augmented Lagrangian relaxation and augmented Lagrangian dual (\\cite{boland2015augmented,gu2020exact}) of the \\MICP\\, are defined as,\n\\begin{align*}\n    \\textbf{(ALR)} \\hspace{0.5cm} z_{\\rho}^{LR+}(\\blambda) & = \\min_{\\x \\in X} f(\\x) + {\\blambda}^{\\top}(\\b - \\A\\,\\x) + \\rho\\, \\psi(\\b - \\A\\,\\x)\\\\\n    \\textbf{(ALD)} \\hspace{1cm} z_{\\rho}^{LD+} & = \\sup_{\\blambda\\in \\R^m}\\min_{\\x \\in X} f(\\x) + {\\blambda}^{\\top}(\\b - \\A\\,\\x) + \\rho\\, \\psi(\\b - \\A\\,\\x)\n\\end{align*}\n","start":11053,"end":11724}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"精确罚表示","source_units":[{"raw_tex":"where $\\psi:\\R^m \\mapsto \\R$ is a real valued function. We further designate $\\psi$ to have an \\textit{exact penalty representation} if $\\exists\\, 0 < \\rho < \\infty$, such that $z^{LD+}_{\\rho} = \\zIP$. In the case $\\psi(\\cdot) = \\norm{\\cdot}$ we augmented Lagrangian relaxation is referred to as sharp augmented Lagrangian relaxation. In particular, \n","start":11724,"end":12075}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"尖锐罚松弛","source_units":[{"raw_tex":"where $\\psi:\\R^m \\mapsto \\R$ is a real valued function. We further designate $\\psi$ to have an \\textit{exact penalty representation} if $\\exists\\, 0 < \\rho < \\infty$, such that $z^{LD+}_{\\rho} = \\zIP$. In the case $\\psi(\\cdot) = \\norm{\\cdot}$ we augmented Lagrangian relaxation is referred to as sharp augmented Lagrangian relaxation. In particular, \n$$\n    \\textbf{(SALR)} \\hspace{1cm} z_{SALR}(\\rho) = \\min_{\\x \\in X} f(\\x) + \\rho\\, \\norm{\\b - \\A\\,\\x}\\\\\n$$\n","start":11724,"end":12183}]},{"id":"S006","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"松弛上界与零乘子","source_units":[{"raw_tex":"It follows that $z^{LR+}_{\\rho}(\\blambda) \\leq \\zIP$ and $z^{LR+}_{\\rho}(\\bz) = z_{SALR}(\\rho)$. \n","start":12183,"end":12281}]},{"id":"S007","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"变量分块与衰退锥","source_units":[{"raw_tex":"\\subsection{Notation}\\label{sec:notation} The sets $\\R^n,\\Z^n,\\mathbb{Q}^n$ denote the set of real numbers, integers and rational numbers in $n-$dimensional vector space, respectively. Additionally, $\\R^n_+,\\Z^n_+,\\mathbb{Q}^n_+$ denote the non-negative counterparts of the respective sets. We use $[n]$ to denote the index set $\\{1,2 \\ldots, n\\}$. For an index set $E \\subset \\Z^+$ and $y\\in \\Z$, we define the translation of an index set as $y + E \\define \\set{x + y}{x \\in E}$. For a mixed integer set $S \\subseteq \\R^n \\times \\Z^m$, we denote by $S_R$ the continuous relaxation of $S$. Throughout the following discussion we define $I$ and $C$ as the index sets corresponding to integer and continuous variables, respectively. Analogously, for any $\\u \\in \\R^n$, let $\\u = (\\u_I,\\u_C)$ denote the partition of $\\u$ into the integer and continuous variables, respectively. Similarly for a symmetric matrix $\\Q \\in \\R^{n\\times n}$, let $Q_{UV}$ denote the submatrix of $\\Q$ formed by rows indexed by $U$ and columns indexed by $V$. For $\\delta > 0$ and $\\bar{\\x} \\in \\R^n$, we define by $\\mathcal{N}_{\\delta}(\\bar{\\x}) \\define \\set{\\x \\in \\R^n}{\\norm{\\x -\\bar{\\x}}\\leq \\delta}$, the ball with center $\\bar{\\x}$ and radius $\\delta$. The recession cone of a set $S$ is denoted by $\\rec(S)$. We also define, $\\rec(f)$, \\emph{recession cone of a convex function} $f:\\R^n \\mapsto \\R$ as the recession cone of a level set $\\set{x \\in \\R^n}{f(\\x)\\leq \\alpha}$, for some $\\alpha \\in \\R$ (\\cite{bertsekas2003convex}, Proposition 2.3.1). For given convex sets $A$ and $B$, we say that the sets $A$ and $B$ have \\emph{no common/distinct non-zero directions of recession} if $\\rec(A)\\cap \\rec(B)\\setminus \\{\\bz\\} = \\emptyset$.\\\\\n","start":12281,"end":14000}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"光滑性定义","source_units":[{"raw_tex":"We say that a function $g : \\R^m \\mapsto \\R$ is $L$ smooth, or alternatively, has \\textbf{Lipschitz-continuous} gradients if there exists a Lipschitz\nconstant, $L < \\infty$, such that\n$$\n    \\norm{\\nabla{g(\\x)} - \\nabla{g(y)}}\\leq\n    L\\norm{\\x-\\y}~~~~ \\forall~~ \\x,\\y \\in \\R^m.\n$$\n","start":14001,"end":14283}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"原文强凸性定义","source_units":[{"raw_tex":"Additionally, we say a function $g : \\R^m \\mapsto \\R$ is $\\mu$ \\textbf{strongly-convex} if there exists $ 0 <\\mu < \\infty$, such that\n$$\n    \\norm{\\nabla{g(\\x)} - \\nabla{g(y)}}\\geq\\mu\\norm{\\x-\\y}~~~~ \\forall~~ \\x,\\y \\in \\R^m.\n$$\n","start":14283,"end":14512}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":["def:valuefunction"],"declared_external":false,"source_only":false,"title":"扰动值函数","source_units":[{"raw_tex":"Consider the hyperplane $H_{\\u} \\define \\set{\\x\\in\\R^n}{\\A\\x = \\b + \\u}$, and let the corresponding value function associated with {\\MICP} be defined as \n\\begin{align}\n    \\phi(\\u) = \\min\\set{f(\\x)}{\\x \\in X\\cap\\Hu}.\\label{def:valuefunction}\n\\end{align}\nIn particular, $\\zIP = \\phi(\\bz) = \\min_{\\x\\in X \\cap \\Ho} f(\\x)$. Finally, we define the set $\\calU$ as the set of all possible perturbation vectors $\\u$ such that the feasible set of \\cref{def:valuefunction} is non-empty. In particular, $\\calU \\define \\set{\\u\\in \\R^m}{X \\cap \\Hu \\neq \\emptyset}$.\n","start":14922,"end":15476}]},{"id":"S011","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"低目标子集约化","source_units":[{"raw_tex":"Consider the partition of the feasible set $X = F_> \\cup F_{\\leq}, F_> \\cap F_{\\leq} = \\emptyset$ where $F_{\\leq} = \\set{\\x \\in X}{f(\\x) \\leq \\zIP}$ and $F_> =\\set{\\x \\in X}{f(\\x) > \\zIP}$. Observe that for $\\x \\in F_>$ any positive $\\rho > 0$ implies $f(\\x) + \\rho\\norm{\\b - \\A\\,\\x} > \\zIP$. \nThus, for MICP to have an exact penalty representation, it suffices to show that $\\exists$ $0 < \\rho < \\infty$ such that $f(\\x) + \\rho \\norm{\\b - \\A\\,\\x} \\geq \\zIP$, $\\forall \\x \\in F_{\\leq}$.\n","start":22903,"end":23390}]},{"id":"S012","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"统一整数界扰动模型","source_units":[{"raw_tex":"Consider the following mixed integer convex programming problem.\n\\begin{optprog*}\n    {$\\hat{\\phi}(\\u) = $ minimize}  & \\objective{\\hspace{1.3cm}f({\\x_I},{\\x_C})}\\\\\n    $\\textup{\\MICP}_\\u$ \\qquad subject to & \\hspace{0cm} \\A_I\\,\\x_I + \\A_C\\x_C = \\b + \\u\\\\\n    & \\E_I\\x_I + \\E_C\\x_C \\leq \\f \\hspace{7mm}\\\\\n    & \\norm{\\x_I}_{\\infty}\\leq M \\hspace{0.5cm}\\\\\n    % & \\x_I\\in \\Z_{+}^{n_1}\\hspace{0.4cm}\\\\ \n    & (\\x_I, \\x_C) \\in \\Z^{n_1}\\times\\R^{n_2}\\hspace{0.3cm}\n\\end{optprog*}\nwhere $0 \\leq M < \\infty$. Denote by $F_{\\u} \\define X \\cap \\Hu \\cap S_I $ the feasible region of $\\textup{\\MICP}_\\u$, where the set $S_I = \\set{\\x_I \\in \\Z^{n_1}}{\\norm{\\x_I}_{\\infty} \\leq M}$ is finite.  As before, we let $S_I = \\{\\x^{(1)}_I, \\x_I^{(2)}, \\ldots, \\x_I^{(k)}\\}$. Using this enumeration we characterize the value functions of the continuous restrictions, parameterized in $\\x_I^{(i)},\\,i \\in [k]$ as,\n","start":54192,"end":55085}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"固定整数连续限制","source_units":[{"raw_tex":"\\begin{optprog*}\n    {$\\Phi(\\u,\\x^{(i)}_I) = $ minimize}  & \\objective{\\hspace{6mm}f(\\x^{(i)}_I,\\x_C)}\\\\\n    $\\CPui$ \\qquad subject to & \\A_C\\,\\x_C = \\b + \\u - \\A_I\\x^{(i)}_{I}\\\\\n    & \\E_C\\x_C \\leq \\f - \\E_I\\x^{(i)}_{I}\\hspace{8mm}\\\\\n     & \\x_C \\in \\R^{n_2}\\hspace{1.8cm}\n\\end{optprog*}\nwhere $\\Phi(\\u,\\x^{(i)}_I)$ is the value function of $\\CPui$, a continuous convex  optimization problem. Since the constraints of $\\CPui$ are linear, Slater's condition holds and therefore the value function is lower semi-continuous.\n","start":55085,"end":55608}]},{"id":"S014","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"连续限制下半连续性","source_units":[{"raw_tex":"where $\\Phi(\\u,\\x^{(i)}_I)$ is the value function of $\\CPui$, a continuous convex  optimization problem. Since the constraints of $\\CPui$ are linear, Slater's condition holds and therefore the value function is lower semi-continuous.\n","start":55374,"end":55608}]},{"id":"S015","semantic_type":"theorem","kind":"theorem","labels":["thm:3"],"declared_external":false,"source_only":false,"title":"定理三精确罚表示","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm:3}\nConsider the following mixed integer convex programming problem,\n\\begin{optprog*}\n    minimize  & \\objective{\\hspace{1cm}f({\\x_I},{\\x_C})}\\\\\n    \\textup{\\MICP} \\hspace{2mm} subject to & \\hspace{0cm} \\A_I\\,\\x_I + \\A_C\\x_C = \\b\\hspace{6.5mm}\\\\\n    & \\E_I\\x_I + \\E_C\\x_C \\leq \\f \\hspace{7mm}\\\\\n    % & \\norm{\\x_I}_{\\infty}\\leq M \\hspace{0.5cm}\\\\\n    % & \\x_I\\in \\Z_{+}^{n_1}\\hspace{0.4cm}\\\\ \n     & (\\x_I, \\x_C) \\in \\Z^{n_1}\\times\\R^{n_2}\\hspace{0.3cm}\n\\end{optprog*}\nLet $F = X\\cap \\Ho$ denote the feasible region of \\textup{\\MICP}.\\\\\n\\begin{enumerate}[label = (\\alph*)]\n    \\item If $\\norm{\\x_I}_{\\infty}\\leq M,$ $\\forall\\,\\x \\in F$ then there exists an exact penalty representation for \\textup{\\MICP}.\n    \\item If the recession cone of $f$ and recession cone of $F_R$, the feasible set of continuous relaxation of \\textup{\\MICP}, have no common non-zero directions of recession, then there exists an exact penalty representation for \\textup{\\MICP}.\n\\end{enumerate} \nAdditionally, if $f$ is $\\mu$-strongly convex and $L$ smooth, then the finite penalty parameter $\\rho = \\mathcal{O}\\left(\\frac{L\\beta\\gamma}{\\mu}\\right)$ where $\\displaystyle \\beta \\define \\max_{B \\in \\mathcal{B}}\\norm{B^{-1}}_F$, $\\mathcal{B}$ being the set of all possible invertible submatrices of $[\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^T_C]$ and $\\gamma$ can be explicitly computed given $f(\\bz)$, $\\norm{\\nabla{f(\\bz)}}$ and $f(\\bar{\\x})$ for any $\\bar{\\x} \\in F$.\n% $\\gamma = \\sqrt{f(\\bar{\\x})}$ for any $\\bar{\\x} \\in F$.\\\\\n\\end{theorem}","start":19365,"end":20905}]},{"id":"S016","semantic_type":"lemma","kind":"lemma","labels":["lem:1"],"declared_external":false,"source_only":false,"title":"SALR值函数表示","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:1}\n$\\displaystyle z_{SALR}(\\rho) = \\min_{\\u \\in \\calU}\\,\\, \\phi(\\u) + \\rho\\norm{\\u}$ . \n\\end{lemma}","start":21629,"end":21752}]},{"id":"S017","semantic_type":"proof","kind":"proof","labels":["eq:1","eq:2"],"declared_external":false,"source_only":false,"title":"值函数表示作者证明","source_units":[{"raw_tex":"\\begin{proof}\nObserve the following inequality which follows immediately from the aforementioned definitions\n\\begin{align}\n    z_{SALR}(\\rho) = \\min_{\\x \\in X} f(\\x) + \\rho\\, \\norm{\\b - \\A\\,\\x} & \\leq \\min_{\\x\\in X \\cap \\Hu} f(\\x) + \\rho\\norm{\\u} ,\\,\\,\\,\\,\\, \\forall\\,\\u \\in \\calU \\nonumber\\\\\n                    & = \\phi(\\u) + \\rho\\norm{\\u} ,\\,\\,\\,\\,\\, \\forall\\,\\u \\in \\calU. \\label{eq:1}\n\\end{align}\nConversely, it holds that $\\forall\\,\\u \\in \\calU$\n\\begin{align*}\n    \\phi(\\u) + \\rho\\norm{\\u} & = \\min_{\\x\\in X \\cap \\Hu} f(\\x) + \\rho\\norm{\\u}\\\\\n                             & \\leq f(\\x) + \\rho\\norm{\\A\\x-\\b} ,\\,\\,\\,\\,\\, \\forall\\,\\x\\in X\\cap \\Hu.\n\\end{align*}\nThis implies that $ \\min_{\\u \\in \\calU} \\phi(\\u) + \\rho \\norm{\\u} \\leq f(\\x) + \\rho \\norm{\\A\\x - \\b} $ $\\forall~ x\\in X \\cap \\Hu,\\,\\forall~\\u \\in U$ which yields that $ \\min_{\\u \\in \\calU} \\phi(\\u) + \\rho \\norm{\\u} \\leq f(\\x) + \\rho \\norm{\\A\\x - \\b} $ $\\forall~ \\x \\in X $. This implies,\n\\begin{align}\n    \\min_{\\u\\in \\calU} \\phi(\\u) + \\rho\\norm{\\u} \\leq \\min_{\\x\\in X} f(\\x) + \\rho\\norm{\\A\\x-\\b}. \\label{eq:2}\n\\end{align}\nThe result follows from \\cref{eq:1} and \\cref{eq:2}.\n\\end{proof}","start":21753,"end":22902}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","labels":["eq:1"],"declared_external":false,"source_only":false,"title":"限制纤维所得上界","source_units":[{"raw_tex":"Observe the following inequality which follows immediately from the aforementioned definitions\n\\begin{align}\n    z_{SALR}(\\rho) = \\min_{\\x \\in X} f(\\x) + \\rho\\, \\norm{\\b - \\A\\,\\x} & \\leq \\min_{\\x\\in X \\cap \\Hu} f(\\x) + \\rho\\norm{\\u} ,\\,\\,\\,\\,\\, \\forall\\,\\u \\in \\calU \\nonumber\\\\\n                    & = \\phi(\\u) + \\rho\\norm{\\u} ,\\,\\,\\,\\,\\, \\forall\\,\\u \\in \\calU. \\label{eq:1}\n\\end{align}\n","start":21767,"end":22155}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","labels":["eq:2"],"declared_external":false,"source_only":false,"title":"遍历纤维所得下界","source_units":[{"raw_tex":"Conversely, it holds that $\\forall\\,\\u \\in \\calU$\n\\begin{align*}\n    \\phi(\\u) + \\rho\\norm{\\u} & = \\min_{\\x\\in X \\cap \\Hu} f(\\x) + \\rho\\norm{\\u}\\\\\n                             & \\leq f(\\x) + \\rho\\norm{\\A\\x-\\b} ,\\,\\,\\,\\,\\, \\forall\\,\\x\\in X\\cap \\Hu.\n\\end{align*}\nThis implies that $ \\min_{\\u \\in \\calU} \\phi(\\u) + \\rho \\norm{\\u} \\leq f(\\x) + \\rho \\norm{\\A\\x - \\b} $ $\\forall~ x\\in X \\cap \\Hu,\\,\\forall~\\u \\in U$ which yields that $ \\min_{\\u \\in \\calU} \\phi(\\u) + \\rho \\norm{\\u} \\leq f(\\x) + \\rho \\norm{\\A\\x - \\b} $ $\\forall~ \\x \\in X $. This implies,\n\\begin{align}\n    \\min_{\\u\\in \\calU} \\phi(\\u) + \\rho\\norm{\\u} \\leq \\min_{\\x\\in X} f(\\x) + \\rho\\norm{\\A\\x-\\b}. \\label{eq:2}\n\\end{align}\nThe result follows from \\cref{eq:1} and \\cref{eq:2}.\n","start":22155,"end":22891}]},{"id":"S020","semantic_type":"proposition","kind":"proposition","labels":["prop:2"],"declared_external":false,"source_only":false,"title":"扰动值的有限下界","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:2}\nConsider the continuous relaxation of MICP, and let the optimal objective value $z_R = \\min_{X_R\\cap \\Ho} f(\\x)$ be attained at $\\x_R$. \nLet $\\blambda_{\\A}$ and $\\blambda_{\\E}$ be the Lagrange multipliers for the constraints $\\A\\x = \\b$ and $\\E\\x\\leq \\f$, respectively.\\vspace{1mm}\n\\begin{enumerate}[label = \\roman*)]\n    \\item $\\phi(\\u) > -\\infty$ for all $\\u \\in \\calU$.\\vspace{1mm}\n","start":23391,"end":23810}]},{"id":"S021","semantic_type":"proposition","kind":"proposition","labels":["prop:2"],"declared_external":false,"source_only":false,"title":"低罚目标残差界","source_units":[{"raw_tex":"    \\item If $\\exists ~ \\boldsymbol{\\alpha} \\in X_R$ such that $f(\\boldsymbol{\\alpha}) + \\rho\\norm{\\b-\\A\\,\\boldsymbol{\\alpha}}\\leq \\zIP$ and $\\rho > \\norm{\\blambda_\\A}$  then \n        \\[\n                \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq \\frac{\\zIP-z_R}{\\rho -\\norm{\\blambda_{\\A}}}.\n        \\]    \n\\end{enumerate}\n\\end{proposition}","start":23810,"end":24147}]},{"id":"S022","semantic_type":"proof","kind":"proof","labels":["eq:4","ineq:1","ineq:2","ineq:gradient"],"declared_external":false,"source_only":false,"title":"命题二作者证明","source_units":[{"raw_tex":"\\begin{proof}\nConsider the Lagrangian function for the continuous relaxation of MICP, i.e. for $\\x \\in \\R^n$\n\\begin{equation*}\n    \\mathcal{L}(\\x,\\blambda_{\\A},\\blambda_{\\E}) = f(\\x) - \\blambda_\\A^{\\top}(\\A\\,\\x-\\b)-\\blambda_\\E^{\\top}(\\f-\\E\\x)\n\\end{equation*}\nAs the relaxed problem has only affine constraints strong duality holds (Chapter 5,Section 5.2.3\\cite{boyd2004convex}).\n\n As strong duality holds, the first order necessary (KKT) conditions for $\\mathcal{L}(\\x,\\blambda_{\\A},\\blambda_{\\E})$ can be characterized as\n\\begin{align}\\label{eq:4}\n\\begin{split}\n    \\nabla{f(\\x)} = \\A^{\\top}\\blambda_\\A - \\E^{\\top}{\\blambda_\\E}\\\\\n    \\blambda_\\E^{\\top}(\\f - \\E\\x) = 0\\\\\n    \\blambda_\\A^{\\top}(\\A\\,\\x - \\b) = 0\\\\\n    \\blambda_\\E \\geq \\bz\n\\end{split}\n\\end{align}\nObserve that $\\x_R$ satisfies the system of equations \\cref{eq:4}.\\\\\n\\begin{enumerate}[label = \\roman*)]\n    \\item To see the result, observe that it suffices to show that $\\phi(\\u) > -\\infty$ for all $\\u \\in \\calU$ such that $\\phi(\\u) \\leq \\phi(\\bz) = \\zIP$. Further, consider the set $F_{\\leq}$ defined as,\n            $$\n                F_{\\leq} \\define \\set{\\x \\in X}{f(\\x) \\leq \\zIP}.\n            $$\n            We have for all $\\x \\in F_{\\leq}$, $\\zIP \\geq f(\\x) \\geq f(\\x_R) + \\nabla{f(\\x_R)}(\\x-\\x_R)= z_R + \\nabla{f(\\x_R)}(\\x-\\x_R)$. Substituting from \\cref{ineq:gradient} we obtain, for all $\\x \\in F_{\\leq}$,\n            $$\n                \\zIP \\geq f(\\x) \\geq z_R + \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b) -  \\blambda_\\E^{\\top} (\\E\\x -  \\f )\n            $$\n            As $\\blambda_{\\E} \\geq \\bz$ and $\\x \\in X_R$,\n            \\begin{align*}\n                \\zIP \\geq f(\\x) & \\geq z_R + \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b)\\\\\n                & \\geq z_R - \\norm{\\blambda_{\\A}}\\norm{(\\A\\,\\x - \\b)}\\\\\n                & = z_R - \\norm{\\blambda_{\\A}}\\norm{\\u} > -\\infty.\n            \\end{align*}\n            Since $f(\\x) > -\\infty$ for all $\\x \\in F_{\\leq}$, the result follows.\\\\\n    \\item Since we have for $\\boldsymbol{\\alpha}\\in X_R$, $f(\\boldsymbol{\\alpha}) + \\rho\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq  \\zIP$; convexity of $f$ yields,\n            \\begin{align}\n                f(\\x_R)+ \\nabla{f(\\x_R)}^{\\top}(\\boldsymbol{\\alpha}-\\x_R) +\\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} &\\leq \\zIP , \\text{ i.e.,}\\nonumber \\\\\n                z_R+ \\nabla{f(\\x_R)}^{\\top}(\\boldsymbol{\\alpha}-\\x_R) +\\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} &\\leq \\zIP. \\label{ineq:1}\n            \\end{align}\n            It follows from strong duality that\n            \\begin{align}\n            \\nabla{f(\\x_R)}^{\\top}(\\x-\\x_R) &= \\blambda_{\\A}^{\\top} \\A(\\x - \\x_R) -\\blambda_\\E^{\\top} \\E(\\x-\\x_R)\\nonumber\\\\\n             &= \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b) -  \\blambda_\\E^{\\top} (\\E\\x -  \\f ). \\label{ineq:gradient}\n            \\end{align}\n            Substituting in \\cref{ineq:1}, we obtain\n            \\begin{align*}\n            z_R+ \\blambda_{\\A}^{\\top} (\\A\\,\\boldsymbol{\\alpha} - \\b) +  \\blambda_\\E^{\\top} (\\f - \\E\\boldsymbol{\\alpha}) +\\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b}  & \\leq \\zIP .\n            \\end{align*}\n            As $\\blambda_{\\E} \\geq \\bz$ and $\\boldsymbol{\\alpha} \\in X_R$, the above inequality can be rewritten as\n            % \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b) +   \\blambda_\\E^{\\top} ( \\f - \\E\\x ) +\\rho \\norm{\\A\\,\\x - \\b} & \\leq \\zIP - z_R\\\\\n            \\begin{align}\n            \\blambda_{\\A}^{\\top}(\\A\\,\\boldsymbol{\\alpha} - \\b) + \\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} & \\leq \\zIP - z_R \\label{ineq:2}\n            \\end{align}\n            Using Cauchy Schwarz inequality in \\cref{ineq:2} yields for all $\\x \\in X_R$,\n            \\begin{align*}\n                -\\norm{\\blambda_\\A}\\,\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} + \\rho\\,\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq \\zIP - z_R\\\\\n            (\\rho - \\norm{\\blambda_\\A})\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b}\\leq \\zIP - z_R\\\\\n            \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq \\frac{\\zIP-z_R}{\\rho - \\norm{\\blambda_\\A}}.\n            \\end{align*}\n\\end{enumerate}\n\\end{proof}","start":24148,"end":28140}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","labels":["eq:4"],"declared_external":false,"source_only":false,"title":"连续松弛KKT条件","source_units":[{"raw_tex":"Consider the Lagrangian function for the continuous relaxation of MICP, i.e. for $\\x \\in \\R^n$\n\\begin{equation*}\n    \\mathcal{L}(\\x,\\blambda_{\\A},\\blambda_{\\E}) = f(\\x) - \\blambda_\\A^{\\top}(\\A\\,\\x-\\b)-\\blambda_\\E^{\\top}(\\f-\\E\\x)\n\\end{equation*}\nAs the relaxed problem has only affine constraints strong duality holds (Chapter 5,Section 5.2.3\\cite{boyd2004convex}).\n\n As strong duality holds, the first order necessary (KKT) conditions for $\\mathcal{L}(\\x,\\blambda_{\\A},\\blambda_{\\E})$ can be characterized as\n\\begin{align}\\label{eq:4}\n\\begin{split}\n    \\nabla{f(\\x)} = \\A^{\\top}\\blambda_\\A - \\E^{\\top}{\\blambda_\\E}\\\\\n    \\blambda_\\E^{\\top}(\\f - \\E\\x) = 0\\\\\n    \\blambda_\\A^{\\top}(\\A\\,\\x - \\b) = 0\\\\\n    \\blambda_\\E \\geq \\bz\n\\end{split}\n\\end{align}\nObserve that $\\x_R$ satisfies the system of equations \\cref{eq:4}.\\\\\n","start":24162,"end":24979}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"固定扰动线性下界","source_units":[{"raw_tex":"    \\item To see the result, observe that it suffices to show that $\\phi(\\u) > -\\infty$ for all $\\u \\in \\calU$ such that $\\phi(\\u) \\leq \\phi(\\bz) = \\zIP$. Further, consider the set $F_{\\leq}$ defined as,\n            $$\n                F_{\\leq} \\define \\set{\\x \\in X}{f(\\x) \\leq \\zIP}.\n            $$\n            We have for all $\\x \\in F_{\\leq}$, $\\zIP \\geq f(\\x) \\geq f(\\x_R) + \\nabla{f(\\x_R)}(\\x-\\x_R)= z_R + \\nabla{f(\\x_R)}(\\x-\\x_R)$. Substituting from \\cref{ineq:gradient} we obtain, for all $\\x \\in F_{\\leq}$,\n            $$\n                \\zIP \\geq f(\\x) \\geq z_R + \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b) -  \\blambda_\\E^{\\top} (\\E\\x -  \\f )\n            $$\n            As $\\blambda_{\\E} \\geq \\bz$ and $\\x \\in X_R$,\n            \\begin{align*}\n                \\zIP \\geq f(\\x) & \\geq z_R + \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b)\\\\\n                & \\geq z_R - \\norm{\\blambda_{\\A}}\\norm{(\\A\\,\\x - \\b)}\\\\\n                & = z_R - \\norm{\\blambda_{\\A}}\\norm{\\u} > -\\infty.\n            \\end{align*}\n            Since $f(\\x) > -\\infty$ for all $\\x \\in F_{\\leq}$, the result follows.\\\\\n","start":25015,"end":26090}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":["ineq:1","ineq:gradient"],"declared_external":false,"source_only":false,"title":"凸下界与梯度展开","source_units":[{"raw_tex":"    \\item Since we have for $\\boldsymbol{\\alpha}\\in X_R$, $f(\\boldsymbol{\\alpha}) + \\rho\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq  \\zIP$; convexity of $f$ yields,\n            \\begin{align}\n                f(\\x_R)+ \\nabla{f(\\x_R)}^{\\top}(\\boldsymbol{\\alpha}-\\x_R) +\\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} &\\leq \\zIP , \\text{ i.e.,}\\nonumber \\\\\n                z_R+ \\nabla{f(\\x_R)}^{\\top}(\\boldsymbol{\\alpha}-\\x_R) +\\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} &\\leq \\zIP. \\label{ineq:1}\n            \\end{align}\n            It follows from strong duality that\n            \\begin{align}\n            \\nabla{f(\\x_R)}^{\\top}(\\x-\\x_R) &= \\blambda_{\\A}^{\\top} \\A(\\x - \\x_R) -\\blambda_\\E^{\\top} \\E(\\x-\\x_R)\\nonumber\\\\\n             &= \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b) -  \\blambda_\\E^{\\top} (\\E\\x -  \\f ). \\label{ineq:gradient}\n            \\end{align}\n","start":26090,"end":26929}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","labels":["ineq:2"],"declared_external":false,"source_only":false,"title":"删除松弛项并除正量","source_units":[{"raw_tex":"            Substituting in \\cref{ineq:1}, we obtain\n            \\begin{align*}\n            z_R+ \\blambda_{\\A}^{\\top} (\\A\\,\\boldsymbol{\\alpha} - \\b) +  \\blambda_\\E^{\\top} (\\f - \\E\\boldsymbol{\\alpha}) +\\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b}  & \\leq \\zIP .\n            \\end{align*}\n            As $\\blambda_{\\E} \\geq \\bz$ and $\\boldsymbol{\\alpha} \\in X_R$, the above inequality can be rewritten as\n            % \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b) +   \\blambda_\\E^{\\top} ( \\f - \\E\\x ) +\\rho \\norm{\\A\\,\\x - \\b} & \\leq \\zIP - z_R\\\\\n            \\begin{align}\n            \\blambda_{\\A}^{\\top}(\\A\\,\\boldsymbol{\\alpha} - \\b) + \\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} & \\leq \\zIP - z_R \\label{ineq:2}\n            \\end{align}\n            Using Cauchy Schwarz inequality in \\cref{ineq:2} yields for all $\\x \\in X_R$,\n            \\begin{align*}\n                -\\norm{\\blambda_\\A}\\,\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} + \\rho\\,\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq \\zIP - z_R\\\\\n            (\\rho - \\norm{\\blambda_\\A})\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b}\\leq \\zIP - z_R\\\\\n            \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq \\frac{\\zIP-z_R}{\\rho - \\norm{\\blambda_\\A}}.\n            \\end{align*}\n","start":26929,"end":28113}]},{"id":"S027","semantic_type":"corollary","kind":"corollary","labels":["cor:3"],"declared_external":false,"source_only":false,"title":"候选扰动半径界","source_units":[{"raw_tex":"\\begin{corollary}\\label{cor:3}\nConsider the set $U_\\rho = \\set{\\u \\in \\calU}{\\phi(\\u) + \\rho\\norm{\\u} \\leq \\phi(\\bz)}$. If $\\rho > \\norm{\\blambda_{\\A}}$ then for all $\\u \\in U_\\rho$, $$\\norm{\\u} \\leq \\frac{\\zIP-z_R}{\\rho -\\norm{\\blambda_\\A}}.$$\n\\end{corollary}","start":28284,"end":28544}]},{"id":"S028","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"候选扰动界作者证明","source_units":[{"raw_tex":"\\begin{proof}\nIf for any $~ \\u \\in U_{\\rho},~\\exists~ \\x \\in X\\cap H_{\\u}$ such that $f(\\x) + \\rho\\norm{\\A\\x - \\b} \\leq z_{IP}$, the result follows from \\Cref{prop:2}. Alternatively, if $f(\\x) + \\rho\\norm{\\A\\x - \\b} > z_{IP}\\,\\, \\forall~ \\x \\in X\\cap H_{\\u}$ for some $\\u \\in U_{\\rho}$. \n\\begin{align*}\n\\phi(\\u) + \\rho(\\norm{\\u}) =  \\inf_{X\\cap H_{\\u}} f(\\x) + \\rho\\norm{\\A\\x - \\b} \\geq z_{IP}\n\\end{align*} \nIf \\(\\inf_{X\\cap H_{\\u}} f(\\x) + \\rho\\norm{\\A\\x - \\b} > z_{IP}\\) then $\\u \\not\\in U_{\\rho}$ leading to a contradiction. If \\(\\inf_{X\\cap H_{\\u}} f(\\x) + \\rho\\norm{\\A\\x - \\b} = z_{IP}\\) and there exists an \\(\\x \\in X\\cap H_{\\u}\\) such that \\(f(\\x) + \\rho \\norm{\\A\\x - \\b} = z_{IP}\\) then the \nresult follows from \\cref{prop:2}. If there doesn't exist \\(\\x \\in X\\cap H_{\\u}\\) such that \\(f(\\x) + \\rho\\norm{\\A\\x - \\b}  = z_{IP}\\) then one can find a sequence of \\(\\x_{p}\\) such that the follows holds\n\\begin{align*}\n    f(\\x_{p}) + \\rho \\norm{\\A\\x_p - \\b} \\leq z_{IP} + \\frac{1}{p} \\text{ (From definition of infimum)}\\\\\n\\end{align*}\nLet \\(\\x_R\\) be the optimal solution of the optimization program \\(\\inf_{\\x \\in F_R} f(\\x)\\). Since we are minimizing a convex function over a rational polyhedron the KKT conditions hold as follows.\n\\begin{align*}   \n\\nabla{f(\\x_{R})} = \\A^{\\top}\\blambda_\\A - \\E^{\\top}{\\blambda_\\E}\\\\\n    \\blambda_\\E^{\\top}(\\f - \\E\\x_R) = 0\\\\\n    \\blambda_\\A^{\\top}(\\A\\,\\x_R - \\b) = 0\\\\\n    \\blambda_\\E \\geq \\bz\n\\end{align*}\n\\begin{align*} \n    f(\\x_R)+ \\nabla{f(\\x_R)}^{\\top}(\\x_{p}-\\x_R) +\\rho \\norm{\\A\\,\\x_p - \\b} &\\leq \\zIP + \\frac{1}{p}\\text{ (Convexity)}\\\\\n    z_{R} + \\blambda_{\\A}^{\\top} (\\A\\,\\x_p - \\b) +  \\blambda_\\E^{\\top} (\\f - \\E\\x_p) +\\rho \\norm{\\A\\,\\x_p - \\b} &\\leq \\zIP + \\frac{1}{p}\\\\\n    % (\\rho - \\norm{\\blambda_{\\A}}) \\norm{\\u} &\\leq \\frac{\\zIP - z_{R}}{\\rho - \\norm{\\blambda_\\A}} + \\frac{1}{p(\\rho - \\norm{\\blambda_{\\A}})}\\\\\n    \\norm{\\u} &\\leq \\frac{\\zIP - z_{R}}{\\rho - \\norm{\\blambda_{\\A}}} + \\frac{1}{p(\\rho - \\norm{\\blambda_{\\A}})}\n\\end{align*}\nwhere the last inequality follows from non-negativity of $\\blambda_\\E$ and $(\\f - \\E\\x_p).$ As \\(p\\) is arbitrary, we have as $p \\to \\infty$,\n\\begin{align*}\n    \\norm{\\u} &\\leq \\frac{\\zIP - z_{R}}{\\rho - \\norm{\\blambda_{\\A}}}\n\\end{align*}\n\\end{proof}","start":28545,"end":30789}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"取到情形调用残差界","source_units":[{"raw_tex":"If for any $~ \\u \\in U_{\\rho},~\\exists~ \\x \\in X\\cap H_{\\u}$ such that $f(\\x) + \\rho\\norm{\\A\\x - \\b} \\leq z_{IP}$, the result follows from \\Cref{prop:2}. Alternatively, if $f(\\x) + \\rho\\norm{\\A\\x - \\b} > z_{IP}\\,\\, \\forall~ \\x \\in X\\cap H_{\\u}$ for some $\\u \\in U_{\\rho}$. \n\\begin{align*}\n\\phi(\\u) + \\rho(\\norm{\\u}) =  \\inf_{X\\cap H_{\\u}} f(\\x) + \\rho\\norm{\\A\\x - \\b} \\geq z_{IP}\n\\end{align*} \nIf \\(\\inf_{X\\cap H_{\\u}} f(\\x) + \\rho\\norm{\\A\\x - \\b} > z_{IP}\\) then $\\u \\not\\in U_{\\rho}$ leading to a contradiction. If \\(\\inf_{X\\cap H_{\\u}} f(\\x) + \\rho\\norm{\\A\\x - \\b} = z_{IP}\\) and there exists an \\(\\x \\in X\\cap H_{\\u}\\) such that \\(f(\\x) + \\rho \\norm{\\A\\x - \\b} = z_{IP}\\) then the \nresult follows from \\cref{prop:2}. If there doesn't exist \\(\\x \\in X\\cap H_{\\u}\\) such that \\(f(\\x) + \\rho\\norm{\\A\\x - \\b}  = z_{IP}\\) then one can find a sequence of \\(\\x_{p}\\) such that the follows holds\n","start":28559,"end":29451}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"逼近序列处理未取到","source_units":[{"raw_tex":"result follows from \\cref{prop:2}. If there doesn't exist \\(\\x \\in X\\cap H_{\\u}\\) such that \\(f(\\x) + \\rho\\norm{\\A\\x - \\b}  = z_{IP}\\) then one can find a sequence of \\(\\x_{p}\\) such that the follows holds\n\\begin{align*}\n    f(\\x_{p}) + \\rho \\norm{\\A\\x_p - \\b} \\leq z_{IP} + \\frac{1}{p} \\text{ (From definition of infimum)}\\\\\n\\end{align*}\nLet \\(\\x_R\\) be the optimal solution of the optimization program \\(\\inf_{\\x \\in F_R} f(\\x)\\). Since we are minimizing a convex function over a rational polyhedron the KKT conditions hold as follows.\n\\begin{align*}   \n\\nabla{f(\\x_{R})} = \\A^{\\top}\\blambda_\\A - \\E^{\\top}{\\blambda_\\E}\\\\\n    \\blambda_\\E^{\\top}(\\f - \\E\\x_R) = 0\\\\\n    \\blambda_\\A^{\\top}(\\A\\,\\x_R - \\b) = 0\\\\\n    \\blambda_\\E \\geq \\bz\n\\end{align*}\n\\begin{align*} \n    f(\\x_R)+ \\nabla{f(\\x_R)}^{\\top}(\\x_{p}-\\x_R) +\\rho \\norm{\\A\\,\\x_p - \\b} &\\leq \\zIP + \\frac{1}{p}\\text{ (Convexity)}\\\\\n    z_{R} + \\blambda_{\\A}^{\\top} (\\A\\,\\x_p - \\b) +  \\blambda_\\E^{\\top} (\\f - \\E\\x_p) +\\rho \\norm{\\A\\,\\x_p - \\b} &\\leq \\zIP + \\frac{1}{p}\\\\\n    % (\\rho - \\norm{\\blambda_{\\A}}) \\norm{\\u} &\\leq \\frac{\\zIP - z_{R}}{\\rho - \\norm{\\blambda_\\A}} + \\frac{1}{p(\\rho - \\norm{\\blambda_{\\A}})}\\\\\n    \\norm{\\u} &\\leq \\frac{\\zIP - z_{R}}{\\rho - \\norm{\\blambda_{\\A}}} + \\frac{1}{p(\\rho - \\norm{\\blambda_{\\A}})}\n\\end{align*}\nwhere the last inequality follows from non-negativity of $\\blambda_\\E$ and $(\\f - \\E\\x_p).$ As \\(p\\) is arbitrary, we have as $p \\to \\infty$,\n\\begin{align*}\n    \\norm{\\u} &\\leq \\frac{\\zIP - z_{R}}{\\rho - \\norm{\\blambda_{\\A}}}\n\\end{align*}\n","start":29245,"end":30778}]},{"id":"S031","semantic_type":"corollary","kind":"corollary","labels":["cor:4"],"declared_external":false,"source_only":false,"title":"仅需控制降值扰动","source_units":[{"raw_tex":"\\begin{corollary}\\label{cor:4}\nConsider the set $\\,\\ocalU \\define \\set{\\u \\in \\calU}{\\phi(\\bz) \\geq \\phi(\\u)}$. For $\\alpha \\geq 0$, $\\phi(\\bz) \\leq \\phi(\\u) + \\alpha$ $\\forall \\,\\u \\in \\calU$ if $\\phi(\\bz) \\leq \\phi(\\u) + \\alpha$ $\\forall \\,\\u \\in \\ocalU$.\n\\end{corollary}","start":30791,"end":31064}]},{"id":"S032","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"降值约化作者证明","source_units":[{"raw_tex":"\\begin{proof}\n    For $\\u \\in \\calU\\setminus\\ocalU$, $\\phi(\\bz) < \\phi(\\u) \\leq \\phi(\\u) + \\alpha$. The result follows.\n\\end{proof}","start":31065,"end":31196}]},{"id":"S033","semantic_type":"lemma","kind":"lemma","labels":["lem:3"],"declared_external":false,"source_only":false,"title":"局部线性界推出精确罚","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:3}\nIf there exist $\\delta, \\kappa > 0$ such that  $\\phi(\\bz) \\leq \\phi(\\u) + \\kappa \\norm{\\u}$ for all $\\u \\in \\mathcal{N}_{\\delta}(\\bz) \\cap \\calU$,  then there exists $0 < \\rho^{*} < \\infty$ such that $z_{SALR}(\\rho) = \\zIP$ for all $\\rho > \\rho^{*}$.\n\\end{lemma}","start":31197,"end":31486}]},{"id":"S034","semantic_type":"proof","kind":"proof","labels":["eq:9"],"declared_external":false,"source_only":false,"title":"局部到全局作者证明","source_units":[{"raw_tex":"\\begin{proof}\nObserve that, for $\\rho > 0$ it suffices to consider $\\u \\in U_\\rho$. We have, from \\cref{lem:1} \n$$ z_{SALR}(\\rho) = \\min_{\\u\\in \\calU} \\phi(\\u) + \\rho \\norm{\\u}.$$\nIt follows that,\n\\begin{align*}\n    z_{SALR}(\\rho) = \\min_{\\u\\in U_\\rho}\\phi(\\u) + \\rho \\norm{\\u}\n\\end{align*}\n\\cref{prop:2} yields that for $\\rho > \\norm{\\blambda_{\\A}}$ and $\\u \\in U_{\\rho}$, $\\norm{\\u}$ is bounded. Thus, we have\n\\begin{equation*}\n    z_{SALR}(\\rho) = \\min_{\\u \\in \\mathcal{N}_{\\delta_{\\rho}}(\\bz) \\cap\\, U_\\rho}\\phi(\\u) + \\rho \\norm{\\u} \n\\end{equation*}\nwhere $\\displaystyle\\delta_{\\rho} = \\frac{\\zIP - z_R}{\\rho -\\norm{\\blambda_{\\A}}}$. From \\cref{prop:2} we see that as $\\rho \\to \\infty,\\, \\mathcal{N}_{\\delta_{\\rho}}(\\bz) \\cap U_\\rho \\to \\{\\bz\\}$. If $\\exists$ $\\delta > 0$, and $0 < \\kappa < \\infty$ such that $\\phi(\\bz) \\leq \\phi(\\u) + \\kappa \\norm{\\u}$ for all $\\u \\in \\mathcal{N}_{\\delta}(\\bz) \\cap U_\\rho$, it follows that,\n\\begin{align}\n    \\min_{\\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap\\, U_\\rho} \\phi(\\u) + \\kappa \\norm{\\u} = \\phi(\\bz) = \\zIP\\label{eq:9}\n\\end{align}\nAlternatively, we can increase $\\rho$ such that $\\forall\\, \\u \\in U_\\rho$, $\\norm{\\u} \\leq \\dfrac{\\zIP - z_R}{\\rho -\\norm{\\blambda_{\\A}}} < \\delta$.  Rearranging the terms, we obtain $\\rho > \\norm{\\blambda_\\A} + \\frac{\\zIP - z_R}{\\delta}$.  It follows from \\cref{eq:9} that for any choice of $\\displaystyle \\rho > \\max\\left\\{\\kappa, \\norm{\\blambda_\\A} + \\frac{\\zIP - z_R}{\\delta}\\right\\}$, we have  \n\\begin{align*}\n    z_{SALR}(\\rho) = \\min_{\\u\\in \\mathcal{N}_{\\delta_{\\rho}}(\\bz) \\cap\\, U_\\rho} \\phi(\\u) + \\rho \\norm{\\u} = \\zIP.\n\\end{align*}\n\\end{proof}","start":31487,"end":33114}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"候选扰动收缩到原点","source_units":[{"raw_tex":"Observe that, for $\\rho > 0$ it suffices to consider $\\u \\in U_\\rho$. We have, from \\cref{lem:1} \n$$ z_{SALR}(\\rho) = \\min_{\\u\\in \\calU} \\phi(\\u) + \\rho \\norm{\\u}.$$\nIt follows that,\n\\begin{align*}\n    z_{SALR}(\\rho) = \\min_{\\u\\in U_\\rho}\\phi(\\u) + \\rho \\norm{\\u}\n\\end{align*}\n\\cref{prop:2} yields that for $\\rho > \\norm{\\blambda_{\\A}}$ and $\\u \\in U_{\\rho}$, $\\norm{\\u}$ is bounded. Thus, we have\n\\begin{equation*}\n    z_{SALR}(\\rho) = \\min_{\\u \\in \\mathcal{N}_{\\delta_{\\rho}}(\\bz) \\cap\\, U_\\rho}\\phi(\\u) + \\rho \\norm{\\u} \n\\end{equation*}\nwhere $\\displaystyle\\delta_{\\rho} = \\frac{\\zIP - z_R}{\\rho -\\norm{\\blambda_{\\A}}}$. From \\cref{prop:2} we see that as $\\rho \\to \\infty,\\, \\mathcal{N}_{\\delta_{\\rho}}(\\bz) \\cap U_\\rho \\to \\{\\bz\\}$. If $\\exists$ $\\delta > 0$, and $0 < \\kappa < \\infty$ such that $\\phi(\\bz) \\leq \\phi(\\u) + \\kappa \\norm{\\u}$ for all $\\u \\in \\mathcal{N}_{\\delta}(\\bz) \\cap U_\\rho$, it follows that,\n","start":31501,"end":32419}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":["eq:9"],"declared_external":false,"source_only":false,"title":"显式有限罚参数阈值","source_units":[{"raw_tex":"where $\\displaystyle\\delta_{\\rho} = \\frac{\\zIP - z_R}{\\rho -\\norm{\\blambda_{\\A}}}$. From \\cref{prop:2} we see that as $\\rho \\to \\infty,\\, \\mathcal{N}_{\\delta_{\\rho}}(\\bz) \\cap U_\\rho \\to \\{\\bz\\}$. If $\\exists$ $\\delta > 0$, and $0 < \\kappa < \\infty$ such that $\\phi(\\bz) \\leq \\phi(\\u) + \\kappa \\norm{\\u}$ for all $\\u \\in \\mathcal{N}_{\\delta}(\\bz) \\cap U_\\rho$, it follows that,\n\\begin{align}\n    \\min_{\\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap\\, U_\\rho} \\phi(\\u) + \\kappa \\norm{\\u} = \\phi(\\bz) = \\zIP\\label{eq:9}\n\\end{align}\nAlternatively, we can increase $\\rho$ such that $\\forall\\, \\u \\in U_\\rho$, $\\norm{\\u} \\leq \\dfrac{\\zIP - z_R}{\\rho -\\norm{\\blambda_{\\A}}} < \\delta$.  Rearranging the terms, we obtain $\\rho > \\norm{\\blambda_\\A} + \\frac{\\zIP - z_R}{\\delta}$.  It follows from \\cref{eq:9} that for any choice of $\\displaystyle \\rho > \\max\\left\\{\\kappa, \\norm{\\blambda_\\A} + \\frac{\\zIP - z_R}{\\delta}\\right\\}$, we have  \n\\begin{align*}\n    z_{SALR}(\\rho) = \\min_{\\u\\in \\mathcal{N}_{\\delta_{\\rho}}(\\bz) \\cap\\, U_\\rho} \\phi(\\u) + \\rho \\norm{\\u} = \\zIP.\n\\end{align*}\n","start":32041,"end":33103}]},{"id":"S037","semantic_type":"lemma","kind":"lemma","labels":["lem:6"],"declared_external":false,"source_only":false,"title":"有限整数最小值下半连续","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:6}\nIf the integer variables are bounded, then the value function $\\displaystyle \\hat{\\phi}(\\u) = \\min_{i \\in [k]} \\Phi(\\u,\\x^{(i)}_I)$ is lower semi-continuous at $\\bz$.\n\\end{lemma}","start":55608,"end":55813}]},{"id":"S038","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"有限最小值作者证明","source_units":[{"raw_tex":"\\begin{proof}\nObserve that for $\\u \\in \\calU, \\x^{(i)}_I \\in S_I$, $\\Phi(\\u,\\x^{(i)}_I)$ is lower semi-continuous at ${\\u}$ if and only if strong duality holds at ${\\u}$ [Proposition 6.5.2,\\cite{bertsekas2003convex}]. If for some $\\x^{(i)}_I$, $\\Phi(\\bz, \\x^{(i)}_I) < +\\infty$ then $\\Phi(\\u, \\x^{(i)}_I)$ is lower semi-continuous at $\\u = \\bz$ due to Slater Condition (in case of affine constraints only feasibility is required). On the contrary, $\\Phi(\\bz,\\x^{(i)}_I) = + \\infty$ implies that the polyhedron $P = \\set{\\x \\in \\R^{n_2}}{\\tilde{\\A}\\x = \\tilde{\\b}, \\,\\,\\x \\geq \\bz}$ is empty, where\n\\begin{align*}\n    \\tilde{\\A} = \\begin{pmatrix}\n      {\\A}_C & -{\\A}_C & \\bz\\\\\n      {\\E}_C & -{\\E}_C & \\mathbb{I}\n    \\end{pmatrix},~\n    \\x = \\begin{pmatrix}\n    \\x_C^+\\\\\n    \\x_C^-\\\\\n    \\s\n    \\end{pmatrix},~\n    \\tilde{\\b} = \\begin{pmatrix}\n     {\\b}  -{\\A}_I{\\x}^{(i)}_I\\\\\n     {\\f} - {\\E}_I{\\x}^{(i)}_I\n    \\end{pmatrix}.\n\\end{align*}\nLet $C$ be the closed convex cone spanned by the columns of $\\tilde{\\A}$. If $P = \\emptyset$ then $\\tilde{\\b} \\not\\in C$. Since $C$ is a closed, \n$\\exists\\,\\,\\delta_{i} > 0$ such that $\\mathcal{N}_{\\delta_i}(\\tilde{\\b}) \\cap C = \\emptyset$. Consider $\\u \\in \\R^m \\cap \\mathcal{N}_{\\delta_i}(\\bz)$, then $\\tilde{\\b} + \\begin{pmatrix}\\u\\\\\\bz\\end{pmatrix} \\not \\in C$.\n\nFor $J = \\set{i\\in [k]}{\\Phi({\\x}^{(i)}_I,\\bz) < \\infty}$, define $\\displaystyle\\delta_{\\min} \\define \\min_{i \\in [k]\\setminus J}\\delta_i$.\n\nConsider y such that $y  < \\hat{\\phi}(\\bz)$, it follows that \n\\begin{align*}\n    y & < \\hat{\\phi}(\\bz) < \\min_{i \\in J}\\Phi(\\bz,{\\x}^{(i)}_I) \\leq \\Phi(\\bz,{\\x}^{(i)}_I) .\n\\end{align*}\nHowever, since every $\\Phi({\\u},{\\x}^{(i)}_I), \\, i \\in J$ is lower semi-continuous at $\\bz$, there exists a $\\delta_i > 0$ such that $\\Phi({\\u},{\\x}^{(i)}_I) > y$ for every ${\\u} \\in \\mathcal{N}_{\\delta_i}(\\bz), \\, i \\in J$. Let $\\delta'_{\\min} = \\min(\\delta_i),\\, i \\in J$ and $\\delta'' = \\min(\\delta_{\\min},\\delta'_{\\min})$. This means that for every ${\\u} \\in \\mathcal{N}_{\\delta''}(\\bz)$ $\\Phi({\\u},{\\x}^{(i)}_I) > y \\implies  \\hat{\\phi}({\\u}) > y$. The result follows.\n\\end{proof}","start":55814,"end":57933}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"可行限制的强对偶分支","source_units":[{"raw_tex":"Observe that for $\\u \\in \\calU, \\x^{(i)}_I \\in S_I$, $\\Phi(\\u,\\x^{(i)}_I)$ is lower semi-continuous at ${\\u}$ if and only if strong duality holds at ${\\u}$ [Proposition 6.5.2,\\cite{bertsekas2003convex}]. If for some $\\x^{(i)}_I$, $\\Phi(\\bz, \\x^{(i)}_I) < +\\infty$ then $\\Phi(\\u, \\x^{(i)}_I)$ is lower semi-continuous at $\\u = \\bz$ due to Slater Condition (in case of affine constraints only feasibility is required). On the contrary, $\\Phi(\\bz,\\x^{(i)}_I) = + \\infty$ implies that the polyhedron $P = \\set{\\x \\in \\R^{n_2}}{\\tilde{\\A}\\x = \\tilde{\\b}, \\,\\,\\x \\geq \\bz}$ is empty, where\n","start":55828,"end":56412}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"不可行限制局部排除","source_units":[{"raw_tex":"Observe that for $\\u \\in \\calU, \\x^{(i)}_I \\in S_I$, $\\Phi(\\u,\\x^{(i)}_I)$ is lower semi-continuous at ${\\u}$ if and only if strong duality holds at ${\\u}$ [Proposition 6.5.2,\\cite{bertsekas2003convex}]. If for some $\\x^{(i)}_I$, $\\Phi(\\bz, \\x^{(i)}_I) < +\\infty$ then $\\Phi(\\u, \\x^{(i)}_I)$ is lower semi-continuous at $\\u = \\bz$ due to Slater Condition (in case of affine constraints only feasibility is required). On the contrary, $\\Phi(\\bz,\\x^{(i)}_I) = + \\infty$ implies that the polyhedron $P = \\set{\\x \\in \\R^{n_2}}{\\tilde{\\A}\\x = \\tilde{\\b}, \\,\\,\\x \\geq \\bz}$ is empty, where\n\\begin{align*}\n    \\tilde{\\A} = \\begin{pmatrix}\n      {\\A}_C & -{\\A}_C & \\bz\\\\\n      {\\E}_C & -{\\E}_C & \\mathbb{I}\n    \\end{pmatrix},~\n    \\x = \\begin{pmatrix}\n    \\x_C^+\\\\\n    \\x_C^-\\\\\n    \\s\n    \\end{pmatrix},~\n    \\tilde{\\b} = \\begin{pmatrix}\n     {\\b}  -{\\A}_I{\\x}^{(i)}_I\\\\\n     {\\f} - {\\E}_I{\\x}^{(i)}_I\n    \\end{pmatrix}.\n\\end{align*}\nLet $C$ be the closed convex cone spanned by the columns of $\\tilde{\\A}$. If $P = \\emptyset$ then $\\tilde{\\b} \\not\\in C$. Since $C$ is a closed, \n$\\exists\\,\\,\\delta_{i} > 0$ such that $\\mathcal{N}_{\\delta_i}(\\tilde{\\b}) \\cap C = \\emptyset$. Consider $\\u \\in \\R^m \\cap \\mathcal{N}_{\\delta_i}(\\bz)$, then $\\tilde{\\b} + \\begin{pmatrix}\\u\\\\\\bz\\end{pmatrix} \\not \\in C$.\n\nFor $J = \\set{i\\in [k]}{\\Phi({\\x}^{(i)}_I,\\bz) < \\infty}$, define $\\displaystyle\\delta_{\\min} \\define \\min_{i \\in [k]\\setminus J}\\delta_i$.\n","start":55828,"end":57261}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"统一有限邻域","source_units":[{"raw_tex":"For $J = \\set{i\\in [k]}{\\Phi({\\x}^{(i)}_I,\\bz) < \\infty}$, define $\\displaystyle\\delta_{\\min} \\define \\min_{i \\in [k]\\setminus J}\\delta_i$.\n\nConsider y such that $y  < \\hat{\\phi}(\\bz)$, it follows that \n\\begin{align*}\n    y & < \\hat{\\phi}(\\bz) < \\min_{i \\in J}\\Phi(\\bz,{\\x}^{(i)}_I) \\leq \\Phi(\\bz,{\\x}^{(i)}_I) .\n\\end{align*}\nHowever, since every $\\Phi({\\u},{\\x}^{(i)}_I), \\, i \\in J$ is lower semi-continuous at $\\bz$, there exists a $\\delta_i > 0$ such that $\\Phi({\\u},{\\x}^{(i)}_I) > y$ for every ${\\u} \\in \\mathcal{N}_{\\delta_i}(\\bz), \\, i \\in J$. Let $\\delta'_{\\min} = \\min(\\delta_i),\\, i \\in J$ and $\\delta'' = \\min(\\delta_{\\min},\\delta'_{\\min})$. This means that for every ${\\u} \\in \\mathcal{N}_{\\delta''}(\\bz)$ $\\Phi({\\u},{\\x}^{(i)}_I) > y \\implies  \\hat{\\phi}({\\u}) > y$. The result follows.\n","start":57121,"end":57922}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"降值侧连续性桥接","source_units":[{"raw_tex":"Consider $\\ocalU$ as defined in \\cref{cor:4}, it follows from \\cref{lem:6} that $\\lim_{\\u\\to \\bz} \\hat{\\phi}(\\u) = \\hat{\\phi}(\\bz)$ where $\\u \\in \\ocalU$. Now, for every $\\epsilon > 0$ there exists a $\\delta > 0$ such that\n\\begin{equation*}\n   \\abs{\\hat{\\phi}(\\u) - \\hat{\\phi}(\\bz)} < \\epsilon\\,\\, \\forall \\u\\,\\, \\in \\mathcal{N}_{\\delta}(\\bz) \\cap \\ocalU\n\\end{equation*}\n","start":57934,"end":58305}]},{"id":"S043","semantic_type":"lemma","kind":"lemma","labels":["lem:8"],"declared_external":false,"source_only":false,"title":"局部仅保留原点最优整数","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:8}\nConsider the value function $\\hat{\\phi}(\\u)$ as defined earlier for $\\textup{\\MICP}_\\u$. There exists a $\\delta > 0$ such that for all $\\u \\in \\mathcal{N}_\\delta(\\bz) \\cap \\ocalU$, $\\displaystyle \\hat{\\phi}(\\u) = \\min_{\\x_I\\in S_{=}^{\\bz}} \\Phi(\\u,\\x_I)$ where $S_{=}^{\\bz} = \\set{\\x_I \\in S_I}{\\Phi(\\bz,\\x_I) = \\hat{\\phi}(\\bz)}$\n\\end{lemma}","start":58305,"end":58673}]},{"id":"S044","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"最优整数筛选作者证明","source_units":[{"raw_tex":"\\begin{proof}\nWe can partition the index sets corresponding to solutions of $\\CPui$ into two sets, \n$$\nS_{>}^{\\bz} = \\set{\\x_I\\in S_I}{\\Phi(\\bz,\\x_I) > \\hat{\\phi}(\\bz)} \\text{ and } S_{=}^{\\bz} = \\set{\\x_I\\in S_I}{\\Phi(\\bz,\\x_I) = \\hat{\\phi}(\\bz)}.\n$$\nFor each $\\x^{(i)}_I \\in S_I$, define $ \\overline{\\calU}_i=\\set{\\u \\in \\calU}{\\Phi({\\u},\\x^{(i)}_I)\\leq \\Phi(\\bz,\\x^{(i)}_I)} $. If there is no limiting sequence to ${\\bz}$ in $\\overline{\\calU}_i$ then there exists a neighbourhood $\\delta_i$ such that $\\mathcal{N}_{\\delta_i}(\\bz) \\cap \\overline{\\calU}_i = \\emptyset$. Alternatively, if there does exist a limiting sequence to $\\bz$ and since $\\Phi({\\u},{\\x}_I^{(i)})$ is lower semi-continuous,\n\\begin{equation*}\n    \\lim_{{\\u}\\to \\bz} \\Phi({\\u},{\\x}^{(i)}_I) = \\Phi(\\bz,{\\x}^{(i)}_I)\n\\end{equation*}\nThus, for every $\\epsilon_i > 0$ there exists a $\\delta_i > 0$ such that\n\\begin{align*}\n    \\abs{\\Phi({\\u},{\\x}^{(i)}_I) - \\Phi(\\bz,{\\x}^{(i)}_I)}< \\epsilon_i \\,\\,\\,\\,\\, \\forall \\,\\,\\,\\,{\\u}\\in \\overline{\\calU}_i\\cap \\mathcal{N}_{\\delta_i}(\\bz)\n\\end{align*}\nFor $\\x^{(i)}_I \\in S_{>}^{\\bz}$, let $\\epsilon_i = \\Phi(\\bz,{\\x}^{(i)}_I) - \\hat{\\phi}(\\bz)$. Assuming that $\\Phi(\\bz,{\\x}_I^{(i)})$ is finite, $\\forall$ $\\u \\in \\overline{\\calU}_i\\cap \\mathcal{N}_{\\delta_i}(\\bz)$\n\\begin{align*}\n    \\Phi(\\bz,{\\x}^{(i)}_I) - \\Phi({\\u},{\\x}_I^{(i)}) & <  \\Phi(\\bz,{\\x}^{(i)}_I) - \\hat{\\phi}(\\bz)\\\\\n    \\hat{\\phi}(\\bz) & < \\Phi({\\u},{\\x}^{(i)}_I)\n\\end{align*}\nThus, for all $\\x^{(i)}_I\\in S_{>}^{\\bz}$ there exists a $\\delta_i$ such that $\\Phi({\\u},{\\x}_I^{(i)}) > \\hat{\\phi}(\\bz)$. Assigning $\\delta_{\\min} = \\min_{\\x^{(i)}_I\\in S^{\\bz}_{>}} \\delta_i$ yields $\\displaystyle \\hat{\\phi}({\\u}) = \\min_{\\x_I\\in S^{\\bz}_{=}}\\Phi({\\u},{\\x}_I)$ for every $\\u \\in \\mathcal{N}_{\\delta_{\\min}}(\\bz)\\cap \\overline{\\calU}$.\n\\end{proof}","start":58674,"end":60491}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"限制值函数的降值侧控制","source_units":[{"raw_tex":"We can partition the index sets corresponding to solutions of $\\CPui$ into two sets, \n$$\nS_{>}^{\\bz} = \\set{\\x_I\\in S_I}{\\Phi(\\bz,\\x_I) > \\hat{\\phi}(\\bz)} \\text{ and } S_{=}^{\\bz} = \\set{\\x_I\\in S_I}{\\Phi(\\bz,\\x_I) = \\hat{\\phi}(\\bz)}.\n$$\nFor each $\\x^{(i)}_I \\in S_I$, define $ \\overline{\\calU}_i=\\set{\\u \\in \\calU}{\\Phi({\\u},\\x^{(i)}_I)\\leq \\Phi(\\bz,\\x^{(i)}_I)} $. If there is no limiting sequence to ${\\bz}$ in $\\overline{\\calU}_i$ then there exists a neighbourhood $\\delta_i$ such that $\\mathcal{N}_{\\delta_i}(\\bz) \\cap \\overline{\\calU}_i = \\emptyset$. Alternatively, if there does exist a limiting sequence to $\\bz$ and since $\\Phi({\\u},{\\x}_I^{(i)})$ is lower semi-continuous,\n\\begin{equation*}\n    \\lim_{{\\u}\\to \\bz} \\Phi({\\u},{\\x}^{(i)}_I) = \\Phi(\\bz,{\\x}^{(i)}_I)\n\\end{equation*}\nThus, for every $\\epsilon_i > 0$ there exists a $\\delta_i > 0$ such that\n\\begin{align*}\n    \\abs{\\Phi({\\u},{\\x}^{(i)}_I) - \\Phi(\\bz,{\\x}^{(i)}_I)}< \\epsilon_i \\,\\,\\,\\,\\, \\forall \\,\\,\\,\\,{\\u}\\in \\overline{\\calU}_i\\cap \\mathcal{N}_{\\delta_i}(\\bz)\n\\end{align*}\n","start":58688,"end":59735}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"有限次优分支统一排除","source_units":[{"raw_tex":"For $\\x^{(i)}_I \\in S_{>}^{\\bz}$, let $\\epsilon_i = \\Phi(\\bz,{\\x}^{(i)}_I) - \\hat{\\phi}(\\bz)$. Assuming that $\\Phi(\\bz,{\\x}_I^{(i)})$ is finite, $\\forall$ $\\u \\in \\overline{\\calU}_i\\cap \\mathcal{N}_{\\delta_i}(\\bz)$\n\\begin{align*}\n    \\Phi(\\bz,{\\x}^{(i)}_I) - \\Phi({\\u},{\\x}_I^{(i)}) & <  \\Phi(\\bz,{\\x}^{(i)}_I) - \\hat{\\phi}(\\bz)\\\\\n    \\hat{\\phi}(\\bz) & < \\Phi({\\u},{\\x}^{(i)}_I)\n\\end{align*}\nThus, for all $\\x^{(i)}_I\\in S_{>}^{\\bz}$ there exists a $\\delta_i$ such that $\\Phi({\\u},{\\x}_I^{(i)}) > \\hat{\\phi}(\\bz)$. Assigning $\\delta_{\\min} = \\min_{\\x^{(i)}_I\\in S^{\\bz}_{>}} \\delta_i$ yields $\\displaystyle \\hat{\\phi}({\\u}) = \\min_{\\x_I\\in S^{\\bz}_{=}}\\Phi({\\u},{\\x}_I)$ for every $\\u \\in \\mathcal{N}_{\\delta_{\\min}}(\\bz)\\cap \\overline{\\calU}$.\n","start":59735,"end":60480}]},{"id":"S047","semantic_type":"lemma","kind":"lemma","labels":["lem:9"],"declared_external":false,"source_only":false,"title":"有界整数局部线性界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:9}\nConsider the value function $\\hat{\\phi}(\\u)$ as defined earlier for $\\textup{\\MICP}_\\u$. There exists a $\\delta > 0$ and $0 < \\Gamma< \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $$\\hat{\\phi}(\\bz)\\leq \\hat{\\phi}(\\u) + \\Gamma\\norm{\\u}.$$\n\\end{lemma}","start":60492,"end":60796}]},{"id":"S048","semantic_type":"proof","kind":"proof","labels":["eq:32"],"declared_external":false,"source_only":false,"title":"局部线性界作者证明","source_units":[{"raw_tex":"\\begin{proof}\nConsider $S^{\\bz}_{=}$ as defined in \\cref{lem:8}. Analogously, define $$S_{=}^{\\u} \\define \\set{\\x_I\\in S_{=}^{\\bz}}{\\Phi(\\u,\\x_{I}) = \\hat{\\phi}(\\u)}.$$ For $\\x^{(i)}_I \\in S^{\\u}_{=}$, the first order necessary conditions for optimality (KKT conditions) for $\\CPui$ can be expressed as, %For  writing the KKT conditions\\\\\n\\begin{align}\\label{eq:32}\n\\begin{split}\n    \\nabla_{C}{f(\\x_I^{(i)},\\x_C}) & = \\A^\\top_C\\blambda^{\\u(i)}_{\\A_C} - \\E_C^\\top\\blambda^{\\u(i)}_{\\E_C}\\\\\n    \\A_C\\x_C & = \\b + \\u - \\A_I\\x^{(i)}_I\\\\\n    \\E_C\\x_C+\\E_I\\x^{(i)}_I & \\leq \\f\\\\\n    {\\blambda^{\\u(i)}_{\\E_C}}^\\top(\\f - \\E_C\\x_C - \\E_I\\x^{(i)}_I) & = \\bz\\\\\n    {\\blambda^{\\u(i)}_{\\A_C}}^\\top(\\A_C\\x_C + \\A_I\\x^{(i)}_I - \\b - \\u) & = \\bz\\\\\n    {\\blambda^{\\u(i)}_{\\E_C}} & \\geq \\bz\n\\end{split}\n\\end{align}\nDefine $H^{(j)}_{\\u} = \\set{\\x_C \\in \\R^{n_2}}{\\A_C\\x_C = \\b +\\u- \\A_I\\x^{(j)}_I}$. Let $(\\x_I^{(j)},\\x_C) \\in X \\cap H^{(j)}_{\\u}$ where $\\x_I^{(j)} \\in S_{=}^{\\u}$.\nAdditionally, let the corresponding Lagrange multipliers at $\\u = \\bz$ for $\\x^{(j)}_I$ be $\\blambda^{(j)}_{\\A_C}$ and $\\blambda^{(j)}_{\\E_C}$.\n\\begin{align*}\n    f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\nabla_C{f(\\x_I^{(j)}}\\x^{\\bz*}_C)^{\\top}(\\x^{\\u}_C -  \\x^{\\bz*}_C )\\\\ \n    & \\hspace{3cm}+\\nabla_I{f(\\x_I^{(j)}}, \\x^{\\bz*}_C)^{\\top}(\\x^{(j)}_I -  \\x^{(j)}_I)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) + ( \\blambda^{(j)^\\top}_{\\A_C}\\A_C - \\blambda^{(j)^\\top}_{\\E_C}\\E_C) (\\x^{\\u}_C -  \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}\\A_C(\\x^{\\u}_C -  \\x^{\\bz*}_C) - \\blambda^{(j)^\\top}_{\\E_C}\\E_C (\\x^{\\u}_C -  \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\A_C \\x^{\\u}_C -  \\A_C \\x^{\\bz*}_C) - \\blambda^{(j)^\\top}_{\\E_C}(\\E_C \\x^{\\u}_C -  \\E_C \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\b+\\u-\\A_I\\x^{(j)}_I - \\b + \\A_I\\x^{(j)}_I)\\\\ \n    & \\hspace{3cm}-\\blambda^{(j)^\\top}_{\\E_C}(\\E_C \\x^{\\u}_C -  \\f + \\E_I\\x^{(j)}_I)\n\\end{align*}\nwhere the equalities follow from the set of equations \\cref{eq:32}. Furthermore, observe that since $\\blambda^{(j)^\\top}_{\\E_C} \\geq \\bz$ and $\\E_C\\x^{\\u}_C + \\E\\x^{(j)}_I \\leq \\f$, it follows that \n\\begin{align*}\n   f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u), \\\\\n      \\inf_{X\\cap H^{(j)}_{\\u}}f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u), \\text{ i.e.,}\\\\\n   \\hat{\\phi}({\\u)} & \\geq \\hat{\\phi}(\\bz) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u)\n\\end{align*}\nRearranging the terms in the above inequality,     $-\\blambda^{(j)^\\top}_{\\A_C}\\u \\geq \\hat{\\phi}(\\bz) - \\hat{\\phi}({\\u)}$. As $\\hat{\\phi}(\\bz)\\geq \\hat{\\phi}(\\u)$ $\\forall\\,\\,\\, \\u \\in \\ocalU$, using Cauchy-Schwarz on the left-hand side of the inequality,\n\\begin{align*}\n    \\norm{\\blambda^{(j)}_{\\A_C}}\\cdot\\norm{\\u} \\geq \\hat{\\phi}(\\bz) - \\hat{\\phi}({\\u)} .\n\\end{align*}\nDefining $\\Gamma = \\max_{i\\in S^{\\bz}_{=} }{\\norm{\\blambda^{(j)}_{\\A_C}}}$ yields\n$$\n    \\hat{\\phi}(\\bz) - \\hat{\\phi}(\\u)\\leq \\Gamma\\norm{\\u}\\,\\,\\forall\\,\\, \\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap\\ocalU\n$$\nwhere $\\delta = \\delta_{min}$ as defined in the proof of \\cref{lem:8}. The result follows.\n\\end{proof}","start":60797,"end":64038}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["eq:32"],"declared_external":false,"source_only":false,"title":"活动整数分支与KKT","source_units":[{"raw_tex":"Consider $S^{\\bz}_{=}$ as defined in \\cref{lem:8}. Analogously, define $$S_{=}^{\\u} \\define \\set{\\x_I\\in S_{=}^{\\bz}}{\\Phi(\\u,\\x_{I}) = \\hat{\\phi}(\\u)}.$$ For $\\x^{(i)}_I \\in S^{\\u}_{=}$, the first order necessary conditions for optimality (KKT conditions) for $\\CPui$ can be expressed as, %For  writing the KKT conditions\\\\\n\\begin{align}\\label{eq:32}\n\\begin{split}\n    \\nabla_{C}{f(\\x_I^{(i)},\\x_C}) & = \\A^\\top_C\\blambda^{\\u(i)}_{\\A_C} - \\E_C^\\top\\blambda^{\\u(i)}_{\\E_C}\\\\\n    \\A_C\\x_C & = \\b + \\u - \\A_I\\x^{(i)}_I\\\\\n    \\E_C\\x_C+\\E_I\\x^{(i)}_I & \\leq \\f\\\\\n    {\\blambda^{\\u(i)}_{\\E_C}}^\\top(\\f - \\E_C\\x_C - \\E_I\\x^{(i)}_I) & = \\bz\\\\\n    {\\blambda^{\\u(i)}_{\\A_C}}^\\top(\\A_C\\x_C + \\A_I\\x^{(i)}_I - \\b - \\u) & = \\bz\\\\\n    {\\blambda^{\\u(i)}_{\\E_C}} & \\geq \\bz\n\\end{split}\n\\end{align}\nDefine $H^{(j)}_{\\u} = \\set{\\x_C \\in \\R^{n_2}}{\\A_C\\x_C = \\b +\\u- \\A_I\\x^{(j)}_I}$. Let $(\\x_I^{(j)},\\x_C) \\in X \\cap H^{(j)}_{\\u}$ where $\\x_I^{(j)} \\in S_{=}^{\\u}$.\nAdditionally, let the corresponding Lagrange multipliers at $\\u = \\bz$ for $\\x^{(j)}_I$ be $\\blambda^{(j)}_{\\A_C}$ and $\\blambda^{(j)}_{\\E_C}$.\n","start":60811,"end":61905}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"乘子给出限制线性下界","source_units":[{"raw_tex":"\\begin{align*}\n    f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\nabla_C{f(\\x_I^{(j)}}\\x^{\\bz*}_C)^{\\top}(\\x^{\\u}_C -  \\x^{\\bz*}_C )\\\\ \n    & \\hspace{3cm}+\\nabla_I{f(\\x_I^{(j)}}, \\x^{\\bz*}_C)^{\\top}(\\x^{(j)}_I -  \\x^{(j)}_I)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) + ( \\blambda^{(j)^\\top}_{\\A_C}\\A_C - \\blambda^{(j)^\\top}_{\\E_C}\\E_C) (\\x^{\\u}_C -  \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}\\A_C(\\x^{\\u}_C -  \\x^{\\bz*}_C) - \\blambda^{(j)^\\top}_{\\E_C}\\E_C (\\x^{\\u}_C -  \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\A_C \\x^{\\u}_C -  \\A_C \\x^{\\bz*}_C) - \\blambda^{(j)^\\top}_{\\E_C}(\\E_C \\x^{\\u}_C -  \\E_C \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\b+\\u-\\A_I\\x^{(j)}_I - \\b + \\A_I\\x^{(j)}_I)\\\\ \n    & \\hspace{3cm}-\\blambda^{(j)^\\top}_{\\E_C}(\\E_C \\x^{\\u}_C -  \\f + \\E_I\\x^{(j)}_I)\n\\end{align*}\nwhere the equalities follow from the set of equations \\cref{eq:32}. Furthermore, observe that since $\\blambda^{(j)^\\top}_{\\E_C} \\geq \\bz$ and $\\E_C\\x^{\\u}_C + \\E\\x^{(j)}_I \\leq \\f$, it follows that \n\\begin{align*}\n   f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u), \\\\\n      \\inf_{X\\cap H^{(j)}_{\\u}}f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u), \\text{ i.e.,}\\\\\n   \\hat{\\phi}({\\u)} & \\geq \\hat{\\phi}(\\bz) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u)\n","start":61905,"end":63346}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"有限乘子范数控制Γ","source_units":[{"raw_tex":"\\end{align*}\nRearranging the terms in the above inequality,     $-\\blambda^{(j)^\\top}_{\\A_C}\\u \\geq \\hat{\\phi}(\\bz) - \\hat{\\phi}({\\u)}$. As $\\hat{\\phi}(\\bz)\\geq \\hat{\\phi}(\\u)$ $\\forall\\,\\,\\, \\u \\in \\ocalU$, using Cauchy-Schwarz on the left-hand side of the inequality,\n\\begin{align*}\n    \\norm{\\blambda^{(j)}_{\\A_C}}\\cdot\\norm{\\u} \\geq \\hat{\\phi}(\\bz) - \\hat{\\phi}({\\u)} .\n\\end{align*}\nDefining $\\Gamma = \\max_{i\\in S^{\\bz}_{=} }{\\norm{\\blambda^{(j)}_{\\A_C}}}$ yields\n$$\n    \\hat{\\phi}(\\bz) - \\hat{\\phi}(\\u)\\leq \\Gamma\\norm{\\u}\\,\\,\\forall\\,\\, \\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap\\ocalU\n$$\nwhere $\\delta = \\delta_{min}$ as defined in the proof of \\cref{lem:8}. The result follows.\n","start":63346,"end":64027}]},{"id":"S052","semantic_type":"proof","kind":"proof","labels":["cor:16","eq:32","eq:stationarity","ineq:lambda","ineq:thm3","lem:15","lem:6","lem:8","lem:9","sol:lambda"],"declared_external":false,"source_only":false,"title":"定理三作者证明汇总","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:6}\nIf the integer variables are bounded, then the value function $\\displaystyle \\hat{\\phi}(\\u) = \\min_{i \\in [k]} \\Phi(\\u,\\x^{(i)}_I)$ is lower semi-continuous at $\\bz$.\n\\end{lemma}","start":55608,"end":55813},{"raw_tex":"\\begin{proof}\nObserve that for $\\u \\in \\calU, \\x^{(i)}_I \\in S_I$, $\\Phi(\\u,\\x^{(i)}_I)$ is lower semi-continuous at ${\\u}$ if and only if strong duality holds at ${\\u}$ [Proposition 6.5.2,\\cite{bertsekas2003convex}]. If for some $\\x^{(i)}_I$, $\\Phi(\\bz, \\x^{(i)}_I) < +\\infty$ then $\\Phi(\\u, \\x^{(i)}_I)$ is lower semi-continuous at $\\u = \\bz$ due to Slater Condition (in case of affine constraints only feasibility is required). On the contrary, $\\Phi(\\bz,\\x^{(i)}_I) = + \\infty$ implies that the polyhedron $P = \\set{\\x \\in \\R^{n_2}}{\\tilde{\\A}\\x = \\tilde{\\b}, \\,\\,\\x \\geq \\bz}$ is empty, where\n\\begin{align*}\n    \\tilde{\\A} = \\begin{pmatrix}\n      {\\A}_C & -{\\A}_C & \\bz\\\\\n      {\\E}_C & -{\\E}_C & \\mathbb{I}\n    \\end{pmatrix},~\n    \\x = \\begin{pmatrix}\n    \\x_C^+\\\\\n    \\x_C^-\\\\\n    \\s\n    \\end{pmatrix},~\n    \\tilde{\\b} = \\begin{pmatrix}\n     {\\b}  -{\\A}_I{\\x}^{(i)}_I\\\\\n     {\\f} - {\\E}_I{\\x}^{(i)}_I\n    \\end{pmatrix}.\n\\end{align*}\nLet $C$ be the closed convex cone spanned by the columns of $\\tilde{\\A}$. If $P = \\emptyset$ then $\\tilde{\\b} \\not\\in C$. Since $C$ is a closed, \n$\\exists\\,\\,\\delta_{i} > 0$ such that $\\mathcal{N}_{\\delta_i}(\\tilde{\\b}) \\cap C = \\emptyset$. Consider $\\u \\in \\R^m \\cap \\mathcal{N}_{\\delta_i}(\\bz)$, then $\\tilde{\\b} + \\begin{pmatrix}\\u\\\\\\bz\\end{pmatrix} \\not \\in C$.\n\nFor $J = \\set{i\\in [k]}{\\Phi({\\x}^{(i)}_I,\\bz) < \\infty}$, define $\\displaystyle\\delta_{\\min} \\define \\min_{i \\in [k]\\setminus J}\\delta_i$.\n\nConsider y such that $y  < \\hat{\\phi}(\\bz)$, it follows that \n\\begin{align*}\n    y & < \\hat{\\phi}(\\bz) < \\min_{i \\in J}\\Phi(\\bz,{\\x}^{(i)}_I) \\leq \\Phi(\\bz,{\\x}^{(i)}_I) .\n\\end{align*}\nHowever, since every $\\Phi({\\u},{\\x}^{(i)}_I), \\, i \\in J$ is lower semi-continuous at $\\bz$, there exists a $\\delta_i > 0$ such that $\\Phi({\\u},{\\x}^{(i)}_I) > y$ for every ${\\u} \\in \\mathcal{N}_{\\delta_i}(\\bz), \\, i \\in J$. Let $\\delta'_{\\min} = \\min(\\delta_i),\\, i \\in J$ and $\\delta'' = \\min(\\delta_{\\min},\\delta'_{\\min})$. This means that for every ${\\u} \\in \\mathcal{N}_{\\delta''}(\\bz)$ $\\Phi({\\u},{\\x}^{(i)}_I) > y \\implies  \\hat{\\phi}({\\u}) > y$. The result follows.\n\\end{proof}","start":55814,"end":57933},{"raw_tex":"Consider $\\ocalU$ as defined in \\cref{cor:4}, it follows from \\cref{lem:6} that $\\lim_{\\u\\to \\bz} \\hat{\\phi}(\\u) = \\hat{\\phi}(\\bz)$ where $\\u \\in \\ocalU$. Now, for every $\\epsilon > 0$ there exists a $\\delta > 0$ such that\n\\begin{equation*}\n   \\abs{\\hat{\\phi}(\\u) - \\hat{\\phi}(\\bz)} < \\epsilon\\,\\, \\forall \\u\\,\\, \\in \\mathcal{N}_{\\delta}(\\bz) \\cap \\ocalU\n\\end{equation*}\n\\begin{lemma}\\label{lem:8}\nConsider the value function $\\hat{\\phi}(\\u)$ as defined earlier for $\\textup{\\MICP}_\\u$. There exists a $\\delta > 0$ such that for all $\\u \\in \\mathcal{N}_\\delta(\\bz) \\cap \\ocalU$, $\\displaystyle \\hat{\\phi}(\\u) = \\min_{\\x_I\\in S_{=}^{\\bz}} \\Phi(\\u,\\x_I)$ where $S_{=}^{\\bz} = \\set{\\x_I \\in S_I}{\\Phi(\\bz,\\x_I) = \\hat{\\phi}(\\bz)}$\n\\end{lemma}","start":57934,"end":58673},{"raw_tex":"\\begin{proof}\nWe can partition the index sets corresponding to solutions of $\\CPui$ into two sets, \n$$\nS_{>}^{\\bz} = \\set{\\x_I\\in S_I}{\\Phi(\\bz,\\x_I) > \\hat{\\phi}(\\bz)} \\text{ and } S_{=}^{\\bz} = \\set{\\x_I\\in S_I}{\\Phi(\\bz,\\x_I) = \\hat{\\phi}(\\bz)}.\n$$\nFor each $\\x^{(i)}_I \\in S_I$, define $ \\overline{\\calU}_i=\\set{\\u \\in \\calU}{\\Phi({\\u},\\x^{(i)}_I)\\leq \\Phi(\\bz,\\x^{(i)}_I)} $. If there is no limiting sequence to ${\\bz}$ in $\\overline{\\calU}_i$ then there exists a neighbourhood $\\delta_i$ such that $\\mathcal{N}_{\\delta_i}(\\bz) \\cap \\overline{\\calU}_i = \\emptyset$. Alternatively, if there does exist a limiting sequence to $\\bz$ and since $\\Phi({\\u},{\\x}_I^{(i)})$ is lower semi-continuous,\n\\begin{equation*}\n    \\lim_{{\\u}\\to \\bz} \\Phi({\\u},{\\x}^{(i)}_I) = \\Phi(\\bz,{\\x}^{(i)}_I)\n\\end{equation*}\nThus, for every $\\epsilon_i > 0$ there exists a $\\delta_i > 0$ such that\n\\begin{align*}\n    \\abs{\\Phi({\\u},{\\x}^{(i)}_I) - \\Phi(\\bz,{\\x}^{(i)}_I)}< \\epsilon_i \\,\\,\\,\\,\\, \\forall \\,\\,\\,\\,{\\u}\\in \\overline{\\calU}_i\\cap \\mathcal{N}_{\\delta_i}(\\bz)\n\\end{align*}\nFor $\\x^{(i)}_I \\in S_{>}^{\\bz}$, let $\\epsilon_i = \\Phi(\\bz,{\\x}^{(i)}_I) - \\hat{\\phi}(\\bz)$. Assuming that $\\Phi(\\bz,{\\x}_I^{(i)})$ is finite, $\\forall$ $\\u \\in \\overline{\\calU}_i\\cap \\mathcal{N}_{\\delta_i}(\\bz)$\n\\begin{align*}\n    \\Phi(\\bz,{\\x}^{(i)}_I) - \\Phi({\\u},{\\x}_I^{(i)}) & <  \\Phi(\\bz,{\\x}^{(i)}_I) - \\hat{\\phi}(\\bz)\\\\\n    \\hat{\\phi}(\\bz) & < \\Phi({\\u},{\\x}^{(i)}_I)\n\\end{align*}\nThus, for all $\\x^{(i)}_I\\in S_{>}^{\\bz}$ there exists a $\\delta_i$ such that $\\Phi({\\u},{\\x}_I^{(i)}) > \\hat{\\phi}(\\bz)$. Assigning $\\delta_{\\min} = \\min_{\\x^{(i)}_I\\in S^{\\bz}_{>}} \\delta_i$ yields $\\displaystyle \\hat{\\phi}({\\u}) = \\min_{\\x_I\\in S^{\\bz}_{=}}\\Phi({\\u},{\\x}_I)$ for every $\\u \\in \\mathcal{N}_{\\delta_{\\min}}(\\bz)\\cap \\overline{\\calU}$.\n\\end{proof}","start":58674,"end":60491},{"raw_tex":"\\begin{lemma}\\label{lem:9}\nConsider the value function $\\hat{\\phi}(\\u)$ as defined earlier for $\\textup{\\MICP}_\\u$. There exists a $\\delta > 0$ and $0 < \\Gamma< \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $$\\hat{\\phi}(\\bz)\\leq \\hat{\\phi}(\\u) + \\Gamma\\norm{\\u}.$$\n\\end{lemma}","start":60492,"end":60796},{"raw_tex":"\\begin{proof}\nConsider $S^{\\bz}_{=}$ as defined in \\cref{lem:8}. Analogously, define $$S_{=}^{\\u} \\define \\set{\\x_I\\in S_{=}^{\\bz}}{\\Phi(\\u,\\x_{I}) = \\hat{\\phi}(\\u)}.$$ For $\\x^{(i)}_I \\in S^{\\u}_{=}$, the first order necessary conditions for optimality (KKT conditions) for $\\CPui$ can be expressed as, %For  writing the KKT conditions\\\\\n\\begin{align}\\label{eq:32}\n\\begin{split}\n    \\nabla_{C}{f(\\x_I^{(i)},\\x_C}) & = \\A^\\top_C\\blambda^{\\u(i)}_{\\A_C} - \\E_C^\\top\\blambda^{\\u(i)}_{\\E_C}\\\\\n    \\A_C\\x_C & = \\b + \\u - \\A_I\\x^{(i)}_I\\\\\n    \\E_C\\x_C+\\E_I\\x^{(i)}_I & \\leq \\f\\\\\n    {\\blambda^{\\u(i)}_{\\E_C}}^\\top(\\f - \\E_C\\x_C - \\E_I\\x^{(i)}_I) & = \\bz\\\\\n    {\\blambda^{\\u(i)}_{\\A_C}}^\\top(\\A_C\\x_C + \\A_I\\x^{(i)}_I - \\b - \\u) & = \\bz\\\\\n    {\\blambda^{\\u(i)}_{\\E_C}} & \\geq \\bz\n\\end{split}\n\\end{align}\nDefine $H^{(j)}_{\\u} = \\set{\\x_C \\in \\R^{n_2}}{\\A_C\\x_C = \\b +\\u- \\A_I\\x^{(j)}_I}$. Let $(\\x_I^{(j)},\\x_C) \\in X \\cap H^{(j)}_{\\u}$ where $\\x_I^{(j)} \\in S_{=}^{\\u}$.\nAdditionally, let the corresponding Lagrange multipliers at $\\u = \\bz$ for $\\x^{(j)}_I$ be $\\blambda^{(j)}_{\\A_C}$ and $\\blambda^{(j)}_{\\E_C}$.\n\\begin{align*}\n    f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\nabla_C{f(\\x_I^{(j)}}\\x^{\\bz*}_C)^{\\top}(\\x^{\\u}_C -  \\x^{\\bz*}_C )\\\\ \n    & \\hspace{3cm}+\\nabla_I{f(\\x_I^{(j)}}, \\x^{\\bz*}_C)^{\\top}(\\x^{(j)}_I -  \\x^{(j)}_I)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) + ( \\blambda^{(j)^\\top}_{\\A_C}\\A_C - \\blambda^{(j)^\\top}_{\\E_C}\\E_C) (\\x^{\\u}_C -  \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}\\A_C(\\x^{\\u}_C -  \\x^{\\bz*}_C) - \\blambda^{(j)^\\top}_{\\E_C}\\E_C (\\x^{\\u}_C -  \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\A_C \\x^{\\u}_C -  \\A_C \\x^{\\bz*}_C) - \\blambda^{(j)^\\top}_{\\E_C}(\\E_C \\x^{\\u}_C -  \\E_C \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\b+\\u-\\A_I\\x^{(j)}_I - \\b + \\A_I\\x^{(j)}_I)\\\\ \n    & \\hspace{3cm}-\\blambda^{(j)^\\top}_{\\E_C}(\\E_C \\x^{\\u}_C -  \\f + \\E_I\\x^{(j)}_I)\n\\end{align*}\nwhere the equalities follow from the set of equations \\cref{eq:32}. Furthermore, observe that since $\\blambda^{(j)^\\top}_{\\E_C} \\geq \\bz$ and $\\E_C\\x^{\\u}_C + \\E\\x^{(j)}_I \\leq \\f$, it follows that \n\\begin{align*}\n   f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u), \\\\\n      \\inf_{X\\cap H^{(j)}_{\\u}}f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u), \\text{ i.e.,}\\\\\n   \\hat{\\phi}({\\u)} & \\geq \\hat{\\phi}(\\bz) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u)\n\\end{align*}\nRearranging the terms in the above inequality,     $-\\blambda^{(j)^\\top}_{\\A_C}\\u \\geq \\hat{\\phi}(\\bz) - \\hat{\\phi}({\\u)}$. As $\\hat{\\phi}(\\bz)\\geq \\hat{\\phi}(\\u)$ $\\forall\\,\\,\\, \\u \\in \\ocalU$, using Cauchy-Schwarz on the left-hand side of the inequality,\n\\begin{align*}\n    \\norm{\\blambda^{(j)}_{\\A_C}}\\cdot\\norm{\\u} \\geq \\hat{\\phi}(\\bz) - \\hat{\\phi}({\\u)} .\n\\end{align*}\nDefining $\\Gamma = \\max_{i\\in S^{\\bz}_{=} }{\\norm{\\blambda^{(j)}_{\\A_C}}}$ yields\n$$\n    \\hat{\\phi}(\\bz) - \\hat{\\phi}(\\u)\\leq \\Gamma\\norm{\\u}\\,\\,\\forall\\,\\, \\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap\\ocalU\n$$\nwhere $\\delta = \\delta_{min}$ as defined in the proof of \\cref{lem:8}. The result follows.\n\\end{proof}","start":60797,"end":64038},{"raw_tex":"This completes the proof of \\Cref{thm:3}(a). In the following section we present some cases where an equivalence can be established between (MICP) and (MICP) with bounded integer variables. In particular, we highlight that  if the objective function $f$ satisfies $\\rec(f)\\cap\\rec(F_R)\\setminus\\{\\bz\\} = \\emptyset$, then there exists $M < \\infty$ such that the following equivalence holds.\n$$ \\min\\set{f(\\x)}{\\A\\x = \\b, \\x \\in X} = \\min\\set{f(\\x)}{\\A\\x = \\b, \\norm{\\x}_{\\infty} \\leq M, \\x \\in X}.$$ Observe that if $f$ is $\\mu$-strongly convex, then $\\rec(f) = \\emptyset$, as the level sets of $f$ are bounded. \n","start":64039,"end":64651},{"raw_tex":"\\begin{lemma}\\label{lem:15}\nLet  $\\delta > 0$, and let $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$. Define\n$$\n    S^\\u \\define  \\set{\\x \\in X \\cap \\Hu}{ f(\\x) \\leq \\zIP}.\n$$\nIf \\textup{\\MICP} is feasible ($F \\neq\\emptyset$) and bounded and the recession cone of $f$ and recession cone of $F_R$, the feasible set of continuous relaxation of \\textup{\\MICP}, have no common non-zero directions of recession, then  $S^\\u$ is bounded.\n\\end{lemma}","start":75106,"end":75549},{"raw_tex":"\\begin{proof}\nConsider $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$. Since $\\norm{\\u}_\\infty \\leq \\norm{\\u} < \\delta$ we have $\\norm{\\b - \\A\\x}_{\\infty} < \\delta$, $\\forall\\,\\x \\in X \\cap \\Hu$. Alternatively, $\\forall\\,\\x \\in X \\cap \\Hu$, $-\\1\\delta < {\\b} - \\A\\x < \\delta\\1$. Define, for $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$\n\\begin{align*}\n  S^\\u \\define  \\set{\\x \\in X \\cap \\Hu}{ f(\\x) \\leq \\zIP}\n\\end{align*}\nAdditionally, consider\n\\begin{align*}\n   S = \\set{\\x\\in X_R}{\\begin{Bmatrix}\n                                    -{\\1}\\delta \\leq {\\b} - \\A\\x \\leq {\\1}\\delta \\\\\n                                    f(\\x) \\leq z_{IP}\n                        \\end{Bmatrix}}\n\\end{align*}\n%Here ${\\u}$ is taken over all feasible polyhedra i.e. this set $\\{\\A\\x = {\\b} + {\\u},\\,\\, {\\E}\\x\\leq {\\f}\\,\\, ,\\x\\in \\mathbb{Z}^{n_1}\\times \\mathbb{R}^{n_2}\\}$ is feasible.\nIndeed, $S^\\u \\subseteq S$. Furthermore, $\\rec(S) = \\rec(F_R) \\cap \\rec(f)$, and since $\\rec(F_R) \\cap \\rec(f) = \\{\\bz\\}$, it follows that $S$ is compact and consequently $S^\\u$ is bounded. In particular, $\\norm{\\x}_\\infty \\leq M < \\infty$, $\\forall$ $\\x \\in S^\\u$. %since MICP is feasible and optimally bounded S'' is non-empty.\\\\\n%This implies that one can prescribe a uniform bound on variables of the problem and hence $\\norm{\\x_I\\norm{\\leq M$.\n\\end{proof}","start":75550,"end":76873},{"raw_tex":"\\begin{corollary}\\label{cor:16}\nIf \\textup{\\MICP} is feasible ($F \\neq\\emptyset$) and bounded and the recession cone of $f$ and recession cone of $F_R$, the feasible set of continuous relaxation of \\textup{\\MICP}, have no common non-zero directions of recession, then for ${\\u} \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$, $\\phi(\\u) = \\hat{\\phi}(\\u)$ i.e.\n\\begin{optprog*}\n    {$\\phi(\\u) = $ minimize}  & \\objective{\\hspace{1.3cm}f({\\x_I},{\\x_C})}\\\\\n    subject to & \\hspace{0cm} \\A_I\\,\\x_I + \\A_C\\x_C = \\b + \\u\\\\\n    & \\E_I\\x_I + \\E_C\\x_C \\leq \\f \\hspace{7mm}\\\\\n    & \\norm{\\x_I}_{\\infty}\\leq M \\hspace{0.5cm}\\\\\n    % & \\x_I\\in \\Z_{+}^{n_1}\\hspace{0.4cm}\\\\ \n     & (\\x_I, \\x_C) \\in \\Z^{n_1}\\times\\R^{n_2}\\hspace{0.3cm}\n\\end{optprog*}\nFurthermore, there exists a $\\delta > 0$ and $0 < \\Gamma < \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $\\phi(\\bz)\\leq \\phi(\\u) + \\Gamma\\norm{\\u}$. Additionally, if $f$ is $\\mu$-strongly convex and $L$ smooth, then $\\Gamma = \\mathcal{O}\\left(\\frac{L\\beta\\gamma}{\\mu}\\right)$ where $\\displaystyle \\beta \\define \\max_{B \\in \\mathcal{B}}\\norm{B^{-1}}_F$, $\\mathcal{B}$ being the set of all possible invertible submatrices of $[\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^T_C]$ and $\\gamma$ depends on $f(\\bz)$, $\\norm{\\nabla{f(\\bz)}}$ and $f(\\bar{\\x})$ for any $\\bar{\\x} \\in F$.\n\\end{corollary}","start":76941,"end":78288},{"raw_tex":"\\begin{proof}\n\\Cref{lem:15}, along with \\cref{lem:8,lem:9} readily implies that $\\phi(\\u) = \\hat{\\phi}(\\u)$ and there exists a $\\delta > 0$ and $0 < \\Gamma < \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $\\phi(\\bz)\\leq \\phi(\\u) + \\Gamma\\norm{\\u}$.\n\nTo see the explicit bound on $\\Gamma$, recall the stationarity condition from set of equations \\cref{eq:32} at $\\u = \\bz$.\n\\begin{align}\\label{eq:stationarity}\n\\begin{split}\n    \\nabla_{C}{f(\\x_I^{(i)},\\x_C}) & = \\A^T_C(\\lambda^{(i)+}_{\\A_C} - \\lambda^{(i)-}_{\\A_C}) - {\\E}_C^T\\lambda_{{\\E}_C}\\\\\n    \\lambda_{\\E_C}^{\\top}(\\E\\x -\\f) &= 0\\\\\n    \\lambda^{+}_{\\A_C},\\lambda^{-}_{\\A_C} & \\geq \\bz\\\\\n     \\lambda_{{\\E}_C} &\\geq \\bz\n\\end{split}\n\\end{align}\n\nFor $(\\x_C,\\x^{(i)}_I)$ consider the complementary slackness conditions, that is if for some $j\\in [m]$, $\\E_{j}\\x < \\f_j$ where $\\E_{j}$ is a row of the matrix $\\E$ then the corresponding Lagrangian multiplier $\\lambda_{\\E_j} = 0$. Partition the set $[m]$ into two sets $J = \\{J_<,J_=\\}$, where $J_<=\\set{j\\in [m]}{\\E_j \\x <\\f_j}$ and $J_= =\\set{j\\in [m]}{\\E_j \\x = \\f_j}$.\\\\\n\nDefine $\\A_{aug} = [\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^{\\top}_C]$ and $\\A_{aug}^J$ as the sub-matrix which has all the columns of $\\A_{aug}$ except the columns $2n_2+J_<$.\n % $\\lambda^{(i)+}_{\\A_C},\\lambda^{(i)-}_{\\A_C}$. \n One possible solution to the system \\cref{eq:stationarity} is,\n \\begin{align}\\label{sol:lambda}\n    \\begin{pmatrix}\n     \\lambda^{+}_{\\A_C}\\\\\n      \\lambda^{-}_{\\A_C}\\\\\n      \\lambda_{{\\E}^{J_{=}}_C}\n    \\end{pmatrix} = B^{-1} \\nabla_C{f(\\x^{(i)}_I,\\x_C)}\n\\end{align}\nwhere $B$ is a basis matrix of $\\A_{aug}^J$.\n\nFurthermore, let $\\lambda_{\\A_C} = \\lambda^{+}_{\\A_C} - \\lambda^{-}_{\\A_C}$, this implies that \n\\begin{align}\n    \\norm{\\lambda_{\\A_C}} \\leq \\norm{\\lambda^{+}_{\\A_C}} + \\norm{\\lambda^{-}_{\\A_C}} \\label{ineq:lambda}    \n\\end{align}\nLet $\\x^*$ be an optimal solution to \\MICP, i.e. $f(\\x^*) = \\zIP$. Since $f$ is $\\mu$-strongly convex, we have\n\n\\begin{align*}\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 + \\nabla{f({\\bz)}}^\\top\\x^*  + f(\\bz) & \\leq z_{IP}\\\\\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 + \\nabla{f({\\bz)}}^\\top\\x^* - z_{IP} + f(\\bz)& \\leq 0\\\\\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 - \\norm{\\nabla{f({\\bz)}}}\\,\\norm{\\x^*} - \\left(z_{IP} - f(\\bz)\\right) & \\leq 0\n\\end{align*}\nIt follows that,\n$$\n    \\norm{\\x^*}\\leq \\frac{\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz})}^2 +2\\,\\mu\\cdot \\left(z_{IP} - f(\\bz)\\right)}}{\\mu}\n$$\nSince $f$ is L-smooth,\n\\begin{align*}\n    \\norm{\\nabla{f(\\x^*)} - \\nabla{f(\\bz)}} &\\leq L\\norm{\\x^*}\\\\\n    \\norm{\\nabla{f(\\x^*)}} &\\leq L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\n\\end{align*}\nFrom \\cref{sol:lambda} and \\cref{ineq:lambda} we obtain\n\\begin{align*}\n    \\norm{\\lambda_{\\A_C}} & \\leq 2\\norm{B^{-1}}_F \\norm{\\nabla{f(\\x^*)}}\\\\\n    & \\leq 2\\norm{B^{-1}}_F\\left(L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\\right)\\\\\n    & \\leq 2\\beta\\left(L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\\right)\n\\end{align*}\nwhere $\\beta \\define \\max_{B \\in \\mathcal{B}}\\norm{B^{-1}}_F$ and $\\mathcal{B}$ is the set of all possible invertible submatrices of $[\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^T_C]$.\n\n\\begin{align}\n   \\norm{\\lambda_{\\A_C}} & \\leq \\dfrac{2\\beta\\left(L\\left(\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz})}^2 +2\\mu\\,\\left(z_{IP} - f(\\bz)\\right)}\\right) + \\mu\\norm{\\nabla{f(\\bz)}}\\right)}{\\mu}\\nonumber\\\\\n    & \\leq  \\dfrac{2\\beta L}{\\mu}\\left(2\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz)}}^2 + 2\\mu\\,\\left(f(\\bar{\\x}) - f(\\bz)\\right)}\\right)\\label{ineq:thm3}\n\\end{align}\nwhere the second inequality follows as $\\zIP \\leq f(\\bar{\\x}),$ $\\forall\\,\\bar{\\x} \\in F$ and $L > \\mu$. Combining \\cref{ineq:thm3} with the construction of $\\Gamma$ in \\cref{lem:9}, yields the result.\n\\end{proof}","start":78289,"end":82023},{"raw_tex":"\\cref{thm:3} follows.\n","start":82024,"end":82046}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"作者收束有界整数部分","source_units":[{"raw_tex":"This completes the proof of \\Cref{thm:3}(a). In the following section we present some cases where an equivalence can be established between (MICP) and (MICP) with bounded integer variables. In particular, we highlight that  if the objective function $f$ satisfies $\\rec(f)\\cap\\rec(F_R)\\setminus\\{\\bz\\} = \\emptyset$, then there exists $M < \\infty$ such that the following equivalence holds.\n","start":64039,"end":64429}]},{"id":"S054","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"隐含有界性的过渡","source_units":[{"raw_tex":"This completes the proof of \\Cref{thm:3}(a). In the following section we present some cases where an equivalence can be established between (MICP) and (MICP) with bounded integer variables. In particular, we highlight that  if the objective function $f$ satisfies $\\rec(f)\\cap\\rec(F_R)\\setminus\\{\\bz\\} = \\emptyset$, then there exists $M < \\infty$ such that the following equivalence holds.\n$$ \\min\\set{f(\\x)}{\\A\\x = \\b, \\x \\in X} = \\min\\set{f(\\x)}{\\A\\x = \\b, \\norm{\\x}_{\\infty} \\leq M, \\x \\in X}.$$ Observe that if $f$ is $\\mu$-strongly convex, then $\\rec(f) = \\emptyset$, as the level sets of $f$ are bounded. \n","start":64039,"end":64651}]},{"id":"S055","semantic_type":"lemma","kind":"lemma","labels":["lem:15"],"declared_external":false,"source_only":false,"title":"扰动低目标集合有界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:15}\nLet  $\\delta > 0$, and let $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$. Define\n$$\n    S^\\u \\define  \\set{\\x \\in X \\cap \\Hu}{ f(\\x) \\leq \\zIP}.\n$$\nIf \\textup{\\MICP} is feasible ($F \\neq\\emptyset$) and bounded and the recession cone of $f$ and recession cone of $F_R$, the feasible set of continuous relaxation of \\textup{\\MICP}, have no common non-zero directions of recession, then  $S^\\u$ is bounded.\n\\end{lemma}","start":75106,"end":75549}]},{"id":"S056","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"低目标有界性作者证明","source_units":[{"raw_tex":"\\begin{proof}\nConsider $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$. Since $\\norm{\\u}_\\infty \\leq \\norm{\\u} < \\delta$ we have $\\norm{\\b - \\A\\x}_{\\infty} < \\delta$, $\\forall\\,\\x \\in X \\cap \\Hu$. Alternatively, $\\forall\\,\\x \\in X \\cap \\Hu$, $-\\1\\delta < {\\b} - \\A\\x < \\delta\\1$. Define, for $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$\n\\begin{align*}\n  S^\\u \\define  \\set{\\x \\in X \\cap \\Hu}{ f(\\x) \\leq \\zIP}\n\\end{align*}\nAdditionally, consider\n\\begin{align*}\n   S = \\set{\\x\\in X_R}{\\begin{Bmatrix}\n                                    -{\\1}\\delta \\leq {\\b} - \\A\\x \\leq {\\1}\\delta \\\\\n                                    f(\\x) \\leq z_{IP}\n                        \\end{Bmatrix}}\n\\end{align*}\n%Here ${\\u}$ is taken over all feasible polyhedra i.e. this set $\\{\\A\\x = {\\b} + {\\u},\\,\\, {\\E}\\x\\leq {\\f}\\,\\, ,\\x\\in \\mathbb{Z}^{n_1}\\times \\mathbb{R}^{n_2}\\}$ is feasible.\nIndeed, $S^\\u \\subseteq S$. Furthermore, $\\rec(S) = \\rec(F_R) \\cap \\rec(f)$, and since $\\rec(F_R) \\cap \\rec(f) = \\{\\bz\\}$, it follows that $S$ is compact and consequently $S^\\u$ is bounded. In particular, $\\norm{\\x}_\\infty \\leq M < \\infty$, $\\forall$ $\\x \\in S^\\u$. %since MICP is feasible and optimally bounded S'' is non-empty.\\\\\n%This implies that one can prescribe a uniform bound on variables of the problem and hence $\\norm{\\x_I\\norm{\\leq M$.\n\\end{proof}","start":75550,"end":76873}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"共同残差盒包络","source_units":[{"raw_tex":"Consider $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$. Since $\\norm{\\u}_\\infty \\leq \\norm{\\u} < \\delta$ we have $\\norm{\\b - \\A\\x}_{\\infty} < \\delta$, $\\forall\\,\\x \\in X \\cap \\Hu$. Alternatively, $\\forall\\,\\x \\in X \\cap \\Hu$, $-\\1\\delta < {\\b} - \\A\\x < \\delta\\1$. Define, for $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$\n\\begin{align*}\n  S^\\u \\define  \\set{\\x \\in X \\cap \\Hu}{ f(\\x) \\leq \\zIP}\n\\end{align*}\nAdditionally, consider\n\\begin{align*}\n   S = \\set{\\x\\in X_R}{\\begin{Bmatrix}\n                                    -{\\1}\\delta \\leq {\\b} - \\A\\x \\leq {\\1}\\delta \\\\\n                                    f(\\x) \\leq z_{IP}\n                        \\end{Bmatrix}}\n\\end{align*}\n","start":75564,"end":76239}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"衰退锥零交推出统一界","source_units":[{"raw_tex":"Indeed, $S^\\u \\subseteq S$. Furthermore, $\\rec(S) = \\rec(F_R) \\cap \\rec(f)$, and since $\\rec(F_R) \\cap \\rec(f) = \\{\\bz\\}$, it follows that $S$ is compact and consequently $S^\\u$ is bounded. In particular, $\\norm{\\x}_\\infty \\leq M < \\infty$, $\\forall$ $\\x \\in S^\\u$. %since MICP is feasible and optimally bounded S'' is non-empty.\\\\\n","start":76413,"end":76745}]},{"id":"S059","semantic_type":"corollary","kind":"corollary","labels":["cor:16"],"declared_external":false,"source_only":false,"title":"局部有界截断值相同","source_units":[{"raw_tex":"\\begin{corollary}\\label{cor:16}\nIf \\textup{\\MICP} is feasible ($F \\neq\\emptyset$) and bounded and the recession cone of $f$ and recession cone of $F_R$, the feasible set of continuous relaxation of \\textup{\\MICP}, have no common non-zero directions of recession, then for ${\\u} \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$, $\\phi(\\u) = \\hat{\\phi}(\\u)$ i.e.\n\\begin{optprog*}\n    {$\\phi(\\u) = $ minimize}  & \\objective{\\hspace{1.3cm}f({\\x_I},{\\x_C})}\\\\\n    subject to & \\hspace{0cm} \\A_I\\,\\x_I + \\A_C\\x_C = \\b + \\u\\\\\n    & \\E_I\\x_I + \\E_C\\x_C \\leq \\f \\hspace{7mm}\\\\\n    & \\norm{\\x_I}_{\\infty}\\leq M \\hspace{0.5cm}\\\\\n    % & \\x_I\\in \\Z_{+}^{n_1}\\hspace{0.4cm}\\\\ \n     & (\\x_I, \\x_C) \\in \\Z^{n_1}\\times\\R^{n_2}\\hspace{0.3cm}\n\\end{optprog*}\n","start":76941,"end":77674}]},{"id":"S060","semantic_type":"corollary","kind":"corollary","labels":["cor:16"],"declared_external":false,"source_only":false,"title":"原值函数局部线性界","source_units":[{"raw_tex":"Furthermore, there exists a $\\delta > 0$ and $0 < \\Gamma < \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $\\phi(\\bz)\\leq \\phi(\\u) + \\Gamma\\norm{\\u}$. Additionally, if $f$ is $\\mu$-strongly convex and $L$ smooth, then $\\Gamma = \\mathcal{O}\\left(\\frac{L\\beta\\gamma}{\\mu}\\right)$ where $\\displaystyle \\beta \\define \\max_{B \\in \\mathcal{B}}\\norm{B^{-1}}_F$, $\\mathcal{B}$ being the set of all possible invertible submatrices of $[\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^T_C]$ and $\\gamma$ depends on $f(\\bz)$, $\\norm{\\nabla{f(\\bz)}}$ and $f(\\bar{\\x})$ for any $\\bar{\\x} \\in F$.\n\\end{corollary}","start":77674,"end":78288}]},{"id":"S061","semantic_type":"corollary","kind":"corollary","labels":["cor:16"],"declared_external":false,"source_only":false,"title":"局部常数的定量阶数","source_units":[{"raw_tex":"Furthermore, there exists a $\\delta > 0$ and $0 < \\Gamma < \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $\\phi(\\bz)\\leq \\phi(\\u) + \\Gamma\\norm{\\u}$. Additionally, if $f$ is $\\mu$-strongly convex and $L$ smooth, then $\\Gamma = \\mathcal{O}\\left(\\frac{L\\beta\\gamma}{\\mu}\\right)$ where $\\displaystyle \\beta \\define \\max_{B \\in \\mathcal{B}}\\norm{B^{-1}}_F$, $\\mathcal{B}$ being the set of all possible invertible submatrices of $[\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^T_C]$ and $\\gamma$ depends on $f(\\bz)$, $\\norm{\\nabla{f(\\bz)}}$ and $f(\\bar{\\x})$ for any $\\bar{\\x} \\in F$.\n\\end{corollary}","start":77674,"end":78288}]},{"id":"S062","semantic_type":"proof","kind":"proof","labels":["eq:stationarity","ineq:lambda","ineq:thm3","sol:lambda"],"declared_external":false,"source_only":false,"title":"推论十六作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\\Cref{lem:15}, along with \\cref{lem:8,lem:9} readily implies that $\\phi(\\u) = \\hat{\\phi}(\\u)$ and there exists a $\\delta > 0$ and $0 < \\Gamma < \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $\\phi(\\bz)\\leq \\phi(\\u) + \\Gamma\\norm{\\u}$.\n\nTo see the explicit bound on $\\Gamma$, recall the stationarity condition from set of equations \\cref{eq:32} at $\\u = \\bz$.\n\\begin{align}\\label{eq:stationarity}\n\\begin{split}\n    \\nabla_{C}{f(\\x_I^{(i)},\\x_C}) & = \\A^T_C(\\lambda^{(i)+}_{\\A_C} - \\lambda^{(i)-}_{\\A_C}) - {\\E}_C^T\\lambda_{{\\E}_C}\\\\\n    \\lambda_{\\E_C}^{\\top}(\\E\\x -\\f) &= 0\\\\\n    \\lambda^{+}_{\\A_C},\\lambda^{-}_{\\A_C} & \\geq \\bz\\\\\n     \\lambda_{{\\E}_C} &\\geq \\bz\n\\end{split}\n\\end{align}\n\nFor $(\\x_C,\\x^{(i)}_I)$ consider the complementary slackness conditions, that is if for some $j\\in [m]$, $\\E_{j}\\x < \\f_j$ where $\\E_{j}$ is a row of the matrix $\\E$ then the corresponding Lagrangian multiplier $\\lambda_{\\E_j} = 0$. Partition the set $[m]$ into two sets $J = \\{J_<,J_=\\}$, where $J_<=\\set{j\\in [m]}{\\E_j \\x <\\f_j}$ and $J_= =\\set{j\\in [m]}{\\E_j \\x = \\f_j}$.\\\\\n\nDefine $\\A_{aug} = [\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^{\\top}_C]$ and $\\A_{aug}^J$ as the sub-matrix which has all the columns of $\\A_{aug}$ except the columns $2n_2+J_<$.\n % $\\lambda^{(i)+}_{\\A_C},\\lambda^{(i)-}_{\\A_C}$. \n One possible solution to the system \\cref{eq:stationarity} is,\n \\begin{align}\\label{sol:lambda}\n    \\begin{pmatrix}\n     \\lambda^{+}_{\\A_C}\\\\\n      \\lambda^{-}_{\\A_C}\\\\\n      \\lambda_{{\\E}^{J_{=}}_C}\n    \\end{pmatrix} = B^{-1} \\nabla_C{f(\\x^{(i)}_I,\\x_C)}\n\\end{align}\nwhere $B$ is a basis matrix of $\\A_{aug}^J$.\n\nFurthermore, let $\\lambda_{\\A_C} = \\lambda^{+}_{\\A_C} - \\lambda^{-}_{\\A_C}$, this implies that \n\\begin{align}\n    \\norm{\\lambda_{\\A_C}} \\leq \\norm{\\lambda^{+}_{\\A_C}} + \\norm{\\lambda^{-}_{\\A_C}} \\label{ineq:lambda}    \n\\end{align}\nLet $\\x^*$ be an optimal solution to \\MICP, i.e. $f(\\x^*) = \\zIP$. Since $f$ is $\\mu$-strongly convex, we have\n\n\\begin{align*}\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 + \\nabla{f({\\bz)}}^\\top\\x^*  + f(\\bz) & \\leq z_{IP}\\\\\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 + \\nabla{f({\\bz)}}^\\top\\x^* - z_{IP} + f(\\bz)& \\leq 0\\\\\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 - \\norm{\\nabla{f({\\bz)}}}\\,\\norm{\\x^*} - \\left(z_{IP} - f(\\bz)\\right) & \\leq 0\n\\end{align*}\nIt follows that,\n$$\n    \\norm{\\x^*}\\leq \\frac{\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz})}^2 +2\\,\\mu\\cdot \\left(z_{IP} - f(\\bz)\\right)}}{\\mu}\n$$\nSince $f$ is L-smooth,\n\\begin{align*}\n    \\norm{\\nabla{f(\\x^*)} - \\nabla{f(\\bz)}} &\\leq L\\norm{\\x^*}\\\\\n    \\norm{\\nabla{f(\\x^*)}} &\\leq L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\n\\end{align*}\nFrom \\cref{sol:lambda} and \\cref{ineq:lambda} we obtain\n\\begin{align*}\n    \\norm{\\lambda_{\\A_C}} & \\leq 2\\norm{B^{-1}}_F \\norm{\\nabla{f(\\x^*)}}\\\\\n    & \\leq 2\\norm{B^{-1}}_F\\left(L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\\right)\\\\\n    & \\leq 2\\beta\\left(L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\\right)\n\\end{align*}\nwhere $\\beta \\define \\max_{B \\in \\mathcal{B}}\\norm{B^{-1}}_F$ and $\\mathcal{B}$ is the set of all possible invertible submatrices of $[\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^T_C]$.\n\n\\begin{align}\n   \\norm{\\lambda_{\\A_C}} & \\leq \\dfrac{2\\beta\\left(L\\left(\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz})}^2 +2\\mu\\,\\left(z_{IP} - f(\\bz)\\right)}\\right) + \\mu\\norm{\\nabla{f(\\bz)}}\\right)}{\\mu}\\nonumber\\\\\n    & \\leq  \\dfrac{2\\beta L}{\\mu}\\left(2\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz)}}^2 + 2\\mu\\,\\left(f(\\bar{\\x}) - f(\\bz)\\right)}\\right)\\label{ineq:thm3}\n\\end{align}\nwhere the second inequality follows as $\\zIP \\leq f(\\bar{\\x}),$ $\\forall\\,\\bar{\\x} \\in F$ and $L > \\mu$. Combining \\cref{ineq:thm3} with the construction of $\\Gamma$ in \\cref{lem:9}, yields the result.\n\\end{proof}","start":78289,"end":82023}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"统一界接回局部线性界","source_units":[{"raw_tex":"\\Cref{lem:15}, along with \\cref{lem:8,lem:9} readily implies that $\\phi(\\u) = \\hat{\\phi}(\\u)$ and there exists a $\\delta > 0$ and $0 < \\Gamma < \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $\\phi(\\bz)\\leq \\phi(\\u) + \\Gamma\\norm{\\u}$.\n","start":78303,"end":78565}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":["eq:stationarity"],"declared_external":false,"source_only":false,"title":"分裂乘子并删除松约束","source_units":[{"raw_tex":"To see the explicit bound on $\\Gamma$, recall the stationarity condition from set of equations \\cref{eq:32} at $\\u = \\bz$.\n\\begin{align}\\label{eq:stationarity}\n\\begin{split}\n    \\nabla_{C}{f(\\x_I^{(i)},\\x_C}) & = \\A^T_C(\\lambda^{(i)+}_{\\A_C} - \\lambda^{(i)-}_{\\A_C}) - {\\E}_C^T\\lambda_{{\\E}_C}\\\\\n    \\lambda_{\\E_C}^{\\top}(\\E\\x -\\f) &= 0\\\\\n    \\lambda^{+}_{\\A_C},\\lambda^{-}_{\\A_C} & \\geq \\bz\\\\\n     \\lambda_{{\\E}_C} &\\geq \\bz\n\\end{split}\n\\end{align}\n\nFor $(\\x_C,\\x^{(i)}_I)$ consider the complementary slackness conditions, that is if for some $j\\in [m]$, $\\E_{j}\\x < \\f_j$ where $\\E_{j}$ is a row of the matrix $\\E$ then the corresponding Lagrangian multiplier $\\lambda_{\\E_j} = 0$. Partition the set $[m]$ into two sets $J = \\{J_<,J_=\\}$, where $J_<=\\set{j\\in [m]}{\\E_j \\x <\\f_j}$ and $J_= =\\set{j\\in [m]}{\\E_j \\x = \\f_j}$.\\\\\n\nDefine $\\A_{aug} = [\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^{\\top}_C]$ and $\\A_{aug}^J$ as the sub-matrix which has all the columns of $\\A_{aug}$ except the columns $2n_2+J_<$.\n","start":78566,"end":79570}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":["sol:lambda"],"declared_external":false,"source_only":false,"title":"乘子的基逆表示","source_units":[{"raw_tex":" One possible solution to the system \\cref{eq:stationarity} is,\n \\begin{align}\\label{sol:lambda}\n    \\begin{pmatrix}\n     \\lambda^{+}_{\\A_C}\\\\\n      \\lambda^{-}_{\\A_C}\\\\\n      \\lambda_{{\\E}^{J_{=}}_C}\n    \\end{pmatrix} = B^{-1} \\nabla_C{f(\\x^{(i)}_I,\\x_C)}\n\\end{align}\nwhere $B$ is a basis matrix of $\\A_{aug}^J$.\n","start":79621,"end":79935}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":["ineq:lambda"],"declared_external":false,"source_only":false,"title":"正负乘子范数比较","source_units":[{"raw_tex":"Furthermore, let $\\lambda_{\\A_C} = \\lambda^{+}_{\\A_C} - \\lambda^{-}_{\\A_C}$, this implies that \n\\begin{align}\n    \\norm{\\lambda_{\\A_C}} \\leq \\norm{\\lambda^{+}_{\\A_C}} + \\norm{\\lambda^{-}_{\\A_C}} \\label{ineq:lambda}    \n\\end{align}\n","start":79936,"end":80167}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"强凸性控制最优点半径","source_units":[{"raw_tex":"Let $\\x^*$ be an optimal solution to \\MICP, i.e. $f(\\x^*) = \\zIP$. Since $f$ is $\\mu$-strongly convex, we have\n\n\\begin{align*}\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 + \\nabla{f({\\bz)}}^\\top\\x^*  + f(\\bz) & \\leq z_{IP}\\\\\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 + \\nabla{f({\\bz)}}^\\top\\x^* - z_{IP} + f(\\bz)& \\leq 0\\\\\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 - \\norm{\\nabla{f({\\bz)}}}\\,\\norm{\\x^*} - \\left(z_{IP} - f(\\bz)\\right) & \\leq 0\n\\end{align*}\nIt follows that,\n$$\n    \\norm{\\x^*}\\leq \\frac{\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz})}^2 +2\\,\\mu\\cdot \\left(z_{IP} - f(\\bz)\\right)}}{\\mu}\n$$\n","start":80167,"end":80744}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"光滑性控制最优点梯度","source_units":[{"raw_tex":"Since $f$ is L-smooth,\n\\begin{align*}\n    \\norm{\\nabla{f(\\x^*)} - \\nabla{f(\\bz)}} &\\leq L\\norm{\\x^*}\\\\\n    \\norm{\\nabla{f(\\x^*)}} &\\leq L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\n\\end{align*}\n","start":80744,"end":80930}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":["ineq:thm3"],"declared_external":false,"source_only":false,"title":"统一乘子界与Γ阶数","source_units":[{"raw_tex":"From \\cref{sol:lambda} and \\cref{ineq:lambda} we obtain\n\\begin{align*}\n    \\norm{\\lambda_{\\A_C}} & \\leq 2\\norm{B^{-1}}_F \\norm{\\nabla{f(\\x^*)}}\\\\\n    & \\leq 2\\norm{B^{-1}}_F\\left(L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\\right)\\\\\n    & \\leq 2\\beta\\left(L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\\right)\n\\end{align*}\nwhere $\\beta \\define \\max_{B \\in \\mathcal{B}}\\norm{B^{-1}}_F$ and $\\mathcal{B}$ is the set of all possible invertible submatrices of $[\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^T_C]$.\n\n\\begin{align}\n   \\norm{\\lambda_{\\A_C}} & \\leq \\dfrac{2\\beta\\left(L\\left(\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz})}^2 +2\\mu\\,\\left(z_{IP} - f(\\bz)\\right)}\\right) + \\mu\\norm{\\nabla{f(\\bz)}}\\right)}{\\mu}\\nonumber\\\\\n    & \\leq  \\dfrac{2\\beta L}{\\mu}\\left(2\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz)}}^2 + 2\\mu\\,\\left(f(\\bar{\\x}) - f(\\bz)\\right)}\\right)\\label{ineq:thm3}\n\\end{align}\nwhere the second inequality follows as $\\zIP \\leq f(\\bar{\\x}),$ $\\forall\\,\\bar{\\x} \\in F$ and $L > \\mu$. Combining \\cref{ineq:thm3} with the construction of $\\Gamma$ in \\cref{lem:9}, yields the result.\n","start":80930,"end":82012}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"作者最终收束定理","source_units":[{"raw_tex":"\\cref{thm:3} follows.\n","start":82024,"end":82046}]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第一项：Strong duality and KKT multipliers for attained differentiable convex optimization over affine constraints; Boyd and Vandenberghe, Section 5.2.3。原文第266行显式引用。"],"source_only":false,"title":"仿射凸优化强对偶与KKT","source_units":[]},{"id":"S072","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第二项：Value-function lower semicontinuity through strong duality; Bertsekas, Proposition 6.5.2, as invoked in B0102。原文第740行显式引用。"],"source_only":false,"title":"强对偶与值函数下半连续","source_units":[]},{"id":"S073","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第三项中的 Closedness of finitely generated cones。第756—757行直接使用，未另给文献定位。"],"source_only":false,"title":"有限生成锥闭性","source_units":[]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第四项：Recession-cone intersection rules and the boundedness criterion for nonempty closed convex sets。第1012行直接使用。"],"source_only":false,"title":"衰退锥交与有界性判据","source_units":[]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第五项中的 Convex first-order inequalities, Cauchy–Schwarz and matrix norm inequalities；三角不等式为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"凸一阶与范数估计","source_units":[]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第五项中的 the standard strong-convexity quadratic lower bound；第1062行实际使用。"],"source_only":false,"title":"标准强凸二次下界","source_units":[]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第三项中的 basic feasible solutions of nonnegative linear systems；第1045—1053行隐含使用。"],"source_only":false,"title":"非负线性系统基本解","source_units":[]},{"id":"R_C01","semantic_type":"retained_source","kind":"retained_source","labels":[],"declared_external":false,"source_only":false,"title":"原文保留片段","source_units":[{"raw_tex":"\n","start":11052,"end":11053}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S015","edges":[{"source_id":"S001","target_id":"S015","kind":"supports"},{"source_id":"S002","target_id":"S015","kind":"supports"},{"source_id":"S004","target_id":"S015","kind":"supports"},{"source_id":"S007","target_id":"S015","kind":"supports"},{"source_id":"S008","target_id":"S015","kind":"supports"},{"source_id":"S009","target_id":"S015","kind":"supports"},{"source_id":"S010","target_id":"S015","kind":"supports"},{"source_id":"S003","target_id":"S006","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S018","kind":"supports"},{"source_id":"S010","target_id":"S018","kind":"supports"},{"source_id":"S005","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S005","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S002","target_id":"S023","kind":"supports"},{"source_id":"S071","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S075","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S075","target_id":"S024","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S075","target_id":"S026","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S022","kind":"supports"},{"source_id":"S026","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S029","kind":"supports"},{"source_id":"S002","target_id":"S030","kind":"supports"},{"source_id":"S071","target_id":"S030","kind":"supports"},{"source_id":"S075","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S016","target_id":"S035","kind":"supports"},{"source_id":"S021","target_id":"S035","kind":"supports"},{"source_id":"S027","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S012","target_id":"S037","kind":"supports"},{"source_id":"S013","target_id":"S037","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S071","target_id":"S014","kind":"supports"},{"source_id":"S072","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S039","kind":"supports"},{"source_id":"S071","target_id":"S039","kind":"supports"},{"source_id":"S072","target_id":"S039","kind":"supports"},{"source_id":"S073","target_id":"S040","kind":"supports"},{"source_id":"S012","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S041","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S031","target_id":"S042","kind":"supports"},{"source_id":"S037","target_id":"S042","kind":"supports"},{"source_id":"S014","target_id":"S045","kind":"supports"},{"source_id":"S039","target_id":"S045","kind":"supports"},{"source_id":"S012","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S049","kind":"supports"},{"source_id":"S013","target_id":"S049","kind":"supports"},{"source_id":"S071","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S075","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S043","target_id":"S051","kind":"supports"},{"source_id":"S075","target_id":"S051","kind":"supports"},{"source_id":"S031","target_id":"S051","kind":"supports"},{"source_id":"S049","target_id":"S048","kind":"supports"},{"source_id":"S050","target_id":"S048","kind":"supports"},{"source_id":"S051","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S053","kind":"supports"},{"source_id":"S033","target_id":"S053","kind":"supports"},{"source_id":"S006","target_id":"S053","kind":"supports"},{"source_id":"S007","target_id":"S054","kind":"supports"},{"source_id":"S074","target_id":"S054","kind":"supports"},{"source_id":"S076","target_id":"S054","kind":"supports"},{"source_id":"S010","target_id":"S057","kind":"supports"},{"source_id":"S011","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S002","target_id":"S058","kind":"supports"},{"source_id":"S074","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S058","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S063","kind":"supports"},{"source_id":"S043","target_id":"S063","kind":"supports"},{"source_id":"S047","target_id":"S063","kind":"supports"},{"source_id":"S011","target_id":"S063","kind":"supports"},{"source_id":"S049","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S077","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S066","kind":"supports"},{"source_id":"S075","target_id":"S066","kind":"supports"},{"source_id":"S002","target_id":"S067","kind":"supports"},{"source_id":"S076","target_id":"S067","kind":"supports"},{"source_id":"S075","target_id":"S067","kind":"supports"},{"source_id":"S008","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S075","target_id":"S068","kind":"supports"},{"source_id":"S065","target_id":"S069","kind":"supports"},{"source_id":"S066","target_id":"S069","kind":"supports"},{"source_id":"S067","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S051","target_id":"S069","kind":"supports"},{"source_id":"S075","target_id":"S069","kind":"supports"},{"source_id":"S063","target_id":"S062","kind":"supports"},{"source_id":"S064","target_id":"S062","kind":"supports"},{"source_id":"S065","target_id":"S062","kind":"supports"},{"source_id":"S066","target_id":"S062","kind":"supports"},{"source_id":"S067","target_id":"S062","kind":"supports"},{"source_id":"S068","target_id":"S062","kind":"supports"},{"source_id":"S069","target_id":"S062","kind":"supports"},{"source_id":"S062","target_id":"S059","kind":"supports"},{"source_id":"S062","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S060","target_id":"S070","kind":"supports"},{"source_id":"S061","target_id":"S070","kind":"supports"},{"source_id":"S033","target_id":"S070","kind":"supports"},{"source_id":"S006","target_id":"S070","kind":"supports"},{"source_id":"S038","target_id":"S052","kind":"supports"},{"source_id":"S042","target_id":"S052","kind":"supports"},{"source_id":"S044","target_id":"S052","kind":"supports"},{"source_id":"S048","target_id":"S052","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S056","target_id":"S052","kind":"supports"},{"source_id":"S062","target_id":"S052","kind":"supports"},{"source_id":"S070","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S015","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S015","edges":[{"source_id":"S001","target_id":"S015","kind":"supports"},{"source_id":"S002","target_id":"S015","kind":"supports"},{"source_id":"S004","target_id":"S015","kind":"supports"},{"source_id":"S007","target_id":"S015","kind":"supports"},{"source_id":"S008","target_id":"S015","kind":"supports"},{"source_id":"S009","target_id":"S015","kind":"supports"},{"source_id":"S010","target_id":"S015","kind":"supports"},{"source_id":"S003","target_id":"S006","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S018","kind":"supports"},{"source_id":"S010","target_id":"S018","kind":"supports"},{"source_id":"S005","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S005","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S002","target_id":"S023","kind":"supports"},{"source_id":"S071","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S075","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S075","target_id":"S024","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S075","target_id":"S026","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S022","kind":"supports"},{"source_id":"S026","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S029","kind":"supports"},{"source_id":"S002","target_id":"S030","kind":"supports"},{"source_id":"S071","target_id":"S030","kind":"supports"},{"source_id":"S075","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S016","target_id":"S035","kind":"supports"},{"source_id":"S021","target_id":"S035","kind":"supports"},{"source_id":"S027","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S012","target_id":"S037","kind":"supports"},{"source_id":"S013","target_id":"S037","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S071","target_id":"S014","kind":"supports"},{"source_id":"S072","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S039","kind":"supports"},{"source_id":"S071","target_id":"S039","kind":"supports"},{"source_id":"S072","target_id":"S039","kind":"supports"},{"source_id":"S073","target_id":"S040","kind":"supports"},{"source_id":"S012","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S041","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S031","target_id":"S042","kind":"supports"},{"source_id":"S037","target_id":"S042","kind":"supports"},{"source_id":"S014","target_id":"S045","kind":"supports"},{"source_id":"S039","target_id":"S045","kind":"supports"},{"source_id":"S012","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S049","kind":"supports"},{"source_id":"S013","target_id":"S049","kind":"supports"},{"source_id":"S071","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S075","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S043","target_id":"S051","kind":"supports"},{"source_id":"S075","target_id":"S051","kind":"supports"},{"source_id":"S031","target_id":"S051","kind":"supports"},{"source_id":"S049","target_id":"S048","kind":"supports"},{"source_id":"S050","target_id":"S048","kind":"supports"},{"source_id":"S051","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S053","kind":"supports"},{"source_id":"S033","target_id":"S053","kind":"supports"},{"source_id":"S006","target_id":"S053","kind":"supports"},{"source_id":"S007","target_id":"S054","kind":"supports"},{"source_id":"S074","target_id":"S054","kind":"supports"},{"source_id":"S076","target_id":"S054","kind":"supports"},{"source_id":"S010","target_id":"S057","kind":"supports"},{"source_id":"S011","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S002","target_id":"S058","kind":"supports"},{"source_id":"S074","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S058","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S063","kind":"supports"},{"source_id":"S043","target_id":"S063","kind":"supports"},{"source_id":"S047","target_id":"S063","kind":"supports"},{"source_id":"S011","target_id":"S063","kind":"supports"},{"source_id":"S049","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S077","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S066","kind":"supports"},{"source_id":"S075","target_id":"S066","kind":"supports"},{"source_id":"S002","target_id":"S067","kind":"supports"},{"source_id":"S076","target_id":"S067","kind":"supports"},{"source_id":"S075","target_id":"S067","kind":"supports"},{"source_id":"S008","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S075","target_id":"S068","kind":"supports"},{"source_id":"S065","target_id":"S069","kind":"supports"},{"source_id":"S066","target_id":"S069","kind":"supports"},{"source_id":"S067","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S051","target_id":"S069","kind":"supports"},{"source_id":"S075","target_id":"S069","kind":"supports"},{"source_id":"S063","target_id":"S062","kind":"supports"},{"source_id":"S064","target_id":"S062","kind":"supports"},{"source_id":"S065","target_id":"S062","kind":"supports"},{"source_id":"S066","target_id":"S062","kind":"supports"},{"source_id":"S067","target_id":"S062","kind":"supports"},{"source_id":"S068","target_id":"S062","kind":"supports"},{"source_id":"S069","target_id":"S062","kind":"supports"},{"source_id":"S062","target_id":"S059","kind":"supports"},{"source_id":"S062","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S060","target_id":"S070","kind":"supports"},{"source_id":"S061","target_id":"S070","kind":"supports"},{"source_id":"S033","target_id":"S070","kind":"supports"},{"source_id":"S006","target_id":"S070","kind":"supports"},{"source_id":"S038","target_id":"S052","kind":"supports"},{"source_id":"S042","target_id":"S052","kind":"supports"},{"source_id":"S044","target_id":"S052","kind":"supports"},{"source_id":"S048","target_id":"S052","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S056","target_id":"S052","kind":"supports"},{"source_id":"S062","target_id":"S052","kind":"supports"},{"source_id":"S070","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S015","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"混合整数凸规划","source_units":[{"raw_tex":"Consider, to begin with, the following mixed integer convex programming (MICP) problem,\n\\begin{optprog*}\n    {$\\zIP = $ minimize}  & \\objective{\\hspace{3mm}f(\\x)}\\\\\n   \\textbf{(MICP)} \\qquad subject to & \\A\\,\\x\\,= \\b\\\\\n     & \\x \\in X \\hspace{1mm}\n\\end{optprog*}\nwhere $X:= \\set{\\x\\in \\Z^{n_1}\\times\\R^{n_2}}{\\E\\x\\leq \\f}$ is a polyhedral mixed integer set, $f:\\R^n \\mapsto \\R$ is a real valued convex function, $\\A,\\E$ and $\\b,\\f$ are full rank matrices and vectors of appropriate dimensions, respectively. Through out the remainder of this discussion we assume that the matrices $\\A, \\E$ are rational matrices and $\\b, \\f$ are rational vectors and $f(\\cdot)$ is differentiable. We further assume that both MICP and MICP's continuous relaxation are feasible and the corresponding optimal solutions exist.\\\\\n","start":10244,"end":11052}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"有理数据与最优取到","source_units":[{"raw_tex":"where $X:= \\set{\\x\\in \\Z^{n_1}\\times\\R^{n_2}}{\\E\\x\\leq \\f}$ is a polyhedral mixed integer set, $f:\\R^n \\mapsto \\R$ is a real valued convex function, $\\A,\\E$ and $\\b,\\f$ are full rank matrices and vectors of appropriate dimensions, respectively. Through out the remainder of this discussion we assume that the matrices $\\A, \\E$ are rational matrices and $\\b, \\f$ are rational vectors and $f(\\cdot)$ is differentiable. We further assume that both MICP and MICP's continuous relaxation are feasible and the corresponding optimal solutions exist.\\\\\n","start":10507,"end":11052}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"拉格朗日增广模型","source_units":[{"raw_tex":"Consider the Lagrangian relaxation of the MICP,\n\\begin{equation*}\n    \\textbf{(LR)} \\hspace{1cm} z^{LR}(\\blambda) = \\min_{\\x \\in X} f(\\x) + {\\blambda}^{\\top}(\\b - \\A\\,\\x).\n\\end{equation*}\nThe corresponding augmented Lagrangian relaxation and augmented Lagrangian dual (\\cite{boland2015augmented,gu2020exact}) of the \\MICP\\, are defined as,\n\\begin{align*}\n    \\textbf{(ALR)} \\hspace{0.5cm} z_{\\rho}^{LR+}(\\blambda) & = \\min_{\\x \\in X} f(\\x) + {\\blambda}^{\\top}(\\b - \\A\\,\\x) + \\rho\\, \\psi(\\b - \\A\\,\\x)\\\\\n    \\textbf{(ALD)} \\hspace{1cm} z_{\\rho}^{LD+} & = \\sup_{\\blambda\\in \\R^m}\\min_{\\x \\in X} f(\\x) + {\\blambda}^{\\top}(\\b - \\A\\,\\x) + \\rho\\, \\psi(\\b - \\A\\,\\x)\n\\end{align*}\n","start":11053,"end":11724}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"精确罚表示","source_units":[{"raw_tex":"where $\\psi:\\R^m \\mapsto \\R$ is a real valued function. We further designate $\\psi$ to have an \\textit{exact penalty representation} if $\\exists\\, 0 < \\rho < \\infty$, such that $z^{LD+}_{\\rho} = \\zIP$. In the case $\\psi(\\cdot) = \\norm{\\cdot}$ we augmented Lagrangian relaxation is referred to as sharp augmented Lagrangian relaxation. In particular, \n","start":11724,"end":12075}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"尖锐罚松弛","source_units":[{"raw_tex":"where $\\psi:\\R^m \\mapsto \\R$ is a real valued function. We further designate $\\psi$ to have an \\textit{exact penalty representation} if $\\exists\\, 0 < \\rho < \\infty$, such that $z^{LD+}_{\\rho} = \\zIP$. In the case $\\psi(\\cdot) = \\norm{\\cdot}$ we augmented Lagrangian relaxation is referred to as sharp augmented Lagrangian relaxation. In particular, \n$$\n    \\textbf{(SALR)} \\hspace{1cm} z_{SALR}(\\rho) = \\min_{\\x \\in X} f(\\x) + \\rho\\, \\norm{\\b - \\A\\,\\x}\\\\\n$$\n","start":11724,"end":12183}]},{"id":"S006","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"松弛上界与零乘子","source_units":[{"raw_tex":"It follows that $z^{LR+}_{\\rho}(\\blambda) \\leq \\zIP$ and $z^{LR+}_{\\rho}(\\bz) = z_{SALR}(\\rho)$. \n","start":12183,"end":12281}]},{"id":"S007","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"变量分块与衰退锥","source_units":[{"raw_tex":"\\subsection{Notation}\\label{sec:notation} The sets $\\R^n,\\Z^n,\\mathbb{Q}^n$ denote the set of real numbers, integers and rational numbers in $n-$dimensional vector space, respectively. Additionally, $\\R^n_+,\\Z^n_+,\\mathbb{Q}^n_+$ denote the non-negative counterparts of the respective sets. We use $[n]$ to denote the index set $\\{1,2 \\ldots, n\\}$. For an index set $E \\subset \\Z^+$ and $y\\in \\Z$, we define the translation of an index set as $y + E \\define \\set{x + y}{x \\in E}$. For a mixed integer set $S \\subseteq \\R^n \\times \\Z^m$, we denote by $S_R$ the continuous relaxation of $S$. Throughout the following discussion we define $I$ and $C$ as the index sets corresponding to integer and continuous variables, respectively. Analogously, for any $\\u \\in \\R^n$, let $\\u = (\\u_I,\\u_C)$ denote the partition of $\\u$ into the integer and continuous variables, respectively. Similarly for a symmetric matrix $\\Q \\in \\R^{n\\times n}$, let $Q_{UV}$ denote the submatrix of $\\Q$ formed by rows indexed by $U$ and columns indexed by $V$. For $\\delta > 0$ and $\\bar{\\x} \\in \\R^n$, we define by $\\mathcal{N}_{\\delta}(\\bar{\\x}) \\define \\set{\\x \\in \\R^n}{\\norm{\\x -\\bar{\\x}}\\leq \\delta}$, the ball with center $\\bar{\\x}$ and radius $\\delta$. The recession cone of a set $S$ is denoted by $\\rec(S)$. We also define, $\\rec(f)$, \\emph{recession cone of a convex function} $f:\\R^n \\mapsto \\R$ as the recession cone of a level set $\\set{x \\in \\R^n}{f(\\x)\\leq \\alpha}$, for some $\\alpha \\in \\R$ (\\cite{bertsekas2003convex}, Proposition 2.3.1). For given convex sets $A$ and $B$, we say that the sets $A$ and $B$ have \\emph{no common/distinct non-zero directions of recession} if $\\rec(A)\\cap \\rec(B)\\setminus \\{\\bz\\} = \\emptyset$.\\\\\n","start":12281,"end":14000}]},{"id":"S008","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"光滑性定义","source_units":[{"raw_tex":"We say that a function $g : \\R^m \\mapsto \\R$ is $L$ smooth, or alternatively, has \\textbf{Lipschitz-continuous} gradients if there exists a Lipschitz\nconstant, $L < \\infty$, such that\n$$\n    \\norm{\\nabla{g(\\x)} - \\nabla{g(y)}}\\leq\n    L\\norm{\\x-\\y}~~~~ \\forall~~ \\x,\\y \\in \\R^m.\n$$\n","start":14001,"end":14283}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"原文强凸性定义","source_units":[{"raw_tex":"Additionally, we say a function $g : \\R^m \\mapsto \\R$ is $\\mu$ \\textbf{strongly-convex} if there exists $ 0 <\\mu < \\infty$, such that\n$$\n    \\norm{\\nabla{g(\\x)} - \\nabla{g(y)}}\\geq\\mu\\norm{\\x-\\y}~~~~ \\forall~~ \\x,\\y \\in \\R^m.\n$$\n","start":14283,"end":14512}]},{"id":"S010","semantic_type":"definition","kind":"definition","labels":["def:valuefunction"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"扰动值函数","source_units":[{"raw_tex":"Consider the hyperplane $H_{\\u} \\define \\set{\\x\\in\\R^n}{\\A\\x = \\b + \\u}$, and let the corresponding value function associated with {\\MICP} be defined as \n\\begin{align}\n    \\phi(\\u) = \\min\\set{f(\\x)}{\\x \\in X\\cap\\Hu}.\\label{def:valuefunction}\n\\end{align}\nIn particular, $\\zIP = \\phi(\\bz) = \\min_{\\x\\in X \\cap \\Ho} f(\\x)$. Finally, we define the set $\\calU$ as the set of all possible perturbation vectors $\\u$ such that the feasible set of \\cref{def:valuefunction} is non-empty. In particular, $\\calU \\define \\set{\\u\\in \\R^m}{X \\cap \\Hu \\neq \\emptyset}$.\n","start":14922,"end":15476}]},{"id":"S011","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"低目标子集约化","source_units":[{"raw_tex":"Consider the partition of the feasible set $X = F_> \\cup F_{\\leq}, F_> \\cap F_{\\leq} = \\emptyset$ where $F_{\\leq} = \\set{\\x \\in X}{f(\\x) \\leq \\zIP}$ and $F_> =\\set{\\x \\in X}{f(\\x) > \\zIP}$. Observe that for $\\x \\in F_>$ any positive $\\rho > 0$ implies $f(\\x) + \\rho\\norm{\\b - \\A\\,\\x} > \\zIP$. \nThus, for MICP to have an exact penalty representation, it suffices to show that $\\exists$ $0 < \\rho < \\infty$ such that $f(\\x) + \\rho \\norm{\\b - \\A\\,\\x} \\geq \\zIP$, $\\forall \\x \\in F_{\\leq}$.\n","start":22903,"end":23390}]},{"id":"S012","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"统一整数界扰动模型","source_units":[{"raw_tex":"Consider the following mixed integer convex programming problem.\n\\begin{optprog*}\n    {$\\hat{\\phi}(\\u) = $ minimize}  & \\objective{\\hspace{1.3cm}f({\\x_I},{\\x_C})}\\\\\n    $\\textup{\\MICP}_\\u$ \\qquad subject to & \\hspace{0cm} \\A_I\\,\\x_I + \\A_C\\x_C = \\b + \\u\\\\\n    & \\E_I\\x_I + \\E_C\\x_C \\leq \\f \\hspace{7mm}\\\\\n    & \\norm{\\x_I}_{\\infty}\\leq M \\hspace{0.5cm}\\\\\n    % & \\x_I\\in \\Z_{+}^{n_1}\\hspace{0.4cm}\\\\ \n    & (\\x_I, \\x_C) \\in \\Z^{n_1}\\times\\R^{n_2}\\hspace{0.3cm}\n\\end{optprog*}\nwhere $0 \\leq M < \\infty$. Denote by $F_{\\u} \\define X \\cap \\Hu \\cap S_I $ the feasible region of $\\textup{\\MICP}_\\u$, where the set $S_I = \\set{\\x_I \\in \\Z^{n_1}}{\\norm{\\x_I}_{\\infty} \\leq M}$ is finite.  As before, we let $S_I = \\{\\x^{(1)}_I, \\x_I^{(2)}, \\ldots, \\x_I^{(k)}\\}$. Using this enumeration we characterize the value functions of the continuous restrictions, parameterized in $\\x_I^{(i)},\\,i \\in [k]$ as,\n","start":54192,"end":55085}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"固定整数连续限制","source_units":[{"raw_tex":"\\begin{optprog*}\n    {$\\Phi(\\u,\\x^{(i)}_I) = $ minimize}  & \\objective{\\hspace{6mm}f(\\x^{(i)}_I,\\x_C)}\\\\\n    $\\CPui$ \\qquad subject to & \\A_C\\,\\x_C = \\b + \\u - \\A_I\\x^{(i)}_{I}\\\\\n    & \\E_C\\x_C \\leq \\f - \\E_I\\x^{(i)}_{I}\\hspace{8mm}\\\\\n     & \\x_C \\in \\R^{n_2}\\hspace{1.8cm}\n\\end{optprog*}\nwhere $\\Phi(\\u,\\x^{(i)}_I)$ is the value function of $\\CPui$, a continuous convex  optimization problem. Since the constraints of $\\CPui$ are linear, Slater's condition holds and therefore the value function is lower semi-continuous.\n","start":55085,"end":55608}]},{"id":"S014","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"连续限制下半连续性","source_units":[{"raw_tex":"where $\\Phi(\\u,\\x^{(i)}_I)$ is the value function of $\\CPui$, a continuous convex  optimization problem. Since the constraints of $\\CPui$ are linear, Slater's condition holds and therefore the value function is lower semi-continuous.\n","start":55374,"end":55608}]},{"id":"S015","semantic_type":"theorem","kind":"theorem","labels":["thm:3"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"定理三精确罚表示","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm:3}\nConsider the following mixed integer convex programming problem,\n\\begin{optprog*}\n    minimize  & \\objective{\\hspace{1cm}f({\\x_I},{\\x_C})}\\\\\n    \\textup{\\MICP} \\hspace{2mm} subject to & \\hspace{0cm} \\A_I\\,\\x_I + \\A_C\\x_C = \\b\\hspace{6.5mm}\\\\\n    & \\E_I\\x_I + \\E_C\\x_C \\leq \\f \\hspace{7mm}\\\\\n    % & \\norm{\\x_I}_{\\infty}\\leq M \\hspace{0.5cm}\\\\\n    % & \\x_I\\in \\Z_{+}^{n_1}\\hspace{0.4cm}\\\\ \n     & (\\x_I, \\x_C) \\in \\Z^{n_1}\\times\\R^{n_2}\\hspace{0.3cm}\n\\end{optprog*}\nLet $F = X\\cap \\Ho$ denote the feasible region of \\textup{\\MICP}.\\\\\n\\begin{enumerate}[label = (\\alph*)]\n    \\item If $\\norm{\\x_I}_{\\infty}\\leq M,$ $\\forall\\,\\x \\in F$ then there exists an exact penalty representation for \\textup{\\MICP}.\n    \\item If the recession cone of $f$ and recession cone of $F_R$, the feasible set of continuous relaxation of \\textup{\\MICP}, have no common non-zero directions of recession, then there exists an exact penalty representation for \\textup{\\MICP}.\n\\end{enumerate} \nAdditionally, if $f$ is $\\mu$-strongly convex and $L$ smooth, then the finite penalty parameter $\\rho = \\mathcal{O}\\left(\\frac{L\\beta\\gamma}{\\mu}\\right)$ where $\\displaystyle \\beta \\define \\max_{B \\in \\mathcal{B}}\\norm{B^{-1}}_F$, $\\mathcal{B}$ being the set of all possible invertible submatrices of $[\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^T_C]$ and $\\gamma$ can be explicitly computed given $f(\\bz)$, $\\norm{\\nabla{f(\\bz)}}$ and $f(\\bar{\\x})$ for any $\\bar{\\x} \\in F$.\n% $\\gamma = \\sqrt{f(\\bar{\\x})}$ for any $\\bar{\\x} \\in F$.\\\\\n\\end{theorem}","start":19365,"end":20905}]},{"id":"S016","semantic_type":"lemma","kind":"lemma","labels":["lem:1"],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"SALR值函数表示","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:1}\n$\\displaystyle z_{SALR}(\\rho) = \\min_{\\u \\in \\calU}\\,\\, \\phi(\\u) + \\rho\\norm{\\u}$ . \n\\end{lemma}","start":21629,"end":21752}]},{"id":"S017","semantic_type":"proof","kind":"proof","labels":["eq:1","eq:2"],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"值函数表示作者证明","source_units":[{"raw_tex":"\\begin{proof}\nObserve the following inequality which follows immediately from the aforementioned definitions\n\\begin{align}\n    z_{SALR}(\\rho) = \\min_{\\x \\in X} f(\\x) + \\rho\\, \\norm{\\b - \\A\\,\\x} & \\leq \\min_{\\x\\in X \\cap \\Hu} f(\\x) + \\rho\\norm{\\u} ,\\,\\,\\,\\,\\, \\forall\\,\\u \\in \\calU \\nonumber\\\\\n                    & = \\phi(\\u) + \\rho\\norm{\\u} ,\\,\\,\\,\\,\\, \\forall\\,\\u \\in \\calU. \\label{eq:1}\n\\end{align}\nConversely, it holds that $\\forall\\,\\u \\in \\calU$\n\\begin{align*}\n    \\phi(\\u) + \\rho\\norm{\\u} & = \\min_{\\x\\in X \\cap \\Hu} f(\\x) + \\rho\\norm{\\u}\\\\\n                             & \\leq f(\\x) + \\rho\\norm{\\A\\x-\\b} ,\\,\\,\\,\\,\\, \\forall\\,\\x\\in X\\cap \\Hu.\n\\end{align*}\nThis implies that $ \\min_{\\u \\in \\calU} \\phi(\\u) + \\rho \\norm{\\u} \\leq f(\\x) + \\rho \\norm{\\A\\x - \\b} $ $\\forall~ x\\in X \\cap \\Hu,\\,\\forall~\\u \\in U$ which yields that $ \\min_{\\u \\in \\calU} \\phi(\\u) + \\rho \\norm{\\u} \\leq f(\\x) + \\rho \\norm{\\A\\x - \\b} $ $\\forall~ \\x \\in X $. This implies,\n\\begin{align}\n    \\min_{\\u\\in \\calU} \\phi(\\u) + \\rho\\norm{\\u} \\leq \\min_{\\x\\in X} f(\\x) + \\rho\\norm{\\A\\x-\\b}. \\label{eq:2}\n\\end{align}\nThe result follows from \\cref{eq:1} and \\cref{eq:2}.\n\\end{proof}","start":21753,"end":22902}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","labels":["eq:1"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"限制纤维所得上界","source_units":[{"raw_tex":"Observe the following inequality which follows immediately from the aforementioned definitions\n\\begin{align}\n    z_{SALR}(\\rho) = \\min_{\\x \\in X} f(\\x) + \\rho\\, \\norm{\\b - \\A\\,\\x} & \\leq \\min_{\\x\\in X \\cap \\Hu} f(\\x) + \\rho\\norm{\\u} ,\\,\\,\\,\\,\\, \\forall\\,\\u \\in \\calU \\nonumber\\\\\n                    & = \\phi(\\u) + \\rho\\norm{\\u} ,\\,\\,\\,\\,\\, \\forall\\,\\u \\in \\calU. \\label{eq:1}\n\\end{align}\n","start":21767,"end":22155}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","labels":["eq:2"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"遍历纤维所得下界","source_units":[{"raw_tex":"Conversely, it holds that $\\forall\\,\\u \\in \\calU$\n\\begin{align*}\n    \\phi(\\u) + \\rho\\norm{\\u} & = \\min_{\\x\\in X \\cap \\Hu} f(\\x) + \\rho\\norm{\\u}\\\\\n                             & \\leq f(\\x) + \\rho\\norm{\\A\\x-\\b} ,\\,\\,\\,\\,\\, \\forall\\,\\x\\in X\\cap \\Hu.\n\\end{align*}\nThis implies that $ \\min_{\\u \\in \\calU} \\phi(\\u) + \\rho \\norm{\\u} \\leq f(\\x) + \\rho \\norm{\\A\\x - \\b} $ $\\forall~ x\\in X \\cap \\Hu,\\,\\forall~\\u \\in U$ which yields that $ \\min_{\\u \\in \\calU} \\phi(\\u) + \\rho \\norm{\\u} \\leq f(\\x) + \\rho \\norm{\\A\\x - \\b} $ $\\forall~ \\x \\in X $. This implies,\n\\begin{align}\n    \\min_{\\u\\in \\calU} \\phi(\\u) + \\rho\\norm{\\u} \\leq \\min_{\\x\\in X} f(\\x) + \\rho\\norm{\\A\\x-\\b}. \\label{eq:2}\n\\end{align}\nThe result follows from \\cref{eq:1} and \\cref{eq:2}.\n","start":22155,"end":22891}]},{"id":"S020","semantic_type":"proposition","kind":"proposition","labels":["prop:2"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"扰动值的有限下界","source_units":[{"raw_tex":"\\begin{proposition}\\label{prop:2}\nConsider the continuous relaxation of MICP, and let the optimal objective value $z_R = \\min_{X_R\\cap \\Ho} f(\\x)$ be attained at $\\x_R$. \nLet $\\blambda_{\\A}$ and $\\blambda_{\\E}$ be the Lagrange multipliers for the constraints $\\A\\x = \\b$ and $\\E\\x\\leq \\f$, respectively.\\vspace{1mm}\n\\begin{enumerate}[label = \\roman*)]\n    \\item $\\phi(\\u) > -\\infty$ for all $\\u \\in \\calU$.\\vspace{1mm}\n","start":23391,"end":23810}]},{"id":"S021","semantic_type":"proposition","kind":"proposition","labels":["prop:2"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"低罚目标残差界","source_units":[{"raw_tex":"    \\item If $\\exists ~ \\boldsymbol{\\alpha} \\in X_R$ such that $f(\\boldsymbol{\\alpha}) + \\rho\\norm{\\b-\\A\\,\\boldsymbol{\\alpha}}\\leq \\zIP$ and $\\rho > \\norm{\\blambda_\\A}$  then \n        \\[\n                \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq \\frac{\\zIP-z_R}{\\rho -\\norm{\\blambda_{\\A}}}.\n        \\]    \n\\end{enumerate}\n\\end{proposition}","start":23810,"end":24147}]},{"id":"S022","semantic_type":"proof","kind":"proof","labels":["eq:4","ineq:1","ineq:2","ineq:gradient"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"命题二作者证明","source_units":[{"raw_tex":"\\begin{proof}\nConsider the Lagrangian function for the continuous relaxation of MICP, i.e. for $\\x \\in \\R^n$\n\\begin{equation*}\n    \\mathcal{L}(\\x,\\blambda_{\\A},\\blambda_{\\E}) = f(\\x) - \\blambda_\\A^{\\top}(\\A\\,\\x-\\b)-\\blambda_\\E^{\\top}(\\f-\\E\\x)\n\\end{equation*}\nAs the relaxed problem has only affine constraints strong duality holds (Chapter 5,Section 5.2.3\\cite{boyd2004convex}).\n\n As strong duality holds, the first order necessary (KKT) conditions for $\\mathcal{L}(\\x,\\blambda_{\\A},\\blambda_{\\E})$ can be characterized as\n\\begin{align}\\label{eq:4}\n\\begin{split}\n    \\nabla{f(\\x)} = \\A^{\\top}\\blambda_\\A - \\E^{\\top}{\\blambda_\\E}\\\\\n    \\blambda_\\E^{\\top}(\\f - \\E\\x) = 0\\\\\n    \\blambda_\\A^{\\top}(\\A\\,\\x - \\b) = 0\\\\\n    \\blambda_\\E \\geq \\bz\n\\end{split}\n\\end{align}\nObserve that $\\x_R$ satisfies the system of equations \\cref{eq:4}.\\\\\n\\begin{enumerate}[label = \\roman*)]\n    \\item To see the result, observe that it suffices to show that $\\phi(\\u) > -\\infty$ for all $\\u \\in \\calU$ such that $\\phi(\\u) \\leq \\phi(\\bz) = \\zIP$. Further, consider the set $F_{\\leq}$ defined as,\n            $$\n                F_{\\leq} \\define \\set{\\x \\in X}{f(\\x) \\leq \\zIP}.\n            $$\n            We have for all $\\x \\in F_{\\leq}$, $\\zIP \\geq f(\\x) \\geq f(\\x_R) + \\nabla{f(\\x_R)}(\\x-\\x_R)= z_R + \\nabla{f(\\x_R)}(\\x-\\x_R)$. Substituting from \\cref{ineq:gradient} we obtain, for all $\\x \\in F_{\\leq}$,\n            $$\n                \\zIP \\geq f(\\x) \\geq z_R + \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b) -  \\blambda_\\E^{\\top} (\\E\\x -  \\f )\n            $$\n            As $\\blambda_{\\E} \\geq \\bz$ and $\\x \\in X_R$,\n            \\begin{align*}\n                \\zIP \\geq f(\\x) & \\geq z_R + \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b)\\\\\n                & \\geq z_R - \\norm{\\blambda_{\\A}}\\norm{(\\A\\,\\x - \\b)}\\\\\n                & = z_R - \\norm{\\blambda_{\\A}}\\norm{\\u} > -\\infty.\n            \\end{align*}\n            Since $f(\\x) > -\\infty$ for all $\\x \\in F_{\\leq}$, the result follows.\\\\\n    \\item Since we have for $\\boldsymbol{\\alpha}\\in X_R$, $f(\\boldsymbol{\\alpha}) + \\rho\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq  \\zIP$; convexity of $f$ yields,\n            \\begin{align}\n                f(\\x_R)+ \\nabla{f(\\x_R)}^{\\top}(\\boldsymbol{\\alpha}-\\x_R) +\\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} &\\leq \\zIP , \\text{ i.e.,}\\nonumber \\\\\n                z_R+ \\nabla{f(\\x_R)}^{\\top}(\\boldsymbol{\\alpha}-\\x_R) +\\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} &\\leq \\zIP. \\label{ineq:1}\n            \\end{align}\n            It follows from strong duality that\n            \\begin{align}\n            \\nabla{f(\\x_R)}^{\\top}(\\x-\\x_R) &= \\blambda_{\\A}^{\\top} \\A(\\x - \\x_R) -\\blambda_\\E^{\\top} \\E(\\x-\\x_R)\\nonumber\\\\\n             &= \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b) -  \\blambda_\\E^{\\top} (\\E\\x -  \\f ). \\label{ineq:gradient}\n            \\end{align}\n            Substituting in \\cref{ineq:1}, we obtain\n            \\begin{align*}\n            z_R+ \\blambda_{\\A}^{\\top} (\\A\\,\\boldsymbol{\\alpha} - \\b) +  \\blambda_\\E^{\\top} (\\f - \\E\\boldsymbol{\\alpha}) +\\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b}  & \\leq \\zIP .\n            \\end{align*}\n            As $\\blambda_{\\E} \\geq \\bz$ and $\\boldsymbol{\\alpha} \\in X_R$, the above inequality can be rewritten as\n            % \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b) +   \\blambda_\\E^{\\top} ( \\f - \\E\\x ) +\\rho \\norm{\\A\\,\\x - \\b} & \\leq \\zIP - z_R\\\\\n            \\begin{align}\n            \\blambda_{\\A}^{\\top}(\\A\\,\\boldsymbol{\\alpha} - \\b) + \\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} & \\leq \\zIP - z_R \\label{ineq:2}\n            \\end{align}\n            Using Cauchy Schwarz inequality in \\cref{ineq:2} yields for all $\\x \\in X_R$,\n            \\begin{align*}\n                -\\norm{\\blambda_\\A}\\,\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} + \\rho\\,\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq \\zIP - z_R\\\\\n            (\\rho - \\norm{\\blambda_\\A})\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b}\\leq \\zIP - z_R\\\\\n            \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq \\frac{\\zIP-z_R}{\\rho - \\norm{\\blambda_\\A}}.\n            \\end{align*}\n\\end{enumerate}\n\\end{proof}","start":24148,"end":28140}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","labels":["eq:4"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"连续松弛KKT条件","source_units":[{"raw_tex":"Consider the Lagrangian function for the continuous relaxation of MICP, i.e. for $\\x \\in \\R^n$\n\\begin{equation*}\n    \\mathcal{L}(\\x,\\blambda_{\\A},\\blambda_{\\E}) = f(\\x) - \\blambda_\\A^{\\top}(\\A\\,\\x-\\b)-\\blambda_\\E^{\\top}(\\f-\\E\\x)\n\\end{equation*}\nAs the relaxed problem has only affine constraints strong duality holds (Chapter 5,Section 5.2.3\\cite{boyd2004convex}).\n\n As strong duality holds, the first order necessary (KKT) conditions for $\\mathcal{L}(\\x,\\blambda_{\\A},\\blambda_{\\E})$ can be characterized as\n\\begin{align}\\label{eq:4}\n\\begin{split}\n    \\nabla{f(\\x)} = \\A^{\\top}\\blambda_\\A - \\E^{\\top}{\\blambda_\\E}\\\\\n    \\blambda_\\E^{\\top}(\\f - \\E\\x) = 0\\\\\n    \\blambda_\\A^{\\top}(\\A\\,\\x - \\b) = 0\\\\\n    \\blambda_\\E \\geq \\bz\n\\end{split}\n\\end{align}\nObserve that $\\x_R$ satisfies the system of equations \\cref{eq:4}.\\\\\n","start":24162,"end":24979}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"固定扰动线性下界","source_units":[{"raw_tex":"    \\item To see the result, observe that it suffices to show that $\\phi(\\u) > -\\infty$ for all $\\u \\in \\calU$ such that $\\phi(\\u) \\leq \\phi(\\bz) = \\zIP$. Further, consider the set $F_{\\leq}$ defined as,\n            $$\n                F_{\\leq} \\define \\set{\\x \\in X}{f(\\x) \\leq \\zIP}.\n            $$\n            We have for all $\\x \\in F_{\\leq}$, $\\zIP \\geq f(\\x) \\geq f(\\x_R) + \\nabla{f(\\x_R)}(\\x-\\x_R)= z_R + \\nabla{f(\\x_R)}(\\x-\\x_R)$. Substituting from \\cref{ineq:gradient} we obtain, for all $\\x \\in F_{\\leq}$,\n            $$\n                \\zIP \\geq f(\\x) \\geq z_R + \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b) -  \\blambda_\\E^{\\top} (\\E\\x -  \\f )\n            $$\n            As $\\blambda_{\\E} \\geq \\bz$ and $\\x \\in X_R$,\n            \\begin{align*}\n                \\zIP \\geq f(\\x) & \\geq z_R + \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b)\\\\\n                & \\geq z_R - \\norm{\\blambda_{\\A}}\\norm{(\\A\\,\\x - \\b)}\\\\\n                & = z_R - \\norm{\\blambda_{\\A}}\\norm{\\u} > -\\infty.\n            \\end{align*}\n            Since $f(\\x) > -\\infty$ for all $\\x \\in F_{\\leq}$, the result follows.\\\\\n","start":25015,"end":26090}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":["ineq:1","ineq:gradient"],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"凸下界与梯度展开","source_units":[{"raw_tex":"    \\item Since we have for $\\boldsymbol{\\alpha}\\in X_R$, $f(\\boldsymbol{\\alpha}) + \\rho\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq  \\zIP$; convexity of $f$ yields,\n            \\begin{align}\n                f(\\x_R)+ \\nabla{f(\\x_R)}^{\\top}(\\boldsymbol{\\alpha}-\\x_R) +\\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} &\\leq \\zIP , \\text{ i.e.,}\\nonumber \\\\\n                z_R+ \\nabla{f(\\x_R)}^{\\top}(\\boldsymbol{\\alpha}-\\x_R) +\\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} &\\leq \\zIP. \\label{ineq:1}\n            \\end{align}\n            It follows from strong duality that\n            \\begin{align}\n            \\nabla{f(\\x_R)}^{\\top}(\\x-\\x_R) &= \\blambda_{\\A}^{\\top} \\A(\\x - \\x_R) -\\blambda_\\E^{\\top} \\E(\\x-\\x_R)\\nonumber\\\\\n             &= \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b) -  \\blambda_\\E^{\\top} (\\E\\x -  \\f ). \\label{ineq:gradient}\n            \\end{align}\n","start":26090,"end":26929}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","labels":["ineq:2"],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"删除松弛项并除正量","source_units":[{"raw_tex":"            Substituting in \\cref{ineq:1}, we obtain\n            \\begin{align*}\n            z_R+ \\blambda_{\\A}^{\\top} (\\A\\,\\boldsymbol{\\alpha} - \\b) +  \\blambda_\\E^{\\top} (\\f - \\E\\boldsymbol{\\alpha}) +\\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b}  & \\leq \\zIP .\n            \\end{align*}\n            As $\\blambda_{\\E} \\geq \\bz$ and $\\boldsymbol{\\alpha} \\in X_R$, the above inequality can be rewritten as\n            % \\blambda_{\\A}^{\\top} (\\A\\,\\x - \\b) +   \\blambda_\\E^{\\top} ( \\f - \\E\\x ) +\\rho \\norm{\\A\\,\\x - \\b} & \\leq \\zIP - z_R\\\\\n            \\begin{align}\n            \\blambda_{\\A}^{\\top}(\\A\\,\\boldsymbol{\\alpha} - \\b) + \\rho \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} & \\leq \\zIP - z_R \\label{ineq:2}\n            \\end{align}\n            Using Cauchy Schwarz inequality in \\cref{ineq:2} yields for all $\\x \\in X_R$,\n            \\begin{align*}\n                -\\norm{\\blambda_\\A}\\,\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} + \\rho\\,\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq \\zIP - z_R\\\\\n            (\\rho - \\norm{\\blambda_\\A})\\norm{\\A\\,\\boldsymbol{\\alpha} - \\b}\\leq \\zIP - z_R\\\\\n            \\norm{\\A\\,\\boldsymbol{\\alpha} - \\b} \\leq \\frac{\\zIP-z_R}{\\rho - \\norm{\\blambda_\\A}}.\n            \\end{align*}\n","start":26929,"end":28113}]},{"id":"S027","semantic_type":"corollary","kind":"corollary","labels":["cor:3"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"候选扰动半径界","source_units":[{"raw_tex":"\\begin{corollary}\\label{cor:3}\nConsider the set $U_\\rho = \\set{\\u \\in \\calU}{\\phi(\\u) + \\rho\\norm{\\u} \\leq \\phi(\\bz)}$. If $\\rho > \\norm{\\blambda_{\\A}}$ then for all $\\u \\in U_\\rho$, $$\\norm{\\u} \\leq \\frac{\\zIP-z_R}{\\rho -\\norm{\\blambda_\\A}}.$$\n\\end{corollary}","start":28284,"end":28544}]},{"id":"S028","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"候选扰动界作者证明","source_units":[{"raw_tex":"\\begin{proof}\nIf for any $~ \\u \\in U_{\\rho},~\\exists~ \\x \\in X\\cap H_{\\u}$ such that $f(\\x) + \\rho\\norm{\\A\\x - \\b} \\leq z_{IP}$, the result follows from \\Cref{prop:2}. Alternatively, if $f(\\x) + \\rho\\norm{\\A\\x - \\b} > z_{IP}\\,\\, \\forall~ \\x \\in X\\cap H_{\\u}$ for some $\\u \\in U_{\\rho}$. \n\\begin{align*}\n\\phi(\\u) + \\rho(\\norm{\\u}) =  \\inf_{X\\cap H_{\\u}} f(\\x) + \\rho\\norm{\\A\\x - \\b} \\geq z_{IP}\n\\end{align*} \nIf \\(\\inf_{X\\cap H_{\\u}} f(\\x) + \\rho\\norm{\\A\\x - \\b} > z_{IP}\\) then $\\u \\not\\in U_{\\rho}$ leading to a contradiction. If \\(\\inf_{X\\cap H_{\\u}} f(\\x) + \\rho\\norm{\\A\\x - \\b} = z_{IP}\\) and there exists an \\(\\x \\in X\\cap H_{\\u}\\) such that \\(f(\\x) + \\rho \\norm{\\A\\x - \\b} = z_{IP}\\) then the \nresult follows from \\cref{prop:2}. If there doesn't exist \\(\\x \\in X\\cap H_{\\u}\\) such that \\(f(\\x) + \\rho\\norm{\\A\\x - \\b}  = z_{IP}\\) then one can find a sequence of \\(\\x_{p}\\) such that the follows holds\n\\begin{align*}\n    f(\\x_{p}) + \\rho \\norm{\\A\\x_p - \\b} \\leq z_{IP} + \\frac{1}{p} \\text{ (From definition of infimum)}\\\\\n\\end{align*}\nLet \\(\\x_R\\) be the optimal solution of the optimization program \\(\\inf_{\\x \\in F_R} f(\\x)\\). Since we are minimizing a convex function over a rational polyhedron the KKT conditions hold as follows.\n\\begin{align*}   \n\\nabla{f(\\x_{R})} = \\A^{\\top}\\blambda_\\A - \\E^{\\top}{\\blambda_\\E}\\\\\n    \\blambda_\\E^{\\top}(\\f - \\E\\x_R) = 0\\\\\n    \\blambda_\\A^{\\top}(\\A\\,\\x_R - \\b) = 0\\\\\n    \\blambda_\\E \\geq \\bz\n\\end{align*}\n\\begin{align*} \n    f(\\x_R)+ \\nabla{f(\\x_R)}^{\\top}(\\x_{p}-\\x_R) +\\rho \\norm{\\A\\,\\x_p - \\b} &\\leq \\zIP + \\frac{1}{p}\\text{ (Convexity)}\\\\\n    z_{R} + \\blambda_{\\A}^{\\top} (\\A\\,\\x_p - \\b) +  \\blambda_\\E^{\\top} (\\f - \\E\\x_p) +\\rho \\norm{\\A\\,\\x_p - \\b} &\\leq \\zIP + \\frac{1}{p}\\\\\n    % (\\rho - \\norm{\\blambda_{\\A}}) \\norm{\\u} &\\leq \\frac{\\zIP - z_{R}}{\\rho - \\norm{\\blambda_\\A}} + \\frac{1}{p(\\rho - \\norm{\\blambda_{\\A}})}\\\\\n    \\norm{\\u} &\\leq \\frac{\\zIP - z_{R}}{\\rho - \\norm{\\blambda_{\\A}}} + \\frac{1}{p(\\rho - \\norm{\\blambda_{\\A}})}\n\\end{align*}\nwhere the last inequality follows from non-negativity of $\\blambda_\\E$ and $(\\f - \\E\\x_p).$ As \\(p\\) is arbitrary, we have as $p \\to \\infty$,\n\\begin{align*}\n    \\norm{\\u} &\\leq \\frac{\\zIP - z_{R}}{\\rho - \\norm{\\blambda_{\\A}}}\n\\end{align*}\n\\end{proof}","start":28545,"end":30789}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"取到情形调用残差界","source_units":[{"raw_tex":"If for any $~ \\u \\in U_{\\rho},~\\exists~ \\x \\in X\\cap H_{\\u}$ such that $f(\\x) + \\rho\\norm{\\A\\x - \\b} \\leq z_{IP}$, the result follows from \\Cref{prop:2}. Alternatively, if $f(\\x) + \\rho\\norm{\\A\\x - \\b} > z_{IP}\\,\\, \\forall~ \\x \\in X\\cap H_{\\u}$ for some $\\u \\in U_{\\rho}$. \n\\begin{align*}\n\\phi(\\u) + \\rho(\\norm{\\u}) =  \\inf_{X\\cap H_{\\u}} f(\\x) + \\rho\\norm{\\A\\x - \\b} \\geq z_{IP}\n\\end{align*} \nIf \\(\\inf_{X\\cap H_{\\u}} f(\\x) + \\rho\\norm{\\A\\x - \\b} > z_{IP}\\) then $\\u \\not\\in U_{\\rho}$ leading to a contradiction. If \\(\\inf_{X\\cap H_{\\u}} f(\\x) + \\rho\\norm{\\A\\x - \\b} = z_{IP}\\) and there exists an \\(\\x \\in X\\cap H_{\\u}\\) such that \\(f(\\x) + \\rho \\norm{\\A\\x - \\b} = z_{IP}\\) then the \nresult follows from \\cref{prop:2}. If there doesn't exist \\(\\x \\in X\\cap H_{\\u}\\) such that \\(f(\\x) + \\rho\\norm{\\A\\x - \\b}  = z_{IP}\\) then one can find a sequence of \\(\\x_{p}\\) such that the follows holds\n","start":28559,"end":29451}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"逼近序列处理未取到","source_units":[{"raw_tex":"result follows from \\cref{prop:2}. If there doesn't exist \\(\\x \\in X\\cap H_{\\u}\\) such that \\(f(\\x) + \\rho\\norm{\\A\\x - \\b}  = z_{IP}\\) then one can find a sequence of \\(\\x_{p}\\) such that the follows holds\n\\begin{align*}\n    f(\\x_{p}) + \\rho \\norm{\\A\\x_p - \\b} \\leq z_{IP} + \\frac{1}{p} \\text{ (From definition of infimum)}\\\\\n\\end{align*}\nLet \\(\\x_R\\) be the optimal solution of the optimization program \\(\\inf_{\\x \\in F_R} f(\\x)\\). Since we are minimizing a convex function over a rational polyhedron the KKT conditions hold as follows.\n\\begin{align*}   \n\\nabla{f(\\x_{R})} = \\A^{\\top}\\blambda_\\A - \\E^{\\top}{\\blambda_\\E}\\\\\n    \\blambda_\\E^{\\top}(\\f - \\E\\x_R) = 0\\\\\n    \\blambda_\\A^{\\top}(\\A\\,\\x_R - \\b) = 0\\\\\n    \\blambda_\\E \\geq \\bz\n\\end{align*}\n\\begin{align*} \n    f(\\x_R)+ \\nabla{f(\\x_R)}^{\\top}(\\x_{p}-\\x_R) +\\rho \\norm{\\A\\,\\x_p - \\b} &\\leq \\zIP + \\frac{1}{p}\\text{ (Convexity)}\\\\\n    z_{R} + \\blambda_{\\A}^{\\top} (\\A\\,\\x_p - \\b) +  \\blambda_\\E^{\\top} (\\f - \\E\\x_p) +\\rho \\norm{\\A\\,\\x_p - \\b} &\\leq \\zIP + \\frac{1}{p}\\\\\n    % (\\rho - \\norm{\\blambda_{\\A}}) \\norm{\\u} &\\leq \\frac{\\zIP - z_{R}}{\\rho - \\norm{\\blambda_\\A}} + \\frac{1}{p(\\rho - \\norm{\\blambda_{\\A}})}\\\\\n    \\norm{\\u} &\\leq \\frac{\\zIP - z_{R}}{\\rho - \\norm{\\blambda_{\\A}}} + \\frac{1}{p(\\rho - \\norm{\\blambda_{\\A}})}\n\\end{align*}\nwhere the last inequality follows from non-negativity of $\\blambda_\\E$ and $(\\f - \\E\\x_p).$ As \\(p\\) is arbitrary, we have as $p \\to \\infty$,\n\\begin{align*}\n    \\norm{\\u} &\\leq \\frac{\\zIP - z_{R}}{\\rho - \\norm{\\blambda_{\\A}}}\n\\end{align*}\n","start":29245,"end":30778}]},{"id":"S031","semantic_type":"corollary","kind":"corollary","labels":["cor:4"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"仅需控制降值扰动","source_units":[{"raw_tex":"\\begin{corollary}\\label{cor:4}\nConsider the set $\\,\\ocalU \\define \\set{\\u \\in \\calU}{\\phi(\\bz) \\geq \\phi(\\u)}$. For $\\alpha \\geq 0$, $\\phi(\\bz) \\leq \\phi(\\u) + \\alpha$ $\\forall \\,\\u \\in \\calU$ if $\\phi(\\bz) \\leq \\phi(\\u) + \\alpha$ $\\forall \\,\\u \\in \\ocalU$.\n\\end{corollary}","start":30791,"end":31064}]},{"id":"S032","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"降值约化作者证明","source_units":[{"raw_tex":"\\begin{proof}\n    For $\\u \\in \\calU\\setminus\\ocalU$, $\\phi(\\bz) < \\phi(\\u) \\leq \\phi(\\u) + \\alpha$. The result follows.\n\\end{proof}","start":31065,"end":31196}]},{"id":"S033","semantic_type":"lemma","kind":"lemma","labels":["lem:3"],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"局部线性界推出精确罚","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:3}\nIf there exist $\\delta, \\kappa > 0$ such that  $\\phi(\\bz) \\leq \\phi(\\u) + \\kappa \\norm{\\u}$ for all $\\u \\in \\mathcal{N}_{\\delta}(\\bz) \\cap \\calU$,  then there exists $0 < \\rho^{*} < \\infty$ such that $z_{SALR}(\\rho) = \\zIP$ for all $\\rho > \\rho^{*}$.\n\\end{lemma}","start":31197,"end":31486}]},{"id":"S034","semantic_type":"proof","kind":"proof","labels":["eq:9"],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"局部到全局作者证明","source_units":[{"raw_tex":"\\begin{proof}\nObserve that, for $\\rho > 0$ it suffices to consider $\\u \\in U_\\rho$. We have, from \\cref{lem:1} \n$$ z_{SALR}(\\rho) = \\min_{\\u\\in \\calU} \\phi(\\u) + \\rho \\norm{\\u}.$$\nIt follows that,\n\\begin{align*}\n    z_{SALR}(\\rho) = \\min_{\\u\\in U_\\rho}\\phi(\\u) + \\rho \\norm{\\u}\n\\end{align*}\n\\cref{prop:2} yields that for $\\rho > \\norm{\\blambda_{\\A}}$ and $\\u \\in U_{\\rho}$, $\\norm{\\u}$ is bounded. Thus, we have\n\\begin{equation*}\n    z_{SALR}(\\rho) = \\min_{\\u \\in \\mathcal{N}_{\\delta_{\\rho}}(\\bz) \\cap\\, U_\\rho}\\phi(\\u) + \\rho \\norm{\\u} \n\\end{equation*}\nwhere $\\displaystyle\\delta_{\\rho} = \\frac{\\zIP - z_R}{\\rho -\\norm{\\blambda_{\\A}}}$. From \\cref{prop:2} we see that as $\\rho \\to \\infty,\\, \\mathcal{N}_{\\delta_{\\rho}}(\\bz) \\cap U_\\rho \\to \\{\\bz\\}$. If $\\exists$ $\\delta > 0$, and $0 < \\kappa < \\infty$ such that $\\phi(\\bz) \\leq \\phi(\\u) + \\kappa \\norm{\\u}$ for all $\\u \\in \\mathcal{N}_{\\delta}(\\bz) \\cap U_\\rho$, it follows that,\n\\begin{align}\n    \\min_{\\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap\\, U_\\rho} \\phi(\\u) + \\kappa \\norm{\\u} = \\phi(\\bz) = \\zIP\\label{eq:9}\n\\end{align}\nAlternatively, we can increase $\\rho$ such that $\\forall\\, \\u \\in U_\\rho$, $\\norm{\\u} \\leq \\dfrac{\\zIP - z_R}{\\rho -\\norm{\\blambda_{\\A}}} < \\delta$.  Rearranging the terms, we obtain $\\rho > \\norm{\\blambda_\\A} + \\frac{\\zIP - z_R}{\\delta}$.  It follows from \\cref{eq:9} that for any choice of $\\displaystyle \\rho > \\max\\left\\{\\kappa, \\norm{\\blambda_\\A} + \\frac{\\zIP - z_R}{\\delta}\\right\\}$, we have  \n\\begin{align*}\n    z_{SALR}(\\rho) = \\min_{\\u\\in \\mathcal{N}_{\\delta_{\\rho}}(\\bz) \\cap\\, U_\\rho} \\phi(\\u) + \\rho \\norm{\\u} = \\zIP.\n\\end{align*}\n\\end{proof}","start":31487,"end":33114}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"候选扰动收缩到原点","source_units":[{"raw_tex":"Observe that, for $\\rho > 0$ it suffices to consider $\\u \\in U_\\rho$. We have, from \\cref{lem:1} \n$$ z_{SALR}(\\rho) = \\min_{\\u\\in \\calU} \\phi(\\u) + \\rho \\norm{\\u}.$$\nIt follows that,\n\\begin{align*}\n    z_{SALR}(\\rho) = \\min_{\\u\\in U_\\rho}\\phi(\\u) + \\rho \\norm{\\u}\n\\end{align*}\n\\cref{prop:2} yields that for $\\rho > \\norm{\\blambda_{\\A}}$ and $\\u \\in U_{\\rho}$, $\\norm{\\u}$ is bounded. Thus, we have\n\\begin{equation*}\n    z_{SALR}(\\rho) = \\min_{\\u \\in \\mathcal{N}_{\\delta_{\\rho}}(\\bz) \\cap\\, U_\\rho}\\phi(\\u) + \\rho \\norm{\\u} \n\\end{equation*}\nwhere $\\displaystyle\\delta_{\\rho} = \\frac{\\zIP - z_R}{\\rho -\\norm{\\blambda_{\\A}}}$. From \\cref{prop:2} we see that as $\\rho \\to \\infty,\\, \\mathcal{N}_{\\delta_{\\rho}}(\\bz) \\cap U_\\rho \\to \\{\\bz\\}$. If $\\exists$ $\\delta > 0$, and $0 < \\kappa < \\infty$ such that $\\phi(\\bz) \\leq \\phi(\\u) + \\kappa \\norm{\\u}$ for all $\\u \\in \\mathcal{N}_{\\delta}(\\bz) \\cap U_\\rho$, it follows that,\n","start":31501,"end":32419}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","labels":["eq:9"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"显式有限罚参数阈值","source_units":[{"raw_tex":"where $\\displaystyle\\delta_{\\rho} = \\frac{\\zIP - z_R}{\\rho -\\norm{\\blambda_{\\A}}}$. From \\cref{prop:2} we see that as $\\rho \\to \\infty,\\, \\mathcal{N}_{\\delta_{\\rho}}(\\bz) \\cap U_\\rho \\to \\{\\bz\\}$. If $\\exists$ $\\delta > 0$, and $0 < \\kappa < \\infty$ such that $\\phi(\\bz) \\leq \\phi(\\u) + \\kappa \\norm{\\u}$ for all $\\u \\in \\mathcal{N}_{\\delta}(\\bz) \\cap U_\\rho$, it follows that,\n\\begin{align}\n    \\min_{\\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap\\, U_\\rho} \\phi(\\u) + \\kappa \\norm{\\u} = \\phi(\\bz) = \\zIP\\label{eq:9}\n\\end{align}\nAlternatively, we can increase $\\rho$ such that $\\forall\\, \\u \\in U_\\rho$, $\\norm{\\u} \\leq \\dfrac{\\zIP - z_R}{\\rho -\\norm{\\blambda_{\\A}}} < \\delta$.  Rearranging the terms, we obtain $\\rho > \\norm{\\blambda_\\A} + \\frac{\\zIP - z_R}{\\delta}$.  It follows from \\cref{eq:9} that for any choice of $\\displaystyle \\rho > \\max\\left\\{\\kappa, \\norm{\\blambda_\\A} + \\frac{\\zIP - z_R}{\\delta}\\right\\}$, we have  \n\\begin{align*}\n    z_{SALR}(\\rho) = \\min_{\\u\\in \\mathcal{N}_{\\delta_{\\rho}}(\\bz) \\cap\\, U_\\rho} \\phi(\\u) + \\rho \\norm{\\u} = \\zIP.\n\\end{align*}\n","start":32041,"end":33103}]},{"id":"S037","semantic_type":"lemma","kind":"lemma","labels":["lem:6"],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"有限整数最小值下半连续","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:6}\nIf the integer variables are bounded, then the value function $\\displaystyle \\hat{\\phi}(\\u) = \\min_{i \\in [k]} \\Phi(\\u,\\x^{(i)}_I)$ is lower semi-continuous at $\\bz$.\n\\end{lemma}","start":55608,"end":55813}]},{"id":"S038","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"有限最小值作者证明","source_units":[{"raw_tex":"\\begin{proof}\nObserve that for $\\u \\in \\calU, \\x^{(i)}_I \\in S_I$, $\\Phi(\\u,\\x^{(i)}_I)$ is lower semi-continuous at ${\\u}$ if and only if strong duality holds at ${\\u}$ [Proposition 6.5.2,\\cite{bertsekas2003convex}]. If for some $\\x^{(i)}_I$, $\\Phi(\\bz, \\x^{(i)}_I) < +\\infty$ then $\\Phi(\\u, \\x^{(i)}_I)$ is lower semi-continuous at $\\u = \\bz$ due to Slater Condition (in case of affine constraints only feasibility is required). On the contrary, $\\Phi(\\bz,\\x^{(i)}_I) = + \\infty$ implies that the polyhedron $P = \\set{\\x \\in \\R^{n_2}}{\\tilde{\\A}\\x = \\tilde{\\b}, \\,\\,\\x \\geq \\bz}$ is empty, where\n\\begin{align*}\n    \\tilde{\\A} = \\begin{pmatrix}\n      {\\A}_C & -{\\A}_C & \\bz\\\\\n      {\\E}_C & -{\\E}_C & \\mathbb{I}\n    \\end{pmatrix},~\n    \\x = \\begin{pmatrix}\n    \\x_C^+\\\\\n    \\x_C^-\\\\\n    \\s\n    \\end{pmatrix},~\n    \\tilde{\\b} = \\begin{pmatrix}\n     {\\b}  -{\\A}_I{\\x}^{(i)}_I\\\\\n     {\\f} - {\\E}_I{\\x}^{(i)}_I\n    \\end{pmatrix}.\n\\end{align*}\nLet $C$ be the closed convex cone spanned by the columns of $\\tilde{\\A}$. If $P = \\emptyset$ then $\\tilde{\\b} \\not\\in C$. Since $C$ is a closed, \n$\\exists\\,\\,\\delta_{i} > 0$ such that $\\mathcal{N}_{\\delta_i}(\\tilde{\\b}) \\cap C = \\emptyset$. Consider $\\u \\in \\R^m \\cap \\mathcal{N}_{\\delta_i}(\\bz)$, then $\\tilde{\\b} + \\begin{pmatrix}\\u\\\\\\bz\\end{pmatrix} \\not \\in C$.\n\nFor $J = \\set{i\\in [k]}{\\Phi({\\x}^{(i)}_I,\\bz) < \\infty}$, define $\\displaystyle\\delta_{\\min} \\define \\min_{i \\in [k]\\setminus J}\\delta_i$.\n\nConsider y such that $y  < \\hat{\\phi}(\\bz)$, it follows that \n\\begin{align*}\n    y & < \\hat{\\phi}(\\bz) < \\min_{i \\in J}\\Phi(\\bz,{\\x}^{(i)}_I) \\leq \\Phi(\\bz,{\\x}^{(i)}_I) .\n\\end{align*}\nHowever, since every $\\Phi({\\u},{\\x}^{(i)}_I), \\, i \\in J$ is lower semi-continuous at $\\bz$, there exists a $\\delta_i > 0$ such that $\\Phi({\\u},{\\x}^{(i)}_I) > y$ for every ${\\u} \\in \\mathcal{N}_{\\delta_i}(\\bz), \\, i \\in J$. Let $\\delta'_{\\min} = \\min(\\delta_i),\\, i \\in J$ and $\\delta'' = \\min(\\delta_{\\min},\\delta'_{\\min})$. This means that for every ${\\u} \\in \\mathcal{N}_{\\delta''}(\\bz)$ $\\Phi({\\u},{\\x}^{(i)}_I) > y \\implies  \\hat{\\phi}({\\u}) > y$. The result follows.\n\\end{proof}","start":55814,"end":57933}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"可行限制的强对偶分支","source_units":[{"raw_tex":"Observe that for $\\u \\in \\calU, \\x^{(i)}_I \\in S_I$, $\\Phi(\\u,\\x^{(i)}_I)$ is lower semi-continuous at ${\\u}$ if and only if strong duality holds at ${\\u}$ [Proposition 6.5.2,\\cite{bertsekas2003convex}]. If for some $\\x^{(i)}_I$, $\\Phi(\\bz, \\x^{(i)}_I) < +\\infty$ then $\\Phi(\\u, \\x^{(i)}_I)$ is lower semi-continuous at $\\u = \\bz$ due to Slater Condition (in case of affine constraints only feasibility is required). On the contrary, $\\Phi(\\bz,\\x^{(i)}_I) = + \\infty$ implies that the polyhedron $P = \\set{\\x \\in \\R^{n_2}}{\\tilde{\\A}\\x = \\tilde{\\b}, \\,\\,\\x \\geq \\bz}$ is empty, where\n","start":55828,"end":56412}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"不可行限制局部排除","source_units":[{"raw_tex":"Observe that for $\\u \\in \\calU, \\x^{(i)}_I \\in S_I$, $\\Phi(\\u,\\x^{(i)}_I)$ is lower semi-continuous at ${\\u}$ if and only if strong duality holds at ${\\u}$ [Proposition 6.5.2,\\cite{bertsekas2003convex}]. If for some $\\x^{(i)}_I$, $\\Phi(\\bz, \\x^{(i)}_I) < +\\infty$ then $\\Phi(\\u, \\x^{(i)}_I)$ is lower semi-continuous at $\\u = \\bz$ due to Slater Condition (in case of affine constraints only feasibility is required). On the contrary, $\\Phi(\\bz,\\x^{(i)}_I) = + \\infty$ implies that the polyhedron $P = \\set{\\x \\in \\R^{n_2}}{\\tilde{\\A}\\x = \\tilde{\\b}, \\,\\,\\x \\geq \\bz}$ is empty, where\n\\begin{align*}\n    \\tilde{\\A} = \\begin{pmatrix}\n      {\\A}_C & -{\\A}_C & \\bz\\\\\n      {\\E}_C & -{\\E}_C & \\mathbb{I}\n    \\end{pmatrix},~\n    \\x = \\begin{pmatrix}\n    \\x_C^+\\\\\n    \\x_C^-\\\\\n    \\s\n    \\end{pmatrix},~\n    \\tilde{\\b} = \\begin{pmatrix}\n     {\\b}  -{\\A}_I{\\x}^{(i)}_I\\\\\n     {\\f} - {\\E}_I{\\x}^{(i)}_I\n    \\end{pmatrix}.\n\\end{align*}\nLet $C$ be the closed convex cone spanned by the columns of $\\tilde{\\A}$. If $P = \\emptyset$ then $\\tilde{\\b} \\not\\in C$. Since $C$ is a closed, \n$\\exists\\,\\,\\delta_{i} > 0$ such that $\\mathcal{N}_{\\delta_i}(\\tilde{\\b}) \\cap C = \\emptyset$. Consider $\\u \\in \\R^m \\cap \\mathcal{N}_{\\delta_i}(\\bz)$, then $\\tilde{\\b} + \\begin{pmatrix}\\u\\\\\\bz\\end{pmatrix} \\not \\in C$.\n\nFor $J = \\set{i\\in [k]}{\\Phi({\\x}^{(i)}_I,\\bz) < \\infty}$, define $\\displaystyle\\delta_{\\min} \\define \\min_{i \\in [k]\\setminus J}\\delta_i$.\n","start":55828,"end":57261}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"统一有限邻域","source_units":[{"raw_tex":"For $J = \\set{i\\in [k]}{\\Phi({\\x}^{(i)}_I,\\bz) < \\infty}$, define $\\displaystyle\\delta_{\\min} \\define \\min_{i \\in [k]\\setminus J}\\delta_i$.\n\nConsider y such that $y  < \\hat{\\phi}(\\bz)$, it follows that \n\\begin{align*}\n    y & < \\hat{\\phi}(\\bz) < \\min_{i \\in J}\\Phi(\\bz,{\\x}^{(i)}_I) \\leq \\Phi(\\bz,{\\x}^{(i)}_I) .\n\\end{align*}\nHowever, since every $\\Phi({\\u},{\\x}^{(i)}_I), \\, i \\in J$ is lower semi-continuous at $\\bz$, there exists a $\\delta_i > 0$ such that $\\Phi({\\u},{\\x}^{(i)}_I) > y$ for every ${\\u} \\in \\mathcal{N}_{\\delta_i}(\\bz), \\, i \\in J$. Let $\\delta'_{\\min} = \\min(\\delta_i),\\, i \\in J$ and $\\delta'' = \\min(\\delta_{\\min},\\delta'_{\\min})$. This means that for every ${\\u} \\in \\mathcal{N}_{\\delta''}(\\bz)$ $\\Phi({\\u},{\\x}^{(i)}_I) > y \\implies  \\hat{\\phi}({\\u}) > y$. The result follows.\n","start":57121,"end":57922}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"降值侧连续性桥接","source_units":[{"raw_tex":"Consider $\\ocalU$ as defined in \\cref{cor:4}, it follows from \\cref{lem:6} that $\\lim_{\\u\\to \\bz} \\hat{\\phi}(\\u) = \\hat{\\phi}(\\bz)$ where $\\u \\in \\ocalU$. Now, for every $\\epsilon > 0$ there exists a $\\delta > 0$ such that\n\\begin{equation*}\n   \\abs{\\hat{\\phi}(\\u) - \\hat{\\phi}(\\bz)} < \\epsilon\\,\\, \\forall \\u\\,\\, \\in \\mathcal{N}_{\\delta}(\\bz) \\cap \\ocalU\n\\end{equation*}\n","start":57934,"end":58305}]},{"id":"S043","semantic_type":"lemma","kind":"lemma","labels":["lem:8"],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"局部仅保留原点最优整数","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:8}\nConsider the value function $\\hat{\\phi}(\\u)$ as defined earlier for $\\textup{\\MICP}_\\u$. There exists a $\\delta > 0$ such that for all $\\u \\in \\mathcal{N}_\\delta(\\bz) \\cap \\ocalU$, $\\displaystyle \\hat{\\phi}(\\u) = \\min_{\\x_I\\in S_{=}^{\\bz}} \\Phi(\\u,\\x_I)$ where $S_{=}^{\\bz} = \\set{\\x_I \\in S_I}{\\Phi(\\bz,\\x_I) = \\hat{\\phi}(\\bz)}$\n\\end{lemma}","start":58305,"end":58673}]},{"id":"S044","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"最优整数筛选作者证明","source_units":[{"raw_tex":"\\begin{proof}\nWe can partition the index sets corresponding to solutions of $\\CPui$ into two sets, \n$$\nS_{>}^{\\bz} = \\set{\\x_I\\in S_I}{\\Phi(\\bz,\\x_I) > \\hat{\\phi}(\\bz)} \\text{ and } S_{=}^{\\bz} = \\set{\\x_I\\in S_I}{\\Phi(\\bz,\\x_I) = \\hat{\\phi}(\\bz)}.\n$$\nFor each $\\x^{(i)}_I \\in S_I$, define $ \\overline{\\calU}_i=\\set{\\u \\in \\calU}{\\Phi({\\u},\\x^{(i)}_I)\\leq \\Phi(\\bz,\\x^{(i)}_I)} $. If there is no limiting sequence to ${\\bz}$ in $\\overline{\\calU}_i$ then there exists a neighbourhood $\\delta_i$ such that $\\mathcal{N}_{\\delta_i}(\\bz) \\cap \\overline{\\calU}_i = \\emptyset$. Alternatively, if there does exist a limiting sequence to $\\bz$ and since $\\Phi({\\u},{\\x}_I^{(i)})$ is lower semi-continuous,\n\\begin{equation*}\n    \\lim_{{\\u}\\to \\bz} \\Phi({\\u},{\\x}^{(i)}_I) = \\Phi(\\bz,{\\x}^{(i)}_I)\n\\end{equation*}\nThus, for every $\\epsilon_i > 0$ there exists a $\\delta_i > 0$ such that\n\\begin{align*}\n    \\abs{\\Phi({\\u},{\\x}^{(i)}_I) - \\Phi(\\bz,{\\x}^{(i)}_I)}< \\epsilon_i \\,\\,\\,\\,\\, \\forall \\,\\,\\,\\,{\\u}\\in \\overline{\\calU}_i\\cap \\mathcal{N}_{\\delta_i}(\\bz)\n\\end{align*}\nFor $\\x^{(i)}_I \\in S_{>}^{\\bz}$, let $\\epsilon_i = \\Phi(\\bz,{\\x}^{(i)}_I) - \\hat{\\phi}(\\bz)$. Assuming that $\\Phi(\\bz,{\\x}_I^{(i)})$ is finite, $\\forall$ $\\u \\in \\overline{\\calU}_i\\cap \\mathcal{N}_{\\delta_i}(\\bz)$\n\\begin{align*}\n    \\Phi(\\bz,{\\x}^{(i)}_I) - \\Phi({\\u},{\\x}_I^{(i)}) & <  \\Phi(\\bz,{\\x}^{(i)}_I) - \\hat{\\phi}(\\bz)\\\\\n    \\hat{\\phi}(\\bz) & < \\Phi({\\u},{\\x}^{(i)}_I)\n\\end{align*}\nThus, for all $\\x^{(i)}_I\\in S_{>}^{\\bz}$ there exists a $\\delta_i$ such that $\\Phi({\\u},{\\x}_I^{(i)}) > \\hat{\\phi}(\\bz)$. Assigning $\\delta_{\\min} = \\min_{\\x^{(i)}_I\\in S^{\\bz}_{>}} \\delta_i$ yields $\\displaystyle \\hat{\\phi}({\\u}) = \\min_{\\x_I\\in S^{\\bz}_{=}}\\Phi({\\u},{\\x}_I)$ for every $\\u \\in \\mathcal{N}_{\\delta_{\\min}}(\\bz)\\cap \\overline{\\calU}$.\n\\end{proof}","start":58674,"end":60491}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"限制值函数的降值侧控制","source_units":[{"raw_tex":"We can partition the index sets corresponding to solutions of $\\CPui$ into two sets, \n$$\nS_{>}^{\\bz} = \\set{\\x_I\\in S_I}{\\Phi(\\bz,\\x_I) > \\hat{\\phi}(\\bz)} \\text{ and } S_{=}^{\\bz} = \\set{\\x_I\\in S_I}{\\Phi(\\bz,\\x_I) = \\hat{\\phi}(\\bz)}.\n$$\nFor each $\\x^{(i)}_I \\in S_I$, define $ \\overline{\\calU}_i=\\set{\\u \\in \\calU}{\\Phi({\\u},\\x^{(i)}_I)\\leq \\Phi(\\bz,\\x^{(i)}_I)} $. If there is no limiting sequence to ${\\bz}$ in $\\overline{\\calU}_i$ then there exists a neighbourhood $\\delta_i$ such that $\\mathcal{N}_{\\delta_i}(\\bz) \\cap \\overline{\\calU}_i = \\emptyset$. Alternatively, if there does exist a limiting sequence to $\\bz$ and since $\\Phi({\\u},{\\x}_I^{(i)})$ is lower semi-continuous,\n\\begin{equation*}\n    \\lim_{{\\u}\\to \\bz} \\Phi({\\u},{\\x}^{(i)}_I) = \\Phi(\\bz,{\\x}^{(i)}_I)\n\\end{equation*}\nThus, for every $\\epsilon_i > 0$ there exists a $\\delta_i > 0$ such that\n\\begin{align*}\n    \\abs{\\Phi({\\u},{\\x}^{(i)}_I) - \\Phi(\\bz,{\\x}^{(i)}_I)}< \\epsilon_i \\,\\,\\,\\,\\, \\forall \\,\\,\\,\\,{\\u}\\in \\overline{\\calU}_i\\cap \\mathcal{N}_{\\delta_i}(\\bz)\n\\end{align*}\n","start":58688,"end":59735}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"有限次优分支统一排除","source_units":[{"raw_tex":"For $\\x^{(i)}_I \\in S_{>}^{\\bz}$, let $\\epsilon_i = \\Phi(\\bz,{\\x}^{(i)}_I) - \\hat{\\phi}(\\bz)$. Assuming that $\\Phi(\\bz,{\\x}_I^{(i)})$ is finite, $\\forall$ $\\u \\in \\overline{\\calU}_i\\cap \\mathcal{N}_{\\delta_i}(\\bz)$\n\\begin{align*}\n    \\Phi(\\bz,{\\x}^{(i)}_I) - \\Phi({\\u},{\\x}_I^{(i)}) & <  \\Phi(\\bz,{\\x}^{(i)}_I) - \\hat{\\phi}(\\bz)\\\\\n    \\hat{\\phi}(\\bz) & < \\Phi({\\u},{\\x}^{(i)}_I)\n\\end{align*}\nThus, for all $\\x^{(i)}_I\\in S_{>}^{\\bz}$ there exists a $\\delta_i$ such that $\\Phi({\\u},{\\x}_I^{(i)}) > \\hat{\\phi}(\\bz)$. Assigning $\\delta_{\\min} = \\min_{\\x^{(i)}_I\\in S^{\\bz}_{>}} \\delta_i$ yields $\\displaystyle \\hat{\\phi}({\\u}) = \\min_{\\x_I\\in S^{\\bz}_{=}}\\Phi({\\u},{\\x}_I)$ for every $\\u \\in \\mathcal{N}_{\\delta_{\\min}}(\\bz)\\cap \\overline{\\calU}$.\n","start":59735,"end":60480}]},{"id":"S047","semantic_type":"lemma","kind":"lemma","labels":["lem:9"],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"有界整数局部线性界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:9}\nConsider the value function $\\hat{\\phi}(\\u)$ as defined earlier for $\\textup{\\MICP}_\\u$. There exists a $\\delta > 0$ and $0 < \\Gamma< \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $$\\hat{\\phi}(\\bz)\\leq \\hat{\\phi}(\\u) + \\Gamma\\norm{\\u}.$$\n\\end{lemma}","start":60492,"end":60796}]},{"id":"S048","semantic_type":"proof","kind":"proof","labels":["eq:32"],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"局部线性界作者证明","source_units":[{"raw_tex":"\\begin{proof}\nConsider $S^{\\bz}_{=}$ as defined in \\cref{lem:8}. Analogously, define $$S_{=}^{\\u} \\define \\set{\\x_I\\in S_{=}^{\\bz}}{\\Phi(\\u,\\x_{I}) = \\hat{\\phi}(\\u)}.$$ For $\\x^{(i)}_I \\in S^{\\u}_{=}$, the first order necessary conditions for optimality (KKT conditions) for $\\CPui$ can be expressed as, %For  writing the KKT conditions\\\\\n\\begin{align}\\label{eq:32}\n\\begin{split}\n    \\nabla_{C}{f(\\x_I^{(i)},\\x_C}) & = \\A^\\top_C\\blambda^{\\u(i)}_{\\A_C} - \\E_C^\\top\\blambda^{\\u(i)}_{\\E_C}\\\\\n    \\A_C\\x_C & = \\b + \\u - \\A_I\\x^{(i)}_I\\\\\n    \\E_C\\x_C+\\E_I\\x^{(i)}_I & \\leq \\f\\\\\n    {\\blambda^{\\u(i)}_{\\E_C}}^\\top(\\f - \\E_C\\x_C - \\E_I\\x^{(i)}_I) & = \\bz\\\\\n    {\\blambda^{\\u(i)}_{\\A_C}}^\\top(\\A_C\\x_C + \\A_I\\x^{(i)}_I - \\b - \\u) & = \\bz\\\\\n    {\\blambda^{\\u(i)}_{\\E_C}} & \\geq \\bz\n\\end{split}\n\\end{align}\nDefine $H^{(j)}_{\\u} = \\set{\\x_C \\in \\R^{n_2}}{\\A_C\\x_C = \\b +\\u- \\A_I\\x^{(j)}_I}$. Let $(\\x_I^{(j)},\\x_C) \\in X \\cap H^{(j)}_{\\u}$ where $\\x_I^{(j)} \\in S_{=}^{\\u}$.\nAdditionally, let the corresponding Lagrange multipliers at $\\u = \\bz$ for $\\x^{(j)}_I$ be $\\blambda^{(j)}_{\\A_C}$ and $\\blambda^{(j)}_{\\E_C}$.\n\\begin{align*}\n    f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\nabla_C{f(\\x_I^{(j)}}\\x^{\\bz*}_C)^{\\top}(\\x^{\\u}_C -  \\x^{\\bz*}_C )\\\\ \n    & \\hspace{3cm}+\\nabla_I{f(\\x_I^{(j)}}, \\x^{\\bz*}_C)^{\\top}(\\x^{(j)}_I -  \\x^{(j)}_I)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) + ( \\blambda^{(j)^\\top}_{\\A_C}\\A_C - \\blambda^{(j)^\\top}_{\\E_C}\\E_C) (\\x^{\\u}_C -  \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}\\A_C(\\x^{\\u}_C -  \\x^{\\bz*}_C) - \\blambda^{(j)^\\top}_{\\E_C}\\E_C (\\x^{\\u}_C -  \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\A_C \\x^{\\u}_C -  \\A_C \\x^{\\bz*}_C) - \\blambda^{(j)^\\top}_{\\E_C}(\\E_C \\x^{\\u}_C -  \\E_C \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\b+\\u-\\A_I\\x^{(j)}_I - \\b + \\A_I\\x^{(j)}_I)\\\\ \n    & \\hspace{3cm}-\\blambda^{(j)^\\top}_{\\E_C}(\\E_C \\x^{\\u}_C -  \\f + \\E_I\\x^{(j)}_I)\n\\end{align*}\nwhere the equalities follow from the set of equations \\cref{eq:32}. Furthermore, observe that since $\\blambda^{(j)^\\top}_{\\E_C} \\geq \\bz$ and $\\E_C\\x^{\\u}_C + \\E\\x^{(j)}_I \\leq \\f$, it follows that \n\\begin{align*}\n   f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u), \\\\\n      \\inf_{X\\cap H^{(j)}_{\\u}}f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u), \\text{ i.e.,}\\\\\n   \\hat{\\phi}({\\u)} & \\geq \\hat{\\phi}(\\bz) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u)\n\\end{align*}\nRearranging the terms in the above inequality,     $-\\blambda^{(j)^\\top}_{\\A_C}\\u \\geq \\hat{\\phi}(\\bz) - \\hat{\\phi}({\\u)}$. As $\\hat{\\phi}(\\bz)\\geq \\hat{\\phi}(\\u)$ $\\forall\\,\\,\\, \\u \\in \\ocalU$, using Cauchy-Schwarz on the left-hand side of the inequality,\n\\begin{align*}\n    \\norm{\\blambda^{(j)}_{\\A_C}}\\cdot\\norm{\\u} \\geq \\hat{\\phi}(\\bz) - \\hat{\\phi}({\\u)} .\n\\end{align*}\nDefining $\\Gamma = \\max_{i\\in S^{\\bz}_{=} }{\\norm{\\blambda^{(j)}_{\\A_C}}}$ yields\n$$\n    \\hat{\\phi}(\\bz) - \\hat{\\phi}(\\u)\\leq \\Gamma\\norm{\\u}\\,\\,\\forall\\,\\, \\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap\\ocalU\n$$\nwhere $\\delta = \\delta_{min}$ as defined in the proof of \\cref{lem:8}. The result follows.\n\\end{proof}","start":60797,"end":64038}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["eq:32"],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"活动整数分支与KKT","source_units":[{"raw_tex":"Consider $S^{\\bz}_{=}$ as defined in \\cref{lem:8}. Analogously, define $$S_{=}^{\\u} \\define \\set{\\x_I\\in S_{=}^{\\bz}}{\\Phi(\\u,\\x_{I}) = \\hat{\\phi}(\\u)}.$$ For $\\x^{(i)}_I \\in S^{\\u}_{=}$, the first order necessary conditions for optimality (KKT conditions) for $\\CPui$ can be expressed as, %For  writing the KKT conditions\\\\\n\\begin{align}\\label{eq:32}\n\\begin{split}\n    \\nabla_{C}{f(\\x_I^{(i)},\\x_C}) & = \\A^\\top_C\\blambda^{\\u(i)}_{\\A_C} - \\E_C^\\top\\blambda^{\\u(i)}_{\\E_C}\\\\\n    \\A_C\\x_C & = \\b + \\u - \\A_I\\x^{(i)}_I\\\\\n    \\E_C\\x_C+\\E_I\\x^{(i)}_I & \\leq \\f\\\\\n    {\\blambda^{\\u(i)}_{\\E_C}}^\\top(\\f - \\E_C\\x_C - \\E_I\\x^{(i)}_I) & = \\bz\\\\\n    {\\blambda^{\\u(i)}_{\\A_C}}^\\top(\\A_C\\x_C + \\A_I\\x^{(i)}_I - \\b - \\u) & = \\bz\\\\\n    {\\blambda^{\\u(i)}_{\\E_C}} & \\geq \\bz\n\\end{split}\n\\end{align}\nDefine $H^{(j)}_{\\u} = \\set{\\x_C \\in \\R^{n_2}}{\\A_C\\x_C = \\b +\\u- \\A_I\\x^{(j)}_I}$. Let $(\\x_I^{(j)},\\x_C) \\in X \\cap H^{(j)}_{\\u}$ where $\\x_I^{(j)} \\in S_{=}^{\\u}$.\nAdditionally, let the corresponding Lagrange multipliers at $\\u = \\bz$ for $\\x^{(j)}_I$ be $\\blambda^{(j)}_{\\A_C}$ and $\\blambda^{(j)}_{\\E_C}$.\n","start":60811,"end":61905}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"乘子给出限制线性下界","source_units":[{"raw_tex":"\\begin{align*}\n    f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\nabla_C{f(\\x_I^{(j)}}\\x^{\\bz*}_C)^{\\top}(\\x^{\\u}_C -  \\x^{\\bz*}_C )\\\\ \n    & \\hspace{3cm}+\\nabla_I{f(\\x_I^{(j)}}, \\x^{\\bz*}_C)^{\\top}(\\x^{(j)}_I -  \\x^{(j)}_I)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) + ( \\blambda^{(j)^\\top}_{\\A_C}\\A_C - \\blambda^{(j)^\\top}_{\\E_C}\\E_C) (\\x^{\\u}_C -  \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}\\A_C(\\x^{\\u}_C -  \\x^{\\bz*}_C) - \\blambda^{(j)^\\top}_{\\E_C}\\E_C (\\x^{\\u}_C -  \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\A_C \\x^{\\u}_C -  \\A_C \\x^{\\bz*}_C) - \\blambda^{(j)^\\top}_{\\E_C}(\\E_C \\x^{\\u}_C -  \\E_C \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\b+\\u-\\A_I\\x^{(j)}_I - \\b + \\A_I\\x^{(j)}_I)\\\\ \n    & \\hspace{3cm}-\\blambda^{(j)^\\top}_{\\E_C}(\\E_C \\x^{\\u}_C -  \\f + \\E_I\\x^{(j)}_I)\n\\end{align*}\nwhere the equalities follow from the set of equations \\cref{eq:32}. Furthermore, observe that since $\\blambda^{(j)^\\top}_{\\E_C} \\geq \\bz$ and $\\E_C\\x^{\\u}_C + \\E\\x^{(j)}_I \\leq \\f$, it follows that \n\\begin{align*}\n   f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u), \\\\\n      \\inf_{X\\cap H^{(j)}_{\\u}}f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u), \\text{ i.e.,}\\\\\n   \\hat{\\phi}({\\u)} & \\geq \\hat{\\phi}(\\bz) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u)\n","start":61905,"end":63346}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"有限乘子范数控制Γ","source_units":[{"raw_tex":"\\end{align*}\nRearranging the terms in the above inequality,     $-\\blambda^{(j)^\\top}_{\\A_C}\\u \\geq \\hat{\\phi}(\\bz) - \\hat{\\phi}({\\u)}$. As $\\hat{\\phi}(\\bz)\\geq \\hat{\\phi}(\\u)$ $\\forall\\,\\,\\, \\u \\in \\ocalU$, using Cauchy-Schwarz on the left-hand side of the inequality,\n\\begin{align*}\n    \\norm{\\blambda^{(j)}_{\\A_C}}\\cdot\\norm{\\u} \\geq \\hat{\\phi}(\\bz) - \\hat{\\phi}({\\u)} .\n\\end{align*}\nDefining $\\Gamma = \\max_{i\\in S^{\\bz}_{=} }{\\norm{\\blambda^{(j)}_{\\A_C}}}$ yields\n$$\n    \\hat{\\phi}(\\bz) - \\hat{\\phi}(\\u)\\leq \\Gamma\\norm{\\u}\\,\\,\\forall\\,\\, \\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap\\ocalU\n$$\nwhere $\\delta = \\delta_{min}$ as defined in the proof of \\cref{lem:8}. The result follows.\n","start":63346,"end":64027}]},{"id":"S052","semantic_type":"proof","kind":"proof","labels":["cor:16","eq:32","eq:stationarity","ineq:lambda","ineq:thm3","lem:15","lem:6","lem:8","lem:9","sol:lambda"],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"定理三作者证明汇总","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:6}\nIf the integer variables are bounded, then the value function $\\displaystyle \\hat{\\phi}(\\u) = \\min_{i \\in [k]} \\Phi(\\u,\\x^{(i)}_I)$ is lower semi-continuous at $\\bz$.\n\\end{lemma}","start":55608,"end":55813},{"raw_tex":"\\begin{proof}\nObserve that for $\\u \\in \\calU, \\x^{(i)}_I \\in S_I$, $\\Phi(\\u,\\x^{(i)}_I)$ is lower semi-continuous at ${\\u}$ if and only if strong duality holds at ${\\u}$ [Proposition 6.5.2,\\cite{bertsekas2003convex}]. If for some $\\x^{(i)}_I$, $\\Phi(\\bz, \\x^{(i)}_I) < +\\infty$ then $\\Phi(\\u, \\x^{(i)}_I)$ is lower semi-continuous at $\\u = \\bz$ due to Slater Condition (in case of affine constraints only feasibility is required). On the contrary, $\\Phi(\\bz,\\x^{(i)}_I) = + \\infty$ implies that the polyhedron $P = \\set{\\x \\in \\R^{n_2}}{\\tilde{\\A}\\x = \\tilde{\\b}, \\,\\,\\x \\geq \\bz}$ is empty, where\n\\begin{align*}\n    \\tilde{\\A} = \\begin{pmatrix}\n      {\\A}_C & -{\\A}_C & \\bz\\\\\n      {\\E}_C & -{\\E}_C & \\mathbb{I}\n    \\end{pmatrix},~\n    \\x = \\begin{pmatrix}\n    \\x_C^+\\\\\n    \\x_C^-\\\\\n    \\s\n    \\end{pmatrix},~\n    \\tilde{\\b} = \\begin{pmatrix}\n     {\\b}  -{\\A}_I{\\x}^{(i)}_I\\\\\n     {\\f} - {\\E}_I{\\x}^{(i)}_I\n    \\end{pmatrix}.\n\\end{align*}\nLet $C$ be the closed convex cone spanned by the columns of $\\tilde{\\A}$. If $P = \\emptyset$ then $\\tilde{\\b} \\not\\in C$. Since $C$ is a closed, \n$\\exists\\,\\,\\delta_{i} > 0$ such that $\\mathcal{N}_{\\delta_i}(\\tilde{\\b}) \\cap C = \\emptyset$. Consider $\\u \\in \\R^m \\cap \\mathcal{N}_{\\delta_i}(\\bz)$, then $\\tilde{\\b} + \\begin{pmatrix}\\u\\\\\\bz\\end{pmatrix} \\not \\in C$.\n\nFor $J = \\set{i\\in [k]}{\\Phi({\\x}^{(i)}_I,\\bz) < \\infty}$, define $\\displaystyle\\delta_{\\min} \\define \\min_{i \\in [k]\\setminus J}\\delta_i$.\n\nConsider y such that $y  < \\hat{\\phi}(\\bz)$, it follows that \n\\begin{align*}\n    y & < \\hat{\\phi}(\\bz) < \\min_{i \\in J}\\Phi(\\bz,{\\x}^{(i)}_I) \\leq \\Phi(\\bz,{\\x}^{(i)}_I) .\n\\end{align*}\nHowever, since every $\\Phi({\\u},{\\x}^{(i)}_I), \\, i \\in J$ is lower semi-continuous at $\\bz$, there exists a $\\delta_i > 0$ such that $\\Phi({\\u},{\\x}^{(i)}_I) > y$ for every ${\\u} \\in \\mathcal{N}_{\\delta_i}(\\bz), \\, i \\in J$. Let $\\delta'_{\\min} = \\min(\\delta_i),\\, i \\in J$ and $\\delta'' = \\min(\\delta_{\\min},\\delta'_{\\min})$. This means that for every ${\\u} \\in \\mathcal{N}_{\\delta''}(\\bz)$ $\\Phi({\\u},{\\x}^{(i)}_I) > y \\implies  \\hat{\\phi}({\\u}) > y$. The result follows.\n\\end{proof}","start":55814,"end":57933},{"raw_tex":"Consider $\\ocalU$ as defined in \\cref{cor:4}, it follows from \\cref{lem:6} that $\\lim_{\\u\\to \\bz} \\hat{\\phi}(\\u) = \\hat{\\phi}(\\bz)$ where $\\u \\in \\ocalU$. Now, for every $\\epsilon > 0$ there exists a $\\delta > 0$ such that\n\\begin{equation*}\n   \\abs{\\hat{\\phi}(\\u) - \\hat{\\phi}(\\bz)} < \\epsilon\\,\\, \\forall \\u\\,\\, \\in \\mathcal{N}_{\\delta}(\\bz) \\cap \\ocalU\n\\end{equation*}\n\\begin{lemma}\\label{lem:8}\nConsider the value function $\\hat{\\phi}(\\u)$ as defined earlier for $\\textup{\\MICP}_\\u$. There exists a $\\delta > 0$ such that for all $\\u \\in \\mathcal{N}_\\delta(\\bz) \\cap \\ocalU$, $\\displaystyle \\hat{\\phi}(\\u) = \\min_{\\x_I\\in S_{=}^{\\bz}} \\Phi(\\u,\\x_I)$ where $S_{=}^{\\bz} = \\set{\\x_I \\in S_I}{\\Phi(\\bz,\\x_I) = \\hat{\\phi}(\\bz)}$\n\\end{lemma}","start":57934,"end":58673},{"raw_tex":"\\begin{proof}\nWe can partition the index sets corresponding to solutions of $\\CPui$ into two sets, \n$$\nS_{>}^{\\bz} = \\set{\\x_I\\in S_I}{\\Phi(\\bz,\\x_I) > \\hat{\\phi}(\\bz)} \\text{ and } S_{=}^{\\bz} = \\set{\\x_I\\in S_I}{\\Phi(\\bz,\\x_I) = \\hat{\\phi}(\\bz)}.\n$$\nFor each $\\x^{(i)}_I \\in S_I$, define $ \\overline{\\calU}_i=\\set{\\u \\in \\calU}{\\Phi({\\u},\\x^{(i)}_I)\\leq \\Phi(\\bz,\\x^{(i)}_I)} $. If there is no limiting sequence to ${\\bz}$ in $\\overline{\\calU}_i$ then there exists a neighbourhood $\\delta_i$ such that $\\mathcal{N}_{\\delta_i}(\\bz) \\cap \\overline{\\calU}_i = \\emptyset$. Alternatively, if there does exist a limiting sequence to $\\bz$ and since $\\Phi({\\u},{\\x}_I^{(i)})$ is lower semi-continuous,\n\\begin{equation*}\n    \\lim_{{\\u}\\to \\bz} \\Phi({\\u},{\\x}^{(i)}_I) = \\Phi(\\bz,{\\x}^{(i)}_I)\n\\end{equation*}\nThus, for every $\\epsilon_i > 0$ there exists a $\\delta_i > 0$ such that\n\\begin{align*}\n    \\abs{\\Phi({\\u},{\\x}^{(i)}_I) - \\Phi(\\bz,{\\x}^{(i)}_I)}< \\epsilon_i \\,\\,\\,\\,\\, \\forall \\,\\,\\,\\,{\\u}\\in \\overline{\\calU}_i\\cap \\mathcal{N}_{\\delta_i}(\\bz)\n\\end{align*}\nFor $\\x^{(i)}_I \\in S_{>}^{\\bz}$, let $\\epsilon_i = \\Phi(\\bz,{\\x}^{(i)}_I) - \\hat{\\phi}(\\bz)$. Assuming that $\\Phi(\\bz,{\\x}_I^{(i)})$ is finite, $\\forall$ $\\u \\in \\overline{\\calU}_i\\cap \\mathcal{N}_{\\delta_i}(\\bz)$\n\\begin{align*}\n    \\Phi(\\bz,{\\x}^{(i)}_I) - \\Phi({\\u},{\\x}_I^{(i)}) & <  \\Phi(\\bz,{\\x}^{(i)}_I) - \\hat{\\phi}(\\bz)\\\\\n    \\hat{\\phi}(\\bz) & < \\Phi({\\u},{\\x}^{(i)}_I)\n\\end{align*}\nThus, for all $\\x^{(i)}_I\\in S_{>}^{\\bz}$ there exists a $\\delta_i$ such that $\\Phi({\\u},{\\x}_I^{(i)}) > \\hat{\\phi}(\\bz)$. Assigning $\\delta_{\\min} = \\min_{\\x^{(i)}_I\\in S^{\\bz}_{>}} \\delta_i$ yields $\\displaystyle \\hat{\\phi}({\\u}) = \\min_{\\x_I\\in S^{\\bz}_{=}}\\Phi({\\u},{\\x}_I)$ for every $\\u \\in \\mathcal{N}_{\\delta_{\\min}}(\\bz)\\cap \\overline{\\calU}$.\n\\end{proof}","start":58674,"end":60491},{"raw_tex":"\\begin{lemma}\\label{lem:9}\nConsider the value function $\\hat{\\phi}(\\u)$ as defined earlier for $\\textup{\\MICP}_\\u$. There exists a $\\delta > 0$ and $0 < \\Gamma< \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $$\\hat{\\phi}(\\bz)\\leq \\hat{\\phi}(\\u) + \\Gamma\\norm{\\u}.$$\n\\end{lemma}","start":60492,"end":60796},{"raw_tex":"\\begin{proof}\nConsider $S^{\\bz}_{=}$ as defined in \\cref{lem:8}. Analogously, define $$S_{=}^{\\u} \\define \\set{\\x_I\\in S_{=}^{\\bz}}{\\Phi(\\u,\\x_{I}) = \\hat{\\phi}(\\u)}.$$ For $\\x^{(i)}_I \\in S^{\\u}_{=}$, the first order necessary conditions for optimality (KKT conditions) for $\\CPui$ can be expressed as, %For  writing the KKT conditions\\\\\n\\begin{align}\\label{eq:32}\n\\begin{split}\n    \\nabla_{C}{f(\\x_I^{(i)},\\x_C}) & = \\A^\\top_C\\blambda^{\\u(i)}_{\\A_C} - \\E_C^\\top\\blambda^{\\u(i)}_{\\E_C}\\\\\n    \\A_C\\x_C & = \\b + \\u - \\A_I\\x^{(i)}_I\\\\\n    \\E_C\\x_C+\\E_I\\x^{(i)}_I & \\leq \\f\\\\\n    {\\blambda^{\\u(i)}_{\\E_C}}^\\top(\\f - \\E_C\\x_C - \\E_I\\x^{(i)}_I) & = \\bz\\\\\n    {\\blambda^{\\u(i)}_{\\A_C}}^\\top(\\A_C\\x_C + \\A_I\\x^{(i)}_I - \\b - \\u) & = \\bz\\\\\n    {\\blambda^{\\u(i)}_{\\E_C}} & \\geq \\bz\n\\end{split}\n\\end{align}\nDefine $H^{(j)}_{\\u} = \\set{\\x_C \\in \\R^{n_2}}{\\A_C\\x_C = \\b +\\u- \\A_I\\x^{(j)}_I}$. Let $(\\x_I^{(j)},\\x_C) \\in X \\cap H^{(j)}_{\\u}$ where $\\x_I^{(j)} \\in S_{=}^{\\u}$.\nAdditionally, let the corresponding Lagrange multipliers at $\\u = \\bz$ for $\\x^{(j)}_I$ be $\\blambda^{(j)}_{\\A_C}$ and $\\blambda^{(j)}_{\\E_C}$.\n\\begin{align*}\n    f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\nabla_C{f(\\x_I^{(j)}}\\x^{\\bz*}_C)^{\\top}(\\x^{\\u}_C -  \\x^{\\bz*}_C )\\\\ \n    & \\hspace{3cm}+\\nabla_I{f(\\x_I^{(j)}}, \\x^{\\bz*}_C)^{\\top}(\\x^{(j)}_I -  \\x^{(j)}_I)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) + ( \\blambda^{(j)^\\top}_{\\A_C}\\A_C - \\blambda^{(j)^\\top}_{\\E_C}\\E_C) (\\x^{\\u}_C -  \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}\\A_C(\\x^{\\u}_C -  \\x^{\\bz*}_C) - \\blambda^{(j)^\\top}_{\\E_C}\\E_C (\\x^{\\u}_C -  \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\A_C \\x^{\\u}_C -  \\A_C \\x^{\\bz*}_C) - \\blambda^{(j)^\\top}_{\\E_C}(\\E_C \\x^{\\u}_C -  \\E_C \\x^{\\bz*}_C)\\\\\n    & = f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\b+\\u-\\A_I\\x^{(j)}_I - \\b + \\A_I\\x^{(j)}_I)\\\\ \n    & \\hspace{3cm}-\\blambda^{(j)^\\top}_{\\E_C}(\\E_C \\x^{\\u}_C -  \\f + \\E_I\\x^{(j)}_I)\n\\end{align*}\nwhere the equalities follow from the set of equations \\cref{eq:32}. Furthermore, observe that since $\\blambda^{(j)^\\top}_{\\E_C} \\geq \\bz$ and $\\E_C\\x^{\\u}_C + \\E\\x^{(j)}_I \\leq \\f$, it follows that \n\\begin{align*}\n   f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u), \\\\\n      \\inf_{X\\cap H^{(j)}_{\\u}}f(\\x^{(j)}_I,\\x^{\\u}_C) & \\geq f(\\x^{(j)}_I,\\x^{\\bz^*}_C) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u), \\text{ i.e.,}\\\\\n   \\hat{\\phi}({\\u)} & \\geq \\hat{\\phi}(\\bz) +  \\blambda^{(j)^\\top}_{\\A_C}(\\u)\n\\end{align*}\nRearranging the terms in the above inequality,     $-\\blambda^{(j)^\\top}_{\\A_C}\\u \\geq \\hat{\\phi}(\\bz) - \\hat{\\phi}({\\u)}$. As $\\hat{\\phi}(\\bz)\\geq \\hat{\\phi}(\\u)$ $\\forall\\,\\,\\, \\u \\in \\ocalU$, using Cauchy-Schwarz on the left-hand side of the inequality,\n\\begin{align*}\n    \\norm{\\blambda^{(j)}_{\\A_C}}\\cdot\\norm{\\u} \\geq \\hat{\\phi}(\\bz) - \\hat{\\phi}({\\u)} .\n\\end{align*}\nDefining $\\Gamma = \\max_{i\\in S^{\\bz}_{=} }{\\norm{\\blambda^{(j)}_{\\A_C}}}$ yields\n$$\n    \\hat{\\phi}(\\bz) - \\hat{\\phi}(\\u)\\leq \\Gamma\\norm{\\u}\\,\\,\\forall\\,\\, \\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap\\ocalU\n$$\nwhere $\\delta = \\delta_{min}$ as defined in the proof of \\cref{lem:8}. The result follows.\n\\end{proof}","start":60797,"end":64038},{"raw_tex":"This completes the proof of \\Cref{thm:3}(a). In the following section we present some cases where an equivalence can be established between (MICP) and (MICP) with bounded integer variables. In particular, we highlight that  if the objective function $f$ satisfies $\\rec(f)\\cap\\rec(F_R)\\setminus\\{\\bz\\} = \\emptyset$, then there exists $M < \\infty$ such that the following equivalence holds.\n$$ \\min\\set{f(\\x)}{\\A\\x = \\b, \\x \\in X} = \\min\\set{f(\\x)}{\\A\\x = \\b, \\norm{\\x}_{\\infty} \\leq M, \\x \\in X}.$$ Observe that if $f$ is $\\mu$-strongly convex, then $\\rec(f) = \\emptyset$, as the level sets of $f$ are bounded. \n","start":64039,"end":64651},{"raw_tex":"\\begin{lemma}\\label{lem:15}\nLet  $\\delta > 0$, and let $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$. Define\n$$\n    S^\\u \\define  \\set{\\x \\in X \\cap \\Hu}{ f(\\x) \\leq \\zIP}.\n$$\nIf \\textup{\\MICP} is feasible ($F \\neq\\emptyset$) and bounded and the recession cone of $f$ and recession cone of $F_R$, the feasible set of continuous relaxation of \\textup{\\MICP}, have no common non-zero directions of recession, then  $S^\\u$ is bounded.\n\\end{lemma}","start":75106,"end":75549},{"raw_tex":"\\begin{proof}\nConsider $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$. Since $\\norm{\\u}_\\infty \\leq \\norm{\\u} < \\delta$ we have $\\norm{\\b - \\A\\x}_{\\infty} < \\delta$, $\\forall\\,\\x \\in X \\cap \\Hu$. Alternatively, $\\forall\\,\\x \\in X \\cap \\Hu$, $-\\1\\delta < {\\b} - \\A\\x < \\delta\\1$. Define, for $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$\n\\begin{align*}\n  S^\\u \\define  \\set{\\x \\in X \\cap \\Hu}{ f(\\x) \\leq \\zIP}\n\\end{align*}\nAdditionally, consider\n\\begin{align*}\n   S = \\set{\\x\\in X_R}{\\begin{Bmatrix}\n                                    -{\\1}\\delta \\leq {\\b} - \\A\\x \\leq {\\1}\\delta \\\\\n                                    f(\\x) \\leq z_{IP}\n                        \\end{Bmatrix}}\n\\end{align*}\n%Here ${\\u}$ is taken over all feasible polyhedra i.e. this set $\\{\\A\\x = {\\b} + {\\u},\\,\\, {\\E}\\x\\leq {\\f}\\,\\, ,\\x\\in \\mathbb{Z}^{n_1}\\times \\mathbb{R}^{n_2}\\}$ is feasible.\nIndeed, $S^\\u \\subseteq S$. Furthermore, $\\rec(S) = \\rec(F_R) \\cap \\rec(f)$, and since $\\rec(F_R) \\cap \\rec(f) = \\{\\bz\\}$, it follows that $S$ is compact and consequently $S^\\u$ is bounded. In particular, $\\norm{\\x}_\\infty \\leq M < \\infty$, $\\forall$ $\\x \\in S^\\u$. %since MICP is feasible and optimally bounded S'' is non-empty.\\\\\n%This implies that one can prescribe a uniform bound on variables of the problem and hence $\\norm{\\x_I\\norm{\\leq M$.\n\\end{proof}","start":75550,"end":76873},{"raw_tex":"\\begin{corollary}\\label{cor:16}\nIf \\textup{\\MICP} is feasible ($F \\neq\\emptyset$) and bounded and the recession cone of $f$ and recession cone of $F_R$, the feasible set of continuous relaxation of \\textup{\\MICP}, have no common non-zero directions of recession, then for ${\\u} \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$, $\\phi(\\u) = \\hat{\\phi}(\\u)$ i.e.\n\\begin{optprog*}\n    {$\\phi(\\u) = $ minimize}  & \\objective{\\hspace{1.3cm}f({\\x_I},{\\x_C})}\\\\\n    subject to & \\hspace{0cm} \\A_I\\,\\x_I + \\A_C\\x_C = \\b + \\u\\\\\n    & \\E_I\\x_I + \\E_C\\x_C \\leq \\f \\hspace{7mm}\\\\\n    & \\norm{\\x_I}_{\\infty}\\leq M \\hspace{0.5cm}\\\\\n    % & \\x_I\\in \\Z_{+}^{n_1}\\hspace{0.4cm}\\\\ \n     & (\\x_I, \\x_C) \\in \\Z^{n_1}\\times\\R^{n_2}\\hspace{0.3cm}\n\\end{optprog*}\nFurthermore, there exists a $\\delta > 0$ and $0 < \\Gamma < \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $\\phi(\\bz)\\leq \\phi(\\u) + \\Gamma\\norm{\\u}$. Additionally, if $f$ is $\\mu$-strongly convex and $L$ smooth, then $\\Gamma = \\mathcal{O}\\left(\\frac{L\\beta\\gamma}{\\mu}\\right)$ where $\\displaystyle \\beta \\define \\max_{B \\in \\mathcal{B}}\\norm{B^{-1}}_F$, $\\mathcal{B}$ being the set of all possible invertible submatrices of $[\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^T_C]$ and $\\gamma$ depends on $f(\\bz)$, $\\norm{\\nabla{f(\\bz)}}$ and $f(\\bar{\\x})$ for any $\\bar{\\x} \\in F$.\n\\end{corollary}","start":76941,"end":78288},{"raw_tex":"\\begin{proof}\n\\Cref{lem:15}, along with \\cref{lem:8,lem:9} readily implies that $\\phi(\\u) = \\hat{\\phi}(\\u)$ and there exists a $\\delta > 0$ and $0 < \\Gamma < \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $\\phi(\\bz)\\leq \\phi(\\u) + \\Gamma\\norm{\\u}$.\n\nTo see the explicit bound on $\\Gamma$, recall the stationarity condition from set of equations \\cref{eq:32} at $\\u = \\bz$.\n\\begin{align}\\label{eq:stationarity}\n\\begin{split}\n    \\nabla_{C}{f(\\x_I^{(i)},\\x_C}) & = \\A^T_C(\\lambda^{(i)+}_{\\A_C} - \\lambda^{(i)-}_{\\A_C}) - {\\E}_C^T\\lambda_{{\\E}_C}\\\\\n    \\lambda_{\\E_C}^{\\top}(\\E\\x -\\f) &= 0\\\\\n    \\lambda^{+}_{\\A_C},\\lambda^{-}_{\\A_C} & \\geq \\bz\\\\\n     \\lambda_{{\\E}_C} &\\geq \\bz\n\\end{split}\n\\end{align}\n\nFor $(\\x_C,\\x^{(i)}_I)$ consider the complementary slackness conditions, that is if for some $j\\in [m]$, $\\E_{j}\\x < \\f_j$ where $\\E_{j}$ is a row of the matrix $\\E$ then the corresponding Lagrangian multiplier $\\lambda_{\\E_j} = 0$. Partition the set $[m]$ into two sets $J = \\{J_<,J_=\\}$, where $J_<=\\set{j\\in [m]}{\\E_j \\x <\\f_j}$ and $J_= =\\set{j\\in [m]}{\\E_j \\x = \\f_j}$.\\\\\n\nDefine $\\A_{aug} = [\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^{\\top}_C]$ and $\\A_{aug}^J$ as the sub-matrix which has all the columns of $\\A_{aug}$ except the columns $2n_2+J_<$.\n % $\\lambda^{(i)+}_{\\A_C},\\lambda^{(i)-}_{\\A_C}$. \n One possible solution to the system \\cref{eq:stationarity} is,\n \\begin{align}\\label{sol:lambda}\n    \\begin{pmatrix}\n     \\lambda^{+}_{\\A_C}\\\\\n      \\lambda^{-}_{\\A_C}\\\\\n      \\lambda_{{\\E}^{J_{=}}_C}\n    \\end{pmatrix} = B^{-1} \\nabla_C{f(\\x^{(i)}_I,\\x_C)}\n\\end{align}\nwhere $B$ is a basis matrix of $\\A_{aug}^J$.\n\nFurthermore, let $\\lambda_{\\A_C} = \\lambda^{+}_{\\A_C} - \\lambda^{-}_{\\A_C}$, this implies that \n\\begin{align}\n    \\norm{\\lambda_{\\A_C}} \\leq \\norm{\\lambda^{+}_{\\A_C}} + \\norm{\\lambda^{-}_{\\A_C}} \\label{ineq:lambda}    \n\\end{align}\nLet $\\x^*$ be an optimal solution to \\MICP, i.e. $f(\\x^*) = \\zIP$. Since $f$ is $\\mu$-strongly convex, we have\n\n\\begin{align*}\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 + \\nabla{f({\\bz)}}^\\top\\x^*  + f(\\bz) & \\leq z_{IP}\\\\\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 + \\nabla{f({\\bz)}}^\\top\\x^* - z_{IP} + f(\\bz)& \\leq 0\\\\\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 - \\norm{\\nabla{f({\\bz)}}}\\,\\norm{\\x^*} - \\left(z_{IP} - f(\\bz)\\right) & \\leq 0\n\\end{align*}\nIt follows that,\n$$\n    \\norm{\\x^*}\\leq \\frac{\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz})}^2 +2\\,\\mu\\cdot \\left(z_{IP} - f(\\bz)\\right)}}{\\mu}\n$$\nSince $f$ is L-smooth,\n\\begin{align*}\n    \\norm{\\nabla{f(\\x^*)} - \\nabla{f(\\bz)}} &\\leq L\\norm{\\x^*}\\\\\n    \\norm{\\nabla{f(\\x^*)}} &\\leq L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\n\\end{align*}\nFrom \\cref{sol:lambda} and \\cref{ineq:lambda} we obtain\n\\begin{align*}\n    \\norm{\\lambda_{\\A_C}} & \\leq 2\\norm{B^{-1}}_F \\norm{\\nabla{f(\\x^*)}}\\\\\n    & \\leq 2\\norm{B^{-1}}_F\\left(L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\\right)\\\\\n    & \\leq 2\\beta\\left(L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\\right)\n\\end{align*}\nwhere $\\beta \\define \\max_{B \\in \\mathcal{B}}\\norm{B^{-1}}_F$ and $\\mathcal{B}$ is the set of all possible invertible submatrices of $[\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^T_C]$.\n\n\\begin{align}\n   \\norm{\\lambda_{\\A_C}} & \\leq \\dfrac{2\\beta\\left(L\\left(\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz})}^2 +2\\mu\\,\\left(z_{IP} - f(\\bz)\\right)}\\right) + \\mu\\norm{\\nabla{f(\\bz)}}\\right)}{\\mu}\\nonumber\\\\\n    & \\leq  \\dfrac{2\\beta L}{\\mu}\\left(2\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz)}}^2 + 2\\mu\\,\\left(f(\\bar{\\x}) - f(\\bz)\\right)}\\right)\\label{ineq:thm3}\n\\end{align}\nwhere the second inequality follows as $\\zIP \\leq f(\\bar{\\x}),$ $\\forall\\,\\bar{\\x} \\in F$ and $L > \\mu$. Combining \\cref{ineq:thm3} with the construction of $\\Gamma$ in \\cref{lem:9}, yields the result.\n\\end{proof}","start":78289,"end":82023},{"raw_tex":"\\cref{thm:3} follows.\n","start":82024,"end":82046}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"作者收束有界整数部分","source_units":[{"raw_tex":"This completes the proof of \\Cref{thm:3}(a). In the following section we present some cases where an equivalence can be established between (MICP) and (MICP) with bounded integer variables. In particular, we highlight that  if the objective function $f$ satisfies $\\rec(f)\\cap\\rec(F_R)\\setminus\\{\\bz\\} = \\emptyset$, then there exists $M < \\infty$ such that the following equivalence holds.\n","start":64039,"end":64429}]},{"id":"S054","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"隐含有界性的过渡","source_units":[{"raw_tex":"This completes the proof of \\Cref{thm:3}(a). In the following section we present some cases where an equivalence can be established between (MICP) and (MICP) with bounded integer variables. In particular, we highlight that  if the objective function $f$ satisfies $\\rec(f)\\cap\\rec(F_R)\\setminus\\{\\bz\\} = \\emptyset$, then there exists $M < \\infty$ such that the following equivalence holds.\n$$ \\min\\set{f(\\x)}{\\A\\x = \\b, \\x \\in X} = \\min\\set{f(\\x)}{\\A\\x = \\b, \\norm{\\x}_{\\infty} \\leq M, \\x \\in X}.$$ Observe that if $f$ is $\\mu$-strongly convex, then $\\rec(f) = \\emptyset$, as the level sets of $f$ are bounded. \n","start":64039,"end":64651}]},{"id":"S055","semantic_type":"lemma","kind":"lemma","labels":["lem:15"],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"扰动低目标集合有界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lem:15}\nLet  $\\delta > 0$, and let $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$. Define\n$$\n    S^\\u \\define  \\set{\\x \\in X \\cap \\Hu}{ f(\\x) \\leq \\zIP}.\n$$\nIf \\textup{\\MICP} is feasible ($F \\neq\\emptyset$) and bounded and the recession cone of $f$ and recession cone of $F_R$, the feasible set of continuous relaxation of \\textup{\\MICP}, have no common non-zero directions of recession, then  $S^\\u$ is bounded.\n\\end{lemma}","start":75106,"end":75549}]},{"id":"S056","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"低目标有界性作者证明","source_units":[{"raw_tex":"\\begin{proof}\nConsider $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$. Since $\\norm{\\u}_\\infty \\leq \\norm{\\u} < \\delta$ we have $\\norm{\\b - \\A\\x}_{\\infty} < \\delta$, $\\forall\\,\\x \\in X \\cap \\Hu$. Alternatively, $\\forall\\,\\x \\in X \\cap \\Hu$, $-\\1\\delta < {\\b} - \\A\\x < \\delta\\1$. Define, for $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$\n\\begin{align*}\n  S^\\u \\define  \\set{\\x \\in X \\cap \\Hu}{ f(\\x) \\leq \\zIP}\n\\end{align*}\nAdditionally, consider\n\\begin{align*}\n   S = \\set{\\x\\in X_R}{\\begin{Bmatrix}\n                                    -{\\1}\\delta \\leq {\\b} - \\A\\x \\leq {\\1}\\delta \\\\\n                                    f(\\x) \\leq z_{IP}\n                        \\end{Bmatrix}}\n\\end{align*}\n%Here ${\\u}$ is taken over all feasible polyhedra i.e. this set $\\{\\A\\x = {\\b} + {\\u},\\,\\, {\\E}\\x\\leq {\\f}\\,\\, ,\\x\\in \\mathbb{Z}^{n_1}\\times \\mathbb{R}^{n_2}\\}$ is feasible.\nIndeed, $S^\\u \\subseteq S$. Furthermore, $\\rec(S) = \\rec(F_R) \\cap \\rec(f)$, and since $\\rec(F_R) \\cap \\rec(f) = \\{\\bz\\}$, it follows that $S$ is compact and consequently $S^\\u$ is bounded. In particular, $\\norm{\\x}_\\infty \\leq M < \\infty$, $\\forall$ $\\x \\in S^\\u$. %since MICP is feasible and optimally bounded S'' is non-empty.\\\\\n%This implies that one can prescribe a uniform bound on variables of the problem and hence $\\norm{\\x_I\\norm{\\leq M$.\n\\end{proof}","start":75550,"end":76873}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"共同残差盒包络","source_units":[{"raw_tex":"Consider $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$. Since $\\norm{\\u}_\\infty \\leq \\norm{\\u} < \\delta$ we have $\\norm{\\b - \\A\\x}_{\\infty} < \\delta$, $\\forall\\,\\x \\in X \\cap \\Hu$. Alternatively, $\\forall\\,\\x \\in X \\cap \\Hu$, $-\\1\\delta < {\\b} - \\A\\x < \\delta\\1$. Define, for $\\u \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$\n\\begin{align*}\n  S^\\u \\define  \\set{\\x \\in X \\cap \\Hu}{ f(\\x) \\leq \\zIP}\n\\end{align*}\nAdditionally, consider\n\\begin{align*}\n   S = \\set{\\x\\in X_R}{\\begin{Bmatrix}\n                                    -{\\1}\\delta \\leq {\\b} - \\A\\x \\leq {\\1}\\delta \\\\\n                                    f(\\x) \\leq z_{IP}\n                        \\end{Bmatrix}}\n\\end{align*}\n","start":75564,"end":76239}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"衰退锥零交推出统一界","source_units":[{"raw_tex":"Indeed, $S^\\u \\subseteq S$. Furthermore, $\\rec(S) = \\rec(F_R) \\cap \\rec(f)$, and since $\\rec(F_R) \\cap \\rec(f) = \\{\\bz\\}$, it follows that $S$ is compact and consequently $S^\\u$ is bounded. In particular, $\\norm{\\x}_\\infty \\leq M < \\infty$, $\\forall$ $\\x \\in S^\\u$. %since MICP is feasible and optimally bounded S'' is non-empty.\\\\\n","start":76413,"end":76745}]},{"id":"S059","semantic_type":"corollary","kind":"corollary","labels":["cor:16"],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"局部有界截断值相同","source_units":[{"raw_tex":"\\begin{corollary}\\label{cor:16}\nIf \\textup{\\MICP} is feasible ($F \\neq\\emptyset$) and bounded and the recession cone of $f$ and recession cone of $F_R$, the feasible set of continuous relaxation of \\textup{\\MICP}, have no common non-zero directions of recession, then for ${\\u} \\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\ocalU$, $\\phi(\\u) = \\hat{\\phi}(\\u)$ i.e.\n\\begin{optprog*}\n    {$\\phi(\\u) = $ minimize}  & \\objective{\\hspace{1.3cm}f({\\x_I},{\\x_C})}\\\\\n    subject to & \\hspace{0cm} \\A_I\\,\\x_I + \\A_C\\x_C = \\b + \\u\\\\\n    & \\E_I\\x_I + \\E_C\\x_C \\leq \\f \\hspace{7mm}\\\\\n    & \\norm{\\x_I}_{\\infty}\\leq M \\hspace{0.5cm}\\\\\n    % & \\x_I\\in \\Z_{+}^{n_1}\\hspace{0.4cm}\\\\ \n     & (\\x_I, \\x_C) \\in \\Z^{n_1}\\times\\R^{n_2}\\hspace{0.3cm}\n\\end{optprog*}\n","start":76941,"end":77674}]},{"id":"S060","semantic_type":"corollary","kind":"corollary","labels":["cor:16"],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"原值函数局部线性界","source_units":[{"raw_tex":"Furthermore, there exists a $\\delta > 0$ and $0 < \\Gamma < \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $\\phi(\\bz)\\leq \\phi(\\u) + \\Gamma\\norm{\\u}$. Additionally, if $f$ is $\\mu$-strongly convex and $L$ smooth, then $\\Gamma = \\mathcal{O}\\left(\\frac{L\\beta\\gamma}{\\mu}\\right)$ where $\\displaystyle \\beta \\define \\max_{B \\in \\mathcal{B}}\\norm{B^{-1}}_F$, $\\mathcal{B}$ being the set of all possible invertible submatrices of $[\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^T_C]$ and $\\gamma$ depends on $f(\\bz)$, $\\norm{\\nabla{f(\\bz)}}$ and $f(\\bar{\\x})$ for any $\\bar{\\x} \\in F$.\n\\end{corollary}","start":77674,"end":78288}]},{"id":"S061","semantic_type":"corollary","kind":"corollary","labels":["cor:16"],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"局部常数的定量阶数","source_units":[{"raw_tex":"Furthermore, there exists a $\\delta > 0$ and $0 < \\Gamma < \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $\\phi(\\bz)\\leq \\phi(\\u) + \\Gamma\\norm{\\u}$. Additionally, if $f$ is $\\mu$-strongly convex and $L$ smooth, then $\\Gamma = \\mathcal{O}\\left(\\frac{L\\beta\\gamma}{\\mu}\\right)$ where $\\displaystyle \\beta \\define \\max_{B \\in \\mathcal{B}}\\norm{B^{-1}}_F$, $\\mathcal{B}$ being the set of all possible invertible submatrices of $[\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^T_C]$ and $\\gamma$ depends on $f(\\bz)$, $\\norm{\\nabla{f(\\bz)}}$ and $f(\\bar{\\x})$ for any $\\bar{\\x} \\in F$.\n\\end{corollary}","start":77674,"end":78288}]},{"id":"S062","semantic_type":"proof","kind":"proof","labels":["eq:stationarity","ineq:lambda","ineq:thm3","sol:lambda"],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"推论十六作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\\Cref{lem:15}, along with \\cref{lem:8,lem:9} readily implies that $\\phi(\\u) = \\hat{\\phi}(\\u)$ and there exists a $\\delta > 0$ and $0 < \\Gamma < \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $\\phi(\\bz)\\leq \\phi(\\u) + \\Gamma\\norm{\\u}$.\n\nTo see the explicit bound on $\\Gamma$, recall the stationarity condition from set of equations \\cref{eq:32} at $\\u = \\bz$.\n\\begin{align}\\label{eq:stationarity}\n\\begin{split}\n    \\nabla_{C}{f(\\x_I^{(i)},\\x_C}) & = \\A^T_C(\\lambda^{(i)+}_{\\A_C} - \\lambda^{(i)-}_{\\A_C}) - {\\E}_C^T\\lambda_{{\\E}_C}\\\\\n    \\lambda_{\\E_C}^{\\top}(\\E\\x -\\f) &= 0\\\\\n    \\lambda^{+}_{\\A_C},\\lambda^{-}_{\\A_C} & \\geq \\bz\\\\\n     \\lambda_{{\\E}_C} &\\geq \\bz\n\\end{split}\n\\end{align}\n\nFor $(\\x_C,\\x^{(i)}_I)$ consider the complementary slackness conditions, that is if for some $j\\in [m]$, $\\E_{j}\\x < \\f_j$ where $\\E_{j}$ is a row of the matrix $\\E$ then the corresponding Lagrangian multiplier $\\lambda_{\\E_j} = 0$. Partition the set $[m]$ into two sets $J = \\{J_<,J_=\\}$, where $J_<=\\set{j\\in [m]}{\\E_j \\x <\\f_j}$ and $J_= =\\set{j\\in [m]}{\\E_j \\x = \\f_j}$.\\\\\n\nDefine $\\A_{aug} = [\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^{\\top}_C]$ and $\\A_{aug}^J$ as the sub-matrix which has all the columns of $\\A_{aug}$ except the columns $2n_2+J_<$.\n % $\\lambda^{(i)+}_{\\A_C},\\lambda^{(i)-}_{\\A_C}$. \n One possible solution to the system \\cref{eq:stationarity} is,\n \\begin{align}\\label{sol:lambda}\n    \\begin{pmatrix}\n     \\lambda^{+}_{\\A_C}\\\\\n      \\lambda^{-}_{\\A_C}\\\\\n      \\lambda_{{\\E}^{J_{=}}_C}\n    \\end{pmatrix} = B^{-1} \\nabla_C{f(\\x^{(i)}_I,\\x_C)}\n\\end{align}\nwhere $B$ is a basis matrix of $\\A_{aug}^J$.\n\nFurthermore, let $\\lambda_{\\A_C} = \\lambda^{+}_{\\A_C} - \\lambda^{-}_{\\A_C}$, this implies that \n\\begin{align}\n    \\norm{\\lambda_{\\A_C}} \\leq \\norm{\\lambda^{+}_{\\A_C}} + \\norm{\\lambda^{-}_{\\A_C}} \\label{ineq:lambda}    \n\\end{align}\nLet $\\x^*$ be an optimal solution to \\MICP, i.e. $f(\\x^*) = \\zIP$. Since $f$ is $\\mu$-strongly convex, we have\n\n\\begin{align*}\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 + \\nabla{f({\\bz)}}^\\top\\x^*  + f(\\bz) & \\leq z_{IP}\\\\\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 + \\nabla{f({\\bz)}}^\\top\\x^* - z_{IP} + f(\\bz)& \\leq 0\\\\\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 - \\norm{\\nabla{f({\\bz)}}}\\,\\norm{\\x^*} - \\left(z_{IP} - f(\\bz)\\right) & \\leq 0\n\\end{align*}\nIt follows that,\n$$\n    \\norm{\\x^*}\\leq \\frac{\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz})}^2 +2\\,\\mu\\cdot \\left(z_{IP} - f(\\bz)\\right)}}{\\mu}\n$$\nSince $f$ is L-smooth,\n\\begin{align*}\n    \\norm{\\nabla{f(\\x^*)} - \\nabla{f(\\bz)}} &\\leq L\\norm{\\x^*}\\\\\n    \\norm{\\nabla{f(\\x^*)}} &\\leq L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\n\\end{align*}\nFrom \\cref{sol:lambda} and \\cref{ineq:lambda} we obtain\n\\begin{align*}\n    \\norm{\\lambda_{\\A_C}} & \\leq 2\\norm{B^{-1}}_F \\norm{\\nabla{f(\\x^*)}}\\\\\n    & \\leq 2\\norm{B^{-1}}_F\\left(L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\\right)\\\\\n    & \\leq 2\\beta\\left(L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\\right)\n\\end{align*}\nwhere $\\beta \\define \\max_{B \\in \\mathcal{B}}\\norm{B^{-1}}_F$ and $\\mathcal{B}$ is the set of all possible invertible submatrices of $[\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^T_C]$.\n\n\\begin{align}\n   \\norm{\\lambda_{\\A_C}} & \\leq \\dfrac{2\\beta\\left(L\\left(\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz})}^2 +2\\mu\\,\\left(z_{IP} - f(\\bz)\\right)}\\right) + \\mu\\norm{\\nabla{f(\\bz)}}\\right)}{\\mu}\\nonumber\\\\\n    & \\leq  \\dfrac{2\\beta L}{\\mu}\\left(2\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz)}}^2 + 2\\mu\\,\\left(f(\\bar{\\x}) - f(\\bz)\\right)}\\right)\\label{ineq:thm3}\n\\end{align}\nwhere the second inequality follows as $\\zIP \\leq f(\\bar{\\x}),$ $\\forall\\,\\bar{\\x} \\in F$ and $L > \\mu$. Combining \\cref{ineq:thm3} with the construction of $\\Gamma$ in \\cref{lem:9}, yields the result.\n\\end{proof}","start":78289,"end":82023}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"统一界接回局部线性界","source_units":[{"raw_tex":"\\Cref{lem:15}, along with \\cref{lem:8,lem:9} readily implies that $\\phi(\\u) = \\hat{\\phi}(\\u)$ and there exists a $\\delta > 0$ and $0 < \\Gamma < \\infty$ such that for every $ \\u\\in \\mathcal{N}_{\\delta}(\\bz)\\cap \\calU$, $\\phi(\\bz)\\leq \\phi(\\u) + \\Gamma\\norm{\\u}$.\n","start":78303,"end":78565}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":["eq:stationarity"],"member_ids":["S064"],"declared_external":false,"source_only":false,"title":"分裂乘子并删除松约束","source_units":[{"raw_tex":"To see the explicit bound on $\\Gamma$, recall the stationarity condition from set of equations \\cref{eq:32} at $\\u = \\bz$.\n\\begin{align}\\label{eq:stationarity}\n\\begin{split}\n    \\nabla_{C}{f(\\x_I^{(i)},\\x_C}) & = \\A^T_C(\\lambda^{(i)+}_{\\A_C} - \\lambda^{(i)-}_{\\A_C}) - {\\E}_C^T\\lambda_{{\\E}_C}\\\\\n    \\lambda_{\\E_C}^{\\top}(\\E\\x -\\f) &= 0\\\\\n    \\lambda^{+}_{\\A_C},\\lambda^{-}_{\\A_C} & \\geq \\bz\\\\\n     \\lambda_{{\\E}_C} &\\geq \\bz\n\\end{split}\n\\end{align}\n\nFor $(\\x_C,\\x^{(i)}_I)$ consider the complementary slackness conditions, that is if for some $j\\in [m]$, $\\E_{j}\\x < \\f_j$ where $\\E_{j}$ is a row of the matrix $\\E$ then the corresponding Lagrangian multiplier $\\lambda_{\\E_j} = 0$. Partition the set $[m]$ into two sets $J = \\{J_<,J_=\\}$, where $J_<=\\set{j\\in [m]}{\\E_j \\x <\\f_j}$ and $J_= =\\set{j\\in [m]}{\\E_j \\x = \\f_j}$.\\\\\n\nDefine $\\A_{aug} = [\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^{\\top}_C]$ and $\\A_{aug}^J$ as the sub-matrix which has all the columns of $\\A_{aug}$ except the columns $2n_2+J_<$.\n","start":78566,"end":79570}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":["sol:lambda"],"member_ids":["S065"],"declared_external":false,"source_only":false,"title":"乘子的基逆表示","source_units":[{"raw_tex":" One possible solution to the system \\cref{eq:stationarity} is,\n \\begin{align}\\label{sol:lambda}\n    \\begin{pmatrix}\n     \\lambda^{+}_{\\A_C}\\\\\n      \\lambda^{-}_{\\A_C}\\\\\n      \\lambda_{{\\E}^{J_{=}}_C}\n    \\end{pmatrix} = B^{-1} \\nabla_C{f(\\x^{(i)}_I,\\x_C)}\n\\end{align}\nwhere $B$ is a basis matrix of $\\A_{aug}^J$.\n","start":79621,"end":79935}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":["ineq:lambda"],"member_ids":["S066"],"declared_external":false,"source_only":false,"title":"正负乘子范数比较","source_units":[{"raw_tex":"Furthermore, let $\\lambda_{\\A_C} = \\lambda^{+}_{\\A_C} - \\lambda^{-}_{\\A_C}$, this implies that \n\\begin{align}\n    \\norm{\\lambda_{\\A_C}} \\leq \\norm{\\lambda^{+}_{\\A_C}} + \\norm{\\lambda^{-}_{\\A_C}} \\label{ineq:lambda}    \n\\end{align}\n","start":79936,"end":80167}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S067"],"declared_external":false,"source_only":false,"title":"强凸性控制最优点半径","source_units":[{"raw_tex":"Let $\\x^*$ be an optimal solution to \\MICP, i.e. $f(\\x^*) = \\zIP$. Since $f$ is $\\mu$-strongly convex, we have\n\n\\begin{align*}\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 + \\nabla{f({\\bz)}}^\\top\\x^*  + f(\\bz) & \\leq z_{IP}\\\\\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 + \\nabla{f({\\bz)}}^\\top\\x^* - z_{IP} + f(\\bz)& \\leq 0\\\\\n    \\frac{\\mu}{2} \\norm{\\x^*}^2 - \\norm{\\nabla{f({\\bz)}}}\\,\\norm{\\x^*} - \\left(z_{IP} - f(\\bz)\\right) & \\leq 0\n\\end{align*}\nIt follows that,\n$$\n    \\norm{\\x^*}\\leq \\frac{\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz})}^2 +2\\,\\mu\\cdot \\left(z_{IP} - f(\\bz)\\right)}}{\\mu}\n$$\n","start":80167,"end":80744}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S068"],"declared_external":false,"source_only":false,"title":"光滑性控制最优点梯度","source_units":[{"raw_tex":"Since $f$ is L-smooth,\n\\begin{align*}\n    \\norm{\\nabla{f(\\x^*)} - \\nabla{f(\\bz)}} &\\leq L\\norm{\\x^*}\\\\\n    \\norm{\\nabla{f(\\x^*)}} &\\leq L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\n\\end{align*}\n","start":80744,"end":80930}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","labels":["ineq:thm3"],"member_ids":["S069"],"declared_external":false,"source_only":false,"title":"统一乘子界与Γ阶数","source_units":[{"raw_tex":"From \\cref{sol:lambda} and \\cref{ineq:lambda} we obtain\n\\begin{align*}\n    \\norm{\\lambda_{\\A_C}} & \\leq 2\\norm{B^{-1}}_F \\norm{\\nabla{f(\\x^*)}}\\\\\n    & \\leq 2\\norm{B^{-1}}_F\\left(L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\\right)\\\\\n    & \\leq 2\\beta\\left(L\\norm{\\x^*} + \\norm{\\nabla{f(\\bz)}}\\right)\n\\end{align*}\nwhere $\\beta \\define \\max_{B \\in \\mathcal{B}}\\norm{B^{-1}}_F$ and $\\mathcal{B}$ is the set of all possible invertible submatrices of $[\\A^T_C \\,\\, -\\A^T_C \\hspace{2mm}-{\\E}^T_C]$.\n\n\\begin{align}\n   \\norm{\\lambda_{\\A_C}} & \\leq \\dfrac{2\\beta\\left(L\\left(\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz})}^2 +2\\mu\\,\\left(z_{IP} - f(\\bz)\\right)}\\right) + \\mu\\norm{\\nabla{f(\\bz)}}\\right)}{\\mu}\\nonumber\\\\\n    & \\leq  \\dfrac{2\\beta L}{\\mu}\\left(2\\norm{\\nabla{f(\\bz)}} + \\sqrt{\\norm{\\nabla{f(\\bz)}}^2 + 2\\mu\\,\\left(f(\\bar{\\x}) - f(\\bz)\\right)}\\right)\\label{ineq:thm3}\n\\end{align}\nwhere the second inequality follows as $\\zIP \\leq f(\\bar{\\x}),$ $\\forall\\,\\bar{\\x} \\in F$ and $L > \\mu$. Combining \\cref{ineq:thm3} with the construction of $\\Gamma$ in \\cref{lem:9}, yields the result.\n","start":80930,"end":82012}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S070"],"declared_external":false,"source_only":false,"title":"作者最终收束定理","source_units":[{"raw_tex":"\\cref{thm:3} follows.\n","start":82024,"end":82046}]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S071"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第一项：Strong duality and KKT multipliers for attained differentiable convex optimization over affine constraints; Boyd and Vandenberghe, Section 5.2.3。原文第266行显式引用。"],"source_only":false,"title":"仿射凸优化强对偶与KKT","source_units":[]},{"id":"S072","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S072"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第二项：Value-function lower semicontinuity through strong duality; Bertsekas, Proposition 6.5.2, as invoked in B0102。原文第740行显式引用。"],"source_only":false,"title":"强对偶与值函数下半连续","source_units":[]},{"id":"S073","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S073"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第三项中的 Closedness of finitely generated cones。第756—757行直接使用，未另给文献定位。"],"source_only":false,"title":"有限生成锥闭性","source_units":[]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S074"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第四项：Recession-cone intersection rules and the boundedness criterion for nonempty closed convex sets。第1012行直接使用。"],"source_only":false,"title":"衰退锥交与有界性判据","source_units":[]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S075"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第五项中的 Convex first-order inequalities, Cauchy–Schwarz and matrix norm inequalities；三角不等式为 implicit standard fact made explicit by annotator。"],"source_only":false,"title":"凸一阶与范数估计","source_units":[]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S076"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第五项中的 the standard strong-convexity quadratic lower bound；第1062行实际使用。"],"source_only":false,"title":"标准强凸二次下界","source_units":[]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S077"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第三项中的 basic feasible solutions of nonnegative linear systems；第1045—1053行隐含使用。"],"source_only":false,"title":"非负线性系统基本解","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076","S077":"S077"}}}}
{"record_id":"2208.13933v2","paper":{"title":"Using Taylor-Approximated Gradients to Improve the Frank-Wolfe Method for Empirical Risk Minimization","authors":["Xiong, Zikai","Freund, Robert M."],"arxiv_id":"2208.13933","version":"v2","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/2208.13933v2","pdf_url":"https://arxiv.org/pdf/2208.13933v2","doi":"10.1137/22M1519286"},"context":{"raw_tex":"\tOur problem of interest is the following Empirical Risk Minimization (ERM) problem (also called the \\textit{finite-sum} problem) with constraints: \n\t\\begin{equation}\\label{pro: ERM}\n\t\t\\text{ERM:} \\ \\ \\ \\ \\    \\text{minimize}_{x \\in \\calC} \\quad F(x):= \\tfrac{1}{n} \\sum_{i=1}^{n} f_i(x) \\ , \n\t\\end{equation}\n\twhere $f_i(\\cdot): \\mathbb{R}^p \\to \\mathbb{R}$ is the loss function of observation $i$ for $i=1,\\dots, n$, and $x$ is comprised of the (model) coefficients/parameters which are to be optimized over the feasibility set $\\calC \\subset \\mathbb{R}^p$. Problem \\cref{pro: ERM} is a fundamental problem in statistical and machine learning, as it is the underlying optimization problem on which models are ``trained'' or ``learned'', see  \\cite{vapnik1991principles,hastie2009elements}. The number of observations $n$ can be significantly larger than the dimension $p$, and our interest here will be in the regime where $n \\gg p$.\n\tWe make the following assumptions for $\\erm$ problems throughout this paper:\n\t\\begin{assumption}\\label{assump: ERM}\n\t\tThe following conditions hold for the $\\erm$ problem \\cref{pro: ERM}:\n\t\t\\begin{enumerate}\n\t\t\t\\item[(i)] the feasibility set $\\mathcal{C}$ is a compact convex set,\n\t\t\t\\item[(ii)] for any $g \\in \\mathbb{R}^p$, the linear minimization problem $\\arg\\min_{x\\in\\calC} g^\\top x$\n\t\t\tcan be easily solved, \n\t\t\t\\item[(iii)] for all $i\\in[n]$, the multivariate loss function $f_i(\\cdot)$ is twice-differentiable and its gradient $\\nabla f_i(\\cdot)$ is $L$-Lipschitz continuous on $\\calC$, namely, \n\t\t\t\\begin{equation}\\label{eq: ERM ass lip 1}\n\t\t\t\t\\| \\nabla f_i(u) - \\nabla f_i(v) \\|_* \\le L \\|u-v\\| ~~\\text{ for any $u,v\\in \\calC$}\n\t\t\t\\end{equation}\n\t\t\twhere $\\|\\cdot\\|_*$ is the dual norm of the norm on the space of variables, and \n\t\t\t\\item[(iv)] for all $i\\in[n]$, the Hessian $\\nabla^2 f_i(\\cdot)$ is $\\hat{L}$-Lipschitz continuous on $\\calC$, namely\n\t\t\t\\begin{equation}\\label{eq: ERM ass lip 2}\n\t\t\t\t\\| \\nabla^2 f_i(u) - \\nabla^2 f_i(v) \\| \\le \\hat{L} \\|u-v\\|~~\\text{ for any $u,v\\in \\calC$}\n\t\t\t\\end{equation}\n\t\t\twhere the norm of the Hessian matrices is the operator norm induced by the norm on the space of variables.\n\t\t\\end{enumerate}\n\t\\end{assumption}\n\n\n\t\\subsection{Notation}\n\t\n\tLet $\\mathbb{N}$ denote the natural numbers.  For any $n \\in \\mathbb{N}$, we use $[n]$ to denote the set $\\{1,2,\\dots,n\\}$.\n\tFor any real number $v$, we use $\\lfloor v \\rfloor$ to denote the largest integer number less than or equal to $v$.\n\tDepending on the context, we will use either $\\langle a,b\\rangle$ or $a^\\top b$ to denote the inner product between $a$ and $b$ in Euclidean space. For a vector $x$ in $\\mathbb{R}^p$, the norm $\\|x\\|_q$ is given by $(\\sum_{j=1}^{p}x_j^q)^{1/q}$ for $q \\in [1,\\infty)$ and $\\|x\\|_\\infty := \\max_{j} |x_j|$. The dual of norm $\\|\\cdot\\|$ is defined as $\\|z\\|_* := \\sup\\{z^\\top x : \\|x\\| \\le 1\\}$.  For a matrix $A \\in \\mathbb{R}^{p \\times p}$, the matrix operator norm $\\|A\\|$ induced by the norm $\\|\\cdot\\|$ is defined as $\\sup_{x: \\|x\\| \\le 1} \\|Ax\\|_*$. For any symmetric positive definite matrix $A\\in \\mathbb{S}^{p\\times p}$, we define its induced norm on $\\mathbb{R}^p$ by $\\|z\\|_A := \\sqrt{z^\\top A z}$.  We define $\\sum_{i=j}^k \\alpha_i := 0$ whenever $j > k$.  We denote the uniform distribution on the set $S$ by $\\calU(S)$, and we denote the Bernoulli distribution with probability $p$ by $\\mathrm{Ber}(p)$.\n\n\n\tIn this section we present a new type of Frank-Wolfe algorithm whose newness derives from the way in which approximate gradients are computed.  We call the algorithm TUFW for ``Taylor-point Updating Frank-Wolfe''. Algorithm TUFW is similar to the standard Frank-Wolfe method except that it computes an inexpensive estimate $g^k$ of the gradient $\\nabla F(x^k)$ at iteration $k$.  And it differs from other Frank-Wolfe methods in the way it computes the estimate $g^k$ in that (i) TUFW explicitly uses second-derivative (Hessian) information of the loss functions $f_i$, and (ii) it can use very different batch-size rules -- both stochastic and deterministic. TUFW is described in \\cref{alg: TUFW in general finite sum}.  We now explain the steps of the method.\n\t\n\t\n\t\\begin{algorithm}\n\t\t\\caption{(TUFW) Frank-Wolfe Method with Taylor Point Updating}\n\t\t\\label{alg: TUFW in general finite sum}\n\t\t\\begin{algorithmic}[1]\n\t\t\t\\STATE \\textbf{input}  initial point $x^0$, step-size sequence $\\{\\gamma_k\\}_k$, and (possibly) number of iterations $K$\n\t\t\t\\STATE \\textbf{initialize Taylor points} $\\calB_0 \\gets [n]$ and $b_i \\leftarrow x^0$ for $i\\in [n]$  \\; \\label{bettina}\n\t\t\t\\FOR{$k=0,1,2,\\dots,K$,}\n\t\t\t\\STATE\\label{algline: 3}\\textbf{If $k>0$, update Taylor points}: \\\\ \\ \\ Use some \\textbf{Rule} to construct set $\\calB_k \\subset [n]$ of indices for Taylor-point updating\\\\ \\ \\ Update Taylor points:  for $i\\in\\calB_k$ update $b_i \\gets x^k$ \\; \\label{fanette}\n\t\t\t\\STATE\\label{algline: 4}  \\textbf{Compute estimates of individual gradients}:\\\\ $g^k_{i}\\gets \\nabla f_{i}(b_{i})  +  \\nabla^2 f_i (b_{i}) ( x^k -  b_{i} ) $ for $i\\in [n]$\\; \\label{charlie}\n\t\t\t\\STATE\\label{algline: 5}   \\textbf{Compute estimate of full gradient}: $g^k\\gets \\frac{1}{n}\\sum_{i=1}^{n}g_i^k$\\; \\label{beverly}\n\t\t\t\\STATE\\label{algline: 6}   \\textbf{Solve LMO}: $s^{k}\\gets \\arg\\min_{s\\in\\calC} \\langle g^k,s\\rangle$\\; \\label{alex} \n\t\t\t\\STATE\\label{algline: 7}   \\textbf{Update feasible solution}: $x^{k+1}\\gets x^k + \\gamma_k (s^{k} - x^k)$. \\label{hotel}\n\t\t\t\\ENDFOR\n\t\t\t\\RETURN $x^{K+1}$\n\t\t\\end{algorithmic}\n\t\\end{algorithm}\n\n\n\tRecall that in order to run \\cref{alg: TUFW in general finite sum} we need to specify the step-sizes $\\{\\gamma_k\\}_k$ and the {\\bf Rule} to construct the sets $\\calB_k$ of indices for Taylor-point updating in \\cref{fanette}.  Our first rule is designed \n\tfor $\\calB_k$ to be comprised of $\\beta_k := n /\\sqrt{k}$ independently drawn samples from $[n]$ without replacement, for all $k \\ge 1$.   Since $\\beta_k$ is not in general an integer, then rather than ensuring $|\\calB_k| = \\beta_k$, we will instead ensure that $\\mathbb{E}|\\calB_k| = \\beta_k$ by proceeding as follows.  \n\t\n\t\n\t\n\t\\begin{definition}\\label{rule 1} $\\rone$.  Define $\\beta_k := n /\\sqrt{k}$ and $p_k := \\beta_k - \\lfloor \\beta_k \\rfloor$.  Sample $\\xi_k$ from the Bernoulli distribution $\\mathrm{Ber}(p_k)$ and then uniformly sample $\\lfloor \\beta_k \\rfloor + \\xi_k$ samples from $[n]$ without replacement.  Then define $\\calB_k$ to be the set of these index samples.\\end{definition}\n\t\n\tNote that by design of $\\rone$ it follows that $\\mathbb{E}_{\\xi_k} |\\calB_k| = \\beta_k$.  We call this rule ``$\\rone$'' for ``Stochastic Batch-size Decreasing at the rate $\\sqrt{k}$,'' and we emphasize that the Taylor points are updated less often as $k$ grows, which is in contrast to many other stochastic Frank-Wolfe methods that compute exact individual gradients more frequently as $k$ grows, see \\cite{hazan2016variance,yurtsever2019conditional}. \n\n\n\tLet $D$ denote the diameter of $\\calC$ under the norm of the variable space, namely\n\t$$\n\tD:= \\max_{u,v\\in\\calC} \\| u-v\\| \\ .\n\t$$\n\n\n\tIn the rest of the paper, we will use $x^\\star$ to denote an optimal solution of the $\\erm$ problem, and let $\\eps_k := F(x^k) - F(x^\\star)$. \n\n\n\tWe begin with the following straightforward properties of smooth functions.\n\t\\begin{proposition}\\label{prop: general basic ineq}\n\t\tUnder \\cref{assump: ERM} for $\\erm$,  for any $x$, $y\\in\\calC$ and $i$, \n\t\t\\begin{align}\n\t\t\t| f_i(y) -  f_i(x) - \\langle\\nabla f_i(x),  y- x\\rangle |\\le  L\\|x-y\\|^2 /2 \\ , \\text{ and } \\label{eq: fact: general basic ineq 1} \\\\\n\t\t\t\\| \\nabla f_i(y) - \\nabla f_i(x) - \\langle\\nabla^2 f_i(x),  y- x\\rangle \\|_*\\le  \\hat{L}\\|x-y\\|^2 /2 \\ . \\label{eq: fact: general basic ineq 2}\n\t\t\\end{align}\n\t\\end{proposition}\n\n\n\tFor any $k \\ge 0$ and $i\\in [n]$, we will be interested in the most recent iteration (up through $k$) at which Taylor point $b_i$ was updated.  This is defined as:\n\t\\begin{equation}\\label{def: tau}\n\t\t\\tau^k_i := \\max\\{t: t\\le k  \\text{ and } i \\in \\calB_t\\} \\ ,\n\t\\end{equation}\n\twhich means that in the first $k$ iterations, $\\tau^k_i$ is the latest iteration in which the Taylor point $b_i$ was updated. In \\cref{alg: TUFW in general finite sum}, the relationship between $\\tau^k_i$ and $\\tau^{k-1}_i$ is then:\n\t\\begin{equation}\\label{eq: tauk relationship}\n\t\t\\tau^{k}_i = \\left\\{\n\t\t\\begin{array}{ll}\n\t\t\tk & \\text{ if } \\  i\\in\\calB_k \\\\\n\t\t\t\\tau^{k-1}_i & \\text{ if } \\ i\\notin\\calB_k \\ . \n\t\t\\end{array}\n\t\t\\right.\n\t\\end{equation}\n","units":[{"raw_tex":"\tOur problem of interest is the following Empirical Risk Minimization (ERM) problem (also called the \\textit{finite-sum} problem) with constraints: \n\t\\begin{equation}\\label{pro: ERM}\n\t\t\\text{ERM:} \\ \\ \\ \\ \\    \\text{minimize}_{x \\in \\calC} \\quad F(x):= \\tfrac{1}{n} \\sum_{i=1}^{n} f_i(x) \\ , \n\t\\end{equation}\n\twhere $f_i(\\cdot): \\mathbb{R}^p \\to \\mathbb{R}$ is the loss function of observation $i$ for $i=1,\\dots, n$, and $x$ is comprised of the (model) coefficients/parameters which are to be optimized over the feasibility set $\\calC \\subset \\mathbb{R}^p$. Problem \\cref{pro: ERM} is a fundamental problem in statistical and machine learning, as it is the underlying optimization problem on which models are ``trained'' or ``learned'', see  \\cite{vapnik1991principles,hastie2009elements}. The number of observations $n$ can be significantly larger than the dimension $p$, and our interest here will be in the regime where $n \\gg p$.\n\tWe make the following assumptions for $\\erm$ problems throughout this paper:\n\t\\begin{assumption}\\label{assump: ERM}\n\t\tThe following conditions hold for the $\\erm$ problem \\cref{pro: ERM}:\n\t\t\\begin{enumerate}\n\t\t\t\\item[(i)] the feasibility set $\\mathcal{C}$ is a compact convex set,\n\t\t\t\\item[(ii)] for any $g \\in \\mathbb{R}^p$, the linear minimization problem $\\arg\\min_{x\\in\\calC} g^\\top x$\n\t\t\tcan be easily solved, \n\t\t\t\\item[(iii)] for all $i\\in[n]$, the multivariate loss function $f_i(\\cdot)$ is twice-differentiable and its gradient $\\nabla f_i(\\cdot)$ is $L$-Lipschitz continuous on $\\calC$, namely, \n\t\t\t\\begin{equation}\\label{eq: ERM ass lip 1}\n\t\t\t\t\\| \\nabla f_i(u) - \\nabla f_i(v) \\|_* \\le L \\|u-v\\| ~~\\text{ for any $u,v\\in \\calC$}\n\t\t\t\\end{equation}\n\t\t\twhere $\\|\\cdot\\|_*$ is the dual norm of the norm on the space of variables, and \n\t\t\t\\item[(iv)] for all $i\\in[n]$, the Hessian $\\nabla^2 f_i(\\cdot)$ is $\\hat{L}$-Lipschitz continuous on $\\calC$, namely\n\t\t\t\\begin{equation}\\label{eq: ERM ass lip 2}\n\t\t\t\t\\| \\nabla^2 f_i(u) - \\nabla^2 f_i(v) \\| \\le \\hat{L} \\|u-v\\|~~\\text{ for any $u,v\\in \\calC$}\n\t\t\t\\end{equation}\n\t\t\twhere the norm of the Hessian matrices is the operator norm induced by the norm on the space of variables.\n\t\t\\end{enumerate}\n\t\\end{assumption}\n","start":12615,"end":14818},{"raw_tex":"\t\\subsection{Notation}\n\t\n\tLet $\\mathbb{N}$ denote the natural numbers.  For any $n \\in \\mathbb{N}$, we use $[n]$ to denote the set $\\{1,2,\\dots,n\\}$.\n\tFor any real number $v$, we use $\\lfloor v \\rfloor$ to denote the largest integer number less than or equal to $v$.\n\tDepending on the context, we will use either $\\langle a,b\\rangle$ or $a^\\top b$ to denote the inner product between $a$ and $b$ in Euclidean space. For a vector $x$ in $\\mathbb{R}^p$, the norm $\\|x\\|_q$ is given by $(\\sum_{j=1}^{p}x_j^q)^{1/q}$ for $q \\in [1,\\infty)$ and $\\|x\\|_\\infty := \\max_{j} |x_j|$. The dual of norm $\\|\\cdot\\|$ is defined as $\\|z\\|_* := \\sup\\{z^\\top x : \\|x\\| \\le 1\\}$.  For a matrix $A \\in \\mathbb{R}^{p \\times p}$, the matrix operator norm $\\|A\\|$ induced by the norm $\\|\\cdot\\|$ is defined as $\\sup_{x: \\|x\\| \\le 1} \\|Ax\\|_*$. For any symmetric positive definite matrix $A\\in \\mathbb{S}^{p\\times p}$, we define its induced norm on $\\mathbb{R}^p$ by $\\|z\\|_A := \\sqrt{z^\\top A z}$.  We define $\\sum_{i=j}^k \\alpha_i := 0$ whenever $j > k$.  We denote the uniform distribution on the set $S$ by $\\calU(S)$, and we denote the Bernoulli distribution with probability $p$ by $\\mathrm{Ber}(p)$.\n","start":34310,"end":35494},{"raw_tex":"\tIn this section we present a new type of Frank-Wolfe algorithm whose newness derives from the way in which approximate gradients are computed.  We call the algorithm TUFW for ``Taylor-point Updating Frank-Wolfe''. Algorithm TUFW is similar to the standard Frank-Wolfe method except that it computes an inexpensive estimate $g^k$ of the gradient $\\nabla F(x^k)$ at iteration $k$.  And it differs from other Frank-Wolfe methods in the way it computes the estimate $g^k$ in that (i) TUFW explicitly uses second-derivative (Hessian) information of the loss functions $f_i$, and (ii) it can use very different batch-size rules -- both stochastic and deterministic. TUFW is described in \\cref{alg: TUFW in general finite sum}.  We now explain the steps of the method.\n\t\n\t\n\t\\begin{algorithm}\n\t\t\\caption{(TUFW) Frank-Wolfe Method with Taylor Point Updating}\n\t\t\\label{alg: TUFW in general finite sum}\n\t\t\\begin{algorithmic}[1]\n\t\t\t\\STATE \\textbf{input}  initial point $x^0$, step-size sequence $\\{\\gamma_k\\}_k$, and (possibly) number of iterations $K$\n\t\t\t\\STATE \\textbf{initialize Taylor points} $\\calB_0 \\gets [n]$ and $b_i \\leftarrow x^0$ for $i\\in [n]$  \\; \\label{bettina}\n\t\t\t\\FOR{$k=0,1,2,\\dots,K$,}\n\t\t\t\\STATE\\label{algline: 3}\\textbf{If $k>0$, update Taylor points}: \\\\ \\ \\ Use some \\textbf{Rule} to construct set $\\calB_k \\subset [n]$ of indices for Taylor-point updating\\\\ \\ \\ Update Taylor points:  for $i\\in\\calB_k$ update $b_i \\gets x^k$ \\; \\label{fanette}\n\t\t\t\\STATE\\label{algline: 4}  \\textbf{Compute estimates of individual gradients}:\\\\ $g^k_{i}\\gets \\nabla f_{i}(b_{i})  +  \\nabla^2 f_i (b_{i}) ( x^k -  b_{i} ) $ for $i\\in [n]$\\; \\label{charlie}\n\t\t\t\\STATE\\label{algline: 5}   \\textbf{Compute estimate of full gradient}: $g^k\\gets \\frac{1}{n}\\sum_{i=1}^{n}g_i^k$\\; \\label{beverly}\n\t\t\t\\STATE\\label{algline: 6}   \\textbf{Solve LMO}: $s^{k}\\gets \\arg\\min_{s\\in\\calC} \\langle g^k,s\\rangle$\\; \\label{alex} \n\t\t\t\\STATE\\label{algline: 7}   \\textbf{Update feasible solution}: $x^{k+1}\\gets x^k + \\gamma_k (s^{k} - x^k)$. \\label{hotel}\n\t\t\t\\ENDFOR\n\t\t\t\\RETURN $x^{K+1}$\n\t\t\\end{algorithmic}\n\t\\end{algorithm}\n","start":35580,"end":37679},{"raw_tex":"\tRecall that in order to run \\cref{alg: TUFW in general finite sum} we need to specify the step-sizes $\\{\\gamma_k\\}_k$ and the {\\bf Rule} to construct the sets $\\calB_k$ of indices for Taylor-point updating in \\cref{fanette}.  Our first rule is designed \n\tfor $\\calB_k$ to be comprised of $\\beta_k := n /\\sqrt{k}$ independently drawn samples from $[n]$ without replacement, for all $k \\ge 1$.   Since $\\beta_k$ is not in general an integer, then rather than ensuring $|\\calB_k| = \\beta_k$, we will instead ensure that $\\mathbb{E}|\\calB_k| = \\beta_k$ by proceeding as follows.  \n\t\n\t\n\t\n\t\\begin{definition}\\label{rule 1} $\\rone$.  Define $\\beta_k := n /\\sqrt{k}$ and $p_k := \\beta_k - \\lfloor \\beta_k \\rfloor$.  Sample $\\xi_k$ from the Bernoulli distribution $\\mathrm{Ber}(p_k)$ and then uniformly sample $\\lfloor \\beta_k \\rfloor + \\xi_k$ samples from $[n]$ without replacement.  Then define $\\calB_k$ to be the set of these index samples.\\end{definition}\n\t\n\tNote that by design of $\\rone$ it follows that $\\mathbb{E}_{\\xi_k} |\\calB_k| = \\beta_k$.  We call this rule ``$\\rone$'' for ``Stochastic Batch-size Decreasing at the rate $\\sqrt{k}$,'' and we emphasize that the Taylor points are updated less often as $k$ grows, which is in contrast to many other stochastic Frank-Wolfe methods that compute exact individual gradients more frequently as $k$ grows, see \\cite{hazan2016variance,yurtsever2019conditional}. \n","start":46870,"end":48280},{"raw_tex":"\tLet $D$ denote the diameter of $\\calC$ under the norm of the variable space, namely\n\t$$\n\tD:= \\max_{u,v\\in\\calC} \\| u-v\\| \\ .\n\t$$\n","start":49654,"end":49784},{"raw_tex":"\tIn the rest of the paper, we will use $x^\\star$ to denote an optimal solution of the $\\erm$ problem, and let $\\eps_k := F(x^k) - F(x^\\star)$. \n","start":62922,"end":63066},{"raw_tex":"\tWe begin with the following straightforward properties of smooth functions.\n\t\\begin{proposition}\\label{prop: general basic ineq}\n\t\tUnder \\cref{assump: ERM} for $\\erm$,  for any $x$, $y\\in\\calC$ and $i$, \n\t\t\\begin{align}\n\t\t\t| f_i(y) -  f_i(x) - \\langle\\nabla f_i(x),  y- x\\rangle |\\le  L\\|x-y\\|^2 /2 \\ , \\text{ and } \\label{eq: fact: general basic ineq 1} \\\\\n\t\t\t\\| \\nabla f_i(y) - \\nabla f_i(x) - \\langle\\nabla^2 f_i(x),  y- x\\rangle \\|_*\\le  \\hat{L}\\|x-y\\|^2 /2 \\ . \\label{eq: fact: general basic ineq 2}\n\t\t\\end{align}\n\t\\end{proposition}\n","start":63068,"end":63607},{"raw_tex":"\tFor any $k \\ge 0$ and $i\\in [n]$, we will be interested in the most recent iteration (up through $k$) at which Taylor point $b_i$ was updated.  This is defined as:\n\t\\begin{equation}\\label{def: tau}\n\t\t\\tau^k_i := \\max\\{t: t\\le k  \\text{ and } i \\in \\calB_t\\} \\ ,\n\t\\end{equation}\n\twhich means that in the first $k$ iterations, $\\tau^k_i$ is the latest iteration in which the Taylor point $b_i$ was updated. In \\cref{alg: TUFW in general finite sum}, the relationship between $\\tau^k_i$ and $\\tau^{k-1}_i$ is then:\n\t\\begin{equation}\\label{eq: tauk relationship}\n\t\t\\tau^{k}_i = \\left\\{\n\t\t\\begin{array}{ll}\n\t\t\tk & \\text{ if } \\  i\\in\\calB_k \\\\\n\t\t\t\\tau^{k-1}_i & \\text{ if } \\ i\\notin\\calB_k \\ . \n\t\t\\end{array}\n\t\t\\right.\n\t\\end{equation}\n","start":63609,"end":64341}]},"statement":{"raw_tex":"\\begin{theorem}\\label{thm: general convergence rate convex}\n\t\tSuppose that $F$ is convex and Assumption \\ref{assump: ERM} holds, and  \\cref{alg: TUFW in general finite sum} with $\\rone$ is applied to the problem \\cref{pro: ERM} with step-sizes defined by $\\gamma_k:={2}/({k+2})$ for all $k \\ge 0$. Then for all $k\\ge 1$ we have:\n\t\t\\begin{equation}\\label{eq of thm: general convergence rate convex}\n\t\t\t\\mathbb{E}[F(x^{k}) - F(x^\\star)] \\le  \\frac{2L D^2 + 134 \\hat{L}D^3}{k+1} \\ .\n\t\t\\end{equation} \n\t\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\\label{thm: general convergence rate convex}\n\t\tSuppose that $F$ is convex and Assumption \\ref{assump: ERM} holds, and  \\cref{alg: TUFW in general finite sum} with $\\rone$ is applied to the problem \\cref{pro: ERM} with step-sizes defined by $\\gamma_k:={2}/({k+2})$ for all $k \\ge 0$. Then for all $k\\ge 1$ we have:\n\t\t\\begin{equation}\\label{eq of thm: general convergence rate convex}\n\t\t\t\\mathbb{E}[F(x^{k}) - F(x^\\star)] \\le  \\frac{2L D^2 + 134 \\hat{L}D^3}{k+1} \\ .\n\t\t\\end{equation} \n\t\\end{theorem}","start":50034,"end":50546}]},"proof":{"raw_tex":"\\begin{lemma}\\label{lm: general 3rd term upper bound}\n\t\tUnder \\cref{assump: ERM} for the problem $\\erm$, for any $u,v \\in\\calC$,\n\t\t\\begin{equation}\\label{eq: lm: general 3rd term upper bound}\n\t\t\t\\left(\n\t\t\t\\nabla F(x^k)   - g^k\n\t\t\t\\right)^\\top (u - v) \\le \\frac{\\hat{L}D^3}{2n}\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j \\Big)^2\n\t\t\\end{equation}\n\t\tholds for iteration $k$ of Algorithm \\ref{alg: TUFW in general finite sum} for all $k \\ge 1$.\n\t\\end{lemma}\n\n\\begin{proof}[Proof of \\cref{lm: general 3rd term upper bound}]\n\t\tFirst of all, $\\left(\n\t\t\\nabla F(x^k)   - g^k\n\t\t\\right)^\\top (u - v)$ can be rewritten as\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: general etimate gradient 0}\n\t\t\t\t\\frac{1}{n}\\sum_{i=1}^{n}\\left(\\nabla f_{i}\\big(x^k \\big) -   \\nabla f_{i}\\big( x^{\\tau^{k}_i}\\big) -  \\nabla^2 f_{i} \\big( x^{\\tau^{k}_i}\\big)\\big(x^k  - x^{\\tau^{k}_i} \\big)    \\right)^\\top (u - v),\n\t\t\t\\end{equation}\t\n\t\t\\end{small}which, since $\\|u-v\\| \\le D$, is smaller than or equal to \n\t\t\\begin{small}\\begin{equation}\\label{ineq: general etimate gradient 1}\n\t\t\t\\frac{D}{n}\\sum_{i=1}^{n}\\left\\| \\nabla f_{i}\\big(x^k \\big) -   \\nabla f_{i}\\big( x^{\\tau^{k}_i}\\big) -  \\nabla^2 f_{i} \\big( x^{\\tau^{k}_i}\\big)\\big(x^k  - x^{\\tau^{k}_i} \\big)    \\right\\|_* \\le    \\frac{\\hat{L}D}{2n}\\sum_{i=1}^{n}\\big\\|x^k  - x^{\\tau^{k}_i}\\big\\|^2,\n\t\t\\end{equation}\\end{small}where the inequality is due to \\cref{eq: fact: general basic ineq 2} in \\cref{prop: general basic ineq}.\n\t\tAdditionally, we have \n\t\t\\begin{small}\\begin{equation*}\n\t\t\t\\|x^k  - x^{\\tau^{k}_i}\n\t\t\t\\|^2 \\le  \\bigg( \\sum_{j = \\tau^k_i}^{k-1} \\| x^{j+1} -  x^{j}\\| \\bigg)^2\\le  \\bigg( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j D \\bigg)^2 \\  ,\n\t\t\\end{equation*}\\end{small}and substituting this last bound into \\cref{ineq: general etimate gradient 1} yields \\cref{eq: lm: general 3rd term upper bound}.\n\t\\end{proof}\n\n\\begin{lemma}\\label{lm: general convergence rate convex}\n\t\tUnder \\cref{assump: ERM} for problem $\\erm$, suppose that $F$ is conevx, and let the step-sizes be given by $\\gamma_k=2/(k+2)$ for all $k \\ge 0$. Then for all $k \\ge 1$ we have\n\t\t\\begin{equation}\\label{eq lm: general convergence rate convex}\n\t\t\t\\eps_k  \\le  \\frac{2L D^2}{k+1}\n\t\t\t+  \\frac{2}{k(k+1)}\\sum_{t=1}^{k} t \\left(\n\t\t\t\\nabla F(x^{t-1})   - g^{t-1}\n\t\t\t\\right)^\\top (s^{t-1} - x^\\star) \\ .\n\t\t\\end{equation}\n\t\\end{lemma}\n\n\\begin{proof}[Proof of \\cref{lm: general convergence rate convex}]\n\t\tUtilizing  \\cref{prop: general basic ineq} we have\n\t\t\\begin{small}\\begin{equation}\\label{eq lm general convergence rate convex 0}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\tF(x^{k+1})\\leq ~ & F(x^k)+\\gamma_{k}\\langle\\nabla F(x^k),s^{k}- x^k \\rangle+ \\gamma_{k}^{2} {L D^2}/{2} \\\\\n\t\t\t\t\t= ~ & F(x^{k})+\\gamma_{k}\\langle \\nabla F(x^{k})-g^k, s^{k}-x^k\\rangle +\\gamma_{k}\\langle{g^k}, s^{k}-x^k\\rangle+ \\gamma_{k}^{2} {L D^2}/{2} \\\\\n\t\t\t\t\t\\leq ~ & F(x^{k})+\\gamma_{k}\\langle\\nabla F(x^{k})-{g^k}, s^{k}-x^k\\rangle + \\gamma_{k}\\langle{g^k}, {x}^{\\star}-x^k\\rangle+ \\gamma_{k}^{2} {L D^2}/{2} \\\\\n\t\t\t\t\t= ~ & F(x^{k})+\\gamma_{k}\\langle \\nabla F(x^{k})-{g^k},s^{k}-{x}^{\\star}\\rangle +\\gamma_{k}\\langle \\nabla F(x^{k}), {x}^{\\star}-x^k\\rangle+ \\gamma_{k}^{2} {L D^2}/{2}\\\\\n\t\t\t\t\t\\le ~& F(x^{k})+\\gamma_{k}\\langle \\nabla F(x^{k})-{g^k},s^{k}-{x}^{\\star}\\rangle +\\gamma_{k}(F(x^\\star) - F(x^k))+ \\gamma_{k}^{2} {L D^2}/{2} \\ , \n\t\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses $s^{k} \\in \\arg\\min_{s\\in\\calC} (g^k)^\\top s$ from \\cref{alex} of the algorithm, and the third inequality uses the gradient inequality applied to $F$.  Subtracting $F^\\star$ from both sides of the above inequality chain, we arrive at:\n\t\t\\begin{small}\\begin{equation}\\label{eq lm general convergence rate convex 1}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t\\eps_{k+1}& \\le (1-\\gamma_k)\\eps_k + \\gamma_k \\big(\n\t\t\t\t\t\\nabla F(x^k)   - g^k\n\t\t\t\t\t\\big)^\\top (s^{k} - x^\\star) +  {\\gamma_k^2 L D^2}/{2} \\ .\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}Multiplying both side by $(k+1)(k+2)$ and telescoping the inequalities yields:\n\t\t\\begin{small}\\begin{equation}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t(k+1)(k+2)\\eps_{k+1}  &\\le  2(k+1)L D^2\n\t\t\t\t\t+  \\sum_{t=0}^{k}2(t+1) (\n\t\t\t\t\t\\nabla F(x^t)   - g^t\n\t\t\t\t\t)^\\top (s^{t} - x^\\star) \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}which yields \\cref{eq lm: general convergence rate convex} after rearranging and re-indexing the counter $t$.\n\t\\end{proof}\n\n\\begin{lemma}\\label{lm: convex EZ} Let the series $\\{d_{ij}\\}_{i\\in[n],j\\in\\mathbb{N}}$ be given, and suppose that there exist $M_{1}$ and $M_{\\infty}$ for which $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$ and  $\\sup_{i,j} d_{ij}\\le M_{\\infty}$.  \n\t\tLet $\\tau_i^k$ be defined in \\cref{def: tau}, in which $\\calB_t$ is constructed using $\\rone$ in \\cref{rule 1}. Let $\\gamma_k$ be ${2}/(k+2)$ for all $k \\ge 0$.  Then it holds that\n\t\t\\begin{equation}\\label{eq: lm: convex EZ}\n\t\t\t\\mathbb{E}\\bigg[\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\Big)\\bigg] \\le\n\t\t\t\\frac{6 M_1}{\\sqrt{k+2}} \\ \\ \\text{ and }   \\ \\\n\t\t\t\\mathbb{E} \\bigg[\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\Big)^2 \\bigg]\\le  \\frac{134M_1M_\\infty}{k+2} \\ .\n\t\t\\end{equation}\n\t\\end{lemma}\n\n\\begin{proposition}\\label{lm: rho}\n\t\tFor integers $v$ and $k$ satisfying $1\\le  v \\le k$ it holds that:\n\t\t\\begin{equation}\\label{eq: fact 2}\n\t\t\t\\sum_{i=1}^{v}\\prod_{j=i}^{k}\\left(1-\\tfrac{1}{\\sqrt{j}}\\right) <\n\t\t\t(\\sqrt{v+1})\\exp(2\\sqrt{v+1} - 2\\sqrt{k+1})  \\   .\n\t\t\\end{equation}\n\t\\end{proposition}\n\n\\begin{proof}[Proof of \\cref{lm: rho}]\n\t\tFor convenience let us define $\\rho_j := 1-\\tfrac{1}{\\sqrt{j}}$ for $j \\ge 1$, and define the function $h(x):= \\exp(2 \\sqrt{x} - 2 \\sqrt{k+1}) $ for $x \\ge 0$.  We will first prove the following inequality:\n\t\t\\begin{small}\\begin{equation}\\label{lulu}\n\t\t\t\\prod_{j=i}^{k}\\left(1-\\tfrac{1}{\\sqrt{j}}\\right) \\le h(i) \\ \\ \\ \\mathrm{for~all~} i \\in [k] \\ . \n\t\t\\end{equation}\\end{small}For $i=1$ the left side above is $0$ (because $\\rho_1 = 0$ and hence $\\prod_{j=1}^{k}\\rho_j = 0$), and the right side is $h(1) >0$, so \\cref{lulu} is true for $i=1$. For $i\\ge2$ we have:\n\t\t\\begin{small}\\begin{equation*}\\begin{aligned}\n\t\t\t\t\\prod_{j=i}^{k}\\rho_j  &= \\exp\\left({\\sum_{j=i}^{k}\\ln(1-\\tfrac{1}{\\sqrt{j}})}\\right) \\le \\exp\\left( -\\sum_{j=i}^{k}\\tfrac{1}{\\sqrt{j}}\\right)  \\\\  \n\t\t\t\t&\\le \\exp\\Big( -\\int_i^{k+1} x^{-\\tfrac{1}{2}} dx\\Big) =\\exp(2\\sqrt{i} - 2\\sqrt{k+1}) = h(i) \\ , \\end{aligned} \n\t\t\\end{equation*}\\end{small}where the first inequality follows from the fact that $\\log(1-x)\\le -x$ for any $0\\le x <1$, and the second inequality is an integral bound. This establishes \\cref{lulu}, whereby we have\n\t\t\\begin{small}$$ \\begin{aligned}\n\t\t\t\t\\sum_{i=1}^{v}   \\prod_{j=i}^{k}\\rho_j  &\\le  \\sum_{i=1}^{v} h(i) \\le \\int_{1}^{v+1}h(x)~dx = \\left. (\\sqrt{x}-\\tfrac{1}{2})\\exp(2\\sqrt{x} - 2\\sqrt{k+1}) \\right|_1^{v+1} \\\\ \n\t\t\t\t  &= (\\sqrt{v+1}-\\tfrac{1}{2})\\exp(2\\sqrt{v+1} - 2\\sqrt{k+1}) - (\\tfrac{1}{2})\\exp(2 - 2\\sqrt{k+1}) \\\\\\\n\t\t\t\t&< (\\sqrt{v+1}-\\tfrac{1}{2})\\exp(2\\sqrt{v+1} - 2\\sqrt{k+1}) \\ ,\n\t\t\\end{aligned}$$\\end{small}where the second inequality is an integral bound using the fact that $h(x)$ is an increasing function. \\end{proof}\n\n\\begin{lemma}\\label{lm: simple ineq}\n\t\tFor any $k\\in\\mathbb{N}$ the following inequality holds:\n\t\t\\begin{equation}\\label{ineq: lm: simple ineq} \n\t\t\t\\sqrt{\\lfloor k/2 \\rfloor + 1} \\exp(2\\sqrt{\\lfloor k/2\\rfloor + 1} - 2\\sqrt{k+1}) \\le \\min\\{{3}(k+2)^{-\\frac{1}{2}},{10}(k+2)^{-1}\\}. \n\t\t\\end{equation}\n\t\\end{lemma}\n\n\\begin{proof}[Proof of \\cref{lm: simple ineq}]\n\t\tSince $h(x):=\\sqrt{x}$ is a concave function on $\\mathbb{R}_+$, we have $h(k+2) \\le h(k+1) + h'(k+1)$, which yields $\\sqrt{k+1}\\ge \\sqrt{k+2} - \\frac{1}{2\\sqrt{k+1}} \\ge \\sqrt{k+2} - \\sqrt{2}/4 $ when $k \\ge 1$.\n\t\tFurthermore $\\sqrt{\\lfloor k/2 \\rfloor + 1} \\le \\frac{\\sqrt{2}}{2} \\sqrt{k+2}$, and therefore \n\t\t\\begin{small}\\begin{equation*}\n\t\t\t\\sqrt{\\lfloor k/2 \\rfloor + 1} \\exp(2\\sqrt{\\lfloor k/2\\rfloor + 1} - 2\\sqrt{k+1}) \\le  \\frac{\\sqrt{2}e^{\\frac{\\sqrt{2}}{2}}}{2}(k+2)^{\\frac{1}{2}}  \\exp(-(2-\\sqrt{2})(k+2)^{\\frac{1}{2}}) \\ .\n\t\t\\end{equation*}\\end{small}Next note that the function $h(x):=e^{-x} x^t$ on the domain $x \\in (0,\\infty)$ for $t > 0$ attains its maximal value at $x = t$. In particular for $t=2$ and $t=3$, respectively, we conclude that $e^{-x} \\le 4e^{-2}x^{-2}$ and $e^{-x} \\le 27e^{-3}x^{-3}$ (respectively), and thus  $\\exp(-(2-\\sqrt{2})(k+2)^{\\frac{1}{2}}) \\le 4 e^{-2}(2-\\sqrt{2})^{-2}(k+2)^{-1}$\n\t\tand $\\exp(-(2-\\sqrt{2})(k+2)^{\\frac{1}{2}}) \\le 27 e^{-3}(2-\\sqrt{2})^{-3}(k+2)^{-3/2}$ (respectively).\n\t\tTherefore, the left-hand side of \\cref{ineq: lm: simple ineq} is bounded by \n\t\t$2\\sqrt{2}e^{\\frac{\\sqrt{2}}{2}-2} (2-\\sqrt{2})^{-2} (k+2)^{-\\frac{1}{2}}$ and \n\t\t$\\frac{27}{2}\\sqrt{2}e^{\\frac{\\sqrt{2}}{2}-3} (2-\\sqrt{2})^{-3} (k+2)^{-1}$, respectively, which is less than $3(k+2)^{-\\frac{1}{2}}$ and $10(k+2)^{-1}$, respectively.\n\t\\end{proof}\n\n\\begin{proof}[Proof of \\cref{lm: convex EZ}]\n\t\t\n\t\tFirst of all, let us define $G_k^i := \\left( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\right)$ and $Z_k^i :=\\left( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\right)^2$ for $i \\in [n]$, and also $G_k :=\\sum_{i=1}^n G_k^i$ and $Z_k :=\\sum_{i=1}^n Z_k^i$. Notice in $G_k^i$ and $Z_k^i$ that the randomness lies only in $\\tau_i^k$, which in turn depends on the update sets $\\calB_0, \\calB_1,\\dots,\\calB_k$. \n\t\t\n\t\tWe first analyze $G_k^i$ and $G_k$.  We note that $G_k^i = 0$ when $i \\in \\calB_k$, which happens with probability $\\beta_k/n$.  Because of the relationship between $\\{\\tau^k_i\\}$ and $\\{\\tau^{k-1}_i\\}$ in \\cref{eq: tauk relationship}, we further note that with probability $1-\\beta_k/n$ that $i \\notin \\calB_k$ in which case it follows that $G_k^i = G_{k-1}^i + \\gamma_{k-1}d_{i,k-1}$.  Defining $\\rho_k := 1 - \\frac{1}{\\sqrt{k}} = 1 - \\beta_k/n $, we have \n\t\t\\begin{small}\\begin{equation}\\label{ncl}\n\t\t\t\\mathbb{E}[ G_k ] = \\sum_{i=1}^n \\mathbb{E} [G_k^i ]= \\rho_k \\sum_{i=1}^n \\mathbb{E} [G_{k-1}^i +\n\t\t\t\\gamma_{k-1} d_{i,k-1}] \\le \\rho_k\\left( \\mathbb{E} [G_{k-1} ]+\n\t\t\t\\gamma_{k-1} M_1 \\right) \\ ,\n\t\t\\end{equation}\\end{small}where the inequality uses $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$. Recurrently applying \\cref{ncl} yields:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent G_k 2}\n\t\t\t \\mathbb{E}[ G_k ] \\le \\rho_k( \\mathbb{E}[ G_{k-1} ]+\n\t\t\t\\gamma_{k-1}  M_1 ) \\le\\cdots\\le \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\gamma_{i-1} M_1 + \\Big(\\prod_{j=1}^{k} \\rho_j\\Big)\\mathbb{E}[ G_0] \\ .\n\t\t\\end{equation}\\end{small}Noting that $G_0 = 0$ (since all Taylor points are set to $x^0$ in \\cref{bettina} of \\cref{alg: TUFW in general finite sum}) and using $\\gamma_{i-1} = \\frac{2}{i+1}$ it follows that:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent G_k 3}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\mathbb{E} [G_k] &\\le \\sum_{i=1}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{i+1}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor {k}/{2}\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{i+1}\\\\\n\t\t\t\t& \\le \\sum_{i=2}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{3}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor {k}/{2}\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor {k}/{2}\\rfloor+2} \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses the fact that $\\rho_1 = 0$ and hence $\\prod_{j=1}^{k}\\rho_j = 0$.\n\t\tBy \\cref{lm: rho} we have:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: G_k recurrent sumpro 1}\n\t\t\t\\sum_{i=1}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)  \\frac{2M_1}{3}\n\t\t\t\\le\n\t\t\t\\sqrt{\\lfloor {k}/{2}\n\t\t\t\t\\rfloor + 1} \\exp\\Big(2 \\sqrt{ \\lfloor {k}/{2} \\rfloor + 1} - 2\\sqrt{k+1}\\Big)  \\frac{2M_1}{3} \\le \\frac{2 M_1}{\\sqrt{k+2}} \\ ,\n\t\t\\end{equation}\\end{small}where the second inequality above uses \\cref{lm: simple ineq}.\n\t\tSimilarly for the second term in the rightmost side of \\cref{ineq: recurrent G_k 3} we have:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: G_k recurrent sumpro 2}\n\t\t\t\\sum_{i=\\lfloor k/2 \\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor k/2\\rfloor+2} \\le   \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor k/2\\rfloor+2} \\le \\frac{\\sqrt{k+1} \\cdot  2M_1}{\\lfloor k/2\\rfloor+2} \\le \\frac{4M_1}{\\sqrt{k+2}} \\ , \n\t\t\\end{equation}\\end{small}where the second inequality uses \\cref{lm: rho}.\n\t\tSubstituting \\cref{ineq: G_k recurrent sumpro 1} and \\cref{ineq: G_k recurrent sumpro 2} back into \\cref{ineq: recurrent G_k 3} yields the first part of \\cref{eq: lm: convex EZ}, namely:\n\t\t\\begin{equation}\\label{ineq: convex EG_k final}\n\t\t\t\\mathbb{E} [G_k] \\le {6 M_1}/{\\sqrt{k+2}} \\ .\n\t\t\\end{equation}\n\t\t\n\t\tWe employ a similar proof approach to establish the second part of \\cref{eq: lm: convex EZ}. We note that $Z_k^i = 0$ when $i \\in \\calB_k$, which happens with probability $\\beta_k/n$.  Because of the relationship between $\\{\\tau^k_i\\}$ and $\\{\\tau^{k-1}_i\\}$ in \\cref{eq: tauk relationship}, we further note that with probability $1-\\beta_k/n$ that $i \\notin \\calB_k$ in which case we have\n\t\t\\begin{small}\\begin{equation*}\n\t\t\t\\begin{aligned} \n\t\t\t\tZ_k^i &= (G_{k-1}^i + \\gamma_{k-1}d_{i,k-1})^2 = Z_{k-1}^i + 2 G_{k-1}^i\\gamma_{k-1}d_{i,k-1} + (\\gamma_{k-1}d_{i,k-1})^2 \\\\ &\\le Z_{k-1}^i + 2 M_\\infty \\gamma_{k-1}G_{k-1}^i + \\gamma_{k-1}^2 M_\\infty d_{i,k-1} \\ . \n\t\t\t\\end{aligned}\\end{equation*}\\end{small}Therefore\n\t\t\\begin{small}\\begin{equation}\\begin{aligned}\\label{nclz}\n\t\t\t\t\\mathbb{E}[ Z_k ] = &\\sum_{i=1}^n \\mathbb{E} [Z_k^i ] \\le \\rho_k \\sum_{i=1}^n \\mathbb{E} [Z_{k-1}^i + 2 M_\\infty \\gamma_{k-1}G_{k-1}^i + \\gamma_{k-1}^2 M_\\infty d_{i,k-1}] \\\\ \n\t\t\t\t&\\le \\rho_k \\left( \\mathbb{E} [Z_{k-1}] + 2   M_\\infty \\gamma_{k-1}\\mathbb{E}[G_{k-1}] +   \\gamma_{k-1}^2 M_\\infty M_1 \\right) \\\\\n\t\t\t\t&\\le \\rho_k \\left(\\mathbb{E} [Z_{k-1}] + \\frac{24M_\\infty M_1}{(k+1)\\sqrt{k+1}}   +  \\frac{4M_\\infty M_1}{(k+1)^2} \\right) \\le \\rho_k \\left(\\mathbb{E} [Z_{k-1}] + \\frac{28M_\\infty M_1}{(k+1)\\sqrt{k+1}}  \\right)    \\ ,\n\t\t\\end{aligned}\\end{equation} \\end{small}where the second inequality uses $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$, and the third inequality uses \\cref{ineq: convex EG_k final}. Recurrently applying \\cref{nclz} yields:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent Z_k 4}\n\t\t\t\\mathbb{E} [Z_k]  \\le \\rho_k\\bigg( \\mathbb{E} [Z_{k-1} ]+\n\t\t\t\\frac{28M_1M_\\infty}{(k+1)^{3/2}}\\bigg) \\le \\cdots\\le \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}  \\ .\n\t\t\\end{equation}\\end{small}Therefore\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent Z_k 5}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\mathbb{E} [Z_k]  & \\le  \\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor k/2\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}\\\\\n\t\t\t\t& \\le \\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{3\\sqrt{3}}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor k/2\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\ .\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}By \\cref{lm: rho} we have \n\t\t\t\\begin{small}\\begin{equation}\\label{ineq: Z_k recurrent sumpro 1}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{3\\sqrt{3}}  &\\le \\sqrt{\\lfloor k/2\n\t\t\t\t\t\\rfloor + 1} \\exp\\big(2 \\sqrt{ \\lfloor k/2 \\rfloor + 1} - 2\\sqrt{k+1}\\big) \\cdot \\frac{28M_1M_\\infty}{(3\\sqrt{3})} \\\\ \n\t\t\t\t&\\le \\frac{{54M_1M_\\infty}}{k+2} \\ , \n\t\t\\end{aligned}\\end{equation}\\end{small}where the second inequality above uses \\cref{lm: simple ineq}. \n\t\tSimilarly for the second term in the rightmost side of \\cref{ineq: recurrent Z_k 5} we have:\n\t\t\t\\begin{small}\\begin{equation}\n\t\t\t\\begin{aligned}\\label{ineq: Z_k recurrent sumpro 2}\n\t\t\t\t\\sum_{i=\\lfloor k/2 \\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} &\\le   \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\\\ \n\t\t\t\t&\\le \\sqrt{k+1} \\cdot \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\le \\frac{80M_1M_\\infty}{k+2} \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses \\cref{lm: rho}.\n\t\tSubstituting \\cref{ineq: Z_k recurrent sumpro 1} and \\cref{ineq: Z_k recurrent sumpro 2} back into \\cref{ineq: recurrent Z_k 5} yields the second part of \\cref{eq: lm: convex EZ}, namely:\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: convex EZ_k final}\n\t\t\t\t\\mathbb{E} [Z_k] \\le  \\frac{134M_1M_\\infty}{k+2} \\ .\n\t\t\t\\end{equation}\n\t\t\\end{small}\\end{proof}\n\n\\begin{proof}[Proof of \\cref{thm: general convergence rate convex}]\n\t\tFrom \\cref{lm: general convergence rate convex} it holds that:\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\n\t\t\t\t\\eps_k  \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{2}{k(k+1)}\\sum_{t=1}^{k} t \\left(\n\t\t\t\t\\nabla F(x^{t-1})   - g^{t-1}\n\t\t\t\t\\right)^\\top (s^{t-1} - x^\\star) \\ ,\n\t\t\t\\end{equation}\n\t\t\\end{small}and from \\cref{lm: general 3rd term upper bound} it follows that:\n\t\t\\begin{small}\n\t\t\t\\begin{equation*}\n\t\t\t\t\\eps_k \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{\\hat{L}D^3}{nk(k+1)}\\sum_{t=1}^{k}t\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^{t-1}_i}^{t-2}  \\gamma_j \\Big)^2 \\ .\n\t\t\t\\end{equation*}\n\t\t\\end{small}Taking expectations on both sides then yields\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: convex thm 1 general}\n\t\t\t\t\\mathbb{E} [\\eps_k]  \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{\\hat{L}D^3}{nk(k+1)}\\sum_{t=1}^{k}t~\\mathbb{E}\\Big[\n\t\t\t\t\\sum_{i=1}^{n}\\Big(  \\sum_{j = \\tau^{t-1}_i}^{t-2}  \\gamma_j \\Big)^2\n\t\t\t\t\\Big] \\ .\n\t\t\t\\end{equation}\n\t\t\\end{small}Now define $d_{ij} := 1$ and $M_1 := n$ and $M_\\infty := 1$.  Then since $\\sup_{j}   \\sum_{i=1}^{n} d_{ij} = M_1$  and  $\\sup_{i,j} d_{ij}$ is equal to $M_\\infty$, we can apply \\cref{lm: convex EZ} to \\cref{ineq: convex thm 1 general}  and obtain\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: convex thm 2 general}\n\t\t\t\t\\mathbb{E} [\\eps_k]   \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{\\hat{L}D^3}{nk(k+1)}\\sum_{t=1}^{k}t~\\frac{134 n}{t+1} \\le \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{134 \\hat{L}D^3}{k+1} \\ , \n\t\t\t\\end{equation}\n\t\t\\end{small}which proves \\cref{eq of thm: general convergence rate convex}.\n\t\\end{proof}","units":[{"raw_tex":"\\begin{lemma}\\label{lm: general 3rd term upper bound}\n\t\tUnder \\cref{assump: ERM} for the problem $\\erm$, for any $u,v \\in\\calC$,\n\t\t\\begin{equation}\\label{eq: lm: general 3rd term upper bound}\n\t\t\t\\left(\n\t\t\t\\nabla F(x^k)   - g^k\n\t\t\t\\right)^\\top (u - v) \\le \\frac{\\hat{L}D^3}{2n}\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j \\Big)^2\n\t\t\\end{equation}\n\t\tholds for iteration $k$ of Algorithm \\ref{alg: TUFW in general finite sum} for all $k \\ge 1$.\n\t\\end{lemma}","start":64436,"end":64901},{"raw_tex":"\\begin{proof}[Proof of \\cref{lm: general 3rd term upper bound}]\n\t\tFirst of all, $\\left(\n\t\t\\nabla F(x^k)   - g^k\n\t\t\\right)^\\top (u - v)$ can be rewritten as\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: general etimate gradient 0}\n\t\t\t\t\\frac{1}{n}\\sum_{i=1}^{n}\\left(\\nabla f_{i}\\big(x^k \\big) -   \\nabla f_{i}\\big( x^{\\tau^{k}_i}\\big) -  \\nabla^2 f_{i} \\big( x^{\\tau^{k}_i}\\big)\\big(x^k  - x^{\\tau^{k}_i} \\big)    \\right)^\\top (u - v),\n\t\t\t\\end{equation}\t\n\t\t\\end{small}which, since $\\|u-v\\| \\le D$, is smaller than or equal to \n\t\t\\begin{small}\\begin{equation}\\label{ineq: general etimate gradient 1}\n\t\t\t\\frac{D}{n}\\sum_{i=1}^{n}\\left\\| \\nabla f_{i}\\big(x^k \\big) -   \\nabla f_{i}\\big( x^{\\tau^{k}_i}\\big) -  \\nabla^2 f_{i} \\big( x^{\\tau^{k}_i}\\big)\\big(x^k  - x^{\\tau^{k}_i} \\big)    \\right\\|_* \\le    \\frac{\\hat{L}D}{2n}\\sum_{i=1}^{n}\\big\\|x^k  - x^{\\tau^{k}_i}\\big\\|^2,\n\t\t\\end{equation}\\end{small}where the inequality is due to \\cref{eq: fact: general basic ineq 2} in \\cref{prop: general basic ineq}.\n\t\tAdditionally, we have \n\t\t\\begin{small}\\begin{equation*}\n\t\t\t\\|x^k  - x^{\\tau^{k}_i}\n\t\t\t\\|^2 \\le  \\bigg( \\sum_{j = \\tau^k_i}^{k-1} \\| x^{j+1} -  x^{j}\\| \\bigg)^2\\le  \\bigg( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j D \\bigg)^2 \\  ,\n\t\t\\end{equation*}\\end{small}and substituting this last bound into \\cref{ineq: general etimate gradient 1} yields \\cref{eq: lm: general 3rd term upper bound}.\n\t\\end{proof}","start":64907,"end":66306},{"raw_tex":"\\begin{lemma}\\label{lm: general convergence rate convex}\n\t\tUnder \\cref{assump: ERM} for problem $\\erm$, suppose that $F$ is conevx, and let the step-sizes be given by $\\gamma_k=2/(k+2)$ for all $k \\ge 0$. Then for all $k \\ge 1$ we have\n\t\t\\begin{equation}\\label{eq lm: general convergence rate convex}\n\t\t\t\\eps_k  \\le  \\frac{2L D^2}{k+1}\n\t\t\t+  \\frac{2}{k(k+1)}\\sum_{t=1}^{k} t \\left(\n\t\t\t\\nabla F(x^{t-1})   - g^{t-1}\n\t\t\t\\right)^\\top (s^{t-1} - x^\\star) \\ .\n\t\t\\end{equation}\n\t\\end{lemma}","start":66455,"end":66939},{"raw_tex":"\\begin{proof}[Proof of \\cref{lm: general convergence rate convex}]\n\t\tUtilizing  \\cref{prop: general basic ineq} we have\n\t\t\\begin{small}\\begin{equation}\\label{eq lm general convergence rate convex 0}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\tF(x^{k+1})\\leq ~ & F(x^k)+\\gamma_{k}\\langle\\nabla F(x^k),s^{k}- x^k \\rangle+ \\gamma_{k}^{2} {L D^2}/{2} \\\\\n\t\t\t\t\t= ~ & F(x^{k})+\\gamma_{k}\\langle \\nabla F(x^{k})-g^k, s^{k}-x^k\\rangle +\\gamma_{k}\\langle{g^k}, s^{k}-x^k\\rangle+ \\gamma_{k}^{2} {L D^2}/{2} \\\\\n\t\t\t\t\t\\leq ~ & F(x^{k})+\\gamma_{k}\\langle\\nabla F(x^{k})-{g^k}, s^{k}-x^k\\rangle + \\gamma_{k}\\langle{g^k}, {x}^{\\star}-x^k\\rangle+ \\gamma_{k}^{2} {L D^2}/{2} \\\\\n\t\t\t\t\t= ~ & F(x^{k})+\\gamma_{k}\\langle \\nabla F(x^{k})-{g^k},s^{k}-{x}^{\\star}\\rangle +\\gamma_{k}\\langle \\nabla F(x^{k}), {x}^{\\star}-x^k\\rangle+ \\gamma_{k}^{2} {L D^2}/{2}\\\\\n\t\t\t\t\t\\le ~& F(x^{k})+\\gamma_{k}\\langle \\nabla F(x^{k})-{g^k},s^{k}-{x}^{\\star}\\rangle +\\gamma_{k}(F(x^\\star) - F(x^k))+ \\gamma_{k}^{2} {L D^2}/{2} \\ , \n\t\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses $s^{k} \\in \\arg\\min_{s\\in\\calC} (g^k)^\\top s$ from \\cref{alex} of the algorithm, and the third inequality uses the gradient inequality applied to $F$.  Subtracting $F^\\star$ from both sides of the above inequality chain, we arrive at:\n\t\t\\begin{small}\\begin{equation}\\label{eq lm general convergence rate convex 1}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t\\eps_{k+1}& \\le (1-\\gamma_k)\\eps_k + \\gamma_k \\big(\n\t\t\t\t\t\\nabla F(x^k)   - g^k\n\t\t\t\t\t\\big)^\\top (s^{k} - x^\\star) +  {\\gamma_k^2 L D^2}/{2} \\ .\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}Multiplying both side by $(k+1)(k+2)$ and telescoping the inequalities yields:\n\t\t\\begin{small}\\begin{equation}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t(k+1)(k+2)\\eps_{k+1}  &\\le  2(k+1)L D^2\n\t\t\t\t\t+  \\sum_{t=0}^{k}2(t+1) (\n\t\t\t\t\t\\nabla F(x^t)   - g^t\n\t\t\t\t\t)^\\top (s^{t} - x^\\star) \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}which yields \\cref{eq lm: general convergence rate convex} after rearranging and re-indexing the counter $t$.\n\t\\end{proof}","start":66941,"end":68945},{"raw_tex":"\\begin{lemma}\\label{lm: convex EZ} Let the series $\\{d_{ij}\\}_{i\\in[n],j\\in\\mathbb{N}}$ be given, and suppose that there exist $M_{1}$ and $M_{\\infty}$ for which $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$ and  $\\sup_{i,j} d_{ij}\\le M_{\\infty}$.  \n\t\tLet $\\tau_i^k$ be defined in \\cref{def: tau}, in which $\\calB_t$ is constructed using $\\rone$ in \\cref{rule 1}. Let $\\gamma_k$ be ${2}/(k+2)$ for all $k \\ge 0$.  Then it holds that\n\t\t\\begin{equation}\\label{eq: lm: convex EZ}\n\t\t\t\\mathbb{E}\\bigg[\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\Big)\\bigg] \\le\n\t\t\t\\frac{6 M_1}{\\sqrt{k+2}} \\ \\ \\text{ and }   \\ \\\n\t\t\t\\mathbb{E} \\bigg[\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\Big)^2 \\bigg]\\le  \\frac{134M_1M_\\infty}{k+2} \\ .\n\t\t\\end{equation}\n\t\\end{lemma}","start":69084,"end":69867},{"raw_tex":"\\begin{proposition}\\label{lm: rho}\n\t\tFor integers $v$ and $k$ satisfying $1\\le  v \\le k$ it holds that:\n\t\t\\begin{equation}\\label{eq: fact 2}\n\t\t\t\\sum_{i=1}^{v}\\prod_{j=i}^{k}\\left(1-\\tfrac{1}{\\sqrt{j}}\\right) <\n\t\t\t(\\sqrt{v+1})\\exp(2\\sqrt{v+1} - 2\\sqrt{k+1})  \\   .\n\t\t\\end{equation}\n\t\\end{proposition}","start":69992,"end":70291},{"raw_tex":"\\begin{proof}[Proof of \\cref{lm: rho}]\n\t\tFor convenience let us define $\\rho_j := 1-\\tfrac{1}{\\sqrt{j}}$ for $j \\ge 1$, and define the function $h(x):= \\exp(2 \\sqrt{x} - 2 \\sqrt{k+1}) $ for $x \\ge 0$.  We will first prove the following inequality:\n\t\t\\begin{small}\\begin{equation}\\label{lulu}\n\t\t\t\\prod_{j=i}^{k}\\left(1-\\tfrac{1}{\\sqrt{j}}\\right) \\le h(i) \\ \\ \\ \\mathrm{for~all~} i \\in [k] \\ . \n\t\t\\end{equation}\\end{small}For $i=1$ the left side above is $0$ (because $\\rho_1 = 0$ and hence $\\prod_{j=1}^{k}\\rho_j = 0$), and the right side is $h(1) >0$, so \\cref{lulu} is true for $i=1$. For $i\\ge2$ we have:\n\t\t\\begin{small}\\begin{equation*}\\begin{aligned}\n\t\t\t\t\\prod_{j=i}^{k}\\rho_j  &= \\exp\\left({\\sum_{j=i}^{k}\\ln(1-\\tfrac{1}{\\sqrt{j}})}\\right) \\le \\exp\\left( -\\sum_{j=i}^{k}\\tfrac{1}{\\sqrt{j}}\\right)  \\\\  \n\t\t\t\t&\\le \\exp\\Big( -\\int_i^{k+1} x^{-\\tfrac{1}{2}} dx\\Big) =\\exp(2\\sqrt{i} - 2\\sqrt{k+1}) = h(i) \\ , \\end{aligned} \n\t\t\\end{equation*}\\end{small}where the first inequality follows from the fact that $\\log(1-x)\\le -x$ for any $0\\le x <1$, and the second inequality is an integral bound. This establishes \\cref{lulu}, whereby we have\n\t\t\\begin{small}$$ \\begin{aligned}\n\t\t\t\t\\sum_{i=1}^{v}   \\prod_{j=i}^{k}\\rho_j  &\\le  \\sum_{i=1}^{v} h(i) \\le \\int_{1}^{v+1}h(x)~dx = \\left. (\\sqrt{x}-\\tfrac{1}{2})\\exp(2\\sqrt{x} - 2\\sqrt{k+1}) \\right|_1^{v+1} \\\\ \n\t\t\t\t  &= (\\sqrt{v+1}-\\tfrac{1}{2})\\exp(2\\sqrt{v+1} - 2\\sqrt{k+1}) - (\\tfrac{1}{2})\\exp(2 - 2\\sqrt{k+1}) \\\\\\\n\t\t\t\t&< (\\sqrt{v+1}-\\tfrac{1}{2})\\exp(2\\sqrt{v+1} - 2\\sqrt{k+1}) \\ ,\n\t\t\\end{aligned}$$\\end{small}where the second inequality is an integral bound using the fact that $h(x)$ is an increasing function. \\end{proof}","start":70293,"end":71962},{"raw_tex":"\\begin{lemma}\\label{lm: simple ineq}\n\t\tFor any $k\\in\\mathbb{N}$ the following inequality holds:\n\t\t\\begin{equation}\\label{ineq: lm: simple ineq} \n\t\t\t\\sqrt{\\lfloor k/2 \\rfloor + 1} \\exp(2\\sqrt{\\lfloor k/2\\rfloor + 1} - 2\\sqrt{k+1}) \\le \\min\\{{3}(k+2)^{-\\frac{1}{2}},{10}(k+2)^{-1}\\}. \n\t\t\\end{equation}\n\t\\end{lemma}","start":72091,"end":72403},{"raw_tex":"\\begin{proof}[Proof of \\cref{lm: simple ineq}]\n\t\tSince $h(x):=\\sqrt{x}$ is a concave function on $\\mathbb{R}_+$, we have $h(k+2) \\le h(k+1) + h'(k+1)$, which yields $\\sqrt{k+1}\\ge \\sqrt{k+2} - \\frac{1}{2\\sqrt{k+1}} \\ge \\sqrt{k+2} - \\sqrt{2}/4 $ when $k \\ge 1$.\n\t\tFurthermore $\\sqrt{\\lfloor k/2 \\rfloor + 1} \\le \\frac{\\sqrt{2}}{2} \\sqrt{k+2}$, and therefore \n\t\t\\begin{small}\\begin{equation*}\n\t\t\t\\sqrt{\\lfloor k/2 \\rfloor + 1} \\exp(2\\sqrt{\\lfloor k/2\\rfloor + 1} - 2\\sqrt{k+1}) \\le  \\frac{\\sqrt{2}e^{\\frac{\\sqrt{2}}{2}}}{2}(k+2)^{\\frac{1}{2}}  \\exp(-(2-\\sqrt{2})(k+2)^{\\frac{1}{2}}) \\ .\n\t\t\\end{equation*}\\end{small}Next note that the function $h(x):=e^{-x} x^t$ on the domain $x \\in (0,\\infty)$ for $t > 0$ attains its maximal value at $x = t$. In particular for $t=2$ and $t=3$, respectively, we conclude that $e^{-x} \\le 4e^{-2}x^{-2}$ and $e^{-x} \\le 27e^{-3}x^{-3}$ (respectively), and thus  $\\exp(-(2-\\sqrt{2})(k+2)^{\\frac{1}{2}}) \\le 4 e^{-2}(2-\\sqrt{2})^{-2}(k+2)^{-1}$\n\t\tand $\\exp(-(2-\\sqrt{2})(k+2)^{\\frac{1}{2}}) \\le 27 e^{-3}(2-\\sqrt{2})^{-3}(k+2)^{-3/2}$ (respectively).\n\t\tTherefore, the left-hand side of \\cref{ineq: lm: simple ineq} is bounded by \n\t\t$2\\sqrt{2}e^{\\frac{\\sqrt{2}}{2}-2} (2-\\sqrt{2})^{-2} (k+2)^{-\\frac{1}{2}}$ and \n\t\t$\\frac{27}{2}\\sqrt{2}e^{\\frac{\\sqrt{2}}{2}-3} (2-\\sqrt{2})^{-3} (k+2)^{-1}$, respectively, which is less than $3(k+2)^{-\\frac{1}{2}}$ and $10(k+2)^{-1}$, respectively.\n\t\\end{proof}","start":72405,"end":73829},{"raw_tex":"\\begin{proof}[Proof of \\cref{lm: convex EZ}]\n\t\t\n\t\tFirst of all, let us define $G_k^i := \\left( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\right)$ and $Z_k^i :=\\left( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\right)^2$ for $i \\in [n]$, and also $G_k :=\\sum_{i=1}^n G_k^i$ and $Z_k :=\\sum_{i=1}^n Z_k^i$. Notice in $G_k^i$ and $Z_k^i$ that the randomness lies only in $\\tau_i^k$, which in turn depends on the update sets $\\calB_0, \\calB_1,\\dots,\\calB_k$. \n\t\t\n\t\tWe first analyze $G_k^i$ and $G_k$.  We note that $G_k^i = 0$ when $i \\in \\calB_k$, which happens with probability $\\beta_k/n$.  Because of the relationship between $\\{\\tau^k_i\\}$ and $\\{\\tau^{k-1}_i\\}$ in \\cref{eq: tauk relationship}, we further note that with probability $1-\\beta_k/n$ that $i \\notin \\calB_k$ in which case it follows that $G_k^i = G_{k-1}^i + \\gamma_{k-1}d_{i,k-1}$.  Defining $\\rho_k := 1 - \\frac{1}{\\sqrt{k}} = 1 - \\beta_k/n $, we have \n\t\t\\begin{small}\\begin{equation}\\label{ncl}\n\t\t\t\\mathbb{E}[ G_k ] = \\sum_{i=1}^n \\mathbb{E} [G_k^i ]= \\rho_k \\sum_{i=1}^n \\mathbb{E} [G_{k-1}^i +\n\t\t\t\\gamma_{k-1} d_{i,k-1}] \\le \\rho_k\\left( \\mathbb{E} [G_{k-1} ]+\n\t\t\t\\gamma_{k-1} M_1 \\right) \\ ,\n\t\t\\end{equation}\\end{small}where the inequality uses $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$. Recurrently applying \\cref{ncl} yields:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent G_k 2}\n\t\t\t \\mathbb{E}[ G_k ] \\le \\rho_k( \\mathbb{E}[ G_{k-1} ]+\n\t\t\t\\gamma_{k-1}  M_1 ) \\le\\cdots\\le \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\gamma_{i-1} M_1 + \\Big(\\prod_{j=1}^{k} \\rho_j\\Big)\\mathbb{E}[ G_0] \\ .\n\t\t\\end{equation}\\end{small}Noting that $G_0 = 0$ (since all Taylor points are set to $x^0$ in \\cref{bettina} of \\cref{alg: TUFW in general finite sum}) and using $\\gamma_{i-1} = \\frac{2}{i+1}$ it follows that:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent G_k 3}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\mathbb{E} [G_k] &\\le \\sum_{i=1}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{i+1}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor {k}/{2}\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{i+1}\\\\\n\t\t\t\t& \\le \\sum_{i=2}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{3}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor {k}/{2}\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor {k}/{2}\\rfloor+2} \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses the fact that $\\rho_1 = 0$ and hence $\\prod_{j=1}^{k}\\rho_j = 0$.\n\t\tBy \\cref{lm: rho} we have:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: G_k recurrent sumpro 1}\n\t\t\t\\sum_{i=1}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)  \\frac{2M_1}{3}\n\t\t\t\\le\n\t\t\t\\sqrt{\\lfloor {k}/{2}\n\t\t\t\t\\rfloor + 1} \\exp\\Big(2 \\sqrt{ \\lfloor {k}/{2} \\rfloor + 1} - 2\\sqrt{k+1}\\Big)  \\frac{2M_1}{3} \\le \\frac{2 M_1}{\\sqrt{k+2}} \\ ,\n\t\t\\end{equation}\\end{small}where the second inequality above uses \\cref{lm: simple ineq}.\n\t\tSimilarly for the second term in the rightmost side of \\cref{ineq: recurrent G_k 3} we have:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: G_k recurrent sumpro 2}\n\t\t\t\\sum_{i=\\lfloor k/2 \\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor k/2\\rfloor+2} \\le   \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor k/2\\rfloor+2} \\le \\frac{\\sqrt{k+1} \\cdot  2M_1}{\\lfloor k/2\\rfloor+2} \\le \\frac{4M_1}{\\sqrt{k+2}} \\ , \n\t\t\\end{equation}\\end{small}where the second inequality uses \\cref{lm: rho}.\n\t\tSubstituting \\cref{ineq: G_k recurrent sumpro 1} and \\cref{ineq: G_k recurrent sumpro 2} back into \\cref{ineq: recurrent G_k 3} yields the first part of \\cref{eq: lm: convex EZ}, namely:\n\t\t\\begin{equation}\\label{ineq: convex EG_k final}\n\t\t\t\\mathbb{E} [G_k] \\le {6 M_1}/{\\sqrt{k+2}} \\ .\n\t\t\\end{equation}\n\t\t\n\t\tWe employ a similar proof approach to establish the second part of \\cref{eq: lm: convex EZ}. We note that $Z_k^i = 0$ when $i \\in \\calB_k$, which happens with probability $\\beta_k/n$.  Because of the relationship between $\\{\\tau^k_i\\}$ and $\\{\\tau^{k-1}_i\\}$ in \\cref{eq: tauk relationship}, we further note that with probability $1-\\beta_k/n$ that $i \\notin \\calB_k$ in which case we have\n\t\t\\begin{small}\\begin{equation*}\n\t\t\t\\begin{aligned} \n\t\t\t\tZ_k^i &= (G_{k-1}^i + \\gamma_{k-1}d_{i,k-1})^2 = Z_{k-1}^i + 2 G_{k-1}^i\\gamma_{k-1}d_{i,k-1} + (\\gamma_{k-1}d_{i,k-1})^2 \\\\ &\\le Z_{k-1}^i + 2 M_\\infty \\gamma_{k-1}G_{k-1}^i + \\gamma_{k-1}^2 M_\\infty d_{i,k-1} \\ . \n\t\t\t\\end{aligned}\\end{equation*}\\end{small}Therefore\n\t\t\\begin{small}\\begin{equation}\\begin{aligned}\\label{nclz}\n\t\t\t\t\\mathbb{E}[ Z_k ] = &\\sum_{i=1}^n \\mathbb{E} [Z_k^i ] \\le \\rho_k \\sum_{i=1}^n \\mathbb{E} [Z_{k-1}^i + 2 M_\\infty \\gamma_{k-1}G_{k-1}^i + \\gamma_{k-1}^2 M_\\infty d_{i,k-1}] \\\\ \n\t\t\t\t&\\le \\rho_k \\left( \\mathbb{E} [Z_{k-1}] + 2   M_\\infty \\gamma_{k-1}\\mathbb{E}[G_{k-1}] +   \\gamma_{k-1}^2 M_\\infty M_1 \\right) \\\\\n\t\t\t\t&\\le \\rho_k \\left(\\mathbb{E} [Z_{k-1}] + \\frac{24M_\\infty M_1}{(k+1)\\sqrt{k+1}}   +  \\frac{4M_\\infty M_1}{(k+1)^2} \\right) \\le \\rho_k \\left(\\mathbb{E} [Z_{k-1}] + \\frac{28M_\\infty M_1}{(k+1)\\sqrt{k+1}}  \\right)    \\ ,\n\t\t\\end{aligned}\\end{equation} \\end{small}where the second inequality uses $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$, and the third inequality uses \\cref{ineq: convex EG_k final}. Recurrently applying \\cref{nclz} yields:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent Z_k 4}\n\t\t\t\\mathbb{E} [Z_k]  \\le \\rho_k\\bigg( \\mathbb{E} [Z_{k-1} ]+\n\t\t\t\\frac{28M_1M_\\infty}{(k+1)^{3/2}}\\bigg) \\le \\cdots\\le \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}  \\ .\n\t\t\\end{equation}\\end{small}Therefore\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent Z_k 5}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\mathbb{E} [Z_k]  & \\le  \\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor k/2\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}\\\\\n\t\t\t\t& \\le \\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{3\\sqrt{3}}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor k/2\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\ .\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}By \\cref{lm: rho} we have \n\t\t\t\\begin{small}\\begin{equation}\\label{ineq: Z_k recurrent sumpro 1}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{3\\sqrt{3}}  &\\le \\sqrt{\\lfloor k/2\n\t\t\t\t\t\\rfloor + 1} \\exp\\big(2 \\sqrt{ \\lfloor k/2 \\rfloor + 1} - 2\\sqrt{k+1}\\big) \\cdot \\frac{28M_1M_\\infty}{(3\\sqrt{3})} \\\\ \n\t\t\t\t&\\le \\frac{{54M_1M_\\infty}}{k+2} \\ , \n\t\t\\end{aligned}\\end{equation}\\end{small}where the second inequality above uses \\cref{lm: simple ineq}. \n\t\tSimilarly for the second term in the rightmost side of \\cref{ineq: recurrent Z_k 5} we have:\n\t\t\t\\begin{small}\\begin{equation}\n\t\t\t\\begin{aligned}\\label{ineq: Z_k recurrent sumpro 2}\n\t\t\t\t\\sum_{i=\\lfloor k/2 \\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} &\\le   \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\\\ \n\t\t\t\t&\\le \\sqrt{k+1} \\cdot \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\le \\frac{80M_1M_\\infty}{k+2} \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses \\cref{lm: rho}.\n\t\tSubstituting \\cref{ineq: Z_k recurrent sumpro 1} and \\cref{ineq: Z_k recurrent sumpro 2} back into \\cref{ineq: recurrent Z_k 5} yields the second part of \\cref{eq: lm: convex EZ}, namely:\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: convex EZ_k final}\n\t\t\t\t\\mathbb{E} [Z_k] \\le  \\frac{134M_1M_\\infty}{k+2} \\ .\n\t\t\t\\end{equation}\n\t\t\\end{small}\\end{proof}","start":73888,"end":81481},{"raw_tex":"\\begin{proof}[Proof of \\cref{thm: general convergence rate convex}]\n\t\tFrom \\cref{lm: general convergence rate convex} it holds that:\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\n\t\t\t\t\\eps_k  \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{2}{k(k+1)}\\sum_{t=1}^{k} t \\left(\n\t\t\t\t\\nabla F(x^{t-1})   - g^{t-1}\n\t\t\t\t\\right)^\\top (s^{t-1} - x^\\star) \\ ,\n\t\t\t\\end{equation}\n\t\t\\end{small}and from \\cref{lm: general 3rd term upper bound} it follows that:\n\t\t\\begin{small}\n\t\t\t\\begin{equation*}\n\t\t\t\t\\eps_k \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{\\hat{L}D^3}{nk(k+1)}\\sum_{t=1}^{k}t\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^{t-1}_i}^{t-2}  \\gamma_j \\Big)^2 \\ .\n\t\t\t\\end{equation*}\n\t\t\\end{small}Taking expectations on both sides then yields\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: convex thm 1 general}\n\t\t\t\t\\mathbb{E} [\\eps_k]  \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{\\hat{L}D^3}{nk(k+1)}\\sum_{t=1}^{k}t~\\mathbb{E}\\Big[\n\t\t\t\t\\sum_{i=1}^{n}\\Big(  \\sum_{j = \\tau^{t-1}_i}^{t-2}  \\gamma_j \\Big)^2\n\t\t\t\t\\Big] \\ .\n\t\t\t\\end{equation}\n\t\t\\end{small}Now define $d_{ij} := 1$ and $M_1 := n$ and $M_\\infty := 1$.  Then since $\\sup_{j}   \\sum_{i=1}^{n} d_{ij} = M_1$  and  $\\sup_{i,j} d_{ij}$ is equal to $M_\\infty$, we can apply \\cref{lm: convex EZ} to \\cref{ineq: convex thm 1 general}  and obtain\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: convex thm 2 general}\n\t\t\t\t\\mathbb{E} [\\eps_k]   \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{\\hat{L}D^3}{nk(k+1)}\\sum_{t=1}^{k}t~\\frac{134 n}{t+1} \\le \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{134 \\hat{L}D^3}{k+1} \\ , \n\t\t\t\\end{equation}\n\t\t\\end{small}which proves \\cref{eq of thm: general convergence rate convex}.\n\t\\end{proof}","start":81564,"end":83163}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["pro: ERM"],"declared_external":false,"source_only":false,"title":"有限和约束优化问题","source_units":[{"raw_tex":"\tOur problem of interest is the following Empirical Risk Minimization (ERM) problem (also called the \\textit{finite-sum} problem) with constraints: \n\t\\begin{equation}\\label{pro: ERM}\n\t\t\\text{ERM:} \\ \\ \\ \\ \\    \\text{minimize}_{x \\in \\calC} \\quad F(x):= \\tfrac{1}{n} \\sum_{i=1}^{n} f_i(x) \\ , \n\t\\end{equation}\n\twhere $f_i(\\cdot): \\mathbb{R}^p \\to \\mathbb{R}$ is the loss function of observation $i$ for $i=1,\\dots, n$, and $x$ is comprised of the (model) coefficients/parameters which are to be optimized over the feasibility set $\\calC \\subset \\mathbb{R}^p$. Problem \\cref{pro: ERM} is a fundamental problem in statistical and machine learning, as it is the underlying optimization problem on which models are ``trained'' or ``learned'', see  \\cite{vapnik1991principles,hastie2009elements}. The number of observations $n$ can be significantly larger than the dimension $p$, and our interest here will be in the regime where $n \\gg p$.\n\tWe make the following assumptions for $\\erm$ problems throughout this paper:\n","start":12615,"end":13628}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":["assump: ERM","eq: ERM ass lip 1","eq: ERM ass lip 2"],"declared_external":false,"source_only":false,"title":"通用ERM假设","source_units":[{"raw_tex":"\t\\begin{assumption}\\label{assump: ERM}\n\t\tThe following conditions hold for the $\\erm$ problem \\cref{pro: ERM}:\n\t\t\\begin{enumerate}\n\t\t\t\\item[(i)] the feasibility set $\\mathcal{C}$ is a compact convex set,\n\t\t\t\\item[(ii)] for any $g \\in \\mathbb{R}^p$, the linear minimization problem $\\arg\\min_{x\\in\\calC} g^\\top x$\n\t\t\tcan be easily solved, \n\t\t\t\\item[(iii)] for all $i\\in[n]$, the multivariate loss function $f_i(\\cdot)$ is twice-differentiable and its gradient $\\nabla f_i(\\cdot)$ is $L$-Lipschitz continuous on $\\calC$, namely, \n\t\t\t\\begin{equation}\\label{eq: ERM ass lip 1}\n\t\t\t\t\\| \\nabla f_i(u) - \\nabla f_i(v) \\|_* \\le L \\|u-v\\| ~~\\text{ for any $u,v\\in \\calC$}\n\t\t\t\\end{equation}\n\t\t\twhere $\\|\\cdot\\|_*$ is the dual norm of the norm on the space of variables, and \n\t\t\t\\item[(iv)] for all $i\\in[n]$, the Hessian $\\nabla^2 f_i(\\cdot)$ is $\\hat{L}$-Lipschitz continuous on $\\calC$, namely\n\t\t\t\\begin{equation}\\label{eq: ERM ass lip 2}\n\t\t\t\t\\| \\nabla^2 f_i(u) - \\nabla^2 f_i(v) \\| \\le \\hat{L} \\|u-v\\|~~\\text{ for any $u,v\\in \\calC$}\n\t\t\t\\end{equation}\n\t\t\twhere the norm of the Hessian matrices is the operator norm induced by the norm on the space of variables.\n\t\t\\end{enumerate}\n\t\\end{assumption}\n","start":13628,"end":14818}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","labels":["assump: ERM"],"declared_external":false,"source_only":false,"title":"可行域与线性优化","source_units":[{"raw_tex":"\t\t\t\\item[(i)] the feasibility set $\\mathcal{C}$ is a compact convex set,\n\t\t\t\\item[(ii)] for any $g \\in \\mathbb{R}^p$, the linear minimization problem $\\arg\\min_{x\\in\\calC} g^\\top x$\n\t\t\tcan be easily solved, \n","start":13759,"end":13967}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","labels":["assump: ERM","eq: ERM ass lip 1"],"declared_external":false,"source_only":false,"title":"梯度光滑性","source_units":[{"raw_tex":"\t\t\t\\item[(iii)] for all $i\\in[n]$, the multivariate loss function $f_i(\\cdot)$ is twice-differentiable and its gradient $\\nabla f_i(\\cdot)$ is $L$-Lipschitz continuous on $\\calC$, namely, \n\t\t\t\\begin{equation}\\label{eq: ERM ass lip 1}\n\t\t\t\t\\| \\nabla f_i(u) - \\nabla f_i(v) \\|_* \\le L \\|u-v\\| ~~\\text{ for any $u,v\\in \\calC$}\n\t\t\t\\end{equation}\n\t\t\twhere $\\|\\cdot\\|_*$ is the dual norm of the norm on the space of variables, and \n","start":13967,"end":14392}]},{"id":"S005","semantic_type":"assumption","kind":"assumption","labels":["assump: ERM","eq: ERM ass lip 2"],"declared_external":false,"source_only":false,"title":"Hessian光滑性","source_units":[{"raw_tex":"\t\t\t\\item[(iv)] for all $i\\in[n]$, the Hessian $\\nabla^2 f_i(\\cdot)$ is $\\hat{L}$-Lipschitz continuous on $\\calC$, namely\n\t\t\t\\begin{equation}\\label{eq: ERM ass lip 2}\n\t\t\t\t\\| \\nabla^2 f_i(u) - \\nabla^2 f_i(v) \\| \\le \\hat{L} \\|u-v\\|~~\\text{ for any $u,v\\in \\calC$}\n\t\t\t\\end{equation}\n\t\t\twhere the norm of the Hessian matrices is the operator norm induced by the norm on the space of variables.\n","start":14392,"end":14782}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"索引范数与概率记号","source_units":[{"raw_tex":"\t\\subsection{Notation}\n\t\n\tLet $\\mathbb{N}$ denote the natural numbers.  For any $n \\in \\mathbb{N}$, we use $[n]$ to denote the set $\\{1,2,\\dots,n\\}$.\n\tFor any real number $v$, we use $\\lfloor v \\rfloor$ to denote the largest integer number less than or equal to $v$.\n\tDepending on the context, we will use either $\\langle a,b\\rangle$ or $a^\\top b$ to denote the inner product between $a$ and $b$ in Euclidean space. For a vector $x$ in $\\mathbb{R}^p$, the norm $\\|x\\|_q$ is given by $(\\sum_{j=1}^{p}x_j^q)^{1/q}$ for $q \\in [1,\\infty)$ and $\\|x\\|_\\infty := \\max_{j} |x_j|$. The dual of norm $\\|\\cdot\\|$ is defined as $\\|z\\|_* := \\sup\\{z^\\top x : \\|x\\| \\le 1\\}$.  For a matrix $A \\in \\mathbb{R}^{p \\times p}$, the matrix operator norm $\\|A\\|$ induced by the norm $\\|\\cdot\\|$ is defined as $\\sup_{x: \\|x\\| \\le 1} \\|Ax\\|_*$. For any symmetric positive definite matrix $A\\in \\mathbb{S}^{p\\times p}$, we define its induced norm on $\\mathbb{R}^p$ by $\\|z\\|_A := \\sqrt{z^\\top A z}$.  We define $\\sum_{i=j}^k \\alpha_i := 0$ whenever $j > k$.  We denote the uniform distribution on the set $S$ by $\\calU(S)$, and we denote the Bernoulli distribution with probability $p$ by $\\mathrm{Ber}(p)$.\n","start":34310,"end":35494}]},{"id":"S007","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"TUFW方法介绍","source_units":[{"raw_tex":"\tIn this section we present a new type of Frank-Wolfe algorithm whose newness derives from the way in which approximate gradients are computed.  We call the algorithm TUFW for ``Taylor-point Updating Frank-Wolfe''. Algorithm TUFW is similar to the standard Frank-Wolfe method except that it computes an inexpensive estimate $g^k$ of the gradient $\\nabla F(x^k)$ at iteration $k$.  And it differs from other Frank-Wolfe methods in the way it computes the estimate $g^k$ in that (i) TUFW explicitly uses second-derivative (Hessian) information of the loss functions $f_i$, and (ii) it can use very different batch-size rules -- both stochastic and deterministic. TUFW is described in \\cref{alg: TUFW in general finite sum}.  We now explain the steps of the method.\n\t\n\t\n","start":35580,"end":36347}]},{"id":"S008","semantic_type":"algorithm","kind":"algorithm","labels":["alex","alg: TUFW in general finite sum","algline: 3","algline: 4","algline: 5","algline: 6","algline: 7","bettina","beverly","charlie","fanette","hotel"],"declared_external":false,"source_only":false,"title":"完整TUFW算法","source_units":[{"raw_tex":"\t\\begin{algorithm}\n\t\t\\caption{(TUFW) Frank-Wolfe Method with Taylor Point Updating}\n\t\t\\label{alg: TUFW in general finite sum}\n\t\t\\begin{algorithmic}[1]\n\t\t\t\\STATE \\textbf{input}  initial point $x^0$, step-size sequence $\\{\\gamma_k\\}_k$, and (possibly) number of iterations $K$\n\t\t\t\\STATE \\textbf{initialize Taylor points} $\\calB_0 \\gets [n]$ and $b_i \\leftarrow x^0$ for $i\\in [n]$  \\; \\label{bettina}\n\t\t\t\\FOR{$k=0,1,2,\\dots,K$,}\n\t\t\t\\STATE\\label{algline: 3}\\textbf{If $k>0$, update Taylor points}: \\\\ \\ \\ Use some \\textbf{Rule} to construct set $\\calB_k \\subset [n]$ of indices for Taylor-point updating\\\\ \\ \\ Update Taylor points:  for $i\\in\\calB_k$ update $b_i \\gets x^k$ \\; \\label{fanette}\n\t\t\t\\STATE\\label{algline: 4}  \\textbf{Compute estimates of individual gradients}:\\\\ $g^k_{i}\\gets \\nabla f_{i}(b_{i})  +  \\nabla^2 f_i (b_{i}) ( x^k -  b_{i} ) $ for $i\\in [n]$\\; \\label{charlie}\n\t\t\t\\STATE\\label{algline: 5}   \\textbf{Compute estimate of full gradient}: $g^k\\gets \\frac{1}{n}\\sum_{i=1}^{n}g_i^k$\\; \\label{beverly}\n\t\t\t\\STATE\\label{algline: 6}   \\textbf{Solve LMO}: $s^{k}\\gets \\arg\\min_{s\\in\\calC} \\langle g^k,s\\rangle$\\; \\label{alex} \n\t\t\t\\STATE\\label{algline: 7}   \\textbf{Update feasible solution}: $x^{k+1}\\gets x^k + \\gamma_k (s^{k} - x^k)$. \\label{hotel}\n\t\t\t\\ENDFOR\n\t\t\t\\RETURN $x^{K+1}$\n\t\t\\end{algorithmic}\n\t\\end{algorithm}\n","start":36347,"end":37679}]},{"id":"S009","semantic_type":"algorithm","kind":"algorithm","labels":["alex","alg: TUFW in general finite sum","algline: 3","algline: 4","algline: 5","algline: 6","algline: 7","bettina","beverly","charlie","fanette","hotel"],"declared_external":false,"source_only":false,"title":"Taylor点初始化与刷新","source_units":[{"raw_tex":"\t\t\t\\STATE \\textbf{input}  initial point $x^0$, step-size sequence $\\{\\gamma_k\\}_k$, and (possibly) number of iterations $K$\n\t\t\t\\STATE \\textbf{initialize Taylor points} $\\calB_0 \\gets [n]$ and $b_i \\leftarrow x^0$ for $i\\in [n]$  \\; \\label{bettina}\n\t\t\t\\FOR{$k=0,1,2,\\dots,K$,}\n\t\t\t\\STATE\\label{algline: 3}\\textbf{If $k>0$, update Taylor points}: \\\\ \\ \\ Use some \\textbf{Rule} to construct set $\\calB_k \\subset [n]$ of indices for Taylor-point updating\\\\ \\ \\ Update Taylor points:  for $i\\in\\calB_k$ update $b_i \\gets x^k$ \\; \\label{fanette}\n","start":36498,"end":37037}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","labels":["alex","alg: TUFW in general finite sum","algline: 3","algline: 4","algline: 5","algline: 6","algline: 7","bettina","beverly","charlie","fanette","hotel"],"declared_external":false,"source_only":false,"title":"Taylor梯度估计","source_units":[{"raw_tex":"\t\t\t\\STATE\\label{algline: 4}  \\textbf{Compute estimates of individual gradients}:\\\\ $g^k_{i}\\gets \\nabla f_{i}(b_{i})  +  \\nabla^2 f_i (b_{i}) ( x^k -  b_{i} ) $ for $i\\in [n]$\\; \\label{charlie}\n\t\t\t\\STATE\\label{algline: 5}   \\textbf{Compute estimate of full gradient}: $g^k\\gets \\frac{1}{n}\\sum_{i=1}^{n}g_i^k$\\; \\label{beverly}\n","start":37037,"end":37365}]},{"id":"S011","semantic_type":"algorithm","kind":"algorithm","labels":["alex","alg: TUFW in general finite sum","algline: 3","algline: 4","algline: 5","algline: 6","algline: 7","bettina","beverly","charlie","fanette","hotel"],"declared_external":false,"source_only":false,"title":"线性优化与迭代更新","source_units":[{"raw_tex":"\t\t\t\\STATE\\label{algline: 6}   \\textbf{Solve LMO}: $s^{k}\\gets \\arg\\min_{s\\in\\calC} \\langle g^k,s\\rangle$\\; \\label{alex} \n\t\t\t\\STATE\\label{algline: 7}   \\textbf{Update feasible solution}: $x^{k+1}\\gets x^k + \\gamma_k (s^{k} - x^k)$. \\label{hotel}\n","start":37365,"end":37610}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["rule 1"],"declared_external":false,"source_only":false,"title":"随机递减批量规则","source_units":[{"raw_tex":"\tRecall that in order to run \\cref{alg: TUFW in general finite sum} we need to specify the step-sizes $\\{\\gamma_k\\}_k$ and the {\\bf Rule} to construct the sets $\\calB_k$ of indices for Taylor-point updating in \\cref{fanette}.  Our first rule is designed \n\tfor $\\calB_k$ to be comprised of $\\beta_k := n /\\sqrt{k}$ independently drawn samples from $[n]$ without replacement, for all $k \\ge 1$.   Since $\\beta_k$ is not in general an integer, then rather than ensuring $|\\calB_k| = \\beta_k$, we will instead ensure that $\\mathbb{E}|\\calB_k| = \\beta_k$ by proceeding as follows.  \n\t\n\t\n\t\n\t\\begin{definition}\\label{rule 1} $\\rone$.  Define $\\beta_k := n /\\sqrt{k}$ and $p_k := \\beta_k - \\lfloor \\beta_k \\rfloor$.  Sample $\\xi_k$ from the Bernoulli distribution $\\mathrm{Ber}(p_k)$ and then uniformly sample $\\lfloor \\beta_k \\rfloor + \\xi_k$ samples from $[n]$ without replacement.  Then define $\\calB_k$ to be the set of these index samples.\\end{definition}\n","start":46870,"end":47823}]},{"id":"S013","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"期望批量大小","source_units":[{"raw_tex":"\t\n\tNote that by design of $\\rone$ it follows that $\\mathbb{E}_{\\xi_k} |\\calB_k| = \\beta_k$.  We call this rule ``$\\rone$'' for ``Stochastic Batch-size Decreasing at the rate $\\sqrt{k}$,'' and we emphasize that the Taylor points are updated less often as $k$ grows, which is in contrast to many other stochastic Frank-Wolfe methods that compute exact individual gradients more frequently as $k$ grows, see \\cite{hazan2016variance,yurtsever2019conditional}. \n","start":47823,"end":48280}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"可行域直径","source_units":[{"raw_tex":"\tLet $D$ denote the diameter of $\\calC$ under the norm of the variable space, namely\n\t$$\n\tD:= \\max_{u,v\\in\\calC} \\| u-v\\| \\ .\n\t$$\n","start":49654,"end":49784}]},{"id":"S015","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"最优解与最优性差距","source_units":[{"raw_tex":"\tIn the rest of the paper, we will use $x^\\star$ to denote an optimal solution of the $\\erm$ problem, and let $\\eps_k := F(x^k) - F(x^\\star)$. \n","start":62922,"end":63066}]},{"id":"S016","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单明确允许B0059作为不附文内证明的标准光滑性前提；这里只保留其原文陈述。"],"source_only":false,"title":"函数一阶展开余项","source_units":[{"raw_tex":"\tWe begin with the following straightforward properties of smooth functions.\n\t\\begin{proposition}\\label{prop: general basic ineq}\n\t\tUnder \\cref{assump: ERM} for $\\erm$,  for any $x$, $y\\in\\calC$ and $i$, \n\t\t\\begin{align}\n\t\t\t| f_i(y) -  f_i(x) - \\langle\\nabla f_i(x),  y- x\\rangle |\\le  L\\|x-y\\|^2 /2 \\ , \\text{ and } \\label{eq: fact: general basic ineq 1} \\\\\n","start":63068,"end":63427}]},{"id":"S017","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始前提清单明确允许B0059作为不附文内证明的标准光滑性前提。"],"source_only":false,"title":"梯度Taylor余项","source_units":[{"raw_tex":"\t\t\t\\| \\nabla f_i(y) - \\nabla f_i(x) - \\langle\\nabla^2 f_i(x),  y- x\\rangle \\|_*\\le  \\hat{L}\\|x-y\\|^2 /2 \\ . \\label{eq: fact: general basic ineq 2}\n\t\t\\end{align}\n\t\\end{proposition}\n","start":63427,"end":63607}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":["def: tau"],"declared_external":false,"source_only":false,"title":"最近刷新时刻","source_units":[{"raw_tex":"\tFor any $k \\ge 0$ and $i\\in [n]$, we will be interested in the most recent iteration (up through $k$) at which Taylor point $b_i$ was updated.  This is defined as:\n\t\\begin{equation}\\label{def: tau}\n\t\t\\tau^k_i := \\max\\{t: t\\le k  \\text{ and } i \\in \\calB_t\\} \\ ,\n\t\\end{equation}\n\twhich means that in the first $k$ iterations, $\\tau^k_i$ is the latest iteration in which the Taylor point $b_i$ was updated. In \\cref{alg: TUFW in general finite sum}, the relationship between $\\tau^k_i$ and $\\tau^{k-1}_i$ is then:\n","start":63609,"end":64122}]},{"id":"S019","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq: tauk relationship"],"declared_external":false,"source_only":false,"title":"刷新时刻递推","source_units":[{"raw_tex":"\t\\begin{equation}\\label{eq: tauk relationship}\n\t\t\\tau^{k}_i = \\left\\{\n\t\t\\begin{array}{ll}\n\t\t\tk & \\text{ if } \\  i\\in\\calB_k \\\\\n\t\t\t\\tau^{k-1}_i & \\text{ if } \\ i\\notin\\calB_k \\ . \n\t\t\\end{array}\n\t\t\\right.\n\t\\end{equation}\n","start":64122,"end":64341}]},{"id":"S020","semantic_type":"theorem","kind":"theorem","labels":["eq of thm: general convergence rate convex","thm: general convergence rate convex"],"declared_external":false,"source_only":false,"title":"随机TUFW凸收敛保证","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm: general convergence rate convex}\n\t\tSuppose that $F$ is convex and Assumption \\ref{assump: ERM} holds, and  \\cref{alg: TUFW in general finite sum} with $\\rone$ is applied to the problem \\cref{pro: ERM} with step-sizes defined by $\\gamma_k:={2}/({k+2})$ for all $k \\ge 0$. Then for all $k\\ge 1$ we have:\n\t\t\\begin{equation}\\label{eq of thm: general convergence rate convex}\n\t\t\t\\mathbb{E}[F(x^{k}) - F(x^\\star)] \\le  \\frac{2L D^2 + 134 \\hat{L}D^3}{k+1} \\ .\n\t\t\\end{equation} \n\t\\end{theorem}","start":50034,"end":50546}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","labels":["eq: lm: general 3rd term upper bound","lm: general 3rd term upper bound"],"declared_external":false,"source_only":false,"title":"Taylor梯度方向误差界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lm: general 3rd term upper bound}\n\t\tUnder \\cref{assump: ERM} for the problem $\\erm$, for any $u,v \\in\\calC$,\n\t\t\\begin{equation}\\label{eq: lm: general 3rd term upper bound}\n\t\t\t\\left(\n\t\t\t\\nabla F(x^k)   - g^k\n\t\t\t\\right)^\\top (u - v) \\le \\frac{\\hat{L}D^3}{2n}\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j \\Big)^2\n\t\t\\end{equation}\n\t\tholds for iteration $k$ of Algorithm \\ref{alg: TUFW in general finite sum} for all $k \\ge 1$.\n\t\\end{lemma}","start":64436,"end":64901}]},{"id":"S022","semantic_type":"proof","kind":"proof","labels":["ineq: general etimate gradient 0","ineq: general etimate gradient 1"],"declared_external":false,"source_only":false,"title":"梯度误差引理完整证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{lm: general 3rd term upper bound}]\n\t\tFirst of all, $\\left(\n\t\t\\nabla F(x^k)   - g^k\n\t\t\\right)^\\top (u - v)$ can be rewritten as\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: general etimate gradient 0}\n\t\t\t\t\\frac{1}{n}\\sum_{i=1}^{n}\\left(\\nabla f_{i}\\big(x^k \\big) -   \\nabla f_{i}\\big( x^{\\tau^{k}_i}\\big) -  \\nabla^2 f_{i} \\big( x^{\\tau^{k}_i}\\big)\\big(x^k  - x^{\\tau^{k}_i} \\big)    \\right)^\\top (u - v),\n\t\t\t\\end{equation}\t\n\t\t\\end{small}which, since $\\|u-v\\| \\le D$, is smaller than or equal to \n\t\t\\begin{small}\\begin{equation}\\label{ineq: general etimate gradient 1}\n\t\t\t\\frac{D}{n}\\sum_{i=1}^{n}\\left\\| \\nabla f_{i}\\big(x^k \\big) -   \\nabla f_{i}\\big( x^{\\tau^{k}_i}\\big) -  \\nabla^2 f_{i} \\big( x^{\\tau^{k}_i}\\big)\\big(x^k  - x^{\\tau^{k}_i} \\big)    \\right\\|_* \\le    \\frac{\\hat{L}D}{2n}\\sum_{i=1}^{n}\\big\\|x^k  - x^{\\tau^{k}_i}\\big\\|^2,\n\t\t\\end{equation}\\end{small}where the inequality is due to \\cref{eq: fact: general basic ineq 2} in \\cref{prop: general basic ineq}.\n\t\tAdditionally, we have \n\t\t\\begin{small}\\begin{equation*}\n\t\t\t\\|x^k  - x^{\\tau^{k}_i}\n\t\t\t\\|^2 \\le  \\bigg( \\sum_{j = \\tau^k_i}^{k-1} \\| x^{j+1} -  x^{j}\\| \\bigg)^2\\le  \\bigg( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j D \\bigg)^2 \\  ,\n\t\t\\end{equation*}\\end{small}and substituting this last bound into \\cref{ineq: general etimate gradient 1} yields \\cref{eq: lm: general 3rd term upper bound}.\n\t\\end{proof}","start":64907,"end":66306}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: general etimate gradient 0"],"declared_external":false,"source_only":false,"title":"误差的分量余项展开","source_units":[{"raw_tex":"\t\tFirst of all, $\\left(\n\t\t\\nabla F(x^k)   - g^k\n\t\t\\right)^\\top (u - v)$ can be rewritten as\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: general etimate gradient 0}\n\t\t\t\t\\frac{1}{n}\\sum_{i=1}^{n}\\left(\\nabla f_{i}\\big(x^k \\big) -   \\nabla f_{i}\\big( x^{\\tau^{k}_i}\\big) -  \\nabla^2 f_{i} \\big( x^{\\tau^{k}_i}\\big)\\big(x^k  - x^{\\tau^{k}_i} \\big)    \\right)^\\top (u - v),\n\t\t\t\\end{equation}\t\n","start":64971,"end":65363}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: general etimate gradient 1"],"declared_external":false,"source_only":false,"title":"余项的范数控制","source_units":[{"raw_tex":"\t\t\\end{small}which, since $\\|u-v\\| \\le D$, is smaller than or equal to \n\t\t\\begin{small}\\begin{equation}\\label{ineq: general etimate gradient 1}\n\t\t\t\\frac{D}{n}\\sum_{i=1}^{n}\\left\\| \\nabla f_{i}\\big(x^k \\big) -   \\nabla f_{i}\\big( x^{\\tau^{k}_i}\\big) -  \\nabla^2 f_{i} \\big( x^{\\tau^{k}_i}\\big)\\big(x^k  - x^{\\tau^{k}_i} \\big)    \\right\\|_* \\le    \\frac{\\hat{L}D}{2n}\\sum_{i=1}^{n}\\big\\|x^k  - x^{\\tau^{k}_i}\\big\\|^2,\n\t\t\\end{equation}\\end{small}where the inequality is due to \\cref{eq: fact: general basic ineq 2} in \\cref{prop: general basic ineq}.\n","start":65363,"end":65911}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"轨迹长度控制陈旧误差","source_units":[{"raw_tex":"\t\tAdditionally, we have \n\t\t\\begin{small}\\begin{equation*}\n\t\t\t\\|x^k  - x^{\\tau^{k}_i}\n\t\t\t\\|^2 \\le  \\bigg( \\sum_{j = \\tau^k_i}^{k-1} \\| x^{j+1} -  x^{j}\\| \\bigg)^2\\le  \\bigg( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j D \\bigg)^2 \\  ,\n\t\t\\end{equation*}\\end{small}and substituting this last bound into \\cref{ineq: general etimate gradient 1} yields \\cref{eq: lm: general 3rd term upper bound}.\n","start":65911,"end":66294}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","labels":["eq lm: general convergence rate convex","lm: general convergence rate convex"],"declared_external":false,"source_only":false,"title":"含梯度误差的凸收敛界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lm: general convergence rate convex}\n\t\tUnder \\cref{assump: ERM} for problem $\\erm$, suppose that $F$ is conevx, and let the step-sizes be given by $\\gamma_k=2/(k+2)$ for all $k \\ge 0$. Then for all $k \\ge 1$ we have\n\t\t\\begin{equation}\\label{eq lm: general convergence rate convex}\n\t\t\t\\eps_k  \\le  \\frac{2L D^2}{k+1}\n\t\t\t+  \\frac{2}{k(k+1)}\\sum_{t=1}^{k} t \\left(\n\t\t\t\\nabla F(x^{t-1})   - g^{t-1}\n\t\t\t\\right)^\\top (s^{t-1} - x^\\star) \\ .\n\t\t\\end{equation}\n\t\\end{lemma}","start":66455,"end":66939}]},{"id":"S027","semantic_type":"proof","kind":"proof","labels":["eq lm general convergence rate convex 0","eq lm general convergence rate convex 1"],"declared_external":false,"source_only":false,"title":"凸差距引理完整证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{lm: general convergence rate convex}]\n\t\tUtilizing  \\cref{prop: general basic ineq} we have\n\t\t\\begin{small}\\begin{equation}\\label{eq lm general convergence rate convex 0}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\tF(x^{k+1})\\leq ~ & F(x^k)+\\gamma_{k}\\langle\\nabla F(x^k),s^{k}- x^k \\rangle+ \\gamma_{k}^{2} {L D^2}/{2} \\\\\n\t\t\t\t\t= ~ & F(x^{k})+\\gamma_{k}\\langle \\nabla F(x^{k})-g^k, s^{k}-x^k\\rangle +\\gamma_{k}\\langle{g^k}, s^{k}-x^k\\rangle+ \\gamma_{k}^{2} {L D^2}/{2} \\\\\n\t\t\t\t\t\\leq ~ & F(x^{k})+\\gamma_{k}\\langle\\nabla F(x^{k})-{g^k}, s^{k}-x^k\\rangle + \\gamma_{k}\\langle{g^k}, {x}^{\\star}-x^k\\rangle+ \\gamma_{k}^{2} {L D^2}/{2} \\\\\n\t\t\t\t\t= ~ & F(x^{k})+\\gamma_{k}\\langle \\nabla F(x^{k})-{g^k},s^{k}-{x}^{\\star}\\rangle +\\gamma_{k}\\langle \\nabla F(x^{k}), {x}^{\\star}-x^k\\rangle+ \\gamma_{k}^{2} {L D^2}/{2}\\\\\n\t\t\t\t\t\\le ~& F(x^{k})+\\gamma_{k}\\langle \\nabla F(x^{k})-{g^k},s^{k}-{x}^{\\star}\\rangle +\\gamma_{k}(F(x^\\star) - F(x^k))+ \\gamma_{k}^{2} {L D^2}/{2} \\ , \n\t\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses $s^{k} \\in \\arg\\min_{s\\in\\calC} (g^k)^\\top s$ from \\cref{alex} of the algorithm, and the third inequality uses the gradient inequality applied to $F$.  Subtracting $F^\\star$ from both sides of the above inequality chain, we arrive at:\n\t\t\\begin{small}\\begin{equation}\\label{eq lm general convergence rate convex 1}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t\\eps_{k+1}& \\le (1-\\gamma_k)\\eps_k + \\gamma_k \\big(\n\t\t\t\t\t\\nabla F(x^k)   - g^k\n\t\t\t\t\t\\big)^\\top (s^{k} - x^\\star) +  {\\gamma_k^2 L D^2}/{2} \\ .\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}Multiplying both side by $(k+1)(k+2)$ and telescoping the inequalities yields:\n\t\t\\begin{small}\\begin{equation}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t(k+1)(k+2)\\eps_{k+1}  &\\le  2(k+1)L D^2\n\t\t\t\t\t+  \\sum_{t=0}^{k}2(t+1) (\n\t\t\t\t\t\\nabla F(x^t)   - g^t\n\t\t\t\t\t)^\\top (s^{t} - x^\\star) \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}which yields \\cref{eq lm: general convergence rate convex} after rearranging and re-indexing the counter $t$.\n\t\\end{proof}","start":66941,"end":68945}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":["eq lm general convergence rate convex 0"],"declared_external":false,"source_only":false,"title":"下降界与凸梯度比较","source_units":[{"raw_tex":"\t\tUtilizing  \\cref{prop: general basic ineq} we have\n\t\t\\begin{small}\\begin{equation}\\label{eq lm general convergence rate convex 0}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\tF(x^{k+1})\\leq ~ & F(x^k)+\\gamma_{k}\\langle\\nabla F(x^k),s^{k}- x^k \\rangle+ \\gamma_{k}^{2} {L D^2}/{2} \\\\\n\t\t\t\t\t= ~ & F(x^{k})+\\gamma_{k}\\langle \\nabla F(x^{k})-g^k, s^{k}-x^k\\rangle +\\gamma_{k}\\langle{g^k}, s^{k}-x^k\\rangle+ \\gamma_{k}^{2} {L D^2}/{2} \\\\\n\t\t\t\t\t\\leq ~ & F(x^{k})+\\gamma_{k}\\langle\\nabla F(x^{k})-{g^k}, s^{k}-x^k\\rangle + \\gamma_{k}\\langle{g^k}, {x}^{\\star}-x^k\\rangle+ \\gamma_{k}^{2} {L D^2}/{2} \\\\\n\t\t\t\t\t= ~ & F(x^{k})+\\gamma_{k}\\langle \\nabla F(x^{k})-{g^k},s^{k}-{x}^{\\star}\\rangle +\\gamma_{k}\\langle \\nabla F(x^{k}), {x}^{\\star}-x^k\\rangle+ \\gamma_{k}^{2} {L D^2}/{2}\\\\\n\t\t\t\t\t\\le ~& F(x^{k})+\\gamma_{k}\\langle \\nabla F(x^{k})-{g^k},s^{k}-{x}^{\\star}\\rangle +\\gamma_{k}(F(x^\\star) - F(x^k))+ \\gamma_{k}^{2} {L D^2}/{2} \\ , \n\t\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses $s^{k} \\in \\arg\\min_{s\\in\\calC} (g^k)^\\top s$ from \\cref{alex} of the algorithm, and the third inequality uses the gradient inequality applied to $F$.  Subtracting $F^\\star$ from both sides of the above inequality chain, we arrive at:\n","start":67008,"end":68220}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":["eq lm general convergence rate convex 1"],"declared_external":false,"source_only":false,"title":"最优性差距递推","source_units":[{"raw_tex":"\t\t\\begin{small}\\begin{equation}\\label{eq lm general convergence rate convex 1}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t\\eps_{k+1}& \\le (1-\\gamma_k)\\eps_k + \\gamma_k \\big(\n\t\t\t\t\t\\nabla F(x^k)   - g^k\n\t\t\t\t\t\\big)^\\top (s^{k} - x^\\star) +  {\\gamma_k^2 L D^2}/{2} \\ .\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}Multiplying both side by $(k+1)(k+2)$ and telescoping the inequalities yields:\n","start":68220,"end":68590}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"加权望远镜求和","source_units":[{"raw_tex":"\t\t\\begin{small}\\begin{equation}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t(k+1)(k+2)\\eps_{k+1}  &\\le  2(k+1)L D^2\n\t\t\t\t\t+  \\sum_{t=0}^{k}2(t+1) (\n\t\t\t\t\t\\nabla F(x^t)   - g^t\n\t\t\t\t\t)^\\top (s^{t} - x^\\star) \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}which yields \\cref{eq lm: general convergence rate convex} after rearranging and re-indexing the counter $t$.\n","start":68590,"end":68933}]},{"id":"S031","semantic_type":"lemma","kind":"lemma","labels":["eq: lm: convex EZ","lm: convex EZ"],"declared_external":false,"source_only":false,"title":"随机陈旧和的矩界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lm: convex EZ} Let the series $\\{d_{ij}\\}_{i\\in[n],j\\in\\mathbb{N}}$ be given, and suppose that there exist $M_{1}$ and $M_{\\infty}$ for which $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$ and  $\\sup_{i,j} d_{ij}\\le M_{\\infty}$.  \n\t\tLet $\\tau_i^k$ be defined in \\cref{def: tau}, in which $\\calB_t$ is constructed using $\\rone$ in \\cref{rule 1}. Let $\\gamma_k$ be ${2}/(k+2)$ for all $k \\ge 0$.  Then it holds that\n\t\t\\begin{equation}\\label{eq: lm: convex EZ}\n\t\t\t\\mathbb{E}\\bigg[\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\Big)\\bigg] \\le\n\t\t\t\\frac{6 M_1}{\\sqrt{k+2}} \\ \\ \\text{ and }   \\ \\\n\t\t\t\\mathbb{E} \\bigg[\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\Big)^2 \\bigg]\\le  \\frac{134M_1M_\\infty}{k+2} \\ .\n\t\t\\end{equation}\n\t\\end{lemma}","start":69084,"end":69867}]},{"id":"S032","semantic_type":"assumption","kind":"assumption","labels":["lm: convex EZ"],"declared_external":false,"source_only":false,"title":"矩界引理的显式条件","source_units":[{"raw_tex":"\\begin{lemma}\\label{lm: convex EZ} Let the series $\\{d_{ij}\\}_{i\\in[n],j\\in\\mathbb{N}}$ be given, and suppose that there exist $M_{1}$ and $M_{\\infty}$ for which $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$ and  $\\sup_{i,j} d_{ij}\\le M_{\\infty}$.  \n\t\tLet $\\tau_i^k$ be defined in \\cref{def: tau}, in which $\\calB_t$ is constructed using $\\rone$ in \\cref{rule 1}. Let $\\gamma_k$ be ${2}/(k+2)$ for all $k \\ge 0$.  Then it holds that\n","start":69084,"end":69512}]},{"id":"S033","semantic_type":"lemma","kind":"lemma","labels":["lm: convex EZ"],"declared_external":false,"source_only":false,"title":"陈旧和一阶矩界","source_units":[{"raw_tex":"\t\t\\begin{equation}\\label{eq: lm: convex EZ}\n\t\t\t\\mathbb{E}\\bigg[\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\Big)\\bigg] \\le\n\t\t\t\\frac{6 M_1}{\\sqrt{k+2}} \\ \\ \\text{ and }   \\ \\\n","start":69512,"end":69705}]},{"id":"S034","semantic_type":"lemma","kind":"lemma","labels":["lm: convex EZ"],"declared_external":false,"source_only":false,"title":"陈旧和平方矩界","source_units":[{"raw_tex":"\t\t\t\\mathbb{E} \\bigg[\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\Big)^2 \\bigg]\\le  \\frac{134M_1M_\\infty}{k+2} \\ .\n\t\t\\end{equation}\n\t\\end{lemma}","start":69705,"end":69867}]},{"id":"S035","semantic_type":"proposition","kind":"proposition","labels":["eq: fact 2","lm: rho"],"declared_external":false,"source_only":false,"title":"刷新存活乘积求和界","source_units":[{"raw_tex":"\\begin{proposition}\\label{lm: rho}\n\t\tFor integers $v$ and $k$ satisfying $1\\le  v \\le k$ it holds that:\n\t\t\\begin{equation}\\label{eq: fact 2}\n\t\t\t\\sum_{i=1}^{v}\\prod_{j=i}^{k}\\left(1-\\tfrac{1}{\\sqrt{j}}\\right) <\n\t\t\t(\\sqrt{v+1})\\exp(2\\sqrt{v+1} - 2\\sqrt{k+1})  \\   .\n\t\t\\end{equation}\n\t\\end{proposition}","start":69992,"end":70291}]},{"id":"S036","semantic_type":"proof","kind":"proof","labels":["lulu"],"declared_external":false,"source_only":false,"title":"乘积求和界完整证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{lm: rho}]\n\t\tFor convenience let us define $\\rho_j := 1-\\tfrac{1}{\\sqrt{j}}$ for $j \\ge 1$, and define the function $h(x):= \\exp(2 \\sqrt{x} - 2 \\sqrt{k+1}) $ for $x \\ge 0$.  We will first prove the following inequality:\n\t\t\\begin{small}\\begin{equation}\\label{lulu}\n\t\t\t\\prod_{j=i}^{k}\\left(1-\\tfrac{1}{\\sqrt{j}}\\right) \\le h(i) \\ \\ \\ \\mathrm{for~all~} i \\in [k] \\ . \n\t\t\\end{equation}\\end{small}For $i=1$ the left side above is $0$ (because $\\rho_1 = 0$ and hence $\\prod_{j=1}^{k}\\rho_j = 0$), and the right side is $h(1) >0$, so \\cref{lulu} is true for $i=1$. For $i\\ge2$ we have:\n\t\t\\begin{small}\\begin{equation*}\\begin{aligned}\n\t\t\t\t\\prod_{j=i}^{k}\\rho_j  &= \\exp\\left({\\sum_{j=i}^{k}\\ln(1-\\tfrac{1}{\\sqrt{j}})}\\right) \\le \\exp\\left( -\\sum_{j=i}^{k}\\tfrac{1}{\\sqrt{j}}\\right)  \\\\  \n\t\t\t\t&\\le \\exp\\Big( -\\int_i^{k+1} x^{-\\tfrac{1}{2}} dx\\Big) =\\exp(2\\sqrt{i} - 2\\sqrt{k+1}) = h(i) \\ , \\end{aligned} \n\t\t\\end{equation*}\\end{small}where the first inequality follows from the fact that $\\log(1-x)\\le -x$ for any $0\\le x <1$, and the second inequality is an integral bound. This establishes \\cref{lulu}, whereby we have\n\t\t\\begin{small}$$ \\begin{aligned}\n\t\t\t\t\\sum_{i=1}^{v}   \\prod_{j=i}^{k}\\rho_j  &\\le  \\sum_{i=1}^{v} h(i) \\le \\int_{1}^{v+1}h(x)~dx = \\left. (\\sqrt{x}-\\tfrac{1}{2})\\exp(2\\sqrt{x} - 2\\sqrt{k+1}) \\right|_1^{v+1} \\\\ \n\t\t\t\t  &= (\\sqrt{v+1}-\\tfrac{1}{2})\\exp(2\\sqrt{v+1} - 2\\sqrt{k+1}) - (\\tfrac{1}{2})\\exp(2 - 2\\sqrt{k+1}) \\\\\\\n\t\t\t\t&< (\\sqrt{v+1}-\\tfrac{1}{2})\\exp(2\\sqrt{v+1} - 2\\sqrt{k+1}) \\ ,\n\t\t\\end{aligned}$$\\end{small}where the second inequality is an integral bound using the fact that $h(x)$ is an increasing function. \\end{proof}","start":70293,"end":71962}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":["lulu"],"declared_external":false,"source_only":false,"title":"单个存活乘积指数界","source_units":[{"raw_tex":"\t\tFor convenience let us define $\\rho_j := 1-\\tfrac{1}{\\sqrt{j}}$ for $j \\ge 1$, and define the function $h(x):= \\exp(2 \\sqrt{x} - 2 \\sqrt{k+1}) $ for $x \\ge 0$.  We will first prove the following inequality:\n\t\t\\begin{small}\\begin{equation}\\label{lulu}\n\t\t\t\\prod_{j=i}^{k}\\left(1-\\tfrac{1}{\\sqrt{j}}\\right) \\le h(i) \\ \\ \\ \\mathrm{for~all~} i \\in [k] \\ . \n\t\t\\end{equation}\\end{small}For $i=1$ the left side above is $0$ (because $\\rho_1 = 0$ and hence $\\prod_{j=1}^{k}\\rho_j = 0$), and the right side is $h(1) >0$, so \\cref{lulu} is true for $i=1$. For $i\\ge2$ we have:\n\t\t\\begin{small}\\begin{equation*}\\begin{aligned}\n\t\t\t\t\\prod_{j=i}^{k}\\rho_j  &= \\exp\\left({\\sum_{j=i}^{k}\\ln(1-\\tfrac{1}{\\sqrt{j}})}\\right) \\le \\exp\\left( -\\sum_{j=i}^{k}\\tfrac{1}{\\sqrt{j}}\\right)  \\\\  \n\t\t\t\t&\\le \\exp\\Big( -\\int_i^{k+1} x^{-\\tfrac{1}{2}} dx\\Big) =\\exp(2\\sqrt{i} - 2\\sqrt{k+1}) = h(i) \\ , \\end{aligned} \n\t\t\\end{equation*}\\end{small}where the first inequality follows from the fact that $\\log(1-x)\\le -x$ for any $0\\le x <1$, and the second inequality is an integral bound. This establishes \\cref{lulu}, whereby we have\n","start":70332,"end":71432}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"指数函数求和积分界","source_units":[{"raw_tex":"\t\t\\begin{small}$$ \\begin{aligned}\n\t\t\t\t\\sum_{i=1}^{v}   \\prod_{j=i}^{k}\\rho_j  &\\le  \\sum_{i=1}^{v} h(i) \\le \\int_{1}^{v+1}h(x)~dx = \\left. (\\sqrt{x}-\\tfrac{1}{2})\\exp(2\\sqrt{x} - 2\\sqrt{k+1}) \\right|_1^{v+1} \\\\ \n\t\t\t\t  &= (\\sqrt{v+1}-\\tfrac{1}{2})\\exp(2\\sqrt{v+1} - 2\\sqrt{k+1}) - (\\tfrac{1}{2})\\exp(2 - 2\\sqrt{k+1}) \\\\\\\n\t\t\t\t&< (\\sqrt{v+1}-\\tfrac{1}{2})\\exp(2\\sqrt{v+1} - 2\\sqrt{k+1}) \\ ,\n\t\t\\end{aligned}$$\\end{small}where the second inequality is an integral bound using the fact that $h(x)$ is an increasing function. \\end{proof}","start":71432,"end":71962}]},{"id":"S039","semantic_type":"lemma","kind":"lemma","labels":["ineq: lm: simple ineq","lm: simple ineq"],"declared_external":false,"source_only":false,"title":"半程指数项多项式界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lm: simple ineq}\n\t\tFor any $k\\in\\mathbb{N}$ the following inequality holds:\n\t\t\\begin{equation}\\label{ineq: lm: simple ineq} \n\t\t\t\\sqrt{\\lfloor k/2 \\rfloor + 1} \\exp(2\\sqrt{\\lfloor k/2\\rfloor + 1} - 2\\sqrt{k+1}) \\le \\min\\{{3}(k+2)^{-\\frac{1}{2}},{10}(k+2)^{-1}\\}. \n\t\t\\end{equation}\n\t\\end{lemma}","start":72091,"end":72403}]},{"id":"S040","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"半程项引理完整证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{lm: simple ineq}]\n\t\tSince $h(x):=\\sqrt{x}$ is a concave function on $\\mathbb{R}_+$, we have $h(k+2) \\le h(k+1) + h'(k+1)$, which yields $\\sqrt{k+1}\\ge \\sqrt{k+2} - \\frac{1}{2\\sqrt{k+1}} \\ge \\sqrt{k+2} - \\sqrt{2}/4 $ when $k \\ge 1$.\n\t\tFurthermore $\\sqrt{\\lfloor k/2 \\rfloor + 1} \\le \\frac{\\sqrt{2}}{2} \\sqrt{k+2}$, and therefore \n\t\t\\begin{small}\\begin{equation*}\n\t\t\t\\sqrt{\\lfloor k/2 \\rfloor + 1} \\exp(2\\sqrt{\\lfloor k/2\\rfloor + 1} - 2\\sqrt{k+1}) \\le  \\frac{\\sqrt{2}e^{\\frac{\\sqrt{2}}{2}}}{2}(k+2)^{\\frac{1}{2}}  \\exp(-(2-\\sqrt{2})(k+2)^{\\frac{1}{2}}) \\ .\n\t\t\\end{equation*}\\end{small}Next note that the function $h(x):=e^{-x} x^t$ on the domain $x \\in (0,\\infty)$ for $t > 0$ attains its maximal value at $x = t$. In particular for $t=2$ and $t=3$, respectively, we conclude that $e^{-x} \\le 4e^{-2}x^{-2}$ and $e^{-x} \\le 27e^{-3}x^{-3}$ (respectively), and thus  $\\exp(-(2-\\sqrt{2})(k+2)^{\\frac{1}{2}}) \\le 4 e^{-2}(2-\\sqrt{2})^{-2}(k+2)^{-1}$\n\t\tand $\\exp(-(2-\\sqrt{2})(k+2)^{\\frac{1}{2}}) \\le 27 e^{-3}(2-\\sqrt{2})^{-3}(k+2)^{-3/2}$ (respectively).\n\t\tTherefore, the left-hand side of \\cref{ineq: lm: simple ineq} is bounded by \n\t\t$2\\sqrt{2}e^{\\frac{\\sqrt{2}}{2}-2} (2-\\sqrt{2})^{-2} (k+2)^{-\\frac{1}{2}}$ and \n\t\t$\\frac{27}{2}\\sqrt{2}e^{\\frac{\\sqrt{2}}{2}-3} (2-\\sqrt{2})^{-3} (k+2)^{-1}$, respectively, which is less than $3(k+2)^{-\\frac{1}{2}}$ and $10(k+2)^{-1}$, respectively.\n\t\\end{proof}","start":72405,"end":73829}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"平方根凹性归一化","source_units":[{"raw_tex":"\t\tSince $h(x):=\\sqrt{x}$ is a concave function on $\\mathbb{R}_+$, we have $h(k+2) \\le h(k+1) + h'(k+1)$, which yields $\\sqrt{k+1}\\ge \\sqrt{k+2} - \\frac{1}{2\\sqrt{k+1}} \\ge \\sqrt{k+2} - \\sqrt{2}/4 $ when $k \\ge 1$.\n\t\tFurthermore $\\sqrt{\\lfloor k/2 \\rfloor + 1} \\le \\frac{\\sqrt{2}}{2} \\sqrt{k+2}$, and therefore \n\t\t\\begin{small}\\begin{equation*}\n\t\t\t\\sqrt{\\lfloor k/2 \\rfloor + 1} \\exp(2\\sqrt{\\lfloor k/2\\rfloor + 1} - 2\\sqrt{k+1}) \\le  \\frac{\\sqrt{2}e^{\\frac{\\sqrt{2}}{2}}}{2}(k+2)^{\\frac{1}{2}}  \\exp(-(2-\\sqrt{2})(k+2)^{\\frac{1}{2}}) \\ .\n","start":72452,"end":72990}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"指数衰减转为幂次界","source_units":[{"raw_tex":"\t\t\\end{equation*}\\end{small}Next note that the function $h(x):=e^{-x} x^t$ on the domain $x \\in (0,\\infty)$ for $t > 0$ attains its maximal value at $x = t$. In particular for $t=2$ and $t=3$, respectively, we conclude that $e^{-x} \\le 4e^{-2}x^{-2}$ and $e^{-x} \\le 27e^{-3}x^{-3}$ (respectively), and thus  $\\exp(-(2-\\sqrt{2})(k+2)^{\\frac{1}{2}}) \\le 4 e^{-2}(2-\\sqrt{2})^{-2}(k+2)^{-1}$\n\t\tand $\\exp(-(2-\\sqrt{2})(k+2)^{\\frac{1}{2}}) \\le 27 e^{-3}(2-\\sqrt{2})^{-3}(k+2)^{-3/2}$ (respectively).\n\t\tTherefore, the left-hand side of \\cref{ineq: lm: simple ineq} is bounded by \n\t\t$2\\sqrt{2}e^{\\frac{\\sqrt{2}}{2}-2} (2-\\sqrt{2})^{-2} (k+2)^{-\\frac{1}{2}}$ and \n\t\t$\\frac{27}{2}\\sqrt{2}e^{\\frac{\\sqrt{2}}{2}-3} (2-\\sqrt{2})^{-3} (k+2)^{-1}$, respectively, which is less than $3(k+2)^{-\\frac{1}{2}}$ and $10(k+2)^{-1}$, respectively.\n","start":72990,"end":73817}]},{"id":"S043","semantic_type":"proof","kind":"proof","labels":["ineq: G_k recurrent sumpro 1","ineq: G_k recurrent sumpro 2","ineq: Z_k recurrent sumpro 1","ineq: Z_k recurrent sumpro 2","ineq: convex EG_k final","ineq: convex EZ_k final","ineq: recurrent G_k 2","ineq: recurrent G_k 3","ineq: recurrent Z_k 4","ineq: recurrent Z_k 5","ncl","nclz"],"declared_external":false,"source_only":false,"title":"随机矩界引理完整证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{lm: convex EZ}]\n\t\t\n\t\tFirst of all, let us define $G_k^i := \\left( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\right)$ and $Z_k^i :=\\left( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\right)^2$ for $i \\in [n]$, and also $G_k :=\\sum_{i=1}^n G_k^i$ and $Z_k :=\\sum_{i=1}^n Z_k^i$. Notice in $G_k^i$ and $Z_k^i$ that the randomness lies only in $\\tau_i^k$, which in turn depends on the update sets $\\calB_0, \\calB_1,\\dots,\\calB_k$. \n\t\t\n\t\tWe first analyze $G_k^i$ and $G_k$.  We note that $G_k^i = 0$ when $i \\in \\calB_k$, which happens with probability $\\beta_k/n$.  Because of the relationship between $\\{\\tau^k_i\\}$ and $\\{\\tau^{k-1}_i\\}$ in \\cref{eq: tauk relationship}, we further note that with probability $1-\\beta_k/n$ that $i \\notin \\calB_k$ in which case it follows that $G_k^i = G_{k-1}^i + \\gamma_{k-1}d_{i,k-1}$.  Defining $\\rho_k := 1 - \\frac{1}{\\sqrt{k}} = 1 - \\beta_k/n $, we have \n\t\t\\begin{small}\\begin{equation}\\label{ncl}\n\t\t\t\\mathbb{E}[ G_k ] = \\sum_{i=1}^n \\mathbb{E} [G_k^i ]= \\rho_k \\sum_{i=1}^n \\mathbb{E} [G_{k-1}^i +\n\t\t\t\\gamma_{k-1} d_{i,k-1}] \\le \\rho_k\\left( \\mathbb{E} [G_{k-1} ]+\n\t\t\t\\gamma_{k-1} M_1 \\right) \\ ,\n\t\t\\end{equation}\\end{small}where the inequality uses $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$. Recurrently applying \\cref{ncl} yields:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent G_k 2}\n\t\t\t \\mathbb{E}[ G_k ] \\le \\rho_k( \\mathbb{E}[ G_{k-1} ]+\n\t\t\t\\gamma_{k-1}  M_1 ) \\le\\cdots\\le \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\gamma_{i-1} M_1 + \\Big(\\prod_{j=1}^{k} \\rho_j\\Big)\\mathbb{E}[ G_0] \\ .\n\t\t\\end{equation}\\end{small}Noting that $G_0 = 0$ (since all Taylor points are set to $x^0$ in \\cref{bettina} of \\cref{alg: TUFW in general finite sum}) and using $\\gamma_{i-1} = \\frac{2}{i+1}$ it follows that:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent G_k 3}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\mathbb{E} [G_k] &\\le \\sum_{i=1}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{i+1}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor {k}/{2}\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{i+1}\\\\\n\t\t\t\t& \\le \\sum_{i=2}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{3}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor {k}/{2}\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor {k}/{2}\\rfloor+2} \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses the fact that $\\rho_1 = 0$ and hence $\\prod_{j=1}^{k}\\rho_j = 0$.\n\t\tBy \\cref{lm: rho} we have:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: G_k recurrent sumpro 1}\n\t\t\t\\sum_{i=1}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)  \\frac{2M_1}{3}\n\t\t\t\\le\n\t\t\t\\sqrt{\\lfloor {k}/{2}\n\t\t\t\t\\rfloor + 1} \\exp\\Big(2 \\sqrt{ \\lfloor {k}/{2} \\rfloor + 1} - 2\\sqrt{k+1}\\Big)  \\frac{2M_1}{3} \\le \\frac{2 M_1}{\\sqrt{k+2}} \\ ,\n\t\t\\end{equation}\\end{small}where the second inequality above uses \\cref{lm: simple ineq}.\n\t\tSimilarly for the second term in the rightmost side of \\cref{ineq: recurrent G_k 3} we have:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: G_k recurrent sumpro 2}\n\t\t\t\\sum_{i=\\lfloor k/2 \\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor k/2\\rfloor+2} \\le   \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor k/2\\rfloor+2} \\le \\frac{\\sqrt{k+1} \\cdot  2M_1}{\\lfloor k/2\\rfloor+2} \\le \\frac{4M_1}{\\sqrt{k+2}} \\ , \n\t\t\\end{equation}\\end{small}where the second inequality uses \\cref{lm: rho}.\n\t\tSubstituting \\cref{ineq: G_k recurrent sumpro 1} and \\cref{ineq: G_k recurrent sumpro 2} back into \\cref{ineq: recurrent G_k 3} yields the first part of \\cref{eq: lm: convex EZ}, namely:\n\t\t\\begin{equation}\\label{ineq: convex EG_k final}\n\t\t\t\\mathbb{E} [G_k] \\le {6 M_1}/{\\sqrt{k+2}} \\ .\n\t\t\\end{equation}\n\t\t\n\t\tWe employ a similar proof approach to establish the second part of \\cref{eq: lm: convex EZ}. We note that $Z_k^i = 0$ when $i \\in \\calB_k$, which happens with probability $\\beta_k/n$.  Because of the relationship between $\\{\\tau^k_i\\}$ and $\\{\\tau^{k-1}_i\\}$ in \\cref{eq: tauk relationship}, we further note that with probability $1-\\beta_k/n$ that $i \\notin \\calB_k$ in which case we have\n\t\t\\begin{small}\\begin{equation*}\n\t\t\t\\begin{aligned} \n\t\t\t\tZ_k^i &= (G_{k-1}^i + \\gamma_{k-1}d_{i,k-1})^2 = Z_{k-1}^i + 2 G_{k-1}^i\\gamma_{k-1}d_{i,k-1} + (\\gamma_{k-1}d_{i,k-1})^2 \\\\ &\\le Z_{k-1}^i + 2 M_\\infty \\gamma_{k-1}G_{k-1}^i + \\gamma_{k-1}^2 M_\\infty d_{i,k-1} \\ . \n\t\t\t\\end{aligned}\\end{equation*}\\end{small}Therefore\n\t\t\\begin{small}\\begin{equation}\\begin{aligned}\\label{nclz}\n\t\t\t\t\\mathbb{E}[ Z_k ] = &\\sum_{i=1}^n \\mathbb{E} [Z_k^i ] \\le \\rho_k \\sum_{i=1}^n \\mathbb{E} [Z_{k-1}^i + 2 M_\\infty \\gamma_{k-1}G_{k-1}^i + \\gamma_{k-1}^2 M_\\infty d_{i,k-1}] \\\\ \n\t\t\t\t&\\le \\rho_k \\left( \\mathbb{E} [Z_{k-1}] + 2   M_\\infty \\gamma_{k-1}\\mathbb{E}[G_{k-1}] +   \\gamma_{k-1}^2 M_\\infty M_1 \\right) \\\\\n\t\t\t\t&\\le \\rho_k \\left(\\mathbb{E} [Z_{k-1}] + \\frac{24M_\\infty M_1}{(k+1)\\sqrt{k+1}}   +  \\frac{4M_\\infty M_1}{(k+1)^2} \\right) \\le \\rho_k \\left(\\mathbb{E} [Z_{k-1}] + \\frac{28M_\\infty M_1}{(k+1)\\sqrt{k+1}}  \\right)    \\ ,\n\t\t\\end{aligned}\\end{equation} \\end{small}where the second inequality uses $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$, and the third inequality uses \\cref{ineq: convex EG_k final}. Recurrently applying \\cref{nclz} yields:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent Z_k 4}\n\t\t\t\\mathbb{E} [Z_k]  \\le \\rho_k\\bigg( \\mathbb{E} [Z_{k-1} ]+\n\t\t\t\\frac{28M_1M_\\infty}{(k+1)^{3/2}}\\bigg) \\le \\cdots\\le \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}  \\ .\n\t\t\\end{equation}\\end{small}Therefore\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent Z_k 5}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\mathbb{E} [Z_k]  & \\le  \\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor k/2\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}\\\\\n\t\t\t\t& \\le \\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{3\\sqrt{3}}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor k/2\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\ .\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}By \\cref{lm: rho} we have \n\t\t\t\\begin{small}\\begin{equation}\\label{ineq: Z_k recurrent sumpro 1}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{3\\sqrt{3}}  &\\le \\sqrt{\\lfloor k/2\n\t\t\t\t\t\\rfloor + 1} \\exp\\big(2 \\sqrt{ \\lfloor k/2 \\rfloor + 1} - 2\\sqrt{k+1}\\big) \\cdot \\frac{28M_1M_\\infty}{(3\\sqrt{3})} \\\\ \n\t\t\t\t&\\le \\frac{{54M_1M_\\infty}}{k+2} \\ , \n\t\t\\end{aligned}\\end{equation}\\end{small}where the second inequality above uses \\cref{lm: simple ineq}. \n\t\tSimilarly for the second term in the rightmost side of \\cref{ineq: recurrent Z_k 5} we have:\n\t\t\t\\begin{small}\\begin{equation}\n\t\t\t\\begin{aligned}\\label{ineq: Z_k recurrent sumpro 2}\n\t\t\t\t\\sum_{i=\\lfloor k/2 \\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} &\\le   \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\\\ \n\t\t\t\t&\\le \\sqrt{k+1} \\cdot \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\le \\frac{80M_1M_\\infty}{k+2} \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses \\cref{lm: rho}.\n\t\tSubstituting \\cref{ineq: Z_k recurrent sumpro 1} and \\cref{ineq: Z_k recurrent sumpro 2} back into \\cref{ineq: recurrent Z_k 5} yields the second part of \\cref{eq: lm: convex EZ}, namely:\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: convex EZ_k final}\n\t\t\t\t\\mathbb{E} [Z_k] \\le  \\frac{134M_1M_\\infty}{k+2} \\ .\n\t\t\t\\end{equation}\n\t\t\\end{small}\\end{proof}","start":73888,"end":81481}]},{"id":"S044","semantic_type":"proof","kind":"proof","labels":["ineq: G_k recurrent sumpro 1","ineq: G_k recurrent sumpro 2","ineq: convex EG_k final","ineq: recurrent G_k 2","ineq: recurrent G_k 3","ncl"],"declared_external":false,"source_only":false,"title":"一阶矩部分作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{lm: convex EZ}]\n\t\t\n\t\tFirst of all, let us define $G_k^i := \\left( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\right)$ and $Z_k^i :=\\left( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\right)^2$ for $i \\in [n]$, and also $G_k :=\\sum_{i=1}^n G_k^i$ and $Z_k :=\\sum_{i=1}^n Z_k^i$. Notice in $G_k^i$ and $Z_k^i$ that the randomness lies only in $\\tau_i^k$, which in turn depends on the update sets $\\calB_0, \\calB_1,\\dots,\\calB_k$. \n\t\t\n\t\tWe first analyze $G_k^i$ and $G_k$.  We note that $G_k^i = 0$ when $i \\in \\calB_k$, which happens with probability $\\beta_k/n$.  Because of the relationship between $\\{\\tau^k_i\\}$ and $\\{\\tau^{k-1}_i\\}$ in \\cref{eq: tauk relationship}, we further note that with probability $1-\\beta_k/n$ that $i \\notin \\calB_k$ in which case it follows that $G_k^i = G_{k-1}^i + \\gamma_{k-1}d_{i,k-1}$.  Defining $\\rho_k := 1 - \\frac{1}{\\sqrt{k}} = 1 - \\beta_k/n $, we have \n\t\t\\begin{small}\\begin{equation}\\label{ncl}\n\t\t\t\\mathbb{E}[ G_k ] = \\sum_{i=1}^n \\mathbb{E} [G_k^i ]= \\rho_k \\sum_{i=1}^n \\mathbb{E} [G_{k-1}^i +\n\t\t\t\\gamma_{k-1} d_{i,k-1}] \\le \\rho_k\\left( \\mathbb{E} [G_{k-1} ]+\n\t\t\t\\gamma_{k-1} M_1 \\right) \\ ,\n\t\t\\end{equation}\\end{small}where the inequality uses $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$. Recurrently applying \\cref{ncl} yields:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent G_k 2}\n\t\t\t \\mathbb{E}[ G_k ] \\le \\rho_k( \\mathbb{E}[ G_{k-1} ]+\n\t\t\t\\gamma_{k-1}  M_1 ) \\le\\cdots\\le \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\gamma_{i-1} M_1 + \\Big(\\prod_{j=1}^{k} \\rho_j\\Big)\\mathbb{E}[ G_0] \\ .\n\t\t\\end{equation}\\end{small}Noting that $G_0 = 0$ (since all Taylor points are set to $x^0$ in \\cref{bettina} of \\cref{alg: TUFW in general finite sum}) and using $\\gamma_{i-1} = \\frac{2}{i+1}$ it follows that:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent G_k 3}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\mathbb{E} [G_k] &\\le \\sum_{i=1}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{i+1}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor {k}/{2}\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{i+1}\\\\\n\t\t\t\t& \\le \\sum_{i=2}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{3}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor {k}/{2}\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor {k}/{2}\\rfloor+2} \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses the fact that $\\rho_1 = 0$ and hence $\\prod_{j=1}^{k}\\rho_j = 0$.\n\t\tBy \\cref{lm: rho} we have:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: G_k recurrent sumpro 1}\n\t\t\t\\sum_{i=1}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)  \\frac{2M_1}{3}\n\t\t\t\\le\n\t\t\t\\sqrt{\\lfloor {k}/{2}\n\t\t\t\t\\rfloor + 1} \\exp\\Big(2 \\sqrt{ \\lfloor {k}/{2} \\rfloor + 1} - 2\\sqrt{k+1}\\Big)  \\frac{2M_1}{3} \\le \\frac{2 M_1}{\\sqrt{k+2}} \\ ,\n\t\t\\end{equation}\\end{small}where the second inequality above uses \\cref{lm: simple ineq}.\n\t\tSimilarly for the second term in the rightmost side of \\cref{ineq: recurrent G_k 3} we have:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: G_k recurrent sumpro 2}\n\t\t\t\\sum_{i=\\lfloor k/2 \\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor k/2\\rfloor+2} \\le   \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor k/2\\rfloor+2} \\le \\frac{\\sqrt{k+1} \\cdot  2M_1}{\\lfloor k/2\\rfloor+2} \\le \\frac{4M_1}{\\sqrt{k+2}} \\ , \n\t\t\\end{equation}\\end{small}where the second inequality uses \\cref{lm: rho}.\n\t\tSubstituting \\cref{ineq: G_k recurrent sumpro 1} and \\cref{ineq: G_k recurrent sumpro 2} back into \\cref{ineq: recurrent G_k 3} yields the first part of \\cref{eq: lm: convex EZ}, namely:\n\t\t\\begin{equation}\\label{ineq: convex EG_k final}\n\t\t\t\\mathbb{E} [G_k] \\le {6 M_1}/{\\sqrt{k+2}} \\ .\n\t\t\\end{equation}\n","start":73888,"end":77572}]},{"id":"S045","semantic_type":"proof","kind":"proof","labels":["ineq: Z_k recurrent sumpro 1","ineq: Z_k recurrent sumpro 2","ineq: convex EZ_k final","ineq: recurrent Z_k 4","ineq: recurrent Z_k 5","nclz"],"declared_external":false,"source_only":false,"title":"平方矩部分作者证明","source_units":[{"raw_tex":"\t\t\n\t\tWe employ a similar proof approach to establish the second part of \\cref{eq: lm: convex EZ}. We note that $Z_k^i = 0$ when $i \\in \\calB_k$, which happens with probability $\\beta_k/n$.  Because of the relationship between $\\{\\tau^k_i\\}$ and $\\{\\tau^{k-1}_i\\}$ in \\cref{eq: tauk relationship}, we further note that with probability $1-\\beta_k/n$ that $i \\notin \\calB_k$ in which case we have\n\t\t\\begin{small}\\begin{equation*}\n\t\t\t\\begin{aligned} \n\t\t\t\tZ_k^i &= (G_{k-1}^i + \\gamma_{k-1}d_{i,k-1})^2 = Z_{k-1}^i + 2 G_{k-1}^i\\gamma_{k-1}d_{i,k-1} + (\\gamma_{k-1}d_{i,k-1})^2 \\\\ &\\le Z_{k-1}^i + 2 M_\\infty \\gamma_{k-1}G_{k-1}^i + \\gamma_{k-1}^2 M_\\infty d_{i,k-1} \\ . \n\t\t\t\\end{aligned}\\end{equation*}\\end{small}Therefore\n\t\t\\begin{small}\\begin{equation}\\begin{aligned}\\label{nclz}\n\t\t\t\t\\mathbb{E}[ Z_k ] = &\\sum_{i=1}^n \\mathbb{E} [Z_k^i ] \\le \\rho_k \\sum_{i=1}^n \\mathbb{E} [Z_{k-1}^i + 2 M_\\infty \\gamma_{k-1}G_{k-1}^i + \\gamma_{k-1}^2 M_\\infty d_{i,k-1}] \\\\ \n\t\t\t\t&\\le \\rho_k \\left( \\mathbb{E} [Z_{k-1}] + 2   M_\\infty \\gamma_{k-1}\\mathbb{E}[G_{k-1}] +   \\gamma_{k-1}^2 M_\\infty M_1 \\right) \\\\\n\t\t\t\t&\\le \\rho_k \\left(\\mathbb{E} [Z_{k-1}] + \\frac{24M_\\infty M_1}{(k+1)\\sqrt{k+1}}   +  \\frac{4M_\\infty M_1}{(k+1)^2} \\right) \\le \\rho_k \\left(\\mathbb{E} [Z_{k-1}] + \\frac{28M_\\infty M_1}{(k+1)\\sqrt{k+1}}  \\right)    \\ ,\n\t\t\\end{aligned}\\end{equation} \\end{small}where the second inequality uses $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$, and the third inequality uses \\cref{ineq: convex EG_k final}. Recurrently applying \\cref{nclz} yields:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent Z_k 4}\n\t\t\t\\mathbb{E} [Z_k]  \\le \\rho_k\\bigg( \\mathbb{E} [Z_{k-1} ]+\n\t\t\t\\frac{28M_1M_\\infty}{(k+1)^{3/2}}\\bigg) \\le \\cdots\\le \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}  \\ .\n\t\t\\end{equation}\\end{small}Therefore\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent Z_k 5}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\mathbb{E} [Z_k]  & \\le  \\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor k/2\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}\\\\\n\t\t\t\t& \\le \\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{3\\sqrt{3}}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor k/2\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\ .\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}By \\cref{lm: rho} we have \n\t\t\t\\begin{small}\\begin{equation}\\label{ineq: Z_k recurrent sumpro 1}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{3\\sqrt{3}}  &\\le \\sqrt{\\lfloor k/2\n\t\t\t\t\t\\rfloor + 1} \\exp\\big(2 \\sqrt{ \\lfloor k/2 \\rfloor + 1} - 2\\sqrt{k+1}\\big) \\cdot \\frac{28M_1M_\\infty}{(3\\sqrt{3})} \\\\ \n\t\t\t\t&\\le \\frac{{54M_1M_\\infty}}{k+2} \\ , \n\t\t\\end{aligned}\\end{equation}\\end{small}where the second inequality above uses \\cref{lm: simple ineq}. \n\t\tSimilarly for the second term in the rightmost side of \\cref{ineq: recurrent Z_k 5} we have:\n\t\t\t\\begin{small}\\begin{equation}\n\t\t\t\\begin{aligned}\\label{ineq: Z_k recurrent sumpro 2}\n\t\t\t\t\\sum_{i=\\lfloor k/2 \\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} &\\le   \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\\\ \n\t\t\t\t&\\le \\sqrt{k+1} \\cdot \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\le \\frac{80M_1M_\\infty}{k+2} \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses \\cref{lm: rho}.\n\t\tSubstituting \\cref{ineq: Z_k recurrent sumpro 1} and \\cref{ineq: Z_k recurrent sumpro 2} back into \\cref{ineq: recurrent Z_k 5} yields the second part of \\cref{eq: lm: convex EZ}, namely:\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: convex EZ_k final}\n\t\t\t\t\\mathbb{E} [Z_k] \\le  \\frac{134M_1M_\\infty}{k+2} \\ .\n\t\t\t\\end{equation}\n\t\t\\end{small}\\end{proof}","start":77572,"end":81481}]},{"id":"S046","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"随机陈旧和与平方和","source_units":[{"raw_tex":"\t\t\n\t\tFirst of all, let us define $G_k^i := \\left( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\right)$ and $Z_k^i :=\\left( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\right)^2$ for $i \\in [n]$, and also $G_k :=\\sum_{i=1}^n G_k^i$ and $Z_k :=\\sum_{i=1}^n Z_k^i$. Notice in $G_k^i$ and $Z_k^i$ that the randomness lies only in $\\tau_i^k$, which in turn depends on the update sets $\\calB_0, \\calB_1,\\dots,\\calB_k$. \n\t\t\n","start":73933,"end":74348}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":["ncl"],"declared_external":false,"source_only":false,"title":"一阶矩刷新递推","source_units":[{"raw_tex":"\t\tWe first analyze $G_k^i$ and $G_k$.  We note that $G_k^i = 0$ when $i \\in \\calB_k$, which happens with probability $\\beta_k/n$.  Because of the relationship between $\\{\\tau^k_i\\}$ and $\\{\\tau^{k-1}_i\\}$ in \\cref{eq: tauk relationship}, we further note that with probability $1-\\beta_k/n$ that $i \\notin \\calB_k$ in which case it follows that $G_k^i = G_{k-1}^i + \\gamma_{k-1}d_{i,k-1}$.  Defining $\\rho_k := 1 - \\frac{1}{\\sqrt{k}} = 1 - \\beta_k/n $, we have \n\t\t\\begin{small}\\begin{equation}\\label{ncl}\n\t\t\t\\mathbb{E}[ G_k ] = \\sum_{i=1}^n \\mathbb{E} [G_k^i ]= \\rho_k \\sum_{i=1}^n \\mathbb{E} [G_{k-1}^i +\n\t\t\t\\gamma_{k-1} d_{i,k-1}] \\le \\rho_k\\left( \\mathbb{E} [G_{k-1} ]+\n\t\t\t\\gamma_{k-1} M_1 \\right) \\ ,\n\t\t\\end{equation}\\end{small}where the inequality uses $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$. Recurrently applying \\cref{ncl} yields:\n","start":74348,"end":75187}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: recurrent G_k 2","ineq: recurrent G_k 3"],"declared_external":false,"source_only":false,"title":"一阶矩乘积展开与分段","source_units":[{"raw_tex":"\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent G_k 2}\n\t\t\t \\mathbb{E}[ G_k ] \\le \\rho_k( \\mathbb{E}[ G_{k-1} ]+\n\t\t\t\\gamma_{k-1}  M_1 ) \\le\\cdots\\le \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\gamma_{i-1} M_1 + \\Big(\\prod_{j=1}^{k} \\rho_j\\Big)\\mathbb{E}[ G_0] \\ .\n\t\t\\end{equation}\\end{small}Noting that $G_0 = 0$ (since all Taylor points are set to $x^0$ in \\cref{bettina} of \\cref{alg: TUFW in general finite sum}) and using $\\gamma_{i-1} = \\frac{2}{i+1}$ it follows that:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent G_k 3}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\mathbb{E} [G_k] &\\le \\sum_{i=1}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{i+1}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor {k}/{2}\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{i+1}\\\\\n\t\t\t\t& \\le \\sum_{i=2}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{3}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor {k}/{2}\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor {k}/{2}\\rfloor+2} \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses the fact that $\\rho_1 = 0$ and hence $\\prod_{j=1}^{k}\\rho_j = 0$.\n","start":75187,"end":76313}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: G_k recurrent sumpro 1"],"declared_external":false,"source_only":false,"title":"一阶矩前半段估计","source_units":[{"raw_tex":"\t\tBy \\cref{lm: rho} we have:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: G_k recurrent sumpro 1}\n\t\t\t\\sum_{i=1}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)  \\frac{2M_1}{3}\n\t\t\t\\le\n\t\t\t\\sqrt{\\lfloor {k}/{2}\n\t\t\t\t\\rfloor + 1} \\exp\\Big(2 \\sqrt{ \\lfloor {k}/{2} \\rfloor + 1} - 2\\sqrt{k+1}\\Big)  \\frac{2M_1}{3} \\le \\frac{2 M_1}{\\sqrt{k+2}} \\ ,\n\t\t\\end{equation}\\end{small}where the second inequality above uses \\cref{lm: simple ineq}.\n","start":76313,"end":76750}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: G_k recurrent sumpro 2"],"declared_external":false,"source_only":false,"title":"一阶矩后半段估计","source_units":[{"raw_tex":"\t\tSimilarly for the second term in the rightmost side of \\cref{ineq: recurrent G_k 3} we have:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: G_k recurrent sumpro 2}\n\t\t\t\\sum_{i=\\lfloor k/2 \\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor k/2\\rfloor+2} \\le   \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor k/2\\rfloor+2} \\le \\frac{\\sqrt{k+1} \\cdot  2M_1}{\\lfloor k/2\\rfloor+2} \\le \\frac{4M_1}{\\sqrt{k+2}} \\ , \n\t\t\\end{equation}\\end{small}where the second inequality uses \\cref{lm: rho}.\n","start":76750,"end":77267}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: convex EG_k final"],"declared_external":false,"source_only":false,"title":"合并一阶矩常数","source_units":[{"raw_tex":"\t\tSubstituting \\cref{ineq: G_k recurrent sumpro 1} and \\cref{ineq: G_k recurrent sumpro 2} back into \\cref{ineq: recurrent G_k 3} yields the first part of \\cref{eq: lm: convex EZ}, namely:\n\t\t\\begin{equation}\\label{ineq: convex EG_k final}\n\t\t\t\\mathbb{E} [G_k] \\le {6 M_1}/{\\sqrt{k+2}} \\ .\n\t\t\\end{equation}\n","start":77267,"end":77572}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":["nclz"],"declared_external":false,"source_only":false,"title":"平方矩刷新递推","source_units":[{"raw_tex":"\t\tWe employ a similar proof approach to establish the second part of \\cref{eq: lm: convex EZ}. We note that $Z_k^i = 0$ when $i \\in \\calB_k$, which happens with probability $\\beta_k/n$.  Because of the relationship between $\\{\\tau^k_i\\}$ and $\\{\\tau^{k-1}_i\\}$ in \\cref{eq: tauk relationship}, we further note that with probability $1-\\beta_k/n$ that $i \\notin \\calB_k$ in which case we have\n\t\t\\begin{small}\\begin{equation*}\n\t\t\t\\begin{aligned} \n\t\t\t\tZ_k^i &= (G_{k-1}^i + \\gamma_{k-1}d_{i,k-1})^2 = Z_{k-1}^i + 2 G_{k-1}^i\\gamma_{k-1}d_{i,k-1} + (\\gamma_{k-1}d_{i,k-1})^2 \\\\ &\\le Z_{k-1}^i + 2 M_\\infty \\gamma_{k-1}G_{k-1}^i + \\gamma_{k-1}^2 M_\\infty d_{i,k-1} \\ . \n\t\t\t\\end{aligned}\\end{equation*}\\end{small}Therefore\n\t\t\\begin{small}\\begin{equation}\\begin{aligned}\\label{nclz}\n\t\t\t\t\\mathbb{E}[ Z_k ] = &\\sum_{i=1}^n \\mathbb{E} [Z_k^i ] \\le \\rho_k \\sum_{i=1}^n \\mathbb{E} [Z_{k-1}^i + 2 M_\\infty \\gamma_{k-1}G_{k-1}^i + \\gamma_{k-1}^2 M_\\infty d_{i,k-1}] \\\\ \n\t\t\t\t&\\le \\rho_k \\left( \\mathbb{E} [Z_{k-1}] + 2   M_\\infty \\gamma_{k-1}\\mathbb{E}[G_{k-1}] +   \\gamma_{k-1}^2 M_\\infty M_1 \\right) \\\\\n\t\t\t\t&\\le \\rho_k \\left(\\mathbb{E} [Z_{k-1}] + \\frac{24M_\\infty M_1}{(k+1)\\sqrt{k+1}}   +  \\frac{4M_\\infty M_1}{(k+1)^2} \\right) \\le \\rho_k \\left(\\mathbb{E} [Z_{k-1}] + \\frac{28M_\\infty M_1}{(k+1)\\sqrt{k+1}}  \\right)    \\ ,\n\t\t\\end{aligned}\\end{equation} \\end{small}where the second inequality uses $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$, and the third inequality uses \\cref{ineq: convex EG_k final}. Recurrently applying \\cref{nclz} yields:\n","start":77575,"end":79106}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: recurrent Z_k 4","ineq: recurrent Z_k 5"],"declared_external":false,"source_only":false,"title":"平方矩乘积展开与分段","source_units":[{"raw_tex":"\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent Z_k 4}\n\t\t\t\\mathbb{E} [Z_k]  \\le \\rho_k\\bigg( \\mathbb{E} [Z_{k-1} ]+\n\t\t\t\\frac{28M_1M_\\infty}{(k+1)^{3/2}}\\bigg) \\le \\cdots\\le \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}  \\ .\n\t\t\\end{equation}\\end{small}Therefore\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent Z_k 5}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\mathbb{E} [Z_k]  & \\le  \\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor k/2\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}\\\\\n\t\t\t\t& \\le \\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{3\\sqrt{3}}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor k/2\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\ .\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}By \\cref{lm: rho} we have \n","start":79106,"end":80027}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: Z_k recurrent sumpro 1"],"declared_external":false,"source_only":false,"title":"平方矩前半段估计","source_units":[{"raw_tex":"\t\t\t\\begin{small}\\begin{equation}\\label{ineq: Z_k recurrent sumpro 1}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{3\\sqrt{3}}  &\\le \\sqrt{\\lfloor k/2\n\t\t\t\t\t\\rfloor + 1} \\exp\\big(2 \\sqrt{ \\lfloor k/2 \\rfloor + 1} - 2\\sqrt{k+1}\\big) \\cdot \\frac{28M_1M_\\infty}{(3\\sqrt{3})} \\\\ \n\t\t\t\t&\\le \\frac{{54M_1M_\\infty}}{k+2} \\ , \n\t\t\\end{aligned}\\end{equation}\\end{small}where the second inequality above uses \\cref{lm: simple ineq}. \n","start":80027,"end":80507}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: Z_k recurrent sumpro 2"],"declared_external":false,"source_only":false,"title":"平方矩后半段估计","source_units":[{"raw_tex":"\t\tSimilarly for the second term in the rightmost side of \\cref{ineq: recurrent Z_k 5} we have:\n\t\t\t\\begin{small}\\begin{equation}\n\t\t\t\\begin{aligned}\\label{ineq: Z_k recurrent sumpro 2}\n\t\t\t\t\\sum_{i=\\lfloor k/2 \\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} &\\le   \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\\\ \n\t\t\t\t&\\le \\sqrt{k+1} \\cdot \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\le \\frac{80M_1M_\\infty}{k+2} \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses \\cref{lm: rho}.\n","start":80507,"end":81125}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: convex EZ_k final"],"declared_external":false,"source_only":false,"title":"合并平方矩常数","source_units":[{"raw_tex":"\t\tSubstituting \\cref{ineq: Z_k recurrent sumpro 1} and \\cref{ineq: Z_k recurrent sumpro 2} back into \\cref{ineq: recurrent Z_k 5} yields the second part of \\cref{eq: lm: convex EZ}, namely:\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: convex EZ_k final}\n\t\t\t\t\\mathbb{E} [Z_k] \\le  \\frac{134M_1M_\\infty}{k+2} \\ .\n\t\t\t\\end{equation}\n\t\t\\end{small}\\end{proof}","start":81125,"end":81481}]},{"id":"S057","semantic_type":"proof","kind":"proof","labels":["ineq: convex thm 1 general","ineq: convex thm 2 general"],"declared_external":false,"source_only":false,"title":"主定理完整作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{thm: general convergence rate convex}]\n\t\tFrom \\cref{lm: general convergence rate convex} it holds that:\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\n\t\t\t\t\\eps_k  \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{2}{k(k+1)}\\sum_{t=1}^{k} t \\left(\n\t\t\t\t\\nabla F(x^{t-1})   - g^{t-1}\n\t\t\t\t\\right)^\\top (s^{t-1} - x^\\star) \\ ,\n\t\t\t\\end{equation}\n\t\t\\end{small}and from \\cref{lm: general 3rd term upper bound} it follows that:\n\t\t\\begin{small}\n\t\t\t\\begin{equation*}\n\t\t\t\t\\eps_k \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{\\hat{L}D^3}{nk(k+1)}\\sum_{t=1}^{k}t\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^{t-1}_i}^{t-2}  \\gamma_j \\Big)^2 \\ .\n\t\t\t\\end{equation*}\n\t\t\\end{small}Taking expectations on both sides then yields\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: convex thm 1 general}\n\t\t\t\t\\mathbb{E} [\\eps_k]  \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{\\hat{L}D^3}{nk(k+1)}\\sum_{t=1}^{k}t~\\mathbb{E}\\Big[\n\t\t\t\t\\sum_{i=1}^{n}\\Big(  \\sum_{j = \\tau^{t-1}_i}^{t-2}  \\gamma_j \\Big)^2\n\t\t\t\t\\Big] \\ .\n\t\t\t\\end{equation}\n\t\t\\end{small}Now define $d_{ij} := 1$ and $M_1 := n$ and $M_\\infty := 1$.  Then since $\\sup_{j}   \\sum_{i=1}^{n} d_{ij} = M_1$  and  $\\sup_{i,j} d_{ij}$ is equal to $M_\\infty$, we can apply \\cref{lm: convex EZ} to \\cref{ineq: convex thm 1 general}  and obtain\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: convex thm 2 general}\n\t\t\t\t\\mathbb{E} [\\eps_k]   \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{\\hat{L}D^3}{nk(k+1)}\\sum_{t=1}^{k}t~\\frac{134 n}{t+1} \\le \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{134 \\hat{L}D^3}{k+1} \\ , \n\t\t\t\\end{equation}\n\t\t\\end{small}which proves \\cref{eq of thm: general convergence rate convex}.\n\t\\end{proof}","start":81564,"end":83163}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"调用凸差距引理","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{thm: general convergence rate convex}]\n\t\tFrom \\cref{lm: general convergence rate convex} it holds that:\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\n\t\t\t\t\\eps_k  \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{2}{k(k+1)}\\sum_{t=1}^{k} t \\left(\n\t\t\t\t\\nabla F(x^{t-1})   - g^{t-1}\n\t\t\t\t\\right)^\\top (s^{t-1} - x^\\star) \\ ,\n\t\t\t\\end{equation}\n","start":81564,"end":81909}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"代入Taylor误差界","source_units":[{"raw_tex":"\t\t\\end{small}and from \\cref{lm: general 3rd term upper bound} it follows that:\n\t\t\\begin{small}\n\t\t\t\\begin{equation*}\n\t\t\t\t\\eps_k \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{\\hat{L}D^3}{nk(k+1)}\\sum_{t=1}^{k}t\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^{t-1}_i}^{t-2}  \\gamma_j \\Big)^2 \\ .\n\t\t\t\\end{equation*}\n","start":81909,"end":82199}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: convex thm 1 general"],"declared_external":false,"source_only":false,"title":"对陈旧误差取期望","source_units":[{"raw_tex":"\t\t\\end{small}Taking expectations on both sides then yields\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: convex thm 1 general}\n\t\t\t\t\\mathbb{E} [\\eps_k]  \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{\\hat{L}D^3}{nk(k+1)}\\sum_{t=1}^{k}t~\\mathbb{E}\\Big[\n\t\t\t\t\\sum_{i=1}^{n}\\Big(  \\sum_{j = \\tau^{t-1}_i}^{t-2}  \\gamma_j \\Big)^2\n\t\t\t\t\\Big] \\ .\n\t\t\t\\end{equation}\n","start":82199,"end":82547}]},{"id":"S061","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"主证明中的常数数组","source_units":[{"raw_tex":"\t\t\\end{small}Now define $d_{ij} := 1$ and $M_1 := n$ and $M_\\infty := 1$.  Then since $\\sup_{j}   \\sum_{i=1}^{n} d_{ij} = M_1$  and  $\\sup_{i,j} d_{ij}$ is equal to $M_\\infty$, we can apply \\cref{lm: convex EZ} to \\cref{ineq: convex thm 1 general}  and obtain\n","start":82547,"end":82807}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: convex thm 2 general"],"declared_external":false,"source_only":false,"title":"代入平方矩并完成求和","source_units":[{"raw_tex":"\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: convex thm 2 general}\n\t\t\t\t\\mathbb{E} [\\eps_k]   \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{\\hat{L}D^3}{nk(k+1)}\\sum_{t=1}^{k}t~\\frac{134 n}{t+1} \\le \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{134 \\hat{L}D^3}{k+1} \\ , \n\t\t\t\\end{equation}\n\t\t\\end{small}which proves \\cref{eq of thm: general convergence rate convex}.\n\t\\end{proof}","start":82807,"end":83163}]},{"id":"S063","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始外部前提清单：First-order gradient inequality for differentiable convex functions；作者在第833行明确说明使用。"],"source_only":false,"title":"凸函数一阶梯度不等式","source_units":[]},{"id":"S064","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始外部前提清单：dual-norm Hölder and triangle inequalities。"],"source_only":false,"title":"对偶范数与三角不等式","source_units":[]},{"id":"S065","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始外部前提清单：Elementary expectation identities；条件历史的使用由标注者显式化。"],"source_only":false,"title":"有限期望与条件期望规则","source_units":[]},{"id":"S066","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始外部前提清单：telescoping sums, logarithmic inequalities, and calculus-based integral comparisons；作者还在第895及899行明确使用凹性与极值事实。"],"source_only":false,"title":"求和与初等微积分工具","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S020","edges":[{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S009","target_id":"S008","kind":"supports"},{"source_id":"S010","target_id":"S008","kind":"supports"},{"source_id":"S011","target_id":"S008","kind":"supports"},{"source_id":"S006","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S065","target_id":"S013","kind":"supports"},{"source_id":"S002","target_id":"S016","kind":"supports"},{"source_id":"S006","target_id":"S016","kind":"supports"},{"source_id":"S002","target_id":"S017","kind":"supports"},{"source_id":"S006","target_id":"S017","kind":"supports"},{"source_id":"S009","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S009","target_id":"S019","kind":"supports"},{"source_id":"S001","target_id":"S020","kind":"supports"},{"source_id":"S002","target_id":"S020","kind":"supports"},{"source_id":"S008","target_id":"S020","kind":"supports"},{"source_id":"S012","target_id":"S020","kind":"supports"},{"source_id":"S014","target_id":"S020","kind":"supports"},{"source_id":"S015","target_id":"S020","kind":"supports"},{"source_id":"S001","target_id":"S023","kind":"supports"},{"source_id":"S009","target_id":"S023","kind":"supports"},{"source_id":"S010","target_id":"S023","kind":"supports"},{"source_id":"S018","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S014","target_id":"S024","kind":"supports"},{"source_id":"S017","target_id":"S024","kind":"supports"},{"source_id":"S064","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S011","target_id":"S025","kind":"supports"},{"source_id":"S014","target_id":"S025","kind":"supports"},{"source_id":"S064","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S001","target_id":"S028","kind":"supports"},{"source_id":"S016","target_id":"S028","kind":"supports"},{"source_id":"S011","target_id":"S028","kind":"supports"},{"source_id":"S014","target_id":"S028","kind":"supports"},{"source_id":"S015","target_id":"S028","kind":"supports"},{"source_id":"S063","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S015","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S066","target_id":"S030","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S018","target_id":"S033","kind":"supports"},{"source_id":"S012","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S018","target_id":"S034","kind":"supports"},{"source_id":"S012","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S034","target_id":"S031","kind":"supports"},{"source_id":"S066","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S066","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S066","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S066","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S042","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S018","target_id":"S046","kind":"supports"},{"source_id":"S032","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S019","target_id":"S047","kind":"supports"},{"source_id":"S012","target_id":"S047","kind":"supports"},{"source_id":"S013","target_id":"S047","kind":"supports"},{"source_id":"S032","target_id":"S047","kind":"supports"},{"source_id":"S065","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S009","target_id":"S048","kind":"supports"},{"source_id":"S032","target_id":"S048","kind":"supports"},{"source_id":"S066","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S035","target_id":"S049","kind":"supports"},{"source_id":"S039","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S035","target_id":"S050","kind":"supports"},{"source_id":"S066","target_id":"S050","kind":"supports"},{"source_id":"S048","target_id":"S051","kind":"supports"},{"source_id":"S049","target_id":"S051","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S044","kind":"supports"},{"source_id":"S049","target_id":"S044","kind":"supports"},{"source_id":"S050","target_id":"S044","kind":"supports"},{"source_id":"S051","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S033","kind":"supports"},{"source_id":"S046","target_id":"S052","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S019","target_id":"S052","kind":"supports"},{"source_id":"S012","target_id":"S052","kind":"supports"},{"source_id":"S032","target_id":"S052","kind":"supports"},{"source_id":"S065","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S046","target_id":"S053","kind":"supports"},{"source_id":"S009","target_id":"S053","kind":"supports"},{"source_id":"S066","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S035","target_id":"S054","kind":"supports"},{"source_id":"S039","target_id":"S054","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S035","target_id":"S055","kind":"supports"},{"source_id":"S066","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S056","kind":"supports"},{"source_id":"S054","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S052","target_id":"S045","kind":"supports"},{"source_id":"S053","target_id":"S045","kind":"supports"},{"source_id":"S054","target_id":"S045","kind":"supports"},{"source_id":"S055","target_id":"S045","kind":"supports"},{"source_id":"S056","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S034","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S031","kind":"supports"},{"source_id":"S026","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S021","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S065","target_id":"S060","kind":"supports"},{"source_id":"S032","target_id":"S061","kind":"supports"},{"source_id":"S060","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S034","target_id":"S062","kind":"supports"},{"source_id":"S066","target_id":"S062","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S059","target_id":"S057","kind":"supports"},{"source_id":"S060","target_id":"S057","kind":"supports"},{"source_id":"S061","target_id":"S057","kind":"supports"},{"source_id":"S062","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S020","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S020","edges":[{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S009","target_id":"S008","kind":"supports"},{"source_id":"S010","target_id":"S008","kind":"supports"},{"source_id":"S011","target_id":"S008","kind":"supports"},{"source_id":"S006","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S065","target_id":"S013","kind":"supports"},{"source_id":"S002","target_id":"S016","kind":"supports"},{"source_id":"S006","target_id":"S016","kind":"supports"},{"source_id":"S002","target_id":"S017","kind":"supports"},{"source_id":"S006","target_id":"S017","kind":"supports"},{"source_id":"S009","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S009","target_id":"S019","kind":"supports"},{"source_id":"S001","target_id":"S020","kind":"supports"},{"source_id":"S002","target_id":"S020","kind":"supports"},{"source_id":"S008","target_id":"S020","kind":"supports"},{"source_id":"S012","target_id":"S020","kind":"supports"},{"source_id":"S014","target_id":"S020","kind":"supports"},{"source_id":"S015","target_id":"S020","kind":"supports"},{"source_id":"S001","target_id":"S023","kind":"supports"},{"source_id":"S009","target_id":"S023","kind":"supports"},{"source_id":"S010","target_id":"S023","kind":"supports"},{"source_id":"S018","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S014","target_id":"S024","kind":"supports"},{"source_id":"S017","target_id":"S024","kind":"supports"},{"source_id":"S064","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S011","target_id":"S025","kind":"supports"},{"source_id":"S014","target_id":"S025","kind":"supports"},{"source_id":"S064","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S001","target_id":"S028","kind":"supports"},{"source_id":"S016","target_id":"S028","kind":"supports"},{"source_id":"S011","target_id":"S028","kind":"supports"},{"source_id":"S014","target_id":"S028","kind":"supports"},{"source_id":"S015","target_id":"S028","kind":"supports"},{"source_id":"S063","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S015","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S066","target_id":"S030","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S018","target_id":"S033","kind":"supports"},{"source_id":"S012","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S018","target_id":"S034","kind":"supports"},{"source_id":"S012","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S034","target_id":"S031","kind":"supports"},{"source_id":"S066","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S066","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S066","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S066","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S042","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S018","target_id":"S046","kind":"supports"},{"source_id":"S032","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S019","target_id":"S047","kind":"supports"},{"source_id":"S012","target_id":"S047","kind":"supports"},{"source_id":"S013","target_id":"S047","kind":"supports"},{"source_id":"S032","target_id":"S047","kind":"supports"},{"source_id":"S065","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S009","target_id":"S048","kind":"supports"},{"source_id":"S032","target_id":"S048","kind":"supports"},{"source_id":"S066","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S035","target_id":"S049","kind":"supports"},{"source_id":"S039","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S035","target_id":"S050","kind":"supports"},{"source_id":"S066","target_id":"S050","kind":"supports"},{"source_id":"S048","target_id":"S051","kind":"supports"},{"source_id":"S049","target_id":"S051","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S044","kind":"supports"},{"source_id":"S049","target_id":"S044","kind":"supports"},{"source_id":"S050","target_id":"S044","kind":"supports"},{"source_id":"S051","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S033","kind":"supports"},{"source_id":"S046","target_id":"S052","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S019","target_id":"S052","kind":"supports"},{"source_id":"S012","target_id":"S052","kind":"supports"},{"source_id":"S032","target_id":"S052","kind":"supports"},{"source_id":"S065","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S046","target_id":"S053","kind":"supports"},{"source_id":"S009","target_id":"S053","kind":"supports"},{"source_id":"S066","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S035","target_id":"S054","kind":"supports"},{"source_id":"S039","target_id":"S054","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S035","target_id":"S055","kind":"supports"},{"source_id":"S066","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S056","kind":"supports"},{"source_id":"S054","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S052","target_id":"S045","kind":"supports"},{"source_id":"S053","target_id":"S045","kind":"supports"},{"source_id":"S054","target_id":"S045","kind":"supports"},{"source_id":"S055","target_id":"S045","kind":"supports"},{"source_id":"S056","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S034","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S031","kind":"supports"},{"source_id":"S026","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S021","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S065","target_id":"S060","kind":"supports"},{"source_id":"S032","target_id":"S061","kind":"supports"},{"source_id":"S060","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S034","target_id":"S062","kind":"supports"},{"source_id":"S066","target_id":"S062","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S059","target_id":"S057","kind":"supports"},{"source_id":"S060","target_id":"S057","kind":"supports"},{"source_id":"S061","target_id":"S057","kind":"supports"},{"source_id":"S062","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S020","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["pro: ERM"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"有限和约束优化问题","source_units":[{"raw_tex":"\tOur problem of interest is the following Empirical Risk Minimization (ERM) problem (also called the \\textit{finite-sum} problem) with constraints: \n\t\\begin{equation}\\label{pro: ERM}\n\t\t\\text{ERM:} \\ \\ \\ \\ \\    \\text{minimize}_{x \\in \\calC} \\quad F(x):= \\tfrac{1}{n} \\sum_{i=1}^{n} f_i(x) \\ , \n\t\\end{equation}\n\twhere $f_i(\\cdot): \\mathbb{R}^p \\to \\mathbb{R}$ is the loss function of observation $i$ for $i=1,\\dots, n$, and $x$ is comprised of the (model) coefficients/parameters which are to be optimized over the feasibility set $\\calC \\subset \\mathbb{R}^p$. Problem \\cref{pro: ERM} is a fundamental problem in statistical and machine learning, as it is the underlying optimization problem on which models are ``trained'' or ``learned'', see  \\cite{vapnik1991principles,hastie2009elements}. The number of observations $n$ can be significantly larger than the dimension $p$, and our interest here will be in the regime where $n \\gg p$.\n\tWe make the following assumptions for $\\erm$ problems throughout this paper:\n","start":12615,"end":13628}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":["assump: ERM","eq: ERM ass lip 1","eq: ERM ass lip 2"],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"通用ERM假设","source_units":[{"raw_tex":"\t\\begin{assumption}\\label{assump: ERM}\n\t\tThe following conditions hold for the $\\erm$ problem \\cref{pro: ERM}:\n\t\t\\begin{enumerate}\n\t\t\t\\item[(i)] the feasibility set $\\mathcal{C}$ is a compact convex set,\n\t\t\t\\item[(ii)] for any $g \\in \\mathbb{R}^p$, the linear minimization problem $\\arg\\min_{x\\in\\calC} g^\\top x$\n\t\t\tcan be easily solved, \n\t\t\t\\item[(iii)] for all $i\\in[n]$, the multivariate loss function $f_i(\\cdot)$ is twice-differentiable and its gradient $\\nabla f_i(\\cdot)$ is $L$-Lipschitz continuous on $\\calC$, namely, \n\t\t\t\\begin{equation}\\label{eq: ERM ass lip 1}\n\t\t\t\t\\| \\nabla f_i(u) - \\nabla f_i(v) \\|_* \\le L \\|u-v\\| ~~\\text{ for any $u,v\\in \\calC$}\n\t\t\t\\end{equation}\n\t\t\twhere $\\|\\cdot\\|_*$ is the dual norm of the norm on the space of variables, and \n\t\t\t\\item[(iv)] for all $i\\in[n]$, the Hessian $\\nabla^2 f_i(\\cdot)$ is $\\hat{L}$-Lipschitz continuous on $\\calC$, namely\n\t\t\t\\begin{equation}\\label{eq: ERM ass lip 2}\n\t\t\t\t\\| \\nabla^2 f_i(u) - \\nabla^2 f_i(v) \\| \\le \\hat{L} \\|u-v\\|~~\\text{ for any $u,v\\in \\calC$}\n\t\t\t\\end{equation}\n\t\t\twhere the norm of the Hessian matrices is the operator norm induced by the norm on the space of variables.\n\t\t\\end{enumerate}\n\t\\end{assumption}\n","start":13628,"end":14818}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","labels":["assump: ERM"],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"可行域与线性优化","source_units":[{"raw_tex":"\t\t\t\\item[(i)] the feasibility set $\\mathcal{C}$ is a compact convex set,\n\t\t\t\\item[(ii)] for any $g \\in \\mathbb{R}^p$, the linear minimization problem $\\arg\\min_{x\\in\\calC} g^\\top x$\n\t\t\tcan be easily solved, \n","start":13759,"end":13967}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","labels":["assump: ERM","eq: ERM ass lip 1"],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"梯度光滑性","source_units":[{"raw_tex":"\t\t\t\\item[(iii)] for all $i\\in[n]$, the multivariate loss function $f_i(\\cdot)$ is twice-differentiable and its gradient $\\nabla f_i(\\cdot)$ is $L$-Lipschitz continuous on $\\calC$, namely, \n\t\t\t\\begin{equation}\\label{eq: ERM ass lip 1}\n\t\t\t\t\\| \\nabla f_i(u) - \\nabla f_i(v) \\|_* \\le L \\|u-v\\| ~~\\text{ for any $u,v\\in \\calC$}\n\t\t\t\\end{equation}\n\t\t\twhere $\\|\\cdot\\|_*$ is the dual norm of the norm on the space of variables, and \n","start":13967,"end":14392}]},{"id":"S005","semantic_type":"assumption","kind":"assumption","labels":["assump: ERM","eq: ERM ass lip 2"],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"Hessian光滑性","source_units":[{"raw_tex":"\t\t\t\\item[(iv)] for all $i\\in[n]$, the Hessian $\\nabla^2 f_i(\\cdot)$ is $\\hat{L}$-Lipschitz continuous on $\\calC$, namely\n\t\t\t\\begin{equation}\\label{eq: ERM ass lip 2}\n\t\t\t\t\\| \\nabla^2 f_i(u) - \\nabla^2 f_i(v) \\| \\le \\hat{L} \\|u-v\\|~~\\text{ for any $u,v\\in \\calC$}\n\t\t\t\\end{equation}\n\t\t\twhere the norm of the Hessian matrices is the operator norm induced by the norm on the space of variables.\n","start":14392,"end":14782}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"索引范数与概率记号","source_units":[{"raw_tex":"\t\\subsection{Notation}\n\t\n\tLet $\\mathbb{N}$ denote the natural numbers.  For any $n \\in \\mathbb{N}$, we use $[n]$ to denote the set $\\{1,2,\\dots,n\\}$.\n\tFor any real number $v$, we use $\\lfloor v \\rfloor$ to denote the largest integer number less than or equal to $v$.\n\tDepending on the context, we will use either $\\langle a,b\\rangle$ or $a^\\top b$ to denote the inner product between $a$ and $b$ in Euclidean space. For a vector $x$ in $\\mathbb{R}^p$, the norm $\\|x\\|_q$ is given by $(\\sum_{j=1}^{p}x_j^q)^{1/q}$ for $q \\in [1,\\infty)$ and $\\|x\\|_\\infty := \\max_{j} |x_j|$. The dual of norm $\\|\\cdot\\|$ is defined as $\\|z\\|_* := \\sup\\{z^\\top x : \\|x\\| \\le 1\\}$.  For a matrix $A \\in \\mathbb{R}^{p \\times p}$, the matrix operator norm $\\|A\\|$ induced by the norm $\\|\\cdot\\|$ is defined as $\\sup_{x: \\|x\\| \\le 1} \\|Ax\\|_*$. For any symmetric positive definite matrix $A\\in \\mathbb{S}^{p\\times p}$, we define its induced norm on $\\mathbb{R}^p$ by $\\|z\\|_A := \\sqrt{z^\\top A z}$.  We define $\\sum_{i=j}^k \\alpha_i := 0$ whenever $j > k$.  We denote the uniform distribution on the set $S$ by $\\calU(S)$, and we denote the Bernoulli distribution with probability $p$ by $\\mathrm{Ber}(p)$.\n","start":34310,"end":35494}]},{"id":"S008","semantic_type":"algorithm","kind":"algorithm","labels":["alex","alg: TUFW in general finite sum","algline: 3","algline: 4","algline: 5","algline: 6","algline: 7","bettina","beverly","charlie","fanette","hotel"],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"完整TUFW算法","source_units":[{"raw_tex":"\t\\begin{algorithm}\n\t\t\\caption{(TUFW) Frank-Wolfe Method with Taylor Point Updating}\n\t\t\\label{alg: TUFW in general finite sum}\n\t\t\\begin{algorithmic}[1]\n\t\t\t\\STATE \\textbf{input}  initial point $x^0$, step-size sequence $\\{\\gamma_k\\}_k$, and (possibly) number of iterations $K$\n\t\t\t\\STATE \\textbf{initialize Taylor points} $\\calB_0 \\gets [n]$ and $b_i \\leftarrow x^0$ for $i\\in [n]$  \\; \\label{bettina}\n\t\t\t\\FOR{$k=0,1,2,\\dots,K$,}\n\t\t\t\\STATE\\label{algline: 3}\\textbf{If $k>0$, update Taylor points}: \\\\ \\ \\ Use some \\textbf{Rule} to construct set $\\calB_k \\subset [n]$ of indices for Taylor-point updating\\\\ \\ \\ Update Taylor points:  for $i\\in\\calB_k$ update $b_i \\gets x^k$ \\; \\label{fanette}\n\t\t\t\\STATE\\label{algline: 4}  \\textbf{Compute estimates of individual gradients}:\\\\ $g^k_{i}\\gets \\nabla f_{i}(b_{i})  +  \\nabla^2 f_i (b_{i}) ( x^k -  b_{i} ) $ for $i\\in [n]$\\; \\label{charlie}\n\t\t\t\\STATE\\label{algline: 5}   \\textbf{Compute estimate of full gradient}: $g^k\\gets \\frac{1}{n}\\sum_{i=1}^{n}g_i^k$\\; \\label{beverly}\n\t\t\t\\STATE\\label{algline: 6}   \\textbf{Solve LMO}: $s^{k}\\gets \\arg\\min_{s\\in\\calC} \\langle g^k,s\\rangle$\\; \\label{alex} \n\t\t\t\\STATE\\label{algline: 7}   \\textbf{Update feasible solution}: $x^{k+1}\\gets x^k + \\gamma_k (s^{k} - x^k)$. \\label{hotel}\n\t\t\t\\ENDFOR\n\t\t\t\\RETURN $x^{K+1}$\n\t\t\\end{algorithmic}\n\t\\end{algorithm}\n","start":36347,"end":37679}]},{"id":"S009","semantic_type":"algorithm","kind":"algorithm","labels":["alex","alg: TUFW in general finite sum","algline: 3","algline: 4","algline: 5","algline: 6","algline: 7","bettina","beverly","charlie","fanette","hotel"],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"Taylor点初始化与刷新","source_units":[{"raw_tex":"\t\t\t\\STATE \\textbf{input}  initial point $x^0$, step-size sequence $\\{\\gamma_k\\}_k$, and (possibly) number of iterations $K$\n\t\t\t\\STATE \\textbf{initialize Taylor points} $\\calB_0 \\gets [n]$ and $b_i \\leftarrow x^0$ for $i\\in [n]$  \\; \\label{bettina}\n\t\t\t\\FOR{$k=0,1,2,\\dots,K$,}\n\t\t\t\\STATE\\label{algline: 3}\\textbf{If $k>0$, update Taylor points}: \\\\ \\ \\ Use some \\textbf{Rule} to construct set $\\calB_k \\subset [n]$ of indices for Taylor-point updating\\\\ \\ \\ Update Taylor points:  for $i\\in\\calB_k$ update $b_i \\gets x^k$ \\; \\label{fanette}\n","start":36498,"end":37037}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","labels":["alex","alg: TUFW in general finite sum","algline: 3","algline: 4","algline: 5","algline: 6","algline: 7","bettina","beverly","charlie","fanette","hotel"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"Taylor梯度估计","source_units":[{"raw_tex":"\t\t\t\\STATE\\label{algline: 4}  \\textbf{Compute estimates of individual gradients}:\\\\ $g^k_{i}\\gets \\nabla f_{i}(b_{i})  +  \\nabla^2 f_i (b_{i}) ( x^k -  b_{i} ) $ for $i\\in [n]$\\; \\label{charlie}\n\t\t\t\\STATE\\label{algline: 5}   \\textbf{Compute estimate of full gradient}: $g^k\\gets \\frac{1}{n}\\sum_{i=1}^{n}g_i^k$\\; \\label{beverly}\n","start":37037,"end":37365}]},{"id":"S011","semantic_type":"algorithm","kind":"algorithm","labels":["alex","alg: TUFW in general finite sum","algline: 3","algline: 4","algline: 5","algline: 6","algline: 7","bettina","beverly","charlie","fanette","hotel"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"线性优化与迭代更新","source_units":[{"raw_tex":"\t\t\t\\STATE\\label{algline: 6}   \\textbf{Solve LMO}: $s^{k}\\gets \\arg\\min_{s\\in\\calC} \\langle g^k,s\\rangle$\\; \\label{alex} \n\t\t\t\\STATE\\label{algline: 7}   \\textbf{Update feasible solution}: $x^{k+1}\\gets x^k + \\gamma_k (s^{k} - x^k)$. \\label{hotel}\n","start":37365,"end":37610}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["rule 1"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"随机递减批量规则","source_units":[{"raw_tex":"\tRecall that in order to run \\cref{alg: TUFW in general finite sum} we need to specify the step-sizes $\\{\\gamma_k\\}_k$ and the {\\bf Rule} to construct the sets $\\calB_k$ of indices for Taylor-point updating in \\cref{fanette}.  Our first rule is designed \n\tfor $\\calB_k$ to be comprised of $\\beta_k := n /\\sqrt{k}$ independently drawn samples from $[n]$ without replacement, for all $k \\ge 1$.   Since $\\beta_k$ is not in general an integer, then rather than ensuring $|\\calB_k| = \\beta_k$, we will instead ensure that $\\mathbb{E}|\\calB_k| = \\beta_k$ by proceeding as follows.  \n\t\n\t\n\t\n\t\\begin{definition}\\label{rule 1} $\\rone$.  Define $\\beta_k := n /\\sqrt{k}$ and $p_k := \\beta_k - \\lfloor \\beta_k \\rfloor$.  Sample $\\xi_k$ from the Bernoulli distribution $\\mathrm{Ber}(p_k)$ and then uniformly sample $\\lfloor \\beta_k \\rfloor + \\xi_k$ samples from $[n]$ without replacement.  Then define $\\calB_k$ to be the set of these index samples.\\end{definition}\n","start":46870,"end":47823}]},{"id":"S013","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"期望批量大小","source_units":[{"raw_tex":"\t\n\tNote that by design of $\\rone$ it follows that $\\mathbb{E}_{\\xi_k} |\\calB_k| = \\beta_k$.  We call this rule ``$\\rone$'' for ``Stochastic Batch-size Decreasing at the rate $\\sqrt{k}$,'' and we emphasize that the Taylor points are updated less often as $k$ grows, which is in contrast to many other stochastic Frank-Wolfe methods that compute exact individual gradients more frequently as $k$ grows, see \\cite{hazan2016variance,yurtsever2019conditional}. \n","start":47823,"end":48280}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"可行域直径","source_units":[{"raw_tex":"\tLet $D$ denote the diameter of $\\calC$ under the norm of the variable space, namely\n\t$$\n\tD:= \\max_{u,v\\in\\calC} \\| u-v\\| \\ .\n\t$$\n","start":49654,"end":49784}]},{"id":"S015","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"最优解与最优性差距","source_units":[{"raw_tex":"\tIn the rest of the paper, we will use $x^\\star$ to denote an optimal solution of the $\\erm$ problem, and let $\\eps_k := F(x^k) - F(x^\\star)$. \n","start":62922,"end":63066}]},{"id":"S016","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S016"],"declared_external":true,"declaration_text":["原始前提清单明确允许B0059作为不附文内证明的标准光滑性前提；这里只保留其原文陈述。"],"source_only":false,"title":"函数一阶展开余项","source_units":[{"raw_tex":"\tWe begin with the following straightforward properties of smooth functions.\n\t\\begin{proposition}\\label{prop: general basic ineq}\n\t\tUnder \\cref{assump: ERM} for $\\erm$,  for any $x$, $y\\in\\calC$ and $i$, \n\t\t\\begin{align}\n\t\t\t| f_i(y) -  f_i(x) - \\langle\\nabla f_i(x),  y- x\\rangle |\\le  L\\|x-y\\|^2 /2 \\ , \\text{ and } \\label{eq: fact: general basic ineq 1} \\\\\n","start":63068,"end":63427}]},{"id":"S017","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S017"],"declared_external":true,"declaration_text":["原始前提清单明确允许B0059作为不附文内证明的标准光滑性前提。"],"source_only":false,"title":"梯度Taylor余项","source_units":[{"raw_tex":"\t\t\t\\| \\nabla f_i(y) - \\nabla f_i(x) - \\langle\\nabla^2 f_i(x),  y- x\\rangle \\|_*\\le  \\hat{L}\\|x-y\\|^2 /2 \\ . \\label{eq: fact: general basic ineq 2}\n\t\t\\end{align}\n\t\\end{proposition}\n","start":63427,"end":63607}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":["def: tau"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"最近刷新时刻","source_units":[{"raw_tex":"\tFor any $k \\ge 0$ and $i\\in [n]$, we will be interested in the most recent iteration (up through $k$) at which Taylor point $b_i$ was updated.  This is defined as:\n\t\\begin{equation}\\label{def: tau}\n\t\t\\tau^k_i := \\max\\{t: t\\le k  \\text{ and } i \\in \\calB_t\\} \\ ,\n\t\\end{equation}\n\twhich means that in the first $k$ iterations, $\\tau^k_i$ is the latest iteration in which the Taylor point $b_i$ was updated. In \\cref{alg: TUFW in general finite sum}, the relationship between $\\tau^k_i$ and $\\tau^{k-1}_i$ is then:\n","start":63609,"end":64122}]},{"id":"S019","semantic_type":"derived_fact","kind":"derived_fact","labels":["eq: tauk relationship"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"刷新时刻递推","source_units":[{"raw_tex":"\t\\begin{equation}\\label{eq: tauk relationship}\n\t\t\\tau^{k}_i = \\left\\{\n\t\t\\begin{array}{ll}\n\t\t\tk & \\text{ if } \\  i\\in\\calB_k \\\\\n\t\t\t\\tau^{k-1}_i & \\text{ if } \\ i\\notin\\calB_k \\ . \n\t\t\\end{array}\n\t\t\\right.\n\t\\end{equation}\n","start":64122,"end":64341}]},{"id":"S020","semantic_type":"theorem","kind":"theorem","labels":["eq of thm: general convergence rate convex","thm: general convergence rate convex"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"随机TUFW凸收敛保证","source_units":[{"raw_tex":"\\begin{theorem}\\label{thm: general convergence rate convex}\n\t\tSuppose that $F$ is convex and Assumption \\ref{assump: ERM} holds, and  \\cref{alg: TUFW in general finite sum} with $\\rone$ is applied to the problem \\cref{pro: ERM} with step-sizes defined by $\\gamma_k:={2}/({k+2})$ for all $k \\ge 0$. Then for all $k\\ge 1$ we have:\n\t\t\\begin{equation}\\label{eq of thm: general convergence rate convex}\n\t\t\t\\mathbb{E}[F(x^{k}) - F(x^\\star)] \\le  \\frac{2L D^2 + 134 \\hat{L}D^3}{k+1} \\ .\n\t\t\\end{equation} \n\t\\end{theorem}","start":50034,"end":50546}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","labels":["eq: lm: general 3rd term upper bound","lm: general 3rd term upper bound"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"Taylor梯度方向误差界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lm: general 3rd term upper bound}\n\t\tUnder \\cref{assump: ERM} for the problem $\\erm$, for any $u,v \\in\\calC$,\n\t\t\\begin{equation}\\label{eq: lm: general 3rd term upper bound}\n\t\t\t\\left(\n\t\t\t\\nabla F(x^k)   - g^k\n\t\t\t\\right)^\\top (u - v) \\le \\frac{\\hat{L}D^3}{2n}\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j \\Big)^2\n\t\t\\end{equation}\n\t\tholds for iteration $k$ of Algorithm \\ref{alg: TUFW in general finite sum} for all $k \\ge 1$.\n\t\\end{lemma}","start":64436,"end":64901}]},{"id":"S022","semantic_type":"proof","kind":"proof","labels":["ineq: general etimate gradient 0","ineq: general etimate gradient 1"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"梯度误差引理完整证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{lm: general 3rd term upper bound}]\n\t\tFirst of all, $\\left(\n\t\t\\nabla F(x^k)   - g^k\n\t\t\\right)^\\top (u - v)$ can be rewritten as\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: general etimate gradient 0}\n\t\t\t\t\\frac{1}{n}\\sum_{i=1}^{n}\\left(\\nabla f_{i}\\big(x^k \\big) -   \\nabla f_{i}\\big( x^{\\tau^{k}_i}\\big) -  \\nabla^2 f_{i} \\big( x^{\\tau^{k}_i}\\big)\\big(x^k  - x^{\\tau^{k}_i} \\big)    \\right)^\\top (u - v),\n\t\t\t\\end{equation}\t\n\t\t\\end{small}which, since $\\|u-v\\| \\le D$, is smaller than or equal to \n\t\t\\begin{small}\\begin{equation}\\label{ineq: general etimate gradient 1}\n\t\t\t\\frac{D}{n}\\sum_{i=1}^{n}\\left\\| \\nabla f_{i}\\big(x^k \\big) -   \\nabla f_{i}\\big( x^{\\tau^{k}_i}\\big) -  \\nabla^2 f_{i} \\big( x^{\\tau^{k}_i}\\big)\\big(x^k  - x^{\\tau^{k}_i} \\big)    \\right\\|_* \\le    \\frac{\\hat{L}D}{2n}\\sum_{i=1}^{n}\\big\\|x^k  - x^{\\tau^{k}_i}\\big\\|^2,\n\t\t\\end{equation}\\end{small}where the inequality is due to \\cref{eq: fact: general basic ineq 2} in \\cref{prop: general basic ineq}.\n\t\tAdditionally, we have \n\t\t\\begin{small}\\begin{equation*}\n\t\t\t\\|x^k  - x^{\\tau^{k}_i}\n\t\t\t\\|^2 \\le  \\bigg( \\sum_{j = \\tau^k_i}^{k-1} \\| x^{j+1} -  x^{j}\\| \\bigg)^2\\le  \\bigg( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j D \\bigg)^2 \\  ,\n\t\t\\end{equation*}\\end{small}and substituting this last bound into \\cref{ineq: general etimate gradient 1} yields \\cref{eq: lm: general 3rd term upper bound}.\n\t\\end{proof}","start":64907,"end":66306}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: general etimate gradient 0"],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"误差的分量余项展开","source_units":[{"raw_tex":"\t\tFirst of all, $\\left(\n\t\t\\nabla F(x^k)   - g^k\n\t\t\\right)^\\top (u - v)$ can be rewritten as\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: general etimate gradient 0}\n\t\t\t\t\\frac{1}{n}\\sum_{i=1}^{n}\\left(\\nabla f_{i}\\big(x^k \\big) -   \\nabla f_{i}\\big( x^{\\tau^{k}_i}\\big) -  \\nabla^2 f_{i} \\big( x^{\\tau^{k}_i}\\big)\\big(x^k  - x^{\\tau^{k}_i} \\big)    \\right)^\\top (u - v),\n\t\t\t\\end{equation}\t\n","start":64971,"end":65363}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: general etimate gradient 1"],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"余项的范数控制","source_units":[{"raw_tex":"\t\t\\end{small}which, since $\\|u-v\\| \\le D$, is smaller than or equal to \n\t\t\\begin{small}\\begin{equation}\\label{ineq: general etimate gradient 1}\n\t\t\t\\frac{D}{n}\\sum_{i=1}^{n}\\left\\| \\nabla f_{i}\\big(x^k \\big) -   \\nabla f_{i}\\big( x^{\\tau^{k}_i}\\big) -  \\nabla^2 f_{i} \\big( x^{\\tau^{k}_i}\\big)\\big(x^k  - x^{\\tau^{k}_i} \\big)    \\right\\|_* \\le    \\frac{\\hat{L}D}{2n}\\sum_{i=1}^{n}\\big\\|x^k  - x^{\\tau^{k}_i}\\big\\|^2,\n\t\t\\end{equation}\\end{small}where the inequality is due to \\cref{eq: fact: general basic ineq 2} in \\cref{prop: general basic ineq}.\n","start":65363,"end":65911}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"轨迹长度控制陈旧误差","source_units":[{"raw_tex":"\t\tAdditionally, we have \n\t\t\\begin{small}\\begin{equation*}\n\t\t\t\\|x^k  - x^{\\tau^{k}_i}\n\t\t\t\\|^2 \\le  \\bigg( \\sum_{j = \\tau^k_i}^{k-1} \\| x^{j+1} -  x^{j}\\| \\bigg)^2\\le  \\bigg( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j D \\bigg)^2 \\  ,\n\t\t\\end{equation*}\\end{small}and substituting this last bound into \\cref{ineq: general etimate gradient 1} yields \\cref{eq: lm: general 3rd term upper bound}.\n","start":65911,"end":66294}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","labels":["eq lm: general convergence rate convex","lm: general convergence rate convex"],"member_ids":["S026"],"declared_external":false,"source_only":false,"title":"含梯度误差的凸收敛界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lm: general convergence rate convex}\n\t\tUnder \\cref{assump: ERM} for problem $\\erm$, suppose that $F$ is conevx, and let the step-sizes be given by $\\gamma_k=2/(k+2)$ for all $k \\ge 0$. Then for all $k \\ge 1$ we have\n\t\t\\begin{equation}\\label{eq lm: general convergence rate convex}\n\t\t\t\\eps_k  \\le  \\frac{2L D^2}{k+1}\n\t\t\t+  \\frac{2}{k(k+1)}\\sum_{t=1}^{k} t \\left(\n\t\t\t\\nabla F(x^{t-1})   - g^{t-1}\n\t\t\t\\right)^\\top (s^{t-1} - x^\\star) \\ .\n\t\t\\end{equation}\n\t\\end{lemma}","start":66455,"end":66939}]},{"id":"S027","semantic_type":"proof","kind":"proof","labels":["eq lm general convergence rate convex 0","eq lm general convergence rate convex 1"],"member_ids":["S027"],"declared_external":false,"source_only":false,"title":"凸差距引理完整证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{lm: general convergence rate convex}]\n\t\tUtilizing  \\cref{prop: general basic ineq} we have\n\t\t\\begin{small}\\begin{equation}\\label{eq lm general convergence rate convex 0}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\tF(x^{k+1})\\leq ~ & F(x^k)+\\gamma_{k}\\langle\\nabla F(x^k),s^{k}- x^k \\rangle+ \\gamma_{k}^{2} {L D^2}/{2} \\\\\n\t\t\t\t\t= ~ & F(x^{k})+\\gamma_{k}\\langle \\nabla F(x^{k})-g^k, s^{k}-x^k\\rangle +\\gamma_{k}\\langle{g^k}, s^{k}-x^k\\rangle+ \\gamma_{k}^{2} {L D^2}/{2} \\\\\n\t\t\t\t\t\\leq ~ & F(x^{k})+\\gamma_{k}\\langle\\nabla F(x^{k})-{g^k}, s^{k}-x^k\\rangle + \\gamma_{k}\\langle{g^k}, {x}^{\\star}-x^k\\rangle+ \\gamma_{k}^{2} {L D^2}/{2} \\\\\n\t\t\t\t\t= ~ & F(x^{k})+\\gamma_{k}\\langle \\nabla F(x^{k})-{g^k},s^{k}-{x}^{\\star}\\rangle +\\gamma_{k}\\langle \\nabla F(x^{k}), {x}^{\\star}-x^k\\rangle+ \\gamma_{k}^{2} {L D^2}/{2}\\\\\n\t\t\t\t\t\\le ~& F(x^{k})+\\gamma_{k}\\langle \\nabla F(x^{k})-{g^k},s^{k}-{x}^{\\star}\\rangle +\\gamma_{k}(F(x^\\star) - F(x^k))+ \\gamma_{k}^{2} {L D^2}/{2} \\ , \n\t\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses $s^{k} \\in \\arg\\min_{s\\in\\calC} (g^k)^\\top s$ from \\cref{alex} of the algorithm, and the third inequality uses the gradient inequality applied to $F$.  Subtracting $F^\\star$ from both sides of the above inequality chain, we arrive at:\n\t\t\\begin{small}\\begin{equation}\\label{eq lm general convergence rate convex 1}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t\\eps_{k+1}& \\le (1-\\gamma_k)\\eps_k + \\gamma_k \\big(\n\t\t\t\t\t\\nabla F(x^k)   - g^k\n\t\t\t\t\t\\big)^\\top (s^{k} - x^\\star) +  {\\gamma_k^2 L D^2}/{2} \\ .\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}Multiplying both side by $(k+1)(k+2)$ and telescoping the inequalities yields:\n\t\t\\begin{small}\\begin{equation}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t(k+1)(k+2)\\eps_{k+1}  &\\le  2(k+1)L D^2\n\t\t\t\t\t+  \\sum_{t=0}^{k}2(t+1) (\n\t\t\t\t\t\\nabla F(x^t)   - g^t\n\t\t\t\t\t)^\\top (s^{t} - x^\\star) \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}which yields \\cref{eq lm: general convergence rate convex} after rearranging and re-indexing the counter $t$.\n\t\\end{proof}","start":66941,"end":68945}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","labels":["eq lm general convergence rate convex 0"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"下降界与凸梯度比较","source_units":[{"raw_tex":"\t\tUtilizing  \\cref{prop: general basic ineq} we have\n\t\t\\begin{small}\\begin{equation}\\label{eq lm general convergence rate convex 0}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\tF(x^{k+1})\\leq ~ & F(x^k)+\\gamma_{k}\\langle\\nabla F(x^k),s^{k}- x^k \\rangle+ \\gamma_{k}^{2} {L D^2}/{2} \\\\\n\t\t\t\t\t= ~ & F(x^{k})+\\gamma_{k}\\langle \\nabla F(x^{k})-g^k, s^{k}-x^k\\rangle +\\gamma_{k}\\langle{g^k}, s^{k}-x^k\\rangle+ \\gamma_{k}^{2} {L D^2}/{2} \\\\\n\t\t\t\t\t\\leq ~ & F(x^{k})+\\gamma_{k}\\langle\\nabla F(x^{k})-{g^k}, s^{k}-x^k\\rangle + \\gamma_{k}\\langle{g^k}, {x}^{\\star}-x^k\\rangle+ \\gamma_{k}^{2} {L D^2}/{2} \\\\\n\t\t\t\t\t= ~ & F(x^{k})+\\gamma_{k}\\langle \\nabla F(x^{k})-{g^k},s^{k}-{x}^{\\star}\\rangle +\\gamma_{k}\\langle \\nabla F(x^{k}), {x}^{\\star}-x^k\\rangle+ \\gamma_{k}^{2} {L D^2}/{2}\\\\\n\t\t\t\t\t\\le ~& F(x^{k})+\\gamma_{k}\\langle \\nabla F(x^{k})-{g^k},s^{k}-{x}^{\\star}\\rangle +\\gamma_{k}(F(x^\\star) - F(x^k))+ \\gamma_{k}^{2} {L D^2}/{2} \\ , \n\t\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses $s^{k} \\in \\arg\\min_{s\\in\\calC} (g^k)^\\top s$ from \\cref{alex} of the algorithm, and the third inequality uses the gradient inequality applied to $F$.  Subtracting $F^\\star$ from both sides of the above inequality chain, we arrive at:\n","start":67008,"end":68220}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","labels":["eq lm general convergence rate convex 1"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"最优性差距递推","source_units":[{"raw_tex":"\t\t\\begin{small}\\begin{equation}\\label{eq lm general convergence rate convex 1}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t\\eps_{k+1}& \\le (1-\\gamma_k)\\eps_k + \\gamma_k \\big(\n\t\t\t\t\t\\nabla F(x^k)   - g^k\n\t\t\t\t\t\\big)^\\top (s^{k} - x^\\star) +  {\\gamma_k^2 L D^2}/{2} \\ .\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}Multiplying both side by $(k+1)(k+2)$ and telescoping the inequalities yields:\n","start":68220,"end":68590}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"加权望远镜求和","source_units":[{"raw_tex":"\t\t\\begin{small}\\begin{equation}\n\t\t\t\t\\begin{aligned}\n\t\t\t\t\t(k+1)(k+2)\\eps_{k+1}  &\\le  2(k+1)L D^2\n\t\t\t\t\t+  \\sum_{t=0}^{k}2(t+1) (\n\t\t\t\t\t\\nabla F(x^t)   - g^t\n\t\t\t\t\t)^\\top (s^{t} - x^\\star) \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}which yields \\cref{eq lm: general convergence rate convex} after rearranging and re-indexing the counter $t$.\n","start":68590,"end":68933}]},{"id":"S031","semantic_type":"lemma","kind":"lemma","labels":["eq: lm: convex EZ","lm: convex EZ"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"随机陈旧和的矩界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lm: convex EZ} Let the series $\\{d_{ij}\\}_{i\\in[n],j\\in\\mathbb{N}}$ be given, and suppose that there exist $M_{1}$ and $M_{\\infty}$ for which $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$ and  $\\sup_{i,j} d_{ij}\\le M_{\\infty}$.  \n\t\tLet $\\tau_i^k$ be defined in \\cref{def: tau}, in which $\\calB_t$ is constructed using $\\rone$ in \\cref{rule 1}. Let $\\gamma_k$ be ${2}/(k+2)$ for all $k \\ge 0$.  Then it holds that\n\t\t\\begin{equation}\\label{eq: lm: convex EZ}\n\t\t\t\\mathbb{E}\\bigg[\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\Big)\\bigg] \\le\n\t\t\t\\frac{6 M_1}{\\sqrt{k+2}} \\ \\ \\text{ and }   \\ \\\n\t\t\t\\mathbb{E} \\bigg[\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\Big)^2 \\bigg]\\le  \\frac{134M_1M_\\infty}{k+2} \\ .\n\t\t\\end{equation}\n\t\\end{lemma}","start":69084,"end":69867}]},{"id":"S032","semantic_type":"assumption","kind":"assumption","labels":["lm: convex EZ"],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"矩界引理的显式条件","source_units":[{"raw_tex":"\\begin{lemma}\\label{lm: convex EZ} Let the series $\\{d_{ij}\\}_{i\\in[n],j\\in\\mathbb{N}}$ be given, and suppose that there exist $M_{1}$ and $M_{\\infty}$ for which $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$ and  $\\sup_{i,j} d_{ij}\\le M_{\\infty}$.  \n\t\tLet $\\tau_i^k$ be defined in \\cref{def: tau}, in which $\\calB_t$ is constructed using $\\rone$ in \\cref{rule 1}. Let $\\gamma_k$ be ${2}/(k+2)$ for all $k \\ge 0$.  Then it holds that\n","start":69084,"end":69512}]},{"id":"S033","semantic_type":"lemma","kind":"lemma","labels":["lm: convex EZ"],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"陈旧和一阶矩界","source_units":[{"raw_tex":"\t\t\\begin{equation}\\label{eq: lm: convex EZ}\n\t\t\t\\mathbb{E}\\bigg[\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\Big)\\bigg] \\le\n\t\t\t\\frac{6 M_1}{\\sqrt{k+2}} \\ \\ \\text{ and }   \\ \\\n","start":69512,"end":69705}]},{"id":"S034","semantic_type":"lemma","kind":"lemma","labels":["lm: convex EZ"],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"陈旧和平方矩界","source_units":[{"raw_tex":"\t\t\t\\mathbb{E} \\bigg[\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\Big)^2 \\bigg]\\le  \\frac{134M_1M_\\infty}{k+2} \\ .\n\t\t\\end{equation}\n\t\\end{lemma}","start":69705,"end":69867}]},{"id":"S035","semantic_type":"proposition","kind":"proposition","labels":["eq: fact 2","lm: rho"],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"刷新存活乘积求和界","source_units":[{"raw_tex":"\\begin{proposition}\\label{lm: rho}\n\t\tFor integers $v$ and $k$ satisfying $1\\le  v \\le k$ it holds that:\n\t\t\\begin{equation}\\label{eq: fact 2}\n\t\t\t\\sum_{i=1}^{v}\\prod_{j=i}^{k}\\left(1-\\tfrac{1}{\\sqrt{j}}\\right) <\n\t\t\t(\\sqrt{v+1})\\exp(2\\sqrt{v+1} - 2\\sqrt{k+1})  \\   .\n\t\t\\end{equation}\n\t\\end{proposition}","start":69992,"end":70291}]},{"id":"S036","semantic_type":"proof","kind":"proof","labels":["lulu"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"乘积求和界完整证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{lm: rho}]\n\t\tFor convenience let us define $\\rho_j := 1-\\tfrac{1}{\\sqrt{j}}$ for $j \\ge 1$, and define the function $h(x):= \\exp(2 \\sqrt{x} - 2 \\sqrt{k+1}) $ for $x \\ge 0$.  We will first prove the following inequality:\n\t\t\\begin{small}\\begin{equation}\\label{lulu}\n\t\t\t\\prod_{j=i}^{k}\\left(1-\\tfrac{1}{\\sqrt{j}}\\right) \\le h(i) \\ \\ \\ \\mathrm{for~all~} i \\in [k] \\ . \n\t\t\\end{equation}\\end{small}For $i=1$ the left side above is $0$ (because $\\rho_1 = 0$ and hence $\\prod_{j=1}^{k}\\rho_j = 0$), and the right side is $h(1) >0$, so \\cref{lulu} is true for $i=1$. For $i\\ge2$ we have:\n\t\t\\begin{small}\\begin{equation*}\\begin{aligned}\n\t\t\t\t\\prod_{j=i}^{k}\\rho_j  &= \\exp\\left({\\sum_{j=i}^{k}\\ln(1-\\tfrac{1}{\\sqrt{j}})}\\right) \\le \\exp\\left( -\\sum_{j=i}^{k}\\tfrac{1}{\\sqrt{j}}\\right)  \\\\  \n\t\t\t\t&\\le \\exp\\Big( -\\int_i^{k+1} x^{-\\tfrac{1}{2}} dx\\Big) =\\exp(2\\sqrt{i} - 2\\sqrt{k+1}) = h(i) \\ , \\end{aligned} \n\t\t\\end{equation*}\\end{small}where the first inequality follows from the fact that $\\log(1-x)\\le -x$ for any $0\\le x <1$, and the second inequality is an integral bound. This establishes \\cref{lulu}, whereby we have\n\t\t\\begin{small}$$ \\begin{aligned}\n\t\t\t\t\\sum_{i=1}^{v}   \\prod_{j=i}^{k}\\rho_j  &\\le  \\sum_{i=1}^{v} h(i) \\le \\int_{1}^{v+1}h(x)~dx = \\left. (\\sqrt{x}-\\tfrac{1}{2})\\exp(2\\sqrt{x} - 2\\sqrt{k+1}) \\right|_1^{v+1} \\\\ \n\t\t\t\t  &= (\\sqrt{v+1}-\\tfrac{1}{2})\\exp(2\\sqrt{v+1} - 2\\sqrt{k+1}) - (\\tfrac{1}{2})\\exp(2 - 2\\sqrt{k+1}) \\\\\\\n\t\t\t\t&< (\\sqrt{v+1}-\\tfrac{1}{2})\\exp(2\\sqrt{v+1} - 2\\sqrt{k+1}) \\ ,\n\t\t\\end{aligned}$$\\end{small}where the second inequality is an integral bound using the fact that $h(x)$ is an increasing function. \\end{proof}","start":70293,"end":71962}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","labels":["lulu"],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"单个存活乘积指数界","source_units":[{"raw_tex":"\t\tFor convenience let us define $\\rho_j := 1-\\tfrac{1}{\\sqrt{j}}$ for $j \\ge 1$, and define the function $h(x):= \\exp(2 \\sqrt{x} - 2 \\sqrt{k+1}) $ for $x \\ge 0$.  We will first prove the following inequality:\n\t\t\\begin{small}\\begin{equation}\\label{lulu}\n\t\t\t\\prod_{j=i}^{k}\\left(1-\\tfrac{1}{\\sqrt{j}}\\right) \\le h(i) \\ \\ \\ \\mathrm{for~all~} i \\in [k] \\ . \n\t\t\\end{equation}\\end{small}For $i=1$ the left side above is $0$ (because $\\rho_1 = 0$ and hence $\\prod_{j=1}^{k}\\rho_j = 0$), and the right side is $h(1) >0$, so \\cref{lulu} is true for $i=1$. For $i\\ge2$ we have:\n\t\t\\begin{small}\\begin{equation*}\\begin{aligned}\n\t\t\t\t\\prod_{j=i}^{k}\\rho_j  &= \\exp\\left({\\sum_{j=i}^{k}\\ln(1-\\tfrac{1}{\\sqrt{j}})}\\right) \\le \\exp\\left( -\\sum_{j=i}^{k}\\tfrac{1}{\\sqrt{j}}\\right)  \\\\  \n\t\t\t\t&\\le \\exp\\Big( -\\int_i^{k+1} x^{-\\tfrac{1}{2}} dx\\Big) =\\exp(2\\sqrt{i} - 2\\sqrt{k+1}) = h(i) \\ , \\end{aligned} \n\t\t\\end{equation*}\\end{small}where the first inequality follows from the fact that $\\log(1-x)\\le -x$ for any $0\\le x <1$, and the second inequality is an integral bound. This establishes \\cref{lulu}, whereby we have\n","start":70332,"end":71432}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"指数函数求和积分界","source_units":[{"raw_tex":"\t\t\\begin{small}$$ \\begin{aligned}\n\t\t\t\t\\sum_{i=1}^{v}   \\prod_{j=i}^{k}\\rho_j  &\\le  \\sum_{i=1}^{v} h(i) \\le \\int_{1}^{v+1}h(x)~dx = \\left. (\\sqrt{x}-\\tfrac{1}{2})\\exp(2\\sqrt{x} - 2\\sqrt{k+1}) \\right|_1^{v+1} \\\\ \n\t\t\t\t  &= (\\sqrt{v+1}-\\tfrac{1}{2})\\exp(2\\sqrt{v+1} - 2\\sqrt{k+1}) - (\\tfrac{1}{2})\\exp(2 - 2\\sqrt{k+1}) \\\\\\\n\t\t\t\t&< (\\sqrt{v+1}-\\tfrac{1}{2})\\exp(2\\sqrt{v+1} - 2\\sqrt{k+1}) \\ ,\n\t\t\\end{aligned}$$\\end{small}where the second inequality is an integral bound using the fact that $h(x)$ is an increasing function. \\end{proof}","start":71432,"end":71962}]},{"id":"S039","semantic_type":"lemma","kind":"lemma","labels":["ineq: lm: simple ineq","lm: simple ineq"],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"半程指数项多项式界","source_units":[{"raw_tex":"\\begin{lemma}\\label{lm: simple ineq}\n\t\tFor any $k\\in\\mathbb{N}$ the following inequality holds:\n\t\t\\begin{equation}\\label{ineq: lm: simple ineq} \n\t\t\t\\sqrt{\\lfloor k/2 \\rfloor + 1} \\exp(2\\sqrt{\\lfloor k/2\\rfloor + 1} - 2\\sqrt{k+1}) \\le \\min\\{{3}(k+2)^{-\\frac{1}{2}},{10}(k+2)^{-1}\\}. \n\t\t\\end{equation}\n\t\\end{lemma}","start":72091,"end":72403}]},{"id":"S040","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"半程项引理完整证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{lm: simple ineq}]\n\t\tSince $h(x):=\\sqrt{x}$ is a concave function on $\\mathbb{R}_+$, we have $h(k+2) \\le h(k+1) + h'(k+1)$, which yields $\\sqrt{k+1}\\ge \\sqrt{k+2} - \\frac{1}{2\\sqrt{k+1}} \\ge \\sqrt{k+2} - \\sqrt{2}/4 $ when $k \\ge 1$.\n\t\tFurthermore $\\sqrt{\\lfloor k/2 \\rfloor + 1} \\le \\frac{\\sqrt{2}}{2} \\sqrt{k+2}$, and therefore \n\t\t\\begin{small}\\begin{equation*}\n\t\t\t\\sqrt{\\lfloor k/2 \\rfloor + 1} \\exp(2\\sqrt{\\lfloor k/2\\rfloor + 1} - 2\\sqrt{k+1}) \\le  \\frac{\\sqrt{2}e^{\\frac{\\sqrt{2}}{2}}}{2}(k+2)^{\\frac{1}{2}}  \\exp(-(2-\\sqrt{2})(k+2)^{\\frac{1}{2}}) \\ .\n\t\t\\end{equation*}\\end{small}Next note that the function $h(x):=e^{-x} x^t$ on the domain $x \\in (0,\\infty)$ for $t > 0$ attains its maximal value at $x = t$. In particular for $t=2$ and $t=3$, respectively, we conclude that $e^{-x} \\le 4e^{-2}x^{-2}$ and $e^{-x} \\le 27e^{-3}x^{-3}$ (respectively), and thus  $\\exp(-(2-\\sqrt{2})(k+2)^{\\frac{1}{2}}) \\le 4 e^{-2}(2-\\sqrt{2})^{-2}(k+2)^{-1}$\n\t\tand $\\exp(-(2-\\sqrt{2})(k+2)^{\\frac{1}{2}}) \\le 27 e^{-3}(2-\\sqrt{2})^{-3}(k+2)^{-3/2}$ (respectively).\n\t\tTherefore, the left-hand side of \\cref{ineq: lm: simple ineq} is bounded by \n\t\t$2\\sqrt{2}e^{\\frac{\\sqrt{2}}{2}-2} (2-\\sqrt{2})^{-2} (k+2)^{-\\frac{1}{2}}$ and \n\t\t$\\frac{27}{2}\\sqrt{2}e^{\\frac{\\sqrt{2}}{2}-3} (2-\\sqrt{2})^{-3} (k+2)^{-1}$, respectively, which is less than $3(k+2)^{-\\frac{1}{2}}$ and $10(k+2)^{-1}$, respectively.\n\t\\end{proof}","start":72405,"end":73829}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"平方根凹性归一化","source_units":[{"raw_tex":"\t\tSince $h(x):=\\sqrt{x}$ is a concave function on $\\mathbb{R}_+$, we have $h(k+2) \\le h(k+1) + h'(k+1)$, which yields $\\sqrt{k+1}\\ge \\sqrt{k+2} - \\frac{1}{2\\sqrt{k+1}} \\ge \\sqrt{k+2} - \\sqrt{2}/4 $ when $k \\ge 1$.\n\t\tFurthermore $\\sqrt{\\lfloor k/2 \\rfloor + 1} \\le \\frac{\\sqrt{2}}{2} \\sqrt{k+2}$, and therefore \n\t\t\\begin{small}\\begin{equation*}\n\t\t\t\\sqrt{\\lfloor k/2 \\rfloor + 1} \\exp(2\\sqrt{\\lfloor k/2\\rfloor + 1} - 2\\sqrt{k+1}) \\le  \\frac{\\sqrt{2}e^{\\frac{\\sqrt{2}}{2}}}{2}(k+2)^{\\frac{1}{2}}  \\exp(-(2-\\sqrt{2})(k+2)^{\\frac{1}{2}}) \\ .\n","start":72452,"end":72990}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"指数衰减转为幂次界","source_units":[{"raw_tex":"\t\t\\end{equation*}\\end{small}Next note that the function $h(x):=e^{-x} x^t$ on the domain $x \\in (0,\\infty)$ for $t > 0$ attains its maximal value at $x = t$. In particular for $t=2$ and $t=3$, respectively, we conclude that $e^{-x} \\le 4e^{-2}x^{-2}$ and $e^{-x} \\le 27e^{-3}x^{-3}$ (respectively), and thus  $\\exp(-(2-\\sqrt{2})(k+2)^{\\frac{1}{2}}) \\le 4 e^{-2}(2-\\sqrt{2})^{-2}(k+2)^{-1}$\n\t\tand $\\exp(-(2-\\sqrt{2})(k+2)^{\\frac{1}{2}}) \\le 27 e^{-3}(2-\\sqrt{2})^{-3}(k+2)^{-3/2}$ (respectively).\n\t\tTherefore, the left-hand side of \\cref{ineq: lm: simple ineq} is bounded by \n\t\t$2\\sqrt{2}e^{\\frac{\\sqrt{2}}{2}-2} (2-\\sqrt{2})^{-2} (k+2)^{-\\frac{1}{2}}$ and \n\t\t$\\frac{27}{2}\\sqrt{2}e^{\\frac{\\sqrt{2}}{2}-3} (2-\\sqrt{2})^{-3} (k+2)^{-1}$, respectively, which is less than $3(k+2)^{-\\frac{1}{2}}$ and $10(k+2)^{-1}$, respectively.\n","start":72990,"end":73817}]},{"id":"S043","semantic_type":"proof","kind":"proof","labels":["ineq: G_k recurrent sumpro 1","ineq: G_k recurrent sumpro 2","ineq: Z_k recurrent sumpro 1","ineq: Z_k recurrent sumpro 2","ineq: convex EG_k final","ineq: convex EZ_k final","ineq: recurrent G_k 2","ineq: recurrent G_k 3","ineq: recurrent Z_k 4","ineq: recurrent Z_k 5","ncl","nclz"],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"随机矩界引理完整证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{lm: convex EZ}]\n\t\t\n\t\tFirst of all, let us define $G_k^i := \\left( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\right)$ and $Z_k^i :=\\left( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\right)^2$ for $i \\in [n]$, and also $G_k :=\\sum_{i=1}^n G_k^i$ and $Z_k :=\\sum_{i=1}^n Z_k^i$. Notice in $G_k^i$ and $Z_k^i$ that the randomness lies only in $\\tau_i^k$, which in turn depends on the update sets $\\calB_0, \\calB_1,\\dots,\\calB_k$. \n\t\t\n\t\tWe first analyze $G_k^i$ and $G_k$.  We note that $G_k^i = 0$ when $i \\in \\calB_k$, which happens with probability $\\beta_k/n$.  Because of the relationship between $\\{\\tau^k_i\\}$ and $\\{\\tau^{k-1}_i\\}$ in \\cref{eq: tauk relationship}, we further note that with probability $1-\\beta_k/n$ that $i \\notin \\calB_k$ in which case it follows that $G_k^i = G_{k-1}^i + \\gamma_{k-1}d_{i,k-1}$.  Defining $\\rho_k := 1 - \\frac{1}{\\sqrt{k}} = 1 - \\beta_k/n $, we have \n\t\t\\begin{small}\\begin{equation}\\label{ncl}\n\t\t\t\\mathbb{E}[ G_k ] = \\sum_{i=1}^n \\mathbb{E} [G_k^i ]= \\rho_k \\sum_{i=1}^n \\mathbb{E} [G_{k-1}^i +\n\t\t\t\\gamma_{k-1} d_{i,k-1}] \\le \\rho_k\\left( \\mathbb{E} [G_{k-1} ]+\n\t\t\t\\gamma_{k-1} M_1 \\right) \\ ,\n\t\t\\end{equation}\\end{small}where the inequality uses $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$. Recurrently applying \\cref{ncl} yields:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent G_k 2}\n\t\t\t \\mathbb{E}[ G_k ] \\le \\rho_k( \\mathbb{E}[ G_{k-1} ]+\n\t\t\t\\gamma_{k-1}  M_1 ) \\le\\cdots\\le \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\gamma_{i-1} M_1 + \\Big(\\prod_{j=1}^{k} \\rho_j\\Big)\\mathbb{E}[ G_0] \\ .\n\t\t\\end{equation}\\end{small}Noting that $G_0 = 0$ (since all Taylor points are set to $x^0$ in \\cref{bettina} of \\cref{alg: TUFW in general finite sum}) and using $\\gamma_{i-1} = \\frac{2}{i+1}$ it follows that:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent G_k 3}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\mathbb{E} [G_k] &\\le \\sum_{i=1}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{i+1}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor {k}/{2}\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{i+1}\\\\\n\t\t\t\t& \\le \\sum_{i=2}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{3}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor {k}/{2}\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor {k}/{2}\\rfloor+2} \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses the fact that $\\rho_1 = 0$ and hence $\\prod_{j=1}^{k}\\rho_j = 0$.\n\t\tBy \\cref{lm: rho} we have:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: G_k recurrent sumpro 1}\n\t\t\t\\sum_{i=1}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)  \\frac{2M_1}{3}\n\t\t\t\\le\n\t\t\t\\sqrt{\\lfloor {k}/{2}\n\t\t\t\t\\rfloor + 1} \\exp\\Big(2 \\sqrt{ \\lfloor {k}/{2} \\rfloor + 1} - 2\\sqrt{k+1}\\Big)  \\frac{2M_1}{3} \\le \\frac{2 M_1}{\\sqrt{k+2}} \\ ,\n\t\t\\end{equation}\\end{small}where the second inequality above uses \\cref{lm: simple ineq}.\n\t\tSimilarly for the second term in the rightmost side of \\cref{ineq: recurrent G_k 3} we have:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: G_k recurrent sumpro 2}\n\t\t\t\\sum_{i=\\lfloor k/2 \\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor k/2\\rfloor+2} \\le   \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor k/2\\rfloor+2} \\le \\frac{\\sqrt{k+1} \\cdot  2M_1}{\\lfloor k/2\\rfloor+2} \\le \\frac{4M_1}{\\sqrt{k+2}} \\ , \n\t\t\\end{equation}\\end{small}where the second inequality uses \\cref{lm: rho}.\n\t\tSubstituting \\cref{ineq: G_k recurrent sumpro 1} and \\cref{ineq: G_k recurrent sumpro 2} back into \\cref{ineq: recurrent G_k 3} yields the first part of \\cref{eq: lm: convex EZ}, namely:\n\t\t\\begin{equation}\\label{ineq: convex EG_k final}\n\t\t\t\\mathbb{E} [G_k] \\le {6 M_1}/{\\sqrt{k+2}} \\ .\n\t\t\\end{equation}\n\t\t\n\t\tWe employ a similar proof approach to establish the second part of \\cref{eq: lm: convex EZ}. We note that $Z_k^i = 0$ when $i \\in \\calB_k$, which happens with probability $\\beta_k/n$.  Because of the relationship between $\\{\\tau^k_i\\}$ and $\\{\\tau^{k-1}_i\\}$ in \\cref{eq: tauk relationship}, we further note that with probability $1-\\beta_k/n$ that $i \\notin \\calB_k$ in which case we have\n\t\t\\begin{small}\\begin{equation*}\n\t\t\t\\begin{aligned} \n\t\t\t\tZ_k^i &= (G_{k-1}^i + \\gamma_{k-1}d_{i,k-1})^2 = Z_{k-1}^i + 2 G_{k-1}^i\\gamma_{k-1}d_{i,k-1} + (\\gamma_{k-1}d_{i,k-1})^2 \\\\ &\\le Z_{k-1}^i + 2 M_\\infty \\gamma_{k-1}G_{k-1}^i + \\gamma_{k-1}^2 M_\\infty d_{i,k-1} \\ . \n\t\t\t\\end{aligned}\\end{equation*}\\end{small}Therefore\n\t\t\\begin{small}\\begin{equation}\\begin{aligned}\\label{nclz}\n\t\t\t\t\\mathbb{E}[ Z_k ] = &\\sum_{i=1}^n \\mathbb{E} [Z_k^i ] \\le \\rho_k \\sum_{i=1}^n \\mathbb{E} [Z_{k-1}^i + 2 M_\\infty \\gamma_{k-1}G_{k-1}^i + \\gamma_{k-1}^2 M_\\infty d_{i,k-1}] \\\\ \n\t\t\t\t&\\le \\rho_k \\left( \\mathbb{E} [Z_{k-1}] + 2   M_\\infty \\gamma_{k-1}\\mathbb{E}[G_{k-1}] +   \\gamma_{k-1}^2 M_\\infty M_1 \\right) \\\\\n\t\t\t\t&\\le \\rho_k \\left(\\mathbb{E} [Z_{k-1}] + \\frac{24M_\\infty M_1}{(k+1)\\sqrt{k+1}}   +  \\frac{4M_\\infty M_1}{(k+1)^2} \\right) \\le \\rho_k \\left(\\mathbb{E} [Z_{k-1}] + \\frac{28M_\\infty M_1}{(k+1)\\sqrt{k+1}}  \\right)    \\ ,\n\t\t\\end{aligned}\\end{equation} \\end{small}where the second inequality uses $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$, and the third inequality uses \\cref{ineq: convex EG_k final}. Recurrently applying \\cref{nclz} yields:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent Z_k 4}\n\t\t\t\\mathbb{E} [Z_k]  \\le \\rho_k\\bigg( \\mathbb{E} [Z_{k-1} ]+\n\t\t\t\\frac{28M_1M_\\infty}{(k+1)^{3/2}}\\bigg) \\le \\cdots\\le \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}  \\ .\n\t\t\\end{equation}\\end{small}Therefore\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent Z_k 5}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\mathbb{E} [Z_k]  & \\le  \\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor k/2\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}\\\\\n\t\t\t\t& \\le \\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{3\\sqrt{3}}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor k/2\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\ .\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}By \\cref{lm: rho} we have \n\t\t\t\\begin{small}\\begin{equation}\\label{ineq: Z_k recurrent sumpro 1}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{3\\sqrt{3}}  &\\le \\sqrt{\\lfloor k/2\n\t\t\t\t\t\\rfloor + 1} \\exp\\big(2 \\sqrt{ \\lfloor k/2 \\rfloor + 1} - 2\\sqrt{k+1}\\big) \\cdot \\frac{28M_1M_\\infty}{(3\\sqrt{3})} \\\\ \n\t\t\t\t&\\le \\frac{{54M_1M_\\infty}}{k+2} \\ , \n\t\t\\end{aligned}\\end{equation}\\end{small}where the second inequality above uses \\cref{lm: simple ineq}. \n\t\tSimilarly for the second term in the rightmost side of \\cref{ineq: recurrent Z_k 5} we have:\n\t\t\t\\begin{small}\\begin{equation}\n\t\t\t\\begin{aligned}\\label{ineq: Z_k recurrent sumpro 2}\n\t\t\t\t\\sum_{i=\\lfloor k/2 \\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} &\\le   \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\\\ \n\t\t\t\t&\\le \\sqrt{k+1} \\cdot \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\le \\frac{80M_1M_\\infty}{k+2} \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses \\cref{lm: rho}.\n\t\tSubstituting \\cref{ineq: Z_k recurrent sumpro 1} and \\cref{ineq: Z_k recurrent sumpro 2} back into \\cref{ineq: recurrent Z_k 5} yields the second part of \\cref{eq: lm: convex EZ}, namely:\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: convex EZ_k final}\n\t\t\t\t\\mathbb{E} [Z_k] \\le  \\frac{134M_1M_\\infty}{k+2} \\ .\n\t\t\t\\end{equation}\n\t\t\\end{small}\\end{proof}","start":73888,"end":81481}]},{"id":"S044","semantic_type":"proof","kind":"proof","labels":["ineq: G_k recurrent sumpro 1","ineq: G_k recurrent sumpro 2","ineq: convex EG_k final","ineq: recurrent G_k 2","ineq: recurrent G_k 3","ncl"],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"一阶矩部分作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{lm: convex EZ}]\n\t\t\n\t\tFirst of all, let us define $G_k^i := \\left( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\right)$ and $Z_k^i :=\\left( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\right)^2$ for $i \\in [n]$, and also $G_k :=\\sum_{i=1}^n G_k^i$ and $Z_k :=\\sum_{i=1}^n Z_k^i$. Notice in $G_k^i$ and $Z_k^i$ that the randomness lies only in $\\tau_i^k$, which in turn depends on the update sets $\\calB_0, \\calB_1,\\dots,\\calB_k$. \n\t\t\n\t\tWe first analyze $G_k^i$ and $G_k$.  We note that $G_k^i = 0$ when $i \\in \\calB_k$, which happens with probability $\\beta_k/n$.  Because of the relationship between $\\{\\tau^k_i\\}$ and $\\{\\tau^{k-1}_i\\}$ in \\cref{eq: tauk relationship}, we further note that with probability $1-\\beta_k/n$ that $i \\notin \\calB_k$ in which case it follows that $G_k^i = G_{k-1}^i + \\gamma_{k-1}d_{i,k-1}$.  Defining $\\rho_k := 1 - \\frac{1}{\\sqrt{k}} = 1 - \\beta_k/n $, we have \n\t\t\\begin{small}\\begin{equation}\\label{ncl}\n\t\t\t\\mathbb{E}[ G_k ] = \\sum_{i=1}^n \\mathbb{E} [G_k^i ]= \\rho_k \\sum_{i=1}^n \\mathbb{E} [G_{k-1}^i +\n\t\t\t\\gamma_{k-1} d_{i,k-1}] \\le \\rho_k\\left( \\mathbb{E} [G_{k-1} ]+\n\t\t\t\\gamma_{k-1} M_1 \\right) \\ ,\n\t\t\\end{equation}\\end{small}where the inequality uses $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$. Recurrently applying \\cref{ncl} yields:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent G_k 2}\n\t\t\t \\mathbb{E}[ G_k ] \\le \\rho_k( \\mathbb{E}[ G_{k-1} ]+\n\t\t\t\\gamma_{k-1}  M_1 ) \\le\\cdots\\le \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\gamma_{i-1} M_1 + \\Big(\\prod_{j=1}^{k} \\rho_j\\Big)\\mathbb{E}[ G_0] \\ .\n\t\t\\end{equation}\\end{small}Noting that $G_0 = 0$ (since all Taylor points are set to $x^0$ in \\cref{bettina} of \\cref{alg: TUFW in general finite sum}) and using $\\gamma_{i-1} = \\frac{2}{i+1}$ it follows that:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent G_k 3}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\mathbb{E} [G_k] &\\le \\sum_{i=1}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{i+1}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor {k}/{2}\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{i+1}\\\\\n\t\t\t\t& \\le \\sum_{i=2}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{3}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor {k}/{2}\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor {k}/{2}\\rfloor+2} \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses the fact that $\\rho_1 = 0$ and hence $\\prod_{j=1}^{k}\\rho_j = 0$.\n\t\tBy \\cref{lm: rho} we have:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: G_k recurrent sumpro 1}\n\t\t\t\\sum_{i=1}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)  \\frac{2M_1}{3}\n\t\t\t\\le\n\t\t\t\\sqrt{\\lfloor {k}/{2}\n\t\t\t\t\\rfloor + 1} \\exp\\Big(2 \\sqrt{ \\lfloor {k}/{2} \\rfloor + 1} - 2\\sqrt{k+1}\\Big)  \\frac{2M_1}{3} \\le \\frac{2 M_1}{\\sqrt{k+2}} \\ ,\n\t\t\\end{equation}\\end{small}where the second inequality above uses \\cref{lm: simple ineq}.\n\t\tSimilarly for the second term in the rightmost side of \\cref{ineq: recurrent G_k 3} we have:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: G_k recurrent sumpro 2}\n\t\t\t\\sum_{i=\\lfloor k/2 \\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor k/2\\rfloor+2} \\le   \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor k/2\\rfloor+2} \\le \\frac{\\sqrt{k+1} \\cdot  2M_1}{\\lfloor k/2\\rfloor+2} \\le \\frac{4M_1}{\\sqrt{k+2}} \\ , \n\t\t\\end{equation}\\end{small}where the second inequality uses \\cref{lm: rho}.\n\t\tSubstituting \\cref{ineq: G_k recurrent sumpro 1} and \\cref{ineq: G_k recurrent sumpro 2} back into \\cref{ineq: recurrent G_k 3} yields the first part of \\cref{eq: lm: convex EZ}, namely:\n\t\t\\begin{equation}\\label{ineq: convex EG_k final}\n\t\t\t\\mathbb{E} [G_k] \\le {6 M_1}/{\\sqrt{k+2}} \\ .\n\t\t\\end{equation}\n","start":73888,"end":77572}]},{"id":"S045","semantic_type":"proof","kind":"proof","labels":["ineq: Z_k recurrent sumpro 1","ineq: Z_k recurrent sumpro 2","ineq: convex EZ_k final","ineq: recurrent Z_k 4","ineq: recurrent Z_k 5","nclz"],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"平方矩部分作者证明","source_units":[{"raw_tex":"\t\t\n\t\tWe employ a similar proof approach to establish the second part of \\cref{eq: lm: convex EZ}. We note that $Z_k^i = 0$ when $i \\in \\calB_k$, which happens with probability $\\beta_k/n$.  Because of the relationship between $\\{\\tau^k_i\\}$ and $\\{\\tau^{k-1}_i\\}$ in \\cref{eq: tauk relationship}, we further note that with probability $1-\\beta_k/n$ that $i \\notin \\calB_k$ in which case we have\n\t\t\\begin{small}\\begin{equation*}\n\t\t\t\\begin{aligned} \n\t\t\t\tZ_k^i &= (G_{k-1}^i + \\gamma_{k-1}d_{i,k-1})^2 = Z_{k-1}^i + 2 G_{k-1}^i\\gamma_{k-1}d_{i,k-1} + (\\gamma_{k-1}d_{i,k-1})^2 \\\\ &\\le Z_{k-1}^i + 2 M_\\infty \\gamma_{k-1}G_{k-1}^i + \\gamma_{k-1}^2 M_\\infty d_{i,k-1} \\ . \n\t\t\t\\end{aligned}\\end{equation*}\\end{small}Therefore\n\t\t\\begin{small}\\begin{equation}\\begin{aligned}\\label{nclz}\n\t\t\t\t\\mathbb{E}[ Z_k ] = &\\sum_{i=1}^n \\mathbb{E} [Z_k^i ] \\le \\rho_k \\sum_{i=1}^n \\mathbb{E} [Z_{k-1}^i + 2 M_\\infty \\gamma_{k-1}G_{k-1}^i + \\gamma_{k-1}^2 M_\\infty d_{i,k-1}] \\\\ \n\t\t\t\t&\\le \\rho_k \\left( \\mathbb{E} [Z_{k-1}] + 2   M_\\infty \\gamma_{k-1}\\mathbb{E}[G_{k-1}] +   \\gamma_{k-1}^2 M_\\infty M_1 \\right) \\\\\n\t\t\t\t&\\le \\rho_k \\left(\\mathbb{E} [Z_{k-1}] + \\frac{24M_\\infty M_1}{(k+1)\\sqrt{k+1}}   +  \\frac{4M_\\infty M_1}{(k+1)^2} \\right) \\le \\rho_k \\left(\\mathbb{E} [Z_{k-1}] + \\frac{28M_\\infty M_1}{(k+1)\\sqrt{k+1}}  \\right)    \\ ,\n\t\t\\end{aligned}\\end{equation} \\end{small}where the second inequality uses $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$, and the third inequality uses \\cref{ineq: convex EG_k final}. Recurrently applying \\cref{nclz} yields:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent Z_k 4}\n\t\t\t\\mathbb{E} [Z_k]  \\le \\rho_k\\bigg( \\mathbb{E} [Z_{k-1} ]+\n\t\t\t\\frac{28M_1M_\\infty}{(k+1)^{3/2}}\\bigg) \\le \\cdots\\le \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}  \\ .\n\t\t\\end{equation}\\end{small}Therefore\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent Z_k 5}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\mathbb{E} [Z_k]  & \\le  \\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor k/2\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}\\\\\n\t\t\t\t& \\le \\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{3\\sqrt{3}}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor k/2\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\ .\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}By \\cref{lm: rho} we have \n\t\t\t\\begin{small}\\begin{equation}\\label{ineq: Z_k recurrent sumpro 1}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{3\\sqrt{3}}  &\\le \\sqrt{\\lfloor k/2\n\t\t\t\t\t\\rfloor + 1} \\exp\\big(2 \\sqrt{ \\lfloor k/2 \\rfloor + 1} - 2\\sqrt{k+1}\\big) \\cdot \\frac{28M_1M_\\infty}{(3\\sqrt{3})} \\\\ \n\t\t\t\t&\\le \\frac{{54M_1M_\\infty}}{k+2} \\ , \n\t\t\\end{aligned}\\end{equation}\\end{small}where the second inequality above uses \\cref{lm: simple ineq}. \n\t\tSimilarly for the second term in the rightmost side of \\cref{ineq: recurrent Z_k 5} we have:\n\t\t\t\\begin{small}\\begin{equation}\n\t\t\t\\begin{aligned}\\label{ineq: Z_k recurrent sumpro 2}\n\t\t\t\t\\sum_{i=\\lfloor k/2 \\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} &\\le   \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\\\ \n\t\t\t\t&\\le \\sqrt{k+1} \\cdot \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\le \\frac{80M_1M_\\infty}{k+2} \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses \\cref{lm: rho}.\n\t\tSubstituting \\cref{ineq: Z_k recurrent sumpro 1} and \\cref{ineq: Z_k recurrent sumpro 2} back into \\cref{ineq: recurrent Z_k 5} yields the second part of \\cref{eq: lm: convex EZ}, namely:\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: convex EZ_k final}\n\t\t\t\t\\mathbb{E} [Z_k] \\le  \\frac{134M_1M_\\infty}{k+2} \\ .\n\t\t\t\\end{equation}\n\t\t\\end{small}\\end{proof}","start":77572,"end":81481}]},{"id":"S046","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"随机陈旧和与平方和","source_units":[{"raw_tex":"\t\t\n\t\tFirst of all, let us define $G_k^i := \\left( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\right)$ and $Z_k^i :=\\left( \\sum_{j = \\tau^k_i}^{k-1}  \\gamma_j d_{ij} \\right)^2$ for $i \\in [n]$, and also $G_k :=\\sum_{i=1}^n G_k^i$ and $Z_k :=\\sum_{i=1}^n Z_k^i$. Notice in $G_k^i$ and $Z_k^i$ that the randomness lies only in $\\tau_i^k$, which in turn depends on the update sets $\\calB_0, \\calB_1,\\dots,\\calB_k$. \n\t\t\n","start":73933,"end":74348}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":["ncl"],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"一阶矩刷新递推","source_units":[{"raw_tex":"\t\tWe first analyze $G_k^i$ and $G_k$.  We note that $G_k^i = 0$ when $i \\in \\calB_k$, which happens with probability $\\beta_k/n$.  Because of the relationship between $\\{\\tau^k_i\\}$ and $\\{\\tau^{k-1}_i\\}$ in \\cref{eq: tauk relationship}, we further note that with probability $1-\\beta_k/n$ that $i \\notin \\calB_k$ in which case it follows that $G_k^i = G_{k-1}^i + \\gamma_{k-1}d_{i,k-1}$.  Defining $\\rho_k := 1 - \\frac{1}{\\sqrt{k}} = 1 - \\beta_k/n $, we have \n\t\t\\begin{small}\\begin{equation}\\label{ncl}\n\t\t\t\\mathbb{E}[ G_k ] = \\sum_{i=1}^n \\mathbb{E} [G_k^i ]= \\rho_k \\sum_{i=1}^n \\mathbb{E} [G_{k-1}^i +\n\t\t\t\\gamma_{k-1} d_{i,k-1}] \\le \\rho_k\\left( \\mathbb{E} [G_{k-1} ]+\n\t\t\t\\gamma_{k-1} M_1 \\right) \\ ,\n\t\t\\end{equation}\\end{small}where the inequality uses $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$. Recurrently applying \\cref{ncl} yields:\n","start":74348,"end":75187}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: recurrent G_k 2","ineq: recurrent G_k 3"],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"一阶矩乘积展开与分段","source_units":[{"raw_tex":"\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent G_k 2}\n\t\t\t \\mathbb{E}[ G_k ] \\le \\rho_k( \\mathbb{E}[ G_{k-1} ]+\n\t\t\t\\gamma_{k-1}  M_1 ) \\le\\cdots\\le \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\gamma_{i-1} M_1 + \\Big(\\prod_{j=1}^{k} \\rho_j\\Big)\\mathbb{E}[ G_0] \\ .\n\t\t\\end{equation}\\end{small}Noting that $G_0 = 0$ (since all Taylor points are set to $x^0$ in \\cref{bettina} of \\cref{alg: TUFW in general finite sum}) and using $\\gamma_{i-1} = \\frac{2}{i+1}$ it follows that:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent G_k 3}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\mathbb{E} [G_k] &\\le \\sum_{i=1}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{i+1}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor {k}/{2}\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{i+1}\\\\\n\t\t\t\t& \\le \\sum_{i=2}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{3}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor {k}/{2}\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor {k}/{2}\\rfloor+2} \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses the fact that $\\rho_1 = 0$ and hence $\\prod_{j=1}^{k}\\rho_j = 0$.\n","start":75187,"end":76313}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: G_k recurrent sumpro 1"],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"一阶矩前半段估计","source_units":[{"raw_tex":"\t\tBy \\cref{lm: rho} we have:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: G_k recurrent sumpro 1}\n\t\t\t\\sum_{i=1}^{\\lfloor {k}/{2}\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)  \\frac{2M_1}{3}\n\t\t\t\\le\n\t\t\t\\sqrt{\\lfloor {k}/{2}\n\t\t\t\t\\rfloor + 1} \\exp\\Big(2 \\sqrt{ \\lfloor {k}/{2} \\rfloor + 1} - 2\\sqrt{k+1}\\Big)  \\frac{2M_1}{3} \\le \\frac{2 M_1}{\\sqrt{k+2}} \\ ,\n\t\t\\end{equation}\\end{small}where the second inequality above uses \\cref{lm: simple ineq}.\n","start":76313,"end":76750}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: G_k recurrent sumpro 2"],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"一阶矩后半段估计","source_units":[{"raw_tex":"\t\tSimilarly for the second term in the rightmost side of \\cref{ineq: recurrent G_k 3} we have:\n\t\t\\begin{small}\\begin{equation}\\label{ineq: G_k recurrent sumpro 2}\n\t\t\t\\sum_{i=\\lfloor k/2 \\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor k/2\\rfloor+2} \\le   \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{2M_1}{\\lfloor k/2\\rfloor+2} \\le \\frac{\\sqrt{k+1} \\cdot  2M_1}{\\lfloor k/2\\rfloor+2} \\le \\frac{4M_1}{\\sqrt{k+2}} \\ , \n\t\t\\end{equation}\\end{small}where the second inequality uses \\cref{lm: rho}.\n","start":76750,"end":77267}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: convex EG_k final"],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"合并一阶矩常数","source_units":[{"raw_tex":"\t\tSubstituting \\cref{ineq: G_k recurrent sumpro 1} and \\cref{ineq: G_k recurrent sumpro 2} back into \\cref{ineq: recurrent G_k 3} yields the first part of \\cref{eq: lm: convex EZ}, namely:\n\t\t\\begin{equation}\\label{ineq: convex EG_k final}\n\t\t\t\\mathbb{E} [G_k] \\le {6 M_1}/{\\sqrt{k+2}} \\ .\n\t\t\\end{equation}\n","start":77267,"end":77572}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":["nclz"],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"平方矩刷新递推","source_units":[{"raw_tex":"\t\tWe employ a similar proof approach to establish the second part of \\cref{eq: lm: convex EZ}. We note that $Z_k^i = 0$ when $i \\in \\calB_k$, which happens with probability $\\beta_k/n$.  Because of the relationship between $\\{\\tau^k_i\\}$ and $\\{\\tau^{k-1}_i\\}$ in \\cref{eq: tauk relationship}, we further note that with probability $1-\\beta_k/n$ that $i \\notin \\calB_k$ in which case we have\n\t\t\\begin{small}\\begin{equation*}\n\t\t\t\\begin{aligned} \n\t\t\t\tZ_k^i &= (G_{k-1}^i + \\gamma_{k-1}d_{i,k-1})^2 = Z_{k-1}^i + 2 G_{k-1}^i\\gamma_{k-1}d_{i,k-1} + (\\gamma_{k-1}d_{i,k-1})^2 \\\\ &\\le Z_{k-1}^i + 2 M_\\infty \\gamma_{k-1}G_{k-1}^i + \\gamma_{k-1}^2 M_\\infty d_{i,k-1} \\ . \n\t\t\t\\end{aligned}\\end{equation*}\\end{small}Therefore\n\t\t\\begin{small}\\begin{equation}\\begin{aligned}\\label{nclz}\n\t\t\t\t\\mathbb{E}[ Z_k ] = &\\sum_{i=1}^n \\mathbb{E} [Z_k^i ] \\le \\rho_k \\sum_{i=1}^n \\mathbb{E} [Z_{k-1}^i + 2 M_\\infty \\gamma_{k-1}G_{k-1}^i + \\gamma_{k-1}^2 M_\\infty d_{i,k-1}] \\\\ \n\t\t\t\t&\\le \\rho_k \\left( \\mathbb{E} [Z_{k-1}] + 2   M_\\infty \\gamma_{k-1}\\mathbb{E}[G_{k-1}] +   \\gamma_{k-1}^2 M_\\infty M_1 \\right) \\\\\n\t\t\t\t&\\le \\rho_k \\left(\\mathbb{E} [Z_{k-1}] + \\frac{24M_\\infty M_1}{(k+1)\\sqrt{k+1}}   +  \\frac{4M_\\infty M_1}{(k+1)^2} \\right) \\le \\rho_k \\left(\\mathbb{E} [Z_{k-1}] + \\frac{28M_\\infty M_1}{(k+1)\\sqrt{k+1}}  \\right)    \\ ,\n\t\t\\end{aligned}\\end{equation} \\end{small}where the second inequality uses $\\sup_{j} \\sum_{i=1}^n d_{ij} \\le M_{1}$, and the third inequality uses \\cref{ineq: convex EG_k final}. Recurrently applying \\cref{nclz} yields:\n","start":77575,"end":79106}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: recurrent Z_k 4","ineq: recurrent Z_k 5"],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"平方矩乘积展开与分段","source_units":[{"raw_tex":"\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent Z_k 4}\n\t\t\t\\mathbb{E} [Z_k]  \\le \\rho_k\\bigg( \\mathbb{E} [Z_{k-1} ]+\n\t\t\t\\frac{28M_1M_\\infty}{(k+1)^{3/2}}\\bigg) \\le \\cdots\\le \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}  \\ .\n\t\t\\end{equation}\\end{small}Therefore\n\t\t\\begin{small}\\begin{equation}\\label{ineq: recurrent Z_k 5}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\mathbb{E} [Z_k]  & \\le  \\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor k/2\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(i+1)^{3/2}}\\\\\n\t\t\t\t& \\le \\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{3\\sqrt{3}}\n\t\t\t\t+\n\t\t\t\t\\sum_{i=\\lfloor k/2\\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\ .\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}By \\cref{lm: rho} we have \n","start":79106,"end":80027}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: Z_k recurrent sumpro 1"],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"平方矩前半段估计","source_units":[{"raw_tex":"\t\t\t\\begin{small}\\begin{equation}\\label{ineq: Z_k recurrent sumpro 1}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\sum_{i=1}^{\\lfloor k/2\\rfloor}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{3\\sqrt{3}}  &\\le \\sqrt{\\lfloor k/2\n\t\t\t\t\t\\rfloor + 1} \\exp\\big(2 \\sqrt{ \\lfloor k/2 \\rfloor + 1} - 2\\sqrt{k+1}\\big) \\cdot \\frac{28M_1M_\\infty}{(3\\sqrt{3})} \\\\ \n\t\t\t\t&\\le \\frac{{54M_1M_\\infty}}{k+2} \\ , \n\t\t\\end{aligned}\\end{equation}\\end{small}where the second inequality above uses \\cref{lm: simple ineq}. \n","start":80027,"end":80507}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: Z_k recurrent sumpro 2"],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"平方矩后半段估计","source_units":[{"raw_tex":"\t\tSimilarly for the second term in the rightmost side of \\cref{ineq: recurrent Z_k 5} we have:\n\t\t\t\\begin{small}\\begin{equation}\n\t\t\t\\begin{aligned}\\label{ineq: Z_k recurrent sumpro 2}\n\t\t\t\t\\sum_{i=\\lfloor k/2 \\rfloor+1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big)\\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} &\\le   \\sum_{i=1}^{k}\\Big(\\prod_{j=i}^{k}\\rho_j\\Big) \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\\\ \n\t\t\t\t&\\le \\sqrt{k+1} \\cdot \\frac{28M_1M_\\infty}{(\\lfloor k/2\\rfloor +2)^{3/2}} \\le \\frac{80M_1M_\\infty}{k+2} \\ ,\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\\end{small}where the second inequality uses \\cref{lm: rho}.\n","start":80507,"end":81125}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: convex EZ_k final"],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"合并平方矩常数","source_units":[{"raw_tex":"\t\tSubstituting \\cref{ineq: Z_k recurrent sumpro 1} and \\cref{ineq: Z_k recurrent sumpro 2} back into \\cref{ineq: recurrent Z_k 5} yields the second part of \\cref{eq: lm: convex EZ}, namely:\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: convex EZ_k final}\n\t\t\t\t\\mathbb{E} [Z_k] \\le  \\frac{134M_1M_\\infty}{k+2} \\ .\n\t\t\t\\end{equation}\n\t\t\\end{small}\\end{proof}","start":81125,"end":81481}]},{"id":"S057","semantic_type":"proof","kind":"proof","labels":["ineq: convex thm 1 general","ineq: convex thm 2 general"],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"主定理完整作者证明","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{thm: general convergence rate convex}]\n\t\tFrom \\cref{lm: general convergence rate convex} it holds that:\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\n\t\t\t\t\\eps_k  \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{2}{k(k+1)}\\sum_{t=1}^{k} t \\left(\n\t\t\t\t\\nabla F(x^{t-1})   - g^{t-1}\n\t\t\t\t\\right)^\\top (s^{t-1} - x^\\star) \\ ,\n\t\t\t\\end{equation}\n\t\t\\end{small}and from \\cref{lm: general 3rd term upper bound} it follows that:\n\t\t\\begin{small}\n\t\t\t\\begin{equation*}\n\t\t\t\t\\eps_k \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{\\hat{L}D^3}{nk(k+1)}\\sum_{t=1}^{k}t\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^{t-1}_i}^{t-2}  \\gamma_j \\Big)^2 \\ .\n\t\t\t\\end{equation*}\n\t\t\\end{small}Taking expectations on both sides then yields\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: convex thm 1 general}\n\t\t\t\t\\mathbb{E} [\\eps_k]  \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{\\hat{L}D^3}{nk(k+1)}\\sum_{t=1}^{k}t~\\mathbb{E}\\Big[\n\t\t\t\t\\sum_{i=1}^{n}\\Big(  \\sum_{j = \\tau^{t-1}_i}^{t-2}  \\gamma_j \\Big)^2\n\t\t\t\t\\Big] \\ .\n\t\t\t\\end{equation}\n\t\t\\end{small}Now define $d_{ij} := 1$ and $M_1 := n$ and $M_\\infty := 1$.  Then since $\\sup_{j}   \\sum_{i=1}^{n} d_{ij} = M_1$  and  $\\sup_{i,j} d_{ij}$ is equal to $M_\\infty$, we can apply \\cref{lm: convex EZ} to \\cref{ineq: convex thm 1 general}  and obtain\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: convex thm 2 general}\n\t\t\t\t\\mathbb{E} [\\eps_k]   \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{\\hat{L}D^3}{nk(k+1)}\\sum_{t=1}^{k}t~\\frac{134 n}{t+1} \\le \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{134 \\hat{L}D^3}{k+1} \\ , \n\t\t\t\\end{equation}\n\t\t\\end{small}which proves \\cref{eq of thm: general convergence rate convex}.\n\t\\end{proof}","start":81564,"end":83163}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"调用凸差距引理","source_units":[{"raw_tex":"\\begin{proof}[Proof of \\cref{thm: general convergence rate convex}]\n\t\tFrom \\cref{lm: general convergence rate convex} it holds that:\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\n\t\t\t\t\\eps_k  \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{2}{k(k+1)}\\sum_{t=1}^{k} t \\left(\n\t\t\t\t\\nabla F(x^{t-1})   - g^{t-1}\n\t\t\t\t\\right)^\\top (s^{t-1} - x^\\star) \\ ,\n\t\t\t\\end{equation}\n","start":81564,"end":81909}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"代入Taylor误差界","source_units":[{"raw_tex":"\t\t\\end{small}and from \\cref{lm: general 3rd term upper bound} it follows that:\n\t\t\\begin{small}\n\t\t\t\\begin{equation*}\n\t\t\t\t\\eps_k \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{\\hat{L}D^3}{nk(k+1)}\\sum_{t=1}^{k}t\\sum_{i=1}^{n}\\Big( \\sum_{j = \\tau^{t-1}_i}^{t-2}  \\gamma_j \\Big)^2 \\ .\n\t\t\t\\end{equation*}\n","start":81909,"end":82199}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: convex thm 1 general"],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"对陈旧误差取期望","source_units":[{"raw_tex":"\t\t\\end{small}Taking expectations on both sides then yields\n\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: convex thm 1 general}\n\t\t\t\t\\mathbb{E} [\\eps_k]  \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{\\hat{L}D^3}{nk(k+1)}\\sum_{t=1}^{k}t~\\mathbb{E}\\Big[\n\t\t\t\t\\sum_{i=1}^{n}\\Big(  \\sum_{j = \\tau^{t-1}_i}^{t-2}  \\gamma_j \\Big)^2\n\t\t\t\t\\Big] \\ .\n\t\t\t\\end{equation}\n","start":82199,"end":82547}]},{"id":"S061","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"主证明中的常数数组","source_units":[{"raw_tex":"\t\t\\end{small}Now define $d_{ij} := 1$ and $M_1 := n$ and $M_\\infty := 1$.  Then since $\\sup_{j}   \\sum_{i=1}^{n} d_{ij} = M_1$  and  $\\sup_{i,j} d_{ij}$ is equal to $M_\\infty$, we can apply \\cref{lm: convex EZ} to \\cref{ineq: convex thm 1 general}  and obtain\n","start":82547,"end":82807}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":["ineq: convex thm 2 general"],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"代入平方矩并完成求和","source_units":[{"raw_tex":"\t\t\\begin{small}\n\t\t\t\\begin{equation}\\label{ineq: convex thm 2 general}\n\t\t\t\t\\mathbb{E} [\\eps_k]   \\le  \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{\\hat{L}D^3}{nk(k+1)}\\sum_{t=1}^{k}t~\\frac{134 n}{t+1} \\le \\frac{2L D^2}{k+1}\n\t\t\t\t+  \\frac{134 \\hat{L}D^3}{k+1} \\ , \n\t\t\t\\end{equation}\n\t\t\\end{small}which proves \\cref{eq of thm: general convergence rate convex}.\n\t\\end{proof}","start":82807,"end":83163}]},{"id":"S063","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S063"],"declared_external":true,"declaration_text":["原始外部前提清单：First-order gradient inequality for differentiable convex functions；作者在第833行明确说明使用。"],"source_only":false,"title":"凸函数一阶梯度不等式","source_units":[]},{"id":"S064","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S064"],"declared_external":true,"declaration_text":["原始外部前提清单：dual-norm Hölder and triangle inequalities。"],"source_only":false,"title":"对偶范数与三角不等式","source_units":[]},{"id":"S065","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S065"],"declared_external":true,"declaration_text":["原始外部前提清单：Elementary expectation identities；条件历史的使用由标注者显式化。"],"source_only":false,"title":"有限期望与条件期望规则","source_units":[]},{"id":"S066","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S066"],"declared_external":true,"declaration_text":["原始外部前提清单：telescoping sums, logarithmic inequalities, and calculus-based integral comparisons；作者还在第895及899行明确使用凹性与极值事实。"],"source_only":false,"title":"求和与初等微积分工具","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S064":"S064","S065":"S065","S066":"S066"}}}}
{"record_id":"2207.06452v1","paper":{"title":"Stochastic trust-region algorithm in random subspaces with convergence and expected complexity analyses","authors":["Dzahini, Kwassi Joseph","Wild, Stefan M."],"arxiv_id":"2207.06452","version":"v1","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/2207.06452v1","pdf_url":"https://arxiv.org/pdf/2207.06452v1","doi":"10.1137/22M1524072"},"context":{"raw_tex":"This paper focuses on unconstrained stochastic DFO, wherein the objective function values are accessible only through a blackbox oracle corrupted by stochastic noise. We consider the problem \n\\begin{equation}\\label{probl1}\n\t\\underset{\\x\\in\\rn}{\\min} f(\\x), \\qquad\\mbox{with}\\quad f(\\x)=\\Esp_{\\ubar{\\theta}}\\left[f_{\\ubar{\\theta}}(\\x)\\right],\n\\end{equation}\n%\\quad\\mbox{with}\\quad f(\\x)=\\Esp_{\\ubar{\\theta}}\\left[f_{\\ubar{\\theta}}(\\x)\\right]\nwhere the values of the continuously differentiable function $f:\\rn\\to\\R$ are available only via $f_{\\ubar{\\theta}}$, a stochastically noisy version of $f$, and where $\\ubar{\\theta}$ is a random variable whose distribution is possibly unknown. \n\n\nUnlike the stochastic trust-region framework STORM~\\cite{blanchet2016convergence,chen2018stochastic}, which builds full space random models $\\ubar{m}_k(\\x)$, $\\x\\in\\rn$, to approximate $f(\\x)$, STARS operates as follows. On iteration $k$, given a current iterate $\\xk$, let $\\mathcal{Y}_k\\subseteq \\rn$ be the affine space randomly chosen through the range of a matrix $\\Qk\\in\\rnp$ \n%satisfying $\\norme{\\Qk}\\leq\\Qmax$ for some $\\Qmax>0$, \nwhose entries are randomly selected; that is, \n\\[\\mathcal{Y}_k = \\accolade{\\xk+\\Qk\\bm{{\\s}}:\\bm{{\\s}}\\in\\R^p}.\\]\nThen, given a trust-region radius $\\dk>0$, a subspace model $\\mhatk(\\s), \\s\\in\\rp$ is built only on $\\mathcal{Y}_k$ using realizations of the stochastically noisy function $f_{\\ubar{\\theta}}$; the model  serves as an approximation of $f(\\x)$ in $\\mathcal{B}(\\xk,\\dk;\\Qk):=\\accolade{\\xk+\\Qk\\s \\in \\mathcal{Y}_k:\\normii{\\s}\\leq\\dk}$. For concreteness, here we employ a quadratic subspace model given by \n\\begin{equation*}\nf(\\xk+\\Qk\\s)\\approx\\mhatk(\\bm{\\s}):= f_k+\\gkhat^\\top\\s+\\s^\\top\\Hkhat\\s,\n\\end{equation*}\nwhere $\\gkhat\\in \\R^p$ and $\\Hkhat\\in \\Sym^{p}$ are the low-dimensional model gradient and Hessian, respectively. A {\\it tentative step} $\\Qk\\sk\\in\\rn$ is produced by using the solution $\\sk\\in\\rp$ obtained by approximately minimizing $\\mhatk$ inside the trust  region; that  is, $\\sk\\approx{\\argmin}\\left\\{ \\mhatk(\\s): \\s\\in\\rp, \\, \\normii{\\bm{{\\s}}}\\leq \\dk\\right\\}$. Inspired by~\\cite{CRsubspace2021,chen2018stochastic}, the {\\it trial step} $\\sk$ has to provide a sufficient decrease in $\\mhatk$ by satisfying the following standard fraction of the Cauchy decrease condition.\n\n\\begin{assumption}\nFor every iteration $k$, a trial step $\\sk\\in\\rp$ is computed so that\\footnote{Throughout the manuscript, the matrix norm $\\norme{\\cdot}$ is supposed to be {\\it consistent} with the Euclidean norm; that is, $\\normii{\\Ab\\xb}\\leq \\norme{\\Ab}\\normii{\\xb}$, which holds for both Frobenius and spectral matrix norms.}\n\\begin{equation}\\label{CauchyDecrease}\n\\mhatk(\\bm{0})-\\mhatk(\\bm{\\sk})\\geq \\frac{\\kappa_{fcd}}{2}\\normii{\\gkhat}\\min\\accolade{\\dk,\\frac{\\normii{\\gkhat}}{\\max\\left(\\norme{\\Hkhat},1\\right)}},\n\\end{equation}\nfor some constant $\\kappa_{fcd}\\in (0,1]$.\n\\end{assumption}\n\n\nThe random variables considered here are all defined on the same probability space $(\\Omega,\\F, \\pr)$, where $\\Omega$, referred to as the {\\it sample space}, is a nonempty set whose elements $\\omega$ and subsets are called {\\it sample points} and {\\it events}, respectively. $\\F$ is a collection of such events, which is called a {\\it $\\sigma$-algebra}. $\\pr$ is a finite measure defined on the measurable space $(\\Omega,\\F)$, satisfying $\\prob{\\Omega}=1$ and referred to as the {\\it probability measure}. An increasing subsequence $\\accolade{\\mathcal{S}_k}_k$ of $\\sigma$-algebras of $\\F$ will be called a {\\it filtration}. Let $\\mathscr{B}(\\rn)$ be the $\\sigma$-algebra generated by the open sets of $\\rn$, also known as the Borel $\\sigma$-algebra of $\\rn$. A random variable $\\ubar{z}$ is a measurable map defined on $(\\Omega,\\F, \\pr)$ into the measurable space $(\\rn, \\mathscr{B}(\\rn))$, where measurability means that $\\accolade{\\ubar{z}\\in A}:=\\accolade{\\omega\\in\\Omega:\\ubar{z}(\\omega)\\in A} =:{\\ubar{z}}^{-1}(A)\\in\\F\\ \\forall A\\in \\mathscr{B}(\\rn)$~\\cite{bhattacharya2007basic}. For the remainder of the manuscript, vectors will be written in lowercase boldface (e.g., $\\x\\in\\rn, n\\geq 2$) while matrices will be written in uppercase boldface (e.g., $\\Qk\\in\\rnp$), and underlined letters (e.g., $\\ubar{z}, \\ubar{\\x}, \\QkRandom$) will be used to denote random quantities; $z=\\ubar{z}(\\omega)$ will denote a realization of $\\ubar{z}$.\n\n\n\\begin{algorithm}[htb]\n\\caption{STARS}\n\\label{algoStoScalTR} \n\\textbf{[0] Initialization}\\\\\n\\hspace*{10mm}Choose constants $\\gamma>1$, $\\eta_1\\in(0,1)$, $\\eta_2>0$, $j_{\\max}\\in\\N$, $\\varepsilon\\in(0,1)$, $\\beta\\in(0,1)$, \\\\ \n\\hspace*{10mm}$c_1\\geq 1$, initial trust-region radius $\\delta_0>0$, and maximum trust-region radius\\\\\n\\hspace*{10mm}$\\dmaxx=\\gamma^{j_{\\max}}\\delta_0$, starting point $\\bm{{\\x}_0}\\in\\rn$, and dimension $p\\in\\intbracket{1,n}$.\\\\ \n\\hspace*{10mm}Set the iteration counter $k \\gets 0$.\\\\\n\\textbf{[1] Construction of subspace model}\\\\\n\\hspace*{10mm}Generate $\\Qk$: a realization of a\\footnotemark {\\em WAM}($1-\\varepsilon,\\beta$), using a distribution $\\mathbb{Q}_{k}$.\\\\\n\\hspace*{10mm}Build model $\\mhatk:\\rp\\to\\R$ that is $(\\kef,\\keg;\\Qk)$-fully linear \n%$\\mhatk(\\s)={\\f}_k+\\gkhat^{\\top}\\s+$\\\\\n%\\hspace*{10mm}$\\frac{1}{2}\\s^{\\top}\\Hkhat\\s\\approx f(\\xk+\\Qk \\s)$, that locally approximates $f$ \nin  $\\mathcal{B}(\\xk,c_1\\dk;\\Qk)$.\\\\ \n%\\hspace*{10mm}with the affine space defined by the columns of $\\Qk$.\\\\\n\\textbf{[2] Step calculation}\\\\\n\\hspace*{10mm}Compute $\\sk\\approx {\\argmin}\\left\\{ \\mhatk(\\s) : \\; \\s\\in\\rp:\\normii{\\s}\\leq \\dk\\right \\}$ satisfying~\\eqref{CauchyDecrease}.\\\\\n\\textbf{[3] Estimate computation}\\\\\n\\hspace*{10mm}Obtain estimates $\\fok\\approx f(\\xk)$ and $\\fsk\\approx f(\\xk+\\Qk\\sk)$ satisfying~\\eqref{accEstimates}.\\\\\n\\textbf{[4] Updates}\\\\\n\\hspace*{10mm}Compute $\\rho_k=\\frac{\\fok-\\fsk}{\\mhatk(\\bm{0})-\\mhatk(\\sk)}.$\\\\\n\\hspace*{10mm}If $\\rho_k\\geq \\eta_1$ and $\\normii{\\gkhat}\\geq \\eta_2\\dk$ (\\textbf{success}): \\\\\n\\hspace*{16mm}set $\\bm{\\xkun}=\\xk+\\Qk\\sk$ and $\\dkun=\\min\\accolade{\\gamma\\dk,\\dmaxx}$. \\\\\n\\hspace*{10mm}Otherwise (\\textbf{failure}): set $\\bm{\\xkun}=\\xk$ and $\\dkun=\\gamma^{-1}\\dk$.\\\\\n\\hspace*{10mm}Update the iteration counter $k \\gets k+1$, and go to~\\textbf{[1]}.\\\\\n\\end{algorithm}\n%\\caption[]{\\small{STARS algorithm}}\n%\\end{figure*}\n\\footnotetext{See Definition~\\ref{wellAlignedDefinition} for the meaning of {\\em WAM}($1-\\varepsilon,\\beta$).}\n\n\nThe deterministic models $\\mhatk$ and the function estimates $\\fok$ and $\\fsk$ are constructed by using evaluations of the noisy function $f_{\\ubar{\\theta}}$ and the randomly selected entries of the deterministic matrix~$\\Qk=\\QkRandom(\\omega)$.  Hence, $\\mhatk$, $\\fok$, and $\\fsk$ can be considered as realizations of random models and estimates $\\Mhatk$, $\\Fok$, and $\\Fsk$, respectively. Each iteration of Algorithm~\\ref{algoStoScalTR} is therefore influenced by the behavior of these random quantities; and consequently, the algorithm results in a stochastic process. The present work shows that under certain conditions on the sequences~$\\accoladekinN{\\QkRandom}$, $\\accoladekinN{\\Fok,\\Fsk}$, and $\\accoladekinN{\\Mhatk}$, the resulting stochastic process has desirable convergence properties, conditioned on the past, where {\\it past} means the {\\it past history} of the algorithm (see~Remark~\\ref{whyTheseSigmaAlgebras}). In particular, for the needs of convergence and expected complexity analyses presented  in Sections~\\ref{Section4} and~\\ref{Section5}, and as will be seen  in Section~\\ref{Section3}, the random models~$\\Mhatk$ and estimates~$\\Fok,\\Fsk$ are required to be probabilistically sufficiently accurate conditioned on the past, as was the case in~\\cite{chen2018stochastic}. Moreover, $\\QkRandom$ must guarantee a high quality of the selected subspace probabilistically and conditioned on the past, as will be detailed in Section~\\ref{Section31}. \n\nFor all $k\\in\\N$, denote by $(\\QkRandom)_{ij},  (i,j)\\in \\intbracket{1,n}\\times\\intbracket{1,p}=:\\mathbb{I}_{n,p}$ the entries of the random matrix $\\QkRandom$, where for $a\\leq b$, $\\intbracket{a,b}:=[a,b]\\cap\\mathbb{Z}$  throughout the manuscript. \nTo formalize conditioning on the past, we consider the three filtrations $\\accolade{\\falgebra}_{k\\in\\N}$, $\\accolade{\\fqalgebra}_{k\\in\\N}$, and $\\accolade{\\fmqalgebra}_{k\\in\\N}$ defined respectively by \n\\begin{equation*}\n\\begin{array}{rcl}\n\\falgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell,  (\\QellRandom)_{ij} \\mbox{ with}\\  \\ell \\in \\intbracket{0,k-1} \\mbox{ \\&}\\ (i,j)\\in \\mathbb{I}_{n,p} \\right)\\\\\n\\fqalgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell, (\\QellRandom)_{ij}, (\\QkRandom)_{ij} \\mbox{ with}\\  \\ell \\in \\intbracket{0,k-1} \\mbox{ \\&}\\ (i,j)\\in \\mathbb{I}_{n,p} \\right)\\\\\n\\fmqalgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell, (\\QellRandom)_{ij}, (\\QkRandom)_{ij}, \\ \\Mhatk \\mbox{ with}\\  \\ell \\in \\intbracket{0,k-1} \\mbox{ \\&}\\ (i,j)\\in \\mathbb{I}_{n,p}  \\right).\n\\end{array}\n\\end{equation*}\n% \\begin{eqnarray*}\n%  \\falgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell\\ \\mbox{and}\\ (\\QellRandom)_{ij}\\ \\mbox{with}\\  \\ell \\in \\intbracket{0,k-1}\\ \\mbox{and}\\ (i,j)\\in \\mathbb{I}_{n,p}\\right)\\\\\n%  \\fqalgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell,\\ (\\QellRandom)_{ij}\\ \\mbox{and}\\ (\\QkRandom)_{ij}\\ \\mbox{with}\\  \\ell \\in \\intbracket{0,k-1}\\ \\mbox{and}\\ (i,j)\\in \\mathbb{I}_{n,p}\\right)\\\\\n%  \\fmqalgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell,\\ (\\QellRandom)_{ij},\\ \\Mhatk\\ \\mbox{and}\\ (\\QkRandom)_{ij}\\ \\mbox{with}\\  \\ell \\in \\intbracket{0,k-1}\\ \\mbox{and}\\ (i,j)\\in \\mathbb{I}_{n,p}\\right).\n% \\end{eqnarray*}\nHere, for {\\it completeness}~\\cite{billingsley1995ThirdEdition}, $\\mathcal{F}_{-1}=\\sigma(\\bm{\\x_0})$.\n%One can easily see that . \nBy construction, $\\falgebra\\subset\\fqalgebra\\subset\\fmqalgebra$, $\\QkRandom$ is both~$\\fqalgebra$ and~$\\fmqalgebra$-measurable, and $\\Mhatk$ is~$\\fmqalgebra$-measurable, which imply that $\\E{\\QkRandom|\\fqalgebra}=\\E{\\QkRandom|\\fmqalgebra}=\\QkRandom$ and $\\E{\\Mhatk|\\fmqalgebra}=\\Mhatk$. \n%The  motivation for  introducing these $\\sigma$-algebras in the formalization of conditioning on the past is explained next.\n\n\\begin{remark}\\label{whyTheseSigmaAlgebras}\nAt a given iteration~$k$, when generating~$\\QkRandom$, because no model or estimates were generated since the start of the iteration, it obviously follows from the construction of Algorithm~\\ref{algoStoScalTR} that its past history can be formalized by~$\\falgebra$. On the other hand, when constructing~$\\Mhatk$, the past of the algorithm that includes~$\\QkRandom$, given that the latter is already generated since the beginning of iteration~$k$, can therefore be formalized by~$\\fqalgebra$. A similar observation  explains why~$\\fmqalgebra$ can formalize the past history of Algorithm~\\ref{algoStoScalTR} before the construction of~$\\Fok$ and~$\\Fsk$. These observations will also play an important role in the formalization of sufficient accuracy of the random models and estimates in Definitions~\\ref{defProbQkfully} and~\\ref{defProbAccEstim} and the formalization of high quality of the random subspaces in Assumption~\\ref{wellAlignedAssumption}.  \n\\end{remark}\n\n\nRecalling Remark~\\ref{whyTheseSigmaAlgebras} and inspired by~\\cite{CRsubspace2021}, we propose a probabilistic quantification of the quality of the subspace selection as follows.\n\n\\begin{definition}\\label{wellAlignedDefinition}\nFor fixed $\\alpha_Q,\\beta_Q\\in (0,1)$, a sequence $\\accoladekinN{\\QkRandom}$ of random matrices \n%determining the selection of subspaces \nis $(1-\\beta_Q)$-probabilistically $\\alpha_{Q}$-well aligned if, for any $\\F_{k-1}$-measurable random vector $\\viRandom$ with realizations $\\vi\\in \\rn$, the events \n$\\mathcal{A}_k:= \\accolade{\\normii{\\QkRandomTranspose\\viRandom}\\geq \\alpha_{{Q}}\\normii{\\viRandom}}$ satisfy \n\\begin{equation}\\label{Akineq}\n\\prob{\\mathcal{A}_k|\\F_{k-1}}=\\E{\\iak|\\F_{k-1}}\\geq 1-\\beta_Q,\n\\end{equation}\nwhere $\\iak$ denotes the indicator function of the event $\\mathcal{A}_k$; that  is, $\\iak(\\omega)=1$ if $\\omega\\in \\mathcal{A}_k$ and $\\iak(\\omega)=0$ otherwise. Random matrices satisfying~\\eqref{Akineq} will be referred to as {WAM}($\\alpha_Q,\\beta_Q$).\n%, in other words, well-aligned matrices with parameters $\\alpha_Q$ and $\\beta_Q$.\n\\end{definition}\n%\\begin{definition}\n%Random matrices satisfying the conditions of Lemma... with parameters $\\varepsilon, \\beta$ will be referred to as a {\\em JLM}($\\varepsilon,\\beta$), i.e., Johnson-Lindenstrauss matrices with parameters~$\\varepsilon$ and~$\\beta$.\n%\\end{definition}\nFor convergence purposes  in Sections~\\ref{Section4} and~\\ref{Section5}, the following will be assumed throughout the manuscript.\n\\begin{assumption}\\label{wellAlignedAssumption}\nThe sequence $\\accoladekinN{\\QkRandom}$ of matrices used by Algorithm~\\ref{algoStoScalTR} is $(1-\\beta_Q)$-probabilistically $\\alpha_{Q}$-well aligned for some fixed $\\alpha_Q\\in (0,1)$ and $\\beta_Q\\in (0,1/2)$.\n\\end{assumption}\n\n\nInspired by~\\cite{CRsubspace2021}, a technical assumption needed for convergence and expected complexity analyses later in Sections~\\ref{Section4} and~\\ref{Section5} is the following. \n\\begin{assumption}\\label{QmaxAssumption}\nAny realization $\\Qk$ of the random matrix used by Algorithm~\\ref{algoStoScalTR} satisfies $\\norme{\\Qk}\\leq \\Qmax$ for all $k\\in\\N$, for some constant $\\Qmax >0$ independent of~$k$.\n\\end{assumption}\n\n\n\\subsection{Probabilistic models and estimates in random subspaces.}\nAs mentioned at the beginning of Section~\\ref{Section3}, the models used by Algorithm~\\ref{algoStoScalTR} need to be sufficiently accurate. Motivated by~\\cite{chen2018stochastic}, this sufficient accuracy is formalized for deterministic subspace models by the following measure of accuracy inspired by~\\cite{CRsubspace2021}.\n\\begin{definition}\\label{QkFullyLinearModels}\nAssume $\\nabla f$ is  Lipschitz continuous. Given $\\Qk\\in\\rnp$ and $\\xk\\in\\rn$, a function $\\mhatk:\\rp\\to\\R$ is a $(\\kef,\\keg; \\Qk)$-fully linear model of $f$ in $\\mathcal{B}(\\xk,\\dk;\\Qk)$ for some $\\dk>0$ if there exist constants $\\kappa_{ef}, \\kappa_{eg} >0$ independent of $k$ such that for all $\\s\\in\\rp$ with $\\normii{\\s}\\leq \\dk$,\n\\begin{equation}\\label{Eq36}\n%\\begin{split}\n\\abs{f(\\xk +\\Qk\\s)-\\mhatk(\\s)}\\leq \\kappa_{ef}\\dk^2\\quad \\mbox{and}\\quad\n\\normii{\\Qk^{\\top}\\nabla f(\\xk +\\Qk\\s)-\\nabla\\mhatk(\\s)}\\leq \\kappa_{eg}\\dk.\n%\\end{split}\n\\end{equation}\n\\end{definition}\nAs pointed out in~\\cite{CRsubspace2021}, the gradient condition in~\\eqref{Eq36} is due to the fact that given any fixed~$\\x$ and~$\\Q$, if $\\hat{f}(\\s):=f(\\x+\\Q\\s)$, then $\\nabla_{\\s} \\hat{f}(\\s)=\\Q^{\\top}\\nabla_{\\x} f(\\x+\\Q\\s)$. Moreover, denoting by $\\bm{I}$ the identity matrix in full-dimensional subspaces where $p=n$, then $(\\kef,\\keg; \\bm{I})$-fully linear models correspond to standard fully linear models~\\cite[Definition~6.1]{CoScVibook}.\n\n%Inspired by~\\cite{chen2018stochastic}, and \nRecalling Remark~\\ref{whyTheseSigmaAlgebras}, the next definition formalizing sufficient accuracy of probabilistic subspace models $\\Mhatk$ is a stochastic variant of Definition~\\ref{QkFullyLinearModels}, which generalizes~\\cite[Definition~3.4]{chen2018stochastic} to low-dimensional subspaces. The existence of such models will be rigorously demonstrated  in Theorem~\\ref{subspaceModel1} and Corollary~\\ref{CorQkFully}.\n\\begin{definition}\\label{defProbQkfully}\nA sequence $\\accolade{\\Mhatk}$ of random models is $\\beta_m$-probabilistically \\\\ $(\\kef,\\keg; \\Qk)$-fully linear with respect to the random sequence $\\accolade{\\mathcal{B}(\\Xk,\\Dk;\\QkRandom)}$ if the events\n\\[I_k^Q=\\accolade{\\mbox{Given}\\ \\Qk\\ \\mbox{and}\\ \\xk, \\Mhatk\\ \\mbox{is}\\ (\\kef,\\keg;\\Qk)\\mbox{-fully linear for}\\ f\\ \\mbox{in}\\ \\mathcal{B}(\\Xk,\\Dk;\\QkRandom)}\\]\nsatisfy the supermartingale-like condition\n\\begin{equation}\\label{ikQProb}\n\\prob{I^Q_k|\\fqalgebra}=\\E{\\iik|\\fqalgebra}\\geq \\beta_m.\n\\end{equation}\n\\end{definition}\nRecalling that estimates of unknown function values also need to be sufficiently accurate, we introduce a formalization of sufficient accuracy inspired by~\\cite{audet2019stomads,chen2018stochastic,dzahini2020expected,dzahini2020constrained,paquette2018stochastic} and motivated by~\\cite{CRsubspace2021} as follows.\n\\begin{definition}\nGiven $\\ef>0$ and $\\Qk\\sk\\in\\rn, \\, \\xk\\in\\rn$, $\\fok$ and $\\fsk$ are called $\\ef$-accurate estimates of $f(\\xk)$ and $f(\\xk+\\Qk\\sk)$, respectively, for a given $\\dk$ if\n\\begin{equation}\\label{accEstimates}\n\\abs{\\fok-f(\\xk)}\\leq\\ef\\dk^2\\quad \\mbox{ and }\\quad  \\abs{\\fsk-f(\\xk+\\Qk\\sk)}\\leq\\ef\\dk^2.\n\\end{equation}\n\\end{definition}\n\n\nRecalling Remark~\\ref{whyTheseSigmaAlgebras},\n%and following~\\cite{audet2019stomads,chen2018stochastic,dzahini2020expected,dzahini2020constrained,paquette2018stochastic}, \nwe  extend  this definition to the following stochastic variant.\n\\begin{definition}\\label{defProbAccEstim}\nA sequence of random estimates $\\accolade{\\Fok,\\Fsk}$ is said to be $\\beta_f$-probabilistically $\\ef$-accurate with respect to the corresponding sequence $\\accolade{\\Xk,\\Dk,\\QkRandom\\Sk}$ if the events \n\\begin{eqnarray*}\nJ_k^Q &=&\\left\\lbrace\\mbox{Given}\\ \\Qk\\ \\mbox{and}\\ \\xk, \\Fok \\mbox{ and } \\Fsk \\mbox{ are }\\ef\\mbox{-accurate estimates of } f(\\xk)  \\mbox{ and}\\right.\\\\\n& &\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad \\left.  f(\\xk+\\Qk\\sk), \\mbox{ respectively, for } \\Dk \\right\\rbrace\n\\end{eqnarray*}\n% \nsatisfy the supermartingale-like condition\n$\\prob{J^Q_k|\\fmqalgebra}\\geq \\beta_f.$\n\\end{definition}\n\n\nBased on Definitions~\\ref{wellAlignedDefinition}, \\ref{defProbQkfully}, and~\\ref{defProbAccEstim}, {\\it true} iterations, in other words, those for which $\\iak\\iik\\ijk=1$, occur with a total probability of at least $\\tilde{\\beta}:=\\beta_f\\beta_m(1-\\beta_Q)$ conditioned on $\\falgebra$. Noting that by construction $\\iak$ is both $\\fqalgebra$-measurable and $\\fmqalgebra$-measurable since $\\fqalgebra\\subset\\fmqalgebra$, and that $\\iik$ is $\\fmqalgebra$-measurable, then for $F:= \\accolade{\\mathcal{A}_k\\cap I^Q_k\\cap J_k^Q}$, we have\n\\begin{equation}\\label{prodBetaIneq}\n\\begin{split}\n\\pr(F|\\falgebra)&=\\E{\\iak\\iik\\ijk|\\falgebra}\n=\\E{\\iak\\iik\\E{\\ijk|\\fmqalgebra} |\\falgebra}\\\\\n&\\geq \\beta_f\\E{\\iak\\E{\\iik|\\fqalgebra} |\\falgebra}\\geq \\beta_f\\beta_m\\E{\\iak|\\falgebra}\\geq \\tilde{\\beta}.\n%&\\geq \\beta_f\\beta_m(1-\\beta_Q),\n\\end{split}\n\\end{equation}\nThe second equality and the first two inequalities used the  {\\it William's Tower Property} for conditional expectations~\\cite[Theorem~34.4.]{billingsley1995ThirdEdition}, namely, the fact that if ${\\ubar{z}}$ is integrable and the $\\sigma$-algebras $\\mathcal{G}_1$ and $\\mathcal{G}_2$ satisfy $\\mathcal{G}_1\\subset\\mathcal{G}_2$, then $\\E{{\\ubar{z}}|\\mathcal{G}_1}=\\E{\\E{{\\ubar{z}}|\\mathcal{G}_2}|\\mathcal{G}_1}$. Moreover, for a $\\sigma$-algebra $\\mathcal{G}$,  $\\E{{\\ubar{r}}{\\ubar{u}}|\\mathcal{G}}={\\ubar{r}}\\E{{\\ubar{u}}|\\mathcal{G}}$ if~${\\ubar{r}}$ is $\\mathcal{G}$-measurable and~${\\ubar{r}}$ and~${\\ubar{r}}{\\ubar{u}}$ are integrable~\\cite[Theorem~34.3.]{billingsley1995ThirdEdition}.\n\nNote that even though the present algorithmic framework does not distinguish true iterations from {\\it false} ones, it holds that $\\prob{\\underset{k\\to\\infty}{\\limsup}\\ \\ubar{w}_k=\\infty}=1$, where\n\\begin{equation}\\label{randomWalk}\n\\ubar{w}_k:=\\sum_{i=0}^k\\left(2\\mathds{1}_{\\mathcal{A}_i}\\mathds{1}_{I^Q_i}\\mathds{1}_{J^Q_i}-1\\right),\n\\end{equation}\nas will be seen later in the proof of Theorem~\\ref{liminfTheorem} provided $\\tilde{\\beta}>1/2$. This shows that true iterations occur sufficiently often for the convergence of Algorithm~\\ref{algoStoScalTR} to hold (see, e.g.,~\\cite[Theorem~3.6]{dzahini2020constrained} for a similar result).\n\n\n\\begin{remark}%\\label{remarkProdBetafBetam}\nCalculations similar to those in~\\eqref{prodBetaIneq} easily yield \n\\begin{eqnarray}\\label{prodBetafBetam}\n\\prob{I^Q_k\\cap J_k^Q|\\fqalgebra}\\geq \\beta_f\\beta_m, & & \\prob{J_k^Q|\\fqalgebra}\\geq \\beta_f,\\\\\n%\\end{equation}\n%\\begin{equation}\n\\label{otherProbIneqs}\n\\prob{I^Q_k|\\falgebra}\\geq\\beta_m & \\qquad \\mbox{ and } \\qquad & \\prob{J^Q_k|\\falgebra}\\geq\\beta_f,\n\\end{eqnarray}\nwhich will be useful in the proofs of Theorems~\\ref{zerothOrderScal} and~\\ref{liminfTheorem}, respectively. \n%and as was considered in~\\cite[Theorems~4.11]{chen2018stochastic} in similar situations.\n\\end{remark}\n\n\nIn the remainder of the manuscript, the following inspired by~\\cite[Assumptions~4.1 and~4.3]{chen2018stochastic}, respectively, will be assumed.\n\\begin{assumption}\\label{fGradfAssumption}\nFor given $\\bm{\\x_0}$, $\\dmaxx$, and $\\bm{Q_0}$, let $\\mathscr{L}(\\bm{\\x_0};\\bm{Q_0})\\subset\\rn$ be the set containing all iterates of Algorithm~\\ref{algoStoScalTR}. Define the region $\\mathscr{L}^{\\star}(\\bm{\\x_0};\\bm{Q_0})$ considered by the algorithm realizations as $\\mathscr{L}^{\\star}(\\bm{\\x_0};\\bm{Q_0})=\\ds{\\underset{\\accolade{k\\, \\geq\\, 0,\\, \\xk\\in\\mathscr{L}(\\bm{\\x_0};\\bm{Q_0})}}{\\bigcup}}\\mathcal{B}(\\xk,\\dmaxx;\\Qk)$. Then $f(\\x)\\geq f_{\\min}$ for all $\\x\\in\\rn$ and for some constant $f_{\\min}>-\\infty$. Moreover, $f$ and its gradient are Lipschitz continuous on $\\mathscr{L}^{\\star}(\\bm{\\x_0};\\bm{Q_0})$.\n\\end{assumption}\n\n\\begin{assumption}\\label{boundedHessian}\nThere exists some $\\kappa_h\\geq 1$ such that for all $k\\geq 0$, the Hessian $\\Hkhat$ of all realizations of $\\Mhatk$ satisfies $\\norme{\\Hkhat}\\leq \\kappa_h.$\n\\end{assumption}\n\n\n\\begin{lemma}\\label{LemmaC}\nLet Assumption~\\ref{boundedHessian} hold. Assume that $\\mhatk$ is $(\\kef,\\keg;\\Qk)$-fully linear and the estimates $\\accolade{\\fok,\\fsk}$ are $\\ef$-accurate with $\\ef\\leq\\kappa_{ef}$. Then the $k$th iteration is successful if \n\\begin{equation}\\label{deltaLemmaC}\n\\dk\\leq\\min\\accolade{\\frac{1}{\\kappa_h}, \\frac{1}{\\eta_2}, \\frac{\\kappa_{fcd}(1-\\eta_1)}{8\\kappa_{ef}}}\\normii{\\gkhat}.\n\\end{equation}\n\\end{lemma}\n\n\\begin{lemma}\\label{LemmaD}\nSuppose that Assumption~\\ref{boundedHessian} holds and the estimates $\\accolade{\\fok,\\fsk}$ are $\\ef$-accurate with $\\ef<\\frac{1}{4}\\eta_1\\eta_2\\kappa_{fcd}\\min\\accolade{\\frac{\\eta_2}{\\kappa_h}, 1}$. If the $k$th iteration is successful, then the improvement in $f$ is such that\n\\begin{equation}\\label{LemmaDimprovement}\nf(\\xkun)-f(\\xk)\\leq -C_2\\dk^2,\n\\end{equation}\nwhere $C_2=\\frac{1}{2}\\eta_1\\eta_2\\kappa_{fcd}\\min\\accolade{\\frac{\\eta_2}{\\kappa_h}, 1} -2\\ef>0$. \n\\end{lemma}\n\n\\subsection{Zeroth-order convergence result.}\\label{Section4p2}\nTo prove in Theorem~\\ref{zerothOrderScal} that the sequence of random trust-region radii converges to zero almost surely, we assume the following.\n\\begin{assumption}\\label{assumptionsOnEstimatesAndModels} \nThe sequences of estimates $\\accolade{\\Fok, \\Fsk}$ and models $\\accolade{\\Mhatk}$ generated by Algorithm~\\ref{algoStoScalTR} are, respectively, $\\beta_f$-probabilistically $\\ef$-accurate for \n$\\ef<\\min\\accolade{\\kappa_{ef}, \\frac{1}{4}\\eta_1\\eta_2\\kappa_{fcd}\\min\\accolade{\\frac{\\eta_2}{\\kappa_h}, 1}}$ and $\\beta_m$-probabilistically $(\\kef,\\keg;\\Qk)$-fully linear for some $\\beta_f, \\beta_m\\in (0,1)$.\n\\end{assumption}\n\n\nA formal definition from~\\cite{blanchet2016convergence} of the stopping time related to a discrete time stochastic process is as follows.\n\\begin{definition}\nA random variable~$\\tauEpsilon$ is called a stopping time with respect to a given discrete time stochastic process $\\accolade{\\X_k}_{k\\in\\N}$ if the event $\\accolade{\\tauEpsilon=k}$ belongs to the $\\sigma$-algebra $\\sigma\\left(\\X_1,\\dots,\\X_k \\right)$ generated by $\\X_1,\\dots,\\X_k$, for each $k\\in\\N$.\n\\end{definition}\n\nConsider a stochastic process $\\accolade{\\left(\\Phik,\\Dk\\right)}_{k\\in\\N}$, where $\\Phik, \\Dk\\in [0,\\infty)$, and introduce on the same probability space as $\\accolade{\\left(\\Phik,\\Dk\\right)}_{k\\in\\N}$ a biased random walk process $\\accolade{\\Wtildek}_{k\\in\\N}$ obeying the following dynamics:\n\\begin{equation}\\label{dynamicsWk}\n\\prob{\\Wtildekun=1|\\F_k}=q\\qquad \\mbox{ and }\\qquad \\prob{\\Wtildekun=-1|\\F_k}=1-q,\n\\end{equation}\nwhere $q\\in (1/2, 1)$ and $\\F_k=\\sigma\\left( \\left(\\PhiZero,\\Dzero,\\WtildeZero\\right),\\dots,\\left(\\Phik,\\Dk,\\Wtildek\\right) \\right)$, with ${\\WtildeZero=1}$.\n\nLet $\\accolade{\\tauEpsilon}_{\\epsilon>0}$ be a family of stopping times with respect to $\\accoladekinN{\\F_k}$, parameterized by $\\epsilon$. A bound on $\\E{\\tauEpsilon}$ is derived in~\\cite{blanchet2016convergence,dzahini2020expected,paquette2018stochastic} under the following assumption.\n\n\\begin{assumption}\\label{assumptionRenewal}\nThe following hold for the stochastic process $\\accoladekinN{\\left(\\Phik,\\Dk,\\Wtildek\\right)}$.\n\\begin{itemize}\n\\item[(i)] There exist $\\lambda\\in (0, \\infty)$ and $\\dmaxx=\\delta_0 e^{\\lambda j_{\\max}}$ for some $j_{\\max}\\in\\Z$ such that $\\Dk\\leq \\dmaxx$ for all $k$.\n\\item[(ii)] There exists $\\depsilon=\\delta_0 e^{\\lambda j_{\\epsilon}}$, for some $j_{\\epsilon}\\in\\Z$ with $j_{\\epsilon}\\leq 0$ such that for all $k$\n\\begin{equation}\\label{dynamicsRenewal}\n\\itksup\\Dkun\\geq \\itksup\\min\\accolade{\\Dk e^{\\lambda\\Wtildekun}, \\depsilon}, \n\\end{equation}\nwhere $\\Wtildekun$ satisfies~\\eqref{dynamicsWk}. \n\\item[(iii)] There exists a nondecreasing function $h:[0,\\infty)\\to (0,\\infty)$ and a constant $\\varrho>0$ such that for all $k$\n\\[\\E{\\Phikun-\\Phik|\\F_k}\\itksup\\leq-\\varrho h\\left(\\Dk\\right)\\itksup.\\]\n\\end{itemize}\n\\end{assumption}\n\nNoticing that the event $\\accolade{\\Dk\\geq\\depsilon}$ occurs sufficiently frequently on average, often $\\E{\\Phikun-\\Phik}$ can be bounded by some negative fixed constant \\cite{blanchet2016convergence}, thus allowing a bound on the expected stopping time $\\E{\\tauEpsilon}$, as stated next.\n\n\\begin{theorem}\\label{theorComplexity1}\nUnder Assumption~\\ref{assumptionRenewal}, \n$\\E{\\tauEpsilon}\\leq \\frac{q}{2q-1}\\cdot\\frac{\\PhiZero}{\\varrho h\\left(\\depsilon\\right)}+1.$\n\\end{theorem}\n\n\n\\subsection{Expected complexity result.}\\label{ExpectedSubsection}\n\nConsider the process $\\accolade{\\left(\\Phik,\\Dk,\\Wtildek\\right)}_{k\\in\\N}$, where~$\\Phik$ is defined in Theorem~\\ref{zerothOrderScal}, $\\Dk$ is the trust-region radius, and $\\Wtildek$ is  defined by $\\Wtildek=2\\left(\\iak\\iik\\ijk-\\frac{1}{2} \\right)$.  Given $\\epsilon\\in (0,1)$, let $\\tauEpsilon$ be the number of iterations required by Algorithm~\\ref{algoStoScalTR} to first drive the norm of the gradient of~$f$ below $\\epsilon$:  \n\\[\\tauEpsilon:=\\inf \\accolade{k\\in\\N: \\normii{\\nabla f(\\Xk)}\\leq\\epsilon}.\\]\nThen $\\tauEpsilon$ is a stopping time for the stochastic process generated by Algorithm~\\ref{algoStoScalTR}, and hence for $\\accolade{\\left(\\Phik,\\Dk,\\Wtildek\\right)}_{k\\in\\N}$ \\cite{blanchet2016convergence,dzahini2020expected,paquette2018stochastic}. More precisely, the occurrence of $\\accolade{\\tauEpsilon=k}$ can be determined by observing $\\left(\\PhiZero,\\Dzero,\\WtildeZero\\right)$, $\\dots$ $(\\Phikmun,\\Dkmun,\\Wtildekmun)$, which means that $\\tauEpsilon$ is a stopping time with respect to the filtration $\\accoladekinN{\\falgebra}$. To bound $\\E{\\tauEpsilon}$ using Theorem~\\ref{theorComplexity1}, we show next that Assumption~\\ref{assumptionRenewal} holds for $\\accolade{\\left(\\Phik,\\Dk,\\Wtildek\\right)}_{k\\in\\N}$.\n\n\nFrom~\\eqref{PhikIncrementBound}, since $\\falgebra\\subset\\fqalgebra$, we observe that\\footnote{Unlike the analysis in~\\cite{blanchet2016convergence}, the first equality of~\\eqref{PhikIncrSecond} is essential and due to the fact that in the present framework $\\tauEpsilon$ is a stopping time with respect to $\\sigma$-algebras $\\falgebra$ that are smaller than $\\fqalgebra$ with respect to which $\\accoladekinN{\\Phik}$ was proved in Theorem~\\ref{zerothOrderScal} Eq.~\\eqref{PhikIncrementBound} to be a supermartingale.} for all $k\\in\\N$:\n\\begin{equation}\\label{PhikIncrSecond}\n\\E{\\Phikun-\\Phik|\\falgebra}=\\E{\\E{\\Phikun-\\Phik|\\fqalgebra}|\\falgebra}\\leq\\E{-\\varrho\\Dk^2|\\falgebra}=-\\varrho\\Dk^2.\n\\end{equation}\nNote that~\\eqref{PhikIncrSecond} holds for any realizations of the random variables $\\E{\\Phikun-\\Phik|\\falgebra}$ and $\\Dk^2$ and hence on the event $\\accolade{\\tauEpsilon >k}$ in particular, which shows that Assumption~\\ref{assumptionRenewal}-$(iii)$ holds with the constant $\\varrho=\\frac{1}{2}\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})$ of Theorem~\\ref{zerothOrderScal} and $h(t)=t^2$. Assumption~\\ref{assumptionRenewal}-$(i)$ automatically follows from the initialization strategy in Algorithm~\\ref{algoStoScalTR} with $\\lambda=\\ln(\\gamma)$. Before showing by means of Lemma~\\ref{smallLemma}, inspired by~\\cite[Lemma~7]{blanchet2016convergence}, that~\\eqref{dynamicsRenewal} is satisfied, we first define the constant \n\\begin{equation}\\label{deltaEpsilonDefinition}\n\\depsilon:=\\frac{\\epsilon}{\\xi}, \\quad\\mbox{with } \\xi\\geq  \\frac{1}{\\alpha_Q}\\max\\accolade{\\kappa_h+\\kappa_{eg}, \\eta_2+\\kappa_{eg}, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}+\\kappa_{eg}}=:\\hat{\\xi}, \n\\end{equation}\ninspired by the proof of Theorem~\\ref{liminfTheorem}, especially~\\eqref{ineq4}. Then, following ~\\cite{blanchet2016convergence} exactly, one can assume without loss of generality that $\\depsilon=\\gamma^i\\delta_0$ for some integer $i=:j_{\\epsilon}\\leq 0$, whence $\\Dk=\\gamma^{{\\ubar{i}}_k}\\depsilon$ for any $k$ and some integer~${\\ubar{i}}_k$.\n","units":[{"raw_tex":"This paper focuses on unconstrained stochastic DFO, wherein the objective function values are accessible only through a blackbox oracle corrupted by stochastic noise. We consider the problem \n\\begin{equation}\\label{probl1}\n\t\\underset{\\x\\in\\rn}{\\min} f(\\x), \\qquad\\mbox{with}\\quad f(\\x)=\\Esp_{\\ubar{\\theta}}\\left[f_{\\ubar{\\theta}}(\\x)\\right],\n\\end{equation}\n%\\quad\\mbox{with}\\quad f(\\x)=\\Esp_{\\ubar{\\theta}}\\left[f_{\\ubar{\\theta}}(\\x)\\right]\nwhere the values of the continuously differentiable function $f:\\rn\\to\\R$ are available only via $f_{\\ubar{\\theta}}$, a stochastically noisy version of $f$, and where $\\ubar{\\theta}$ is a random variable whose distribution is possibly unknown. \n","start":28799,"end":29485},{"raw_tex":"Unlike the stochastic trust-region framework STORM~\\cite{blanchet2016convergence,chen2018stochastic}, which builds full space random models $\\ubar{m}_k(\\x)$, $\\x\\in\\rn$, to approximate $f(\\x)$, STARS operates as follows. On iteration $k$, given a current iterate $\\xk$, let $\\mathcal{Y}_k\\subseteq \\rn$ be the affine space randomly chosen through the range of a matrix $\\Qk\\in\\rnp$ \n%satisfying $\\norme{\\Qk}\\leq\\Qmax$ for some $\\Qmax>0$, \nwhose entries are randomly selected; that is, \n\\[\\mathcal{Y}_k = \\accolade{\\xk+\\Qk\\bm{{\\s}}:\\bm{{\\s}}\\in\\R^p}.\\]\nThen, given a trust-region radius $\\dk>0$, a subspace model $\\mhatk(\\s), \\s\\in\\rp$ is built only on $\\mathcal{Y}_k$ using realizations of the stochastically noisy function $f_{\\ubar{\\theta}}$; the model  serves as an approximation of $f(\\x)$ in $\\mathcal{B}(\\xk,\\dk;\\Qk):=\\accolade{\\xk+\\Qk\\s \\in \\mathcal{Y}_k:\\normii{\\s}\\leq\\dk}$. For concreteness, here we employ a quadratic subspace model given by \n\\begin{equation*}\nf(\\xk+\\Qk\\s)\\approx\\mhatk(\\bm{\\s}):= f_k+\\gkhat^\\top\\s+\\s^\\top\\Hkhat\\s,\n\\end{equation*}\nwhere $\\gkhat\\in \\R^p$ and $\\Hkhat\\in \\Sym^{p}$ are the low-dimensional model gradient and Hessian, respectively. A {\\it tentative step} $\\Qk\\sk\\in\\rn$ is produced by using the solution $\\sk\\in\\rp$ obtained by approximately minimizing $\\mhatk$ inside the trust  region; that  is, $\\sk\\approx{\\argmin}\\left\\{ \\mhatk(\\s): \\s\\in\\rp, \\, \\normii{\\bm{{\\s}}}\\leq \\dk\\right\\}$. Inspired by~\\cite{CRsubspace2021,chen2018stochastic}, the {\\it trial step} $\\sk$ has to provide a sufficient decrease in $\\mhatk$ by satisfying the following standard fraction of the Cauchy decrease condition.\n\n\\begin{assumption}\nFor every iteration $k$, a trial step $\\sk\\in\\rp$ is computed so that\\footnote{Throughout the manuscript, the matrix norm $\\norme{\\cdot}$ is supposed to be {\\it consistent} with the Euclidean norm; that is, $\\normii{\\Ab\\xb}\\leq \\norme{\\Ab}\\normii{\\xb}$, which holds for both Frobenius and spectral matrix norms.}\n\\begin{equation}\\label{CauchyDecrease}\n\\mhatk(\\bm{0})-\\mhatk(\\bm{\\sk})\\geq \\frac{\\kappa_{fcd}}{2}\\normii{\\gkhat}\\min\\accolade{\\dk,\\frac{\\normii{\\gkhat}}{\\max\\left(\\norme{\\Hkhat},1\\right)}},\n\\end{equation}\nfor some constant $\\kappa_{fcd}\\in (0,1]$.\n\\end{assumption}\n","start":39790,"end":42028},{"raw_tex":"The random variables considered here are all defined on the same probability space $(\\Omega,\\F, \\pr)$, where $\\Omega$, referred to as the {\\it sample space}, is a nonempty set whose elements $\\omega$ and subsets are called {\\it sample points} and {\\it events}, respectively. $\\F$ is a collection of such events, which is called a {\\it $\\sigma$-algebra}. $\\pr$ is a finite measure defined on the measurable space $(\\Omega,\\F)$, satisfying $\\prob{\\Omega}=1$ and referred to as the {\\it probability measure}. An increasing subsequence $\\accolade{\\mathcal{S}_k}_k$ of $\\sigma$-algebras of $\\F$ will be called a {\\it filtration}. Let $\\mathscr{B}(\\rn)$ be the $\\sigma$-algebra generated by the open sets of $\\rn$, also known as the Borel $\\sigma$-algebra of $\\rn$. A random variable $\\ubar{z}$ is a measurable map defined on $(\\Omega,\\F, \\pr)$ into the measurable space $(\\rn, \\mathscr{B}(\\rn))$, where measurability means that $\\accolade{\\ubar{z}\\in A}:=\\accolade{\\omega\\in\\Omega:\\ubar{z}(\\omega)\\in A} =:{\\ubar{z}}^{-1}(A)\\in\\F\\ \\forall A\\in \\mathscr{B}(\\rn)$~\\cite{bhattacharya2007basic}. For the remainder of the manuscript, vectors will be written in lowercase boldface (e.g., $\\x\\in\\rn, n\\geq 2$) while matrices will be written in uppercase boldface (e.g., $\\Qk\\in\\rnp$), and underlined letters (e.g., $\\ubar{z}, \\ubar{\\x}, \\QkRandom$) will be used to denote random quantities; $z=\\ubar{z}(\\omega)$ will denote a realization of $\\ubar{z}$.\n","start":43183,"end":44624},{"raw_tex":"\\begin{algorithm}[htb]\n\\caption{STARS}\n\\label{algoStoScalTR} \n\\textbf{[0] Initialization}\\\\\n\\hspace*{10mm}Choose constants $\\gamma>1$, $\\eta_1\\in(0,1)$, $\\eta_2>0$, $j_{\\max}\\in\\N$, $\\varepsilon\\in(0,1)$, $\\beta\\in(0,1)$, \\\\ \n\\hspace*{10mm}$c_1\\geq 1$, initial trust-region radius $\\delta_0>0$, and maximum trust-region radius\\\\\n\\hspace*{10mm}$\\dmaxx=\\gamma^{j_{\\max}}\\delta_0$, starting point $\\bm{{\\x}_0}\\in\\rn$, and dimension $p\\in\\intbracket{1,n}$.\\\\ \n\\hspace*{10mm}Set the iteration counter $k \\gets 0$.\\\\\n\\textbf{[1] Construction of subspace model}\\\\\n\\hspace*{10mm}Generate $\\Qk$: a realization of a\\footnotemark {\\em WAM}($1-\\varepsilon,\\beta$), using a distribution $\\mathbb{Q}_{k}$.\\\\\n\\hspace*{10mm}Build model $\\mhatk:\\rp\\to\\R$ that is $(\\kef,\\keg;\\Qk)$-fully linear \n%$\\mhatk(\\s)={\\f}_k+\\gkhat^{\\top}\\s+$\\\\\n%\\hspace*{10mm}$\\frac{1}{2}\\s^{\\top}\\Hkhat\\s\\approx f(\\xk+\\Qk \\s)$, that locally approximates $f$ \nin  $\\mathcal{B}(\\xk,c_1\\dk;\\Qk)$.\\\\ \n%\\hspace*{10mm}with the affine space defined by the columns of $\\Qk$.\\\\\n\\textbf{[2] Step calculation}\\\\\n\\hspace*{10mm}Compute $\\sk\\approx {\\argmin}\\left\\{ \\mhatk(\\s) : \\; \\s\\in\\rp:\\normii{\\s}\\leq \\dk\\right \\}$ satisfying~\\eqref{CauchyDecrease}.\\\\\n\\textbf{[3] Estimate computation}\\\\\n\\hspace*{10mm}Obtain estimates $\\fok\\approx f(\\xk)$ and $\\fsk\\approx f(\\xk+\\Qk\\sk)$ satisfying~\\eqref{accEstimates}.\\\\\n\\textbf{[4] Updates}\\\\\n\\hspace*{10mm}Compute $\\rho_k=\\frac{\\fok-\\fsk}{\\mhatk(\\bm{0})-\\mhatk(\\sk)}.$\\\\\n\\hspace*{10mm}If $\\rho_k\\geq \\eta_1$ and $\\normii{\\gkhat}\\geq \\eta_2\\dk$ (\\textbf{success}): \\\\\n\\hspace*{16mm}set $\\bm{\\xkun}=\\xk+\\Qk\\sk$ and $\\dkun=\\min\\accolade{\\gamma\\dk,\\dmaxx}$. \\\\\n\\hspace*{10mm}Otherwise (\\textbf{failure}): set $\\bm{\\xkun}=\\xk$ and $\\dkun=\\gamma^{-1}\\dk$.\\\\\n\\hspace*{10mm}Update the iteration counter $k \\gets k+1$, and go to~\\textbf{[1]}.\\\\\n\\end{algorithm}\n%\\caption[]{\\small{STARS algorithm}}\n%\\end{figure*}\n\\footnotetext{See Definition~\\ref{wellAlignedDefinition} for the meaning of {\\em WAM}($1-\\varepsilon,\\beta$).}\n","start":44648,"end":46651},{"raw_tex":"The deterministic models $\\mhatk$ and the function estimates $\\fok$ and $\\fsk$ are constructed by using evaluations of the noisy function $f_{\\ubar{\\theta}}$ and the randomly selected entries of the deterministic matrix~$\\Qk=\\QkRandom(\\omega)$.  Hence, $\\mhatk$, $\\fok$, and $\\fsk$ can be considered as realizations of random models and estimates $\\Mhatk$, $\\Fok$, and $\\Fsk$, respectively. Each iteration of Algorithm~\\ref{algoStoScalTR} is therefore influenced by the behavior of these random quantities; and consequently, the algorithm results in a stochastic process. The present work shows that under certain conditions on the sequences~$\\accoladekinN{\\QkRandom}$, $\\accoladekinN{\\Fok,\\Fsk}$, and $\\accoladekinN{\\Mhatk}$, the resulting stochastic process has desirable convergence properties, conditioned on the past, where {\\it past} means the {\\it past history} of the algorithm (see~Remark~\\ref{whyTheseSigmaAlgebras}). In particular, for the needs of convergence and expected complexity analyses presented  in Sections~\\ref{Section4} and~\\ref{Section5}, and as will be seen  in Section~\\ref{Section3}, the random models~$\\Mhatk$ and estimates~$\\Fok,\\Fsk$ are required to be probabilistically sufficiently accurate conditioned on the past, as was the case in~\\cite{chen2018stochastic}. Moreover, $\\QkRandom$ must guarantee a high quality of the selected subspace probabilistically and conditioned on the past, as will be detailed in Section~\\ref{Section31}. \n\nFor all $k\\in\\N$, denote by $(\\QkRandom)_{ij},  (i,j)\\in \\intbracket{1,n}\\times\\intbracket{1,p}=:\\mathbb{I}_{n,p}$ the entries of the random matrix $\\QkRandom$, where for $a\\leq b$, $\\intbracket{a,b}:=[a,b]\\cap\\mathbb{Z}$  throughout the manuscript. \nTo formalize conditioning on the past, we consider the three filtrations $\\accolade{\\falgebra}_{k\\in\\N}$, $\\accolade{\\fqalgebra}_{k\\in\\N}$, and $\\accolade{\\fmqalgebra}_{k\\in\\N}$ defined respectively by \n\\begin{equation*}\n\\begin{array}{rcl}\n\\falgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell,  (\\QellRandom)_{ij} \\mbox{ with}\\  \\ell \\in \\intbracket{0,k-1} \\mbox{ \\&}\\ (i,j)\\in \\mathbb{I}_{n,p} \\right)\\\\\n\\fqalgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell, (\\QellRandom)_{ij}, (\\QkRandom)_{ij} \\mbox{ with}\\  \\ell \\in \\intbracket{0,k-1} \\mbox{ \\&}\\ (i,j)\\in \\mathbb{I}_{n,p} \\right)\\\\\n\\fmqalgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell, (\\QellRandom)_{ij}, (\\QkRandom)_{ij}, \\ \\Mhatk \\mbox{ with}\\  \\ell \\in \\intbracket{0,k-1} \\mbox{ \\&}\\ (i,j)\\in \\mathbb{I}_{n,p}  \\right).\n\\end{array}\n\\end{equation*}\n% \\begin{eqnarray*}\n%  \\falgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell\\ \\mbox{and}\\ (\\QellRandom)_{ij}\\ \\mbox{with}\\  \\ell \\in \\intbracket{0,k-1}\\ \\mbox{and}\\ (i,j)\\in \\mathbb{I}_{n,p}\\right)\\\\\n%  \\fqalgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell,\\ (\\QellRandom)_{ij}\\ \\mbox{and}\\ (\\QkRandom)_{ij}\\ \\mbox{with}\\  \\ell \\in \\intbracket{0,k-1}\\ \\mbox{and}\\ (i,j)\\in \\mathbb{I}_{n,p}\\right)\\\\\n%  \\fmqalgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell,\\ (\\QellRandom)_{ij},\\ \\Mhatk\\ \\mbox{and}\\ (\\QkRandom)_{ij}\\ \\mbox{with}\\  \\ell \\in \\intbracket{0,k-1}\\ \\mbox{and}\\ (i,j)\\in \\mathbb{I}_{n,p}\\right).\n% \\end{eqnarray*}\nHere, for {\\it completeness}~\\cite{billingsley1995ThirdEdition}, $\\mathcal{F}_{-1}=\\sigma(\\bm{\\x_0})$.\n%One can easily see that . \nBy construction, $\\falgebra\\subset\\fqalgebra\\subset\\fmqalgebra$, $\\QkRandom$ is both~$\\fqalgebra$ and~$\\fmqalgebra$-measurable, and $\\Mhatk$ is~$\\fmqalgebra$-measurable, which imply that $\\E{\\QkRandom|\\fqalgebra}=\\E{\\QkRandom|\\fmqalgebra}=\\QkRandom$ and $\\E{\\Mhatk|\\fmqalgebra}=\\Mhatk$. \n%The  motivation for  introducing these $\\sigma$-algebras in the formalization of conditioning on the past is explained next.\n\n\\begin{remark}\\label{whyTheseSigmaAlgebras}\nAt a given iteration~$k$, when generating~$\\QkRandom$, because no model or estimates were generated since the start of the iteration, it obviously follows from the construction of Algorithm~\\ref{algoStoScalTR} that its past history can be formalized by~$\\falgebra$. On the other hand, when constructing~$\\Mhatk$, the past of the algorithm that includes~$\\QkRandom$, given that the latter is already generated since the beginning of iteration~$k$, can therefore be formalized by~$\\fqalgebra$. A similar observation  explains why~$\\fmqalgebra$ can formalize the past history of Algorithm~\\ref{algoStoScalTR} before the construction of~$\\Fok$ and~$\\Fsk$. These observations will also play an important role in the formalization of sufficient accuracy of the random models and estimates in Definitions~\\ref{defProbQkfully} and~\\ref{defProbAccEstim} and the formalization of high quality of the random subspaces in Assumption~\\ref{wellAlignedAssumption}.  \n\\end{remark}\n","start":46654,"end":51356},{"raw_tex":"Recalling Remark~\\ref{whyTheseSigmaAlgebras} and inspired by~\\cite{CRsubspace2021}, we propose a probabilistic quantification of the quality of the subspace selection as follows.\n\n\\begin{definition}\\label{wellAlignedDefinition}\nFor fixed $\\alpha_Q,\\beta_Q\\in (0,1)$, a sequence $\\accoladekinN{\\QkRandom}$ of random matrices \n%determining the selection of subspaces \nis $(1-\\beta_Q)$-probabilistically $\\alpha_{Q}$-well aligned if, for any $\\F_{k-1}$-measurable random vector $\\viRandom$ with realizations $\\vi\\in \\rn$, the events \n$\\mathcal{A}_k:= \\accolade{\\normii{\\QkRandomTranspose\\viRandom}\\geq \\alpha_{{Q}}\\normii{\\viRandom}}$ satisfy \n\\begin{equation}\\label{Akineq}\n\\prob{\\mathcal{A}_k|\\F_{k-1}}=\\E{\\iak|\\F_{k-1}}\\geq 1-\\beta_Q,\n\\end{equation}\nwhere $\\iak$ denotes the indicator function of the event $\\mathcal{A}_k$; that  is, $\\iak(\\omega)=1$ if $\\omega\\in \\mathcal{A}_k$ and $\\iak(\\omega)=0$ otherwise. Random matrices satisfying~\\eqref{Akineq} will be referred to as {WAM}($\\alpha_Q,\\beta_Q$).\n%, in other words, well-aligned matrices with parameters $\\alpha_Q$ and $\\beta_Q$.\n\\end{definition}\n%\\begin{definition}\n%Random matrices satisfying the conditions of Lemma... with parameters $\\varepsilon, \\beta$ will be referred to as a {\\em JLM}($\\varepsilon,\\beta$), i.e., Johnson-Lindenstrauss matrices with parameters~$\\varepsilon$ and~$\\beta$.\n%\\end{definition}\nFor convergence purposes  in Sections~\\ref{Section4} and~\\ref{Section5}, the following will be assumed throughout the manuscript.\n\\begin{assumption}\\label{wellAlignedAssumption}\nThe sequence $\\accoladekinN{\\QkRandom}$ of matrices used by Algorithm~\\ref{algoStoScalTR} is $(1-\\beta_Q)$-probabilistically $\\alpha_{Q}$-well aligned for some fixed $\\alpha_Q\\in (0,1)$ and $\\beta_Q\\in (0,1/2)$.\n\\end{assumption}\n","start":53148,"end":54926},{"raw_tex":"Inspired by~\\cite{CRsubspace2021}, a technical assumption needed for convergence and expected complexity analyses later in Sections~\\ref{Section4} and~\\ref{Section5} is the following. \n\\begin{assumption}\\label{QmaxAssumption}\nAny realization $\\Qk$ of the random matrix used by Algorithm~\\ref{algoStoScalTR} satisfies $\\norme{\\Qk}\\leq \\Qmax$ for all $k\\in\\N$, for some constant $\\Qmax >0$ independent of~$k$.\n\\end{assumption}\n","start":63629,"end":64054},{"raw_tex":"\\subsection{Probabilistic models and estimates in random subspaces.}\nAs mentioned at the beginning of Section~\\ref{Section3}, the models used by Algorithm~\\ref{algoStoScalTR} need to be sufficiently accurate. Motivated by~\\cite{chen2018stochastic}, this sufficient accuracy is formalized for deterministic subspace models by the following measure of accuracy inspired by~\\cite{CRsubspace2021}.\n\\begin{definition}\\label{QkFullyLinearModels}\nAssume $\\nabla f$ is  Lipschitz continuous. Given $\\Qk\\in\\rnp$ and $\\xk\\in\\rn$, a function $\\mhatk:\\rp\\to\\R$ is a $(\\kef,\\keg; \\Qk)$-fully linear model of $f$ in $\\mathcal{B}(\\xk,\\dk;\\Qk)$ for some $\\dk>0$ if there exist constants $\\kappa_{ef}, \\kappa_{eg} >0$ independent of $k$ such that for all $\\s\\in\\rp$ with $\\normii{\\s}\\leq \\dk$,\n\\begin{equation}\\label{Eq36}\n%\\begin{split}\n\\abs{f(\\xk +\\Qk\\s)-\\mhatk(\\s)}\\leq \\kappa_{ef}\\dk^2\\quad \\mbox{and}\\quad\n\\normii{\\Qk^{\\top}\\nabla f(\\xk +\\Qk\\s)-\\nabla\\mhatk(\\s)}\\leq \\kappa_{eg}\\dk.\n%\\end{split}\n\\end{equation}\n\\end{definition}\nAs pointed out in~\\cite{CRsubspace2021}, the gradient condition in~\\eqref{Eq36} is due to the fact that given any fixed~$\\x$ and~$\\Q$, if $\\hat{f}(\\s):=f(\\x+\\Q\\s)$, then $\\nabla_{\\s} \\hat{f}(\\s)=\\Q^{\\top}\\nabla_{\\x} f(\\x+\\Q\\s)$. Moreover, denoting by $\\bm{I}$ the identity matrix in full-dimensional subspaces where $p=n$, then $(\\kef,\\keg; \\bm{I})$-fully linear models correspond to standard fully linear models~\\cite[Definition~6.1]{CoScVibook}.\n\n%Inspired by~\\cite{chen2018stochastic}, and \nRecalling Remark~\\ref{whyTheseSigmaAlgebras}, the next definition formalizing sufficient accuracy of probabilistic subspace models $\\Mhatk$ is a stochastic variant of Definition~\\ref{QkFullyLinearModels}, which generalizes~\\cite[Definition~3.4]{chen2018stochastic} to low-dimensional subspaces. The existence of such models will be rigorously demonstrated  in Theorem~\\ref{subspaceModel1} and Corollary~\\ref{CorQkFully}.\n\\begin{definition}\\label{defProbQkfully}\nA sequence $\\accolade{\\Mhatk}$ of random models is $\\beta_m$-probabilistically \\\\ $(\\kef,\\keg; \\Qk)$-fully linear with respect to the random sequence $\\accolade{\\mathcal{B}(\\Xk,\\Dk;\\QkRandom)}$ if the events\n\\[I_k^Q=\\accolade{\\mbox{Given}\\ \\Qk\\ \\mbox{and}\\ \\xk, \\Mhatk\\ \\mbox{is}\\ (\\kef,\\keg;\\Qk)\\mbox{-fully linear for}\\ f\\ \\mbox{in}\\ \\mathcal{B}(\\Xk,\\Dk;\\QkRandom)}\\]\nsatisfy the supermartingale-like condition\n\\begin{equation}\\label{ikQProb}\n\\prob{I^Q_k|\\fqalgebra}=\\E{\\iik|\\fqalgebra}\\geq \\beta_m.\n\\end{equation}\n\\end{definition}\nRecalling that estimates of unknown function values also need to be sufficiently accurate, we introduce a formalization of sufficient accuracy inspired by~\\cite{audet2019stomads,chen2018stochastic,dzahini2020expected,dzahini2020constrained,paquette2018stochastic} and motivated by~\\cite{CRsubspace2021} as follows.\n\\begin{definition}\nGiven $\\ef>0$ and $\\Qk\\sk\\in\\rn, \\, \\xk\\in\\rn$, $\\fok$ and $\\fsk$ are called $\\ef$-accurate estimates of $f(\\xk)$ and $f(\\xk+\\Qk\\sk)$, respectively, for a given $\\dk$ if\n\\begin{equation}\\label{accEstimates}\n\\abs{\\fok-f(\\xk)}\\leq\\ef\\dk^2\\quad \\mbox{ and }\\quad  \\abs{\\fsk-f(\\xk+\\Qk\\sk)}\\leq\\ef\\dk^2.\n\\end{equation}\n\\end{definition}\n\n\nRecalling Remark~\\ref{whyTheseSigmaAlgebras},\n%and following~\\cite{audet2019stomads,chen2018stochastic,dzahini2020expected,dzahini2020constrained,paquette2018stochastic}, \nwe  extend  this definition to the following stochastic variant.\n\\begin{definition}\\label{defProbAccEstim}\nA sequence of random estimates $\\accolade{\\Fok,\\Fsk}$ is said to be $\\beta_f$-probabilistically $\\ef$-accurate with respect to the corresponding sequence $\\accolade{\\Xk,\\Dk,\\QkRandom\\Sk}$ if the events \n\\begin{eqnarray*}\nJ_k^Q &=&\\left\\lbrace\\mbox{Given}\\ \\Qk\\ \\mbox{and}\\ \\xk, \\Fok \\mbox{ and } \\Fsk \\mbox{ are }\\ef\\mbox{-accurate estimates of } f(\\xk)  \\mbox{ and}\\right.\\\\\n& &\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad \\left.  f(\\xk+\\Qk\\sk), \\mbox{ respectively, for } \\Dk \\right\\rbrace\n\\end{eqnarray*}\n% \nsatisfy the supermartingale-like condition\n$\\prob{J^Q_k|\\fmqalgebra}\\geq \\beta_f.$\n\\end{definition}\n\n\nBased on Definitions~\\ref{wellAlignedDefinition}, \\ref{defProbQkfully}, and~\\ref{defProbAccEstim}, {\\it true} iterations, in other words, those for which $\\iak\\iik\\ijk=1$, occur with a total probability of at least $\\tilde{\\beta}:=\\beta_f\\beta_m(1-\\beta_Q)$ conditioned on $\\falgebra$. Noting that by construction $\\iak$ is both $\\fqalgebra$-measurable and $\\fmqalgebra$-measurable since $\\fqalgebra\\subset\\fmqalgebra$, and that $\\iik$ is $\\fmqalgebra$-measurable, then for $F:= \\accolade{\\mathcal{A}_k\\cap I^Q_k\\cap J_k^Q}$, we have\n\\begin{equation}\\label{prodBetaIneq}\n\\begin{split}\n\\pr(F|\\falgebra)&=\\E{\\iak\\iik\\ijk|\\falgebra}\n=\\E{\\iak\\iik\\E{\\ijk|\\fmqalgebra} |\\falgebra}\\\\\n&\\geq \\beta_f\\E{\\iak\\E{\\iik|\\fqalgebra} |\\falgebra}\\geq \\beta_f\\beta_m\\E{\\iak|\\falgebra}\\geq \\tilde{\\beta}.\n%&\\geq \\beta_f\\beta_m(1-\\beta_Q),\n\\end{split}\n\\end{equation}\nThe second equality and the first two inequalities used the  {\\it William's Tower Property} for conditional expectations~\\cite[Theorem~34.4.]{billingsley1995ThirdEdition}, namely, the fact that if ${\\ubar{z}}$ is integrable and the $\\sigma$-algebras $\\mathcal{G}_1$ and $\\mathcal{G}_2$ satisfy $\\mathcal{G}_1\\subset\\mathcal{G}_2$, then $\\E{{\\ubar{z}}|\\mathcal{G}_1}=\\E{\\E{{\\ubar{z}}|\\mathcal{G}_2}|\\mathcal{G}_1}$. Moreover, for a $\\sigma$-algebra $\\mathcal{G}$,  $\\E{{\\ubar{r}}{\\ubar{u}}|\\mathcal{G}}={\\ubar{r}}\\E{{\\ubar{u}}|\\mathcal{G}}$ if~${\\ubar{r}}$ is $\\mathcal{G}$-measurable and~${\\ubar{r}}$ and~${\\ubar{r}}{\\ubar{u}}$ are integrable~\\cite[Theorem~34.3.]{billingsley1995ThirdEdition}.\n\nNote that even though the present algorithmic framework does not distinguish true iterations from {\\it false} ones, it holds that $\\prob{\\underset{k\\to\\infty}{\\limsup}\\ \\ubar{w}_k=\\infty}=1$, where\n\\begin{equation}\\label{randomWalk}\n\\ubar{w}_k:=\\sum_{i=0}^k\\left(2\\mathds{1}_{\\mathcal{A}_i}\\mathds{1}_{I^Q_i}\\mathds{1}_{J^Q_i}-1\\right),\n\\end{equation}\nas will be seen later in the proof of Theorem~\\ref{liminfTheorem} provided $\\tilde{\\beta}>1/2$. This shows that true iterations occur sufficiently often for the convergence of Algorithm~\\ref{algoStoScalTR} to hold (see, e.g.,~\\cite[Theorem~3.6]{dzahini2020constrained} for a similar result).\n\n\n\\begin{remark}%\\label{remarkProdBetafBetam}\nCalculations similar to those in~\\eqref{prodBetaIneq} easily yield \n\\begin{eqnarray}\\label{prodBetafBetam}\n\\prob{I^Q_k\\cap J_k^Q|\\fqalgebra}\\geq \\beta_f\\beta_m, & & \\prob{J_k^Q|\\fqalgebra}\\geq \\beta_f,\\\\\n%\\end{equation}\n%\\begin{equation}\n\\label{otherProbIneqs}\n\\prob{I^Q_k|\\falgebra}\\geq\\beta_m & \\qquad \\mbox{ and } \\qquad & \\prob{J^Q_k|\\falgebra}\\geq\\beta_f,\n\\end{eqnarray}\nwhich will be useful in the proofs of Theorems~\\ref{zerothOrderScal} and~\\ref{liminfTheorem}, respectively. \n%and as was considered in~\\cite[Theorems~4.11]{chen2018stochastic} in similar situations.\n\\end{remark}\n","start":67168,"end":74056},{"raw_tex":"In the remainder of the manuscript, the following inspired by~\\cite[Assumptions~4.1 and~4.3]{chen2018stochastic}, respectively, will be assumed.\n\\begin{assumption}\\label{fGradfAssumption}\nFor given $\\bm{\\x_0}$, $\\dmaxx$, and $\\bm{Q_0}$, let $\\mathscr{L}(\\bm{\\x_0};\\bm{Q_0})\\subset\\rn$ be the set containing all iterates of Algorithm~\\ref{algoStoScalTR}. Define the region $\\mathscr{L}^{\\star}(\\bm{\\x_0};\\bm{Q_0})$ considered by the algorithm realizations as $\\mathscr{L}^{\\star}(\\bm{\\x_0};\\bm{Q_0})=\\ds{\\underset{\\accolade{k\\, \\geq\\, 0,\\, \\xk\\in\\mathscr{L}(\\bm{\\x_0};\\bm{Q_0})}}{\\bigcup}}\\mathcal{B}(\\xk,\\dmaxx;\\Qk)$. Then $f(\\x)\\geq f_{\\min}$ for all $\\x\\in\\rn$ and for some constant $f_{\\min}>-\\infty$. Moreover, $f$ and its gradient are Lipschitz continuous on $\\mathscr{L}^{\\star}(\\bm{\\x_0};\\bm{Q_0})$.\n\\end{assumption}\n\n\\begin{assumption}\\label{boundedHessian}\nThere exists some $\\kappa_h\\geq 1$ such that for all $k\\geq 0$, the Hessian $\\Hkhat$ of all realizations of $\\Mhatk$ satisfies $\\norme{\\Hkhat}\\leq \\kappa_h.$\n\\end{assumption}\n","start":90450,"end":91491},{"raw_tex":"\\begin{lemma}\\label{LemmaC}\nLet Assumption~\\ref{boundedHessian} hold. Assume that $\\mhatk$ is $(\\kef,\\keg;\\Qk)$-fully linear and the estimates $\\accolade{\\fok,\\fsk}$ are $\\ef$-accurate with $\\ef\\leq\\kappa_{ef}$. Then the $k$th iteration is successful if \n\\begin{equation}\\label{deltaLemmaC}\n\\dk\\leq\\min\\accolade{\\frac{1}{\\kappa_h}, \\frac{1}{\\eta_2}, \\frac{\\kappa_{fcd}(1-\\eta_1)}{8\\kappa_{ef}}}\\normii{\\gkhat}.\n\\end{equation}\n\\end{lemma}","start":93481,"end":93918},{"raw_tex":"\\begin{lemma}\\label{LemmaD}\nSuppose that Assumption~\\ref{boundedHessian} holds and the estimates $\\accolade{\\fok,\\fsk}$ are $\\ef$-accurate with $\\ef<\\frac{1}{4}\\eta_1\\eta_2\\kappa_{fcd}\\min\\accolade{\\frac{\\eta_2}{\\kappa_h}, 1}$. If the $k$th iteration is successful, then the improvement in $f$ is such that\n\\begin{equation}\\label{LemmaDimprovement}\nf(\\xkun)-f(\\xk)\\leq -C_2\\dk^2,\n\\end{equation}\nwhere $C_2=\\frac{1}{2}\\eta_1\\eta_2\\kappa_{fcd}\\min\\accolade{\\frac{\\eta_2}{\\kappa_h}, 1} -2\\ef>0$. \n\\end{lemma}","start":94242,"end":94747},{"raw_tex":"\\subsection{Zeroth-order convergence result.}\\label{Section4p2}\nTo prove in Theorem~\\ref{zerothOrderScal} that the sequence of random trust-region radii converges to zero almost surely, we assume the following.\n\\begin{assumption}\\label{assumptionsOnEstimatesAndModels} \nThe sequences of estimates $\\accolade{\\Fok, \\Fsk}$ and models $\\accolade{\\Mhatk}$ generated by Algorithm~\\ref{algoStoScalTR} are, respectively, $\\beta_f$-probabilistically $\\ef$-accurate for \n$\\ef<\\min\\accolade{\\kappa_{ef}, \\frac{1}{4}\\eta_1\\eta_2\\kappa_{fcd}\\min\\accolade{\\frac{\\eta_2}{\\kappa_h}, 1}}$ and $\\beta_m$-probabilistically $(\\kef,\\keg;\\Qk)$-fully linear for some $\\beta_f, \\beta_m\\in (0,1)$.\n\\end{assumption}\n","start":99141,"end":99832},{"raw_tex":"A formal definition from~\\cite{blanchet2016convergence} of the stopping time related to a discrete time stochastic process is as follows.\n\\begin{definition}\nA random variable~$\\tauEpsilon$ is called a stopping time with respect to a given discrete time stochastic process $\\accolade{\\X_k}_{k\\in\\N}$ if the event $\\accolade{\\tauEpsilon=k}$ belongs to the $\\sigma$-algebra $\\sigma\\left(\\X_1,\\dots,\\X_k \\right)$ generated by $\\X_1,\\dots,\\X_k$, for each $k\\in\\N$.\n\\end{definition}\n\nConsider a stochastic process $\\accolade{\\left(\\Phik,\\Dk\\right)}_{k\\in\\N}$, where $\\Phik, \\Dk\\in [0,\\infty)$, and introduce on the same probability space as $\\accolade{\\left(\\Phik,\\Dk\\right)}_{k\\in\\N}$ a biased random walk process $\\accolade{\\Wtildek}_{k\\in\\N}$ obeying the following dynamics:\n\\begin{equation}\\label{dynamicsWk}\n\\prob{\\Wtildekun=1|\\F_k}=q\\qquad \\mbox{ and }\\qquad \\prob{\\Wtildekun=-1|\\F_k}=1-q,\n\\end{equation}\nwhere $q\\in (1/2, 1)$ and $\\F_k=\\sigma\\left( \\left(\\PhiZero,\\Dzero,\\WtildeZero\\right),\\dots,\\left(\\Phik,\\Dk,\\Wtildek\\right) \\right)$, with ${\\WtildeZero=1}$.\n\nLet $\\accolade{\\tauEpsilon}_{\\epsilon>0}$ be a family of stopping times with respect to $\\accoladekinN{\\F_k}$, parameterized by $\\epsilon$. A bound on $\\E{\\tauEpsilon}$ is derived in~\\cite{blanchet2016convergence,dzahini2020expected,paquette2018stochastic} under the following assumption.\n\n\\begin{assumption}\\label{assumptionRenewal}\nThe following hold for the stochastic process $\\accoladekinN{\\left(\\Phik,\\Dk,\\Wtildek\\right)}$.\n\\begin{itemize}\n\\item[(i)] There exist $\\lambda\\in (0, \\infty)$ and $\\dmaxx=\\delta_0 e^{\\lambda j_{\\max}}$ for some $j_{\\max}\\in\\Z$ such that $\\Dk\\leq \\dmaxx$ for all $k$.\n\\item[(ii)] There exists $\\depsilon=\\delta_0 e^{\\lambda j_{\\epsilon}}$, for some $j_{\\epsilon}\\in\\Z$ with $j_{\\epsilon}\\leq 0$ such that for all $k$\n\\begin{equation}\\label{dynamicsRenewal}\n\\itksup\\Dkun\\geq \\itksup\\min\\accolade{\\Dk e^{\\lambda\\Wtildekun}, \\depsilon}, \n\\end{equation}\nwhere $\\Wtildekun$ satisfies~\\eqref{dynamicsWk}. \n\\item[(iii)] There exists a nondecreasing function $h:[0,\\infty)\\to (0,\\infty)$ and a constant $\\varrho>0$ such that for all $k$\n\\[\\E{\\Phikun-\\Phik|\\F_k}\\itksup\\leq-\\varrho h\\left(\\Dk\\right)\\itksup.\\]\n\\end{itemize}\n\\end{assumption}\n\nNoticing that the event $\\accolade{\\Dk\\geq\\depsilon}$ occurs sufficiently frequently on average, often $\\E{\\Phikun-\\Phik}$ can be bounded by some negative fixed constant \\cite{blanchet2016convergence}, thus allowing a bound on the expected stopping time $\\E{\\tauEpsilon}$, as stated next.\n\n\\begin{theorem}\\label{theorComplexity1}\nUnder Assumption~\\ref{assumptionRenewal}, \n$\\E{\\tauEpsilon}\\leq \\frac{q}{2q-1}\\cdot\\frac{\\PhiZero}{\\varrho h\\left(\\depsilon\\right)}+1.$\n\\end{theorem}\n","start":119853,"end":122564},{"raw_tex":"\\subsection{Expected complexity result.}\\label{ExpectedSubsection}\n\nConsider the process $\\accolade{\\left(\\Phik,\\Dk,\\Wtildek\\right)}_{k\\in\\N}$, where~$\\Phik$ is defined in Theorem~\\ref{zerothOrderScal}, $\\Dk$ is the trust-region radius, and $\\Wtildek$ is  defined by $\\Wtildek=2\\left(\\iak\\iik\\ijk-\\frac{1}{2} \\right)$.  Given $\\epsilon\\in (0,1)$, let $\\tauEpsilon$ be the number of iterations required by Algorithm~\\ref{algoStoScalTR} to first drive the norm of the gradient of~$f$ below $\\epsilon$:  \n\\[\\tauEpsilon:=\\inf \\accolade{k\\in\\N: \\normii{\\nabla f(\\Xk)}\\leq\\epsilon}.\\]\nThen $\\tauEpsilon$ is a stopping time for the stochastic process generated by Algorithm~\\ref{algoStoScalTR}, and hence for $\\accolade{\\left(\\Phik,\\Dk,\\Wtildek\\right)}_{k\\in\\N}$ \\cite{blanchet2016convergence,dzahini2020expected,paquette2018stochastic}. More precisely, the occurrence of $\\accolade{\\tauEpsilon=k}$ can be determined by observing $\\left(\\PhiZero,\\Dzero,\\WtildeZero\\right)$, $\\dots$ $(\\Phikmun,\\Dkmun,\\Wtildekmun)$, which means that $\\tauEpsilon$ is a stopping time with respect to the filtration $\\accoladekinN{\\falgebra}$. To bound $\\E{\\tauEpsilon}$ using Theorem~\\ref{theorComplexity1}, we show next that Assumption~\\ref{assumptionRenewal} holds for $\\accolade{\\left(\\Phik,\\Dk,\\Wtildek\\right)}_{k\\in\\N}$.\n\n\nFrom~\\eqref{PhikIncrementBound}, since $\\falgebra\\subset\\fqalgebra$, we observe that\\footnote{Unlike the analysis in~\\cite{blanchet2016convergence}, the first equality of~\\eqref{PhikIncrSecond} is essential and due to the fact that in the present framework $\\tauEpsilon$ is a stopping time with respect to $\\sigma$-algebras $\\falgebra$ that are smaller than $\\fqalgebra$ with respect to which $\\accoladekinN{\\Phik}$ was proved in Theorem~\\ref{zerothOrderScal} Eq.~\\eqref{PhikIncrementBound} to be a supermartingale.} for all $k\\in\\N$:\n\\begin{equation}\\label{PhikIncrSecond}\n\\E{\\Phikun-\\Phik|\\falgebra}=\\E{\\E{\\Phikun-\\Phik|\\fqalgebra}|\\falgebra}\\leq\\E{-\\varrho\\Dk^2|\\falgebra}=-\\varrho\\Dk^2.\n\\end{equation}\nNote that~\\eqref{PhikIncrSecond} holds for any realizations of the random variables $\\E{\\Phikun-\\Phik|\\falgebra}$ and $\\Dk^2$ and hence on the event $\\accolade{\\tauEpsilon >k}$ in particular, which shows that Assumption~\\ref{assumptionRenewal}-$(iii)$ holds with the constant $\\varrho=\\frac{1}{2}\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})$ of Theorem~\\ref{zerothOrderScal} and $h(t)=t^2$. Assumption~\\ref{assumptionRenewal}-$(i)$ automatically follows from the initialization strategy in Algorithm~\\ref{algoStoScalTR} with $\\lambda=\\ln(\\gamma)$. Before showing by means of Lemma~\\ref{smallLemma}, inspired by~\\cite[Lemma~7]{blanchet2016convergence}, that~\\eqref{dynamicsRenewal} is satisfied, we first define the constant \n\\begin{equation}\\label{deltaEpsilonDefinition}\n\\depsilon:=\\frac{\\epsilon}{\\xi}, \\quad\\mbox{with } \\xi\\geq  \\frac{1}{\\alpha_Q}\\max\\accolade{\\kappa_h+\\kappa_{eg}, \\eta_2+\\kappa_{eg}, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}+\\kappa_{eg}}=:\\hat{\\xi}, \n\\end{equation}\ninspired by the proof of Theorem~\\ref{liminfTheorem}, especially~\\eqref{ineq4}. Then, following ~\\cite{blanchet2016convergence} exactly, one can assume without loss of generality that $\\depsilon=\\gamma^i\\delta_0$ for some integer $i=:j_{\\epsilon}\\leq 0$, whence $\\Dk=\\gamma^{{\\ubar{i}}_k}\\depsilon$ for any $k$ and some integer~${\\ubar{i}}_k$.\n","start":122565,"end":125900}]},"statement":{"raw_tex":"\\begin{theorem}\nLet all the assumptions made in Theorem~\\ref{liminfTheorem} hold. Then \n\\[\\E{\\tauEpsilon}\\leq \\frac{\\tilde{\\beta}}{2\\tilde{\\beta}-1}\\cdot\\frac{\\PhiZero\\tilde{\\xi}^2}{\\varrho \\epsilon^2}+1\\]\nfor some $\\tilde{\\xi}\\geq\\hat{\\xi}$, where $\\varrho$ is the constant of Theorem~\\ref{zerothOrderScal}, $\\tilde{\\beta}$ is the same constant of Lemma~\\ref{smallLemma}, and $\\hat{\\xi}$ is defined in~\\eqref{deltaEpsilonDefinition}.\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\nLet all the assumptions made in Theorem~\\ref{liminfTheorem} hold. Then \n\\[\\E{\\tauEpsilon}\\leq \\frac{\\tilde{\\beta}}{2\\tilde{\\beta}-1}\\cdot\\frac{\\PhiZero\\tilde{\\xi}^2}{\\varrho \\epsilon^2}+1\\]\nfor some $\\tilde{\\xi}\\geq\\hat{\\xi}$, where $\\varrho$ is the constant of Theorem~\\ref{zerothOrderScal}, $\\tilde{\\beta}$ is the same constant of Lemma~\\ref{smallLemma}, and $\\hat{\\xi}$ is defined in~\\eqref{deltaEpsilonDefinition}.\n\\end{theorem}","start":127519,"end":127967}]},"proof":{"raw_tex":"\\begin{lemma}\\label{LemmaA}\nSuppose that the function $\\mhatk$ is a $(\\kef,\\keg;\\Qk)$-fully linear model for~$f$ in $\\B{\\xk}{\\dk;\\Qk}$. Then the trial step $\\sk$ leads to an improvement in $f$ such that\n\\begin{equation}\\label{LemmaAimprovement}\nf(\\xk+\\Qk\\sk)-f(\\xk)\\leq -\\frac{\\kappa_{fcd}}{4}\\normii{\\gkhat}\\dk,\n%\\end{equation}\n%\\begin{equation}\\label{deltaLemmaA}\n\\quad\n\\mbox{if }\n\\dk\\leq \\min\\accolade{\\frac{1}{\\kappa_h},\\frac{\\kappa_{fcd}}{8\\kappa_{ef}}}\\normii{\\gkhat}.\n\\end{equation}\n\\end{lemma}\n\n\\begin{lemma}\\label{LemmaB2}\nLet Assumption~\\ref{boundedHessian} hold, and suppose the model $\\mhatk$ is $(\\kef,\\keg;\\Qk)$-fully linear. Define $C_1=\\frac{\\kappa_{fcd}}{4} \\max\\accolade{\\frac{\\kappa_h}{\\kappa_h+\\kappa_{eg}}, \\frac{8\\kappa_{ef}}{8\\kappa_{ef}+\\kappa_{fcd}\\kappa_{eg}}}$. Then the trial step $\\sk$ leads to an improvement in $f$ such that \n\n\\begin{equation}\\label{LemmaB2improvement}\nf(\\xk+\\Qk\\sk)-f(\\xk)\\leq -C_1\\normii{\\QkTranspose\\nabla f(\\xk)}\\dk,\n\\end{equation}\n\\begin{equation}\\label{deltaLemmaB2}\n\\mbox{if}\\quad\n\\dk\\leq \\min\\accolade{\\frac{1}{\\kappa_h+\\kappa_{eg}}, \\frac{1}{\\frac{8\\kappa_{ef}}{\\kappa_{fcd}}+\\kappa_{eg}}}\\normii{\\QkTranspose\\nabla f(\\xk)}. \n\\end{equation}\n\\end{lemma}\n\n\\begin{lemma}\\label{LemmaE}\nUnder Assumptions~\\ref{QmaxAssumption} and~\\ref{fGradfAssumption}, assume that Algorithm~\\ref{algoStoScalTR} erroneously accepts a step $\\sk$ leading to an increase in the objective function when $\\norme{\\QkTranspose\\nabla f(\\xk)}\\geq \\zeta\\dk$ for some constant $\\zeta>0$, and let $C_3(\\zeta)=1+\\frac{3L_g}{2\\zeta}\\Qmax^2$. Then the increase in $f$ is such that\n\\begin{equation}\\label{LemmaEincrease}\nf(\\xk+\\Qk\\sk)-f(\\xk)\\leq C_3(\\zeta)\\norme{\\QkTranspose\\nabla f(\\xk)}\\dk.\n\\end{equation}\n\\end{lemma}\n\n\\begin{lemma}\\label{partialMeasurable}\nAny partial derivative \n$\\frac{\\partial f}{\\partial x^i}: \\left(\\rn, \\mathscr{B}(\\rn)\\right)\\to \\left(\\R, \\mathscr{B}(\\R)\\right),\\ i\\in\\intbracket{1,n} $,\nof a differentiable function $f$ is a Borel measurable function.\n\\end{lemma}\n\n\\begin{proof}\nLet $(\\bm{e_1},\\bm{e_2},\\dots,\\bm{e_n})$ be the canonical basis of $\\rn$. For any $i\\in\\intbracket{1,n} $ and $\\bm{x}=(x^1,x^2,\\dots,x^n)$, define, for all $j\\in\\N$,\n$D_{ij}(\\bm{x})=\\frac{f(\\bm{x}+h_j\\bm{e_i})-f(\\bm{x})}{h_j}$,\nwhere $\\accolade{h_j}_{j\\in\\N}$ is a sequence of positive real numbers converging to~$0$; $f$ is differentiable and hence continuous, and thus is Borel measurable~\\cite[Theorem~13.2]{billingsley1995ThirdEdition}. Therefore, since for all $i\\in\\intbracket{1,n}$, $\\underset{j\\to\\infty}{\\lim}D_{ij}(\\bm{x})=\\frac{\\partial f}{\\partial x^i}(\\bm{x}),$\nthen $\\frac{\\partial f}{\\partial x^i}$ is also Borel measurable~\\cite[Theorem~13.3 and Theorem~13.4-(ii)]{billingsley1995ThirdEdition}.\n\\end{proof}\n\n\\begin{lemma}\\label{indicatorMeasurable}\nLet $\\zeta\\geq 0$ be a constant, and consider the event $\\Gamma_k:=\\accolade{\\Gk\\geq 0}$, where $\\Gk:=\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}-\\zeta\\Dk$. Then $\\Gk$, $\\igk$, and $\\igkc$ are $\\fqalgebra$-measurable.\n\\end{lemma}\n\n\\begin{proof}\nBy construction, all the entries $\\left(\\QkRandom\\right)_{ij},\\ (i,j)\\in\\intbracket{1,n}\\times\\intbracket{1,p}$ of $\\QkRandom$ are $\\fqalgebra$-measurable, while $\\Dk$ and the components $\\Xk^i, \\ i\\in\\intbracket{1, n}$ of $\\Xk$ are $\\falgebra$-measurable and hence $\\fqalgebra$-measurable since $\\falgebra\\subseteq\\fqalgebra$. Thus, for any $i\\in\\intbracket{1, n}$, $\\frac{\\partial f}{\\partial x^i}(\\Xk)$ is $\\fqalgebra$-measurable as the composition of the Borel measurable function $\\frac{\\partial f}{\\partial x^i}$ (from Lemma~\\ref{partialMeasurable}) and the random map $\\Xk:(\\Omega,\\fqalgebra)\\to\\left(\\rn, \\mathscr{B}(\\rn)\\right)$~\\cite[Theorem~13.3 and Theorem~13.1-(ii)]{billingsley1995ThirdEdition}. Recall that  sums and products of measurable functions are also measurable~\\cite[Theorem~13.3]{billingsley1995ThirdEdition}, \n%%consequence of Theorem~13.3\nand note that the random variable \n\\begin{equation}\\label{ukRandomMeasurable}\n\\ubar{u}_k:=\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}^2=\\sum_{j=1}^p\\left(\\sum_{i=1}^n\\left(\\QkRandom\\right)_{ij}\\frac{\\partial f}{\\partial x^i}(\\Xk) \\right)^2 \n\\end{equation}\nis consequently $\\fqalgebra$-measurable. Thus $\\Gk=\\sqrt{{\\ubar{u}_k}}-\\zeta\\Dk$ is also $\\fqalgebra$-measurable. The $\\fqalgebra$-measurability of $\\Gk$ ensures in particular that $\\Gamma_k=\\Gk^{-1}([0, \\infty))\\in\\fqalgebra$ since $[0, \\infty)$ is a Borel set of $\\R$, whence the {\\it simple} real functions $\\igk$ and $\\igkc$~\\cite[Equation~(13.3)]{billingsley1995ThirdEdition} are also $\\fqalgebra$-measurable. \n\\end{proof}\n\n\\begin{theorem}\\label{zerothOrderScal}\nLet all assumptions that were made in Lemmas~\\ref{LemmaA}-\\ref{LemmaD} hold with the same constants $C_1, C_2$, and $C_3:=C_3(\\zeta)$, for some fixed $\\zeta>0$ satisfying\n\\begin{equation}\\label{zetaChoice}\n\\zeta\\geq \\kappa_{eg}+\\max\\accolade{\\eta_2,\\kappa_h, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}}.\n\\end{equation}\nLet $\\nu\\in (0,1)$ be chosen according to \n\\begin{equation}\\label{nuChoiceScal}\n\\frac{\\nu}{1-\\nu} \\geq \\frac{8\\gamma^2}{\\min\\accolade{C_2, \\zeta C_1}}.  \n\\end{equation}\nAssume further that Assumption~\\ref{assumptionsOnEstimatesAndModels} holds with $\\beta_f, \\beta_m\\in (0,1)$ satisfying\\footnote{In~\\eqref{probChoices}, the result is intentionally presented with $(1-\\beta_f)+(1-\\beta_m)$ instead of $2-\\beta_f-\\beta_m$ in order to emphasize one of its differences compared with~\\cite{blanchet2016convergence,chen2018stochastic} in which a term similar to $(1-\\beta_f)(1-\\beta_m)$ was derived.}\n\\begin{equation}\\label{probChoices}\n\\frac{\\beta_f\\beta_m-\\frac{1}{2}}{(1-\\beta_f)+(1-\\beta_m)}\\geq \\frac{C_3}{C_1}\\quad \\mbox{ and }\\quad \\frac{\\beta_f}{1-\\beta_f}\\geq \\frac{2\\nu\\zeta\\gamma^2C_3}{(1-\\nu)(\\gamma^2-1)}+2\\gamma^2.\n\\end{equation}\nDefine $\\varrho=\\frac{1}{2}\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})>0$. Then, the random function $\\Phik:=\\nu (f(\\Xk)-f_{\\min})+(1-\\nu)\\Dk^2$ satisfies\n\\begin{equation}\\label{PhikIncrementBound}\n\\E{\\Phikun-\\Phik|\\fqalgebra}\\leq -\\varrho\\Dk^2 \\quad \\mbox{for all}\\ k\\in\\N,\n\\end{equation}\nwhich implies that\n\\begin{equation}\\label{seriesResult}\n\\sum_{k=0}^{\\infty}\\Dk^2<\\infty\\quad \\mbox{almost surely.}\n\\end{equation}\n\\end{theorem}\n\n\\begin{proof}\nThe proof is inspired by those of~\\cite[Theorem~3]{blanchet2016convergence} and~\\cite[Theorem~4.11]{chen2018stochastic}. \nThe overall goal is to prove~\\eqref{PhikIncrementBound}. Indeed noticing that $\\Phik>0$ and then taking expectations on both sides of the inequality in~\\eqref{PhikIncrementBound}, we obtain~\\eqref{seriesResult} (see, e.g.,~\\cite[Theorem~3]{dzahini2020expected} for details). Let $\\phi_k$ denote realizations of $\\Phik$, and recall that on all successful iterations, $\\xkun=\\xk+\\Qk\\sk$ and $\\dkun=\\min\\accolade{\\gamma\\dk,\\dmaxx}$,  implying\n\\begin{equation}\\label{un}\n\\phi_{k+1}-\\phi_k\\leq \\nu\\left(f(\\xkun)-f(\\xk)\\right)+(1-\\nu)(\\gamma^2-1)\\dk^2,\n\\end{equation}\nwhile on unsuccessful iterations, $\\xkun=\\xk$ and $\\dkun=\\gamma^{-1}\\dk$, in which case\n\\begin{equation}\\label{bOne}\n\\phi_{k+1}-\\phi_k\\leq(1-\\nu)\\left(\\frac{1}{\\gamma^2}-1\\right)\\dk^2=:b_1<0.\n\\end{equation}\n\nRecall the event $\\Gamma_k=\\accolade{\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\geq\\zeta\\Dk}$ of Lemma~\\ref{indicatorMeasurable} with $\\zeta$ satisfying ~\\eqref{zetaChoice}. The proof considers two cases: $\\igk=1$ and $\\igkc=1$. Inspired by~\\cite[proof of Theorem~4.11]{chen2018stochastic}, Case~1 aims to show that\n\\begin{equation}\\label{huit}\n%\\begin{split}\n\\E{\\igk\\left(\\Phikun-\\Phik\\right)|\\fqalgebra}\\leq-2\\igk(1-\\nu)(\\gamma^2-1)\\Dk^2\n\\leq -\\frac{1}{2}\\igk\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})\\Dk^2, \n%\\end{split}\n\\end{equation}\nwhere the last inequality follows from $1-\\frac{1}{\\gamma^2}<\\gamma^2-1$ and $\\beta_f\\leq 1$. On the other hand, inspired by~\\cite[proof of Theorem~3]{blanchet2016convergence}, Case~2 shows that\n\\begin{equation}\\label{last}\n\\E{\\igkc\\left(\\Phikun-\\Phik\\right)|\\fqalgebra}\\leq-\\frac{1}{2}\\igkc\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})\\Dk^2.\n\\end{equation}\nThus, combining~\\eqref{huit} and~\\eqref{last} leads to~\\eqref{PhikIncrementBound}.\n\n$ $\\\\\n\\textbf{Case~1: $\\igk=1$}\\\\\n$ $\\\\\nUnlike the proof of~\\cite[Theorem~4.11, Case~1]{chen2018stochastic} where four subcases were considered, only three are analyzed next.\n\n%\\begin{itemize}\n%\\item[(i)] \n\\noindent \\textit{(Subcase 1i)}\nGood model ($\\iik=1$) and good estimates ($\\ijk=1$). By~\\eqref{zetaChoice}, \n\\[\\normii{\\QkTranspose\\nabla f(\\xk)}\\geq \\max\\accolade{\\eta_2+\\kappa_{eg}, \\kappa_h+\\kappa_{eg}, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}}+\\kappa_{eg}}\\dk,\\]\nimplying in particular~\\eqref{deltaLemmaB2} and hence a decrease in $f$ according to~\\eqref{LemmaB2improvement}. Moreover, by $(\\kef,\\keg;\\Qk)$-full linearity, \n\\[\\normii{\\gkhat}\\geq \\normii{\\QkTranspose\\nabla f(\\xk)}-\\kappa_{eg}\\dk\\geq \\max\\accolade{\\eta_2,\\kappa_h, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}}\\dk, \\]\nthus implying~\\eqref{deltaLemmaC}; and since $\\ef\\leq \\kappa_{ef}$ per assumptions of Lemma~\\ref{LemmaC}, the iteration is successful. Consequently,~\\eqref{un} together with~\\eqref{LemmaB2improvement} yields\n\\begin{equation}\\label{bTwo}\n\\begin{split}\n\t\\phi_{k+1}-\\phi_k &\\leq -\\nu C_1\\normii{\\QkTranspose\\nabla f(\\xk)}\\dk+(1-\\nu)(\\gamma^2-1)\\dk^2=:b_2\\\\\n\t&\\leq \\left(-\\nu C_1\\zeta+(1-\\nu)(\\gamma^2-1)\\right)\\dk^2<0,\n\\end{split}\n\\end{equation}\nwhere the last inequality follows from~\\eqref{nuChoiceScal}. Denoting by ${\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)$ the random variable with realizations $b_2<0$, with ${\\ubar{u}}_k$ defined in~\\eqref{ukRandomMeasurable}, we have from~\\eqref{bTwo} that\n\\begin{equation}\\label{bTwoRandom}\n\\igk\\iik\\ijk\\left(\\Phikun-\\Phik\\right)\\leq \\igk\\iik\\ijk{\\ubar{b}}_2({\\ubar{u}}_k,\\Dk).\n\\end{equation}\n%\\item[(ii)] \n\\noindent \\textit{(Subcase 1ii)}\nBad model ($\\iikc=1$) and good estimates ($\\ijk=1$). In this case, regardless of the iteration type (i.e., successful or unsuccessful), the change in $\\phi_k$ can always be bounded by using~\\eqref{un} and~\\eqref{LemmaEincrease} as follows:\n\\begin{equation}\\label{bThree}\n\\phi_{k+1}-\\phi_k\\leq\\nu C_3\\norme{\\QkTranspose\\nabla f(\\xk)}\\dk+(1-\\nu)(\\gamma^2-1)\\dk^2=:b_3.\n\\end{equation}\nDenoting by ${\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)$ the random variable with realizations $b_3>0$, from~\\eqref{bThree} we have that\n\\begin{equation}\\label{bThreeRandom2}\n\\igk\\iikc\\ijk\\left(\\Phikun-\\Phik\\right)\\leq \\igk\\iikc\\ijk{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)\\leq \\igk\\iikc{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk).\n\\end{equation}\n%\\item[(iii)] \n\\noindent \\textit{(Subcase 1iii)}\nBad estimates ($\\ijkc=1$). From \\textit{Subcase 1ii}, it always holds that \n\\begin{equation}\\label{bThreeRandom3}\n\\igk\\ijkc\\left(\\Phikun-\\Phik\\right)\\leq \\igk\\ijkc{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk).\n\\end{equation}\n%\\end{itemize}\n\nWith the subcases thus complete, since the random variables $\\igk, {\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)$, and ${\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)$ are $\\fqalgebra$-measurable thanks to the proof of Lemma~\\ref{indicatorMeasurable}, combining~\\eqref{bTwoRandom},~\\eqref{bThreeRandom2}, and~\\eqref{bThreeRandom3} and taking expectations with respect to $\\fqalgebra$, we have\n\\begin{equation}\\label{bTwoThree}\n\\begin{split}\n\\E{\\igk\\left(\\Phikun-\\Phik\\right)|\\fqalgebra}&\\leq \\igk{\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)\\E{\\iik\\ijk|\\fqalgebra}\\\\\n&+\\igk{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)\\left(\\E{\\iikc|\\fqalgebra}+\\E{\\ijkc|\\fqalgebra}\\right)\\\\\n&\\leq \\igk\\left(\\beta_f\\beta_m{\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)+{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)(1-\\beta_m+1-\\beta_f)\\right)=:\\igk {\\ubar{b}}_{2,3}({\\ubar{u}}_k,\\Dk),\n\\end{split}\n\\end{equation}\nwhere the last inequality follows from~\\eqref{ikQProb} and~\\eqref{prodBetafBetam} and the fact that ${\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)<0$ while ${\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)>0$.  Note that \n\\begin{eqnarray*}\n{\\ubar{b}}_{2,3}({\\ubar{u}}_k,\\Dk)&=&\\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\left[-\\beta_f\\beta_m C_1+(2-\\beta_f-\\beta_m)C_3\\right]\\\\\n& &+(1-\\nu)(\\gamma^2-1)(2-\\beta_f-\\beta_m+\\beta_f\\beta_m)\\Dk^2\\\\\n&\\leq& \\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\left[-\\beta_f\\beta_m C_1+(2-\\beta_f-\\beta_m)C_3\\right] + 2(1-\\nu)(\\gamma^2-1)\\Dk^2,\n\\end{eqnarray*}\nwhere the last inequality follows from $2-\\beta_f-\\beta_m+\\beta_f\\beta_m=1+(1-\\beta_f)(1-\\beta_m)\\leq 2$.\n\nFor $\\beta_f$ and $\\beta_m$ chosen according to the first condition in~\\eqref{probChoices}, and for $\\nu$ satisfying~\\eqref{nuChoiceScal}, the following holds:\n\\[\\beta_f\\beta_m C_1 - (2-\\beta_f-\\beta_m)C_3\\geq \\frac{1}{2}C_1\\geq 2\\frac{2(1-\\nu)(\\gamma^2-1)}{\\nu\\zeta}.\\]\nHence, since $\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\geq \\zeta\\Dk$, then \n\\begin{eqnarray*}\n{\\ubar{b}}_{2,3}({\\ubar{u}}_k,\\Dk)&\\leq& -\\frac{1}{2}\\left[\\beta_f\\beta_m C_1-(2-\\beta_f-\\beta_m)C_3\\right]\\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\\\\n&\\leq& -\\frac{1}{4}C_1\\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\leq -2(1-\\nu)(\\gamma^2-1)\\Dk^2,\n\\end{eqnarray*}\nwhich, together with~\\eqref{bTwoThree}, leads to~\\eqref{huit}.\n\n$ $\\\\\n\\textbf{Case~2: $\\igkc=1$}\n$ $\\\\ \nConsider the event $G_k:=\\accolade{\\normii{\\Gkhat}\\geq \\eta_2\\Dk}$, and note that since $\\Mhatk:={\\ubar{f}_k}+\\Gkhat^{\\top}\\sRandom +\\frac{1}{2}\\sRandom^{\\top}\\HkhatRandom\\sRandom$ is $\\fmqalgebra$-measurable by construction, then in particular $\\normii{\\Gkhat}$ and hence $\\igkhat, \\igkhatc$ are also $\\fmqalgebra$-measurable. Observe that if $\\igkhatc=1$, then the iteration is unsuccessful since $\\normii{\\gkhat}<\\eta_2\\dk$, leading to~\\eqref{bOne}. Hence, \n\\begin{equation}\\label{GkbarCase2}\n\\begin{split}\n\\E{\\igkc\\igkhatc(\\Phikun-\\Phik)|\\fmqalgebra}&\\leq -(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\E{\\igkc\\igkhatc\\Dk^2|\\fmqalgebra}\\\\\n&\\leq -\\frac{1}{2}\\igkc\\igkhatc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{split}\n\\end{equation}\nwhere the last inequality is due to the $\\fmqalgebra$-measurability of $\\igkc$ and $\\Dk$, given that $\\falgebra\\subset\\fqalgebra\\subset\\fmqalgebra$. \n\nInspired by~\\cite[Theorem~3]{blanchet2016convergence}, which improved on the proof of~\\cite[Theorem~4.11, Case~2]{chen2018stochastic} in which four subcases were considered, only two are analyzed next. The overall goal is to prove that\n\\begin{equation}\\label{GkCase2}\n\\E{\\igkc\\igkhat(\\Phikun-\\Phik)|\\fmqalgebra}\\leq -\\frac{1}{2}\\igkc\\igkhat\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{equation}\nwhich, combined with~\\eqref{GkbarCase2}, leads to \n\\begin{equation*}%\\label{GammaBarFmq}\n\\E{\\igkc(\\Phikun-\\Phik)|\\fmqalgebra}\\leq -\\frac{1}{2}\\igkc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2\n\\end{equation*}\nand consequently\n\\begin{equation*}%\\label{GammaBarFq}\n\\begin{split}\n\\E{\\igkc(\\Phikun-\\Phik)|\\fqalgebra}&=\\E{\\E{\\igkc(\\Phikun-\\Phik)|\\fmqalgebra}|\\fqalgebra}\\\\\n&\\leq \\E{-\\frac{1}{2}\\igkc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2 \\, |\\fqalgebra}=-\\frac{1}{2}\\igkc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{split}\n\\end{equation*}\n showing~\\eqref{last}, where the last equality follows from the $\\fqalgebra$-measurability of $\\igkc$ and $\\Dk$.  What remains to be shown is~\\eqref{GkCase2} as achieved next, assuming that $\\igkhat=1$.\n\n\\noindent \\textit{(Subcase 2i)}\n%\\begin{itemize}\n%\\item[(i)] \nGood estimates ($\\ijk=1$) and $\\igkhat=1$. While $f$ is decreased on successful iterations because  of good estimates thanks to Lemma~\\ref{LemmaD} and in this case~\\eqref{LemmaDimprovement} holds,  $\\dk$ is reduced on unsuccessful iterations, leading to~\\eqref{bOne}. However, combining~\\eqref{LemmaDimprovement} and~\\eqref{un} yields\n$\\phi_{k+1}-\\phi_k\\leq \\left[-\\nu C_2+(1-\\nu)(\\gamma^2-1)\\right]\\dk^2\\leq b_1,$\nwhere the last inequality is due to~\\eqref{nuChoiceScal}, which shows that~\\eqref{bOne} holds in any case. Consequently,\n\\begin{equation}\n\\label{GammakBarGkJk}\n\\igkc\\igkhat\\ijk(\\Phikun-\\Phik) \\leq -\\igkc\\igkhat\\ijk(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2.\n\\end{equation}\n\n%\\item[(ii)] \n\\noindent \\textit{(Subcase 2ii)}\nBad estimates ($\\ijkc=1$) and $\\igkhat=1$. In this case, a successful step can lead to an increase in $f$\naccording to~\\eqref{LemmaEincrease}, which combined with~\\eqref{un} yield~\\eqref{bThree} and hence\n\\begin{equation}\\label{GammakBarGkJkBar}\n\\igkc\\igkhat\\ijkc(\\Phikun-\\Phik) \\leq -\\igkc\\igkhat\\ijkc\\left(\\nu C_3\\zeta+(1-\\nu)(\\gamma^2-1)\\right)\\Dk^2.\n\\end{equation}\n%\\end{itemize}\n\nWith the subcases complete, recall that $\\E{\\ijk|\\fmqalgebra}\\geq\\beta_f$. Then  combining~\\eqref{GammakBarGkJk} and~\\eqref{GammakBarGkJkBar} and taking expectations with respect to~$\\fmqalgebra$ lead to\n\\begin{eqnarray*}%\\label{GkCase2Proved}\n\\E{\\igkc\\igkhat(\\Phikun-\\Phik)|\\fmqalgebra}&\\leq& \\igkc\\igkhat\\left(-\\beta_f(1-\\nu)(1-1/\\gamma^2)\\right.\\\\ \n& &\\left. +\\ (1-\\beta_f)\\left(\\nu C_3\\zeta+(1-\\nu)(\\gamma^2-1)\\right) \\right)\\Dk^2\\\\\n&\\leq& -\\frac{1}{2}\\igkc\\igkhat\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{eqnarray*}\nthus proving~\\eqref{GkCase2}, where the last inequality follows from the second condition in~\\eqref{probChoices}, and the proof is complete.\n\\end{proof}\n\n\\begin{theorem}\\label{liminfTheorem}\nLet all the assumptions made in Theorem~\\ref{zerothOrderScal} hold. Assume further that Assumption~\\ref{wellAlignedAssumption} holds with $0\\leq\\beta_Q<1-\\frac{1}{2\\beta_f\\beta_m}$. Then, the sequence $\\accolade{\\Xk}_{k\\in\\N}$ of random iterates generated by Algorithm~\\ref{algoStoScalTR} satisfies\n\\[\\underset{k\\to\\infty}{\\liminf}\\normii{\\nabla f(\\Xk)}=0\\quad\\mbox{ almost surely.}\\]\n\\end{theorem}\n\n\\begin{proof}\nThe result is proved by contradiction conditioned on the almost sure event $E_0:=\\accolade{\\Dk\\to 0}$ (thanks to Theorem~\\ref{zerothOrderScal}), inspired by the proof of~\\cite[Theorem~4.16]{chen2018stochastic} and also using ideas from~\\cite{audet2019stomads,dzahini2020constrained}. Assume that with nonzero probability there exists a random variable ${\\ubar{\\varepsilon}}'>0$ such that \n$\\normii{\\nabla f(\\Xk)}\\geq {\\ubar{\\varepsilon}}' \\mbox{ for all } k\\in\\N.$\n\nLet $\\accolade{\\xk}, \\accolade{\\dk}$, and ${{\\varepsilon}}'$ be realizations of $\\accolade{\\Xk}, \\accolade{\\Dk}$, and ${\\ubar{\\varepsilon}}'$, respectively, for which $\\normii{\\nabla f(\\xk)}\\geq {{\\varepsilon}}'$ for all $k$. Since $\\dk\\to 0$, there exists $k_0\\in\\N$ such that\n\\begin{equation}\\label{bChoice}\n\\dk<b:=\\alpha_Q\\cdot\\min\\accolade{\\frac{{{\\varepsilon}}'}{2(\\kappa_h+\\kappa_{eg})}, \\frac{{{\\varepsilon}}'}{2(\\eta_2+\\kappa_{eg})}, \\frac{{{\\varepsilon}}'}{\\frac{16\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}+2\\kappa_{eg}}, \\frac{\\dmaxx}{\\gamma}}\n\\end{equation}\nfor all $k\\geq k_0$.\nLet $\\ubar{r}_k$ be the random variable with realizations $r_k=\\log_{\\gamma}\\left(\\frac{\\dk}{b}\\right)$. Then $r_k<0$ for all $k\\geq k_0$. The main idea is to show that such realizations occur only with probability zero, leading to a contradiction. \n\nTo prove that $\\accolade{\\ubar{r}_k}$ is a submartingale, recall the events $I^Q_k$ and $J^Q_k$ satisfying~\\eqref{otherProbIneqs} and $\\mathcal{A}_k$ satisfying~\\eqref{Akineq}.\n%$\\prob{\\mathcal{A}_k|\\falgebra}\\geq 1-\\beta_Q$, \n%$\\prob{I^Q_k|\\falgebra}\\geq\\beta_m$ and  $\\prob{J^Q_k|\\falgebra}\\geq\\beta_f$, \n%where $\\mathcal{F}_{k-1}^Q :=\\sigma\\left(\\mathcal{A}_0,\\mathcal{A}_1,\\dots,\\mathcal{A}_{k-1}\\right)\\cap \\sigma\\left(I^Q_0,I^Q_1,\\dots,I^Q_{k-1}\\right)\\cap \\sigma\\left(J^Q_0,J^Q_1,\\dots,J^Q_{k-1}\\right)$. \nConsider some iteration $k\\geq k_0$ for which $\\mathcal{A}_k$, $I^Q_k$, and $J^Q_k$ all occur, which happens with probability at least $\\beta_f\\beta_m(1-\\beta_Q)$ conditioned on $\\falgebra$, as was shown in~\\eqref{prodBetaIneq}. It follows from~\\eqref{bChoice} that \n\\begin{equation}\\label{ineq4}\n\\normii{\\nabla f(\\xk)}\\geq \\frac{1}{\\alpha_Q}\\max\\accolade{\\kappa_h+\\kappa_{eg}, \\eta_2+\\kappa_{eg}, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}+\\kappa_{eg}}\\dk,\n\\end{equation}\nwhich, together with the $(\\kef,\\keg;\\Qk)$-full linearity and $\\alpha_Q$-well alignment, yield\n\\begin{equation}\\label{ineq5}\n%\\begin{split}\n\\normii{\\gkhat}\\geq\\normii{\\QkTranspose\\nabla f(\\xk)}-\\kappa_{eg}\\dk\\geq \\alpha_Q \\normii{\\nabla f(\\xk)}-\\kappa_{eg}\\dk\n\\geq \\max\\accolade{\\kappa_h, \\eta_2, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}}\\dk.\n%\\end{split}\n\\end{equation}\nThus, the $k$th iteration is successful according to Lemma~\\ref{LemmaC}. Hence, $\\dkun=\\gamma\\dk$, and hence $r_{k+1}=r_k+1$. For all other outcomes of $\\mathcal{A}_k$, $I^Q_k$, and $J^Q_k$, which occur with a total probability of at most $1-\\beta_f\\beta_m(1-\\beta_Q)$, it always holds that $\\dkun\\geq\\gamma^{-1}\\dk$, which implies that $r_{k+1}\\geq r_k-1$. Hence, \n\\begin{eqnarray*}\n\\E{\\mathds{1}_{\\mathcal{A}_k\\cap I^Q_k\\cap J^Q_k} \\left(\\ubar{r}_{k+1}-\\ubar{r}_k\\right)|\\falgebra}&\\geq& \\beta_f\\beta_m(1-\\beta_Q)\\quad \\quad\\mbox{and}\\\\\n\\E{\\mathds{1}_{\\overline{\\mathcal{A}_k\\cap I^Q_k\\cap J^Q_k}} \\left(\\ubar{r}_{k+1}-\\ubar{r}_k\\right)|\\falgebra}&\\geq& \\beta_f\\beta_m(1-\\beta_Q)-1, \n\\end{eqnarray*}\nwhich shows that $\\accolade{\\ubar{r}_k}$ is a submartingale if $0\\leq\\beta_Q<1-\\frac{1}{2\\beta_f\\beta_m}$.\n\nUsing the random walk defined in~\\eqref{randomWalk}, one  can easily  show (following, e.g.,~\\cite[Theorem~4]{audet2019stomads}, \\cite[Theorem~4.16]{chen2018stochastic} and~\\cite[Theorem~3.6 and Lemmas~4.7 and~4.12]{dzahini2020constrained}) that $\\accolade{\\ubar{w}_k}$ is also a submartingale with bounded increments and, as such, cannot converge to a finite value, and therefore $\\prob{\\underset{k\\to\\infty}{\\limsup}\\ \\ubar{w}_k=\\infty}=1$ (thanks to~\\cite[Theorem~4.4]{chen2018stochastic}). However, since $r_k-r_{k_0}\\geq w_k-w_{k_0}$ by construction,  the sequence of realizations~$r_k$ such that $r_k<0\\ \\forall k\\ \\geq k_0$ occurs with probability zero, leading to a contradiction, which achieves the proof.\n\\end{proof}\n\n\\begin{lemma}\\label{smallLemma}\nLet all the assumptions made in Theorem~\\ref{liminfTheorem} hold. Then~\\eqref{dynamicsRenewal} is satisfied for\\\\ $\\Wtildek=2\\left(\\iak\\iik\\ijk-\\frac{1}{2} \\right)$, with $\\lambda=\\ln(\\gamma)$ and some fixed $q\\in (\\tilde{\\beta}, 1)$ with $\\tilde{\\beta}=\\beta_f\\beta_m(1-\\beta_Q)$.\n\\end{lemma}\n\n\\begin{proof}\nThe result is proved by suitably adapting the proof of~\\cite[Lemma~7]{blanchet2016convergence} as was done in~\\cite{dzahini2020expected}. First, we notice that~\\eqref{dynamicsRenewal} trivially holds when $\\itksup=0$. Next, we show that when $\\itksup=1$, then \n\\[\\Dkun\\geq \\min\\accolade{\\depsilon, \\min\\accolade{\\dmaxx,\\gamma\\Dk}\\iak\\iik\\ijk+\\gamma^{-1}\\Dk\\left(1-\\iak\\iik\\ijk\\right)}. \\]\nRecall that $\\dk=\\gamma^{i_k}\\depsilon$ for some integer $i_k$. Therefore, if $\\dk>\\depsilon$, then $\\dk\\geq \\gamma\\depsilon$, which implies that $\\dkun\\geq\\gamma^{-1}\\dk\\geq\\depsilon$. Now assume that $\\dk\\leq \\depsilon$. Since $\\tauEpsilon>k$, we have $\\normii{\\nabla f(\\xk)}>\\epsilon=\\xi\\depsilon\\geq\\xi\\dk$. Thus, it follows from the definition of~$\\xi$ that~\\eqref{ineq4} holds. If $\\iak=1$, $\\iik=1$, and $\\ijk=1$, then~\\eqref{ineq5} holds, and consequently the $k$th iteration is successful, as was  explained in the proof of Theorem~\\ref{liminfTheorem}. Hence, $\\xkun=\\xk+\\Qk\\sk$ and $\\dkun=\\min\\accolade{\\dmaxx,\\gamma\\dk}$. If $\\iak\\iik\\ijk=0$, then it always holds that $\\dkun\\geq \\gamma^{-1}\\dk$. The proof is completed by  observing that $\\prob{\\iak\\iik\\ijk=1}=q$ for some fixed $q\\geq \\tilde{\\beta}$.\n\\end{proof}\n\n\\begin{proof}\nRecall the choice of $q$ in Lemma~\\ref{smallLemma}, and pick some $\\tilde{\\xi}\\geq\\hat{\\xi}$ such that $\\depsilon=\\frac{\\epsilon}{\\tilde{\\xi}}=\\gamma^{i}\\delta_0$ for some integer $i\\leq 0$ (without loss of generality), as discussed above. The proof follows by employing Theorem~\\ref{theorComplexity1} with $h(t)=t^2$, which yields\n\\[\\E{\\tauEpsilon}\\leq \\frac{q}{2q-1}\\cdot\\frac{\\PhiZero\\tilde{\\xi}^2}{\\varrho \\epsilon^2}+1\\leq \\frac{\\tilde{\\beta}}{2\\tilde{\\beta}-1}\\cdot\\frac{\\PhiZero\\tilde{\\xi}^2}{\\varrho \\epsilon^2}+1.\\]\n\\end{proof}\n\n\\begin{proof}\nThe proof is similar to that of~\\cite[Lemma~4.5]{chen2018stochastic} and is not detailed here. It follows from~\\eqref{CauchyDecrease} together with the inequalities $\\kappa_h\\geq \\max\\accolade{\\norme{\\Hkhat}, 1}$ and $\\normii{\\gkhat}\\geq\\kappa_h\\dk$ that  $\\mhatk(\\bm{0})-\\mhatk(\\sk)\\geq\\frac{\\kappa_{fcd}}{2}\\normii{\\gkhat}\\dk.$ Since the model is $(\\kef,\\keg;\\Qk)$-fully linear, then \n$f(\\xk+\\Qk\\sk)-f(\\xk)\\leq 2\\kappa_{ef}\\dk^2+\\mhatk(\\sk)-\\mhatk(\\bm{0})\\leq -\\frac{\\kappa_{fcd}}{4}\\normii{\\gkhat}\\dk,$\nwhere the last inequality follows from the fact that $\\dk\\leq \\frac{\\kappa_{fcd}}{8\\kappa_{ef}}\\normii{\\gkhat}$.\n\\end{proof}\n\n\\begin{proof}\nThe proof is inspired by (and similar to) that of~\\cite[Lemma~4.6]{chen2018stochastic}. By $(\\kef,\\keg;\\Qk)$-full linearity and~\\eqref{deltaLemmaB2}, it holds\n\\begin{equation}\\label{ineq1}\n\\normii{\\gkhat}\\geq \\normii{\\QkTranspose\\nabla f(\\xk)}-\\kappa_{eg}\\dk\\geq \\max\\accolade{\\kappa_h, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}}}\\dk, \n\\end{equation}\nwhich shows that condition~\\eqref{LemmaAimprovement} is satisfied. Consequently $f$ decreases as in~\\eqref{LemmaAimprovement}. The first inequality in~\\eqref{ineq1}, together with~\\eqref{deltaLemmaB2}, yields\n$\\normii{\\gkhat}\\geq \\frac{4C_1}{\\kappa_{fcd}}\\normii{\\QkTranspose\\nabla f(\\xk)},$\nwhich, combined with~\\eqref{LemmaAimprovement}, implies~\\eqref{LemmaB2improvement}, and the proof is complete.\n\\end{proof}\n\n\\begin{proof}\nAccording to the Fundamental Theorem of Calculus in $\\rn$ (see, e.g.,~\\cite[Lemma~9.3]{AuHa2017}),\\\\\n$\\abs{f(\\xk)-f(\\xk+\\Qk\\sk)-\\nabla f(\\xk+\\Qk\\sk)^{\\top}(-\\Qk\\sk)}\\leq\\frac{1}{2}L_g\\Qmax^2\\dk^2,$  with the inequality also following from $\\normii{\\Qk\\sk}\\leq \\Qmax\\dk$, which implies that\n\\begin{equation}\\label{ineq2}\nf(\\xk+\\Qk\\sk)-f(\\xk)\\leq \\sk^{\\top}\\left[\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)\\right] + \\frac{1}{2}L_g\\Qmax^2\\dk^2. \n\\end{equation}\nPer Lipschitz continuity of $\\nabla f$, \n$\\normii{\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)-\\QkTranspose \\nabla f(\\xk)} \\leq L_g\\Qmax^2\\dk,$\nwhich implies that\n\\begin{equation}\\label{ineq3}\n\\normii{\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)}\\leq L_g\\Qmax^2\\dk+\\normii{\\QkTranspose \\nabla f(\\xk)}.\n\\end{equation}\nThus, using~\\eqref{ineq2} and~\\eqref{ineq3}, together with the inequality $\\dk\\leq \\zeta^{-1}\\normii{\\QkTranspose \\nabla f(\\xk)}$, yields\n\\begin{eqnarray*}\nf(\\xk+\\Qk\\sk)&-&f(\\xk)\\leq \\normii{\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)}\\dk+\\frac{L_g}{2\\zeta}\\Qmax^2\\normii{\\QkTranspose \\nabla f(\\xk)}\\dk\\\\\n&\\leq& \\left(\\frac{L_g}{\\zeta}\\Qmax^2\\normii{\\QkTranspose \\nabla f(\\xk)} +\\normii{\\QkTranspose \\nabla f(\\xk)}\\right)\\dk\\\\\n&+&\\frac{L_g}{2\\zeta}\\Qmax^2\\normii{\\QkTranspose \\nabla f(\\xk)}\\dk= \\left(1+\\frac{3L_g}{2\\zeta}\\Qmax^2\\right)\\normii{\\QkTranspose \\nabla f(\\xk)}\\dk,\n\\end{eqnarray*}\nwhich completes the proof.\n\\end{proof}","units":[{"raw_tex":"\\begin{lemma}\\label{LemmaA}\nSuppose that the function $\\mhatk$ is a $(\\kef,\\keg;\\Qk)$-fully linear model for~$f$ in $\\B{\\xk}{\\dk;\\Qk}$. Then the trial step $\\sk$ leads to an improvement in $f$ such that\n\\begin{equation}\\label{LemmaAimprovement}\nf(\\xk+\\Qk\\sk)-f(\\xk)\\leq -\\frac{\\kappa_{fcd}}{4}\\normii{\\gkhat}\\dk,\n%\\end{equation}\n%\\begin{equation}\\label{deltaLemmaA}\n\\quad\n\\mbox{if }\n\\dk\\leq \\min\\accolade{\\frac{1}{\\kappa_h},\\frac{\\kappa_{fcd}}{8\\kappa_{ef}}}\\normii{\\gkhat}.\n\\end{equation}\n\\end{lemma}","start":91779,"end":92280},{"raw_tex":"\\begin{lemma}\\label{LemmaB2}\nLet Assumption~\\ref{boundedHessian} hold, and suppose the model $\\mhatk$ is $(\\kef,\\keg;\\Qk)$-fully linear. Define $C_1=\\frac{\\kappa_{fcd}}{4} \\max\\accolade{\\frac{\\kappa_h}{\\kappa_h+\\kappa_{eg}}, \\frac{8\\kappa_{ef}}{8\\kappa_{ef}+\\kappa_{fcd}\\kappa_{eg}}}$. Then the trial step $\\sk$ leads to an improvement in $f$ such that \n\n\\begin{equation}\\label{LemmaB2improvement}\nf(\\xk+\\Qk\\sk)-f(\\xk)\\leq -C_1\\normii{\\QkTranspose\\nabla f(\\xk)}\\dk,\n\\end{equation}\n\\begin{equation}\\label{deltaLemmaB2}\n\\mbox{if}\\quad\n\\dk\\leq \\min\\accolade{\\frac{1}{\\kappa_h+\\kappa_{eg}}, \\frac{1}{\\frac{8\\kappa_{ef}}{\\kappa_{fcd}}+\\kappa_{eg}}}\\normii{\\QkTranspose\\nabla f(\\xk)}. \n\\end{equation}\n\\end{lemma}","start":92546,"end":93252},{"raw_tex":"\\begin{lemma}\\label{LemmaE}\nUnder Assumptions~\\ref{QmaxAssumption} and~\\ref{fGradfAssumption}, assume that Algorithm~\\ref{algoStoScalTR} erroneously accepts a step $\\sk$ leading to an increase in the objective function when $\\norme{\\QkTranspose\\nabla f(\\xk)}\\geq \\zeta\\dk$ for some constant $\\zeta>0$, and let $C_3(\\zeta)=1+\\frac{3L_g}{2\\zeta}\\Qmax^2$. Then the increase in $f$ is such that\n\\begin{equation}\\label{LemmaEincrease}\nf(\\xk+\\Qk\\sk)-f(\\xk)\\leq C_3(\\zeta)\\norme{\\QkTranspose\\nabla f(\\xk)}\\dk.\n\\end{equation}\n\\end{lemma}","start":95221,"end":95750},{"raw_tex":"\\begin{lemma}\\label{partialMeasurable}\nAny partial derivative \n$\\frac{\\partial f}{\\partial x^i}: \\left(\\rn, \\mathscr{B}(\\rn)\\right)\\to \\left(\\R, \\mathscr{B}(\\R)\\right),\\ i\\in\\intbracket{1,n} $,\nof a differentiable function $f$ is a Borel measurable function.\n\\end{lemma}","start":96299,"end":96569},{"raw_tex":"\\begin{proof}\nLet $(\\bm{e_1},\\bm{e_2},\\dots,\\bm{e_n})$ be the canonical basis of $\\rn$. For any $i\\in\\intbracket{1,n} $ and $\\bm{x}=(x^1,x^2,\\dots,x^n)$, define, for all $j\\in\\N$,\n$D_{ij}(\\bm{x})=\\frac{f(\\bm{x}+h_j\\bm{e_i})-f(\\bm{x})}{h_j}$,\nwhere $\\accolade{h_j}_{j\\in\\N}$ is a sequence of positive real numbers converging to~$0$; $f$ is differentiable and hence continuous, and thus is Borel measurable~\\cite[Theorem~13.2]{billingsley1995ThirdEdition}. Therefore, since for all $i\\in\\intbracket{1,n}$, $\\underset{j\\to\\infty}{\\lim}D_{ij}(\\bm{x})=\\frac{\\partial f}{\\partial x^i}(\\bm{x}),$\nthen $\\frac{\\partial f}{\\partial x^i}$ is also Borel measurable~\\cite[Theorem~13.3 and Theorem~13.4-(ii)]{billingsley1995ThirdEdition}.\n\\end{proof}","start":96571,"end":97307},{"raw_tex":"\\begin{lemma}\\label{indicatorMeasurable}\nLet $\\zeta\\geq 0$ be a constant, and consider the event $\\Gamma_k:=\\accolade{\\Gk\\geq 0}$, where $\\Gk:=\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}-\\zeta\\Dk$. Then $\\Gk$, $\\igk$, and $\\igkc$ are $\\fqalgebra$-measurable.\n\\end{lemma}","start":97310,"end":97577},{"raw_tex":"\\begin{proof}\nBy construction, all the entries $\\left(\\QkRandom\\right)_{ij},\\ (i,j)\\in\\intbracket{1,n}\\times\\intbracket{1,p}$ of $\\QkRandom$ are $\\fqalgebra$-measurable, while $\\Dk$ and the components $\\Xk^i, \\ i\\in\\intbracket{1, n}$ of $\\Xk$ are $\\falgebra$-measurable and hence $\\fqalgebra$-measurable since $\\falgebra\\subseteq\\fqalgebra$. Thus, for any $i\\in\\intbracket{1, n}$, $\\frac{\\partial f}{\\partial x^i}(\\Xk)$ is $\\fqalgebra$-measurable as the composition of the Borel measurable function $\\frac{\\partial f}{\\partial x^i}$ (from Lemma~\\ref{partialMeasurable}) and the random map $\\Xk:(\\Omega,\\fqalgebra)\\to\\left(\\rn, \\mathscr{B}(\\rn)\\right)$~\\cite[Theorem~13.3 and Theorem~13.1-(ii)]{billingsley1995ThirdEdition}. Recall that  sums and products of measurable functions are also measurable~\\cite[Theorem~13.3]{billingsley1995ThirdEdition}, \n%%consequence of Theorem~13.3\nand note that the random variable \n\\begin{equation}\\label{ukRandomMeasurable}\n\\ubar{u}_k:=\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}^2=\\sum_{j=1}^p\\left(\\sum_{i=1}^n\\left(\\QkRandom\\right)_{ij}\\frac{\\partial f}{\\partial x^i}(\\Xk) \\right)^2 \n\\end{equation}\nis consequently $\\fqalgebra$-measurable. Thus $\\Gk=\\sqrt{{\\ubar{u}_k}}-\\zeta\\Dk$ is also $\\fqalgebra$-measurable. The $\\fqalgebra$-measurability of $\\Gk$ ensures in particular that $\\Gamma_k=\\Gk^{-1}([0, \\infty))\\in\\fqalgebra$ since $[0, \\infty)$ is a Borel set of $\\R$, whence the {\\it simple} real functions $\\igk$ and $\\igkc$~\\cite[Equation~(13.3)]{billingsley1995ThirdEdition} are also $\\fqalgebra$-measurable. \n\\end{proof}","start":97579,"end":99139},{"raw_tex":"\\begin{theorem}\\label{zerothOrderScal}\nLet all assumptions that were made in Lemmas~\\ref{LemmaA}-\\ref{LemmaD} hold with the same constants $C_1, C_2$, and $C_3:=C_3(\\zeta)$, for some fixed $\\zeta>0$ satisfying\n\\begin{equation}\\label{zetaChoice}\n\\zeta\\geq \\kappa_{eg}+\\max\\accolade{\\eta_2,\\kappa_h, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}}.\n\\end{equation}\nLet $\\nu\\in (0,1)$ be chosen according to \n\\begin{equation}\\label{nuChoiceScal}\n\\frac{\\nu}{1-\\nu} \\geq \\frac{8\\gamma^2}{\\min\\accolade{C_2, \\zeta C_1}}.  \n\\end{equation}\nAssume further that Assumption~\\ref{assumptionsOnEstimatesAndModels} holds with $\\beta_f, \\beta_m\\in (0,1)$ satisfying\\footnote{In~\\eqref{probChoices}, the result is intentionally presented with $(1-\\beta_f)+(1-\\beta_m)$ instead of $2-\\beta_f-\\beta_m$ in order to emphasize one of its differences compared with~\\cite{blanchet2016convergence,chen2018stochastic} in which a term similar to $(1-\\beta_f)(1-\\beta_m)$ was derived.}\n\\begin{equation}\\label{probChoices}\n\\frac{\\beta_f\\beta_m-\\frac{1}{2}}{(1-\\beta_f)+(1-\\beta_m)}\\geq \\frac{C_3}{C_1}\\quad \\mbox{ and }\\quad \\frac{\\beta_f}{1-\\beta_f}\\geq \\frac{2\\nu\\zeta\\gamma^2C_3}{(1-\\nu)(\\gamma^2-1)}+2\\gamma^2.\n\\end{equation}\nDefine $\\varrho=\\frac{1}{2}\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})>0$. Then, the random function $\\Phik:=\\nu (f(\\Xk)-f_{\\min})+(1-\\nu)\\Dk^2$ satisfies\n\\begin{equation}\\label{PhikIncrementBound}\n\\E{\\Phikun-\\Phik|\\fqalgebra}\\leq -\\varrho\\Dk^2 \\quad \\mbox{for all}\\ k\\in\\N,\n\\end{equation}\nwhich implies that\n\\begin{equation}\\label{seriesResult}\n\\sum_{k=0}^{\\infty}\\Dk^2<\\infty\\quad \\mbox{almost surely.}\n\\end{equation}\n\\end{theorem}","start":101177,"end":102801},{"raw_tex":"\\begin{proof}\nThe proof is inspired by those of~\\cite[Theorem~3]{blanchet2016convergence} and~\\cite[Theorem~4.11]{chen2018stochastic}. \nThe overall goal is to prove~\\eqref{PhikIncrementBound}. Indeed noticing that $\\Phik>0$ and then taking expectations on both sides of the inequality in~\\eqref{PhikIncrementBound}, we obtain~\\eqref{seriesResult} (see, e.g.,~\\cite[Theorem~3]{dzahini2020expected} for details). Let $\\phi_k$ denote realizations of $\\Phik$, and recall that on all successful iterations, $\\xkun=\\xk+\\Qk\\sk$ and $\\dkun=\\min\\accolade{\\gamma\\dk,\\dmaxx}$,  implying\n\\begin{equation}\\label{un}\n\\phi_{k+1}-\\phi_k\\leq \\nu\\left(f(\\xkun)-f(\\xk)\\right)+(1-\\nu)(\\gamma^2-1)\\dk^2,\n\\end{equation}\nwhile on unsuccessful iterations, $\\xkun=\\xk$ and $\\dkun=\\gamma^{-1}\\dk$, in which case\n\\begin{equation}\\label{bOne}\n\\phi_{k+1}-\\phi_k\\leq(1-\\nu)\\left(\\frac{1}{\\gamma^2}-1\\right)\\dk^2=:b_1<0.\n\\end{equation}\n\nRecall the event $\\Gamma_k=\\accolade{\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\geq\\zeta\\Dk}$ of Lemma~\\ref{indicatorMeasurable} with $\\zeta$ satisfying ~\\eqref{zetaChoice}. The proof considers two cases: $\\igk=1$ and $\\igkc=1$. Inspired by~\\cite[proof of Theorem~4.11]{chen2018stochastic}, Case~1 aims to show that\n\\begin{equation}\\label{huit}\n%\\begin{split}\n\\E{\\igk\\left(\\Phikun-\\Phik\\right)|\\fqalgebra}\\leq-2\\igk(1-\\nu)(\\gamma^2-1)\\Dk^2\n\\leq -\\frac{1}{2}\\igk\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})\\Dk^2, \n%\\end{split}\n\\end{equation}\nwhere the last inequality follows from $1-\\frac{1}{\\gamma^2}<\\gamma^2-1$ and $\\beta_f\\leq 1$. On the other hand, inspired by~\\cite[proof of Theorem~3]{blanchet2016convergence}, Case~2 shows that\n\\begin{equation}\\label{last}\n\\E{\\igkc\\left(\\Phikun-\\Phik\\right)|\\fqalgebra}\\leq-\\frac{1}{2}\\igkc\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})\\Dk^2.\n\\end{equation}\nThus, combining~\\eqref{huit} and~\\eqref{last} leads to~\\eqref{PhikIncrementBound}.\n\n$ $\\\\\n\\textbf{Case~1: $\\igk=1$}\\\\\n$ $\\\\\nUnlike the proof of~\\cite[Theorem~4.11, Case~1]{chen2018stochastic} where four subcases were considered, only three are analyzed next.\n\n%\\begin{itemize}\n%\\item[(i)] \n\\noindent \\textit{(Subcase 1i)}\nGood model ($\\iik=1$) and good estimates ($\\ijk=1$). By~\\eqref{zetaChoice}, \n\\[\\normii{\\QkTranspose\\nabla f(\\xk)}\\geq \\max\\accolade{\\eta_2+\\kappa_{eg}, \\kappa_h+\\kappa_{eg}, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}}+\\kappa_{eg}}\\dk,\\]\nimplying in particular~\\eqref{deltaLemmaB2} and hence a decrease in $f$ according to~\\eqref{LemmaB2improvement}. Moreover, by $(\\kef,\\keg;\\Qk)$-full linearity, \n\\[\\normii{\\gkhat}\\geq \\normii{\\QkTranspose\\nabla f(\\xk)}-\\kappa_{eg}\\dk\\geq \\max\\accolade{\\eta_2,\\kappa_h, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}}\\dk, \\]\nthus implying~\\eqref{deltaLemmaC}; and since $\\ef\\leq \\kappa_{ef}$ per assumptions of Lemma~\\ref{LemmaC}, the iteration is successful. Consequently,~\\eqref{un} together with~\\eqref{LemmaB2improvement} yields\n\\begin{equation}\\label{bTwo}\n\\begin{split}\n\t\\phi_{k+1}-\\phi_k &\\leq -\\nu C_1\\normii{\\QkTranspose\\nabla f(\\xk)}\\dk+(1-\\nu)(\\gamma^2-1)\\dk^2=:b_2\\\\\n\t&\\leq \\left(-\\nu C_1\\zeta+(1-\\nu)(\\gamma^2-1)\\right)\\dk^2<0,\n\\end{split}\n\\end{equation}\nwhere the last inequality follows from~\\eqref{nuChoiceScal}. Denoting by ${\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)$ the random variable with realizations $b_2<0$, with ${\\ubar{u}}_k$ defined in~\\eqref{ukRandomMeasurable}, we have from~\\eqref{bTwo} that\n\\begin{equation}\\label{bTwoRandom}\n\\igk\\iik\\ijk\\left(\\Phikun-\\Phik\\right)\\leq \\igk\\iik\\ijk{\\ubar{b}}_2({\\ubar{u}}_k,\\Dk).\n\\end{equation}\n%\\item[(ii)] \n\\noindent \\textit{(Subcase 1ii)}\nBad model ($\\iikc=1$) and good estimates ($\\ijk=1$). In this case, regardless of the iteration type (i.e., successful or unsuccessful), the change in $\\phi_k$ can always be bounded by using~\\eqref{un} and~\\eqref{LemmaEincrease} as follows:\n\\begin{equation}\\label{bThree}\n\\phi_{k+1}-\\phi_k\\leq\\nu C_3\\norme{\\QkTranspose\\nabla f(\\xk)}\\dk+(1-\\nu)(\\gamma^2-1)\\dk^2=:b_3.\n\\end{equation}\nDenoting by ${\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)$ the random variable with realizations $b_3>0$, from~\\eqref{bThree} we have that\n\\begin{equation}\\label{bThreeRandom2}\n\\igk\\iikc\\ijk\\left(\\Phikun-\\Phik\\right)\\leq \\igk\\iikc\\ijk{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)\\leq \\igk\\iikc{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk).\n\\end{equation}\n%\\item[(iii)] \n\\noindent \\textit{(Subcase 1iii)}\nBad estimates ($\\ijkc=1$). From \\textit{Subcase 1ii}, it always holds that \n\\begin{equation}\\label{bThreeRandom3}\n\\igk\\ijkc\\left(\\Phikun-\\Phik\\right)\\leq \\igk\\ijkc{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk).\n\\end{equation}\n%\\end{itemize}\n\nWith the subcases thus complete, since the random variables $\\igk, {\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)$, and ${\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)$ are $\\fqalgebra$-measurable thanks to the proof of Lemma~\\ref{indicatorMeasurable}, combining~\\eqref{bTwoRandom},~\\eqref{bThreeRandom2}, and~\\eqref{bThreeRandom3} and taking expectations with respect to $\\fqalgebra$, we have\n\\begin{equation}\\label{bTwoThree}\n\\begin{split}\n\\E{\\igk\\left(\\Phikun-\\Phik\\right)|\\fqalgebra}&\\leq \\igk{\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)\\E{\\iik\\ijk|\\fqalgebra}\\\\\n&+\\igk{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)\\left(\\E{\\iikc|\\fqalgebra}+\\E{\\ijkc|\\fqalgebra}\\right)\\\\\n&\\leq \\igk\\left(\\beta_f\\beta_m{\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)+{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)(1-\\beta_m+1-\\beta_f)\\right)=:\\igk {\\ubar{b}}_{2,3}({\\ubar{u}}_k,\\Dk),\n\\end{split}\n\\end{equation}\nwhere the last inequality follows from~\\eqref{ikQProb} and~\\eqref{prodBetafBetam} and the fact that ${\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)<0$ while ${\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)>0$.  Note that \n\\begin{eqnarray*}\n{\\ubar{b}}_{2,3}({\\ubar{u}}_k,\\Dk)&=&\\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\left[-\\beta_f\\beta_m C_1+(2-\\beta_f-\\beta_m)C_3\\right]\\\\\n& &+(1-\\nu)(\\gamma^2-1)(2-\\beta_f-\\beta_m+\\beta_f\\beta_m)\\Dk^2\\\\\n&\\leq& \\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\left[-\\beta_f\\beta_m C_1+(2-\\beta_f-\\beta_m)C_3\\right] + 2(1-\\nu)(\\gamma^2-1)\\Dk^2,\n\\end{eqnarray*}\nwhere the last inequality follows from $2-\\beta_f-\\beta_m+\\beta_f\\beta_m=1+(1-\\beta_f)(1-\\beta_m)\\leq 2$.\n\nFor $\\beta_f$ and $\\beta_m$ chosen according to the first condition in~\\eqref{probChoices}, and for $\\nu$ satisfying~\\eqref{nuChoiceScal}, the following holds:\n\\[\\beta_f\\beta_m C_1 - (2-\\beta_f-\\beta_m)C_3\\geq \\frac{1}{2}C_1\\geq 2\\frac{2(1-\\nu)(\\gamma^2-1)}{\\nu\\zeta}.\\]\nHence, since $\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\geq \\zeta\\Dk$, then \n\\begin{eqnarray*}\n{\\ubar{b}}_{2,3}({\\ubar{u}}_k,\\Dk)&\\leq& -\\frac{1}{2}\\left[\\beta_f\\beta_m C_1-(2-\\beta_f-\\beta_m)C_3\\right]\\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\\\\n&\\leq& -\\frac{1}{4}C_1\\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\leq -2(1-\\nu)(\\gamma^2-1)\\Dk^2,\n\\end{eqnarray*}\nwhich, together with~\\eqref{bTwoThree}, leads to~\\eqref{huit}.\n\n$ $\\\\\n\\textbf{Case~2: $\\igkc=1$}\n$ $\\\\ \nConsider the event $G_k:=\\accolade{\\normii{\\Gkhat}\\geq \\eta_2\\Dk}$, and note that since $\\Mhatk:={\\ubar{f}_k}+\\Gkhat^{\\top}\\sRandom +\\frac{1}{2}\\sRandom^{\\top}\\HkhatRandom\\sRandom$ is $\\fmqalgebra$-measurable by construction, then in particular $\\normii{\\Gkhat}$ and hence $\\igkhat, \\igkhatc$ are also $\\fmqalgebra$-measurable. Observe that if $\\igkhatc=1$, then the iteration is unsuccessful since $\\normii{\\gkhat}<\\eta_2\\dk$, leading to~\\eqref{bOne}. Hence, \n\\begin{equation}\\label{GkbarCase2}\n\\begin{split}\n\\E{\\igkc\\igkhatc(\\Phikun-\\Phik)|\\fmqalgebra}&\\leq -(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\E{\\igkc\\igkhatc\\Dk^2|\\fmqalgebra}\\\\\n&\\leq -\\frac{1}{2}\\igkc\\igkhatc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{split}\n\\end{equation}\nwhere the last inequality is due to the $\\fmqalgebra$-measurability of $\\igkc$ and $\\Dk$, given that $\\falgebra\\subset\\fqalgebra\\subset\\fmqalgebra$. \n\nInspired by~\\cite[Theorem~3]{blanchet2016convergence}, which improved on the proof of~\\cite[Theorem~4.11, Case~2]{chen2018stochastic} in which four subcases were considered, only two are analyzed next. The overall goal is to prove that\n\\begin{equation}\\label{GkCase2}\n\\E{\\igkc\\igkhat(\\Phikun-\\Phik)|\\fmqalgebra}\\leq -\\frac{1}{2}\\igkc\\igkhat\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{equation}\nwhich, combined with~\\eqref{GkbarCase2}, leads to \n\\begin{equation*}%\\label{GammaBarFmq}\n\\E{\\igkc(\\Phikun-\\Phik)|\\fmqalgebra}\\leq -\\frac{1}{2}\\igkc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2\n\\end{equation*}\nand consequently\n\\begin{equation*}%\\label{GammaBarFq}\n\\begin{split}\n\\E{\\igkc(\\Phikun-\\Phik)|\\fqalgebra}&=\\E{\\E{\\igkc(\\Phikun-\\Phik)|\\fmqalgebra}|\\fqalgebra}\\\\\n&\\leq \\E{-\\frac{1}{2}\\igkc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2 \\, |\\fqalgebra}=-\\frac{1}{2}\\igkc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{split}\n\\end{equation*}\n showing~\\eqref{last}, where the last equality follows from the $\\fqalgebra$-measurability of $\\igkc$ and $\\Dk$.  What remains to be shown is~\\eqref{GkCase2} as achieved next, assuming that $\\igkhat=1$.\n\n\\noindent \\textit{(Subcase 2i)}\n%\\begin{itemize}\n%\\item[(i)] \nGood estimates ($\\ijk=1$) and $\\igkhat=1$. While $f$ is decreased on successful iterations because  of good estimates thanks to Lemma~\\ref{LemmaD} and in this case~\\eqref{LemmaDimprovement} holds,  $\\dk$ is reduced on unsuccessful iterations, leading to~\\eqref{bOne}. However, combining~\\eqref{LemmaDimprovement} and~\\eqref{un} yields\n$\\phi_{k+1}-\\phi_k\\leq \\left[-\\nu C_2+(1-\\nu)(\\gamma^2-1)\\right]\\dk^2\\leq b_1,$\nwhere the last inequality is due to~\\eqref{nuChoiceScal}, which shows that~\\eqref{bOne} holds in any case. Consequently,\n\\begin{equation}\n\\label{GammakBarGkJk}\n\\igkc\\igkhat\\ijk(\\Phikun-\\Phik) \\leq -\\igkc\\igkhat\\ijk(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2.\n\\end{equation}\n\n%\\item[(ii)] \n\\noindent \\textit{(Subcase 2ii)}\nBad estimates ($\\ijkc=1$) and $\\igkhat=1$. In this case, a successful step can lead to an increase in $f$\naccording to~\\eqref{LemmaEincrease}, which combined with~\\eqref{un} yield~\\eqref{bThree} and hence\n\\begin{equation}\\label{GammakBarGkJkBar}\n\\igkc\\igkhat\\ijkc(\\Phikun-\\Phik) \\leq -\\igkc\\igkhat\\ijkc\\left(\\nu C_3\\zeta+(1-\\nu)(\\gamma^2-1)\\right)\\Dk^2.\n\\end{equation}\n%\\end{itemize}\n\nWith the subcases complete, recall that $\\E{\\ijk|\\fmqalgebra}\\geq\\beta_f$. Then  combining~\\eqref{GammakBarGkJk} and~\\eqref{GammakBarGkJkBar} and taking expectations with respect to~$\\fmqalgebra$ lead to\n\\begin{eqnarray*}%\\label{GkCase2Proved}\n\\E{\\igkc\\igkhat(\\Phikun-\\Phik)|\\fmqalgebra}&\\leq& \\igkc\\igkhat\\left(-\\beta_f(1-\\nu)(1-1/\\gamma^2)\\right.\\\\ \n& &\\left. +\\ (1-\\beta_f)\\left(\\nu C_3\\zeta+(1-\\nu)(\\gamma^2-1)\\right) \\right)\\Dk^2\\\\\n&\\leq& -\\frac{1}{2}\\igkc\\igkhat\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{eqnarray*}\nthus proving~\\eqref{GkCase2}, where the last inequality follows from the second condition in~\\eqref{probChoices}, and the proof is complete.\n\\end{proof}","start":102803,"end":113478},{"raw_tex":"\\begin{theorem}\\label{liminfTheorem}\nLet all the assumptions made in Theorem~\\ref{zerothOrderScal} hold. Assume further that Assumption~\\ref{wellAlignedAssumption} holds with $0\\leq\\beta_Q<1-\\frac{1}{2\\beta_f\\beta_m}$. Then, the sequence $\\accolade{\\Xk}_{k\\in\\N}$ of random iterates generated by Algorithm~\\ref{algoStoScalTR} satisfies\n\\[\\underset{k\\to\\infty}{\\liminf}\\normii{\\nabla f(\\Xk)}=0\\quad\\mbox{ almost surely.}\\]\n\\end{theorem}","start":114364,"end":114799},{"raw_tex":"\\begin{proof}\nThe result is proved by contradiction conditioned on the almost sure event $E_0:=\\accolade{\\Dk\\to 0}$ (thanks to Theorem~\\ref{zerothOrderScal}), inspired by the proof of~\\cite[Theorem~4.16]{chen2018stochastic} and also using ideas from~\\cite{audet2019stomads,dzahini2020constrained}. Assume that with nonzero probability there exists a random variable ${\\ubar{\\varepsilon}}'>0$ such that \n$\\normii{\\nabla f(\\Xk)}\\geq {\\ubar{\\varepsilon}}' \\mbox{ for all } k\\in\\N.$\n\nLet $\\accolade{\\xk}, \\accolade{\\dk}$, and ${{\\varepsilon}}'$ be realizations of $\\accolade{\\Xk}, \\accolade{\\Dk}$, and ${\\ubar{\\varepsilon}}'$, respectively, for which $\\normii{\\nabla f(\\xk)}\\geq {{\\varepsilon}}'$ for all $k$. Since $\\dk\\to 0$, there exists $k_0\\in\\N$ such that\n\\begin{equation}\\label{bChoice}\n\\dk<b:=\\alpha_Q\\cdot\\min\\accolade{\\frac{{{\\varepsilon}}'}{2(\\kappa_h+\\kappa_{eg})}, \\frac{{{\\varepsilon}}'}{2(\\eta_2+\\kappa_{eg})}, \\frac{{{\\varepsilon}}'}{\\frac{16\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}+2\\kappa_{eg}}, \\frac{\\dmaxx}{\\gamma}}\n\\end{equation}\nfor all $k\\geq k_0$.\nLet $\\ubar{r}_k$ be the random variable with realizations $r_k=\\log_{\\gamma}\\left(\\frac{\\dk}{b}\\right)$. Then $r_k<0$ for all $k\\geq k_0$. The main idea is to show that such realizations occur only with probability zero, leading to a contradiction. \n\nTo prove that $\\accolade{\\ubar{r}_k}$ is a submartingale, recall the events $I^Q_k$ and $J^Q_k$ satisfying~\\eqref{otherProbIneqs} and $\\mathcal{A}_k$ satisfying~\\eqref{Akineq}.\n%$\\prob{\\mathcal{A}_k|\\falgebra}\\geq 1-\\beta_Q$, \n%$\\prob{I^Q_k|\\falgebra}\\geq\\beta_m$ and  $\\prob{J^Q_k|\\falgebra}\\geq\\beta_f$, \n%where $\\mathcal{F}_{k-1}^Q :=\\sigma\\left(\\mathcal{A}_0,\\mathcal{A}_1,\\dots,\\mathcal{A}_{k-1}\\right)\\cap \\sigma\\left(I^Q_0,I^Q_1,\\dots,I^Q_{k-1}\\right)\\cap \\sigma\\left(J^Q_0,J^Q_1,\\dots,J^Q_{k-1}\\right)$. \nConsider some iteration $k\\geq k_0$ for which $\\mathcal{A}_k$, $I^Q_k$, and $J^Q_k$ all occur, which happens with probability at least $\\beta_f\\beta_m(1-\\beta_Q)$ conditioned on $\\falgebra$, as was shown in~\\eqref{prodBetaIneq}. It follows from~\\eqref{bChoice} that \n\\begin{equation}\\label{ineq4}\n\\normii{\\nabla f(\\xk)}\\geq \\frac{1}{\\alpha_Q}\\max\\accolade{\\kappa_h+\\kappa_{eg}, \\eta_2+\\kappa_{eg}, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}+\\kappa_{eg}}\\dk,\n\\end{equation}\nwhich, together with the $(\\kef,\\keg;\\Qk)$-full linearity and $\\alpha_Q$-well alignment, yield\n\\begin{equation}\\label{ineq5}\n%\\begin{split}\n\\normii{\\gkhat}\\geq\\normii{\\QkTranspose\\nabla f(\\xk)}-\\kappa_{eg}\\dk\\geq \\alpha_Q \\normii{\\nabla f(\\xk)}-\\kappa_{eg}\\dk\n\\geq \\max\\accolade{\\kappa_h, \\eta_2, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}}\\dk.\n%\\end{split}\n\\end{equation}\nThus, the $k$th iteration is successful according to Lemma~\\ref{LemmaC}. Hence, $\\dkun=\\gamma\\dk$, and hence $r_{k+1}=r_k+1$. For all other outcomes of $\\mathcal{A}_k$, $I^Q_k$, and $J^Q_k$, which occur with a total probability of at most $1-\\beta_f\\beta_m(1-\\beta_Q)$, it always holds that $\\dkun\\geq\\gamma^{-1}\\dk$, which implies that $r_{k+1}\\geq r_k-1$. Hence, \n\\begin{eqnarray*}\n\\E{\\mathds{1}_{\\mathcal{A}_k\\cap I^Q_k\\cap J^Q_k} \\left(\\ubar{r}_{k+1}-\\ubar{r}_k\\right)|\\falgebra}&\\geq& \\beta_f\\beta_m(1-\\beta_Q)\\quad \\quad\\mbox{and}\\\\\n\\E{\\mathds{1}_{\\overline{\\mathcal{A}_k\\cap I^Q_k\\cap J^Q_k}} \\left(\\ubar{r}_{k+1}-\\ubar{r}_k\\right)|\\falgebra}&\\geq& \\beta_f\\beta_m(1-\\beta_Q)-1, \n\\end{eqnarray*}\nwhich shows that $\\accolade{\\ubar{r}_k}$ is a submartingale if $0\\leq\\beta_Q<1-\\frac{1}{2\\beta_f\\beta_m}$.\n\nUsing the random walk defined in~\\eqref{randomWalk}, one  can easily  show (following, e.g.,~\\cite[Theorem~4]{audet2019stomads}, \\cite[Theorem~4.16]{chen2018stochastic} and~\\cite[Theorem~3.6 and Lemmas~4.7 and~4.12]{dzahini2020constrained}) that $\\accolade{\\ubar{w}_k}$ is also a submartingale with bounded increments and, as such, cannot converge to a finite value, and therefore $\\prob{\\underset{k\\to\\infty}{\\limsup}\\ \\ubar{w}_k=\\infty}=1$ (thanks to~\\cite[Theorem~4.4]{chen2018stochastic}). However, since $r_k-r_{k_0}\\geq w_k-w_{k_0}$ by construction,  the sequence of realizations~$r_k$ such that $r_k<0\\ \\forall k\\ \\geq k_0$ occurs with probability zero, leading to a contradiction, which achieves the proof.\n\\end{proof}","start":114801,"end":119014},{"raw_tex":"\\begin{lemma}\\label{smallLemma}\nLet all the assumptions made in Theorem~\\ref{liminfTheorem} hold. Then~\\eqref{dynamicsRenewal} is satisfied for\\\\ $\\Wtildek=2\\left(\\iak\\iik\\ijk-\\frac{1}{2} \\right)$, with $\\lambda=\\ln(\\gamma)$ and some fixed $q\\in (\\tilde{\\beta}, 1)$ with $\\tilde{\\beta}=\\beta_f\\beta_m(1-\\beta_Q)$.\n\\end{lemma}","start":125901,"end":126226},{"raw_tex":"\\begin{proof}\nThe result is proved by suitably adapting the proof of~\\cite[Lemma~7]{blanchet2016convergence} as was done in~\\cite{dzahini2020expected}. First, we notice that~\\eqref{dynamicsRenewal} trivially holds when $\\itksup=0$. Next, we show that when $\\itksup=1$, then \n\\[\\Dkun\\geq \\min\\accolade{\\depsilon, \\min\\accolade{\\dmaxx,\\gamma\\Dk}\\iak\\iik\\ijk+\\gamma^{-1}\\Dk\\left(1-\\iak\\iik\\ijk\\right)}. \\]\nRecall that $\\dk=\\gamma^{i_k}\\depsilon$ for some integer $i_k$. Therefore, if $\\dk>\\depsilon$, then $\\dk\\geq \\gamma\\depsilon$, which implies that $\\dkun\\geq\\gamma^{-1}\\dk\\geq\\depsilon$. Now assume that $\\dk\\leq \\depsilon$. Since $\\tauEpsilon>k$, we have $\\normii{\\nabla f(\\xk)}>\\epsilon=\\xi\\depsilon\\geq\\xi\\dk$. Thus, it follows from the definition of~$\\xi$ that~\\eqref{ineq4} holds. If $\\iak=1$, $\\iik=1$, and $\\ijk=1$, then~\\eqref{ineq5} holds, and consequently the $k$th iteration is successful, as was  explained in the proof of Theorem~\\ref{liminfTheorem}. Hence, $\\xkun=\\xk+\\Qk\\sk$ and $\\dkun=\\min\\accolade{\\dmaxx,\\gamma\\dk}$. If $\\iak\\iik\\ijk=0$, then it always holds that $\\dkun\\geq \\gamma^{-1}\\dk$. The proof is completed by  observing that $\\prob{\\iak\\iik\\ijk=1}=q$ for some fixed $q\\geq \\tilde{\\beta}$.\n\\end{proof}","start":126228,"end":127456},{"raw_tex":"\\begin{proof}\nRecall the choice of $q$ in Lemma~\\ref{smallLemma}, and pick some $\\tilde{\\xi}\\geq\\hat{\\xi}$ such that $\\depsilon=\\frac{\\epsilon}{\\tilde{\\xi}}=\\gamma^{i}\\delta_0$ for some integer $i\\leq 0$ (without loss of generality), as discussed above. The proof follows by employing Theorem~\\ref{theorComplexity1} with $h(t)=t^2$, which yields\n\\[\\E{\\tauEpsilon}\\leq \\frac{q}{2q-1}\\cdot\\frac{\\PhiZero\\tilde{\\xi}^2}{\\varrho \\epsilon^2}+1\\leq \\frac{\\tilde{\\beta}}{2\\tilde{\\beta}-1}\\cdot\\frac{\\PhiZero\\tilde{\\xi}^2}{\\varrho \\epsilon^2}+1.\\]\n\\end{proof}","start":127969,"end":128519},{"raw_tex":"\\begin{proof}\nThe proof is similar to that of~\\cite[Lemma~4.5]{chen2018stochastic} and is not detailed here. It follows from~\\eqref{CauchyDecrease} together with the inequalities $\\kappa_h\\geq \\max\\accolade{\\norme{\\Hkhat}, 1}$ and $\\normii{\\gkhat}\\geq\\kappa_h\\dk$ that  $\\mhatk(\\bm{0})-\\mhatk(\\sk)\\geq\\frac{\\kappa_{fcd}}{2}\\normii{\\gkhat}\\dk.$ Since the model is $(\\kef,\\keg;\\Qk)$-fully linear, then \n$f(\\xk+\\Qk\\sk)-f(\\xk)\\leq 2\\kappa_{ef}\\dk^2+\\mhatk(\\sk)-\\mhatk(\\bm{0})\\leq -\\frac{\\kappa_{fcd}}{4}\\normii{\\gkhat}\\dk,$\nwhere the last inequality follows from the fact that $\\dk\\leq \\frac{\\kappa_{fcd}}{8\\kappa_{ef}}\\normii{\\gkhat}$.\n\\end{proof}","start":141109,"end":141753},{"raw_tex":"\\begin{proof}\nThe proof is inspired by (and similar to) that of~\\cite[Lemma~4.6]{chen2018stochastic}. By $(\\kef,\\keg;\\Qk)$-full linearity and~\\eqref{deltaLemmaB2}, it holds\n\\begin{equation}\\label{ineq1}\n\\normii{\\gkhat}\\geq \\normii{\\QkTranspose\\nabla f(\\xk)}-\\kappa_{eg}\\dk\\geq \\max\\accolade{\\kappa_h, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}}}\\dk, \n\\end{equation}\nwhich shows that condition~\\eqref{LemmaAimprovement} is satisfied. Consequently $f$ decreases as in~\\eqref{LemmaAimprovement}. The first inequality in~\\eqref{ineq1}, together with~\\eqref{deltaLemmaB2}, yields\n$\\normii{\\gkhat}\\geq \\frac{4C_1}{\\kappa_{fcd}}\\normii{\\QkTranspose\\nabla f(\\xk)},$\nwhich, combined with~\\eqref{LemmaAimprovement}, implies~\\eqref{LemmaB2improvement}, and the proof is complete.\n\\end{proof}","start":141801,"end":142571},{"raw_tex":"\\begin{proof}\nAccording to the Fundamental Theorem of Calculus in $\\rn$ (see, e.g.,~\\cite[Lemma~9.3]{AuHa2017}),\\\\\n$\\abs{f(\\xk)-f(\\xk+\\Qk\\sk)-\\nabla f(\\xk+\\Qk\\sk)^{\\top}(-\\Qk\\sk)}\\leq\\frac{1}{2}L_g\\Qmax^2\\dk^2,$  with the inequality also following from $\\normii{\\Qk\\sk}\\leq \\Qmax\\dk$, which implies that\n\\begin{equation}\\label{ineq2}\nf(\\xk+\\Qk\\sk)-f(\\xk)\\leq \\sk^{\\top}\\left[\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)\\right] + \\frac{1}{2}L_g\\Qmax^2\\dk^2. \n\\end{equation}\nPer Lipschitz continuity of $\\nabla f$, \n$\\normii{\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)-\\QkTranspose \\nabla f(\\xk)} \\leq L_g\\Qmax^2\\dk,$\nwhich implies that\n\\begin{equation}\\label{ineq3}\n\\normii{\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)}\\leq L_g\\Qmax^2\\dk+\\normii{\\QkTranspose \\nabla f(\\xk)}.\n\\end{equation}\nThus, using~\\eqref{ineq2} and~\\eqref{ineq3}, together with the inequality $\\dk\\leq \\zeta^{-1}\\normii{\\QkTranspose \\nabla f(\\xk)}$, yields\n\\begin{eqnarray*}\nf(\\xk+\\Qk\\sk)&-&f(\\xk)\\leq \\normii{\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)}\\dk+\\frac{L_g}{2\\zeta}\\Qmax^2\\normii{\\QkTranspose \\nabla f(\\xk)}\\dk\\\\\n&\\leq& \\left(\\frac{L_g}{\\zeta}\\Qmax^2\\normii{\\QkTranspose \\nabla f(\\xk)} +\\normii{\\QkTranspose \\nabla f(\\xk)}\\right)\\dk\\\\\n&+&\\frac{L_g}{2\\zeta}\\Qmax^2\\normii{\\QkTranspose \\nabla f(\\xk)}\\dk= \\left(1+\\frac{3L_g}{2\\zeta}\\Qmax^2\\right)\\normii{\\QkTranspose \\nabla f(\\xk)}\\dk,\n\\end{eqnarray*}\nwhich completes the proof.\n\\end{proof}","start":142618,"end":143995}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["probl1"],"declared_external":false,"source_only":false,"title":"随机无约束优化","source_units":[{"raw_tex":"This paper focuses on unconstrained stochastic DFO, wherein the objective function values are accessible only through a blackbox oracle corrupted by stochastic noise. We consider the problem \n\\begin{equation}\\label{probl1}\n\t\\underset{\\x\\in\\rn}{\\min} f(\\x), \\qquad\\mbox{with}\\quad f(\\x)=\\Esp_{\\ubar{\\theta}}\\left[f_{\\ubar{\\theta}}(\\x)\\right],\n\\end{equation}\n%\\quad\\mbox{with}\\quad f(\\x)=\\Esp_{\\ubar{\\theta}}\\left[f_{\\ubar{\\theta}}(\\x)\\right]\nwhere the values of the continuously differentiable function $f:\\rn\\to\\R$ are available only via $f_{\\ubar{\\theta}}$, a stochastically noisy version of $f$, and where $\\ubar{\\theta}$ is a random variable whose distribution is possibly unknown. \n","start":28799,"end":29485}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"子空间模型与试探步","source_units":[{"raw_tex":"Unlike the stochastic trust-region framework STORM~\\cite{blanchet2016convergence,chen2018stochastic}, which builds full space random models $\\ubar{m}_k(\\x)$, $\\x\\in\\rn$, to approximate $f(\\x)$, STARS operates as follows. On iteration $k$, given a current iterate $\\xk$, let $\\mathcal{Y}_k\\subseteq \\rn$ be the affine space randomly chosen through the range of a matrix $\\Qk\\in\\rnp$ \n%satisfying $\\norme{\\Qk}\\leq\\Qmax$ for some $\\Qmax>0$, \nwhose entries are randomly selected; that is, \n\\[\\mathcal{Y}_k = \\accolade{\\xk+\\Qk\\bm{{\\s}}:\\bm{{\\s}}\\in\\R^p}.\\]\nThen, given a trust-region radius $\\dk>0$, a subspace model $\\mhatk(\\s), \\s\\in\\rp$ is built only on $\\mathcal{Y}_k$ using realizations of the stochastically noisy function $f_{\\ubar{\\theta}}$; the model  serves as an approximation of $f(\\x)$ in $\\mathcal{B}(\\xk,\\dk;\\Qk):=\\accolade{\\xk+\\Qk\\s \\in \\mathcal{Y}_k:\\normii{\\s}\\leq\\dk}$. For concreteness, here we employ a quadratic subspace model given by \n\\begin{equation*}\nf(\\xk+\\Qk\\s)\\approx\\mhatk(\\bm{\\s}):= f_k+\\gkhat^\\top\\s+\\s^\\top\\Hkhat\\s,\n\\end{equation*}\nwhere $\\gkhat\\in \\R^p$ and $\\Hkhat\\in \\Sym^{p}$ are the low-dimensional model gradient and Hessian, respectively. A {\\it tentative step} $\\Qk\\sk\\in\\rn$ is produced by using the solution $\\sk\\in\\rp$ obtained by approximately minimizing $\\mhatk$ inside the trust  region; that  is, $\\sk\\approx{\\argmin}\\left\\{ \\mhatk(\\s): \\s\\in\\rp, \\, \\normii{\\bm{{\\s}}}\\leq \\dk\\right\\}$. Inspired by~\\cite{CRsubspace2021,chen2018stochastic}, the {\\it trial step} $\\sk$ has to provide a sufficient decrease in $\\mhatk$ by satisfying the following standard fraction of the Cauchy decrease condition.\n\n","start":39790,"end":41431}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","labels":["CauchyDecrease"],"declared_external":false,"source_only":false,"title":"柯西充分下降条件","source_units":[{"raw_tex":"\\begin{assumption}\nFor every iteration $k$, a trial step $\\sk\\in\\rp$ is computed so that\\footnote{Throughout the manuscript, the matrix norm $\\norme{\\cdot}$ is supposed to be {\\it consistent} with the Euclidean norm; that is, $\\normii{\\Ab\\xb}\\leq \\norme{\\Ab}\\normii{\\xb}$, which holds for both Frobenius and spectral matrix norms.}\n\\begin{equation}\\label{CauchyDecrease}\n\\mhatk(\\bm{0})-\\mhatk(\\bm{\\sk})\\geq \\frac{\\kappa_{fcd}}{2}\\normii{\\gkhat}\\min\\accolade{\\dk,\\frac{\\normii{\\gkhat}}{\\max\\left(\\norme{\\Hkhat},1\\right)}},\n\\end{equation}\nfor some constant $\\kappa_{fcd}\\in (0,1]$.\n\\end{assumption}\n","start":41431,"end":42028}]},{"id":"S004","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"概率空间与随机记号","source_units":[{"raw_tex":"The random variables considered here are all defined on the same probability space $(\\Omega,\\F, \\pr)$, where $\\Omega$, referred to as the {\\it sample space}, is a nonempty set whose elements $\\omega$ and subsets are called {\\it sample points} and {\\it events}, respectively. $\\F$ is a collection of such events, which is called a {\\it $\\sigma$-algebra}. $\\pr$ is a finite measure defined on the measurable space $(\\Omega,\\F)$, satisfying $\\prob{\\Omega}=1$ and referred to as the {\\it probability measure}. An increasing subsequence $\\accolade{\\mathcal{S}_k}_k$ of $\\sigma$-algebras of $\\F$ will be called a {\\it filtration}. Let $\\mathscr{B}(\\rn)$ be the $\\sigma$-algebra generated by the open sets of $\\rn$, also known as the Borel $\\sigma$-algebra of $\\rn$. A random variable $\\ubar{z}$ is a measurable map defined on $(\\Omega,\\F, \\pr)$ into the measurable space $(\\rn, \\mathscr{B}(\\rn))$, where measurability means that $\\accolade{\\ubar{z}\\in A}:=\\accolade{\\omega\\in\\Omega:\\ubar{z}(\\omega)\\in A} =:{\\ubar{z}}^{-1}(A)\\in\\F\\ \\forall A\\in \\mathscr{B}(\\rn)$~\\cite{bhattacharya2007basic}. For the remainder of the manuscript, vectors will be written in lowercase boldface (e.g., $\\x\\in\\rn, n\\geq 2$) while matrices will be written in uppercase boldface (e.g., $\\Qk\\in\\rnp$), and underlined letters (e.g., $\\ubar{z}, \\ubar{\\x}, \\QkRandom$) will be used to denote random quantities; $z=\\ubar{z}(\\omega)$ will denote a realization of $\\ubar{z}$.\n","start":43183,"end":44624}]},{"id":"S005","semantic_type":"algorithm","kind":"algorithm","labels":["algoStoScalTR"],"declared_external":false,"source_only":false,"title":"STARS 完整算法","source_units":[{"raw_tex":"\\begin{algorithm}[htb]\n\\caption{STARS}\n\\label{algoStoScalTR} \n\\textbf{[0] Initialization}\\\\\n\\hspace*{10mm}Choose constants $\\gamma>1$, $\\eta_1\\in(0,1)$, $\\eta_2>0$, $j_{\\max}\\in\\N$, $\\varepsilon\\in(0,1)$, $\\beta\\in(0,1)$, \\\\ \n\\hspace*{10mm}$c_1\\geq 1$, initial trust-region radius $\\delta_0>0$, and maximum trust-region radius\\\\\n\\hspace*{10mm}$\\dmaxx=\\gamma^{j_{\\max}}\\delta_0$, starting point $\\bm{{\\x}_0}\\in\\rn$, and dimension $p\\in\\intbracket{1,n}$.\\\\ \n\\hspace*{10mm}Set the iteration counter $k \\gets 0$.\\\\\n\\textbf{[1] Construction of subspace model}\\\\\n\\hspace*{10mm}Generate $\\Qk$: a realization of a\\footnotemark {\\em WAM}($1-\\varepsilon,\\beta$), using a distribution $\\mathbb{Q}_{k}$.\\\\\n\\hspace*{10mm}Build model $\\mhatk:\\rp\\to\\R$ that is $(\\kef,\\keg;\\Qk)$-fully linear \n%$\\mhatk(\\s)={\\f}_k+\\gkhat^{\\top}\\s+$\\\\\n%\\hspace*{10mm}$\\frac{1}{2}\\s^{\\top}\\Hkhat\\s\\approx f(\\xk+\\Qk \\s)$, that locally approximates $f$ \nin  $\\mathcal{B}(\\xk,c_1\\dk;\\Qk)$.\\\\ \n%\\hspace*{10mm}with the affine space defined by the columns of $\\Qk$.\\\\\n\\textbf{[2] Step calculation}\\\\\n\\hspace*{10mm}Compute $\\sk\\approx {\\argmin}\\left\\{ \\mhatk(\\s) : \\; \\s\\in\\rp:\\normii{\\s}\\leq \\dk\\right \\}$ satisfying~\\eqref{CauchyDecrease}.\\\\\n\\textbf{[3] Estimate computation}\\\\\n\\hspace*{10mm}Obtain estimates $\\fok\\approx f(\\xk)$ and $\\fsk\\approx f(\\xk+\\Qk\\sk)$ satisfying~\\eqref{accEstimates}.\\\\\n\\textbf{[4] Updates}\\\\\n\\hspace*{10mm}Compute $\\rho_k=\\frac{\\fok-\\fsk}{\\mhatk(\\bm{0})-\\mhatk(\\sk)}.$\\\\\n\\hspace*{10mm}If $\\rho_k\\geq \\eta_1$ and $\\normii{\\gkhat}\\geq \\eta_2\\dk$ (\\textbf{success}): \\\\\n\\hspace*{16mm}set $\\bm{\\xkun}=\\xk+\\Qk\\sk$ and $\\dkun=\\min\\accolade{\\gamma\\dk,\\dmaxx}$. \\\\\n\\hspace*{10mm}Otherwise (\\textbf{failure}): set $\\bm{\\xkun}=\\xk$ and $\\dkun=\\gamma^{-1}\\dk$.\\\\\n\\hspace*{10mm}Update the iteration counter $k \\gets k+1$, and go to~\\textbf{[1]}.\\\\\n\\end{algorithm}\n%\\caption[]{\\small{STARS algorithm}}\n%\\end{figure*}\n\\footnotetext{See Definition~\\ref{wellAlignedDefinition} for the meaning of {\\em WAM}($1-\\varepsilon,\\beta$).}\n","start":44648,"end":46651}]},{"id":"S006","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"算法生成随机过程","source_units":[{"raw_tex":"The deterministic models $\\mhatk$ and the function estimates $\\fok$ and $\\fsk$ are constructed by using evaluations of the noisy function $f_{\\ubar{\\theta}}$ and the randomly selected entries of the deterministic matrix~$\\Qk=\\QkRandom(\\omega)$.  Hence, $\\mhatk$, $\\fok$, and $\\fsk$ can be considered as realizations of random models and estimates $\\Mhatk$, $\\Fok$, and $\\Fsk$, respectively. Each iteration of Algorithm~\\ref{algoStoScalTR} is therefore influenced by the behavior of these random quantities; and consequently, the algorithm results in a stochastic process. The present work shows that under certain conditions on the sequences~$\\accoladekinN{\\QkRandom}$, $\\accoladekinN{\\Fok,\\Fsk}$, and $\\accoladekinN{\\Mhatk}$, the resulting stochastic process has desirable convergence properties, conditioned on the past, where {\\it past} means the {\\it past history} of the algorithm (see~Remark~\\ref{whyTheseSigmaAlgebras}). In particular, for the needs of convergence and expected complexity analyses presented  in Sections~\\ref{Section4} and~\\ref{Section5}, and as will be seen  in Section~\\ref{Section3}, the random models~$\\Mhatk$ and estimates~$\\Fok,\\Fsk$ are required to be probabilistically sufficiently accurate conditioned on the past, as was the case in~\\cite{chen2018stochastic}. Moreover, $\\QkRandom$ must guarantee a high quality of the selected subspace probabilistically and conditioned on the past, as will be detailed in Section~\\ref{Section31}. \n\nFor all $k\\in\\N$, denote by $(\\QkRandom)_{ij},  (i,j)\\in \\intbracket{1,n}\\times\\intbracket{1,p}=:\\mathbb{I}_{n,p}$ the entries of the random matrix $\\QkRandom$, where for $a\\leq b$, $\\intbracket{a,b}:=[a,b]\\cap\\mathbb{Z}$  throughout the manuscript. \n","start":46654,"end":48373}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"三层历史滤过","source_units":[{"raw_tex":"To formalize conditioning on the past, we consider the three filtrations $\\accolade{\\falgebra}_{k\\in\\N}$, $\\accolade{\\fqalgebra}_{k\\in\\N}$, and $\\accolade{\\fmqalgebra}_{k\\in\\N}$ defined respectively by \n\\begin{equation*}\n\\begin{array}{rcl}\n\\falgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell,  (\\QellRandom)_{ij} \\mbox{ with}\\  \\ell \\in \\intbracket{0,k-1} \\mbox{ \\&}\\ (i,j)\\in \\mathbb{I}_{n,p} \\right)\\\\\n\\fqalgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell, (\\QellRandom)_{ij}, (\\QkRandom)_{ij} \\mbox{ with}\\  \\ell \\in \\intbracket{0,k-1} \\mbox{ \\&}\\ (i,j)\\in \\mathbb{I}_{n,p} \\right)\\\\\n\\fmqalgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell, (\\QellRandom)_{ij}, (\\QkRandom)_{ij}, \\ \\Mhatk \\mbox{ with}\\  \\ell \\in \\intbracket{0,k-1} \\mbox{ \\&}\\ (i,j)\\in \\mathbb{I}_{n,p}  \\right).\n\\end{array}\n\\end{equation*}\n% \\begin{eqnarray*}\n%  \\falgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell\\ \\mbox{and}\\ (\\QellRandom)_{ij}\\ \\mbox{with}\\  \\ell \\in \\intbracket{0,k-1}\\ \\mbox{and}\\ (i,j)\\in \\mathbb{I}_{n,p}\\right)\\\\\n%  \\fqalgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell,\\ (\\QellRandom)_{ij}\\ \\mbox{and}\\ (\\QkRandom)_{ij}\\ \\mbox{with}\\  \\ell \\in \\intbracket{0,k-1}\\ \\mbox{and}\\ (i,j)\\in \\mathbb{I}_{n,p}\\right)\\\\\n%  \\fmqalgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell,\\ (\\QellRandom)_{ij},\\ \\Mhatk\\ \\mbox{and}\\ (\\QkRandom)_{ij}\\ \\mbox{with}\\  \\ell \\in \\intbracket{0,k-1}\\ \\mbox{and}\\ (i,j)\\in \\mathbb{I}_{n,p}\\right).\n% \\end{eqnarray*}\nHere, for {\\it completeness}~\\cite{billingsley1995ThirdEdition}, $\\mathcal{F}_{-1}=\\sigma(\\bm{\\x_0})$.\n%One can easily see that . \nBy construction, $\\falgebra\\subset\\fqalgebra\\subset\\fmqalgebra$, $\\QkRandom$ is both~$\\fqalgebra$ and~$\\fmqalgebra$-measurable, and $\\Mhatk$ is~$\\fmqalgebra$-measurable, which imply that $\\E{\\QkRandom|\\fqalgebra}=\\E{\\QkRandom|\\fmqalgebra}=\\QkRandom$ and $\\E{\\Mhatk|\\fmqalgebra}=\\Mhatk$. \n%The  motivation for  introducing these $\\sigma$-algebras in the formalization of conditioning on the past is explained next.\n\n","start":48373,"end":50347}]},{"id":"S008","semantic_type":"interpretation","kind":"interpretation","labels":["whyTheseSigmaAlgebras"],"declared_external":false,"source_only":false,"title":"历史信息的阶段含义","source_units":[{"raw_tex":"\\begin{remark}\\label{whyTheseSigmaAlgebras}\nAt a given iteration~$k$, when generating~$\\QkRandom$, because no model or estimates were generated since the start of the iteration, it obviously follows from the construction of Algorithm~\\ref{algoStoScalTR} that its past history can be formalized by~$\\falgebra$. On the other hand, when constructing~$\\Mhatk$, the past of the algorithm that includes~$\\QkRandom$, given that the latter is already generated since the beginning of iteration~$k$, can therefore be formalized by~$\\fqalgebra$. A similar observation  explains why~$\\fmqalgebra$ can formalize the past history of Algorithm~\\ref{algoStoScalTR} before the construction of~$\\Fok$ and~$\\Fsk$. These observations will also play an important role in the formalization of sufficient accuracy of the random models and estimates in Definitions~\\ref{defProbQkfully} and~\\ref{defProbAccEstim} and the formalization of high quality of the random subspaces in Assumption~\\ref{wellAlignedAssumption}.  \n\\end{remark}\n","start":50347,"end":51356}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":["Akineq","wellAlignedDefinition"],"declared_external":false,"source_only":false,"title":"概率良好对齐","source_units":[{"raw_tex":"Recalling Remark~\\ref{whyTheseSigmaAlgebras} and inspired by~\\cite{CRsubspace2021}, we propose a probabilistic quantification of the quality of the subspace selection as follows.\n\n\\begin{definition}\\label{wellAlignedDefinition}\nFor fixed $\\alpha_Q,\\beta_Q\\in (0,1)$, a sequence $\\accoladekinN{\\QkRandom}$ of random matrices \n%determining the selection of subspaces \nis $(1-\\beta_Q)$-probabilistically $\\alpha_{Q}$-well aligned if, for any $\\F_{k-1}$-measurable random vector $\\viRandom$ with realizations $\\vi\\in \\rn$, the events \n$\\mathcal{A}_k:= \\accolade{\\normii{\\QkRandomTranspose\\viRandom}\\geq \\alpha_{{Q}}\\normii{\\viRandom}}$ satisfy \n\\begin{equation}\\label{Akineq}\n\\prob{\\mathcal{A}_k|\\F_{k-1}}=\\E{\\iak|\\F_{k-1}}\\geq 1-\\beta_Q,\n\\end{equation}\nwhere $\\iak$ denotes the indicator function of the event $\\mathcal{A}_k$; that  is, $\\iak(\\omega)=1$ if $\\omega\\in \\mathcal{A}_k$ and $\\iak(\\omega)=0$ otherwise. Random matrices satisfying~\\eqref{Akineq} will be referred to as {WAM}($\\alpha_Q,\\beta_Q$).\n%, in other words, well-aligned matrices with parameters $\\alpha_Q$ and $\\beta_Q$.\n\\end{definition}\n","start":53148,"end":54252}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","labels":["wellAlignedAssumption"],"declared_external":false,"source_only":false,"title":"子空间对齐假设","source_units":[{"raw_tex":"%\\begin{definition}\n%Random matrices satisfying the conditions of Lemma... with parameters $\\varepsilon, \\beta$ will be referred to as a {\\em JLM}($\\varepsilon,\\beta$), i.e., Johnson-Lindenstrauss matrices with parameters~$\\varepsilon$ and~$\\beta$.\n%\\end{definition}\nFor convergence purposes  in Sections~\\ref{Section4} and~\\ref{Section5}, the following will be assumed throughout the manuscript.\n\\begin{assumption}\\label{wellAlignedAssumption}\nThe sequence $\\accoladekinN{\\QkRandom}$ of matrices used by Algorithm~\\ref{algoStoScalTR} is $(1-\\beta_Q)$-probabilistically $\\alpha_{Q}$-well aligned for some fixed $\\alpha_Q\\in (0,1)$ and $\\beta_Q\\in (0,1/2)$.\n\\end{assumption}\n","start":54252,"end":54926}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","labels":["QmaxAssumption"],"declared_external":false,"source_only":false,"title":"子空间矩阵一致有界","source_units":[{"raw_tex":"Inspired by~\\cite{CRsubspace2021}, a technical assumption needed for convergence and expected complexity analyses later in Sections~\\ref{Section4} and~\\ref{Section5} is the following. \n\\begin{assumption}\\label{QmaxAssumption}\nAny realization $\\Qk$ of the random matrix used by Algorithm~\\ref{algoStoScalTR} satisfies $\\norme{\\Qk}\\leq \\Qmax$ for all $k\\in\\N$, for some constant $\\Qmax >0$ independent of~$k$.\n\\end{assumption}\n","start":63629,"end":64054}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["Eq36","QkFullyLinearModels"],"declared_external":false,"source_only":false,"title":"子空间完全线性模型","source_units":[{"raw_tex":"\\subsection{Probabilistic models and estimates in random subspaces.}\nAs mentioned at the beginning of Section~\\ref{Section3}, the models used by Algorithm~\\ref{algoStoScalTR} need to be sufficiently accurate. Motivated by~\\cite{chen2018stochastic}, this sufficient accuracy is formalized for deterministic subspace models by the following measure of accuracy inspired by~\\cite{CRsubspace2021}.\n\\begin{definition}\\label{QkFullyLinearModels}\nAssume $\\nabla f$ is  Lipschitz continuous. Given $\\Qk\\in\\rnp$ and $\\xk\\in\\rn$, a function $\\mhatk:\\rp\\to\\R$ is a $(\\kef,\\keg; \\Qk)$-fully linear model of $f$ in $\\mathcal{B}(\\xk,\\dk;\\Qk)$ for some $\\dk>0$ if there exist constants $\\kappa_{ef}, \\kappa_{eg} >0$ independent of $k$ such that for all $\\s\\in\\rp$ with $\\normii{\\s}\\leq \\dk$,\n\\begin{equation}\\label{Eq36}\n%\\begin{split}\n\\abs{f(\\xk +\\Qk\\s)-\\mhatk(\\s)}\\leq \\kappa_{ef}\\dk^2\\quad \\mbox{and}\\quad\n\\normii{\\Qk^{\\top}\\nabla f(\\xk +\\Qk\\s)-\\nabla\\mhatk(\\s)}\\leq \\kappa_{eg}\\dk.\n%\\end{split}\n\\end{equation}\n\\end{definition}\nAs pointed out in~\\cite{CRsubspace2021}, the gradient condition in~\\eqref{Eq36} is due to the fact that given any fixed~$\\x$ and~$\\Q$, if $\\hat{f}(\\s):=f(\\x+\\Q\\s)$, then $\\nabla_{\\s} \\hat{f}(\\s)=\\Q^{\\top}\\nabla_{\\x} f(\\x+\\Q\\s)$. Moreover, denoting by $\\bm{I}$ the identity matrix in full-dimensional subspaces where $p=n$, then $(\\kef,\\keg; \\bm{I})$-fully linear models correspond to standard fully linear models~\\cite[Definition~6.1]{CoScVibook}.\n\n%Inspired by~\\cite{chen2018stochastic}, and \nRecalling Remark~\\ref{whyTheseSigmaAlgebras}, the next definition formalizing sufficient accuracy of probabilistic subspace models $\\Mhatk$ is a stochastic variant of Definition~\\ref{QkFullyLinearModels}, which generalizes~\\cite[Definition~3.4]{chen2018stochastic} to low-dimensional subspaces. The existence of such models will be rigorously demonstrated  in Theorem~\\ref{subspaceModel1} and Corollary~\\ref{CorQkFully}.\n","start":67168,"end":69099}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["defProbQkfully","ikQProb"],"declared_external":false,"source_only":false,"title":"模型条件精度事件","source_units":[{"raw_tex":"\\begin{definition}\\label{defProbQkfully}\nA sequence $\\accolade{\\Mhatk}$ of random models is $\\beta_m$-probabilistically \\\\ $(\\kef,\\keg; \\Qk)$-fully linear with respect to the random sequence $\\accolade{\\mathcal{B}(\\Xk,\\Dk;\\QkRandom)}$ if the events\n\\[I_k^Q=\\accolade{\\mbox{Given}\\ \\Qk\\ \\mbox{and}\\ \\xk, \\Mhatk\\ \\mbox{is}\\ (\\kef,\\keg;\\Qk)\\mbox{-fully linear for}\\ f\\ \\mbox{in}\\ \\mathcal{B}(\\Xk,\\Dk;\\QkRandom)}\\]\nsatisfy the supermartingale-like condition\n\\begin{equation}\\label{ikQProb}\n\\prob{I^Q_k|\\fqalgebra}=\\E{\\iik|\\fqalgebra}\\geq \\beta_m.\n\\end{equation}\n\\end{definition}\nRecalling that estimates of unknown function values also need to be sufficiently accurate, we introduce a formalization of sufficient accuracy inspired by~\\cite{audet2019stomads,chen2018stochastic,dzahini2020expected,dzahini2020constrained,paquette2018stochastic} and motivated by~\\cite{CRsubspace2021} as follows.\n","start":69099,"end":69989}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":["accEstimates"],"declared_external":false,"source_only":false,"title":"函数估计精度","source_units":[{"raw_tex":"\\begin{definition}\nGiven $\\ef>0$ and $\\Qk\\sk\\in\\rn, \\, \\xk\\in\\rn$, $\\fok$ and $\\fsk$ are called $\\ef$-accurate estimates of $f(\\xk)$ and $f(\\xk+\\Qk\\sk)$, respectively, for a given $\\dk$ if\n\\begin{equation}\\label{accEstimates}\n\\abs{\\fok-f(\\xk)}\\leq\\ef\\dk^2\\quad \\mbox{ and }\\quad  \\abs{\\fsk-f(\\xk+\\Qk\\sk)}\\leq\\ef\\dk^2.\n\\end{equation}\n\\end{definition}\n\n\nRecalling Remark~\\ref{whyTheseSigmaAlgebras},\n%and following~\\cite{audet2019stomads,chen2018stochastic,dzahini2020expected,dzahini2020constrained,paquette2018stochastic}, \nwe  extend  this definition to the following stochastic variant.\n","start":69989,"end":70578}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":["defProbAccEstim"],"declared_external":false,"source_only":false,"title":"估计条件精度事件","source_units":[{"raw_tex":"\\begin{definition}\\label{defProbAccEstim}\nA sequence of random estimates $\\accolade{\\Fok,\\Fsk}$ is said to be $\\beta_f$-probabilistically $\\ef$-accurate with respect to the corresponding sequence $\\accolade{\\Xk,\\Dk,\\QkRandom\\Sk}$ if the events \n\\begin{eqnarray*}\nJ_k^Q &=&\\left\\lbrace\\mbox{Given}\\ \\Qk\\ \\mbox{and}\\ \\xk, \\Fok \\mbox{ and } \\Fsk \\mbox{ are }\\ef\\mbox{-accurate estimates of } f(\\xk)  \\mbox{ and}\\right.\\\\\n& &\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad \\left.  f(\\xk+\\Qk\\sk), \\mbox{ respectively, for } \\Dk \\right\\rbrace\n\\end{eqnarray*}\n% \nsatisfy the supermartingale-like condition\n$\\prob{J^Q_k|\\fmqalgebra}\\geq \\beta_f.$\n\\end{definition}\n\n\n","start":70578,"end":71237}]},{"id":"S016","semantic_type":"derived_fact","kind":"derived_fact","labels":["prodBetaIneq"],"declared_external":false,"source_only":false,"title":"真实迭代联合概率下界","source_units":[{"raw_tex":"Based on Definitions~\\ref{wellAlignedDefinition}, \\ref{defProbQkfully}, and~\\ref{defProbAccEstim}, {\\it true} iterations, in other words, those for which $\\iak\\iik\\ijk=1$, occur with a total probability of at least $\\tilde{\\beta}:=\\beta_f\\beta_m(1-\\beta_Q)$ conditioned on $\\falgebra$. Noting that by construction $\\iak$ is both $\\fqalgebra$-measurable and $\\fmqalgebra$-measurable since $\\fqalgebra\\subset\\fmqalgebra$, and that $\\iik$ is $\\fmqalgebra$-measurable, then for $F:= \\accolade{\\mathcal{A}_k\\cap I^Q_k\\cap J_k^Q}$, we have\n\\begin{equation}\\label{prodBetaIneq}\n\\begin{split}\n\\pr(F|\\falgebra)&=\\E{\\iak\\iik\\ijk|\\falgebra}\n=\\E{\\iak\\iik\\E{\\ijk|\\fmqalgebra} |\\falgebra}\\\\\n&\\geq \\beta_f\\E{\\iak\\E{\\iik|\\fqalgebra} |\\falgebra}\\geq \\beta_f\\beta_m\\E{\\iak|\\falgebra}\\geq \\tilde{\\beta}.\n%&\\geq \\beta_f\\beta_m(1-\\beta_Q),\n\\end{split}\n\\end{equation}\n","start":71237,"end":72083}]},{"id":"S017","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件列出的 billingsley1995ThirdEdition，Theorems 34.3 and 34.4；L1055 明确陈述并引用。"],"source_only":false,"title":"条件期望运算规则","source_units":[{"raw_tex":"The second equality and the first two inequalities used the  {\\it William's Tower Property} for conditional expectations~\\cite[Theorem~34.4.]{billingsley1995ThirdEdition}, namely, the fact that if ${\\ubar{z}}$ is integrable and the $\\sigma$-algebras $\\mathcal{G}_1$ and $\\mathcal{G}_2$ satisfy $\\mathcal{G}_1\\subset\\mathcal{G}_2$, then $\\E{{\\ubar{z}}|\\mathcal{G}_1}=\\E{\\E{{\\ubar{z}}|\\mathcal{G}_2}|\\mathcal{G}_1}$. Moreover, for a $\\sigma$-algebra $\\mathcal{G}$,  $\\E{{\\ubar{r}}{\\ubar{u}}|\\mathcal{G}}={\\ubar{r}}\\E{{\\ubar{u}}|\\mathcal{G}}$ if~${\\ubar{r}}$ is $\\mathcal{G}$-measurable and~${\\ubar{r}}$ and~${\\ubar{r}}{\\ubar{u}}$ are integrable~\\cite[Theorem~34.3.]{billingsley1995ThirdEdition}.\n","start":72083,"end":72777}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":["randomWalk"],"declared_external":false,"source_only":false,"title":"真实迭代随机游走","source_units":[{"raw_tex":"\nNote that even though the present algorithmic framework does not distinguish true iterations from {\\it false} ones, it holds that $\\prob{\\underset{k\\to\\infty}{\\limsup}\\ \\ubar{w}_k=\\infty}=1$, where\n\\begin{equation}\\label{randomWalk}\n\\ubar{w}_k:=\\sum_{i=0}^k\\left(2\\mathds{1}_{\\mathcal{A}_i}\\mathds{1}_{I^Q_i}\\mathds{1}_{J^Q_i}-1\\right),\n\\end{equation}\nas will be seen later in the proof of Theorem~\\ref{liminfTheorem} provided $\\tilde{\\beta}>1/2$. This shows that true iterations occur sufficiently often for the convergence of Algorithm~\\ref{algoStoScalTR} to hold (see, e.g.,~\\cite[Theorem~3.6]{dzahini2020constrained} for a similar result).\n\n\n","start":72777,"end":73424}]},{"id":"S019","semantic_type":"derived_fact","kind":"derived_fact","labels":["otherProbIneqs","prodBetafBetam"],"declared_external":false,"source_only":false,"title":"其余条件概率下界","source_units":[{"raw_tex":"\\begin{remark}%\\label{remarkProdBetafBetam}\nCalculations similar to those in~\\eqref{prodBetaIneq} easily yield \n\\begin{eqnarray}\\label{prodBetafBetam}\n\\prob{I^Q_k\\cap J_k^Q|\\fqalgebra}\\geq \\beta_f\\beta_m, & & \\prob{J_k^Q|\\fqalgebra}\\geq \\beta_f,\\\\\n%\\end{equation}\n%\\begin{equation}\n\\label{otherProbIneqs}\n\\prob{I^Q_k|\\falgebra}\\geq\\beta_m & \\qquad \\mbox{ and } \\qquad & \\prob{J^Q_k|\\falgebra}\\geq\\beta_f,\n\\end{eqnarray}\nwhich will be useful in the proofs of Theorems~\\ref{zerothOrderScal} and~\\ref{liminfTheorem}, respectively. \n%and as was considered in~\\cite[Theorems~4.11]{chen2018stochastic} in similar situations.\n\\end{remark}\n","start":73424,"end":74056}]},{"id":"S020","semantic_type":"assumption","kind":"assumption","labels":["fGradfAssumption"],"declared_external":false,"source_only":false,"title":"目标下界与正则性","source_units":[{"raw_tex":"In the remainder of the manuscript, the following inspired by~\\cite[Assumptions~4.1 and~4.3]{chen2018stochastic}, respectively, will be assumed.\n\\begin{assumption}\\label{fGradfAssumption}\nFor given $\\bm{\\x_0}$, $\\dmaxx$, and $\\bm{Q_0}$, let $\\mathscr{L}(\\bm{\\x_0};\\bm{Q_0})\\subset\\rn$ be the set containing all iterates of Algorithm~\\ref{algoStoScalTR}. Define the region $\\mathscr{L}^{\\star}(\\bm{\\x_0};\\bm{Q_0})$ considered by the algorithm realizations as $\\mathscr{L}^{\\star}(\\bm{\\x_0};\\bm{Q_0})=\\ds{\\underset{\\accolade{k\\, \\geq\\, 0,\\, \\xk\\in\\mathscr{L}(\\bm{\\x_0};\\bm{Q_0})}}{\\bigcup}}\\mathcal{B}(\\xk,\\dmaxx;\\Qk)$. Then $f(\\x)\\geq f_{\\min}$ for all $\\x\\in\\rn$ and for some constant $f_{\\min}>-\\infty$. Moreover, $f$ and its gradient are Lipschitz continuous on $\\mathscr{L}^{\\star}(\\bm{\\x_0};\\bm{Q_0})$.\n\\end{assumption}\n\n","start":90450,"end":91275}]},{"id":"S021","semantic_type":"assumption","kind":"assumption","labels":["boundedHessian"],"declared_external":false,"source_only":false,"title":"模型海森矩阵界","source_units":[{"raw_tex":"\\begin{assumption}\\label{boundedHessian}\nThere exists some $\\kappa_h\\geq 1$ such that for all $k\\geq 0$, the Hessian $\\Hkhat$ of all realizations of $\\Mhatk$ satisfies $\\norme{\\Hkhat}\\leq \\kappa_h.$\n\\end{assumption}\n","start":91275,"end":91491}]},{"id":"S022","semantic_type":"lemma","kind":"lemma","labels":["LemmaC","deltaLemmaC"],"declared_external":false,"source_only":false,"title":"准确模型估计保证成功","source_units":[{"raw_tex":"\\begin{lemma}\\label{LemmaC}\nLet Assumption~\\ref{boundedHessian} hold. Assume that $\\mhatk$ is $(\\kef,\\keg;\\Qk)$-fully linear and the estimates $\\accolade{\\fok,\\fsk}$ are $\\ef$-accurate with $\\ef\\leq\\kappa_{ef}$. Then the $k$th iteration is successful if \n\\begin{equation}\\label{deltaLemmaC}\n\\dk\\leq\\min\\accolade{\\frac{1}{\\kappa_h}, \\frac{1}{\\eta_2}, \\frac{\\kappa_{fcd}(1-\\eta_1)}{8\\kappa_{ef}}}\\normii{\\gkhat}.\n\\end{equation}\n\\end{lemma}","start":93481,"end":93918}]},{"id":"S023","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"成功引理的转引证明","source_units":[{"raw_tex":"\\begin{proof}\nThe proof immediately follows from that of~\\cite[Lemma~4.7]{chen2018stochastic} with minor modifications and is not presented here again.\n\\end{proof}\n","start":93920,"end":94084}]},{"id":"S024","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件列出的 chen2018stochastic, Lemma 4.7；作者于 L1292 调用。"],"source_only":false,"title":"外部成功性引理","source_units":[]},{"id":"S025","semantic_type":"lemma","kind":"lemma","labels":["LemmaD","LemmaDimprovement"],"declared_external":false,"source_only":false,"title":"准确成功步的目标下降","source_units":[{"raw_tex":"\\begin{lemma}\\label{LemmaD}\nSuppose that Assumption~\\ref{boundedHessian} holds and the estimates $\\accolade{\\fok,\\fsk}$ are $\\ef$-accurate with $\\ef<\\frac{1}{4}\\eta_1\\eta_2\\kappa_{fcd}\\min\\accolade{\\frac{\\eta_2}{\\kappa_h}, 1}$. If the $k$th iteration is successful, then the improvement in $f$ is such that\n\\begin{equation}\\label{LemmaDimprovement}\nf(\\xkun)-f(\\xk)\\leq -C_2\\dk^2,\n\\end{equation}\nwhere $C_2=\\frac{1}{2}\\eta_1\\eta_2\\kappa_{fcd}\\min\\accolade{\\frac{\\eta_2}{\\kappa_h}, 1} -2\\ef>0$. \n\\end{lemma}","start":94242,"end":94747}]},{"id":"S026","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":true,"title":"准确下降引理的转引证明","source_units":[{"raw_tex":"\\begin{proof}\nAgain, the proof immediately follows from that of~\\cite[Lemma~4.8]{chen2018stochastic} with minor modifications and is not presented here.\n\\end{proof}\n","start":94749,"end":94914}]},{"id":"S027","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件列出的 chen2018stochastic, Lemma 4.8；作者于 L1304 调用。"],"source_only":false,"title":"外部准确下降引理","source_units":[]},{"id":"S028","semantic_type":"assumption","kind":"assumption","labels":["assumptionsOnEstimatesAndModels"],"declared_external":false,"source_only":false,"title":"随机模型估计精度假设","source_units":[{"raw_tex":"\\subsection{Zeroth-order convergence result.}\\label{Section4p2}\nTo prove in Theorem~\\ref{zerothOrderScal} that the sequence of random trust-region radii converges to zero almost surely, we assume the following.\n\\begin{assumption}\\label{assumptionsOnEstimatesAndModels} \nThe sequences of estimates $\\accolade{\\Fok, \\Fsk}$ and models $\\accolade{\\Mhatk}$ generated by Algorithm~\\ref{algoStoScalTR} are, respectively, $\\beta_f$-probabilistically $\\ef$-accurate for \n$\\ef<\\min\\accolade{\\kappa_{ef}, \\frac{1}{4}\\eta_1\\eta_2\\kappa_{fcd}\\min\\accolade{\\frac{\\eta_2}{\\kappa_h}, 1}}$ and $\\beta_m$-probabilistically $(\\kef,\\keg;\\Qk)$-fully linear for some $\\beta_f, \\beta_m\\in (0,1)$.\n\\end{assumption}\n","start":99141,"end":99832}]},{"id":"S029","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"源文停止时刻定义","source_units":[{"raw_tex":"A formal definition from~\\cite{blanchet2016convergence} of the stopping time related to a discrete time stochastic process is as follows.\n\\begin{definition}\nA random variable~$\\tauEpsilon$ is called a stopping time with respect to a given discrete time stochastic process $\\accolade{\\X_k}_{k\\in\\N}$ if the event $\\accolade{\\tauEpsilon=k}$ belongs to the $\\sigma$-algebra $\\sigma\\left(\\X_1,\\dots,\\X_k \\right)$ generated by $\\X_1,\\dots,\\X_k$, for each $k\\in\\N$.\n\\end{definition}\n\n","start":119853,"end":120331}]},{"id":"S030","semantic_type":"definition","kind":"definition","labels":["dynamicsWk"],"declared_external":false,"source_only":false,"title":"更新报酬框架的随机游走","source_units":[{"raw_tex":"Consider a stochastic process $\\accolade{\\left(\\Phik,\\Dk\\right)}_{k\\in\\N}$, where $\\Phik, \\Dk\\in [0,\\infty)$, and introduce on the same probability space as $\\accolade{\\left(\\Phik,\\Dk\\right)}_{k\\in\\N}$ a biased random walk process $\\accolade{\\Wtildek}_{k\\in\\N}$ obeying the following dynamics:\n\\begin{equation}\\label{dynamicsWk}\n\\prob{\\Wtildekun=1|\\F_k}=q\\qquad \\mbox{ and }\\qquad \\prob{\\Wtildekun=-1|\\F_k}=1-q,\n\\end{equation}\nwhere $q\\in (1/2, 1)$ and $\\F_k=\\sigma\\left( \\left(\\PhiZero,\\Dzero,\\WtildeZero\\right),\\dots,\\left(\\Phik,\\Dk,\\Wtildek\\right) \\right)$, with ${\\WtildeZero=1}$.\n\nLet $\\accolade{\\tauEpsilon}_{\\epsilon>0}$ be a family of stopping times with respect to $\\accoladekinN{\\F_k}$, parameterized by $\\epsilon$. A bound on $\\E{\\tauEpsilon}$ is derived in~\\cite{blanchet2016convergence,dzahini2020expected,paquette2018stochastic} under the following assumption.\n\n","start":120331,"end":121207}]},{"id":"S031","semantic_type":"assumption","kind":"assumption","labels":["assumptionRenewal","dynamicsRenewal"],"declared_external":false,"source_only":false,"title":"更新报酬三项条件","source_units":[{"raw_tex":"\\begin{assumption}\\label{assumptionRenewal}\nThe following hold for the stochastic process $\\accoladekinN{\\left(\\Phik,\\Dk,\\Wtildek\\right)}$.\n\\begin{itemize}\n\\item[(i)] There exist $\\lambda\\in (0, \\infty)$ and $\\dmaxx=\\delta_0 e^{\\lambda j_{\\max}}$ for some $j_{\\max}\\in\\Z$ such that $\\Dk\\leq \\dmaxx$ for all $k$.\n\\item[(ii)] There exists $\\depsilon=\\delta_0 e^{\\lambda j_{\\epsilon}}$, for some $j_{\\epsilon}\\in\\Z$ with $j_{\\epsilon}\\leq 0$ such that for all $k$\n\\begin{equation}\\label{dynamicsRenewal}\n\\itksup\\Dkun\\geq \\itksup\\min\\accolade{\\Dk e^{\\lambda\\Wtildekun}, \\depsilon}, \n\\end{equation}\nwhere $\\Wtildekun$ satisfies~\\eqref{dynamicsWk}. \n\\item[(iii)] There exists a nondecreasing function $h:[0,\\infty)\\to (0,\\infty)$ and a constant $\\varrho>0$ such that for all $k$\n\\[\\E{\\Phikun-\\Phik|\\F_k}\\itksup\\leq-\\varrho h\\left(\\Dk\\right)\\itksup.\\]\n\\end{itemize}\n\\end{assumption}\n\nNoticing that the event $\\accolade{\\Dk\\geq\\depsilon}$ occurs sufficiently frequently on average, often $\\E{\\Phikun-\\Phik}$ can be bounded by some negative fixed constant \\cite{blanchet2016convergence}, thus allowing a bound on the expected stopping time $\\E{\\tauEpsilon}$, as stated next.\n\n","start":121207,"end":122374}]},{"id":"S032","semantic_type":"theorem","kind":"theorem","labels":["theorComplexity1"],"declared_external":true,"declaration_text":["原先决条件明确允许 B0140：从 blanchet2016convergence 回顾的 renewal-reward expected stopping-time theorem，无文内证明。"],"source_only":false,"title":"外部期望停止时间界","source_units":[{"raw_tex":"\\begin{theorem}\\label{theorComplexity1}\nUnder Assumption~\\ref{assumptionRenewal}, \n$\\E{\\tauEpsilon}\\leq \\frac{q}{2q-1}\\cdot\\frac{\\PhiZero}{\\varrho h\\left(\\depsilon\\right)}+1.$\n\\end{theorem}\n","start":122374,"end":122564}]},{"id":"S033","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"算法停止指标与过程识别","source_units":[{"raw_tex":"\\subsection{Expected complexity result.}\\label{ExpectedSubsection}\n\nConsider the process $\\accolade{\\left(\\Phik,\\Dk,\\Wtildek\\right)}_{k\\in\\N}$, where~$\\Phik$ is defined in Theorem~\\ref{zerothOrderScal}, $\\Dk$ is the trust-region radius, and $\\Wtildek$ is  defined by $\\Wtildek=2\\left(\\iak\\iik\\ijk-\\frac{1}{2} \\right)$.  Given $\\epsilon\\in (0,1)$, let $\\tauEpsilon$ be the number of iterations required by Algorithm~\\ref{algoStoScalTR} to first drive the norm of the gradient of~$f$ below $\\epsilon$:  \n\\[\\tauEpsilon:=\\inf \\accolade{k\\in\\N: \\normii{\\nabla f(\\Xk)}\\leq\\epsilon}.\\]\n","start":122565,"end":123144}]},{"id":"S034","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"停止时刻滤过识别断言","source_units":[{"raw_tex":"Then $\\tauEpsilon$ is a stopping time for the stochastic process generated by Algorithm~\\ref{algoStoScalTR}, and hence for $\\accolade{\\left(\\Phik,\\Dk,\\Wtildek\\right)}_{k\\in\\N}$ \\cite{blanchet2016convergence,dzahini2020expected,paquette2018stochastic}. More precisely, the occurrence of $\\accolade{\\tauEpsilon=k}$ can be determined by observing $\\left(\\PhiZero,\\Dzero,\\WtildeZero\\right)$, $\\dots$ $(\\Phikmun,\\Dkmun,\\Wtildekmun)$, which means that $\\tauEpsilon$ is a stopping time with respect to the filtration $\\accoladekinN{\\falgebra}$. To bound $\\E{\\tauEpsilon}$ using Theorem~\\ref{theorComplexity1}, we show next that Assumption~\\ref{assumptionRenewal} holds for $\\accolade{\\left(\\Phik,\\Dk,\\Wtildek\\right)}_{k\\in\\N}$.\n\n\n","start":123144,"end":123867}]},{"id":"S035","semantic_type":"derived_fact","kind":"derived_fact","labels":["PhikIncrSecond"],"declared_external":false,"source_only":false,"title":"向较小历史转移漂移","source_units":[{"raw_tex":"From~\\eqref{PhikIncrementBound}, since $\\falgebra\\subset\\fqalgebra$, we observe that\\footnote{Unlike the analysis in~\\cite{blanchet2016convergence}, the first equality of~\\eqref{PhikIncrSecond} is essential and due to the fact that in the present framework $\\tauEpsilon$ is a stopping time with respect to $\\sigma$-algebras $\\falgebra$ that are smaller than $\\fqalgebra$ with respect to which $\\accoladekinN{\\Phik}$ was proved in Theorem~\\ref{zerothOrderScal} Eq.~\\eqref{PhikIncrementBound} to be a supermartingale.} for all $k\\in\\N$:\n\\begin{equation}\\label{PhikIncrSecond}\n\\E{\\Phikun-\\Phik|\\falgebra}=\\E{\\E{\\Phikun-\\Phik|\\fqalgebra}|\\falgebra}\\leq\\E{-\\varrho\\Dk^2|\\falgebra}=-\\varrho\\Dk^2.\n\\end{equation}\nNote that~\\eqref{PhikIncrSecond} holds for any realizations of the random variables $\\E{\\Phikun-\\Phik|\\falgebra}$ and $\\Dk^2$ and hence on the event $\\accolade{\\tauEpsilon >k}$ in particular, which shows that Assumption~\\ref{assumptionRenewal}-$(iii)$ holds with the constant $\\varrho=\\frac{1}{2}\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})$ of Theorem~\\ref{zerothOrderScal} and $h(t)=t^2$. Assumption~\\ref{assumptionRenewal}-$(i)$ automatically follows from the initialization strategy in Algorithm~\\ref{algoStoScalTR} with $\\lambda=\\ln(\\gamma)$. Before showing by means of Lemma~\\ref{smallLemma}, inspired by~\\cite[Lemma~7]{blanchet2016convergence}, that~\\eqref{dynamicsRenewal} is satisfied, we first define the constant \n","start":123867,"end":125290}]},{"id":"S036","semantic_type":"definition","kind":"definition","labels":["deltaEpsilonDefinition"],"declared_external":false,"source_only":false,"title":"复杂度半径阈值","source_units":[{"raw_tex":"\\begin{equation}\\label{deltaEpsilonDefinition}\n\\depsilon:=\\frac{\\epsilon}{\\xi}, \\quad\\mbox{with } \\xi\\geq  \\frac{1}{\\alpha_Q}\\max\\accolade{\\kappa_h+\\kappa_{eg}, \\eta_2+\\kappa_{eg}, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}+\\kappa_{eg}}=:\\hat{\\xi}, \n\\end{equation}\n","start":125290,"end":125556}]},{"id":"S037","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"半径格点约定","source_units":[{"raw_tex":"inspired by the proof of Theorem~\\ref{liminfTheorem}, especially~\\eqref{ineq4}. Then, following ~\\cite{blanchet2016convergence} exactly, one can assume without loss of generality that $\\depsilon=\\gamma^i\\delta_0$ for some integer $i=:j_{\\epsilon}\\leq 0$, whence $\\Dk=\\gamma^{{\\ubar{i}}_k}\\depsilon$ for any $k$ and some integer~${\\ubar{i}}_k$.\n","start":125556,"end":125900}]},{"id":"S038","semantic_type":"theorem","kind":"theorem","labels":[],"declared_external":false,"source_only":false,"title":"STARS 期望迭代复杂度","source_units":[{"raw_tex":"\\begin{theorem}\nLet all the assumptions made in Theorem~\\ref{liminfTheorem} hold. Then \n\\[\\E{\\tauEpsilon}\\leq \\frac{\\tilde{\\beta}}{2\\tilde{\\beta}-1}\\cdot\\frac{\\PhiZero\\tilde{\\xi}^2}{\\varrho \\epsilon^2}+1\\]\nfor some $\\tilde{\\xi}\\geq\\hat{\\xi}$, where $\\varrho$ is the constant of Theorem~\\ref{zerothOrderScal}, $\\tilde{\\beta}$ is the same constant of Lemma~\\ref{smallLemma}, and $\\hat{\\xi}$ is defined in~\\eqref{deltaEpsilonDefinition}.\n\\end{theorem}","start":127519,"end":127967}]},{"id":"S039","semantic_type":"lemma","kind":"lemma","labels":["LemmaA","LemmaAimprovement"],"declared_external":false,"source_only":false,"title":"按模型梯度保证下降","source_units":[{"raw_tex":"\\begin{lemma}\\label{LemmaA}\nSuppose that the function $\\mhatk$ is a $(\\kef,\\keg;\\Qk)$-fully linear model for~$f$ in $\\B{\\xk}{\\dk;\\Qk}$. Then the trial step $\\sk$ leads to an improvement in $f$ such that\n\\begin{equation}\\label{LemmaAimprovement}\nf(\\xk+\\Qk\\sk)-f(\\xk)\\leq -\\frac{\\kappa_{fcd}}{4}\\normii{\\gkhat}\\dk,\n%\\end{equation}\n%\\begin{equation}\\label{deltaLemmaA}\n\\quad\n\\mbox{if }\n\\dk\\leq \\min\\accolade{\\frac{1}{\\kappa_h},\\frac{\\kappa_{fcd}}{8\\kappa_{ef}}}\\normii{\\gkhat}.\n\\end{equation}\n\\end{lemma}","start":91779,"end":92280}]},{"id":"S040","semantic_type":"lemma","kind":"lemma","labels":["LemmaB2","LemmaB2improvement","deltaLemmaB2"],"declared_external":false,"source_only":false,"title":"按投影真梯度保证下降","source_units":[{"raw_tex":"\\begin{lemma}\\label{LemmaB2}\nLet Assumption~\\ref{boundedHessian} hold, and suppose the model $\\mhatk$ is $(\\kef,\\keg;\\Qk)$-fully linear. Define $C_1=\\frac{\\kappa_{fcd}}{4} \\max\\accolade{\\frac{\\kappa_h}{\\kappa_h+\\kappa_{eg}}, \\frac{8\\kappa_{ef}}{8\\kappa_{ef}+\\kappa_{fcd}\\kappa_{eg}}}$. Then the trial step $\\sk$ leads to an improvement in $f$ such that \n\n\\begin{equation}\\label{LemmaB2improvement}\nf(\\xk+\\Qk\\sk)-f(\\xk)\\leq -C_1\\normii{\\QkTranspose\\nabla f(\\xk)}\\dk,\n\\end{equation}\n\\begin{equation}\\label{deltaLemmaB2}\n\\mbox{if}\\quad\n\\dk\\leq \\min\\accolade{\\frac{1}{\\kappa_h+\\kappa_{eg}}, \\frac{1}{\\frac{8\\kappa_{ef}}{\\kappa_{fcd}}+\\kappa_{eg}}}\\normii{\\QkTranspose\\nabla f(\\xk)}. \n\\end{equation}\n\\end{lemma}","start":92546,"end":93252}]},{"id":"S041","semantic_type":"lemma","kind":"lemma","labels":["LemmaE","LemmaEincrease"],"declared_external":false,"source_only":false,"title":"错误接受步的增幅界","source_units":[{"raw_tex":"\\begin{lemma}\\label{LemmaE}\nUnder Assumptions~\\ref{QmaxAssumption} and~\\ref{fGradfAssumption}, assume that Algorithm~\\ref{algoStoScalTR} erroneously accepts a step $\\sk$ leading to an increase in the objective function when $\\norme{\\QkTranspose\\nabla f(\\xk)}\\geq \\zeta\\dk$ for some constant $\\zeta>0$, and let $C_3(\\zeta)=1+\\frac{3L_g}{2\\zeta}\\Qmax^2$. Then the increase in $f$ is such that\n\\begin{equation}\\label{LemmaEincrease}\nf(\\xk+\\Qk\\sk)-f(\\xk)\\leq C_3(\\zeta)\\norme{\\QkTranspose\\nabla f(\\xk)}\\dk.\n\\end{equation}\n\\end{lemma}","start":95221,"end":95750}]},{"id":"S042","semantic_type":"lemma","kind":"lemma","labels":["partialMeasurable"],"declared_external":false,"source_only":false,"title":"偏导数的 Borel 可测性","source_units":[{"raw_tex":"\\begin{lemma}\\label{partialMeasurable}\nAny partial derivative \n$\\frac{\\partial f}{\\partial x^i}: \\left(\\rn, \\mathscr{B}(\\rn)\\right)\\to \\left(\\R, \\mathscr{B}(\\R)\\right),\\ i\\in\\intbracket{1,n} $,\nof a differentiable function $f$ is a Borel measurable function.\n\\end{lemma}","start":96299,"end":96569}]},{"id":"S043","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"偏导可测性的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $(\\bm{e_1},\\bm{e_2},\\dots,\\bm{e_n})$ be the canonical basis of $\\rn$. For any $i\\in\\intbracket{1,n} $ and $\\bm{x}=(x^1,x^2,\\dots,x^n)$, define, for all $j\\in\\N$,\n$D_{ij}(\\bm{x})=\\frac{f(\\bm{x}+h_j\\bm{e_i})-f(\\bm{x})}{h_j}$,\nwhere $\\accolade{h_j}_{j\\in\\N}$ is a sequence of positive real numbers converging to~$0$; $f$ is differentiable and hence continuous, and thus is Borel measurable~\\cite[Theorem~13.2]{billingsley1995ThirdEdition}. Therefore, since for all $i\\in\\intbracket{1,n}$, $\\underset{j\\to\\infty}{\\lim}D_{ij}(\\bm{x})=\\frac{\\partial f}{\\partial x^i}(\\bm{x}),$\nthen $\\frac{\\partial f}{\\partial x^i}$ is also Borel measurable~\\cite[Theorem~13.3 and Theorem~13.4-(ii)]{billingsley1995ThirdEdition}.\n\\end{proof}","start":96571,"end":97307}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"差商极限保持可测","source_units":[{"raw_tex":"Let $(\\bm{e_1},\\bm{e_2},\\dots,\\bm{e_n})$ be the canonical basis of $\\rn$. For any $i\\in\\intbracket{1,n} $ and $\\bm{x}=(x^1,x^2,\\dots,x^n)$, define, for all $j\\in\\N$,\n$D_{ij}(\\bm{x})=\\frac{f(\\bm{x}+h_j\\bm{e_i})-f(\\bm{x})}{h_j}$,\nwhere $\\accolade{h_j}_{j\\in\\N}$ is a sequence of positive real numbers converging to~$0$; $f$ is differentiable and hence continuous, and thus is Borel measurable~\\cite[Theorem~13.2]{billingsley1995ThirdEdition}. Therefore, since for all $i\\in\\intbracket{1,n}$, $\\underset{j\\to\\infty}{\\lim}D_{ij}(\\bm{x})=\\frac{\\partial f}{\\partial x^i}(\\bm{x}),$\nthen $\\frac{\\partial f}{\\partial x^i}$ is also Borel measurable~\\cite[Theorem~13.3 and Theorem~13.4-(ii)]{billingsley1995ThirdEdition}.\n","start":96585,"end":97296}]},{"id":"S045","semantic_type":"lemma","kind":"lemma","labels":["indicatorMeasurable"],"declared_external":false,"source_only":false,"title":"梯度分区指标的可测性","source_units":[{"raw_tex":"\\begin{lemma}\\label{indicatorMeasurable}\nLet $\\zeta\\geq 0$ be a constant, and consider the event $\\Gamma_k:=\\accolade{\\Gk\\geq 0}$, where $\\Gk:=\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}-\\zeta\\Dk$. Then $\\Gk$, $\\igk$, and $\\igkc$ are $\\fqalgebra$-measurable.\n\\end{lemma}","start":97310,"end":97577}]},{"id":"S046","semantic_type":"proof","kind":"proof","labels":["ukRandomMeasurable"],"declared_external":false,"source_only":false,"title":"分区指标的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nBy construction, all the entries $\\left(\\QkRandom\\right)_{ij},\\ (i,j)\\in\\intbracket{1,n}\\times\\intbracket{1,p}$ of $\\QkRandom$ are $\\fqalgebra$-measurable, while $\\Dk$ and the components $\\Xk^i, \\ i\\in\\intbracket{1, n}$ of $\\Xk$ are $\\falgebra$-measurable and hence $\\fqalgebra$-measurable since $\\falgebra\\subseteq\\fqalgebra$. Thus, for any $i\\in\\intbracket{1, n}$, $\\frac{\\partial f}{\\partial x^i}(\\Xk)$ is $\\fqalgebra$-measurable as the composition of the Borel measurable function $\\frac{\\partial f}{\\partial x^i}$ (from Lemma~\\ref{partialMeasurable}) and the random map $\\Xk:(\\Omega,\\fqalgebra)\\to\\left(\\rn, \\mathscr{B}(\\rn)\\right)$~\\cite[Theorem~13.3 and Theorem~13.1-(ii)]{billingsley1995ThirdEdition}. Recall that  sums and products of measurable functions are also measurable~\\cite[Theorem~13.3]{billingsley1995ThirdEdition}, \n%%consequence of Theorem~13.3\nand note that the random variable \n\\begin{equation}\\label{ukRandomMeasurable}\n\\ubar{u}_k:=\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}^2=\\sum_{j=1}^p\\left(\\sum_{i=1}^n\\left(\\QkRandom\\right)_{ij}\\frac{\\partial f}{\\partial x^i}(\\Xk) \\right)^2 \n\\end{equation}\nis consequently $\\fqalgebra$-measurable. Thus $\\Gk=\\sqrt{{\\ubar{u}_k}}-\\zeta\\Dk$ is also $\\fqalgebra$-measurable. The $\\fqalgebra$-measurability of $\\Gk$ ensures in particular that $\\Gamma_k=\\Gk^{-1}([0, \\infty))\\in\\fqalgebra$ since $[0, \\infty)$ is a Borel set of $\\R$, whence the {\\it simple} real functions $\\igk$ and $\\igkc$~\\cite[Equation~(13.3)]{billingsley1995ThirdEdition} are also $\\fqalgebra$-measurable. \n\\end{proof}","start":97579,"end":99139}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":["ukRandomMeasurable"],"declared_external":false,"source_only":false,"title":"组合可测量并取原像","source_units":[{"raw_tex":"By construction, all the entries $\\left(\\QkRandom\\right)_{ij},\\ (i,j)\\in\\intbracket{1,n}\\times\\intbracket{1,p}$ of $\\QkRandom$ are $\\fqalgebra$-measurable, while $\\Dk$ and the components $\\Xk^i, \\ i\\in\\intbracket{1, n}$ of $\\Xk$ are $\\falgebra$-measurable and hence $\\fqalgebra$-measurable since $\\falgebra\\subseteq\\fqalgebra$. Thus, for any $i\\in\\intbracket{1, n}$, $\\frac{\\partial f}{\\partial x^i}(\\Xk)$ is $\\fqalgebra$-measurable as the composition of the Borel measurable function $\\frac{\\partial f}{\\partial x^i}$ (from Lemma~\\ref{partialMeasurable}) and the random map $\\Xk:(\\Omega,\\fqalgebra)\\to\\left(\\rn, \\mathscr{B}(\\rn)\\right)$~\\cite[Theorem~13.3 and Theorem~13.1-(ii)]{billingsley1995ThirdEdition}. Recall that  sums and products of measurable functions are also measurable~\\cite[Theorem~13.3]{billingsley1995ThirdEdition}, \n%%consequence of Theorem~13.3\nand note that the random variable \n\\begin{equation}\\label{ukRandomMeasurable}\n\\ubar{u}_k:=\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}^2=\\sum_{j=1}^p\\left(\\sum_{i=1}^n\\left(\\QkRandom\\right)_{ij}\\frac{\\partial f}{\\partial x^i}(\\Xk) \\right)^2 \n\\end{equation}\nis consequently $\\fqalgebra$-measurable. Thus $\\Gk=\\sqrt{{\\ubar{u}_k}}-\\zeta\\Dk$ is also $\\fqalgebra$-measurable. The $\\fqalgebra$-measurability of $\\Gk$ ensures in particular that $\\Gamma_k=\\Gk^{-1}([0, \\infty))\\in\\fqalgebra$ since $[0, \\infty)$ is a Borel set of $\\R$, whence the {\\it simple} real functions $\\igk$ and $\\igkc$~\\cite[Equation~(13.3)]{billingsley1995ThirdEdition} are also $\\fqalgebra$-measurable. \n","start":97593,"end":99128}]},{"id":"S048","semantic_type":"assumption","kind":"assumption","labels":["nuChoiceScal","probChoices","zerothOrderScal","zetaChoice"],"declared_external":false,"source_only":false,"title":"漂移分析参数限制","source_units":[{"raw_tex":"Let all assumptions that were made in Lemmas~\\ref{LemmaA}-\\ref{LemmaD} hold with the same constants $C_1, C_2$, and $C_3:=C_3(\\zeta)$, for some fixed $\\zeta>0$ satisfying\n\\begin{equation}\\label{zetaChoice}\n\\zeta\\geq \\kappa_{eg}+\\max\\accolade{\\eta_2,\\kappa_h, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}}.\n\\end{equation}\nLet $\\nu\\in (0,1)$ be chosen according to \n\\begin{equation}\\label{nuChoiceScal}\n\\frac{\\nu}{1-\\nu} \\geq \\frac{8\\gamma^2}{\\min\\accolade{C_2, \\zeta C_1}}.  \n\\end{equation}\nAssume further that Assumption~\\ref{assumptionsOnEstimatesAndModels} holds with $\\beta_f, \\beta_m\\in (0,1)$ satisfying\\footnote{In~\\eqref{probChoices}, the result is intentionally presented with $(1-\\beta_f)+(1-\\beta_m)$ instead of $2-\\beta_f-\\beta_m$ in order to emphasize one of its differences compared with~\\cite{blanchet2016convergence,chen2018stochastic} in which a term similar to $(1-\\beta_f)(1-\\beta_m)$ was derived.}\n\\begin{equation}\\label{probChoices}\n\\frac{\\beta_f\\beta_m-\\frac{1}{2}}{(1-\\beta_f)+(1-\\beta_m)}\\geq \\frac{C_3}{C_1}\\quad \\mbox{ and }\\quad \\frac{\\beta_f}{1-\\beta_f}\\geq \\frac{2\\nu\\zeta\\gamma^2C_3}{(1-\\nu)(\\gamma^2-1)}+2\\gamma^2.\n\\end{equation}\n","start":101216,"end":102375}]},{"id":"S049","semantic_type":"definition","kind":"definition","labels":["zerothOrderScal"],"declared_external":false,"source_only":false,"title":"势函数与漂移常数","source_units":[{"raw_tex":"Define $\\varrho=\\frac{1}{2}\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})>0$. Then, the random function $\\Phik:=\\nu (f(\\Xk)-f_{\\min})+(1-\\nu)\\Dk^2$ satisfies\n","start":102375,"end":102523}]},{"id":"S050","semantic_type":"derived_fact","kind":"derived_fact","labels":["PhikIncrementBound","zerothOrderScal"],"declared_external":false,"source_only":false,"title":"条件负漂移结论","source_units":[{"raw_tex":"\\begin{equation}\\label{PhikIncrementBound}\n\\E{\\Phikun-\\Phik|\\fqalgebra}\\leq -\\varrho\\Dk^2 \\quad \\mbox{for all}\\ k\\in\\N,\n\\end{equation}\n","start":102523,"end":102658}]},{"id":"S051","semantic_type":"derived_fact","kind":"derived_fact","labels":["seriesResult","zerothOrderScal"],"declared_external":false,"source_only":false,"title":"半径平方几乎处处可和","source_units":[{"raw_tex":"which implies that\n\\begin{equation}\\label{seriesResult}\n\\sum_{k=0}^{\\infty}\\Dk^2<\\infty\\quad \\mbox{almost surely.}\n\\end{equation}\n\\end{theorem}","start":102658,"end":102801}]},{"id":"S052","semantic_type":"theorem","kind":"theorem","labels":["PhikIncrementBound","nuChoiceScal","probChoices","seriesResult","zerothOrderScal","zetaChoice"],"declared_external":false,"source_only":false,"title":"负漂移与半径平方可和","source_units":[{"raw_tex":"\\begin{theorem}\\label{zerothOrderScal}\nLet all assumptions that were made in Lemmas~\\ref{LemmaA}-\\ref{LemmaD} hold with the same constants $C_1, C_2$, and $C_3:=C_3(\\zeta)$, for some fixed $\\zeta>0$ satisfying\n\\begin{equation}\\label{zetaChoice}\n\\zeta\\geq \\kappa_{eg}+\\max\\accolade{\\eta_2,\\kappa_h, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}}.\n\\end{equation}\nLet $\\nu\\in (0,1)$ be chosen according to \n\\begin{equation}\\label{nuChoiceScal}\n\\frac{\\nu}{1-\\nu} \\geq \\frac{8\\gamma^2}{\\min\\accolade{C_2, \\zeta C_1}}.  \n\\end{equation}\nAssume further that Assumption~\\ref{assumptionsOnEstimatesAndModels} holds with $\\beta_f, \\beta_m\\in (0,1)$ satisfying\\footnote{In~\\eqref{probChoices}, the result is intentionally presented with $(1-\\beta_f)+(1-\\beta_m)$ instead of $2-\\beta_f-\\beta_m$ in order to emphasize one of its differences compared with~\\cite{blanchet2016convergence,chen2018stochastic} in which a term similar to $(1-\\beta_f)(1-\\beta_m)$ was derived.}\n\\begin{equation}\\label{probChoices}\n\\frac{\\beta_f\\beta_m-\\frac{1}{2}}{(1-\\beta_f)+(1-\\beta_m)}\\geq \\frac{C_3}{C_1}\\quad \\mbox{ and }\\quad \\frac{\\beta_f}{1-\\beta_f}\\geq \\frac{2\\nu\\zeta\\gamma^2C_3}{(1-\\nu)(\\gamma^2-1)}+2\\gamma^2.\n\\end{equation}\nDefine $\\varrho=\\frac{1}{2}\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})>0$. Then, the random function $\\Phik:=\\nu (f(\\Xk)-f_{\\min})+(1-\\nu)\\Dk^2$ satisfies\n\\begin{equation}\\label{PhikIncrementBound}\n\\E{\\Phikun-\\Phik|\\fqalgebra}\\leq -\\varrho\\Dk^2 \\quad \\mbox{for all}\\ k\\in\\N,\n\\end{equation}\nwhich implies that\n\\begin{equation}\\label{seriesResult}\n\\sum_{k=0}^{\\infty}\\Dk^2<\\infty\\quad \\mbox{almost surely.}\n\\end{equation}\n\\end{theorem}","start":101177,"end":102801}]},{"id":"S053","semantic_type":"proof","kind":"proof","labels":["GammakBarGkJk","GammakBarGkJkBar","GkCase2","GkbarCase2","bOne","bThree","bThreeRandom2","bThreeRandom3","bTwo","bTwoRandom","bTwoThree","huit","last","un"],"declared_external":false,"source_only":false,"title":"负漂移定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nThe proof is inspired by those of~\\cite[Theorem~3]{blanchet2016convergence} and~\\cite[Theorem~4.11]{chen2018stochastic}. \nThe overall goal is to prove~\\eqref{PhikIncrementBound}. Indeed noticing that $\\Phik>0$ and then taking expectations on both sides of the inequality in~\\eqref{PhikIncrementBound}, we obtain~\\eqref{seriesResult} (see, e.g.,~\\cite[Theorem~3]{dzahini2020expected} for details). Let $\\phi_k$ denote realizations of $\\Phik$, and recall that on all successful iterations, $\\xkun=\\xk+\\Qk\\sk$ and $\\dkun=\\min\\accolade{\\gamma\\dk,\\dmaxx}$,  implying\n\\begin{equation}\\label{un}\n\\phi_{k+1}-\\phi_k\\leq \\nu\\left(f(\\xkun)-f(\\xk)\\right)+(1-\\nu)(\\gamma^2-1)\\dk^2,\n\\end{equation}\nwhile on unsuccessful iterations, $\\xkun=\\xk$ and $\\dkun=\\gamma^{-1}\\dk$, in which case\n\\begin{equation}\\label{bOne}\n\\phi_{k+1}-\\phi_k\\leq(1-\\nu)\\left(\\frac{1}{\\gamma^2}-1\\right)\\dk^2=:b_1<0.\n\\end{equation}\n\nRecall the event $\\Gamma_k=\\accolade{\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\geq\\zeta\\Dk}$ of Lemma~\\ref{indicatorMeasurable} with $\\zeta$ satisfying ~\\eqref{zetaChoice}. The proof considers two cases: $\\igk=1$ and $\\igkc=1$. Inspired by~\\cite[proof of Theorem~4.11]{chen2018stochastic}, Case~1 aims to show that\n\\begin{equation}\\label{huit}\n%\\begin{split}\n\\E{\\igk\\left(\\Phikun-\\Phik\\right)|\\fqalgebra}\\leq-2\\igk(1-\\nu)(\\gamma^2-1)\\Dk^2\n\\leq -\\frac{1}{2}\\igk\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})\\Dk^2, \n%\\end{split}\n\\end{equation}\nwhere the last inequality follows from $1-\\frac{1}{\\gamma^2}<\\gamma^2-1$ and $\\beta_f\\leq 1$. On the other hand, inspired by~\\cite[proof of Theorem~3]{blanchet2016convergence}, Case~2 shows that\n\\begin{equation}\\label{last}\n\\E{\\igkc\\left(\\Phikun-\\Phik\\right)|\\fqalgebra}\\leq-\\frac{1}{2}\\igkc\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})\\Dk^2.\n\\end{equation}\nThus, combining~\\eqref{huit} and~\\eqref{last} leads to~\\eqref{PhikIncrementBound}.\n\n$ $\\\\\n\\textbf{Case~1: $\\igk=1$}\\\\\n$ $\\\\\nUnlike the proof of~\\cite[Theorem~4.11, Case~1]{chen2018stochastic} where four subcases were considered, only three are analyzed next.\n\n%\\begin{itemize}\n%\\item[(i)] \n\\noindent \\textit{(Subcase 1i)}\nGood model ($\\iik=1$) and good estimates ($\\ijk=1$). By~\\eqref{zetaChoice}, \n\\[\\normii{\\QkTranspose\\nabla f(\\xk)}\\geq \\max\\accolade{\\eta_2+\\kappa_{eg}, \\kappa_h+\\kappa_{eg}, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}}+\\kappa_{eg}}\\dk,\\]\nimplying in particular~\\eqref{deltaLemmaB2} and hence a decrease in $f$ according to~\\eqref{LemmaB2improvement}. Moreover, by $(\\kef,\\keg;\\Qk)$-full linearity, \n\\[\\normii{\\gkhat}\\geq \\normii{\\QkTranspose\\nabla f(\\xk)}-\\kappa_{eg}\\dk\\geq \\max\\accolade{\\eta_2,\\kappa_h, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}}\\dk, \\]\nthus implying~\\eqref{deltaLemmaC}; and since $\\ef\\leq \\kappa_{ef}$ per assumptions of Lemma~\\ref{LemmaC}, the iteration is successful. Consequently,~\\eqref{un} together with~\\eqref{LemmaB2improvement} yields\n\\begin{equation}\\label{bTwo}\n\\begin{split}\n\t\\phi_{k+1}-\\phi_k &\\leq -\\nu C_1\\normii{\\QkTranspose\\nabla f(\\xk)}\\dk+(1-\\nu)(\\gamma^2-1)\\dk^2=:b_2\\\\\n\t&\\leq \\left(-\\nu C_1\\zeta+(1-\\nu)(\\gamma^2-1)\\right)\\dk^2<0,\n\\end{split}\n\\end{equation}\nwhere the last inequality follows from~\\eqref{nuChoiceScal}. Denoting by ${\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)$ the random variable with realizations $b_2<0$, with ${\\ubar{u}}_k$ defined in~\\eqref{ukRandomMeasurable}, we have from~\\eqref{bTwo} that\n\\begin{equation}\\label{bTwoRandom}\n\\igk\\iik\\ijk\\left(\\Phikun-\\Phik\\right)\\leq \\igk\\iik\\ijk{\\ubar{b}}_2({\\ubar{u}}_k,\\Dk).\n\\end{equation}\n%\\item[(ii)] \n\\noindent \\textit{(Subcase 1ii)}\nBad model ($\\iikc=1$) and good estimates ($\\ijk=1$). In this case, regardless of the iteration type (i.e., successful or unsuccessful), the change in $\\phi_k$ can always be bounded by using~\\eqref{un} and~\\eqref{LemmaEincrease} as follows:\n\\begin{equation}\\label{bThree}\n\\phi_{k+1}-\\phi_k\\leq\\nu C_3\\norme{\\QkTranspose\\nabla f(\\xk)}\\dk+(1-\\nu)(\\gamma^2-1)\\dk^2=:b_3.\n\\end{equation}\nDenoting by ${\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)$ the random variable with realizations $b_3>0$, from~\\eqref{bThree} we have that\n\\begin{equation}\\label{bThreeRandom2}\n\\igk\\iikc\\ijk\\left(\\Phikun-\\Phik\\right)\\leq \\igk\\iikc\\ijk{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)\\leq \\igk\\iikc{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk).\n\\end{equation}\n%\\item[(iii)] \n\\noindent \\textit{(Subcase 1iii)}\nBad estimates ($\\ijkc=1$). From \\textit{Subcase 1ii}, it always holds that \n\\begin{equation}\\label{bThreeRandom3}\n\\igk\\ijkc\\left(\\Phikun-\\Phik\\right)\\leq \\igk\\ijkc{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk).\n\\end{equation}\n%\\end{itemize}\n\nWith the subcases thus complete, since the random variables $\\igk, {\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)$, and ${\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)$ are $\\fqalgebra$-measurable thanks to the proof of Lemma~\\ref{indicatorMeasurable}, combining~\\eqref{bTwoRandom},~\\eqref{bThreeRandom2}, and~\\eqref{bThreeRandom3} and taking expectations with respect to $\\fqalgebra$, we have\n\\begin{equation}\\label{bTwoThree}\n\\begin{split}\n\\E{\\igk\\left(\\Phikun-\\Phik\\right)|\\fqalgebra}&\\leq \\igk{\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)\\E{\\iik\\ijk|\\fqalgebra}\\\\\n&+\\igk{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)\\left(\\E{\\iikc|\\fqalgebra}+\\E{\\ijkc|\\fqalgebra}\\right)\\\\\n&\\leq \\igk\\left(\\beta_f\\beta_m{\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)+{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)(1-\\beta_m+1-\\beta_f)\\right)=:\\igk {\\ubar{b}}_{2,3}({\\ubar{u}}_k,\\Dk),\n\\end{split}\n\\end{equation}\nwhere the last inequality follows from~\\eqref{ikQProb} and~\\eqref{prodBetafBetam} and the fact that ${\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)<0$ while ${\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)>0$.  Note that \n\\begin{eqnarray*}\n{\\ubar{b}}_{2,3}({\\ubar{u}}_k,\\Dk)&=&\\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\left[-\\beta_f\\beta_m C_1+(2-\\beta_f-\\beta_m)C_3\\right]\\\\\n& &+(1-\\nu)(\\gamma^2-1)(2-\\beta_f-\\beta_m+\\beta_f\\beta_m)\\Dk^2\\\\\n&\\leq& \\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\left[-\\beta_f\\beta_m C_1+(2-\\beta_f-\\beta_m)C_3\\right] + 2(1-\\nu)(\\gamma^2-1)\\Dk^2,\n\\end{eqnarray*}\nwhere the last inequality follows from $2-\\beta_f-\\beta_m+\\beta_f\\beta_m=1+(1-\\beta_f)(1-\\beta_m)\\leq 2$.\n\nFor $\\beta_f$ and $\\beta_m$ chosen according to the first condition in~\\eqref{probChoices}, and for $\\nu$ satisfying~\\eqref{nuChoiceScal}, the following holds:\n\\[\\beta_f\\beta_m C_1 - (2-\\beta_f-\\beta_m)C_3\\geq \\frac{1}{2}C_1\\geq 2\\frac{2(1-\\nu)(\\gamma^2-1)}{\\nu\\zeta}.\\]\nHence, since $\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\geq \\zeta\\Dk$, then \n\\begin{eqnarray*}\n{\\ubar{b}}_{2,3}({\\ubar{u}}_k,\\Dk)&\\leq& -\\frac{1}{2}\\left[\\beta_f\\beta_m C_1-(2-\\beta_f-\\beta_m)C_3\\right]\\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\\\\n&\\leq& -\\frac{1}{4}C_1\\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\leq -2(1-\\nu)(\\gamma^2-1)\\Dk^2,\n\\end{eqnarray*}\nwhich, together with~\\eqref{bTwoThree}, leads to~\\eqref{huit}.\n\n$ $\\\\\n\\textbf{Case~2: $\\igkc=1$}\n$ $\\\\ \nConsider the event $G_k:=\\accolade{\\normii{\\Gkhat}\\geq \\eta_2\\Dk}$, and note that since $\\Mhatk:={\\ubar{f}_k}+\\Gkhat^{\\top}\\sRandom +\\frac{1}{2}\\sRandom^{\\top}\\HkhatRandom\\sRandom$ is $\\fmqalgebra$-measurable by construction, then in particular $\\normii{\\Gkhat}$ and hence $\\igkhat, \\igkhatc$ are also $\\fmqalgebra$-measurable. Observe that if $\\igkhatc=1$, then the iteration is unsuccessful since $\\normii{\\gkhat}<\\eta_2\\dk$, leading to~\\eqref{bOne}. Hence, \n\\begin{equation}\\label{GkbarCase2}\n\\begin{split}\n\\E{\\igkc\\igkhatc(\\Phikun-\\Phik)|\\fmqalgebra}&\\leq -(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\E{\\igkc\\igkhatc\\Dk^2|\\fmqalgebra}\\\\\n&\\leq -\\frac{1}{2}\\igkc\\igkhatc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{split}\n\\end{equation}\nwhere the last inequality is due to the $\\fmqalgebra$-measurability of $\\igkc$ and $\\Dk$, given that $\\falgebra\\subset\\fqalgebra\\subset\\fmqalgebra$. \n\nInspired by~\\cite[Theorem~3]{blanchet2016convergence}, which improved on the proof of~\\cite[Theorem~4.11, Case~2]{chen2018stochastic} in which four subcases were considered, only two are analyzed next. The overall goal is to prove that\n\\begin{equation}\\label{GkCase2}\n\\E{\\igkc\\igkhat(\\Phikun-\\Phik)|\\fmqalgebra}\\leq -\\frac{1}{2}\\igkc\\igkhat\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{equation}\nwhich, combined with~\\eqref{GkbarCase2}, leads to \n\\begin{equation*}%\\label{GammaBarFmq}\n\\E{\\igkc(\\Phikun-\\Phik)|\\fmqalgebra}\\leq -\\frac{1}{2}\\igkc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2\n\\end{equation*}\nand consequently\n\\begin{equation*}%\\label{GammaBarFq}\n\\begin{split}\n\\E{\\igkc(\\Phikun-\\Phik)|\\fqalgebra}&=\\E{\\E{\\igkc(\\Phikun-\\Phik)|\\fmqalgebra}|\\fqalgebra}\\\\\n&\\leq \\E{-\\frac{1}{2}\\igkc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2 \\, |\\fqalgebra}=-\\frac{1}{2}\\igkc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{split}\n\\end{equation*}\n showing~\\eqref{last}, where the last equality follows from the $\\fqalgebra$-measurability of $\\igkc$ and $\\Dk$.  What remains to be shown is~\\eqref{GkCase2} as achieved next, assuming that $\\igkhat=1$.\n\n\\noindent \\textit{(Subcase 2i)}\n%\\begin{itemize}\n%\\item[(i)] \nGood estimates ($\\ijk=1$) and $\\igkhat=1$. While $f$ is decreased on successful iterations because  of good estimates thanks to Lemma~\\ref{LemmaD} and in this case~\\eqref{LemmaDimprovement} holds,  $\\dk$ is reduced on unsuccessful iterations, leading to~\\eqref{bOne}. However, combining~\\eqref{LemmaDimprovement} and~\\eqref{un} yields\n$\\phi_{k+1}-\\phi_k\\leq \\left[-\\nu C_2+(1-\\nu)(\\gamma^2-1)\\right]\\dk^2\\leq b_1,$\nwhere the last inequality is due to~\\eqref{nuChoiceScal}, which shows that~\\eqref{bOne} holds in any case. Consequently,\n\\begin{equation}\n\\label{GammakBarGkJk}\n\\igkc\\igkhat\\ijk(\\Phikun-\\Phik) \\leq -\\igkc\\igkhat\\ijk(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2.\n\\end{equation}\n\n%\\item[(ii)] \n\\noindent \\textit{(Subcase 2ii)}\nBad estimates ($\\ijkc=1$) and $\\igkhat=1$. In this case, a successful step can lead to an increase in $f$\naccording to~\\eqref{LemmaEincrease}, which combined with~\\eqref{un} yield~\\eqref{bThree} and hence\n\\begin{equation}\\label{GammakBarGkJkBar}\n\\igkc\\igkhat\\ijkc(\\Phikun-\\Phik) \\leq -\\igkc\\igkhat\\ijkc\\left(\\nu C_3\\zeta+(1-\\nu)(\\gamma^2-1)\\right)\\Dk^2.\n\\end{equation}\n%\\end{itemize}\n\nWith the subcases complete, recall that $\\E{\\ijk|\\fmqalgebra}\\geq\\beta_f$. Then  combining~\\eqref{GammakBarGkJk} and~\\eqref{GammakBarGkJkBar} and taking expectations with respect to~$\\fmqalgebra$ lead to\n\\begin{eqnarray*}%\\label{GkCase2Proved}\n\\E{\\igkc\\igkhat(\\Phikun-\\Phik)|\\fmqalgebra}&\\leq& \\igkc\\igkhat\\left(-\\beta_f(1-\\nu)(1-1/\\gamma^2)\\right.\\\\ \n& &\\left. +\\ (1-\\beta_f)\\left(\\nu C_3\\zeta+(1-\\nu)(\\gamma^2-1)\\right) \\right)\\Dk^2\\\\\n&\\leq& -\\frac{1}{2}\\igkc\\igkhat\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{eqnarray*}\nthus proving~\\eqref{GkCase2}, where the last inequality follows from the second condition in~\\eqref{probChoices}, and the proof is complete.\n\\end{proof}","start":102803,"end":113478}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":["bOne","un"],"declared_external":false,"source_only":false,"title":"成功与失败的势增量","source_units":[{"raw_tex":"\\begin{equation}\\label{un}\n\\phi_{k+1}-\\phi_k\\leq \\nu\\left(f(\\xkun)-f(\\xk)\\right)+(1-\\nu)(\\gamma^2-1)\\dk^2,\n\\end{equation}\nwhile on unsuccessful iterations, $\\xkun=\\xk$ and $\\dkun=\\gamma^{-1}\\dk$, in which case\n\\begin{equation}\\label{bOne}\n\\phi_{k+1}-\\phi_k\\leq(1-\\nu)\\left(\\frac{1}{\\gamma^2}-1\\right)\\dk^2=:b_1<0.\n\\end{equation}\n","start":103379,"end":103708}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["huit","last"],"declared_external":false,"source_only":false,"title":"按投影真梯度分区","source_units":[{"raw_tex":"\nRecall the event $\\Gamma_k=\\accolade{\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\geq\\zeta\\Dk}$ of Lemma~\\ref{indicatorMeasurable} with $\\zeta$ satisfying ~\\eqref{zetaChoice}. The proof considers two cases: $\\igk=1$ and $\\igkc=1$. Inspired by~\\cite[proof of Theorem~4.11]{chen2018stochastic}, Case~1 aims to show that\n\\begin{equation}\\label{huit}\n%\\begin{split}\n\\E{\\igk\\left(\\Phikun-\\Phik\\right)|\\fqalgebra}\\leq-2\\igk(1-\\nu)(\\gamma^2-1)\\Dk^2\n\\leq -\\frac{1}{2}\\igk\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})\\Dk^2, \n%\\end{split}\n\\end{equation}\nwhere the last inequality follows from $1-\\frac{1}{\\gamma^2}<\\gamma^2-1$ and $\\beta_f\\leq 1$. On the other hand, inspired by~\\cite[proof of Theorem~3]{blanchet2016convergence}, Case~2 shows that\n\\begin{equation}\\label{last}\n\\E{\\igkc\\left(\\Phikun-\\Phik\\right)|\\fqalgebra}\\leq-\\frac{1}{2}\\igkc\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})\\Dk^2.\n\\end{equation}\nThus, combining~\\eqref{huit} and~\\eqref{last} leads to~\\eqref{PhikIncrementBound}.\n\n$ $\\\\\n\\textbf{Case~1: $\\igk=1$}\\\\\n$ $\\\\\nUnlike the proof of~\\cite[Theorem~4.11, Case~1]{chen2018stochastic} where four subcases were considered, only three are analyzed next.\n\n%\\begin{itemize}\n%\\item[(i)] \n","start":103708,"end":104879}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":["bTwo","bTwoRandom"],"declared_external":false,"source_only":false,"title":"大梯度且模型估计良好","source_units":[{"raw_tex":"\\noindent \\textit{(Subcase 1i)}\nGood model ($\\iik=1$) and good estimates ($\\ijk=1$). By~\\eqref{zetaChoice}, \n\\[\\normii{\\QkTranspose\\nabla f(\\xk)}\\geq \\max\\accolade{\\eta_2+\\kappa_{eg}, \\kappa_h+\\kappa_{eg}, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}}+\\kappa_{eg}}\\dk,\\]\nimplying in particular~\\eqref{deltaLemmaB2} and hence a decrease in $f$ according to~\\eqref{LemmaB2improvement}. Moreover, by $(\\kef,\\keg;\\Qk)$-full linearity, \n\\[\\normii{\\gkhat}\\geq \\normii{\\QkTranspose\\nabla f(\\xk)}-\\kappa_{eg}\\dk\\geq \\max\\accolade{\\eta_2,\\kappa_h, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}}\\dk, \\]\nthus implying~\\eqref{deltaLemmaC}; and since $\\ef\\leq \\kappa_{ef}$ per assumptions of Lemma~\\ref{LemmaC}, the iteration is successful. Consequently,~\\eqref{un} together with~\\eqref{LemmaB2improvement} yields\n\\begin{equation}\\label{bTwo}\n\\begin{split}\n\t\\phi_{k+1}-\\phi_k &\\leq -\\nu C_1\\normii{\\QkTranspose\\nabla f(\\xk)}\\dk+(1-\\nu)(\\gamma^2-1)\\dk^2=:b_2\\\\\n\t&\\leq \\left(-\\nu C_1\\zeta+(1-\\nu)(\\gamma^2-1)\\right)\\dk^2<0,\n\\end{split}\n\\end{equation}\nwhere the last inequality follows from~\\eqref{nuChoiceScal}. Denoting by ${\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)$ the random variable with realizations $b_2<0$, with ${\\ubar{u}}_k$ defined in~\\eqref{ukRandomMeasurable}, we have from~\\eqref{bTwo} that\n\\begin{equation}\\label{bTwoRandom}\n\\igk\\iik\\ijk\\left(\\Phikun-\\Phik\\right)\\leq \\igk\\iik\\ijk{\\ubar{b}}_2({\\ubar{u}}_k,\\Dk).\n\\end{equation}\n%\\item[(ii)] \n","start":104879,"end":106295}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":["bThree","bThreeRandom2"],"declared_external":false,"source_only":false,"title":"大梯度且模型不良","source_units":[{"raw_tex":"\\noindent \\textit{(Subcase 1ii)}\nBad model ($\\iikc=1$) and good estimates ($\\ijk=1$). In this case, regardless of the iteration type (i.e., successful or unsuccessful), the change in $\\phi_k$ can always be bounded by using~\\eqref{un} and~\\eqref{LemmaEincrease} as follows:\n\\begin{equation}\\label{bThree}\n\\phi_{k+1}-\\phi_k\\leq\\nu C_3\\norme{\\QkTranspose\\nabla f(\\xk)}\\dk+(1-\\nu)(\\gamma^2-1)\\dk^2=:b_3.\n\\end{equation}\nDenoting by ${\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)$ the random variable with realizations $b_3>0$, from~\\eqref{bThree} we have that\n\\begin{equation}\\label{bThreeRandom2}\n\\igk\\iikc\\ijk\\left(\\Phikun-\\Phik\\right)\\leq \\igk\\iikc\\ijk{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)\\leq \\igk\\iikc{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk).\n\\end{equation}\n%\\item[(iii)] \n","start":106295,"end":107036}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":["bThreeRandom3"],"declared_external":false,"source_only":false,"title":"大梯度且估计不良","source_units":[{"raw_tex":"\\noindent \\textit{(Subcase 1iii)}\nBad estimates ($\\ijkc=1$). From \\textit{Subcase 1ii}, it always holds that \n\\begin{equation}\\label{bThreeRandom3}\n\\igk\\ijkc\\left(\\Phikun-\\Phik\\right)\\leq \\igk\\ijkc{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk).\n\\end{equation}\n%\\end{itemize}\n\n","start":107036,"end":107296}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":["bTwoThree"],"declared_external":false,"source_only":false,"title":"大梯度事件的概率加权","source_units":[{"raw_tex":"With the subcases thus complete, since the random variables $\\igk, {\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)$, and ${\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)$ are $\\fqalgebra$-measurable thanks to the proof of Lemma~\\ref{indicatorMeasurable}, combining~\\eqref{bTwoRandom},~\\eqref{bThreeRandom2}, and~\\eqref{bThreeRandom3} and taking expectations with respect to $\\fqalgebra$, we have\n\\begin{equation}\\label{bTwoThree}\n\\begin{split}\n\\E{\\igk\\left(\\Phikun-\\Phik\\right)|\\fqalgebra}&\\leq \\igk{\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)\\E{\\iik\\ijk|\\fqalgebra}\\\\\n&+\\igk{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)\\left(\\E{\\iikc|\\fqalgebra}+\\E{\\ijkc|\\fqalgebra}\\right)\\\\\n&\\leq \\igk\\left(\\beta_f\\beta_m{\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)+{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)(1-\\beta_m+1-\\beta_f)\\right)=:\\igk {\\ubar{b}}_{2,3}({\\ubar{u}}_k,\\Dk),\n\\end{split}\n\\end{equation}\nwhere the last inequality follows from~\\eqref{ikQProb} and~\\eqref{prodBetafBetam} and the fact that ${\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)<0$ while ${\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)>0$.  Note that \n\\begin{eqnarray*}\n{\\ubar{b}}_{2,3}({\\ubar{u}}_k,\\Dk)&=&\\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\left[-\\beta_f\\beta_m C_1+(2-\\beta_f-\\beta_m)C_3\\right]\\\\\n& &+(1-\\nu)(\\gamma^2-1)(2-\\beta_f-\\beta_m+\\beta_f\\beta_m)\\Dk^2\\\\\n&\\leq& \\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\left[-\\beta_f\\beta_m C_1+(2-\\beta_f-\\beta_m)C_3\\right] + 2(1-\\nu)(\\gamma^2-1)\\Dk^2,\n\\end{eqnarray*}\nwhere the last inequality follows from $2-\\beta_f-\\beta_m+\\beta_f\\beta_m=1+(1-\\beta_f)(1-\\beta_m)\\leq 2$.\n","start":107296,"end":108774}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"完成大梯度负漂移","source_units":[{"raw_tex":"\nFor $\\beta_f$ and $\\beta_m$ chosen according to the first condition in~\\eqref{probChoices}, and for $\\nu$ satisfying~\\eqref{nuChoiceScal}, the following holds:\n\\[\\beta_f\\beta_m C_1 - (2-\\beta_f-\\beta_m)C_3\\geq \\frac{1}{2}C_1\\geq 2\\frac{2(1-\\nu)(\\gamma^2-1)}{\\nu\\zeta}.\\]\nHence, since $\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\geq \\zeta\\Dk$, then \n\\begin{eqnarray*}\n{\\ubar{b}}_{2,3}({\\ubar{u}}_k,\\Dk)&\\leq& -\\frac{1}{2}\\left[\\beta_f\\beta_m C_1-(2-\\beta_f-\\beta_m)C_3\\right]\\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\\\\n&\\leq& -\\frac{1}{4}C_1\\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\leq -2(1-\\nu)(\\gamma^2-1)\\Dk^2,\n\\end{eqnarray*}\nwhich, together with~\\eqref{bTwoThree}, leads to~\\eqref{huit}.\n\n","start":108774,"end":109479}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":["GkbarCase2"],"declared_external":false,"source_only":false,"title":"小梯度下的模型梯度门槛","source_units":[{"raw_tex":"$ $\\\\\n\\textbf{Case~2: $\\igkc=1$}\n$ $\\\\ \nConsider the event $G_k:=\\accolade{\\normii{\\Gkhat}\\geq \\eta_2\\Dk}$, and note that since $\\Mhatk:={\\ubar{f}_k}+\\Gkhat^{\\top}\\sRandom +\\frac{1}{2}\\sRandom^{\\top}\\HkhatRandom\\sRandom$ is $\\fmqalgebra$-measurable by construction, then in particular $\\normii{\\Gkhat}$ and hence $\\igkhat, \\igkhatc$ are also $\\fmqalgebra$-measurable. Observe that if $\\igkhatc=1$, then the iteration is unsuccessful since $\\normii{\\gkhat}<\\eta_2\\dk$, leading to~\\eqref{bOne}. Hence, \n\\begin{equation}\\label{GkbarCase2}\n\\begin{split}\n\\E{\\igkc\\igkhatc(\\Phikun-\\Phik)|\\fmqalgebra}&\\leq -(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\E{\\igkc\\igkhatc\\Dk^2|\\fmqalgebra}\\\\\n&\\leq -\\frac{1}{2}\\igkc\\igkhatc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{split}\n\\end{equation}\nwhere the last inequality is due to the $\\fmqalgebra$-measurability of $\\igkc$ and $\\Dk$, given that $\\falgebra\\subset\\fqalgebra\\subset\\fmqalgebra$. \n\n","start":109479,"end":110420}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":["GkCase2"],"declared_external":false,"source_only":false,"title":"组合小梯度分支并降滤过","source_units":[{"raw_tex":"Inspired by~\\cite[Theorem~3]{blanchet2016convergence}, which improved on the proof of~\\cite[Theorem~4.11, Case~2]{chen2018stochastic} in which four subcases were considered, only two are analyzed next. The overall goal is to prove that\n\\begin{equation}\\label{GkCase2}\n\\E{\\igkc\\igkhat(\\Phikun-\\Phik)|\\fmqalgebra}\\leq -\\frac{1}{2}\\igkc\\igkhat\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{equation}\nwhich, combined with~\\eqref{GkbarCase2}, leads to \n\\begin{equation*}%\\label{GammaBarFmq}\n\\E{\\igkc(\\Phikun-\\Phik)|\\fmqalgebra}\\leq -\\frac{1}{2}\\igkc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2\n\\end{equation*}\nand consequently\n\\begin{equation*}%\\label{GammaBarFq}\n\\begin{split}\n\\E{\\igkc(\\Phikun-\\Phik)|\\fqalgebra}&=\\E{\\E{\\igkc(\\Phikun-\\Phik)|\\fmqalgebra}|\\fqalgebra}\\\\\n&\\leq \\E{-\\frac{1}{2}\\igkc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2 \\, |\\fqalgebra}=-\\frac{1}{2}\\igkc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{split}\n\\end{equation*}\n showing~\\eqref{last}, where the last equality follows from the $\\fqalgebra$-measurability of $\\igkc$ and $\\Dk$.  What remains to be shown is~\\eqref{GkCase2} as achieved next, assuming that $\\igkhat=1$.\n\n","start":110420,"end":111602}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":["GammakBarGkJk"],"declared_external":false,"source_only":false,"title":"小梯度且估计良好","source_units":[{"raw_tex":"\\noindent \\textit{(Subcase 2i)}\n%\\begin{itemize}\n%\\item[(i)] \nGood estimates ($\\ijk=1$) and $\\igkhat=1$. While $f$ is decreased on successful iterations because  of good estimates thanks to Lemma~\\ref{LemmaD} and in this case~\\eqref{LemmaDimprovement} holds,  $\\dk$ is reduced on unsuccessful iterations, leading to~\\eqref{bOne}. However, combining~\\eqref{LemmaDimprovement} and~\\eqref{un} yields\n$\\phi_{k+1}-\\phi_k\\leq \\left[-\\nu C_2+(1-\\nu)(\\gamma^2-1)\\right]\\dk^2\\leq b_1,$\nwhere the last inequality is due to~\\eqref{nuChoiceScal}, which shows that~\\eqref{bOne} holds in any case. Consequently,\n\\begin{equation}\n\\label{GammakBarGkJk}\n\\igkc\\igkhat\\ijk(\\Phikun-\\Phik) \\leq -\\igkc\\igkhat\\ijk(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2.\n\\end{equation}\n\n%\\item[(ii)] \n","start":111602,"end":112370}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":["GammakBarGkJkBar"],"declared_external":false,"source_only":false,"title":"小梯度且估计不良","source_units":[{"raw_tex":"\\noindent \\textit{(Subcase 2ii)}\nBad estimates ($\\ijkc=1$) and $\\igkhat=1$. In this case, a successful step can lead to an increase in $f$\naccording to~\\eqref{LemmaEincrease}, which combined with~\\eqref{un} yield~\\eqref{bThree} and hence\n\\begin{equation}\\label{GammakBarGkJkBar}\n\\igkc\\igkhat\\ijkc(\\Phikun-\\Phik) \\leq -\\igkc\\igkhat\\ijkc\\left(\\nu C_3\\zeta+(1-\\nu)(\\gamma^2-1)\\right)\\Dk^2.\n\\end{equation}\n%\\end{itemize}\n\n","start":112370,"end":112788}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"完成模型门槛分支漂移","source_units":[{"raw_tex":"With the subcases complete, recall that $\\E{\\ijk|\\fmqalgebra}\\geq\\beta_f$. Then  combining~\\eqref{GammakBarGkJk} and~\\eqref{GammakBarGkJkBar} and taking expectations with respect to~$\\fmqalgebra$ lead to\n\\begin{eqnarray*}%\\label{GkCase2Proved}\n\\E{\\igkc\\igkhat(\\Phikun-\\Phik)|\\fmqalgebra}&\\leq& \\igkc\\igkhat\\left(-\\beta_f(1-\\nu)(1-1/\\gamma^2)\\right.\\\\ \n& &\\left. +\\ (1-\\beta_f)\\left(\\nu C_3\\zeta+(1-\\nu)(\\gamma^2-1)\\right) \\right)\\Dk^2\\\\\n&\\leq& -\\frac{1}{2}\\igkc\\igkhat\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{eqnarray*}\nthus proving~\\eqref{GkCase2}, where the last inequality follows from the second condition in~\\eqref{probChoices}, and the proof is complete.\n\\end{proof}","start":112788,"end":113478}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"由非负漂移求和","source_units":[{"raw_tex":"\\begin{proof}\nThe proof is inspired by those of~\\cite[Theorem~3]{blanchet2016convergence} and~\\cite[Theorem~4.11]{chen2018stochastic}. \nThe overall goal is to prove~\\eqref{PhikIncrementBound}. Indeed noticing that $\\Phik>0$ and then taking expectations on both sides of the inequality in~\\eqref{PhikIncrementBound}, we obtain~\\eqref{seriesResult} (see, e.g.,~\\cite[Theorem~3]{dzahini2020expected} for details). Let $\\phi_k$ denote realizations of $\\Phik$, and recall that on all successful iterations, $\\xkun=\\xk+\\Qk\\sk$ and $\\dkun=\\min\\accolade{\\gamma\\dk,\\dmaxx}$,  implying\n","start":102803,"end":103379}]},{"id":"S067","semantic_type":"theorem","kind":"theorem","labels":["liminfTheorem"],"declared_external":false,"source_only":false,"title":"梯度下极限收敛","source_units":[{"raw_tex":"\\begin{theorem}\\label{liminfTheorem}\nLet all the assumptions made in Theorem~\\ref{zerothOrderScal} hold. Assume further that Assumption~\\ref{wellAlignedAssumption} holds with $0\\leq\\beta_Q<1-\\frac{1}{2\\beta_f\\beta_m}$. Then, the sequence $\\accolade{\\Xk}_{k\\in\\N}$ of random iterates generated by Algorithm~\\ref{algoStoScalTR} satisfies\n\\[\\underset{k\\to\\infty}{\\liminf}\\normii{\\nabla f(\\Xk)}=0\\quad\\mbox{ almost surely.}\\]\n\\end{theorem}","start":114364,"end":114799}]},{"id":"S068","semantic_type":"assumption","kind":"assumption","labels":["liminfTheorem"],"declared_external":false,"source_only":false,"title":"严格正偏置的参数限制","source_units":[{"raw_tex":"Let all the assumptions made in Theorem~\\ref{zerothOrderScal} hold. Assume further that Assumption~\\ref{wellAlignedAssumption} holds with $0\\leq\\beta_Q<1-\\frac{1}{2\\beta_f\\beta_m}$. Then, the sequence $\\accolade{\\Xk}_{k\\in\\N}$ of random iterates generated by Algorithm~\\ref{algoStoScalTR} satisfies\n","start":114401,"end":114700}]},{"id":"S069","semantic_type":"proof","kind":"proof","labels":["bChoice","ineq4","ineq5"],"declared_external":false,"source_only":false,"title":"下极限定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nThe result is proved by contradiction conditioned on the almost sure event $E_0:=\\accolade{\\Dk\\to 0}$ (thanks to Theorem~\\ref{zerothOrderScal}), inspired by the proof of~\\cite[Theorem~4.16]{chen2018stochastic} and also using ideas from~\\cite{audet2019stomads,dzahini2020constrained}. Assume that with nonzero probability there exists a random variable ${\\ubar{\\varepsilon}}'>0$ such that \n$\\normii{\\nabla f(\\Xk)}\\geq {\\ubar{\\varepsilon}}' \\mbox{ for all } k\\in\\N.$\n\nLet $\\accolade{\\xk}, \\accolade{\\dk}$, and ${{\\varepsilon}}'$ be realizations of $\\accolade{\\Xk}, \\accolade{\\Dk}$, and ${\\ubar{\\varepsilon}}'$, respectively, for which $\\normii{\\nabla f(\\xk)}\\geq {{\\varepsilon}}'$ for all $k$. Since $\\dk\\to 0$, there exists $k_0\\in\\N$ such that\n\\begin{equation}\\label{bChoice}\n\\dk<b:=\\alpha_Q\\cdot\\min\\accolade{\\frac{{{\\varepsilon}}'}{2(\\kappa_h+\\kappa_{eg})}, \\frac{{{\\varepsilon}}'}{2(\\eta_2+\\kappa_{eg})}, \\frac{{{\\varepsilon}}'}{\\frac{16\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}+2\\kappa_{eg}}, \\frac{\\dmaxx}{\\gamma}}\n\\end{equation}\nfor all $k\\geq k_0$.\nLet $\\ubar{r}_k$ be the random variable with realizations $r_k=\\log_{\\gamma}\\left(\\frac{\\dk}{b}\\right)$. Then $r_k<0$ for all $k\\geq k_0$. The main idea is to show that such realizations occur only with probability zero, leading to a contradiction. \n\nTo prove that $\\accolade{\\ubar{r}_k}$ is a submartingale, recall the events $I^Q_k$ and $J^Q_k$ satisfying~\\eqref{otherProbIneqs} and $\\mathcal{A}_k$ satisfying~\\eqref{Akineq}.\n%$\\prob{\\mathcal{A}_k|\\falgebra}\\geq 1-\\beta_Q$, \n%$\\prob{I^Q_k|\\falgebra}\\geq\\beta_m$ and  $\\prob{J^Q_k|\\falgebra}\\geq\\beta_f$, \n%where $\\mathcal{F}_{k-1}^Q :=\\sigma\\left(\\mathcal{A}_0,\\mathcal{A}_1,\\dots,\\mathcal{A}_{k-1}\\right)\\cap \\sigma\\left(I^Q_0,I^Q_1,\\dots,I^Q_{k-1}\\right)\\cap \\sigma\\left(J^Q_0,J^Q_1,\\dots,J^Q_{k-1}\\right)$. \nConsider some iteration $k\\geq k_0$ for which $\\mathcal{A}_k$, $I^Q_k$, and $J^Q_k$ all occur, which happens with probability at least $\\beta_f\\beta_m(1-\\beta_Q)$ conditioned on $\\falgebra$, as was shown in~\\eqref{prodBetaIneq}. It follows from~\\eqref{bChoice} that \n\\begin{equation}\\label{ineq4}\n\\normii{\\nabla f(\\xk)}\\geq \\frac{1}{\\alpha_Q}\\max\\accolade{\\kappa_h+\\kappa_{eg}, \\eta_2+\\kappa_{eg}, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}+\\kappa_{eg}}\\dk,\n\\end{equation}\nwhich, together with the $(\\kef,\\keg;\\Qk)$-full linearity and $\\alpha_Q$-well alignment, yield\n\\begin{equation}\\label{ineq5}\n%\\begin{split}\n\\normii{\\gkhat}\\geq\\normii{\\QkTranspose\\nabla f(\\xk)}-\\kappa_{eg}\\dk\\geq \\alpha_Q \\normii{\\nabla f(\\xk)}-\\kappa_{eg}\\dk\n\\geq \\max\\accolade{\\kappa_h, \\eta_2, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}}\\dk.\n%\\end{split}\n\\end{equation}\nThus, the $k$th iteration is successful according to Lemma~\\ref{LemmaC}. Hence, $\\dkun=\\gamma\\dk$, and hence $r_{k+1}=r_k+1$. For all other outcomes of $\\mathcal{A}_k$, $I^Q_k$, and $J^Q_k$, which occur with a total probability of at most $1-\\beta_f\\beta_m(1-\\beta_Q)$, it always holds that $\\dkun\\geq\\gamma^{-1}\\dk$, which implies that $r_{k+1}\\geq r_k-1$. Hence, \n\\begin{eqnarray*}\n\\E{\\mathds{1}_{\\mathcal{A}_k\\cap I^Q_k\\cap J^Q_k} \\left(\\ubar{r}_{k+1}-\\ubar{r}_k\\right)|\\falgebra}&\\geq& \\beta_f\\beta_m(1-\\beta_Q)\\quad \\quad\\mbox{and}\\\\\n\\E{\\mathds{1}_{\\overline{\\mathcal{A}_k\\cap I^Q_k\\cap J^Q_k}} \\left(\\ubar{r}_{k+1}-\\ubar{r}_k\\right)|\\falgebra}&\\geq& \\beta_f\\beta_m(1-\\beta_Q)-1, \n\\end{eqnarray*}\nwhich shows that $\\accolade{\\ubar{r}_k}$ is a submartingale if $0\\leq\\beta_Q<1-\\frac{1}{2\\beta_f\\beta_m}$.\n\nUsing the random walk defined in~\\eqref{randomWalk}, one  can easily  show (following, e.g.,~\\cite[Theorem~4]{audet2019stomads}, \\cite[Theorem~4.16]{chen2018stochastic} and~\\cite[Theorem~3.6 and Lemmas~4.7 and~4.12]{dzahini2020constrained}) that $\\accolade{\\ubar{w}_k}$ is also a submartingale with bounded increments and, as such, cannot converge to a finite value, and therefore $\\prob{\\underset{k\\to\\infty}{\\limsup}\\ \\ubar{w}_k=\\infty}=1$ (thanks to~\\cite[Theorem~4.4]{chen2018stochastic}). However, since $r_k-r_{k_0}\\geq w_k-w_{k_0}$ by construction,  the sequence of realizations~$r_k$ such that $r_k<0\\ \\forall k\\ \\geq k_0$ occurs with probability zero, leading to a contradiction, which achieves the proof.\n\\end{proof}","start":114801,"end":119014}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"在半径趋零事件上反设","source_units":[{"raw_tex":"\\begin{proof}\nThe result is proved by contradiction conditioned on the almost sure event $E_0:=\\accolade{\\Dk\\to 0}$ (thanks to Theorem~\\ref{zerothOrderScal}), inspired by the proof of~\\cite[Theorem~4.16]{chen2018stochastic} and also using ideas from~\\cite{audet2019stomads,dzahini2020constrained}. Assume that with nonzero probability there exists a random variable ${\\ubar{\\varepsilon}}'>0$ such that \n$\\normii{\\nabla f(\\Xk)}\\geq {\\ubar{\\varepsilon}}' \\mbox{ for all } k\\in\\N.$\n\nLet $\\accolade{\\xk}, \\accolade{\\dk}$, and ${{\\varepsilon}}'$ be realizations of $\\accolade{\\Xk}, \\accolade{\\Dk}$, and ${\\ubar{\\varepsilon}}'$, respectively, for which $\\normii{\\nabla f(\\xk)}\\geq {{\\varepsilon}}'$ for all $k$. Since $\\dk\\to 0$, there exists $k_0\\in\\N$ such that\n","start":114801,"end":115559}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":["bChoice"],"declared_external":false,"source_only":false,"title":"构造对数半径门槛","source_units":[{"raw_tex":"\\begin{equation}\\label{bChoice}\n\\dk<b:=\\alpha_Q\\cdot\\min\\accolade{\\frac{{{\\varepsilon}}'}{2(\\kappa_h+\\kappa_{eg})}, \\frac{{{\\varepsilon}}'}{2(\\eta_2+\\kappa_{eg})}, \\frac{{{\\varepsilon}}'}{\\frac{16\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}+2\\kappa_{eg}}, \\frac{\\dmaxx}{\\gamma}}\n\\end{equation}\nfor all $k\\geq k_0$.\nLet $\\ubar{r}_k$ be the random variable with realizations $r_k=\\log_{\\gamma}\\left(\\frac{\\dk}{b}\\right)$. Then $r_k<0$ for all $k\\geq k_0$. The main idea is to show that such realizations occur only with probability zero, leading to a contradiction. \n\n","start":115559,"end":116117}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":["ineq4","ineq5"],"declared_external":false,"source_only":false,"title":"对齐准确小半径必成功","source_units":[{"raw_tex":"To prove that $\\accolade{\\ubar{r}_k}$ is a submartingale, recall the events $I^Q_k$ and $J^Q_k$ satisfying~\\eqref{otherProbIneqs} and $\\mathcal{A}_k$ satisfying~\\eqref{Akineq}.\n%$\\prob{\\mathcal{A}_k|\\falgebra}\\geq 1-\\beta_Q$, \n%$\\prob{I^Q_k|\\falgebra}\\geq\\beta_m$ and  $\\prob{J^Q_k|\\falgebra}\\geq\\beta_f$, \n%where $\\mathcal{F}_{k-1}^Q :=\\sigma\\left(\\mathcal{A}_0,\\mathcal{A}_1,\\dots,\\mathcal{A}_{k-1}\\right)\\cap \\sigma\\left(I^Q_0,I^Q_1,\\dots,I^Q_{k-1}\\right)\\cap \\sigma\\left(J^Q_0,J^Q_1,\\dots,J^Q_{k-1}\\right)$. \nConsider some iteration $k\\geq k_0$ for which $\\mathcal{A}_k$, $I^Q_k$, and $J^Q_k$ all occur, which happens with probability at least $\\beta_f\\beta_m(1-\\beta_Q)$ conditioned on $\\falgebra$, as was shown in~\\eqref{prodBetaIneq}. It follows from~\\eqref{bChoice} that \n\\begin{equation}\\label{ineq4}\n\\normii{\\nabla f(\\xk)}\\geq \\frac{1}{\\alpha_Q}\\max\\accolade{\\kappa_h+\\kappa_{eg}, \\eta_2+\\kappa_{eg}, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}+\\kappa_{eg}}\\dk,\n\\end{equation}\nwhich, together with the $(\\kef,\\keg;\\Qk)$-full linearity and $\\alpha_Q$-well alignment, yield\n\\begin{equation}\\label{ineq5}\n%\\begin{split}\n\\normii{\\gkhat}\\geq\\normii{\\QkTranspose\\nabla f(\\xk)}-\\kappa_{eg}\\dk\\geq \\alpha_Q \\normii{\\nabla f(\\xk)}-\\kappa_{eg}\\dk\n\\geq \\max\\accolade{\\kappa_h, \\eta_2, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}}\\dk.\n%\\end{split}\n\\end{equation}\nThus, the $k$th iteration is successful according to Lemma~\\ref{LemmaC}. Hence, $\\dkun=\\gamma\\dk$, and hence $r_{k+1}=r_k+1$. For all other outcomes of $\\mathcal{A}_k$, $I^Q_k$, and $J^Q_k$, which occur with a total probability of at most $1-\\beta_f\\beta_m(1-\\beta_Q)$, it always holds that $\\dkun\\geq\\gamma^{-1}\\dk$, which implies that $r_{k+1}\\geq r_k-1$. Hence, \n","start":116117,"end":117844}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"对数半径的偏置计算","source_units":[{"raw_tex":"\\begin{eqnarray*}\n\\E{\\mathds{1}_{\\mathcal{A}_k\\cap I^Q_k\\cap J^Q_k} \\left(\\ubar{r}_{k+1}-\\ubar{r}_k\\right)|\\falgebra}&\\geq& \\beta_f\\beta_m(1-\\beta_Q)\\quad \\quad\\mbox{and}\\\\\n\\E{\\mathds{1}_{\\overline{\\mathcal{A}_k\\cap I^Q_k\\cap J^Q_k}} \\left(\\ubar{r}_{k+1}-\\ubar{r}_k\\right)|\\falgebra}&\\geq& \\beta_f\\beta_m(1-\\beta_Q)-1, \n\\end{eqnarray*}\nwhich shows that $\\accolade{\\ubar{r}_k}$ is a submartingale if $0\\leq\\beta_Q<1-\\frac{1}{2\\beta_f\\beta_m}$.\n","start":117844,"end":118287}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"随机游走上极限排除反设","source_units":[{"raw_tex":"\nUsing the random walk defined in~\\eqref{randomWalk}, one  can easily  show (following, e.g.,~\\cite[Theorem~4]{audet2019stomads}, \\cite[Theorem~4.16]{chen2018stochastic} and~\\cite[Theorem~3.6 and Lemmas~4.7 and~4.12]{dzahini2020constrained}) that $\\accolade{\\ubar{w}_k}$ is also a submartingale with bounded increments and, as such, cannot converge to a finite value, and therefore $\\prob{\\underset{k\\to\\infty}{\\limsup}\\ \\ubar{w}_k=\\infty}=1$ (thanks to~\\cite[Theorem~4.4]{chen2018stochastic}). However, since $r_k-r_{k_0}\\geq w_k-w_{k_0}$ by construction,  the sequence of realizations~$r_k$ such that $r_k<0\\ \\forall k\\ \\geq k_0$ occurs with probability zero, leading to a contradiction, which achieves the proof.\n\\end{proof}","start":118287,"end":119014}]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件列出的 chen2018stochastic Theorem 4.4，audet2019stomads Theorem 4，dzahini2020constrained Theorem 3.6 and Lemmas 4.7 and 4.12；L1566 另比较 chen2018stochastic Theorem 4.16。"],"source_only":false,"title":"外部随机游走论证","source_units":[]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件列出的 billingsley1995ThirdEdition Chapter 13；L1325–L1326、L1335、L1341 具体引用 Theorems 13.1–13.4 和 Equation (13.3)。"],"source_only":false,"title":"可测函数闭包与极限定理","source_units":[]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件列出的 Fundamental Theorem of Calculus and Lipschitz-gradient remainder bound，AuHa2017 Lemma 9.3；L1806 明确调用。"],"source_only":false,"title":"微积分基本定理与余项界","source_units":[]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原先决条件列出的 standard nonnegative drift summation argument，参考 dzahini2020expected Theorem 3；L1378 明确转引。"],"source_only":false,"title":"非负过程漂移求和规则","source_units":[]},{"id":"S079","semantic_type":"lemma","kind":"lemma","labels":["smallLemma"],"declared_external":false,"source_only":false,"title":"算法半径满足更新下界","source_units":[{"raw_tex":"\\begin{lemma}\\label{smallLemma}\nLet all the assumptions made in Theorem~\\ref{liminfTheorem} hold. Then~\\eqref{dynamicsRenewal} is satisfied for\\\\ $\\Wtildek=2\\left(\\iak\\iik\\ijk-\\frac{1}{2} \\right)$, with $\\lambda=\\ln(\\gamma)$ and some fixed $q\\in (\\tilde{\\beta}, 1)$ with $\\tilde{\\beta}=\\beta_f\\beta_m(1-\\beta_Q)$.\n\\end{lemma}","start":125901,"end":126226}]},{"id":"S080","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"半径动力学的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nThe result is proved by suitably adapting the proof of~\\cite[Lemma~7]{blanchet2016convergence} as was done in~\\cite{dzahini2020expected}. First, we notice that~\\eqref{dynamicsRenewal} trivially holds when $\\itksup=0$. Next, we show that when $\\itksup=1$, then \n\\[\\Dkun\\geq \\min\\accolade{\\depsilon, \\min\\accolade{\\dmaxx,\\gamma\\Dk}\\iak\\iik\\ijk+\\gamma^{-1}\\Dk\\left(1-\\iak\\iik\\ijk\\right)}. \\]\nRecall that $\\dk=\\gamma^{i_k}\\depsilon$ for some integer $i_k$. Therefore, if $\\dk>\\depsilon$, then $\\dk\\geq \\gamma\\depsilon$, which implies that $\\dkun\\geq\\gamma^{-1}\\dk\\geq\\depsilon$. Now assume that $\\dk\\leq \\depsilon$. Since $\\tauEpsilon>k$, we have $\\normii{\\nabla f(\\xk)}>\\epsilon=\\xi\\depsilon\\geq\\xi\\dk$. Thus, it follows from the definition of~$\\xi$ that~\\eqref{ineq4} holds. If $\\iak=1$, $\\iik=1$, and $\\ijk=1$, then~\\eqref{ineq5} holds, and consequently the $k$th iteration is successful, as was  explained in the proof of Theorem~\\ref{liminfTheorem}. Hence, $\\xkun=\\xk+\\Qk\\sk$ and $\\dkun=\\min\\accolade{\\dmaxx,\\gamma\\dk}$. If $\\iak\\iik\\ijk=0$, then it always holds that $\\dkun\\geq \\gamma^{-1}\\dk$. The proof is completed by  observing that $\\prob{\\iak\\iik\\ijk=1}=q$ for some fixed $q\\geq \\tilde{\\beta}$.\n\\end{proof}","start":126228,"end":127456}]},{"id":"S081","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"停止后情形与目标不等式","source_units":[{"raw_tex":"\\begin{proof}\nThe result is proved by suitably adapting the proof of~\\cite[Lemma~7]{blanchet2016convergence} as was done in~\\cite{dzahini2020expected}. First, we notice that~\\eqref{dynamicsRenewal} trivially holds when $\\itksup=0$. Next, we show that when $\\itksup=1$, then \n\\[\\Dkun\\geq \\min\\accolade{\\depsilon, \\min\\accolade{\\dmaxx,\\gamma\\Dk}\\iak\\iik\\ijk+\\gamma^{-1}\\Dk\\left(1-\\iak\\iik\\ijk\\right)}. \\]\n","start":126228,"end":126631}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"阈值两侧的半径控制","source_units":[{"raw_tex":"Recall that $\\dk=\\gamma^{i_k}\\depsilon$ for some integer $i_k$. Therefore, if $\\dk>\\depsilon$, then $\\dk\\geq \\gamma\\depsilon$, which implies that $\\dkun\\geq\\gamma^{-1}\\dk\\geq\\depsilon$. Now assume that $\\dk\\leq \\depsilon$. Since $\\tauEpsilon>k$, we have $\\normii{\\nabla f(\\xk)}>\\epsilon=\\xi\\depsilon\\geq\\xi\\dk$. Thus, it follows from the definition of~$\\xi$ that~\\eqref{ineq4} holds. If $\\iak=1$, $\\iik=1$, and $\\ijk=1$, then~\\eqref{ineq5} holds, and consequently the $k$th iteration is successful, as was  explained in the proof of Theorem~\\ref{liminfTheorem}. Hence, $\\xkun=\\xk+\\Qk\\sk$ and $\\dkun=\\min\\accolade{\\dmaxx,\\gamma\\dk}$. If $\\iak\\iik\\ijk=0$, then it always holds that $\\dkun\\geq \\gamma^{-1}\\dk$. The proof is completed by  observing that $\\prob{\\iak\\iik\\ijk=1}=q$ for some fixed $q\\geq \\tilde{\\beta}$.\n","start":126631,"end":127445}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"固定概率的未证断言","source_units":[{"raw_tex":"Recall that $\\dk=\\gamma^{i_k}\\depsilon$ for some integer $i_k$. Therefore, if $\\dk>\\depsilon$, then $\\dk\\geq \\gamma\\depsilon$, which implies that $\\dkun\\geq\\gamma^{-1}\\dk\\geq\\depsilon$. Now assume that $\\dk\\leq \\depsilon$. Since $\\tauEpsilon>k$, we have $\\normii{\\nabla f(\\xk)}>\\epsilon=\\xi\\depsilon\\geq\\xi\\dk$. Thus, it follows from the definition of~$\\xi$ that~\\eqref{ineq4} holds. If $\\iak=1$, $\\iik=1$, and $\\ijk=1$, then~\\eqref{ineq5} holds, and consequently the $k$th iteration is successful, as was  explained in the proof of Theorem~\\ref{liminfTheorem}. Hence, $\\xkun=\\xk+\\Qk\\sk$ and $\\dkun=\\min\\accolade{\\dmaxx,\\gamma\\dk}$. If $\\iak\\iik\\ijk=0$, then it always holds that $\\dkun\\geq \\gamma^{-1}\\dk$. The proof is completed by  observing that $\\prob{\\iak\\iik\\ijk=1}=q$ for some fixed $q\\geq \\tilde{\\beta}$.\n","start":126631,"end":127445}]},{"id":"S084","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"复杂度定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nRecall the choice of $q$ in Lemma~\\ref{smallLemma}, and pick some $\\tilde{\\xi}\\geq\\hat{\\xi}$ such that $\\depsilon=\\frac{\\epsilon}{\\tilde{\\xi}}=\\gamma^{i}\\delta_0$ for some integer $i\\leq 0$ (without loss of generality), as discussed above. The proof follows by employing Theorem~\\ref{theorComplexity1} with $h(t)=t^2$, which yields\n\\[\\E{\\tauEpsilon}\\leq \\frac{q}{2q-1}\\cdot\\frac{\\PhiZero\\tilde{\\xi}^2}{\\varrho \\epsilon^2}+1\\leq \\frac{\\tilde{\\beta}}{2\\tilde{\\beta}-1}\\cdot\\frac{\\PhiZero\\tilde{\\xi}^2}{\\varrho \\epsilon^2}+1.\\]\n\\end{proof}","start":127969,"end":128519}]},{"id":"S085","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"选择格点兼容阈值","source_units":[{"raw_tex":"Recall the choice of $q$ in Lemma~\\ref{smallLemma}, and pick some $\\tilde{\\xi}\\geq\\hat{\\xi}$ such that $\\depsilon=\\frac{\\epsilon}{\\tilde{\\xi}}=\\gamma^{i}\\delta_0$ for some integer $i\\leq 0$ (without loss of generality), as discussed above. The proof follows by employing Theorem~\\ref{theorComplexity1} with $h(t)=t^2$, which yields\n","start":127983,"end":128315}]},{"id":"S086","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"应用期望停止时间定理","source_units":[{"raw_tex":"Recall the choice of $q$ in Lemma~\\ref{smallLemma}, and pick some $\\tilde{\\xi}\\geq\\hat{\\xi}$ such that $\\depsilon=\\frac{\\epsilon}{\\tilde{\\xi}}=\\gamma^{i}\\delta_0$ for some integer $i\\leq 0$ (without loss of generality), as discussed above. The proof follows by employing Theorem~\\ref{theorComplexity1} with $h(t)=t^2$, which yields\n\\[\\E{\\tauEpsilon}\\leq \\frac{q}{2q-1}\\cdot\\frac{\\PhiZero\\tilde{\\xi}^2}{\\varrho \\epsilon^2}+1\\leq \\frac{\\tilde{\\beta}}{2\\tilde{\\beta}-1}\\cdot\\frac{\\PhiZero\\tilde{\\xi}^2}{\\varrho \\epsilon^2}+1.\\]\n","start":127983,"end":128508}]},{"id":"S087","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"以联合下界替换概率因子","source_units":[{"raw_tex":"\\[\\E{\\tauEpsilon}\\leq \\frac{q}{2q-1}\\cdot\\frac{\\PhiZero\\tilde{\\xi}^2}{\\varrho \\epsilon^2}+1\\leq \\frac{\\tilde{\\beta}}{2\\tilde{\\beta}-1}\\cdot\\frac{\\PhiZero\\tilde{\\xi}^2}{\\varrho \\epsilon^2}+1.\\]\n","start":128315,"end":128508}]},{"id":"S088","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"模型梯度下降的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nThe proof is similar to that of~\\cite[Lemma~4.5]{chen2018stochastic} and is not detailed here. It follows from~\\eqref{CauchyDecrease} together with the inequalities $\\kappa_h\\geq \\max\\accolade{\\norme{\\Hkhat}, 1}$ and $\\normii{\\gkhat}\\geq\\kappa_h\\dk$ that  $\\mhatk(\\bm{0})-\\mhatk(\\sk)\\geq\\frac{\\kappa_{fcd}}{2}\\normii{\\gkhat}\\dk.$ Since the model is $(\\kef,\\keg;\\Qk)$-fully linear, then \n$f(\\xk+\\Qk\\sk)-f(\\xk)\\leq 2\\kappa_{ef}\\dk^2+\\mhatk(\\sk)-\\mhatk(\\bm{0})\\leq -\\frac{\\kappa_{fcd}}{4}\\normii{\\gkhat}\\dk,$\nwhere the last inequality follows from the fact that $\\dk\\leq \\frac{\\kappa_{fcd}}{8\\kappa_{ef}}\\normii{\\gkhat}$.\n\\end{proof}","start":141109,"end":141753}]},{"id":"S089","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"柯西下降抵消模型误差","source_units":[{"raw_tex":"The proof is similar to that of~\\cite[Lemma~4.5]{chen2018stochastic} and is not detailed here. It follows from~\\eqref{CauchyDecrease} together with the inequalities $\\kappa_h\\geq \\max\\accolade{\\norme{\\Hkhat}, 1}$ and $\\normii{\\gkhat}\\geq\\kappa_h\\dk$ that  $\\mhatk(\\bm{0})-\\mhatk(\\sk)\\geq\\frac{\\kappa_{fcd}}{2}\\normii{\\gkhat}\\dk.$ Since the model is $(\\kef,\\keg;\\Qk)$-fully linear, then \n$f(\\xk+\\Qk\\sk)-f(\\xk)\\leq 2\\kappa_{ef}\\dk^2+\\mhatk(\\sk)-\\mhatk(\\bm{0})\\leq -\\frac{\\kappa_{fcd}}{4}\\normii{\\gkhat}\\dk,$\nwhere the last inequality follows from the fact that $\\dk\\leq \\frac{\\kappa_{fcd}}{8\\kappa_{ef}}\\normii{\\gkhat}$.\n","start":141123,"end":141742}]},{"id":"S090","semantic_type":"proof","kind":"proof","labels":["ineq1"],"declared_external":false,"source_only":false,"title":"投影梯度下降的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nThe proof is inspired by (and similar to) that of~\\cite[Lemma~4.6]{chen2018stochastic}. By $(\\kef,\\keg;\\Qk)$-full linearity and~\\eqref{deltaLemmaB2}, it holds\n\\begin{equation}\\label{ineq1}\n\\normii{\\gkhat}\\geq \\normii{\\QkTranspose\\nabla f(\\xk)}-\\kappa_{eg}\\dk\\geq \\max\\accolade{\\kappa_h, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}}}\\dk, \n\\end{equation}\nwhich shows that condition~\\eqref{LemmaAimprovement} is satisfied. Consequently $f$ decreases as in~\\eqref{LemmaAimprovement}. The first inequality in~\\eqref{ineq1}, together with~\\eqref{deltaLemmaB2}, yields\n$\\normii{\\gkhat}\\geq \\frac{4C_1}{\\kappa_{fcd}}\\normii{\\QkTranspose\\nabla f(\\xk)},$\nwhich, combined with~\\eqref{LemmaAimprovement}, implies~\\eqref{LemmaB2improvement}, and the proof is complete.\n\\end{proof}","start":141801,"end":142571}]},{"id":"S091","semantic_type":"proof_step","kind":"proof_step","labels":["ineq1"],"declared_external":false,"source_only":false,"title":"投影梯度转为模型梯度门槛","source_units":[{"raw_tex":"The proof is inspired by (and similar to) that of~\\cite[Lemma~4.6]{chen2018stochastic}. By $(\\kef,\\keg;\\Qk)$-full linearity and~\\eqref{deltaLemmaB2}, it holds\n\\begin{equation}\\label{ineq1}\n\\normii{\\gkhat}\\geq \\normii{\\QkTranspose\\nabla f(\\xk)}-\\kappa_{eg}\\dk\\geq \\max\\accolade{\\kappa_h, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}}}\\dk, \n\\end{equation}\n","start":141815,"end":142157}]},{"id":"S092","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"转移下降系数","source_units":[{"raw_tex":"which shows that condition~\\eqref{LemmaAimprovement} is satisfied. Consequently $f$ decreases as in~\\eqref{LemmaAimprovement}. The first inequality in~\\eqref{ineq1}, together with~\\eqref{deltaLemmaB2}, yields\n$\\normii{\\gkhat}\\geq \\frac{4C_1}{\\kappa_{fcd}}\\normii{\\QkTranspose\\nabla f(\\xk)},$\nwhich, combined with~\\eqref{LemmaAimprovement}, implies~\\eqref{LemmaB2improvement}, and the proof is complete.\n","start":142157,"end":142560}]},{"id":"S093","semantic_type":"proof","kind":"proof","labels":["ineq2","ineq3"],"declared_external":false,"source_only":false,"title":"错误接受增幅的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nAccording to the Fundamental Theorem of Calculus in $\\rn$ (see, e.g.,~\\cite[Lemma~9.3]{AuHa2017}),\\\\\n$\\abs{f(\\xk)-f(\\xk+\\Qk\\sk)-\\nabla f(\\xk+\\Qk\\sk)^{\\top}(-\\Qk\\sk)}\\leq\\frac{1}{2}L_g\\Qmax^2\\dk^2,$  with the inequality also following from $\\normii{\\Qk\\sk}\\leq \\Qmax\\dk$, which implies that\n\\begin{equation}\\label{ineq2}\nf(\\xk+\\Qk\\sk)-f(\\xk)\\leq \\sk^{\\top}\\left[\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)\\right] + \\frac{1}{2}L_g\\Qmax^2\\dk^2. \n\\end{equation}\nPer Lipschitz continuity of $\\nabla f$, \n$\\normii{\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)-\\QkTranspose \\nabla f(\\xk)} \\leq L_g\\Qmax^2\\dk,$\nwhich implies that\n\\begin{equation}\\label{ineq3}\n\\normii{\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)}\\leq L_g\\Qmax^2\\dk+\\normii{\\QkTranspose \\nabla f(\\xk)}.\n\\end{equation}\nThus, using~\\eqref{ineq2} and~\\eqref{ineq3}, together with the inequality $\\dk\\leq \\zeta^{-1}\\normii{\\QkTranspose \\nabla f(\\xk)}$, yields\n\\begin{eqnarray*}\nf(\\xk+\\Qk\\sk)&-&f(\\xk)\\leq \\normii{\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)}\\dk+\\frac{L_g}{2\\zeta}\\Qmax^2\\normii{\\QkTranspose \\nabla f(\\xk)}\\dk\\\\\n&\\leq& \\left(\\frac{L_g}{\\zeta}\\Qmax^2\\normii{\\QkTranspose \\nabla f(\\xk)} +\\normii{\\QkTranspose \\nabla f(\\xk)}\\right)\\dk\\\\\n&+&\\frac{L_g}{2\\zeta}\\Qmax^2\\normii{\\QkTranspose \\nabla f(\\xk)}\\dk= \\left(1+\\frac{3L_g}{2\\zeta}\\Qmax^2\\right)\\normii{\\QkTranspose \\nabla f(\\xk)}\\dk,\n\\end{eqnarray*}\nwhich completes the proof.\n\\end{proof}","start":142618,"end":143995}]},{"id":"S094","semantic_type":"proof_step","kind":"proof_step","labels":["ineq2"],"declared_external":false,"source_only":false,"title":"以终点梯度控制目标差","source_units":[{"raw_tex":"According to the Fundamental Theorem of Calculus in $\\rn$ (see, e.g.,~\\cite[Lemma~9.3]{AuHa2017}),\\\\\n$\\abs{f(\\xk)-f(\\xk+\\Qk\\sk)-\\nabla f(\\xk+\\Qk\\sk)^{\\top}(-\\Qk\\sk)}\\leq\\frac{1}{2}L_g\\Qmax^2\\dk^2,$  with the inequality also following from $\\normii{\\Qk\\sk}\\leq \\Qmax\\dk$, which implies that\n\\begin{equation}\\label{ineq2}\nf(\\xk+\\Qk\\sk)-f(\\xk)\\leq \\sk^{\\top}\\left[\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)\\right] + \\frac{1}{2}L_g\\Qmax^2\\dk^2. \n\\end{equation}\n","start":142632,"end":143080}]},{"id":"S095","semantic_type":"proof_step","kind":"proof_step","labels":["ineq3"],"declared_external":false,"source_only":false,"title":"终点投影梯度变化界","source_units":[{"raw_tex":"Per Lipschitz continuity of $\\nabla f$, \n$\\normii{\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)-\\QkTranspose \\nabla f(\\xk)} \\leq L_g\\Qmax^2\\dk,$\nwhich implies that\n\\begin{equation}\\label{ineq3}\n\\normii{\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)}\\leq L_g\\Qmax^2\\dk+\\normii{\\QkTranspose \\nabla f(\\xk)}.\n\\end{equation}\n","start":143080,"end":143375}]},{"id":"S096","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"利用大梯度吸收二次项","source_units":[{"raw_tex":"Thus, using~\\eqref{ineq2} and~\\eqref{ineq3}, together with the inequality $\\dk\\leq \\zeta^{-1}\\normii{\\QkTranspose \\nabla f(\\xk)}$, yields\n\\begin{eqnarray*}\nf(\\xk+\\Qk\\sk)&-&f(\\xk)\\leq \\normii{\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)}\\dk+\\frac{L_g}{2\\zeta}\\Qmax^2\\normii{\\QkTranspose \\nabla f(\\xk)}\\dk\\\\\n&\\leq& \\left(\\frac{L_g}{\\zeta}\\Qmax^2\\normii{\\QkTranspose \\nabla f(\\xk)} +\\normii{\\QkTranspose \\nabla f(\\xk)}\\right)\\dk\\\\\n&+&\\frac{L_g}{2\\zeta}\\Qmax^2\\normii{\\QkTranspose \\nabla f(\\xk)}\\dk= \\left(1+\\frac{3L_g}{2\\zeta}\\Qmax^2\\right)\\normii{\\QkTranspose \\nabla f(\\xk)}\\dk,\n\\end{eqnarray*}\nwhich completes the proof.\n","start":143375,"end":143984}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S038","edges":[{"source_id":"S001","target_id":"S012","kind":"supports"},{"source_id":"S002","target_id":"S012","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S009","target_id":"S005","kind":"supports"},{"source_id":"S012","target_id":"S005","kind":"supports"},{"source_id":"S014","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S005","target_id":"S008","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S007","target_id":"S013","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S007","target_id":"S015","kind":"supports"},{"source_id":"S009","target_id":"S016","kind":"supports"},{"source_id":"S013","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S007","target_id":"S016","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S013","target_id":"S019","kind":"supports"},{"source_id":"S015","target_id":"S019","kind":"supports"},{"source_id":"S007","target_id":"S019","kind":"supports"},{"source_id":"S017","target_id":"S019","kind":"supports"},{"source_id":"S016","target_id":"S019","kind":"supports"},{"source_id":"S013","target_id":"S028","kind":"supports"},{"source_id":"S015","target_id":"S028","kind":"supports"},{"source_id":"S014","target_id":"S028","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S001","target_id":"S044","kind":"supports"},{"source_id":"S076","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S047","kind":"supports"},{"source_id":"S007","target_id":"S047","kind":"supports"},{"source_id":"S076","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S003","target_id":"S089","kind":"supports"},{"source_id":"S021","target_id":"S089","kind":"supports"},{"source_id":"S012","target_id":"S089","kind":"supports"},{"source_id":"S089","target_id":"S088","kind":"supports"},{"source_id":"S088","target_id":"S039","kind":"supports"},{"source_id":"S012","target_id":"S091","kind":"supports"},{"source_id":"S091","target_id":"S092","kind":"supports"},{"source_id":"S039","target_id":"S092","kind":"supports"},{"source_id":"S091","target_id":"S090","kind":"supports"},{"source_id":"S092","target_id":"S090","kind":"supports"},{"source_id":"S090","target_id":"S040","kind":"supports"},{"source_id":"S077","target_id":"S094","kind":"supports"},{"source_id":"S020","target_id":"S094","kind":"supports"},{"source_id":"S011","target_id":"S094","kind":"supports"},{"source_id":"S002","target_id":"S094","kind":"supports"},{"source_id":"S020","target_id":"S095","kind":"supports"},{"source_id":"S011","target_id":"S095","kind":"supports"},{"source_id":"S002","target_id":"S095","kind":"supports"},{"source_id":"S094","target_id":"S096","kind":"supports"},{"source_id":"S095","target_id":"S096","kind":"supports"},{"source_id":"S094","target_id":"S093","kind":"supports"},{"source_id":"S095","target_id":"S093","kind":"supports"},{"source_id":"S096","target_id":"S093","kind":"supports"},{"source_id":"S093","target_id":"S041","kind":"supports"},{"source_id":"S005","target_id":"S054","kind":"supports"},{"source_id":"S049","target_id":"S054","kind":"supports"},{"source_id":"S045","target_id":"S055","kind":"supports"},{"source_id":"S048","target_id":"S055","kind":"supports"},{"source_id":"S040","target_id":"S056","kind":"supports"},{"source_id":"S022","target_id":"S056","kind":"supports"},{"source_id":"S012","target_id":"S056","kind":"supports"},{"source_id":"S048","target_id":"S056","kind":"supports"},{"source_id":"S054","target_id":"S056","kind":"supports"},{"source_id":"S028","target_id":"S056","kind":"supports"},{"source_id":"S041","target_id":"S057","kind":"supports"},{"source_id":"S054","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S059","kind":"supports"},{"source_id":"S057","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S045","target_id":"S059","kind":"supports"},{"source_id":"S013","target_id":"S059","kind":"supports"},{"source_id":"S019","target_id":"S059","kind":"supports"},{"source_id":"S017","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S048","target_id":"S060","kind":"supports"},{"source_id":"S005","target_id":"S061","kind":"supports"},{"source_id":"S007","target_id":"S061","kind":"supports"},{"source_id":"S054","target_id":"S061","kind":"supports"},{"source_id":"S045","target_id":"S061","kind":"supports"},{"source_id":"S025","target_id":"S063","kind":"supports"},{"source_id":"S054","target_id":"S063","kind":"supports"},{"source_id":"S048","target_id":"S063","kind":"supports"},{"source_id":"S028","target_id":"S063","kind":"supports"},{"source_id":"S041","target_id":"S064","kind":"supports"},{"source_id":"S054","target_id":"S064","kind":"supports"},{"source_id":"S057","target_id":"S064","kind":"supports"},{"source_id":"S063","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S015","target_id":"S065","kind":"supports"},{"source_id":"S048","target_id":"S065","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S065","target_id":"S062","kind":"supports"},{"source_id":"S007","target_id":"S062","kind":"supports"},{"source_id":"S045","target_id":"S062","kind":"supports"},{"source_id":"S017","target_id":"S062","kind":"supports"},{"source_id":"S060","target_id":"S050","kind":"supports"},{"source_id":"S062","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S066","kind":"supports"},{"source_id":"S049","target_id":"S066","kind":"supports"},{"source_id":"S020","target_id":"S066","kind":"supports"},{"source_id":"S078","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S055","target_id":"S053","kind":"supports"},{"source_id":"S056","target_id":"S053","kind":"supports"},{"source_id":"S057","target_id":"S053","kind":"supports"},{"source_id":"S058","target_id":"S053","kind":"supports"},{"source_id":"S059","target_id":"S053","kind":"supports"},{"source_id":"S060","target_id":"S053","kind":"supports"},{"source_id":"S061","target_id":"S053","kind":"supports"},{"source_id":"S062","target_id":"S053","kind":"supports"},{"source_id":"S063","target_id":"S053","kind":"supports"},{"source_id":"S064","target_id":"S053","kind":"supports"},{"source_id":"S065","target_id":"S053","kind":"supports"},{"source_id":"S066","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S051","target_id":"S070","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S010","target_id":"S071","kind":"supports"},{"source_id":"S005","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S016","target_id":"S072","kind":"supports"},{"source_id":"S019","target_id":"S072","kind":"supports"},{"source_id":"S009","target_id":"S072","kind":"supports"},{"source_id":"S012","target_id":"S072","kind":"supports"},{"source_id":"S022","target_id":"S072","kind":"supports"},{"source_id":"S005","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S016","target_id":"S073","kind":"supports"},{"source_id":"S068","target_id":"S073","kind":"supports"},{"source_id":"S018","target_id":"S074","kind":"supports"},{"source_id":"S016","target_id":"S074","kind":"supports"},{"source_id":"S068","target_id":"S074","kind":"supports"},{"source_id":"S072","target_id":"S074","kind":"supports"},{"source_id":"S071","target_id":"S074","kind":"supports"},{"source_id":"S075","target_id":"S074","kind":"supports"},{"source_id":"S070","target_id":"S069","kind":"supports"},{"source_id":"S071","target_id":"S069","kind":"supports"},{"source_id":"S072","target_id":"S069","kind":"supports"},{"source_id":"S073","target_id":"S069","kind":"supports"},{"source_id":"S074","target_id":"S069","kind":"supports"},{"source_id":"S069","target_id":"S067","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S029","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S049","target_id":"S033","kind":"supports"},{"source_id":"S005","target_id":"S033","kind":"supports"},{"source_id":"S009","target_id":"S033","kind":"supports"},{"source_id":"S013","target_id":"S033","kind":"supports"},{"source_id":"S015","target_id":"S033","kind":"supports"},{"source_id":"S029","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S007","target_id":"S034","kind":"supports"},{"source_id":"S050","target_id":"S035","kind":"supports"},{"source_id":"S007","target_id":"S035","kind":"supports"},{"source_id":"S017","target_id":"S035","kind":"supports"},{"source_id":"S005","target_id":"S035","kind":"supports"},{"source_id":"S049","target_id":"S035","kind":"supports"},{"source_id":"S072","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S005","target_id":"S037","kind":"supports"},{"source_id":"S033","target_id":"S081","kind":"supports"},{"source_id":"S031","target_id":"S081","kind":"supports"},{"source_id":"S037","target_id":"S082","kind":"supports"},{"source_id":"S036","target_id":"S082","kind":"supports"},{"source_id":"S033","target_id":"S082","kind":"supports"},{"source_id":"S005","target_id":"S082","kind":"supports"},{"source_id":"S072","target_id":"S082","kind":"supports"},{"source_id":"S009","target_id":"S083","kind":"supports"},{"source_id":"S013","target_id":"S083","kind":"supports"},{"source_id":"S015","target_id":"S083","kind":"supports"},{"source_id":"S081","target_id":"S080","kind":"supports"},{"source_id":"S082","target_id":"S080","kind":"supports"},{"source_id":"S083","target_id":"S080","kind":"supports"},{"source_id":"S080","target_id":"S079","kind":"supports"},{"source_id":"S036","target_id":"S085","kind":"supports"},{"source_id":"S037","target_id":"S085","kind":"supports"},{"source_id":"S079","target_id":"S085","kind":"supports"},{"source_id":"S032","target_id":"S086","kind":"supports"},{"source_id":"S079","target_id":"S086","kind":"supports"},{"source_id":"S033","target_id":"S086","kind":"supports"},{"source_id":"S034","target_id":"S086","kind":"supports"},{"source_id":"S035","target_id":"S086","kind":"supports"},{"source_id":"S085","target_id":"S086","kind":"supports"},{"source_id":"S086","target_id":"S087","kind":"supports"},{"source_id":"S079","target_id":"S087","kind":"supports"},{"source_id":"S068","target_id":"S087","kind":"supports"},{"source_id":"S085","target_id":"S084","kind":"supports"},{"source_id":"S086","target_id":"S084","kind":"supports"},{"source_id":"S087","target_id":"S084","kind":"supports"},{"source_id":"S084","target_id":"S038","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S038","edges":[{"source_id":"S001","target_id":"S012","kind":"supports"},{"source_id":"S002","target_id":"S012","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S009","target_id":"S005","kind":"supports"},{"source_id":"S012","target_id":"S005","kind":"supports"},{"source_id":"S014","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S007","target_id":"S013","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S007","target_id":"S015","kind":"supports"},{"source_id":"S009","target_id":"S016","kind":"supports"},{"source_id":"S013","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S007","target_id":"S016","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S013","target_id":"S019","kind":"supports"},{"source_id":"S015","target_id":"S019","kind":"supports"},{"source_id":"S007","target_id":"S019","kind":"supports"},{"source_id":"S017","target_id":"S019","kind":"supports"},{"source_id":"S016","target_id":"S019","kind":"supports"},{"source_id":"S013","target_id":"S028","kind":"supports"},{"source_id":"S015","target_id":"S028","kind":"supports"},{"source_id":"S014","target_id":"S028","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S001","target_id":"S044","kind":"supports"},{"source_id":"S076","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S047","kind":"supports"},{"source_id":"S007","target_id":"S047","kind":"supports"},{"source_id":"S076","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S003","target_id":"S089","kind":"supports"},{"source_id":"S021","target_id":"S089","kind":"supports"},{"source_id":"S012","target_id":"S089","kind":"supports"},{"source_id":"S089","target_id":"S088","kind":"supports"},{"source_id":"S088","target_id":"S039","kind":"supports"},{"source_id":"S012","target_id":"S091","kind":"supports"},{"source_id":"S091","target_id":"S092","kind":"supports"},{"source_id":"S039","target_id":"S092","kind":"supports"},{"source_id":"S091","target_id":"S090","kind":"supports"},{"source_id":"S092","target_id":"S090","kind":"supports"},{"source_id":"S090","target_id":"S040","kind":"supports"},{"source_id":"S077","target_id":"S094","kind":"supports"},{"source_id":"S020","target_id":"S094","kind":"supports"},{"source_id":"S011","target_id":"S094","kind":"supports"},{"source_id":"S002","target_id":"S094","kind":"supports"},{"source_id":"S020","target_id":"S095","kind":"supports"},{"source_id":"S011","target_id":"S095","kind":"supports"},{"source_id":"S002","target_id":"S095","kind":"supports"},{"source_id":"S094","target_id":"S096","kind":"supports"},{"source_id":"S095","target_id":"S096","kind":"supports"},{"source_id":"S094","target_id":"S093","kind":"supports"},{"source_id":"S095","target_id":"S093","kind":"supports"},{"source_id":"S096","target_id":"S093","kind":"supports"},{"source_id":"S093","target_id":"S041","kind":"supports"},{"source_id":"S005","target_id":"S054","kind":"supports"},{"source_id":"S049","target_id":"S054","kind":"supports"},{"source_id":"S045","target_id":"S055","kind":"supports"},{"source_id":"S048","target_id":"S055","kind":"supports"},{"source_id":"S040","target_id":"S056","kind":"supports"},{"source_id":"S022","target_id":"S056","kind":"supports"},{"source_id":"S012","target_id":"S056","kind":"supports"},{"source_id":"S048","target_id":"S056","kind":"supports"},{"source_id":"S054","target_id":"S056","kind":"supports"},{"source_id":"S028","target_id":"S056","kind":"supports"},{"source_id":"S041","target_id":"S057","kind":"supports"},{"source_id":"S054","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S059","kind":"supports"},{"source_id":"S057","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S045","target_id":"S059","kind":"supports"},{"source_id":"S013","target_id":"S059","kind":"supports"},{"source_id":"S019","target_id":"S059","kind":"supports"},{"source_id":"S017","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S048","target_id":"S060","kind":"supports"},{"source_id":"S005","target_id":"S061","kind":"supports"},{"source_id":"S007","target_id":"S061","kind":"supports"},{"source_id":"S054","target_id":"S061","kind":"supports"},{"source_id":"S045","target_id":"S061","kind":"supports"},{"source_id":"S025","target_id":"S063","kind":"supports"},{"source_id":"S054","target_id":"S063","kind":"supports"},{"source_id":"S048","target_id":"S063","kind":"supports"},{"source_id":"S028","target_id":"S063","kind":"supports"},{"source_id":"S041","target_id":"S064","kind":"supports"},{"source_id":"S054","target_id":"S064","kind":"supports"},{"source_id":"S057","target_id":"S064","kind":"supports"},{"source_id":"S063","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S015","target_id":"S065","kind":"supports"},{"source_id":"S048","target_id":"S065","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S065","target_id":"S062","kind":"supports"},{"source_id":"S007","target_id":"S062","kind":"supports"},{"source_id":"S045","target_id":"S062","kind":"supports"},{"source_id":"S017","target_id":"S062","kind":"supports"},{"source_id":"S060","target_id":"S050","kind":"supports"},{"source_id":"S062","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S066","kind":"supports"},{"source_id":"S049","target_id":"S066","kind":"supports"},{"source_id":"S020","target_id":"S066","kind":"supports"},{"source_id":"S078","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S055","target_id":"S053","kind":"supports"},{"source_id":"S056","target_id":"S053","kind":"supports"},{"source_id":"S057","target_id":"S053","kind":"supports"},{"source_id":"S058","target_id":"S053","kind":"supports"},{"source_id":"S059","target_id":"S053","kind":"supports"},{"source_id":"S060","target_id":"S053","kind":"supports"},{"source_id":"S061","target_id":"S053","kind":"supports"},{"source_id":"S062","target_id":"S053","kind":"supports"},{"source_id":"S063","target_id":"S053","kind":"supports"},{"source_id":"S064","target_id":"S053","kind":"supports"},{"source_id":"S065","target_id":"S053","kind":"supports"},{"source_id":"S066","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S051","target_id":"S070","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S010","target_id":"S071","kind":"supports"},{"source_id":"S005","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S016","target_id":"S072","kind":"supports"},{"source_id":"S019","target_id":"S072","kind":"supports"},{"source_id":"S009","target_id":"S072","kind":"supports"},{"source_id":"S012","target_id":"S072","kind":"supports"},{"source_id":"S022","target_id":"S072","kind":"supports"},{"source_id":"S005","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S016","target_id":"S073","kind":"supports"},{"source_id":"S068","target_id":"S073","kind":"supports"},{"source_id":"S018","target_id":"S074","kind":"supports"},{"source_id":"S016","target_id":"S074","kind":"supports"},{"source_id":"S068","target_id":"S074","kind":"supports"},{"source_id":"S072","target_id":"S074","kind":"supports"},{"source_id":"S071","target_id":"S074","kind":"supports"},{"source_id":"S075","target_id":"S074","kind":"supports"},{"source_id":"S070","target_id":"S069","kind":"supports"},{"source_id":"S071","target_id":"S069","kind":"supports"},{"source_id":"S072","target_id":"S069","kind":"supports"},{"source_id":"S073","target_id":"S069","kind":"supports"},{"source_id":"S074","target_id":"S069","kind":"supports"},{"source_id":"S069","target_id":"S067","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S029","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S049","target_id":"S033","kind":"supports"},{"source_id":"S005","target_id":"S033","kind":"supports"},{"source_id":"S009","target_id":"S033","kind":"supports"},{"source_id":"S013","target_id":"S033","kind":"supports"},{"source_id":"S015","target_id":"S033","kind":"supports"},{"source_id":"S029","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S007","target_id":"S034","kind":"supports"},{"source_id":"S050","target_id":"S035","kind":"supports"},{"source_id":"S007","target_id":"S035","kind":"supports"},{"source_id":"S017","target_id":"S035","kind":"supports"},{"source_id":"S005","target_id":"S035","kind":"supports"},{"source_id":"S049","target_id":"S035","kind":"supports"},{"source_id":"S072","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S005","target_id":"S037","kind":"supports"},{"source_id":"S033","target_id":"S081","kind":"supports"},{"source_id":"S031","target_id":"S081","kind":"supports"},{"source_id":"S037","target_id":"S082","kind":"supports"},{"source_id":"S036","target_id":"S082","kind":"supports"},{"source_id":"S033","target_id":"S082","kind":"supports"},{"source_id":"S005","target_id":"S082","kind":"supports"},{"source_id":"S072","target_id":"S082","kind":"supports"},{"source_id":"S009","target_id":"S083","kind":"supports"},{"source_id":"S013","target_id":"S083","kind":"supports"},{"source_id":"S015","target_id":"S083","kind":"supports"},{"source_id":"S081","target_id":"S080","kind":"supports"},{"source_id":"S082","target_id":"S080","kind":"supports"},{"source_id":"S083","target_id":"S080","kind":"supports"},{"source_id":"S080","target_id":"S079","kind":"supports"},{"source_id":"S036","target_id":"S085","kind":"supports"},{"source_id":"S037","target_id":"S085","kind":"supports"},{"source_id":"S079","target_id":"S085","kind":"supports"},{"source_id":"S032","target_id":"S086","kind":"supports"},{"source_id":"S079","target_id":"S086","kind":"supports"},{"source_id":"S033","target_id":"S086","kind":"supports"},{"source_id":"S034","target_id":"S086","kind":"supports"},{"source_id":"S035","target_id":"S086","kind":"supports"},{"source_id":"S085","target_id":"S086","kind":"supports"},{"source_id":"S086","target_id":"S087","kind":"supports"},{"source_id":"S079","target_id":"S087","kind":"supports"},{"source_id":"S068","target_id":"S087","kind":"supports"},{"source_id":"S085","target_id":"S084","kind":"supports"},{"source_id":"S086","target_id":"S084","kind":"supports"},{"source_id":"S087","target_id":"S084","kind":"supports"},{"source_id":"S084","target_id":"S038","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["probl1"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"随机无约束优化","source_units":[{"raw_tex":"This paper focuses on unconstrained stochastic DFO, wherein the objective function values are accessible only through a blackbox oracle corrupted by stochastic noise. We consider the problem \n\\begin{equation}\\label{probl1}\n\t\\underset{\\x\\in\\rn}{\\min} f(\\x), \\qquad\\mbox{with}\\quad f(\\x)=\\Esp_{\\ubar{\\theta}}\\left[f_{\\ubar{\\theta}}(\\x)\\right],\n\\end{equation}\n%\\quad\\mbox{with}\\quad f(\\x)=\\Esp_{\\ubar{\\theta}}\\left[f_{\\ubar{\\theta}}(\\x)\\right]\nwhere the values of the continuously differentiable function $f:\\rn\\to\\R$ are available only via $f_{\\ubar{\\theta}}$, a stochastically noisy version of $f$, and where $\\ubar{\\theta}$ is a random variable whose distribution is possibly unknown. \n","start":28799,"end":29485}]},{"id":"S002","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"子空间模型与试探步","source_units":[{"raw_tex":"Unlike the stochastic trust-region framework STORM~\\cite{blanchet2016convergence,chen2018stochastic}, which builds full space random models $\\ubar{m}_k(\\x)$, $\\x\\in\\rn$, to approximate $f(\\x)$, STARS operates as follows. On iteration $k$, given a current iterate $\\xk$, let $\\mathcal{Y}_k\\subseteq \\rn$ be the affine space randomly chosen through the range of a matrix $\\Qk\\in\\rnp$ \n%satisfying $\\norme{\\Qk}\\leq\\Qmax$ for some $\\Qmax>0$, \nwhose entries are randomly selected; that is, \n\\[\\mathcal{Y}_k = \\accolade{\\xk+\\Qk\\bm{{\\s}}:\\bm{{\\s}}\\in\\R^p}.\\]\nThen, given a trust-region radius $\\dk>0$, a subspace model $\\mhatk(\\s), \\s\\in\\rp$ is built only on $\\mathcal{Y}_k$ using realizations of the stochastically noisy function $f_{\\ubar{\\theta}}$; the model  serves as an approximation of $f(\\x)$ in $\\mathcal{B}(\\xk,\\dk;\\Qk):=\\accolade{\\xk+\\Qk\\s \\in \\mathcal{Y}_k:\\normii{\\s}\\leq\\dk}$. For concreteness, here we employ a quadratic subspace model given by \n\\begin{equation*}\nf(\\xk+\\Qk\\s)\\approx\\mhatk(\\bm{\\s}):= f_k+\\gkhat^\\top\\s+\\s^\\top\\Hkhat\\s,\n\\end{equation*}\nwhere $\\gkhat\\in \\R^p$ and $\\Hkhat\\in \\Sym^{p}$ are the low-dimensional model gradient and Hessian, respectively. A {\\it tentative step} $\\Qk\\sk\\in\\rn$ is produced by using the solution $\\sk\\in\\rp$ obtained by approximately minimizing $\\mhatk$ inside the trust  region; that  is, $\\sk\\approx{\\argmin}\\left\\{ \\mhatk(\\s): \\s\\in\\rp, \\, \\normii{\\bm{{\\s}}}\\leq \\dk\\right\\}$. Inspired by~\\cite{CRsubspace2021,chen2018stochastic}, the {\\it trial step} $\\sk$ has to provide a sufficient decrease in $\\mhatk$ by satisfying the following standard fraction of the Cauchy decrease condition.\n\n","start":39790,"end":41431}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","labels":["CauchyDecrease"],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"柯西充分下降条件","source_units":[{"raw_tex":"\\begin{assumption}\nFor every iteration $k$, a trial step $\\sk\\in\\rp$ is computed so that\\footnote{Throughout the manuscript, the matrix norm $\\norme{\\cdot}$ is supposed to be {\\it consistent} with the Euclidean norm; that is, $\\normii{\\Ab\\xb}\\leq \\norme{\\Ab}\\normii{\\xb}$, which holds for both Frobenius and spectral matrix norms.}\n\\begin{equation}\\label{CauchyDecrease}\n\\mhatk(\\bm{0})-\\mhatk(\\bm{\\sk})\\geq \\frac{\\kappa_{fcd}}{2}\\normii{\\gkhat}\\min\\accolade{\\dk,\\frac{\\normii{\\gkhat}}{\\max\\left(\\norme{\\Hkhat},1\\right)}},\n\\end{equation}\nfor some constant $\\kappa_{fcd}\\in (0,1]$.\n\\end{assumption}\n","start":41431,"end":42028}]},{"id":"S004","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"概率空间与随机记号","source_units":[{"raw_tex":"The random variables considered here are all defined on the same probability space $(\\Omega,\\F, \\pr)$, where $\\Omega$, referred to as the {\\it sample space}, is a nonempty set whose elements $\\omega$ and subsets are called {\\it sample points} and {\\it events}, respectively. $\\F$ is a collection of such events, which is called a {\\it $\\sigma$-algebra}. $\\pr$ is a finite measure defined on the measurable space $(\\Omega,\\F)$, satisfying $\\prob{\\Omega}=1$ and referred to as the {\\it probability measure}. An increasing subsequence $\\accolade{\\mathcal{S}_k}_k$ of $\\sigma$-algebras of $\\F$ will be called a {\\it filtration}. Let $\\mathscr{B}(\\rn)$ be the $\\sigma$-algebra generated by the open sets of $\\rn$, also known as the Borel $\\sigma$-algebra of $\\rn$. A random variable $\\ubar{z}$ is a measurable map defined on $(\\Omega,\\F, \\pr)$ into the measurable space $(\\rn, \\mathscr{B}(\\rn))$, where measurability means that $\\accolade{\\ubar{z}\\in A}:=\\accolade{\\omega\\in\\Omega:\\ubar{z}(\\omega)\\in A} =:{\\ubar{z}}^{-1}(A)\\in\\F\\ \\forall A\\in \\mathscr{B}(\\rn)$~\\cite{bhattacharya2007basic}. For the remainder of the manuscript, vectors will be written in lowercase boldface (e.g., $\\x\\in\\rn, n\\geq 2$) while matrices will be written in uppercase boldface (e.g., $\\Qk\\in\\rnp$), and underlined letters (e.g., $\\ubar{z}, \\ubar{\\x}, \\QkRandom$) will be used to denote random quantities; $z=\\ubar{z}(\\omega)$ will denote a realization of $\\ubar{z}$.\n","start":43183,"end":44624}]},{"id":"S005","semantic_type":"algorithm","kind":"algorithm","labels":["algoStoScalTR"],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"STARS 完整算法","source_units":[{"raw_tex":"\\begin{algorithm}[htb]\n\\caption{STARS}\n\\label{algoStoScalTR} \n\\textbf{[0] Initialization}\\\\\n\\hspace*{10mm}Choose constants $\\gamma>1$, $\\eta_1\\in(0,1)$, $\\eta_2>0$, $j_{\\max}\\in\\N$, $\\varepsilon\\in(0,1)$, $\\beta\\in(0,1)$, \\\\ \n\\hspace*{10mm}$c_1\\geq 1$, initial trust-region radius $\\delta_0>0$, and maximum trust-region radius\\\\\n\\hspace*{10mm}$\\dmaxx=\\gamma^{j_{\\max}}\\delta_0$, starting point $\\bm{{\\x}_0}\\in\\rn$, and dimension $p\\in\\intbracket{1,n}$.\\\\ \n\\hspace*{10mm}Set the iteration counter $k \\gets 0$.\\\\\n\\textbf{[1] Construction of subspace model}\\\\\n\\hspace*{10mm}Generate $\\Qk$: a realization of a\\footnotemark {\\em WAM}($1-\\varepsilon,\\beta$), using a distribution $\\mathbb{Q}_{k}$.\\\\\n\\hspace*{10mm}Build model $\\mhatk:\\rp\\to\\R$ that is $(\\kef,\\keg;\\Qk)$-fully linear \n%$\\mhatk(\\s)={\\f}_k+\\gkhat^{\\top}\\s+$\\\\\n%\\hspace*{10mm}$\\frac{1}{2}\\s^{\\top}\\Hkhat\\s\\approx f(\\xk+\\Qk \\s)$, that locally approximates $f$ \nin  $\\mathcal{B}(\\xk,c_1\\dk;\\Qk)$.\\\\ \n%\\hspace*{10mm}with the affine space defined by the columns of $\\Qk$.\\\\\n\\textbf{[2] Step calculation}\\\\\n\\hspace*{10mm}Compute $\\sk\\approx {\\argmin}\\left\\{ \\mhatk(\\s) : \\; \\s\\in\\rp:\\normii{\\s}\\leq \\dk\\right \\}$ satisfying~\\eqref{CauchyDecrease}.\\\\\n\\textbf{[3] Estimate computation}\\\\\n\\hspace*{10mm}Obtain estimates $\\fok\\approx f(\\xk)$ and $\\fsk\\approx f(\\xk+\\Qk\\sk)$ satisfying~\\eqref{accEstimates}.\\\\\n\\textbf{[4] Updates}\\\\\n\\hspace*{10mm}Compute $\\rho_k=\\frac{\\fok-\\fsk}{\\mhatk(\\bm{0})-\\mhatk(\\sk)}.$\\\\\n\\hspace*{10mm}If $\\rho_k\\geq \\eta_1$ and $\\normii{\\gkhat}\\geq \\eta_2\\dk$ (\\textbf{success}): \\\\\n\\hspace*{16mm}set $\\bm{\\xkun}=\\xk+\\Qk\\sk$ and $\\dkun=\\min\\accolade{\\gamma\\dk,\\dmaxx}$. \\\\\n\\hspace*{10mm}Otherwise (\\textbf{failure}): set $\\bm{\\xkun}=\\xk$ and $\\dkun=\\gamma^{-1}\\dk$.\\\\\n\\hspace*{10mm}Update the iteration counter $k \\gets k+1$, and go to~\\textbf{[1]}.\\\\\n\\end{algorithm}\n%\\caption[]{\\small{STARS algorithm}}\n%\\end{figure*}\n\\footnotetext{See Definition~\\ref{wellAlignedDefinition} for the meaning of {\\em WAM}($1-\\varepsilon,\\beta$).}\n","start":44648,"end":46651}]},{"id":"S006","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"算法生成随机过程","source_units":[{"raw_tex":"The deterministic models $\\mhatk$ and the function estimates $\\fok$ and $\\fsk$ are constructed by using evaluations of the noisy function $f_{\\ubar{\\theta}}$ and the randomly selected entries of the deterministic matrix~$\\Qk=\\QkRandom(\\omega)$.  Hence, $\\mhatk$, $\\fok$, and $\\fsk$ can be considered as realizations of random models and estimates $\\Mhatk$, $\\Fok$, and $\\Fsk$, respectively. Each iteration of Algorithm~\\ref{algoStoScalTR} is therefore influenced by the behavior of these random quantities; and consequently, the algorithm results in a stochastic process. The present work shows that under certain conditions on the sequences~$\\accoladekinN{\\QkRandom}$, $\\accoladekinN{\\Fok,\\Fsk}$, and $\\accoladekinN{\\Mhatk}$, the resulting stochastic process has desirable convergence properties, conditioned on the past, where {\\it past} means the {\\it past history} of the algorithm (see~Remark~\\ref{whyTheseSigmaAlgebras}). In particular, for the needs of convergence and expected complexity analyses presented  in Sections~\\ref{Section4} and~\\ref{Section5}, and as will be seen  in Section~\\ref{Section3}, the random models~$\\Mhatk$ and estimates~$\\Fok,\\Fsk$ are required to be probabilistically sufficiently accurate conditioned on the past, as was the case in~\\cite{chen2018stochastic}. Moreover, $\\QkRandom$ must guarantee a high quality of the selected subspace probabilistically and conditioned on the past, as will be detailed in Section~\\ref{Section31}. \n\nFor all $k\\in\\N$, denote by $(\\QkRandom)_{ij},  (i,j)\\in \\intbracket{1,n}\\times\\intbracket{1,p}=:\\mathbb{I}_{n,p}$ the entries of the random matrix $\\QkRandom$, where for $a\\leq b$, $\\intbracket{a,b}:=[a,b]\\cap\\mathbb{Z}$  throughout the manuscript. \n","start":46654,"end":48373}]},{"id":"S007","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S007"],"declared_external":false,"source_only":false,"title":"三层历史滤过","source_units":[{"raw_tex":"To formalize conditioning on the past, we consider the three filtrations $\\accolade{\\falgebra}_{k\\in\\N}$, $\\accolade{\\fqalgebra}_{k\\in\\N}$, and $\\accolade{\\fmqalgebra}_{k\\in\\N}$ defined respectively by \n\\begin{equation*}\n\\begin{array}{rcl}\n\\falgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell,  (\\QellRandom)_{ij} \\mbox{ with}\\  \\ell \\in \\intbracket{0,k-1} \\mbox{ \\&}\\ (i,j)\\in \\mathbb{I}_{n,p} \\right)\\\\\n\\fqalgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell, (\\QellRandom)_{ij}, (\\QkRandom)_{ij} \\mbox{ with}\\  \\ell \\in \\intbracket{0,k-1} \\mbox{ \\&}\\ (i,j)\\in \\mathbb{I}_{n,p} \\right)\\\\\n\\fmqalgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell, (\\QellRandom)_{ij}, (\\QkRandom)_{ij}, \\ \\Mhatk \\mbox{ with}\\  \\ell \\in \\intbracket{0,k-1} \\mbox{ \\&}\\ (i,j)\\in \\mathbb{I}_{n,p}  \\right).\n\\end{array}\n\\end{equation*}\n% \\begin{eqnarray*}\n%  \\falgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell\\ \\mbox{and}\\ (\\QellRandom)_{ij}\\ \\mbox{with}\\  \\ell \\in \\intbracket{0,k-1}\\ \\mbox{and}\\ (i,j)\\in \\mathbb{I}_{n,p}\\right)\\\\\n%  \\fqalgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell,\\ (\\QellRandom)_{ij}\\ \\mbox{and}\\ (\\QkRandom)_{ij}\\ \\mbox{with}\\  \\ell \\in \\intbracket{0,k-1}\\ \\mbox{and}\\ (i,j)\\in \\mathbb{I}_{n,p}\\right)\\\\\n%  \\fmqalgebra &:=& \\sigma\\left(\\Foell, \\Fsell, \\Mhatell,\\ (\\QellRandom)_{ij},\\ \\Mhatk\\ \\mbox{and}\\ (\\QkRandom)_{ij}\\ \\mbox{with}\\  \\ell \\in \\intbracket{0,k-1}\\ \\mbox{and}\\ (i,j)\\in \\mathbb{I}_{n,p}\\right).\n% \\end{eqnarray*}\nHere, for {\\it completeness}~\\cite{billingsley1995ThirdEdition}, $\\mathcal{F}_{-1}=\\sigma(\\bm{\\x_0})$.\n%One can easily see that . \nBy construction, $\\falgebra\\subset\\fqalgebra\\subset\\fmqalgebra$, $\\QkRandom$ is both~$\\fqalgebra$ and~$\\fmqalgebra$-measurable, and $\\Mhatk$ is~$\\fmqalgebra$-measurable, which imply that $\\E{\\QkRandom|\\fqalgebra}=\\E{\\QkRandom|\\fmqalgebra}=\\QkRandom$ and $\\E{\\Mhatk|\\fmqalgebra}=\\Mhatk$. \n%The  motivation for  introducing these $\\sigma$-algebras in the formalization of conditioning on the past is explained next.\n\n","start":48373,"end":50347}]},{"id":"S009","semantic_type":"definition","kind":"definition","labels":["Akineq","wellAlignedDefinition"],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"概率良好对齐","source_units":[{"raw_tex":"Recalling Remark~\\ref{whyTheseSigmaAlgebras} and inspired by~\\cite{CRsubspace2021}, we propose a probabilistic quantification of the quality of the subspace selection as follows.\n\n\\begin{definition}\\label{wellAlignedDefinition}\nFor fixed $\\alpha_Q,\\beta_Q\\in (0,1)$, a sequence $\\accoladekinN{\\QkRandom}$ of random matrices \n%determining the selection of subspaces \nis $(1-\\beta_Q)$-probabilistically $\\alpha_{Q}$-well aligned if, for any $\\F_{k-1}$-measurable random vector $\\viRandom$ with realizations $\\vi\\in \\rn$, the events \n$\\mathcal{A}_k:= \\accolade{\\normii{\\QkRandomTranspose\\viRandom}\\geq \\alpha_{{Q}}\\normii{\\viRandom}}$ satisfy \n\\begin{equation}\\label{Akineq}\n\\prob{\\mathcal{A}_k|\\F_{k-1}}=\\E{\\iak|\\F_{k-1}}\\geq 1-\\beta_Q,\n\\end{equation}\nwhere $\\iak$ denotes the indicator function of the event $\\mathcal{A}_k$; that  is, $\\iak(\\omega)=1$ if $\\omega\\in \\mathcal{A}_k$ and $\\iak(\\omega)=0$ otherwise. Random matrices satisfying~\\eqref{Akineq} will be referred to as {WAM}($\\alpha_Q,\\beta_Q$).\n%, in other words, well-aligned matrices with parameters $\\alpha_Q$ and $\\beta_Q$.\n\\end{definition}\n","start":53148,"end":54252}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","labels":["wellAlignedAssumption"],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"子空间对齐假设","source_units":[{"raw_tex":"%\\begin{definition}\n%Random matrices satisfying the conditions of Lemma... with parameters $\\varepsilon, \\beta$ will be referred to as a {\\em JLM}($\\varepsilon,\\beta$), i.e., Johnson-Lindenstrauss matrices with parameters~$\\varepsilon$ and~$\\beta$.\n%\\end{definition}\nFor convergence purposes  in Sections~\\ref{Section4} and~\\ref{Section5}, the following will be assumed throughout the manuscript.\n\\begin{assumption}\\label{wellAlignedAssumption}\nThe sequence $\\accoladekinN{\\QkRandom}$ of matrices used by Algorithm~\\ref{algoStoScalTR} is $(1-\\beta_Q)$-probabilistically $\\alpha_{Q}$-well aligned for some fixed $\\alpha_Q\\in (0,1)$ and $\\beta_Q\\in (0,1/2)$.\n\\end{assumption}\n","start":54252,"end":54926}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","labels":["QmaxAssumption"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"子空间矩阵一致有界","source_units":[{"raw_tex":"Inspired by~\\cite{CRsubspace2021}, a technical assumption needed for convergence and expected complexity analyses later in Sections~\\ref{Section4} and~\\ref{Section5} is the following. \n\\begin{assumption}\\label{QmaxAssumption}\nAny realization $\\Qk$ of the random matrix used by Algorithm~\\ref{algoStoScalTR} satisfies $\\norme{\\Qk}\\leq \\Qmax$ for all $k\\in\\N$, for some constant $\\Qmax >0$ independent of~$k$.\n\\end{assumption}\n","start":63629,"end":64054}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["Eq36","QkFullyLinearModels"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"子空间完全线性模型","source_units":[{"raw_tex":"\\subsection{Probabilistic models and estimates in random subspaces.}\nAs mentioned at the beginning of Section~\\ref{Section3}, the models used by Algorithm~\\ref{algoStoScalTR} need to be sufficiently accurate. Motivated by~\\cite{chen2018stochastic}, this sufficient accuracy is formalized for deterministic subspace models by the following measure of accuracy inspired by~\\cite{CRsubspace2021}.\n\\begin{definition}\\label{QkFullyLinearModels}\nAssume $\\nabla f$ is  Lipschitz continuous. Given $\\Qk\\in\\rnp$ and $\\xk\\in\\rn$, a function $\\mhatk:\\rp\\to\\R$ is a $(\\kef,\\keg; \\Qk)$-fully linear model of $f$ in $\\mathcal{B}(\\xk,\\dk;\\Qk)$ for some $\\dk>0$ if there exist constants $\\kappa_{ef}, \\kappa_{eg} >0$ independent of $k$ such that for all $\\s\\in\\rp$ with $\\normii{\\s}\\leq \\dk$,\n\\begin{equation}\\label{Eq36}\n%\\begin{split}\n\\abs{f(\\xk +\\Qk\\s)-\\mhatk(\\s)}\\leq \\kappa_{ef}\\dk^2\\quad \\mbox{and}\\quad\n\\normii{\\Qk^{\\top}\\nabla f(\\xk +\\Qk\\s)-\\nabla\\mhatk(\\s)}\\leq \\kappa_{eg}\\dk.\n%\\end{split}\n\\end{equation}\n\\end{definition}\nAs pointed out in~\\cite{CRsubspace2021}, the gradient condition in~\\eqref{Eq36} is due to the fact that given any fixed~$\\x$ and~$\\Q$, if $\\hat{f}(\\s):=f(\\x+\\Q\\s)$, then $\\nabla_{\\s} \\hat{f}(\\s)=\\Q^{\\top}\\nabla_{\\x} f(\\x+\\Q\\s)$. Moreover, denoting by $\\bm{I}$ the identity matrix in full-dimensional subspaces where $p=n$, then $(\\kef,\\keg; \\bm{I})$-fully linear models correspond to standard fully linear models~\\cite[Definition~6.1]{CoScVibook}.\n\n%Inspired by~\\cite{chen2018stochastic}, and \nRecalling Remark~\\ref{whyTheseSigmaAlgebras}, the next definition formalizing sufficient accuracy of probabilistic subspace models $\\Mhatk$ is a stochastic variant of Definition~\\ref{QkFullyLinearModels}, which generalizes~\\cite[Definition~3.4]{chen2018stochastic} to low-dimensional subspaces. The existence of such models will be rigorously demonstrated  in Theorem~\\ref{subspaceModel1} and Corollary~\\ref{CorQkFully}.\n","start":67168,"end":69099}]},{"id":"S013","semantic_type":"definition","kind":"definition","labels":["defProbQkfully","ikQProb"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"模型条件精度事件","source_units":[{"raw_tex":"\\begin{definition}\\label{defProbQkfully}\nA sequence $\\accolade{\\Mhatk}$ of random models is $\\beta_m$-probabilistically \\\\ $(\\kef,\\keg; \\Qk)$-fully linear with respect to the random sequence $\\accolade{\\mathcal{B}(\\Xk,\\Dk;\\QkRandom)}$ if the events\n\\[I_k^Q=\\accolade{\\mbox{Given}\\ \\Qk\\ \\mbox{and}\\ \\xk, \\Mhatk\\ \\mbox{is}\\ (\\kef,\\keg;\\Qk)\\mbox{-fully linear for}\\ f\\ \\mbox{in}\\ \\mathcal{B}(\\Xk,\\Dk;\\QkRandom)}\\]\nsatisfy the supermartingale-like condition\n\\begin{equation}\\label{ikQProb}\n\\prob{I^Q_k|\\fqalgebra}=\\E{\\iik|\\fqalgebra}\\geq \\beta_m.\n\\end{equation}\n\\end{definition}\nRecalling that estimates of unknown function values also need to be sufficiently accurate, we introduce a formalization of sufficient accuracy inspired by~\\cite{audet2019stomads,chen2018stochastic,dzahini2020expected,dzahini2020constrained,paquette2018stochastic} and motivated by~\\cite{CRsubspace2021} as follows.\n","start":69099,"end":69989}]},{"id":"S014","semantic_type":"definition","kind":"definition","labels":["accEstimates"],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"函数估计精度","source_units":[{"raw_tex":"\\begin{definition}\nGiven $\\ef>0$ and $\\Qk\\sk\\in\\rn, \\, \\xk\\in\\rn$, $\\fok$ and $\\fsk$ are called $\\ef$-accurate estimates of $f(\\xk)$ and $f(\\xk+\\Qk\\sk)$, respectively, for a given $\\dk$ if\n\\begin{equation}\\label{accEstimates}\n\\abs{\\fok-f(\\xk)}\\leq\\ef\\dk^2\\quad \\mbox{ and }\\quad  \\abs{\\fsk-f(\\xk+\\Qk\\sk)}\\leq\\ef\\dk^2.\n\\end{equation}\n\\end{definition}\n\n\nRecalling Remark~\\ref{whyTheseSigmaAlgebras},\n%and following~\\cite{audet2019stomads,chen2018stochastic,dzahini2020expected,dzahini2020constrained,paquette2018stochastic}, \nwe  extend  this definition to the following stochastic variant.\n","start":69989,"end":70578}]},{"id":"S015","semantic_type":"definition","kind":"definition","labels":["defProbAccEstim"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"估计条件精度事件","source_units":[{"raw_tex":"\\begin{definition}\\label{defProbAccEstim}\nA sequence of random estimates $\\accolade{\\Fok,\\Fsk}$ is said to be $\\beta_f$-probabilistically $\\ef$-accurate with respect to the corresponding sequence $\\accolade{\\Xk,\\Dk,\\QkRandom\\Sk}$ if the events \n\\begin{eqnarray*}\nJ_k^Q &=&\\left\\lbrace\\mbox{Given}\\ \\Qk\\ \\mbox{and}\\ \\xk, \\Fok \\mbox{ and } \\Fsk \\mbox{ are }\\ef\\mbox{-accurate estimates of } f(\\xk)  \\mbox{ and}\\right.\\\\\n& &\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad \\left.  f(\\xk+\\Qk\\sk), \\mbox{ respectively, for } \\Dk \\right\\rbrace\n\\end{eqnarray*}\n% \nsatisfy the supermartingale-like condition\n$\\prob{J^Q_k|\\fmqalgebra}\\geq \\beta_f.$\n\\end{definition}\n\n\n","start":70578,"end":71237}]},{"id":"S016","semantic_type":"derived_fact","kind":"derived_fact","labels":["prodBetaIneq"],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"真实迭代联合概率下界","source_units":[{"raw_tex":"Based on Definitions~\\ref{wellAlignedDefinition}, \\ref{defProbQkfully}, and~\\ref{defProbAccEstim}, {\\it true} iterations, in other words, those for which $\\iak\\iik\\ijk=1$, occur with a total probability of at least $\\tilde{\\beta}:=\\beta_f\\beta_m(1-\\beta_Q)$ conditioned on $\\falgebra$. Noting that by construction $\\iak$ is both $\\fqalgebra$-measurable and $\\fmqalgebra$-measurable since $\\fqalgebra\\subset\\fmqalgebra$, and that $\\iik$ is $\\fmqalgebra$-measurable, then for $F:= \\accolade{\\mathcal{A}_k\\cap I^Q_k\\cap J_k^Q}$, we have\n\\begin{equation}\\label{prodBetaIneq}\n\\begin{split}\n\\pr(F|\\falgebra)&=\\E{\\iak\\iik\\ijk|\\falgebra}\n=\\E{\\iak\\iik\\E{\\ijk|\\fmqalgebra} |\\falgebra}\\\\\n&\\geq \\beta_f\\E{\\iak\\E{\\iik|\\fqalgebra} |\\falgebra}\\geq \\beta_f\\beta_m\\E{\\iak|\\falgebra}\\geq \\tilde{\\beta}.\n%&\\geq \\beta_f\\beta_m(1-\\beta_Q),\n\\end{split}\n\\end{equation}\n","start":71237,"end":72083}]},{"id":"S017","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S017"],"declared_external":true,"declaration_text":["原先决条件列出的 billingsley1995ThirdEdition，Theorems 34.3 and 34.4；L1055 明确陈述并引用。"],"source_only":false,"title":"条件期望运算规则","source_units":[{"raw_tex":"The second equality and the first two inequalities used the  {\\it William's Tower Property} for conditional expectations~\\cite[Theorem~34.4.]{billingsley1995ThirdEdition}, namely, the fact that if ${\\ubar{z}}$ is integrable and the $\\sigma$-algebras $\\mathcal{G}_1$ and $\\mathcal{G}_2$ satisfy $\\mathcal{G}_1\\subset\\mathcal{G}_2$, then $\\E{{\\ubar{z}}|\\mathcal{G}_1}=\\E{\\E{{\\ubar{z}}|\\mathcal{G}_2}|\\mathcal{G}_1}$. Moreover, for a $\\sigma$-algebra $\\mathcal{G}$,  $\\E{{\\ubar{r}}{\\ubar{u}}|\\mathcal{G}}={\\ubar{r}}\\E{{\\ubar{u}}|\\mathcal{G}}$ if~${\\ubar{r}}$ is $\\mathcal{G}$-measurable and~${\\ubar{r}}$ and~${\\ubar{r}}{\\ubar{u}}$ are integrable~\\cite[Theorem~34.3.]{billingsley1995ThirdEdition}.\n","start":72083,"end":72777}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":["randomWalk"],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"真实迭代随机游走","source_units":[{"raw_tex":"\nNote that even though the present algorithmic framework does not distinguish true iterations from {\\it false} ones, it holds that $\\prob{\\underset{k\\to\\infty}{\\limsup}\\ \\ubar{w}_k=\\infty}=1$, where\n\\begin{equation}\\label{randomWalk}\n\\ubar{w}_k:=\\sum_{i=0}^k\\left(2\\mathds{1}_{\\mathcal{A}_i}\\mathds{1}_{I^Q_i}\\mathds{1}_{J^Q_i}-1\\right),\n\\end{equation}\nas will be seen later in the proof of Theorem~\\ref{liminfTheorem} provided $\\tilde{\\beta}>1/2$. This shows that true iterations occur sufficiently often for the convergence of Algorithm~\\ref{algoStoScalTR} to hold (see, e.g.,~\\cite[Theorem~3.6]{dzahini2020constrained} for a similar result).\n\n\n","start":72777,"end":73424}]},{"id":"S019","semantic_type":"derived_fact","kind":"derived_fact","labels":["otherProbIneqs","prodBetafBetam"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"其余条件概率下界","source_units":[{"raw_tex":"\\begin{remark}%\\label{remarkProdBetafBetam}\nCalculations similar to those in~\\eqref{prodBetaIneq} easily yield \n\\begin{eqnarray}\\label{prodBetafBetam}\n\\prob{I^Q_k\\cap J_k^Q|\\fqalgebra}\\geq \\beta_f\\beta_m, & & \\prob{J_k^Q|\\fqalgebra}\\geq \\beta_f,\\\\\n%\\end{equation}\n%\\begin{equation}\n\\label{otherProbIneqs}\n\\prob{I^Q_k|\\falgebra}\\geq\\beta_m & \\qquad \\mbox{ and } \\qquad & \\prob{J^Q_k|\\falgebra}\\geq\\beta_f,\n\\end{eqnarray}\nwhich will be useful in the proofs of Theorems~\\ref{zerothOrderScal} and~\\ref{liminfTheorem}, respectively. \n%and as was considered in~\\cite[Theorems~4.11]{chen2018stochastic} in similar situations.\n\\end{remark}\n","start":73424,"end":74056}]},{"id":"S020","semantic_type":"assumption","kind":"assumption","labels":["fGradfAssumption"],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"目标下界与正则性","source_units":[{"raw_tex":"In the remainder of the manuscript, the following inspired by~\\cite[Assumptions~4.1 and~4.3]{chen2018stochastic}, respectively, will be assumed.\n\\begin{assumption}\\label{fGradfAssumption}\nFor given $\\bm{\\x_0}$, $\\dmaxx$, and $\\bm{Q_0}$, let $\\mathscr{L}(\\bm{\\x_0};\\bm{Q_0})\\subset\\rn$ be the set containing all iterates of Algorithm~\\ref{algoStoScalTR}. Define the region $\\mathscr{L}^{\\star}(\\bm{\\x_0};\\bm{Q_0})$ considered by the algorithm realizations as $\\mathscr{L}^{\\star}(\\bm{\\x_0};\\bm{Q_0})=\\ds{\\underset{\\accolade{k\\, \\geq\\, 0,\\, \\xk\\in\\mathscr{L}(\\bm{\\x_0};\\bm{Q_0})}}{\\bigcup}}\\mathcal{B}(\\xk,\\dmaxx;\\Qk)$. Then $f(\\x)\\geq f_{\\min}$ for all $\\x\\in\\rn$ and for some constant $f_{\\min}>-\\infty$. Moreover, $f$ and its gradient are Lipschitz continuous on $\\mathscr{L}^{\\star}(\\bm{\\x_0};\\bm{Q_0})$.\n\\end{assumption}\n\n","start":90450,"end":91275}]},{"id":"S021","semantic_type":"assumption","kind":"assumption","labels":["boundedHessian"],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"模型海森矩阵界","source_units":[{"raw_tex":"\\begin{assumption}\\label{boundedHessian}\nThere exists some $\\kappa_h\\geq 1$ such that for all $k\\geq 0$, the Hessian $\\Hkhat$ of all realizations of $\\Mhatk$ satisfies $\\norme{\\Hkhat}\\leq \\kappa_h.$\n\\end{assumption}\n","start":91275,"end":91491}]},{"id":"S022","semantic_type":"lemma","kind":"lemma","labels":["LemmaC","deltaLemmaC"],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"准确模型估计保证成功","source_units":[{"raw_tex":"\\begin{lemma}\\label{LemmaC}\nLet Assumption~\\ref{boundedHessian} hold. Assume that $\\mhatk$ is $(\\kef,\\keg;\\Qk)$-fully linear and the estimates $\\accolade{\\fok,\\fsk}$ are $\\ef$-accurate with $\\ef\\leq\\kappa_{ef}$. Then the $k$th iteration is successful if \n\\begin{equation}\\label{deltaLemmaC}\n\\dk\\leq\\min\\accolade{\\frac{1}{\\kappa_h}, \\frac{1}{\\eta_2}, \\frac{\\kappa_{fcd}(1-\\eta_1)}{8\\kappa_{ef}}}\\normii{\\gkhat}.\n\\end{equation}\n\\end{lemma}","start":93481,"end":93918}]},{"id":"S023","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S023"],"declared_external":false,"source_only":true,"title":"成功引理的转引证明","source_units":[{"raw_tex":"\\begin{proof}\nThe proof immediately follows from that of~\\cite[Lemma~4.7]{chen2018stochastic} with minor modifications and is not presented here again.\n\\end{proof}\n","start":93920,"end":94084}]},{"id":"S024","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S024"],"declared_external":true,"declaration_text":["原先决条件列出的 chen2018stochastic, Lemma 4.7；作者于 L1292 调用。"],"source_only":false,"title":"外部成功性引理","source_units":[]},{"id":"S025","semantic_type":"lemma","kind":"lemma","labels":["LemmaD","LemmaDimprovement"],"member_ids":["S025"],"declared_external":false,"source_only":false,"title":"准确成功步的目标下降","source_units":[{"raw_tex":"\\begin{lemma}\\label{LemmaD}\nSuppose that Assumption~\\ref{boundedHessian} holds and the estimates $\\accolade{\\fok,\\fsk}$ are $\\ef$-accurate with $\\ef<\\frac{1}{4}\\eta_1\\eta_2\\kappa_{fcd}\\min\\accolade{\\frac{\\eta_2}{\\kappa_h}, 1}$. If the $k$th iteration is successful, then the improvement in $f$ is such that\n\\begin{equation}\\label{LemmaDimprovement}\nf(\\xkun)-f(\\xk)\\leq -C_2\\dk^2,\n\\end{equation}\nwhere $C_2=\\frac{1}{2}\\eta_1\\eta_2\\kappa_{fcd}\\min\\accolade{\\frac{\\eta_2}{\\kappa_h}, 1} -2\\ef>0$. \n\\end{lemma}","start":94242,"end":94747}]},{"id":"S026","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S026"],"declared_external":false,"source_only":true,"title":"准确下降引理的转引证明","source_units":[{"raw_tex":"\\begin{proof}\nAgain, the proof immediately follows from that of~\\cite[Lemma~4.8]{chen2018stochastic} with minor modifications and is not presented here.\n\\end{proof}\n","start":94749,"end":94914}]},{"id":"S027","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S027"],"declared_external":true,"declaration_text":["原先决条件列出的 chen2018stochastic, Lemma 4.8；作者于 L1304 调用。"],"source_only":false,"title":"外部准确下降引理","source_units":[]},{"id":"S028","semantic_type":"assumption","kind":"assumption","labels":["assumptionsOnEstimatesAndModels"],"member_ids":["S028"],"declared_external":false,"source_only":false,"title":"随机模型估计精度假设","source_units":[{"raw_tex":"\\subsection{Zeroth-order convergence result.}\\label{Section4p2}\nTo prove in Theorem~\\ref{zerothOrderScal} that the sequence of random trust-region radii converges to zero almost surely, we assume the following.\n\\begin{assumption}\\label{assumptionsOnEstimatesAndModels} \nThe sequences of estimates $\\accolade{\\Fok, \\Fsk}$ and models $\\accolade{\\Mhatk}$ generated by Algorithm~\\ref{algoStoScalTR} are, respectively, $\\beta_f$-probabilistically $\\ef$-accurate for \n$\\ef<\\min\\accolade{\\kappa_{ef}, \\frac{1}{4}\\eta_1\\eta_2\\kappa_{fcd}\\min\\accolade{\\frac{\\eta_2}{\\kappa_h}, 1}}$ and $\\beta_m$-probabilistically $(\\kef,\\keg;\\Qk)$-fully linear for some $\\beta_f, \\beta_m\\in (0,1)$.\n\\end{assumption}\n","start":99141,"end":99832}]},{"id":"S029","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"源文停止时刻定义","source_units":[{"raw_tex":"A formal definition from~\\cite{blanchet2016convergence} of the stopping time related to a discrete time stochastic process is as follows.\n\\begin{definition}\nA random variable~$\\tauEpsilon$ is called a stopping time with respect to a given discrete time stochastic process $\\accolade{\\X_k}_{k\\in\\N}$ if the event $\\accolade{\\tauEpsilon=k}$ belongs to the $\\sigma$-algebra $\\sigma\\left(\\X_1,\\dots,\\X_k \\right)$ generated by $\\X_1,\\dots,\\X_k$, for each $k\\in\\N$.\n\\end{definition}\n\n","start":119853,"end":120331}]},{"id":"S030","semantic_type":"definition","kind":"definition","labels":["dynamicsWk"],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"更新报酬框架的随机游走","source_units":[{"raw_tex":"Consider a stochastic process $\\accolade{\\left(\\Phik,\\Dk\\right)}_{k\\in\\N}$, where $\\Phik, \\Dk\\in [0,\\infty)$, and introduce on the same probability space as $\\accolade{\\left(\\Phik,\\Dk\\right)}_{k\\in\\N}$ a biased random walk process $\\accolade{\\Wtildek}_{k\\in\\N}$ obeying the following dynamics:\n\\begin{equation}\\label{dynamicsWk}\n\\prob{\\Wtildekun=1|\\F_k}=q\\qquad \\mbox{ and }\\qquad \\prob{\\Wtildekun=-1|\\F_k}=1-q,\n\\end{equation}\nwhere $q\\in (1/2, 1)$ and $\\F_k=\\sigma\\left( \\left(\\PhiZero,\\Dzero,\\WtildeZero\\right),\\dots,\\left(\\Phik,\\Dk,\\Wtildek\\right) \\right)$, with ${\\WtildeZero=1}$.\n\nLet $\\accolade{\\tauEpsilon}_{\\epsilon>0}$ be a family of stopping times with respect to $\\accoladekinN{\\F_k}$, parameterized by $\\epsilon$. A bound on $\\E{\\tauEpsilon}$ is derived in~\\cite{blanchet2016convergence,dzahini2020expected,paquette2018stochastic} under the following assumption.\n\n","start":120331,"end":121207}]},{"id":"S031","semantic_type":"assumption","kind":"assumption","labels":["assumptionRenewal","dynamicsRenewal"],"member_ids":["S031"],"declared_external":false,"source_only":false,"title":"更新报酬三项条件","source_units":[{"raw_tex":"\\begin{assumption}\\label{assumptionRenewal}\nThe following hold for the stochastic process $\\accoladekinN{\\left(\\Phik,\\Dk,\\Wtildek\\right)}$.\n\\begin{itemize}\n\\item[(i)] There exist $\\lambda\\in (0, \\infty)$ and $\\dmaxx=\\delta_0 e^{\\lambda j_{\\max}}$ for some $j_{\\max}\\in\\Z$ such that $\\Dk\\leq \\dmaxx$ for all $k$.\n\\item[(ii)] There exists $\\depsilon=\\delta_0 e^{\\lambda j_{\\epsilon}}$, for some $j_{\\epsilon}\\in\\Z$ with $j_{\\epsilon}\\leq 0$ such that for all $k$\n\\begin{equation}\\label{dynamicsRenewal}\n\\itksup\\Dkun\\geq \\itksup\\min\\accolade{\\Dk e^{\\lambda\\Wtildekun}, \\depsilon}, \n\\end{equation}\nwhere $\\Wtildekun$ satisfies~\\eqref{dynamicsWk}. \n\\item[(iii)] There exists a nondecreasing function $h:[0,\\infty)\\to (0,\\infty)$ and a constant $\\varrho>0$ such that for all $k$\n\\[\\E{\\Phikun-\\Phik|\\F_k}\\itksup\\leq-\\varrho h\\left(\\Dk\\right)\\itksup.\\]\n\\end{itemize}\n\\end{assumption}\n\nNoticing that the event $\\accolade{\\Dk\\geq\\depsilon}$ occurs sufficiently frequently on average, often $\\E{\\Phikun-\\Phik}$ can be bounded by some negative fixed constant \\cite{blanchet2016convergence}, thus allowing a bound on the expected stopping time $\\E{\\tauEpsilon}$, as stated next.\n\n","start":121207,"end":122374}]},{"id":"S032","semantic_type":"theorem","kind":"theorem","labels":["theorComplexity1"],"member_ids":["S032"],"declared_external":true,"declaration_text":["原先决条件明确允许 B0140：从 blanchet2016convergence 回顾的 renewal-reward expected stopping-time theorem，无文内证明。"],"source_only":false,"title":"外部期望停止时间界","source_units":[{"raw_tex":"\\begin{theorem}\\label{theorComplexity1}\nUnder Assumption~\\ref{assumptionRenewal}, \n$\\E{\\tauEpsilon}\\leq \\frac{q}{2q-1}\\cdot\\frac{\\PhiZero}{\\varrho h\\left(\\depsilon\\right)}+1.$\n\\end{theorem}\n","start":122374,"end":122564}]},{"id":"S033","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"算法停止指标与过程识别","source_units":[{"raw_tex":"\\subsection{Expected complexity result.}\\label{ExpectedSubsection}\n\nConsider the process $\\accolade{\\left(\\Phik,\\Dk,\\Wtildek\\right)}_{k\\in\\N}$, where~$\\Phik$ is defined in Theorem~\\ref{zerothOrderScal}, $\\Dk$ is the trust-region radius, and $\\Wtildek$ is  defined by $\\Wtildek=2\\left(\\iak\\iik\\ijk-\\frac{1}{2} \\right)$.  Given $\\epsilon\\in (0,1)$, let $\\tauEpsilon$ be the number of iterations required by Algorithm~\\ref{algoStoScalTR} to first drive the norm of the gradient of~$f$ below $\\epsilon$:  \n\\[\\tauEpsilon:=\\inf \\accolade{k\\in\\N: \\normii{\\nabla f(\\Xk)}\\leq\\epsilon}.\\]\n","start":122565,"end":123144}]},{"id":"S034","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"停止时刻滤过识别断言","source_units":[{"raw_tex":"Then $\\tauEpsilon$ is a stopping time for the stochastic process generated by Algorithm~\\ref{algoStoScalTR}, and hence for $\\accolade{\\left(\\Phik,\\Dk,\\Wtildek\\right)}_{k\\in\\N}$ \\cite{blanchet2016convergence,dzahini2020expected,paquette2018stochastic}. More precisely, the occurrence of $\\accolade{\\tauEpsilon=k}$ can be determined by observing $\\left(\\PhiZero,\\Dzero,\\WtildeZero\\right)$, $\\dots$ $(\\Phikmun,\\Dkmun,\\Wtildekmun)$, which means that $\\tauEpsilon$ is a stopping time with respect to the filtration $\\accoladekinN{\\falgebra}$. To bound $\\E{\\tauEpsilon}$ using Theorem~\\ref{theorComplexity1}, we show next that Assumption~\\ref{assumptionRenewal} holds for $\\accolade{\\left(\\Phik,\\Dk,\\Wtildek\\right)}_{k\\in\\N}$.\n\n\n","start":123144,"end":123867}]},{"id":"S035","semantic_type":"derived_fact","kind":"derived_fact","labels":["PhikIncrSecond"],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"向较小历史转移漂移","source_units":[{"raw_tex":"From~\\eqref{PhikIncrementBound}, since $\\falgebra\\subset\\fqalgebra$, we observe that\\footnote{Unlike the analysis in~\\cite{blanchet2016convergence}, the first equality of~\\eqref{PhikIncrSecond} is essential and due to the fact that in the present framework $\\tauEpsilon$ is a stopping time with respect to $\\sigma$-algebras $\\falgebra$ that are smaller than $\\fqalgebra$ with respect to which $\\accoladekinN{\\Phik}$ was proved in Theorem~\\ref{zerothOrderScal} Eq.~\\eqref{PhikIncrementBound} to be a supermartingale.} for all $k\\in\\N$:\n\\begin{equation}\\label{PhikIncrSecond}\n\\E{\\Phikun-\\Phik|\\falgebra}=\\E{\\E{\\Phikun-\\Phik|\\fqalgebra}|\\falgebra}\\leq\\E{-\\varrho\\Dk^2|\\falgebra}=-\\varrho\\Dk^2.\n\\end{equation}\nNote that~\\eqref{PhikIncrSecond} holds for any realizations of the random variables $\\E{\\Phikun-\\Phik|\\falgebra}$ and $\\Dk^2$ and hence on the event $\\accolade{\\tauEpsilon >k}$ in particular, which shows that Assumption~\\ref{assumptionRenewal}-$(iii)$ holds with the constant $\\varrho=\\frac{1}{2}\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})$ of Theorem~\\ref{zerothOrderScal} and $h(t)=t^2$. Assumption~\\ref{assumptionRenewal}-$(i)$ automatically follows from the initialization strategy in Algorithm~\\ref{algoStoScalTR} with $\\lambda=\\ln(\\gamma)$. Before showing by means of Lemma~\\ref{smallLemma}, inspired by~\\cite[Lemma~7]{blanchet2016convergence}, that~\\eqref{dynamicsRenewal} is satisfied, we first define the constant \n","start":123867,"end":125290}]},{"id":"S036","semantic_type":"definition","kind":"definition","labels":["deltaEpsilonDefinition"],"member_ids":["S036"],"declared_external":false,"source_only":false,"title":"复杂度半径阈值","source_units":[{"raw_tex":"\\begin{equation}\\label{deltaEpsilonDefinition}\n\\depsilon:=\\frac{\\epsilon}{\\xi}, \\quad\\mbox{with } \\xi\\geq  \\frac{1}{\\alpha_Q}\\max\\accolade{\\kappa_h+\\kappa_{eg}, \\eta_2+\\kappa_{eg}, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}+\\kappa_{eg}}=:\\hat{\\xi}, \n\\end{equation}\n","start":125290,"end":125556}]},{"id":"S037","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"半径格点约定","source_units":[{"raw_tex":"inspired by the proof of Theorem~\\ref{liminfTheorem}, especially~\\eqref{ineq4}. Then, following ~\\cite{blanchet2016convergence} exactly, one can assume without loss of generality that $\\depsilon=\\gamma^i\\delta_0$ for some integer $i=:j_{\\epsilon}\\leq 0$, whence $\\Dk=\\gamma^{{\\ubar{i}}_k}\\depsilon$ for any $k$ and some integer~${\\ubar{i}}_k$.\n","start":125556,"end":125900}]},{"id":"S038","semantic_type":"theorem","kind":"theorem","labels":[],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"STARS 期望迭代复杂度","source_units":[{"raw_tex":"\\begin{theorem}\nLet all the assumptions made in Theorem~\\ref{liminfTheorem} hold. Then \n\\[\\E{\\tauEpsilon}\\leq \\frac{\\tilde{\\beta}}{2\\tilde{\\beta}-1}\\cdot\\frac{\\PhiZero\\tilde{\\xi}^2}{\\varrho \\epsilon^2}+1\\]\nfor some $\\tilde{\\xi}\\geq\\hat{\\xi}$, where $\\varrho$ is the constant of Theorem~\\ref{zerothOrderScal}, $\\tilde{\\beta}$ is the same constant of Lemma~\\ref{smallLemma}, and $\\hat{\\xi}$ is defined in~\\eqref{deltaEpsilonDefinition}.\n\\end{theorem}","start":127519,"end":127967}]},{"id":"S039","semantic_type":"lemma","kind":"lemma","labels":["LemmaA","LemmaAimprovement"],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"按模型梯度保证下降","source_units":[{"raw_tex":"\\begin{lemma}\\label{LemmaA}\nSuppose that the function $\\mhatk$ is a $(\\kef,\\keg;\\Qk)$-fully linear model for~$f$ in $\\B{\\xk}{\\dk;\\Qk}$. Then the trial step $\\sk$ leads to an improvement in $f$ such that\n\\begin{equation}\\label{LemmaAimprovement}\nf(\\xk+\\Qk\\sk)-f(\\xk)\\leq -\\frac{\\kappa_{fcd}}{4}\\normii{\\gkhat}\\dk,\n%\\end{equation}\n%\\begin{equation}\\label{deltaLemmaA}\n\\quad\n\\mbox{if }\n\\dk\\leq \\min\\accolade{\\frac{1}{\\kappa_h},\\frac{\\kappa_{fcd}}{8\\kappa_{ef}}}\\normii{\\gkhat}.\n\\end{equation}\n\\end{lemma}","start":91779,"end":92280}]},{"id":"S040","semantic_type":"lemma","kind":"lemma","labels":["LemmaB2","LemmaB2improvement","deltaLemmaB2"],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"按投影真梯度保证下降","source_units":[{"raw_tex":"\\begin{lemma}\\label{LemmaB2}\nLet Assumption~\\ref{boundedHessian} hold, and suppose the model $\\mhatk$ is $(\\kef,\\keg;\\Qk)$-fully linear. Define $C_1=\\frac{\\kappa_{fcd}}{4} \\max\\accolade{\\frac{\\kappa_h}{\\kappa_h+\\kappa_{eg}}, \\frac{8\\kappa_{ef}}{8\\kappa_{ef}+\\kappa_{fcd}\\kappa_{eg}}}$. Then the trial step $\\sk$ leads to an improvement in $f$ such that \n\n\\begin{equation}\\label{LemmaB2improvement}\nf(\\xk+\\Qk\\sk)-f(\\xk)\\leq -C_1\\normii{\\QkTranspose\\nabla f(\\xk)}\\dk,\n\\end{equation}\n\\begin{equation}\\label{deltaLemmaB2}\n\\mbox{if}\\quad\n\\dk\\leq \\min\\accolade{\\frac{1}{\\kappa_h+\\kappa_{eg}}, \\frac{1}{\\frac{8\\kappa_{ef}}{\\kappa_{fcd}}+\\kappa_{eg}}}\\normii{\\QkTranspose\\nabla f(\\xk)}. \n\\end{equation}\n\\end{lemma}","start":92546,"end":93252}]},{"id":"S041","semantic_type":"lemma","kind":"lemma","labels":["LemmaE","LemmaEincrease"],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"错误接受步的增幅界","source_units":[{"raw_tex":"\\begin{lemma}\\label{LemmaE}\nUnder Assumptions~\\ref{QmaxAssumption} and~\\ref{fGradfAssumption}, assume that Algorithm~\\ref{algoStoScalTR} erroneously accepts a step $\\sk$ leading to an increase in the objective function when $\\norme{\\QkTranspose\\nabla f(\\xk)}\\geq \\zeta\\dk$ for some constant $\\zeta>0$, and let $C_3(\\zeta)=1+\\frac{3L_g}{2\\zeta}\\Qmax^2$. Then the increase in $f$ is such that\n\\begin{equation}\\label{LemmaEincrease}\nf(\\xk+\\Qk\\sk)-f(\\xk)\\leq C_3(\\zeta)\\norme{\\QkTranspose\\nabla f(\\xk)}\\dk.\n\\end{equation}\n\\end{lemma}","start":95221,"end":95750}]},{"id":"S042","semantic_type":"lemma","kind":"lemma","labels":["partialMeasurable"],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"偏导数的 Borel 可测性","source_units":[{"raw_tex":"\\begin{lemma}\\label{partialMeasurable}\nAny partial derivative \n$\\frac{\\partial f}{\\partial x^i}: \\left(\\rn, \\mathscr{B}(\\rn)\\right)\\to \\left(\\R, \\mathscr{B}(\\R)\\right),\\ i\\in\\intbracket{1,n} $,\nof a differentiable function $f$ is a Borel measurable function.\n\\end{lemma}","start":96299,"end":96569}]},{"id":"S043","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"偏导可测性的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nLet $(\\bm{e_1},\\bm{e_2},\\dots,\\bm{e_n})$ be the canonical basis of $\\rn$. For any $i\\in\\intbracket{1,n} $ and $\\bm{x}=(x^1,x^2,\\dots,x^n)$, define, for all $j\\in\\N$,\n$D_{ij}(\\bm{x})=\\frac{f(\\bm{x}+h_j\\bm{e_i})-f(\\bm{x})}{h_j}$,\nwhere $\\accolade{h_j}_{j\\in\\N}$ is a sequence of positive real numbers converging to~$0$; $f$ is differentiable and hence continuous, and thus is Borel measurable~\\cite[Theorem~13.2]{billingsley1995ThirdEdition}. Therefore, since for all $i\\in\\intbracket{1,n}$, $\\underset{j\\to\\infty}{\\lim}D_{ij}(\\bm{x})=\\frac{\\partial f}{\\partial x^i}(\\bm{x}),$\nthen $\\frac{\\partial f}{\\partial x^i}$ is also Borel measurable~\\cite[Theorem~13.3 and Theorem~13.4-(ii)]{billingsley1995ThirdEdition}.\n\\end{proof}","start":96571,"end":97307}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"差商极限保持可测","source_units":[{"raw_tex":"Let $(\\bm{e_1},\\bm{e_2},\\dots,\\bm{e_n})$ be the canonical basis of $\\rn$. For any $i\\in\\intbracket{1,n} $ and $\\bm{x}=(x^1,x^2,\\dots,x^n)$, define, for all $j\\in\\N$,\n$D_{ij}(\\bm{x})=\\frac{f(\\bm{x}+h_j\\bm{e_i})-f(\\bm{x})}{h_j}$,\nwhere $\\accolade{h_j}_{j\\in\\N}$ is a sequence of positive real numbers converging to~$0$; $f$ is differentiable and hence continuous, and thus is Borel measurable~\\cite[Theorem~13.2]{billingsley1995ThirdEdition}. Therefore, since for all $i\\in\\intbracket{1,n}$, $\\underset{j\\to\\infty}{\\lim}D_{ij}(\\bm{x})=\\frac{\\partial f}{\\partial x^i}(\\bm{x}),$\nthen $\\frac{\\partial f}{\\partial x^i}$ is also Borel measurable~\\cite[Theorem~13.3 and Theorem~13.4-(ii)]{billingsley1995ThirdEdition}.\n","start":96585,"end":97296}]},{"id":"S045","semantic_type":"lemma","kind":"lemma","labels":["indicatorMeasurable"],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"梯度分区指标的可测性","source_units":[{"raw_tex":"\\begin{lemma}\\label{indicatorMeasurable}\nLet $\\zeta\\geq 0$ be a constant, and consider the event $\\Gamma_k:=\\accolade{\\Gk\\geq 0}$, where $\\Gk:=\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}-\\zeta\\Dk$. Then $\\Gk$, $\\igk$, and $\\igkc$ are $\\fqalgebra$-measurable.\n\\end{lemma}","start":97310,"end":97577}]},{"id":"S046","semantic_type":"proof","kind":"proof","labels":["ukRandomMeasurable"],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"分区指标的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nBy construction, all the entries $\\left(\\QkRandom\\right)_{ij},\\ (i,j)\\in\\intbracket{1,n}\\times\\intbracket{1,p}$ of $\\QkRandom$ are $\\fqalgebra$-measurable, while $\\Dk$ and the components $\\Xk^i, \\ i\\in\\intbracket{1, n}$ of $\\Xk$ are $\\falgebra$-measurable and hence $\\fqalgebra$-measurable since $\\falgebra\\subseteq\\fqalgebra$. Thus, for any $i\\in\\intbracket{1, n}$, $\\frac{\\partial f}{\\partial x^i}(\\Xk)$ is $\\fqalgebra$-measurable as the composition of the Borel measurable function $\\frac{\\partial f}{\\partial x^i}$ (from Lemma~\\ref{partialMeasurable}) and the random map $\\Xk:(\\Omega,\\fqalgebra)\\to\\left(\\rn, \\mathscr{B}(\\rn)\\right)$~\\cite[Theorem~13.3 and Theorem~13.1-(ii)]{billingsley1995ThirdEdition}. Recall that  sums and products of measurable functions are also measurable~\\cite[Theorem~13.3]{billingsley1995ThirdEdition}, \n%%consequence of Theorem~13.3\nand note that the random variable \n\\begin{equation}\\label{ukRandomMeasurable}\n\\ubar{u}_k:=\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}^2=\\sum_{j=1}^p\\left(\\sum_{i=1}^n\\left(\\QkRandom\\right)_{ij}\\frac{\\partial f}{\\partial x^i}(\\Xk) \\right)^2 \n\\end{equation}\nis consequently $\\fqalgebra$-measurable. Thus $\\Gk=\\sqrt{{\\ubar{u}_k}}-\\zeta\\Dk$ is also $\\fqalgebra$-measurable. The $\\fqalgebra$-measurability of $\\Gk$ ensures in particular that $\\Gamma_k=\\Gk^{-1}([0, \\infty))\\in\\fqalgebra$ since $[0, \\infty)$ is a Borel set of $\\R$, whence the {\\it simple} real functions $\\igk$ and $\\igkc$~\\cite[Equation~(13.3)]{billingsley1995ThirdEdition} are also $\\fqalgebra$-measurable. \n\\end{proof}","start":97579,"end":99139}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":["ukRandomMeasurable"],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"组合可测量并取原像","source_units":[{"raw_tex":"By construction, all the entries $\\left(\\QkRandom\\right)_{ij},\\ (i,j)\\in\\intbracket{1,n}\\times\\intbracket{1,p}$ of $\\QkRandom$ are $\\fqalgebra$-measurable, while $\\Dk$ and the components $\\Xk^i, \\ i\\in\\intbracket{1, n}$ of $\\Xk$ are $\\falgebra$-measurable and hence $\\fqalgebra$-measurable since $\\falgebra\\subseteq\\fqalgebra$. Thus, for any $i\\in\\intbracket{1, n}$, $\\frac{\\partial f}{\\partial x^i}(\\Xk)$ is $\\fqalgebra$-measurable as the composition of the Borel measurable function $\\frac{\\partial f}{\\partial x^i}$ (from Lemma~\\ref{partialMeasurable}) and the random map $\\Xk:(\\Omega,\\fqalgebra)\\to\\left(\\rn, \\mathscr{B}(\\rn)\\right)$~\\cite[Theorem~13.3 and Theorem~13.1-(ii)]{billingsley1995ThirdEdition}. Recall that  sums and products of measurable functions are also measurable~\\cite[Theorem~13.3]{billingsley1995ThirdEdition}, \n%%consequence of Theorem~13.3\nand note that the random variable \n\\begin{equation}\\label{ukRandomMeasurable}\n\\ubar{u}_k:=\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}^2=\\sum_{j=1}^p\\left(\\sum_{i=1}^n\\left(\\QkRandom\\right)_{ij}\\frac{\\partial f}{\\partial x^i}(\\Xk) \\right)^2 \n\\end{equation}\nis consequently $\\fqalgebra$-measurable. Thus $\\Gk=\\sqrt{{\\ubar{u}_k}}-\\zeta\\Dk$ is also $\\fqalgebra$-measurable. The $\\fqalgebra$-measurability of $\\Gk$ ensures in particular that $\\Gamma_k=\\Gk^{-1}([0, \\infty))\\in\\fqalgebra$ since $[0, \\infty)$ is a Borel set of $\\R$, whence the {\\it simple} real functions $\\igk$ and $\\igkc$~\\cite[Equation~(13.3)]{billingsley1995ThirdEdition} are also $\\fqalgebra$-measurable. \n","start":97593,"end":99128}]},{"id":"S048","semantic_type":"assumption","kind":"assumption","labels":["nuChoiceScal","probChoices","zerothOrderScal","zetaChoice"],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"漂移分析参数限制","source_units":[{"raw_tex":"Let all assumptions that were made in Lemmas~\\ref{LemmaA}-\\ref{LemmaD} hold with the same constants $C_1, C_2$, and $C_3:=C_3(\\zeta)$, for some fixed $\\zeta>0$ satisfying\n\\begin{equation}\\label{zetaChoice}\n\\zeta\\geq \\kappa_{eg}+\\max\\accolade{\\eta_2,\\kappa_h, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}}.\n\\end{equation}\nLet $\\nu\\in (0,1)$ be chosen according to \n\\begin{equation}\\label{nuChoiceScal}\n\\frac{\\nu}{1-\\nu} \\geq \\frac{8\\gamma^2}{\\min\\accolade{C_2, \\zeta C_1}}.  \n\\end{equation}\nAssume further that Assumption~\\ref{assumptionsOnEstimatesAndModels} holds with $\\beta_f, \\beta_m\\in (0,1)$ satisfying\\footnote{In~\\eqref{probChoices}, the result is intentionally presented with $(1-\\beta_f)+(1-\\beta_m)$ instead of $2-\\beta_f-\\beta_m$ in order to emphasize one of its differences compared with~\\cite{blanchet2016convergence,chen2018stochastic} in which a term similar to $(1-\\beta_f)(1-\\beta_m)$ was derived.}\n\\begin{equation}\\label{probChoices}\n\\frac{\\beta_f\\beta_m-\\frac{1}{2}}{(1-\\beta_f)+(1-\\beta_m)}\\geq \\frac{C_3}{C_1}\\quad \\mbox{ and }\\quad \\frac{\\beta_f}{1-\\beta_f}\\geq \\frac{2\\nu\\zeta\\gamma^2C_3}{(1-\\nu)(\\gamma^2-1)}+2\\gamma^2.\n\\end{equation}\n","start":101216,"end":102375}]},{"id":"S049","semantic_type":"definition","kind":"definition","labels":["zerothOrderScal"],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"势函数与漂移常数","source_units":[{"raw_tex":"Define $\\varrho=\\frac{1}{2}\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})>0$. Then, the random function $\\Phik:=\\nu (f(\\Xk)-f_{\\min})+(1-\\nu)\\Dk^2$ satisfies\n","start":102375,"end":102523}]},{"id":"S050","semantic_type":"derived_fact","kind":"derived_fact","labels":["PhikIncrementBound","zerothOrderScal"],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"条件负漂移结论","source_units":[{"raw_tex":"\\begin{equation}\\label{PhikIncrementBound}\n\\E{\\Phikun-\\Phik|\\fqalgebra}\\leq -\\varrho\\Dk^2 \\quad \\mbox{for all}\\ k\\in\\N,\n\\end{equation}\n","start":102523,"end":102658}]},{"id":"S051","semantic_type":"derived_fact","kind":"derived_fact","labels":["seriesResult","zerothOrderScal"],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"半径平方几乎处处可和","source_units":[{"raw_tex":"which implies that\n\\begin{equation}\\label{seriesResult}\n\\sum_{k=0}^{\\infty}\\Dk^2<\\infty\\quad \\mbox{almost surely.}\n\\end{equation}\n\\end{theorem}","start":102658,"end":102801}]},{"id":"S052","semantic_type":"theorem","kind":"theorem","labels":["PhikIncrementBound","nuChoiceScal","probChoices","seriesResult","zerothOrderScal","zetaChoice"],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"负漂移与半径平方可和","source_units":[{"raw_tex":"\\begin{theorem}\\label{zerothOrderScal}\nLet all assumptions that were made in Lemmas~\\ref{LemmaA}-\\ref{LemmaD} hold with the same constants $C_1, C_2$, and $C_3:=C_3(\\zeta)$, for some fixed $\\zeta>0$ satisfying\n\\begin{equation}\\label{zetaChoice}\n\\zeta\\geq \\kappa_{eg}+\\max\\accolade{\\eta_2,\\kappa_h, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}}.\n\\end{equation}\nLet $\\nu\\in (0,1)$ be chosen according to \n\\begin{equation}\\label{nuChoiceScal}\n\\frac{\\nu}{1-\\nu} \\geq \\frac{8\\gamma^2}{\\min\\accolade{C_2, \\zeta C_1}}.  \n\\end{equation}\nAssume further that Assumption~\\ref{assumptionsOnEstimatesAndModels} holds with $\\beta_f, \\beta_m\\in (0,1)$ satisfying\\footnote{In~\\eqref{probChoices}, the result is intentionally presented with $(1-\\beta_f)+(1-\\beta_m)$ instead of $2-\\beta_f-\\beta_m$ in order to emphasize one of its differences compared with~\\cite{blanchet2016convergence,chen2018stochastic} in which a term similar to $(1-\\beta_f)(1-\\beta_m)$ was derived.}\n\\begin{equation}\\label{probChoices}\n\\frac{\\beta_f\\beta_m-\\frac{1}{2}}{(1-\\beta_f)+(1-\\beta_m)}\\geq \\frac{C_3}{C_1}\\quad \\mbox{ and }\\quad \\frac{\\beta_f}{1-\\beta_f}\\geq \\frac{2\\nu\\zeta\\gamma^2C_3}{(1-\\nu)(\\gamma^2-1)}+2\\gamma^2.\n\\end{equation}\nDefine $\\varrho=\\frac{1}{2}\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})>0$. Then, the random function $\\Phik:=\\nu (f(\\Xk)-f_{\\min})+(1-\\nu)\\Dk^2$ satisfies\n\\begin{equation}\\label{PhikIncrementBound}\n\\E{\\Phikun-\\Phik|\\fqalgebra}\\leq -\\varrho\\Dk^2 \\quad \\mbox{for all}\\ k\\in\\N,\n\\end{equation}\nwhich implies that\n\\begin{equation}\\label{seriesResult}\n\\sum_{k=0}^{\\infty}\\Dk^2<\\infty\\quad \\mbox{almost surely.}\n\\end{equation}\n\\end{theorem}","start":101177,"end":102801}]},{"id":"S053","semantic_type":"proof","kind":"proof","labels":["GammakBarGkJk","GammakBarGkJkBar","GkCase2","GkbarCase2","bOne","bThree","bThreeRandom2","bThreeRandom3","bTwo","bTwoRandom","bTwoThree","huit","last","un"],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"负漂移定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nThe proof is inspired by those of~\\cite[Theorem~3]{blanchet2016convergence} and~\\cite[Theorem~4.11]{chen2018stochastic}. \nThe overall goal is to prove~\\eqref{PhikIncrementBound}. Indeed noticing that $\\Phik>0$ and then taking expectations on both sides of the inequality in~\\eqref{PhikIncrementBound}, we obtain~\\eqref{seriesResult} (see, e.g.,~\\cite[Theorem~3]{dzahini2020expected} for details). Let $\\phi_k$ denote realizations of $\\Phik$, and recall that on all successful iterations, $\\xkun=\\xk+\\Qk\\sk$ and $\\dkun=\\min\\accolade{\\gamma\\dk,\\dmaxx}$,  implying\n\\begin{equation}\\label{un}\n\\phi_{k+1}-\\phi_k\\leq \\nu\\left(f(\\xkun)-f(\\xk)\\right)+(1-\\nu)(\\gamma^2-1)\\dk^2,\n\\end{equation}\nwhile on unsuccessful iterations, $\\xkun=\\xk$ and $\\dkun=\\gamma^{-1}\\dk$, in which case\n\\begin{equation}\\label{bOne}\n\\phi_{k+1}-\\phi_k\\leq(1-\\nu)\\left(\\frac{1}{\\gamma^2}-1\\right)\\dk^2=:b_1<0.\n\\end{equation}\n\nRecall the event $\\Gamma_k=\\accolade{\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\geq\\zeta\\Dk}$ of Lemma~\\ref{indicatorMeasurable} with $\\zeta$ satisfying ~\\eqref{zetaChoice}. The proof considers two cases: $\\igk=1$ and $\\igkc=1$. Inspired by~\\cite[proof of Theorem~4.11]{chen2018stochastic}, Case~1 aims to show that\n\\begin{equation}\\label{huit}\n%\\begin{split}\n\\E{\\igk\\left(\\Phikun-\\Phik\\right)|\\fqalgebra}\\leq-2\\igk(1-\\nu)(\\gamma^2-1)\\Dk^2\n\\leq -\\frac{1}{2}\\igk\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})\\Dk^2, \n%\\end{split}\n\\end{equation}\nwhere the last inequality follows from $1-\\frac{1}{\\gamma^2}<\\gamma^2-1$ and $\\beta_f\\leq 1$. On the other hand, inspired by~\\cite[proof of Theorem~3]{blanchet2016convergence}, Case~2 shows that\n\\begin{equation}\\label{last}\n\\E{\\igkc\\left(\\Phikun-\\Phik\\right)|\\fqalgebra}\\leq-\\frac{1}{2}\\igkc\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})\\Dk^2.\n\\end{equation}\nThus, combining~\\eqref{huit} and~\\eqref{last} leads to~\\eqref{PhikIncrementBound}.\n\n$ $\\\\\n\\textbf{Case~1: $\\igk=1$}\\\\\n$ $\\\\\nUnlike the proof of~\\cite[Theorem~4.11, Case~1]{chen2018stochastic} where four subcases were considered, only three are analyzed next.\n\n%\\begin{itemize}\n%\\item[(i)] \n\\noindent \\textit{(Subcase 1i)}\nGood model ($\\iik=1$) and good estimates ($\\ijk=1$). By~\\eqref{zetaChoice}, \n\\[\\normii{\\QkTranspose\\nabla f(\\xk)}\\geq \\max\\accolade{\\eta_2+\\kappa_{eg}, \\kappa_h+\\kappa_{eg}, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}}+\\kappa_{eg}}\\dk,\\]\nimplying in particular~\\eqref{deltaLemmaB2} and hence a decrease in $f$ according to~\\eqref{LemmaB2improvement}. Moreover, by $(\\kef,\\keg;\\Qk)$-full linearity, \n\\[\\normii{\\gkhat}\\geq \\normii{\\QkTranspose\\nabla f(\\xk)}-\\kappa_{eg}\\dk\\geq \\max\\accolade{\\eta_2,\\kappa_h, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}}\\dk, \\]\nthus implying~\\eqref{deltaLemmaC}; and since $\\ef\\leq \\kappa_{ef}$ per assumptions of Lemma~\\ref{LemmaC}, the iteration is successful. Consequently,~\\eqref{un} together with~\\eqref{LemmaB2improvement} yields\n\\begin{equation}\\label{bTwo}\n\\begin{split}\n\t\\phi_{k+1}-\\phi_k &\\leq -\\nu C_1\\normii{\\QkTranspose\\nabla f(\\xk)}\\dk+(1-\\nu)(\\gamma^2-1)\\dk^2=:b_2\\\\\n\t&\\leq \\left(-\\nu C_1\\zeta+(1-\\nu)(\\gamma^2-1)\\right)\\dk^2<0,\n\\end{split}\n\\end{equation}\nwhere the last inequality follows from~\\eqref{nuChoiceScal}. Denoting by ${\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)$ the random variable with realizations $b_2<0$, with ${\\ubar{u}}_k$ defined in~\\eqref{ukRandomMeasurable}, we have from~\\eqref{bTwo} that\n\\begin{equation}\\label{bTwoRandom}\n\\igk\\iik\\ijk\\left(\\Phikun-\\Phik\\right)\\leq \\igk\\iik\\ijk{\\ubar{b}}_2({\\ubar{u}}_k,\\Dk).\n\\end{equation}\n%\\item[(ii)] \n\\noindent \\textit{(Subcase 1ii)}\nBad model ($\\iikc=1$) and good estimates ($\\ijk=1$). In this case, regardless of the iteration type (i.e., successful or unsuccessful), the change in $\\phi_k$ can always be bounded by using~\\eqref{un} and~\\eqref{LemmaEincrease} as follows:\n\\begin{equation}\\label{bThree}\n\\phi_{k+1}-\\phi_k\\leq\\nu C_3\\norme{\\QkTranspose\\nabla f(\\xk)}\\dk+(1-\\nu)(\\gamma^2-1)\\dk^2=:b_3.\n\\end{equation}\nDenoting by ${\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)$ the random variable with realizations $b_3>0$, from~\\eqref{bThree} we have that\n\\begin{equation}\\label{bThreeRandom2}\n\\igk\\iikc\\ijk\\left(\\Phikun-\\Phik\\right)\\leq \\igk\\iikc\\ijk{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)\\leq \\igk\\iikc{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk).\n\\end{equation}\n%\\item[(iii)] \n\\noindent \\textit{(Subcase 1iii)}\nBad estimates ($\\ijkc=1$). From \\textit{Subcase 1ii}, it always holds that \n\\begin{equation}\\label{bThreeRandom3}\n\\igk\\ijkc\\left(\\Phikun-\\Phik\\right)\\leq \\igk\\ijkc{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk).\n\\end{equation}\n%\\end{itemize}\n\nWith the subcases thus complete, since the random variables $\\igk, {\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)$, and ${\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)$ are $\\fqalgebra$-measurable thanks to the proof of Lemma~\\ref{indicatorMeasurable}, combining~\\eqref{bTwoRandom},~\\eqref{bThreeRandom2}, and~\\eqref{bThreeRandom3} and taking expectations with respect to $\\fqalgebra$, we have\n\\begin{equation}\\label{bTwoThree}\n\\begin{split}\n\\E{\\igk\\left(\\Phikun-\\Phik\\right)|\\fqalgebra}&\\leq \\igk{\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)\\E{\\iik\\ijk|\\fqalgebra}\\\\\n&+\\igk{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)\\left(\\E{\\iikc|\\fqalgebra}+\\E{\\ijkc|\\fqalgebra}\\right)\\\\\n&\\leq \\igk\\left(\\beta_f\\beta_m{\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)+{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)(1-\\beta_m+1-\\beta_f)\\right)=:\\igk {\\ubar{b}}_{2,3}({\\ubar{u}}_k,\\Dk),\n\\end{split}\n\\end{equation}\nwhere the last inequality follows from~\\eqref{ikQProb} and~\\eqref{prodBetafBetam} and the fact that ${\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)<0$ while ${\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)>0$.  Note that \n\\begin{eqnarray*}\n{\\ubar{b}}_{2,3}({\\ubar{u}}_k,\\Dk)&=&\\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\left[-\\beta_f\\beta_m C_1+(2-\\beta_f-\\beta_m)C_3\\right]\\\\\n& &+(1-\\nu)(\\gamma^2-1)(2-\\beta_f-\\beta_m+\\beta_f\\beta_m)\\Dk^2\\\\\n&\\leq& \\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\left[-\\beta_f\\beta_m C_1+(2-\\beta_f-\\beta_m)C_3\\right] + 2(1-\\nu)(\\gamma^2-1)\\Dk^2,\n\\end{eqnarray*}\nwhere the last inequality follows from $2-\\beta_f-\\beta_m+\\beta_f\\beta_m=1+(1-\\beta_f)(1-\\beta_m)\\leq 2$.\n\nFor $\\beta_f$ and $\\beta_m$ chosen according to the first condition in~\\eqref{probChoices}, and for $\\nu$ satisfying~\\eqref{nuChoiceScal}, the following holds:\n\\[\\beta_f\\beta_m C_1 - (2-\\beta_f-\\beta_m)C_3\\geq \\frac{1}{2}C_1\\geq 2\\frac{2(1-\\nu)(\\gamma^2-1)}{\\nu\\zeta}.\\]\nHence, since $\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\geq \\zeta\\Dk$, then \n\\begin{eqnarray*}\n{\\ubar{b}}_{2,3}({\\ubar{u}}_k,\\Dk)&\\leq& -\\frac{1}{2}\\left[\\beta_f\\beta_m C_1-(2-\\beta_f-\\beta_m)C_3\\right]\\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\\\\n&\\leq& -\\frac{1}{4}C_1\\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\leq -2(1-\\nu)(\\gamma^2-1)\\Dk^2,\n\\end{eqnarray*}\nwhich, together with~\\eqref{bTwoThree}, leads to~\\eqref{huit}.\n\n$ $\\\\\n\\textbf{Case~2: $\\igkc=1$}\n$ $\\\\ \nConsider the event $G_k:=\\accolade{\\normii{\\Gkhat}\\geq \\eta_2\\Dk}$, and note that since $\\Mhatk:={\\ubar{f}_k}+\\Gkhat^{\\top}\\sRandom +\\frac{1}{2}\\sRandom^{\\top}\\HkhatRandom\\sRandom$ is $\\fmqalgebra$-measurable by construction, then in particular $\\normii{\\Gkhat}$ and hence $\\igkhat, \\igkhatc$ are also $\\fmqalgebra$-measurable. Observe that if $\\igkhatc=1$, then the iteration is unsuccessful since $\\normii{\\gkhat}<\\eta_2\\dk$, leading to~\\eqref{bOne}. Hence, \n\\begin{equation}\\label{GkbarCase2}\n\\begin{split}\n\\E{\\igkc\\igkhatc(\\Phikun-\\Phik)|\\fmqalgebra}&\\leq -(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\E{\\igkc\\igkhatc\\Dk^2|\\fmqalgebra}\\\\\n&\\leq -\\frac{1}{2}\\igkc\\igkhatc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{split}\n\\end{equation}\nwhere the last inequality is due to the $\\fmqalgebra$-measurability of $\\igkc$ and $\\Dk$, given that $\\falgebra\\subset\\fqalgebra\\subset\\fmqalgebra$. \n\nInspired by~\\cite[Theorem~3]{blanchet2016convergence}, which improved on the proof of~\\cite[Theorem~4.11, Case~2]{chen2018stochastic} in which four subcases were considered, only two are analyzed next. The overall goal is to prove that\n\\begin{equation}\\label{GkCase2}\n\\E{\\igkc\\igkhat(\\Phikun-\\Phik)|\\fmqalgebra}\\leq -\\frac{1}{2}\\igkc\\igkhat\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{equation}\nwhich, combined with~\\eqref{GkbarCase2}, leads to \n\\begin{equation*}%\\label{GammaBarFmq}\n\\E{\\igkc(\\Phikun-\\Phik)|\\fmqalgebra}\\leq -\\frac{1}{2}\\igkc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2\n\\end{equation*}\nand consequently\n\\begin{equation*}%\\label{GammaBarFq}\n\\begin{split}\n\\E{\\igkc(\\Phikun-\\Phik)|\\fqalgebra}&=\\E{\\E{\\igkc(\\Phikun-\\Phik)|\\fmqalgebra}|\\fqalgebra}\\\\\n&\\leq \\E{-\\frac{1}{2}\\igkc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2 \\, |\\fqalgebra}=-\\frac{1}{2}\\igkc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{split}\n\\end{equation*}\n showing~\\eqref{last}, where the last equality follows from the $\\fqalgebra$-measurability of $\\igkc$ and $\\Dk$.  What remains to be shown is~\\eqref{GkCase2} as achieved next, assuming that $\\igkhat=1$.\n\n\\noindent \\textit{(Subcase 2i)}\n%\\begin{itemize}\n%\\item[(i)] \nGood estimates ($\\ijk=1$) and $\\igkhat=1$. While $f$ is decreased on successful iterations because  of good estimates thanks to Lemma~\\ref{LemmaD} and in this case~\\eqref{LemmaDimprovement} holds,  $\\dk$ is reduced on unsuccessful iterations, leading to~\\eqref{bOne}. However, combining~\\eqref{LemmaDimprovement} and~\\eqref{un} yields\n$\\phi_{k+1}-\\phi_k\\leq \\left[-\\nu C_2+(1-\\nu)(\\gamma^2-1)\\right]\\dk^2\\leq b_1,$\nwhere the last inequality is due to~\\eqref{nuChoiceScal}, which shows that~\\eqref{bOne} holds in any case. Consequently,\n\\begin{equation}\n\\label{GammakBarGkJk}\n\\igkc\\igkhat\\ijk(\\Phikun-\\Phik) \\leq -\\igkc\\igkhat\\ijk(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2.\n\\end{equation}\n\n%\\item[(ii)] \n\\noindent \\textit{(Subcase 2ii)}\nBad estimates ($\\ijkc=1$) and $\\igkhat=1$. In this case, a successful step can lead to an increase in $f$\naccording to~\\eqref{LemmaEincrease}, which combined with~\\eqref{un} yield~\\eqref{bThree} and hence\n\\begin{equation}\\label{GammakBarGkJkBar}\n\\igkc\\igkhat\\ijkc(\\Phikun-\\Phik) \\leq -\\igkc\\igkhat\\ijkc\\left(\\nu C_3\\zeta+(1-\\nu)(\\gamma^2-1)\\right)\\Dk^2.\n\\end{equation}\n%\\end{itemize}\n\nWith the subcases complete, recall that $\\E{\\ijk|\\fmqalgebra}\\geq\\beta_f$. Then  combining~\\eqref{GammakBarGkJk} and~\\eqref{GammakBarGkJkBar} and taking expectations with respect to~$\\fmqalgebra$ lead to\n\\begin{eqnarray*}%\\label{GkCase2Proved}\n\\E{\\igkc\\igkhat(\\Phikun-\\Phik)|\\fmqalgebra}&\\leq& \\igkc\\igkhat\\left(-\\beta_f(1-\\nu)(1-1/\\gamma^2)\\right.\\\\ \n& &\\left. +\\ (1-\\beta_f)\\left(\\nu C_3\\zeta+(1-\\nu)(\\gamma^2-1)\\right) \\right)\\Dk^2\\\\\n&\\leq& -\\frac{1}{2}\\igkc\\igkhat\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{eqnarray*}\nthus proving~\\eqref{GkCase2}, where the last inequality follows from the second condition in~\\eqref{probChoices}, and the proof is complete.\n\\end{proof}","start":102803,"end":113478}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":["bOne","un"],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"成功与失败的势增量","source_units":[{"raw_tex":"\\begin{equation}\\label{un}\n\\phi_{k+1}-\\phi_k\\leq \\nu\\left(f(\\xkun)-f(\\xk)\\right)+(1-\\nu)(\\gamma^2-1)\\dk^2,\n\\end{equation}\nwhile on unsuccessful iterations, $\\xkun=\\xk$ and $\\dkun=\\gamma^{-1}\\dk$, in which case\n\\begin{equation}\\label{bOne}\n\\phi_{k+1}-\\phi_k\\leq(1-\\nu)\\left(\\frac{1}{\\gamma^2}-1\\right)\\dk^2=:b_1<0.\n\\end{equation}\n","start":103379,"end":103708}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","labels":["huit","last"],"member_ids":["S055"],"declared_external":false,"source_only":false,"title":"按投影真梯度分区","source_units":[{"raw_tex":"\nRecall the event $\\Gamma_k=\\accolade{\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\geq\\zeta\\Dk}$ of Lemma~\\ref{indicatorMeasurable} with $\\zeta$ satisfying ~\\eqref{zetaChoice}. The proof considers two cases: $\\igk=1$ and $\\igkc=1$. Inspired by~\\cite[proof of Theorem~4.11]{chen2018stochastic}, Case~1 aims to show that\n\\begin{equation}\\label{huit}\n%\\begin{split}\n\\E{\\igk\\left(\\Phikun-\\Phik\\right)|\\fqalgebra}\\leq-2\\igk(1-\\nu)(\\gamma^2-1)\\Dk^2\n\\leq -\\frac{1}{2}\\igk\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})\\Dk^2, \n%\\end{split}\n\\end{equation}\nwhere the last inequality follows from $1-\\frac{1}{\\gamma^2}<\\gamma^2-1$ and $\\beta_f\\leq 1$. On the other hand, inspired by~\\cite[proof of Theorem~3]{blanchet2016convergence}, Case~2 shows that\n\\begin{equation}\\label{last}\n\\E{\\igkc\\left(\\Phikun-\\Phik\\right)|\\fqalgebra}\\leq-\\frac{1}{2}\\igkc\\beta_f(1-\\nu)(1-\\frac{1}{\\gamma^2})\\Dk^2.\n\\end{equation}\nThus, combining~\\eqref{huit} and~\\eqref{last} leads to~\\eqref{PhikIncrementBound}.\n\n$ $\\\\\n\\textbf{Case~1: $\\igk=1$}\\\\\n$ $\\\\\nUnlike the proof of~\\cite[Theorem~4.11, Case~1]{chen2018stochastic} where four subcases were considered, only three are analyzed next.\n\n%\\begin{itemize}\n%\\item[(i)] \n","start":103708,"end":104879}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","labels":["bTwo","bTwoRandom"],"member_ids":["S056"],"declared_external":false,"source_only":false,"title":"大梯度且模型估计良好","source_units":[{"raw_tex":"\\noindent \\textit{(Subcase 1i)}\nGood model ($\\iik=1$) and good estimates ($\\ijk=1$). By~\\eqref{zetaChoice}, \n\\[\\normii{\\QkTranspose\\nabla f(\\xk)}\\geq \\max\\accolade{\\eta_2+\\kappa_{eg}, \\kappa_h+\\kappa_{eg}, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}}+\\kappa_{eg}}\\dk,\\]\nimplying in particular~\\eqref{deltaLemmaB2} and hence a decrease in $f$ according to~\\eqref{LemmaB2improvement}. Moreover, by $(\\kef,\\keg;\\Qk)$-full linearity, \n\\[\\normii{\\gkhat}\\geq \\normii{\\QkTranspose\\nabla f(\\xk)}-\\kappa_{eg}\\dk\\geq \\max\\accolade{\\eta_2,\\kappa_h, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}}\\dk, \\]\nthus implying~\\eqref{deltaLemmaC}; and since $\\ef\\leq \\kappa_{ef}$ per assumptions of Lemma~\\ref{LemmaC}, the iteration is successful. Consequently,~\\eqref{un} together with~\\eqref{LemmaB2improvement} yields\n\\begin{equation}\\label{bTwo}\n\\begin{split}\n\t\\phi_{k+1}-\\phi_k &\\leq -\\nu C_1\\normii{\\QkTranspose\\nabla f(\\xk)}\\dk+(1-\\nu)(\\gamma^2-1)\\dk^2=:b_2\\\\\n\t&\\leq \\left(-\\nu C_1\\zeta+(1-\\nu)(\\gamma^2-1)\\right)\\dk^2<0,\n\\end{split}\n\\end{equation}\nwhere the last inequality follows from~\\eqref{nuChoiceScal}. Denoting by ${\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)$ the random variable with realizations $b_2<0$, with ${\\ubar{u}}_k$ defined in~\\eqref{ukRandomMeasurable}, we have from~\\eqref{bTwo} that\n\\begin{equation}\\label{bTwoRandom}\n\\igk\\iik\\ijk\\left(\\Phikun-\\Phik\\right)\\leq \\igk\\iik\\ijk{\\ubar{b}}_2({\\ubar{u}}_k,\\Dk).\n\\end{equation}\n%\\item[(ii)] \n","start":104879,"end":106295}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","labels":["bThree","bThreeRandom2"],"member_ids":["S057"],"declared_external":false,"source_only":false,"title":"大梯度且模型不良","source_units":[{"raw_tex":"\\noindent \\textit{(Subcase 1ii)}\nBad model ($\\iikc=1$) and good estimates ($\\ijk=1$). In this case, regardless of the iteration type (i.e., successful or unsuccessful), the change in $\\phi_k$ can always be bounded by using~\\eqref{un} and~\\eqref{LemmaEincrease} as follows:\n\\begin{equation}\\label{bThree}\n\\phi_{k+1}-\\phi_k\\leq\\nu C_3\\norme{\\QkTranspose\\nabla f(\\xk)}\\dk+(1-\\nu)(\\gamma^2-1)\\dk^2=:b_3.\n\\end{equation}\nDenoting by ${\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)$ the random variable with realizations $b_3>0$, from~\\eqref{bThree} we have that\n\\begin{equation}\\label{bThreeRandom2}\n\\igk\\iikc\\ijk\\left(\\Phikun-\\Phik\\right)\\leq \\igk\\iikc\\ijk{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)\\leq \\igk\\iikc{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk).\n\\end{equation}\n%\\item[(iii)] \n","start":106295,"end":107036}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","labels":["bThreeRandom3"],"member_ids":["S058"],"declared_external":false,"source_only":false,"title":"大梯度且估计不良","source_units":[{"raw_tex":"\\noindent \\textit{(Subcase 1iii)}\nBad estimates ($\\ijkc=1$). From \\textit{Subcase 1ii}, it always holds that \n\\begin{equation}\\label{bThreeRandom3}\n\\igk\\ijkc\\left(\\Phikun-\\Phik\\right)\\leq \\igk\\ijkc{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk).\n\\end{equation}\n%\\end{itemize}\n\n","start":107036,"end":107296}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","labels":["bTwoThree"],"member_ids":["S059"],"declared_external":false,"source_only":false,"title":"大梯度事件的概率加权","source_units":[{"raw_tex":"With the subcases thus complete, since the random variables $\\igk, {\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)$, and ${\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)$ are $\\fqalgebra$-measurable thanks to the proof of Lemma~\\ref{indicatorMeasurable}, combining~\\eqref{bTwoRandom},~\\eqref{bThreeRandom2}, and~\\eqref{bThreeRandom3} and taking expectations with respect to $\\fqalgebra$, we have\n\\begin{equation}\\label{bTwoThree}\n\\begin{split}\n\\E{\\igk\\left(\\Phikun-\\Phik\\right)|\\fqalgebra}&\\leq \\igk{\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)\\E{\\iik\\ijk|\\fqalgebra}\\\\\n&+\\igk{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)\\left(\\E{\\iikc|\\fqalgebra}+\\E{\\ijkc|\\fqalgebra}\\right)\\\\\n&\\leq \\igk\\left(\\beta_f\\beta_m{\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)+{\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)(1-\\beta_m+1-\\beta_f)\\right)=:\\igk {\\ubar{b}}_{2,3}({\\ubar{u}}_k,\\Dk),\n\\end{split}\n\\end{equation}\nwhere the last inequality follows from~\\eqref{ikQProb} and~\\eqref{prodBetafBetam} and the fact that ${\\ubar{b}}_2({\\ubar{u}}_k,\\Dk)<0$ while ${\\ubar{b}}_3({\\ubar{u}}_k,\\Dk)>0$.  Note that \n\\begin{eqnarray*}\n{\\ubar{b}}_{2,3}({\\ubar{u}}_k,\\Dk)&=&\\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\left[-\\beta_f\\beta_m C_1+(2-\\beta_f-\\beta_m)C_3\\right]\\\\\n& &+(1-\\nu)(\\gamma^2-1)(2-\\beta_f-\\beta_m+\\beta_f\\beta_m)\\Dk^2\\\\\n&\\leq& \\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\left[-\\beta_f\\beta_m C_1+(2-\\beta_f-\\beta_m)C_3\\right] + 2(1-\\nu)(\\gamma^2-1)\\Dk^2,\n\\end{eqnarray*}\nwhere the last inequality follows from $2-\\beta_f-\\beta_m+\\beta_f\\beta_m=1+(1-\\beta_f)(1-\\beta_m)\\leq 2$.\n","start":107296,"end":108774}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S060"],"declared_external":false,"source_only":false,"title":"完成大梯度负漂移","source_units":[{"raw_tex":"\nFor $\\beta_f$ and $\\beta_m$ chosen according to the first condition in~\\eqref{probChoices}, and for $\\nu$ satisfying~\\eqref{nuChoiceScal}, the following holds:\n\\[\\beta_f\\beta_m C_1 - (2-\\beta_f-\\beta_m)C_3\\geq \\frac{1}{2}C_1\\geq 2\\frac{2(1-\\nu)(\\gamma^2-1)}{\\nu\\zeta}.\\]\nHence, since $\\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\geq \\zeta\\Dk$, then \n\\begin{eqnarray*}\n{\\ubar{b}}_{2,3}({\\ubar{u}}_k,\\Dk)&\\leq& -\\frac{1}{2}\\left[\\beta_f\\beta_m C_1-(2-\\beta_f-\\beta_m)C_3\\right]\\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\\\\n&\\leq& -\\frac{1}{4}C_1\\nu \\normii{\\QkRandomTranspose\\nabla f(\\Xk)}\\Dk\\leq -2(1-\\nu)(\\gamma^2-1)\\Dk^2,\n\\end{eqnarray*}\nwhich, together with~\\eqref{bTwoThree}, leads to~\\eqref{huit}.\n\n","start":108774,"end":109479}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","labels":["GkbarCase2"],"member_ids":["S061"],"declared_external":false,"source_only":false,"title":"小梯度下的模型梯度门槛","source_units":[{"raw_tex":"$ $\\\\\n\\textbf{Case~2: $\\igkc=1$}\n$ $\\\\ \nConsider the event $G_k:=\\accolade{\\normii{\\Gkhat}\\geq \\eta_2\\Dk}$, and note that since $\\Mhatk:={\\ubar{f}_k}+\\Gkhat^{\\top}\\sRandom +\\frac{1}{2}\\sRandom^{\\top}\\HkhatRandom\\sRandom$ is $\\fmqalgebra$-measurable by construction, then in particular $\\normii{\\Gkhat}$ and hence $\\igkhat, \\igkhatc$ are also $\\fmqalgebra$-measurable. Observe that if $\\igkhatc=1$, then the iteration is unsuccessful since $\\normii{\\gkhat}<\\eta_2\\dk$, leading to~\\eqref{bOne}. Hence, \n\\begin{equation}\\label{GkbarCase2}\n\\begin{split}\n\\E{\\igkc\\igkhatc(\\Phikun-\\Phik)|\\fmqalgebra}&\\leq -(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\E{\\igkc\\igkhatc\\Dk^2|\\fmqalgebra}\\\\\n&\\leq -\\frac{1}{2}\\igkc\\igkhatc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{split}\n\\end{equation}\nwhere the last inequality is due to the $\\fmqalgebra$-measurability of $\\igkc$ and $\\Dk$, given that $\\falgebra\\subset\\fqalgebra\\subset\\fmqalgebra$. \n\n","start":109479,"end":110420}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","labels":["GkCase2"],"member_ids":["S062"],"declared_external":false,"source_only":false,"title":"组合小梯度分支并降滤过","source_units":[{"raw_tex":"Inspired by~\\cite[Theorem~3]{blanchet2016convergence}, which improved on the proof of~\\cite[Theorem~4.11, Case~2]{chen2018stochastic} in which four subcases were considered, only two are analyzed next. The overall goal is to prove that\n\\begin{equation}\\label{GkCase2}\n\\E{\\igkc\\igkhat(\\Phikun-\\Phik)|\\fmqalgebra}\\leq -\\frac{1}{2}\\igkc\\igkhat\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{equation}\nwhich, combined with~\\eqref{GkbarCase2}, leads to \n\\begin{equation*}%\\label{GammaBarFmq}\n\\E{\\igkc(\\Phikun-\\Phik)|\\fmqalgebra}\\leq -\\frac{1}{2}\\igkc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2\n\\end{equation*}\nand consequently\n\\begin{equation*}%\\label{GammaBarFq}\n\\begin{split}\n\\E{\\igkc(\\Phikun-\\Phik)|\\fqalgebra}&=\\E{\\E{\\igkc(\\Phikun-\\Phik)|\\fmqalgebra}|\\fqalgebra}\\\\\n&\\leq \\E{-\\frac{1}{2}\\igkc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2 \\, |\\fqalgebra}=-\\frac{1}{2}\\igkc\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{split}\n\\end{equation*}\n showing~\\eqref{last}, where the last equality follows from the $\\fqalgebra$-measurability of $\\igkc$ and $\\Dk$.  What remains to be shown is~\\eqref{GkCase2} as achieved next, assuming that $\\igkhat=1$.\n\n","start":110420,"end":111602}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","labels":["GammakBarGkJk"],"member_ids":["S063"],"declared_external":false,"source_only":false,"title":"小梯度且估计良好","source_units":[{"raw_tex":"\\noindent \\textit{(Subcase 2i)}\n%\\begin{itemize}\n%\\item[(i)] \nGood estimates ($\\ijk=1$) and $\\igkhat=1$. While $f$ is decreased on successful iterations because  of good estimates thanks to Lemma~\\ref{LemmaD} and in this case~\\eqref{LemmaDimprovement} holds,  $\\dk$ is reduced on unsuccessful iterations, leading to~\\eqref{bOne}. However, combining~\\eqref{LemmaDimprovement} and~\\eqref{un} yields\n$\\phi_{k+1}-\\phi_k\\leq \\left[-\\nu C_2+(1-\\nu)(\\gamma^2-1)\\right]\\dk^2\\leq b_1,$\nwhere the last inequality is due to~\\eqref{nuChoiceScal}, which shows that~\\eqref{bOne} holds in any case. Consequently,\n\\begin{equation}\n\\label{GammakBarGkJk}\n\\igkc\\igkhat\\ijk(\\Phikun-\\Phik) \\leq -\\igkc\\igkhat\\ijk(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2.\n\\end{equation}\n\n%\\item[(ii)] \n","start":111602,"end":112370}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","labels":["GammakBarGkJkBar"],"member_ids":["S064"],"declared_external":false,"source_only":false,"title":"小梯度且估计不良","source_units":[{"raw_tex":"\\noindent \\textit{(Subcase 2ii)}\nBad estimates ($\\ijkc=1$) and $\\igkhat=1$. In this case, a successful step can lead to an increase in $f$\naccording to~\\eqref{LemmaEincrease}, which combined with~\\eqref{un} yield~\\eqref{bThree} and hence\n\\begin{equation}\\label{GammakBarGkJkBar}\n\\igkc\\igkhat\\ijkc(\\Phikun-\\Phik) \\leq -\\igkc\\igkhat\\ijkc\\left(\\nu C_3\\zeta+(1-\\nu)(\\gamma^2-1)\\right)\\Dk^2.\n\\end{equation}\n%\\end{itemize}\n\n","start":112370,"end":112788}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S065"],"declared_external":false,"source_only":false,"title":"完成模型门槛分支漂移","source_units":[{"raw_tex":"With the subcases complete, recall that $\\E{\\ijk|\\fmqalgebra}\\geq\\beta_f$. Then  combining~\\eqref{GammakBarGkJk} and~\\eqref{GammakBarGkJkBar} and taking expectations with respect to~$\\fmqalgebra$ lead to\n\\begin{eqnarray*}%\\label{GkCase2Proved}\n\\E{\\igkc\\igkhat(\\Phikun-\\Phik)|\\fmqalgebra}&\\leq& \\igkc\\igkhat\\left(-\\beta_f(1-\\nu)(1-1/\\gamma^2)\\right.\\\\ \n& &\\left. +\\ (1-\\beta_f)\\left(\\nu C_3\\zeta+(1-\\nu)(\\gamma^2-1)\\right) \\right)\\Dk^2\\\\\n&\\leq& -\\frac{1}{2}\\igkc\\igkhat\\beta_f(1-\\nu)\\left(1-\\frac{1}{\\gamma^2}\\right)\\Dk^2,\n\\end{eqnarray*}\nthus proving~\\eqref{GkCase2}, where the last inequality follows from the second condition in~\\eqref{probChoices}, and the proof is complete.\n\\end{proof}","start":112788,"end":113478}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S066"],"declared_external":false,"source_only":false,"title":"由非负漂移求和","source_units":[{"raw_tex":"\\begin{proof}\nThe proof is inspired by those of~\\cite[Theorem~3]{blanchet2016convergence} and~\\cite[Theorem~4.11]{chen2018stochastic}. \nThe overall goal is to prove~\\eqref{PhikIncrementBound}. Indeed noticing that $\\Phik>0$ and then taking expectations on both sides of the inequality in~\\eqref{PhikIncrementBound}, we obtain~\\eqref{seriesResult} (see, e.g.,~\\cite[Theorem~3]{dzahini2020expected} for details). Let $\\phi_k$ denote realizations of $\\Phik$, and recall that on all successful iterations, $\\xkun=\\xk+\\Qk\\sk$ and $\\dkun=\\min\\accolade{\\gamma\\dk,\\dmaxx}$,  implying\n","start":102803,"end":103379}]},{"id":"S067","semantic_type":"theorem","kind":"theorem","labels":["liminfTheorem"],"member_ids":["S067"],"declared_external":false,"source_only":false,"title":"梯度下极限收敛","source_units":[{"raw_tex":"\\begin{theorem}\\label{liminfTheorem}\nLet all the assumptions made in Theorem~\\ref{zerothOrderScal} hold. Assume further that Assumption~\\ref{wellAlignedAssumption} holds with $0\\leq\\beta_Q<1-\\frac{1}{2\\beta_f\\beta_m}$. Then, the sequence $\\accolade{\\Xk}_{k\\in\\N}$ of random iterates generated by Algorithm~\\ref{algoStoScalTR} satisfies\n\\[\\underset{k\\to\\infty}{\\liminf}\\normii{\\nabla f(\\Xk)}=0\\quad\\mbox{ almost surely.}\\]\n\\end{theorem}","start":114364,"end":114799}]},{"id":"S068","semantic_type":"assumption","kind":"assumption","labels":["liminfTheorem"],"member_ids":["S068"],"declared_external":false,"source_only":false,"title":"严格正偏置的参数限制","source_units":[{"raw_tex":"Let all the assumptions made in Theorem~\\ref{zerothOrderScal} hold. Assume further that Assumption~\\ref{wellAlignedAssumption} holds with $0\\leq\\beta_Q<1-\\frac{1}{2\\beta_f\\beta_m}$. Then, the sequence $\\accolade{\\Xk}_{k\\in\\N}$ of random iterates generated by Algorithm~\\ref{algoStoScalTR} satisfies\n","start":114401,"end":114700}]},{"id":"S069","semantic_type":"proof","kind":"proof","labels":["bChoice","ineq4","ineq5"],"member_ids":["S069"],"declared_external":false,"source_only":false,"title":"下极限定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nThe result is proved by contradiction conditioned on the almost sure event $E_0:=\\accolade{\\Dk\\to 0}$ (thanks to Theorem~\\ref{zerothOrderScal}), inspired by the proof of~\\cite[Theorem~4.16]{chen2018stochastic} and also using ideas from~\\cite{audet2019stomads,dzahini2020constrained}. Assume that with nonzero probability there exists a random variable ${\\ubar{\\varepsilon}}'>0$ such that \n$\\normii{\\nabla f(\\Xk)}\\geq {\\ubar{\\varepsilon}}' \\mbox{ for all } k\\in\\N.$\n\nLet $\\accolade{\\xk}, \\accolade{\\dk}$, and ${{\\varepsilon}}'$ be realizations of $\\accolade{\\Xk}, \\accolade{\\Dk}$, and ${\\ubar{\\varepsilon}}'$, respectively, for which $\\normii{\\nabla f(\\xk)}\\geq {{\\varepsilon}}'$ for all $k$. Since $\\dk\\to 0$, there exists $k_0\\in\\N$ such that\n\\begin{equation}\\label{bChoice}\n\\dk<b:=\\alpha_Q\\cdot\\min\\accolade{\\frac{{{\\varepsilon}}'}{2(\\kappa_h+\\kappa_{eg})}, \\frac{{{\\varepsilon}}'}{2(\\eta_2+\\kappa_{eg})}, \\frac{{{\\varepsilon}}'}{\\frac{16\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}+2\\kappa_{eg}}, \\frac{\\dmaxx}{\\gamma}}\n\\end{equation}\nfor all $k\\geq k_0$.\nLet $\\ubar{r}_k$ be the random variable with realizations $r_k=\\log_{\\gamma}\\left(\\frac{\\dk}{b}\\right)$. Then $r_k<0$ for all $k\\geq k_0$. The main idea is to show that such realizations occur only with probability zero, leading to a contradiction. \n\nTo prove that $\\accolade{\\ubar{r}_k}$ is a submartingale, recall the events $I^Q_k$ and $J^Q_k$ satisfying~\\eqref{otherProbIneqs} and $\\mathcal{A}_k$ satisfying~\\eqref{Akineq}.\n%$\\prob{\\mathcal{A}_k|\\falgebra}\\geq 1-\\beta_Q$, \n%$\\prob{I^Q_k|\\falgebra}\\geq\\beta_m$ and  $\\prob{J^Q_k|\\falgebra}\\geq\\beta_f$, \n%where $\\mathcal{F}_{k-1}^Q :=\\sigma\\left(\\mathcal{A}_0,\\mathcal{A}_1,\\dots,\\mathcal{A}_{k-1}\\right)\\cap \\sigma\\left(I^Q_0,I^Q_1,\\dots,I^Q_{k-1}\\right)\\cap \\sigma\\left(J^Q_0,J^Q_1,\\dots,J^Q_{k-1}\\right)$. \nConsider some iteration $k\\geq k_0$ for which $\\mathcal{A}_k$, $I^Q_k$, and $J^Q_k$ all occur, which happens with probability at least $\\beta_f\\beta_m(1-\\beta_Q)$ conditioned on $\\falgebra$, as was shown in~\\eqref{prodBetaIneq}. It follows from~\\eqref{bChoice} that \n\\begin{equation}\\label{ineq4}\n\\normii{\\nabla f(\\xk)}\\geq \\frac{1}{\\alpha_Q}\\max\\accolade{\\kappa_h+\\kappa_{eg}, \\eta_2+\\kappa_{eg}, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}+\\kappa_{eg}}\\dk,\n\\end{equation}\nwhich, together with the $(\\kef,\\keg;\\Qk)$-full linearity and $\\alpha_Q$-well alignment, yield\n\\begin{equation}\\label{ineq5}\n%\\begin{split}\n\\normii{\\gkhat}\\geq\\normii{\\QkTranspose\\nabla f(\\xk)}-\\kappa_{eg}\\dk\\geq \\alpha_Q \\normii{\\nabla f(\\xk)}-\\kappa_{eg}\\dk\n\\geq \\max\\accolade{\\kappa_h, \\eta_2, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}}\\dk.\n%\\end{split}\n\\end{equation}\nThus, the $k$th iteration is successful according to Lemma~\\ref{LemmaC}. Hence, $\\dkun=\\gamma\\dk$, and hence $r_{k+1}=r_k+1$. For all other outcomes of $\\mathcal{A}_k$, $I^Q_k$, and $J^Q_k$, which occur with a total probability of at most $1-\\beta_f\\beta_m(1-\\beta_Q)$, it always holds that $\\dkun\\geq\\gamma^{-1}\\dk$, which implies that $r_{k+1}\\geq r_k-1$. Hence, \n\\begin{eqnarray*}\n\\E{\\mathds{1}_{\\mathcal{A}_k\\cap I^Q_k\\cap J^Q_k} \\left(\\ubar{r}_{k+1}-\\ubar{r}_k\\right)|\\falgebra}&\\geq& \\beta_f\\beta_m(1-\\beta_Q)\\quad \\quad\\mbox{and}\\\\\n\\E{\\mathds{1}_{\\overline{\\mathcal{A}_k\\cap I^Q_k\\cap J^Q_k}} \\left(\\ubar{r}_{k+1}-\\ubar{r}_k\\right)|\\falgebra}&\\geq& \\beta_f\\beta_m(1-\\beta_Q)-1, \n\\end{eqnarray*}\nwhich shows that $\\accolade{\\ubar{r}_k}$ is a submartingale if $0\\leq\\beta_Q<1-\\frac{1}{2\\beta_f\\beta_m}$.\n\nUsing the random walk defined in~\\eqref{randomWalk}, one  can easily  show (following, e.g.,~\\cite[Theorem~4]{audet2019stomads}, \\cite[Theorem~4.16]{chen2018stochastic} and~\\cite[Theorem~3.6 and Lemmas~4.7 and~4.12]{dzahini2020constrained}) that $\\accolade{\\ubar{w}_k}$ is also a submartingale with bounded increments and, as such, cannot converge to a finite value, and therefore $\\prob{\\underset{k\\to\\infty}{\\limsup}\\ \\ubar{w}_k=\\infty}=1$ (thanks to~\\cite[Theorem~4.4]{chen2018stochastic}). However, since $r_k-r_{k_0}\\geq w_k-w_{k_0}$ by construction,  the sequence of realizations~$r_k$ such that $r_k<0\\ \\forall k\\ \\geq k_0$ occurs with probability zero, leading to a contradiction, which achieves the proof.\n\\end{proof}","start":114801,"end":119014}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S070"],"declared_external":false,"source_only":false,"title":"在半径趋零事件上反设","source_units":[{"raw_tex":"\\begin{proof}\nThe result is proved by contradiction conditioned on the almost sure event $E_0:=\\accolade{\\Dk\\to 0}$ (thanks to Theorem~\\ref{zerothOrderScal}), inspired by the proof of~\\cite[Theorem~4.16]{chen2018stochastic} and also using ideas from~\\cite{audet2019stomads,dzahini2020constrained}. Assume that with nonzero probability there exists a random variable ${\\ubar{\\varepsilon}}'>0$ such that \n$\\normii{\\nabla f(\\Xk)}\\geq {\\ubar{\\varepsilon}}' \\mbox{ for all } k\\in\\N.$\n\nLet $\\accolade{\\xk}, \\accolade{\\dk}$, and ${{\\varepsilon}}'$ be realizations of $\\accolade{\\Xk}, \\accolade{\\Dk}$, and ${\\ubar{\\varepsilon}}'$, respectively, for which $\\normii{\\nabla f(\\xk)}\\geq {{\\varepsilon}}'$ for all $k$. Since $\\dk\\to 0$, there exists $k_0\\in\\N$ such that\n","start":114801,"end":115559}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","labels":["bChoice"],"member_ids":["S071"],"declared_external":false,"source_only":false,"title":"构造对数半径门槛","source_units":[{"raw_tex":"\\begin{equation}\\label{bChoice}\n\\dk<b:=\\alpha_Q\\cdot\\min\\accolade{\\frac{{{\\varepsilon}}'}{2(\\kappa_h+\\kappa_{eg})}, \\frac{{{\\varepsilon}}'}{2(\\eta_2+\\kappa_{eg})}, \\frac{{{\\varepsilon}}'}{\\frac{16\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}+2\\kappa_{eg}}, \\frac{\\dmaxx}{\\gamma}}\n\\end{equation}\nfor all $k\\geq k_0$.\nLet $\\ubar{r}_k$ be the random variable with realizations $r_k=\\log_{\\gamma}\\left(\\frac{\\dk}{b}\\right)$. Then $r_k<0$ for all $k\\geq k_0$. The main idea is to show that such realizations occur only with probability zero, leading to a contradiction. \n\n","start":115559,"end":116117}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","labels":["ineq4","ineq5"],"member_ids":["S072"],"declared_external":false,"source_only":false,"title":"对齐准确小半径必成功","source_units":[{"raw_tex":"To prove that $\\accolade{\\ubar{r}_k}$ is a submartingale, recall the events $I^Q_k$ and $J^Q_k$ satisfying~\\eqref{otherProbIneqs} and $\\mathcal{A}_k$ satisfying~\\eqref{Akineq}.\n%$\\prob{\\mathcal{A}_k|\\falgebra}\\geq 1-\\beta_Q$, \n%$\\prob{I^Q_k|\\falgebra}\\geq\\beta_m$ and  $\\prob{J^Q_k|\\falgebra}\\geq\\beta_f$, \n%where $\\mathcal{F}_{k-1}^Q :=\\sigma\\left(\\mathcal{A}_0,\\mathcal{A}_1,\\dots,\\mathcal{A}_{k-1}\\right)\\cap \\sigma\\left(I^Q_0,I^Q_1,\\dots,I^Q_{k-1}\\right)\\cap \\sigma\\left(J^Q_0,J^Q_1,\\dots,J^Q_{k-1}\\right)$. \nConsider some iteration $k\\geq k_0$ for which $\\mathcal{A}_k$, $I^Q_k$, and $J^Q_k$ all occur, which happens with probability at least $\\beta_f\\beta_m(1-\\beta_Q)$ conditioned on $\\falgebra$, as was shown in~\\eqref{prodBetaIneq}. It follows from~\\eqref{bChoice} that \n\\begin{equation}\\label{ineq4}\n\\normii{\\nabla f(\\xk)}\\geq \\frac{1}{\\alpha_Q}\\max\\accolade{\\kappa_h+\\kappa_{eg}, \\eta_2+\\kappa_{eg}, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}+\\kappa_{eg}}\\dk,\n\\end{equation}\nwhich, together with the $(\\kef,\\keg;\\Qk)$-full linearity and $\\alpha_Q$-well alignment, yield\n\\begin{equation}\\label{ineq5}\n%\\begin{split}\n\\normii{\\gkhat}\\geq\\normii{\\QkTranspose\\nabla f(\\xk)}-\\kappa_{eg}\\dk\\geq \\alpha_Q \\normii{\\nabla f(\\xk)}-\\kappa_{eg}\\dk\n\\geq \\max\\accolade{\\kappa_h, \\eta_2, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}(1-\\eta_1)}}\\dk.\n%\\end{split}\n\\end{equation}\nThus, the $k$th iteration is successful according to Lemma~\\ref{LemmaC}. Hence, $\\dkun=\\gamma\\dk$, and hence $r_{k+1}=r_k+1$. For all other outcomes of $\\mathcal{A}_k$, $I^Q_k$, and $J^Q_k$, which occur with a total probability of at most $1-\\beta_f\\beta_m(1-\\beta_Q)$, it always holds that $\\dkun\\geq\\gamma^{-1}\\dk$, which implies that $r_{k+1}\\geq r_k-1$. Hence, \n","start":116117,"end":117844}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S073"],"declared_external":false,"source_only":false,"title":"对数半径的偏置计算","source_units":[{"raw_tex":"\\begin{eqnarray*}\n\\E{\\mathds{1}_{\\mathcal{A}_k\\cap I^Q_k\\cap J^Q_k} \\left(\\ubar{r}_{k+1}-\\ubar{r}_k\\right)|\\falgebra}&\\geq& \\beta_f\\beta_m(1-\\beta_Q)\\quad \\quad\\mbox{and}\\\\\n\\E{\\mathds{1}_{\\overline{\\mathcal{A}_k\\cap I^Q_k\\cap J^Q_k}} \\left(\\ubar{r}_{k+1}-\\ubar{r}_k\\right)|\\falgebra}&\\geq& \\beta_f\\beta_m(1-\\beta_Q)-1, \n\\end{eqnarray*}\nwhich shows that $\\accolade{\\ubar{r}_k}$ is a submartingale if $0\\leq\\beta_Q<1-\\frac{1}{2\\beta_f\\beta_m}$.\n","start":117844,"end":118287}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S074"],"declared_external":false,"source_only":false,"title":"随机游走上极限排除反设","source_units":[{"raw_tex":"\nUsing the random walk defined in~\\eqref{randomWalk}, one  can easily  show (following, e.g.,~\\cite[Theorem~4]{audet2019stomads}, \\cite[Theorem~4.16]{chen2018stochastic} and~\\cite[Theorem~3.6 and Lemmas~4.7 and~4.12]{dzahini2020constrained}) that $\\accolade{\\ubar{w}_k}$ is also a submartingale with bounded increments and, as such, cannot converge to a finite value, and therefore $\\prob{\\underset{k\\to\\infty}{\\limsup}\\ \\ubar{w}_k=\\infty}=1$ (thanks to~\\cite[Theorem~4.4]{chen2018stochastic}). However, since $r_k-r_{k_0}\\geq w_k-w_{k_0}$ by construction,  the sequence of realizations~$r_k$ such that $r_k<0\\ \\forall k\\ \\geq k_0$ occurs with probability zero, leading to a contradiction, which achieves the proof.\n\\end{proof}","start":118287,"end":119014}]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S075"],"declared_external":true,"declaration_text":["原先决条件列出的 chen2018stochastic Theorem 4.4，audet2019stomads Theorem 4，dzahini2020constrained Theorem 3.6 and Lemmas 4.7 and 4.12；L1566 另比较 chen2018stochastic Theorem 4.16。"],"source_only":false,"title":"外部随机游走论证","source_units":[]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S076"],"declared_external":true,"declaration_text":["原先决条件列出的 billingsley1995ThirdEdition Chapter 13；L1325–L1326、L1335、L1341 具体引用 Theorems 13.1–13.4 和 Equation (13.3)。"],"source_only":false,"title":"可测函数闭包与极限定理","source_units":[]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S077"],"declared_external":true,"declaration_text":["原先决条件列出的 Fundamental Theorem of Calculus and Lipschitz-gradient remainder bound，AuHa2017 Lemma 9.3；L1806 明确调用。"],"source_only":false,"title":"微积分基本定理与余项界","source_units":[]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S078"],"declared_external":true,"declaration_text":["原先决条件列出的 standard nonnegative drift summation argument，参考 dzahini2020expected Theorem 3；L1378 明确转引。"],"source_only":false,"title":"非负过程漂移求和规则","source_units":[]},{"id":"S079","semantic_type":"lemma","kind":"lemma","labels":["smallLemma"],"member_ids":["S079"],"declared_external":false,"source_only":false,"title":"算法半径满足更新下界","source_units":[{"raw_tex":"\\begin{lemma}\\label{smallLemma}\nLet all the assumptions made in Theorem~\\ref{liminfTheorem} hold. Then~\\eqref{dynamicsRenewal} is satisfied for\\\\ $\\Wtildek=2\\left(\\iak\\iik\\ijk-\\frac{1}{2} \\right)$, with $\\lambda=\\ln(\\gamma)$ and some fixed $q\\in (\\tilde{\\beta}, 1)$ with $\\tilde{\\beta}=\\beta_f\\beta_m(1-\\beta_Q)$.\n\\end{lemma}","start":125901,"end":126226}]},{"id":"S080","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S080"],"declared_external":false,"source_only":false,"title":"半径动力学的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nThe result is proved by suitably adapting the proof of~\\cite[Lemma~7]{blanchet2016convergence} as was done in~\\cite{dzahini2020expected}. First, we notice that~\\eqref{dynamicsRenewal} trivially holds when $\\itksup=0$. Next, we show that when $\\itksup=1$, then \n\\[\\Dkun\\geq \\min\\accolade{\\depsilon, \\min\\accolade{\\dmaxx,\\gamma\\Dk}\\iak\\iik\\ijk+\\gamma^{-1}\\Dk\\left(1-\\iak\\iik\\ijk\\right)}. \\]\nRecall that $\\dk=\\gamma^{i_k}\\depsilon$ for some integer $i_k$. Therefore, if $\\dk>\\depsilon$, then $\\dk\\geq \\gamma\\depsilon$, which implies that $\\dkun\\geq\\gamma^{-1}\\dk\\geq\\depsilon$. Now assume that $\\dk\\leq \\depsilon$. Since $\\tauEpsilon>k$, we have $\\normii{\\nabla f(\\xk)}>\\epsilon=\\xi\\depsilon\\geq\\xi\\dk$. Thus, it follows from the definition of~$\\xi$ that~\\eqref{ineq4} holds. If $\\iak=1$, $\\iik=1$, and $\\ijk=1$, then~\\eqref{ineq5} holds, and consequently the $k$th iteration is successful, as was  explained in the proof of Theorem~\\ref{liminfTheorem}. Hence, $\\xkun=\\xk+\\Qk\\sk$ and $\\dkun=\\min\\accolade{\\dmaxx,\\gamma\\dk}$. If $\\iak\\iik\\ijk=0$, then it always holds that $\\dkun\\geq \\gamma^{-1}\\dk$. The proof is completed by  observing that $\\prob{\\iak\\iik\\ijk=1}=q$ for some fixed $q\\geq \\tilde{\\beta}$.\n\\end{proof}","start":126228,"end":127456}]},{"id":"S081","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S081"],"declared_external":false,"source_only":false,"title":"停止后情形与目标不等式","source_units":[{"raw_tex":"\\begin{proof}\nThe result is proved by suitably adapting the proof of~\\cite[Lemma~7]{blanchet2016convergence} as was done in~\\cite{dzahini2020expected}. First, we notice that~\\eqref{dynamicsRenewal} trivially holds when $\\itksup=0$. Next, we show that when $\\itksup=1$, then \n\\[\\Dkun\\geq \\min\\accolade{\\depsilon, \\min\\accolade{\\dmaxx,\\gamma\\Dk}\\iak\\iik\\ijk+\\gamma^{-1}\\Dk\\left(1-\\iak\\iik\\ijk\\right)}. \\]\n","start":126228,"end":126631}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S082"],"declared_external":false,"source_only":false,"title":"阈值两侧的半径控制","source_units":[{"raw_tex":"Recall that $\\dk=\\gamma^{i_k}\\depsilon$ for some integer $i_k$. Therefore, if $\\dk>\\depsilon$, then $\\dk\\geq \\gamma\\depsilon$, which implies that $\\dkun\\geq\\gamma^{-1}\\dk\\geq\\depsilon$. Now assume that $\\dk\\leq \\depsilon$. Since $\\tauEpsilon>k$, we have $\\normii{\\nabla f(\\xk)}>\\epsilon=\\xi\\depsilon\\geq\\xi\\dk$. Thus, it follows from the definition of~$\\xi$ that~\\eqref{ineq4} holds. If $\\iak=1$, $\\iik=1$, and $\\ijk=1$, then~\\eqref{ineq5} holds, and consequently the $k$th iteration is successful, as was  explained in the proof of Theorem~\\ref{liminfTheorem}. Hence, $\\xkun=\\xk+\\Qk\\sk$ and $\\dkun=\\min\\accolade{\\dmaxx,\\gamma\\dk}$. If $\\iak\\iik\\ijk=0$, then it always holds that $\\dkun\\geq \\gamma^{-1}\\dk$. The proof is completed by  observing that $\\prob{\\iak\\iik\\ijk=1}=q$ for some fixed $q\\geq \\tilde{\\beta}$.\n","start":126631,"end":127445}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S083"],"declared_external":false,"source_only":false,"title":"固定概率的未证断言","source_units":[{"raw_tex":"Recall that $\\dk=\\gamma^{i_k}\\depsilon$ for some integer $i_k$. Therefore, if $\\dk>\\depsilon$, then $\\dk\\geq \\gamma\\depsilon$, which implies that $\\dkun\\geq\\gamma^{-1}\\dk\\geq\\depsilon$. Now assume that $\\dk\\leq \\depsilon$. Since $\\tauEpsilon>k$, we have $\\normii{\\nabla f(\\xk)}>\\epsilon=\\xi\\depsilon\\geq\\xi\\dk$. Thus, it follows from the definition of~$\\xi$ that~\\eqref{ineq4} holds. If $\\iak=1$, $\\iik=1$, and $\\ijk=1$, then~\\eqref{ineq5} holds, and consequently the $k$th iteration is successful, as was  explained in the proof of Theorem~\\ref{liminfTheorem}. Hence, $\\xkun=\\xk+\\Qk\\sk$ and $\\dkun=\\min\\accolade{\\dmaxx,\\gamma\\dk}$. If $\\iak\\iik\\ijk=0$, then it always holds that $\\dkun\\geq \\gamma^{-1}\\dk$. The proof is completed by  observing that $\\prob{\\iak\\iik\\ijk=1}=q$ for some fixed $q\\geq \\tilde{\\beta}$.\n","start":126631,"end":127445}]},{"id":"S084","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S084"],"declared_external":false,"source_only":false,"title":"复杂度定理的完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\nRecall the choice of $q$ in Lemma~\\ref{smallLemma}, and pick some $\\tilde{\\xi}\\geq\\hat{\\xi}$ such that $\\depsilon=\\frac{\\epsilon}{\\tilde{\\xi}}=\\gamma^{i}\\delta_0$ for some integer $i\\leq 0$ (without loss of generality), as discussed above. The proof follows by employing Theorem~\\ref{theorComplexity1} with $h(t)=t^2$, which yields\n\\[\\E{\\tauEpsilon}\\leq \\frac{q}{2q-1}\\cdot\\frac{\\PhiZero\\tilde{\\xi}^2}{\\varrho \\epsilon^2}+1\\leq \\frac{\\tilde{\\beta}}{2\\tilde{\\beta}-1}\\cdot\\frac{\\PhiZero\\tilde{\\xi}^2}{\\varrho \\epsilon^2}+1.\\]\n\\end{proof}","start":127969,"end":128519}]},{"id":"S085","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S085"],"declared_external":false,"source_only":false,"title":"选择格点兼容阈值","source_units":[{"raw_tex":"Recall the choice of $q$ in Lemma~\\ref{smallLemma}, and pick some $\\tilde{\\xi}\\geq\\hat{\\xi}$ such that $\\depsilon=\\frac{\\epsilon}{\\tilde{\\xi}}=\\gamma^{i}\\delta_0$ for some integer $i\\leq 0$ (without loss of generality), as discussed above. The proof follows by employing Theorem~\\ref{theorComplexity1} with $h(t)=t^2$, which yields\n","start":127983,"end":128315}]},{"id":"S086","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S086"],"declared_external":false,"source_only":false,"title":"应用期望停止时间定理","source_units":[{"raw_tex":"Recall the choice of $q$ in Lemma~\\ref{smallLemma}, and pick some $\\tilde{\\xi}\\geq\\hat{\\xi}$ such that $\\depsilon=\\frac{\\epsilon}{\\tilde{\\xi}}=\\gamma^{i}\\delta_0$ for some integer $i\\leq 0$ (without loss of generality), as discussed above. The proof follows by employing Theorem~\\ref{theorComplexity1} with $h(t)=t^2$, which yields\n\\[\\E{\\tauEpsilon}\\leq \\frac{q}{2q-1}\\cdot\\frac{\\PhiZero\\tilde{\\xi}^2}{\\varrho \\epsilon^2}+1\\leq \\frac{\\tilde{\\beta}}{2\\tilde{\\beta}-1}\\cdot\\frac{\\PhiZero\\tilde{\\xi}^2}{\\varrho \\epsilon^2}+1.\\]\n","start":127983,"end":128508}]},{"id":"S087","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S087"],"declared_external":false,"source_only":false,"title":"以联合下界替换概率因子","source_units":[{"raw_tex":"\\[\\E{\\tauEpsilon}\\leq \\frac{q}{2q-1}\\cdot\\frac{\\PhiZero\\tilde{\\xi}^2}{\\varrho \\epsilon^2}+1\\leq \\frac{\\tilde{\\beta}}{2\\tilde{\\beta}-1}\\cdot\\frac{\\PhiZero\\tilde{\\xi}^2}{\\varrho \\epsilon^2}+1.\\]\n","start":128315,"end":128508}]},{"id":"S088","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S088"],"declared_external":false,"source_only":false,"title":"模型梯度下降的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nThe proof is similar to that of~\\cite[Lemma~4.5]{chen2018stochastic} and is not detailed here. It follows from~\\eqref{CauchyDecrease} together with the inequalities $\\kappa_h\\geq \\max\\accolade{\\norme{\\Hkhat}, 1}$ and $\\normii{\\gkhat}\\geq\\kappa_h\\dk$ that  $\\mhatk(\\bm{0})-\\mhatk(\\sk)\\geq\\frac{\\kappa_{fcd}}{2}\\normii{\\gkhat}\\dk.$ Since the model is $(\\kef,\\keg;\\Qk)$-fully linear, then \n$f(\\xk+\\Qk\\sk)-f(\\xk)\\leq 2\\kappa_{ef}\\dk^2+\\mhatk(\\sk)-\\mhatk(\\bm{0})\\leq -\\frac{\\kappa_{fcd}}{4}\\normii{\\gkhat}\\dk,$\nwhere the last inequality follows from the fact that $\\dk\\leq \\frac{\\kappa_{fcd}}{8\\kappa_{ef}}\\normii{\\gkhat}$.\n\\end{proof}","start":141109,"end":141753}]},{"id":"S089","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S089"],"declared_external":false,"source_only":false,"title":"柯西下降抵消模型误差","source_units":[{"raw_tex":"The proof is similar to that of~\\cite[Lemma~4.5]{chen2018stochastic} and is not detailed here. It follows from~\\eqref{CauchyDecrease} together with the inequalities $\\kappa_h\\geq \\max\\accolade{\\norme{\\Hkhat}, 1}$ and $\\normii{\\gkhat}\\geq\\kappa_h\\dk$ that  $\\mhatk(\\bm{0})-\\mhatk(\\sk)\\geq\\frac{\\kappa_{fcd}}{2}\\normii{\\gkhat}\\dk.$ Since the model is $(\\kef,\\keg;\\Qk)$-fully linear, then \n$f(\\xk+\\Qk\\sk)-f(\\xk)\\leq 2\\kappa_{ef}\\dk^2+\\mhatk(\\sk)-\\mhatk(\\bm{0})\\leq -\\frac{\\kappa_{fcd}}{4}\\normii{\\gkhat}\\dk,$\nwhere the last inequality follows from the fact that $\\dk\\leq \\frac{\\kappa_{fcd}}{8\\kappa_{ef}}\\normii{\\gkhat}$.\n","start":141123,"end":141742}]},{"id":"S090","semantic_type":"proof","kind":"proof","labels":["ineq1"],"member_ids":["S090"],"declared_external":false,"source_only":false,"title":"投影梯度下降的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nThe proof is inspired by (and similar to) that of~\\cite[Lemma~4.6]{chen2018stochastic}. By $(\\kef,\\keg;\\Qk)$-full linearity and~\\eqref{deltaLemmaB2}, it holds\n\\begin{equation}\\label{ineq1}\n\\normii{\\gkhat}\\geq \\normii{\\QkTranspose\\nabla f(\\xk)}-\\kappa_{eg}\\dk\\geq \\max\\accolade{\\kappa_h, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}}}\\dk, \n\\end{equation}\nwhich shows that condition~\\eqref{LemmaAimprovement} is satisfied. Consequently $f$ decreases as in~\\eqref{LemmaAimprovement}. The first inequality in~\\eqref{ineq1}, together with~\\eqref{deltaLemmaB2}, yields\n$\\normii{\\gkhat}\\geq \\frac{4C_1}{\\kappa_{fcd}}\\normii{\\QkTranspose\\nabla f(\\xk)},$\nwhich, combined with~\\eqref{LemmaAimprovement}, implies~\\eqref{LemmaB2improvement}, and the proof is complete.\n\\end{proof}","start":141801,"end":142571}]},{"id":"S091","semantic_type":"proof_step","kind":"proof_step","labels":["ineq1"],"member_ids":["S091"],"declared_external":false,"source_only":false,"title":"投影梯度转为模型梯度门槛","source_units":[{"raw_tex":"The proof is inspired by (and similar to) that of~\\cite[Lemma~4.6]{chen2018stochastic}. By $(\\kef,\\keg;\\Qk)$-full linearity and~\\eqref{deltaLemmaB2}, it holds\n\\begin{equation}\\label{ineq1}\n\\normii{\\gkhat}\\geq \\normii{\\QkTranspose\\nabla f(\\xk)}-\\kappa_{eg}\\dk\\geq \\max\\accolade{\\kappa_h, \\frac{8\\kappa_{ef}}{\\kappa_{fcd}}}\\dk, \n\\end{equation}\n","start":141815,"end":142157}]},{"id":"S092","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S092"],"declared_external":false,"source_only":false,"title":"转移下降系数","source_units":[{"raw_tex":"which shows that condition~\\eqref{LemmaAimprovement} is satisfied. Consequently $f$ decreases as in~\\eqref{LemmaAimprovement}. The first inequality in~\\eqref{ineq1}, together with~\\eqref{deltaLemmaB2}, yields\n$\\normii{\\gkhat}\\geq \\frac{4C_1}{\\kappa_{fcd}}\\normii{\\QkTranspose\\nabla f(\\xk)},$\nwhich, combined with~\\eqref{LemmaAimprovement}, implies~\\eqref{LemmaB2improvement}, and the proof is complete.\n","start":142157,"end":142560}]},{"id":"S093","semantic_type":"proof","kind":"proof","labels":["ineq2","ineq3"],"member_ids":["S093"],"declared_external":false,"source_only":false,"title":"错误接受增幅的作者证明","source_units":[{"raw_tex":"\\begin{proof}\nAccording to the Fundamental Theorem of Calculus in $\\rn$ (see, e.g.,~\\cite[Lemma~9.3]{AuHa2017}),\\\\\n$\\abs{f(\\xk)-f(\\xk+\\Qk\\sk)-\\nabla f(\\xk+\\Qk\\sk)^{\\top}(-\\Qk\\sk)}\\leq\\frac{1}{2}L_g\\Qmax^2\\dk^2,$  with the inequality also following from $\\normii{\\Qk\\sk}\\leq \\Qmax\\dk$, which implies that\n\\begin{equation}\\label{ineq2}\nf(\\xk+\\Qk\\sk)-f(\\xk)\\leq \\sk^{\\top}\\left[\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)\\right] + \\frac{1}{2}L_g\\Qmax^2\\dk^2. \n\\end{equation}\nPer Lipschitz continuity of $\\nabla f$, \n$\\normii{\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)-\\QkTranspose \\nabla f(\\xk)} \\leq L_g\\Qmax^2\\dk,$\nwhich implies that\n\\begin{equation}\\label{ineq3}\n\\normii{\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)}\\leq L_g\\Qmax^2\\dk+\\normii{\\QkTranspose \\nabla f(\\xk)}.\n\\end{equation}\nThus, using~\\eqref{ineq2} and~\\eqref{ineq3}, together with the inequality $\\dk\\leq \\zeta^{-1}\\normii{\\QkTranspose \\nabla f(\\xk)}$, yields\n\\begin{eqnarray*}\nf(\\xk+\\Qk\\sk)&-&f(\\xk)\\leq \\normii{\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)}\\dk+\\frac{L_g}{2\\zeta}\\Qmax^2\\normii{\\QkTranspose \\nabla f(\\xk)}\\dk\\\\\n&\\leq& \\left(\\frac{L_g}{\\zeta}\\Qmax^2\\normii{\\QkTranspose \\nabla f(\\xk)} +\\normii{\\QkTranspose \\nabla f(\\xk)}\\right)\\dk\\\\\n&+&\\frac{L_g}{2\\zeta}\\Qmax^2\\normii{\\QkTranspose \\nabla f(\\xk)}\\dk= \\left(1+\\frac{3L_g}{2\\zeta}\\Qmax^2\\right)\\normii{\\QkTranspose \\nabla f(\\xk)}\\dk,\n\\end{eqnarray*}\nwhich completes the proof.\n\\end{proof}","start":142618,"end":143995}]},{"id":"S094","semantic_type":"proof_step","kind":"proof_step","labels":["ineq2"],"member_ids":["S094"],"declared_external":false,"source_only":false,"title":"以终点梯度控制目标差","source_units":[{"raw_tex":"According to the Fundamental Theorem of Calculus in $\\rn$ (see, e.g.,~\\cite[Lemma~9.3]{AuHa2017}),\\\\\n$\\abs{f(\\xk)-f(\\xk+\\Qk\\sk)-\\nabla f(\\xk+\\Qk\\sk)^{\\top}(-\\Qk\\sk)}\\leq\\frac{1}{2}L_g\\Qmax^2\\dk^2,$  with the inequality also following from $\\normii{\\Qk\\sk}\\leq \\Qmax\\dk$, which implies that\n\\begin{equation}\\label{ineq2}\nf(\\xk+\\Qk\\sk)-f(\\xk)\\leq \\sk^{\\top}\\left[\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)\\right] + \\frac{1}{2}L_g\\Qmax^2\\dk^2. \n\\end{equation}\n","start":142632,"end":143080}]},{"id":"S095","semantic_type":"proof_step","kind":"proof_step","labels":["ineq3"],"member_ids":["S095"],"declared_external":false,"source_only":false,"title":"终点投影梯度变化界","source_units":[{"raw_tex":"Per Lipschitz continuity of $\\nabla f$, \n$\\normii{\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)-\\QkTranspose \\nabla f(\\xk)} \\leq L_g\\Qmax^2\\dk,$\nwhich implies that\n\\begin{equation}\\label{ineq3}\n\\normii{\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)}\\leq L_g\\Qmax^2\\dk+\\normii{\\QkTranspose \\nabla f(\\xk)}.\n\\end{equation}\n","start":143080,"end":143375}]},{"id":"S096","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S096"],"declared_external":false,"source_only":false,"title":"利用大梯度吸收二次项","source_units":[{"raw_tex":"Thus, using~\\eqref{ineq2} and~\\eqref{ineq3}, together with the inequality $\\dk\\leq \\zeta^{-1}\\normii{\\QkTranspose \\nabla f(\\xk)}$, yields\n\\begin{eqnarray*}\nf(\\xk+\\Qk\\sk)&-&f(\\xk)\\leq \\normii{\\QkTranspose \\nabla f(\\xk+\\Qk\\sk)}\\dk+\\frac{L_g}{2\\zeta}\\Qmax^2\\normii{\\QkTranspose \\nabla f(\\xk)}\\dk\\\\\n&\\leq& \\left(\\frac{L_g}{\\zeta}\\Qmax^2\\normii{\\QkTranspose \\nabla f(\\xk)} +\\normii{\\QkTranspose \\nabla f(\\xk)}\\right)\\dk\\\\\n&+&\\frac{L_g}{2\\zeta}\\Qmax^2\\normii{\\QkTranspose \\nabla f(\\xk)}\\dk= \\left(1+\\frac{3L_g}{2\\zeta}\\Qmax^2\\right)\\normii{\\QkTranspose \\nabla f(\\xk)}\\dk,\n\\end{eqnarray*}\nwhich completes the proof.\n","start":143375,"end":143984}]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S007":"S007","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S031":"S031","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055","S056":"S056","S057":"S057","S058":"S058","S059":"S059","S060":"S060","S061":"S061","S062":"S062","S063":"S063","S064":"S064","S065":"S065","S066":"S066","S067":"S067","S068":"S068","S069":"S069","S070":"S070","S071":"S071","S072":"S072","S073":"S073","S074":"S074","S075":"S075","S076":"S076","S077":"S077","S078":"S078","S079":"S079","S080":"S080","S081":"S081","S082":"S082","S083":"S083","S084":"S084","S085":"S085","S086":"S086","S087":"S087","S088":"S088","S089":"S089","S090":"S090","S091":"S091","S092":"S092","S093":"S093","S094":"S094","S095":"S095","S096":"S096"}}}}
{"record_id":"2205.12772v2","paper":{"title":"A systematic approach to Lyapunov analyses of continuous-time models in convex optimization","authors":["Moucer, Céline","Taylor, Adrien","Bach, Francis"],"arxiv_id":"2205.12772","version":"v2","journal":"SIAM Journal on Optimization","abstract_url":"https://arxiv.org/abs/2205.12772v2","pdf_url":"https://arxiv.org/pdf/2205.12772v2","doi":"10.1137/22M1498486"},"context":{"raw_tex":"Convex optimization is an important tool in the numerical analyst toolbox. It serves, among others, for framing modeling problems in data science and signal processing. We consider optimization problems of the form:\n\\begin{equation}\n    \\min_{x \\in \\mathbf{R}^d} f(x),\n    \\label{eq:optim}\n\\end{equation}\n\n\n\\subsection{Assumptions}\n\\label{sec:assumptions}\nThroughout this work, functions to be minimized are convex (see Problem~\\eqref{eq:optim}). Under this assumption, stationary points are global minimizers. We restrict ourselves to continuous-time versions of gradient descent, accelerated gradient descent and stochastic gradient descent.\n\\smallbreak\nLet us recall a few basic definitions and properties characterizing the classes of functions under consideration within the next sections. A function $f : \\mathbf{R}^d \\rightarrow \\mathbf{R}$ is convex if for all $x,y \\in \\mathbf{R}^d$, and for all $\\lambda \\in [0,1]$, $f(\\lambda x + (1-\\lambda)y) \\leqslant \\lambda f(x) + (1-\\lambda)f(y)$. Such functions (i.e. with full domain, $\\text{dom}(f) = \\mathbf{R}^d$) are convex closed proper (CCP) (i.e., their epigraphs are non-empty closed convex sets). For simplicity, we assume in addition differentiability of $f$.  Such a differentiable function $f$ is convex if and only if for all $x,y \\in \\mathbf{R}^d$, $f(y) \\geqslant f(x) + \\langle \\nabla f(x), y-x \\rangle$. A differentiable function $f$ is $L$-smooth if its gradient is $L$-Lipschitz, that is if for any $x,y  \\ \\in \\mathbf{R}^d$:\n\\begin{equation*}\n        \\|\\nabla f(x) - \\nabla f(y)\\| \\leqslant L \\|x-y\\|.\n\\end{equation*}\nSmoothness is a common assumption for analyzing optimization methods, that limits the growth rate of the function. A convex differentiable function $f$ is $\\mu$-strongly convex if for any $x, y \\in \\mathbf{R}^d$ it satisfies:\n\\begin{equation*}\n         \\|\\nabla f(x) - \\nabla f(y)\\| \\geqslant \\mu \\|x-y\\|.\n\\end{equation*}\nStrong convexity ensures the function is not too flat, and the unicity of the minimizer~$x_\\star$. We denote by $\\mathcal{F}_{\\mu,L}$ the family of $L$-smooth $\\mu$-strongly convex functions from $\\mathbf{R}^d$ to $\\mathbf{R}$, with $0 \\leqslant \\mu \\leqslant L \\leqslant +\\infty$. Weaker assumptions than strong convexity are also encountered in the literature for analyzing gradient algorithms, and leads to similar convergence guarantees. Among them, Bolte~et~al.~\\cite[Appendix 5.2]{bolte2009} proved linear convergence of gradient descent under Łojasiewicz inequality, first introduced by Łojasiewicz~\\cite{Lojasiewicz1963}. Other relaxed versions of strong convexity followed~\\cite{necoara2019linear}.\n\n\nLet $f$ be $\\mu$-strongly convex (i.e., $f\\in\\mathcal{F}_{\\mu,\\infty}$) with $\\mu>0$, and $x_\\star$ be its unique minimizer such that $f(x_\\star) = f_\\star$. In this context, it is possible to prove linear convergence of the gradient flow to its stationary point. Scieur et~al.~proved in~\\cite[Proposition 1.1]{scieur2017integration} the following convergence bound in function values for the gradient flow:\n\\begin{equation}\n\\label{reflyapmu}\nf(X_t) - f_\\star \\leqslant e^{- 2 \\mu t}(f(x_0) - f_\\star).\n\\end{equation}\nThis convergence guarantee follows directly from the derivative with respect to time of the Lyapunov function $\\mathcal{V}(X_t) = f(X_t) - f_\\star$, together with strong convexity (or Łojasiewicz inequality): $\\frac{d}{dt}\\mathcal{V}(X_t) = \\dot{X_t}^\\top \\nabla f(X_t) = -\\|\\nabla f(X_t)\\|^2 \\leqslant -2 \\mu (f(X_t) - f_\\star) = -2 \\mu \\mathcal{V}(X_t)$.\n\n\\bigbreak\nGiven the specific gradient flows studied in this work, it is reasonable to search for Lyapunov functions made of linear combinations of function values, and a quadratic form in the trajectory $X_t$. We simply refer to them as quadratic Lyapunov functions:\n\\begin{equation}\n\\label{lyapunov}\n    \\mathcal{V}_{a,c}(X_t) = a \\cdot (f(X_t) - f_\\star) + c \\cdot \\|X_t - x_\\star\\|^2,\n\\end{equation}\nwhere $a, c \\geqslant 0$ are constants that do not depend on $t$, such that the function $ \\mathcal{V}_{a,c}$ is nonnegative and nonincreasing along the flow. Quadratic Lyapunov functions are common for proving convergence of gradient flows and cover for instance the Lyapunov function used to prove convergence of the gradient flow under strong convexity~\\eqref{reflyapmu}. Given a Lyapunov function $\\mathcal{V}_{a,c}$, the idea is to find the largest nonnegative value $\\tau_\\star$ such that the condition\n\\begin{equation}\n\\label{wc_mu}\n    \\frac{d}{dt}\\mathcal{V}_{a,c}(X_t) \\leqslant -\\tau_\\star \\mathcal{V}_{a,c}(X_t),\n\\end{equation}\nholds for any dimension $d \\in \\mathbf{N}$, any function $f \\in \\mathcal{F}_{\\mu, \\infty}$ and any trajectory $X_t$ generated by the gradient flow. After integration, \\eqref{wc_mu} allows to obtain a convergence guarantee of the form $\\mathcal{V}_{a,c}(X_t) \\leqslant e^{-\\tau_\\star t} \\mathcal{V}_{a,c}(X_0)$. Given a certain Lyapunov function $\\mathcal{V}_{a,c}$ and a time $t$, we get that the largest acceptable $\\tau_\\star$ is a solution to:\n\\begin{equation}\n\\label{eq:str_pep}\n    \\begin{aligned}\n      -\\tau_\\star = \\max_{X_t \\in \\mathbf{R}^d, d \\in \\mathbf{N}, f\\in \\mathcal{F}_{\\mu, \\infty}} \\ & \\  \\frac{d}{dt}\\mathcal{V}_{a,c}(X_t), \\\\\n      \\text{subject to }\n    \\mathcal{V}_{a,c}(X_t) &= 1, \\\\\n    \\dot{X_t} &= - \\nabla f (X_t).\n\\end{aligned}\n\\end{equation}\nThis minimization problem is invariant with respect to $t$. \nIt is established in~\\cite{taylor2016smooth, drori2012performance} that these so-called performance estimation problems (PEP) can be formulated as SDPs. Because of the variable $f \\in \\mathcal{F}_{\\mu, \\infty}$, the maximization problem~\\eqref{eq:str_pep} is infinite-dimensional. Recall that a differentiable function $f \\in \\mathcal{F}_{\\mu, \\infty}$ verifies for all points $x,y \\in \\mathbf{R}^d$, $f(x) - f(y) - \\langle \\nabla f(y), x-y \\rangle \\geqslant \\frac{\\mu}{2} \\|x-y\\|^2$. Using alternate variables $f_t$, $f_\\star$, $g_t$ and $g_\\star$ (informally: $f_t = f(X_t)$, $f_\\star = f(x_\\star)$, $ g_t = \\nabla f(X_t)$ and $g_\\star = \\nabla f(x_\\star) = 0$), it holds that:\n\\begin{equation}\n\\label{eq:interpolation_step}\n    \\begin{aligned}\n      -\\tau_\\star = \\max_{X_t \\in \\mathbf{R}^d, d \\in \\mathbf{N}} \\ & \\ \\frac{d}{dt}\\mathcal{V}_{a, c}(X_t), \\\\\n      \\text{subject to } &\\mathcal{V}_{a,c}(X_t) = 1, \\\\\n    &\\dot{X}_t = - g_t, \\\\\n    & f_i - f_j -\\langle g_j, X_i-X_j\\rangle \\geqslant \\frac{\\mu}{2}\\|X_i-X_j\\|^2, \\ \\forall i,j=t, \\star.\n\\end{aligned}\n\\end{equation}\nThe fact that~\\eqref{eq:interpolation_step} produces an upper bound on $-\\tau_\\star$ directly follows from the fact that any sampled strongly convex function satisfy these inequalities at the sampled points (here $X_t$ and $x_\\star$). Thereby, any feasible point to~\\eqref{eq:str_pep} corresponds to a feasible point for~\\eqref{eq:interpolation_step} with the same objective value. In the other direction,~\\cite[Corollary 2]{taylor2016smooth} (which provides a constructive way to obtain some $f \\in \\mathcal{F}_{\\mu, \\infty}$ that interpolates the triplets $(X_i, g_i, f_i)_{i=t, \\star}$) ensures that any feasible point to~\\eqref{eq:interpolation_step} can be translated to a feasible point to~\\eqref{eq:str_pep} with also the same objective value, thereby reaching the equivalence between formulations~\\eqref{eq:str_pep} and~\\eqref{eq:interpolation_step}.\n\\smallbreak\nIn a second stage, we introduce $G  = \\begin{pmatrix} \\|X_t - x_\\star\\|^2 &  \\langle X_t - x_\\star, g_t \\rangle  \\\\  \\langle X_t-x_\\star, g_t \\rangle & \\| g_t \\|^2 \\end{pmatrix} \\succcurlyeq 0$, a Gram matrix and a vector $F = [f_t, f_\\star]$, thereby obtaining a semidefinite reformulation:\n\\begin{equation}\n\\label{SDP}\n    \\begin{aligned}\n      -\\tau_\\star = \\max_{G\\succcurlyeq 0, F \\in \\mathbf{R}^2} & b_0^\\top F + {\\rm Tr}(A_0G), \\\\\n      \\text{subject to }\n    & b_1^\\top F + {\\rm Tr}(A_1G) \\geqslant 0, \\\\\n    & b_2^\\top F + {\\rm Tr}(A_2G) \\geqslant 0, \\\\\n    & b_3^\\top F + {\\rm Tr}(A_3G) = 1. \\\\\n\\end{aligned}\n\\end{equation}\nwhere $A_0 = \\begin{pmatrix} 0 & -c \\\\ -c & -a \\end{pmatrix}$, $A_1 = \\begin{pmatrix}-\\mu/2 & 1/2 \\\\ 1/2 & 0 \\end{pmatrix}$, $A_2= \\begin{pmatrix}-\\mu/2 & 0 \\\\ 0 & 0\\end{pmatrix}$, $A_3 = \\begin{pmatrix}c & 0 \\\\ 0 & 0 \\end{pmatrix}$, $b_0 = [0, 0]^\\top$, $b_1 = [-1,\\ 1]^\\top $, $b_2=[1, \\ -1]^\\top $ and $b_3 = a[1, \\ -1]^\\top $. Consider the corresponding Lagrangian, for $F \\in \\mathbf{R}^2$, $G \\succcurlyeq 0$, $\\tau \\in \\mathbf{R}$ and $\\lambda_1, \\lambda_2 \\geqslant 0$: \n\\begin{equation*}\n    \\begin{aligned}\n    \\mathcal{L}(F, G, \\tau, \\lambda_1, \\lambda_2) =& b_0^\\top F + {\\rm Tr}(A_0G) + \\tau \\cdot (b_3^\\top F + {\\rm Tr}(A_3G) - 1) \\\\ \n    & + \\lambda_1 \\cdot(b_1^\\top F + {\\rm Tr}(A_1G)) + \\lambda_2 \\cdot(b_2^\\top F + {\\rm Tr}(A_2G)).\n    \\end{aligned}\n\\end{equation*}\nThe saddle point of the Lagrangian is given by:\n\\begin{equation*}\n    \\begin{aligned}\n    \\tau_\\star &= \\min_{\\tau, \\lambda_1 \\geqslant 0, \\lambda_2 \\geqslant 0} \\max_{F, G \\succcurlyeq 0} \\mathcal{L}(F, G, \\tau, \\lambda_1, \\lambda_2).\n\\end{aligned}\n\\end{equation*}\nThe Lagrangian dual of the SDP~\\eqref{SDP} is obtained by maximizing over $F \\in \\mathbf{R}^2$, $G \\succcurlyeq 0$:\n\\begin{equation*}\n    \\begin{aligned}\n    -\\tau_\\star =  \\min_{\\tau, \\lambda_1, \\lambda_2} & \\ -\\tau, \\\\\n      \\text{subject to } & S = A_0 + \\lambda_1 A_1 + \\lambda_2 A_2 + \\tau A_3 \\preccurlyeq 0 \\\\ \n    &  b_0 +  \\lambda_1 b_1 +\\lambda_2 b_2  + \\tau b_3 = 0, \\\\\n    &  \\tau\\in \\mathbf{R}, \\ \\lambda_1, \\lambda_2 \\geqslant 0.\n\\end{aligned}\n\\end{equation*}\nThe equality with $\\tau_\\star$ comes from strong duality, that holds via Slater's conditions (it is relatively easy to show that there exists a Slater's point using the same construction as in~\\cite[Theorem 6]{taylor2016smooth}). Finally, since any feasible $\\tau$ is a lower bound for $\\tau_\\star$, those developments allow arriving to the equivalence between verifying a quadratic Lyapunov function and verifying the feasibility of an LMI.\n\n\nThe corresponding SDE is given by:\n\\begin{equation}\n\\label{eq:sme}\n    dX_t = -\\nabla f(X_t)dt + (\\gamma \\Sigma(X_t))^{1/2}dB_t,\n\\end{equation}\nwhere $B_t$ is a standard Brownian motion. The SDE~\\eqref{eq:sme} is an (order-$1$ weak) approximation of  SGD~(\\cite[Theorem 1]{li2017stochastic},~\\cite{Li2019}), which allows to take into account the role of constant step size in the dynamics of SGD (while keeping them small). Under mild assumptions on $f$, Li~et~al.~\\cite{Li2019} proved the weak approximation of SGD by this SDE on a finite interval $[0, T]$: there exists $C>0$ such that $\\|\\mathbf{E}[x_k] - \\mathbf{E}[X(k\\gamma)] \\| \\leqslant C\\gamma$ for $k \\in [0, \\frac{T}{\\gamma}]$. However, the approximation point of view from this approach is relatively limited since $C$ depends exponentially on~$T$. \n\n\nCompared to gradient descent, SGD does not converge to a stationary point under constant step sizes~\\cite{bach2011, taylor2021stochastic}. Convergence to a stationary point requires diminishing step sizes such as $\\gamma_k = \\frac{1}{\\sqrt{k}}$. Li~et~al.~\\cite[Section 4.1]{li2017stochastic} (and later Orvieto~and~Lucchi~\\cite[Section 2.1]{orvieto2020continuoustime}) proposed to include this varying learning rate in the dynamics:\n\\begin{equation*}\n    x_{k+1} = x_k - \\gamma h_k \\nabla f(x_k),\n\\end{equation*}\nwhere $\\gamma$ is the maximum allowed learning rate and $h_k \\in [0, 1]$ is the time-varying part. For $h_t \\geqslant 0$ a continuous function playing the role of step sizes $h_k$, the SDE is:\n\\begin{equation}\n\\label{sme_general}\n    dX_t = - h_t \\nabla f(X_t)dt + h_t(\\gamma \\Sigma(X_t))^{1/2}dB_t.\n\\end{equation}\nWe treat the covariance matrix $\\Sigma(X_t)$ as symmetric, already implied by the notation $\\Sigma(X_t)^{1/2}$, but unstructured with bounded variance $\\Sigma(X_t) \\preccurlyeq \\Sigma$ along any trajectory $X_t$ generated by the approximating SDE~\\eqref{sme_general}. Compared to ODEs, functions $f \\in \\mathcal{F}_{0, L}$ to be optimized using SDEs are in addition assumed to be possibly $L$-smooth with $L\\in (0, \\infty]$ and to be twice continuously differentiable.\n\n\n\\smallbreak\nIn SDE theory, a differential with respect to time of a function of a solution to a stochastic process is given by Ito's Lemma~\\cite[Theorem 4.2]{sarkka_solin_2019}.\n\\begin{lemma}[Ito's Lemma]\nLet $g$ be a twice continuously differentiable function, and $X_t$ be a stochastic process solution to the SDE~\\eqref{eq:sme}, then\n\\begin{equation*}\n    dg(X_t,t) = \\frac{\\partial}{\\partial t}g(X_t, t)dt + \\frac{\\partial}{\\partial x}g(X_t, t) dX_t + \\frac{1}{2}\\gamma {\\rm Tr}(\\frac{\\partial^2}{\\partial x^2}g(X_t, t)\\Sigma(X_t))dt.\n\\end{equation*}\n\\end{lemma}\n\n\n\\subsubsection{Polyak-Ruppert averaging}\nIn this section, we analyze convergence properties of an SDE~\\eqref{sme_general} approximating SGD with time-varying step sizes under Polyak-Ruppert averaging. Taylor~and~Bach~\\cite[Theorem 6]{taylor2021stochastic} provided a systematic design of Lyapunov functions for~\\eqref{eq:pr_avg}, and a condition on the step size for convergence to the optimum. An approximating SDE for Polyak-Ruppert averaging is given by:\n\\begin{equation}\n\\label{ODE_mean1}\n    \\begin{aligned}\n        dX_t &= -h_t\\nabla f(X_t)dt + h_t(\\gamma \\Sigma(X_t))^{1/2} dB_t, \\\\\n        d\\bar{X}_t &= \\frac{X_t - \\bar{X}_t}{t}dt,\n        \\end{aligned}\n    \\end{equation}\nwith step size $\\gamma>0$ that is taken close to zero, and a variable term $h_t\\in [0, 1]$. We introduce the family of quadratic functions taking the averaged sequence into account:\n\\begin{equation}\n\\label{lyap_avg}\n    \\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}(X_t, \\bar{X}_t, t) = \\begin{pmatrix}a^{(1)}_t \\\\ a^{(2)}_t \\end{pmatrix}^\\top \\!\\!\\!\\begin{pmatrix} f(X_t) - f_\\star \\\\ f(\\bar{X_t}) - f_\\star \\end{pmatrix} + \\begin{pmatrix} X_t - X_\\star\\\\ \\bar{X_t} - X_\\star\\end{pmatrix}^\\top \\!\\!\\! \\left(P_t\\otimes I_d\\right) \\! \\begin{pmatrix} X_t - X_\\star\\\\ \\bar{X_t} - X_\\star\\end{pmatrix},\n\\end{equation}\nwhere $a^{(1)}_t, a^{(2)}_t \\geqslant 0$ and $P_t = \\begin{pmatrix}p^{(11)}_t & p^{(12)}_t \\\\ p^{(12)}_t & p^{(22)}_t \\end{pmatrix}\\succcurlyeq 0$ are continuously differentiable with respect to time. Given a quadratic function $ \\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}$ and an SDE~\\eqref{ODE_mean1} under Polyak-Ruppert averaging, we present in Theorem~\\ref{pr_avg_conv} a  way to control the quantity:\n\\begin{equation*}\n    \\begin{aligned}\n      \\max_{f \\in \\mathcal{F}_{0, \\infty}, d \\in \\mathbf{N}, X_t \\in \\mathbf{R}^d} \\ & \\ \\frac{d}{dt}\\mathbf{E}\\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}(X_t, \\bar{X}_t, t), \\\\\n       \\text{subject to} \\ & dX_t = -h_t\\nabla f(X_t)dt + h_t(\\gamma \\Sigma(X_t))^{1/2} dB_t, \\\\\n        & d\\bar{X}_t = \\frac{X_t - \\bar{X}_t}{t}dt.\n    \\end{aligned}\n\\end{equation*}\n","units":[{"raw_tex":"Convex optimization is an important tool in the numerical analyst toolbox. It serves, among others, for framing modeling problems in data science and signal processing. We consider optimization problems of the form:\n\\begin{equation}\n    \\min_{x \\in \\mathbf{R}^d} f(x),\n    \\label{eq:optim}\n\\end{equation}\n","start":7680,"end":7985},{"raw_tex":"\\subsection{Assumptions}\n\\label{sec:assumptions}\nThroughout this work, functions to be minimized are convex (see Problem~\\eqref{eq:optim}). Under this assumption, stationary points are global minimizers. We restrict ourselves to continuous-time versions of gradient descent, accelerated gradient descent and stochastic gradient descent.\n\\smallbreak\nLet us recall a few basic definitions and properties characterizing the classes of functions under consideration within the next sections. A function $f : \\mathbf{R}^d \\rightarrow \\mathbf{R}$ is convex if for all $x,y \\in \\mathbf{R}^d$, and for all $\\lambda \\in [0,1]$, $f(\\lambda x + (1-\\lambda)y) \\leqslant \\lambda f(x) + (1-\\lambda)f(y)$. Such functions (i.e. with full domain, $\\text{dom}(f) = \\mathbf{R}^d$) are convex closed proper (CCP) (i.e., their epigraphs are non-empty closed convex sets). For simplicity, we assume in addition differentiability of $f$.  Such a differentiable function $f$ is convex if and only if for all $x,y \\in \\mathbf{R}^d$, $f(y) \\geqslant f(x) + \\langle \\nabla f(x), y-x \\rangle$. A differentiable function $f$ is $L$-smooth if its gradient is $L$-Lipschitz, that is if for any $x,y  \\ \\in \\mathbf{R}^d$:\n\\begin{equation*}\n        \\|\\nabla f(x) - \\nabla f(y)\\| \\leqslant L \\|x-y\\|.\n\\end{equation*}\nSmoothness is a common assumption for analyzing optimization methods, that limits the growth rate of the function. A convex differentiable function $f$ is $\\mu$-strongly convex if for any $x, y \\in \\mathbf{R}^d$ it satisfies:\n\\begin{equation*}\n         \\|\\nabla f(x) - \\nabla f(y)\\| \\geqslant \\mu \\|x-y\\|.\n\\end{equation*}\nStrong convexity ensures the function is not too flat, and the unicity of the minimizer~$x_\\star$. We denote by $\\mathcal{F}_{\\mu,L}$ the family of $L$-smooth $\\mu$-strongly convex functions from $\\mathbf{R}^d$ to $\\mathbf{R}$, with $0 \\leqslant \\mu \\leqslant L \\leqslant +\\infty$. Weaker assumptions than strong convexity are also encountered in the literature for analyzing gradient algorithms, and leads to similar convergence guarantees. Among them, Bolte~et~al.~\\cite[Appendix 5.2]{bolte2009} proved linear convergence of gradient descent under Łojasiewicz inequality, first introduced by Łojasiewicz~\\cite{Lojasiewicz1963}. Other relaxed versions of strong convexity followed~\\cite{necoara2019linear}.\n","start":20075,"end":22388},{"raw_tex":"Let $f$ be $\\mu$-strongly convex (i.e., $f\\in\\mathcal{F}_{\\mu,\\infty}$) with $\\mu>0$, and $x_\\star$ be its unique minimizer such that $f(x_\\star) = f_\\star$. In this context, it is possible to prove linear convergence of the gradient flow to its stationary point. Scieur et~al.~proved in~\\cite[Proposition 1.1]{scieur2017integration} the following convergence bound in function values for the gradient flow:\n\\begin{equation}\n\\label{reflyapmu}\nf(X_t) - f_\\star \\leqslant e^{- 2 \\mu t}(f(x_0) - f_\\star).\n\\end{equation}\nThis convergence guarantee follows directly from the derivative with respect to time of the Lyapunov function $\\mathcal{V}(X_t) = f(X_t) - f_\\star$, together with strong convexity (or Łojasiewicz inequality): $\\frac{d}{dt}\\mathcal{V}(X_t) = \\dot{X_t}^\\top \\nabla f(X_t) = -\\|\\nabla f(X_t)\\|^2 \\leqslant -2 \\mu (f(X_t) - f_\\star) = -2 \\mu \\mathcal{V}(X_t)$.\n\n\\bigbreak\nGiven the specific gradient flows studied in this work, it is reasonable to search for Lyapunov functions made of linear combinations of function values, and a quadratic form in the trajectory $X_t$. We simply refer to them as quadratic Lyapunov functions:\n\\begin{equation}\n\\label{lyapunov}\n    \\mathcal{V}_{a,c}(X_t) = a \\cdot (f(X_t) - f_\\star) + c \\cdot \\|X_t - x_\\star\\|^2,\n\\end{equation}\nwhere $a, c \\geqslant 0$ are constants that do not depend on $t$, such that the function $ \\mathcal{V}_{a,c}$ is nonnegative and nonincreasing along the flow. Quadratic Lyapunov functions are common for proving convergence of gradient flows and cover for instance the Lyapunov function used to prove convergence of the gradient flow under strong convexity~\\eqref{reflyapmu}. Given a Lyapunov function $\\mathcal{V}_{a,c}$, the idea is to find the largest nonnegative value $\\tau_\\star$ such that the condition\n\\begin{equation}\n\\label{wc_mu}\n    \\frac{d}{dt}\\mathcal{V}_{a,c}(X_t) \\leqslant -\\tau_\\star \\mathcal{V}_{a,c}(X_t),\n\\end{equation}\nholds for any dimension $d \\in \\mathbf{N}$, any function $f \\in \\mathcal{F}_{\\mu, \\infty}$ and any trajectory $X_t$ generated by the gradient flow. After integration, \\eqref{wc_mu} allows to obtain a convergence guarantee of the form $\\mathcal{V}_{a,c}(X_t) \\leqslant e^{-\\tau_\\star t} \\mathcal{V}_{a,c}(X_0)$. Given a certain Lyapunov function $\\mathcal{V}_{a,c}$ and a time $t$, we get that the largest acceptable $\\tau_\\star$ is a solution to:\n\\begin{equation}\n\\label{eq:str_pep}\n    \\begin{aligned}\n      -\\tau_\\star = \\max_{X_t \\in \\mathbf{R}^d, d \\in \\mathbf{N}, f\\in \\mathcal{F}_{\\mu, \\infty}} \\ & \\  \\frac{d}{dt}\\mathcal{V}_{a,c}(X_t), \\\\\n      \\text{subject to }\n    \\mathcal{V}_{a,c}(X_t) &= 1, \\\\\n    \\dot{X_t} &= - \\nabla f (X_t).\n\\end{aligned}\n\\end{equation}\nThis minimization problem is invariant with respect to $t$. \nIt is established in~\\cite{taylor2016smooth, drori2012performance} that these so-called performance estimation problems (PEP) can be formulated as SDPs. Because of the variable $f \\in \\mathcal{F}_{\\mu, \\infty}$, the maximization problem~\\eqref{eq:str_pep} is infinite-dimensional. Recall that a differentiable function $f \\in \\mathcal{F}_{\\mu, \\infty}$ verifies for all points $x,y \\in \\mathbf{R}^d$, $f(x) - f(y) - \\langle \\nabla f(y), x-y \\rangle \\geqslant \\frac{\\mu}{2} \\|x-y\\|^2$. Using alternate variables $f_t$, $f_\\star$, $g_t$ and $g_\\star$ (informally: $f_t = f(X_t)$, $f_\\star = f(x_\\star)$, $ g_t = \\nabla f(X_t)$ and $g_\\star = \\nabla f(x_\\star) = 0$), it holds that:\n\\begin{equation}\n\\label{eq:interpolation_step}\n    \\begin{aligned}\n      -\\tau_\\star = \\max_{X_t \\in \\mathbf{R}^d, d \\in \\mathbf{N}} \\ & \\ \\frac{d}{dt}\\mathcal{V}_{a, c}(X_t), \\\\\n      \\text{subject to } &\\mathcal{V}_{a,c}(X_t) = 1, \\\\\n    &\\dot{X}_t = - g_t, \\\\\n    & f_i - f_j -\\langle g_j, X_i-X_j\\rangle \\geqslant \\frac{\\mu}{2}\\|X_i-X_j\\|^2, \\ \\forall i,j=t, \\star.\n\\end{aligned}\n\\end{equation}\nThe fact that~\\eqref{eq:interpolation_step} produces an upper bound on $-\\tau_\\star$ directly follows from the fact that any sampled strongly convex function satisfy these inequalities at the sampled points (here $X_t$ and $x_\\star$). Thereby, any feasible point to~\\eqref{eq:str_pep} corresponds to a feasible point for~\\eqref{eq:interpolation_step} with the same objective value. In the other direction,~\\cite[Corollary 2]{taylor2016smooth} (which provides a constructive way to obtain some $f \\in \\mathcal{F}_{\\mu, \\infty}$ that interpolates the triplets $(X_i, g_i, f_i)_{i=t, \\star}$) ensures that any feasible point to~\\eqref{eq:interpolation_step} can be translated to a feasible point to~\\eqref{eq:str_pep} with also the same objective value, thereby reaching the equivalence between formulations~\\eqref{eq:str_pep} and~\\eqref{eq:interpolation_step}.\n\\smallbreak\nIn a second stage, we introduce $G  = \\begin{pmatrix} \\|X_t - x_\\star\\|^2 &  \\langle X_t - x_\\star, g_t \\rangle  \\\\  \\langle X_t-x_\\star, g_t \\rangle & \\| g_t \\|^2 \\end{pmatrix} \\succcurlyeq 0$, a Gram matrix and a vector $F = [f_t, f_\\star]$, thereby obtaining a semidefinite reformulation:\n\\begin{equation}\n\\label{SDP}\n    \\begin{aligned}\n      -\\tau_\\star = \\max_{G\\succcurlyeq 0, F \\in \\mathbf{R}^2} & b_0^\\top F + {\\rm Tr}(A_0G), \\\\\n      \\text{subject to }\n    & b_1^\\top F + {\\rm Tr}(A_1G) \\geqslant 0, \\\\\n    & b_2^\\top F + {\\rm Tr}(A_2G) \\geqslant 0, \\\\\n    & b_3^\\top F + {\\rm Tr}(A_3G) = 1. \\\\\n\\end{aligned}\n\\end{equation}\nwhere $A_0 = \\begin{pmatrix} 0 & -c \\\\ -c & -a \\end{pmatrix}$, $A_1 = \\begin{pmatrix}-\\mu/2 & 1/2 \\\\ 1/2 & 0 \\end{pmatrix}$, $A_2= \\begin{pmatrix}-\\mu/2 & 0 \\\\ 0 & 0\\end{pmatrix}$, $A_3 = \\begin{pmatrix}c & 0 \\\\ 0 & 0 \\end{pmatrix}$, $b_0 = [0, 0]^\\top$, $b_1 = [-1,\\ 1]^\\top $, $b_2=[1, \\ -1]^\\top $ and $b_3 = a[1, \\ -1]^\\top $. Consider the corresponding Lagrangian, for $F \\in \\mathbf{R}^2$, $G \\succcurlyeq 0$, $\\tau \\in \\mathbf{R}$ and $\\lambda_1, \\lambda_2 \\geqslant 0$: \n\\begin{equation*}\n    \\begin{aligned}\n    \\mathcal{L}(F, G, \\tau, \\lambda_1, \\lambda_2) =& b_0^\\top F + {\\rm Tr}(A_0G) + \\tau \\cdot (b_3^\\top F + {\\rm Tr}(A_3G) - 1) \\\\ \n    & + \\lambda_1 \\cdot(b_1^\\top F + {\\rm Tr}(A_1G)) + \\lambda_2 \\cdot(b_2^\\top F + {\\rm Tr}(A_2G)).\n    \\end{aligned}\n\\end{equation*}\nThe saddle point of the Lagrangian is given by:\n\\begin{equation*}\n    \\begin{aligned}\n    \\tau_\\star &= \\min_{\\tau, \\lambda_1 \\geqslant 0, \\lambda_2 \\geqslant 0} \\max_{F, G \\succcurlyeq 0} \\mathcal{L}(F, G, \\tau, \\lambda_1, \\lambda_2).\n\\end{aligned}\n\\end{equation*}\nThe Lagrangian dual of the SDP~\\eqref{SDP} is obtained by maximizing over $F \\in \\mathbf{R}^2$, $G \\succcurlyeq 0$:\n\\begin{equation*}\n    \\begin{aligned}\n    -\\tau_\\star =  \\min_{\\tau, \\lambda_1, \\lambda_2} & \\ -\\tau, \\\\\n      \\text{subject to } & S = A_0 + \\lambda_1 A_1 + \\lambda_2 A_2 + \\tau A_3 \\preccurlyeq 0 \\\\ \n    &  b_0 +  \\lambda_1 b_1 +\\lambda_2 b_2  + \\tau b_3 = 0, \\\\\n    &  \\tau\\in \\mathbf{R}, \\ \\lambda_1, \\lambda_2 \\geqslant 0.\n\\end{aligned}\n\\end{equation*}\nThe equality with $\\tau_\\star$ comes from strong duality, that holds via Slater's conditions (it is relatively easy to show that there exists a Slater's point using the same construction as in~\\cite[Theorem 6]{taylor2016smooth}). Finally, since any feasible $\\tau$ is a lower bound for $\\tau_\\star$, those developments allow arriving to the equivalence between verifying a quadratic Lyapunov function and verifying the feasibility of an LMI.\n","start":23778,"end":31080},{"raw_tex":"The corresponding SDE is given by:\n\\begin{equation}\n\\label{eq:sme}\n    dX_t = -\\nabla f(X_t)dt + (\\gamma \\Sigma(X_t))^{1/2}dB_t,\n\\end{equation}\nwhere $B_t$ is a standard Brownian motion. The SDE~\\eqref{eq:sme} is an (order-$1$ weak) approximation of  SGD~(\\cite[Theorem 1]{li2017stochastic},~\\cite{Li2019}), which allows to take into account the role of constant step size in the dynamics of SGD (while keeping them small). Under mild assumptions on $f$, Li~et~al.~\\cite{Li2019} proved the weak approximation of SGD by this SDE on a finite interval $[0, T]$: there exists $C>0$ such that $\\|\\mathbf{E}[x_k] - \\mathbf{E}[X(k\\gamma)] \\| \\leqslant C\\gamma$ for $k \\in [0, \\frac{T}{\\gamma}]$. However, the approximation point of view from this approach is relatively limited since $C$ depends exponentially on~$T$. \n","start":69903,"end":70715},{"raw_tex":"Compared to gradient descent, SGD does not converge to a stationary point under constant step sizes~\\cite{bach2011, taylor2021stochastic}. Convergence to a stationary point requires diminishing step sizes such as $\\gamma_k = \\frac{1}{\\sqrt{k}}$. Li~et~al.~\\cite[Section 4.1]{li2017stochastic} (and later Orvieto~and~Lucchi~\\cite[Section 2.1]{orvieto2020continuoustime}) proposed to include this varying learning rate in the dynamics:\n\\begin{equation*}\n    x_{k+1} = x_k - \\gamma h_k \\nabla f(x_k),\n\\end{equation*}\nwhere $\\gamma$ is the maximum allowed learning rate and $h_k \\in [0, 1]$ is the time-varying part. For $h_t \\geqslant 0$ a continuous function playing the role of step sizes $h_k$, the SDE is:\n\\begin{equation}\n\\label{sme_general}\n    dX_t = - h_t \\nabla f(X_t)dt + h_t(\\gamma \\Sigma(X_t))^{1/2}dB_t.\n\\end{equation}\nWe treat the covariance matrix $\\Sigma(X_t)$ as symmetric, already implied by the notation $\\Sigma(X_t)^{1/2}$, but unstructured with bounded variance $\\Sigma(X_t) \\preccurlyeq \\Sigma$ along any trajectory $X_t$ generated by the approximating SDE~\\eqref{sme_general}. Compared to ODEs, functions $f \\in \\mathcal{F}_{0, L}$ to be optimized using SDEs are in addition assumed to be possibly $L$-smooth with $L\\in (0, \\infty]$ and to be twice continuously differentiable.\n","start":71221,"end":72519},{"raw_tex":"\\smallbreak\nIn SDE theory, a differential with respect to time of a function of a solution to a stochastic process is given by Ito's Lemma~\\cite[Theorem 4.2]{sarkka_solin_2019}.\n\\begin{lemma}[Ito's Lemma]\nLet $g$ be a twice continuously differentiable function, and $X_t$ be a stochastic process solution to the SDE~\\eqref{eq:sme}, then\n\\begin{equation*}\n    dg(X_t,t) = \\frac{\\partial}{\\partial t}g(X_t, t)dt + \\frac{\\partial}{\\partial x}g(X_t, t) dX_t + \\frac{1}{2}\\gamma {\\rm Tr}(\\frac{\\partial^2}{\\partial x^2}g(X_t, t)\\Sigma(X_t))dt.\n\\end{equation*}\n\\end{lemma}\n","start":73421,"end":73988},{"raw_tex":"\\subsubsection{Polyak-Ruppert averaging}\nIn this section, we analyze convergence properties of an SDE~\\eqref{sme_general} approximating SGD with time-varying step sizes under Polyak-Ruppert averaging. Taylor~and~Bach~\\cite[Theorem 6]{taylor2021stochastic} provided a systematic design of Lyapunov functions for~\\eqref{eq:pr_avg}, and a condition on the step size for convergence to the optimum. An approximating SDE for Polyak-Ruppert averaging is given by:\n\\begin{equation}\n\\label{ODE_mean1}\n    \\begin{aligned}\n        dX_t &= -h_t\\nabla f(X_t)dt + h_t(\\gamma \\Sigma(X_t))^{1/2} dB_t, \\\\\n        d\\bar{X}_t &= \\frac{X_t - \\bar{X}_t}{t}dt,\n        \\end{aligned}\n    \\end{equation}\nwith step size $\\gamma>0$ that is taken close to zero, and a variable term $h_t\\in [0, 1]$. We introduce the family of quadratic functions taking the averaged sequence into account:\n\\begin{equation}\n\\label{lyap_avg}\n    \\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}(X_t, \\bar{X}_t, t) = \\begin{pmatrix}a^{(1)}_t \\\\ a^{(2)}_t \\end{pmatrix}^\\top \\!\\!\\!\\begin{pmatrix} f(X_t) - f_\\star \\\\ f(\\bar{X_t}) - f_\\star \\end{pmatrix} + \\begin{pmatrix} X_t - X_\\star\\\\ \\bar{X_t} - X_\\star\\end{pmatrix}^\\top \\!\\!\\! \\left(P_t\\otimes I_d\\right) \\! \\begin{pmatrix} X_t - X_\\star\\\\ \\bar{X_t} - X_\\star\\end{pmatrix},\n\\end{equation}\nwhere $a^{(1)}_t, a^{(2)}_t \\geqslant 0$ and $P_t = \\begin{pmatrix}p^{(11)}_t & p^{(12)}_t \\\\ p^{(12)}_t & p^{(22)}_t \\end{pmatrix}\\succcurlyeq 0$ are continuously differentiable with respect to time. Given a quadratic function $ \\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}$ and an SDE~\\eqref{ODE_mean1} under Polyak-Ruppert averaging, we present in Theorem~\\ref{pr_avg_conv} a  way to control the quantity:\n\\begin{equation*}\n    \\begin{aligned}\n      \\max_{f \\in \\mathcal{F}_{0, \\infty}, d \\in \\mathbf{N}, X_t \\in \\mathbf{R}^d} \\ & \\ \\frac{d}{dt}\\mathbf{E}\\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}(X_t, \\bar{X}_t, t), \\\\\n       \\text{subject to} \\ & dX_t = -h_t\\nabla f(X_t)dt + h_t(\\gamma \\Sigma(X_t))^{1/2} dB_t, \\\\\n        & d\\bar{X}_t = \\frac{X_t - \\bar{X}_t}{t}dt.\n    \\end{aligned}\n\\end{equation*}\n","start":83501,"end":85590}]},"statement":{"raw_tex":"\\begin{theorem}\n\\label{pr_avg_conv}\nLet $h_t \\geqslant 0$, and $a_t \\geqslant 0$, $P_t \\succcurlyeq 0$ be continuously differentiable with respect to time. Let  $f \\in \\mathcal{F}_{0, \\infty}$ be a twice continuously differentiable function and $(X_t, \\bar{X}_t) \\in \\mathbf{R}^d \\times \\mathbf{R}^d$ be a trajectory generated by the SDE under Polyak-Ruppert averaging~\\eqref{ODE_mean1}. Let $\\mathcal{V}$ be a quadratic function~\\eqref{lyap_avg}, such that there exist $\\lambda^{(1)}_t, ..., \\lambda^{(6)}_t \\geqslant 0$ verifying:\n\\begin{equation*}\n    \\begin{aligned}\n&\\begin{pmatrix} \\dot{p}^{(11)}_t + \\frac{2p^{(12)}_t}{t} & \\dot{p}^{(12)}_t + \\frac{p^{(22)}_t - p^{(12)}_t}{t}& \\frac{\\lambda^{(6)}_t + \\lambda^{(4)}_t - 2h_tp^{(11)}_t}{2} & \\frac{a^{(2)}_t}{2t} - \\frac{\\lambda^{(5)}_t}{2}\\\\ \n\n\\dot{p}^{(12)}_t + \\frac{p^{(22)}_t - p^{(12)}_t}{t} & \\dot{p}^{(22)}_t - \\frac{2p^{(22)}_t}{t} & -\\frac{\\lambda^{(6)}_t - 2h_tp^{(12)}_t}{2} & -\\frac{a^{(2)}_t}{2t} + \\frac{\\lambda^{(5)}_t + \\lambda^{(3)}_t}{2}\\\\ \n\n \\frac{\\lambda^{(6)}_t + \\lambda^{(4)}_t - 2h_tp^{(11)}_t}{2} & -\\frac{\\lambda^{(6)}_t - 2h_tp^{(12)}_t}{2} & -a^{(1)}_t & 0 \\\\ \n\n\\frac{a^{(2)}_t}{2t} - \\frac{\\lambda^{(5)}_t}{2} & -\\frac{a^{(2)}_t}{2t} + \\frac{\\lambda^{(5)}_t + \\lambda^{(3)}_t}{2} & 0 &  0\\end{pmatrix} \\preccurlyeq 0, \\\\ \n&\\dot{a}^{(1)}_t + \\lambda^{(1)}_t + \\lambda^{(5)}_t = \\lambda^{(4)}_t + \\lambda^{(6)}_t, \\\\ \n&\\dot{a}^{(2)}_t + \\lambda^{(2)}_t + \\lambda^{(6)}_t = \\lambda^{(3)}_t + \\lambda^{(5)}_t + \\dot{a}^{(1)}_t. \n\\end{aligned}\n\\end{equation*}\nThen, the following inequality is satisfied:\n\\smallbreak \n$\\displaystyle\\frac{d}{dt}\\mathbf{E}\\mathcal{V}(X_t, \\bar{X}_t, t)\\leqslant \\frac{1}{2}\\mathbf{E}{\\rm Tr}\\left((a^{(1)}_t\\nabla_{xx} f(X_t) + 2p^{(11)}_t I_d)\\Sigma(X_t)\\right)h_t^2\\gamma$.\n\\end{theorem}","units":[{"raw_tex":"\\begin{theorem}\n\\label{pr_avg_conv}\nLet $h_t \\geqslant 0$, and $a_t \\geqslant 0$, $P_t \\succcurlyeq 0$ be continuously differentiable with respect to time. Let  $f \\in \\mathcal{F}_{0, \\infty}$ be a twice continuously differentiable function and $(X_t, \\bar{X}_t) \\in \\mathbf{R}^d \\times \\mathbf{R}^d$ be a trajectory generated by the SDE under Polyak-Ruppert averaging~\\eqref{ODE_mean1}. Let $\\mathcal{V}$ be a quadratic function~\\eqref{lyap_avg}, such that there exist $\\lambda^{(1)}_t, ..., \\lambda^{(6)}_t \\geqslant 0$ verifying:\n\\begin{equation*}\n    \\begin{aligned}\n&\\begin{pmatrix} \\dot{p}^{(11)}_t + \\frac{2p^{(12)}_t}{t} & \\dot{p}^{(12)}_t + \\frac{p^{(22)}_t - p^{(12)}_t}{t}& \\frac{\\lambda^{(6)}_t + \\lambda^{(4)}_t - 2h_tp^{(11)}_t}{2} & \\frac{a^{(2)}_t}{2t} - \\frac{\\lambda^{(5)}_t}{2}\\\\ \n\n\\dot{p}^{(12)}_t + \\frac{p^{(22)}_t - p^{(12)}_t}{t} & \\dot{p}^{(22)}_t - \\frac{2p^{(22)}_t}{t} & -\\frac{\\lambda^{(6)}_t - 2h_tp^{(12)}_t}{2} & -\\frac{a^{(2)}_t}{2t} + \\frac{\\lambda^{(5)}_t + \\lambda^{(3)}_t}{2}\\\\ \n\n \\frac{\\lambda^{(6)}_t + \\lambda^{(4)}_t - 2h_tp^{(11)}_t}{2} & -\\frac{\\lambda^{(6)}_t - 2h_tp^{(12)}_t}{2} & -a^{(1)}_t & 0 \\\\ \n\n\\frac{a^{(2)}_t}{2t} - \\frac{\\lambda^{(5)}_t}{2} & -\\frac{a^{(2)}_t}{2t} + \\frac{\\lambda^{(5)}_t + \\lambda^{(3)}_t}{2} & 0 &  0\\end{pmatrix} \\preccurlyeq 0, \\\\ \n&\\dot{a}^{(1)}_t + \\lambda^{(1)}_t + \\lambda^{(5)}_t = \\lambda^{(4)}_t + \\lambda^{(6)}_t, \\\\ \n&\\dot{a}^{(2)}_t + \\lambda^{(2)}_t + \\lambda^{(6)}_t = \\lambda^{(3)}_t + \\lambda^{(5)}_t + \\dot{a}^{(1)}_t. \n\\end{aligned}\n\\end{equation*}\nThen, the following inequality is satisfied:\n\\smallbreak \n$\\displaystyle\\frac{d}{dt}\\mathbf{E}\\mathcal{V}(X_t, \\bar{X}_t, t)\\leqslant \\frac{1}{2}\\mathbf{E}{\\rm Tr}\\left((a^{(1)}_t\\nabla_{xx} f(X_t) + 2p^{(11)}_t I_d)\\Sigma(X_t)\\right)h_t^2\\gamma$.\n\\end{theorem}","start":85591,"end":87393}]},"proof":{"raw_tex":"\\begin{proof}\nThe proof follows the method from Section~\\ref{sec:methodo} (see Appendix~\\ref{sec:A11}).\n\\end{proof}\n\n\\begin{proof}\n\nWe rewrite the SDE into \n\\begin{equation*}\n\\begin{aligned}\n    \\begin{pmatrix} dX_t \\\\ d\\bar{X}_t\\end{pmatrix} =& \\left(\\begin{pmatrix} 0 & 0 \\\\ \\frac{1}{t} & -\\frac{1}{t} \\end{pmatrix}\\otimes I_d\\right)\\begin{pmatrix} X_t \\\\ \\bar{X}_t\\end{pmatrix}dt +\\left(\\begin{pmatrix}- h_t & 0 \\\\ 0  & 0  \\end{pmatrix}\\otimes I_d\\right)\\begin{pmatrix} \\nabla f(X_t) \\\\ \\nabla f(\\bar{X}_t) \\end{pmatrix}dt \\\\\n    &+\\left(\\begin{pmatrix}h_t (\\gamma\\Sigma(X_t))^{1/2} \\\\ 0\\end{pmatrix}\\otimes I_d\\right)dB_t\n\\end{aligned}\n\\end{equation*}\n and denote $Y_t = \\begin{pmatrix} X_t \\\\ \\bar{X}_t \\end{pmatrix}$. \n \\smallbreak\n We consider the quadratic function~\\eqref{lyap_avg} $\\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}(X_t, \\bar{X}_t ,t) = \\mathcal{V}(Y_t,t)$. Applying Ito's formula, its derivative with respect to time is $\\frac{d}{dt} \\mathcal{V}(Y_t,t) = \\frac{\\partial}{\\partial t}\\mathcal{V}(Y_t,t) + \\frac{\\partial}{\\partial y}\\mathcal{V}\\frac{dY_t}{dt} + \\frac{1}{2}\\gamma h^2_t {\\rm Tr}[\\frac{\\partial^2}{\\partial Y_t^2}\\mathcal{V}(Y_t,t)^\\top \\begin{pmatrix}\\Sigma(X_t) \\\\ 0\\end{pmatrix}]$. By taking the expectation of Ito's formula, we have $\\frac{d}{dt} \\mathbf{E}\\mathcal{V}(Y_t,t) = \\mathbf{E}[\\frac{\\partial}{\\partial t}\\mathcal{V}(Y_t,t) + \\frac{\\partial}{\\partial y}\\mathcal{V}\\frac{dY_t}{dt}] +\\frac{1}{2}\\gamma h^2_t \\mathbf{E}{\\rm Tr}(\\frac{\\partial^2}{\\partial Y_t^2}\\mathcal{V}(Y_t,t)^\\top \\begin{pmatrix}\\Sigma(X_t) \\\\ 0\\end{pmatrix})$.\n \\smallbreak\n The variance term $\\frac{1}{2}\\gamma h^2_t \\mathbf{E}{\\rm Tr}(\\frac{\\partial^2}{\\partial Y_t^2}\\mathcal{V}(Y_t,t)^\\top \\begin{pmatrix}\\Sigma(X_t) \\\\ 0\\end{pmatrix})$ depends on the second derivative of $f$ in the space variable $(X_t)$, and on the covariance $\\Sigma_t$. We do not take this term into account in the performance estimation frawework.\n \\smallbreak\n We formulate the problem of verifying a quadratic function as verifying the inequality $\\frac{d}{dt}\\mathbf{E}\\mathcal{V}(X_t, \\bar{X}_t,t) \\leqslant \\frac{1}{2}\\gamma h^2_t \\mathbf{E}{\\rm Tr}(\\frac{\\partial^2}{\\partial X_t^2}\\mathcal{V}(X_t, \\bar{X}_t, t)^\\top \\Sigma(X_t))$ (that is $\\mathbf{E}[\\frac{\\partial}{\\partial t}\\mathcal{V}(Y_t,t) + \\frac{\\partial}{\\partial y}\\mathcal{V}\\frac{dY_t}{dt}] \\leqslant 0$) holds for any twice continuously differentiable function $f \\in \\mathcal{F}_{0, \\infty}$, and any trajectory $(X_t, \\bar{X}_t)$ generated by the SDE under Polyak-Ruppert averaging~\\eqref{ODE_mean1}. \n \\smallbreak\n This problem is equivalent with verifying that $\\frac{d}{dt}\\mathcal{V}(\\tilde{Y}_t,t) \\leqslant 0$ holds for any function twice continuously differentiable $f \\in \\mathcal{F}_{0, \\infty}$ and any trajectory $\\tilde{Y}_t = \\begin{pmatrix} \\tilde{X}_t & \\tilde{\\bar{X}}_t\\end{pmatrix}^\\top $ generated by deterministic gradient flow from the SDE~\\eqref{ODE_mean1} with $\\gamma = 0$. \n\\smallbreak\nWe follow the methodology developed in Section~\\ref{sec:deterministic} for ODEs. We formulate the verification of such a Lyapunov function as the maximization problem\n\\begin{equation*}\n    \\begin{aligned}\n      0\\geqslant \\max_{X_t \\in \\mathbf{R}^d,  d\\in \\mathbf{N}, \\ f\\in \\mathcal{F}_{0, \\infty}} \\ & \\ \\frac{d}{dt}  \\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}(X_t, \\bar{X}_t, t)\\\\\n      \\text{subject to }\n    \\ \\dot{X}_t &= -h_t\\nabla f(\\bar{X}_t), \\\n        \\dot{\\bar{X}}_t = \\frac{X_t - \\bar{X}_t}{t}dt.\n\\end{aligned}\n\\end{equation*}\nThis maximization problem can be formulated as the following SDP program\n\n\\begin{equation*}\n    \\begin{aligned}\n    & \\min_{\\lambda_t^{(i)} \\geqslant 0, \\ i \\in \\{1, ..., 6 \\}} \\ 0 \\, \\\\\n&\\begin{pmatrix} \\dot{p}^{(11)}_t + \\frac{2p^{(12)}_t}{t} & \\dot{p}^{(12)}_t + \\frac{p^{(22)}_t - p^{(12)}_t}{t}& \\frac{\\lambda^{(6)}_t + \\lambda^{(4)}_t - 2h_tp^{(11)}_t}{2} & \\frac{a^{(2)}_t}{2t} - \\frac{\\lambda^{(5)}_t}{2}\\\\ \n\n\\dot{p}^{(12)}_t + \\frac{p^{(22)}_t - p^{(12)}_t}{t} & \\dot{p}^{(22)}_t - \\frac{2p^{(22)}_t}{t} & -\\frac{\\lambda^{(6)}_t - 2h_tp^{(12)}_t}{2} & -\\frac{a^{(2)}_t}{2t} + \\frac{\\lambda^{(5)}_t + \\lambda^{(3)}_t}{2}\\\\ \n\n \\frac{\\lambda^{(6)}_t + \\lambda^{(4)}_t - 2h_tp^{(11)}_t}{2} & -\\frac{\\lambda^{(6)}_t - 2h_tp^{(12)}_t}{2} & -a^{(1)}_t & 0 \\\\ \n\n\\frac{a^{(2)}_t}{2t} - \\frac{\\lambda^{(5)}_t}{2} & -\\frac{a^{(2)}_t}{2t} + \\frac{\\lambda^{(5)}_t + \\lambda^{(3)}_t}{2} & 0 &  0\\end{pmatrix} \\preccurlyeq 0, \\\\ \n&\\dot{a}^{(1)}_t + \\lambda^{(1)}_t + \\lambda^{(5)}_t = \\lambda^{(4)}_t + \\lambda^{(6)}_t, \\\\ \n&\\dot{a}^{(2)}_t + \\lambda^{(2)}_t + \\lambda^{(6)}_t = \\lambda^{(3)}_t + \\lambda^{(5)}_t + \\dot{a}^{(1)}_t. \n\\end{aligned}\n\\end{equation*}\nNote that the Gram matrix $G$ and function value vector $F$ to be introduced are different from the setting explored in Section~\\ref{sec:deterministic}, more precisely $G = Q^\\top Q \\succcurlyeq 0, \\ Q = [X_t - x_\\star, \\bar{X}_t - x_\\star, g_t, \\bar{g}_t]$ and $F = (f(X_t), f(\\bar{X}_t), f_\\star)$.\n\nThe final inequality follows from Ito's formula.\n\\end{proof}","units":[{"raw_tex":"\\begin{proof}\nThe proof follows the method from Section~\\ref{sec:methodo} (see Appendix~\\ref{sec:A11}).\n\\end{proof}","start":87394,"end":87509},{"raw_tex":"\\begin{proof}\n\nWe rewrite the SDE into \n\\begin{equation*}\n\\begin{aligned}\n    \\begin{pmatrix} dX_t \\\\ d\\bar{X}_t\\end{pmatrix} =& \\left(\\begin{pmatrix} 0 & 0 \\\\ \\frac{1}{t} & -\\frac{1}{t} \\end{pmatrix}\\otimes I_d\\right)\\begin{pmatrix} X_t \\\\ \\bar{X}_t\\end{pmatrix}dt +\\left(\\begin{pmatrix}- h_t & 0 \\\\ 0  & 0  \\end{pmatrix}\\otimes I_d\\right)\\begin{pmatrix} \\nabla f(X_t) \\\\ \\nabla f(\\bar{X}_t) \\end{pmatrix}dt \\\\\n    &+\\left(\\begin{pmatrix}h_t (\\gamma\\Sigma(X_t))^{1/2} \\\\ 0\\end{pmatrix}\\otimes I_d\\right)dB_t\n\\end{aligned}\n\\end{equation*}\n and denote $Y_t = \\begin{pmatrix} X_t \\\\ \\bar{X}_t \\end{pmatrix}$. \n \\smallbreak\n We consider the quadratic function~\\eqref{lyap_avg} $\\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}(X_t, \\bar{X}_t ,t) = \\mathcal{V}(Y_t,t)$. Applying Ito's formula, its derivative with respect to time is $\\frac{d}{dt} \\mathcal{V}(Y_t,t) = \\frac{\\partial}{\\partial t}\\mathcal{V}(Y_t,t) + \\frac{\\partial}{\\partial y}\\mathcal{V}\\frac{dY_t}{dt} + \\frac{1}{2}\\gamma h^2_t {\\rm Tr}[\\frac{\\partial^2}{\\partial Y_t^2}\\mathcal{V}(Y_t,t)^\\top \\begin{pmatrix}\\Sigma(X_t) \\\\ 0\\end{pmatrix}]$. By taking the expectation of Ito's formula, we have $\\frac{d}{dt} \\mathbf{E}\\mathcal{V}(Y_t,t) = \\mathbf{E}[\\frac{\\partial}{\\partial t}\\mathcal{V}(Y_t,t) + \\frac{\\partial}{\\partial y}\\mathcal{V}\\frac{dY_t}{dt}] +\\frac{1}{2}\\gamma h^2_t \\mathbf{E}{\\rm Tr}(\\frac{\\partial^2}{\\partial Y_t^2}\\mathcal{V}(Y_t,t)^\\top \\begin{pmatrix}\\Sigma(X_t) \\\\ 0\\end{pmatrix})$.\n \\smallbreak\n The variance term $\\frac{1}{2}\\gamma h^2_t \\mathbf{E}{\\rm Tr}(\\frac{\\partial^2}{\\partial Y_t^2}\\mathcal{V}(Y_t,t)^\\top \\begin{pmatrix}\\Sigma(X_t) \\\\ 0\\end{pmatrix})$ depends on the second derivative of $f$ in the space variable $(X_t)$, and on the covariance $\\Sigma_t$. We do not take this term into account in the performance estimation frawework.\n \\smallbreak\n We formulate the problem of verifying a quadratic function as verifying the inequality $\\frac{d}{dt}\\mathbf{E}\\mathcal{V}(X_t, \\bar{X}_t,t) \\leqslant \\frac{1}{2}\\gamma h^2_t \\mathbf{E}{\\rm Tr}(\\frac{\\partial^2}{\\partial X_t^2}\\mathcal{V}(X_t, \\bar{X}_t, t)^\\top \\Sigma(X_t))$ (that is $\\mathbf{E}[\\frac{\\partial}{\\partial t}\\mathcal{V}(Y_t,t) + \\frac{\\partial}{\\partial y}\\mathcal{V}\\frac{dY_t}{dt}] \\leqslant 0$) holds for any twice continuously differentiable function $f \\in \\mathcal{F}_{0, \\infty}$, and any trajectory $(X_t, \\bar{X}_t)$ generated by the SDE under Polyak-Ruppert averaging~\\eqref{ODE_mean1}. \n \\smallbreak\n This problem is equivalent with verifying that $\\frac{d}{dt}\\mathcal{V}(\\tilde{Y}_t,t) \\leqslant 0$ holds for any function twice continuously differentiable $f \\in \\mathcal{F}_{0, \\infty}$ and any trajectory $\\tilde{Y}_t = \\begin{pmatrix} \\tilde{X}_t & \\tilde{\\bar{X}}_t\\end{pmatrix}^\\top $ generated by deterministic gradient flow from the SDE~\\eqref{ODE_mean1} with $\\gamma = 0$. \n\\smallbreak\nWe follow the methodology developed in Section~\\ref{sec:deterministic} for ODEs. We formulate the verification of such a Lyapunov function as the maximization problem\n\\begin{equation*}\n    \\begin{aligned}\n      0\\geqslant \\max_{X_t \\in \\mathbf{R}^d,  d\\in \\mathbf{N}, \\ f\\in \\mathcal{F}_{0, \\infty}} \\ & \\ \\frac{d}{dt}  \\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}(X_t, \\bar{X}_t, t)\\\\\n      \\text{subject to }\n    \\ \\dot{X}_t &= -h_t\\nabla f(\\bar{X}_t), \\\n        \\dot{\\bar{X}}_t = \\frac{X_t - \\bar{X}_t}{t}dt.\n\\end{aligned}\n\\end{equation*}\nThis maximization problem can be formulated as the following SDP program\n\n\\begin{equation*}\n    \\begin{aligned}\n    & \\min_{\\lambda_t^{(i)} \\geqslant 0, \\ i \\in \\{1, ..., 6 \\}} \\ 0 \\, \\\\\n&\\begin{pmatrix} \\dot{p}^{(11)}_t + \\frac{2p^{(12)}_t}{t} & \\dot{p}^{(12)}_t + \\frac{p^{(22)}_t - p^{(12)}_t}{t}& \\frac{\\lambda^{(6)}_t + \\lambda^{(4)}_t - 2h_tp^{(11)}_t}{2} & \\frac{a^{(2)}_t}{2t} - \\frac{\\lambda^{(5)}_t}{2}\\\\ \n\n\\dot{p}^{(12)}_t + \\frac{p^{(22)}_t - p^{(12)}_t}{t} & \\dot{p}^{(22)}_t - \\frac{2p^{(22)}_t}{t} & -\\frac{\\lambda^{(6)}_t - 2h_tp^{(12)}_t}{2} & -\\frac{a^{(2)}_t}{2t} + \\frac{\\lambda^{(5)}_t + \\lambda^{(3)}_t}{2}\\\\ \n\n \\frac{\\lambda^{(6)}_t + \\lambda^{(4)}_t - 2h_tp^{(11)}_t}{2} & -\\frac{\\lambda^{(6)}_t - 2h_tp^{(12)}_t}{2} & -a^{(1)}_t & 0 \\\\ \n\n\\frac{a^{(2)}_t}{2t} - \\frac{\\lambda^{(5)}_t}{2} & -\\frac{a^{(2)}_t}{2t} + \\frac{\\lambda^{(5)}_t + \\lambda^{(3)}_t}{2} & 0 &  0\\end{pmatrix} \\preccurlyeq 0, \\\\ \n&\\dot{a}^{(1)}_t + \\lambda^{(1)}_t + \\lambda^{(5)}_t = \\lambda^{(4)}_t + \\lambda^{(6)}_t, \\\\ \n&\\dot{a}^{(2)}_t + \\lambda^{(2)}_t + \\lambda^{(6)}_t = \\lambda^{(3)}_t + \\lambda^{(5)}_t + \\dot{a}^{(1)}_t. \n\\end{aligned}\n\\end{equation*}\nNote that the Gram matrix $G$ and function value vector $F$ to be introduced are different from the setting explored in Section~\\ref{sec:deterministic}, more precisely $G = Q^\\top Q \\succcurlyeq 0, \\ Q = [X_t - x_\\star, \\bar{X}_t - x_\\star, g_t, \\bar{g}_t]$ and $F = (f(X_t), f(\\bar{X}_t), f_\\star)$.\n\nThe final inequality follows from Ito's formula.\n\\end{proof}","start":111630,"end":116544}]},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:optim"],"declared_external":false,"source_only":false,"title":"全空间优化问题","source_units":[{"raw_tex":"Convex optimization is an important tool in the numerical analyst toolbox. It serves, among others, for framing modeling problems in data science and signal processing. We consider optimization problems of the form:\n\\begin{equation}\n    \\min_{x \\in \\mathbf{R}^d} f(x),\n    \\label{eq:optim}\n\\end{equation}\n","start":7680,"end":7985}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"全篇凸性约定","source_units":[{"raw_tex":"\\subsection{Assumptions}\n\\label{sec:assumptions}\nThroughout this work, functions to be minimized are convex (see Problem~\\eqref{eq:optim}). Under this assumption, stationary points are global minimizers. We restrict ourselves to continuous-time versions of gradient descent, accelerated gradient descent and stochastic gradient descent.\n\\smallbreak\n","start":20075,"end":20424}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"凸性与支撑超平面","source_units":[{"raw_tex":"Let us recall a few basic definitions and properties characterizing the classes of functions under consideration within the next sections. A function $f : \\mathbf{R}^d \\rightarrow \\mathbf{R}$ is convex if for all $x,y \\in \\mathbf{R}^d$, and for all $\\lambda \\in [0,1]$, $f(\\lambda x + (1-\\lambda)y) \\leqslant \\lambda f(x) + (1-\\lambda)f(y)$. Such functions (i.e. with full domain, $\\text{dom}(f) = \\mathbf{R}^d$) are convex closed proper (CCP) (i.e., their epigraphs are non-empty closed convex sets). For simplicity, we assume in addition differentiability of $f$.  Such a differentiable function $f$ is convex if and only if for all $x,y \\in \\mathbf{R}^d$, $f(y) \\geqslant f(x) + \\langle \\nabla f(x), y-x \\rangle$. A differentiable function $f$ is $L$-smooth if its gradient is $L$-Lipschitz, that is if for any $x,y  \\ \\in \\mathbf{R}^d$:\n","start":20424,"end":21265}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"梯度光滑性","source_units":[{"raw_tex":"Let us recall a few basic definitions and properties characterizing the classes of functions under consideration within the next sections. A function $f : \\mathbf{R}^d \\rightarrow \\mathbf{R}$ is convex if for all $x,y \\in \\mathbf{R}^d$, and for all $\\lambda \\in [0,1]$, $f(\\lambda x + (1-\\lambda)y) \\leqslant \\lambda f(x) + (1-\\lambda)f(y)$. Such functions (i.e. with full domain, $\\text{dom}(f) = \\mathbf{R}^d$) are convex closed proper (CCP) (i.e., their epigraphs are non-empty closed convex sets). For simplicity, we assume in addition differentiability of $f$.  Such a differentiable function $f$ is convex if and only if for all $x,y \\in \\mathbf{R}^d$, $f(y) \\geqslant f(x) + \\langle \\nabla f(x), y-x \\rangle$. A differentiable function $f$ is $L$-smooth if its gradient is $L$-Lipschitz, that is if for any $x,y  \\ \\in \\mathbf{R}^d$:\n\\begin{equation*}\n        \\|\\nabla f(x) - \\nabla f(y)\\| \\leqslant L \\|x-y\\|.\n\\end{equation*}\n","start":20424,"end":21358}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"强凸性背景约定","source_units":[{"raw_tex":"Smoothness is a common assumption for analyzing optimization methods, that limits the growth rate of the function. A convex differentiable function $f$ is $\\mu$-strongly convex if for any $x, y \\in \\mathbf{R}^d$ it satisfies:\n\\begin{equation*}\n         \\|\\nabla f(x) - \\nabla f(y)\\| \\geqslant \\mu \\|x-y\\|.\n\\end{equation*}\n","start":21358,"end":21680}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"函数类记号","source_units":[{"raw_tex":"Strong convexity ensures the function is not too flat, and the unicity of the minimizer~$x_\\star$. We denote by $\\mathcal{F}_{\\mu,L}$ the family of $L$-smooth $\\mu$-strongly convex functions from $\\mathbf{R}^d$ to $\\mathbf{R}$, with $0 \\leqslant \\mu \\leqslant L \\leqslant +\\infty$. Weaker assumptions than strong convexity are also encountered in the literature for analyzing gradient algorithms, and leads to similar convergence guarantees. Among them, Bolte~et~al.~\\cite[Appendix 5.2]{bolte2009} proved linear convergence of gradient descent under Łojasiewicz inequality, first introduced by Łojasiewicz~\\cite{Lojasiewicz1963}. Other relaxed versions of strong convexity followed~\\cite{necoara2019linear}.\n","start":21680,"end":22388}]},{"id":"S007","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"其他增长条件背景","source_units":[{"raw_tex":"Strong convexity ensures the function is not too flat, and the unicity of the minimizer~$x_\\star$. We denote by $\\mathcal{F}_{\\mu,L}$ the family of $L$-smooth $\\mu$-strongly convex functions from $\\mathbf{R}^d$ to $\\mathbf{R}$, with $0 \\leqslant \\mu \\leqslant L \\leqslant +\\infty$. Weaker assumptions than strong convexity are also encountered in the literature for analyzing gradient algorithms, and leads to similar convergence guarantees. Among them, Bolte~et~al.~\\cite[Appendix 5.2]{bolte2009} proved linear convergence of gradient descent under Łojasiewicz inequality, first introduced by Łojasiewicz~\\cite{Lojasiewicz1963}. Other relaxed versions of strong convexity followed~\\cite{necoara2019linear}.\n","start":21680,"end":22388}]},{"id":"S008","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"极小点与最优值","source_units":[{"raw_tex":"Let $f$ be $\\mu$-strongly convex (i.e., $f\\in\\mathcal{F}_{\\mu,\\infty}$) with $\\mu>0$, and $x_\\star$ be its unique minimizer such that $f(x_\\star) = f_\\star$. In this context, it is possible to prove linear convergence of the gradient flow to its stationary point. Scieur et~al.~proved in~\\cite[Proposition 1.1]{scieur2017integration} the following convergence bound in function values for the gradient flow:\n","start":23778,"end":24186}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","labels":["reflyapmu"],"declared_external":false,"source_only":false,"title":"强凸梯度流速率示例","source_units":[{"raw_tex":"Let $f$ be $\\mu$-strongly convex (i.e., $f\\in\\mathcal{F}_{\\mu,\\infty}$) with $\\mu>0$, and $x_\\star$ be its unique minimizer such that $f(x_\\star) = f_\\star$. In this context, it is possible to prove linear convergence of the gradient flow to its stationary point. Scieur et~al.~proved in~\\cite[Proposition 1.1]{scieur2017integration} the following convergence bound in function values for the gradient flow:\n\\begin{equation}\n\\label{reflyapmu}\nf(X_t) - f_\\star \\leqslant e^{- 2 \\mu t}(f(x_0) - f_\\star).\n\\end{equation}\n","start":23778,"end":24296}]},{"id":"S010","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"强凸示例的作者推导","source_units":[{"raw_tex":"This convergence guarantee follows directly from the derivative with respect to time of the Lyapunov function $\\mathcal{V}(X_t) = f(X_t) - f_\\star$, together with strong convexity (or Łojasiewicz inequality): $\\frac{d}{dt}\\mathcal{V}(X_t) = \\dot{X_t}^\\top \\nabla f(X_t) = -\\|\\nabla f(X_t)\\|^2 \\leqslant -2 \\mu (f(X_t) - f_\\star) = -2 \\mu \\mathcal{V}(X_t)$.\n","start":24296,"end":24653}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":["lyapunov"],"declared_external":false,"source_only":false,"title":"确定性二次势函数","source_units":[{"raw_tex":"\n\\bigbreak\nGiven the specific gradient flows studied in this work, it is reasonable to search for Lyapunov functions made of linear combinations of function values, and a quadratic form in the trajectory $X_t$. We simply refer to them as quadratic Lyapunov functions:\n\\begin{equation}\n\\label{lyapunov}\n    \\mathcal{V}_{a,c}(X_t) = a \\cdot (f(X_t) - f_\\star) + c \\cdot \\|X_t - x_\\star\\|^2,\n\\end{equation}\nwhere $a, c \\geqslant 0$ are constants that do not depend on $t$, such that the function $ \\mathcal{V}_{a,c}$ is nonnegative and nonincreasing along the flow. Quadratic Lyapunov functions are common for proving convergence of gradient flows and cover for instance the Lyapunov function used to prove convergence of the gradient flow under strong convexity~\\eqref{reflyapmu}. Given a Lyapunov function $\\mathcal{V}_{a,c}$, the idea is to find the largest nonnegative value $\\tau_\\star$ such that the condition\n","start":24653,"end":25566}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["wc_mu"],"declared_external":false,"source_only":false,"title":"统一衰减验证目标","source_units":[{"raw_tex":"\\begin{equation}\n\\label{wc_mu}\n    \\frac{d}{dt}\\mathcal{V}_{a,c}(X_t) \\leqslant -\\tau_\\star \\mathcal{V}_{a,c}(X_t),\n\\end{equation}\nholds for any dimension $d \\in \\mathbf{N}$, any function $f \\in \\mathcal{F}_{\\mu, \\infty}$ and any trajectory $X_t$ generated by the gradient flow. After integration, \\eqref{wc_mu} allows to obtain a convergence guarantee of the form $\\mathcal{V}_{a,c}(X_t) \\leqslant e^{-\\tau_\\star t} \\mathcal{V}_{a,c}(X_0)$. Given a certain Lyapunov function $\\mathcal{V}_{a,c}$ and a time $t$, we get that the largest acceptable $\\tau_\\star$ is a solution to:\n","start":25566,"end":26144}]},{"id":"S013","semantic_type":"algorithm","kind":"algorithm","labels":["eq:str_pep"],"declared_external":false,"source_only":false,"title":"性能估计极值表述","source_units":[{"raw_tex":"\\begin{equation}\n\\label{eq:str_pep}\n    \\begin{aligned}\n      -\\tau_\\star = \\max_{X_t \\in \\mathbf{R}^d, d \\in \\mathbf{N}, f\\in \\mathcal{F}_{\\mu, \\infty}} \\ & \\  \\frac{d}{dt}\\mathcal{V}_{a,c}(X_t), \\\\\n      \\text{subject to }\n    \\mathcal{V}_{a,c}(X_t) &= 1, \\\\\n    \\dot{X_t} &= - \\nabla f (X_t).\n\\end{aligned}\n\\end{equation}\nThis minimization problem is invariant with respect to $t$. \n","start":26144,"end":26530}]},{"id":"S014","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"采样函数值与梯度","source_units":[{"raw_tex":"It is established in~\\cite{taylor2016smooth, drori2012performance} that these so-called performance estimation problems (PEP) can be formulated as SDPs. Because of the variable $f \\in \\mathcal{F}_{\\mu, \\infty}$, the maximization problem~\\eqref{eq:str_pep} is infinite-dimensional. Recall that a differentiable function $f \\in \\mathcal{F}_{\\mu, \\infty}$ verifies for all points $x,y \\in \\mathbf{R}^d$, $f(x) - f(y) - \\langle \\nabla f(y), x-y \\rangle \\geqslant \\frac{\\mu}{2} \\|x-y\\|^2$. Using alternate variables $f_t$, $f_\\star$, $g_t$ and $g_\\star$ (informally: $f_t = f(X_t)$, $f_\\star = f(x_\\star)$, $ g_t = \\nabla f(X_t)$ and $g_\\star = \\nabla f(x_\\star) = 0$), it holds that:\n","start":26530,"end":27210}]},{"id":"S015","semantic_type":"algorithm","kind":"algorithm","labels":["eq:interpolation_step"],"declared_external":false,"source_only":false,"title":"有限采样不等式替换","source_units":[{"raw_tex":"\\begin{equation}\n\\label{eq:interpolation_step}\n    \\begin{aligned}\n      -\\tau_\\star = \\max_{X_t \\in \\mathbf{R}^d, d \\in \\mathbf{N}} \\ & \\ \\frac{d}{dt}\\mathcal{V}_{a, c}(X_t), \\\\\n      \\text{subject to } &\\mathcal{V}_{a,c}(X_t) = 1, \\\\\n    &\\dot{X}_t = - g_t, \\\\\n    & f_i - f_j -\\langle g_j, X_i-X_j\\rangle \\geqslant \\frac{\\mu}{2}\\|X_i-X_j\\|^2, \\ \\forall i,j=t, \\star.\n\\end{aligned}\n\\end{equation}\n","start":27210,"end":27609}]},{"id":"S016","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"采样松弛给出上界","source_units":[{"raw_tex":"The fact that~\\eqref{eq:interpolation_step} produces an upper bound on $-\\tau_\\star$ directly follows from the fact that any sampled strongly convex function satisfy these inequalities at the sampled points (here $X_t$ and $x_\\star$). Thereby, any feasible point to~\\eqref{eq:str_pep} corresponds to a feasible point for~\\eqref{eq:interpolation_step} with the same objective value. In the other direction,~\\cite[Corollary 2]{taylor2016smooth} (which provides a constructive way to obtain some $f \\in \\mathcal{F}_{\\mu, \\infty}$ that interpolates the triplets $(X_i, g_i, f_i)_{i=t, \\star}$) ensures that any feasible point to~\\eqref{eq:interpolation_step} can be translated to a feasible point to~\\eqref{eq:str_pep} with also the same objective value, thereby reaching the equivalence between formulations~\\eqref{eq:str_pep} and~\\eqref{eq:interpolation_step}.\n","start":27609,"end":28468}]},{"id":"S017","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"插值逆向实现与精确性","source_units":[{"raw_tex":"The fact that~\\eqref{eq:interpolation_step} produces an upper bound on $-\\tau_\\star$ directly follows from the fact that any sampled strongly convex function satisfy these inequalities at the sampled points (here $X_t$ and $x_\\star$). Thereby, any feasible point to~\\eqref{eq:str_pep} corresponds to a feasible point for~\\eqref{eq:interpolation_step} with the same objective value. In the other direction,~\\cite[Corollary 2]{taylor2016smooth} (which provides a constructive way to obtain some $f \\in \\mathcal{F}_{\\mu, \\infty}$ that interpolates the triplets $(X_i, g_i, f_i)_{i=t, \\star}$) ensures that any feasible point to~\\eqref{eq:interpolation_step} can be translated to a feasible point to~\\eqref{eq:str_pep} with also the same objective value, thereby reaching the equivalence between formulations~\\eqref{eq:str_pep} and~\\eqref{eq:interpolation_step}.\n","start":27609,"end":28468}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":[],"declared_external":false,"source_only":false,"title":"二向量 Gram 编码","source_units":[{"raw_tex":"\\smallbreak\nIn a second stage, we introduce $G  = \\begin{pmatrix} \\|X_t - x_\\star\\|^2 &  \\langle X_t - x_\\star, g_t \\rangle  \\\\  \\langle X_t-x_\\star, g_t \\rangle & \\| g_t \\|^2 \\end{pmatrix} \\succcurlyeq 0$, a Gram matrix and a vector $F = [f_t, f_\\star]$, thereby obtaining a semidefinite reformulation:\n","start":28468,"end":28772}]},{"id":"S019","semantic_type":"algorithm","kind":"algorithm","labels":["SDP"],"declared_external":false,"source_only":false,"title":"性能估计的 SDP 编码","source_units":[{"raw_tex":"\\begin{equation}\n\\label{SDP}\n    \\begin{aligned}\n      -\\tau_\\star = \\max_{G\\succcurlyeq 0, F \\in \\mathbf{R}^2} & b_0^\\top F + {\\rm Tr}(A_0G), \\\\\n      \\text{subject to }\n    & b_1^\\top F + {\\rm Tr}(A_1G) \\geqslant 0, \\\\\n    & b_2^\\top F + {\\rm Tr}(A_2G) \\geqslant 0, \\\\\n    & b_3^\\top F + {\\rm Tr}(A_3G) = 1. \\\\\n\\end{aligned}\n\\end{equation}\nwhere $A_0 = \\begin{pmatrix} 0 & -c \\\\ -c & -a \\end{pmatrix}$, $A_1 = \\begin{pmatrix}-\\mu/2 & 1/2 \\\\ 1/2 & 0 \\end{pmatrix}$, $A_2= \\begin{pmatrix}-\\mu/2 & 0 \\\\ 0 & 0\\end{pmatrix}$, $A_3 = \\begin{pmatrix}c & 0 \\\\ 0 & 0 \\end{pmatrix}$, $b_0 = [0, 0]^\\top$, $b_1 = [-1,\\ 1]^\\top $, $b_2=[1, \\ -1]^\\top $ and $b_3 = a[1, \\ -1]^\\top $. Consider the corresponding Lagrangian, for $F \\in \\mathbf{R}^2$, $G \\succcurlyeq 0$, $\\tau \\in \\mathbf{R}$ and $\\lambda_1, \\lambda_2 \\geqslant 0$: \n","start":28772,"end":29593}]},{"id":"S020","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"非负乘子的拉格朗日函数","source_units":[{"raw_tex":"where $A_0 = \\begin{pmatrix} 0 & -c \\\\ -c & -a \\end{pmatrix}$, $A_1 = \\begin{pmatrix}-\\mu/2 & 1/2 \\\\ 1/2 & 0 \\end{pmatrix}$, $A_2= \\begin{pmatrix}-\\mu/2 & 0 \\\\ 0 & 0\\end{pmatrix}$, $A_3 = \\begin{pmatrix}c & 0 \\\\ 0 & 0 \\end{pmatrix}$, $b_0 = [0, 0]^\\top$, $b_1 = [-1,\\ 1]^\\top $, $b_2=[1, \\ -1]^\\top $ and $b_3 = a[1, \\ -1]^\\top $. Consider the corresponding Lagrangian, for $F \\in \\mathbf{R}^2$, $G \\succcurlyeq 0$, $\\tau \\in \\mathbf{R}$ and $\\lambda_1, \\lambda_2 \\geqslant 0$: \n\\begin{equation*}\n    \\begin{aligned}\n    \\mathcal{L}(F, G, \\tau, \\lambda_1, \\lambda_2) =& b_0^\\top F + {\\rm Tr}(A_0G) + \\tau \\cdot (b_3^\\top F + {\\rm Tr}(A_3G) - 1) \\\\ \n    & + \\lambda_1 \\cdot(b_1^\\top F + {\\rm Tr}(A_1G)) + \\lambda_2 \\cdot(b_2^\\top F + {\\rm Tr}(A_2G)).\n    \\end{aligned}\n\\end{equation*}\n","start":29114,"end":29898}]},{"id":"S021","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"方法示例的鞍点表述","source_units":[{"raw_tex":"The saddle point of the Lagrangian is given by:\n\\begin{equation*}\n    \\begin{aligned}\n    \\tau_\\star &= \\min_{\\tau, \\lambda_1 \\geqslant 0, \\lambda_2 \\geqslant 0} \\max_{F, G \\succcurlyeq 0} \\mathcal{L}(F, G, \\tau, \\lambda_1, \\lambda_2).\n\\end{aligned}\n\\end{equation*}\n","start":29898,"end":30164}]},{"id":"S022","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"对偶矩阵与系数平衡","source_units":[{"raw_tex":"The Lagrangian dual of the SDP~\\eqref{SDP} is obtained by maximizing over $F \\in \\mathbf{R}^2$, $G \\succcurlyeq 0$:\n\\begin{equation*}\n    \\begin{aligned}\n    -\\tau_\\star =  \\min_{\\tau, \\lambda_1, \\lambda_2} & \\ -\\tau, \\\\\n      \\text{subject to } & S = A_0 + \\lambda_1 A_1 + \\lambda_2 A_2 + \\tau A_3 \\preccurlyeq 0 \\\\ \n    &  b_0 +  \\lambda_1 b_1 +\\lambda_2 b_2  + \\tau b_3 = 0, \\\\\n    &  \\tau\\in \\mathbf{R}, \\ \\lambda_1, \\lambda_2 \\geqslant 0.\n\\end{aligned}\n\\end{equation*}\n","start":30164,"end":30638}]},{"id":"S023","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"强对偶与方法精确性","source_units":[{"raw_tex":"The equality with $\\tau_\\star$ comes from strong duality, that holds via Slater's conditions (it is relatively easy to show that there exists a Slater's point using the same construction as in~\\cite[Theorem 6]{taylor2016smooth}). Finally, since any feasible $\\tau$ is a lower bound for $\\tau_\\star$, those developments allow arriving to the equivalence between verifying a quadratic Lyapunov function and verifying the feasibility of an LMI.\n","start":30638,"end":31080}]},{"id":"S024","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"declared_external":false,"source_only":false,"title":"可行对偶证书的充分性","source_units":[{"raw_tex":"The equality with $\\tau_\\star$ comes from strong duality, that holds via Slater's conditions (it is relatively easy to show that there exists a Slater's point using the same construction as in~\\cite[Theorem 6]{taylor2016smooth}). Finally, since any feasible $\\tau$ is a lower bound for $\\tau_\\star$, those developments allow arriving to the equivalence between verifying a quadratic Lyapunov function and verifying the feasibility of an LMI.\n","start":30638,"end":31080}]},{"id":"S025","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第三项：taylor2016smooth, Corollary 2；原文第397行明确引用。"],"source_only":false,"title":"强凸数据插值定理","source_units":[]},{"id":"S026","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第三项：Slater's condition for strong duality；原文第432行援引。"],"source_only":false,"title":"Slater 强对偶准则","source_units":[]},{"id":"S027","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第二项：positive semidefinite Gram representations。"],"source_only":false,"title":"半正定 Gram 表示","source_units":[]},{"id":"S028","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第二项：SDP weak duality。"],"source_only":false,"title":"半定规划弱对偶","source_units":[]},{"id":"S029","semantic_type":"definition","kind":"definition","labels":["eq:sme"],"declared_external":false,"source_only":false,"title":"基本扩散模型","source_units":[{"raw_tex":"The corresponding SDE is given by:\n\\begin{equation}\n\\label{eq:sme}\n    dX_t = -\\nabla f(X_t)dt + (\\gamma \\Sigma(X_t))^{1/2}dB_t,\n\\end{equation}\n","start":69903,"end":70047}]},{"id":"S030","semantic_type":"notation","kind":"notation","labels":[],"declared_external":false,"source_only":false,"title":"标准布朗运动记号","source_units":[{"raw_tex":"where $B_t$ is a standard Brownian motion. The SDE~\\eqref{eq:sme} is an (order-$1$ weak) approximation of  SGD~(\\cite[Theorem 1]{li2017stochastic},~\\cite{Li2019}), which allows to take into account the role of constant step size in the dynamics of SGD (while keeping them small). Under mild assumptions on $f$, Li~et~al.~\\cite{Li2019} proved the weak approximation of SGD by this SDE on a finite interval $[0, T]$: there exists $C>0$ such that $\\|\\mathbf{E}[x_k] - \\mathbf{E}[X(k\\gamma)] \\| \\leqslant C\\gamma$ for $k \\in [0, \\frac{T}{\\gamma}]$. However, the approximation point of view from this approach is relatively limited since $C$ depends exponentially on~$T$. \n","start":70047,"end":70715}]},{"id":"S031","semantic_type":"interpretation","kind":"interpretation","labels":[],"declared_external":false,"source_only":false,"title":"SGD 弱近似背景","source_units":[{"raw_tex":"where $B_t$ is a standard Brownian motion. The SDE~\\eqref{eq:sme} is an (order-$1$ weak) approximation of  SGD~(\\cite[Theorem 1]{li2017stochastic},~\\cite{Li2019}), which allows to take into account the role of constant step size in the dynamics of SGD (while keeping them small). Under mild assumptions on $f$, Li~et~al.~\\cite{Li2019} proved the weak approximation of SGD by this SDE on a finite interval $[0, T]$: there exists $C>0$ such that $\\|\\mathbf{E}[x_k] - \\mathbf{E}[X(k\\gamma)] \\| \\leqslant C\\gamma$ for $k \\in [0, \\frac{T}{\\gamma}]$. However, the approximation point of view from this approach is relatively limited since $C$ depends exponentially on~$T$. \n","start":70047,"end":70715}]},{"id":"S032","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"变化步长的建模背景","source_units":[{"raw_tex":"Compared to gradient descent, SGD does not converge to a stationary point under constant step sizes~\\cite{bach2011, taylor2021stochastic}. Convergence to a stationary point requires diminishing step sizes such as $\\gamma_k = \\frac{1}{\\sqrt{k}}$. Li~et~al.~\\cite[Section 4.1]{li2017stochastic} (and later Orvieto~and~Lucchi~\\cite[Section 2.1]{orvieto2020continuoustime}) proposed to include this varying learning rate in the dynamics:\n\\begin{equation*}\n    x_{k+1} = x_k - \\gamma h_k \\nabla f(x_k),\n\\end{equation*}\nwhere $\\gamma$ is the maximum allowed learning rate and $h_k \\in [0, 1]$ is the time-varying part. For $h_t \\geqslant 0$ a continuous function playing the role of step sizes $h_k$, the SDE is:\n","start":71221,"end":71928}]},{"id":"S033","semantic_type":"definition","kind":"definition","labels":["sme_general"],"declared_external":false,"source_only":false,"title":"时变步长随机梯度流","source_units":[{"raw_tex":"where $\\gamma$ is the maximum allowed learning rate and $h_k \\in [0, 1]$ is the time-varying part. For $h_t \\geqslant 0$ a continuous function playing the role of step sizes $h_k$, the SDE is:\n\\begin{equation}\n\\label{sme_general}\n    dX_t = - h_t \\nabla f(X_t)dt + h_t(\\gamma \\Sigma(X_t))^{1/2}dB_t.\n\\end{equation}\n","start":71735,"end":72050}]},{"id":"S034","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"协方差与方差界","source_units":[{"raw_tex":"We treat the covariance matrix $\\Sigma(X_t)$ as symmetric, already implied by the notation $\\Sigma(X_t)^{1/2}$, but unstructured with bounded variance $\\Sigma(X_t) \\preccurlyeq \\Sigma$ along any trajectory $X_t$ generated by the approximating SDE~\\eqref{sme_general}. Compared to ODEs, functions $f \\in \\mathcal{F}_{0, L}$ to be optimized using SDEs are in addition assumed to be possibly $L$-smooth with $L\\in (0, \\infty]$ and to be twice continuously differentiable.\n","start":72050,"end":72519}]},{"id":"S035","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"随机分析的函数正则性","source_units":[{"raw_tex":"We treat the covariance matrix $\\Sigma(X_t)$ as symmetric, already implied by the notation $\\Sigma(X_t)^{1/2}$, but unstructured with bounded variance $\\Sigma(X_t) \\preccurlyeq \\Sigma$ along any trajectory $X_t$ generated by the approximating SDE~\\eqref{sme_general}. Compared to ODEs, functions $f \\in \\mathcal{F}_{0, L}$ to be optimized using SDEs are in addition assumed to be possibly $L$-smooth with $L\\in (0, \\infty]$ and to be twice continuously differentiable.\n","start":72050,"end":72519}]},{"id":"S036","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["原始 external_prerequisites 第一项及 declared_permitted_prerequisite_ids B0088：标准多维含时 Itô 公式，原文第881行引用 sarkka_solin_2019, Theorem 4.2。"],"source_only":false,"title":"允许使用的 Itô 公式","source_units":[{"raw_tex":"\\smallbreak\nIn SDE theory, a differential with respect to time of a function of a solution to a stochastic process is given by Ito's Lemma~\\cite[Theorem 4.2]{sarkka_solin_2019}.\n\\begin{lemma}[Ito's Lemma]\nLet $g$ be a twice continuously differentiable function, and $X_t$ be a stochastic process solution to the SDE~\\eqref{eq:sme}, then\n\\begin{equation*}\n    dg(X_t,t) = \\frac{\\partial}{\\partial t}g(X_t, t)dt + \\frac{\\partial}{\\partial x}g(X_t, t) dX_t + \\frac{1}{2}\\gamma {\\rm Tr}(\\frac{\\partial^2}{\\partial x^2}g(X_t, t)\\Sigma(X_t))dt.\n\\end{equation*}\n\\end{lemma}\n","start":73421,"end":73988}]},{"id":"S037","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"declared_external":false,"source_only":false,"title":"平均模型的研究背景","source_units":[{"raw_tex":"\\subsubsection{Polyak-Ruppert averaging}\nIn this section, we analyze convergence properties of an SDE~\\eqref{sme_general} approximating SGD with time-varying step sizes under Polyak-Ruppert averaging. Taylor~and~Bach~\\cite[Theorem 6]{taylor2021stochastic} provided a systematic design of Lyapunov functions for~\\eqref{eq:pr_avg}, and a condition on the step size for convergence to the optimum. An approximating SDE for Polyak-Ruppert averaging is given by:\n","start":83501,"end":83959}]},{"id":"S038","semantic_type":"definition","kind":"definition","labels":["ODE_mean1"],"declared_external":false,"source_only":false,"title":"Polyak–Ruppert 平均 SDE","source_units":[{"raw_tex":"\\begin{equation}\n\\label{ODE_mean1}\n    \\begin{aligned}\n        dX_t &= -h_t\\nabla f(X_t)dt + h_t(\\gamma \\Sigma(X_t))^{1/2} dB_t, \\\\\n        d\\bar{X}_t &= \\frac{X_t - \\bar{X}_t}{t}dt,\n        \\end{aligned}\n    \\end{equation}\n","start":83959,"end":84183}]},{"id":"S039","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"平均模型的步长范围","source_units":[{"raw_tex":"with step size $\\gamma>0$ that is taken close to zero, and a variable term $h_t\\in [0, 1]$. We introduce the family of quadratic functions taking the averaged sequence into account:\n","start":84183,"end":84365}]},{"id":"S040","semantic_type":"definition","kind":"definition","labels":["lyap_avg"],"declared_external":false,"source_only":false,"title":"含平均点的二次势函数","source_units":[{"raw_tex":"\\begin{equation}\n\\label{lyap_avg}\n    \\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}(X_t, \\bar{X}_t, t) = \\begin{pmatrix}a^{(1)}_t \\\\ a^{(2)}_t \\end{pmatrix}^\\top \\!\\!\\!\\begin{pmatrix} f(X_t) - f_\\star \\\\ f(\\bar{X_t}) - f_\\star \\end{pmatrix} + \\begin{pmatrix} X_t - X_\\star\\\\ \\bar{X_t} - X_\\star\\end{pmatrix}^\\top \\!\\!\\! \\left(P_t\\otimes I_d\\right) \\! \\begin{pmatrix} X_t - X_\\star\\\\ \\bar{X_t} - X_\\star\\end{pmatrix},\n\\end{equation}\n","start":84365,"end":84791}]},{"id":"S041","semantic_type":"assumption","kind":"assumption","labels":[],"declared_external":false,"source_only":false,"title":"势函数系数正则性","source_units":[{"raw_tex":"where $a^{(1)}_t, a^{(2)}_t \\geqslant 0$ and $P_t = \\begin{pmatrix}p^{(11)}_t & p^{(12)}_t \\\\ p^{(12)}_t & p^{(22)}_t \\end{pmatrix}\\succcurlyeq 0$ are continuously differentiable with respect to time. Given a quadratic function $ \\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}$ and an SDE~\\eqref{ODE_mean1} under Polyak-Ruppert averaging, we present in Theorem~\\ref{pr_avg_conv} a  way to control the quantity:\n","start":84791,"end":85195}]},{"id":"S042","semantic_type":"algorithm","kind":"algorithm","labels":[],"declared_external":false,"source_only":false,"title":"平均势函数的最坏漂移问题","source_units":[{"raw_tex":"\\begin{equation*}\n    \\begin{aligned}\n      \\max_{f \\in \\mathcal{F}_{0, \\infty}, d \\in \\mathbf{N}, X_t \\in \\mathbf{R}^d} \\ & \\ \\frac{d}{dt}\\mathbf{E}\\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}(X_t, \\bar{X}_t, t), \\\\\n       \\text{subject to} \\ & dX_t = -h_t\\nabla f(X_t)dt + h_t(\\gamma \\Sigma(X_t))^{1/2} dB_t, \\\\\n        & d\\bar{X}_t = \\frac{X_t - \\bar{X}_t}{t}dt.\n    \\end{aligned}\n\\end{equation*}\n","start":85195,"end":85590}]},{"id":"S043","semantic_type":"theorem","kind":"theorem","labels":["pr_avg_conv"],"declared_external":false,"source_only":false,"title":"平均随机流的充分 LMI 证书","source_units":[{"raw_tex":"\\begin{theorem}\n\\label{pr_avg_conv}\nLet $h_t \\geqslant 0$, and $a_t \\geqslant 0$, $P_t \\succcurlyeq 0$ be continuously differentiable with respect to time. Let  $f \\in \\mathcal{F}_{0, \\infty}$ be a twice continuously differentiable function and $(X_t, \\bar{X}_t) \\in \\mathbf{R}^d \\times \\mathbf{R}^d$ be a trajectory generated by the SDE under Polyak-Ruppert averaging~\\eqref{ODE_mean1}. Let $\\mathcal{V}$ be a quadratic function~\\eqref{lyap_avg}, such that there exist $\\lambda^{(1)}_t, ..., \\lambda^{(6)}_t \\geqslant 0$ verifying:\n\\begin{equation*}\n    \\begin{aligned}\n&\\begin{pmatrix} \\dot{p}^{(11)}_t + \\frac{2p^{(12)}_t}{t} & \\dot{p}^{(12)}_t + \\frac{p^{(22)}_t - p^{(12)}_t}{t}& \\frac{\\lambda^{(6)}_t + \\lambda^{(4)}_t - 2h_tp^{(11)}_t}{2} & \\frac{a^{(2)}_t}{2t} - \\frac{\\lambda^{(5)}_t}{2}\\\\ \n\n\\dot{p}^{(12)}_t + \\frac{p^{(22)}_t - p^{(12)}_t}{t} & \\dot{p}^{(22)}_t - \\frac{2p^{(22)}_t}{t} & -\\frac{\\lambda^{(6)}_t - 2h_tp^{(12)}_t}{2} & -\\frac{a^{(2)}_t}{2t} + \\frac{\\lambda^{(5)}_t + \\lambda^{(3)}_t}{2}\\\\ \n\n \\frac{\\lambda^{(6)}_t + \\lambda^{(4)}_t - 2h_tp^{(11)}_t}{2} & -\\frac{\\lambda^{(6)}_t - 2h_tp^{(12)}_t}{2} & -a^{(1)}_t & 0 \\\\ \n\n\\frac{a^{(2)}_t}{2t} - \\frac{\\lambda^{(5)}_t}{2} & -\\frac{a^{(2)}_t}{2t} + \\frac{\\lambda^{(5)}_t + \\lambda^{(3)}_t}{2} & 0 &  0\\end{pmatrix} \\preccurlyeq 0, \\\\ \n&\\dot{a}^{(1)}_t + \\lambda^{(1)}_t + \\lambda^{(5)}_t = \\lambda^{(4)}_t + \\lambda^{(6)}_t, \\\\ \n&\\dot{a}^{(2)}_t + \\lambda^{(2)}_t + \\lambda^{(6)}_t = \\lambda^{(3)}_t + \\lambda^{(5)}_t + \\dot{a}^{(1)}_t. \n\\end{aligned}\n\\end{equation*}\nThen, the following inequality is satisfied:\n\\smallbreak \n$\\displaystyle\\frac{d}{dt}\\mathbf{E}\\mathcal{V}(X_t, \\bar{X}_t, t)\\leqslant \\frac{1}{2}\\mathbf{E}{\\rm Tr}\\left((a^{(1)}_t\\nabla_{xx} f(X_t) + 2p^{(11)}_t I_d)\\Sigma(X_t)\\right)h_t^2\\gamma$.\n\\end{theorem}","start":85591,"end":87393}]},{"id":"S044","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"正文证明指引","source_units":[{"raw_tex":"\\begin{proof}\nThe proof follows the method from Section~\\ref{sec:methodo} (see Appendix~\\ref{sec:A11}).\n\\end{proof}","start":87394,"end":87509}]},{"id":"S045","semantic_type":"proof","kind":"proof","labels":[],"declared_external":false,"source_only":false,"title":"附录完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\nWe rewrite the SDE into \n\\begin{equation*}\n\\begin{aligned}\n    \\begin{pmatrix} dX_t \\\\ d\\bar{X}_t\\end{pmatrix} =& \\left(\\begin{pmatrix} 0 & 0 \\\\ \\frac{1}{t} & -\\frac{1}{t} \\end{pmatrix}\\otimes I_d\\right)\\begin{pmatrix} X_t \\\\ \\bar{X}_t\\end{pmatrix}dt +\\left(\\begin{pmatrix}- h_t & 0 \\\\ 0  & 0  \\end{pmatrix}\\otimes I_d\\right)\\begin{pmatrix} \\nabla f(X_t) \\\\ \\nabla f(\\bar{X}_t) \\end{pmatrix}dt \\\\\n    &+\\left(\\begin{pmatrix}h_t (\\gamma\\Sigma(X_t))^{1/2} \\\\ 0\\end{pmatrix}\\otimes I_d\\right)dB_t\n\\end{aligned}\n\\end{equation*}\n and denote $Y_t = \\begin{pmatrix} X_t \\\\ \\bar{X}_t \\end{pmatrix}$. \n \\smallbreak\n We consider the quadratic function~\\eqref{lyap_avg} $\\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}(X_t, \\bar{X}_t ,t) = \\mathcal{V}(Y_t,t)$. Applying Ito's formula, its derivative with respect to time is $\\frac{d}{dt} \\mathcal{V}(Y_t,t) = \\frac{\\partial}{\\partial t}\\mathcal{V}(Y_t,t) + \\frac{\\partial}{\\partial y}\\mathcal{V}\\frac{dY_t}{dt} + \\frac{1}{2}\\gamma h^2_t {\\rm Tr}[\\frac{\\partial^2}{\\partial Y_t^2}\\mathcal{V}(Y_t,t)^\\top \\begin{pmatrix}\\Sigma(X_t) \\\\ 0\\end{pmatrix}]$. By taking the expectation of Ito's formula, we have $\\frac{d}{dt} \\mathbf{E}\\mathcal{V}(Y_t,t) = \\mathbf{E}[\\frac{\\partial}{\\partial t}\\mathcal{V}(Y_t,t) + \\frac{\\partial}{\\partial y}\\mathcal{V}\\frac{dY_t}{dt}] +\\frac{1}{2}\\gamma h^2_t \\mathbf{E}{\\rm Tr}(\\frac{\\partial^2}{\\partial Y_t^2}\\mathcal{V}(Y_t,t)^\\top \\begin{pmatrix}\\Sigma(X_t) \\\\ 0\\end{pmatrix})$.\n \\smallbreak\n The variance term $\\frac{1}{2}\\gamma h^2_t \\mathbf{E}{\\rm Tr}(\\frac{\\partial^2}{\\partial Y_t^2}\\mathcal{V}(Y_t,t)^\\top \\begin{pmatrix}\\Sigma(X_t) \\\\ 0\\end{pmatrix})$ depends on the second derivative of $f$ in the space variable $(X_t)$, and on the covariance $\\Sigma_t$. We do not take this term into account in the performance estimation frawework.\n \\smallbreak\n We formulate the problem of verifying a quadratic function as verifying the inequality $\\frac{d}{dt}\\mathbf{E}\\mathcal{V}(X_t, \\bar{X}_t,t) \\leqslant \\frac{1}{2}\\gamma h^2_t \\mathbf{E}{\\rm Tr}(\\frac{\\partial^2}{\\partial X_t^2}\\mathcal{V}(X_t, \\bar{X}_t, t)^\\top \\Sigma(X_t))$ (that is $\\mathbf{E}[\\frac{\\partial}{\\partial t}\\mathcal{V}(Y_t,t) + \\frac{\\partial}{\\partial y}\\mathcal{V}\\frac{dY_t}{dt}] \\leqslant 0$) holds for any twice continuously differentiable function $f \\in \\mathcal{F}_{0, \\infty}$, and any trajectory $(X_t, \\bar{X}_t)$ generated by the SDE under Polyak-Ruppert averaging~\\eqref{ODE_mean1}. \n \\smallbreak\n This problem is equivalent with verifying that $\\frac{d}{dt}\\mathcal{V}(\\tilde{Y}_t,t) \\leqslant 0$ holds for any function twice continuously differentiable $f \\in \\mathcal{F}_{0, \\infty}$ and any trajectory $\\tilde{Y}_t = \\begin{pmatrix} \\tilde{X}_t & \\tilde{\\bar{X}}_t\\end{pmatrix}^\\top $ generated by deterministic gradient flow from the SDE~\\eqref{ODE_mean1} with $\\gamma = 0$. \n\\smallbreak\nWe follow the methodology developed in Section~\\ref{sec:deterministic} for ODEs. We formulate the verification of such a Lyapunov function as the maximization problem\n\\begin{equation*}\n    \\begin{aligned}\n      0\\geqslant \\max_{X_t \\in \\mathbf{R}^d,  d\\in \\mathbf{N}, \\ f\\in \\mathcal{F}_{0, \\infty}} \\ & \\ \\frac{d}{dt}  \\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}(X_t, \\bar{X}_t, t)\\\\\n      \\text{subject to }\n    \\ \\dot{X}_t &= -h_t\\nabla f(\\bar{X}_t), \\\n        \\dot{\\bar{X}}_t = \\frac{X_t - \\bar{X}_t}{t}dt.\n\\end{aligned}\n\\end{equation*}\nThis maximization problem can be formulated as the following SDP program\n\n\\begin{equation*}\n    \\begin{aligned}\n    & \\min_{\\lambda_t^{(i)} \\geqslant 0, \\ i \\in \\{1, ..., 6 \\}} \\ 0 \\, \\\\\n&\\begin{pmatrix} \\dot{p}^{(11)}_t + \\frac{2p^{(12)}_t}{t} & \\dot{p}^{(12)}_t + \\frac{p^{(22)}_t - p^{(12)}_t}{t}& \\frac{\\lambda^{(6)}_t + \\lambda^{(4)}_t - 2h_tp^{(11)}_t}{2} & \\frac{a^{(2)}_t}{2t} - \\frac{\\lambda^{(5)}_t}{2}\\\\ \n\n\\dot{p}^{(12)}_t + \\frac{p^{(22)}_t - p^{(12)}_t}{t} & \\dot{p}^{(22)}_t - \\frac{2p^{(22)}_t}{t} & -\\frac{\\lambda^{(6)}_t - 2h_tp^{(12)}_t}{2} & -\\frac{a^{(2)}_t}{2t} + \\frac{\\lambda^{(5)}_t + \\lambda^{(3)}_t}{2}\\\\ \n\n \\frac{\\lambda^{(6)}_t + \\lambda^{(4)}_t - 2h_tp^{(11)}_t}{2} & -\\frac{\\lambda^{(6)}_t - 2h_tp^{(12)}_t}{2} & -a^{(1)}_t & 0 \\\\ \n\n\\frac{a^{(2)}_t}{2t} - \\frac{\\lambda^{(5)}_t}{2} & -\\frac{a^{(2)}_t}{2t} + \\frac{\\lambda^{(5)}_t + \\lambda^{(3)}_t}{2} & 0 &  0\\end{pmatrix} \\preccurlyeq 0, \\\\ \n&\\dot{a}^{(1)}_t + \\lambda^{(1)}_t + \\lambda^{(5)}_t = \\lambda^{(4)}_t + \\lambda^{(6)}_t, \\\\ \n&\\dot{a}^{(2)}_t + \\lambda^{(2)}_t + \\lambda^{(6)}_t = \\lambda^{(3)}_t + \\lambda^{(5)}_t + \\dot{a}^{(1)}_t. \n\\end{aligned}\n\\end{equation*}\nNote that the Gram matrix $G$ and function value vector $F$ to be introduced are different from the setting explored in Section~\\ref{sec:deterministic}, more precisely $G = Q^\\top Q \\succcurlyeq 0, \\ Q = [X_t - x_\\star, \\bar{X}_t - x_\\star, g_t, \\bar{g}_t]$ and $F = (f(X_t), f(\\bar{X}_t), f_\\star)$.\n\nThe final inequality follows from Ito's formula.\n\\end{proof}","start":111630,"end":116544}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"增广状态与块 SDE","source_units":[{"raw_tex":"\\begin{proof}\n\nWe rewrite the SDE into \n\\begin{equation*}\n\\begin{aligned}\n    \\begin{pmatrix} dX_t \\\\ d\\bar{X}_t\\end{pmatrix} =& \\left(\\begin{pmatrix} 0 & 0 \\\\ \\frac{1}{t} & -\\frac{1}{t} \\end{pmatrix}\\otimes I_d\\right)\\begin{pmatrix} X_t \\\\ \\bar{X}_t\\end{pmatrix}dt +\\left(\\begin{pmatrix}- h_t & 0 \\\\ 0  & 0  \\end{pmatrix}\\otimes I_d\\right)\\begin{pmatrix} \\nabla f(X_t) \\\\ \\nabla f(\\bar{X}_t) \\end{pmatrix}dt \\\\\n    &+\\left(\\begin{pmatrix}h_t (\\gamma\\Sigma(X_t))^{1/2} \\\\ 0\\end{pmatrix}\\otimes I_d\\right)dB_t\n\\end{aligned}\n\\end{equation*}\n and denote $Y_t = \\begin{pmatrix} X_t \\\\ \\bar{X}_t \\end{pmatrix}$. \n \\smallbreak\n","start":111630,"end":112251}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"应用 Itô 并取期望","source_units":[{"raw_tex":" We consider the quadratic function~\\eqref{lyap_avg} $\\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}(X_t, \\bar{X}_t ,t) = \\mathcal{V}(Y_t,t)$. Applying Ito's formula, its derivative with respect to time is $\\frac{d}{dt} \\mathcal{V}(Y_t,t) = \\frac{\\partial}{\\partial t}\\mathcal{V}(Y_t,t) + \\frac{\\partial}{\\partial y}\\mathcal{V}\\frac{dY_t}{dt} + \\frac{1}{2}\\gamma h^2_t {\\rm Tr}[\\frac{\\partial^2}{\\partial Y_t^2}\\mathcal{V}(Y_t,t)^\\top \\begin{pmatrix}\\Sigma(X_t) \\\\ 0\\end{pmatrix}]$. By taking the expectation of Ito's formula, we have $\\frac{d}{dt} \\mathbf{E}\\mathcal{V}(Y_t,t) = \\mathbf{E}[\\frac{\\partial}{\\partial t}\\mathcal{V}(Y_t,t) + \\frac{\\partial}{\\partial y}\\mathcal{V}\\frac{dY_t}{dt}] +\\frac{1}{2}\\gamma h^2_t \\mathbf{E}{\\rm Tr}(\\frac{\\partial^2}{\\partial Y_t^2}\\mathcal{V}(Y_t,t)^\\top \\begin{pmatrix}\\Sigma(X_t) \\\\ 0\\end{pmatrix})$.\n \\smallbreak\n","start":112251,"end":113100}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"将方差项留在证书外","source_units":[{"raw_tex":" The variance term $\\frac{1}{2}\\gamma h^2_t \\mathbf{E}{\\rm Tr}(\\frac{\\partial^2}{\\partial Y_t^2}\\mathcal{V}(Y_t,t)^\\top \\begin{pmatrix}\\Sigma(X_t) \\\\ 0\\end{pmatrix})$ depends on the second derivative of $f$ in the space variable $(X_t)$, and on the covariance $\\Sigma_t$. We do not take this term into account in the performance estimation frawework.\n \\smallbreak\n","start":113100,"end":113464}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"归约为非正一阶漂移","source_units":[{"raw_tex":" We formulate the problem of verifying a quadratic function as verifying the inequality $\\frac{d}{dt}\\mathbf{E}\\mathcal{V}(X_t, \\bar{X}_t,t) \\leqslant \\frac{1}{2}\\gamma h^2_t \\mathbf{E}{\\rm Tr}(\\frac{\\partial^2}{\\partial X_t^2}\\mathcal{V}(X_t, \\bar{X}_t, t)^\\top \\Sigma(X_t))$ (that is $\\mathbf{E}[\\frac{\\partial}{\\partial t}\\mathcal{V}(Y_t,t) + \\frac{\\partial}{\\partial y}\\mathcal{V}\\frac{dY_t}{dt}] \\leqslant 0$) holds for any twice continuously differentiable function $f \\in \\mathcal{F}_{0, \\infty}$, and any trajectory $(X_t, \\bar{X}_t)$ generated by the SDE under Polyak-Ruppert averaging~\\eqref{ODE_mean1}. \n \\smallbreak\n","start":113464,"end":114092}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"转向零噪声确定性验证","source_units":[{"raw_tex":" This problem is equivalent with verifying that $\\frac{d}{dt}\\mathcal{V}(\\tilde{Y}_t,t) \\leqslant 0$ holds for any function twice continuously differentiable $f \\in \\mathcal{F}_{0, \\infty}$ and any trajectory $\\tilde{Y}_t = \\begin{pmatrix} \\tilde{X}_t & \\tilde{\\bar{X}}_t\\end{pmatrix}^\\top $ generated by deterministic gradient flow from the SDE~\\eqref{ODE_mean1} with $\\gamma = 0$. \n\\smallbreak\n","start":114092,"end":114488}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"确定性最坏导数表述","source_units":[{"raw_tex":"We follow the methodology developed in Section~\\ref{sec:deterministic} for ODEs. We formulate the verification of such a Lyapunov function as the maximization problem\n\\begin{equation*}\n    \\begin{aligned}\n      0\\geqslant \\max_{X_t \\in \\mathbf{R}^d,  d\\in \\mathbf{N}, \\ f\\in \\mathcal{F}_{0, \\infty}} \\ & \\ \\frac{d}{dt}  \\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}(X_t, \\bar{X}_t, t)\\\\\n      \\text{subject to }\n    \\ \\dot{X}_t &= -h_t\\nabla f(\\bar{X}_t), \\\n        \\dot{\\bar{X}}_t = \\frac{X_t - \\bar{X}_t}{t}dt.\n\\end{aligned}\n\\end{equation*}\n","start":114488,"end":115025}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"给出六乘子 LMI","source_units":[{"raw_tex":"This maximization problem can be formulated as the following SDP program\n\n\\begin{equation*}\n    \\begin{aligned}\n    & \\min_{\\lambda_t^{(i)} \\geqslant 0, \\ i \\in \\{1, ..., 6 \\}} \\ 0 \\, \\\\\n&\\begin{pmatrix} \\dot{p}^{(11)}_t + \\frac{2p^{(12)}_t}{t} & \\dot{p}^{(12)}_t + \\frac{p^{(22)}_t - p^{(12)}_t}{t}& \\frac{\\lambda^{(6)}_t + \\lambda^{(4)}_t - 2h_tp^{(11)}_t}{2} & \\frac{a^{(2)}_t}{2t} - \\frac{\\lambda^{(5)}_t}{2}\\\\ \n\n\\dot{p}^{(12)}_t + \\frac{p^{(22)}_t - p^{(12)}_t}{t} & \\dot{p}^{(22)}_t - \\frac{2p^{(22)}_t}{t} & -\\frac{\\lambda^{(6)}_t - 2h_tp^{(12)}_t}{2} & -\\frac{a^{(2)}_t}{2t} + \\frac{\\lambda^{(5)}_t + \\lambda^{(3)}_t}{2}\\\\ \n\n \\frac{\\lambda^{(6)}_t + \\lambda^{(4)}_t - 2h_tp^{(11)}_t}{2} & -\\frac{\\lambda^{(6)}_t - 2h_tp^{(12)}_t}{2} & -a^{(1)}_t & 0 \\\\ \n\n\\frac{a^{(2)}_t}{2t} - \\frac{\\lambda^{(5)}_t}{2} & -\\frac{a^{(2)}_t}{2t} + \\frac{\\lambda^{(5)}_t + \\lambda^{(3)}_t}{2} & 0 &  0\\end{pmatrix} \\preccurlyeq 0, \\\\ \n&\\dot{a}^{(1)}_t + \\lambda^{(1)}_t + \\lambda^{(5)}_t = \\lambda^{(4)}_t + \\lambda^{(6)}_t, \\\\ \n&\\dot{a}^{(2)}_t + \\lambda^{(2)}_t + \\lambda^{(6)}_t = \\lambda^{(3)}_t + \\lambda^{(5)}_t + \\dot{a}^{(1)}_t. \n\\end{aligned}\n\\end{equation*}\n","start":115025,"end":116182}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"指定四向量 Gram 数据","source_units":[{"raw_tex":"Note that the Gram matrix $G$ and function value vector $F$ to be introduced are different from the setting explored in Section~\\ref{sec:deterministic}, more precisely $G = Q^\\top Q \\succcurlyeq 0, \\ Q = [X_t - x_\\star, \\bar{X}_t - x_\\star, g_t, \\bar{g}_t]$ and $F = (f(X_t), f(\\bar{X}_t), f_\\star)$.\n\n","start":116182,"end":116484}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":[],"declared_external":false,"source_only":false,"title":"以 Itô 公式收束","source_units":[{"raw_tex":"The final inequality follows from Ito's formula.\n\\end{proof}","start":116484,"end":116544}]},{"id":"S055","semantic_type":"external_fact","kind":"external_fact","labels":[],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；用于解释原文第1237行“取 Itô 公式的期望”所省略的条件性背景。"],"source_only":false,"title":"取期望的条件性规则","source_units":[]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S043","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S010","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S011","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S013","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S013","target_id":"S017","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S025","target_id":"S017","kind":"supports"},{"source_id":"S014","target_id":"S018","kind":"supports"},{"source_id":"S015","target_id":"S018","kind":"supports"},{"source_id":"S027","target_id":"S018","kind":"supports"},{"source_id":"S015","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S019","target_id":"S022","kind":"supports"},{"source_id":"S020","target_id":"S022","kind":"supports"},{"source_id":"S019","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S026","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S024","kind":"supports"},{"source_id":"S028","target_id":"S024","kind":"supports"},{"source_id":"S029","target_id":"S036","kind":"supports"},{"source_id":"S030","target_id":"S036","kind":"supports"},{"source_id":"S033","target_id":"S038","kind":"supports"},{"source_id":"S008","target_id":"S040","kind":"supports"},{"source_id":"S038","target_id":"S042","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S006","target_id":"S042","kind":"supports"},{"source_id":"S038","target_id":"S043","kind":"supports"},{"source_id":"S040","target_id":"S043","kind":"supports"},{"source_id":"S006","target_id":"S043","kind":"supports"},{"source_id":"S035","target_id":"S043","kind":"supports"},{"source_id":"S015","target_id":"S044","kind":"supports"},{"source_id":"S018","target_id":"S044","kind":"supports"},{"source_id":"S022","target_id":"S044","kind":"supports"},{"source_id":"S024","target_id":"S044","kind":"supports"},{"source_id":"S038","target_id":"S046","kind":"supports"},{"source_id":"S040","target_id":"S047","kind":"supports"},{"source_id":"S041","target_id":"S047","kind":"supports"},{"source_id":"S036","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S035","target_id":"S047","kind":"supports"},{"source_id":"S055","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S038","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S038","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S013","target_id":"S051","kind":"supports"},{"source_id":"S016","target_id":"S051","kind":"supports"},{"source_id":"S027","target_id":"S053","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S003","target_id":"S052","kind":"supports"},{"source_id":"S022","target_id":"S052","kind":"supports"},{"source_id":"S028","target_id":"S052","kind":"supports"},{"source_id":"S047","target_id":"S054","kind":"supports"},{"source_id":"S049","target_id":"S054","kind":"supports"},{"source_id":"S052","target_id":"S054","kind":"supports"},{"source_id":"S040","target_id":"S054","kind":"supports"},{"source_id":"S036","target_id":"S054","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S048","target_id":"S045","kind":"supports"},{"source_id":"S049","target_id":"S045","kind":"supports"},{"source_id":"S050","target_id":"S045","kind":"supports"},{"source_id":"S051","target_id":"S045","kind":"supports"},{"source_id":"S052","target_id":"S045","kind":"supports"},{"source_id":"S053","target_id":"S045","kind":"supports"},{"source_id":"S054","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"}],"overview":{"direction_convention":"supporter_to_supported","main_result_id":"S043","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S010","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S011","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S013","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S013","target_id":"S017","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S025","target_id":"S017","kind":"supports"},{"source_id":"S014","target_id":"S018","kind":"supports"},{"source_id":"S015","target_id":"S018","kind":"supports"},{"source_id":"S027","target_id":"S018","kind":"supports"},{"source_id":"S015","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S019","target_id":"S022","kind":"supports"},{"source_id":"S020","target_id":"S022","kind":"supports"},{"source_id":"S019","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S026","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S024","kind":"supports"},{"source_id":"S028","target_id":"S024","kind":"supports"},{"source_id":"S029","target_id":"S036","kind":"supports"},{"source_id":"S030","target_id":"S036","kind":"supports"},{"source_id":"S033","target_id":"S038","kind":"supports"},{"source_id":"S008","target_id":"S040","kind":"supports"},{"source_id":"S038","target_id":"S042","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S006","target_id":"S042","kind":"supports"},{"source_id":"S038","target_id":"S043","kind":"supports"},{"source_id":"S040","target_id":"S043","kind":"supports"},{"source_id":"S006","target_id":"S043","kind":"supports"},{"source_id":"S035","target_id":"S043","kind":"supports"},{"source_id":"S015","target_id":"S044","kind":"supports"},{"source_id":"S018","target_id":"S044","kind":"supports"},{"source_id":"S022","target_id":"S044","kind":"supports"},{"source_id":"S024","target_id":"S044","kind":"supports"},{"source_id":"S038","target_id":"S046","kind":"supports"},{"source_id":"S040","target_id":"S047","kind":"supports"},{"source_id":"S041","target_id":"S047","kind":"supports"},{"source_id":"S036","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S035","target_id":"S047","kind":"supports"},{"source_id":"S055","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S038","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S038","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S013","target_id":"S051","kind":"supports"},{"source_id":"S016","target_id":"S051","kind":"supports"},{"source_id":"S027","target_id":"S053","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S003","target_id":"S052","kind":"supports"},{"source_id":"S022","target_id":"S052","kind":"supports"},{"source_id":"S028","target_id":"S052","kind":"supports"},{"source_id":"S047","target_id":"S054","kind":"supports"},{"source_id":"S049","target_id":"S054","kind":"supports"},{"source_id":"S052","target_id":"S054","kind":"supports"},{"source_id":"S040","target_id":"S054","kind":"supports"},{"source_id":"S036","target_id":"S054","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S048","target_id":"S045","kind":"supports"},{"source_id":"S049","target_id":"S045","kind":"supports"},{"source_id":"S050","target_id":"S045","kind":"supports"},{"source_id":"S051","target_id":"S045","kind":"supports"},{"source_id":"S052","target_id":"S045","kind":"supports"},{"source_id":"S053","target_id":"S045","kind":"supports"},{"source_id":"S054","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"}],"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","labels":["eq:optim"],"member_ids":["S001"],"declared_external":false,"source_only":false,"title":"全空间优化问题","source_units":[{"raw_tex":"Convex optimization is an important tool in the numerical analyst toolbox. It serves, among others, for framing modeling problems in data science and signal processing. We consider optimization problems of the form:\n\\begin{equation}\n    \\min_{x \\in \\mathbf{R}^d} f(x),\n    \\label{eq:optim}\n\\end{equation}\n","start":7680,"end":7985}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S002"],"declared_external":false,"source_only":false,"title":"全篇凸性约定","source_units":[{"raw_tex":"\\subsection{Assumptions}\n\\label{sec:assumptions}\nThroughout this work, functions to be minimized are convex (see Problem~\\eqref{eq:optim}). Under this assumption, stationary points are global minimizers. We restrict ourselves to continuous-time versions of gradient descent, accelerated gradient descent and stochastic gradient descent.\n\\smallbreak\n","start":20075,"end":20424}]},{"id":"S003","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S003"],"declared_external":false,"source_only":false,"title":"凸性与支撑超平面","source_units":[{"raw_tex":"Let us recall a few basic definitions and properties characterizing the classes of functions under consideration within the next sections. A function $f : \\mathbf{R}^d \\rightarrow \\mathbf{R}$ is convex if for all $x,y \\in \\mathbf{R}^d$, and for all $\\lambda \\in [0,1]$, $f(\\lambda x + (1-\\lambda)y) \\leqslant \\lambda f(x) + (1-\\lambda)f(y)$. Such functions (i.e. with full domain, $\\text{dom}(f) = \\mathbf{R}^d$) are convex closed proper (CCP) (i.e., their epigraphs are non-empty closed convex sets). For simplicity, we assume in addition differentiability of $f$.  Such a differentiable function $f$ is convex if and only if for all $x,y \\in \\mathbf{R}^d$, $f(y) \\geqslant f(x) + \\langle \\nabla f(x), y-x \\rangle$. A differentiable function $f$ is $L$-smooth if its gradient is $L$-Lipschitz, that is if for any $x,y  \\ \\in \\mathbf{R}^d$:\n","start":20424,"end":21265}]},{"id":"S004","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S004"],"declared_external":false,"source_only":false,"title":"梯度光滑性","source_units":[{"raw_tex":"Let us recall a few basic definitions and properties characterizing the classes of functions under consideration within the next sections. A function $f : \\mathbf{R}^d \\rightarrow \\mathbf{R}$ is convex if for all $x,y \\in \\mathbf{R}^d$, and for all $\\lambda \\in [0,1]$, $f(\\lambda x + (1-\\lambda)y) \\leqslant \\lambda f(x) + (1-\\lambda)f(y)$. Such functions (i.e. with full domain, $\\text{dom}(f) = \\mathbf{R}^d$) are convex closed proper (CCP) (i.e., their epigraphs are non-empty closed convex sets). For simplicity, we assume in addition differentiability of $f$.  Such a differentiable function $f$ is convex if and only if for all $x,y \\in \\mathbf{R}^d$, $f(y) \\geqslant f(x) + \\langle \\nabla f(x), y-x \\rangle$. A differentiable function $f$ is $L$-smooth if its gradient is $L$-Lipschitz, that is if for any $x,y  \\ \\in \\mathbf{R}^d$:\n\\begin{equation*}\n        \\|\\nabla f(x) - \\nabla f(y)\\| \\leqslant L \\|x-y\\|.\n\\end{equation*}\n","start":20424,"end":21358}]},{"id":"S005","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S005"],"declared_external":false,"source_only":false,"title":"强凸性背景约定","source_units":[{"raw_tex":"Smoothness is a common assumption for analyzing optimization methods, that limits the growth rate of the function. A convex differentiable function $f$ is $\\mu$-strongly convex if for any $x, y \\in \\mathbf{R}^d$ it satisfies:\n\\begin{equation*}\n         \\|\\nabla f(x) - \\nabla f(y)\\| \\geqslant \\mu \\|x-y\\|.\n\\end{equation*}\n","start":21358,"end":21680}]},{"id":"S006","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S006"],"declared_external":false,"source_only":false,"title":"函数类记号","source_units":[{"raw_tex":"Strong convexity ensures the function is not too flat, and the unicity of the minimizer~$x_\\star$. We denote by $\\mathcal{F}_{\\mu,L}$ the family of $L$-smooth $\\mu$-strongly convex functions from $\\mathbf{R}^d$ to $\\mathbf{R}$, with $0 \\leqslant \\mu \\leqslant L \\leqslant +\\infty$. Weaker assumptions than strong convexity are also encountered in the literature for analyzing gradient algorithms, and leads to similar convergence guarantees. Among them, Bolte~et~al.~\\cite[Appendix 5.2]{bolte2009} proved linear convergence of gradient descent under Łojasiewicz inequality, first introduced by Łojasiewicz~\\cite{Lojasiewicz1963}. Other relaxed versions of strong convexity followed~\\cite{necoara2019linear}.\n","start":21680,"end":22388}]},{"id":"S008","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S008"],"declared_external":false,"source_only":false,"title":"极小点与最优值","source_units":[{"raw_tex":"Let $f$ be $\\mu$-strongly convex (i.e., $f\\in\\mathcal{F}_{\\mu,\\infty}$) with $\\mu>0$, and $x_\\star$ be its unique minimizer such that $f(x_\\star) = f_\\star$. In this context, it is possible to prove linear convergence of the gradient flow to its stationary point. Scieur et~al.~proved in~\\cite[Proposition 1.1]{scieur2017integration} the following convergence bound in function values for the gradient flow:\n","start":23778,"end":24186}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","labels":["reflyapmu"],"member_ids":["S009"],"declared_external":false,"source_only":false,"title":"强凸梯度流速率示例","source_units":[{"raw_tex":"Let $f$ be $\\mu$-strongly convex (i.e., $f\\in\\mathcal{F}_{\\mu,\\infty}$) with $\\mu>0$, and $x_\\star$ be its unique minimizer such that $f(x_\\star) = f_\\star$. In this context, it is possible to prove linear convergence of the gradient flow to its stationary point. Scieur et~al.~proved in~\\cite[Proposition 1.1]{scieur2017integration} the following convergence bound in function values for the gradient flow:\n\\begin{equation}\n\\label{reflyapmu}\nf(X_t) - f_\\star \\leqslant e^{- 2 \\mu t}(f(x_0) - f_\\star).\n\\end{equation}\n","start":23778,"end":24296}]},{"id":"S010","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S010"],"declared_external":false,"source_only":false,"title":"强凸示例的作者推导","source_units":[{"raw_tex":"This convergence guarantee follows directly from the derivative with respect to time of the Lyapunov function $\\mathcal{V}(X_t) = f(X_t) - f_\\star$, together with strong convexity (or Łojasiewicz inequality): $\\frac{d}{dt}\\mathcal{V}(X_t) = \\dot{X_t}^\\top \\nabla f(X_t) = -\\|\\nabla f(X_t)\\|^2 \\leqslant -2 \\mu (f(X_t) - f_\\star) = -2 \\mu \\mathcal{V}(X_t)$.\n","start":24296,"end":24653}]},{"id":"S011","semantic_type":"definition","kind":"definition","labels":["lyapunov"],"member_ids":["S011"],"declared_external":false,"source_only":false,"title":"确定性二次势函数","source_units":[{"raw_tex":"\n\\bigbreak\nGiven the specific gradient flows studied in this work, it is reasonable to search for Lyapunov functions made of linear combinations of function values, and a quadratic form in the trajectory $X_t$. We simply refer to them as quadratic Lyapunov functions:\n\\begin{equation}\n\\label{lyapunov}\n    \\mathcal{V}_{a,c}(X_t) = a \\cdot (f(X_t) - f_\\star) + c \\cdot \\|X_t - x_\\star\\|^2,\n\\end{equation}\nwhere $a, c \\geqslant 0$ are constants that do not depend on $t$, such that the function $ \\mathcal{V}_{a,c}$ is nonnegative and nonincreasing along the flow. Quadratic Lyapunov functions are common for proving convergence of gradient flows and cover for instance the Lyapunov function used to prove convergence of the gradient flow under strong convexity~\\eqref{reflyapmu}. Given a Lyapunov function $\\mathcal{V}_{a,c}$, the idea is to find the largest nonnegative value $\\tau_\\star$ such that the condition\n","start":24653,"end":25566}]},{"id":"S012","semantic_type":"definition","kind":"definition","labels":["wc_mu"],"member_ids":["S012"],"declared_external":false,"source_only":false,"title":"统一衰减验证目标","source_units":[{"raw_tex":"\\begin{equation}\n\\label{wc_mu}\n    \\frac{d}{dt}\\mathcal{V}_{a,c}(X_t) \\leqslant -\\tau_\\star \\mathcal{V}_{a,c}(X_t),\n\\end{equation}\nholds for any dimension $d \\in \\mathbf{N}$, any function $f \\in \\mathcal{F}_{\\mu, \\infty}$ and any trajectory $X_t$ generated by the gradient flow. After integration, \\eqref{wc_mu} allows to obtain a convergence guarantee of the form $\\mathcal{V}_{a,c}(X_t) \\leqslant e^{-\\tau_\\star t} \\mathcal{V}_{a,c}(X_0)$. Given a certain Lyapunov function $\\mathcal{V}_{a,c}$ and a time $t$, we get that the largest acceptable $\\tau_\\star$ is a solution to:\n","start":25566,"end":26144}]},{"id":"S013","semantic_type":"algorithm","kind":"algorithm","labels":["eq:str_pep"],"member_ids":["S013"],"declared_external":false,"source_only":false,"title":"性能估计极值表述","source_units":[{"raw_tex":"\\begin{equation}\n\\label{eq:str_pep}\n    \\begin{aligned}\n      -\\tau_\\star = \\max_{X_t \\in \\mathbf{R}^d, d \\in \\mathbf{N}, f\\in \\mathcal{F}_{\\mu, \\infty}} \\ & \\  \\frac{d}{dt}\\mathcal{V}_{a,c}(X_t), \\\\\n      \\text{subject to }\n    \\mathcal{V}_{a,c}(X_t) &= 1, \\\\\n    \\dot{X_t} &= - \\nabla f (X_t).\n\\end{aligned}\n\\end{equation}\nThis minimization problem is invariant with respect to $t$. \n","start":26144,"end":26530}]},{"id":"S014","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S014"],"declared_external":false,"source_only":false,"title":"采样函数值与梯度","source_units":[{"raw_tex":"It is established in~\\cite{taylor2016smooth, drori2012performance} that these so-called performance estimation problems (PEP) can be formulated as SDPs. Because of the variable $f \\in \\mathcal{F}_{\\mu, \\infty}$, the maximization problem~\\eqref{eq:str_pep} is infinite-dimensional. Recall that a differentiable function $f \\in \\mathcal{F}_{\\mu, \\infty}$ verifies for all points $x,y \\in \\mathbf{R}^d$, $f(x) - f(y) - \\langle \\nabla f(y), x-y \\rangle \\geqslant \\frac{\\mu}{2} \\|x-y\\|^2$. Using alternate variables $f_t$, $f_\\star$, $g_t$ and $g_\\star$ (informally: $f_t = f(X_t)$, $f_\\star = f(x_\\star)$, $ g_t = \\nabla f(X_t)$ and $g_\\star = \\nabla f(x_\\star) = 0$), it holds that:\n","start":26530,"end":27210}]},{"id":"S015","semantic_type":"algorithm","kind":"algorithm","labels":["eq:interpolation_step"],"member_ids":["S015"],"declared_external":false,"source_only":false,"title":"有限采样不等式替换","source_units":[{"raw_tex":"\\begin{equation}\n\\label{eq:interpolation_step}\n    \\begin{aligned}\n      -\\tau_\\star = \\max_{X_t \\in \\mathbf{R}^d, d \\in \\mathbf{N}} \\ & \\ \\frac{d}{dt}\\mathcal{V}_{a, c}(X_t), \\\\\n      \\text{subject to } &\\mathcal{V}_{a,c}(X_t) = 1, \\\\\n    &\\dot{X}_t = - g_t, \\\\\n    & f_i - f_j -\\langle g_j, X_i-X_j\\rangle \\geqslant \\frac{\\mu}{2}\\|X_i-X_j\\|^2, \\ \\forall i,j=t, \\star.\n\\end{aligned}\n\\end{equation}\n","start":27210,"end":27609}]},{"id":"S016","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S016"],"declared_external":false,"source_only":false,"title":"采样松弛给出上界","source_units":[{"raw_tex":"The fact that~\\eqref{eq:interpolation_step} produces an upper bound on $-\\tau_\\star$ directly follows from the fact that any sampled strongly convex function satisfy these inequalities at the sampled points (here $X_t$ and $x_\\star$). Thereby, any feasible point to~\\eqref{eq:str_pep} corresponds to a feasible point for~\\eqref{eq:interpolation_step} with the same objective value. In the other direction,~\\cite[Corollary 2]{taylor2016smooth} (which provides a constructive way to obtain some $f \\in \\mathcal{F}_{\\mu, \\infty}$ that interpolates the triplets $(X_i, g_i, f_i)_{i=t, \\star}$) ensures that any feasible point to~\\eqref{eq:interpolation_step} can be translated to a feasible point to~\\eqref{eq:str_pep} with also the same objective value, thereby reaching the equivalence between formulations~\\eqref{eq:str_pep} and~\\eqref{eq:interpolation_step}.\n","start":27609,"end":28468}]},{"id":"S017","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S017"],"declared_external":false,"source_only":false,"title":"插值逆向实现与精确性","source_units":[{"raw_tex":"The fact that~\\eqref{eq:interpolation_step} produces an upper bound on $-\\tau_\\star$ directly follows from the fact that any sampled strongly convex function satisfy these inequalities at the sampled points (here $X_t$ and $x_\\star$). Thereby, any feasible point to~\\eqref{eq:str_pep} corresponds to a feasible point for~\\eqref{eq:interpolation_step} with the same objective value. In the other direction,~\\cite[Corollary 2]{taylor2016smooth} (which provides a constructive way to obtain some $f \\in \\mathcal{F}_{\\mu, \\infty}$ that interpolates the triplets $(X_i, g_i, f_i)_{i=t, \\star}$) ensures that any feasible point to~\\eqref{eq:interpolation_step} can be translated to a feasible point to~\\eqref{eq:str_pep} with also the same objective value, thereby reaching the equivalence between formulations~\\eqref{eq:str_pep} and~\\eqref{eq:interpolation_step}.\n","start":27609,"end":28468}]},{"id":"S018","semantic_type":"definition","kind":"definition","labels":[],"member_ids":["S018"],"declared_external":false,"source_only":false,"title":"二向量 Gram 编码","source_units":[{"raw_tex":"\\smallbreak\nIn a second stage, we introduce $G  = \\begin{pmatrix} \\|X_t - x_\\star\\|^2 &  \\langle X_t - x_\\star, g_t \\rangle  \\\\  \\langle X_t-x_\\star, g_t \\rangle & \\| g_t \\|^2 \\end{pmatrix} \\succcurlyeq 0$, a Gram matrix and a vector $F = [f_t, f_\\star]$, thereby obtaining a semidefinite reformulation:\n","start":28468,"end":28772}]},{"id":"S019","semantic_type":"algorithm","kind":"algorithm","labels":["SDP"],"member_ids":["S019"],"declared_external":false,"source_only":false,"title":"性能估计的 SDP 编码","source_units":[{"raw_tex":"\\begin{equation}\n\\label{SDP}\n    \\begin{aligned}\n      -\\tau_\\star = \\max_{G\\succcurlyeq 0, F \\in \\mathbf{R}^2} & b_0^\\top F + {\\rm Tr}(A_0G), \\\\\n      \\text{subject to }\n    & b_1^\\top F + {\\rm Tr}(A_1G) \\geqslant 0, \\\\\n    & b_2^\\top F + {\\rm Tr}(A_2G) \\geqslant 0, \\\\\n    & b_3^\\top F + {\\rm Tr}(A_3G) = 1. \\\\\n\\end{aligned}\n\\end{equation}\nwhere $A_0 = \\begin{pmatrix} 0 & -c \\\\ -c & -a \\end{pmatrix}$, $A_1 = \\begin{pmatrix}-\\mu/2 & 1/2 \\\\ 1/2 & 0 \\end{pmatrix}$, $A_2= \\begin{pmatrix}-\\mu/2 & 0 \\\\ 0 & 0\\end{pmatrix}$, $A_3 = \\begin{pmatrix}c & 0 \\\\ 0 & 0 \\end{pmatrix}$, $b_0 = [0, 0]^\\top$, $b_1 = [-1,\\ 1]^\\top $, $b_2=[1, \\ -1]^\\top $ and $b_3 = a[1, \\ -1]^\\top $. Consider the corresponding Lagrangian, for $F \\in \\mathbf{R}^2$, $G \\succcurlyeq 0$, $\\tau \\in \\mathbf{R}$ and $\\lambda_1, \\lambda_2 \\geqslant 0$: \n","start":28772,"end":29593}]},{"id":"S020","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S020"],"declared_external":false,"source_only":false,"title":"非负乘子的拉格朗日函数","source_units":[{"raw_tex":"where $A_0 = \\begin{pmatrix} 0 & -c \\\\ -c & -a \\end{pmatrix}$, $A_1 = \\begin{pmatrix}-\\mu/2 & 1/2 \\\\ 1/2 & 0 \\end{pmatrix}$, $A_2= \\begin{pmatrix}-\\mu/2 & 0 \\\\ 0 & 0\\end{pmatrix}$, $A_3 = \\begin{pmatrix}c & 0 \\\\ 0 & 0 \\end{pmatrix}$, $b_0 = [0, 0]^\\top$, $b_1 = [-1,\\ 1]^\\top $, $b_2=[1, \\ -1]^\\top $ and $b_3 = a[1, \\ -1]^\\top $. Consider the corresponding Lagrangian, for $F \\in \\mathbf{R}^2$, $G \\succcurlyeq 0$, $\\tau \\in \\mathbf{R}$ and $\\lambda_1, \\lambda_2 \\geqslant 0$: \n\\begin{equation*}\n    \\begin{aligned}\n    \\mathcal{L}(F, G, \\tau, \\lambda_1, \\lambda_2) =& b_0^\\top F + {\\rm Tr}(A_0G) + \\tau \\cdot (b_3^\\top F + {\\rm Tr}(A_3G) - 1) \\\\ \n    & + \\lambda_1 \\cdot(b_1^\\top F + {\\rm Tr}(A_1G)) + \\lambda_2 \\cdot(b_2^\\top F + {\\rm Tr}(A_2G)).\n    \\end{aligned}\n\\end{equation*}\n","start":29114,"end":29898}]},{"id":"S021","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S021"],"declared_external":false,"source_only":false,"title":"方法示例的鞍点表述","source_units":[{"raw_tex":"The saddle point of the Lagrangian is given by:\n\\begin{equation*}\n    \\begin{aligned}\n    \\tau_\\star &= \\min_{\\tau, \\lambda_1 \\geqslant 0, \\lambda_2 \\geqslant 0} \\max_{F, G \\succcurlyeq 0} \\mathcal{L}(F, G, \\tau, \\lambda_1, \\lambda_2).\n\\end{aligned}\n\\end{equation*}\n","start":29898,"end":30164}]},{"id":"S022","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S022"],"declared_external":false,"source_only":false,"title":"对偶矩阵与系数平衡","source_units":[{"raw_tex":"The Lagrangian dual of the SDP~\\eqref{SDP} is obtained by maximizing over $F \\in \\mathbf{R}^2$, $G \\succcurlyeq 0$:\n\\begin{equation*}\n    \\begin{aligned}\n    -\\tau_\\star =  \\min_{\\tau, \\lambda_1, \\lambda_2} & \\ -\\tau, \\\\\n      \\text{subject to } & S = A_0 + \\lambda_1 A_1 + \\lambda_2 A_2 + \\tau A_3 \\preccurlyeq 0 \\\\ \n    &  b_0 +  \\lambda_1 b_1 +\\lambda_2 b_2  + \\tau b_3 = 0, \\\\\n    &  \\tau\\in \\mathbf{R}, \\ \\lambda_1, \\lambda_2 \\geqslant 0.\n\\end{aligned}\n\\end{equation*}\n","start":30164,"end":30638}]},{"id":"S023","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S023"],"declared_external":false,"source_only":false,"title":"强对偶与方法精确性","source_units":[{"raw_tex":"The equality with $\\tau_\\star$ comes from strong duality, that holds via Slater's conditions (it is relatively easy to show that there exists a Slater's point using the same construction as in~\\cite[Theorem 6]{taylor2016smooth}). Finally, since any feasible $\\tau$ is a lower bound for $\\tau_\\star$, those developments allow arriving to the equivalence between verifying a quadratic Lyapunov function and verifying the feasibility of an LMI.\n","start":30638,"end":31080}]},{"id":"S024","semantic_type":"derived_fact","kind":"derived_fact","labels":[],"member_ids":["S024"],"declared_external":false,"source_only":false,"title":"可行对偶证书的充分性","source_units":[{"raw_tex":"The equality with $\\tau_\\star$ comes from strong duality, that holds via Slater's conditions (it is relatively easy to show that there exists a Slater's point using the same construction as in~\\cite[Theorem 6]{taylor2016smooth}). Finally, since any feasible $\\tau$ is a lower bound for $\\tau_\\star$, those developments allow arriving to the equivalence between verifying a quadratic Lyapunov function and verifying the feasibility of an LMI.\n","start":30638,"end":31080}]},{"id":"S025","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S025"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第三项：taylor2016smooth, Corollary 2；原文第397行明确引用。"],"source_only":false,"title":"强凸数据插值定理","source_units":[]},{"id":"S026","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S026"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第三项：Slater's condition for strong duality；原文第432行援引。"],"source_only":false,"title":"Slater 强对偶准则","source_units":[]},{"id":"S027","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S027"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第二项：positive semidefinite Gram representations。"],"source_only":false,"title":"半正定 Gram 表示","source_units":[]},{"id":"S028","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S028"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第二项：SDP weak duality。"],"source_only":false,"title":"半定规划弱对偶","source_units":[]},{"id":"S029","semantic_type":"definition","kind":"definition","labels":["eq:sme"],"member_ids":["S029"],"declared_external":false,"source_only":false,"title":"基本扩散模型","source_units":[{"raw_tex":"The corresponding SDE is given by:\n\\begin{equation}\n\\label{eq:sme}\n    dX_t = -\\nabla f(X_t)dt + (\\gamma \\Sigma(X_t))^{1/2}dB_t,\n\\end{equation}\n","start":69903,"end":70047}]},{"id":"S030","semantic_type":"notation","kind":"notation","labels":[],"member_ids":["S030"],"declared_external":false,"source_only":false,"title":"标准布朗运动记号","source_units":[{"raw_tex":"where $B_t$ is a standard Brownian motion. The SDE~\\eqref{eq:sme} is an (order-$1$ weak) approximation of  SGD~(\\cite[Theorem 1]{li2017stochastic},~\\cite{Li2019}), which allows to take into account the role of constant step size in the dynamics of SGD (while keeping them small). Under mild assumptions on $f$, Li~et~al.~\\cite{Li2019} proved the weak approximation of SGD by this SDE on a finite interval $[0, T]$: there exists $C>0$ such that $\\|\\mathbf{E}[x_k] - \\mathbf{E}[X(k\\gamma)] \\| \\leqslant C\\gamma$ for $k \\in [0, \\frac{T}{\\gamma}]$. However, the approximation point of view from this approach is relatively limited since $C$ depends exponentially on~$T$. \n","start":70047,"end":70715}]},{"id":"S032","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S032"],"declared_external":false,"source_only":false,"title":"变化步长的建模背景","source_units":[{"raw_tex":"Compared to gradient descent, SGD does not converge to a stationary point under constant step sizes~\\cite{bach2011, taylor2021stochastic}. Convergence to a stationary point requires diminishing step sizes such as $\\gamma_k = \\frac{1}{\\sqrt{k}}$. Li~et~al.~\\cite[Section 4.1]{li2017stochastic} (and later Orvieto~and~Lucchi~\\cite[Section 2.1]{orvieto2020continuoustime}) proposed to include this varying learning rate in the dynamics:\n\\begin{equation*}\n    x_{k+1} = x_k - \\gamma h_k \\nabla f(x_k),\n\\end{equation*}\nwhere $\\gamma$ is the maximum allowed learning rate and $h_k \\in [0, 1]$ is the time-varying part. For $h_t \\geqslant 0$ a continuous function playing the role of step sizes $h_k$, the SDE is:\n","start":71221,"end":71928}]},{"id":"S033","semantic_type":"definition","kind":"definition","labels":["sme_general"],"member_ids":["S033"],"declared_external":false,"source_only":false,"title":"时变步长随机梯度流","source_units":[{"raw_tex":"where $\\gamma$ is the maximum allowed learning rate and $h_k \\in [0, 1]$ is the time-varying part. For $h_t \\geqslant 0$ a continuous function playing the role of step sizes $h_k$, the SDE is:\n\\begin{equation}\n\\label{sme_general}\n    dX_t = - h_t \\nabla f(X_t)dt + h_t(\\gamma \\Sigma(X_t))^{1/2}dB_t.\n\\end{equation}\n","start":71735,"end":72050}]},{"id":"S034","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S034"],"declared_external":false,"source_only":false,"title":"协方差与方差界","source_units":[{"raw_tex":"We treat the covariance matrix $\\Sigma(X_t)$ as symmetric, already implied by the notation $\\Sigma(X_t)^{1/2}$, but unstructured with bounded variance $\\Sigma(X_t) \\preccurlyeq \\Sigma$ along any trajectory $X_t$ generated by the approximating SDE~\\eqref{sme_general}. Compared to ODEs, functions $f \\in \\mathcal{F}_{0, L}$ to be optimized using SDEs are in addition assumed to be possibly $L$-smooth with $L\\in (0, \\infty]$ and to be twice continuously differentiable.\n","start":72050,"end":72519}]},{"id":"S035","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S035"],"declared_external":false,"source_only":false,"title":"随机分析的函数正则性","source_units":[{"raw_tex":"We treat the covariance matrix $\\Sigma(X_t)$ as symmetric, already implied by the notation $\\Sigma(X_t)^{1/2}$, but unstructured with bounded variance $\\Sigma(X_t) \\preccurlyeq \\Sigma$ along any trajectory $X_t$ generated by the approximating SDE~\\eqref{sme_general}. Compared to ODEs, functions $f \\in \\mathcal{F}_{0, L}$ to be optimized using SDEs are in addition assumed to be possibly $L$-smooth with $L\\in (0, \\infty]$ and to be twice continuously differentiable.\n","start":72050,"end":72519}]},{"id":"S036","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S036"],"declared_external":true,"declaration_text":["原始 external_prerequisites 第一项及 declared_permitted_prerequisite_ids B0088：标准多维含时 Itô 公式，原文第881行引用 sarkka_solin_2019, Theorem 4.2。"],"source_only":false,"title":"允许使用的 Itô 公式","source_units":[{"raw_tex":"\\smallbreak\nIn SDE theory, a differential with respect to time of a function of a solution to a stochastic process is given by Ito's Lemma~\\cite[Theorem 4.2]{sarkka_solin_2019}.\n\\begin{lemma}[Ito's Lemma]\nLet $g$ be a twice continuously differentiable function, and $X_t$ be a stochastic process solution to the SDE~\\eqref{eq:sme}, then\n\\begin{equation*}\n    dg(X_t,t) = \\frac{\\partial}{\\partial t}g(X_t, t)dt + \\frac{\\partial}{\\partial x}g(X_t, t) dX_t + \\frac{1}{2}\\gamma {\\rm Tr}(\\frac{\\partial^2}{\\partial x^2}g(X_t, t)\\Sigma(X_t))dt.\n\\end{equation*}\n\\end{lemma}\n","start":73421,"end":73988}]},{"id":"S037","semantic_type":"problem_setting","kind":"problem_setting","labels":[],"member_ids":["S037"],"declared_external":false,"source_only":false,"title":"平均模型的研究背景","source_units":[{"raw_tex":"\\subsubsection{Polyak-Ruppert averaging}\nIn this section, we analyze convergence properties of an SDE~\\eqref{sme_general} approximating SGD with time-varying step sizes under Polyak-Ruppert averaging. Taylor~and~Bach~\\cite[Theorem 6]{taylor2021stochastic} provided a systematic design of Lyapunov functions for~\\eqref{eq:pr_avg}, and a condition on the step size for convergence to the optimum. An approximating SDE for Polyak-Ruppert averaging is given by:\n","start":83501,"end":83959}]},{"id":"S038","semantic_type":"definition","kind":"definition","labels":["ODE_mean1"],"member_ids":["S038"],"declared_external":false,"source_only":false,"title":"Polyak–Ruppert 平均 SDE","source_units":[{"raw_tex":"\\begin{equation}\n\\label{ODE_mean1}\n    \\begin{aligned}\n        dX_t &= -h_t\\nabla f(X_t)dt + h_t(\\gamma \\Sigma(X_t))^{1/2} dB_t, \\\\\n        d\\bar{X}_t &= \\frac{X_t - \\bar{X}_t}{t}dt,\n        \\end{aligned}\n    \\end{equation}\n","start":83959,"end":84183}]},{"id":"S039","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S039"],"declared_external":false,"source_only":false,"title":"平均模型的步长范围","source_units":[{"raw_tex":"with step size $\\gamma>0$ that is taken close to zero, and a variable term $h_t\\in [0, 1]$. We introduce the family of quadratic functions taking the averaged sequence into account:\n","start":84183,"end":84365}]},{"id":"S040","semantic_type":"definition","kind":"definition","labels":["lyap_avg"],"member_ids":["S040"],"declared_external":false,"source_only":false,"title":"含平均点的二次势函数","source_units":[{"raw_tex":"\\begin{equation}\n\\label{lyap_avg}\n    \\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}(X_t, \\bar{X}_t, t) = \\begin{pmatrix}a^{(1)}_t \\\\ a^{(2)}_t \\end{pmatrix}^\\top \\!\\!\\!\\begin{pmatrix} f(X_t) - f_\\star \\\\ f(\\bar{X_t}) - f_\\star \\end{pmatrix} + \\begin{pmatrix} X_t - X_\\star\\\\ \\bar{X_t} - X_\\star\\end{pmatrix}^\\top \\!\\!\\! \\left(P_t\\otimes I_d\\right) \\! \\begin{pmatrix} X_t - X_\\star\\\\ \\bar{X_t} - X_\\star\\end{pmatrix},\n\\end{equation}\n","start":84365,"end":84791}]},{"id":"S041","semantic_type":"assumption","kind":"assumption","labels":[],"member_ids":["S041"],"declared_external":false,"source_only":false,"title":"势函数系数正则性","source_units":[{"raw_tex":"where $a^{(1)}_t, a^{(2)}_t \\geqslant 0$ and $P_t = \\begin{pmatrix}p^{(11)}_t & p^{(12)}_t \\\\ p^{(12)}_t & p^{(22)}_t \\end{pmatrix}\\succcurlyeq 0$ are continuously differentiable with respect to time. Given a quadratic function $ \\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}$ and an SDE~\\eqref{ODE_mean1} under Polyak-Ruppert averaging, we present in Theorem~\\ref{pr_avg_conv} a  way to control the quantity:\n","start":84791,"end":85195}]},{"id":"S042","semantic_type":"algorithm","kind":"algorithm","labels":[],"member_ids":["S042"],"declared_external":false,"source_only":false,"title":"平均势函数的最坏漂移问题","source_units":[{"raw_tex":"\\begin{equation*}\n    \\begin{aligned}\n      \\max_{f \\in \\mathcal{F}_{0, \\infty}, d \\in \\mathbf{N}, X_t \\in \\mathbf{R}^d} \\ & \\ \\frac{d}{dt}\\mathbf{E}\\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}(X_t, \\bar{X}_t, t), \\\\\n       \\text{subject to} \\ & dX_t = -h_t\\nabla f(X_t)dt + h_t(\\gamma \\Sigma(X_t))^{1/2} dB_t, \\\\\n        & d\\bar{X}_t = \\frac{X_t - \\bar{X}_t}{t}dt.\n    \\end{aligned}\n\\end{equation*}\n","start":85195,"end":85590}]},{"id":"S043","semantic_type":"theorem","kind":"theorem","labels":["pr_avg_conv"],"member_ids":["S043"],"declared_external":false,"source_only":false,"title":"平均随机流的充分 LMI 证书","source_units":[{"raw_tex":"\\begin{theorem}\n\\label{pr_avg_conv}\nLet $h_t \\geqslant 0$, and $a_t \\geqslant 0$, $P_t \\succcurlyeq 0$ be continuously differentiable with respect to time. Let  $f \\in \\mathcal{F}_{0, \\infty}$ be a twice continuously differentiable function and $(X_t, \\bar{X}_t) \\in \\mathbf{R}^d \\times \\mathbf{R}^d$ be a trajectory generated by the SDE under Polyak-Ruppert averaging~\\eqref{ODE_mean1}. Let $\\mathcal{V}$ be a quadratic function~\\eqref{lyap_avg}, such that there exist $\\lambda^{(1)}_t, ..., \\lambda^{(6)}_t \\geqslant 0$ verifying:\n\\begin{equation*}\n    \\begin{aligned}\n&\\begin{pmatrix} \\dot{p}^{(11)}_t + \\frac{2p^{(12)}_t}{t} & \\dot{p}^{(12)}_t + \\frac{p^{(22)}_t - p^{(12)}_t}{t}& \\frac{\\lambda^{(6)}_t + \\lambda^{(4)}_t - 2h_tp^{(11)}_t}{2} & \\frac{a^{(2)}_t}{2t} - \\frac{\\lambda^{(5)}_t}{2}\\\\ \n\n\\dot{p}^{(12)}_t + \\frac{p^{(22)}_t - p^{(12)}_t}{t} & \\dot{p}^{(22)}_t - \\frac{2p^{(22)}_t}{t} & -\\frac{\\lambda^{(6)}_t - 2h_tp^{(12)}_t}{2} & -\\frac{a^{(2)}_t}{2t} + \\frac{\\lambda^{(5)}_t + \\lambda^{(3)}_t}{2}\\\\ \n\n \\frac{\\lambda^{(6)}_t + \\lambda^{(4)}_t - 2h_tp^{(11)}_t}{2} & -\\frac{\\lambda^{(6)}_t - 2h_tp^{(12)}_t}{2} & -a^{(1)}_t & 0 \\\\ \n\n\\frac{a^{(2)}_t}{2t} - \\frac{\\lambda^{(5)}_t}{2} & -\\frac{a^{(2)}_t}{2t} + \\frac{\\lambda^{(5)}_t + \\lambda^{(3)}_t}{2} & 0 &  0\\end{pmatrix} \\preccurlyeq 0, \\\\ \n&\\dot{a}^{(1)}_t + \\lambda^{(1)}_t + \\lambda^{(5)}_t = \\lambda^{(4)}_t + \\lambda^{(6)}_t, \\\\ \n&\\dot{a}^{(2)}_t + \\lambda^{(2)}_t + \\lambda^{(6)}_t = \\lambda^{(3)}_t + \\lambda^{(5)}_t + \\dot{a}^{(1)}_t. \n\\end{aligned}\n\\end{equation*}\nThen, the following inequality is satisfied:\n\\smallbreak \n$\\displaystyle\\frac{d}{dt}\\mathbf{E}\\mathcal{V}(X_t, \\bar{X}_t, t)\\leqslant \\frac{1}{2}\\mathbf{E}{\\rm Tr}\\left((a^{(1)}_t\\nabla_{xx} f(X_t) + 2p^{(11)}_t I_d)\\Sigma(X_t)\\right)h_t^2\\gamma$.\n\\end{theorem}","start":85591,"end":87393}]},{"id":"S044","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S044"],"declared_external":false,"source_only":false,"title":"正文证明指引","source_units":[{"raw_tex":"\\begin{proof}\nThe proof follows the method from Section~\\ref{sec:methodo} (see Appendix~\\ref{sec:A11}).\n\\end{proof}","start":87394,"end":87509}]},{"id":"S045","semantic_type":"proof","kind":"proof","labels":[],"member_ids":["S045"],"declared_external":false,"source_only":false,"title":"附录完整作者证明","source_units":[{"raw_tex":"\\begin{proof}\n\nWe rewrite the SDE into \n\\begin{equation*}\n\\begin{aligned}\n    \\begin{pmatrix} dX_t \\\\ d\\bar{X}_t\\end{pmatrix} =& \\left(\\begin{pmatrix} 0 & 0 \\\\ \\frac{1}{t} & -\\frac{1}{t} \\end{pmatrix}\\otimes I_d\\right)\\begin{pmatrix} X_t \\\\ \\bar{X}_t\\end{pmatrix}dt +\\left(\\begin{pmatrix}- h_t & 0 \\\\ 0  & 0  \\end{pmatrix}\\otimes I_d\\right)\\begin{pmatrix} \\nabla f(X_t) \\\\ \\nabla f(\\bar{X}_t) \\end{pmatrix}dt \\\\\n    &+\\left(\\begin{pmatrix}h_t (\\gamma\\Sigma(X_t))^{1/2} \\\\ 0\\end{pmatrix}\\otimes I_d\\right)dB_t\n\\end{aligned}\n\\end{equation*}\n and denote $Y_t = \\begin{pmatrix} X_t \\\\ \\bar{X}_t \\end{pmatrix}$. \n \\smallbreak\n We consider the quadratic function~\\eqref{lyap_avg} $\\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}(X_t, \\bar{X}_t ,t) = \\mathcal{V}(Y_t,t)$. Applying Ito's formula, its derivative with respect to time is $\\frac{d}{dt} \\mathcal{V}(Y_t,t) = \\frac{\\partial}{\\partial t}\\mathcal{V}(Y_t,t) + \\frac{\\partial}{\\partial y}\\mathcal{V}\\frac{dY_t}{dt} + \\frac{1}{2}\\gamma h^2_t {\\rm Tr}[\\frac{\\partial^2}{\\partial Y_t^2}\\mathcal{V}(Y_t,t)^\\top \\begin{pmatrix}\\Sigma(X_t) \\\\ 0\\end{pmatrix}]$. By taking the expectation of Ito's formula, we have $\\frac{d}{dt} \\mathbf{E}\\mathcal{V}(Y_t,t) = \\mathbf{E}[\\frac{\\partial}{\\partial t}\\mathcal{V}(Y_t,t) + \\frac{\\partial}{\\partial y}\\mathcal{V}\\frac{dY_t}{dt}] +\\frac{1}{2}\\gamma h^2_t \\mathbf{E}{\\rm Tr}(\\frac{\\partial^2}{\\partial Y_t^2}\\mathcal{V}(Y_t,t)^\\top \\begin{pmatrix}\\Sigma(X_t) \\\\ 0\\end{pmatrix})$.\n \\smallbreak\n The variance term $\\frac{1}{2}\\gamma h^2_t \\mathbf{E}{\\rm Tr}(\\frac{\\partial^2}{\\partial Y_t^2}\\mathcal{V}(Y_t,t)^\\top \\begin{pmatrix}\\Sigma(X_t) \\\\ 0\\end{pmatrix})$ depends on the second derivative of $f$ in the space variable $(X_t)$, and on the covariance $\\Sigma_t$. We do not take this term into account in the performance estimation frawework.\n \\smallbreak\n We formulate the problem of verifying a quadratic function as verifying the inequality $\\frac{d}{dt}\\mathbf{E}\\mathcal{V}(X_t, \\bar{X}_t,t) \\leqslant \\frac{1}{2}\\gamma h^2_t \\mathbf{E}{\\rm Tr}(\\frac{\\partial^2}{\\partial X_t^2}\\mathcal{V}(X_t, \\bar{X}_t, t)^\\top \\Sigma(X_t))$ (that is $\\mathbf{E}[\\frac{\\partial}{\\partial t}\\mathcal{V}(Y_t,t) + \\frac{\\partial}{\\partial y}\\mathcal{V}\\frac{dY_t}{dt}] \\leqslant 0$) holds for any twice continuously differentiable function $f \\in \\mathcal{F}_{0, \\infty}$, and any trajectory $(X_t, \\bar{X}_t)$ generated by the SDE under Polyak-Ruppert averaging~\\eqref{ODE_mean1}. \n \\smallbreak\n This problem is equivalent with verifying that $\\frac{d}{dt}\\mathcal{V}(\\tilde{Y}_t,t) \\leqslant 0$ holds for any function twice continuously differentiable $f \\in \\mathcal{F}_{0, \\infty}$ and any trajectory $\\tilde{Y}_t = \\begin{pmatrix} \\tilde{X}_t & \\tilde{\\bar{X}}_t\\end{pmatrix}^\\top $ generated by deterministic gradient flow from the SDE~\\eqref{ODE_mean1} with $\\gamma = 0$. \n\\smallbreak\nWe follow the methodology developed in Section~\\ref{sec:deterministic} for ODEs. We formulate the verification of such a Lyapunov function as the maximization problem\n\\begin{equation*}\n    \\begin{aligned}\n      0\\geqslant \\max_{X_t \\in \\mathbf{R}^d,  d\\in \\mathbf{N}, \\ f\\in \\mathcal{F}_{0, \\infty}} \\ & \\ \\frac{d}{dt}  \\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}(X_t, \\bar{X}_t, t)\\\\\n      \\text{subject to }\n    \\ \\dot{X}_t &= -h_t\\nabla f(\\bar{X}_t), \\\n        \\dot{\\bar{X}}_t = \\frac{X_t - \\bar{X}_t}{t}dt.\n\\end{aligned}\n\\end{equation*}\nThis maximization problem can be formulated as the following SDP program\n\n\\begin{equation*}\n    \\begin{aligned}\n    & \\min_{\\lambda_t^{(i)} \\geqslant 0, \\ i \\in \\{1, ..., 6 \\}} \\ 0 \\, \\\\\n&\\begin{pmatrix} \\dot{p}^{(11)}_t + \\frac{2p^{(12)}_t}{t} & \\dot{p}^{(12)}_t + \\frac{p^{(22)}_t - p^{(12)}_t}{t}& \\frac{\\lambda^{(6)}_t + \\lambda^{(4)}_t - 2h_tp^{(11)}_t}{2} & \\frac{a^{(2)}_t}{2t} - \\frac{\\lambda^{(5)}_t}{2}\\\\ \n\n\\dot{p}^{(12)}_t + \\frac{p^{(22)}_t - p^{(12)}_t}{t} & \\dot{p}^{(22)}_t - \\frac{2p^{(22)}_t}{t} & -\\frac{\\lambda^{(6)}_t - 2h_tp^{(12)}_t}{2} & -\\frac{a^{(2)}_t}{2t} + \\frac{\\lambda^{(5)}_t + \\lambda^{(3)}_t}{2}\\\\ \n\n \\frac{\\lambda^{(6)}_t + \\lambda^{(4)}_t - 2h_tp^{(11)}_t}{2} & -\\frac{\\lambda^{(6)}_t - 2h_tp^{(12)}_t}{2} & -a^{(1)}_t & 0 \\\\ \n\n\\frac{a^{(2)}_t}{2t} - \\frac{\\lambda^{(5)}_t}{2} & -\\frac{a^{(2)}_t}{2t} + \\frac{\\lambda^{(5)}_t + \\lambda^{(3)}_t}{2} & 0 &  0\\end{pmatrix} \\preccurlyeq 0, \\\\ \n&\\dot{a}^{(1)}_t + \\lambda^{(1)}_t + \\lambda^{(5)}_t = \\lambda^{(4)}_t + \\lambda^{(6)}_t, \\\\ \n&\\dot{a}^{(2)}_t + \\lambda^{(2)}_t + \\lambda^{(6)}_t = \\lambda^{(3)}_t + \\lambda^{(5)}_t + \\dot{a}^{(1)}_t. \n\\end{aligned}\n\\end{equation*}\nNote that the Gram matrix $G$ and function value vector $F$ to be introduced are different from the setting explored in Section~\\ref{sec:deterministic}, more precisely $G = Q^\\top Q \\succcurlyeq 0, \\ Q = [X_t - x_\\star, \\bar{X}_t - x_\\star, g_t, \\bar{g}_t]$ and $F = (f(X_t), f(\\bar{X}_t), f_\\star)$.\n\nThe final inequality follows from Ito's formula.\n\\end{proof}","start":111630,"end":116544}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S046"],"declared_external":false,"source_only":false,"title":"增广状态与块 SDE","source_units":[{"raw_tex":"\\begin{proof}\n\nWe rewrite the SDE into \n\\begin{equation*}\n\\begin{aligned}\n    \\begin{pmatrix} dX_t \\\\ d\\bar{X}_t\\end{pmatrix} =& \\left(\\begin{pmatrix} 0 & 0 \\\\ \\frac{1}{t} & -\\frac{1}{t} \\end{pmatrix}\\otimes I_d\\right)\\begin{pmatrix} X_t \\\\ \\bar{X}_t\\end{pmatrix}dt +\\left(\\begin{pmatrix}- h_t & 0 \\\\ 0  & 0  \\end{pmatrix}\\otimes I_d\\right)\\begin{pmatrix} \\nabla f(X_t) \\\\ \\nabla f(\\bar{X}_t) \\end{pmatrix}dt \\\\\n    &+\\left(\\begin{pmatrix}h_t (\\gamma\\Sigma(X_t))^{1/2} \\\\ 0\\end{pmatrix}\\otimes I_d\\right)dB_t\n\\end{aligned}\n\\end{equation*}\n and denote $Y_t = \\begin{pmatrix} X_t \\\\ \\bar{X}_t \\end{pmatrix}$. \n \\smallbreak\n","start":111630,"end":112251}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S047"],"declared_external":false,"source_only":false,"title":"应用 Itô 并取期望","source_units":[{"raw_tex":" We consider the quadratic function~\\eqref{lyap_avg} $\\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}(X_t, \\bar{X}_t ,t) = \\mathcal{V}(Y_t,t)$. Applying Ito's formula, its derivative with respect to time is $\\frac{d}{dt} \\mathcal{V}(Y_t,t) = \\frac{\\partial}{\\partial t}\\mathcal{V}(Y_t,t) + \\frac{\\partial}{\\partial y}\\mathcal{V}\\frac{dY_t}{dt} + \\frac{1}{2}\\gamma h^2_t {\\rm Tr}[\\frac{\\partial^2}{\\partial Y_t^2}\\mathcal{V}(Y_t,t)^\\top \\begin{pmatrix}\\Sigma(X_t) \\\\ 0\\end{pmatrix}]$. By taking the expectation of Ito's formula, we have $\\frac{d}{dt} \\mathbf{E}\\mathcal{V}(Y_t,t) = \\mathbf{E}[\\frac{\\partial}{\\partial t}\\mathcal{V}(Y_t,t) + \\frac{\\partial}{\\partial y}\\mathcal{V}\\frac{dY_t}{dt}] +\\frac{1}{2}\\gamma h^2_t \\mathbf{E}{\\rm Tr}(\\frac{\\partial^2}{\\partial Y_t^2}\\mathcal{V}(Y_t,t)^\\top \\begin{pmatrix}\\Sigma(X_t) \\\\ 0\\end{pmatrix})$.\n \\smallbreak\n","start":112251,"end":113100}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S048"],"declared_external":false,"source_only":false,"title":"将方差项留在证书外","source_units":[{"raw_tex":" The variance term $\\frac{1}{2}\\gamma h^2_t \\mathbf{E}{\\rm Tr}(\\frac{\\partial^2}{\\partial Y_t^2}\\mathcal{V}(Y_t,t)^\\top \\begin{pmatrix}\\Sigma(X_t) \\\\ 0\\end{pmatrix})$ depends on the second derivative of $f$ in the space variable $(X_t)$, and on the covariance $\\Sigma_t$. We do not take this term into account in the performance estimation frawework.\n \\smallbreak\n","start":113100,"end":113464}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S049"],"declared_external":false,"source_only":false,"title":"归约为非正一阶漂移","source_units":[{"raw_tex":" We formulate the problem of verifying a quadratic function as verifying the inequality $\\frac{d}{dt}\\mathbf{E}\\mathcal{V}(X_t, \\bar{X}_t,t) \\leqslant \\frac{1}{2}\\gamma h^2_t \\mathbf{E}{\\rm Tr}(\\frac{\\partial^2}{\\partial X_t^2}\\mathcal{V}(X_t, \\bar{X}_t, t)^\\top \\Sigma(X_t))$ (that is $\\mathbf{E}[\\frac{\\partial}{\\partial t}\\mathcal{V}(Y_t,t) + \\frac{\\partial}{\\partial y}\\mathcal{V}\\frac{dY_t}{dt}] \\leqslant 0$) holds for any twice continuously differentiable function $f \\in \\mathcal{F}_{0, \\infty}$, and any trajectory $(X_t, \\bar{X}_t)$ generated by the SDE under Polyak-Ruppert averaging~\\eqref{ODE_mean1}. \n \\smallbreak\n","start":113464,"end":114092}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S050"],"declared_external":false,"source_only":false,"title":"转向零噪声确定性验证","source_units":[{"raw_tex":" This problem is equivalent with verifying that $\\frac{d}{dt}\\mathcal{V}(\\tilde{Y}_t,t) \\leqslant 0$ holds for any function twice continuously differentiable $f \\in \\mathcal{F}_{0, \\infty}$ and any trajectory $\\tilde{Y}_t = \\begin{pmatrix} \\tilde{X}_t & \\tilde{\\bar{X}}_t\\end{pmatrix}^\\top $ generated by deterministic gradient flow from the SDE~\\eqref{ODE_mean1} with $\\gamma = 0$. \n\\smallbreak\n","start":114092,"end":114488}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S051"],"declared_external":false,"source_only":false,"title":"确定性最坏导数表述","source_units":[{"raw_tex":"We follow the methodology developed in Section~\\ref{sec:deterministic} for ODEs. We formulate the verification of such a Lyapunov function as the maximization problem\n\\begin{equation*}\n    \\begin{aligned}\n      0\\geqslant \\max_{X_t \\in \\mathbf{R}^d,  d\\in \\mathbf{N}, \\ f\\in \\mathcal{F}_{0, \\infty}} \\ & \\ \\frac{d}{dt}  \\mathcal{V}_{a^{(1)}_t, a^{(2)}_t, P_t}(X_t, \\bar{X}_t, t)\\\\\n      \\text{subject to }\n    \\ \\dot{X}_t &= -h_t\\nabla f(\\bar{X}_t), \\\n        \\dot{\\bar{X}}_t = \\frac{X_t - \\bar{X}_t}{t}dt.\n\\end{aligned}\n\\end{equation*}\n","start":114488,"end":115025}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S052"],"declared_external":false,"source_only":false,"title":"给出六乘子 LMI","source_units":[{"raw_tex":"This maximization problem can be formulated as the following SDP program\n\n\\begin{equation*}\n    \\begin{aligned}\n    & \\min_{\\lambda_t^{(i)} \\geqslant 0, \\ i \\in \\{1, ..., 6 \\}} \\ 0 \\, \\\\\n&\\begin{pmatrix} \\dot{p}^{(11)}_t + \\frac{2p^{(12)}_t}{t} & \\dot{p}^{(12)}_t + \\frac{p^{(22)}_t - p^{(12)}_t}{t}& \\frac{\\lambda^{(6)}_t + \\lambda^{(4)}_t - 2h_tp^{(11)}_t}{2} & \\frac{a^{(2)}_t}{2t} - \\frac{\\lambda^{(5)}_t}{2}\\\\ \n\n\\dot{p}^{(12)}_t + \\frac{p^{(22)}_t - p^{(12)}_t}{t} & \\dot{p}^{(22)}_t - \\frac{2p^{(22)}_t}{t} & -\\frac{\\lambda^{(6)}_t - 2h_tp^{(12)}_t}{2} & -\\frac{a^{(2)}_t}{2t} + \\frac{\\lambda^{(5)}_t + \\lambda^{(3)}_t}{2}\\\\ \n\n \\frac{\\lambda^{(6)}_t + \\lambda^{(4)}_t - 2h_tp^{(11)}_t}{2} & -\\frac{\\lambda^{(6)}_t - 2h_tp^{(12)}_t}{2} & -a^{(1)}_t & 0 \\\\ \n\n\\frac{a^{(2)}_t}{2t} - \\frac{\\lambda^{(5)}_t}{2} & -\\frac{a^{(2)}_t}{2t} + \\frac{\\lambda^{(5)}_t + \\lambda^{(3)}_t}{2} & 0 &  0\\end{pmatrix} \\preccurlyeq 0, \\\\ \n&\\dot{a}^{(1)}_t + \\lambda^{(1)}_t + \\lambda^{(5)}_t = \\lambda^{(4)}_t + \\lambda^{(6)}_t, \\\\ \n&\\dot{a}^{(2)}_t + \\lambda^{(2)}_t + \\lambda^{(6)}_t = \\lambda^{(3)}_t + \\lambda^{(5)}_t + \\dot{a}^{(1)}_t. \n\\end{aligned}\n\\end{equation*}\n","start":115025,"end":116182}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S053"],"declared_external":false,"source_only":false,"title":"指定四向量 Gram 数据","source_units":[{"raw_tex":"Note that the Gram matrix $G$ and function value vector $F$ to be introduced are different from the setting explored in Section~\\ref{sec:deterministic}, more precisely $G = Q^\\top Q \\succcurlyeq 0, \\ Q = [X_t - x_\\star, \\bar{X}_t - x_\\star, g_t, \\bar{g}_t]$ and $F = (f(X_t), f(\\bar{X}_t), f_\\star)$.\n\n","start":116182,"end":116484}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","labels":[],"member_ids":["S054"],"declared_external":false,"source_only":false,"title":"以 Itô 公式收束","source_units":[{"raw_tex":"The final inequality follows from Ito's formula.\n\\end{proof}","start":116484,"end":116544}]},{"id":"S055","semantic_type":"external_fact","kind":"external_fact","labels":[],"member_ids":["S055"],"declared_external":true,"declaration_text":["implicit standard fact made explicit by annotator；用于解释原文第1237行“取 Itô 公式的期望”所省略的条件性背景。"],"source_only":false,"title":"取期望的条件性规则","source_units":[]}],"collapse_map":{"S001":"S001","S002":"S002","S003":"S003","S004":"S004","S005":"S005","S006":"S006","S008":"S008","S009":"S009","S010":"S010","S011":"S011","S012":"S012","S013":"S013","S014":"S014","S015":"S015","S016":"S016","S017":"S017","S018":"S018","S019":"S019","S020":"S020","S021":"S021","S022":"S022","S023":"S023","S024":"S024","S025":"S025","S026":"S026","S027":"S027","S028":"S028","S029":"S029","S030":"S030","S032":"S032","S033":"S033","S034":"S034","S035":"S035","S036":"S036","S037":"S037","S038":"S038","S039":"S039","S040":"S040","S041":"S041","S042":"S042","S043":"S043","S044":"S044","S045":"S045","S046":"S046","S047":"S047","S048":"S048","S049":"S049","S050":"S050","S051":"S051","S052":"S052","S053":"S053","S054":"S054","S055":"S055"}}}}
{"record_id":"1706.05681v2","paper":{"title":"On the convergence of mirror descent beyond stochastic convex programming","authors":["Zhengyuan Zhou","Panayotis Mertikopoulos","Nicholas Bambos","Stephen Boyd","Peter Glynn"],"arxiv_id":"1706.05681","version":"v2","journal":"SIAM Journal on Optimization","doi":"10.1137/17m1134925","abstract_url":"https://arxiv.org/abs/1706.05681v2","pdf_url":"https://arxiv.org/pdf/1706.05681v2"},"context":{"raw_tex":"Let $\\states$ be a convex compact subset of a $\\vdim$-di\\-men\\-sio\\-nal vector space $\\vecspace$ with norm $\\norm{\\argdot}$.\n%and let $\\sample$ be a random variable on an underlying (complete) probability space $\\pspace$\nThroughout this paper, we will focus on stochastic optimization problems of the general form\n\\begin{equation}\n\\label{eq:opt}\n\\tag{Opt}\n\\begin{aligned}\n\\text{minimize}\n&\\quad\n\\obj(\\state),\n\\\\\n\\text{subject to}\n&\\quad\n\\state\\in\\states,\n\\end{aligned}\n\\end{equation}\n%where the objective function $\\obj\\from\\states\\to\\R$ is of the form\nwhere\n\\begin{equation}\n\\label{eq:obj}\n\\obj(\\state)\n\t= \\exof{\\sobj(\\state;\\sample)}\n\\end{equation}\nfor some stochastic objective function $\\sobj\\from\\states\\times\\samples\\to\\R$ defined on an underlying (complete) probability space $\\pspace$.\n\n\n\\begin{assumption}\n\\label{asm:diff}\n$\\sobj(\\state, \\sample)$ is continuously differentiable in $\\state$ for almost all $\\sample\\in\\samples$.\n\\end{assumption}\n\n\\begin{assumption}\n\\label{asm:L2}\nThe gradient of $\\sobj$ is uniformly bounded in $L^{2}$, \\ie $\\exof{\\dnorm{\\nabla\\sobj(\\state;\\sample)}^{2}} \\leq \\vbound^{2}$ for some finite $\\vbound\\geq0$ and all $\\state\\in\\states$.\n\\end{assumption}\n\n\\begin{remark*}\nIn the above, gradients are treated as elements of the dual space $\\dstates\\equiv\\dual\\vecspace$ of $\\vecspace$, and $\\dnorm{\\dstate} = \\sup\\setdef{\\braket{\\dstate}{\\state}}{\\norm{\\state}\\leq1}$ denotes the dual norm of $\\dstate\\in\\dstates$.\nWe also note that $\\nabla\\sobj(\\state;\\sample)$ refers to the gradient of $\\sobj(\\state;\\sample)$ with respect to $\\state$;\nsince $\\samples$ is not assumed to carry a differential structure, there is no danger of confusion.\n\\end{remark*}\n\n\\cref{asm:L2} is also standard in the stochastic optimization literature:\nit holds trivially if $\\sobj$ is uniformly Lipschitz (another commonly used condition) and, by the dominated convergence theorem, it further implies that $\\obj$ is smooth and $\\nabla\\obj(\\state) = \\nabla\\exof{\\sobj(\\state;\\sample)} = \\exof{\\nabla\\sobj(\\state;\\sample)}$ is bounded.\nAs a result, the solution set\n\\begin{equation}\n\\label{eq:solset}\n\\sols\n\t= \\argmin\\obj\n\\end{equation}\nof \\eqref{eq:opt} is closed and nonempty (by the compactness of $\\states$ and the continuity of $\\obj$).\n\n\n\\begin{definition}\n\\label{def:VC}\nWe say that \\eqref{eq:opt} is \\emph{variationally coherent} if\n\\begin{equation}\n\\label{eq:VC}\n\\tag{VC}\n\\braket{\\nabla\\obj(\\state)}{\\state - \\sol}\n\t\\geq 0\n\t\\quad\n\t\\text{for all $\\state\\in\\states$, $\\sol\\in\\sols$},\n\\end{equation}\nand there exists some $\\sol\\in\\sols$ such that equality holds in \\eqref{eq:VC} only if $\\state \\in \\sols$.\n\\end{definition}\n\nThe main idea of the method is as follows:\nAt each iteration, the algorithm takes as input an \\ac{iid} sample of the gradient of $\\sobj$ at the algorithm's current state.\nSubsequently, the method takes a step along this stochastic gradient in the dual space $\\dstates\\equiv\\dspace$ of $\\vecspace$ (where gradients live), the result is ``mirrored'' back to the problem's feasible region $\\states$, and the process repeats.\nFormally, this gives rise to the recursion\n\\begin{equation}\n\\label{eq:SMD}\n\\tag{SMD}\n\\begin{aligned}\n\\act_{\\run}\n\t&= \\mirror(\\score_{\\run})\n\t\\\\\n\\score_{\\run+1}\n\t&= \\score_{\\run} - \\step_{\\run} \\nabla\\sobj(\\act_{\\run};\\sample_{\\run})\n\\end{aligned}\n\\end{equation}\nwhere:\n\\begin{enumerate}\n\\item\n$\\run=\\running$ denotes the algorithm's running counter.\n\\item\n$\\score_{\\run}\\in\\dstates$ is a score variable that aggregates gradient steps up to stage $\\run$.\n\\item\n$\\mirror\\from\\dstates\\to\\states$ is the \\emph{mirror map} that outputs a solution candidate $\\act_{\\run}\\in\\states$ as a function of the score variable $\\score_{\\run}\\in\\dspace$.\n\\item\n$\\sample_{\\run}\\in\\samples$ is a sequence of \\ac{iid} samples.%\n\\footnote{The indexing convention for $\\sample_{\\run}$ means that $\\score_{\\run}$ and $\\act_{\\run}$ are \\emph{predictable} relative to the natural filtration $\\filter_{\\run} = \\sigma(\\sample_{1},\\dotsc,\\sample_{\\run})$ of $\\sample_{\\run}$, \\ie $\\score_{\\run+1}$ and $\\act_{\\run+1}$ are both $\\filter_{\\run}$-measurable.\nTo this history, we also attach the trivial $\\sigma$-algebra as $\\filter_{0}$ for completeness.}\n\\item\n$\\step_{\\run}>0$ is the algorithm's step-size sequence, assumed in what follows to satisfy the Robbins\\textendash Monro summability condition\n\\begin{equation}\n\\label{eq:L2L1}\n\\sum_{\\run=1}^{\\infty} \\step_{\\run}^{2}\n\t<\\infty\n\t\\quad\n\t\\text{and}\n\t\\quad\n\t\\sum_{\\run=1}^{\\infty} \\step_{\\run}\n\t= \\infty.\n\\end{equation}\n\\end{enumerate}\n\n\n\\begin{algorithm}[tbp]\n\\caption{Stochastic mirror descent (\\ac{SMD})}\n\\label{alg:SMD}\n%----------------------------------------------------------------------\n%%% SMD ALGORITHM\n%----------------------------------------------------------------------\n% !TEX root = ../Main.tex\n\n\n\\tt\n\\begin{algorithmic}[1]\n\\Require\n\tmirror map $\\mirror\\from\\dstates\\to\\states$;\n\tstep-size sequence $\\step_{\\run}>0$\n\\State\n\tchoose $\\score\\in\\dstates \\equiv \\dspace$\n\t\\Comment{initialization}\n\\For{$\\run=\\running$}\n%\t\\State $\\act_{\\run} = \\mirror(\\score_{\\run-1})$\n%\t\\State $\\score_{\\run} = \\score_{\\run-1} - \\step_{\\run} \\nabla\\sobj(\\act_{\\run},\\sample_{\\run})$\n%\tat each stage $\\run=\\running$\n\t\\State\n\t\tset $\\act \\leftarrow \\mirror(\\score)$\n\t\t\\Comment{set state}\n\t\\State\n\t\tdraw $\\sample\\in\\samples$\n\t\t\\Comment{gradient sample}\n\t\\State\n\t\tget $\\est\\payv = -\\nabla\\sobj(\\act;\\sample)$\n\t\t\\Comment{get oracle feedback}\n\t\\State\n\t\tset $\\score \\leftarrow \\score + \\step_{\\run} \\est\\payv$\n\t\t\\Comment{update score variable}\n\\EndFor\n\\\\\n\\Return $\\act$\n\t\\Comment{output}\n\\end{algorithmic}\n\\end{algorithm}\n\nIn more detail, the algorithm's mirror map $\\mirror\\from\\dstates\\to\\states$ is defined as\n\\begin{equation}\n\\label{eq:mirror}\n\\mirror(\\dstate)\n\t= \\argmax_{\\state\\in\\states} \\{\\braket{\\dstate}{\\state} - \\hreg(\\state)\\},\n\\end{equation}\nwhere the \\emph{regularizer} (or \\emph{penalty function}) $\\hreg\\from\\states\\to\\R$ is assumed to be continuous and strongly convex on $\\states$, \\ie there exists some $\\strong>0$ such that\n\\begin{equation}\n\\label{eq:strong}\n\\hreg(\\tau\\state + (1-\\tau)\\alt\\state)\n\t\\leq \\tau \\hreg(\\state) + (1-\\tau) \\hreg(\\alt\\state) - \\tfrac{1}{2} \\strong\\tau(1-\\tau) \\norm{\\alt\\state - \\state}^{2},\n\\end{equation}\nfor all $\\state,\\alt\\state\\in\\states$ and all $\\tau\\in[0,1]$.\nThe mapping $\\mirror\\from\\dspace\\to\\states$ defined by \\eqref{eq:mirror} is then called the \\emph{mirror map induced by $\\hreg$}.\n\n\n\\begin{definition}\n\\label{def:Fenchel}\nLet $h\\from\\states \\to \\R$ be a regularizer on $\\states$.\nThe induced \\emph{Fenchel coupling} $\\fench(\\base,\\dstate)$ between a base-point $\\base\\in\\states$ and a dual vector $\\dstate\\in\\dstates$ is defined as\n\\begin{equation}\n\\label{eq:Fenchel}\n\\fench(\\base,\\dstate)\n\t= \\hreg(\\base) + \\hreg^{\\ast}(\\dstate) - \\braket{\\dstate}{\\base}\n\\end{equation}\nwhere $\\hreg^{\\ast}(\\dstate) = \\max_{\\state\\in\\states} \\{ \\braket{\\dstate}{\\state} - \\hreg(\\state) \\}$ denotes the convex conjugate of $h$.\n\\end{definition}\n\n\\begin{assumption}\n\\label{asm:reciprocity}\n$\\fench(\\base,\\dstate_{\\run}) \\to 0$ whenever $\\mirror(\\dstate_{\\run})\\to\\base$.\n\\end{assumption}\n\n\\begin{definition}\n\\label{def:metrics}\n%Let $h$ be a regularizer on $\\states$ and\nLet $\\region$ be a subset of $\\states$.\n\\begin{subequations}\n\\begin{enumerate}\n\\item\nThe \\emph{distance} between $\\region$ and $\\state\\in\\states$ is defined as $\\dist(\\region,\\state) = \\inf_{\\alt\\state\\in\\region} \\norm{\\alt\\state - \\state}$,\nand the corresponding \\emph{$\\eps$-neighborhood of $\\region$} is\n\\begin{alignat}{2}\n\\label{eq:ball-norm}\n\\ball(\\region,\\eps)\n\t&= \\setdef{\\state\\in\\states}{\\dist(\\region,\\state) < \\eps}.\n\\intertext{%\n\\item\nThe (setwise) \\emph{Fenchel coupling} between $\\region$ and $\\dstate\\in\\dstates$ is defined as $\\fench(\\region,\\dstate) = \\inf_{\\state\\in\\region} \\fench(\\state,\\dstate)$,\nand the corresponding \\emph{Fenchel $\\delta$-zone} of $\\region$ under $h$ is}\n\\label{eq:ball-Fench}\n\\fball(\\region,\\delta)\n\t&= \\setdef{\\state\\in\\states}{\\text{$\\state=\\mirror(\\dstate)$ for some $\\dstate\\in\\dstates$ with $\\fench(\\region,\\dstate) < \\delta$}}.\n\\end{alignat}\n\\end{enumerate}\n\\end{subequations}\n\\end{definition}","units":[{"start":42684,"end":43478,"start_line":920,"end_line":943,"raw_tex":"Let $\\states$ be a convex compact subset of a $\\vdim$-di\\-men\\-sio\\-nal vector space $\\vecspace$ with norm $\\norm{\\argdot}$.\n%and let $\\sample$ be a random variable on an underlying (complete) probability space $\\pspace$\nThroughout this paper, we will focus on stochastic optimization problems of the general form\n\\begin{equation}\n\\label{eq:opt}\n\\tag{Opt}\n\\begin{aligned}\n\\text{minimize}\n&\\quad\n\\obj(\\state),\n\\\\\n\\text{subject to}\n&\\quad\n\\state\\in\\states,\n\\end{aligned}\n\\end{equation}\n%where the objective function $\\obj\\from\\states\\to\\R$ is of the form\nwhere\n\\begin{equation}\n\\label{eq:obj}\n\\obj(\\state)\n\t= \\exof{\\sobj(\\state;\\sample)}\n\\end{equation}\nfor some stochastic objective function $\\sobj\\from\\states\\times\\samples\\to\\R$ defined on an underlying (complete) probability space $\\pspace$.\n","sha256_utf8":"a50156b9158f6c645de2a1d79b23eaa52ec4364c1434c745da99f23a7752d3a4"},{"start":43566,"end":43723,"start_line":946,"end_line":949,"raw_tex":"\\begin{assumption}\n\\label{asm:diff}\n$\\sobj(\\state, \\sample)$ is continuously differentiable in $\\state$ for almost all $\\sample\\in\\samples$.\n\\end{assumption}","sha256_utf8":"ce0bf661b27fe331b84dbc963dbe768954e7cfa553ab2d33083d3acdf5237bd8"},{"start":43725,"end":43961,"start_line":951,"end_line":954,"raw_tex":"\\begin{assumption}\n\\label{asm:L2}\nThe gradient of $\\sobj$ is uniformly bounded in $L^{2}$, \\ie $\\exof{\\dnorm{\\nabla\\sobj(\\state;\\sample)}^{2}} \\leq \\vbound^{2}$ for some finite $\\vbound\\geq0$ and all $\\state\\in\\states$.\n\\end{assumption}","sha256_utf8":"dac713086aee94d84c6336762a457d0e21431122e746b4241dcbde7d2d829690"},{"start":43976,"end":44471,"start_line":957,"end_line":961,"raw_tex":"\\begin{remark*}\nIn the above, gradients are treated as elements of the dual space $\\dstates\\equiv\\dual\\vecspace$ of $\\vecspace$, and $\\dnorm{\\dstate} = \\sup\\setdef{\\braket{\\dstate}{\\state}}{\\norm{\\state}\\leq1}$ denotes the dual norm of $\\dstate\\in\\dstates$.\nWe also note that $\\nabla\\sobj(\\state;\\sample)$ refers to the gradient of $\\sobj(\\state;\\sample)$ with respect to $\\state$;\nsince $\\samples$ is not assumed to carry a differential structure, there is no danger of confusion.\n\\end{remark*}","sha256_utf8":"b4fc8807952a53682934f8e7cebccdfec1926bde41e65a41217b5b3d193d51c5"},{"start":44768,"end":45330,"start_line":966,"end_line":974,"raw_tex":"\\cref{asm:L2} is also standard in the stochastic optimization literature:\nit holds trivially if $\\sobj$ is uniformly Lipschitz (another commonly used condition) and, by the dominated convergence theorem, it further implies that $\\obj$ is smooth and $\\nabla\\obj(\\state) = \\nabla\\exof{\\sobj(\\state;\\sample)} = \\exof{\\nabla\\sobj(\\state;\\sample)}$ is bounded.\nAs a result, the solution set\n\\begin{equation}\n\\label{eq:solset}\n\\sols\n\t= \\argmin\\obj\n\\end{equation}\nof \\eqref{eq:opt} is closed and nonempty (by the compactness of $\\states$ and the continuity of $\\obj$).\n","sha256_utf8":"0d2a1f6b5d2e36b8e98cfdc80a451646291ee0f58c527ce1eacc7106b8dab45c"},{"start":46920,"end":47305,"start_line":1015,"end_line":1027,"raw_tex":"\\begin{definition}\n\\label{def:VC}\nWe say that \\eqref{eq:opt} is \\emph{variationally coherent} if\n\\begin{equation}\n\\label{eq:VC}\n\\tag{VC}\n\\braket{\\nabla\\obj(\\state)}{\\state - \\sol}\n\t\\geq 0\n\t\\quad\n\t\\text{for all $\\state\\in\\states$, $\\sol\\in\\sols$},\n\\end{equation}\nand there exists some $\\sol\\in\\sols$ such that equality holds in \\eqref{eq:VC} only if $\\state \\in \\sols$.\n\\end{definition}","sha256_utf8":"fbd7baeeaa7eeb60770e9da2e6a49c16a0d47b1c4a7f0ee177b6d180b08f4e10"},{"start":56232,"end":58115,"start_line":1194,"end_line":1233,"raw_tex":"The main idea of the method is as follows:\nAt each iteration, the algorithm takes as input an \\ac{iid} sample of the gradient of $\\sobj$ at the algorithm's current state.\nSubsequently, the method takes a step along this stochastic gradient in the dual space $\\dstates\\equiv\\dspace$ of $\\vecspace$ (where gradients live), the result is ``mirrored'' back to the problem's feasible region $\\states$, and the process repeats.\nFormally, this gives rise to the recursion\n\\begin{equation}\n\\label{eq:SMD}\n\\tag{SMD}\n\\begin{aligned}\n\\act_{\\run}\n\t&= \\mirror(\\score_{\\run})\n\t\\\\\n\\score_{\\run+1}\n\t&= \\score_{\\run} - \\step_{\\run} \\nabla\\sobj(\\act_{\\run};\\sample_{\\run})\n\\end{aligned}\n\\end{equation}\nwhere:\n\\begin{enumerate}\n\\item\n$\\run=\\running$ denotes the algorithm's running counter.\n\\item\n$\\score_{\\run}\\in\\dstates$ is a score variable that aggregates gradient steps up to stage $\\run$.\n\\item\n$\\mirror\\from\\dstates\\to\\states$ is the \\emph{mirror map} that outputs a solution candidate $\\act_{\\run}\\in\\states$ as a function of the score variable $\\score_{\\run}\\in\\dspace$.\n\\item\n$\\sample_{\\run}\\in\\samples$ is a sequence of \\ac{iid} samples.%\n\\footnote{The indexing convention for $\\sample_{\\run}$ means that $\\score_{\\run}$ and $\\act_{\\run}$ are \\emph{predictable} relative to the natural filtration $\\filter_{\\run} = \\sigma(\\sample_{1},\\dotsc,\\sample_{\\run})$ of $\\sample_{\\run}$, \\ie $\\score_{\\run+1}$ and $\\act_{\\run+1}$ are both $\\filter_{\\run}$-measurable.\nTo this history, we also attach the trivial $\\sigma$-algebra as $\\filter_{0}$ for completeness.}\n\\item\n$\\step_{\\run}>0$ is the algorithm's step-size sequence, assumed in what follows to satisfy the Robbins\\textendash Monro summability condition\n\\begin{equation}\n\\label{eq:L2L1}\n\\sum_{\\run=1}^{\\infty} \\step_{\\run}^{2}\n\t<\\infty\n\t\\quad\n\t\\text{and}\n\t\\quad\n\t\\sum_{\\run=1}^{\\infty} \\step_{\\run}\n\t= \\infty.\n\\end{equation}\n\\end{enumerate}\n","sha256_utf8":"0f6ac13b1131eeb19e3ef6b48bf29a364b874cabce797f2cd80e703d3d3323f2"},{"start":58360,"end":59431,"start_line":1241,"end_line":1279,"raw_tex":"\\begin{algorithm}[tbp]\n\\caption{Stochastic mirror descent (\\ac{SMD})}\n\\label{alg:SMD}\n%----------------------------------------------------------------------\n%%% SMD ALGORITHM\n%----------------------------------------------------------------------\n% !TEX root = ../Main.tex\n\n\n\\tt\n\\begin{algorithmic}[1]\n\\Require\n\tmirror map $\\mirror\\from\\dstates\\to\\states$;\n\tstep-size sequence $\\step_{\\run}>0$\n\\State\n\tchoose $\\score\\in\\dstates \\equiv \\dspace$\n\t\\Comment{initialization}\n\\For{$\\run=\\running$}\n%\t\\State $\\act_{\\run} = \\mirror(\\score_{\\run-1})$\n%\t\\State $\\score_{\\run} = \\score_{\\run-1} - \\step_{\\run} \\nabla\\sobj(\\act_{\\run},\\sample_{\\run})$\n%\tat each stage $\\run=\\running$\n\t\\State\n\t\tset $\\act \\leftarrow \\mirror(\\score)$\n\t\t\\Comment{set state}\n\t\\State\n\t\tdraw $\\sample\\in\\samples$\n\t\t\\Comment{gradient sample}\n\t\\State\n\t\tget $\\est\\payv = -\\nabla\\sobj(\\act;\\sample)$\n\t\t\\Comment{get oracle feedback}\n\t\\State\n\t\tset $\\score \\leftarrow \\score + \\step_{\\run} \\est\\payv$\n\t\t\\Comment{update score variable}\n\\EndFor\n\\\\\n\\Return $\\act$\n\t\\Comment{output}\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"3f1cc338409cd7e9c606d0699f78711d2998329da62334b3100b91a2eec224ff"},{"start":63585,"end":64409,"start_line":1387,"end_line":1400,"raw_tex":"In more detail, the algorithm's mirror map $\\mirror\\from\\dstates\\to\\states$ is defined as\n\\begin{equation}\n\\label{eq:mirror}\n\\mirror(\\dstate)\n\t= \\argmax_{\\state\\in\\states} \\{\\braket{\\dstate}{\\state} - \\hreg(\\state)\\},\n\\end{equation}\nwhere the \\emph{regularizer} (or \\emph{penalty function}) $\\hreg\\from\\states\\to\\R$ is assumed to be continuous and strongly convex on $\\states$, \\ie there exists some $\\strong>0$ such that\n\\begin{equation}\n\\label{eq:strong}\n\\hreg(\\tau\\state + (1-\\tau)\\alt\\state)\n\t\\leq \\tau \\hreg(\\state) + (1-\\tau) \\hreg(\\alt\\state) - \\tfrac{1}{2} \\strong\\tau(1-\\tau) \\norm{\\alt\\state - \\state}^{2},\n\\end{equation}\nfor all $\\state,\\alt\\state\\in\\states$ and all $\\tau\\in[0,1]$.\nThe mapping $\\mirror\\from\\dspace\\to\\states$ defined by \\eqref{eq:mirror} is then called the \\emph{mirror map induced by $\\hreg$}.\n","sha256_utf8":"ef32bab0f141b0c17ab2ff96c028ccf9c3d0f947c6dd0b7805bf58a1f59c3e5c"},{"start":70370,"end":70914,"start_line":1501,"end_line":1511,"raw_tex":"\\begin{definition}\n\\label{def:Fenchel}\nLet $h\\from\\states \\to \\R$ be a regularizer on $\\states$.\nThe induced \\emph{Fenchel coupling} $\\fench(\\base,\\dstate)$ between a base-point $\\base\\in\\states$ and a dual vector $\\dstate\\in\\dstates$ is defined as\n\\begin{equation}\n\\label{eq:Fenchel}\n\\fench(\\base,\\dstate)\n\t= \\hreg(\\base) + \\hreg^{\\ast}(\\dstate) - \\braket{\\dstate}{\\base}\n\\end{equation}\nwhere $\\hreg^{\\ast}(\\dstate) = \\max_{\\state\\in\\states} \\{ \\braket{\\dstate}{\\state} - \\hreg(\\state) \\}$ denotes the convex conjugate of $h$.\n\\end{definition}","sha256_utf8":"54236f05ae1d328daaccb9cfe4775a61116eea18b6e24d27f07fab9cd2401b71"},{"start":72798,"end":72938,"start_line":1549,"end_line":1552,"raw_tex":"\\begin{assumption}\n\\label{asm:reciprocity}\n$\\fench(\\base,\\dstate_{\\run}) \\to 0$ whenever $\\mirror(\\dstate_{\\run})\\to\\base$.\n\\end{assumption}","sha256_utf8":"98e047832d10e3eeaa2826a7ad0ea3c675d86851597d1c199f73e548d51d77ff"},{"start":73884,"end":74907,"start_line":1570,"end_line":1593,"raw_tex":"\\begin{definition}\n\\label{def:metrics}\n%Let $h$ be a regularizer on $\\states$ and\nLet $\\region$ be a subset of $\\states$.\n\\begin{subequations}\n\\begin{enumerate}\n\\item\nThe \\emph{distance} between $\\region$ and $\\state\\in\\states$ is defined as $\\dist(\\region,\\state) = \\inf_{\\alt\\state\\in\\region} \\norm{\\alt\\state - \\state}$,\nand the corresponding \\emph{$\\eps$-neighborhood of $\\region$} is\n\\begin{alignat}{2}\n\\label{eq:ball-norm}\n\\ball(\\region,\\eps)\n\t&= \\setdef{\\state\\in\\states}{\\dist(\\region,\\state) < \\eps}.\n\\intertext{%\n\\item\nThe (setwise) \\emph{Fenchel coupling} between $\\region$ and $\\dstate\\in\\dstates$ is defined as $\\fench(\\region,\\dstate) = \\inf_{\\state\\in\\region} \\fench(\\state,\\dstate)$,\nand the corresponding \\emph{Fenchel $\\delta$-zone} of $\\region$ under $h$ is}\n\\label{eq:ball-Fench}\n\\fball(\\region,\\delta)\n\t&= \\setdef{\\state\\in\\states}{\\text{$\\state=\\mirror(\\dstate)$ for some $\\dstate\\in\\dstates$ with $\\fench(\\region,\\dstate) < \\delta$}}.\n\\end{alignat}\n\\end{enumerate}\n\\end{subequations}\n\\end{definition}","sha256_utf8":"22aea0789a66d333e732a4f8e16cd7235c1325f33cfed4109993f9cc5fea945c"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}[Almost sure global convergence]\n\\label{thm:global}\nSuppose that \\eqref{eq:opt} is variationally coherent.\nThen, under \\cref{asm:diff,asm:L2,asm:reciprocity}, $\\act_{\\run}$ converges with probability $1$ to a \\textpar{possibly random} minimum point of \\eqref{eq:opt}.\n\\end{theorem}","units":[{"start":79102,"end":79398,"start_line":1689,"end_line":1693,"raw_tex":"\\begin{theorem}[Almost sure global convergence]\n\\label{thm:global}\nSuppose that \\eqref{eq:opt} is variationally coherent.\nThen, under \\cref{asm:diff,asm:L2,asm:reciprocity}, $\\act_{\\run}$ converges with probability $1$ to a \\textpar{possibly random} minimum point of \\eqref{eq:opt}.\n\\end{theorem}","sha256_utf8":"972f78cca042cd5e82bc17544413b285f269738afb50b51d09d9221b3da4ad8c"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\n\\label{lem:Fenchel}\nLet $h$ be a $\\strong$-strongly convex regularizer on $\\states$.\nThen, for all $\\base\\in\\states$ and all $\\dstate,\\alt\\dstate\\in\\dstates$, we have:\n\\begin{subequations}\n\\label{eq:Fench-properties}\n\\begin{alignat}{2}\n\\label{eq:Fench-norm}\n&a)\n\t\\;\\;\n\t\\fench(\\base,\\dstate)\n\t&&\\geq \\frac{\\strong}{2} \\, \\norm{\\mirror(\\dstate) - \\base}^{2}.\n\t\\hspace{17em}\n\t\\\\[3pt]\n\\label{eq:Fench-bound}\n&b)\n\t\\;\\;\n\t\\fench(\\base,\\alt\\dstate)\n\t&&\\leq \\fench(\\base,\\dstate) + \\braket{\\alt\\dstate - \\dstate}{\\mirror(\\dstate) - \\base} + \\frac{1}{2\\strong} \\dnorm{\\alt\\dstate - \\dstate}^{2}.\n\\end{alignat}\n\\end{subequations}\n%\\begin{enumerate}\n%\\item\n%$\\fench(\\base, \\dstate) \\geq \\frac{1}{2} \\strong\\norm{\\mirror(\\dstate) - \\base}^{2}, \\forall x \\in \\states, \\forall y \\in \\R^{n}.$\n%\\item \n%$\\fench(\\state, \\tilde{\\dstate}) \\le \\fench(\\state, \\dstate) + \\braket{\\tilde{\\dstate} - y}{\\mirror(\\dstate) - x} + \\frac{1}{2\\strong} \\|\\tilde{\\dstate} - y \\|_*^2, \\forall x \\in \\states, \\forall \\tilde{\\dstate}, y \\in \\R^{n}.$\n%\\end{enumerate}\n\\end{lemma}\n\n\\begin{proposition}\n\\label{prop:recurrence}\nFix some $\\eps>0$ and $\\delta>0$.\nIf \\eqref{eq:opt} is variationally coherent and \\cref{asm:diff,asm:L2,asm:reciprocity} hold, the \\textpar{random} iterates $\\act_{\\run}$ of \\cref{alg:SMD} enter $\\ball(\\sols,\\eps)$ and $\\fball(\\sols,\\delta)$ infinitely many times \\as.\n\\end{proposition}\n\n\\begin{corollary}\n\\label{cor:recurrence}\nWith probability $1$, there exists a subsequence $\\act_{\\run_{\\runalt}}$ of $\\act_{\\run}$ converging to a \\textpar{random} minimum point $\\sol$ of \\eqref{eq:opt}.\n\\end{corollary}\n\n\\begin{proofstep}{Martingale properties of $\\score_{\\run}$}\nFirst, let\n\\begin{equation}\n\\payv(\\state)\n\t= - \\exof{\\nabla\\sobj(\\state;\\sample)} = -\\nabla\\obj(\\state)\n\\end{equation}\ndenote the negative gradient of $\\obj$ at $\\state\\in\\states$, and write\n\\begin{equation}\n\\label{eq:oracle}\n\\est\\payv_{\\run}\n\t= -\\nabla\\sobj(\\act_{\\run};\\sample_{\\run})\n\\end{equation}\nfor the corresponding oracle feedback at stage $\\run$.\nThen, \\cref{alg:SMD} may be written in Robbins\\textendash Monro form as\n\\begin{equation}\n\\label{eq:SA}\n\\score_{\\run+1}\n\t= \\score_{\\run} + \\step_{\\run} \\est\\payv_{\\run}\n\t= \\score_{\\run} + \\step_{\\run} \\bracks{\\payv(\\act_{\\run}) + \\noise_{\\run}},\n\\end{equation}\nwhere\n\\begin{equation}\n\\noise_{\\run}\n\t= \\nabla\\obj(\\act_{\\run}) - \\nabla\\sobj(\\act_{\\run};\\sample_{\\run})\n\\end{equation}\ndenotes the difference between the mean gradient of $\\obj$ at $\\act_{\\run}$ and the $\\run$-th stage gradient sample.%\n\\footnote{Recall here that there is a one-step offset between $\\act_{\\run}$ and $\\sample_{\\run+1}$ at the $\\run$-th iteration of \\ac{SMD}.}\n\\begin{subequations}\n\\label{eq:MDS}\nBy construction, $\\noise_{\\run}$ is a \\acl{MDS} relative to the history (natural filtration) $\\filter_{\\run} = \\sigma(\\sample_{1},\\dotsc,\\sample_{\\run})$ of $\\sample_{\\run}$, \\ie\n\\begin{equation}\n\\label{eq:zero-mean}\n\\exof{\\noise_{\\run}\\given\\filter_{\\run-1}}\n\t= 0\n\t\\quad\n\t\\text{for all $\\run$.}\n\\end{equation}\nFurthermore, by \\cref{asm:L2}, it readily follows that $\\noise_{\\run}$ has uniformly bounded second moments, \\ie there exists some finite $\\noisedev\\geq0$ such that\n\\begin{equation}\n\\label{eq:MSE}\n\\exof{\\dnorm{\\noise_{\\run}}^{2} \\given \\filter_{\\run-1}}\n\t\\leq \\noisevar\n\t\\quad\n\t\\text{for all $\\run$},\n\\end{equation}\nimplying in turn that $\\noise_{\\run}$ is bounded in $L^{2}$ (for a more detailed treatment, see \\cref{app:proofs}).\n\\end{subequations}\n\\end{proofstep}\n\n\\begin{proofstep}{Recurrence of $\\eps$-neighborhoods}\nInvoking the \\acl{LLN} for $L^{2}$-bounded \\aclp{MDS} and using the Fenchel coupling as an energy function (cf.~\\cref{app:proofs}), we show that if $\\act_{\\run}$ remains outside $\\ball(\\sols,\\eps)$ for sufficiently large $\\run$, we must also have $\\fench(\\sols,\\score_{\\run})\\to-\\infty$ \\as.\nThis contradicts the positive-definiteness of $\\fench$, so $\\act_{\\run}$ must enter $\\ball(\\sols,\\eps)$ infinitely often \\as.\n\\end{proofstep}\n\n\\begin{proofstep}{Recurrence of Fenchel zones}\nBy reciprocity (\\cref{asm:reciprocity}), $\\fball(\\sols, \\delta)$ always contains an $\\eps$-neighborhood of $\\sols$.\n%within it: $\\forall \\delta > 0$, $\\exists \\eps(\\delta) > 0$ such that: $\\ball(\\sols,\\eps) \\subset \\fball(\\sols, \\delta)$.\nSince $\\act_{\\run}$ enters $\\ball(\\sols,\\eps)$ infinitely many times \\as, the same must hold for $\\fball(\\sols, \\delta)$.\nOur claim and \\cref{cor:recurrence} then follow immediately.\n\\end{proofstep}\n\n\\begin{proof}[Proof of \\cref{thm:global}]\nLet $\\sol\\in\\sols$ be a minimum point of \\eqref{eq:opt}.\nThen, letting $\\fench_{\\run} = \\fench(\\sol,\\score_{\\run})$, \\cref{lem:Fenchel} gives\n\\begin{flalign}\n\\label{eq:Fench-bound1}\n\\fench_{\\run+1}\n\t= \\fench(\\sol,\\score_{\\run+1})\n\t&= \\fench(\\sol,\\score_{\\run} + \\step_{\\run}\\est\\payv_{\\run})\n\t\\notag\\\\\n\t&\\leq \\fench(\\sol,\\score_{\\run})\n\t+ \\step_{\\run} \\braket{\\est\\payv_{\\run}}{\\act_{\\run} - \\sol}\n\t+ \\frac{\\step_{\\run}^{2}}{2\\strong} \\dnorm{\\est\\payv_{\\run}}^{2}\n\t\\notag\\\\\n\t&= \\fench_{\\run}\n\t+ \\step_{\\run} \\braket{\\payv(\\act_{\\run})}{\\act_{\\run} - \\sol}\n\t+ \\step_{\\run} \\snoise_{\\run}\n%\t+ \\step_{\\run} \\braket{\\noise_{\\run}}{\\act_{\\run} - \\sol}\n\t+ \\frac{\\step_{\\run}^{2}}{2\\strong} \\dnorm{\\est\\payv_{\\run}}^{2}\n\t\\notag\\\\\n\t&\\leq \\fench_{\\run}\n\t+ \\step_{\\run} \\snoise_{\\run}\n\t+ \\frac{\\step_{\\run}^{2}}{2\\strong} \\dnorm{\\est\\payv_{\\run}}^{2},\n\\end{flalign}\nwhere we set $\\snoise_{\\run} = \\braket{\\noise_{\\run}}{\\act_{\\run} - \\sol}$ in the third line and used the fact that $\\obj$ satisfies \\eqref{eq:VC} in the last one.\nSince $\\score_{\\run}$ is predictable relative to $\\filter_{\\run}$ (\\ie $\\score_{\\run}$ is $\\filter_{\\run-1}$-measurable), the process $\\fench_{\\run} = \\fench(\\sol,\\score_{\\run})$ is itself adapted to the shifted filtration $\\alt\\filter_{\\run} = \\sigma(\\sample_{\\start},\\score_{2}\\dotsc,\\sample_{\\run-1},\\score_{\\run}) = \\filter_{\\run-1}$.\nThus, taking conditional expectations and invoking \\cref{asm:L2}, the bound \\eqref{eq:Fench-bound1} becomes:\n\\begin{flalign}\n\\label{eq:Fench-bound-cond}\n\\exof{\\fench_{\\run+1} \\given \\alt\\filter_{\\run}}\n\t&\\leq \\fench_{\\run}\n\t+ \\exof{\\snoise_{\\run} \\given \\alt\\filter_{\\run}}\n\t+ \\frac{\\step_{\\run}^{2}}{2\\strong} \\exof{\\dnorm{\\est\\payv_{\\run}}^{2} \\given \\alt\\filter_{\\run}}\n\t\\notag\\\\\n\t&= \\fench_{\\run}\n\t+ \\exof{\\snoise_{\\run} \\given \\filter_{\\run-1}}\n\t+ \\frac{\\step_{\\run}^{2}}{2\\strong} \\exof{\\dnorm{\\nabla\\sobj(\\act_{\\run};\\sample_{\\run})}^{2} \\given \\filter_{\\run-1}}\n\t\\notag\\\\\n\t&\\leq \\fench_{\\run}\n\t+ \\frac{\\step_{\\run}^{2} \\vbound^{2}}{2\\strong},\n\\end{flalign}\nwhere, in the last line, we used \\cref{asm:L2} and the fact that $\\noise_{\\run}$ is a \\acl{MDS} (so $\\exof{\\snoise_{\\run} \\given \\filter_{\\run-1}} = 0$; for a detailed derivation, see the proof of \\cref{prop:recurrence} in \\cref{app:proofs}).\nHence, with $\\sum_{\\run=\\start}^{\\infty} \\step_{\\run}^{2} < \\infty$, \\cref{lem:Gladyshev} implies that $\\fench_{\\run}$ converges \\as to some finite limit $\\fench_{\\infty}$.\n\nNow, by \\cref{prop:recurrence}, there exists \\as a subsequence $\\score_{\\run_{\\runalt}}$ of $\\score_{\\run}$ and some (possibly random) $\\sol\\in\\sols$ such that $\\lim_{\\runalt\\to\\infty} \\fench(\\sol,\\score_{\\run_{\\runalt}}) = 0$.\nSince the limit $\\lim_{\\run\\to\\infty} \\fench(\\sol,\\score_{\\run})$ exists \\as, it follows that $\\lim_{\\run\\to\\infty} \\fench(\\sol,\\score_{\\run}) = 0$.\nThis shows that, with probability $1$, $\\act_{\\run} = \\mirror(\\score_{\\run})$ converges to some (random) minimum point $\\sol$ of \\eqref{eq:opt}, as claimed.\n\\end{proof}\n\n\\begin{proofof}{Proof of \\cref{lem:Fenchel}}\nTo prove the first claim, let $\\state =\\mirror(\\dstate) = \\argmax_{\\alt\\state\\in\\states}\\{\\braket{\\dstate}{\\alt\\state} - \\hreg(\\alt\\state)\\}$, so $\\dstate\\in\\pd \\hreg(\\state)$ from standard results in convex analysis \\cite{Roc70}.\n%($y$ is a subgradient of $\\hreg(p)$ if and only if $p=\\mirror(\\dstate)$).\nWe thus get:\n\\begin{equation}\n\\label{eq:Fench-sub}\n\\fench(\\base,\\dstate) = \\hreg(\\base) + \\hreg^{\\ast}(\\dstate) - \\braket{\\dstate}{\\base}\n%\t= \\hreg(\\base) + \\braket{\\dstate}{\\mirror(\\dstate)} - \\hreg(\\mirror(\\dstate)) - \\braket{\\dstate}{\\base}\n\t= \\hreg(\\base) - \\hreg(\\state) - \\braket{\\dstate}{\\base - \\state}.\n\\end{equation}\nSince $\\dstate \\in \\pd \\hreg(\\state)$ and $\\hreg$ is $\\strong$-strongly convex, we also have\n\\begin{equation}\n\\hreg(\\state) + \\tau\\braket{\\dstate}{\\base - \\state}\n\t\\leq \\hreg(\\state + \\tau(\\base - \\state))\n\t\\leq (1-\\tau) \\hreg(\\state) + \\tau \\hreg(\\base) - \\tfrac{1}{2} \\strong \\tau(1-\\tau) \\norm{\\state - \\base}^{2}\n\\end{equation}\nfor all $\\tau\\in[0,1]$, thereby leading to the bound\n\\begin{equation}\n\\label{eq:divbound}\n\\tfrac{1}{2} \\strong(1-\\tau) \\norm{\\state - \\base}^{2}\n\t\\leq \\hreg(\\base) - \\hreg(\\state) - \\braket{\\dstate}{\\base - \\state}\n\t= \\fench(\\base,\\dstate).\n\\end{equation}\nOur claim then follows by letting $\\tau\\to0^{+}$ in \\eqref{eq:divbound}.\n\nFor our second claim, we start by citing a well-known duality principle for strongly convex functions \\cite[Theorem 12.60]{RW98}:\nIf $f\\from\\R^{\\vdim}\\to\\R\\cup\\{-\\infty,+\\infty\\}$ is proper, lower semi-continuous and convex, its convex conjugate $f^{\\ast}$ is $\\sigma$-strongly convex if and only if $f$ is differentiable and satisfies\n\\begin{equation}\nf(\\alt\\state)\n\t\\leq f(\\state) + \\braket{\\nabla f(\\state)}{\\alt\\state - \\state} + \\frac{1}{2\\sigma} \\norm{\\alt\\state - \\state}^{2}\n\\end{equation}\nfor all $\\state,\\alt\\state\\in\\R^{\\vdim}$.\nIn our case, if we extend $\\hreg$ to all of $\\vecspace$ by setting $\\hreg\\equiv+\\infty$ outside $\\states$, we have that $\\hreg$ is $\\strong$-strongly convex, lower semi-continuous and proper, so $(\\hreg^{\\ast})^{\\ast} = \\hreg$ \\cite[Theorem 11.1]{RW98}.\nIt is also easy to see that $\\hreg^{\\ast}$ is proper, lower semi-continuous and convex (since it is a point-wise maximum of affine functions by definition), so the $\\strong$-strong convexity of $\\hreg = (\\hreg^{\\ast})^{\\ast}$ implies that $\\hreg^{\\ast}$ is differentiable and satisfies\n\\begin{flalign}\n\\label{eq:smoothness}\n\\hreg^{\\ast}(\\alt\\dstate)\n\t&\\leq \\hreg^{\\ast}(\\dstate)\n\t+ \\braket{\\alt\\dstate - \\dstate}{\\nabla \\hreg^{\\ast}(\\dstate)}\n\t+ \\frac{1}{2\\strong} \\dnorm{\\alt\\dstate - \\dstate}^{2}\n\t\\\\\n\t&= \\hreg^{\\ast}(\\dstate)\n\t+ \\braket{\\alt\\dstate - \\dstate}{\\mirror(\\dstate)} +\n\t\\frac{1}{2\\strong} \\dnorm{\\alt\\dstate - \\dstate}^{2}\n\\end{flalign}\nfor all $\\dstate,\\alt\\dstate\\in\\dstates$, where the last equality follows from the fact that $\\nabla \\hreg^{\\ast}(\\dstate) = \\mirror(\\dstate)$.\nTherefore, substituting the preceding inequality in the definition of the Fenchel coupling, we obtain\n\\begin{flalign}\n\\fench(\\state,\\alt\\dstate)\n\t&= \\hreg(\\state) + \\hreg^{\\ast}(\\alt\\dstate) - \\braket{\\alt\\dstate}{\\state}\n\t\\notag\\\\\n\t&\\leq \\hreg(\\state) + \\hreg^{\\ast}(\\dstate) + \\braket{\\alt\\dstate - \\dstate}{\\nabla \\hreg^{\\ast}(\\dstate)} + \\frac{1}{2\\strong} \\dnorm{\\alt\\dstate - \\dstate}^{2} - \\braket{\\alt\\dstate}{\\state}\n\t\\notag\\\\\n\t&= \\fench(\\state,\\dstate) + \\braket{\\alt\\dstate - \\dstate}{\\mirror(\\dstate) - \\state} + \\frac{1}{2\\strong} \\dnorm{\\alt\\dstate - \\dstate}^{2},\n\\end{flalign}\nand our assertion follows.\n\\end{proofof}\n\n\\begin{proofof}{Proof of \\cref{prop:recurrence}}\nOur proof proceeds step-by-step, as discussed in \\cref{sec:recurrence}:\n\n\\setcounter{proofstep}{0}\n\\begin{proofstep}{Martingale properties of $\\score_{\\run}$}\nBy \\cref{asm:L2} and the fact that finiteness of second moments implies finiteness of first moments, we get $\\exof{\\dnorm{\\sobj(\\state;\\sample_{\\run})}} < \\infty$.\nWe then claim that $\\noise_{\\run} = \\nabla\\obj(\\act_{\\run}) - \\nabla\\sobj(\\act_{\\run};\\sample_{\\run})$ is an $L^{2}$-bounded \\acl{MDS} with respect to the natural filtration of $\\sample_{\\run}$.\nIndeed, we have:\n\\begin{enumerate}\n\\addtolength{\\itemsep}{1ex}\n\\item\nSince $\\act_{\\run}$ is $\\filter_{\\run-1}$-measurable and $\\sample_{\\run}$ is \\ac{iid}, we readily get\n\\begin{flalign}\n\\exof{\\noise_{\\run} \\given \\filter_{\\run-1}}\n\t&= \\exof{\\nabla\\obj(\\act_{\\run}) - \\nabla\\sobj(\\act_{\\run};\\sample_{\\run}) \\given \\filter_{\\run-1}}\n\t\\notag\\\\\n\t&= \\exof{\\nabla\\obj(\\act_{\\run}) - \\nabla\\sobj(\\act_{\\run};\\sample_{\\run}) \\given \\sample_{1},\\dotsc,\\sample_{\\run-1}}\n\t\\notag\\\\\n\t&= \\nabla\\obj(\\act_{\\run}) - \\nabla\\obj(\\act_{\\run})\n\t\\notag\\\\\n\t&= 0.\n\\end{flalign}\n%so $\\exof{\\noise_{\\run}} = \\exof{\\exof{\\noise_{\\run} \\given \\filter_{\\run-1}}} = 0$, i.e. $\\noise_{\\run}$ has zero mean.\n\n\\item\nFurthermore, by \\cref{asm:L2}, the $L^{2}$ norm of $\\noise$ satisfies\n\\begin{flalign}\n\\label{eq:vbound}\n\\exof{\\dnorm{\\noise_{\\run}}^{2} \\given \\filter_{\\run-1}}\n\t&= \\exof{\\dnorm{\\nabla\\obj(\\act_{\\run}) - \\nabla\\sobj(\\act_{\\run};\\sample_{\\run})}^{2} \\given \\filter_{\\run-1}}\n\t\\notag\\\\\n\t&\\leq 2\\exof{\\dnorm{\\nabla\\obj(\\act_{\\run})}^{2} \\given \\filter_{\\run-1}}\n\t+ 2 \\exof{\\dnorm{\\nabla\\sobj(\\act_{\\run};\\sample_{\\run})}^{2} \\given \\filter_{\\run-1}}\n\t\\notag\\\\\n\t&\\leq 2 \\dnorm{\\nabla\\obj(\\act_{\\run})}^{2}\n\t+ 2 \\vbound^{2}\n%\t+ 2 \\dnorm{\\exof{\\nabla\\sobj(\\act_{\\run};\\sample_{\\run}) \\given \\filter_{\\run-1}}}^{2}\n\t\\notag\\\\\n\t&= 2\\dnorm{\\exof{\\nabla\\sobj(\\act_{\\run};\\sample)}}^{2}\n\t+ 2\\vbound^{2}\n\t\\notag\\\\\n\t&\\leq 2 \\exof{\\dnorm{\\nabla\\sobj(\\act_{\\run};\\sample)}^{2}}\n\t+ 2\\vbound^{2}\n\t\\notag\\\\\n\t&\\leq \\noisevar,\n\\end{flalign}\nwhere we set $\\noisevar = 4\\vbound^{2}$, and we used the dominated convergence theorem to interchange expectation and differentiation in the second line, and Jensen's inequality in the penultimate one.\n%Since $\\states$ is compact, there exists some finite $\\bound$ such that $\\dnorm{\\nabla\\obj(\\state)} \\leq \\bound$ for all $\\state\\in\\states$.\n%Thus, letting $\\vbound = 2\\bound$, we finally obtain\n%\\begin{equation}\n%\\label{eq:vbound}\n%%\\exof{\\dnorm{\\noise_{\\run}}^{2}}\n%%\t= \\exof{\\exof{\\dnorm{\\noise_{\\run}}^{2} \\given \\filter_{\\run-1}}}\n%\\exof{\\dnorm{\\noise_{\\run}}^{2} \\given \\filter_{\\run-1}}\n%\t\\leq \\vbound^{2}.\n%\\end{equation}\n\\end{enumerate}\n\\end{proofstep}\n\n\\begin{proofstep}{Recurrence of $\\eps$-neighborhoods}\nWe proceed to show that every $\\eps$-neigh\\-bor\\-hood $\\ball(\\sols,\\eps)$ of $\\sols$ is recurrent under $\\act_{\\run}$.\nTo do so, fix some $\\eps>0$ and assume ad absurdum that $\\act_{\\run}$ enters $\\ball(\\sols,\\eps)$ only a finite number of times, so there exists some finite $\\run_{0}$ such that $\\dist(\\sols,\\act_{\\run}) \\geq \\eps$ for all $\\run\\geq\\run_{0}$.\nSince $\\states\\setminus\\ball(\\sols,\\eps)$ is compact, $\\payv(\\state) = -\\nabla \\obj(\\state)$ is continuous in $\\state$;\nfurthermore, letting $\\sol$ be such that $\\braket{\\nabla\\obj(\\state)}{\\state - \\sol} = 0$ only if $\\state\\in\\sols$ (cf.~\\cref{def:VC}),  it follows that there exists some $\\const \\equiv \\const(\\eps) > 0$ such that\n\\begin{equation}\n\\braket{\\payv(\\state)}{\\state - \\sol}\n\t\\leq - \\const\n\t< 0\n\t\\quad\n\t\\text{for all $\\state\\in\\states\\setminus\\ball(\\sols,\\eps)$}.\n\\end{equation}\nTo proceed, let $R = \\max_{\\state \\in \\states} \\norm{\\state}$ and set $\\beta_{\\run} = \\step_{\\run}^{2}/(2\\strong)$.\nThen, letting $\\fench_{\\run} = \\fench(\\sol,\\score_{\\run})$ and $\\snoise_{\\run} = \\braket{\\noise_{\\run}}{\\act_{\\run}-\\sol}$, \\cref{lem:Fenchel} yields\n\\begin{flalign}\n\\fench_{\\run+1}\n\t= \\fench(\\sol,\\score_{\\run+1})\n\t&= \\fench(\\sol, \\score_{\\run} + \\step_{\\run} \\est\\payv_{\\run})\n\t\\notag\\\\\n\t&\\leq \\fench(\\sol, \\score_{\\run})\n\t+ \\step_{\\run} \\braket{\\payv(\\act_{\\run}) + \\noise_{\\run}}{\\act_{\\run} - \\sol}\n\t+ \\beta_{\\run} \\dnorm{\\est\\payv_{\\run}}^{2}\n\t\\notag\\\\\n\t&=\\fench_{\\run}\n\t+ \\step_{\\run} \\braket{\\payv(\\act_{\\run})}{\\act_{\\run} - \\sol}\n\t+ \\step_{\\run} \\snoise_{\\run}\n\t+ \\beta_{\\run} \\dnorm{\\est\\payv_{\\run}}^{2}\n\t\\notag\\\\\n\t&\\leq \\fench_{\\run}\n\t- \\step_{\\run} \\const\n\t+ \\step_{\\run} \\snoise_{\\run}\n\t+ \\beta_{\\run} \\dnorm{\\est\\payv_{\\run}}^{2}.\n\\end{flalign}\nHence, letting $\\tau_{\\run} = \\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt}$ and telescoping from $\\run_{0}$ to $\\run$, we get\n\\begin{flalign}\n\\label{eq:bound-Fench}\n\\fench_{\\run+1}\n\t&\\leq \\fench_{\\run_{0}}\n\t- \\const \\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt}\n\t+ \\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt} \\snoise_{\\runalt}\n\t+ \\sum_{\\runalt=\\run_{0}}^{\\run} \\beta_{\\runalt} \\dnorm{\\est\\payv_{\\runalt}}^{2}\n\t\\notag\\\\\n\t&= \\fench_{\\run_{0}}\n\t- \\tau_{\\run} \\bracks*{\\const - \\frac{\\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt}\\snoise_{\\runalt}}{\\tau_{\\run}}}\n\t+ \\sum_{\\runalt=\\run_{0}}^{\\run} \\beta_{\\runalt} \\dnorm{\\est\\payv_{\\runalt}}^{2}\n%\t\\notag\\\\\n%\t&\\leq \\fench_{\\run_{0}}\n%\t- \\tau_{\\run} \\bracks*{\\const - \\frac{\\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt}\\snoise_{\\runalt}}{\\tau_{\\run}}}\n%\t+ 2 \\smashoperator{\\sum_{\\runalt=\\run_{0}}^{\\run}} \\beta_{\\runalt} \\bound^{2}\n\\end{flalign}\n\nWe now proceed to bound each term of \\eqref{eq:bound-Fench}.\nFirst, by construction, we have\n\\begin{equation}\n\\exof{\\snoise_{\\run} \\given \\filter_{\\run-1}}\n\t= \\exof{\\braket{\\noise_{\\run}}{\\act_{\\run} - \\sol} \\given \\filter_{\\run-1}}\n\t= \\braket{\\exof{\\noise_{\\run} \\given \\filter_{\\run-1}}}{\\act_{\\run} - \\sol}\n\t= 0,\n\\end{equation}\nwhere we used the fact that $\\act_{\\run}$ is $\\filter_{\\run-1}$-measurable.\nMoreover, by Young's inequality, we also have\n\\begin{equation}\n\\abs{\\snoise_{\\run}}\n\t= \\abs{\\braket{\\noise_{\\run}}{\\act_{\\run} - \\sol}}\n\t\\leq \\dnorm{\\noise_{\\run}} \\norm{\\act_{\\run} - \\sol}\n\t\\leq 2R\\dnorm{\\noise_{\\run}},\n\\end{equation}\nwhere, as before, $R = \\max_{\\state\\in\\states} \\norm{\\state}$.\n\\cref{eq:vbound} then gives\n\\begin{equation}\n\\exof{\\snoise_{\\run}^{2} \\given \\filter_{\\run-1}}\n\t\\leq (2R)^{2} \\exof{\\dnorm{\\noise_{\\run}}^{2} \\given \\filter_{\\run-1}}\n\t\\leq 4 R^{2} \\noisevar,\n\\end{equation}\nimplying in turn that $\\snoise_{\\run}$ is an $L^{2}$-bounded \\acl{MDS}.\nIt then follows that $\\snoise_{\\run}$ satisfies the summability condition\n\\begin{equation}\n\\sum_{\\run=\\run_{0}}^{\\infty} \\frac{\\exof{\\abs{\\step_{\\run} \\snoise_{\\run}}^{2} \\given \\filter_{\\run-1}}}{\\tau_{\\run}^{2}}\n\t\\leq 4R^{2}\\noisevar \\sum_{\\run=\\run_{0}}^{\\infty} \\frac{\\step_{\\run}^{2}}{\\tau_{\\run}^{2}}\n\t< \\infty,\n\\end{equation}\nwhere the last inequality follows from the assumption that $\\step_{\\run}$ is square-summable.\nThus, by the \\acl{LLN} for \\aclp{MDS} (\\cref{thm:mg_convergence1}), we get\n\\begin{equation}\n\\frac{\\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt} \\snoise_{\\runalt}}{\\tau_{\\run}}\n\t\\to 0\n\t\\quad\n\t\\text{as $\\run\\to\\infty$ \\as},\n\\end{equation}\nand, with $\\sum_{\\runalt=\\run_{0}}^{\\infty} \\step_{\\runalt} = \\infty$, we finally obtain\n\\begin{equation}\n\\lim_{\\run\\to\\infty} \\tau_{\\run} \\bracks*{\\const - \\frac{\\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt}\\snoise_{\\runalt}}{\\tau_{\\run}}}\n\t= \\infty\n\t\\quad\n\t\\as.\n\\end{equation}\n\n%The third term of \\eqref{eq:bound-Fench} is trivially bounded because $\\step_{\\run}$ is square-summable.\nFor the last term of \\eqref{eq:bound-Fench}, let $S_{\\run} = \\sum_{\\runalt=\\run_{0}}^{\\run} \\beta_{\\runalt} \\dnorm{\\est\\payv_{\\runalt}}^{2}$, so $S_{\\run}$ is $\\filter_{\\run}$-measurable and nondecrasing.\nIn addition, we have\n\\begin{equation}\n\\exof{S_{\\run}}\n\t= \\sum_{\\runalt=\\run_{0}}^{\\run} \\beta_{\\runalt} \\exof{\\dnorm{\\est\\payv_{\\runalt}}^{2}}\n\t\\leq \\vbound^{2} \\sum_{\\runalt=\\run_{0}}^{\\run} \\beta_{\\runalt}\n\t< \\infty,\n\\end{equation}\nwith the last step following from \\eqref{eq:vbound}.\nThis implies that $S_{\\run}$ is an $L^{1}$-bounded submartingale so, by Doob's submartingale convergence theorem (\\cref{thm:mg_convergence2}), $S_{\\run}$ converges almost surely to some random variable $S_{\\infty}$, \\ie the last term of \\eqref{eq:bound-Fench} is bounded.\nHence, combining all of the above, we finally obtain\n\\begin{equation}\n\\limsup_{\\run\\to\\infty} \\fench_{\\run}\n\t= - \\infty\n\t\\quad\n\t\\as,\n\\end{equation}\ncontradicting the positive-definiteness of the Fenchel coupling (cf.~\\cref{lem:Fenchel}).\nWe thus conclude that $\\act_{\\run}$ enters $\\ball(\\sols,\\eps)$ infinitely many times \\as, as claimed.\n\\end{proofstep}\n\n\\begin{proofstep}{Recurrence of Fenchel zones}\nUsing the reciprocity of the Fenchel coupling (\\cref{asm:reciprocity}), we show below that every Fenchel zone $\\fball(\\sols,\\delta)$ of $\\sols$ contains an $\\eps$-neighborhood of $\\sols$.\n%\\begin{equation}\\label{eq:contains}\n%\\forall \\delta > 0,\\exists \\eps(\\delta) > 0, \\ball(\\sols,\\eps) \\subset \\fball(\\sols, \\delta).\n%\\end{equation} \nThen, given that $\\act_{\\run}$ enters $\\ball(\\sols,\\eps)$ infinitely often (per the previous step), it will also enter $\\fball(\\sols, \\delta)$ infinitely often.\n\nTo establish this claim, assume instead that there is no $\\eps$-ball $\\ball(\\sols,\\eps)$ contained in $\\fball(\\sols, \\delta)$.\nThen, for all $\\runalt>0$ there exists some $\\dstate_{\\runalt}\\in\\dstates$ such that $\\dist(\\sols,\\mirror(\\dstate_{\\runalt})) = 1/\\runalt$ but $\\fench(\\sols, y_{\\delta}) \\geq \\eps$.\nThis produces a sequence $(\\dstate_{\\runalt})_{\\runalt=1}^{\\infty}$ such that $\\dist(\\sols,\\mirror(\\dstate_{\\runalt})) \\to 0$ but $\\fench(\\sols, \\dstate_{\\runalt}) \\geq \\eps$.\nSince $\\states$ is compact and $\\sols$ is closed, we can assume without loss of generality that $\\mirror(\\dstate_{\\runalt})\\to\\base$ for some $\\base\\in\\sols$ (at worst, we only need to descend to a subsequence of $\\dstate_{\\runalt}$).\nWe thus get\n\\begin{equation}\n\\eps\n\t\\leq \\fench(\\sols,\\dstate_{\\runalt})\n\t\\leq \\fench(\\base,\\dstate_{\\runalt}).\n\\end{equation}\nHowever, since $\\mirror(\\dstate_{\\runalt})\\to\\base$, \\cref{asm:reciprocity} gives $\\fench(\\base,\\dstate_{\\runalt})\\to0$, a contradiction.\nWe conclude that $\\fball(\\sols,\\delta)$ contains an $\\eps$-neighborhood of $\\sols$, as required.\n\\end{proofstep}\n\\end{proofof}","units":[{"start":71335,"end":72391,"start_line":1517,"end_line":1544,"raw_tex":"\\begin{lemma}\n\\label{lem:Fenchel}\nLet $h$ be a $\\strong$-strongly convex regularizer on $\\states$.\nThen, for all $\\base\\in\\states$ and all $\\dstate,\\alt\\dstate\\in\\dstates$, we have:\n\\begin{subequations}\n\\label{eq:Fench-properties}\n\\begin{alignat}{2}\n\\label{eq:Fench-norm}\n&a)\n\t\\;\\;\n\t\\fench(\\base,\\dstate)\n\t&&\\geq \\frac{\\strong}{2} \\, \\norm{\\mirror(\\dstate) - \\base}^{2}.\n\t\\hspace{17em}\n\t\\\\[3pt]\n\\label{eq:Fench-bound}\n&b)\n\t\\;\\;\n\t\\fench(\\base,\\alt\\dstate)\n\t&&\\leq \\fench(\\base,\\dstate) + \\braket{\\alt\\dstate - \\dstate}{\\mirror(\\dstate) - \\base} + \\frac{1}{2\\strong} \\dnorm{\\alt\\dstate - \\dstate}^{2}.\n\\end{alignat}\n\\end{subequations}\n%\\begin{enumerate}\n%\\item\n%$\\fench(\\base, \\dstate) \\geq \\frac{1}{2} \\strong\\norm{\\mirror(\\dstate) - \\base}^{2}, \\forall x \\in \\states, \\forall y \\in \\R^{n}.$\n%\\item \n%$\\fench(\\state, \\tilde{\\dstate}) \\le \\fench(\\state, \\dstate) + \\braket{\\tilde{\\dstate} - y}{\\mirror(\\dstate) - x} + \\frac{1}{2\\strong} \\|\\tilde{\\dstate} - y \\|_*^2, \\forall x \\in \\states, \\forall \\tilde{\\dstate}, y \\in \\R^{n}.$\n%\\end{enumerate}\n\\end{lemma}","sha256_utf8":"89f22bcaca148ff861769b74b23c0acdddf40d21c87df1eab10dbd4ab52ad2dc"},{"start":74993,"end":75323,"start_line":1597,"end_line":1601,"raw_tex":"\\begin{proposition}\n\\label{prop:recurrence}\nFix some $\\eps>0$ and $\\delta>0$.\nIf \\eqref{eq:opt} is variationally coherent and \\cref{asm:diff,asm:L2,asm:reciprocity} hold, the \\textpar{random} iterates $\\act_{\\run}$ of \\cref{alg:SMD} enter $\\ball(\\sols,\\eps)$ and $\\fball(\\sols,\\delta)$ infinitely many times \\as.\n\\end{proposition}","sha256_utf8":"9c6e4c5ffeccbacc6dceebab94295ef2372a497fae92cb30c2fdffc25a607f00"},{"start":75325,"end":75544,"start_line":1603,"end_line":1606,"raw_tex":"\\begin{corollary}\n\\label{cor:recurrence}\nWith probability $1$, there exists a subsequence $\\act_{\\run_{\\runalt}}$ of $\\act_{\\run}$ converging to a \\textpar{random} minimum point $\\sol$ of \\eqref{eq:opt}.\n\\end{corollary}","sha256_utf8":"e0e9469b26142cbb175c30c0dd1299c5d4c1db322b6222744c8167823ed6b577"},{"start":75674,"end":77543,"start_line":1613,"end_line":1660,"raw_tex":"\\begin{proofstep}{Martingale properties of $\\score_{\\run}$}\nFirst, let\n\\begin{equation}\n\\payv(\\state)\n\t= - \\exof{\\nabla\\sobj(\\state;\\sample)} = -\\nabla\\obj(\\state)\n\\end{equation}\ndenote the negative gradient of $\\obj$ at $\\state\\in\\states$, and write\n\\begin{equation}\n\\label{eq:oracle}\n\\est\\payv_{\\run}\n\t= -\\nabla\\sobj(\\act_{\\run};\\sample_{\\run})\n\\end{equation}\nfor the corresponding oracle feedback at stage $\\run$.\nThen, \\cref{alg:SMD} may be written in Robbins\\textendash Monro form as\n\\begin{equation}\n\\label{eq:SA}\n\\score_{\\run+1}\n\t= \\score_{\\run} + \\step_{\\run} \\est\\payv_{\\run}\n\t= \\score_{\\run} + \\step_{\\run} \\bracks{\\payv(\\act_{\\run}) + \\noise_{\\run}},\n\\end{equation}\nwhere\n\\begin{equation}\n\\noise_{\\run}\n\t= \\nabla\\obj(\\act_{\\run}) - \\nabla\\sobj(\\act_{\\run};\\sample_{\\run})\n\\end{equation}\ndenotes the difference between the mean gradient of $\\obj$ at $\\act_{\\run}$ and the $\\run$-th stage gradient sample.%\n\\footnote{Recall here that there is a one-step offset between $\\act_{\\run}$ and $\\sample_{\\run+1}$ at the $\\run$-th iteration of \\ac{SMD}.}\n\\begin{subequations}\n\\label{eq:MDS}\nBy construction, $\\noise_{\\run}$ is a \\acl{MDS} relative to the history (natural filtration) $\\filter_{\\run} = \\sigma(\\sample_{1},\\dotsc,\\sample_{\\run})$ of $\\sample_{\\run}$, \\ie\n\\begin{equation}\n\\label{eq:zero-mean}\n\\exof{\\noise_{\\run}\\given\\filter_{\\run-1}}\n\t= 0\n\t\\quad\n\t\\text{for all $\\run$.}\n\\end{equation}\nFurthermore, by \\cref{asm:L2}, it readily follows that $\\noise_{\\run}$ has uniformly bounded second moments, \\ie there exists some finite $\\noisedev\\geq0$ such that\n\\begin{equation}\n\\label{eq:MSE}\n\\exof{\\dnorm{\\noise_{\\run}}^{2} \\given \\filter_{\\run-1}}\n\t\\leq \\noisevar\n\t\\quad\n\t\\text{for all $\\run$},\n\\end{equation}\nimplying in turn that $\\noise_{\\run}$ is bounded in $L^{2}$ (for a more detailed treatment, see \\cref{app:proofs}).\n\\end{subequations}\n\\end{proofstep}","sha256_utf8":"33e2c81f9e493a6e6e3a01ef6583093441f5f1546691ee9fa1c0dc7332b54ab5"},{"start":77545,"end":78032,"start_line":1662,"end_line":1665,"raw_tex":"\\begin{proofstep}{Recurrence of $\\eps$-neighborhoods}\nInvoking the \\acl{LLN} for $L^{2}$-bounded \\aclp{MDS} and using the Fenchel coupling as an energy function (cf.~\\cref{app:proofs}), we show that if $\\act_{\\run}$ remains outside $\\ball(\\sols,\\eps)$ for sufficiently large $\\run$, we must also have $\\fench(\\sols,\\score_{\\run})\\to-\\infty$ \\as.\nThis contradicts the positive-definiteness of $\\fench$, so $\\act_{\\run}$ must enter $\\ball(\\sols,\\eps)$ infinitely often \\as.\n\\end{proofstep}","sha256_utf8":"a959a5d2a46bb3bcd10a91348c65d3f385900b7ada91c80504c035f912094bd9"},{"start":78035,"end":78519,"start_line":1668,"end_line":1673,"raw_tex":"\\begin{proofstep}{Recurrence of Fenchel zones}\nBy reciprocity (\\cref{asm:reciprocity}), $\\fball(\\sols, \\delta)$ always contains an $\\eps$-neighborhood of $\\sols$.\n%within it: $\\forall \\delta > 0$, $\\exists \\eps(\\delta) > 0$ such that: $\\ball(\\sols,\\eps) \\subset \\fball(\\sols, \\delta)$.\nSince $\\act_{\\run}$ enters $\\ball(\\sols,\\eps)$ infinitely many times \\as, the same must hold for $\\fball(\\sols, \\delta)$.\nOur claim and \\cref{cor:recurrence} then follow immediately.\n\\end{proofstep}","sha256_utf8":"79dbe2091243ece56cb987490d85387a8348eaf222700a78a6c6037676da3337"},{"start":82158,"end":85186,"start_line":1736,"end_line":1782,"raw_tex":"\\begin{proof}[Proof of \\cref{thm:global}]\nLet $\\sol\\in\\sols$ be a minimum point of \\eqref{eq:opt}.\nThen, letting $\\fench_{\\run} = \\fench(\\sol,\\score_{\\run})$, \\cref{lem:Fenchel} gives\n\\begin{flalign}\n\\label{eq:Fench-bound1}\n\\fench_{\\run+1}\n\t= \\fench(\\sol,\\score_{\\run+1})\n\t&= \\fench(\\sol,\\score_{\\run} + \\step_{\\run}\\est\\payv_{\\run})\n\t\\notag\\\\\n\t&\\leq \\fench(\\sol,\\score_{\\run})\n\t+ \\step_{\\run} \\braket{\\est\\payv_{\\run}}{\\act_{\\run} - \\sol}\n\t+ \\frac{\\step_{\\run}^{2}}{2\\strong} \\dnorm{\\est\\payv_{\\run}}^{2}\n\t\\notag\\\\\n\t&= \\fench_{\\run}\n\t+ \\step_{\\run} \\braket{\\payv(\\act_{\\run})}{\\act_{\\run} - \\sol}\n\t+ \\step_{\\run} \\snoise_{\\run}\n%\t+ \\step_{\\run} \\braket{\\noise_{\\run}}{\\act_{\\run} - \\sol}\n\t+ \\frac{\\step_{\\run}^{2}}{2\\strong} \\dnorm{\\est\\payv_{\\run}}^{2}\n\t\\notag\\\\\n\t&\\leq \\fench_{\\run}\n\t+ \\step_{\\run} \\snoise_{\\run}\n\t+ \\frac{\\step_{\\run}^{2}}{2\\strong} \\dnorm{\\est\\payv_{\\run}}^{2},\n\\end{flalign}\nwhere we set $\\snoise_{\\run} = \\braket{\\noise_{\\run}}{\\act_{\\run} - \\sol}$ in the third line and used the fact that $\\obj$ satisfies \\eqref{eq:VC} in the last one.\nSince $\\score_{\\run}$ is predictable relative to $\\filter_{\\run}$ (\\ie $\\score_{\\run}$ is $\\filter_{\\run-1}$-measurable), the process $\\fench_{\\run} = \\fench(\\sol,\\score_{\\run})$ is itself adapted to the shifted filtration $\\alt\\filter_{\\run} = \\sigma(\\sample_{\\start},\\score_{2}\\dotsc,\\sample_{\\run-1},\\score_{\\run}) = \\filter_{\\run-1}$.\nThus, taking conditional expectations and invoking \\cref{asm:L2}, the bound \\eqref{eq:Fench-bound1} becomes:\n\\begin{flalign}\n\\label{eq:Fench-bound-cond}\n\\exof{\\fench_{\\run+1} \\given \\alt\\filter_{\\run}}\n\t&\\leq \\fench_{\\run}\n\t+ \\exof{\\snoise_{\\run} \\given \\alt\\filter_{\\run}}\n\t+ \\frac{\\step_{\\run}^{2}}{2\\strong} \\exof{\\dnorm{\\est\\payv_{\\run}}^{2} \\given \\alt\\filter_{\\run}}\n\t\\notag\\\\\n\t&= \\fench_{\\run}\n\t+ \\exof{\\snoise_{\\run} \\given \\filter_{\\run-1}}\n\t+ \\frac{\\step_{\\run}^{2}}{2\\strong} \\exof{\\dnorm{\\nabla\\sobj(\\act_{\\run};\\sample_{\\run})}^{2} \\given \\filter_{\\run-1}}\n\t\\notag\\\\\n\t&\\leq \\fench_{\\run}\n\t+ \\frac{\\step_{\\run}^{2} \\vbound^{2}}{2\\strong},\n\\end{flalign}\nwhere, in the last line, we used \\cref{asm:L2} and the fact that $\\noise_{\\run}$ is a \\acl{MDS} (so $\\exof{\\snoise_{\\run} \\given \\filter_{\\run-1}} = 0$; for a detailed derivation, see the proof of \\cref{prop:recurrence} in \\cref{app:proofs}).\nHence, with $\\sum_{\\run=\\start}^{\\infty} \\step_{\\run}^{2} < \\infty$, \\cref{lem:Gladyshev} implies that $\\fench_{\\run}$ converges \\as to some finite limit $\\fench_{\\infty}$.\n\nNow, by \\cref{prop:recurrence}, there exists \\as a subsequence $\\score_{\\run_{\\runalt}}$ of $\\score_{\\run}$ and some (possibly random) $\\sol\\in\\sols$ such that $\\lim_{\\runalt\\to\\infty} \\fench(\\sol,\\score_{\\run_{\\runalt}}) = 0$.\nSince the limit $\\lim_{\\run\\to\\infty} \\fench(\\sol,\\score_{\\run})$ exists \\as, it follows that $\\lim_{\\run\\to\\infty} \\fench(\\sol,\\score_{\\run}) = 0$.\nThis shows that, with probability $1$, $\\act_{\\run} = \\mirror(\\score_{\\run})$ converges to some (random) minimum point $\\sol$ of \\eqref{eq:opt}, as claimed.\n\\end{proof}","sha256_utf8":"736fc639b005fee07396094d28d98f33cfd36841eb9d610823617009edfe5dea"},{"start":140363,"end":143925,"start_line":2700,"end_line":2756,"raw_tex":"\\begin{proofof}{Proof of \\cref{lem:Fenchel}}\nTo prove the first claim, let $\\state =\\mirror(\\dstate) = \\argmax_{\\alt\\state\\in\\states}\\{\\braket{\\dstate}{\\alt\\state} - \\hreg(\\alt\\state)\\}$, so $\\dstate\\in\\pd \\hreg(\\state)$ from standard results in convex analysis \\cite{Roc70}.\n%($y$ is a subgradient of $\\hreg(p)$ if and only if $p=\\mirror(\\dstate)$).\nWe thus get:\n\\begin{equation}\n\\label{eq:Fench-sub}\n\\fench(\\base,\\dstate) = \\hreg(\\base) + \\hreg^{\\ast}(\\dstate) - \\braket{\\dstate}{\\base}\n%\t= \\hreg(\\base) + \\braket{\\dstate}{\\mirror(\\dstate)} - \\hreg(\\mirror(\\dstate)) - \\braket{\\dstate}{\\base}\n\t= \\hreg(\\base) - \\hreg(\\state) - \\braket{\\dstate}{\\base - \\state}.\n\\end{equation}\nSince $\\dstate \\in \\pd \\hreg(\\state)$ and $\\hreg$ is $\\strong$-strongly convex, we also have\n\\begin{equation}\n\\hreg(\\state) + \\tau\\braket{\\dstate}{\\base - \\state}\n\t\\leq \\hreg(\\state + \\tau(\\base - \\state))\n\t\\leq (1-\\tau) \\hreg(\\state) + \\tau \\hreg(\\base) - \\tfrac{1}{2} \\strong \\tau(1-\\tau) \\norm{\\state - \\base}^{2}\n\\end{equation}\nfor all $\\tau\\in[0,1]$, thereby leading to the bound\n\\begin{equation}\n\\label{eq:divbound}\n\\tfrac{1}{2} \\strong(1-\\tau) \\norm{\\state - \\base}^{2}\n\t\\leq \\hreg(\\base) - \\hreg(\\state) - \\braket{\\dstate}{\\base - \\state}\n\t= \\fench(\\base,\\dstate).\n\\end{equation}\nOur claim then follows by letting $\\tau\\to0^{+}$ in \\eqref{eq:divbound}.\n\nFor our second claim, we start by citing a well-known duality principle for strongly convex functions \\cite[Theorem 12.60]{RW98}:\nIf $f\\from\\R^{\\vdim}\\to\\R\\cup\\{-\\infty,+\\infty\\}$ is proper, lower semi-continuous and convex, its convex conjugate $f^{\\ast}$ is $\\sigma$-strongly convex if and only if $f$ is differentiable and satisfies\n\\begin{equation}\nf(\\alt\\state)\n\t\\leq f(\\state) + \\braket{\\nabla f(\\state)}{\\alt\\state - \\state} + \\frac{1}{2\\sigma} \\norm{\\alt\\state - \\state}^{2}\n\\end{equation}\nfor all $\\state,\\alt\\state\\in\\R^{\\vdim}$.\nIn our case, if we extend $\\hreg$ to all of $\\vecspace$ by setting $\\hreg\\equiv+\\infty$ outside $\\states$, we have that $\\hreg$ is $\\strong$-strongly convex, lower semi-continuous and proper, so $(\\hreg^{\\ast})^{\\ast} = \\hreg$ \\cite[Theorem 11.1]{RW98}.\nIt is also easy to see that $\\hreg^{\\ast}$ is proper, lower semi-continuous and convex (since it is a point-wise maximum of affine functions by definition), so the $\\strong$-strong convexity of $\\hreg = (\\hreg^{\\ast})^{\\ast}$ implies that $\\hreg^{\\ast}$ is differentiable and satisfies\n\\begin{flalign}\n\\label{eq:smoothness}\n\\hreg^{\\ast}(\\alt\\dstate)\n\t&\\leq \\hreg^{\\ast}(\\dstate)\n\t+ \\braket{\\alt\\dstate - \\dstate}{\\nabla \\hreg^{\\ast}(\\dstate)}\n\t+ \\frac{1}{2\\strong} \\dnorm{\\alt\\dstate - \\dstate}^{2}\n\t\\\\\n\t&= \\hreg^{\\ast}(\\dstate)\n\t+ \\braket{\\alt\\dstate - \\dstate}{\\mirror(\\dstate)} +\n\t\\frac{1}{2\\strong} \\dnorm{\\alt\\dstate - \\dstate}^{2}\n\\end{flalign}\nfor all $\\dstate,\\alt\\dstate\\in\\dstates$, where the last equality follows from the fact that $\\nabla \\hreg^{\\ast}(\\dstate) = \\mirror(\\dstate)$.\nTherefore, substituting the preceding inequality in the definition of the Fenchel coupling, we obtain\n\\begin{flalign}\n\\fench(\\state,\\alt\\dstate)\n\t&= \\hreg(\\state) + \\hreg^{\\ast}(\\alt\\dstate) - \\braket{\\alt\\dstate}{\\state}\n\t\\notag\\\\\n\t&\\leq \\hreg(\\state) + \\hreg^{\\ast}(\\dstate) + \\braket{\\alt\\dstate - \\dstate}{\\nabla \\hreg^{\\ast}(\\dstate)} + \\frac{1}{2\\strong} \\dnorm{\\alt\\dstate - \\dstate}^{2} - \\braket{\\alt\\dstate}{\\state}\n\t\\notag\\\\\n\t&= \\fench(\\state,\\dstate) + \\braket{\\alt\\dstate - \\dstate}{\\mirror(\\dstate) - \\state} + \\frac{1}{2\\strong} \\dnorm{\\alt\\dstate - \\dstate}^{2},\n\\end{flalign}\nand our assertion follows.\n\\end{proofof}","sha256_utf8":"dd5c26f3d9e37c37b595171eca73de02007e5159a6fcea4dda22f264424744aa"},{"start":143983,"end":154219,"start_line":2761,"end_line":2959,"raw_tex":"\\begin{proofof}{Proof of \\cref{prop:recurrence}}\nOur proof proceeds step-by-step, as discussed in \\cref{sec:recurrence}:\n\n\\setcounter{proofstep}{0}\n\\begin{proofstep}{Martingale properties of $\\score_{\\run}$}\nBy \\cref{asm:L2} and the fact that finiteness of second moments implies finiteness of first moments, we get $\\exof{\\dnorm{\\sobj(\\state;\\sample_{\\run})}} < \\infty$.\nWe then claim that $\\noise_{\\run} = \\nabla\\obj(\\act_{\\run}) - \\nabla\\sobj(\\act_{\\run};\\sample_{\\run})$ is an $L^{2}$-bounded \\acl{MDS} with respect to the natural filtration of $\\sample_{\\run}$.\nIndeed, we have:\n\\begin{enumerate}\n\\addtolength{\\itemsep}{1ex}\n\\item\nSince $\\act_{\\run}$ is $\\filter_{\\run-1}$-measurable and $\\sample_{\\run}$ is \\ac{iid}, we readily get\n\\begin{flalign}\n\\exof{\\noise_{\\run} \\given \\filter_{\\run-1}}\n\t&= \\exof{\\nabla\\obj(\\act_{\\run}) - \\nabla\\sobj(\\act_{\\run};\\sample_{\\run}) \\given \\filter_{\\run-1}}\n\t\\notag\\\\\n\t&= \\exof{\\nabla\\obj(\\act_{\\run}) - \\nabla\\sobj(\\act_{\\run};\\sample_{\\run}) \\given \\sample_{1},\\dotsc,\\sample_{\\run-1}}\n\t\\notag\\\\\n\t&= \\nabla\\obj(\\act_{\\run}) - \\nabla\\obj(\\act_{\\run})\n\t\\notag\\\\\n\t&= 0.\n\\end{flalign}\n%so $\\exof{\\noise_{\\run}} = \\exof{\\exof{\\noise_{\\run} \\given \\filter_{\\run-1}}} = 0$, i.e. $\\noise_{\\run}$ has zero mean.\n\n\\item\nFurthermore, by \\cref{asm:L2}, the $L^{2}$ norm of $\\noise$ satisfies\n\\begin{flalign}\n\\label{eq:vbound}\n\\exof{\\dnorm{\\noise_{\\run}}^{2} \\given \\filter_{\\run-1}}\n\t&= \\exof{\\dnorm{\\nabla\\obj(\\act_{\\run}) - \\nabla\\sobj(\\act_{\\run};\\sample_{\\run})}^{2} \\given \\filter_{\\run-1}}\n\t\\notag\\\\\n\t&\\leq 2\\exof{\\dnorm{\\nabla\\obj(\\act_{\\run})}^{2} \\given \\filter_{\\run-1}}\n\t+ 2 \\exof{\\dnorm{\\nabla\\sobj(\\act_{\\run};\\sample_{\\run})}^{2} \\given \\filter_{\\run-1}}\n\t\\notag\\\\\n\t&\\leq 2 \\dnorm{\\nabla\\obj(\\act_{\\run})}^{2}\n\t+ 2 \\vbound^{2}\n%\t+ 2 \\dnorm{\\exof{\\nabla\\sobj(\\act_{\\run};\\sample_{\\run}) \\given \\filter_{\\run-1}}}^{2}\n\t\\notag\\\\\n\t&= 2\\dnorm{\\exof{\\nabla\\sobj(\\act_{\\run};\\sample)}}^{2}\n\t+ 2\\vbound^{2}\n\t\\notag\\\\\n\t&\\leq 2 \\exof{\\dnorm{\\nabla\\sobj(\\act_{\\run};\\sample)}^{2}}\n\t+ 2\\vbound^{2}\n\t\\notag\\\\\n\t&\\leq \\noisevar,\n\\end{flalign}\nwhere we set $\\noisevar = 4\\vbound^{2}$, and we used the dominated convergence theorem to interchange expectation and differentiation in the second line, and Jensen's inequality in the penultimate one.\n%Since $\\states$ is compact, there exists some finite $\\bound$ such that $\\dnorm{\\nabla\\obj(\\state)} \\leq \\bound$ for all $\\state\\in\\states$.\n%Thus, letting $\\vbound = 2\\bound$, we finally obtain\n%\\begin{equation}\n%\\label{eq:vbound}\n%%\\exof{\\dnorm{\\noise_{\\run}}^{2}}\n%%\t= \\exof{\\exof{\\dnorm{\\noise_{\\run}}^{2} \\given \\filter_{\\run-1}}}\n%\\exof{\\dnorm{\\noise_{\\run}}^{2} \\given \\filter_{\\run-1}}\n%\t\\leq \\vbound^{2}.\n%\\end{equation}\n\\end{enumerate}\n\\end{proofstep}\n\n\\begin{proofstep}{Recurrence of $\\eps$-neighborhoods}\nWe proceed to show that every $\\eps$-neigh\\-bor\\-hood $\\ball(\\sols,\\eps)$ of $\\sols$ is recurrent under $\\act_{\\run}$.\nTo do so, fix some $\\eps>0$ and assume ad absurdum that $\\act_{\\run}$ enters $\\ball(\\sols,\\eps)$ only a finite number of times, so there exists some finite $\\run_{0}$ such that $\\dist(\\sols,\\act_{\\run}) \\geq \\eps$ for all $\\run\\geq\\run_{0}$.\nSince $\\states\\setminus\\ball(\\sols,\\eps)$ is compact, $\\payv(\\state) = -\\nabla \\obj(\\state)$ is continuous in $\\state$;\nfurthermore, letting $\\sol$ be such that $\\braket{\\nabla\\obj(\\state)}{\\state - \\sol} = 0$ only if $\\state\\in\\sols$ (cf.~\\cref{def:VC}),  it follows that there exists some $\\const \\equiv \\const(\\eps) > 0$ such that\n\\begin{equation}\n\\braket{\\payv(\\state)}{\\state - \\sol}\n\t\\leq - \\const\n\t< 0\n\t\\quad\n\t\\text{for all $\\state\\in\\states\\setminus\\ball(\\sols,\\eps)$}.\n\\end{equation}\nTo proceed, let $R = \\max_{\\state \\in \\states} \\norm{\\state}$ and set $\\beta_{\\run} = \\step_{\\run}^{2}/(2\\strong)$.\nThen, letting $\\fench_{\\run} = \\fench(\\sol,\\score_{\\run})$ and $\\snoise_{\\run} = \\braket{\\noise_{\\run}}{\\act_{\\run}-\\sol}$, \\cref{lem:Fenchel} yields\n\\begin{flalign}\n\\fench_{\\run+1}\n\t= \\fench(\\sol,\\score_{\\run+1})\n\t&= \\fench(\\sol, \\score_{\\run} + \\step_{\\run} \\est\\payv_{\\run})\n\t\\notag\\\\\n\t&\\leq \\fench(\\sol, \\score_{\\run})\n\t+ \\step_{\\run} \\braket{\\payv(\\act_{\\run}) + \\noise_{\\run}}{\\act_{\\run} - \\sol}\n\t+ \\beta_{\\run} \\dnorm{\\est\\payv_{\\run}}^{2}\n\t\\notag\\\\\n\t&=\\fench_{\\run}\n\t+ \\step_{\\run} \\braket{\\payv(\\act_{\\run})}{\\act_{\\run} - \\sol}\n\t+ \\step_{\\run} \\snoise_{\\run}\n\t+ \\beta_{\\run} \\dnorm{\\est\\payv_{\\run}}^{2}\n\t\\notag\\\\\n\t&\\leq \\fench_{\\run}\n\t- \\step_{\\run} \\const\n\t+ \\step_{\\run} \\snoise_{\\run}\n\t+ \\beta_{\\run} \\dnorm{\\est\\payv_{\\run}}^{2}.\n\\end{flalign}\nHence, letting $\\tau_{\\run} = \\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt}$ and telescoping from $\\run_{0}$ to $\\run$, we get\n\\begin{flalign}\n\\label{eq:bound-Fench}\n\\fench_{\\run+1}\n\t&\\leq \\fench_{\\run_{0}}\n\t- \\const \\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt}\n\t+ \\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt} \\snoise_{\\runalt}\n\t+ \\sum_{\\runalt=\\run_{0}}^{\\run} \\beta_{\\runalt} \\dnorm{\\est\\payv_{\\runalt}}^{2}\n\t\\notag\\\\\n\t&= \\fench_{\\run_{0}}\n\t- \\tau_{\\run} \\bracks*{\\const - \\frac{\\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt}\\snoise_{\\runalt}}{\\tau_{\\run}}}\n\t+ \\sum_{\\runalt=\\run_{0}}^{\\run} \\beta_{\\runalt} \\dnorm{\\est\\payv_{\\runalt}}^{2}\n%\t\\notag\\\\\n%\t&\\leq \\fench_{\\run_{0}}\n%\t- \\tau_{\\run} \\bracks*{\\const - \\frac{\\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt}\\snoise_{\\runalt}}{\\tau_{\\run}}}\n%\t+ 2 \\smashoperator{\\sum_{\\runalt=\\run_{0}}^{\\run}} \\beta_{\\runalt} \\bound^{2}\n\\end{flalign}\n\nWe now proceed to bound each term of \\eqref{eq:bound-Fench}.\nFirst, by construction, we have\n\\begin{equation}\n\\exof{\\snoise_{\\run} \\given \\filter_{\\run-1}}\n\t= \\exof{\\braket{\\noise_{\\run}}{\\act_{\\run} - \\sol} \\given \\filter_{\\run-1}}\n\t= \\braket{\\exof{\\noise_{\\run} \\given \\filter_{\\run-1}}}{\\act_{\\run} - \\sol}\n\t= 0,\n\\end{equation}\nwhere we used the fact that $\\act_{\\run}$ is $\\filter_{\\run-1}$-measurable.\nMoreover, by Young's inequality, we also have\n\\begin{equation}\n\\abs{\\snoise_{\\run}}\n\t= \\abs{\\braket{\\noise_{\\run}}{\\act_{\\run} - \\sol}}\n\t\\leq \\dnorm{\\noise_{\\run}} \\norm{\\act_{\\run} - \\sol}\n\t\\leq 2R\\dnorm{\\noise_{\\run}},\n\\end{equation}\nwhere, as before, $R = \\max_{\\state\\in\\states} \\norm{\\state}$.\n\\cref{eq:vbound} then gives\n\\begin{equation}\n\\exof{\\snoise_{\\run}^{2} \\given \\filter_{\\run-1}}\n\t\\leq (2R)^{2} \\exof{\\dnorm{\\noise_{\\run}}^{2} \\given \\filter_{\\run-1}}\n\t\\leq 4 R^{2} \\noisevar,\n\\end{equation}\nimplying in turn that $\\snoise_{\\run}$ is an $L^{2}$-bounded \\acl{MDS}.\nIt then follows that $\\snoise_{\\run}$ satisfies the summability condition\n\\begin{equation}\n\\sum_{\\run=\\run_{0}}^{\\infty} \\frac{\\exof{\\abs{\\step_{\\run} \\snoise_{\\run}}^{2} \\given \\filter_{\\run-1}}}{\\tau_{\\run}^{2}}\n\t\\leq 4R^{2}\\noisevar \\sum_{\\run=\\run_{0}}^{\\infty} \\frac{\\step_{\\run}^{2}}{\\tau_{\\run}^{2}}\n\t< \\infty,\n\\end{equation}\nwhere the last inequality follows from the assumption that $\\step_{\\run}$ is square-summable.\nThus, by the \\acl{LLN} for \\aclp{MDS} (\\cref{thm:mg_convergence1}), we get\n\\begin{equation}\n\\frac{\\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt} \\snoise_{\\runalt}}{\\tau_{\\run}}\n\t\\to 0\n\t\\quad\n\t\\text{as $\\run\\to\\infty$ \\as},\n\\end{equation}\nand, with $\\sum_{\\runalt=\\run_{0}}^{\\infty} \\step_{\\runalt} = \\infty$, we finally obtain\n\\begin{equation}\n\\lim_{\\run\\to\\infty} \\tau_{\\run} \\bracks*{\\const - \\frac{\\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt}\\snoise_{\\runalt}}{\\tau_{\\run}}}\n\t= \\infty\n\t\\quad\n\t\\as.\n\\end{equation}\n\n%The third term of \\eqref{eq:bound-Fench} is trivially bounded because $\\step_{\\run}$ is square-summable.\nFor the last term of \\eqref{eq:bound-Fench}, let $S_{\\run} = \\sum_{\\runalt=\\run_{0}}^{\\run} \\beta_{\\runalt} \\dnorm{\\est\\payv_{\\runalt}}^{2}$, so $S_{\\run}$ is $\\filter_{\\run}$-measurable and nondecrasing.\nIn addition, we have\n\\begin{equation}\n\\exof{S_{\\run}}\n\t= \\sum_{\\runalt=\\run_{0}}^{\\run} \\beta_{\\runalt} \\exof{\\dnorm{\\est\\payv_{\\runalt}}^{2}}\n\t\\leq \\vbound^{2} \\sum_{\\runalt=\\run_{0}}^{\\run} \\beta_{\\runalt}\n\t< \\infty,\n\\end{equation}\nwith the last step following from \\eqref{eq:vbound}.\nThis implies that $S_{\\run}$ is an $L^{1}$-bounded submartingale so, by Doob's submartingale convergence theorem (\\cref{thm:mg_convergence2}), $S_{\\run}$ converges almost surely to some random variable $S_{\\infty}$, \\ie the last term of \\eqref{eq:bound-Fench} is bounded.\nHence, combining all of the above, we finally obtain\n\\begin{equation}\n\\limsup_{\\run\\to\\infty} \\fench_{\\run}\n\t= - \\infty\n\t\\quad\n\t\\as,\n\\end{equation}\ncontradicting the positive-definiteness of the Fenchel coupling (cf.~\\cref{lem:Fenchel}).\nWe thus conclude that $\\act_{\\run}$ enters $\\ball(\\sols,\\eps)$ infinitely many times \\as, as claimed.\n\\end{proofstep}\n\n\\begin{proofstep}{Recurrence of Fenchel zones}\nUsing the reciprocity of the Fenchel coupling (\\cref{asm:reciprocity}), we show below that every Fenchel zone $\\fball(\\sols,\\delta)$ of $\\sols$ contains an $\\eps$-neighborhood of $\\sols$.\n%\\begin{equation}\\label{eq:contains}\n%\\forall \\delta > 0,\\exists \\eps(\\delta) > 0, \\ball(\\sols,\\eps) \\subset \\fball(\\sols, \\delta).\n%\\end{equation} \nThen, given that $\\act_{\\run}$ enters $\\ball(\\sols,\\eps)$ infinitely often (per the previous step), it will also enter $\\fball(\\sols, \\delta)$ infinitely often.\n\nTo establish this claim, assume instead that there is no $\\eps$-ball $\\ball(\\sols,\\eps)$ contained in $\\fball(\\sols, \\delta)$.\nThen, for all $\\runalt>0$ there exists some $\\dstate_{\\runalt}\\in\\dstates$ such that $\\dist(\\sols,\\mirror(\\dstate_{\\runalt})) = 1/\\runalt$ but $\\fench(\\sols, y_{\\delta}) \\geq \\eps$.\nThis produces a sequence $(\\dstate_{\\runalt})_{\\runalt=1}^{\\infty}$ such that $\\dist(\\sols,\\mirror(\\dstate_{\\runalt})) \\to 0$ but $\\fench(\\sols, \\dstate_{\\runalt}) \\geq \\eps$.\nSince $\\states$ is compact and $\\sols$ is closed, we can assume without loss of generality that $\\mirror(\\dstate_{\\runalt})\\to\\base$ for some $\\base\\in\\sols$ (at worst, we only need to descend to a subsequence of $\\dstate_{\\runalt}$).\nWe thus get\n\\begin{equation}\n\\eps\n\t\\leq \\fench(\\sols,\\dstate_{\\runalt})\n\t\\leq \\fench(\\base,\\dstate_{\\runalt}).\n\\end{equation}\nHowever, since $\\mirror(\\dstate_{\\runalt})\\to\\base$, \\cref{asm:reciprocity} gives $\\fench(\\base,\\dstate_{\\runalt})\\to0$, a contradiction.\nWe conclude that $\\fball(\\sols,\\delta)$ contains an $\\eps$-neighborhood of $\\sols$, as required.\n\\end{proofstep}\n\\end{proofof}","sha256_utf8":"c99cd47f6555315a1c33d2526ea840c9f3534a5793292d3219d2b7124e0dcf1a"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"随机镜像下降几乎必然全局收敛","source_units":[{"start":79102,"end":79398,"start_line":1689,"end_line":1693,"raw_tex":"\\begin{theorem}[Almost sure global convergence]\n\\label{thm:global}\nSuppose that \\eqref{eq:opt} is variationally coherent.\nThen, under \\cref{asm:diff,asm:L2,asm:reciprocity}, $\\act_{\\run}$ converges with probability $1$ to a \\textpar{possibly random} minimum point of \\eqref{eq:opt}.\n\\end{theorem}","sha256_utf8":"972f78cca042cd5e82bc17544413b285f269738afb50b51d09d9221b3da4ad8c"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"全局收敛完整证明","source_units":[{"start":82158,"end":85186,"start_line":1736,"end_line":1782,"raw_tex":"\\begin{proof}[Proof of \\cref{thm:global}]\nLet $\\sol\\in\\sols$ be a minimum point of \\eqref{eq:opt}.\nThen, letting $\\fench_{\\run} = \\fench(\\sol,\\score_{\\run})$, \\cref{lem:Fenchel} gives\n\\begin{flalign}\n\\label{eq:Fench-bound1}\n\\fench_{\\run+1}\n\t= \\fench(\\sol,\\score_{\\run+1})\n\t&= \\fench(\\sol,\\score_{\\run} + \\step_{\\run}\\est\\payv_{\\run})\n\t\\notag\\\\\n\t&\\leq \\fench(\\sol,\\score_{\\run})\n\t+ \\step_{\\run} \\braket{\\est\\payv_{\\run}}{\\act_{\\run} - \\sol}\n\t+ \\frac{\\step_{\\run}^{2}}{2\\strong} \\dnorm{\\est\\payv_{\\run}}^{2}\n\t\\notag\\\\\n\t&= \\fench_{\\run}\n\t+ \\step_{\\run} \\braket{\\payv(\\act_{\\run})}{\\act_{\\run} - \\sol}\n\t+ \\step_{\\run} \\snoise_{\\run}\n%\t+ \\step_{\\run} \\braket{\\noise_{\\run}}{\\act_{\\run} - \\sol}\n\t+ \\frac{\\step_{\\run}^{2}}{2\\strong} \\dnorm{\\est\\payv_{\\run}}^{2}\n\t\\notag\\\\\n\t&\\leq \\fench_{\\run}\n\t+ \\step_{\\run} \\snoise_{\\run}\n\t+ \\frac{\\step_{\\run}^{2}}{2\\strong} \\dnorm{\\est\\payv_{\\run}}^{2},\n\\end{flalign}\nwhere we set $\\snoise_{\\run} = \\braket{\\noise_{\\run}}{\\act_{\\run} - \\sol}$ in the third line and used the fact that $\\obj$ satisfies \\eqref{eq:VC} in the last one.\nSince $\\score_{\\run}$ is predictable relative to $\\filter_{\\run}$ (\\ie $\\score_{\\run}$ is $\\filter_{\\run-1}$-measurable), the process $\\fench_{\\run} = \\fench(\\sol,\\score_{\\run})$ is itself adapted to the shifted filtration $\\alt\\filter_{\\run} = \\sigma(\\sample_{\\start},\\score_{2}\\dotsc,\\sample_{\\run-1},\\score_{\\run}) = \\filter_{\\run-1}$.\nThus, taking conditional expectations and invoking \\cref{asm:L2}, the bound \\eqref{eq:Fench-bound1} becomes:\n\\begin{flalign}\n\\label{eq:Fench-bound-cond}\n\\exof{\\fench_{\\run+1} \\given \\alt\\filter_{\\run}}\n\t&\\leq \\fench_{\\run}\n\t+ \\exof{\\snoise_{\\run} \\given \\alt\\filter_{\\run}}\n\t+ \\frac{\\step_{\\run}^{2}}{2\\strong} \\exof{\\dnorm{\\est\\payv_{\\run}}^{2} \\given \\alt\\filter_{\\run}}\n\t\\notag\\\\\n\t&= \\fench_{\\run}\n\t+ \\exof{\\snoise_{\\run} \\given \\filter_{\\run-1}}\n\t+ \\frac{\\step_{\\run}^{2}}{2\\strong} \\exof{\\dnorm{\\nabla\\sobj(\\act_{\\run};\\sample_{\\run})}^{2} \\given \\filter_{\\run-1}}\n\t\\notag\\\\\n\t&\\leq \\fench_{\\run}\n\t+ \\frac{\\step_{\\run}^{2} \\vbound^{2}}{2\\strong},\n\\end{flalign}\nwhere, in the last line, we used \\cref{asm:L2} and the fact that $\\noise_{\\run}$ is a \\acl{MDS} (so $\\exof{\\snoise_{\\run} \\given \\filter_{\\run-1}} = 0$; for a detailed derivation, see the proof of \\cref{prop:recurrence} in \\cref{app:proofs}).\nHence, with $\\sum_{\\run=\\start}^{\\infty} \\step_{\\run}^{2} < \\infty$, \\cref{lem:Gladyshev} implies that $\\fench_{\\run}$ converges \\as to some finite limit $\\fench_{\\infty}$.\n\nNow, by \\cref{prop:recurrence}, there exists \\as a subsequence $\\score_{\\run_{\\runalt}}$ of $\\score_{\\run}$ and some (possibly random) $\\sol\\in\\sols$ such that $\\lim_{\\runalt\\to\\infty} \\fench(\\sol,\\score_{\\run_{\\runalt}}) = 0$.\nSince the limit $\\lim_{\\run\\to\\infty} \\fench(\\sol,\\score_{\\run})$ exists \\as, it follows that $\\lim_{\\run\\to\\infty} \\fench(\\sol,\\score_{\\run}) = 0$.\nThis shows that, with probability $1$, $\\act_{\\run} = \\mirror(\\score_{\\run})$ converges to some (random) minimum point $\\sol$ of \\eqref{eq:opt}, as claimed.\n\\end{proof}","sha256_utf8":"736fc639b005fee07396094d28d98f33cfd36841eb9d610823617009edfe5dea"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"紧凸域上的随机优化问题","source_units":[{"start":42684,"end":43478,"start_line":920,"end_line":943,"raw_tex":"Let $\\states$ be a convex compact subset of a $\\vdim$-di\\-men\\-sio\\-nal vector space $\\vecspace$ with norm $\\norm{\\argdot}$.\n%and let $\\sample$ be a random variable on an underlying (complete) probability space $\\pspace$\nThroughout this paper, we will focus on stochastic optimization problems of the general form\n\\begin{equation}\n\\label{eq:opt}\n\\tag{Opt}\n\\begin{aligned}\n\\text{minimize}\n&\\quad\n\\obj(\\state),\n\\\\\n\\text{subject to}\n&\\quad\n\\state\\in\\states,\n\\end{aligned}\n\\end{equation}\n%where the objective function $\\obj\\from\\states\\to\\R$ is of the form\nwhere\n\\begin{equation}\n\\label{eq:obj}\n\\obj(\\state)\n\t= \\exof{\\sobj(\\state;\\sample)}\n\\end{equation}\nfor some stochastic objective function $\\sobj\\from\\states\\times\\samples\\to\\R$ defined on an underlying (complete) probability space $\\pspace$.\n","sha256_utf8":"a50156b9158f6c645de2a1d79b23eaa52ec4364c1434c745da99f23a7752d3a4"}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","title":"样本目标连续可微","source_units":[{"start":43566,"end":43723,"start_line":946,"end_line":949,"raw_tex":"\\begin{assumption}\n\\label{asm:diff}\n$\\sobj(\\state, \\sample)$ is continuously differentiable in $\\state$ for almost all $\\sample\\in\\samples$.\n\\end{assumption}","sha256_utf8":"ce0bf661b27fe331b84dbc963dbe768954e7cfa553ab2d33083d3acdf5237bd8"}]},{"id":"S005","semantic_type":"assumption","kind":"assumption","title":"随机梯度二阶矩一致有界","source_units":[{"start":43725,"end":43961,"start_line":951,"end_line":954,"raw_tex":"\\begin{assumption}\n\\label{asm:L2}\nThe gradient of $\\sobj$ is uniformly bounded in $L^{2}$, \\ie $\\exof{\\dnorm{\\nabla\\sobj(\\state;\\sample)}^{2}} \\leq \\vbound^{2}$ for some finite $\\vbound\\geq0$ and all $\\state\\in\\states$.\n\\end{assumption}","sha256_utf8":"dac713086aee94d84c6336762a457d0e21431122e746b4241dcbde7d2d829690"}]},{"id":"S006","semantic_type":"notation","kind":"notation","title":"梯度对偶空间与对偶范数","source_units":[{"start":43976,"end":44471,"start_line":957,"end_line":961,"raw_tex":"\\begin{remark*}\nIn the above, gradients are treated as elements of the dual space $\\dstates\\equiv\\dual\\vecspace$ of $\\vecspace$, and $\\dnorm{\\dstate} = \\sup\\setdef{\\braket{\\dstate}{\\state}}{\\norm{\\state}\\leq1}$ denotes the dual norm of $\\dstate\\in\\dstates$.\nWe also note that $\\nabla\\sobj(\\state;\\sample)$ refers to the gradient of $\\sobj(\\state;\\sample)$ with respect to $\\state$;\nsince $\\samples$ is not assumed to carry a differential structure, there is no danger of confusion.\n\\end{remark*}","sha256_utf8":"b4fc8807952a53682934f8e7cebccdfec1926bde41e65a41217b5b3d193d51c5"}]},{"id":"S007","semantic_type":"derived_fact","kind":"derived_fact","title":"平均梯度与非空闭解集","source_units":[{"start":44768,"end":45330,"start_line":966,"end_line":974,"raw_tex":"\\cref{asm:L2} is also standard in the stochastic optimization literature:\nit holds trivially if $\\sobj$ is uniformly Lipschitz (another commonly used condition) and, by the dominated convergence theorem, it further implies that $\\obj$ is smooth and $\\nabla\\obj(\\state) = \\nabla\\exof{\\sobj(\\state;\\sample)} = \\exof{\\nabla\\sobj(\\state;\\sample)}$ is bounded.\nAs a result, the solution set\n\\begin{equation}\n\\label{eq:solset}\n\\sols\n\t= \\argmin\\obj\n\\end{equation}\nof \\eqref{eq:opt} is closed and nonempty (by the compactness of $\\states$ and the continuity of $\\obj$).\n","sha256_utf8":"0d2a1f6b5d2e36b8e98cfdc80a451646291ee0f58c527ce1eacc7106b8dab45c"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"变分相干性","source_units":[{"start":46920,"end":47305,"start_line":1015,"end_line":1027,"raw_tex":"\\begin{definition}\n\\label{def:VC}\nWe say that \\eqref{eq:opt} is \\emph{variationally coherent} if\n\\begin{equation}\n\\label{eq:VC}\n\\tag{VC}\n\\braket{\\nabla\\obj(\\state)}{\\state - \\sol}\n\t\\geq 0\n\t\\quad\n\t\\text{for all $\\state\\in\\states$, $\\sol\\in\\sols$},\n\\end{equation}\nand there exists some $\\sol\\in\\sols$ such that equality holds in \\eqref{eq:VC} only if $\\state \\in \\sols$.\n\\end{definition}","sha256_utf8":"fbd7baeeaa7eeb60770e9da2e6a49c16a0d47b1c4a7f0ee177b6d180b08f4e10"}]},{"id":"S009","semantic_type":"algorithm","kind":"algorithm","title":"随机递推、采样与步长条件","source_units":[{"start":56232,"end":58115,"start_line":1194,"end_line":1233,"raw_tex":"The main idea of the method is as follows:\nAt each iteration, the algorithm takes as input an \\ac{iid} sample of the gradient of $\\sobj$ at the algorithm's current state.\nSubsequently, the method takes a step along this stochastic gradient in the dual space $\\dstates\\equiv\\dspace$ of $\\vecspace$ (where gradients live), the result is ``mirrored'' back to the problem's feasible region $\\states$, and the process repeats.\nFormally, this gives rise to the recursion\n\\begin{equation}\n\\label{eq:SMD}\n\\tag{SMD}\n\\begin{aligned}\n\\act_{\\run}\n\t&= \\mirror(\\score_{\\run})\n\t\\\\\n\\score_{\\run+1}\n\t&= \\score_{\\run} - \\step_{\\run} \\nabla\\sobj(\\act_{\\run};\\sample_{\\run})\n\\end{aligned}\n\\end{equation}\nwhere:\n\\begin{enumerate}\n\\item\n$\\run=\\running$ denotes the algorithm's running counter.\n\\item\n$\\score_{\\run}\\in\\dstates$ is a score variable that aggregates gradient steps up to stage $\\run$.\n\\item\n$\\mirror\\from\\dstates\\to\\states$ is the \\emph{mirror map} that outputs a solution candidate $\\act_{\\run}\\in\\states$ as a function of the score variable $\\score_{\\run}\\in\\dspace$.\n\\item\n$\\sample_{\\run}\\in\\samples$ is a sequence of \\ac{iid} samples.%\n\\footnote{The indexing convention for $\\sample_{\\run}$ means that $\\score_{\\run}$ and $\\act_{\\run}$ are \\emph{predictable} relative to the natural filtration $\\filter_{\\run} = \\sigma(\\sample_{1},\\dotsc,\\sample_{\\run})$ of $\\sample_{\\run}$, \\ie $\\score_{\\run+1}$ and $\\act_{\\run+1}$ are both $\\filter_{\\run}$-measurable.\nTo this history, we also attach the trivial $\\sigma$-algebra as $\\filter_{0}$ for completeness.}\n\\item\n$\\step_{\\run}>0$ is the algorithm's step-size sequence, assumed in what follows to satisfy the Robbins\\textendash Monro summability condition\n\\begin{equation}\n\\label{eq:L2L1}\n\\sum_{\\run=1}^{\\infty} \\step_{\\run}^{2}\n\t<\\infty\n\t\\quad\n\t\\text{and}\n\t\\quad\n\t\\sum_{\\run=1}^{\\infty} \\step_{\\run}\n\t= \\infty.\n\\end{equation}\n\\end{enumerate}\n","sha256_utf8":"0f6ac13b1131eeb19e3ef6b48bf29a364b874cabce797f2cd80e703d3d3323f2"}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","title":"随机镜像下降伪代码","source_units":[{"start":58360,"end":59431,"start_line":1241,"end_line":1279,"raw_tex":"\\begin{algorithm}[tbp]\n\\caption{Stochastic mirror descent (\\ac{SMD})}\n\\label{alg:SMD}\n%----------------------------------------------------------------------\n%%% SMD ALGORITHM\n%----------------------------------------------------------------------\n% !TEX root = ../Main.tex\n\n\n\\tt\n\\begin{algorithmic}[1]\n\\Require\n\tmirror map $\\mirror\\from\\dstates\\to\\states$;\n\tstep-size sequence $\\step_{\\run}>0$\n\\State\n\tchoose $\\score\\in\\dstates \\equiv \\dspace$\n\t\\Comment{initialization}\n\\For{$\\run=\\running$}\n%\t\\State $\\act_{\\run} = \\mirror(\\score_{\\run-1})$\n%\t\\State $\\score_{\\run} = \\score_{\\run-1} - \\step_{\\run} \\nabla\\sobj(\\act_{\\run},\\sample_{\\run})$\n%\tat each stage $\\run=\\running$\n\t\\State\n\t\tset $\\act \\leftarrow \\mirror(\\score)$\n\t\t\\Comment{set state}\n\t\\State\n\t\tdraw $\\sample\\in\\samples$\n\t\t\\Comment{gradient sample}\n\t\\State\n\t\tget $\\est\\payv = -\\nabla\\sobj(\\act;\\sample)$\n\t\t\\Comment{get oracle feedback}\n\t\\State\n\t\tset $\\score \\leftarrow \\score + \\step_{\\run} \\est\\payv$\n\t\t\\Comment{update score variable}\n\\EndFor\n\\\\\n\\Return $\\act$\n\t\\Comment{output}\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"3f1cc338409cd7e9c606d0699f78711d2998329da62334b3100b91a2eec224ff"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"强凸正则项与镜像映射","source_units":[{"start":63585,"end":64409,"start_line":1387,"end_line":1400,"raw_tex":"In more detail, the algorithm's mirror map $\\mirror\\from\\dstates\\to\\states$ is defined as\n\\begin{equation}\n\\label{eq:mirror}\n\\mirror(\\dstate)\n\t= \\argmax_{\\state\\in\\states} \\{\\braket{\\dstate}{\\state} - \\hreg(\\state)\\},\n\\end{equation}\nwhere the \\emph{regularizer} (or \\emph{penalty function}) $\\hreg\\from\\states\\to\\R$ is assumed to be continuous and strongly convex on $\\states$, \\ie there exists some $\\strong>0$ such that\n\\begin{equation}\n\\label{eq:strong}\n\\hreg(\\tau\\state + (1-\\tau)\\alt\\state)\n\t\\leq \\tau \\hreg(\\state) + (1-\\tau) \\hreg(\\alt\\state) - \\tfrac{1}{2} \\strong\\tau(1-\\tau) \\norm{\\alt\\state - \\state}^{2},\n\\end{equation}\nfor all $\\state,\\alt\\state\\in\\states$ and all $\\tau\\in[0,1]$.\nThe mapping $\\mirror\\from\\dspace\\to\\states$ defined by \\eqref{eq:mirror} is then called the \\emph{mirror map induced by $\\hreg$}.\n","sha256_utf8":"ef32bab0f141b0c17ab2ff96c028ccf9c3d0f947c6dd0b7805bf58a1f59c3e5c"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"Fenchel耦合与凸共轭","source_units":[{"start":70370,"end":70914,"start_line":1501,"end_line":1511,"raw_tex":"\\begin{definition}\n\\label{def:Fenchel}\nLet $h\\from\\states \\to \\R$ be a regularizer on $\\states$.\nThe induced \\emph{Fenchel coupling} $\\fench(\\base,\\dstate)$ between a base-point $\\base\\in\\states$ and a dual vector $\\dstate\\in\\dstates$ is defined as\n\\begin{equation}\n\\label{eq:Fenchel}\n\\fench(\\base,\\dstate)\n\t= \\hreg(\\base) + \\hreg^{\\ast}(\\dstate) - \\braket{\\dstate}{\\base}\n\\end{equation}\nwhere $\\hreg^{\\ast}(\\dstate) = \\max_{\\state\\in\\states} \\{ \\braket{\\dstate}{\\state} - \\hreg(\\state) \\}$ denotes the convex conjugate of $h$.\n\\end{definition}","sha256_utf8":"54236f05ae1d328daaccb9cfe4775a61116eea18b6e24d27f07fab9cd2401b71"}]},{"id":"S013","semantic_type":"assumption","kind":"assumption","title":"Fenchel耦合互易性","source_units":[{"start":72798,"end":72938,"start_line":1549,"end_line":1552,"raw_tex":"\\begin{assumption}\n\\label{asm:reciprocity}\n$\\fench(\\base,\\dstate_{\\run}) \\to 0$ whenever $\\mirror(\\dstate_{\\run})\\to\\base$.\n\\end{assumption}","sha256_utf8":"98e047832d10e3eeaa2826a7ad0ea3c675d86851597d1c199f73e548d51d77ff"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"集合邻域与Fenchel区域","source_units":[{"start":73884,"end":74907,"start_line":1570,"end_line":1593,"raw_tex":"\\begin{definition}\n\\label{def:metrics}\n%Let $h$ be a regularizer on $\\states$ and\nLet $\\region$ be a subset of $\\states$.\n\\begin{subequations}\n\\begin{enumerate}\n\\item\nThe \\emph{distance} between $\\region$ and $\\state\\in\\states$ is defined as $\\dist(\\region,\\state) = \\inf_{\\alt\\state\\in\\region} \\norm{\\alt\\state - \\state}$,\nand the corresponding \\emph{$\\eps$-neighborhood of $\\region$} is\n\\begin{alignat}{2}\n\\label{eq:ball-norm}\n\\ball(\\region,\\eps)\n\t&= \\setdef{\\state\\in\\states}{\\dist(\\region,\\state) < \\eps}.\n\\intertext{%\n\\item\nThe (setwise) \\emph{Fenchel coupling} between $\\region$ and $\\dstate\\in\\dstates$ is defined as $\\fench(\\region,\\dstate) = \\inf_{\\state\\in\\region} \\fench(\\state,\\dstate)$,\nand the corresponding \\emph{Fenchel $\\delta$-zone} of $\\region$ under $h$ is}\n\\label{eq:ball-Fench}\n\\fball(\\region,\\delta)\n\t&= \\setdef{\\state\\in\\states}{\\text{$\\state=\\mirror(\\dstate)$ for some $\\dstate\\in\\dstates$ with $\\fench(\\region,\\dstate) < \\delta$}}.\n\\end{alignat}\n\\end{enumerate}\n\\end{subequations}\n\\end{definition}","sha256_utf8":"22aea0789a66d333e732a4f8e16cd7235c1325f33cfed4109993f9cc5fea945c"}]},{"id":"S015","semantic_type":"lemma","kind":"lemma","title":"Fenchel耦合下界与递推界","source_units":[{"start":71335,"end":72391,"start_line":1517,"end_line":1544,"raw_tex":"\\begin{lemma}\n\\label{lem:Fenchel}\nLet $h$ be a $\\strong$-strongly convex regularizer on $\\states$.\nThen, for all $\\base\\in\\states$ and all $\\dstate,\\alt\\dstate\\in\\dstates$, we have:\n\\begin{subequations}\n\\label{eq:Fench-properties}\n\\begin{alignat}{2}\n\\label{eq:Fench-norm}\n&a)\n\t\\;\\;\n\t\\fench(\\base,\\dstate)\n\t&&\\geq \\frac{\\strong}{2} \\, \\norm{\\mirror(\\dstate) - \\base}^{2}.\n\t\\hspace{17em}\n\t\\\\[3pt]\n\\label{eq:Fench-bound}\n&b)\n\t\\;\\;\n\t\\fench(\\base,\\alt\\dstate)\n\t&&\\leq \\fench(\\base,\\dstate) + \\braket{\\alt\\dstate - \\dstate}{\\mirror(\\dstate) - \\base} + \\frac{1}{2\\strong} \\dnorm{\\alt\\dstate - \\dstate}^{2}.\n\\end{alignat}\n\\end{subequations}\n%\\begin{enumerate}\n%\\item\n%$\\fench(\\base, \\dstate) \\geq \\frac{1}{2} \\strong\\norm{\\mirror(\\dstate) - \\base}^{2}, \\forall x \\in \\states, \\forall y \\in \\R^{n}.$\n%\\item \n%$\\fench(\\state, \\tilde{\\dstate}) \\le \\fench(\\state, \\dstate) + \\braket{\\tilde{\\dstate} - y}{\\mirror(\\dstate) - x} + \\frac{1}{2\\strong} \\|\\tilde{\\dstate} - y \\|_*^2, \\forall x \\in \\states, \\forall \\tilde{\\dstate}, y \\in \\R^{n}.$\n%\\end{enumerate}\n\\end{lemma}","sha256_utf8":"89f22bcaca148ff861769b74b23c0acdddf40d21c87df1eab10dbd4ab52ad2dc"}]},{"id":"S016","semantic_type":"proposition","kind":"proposition","title":"解集邻域几乎必然反复进入","source_units":[{"start":74993,"end":75323,"start_line":1597,"end_line":1601,"raw_tex":"\\begin{proposition}\n\\label{prop:recurrence}\nFix some $\\eps>0$ and $\\delta>0$.\nIf \\eqref{eq:opt} is variationally coherent and \\cref{asm:diff,asm:L2,asm:reciprocity} hold, the \\textpar{random} iterates $\\act_{\\run}$ of \\cref{alg:SMD} enter $\\ball(\\sols,\\eps)$ and $\\fball(\\sols,\\delta)$ infinitely many times \\as.\n\\end{proposition}","sha256_utf8":"9c6e4c5ffeccbacc6dceebab94295ef2372a497fae92cb30c2fdffc25a607f00"}]},{"id":"S017","semantic_type":"corollary","kind":"corollary","title":"存在收敛至最优点的子序列","source_units":[{"start":75325,"end":75544,"start_line":1603,"end_line":1606,"raw_tex":"\\begin{corollary}\n\\label{cor:recurrence}\nWith probability $1$, there exists a subsequence $\\act_{\\run_{\\runalt}}$ of $\\act_{\\run}$ converging to a \\textpar{random} minimum point $\\sol$ of \\eqref{eq:opt}.\n\\end{corollary}","sha256_utf8":"e0e9469b26142cbb175c30c0dd1299c5d4c1db322b6222744c8167823ed6b577"}]},{"id":"S018","semantic_type":"proof","kind":"proof","title":"重返性与子序列结论证明纲要","source_units":[{"start":75674,"end":77543,"start_line":1613,"end_line":1660,"raw_tex":"\\begin{proofstep}{Martingale properties of $\\score_{\\run}$}\nFirst, let\n\\begin{equation}\n\\payv(\\state)\n\t= - \\exof{\\nabla\\sobj(\\state;\\sample)} = -\\nabla\\obj(\\state)\n\\end{equation}\ndenote the negative gradient of $\\obj$ at $\\state\\in\\states$, and write\n\\begin{equation}\n\\label{eq:oracle}\n\\est\\payv_{\\run}\n\t= -\\nabla\\sobj(\\act_{\\run};\\sample_{\\run})\n\\end{equation}\nfor the corresponding oracle feedback at stage $\\run$.\nThen, \\cref{alg:SMD} may be written in Robbins\\textendash Monro form as\n\\begin{equation}\n\\label{eq:SA}\n\\score_{\\run+1}\n\t= \\score_{\\run} + \\step_{\\run} \\est\\payv_{\\run}\n\t= \\score_{\\run} + \\step_{\\run} \\bracks{\\payv(\\act_{\\run}) + \\noise_{\\run}},\n\\end{equation}\nwhere\n\\begin{equation}\n\\noise_{\\run}\n\t= \\nabla\\obj(\\act_{\\run}) - \\nabla\\sobj(\\act_{\\run};\\sample_{\\run})\n\\end{equation}\ndenotes the difference between the mean gradient of $\\obj$ at $\\act_{\\run}$ and the $\\run$-th stage gradient sample.%\n\\footnote{Recall here that there is a one-step offset between $\\act_{\\run}$ and $\\sample_{\\run+1}$ at the $\\run$-th iteration of \\ac{SMD}.}\n\\begin{subequations}\n\\label{eq:MDS}\nBy construction, $\\noise_{\\run}$ is a \\acl{MDS} relative to the history (natural filtration) $\\filter_{\\run} = \\sigma(\\sample_{1},\\dotsc,\\sample_{\\run})$ of $\\sample_{\\run}$, \\ie\n\\begin{equation}\n\\label{eq:zero-mean}\n\\exof{\\noise_{\\run}\\given\\filter_{\\run-1}}\n\t= 0\n\t\\quad\n\t\\text{for all $\\run$.}\n\\end{equation}\nFurthermore, by \\cref{asm:L2}, it readily follows that $\\noise_{\\run}$ has uniformly bounded second moments, \\ie there exists some finite $\\noisedev\\geq0$ such that\n\\begin{equation}\n\\label{eq:MSE}\n\\exof{\\dnorm{\\noise_{\\run}}^{2} \\given \\filter_{\\run-1}}\n\t\\leq \\noisevar\n\t\\quad\n\t\\text{for all $\\run$},\n\\end{equation}\nimplying in turn that $\\noise_{\\run}$ is bounded in $L^{2}$ (for a more detailed treatment, see \\cref{app:proofs}).\n\\end{subequations}\n\\end{proofstep}","sha256_utf8":"33e2c81f9e493a6e6e3a01ef6583093441f5f1546691ee9fa1c0dc7332b54ab5"},{"start":77545,"end":78032,"start_line":1662,"end_line":1665,"raw_tex":"\\begin{proofstep}{Recurrence of $\\eps$-neighborhoods}\nInvoking the \\acl{LLN} for $L^{2}$-bounded \\aclp{MDS} and using the Fenchel coupling as an energy function (cf.~\\cref{app:proofs}), we show that if $\\act_{\\run}$ remains outside $\\ball(\\sols,\\eps)$ for sufficiently large $\\run$, we must also have $\\fench(\\sols,\\score_{\\run})\\to-\\infty$ \\as.\nThis contradicts the positive-definiteness of $\\fench$, so $\\act_{\\run}$ must enter $\\ball(\\sols,\\eps)$ infinitely often \\as.\n\\end{proofstep}","sha256_utf8":"a959a5d2a46bb3bcd10a91348c65d3f385900b7ada91c80504c035f912094bd9"},{"start":78035,"end":78519,"start_line":1668,"end_line":1673,"raw_tex":"\\begin{proofstep}{Recurrence of Fenchel zones}\nBy reciprocity (\\cref{asm:reciprocity}), $\\fball(\\sols, \\delta)$ always contains an $\\eps$-neighborhood of $\\sols$.\n%within it: $\\forall \\delta > 0$, $\\exists \\eps(\\delta) > 0$ such that: $\\ball(\\sols,\\eps) \\subset \\fball(\\sols, \\delta)$.\nSince $\\act_{\\run}$ enters $\\ball(\\sols,\\eps)$ infinitely many times \\as, the same must hold for $\\fball(\\sols, \\delta)$.\nOur claim and \\cref{cor:recurrence} then follow immediately.\n\\end{proofstep}","sha256_utf8":"79dbe2091243ece56cb987490d85387a8348eaf222700a78a6c6037676da3337"}]},{"id":"S019","semantic_type":"proof","kind":"proof","title":"Fenchel耦合性质完整证明","source_units":[{"start":140363,"end":143925,"start_line":2700,"end_line":2756,"raw_tex":"\\begin{proofof}{Proof of \\cref{lem:Fenchel}}\nTo prove the first claim, let $\\state =\\mirror(\\dstate) = \\argmax_{\\alt\\state\\in\\states}\\{\\braket{\\dstate}{\\alt\\state} - \\hreg(\\alt\\state)\\}$, so $\\dstate\\in\\pd \\hreg(\\state)$ from standard results in convex analysis \\cite{Roc70}.\n%($y$ is a subgradient of $\\hreg(p)$ if and only if $p=\\mirror(\\dstate)$).\nWe thus get:\n\\begin{equation}\n\\label{eq:Fench-sub}\n\\fench(\\base,\\dstate) = \\hreg(\\base) + \\hreg^{\\ast}(\\dstate) - \\braket{\\dstate}{\\base}\n%\t= \\hreg(\\base) + \\braket{\\dstate}{\\mirror(\\dstate)} - \\hreg(\\mirror(\\dstate)) - \\braket{\\dstate}{\\base}\n\t= \\hreg(\\base) - \\hreg(\\state) - \\braket{\\dstate}{\\base - \\state}.\n\\end{equation}\nSince $\\dstate \\in \\pd \\hreg(\\state)$ and $\\hreg$ is $\\strong$-strongly convex, we also have\n\\begin{equation}\n\\hreg(\\state) + \\tau\\braket{\\dstate}{\\base - \\state}\n\t\\leq \\hreg(\\state + \\tau(\\base - \\state))\n\t\\leq (1-\\tau) \\hreg(\\state) + \\tau \\hreg(\\base) - \\tfrac{1}{2} \\strong \\tau(1-\\tau) \\norm{\\state - \\base}^{2}\n\\end{equation}\nfor all $\\tau\\in[0,1]$, thereby leading to the bound\n\\begin{equation}\n\\label{eq:divbound}\n\\tfrac{1}{2} \\strong(1-\\tau) \\norm{\\state - \\base}^{2}\n\t\\leq \\hreg(\\base) - \\hreg(\\state) - \\braket{\\dstate}{\\base - \\state}\n\t= \\fench(\\base,\\dstate).\n\\end{equation}\nOur claim then follows by letting $\\tau\\to0^{+}$ in \\eqref{eq:divbound}.\n\nFor our second claim, we start by citing a well-known duality principle for strongly convex functions \\cite[Theorem 12.60]{RW98}:\nIf $f\\from\\R^{\\vdim}\\to\\R\\cup\\{-\\infty,+\\infty\\}$ is proper, lower semi-continuous and convex, its convex conjugate $f^{\\ast}$ is $\\sigma$-strongly convex if and only if $f$ is differentiable and satisfies\n\\begin{equation}\nf(\\alt\\state)\n\t\\leq f(\\state) + \\braket{\\nabla f(\\state)}{\\alt\\state - \\state} + \\frac{1}{2\\sigma} \\norm{\\alt\\state - \\state}^{2}\n\\end{equation}\nfor all $\\state,\\alt\\state\\in\\R^{\\vdim}$.\nIn our case, if we extend $\\hreg$ to all of $\\vecspace$ by setting $\\hreg\\equiv+\\infty$ outside $\\states$, we have that $\\hreg$ is $\\strong$-strongly convex, lower semi-continuous and proper, so $(\\hreg^{\\ast})^{\\ast} = \\hreg$ \\cite[Theorem 11.1]{RW98}.\nIt is also easy to see that $\\hreg^{\\ast}$ is proper, lower semi-continuous and convex (since it is a point-wise maximum of affine functions by definition), so the $\\strong$-strong convexity of $\\hreg = (\\hreg^{\\ast})^{\\ast}$ implies that $\\hreg^{\\ast}$ is differentiable and satisfies\n\\begin{flalign}\n\\label{eq:smoothness}\n\\hreg^{\\ast}(\\alt\\dstate)\n\t&\\leq \\hreg^{\\ast}(\\dstate)\n\t+ \\braket{\\alt\\dstate - \\dstate}{\\nabla \\hreg^{\\ast}(\\dstate)}\n\t+ \\frac{1}{2\\strong} \\dnorm{\\alt\\dstate - \\dstate}^{2}\n\t\\\\\n\t&= \\hreg^{\\ast}(\\dstate)\n\t+ \\braket{\\alt\\dstate - \\dstate}{\\mirror(\\dstate)} +\n\t\\frac{1}{2\\strong} \\dnorm{\\alt\\dstate - \\dstate}^{2}\n\\end{flalign}\nfor all $\\dstate,\\alt\\dstate\\in\\dstates$, where the last equality follows from the fact that $\\nabla \\hreg^{\\ast}(\\dstate) = \\mirror(\\dstate)$.\nTherefore, substituting the preceding inequality in the definition of the Fenchel coupling, we obtain\n\\begin{flalign}\n\\fench(\\state,\\alt\\dstate)\n\t&= \\hreg(\\state) + \\hreg^{\\ast}(\\alt\\dstate) - \\braket{\\alt\\dstate}{\\state}\n\t\\notag\\\\\n\t&\\leq \\hreg(\\state) + \\hreg^{\\ast}(\\dstate) + \\braket{\\alt\\dstate - \\dstate}{\\nabla \\hreg^{\\ast}(\\dstate)} + \\frac{1}{2\\strong} \\dnorm{\\alt\\dstate - \\dstate}^{2} - \\braket{\\alt\\dstate}{\\state}\n\t\\notag\\\\\n\t&= \\fench(\\state,\\dstate) + \\braket{\\alt\\dstate - \\dstate}{\\mirror(\\dstate) - \\state} + \\frac{1}{2\\strong} \\dnorm{\\alt\\dstate - \\dstate}^{2},\n\\end{flalign}\nand our assertion follows.\n\\end{proofof}","sha256_utf8":"dd5c26f3d9e37c37b595171eca73de02007e5159a6fcea4dda22f264424744aa"}]},{"id":"S020","semantic_type":"proof","kind":"proof","title":"重返性命题完整技术证明","source_units":[{"start":143983,"end":154219,"start_line":2761,"end_line":2959,"raw_tex":"\\begin{proofof}{Proof of \\cref{prop:recurrence}}\nOur proof proceeds step-by-step, as discussed in \\cref{sec:recurrence}:\n\n\\setcounter{proofstep}{0}\n\\begin{proofstep}{Martingale properties of $\\score_{\\run}$}\nBy \\cref{asm:L2} and the fact that finiteness of second moments implies finiteness of first moments, we get $\\exof{\\dnorm{\\sobj(\\state;\\sample_{\\run})}} < \\infty$.\nWe then claim that $\\noise_{\\run} = \\nabla\\obj(\\act_{\\run}) - \\nabla\\sobj(\\act_{\\run};\\sample_{\\run})$ is an $L^{2}$-bounded \\acl{MDS} with respect to the natural filtration of $\\sample_{\\run}$.\nIndeed, we have:\n\\begin{enumerate}\n\\addtolength{\\itemsep}{1ex}\n\\item\nSince $\\act_{\\run}$ is $\\filter_{\\run-1}$-measurable and $\\sample_{\\run}$ is \\ac{iid}, we readily get\n\\begin{flalign}\n\\exof{\\noise_{\\run} \\given \\filter_{\\run-1}}\n\t&= \\exof{\\nabla\\obj(\\act_{\\run}) - \\nabla\\sobj(\\act_{\\run};\\sample_{\\run}) \\given \\filter_{\\run-1}}\n\t\\notag\\\\\n\t&= \\exof{\\nabla\\obj(\\act_{\\run}) - \\nabla\\sobj(\\act_{\\run};\\sample_{\\run}) \\given \\sample_{1},\\dotsc,\\sample_{\\run-1}}\n\t\\notag\\\\\n\t&= \\nabla\\obj(\\act_{\\run}) - \\nabla\\obj(\\act_{\\run})\n\t\\notag\\\\\n\t&= 0.\n\\end{flalign}\n%so $\\exof{\\noise_{\\run}} = \\exof{\\exof{\\noise_{\\run} \\given \\filter_{\\run-1}}} = 0$, i.e. $\\noise_{\\run}$ has zero mean.\n\n\\item\nFurthermore, by \\cref{asm:L2}, the $L^{2}$ norm of $\\noise$ satisfies\n\\begin{flalign}\n\\label{eq:vbound}\n\\exof{\\dnorm{\\noise_{\\run}}^{2} \\given \\filter_{\\run-1}}\n\t&= \\exof{\\dnorm{\\nabla\\obj(\\act_{\\run}) - \\nabla\\sobj(\\act_{\\run};\\sample_{\\run})}^{2} \\given \\filter_{\\run-1}}\n\t\\notag\\\\\n\t&\\leq 2\\exof{\\dnorm{\\nabla\\obj(\\act_{\\run})}^{2} \\given \\filter_{\\run-1}}\n\t+ 2 \\exof{\\dnorm{\\nabla\\sobj(\\act_{\\run};\\sample_{\\run})}^{2} \\given \\filter_{\\run-1}}\n\t\\notag\\\\\n\t&\\leq 2 \\dnorm{\\nabla\\obj(\\act_{\\run})}^{2}\n\t+ 2 \\vbound^{2}\n%\t+ 2 \\dnorm{\\exof{\\nabla\\sobj(\\act_{\\run};\\sample_{\\run}) \\given \\filter_{\\run-1}}}^{2}\n\t\\notag\\\\\n\t&= 2\\dnorm{\\exof{\\nabla\\sobj(\\act_{\\run};\\sample)}}^{2}\n\t+ 2\\vbound^{2}\n\t\\notag\\\\\n\t&\\leq 2 \\exof{\\dnorm{\\nabla\\sobj(\\act_{\\run};\\sample)}^{2}}\n\t+ 2\\vbound^{2}\n\t\\notag\\\\\n\t&\\leq \\noisevar,\n\\end{flalign}\nwhere we set $\\noisevar = 4\\vbound^{2}$, and we used the dominated convergence theorem to interchange expectation and differentiation in the second line, and Jensen's inequality in the penultimate one.\n%Since $\\states$ is compact, there exists some finite $\\bound$ such that $\\dnorm{\\nabla\\obj(\\state)} \\leq \\bound$ for all $\\state\\in\\states$.\n%Thus, letting $\\vbound = 2\\bound$, we finally obtain\n%\\begin{equation}\n%\\label{eq:vbound}\n%%\\exof{\\dnorm{\\noise_{\\run}}^{2}}\n%%\t= \\exof{\\exof{\\dnorm{\\noise_{\\run}}^{2} \\given \\filter_{\\run-1}}}\n%\\exof{\\dnorm{\\noise_{\\run}}^{2} \\given \\filter_{\\run-1}}\n%\t\\leq \\vbound^{2}.\n%\\end{equation}\n\\end{enumerate}\n\\end{proofstep}\n\n\\begin{proofstep}{Recurrence of $\\eps$-neighborhoods}\nWe proceed to show that every $\\eps$-neigh\\-bor\\-hood $\\ball(\\sols,\\eps)$ of $\\sols$ is recurrent under $\\act_{\\run}$.\nTo do so, fix some $\\eps>0$ and assume ad absurdum that $\\act_{\\run}$ enters $\\ball(\\sols,\\eps)$ only a finite number of times, so there exists some finite $\\run_{0}$ such that $\\dist(\\sols,\\act_{\\run}) \\geq \\eps$ for all $\\run\\geq\\run_{0}$.\nSince $\\states\\setminus\\ball(\\sols,\\eps)$ is compact, $\\payv(\\state) = -\\nabla \\obj(\\state)$ is continuous in $\\state$;\nfurthermore, letting $\\sol$ be such that $\\braket{\\nabla\\obj(\\state)}{\\state - \\sol} = 0$ only if $\\state\\in\\sols$ (cf.~\\cref{def:VC}),  it follows that there exists some $\\const \\equiv \\const(\\eps) > 0$ such that\n\\begin{equation}\n\\braket{\\payv(\\state)}{\\state - \\sol}\n\t\\leq - \\const\n\t< 0\n\t\\quad\n\t\\text{for all $\\state\\in\\states\\setminus\\ball(\\sols,\\eps)$}.\n\\end{equation}\nTo proceed, let $R = \\max_{\\state \\in \\states} \\norm{\\state}$ and set $\\beta_{\\run} = \\step_{\\run}^{2}/(2\\strong)$.\nThen, letting $\\fench_{\\run} = \\fench(\\sol,\\score_{\\run})$ and $\\snoise_{\\run} = \\braket{\\noise_{\\run}}{\\act_{\\run}-\\sol}$, \\cref{lem:Fenchel} yields\n\\begin{flalign}\n\\fench_{\\run+1}\n\t= \\fench(\\sol,\\score_{\\run+1})\n\t&= \\fench(\\sol, \\score_{\\run} + \\step_{\\run} \\est\\payv_{\\run})\n\t\\notag\\\\\n\t&\\leq \\fench(\\sol, \\score_{\\run})\n\t+ \\step_{\\run} \\braket{\\payv(\\act_{\\run}) + \\noise_{\\run}}{\\act_{\\run} - \\sol}\n\t+ \\beta_{\\run} \\dnorm{\\est\\payv_{\\run}}^{2}\n\t\\notag\\\\\n\t&=\\fench_{\\run}\n\t+ \\step_{\\run} \\braket{\\payv(\\act_{\\run})}{\\act_{\\run} - \\sol}\n\t+ \\step_{\\run} \\snoise_{\\run}\n\t+ \\beta_{\\run} \\dnorm{\\est\\payv_{\\run}}^{2}\n\t\\notag\\\\\n\t&\\leq \\fench_{\\run}\n\t- \\step_{\\run} \\const\n\t+ \\step_{\\run} \\snoise_{\\run}\n\t+ \\beta_{\\run} \\dnorm{\\est\\payv_{\\run}}^{2}.\n\\end{flalign}\nHence, letting $\\tau_{\\run} = \\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt}$ and telescoping from $\\run_{0}$ to $\\run$, we get\n\\begin{flalign}\n\\label{eq:bound-Fench}\n\\fench_{\\run+1}\n\t&\\leq \\fench_{\\run_{0}}\n\t- \\const \\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt}\n\t+ \\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt} \\snoise_{\\runalt}\n\t+ \\sum_{\\runalt=\\run_{0}}^{\\run} \\beta_{\\runalt} \\dnorm{\\est\\payv_{\\runalt}}^{2}\n\t\\notag\\\\\n\t&= \\fench_{\\run_{0}}\n\t- \\tau_{\\run} \\bracks*{\\const - \\frac{\\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt}\\snoise_{\\runalt}}{\\tau_{\\run}}}\n\t+ \\sum_{\\runalt=\\run_{0}}^{\\run} \\beta_{\\runalt} \\dnorm{\\est\\payv_{\\runalt}}^{2}\n%\t\\notag\\\\\n%\t&\\leq \\fench_{\\run_{0}}\n%\t- \\tau_{\\run} \\bracks*{\\const - \\frac{\\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt}\\snoise_{\\runalt}}{\\tau_{\\run}}}\n%\t+ 2 \\smashoperator{\\sum_{\\runalt=\\run_{0}}^{\\run}} \\beta_{\\runalt} \\bound^{2}\n\\end{flalign}\n\nWe now proceed to bound each term of \\eqref{eq:bound-Fench}.\nFirst, by construction, we have\n\\begin{equation}\n\\exof{\\snoise_{\\run} \\given \\filter_{\\run-1}}\n\t= \\exof{\\braket{\\noise_{\\run}}{\\act_{\\run} - \\sol} \\given \\filter_{\\run-1}}\n\t= \\braket{\\exof{\\noise_{\\run} \\given \\filter_{\\run-1}}}{\\act_{\\run} - \\sol}\n\t= 0,\n\\end{equation}\nwhere we used the fact that $\\act_{\\run}$ is $\\filter_{\\run-1}$-measurable.\nMoreover, by Young's inequality, we also have\n\\begin{equation}\n\\abs{\\snoise_{\\run}}\n\t= \\abs{\\braket{\\noise_{\\run}}{\\act_{\\run} - \\sol}}\n\t\\leq \\dnorm{\\noise_{\\run}} \\norm{\\act_{\\run} - \\sol}\n\t\\leq 2R\\dnorm{\\noise_{\\run}},\n\\end{equation}\nwhere, as before, $R = \\max_{\\state\\in\\states} \\norm{\\state}$.\n\\cref{eq:vbound} then gives\n\\begin{equation}\n\\exof{\\snoise_{\\run}^{2} \\given \\filter_{\\run-1}}\n\t\\leq (2R)^{2} \\exof{\\dnorm{\\noise_{\\run}}^{2} \\given \\filter_{\\run-1}}\n\t\\leq 4 R^{2} \\noisevar,\n\\end{equation}\nimplying in turn that $\\snoise_{\\run}$ is an $L^{2}$-bounded \\acl{MDS}.\nIt then follows that $\\snoise_{\\run}$ satisfies the summability condition\n\\begin{equation}\n\\sum_{\\run=\\run_{0}}^{\\infty} \\frac{\\exof{\\abs{\\step_{\\run} \\snoise_{\\run}}^{2} \\given \\filter_{\\run-1}}}{\\tau_{\\run}^{2}}\n\t\\leq 4R^{2}\\noisevar \\sum_{\\run=\\run_{0}}^{\\infty} \\frac{\\step_{\\run}^{2}}{\\tau_{\\run}^{2}}\n\t< \\infty,\n\\end{equation}\nwhere the last inequality follows from the assumption that $\\step_{\\run}$ is square-summable.\nThus, by the \\acl{LLN} for \\aclp{MDS} (\\cref{thm:mg_convergence1}), we get\n\\begin{equation}\n\\frac{\\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt} \\snoise_{\\runalt}}{\\tau_{\\run}}\n\t\\to 0\n\t\\quad\n\t\\text{as $\\run\\to\\infty$ \\as},\n\\end{equation}\nand, with $\\sum_{\\runalt=\\run_{0}}^{\\infty} \\step_{\\runalt} = \\infty$, we finally obtain\n\\begin{equation}\n\\lim_{\\run\\to\\infty} \\tau_{\\run} \\bracks*{\\const - \\frac{\\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt}\\snoise_{\\runalt}}{\\tau_{\\run}}}\n\t= \\infty\n\t\\quad\n\t\\as.\n\\end{equation}\n\n%The third term of \\eqref{eq:bound-Fench} is trivially bounded because $\\step_{\\run}$ is square-summable.\nFor the last term of \\eqref{eq:bound-Fench}, let $S_{\\run} = \\sum_{\\runalt=\\run_{0}}^{\\run} \\beta_{\\runalt} \\dnorm{\\est\\payv_{\\runalt}}^{2}$, so $S_{\\run}$ is $\\filter_{\\run}$-measurable and nondecrasing.\nIn addition, we have\n\\begin{equation}\n\\exof{S_{\\run}}\n\t= \\sum_{\\runalt=\\run_{0}}^{\\run} \\beta_{\\runalt} \\exof{\\dnorm{\\est\\payv_{\\runalt}}^{2}}\n\t\\leq \\vbound^{2} \\sum_{\\runalt=\\run_{0}}^{\\run} \\beta_{\\runalt}\n\t< \\infty,\n\\end{equation}\nwith the last step following from \\eqref{eq:vbound}.\nThis implies that $S_{\\run}$ is an $L^{1}$-bounded submartingale so, by Doob's submartingale convergence theorem (\\cref{thm:mg_convergence2}), $S_{\\run}$ converges almost surely to some random variable $S_{\\infty}$, \\ie the last term of \\eqref{eq:bound-Fench} is bounded.\nHence, combining all of the above, we finally obtain\n\\begin{equation}\n\\limsup_{\\run\\to\\infty} \\fench_{\\run}\n\t= - \\infty\n\t\\quad\n\t\\as,\n\\end{equation}\ncontradicting the positive-definiteness of the Fenchel coupling (cf.~\\cref{lem:Fenchel}).\nWe thus conclude that $\\act_{\\run}$ enters $\\ball(\\sols,\\eps)$ infinitely many times \\as, as claimed.\n\\end{proofstep}\n\n\\begin{proofstep}{Recurrence of Fenchel zones}\nUsing the reciprocity of the Fenchel coupling (\\cref{asm:reciprocity}), we show below that every Fenchel zone $\\fball(\\sols,\\delta)$ of $\\sols$ contains an $\\eps$-neighborhood of $\\sols$.\n%\\begin{equation}\\label{eq:contains}\n%\\forall \\delta > 0,\\exists \\eps(\\delta) > 0, \\ball(\\sols,\\eps) \\subset \\fball(\\sols, \\delta).\n%\\end{equation} \nThen, given that $\\act_{\\run}$ enters $\\ball(\\sols,\\eps)$ infinitely often (per the previous step), it will also enter $\\fball(\\sols, \\delta)$ infinitely often.\n\nTo establish this claim, assume instead that there is no $\\eps$-ball $\\ball(\\sols,\\eps)$ contained in $\\fball(\\sols, \\delta)$.\nThen, for all $\\runalt>0$ there exists some $\\dstate_{\\runalt}\\in\\dstates$ such that $\\dist(\\sols,\\mirror(\\dstate_{\\runalt})) = 1/\\runalt$ but $\\fench(\\sols, y_{\\delta}) \\geq \\eps$.\nThis produces a sequence $(\\dstate_{\\runalt})_{\\runalt=1}^{\\infty}$ such that $\\dist(\\sols,\\mirror(\\dstate_{\\runalt})) \\to 0$ but $\\fench(\\sols, \\dstate_{\\runalt}) \\geq \\eps$.\nSince $\\states$ is compact and $\\sols$ is closed, we can assume without loss of generality that $\\mirror(\\dstate_{\\runalt})\\to\\base$ for some $\\base\\in\\sols$ (at worst, we only need to descend to a subsequence of $\\dstate_{\\runalt}$).\nWe thus get\n\\begin{equation}\n\\eps\n\t\\leq \\fench(\\sols,\\dstate_{\\runalt})\n\t\\leq \\fench(\\base,\\dstate_{\\runalt}).\n\\end{equation}\nHowever, since $\\mirror(\\dstate_{\\runalt})\\to\\base$, \\cref{asm:reciprocity} gives $\\fench(\\base,\\dstate_{\\runalt})\\to0$, a contradiction.\nWe conclude that $\\fball(\\sols,\\delta)$ contains an $\\eps$-neighborhood of $\\sols$, as required.\n\\end{proofstep}\n\\end{proofof}","sha256_utf8":"c99cd47f6555315a1c33d2526ea840c9f3534a5793292d3219d2b7124e0dcf1a"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"相干性消去能量递推漂移项","source_units":[{"start":82158,"end":83220,"start_line":1736,"end_line":1759,"raw_tex":"\\begin{proof}[Proof of \\cref{thm:global}]\nLet $\\sol\\in\\sols$ be a minimum point of \\eqref{eq:opt}.\nThen, letting $\\fench_{\\run} = \\fench(\\sol,\\score_{\\run})$, \\cref{lem:Fenchel} gives\n\\begin{flalign}\n\\label{eq:Fench-bound1}\n\\fench_{\\run+1}\n\t= \\fench(\\sol,\\score_{\\run+1})\n\t&= \\fench(\\sol,\\score_{\\run} + \\step_{\\run}\\est\\payv_{\\run})\n\t\\notag\\\\\n\t&\\leq \\fench(\\sol,\\score_{\\run})\n\t+ \\step_{\\run} \\braket{\\est\\payv_{\\run}}{\\act_{\\run} - \\sol}\n\t+ \\frac{\\step_{\\run}^{2}}{2\\strong} \\dnorm{\\est\\payv_{\\run}}^{2}\n\t\\notag\\\\\n\t&= \\fench_{\\run}\n\t+ \\step_{\\run} \\braket{\\payv(\\act_{\\run})}{\\act_{\\run} - \\sol}\n\t+ \\step_{\\run} \\snoise_{\\run}\n%\t+ \\step_{\\run} \\braket{\\noise_{\\run}}{\\act_{\\run} - \\sol}\n\t+ \\frac{\\step_{\\run}^{2}}{2\\strong} \\dnorm{\\est\\payv_{\\run}}^{2}\n\t\\notag\\\\\n\t&\\leq \\fench_{\\run}\n\t+ \\step_{\\run} \\snoise_{\\run}\n\t+ \\frac{\\step_{\\run}^{2}}{2\\strong} \\dnorm{\\est\\payv_{\\run}}^{2},\n\\end{flalign}\nwhere we set $\\snoise_{\\run} = \\braket{\\noise_{\\run}}{\\act_{\\run} - \\sol}$ in the third line and used the fact that $\\obj$ satisfies \\eqref{eq:VC} in the last one.\n","sha256_utf8":"67c680800cf968a2f525166a6cd91196624a7cdd3120966424bc7d6875d6643d"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"条件期望界保证耦合极限存在","source_units":[{"start":83220,"end":84640,"start_line":1760,"end_line":1777,"raw_tex":"Since $\\score_{\\run}$ is predictable relative to $\\filter_{\\run}$ (\\ie $\\score_{\\run}$ is $\\filter_{\\run-1}$-measurable), the process $\\fench_{\\run} = \\fench(\\sol,\\score_{\\run})$ is itself adapted to the shifted filtration $\\alt\\filter_{\\run} = \\sigma(\\sample_{\\start},\\score_{2}\\dotsc,\\sample_{\\run-1},\\score_{\\run}) = \\filter_{\\run-1}$.\nThus, taking conditional expectations and invoking \\cref{asm:L2}, the bound \\eqref{eq:Fench-bound1} becomes:\n\\begin{flalign}\n\\label{eq:Fench-bound-cond}\n\\exof{\\fench_{\\run+1} \\given \\alt\\filter_{\\run}}\n\t&\\leq \\fench_{\\run}\n\t+ \\exof{\\snoise_{\\run} \\given \\alt\\filter_{\\run}}\n\t+ \\frac{\\step_{\\run}^{2}}{2\\strong} \\exof{\\dnorm{\\est\\payv_{\\run}}^{2} \\given \\alt\\filter_{\\run}}\n\t\\notag\\\\\n\t&= \\fench_{\\run}\n\t+ \\exof{\\snoise_{\\run} \\given \\filter_{\\run-1}}\n\t+ \\frac{\\step_{\\run}^{2}}{2\\strong} \\exof{\\dnorm{\\nabla\\sobj(\\act_{\\run};\\sample_{\\run})}^{2} \\given \\filter_{\\run-1}}\n\t\\notag\\\\\n\t&\\leq \\fench_{\\run}\n\t+ \\frac{\\step_{\\run}^{2} \\vbound^{2}}{2\\strong},\n\\end{flalign}\nwhere, in the last line, we used \\cref{asm:L2} and the fact that $\\noise_{\\run}$ is a \\acl{MDS} (so $\\exof{\\snoise_{\\run} \\given \\filter_{\\run-1}} = 0$; for a detailed derivation, see the proof of \\cref{prop:recurrence} in \\cref{app:proofs}).\nHence, with $\\sum_{\\run=\\start}^{\\infty} \\step_{\\run}^{2} < \\infty$, \\cref{lem:Gladyshev} implies that $\\fench_{\\run}$ converges \\as to some finite limit $\\fench_{\\infty}$.\n","sha256_utf8":"ae23f678bd91c7a2dba511326315b6af909aafa9df1b8f2b779338bc21e185b9"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"重返子序列迫使全序列收敛","source_units":[{"start":84641,"end":85186,"start_line":1779,"end_line":1782,"raw_tex":"Now, by \\cref{prop:recurrence}, there exists \\as a subsequence $\\score_{\\run_{\\runalt}}$ of $\\score_{\\run}$ and some (possibly random) $\\sol\\in\\sols$ such that $\\lim_{\\runalt\\to\\infty} \\fench(\\sol,\\score_{\\run_{\\runalt}}) = 0$.\nSince the limit $\\lim_{\\run\\to\\infty} \\fench(\\sol,\\score_{\\run})$ exists \\as, it follows that $\\lim_{\\run\\to\\infty} \\fench(\\sol,\\score_{\\run}) = 0$.\nThis shows that, with probability $1$, $\\act_{\\run} = \\mirror(\\score_{\\run})$ converges to some (random) minimum point $\\sol$ of \\eqref{eq:opt}, as claimed.\n\\end{proof}","sha256_utf8":"a40e45fb00c81c274c7589561280826e1048f57d4a6b19470318202a01dbab14"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"定义平均方向、反馈与噪声分解","source_units":[{"start":75674,"end":76730,"start_line":1613,"end_line":1639,"raw_tex":"\\begin{proofstep}{Martingale properties of $\\score_{\\run}$}\nFirst, let\n\\begin{equation}\n\\payv(\\state)\n\t= - \\exof{\\nabla\\sobj(\\state;\\sample)} = -\\nabla\\obj(\\state)\n\\end{equation}\ndenote the negative gradient of $\\obj$ at $\\state\\in\\states$, and write\n\\begin{equation}\n\\label{eq:oracle}\n\\est\\payv_{\\run}\n\t= -\\nabla\\sobj(\\act_{\\run};\\sample_{\\run})\n\\end{equation}\nfor the corresponding oracle feedback at stage $\\run$.\nThen, \\cref{alg:SMD} may be written in Robbins\\textendash Monro form as\n\\begin{equation}\n\\label{eq:SA}\n\\score_{\\run+1}\n\t= \\score_{\\run} + \\step_{\\run} \\est\\payv_{\\run}\n\t= \\score_{\\run} + \\step_{\\run} \\bracks{\\payv(\\act_{\\run}) + \\noise_{\\run}},\n\\end{equation}\nwhere\n\\begin{equation}\n\\noise_{\\run}\n\t= \\nabla\\obj(\\act_{\\run}) - \\nabla\\sobj(\\act_{\\run};\\sample_{\\run})\n\\end{equation}\ndenotes the difference between the mean gradient of $\\obj$ at $\\act_{\\run}$ and the $\\run$-th stage gradient sample.%\n\\footnote{Recall here that there is a one-step offset between $\\act_{\\run}$ and $\\sample_{\\run+1}$ at the $\\run$-th iteration of \\ac{SMD}.}\n","sha256_utf8":"a69fad6f4ee26b8a470f583c21de38bb7e59433c947158d7e511c2270cb35826"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"噪声鞅差与二阶矩界纲要","source_units":[{"start":76730,"end":77543,"start_line":1640,"end_line":1660,"raw_tex":"\\begin{subequations}\n\\label{eq:MDS}\nBy construction, $\\noise_{\\run}$ is a \\acl{MDS} relative to the history (natural filtration) $\\filter_{\\run} = \\sigma(\\sample_{1},\\dotsc,\\sample_{\\run})$ of $\\sample_{\\run}$, \\ie\n\\begin{equation}\n\\label{eq:zero-mean}\n\\exof{\\noise_{\\run}\\given\\filter_{\\run-1}}\n\t= 0\n\t\\quad\n\t\\text{for all $\\run$.}\n\\end{equation}\nFurthermore, by \\cref{asm:L2}, it readily follows that $\\noise_{\\run}$ has uniformly bounded second moments, \\ie there exists some finite $\\noisedev\\geq0$ such that\n\\begin{equation}\n\\label{eq:MSE}\n\\exof{\\dnorm{\\noise_{\\run}}^{2} \\given \\filter_{\\run-1}}\n\t\\leq \\noisevar\n\t\\quad\n\t\\text{for all $\\run$},\n\\end{equation}\nimplying in turn that $\\noise_{\\run}$ is bounded in $L^{2}$ (for a more detailed treatment, see \\cref{app:proofs}).\n\\end{subequations}\n\\end{proofstep}","sha256_utf8":"693f6931ce42bf6c7528c658e4a20aa610a834a318fd5a74372843c8eaa66e81"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"负无穷能量反证邻域重返","source_units":[{"start":77545,"end":78032,"start_line":1662,"end_line":1665,"raw_tex":"\\begin{proofstep}{Recurrence of $\\eps$-neighborhoods}\nInvoking the \\acl{LLN} for $L^{2}$-bounded \\aclp{MDS} and using the Fenchel coupling as an energy function (cf.~\\cref{app:proofs}), we show that if $\\act_{\\run}$ remains outside $\\ball(\\sols,\\eps)$ for sufficiently large $\\run$, we must also have $\\fench(\\sols,\\score_{\\run})\\to-\\infty$ \\as.\nThis contradicts the positive-definiteness of $\\fench$, so $\\act_{\\run}$ must enter $\\ball(\\sols,\\eps)$ infinitely often \\as.\n\\end{proofstep}","sha256_utf8":"a959a5d2a46bb3bcd10a91348c65d3f385900b7ada91c80504c035f912094bd9"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"互易性推出区域重返与子序列","source_units":[{"start":78035,"end":78519,"start_line":1668,"end_line":1673,"raw_tex":"\\begin{proofstep}{Recurrence of Fenchel zones}\nBy reciprocity (\\cref{asm:reciprocity}), $\\fball(\\sols, \\delta)$ always contains an $\\eps$-neighborhood of $\\sols$.\n%within it: $\\forall \\delta > 0$, $\\exists \\eps(\\delta) > 0$ such that: $\\ball(\\sols,\\eps) \\subset \\fball(\\sols, \\delta)$.\nSince $\\act_{\\run}$ enters $\\ball(\\sols,\\eps)$ infinitely many times \\as, the same must hold for $\\fball(\\sols, \\delta)$.\nOur claim and \\cref{cor:recurrence} then follow immediately.\n\\end{proofstep}","sha256_utf8":"79dbe2091243ece56cb987490d85387a8348eaf222700a78a6c6037676da3337"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"次梯度与强凸性推出距离下界","source_units":[{"start":140363,"end":141702,"start_line":2700,"end_line":2723,"raw_tex":"\\begin{proofof}{Proof of \\cref{lem:Fenchel}}\nTo prove the first claim, let $\\state =\\mirror(\\dstate) = \\argmax_{\\alt\\state\\in\\states}\\{\\braket{\\dstate}{\\alt\\state} - \\hreg(\\alt\\state)\\}$, so $\\dstate\\in\\pd \\hreg(\\state)$ from standard results in convex analysis \\cite{Roc70}.\n%($y$ is a subgradient of $\\hreg(p)$ if and only if $p=\\mirror(\\dstate)$).\nWe thus get:\n\\begin{equation}\n\\label{eq:Fench-sub}\n\\fench(\\base,\\dstate) = \\hreg(\\base) + \\hreg^{\\ast}(\\dstate) - \\braket{\\dstate}{\\base}\n%\t= \\hreg(\\base) + \\braket{\\dstate}{\\mirror(\\dstate)} - \\hreg(\\mirror(\\dstate)) - \\braket{\\dstate}{\\base}\n\t= \\hreg(\\base) - \\hreg(\\state) - \\braket{\\dstate}{\\base - \\state}.\n\\end{equation}\nSince $\\dstate \\in \\pd \\hreg(\\state)$ and $\\hreg$ is $\\strong$-strongly convex, we also have\n\\begin{equation}\n\\hreg(\\state) + \\tau\\braket{\\dstate}{\\base - \\state}\n\t\\leq \\hreg(\\state + \\tau(\\base - \\state))\n\t\\leq (1-\\tau) \\hreg(\\state) + \\tau \\hreg(\\base) - \\tfrac{1}{2} \\strong \\tau(1-\\tau) \\norm{\\state - \\base}^{2}\n\\end{equation}\nfor all $\\tau\\in[0,1]$, thereby leading to the bound\n\\begin{equation}\n\\label{eq:divbound}\n\\tfrac{1}{2} \\strong(1-\\tau) \\norm{\\state - \\base}^{2}\n\t\\leq \\hreg(\\base) - \\hreg(\\state) - \\braket{\\dstate}{\\base - \\state}\n\t= \\fench(\\base,\\dstate).\n\\end{equation}\nOur claim then follows by letting $\\tau\\to0^{+}$ in \\eqref{eq:divbound}.\n","sha256_utf8":"d5d7d3c400419f0ce43bd8c2214272380a240195119d26d0a0fd4f1c4802930c"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"共轭对偶性推出光滑上界","source_units":[{"start":141703,"end":143292,"start_line":2725,"end_line":2745,"raw_tex":"For our second claim, we start by citing a well-known duality principle for strongly convex functions \\cite[Theorem 12.60]{RW98}:\nIf $f\\from\\R^{\\vdim}\\to\\R\\cup\\{-\\infty,+\\infty\\}$ is proper, lower semi-continuous and convex, its convex conjugate $f^{\\ast}$ is $\\sigma$-strongly convex if and only if $f$ is differentiable and satisfies\n\\begin{equation}\nf(\\alt\\state)\n\t\\leq f(\\state) + \\braket{\\nabla f(\\state)}{\\alt\\state - \\state} + \\frac{1}{2\\sigma} \\norm{\\alt\\state - \\state}^{2}\n\\end{equation}\nfor all $\\state,\\alt\\state\\in\\R^{\\vdim}$.\nIn our case, if we extend $\\hreg$ to all of $\\vecspace$ by setting $\\hreg\\equiv+\\infty$ outside $\\states$, we have that $\\hreg$ is $\\strong$-strongly convex, lower semi-continuous and proper, so $(\\hreg^{\\ast})^{\\ast} = \\hreg$ \\cite[Theorem 11.1]{RW98}.\nIt is also easy to see that $\\hreg^{\\ast}$ is proper, lower semi-continuous and convex (since it is a point-wise maximum of affine functions by definition), so the $\\strong$-strong convexity of $\\hreg = (\\hreg^{\\ast})^{\\ast}$ implies that $\\hreg^{\\ast}$ is differentiable and satisfies\n\\begin{flalign}\n\\label{eq:smoothness}\n\\hreg^{\\ast}(\\alt\\dstate)\n\t&\\leq \\hreg^{\\ast}(\\dstate)\n\t+ \\braket{\\alt\\dstate - \\dstate}{\\nabla \\hreg^{\\ast}(\\dstate)}\n\t+ \\frac{1}{2\\strong} \\dnorm{\\alt\\dstate - \\dstate}^{2}\n\t\\\\\n\t&= \\hreg^{\\ast}(\\dstate)\n\t+ \\braket{\\alt\\dstate - \\dstate}{\\mirror(\\dstate)} +\n\t\\frac{1}{2\\strong} \\dnorm{\\alt\\dstate - \\dstate}^{2}\n\\end{flalign}\nfor all $\\dstate,\\alt\\dstate\\in\\dstates$, where the last equality follows from the fact that $\\nabla \\hreg^{\\ast}(\\dstate) = \\mirror(\\dstate)$.\n","sha256_utf8":"ed3fa368c91d3618308b48cfdb0082cf8cb5200840dd9c8db6ca55d8a17b988a"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"代入耦合定义得到递推上界","source_units":[{"start":143292,"end":143925,"start_line":2746,"end_line":2756,"raw_tex":"Therefore, substituting the preceding inequality in the definition of the Fenchel coupling, we obtain\n\\begin{flalign}\n\\fench(\\state,\\alt\\dstate)\n\t&= \\hreg(\\state) + \\hreg^{\\ast}(\\alt\\dstate) - \\braket{\\alt\\dstate}{\\state}\n\t\\notag\\\\\n\t&\\leq \\hreg(\\state) + \\hreg^{\\ast}(\\dstate) + \\braket{\\alt\\dstate - \\dstate}{\\nabla \\hreg^{\\ast}(\\dstate)} + \\frac{1}{2\\strong} \\dnorm{\\alt\\dstate - \\dstate}^{2} - \\braket{\\alt\\dstate}{\\state}\n\t\\notag\\\\\n\t&= \\fench(\\state,\\dstate) + \\braket{\\alt\\dstate - \\dstate}{\\mirror(\\dstate) - \\state} + \\frac{1}{2\\strong} \\dnorm{\\alt\\dstate - \\dstate}^{2},\n\\end{flalign}\nand our assertion follows.\n\\end{proofof}","sha256_utf8":"3de26efc4ca067a010d2e058f39e7772cd1edf9290f54fcdd4e4f5f73b45420e"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"独立采样推出噪声条件均值为零","source_units":[{"start":143983,"end":145230,"start_line":2761,"end_line":2783,"raw_tex":"\\begin{proofof}{Proof of \\cref{prop:recurrence}}\nOur proof proceeds step-by-step, as discussed in \\cref{sec:recurrence}:\n\n\\setcounter{proofstep}{0}\n\\begin{proofstep}{Martingale properties of $\\score_{\\run}$}\nBy \\cref{asm:L2} and the fact that finiteness of second moments implies finiteness of first moments, we get $\\exof{\\dnorm{\\sobj(\\state;\\sample_{\\run})}} < \\infty$.\nWe then claim that $\\noise_{\\run} = \\nabla\\obj(\\act_{\\run}) - \\nabla\\sobj(\\act_{\\run};\\sample_{\\run})$ is an $L^{2}$-bounded \\acl{MDS} with respect to the natural filtration of $\\sample_{\\run}$.\nIndeed, we have:\n\\begin{enumerate}\n\\addtolength{\\itemsep}{1ex}\n\\item\nSince $\\act_{\\run}$ is $\\filter_{\\run-1}$-measurable and $\\sample_{\\run}$ is \\ac{iid}, we readily get\n\\begin{flalign}\n\\exof{\\noise_{\\run} \\given \\filter_{\\run-1}}\n\t&= \\exof{\\nabla\\obj(\\act_{\\run}) - \\nabla\\sobj(\\act_{\\run};\\sample_{\\run}) \\given \\filter_{\\run-1}}\n\t\\notag\\\\\n\t&= \\exof{\\nabla\\obj(\\act_{\\run}) - \\nabla\\sobj(\\act_{\\run};\\sample_{\\run}) \\given \\sample_{1},\\dotsc,\\sample_{\\run-1}}\n\t\\notag\\\\\n\t&= \\nabla\\obj(\\act_{\\run}) - \\nabla\\obj(\\act_{\\run})\n\t\\notag\\\\\n\t&= 0.\n\\end{flalign}\n%so $\\exof{\\noise_{\\run}} = \\exof{\\exof{\\noise_{\\run} \\given \\filter_{\\run-1}}} = 0$, i.e. $\\noise_{\\run}$ has zero mean.\n","sha256_utf8":"7a18af9912c35c42aaeac897709fa38e4f90a60743f097b186a780198006b987"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"Jensen不等式控制噪声二阶矩","source_units":[{"start":145231,"end":146722,"start_line":2785,"end_line":2818,"raw_tex":"\\item\nFurthermore, by \\cref{asm:L2}, the $L^{2}$ norm of $\\noise$ satisfies\n\\begin{flalign}\n\\label{eq:vbound}\n\\exof{\\dnorm{\\noise_{\\run}}^{2} \\given \\filter_{\\run-1}}\n\t&= \\exof{\\dnorm{\\nabla\\obj(\\act_{\\run}) - \\nabla\\sobj(\\act_{\\run};\\sample_{\\run})}^{2} \\given \\filter_{\\run-1}}\n\t\\notag\\\\\n\t&\\leq 2\\exof{\\dnorm{\\nabla\\obj(\\act_{\\run})}^{2} \\given \\filter_{\\run-1}}\n\t+ 2 \\exof{\\dnorm{\\nabla\\sobj(\\act_{\\run};\\sample_{\\run})}^{2} \\given \\filter_{\\run-1}}\n\t\\notag\\\\\n\t&\\leq 2 \\dnorm{\\nabla\\obj(\\act_{\\run})}^{2}\n\t+ 2 \\vbound^{2}\n%\t+ 2 \\dnorm{\\exof{\\nabla\\sobj(\\act_{\\run};\\sample_{\\run}) \\given \\filter_{\\run-1}}}^{2}\n\t\\notag\\\\\n\t&= 2\\dnorm{\\exof{\\nabla\\sobj(\\act_{\\run};\\sample)}}^{2}\n\t+ 2\\vbound^{2}\n\t\\notag\\\\\n\t&\\leq 2 \\exof{\\dnorm{\\nabla\\sobj(\\act_{\\run};\\sample)}^{2}}\n\t+ 2\\vbound^{2}\n\t\\notag\\\\\n\t&\\leq \\noisevar,\n\\end{flalign}\nwhere we set $\\noisevar = 4\\vbound^{2}$, and we used the dominated convergence theorem to interchange expectation and differentiation in the second line, and Jensen's inequality in the penultimate one.\n%Since $\\states$ is compact, there exists some finite $\\bound$ such that $\\dnorm{\\nabla\\obj(\\state)} \\leq \\bound$ for all $\\state\\in\\states$.\n%Thus, letting $\\vbound = 2\\bound$, we finally obtain\n%\\begin{equation}\n%\\label{eq:vbound}\n%%\\exof{\\dnorm{\\noise_{\\run}}^{2}}\n%%\t= \\exof{\\exof{\\dnorm{\\noise_{\\run}}^{2} \\given \\filter_{\\run-1}}}\n%\\exof{\\dnorm{\\noise_{\\run}}^{2} \\given \\filter_{\\run-1}}\n%\t\\leq \\vbound^{2}.\n%\\end{equation}\n\\end{enumerate}\n\\end{proofstep}\n","sha256_utf8":"ab64a6a900e387072c0d9e4126ffcd5e1888be9645eba0a65a3f8b8014bab524"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"紧性与严格相干性给出负漂移","source_units":[{"start":146723,"end":147631,"start_line":2820,"end_line":2831,"raw_tex":"\\begin{proofstep}{Recurrence of $\\eps$-neighborhoods}\nWe proceed to show that every $\\eps$-neigh\\-bor\\-hood $\\ball(\\sols,\\eps)$ of $\\sols$ is recurrent under $\\act_{\\run}$.\nTo do so, fix some $\\eps>0$ and assume ad absurdum that $\\act_{\\run}$ enters $\\ball(\\sols,\\eps)$ only a finite number of times, so there exists some finite $\\run_{0}$ such that $\\dist(\\sols,\\act_{\\run}) \\geq \\eps$ for all $\\run\\geq\\run_{0}$.\nSince $\\states\\setminus\\ball(\\sols,\\eps)$ is compact, $\\payv(\\state) = -\\nabla \\obj(\\state)$ is continuous in $\\state$;\nfurthermore, letting $\\sol$ be such that $\\braket{\\nabla\\obj(\\state)}{\\state - \\sol} = 0$ only if $\\state\\in\\sols$ (cf.~\\cref{def:VC}),  it follows that there exists some $\\const \\equiv \\const(\\eps) > 0$ such that\n\\begin{equation}\n\\braket{\\payv(\\state)}{\\state - \\sol}\n\t\\leq - \\const\n\t< 0\n\t\\quad\n\t\\text{for all $\\state\\in\\states\\setminus\\ball(\\sols,\\eps)$}.\n\\end{equation}\n","sha256_utf8":"4c1be3ceaa74d41651d52482c50537772d13e350900e3cb6b7da009814682f10"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"耦合递推展开并累加","source_units":[{"start":147631,"end":149404,"start_line":2832,"end_line":2869,"raw_tex":"To proceed, let $R = \\max_{\\state \\in \\states} \\norm{\\state}$ and set $\\beta_{\\run} = \\step_{\\run}^{2}/(2\\strong)$.\nThen, letting $\\fench_{\\run} = \\fench(\\sol,\\score_{\\run})$ and $\\snoise_{\\run} = \\braket{\\noise_{\\run}}{\\act_{\\run}-\\sol}$, \\cref{lem:Fenchel} yields\n\\begin{flalign}\n\\fench_{\\run+1}\n\t= \\fench(\\sol,\\score_{\\run+1})\n\t&= \\fench(\\sol, \\score_{\\run} + \\step_{\\run} \\est\\payv_{\\run})\n\t\\notag\\\\\n\t&\\leq \\fench(\\sol, \\score_{\\run})\n\t+ \\step_{\\run} \\braket{\\payv(\\act_{\\run}) + \\noise_{\\run}}{\\act_{\\run} - \\sol}\n\t+ \\beta_{\\run} \\dnorm{\\est\\payv_{\\run}}^{2}\n\t\\notag\\\\\n\t&=\\fench_{\\run}\n\t+ \\step_{\\run} \\braket{\\payv(\\act_{\\run})}{\\act_{\\run} - \\sol}\n\t+ \\step_{\\run} \\snoise_{\\run}\n\t+ \\beta_{\\run} \\dnorm{\\est\\payv_{\\run}}^{2}\n\t\\notag\\\\\n\t&\\leq \\fench_{\\run}\n\t- \\step_{\\run} \\const\n\t+ \\step_{\\run} \\snoise_{\\run}\n\t+ \\beta_{\\run} \\dnorm{\\est\\payv_{\\run}}^{2}.\n\\end{flalign}\nHence, letting $\\tau_{\\run} = \\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt}$ and telescoping from $\\run_{0}$ to $\\run$, we get\n\\begin{flalign}\n\\label{eq:bound-Fench}\n\\fench_{\\run+1}\n\t&\\leq \\fench_{\\run_{0}}\n\t- \\const \\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt}\n\t+ \\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt} \\snoise_{\\runalt}\n\t+ \\sum_{\\runalt=\\run_{0}}^{\\run} \\beta_{\\runalt} \\dnorm{\\est\\payv_{\\runalt}}^{2}\n\t\\notag\\\\\n\t&= \\fench_{\\run_{0}}\n\t- \\tau_{\\run} \\bracks*{\\const - \\frac{\\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt}\\snoise_{\\runalt}}{\\tau_{\\run}}}\n\t+ \\sum_{\\runalt=\\run_{0}}^{\\run} \\beta_{\\runalt} \\dnorm{\\est\\payv_{\\runalt}}^{2}\n%\t\\notag\\\\\n%\t&\\leq \\fench_{\\run_{0}}\n%\t- \\tau_{\\run} \\bracks*{\\const - \\frac{\\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt}\\snoise_{\\runalt}}{\\tau_{\\run}}}\n%\t+ 2 \\smashoperator{\\sum_{\\runalt=\\run_{0}}^{\\run}} \\beta_{\\runalt} \\bound^{2}\n\\end{flalign}\n","sha256_utf8":"997f68549d093084449ff2ed2dd9de4d8f4437ab633b95d53625701cf9bc2aed"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"标量噪声保持鞅差性质","source_units":[{"start":149405,"end":149812,"start_line":2871,"end_line":2879,"raw_tex":"We now proceed to bound each term of \\eqref{eq:bound-Fench}.\nFirst, by construction, we have\n\\begin{equation}\n\\exof{\\snoise_{\\run} \\given \\filter_{\\run-1}}\n\t= \\exof{\\braket{\\noise_{\\run}}{\\act_{\\run} - \\sol} \\given \\filter_{\\run-1}}\n\t= \\braket{\\exof{\\noise_{\\run} \\given \\filter_{\\run-1}}}{\\act_{\\run} - \\sol}\n\t= 0,\n\\end{equation}\nwhere we used the fact that $\\act_{\\run}$ is $\\filter_{\\run-1}$-measurable.\n","sha256_utf8":"0e79c95e3ea6b5b21f0af7faa1a295a7ff12063066188682e6a31468c345aed8"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"紧域控制标量噪声二阶矩","source_units":[{"start":149812,"end":150390,"start_line":2880,"end_line":2894,"raw_tex":"Moreover, by Young's inequality, we also have\n\\begin{equation}\n\\abs{\\snoise_{\\run}}\n\t= \\abs{\\braket{\\noise_{\\run}}{\\act_{\\run} - \\sol}}\n\t\\leq \\dnorm{\\noise_{\\run}} \\norm{\\act_{\\run} - \\sol}\n\t\\leq 2R\\dnorm{\\noise_{\\run}},\n\\end{equation}\nwhere, as before, $R = \\max_{\\state\\in\\states} \\norm{\\state}$.\n\\cref{eq:vbound} then gives\n\\begin{equation}\n\\exof{\\snoise_{\\run}^{2} \\given \\filter_{\\run-1}}\n\t\\leq (2R)^{2} \\exof{\\dnorm{\\noise_{\\run}}^{2} \\given \\filter_{\\run-1}}\n\t\\leq 4 R^{2} \\noisevar,\n\\end{equation}\nimplying in turn that $\\snoise_{\\run}$ is an $L^{2}$-bounded \\acl{MDS}.\n","sha256_utf8":"79b8c09bb9c7206578eee3f98645cb75ce1a44611ca516982288b9e103a71b32"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"鞅差大数律消去归一化噪声","source_units":[{"start":150390,"end":151335,"start_line":2895,"end_line":2915,"raw_tex":"It then follows that $\\snoise_{\\run}$ satisfies the summability condition\n\\begin{equation}\n\\sum_{\\run=\\run_{0}}^{\\infty} \\frac{\\exof{\\abs{\\step_{\\run} \\snoise_{\\run}}^{2} \\given \\filter_{\\run-1}}}{\\tau_{\\run}^{2}}\n\t\\leq 4R^{2}\\noisevar \\sum_{\\run=\\run_{0}}^{\\infty} \\frac{\\step_{\\run}^{2}}{\\tau_{\\run}^{2}}\n\t< \\infty,\n\\end{equation}\nwhere the last inequality follows from the assumption that $\\step_{\\run}$ is square-summable.\nThus, by the \\acl{LLN} for \\aclp{MDS} (\\cref{thm:mg_convergence1}), we get\n\\begin{equation}\n\\frac{\\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt} \\snoise_{\\runalt}}{\\tau_{\\run}}\n\t\\to 0\n\t\\quad\n\t\\text{as $\\run\\to\\infty$ \\as},\n\\end{equation}\nand, with $\\sum_{\\runalt=\\run_{0}}^{\\infty} \\step_{\\runalt} = \\infty$, we finally obtain\n\\begin{equation}\n\\lim_{\\run\\to\\infty} \\tau_{\\run} \\bracks*{\\const - \\frac{\\sum_{\\runalt=\\run_{0}}^{\\run} \\step_{\\runalt}\\snoise_{\\runalt}}{\\tau_{\\run}}}\n\t= \\infty\n\t\\quad\n\t\\as.\n\\end{equation}\n","sha256_utf8":"1b75e1430b7f85c0b6be2fc228cbb13064ef02fe4cfcdbbc4352fc8de2905c3e"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"次鞅收敛控制二阶累积误差","source_units":[{"start":151442,"end":152206,"start_line":2918,"end_line":2927,"raw_tex":"For the last term of \\eqref{eq:bound-Fench}, let $S_{\\run} = \\sum_{\\runalt=\\run_{0}}^{\\run} \\beta_{\\runalt} \\dnorm{\\est\\payv_{\\runalt}}^{2}$, so $S_{\\run}$ is $\\filter_{\\run}$-measurable and nondecrasing.\nIn addition, we have\n\\begin{equation}\n\\exof{S_{\\run}}\n\t= \\sum_{\\runalt=\\run_{0}}^{\\run} \\beta_{\\runalt} \\exof{\\dnorm{\\est\\payv_{\\runalt}}^{2}}\n\t\\leq \\vbound^{2} \\sum_{\\runalt=\\run_{0}}^{\\run} \\beta_{\\runalt}\n\t< \\infty,\n\\end{equation}\nwith the last step following from \\eqref{eq:vbound}.\nThis implies that $S_{\\run}$ is an $L^{1}$-bounded submartingale so, by Doob's submartingale convergence theorem (\\cref{thm:mg_convergence2}), $S_{\\run}$ converges almost surely to some random variable $S_{\\infty}$, \\ie the last term of \\eqref{eq:bound-Fench} is bounded.\n","sha256_utf8":"b37e3901cda9f62d2bcc88936838b89327ccddd597011c4f3a7f1d705402390d"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"能量非负性排除永久远离解集","source_units":[{"start":152206,"end":152562,"start_line":2928,"end_line":2937,"raw_tex":"Hence, combining all of the above, we finally obtain\n\\begin{equation}\n\\limsup_{\\run\\to\\infty} \\fench_{\\run}\n\t= - \\infty\n\t\\quad\n\t\\as,\n\\end{equation}\ncontradicting the positive-definiteness of the Fenchel coupling (cf.~\\cref{lem:Fenchel}).\nWe thus conclude that $\\act_{\\run}$ enters $\\ball(\\sols,\\eps)$ infinitely many times \\as, as claimed.\n\\end{proofstep}\n","sha256_utf8":"5ab06b52c419e3754529eaa0f55bbcfa2a8a9052f3d4a392798de70fdb750dc1"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"邻域重返传递至Fenchel区域","source_units":[{"start":152563,"end":153108,"start_line":2939,"end_line":2944,"raw_tex":"\\begin{proofstep}{Recurrence of Fenchel zones}\nUsing the reciprocity of the Fenchel coupling (\\cref{asm:reciprocity}), we show below that every Fenchel zone $\\fball(\\sols,\\delta)$ of $\\sols$ contains an $\\eps$-neighborhood of $\\sols$.\n%\\begin{equation}\\label{eq:contains}\n%\\forall \\delta > 0,\\exists \\eps(\\delta) > 0, \\ball(\\sols,\\eps) \\subset \\fball(\\sols, \\delta).\n%\\end{equation} \nThen, given that $\\act_{\\run}$ enters $\\ball(\\sols,\\eps)$ infinitely often (per the previous step), it will also enter $\\fball(\\sols, \\delta)$ infinitely often.\n","sha256_utf8":"44f89e04d4af677ef79d5f22b55f2e16e8b2ecf27bfcb7ea8fe96e38fd11af4a"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"紧性与互易性证明区域包含邻域","source_units":[{"start":153109,"end":154219,"start_line":2946,"end_line":2959,"raw_tex":"To establish this claim, assume instead that there is no $\\eps$-ball $\\ball(\\sols,\\eps)$ contained in $\\fball(\\sols, \\delta)$.\nThen, for all $\\runalt>0$ there exists some $\\dstate_{\\runalt}\\in\\dstates$ such that $\\dist(\\sols,\\mirror(\\dstate_{\\runalt})) = 1/\\runalt$ but $\\fench(\\sols, y_{\\delta}) \\geq \\eps$.\nThis produces a sequence $(\\dstate_{\\runalt})_{\\runalt=1}^{\\infty}$ such that $\\dist(\\sols,\\mirror(\\dstate_{\\runalt})) \\to 0$ but $\\fench(\\sols, \\dstate_{\\runalt}) \\geq \\eps$.\nSince $\\states$ is compact and $\\sols$ is closed, we can assume without loss of generality that $\\mirror(\\dstate_{\\runalt})\\to\\base$ for some $\\base\\in\\sols$ (at worst, we only need to descend to a subsequence of $\\dstate_{\\runalt}$).\nWe thus get\n\\begin{equation}\n\\eps\n\t\\leq \\fench(\\sols,\\dstate_{\\runalt})\n\t\\leq \\fench(\\base,\\dstate_{\\runalt}).\n\\end{equation}\nHowever, since $\\mirror(\\dstate_{\\runalt})\\to\\base$, \\cref{asm:reciprocity} gives $\\fench(\\base,\\dstate_{\\runalt})\\to0$, a contradiction.\nWe conclude that $\\fball(\\sols,\\delta)$ contains an $\\eps$-neighborhood of $\\sols$, as required.\n\\end{proofstep}\n\\end{proofof}","sha256_utf8":"50f63c75c8b179f0851497bbfa2c1f7f1421620f454166b8fb2a987b4fd3302e"}]},{"id":"S042","semantic_type":"external_fact","kind":"external_fact","title":"凸次梯度最优性","source_units":[],"declared_external":true,"declaration_text":["For a proper closed convex regularizer h extended by +infinity outside its domain, x maximizes <y,x>-h(x) exactly when y belongs to the convex subdifferential of h at x. When h* is differentiable, its gradient equals this unique maximizer. These are the convex-analysis facts cited to Roc70."]},{"id":"S043","semantic_type":"external_fact","kind":"external_fact","title":"强凸共轭光滑性与双共轭","source_units":[],"declared_external":true,"declaration_text":["A proper lower-semicontinuous convex function equals its biconjugate. If h is K-strongly convex, h* is differentiable and satisfies h*(y') <= h*(y)+<grad h*(y),y'-y>+||y'-y||_*^2/(2K). The author cites RW98, Theorems 11.1 and 12.60."]},{"id":"S044","semantic_type":"external_fact","kind":"external_fact","title":"鞅差强大数律","source_units":[],"declared_external":true,"declaration_text":["For a martingale M_n=sum_{k=1}^n d_k and deterministic nondecreasing positive tau_n tending to infinity, if sum_n tau_n^(-p) E[|d_n|^p | F_{n-1}] is finite almost surely for some p in [1,2], then M_n/tau_n tends to zero almost surely. This is the standard theorem stated in B0145 and attributed to HH80."]},{"id":"S045","semantic_type":"external_fact","kind":"external_fact","title":"Doob次鞅收敛定理","source_units":[],"declared_external":true,"declaration_text":["An L1-bounded submartingale converges almost surely to an integrable random variable. This is the standard theorem stated in B0147 and attributed to HH80."]},{"id":"S046","semantic_type":"external_fact","kind":"external_fact","title":"Gladyshev收敛引理","source_units":[],"declared_external":true,"declaration_text":["If nonnegative random variables a_n satisfy E[a_{n+1}|a_1,...,a_n] <= (1+delta_n)a_n+epsilon_n for nonnegative deterministic summable sequences delta_n and epsilon_n, then a_n converges almost surely to a finite nonnegative random variable. This is B0072, attributed to Pol87, p. 49."]},{"id":"S047","semantic_type":"external_fact","kind":"external_fact","title":"期望交换与条件期望基本性质","source_units":[],"declared_external":true,"declaration_text":["Dominated convergence permits differentiation under expectation when its domination hypotheses hold. Jensen's inequality bounds the squared norm of an expectation by the expectation of the squared norm. Independence of a fresh sample from the past permits conditional averaging at a predictable state, and predictable factors can be taken outside conditional expectations."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S007","kind":"supports"},{"source_id":"S003","target_id":"S033","kind":"supports"},{"source_id":"S003","target_id":"S036","kind":"supports"},{"source_id":"S003","target_id":"S041","kind":"supports"},{"source_id":"S004","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S025","kind":"supports"},{"source_id":"S005","target_id":"S032","kind":"supports"},{"source_id":"S005","target_id":"S038","kind":"supports"},{"source_id":"S005","target_id":"S022","kind":"supports"},{"source_id":"S006","target_id":"S032","kind":"supports"},{"source_id":"S006","target_id":"S036","kind":"supports"},{"source_id":"S007","target_id":"S024","kind":"supports"},{"source_id":"S007","target_id":"S031","kind":"supports"},{"source_id":"S007","target_id":"S033","kind":"supports"},{"source_id":"S007","target_id":"S041","kind":"supports"},{"source_id":"S008","target_id":"S021","kind":"supports"},{"source_id":"S008","target_id":"S033","kind":"supports"},{"source_id":"S009","target_id":"S024","kind":"supports"},{"source_id":"S009","target_id":"S031","kind":"supports"},{"source_id":"S009","target_id":"S034","kind":"supports"},{"source_id":"S009","target_id":"S037","kind":"supports"},{"source_id":"S009","target_id":"S038","kind":"supports"},{"source_id":"S009","target_id":"S022","kind":"supports"},{"source_id":"S010","target_id":"S024","kind":"supports"},{"source_id":"S011","target_id":"S028","kind":"supports"},{"source_id":"S011","target_id":"S029","kind":"supports"},{"source_id":"S012","target_id":"S028","kind":"supports"},{"source_id":"S012","target_id":"S030","kind":"supports"},{"source_id":"S013","target_id":"S027","kind":"supports"},{"source_id":"S013","target_id":"S041","kind":"supports"},{"source_id":"S013","target_id":"S023","kind":"supports"},{"source_id":"S014","target_id":"S026","kind":"supports"},{"source_id":"S014","target_id":"S027","kind":"supports"},{"source_id":"S014","target_id":"S033","kind":"supports"},{"source_id":"S014","target_id":"S040","kind":"supports"},{"source_id":"S014","target_id":"S041","kind":"supports"},{"source_id":"S015","target_id":"S021","kind":"supports"},{"source_id":"S015","target_id":"S026","kind":"supports"},{"source_id":"S015","target_id":"S034","kind":"supports"},{"source_id":"S015","target_id":"S039","kind":"supports"},{"source_id":"S015","target_id":"S023","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S023","kind":"supports"},{"source_id":"S017","target_id":"S023","kind":"supports"},{"source_id":"S018","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S015","kind":"supports"},{"source_id":"S020","target_id":"S016","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S002","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S002","kind":"supports"},{"source_id":"S023","target_id":"S002","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S031","kind":"supports"},{"source_id":"S024","target_id":"S034","kind":"supports"},{"source_id":"S024","target_id":"S021","kind":"supports"},{"source_id":"S024","target_id":"S018","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S018","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S018","kind":"supports"},{"source_id":"S027","target_id":"S018","kind":"supports"},{"source_id":"S028","target_id":"S019","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S019","kind":"supports"},{"source_id":"S030","target_id":"S019","kind":"supports"},{"source_id":"S031","target_id":"S035","kind":"supports"},{"source_id":"S031","target_id":"S022","kind":"supports"},{"source_id":"S031","target_id":"S020","kind":"supports"},{"source_id":"S032","target_id":"S036","kind":"supports"},{"source_id":"S032","target_id":"S020","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S020","kind":"supports"},{"source_id":"S034","target_id":"S039","kind":"supports"},{"source_id":"S034","target_id":"S020","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S035","target_id":"S020","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S020","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S020","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S020","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S020","kind":"supports"},{"source_id":"S040","target_id":"S020","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S041","target_id":"S020","kind":"supports"},{"source_id":"S042","target_id":"S028","kind":"supports"},{"source_id":"S042","target_id":"S029","kind":"supports"},{"source_id":"S043","target_id":"S029","kind":"supports"},{"source_id":"S044","target_id":"S026","kind":"supports"},{"source_id":"S044","target_id":"S037","kind":"supports"},{"source_id":"S045","target_id":"S038","kind":"supports"},{"source_id":"S046","target_id":"S022","kind":"supports"},{"source_id":"S047","target_id":"S007","kind":"supports"},{"source_id":"S047","target_id":"S025","kind":"supports"},{"source_id":"S047","target_id":"S031","kind":"supports"},{"source_id":"S047","target_id":"S032","kind":"supports"},{"source_id":"S047","target_id":"S035","kind":"supports"},{"source_id":"S047","target_id":"S022","kind":"supports"}]}}
{"record_id":"1709.05747v4","paper":{"title":"Douglas-Rachford splitting and ADMM for nonconvex optimization: tight convergence results","authors":["Andreas Themelis","Panagiotis Patrinos"],"arxiv_id":"1709.05747","version":"v4","journal":"SIAM Journal on Optimization","doi":"10.1137/18m1163993","abstract_url":"https://arxiv.org/abs/1709.05747v4","pdf_url":"https://arxiv.org/pdf/1709.05747v4"},"context":{"raw_tex":"\\begin{equation}\\label{eq:P}\n\t\\minimize_{s\\in\\R^p} \\varphi(s)\\equiv\\sDRS(s)+\\nsDRS(s).\n\\end{equation}\n\nHere, \\(\\func{\\sDRS,\\nsDRS}{\\R^p}{\\Rinf}\\) are proper, lower semicontinuous (lsc), extended-real-valued functions (\\(\\Rinf\\coloneqq\\R\\cup\\set\\infty\\) denotes the extended-real line).\nStarting from some \\(s\\in\\R^p\\), one DR-iteration applied to \\eqref{eq:P} with \\emph{stepsize} \\(\\gamma>0\\) and \\emph{relaxation} parameter \\(\\lambda>0\\) amounts to\n\\[\n\t\\tag{DRS}\\label{DRS}\n% \t\\renewcommand\\arraystretch{1.5}\n\t\\begin{cases}[rl]\n\t\tu\\phantom{^+}\n\t{}\\in{} &\n\t\t\\prox_{\\gamma\\sDRS}(s)\n\t\\\\\n\t\tv\\phantom{^+}\n\t{}\\in{} &\n\t\t\\prox_{\\gamma\\nsDRS}(2u-s)\n\t\\\\\n\t\ts^+\n\t{}={} &\n\t\ts+\\lambda(v-u).\n\t\\end{cases}\n\\]\n\n\t\t\tThe extended-real line is \\(\\Rinf=\\R\\cup\\set{\\infty}\\).\nThe positive and negative parts of \\(r\\in\\R\\) are defined respectively as\n\\(\n\t[r]_+\n{}\\coloneqq{}\n\t\\max\\set{0,r}\n\\)\nand\n\\(\n\t[r]_-\n{}\\coloneqq{}\n\t\\max\\set{0,-r}\n\\),\nso that \\(r=[r]_+-[r]_-\\).\nWe adopt the convention that \\(\\nicefrac10=\\infty\\).\n\n\nFor a set \\(E\\) and a sequence \\(\\seq{x^k}\\) we write \\(\\seq{x^k}\\subset E\\) to indicate that \\(x^k\\in E\\) for all \\(k\\in\\N\\).\nWe say that \\(\\seq{x^k}\\subset\\R^n\\) is \\DEF{summable} if \\(\\sum_{k\\in\\N}\\|x^k\\|\\) is finite, and \\DEF{square-summable} if \\(\\seq{\\|x^k\\|^2}\\) is summable.\n\n\nThe \\DEF{domain} of an extended-real-valued function \\(\\func{h}{\\R^n}{\\Rinf}\\) is the set\n\\(\n\t\\dom h\n{}\\coloneqq{}\n\t\\set{x\\in\\R^n}[\n\t\th(x)<\\infty\n\t]\n\\),\nwhile its \\DEF{epigraph} is the set\n\\(\n\t\\epi h\n{}\\coloneqq{}\n\t\\set{(x,\\alpha)\\in\\R^n\\times\\R}[\n\t\th(x)\\leq\\alpha\n\t]\n\\).\n\\(h\\) is said to be \\DEF{proper} if \\(\\dom h\\neq\\emptyset\\), and \\DEF{lower semicontinuous (lsc)} if \\(\\epi h\\) is a closed subset of \\(\\R^{n+1}\\).\nFor \\(\\alpha\\in\\R\\), \\(\\lev_{\\leq\\alpha}h\\) is the \\DEF{\\(\\alpha\\)-level set} of \\(h\\), \\ie\n\\(\n\t\\lev_{\\leq\\alpha}h\n{}\\coloneqq{}\n\t\\set{x\\in\\R^n}[\n\t\th(x)\\leq\\alpha\n\t]\n\\).\nWe say that \\(h\\) is \\emph{level bounded} if \\(\\lev_{\\leq\\alpha}h\\) is bounded for all \\(\\alpha\\in\\R\\).\n\n\nWe use the notation \\(\\ffunc H{\\R^n}{\\R^m}\\) to indicate a point-to-set mapping \\(\\func H{\\R^n}{\\mathcal P(\\R^m)}\\), where \\(\\mathcal P(\\R^m)\\) is the power set of \\(\\R^m\\) (the set of all subsets of \\(\\R^m\\)).\nThe \\DEF{graph} of \\(H\\) is the set\n\\(\n\t\\graph H\n{}\\coloneqq{}\n\t\\set{(x,y)\\in\\R^n\\times\\R^m}[\n\t\ty\\in H(x)\n\t]\n\\).\n \n% =============================================================================================== %\n\nWe denote by \\(\\ffunc{\\hat\\partial h}{\\R^n}{\\R^n}\\) the \\DEF{regular subdifferential} of \\(h\\), where\n\\begin{equation}\\label{eq:hatpartial}\n\tv\\in\\hat\\partial h(\\bar x)\n\\quad\\Leftrightarrow\\quad\n\t\\liminf_{\\limsubstack{x&\\to&\\bar x\\\\x&\\neq&\\bar x}}{\n\t\t\\frac{h(x)-h(\\bar x)-\\innprod{v}{x-\\bar x}}{\\|x-\\bar x\\|}\n\t}\n{}\\geq{}\n\t0.\n\\end{equation}\nA necessary condition for local minimality of \\(x\\) for \\(h\\) is \\(0\\in\\hat\\partial h(x)\\), see \\cite[Thm. 10.1]{rockafellar2011variational}.\nThe (limiting) \\DEF{subdifferential} of \\(h\\) is \\(\\ffunc{\\partial h}{\\R^n}{\\R^n}\\), where\n\\(\n\tv\\in\\partial h(\\bar x)\n\\)\niff there exists a sequence \\(\\seq{x^k,v^k}\\subseteq\\graph\\hat\\partial h\\) such that\n\\[\n\t\\lim_{k\\to\\infty}(x^k,h(x^k),v^k)\n{}={}\n\t(x,h(x),v).\n\\]\n\n\n\t\t\tThe class of functions \\(\\func{h}{\\R^n}{\\R}\\) that are \\(k\\) times continuously differentiable is denoted as \\(\\cont^k(\\R^n)\\).\nWe write \\(h\\in\\cont^{1,1}(\\R^n)\\) to indicate that \\(h\\in\\cont^1(\\R^n)\\) and that \\(\\nabla h\\) is Lipschitz continuous with modulus \\(L_h\\).\nTo simplify the terminology, we will say that such an \\(h\\) is \\DEF{\\(L_h\\)-smooth}.\nIt follows from \\cite[Prop. A.24]{bertsekas2016nonlinear} that if \\(h\\) is \\(L_h\\)-smooth, then\n\\(\n\t|h(y)-h(x)-\\innprod{\\nabla h(x)}{y-x}|\n{}\\leq{}\n\t\\tfrac{L_h}{2}\\|y-x\\|^2\n\\)\nfor all \\(x,y\\in\\R^n\\).\nIn particular, there exists \\(\\sigma_h\\in[-L_h,L_h]\\) such that \\(h\\) is \\DEF{\\(\\sigma_h\\)-hypoconvex}, in the sense that \\(h-\\tfrac{\\sigma_h}{2}\\|{}\\cdot{}\\|^2\\) is a convex function.\nThus, every \\(L_h\\)-smooth and \\(\\sigma_h\\)-hypoconvex function \\(h\\) satisfies\n\\begin{equation}\\label{eq:LipBound}\n\t\\tfrac{\\sigma_h}{2}\n\t\\|y-x\\|^2\n{}\\leq{}\n\th(y)-h(x)-\\innprod{\\nabla h(x)}{y-x}\n{}\\leq{}\n\t\\tfrac{L_h}{2}\n\t\\|y-x\\|^2\n\\quad\n\t\\forall x,y\\in\\R^n.\n\\end{equation}\nBy applying \\cite[Thm. 2.1.5]{nesterov2003introductory} to the (convex) function \\(\\psi=h-\\tfrac\\sigma2\\|{}\\cdot{}\\|^2\\) we obtain that this is equivalent to having\n\\begin{equation}\\label{eq:innprod}\n\t\\sigma_h\n\t\\|y-x\\|^2\n{}\\leq{}\n\t\\innprod{\\nabla h(y)-\\nabla h(x)}{y-x}\n{}\\leq{}\n\tL_h\n\t\\|y-x\\|^2\n\\quad\n\t\\forall x,y\\in\\R^n.\n\\end{equation}\nNote that \\(\\sigma\\)-hypoconvexity generalizes the notion of (strong) convexity by allowing negative strong convexity moduli.\nIn fact, if \\(\\sigma=0\\) then \\(\\sigma\\)-hypoconvexity reduces to convexity, while for \\(\\sigma>0\\) it denotes \\(\\sigma\\)-strong convexity.\n\n\n\t\t\tThe \\DEF{proximal mapping} of \\(\\func{h}{\\R^n}{\\Rinf}\\) with parameter \\(\\gamma>0\\) is \\(\\ffunc{\\prox_{\\gamma h}}{\\R^n}{\\dom h}\\) defined as\n\\begin{equation}\n\t\\prox_{\\gamma h}(x)\n{}\\coloneqq{}\n\t\\argmin_{w\\in\\R^n}\\set{\n\t\th(w) + \\tfrac{1}{2\\gamma}\\|w-x\\|^2\n\t}.\n\\end{equation}\nWe say that a function \\(h\\) is \\DEF{prox-bounded} if \\(h+\\tfrac{1}{2\\gamma}\\|{}\\cdot{}\\|^2\\) is lower bounded for some \\(\\gamma>0\\).\nThe supremum of all such \\(\\gamma\\)\n% --- which is possibly infinite, as it is the case when \\(h\\) is lower bounded or convex ---\nis the \\emph{threshold of prox-boundedness of \\(h\\)}, denoted as \\(\\gamma_h\\).\nIf \\(h\\) is lsc, then \\(\\prox_{\\gamma h}\\) is nonempty- and compact-valued over \\(\\R^n\\) for all \\(\\gamma\\in(0,\\gamma_h)\\) \\cite[Thm. 1.25]{rockafellar2011variational}.\nConsequently, the value function of the minimization problem defining the proximal mapping, namely the \\DEF{Moreau envelope} with stepsize \\(\\gamma\\in(0,\\gamma_h)\\), denoted by \\(\\func{h^\\gamma}{\\R^n}{\\R}\\) and defined as\n\\begin{equation}\n\t\\fillwidthof[r]{\n\t\t\\prox_{\\gamma h}(x)\n\t}{\n\t\th^\\gamma(x)\n\t}\n{}\\coloneqq{}\n\t\\fillwidthof[l]{\n\t\t\\argmin_{w\\in\\R^n}\\set{\n\t\t\th(w) + \\tfrac{1}{2\\gamma}\\|w-x\\|^2\n\t\t}.\n\t}{\n\t\t\\inf_{w\\in\\R^n}\\set{\n\t\t\th(w) + \\tfrac{1}{2\\gamma}\\|w-x\\|^2\n\t\t},\n\t}\n\\end{equation}\nis everywhere finite and, in fact, strictly continuous \\cite[Ex. 10.32]{rockafellar2011variational}.\nMoreover, the necessary optimality conditions of the problem defining \\(\\prox_{\\gamma g}\\) together with \\cite[Thm. 10.1 and Ex. 8.8]{rockafellar2011variational} imply that\n\\begin{equation}\\label{eq:prox:subdiff}\n\t\\tfrac1\\gamma(x-\\bar x)\n{}\\in{}\n\t\\hat\\partial h(\\bar x)\n\\quad\n\t\\forall\\bar x\\in\\prox_{\\gamma h}(x).\n\\end{equation}\n\n\n\\begin{ass}[Requirements for the DRS formulation \\eqref{eq:P}]\\label{ass:DRS}%\n\tThe following hold\n\t\\begin{enumerate}\n\t\\item\\label{ass:DRS:f}%\n\t\t\\(\\sDRS\\in\\cont^{1,1}(\\R^n)\\) is \\(L_{\\sDRS}\\)-smooth, hence \\(\\sigma_{\\sDRS}\\)-hypoconvex for some \\(\\sigma_{\\sDRS}\\in[-L_{\\sDRS},L_{\\sDRS}]\\).\n\t\\item\\label{ass:DRS:g}%\n\t\t\\(\\nsDRS\\) is proper and lsc.\n\t\\item\\label{ass:DRS:LB}%\n\t\tProblem \\eqref{eq:P} has a solution, that is, \\(\\argmin\\varphi\\neq\\emptyset\\).\n\t\\end{enumerate}\n\\end{ass}\n\n\\begin{rem}[Feasible stepsizes for \\ref{DRS}]\\label{thm:proxbounded}%\n\tUnder \\Cref{ass:DRS}, both \\(\\sDRS\\) and \\(\\nsDRS\\) are prox-bounded with threshold at least \\(\\nicefrac{1}{L_{\\sDRS}}\\), and in particular \\ref{DRS} iterations are well defined for all \\(\\gamma\\in(0,\\nicefrac{1}{L_{\\sDRS}})\\).\n\n\nStarting from \\(s\\in\\R^p\\), let us consider variables \\((u,v)\\) generated by a \\ref{DRS} step under \\Cref{ass:DRS}.\nAs first noted in \\cite{patrinos2014douglas}, from the relation \\(s=u+\\gamma\\nabla\\sDRS(u)\\) (see \\cref{thm:proxfEquiv}) it follows that\n\\begin{equation}\\label{eq:vFBS}\n\tv\\in\\FB u\n\\end{equation}\nis the result of a \\DEF{forward-backward step} at \\(u\\), amounting to\n\\begin{align}\\label{eq:DRSv}\n\tv\n{}\\in{}\n\t\\argmin_{w\\in\\R^p}{} &\n\t\\set{\n\t\t\\nsDRS(w)\n\t\t{}+{}\n\t\t\\smashunderbracket{\n\t\t\t\\sDRS(u)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla\\sDRS(u)}{w-u}\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2\\gamma}\\|w-u\\|^2\n\t\t}{}\n\t},\n\\shortintertext{%\n\tsee \\eg \\cite{bolte2014proximal,themelis2018forward} for an extensive discussion on nonconvex forward-backward splitting (FBS).\n\tThis shows that \\(v\\) is the result of the minimization of a \\emph{majorization model} for the original function \\(\\varphi=\\sDRS+\\nsDRS\\), where the smooth function \\(\\sDRS\\) is replaced by the quadratic upper bound emphasized by the under-bracket in \\eqref{eq:DRSv}.\n\tFirst introduced in \\cite{patrinos2014douglas} for convex problems, the \\DEF{Douglas-Rachford envelope} (DRE) is the function \\(\\func{\\DRE}{\\R^n}{\\R}\\) defined as%\n}\n\\label{eq:DRE}\n\t\\DRE(s)\n{}\\coloneqq{}\n\t\\min_{w\\in\\R^p}{} &\n\t\\set{\n\t\t\\nsDRS(w)\n\t\t{}+{}\n\t\t\\sDRS(u)\n\t\t{}+{}\n\t\t\\innprod{\\nabla\\sDRS(u)}{w-u}\n\t\t{}+{}\n\t\t\\tfrac{1}{2\\gamma}\\|w-u\\|^2\n\t}.\n\\end{align}\n\nNamely, rather than the \\emph{minimizer} \\(v\\), \\(\\DRE(s)\\) is the \\emph{value} of the minimization problem \\eqref{eq:DRSv} defining the \\(v\\)-update in \\eqref{DRS}.\nThe expression \\eqref{eq:DRE} emphasizes the close connection that the DRE has with the \\DEF{forward-backward envelope} (FBE) as in \\cite{themelis2018forward}, here denoted \\(\\FBE\\), namely\n\\begin{equation}\\label{eq:FBEu}\n\t\\DRE(s)\n{}={}\n\t\\FBE(u),\n\\quad\n\t\\text{where \\(u=\\prox_{\\gamma\\sDRS}(s)\\).}\n\\end{equation}","units":[{"start":4734,"end":4835,"start_line":117,"end_line":119,"raw_tex":"\\begin{equation}\\label{eq:P}\n\t\\minimize_{s\\in\\R^p} \\varphi(s)\\equiv\\sDRS(s)+\\nsDRS(s).\n\\end{equation}","sha256_utf8":"70721885044c0851f87e8df23eb94f7efd6499744c02415ba663ebff5e4bedba"},{"start":4836,"end":5427,"start_line":120,"end_line":138,"raw_tex":"Here, \\(\\func{\\sDRS,\\nsDRS}{\\R^p}{\\Rinf}\\) are proper, lower semicontinuous (lsc), extended-real-valued functions (\\(\\Rinf\\coloneqq\\R\\cup\\set\\infty\\) denotes the extended-real line).\nStarting from some \\(s\\in\\R^p\\), one DR-iteration applied to \\eqref{eq:P} with \\emph{stepsize} \\(\\gamma>0\\) and \\emph{relaxation} parameter \\(\\lambda>0\\) amounts to\n\\[\n\t\\tag{DRS}\\label{DRS}\n% \t\\renewcommand\\arraystretch{1.5}\n\t\\begin{cases}[rl]\n\t\tu\\phantom{^+}\n\t{}\\in{} &\n\t\t\\prox_{\\gamma\\sDRS}(s)\n\t\\\\\n\t\tv\\phantom{^+}\n\t{}\\in{} &\n\t\t\\prox_{\\gamma\\nsDRS}(2u-s)\n\t\\\\\n\t\ts^+\n\t{}={} &\n\t\ts+\\lambda(v-u).\n\t\\end{cases}\n\\]","sha256_utf8":"0f28b0447d4ddb4d6bff1aeee06cb7e313d37ef6b860e509e3892bb0c36640fa"},{"start":33731,"end":34034,"start_line":479,"end_line":493,"raw_tex":"\t\t\tThe extended-real line is \\(\\Rinf=\\R\\cup\\set{\\infty}\\).\nThe positive and negative parts of \\(r\\in\\R\\) are defined respectively as\n\\(\n\t[r]_+\n{}\\coloneqq{}\n\t\\max\\set{0,r}\n\\)\nand\n\\(\n\t[r]_-\n{}\\coloneqq{}\n\t\\max\\set{0,-r}\n\\),\nso that \\(r=[r]_+-[r]_-\\).\nWe adopt the convention that \\(\\nicefrac10=\\infty\\).\n","sha256_utf8":"31933ae8c30d33ebcdb903c6cbc4c325eb57d72d5442cd1fc80ef8bd8083ed4e"},{"start":34591,"end":34874,"start_line":500,"end_line":501,"raw_tex":"For a set \\(E\\) and a sequence \\(\\seq{x^k}\\) we write \\(\\seq{x^k}\\subset E\\) to indicate that \\(x^k\\in E\\) for all \\(k\\in\\N\\).\nWe say that \\(\\seq{x^k}\\subset\\R^n\\) is \\DEF{summable} if \\(\\sum_{k\\in\\N}\\|x^k\\|\\) is finite, and \\DEF{square-summable} if \\(\\seq{\\|x^k\\|^2}\\) is summable.\n","sha256_utf8":"fc009399c22424cb0ccf7c5deb83e51d512d5248ae2e29c6497638a6e9bfc5fe"},{"start":34976,"end":35670,"start_line":505,"end_line":530,"raw_tex":"The \\DEF{domain} of an extended-real-valued function \\(\\func{h}{\\R^n}{\\Rinf}\\) is the set\n\\(\n\t\\dom h\n{}\\coloneqq{}\n\t\\set{x\\in\\R^n}[\n\t\th(x)<\\infty\n\t]\n\\),\nwhile its \\DEF{epigraph} is the set\n\\(\n\t\\epi h\n{}\\coloneqq{}\n\t\\set{(x,\\alpha)\\in\\R^n\\times\\R}[\n\t\th(x)\\leq\\alpha\n\t]\n\\).\n\\(h\\) is said to be \\DEF{proper} if \\(\\dom h\\neq\\emptyset\\), and \\DEF{lower semicontinuous (lsc)} if \\(\\epi h\\) is a closed subset of \\(\\R^{n+1}\\).\nFor \\(\\alpha\\in\\R\\), \\(\\lev_{\\leq\\alpha}h\\) is the \\DEF{\\(\\alpha\\)-level set} of \\(h\\), \\ie\n\\(\n\t\\lev_{\\leq\\alpha}h\n{}\\coloneqq{}\n\t\\set{x\\in\\R^n}[\n\t\th(x)\\leq\\alpha\n\t]\n\\).\nWe say that \\(h\\) is \\emph{level bounded} if \\(\\lev_{\\leq\\alpha}h\\) is bounded for all \\(\\alpha\\in\\R\\).\n","sha256_utf8":"1c28b3b97b790e88faea634beb509444b69e0827ce55b39e339ded7577f15d22"},{"start":36182,"end":37356,"start_line":538,"end_line":570,"raw_tex":"We use the notation \\(\\ffunc H{\\R^n}{\\R^m}\\) to indicate a point-to-set mapping \\(\\func H{\\R^n}{\\mathcal P(\\R^m)}\\), where \\(\\mathcal P(\\R^m)\\) is the power set of \\(\\R^m\\) (the set of all subsets of \\(\\R^m\\)).\nThe \\DEF{graph} of \\(H\\) is the set\n\\(\n\t\\graph H\n{}\\coloneqq{}\n\t\\set{(x,y)\\in\\R^n\\times\\R^m}[\n\t\ty\\in H(x)\n\t]\n\\).\n \n% =============================================================================================== %\n\nWe denote by \\(\\ffunc{\\hat\\partial h}{\\R^n}{\\R^n}\\) the \\DEF{regular subdifferential} of \\(h\\), where\n\\begin{equation}\\label{eq:hatpartial}\n\tv\\in\\hat\\partial h(\\bar x)\n\\quad\\Leftrightarrow\\quad\n\t\\liminf_{\\limsubstack{x&\\to&\\bar x\\\\x&\\neq&\\bar x}}{\n\t\t\\frac{h(x)-h(\\bar x)-\\innprod{v}{x-\\bar x}}{\\|x-\\bar x\\|}\n\t}\n{}\\geq{}\n\t0.\n\\end{equation}\nA necessary condition for local minimality of \\(x\\) for \\(h\\) is \\(0\\in\\hat\\partial h(x)\\), see \\cite[Thm. 10.1]{rockafellar2011variational}.\nThe (limiting) \\DEF{subdifferential} of \\(h\\) is \\(\\ffunc{\\partial h}{\\R^n}{\\R^n}\\), where\n\\(\n\tv\\in\\partial h(\\bar x)\n\\)\niff there exists a sequence \\(\\seq{x^k,v^k}\\subseteq\\graph\\hat\\partial h\\) such that\n\\[\n\t\\lim_{k\\to\\infty}(x^k,h(x^k),v^k)\n{}={}\n\t(x,h(x),v).\n\\]\n","sha256_utf8":"f66adedf0c4f5b123ae5513d37148e32f8907c27d2cf32f915c014f9a44f3619"},{"start":37661,"end":39280,"start_line":576,"end_line":612,"raw_tex":"\t\t\tThe class of functions \\(\\func{h}{\\R^n}{\\R}\\) that are \\(k\\) times continuously differentiable is denoted as \\(\\cont^k(\\R^n)\\).\nWe write \\(h\\in\\cont^{1,1}(\\R^n)\\) to indicate that \\(h\\in\\cont^1(\\R^n)\\) and that \\(\\nabla h\\) is Lipschitz continuous with modulus \\(L_h\\).\nTo simplify the terminology, we will say that such an \\(h\\) is \\DEF{\\(L_h\\)-smooth}.\nIt follows from \\cite[Prop. A.24]{bertsekas2016nonlinear} that if \\(h\\) is \\(L_h\\)-smooth, then\n\\(\n\t|h(y)-h(x)-\\innprod{\\nabla h(x)}{y-x}|\n{}\\leq{}\n\t\\tfrac{L_h}{2}\\|y-x\\|^2\n\\)\nfor all \\(x,y\\in\\R^n\\).\nIn particular, there exists \\(\\sigma_h\\in[-L_h,L_h]\\) such that \\(h\\) is \\DEF{\\(\\sigma_h\\)-hypoconvex}, in the sense that \\(h-\\tfrac{\\sigma_h}{2}\\|{}\\cdot{}\\|^2\\) is a convex function.\nThus, every \\(L_h\\)-smooth and \\(\\sigma_h\\)-hypoconvex function \\(h\\) satisfies\n\\begin{equation}\\label{eq:LipBound}\n\t\\tfrac{\\sigma_h}{2}\n\t\\|y-x\\|^2\n{}\\leq{}\n\th(y)-h(x)-\\innprod{\\nabla h(x)}{y-x}\n{}\\leq{}\n\t\\tfrac{L_h}{2}\n\t\\|y-x\\|^2\n\\quad\n\t\\forall x,y\\in\\R^n.\n\\end{equation}\nBy applying \\cite[Thm. 2.1.5]{nesterov2003introductory} to the (convex) function \\(\\psi=h-\\tfrac\\sigma2\\|{}\\cdot{}\\|^2\\) we obtain that this is equivalent to having\n\\begin{equation}\\label{eq:innprod}\n\t\\sigma_h\n\t\\|y-x\\|^2\n{}\\leq{}\n\t\\innprod{\\nabla h(y)-\\nabla h(x)}{y-x}\n{}\\leq{}\n\tL_h\n\t\\|y-x\\|^2\n\\quad\n\t\\forall x,y\\in\\R^n.\n\\end{equation}\nNote that \\(\\sigma\\)-hypoconvexity generalizes the notion of (strong) convexity by allowing negative strong convexity moduli.\nIn fact, if \\(\\sigma=0\\) then \\(\\sigma\\)-hypoconvexity reduces to convexity, while for \\(\\sigma>0\\) it denotes \\(\\sigma\\)-strong convexity.\n","sha256_utf8":"0b2187e92058ee778e819841cf31d3dccc2ea49daef1c63d691b77957fb1180b"},{"start":42319,"end":44027,"start_line":725,"end_line":764,"raw_tex":"\t\t\tThe \\DEF{proximal mapping} of \\(\\func{h}{\\R^n}{\\Rinf}\\) with parameter \\(\\gamma>0\\) is \\(\\ffunc{\\prox_{\\gamma h}}{\\R^n}{\\dom h}\\) defined as\n\\begin{equation}\n\t\\prox_{\\gamma h}(x)\n{}\\coloneqq{}\n\t\\argmin_{w\\in\\R^n}\\set{\n\t\th(w) + \\tfrac{1}{2\\gamma}\\|w-x\\|^2\n\t}.\n\\end{equation}\nWe say that a function \\(h\\) is \\DEF{prox-bounded} if \\(h+\\tfrac{1}{2\\gamma}\\|{}\\cdot{}\\|^2\\) is lower bounded for some \\(\\gamma>0\\).\nThe supremum of all such \\(\\gamma\\)\n% --- which is possibly infinite, as it is the case when \\(h\\) is lower bounded or convex ---\nis the \\emph{threshold of prox-boundedness of \\(h\\)}, denoted as \\(\\gamma_h\\).\nIf \\(h\\) is lsc, then \\(\\prox_{\\gamma h}\\) is nonempty- and compact-valued over \\(\\R^n\\) for all \\(\\gamma\\in(0,\\gamma_h)\\) \\cite[Thm. 1.25]{rockafellar2011variational}.\nConsequently, the value function of the minimization problem defining the proximal mapping, namely the \\DEF{Moreau envelope} with stepsize \\(\\gamma\\in(0,\\gamma_h)\\), denoted by \\(\\func{h^\\gamma}{\\R^n}{\\R}\\) and defined as\n\\begin{equation}\n\t\\fillwidthof[r]{\n\t\t\\prox_{\\gamma h}(x)\n\t}{\n\t\th^\\gamma(x)\n\t}\n{}\\coloneqq{}\n\t\\fillwidthof[l]{\n\t\t\\argmin_{w\\in\\R^n}\\set{\n\t\t\th(w) + \\tfrac{1}{2\\gamma}\\|w-x\\|^2\n\t\t}.\n\t}{\n\t\t\\inf_{w\\in\\R^n}\\set{\n\t\t\th(w) + \\tfrac{1}{2\\gamma}\\|w-x\\|^2\n\t\t},\n\t}\n\\end{equation}\nis everywhere finite and, in fact, strictly continuous \\cite[Ex. 10.32]{rockafellar2011variational}.\nMoreover, the necessary optimality conditions of the problem defining \\(\\prox_{\\gamma g}\\) together with \\cite[Thm. 10.1 and Ex. 8.8]{rockafellar2011variational} imply that\n\\begin{equation}\\label{eq:prox:subdiff}\n\t\\tfrac1\\gamma(x-\\bar x)\n{}\\in{}\n\t\\hat\\partial h(\\bar x)\n\\quad\n\t\\forall\\bar x\\in\\prox_{\\gamma h}(x).\n\\end{equation}\n","sha256_utf8":"0c87c508d90114d99417b88c65e5bef55e39566223ce33ba66fed73d115e650c"},{"start":46558,"end":47038,"start_line":843,"end_line":853,"raw_tex":"\\begin{ass}[Requirements for the DRS formulation \\eqref{eq:P}]\\label{ass:DRS}%\n\tThe following hold\n\t\\begin{enumerate}\n\t\\item\\label{ass:DRS:f}%\n\t\t\\(\\sDRS\\in\\cont^{1,1}(\\R^n)\\) is \\(L_{\\sDRS}\\)-smooth, hence \\(\\sigma_{\\sDRS}\\)-hypoconvex for some \\(\\sigma_{\\sDRS}\\in[-L_{\\sDRS},L_{\\sDRS}]\\).\n\t\\item\\label{ass:DRS:g}%\n\t\t\\(\\nsDRS\\) is proper and lsc.\n\t\\item\\label{ass:DRS:LB}%\n\t\tProblem \\eqref{eq:P} has a solution, that is, \\(\\argmin\\varphi\\neq\\emptyset\\).\n\t\\end{enumerate}\n\\end{ass}","sha256_utf8":"f589fe62d24de882b197f676d47c3595d0bc2cfa5dd9f80ca0cba83d8837e020"},{"start":47039,"end":47338,"start_line":854,"end_line":855,"raw_tex":"\\begin{rem}[Feasible stepsizes for \\ref{DRS}]\\label{thm:proxbounded}%\n\tUnder \\Cref{ass:DRS}, both \\(\\sDRS\\) and \\(\\nsDRS\\) are prox-bounded with threshold at least \\(\\nicefrac{1}{L_{\\sDRS}}\\), and in particular \\ref{DRS} iterations are well defined for all \\(\\gamma\\in(0,\\nicefrac{1}{L_{\\sDRS}})\\).\n","sha256_utf8":"f0e61e49f780fbe674f443f86f92a60c1ed818a005cd0d0ac23cfa817166aa24"},{"start":48777,"end":50148,"start_line":879,"end_line":918,"raw_tex":"Starting from \\(s\\in\\R^p\\), let us consider variables \\((u,v)\\) generated by a \\ref{DRS} step under \\Cref{ass:DRS}.\nAs first noted in \\cite{patrinos2014douglas}, from the relation \\(s=u+\\gamma\\nabla\\sDRS(u)\\) (see \\cref{thm:proxfEquiv}) it follows that\n\\begin{equation}\\label{eq:vFBS}\n\tv\\in\\FB u\n\\end{equation}\nis the result of a \\DEF{forward-backward step} at \\(u\\), amounting to\n\\begin{align}\\label{eq:DRSv}\n\tv\n{}\\in{}\n\t\\argmin_{w\\in\\R^p}{} &\n\t\\set{\n\t\t\\nsDRS(w)\n\t\t{}+{}\n\t\t\\smashunderbracket{\n\t\t\t\\sDRS(u)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla\\sDRS(u)}{w-u}\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2\\gamma}\\|w-u\\|^2\n\t\t}{}\n\t},\n\\shortintertext{%\n\tsee \\eg \\cite{bolte2014proximal,themelis2018forward} for an extensive discussion on nonconvex forward-backward splitting (FBS).\n\tThis shows that \\(v\\) is the result of the minimization of a \\emph{majorization model} for the original function \\(\\varphi=\\sDRS+\\nsDRS\\), where the smooth function \\(\\sDRS\\) is replaced by the quadratic upper bound emphasized by the under-bracket in \\eqref{eq:DRSv}.\n\tFirst introduced in \\cite{patrinos2014douglas} for convex problems, the \\DEF{Douglas-Rachford envelope} (DRE) is the function \\(\\func{\\DRE}{\\R^n}{\\R}\\) defined as%\n}\n\\label{eq:DRE}\n\t\\DRE(s)\n{}\\coloneqq{}\n\t\\min_{w\\in\\R^p}{} &\n\t\\set{\n\t\t\\nsDRS(w)\n\t\t{}+{}\n\t\t\\sDRS(u)\n\t\t{}+{}\n\t\t\\innprod{\\nabla\\sDRS(u)}{w-u}\n\t\t{}+{}\n\t\t\\tfrac{1}{2\\gamma}\\|w-u\\|^2\n\t}.\n\\end{align}","sha256_utf8":"f2e3eed690dbe8488e91b5b5ea581f7027a2d49df89aea9dd0e97e387f5129ba"},{"start":50149,"end":50626,"start_line":919,"end_line":927,"raw_tex":"Namely, rather than the \\emph{minimizer} \\(v\\), \\(\\DRE(s)\\) is the \\emph{value} of the minimization problem \\eqref{eq:DRSv} defining the \\(v\\)-update in \\eqref{DRS}.\nThe expression \\eqref{eq:DRE} emphasizes the close connection that the DRE has with the \\DEF{forward-backward envelope} (FBE) as in \\cite{themelis2018forward}, here denoted \\(\\FBE\\), namely\n\\begin{equation}\\label{eq:FBEu}\n\t\\DRE(s)\n{}={}\n\t\\FBE(u),\n\\quad\n\t\\text{where \\(u=\\prox_{\\gamma\\sDRS}(s)\\).}\n\\end{equation}","sha256_utf8":"c73e46b1972aef34e074855f516e4e20888ffe5698c5b6673151972048d8f07a"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{thm}[Subsequential convergence]\\label{thm:DRS}%\n\tSuppose that \\Cref{ass:DRS} is satisfied, and consider a sequence \\(\\seq{s^k,u^k,v^k}\\) generated by \\ref{DRS} with stepsize \\(\\gamma\\) and relaxation \\(\\lambda\\) as in \\Cref{thm:DRE:SD}, starting from \\(s^0\\in\\R^p\\).\n\tThe following hold:%\n\t\\begin{enumerate}\n\t\\item\\label{thm:DRS:r}\n\t\tThe residual \\(\\seq{u^k-v^k}\\) vanishes with rate\n\t\t\\(\n\t\t\t\\min_{i\\leq k}\\|u^i-v^i\\|\n\t\t{}={}\n\t\t\to(\\nicefrac{1}{\\sqrt k})\n\t\t\\).\n\t\\item\\label{thm:DRS:uv}\n\t\t\\(\\seq{u^k}\\) and \\(\\seq{v^k}\\) have same cluster points, all of which are stationary for \\(\\varphi\\) and on which \\(\\varphi\\) has same value, this being the limit of \\(\\seq{\\DRE(s^k)}\\).\n\t\tIn fact, for each \\(k\\) one has\n\t\t\\(\n\t\t\t\\dist(0,\\hat\\partial\\varphi(v^k))\n\t\t{}\\leq{}\n\t\t\t\\frac{1-\\gamma\\sigma_{\\sDRS}}{2\\gamma}\n\t\t\t\\|u^k-v^k\\|\n\t\t\\).\n\t\\item\\label{thm:DRS:bounded}\n\t\tIf \\(\\varphi\\) has bounded level sets, then the sequence \\(\\seq{s^k,u^k,v^k}\\) is bounded.\n\t\\end{enumerate}\n\t\n\n\n\\end{thm}","units":[{"start":93135,"end":94108,"start_line":2362,"end_line":2385,"raw_tex":"\\begin{thm}[Subsequential convergence]\\label{thm:DRS}%\n\tSuppose that \\Cref{ass:DRS} is satisfied, and consider a sequence \\(\\seq{s^k,u^k,v^k}\\) generated by \\ref{DRS} with stepsize \\(\\gamma\\) and relaxation \\(\\lambda\\) as in \\Cref{thm:DRE:SD}, starting from \\(s^0\\in\\R^p\\).\n\tThe following hold:%\n\t\\begin{enumerate}\n\t\\item\\label{thm:DRS:r}\n\t\tThe residual \\(\\seq{u^k-v^k}\\) vanishes with rate\n\t\t\\(\n\t\t\t\\min_{i\\leq k}\\|u^i-v^i\\|\n\t\t{}={}\n\t\t\to(\\nicefrac{1}{\\sqrt k})\n\t\t\\).\n\t\\item\\label{thm:DRS:uv}\n\t\t\\(\\seq{u^k}\\) and \\(\\seq{v^k}\\) have same cluster points, all of which are stationary for \\(\\varphi\\) and on which \\(\\varphi\\) has same value, this being the limit of \\(\\seq{\\DRE(s^k)}\\).\n\t\tIn fact, for each \\(k\\) one has\n\t\t\\(\n\t\t\t\\dist(0,\\hat\\partial\\varphi(v^k))\n\t\t{}\\leq{}\n\t\t\t\\frac{1-\\gamma\\sigma_{\\sDRS}}{2\\gamma}\n\t\t\t\\|u^k-v^k\\|\n\t\t\\).\n\t\\item\\label{thm:DRS:bounded}\n\t\tIf \\(\\varphi\\) has bounded level sets, then the sequence \\(\\seq{s^k,u^k,v^k}\\) is bounded.\n\t\\end{enumerate}\n\t","sha256_utf8":"1db1e47d60d7f2f2e9beef27ac69ef8efa4931fe8461c081777f8fd7b5fc6dae"},{"start":97751,"end":97761,"start_line":2490,"end_line":2491,"raw_tex":"\n\\end{thm}","sha256_utf8":"27661e277ceb78e4dd676b4ea4b5c8d9a4d3c607a1a758eac095282a4f311f77"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{thm}[Lower bounds for smooth functions]\\label{thm:rho}%\n\tLet \\(h\\in C^{1,1}(\\R^n)\\) be \\(L_h\\)-smooth and \\(\\sigma_h\\)-hypoconvex.\n\tThen, for all \\(x,y\\in\\R^n\\) it holds that\n\t\\[\n\t\th(y)\n\t{}\\geq{}\n\t\th(x)\n\t\t{}+{}\n\t\t\\innprod{\\nabla h(x)}{y-x}\n\t\t{}+{}\n\t\t\\rho(y,x),\n\t\\]\n\twhere\n\t\\begin{enumerate}\n\t\\item\\label{thm:hypo:rho0}%\n\t\teither~\n\t\t\\(\n\t\t\t\\rho(y,x)\n\t\t{}={}\n\t\t\t\\tfrac{\\sigma_h}{2}\n\t\t\t\\|y-x\\|^2\n\t\t\\),\n\t\\item\\label{thm:hypo:rho1}%\n\t\tor~\n\t\t\\(\n\t\t\t\\rho(y,x)\n\t\t{}={}\n\t\t\t\\tfrac{\\sigma_hL_h}{2(L_h+\\sigma_h)}\n\t\t\t\\|y-x\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2(L_h+\\sigma_h)}\n\t\t\t\\|\\nabla h(y)-\\nabla h(x)\\|^2\n\t\t\\),~\n\t\tprovided that~\n\t\t\\(\n\t\t\t-L_h<\\sigma_h\\leq 0\n\t\t\\).\n\t\\end{enumerate}\n\tClearly, all inequalities remain valid if one replaces \\(L_h\\) with any \\(L\\geq L_h\\) and \\(\\sigma_h\\) with any \\(\\sigma\\in[-L,\\sigma_h]\\).\n\t\\begin{proof}\n\t\tSee \\Cref{proof:thm:rho}.\n\t\\end{proof}\n\\end{thm}\n\n\\begin{prop}[Proximal properties of smooth functions]\\label{thm:proxf}%\n\tLet \\(h\\in \\cont^{1,1}(\\dom h)\\) be \\(L_h\\)-smooth, hence \\(\\sigma_h\\)-hypoconvex for some \\(\\sigma_h\\in[-L_h,L_h]\\).\n\tThen, \\(h\\) is prox-bounded with \\(\\gamma_{h}\\geq\\nicefrac{1}{[\\sigma_{h}]_-}\\) and for all \\(\\gamma<\\nicefrac{1}{[\\sigma_{h}]_-}\\) the following hold:\n\t\\begin{enumerate}\n\t\\item\\label{thm:proxfEquiv}\n\t\t\\(\\prox_{\\gamma h}\\) is single valued, and for all \\(s\\in\\R^n\\) it holds that \\(u=\\prox_{\\gamma h}(s)\\) iff\n\t\t\\(\n\t\t\ts=u+\\gamma\\nabla h(u)\n\t\t\\).\n\t\\item\\label{thm:proxfLip}\n\t\t\\(\\prox_{\\gamma h}\\) is \\((\\tfrac{1}{1+\\gamma L_{h}})\\)-strongly monotone and \\((1+\\gamma\\sigma_h)\\)-cocoercive, in the sense that\n\t\t\\[\n\t\t\t\\innprod{u-u'}{s-s'}\n\t\t{}\\geq{}\n\t\t\t\\tfrac{1}{1+\\gamma L_h}\n\t\t\t\\|s-s'\\|^2\n\t\t\\quad\\text{and}\\quad\n\t\t\t\\innprod{u-u'}{s-s'}\n\t\t{}\\geq{}\n\t\t\t(1+\\gamma\\sigma_h)\n\t\t\t\\|u-u'\\|^2\n\t\t\\]\n\t\tfor all \\(s,s'\\in\\R^n\\), where \\(u=\\prox_{\\gamma h}(s)\\) and \\(u'=\\prox_{\\gamma h}(s')\\).\n\t\tIn particular,\n\t\t\\begin{equation}\\label{eq:biLip}\n\t\t\t\\tfrac{1}{1+\\gamma L_h}\n\t\t\t\\|s-s'\\|\n\t\t{}\\leq{}\n\t\t\t\\|u-u'\\|\n\t\t{}\\leq{}\n\t\t\t\\tfrac{1}{1+\\gamma\\sigma_h}\n\t\t\t\\|s-s'\\|.\n\t\t\\end{equation}\n\t\tThus, \\(\\prox_{\\gamma h}\\) is a \\(\\tfrac{1}{1+\\gamma\\sigma_h}\\)-Lipschitz and invertible mapping, and its inverse \\(\\id+\\gamma\\nabla h\\) is \\((1+\\gamma L_h)\\)-Lipschitz continuous.\n\t\\item\\label{thm:MoreaufC1}\n\t\t\\(h^\\gamma\\in \\cont^{1,1}(\\R^n)\\) is \\(L_{h^\\gamma}\\)-smooth and \\(\\sigma_{h^\\gamma}\\)-hypoconvex, with\n\t\t\\(\n\t\t\tL_{h^\\gamma}\n\t\t{}={}\n\t\t\t\\max\\set{\n\t\t\t\t\\tfrac{L_{h}}{1+\\gamma L_{h}},~\n\t\t\t\t\\tfrac{[\\sigma_{h}]_-}{1+\\gamma\\sigma_{h}}\n\t\t\t}\n\t\t\\)\n\t\tand\n\t\t\\(\n\t\t\t\\sigma_{h^\\gamma}\n\t\t{}={}\n\t\t\t\\tfrac{\\sigma_{h}}{1+\\gamma\\sigma_{h}}\n\t\t\\).\n\t\tMoreover,\n\t\t\\(\n\t\t\t\\nabla h^\\gamma(s)\n\t\t{}={}\n\t\t\t\\tfrac{1}{\\gamma}(s-\\prox_{\\gamma h}(s))\n\t\t\\)\n\t\tand\n\t\t\\(\n\t\t\t\\nabla h(\\prox_{\\gamma h}(s))\n\t\t{}={}\n\t\t\t\\tfrac{1}{\\gamma}\\bigl(s-\\prox_{\\gamma h}(s)\\bigr)\n\t\t\\).\n\t\\end{enumerate}\n\t\\begin{proof}\n\t\tSee \\Cref{proof:thm:proxf}.\n\t\\end{proof}\n\\end{prop}\n\n\tThat \\(\\gamma_{\\sDRS}\\geq\\nicefrac{1}{L{\\sDRS}}\\) follows from \\Cref{thm:proxf}, having \\(\\nicefrac{1}{[\\sigma_{\\sDRS}]_-}\\geq\\nicefrac{1}{L_{\\sDRS}}\\).\n\tAs for \\(\\nsDRS\\), for all \\(s\\in\\R^p\\) it holds that\n\t\\[\n\t\t\\inf\\varphi\n\t{}\\leq{}\n\t\t\\sDRS(s)+\\nsDRS(s)\n\t{}\\overrel*[\\leq]{\\eqref{eq:LipBound}}{}\n\t\t\\sDRS(0)+\\innprod{\\nabla\\sDRS(0)}{s}+\\tfrac{L_{\\sDRS}}{2}\\|s\\|^2\n\t\t{}+{}\n\t\t\\nsDRS(s),\n\t\\]\n\thence, for all \\(\\gamma<\\nicefrac{1}{L_{\\sDRS}}\\) the function \\(s\\mapsto\\nsDRS(s)+\\tfrac{1}{2\\gamma}\\|s\\|^2\\) is lower bounded.\n\n\nWe start by observing that \\(\\DRE\\) is a strictly continuous function for \\(\\gamma<\\nicefrac{1}{L_{\\sDRS}}\\), owing to the fact that so is the FBE \\cite[Prop. 4.2]{themelis2018forward}, and that \\(\\prox_{\\gamma\\sDRS}\\) is Lipschitz continuous as shown in \\cref{thm:proxfLip}.\n\\begin{prop}[Strict continuity]\\label{thm:C0}%\n\tSuppose that \\Cref{ass:DRS} is satisfied.\n\tFor all \\(\\gamma<\\nicefrac{1}{L_{\\sDRS}}\\) the DRE \\(\\DRE\\) is a real-valued and strictly continuous function.\n\\end{prop}\n\n\\begin{prop}[Sandwiching property]\\label{thm:sandwich}%\n\tSuppose that \\Cref{ass:DRS} is satisfied.\n\tLet \\(\\gamma<\\nicefrac{1}{L_{\\sDRS}}\\) be fixed, and consider \\(u,v\\) generated by one \\ref{DRS} iteration starting from \\(s\\in\\R^p\\).\n\tThen,\n\t\\begin{enumerate}\n\t\\item\\label{thm:DREleq}\n\t\t\\(\\DRE(s)\\leq\\varphi(u)\\).\n\t\\item\\label{thm:DREgeq}\n\t\t\\(\n\t\t\t\\varphi(v)\n\t\t{}\\leq{}\n\t\t\t\\DRE(s)\n\t\t\t{}-{}\n\t\t\t\\tfrac{1-\\gamma L_{\\sDRS}}{2\\gamma}\\|u-v\\|^2\n\t\t\\).\n% \t\t\\(\n% \t\t\t\\varphi(v)+\\tfrac{1-\\gamma L_{\\sDRS}}{2\\gamma}\\|u-v\\|^2\n% \t\t{}\\leq{}\n% \t\t\t\\DRE(s)\n% \t\t{}\\leq{}\n% \t\t\t\\varphi(v)+\\tfrac{1-\\gamma\\sigma_{\\sDRS}}{2\\gamma}\\|u-v\\|^2\n% \t\t\\).\n\t\\end{enumerate}\n\t\\begin{proof}\n\t\t\\ref{thm:DREleq} is easily inferred from definition \\eqref{eq:DRE} by considering \\(w=u\\).\n\t\tMoreover, it follows from \\cite[Prop. 4.3]{themelis2018forward} and the fact that \\(v\\in\\FB u\\), cf. \\eqref{eq:vFBS}, that\n\t\t\\(\n\t\t\t\\varphi(v)\n\t\t{}\\leq{}\n\t\t\t\\FBE(u)\n\t\t\t{}-{}\n\t\t\t\\tfrac{1-\\gamma L_{\\sDRS}}{2\\gamma}\\|u-v\\|^2\n\t\t\\).\n\t\t\\ref{thm:DREgeq} then follows from \\eqref{eq:FBEu}.\n\t\\end{proof}\n\\end{prop}\n\n\\begin{thm}[Minimization and level-boundedness equivalence]\\label{thm:DREequiv}%\n\tSuppose that \\Cref{ass:DRS} is satisfied.\n\tFor any \\(\\gamma<\\nicefrac{1}{L_{\\sDRS}}\\) the following hold:\n\t\\begin{enumerate}\n\t\\item\\label{thm:inf}%\n\t\t\\(\\inf\\varphi=\\inf\\DRE\\).\n\t\\item\\label{thm:argmin}%\n\t\t\\(\\argmin\\varphi=\\prox_{\\gamma\\sDRS}\\bigl(\\argmin\\DRE\\bigr)\\).\n\t\\item\\label{thm:boundedlev}%\n\t\t\\(\\varphi\\) is level bounded iff so is \\(\\DRE\\).\n\t\\end{enumerate}\n\t\\begin{proof}\n\t\tIt follows from \\cite[Thm. 4.4]{themelis2018forward} that the FBE satisfies \\(\\inf\\varphi=\\inf\\FBE\\) and \\(\\argmin\\varphi=\\argmin\\FBE\\).\n\t\tThe similar properties \\ref{thm:inf} and \\ref{thm:argmin} of the DRE then follow from the identity \\(\\DRE=\\FBE\\circ\\prox_{\\gamma\\sDRS}\\), cf. \\eqref{eq:FBEu}, and the fact that \\(\\prox_{\\gamma\\sDRS}\\) is invertible, as shown in \\cref{thm:proxf}.\n\t\t\n\t\tWe now show \\ref{thm:boundedlev}.\n\t\tDenote\n\t\t\\(\n\t\t\t\\varphi_\\star\n\t\t{}\\coloneqq{}\n\t\t\t\\inf\\varphi\n\t\t{}={}\n\t\t\t\\inf\\DRE\n\t\t\\),\n\t\twhich is finite by assumption.\n\t\t\\begin{proofitemize}\n\t\t\\item\n\t\t\tSuppose that \\(\\DRE\\) is level bounded, and let \\(u\\in\\lev_{\\leq\\alpha}\\varphi\\) for some \\(\\alpha>\\varphi_\\star\\).\n\t\t\tThen, \\(s\\coloneqq u+\\gamma\\nabla\\sDRS(u)\\) is such that \\(\\prox_{\\gamma\\sDRS}(s)=u\\), as shown in \\cref{thm:proxfEquiv}.\n\t\t\tThus, from \\cref{thm:sandwich} it follows that \\(s\\in\\lev_{\\leq\\alpha}\\DRE\\).\n\t\t\tIn particular,\n\t\t\t\\(\n\t\t\t\t\\lev_{\\leq\\alpha}\\varphi\n\t\t\t{}\\subseteq{}\n\t\t\t\t[I+\\gamma\\nabla\\sDRS]\n\t\t\t\t(\\lev_{\\leq\\alpha}\\DRE),\n\t\t\t\\)\n\t\t\tand since \\(\\I+\\gamma\\nabla\\sDRS\\) is Lipschitz continuous and \\(\\lev_{\\leq\\alpha}\\DRE\\) is bounded by assumption, it follows that \\(\\lev_{\\leq\\alpha}\\varphi\\) is also bounded.\n\t\t\\item\n\t\t\tSuppose now that \\(\\DRE\\) is not level bounded.\n\t\t\tThen, there exists \\(\\alpha>\\varphi_\\star\\) together with a sequence \\(\\seq{s_k}\\) satisfying\n\t\t\t\\(\n\t\t\t\ts_k\n\t\t\t{}\\in{}\n\t\t\t\t\\lev_{\\leq\\alpha}\\DRE\n\t\t\t\t{}\\setminus{}\n\t\t\t\t\\ball 0k\n\t\t\t\\)\n\t\t\tfor all \\(k\\in\\N\\).\n\t\t\tLet \\(u_k\\coloneqq\\prox_{\\gamma\\sDRS}(s_k)\\), so that \\(s_k=u_k+\\gamma\\nabla\\sDRS(u_k)\\) (cf. \\cref{thm:proxfEquiv}), and let \\(v_k\\in\\FB{u_k}\\).\n\t\t\tFrom \\cref{thm:DREgeq} it then follows that \\(v_k\\in\\lev_{\\leq\\alpha}\\varphi\\), and that\n\t\t\t\\[\n\t\t\t\t\\alpha\n\t\t\t\t{}-{}\n\t\t\t\t\\varphi_\\star\n\t\t\t{}\\geq{}\n\t\t\t\t\\DRE(s_k)\n\t\t\t\t{}-{}\n\t\t\t\t\\varphi_\\star\n\t\t\t{}\\geq{}\n\t\t\t\t\\DRE(s_k)\n\t\t\t\t{}-{}\n\t\t\t\t\\varphi(v_k)\n\t\t\t{}\\geq{}\n\t\t\t\t\\tfrac{1-\\gamma L_{\\sDRS}}{2\\gamma}\n\t\t\t\t\\|u_k-v_k\\|^2.\n\t\t\t\\]\n\t\t\tTherefore,\n\t\t\t\\(\n\t\t\t\t\\|u_k-v_k\\|^2\n\t\t\t{}\\leq{}\n\t\t\t\t\\tfrac{\n\t\t\t\t\t2\\gamma(\\alpha-\\varphi_\\star)\n\t\t\t\t}{\n\t\t\t\t\t1-\\gamma L_{\\sDRS}\n\t\t\t\t}\n\t\t\t\\)\n\t\t\tand\n\t\t\t\\begin{align*}\n\t\t\t\t\\|v_k\\|\n\t\t\t{}\\geq{} &\n\t\t\t\t\\|u_k-u_0\\|\n\t\t\t\t{}-{}\n\t\t\t\t\\|u_0\\|\n\t\t\t\t{}-{}\n\t\t\t\t\\|u_k-v_k\\|\n\t\t\t\\smash{{}\\overrel[\\geq]{\\ref{thm:proxfLip}}{}}\n\t\t\t\t\\tfrac{1}{1+\\gamma L_{\\sDRS}}\n\t\t\t\t\\|s_k-s_0\\|\n\t\t\t\t{}-{}\n\t\t\t\t\\|u_0\\|\n\t\t\t\t{}-{}\n\t\t\t\t\\|u_k-v_k\\|\n\t\t\t\\\\\n\t\t\t{}\\geq{} &\n\t\t\t\t\\tfrac{\n\t\t\t\t\tk-\\|s_0\\|\n\t\t\t\t}{\n\t\t\t\t\t1+\\gamma L_{\\sDRS}\n\t\t\t\t}\n\t\t\t\t{}-{}\n\t\t\t\t\\|u_0\\|\n\t\t\t\t{}-{}\n\t\t\t\t\\sqrt{\n\t\t\t\t\t\\tfrac{\n\t\t\t\t\t\t2\\gamma(\\alpha-\\varphi_\\star)\n\t\t\t\t\t}{\n\t\t\t\t\t\t1-\\gamma L_{\\sDRS}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t{}\\to{}\n\t\t\t\t+\\infty\n\t\t\t\\quad\\text{as \\(k\\to\\infty\\).}\n\t\t\t\\end{align*}\n\t\t\tThis shows that \\(\\lev_{\\leq\\alpha}\\varphi\\) is also unbounded.\n\t\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}\n\\end{thm}\n\n\\begin{thm}[Sufficient decrease on the DRE]\\label{thm:DRE:SD}%\n\tSuppose that \\Cref{ass:DRS} is satisfied, and consider one \\ref{DRS} update\n\t\\(\n\t\ts\n\t{}\\mapsto{}\n\t\t(u,v,s^+)\n\t\\)\n\tfor some stepsize\n\t\\(\n\t\t\\gamma\n\t{}<{}\n\t\t\\min\\set{\n\t\t\t\\tfrac{2-\\lambda}{2[\\sigma_{\\sDRS}]_-},\\,\n\t\t\t\\tfrac{1}{L_{\\sDRS}}\n\t\t}\n\t\\)\n\tand relaxation \\(\\lambda\\in(0,2)\\).\n\tThen,\n\t\\begin{equation}\\label{eq:DRE:SD}\n\t\t\\DRE(s)-\\DRE(s^+)\n\t{}\\geq{}\n\t\t\\tfrac{c}{(1+\\gamma L_{\\sDRS})^2}\\,\n\t\t\\|s-s^+\\|^2,\n\t\\end{equation}\n\twhere, denoting \\(p_{\\sDRS}\\coloneqq\\nicefrac{\\sigma_{\\sDRS}}{L_{\\sDRS}}\\in[-1,1]\\), \\(c\\) is a strictly positive constant defined as%\n\t\\footnote{%\n\t\tA one-line expression for the constant is\n\t\t\\(\n\t\t\tc\n\t\t{}={}\n\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n\t\t\t{}-{}\n\t\t\t\\min\\set{\n\t\t\t\t\\tfrac{[p_{\\sDRS}]_-}{\\lambda},~\n\t\t\t\tL_{\\sDRS}\\max\\set{\n\t\t\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{2(1-[p_{\\sDRS}]_-)},\\,\n\t\t\t\t\t\\tfrac12-\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\n\t\t\t\t}\n\t\t\t}\n\t\t\\).\n\t}\n\t\\begin{equation}\\label{eq:DRE:c0}\n\t\tc\n\t{}={}\n\t\t\\frac{2-\\lambda}{2\\lambda\\gamma}\n\t\t{}-{}\n\t\t\\begin{ifcases}\n\t\t\tL_{\\sDRS}\\max\\set{\n\t\t\t\t\\tfrac{[p_{\\sDRS}]_-}{2(1-[p_{\\sDRS}]_-)},\\,\n\t\t\t\t\\tfrac12-\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\n\t\t\t}\n\t\t&\n\t\t\t\tp_{\\sDRS}\n\t\t\t{}\\geq{}\n\t\t\t\t\\tfrac\\lambda2-1\n% \t\t\t0<\\lambda<2(1-[p_{\\sDRS}]_-),\n\t\t\\\\\n\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{\\lambda}\n\t\t\\otherwise[otherwise.]\n\t\t\\end{ifcases}\n\t\\end{equation}\n\tIf \\(\\sDRS\\) is strongly convex, then \\eqref{eq:DRE:SD} also holds for\n\t\\begin{equation}\\label{eq:lambda>=2}\n\t\t2\n\t{}\\leq{}\n\t\t\\lambda\n\t{}<{}\n\t\t\\tfrac{4}{1+\\sqrt{1-p_{\\sDRS}}}\n\t\\quad\\text{and}\\quad\n\t\t\\tfrac{\n\t\t\tp_{\\sDRS}\\lambda\n\t\t\t{}-{}\n\t\t\t\\delta\n\t\t}{\n\t\t\t4\\sigma_{\\sDRS}\n\t\t}\n\t{}<{}\n\t\t\\gamma\n\t{}<{}\n\t\t\\tfrac{\n\t\t\tp_{\\sDRS}\\lambda\n\t\t\t{}+{}\n\t\t\t\\delta\n\t\t}{\n\t\t\t4\\sigma_{\\sDRS}\n\t\t},\n\t\\end{equation}\n\twhere\n\t\\(\n\t\t\\delta\n\t{}\\coloneqq{}\n\t\t\\sqrt{(p_{\\sDRS}\\lambda)^2-8p_{\\sDRS}(\\lambda-2)}\n\t\\),\n\tin which case\n\t\\begin{equation}\\label{eq:DRE:c1}\n\t\tc\n\t{}={}\n\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n\t\t{}+{}\n\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n\t\t(\\tfrac12-\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}).\n\t\\end{equation}\n\t\\begin{proof}\n\t\t\\newcommand\\x\\xi%\n\t\tLet \\((u^+,v^+)\\) be generated by one \\ref{DRS} iteration starting at \\(s^+\\).\n\t\tThen,\n\t\t\\begin{align*}\n\t\t\t\\DRE(s^+)\n\t\t{}={} &\n\t\t\t\\min_{w\\in\\R^n}\\set{\n\t\t\t\t\\sDRS(u^+)\n\t\t\t\t{}+{}\n\t\t\t\t\\nsDRS(w)\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)}{w-u^+}\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|w-u^+\\|^2\n\t\t\t}\n\t\t\\end{align*}\n\t\tand the minimum is attained at \\(w=v^+\\).\n\t\tTherefore, letting \\(\\rho\\) be as in \\cref{thm:rho},\n\t\t{\\mathtight[0.9]%\n\t\t\t\\begin{align*}\n\t\t\t\t\\DRE(s^+)\n\t\t\t{}\\leq{} &\n\t\t\t\t\\sDRS(u^+)\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)}{v-u^+}\n\t\t\t\t{}+{}\n\t\t\t\t\\nsDRS(v)\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|u^+-v\\|^2\n\t\t\t\\\\\n\t\t\t{}={} &\n\t\t\t\t\\smashunderbracket[0.5pt]{\n\t\t\t\t\t\\sDRS(u^+)\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)}{u-u^+}\n\t\t\t\t}{}\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)}{v-u}\n\t\t\t\t{}+{}\n\t\t\t\t\\nsDRS(v)\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|u^+-v\\|^2\n\t\t\t\\\\\n\t\t\t% \t\t\t\\dueto{\\cref{thm:rho}}\n\t\t\t{}\\overrel[\\leq]{\\llap{\\cref{thm:rho}}}{} &\n\t\t\t\t\\smashoverbracket[0.5pt]{\n\t\t\t\t\t\\sDRS(u)\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\rho(u,u^+)\n\t\t\t\t}{}\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)}{v-u}\n\t\t\t\t{}+{}\n\t\t\t\t\\nsDRS(v)\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|u^+-v\\|^2\n\t\t\t\\\\\n\t\t\t{}={} &\n\t\t\t\t\\sDRS(u)\n\t\t\t\t{}-{}\n\t\t\t\t\\rho(u,u^+)\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u)}{v-u}\n% \t\t\t\t\\innprod{\\nabla\\sDRS(\\fillwidthof[c]{u^+}{u})}{v-u}\n\t\t\t\t{}+{}\n\t\t\t\t\\nsDRS(v)\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|u^+-v\\|^2\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t{+{}}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)-\\nabla\\sDRS(u)}{v-u}\n\t\t\t\\\\\n\t\t\t{}={} &\n\t\t\t\t\\DRE(s)\n\t\t\t\t{}-{}\n\t\t\t\t\\rho(u,u^+)\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)-\\nabla\\sDRS(u)}{v-u}\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|u-u^+\\|^2\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac1\\gamma\n\t\t\t\t\\innprod{u^+-u}{u-v}.\n\t\t\t\\end{align*}%\n\t\t}%\n\t\tSince\n\t\t\\(\n\t\t\tu-v\n\t\t{}={}\n\t\t\t\\tfrac1\\lambda(s-s^+)\n\t\t{}={}\n\t\t\t\\tfrac1\\lambda(u-u^+)\n\t\t\t{}+{}\n\t\t\t\\tfrac\\gamma\\lambda(\\nabla\\sDRS(u)-\\nabla\\sDRS(u^+))\n\t\t\\),\n\t\tas it follows from \\cref{thm:proxfEquiv}, it all simplifies to\n\t\t\\begin{equation}\\label{eq:DRE:SD0}\n\t\t\t\\DRE(s)\n\t\t\t{}-{}\n\t\t\t\\DRE(s^+)\n\t\t{}\\geq{}\n\t\t\t\\tfrac{2-\\lambda}{2\\gamma\\lambda}\n\t\t\t\\|u-u^+\\|^2\n\t\t\t{}-{}\n\t\t\t\\tfrac\\gamma\\lambda\n\t\t\t\\|\\nabla\\sDRS(u^+)-\\nabla\\sDRS(u)\\|^2\n\t\t\t{}+{}\n\t\t\t\\rho(u,u^+).\n\t\t\\end{equation}\n\t\tIt will suffice to show that\n\t\t\\[\n\t\t\t\\DRE(s)-\\DRE(s^+)\n\t\t{}\\geq{}\n\t\t\tc\n\t\t\t\\|u-u^+\\|^2;\n\t\t\\]\n\t\tinequality \\eqref{eq:DRE:SD} will then follow from the \\(\\tfrac{1}{1+\\gamma L_{\\sDRS}}\\)-strong monotonicity of \\(\\prox_{\\gamma\\sDRS}\\), see \\cref{thm:proxfLip}.\n% \t\t\\[\n% \t\t\t\\|u-u^+\\|^2\n% \t\t{}\\geq{}\n% \t\t\t\\tfrac{1}{(1+\\gamma L_{\\sDRS})^2}\n% \t\t\t\\|s-s^+\\|^2\n% \t\t\\]\n\t\tWe now proceed by cases.\n\t\t\\begin{proofitemize}\n\t\t\\item\\textbf{Case 1: \\(\\lambda\\in(0,2)\\).}\n\t\t\t\n\t\t\t\\noindent\n\t\t\tLet \\(\\sigma\\coloneqq-[\\sigma_{\\sDRS}]_-=\\min\\set{\\sigma_{\\sDRS},0}\\) and \\(L\\geq L_{\\sDRS}\\) be such that \\(L+\\sigma>0\\); the value of such an \\(L\\) will be fixed later.\n\t\t\tThen, \\(\\sigma\\leq0\\) and \\(\\sDRS\\) is \\(L\\)-smooth and \\(\\sigma\\)-hypoconvex.\n\t\t\tWe may thus choose \\(\\rho(u,u^+)\\) as in \\cref{thm:hypo:rho1} with these values of \\(L\\) and \\(\\sigma\\).\n\t\t\tInequality \\eqref{eq:DRE:SD0} then becomes\n\t\t\t\\[\n\t\t\t\t\\tfrac{\\DRE(s)-\\DRE(s^+)}{L}\n\t\t\t{}\\geq{}\n\t\t\t\t\\left(\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\x}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{p}{2(1+p)}\n\t\t\t\t\\right)\n\t\t\t\t\\|u^+-u\\|^2\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{L^2}\n\t\t\t\t\\left(\n\t\t\t\t\t\\tfrac{1}{2(1+p)}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{\\x}{\\lambda}\n\t\t\t\t\\right)\n\t\t\t\t\\|\\nabla\\sDRS(u^+)-\\nabla\\sDRS(u)\\|^2,\n\t\t\t\\]\n\t\t\twhere \\(\\x\\coloneqq\\gamma L\\) and \\(p\\coloneqq\\nicefrac\\sigma L\\in(-1,0]\\).\n\t\t\tSince \\(\\nabla\\sDRS\\) is \\(L_{\\sDRS}\\)-Lipschitz continuous, the claim holds provided that the constant\n\t\t\t\\begin{equation}\\label{eq:c}\n\t\t\t\t\\frac cL\n\t\t\t{}={}\n\t\t\t\t\\begin{ifcases}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\x}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{p}{2(1+p)}\n\t\t\t\t&\n\t\t\t\t\t0\n\t\t\t\t\t{}<{}\n\t\t\t\t\t\\tfrac{1}{2(1+p)}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{\\x}{\\lambda}\n\t\t\t\t\\\\[3pt]\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\x}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{p}{2(1+p)}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{L_{\\sDRS}^2}{L^2}\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t\\tfrac{1}{2(1+p)}\n\t\t\t\t\t\t{}-{}\n\t\t\t\t\t\t\\tfrac{\\x}{\\lambda}\n\t\t\t\t\t\\right)\n\t\t\t\t\\otherwise[otherwise,]\n\t\t\t\t\\end{ifcases}\n\t\t\t\\end{equation}\n\t\t\tis strictly positive.\n% \t\t\t\n\t\t\tNow, let us consider two subcases:\n\t\t\t\\begin{itemize}\n\t\t\t\\item\\textbf{Case 1a:} \\(0<\\lambda\\leq 2(1+\\nicefrac{\\sigma}{L_{\\sDRS}})\\).\n\t\t\t\n\t\t\t\tThen,\n\t\t\t\t\\(\n\t\t\t\t\t\\sigma\\geq -\\frac{2-\\lambda}{2}L_{\\sDRS}>-L_{\\sDRS}\n\t\t\t\t\\)\n\t\t\t\tand we can take \\(L=L_{\\sDRS}\\).\n\t\t\t\tConsequently, \\(p=\\nicefrac{\\sigma}{L_{\\sDRS}}\\), \\(\\x=\\gamma L_{\\sDRS}\\), and \\eqref{eq:c} becomes\n\t\t\t\t\\begin{equation}\\label{eq:cbar}\n\t\t\t\t\t\\frac{c}{L_{\\sDRS}}\n\t\t\t\t{}={}\n\t\t\t\t\t\\frac{2-\\lambda}{2\\lambda\\gamma L_{\\sDRS}}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\begin{ifcases}\n\t\t\t\t\t\t\\tfrac{p}{2(1+p)}\n\t\t\t\t\t&\n\t\t\t\t\t\t\\gamma\n\t\t\t\t\t\t{}<{}\n\t\t\t\t\t\t\\tfrac{\\lambda}{2(1+p)}\n\t\t\t\t\t\\\\[3pt]\n\t\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t\t{}-{}\n\t\t\t\t\t\t\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\n\t\t\t\t\t\\otherwise.\n\t\t\t\t\t\\end{ifcases}\n\t\t\t\t\\end{equation}\n\t\t\t\tLet us verify that in this case any \\(\\gamma\\) such that \\(\\gamma<\\nicefrac{1}{L_{\\sDRS}}\\) yields a strictly positive coefficient \\(c\\).\n\t\t\t\tIf \\(0<\\gamma L_{\\sDRS}<\\frac{\\lambda}{2(1+p)}\\leq 1\\), then\n\t\t\t\t\\[\n\t\t\t\t\t\\tfrac{c}{L_{\\sDRS}}\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma L_{\\sDRS}}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{p}{2(1+p)}\n\t\t\t\t{}>{}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{p}{\\lambda}\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac{1+p}{\\lambda}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t{}\\geq{}\n\t\t\t\t\t0,\n\t\t\t\t\\]\n\t\t\t\twhere in the inequality we used the fact that \\(\\lambda<2\\) and \\(p\\leq0\\).\n\t\t\t\tIf instead \\(\\frac{\\lambda}{2(1+p)}<\\gamma L_{\\sDRS}<1\\), then\n\t\t\t\t\\[\n\t\t\t\t\t\\tfrac{c}{L_{\\sDRS}}\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma L_{\\sDRS}}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\n\t\t\t\t{}>{}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{1}{\\lambda}\n\t\t\t\t{}={}\n\t\t\t\t\t0.\n\t\t\t\t\\]\n\t\t\t\tEither way, the sufficient decrease constant \\(c\\) is strictly positive.\n% \t\t\t\tBy imposing \\(c>0\\) we obtain the following conditions on \\(\\gamma\\) and \\(\\lambda\\):\n% \t\t\t\t\\begin{align*}\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\tfrac{\\sigma}{2(1+p)}\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{\\lambda}{2(L_{\\sDRS}+\\sigma)}\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\quad\\cup\\quad &\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\tfrac{L_{\\sDRS}}{2}\n% \t\t\t\t\t\t{}-{}\n% \t\t\t\t\t\t\\tfrac{\\gamma L_{\\sDRS}^2}{\\lambda}\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}\\geq{} &\n% \t\t\t\t\t\t\\tfrac{\\lambda}{2(L_{\\sDRS}+\\sigma)}.\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\shortintertext{%\n% \t\t\t\t\tEquivalently,\n% \t\t\t\t}\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t(1+p)(2-\\lambda)\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\lambda p\\xi\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\xi\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{\\lambda}{2(1+p)}\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\quad\\cup\\quad &\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}>{} &\n% \t\t\t\t\t\t2\\xi^2\n% \t\t\t\t\t\t{}-{}\n% \t\t\t\t\t\t\\lambda\\xi\n% \t\t\t\t\t\t{}-{}\n% \t\t\t\t\t\t(2-\\lambda)\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\xi\n% \t\t\t\t\t{}\\geq{} &\n% \t\t\t\t\t\t\\tfrac{\\lambda}{2(1+p)}.\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\shortintertext{%\n% \t\t\t\t\tSince \\(\\xi<1\\), notice that\n% \t\t\t\t\t\\(\n% \t\t\t\t\t\t(1+p)(2-\\lambda)\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\lambda p\\xi\n% \t\t\t\t\t{}>{}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t\\)\n% \t\t\t\t\tis always satisfied when \\(2-\\lambda\\geq-2p\\), as in the case being considered.\n% \t\t\t\t\tThis is evident if \\(p=0\\) (since \\(\\lambda<2\\)); if instead \\(p<0\\), one has\n% \t\t\t\t\t\\(\n% \t\t\t\t\t\t(1+p)(2-\\lambda)\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\lambda p\\xi\n% \t\t\t\t\t{}\\geq{}\n% \t\t\t\t\t\t-2p(1+p)+p\\lambda\\xi\n% \t\t\t\t\t{}>{}\n% \t\t\t\t\t\t-p(2(1+p)-\\lambda)\n% \t\t\t\t\t{}\\geq{}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t\\).\n% \t\t\t\t\tAll conditions then reduce to\n% \t\t\t\t}\n% \t\t\t\t\t\\xi\n% \t\t\t\t{}<{}\n% \t\t\t\t\t\\tfrac{\\lambda}{2(1+p)}\n% \t\t\t\t\\quad\\cup\\quad &\n% \t\t\t\t\t\\tfrac{\\lambda}{2(1+p)}\n% \t\t\t\t{}\\leq{}\n% \t\t\t\t\t\\xi\n% \t\t\t\t{}<{}\n% \t\t\t\t\t1,\n% \t\t\t\t\\end{align*}\n% \t\t\t\tthat is, \\(\\gamma L_{\\sDRS}=\\xi<1\\).\n\t\t\t\tSince \\(\\sigma=-[\\sigma_{\\sDRS}]_-\\) and\n\t\t\t\t\\[\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{\\sigma}{2(1+p)}\n\t\t\t\t{}\\leq{}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{L_{\\sDRS}}{2}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{\\gamma L_{\\sDRS}^2}{\\lambda}\n\t\t\t\t\\quad\\Leftrightarrow\\quad\n\t\t\t\t\t\\gamma\n\t\t\t\t{}\\leq{}\n\t\t\t\t\t\\tfrac{\\lambda}{2(L_{\\sDRS}+\\sigma)},\n\t\t\t\t\\]\n\t\t\t\tfrom \\eqref{eq:cbar} we conclude that \\(c\\) is as in \\eqref{eq:DRE:SD}.\n\t\t\t%%%%\n\t\t\t%%%%\n\t\t\t%%%%\n\t\t\t\\item\\textbf{Case 1b:} \\(2(1+\\nicefrac{\\sigma}{L_{\\sDRS}})<\\lambda<2\\).\n\t\t\t\t\n\t\t\t\tNecessarily \\(\\sigma<0\\), for otherwise the range of \\(\\lambda\\) would be empty.\n\t\t\t\tIn particular, \\(\\sigma=\\sigma_{\\sDRS}\\), and the lower bound on \\(\\lambda\\) can be expressed as\n\t\t\t\t\\(\n\t\t\t\t\t\\sigma_{\\sDRS}<-\\frac{2-\\lambda}{2}L_{\\sDRS}\n\t\t\t\t\\).\n\t\t\t\tConsequently, \\(L\\coloneqq\\tfrac{-2\\sigma_{\\sDRS}}{2-\\lambda}\\) is strictly larger than \\(L_{\\sDRS}\\), and in particular \\(\\sigma+L=\\sigma_{\\sDRS}+L>0\\).\n\t\t\t\tThe ratio of \\(\\sigma\\) and \\(L\\) is thus \\(p=\\tfrac\\lambda2-1\\), and \\eqref{eq:c} becomes\n\t\t\t\t\\begin{equation}\\label{eq:c2}\n\t\t\t\t\tc\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\begin{ifcases}\n\t\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n\t\t\t\t\t&\n\t\t\t\t\t\t\\gamma\n\t\t\t\t\t\t{}<{}\n\t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n\t\t\t\t\t\\\\[3pt]\n\t\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}-\\tfrac{\\gamma L_{\\sDRS}^2}{\\lambda}+\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}\\lambda}L_{\\sDRS}^2\n\t\t\t\t\t\\otherwise.\n\t\t\t\t\t\\end{ifcases}\n\t\t\t\t\\end{equation}\n\t\t\t\tLet us show that, when\n\t\t\t\t\\(\n\t\t\t\t\t\\gamma\n\t\t\t\t{}<{}\n\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac1L\n\t\t\t\t\\),\n\t\t\t\talso in this case the sufficient decrease constant \\(c\\) is strictly positive.\n\t\t\t\tWe have\n\t\t\t\t\\[\n\t\t\t\t\t\\tfrac cL\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma L}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n\t\t\t\t\t\\tfrac1L\n\t\t\t\t{}>{}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n\t\t\t\t{}={}\n\t\t\t\t\t0,\n\t\t\t\t\\]\n\t\t\t\thence the claim.\n\t\t\t\tThis concludes the proof for the case \\(\\lambda\\in(0,2)\\).\n% \t\t\t\tBy imposing \\(c>0\\) this reduces to\n% \t\t\t\t\\begin{align*}\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\quad\\cup\\quad &\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n% \t\t\t\t\t\t{}-{}\n% \t\t\t\t\t\t\\tfrac{\\gamma L_{\\sDRS}^2}{\\lambda}\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}\\lambda}L_{\\sDRS}^2\n% \t\t\t\t\t{}={}\n% \t\t\t\t\t\t\\smashoverbracket{\n% \t\t\t\t\t\t\t\\tfrac1\\lambda\n% \t\t\t\t\t\t\t\\bigl(\n% \t\t\t\t\t\t\t\t\\gamma L_{\\sDRS}^2-\\sigma_{\\sDRS}\n% \t\t\t\t\t\t\t\\bigr)\n% \t\t\t\t\t\t}{>0}\n% \t\t\t\t\t\t\\bigl(\n% \t\t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\gamma\\sigma_{\\sDRS}}\n% \t\t\t\t\t\t\t{}-{}\n% \t\t\t\t\t\t\t1\n% \t\t\t\t\t\t\\bigr)\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}\\geq{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}.\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\shortintertext{Equivalently,}\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t\t{}<{}\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t\t{}<{}\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\quad\\cup\\quad &\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}<{}\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}\\geq{}\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}},\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\end{align*}\n% \t\t\t\tthat is,\n% \t\t\t\t\\(\n% \t\t\t\t\t\\gamma\n% \t\t\t\t{}<{}\n% \t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t\\).\n% \t\t\t\tFrom \\eqref{eq:c2} it follows that\n% \t\t\t\t\\(\n% \t\t\t\t\tc\n% \t\t\t\t{}={}\n% \t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t{}+{}\n% \t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n% \t\t\t\t\\).%\n\t\t\t\\end{itemize}\n% \t\t\tLet us recap {\\bf Case 1}:\n% \t\t\t\\[\n% \t\t\t\t\\begin{cases}[@{\\gamma<{}} l<{,} @{~~c={}} l @{\\quad\\text{if~~}} l]\n% \t\t\t\t\t\\tfrac{1}{L_{\\sDRS}}\n% \t\t\t\t&\n% \t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t{}-{}\n% \t\t\t\t\t\\max\\set{\n% \t\t\t\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{2(1-[p_{\\sDRS}]_-)},\\,\n% \t\t\t\t\t\tL_{\\sDRS}\\bigl(\\tfrac12-\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\\bigr)\n% \t\t\t\t\t}\n% \t\t\t\t&\n% \t\t\t\t\t0<\\lambda<2(1-[p_{\\sDRS}]_-),\n% \t\t\t\t\\\\\n% \t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t&\n% \t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t{}-{}\n% \t\t\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{\\lambda}\n% \t\t\t\t&\n% \t\t\t\t\t2(1-[p_{\\sDRS}]_-)<\\lambda<2.\n% \t\t\t\t\\end{cases}\n% \t\t\t\\]\n% \t\t\tWe may unify the two cases with the more compact notation\n% \t\t\t\\[\n% \t\t\t\t\\gamma\n% \t\t\t{}<{}\n% \t\t\t\t\\min\\set{\n% \t\t\t\t\t\\tfrac{1}{L_{\\sDRS}},\\,\n% \t\t\t\t\t\\tfrac{2-\\lambda}{-2[\\sigma_{\\sDRS}]_-}\n% \t\t\t\t}\n% \t\t\t\\quad\\text{and}\\quad\n% \t\t\t\tc\n% \t\t\t{}={}\n% \t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t{}-{}\n% \t\t\t\t\\min\\set{\n% \t\t\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{\\lambda},~\n% \t\t\t\t\t\\max\\set{\n% \t\t\t\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{2(1-[p_{\\sDRS}]_-)},\\,\n% \t\t\t\t\t\tL_{\\sDRS}\\bigl(\\tfrac12-\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\\bigr)\n% \t\t\t\t\t}\n% \t\t\t\t},\n% \t\t\t\\]\n% \t\t\tthus obtaining the range of \\(\\gamma\\) and the value of \\(c\\) indicated in \\eqref{eq:c_nonstr}.\n\t\t\t%%%\n\t\t\t%%%\n\t\t\t%%%\n\t\t\\item\\textbf{Case 2: \\(\\lambda\\geq2\\).}\n\n\t\t\t\\noindent\n\t\t\tIn this case we need to assume that \\(\\sDRS\\) is strongly convex, that is, that \\(\\sigma_{\\sDRS}>0\\).\n\t\t\tInstead of considering a single expression of \\(\\rho\\), we will rather take a convex combination of those in \\cref{thm:hypo:rho0,thm:hypo:rho1}, namely\n\t\t\t\\[\n\t\t\t\t\\rho(u,u^+)\n\t\t\t{}={}\n\t\t\t\t(1-\\alpha)\\tfrac{\\sigma_{\\sDRS}}{2}\\|u-u^+\\|^2\n\t\t\t\t{}+{}\n\t\t\t\t\\alpha\\tfrac{1}{2L_{\\sDRS}}\\|\\nabla\\sDRS(u)-\\nabla\\sDRS(u^+)\\|^2\n\t\t\t\\]\n\t\t\tfor some \\(\\alpha\\in[0,1]\\) to be determined.\n\t\t\t\\eqref{eq:DRE:SD0} then becomes\n\t\t\t\\[\n\t\t\t\t\\tfrac{\n\t\t\t\t\t\\DRE(s)-\\DRE(s^+)\n\t\t\t\t}{\n\t\t\t\t\tL_{\\sDRS}\n\t\t\t\t}\n\t\t\t{}\\geq{}\n\t\t\t\t\\left(\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\xi}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{(1-\\alpha)p}{2}\n\t\t\t\t\\right)\n\t\t\t\t\\|u-u^+\\|^2\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{L_{\\sDRS}^2}\n\t\t\t\t\\left(\n\t\t\t\t\t\\tfrac{\\alpha}{2}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{\\xi}{\\lambda}\n\t\t\t\t\\right)\n\t\t\t\t\\|\\nabla\\sDRS(u)-\\nabla\\sDRS(u^+)\\|^2,\n\t\t\t\\]\n\t\t\twhere \\(\\x\\coloneqq\\gamma L_{\\sDRS}\\) and \\(p\\coloneqq\\nicefrac{\\sigma_{\\sDRS}}{L_{\\sDRS}}\\in(0,1]\\).\n\t\t\tBy restricting \\(\\xi\\in(0,1)\\), since \\(\\lambda\\geq2\\) one can take\n\t\t\t\\(\n\t\t\t\t\\alpha\n\t\t\t{}\\coloneqq{}\n\t\t\t\t\\nicefrac{2\\xi}{\\lambda}\n\t\t\t{}\\in{}\n\t\t\t\t(0,1)\n\t\t\t\\)\n\t\t\tto make the coefficient multiplying the gradient norm vanish.\n\t\t\tWe then obtain\n\t\t\t\\begin{equation}\\label{eq:clambda2}\n\t\t\t\t\\tfrac{c}{L_{\\sDRS}}\n\t\t\t{}={}\n\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\xi}\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{(\\lambda-2\\xi)p}{2\\lambda}.\n\t\t\t\\end{equation}\n\t\t\tImposing \\(c>0\\) results in the following second-order equation in variable \\(\\xi\\),\n\t\t\t\\begin{equation}\\label{eq:xi}\n\t\t\t\t2p\\xi^2-p\\lambda\\xi+(\\lambda-2)\n\t\t\t{}<{}\n\t\t\t\t0.\n\t\t\t\\end{equation}\n\t\t\tThe discriminant is\n\t\t\t\\(\n\t\t\t\t\\Delta\n\t\t\t{}\\coloneqq{}\n\t\t\t\t(p\\lambda)^2\n\t\t\t\t{}-{}\n\t\t\t\t8p(\\lambda-2)\n\t\t\t\\),\n\t\t\twhich, for \\(\\lambda\\geq2\\), is strictly positive iff\n\t\t\t\\[\n\t\t\t\t2\n\t\t\t{}\\leq{}\n\t\t\t\t\\lambda\n\t\t\t{}<{}\n\t\t\t\t\\tfrac{4}{1+\\sqrt{1-p}}\n\t\t\t~~\\vee~~\n\t\t\t\t\\lambda\n\t\t\t{}>{}\n\t\t\t\t\\tfrac{4}{1-\\sqrt{1-p}}.\n\t\t\t\\]\n\t\t\tDenoting\n\t\t\t\\(\n\t\t\t\t\\delta\n\t\t\t{}\\coloneqq{}\n\t\t\t\t\\sqrt\\Delta\n\t\t\t{}={}\n\t\t\t\t\\sqrt{\n\t\t\t\t\t(p\\lambda)^2\n\t\t\t\t\t{}-{}\n\t\t\t\t\t8p(\\lambda-2)\n\t\t\t\t}\n\t\t\t\\),\n\t\t\tthe solution to \\eqref{eq:xi} is\n\t\t\t\\(\n\t\t\t\t\\tfrac{\n\t\t\t\t\tp\\lambda\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\delta\n\t\t\t\t}{\n\t\t\t\t\t4p\n\t\t\t\t}\n\t\t\t{}<{}\n\t\t\t\t\\x\n\t\t\t{}<{}\n\t\t\t\t\\tfrac{\n\t\t\t\t\tp\\lambda\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\delta\n\t\t\t\t}{\n\t\t\t\t\t4p\n\t\t\t\t}\n\t\t\t\\).\n\t\t\tHowever, the case \\(\\lambda\\geq 4\\) has to be discarded, as\n\t\t\t\\(\n\t\t\t\t\\tfrac{p\\lambda-\\delta}{4p}\n\t\t\t{}>{}\n\t\t\t\t1\n\t\t\t\\)\n\t\t\tin this case, contradicting the fact that \\(p\\leq1\\).\n\t\t\tTo see this, suppose \\(\\lambda\\geq4\\).\n\t\t\tThen,\n\t\t\t\\begin{align*}\n\t\t\t\t\\tfrac{\n\t\t\t\t\tp\\lambda\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\delta\n\t\t\t\t}{\n\t\t\t\t\t4p\n\t\t\t\t}\n\t\t\t{}<{}\n\t\t\t\t1\n\t\t\t~~\\Leftrightarrow~~ &\n\t\t\t\tp(\\lambda-4)\n\t\t\t{}<{}\n\t\t\t\t\\delta\n\t\t\t\\\\\n\t\t\t~~\\Leftrightarrow~~ &\n\t\t\t\tp^2(\\lambda-4)^2\n\t\t\t{}<{}\n\t\t\t\t\\Delta\n\t\t\t{}={}\n\t\t\t\t(p\\lambda)^2\n\t\t\t\t{}-{}\n\t\t\t\t8p(\\lambda-2)\n\t\t\t\\\\\n\t\t\t~~\\Leftrightarrow~~ &\n\t\t\t\tp(2-\\lambda)\n\t\t\t{}<{}\n\t\t\t\t2-\\lambda,\n\t\t\t\\end{align*}\n\t\t\thence \\(p>1\\), which contradicts the fact that \\(\\sigma_{\\sDRS}\\leq L_{\\sDRS}\\).\n% \t\t\tHere and in the second implication we used the assumption \\(\\lambda\\geq4\\).\n\t\t\tThus, the only feasible ranges are the ones given in \\eqref{eq:lambda>=2}, hence the claimed sufficient decrease constant \\(c\\), cf. \\eqref{eq:clambda2}.\n\t\t\t\\qedhere\n\t\t\\end{proofitemize}\n% \t\tAlso in this case, the claimed range of \\(\\gamma\\) and value of \\(c\\) have been found.\n\t\\end{proof}\n\\end{thm}\n\n\\begin{proof}\n\t\tTo avoid trivialities, we assume that a fixed point is not found in a finite number of iterations, hence that \\(v^k\\neq u^k\\) for all \\(k\\)'s.\n\t\t\\begin{proofitemize}\n\t\t\\item\\ref{thm:DRS:r}\n\t\t\tLet \\(c=c(\\gamma,\\lambda)\\) be as in \\cref{thm:DRE:SD}.\n\t\t\tTelescoping the inequality \\eqref{eq:DRE:SD} yields\n\t\t\t\\begin{align*}\n\t\t\t\\textstyle\n\t\t\t\t\\tfrac{c\\lambda^2}{(1+\\gamma L_{\\sDRS})^2}\n\t\t\t\t\\sum_{k\\in\\N}{\n\t\t\t\t\t\\|u^k-v^k\\|^2\n\t\t\t\t}\n\t\t\t{}\\leq{} &\n\t\t\t\\textstyle\n\t\t\t\t\\sum_{k\\in\\N}{\n\t\t\t\t\\bigl[\n\t\t\t\t\t\\DRE(s^k)\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\DRE(s^{k+1})\n\t\t\t\t\\bigr]}\n\t\t\t{}\\leq{}\n\t\t\t\t\\DRE(s^0)-\\inf\\DRE.\n\t\t\t\\end{align*}\n\t\t\tSince \\(\\inf\\DRE=\\inf\\varphi>-\\infty\\) and \\(\\DRE\\) is real valued (cf. \\cref{thm:C0,thm:DREequiv}), it follows that \\(\\seq{u^k-v^k}\\) is square summable, hence the claimed rate of convergence.\n\t\t\tMoreover, since \\(\\DRE(s^k)\\) is decreasing it admits a (finite) limit, be it \\(\\varphi_\\star\\).\n\t%%%%%%%%\n\t\t\\item\\ref{thm:DRS:uv}\n\t\t\tSince \\(\\seq{u^k-v^k}\\to 0\\), necessarily \\(\\seq{u^k}\\) and \\(\\seq{v^k}\\) have same cluster points.\n\t\t\tSuppose that \\(\\seq{u^k}[k\\in K]\\to u'\\) for some \\(K\\subseteq\\N\\) and \\(u'\\in\\R^p\\).\n\t\t\tThen, \\(\\seq{v^k}[k\\in K]\\to u'\\), and since \\(s^k=u^k+\\nabla\\sDRS(u^k)\\) (cf. \\cref{thm:proxfEquiv}), continuity of \\(\\nabla\\sDRS\\) implies that\n\t\t\t\\(\n\t\t\t\t\\seq{s^k}[k\\in K]\n\t\t\t{}\\to{}\n\t\t\t\ts'\n\t\t\t{}={}\n\t\t\t\tu'+\\gamma\\nabla\\sDRS(u')\n\t\t\t\\).\n\t\t\tFrom \\cref{thm:proxfEquiv} we infer that \\(u'=\\prox_{\\gamma\\sDRS}(s')\\).\n\t\t\t\n\t\t\tSimilarly, \\(\\seq{\\Fw{u^k}}[k\\in K]\\to \\Fw{u'}\\), and the outer semicontinuity of \\(\\prox_{\\gamma\\nsDRS}\\) \\cite[Ex. 5.23(b)]{rockafellar2011variational} combined with \\eqref{eq:vFBS} implies that\n\t\t\t\\[\n\t\t\t\tu'\n\t\t\t{}={}\n\t\t\t\t\\lim_{K\\ni k\\to\\infty}{\n\t\t\t\t\t\\hspace*{-5pt}\n\t\t\t\t\tv^k\n\t\t\t\t}\n\t\t\t{}\\in{}\n\t\t\t\t\\limsup_{K\\ni k\\to\\infty}{\n\t\t\t\t\t\\FB{u^k}\n\t\t\t\t}\n\t\t\t{}\\subseteq{}\n\t\t\t\t\\FB{u'}.\n\t\t\t\\]\n\t\t\tFrom \\eqref{eq:prox:subdiff} we then have that\n\t\t\t\\(\n\t\t\t\t-\\nabla\\sDRS(u')\n\t\t\t{}\\in{}\n\t\t\t\t\\hat\\partial\\nsDRS(u')\n\t\t\t\\),\n\t\t\thence \\(0\\in\\hat\\partial\\varphi(u')\\), as it follows from \\cite[Ex. 8.8]{rockafellar2011variational}.\n\t\t\tFinally, since \\(v^k\\to u'\\),\n\t\t\t\\[\n\t\t\t\t\\varphi(u')\n\t\t\t{}\\leq{}\n\t\t\t\t\\liminf_{K\\ni k\\to\\infty}{\n\t\t\t\t\t\\varphi(v^k)\n\t\t\t\t}\n\t\t\t{}\\leq{}\n\t\t\t\t\\limsup_{K\\ni k\\to\\infty}{\n\t\t\t\t\t\\varphi(v^k)\n\t\t\t\t}\n\t\t\t{}\\leq{}\n\t\t\t\t\\limsup_{K\\ni k\\to\\infty}{\n\t\t\t\t\t\\DRE(s^k)\n\t\t\t\t}\n\t\t\t{}={}\n\t\t\t\t\\DRE(s')\n\t\t\t{}\\leq{}\n\t\t\t\t\\varphi(u'),\n\t\t\t\\]\n\t\t\twhere the first inequality is due to lower semicontinuity of \\(\\varphi\\), the third and the last to the sandwiching property (\\cref{thm:sandwich}), and the equality to the continuity of \\(\\DRE\\) (\\cref{thm:C0}).\n\t\t\tThis shows that \\(\\seq{\\varphi(u^k)}[k\\in K]\\to\\varphi(u')=\\DRE(s')\\), and since \\(\\seq{\\DRE(s^k)}\\to\\varphi_\\star\\), then necessarily \\(\\varphi(u')=\\DRE(s')=\\varphi_\\star\\) independently of the cluster point \\(u'\\).\n\t\t\tThe last assert follows from the optimality condition of \\(v^k\\) as in \\eqref{eq:vFBS}, namely,\n\t\t\t\\(\n\t\t\t\t\\tfrac1\\gamma(u^k-v^k)\n\t\t\t\t{}-{}\n\t\t\t\t\\nabla\\sDRS(u^k)\n\t\t\t{}\\in{}\n\t\t\t\t\\hat\\partial\\nsDRS(v^k)\n\t\t\t\\)\n\t\t\tdue to \\eqref{eq:prox:subdiff}, together with \\(\\sigma_{\\sDRS}\\)-hypoconvexity of \\(\\sDRS\\).\n\t\t\\item\\ref{thm:DRS:bounded}.\n\t\t\tSuppose that \\(\\varphi\\) has bounded level sets.\n\t\t\tThen, it follows from \\cref{thm:boundedlev} that so does \\(\\DRE\\), and since\n\t\t\t\\(\n\t\t\t\ts^k\n\t\t\t{}\\in{}\n\t\t\t\t\\lev_{\\leq\\DRE(s^0)}\\DRE\n\t\t\t\\)\n\t\t\tfor all \\(k\\in\\N\\), then the sequence \\(\\seq{s^k}\\) is bounded.\n\t\t\tDue to Lipschitz continuity of \\(\\prox_{\\gamma\\sDRS}\\) (cf. \\cref{thm:proxfLip}), also \\(\\seq{u^k}\\) is bounded.\n\t\t\tIn turn, since \\(v^k-u^k\\to 0\\) we conclude that also \\(\\seq{v^k}\\) is bounded.\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}\n\n\\begin{appendixproof}{thm:rho}%\n\t\\begin{proofitemize}\n\t\\item\\ref{thm:hypo:rho0}.\n\t\tThis is the lower bound in \\eqref{eq:LipBound}.\n\t\\item\\ref{thm:hypo:rho1}.\n\t\tLet \\(L\\geq L_h\\) and \\(\\sigma\\in(-L,\\min\\set{0,\\sigma_h}]\\) be fixed.\n\t\tThen, \\(h\\) is \\(L\\)-smooth and \\(\\sigma\\)-hypoconvex, and from \\cite[Thm. 2.1.12]{nesterov2003introductory} we obtain that\n\t\t\\begin{equation}\\label{eq:LipBound:mix}\n\t\t\t\\innprod{\n\t\t\t\t\\nabla h(y)-\\nabla h(x)\n\t\t\t}{\n\t\t\t\ty-x\n\t\t\t}\n\t\t{}\\geq{}\n\t\t\t\\tfrac{\\sigma L}{L+\\sigma}\\|x-y\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{L+\\sigma}\\|\\nabla h(x)-\\nabla h(y)\\|^2\n\t\t\\end{equation}\n\t\tfor all \\(x,y\\in\\R^n\\).\n\t\t(Although \\cite[Thm. 2.1.12]{nesterov2003introductory} assumes \\(\\sigma>0\\), the given proof does not necessitate this restriction).\n\t\tMoreover, \\(\\psi\\coloneqq h-\\tfrac\\sigma2\\|{}\\cdot{}\\|^2\\) is convex and \\(L_\\psi\\)-smooth, with \\(L_\\psi=L-\\sigma\\).\n\t\tConsequently, for all \\(x,y\\in\\R^n\\) one has\n\t\t\\(\n\t\t\t\\psi(y)\n\t\t{}\\geq{}\n\t\t\t\\psi(x)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla\\psi(x)}{y-x}\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2L_\\psi}\\|\\nabla\\psi(y)-\\nabla\\psi(x)\\|^2\n\t\t\\),\n\t\tsee \\cite[Thm. 2.1.5]{nesterov2003introductory}, resulting in\n\t\t\\begin{align*}\n\t\t\th(y)\n\t\t{}\\geq{} &\n\t\t\th(x)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla h(x)}{y-x}\n\t\t\t{}+{}\n\t\t\t\\tfrac{\\sigma L}{2(L-\\sigma)}\n\t\t\t\\|y-x\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2(L-\\sigma)}\n\t\t\t\\|\\nabla h(y)-\\nabla h(x)\\|^2\n\t\t\\\\\n\t\t&\n\t\t\t{}-{}\n\t\t\t\\tfrac{\\sigma}{2(L-\\sigma)}\n\t\t\t\\innprod{\n\t\t\t\t\\nabla h(y)-\\nabla h(x)\n\t\t\t}{\n\t\t\t\ty-x\n\t\t\t}.\n\t\t\\end{align*}\n\t\tSince \\(\\sigma\\leq0\\), the coefficient of the scalar product in the second line is positive.\n\t\tWe may thus invoke the inequality \\eqref{eq:LipBound:mix} to arrive to\n\t\t\\begin{align*}\n\t\t\th(y)\n\t\t{}\\geq{} &\n\t\t\th(x)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla h(x)}{y-x}\n\t\t\t{}+{}\n\t\t\t\\tfrac{\\sigma L}{2(L-\\sigma)}\n\t\t\t\\|y-x\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2(L-\\sigma)}\n\t\t\t\\|\\nabla h(y)-\\nabla h(x)\\|^2\n\t\t\\\\\n\t\t&\n\t\t\t{}-{}\n\t\t\t\\tfrac{\\sigma}{2(L-\\sigma)}\n\t\t\t\\left[\n\t\t\t\t\\tfrac{\\sigma L}{L+\\sigma}\\|x-y\\|^2\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{L+\\sigma}\\|\\nabla h(x)-\\nabla h(y)\\|^2\n\t\t\t\\right]\n\t\t\\\\\n\t\t{}={} &\n\t\t\th(x)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla h(x)}{y-x}\n\t\t\t{}+{}\n\t\t\t\\tfrac{\\sigma L}{2(L+\\sigma)}\n\t\t\t\\|y-x\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2(L+\\sigma)}\n\t\t\t\\|\\nabla h(y)-\\nabla h(x)\\|^2,\n\t\t\\end{align*}\n\t\thence the claimed inequality.\n\t\\qedhere\n\t\\end{proofitemize}\n\\end{appendixproof}\n\n\\begin{appendixproof}{thm:proxf}%\n\tLet \\(\\gamma\\in(0,\\nicefrac{1}{[\\sigma_h]_-})\\) be fixed, and let \\(\\psi\\coloneqq\\gamma h+\\tfrac12\\|{}\\cdot{}\\|^2\\).\n\tObserve that \\(\\psi\\in\\cont^{1,1}(\\R^n)\\) is \\(L_\\psi\\)-smooth and \\(\\sigma_\\psi\\)-strongly convex, with \\(L_\\psi=1+\\gamma L_h\\) and \\(\\sigma_\\psi=1+\\gamma\\sigma_h\\).\n\tIn particular, due to strong convexity \\(\\inf\\psi>-\\infty\\), and by definition of prox-boundedness it then follows that \\(\\gamma_h\\geq\\nicefrac{1}{[\\sigma_h]_-}\\).\n\t\\begin{proofitemize}\n\t\\item\\ref{thm:proxfEquiv}.\n\t\tFollows from \\eqref{eq:prox:subdiff}, by observing that \\(h+\\tfrac{1}{2\\gamma}\\|{}\\cdot{}-s\\|^2\\) is strongly convex, hence that a minimizer is characterized by stationarity.\n\t\\item\\ref{thm:proxfLip}.\n\t\tFor \\(s,s'\\in\\R^n\\), let \\(u=\\prox_{\\gamma h}(s)\\) and \\(u'=\\prox_{\\gamma h}(s')\\).\n\t\tThen,\n\t\t\\[\n\t\t\t\\innprod{s-s'}{u-u'}\n\t\t{}={}\n\t\t\t\\innprod{\n\t\t\t\t\\nabla\\psi(u)\n\t\t\t\t{}-{}\n\t\t\t\t\\nabla\\psi(u')\n\t\t\t}{u-u'}\n\t\t{}\\geq{}\n\t\t\t\\sigma_\\psi\n\t\t\t\\|u-u'\\|^2\n\t\t{}={}\n\t\t\t(1+\\gamma\\sigma_h)\n\t\t\t\\|u-u'\\|^2,\n\t\t\\]\n\t\twhere the first equality was shown in \\ref{thm:proxfEquiv} and the inequality follows from \\eqref{eq:innprod}.\n\t\tBy using the \\(\\frac{1}{L_\\psi}\\)-cocoercivity of \\(\\nabla\\psi\\) \\cite[Thm. 2.1.10]{nesterov2003introductory}, also the claimed strong monotonicity follows.\n\t\tIn turn, the Cauchy-Schwartz inequality on the inner product yields \\eqref{eq:biLip}.\n\t\\item\\ref{thm:MoreaufC1}.\n\t\tFrom \\cite[Ex. 10.32]{rockafellar2011variational} it follows that \\(h^\\gamma\\) is strictly continuous and that\n\t\t\\(\n\t\t\t\\partial h^\\gamma(s)\n\t\t{}\\subseteq{}\n\t\t\t\\tfrac1\\gamma(s-\\prox_{\\gamma h}(s))\n\t\t\\).\n\t\tBecause of single valuedness of \\(\\prox_{\\gamma h}\\), by invoking \\cite[Thm. 9.18]{rockafellar2011variational} we conclude that \\(h^\\gamma\\) is everywhere differentiable with\n\t\t\\(\n\t\t\t\\nabla h^\\gamma(s)\n\t\t{}={}\n\t\t\t\\tfrac1\\gamma(s-\\prox_{\\gamma h}(s))\n\t\t\\).\n% \t\tMoreover, \\(u\\) minimizes \\(h+\\tfrac{1}{2\\gamma}\\|{}\\cdot{}-s\\|^2\\) and in particular \\(\\tfrac{1}{\\gamma}(s-u)\\in\\partial h(u)\\).\n% \t\tTherefore,\n% \t\t\\(\n% \t\t\t\\nabla h(u)\n% \t\t{}\\overrel*{\\eqref{eq:NablaProj}}{}\n% \t\t\t\\proj_{\\dom h}\\partial h(u)\n% \t\t{}={}\n% \t\t\t\\tfrac{1}{\\gamma}(\\proj_{\\dom h}s-u)\n% \t\t\\),\n% \t\twhere the last equality follows from the fact that \\(u\\in\\dom h\\).\n\t\tThus,\n\t\t\\begin{align*}\n\t\t\t\\innprod{\\nabla h^\\gamma(s)-\\nabla h^\\gamma(s')}{s-s'}\n\t\t{}={} &\n\t\t\t\\tfrac{1}{\\gamma}\n\t\t\t\\left(\n\t\t\t\t\\|s-s'\\|^2\n\t\t\t\t{}-{}\n\t\t\t\t\\innprod{s-s'}{u-u'}\n\t\t\t\\right),\n% \t\t\\\\\n% \t\t{}\\leq{} &\n% \t\t\t\\tfrac{1}{\\gamma}\n% \t\t\t\\bigl(\n% \t\t\t\t1\n% \t\t\t\t{}-{}\n% \t\t\t\t\\tfrac{1}{1+\\gamma L_{h}}\n% \t\t\t\\bigr)\n% \t\t\t\t\\|s-s'\\|^2\n% \t\t{}={}\n% \t\t\t\\tfrac{L_{h}}{1+\\gamma L_{h}}\n% \t\t\t\\|s-s'\\|^2.\n\t\t\\end{align*}\n\t\tand from the bounds in \\ref{thm:proxfLip} we conclude that\n% \t\tSimilarly, by using also the second inequality in \\eqref{eq:coco} we arrive to\n\t\t\\[\n\t\t\t\\tfrac{\\sigma_{h}}{1+\\gamma\\sigma_{h}}\n\t\t\t\\|s-s'\\|^2\n\t\t{}\\leq{}\n\t\t\t\\innprod{\\nabla h^\\gamma(s)-\\nabla h^\\gamma(s')}{s-s'}\n\t\t{}\\leq{}\n\t\t\t\\tfrac{L_{h}}{1+\\gamma L_{h}}\n\t\t\t\\|s-s'\\|^2.\n\t\t\\]\n\t\tThe claimed smoothness and hypoconvexity follow from the characterization of \\eqref{eq:innprod}.\n\t\t\\qedhere\n\t\\end{proofitemize}\n\\end{appendixproof}","units":[{"start":41391,"end":42263,"start_line":678,"end_line":720,"raw_tex":"\\begin{thm}[Lower bounds for smooth functions]\\label{thm:rho}%\n\tLet \\(h\\in C^{1,1}(\\R^n)\\) be \\(L_h\\)-smooth and \\(\\sigma_h\\)-hypoconvex.\n\tThen, for all \\(x,y\\in\\R^n\\) it holds that\n\t\\[\n\t\th(y)\n\t{}\\geq{}\n\t\th(x)\n\t\t{}+{}\n\t\t\\innprod{\\nabla h(x)}{y-x}\n\t\t{}+{}\n\t\t\\rho(y,x),\n\t\\]\n\twhere\n\t\\begin{enumerate}\n\t\\item\\label{thm:hypo:rho0}%\n\t\teither~\n\t\t\\(\n\t\t\t\\rho(y,x)\n\t\t{}={}\n\t\t\t\\tfrac{\\sigma_h}{2}\n\t\t\t\\|y-x\\|^2\n\t\t\\),\n\t\\item\\label{thm:hypo:rho1}%\n\t\tor~\n\t\t\\(\n\t\t\t\\rho(y,x)\n\t\t{}={}\n\t\t\t\\tfrac{\\sigma_hL_h}{2(L_h+\\sigma_h)}\n\t\t\t\\|y-x\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2(L_h+\\sigma_h)}\n\t\t\t\\|\\nabla h(y)-\\nabla h(x)\\|^2\n\t\t\\),~\n\t\tprovided that~\n\t\t\\(\n\t\t\t-L_h<\\sigma_h\\leq 0\n\t\t\\).\n\t\\end{enumerate}\n\tClearly, all inequalities remain valid if one replaces \\(L_h\\) with any \\(L\\geq L_h\\) and \\(\\sigma_h\\) with any \\(\\sigma\\in[-L,\\sigma_h]\\).\n\t\\begin{proof}\n\t\tSee \\Cref{proof:thm:rho}.\n\t\\end{proof}\n\\end{thm}","sha256_utf8":"27efc774ff1eb82f641c7bf78f71c5496129d9fc12b16052b6b40f38bb486355"},{"start":44157,"end":46146,"start_line":766,"end_line":832,"raw_tex":"\\begin{prop}[Proximal properties of smooth functions]\\label{thm:proxf}%\n\tLet \\(h\\in \\cont^{1,1}(\\dom h)\\) be \\(L_h\\)-smooth, hence \\(\\sigma_h\\)-hypoconvex for some \\(\\sigma_h\\in[-L_h,L_h]\\).\n\tThen, \\(h\\) is prox-bounded with \\(\\gamma_{h}\\geq\\nicefrac{1}{[\\sigma_{h}]_-}\\) and for all \\(\\gamma<\\nicefrac{1}{[\\sigma_{h}]_-}\\) the following hold:\n\t\\begin{enumerate}\n\t\\item\\label{thm:proxfEquiv}\n\t\t\\(\\prox_{\\gamma h}\\) is single valued, and for all \\(s\\in\\R^n\\) it holds that \\(u=\\prox_{\\gamma h}(s)\\) iff\n\t\t\\(\n\t\t\ts=u+\\gamma\\nabla h(u)\n\t\t\\).\n\t\\item\\label{thm:proxfLip}\n\t\t\\(\\prox_{\\gamma h}\\) is \\((\\tfrac{1}{1+\\gamma L_{h}})\\)-strongly monotone and \\((1+\\gamma\\sigma_h)\\)-cocoercive, in the sense that\n\t\t\\[\n\t\t\t\\innprod{u-u'}{s-s'}\n\t\t{}\\geq{}\n\t\t\t\\tfrac{1}{1+\\gamma L_h}\n\t\t\t\\|s-s'\\|^2\n\t\t\\quad\\text{and}\\quad\n\t\t\t\\innprod{u-u'}{s-s'}\n\t\t{}\\geq{}\n\t\t\t(1+\\gamma\\sigma_h)\n\t\t\t\\|u-u'\\|^2\n\t\t\\]\n\t\tfor all \\(s,s'\\in\\R^n\\), where \\(u=\\prox_{\\gamma h}(s)\\) and \\(u'=\\prox_{\\gamma h}(s')\\).\n\t\tIn particular,\n\t\t\\begin{equation}\\label{eq:biLip}\n\t\t\t\\tfrac{1}{1+\\gamma L_h}\n\t\t\t\\|s-s'\\|\n\t\t{}\\leq{}\n\t\t\t\\|u-u'\\|\n\t\t{}\\leq{}\n\t\t\t\\tfrac{1}{1+\\gamma\\sigma_h}\n\t\t\t\\|s-s'\\|.\n\t\t\\end{equation}\n\t\tThus, \\(\\prox_{\\gamma h}\\) is a \\(\\tfrac{1}{1+\\gamma\\sigma_h}\\)-Lipschitz and invertible mapping, and its inverse \\(\\id+\\gamma\\nabla h\\) is \\((1+\\gamma L_h)\\)-Lipschitz continuous.\n\t\\item\\label{thm:MoreaufC1}\n\t\t\\(h^\\gamma\\in \\cont^{1,1}(\\R^n)\\) is \\(L_{h^\\gamma}\\)-smooth and \\(\\sigma_{h^\\gamma}\\)-hypoconvex, with\n\t\t\\(\n\t\t\tL_{h^\\gamma}\n\t\t{}={}\n\t\t\t\\max\\set{\n\t\t\t\t\\tfrac{L_{h}}{1+\\gamma L_{h}},~\n\t\t\t\t\\tfrac{[\\sigma_{h}]_-}{1+\\gamma\\sigma_{h}}\n\t\t\t}\n\t\t\\)\n\t\tand\n\t\t\\(\n\t\t\t\\sigma_{h^\\gamma}\n\t\t{}={}\n\t\t\t\\tfrac{\\sigma_{h}}{1+\\gamma\\sigma_{h}}\n\t\t\\).\n\t\tMoreover,\n\t\t\\(\n\t\t\t\\nabla h^\\gamma(s)\n\t\t{}={}\n\t\t\t\\tfrac{1}{\\gamma}(s-\\prox_{\\gamma h}(s))\n\t\t\\)\n\t\tand\n\t\t\\(\n\t\t\t\\nabla h(\\prox_{\\gamma h}(s))\n\t\t{}={}\n\t\t\t\\tfrac{1}{\\gamma}\\bigl(s-\\prox_{\\gamma h}(s)\\bigr)\n\t\t\\).\n\t\\end{enumerate}\n\t\\begin{proof}\n\t\tSee \\Cref{proof:thm:proxf}.\n\t\\end{proof}\n\\end{prop}","sha256_utf8":"00fa64e786e8a54819a783fc11c11232f48429ea54047bd507d09487388653a8"},{"start":47338,"end":47860,"start_line":856,"end_line":867,"raw_tex":"\tThat \\(\\gamma_{\\sDRS}\\geq\\nicefrac{1}{L{\\sDRS}}\\) follows from \\Cref{thm:proxf}, having \\(\\nicefrac{1}{[\\sigma_{\\sDRS}]_-}\\geq\\nicefrac{1}{L_{\\sDRS}}\\).\n\tAs for \\(\\nsDRS\\), for all \\(s\\in\\R^p\\) it holds that\n\t\\[\n\t\t\\inf\\varphi\n\t{}\\leq{}\n\t\t\\sDRS(s)+\\nsDRS(s)\n\t{}\\overrel*[\\leq]{\\eqref{eq:LipBound}}{}\n\t\t\\sDRS(0)+\\innprod{\\nabla\\sDRS(0)}{s}+\\tfrac{L_{\\sDRS}}{2}\\|s\\|^2\n\t\t{}+{}\n\t\t\\nsDRS(s),\n\t\\]\n\thence, for all \\(\\gamma<\\nicefrac{1}{L_{\\sDRS}}\\) the function \\(s\\mapsto\\nsDRS(s)+\\tfrac{1}{2\\gamma}\\|s\\|^2\\) is lower bounded.\n","sha256_utf8":"1cfcb27102915c4abcadda6b1c311e8019f78063c4f1bc8b692fd7c7c1cf4161"},{"start":52744,"end":53232,"start_line":977,"end_line":981,"raw_tex":"We start by observing that \\(\\DRE\\) is a strictly continuous function for \\(\\gamma<\\nicefrac{1}{L_{\\sDRS}}\\), owing to the fact that so is the FBE \\cite[Prop. 4.2]{themelis2018forward}, and that \\(\\prox_{\\gamma\\sDRS}\\) is Lipschitz continuous as shown in \\cref{thm:proxfLip}.\n\\begin{prop}[Strict continuity]\\label{thm:C0}%\n\tSuppose that \\Cref{ass:DRS} is satisfied.\n\tFor all \\(\\gamma<\\nicefrac{1}{L_{\\sDRS}}\\) the DRE \\(\\DRE\\) is a real-valued and strictly continuous function.\n\\end{prop}","sha256_utf8":"3cfb633a77f9264ea2bc67feb8fd81a88037c3d706b08d59007f737cb7bf5306"},{"start":53615,"end":54670,"start_line":985,"end_line":1020,"raw_tex":"\\begin{prop}[Sandwiching property]\\label{thm:sandwich}%\n\tSuppose that \\Cref{ass:DRS} is satisfied.\n\tLet \\(\\gamma<\\nicefrac{1}{L_{\\sDRS}}\\) be fixed, and consider \\(u,v\\) generated by one \\ref{DRS} iteration starting from \\(s\\in\\R^p\\).\n\tThen,\n\t\\begin{enumerate}\n\t\\item\\label{thm:DREleq}\n\t\t\\(\\DRE(s)\\leq\\varphi(u)\\).\n\t\\item\\label{thm:DREgeq}\n\t\t\\(\n\t\t\t\\varphi(v)\n\t\t{}\\leq{}\n\t\t\t\\DRE(s)\n\t\t\t{}-{}\n\t\t\t\\tfrac{1-\\gamma L_{\\sDRS}}{2\\gamma}\\|u-v\\|^2\n\t\t\\).\n% \t\t\\(\n% \t\t\t\\varphi(v)+\\tfrac{1-\\gamma L_{\\sDRS}}{2\\gamma}\\|u-v\\|^2\n% \t\t{}\\leq{}\n% \t\t\t\\DRE(s)\n% \t\t{}\\leq{}\n% \t\t\t\\varphi(v)+\\tfrac{1-\\gamma\\sigma_{\\sDRS}}{2\\gamma}\\|u-v\\|^2\n% \t\t\\).\n\t\\end{enumerate}\n\t\\begin{proof}\n\t\t\\ref{thm:DREleq} is easily inferred from definition \\eqref{eq:DRE} by considering \\(w=u\\).\n\t\tMoreover, it follows from \\cite[Prop. 4.3]{themelis2018forward} and the fact that \\(v\\in\\FB u\\), cf. \\eqref{eq:vFBS}, that\n\t\t\\(\n\t\t\t\\varphi(v)\n\t\t{}\\leq{}\n\t\t\t\\FBE(u)\n\t\t\t{}-{}\n\t\t\t\\tfrac{1-\\gamma L_{\\sDRS}}{2\\gamma}\\|u-v\\|^2\n\t\t\\).\n\t\t\\ref{thm:DREgeq} then follows from \\eqref{eq:FBEu}.\n\t\\end{proof}\n\\end{prop}","sha256_utf8":"b32d8a09f053f4f111121bd8dea8f5d1aab1eb6eebf2e19e2b8129d5ee011b90"},{"start":54677,"end":57910,"start_line":1023,"end_line":1141,"raw_tex":"\\begin{thm}[Minimization and level-boundedness equivalence]\\label{thm:DREequiv}%\n\tSuppose that \\Cref{ass:DRS} is satisfied.\n\tFor any \\(\\gamma<\\nicefrac{1}{L_{\\sDRS}}\\) the following hold:\n\t\\begin{enumerate}\n\t\\item\\label{thm:inf}%\n\t\t\\(\\inf\\varphi=\\inf\\DRE\\).\n\t\\item\\label{thm:argmin}%\n\t\t\\(\\argmin\\varphi=\\prox_{\\gamma\\sDRS}\\bigl(\\argmin\\DRE\\bigr)\\).\n\t\\item\\label{thm:boundedlev}%\n\t\t\\(\\varphi\\) is level bounded iff so is \\(\\DRE\\).\n\t\\end{enumerate}\n\t\\begin{proof}\n\t\tIt follows from \\cite[Thm. 4.4]{themelis2018forward} that the FBE satisfies \\(\\inf\\varphi=\\inf\\FBE\\) and \\(\\argmin\\varphi=\\argmin\\FBE\\).\n\t\tThe similar properties \\ref{thm:inf} and \\ref{thm:argmin} of the DRE then follow from the identity \\(\\DRE=\\FBE\\circ\\prox_{\\gamma\\sDRS}\\), cf. \\eqref{eq:FBEu}, and the fact that \\(\\prox_{\\gamma\\sDRS}\\) is invertible, as shown in \\cref{thm:proxf}.\n\t\t\n\t\tWe now show \\ref{thm:boundedlev}.\n\t\tDenote\n\t\t\\(\n\t\t\t\\varphi_\\star\n\t\t{}\\coloneqq{}\n\t\t\t\\inf\\varphi\n\t\t{}={}\n\t\t\t\\inf\\DRE\n\t\t\\),\n\t\twhich is finite by assumption.\n\t\t\\begin{proofitemize}\n\t\t\\item\n\t\t\tSuppose that \\(\\DRE\\) is level bounded, and let \\(u\\in\\lev_{\\leq\\alpha}\\varphi\\) for some \\(\\alpha>\\varphi_\\star\\).\n\t\t\tThen, \\(s\\coloneqq u+\\gamma\\nabla\\sDRS(u)\\) is such that \\(\\prox_{\\gamma\\sDRS}(s)=u\\), as shown in \\cref{thm:proxfEquiv}.\n\t\t\tThus, from \\cref{thm:sandwich} it follows that \\(s\\in\\lev_{\\leq\\alpha}\\DRE\\).\n\t\t\tIn particular,\n\t\t\t\\(\n\t\t\t\t\\lev_{\\leq\\alpha}\\varphi\n\t\t\t{}\\subseteq{}\n\t\t\t\t[I+\\gamma\\nabla\\sDRS]\n\t\t\t\t(\\lev_{\\leq\\alpha}\\DRE),\n\t\t\t\\)\n\t\t\tand since \\(\\I+\\gamma\\nabla\\sDRS\\) is Lipschitz continuous and \\(\\lev_{\\leq\\alpha}\\DRE\\) is bounded by assumption, it follows that \\(\\lev_{\\leq\\alpha}\\varphi\\) is also bounded.\n\t\t\\item\n\t\t\tSuppose now that \\(\\DRE\\) is not level bounded.\n\t\t\tThen, there exists \\(\\alpha>\\varphi_\\star\\) together with a sequence \\(\\seq{s_k}\\) satisfying\n\t\t\t\\(\n\t\t\t\ts_k\n\t\t\t{}\\in{}\n\t\t\t\t\\lev_{\\leq\\alpha}\\DRE\n\t\t\t\t{}\\setminus{}\n\t\t\t\t\\ball 0k\n\t\t\t\\)\n\t\t\tfor all \\(k\\in\\N\\).\n\t\t\tLet \\(u_k\\coloneqq\\prox_{\\gamma\\sDRS}(s_k)\\), so that \\(s_k=u_k+\\gamma\\nabla\\sDRS(u_k)\\) (cf. \\cref{thm:proxfEquiv}), and let \\(v_k\\in\\FB{u_k}\\).\n\t\t\tFrom \\cref{thm:DREgeq} it then follows that \\(v_k\\in\\lev_{\\leq\\alpha}\\varphi\\), and that\n\t\t\t\\[\n\t\t\t\t\\alpha\n\t\t\t\t{}-{}\n\t\t\t\t\\varphi_\\star\n\t\t\t{}\\geq{}\n\t\t\t\t\\DRE(s_k)\n\t\t\t\t{}-{}\n\t\t\t\t\\varphi_\\star\n\t\t\t{}\\geq{}\n\t\t\t\t\\DRE(s_k)\n\t\t\t\t{}-{}\n\t\t\t\t\\varphi(v_k)\n\t\t\t{}\\geq{}\n\t\t\t\t\\tfrac{1-\\gamma L_{\\sDRS}}{2\\gamma}\n\t\t\t\t\\|u_k-v_k\\|^2.\n\t\t\t\\]\n\t\t\tTherefore,\n\t\t\t\\(\n\t\t\t\t\\|u_k-v_k\\|^2\n\t\t\t{}\\leq{}\n\t\t\t\t\\tfrac{\n\t\t\t\t\t2\\gamma(\\alpha-\\varphi_\\star)\n\t\t\t\t}{\n\t\t\t\t\t1-\\gamma L_{\\sDRS}\n\t\t\t\t}\n\t\t\t\\)\n\t\t\tand\n\t\t\t\\begin{align*}\n\t\t\t\t\\|v_k\\|\n\t\t\t{}\\geq{} &\n\t\t\t\t\\|u_k-u_0\\|\n\t\t\t\t{}-{}\n\t\t\t\t\\|u_0\\|\n\t\t\t\t{}-{}\n\t\t\t\t\\|u_k-v_k\\|\n\t\t\t\\smash{{}\\overrel[\\geq]{\\ref{thm:proxfLip}}{}}\n\t\t\t\t\\tfrac{1}{1+\\gamma L_{\\sDRS}}\n\t\t\t\t\\|s_k-s_0\\|\n\t\t\t\t{}-{}\n\t\t\t\t\\|u_0\\|\n\t\t\t\t{}-{}\n\t\t\t\t\\|u_k-v_k\\|\n\t\t\t\\\\\n\t\t\t{}\\geq{} &\n\t\t\t\t\\tfrac{\n\t\t\t\t\tk-\\|s_0\\|\n\t\t\t\t}{\n\t\t\t\t\t1+\\gamma L_{\\sDRS}\n\t\t\t\t}\n\t\t\t\t{}-{}\n\t\t\t\t\\|u_0\\|\n\t\t\t\t{}-{}\n\t\t\t\t\\sqrt{\n\t\t\t\t\t\\tfrac{\n\t\t\t\t\t\t2\\gamma(\\alpha-\\varphi_\\star)\n\t\t\t\t\t}{\n\t\t\t\t\t\t1-\\gamma L_{\\sDRS}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t{}\\to{}\n\t\t\t\t+\\infty\n\t\t\t\\quad\\text{as \\(k\\to\\infty\\).}\n\t\t\t\\end{align*}\n\t\t\tThis shows that \\(\\lev_{\\leq\\alpha}\\varphi\\) is also unbounded.\n\t\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}\n\\end{thm}","sha256_utf8":"6d9dba970caf57b123383b6242e1cc7f6df431aeffe3f8e330d6279490c6036d"},{"start":71396,"end":89754,"start_line":1445,"end_line":2247,"raw_tex":"\\begin{thm}[Sufficient decrease on the DRE]\\label{thm:DRE:SD}%\n\tSuppose that \\Cref{ass:DRS} is satisfied, and consider one \\ref{DRS} update\n\t\\(\n\t\ts\n\t{}\\mapsto{}\n\t\t(u,v,s^+)\n\t\\)\n\tfor some stepsize\n\t\\(\n\t\t\\gamma\n\t{}<{}\n\t\t\\min\\set{\n\t\t\t\\tfrac{2-\\lambda}{2[\\sigma_{\\sDRS}]_-},\\,\n\t\t\t\\tfrac{1}{L_{\\sDRS}}\n\t\t}\n\t\\)\n\tand relaxation \\(\\lambda\\in(0,2)\\).\n\tThen,\n\t\\begin{equation}\\label{eq:DRE:SD}\n\t\t\\DRE(s)-\\DRE(s^+)\n\t{}\\geq{}\n\t\t\\tfrac{c}{(1+\\gamma L_{\\sDRS})^2}\\,\n\t\t\\|s-s^+\\|^2,\n\t\\end{equation}\n\twhere, denoting \\(p_{\\sDRS}\\coloneqq\\nicefrac{\\sigma_{\\sDRS}}{L_{\\sDRS}}\\in[-1,1]\\), \\(c\\) is a strictly positive constant defined as%\n\t\\footnote{%\n\t\tA one-line expression for the constant is\n\t\t\\(\n\t\t\tc\n\t\t{}={}\n\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n\t\t\t{}-{}\n\t\t\t\\min\\set{\n\t\t\t\t\\tfrac{[p_{\\sDRS}]_-}{\\lambda},~\n\t\t\t\tL_{\\sDRS}\\max\\set{\n\t\t\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{2(1-[p_{\\sDRS}]_-)},\\,\n\t\t\t\t\t\\tfrac12-\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\n\t\t\t\t}\n\t\t\t}\n\t\t\\).\n\t}\n\t\\begin{equation}\\label{eq:DRE:c0}\n\t\tc\n\t{}={}\n\t\t\\frac{2-\\lambda}{2\\lambda\\gamma}\n\t\t{}-{}\n\t\t\\begin{ifcases}\n\t\t\tL_{\\sDRS}\\max\\set{\n\t\t\t\t\\tfrac{[p_{\\sDRS}]_-}{2(1-[p_{\\sDRS}]_-)},\\,\n\t\t\t\t\\tfrac12-\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\n\t\t\t}\n\t\t&\n\t\t\t\tp_{\\sDRS}\n\t\t\t{}\\geq{}\n\t\t\t\t\\tfrac\\lambda2-1\n% \t\t\t0<\\lambda<2(1-[p_{\\sDRS}]_-),\n\t\t\\\\\n\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{\\lambda}\n\t\t\\otherwise[otherwise.]\n\t\t\\end{ifcases}\n\t\\end{equation}\n\tIf \\(\\sDRS\\) is strongly convex, then \\eqref{eq:DRE:SD} also holds for\n\t\\begin{equation}\\label{eq:lambda>=2}\n\t\t2\n\t{}\\leq{}\n\t\t\\lambda\n\t{}<{}\n\t\t\\tfrac{4}{1+\\sqrt{1-p_{\\sDRS}}}\n\t\\quad\\text{and}\\quad\n\t\t\\tfrac{\n\t\t\tp_{\\sDRS}\\lambda\n\t\t\t{}-{}\n\t\t\t\\delta\n\t\t}{\n\t\t\t4\\sigma_{\\sDRS}\n\t\t}\n\t{}<{}\n\t\t\\gamma\n\t{}<{}\n\t\t\\tfrac{\n\t\t\tp_{\\sDRS}\\lambda\n\t\t\t{}+{}\n\t\t\t\\delta\n\t\t}{\n\t\t\t4\\sigma_{\\sDRS}\n\t\t},\n\t\\end{equation}\n\twhere\n\t\\(\n\t\t\\delta\n\t{}\\coloneqq{}\n\t\t\\sqrt{(p_{\\sDRS}\\lambda)^2-8p_{\\sDRS}(\\lambda-2)}\n\t\\),\n\tin which case\n\t\\begin{equation}\\label{eq:DRE:c1}\n\t\tc\n\t{}={}\n\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n\t\t{}+{}\n\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n\t\t(\\tfrac12-\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}).\n\t\\end{equation}\n\t\\begin{proof}\n\t\t\\newcommand\\x\\xi%\n\t\tLet \\((u^+,v^+)\\) be generated by one \\ref{DRS} iteration starting at \\(s^+\\).\n\t\tThen,\n\t\t\\begin{align*}\n\t\t\t\\DRE(s^+)\n\t\t{}={} &\n\t\t\t\\min_{w\\in\\R^n}\\set{\n\t\t\t\t\\sDRS(u^+)\n\t\t\t\t{}+{}\n\t\t\t\t\\nsDRS(w)\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)}{w-u^+}\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|w-u^+\\|^2\n\t\t\t}\n\t\t\\end{align*}\n\t\tand the minimum is attained at \\(w=v^+\\).\n\t\tTherefore, letting \\(\\rho\\) be as in \\cref{thm:rho},\n\t\t{\\mathtight[0.9]%\n\t\t\t\\begin{align*}\n\t\t\t\t\\DRE(s^+)\n\t\t\t{}\\leq{} &\n\t\t\t\t\\sDRS(u^+)\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)}{v-u^+}\n\t\t\t\t{}+{}\n\t\t\t\t\\nsDRS(v)\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|u^+-v\\|^2\n\t\t\t\\\\\n\t\t\t{}={} &\n\t\t\t\t\\smashunderbracket[0.5pt]{\n\t\t\t\t\t\\sDRS(u^+)\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)}{u-u^+}\n\t\t\t\t}{}\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)}{v-u}\n\t\t\t\t{}+{}\n\t\t\t\t\\nsDRS(v)\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|u^+-v\\|^2\n\t\t\t\\\\\n\t\t\t% \t\t\t\\dueto{\\cref{thm:rho}}\n\t\t\t{}\\overrel[\\leq]{\\llap{\\cref{thm:rho}}}{} &\n\t\t\t\t\\smashoverbracket[0.5pt]{\n\t\t\t\t\t\\sDRS(u)\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\rho(u,u^+)\n\t\t\t\t}{}\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)}{v-u}\n\t\t\t\t{}+{}\n\t\t\t\t\\nsDRS(v)\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|u^+-v\\|^2\n\t\t\t\\\\\n\t\t\t{}={} &\n\t\t\t\t\\sDRS(u)\n\t\t\t\t{}-{}\n\t\t\t\t\\rho(u,u^+)\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u)}{v-u}\n% \t\t\t\t\\innprod{\\nabla\\sDRS(\\fillwidthof[c]{u^+}{u})}{v-u}\n\t\t\t\t{}+{}\n\t\t\t\t\\nsDRS(v)\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|u^+-v\\|^2\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t{+{}}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)-\\nabla\\sDRS(u)}{v-u}\n\t\t\t\\\\\n\t\t\t{}={} &\n\t\t\t\t\\DRE(s)\n\t\t\t\t{}-{}\n\t\t\t\t\\rho(u,u^+)\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)-\\nabla\\sDRS(u)}{v-u}\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|u-u^+\\|^2\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac1\\gamma\n\t\t\t\t\\innprod{u^+-u}{u-v}.\n\t\t\t\\end{align*}%\n\t\t}%\n\t\tSince\n\t\t\\(\n\t\t\tu-v\n\t\t{}={}\n\t\t\t\\tfrac1\\lambda(s-s^+)\n\t\t{}={}\n\t\t\t\\tfrac1\\lambda(u-u^+)\n\t\t\t{}+{}\n\t\t\t\\tfrac\\gamma\\lambda(\\nabla\\sDRS(u)-\\nabla\\sDRS(u^+))\n\t\t\\),\n\t\tas it follows from \\cref{thm:proxfEquiv}, it all simplifies to\n\t\t\\begin{equation}\\label{eq:DRE:SD0}\n\t\t\t\\DRE(s)\n\t\t\t{}-{}\n\t\t\t\\DRE(s^+)\n\t\t{}\\geq{}\n\t\t\t\\tfrac{2-\\lambda}{2\\gamma\\lambda}\n\t\t\t\\|u-u^+\\|^2\n\t\t\t{}-{}\n\t\t\t\\tfrac\\gamma\\lambda\n\t\t\t\\|\\nabla\\sDRS(u^+)-\\nabla\\sDRS(u)\\|^2\n\t\t\t{}+{}\n\t\t\t\\rho(u,u^+).\n\t\t\\end{equation}\n\t\tIt will suffice to show that\n\t\t\\[\n\t\t\t\\DRE(s)-\\DRE(s^+)\n\t\t{}\\geq{}\n\t\t\tc\n\t\t\t\\|u-u^+\\|^2;\n\t\t\\]\n\t\tinequality \\eqref{eq:DRE:SD} will then follow from the \\(\\tfrac{1}{1+\\gamma L_{\\sDRS}}\\)-strong monotonicity of \\(\\prox_{\\gamma\\sDRS}\\), see \\cref{thm:proxfLip}.\n% \t\t\\[\n% \t\t\t\\|u-u^+\\|^2\n% \t\t{}\\geq{}\n% \t\t\t\\tfrac{1}{(1+\\gamma L_{\\sDRS})^2}\n% \t\t\t\\|s-s^+\\|^2\n% \t\t\\]\n\t\tWe now proceed by cases.\n\t\t\\begin{proofitemize}\n\t\t\\item\\textbf{Case 1: \\(\\lambda\\in(0,2)\\).}\n\t\t\t\n\t\t\t\\noindent\n\t\t\tLet \\(\\sigma\\coloneqq-[\\sigma_{\\sDRS}]_-=\\min\\set{\\sigma_{\\sDRS},0}\\) and \\(L\\geq L_{\\sDRS}\\) be such that \\(L+\\sigma>0\\); the value of such an \\(L\\) will be fixed later.\n\t\t\tThen, \\(\\sigma\\leq0\\) and \\(\\sDRS\\) is \\(L\\)-smooth and \\(\\sigma\\)-hypoconvex.\n\t\t\tWe may thus choose \\(\\rho(u,u^+)\\) as in \\cref{thm:hypo:rho1} with these values of \\(L\\) and \\(\\sigma\\).\n\t\t\tInequality \\eqref{eq:DRE:SD0} then becomes\n\t\t\t\\[\n\t\t\t\t\\tfrac{\\DRE(s)-\\DRE(s^+)}{L}\n\t\t\t{}\\geq{}\n\t\t\t\t\\left(\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\x}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{p}{2(1+p)}\n\t\t\t\t\\right)\n\t\t\t\t\\|u^+-u\\|^2\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{L^2}\n\t\t\t\t\\left(\n\t\t\t\t\t\\tfrac{1}{2(1+p)}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{\\x}{\\lambda}\n\t\t\t\t\\right)\n\t\t\t\t\\|\\nabla\\sDRS(u^+)-\\nabla\\sDRS(u)\\|^2,\n\t\t\t\\]\n\t\t\twhere \\(\\x\\coloneqq\\gamma L\\) and \\(p\\coloneqq\\nicefrac\\sigma L\\in(-1,0]\\).\n\t\t\tSince \\(\\nabla\\sDRS\\) is \\(L_{\\sDRS}\\)-Lipschitz continuous, the claim holds provided that the constant\n\t\t\t\\begin{equation}\\label{eq:c}\n\t\t\t\t\\frac cL\n\t\t\t{}={}\n\t\t\t\t\\begin{ifcases}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\x}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{p}{2(1+p)}\n\t\t\t\t&\n\t\t\t\t\t0\n\t\t\t\t\t{}<{}\n\t\t\t\t\t\\tfrac{1}{2(1+p)}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{\\x}{\\lambda}\n\t\t\t\t\\\\[3pt]\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\x}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{p}{2(1+p)}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{L_{\\sDRS}^2}{L^2}\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t\\tfrac{1}{2(1+p)}\n\t\t\t\t\t\t{}-{}\n\t\t\t\t\t\t\\tfrac{\\x}{\\lambda}\n\t\t\t\t\t\\right)\n\t\t\t\t\\otherwise[otherwise,]\n\t\t\t\t\\end{ifcases}\n\t\t\t\\end{equation}\n\t\t\tis strictly positive.\n% \t\t\t\n\t\t\tNow, let us consider two subcases:\n\t\t\t\\begin{itemize}\n\t\t\t\\item\\textbf{Case 1a:} \\(0<\\lambda\\leq 2(1+\\nicefrac{\\sigma}{L_{\\sDRS}})\\).\n\t\t\t\n\t\t\t\tThen,\n\t\t\t\t\\(\n\t\t\t\t\t\\sigma\\geq -\\frac{2-\\lambda}{2}L_{\\sDRS}>-L_{\\sDRS}\n\t\t\t\t\\)\n\t\t\t\tand we can take \\(L=L_{\\sDRS}\\).\n\t\t\t\tConsequently, \\(p=\\nicefrac{\\sigma}{L_{\\sDRS}}\\), \\(\\x=\\gamma L_{\\sDRS}\\), and \\eqref{eq:c} becomes\n\t\t\t\t\\begin{equation}\\label{eq:cbar}\n\t\t\t\t\t\\frac{c}{L_{\\sDRS}}\n\t\t\t\t{}={}\n\t\t\t\t\t\\frac{2-\\lambda}{2\\lambda\\gamma L_{\\sDRS}}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\begin{ifcases}\n\t\t\t\t\t\t\\tfrac{p}{2(1+p)}\n\t\t\t\t\t&\n\t\t\t\t\t\t\\gamma\n\t\t\t\t\t\t{}<{}\n\t\t\t\t\t\t\\tfrac{\\lambda}{2(1+p)}\n\t\t\t\t\t\\\\[3pt]\n\t\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t\t{}-{}\n\t\t\t\t\t\t\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\n\t\t\t\t\t\\otherwise.\n\t\t\t\t\t\\end{ifcases}\n\t\t\t\t\\end{equation}\n\t\t\t\tLet us verify that in this case any \\(\\gamma\\) such that \\(\\gamma<\\nicefrac{1}{L_{\\sDRS}}\\) yields a strictly positive coefficient \\(c\\).\n\t\t\t\tIf \\(0<\\gamma L_{\\sDRS}<\\frac{\\lambda}{2(1+p)}\\leq 1\\), then\n\t\t\t\t\\[\n\t\t\t\t\t\\tfrac{c}{L_{\\sDRS}}\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma L_{\\sDRS}}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{p}{2(1+p)}\n\t\t\t\t{}>{}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{p}{\\lambda}\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac{1+p}{\\lambda}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t{}\\geq{}\n\t\t\t\t\t0,\n\t\t\t\t\\]\n\t\t\t\twhere in the inequality we used the fact that \\(\\lambda<2\\) and \\(p\\leq0\\).\n\t\t\t\tIf instead \\(\\frac{\\lambda}{2(1+p)}<\\gamma L_{\\sDRS}<1\\), then\n\t\t\t\t\\[\n\t\t\t\t\t\\tfrac{c}{L_{\\sDRS}}\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma L_{\\sDRS}}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\n\t\t\t\t{}>{}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{1}{\\lambda}\n\t\t\t\t{}={}\n\t\t\t\t\t0.\n\t\t\t\t\\]\n\t\t\t\tEither way, the sufficient decrease constant \\(c\\) is strictly positive.\n% \t\t\t\tBy imposing \\(c>0\\) we obtain the following conditions on \\(\\gamma\\) and \\(\\lambda\\):\n% \t\t\t\t\\begin{align*}\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\tfrac{\\sigma}{2(1+p)}\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{\\lambda}{2(L_{\\sDRS}+\\sigma)}\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\quad\\cup\\quad &\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\tfrac{L_{\\sDRS}}{2}\n% \t\t\t\t\t\t{}-{}\n% \t\t\t\t\t\t\\tfrac{\\gamma L_{\\sDRS}^2}{\\lambda}\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}\\geq{} &\n% \t\t\t\t\t\t\\tfrac{\\lambda}{2(L_{\\sDRS}+\\sigma)}.\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\shortintertext{%\n% \t\t\t\t\tEquivalently,\n% \t\t\t\t}\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t(1+p)(2-\\lambda)\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\lambda p\\xi\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\xi\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{\\lambda}{2(1+p)}\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\quad\\cup\\quad &\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}>{} &\n% \t\t\t\t\t\t2\\xi^2\n% \t\t\t\t\t\t{}-{}\n% \t\t\t\t\t\t\\lambda\\xi\n% \t\t\t\t\t\t{}-{}\n% \t\t\t\t\t\t(2-\\lambda)\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\xi\n% \t\t\t\t\t{}\\geq{} &\n% \t\t\t\t\t\t\\tfrac{\\lambda}{2(1+p)}.\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\shortintertext{%\n% \t\t\t\t\tSince \\(\\xi<1\\), notice that\n% \t\t\t\t\t\\(\n% \t\t\t\t\t\t(1+p)(2-\\lambda)\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\lambda p\\xi\n% \t\t\t\t\t{}>{}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t\\)\n% \t\t\t\t\tis always satisfied when \\(2-\\lambda\\geq-2p\\), as in the case being considered.\n% \t\t\t\t\tThis is evident if \\(p=0\\) (since \\(\\lambda<2\\)); if instead \\(p<0\\), one has\n% \t\t\t\t\t\\(\n% \t\t\t\t\t\t(1+p)(2-\\lambda)\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\lambda p\\xi\n% \t\t\t\t\t{}\\geq{}\n% \t\t\t\t\t\t-2p(1+p)+p\\lambda\\xi\n% \t\t\t\t\t{}>{}\n% \t\t\t\t\t\t-p(2(1+p)-\\lambda)\n% \t\t\t\t\t{}\\geq{}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t\\).\n% \t\t\t\t\tAll conditions then reduce to\n% \t\t\t\t}\n% \t\t\t\t\t\\xi\n% \t\t\t\t{}<{}\n% \t\t\t\t\t\\tfrac{\\lambda}{2(1+p)}\n% \t\t\t\t\\quad\\cup\\quad &\n% \t\t\t\t\t\\tfrac{\\lambda}{2(1+p)}\n% \t\t\t\t{}\\leq{}\n% \t\t\t\t\t\\xi\n% \t\t\t\t{}<{}\n% \t\t\t\t\t1,\n% \t\t\t\t\\end{align*}\n% \t\t\t\tthat is, \\(\\gamma L_{\\sDRS}=\\xi<1\\).\n\t\t\t\tSince \\(\\sigma=-[\\sigma_{\\sDRS}]_-\\) and\n\t\t\t\t\\[\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{\\sigma}{2(1+p)}\n\t\t\t\t{}\\leq{}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{L_{\\sDRS}}{2}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{\\gamma L_{\\sDRS}^2}{\\lambda}\n\t\t\t\t\\quad\\Leftrightarrow\\quad\n\t\t\t\t\t\\gamma\n\t\t\t\t{}\\leq{}\n\t\t\t\t\t\\tfrac{\\lambda}{2(L_{\\sDRS}+\\sigma)},\n\t\t\t\t\\]\n\t\t\t\tfrom \\eqref{eq:cbar} we conclude that \\(c\\) is as in \\eqref{eq:DRE:SD}.\n\t\t\t%%%%\n\t\t\t%%%%\n\t\t\t%%%%\n\t\t\t\\item\\textbf{Case 1b:} \\(2(1+\\nicefrac{\\sigma}{L_{\\sDRS}})<\\lambda<2\\).\n\t\t\t\t\n\t\t\t\tNecessarily \\(\\sigma<0\\), for otherwise the range of \\(\\lambda\\) would be empty.\n\t\t\t\tIn particular, \\(\\sigma=\\sigma_{\\sDRS}\\), and the lower bound on \\(\\lambda\\) can be expressed as\n\t\t\t\t\\(\n\t\t\t\t\t\\sigma_{\\sDRS}<-\\frac{2-\\lambda}{2}L_{\\sDRS}\n\t\t\t\t\\).\n\t\t\t\tConsequently, \\(L\\coloneqq\\tfrac{-2\\sigma_{\\sDRS}}{2-\\lambda}\\) is strictly larger than \\(L_{\\sDRS}\\), and in particular \\(\\sigma+L=\\sigma_{\\sDRS}+L>0\\).\n\t\t\t\tThe ratio of \\(\\sigma\\) and \\(L\\) is thus \\(p=\\tfrac\\lambda2-1\\), and \\eqref{eq:c} becomes\n\t\t\t\t\\begin{equation}\\label{eq:c2}\n\t\t\t\t\tc\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\begin{ifcases}\n\t\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n\t\t\t\t\t&\n\t\t\t\t\t\t\\gamma\n\t\t\t\t\t\t{}<{}\n\t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n\t\t\t\t\t\\\\[3pt]\n\t\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}-\\tfrac{\\gamma L_{\\sDRS}^2}{\\lambda}+\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}\\lambda}L_{\\sDRS}^2\n\t\t\t\t\t\\otherwise.\n\t\t\t\t\t\\end{ifcases}\n\t\t\t\t\\end{equation}\n\t\t\t\tLet us show that, when\n\t\t\t\t\\(\n\t\t\t\t\t\\gamma\n\t\t\t\t{}<{}\n\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac1L\n\t\t\t\t\\),\n\t\t\t\talso in this case the sufficient decrease constant \\(c\\) is strictly positive.\n\t\t\t\tWe have\n\t\t\t\t\\[\n\t\t\t\t\t\\tfrac cL\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma L}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n\t\t\t\t\t\\tfrac1L\n\t\t\t\t{}>{}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n\t\t\t\t{}={}\n\t\t\t\t\t0,\n\t\t\t\t\\]\n\t\t\t\thence the claim.\n\t\t\t\tThis concludes the proof for the case \\(\\lambda\\in(0,2)\\).\n% \t\t\t\tBy imposing \\(c>0\\) this reduces to\n% \t\t\t\t\\begin{align*}\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\quad\\cup\\quad &\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n% \t\t\t\t\t\t{}-{}\n% \t\t\t\t\t\t\\tfrac{\\gamma L_{\\sDRS}^2}{\\lambda}\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}\\lambda}L_{\\sDRS}^2\n% \t\t\t\t\t{}={}\n% \t\t\t\t\t\t\\smashoverbracket{\n% \t\t\t\t\t\t\t\\tfrac1\\lambda\n% \t\t\t\t\t\t\t\\bigl(\n% \t\t\t\t\t\t\t\t\\gamma L_{\\sDRS}^2-\\sigma_{\\sDRS}\n% \t\t\t\t\t\t\t\\bigr)\n% \t\t\t\t\t\t}{>0}\n% \t\t\t\t\t\t\\bigl(\n% \t\t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\gamma\\sigma_{\\sDRS}}\n% \t\t\t\t\t\t\t{}-{}\n% \t\t\t\t\t\t\t1\n% \t\t\t\t\t\t\\bigr)\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}\\geq{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}.\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\shortintertext{Equivalently,}\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t\t{}<{}\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t\t{}<{}\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\quad\\cup\\quad &\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}<{}\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}\\geq{}\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}},\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\end{align*}\n% \t\t\t\tthat is,\n% \t\t\t\t\\(\n% \t\t\t\t\t\\gamma\n% \t\t\t\t{}<{}\n% \t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t\\).\n% \t\t\t\tFrom \\eqref{eq:c2} it follows that\n% \t\t\t\t\\(\n% \t\t\t\t\tc\n% \t\t\t\t{}={}\n% \t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t{}+{}\n% \t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n% \t\t\t\t\\).%\n\t\t\t\\end{itemize}\n% \t\t\tLet us recap {\\bf Case 1}:\n% \t\t\t\\[\n% \t\t\t\t\\begin{cases}[@{\\gamma<{}} l<{,} @{~~c={}} l @{\\quad\\text{if~~}} l]\n% \t\t\t\t\t\\tfrac{1}{L_{\\sDRS}}\n% \t\t\t\t&\n% \t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t{}-{}\n% \t\t\t\t\t\\max\\set{\n% \t\t\t\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{2(1-[p_{\\sDRS}]_-)},\\,\n% \t\t\t\t\t\tL_{\\sDRS}\\bigl(\\tfrac12-\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\\bigr)\n% \t\t\t\t\t}\n% \t\t\t\t&\n% \t\t\t\t\t0<\\lambda<2(1-[p_{\\sDRS}]_-),\n% \t\t\t\t\\\\\n% \t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t&\n% \t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t{}-{}\n% \t\t\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{\\lambda}\n% \t\t\t\t&\n% \t\t\t\t\t2(1-[p_{\\sDRS}]_-)<\\lambda<2.\n% \t\t\t\t\\end{cases}\n% \t\t\t\\]\n% \t\t\tWe may unify the two cases with the more compact notation\n% \t\t\t\\[\n% \t\t\t\t\\gamma\n% \t\t\t{}<{}\n% \t\t\t\t\\min\\set{\n% \t\t\t\t\t\\tfrac{1}{L_{\\sDRS}},\\,\n% \t\t\t\t\t\\tfrac{2-\\lambda}{-2[\\sigma_{\\sDRS}]_-}\n% \t\t\t\t}\n% \t\t\t\\quad\\text{and}\\quad\n% \t\t\t\tc\n% \t\t\t{}={}\n% \t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t{}-{}\n% \t\t\t\t\\min\\set{\n% \t\t\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{\\lambda},~\n% \t\t\t\t\t\\max\\set{\n% \t\t\t\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{2(1-[p_{\\sDRS}]_-)},\\,\n% \t\t\t\t\t\tL_{\\sDRS}\\bigl(\\tfrac12-\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\\bigr)\n% \t\t\t\t\t}\n% \t\t\t\t},\n% \t\t\t\\]\n% \t\t\tthus obtaining the range of \\(\\gamma\\) and the value of \\(c\\) indicated in \\eqref{eq:c_nonstr}.\n\t\t\t%%%\n\t\t\t%%%\n\t\t\t%%%\n\t\t\\item\\textbf{Case 2: \\(\\lambda\\geq2\\).}\n\n\t\t\t\\noindent\n\t\t\tIn this case we need to assume that \\(\\sDRS\\) is strongly convex, that is, that \\(\\sigma_{\\sDRS}>0\\).\n\t\t\tInstead of considering a single expression of \\(\\rho\\), we will rather take a convex combination of those in \\cref{thm:hypo:rho0,thm:hypo:rho1}, namely\n\t\t\t\\[\n\t\t\t\t\\rho(u,u^+)\n\t\t\t{}={}\n\t\t\t\t(1-\\alpha)\\tfrac{\\sigma_{\\sDRS}}{2}\\|u-u^+\\|^2\n\t\t\t\t{}+{}\n\t\t\t\t\\alpha\\tfrac{1}{2L_{\\sDRS}}\\|\\nabla\\sDRS(u)-\\nabla\\sDRS(u^+)\\|^2\n\t\t\t\\]\n\t\t\tfor some \\(\\alpha\\in[0,1]\\) to be determined.\n\t\t\t\\eqref{eq:DRE:SD0} then becomes\n\t\t\t\\[\n\t\t\t\t\\tfrac{\n\t\t\t\t\t\\DRE(s)-\\DRE(s^+)\n\t\t\t\t}{\n\t\t\t\t\tL_{\\sDRS}\n\t\t\t\t}\n\t\t\t{}\\geq{}\n\t\t\t\t\\left(\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\xi}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{(1-\\alpha)p}{2}\n\t\t\t\t\\right)\n\t\t\t\t\\|u-u^+\\|^2\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{L_{\\sDRS}^2}\n\t\t\t\t\\left(\n\t\t\t\t\t\\tfrac{\\alpha}{2}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{\\xi}{\\lambda}\n\t\t\t\t\\right)\n\t\t\t\t\\|\\nabla\\sDRS(u)-\\nabla\\sDRS(u^+)\\|^2,\n\t\t\t\\]\n\t\t\twhere \\(\\x\\coloneqq\\gamma L_{\\sDRS}\\) and \\(p\\coloneqq\\nicefrac{\\sigma_{\\sDRS}}{L_{\\sDRS}}\\in(0,1]\\).\n\t\t\tBy restricting \\(\\xi\\in(0,1)\\), since \\(\\lambda\\geq2\\) one can take\n\t\t\t\\(\n\t\t\t\t\\alpha\n\t\t\t{}\\coloneqq{}\n\t\t\t\t\\nicefrac{2\\xi}{\\lambda}\n\t\t\t{}\\in{}\n\t\t\t\t(0,1)\n\t\t\t\\)\n\t\t\tto make the coefficient multiplying the gradient norm vanish.\n\t\t\tWe then obtain\n\t\t\t\\begin{equation}\\label{eq:clambda2}\n\t\t\t\t\\tfrac{c}{L_{\\sDRS}}\n\t\t\t{}={}\n\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\xi}\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{(\\lambda-2\\xi)p}{2\\lambda}.\n\t\t\t\\end{equation}\n\t\t\tImposing \\(c>0\\) results in the following second-order equation in variable \\(\\xi\\),\n\t\t\t\\begin{equation}\\label{eq:xi}\n\t\t\t\t2p\\xi^2-p\\lambda\\xi+(\\lambda-2)\n\t\t\t{}<{}\n\t\t\t\t0.\n\t\t\t\\end{equation}\n\t\t\tThe discriminant is\n\t\t\t\\(\n\t\t\t\t\\Delta\n\t\t\t{}\\coloneqq{}\n\t\t\t\t(p\\lambda)^2\n\t\t\t\t{}-{}\n\t\t\t\t8p(\\lambda-2)\n\t\t\t\\),\n\t\t\twhich, for \\(\\lambda\\geq2\\), is strictly positive iff\n\t\t\t\\[\n\t\t\t\t2\n\t\t\t{}\\leq{}\n\t\t\t\t\\lambda\n\t\t\t{}<{}\n\t\t\t\t\\tfrac{4}{1+\\sqrt{1-p}}\n\t\t\t~~\\vee~~\n\t\t\t\t\\lambda\n\t\t\t{}>{}\n\t\t\t\t\\tfrac{4}{1-\\sqrt{1-p}}.\n\t\t\t\\]\n\t\t\tDenoting\n\t\t\t\\(\n\t\t\t\t\\delta\n\t\t\t{}\\coloneqq{}\n\t\t\t\t\\sqrt\\Delta\n\t\t\t{}={}\n\t\t\t\t\\sqrt{\n\t\t\t\t\t(p\\lambda)^2\n\t\t\t\t\t{}-{}\n\t\t\t\t\t8p(\\lambda-2)\n\t\t\t\t}\n\t\t\t\\),\n\t\t\tthe solution to \\eqref{eq:xi} is\n\t\t\t\\(\n\t\t\t\t\\tfrac{\n\t\t\t\t\tp\\lambda\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\delta\n\t\t\t\t}{\n\t\t\t\t\t4p\n\t\t\t\t}\n\t\t\t{}<{}\n\t\t\t\t\\x\n\t\t\t{}<{}\n\t\t\t\t\\tfrac{\n\t\t\t\t\tp\\lambda\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\delta\n\t\t\t\t}{\n\t\t\t\t\t4p\n\t\t\t\t}\n\t\t\t\\).\n\t\t\tHowever, the case \\(\\lambda\\geq 4\\) has to be discarded, as\n\t\t\t\\(\n\t\t\t\t\\tfrac{p\\lambda-\\delta}{4p}\n\t\t\t{}>{}\n\t\t\t\t1\n\t\t\t\\)\n\t\t\tin this case, contradicting the fact that \\(p\\leq1\\).\n\t\t\tTo see this, suppose \\(\\lambda\\geq4\\).\n\t\t\tThen,\n\t\t\t\\begin{align*}\n\t\t\t\t\\tfrac{\n\t\t\t\t\tp\\lambda\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\delta\n\t\t\t\t}{\n\t\t\t\t\t4p\n\t\t\t\t}\n\t\t\t{}<{}\n\t\t\t\t1\n\t\t\t~~\\Leftrightarrow~~ &\n\t\t\t\tp(\\lambda-4)\n\t\t\t{}<{}\n\t\t\t\t\\delta\n\t\t\t\\\\\n\t\t\t~~\\Leftrightarrow~~ &\n\t\t\t\tp^2(\\lambda-4)^2\n\t\t\t{}<{}\n\t\t\t\t\\Delta\n\t\t\t{}={}\n\t\t\t\t(p\\lambda)^2\n\t\t\t\t{}-{}\n\t\t\t\t8p(\\lambda-2)\n\t\t\t\\\\\n\t\t\t~~\\Leftrightarrow~~ &\n\t\t\t\tp(2-\\lambda)\n\t\t\t{}<{}\n\t\t\t\t2-\\lambda,\n\t\t\t\\end{align*}\n\t\t\thence \\(p>1\\), which contradicts the fact that \\(\\sigma_{\\sDRS}\\leq L_{\\sDRS}\\).\n% \t\t\tHere and in the second implication we used the assumption \\(\\lambda\\geq4\\).\n\t\t\tThus, the only feasible ranges are the ones given in \\eqref{eq:lambda>=2}, hence the claimed sufficient decrease constant \\(c\\), cf. \\eqref{eq:clambda2}.\n\t\t\t\\qedhere\n\t\t\\end{proofitemize}\n% \t\tAlso in this case, the claimed range of \\(\\gamma\\) and value of \\(c\\) have been found.\n\t\\end{proof}\n\\end{thm}","sha256_utf8":"93e455cd94d5571dfb623e9419c7fbe58c0f67bcff8c8e6ed100ad36ccaac30d"},{"start":94108,"end":97751,"start_line":2385,"end_line":2490,"raw_tex":"\\begin{proof}\n\t\tTo avoid trivialities, we assume that a fixed point is not found in a finite number of iterations, hence that \\(v^k\\neq u^k\\) for all \\(k\\)'s.\n\t\t\\begin{proofitemize}\n\t\t\\item\\ref{thm:DRS:r}\n\t\t\tLet \\(c=c(\\gamma,\\lambda)\\) be as in \\cref{thm:DRE:SD}.\n\t\t\tTelescoping the inequality \\eqref{eq:DRE:SD} yields\n\t\t\t\\begin{align*}\n\t\t\t\\textstyle\n\t\t\t\t\\tfrac{c\\lambda^2}{(1+\\gamma L_{\\sDRS})^2}\n\t\t\t\t\\sum_{k\\in\\N}{\n\t\t\t\t\t\\|u^k-v^k\\|^2\n\t\t\t\t}\n\t\t\t{}\\leq{} &\n\t\t\t\\textstyle\n\t\t\t\t\\sum_{k\\in\\N}{\n\t\t\t\t\\bigl[\n\t\t\t\t\t\\DRE(s^k)\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\DRE(s^{k+1})\n\t\t\t\t\\bigr]}\n\t\t\t{}\\leq{}\n\t\t\t\t\\DRE(s^0)-\\inf\\DRE.\n\t\t\t\\end{align*}\n\t\t\tSince \\(\\inf\\DRE=\\inf\\varphi>-\\infty\\) and \\(\\DRE\\) is real valued (cf. \\cref{thm:C0,thm:DREequiv}), it follows that \\(\\seq{u^k-v^k}\\) is square summable, hence the claimed rate of convergence.\n\t\t\tMoreover, since \\(\\DRE(s^k)\\) is decreasing it admits a (finite) limit, be it \\(\\varphi_\\star\\).\n\t%%%%%%%%\n\t\t\\item\\ref{thm:DRS:uv}\n\t\t\tSince \\(\\seq{u^k-v^k}\\to 0\\), necessarily \\(\\seq{u^k}\\) and \\(\\seq{v^k}\\) have same cluster points.\n\t\t\tSuppose that \\(\\seq{u^k}[k\\in K]\\to u'\\) for some \\(K\\subseteq\\N\\) and \\(u'\\in\\R^p\\).\n\t\t\tThen, \\(\\seq{v^k}[k\\in K]\\to u'\\), and since \\(s^k=u^k+\\nabla\\sDRS(u^k)\\) (cf. \\cref{thm:proxfEquiv}), continuity of \\(\\nabla\\sDRS\\) implies that\n\t\t\t\\(\n\t\t\t\t\\seq{s^k}[k\\in K]\n\t\t\t{}\\to{}\n\t\t\t\ts'\n\t\t\t{}={}\n\t\t\t\tu'+\\gamma\\nabla\\sDRS(u')\n\t\t\t\\).\n\t\t\tFrom \\cref{thm:proxfEquiv} we infer that \\(u'=\\prox_{\\gamma\\sDRS}(s')\\).\n\t\t\t\n\t\t\tSimilarly, \\(\\seq{\\Fw{u^k}}[k\\in K]\\to \\Fw{u'}\\), and the outer semicontinuity of \\(\\prox_{\\gamma\\nsDRS}\\) \\cite[Ex. 5.23(b)]{rockafellar2011variational} combined with \\eqref{eq:vFBS} implies that\n\t\t\t\\[\n\t\t\t\tu'\n\t\t\t{}={}\n\t\t\t\t\\lim_{K\\ni k\\to\\infty}{\n\t\t\t\t\t\\hspace*{-5pt}\n\t\t\t\t\tv^k\n\t\t\t\t}\n\t\t\t{}\\in{}\n\t\t\t\t\\limsup_{K\\ni k\\to\\infty}{\n\t\t\t\t\t\\FB{u^k}\n\t\t\t\t}\n\t\t\t{}\\subseteq{}\n\t\t\t\t\\FB{u'}.\n\t\t\t\\]\n\t\t\tFrom \\eqref{eq:prox:subdiff} we then have that\n\t\t\t\\(\n\t\t\t\t-\\nabla\\sDRS(u')\n\t\t\t{}\\in{}\n\t\t\t\t\\hat\\partial\\nsDRS(u')\n\t\t\t\\),\n\t\t\thence \\(0\\in\\hat\\partial\\varphi(u')\\), as it follows from \\cite[Ex. 8.8]{rockafellar2011variational}.\n\t\t\tFinally, since \\(v^k\\to u'\\),\n\t\t\t\\[\n\t\t\t\t\\varphi(u')\n\t\t\t{}\\leq{}\n\t\t\t\t\\liminf_{K\\ni k\\to\\infty}{\n\t\t\t\t\t\\varphi(v^k)\n\t\t\t\t}\n\t\t\t{}\\leq{}\n\t\t\t\t\\limsup_{K\\ni k\\to\\infty}{\n\t\t\t\t\t\\varphi(v^k)\n\t\t\t\t}\n\t\t\t{}\\leq{}\n\t\t\t\t\\limsup_{K\\ni k\\to\\infty}{\n\t\t\t\t\t\\DRE(s^k)\n\t\t\t\t}\n\t\t\t{}={}\n\t\t\t\t\\DRE(s')\n\t\t\t{}\\leq{}\n\t\t\t\t\\varphi(u'),\n\t\t\t\\]\n\t\t\twhere the first inequality is due to lower semicontinuity of \\(\\varphi\\), the third and the last to the sandwiching property (\\cref{thm:sandwich}), and the equality to the continuity of \\(\\DRE\\) (\\cref{thm:C0}).\n\t\t\tThis shows that \\(\\seq{\\varphi(u^k)}[k\\in K]\\to\\varphi(u')=\\DRE(s')\\), and since \\(\\seq{\\DRE(s^k)}\\to\\varphi_\\star\\), then necessarily \\(\\varphi(u')=\\DRE(s')=\\varphi_\\star\\) independently of the cluster point \\(u'\\).\n\t\t\tThe last assert follows from the optimality condition of \\(v^k\\) as in \\eqref{eq:vFBS}, namely,\n\t\t\t\\(\n\t\t\t\t\\tfrac1\\gamma(u^k-v^k)\n\t\t\t\t{}-{}\n\t\t\t\t\\nabla\\sDRS(u^k)\n\t\t\t{}\\in{}\n\t\t\t\t\\hat\\partial\\nsDRS(v^k)\n\t\t\t\\)\n\t\t\tdue to \\eqref{eq:prox:subdiff}, together with \\(\\sigma_{\\sDRS}\\)-hypoconvexity of \\(\\sDRS\\).\n\t\t\\item\\ref{thm:DRS:bounded}.\n\t\t\tSuppose that \\(\\varphi\\) has bounded level sets.\n\t\t\tThen, it follows from \\cref{thm:boundedlev} that so does \\(\\DRE\\), and since\n\t\t\t\\(\n\t\t\t\ts^k\n\t\t\t{}\\in{}\n\t\t\t\t\\lev_{\\leq\\DRE(s^0)}\\DRE\n\t\t\t\\)\n\t\t\tfor all \\(k\\in\\N\\), then the sequence \\(\\seq{s^k}\\) is bounded.\n\t\t\tDue to Lipschitz continuity of \\(\\prox_{\\gamma\\sDRS}\\) (cf. \\cref{thm:proxfLip}), also \\(\\seq{u^k}\\) is bounded.\n\t\t\tIn turn, since \\(v^k-u^k\\to 0\\) we conclude that also \\(\\seq{v^k}\\) is bounded.\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}","sha256_utf8":"8a3a214b5dda9c2b688b6556af15725c4075220976c1d79df7fee8369b791b18"},{"start":199331,"end":201623,"start_line":5441,"end_line":5533,"raw_tex":"\\begin{appendixproof}{thm:rho}%\n\t\\begin{proofitemize}\n\t\\item\\ref{thm:hypo:rho0}.\n\t\tThis is the lower bound in \\eqref{eq:LipBound}.\n\t\\item\\ref{thm:hypo:rho1}.\n\t\tLet \\(L\\geq L_h\\) and \\(\\sigma\\in(-L,\\min\\set{0,\\sigma_h}]\\) be fixed.\n\t\tThen, \\(h\\) is \\(L\\)-smooth and \\(\\sigma\\)-hypoconvex, and from \\cite[Thm. 2.1.12]{nesterov2003introductory} we obtain that\n\t\t\\begin{equation}\\label{eq:LipBound:mix}\n\t\t\t\\innprod{\n\t\t\t\t\\nabla h(y)-\\nabla h(x)\n\t\t\t}{\n\t\t\t\ty-x\n\t\t\t}\n\t\t{}\\geq{}\n\t\t\t\\tfrac{\\sigma L}{L+\\sigma}\\|x-y\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{L+\\sigma}\\|\\nabla h(x)-\\nabla h(y)\\|^2\n\t\t\\end{equation}\n\t\tfor all \\(x,y\\in\\R^n\\).\n\t\t(Although \\cite[Thm. 2.1.12]{nesterov2003introductory} assumes \\(\\sigma>0\\), the given proof does not necessitate this restriction).\n\t\tMoreover, \\(\\psi\\coloneqq h-\\tfrac\\sigma2\\|{}\\cdot{}\\|^2\\) is convex and \\(L_\\psi\\)-smooth, with \\(L_\\psi=L-\\sigma\\).\n\t\tConsequently, for all \\(x,y\\in\\R^n\\) one has\n\t\t\\(\n\t\t\t\\psi(y)\n\t\t{}\\geq{}\n\t\t\t\\psi(x)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla\\psi(x)}{y-x}\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2L_\\psi}\\|\\nabla\\psi(y)-\\nabla\\psi(x)\\|^2\n\t\t\\),\n\t\tsee \\cite[Thm. 2.1.5]{nesterov2003introductory}, resulting in\n\t\t\\begin{align*}\n\t\t\th(y)\n\t\t{}\\geq{} &\n\t\t\th(x)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla h(x)}{y-x}\n\t\t\t{}+{}\n\t\t\t\\tfrac{\\sigma L}{2(L-\\sigma)}\n\t\t\t\\|y-x\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2(L-\\sigma)}\n\t\t\t\\|\\nabla h(y)-\\nabla h(x)\\|^2\n\t\t\\\\\n\t\t&\n\t\t\t{}-{}\n\t\t\t\\tfrac{\\sigma}{2(L-\\sigma)}\n\t\t\t\\innprod{\n\t\t\t\t\\nabla h(y)-\\nabla h(x)\n\t\t\t}{\n\t\t\t\ty-x\n\t\t\t}.\n\t\t\\end{align*}\n\t\tSince \\(\\sigma\\leq0\\), the coefficient of the scalar product in the second line is positive.\n\t\tWe may thus invoke the inequality \\eqref{eq:LipBound:mix} to arrive to\n\t\t\\begin{align*}\n\t\t\th(y)\n\t\t{}\\geq{} &\n\t\t\th(x)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla h(x)}{y-x}\n\t\t\t{}+{}\n\t\t\t\\tfrac{\\sigma L}{2(L-\\sigma)}\n\t\t\t\\|y-x\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2(L-\\sigma)}\n\t\t\t\\|\\nabla h(y)-\\nabla h(x)\\|^2\n\t\t\\\\\n\t\t&\n\t\t\t{}-{}\n\t\t\t\\tfrac{\\sigma}{2(L-\\sigma)}\n\t\t\t\\left[\n\t\t\t\t\\tfrac{\\sigma L}{L+\\sigma}\\|x-y\\|^2\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{L+\\sigma}\\|\\nabla h(x)-\\nabla h(y)\\|^2\n\t\t\t\\right]\n\t\t\\\\\n\t\t{}={} &\n\t\t\th(x)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla h(x)}{y-x}\n\t\t\t{}+{}\n\t\t\t\\tfrac{\\sigma L}{2(L+\\sigma)}\n\t\t\t\\|y-x\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2(L+\\sigma)}\n\t\t\t\\|\\nabla h(y)-\\nabla h(x)\\|^2,\n\t\t\\end{align*}\n\t\thence the claimed inequality.\n\t\\qedhere\n\t\\end{proofitemize}\n\\end{appendixproof}","sha256_utf8":"55109ad0b6fa2d44137d298b88b0669d0a5ccdb438f95a0b7a81edd06480e847"},{"start":201627,"end":204771,"start_line":5537,"end_line":5625,"raw_tex":"\\begin{appendixproof}{thm:proxf}%\n\tLet \\(\\gamma\\in(0,\\nicefrac{1}{[\\sigma_h]_-})\\) be fixed, and let \\(\\psi\\coloneqq\\gamma h+\\tfrac12\\|{}\\cdot{}\\|^2\\).\n\tObserve that \\(\\psi\\in\\cont^{1,1}(\\R^n)\\) is \\(L_\\psi\\)-smooth and \\(\\sigma_\\psi\\)-strongly convex, with \\(L_\\psi=1+\\gamma L_h\\) and \\(\\sigma_\\psi=1+\\gamma\\sigma_h\\).\n\tIn particular, due to strong convexity \\(\\inf\\psi>-\\infty\\), and by definition of prox-boundedness it then follows that \\(\\gamma_h\\geq\\nicefrac{1}{[\\sigma_h]_-}\\).\n\t\\begin{proofitemize}\n\t\\item\\ref{thm:proxfEquiv}.\n\t\tFollows from \\eqref{eq:prox:subdiff}, by observing that \\(h+\\tfrac{1}{2\\gamma}\\|{}\\cdot{}-s\\|^2\\) is strongly convex, hence that a minimizer is characterized by stationarity.\n\t\\item\\ref{thm:proxfLip}.\n\t\tFor \\(s,s'\\in\\R^n\\), let \\(u=\\prox_{\\gamma h}(s)\\) and \\(u'=\\prox_{\\gamma h}(s')\\).\n\t\tThen,\n\t\t\\[\n\t\t\t\\innprod{s-s'}{u-u'}\n\t\t{}={}\n\t\t\t\\innprod{\n\t\t\t\t\\nabla\\psi(u)\n\t\t\t\t{}-{}\n\t\t\t\t\\nabla\\psi(u')\n\t\t\t}{u-u'}\n\t\t{}\\geq{}\n\t\t\t\\sigma_\\psi\n\t\t\t\\|u-u'\\|^2\n\t\t{}={}\n\t\t\t(1+\\gamma\\sigma_h)\n\t\t\t\\|u-u'\\|^2,\n\t\t\\]\n\t\twhere the first equality was shown in \\ref{thm:proxfEquiv} and the inequality follows from \\eqref{eq:innprod}.\n\t\tBy using the \\(\\frac{1}{L_\\psi}\\)-cocoercivity of \\(\\nabla\\psi\\) \\cite[Thm. 2.1.10]{nesterov2003introductory}, also the claimed strong monotonicity follows.\n\t\tIn turn, the Cauchy-Schwartz inequality on the inner product yields \\eqref{eq:biLip}.\n\t\\item\\ref{thm:MoreaufC1}.\n\t\tFrom \\cite[Ex. 10.32]{rockafellar2011variational} it follows that \\(h^\\gamma\\) is strictly continuous and that\n\t\t\\(\n\t\t\t\\partial h^\\gamma(s)\n\t\t{}\\subseteq{}\n\t\t\t\\tfrac1\\gamma(s-\\prox_{\\gamma h}(s))\n\t\t\\).\n\t\tBecause of single valuedness of \\(\\prox_{\\gamma h}\\), by invoking \\cite[Thm. 9.18]{rockafellar2011variational} we conclude that \\(h^\\gamma\\) is everywhere differentiable with\n\t\t\\(\n\t\t\t\\nabla h^\\gamma(s)\n\t\t{}={}\n\t\t\t\\tfrac1\\gamma(s-\\prox_{\\gamma h}(s))\n\t\t\\).\n% \t\tMoreover, \\(u\\) minimizes \\(h+\\tfrac{1}{2\\gamma}\\|{}\\cdot{}-s\\|^2\\) and in particular \\(\\tfrac{1}{\\gamma}(s-u)\\in\\partial h(u)\\).\n% \t\tTherefore,\n% \t\t\\(\n% \t\t\t\\nabla h(u)\n% \t\t{}\\overrel*{\\eqref{eq:NablaProj}}{}\n% \t\t\t\\proj_{\\dom h}\\partial h(u)\n% \t\t{}={}\n% \t\t\t\\tfrac{1}{\\gamma}(\\proj_{\\dom h}s-u)\n% \t\t\\),\n% \t\twhere the last equality follows from the fact that \\(u\\in\\dom h\\).\n\t\tThus,\n\t\t\\begin{align*}\n\t\t\t\\innprod{\\nabla h^\\gamma(s)-\\nabla h^\\gamma(s')}{s-s'}\n\t\t{}={} &\n\t\t\t\\tfrac{1}{\\gamma}\n\t\t\t\\left(\n\t\t\t\t\\|s-s'\\|^2\n\t\t\t\t{}-{}\n\t\t\t\t\\innprod{s-s'}{u-u'}\n\t\t\t\\right),\n% \t\t\\\\\n% \t\t{}\\leq{} &\n% \t\t\t\\tfrac{1}{\\gamma}\n% \t\t\t\\bigl(\n% \t\t\t\t1\n% \t\t\t\t{}-{}\n% \t\t\t\t\\tfrac{1}{1+\\gamma L_{h}}\n% \t\t\t\\bigr)\n% \t\t\t\t\\|s-s'\\|^2\n% \t\t{}={}\n% \t\t\t\\tfrac{L_{h}}{1+\\gamma L_{h}}\n% \t\t\t\\|s-s'\\|^2.\n\t\t\\end{align*}\n\t\tand from the bounds in \\ref{thm:proxfLip} we conclude that\n% \t\tSimilarly, by using also the second inequality in \\eqref{eq:coco} we arrive to\n\t\t\\[\n\t\t\t\\tfrac{\\sigma_{h}}{1+\\gamma\\sigma_{h}}\n\t\t\t\\|s-s'\\|^2\n\t\t{}\\leq{}\n\t\t\t\\innprod{\\nabla h^\\gamma(s)-\\nabla h^\\gamma(s')}{s-s'}\n\t\t{}\\leq{}\n\t\t\t\\tfrac{L_{h}}{1+\\gamma L_{h}}\n\t\t\t\\|s-s'\\|^2.\n\t\t\\]\n\t\tThe claimed smoothness and hypoconvexity follow from the characterization of \\eqref{eq:innprod}.\n\t\t\\qedhere\n\t\\end{proofitemize}\n\\end{appendixproof}","sha256_utf8":"c177f59a3b718e9410a4198c001b3f3570fa421308c49b3919a0740e9462b043"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"DRS子序列收敛与残差速率","source_units":[{"start":93135,"end":94108,"start_line":2362,"end_line":2385,"raw_tex":"\\begin{thm}[Subsequential convergence]\\label{thm:DRS}%\n\tSuppose that \\Cref{ass:DRS} is satisfied, and consider a sequence \\(\\seq{s^k,u^k,v^k}\\) generated by \\ref{DRS} with stepsize \\(\\gamma\\) and relaxation \\(\\lambda\\) as in \\Cref{thm:DRE:SD}, starting from \\(s^0\\in\\R^p\\).\n\tThe following hold:%\n\t\\begin{enumerate}\n\t\\item\\label{thm:DRS:r}\n\t\tThe residual \\(\\seq{u^k-v^k}\\) vanishes with rate\n\t\t\\(\n\t\t\t\\min_{i\\leq k}\\|u^i-v^i\\|\n\t\t{}={}\n\t\t\to(\\nicefrac{1}{\\sqrt k})\n\t\t\\).\n\t\\item\\label{thm:DRS:uv}\n\t\t\\(\\seq{u^k}\\) and \\(\\seq{v^k}\\) have same cluster points, all of which are stationary for \\(\\varphi\\) and on which \\(\\varphi\\) has same value, this being the limit of \\(\\seq{\\DRE(s^k)}\\).\n\t\tIn fact, for each \\(k\\) one has\n\t\t\\(\n\t\t\t\\dist(0,\\hat\\partial\\varphi(v^k))\n\t\t{}\\leq{}\n\t\t\t\\frac{1-\\gamma\\sigma_{\\sDRS}}{2\\gamma}\n\t\t\t\\|u^k-v^k\\|\n\t\t\\).\n\t\\item\\label{thm:DRS:bounded}\n\t\tIf \\(\\varphi\\) has bounded level sets, then the sequence \\(\\seq{s^k,u^k,v^k}\\) is bounded.\n\t\\end{enumerate}\n\t","sha256_utf8":"1db1e47d60d7f2f2e9beef27ac69ef8efa4931fe8461c081777f8fd7b5fc6dae"},{"start":97751,"end":97761,"start_line":2490,"end_line":2491,"raw_tex":"\n\\end{thm}","sha256_utf8":"27661e277ceb78e4dd676b4ea4b5c8d9a4d3c607a1a758eac095282a4f311f77"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"主收敛定理完整证明","source_units":[{"start":94108,"end":97751,"start_line":2385,"end_line":2490,"raw_tex":"\\begin{proof}\n\t\tTo avoid trivialities, we assume that a fixed point is not found in a finite number of iterations, hence that \\(v^k\\neq u^k\\) for all \\(k\\)'s.\n\t\t\\begin{proofitemize}\n\t\t\\item\\ref{thm:DRS:r}\n\t\t\tLet \\(c=c(\\gamma,\\lambda)\\) be as in \\cref{thm:DRE:SD}.\n\t\t\tTelescoping the inequality \\eqref{eq:DRE:SD} yields\n\t\t\t\\begin{align*}\n\t\t\t\\textstyle\n\t\t\t\t\\tfrac{c\\lambda^2}{(1+\\gamma L_{\\sDRS})^2}\n\t\t\t\t\\sum_{k\\in\\N}{\n\t\t\t\t\t\\|u^k-v^k\\|^2\n\t\t\t\t}\n\t\t\t{}\\leq{} &\n\t\t\t\\textstyle\n\t\t\t\t\\sum_{k\\in\\N}{\n\t\t\t\t\\bigl[\n\t\t\t\t\t\\DRE(s^k)\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\DRE(s^{k+1})\n\t\t\t\t\\bigr]}\n\t\t\t{}\\leq{}\n\t\t\t\t\\DRE(s^0)-\\inf\\DRE.\n\t\t\t\\end{align*}\n\t\t\tSince \\(\\inf\\DRE=\\inf\\varphi>-\\infty\\) and \\(\\DRE\\) is real valued (cf. \\cref{thm:C0,thm:DREequiv}), it follows that \\(\\seq{u^k-v^k}\\) is square summable, hence the claimed rate of convergence.\n\t\t\tMoreover, since \\(\\DRE(s^k)\\) is decreasing it admits a (finite) limit, be it \\(\\varphi_\\star\\).\n\t%%%%%%%%\n\t\t\\item\\ref{thm:DRS:uv}\n\t\t\tSince \\(\\seq{u^k-v^k}\\to 0\\), necessarily \\(\\seq{u^k}\\) and \\(\\seq{v^k}\\) have same cluster points.\n\t\t\tSuppose that \\(\\seq{u^k}[k\\in K]\\to u'\\) for some \\(K\\subseteq\\N\\) and \\(u'\\in\\R^p\\).\n\t\t\tThen, \\(\\seq{v^k}[k\\in K]\\to u'\\), and since \\(s^k=u^k+\\nabla\\sDRS(u^k)\\) (cf. \\cref{thm:proxfEquiv}), continuity of \\(\\nabla\\sDRS\\) implies that\n\t\t\t\\(\n\t\t\t\t\\seq{s^k}[k\\in K]\n\t\t\t{}\\to{}\n\t\t\t\ts'\n\t\t\t{}={}\n\t\t\t\tu'+\\gamma\\nabla\\sDRS(u')\n\t\t\t\\).\n\t\t\tFrom \\cref{thm:proxfEquiv} we infer that \\(u'=\\prox_{\\gamma\\sDRS}(s')\\).\n\t\t\t\n\t\t\tSimilarly, \\(\\seq{\\Fw{u^k}}[k\\in K]\\to \\Fw{u'}\\), and the outer semicontinuity of \\(\\prox_{\\gamma\\nsDRS}\\) \\cite[Ex. 5.23(b)]{rockafellar2011variational} combined with \\eqref{eq:vFBS} implies that\n\t\t\t\\[\n\t\t\t\tu'\n\t\t\t{}={}\n\t\t\t\t\\lim_{K\\ni k\\to\\infty}{\n\t\t\t\t\t\\hspace*{-5pt}\n\t\t\t\t\tv^k\n\t\t\t\t}\n\t\t\t{}\\in{}\n\t\t\t\t\\limsup_{K\\ni k\\to\\infty}{\n\t\t\t\t\t\\FB{u^k}\n\t\t\t\t}\n\t\t\t{}\\subseteq{}\n\t\t\t\t\\FB{u'}.\n\t\t\t\\]\n\t\t\tFrom \\eqref{eq:prox:subdiff} we then have that\n\t\t\t\\(\n\t\t\t\t-\\nabla\\sDRS(u')\n\t\t\t{}\\in{}\n\t\t\t\t\\hat\\partial\\nsDRS(u')\n\t\t\t\\),\n\t\t\thence \\(0\\in\\hat\\partial\\varphi(u')\\), as it follows from \\cite[Ex. 8.8]{rockafellar2011variational}.\n\t\t\tFinally, since \\(v^k\\to u'\\),\n\t\t\t\\[\n\t\t\t\t\\varphi(u')\n\t\t\t{}\\leq{}\n\t\t\t\t\\liminf_{K\\ni k\\to\\infty}{\n\t\t\t\t\t\\varphi(v^k)\n\t\t\t\t}\n\t\t\t{}\\leq{}\n\t\t\t\t\\limsup_{K\\ni k\\to\\infty}{\n\t\t\t\t\t\\varphi(v^k)\n\t\t\t\t}\n\t\t\t{}\\leq{}\n\t\t\t\t\\limsup_{K\\ni k\\to\\infty}{\n\t\t\t\t\t\\DRE(s^k)\n\t\t\t\t}\n\t\t\t{}={}\n\t\t\t\t\\DRE(s')\n\t\t\t{}\\leq{}\n\t\t\t\t\\varphi(u'),\n\t\t\t\\]\n\t\t\twhere the first inequality is due to lower semicontinuity of \\(\\varphi\\), the third and the last to the sandwiching property (\\cref{thm:sandwich}), and the equality to the continuity of \\(\\DRE\\) (\\cref{thm:C0}).\n\t\t\tThis shows that \\(\\seq{\\varphi(u^k)}[k\\in K]\\to\\varphi(u')=\\DRE(s')\\), and since \\(\\seq{\\DRE(s^k)}\\to\\varphi_\\star\\), then necessarily \\(\\varphi(u')=\\DRE(s')=\\varphi_\\star\\) independently of the cluster point \\(u'\\).\n\t\t\tThe last assert follows from the optimality condition of \\(v^k\\) as in \\eqref{eq:vFBS}, namely,\n\t\t\t\\(\n\t\t\t\t\\tfrac1\\gamma(u^k-v^k)\n\t\t\t\t{}-{}\n\t\t\t\t\\nabla\\sDRS(u^k)\n\t\t\t{}\\in{}\n\t\t\t\t\\hat\\partial\\nsDRS(v^k)\n\t\t\t\\)\n\t\t\tdue to \\eqref{eq:prox:subdiff}, together with \\(\\sigma_{\\sDRS}\\)-hypoconvexity of \\(\\sDRS\\).\n\t\t\\item\\ref{thm:DRS:bounded}.\n\t\t\tSuppose that \\(\\varphi\\) has bounded level sets.\n\t\t\tThen, it follows from \\cref{thm:boundedlev} that so does \\(\\DRE\\), and since\n\t\t\t\\(\n\t\t\t\ts^k\n\t\t\t{}\\in{}\n\t\t\t\t\\lev_{\\leq\\DRE(s^0)}\\DRE\n\t\t\t\\)\n\t\t\tfor all \\(k\\in\\N\\), then the sequence \\(\\seq{s^k}\\) is bounded.\n\t\t\tDue to Lipschitz continuity of \\(\\prox_{\\gamma\\sDRS}\\) (cf. \\cref{thm:proxfLip}), also \\(\\seq{u^k}\\) is bounded.\n\t\t\tIn turn, since \\(v^k-u^k\\to 0\\) we conclude that also \\(\\seq{v^k}\\) is bounded.\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}","sha256_utf8":"8a3a214b5dda9c2b688b6556af15725c4075220976c1d79df7fee8369b791b18"}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","title":"排除有限步到达不动点","source_units":[{"start":94122,"end":94267,"start_line":2386,"end_line":2386,"raw_tex":"\t\tTo avoid trivialities, we assume that a fixed point is not found in a finite number of iterations, hence that \\(v^k\\neq u^k\\) for all \\(k\\)'s.\n","sha256_utf8":"df3642124e7f021b739a0e4b2448868287d87107ac2b508ba617831de2c02afc"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"处理有限终止的平凡情形","source_units":[{"start":94108,"end":94290,"start_line":2385,"end_line":2387,"raw_tex":"\\begin{proof}\n\t\tTo avoid trivialities, we assume that a fixed point is not found in a finite number of iterations, hence that \\(v^k\\neq u^k\\) for all \\(k\\)'s.\n\t\t\\begin{proofitemize}\n","sha256_utf8":"6a6b881595fdcc97084cf5ca45ef6edf486b07dd31bb254dd846d13bae967da5"}]},{"id":"S005","semantic_type":"proof_step","kind":"proof_step","title":"望远镜求和与残差速率","source_units":[{"start":94290,"end":94914,"start_line":2388,"end_line":2408,"raw_tex":"\t\t\\item\\ref{thm:DRS:r}\n\t\t\tLet \\(c=c(\\gamma,\\lambda)\\) be as in \\cref{thm:DRE:SD}.\n\t\t\tTelescoping the inequality \\eqref{eq:DRE:SD} yields\n\t\t\t\\begin{align*}\n\t\t\t\\textstyle\n\t\t\t\t\\tfrac{c\\lambda^2}{(1+\\gamma L_{\\sDRS})^2}\n\t\t\t\t\\sum_{k\\in\\N}{\n\t\t\t\t\t\\|u^k-v^k\\|^2\n\t\t\t\t}\n\t\t\t{}\\leq{} &\n\t\t\t\\textstyle\n\t\t\t\t\\sum_{k\\in\\N}{\n\t\t\t\t\\bigl[\n\t\t\t\t\t\\DRE(s^k)\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\DRE(s^{k+1})\n\t\t\t\t\\bigr]}\n\t\t\t{}\\leq{}\n\t\t\t\t\\DRE(s^0)-\\inf\\DRE.\n\t\t\t\\end{align*}\n\t\t\tSince \\(\\inf\\DRE=\\inf\\varphi>-\\infty\\) and \\(\\DRE\\) is real valued (cf. \\cref{thm:C0,thm:DREequiv}), it follows that \\(\\seq{u^k-v^k}\\) is square summable, hence the claimed rate of convergence.\n","sha256_utf8":"d31c969326694ffbdf5b64035a467d1460b93df0b1f7f0dab657da74e6c10d42"}]},{"id":"S006","semantic_type":"proof_step","kind":"proof_step","title":"包络值单调趋于有限极限","source_units":[{"start":94914,"end":95024,"start_line":2409,"end_line":2410,"raw_tex":"\t\t\tMoreover, since \\(\\DRE(s^k)\\) is decreasing it admits a (finite) limit, be it \\(\\varphi_\\star\\).\n\t%%%%%%%%\n","sha256_utf8":"9b23e9a27806e931312f0b3dbae59474eb3a8304c9f954919f6b7504a2104b4c"}]},{"id":"S007","semantic_type":"proof_step","kind":"proof_step","title":"共同聚点与对应原像","source_units":[{"start":95024,"end":95556,"start_line":2411,"end_line":2422,"raw_tex":"\t\t\\item\\ref{thm:DRS:uv}\n\t\t\tSince \\(\\seq{u^k-v^k}\\to 0\\), necessarily \\(\\seq{u^k}\\) and \\(\\seq{v^k}\\) have same cluster points.\n\t\t\tSuppose that \\(\\seq{u^k}[k\\in K]\\to u'\\) for some \\(K\\subseteq\\N\\) and \\(u'\\in\\R^p\\).\n\t\t\tThen, \\(\\seq{v^k}[k\\in K]\\to u'\\), and since \\(s^k=u^k+\\nabla\\sDRS(u^k)\\) (cf. \\cref{thm:proxfEquiv}), continuity of \\(\\nabla\\sDRS\\) implies that\n\t\t\t\\(\n\t\t\t\t\\seq{s^k}[k\\in K]\n\t\t\t{}\\to{}\n\t\t\t\ts'\n\t\t\t{}={}\n\t\t\t\tu'+\\gamma\\nabla\\sDRS(u')\n\t\t\t\\).\n\t\t\tFrom \\cref{thm:proxfEquiv} we infer that \\(u'=\\prox_{\\gamma\\sDRS}(s')\\).\n","sha256_utf8":"3ca5e5c1e7773596324cd90ef9d38b00452bdf1853870f997197a214bd84a2b4"}]},{"id":"S008","semantic_type":"proof_step","kind":"proof_step","title":"近端外半连续性给出不动点","source_units":[{"start":95556,"end":95943,"start_line":2423,"end_line":2438,"raw_tex":"\t\t\t\n\t\t\tSimilarly, \\(\\seq{\\Fw{u^k}}[k\\in K]\\to \\Fw{u'}\\), and the outer semicontinuity of \\(\\prox_{\\gamma\\nsDRS}\\) \\cite[Ex. 5.23(b)]{rockafellar2011variational} combined with \\eqref{eq:vFBS} implies that\n\t\t\t\\[\n\t\t\t\tu'\n\t\t\t{}={}\n\t\t\t\t\\lim_{K\\ni k\\to\\infty}{\n\t\t\t\t\t\\hspace*{-5pt}\n\t\t\t\t\tv^k\n\t\t\t\t}\n\t\t\t{}\\in{}\n\t\t\t\t\\limsup_{K\\ni k\\to\\infty}{\n\t\t\t\t\t\\FB{u^k}\n\t\t\t\t}\n\t\t\t{}\\subseteq{}\n\t\t\t\t\\FB{u'}.\n\t\t\t\\]\n","sha256_utf8":"380e95ee9f307073c4d26897589f8882e85395be8734ce4e0c88c19a35ef2779"}]},{"id":"S009","semantic_type":"proof_step","kind":"proof_step","title":"聚点满足驻点条件","source_units":[{"start":95943,"end":96170,"start_line":2439,"end_line":2445,"raw_tex":"\t\t\tFrom \\eqref{eq:prox:subdiff} we then have that\n\t\t\t\\(\n\t\t\t\t-\\nabla\\sDRS(u')\n\t\t\t{}\\in{}\n\t\t\t\t\\hat\\partial\\nsDRS(u')\n\t\t\t\\),\n\t\t\thence \\(0\\in\\hat\\partial\\varphi(u')\\), as it follows from \\cite[Ex. 8.8]{rockafellar2011variational}.\n","sha256_utf8":"d8ebde20e1e1372fea1bdbbf05a93242ebee7d5563538f98829970cd311a2907"}]},{"id":"S010","semantic_type":"proof_step","kind":"proof_step","title":"夹逼确定所有聚点目标值","source_units":[{"start":96170,"end":96915,"start_line":2446,"end_line":2467,"raw_tex":"\t\t\tFinally, since \\(v^k\\to u'\\),\n\t\t\t\\[\n\t\t\t\t\\varphi(u')\n\t\t\t{}\\leq{}\n\t\t\t\t\\liminf_{K\\ni k\\to\\infty}{\n\t\t\t\t\t\\varphi(v^k)\n\t\t\t\t}\n\t\t\t{}\\leq{}\n\t\t\t\t\\limsup_{K\\ni k\\to\\infty}{\n\t\t\t\t\t\\varphi(v^k)\n\t\t\t\t}\n\t\t\t{}\\leq{}\n\t\t\t\t\\limsup_{K\\ni k\\to\\infty}{\n\t\t\t\t\t\\DRE(s^k)\n\t\t\t\t}\n\t\t\t{}={}\n\t\t\t\t\\DRE(s')\n\t\t\t{}\\leq{}\n\t\t\t\t\\varphi(u'),\n\t\t\t\\]\n\t\t\twhere the first inequality is due to lower semicontinuity of \\(\\varphi\\), the third and the last to the sandwiching property (\\cref{thm:sandwich}), and the equality to the continuity of \\(\\DRE\\) (\\cref{thm:C0}).\n\t\t\tThis shows that \\(\\seq{\\varphi(u^k)}[k\\in K]\\to\\varphi(u')=\\DRE(s')\\), and since \\(\\seq{\\DRE(s^k)}\\to\\varphi_\\star\\), then necessarily \\(\\varphi(u')=\\DRE(s')=\\varphi_\\star\\) independently of the cluster point \\(u'\\).\n","sha256_utf8":"674faf8d857cef60bd6dc153b366c159ba43c258d1aea57099e2c25de459c39c"}]},{"id":"S011","semantic_type":"proof_step","kind":"proof_step","title":"近端最优性给出驻点残差界","source_units":[{"start":96915,"end":97219,"start_line":2468,"end_line":2476,"raw_tex":"\t\t\tThe last assert follows from the optimality condition of \\(v^k\\) as in \\eqref{eq:vFBS}, namely,\n\t\t\t\\(\n\t\t\t\t\\tfrac1\\gamma(u^k-v^k)\n\t\t\t\t{}-{}\n\t\t\t\t\\nabla\\sDRS(u^k)\n\t\t\t{}\\in{}\n\t\t\t\t\\hat\\partial\\nsDRS(v^k)\n\t\t\t\\)\n\t\t\tdue to \\eqref{eq:prox:subdiff}, together with \\(\\sigma_{\\sDRS}\\)-hypoconvexity of \\(\\sDRS\\).\n","sha256_utf8":"5ead196043039bda646ee2680f9d68be30303b52106c8288cfceea3f7248ec69"}]},{"id":"S012","semantic_type":"proof_step","kind":"proof_step","title":"水平集有界推出迭代有界","source_units":[{"start":97219,"end":97751,"start_line":2477,"end_line":2490,"raw_tex":"\t\t\\item\\ref{thm:DRS:bounded}.\n\t\t\tSuppose that \\(\\varphi\\) has bounded level sets.\n\t\t\tThen, it follows from \\cref{thm:boundedlev} that so does \\(\\DRE\\), and since\n\t\t\t\\(\n\t\t\t\ts^k\n\t\t\t{}\\in{}\n\t\t\t\t\\lev_{\\leq\\DRE(s^0)}\\DRE\n\t\t\t\\)\n\t\t\tfor all \\(k\\in\\N\\), then the sequence \\(\\seq{s^k}\\) is bounded.\n\t\t\tDue to Lipschitz continuity of \\(\\prox_{\\gamma\\sDRS}\\) (cf. \\cref{thm:proxfLip}), also \\(\\seq{u^k}\\) is bounded.\n\t\t\tIn turn, since \\(v^k-u^k\\to 0\\) we conclude that also \\(\\seq{v^k}\\) is bounded.\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}","sha256_utf8":"8f2b41f1cd50529d82d1385115aef3edc78471b86e80b4bbd6f515b70a18b6d8"}]},{"id":"S013","semantic_type":"problem_setting","kind":"problem_setting","title":"两函数复合最小化问题","source_units":[{"start":4734,"end":4835,"start_line":117,"end_line":119,"raw_tex":"\\begin{equation}\\label{eq:P}\n\t\\minimize_{s\\in\\R^p} \\varphi(s)\\equiv\\sDRS(s)+\\nsDRS(s).\n\\end{equation}","sha256_utf8":"70721885044c0851f87e8df23eb94f7efd6499744c02415ba663ebff5e4bedba"},{"start":4836,"end":5019,"start_line":120,"end_line":120,"raw_tex":"Here, \\(\\func{\\sDRS,\\nsDRS}{\\R^p}{\\Rinf}\\) are proper, lower semicontinuous (lsc), extended-real-valued functions (\\(\\Rinf\\coloneqq\\R\\cup\\set\\infty\\) denotes the extended-real line).\n","sha256_utf8":"779f3bc6060dc1f97c2c50268dbd57bc1d4fe5f4467b59838a9150e9693420a5"}]},{"id":"S014","semantic_type":"algorithm","kind":"algorithm","title":"带松弛参数的DRS迭代","source_units":[{"start":5019,"end":5427,"start_line":121,"end_line":138,"raw_tex":"Starting from some \\(s\\in\\R^p\\), one DR-iteration applied to \\eqref{eq:P} with \\emph{stepsize} \\(\\gamma>0\\) and \\emph{relaxation} parameter \\(\\lambda>0\\) amounts to\n\\[\n\t\\tag{DRS}\\label{DRS}\n% \t\\renewcommand\\arraystretch{1.5}\n\t\\begin{cases}[rl]\n\t\tu\\phantom{^+}\n\t{}\\in{} &\n\t\t\\prox_{\\gamma\\sDRS}(s)\n\t\\\\\n\t\tv\\phantom{^+}\n\t{}\\in{} &\n\t\t\\prox_{\\gamma\\nsDRS}(2u-s)\n\t\\\\\n\t\ts^+\n\t{}={} &\n\t\ts+\\lambda(v-u).\n\t\\end{cases}\n\\]","sha256_utf8":"12b5b1c9a91adf170198722de507f98e8b161d0479d428850f6e173daab8ccee"}]},{"id":"S015","semantic_type":"notation","kind":"notation","title":"扩展实数与正负部约定","source_units":[{"start":33731,"end":34034,"start_line":479,"end_line":493,"raw_tex":"\t\t\tThe extended-real line is \\(\\Rinf=\\R\\cup\\set{\\infty}\\).\nThe positive and negative parts of \\(r\\in\\R\\) are defined respectively as\n\\(\n\t[r]_+\n{}\\coloneqq{}\n\t\\max\\set{0,r}\n\\)\nand\n\\(\n\t[r]_-\n{}\\coloneqq{}\n\t\\max\\set{0,-r}\n\\),\nso that \\(r=[r]_+-[r]_-\\).\nWe adopt the convention that \\(\\nicefrac10=\\infty\\).\n","sha256_utf8":"31933ae8c30d33ebcdb903c6cbc4c325eb57d72d5442cd1fc80ef8bd8083ed4e"}]},{"id":"S016","semantic_type":"definition","kind":"definition","title":"可和与平方可和序列","source_units":[{"start":34591,"end":34874,"start_line":500,"end_line":501,"raw_tex":"For a set \\(E\\) and a sequence \\(\\seq{x^k}\\) we write \\(\\seq{x^k}\\subset E\\) to indicate that \\(x^k\\in E\\) for all \\(k\\in\\N\\).\nWe say that \\(\\seq{x^k}\\subset\\R^n\\) is \\DEF{summable} if \\(\\sum_{k\\in\\N}\\|x^k\\|\\) is finite, and \\DEF{square-summable} if \\(\\seq{\\|x^k\\|^2}\\) is summable.\n","sha256_utf8":"fc009399c22424cb0ccf7c5deb83e51d512d5248ae2e29c6497638a6e9bfc5fe"}]},{"id":"S017","semantic_type":"definition","kind":"definition","title":"函数定义域与水平有界性","source_units":[{"start":34976,"end":35670,"start_line":505,"end_line":530,"raw_tex":"The \\DEF{domain} of an extended-real-valued function \\(\\func{h}{\\R^n}{\\Rinf}\\) is the set\n\\(\n\t\\dom h\n{}\\coloneqq{}\n\t\\set{x\\in\\R^n}[\n\t\th(x)<\\infty\n\t]\n\\),\nwhile its \\DEF{epigraph} is the set\n\\(\n\t\\epi h\n{}\\coloneqq{}\n\t\\set{(x,\\alpha)\\in\\R^n\\times\\R}[\n\t\th(x)\\leq\\alpha\n\t]\n\\).\n\\(h\\) is said to be \\DEF{proper} if \\(\\dom h\\neq\\emptyset\\), and \\DEF{lower semicontinuous (lsc)} if \\(\\epi h\\) is a closed subset of \\(\\R^{n+1}\\).\nFor \\(\\alpha\\in\\R\\), \\(\\lev_{\\leq\\alpha}h\\) is the \\DEF{\\(\\alpha\\)-level set} of \\(h\\), \\ie\n\\(\n\t\\lev_{\\leq\\alpha}h\n{}\\coloneqq{}\n\t\\set{x\\in\\R^n}[\n\t\th(x)\\leq\\alpha\n\t]\n\\).\nWe say that \\(h\\) is \\emph{level bounded} if \\(\\lev_{\\leq\\alpha}h\\) is bounded for all \\(\\alpha\\in\\R\\).\n","sha256_utf8":"1c28b3b97b790e88faea634beb509444b69e0827ce55b39e339ded7577f15d22"}]},{"id":"S018","semantic_type":"definition","kind":"definition","title":"集合映射与次微分","source_units":[{"start":36182,"end":37356,"start_line":538,"end_line":570,"raw_tex":"We use the notation \\(\\ffunc H{\\R^n}{\\R^m}\\) to indicate a point-to-set mapping \\(\\func H{\\R^n}{\\mathcal P(\\R^m)}\\), where \\(\\mathcal P(\\R^m)\\) is the power set of \\(\\R^m\\) (the set of all subsets of \\(\\R^m\\)).\nThe \\DEF{graph} of \\(H\\) is the set\n\\(\n\t\\graph H\n{}\\coloneqq{}\n\t\\set{(x,y)\\in\\R^n\\times\\R^m}[\n\t\ty\\in H(x)\n\t]\n\\).\n \n% =============================================================================================== %\n\nWe denote by \\(\\ffunc{\\hat\\partial h}{\\R^n}{\\R^n}\\) the \\DEF{regular subdifferential} of \\(h\\), where\n\\begin{equation}\\label{eq:hatpartial}\n\tv\\in\\hat\\partial h(\\bar x)\n\\quad\\Leftrightarrow\\quad\n\t\\liminf_{\\limsubstack{x&\\to&\\bar x\\\\x&\\neq&\\bar x}}{\n\t\t\\frac{h(x)-h(\\bar x)-\\innprod{v}{x-\\bar x}}{\\|x-\\bar x\\|}\n\t}\n{}\\geq{}\n\t0.\n\\end{equation}\nA necessary condition for local minimality of \\(x\\) for \\(h\\) is \\(0\\in\\hat\\partial h(x)\\), see \\cite[Thm. 10.1]{rockafellar2011variational}.\nThe (limiting) \\DEF{subdifferential} of \\(h\\) is \\(\\ffunc{\\partial h}{\\R^n}{\\R^n}\\), where\n\\(\n\tv\\in\\partial h(\\bar x)\n\\)\niff there exists a sequence \\(\\seq{x^k,v^k}\\subseteq\\graph\\hat\\partial h\\) such that\n\\[\n\t\\lim_{k\\to\\infty}(x^k,h(x^k),v^k)\n{}={}\n\t(x,h(x),v).\n\\]\n","sha256_utf8":"f66adedf0c4f5b123ae5513d37148e32f8907c27d2cf32f915c014f9a44f3619"}]},{"id":"S019","semantic_type":"definition","kind":"definition","title":"光滑性与弱凸性及基本界","source_units":[{"start":37661,"end":39280,"start_line":576,"end_line":612,"raw_tex":"\t\t\tThe class of functions \\(\\func{h}{\\R^n}{\\R}\\) that are \\(k\\) times continuously differentiable is denoted as \\(\\cont^k(\\R^n)\\).\nWe write \\(h\\in\\cont^{1,1}(\\R^n)\\) to indicate that \\(h\\in\\cont^1(\\R^n)\\) and that \\(\\nabla h\\) is Lipschitz continuous with modulus \\(L_h\\).\nTo simplify the terminology, we will say that such an \\(h\\) is \\DEF{\\(L_h\\)-smooth}.\nIt follows from \\cite[Prop. A.24]{bertsekas2016nonlinear} that if \\(h\\) is \\(L_h\\)-smooth, then\n\\(\n\t|h(y)-h(x)-\\innprod{\\nabla h(x)}{y-x}|\n{}\\leq{}\n\t\\tfrac{L_h}{2}\\|y-x\\|^2\n\\)\nfor all \\(x,y\\in\\R^n\\).\nIn particular, there exists \\(\\sigma_h\\in[-L_h,L_h]\\) such that \\(h\\) is \\DEF{\\(\\sigma_h\\)-hypoconvex}, in the sense that \\(h-\\tfrac{\\sigma_h}{2}\\|{}\\cdot{}\\|^2\\) is a convex function.\nThus, every \\(L_h\\)-smooth and \\(\\sigma_h\\)-hypoconvex function \\(h\\) satisfies\n\\begin{equation}\\label{eq:LipBound}\n\t\\tfrac{\\sigma_h}{2}\n\t\\|y-x\\|^2\n{}\\leq{}\n\th(y)-h(x)-\\innprod{\\nabla h(x)}{y-x}\n{}\\leq{}\n\t\\tfrac{L_h}{2}\n\t\\|y-x\\|^2\n\\quad\n\t\\forall x,y\\in\\R^n.\n\\end{equation}\nBy applying \\cite[Thm. 2.1.5]{nesterov2003introductory} to the (convex) function \\(\\psi=h-\\tfrac\\sigma2\\|{}\\cdot{}\\|^2\\) we obtain that this is equivalent to having\n\\begin{equation}\\label{eq:innprod}\n\t\\sigma_h\n\t\\|y-x\\|^2\n{}\\leq{}\n\t\\innprod{\\nabla h(y)-\\nabla h(x)}{y-x}\n{}\\leq{}\n\tL_h\n\t\\|y-x\\|^2\n\\quad\n\t\\forall x,y\\in\\R^n.\n\\end{equation}\nNote that \\(\\sigma\\)-hypoconvexity generalizes the notion of (strong) convexity by allowing negative strong convexity moduli.\nIn fact, if \\(\\sigma=0\\) then \\(\\sigma\\)-hypoconvexity reduces to convexity, while for \\(\\sigma>0\\) it denotes \\(\\sigma\\)-strong convexity.\n","sha256_utf8":"0b2187e92058ee778e819841cf31d3dccc2ea49daef1c63d691b77957fb1180b"}]},{"id":"S020","semantic_type":"definition","kind":"definition","title":"近端映射与Moreau包络","source_units":[{"start":42319,"end":44027,"start_line":725,"end_line":764,"raw_tex":"\t\t\tThe \\DEF{proximal mapping} of \\(\\func{h}{\\R^n}{\\Rinf}\\) with parameter \\(\\gamma>0\\) is \\(\\ffunc{\\prox_{\\gamma h}}{\\R^n}{\\dom h}\\) defined as\n\\begin{equation}\n\t\\prox_{\\gamma h}(x)\n{}\\coloneqq{}\n\t\\argmin_{w\\in\\R^n}\\set{\n\t\th(w) + \\tfrac{1}{2\\gamma}\\|w-x\\|^2\n\t}.\n\\end{equation}\nWe say that a function \\(h\\) is \\DEF{prox-bounded} if \\(h+\\tfrac{1}{2\\gamma}\\|{}\\cdot{}\\|^2\\) is lower bounded for some \\(\\gamma>0\\).\nThe supremum of all such \\(\\gamma\\)\n% --- which is possibly infinite, as it is the case when \\(h\\) is lower bounded or convex ---\nis the \\emph{threshold of prox-boundedness of \\(h\\)}, denoted as \\(\\gamma_h\\).\nIf \\(h\\) is lsc, then \\(\\prox_{\\gamma h}\\) is nonempty- and compact-valued over \\(\\R^n\\) for all \\(\\gamma\\in(0,\\gamma_h)\\) \\cite[Thm. 1.25]{rockafellar2011variational}.\nConsequently, the value function of the minimization problem defining the proximal mapping, namely the \\DEF{Moreau envelope} with stepsize \\(\\gamma\\in(0,\\gamma_h)\\), denoted by \\(\\func{h^\\gamma}{\\R^n}{\\R}\\) and defined as\n\\begin{equation}\n\t\\fillwidthof[r]{\n\t\t\\prox_{\\gamma h}(x)\n\t}{\n\t\th^\\gamma(x)\n\t}\n{}\\coloneqq{}\n\t\\fillwidthof[l]{\n\t\t\\argmin_{w\\in\\R^n}\\set{\n\t\t\th(w) + \\tfrac{1}{2\\gamma}\\|w-x\\|^2\n\t\t}.\n\t}{\n\t\t\\inf_{w\\in\\R^n}\\set{\n\t\t\th(w) + \\tfrac{1}{2\\gamma}\\|w-x\\|^2\n\t\t},\n\t}\n\\end{equation}\nis everywhere finite and, in fact, strictly continuous \\cite[Ex. 10.32]{rockafellar2011variational}.\nMoreover, the necessary optimality conditions of the problem defining \\(\\prox_{\\gamma g}\\) together with \\cite[Thm. 10.1 and Ex. 8.8]{rockafellar2011variational} imply that\n\\begin{equation}\\label{eq:prox:subdiff}\n\t\\tfrac1\\gamma(x-\\bar x)\n{}\\in{}\n\t\\hat\\partial h(\\bar x)\n\\quad\n\t\\forall\\bar x\\in\\prox_{\\gamma h}(x).\n\\end{equation}\n","sha256_utf8":"0c87c508d90114d99417b88c65e5bef55e39566223ce33ba66fed73d115e650c"}]},{"id":"S021","semantic_type":"assumption","kind":"assumption","title":"DRS的基本假设","source_units":[{"start":46558,"end":47038,"start_line":843,"end_line":853,"raw_tex":"\\begin{ass}[Requirements for the DRS formulation \\eqref{eq:P}]\\label{ass:DRS}%\n\tThe following hold\n\t\\begin{enumerate}\n\t\\item\\label{ass:DRS:f}%\n\t\t\\(\\sDRS\\in\\cont^{1,1}(\\R^n)\\) is \\(L_{\\sDRS}\\)-smooth, hence \\(\\sigma_{\\sDRS}\\)-hypoconvex for some \\(\\sigma_{\\sDRS}\\in[-L_{\\sDRS},L_{\\sDRS}]\\).\n\t\\item\\label{ass:DRS:g}%\n\t\t\\(\\nsDRS\\) is proper and lsc.\n\t\\item\\label{ass:DRS:LB}%\n\t\tProblem \\eqref{eq:P} has a solution, that is, \\(\\argmin\\varphi\\neq\\emptyset\\).\n\t\\end{enumerate}\n\\end{ass}","sha256_utf8":"f589fe62d24de882b197f676d47c3595d0bc2cfa5dd9f80ca0cba83d8837e020"}]},{"id":"S022","semantic_type":"derived_fact","kind":"derived_fact","title":"可行步长保证迭代有定义","source_units":[{"start":47039,"end":47338,"start_line":854,"end_line":855,"raw_tex":"\\begin{rem}[Feasible stepsizes for \\ref{DRS}]\\label{thm:proxbounded}%\n\tUnder \\Cref{ass:DRS}, both \\(\\sDRS\\) and \\(\\nsDRS\\) are prox-bounded with threshold at least \\(\\nicefrac{1}{L_{\\sDRS}}\\), and in particular \\ref{DRS} iterations are well defined for all \\(\\gamma\\in(0,\\nicefrac{1}{L_{\\sDRS}})\\).\n","sha256_utf8":"f0e61e49f780fbe674f443f86f92a60c1ed818a005cd0d0ac23cfa817166aa24"}]},{"id":"S023","semantic_type":"definition","kind":"definition","title":"前向后向步与DR包络","source_units":[{"start":48777,"end":50148,"start_line":879,"end_line":918,"raw_tex":"Starting from \\(s\\in\\R^p\\), let us consider variables \\((u,v)\\) generated by a \\ref{DRS} step under \\Cref{ass:DRS}.\nAs first noted in \\cite{patrinos2014douglas}, from the relation \\(s=u+\\gamma\\nabla\\sDRS(u)\\) (see \\cref{thm:proxfEquiv}) it follows that\n\\begin{equation}\\label{eq:vFBS}\n\tv\\in\\FB u\n\\end{equation}\nis the result of a \\DEF{forward-backward step} at \\(u\\), amounting to\n\\begin{align}\\label{eq:DRSv}\n\tv\n{}\\in{}\n\t\\argmin_{w\\in\\R^p}{} &\n\t\\set{\n\t\t\\nsDRS(w)\n\t\t{}+{}\n\t\t\\smashunderbracket{\n\t\t\t\\sDRS(u)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla\\sDRS(u)}{w-u}\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2\\gamma}\\|w-u\\|^2\n\t\t}{}\n\t},\n\\shortintertext{%\n\tsee \\eg \\cite{bolte2014proximal,themelis2018forward} for an extensive discussion on nonconvex forward-backward splitting (FBS).\n\tThis shows that \\(v\\) is the result of the minimization of a \\emph{majorization model} for the original function \\(\\varphi=\\sDRS+\\nsDRS\\), where the smooth function \\(\\sDRS\\) is replaced by the quadratic upper bound emphasized by the under-bracket in \\eqref{eq:DRSv}.\n\tFirst introduced in \\cite{patrinos2014douglas} for convex problems, the \\DEF{Douglas-Rachford envelope} (DRE) is the function \\(\\func{\\DRE}{\\R^n}{\\R}\\) defined as%\n}\n\\label{eq:DRE}\n\t\\DRE(s)\n{}\\coloneqq{}\n\t\\min_{w\\in\\R^p}{} &\n\t\\set{\n\t\t\\nsDRS(w)\n\t\t{}+{}\n\t\t\\sDRS(u)\n\t\t{}+{}\n\t\t\\innprod{\\nabla\\sDRS(u)}{w-u}\n\t\t{}+{}\n\t\t\\tfrac{1}{2\\gamma}\\|w-u\\|^2\n\t}.\n\\end{align}","sha256_utf8":"f2e3eed690dbe8488e91b5b5ea581f7027a2d49df89aea9dd0e97e387f5129ba"},{"start":50149,"end":50315,"start_line":919,"end_line":919,"raw_tex":"Namely, rather than the \\emph{minimizer} \\(v\\), \\(\\DRE(s)\\) is the \\emph{value} of the minimization problem \\eqref{eq:DRSv} defining the \\(v\\)-update in \\eqref{DRS}.\n","sha256_utf8":"b2fdd9d990cc1092918a76b1bb58651a115702d94d0de8ccf7130277b523d8dc"}]},{"id":"S024","semantic_type":"derived_fact","kind":"derived_fact","title":"DR包络的复合表达式","source_units":[{"start":50315,"end":50626,"start_line":920,"end_line":927,"raw_tex":"The expression \\eqref{eq:DRE} emphasizes the close connection that the DRE has with the \\DEF{forward-backward envelope} (FBE) as in \\cite{themelis2018forward}, here denoted \\(\\FBE\\), namely\n\\begin{equation}\\label{eq:FBEu}\n\t\\DRE(s)\n{}={}\n\t\\FBE(u),\n\\quad\n\t\\text{where \\(u=\\prox_{\\gamma\\sDRS}(s)\\).}\n\\end{equation}","sha256_utf8":"2c8f65a349e5463b6cec5e85ffe26134d6d46581e0a06165c8190703877452c7"}]},{"id":"S025","semantic_type":"theorem","kind":"theorem","title":"光滑函数的两种下界","source_units":[{"start":41391,"end":42199,"start_line":678,"end_line":717,"raw_tex":"\\begin{thm}[Lower bounds for smooth functions]\\label{thm:rho}%\n\tLet \\(h\\in C^{1,1}(\\R^n)\\) be \\(L_h\\)-smooth and \\(\\sigma_h\\)-hypoconvex.\n\tThen, for all \\(x,y\\in\\R^n\\) it holds that\n\t\\[\n\t\th(y)\n\t{}\\geq{}\n\t\th(x)\n\t\t{}+{}\n\t\t\\innprod{\\nabla h(x)}{y-x}\n\t\t{}+{}\n\t\t\\rho(y,x),\n\t\\]\n\twhere\n\t\\begin{enumerate}\n\t\\item\\label{thm:hypo:rho0}%\n\t\teither~\n\t\t\\(\n\t\t\t\\rho(y,x)\n\t\t{}={}\n\t\t\t\\tfrac{\\sigma_h}{2}\n\t\t\t\\|y-x\\|^2\n\t\t\\),\n\t\\item\\label{thm:hypo:rho1}%\n\t\tor~\n\t\t\\(\n\t\t\t\\rho(y,x)\n\t\t{}={}\n\t\t\t\\tfrac{\\sigma_hL_h}{2(L_h+\\sigma_h)}\n\t\t\t\\|y-x\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2(L_h+\\sigma_h)}\n\t\t\t\\|\\nabla h(y)-\\nabla h(x)\\|^2\n\t\t\\),~\n\t\tprovided that~\n\t\t\\(\n\t\t\t-L_h<\\sigma_h\\leq 0\n\t\t\\).\n\t\\end{enumerate}\n\tClearly, all inequalities remain valid if one replaces \\(L_h\\) with any \\(L\\geq L_h\\) and \\(\\sigma_h\\) with any \\(\\sigma\\in[-L,\\sigma_h]\\).\n\t","sha256_utf8":"a68efa22ecf970a865a239ad55f371e2b631d2624971c4546e3814b34d9463a8"},{"start":42253,"end":42263,"start_line":719,"end_line":720,"raw_tex":"\n\\end{thm}","sha256_utf8":"27661e277ceb78e4dd676b4ea4b5c8d9a4d3c607a1a758eac095282a4f311f77"}]},{"id":"S026","semantic_type":"proof","kind":"proof","title":"光滑函数下界完整证明","source_units":[{"start":42199,"end":42253,"start_line":717,"end_line":719,"raw_tex":"\\begin{proof}\n\t\tSee \\Cref{proof:thm:rho}.\n\t\\end{proof}","sha256_utf8":"b37111579588e7a16875f9b03217c628101601d204f83d722d4d9138724ea60b"},{"start":199331,"end":201623,"start_line":5441,"end_line":5533,"raw_tex":"\\begin{appendixproof}{thm:rho}%\n\t\\begin{proofitemize}\n\t\\item\\ref{thm:hypo:rho0}.\n\t\tThis is the lower bound in \\eqref{eq:LipBound}.\n\t\\item\\ref{thm:hypo:rho1}.\n\t\tLet \\(L\\geq L_h\\) and \\(\\sigma\\in(-L,\\min\\set{0,\\sigma_h}]\\) be fixed.\n\t\tThen, \\(h\\) is \\(L\\)-smooth and \\(\\sigma\\)-hypoconvex, and from \\cite[Thm. 2.1.12]{nesterov2003introductory} we obtain that\n\t\t\\begin{equation}\\label{eq:LipBound:mix}\n\t\t\t\\innprod{\n\t\t\t\t\\nabla h(y)-\\nabla h(x)\n\t\t\t}{\n\t\t\t\ty-x\n\t\t\t}\n\t\t{}\\geq{}\n\t\t\t\\tfrac{\\sigma L}{L+\\sigma}\\|x-y\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{L+\\sigma}\\|\\nabla h(x)-\\nabla h(y)\\|^2\n\t\t\\end{equation}\n\t\tfor all \\(x,y\\in\\R^n\\).\n\t\t(Although \\cite[Thm. 2.1.12]{nesterov2003introductory} assumes \\(\\sigma>0\\), the given proof does not necessitate this restriction).\n\t\tMoreover, \\(\\psi\\coloneqq h-\\tfrac\\sigma2\\|{}\\cdot{}\\|^2\\) is convex and \\(L_\\psi\\)-smooth, with \\(L_\\psi=L-\\sigma\\).\n\t\tConsequently, for all \\(x,y\\in\\R^n\\) one has\n\t\t\\(\n\t\t\t\\psi(y)\n\t\t{}\\geq{}\n\t\t\t\\psi(x)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla\\psi(x)}{y-x}\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2L_\\psi}\\|\\nabla\\psi(y)-\\nabla\\psi(x)\\|^2\n\t\t\\),\n\t\tsee \\cite[Thm. 2.1.5]{nesterov2003introductory}, resulting in\n\t\t\\begin{align*}\n\t\t\th(y)\n\t\t{}\\geq{} &\n\t\t\th(x)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla h(x)}{y-x}\n\t\t\t{}+{}\n\t\t\t\\tfrac{\\sigma L}{2(L-\\sigma)}\n\t\t\t\\|y-x\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2(L-\\sigma)}\n\t\t\t\\|\\nabla h(y)-\\nabla h(x)\\|^2\n\t\t\\\\\n\t\t&\n\t\t\t{}-{}\n\t\t\t\\tfrac{\\sigma}{2(L-\\sigma)}\n\t\t\t\\innprod{\n\t\t\t\t\\nabla h(y)-\\nabla h(x)\n\t\t\t}{\n\t\t\t\ty-x\n\t\t\t}.\n\t\t\\end{align*}\n\t\tSince \\(\\sigma\\leq0\\), the coefficient of the scalar product in the second line is positive.\n\t\tWe may thus invoke the inequality \\eqref{eq:LipBound:mix} to arrive to\n\t\t\\begin{align*}\n\t\t\th(y)\n\t\t{}\\geq{} &\n\t\t\th(x)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla h(x)}{y-x}\n\t\t\t{}+{}\n\t\t\t\\tfrac{\\sigma L}{2(L-\\sigma)}\n\t\t\t\\|y-x\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2(L-\\sigma)}\n\t\t\t\\|\\nabla h(y)-\\nabla h(x)\\|^2\n\t\t\\\\\n\t\t&\n\t\t\t{}-{}\n\t\t\t\\tfrac{\\sigma}{2(L-\\sigma)}\n\t\t\t\\left[\n\t\t\t\t\\tfrac{\\sigma L}{L+\\sigma}\\|x-y\\|^2\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{L+\\sigma}\\|\\nabla h(x)-\\nabla h(y)\\|^2\n\t\t\t\\right]\n\t\t\\\\\n\t\t{}={} &\n\t\t\th(x)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla h(x)}{y-x}\n\t\t\t{}+{}\n\t\t\t\\tfrac{\\sigma L}{2(L+\\sigma)}\n\t\t\t\\|y-x\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2(L+\\sigma)}\n\t\t\t\\|\\nabla h(y)-\\nabla h(x)\\|^2,\n\t\t\\end{align*}\n\t\thence the claimed inequality.\n\t\\qedhere\n\t\\end{proofitemize}\n\\end{appendixproof}","sha256_utf8":"55109ad0b6fa2d44137d298b88b0669d0a5ccdb438f95a0b7a81edd06480e847"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"下界证明的附录指引","source_units":[{"start":42199,"end":42253,"start_line":717,"end_line":719,"raw_tex":"\\begin{proof}\n\t\tSee \\Cref{proof:thm:rho}.\n\t\\end{proof}","sha256_utf8":"b37111579588e7a16875f9b03217c628101601d204f83d722d4d9138724ea60b"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"第一下界由弱凸性得到","source_units":[{"start":199331,"end":199462,"start_line":5441,"end_line":5444,"raw_tex":"\\begin{appendixproof}{thm:rho}%\n\t\\begin{proofitemize}\n\t\\item\\ref{thm:hypo:rho0}.\n\t\tThis is the lower bound in \\eqref{eq:LipBound}.\n","sha256_utf8":"0d78b1d02c86b973cf65db9032b0160a56dd301bedbd632ecb41fab4b7cb86a7"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"引入混合梯度不等式","source_units":[{"start":199462,"end":200079,"start_line":5445,"end_line":5460,"raw_tex":"\t\\item\\ref{thm:hypo:rho1}.\n\t\tLet \\(L\\geq L_h\\) and \\(\\sigma\\in(-L,\\min\\set{0,\\sigma_h}]\\) be fixed.\n\t\tThen, \\(h\\) is \\(L\\)-smooth and \\(\\sigma\\)-hypoconvex, and from \\cite[Thm. 2.1.12]{nesterov2003introductory} we obtain that\n\t\t\\begin{equation}\\label{eq:LipBound:mix}\n\t\t\t\\innprod{\n\t\t\t\t\\nabla h(y)-\\nabla h(x)\n\t\t\t}{\n\t\t\t\ty-x\n\t\t\t}\n\t\t{}\\geq{}\n\t\t\t\\tfrac{\\sigma L}{L+\\sigma}\\|x-y\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{L+\\sigma}\\|\\nabla h(x)-\\nabla h(y)\\|^2\n\t\t\\end{equation}\n\t\tfor all \\(x,y\\in\\R^n\\).\n\t\t(Although \\cite[Thm. 2.1.12]{nesterov2003introductory} assumes \\(\\sigma>0\\), the given proof does not necessitate this restriction).\n","sha256_utf8":"77cb4a315686561b00625e9810db1dc6e9255da26b7588e859f4da60d51be20a"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"凸化函数并展开光滑下界","source_units":[{"start":200079,"end":200792,"start_line":5461,"end_line":5494,"raw_tex":"\t\tMoreover, \\(\\psi\\coloneqq h-\\tfrac\\sigma2\\|{}\\cdot{}\\|^2\\) is convex and \\(L_\\psi\\)-smooth, with \\(L_\\psi=L-\\sigma\\).\n\t\tConsequently, for all \\(x,y\\in\\R^n\\) one has\n\t\t\\(\n\t\t\t\\psi(y)\n\t\t{}\\geq{}\n\t\t\t\\psi(x)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla\\psi(x)}{y-x}\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2L_\\psi}\\|\\nabla\\psi(y)-\\nabla\\psi(x)\\|^2\n\t\t\\),\n\t\tsee \\cite[Thm. 2.1.5]{nesterov2003introductory}, resulting in\n\t\t\\begin{align*}\n\t\t\th(y)\n\t\t{}\\geq{} &\n\t\t\th(x)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla h(x)}{y-x}\n\t\t\t{}+{}\n\t\t\t\\tfrac{\\sigma L}{2(L-\\sigma)}\n\t\t\t\\|y-x\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2(L-\\sigma)}\n\t\t\t\\|\\nabla h(y)-\\nabla h(x)\\|^2\n\t\t\\\\\n\t\t&\n\t\t\t{}-{}\n\t\t\t\\tfrac{\\sigma}{2(L-\\sigma)}\n\t\t\t\\innprod{\n\t\t\t\t\\nabla h(y)-\\nabla h(x)\n\t\t\t}{\n\t\t\t\ty-x\n\t\t\t}.\n\t\t\\end{align*}\n","sha256_utf8":"6323b4cbbca0d73d83257b8c8c54e8447f668d5059c8a70bd5066cdb51715b5b"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"代入混合界完成第二下界","source_units":[{"start":200792,"end":201623,"start_line":5495,"end_line":5533,"raw_tex":"\t\tSince \\(\\sigma\\leq0\\), the coefficient of the scalar product in the second line is positive.\n\t\tWe may thus invoke the inequality \\eqref{eq:LipBound:mix} to arrive to\n\t\t\\begin{align*}\n\t\t\th(y)\n\t\t{}\\geq{} &\n\t\t\th(x)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla h(x)}{y-x}\n\t\t\t{}+{}\n\t\t\t\\tfrac{\\sigma L}{2(L-\\sigma)}\n\t\t\t\\|y-x\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2(L-\\sigma)}\n\t\t\t\\|\\nabla h(y)-\\nabla h(x)\\|^2\n\t\t\\\\\n\t\t&\n\t\t\t{}-{}\n\t\t\t\\tfrac{\\sigma}{2(L-\\sigma)}\n\t\t\t\\left[\n\t\t\t\t\\tfrac{\\sigma L}{L+\\sigma}\\|x-y\\|^2\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{L+\\sigma}\\|\\nabla h(x)-\\nabla h(y)\\|^2\n\t\t\t\\right]\n\t\t\\\\\n\t\t{}={} &\n\t\t\th(x)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla h(x)}{y-x}\n\t\t\t{}+{}\n\t\t\t\\tfrac{\\sigma L}{2(L+\\sigma)}\n\t\t\t\\|y-x\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2(L+\\sigma)}\n\t\t\t\\|\\nabla h(y)-\\nabla h(x)\\|^2,\n\t\t\\end{align*}\n\t\thence the claimed inequality.\n\t\\qedhere\n\t\\end{proofitemize}\n\\end{appendixproof}","sha256_utf8":"177c801d0bcd9ef50a0eae81cdbbc318e0736245b915954935269c5fca79654a"}]},{"id":"S032","semantic_type":"proposition","kind":"proposition","title":"光滑函数的近端性质","source_units":[{"start":44157,"end":46079,"start_line":766,"end_line":829,"raw_tex":"\\begin{prop}[Proximal properties of smooth functions]\\label{thm:proxf}%\n\tLet \\(h\\in \\cont^{1,1}(\\dom h)\\) be \\(L_h\\)-smooth, hence \\(\\sigma_h\\)-hypoconvex for some \\(\\sigma_h\\in[-L_h,L_h]\\).\n\tThen, \\(h\\) is prox-bounded with \\(\\gamma_{h}\\geq\\nicefrac{1}{[\\sigma_{h}]_-}\\) and for all \\(\\gamma<\\nicefrac{1}{[\\sigma_{h}]_-}\\) the following hold:\n\t\\begin{enumerate}\n\t\\item\\label{thm:proxfEquiv}\n\t\t\\(\\prox_{\\gamma h}\\) is single valued, and for all \\(s\\in\\R^n\\) it holds that \\(u=\\prox_{\\gamma h}(s)\\) iff\n\t\t\\(\n\t\t\ts=u+\\gamma\\nabla h(u)\n\t\t\\).\n\t\\item\\label{thm:proxfLip}\n\t\t\\(\\prox_{\\gamma h}\\) is \\((\\tfrac{1}{1+\\gamma L_{h}})\\)-strongly monotone and \\((1+\\gamma\\sigma_h)\\)-cocoercive, in the sense that\n\t\t\\[\n\t\t\t\\innprod{u-u'}{s-s'}\n\t\t{}\\geq{}\n\t\t\t\\tfrac{1}{1+\\gamma L_h}\n\t\t\t\\|s-s'\\|^2\n\t\t\\quad\\text{and}\\quad\n\t\t\t\\innprod{u-u'}{s-s'}\n\t\t{}\\geq{}\n\t\t\t(1+\\gamma\\sigma_h)\n\t\t\t\\|u-u'\\|^2\n\t\t\\]\n\t\tfor all \\(s,s'\\in\\R^n\\), where \\(u=\\prox_{\\gamma h}(s)\\) and \\(u'=\\prox_{\\gamma h}(s')\\).\n\t\tIn particular,\n\t\t\\begin{equation}\\label{eq:biLip}\n\t\t\t\\tfrac{1}{1+\\gamma L_h}\n\t\t\t\\|s-s'\\|\n\t\t{}\\leq{}\n\t\t\t\\|u-u'\\|\n\t\t{}\\leq{}\n\t\t\t\\tfrac{1}{1+\\gamma\\sigma_h}\n\t\t\t\\|s-s'\\|.\n\t\t\\end{equation}\n\t\tThus, \\(\\prox_{\\gamma h}\\) is a \\(\\tfrac{1}{1+\\gamma\\sigma_h}\\)-Lipschitz and invertible mapping, and its inverse \\(\\id+\\gamma\\nabla h\\) is \\((1+\\gamma L_h)\\)-Lipschitz continuous.\n\t\\item\\label{thm:MoreaufC1}\n\t\t\\(h^\\gamma\\in \\cont^{1,1}(\\R^n)\\) is \\(L_{h^\\gamma}\\)-smooth and \\(\\sigma_{h^\\gamma}\\)-hypoconvex, with\n\t\t\\(\n\t\t\tL_{h^\\gamma}\n\t\t{}={}\n\t\t\t\\max\\set{\n\t\t\t\t\\tfrac{L_{h}}{1+\\gamma L_{h}},~\n\t\t\t\t\\tfrac{[\\sigma_{h}]_-}{1+\\gamma\\sigma_{h}}\n\t\t\t}\n\t\t\\)\n\t\tand\n\t\t\\(\n\t\t\t\\sigma_{h^\\gamma}\n\t\t{}={}\n\t\t\t\\tfrac{\\sigma_{h}}{1+\\gamma\\sigma_{h}}\n\t\t\\).\n\t\tMoreover,\n\t\t\\(\n\t\t\t\\nabla h^\\gamma(s)\n\t\t{}={}\n\t\t\t\\tfrac{1}{\\gamma}(s-\\prox_{\\gamma h}(s))\n\t\t\\)\n\t\tand\n\t\t\\(\n\t\t\t\\nabla h(\\prox_{\\gamma h}(s))\n\t\t{}={}\n\t\t\t\\tfrac{1}{\\gamma}\\bigl(s-\\prox_{\\gamma h}(s)\\bigr)\n\t\t\\).\n\t\\end{enumerate}\n\t","sha256_utf8":"2ca6f2ee237d21f5b0eade954ca58dccb82a8bb300ae1332cbc89e635b25493f"},{"start":46135,"end":46146,"start_line":831,"end_line":832,"raw_tex":"\n\\end{prop}","sha256_utf8":"05d52a2387e348b9b8c6add2d64ac5b19b3c01cd0607238154a9f309962fa7aa"}]},{"id":"S033","semantic_type":"proof","kind":"proof","title":"近端性质完整证明","source_units":[{"start":46079,"end":46135,"start_line":829,"end_line":831,"raw_tex":"\\begin{proof}\n\t\tSee \\Cref{proof:thm:proxf}.\n\t\\end{proof}","sha256_utf8":"8b35bd666996567ea3e4da5e3104493ecfcc1e2c643d7e799e199392ecc2abef"},{"start":201627,"end":204771,"start_line":5537,"end_line":5625,"raw_tex":"\\begin{appendixproof}{thm:proxf}%\n\tLet \\(\\gamma\\in(0,\\nicefrac{1}{[\\sigma_h]_-})\\) be fixed, and let \\(\\psi\\coloneqq\\gamma h+\\tfrac12\\|{}\\cdot{}\\|^2\\).\n\tObserve that \\(\\psi\\in\\cont^{1,1}(\\R^n)\\) is \\(L_\\psi\\)-smooth and \\(\\sigma_\\psi\\)-strongly convex, with \\(L_\\psi=1+\\gamma L_h\\) and \\(\\sigma_\\psi=1+\\gamma\\sigma_h\\).\n\tIn particular, due to strong convexity \\(\\inf\\psi>-\\infty\\), and by definition of prox-boundedness it then follows that \\(\\gamma_h\\geq\\nicefrac{1}{[\\sigma_h]_-}\\).\n\t\\begin{proofitemize}\n\t\\item\\ref{thm:proxfEquiv}.\n\t\tFollows from \\eqref{eq:prox:subdiff}, by observing that \\(h+\\tfrac{1}{2\\gamma}\\|{}\\cdot{}-s\\|^2\\) is strongly convex, hence that a minimizer is characterized by stationarity.\n\t\\item\\ref{thm:proxfLip}.\n\t\tFor \\(s,s'\\in\\R^n\\), let \\(u=\\prox_{\\gamma h}(s)\\) and \\(u'=\\prox_{\\gamma h}(s')\\).\n\t\tThen,\n\t\t\\[\n\t\t\t\\innprod{s-s'}{u-u'}\n\t\t{}={}\n\t\t\t\\innprod{\n\t\t\t\t\\nabla\\psi(u)\n\t\t\t\t{}-{}\n\t\t\t\t\\nabla\\psi(u')\n\t\t\t}{u-u'}\n\t\t{}\\geq{}\n\t\t\t\\sigma_\\psi\n\t\t\t\\|u-u'\\|^2\n\t\t{}={}\n\t\t\t(1+\\gamma\\sigma_h)\n\t\t\t\\|u-u'\\|^2,\n\t\t\\]\n\t\twhere the first equality was shown in \\ref{thm:proxfEquiv} and the inequality follows from \\eqref{eq:innprod}.\n\t\tBy using the \\(\\frac{1}{L_\\psi}\\)-cocoercivity of \\(\\nabla\\psi\\) \\cite[Thm. 2.1.10]{nesterov2003introductory}, also the claimed strong monotonicity follows.\n\t\tIn turn, the Cauchy-Schwartz inequality on the inner product yields \\eqref{eq:biLip}.\n\t\\item\\ref{thm:MoreaufC1}.\n\t\tFrom \\cite[Ex. 10.32]{rockafellar2011variational} it follows that \\(h^\\gamma\\) is strictly continuous and that\n\t\t\\(\n\t\t\t\\partial h^\\gamma(s)\n\t\t{}\\subseteq{}\n\t\t\t\\tfrac1\\gamma(s-\\prox_{\\gamma h}(s))\n\t\t\\).\n\t\tBecause of single valuedness of \\(\\prox_{\\gamma h}\\), by invoking \\cite[Thm. 9.18]{rockafellar2011variational} we conclude that \\(h^\\gamma\\) is everywhere differentiable with\n\t\t\\(\n\t\t\t\\nabla h^\\gamma(s)\n\t\t{}={}\n\t\t\t\\tfrac1\\gamma(s-\\prox_{\\gamma h}(s))\n\t\t\\).\n% \t\tMoreover, \\(u\\) minimizes \\(h+\\tfrac{1}{2\\gamma}\\|{}\\cdot{}-s\\|^2\\) and in particular \\(\\tfrac{1}{\\gamma}(s-u)\\in\\partial h(u)\\).\n% \t\tTherefore,\n% \t\t\\(\n% \t\t\t\\nabla h(u)\n% \t\t{}\\overrel*{\\eqref{eq:NablaProj}}{}\n% \t\t\t\\proj_{\\dom h}\\partial h(u)\n% \t\t{}={}\n% \t\t\t\\tfrac{1}{\\gamma}(\\proj_{\\dom h}s-u)\n% \t\t\\),\n% \t\twhere the last equality follows from the fact that \\(u\\in\\dom h\\).\n\t\tThus,\n\t\t\\begin{align*}\n\t\t\t\\innprod{\\nabla h^\\gamma(s)-\\nabla h^\\gamma(s')}{s-s'}\n\t\t{}={} &\n\t\t\t\\tfrac{1}{\\gamma}\n\t\t\t\\left(\n\t\t\t\t\\|s-s'\\|^2\n\t\t\t\t{}-{}\n\t\t\t\t\\innprod{s-s'}{u-u'}\n\t\t\t\\right),\n% \t\t\\\\\n% \t\t{}\\leq{} &\n% \t\t\t\\tfrac{1}{\\gamma}\n% \t\t\t\\bigl(\n% \t\t\t\t1\n% \t\t\t\t{}-{}\n% \t\t\t\t\\tfrac{1}{1+\\gamma L_{h}}\n% \t\t\t\\bigr)\n% \t\t\t\t\\|s-s'\\|^2\n% \t\t{}={}\n% \t\t\t\\tfrac{L_{h}}{1+\\gamma L_{h}}\n% \t\t\t\\|s-s'\\|^2.\n\t\t\\end{align*}\n\t\tand from the bounds in \\ref{thm:proxfLip} we conclude that\n% \t\tSimilarly, by using also the second inequality in \\eqref{eq:coco} we arrive to\n\t\t\\[\n\t\t\t\\tfrac{\\sigma_{h}}{1+\\gamma\\sigma_{h}}\n\t\t\t\\|s-s'\\|^2\n\t\t{}\\leq{}\n\t\t\t\\innprod{\\nabla h^\\gamma(s)-\\nabla h^\\gamma(s')}{s-s'}\n\t\t{}\\leq{}\n\t\t\t\\tfrac{L_{h}}{1+\\gamma L_{h}}\n\t\t\t\\|s-s'\\|^2.\n\t\t\\]\n\t\tThe claimed smoothness and hypoconvexity follow from the characterization of \\eqref{eq:innprod}.\n\t\t\\qedhere\n\t\\end{proofitemize}\n\\end{appendixproof}","sha256_utf8":"c177f59a3b718e9410a4198c001b3f3570fa421308c49b3919a0740e9462b043"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"近端性质证明的附录指引","source_units":[{"start":46079,"end":46135,"start_line":829,"end_line":831,"raw_tex":"\\begin{proof}\n\t\tSee \\Cref{proof:thm:proxf}.\n\t\\end{proof}","sha256_utf8":"8b35bd666996567ea3e4da5e3104493ecfcc1e2c643d7e799e199392ecc2abef"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"强凸辅助函数与近端阈值","source_units":[{"start":201627,"end":202112,"start_line":5537,"end_line":5540,"raw_tex":"\\begin{appendixproof}{thm:proxf}%\n\tLet \\(\\gamma\\in(0,\\nicefrac{1}{[\\sigma_h]_-})\\) be fixed, and let \\(\\psi\\coloneqq\\gamma h+\\tfrac12\\|{}\\cdot{}\\|^2\\).\n\tObserve that \\(\\psi\\in\\cont^{1,1}(\\R^n)\\) is \\(L_\\psi\\)-smooth and \\(\\sigma_\\psi\\)-strongly convex, with \\(L_\\psi=1+\\gamma L_h\\) and \\(\\sigma_\\psi=1+\\gamma\\sigma_h\\).\n\tIn particular, due to strong convexity \\(\\inf\\psi>-\\infty\\), and by definition of prox-boundedness it then follows that \\(\\gamma_h\\geq\\nicefrac{1}{[\\sigma_h]_-}\\).\n","sha256_utf8":"7a9ace5f4e86ecb91356c5149aac05c06a80b61daaa89fc2ca5e2a75401b0473"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"近端点的唯一性与刻画","source_units":[{"start":202112,"end":202339,"start_line":5541,"end_line":5543,"raw_tex":"\t\\begin{proofitemize}\n\t\\item\\ref{thm:proxfEquiv}.\n\t\tFollows from \\eqref{eq:prox:subdiff}, by observing that \\(h+\\tfrac{1}{2\\gamma}\\|{}\\cdot{}-s\\|^2\\) is strongly convex, hence that a minimizer is characterized by stationarity.\n","sha256_utf8":"2a506d1dbe52ab362d2083ca00e0abc414050ddcfaea0b51de4bc617a5f9ca16"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"近端单调性与双Lipschitz界","source_units":[{"start":202339,"end":203017,"start_line":5544,"end_line":5564,"raw_tex":"\t\\item\\ref{thm:proxfLip}.\n\t\tFor \\(s,s'\\in\\R^n\\), let \\(u=\\prox_{\\gamma h}(s)\\) and \\(u'=\\prox_{\\gamma h}(s')\\).\n\t\tThen,\n\t\t\\[\n\t\t\t\\innprod{s-s'}{u-u'}\n\t\t{}={}\n\t\t\t\\innprod{\n\t\t\t\t\\nabla\\psi(u)\n\t\t\t\t{}-{}\n\t\t\t\t\\nabla\\psi(u')\n\t\t\t}{u-u'}\n\t\t{}\\geq{}\n\t\t\t\\sigma_\\psi\n\t\t\t\\|u-u'\\|^2\n\t\t{}={}\n\t\t\t(1+\\gamma\\sigma_h)\n\t\t\t\\|u-u'\\|^2,\n\t\t\\]\n\t\twhere the first equality was shown in \\ref{thm:proxfEquiv} and the inequality follows from \\eqref{eq:innprod}.\n\t\tBy using the \\(\\frac{1}{L_\\psi}\\)-cocoercivity of \\(\\nabla\\psi\\) \\cite[Thm. 2.1.10]{nesterov2003introductory}, also the claimed strong monotonicity follows.\n\t\tIn turn, the Cauchy-Schwartz inequality on the inner product yields \\eqref{eq:biLip}.\n","sha256_utf8":"b9cd0e473c750b4a29ebc3a918cc9d7ea9200b23c86e03ae261ad719394ae56b"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"Moreau包络可微性","source_units":[{"start":203017,"end":203891,"start_line":5565,"end_line":5588,"raw_tex":"\t\\item\\ref{thm:MoreaufC1}.\n\t\tFrom \\cite[Ex. 10.32]{rockafellar2011variational} it follows that \\(h^\\gamma\\) is strictly continuous and that\n\t\t\\(\n\t\t\t\\partial h^\\gamma(s)\n\t\t{}\\subseteq{}\n\t\t\t\\tfrac1\\gamma(s-\\prox_{\\gamma h}(s))\n\t\t\\).\n\t\tBecause of single valuedness of \\(\\prox_{\\gamma h}\\), by invoking \\cite[Thm. 9.18]{rockafellar2011variational} we conclude that \\(h^\\gamma\\) is everywhere differentiable with\n\t\t\\(\n\t\t\t\\nabla h^\\gamma(s)\n\t\t{}={}\n\t\t\t\\tfrac1\\gamma(s-\\prox_{\\gamma h}(s))\n\t\t\\).\n% \t\tMoreover, \\(u\\) minimizes \\(h+\\tfrac{1}{2\\gamma}\\|{}\\cdot{}-s\\|^2\\) and in particular \\(\\tfrac{1}{\\gamma}(s-u)\\in\\partial h(u)\\).\n% \t\tTherefore,\n% \t\t\\(\n% \t\t\t\\nabla h(u)\n% \t\t{}\\overrel*{\\eqref{eq:NablaProj}}{}\n% \t\t\t\\proj_{\\dom h}\\partial h(u)\n% \t\t{}={}\n% \t\t\t\\tfrac{1}{\\gamma}(\\proj_{\\dom h}s-u)\n% \t\t\\),\n% \t\twhere the last equality follows from the fact that \\(u\\in\\dom h\\).\n\t\tThus,\n","sha256_utf8":"74f541dc67212eb6cf34f6eeda91bf37bb72a7a4dc2882acd77893dd804e4924"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"Moreau包络的光滑弱凸模量","source_units":[{"start":203891,"end":204771,"start_line":5589,"end_line":5625,"raw_tex":"\t\t\\begin{align*}\n\t\t\t\\innprod{\\nabla h^\\gamma(s)-\\nabla h^\\gamma(s')}{s-s'}\n\t\t{}={} &\n\t\t\t\\tfrac{1}{\\gamma}\n\t\t\t\\left(\n\t\t\t\t\\|s-s'\\|^2\n\t\t\t\t{}-{}\n\t\t\t\t\\innprod{s-s'}{u-u'}\n\t\t\t\\right),\n% \t\t\\\\\n% \t\t{}\\leq{} &\n% \t\t\t\\tfrac{1}{\\gamma}\n% \t\t\t\\bigl(\n% \t\t\t\t1\n% \t\t\t\t{}-{}\n% \t\t\t\t\\tfrac{1}{1+\\gamma L_{h}}\n% \t\t\t\\bigr)\n% \t\t\t\t\\|s-s'\\|^2\n% \t\t{}={}\n% \t\t\t\\tfrac{L_{h}}{1+\\gamma L_{h}}\n% \t\t\t\\|s-s'\\|^2.\n\t\t\\end{align*}\n\t\tand from the bounds in \\ref{thm:proxfLip} we conclude that\n% \t\tSimilarly, by using also the second inequality in \\eqref{eq:coco} we arrive to\n\t\t\\[\n\t\t\t\\tfrac{\\sigma_{h}}{1+\\gamma\\sigma_{h}}\n\t\t\t\\|s-s'\\|^2\n\t\t{}\\leq{}\n\t\t\t\\innprod{\\nabla h^\\gamma(s)-\\nabla h^\\gamma(s')}{s-s'}\n\t\t{}\\leq{}\n\t\t\t\\tfrac{L_{h}}{1+\\gamma L_{h}}\n\t\t\t\\|s-s'\\|^2.\n\t\t\\]\n\t\tThe claimed smoothness and hypoconvexity follow from the characterization of \\eqref{eq:innprod}.\n\t\t\\qedhere\n\t\\end{proofitemize}\n\\end{appendixproof}","sha256_utf8":"70b1febc233a6dfd5726f54bf13336bebb78014494c99ce927786e0069c28228"}]},{"id":"S040","semantic_type":"proof","kind":"proof","title":"步长可行性的完整论证","source_units":[{"start":47338,"end":47860,"start_line":856,"end_line":867,"raw_tex":"\tThat \\(\\gamma_{\\sDRS}\\geq\\nicefrac{1}{L{\\sDRS}}\\) follows from \\Cref{thm:proxf}, having \\(\\nicefrac{1}{[\\sigma_{\\sDRS}]_-}\\geq\\nicefrac{1}{L_{\\sDRS}}\\).\n\tAs for \\(\\nsDRS\\), for all \\(s\\in\\R^p\\) it holds that\n\t\\[\n\t\t\\inf\\varphi\n\t{}\\leq{}\n\t\t\\sDRS(s)+\\nsDRS(s)\n\t{}\\overrel*[\\leq]{\\eqref{eq:LipBound}}{}\n\t\t\\sDRS(0)+\\innprod{\\nabla\\sDRS(0)}{s}+\\tfrac{L_{\\sDRS}}{2}\\|s\\|^2\n\t\t{}+{}\n\t\t\\nsDRS(s),\n\t\\]\n\thence, for all \\(\\gamma<\\nicefrac{1}{L_{\\sDRS}}\\) the function \\(s\\mapsto\\nsDRS(s)+\\tfrac{1}{2\\gamma}\\|s\\|^2\\) is lower bounded.\n","sha256_utf8":"1cfcb27102915c4abcadda6b1c311e8019f78063c4f1bc8b692fd7c7c1cf4161"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"光滑项的近端阈值","source_units":[{"start":47338,"end":47492,"start_line":856,"end_line":856,"raw_tex":"\tThat \\(\\gamma_{\\sDRS}\\geq\\nicefrac{1}{L{\\sDRS}}\\) follows from \\Cref{thm:proxf}, having \\(\\nicefrac{1}{[\\sigma_{\\sDRS}]_-}\\geq\\nicefrac{1}{L_{\\sDRS}}\\).\n","sha256_utf8":"a32143c6d9cf86c621c1f8e066909f6eb2eae02c0fc35321c9a7a13a973f0300"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"由目标下界控制非光滑项","source_units":[{"start":47492,"end":47860,"start_line":857,"end_line":867,"raw_tex":"\tAs for \\(\\nsDRS\\), for all \\(s\\in\\R^p\\) it holds that\n\t\\[\n\t\t\\inf\\varphi\n\t{}\\leq{}\n\t\t\\sDRS(s)+\\nsDRS(s)\n\t{}\\overrel*[\\leq]{\\eqref{eq:LipBound}}{}\n\t\t\\sDRS(0)+\\innprod{\\nabla\\sDRS(0)}{s}+\\tfrac{L_{\\sDRS}}{2}\\|s\\|^2\n\t\t{}+{}\n\t\t\\nsDRS(s),\n\t\\]\n\thence, for all \\(\\gamma<\\nicefrac{1}{L_{\\sDRS}}\\) the function \\(s\\mapsto\\nsDRS(s)+\\tfrac{1}{2\\gamma}\\|s\\|^2\\) is lower bounded.\n","sha256_utf8":"91313e42e13de6509b82580a72782917866687c85a8a5009891f6d9b3fd1a058"}]},{"id":"S043","semantic_type":"proposition","kind":"proposition","title":"DR包络的严格连续性","source_units":[{"start":53020,"end":53232,"start_line":978,"end_line":981,"raw_tex":"\\begin{prop}[Strict continuity]\\label{thm:C0}%\n\tSuppose that \\Cref{ass:DRS} is satisfied.\n\tFor all \\(\\gamma<\\nicefrac{1}{L_{\\sDRS}}\\) the DRE \\(\\DRE\\) is a real-valued and strictly continuous function.\n\\end{prop}","sha256_utf8":"4d9cdbd911ed084f801c6207f71ef7d09ffebe9e882354cc9237363b7912d7b8"}]},{"id":"S044","semantic_type":"proof","kind":"proof","title":"DR包络连续性的完整论证","source_units":[{"start":52744,"end":53020,"start_line":977,"end_line":977,"raw_tex":"We start by observing that \\(\\DRE\\) is a strictly continuous function for \\(\\gamma<\\nicefrac{1}{L_{\\sDRS}}\\), owing to the fact that so is the FBE \\cite[Prop. 4.2]{themelis2018forward}, and that \\(\\prox_{\\gamma\\sDRS}\\) is Lipschitz continuous as shown in \\cref{thm:proxfLip}.\n","sha256_utf8":"a3fbdc43a69870bad8f5a909bb265b3cab89ad3d639c699e5c0cccc19d51a81d"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"连续函数复合给出包络连续性","source_units":[{"start":52744,"end":53020,"start_line":977,"end_line":977,"raw_tex":"We start by observing that \\(\\DRE\\) is a strictly continuous function for \\(\\gamma<\\nicefrac{1}{L_{\\sDRS}}\\), owing to the fact that so is the FBE \\cite[Prop. 4.2]{themelis2018forward}, and that \\(\\prox_{\\gamma\\sDRS}\\) is Lipschitz continuous as shown in \\cref{thm:proxfLip}.\n","sha256_utf8":"a3fbdc43a69870bad8f5a909bb265b3cab89ad3d639c699e5c0cccc19d51a81d"}]},{"id":"S046","semantic_type":"proposition","kind":"proposition","title":"DR包络的夹逼性质","source_units":[{"start":53615,"end":54257,"start_line":985,"end_line":1008,"raw_tex":"\\begin{prop}[Sandwiching property]\\label{thm:sandwich}%\n\tSuppose that \\Cref{ass:DRS} is satisfied.\n\tLet \\(\\gamma<\\nicefrac{1}{L_{\\sDRS}}\\) be fixed, and consider \\(u,v\\) generated by one \\ref{DRS} iteration starting from \\(s\\in\\R^p\\).\n\tThen,\n\t\\begin{enumerate}\n\t\\item\\label{thm:DREleq}\n\t\t\\(\\DRE(s)\\leq\\varphi(u)\\).\n\t\\item\\label{thm:DREgeq}\n\t\t\\(\n\t\t\t\\varphi(v)\n\t\t{}\\leq{}\n\t\t\t\\DRE(s)\n\t\t\t{}-{}\n\t\t\t\\tfrac{1-\\gamma L_{\\sDRS}}{2\\gamma}\\|u-v\\|^2\n\t\t\\).\n% \t\t\\(\n% \t\t\t\\varphi(v)+\\tfrac{1-\\gamma L_{\\sDRS}}{2\\gamma}\\|u-v\\|^2\n% \t\t{}\\leq{}\n% \t\t\t\\DRE(s)\n% \t\t{}\\leq{}\n% \t\t\t\\varphi(v)+\\tfrac{1-\\gamma\\sigma_{\\sDRS}}{2\\gamma}\\|u-v\\|^2\n% \t\t\\).\n\t\\end{enumerate}\n\t","sha256_utf8":"2aa530b3e374844f8787b997ef15eb1b06285df4cbefcbc721c64990e761b574"},{"start":54659,"end":54670,"start_line":1019,"end_line":1020,"raw_tex":"\n\\end{prop}","sha256_utf8":"05d52a2387e348b9b8c6add2d64ac5b19b3c01cd0607238154a9f309962fa7aa"}]},{"id":"S047","semantic_type":"proof","kind":"proof","title":"夹逼性质完整证明","source_units":[{"start":54257,"end":54659,"start_line":1008,"end_line":1019,"raw_tex":"\\begin{proof}\n\t\t\\ref{thm:DREleq} is easily inferred from definition \\eqref{eq:DRE} by considering \\(w=u\\).\n\t\tMoreover, it follows from \\cite[Prop. 4.3]{themelis2018forward} and the fact that \\(v\\in\\FB u\\), cf. \\eqref{eq:vFBS}, that\n\t\t\\(\n\t\t\t\\varphi(v)\n\t\t{}\\leq{}\n\t\t\t\\FBE(u)\n\t\t\t{}-{}\n\t\t\t\\tfrac{1-\\gamma L_{\\sDRS}}{2\\gamma}\\|u-v\\|^2\n\t\t\\).\n\t\t\\ref{thm:DREgeq} then follows from \\eqref{eq:FBEu}.\n\t\\end{proof}","sha256_utf8":"1a47ab615626fe7eceaeeba54d1928abc67931cf66759995a7b09a8863c3bbb4"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"代入原近端点得到上界","source_units":[{"start":54257,"end":54364,"start_line":1008,"end_line":1009,"raw_tex":"\\begin{proof}\n\t\t\\ref{thm:DREleq} is easily inferred from definition \\eqref{eq:DRE} by considering \\(w=u\\).\n","sha256_utf8":"2e892597b1fce7e7042de5ab1325dbbcb0a0d5724b4bf0977af3575eb871ca47"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"前向后向包络界得到下界","source_units":[{"start":54364,"end":54659,"start_line":1010,"end_line":1019,"raw_tex":"\t\tMoreover, it follows from \\cite[Prop. 4.3]{themelis2018forward} and the fact that \\(v\\in\\FB u\\), cf. \\eqref{eq:vFBS}, that\n\t\t\\(\n\t\t\t\\varphi(v)\n\t\t{}\\leq{}\n\t\t\t\\FBE(u)\n\t\t\t{}-{}\n\t\t\t\\tfrac{1-\\gamma L_{\\sDRS}}{2\\gamma}\\|u-v\\|^2\n\t\t\\).\n\t\t\\ref{thm:DREgeq} then follows from \\eqref{eq:FBEu}.\n\t\\end{proof}","sha256_utf8":"ebe2285b4811b6a6d84112871aa4e954cef7f4e307971bc34315364f9509bf93"}]},{"id":"S050","semantic_type":"theorem","kind":"theorem","title":"极小化与水平有界性等价","source_units":[{"start":54677,"end":55125,"start_line":1023,"end_line":1034,"raw_tex":"\\begin{thm}[Minimization and level-boundedness equivalence]\\label{thm:DREequiv}%\n\tSuppose that \\Cref{ass:DRS} is satisfied.\n\tFor any \\(\\gamma<\\nicefrac{1}{L_{\\sDRS}}\\) the following hold:\n\t\\begin{enumerate}\n\t\\item\\label{thm:inf}%\n\t\t\\(\\inf\\varphi=\\inf\\DRE\\).\n\t\\item\\label{thm:argmin}%\n\t\t\\(\\argmin\\varphi=\\prox_{\\gamma\\sDRS}\\bigl(\\argmin\\DRE\\bigr)\\).\n\t\\item\\label{thm:boundedlev}%\n\t\t\\(\\varphi\\) is level bounded iff so is \\(\\DRE\\).\n\t\\end{enumerate}\n\t","sha256_utf8":"f9f705a90d4335e9e9d779e383753afc549e66ba50f6f05aeb443be1bc5e4ccc"},{"start":57900,"end":57910,"start_line":1140,"end_line":1141,"raw_tex":"\n\\end{thm}","sha256_utf8":"27661e277ceb78e4dd676b4ea4b5c8d9a4d3c607a1a758eac095282a4f311f77"}]},{"id":"S051","semantic_type":"proof","kind":"proof","title":"包络等价性完整证明","source_units":[{"start":55125,"end":57900,"start_line":1034,"end_line":1140,"raw_tex":"\\begin{proof}\n\t\tIt follows from \\cite[Thm. 4.4]{themelis2018forward} that the FBE satisfies \\(\\inf\\varphi=\\inf\\FBE\\) and \\(\\argmin\\varphi=\\argmin\\FBE\\).\n\t\tThe similar properties \\ref{thm:inf} and \\ref{thm:argmin} of the DRE then follow from the identity \\(\\DRE=\\FBE\\circ\\prox_{\\gamma\\sDRS}\\), cf. \\eqref{eq:FBEu}, and the fact that \\(\\prox_{\\gamma\\sDRS}\\) is invertible, as shown in \\cref{thm:proxf}.\n\t\t\n\t\tWe now show \\ref{thm:boundedlev}.\n\t\tDenote\n\t\t\\(\n\t\t\t\\varphi_\\star\n\t\t{}\\coloneqq{}\n\t\t\t\\inf\\varphi\n\t\t{}={}\n\t\t\t\\inf\\DRE\n\t\t\\),\n\t\twhich is finite by assumption.\n\t\t\\begin{proofitemize}\n\t\t\\item\n\t\t\tSuppose that \\(\\DRE\\) is level bounded, and let \\(u\\in\\lev_{\\leq\\alpha}\\varphi\\) for some \\(\\alpha>\\varphi_\\star\\).\n\t\t\tThen, \\(s\\coloneqq u+\\gamma\\nabla\\sDRS(u)\\) is such that \\(\\prox_{\\gamma\\sDRS}(s)=u\\), as shown in \\cref{thm:proxfEquiv}.\n\t\t\tThus, from \\cref{thm:sandwich} it follows that \\(s\\in\\lev_{\\leq\\alpha}\\DRE\\).\n\t\t\tIn particular,\n\t\t\t\\(\n\t\t\t\t\\lev_{\\leq\\alpha}\\varphi\n\t\t\t{}\\subseteq{}\n\t\t\t\t[I+\\gamma\\nabla\\sDRS]\n\t\t\t\t(\\lev_{\\leq\\alpha}\\DRE),\n\t\t\t\\)\n\t\t\tand since \\(\\I+\\gamma\\nabla\\sDRS\\) is Lipschitz continuous and \\(\\lev_{\\leq\\alpha}\\DRE\\) is bounded by assumption, it follows that \\(\\lev_{\\leq\\alpha}\\varphi\\) is also bounded.\n\t\t\\item\n\t\t\tSuppose now that \\(\\DRE\\) is not level bounded.\n\t\t\tThen, there exists \\(\\alpha>\\varphi_\\star\\) together with a sequence \\(\\seq{s_k}\\) satisfying\n\t\t\t\\(\n\t\t\t\ts_k\n\t\t\t{}\\in{}\n\t\t\t\t\\lev_{\\leq\\alpha}\\DRE\n\t\t\t\t{}\\setminus{}\n\t\t\t\t\\ball 0k\n\t\t\t\\)\n\t\t\tfor all \\(k\\in\\N\\).\n\t\t\tLet \\(u_k\\coloneqq\\prox_{\\gamma\\sDRS}(s_k)\\), so that \\(s_k=u_k+\\gamma\\nabla\\sDRS(u_k)\\) (cf. \\cref{thm:proxfEquiv}), and let \\(v_k\\in\\FB{u_k}\\).\n\t\t\tFrom \\cref{thm:DREgeq} it then follows that \\(v_k\\in\\lev_{\\leq\\alpha}\\varphi\\), and that\n\t\t\t\\[\n\t\t\t\t\\alpha\n\t\t\t\t{}-{}\n\t\t\t\t\\varphi_\\star\n\t\t\t{}\\geq{}\n\t\t\t\t\\DRE(s_k)\n\t\t\t\t{}-{}\n\t\t\t\t\\varphi_\\star\n\t\t\t{}\\geq{}\n\t\t\t\t\\DRE(s_k)\n\t\t\t\t{}-{}\n\t\t\t\t\\varphi(v_k)\n\t\t\t{}\\geq{}\n\t\t\t\t\\tfrac{1-\\gamma L_{\\sDRS}}{2\\gamma}\n\t\t\t\t\\|u_k-v_k\\|^2.\n\t\t\t\\]\n\t\t\tTherefore,\n\t\t\t\\(\n\t\t\t\t\\|u_k-v_k\\|^2\n\t\t\t{}\\leq{}\n\t\t\t\t\\tfrac{\n\t\t\t\t\t2\\gamma(\\alpha-\\varphi_\\star)\n\t\t\t\t}{\n\t\t\t\t\t1-\\gamma L_{\\sDRS}\n\t\t\t\t}\n\t\t\t\\)\n\t\t\tand\n\t\t\t\\begin{align*}\n\t\t\t\t\\|v_k\\|\n\t\t\t{}\\geq{} &\n\t\t\t\t\\|u_k-u_0\\|\n\t\t\t\t{}-{}\n\t\t\t\t\\|u_0\\|\n\t\t\t\t{}-{}\n\t\t\t\t\\|u_k-v_k\\|\n\t\t\t\\smash{{}\\overrel[\\geq]{\\ref{thm:proxfLip}}{}}\n\t\t\t\t\\tfrac{1}{1+\\gamma L_{\\sDRS}}\n\t\t\t\t\\|s_k-s_0\\|\n\t\t\t\t{}-{}\n\t\t\t\t\\|u_0\\|\n\t\t\t\t{}-{}\n\t\t\t\t\\|u_k-v_k\\|\n\t\t\t\\\\\n\t\t\t{}\\geq{} &\n\t\t\t\t\\tfrac{\n\t\t\t\t\tk-\\|s_0\\|\n\t\t\t\t}{\n\t\t\t\t\t1+\\gamma L_{\\sDRS}\n\t\t\t\t}\n\t\t\t\t{}-{}\n\t\t\t\t\\|u_0\\|\n\t\t\t\t{}-{}\n\t\t\t\t\\sqrt{\n\t\t\t\t\t\\tfrac{\n\t\t\t\t\t\t2\\gamma(\\alpha-\\varphi_\\star)\n\t\t\t\t\t}{\n\t\t\t\t\t\t1-\\gamma L_{\\sDRS}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t{}\\to{}\n\t\t\t\t+\\infty\n\t\t\t\\quad\\text{as \\(k\\to\\infty\\).}\n\t\t\t\\end{align*}\n\t\t\tThis shows that \\(\\lev_{\\leq\\alpha}\\varphi\\) is also unbounded.\n\t\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}","sha256_utf8":"a4f4523bc7a0f3d9de278d8aedb8053c00f87b4d1aa23a03dbd99bc2b5554137"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"可逆复合保持下确界与极小点","source_units":[{"start":55125,"end":55526,"start_line":1034,"end_line":1036,"raw_tex":"\\begin{proof}\n\t\tIt follows from \\cite[Thm. 4.4]{themelis2018forward} that the FBE satisfies \\(\\inf\\varphi=\\inf\\FBE\\) and \\(\\argmin\\varphi=\\argmin\\FBE\\).\n\t\tThe similar properties \\ref{thm:inf} and \\ref{thm:argmin} of the DRE then follow from the identity \\(\\DRE=\\FBE\\circ\\prox_{\\gamma\\sDRS}\\), cf. \\eqref{eq:FBEu}, and the fact that \\(\\prox_{\\gamma\\sDRS}\\) is invertible, as shown in \\cref{thm:proxf}.\n","sha256_utf8":"432e17762bc141263ff4ef0ab9455aa219f0f8117e6ce961b70b6db743ee03b2"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"固定共同有限下确界","source_units":[{"start":55526,"end":55686,"start_line":1037,"end_line":1047,"raw_tex":"\t\t\n\t\tWe now show \\ref{thm:boundedlev}.\n\t\tDenote\n\t\t\\(\n\t\t\t\\varphi_\\star\n\t\t{}\\coloneqq{}\n\t\t\t\\inf\\varphi\n\t\t{}={}\n\t\t\t\\inf\\DRE\n\t\t\\),\n\t\twhich is finite by assumption.\n","sha256_utf8":"2dff6c22475155efe15de5bf73ec7c1c6cf1642f896e7911eab5199e06443492"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"包络水平有界推出目标水平有界","source_units":[{"start":55686,"end":56353,"start_line":1048,"end_line":1060,"raw_tex":"\t\t\\begin{proofitemize}\n\t\t\\item\n\t\t\tSuppose that \\(\\DRE\\) is level bounded, and let \\(u\\in\\lev_{\\leq\\alpha}\\varphi\\) for some \\(\\alpha>\\varphi_\\star\\).\n\t\t\tThen, \\(s\\coloneqq u+\\gamma\\nabla\\sDRS(u)\\) is such that \\(\\prox_{\\gamma\\sDRS}(s)=u\\), as shown in \\cref{thm:proxfEquiv}.\n\t\t\tThus, from \\cref{thm:sandwich} it follows that \\(s\\in\\lev_{\\leq\\alpha}\\DRE\\).\n\t\t\tIn particular,\n\t\t\t\\(\n\t\t\t\t\\lev_{\\leq\\alpha}\\varphi\n\t\t\t{}\\subseteq{}\n\t\t\t\t[I+\\gamma\\nabla\\sDRS]\n\t\t\t\t(\\lev_{\\leq\\alpha}\\DRE),\n\t\t\t\\)\n\t\t\tand since \\(\\I+\\gamma\\nabla\\sDRS\\) is Lipschitz continuous and \\(\\lev_{\\leq\\alpha}\\DRE\\) is bounded by assumption, it follows that \\(\\lev_{\\leq\\alpha}\\varphi\\) is also bounded.\n","sha256_utf8":"d47cb0daef8d7acf9138175d2ba2239cf65c760b71c4fb14816164b791ec12da"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"无界包络水平集产生受控近端对","source_units":[{"start":56353,"end":57237,"start_line":1061,"end_line":1100,"raw_tex":"\t\t\\item\n\t\t\tSuppose now that \\(\\DRE\\) is not level bounded.\n\t\t\tThen, there exists \\(\\alpha>\\varphi_\\star\\) together with a sequence \\(\\seq{s_k}\\) satisfying\n\t\t\t\\(\n\t\t\t\ts_k\n\t\t\t{}\\in{}\n\t\t\t\t\\lev_{\\leq\\alpha}\\DRE\n\t\t\t\t{}\\setminus{}\n\t\t\t\t\\ball 0k\n\t\t\t\\)\n\t\t\tfor all \\(k\\in\\N\\).\n\t\t\tLet \\(u_k\\coloneqq\\prox_{\\gamma\\sDRS}(s_k)\\), so that \\(s_k=u_k+\\gamma\\nabla\\sDRS(u_k)\\) (cf. \\cref{thm:proxfEquiv}), and let \\(v_k\\in\\FB{u_k}\\).\n\t\t\tFrom \\cref{thm:DREgeq} it then follows that \\(v_k\\in\\lev_{\\leq\\alpha}\\varphi\\), and that\n\t\t\t\\[\n\t\t\t\t\\alpha\n\t\t\t\t{}-{}\n\t\t\t\t\\varphi_\\star\n\t\t\t{}\\geq{}\n\t\t\t\t\\DRE(s_k)\n\t\t\t\t{}-{}\n\t\t\t\t\\varphi_\\star\n\t\t\t{}\\geq{}\n\t\t\t\t\\DRE(s_k)\n\t\t\t\t{}-{}\n\t\t\t\t\\varphi(v_k)\n\t\t\t{}\\geq{}\n\t\t\t\t\\tfrac{1-\\gamma L_{\\sDRS}}{2\\gamma}\n\t\t\t\t\\|u_k-v_k\\|^2.\n\t\t\t\\]\n\t\t\tTherefore,\n\t\t\t\\(\n\t\t\t\t\\|u_k-v_k\\|^2\n\t\t\t{}\\leq{}\n\t\t\t\t\\tfrac{\n\t\t\t\t\t2\\gamma(\\alpha-\\varphi_\\star)\n\t\t\t\t}{\n\t\t\t\t\t1-\\gamma L_{\\sDRS}\n\t\t\t\t}\n\t\t\t\\)\n\t\t\tand\n","sha256_utf8":"74366eef9671d4bc682930f8c5e676ad98812ceae190edd5783b7f65af755c98"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"近端双边界推出目标水平集无界","source_units":[{"start":57237,"end":57900,"start_line":1101,"end_line":1140,"raw_tex":"\t\t\t\\begin{align*}\n\t\t\t\t\\|v_k\\|\n\t\t\t{}\\geq{} &\n\t\t\t\t\\|u_k-u_0\\|\n\t\t\t\t{}-{}\n\t\t\t\t\\|u_0\\|\n\t\t\t\t{}-{}\n\t\t\t\t\\|u_k-v_k\\|\n\t\t\t\\smash{{}\\overrel[\\geq]{\\ref{thm:proxfLip}}{}}\n\t\t\t\t\\tfrac{1}{1+\\gamma L_{\\sDRS}}\n\t\t\t\t\\|s_k-s_0\\|\n\t\t\t\t{}-{}\n\t\t\t\t\\|u_0\\|\n\t\t\t\t{}-{}\n\t\t\t\t\\|u_k-v_k\\|\n\t\t\t\\\\\n\t\t\t{}\\geq{} &\n\t\t\t\t\\tfrac{\n\t\t\t\t\tk-\\|s_0\\|\n\t\t\t\t}{\n\t\t\t\t\t1+\\gamma L_{\\sDRS}\n\t\t\t\t}\n\t\t\t\t{}-{}\n\t\t\t\t\\|u_0\\|\n\t\t\t\t{}-{}\n\t\t\t\t\\sqrt{\n\t\t\t\t\t\\tfrac{\n\t\t\t\t\t\t2\\gamma(\\alpha-\\varphi_\\star)\n\t\t\t\t\t}{\n\t\t\t\t\t\t1-\\gamma L_{\\sDRS}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t{}\\to{}\n\t\t\t\t+\\infty\n\t\t\t\\quad\\text{as \\(k\\to\\infty\\).}\n\t\t\t\\end{align*}\n\t\t\tThis shows that \\(\\lev_{\\leq\\alpha}\\varphi\\) is also unbounded.\n\t\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}","sha256_utf8":"1764a7038646d155118cd09e12e33fe7524b63912e6fb94537af1a73696697ac"}]},{"id":"S057","semantic_type":"theorem","kind":"theorem","title":"DR包络充分下降与参数范围","source_units":[{"start":71396,"end":73436,"start_line":1445,"end_line":1547,"raw_tex":"\\begin{thm}[Sufficient decrease on the DRE]\\label{thm:DRE:SD}%\n\tSuppose that \\Cref{ass:DRS} is satisfied, and consider one \\ref{DRS} update\n\t\\(\n\t\ts\n\t{}\\mapsto{}\n\t\t(u,v,s^+)\n\t\\)\n\tfor some stepsize\n\t\\(\n\t\t\\gamma\n\t{}<{}\n\t\t\\min\\set{\n\t\t\t\\tfrac{2-\\lambda}{2[\\sigma_{\\sDRS}]_-},\\,\n\t\t\t\\tfrac{1}{L_{\\sDRS}}\n\t\t}\n\t\\)\n\tand relaxation \\(\\lambda\\in(0,2)\\).\n\tThen,\n\t\\begin{equation}\\label{eq:DRE:SD}\n\t\t\\DRE(s)-\\DRE(s^+)\n\t{}\\geq{}\n\t\t\\tfrac{c}{(1+\\gamma L_{\\sDRS})^2}\\,\n\t\t\\|s-s^+\\|^2,\n\t\\end{equation}\n\twhere, denoting \\(p_{\\sDRS}\\coloneqq\\nicefrac{\\sigma_{\\sDRS}}{L_{\\sDRS}}\\in[-1,1]\\), \\(c\\) is a strictly positive constant defined as%\n\t\\footnote{%\n\t\tA one-line expression for the constant is\n\t\t\\(\n\t\t\tc\n\t\t{}={}\n\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n\t\t\t{}-{}\n\t\t\t\\min\\set{\n\t\t\t\t\\tfrac{[p_{\\sDRS}]_-}{\\lambda},~\n\t\t\t\tL_{\\sDRS}\\max\\set{\n\t\t\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{2(1-[p_{\\sDRS}]_-)},\\,\n\t\t\t\t\t\\tfrac12-\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\n\t\t\t\t}\n\t\t\t}\n\t\t\\).\n\t}\n\t\\begin{equation}\\label{eq:DRE:c0}\n\t\tc\n\t{}={}\n\t\t\\frac{2-\\lambda}{2\\lambda\\gamma}\n\t\t{}-{}\n\t\t\\begin{ifcases}\n\t\t\tL_{\\sDRS}\\max\\set{\n\t\t\t\t\\tfrac{[p_{\\sDRS}]_-}{2(1-[p_{\\sDRS}]_-)},\\,\n\t\t\t\t\\tfrac12-\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\n\t\t\t}\n\t\t&\n\t\t\t\tp_{\\sDRS}\n\t\t\t{}\\geq{}\n\t\t\t\t\\tfrac\\lambda2-1\n% \t\t\t0<\\lambda<2(1-[p_{\\sDRS}]_-),\n\t\t\\\\\n\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{\\lambda}\n\t\t\\otherwise[otherwise.]\n\t\t\\end{ifcases}\n\t\\end{equation}\n\tIf \\(\\sDRS\\) is strongly convex, then \\eqref{eq:DRE:SD} also holds for\n\t\\begin{equation}\\label{eq:lambda>=2}\n\t\t2\n\t{}\\leq{}\n\t\t\\lambda\n\t{}<{}\n\t\t\\tfrac{4}{1+\\sqrt{1-p_{\\sDRS}}}\n\t\\quad\\text{and}\\quad\n\t\t\\tfrac{\n\t\t\tp_{\\sDRS}\\lambda\n\t\t\t{}-{}\n\t\t\t\\delta\n\t\t}{\n\t\t\t4\\sigma_{\\sDRS}\n\t\t}\n\t{}<{}\n\t\t\\gamma\n\t{}<{}\n\t\t\\tfrac{\n\t\t\tp_{\\sDRS}\\lambda\n\t\t\t{}+{}\n\t\t\t\\delta\n\t\t}{\n\t\t\t4\\sigma_{\\sDRS}\n\t\t},\n\t\\end{equation}\n\twhere\n\t\\(\n\t\t\\delta\n\t{}\\coloneqq{}\n\t\t\\sqrt{(p_{\\sDRS}\\lambda)^2-8p_{\\sDRS}(\\lambda-2)}\n\t\\),\n\tin which case\n\t\\begin{equation}\\label{eq:DRE:c1}\n\t\tc\n\t{}={}\n\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n\t\t{}+{}\n\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n\t\t(\\tfrac12-\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}).\n\t\\end{equation}\n\t","sha256_utf8":"667aa2bde2eddcab8123542282ac774b8d662e07ef62ad43a9a78ea1035bf8f5"},{"start":89744,"end":89754,"start_line":2246,"end_line":2247,"raw_tex":"\n\\end{thm}","sha256_utf8":"27661e277ceb78e4dd676b4ea4b5c8d9a4d3c607a1a758eac095282a4f311f77"}]},{"id":"S058","semantic_type":"proof","kind":"proof","title":"充分下降定理完整证明","source_units":[{"start":73436,"end":89744,"start_line":1547,"end_line":2246,"raw_tex":"\\begin{proof}\n\t\t\\newcommand\\x\\xi%\n\t\tLet \\((u^+,v^+)\\) be generated by one \\ref{DRS} iteration starting at \\(s^+\\).\n\t\tThen,\n\t\t\\begin{align*}\n\t\t\t\\DRE(s^+)\n\t\t{}={} &\n\t\t\t\\min_{w\\in\\R^n}\\set{\n\t\t\t\t\\sDRS(u^+)\n\t\t\t\t{}+{}\n\t\t\t\t\\nsDRS(w)\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)}{w-u^+}\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|w-u^+\\|^2\n\t\t\t}\n\t\t\\end{align*}\n\t\tand the minimum is attained at \\(w=v^+\\).\n\t\tTherefore, letting \\(\\rho\\) be as in \\cref{thm:rho},\n\t\t{\\mathtight[0.9]%\n\t\t\t\\begin{align*}\n\t\t\t\t\\DRE(s^+)\n\t\t\t{}\\leq{} &\n\t\t\t\t\\sDRS(u^+)\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)}{v-u^+}\n\t\t\t\t{}+{}\n\t\t\t\t\\nsDRS(v)\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|u^+-v\\|^2\n\t\t\t\\\\\n\t\t\t{}={} &\n\t\t\t\t\\smashunderbracket[0.5pt]{\n\t\t\t\t\t\\sDRS(u^+)\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)}{u-u^+}\n\t\t\t\t}{}\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)}{v-u}\n\t\t\t\t{}+{}\n\t\t\t\t\\nsDRS(v)\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|u^+-v\\|^2\n\t\t\t\\\\\n\t\t\t% \t\t\t\\dueto{\\cref{thm:rho}}\n\t\t\t{}\\overrel[\\leq]{\\llap{\\cref{thm:rho}}}{} &\n\t\t\t\t\\smashoverbracket[0.5pt]{\n\t\t\t\t\t\\sDRS(u)\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\rho(u,u^+)\n\t\t\t\t}{}\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)}{v-u}\n\t\t\t\t{}+{}\n\t\t\t\t\\nsDRS(v)\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|u^+-v\\|^2\n\t\t\t\\\\\n\t\t\t{}={} &\n\t\t\t\t\\sDRS(u)\n\t\t\t\t{}-{}\n\t\t\t\t\\rho(u,u^+)\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u)}{v-u}\n% \t\t\t\t\\innprod{\\nabla\\sDRS(\\fillwidthof[c]{u^+}{u})}{v-u}\n\t\t\t\t{}+{}\n\t\t\t\t\\nsDRS(v)\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|u^+-v\\|^2\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t{+{}}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)-\\nabla\\sDRS(u)}{v-u}\n\t\t\t\\\\\n\t\t\t{}={} &\n\t\t\t\t\\DRE(s)\n\t\t\t\t{}-{}\n\t\t\t\t\\rho(u,u^+)\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)-\\nabla\\sDRS(u)}{v-u}\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|u-u^+\\|^2\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac1\\gamma\n\t\t\t\t\\innprod{u^+-u}{u-v}.\n\t\t\t\\end{align*}%\n\t\t}%\n\t\tSince\n\t\t\\(\n\t\t\tu-v\n\t\t{}={}\n\t\t\t\\tfrac1\\lambda(s-s^+)\n\t\t{}={}\n\t\t\t\\tfrac1\\lambda(u-u^+)\n\t\t\t{}+{}\n\t\t\t\\tfrac\\gamma\\lambda(\\nabla\\sDRS(u)-\\nabla\\sDRS(u^+))\n\t\t\\),\n\t\tas it follows from \\cref{thm:proxfEquiv}, it all simplifies to\n\t\t\\begin{equation}\\label{eq:DRE:SD0}\n\t\t\t\\DRE(s)\n\t\t\t{}-{}\n\t\t\t\\DRE(s^+)\n\t\t{}\\geq{}\n\t\t\t\\tfrac{2-\\lambda}{2\\gamma\\lambda}\n\t\t\t\\|u-u^+\\|^2\n\t\t\t{}-{}\n\t\t\t\\tfrac\\gamma\\lambda\n\t\t\t\\|\\nabla\\sDRS(u^+)-\\nabla\\sDRS(u)\\|^2\n\t\t\t{}+{}\n\t\t\t\\rho(u,u^+).\n\t\t\\end{equation}\n\t\tIt will suffice to show that\n\t\t\\[\n\t\t\t\\DRE(s)-\\DRE(s^+)\n\t\t{}\\geq{}\n\t\t\tc\n\t\t\t\\|u-u^+\\|^2;\n\t\t\\]\n\t\tinequality \\eqref{eq:DRE:SD} will then follow from the \\(\\tfrac{1}{1+\\gamma L_{\\sDRS}}\\)-strong monotonicity of \\(\\prox_{\\gamma\\sDRS}\\), see \\cref{thm:proxfLip}.\n% \t\t\\[\n% \t\t\t\\|u-u^+\\|^2\n% \t\t{}\\geq{}\n% \t\t\t\\tfrac{1}{(1+\\gamma L_{\\sDRS})^2}\n% \t\t\t\\|s-s^+\\|^2\n% \t\t\\]\n\t\tWe now proceed by cases.\n\t\t\\begin{proofitemize}\n\t\t\\item\\textbf{Case 1: \\(\\lambda\\in(0,2)\\).}\n\t\t\t\n\t\t\t\\noindent\n\t\t\tLet \\(\\sigma\\coloneqq-[\\sigma_{\\sDRS}]_-=\\min\\set{\\sigma_{\\sDRS},0}\\) and \\(L\\geq L_{\\sDRS}\\) be such that \\(L+\\sigma>0\\); the value of such an \\(L\\) will be fixed later.\n\t\t\tThen, \\(\\sigma\\leq0\\) and \\(\\sDRS\\) is \\(L\\)-smooth and \\(\\sigma\\)-hypoconvex.\n\t\t\tWe may thus choose \\(\\rho(u,u^+)\\) as in \\cref{thm:hypo:rho1} with these values of \\(L\\) and \\(\\sigma\\).\n\t\t\tInequality \\eqref{eq:DRE:SD0} then becomes\n\t\t\t\\[\n\t\t\t\t\\tfrac{\\DRE(s)-\\DRE(s^+)}{L}\n\t\t\t{}\\geq{}\n\t\t\t\t\\left(\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\x}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{p}{2(1+p)}\n\t\t\t\t\\right)\n\t\t\t\t\\|u^+-u\\|^2\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{L^2}\n\t\t\t\t\\left(\n\t\t\t\t\t\\tfrac{1}{2(1+p)}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{\\x}{\\lambda}\n\t\t\t\t\\right)\n\t\t\t\t\\|\\nabla\\sDRS(u^+)-\\nabla\\sDRS(u)\\|^2,\n\t\t\t\\]\n\t\t\twhere \\(\\x\\coloneqq\\gamma L\\) and \\(p\\coloneqq\\nicefrac\\sigma L\\in(-1,0]\\).\n\t\t\tSince \\(\\nabla\\sDRS\\) is \\(L_{\\sDRS}\\)-Lipschitz continuous, the claim holds provided that the constant\n\t\t\t\\begin{equation}\\label{eq:c}\n\t\t\t\t\\frac cL\n\t\t\t{}={}\n\t\t\t\t\\begin{ifcases}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\x}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{p}{2(1+p)}\n\t\t\t\t&\n\t\t\t\t\t0\n\t\t\t\t\t{}<{}\n\t\t\t\t\t\\tfrac{1}{2(1+p)}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{\\x}{\\lambda}\n\t\t\t\t\\\\[3pt]\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\x}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{p}{2(1+p)}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{L_{\\sDRS}^2}{L^2}\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t\\tfrac{1}{2(1+p)}\n\t\t\t\t\t\t{}-{}\n\t\t\t\t\t\t\\tfrac{\\x}{\\lambda}\n\t\t\t\t\t\\right)\n\t\t\t\t\\otherwise[otherwise,]\n\t\t\t\t\\end{ifcases}\n\t\t\t\\end{equation}\n\t\t\tis strictly positive.\n% \t\t\t\n\t\t\tNow, let us consider two subcases:\n\t\t\t\\begin{itemize}\n\t\t\t\\item\\textbf{Case 1a:} \\(0<\\lambda\\leq 2(1+\\nicefrac{\\sigma}{L_{\\sDRS}})\\).\n\t\t\t\n\t\t\t\tThen,\n\t\t\t\t\\(\n\t\t\t\t\t\\sigma\\geq -\\frac{2-\\lambda}{2}L_{\\sDRS}>-L_{\\sDRS}\n\t\t\t\t\\)\n\t\t\t\tand we can take \\(L=L_{\\sDRS}\\).\n\t\t\t\tConsequently, \\(p=\\nicefrac{\\sigma}{L_{\\sDRS}}\\), \\(\\x=\\gamma L_{\\sDRS}\\), and \\eqref{eq:c} becomes\n\t\t\t\t\\begin{equation}\\label{eq:cbar}\n\t\t\t\t\t\\frac{c}{L_{\\sDRS}}\n\t\t\t\t{}={}\n\t\t\t\t\t\\frac{2-\\lambda}{2\\lambda\\gamma L_{\\sDRS}}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\begin{ifcases}\n\t\t\t\t\t\t\\tfrac{p}{2(1+p)}\n\t\t\t\t\t&\n\t\t\t\t\t\t\\gamma\n\t\t\t\t\t\t{}<{}\n\t\t\t\t\t\t\\tfrac{\\lambda}{2(1+p)}\n\t\t\t\t\t\\\\[3pt]\n\t\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t\t{}-{}\n\t\t\t\t\t\t\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\n\t\t\t\t\t\\otherwise.\n\t\t\t\t\t\\end{ifcases}\n\t\t\t\t\\end{equation}\n\t\t\t\tLet us verify that in this case any \\(\\gamma\\) such that \\(\\gamma<\\nicefrac{1}{L_{\\sDRS}}\\) yields a strictly positive coefficient \\(c\\).\n\t\t\t\tIf \\(0<\\gamma L_{\\sDRS}<\\frac{\\lambda}{2(1+p)}\\leq 1\\), then\n\t\t\t\t\\[\n\t\t\t\t\t\\tfrac{c}{L_{\\sDRS}}\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma L_{\\sDRS}}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{p}{2(1+p)}\n\t\t\t\t{}>{}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{p}{\\lambda}\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac{1+p}{\\lambda}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t{}\\geq{}\n\t\t\t\t\t0,\n\t\t\t\t\\]\n\t\t\t\twhere in the inequality we used the fact that \\(\\lambda<2\\) and \\(p\\leq0\\).\n\t\t\t\tIf instead \\(\\frac{\\lambda}{2(1+p)}<\\gamma L_{\\sDRS}<1\\), then\n\t\t\t\t\\[\n\t\t\t\t\t\\tfrac{c}{L_{\\sDRS}}\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma L_{\\sDRS}}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\n\t\t\t\t{}>{}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{1}{\\lambda}\n\t\t\t\t{}={}\n\t\t\t\t\t0.\n\t\t\t\t\\]\n\t\t\t\tEither way, the sufficient decrease constant \\(c\\) is strictly positive.\n% \t\t\t\tBy imposing \\(c>0\\) we obtain the following conditions on \\(\\gamma\\) and \\(\\lambda\\):\n% \t\t\t\t\\begin{align*}\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\tfrac{\\sigma}{2(1+p)}\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{\\lambda}{2(L_{\\sDRS}+\\sigma)}\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\quad\\cup\\quad &\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\tfrac{L_{\\sDRS}}{2}\n% \t\t\t\t\t\t{}-{}\n% \t\t\t\t\t\t\\tfrac{\\gamma L_{\\sDRS}^2}{\\lambda}\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}\\geq{} &\n% \t\t\t\t\t\t\\tfrac{\\lambda}{2(L_{\\sDRS}+\\sigma)}.\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\shortintertext{%\n% \t\t\t\t\tEquivalently,\n% \t\t\t\t}\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t(1+p)(2-\\lambda)\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\lambda p\\xi\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\xi\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{\\lambda}{2(1+p)}\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\quad\\cup\\quad &\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}>{} &\n% \t\t\t\t\t\t2\\xi^2\n% \t\t\t\t\t\t{}-{}\n% \t\t\t\t\t\t\\lambda\\xi\n% \t\t\t\t\t\t{}-{}\n% \t\t\t\t\t\t(2-\\lambda)\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\xi\n% \t\t\t\t\t{}\\geq{} &\n% \t\t\t\t\t\t\\tfrac{\\lambda}{2(1+p)}.\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\shortintertext{%\n% \t\t\t\t\tSince \\(\\xi<1\\), notice that\n% \t\t\t\t\t\\(\n% \t\t\t\t\t\t(1+p)(2-\\lambda)\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\lambda p\\xi\n% \t\t\t\t\t{}>{}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t\\)\n% \t\t\t\t\tis always satisfied when \\(2-\\lambda\\geq-2p\\), as in the case being considered.\n% \t\t\t\t\tThis is evident if \\(p=0\\) (since \\(\\lambda<2\\)); if instead \\(p<0\\), one has\n% \t\t\t\t\t\\(\n% \t\t\t\t\t\t(1+p)(2-\\lambda)\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\lambda p\\xi\n% \t\t\t\t\t{}\\geq{}\n% \t\t\t\t\t\t-2p(1+p)+p\\lambda\\xi\n% \t\t\t\t\t{}>{}\n% \t\t\t\t\t\t-p(2(1+p)-\\lambda)\n% \t\t\t\t\t{}\\geq{}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t\\).\n% \t\t\t\t\tAll conditions then reduce to\n% \t\t\t\t}\n% \t\t\t\t\t\\xi\n% \t\t\t\t{}<{}\n% \t\t\t\t\t\\tfrac{\\lambda}{2(1+p)}\n% \t\t\t\t\\quad\\cup\\quad &\n% \t\t\t\t\t\\tfrac{\\lambda}{2(1+p)}\n% \t\t\t\t{}\\leq{}\n% \t\t\t\t\t\\xi\n% \t\t\t\t{}<{}\n% \t\t\t\t\t1,\n% \t\t\t\t\\end{align*}\n% \t\t\t\tthat is, \\(\\gamma L_{\\sDRS}=\\xi<1\\).\n\t\t\t\tSince \\(\\sigma=-[\\sigma_{\\sDRS}]_-\\) and\n\t\t\t\t\\[\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{\\sigma}{2(1+p)}\n\t\t\t\t{}\\leq{}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{L_{\\sDRS}}{2}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{\\gamma L_{\\sDRS}^2}{\\lambda}\n\t\t\t\t\\quad\\Leftrightarrow\\quad\n\t\t\t\t\t\\gamma\n\t\t\t\t{}\\leq{}\n\t\t\t\t\t\\tfrac{\\lambda}{2(L_{\\sDRS}+\\sigma)},\n\t\t\t\t\\]\n\t\t\t\tfrom \\eqref{eq:cbar} we conclude that \\(c\\) is as in \\eqref{eq:DRE:SD}.\n\t\t\t%%%%\n\t\t\t%%%%\n\t\t\t%%%%\n\t\t\t\\item\\textbf{Case 1b:} \\(2(1+\\nicefrac{\\sigma}{L_{\\sDRS}})<\\lambda<2\\).\n\t\t\t\t\n\t\t\t\tNecessarily \\(\\sigma<0\\), for otherwise the range of \\(\\lambda\\) would be empty.\n\t\t\t\tIn particular, \\(\\sigma=\\sigma_{\\sDRS}\\), and the lower bound on \\(\\lambda\\) can be expressed as\n\t\t\t\t\\(\n\t\t\t\t\t\\sigma_{\\sDRS}<-\\frac{2-\\lambda}{2}L_{\\sDRS}\n\t\t\t\t\\).\n\t\t\t\tConsequently, \\(L\\coloneqq\\tfrac{-2\\sigma_{\\sDRS}}{2-\\lambda}\\) is strictly larger than \\(L_{\\sDRS}\\), and in particular \\(\\sigma+L=\\sigma_{\\sDRS}+L>0\\).\n\t\t\t\tThe ratio of \\(\\sigma\\) and \\(L\\) is thus \\(p=\\tfrac\\lambda2-1\\), and \\eqref{eq:c} becomes\n\t\t\t\t\\begin{equation}\\label{eq:c2}\n\t\t\t\t\tc\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\begin{ifcases}\n\t\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n\t\t\t\t\t&\n\t\t\t\t\t\t\\gamma\n\t\t\t\t\t\t{}<{}\n\t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n\t\t\t\t\t\\\\[3pt]\n\t\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}-\\tfrac{\\gamma L_{\\sDRS}^2}{\\lambda}+\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}\\lambda}L_{\\sDRS}^2\n\t\t\t\t\t\\otherwise.\n\t\t\t\t\t\\end{ifcases}\n\t\t\t\t\\end{equation}\n\t\t\t\tLet us show that, when\n\t\t\t\t\\(\n\t\t\t\t\t\\gamma\n\t\t\t\t{}<{}\n\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac1L\n\t\t\t\t\\),\n\t\t\t\talso in this case the sufficient decrease constant \\(c\\) is strictly positive.\n\t\t\t\tWe have\n\t\t\t\t\\[\n\t\t\t\t\t\\tfrac cL\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma L}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n\t\t\t\t\t\\tfrac1L\n\t\t\t\t{}>{}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n\t\t\t\t{}={}\n\t\t\t\t\t0,\n\t\t\t\t\\]\n\t\t\t\thence the claim.\n\t\t\t\tThis concludes the proof for the case \\(\\lambda\\in(0,2)\\).\n% \t\t\t\tBy imposing \\(c>0\\) this reduces to\n% \t\t\t\t\\begin{align*}\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\quad\\cup\\quad &\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n% \t\t\t\t\t\t{}-{}\n% \t\t\t\t\t\t\\tfrac{\\gamma L_{\\sDRS}^2}{\\lambda}\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}\\lambda}L_{\\sDRS}^2\n% \t\t\t\t\t{}={}\n% \t\t\t\t\t\t\\smashoverbracket{\n% \t\t\t\t\t\t\t\\tfrac1\\lambda\n% \t\t\t\t\t\t\t\\bigl(\n% \t\t\t\t\t\t\t\t\\gamma L_{\\sDRS}^2-\\sigma_{\\sDRS}\n% \t\t\t\t\t\t\t\\bigr)\n% \t\t\t\t\t\t}{>0}\n% \t\t\t\t\t\t\\bigl(\n% \t\t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\gamma\\sigma_{\\sDRS}}\n% \t\t\t\t\t\t\t{}-{}\n% \t\t\t\t\t\t\t1\n% \t\t\t\t\t\t\\bigr)\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}\\geq{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}.\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\shortintertext{Equivalently,}\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t\t{}<{}\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t\t{}<{}\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\quad\\cup\\quad &\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}<{}\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}\\geq{}\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}},\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\end{align*}\n% \t\t\t\tthat is,\n% \t\t\t\t\\(\n% \t\t\t\t\t\\gamma\n% \t\t\t\t{}<{}\n% \t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t\\).\n% \t\t\t\tFrom \\eqref{eq:c2} it follows that\n% \t\t\t\t\\(\n% \t\t\t\t\tc\n% \t\t\t\t{}={}\n% \t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t{}+{}\n% \t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n% \t\t\t\t\\).%\n\t\t\t\\end{itemize}\n% \t\t\tLet us recap {\\bf Case 1}:\n% \t\t\t\\[\n% \t\t\t\t\\begin{cases}[@{\\gamma<{}} l<{,} @{~~c={}} l @{\\quad\\text{if~~}} l]\n% \t\t\t\t\t\\tfrac{1}{L_{\\sDRS}}\n% \t\t\t\t&\n% \t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t{}-{}\n% \t\t\t\t\t\\max\\set{\n% \t\t\t\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{2(1-[p_{\\sDRS}]_-)},\\,\n% \t\t\t\t\t\tL_{\\sDRS}\\bigl(\\tfrac12-\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\\bigr)\n% \t\t\t\t\t}\n% \t\t\t\t&\n% \t\t\t\t\t0<\\lambda<2(1-[p_{\\sDRS}]_-),\n% \t\t\t\t\\\\\n% \t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t&\n% \t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t{}-{}\n% \t\t\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{\\lambda}\n% \t\t\t\t&\n% \t\t\t\t\t2(1-[p_{\\sDRS}]_-)<\\lambda<2.\n% \t\t\t\t\\end{cases}\n% \t\t\t\\]\n% \t\t\tWe may unify the two cases with the more compact notation\n% \t\t\t\\[\n% \t\t\t\t\\gamma\n% \t\t\t{}<{}\n% \t\t\t\t\\min\\set{\n% \t\t\t\t\t\\tfrac{1}{L_{\\sDRS}},\\,\n% \t\t\t\t\t\\tfrac{2-\\lambda}{-2[\\sigma_{\\sDRS}]_-}\n% \t\t\t\t}\n% \t\t\t\\quad\\text{and}\\quad\n% \t\t\t\tc\n% \t\t\t{}={}\n% \t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t{}-{}\n% \t\t\t\t\\min\\set{\n% \t\t\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{\\lambda},~\n% \t\t\t\t\t\\max\\set{\n% \t\t\t\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{2(1-[p_{\\sDRS}]_-)},\\,\n% \t\t\t\t\t\tL_{\\sDRS}\\bigl(\\tfrac12-\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\\bigr)\n% \t\t\t\t\t}\n% \t\t\t\t},\n% \t\t\t\\]\n% \t\t\tthus obtaining the range of \\(\\gamma\\) and the value of \\(c\\) indicated in \\eqref{eq:c_nonstr}.\n\t\t\t%%%\n\t\t\t%%%\n\t\t\t%%%\n\t\t\\item\\textbf{Case 2: \\(\\lambda\\geq2\\).}\n\n\t\t\t\\noindent\n\t\t\tIn this case we need to assume that \\(\\sDRS\\) is strongly convex, that is, that \\(\\sigma_{\\sDRS}>0\\).\n\t\t\tInstead of considering a single expression of \\(\\rho\\), we will rather take a convex combination of those in \\cref{thm:hypo:rho0,thm:hypo:rho1}, namely\n\t\t\t\\[\n\t\t\t\t\\rho(u,u^+)\n\t\t\t{}={}\n\t\t\t\t(1-\\alpha)\\tfrac{\\sigma_{\\sDRS}}{2}\\|u-u^+\\|^2\n\t\t\t\t{}+{}\n\t\t\t\t\\alpha\\tfrac{1}{2L_{\\sDRS}}\\|\\nabla\\sDRS(u)-\\nabla\\sDRS(u^+)\\|^2\n\t\t\t\\]\n\t\t\tfor some \\(\\alpha\\in[0,1]\\) to be determined.\n\t\t\t\\eqref{eq:DRE:SD0} then becomes\n\t\t\t\\[\n\t\t\t\t\\tfrac{\n\t\t\t\t\t\\DRE(s)-\\DRE(s^+)\n\t\t\t\t}{\n\t\t\t\t\tL_{\\sDRS}\n\t\t\t\t}\n\t\t\t{}\\geq{}\n\t\t\t\t\\left(\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\xi}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{(1-\\alpha)p}{2}\n\t\t\t\t\\right)\n\t\t\t\t\\|u-u^+\\|^2\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{L_{\\sDRS}^2}\n\t\t\t\t\\left(\n\t\t\t\t\t\\tfrac{\\alpha}{2}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{\\xi}{\\lambda}\n\t\t\t\t\\right)\n\t\t\t\t\\|\\nabla\\sDRS(u)-\\nabla\\sDRS(u^+)\\|^2,\n\t\t\t\\]\n\t\t\twhere \\(\\x\\coloneqq\\gamma L_{\\sDRS}\\) and \\(p\\coloneqq\\nicefrac{\\sigma_{\\sDRS}}{L_{\\sDRS}}\\in(0,1]\\).\n\t\t\tBy restricting \\(\\xi\\in(0,1)\\), since \\(\\lambda\\geq2\\) one can take\n\t\t\t\\(\n\t\t\t\t\\alpha\n\t\t\t{}\\coloneqq{}\n\t\t\t\t\\nicefrac{2\\xi}{\\lambda}\n\t\t\t{}\\in{}\n\t\t\t\t(0,1)\n\t\t\t\\)\n\t\t\tto make the coefficient multiplying the gradient norm vanish.\n\t\t\tWe then obtain\n\t\t\t\\begin{equation}\\label{eq:clambda2}\n\t\t\t\t\\tfrac{c}{L_{\\sDRS}}\n\t\t\t{}={}\n\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\xi}\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{(\\lambda-2\\xi)p}{2\\lambda}.\n\t\t\t\\end{equation}\n\t\t\tImposing \\(c>0\\) results in the following second-order equation in variable \\(\\xi\\),\n\t\t\t\\begin{equation}\\label{eq:xi}\n\t\t\t\t2p\\xi^2-p\\lambda\\xi+(\\lambda-2)\n\t\t\t{}<{}\n\t\t\t\t0.\n\t\t\t\\end{equation}\n\t\t\tThe discriminant is\n\t\t\t\\(\n\t\t\t\t\\Delta\n\t\t\t{}\\coloneqq{}\n\t\t\t\t(p\\lambda)^2\n\t\t\t\t{}-{}\n\t\t\t\t8p(\\lambda-2)\n\t\t\t\\),\n\t\t\twhich, for \\(\\lambda\\geq2\\), is strictly positive iff\n\t\t\t\\[\n\t\t\t\t2\n\t\t\t{}\\leq{}\n\t\t\t\t\\lambda\n\t\t\t{}<{}\n\t\t\t\t\\tfrac{4}{1+\\sqrt{1-p}}\n\t\t\t~~\\vee~~\n\t\t\t\t\\lambda\n\t\t\t{}>{}\n\t\t\t\t\\tfrac{4}{1-\\sqrt{1-p}}.\n\t\t\t\\]\n\t\t\tDenoting\n\t\t\t\\(\n\t\t\t\t\\delta\n\t\t\t{}\\coloneqq{}\n\t\t\t\t\\sqrt\\Delta\n\t\t\t{}={}\n\t\t\t\t\\sqrt{\n\t\t\t\t\t(p\\lambda)^2\n\t\t\t\t\t{}-{}\n\t\t\t\t\t8p(\\lambda-2)\n\t\t\t\t}\n\t\t\t\\),\n\t\t\tthe solution to \\eqref{eq:xi} is\n\t\t\t\\(\n\t\t\t\t\\tfrac{\n\t\t\t\t\tp\\lambda\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\delta\n\t\t\t\t}{\n\t\t\t\t\t4p\n\t\t\t\t}\n\t\t\t{}<{}\n\t\t\t\t\\x\n\t\t\t{}<{}\n\t\t\t\t\\tfrac{\n\t\t\t\t\tp\\lambda\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\delta\n\t\t\t\t}{\n\t\t\t\t\t4p\n\t\t\t\t}\n\t\t\t\\).\n\t\t\tHowever, the case \\(\\lambda\\geq 4\\) has to be discarded, as\n\t\t\t\\(\n\t\t\t\t\\tfrac{p\\lambda-\\delta}{4p}\n\t\t\t{}>{}\n\t\t\t\t1\n\t\t\t\\)\n\t\t\tin this case, contradicting the fact that \\(p\\leq1\\).\n\t\t\tTo see this, suppose \\(\\lambda\\geq4\\).\n\t\t\tThen,\n\t\t\t\\begin{align*}\n\t\t\t\t\\tfrac{\n\t\t\t\t\tp\\lambda\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\delta\n\t\t\t\t}{\n\t\t\t\t\t4p\n\t\t\t\t}\n\t\t\t{}<{}\n\t\t\t\t1\n\t\t\t~~\\Leftrightarrow~~ &\n\t\t\t\tp(\\lambda-4)\n\t\t\t{}<{}\n\t\t\t\t\\delta\n\t\t\t\\\\\n\t\t\t~~\\Leftrightarrow~~ &\n\t\t\t\tp^2(\\lambda-4)^2\n\t\t\t{}<{}\n\t\t\t\t\\Delta\n\t\t\t{}={}\n\t\t\t\t(p\\lambda)^2\n\t\t\t\t{}-{}\n\t\t\t\t8p(\\lambda-2)\n\t\t\t\\\\\n\t\t\t~~\\Leftrightarrow~~ &\n\t\t\t\tp(2-\\lambda)\n\t\t\t{}<{}\n\t\t\t\t2-\\lambda,\n\t\t\t\\end{align*}\n\t\t\thence \\(p>1\\), which contradicts the fact that \\(\\sigma_{\\sDRS}\\leq L_{\\sDRS}\\).\n% \t\t\tHere and in the second implication we used the assumption \\(\\lambda\\geq4\\).\n\t\t\tThus, the only feasible ranges are the ones given in \\eqref{eq:lambda>=2}, hence the claimed sufficient decrease constant \\(c\\), cf. \\eqref{eq:clambda2}.\n\t\t\t\\qedhere\n\t\t\\end{proofitemize}\n% \t\tAlso in this case, the claimed range of \\(\\gamma\\) and value of \\(c\\) have been found.\n\t\\end{proof}","sha256_utf8":"0e3ddc9aa22c3c356ae3ebb0f416a53fca0a82ba2228d37170f7edb973731d50"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"写出下一迭代的包络最小值","source_units":[{"start":73436,"end":73823,"start_line":1547,"end_line":1565,"raw_tex":"\\begin{proof}\n\t\t\\newcommand\\x\\xi%\n\t\tLet \\((u^+,v^+)\\) be generated by one \\ref{DRS} iteration starting at \\(s^+\\).\n\t\tThen,\n\t\t\\begin{align*}\n\t\t\t\\DRE(s^+)\n\t\t{}={} &\n\t\t\t\\min_{w\\in\\R^n}\\set{\n\t\t\t\t\\sDRS(u^+)\n\t\t\t\t{}+{}\n\t\t\t\t\\nsDRS(w)\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)}{w-u^+}\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|w-u^+\\|^2\n\t\t\t}\n\t\t\\end{align*}\n\t\tand the minimum is attained at \\(w=v^+\\).\n","sha256_utf8":"cee0426bcebbef7fa6f05f77fbd92926115e48bdb7412b63b8c8c0ddc472ad0a"}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","title":"以旧近端点比较两次包络值","source_units":[{"start":73823,"end":75148,"start_line":1566,"end_line":1639,"raw_tex":"\t\tTherefore, letting \\(\\rho\\) be as in \\cref{thm:rho},\n\t\t{\\mathtight[0.9]%\n\t\t\t\\begin{align*}\n\t\t\t\t\\DRE(s^+)\n\t\t\t{}\\leq{} &\n\t\t\t\t\\sDRS(u^+)\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)}{v-u^+}\n\t\t\t\t{}+{}\n\t\t\t\t\\nsDRS(v)\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|u^+-v\\|^2\n\t\t\t\\\\\n\t\t\t{}={} &\n\t\t\t\t\\smashunderbracket[0.5pt]{\n\t\t\t\t\t\\sDRS(u^+)\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)}{u-u^+}\n\t\t\t\t}{}\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)}{v-u}\n\t\t\t\t{}+{}\n\t\t\t\t\\nsDRS(v)\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|u^+-v\\|^2\n\t\t\t\\\\\n\t\t\t% \t\t\t\\dueto{\\cref{thm:rho}}\n\t\t\t{}\\overrel[\\leq]{\\llap{\\cref{thm:rho}}}{} &\n\t\t\t\t\\smashoverbracket[0.5pt]{\n\t\t\t\t\t\\sDRS(u)\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\rho(u,u^+)\n\t\t\t\t}{}\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)}{v-u}\n\t\t\t\t{}+{}\n\t\t\t\t\\nsDRS(v)\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|u^+-v\\|^2\n\t\t\t\\\\\n\t\t\t{}={} &\n\t\t\t\t\\sDRS(u)\n\t\t\t\t{}-{}\n\t\t\t\t\\rho(u,u^+)\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u)}{v-u}\n% \t\t\t\t\\innprod{\\nabla\\sDRS(\\fillwidthof[c]{u^+}{u})}{v-u}\n\t\t\t\t{}+{}\n\t\t\t\t\\nsDRS(v)\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|u^+-v\\|^2\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t{+{}}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)-\\nabla\\sDRS(u)}{v-u}\n\t\t\t\\\\\n\t\t\t{}={} &\n\t\t\t\t\\DRE(s)\n\t\t\t\t{}-{}\n\t\t\t\t\\rho(u,u^+)\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla\\sDRS(u^+)-\\nabla\\sDRS(u)}{v-u}\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\n\t\t\t\t\\|u-u^+\\|^2\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac1\\gamma\n\t\t\t\t\\innprod{u^+-u}{u-v}.\n\t\t\t\\end{align*}%\n\t\t}%\n","sha256_utf8":"6a3306ced4728e264c466eda4ce967676a7a93d9c96726a8ec89ca187f2920b9"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"近端刻画化简为基本下降式","source_units":[{"start":75148,"end":75618,"start_line":1640,"end_line":1663,"raw_tex":"\t\tSince\n\t\t\\(\n\t\t\tu-v\n\t\t{}={}\n\t\t\t\\tfrac1\\lambda(s-s^+)\n\t\t{}={}\n\t\t\t\\tfrac1\\lambda(u-u^+)\n\t\t\t{}+{}\n\t\t\t\\tfrac\\gamma\\lambda(\\nabla\\sDRS(u)-\\nabla\\sDRS(u^+))\n\t\t\\),\n\t\tas it follows from \\cref{thm:proxfEquiv}, it all simplifies to\n\t\t\\begin{equation}\\label{eq:DRE:SD0}\n\t\t\t\\DRE(s)\n\t\t\t{}-{}\n\t\t\t\\DRE(s^+)\n\t\t{}\\geq{}\n\t\t\t\\tfrac{2-\\lambda}{2\\gamma\\lambda}\n\t\t\t\\|u-u^+\\|^2\n\t\t\t{}-{}\n\t\t\t\\tfrac\\gamma\\lambda\n\t\t\t\\|\\nabla\\sDRS(u^+)-\\nabla\\sDRS(u)\\|^2\n\t\t\t{}+{}\n\t\t\t\\rho(u,u^+).\n\t\t\\end{equation}\n","sha256_utf8":"1047f5edd1c11d77cd9a4404ee2a41df89dd71f3bda1ef06708e40d4f2a1884e"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"近端强单调性转换下降度量","source_units":[{"start":75618,"end":76003,"start_line":1664,"end_line":1678,"raw_tex":"\t\tIt will suffice to show that\n\t\t\\[\n\t\t\t\\DRE(s)-\\DRE(s^+)\n\t\t{}\\geq{}\n\t\t\tc\n\t\t\t\\|u-u^+\\|^2;\n\t\t\\]\n\t\tinequality \\eqref{eq:DRE:SD} will then follow from the \\(\\tfrac{1}{1+\\gamma L_{\\sDRS}}\\)-strong monotonicity of \\(\\prox_{\\gamma\\sDRS}\\), see \\cref{thm:proxfLip}.\n% \t\t\\[\n% \t\t\t\\|u-u^+\\|^2\n% \t\t{}\\geq{}\n% \t\t\t\\tfrac{1}{(1+\\gamma L_{\\sDRS})^2}\n% \t\t\t\\|s-s^+\\|^2\n% \t\t\\]\n\t\tWe now proceed by cases.\n","sha256_utf8":"35479890b42d339ea149d57e8ea58bac54568f6916343c2b63c589de5cf7bc7f"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"小松弛参数下构造下降系数","source_units":[{"start":76003,"end":77532,"start_line":1679,"end_line":1735,"raw_tex":"\t\t\\begin{proofitemize}\n\t\t\\item\\textbf{Case 1: \\(\\lambda\\in(0,2)\\).}\n\t\t\t\n\t\t\t\\noindent\n\t\t\tLet \\(\\sigma\\coloneqq-[\\sigma_{\\sDRS}]_-=\\min\\set{\\sigma_{\\sDRS},0}\\) and \\(L\\geq L_{\\sDRS}\\) be such that \\(L+\\sigma>0\\); the value of such an \\(L\\) will be fixed later.\n\t\t\tThen, \\(\\sigma\\leq0\\) and \\(\\sDRS\\) is \\(L\\)-smooth and \\(\\sigma\\)-hypoconvex.\n\t\t\tWe may thus choose \\(\\rho(u,u^+)\\) as in \\cref{thm:hypo:rho1} with these values of \\(L\\) and \\(\\sigma\\).\n\t\t\tInequality \\eqref{eq:DRE:SD0} then becomes\n\t\t\t\\[\n\t\t\t\t\\tfrac{\\DRE(s)-\\DRE(s^+)}{L}\n\t\t\t{}\\geq{}\n\t\t\t\t\\left(\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\x}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{p}{2(1+p)}\n\t\t\t\t\\right)\n\t\t\t\t\\|u^+-u\\|^2\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{L^2}\n\t\t\t\t\\left(\n\t\t\t\t\t\\tfrac{1}{2(1+p)}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{\\x}{\\lambda}\n\t\t\t\t\\right)\n\t\t\t\t\\|\\nabla\\sDRS(u^+)-\\nabla\\sDRS(u)\\|^2,\n\t\t\t\\]\n\t\t\twhere \\(\\x\\coloneqq\\gamma L\\) and \\(p\\coloneqq\\nicefrac\\sigma L\\in(-1,0]\\).\n\t\t\tSince \\(\\nabla\\sDRS\\) is \\(L_{\\sDRS}\\)-Lipschitz continuous, the claim holds provided that the constant\n\t\t\t\\begin{equation}\\label{eq:c}\n\t\t\t\t\\frac cL\n\t\t\t{}={}\n\t\t\t\t\\begin{ifcases}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\x}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{p}{2(1+p)}\n\t\t\t\t&\n\t\t\t\t\t0\n\t\t\t\t\t{}<{}\n\t\t\t\t\t\\tfrac{1}{2(1+p)}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{\\x}{\\lambda}\n\t\t\t\t\\\\[3pt]\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\x}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{p}{2(1+p)}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{L_{\\sDRS}^2}{L^2}\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t\\tfrac{1}{2(1+p)}\n\t\t\t\t\t\t{}-{}\n\t\t\t\t\t\t\\tfrac{\\x}{\\lambda}\n\t\t\t\t\t\\right)\n\t\t\t\t\\otherwise[otherwise,]\n\t\t\t\t\\end{ifcases}\n\t\t\t\\end{equation}\n\t\t\tis strictly positive.\n% \t\t\t\n","sha256_utf8":"d2c27b8a6e36cd40daf2fec0a04fe10146fb2b0c59b2f08e1961a3b0810b4a33"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"第一子情形固定光滑模量","source_units":[{"start":77532,"end":78270,"start_line":1736,"end_line":1763,"raw_tex":"\t\t\tNow, let us consider two subcases:\n\t\t\t\\begin{itemize}\n\t\t\t\\item\\textbf{Case 1a:} \\(0<\\lambda\\leq 2(1+\\nicefrac{\\sigma}{L_{\\sDRS}})\\).\n\t\t\t\n\t\t\t\tThen,\n\t\t\t\t\\(\n\t\t\t\t\t\\sigma\\geq -\\frac{2-\\lambda}{2}L_{\\sDRS}>-L_{\\sDRS}\n\t\t\t\t\\)\n\t\t\t\tand we can take \\(L=L_{\\sDRS}\\).\n\t\t\t\tConsequently, \\(p=\\nicefrac{\\sigma}{L_{\\sDRS}}\\), \\(\\x=\\gamma L_{\\sDRS}\\), and \\eqref{eq:c} becomes\n\t\t\t\t\\begin{equation}\\label{eq:cbar}\n\t\t\t\t\t\\frac{c}{L_{\\sDRS}}\n\t\t\t\t{}={}\n\t\t\t\t\t\\frac{2-\\lambda}{2\\lambda\\gamma L_{\\sDRS}}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\begin{ifcases}\n\t\t\t\t\t\t\\tfrac{p}{2(1+p)}\n\t\t\t\t\t&\n\t\t\t\t\t\t\\gamma\n\t\t\t\t\t\t{}<{}\n\t\t\t\t\t\t\\tfrac{\\lambda}{2(1+p)}\n\t\t\t\t\t\\\\[3pt]\n\t\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t\t{}-{}\n\t\t\t\t\t\t\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\n\t\t\t\t\t\\otherwise.\n\t\t\t\t\t\\end{ifcases}\n\t\t\t\t\\end{equation}\n","sha256_utf8":"f39487b990e1ae430bb20314df9d44828696a937bee06c4c9cbc324cab76924e"}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","title":"第一子情形的小步长系数为正","source_units":[{"start":78270,"end":78854,"start_line":1764,"end_line":1783,"raw_tex":"\t\t\t\tLet us verify that in this case any \\(\\gamma\\) such that \\(\\gamma<\\nicefrac{1}{L_{\\sDRS}}\\) yields a strictly positive coefficient \\(c\\).\n\t\t\t\tIf \\(0<\\gamma L_{\\sDRS}<\\frac{\\lambda}{2(1+p)}\\leq 1\\), then\n\t\t\t\t\\[\n\t\t\t\t\t\\tfrac{c}{L_{\\sDRS}}\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma L_{\\sDRS}}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{p}{2(1+p)}\n\t\t\t\t{}>{}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{p}{\\lambda}\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac{1+p}{\\lambda}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t{}\\geq{}\n\t\t\t\t\t0,\n\t\t\t\t\\]\n\t\t\t\twhere in the inequality we used the fact that \\(\\lambda<2\\) and \\(p\\leq0\\).\n","sha256_utf8":"753a89f2b3e0140445aeadcc9c841eee6fefb8c9d19527d5185939aba505e155"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"第一子情形的另一分支系数为正","source_units":[{"start":78854,"end":81251,"start_line":1784,"end_line":1892,"raw_tex":"\t\t\t\tIf instead \\(\\frac{\\lambda}{2(1+p)}<\\gamma L_{\\sDRS}<1\\), then\n\t\t\t\t\\[\n\t\t\t\t\t\\tfrac{c}{L_{\\sDRS}}\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma L_{\\sDRS}}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\n\t\t\t\t{}>{}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{1}{\\lambda}\n\t\t\t\t{}={}\n\t\t\t\t\t0.\n\t\t\t\t\\]\n\t\t\t\tEither way, the sufficient decrease constant \\(c\\) is strictly positive.\n% \t\t\t\tBy imposing \\(c>0\\) we obtain the following conditions on \\(\\gamma\\) and \\(\\lambda\\):\n% \t\t\t\t\\begin{align*}\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\tfrac{\\sigma}{2(1+p)}\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{\\lambda}{2(L_{\\sDRS}+\\sigma)}\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\quad\\cup\\quad &\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\tfrac{L_{\\sDRS}}{2}\n% \t\t\t\t\t\t{}-{}\n% \t\t\t\t\t\t\\tfrac{\\gamma L_{\\sDRS}^2}{\\lambda}\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}\\geq{} &\n% \t\t\t\t\t\t\\tfrac{\\lambda}{2(L_{\\sDRS}+\\sigma)}.\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\shortintertext{%\n% \t\t\t\t\tEquivalently,\n% \t\t\t\t}\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t(1+p)(2-\\lambda)\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\lambda p\\xi\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\xi\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{\\lambda}{2(1+p)}\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\quad\\cup\\quad &\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}>{} &\n% \t\t\t\t\t\t2\\xi^2\n% \t\t\t\t\t\t{}-{}\n% \t\t\t\t\t\t\\lambda\\xi\n% \t\t\t\t\t\t{}-{}\n% \t\t\t\t\t\t(2-\\lambda)\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\xi\n% \t\t\t\t\t{}\\geq{} &\n% \t\t\t\t\t\t\\tfrac{\\lambda}{2(1+p)}.\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\shortintertext{%\n% \t\t\t\t\tSince \\(\\xi<1\\), notice that\n% \t\t\t\t\t\\(\n% \t\t\t\t\t\t(1+p)(2-\\lambda)\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\lambda p\\xi\n% \t\t\t\t\t{}>{}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t\\)\n% \t\t\t\t\tis always satisfied when \\(2-\\lambda\\geq-2p\\), as in the case being considered.\n% \t\t\t\t\tThis is evident if \\(p=0\\) (since \\(\\lambda<2\\)); if instead \\(p<0\\), one has\n% \t\t\t\t\t\\(\n% \t\t\t\t\t\t(1+p)(2-\\lambda)\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\lambda p\\xi\n% \t\t\t\t\t{}\\geq{}\n% \t\t\t\t\t\t-2p(1+p)+p\\lambda\\xi\n% \t\t\t\t\t{}>{}\n% \t\t\t\t\t\t-p(2(1+p)-\\lambda)\n% \t\t\t\t\t{}\\geq{}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t\\).\n% \t\t\t\t\tAll conditions then reduce to\n% \t\t\t\t}\n% \t\t\t\t\t\\xi\n% \t\t\t\t{}<{}\n% \t\t\t\t\t\\tfrac{\\lambda}{2(1+p)}\n% \t\t\t\t\\quad\\cup\\quad &\n% \t\t\t\t\t\\tfrac{\\lambda}{2(1+p)}\n% \t\t\t\t{}\\leq{}\n% \t\t\t\t\t\\xi\n% \t\t\t\t{}<{}\n% \t\t\t\t\t1,\n% \t\t\t\t\\end{align*}\n% \t\t\t\tthat is, \\(\\gamma L_{\\sDRS}=\\xi<1\\).\n","sha256_utf8":"f0540441eef31686f9c1e41b1b6854377207f525c78cda8620bd2c5c36d21f90"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"统一第一子情形的系数表达式","source_units":[{"start":81251,"end":81727,"start_line":1893,"end_line":1912,"raw_tex":"\t\t\t\tSince \\(\\sigma=-[\\sigma_{\\sDRS}]_-\\) and\n\t\t\t\t\\[\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{\\sigma}{2(1+p)}\n\t\t\t\t{}\\leq{}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{L_{\\sDRS}}{2}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{\\gamma L_{\\sDRS}^2}{\\lambda}\n\t\t\t\t\\quad\\Leftrightarrow\\quad\n\t\t\t\t\t\\gamma\n\t\t\t\t{}\\leq{}\n\t\t\t\t\t\\tfrac{\\lambda}{2(L_{\\sDRS}+\\sigma)},\n\t\t\t\t\\]\n\t\t\t\tfrom \\eqref{eq:cbar} we conclude that \\(c\\) is as in \\eqref{eq:DRE:SD}.\n\t\t\t%%%%\n\t\t\t%%%%\n\t\t\t%%%%\n","sha256_utf8":"3d0addf260be462ca9b88107b1485075300cb545daeea4ee405df7bc9f2dd5d2"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"第二子情形增大光滑模量","source_units":[{"start":81727,"end":82741,"start_line":1913,"end_line":1937,"raw_tex":"\t\t\t\\item\\textbf{Case 1b:} \\(2(1+\\nicefrac{\\sigma}{L_{\\sDRS}})<\\lambda<2\\).\n\t\t\t\t\n\t\t\t\tNecessarily \\(\\sigma<0\\), for otherwise the range of \\(\\lambda\\) would be empty.\n\t\t\t\tIn particular, \\(\\sigma=\\sigma_{\\sDRS}\\), and the lower bound on \\(\\lambda\\) can be expressed as\n\t\t\t\t\\(\n\t\t\t\t\t\\sigma_{\\sDRS}<-\\frac{2-\\lambda}{2}L_{\\sDRS}\n\t\t\t\t\\).\n\t\t\t\tConsequently, \\(L\\coloneqq\\tfrac{-2\\sigma_{\\sDRS}}{2-\\lambda}\\) is strictly larger than \\(L_{\\sDRS}\\), and in particular \\(\\sigma+L=\\sigma_{\\sDRS}+L>0\\).\n\t\t\t\tThe ratio of \\(\\sigma\\) and \\(L\\) is thus \\(p=\\tfrac\\lambda2-1\\), and \\eqref{eq:c} becomes\n\t\t\t\t\\begin{equation}\\label{eq:c2}\n\t\t\t\t\tc\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\begin{ifcases}\n\t\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n\t\t\t\t\t&\n\t\t\t\t\t\t\\gamma\n\t\t\t\t\t\t{}<{}\n\t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n\t\t\t\t\t\\\\[3pt]\n\t\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}-\\tfrac{\\gamma L_{\\sDRS}^2}{\\lambda}+\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}\\lambda}L_{\\sDRS}^2\n\t\t\t\t\t\\otherwise.\n\t\t\t\t\t\\end{ifcases}\n\t\t\t\t\\end{equation}\n","sha256_utf8":"9fe6785882e5843f971bc569f016fafa3b8e0990d3878190e5f4a5abc97bb983"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"验证第二子情形的严格下降","source_units":[{"start":82741,"end":86367,"start_line":1938,"end_line":2089,"raw_tex":"\t\t\t\tLet us show that, when\n\t\t\t\t\\(\n\t\t\t\t\t\\gamma\n\t\t\t\t{}<{}\n\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac1L\n\t\t\t\t\\),\n\t\t\t\talso in this case the sufficient decrease constant \\(c\\) is strictly positive.\n\t\t\t\tWe have\n\t\t\t\t\\[\n\t\t\t\t\t\\tfrac cL\n\t\t\t\t{}={}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma L}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n\t\t\t\t\t\\tfrac1L\n\t\t\t\t{}>{}\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n\t\t\t\t{}={}\n\t\t\t\t\t0,\n\t\t\t\t\\]\n\t\t\t\thence the claim.\n\t\t\t\tThis concludes the proof for the case \\(\\lambda\\in(0,2)\\).\n% \t\t\t\tBy imposing \\(c>0\\) this reduces to\n% \t\t\t\t\\begin{align*}\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\quad\\cup\\quad &\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t0\n% \t\t\t\t\t{}<{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n% \t\t\t\t\t\t{}-{}\n% \t\t\t\t\t\t\\tfrac{\\gamma L_{\\sDRS}^2}{\\lambda}\n% \t\t\t\t\t\t{}+{}\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}\\lambda}L_{\\sDRS}^2\n% \t\t\t\t\t{}={}\n% \t\t\t\t\t\t\\smashoverbracket{\n% \t\t\t\t\t\t\t\\tfrac1\\lambda\n% \t\t\t\t\t\t\t\\bigl(\n% \t\t\t\t\t\t\t\t\\gamma L_{\\sDRS}^2-\\sigma_{\\sDRS}\n% \t\t\t\t\t\t\t\\bigr)\n% \t\t\t\t\t\t}{>0}\n% \t\t\t\t\t\t\\bigl(\n% \t\t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\gamma\\sigma_{\\sDRS}}\n% \t\t\t\t\t\t\t{}-{}\n% \t\t\t\t\t\t\t1\n% \t\t\t\t\t\t\\bigr)\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}\\geq{} &\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}.\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\shortintertext{Equivalently,}\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t\t{}<{}\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t\t{}<{}\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\quad\\cup\\quad &\n% \t\t\t\t\t\\begin{cases}\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}<{}\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t\t\\\\\n% \t\t\t\t\t\t\\gamma\n% \t\t\t\t\t{}\\geq{}\n% \t\t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}},\n% \t\t\t\t\t\\end{cases}\n% \t\t\t\t\\end{align*}\n% \t\t\t\tthat is,\n% \t\t\t\t\\(\n% \t\t\t\t\t\\gamma\n% \t\t\t\t{}<{}\n% \t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t\\).\n% \t\t\t\tFrom \\eqref{eq:c2} it follows that\n% \t\t\t\t\\(\n% \t\t\t\t\tc\n% \t\t\t\t{}={}\n% \t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t{}+{}\n% \t\t\t\t\t\\tfrac{\\sigma_{\\sDRS}}{\\lambda}\n% \t\t\t\t\\).%\n\t\t\t\\end{itemize}\n% \t\t\tLet us recap {\\bf Case 1}:\n% \t\t\t\\[\n% \t\t\t\t\\begin{cases}[@{\\gamma<{}} l<{,} @{~~c={}} l @{\\quad\\text{if~~}} l]\n% \t\t\t\t\t\\tfrac{1}{L_{\\sDRS}}\n% \t\t\t\t&\n% \t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t{}-{}\n% \t\t\t\t\t\\max\\set{\n% \t\t\t\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{2(1-[p_{\\sDRS}]_-)},\\,\n% \t\t\t\t\t\tL_{\\sDRS}\\bigl(\\tfrac12-\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\\bigr)\n% \t\t\t\t\t}\n% \t\t\t\t&\n% \t\t\t\t\t0<\\lambda<2(1-[p_{\\sDRS}]_-),\n% \t\t\t\t\\\\\n% \t\t\t\t\t\\tfrac{2-\\lambda}{-2\\sigma_{\\sDRS}}\n% \t\t\t\t&\n% \t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t\t{}-{}\n% \t\t\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{\\lambda}\n% \t\t\t\t&\n% \t\t\t\t\t2(1-[p_{\\sDRS}]_-)<\\lambda<2.\n% \t\t\t\t\\end{cases}\n% \t\t\t\\]\n% \t\t\tWe may unify the two cases with the more compact notation\n% \t\t\t\\[\n% \t\t\t\t\\gamma\n% \t\t\t{}<{}\n% \t\t\t\t\\min\\set{\n% \t\t\t\t\t\\tfrac{1}{L_{\\sDRS}},\\,\n% \t\t\t\t\t\\tfrac{2-\\lambda}{-2[\\sigma_{\\sDRS}]_-}\n% \t\t\t\t}\n% \t\t\t\\quad\\text{and}\\quad\n% \t\t\t\tc\n% \t\t\t{}={}\n% \t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\gamma}\n% \t\t\t\t{}-{}\n% \t\t\t\t\\min\\set{\n% \t\t\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{\\lambda},~\n% \t\t\t\t\t\\max\\set{\n% \t\t\t\t\t\t\\tfrac{[\\sigma_{\\sDRS}]_-}{2(1-[p_{\\sDRS}]_-)},\\,\n% \t\t\t\t\t\tL_{\\sDRS}\\bigl(\\tfrac12-\\tfrac{\\gamma L_{\\sDRS}}{\\lambda}\\bigr)\n% \t\t\t\t\t}\n% \t\t\t\t},\n% \t\t\t\\]\n% \t\t\tthus obtaining the range of \\(\\gamma\\) and the value of \\(c\\) indicated in \\eqref{eq:c_nonstr}.\n\t\t\t%%%\n\t\t\t%%%\n\t\t\t%%%\n","sha256_utf8":"5654a92d2888377b25eeaec94a49f100323597a02087a7e706d5e695b7d6f800"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"大松弛参数下组合两种下界","source_units":[{"start":86367,"end":87403,"start_line":2090,"end_line":2126,"raw_tex":"\t\t\\item\\textbf{Case 2: \\(\\lambda\\geq2\\).}\n\n\t\t\t\\noindent\n\t\t\tIn this case we need to assume that \\(\\sDRS\\) is strongly convex, that is, that \\(\\sigma_{\\sDRS}>0\\).\n\t\t\tInstead of considering a single expression of \\(\\rho\\), we will rather take a convex combination of those in \\cref{thm:hypo:rho0,thm:hypo:rho1}, namely\n\t\t\t\\[\n\t\t\t\t\\rho(u,u^+)\n\t\t\t{}={}\n\t\t\t\t(1-\\alpha)\\tfrac{\\sigma_{\\sDRS}}{2}\\|u-u^+\\|^2\n\t\t\t\t{}+{}\n\t\t\t\t\\alpha\\tfrac{1}{2L_{\\sDRS}}\\|\\nabla\\sDRS(u)-\\nabla\\sDRS(u^+)\\|^2\n\t\t\t\\]\n\t\t\tfor some \\(\\alpha\\in[0,1]\\) to be determined.\n\t\t\t\\eqref{eq:DRE:SD0} then becomes\n\t\t\t\\[\n\t\t\t\t\\tfrac{\n\t\t\t\t\t\\DRE(s)-\\DRE(s^+)\n\t\t\t\t}{\n\t\t\t\t\tL_{\\sDRS}\n\t\t\t\t}\n\t\t\t{}\\geq{}\n\t\t\t\t\\left(\n\t\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\xi}\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tfrac{(1-\\alpha)p}{2}\n\t\t\t\t\\right)\n\t\t\t\t\\|u-u^+\\|^2\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{L_{\\sDRS}^2}\n\t\t\t\t\\left(\n\t\t\t\t\t\\tfrac{\\alpha}{2}\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\tfrac{\\xi}{\\lambda}\n\t\t\t\t\\right)\n\t\t\t\t\\|\\nabla\\sDRS(u)-\\nabla\\sDRS(u^+)\\|^2,\n\t\t\t\\]\n\t\t\twhere \\(\\x\\coloneqq\\gamma L_{\\sDRS}\\) and \\(p\\coloneqq\\nicefrac{\\sigma_{\\sDRS}}{L_{\\sDRS}}\\in(0,1]\\).\n","sha256_utf8":"0e3823ac045e6bf80a876ba650cf6bfdb96a5fa88ada3ab2792b4303d0d5705a"}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","title":"选择组合权重消去梯度项","source_units":[{"start":87403,"end":87822,"start_line":2127,"end_line":2143,"raw_tex":"\t\t\tBy restricting \\(\\xi\\in(0,1)\\), since \\(\\lambda\\geq2\\) one can take\n\t\t\t\\(\n\t\t\t\t\\alpha\n\t\t\t{}\\coloneqq{}\n\t\t\t\t\\nicefrac{2\\xi}{\\lambda}\n\t\t\t{}\\in{}\n\t\t\t\t(0,1)\n\t\t\t\\)\n\t\t\tto make the coefficient multiplying the gradient norm vanish.\n\t\t\tWe then obtain\n\t\t\t\\begin{equation}\\label{eq:clambda2}\n\t\t\t\t\\tfrac{c}{L_{\\sDRS}}\n\t\t\t{}={}\n\t\t\t\t\\tfrac{2-\\lambda}{2\\lambda\\xi}\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{(\\lambda-2\\xi)p}{2\\lambda}.\n\t\t\t\\end{equation}\n","sha256_utf8":"e26265c8547a5ea6acb13b675e9bb7ded052b4e6b096fdd56ee49e4993e18e4f"}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","title":"求解正下降系数的二次条件","source_units":[{"start":87822,"end":88677,"start_line":2144,"end_line":2201,"raw_tex":"\t\t\tImposing \\(c>0\\) results in the following second-order equation in variable \\(\\xi\\),\n\t\t\t\\begin{equation}\\label{eq:xi}\n\t\t\t\t2p\\xi^2-p\\lambda\\xi+(\\lambda-2)\n\t\t\t{}<{}\n\t\t\t\t0.\n\t\t\t\\end{equation}\n\t\t\tThe discriminant is\n\t\t\t\\(\n\t\t\t\t\\Delta\n\t\t\t{}\\coloneqq{}\n\t\t\t\t(p\\lambda)^2\n\t\t\t\t{}-{}\n\t\t\t\t8p(\\lambda-2)\n\t\t\t\\),\n\t\t\twhich, for \\(\\lambda\\geq2\\), is strictly positive iff\n\t\t\t\\[\n\t\t\t\t2\n\t\t\t{}\\leq{}\n\t\t\t\t\\lambda\n\t\t\t{}<{}\n\t\t\t\t\\tfrac{4}{1+\\sqrt{1-p}}\n\t\t\t~~\\vee~~\n\t\t\t\t\\lambda\n\t\t\t{}>{}\n\t\t\t\t\\tfrac{4}{1-\\sqrt{1-p}}.\n\t\t\t\\]\n\t\t\tDenoting\n\t\t\t\\(\n\t\t\t\t\\delta\n\t\t\t{}\\coloneqq{}\n\t\t\t\t\\sqrt\\Delta\n\t\t\t{}={}\n\t\t\t\t\\sqrt{\n\t\t\t\t\t(p\\lambda)^2\n\t\t\t\t\t{}-{}\n\t\t\t\t\t8p(\\lambda-2)\n\t\t\t\t}\n\t\t\t\\),\n\t\t\tthe solution to \\eqref{eq:xi} is\n\t\t\t\\(\n\t\t\t\t\\tfrac{\n\t\t\t\t\tp\\lambda\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\delta\n\t\t\t\t}{\n\t\t\t\t\t4p\n\t\t\t\t}\n\t\t\t{}<{}\n\t\t\t\t\\x\n\t\t\t{}<{}\n\t\t\t\t\\tfrac{\n\t\t\t\t\tp\\lambda\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\delta\n\t\t\t\t}{\n\t\t\t\t\t4p\n\t\t\t\t}\n\t\t\t\\).\n","sha256_utf8":"f060ebfc21b19b11a52c0758a52bd0aeb1620c5db7fd86f8844b8e94d23ff189"}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","title":"排除过大松弛并完成参数界","source_units":[{"start":88677,"end":89744,"start_line":2202,"end_line":2246,"raw_tex":"\t\t\tHowever, the case \\(\\lambda\\geq 4\\) has to be discarded, as\n\t\t\t\\(\n\t\t\t\t\\tfrac{p\\lambda-\\delta}{4p}\n\t\t\t{}>{}\n\t\t\t\t1\n\t\t\t\\)\n\t\t\tin this case, contradicting the fact that \\(p\\leq1\\).\n\t\t\tTo see this, suppose \\(\\lambda\\geq4\\).\n\t\t\tThen,\n\t\t\t\\begin{align*}\n\t\t\t\t\\tfrac{\n\t\t\t\t\tp\\lambda\n\t\t\t\t\t{}-{}\n\t\t\t\t\t\\delta\n\t\t\t\t}{\n\t\t\t\t\t4p\n\t\t\t\t}\n\t\t\t{}<{}\n\t\t\t\t1\n\t\t\t~~\\Leftrightarrow~~ &\n\t\t\t\tp(\\lambda-4)\n\t\t\t{}<{}\n\t\t\t\t\\delta\n\t\t\t\\\\\n\t\t\t~~\\Leftrightarrow~~ &\n\t\t\t\tp^2(\\lambda-4)^2\n\t\t\t{}<{}\n\t\t\t\t\\Delta\n\t\t\t{}={}\n\t\t\t\t(p\\lambda)^2\n\t\t\t\t{}-{}\n\t\t\t\t8p(\\lambda-2)\n\t\t\t\\\\\n\t\t\t~~\\Leftrightarrow~~ &\n\t\t\t\tp(2-\\lambda)\n\t\t\t{}<{}\n\t\t\t\t2-\\lambda,\n\t\t\t\\end{align*}\n\t\t\thence \\(p>1\\), which contradicts the fact that \\(\\sigma_{\\sDRS}\\leq L_{\\sDRS}\\).\n% \t\t\tHere and in the second implication we used the assumption \\(\\lambda\\geq4\\).\n\t\t\tThus, the only feasible ranges are the ones given in \\eqref{eq:lambda>=2}, hence the claimed sufficient decrease constant \\(c\\), cf. \\eqref{eq:clambda2}.\n\t\t\t\\qedhere\n\t\t\\end{proofitemize}\n% \t\tAlso in this case, the claimed range of \\(\\gamma\\) and value of \\(c\\) have been found.\n\t\\end{proof}","sha256_utf8":"716bcdcc1a8cf7c08f9bd8340053fcf821ee6ed83cc656c34d7ca9b17a69b264"}]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","title":"凸光滑函数的标准不等式","source_units":[],"declared_external":true,"declaration_text":["An L-smooth function satisfies the quadratic descent bound. For a convex L-smooth function q, q(y) >= q(x) + <grad q(x),y-x> + ||grad q(y)-grad q(x)||^2/(2L), and grad q is 1/L-cocoercive. Strong convexity gives strong gradient monotonicity and the usual strongly-convex smooth interpolation inequality. These are the cited Bertsekas and Nesterov boundary facts; the author's mixed-modulus argument is retained in B0214."]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","title":"近端存在性与Moreau正则性","source_units":[],"declared_external":true,"declaration_text":["For a proper lsc prox-bounded h and 0 < gamma < gamma_h, prox_{gamma h} is nonempty and compact valued and h^gamma is finite and locally Lipschitz. Its limiting subdifferential is contained in gamma^{-1}(s-prox_{gamma h}(s)). The strict-differentiability criterion cited as Rockafellar–Wets, Theorem 9.18, supplies differentiability in the singleton case used in B0219. Strongly convex coercive proximal objectives have unique minimizers characterized by stationarity."]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","title":"最优性与光滑次微分运算","source_units":[],"declared_external":true,"declaration_text":["A local minimizer of a proper function satisfies 0 in its regular subdifferential. If f is continuously differentiable, then the regular subdifferential of f+g at x equals grad f(x)+the regular subdifferential of g at x. These are the Fermat and smooth-sum rules cited from Rockafellar–Wets."]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","title":"前向后向包络的连续性","source_units":[],"declared_external":true,"declaration_text":["Under the smooth-plus-proper-lsc assumptions and 0 < gamma < 1/L, the forward-backward envelope is finite and locally Lipschitz, as cited in Themelis et al., Proposition 4.2."]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","title":"前向后向包络的下降界","source_units":[],"declared_external":true,"declaration_text":["For phi=f+g with f L-smooth and v in prox_{gamma g}(u-gamma grad f(u)), the forward-backward envelope satisfies phi(v) <= FBE(u) - (1-gamma L)||u-v||^2/(2 gamma), as cited in Themelis et al., Proposition 4.3."]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","title":"前向后向包络的极小化等价","source_units":[],"declared_external":true,"declaration_text":["For the composite problem under the stated assumptions and 0 < gamma < 1/L, inf phi = inf FBE and argmin phi = argmin FBE, as cited in Themelis et al., Theorem 4.4."]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","title":"平方可和给出最优残差速率","source_units":[],"declared_external":true,"declaration_text":["If a_k >= 0 and sum_k a_k < infinity, then min_{0<=i<=k} a_i = o(1/k). Applying this to squared residual norms gives min_{0<=i<=k} ||r_i|| = o(1/sqrt(k))."]},{"id":"S081","semantic_type":"external_fact","kind":"external_fact","title":"近端映射的外半连续性","source_units":[],"declared_external":true,"declaration_text":["For a proper lsc prox-bounded h and a fixed stepsize below its threshold, if s_j tends to s and z_j in prox_{gamma h}(s_j) tends to z, then z belongs to prox_{gamma h}(s), as invoked through Rockafellar–Wets, Example 5.23(b)."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S006","target_id":"S002","kind":"supports"},{"source_id":"S007","target_id":"S002","kind":"supports"},{"source_id":"S008","target_id":"S002","kind":"supports"},{"source_id":"S009","target_id":"S002","kind":"supports"},{"source_id":"S010","target_id":"S002","kind":"supports"},{"source_id":"S011","target_id":"S002","kind":"supports"},{"source_id":"S012","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S005","kind":"supports"},{"source_id":"S014","target_id":"S059","kind":"supports"},{"source_id":"S014","target_id":"S061","kind":"supports"},{"source_id":"S015","target_id":"S035","kind":"supports"},{"source_id":"S015","target_id":"S041","kind":"supports"},{"source_id":"S015","target_id":"S063","kind":"supports"},{"source_id":"S016","target_id":"S005","kind":"supports"},{"source_id":"S017","target_id":"S010","kind":"supports"},{"source_id":"S017","target_id":"S053","kind":"supports"},{"source_id":"S017","target_id":"S054","kind":"supports"},{"source_id":"S017","target_id":"S055","kind":"supports"},{"source_id":"S017","target_id":"S012","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S018","target_id":"S009","kind":"supports"},{"source_id":"S018","target_id":"S011","kind":"supports"},{"source_id":"S019","target_id":"S028","kind":"supports"},{"source_id":"S019","target_id":"S029","kind":"supports"},{"source_id":"S019","target_id":"S030","kind":"supports"},{"source_id":"S019","target_id":"S035","kind":"supports"},{"source_id":"S019","target_id":"S037","kind":"supports"},{"source_id":"S019","target_id":"S039","kind":"supports"},{"source_id":"S019","target_id":"S042","kind":"supports"},{"source_id":"S019","target_id":"S063","kind":"supports"},{"source_id":"S019","target_id":"S070","kind":"supports"},{"source_id":"S019","target_id":"S007","kind":"supports"},{"source_id":"S019","target_id":"S011","kind":"supports"},{"source_id":"S020","target_id":"S035","kind":"supports"},{"source_id":"S020","target_id":"S036","kind":"supports"},{"source_id":"S020","target_id":"S038","kind":"supports"},{"source_id":"S020","target_id":"S040","kind":"supports"},{"source_id":"S020","target_id":"S023","kind":"supports"},{"source_id":"S020","target_id":"S009","kind":"supports"},{"source_id":"S020","target_id":"S011","kind":"supports"},{"source_id":"S021","target_id":"S040","kind":"supports"},{"source_id":"S021","target_id":"S045","kind":"supports"},{"source_id":"S021","target_id":"S047","kind":"supports"},{"source_id":"S021","target_id":"S053","kind":"supports"},{"source_id":"S021","target_id":"S058","kind":"supports"},{"source_id":"S021","target_id":"S005","kind":"supports"},{"source_id":"S022","target_id":"S002","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S048","kind":"supports"},{"source_id":"S023","target_id":"S049","kind":"supports"},{"source_id":"S023","target_id":"S055","kind":"supports"},{"source_id":"S023","target_id":"S059","kind":"supports"},{"source_id":"S023","target_id":"S060","kind":"supports"},{"source_id":"S023","target_id":"S008","kind":"supports"},{"source_id":"S023","target_id":"S011","kind":"supports"},{"source_id":"S024","target_id":"S045","kind":"supports"},{"source_id":"S024","target_id":"S049","kind":"supports"},{"source_id":"S024","target_id":"S052","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S026","kind":"supports"},{"source_id":"S031","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S060","kind":"supports"},{"source_id":"S025","target_id":"S063","kind":"supports"},{"source_id":"S025","target_id":"S070","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S035","target_id":"S033","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S033","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S033","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S033","kind":"supports"},{"source_id":"S039","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S041","kind":"supports"},{"source_id":"S032","target_id":"S023","kind":"supports"},{"source_id":"S032","target_id":"S045","kind":"supports"},{"source_id":"S032","target_id":"S052","kind":"supports"},{"source_id":"S032","target_id":"S054","kind":"supports"},{"source_id":"S032","target_id":"S055","kind":"supports"},{"source_id":"S032","target_id":"S056","kind":"supports"},{"source_id":"S032","target_id":"S061","kind":"supports"},{"source_id":"S032","target_id":"S062","kind":"supports"},{"source_id":"S032","target_id":"S007","kind":"supports"},{"source_id":"S032","target_id":"S012","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S042","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S022","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S005","kind":"supports"},{"source_id":"S043","target_id":"S010","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S049","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S054","kind":"supports"},{"source_id":"S046","target_id":"S055","kind":"supports"},{"source_id":"S046","target_id":"S010","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S051","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S051","kind":"supports"},{"source_id":"S056","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S005","kind":"supports"},{"source_id":"S050","target_id":"S012","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S059","target_id":"S058","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S060","target_id":"S058","kind":"supports"},{"source_id":"S061","target_id":"S063","kind":"supports"},{"source_id":"S061","target_id":"S070","kind":"supports"},{"source_id":"S061","target_id":"S058","kind":"supports"},{"source_id":"S062","target_id":"S058","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S063","target_id":"S068","kind":"supports"},{"source_id":"S063","target_id":"S058","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S066","kind":"supports"},{"source_id":"S064","target_id":"S067","kind":"supports"},{"source_id":"S064","target_id":"S058","kind":"supports"},{"source_id":"S065","target_id":"S067","kind":"supports"},{"source_id":"S065","target_id":"S058","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S058","kind":"supports"},{"source_id":"S067","target_id":"S058","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S058","kind":"supports"},{"source_id":"S069","target_id":"S058","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S058","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S058","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S058","kind":"supports"},{"source_id":"S073","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S005","kind":"supports"},{"source_id":"S057","target_id":"S006","kind":"supports"},{"source_id":"S057","target_id":"S012","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S012","kind":"supports"},{"source_id":"S006","target_id":"S010","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S010","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S074","target_id":"S019","kind":"supports"},{"source_id":"S074","target_id":"S029","kind":"supports"},{"source_id":"S074","target_id":"S030","kind":"supports"},{"source_id":"S074","target_id":"S035","kind":"supports"},{"source_id":"S074","target_id":"S037","kind":"supports"},{"source_id":"S074","target_id":"S070","kind":"supports"},{"source_id":"S075","target_id":"S020","kind":"supports"},{"source_id":"S075","target_id":"S040","kind":"supports"},{"source_id":"S075","target_id":"S036","kind":"supports"},{"source_id":"S075","target_id":"S038","kind":"supports"},{"source_id":"S076","target_id":"S020","kind":"supports"},{"source_id":"S076","target_id":"S009","kind":"supports"},{"source_id":"S076","target_id":"S011","kind":"supports"},{"source_id":"S077","target_id":"S045","kind":"supports"},{"source_id":"S078","target_id":"S049","kind":"supports"},{"source_id":"S079","target_id":"S052","kind":"supports"},{"source_id":"S080","target_id":"S005","kind":"supports"},{"source_id":"S081","target_id":"S008","kind":"supports"}]}}
{"record_id":"1710.05509v3","paper":{"title":"On stochastic and deterministic quasi-Newton methods for non-Strongly convex optimization: Asymptotic convergence and rate analysis","authors":["Farzad Yousefian","Angelia Nedić","Uday Shanbhag"],"arxiv_id":"1710.05509","version":"v3","journal":"SIAM Journal on Optimization","doi":"10.1137/17m1152474","abstract_url":"https://arxiv.org/abs/1710.05509v3","pdf_url":"https://arxiv.org/pdf/1710.05509v3"},"context":{"raw_tex":"\\fy{We} consider the following stochastic optimization problem:\n\\begin{equation}\\label{eqn:problem}\n\\min_{x \\in \\Real^n} f(x)\\fy{\\ \\triangleq \\ }\\EXP{F(x,\\xi(\\us{\\omega}))},\\tag{SO}\n\\end{equation}\nwhere $F: \\Real^n\\times\\us{\\Real^d}\\to\\mathbb{R}$ is a \\fy{real-valued} function, the random\nvector $\\xi$ is defined as {$\\xi:\\Omega\n\\rightarrow \\Real^d$}, \\us{$(\\Omega,{\\cal F}, \\mathbb{P})$ denotes the\nassociated probability space\\fy{,} and the expectation \\us{$\\EXP{F(x,\\xi)}$} is\ntaken with respect to $\\mathbb{P}$}. Problem \\eqref{eqn:problem} provides a general framework that can capture a wide range of applications in operations research, machine learning, statistics and control to name a few \\fys{(\\vus{cf.}\\cite{bottou-2010,nocedal15}). Addressing problem \\eqref{eqn:problem}} has led\n\n\n \\textbf{Notation:} A vector $x$ is assumed to be a column vector and\n$x^T$ denotes its transpose, \\an{while} $\\|x\\|$ {denotes} the Euclidean\nvector norm, i.e., $\\|x\\|=\\sqrt{x^Tx}$.  We write \\textit{a.s.} as the\nabbreviation for ``almost surely''.  \\an{For a symmetric matrix $B$, \n$\\lambda_{\\min}(B)$ and $\\lambda_{\\max}(B)$ \\vus{denotes the\nsmallest and largest eigenvalue of $B$}, respectively.} We use $\\EXP{z}$ to\ndenote the expectation of a random variable~$z$. A function $f:X\n\\subset \\mathbb{R}^n\\rightarrow \\mathbb{R}$ is said to be strongly\nconvex with parameter $\\mu>0$, if  $f(y)\\geq f(x)+\\nabla\nf(x)^T(y-x)+\\frac{\\mu}{2}\\|x-y\\|^2,$ for any $x,y \\in X$.  A mapping\n$F:X \\subset \\mathbb{R}^n\\rightarrow \\mathbb{R}$ is \\vus{said to be} Lipschitz\ncontinuous with parameter $L>0$ if for any $x, y \\in X$, we have\n$\\|F(x)-F(y)\\|\\leq L\\|x-y\\|$. For a continuously differentiable\nfunction $f$ with Lipschitz gradients with parameter $L>0$, we have\n$f(y)\\leq f(x)+\\nabla f(x)^T(y-x)+\\frac{L}{2}\\|x-y\\|^2,$ for any $x,y\n\\in X$. For a vector $x \\in \\Real^n$ and a \\fys{nonempty} set $X\n\\subset \\Real^n$, the Euclidean distance of $x$ from $X$ is denoted by\n$dist(x,X)$. We denote the optimal objective value of problem\n\\eqref{eqn:problem} by $f^*$ and the set of the optimal solutions by\n$X^*$.\n\n\nIn this section, we describe a general SQN scheme for solving problem~\\eqref{eqn:problem} and present the main assumptions. Let $x_0\\in \\Real^n$ be an arbitrary initial point, and $x_k$ be generated by the following recursive rule\n\\begin{align}\\label{eqn:LM-cyclic-reg-BFGS}\\tag{IR-SQN}\nx_{k+1}:=x_k -\\gamma_kH_k\\left(\\nabla F(x_k,\\xi_k)+ \\mu_k(x_k-x_0)\\right), \\quad \\hbox{for all } k \\geq 0.\n\\end{align}\n\nHere, $\\g_k$ and $\\mu_k$ are the steplength and the regularization parameter, respectively. $H_k \\in \\Real^{n\\times n}$ is a matrix that \\fy{contains} the curvature information of the objective function. The \\eqref{eqn:LM-cyclic-reg-BFGS} scheme can be seen as a regularized variant of \\fy{the} classical stochastic SQN \\fy{method}. Here we regularize the gradient map by the term $\\mu_k(x_k-x_0)$ to \\fy{induce} the strong monotonicity property. In \\fy{the} absence of strong convexity of $f$, unlike the classical schemes where $\\mu_k$ is maintained fixed, we let $\\mu_k$ be updated and \\fy{decreased} to zero. Throughout, we let $\\sF_k$ denote the history of the method up to time $k$, i.e., \n$\\sF_k\\fy{\\ \\triangleq \\ }\\{x_0,\\xi_0,\\xi_1,\\ldots,\\xi_{k-1}\\}$ for $k\\ge 1$, and $\\sF_0\\triangleq\\{x_0\\}$.  \n\\begin{assumption}\\label{assum:convex}  Consider problem~\\eqref{eqn:problem}. Let the following hold:\n\\begin{enumerate}\n\\item [(a)]  \\an{The function $f(x)$ is convex over $\\Real^n$.}  \n%\\an{The function $f(x)$ is convex on $\\mathbb{R}^n$.} \n\\item[(b)] $f(x)$ is continuously differentiable \\us{with Lipschitz\n\tcontinuous gradients} over $\\Real^n$ with parameter $L>0$. %Also, $\\lim_{x\\to \\infty}f(x)< \\infty$.\n\t\\item[(c)] \\ The optimal solution set of \\fy{the problem} is nonempty. \n\\end{enumerate}\n\\end{assumption}\n\n\\begin{assumption}[Random variable $\\xi$]\\label{assum:main}\n\\begin{enumerate}\n\\item[(a)]  \\fy{Vectors $\\xi_k \\in \\mathbb{R}^d$ are i.i.d.\\ realizations of the random variable $\\xi$} for any $k \\geq 0$; \n\\item[(b)] The stochastic gradient mapping $\\nabla F(x, \\xi)$ is an unbiased estimator of $\\nabla f(x)$, i.e.\n$\\EXP{\\nabla F(x,\\xi)}=\\nabla f(x)$ for all $x$, and has a bounded variance,\n\ti.e., there exists a \\us{scalar} $\\nu>0$ such that $\\EXP{\\|\\nabla F(x,\\xi)-\\nabla f(x)\\|^2} \\leq \\nu^2,$ for all $x \\in \\mathbb{R}^n$.\n%Also, $\\lim_{x\\to \\infty}f(x)< \\infty$.\n\\end{enumerate}\n\\end{assumption}\n\n\\begin{assumption}[Conditions on matrix $H_k$]\\label{assump:Hk}\nLet the following hold for all $k\\geq 0$:\\\\ \n(a) \\  The matrix $H_k\\in \\Real^{n\\times n}$ is $\\sF_{k}$-measurable, i.e., $\\EXP{H_k \\mid \\sF_{k}}=H_k$. \\\\\n(b) \\ Matrix $H_k$ is symmetric and positive definite and satisfies the following condition: There exist positive scalars $\\lambda_{\\min},\\lambda$ and scalar $\\alpha\\leq 0$ such that $$\\lambda_{\\min}\\mathbf{I} \\preceq H_k \\preceq \\lambda \\mu_k^\\alpha\\mathbf{I},\\qquad  \\hbox{for all }k\\geq 0,$$ \\fys{where $\\mu_k$ is the regularization parameter in \\eqref{eqn:LM-cyclic-reg-BFGS}.}\n\\end{assumption}\n\n\\begin{definition}[Regularized function and gradient mapping]\\label{def:regularizedF}\nConsider the sequence $\\{\\mu_k\\}$ of positive \\an{scalars} and the starting point of the algorithm \\eqref{eqn:LM-cyclic-reg-BFGS}, i.e., $x_0$.\n% for $k \\geq 0$. \nThe regularized function $f_k$ and its gradient are defined as follows \\fys{for all $k\\geq 0$}:\n\\begin{align*}\nf_k(x)\\triangleq  f(x)+\\frac{\\mu_k}{2}{\\|x-x_0\\|^2},\\quad \\nabla f_k(x)\\triangleq\\nabla f(x)+\\mu_k(x-x_0).\n\\end{align*}\n\\end{definition}\n\nIn a similar way, we denote the regularized stochastic function $F(x,\\xi)$ and its gradient with $F_k$ and $\\nabla F_k$ for any $\\xi$, respectively. \n\n\nThroughout, we denote the stochastic error of the gradient estimator by \n\\begin{align}\n\\label{def:wk}w_k\\triangleq \\nabla F(x_k,\\xi_k)-\\nabla f(x_k), \\an{\\quad\\hbox{for all }k\\ge0.} \n\\end{align}\n\nNote that under Assumption \\ref{assum:main}, from the definition of $w_k$ in \\eqref{def:wk}, we obtain $\\EXP{w_k\\mid \\sF_k}=0$ and $\\EXP{\\|w_k\\|^2\\mid \\sF_k}\\fy{\\leq}  \\nu^2$. \\fy{The following result plays a key role in the convergence and rate analysis of the proposed schemes.}\n\n\n\\begin{assumption}[\\fy{Conditions} on sequences for a.s. convergence]\\label{assum:sequences}\nLet the sequences $\\{\\gamma_k\\}$ and $\\{\\mu_k\\}$ be positive and satisfy the following conditions:\n\\begin{flalign*}\n\\begin{array}{lll}\n\\vspace{.2in}(a)\\  \\fy{\\lim_{k \\to \\infty }\\g_k\\mu_k^{2\\alpha-1} =0;}& &(b)\\ \\{\\mu_k\\}\\hbox{ is non-increasing and }\\mu_k \\to 0;\\\\\n\\vspace{.2in}(c)\\  \\lambda_{\\min}\\g_k\\mu_k \\leq 1\\hbox{ for }k \\geq 0; && (d)\\ \\sum_{k=0}^\\infty \\g_k\\mu_k =\\infty;\\\\\n(e)\\ \\sum_{k=0}^\\infty \\mu_k^2\\g_k <\\infty;&&(f)\\  \\sum_{k=0}^\\infty \\g_k^2\\mu_k^{2\\alpha} <\\infty\\fy{.}\n\\end{array}\n\\end{flalign*}\n\\end{assumption}\n\n\\begin{assumption}[\\fy{Conditions} on sequences for convergence in mean]\\label{assum:sequences-ms-convergence}\nLet the sequences $\\{\\gamma_k\\}$ and $\\{\\mu_k\\}$ be positive and satisfy the following conditions:\n\\begin{itemize}\n\\item [(a)] $\\lim_{k \\to \\infty }\\g_k\\mu_k^{2\\alpha-1} =0$;\n\\item [(b)] $\\{\\mu_k\\}\\hbox{ is non-increasing and }\\mu_k \\to 0$;\n\\item [(c)] $\\lambda_{\\min}\\g_k\\mu_{k} \\leq 1$ for $k \\geq 0$;\n%\\item [(d)] $\\g_k\\mu_k^{2\\alpha}\\leq \\frac{\\lambda{\\min}}{\\lambda^2(L+\\mu_0)}$ for all $k\\geq 0$;\n\\item [(d)] There exist $K_0$ and $0<\\beta<1$ such that \\begin{align*}\\g_{k-1}\\mu_{k-1}^{2\\alpha-1}\\leq \\g_{k}\\mu_{k}^{2\\alpha-1}(1+\\beta \\lambda_{\\min}\\g_{k}\\mu_{k}),\\quad \\hbox{for all }k \\geq K_0;\\end{align*}\n\\item [(e)] There exists a scalar $\\rho>0$ such that $ \\mu_k^{2-2\\alpha} \\leq \\rho\\g_k$ for all $k \\geq 0$\\fy{.}\n\\end{itemize}\n\\end{assumption}\n\n\\begin{align}\\label{eqn:mu-k}\n  \\begin{cases}\n   \\mu_{k}\\fy{:=}\\mu_{k-1},      &   \\text{if } k \\text{ is odd},\\\\\n    \\mu_{k}<\\mu_{k-1},  &    \\text{otherwise}\\fy{.}\n  \\end{cases}\n\\end{align}\n\nNote that we allow for updating the stepsize sequence at each iteration.\n\n\n\\begin{align}\\label{equ:siyi-LBFGS}&s_{\\lceil k/2\\rceil}:= x_{k}-x_{k-1},\\cr \n&y_{\\lceil k/2\\rceil}:=  \\nabla F(x_{k},\\xi_{k-1}) -\\nabla F(x_{k-1},\\xi_{k-1})  +  \\fys{\\tau}\\mu_k^\\delta s_{\\lceil k/2\\rceil},\\end{align}\n\n\\begin{align}\\label{eqn:H-k}H_{k}\\fy{\\triangleq}\n  \\begin{cases}\n    H_{k,m},  &    \\text{if } k \\text{ is odd}, \\\\\n    H_{k-1},  &   \\text{otherwise},\n  \\end{cases}\n\\end{align}\n\nwhere $m<n$ (in the large scale settings, \\fys{$m\\ll n$}) is \\fy{the memory parameter and represents} the number of pairs \\fy{$(s_i,y_i)$} to be \\fys{stored} to estimate $H_k$. Matrix $H_{k,m}$, for any \\fy{odd $k\\geq 2m-1$}, is updated using the following recursive formula:\n\\begin{align}\\label{eqn:H-k-m}\nH_{k,j}:=\\left(\\mathbf{I}-\\frac{y_is_i^T}{{y_i^Ts_i}}\\right)^TH_{k,j-1}\\left(\\mathbf{I}-\\frac{y_is_i^T}{y_i^Ts_i}\\right)+\\frac{s_is_i^T}{y_i^Ts_i}, \\quad 1 \\leq j\\leq m,\n\\end{align}\n\nwhere \\fy{$i\\triangleq \\lceil k/2\\rceil-(m-j)$ and we set $H_{k,0}:=\\frac{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}{y_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}\\mathbf{I}$}. Here, at odd iterations, matrix $H_k$ is obtained recursively from $H_{k,0},H_{k,1},\\ldots, H_{k,m-1}$. Note that \\fy{computation of} $H_k$ at an odd $k$ \\fy{needs} $m$ pairs \\fy{of} \\fy{$(s_i,y_i)$}. More precisely, $H_k$ uses the following curvature information: $\\left\\{(s_i,y_i)\\mid i=\\lceil k/2\\rceil-m+1, \\lceil k/2\\rceil-m+2,\\ldots,\\lceil k/2\\rceil\\right\\}.$ \nFor convenience, in the first \\fys{$2m-2$} iterations, we \\fy{let $H_k$ be the identity matrix}. \\fy{This allows for collecting the first set of $m$ pairs $(s_i,y_i)$, where $i=1,2,\\ldots,m$, that is used at iteration $k:=2m-1$ to obtain $H_{2m-1}$}. \\fys{The main differences between update rule \\eqref{eqn:H-k-m} and that of the standard SQN schemes~\\cite{Mokhtari15,nocedal15} are as follows:} (i) The first distinction is with respect to the definition of $y_i$ in \\eqref{equ:siyi-LBFGS}. Here the term $\\fy{\\mu_k^\\delta s_{\\lceil k/2\\rceil}}$ \\vus{compensates} for the lack of strong monotonicity of the gradient mapping and \\vus{aids in} establishing the \\fy{curvature condition}. (ii) Second, instead of obtaining the pair $(s_i,y_i)$ at every iteration, we \\fy{evaluate these terms only at} odd iterations to allow for updating the regularization parameter satisfying \\eqref{eqn:mu-k}. %\\vus{\\bf Farzad -- doesnt the paper by Mokhtari also do odd updates...clarify} \n\n\n\\begin{assumption}\\label{assum:convex2}  \n\\begin{enumerate}\n\\item [(a)]  \\an{The function $F(x,\\xi)$ is convex over $\\Real^n$ for any $\\xi \\in \\Omega$.}  \n%\\an{The function $f(x)$ is convex on $\\mathbb{R}^n$.} \n\\item[(b)] For any $\\xi \\in \\Omega$, $F(\\cdot,\\xi)$ is continuously differentiable \\us{with Lipschitz\n\tcontinuous gradients} over $\\Real^n$ with parameter $L_\\xi>0$. Moreover, $L:=\\sup_{\\xi \\in \\Omega}L_\\xi < \\infty$. %Also, $\\lim_{x\\to \\infty}f(x)< \\infty$.\n\t\\item[(c)] \\ The optimal solution set $X^*$ of problem \\eqref{eqn:problem} is nonempty. \n\\end{enumerate}\n\\end{assumption}\n\n\\begin{algorithm}\n  \\caption{Iteratively Regularized Stochastic Limited-memory BFGS}\n\\label{algorithm:IR-S-BFGS}\n%\\begin{boxedminipage}{165mm}\n    \\begin{algorithmic}[1]\n    \\State\\textbf{Input:} LBFGS memory parameter $m\\geq 1$, Lipschitzian parameter $L>0$, random initial point $x_0 \\in \\Real^n$, initial stepsize $\\g_0>0$, and initial regularization parameter $\\mu_0>0$ such that $\\g_0\\mu_0 \\leq  (m+n) L$, \\fys{scalars $0<\\epsilon < \\frac{1}{3}$, $\\delta \\in \\left(0,\\frac{1.5\\epsilon}{n+m}\\right)$, and $\\tau>0$}; \n    \\State Set $a:=\\frac{2}{3}-\\epsilon+\\frac{2\\delta(n+m)}{3}$, $b:=\\frac{1}{3}$;\n    \\For {$k=0,1,\\ldots,$}\n    \t\t\\State Compute $\\g_k :=\\frac{\\g_0}{(k+1)^a}$ and $\\mu_k:=\\frac{\\mu_02^b}{\\left(k+1+\\text{mod}(k+1,2)\\right)^b}$;\n        \\State  {Evaluate the stochastic gradient $\\nabla F(x_k,\\xi_k)$;}\n     \\If {mod$(k,2)=1$}\n    \\State  Compute index $i:=\\lceil k/2\\rceil$;\n     \\State   Compute vector $s_i:=x_k-x_{k-1}$; \n     \\State   Compute vector  $y_i:= \\nabla F(x_k,\\xi_{k-1})-\\nabla F(x_{k-1},\\xi_{k-1})+\\fys{\\tau}\\mu_k^\\delta s_i$;\n      \\If {$k> 2m$}\n      \\State Discard the vector pair $\\{s_{i-m},y_{i-m}\\}$ from storage;\n      \\EndIf\n   \\EndIf\n \\If {$k < 2m-1$}\n   \\State Update solution iterate $x_{k+1}:=x_k-\\g_k\\left(\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)\\right)$;\n     \\Else\n     \\State Initialize Hessian inverse $H_{k,0}:=\\frac{s_i^Ty_i}{y_i^Ty_i}\\mathbf{I}$;\\hspace{.4in} \\tikzmark{right}\\tikzmark{top}\n     \\State Initialize $q:=\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)$;\n     \\For {$t=i:i-m+1$}\n     \\State Compute scalar $\\alpha_{i-t+1}:=\\frac{s_t^Tq}{s_t^Ty_t}$;\n     \\State Update vector $q:=q-\\alpha_{i-t+1}y_t$;\n     \\EndFor\n     \\State Initialize vector $r:=H_{k,0}q$;\n     \\For {$t=i-m+1:i$}\n     %\\State Compute scalar $\\beta:=\\frac{y_t^Tr}{s_t^Ty_t}$;\n     \\State  Update vector $r:=r+\\left(\\alpha_{i-t+1}-\\frac{y_t^Tr}{s_t^Ty_t}\\right)s_t$;\n     \\EndFor\\tikzmark{bottom}\n%\\State Given $\\mathcal{M}=\\{(s_t,y_t)\\}_{t=i-m+1}^i$, call Alg. \\ref{algorithm:Hk-update} to compute the following vector\\[G_k \\triangleq H_k\\left(\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)\\right);\\]\n             \\State Update solution iterate $x_{k+1}:=x_k-\\g_kr$;\\Comment{\\footnotesize{LBFGS update}} \n   \n\\EndIf\n    \n    \\EndFor\n   \\end{algorithmic}\n   \\AddNote{top}{bottom}{right}{\\footnotesize{LBFGS two-loop recursion}}\n%\\end{boxedminipage}\n% \\vskip17.5pt\n\\end{algorithm}","units":[{"start":9925,"end":10719,"start_line":253,"end_line":261,"raw_tex":"\\fy{We} consider the following stochastic optimization problem:\n\\begin{equation}\\label{eqn:problem}\n\\min_{x \\in \\Real^n} f(x)\\fy{\\ \\triangleq \\ }\\EXP{F(x,\\xi(\\us{\\omega}))},\\tag{SO}\n\\end{equation}\nwhere $F: \\Real^n\\times\\us{\\Real^d}\\to\\mathbb{R}$ is a \\fy{real-valued} function, the random\nvector $\\xi$ is defined as {$\\xi:\\Omega\n\\rightarrow \\Real^d$}, \\us{$(\\Omega,{\\cal F}, \\mathbb{P})$ denotes the\nassociated probability space\\fy{,} and the expectation \\us{$\\EXP{F(x,\\xi)}$} is\ntaken with respect to $\\mathbb{P}$}. Problem \\eqref{eqn:problem} provides a general framework that can capture a wide range of applications in operations research, machine learning, statistics and control to name a few \\fys{(\\vus{cf.}\\cite{bottou-2010,nocedal15}). Addressing problem \\eqref{eqn:problem}} has led\n","sha256_utf8":"2185a6dedab61e32535871ef890c858b0df1fba026673f468a015f7d89daa884"},{"start":23003,"end":24301,"start_line":409,"end_line":428,"raw_tex":" \\textbf{Notation:} A vector $x$ is assumed to be a column vector and\n$x^T$ denotes its transpose, \\an{while} $\\|x\\|$ {denotes} the Euclidean\nvector norm, i.e., $\\|x\\|=\\sqrt{x^Tx}$.  We write \\textit{a.s.} as the\nabbreviation for ``almost surely''.  \\an{For a symmetric matrix $B$, \n$\\lambda_{\\min}(B)$ and $\\lambda_{\\max}(B)$ \\vus{denotes the\nsmallest and largest eigenvalue of $B$}, respectively.} We use $\\EXP{z}$ to\ndenote the expectation of a random variable~$z$. A function $f:X\n\\subset \\mathbb{R}^n\\rightarrow \\mathbb{R}$ is said to be strongly\nconvex with parameter $\\mu>0$, if  $f(y)\\geq f(x)+\\nabla\nf(x)^T(y-x)+\\frac{\\mu}{2}\\|x-y\\|^2,$ for any $x,y \\in X$.  A mapping\n$F:X \\subset \\mathbb{R}^n\\rightarrow \\mathbb{R}$ is \\vus{said to be} Lipschitz\ncontinuous with parameter $L>0$ if for any $x, y \\in X$, we have\n$\\|F(x)-F(y)\\|\\leq L\\|x-y\\|$. For a continuously differentiable\nfunction $f$ with Lipschitz gradients with parameter $L>0$, we have\n$f(y)\\leq f(x)+\\nabla f(x)^T(y-x)+\\frac{L}{2}\\|x-y\\|^2,$ for any $x,y\n\\in X$. For a vector $x \\in \\Real^n$ and a \\fys{nonempty} set $X\n\\subset \\Real^n$, the Euclidean distance of $x$ from $X$ is denoted by\n$dist(x,X)$. We denote the optimal objective value of problem\n\\eqref{eqn:problem} by $f^*$ and the set of the optimal solutions by\n$X^*$.\n","sha256_utf8":"db37fc7f4f4e719c323e870f227c11b58a6d92a3ab445360f8956111f3c1ce95"},{"start":24356,"end":24761,"start_line":433,"end_line":436,"raw_tex":"In this section, we describe a general SQN scheme for solving problem~\\eqref{eqn:problem} and present the main assumptions. Let $x_0\\in \\Real^n$ be an arbitrary initial point, and $x_k$ be generated by the following recursive rule\n\\begin{align}\\label{eqn:LM-cyclic-reg-BFGS}\\tag{IR-SQN}\nx_{k+1}:=x_k -\\gamma_kH_k\\left(\\nabla F(x_k,\\xi_k)+ \\mu_k(x_k-x_0)\\right), \\quad \\hbox{for all } k \\geq 0.\n\\end{align}","sha256_utf8":"030898d70793c86ac87f42fdd540f71efa652d379f25e44b6cbff9fe5d6eecf0"},{"start":24762,"end":26084,"start_line":437,"end_line":447,"raw_tex":"Here, $\\g_k$ and $\\mu_k$ are the steplength and the regularization parameter, respectively. $H_k \\in \\Real^{n\\times n}$ is a matrix that \\fy{contains} the curvature information of the objective function. The \\eqref{eqn:LM-cyclic-reg-BFGS} scheme can be seen as a regularized variant of \\fy{the} classical stochastic SQN \\fy{method}. Here we regularize the gradient map by the term $\\mu_k(x_k-x_0)$ to \\fy{induce} the strong monotonicity property. In \\fy{the} absence of strong convexity of $f$, unlike the classical schemes where $\\mu_k$ is maintained fixed, we let $\\mu_k$ be updated and \\fy{decreased} to zero. Throughout, we let $\\sF_k$ denote the history of the method up to time $k$, i.e., \n$\\sF_k\\fy{\\ \\triangleq \\ }\\{x_0,\\xi_0,\\xi_1,\\ldots,\\xi_{k-1}\\}$ for $k\\ge 1$, and $\\sF_0\\triangleq\\{x_0\\}$.  \n\\begin{assumption}\\label{assum:convex}  Consider problem~\\eqref{eqn:problem}. Let the following hold:\n\\begin{enumerate}\n\\item [(a)]  \\an{The function $f(x)$ is convex over $\\Real^n$.}  \n%\\an{The function $f(x)$ is convex on $\\mathbb{R}^n$.} \n\\item[(b)] $f(x)$ is continuously differentiable \\us{with Lipschitz\n\tcontinuous gradients} over $\\Real^n$ with parameter $L>0$. %Also, $\\lim_{x\\to \\infty}f(x)< \\infty$.\n\t\\item[(c)] \\ The optimal solution set of \\fy{the problem} is nonempty. \n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"e248939c9f3b5f7ff6c1ebf0e2cd975054c3025221cc467f5b4b307d7c143a4f"},{"start":26219,"end":26818,"start_line":449,"end_line":457,"raw_tex":"\\begin{assumption}[Random variable $\\xi$]\\label{assum:main}\n\\begin{enumerate}\n\\item[(a)]  \\fy{Vectors $\\xi_k \\in \\mathbb{R}^d$ are i.i.d.\\ realizations of the random variable $\\xi$} for any $k \\geq 0$; \n\\item[(b)] The stochastic gradient mapping $\\nabla F(x, \\xi)$ is an unbiased estimator of $\\nabla f(x)$, i.e.\n$\\EXP{\\nabla F(x,\\xi)}=\\nabla f(x)$ for all $x$, and has a bounded variance,\n\ti.e., there exists a \\us{scalar} $\\nu>0$ such that $\\EXP{\\|\\nabla F(x,\\xi)-\\nabla f(x)\\|^2} \\leq \\nu^2,$ for all $x \\in \\mathbb{R}^n$.\n%Also, $\\lim_{x\\to \\infty}f(x)< \\infty$.\n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"baf6075aff4c35514acd283bd57e8724454db82e0c737338a58562c3a1868ea5"},{"start":26879,"end":27495,"start_line":459,"end_line":463,"raw_tex":"\\begin{assumption}[Conditions on matrix $H_k$]\\label{assump:Hk}\nLet the following hold for all $k\\geq 0$:\\\\ \n(a) \\  The matrix $H_k\\in \\Real^{n\\times n}$ is $\\sF_{k}$-measurable, i.e., $\\EXP{H_k \\mid \\sF_{k}}=H_k$. \\\\\n(b) \\ Matrix $H_k$ is symmetric and positive definite and satisfies the following condition: There exist positive scalars $\\lambda_{\\min},\\lambda$ and scalar $\\alpha\\leq 0$ such that $$\\lambda_{\\min}\\mathbf{I} \\preceq H_k \\preceq \\lambda \\mu_k^\\alpha\\mathbf{I},\\qquad  \\hbox{for all }k\\geq 0,$$ \\fys{where $\\mu_k$ is the regularization parameter in \\eqref{eqn:LM-cyclic-reg-BFGS}.}\n\\end{assumption}","sha256_utf8":"793bb3d1d98413c5d369f6d21f98f1a69ef6849edb6ff4a461ca78399b15ad88"},{"start":28281,"end":28777,"start_line":469,"end_line":476,"raw_tex":"\\begin{definition}[Regularized function and gradient mapping]\\label{def:regularizedF}\nConsider the sequence $\\{\\mu_k\\}$ of positive \\an{scalars} and the starting point of the algorithm \\eqref{eqn:LM-cyclic-reg-BFGS}, i.e., $x_0$.\n% for $k \\geq 0$. \nThe regularized function $f_k$ and its gradient are defined as follows \\fys{for all $k\\geq 0$}:\n\\begin{align*}\nf_k(x)\\triangleq  f(x)+\\frac{\\mu_k}{2}{\\|x-x_0\\|^2},\\quad \\nabla f_k(x)\\triangleq\\nabla f(x)+\\mu_k(x-x_0).\n\\end{align*}\n\\end{definition}","sha256_utf8":"f2ea9d08f5de185bcbabdd99de1f4a9e5e7c4c9fc90fe0fb94e46c21d561a87c"},{"start":28778,"end":28928,"start_line":477,"end_line":477,"raw_tex":"In a similar way, we denote the regularized stochastic function $F(x,\\xi)$ and its gradient with $F_k$ and $\\nabla F_k$ for any $\\xi$, respectively. \n","sha256_utf8":"9c15ece068a4275f8baaead7ccc33d7f3691b0666d4c34e5b17a7684029c83bc"},{"start":30561,"end":30755,"start_line":506,"end_line":509,"raw_tex":"Throughout, we denote the stochastic error of the gradient estimator by \n\\begin{align}\n\\label{def:wk}w_k\\triangleq \\nabla F(x_k,\\xi_k)-\\nabla f(x_k), \\an{\\quad\\hbox{for all }k\\ge0.} \n\\end{align}","sha256_utf8":"69200a9c7b1d333ac4d2af465a7ed25104ace918bb1b4fbf574ff25a523a603f"},{"start":30756,"end":31037,"start_line":510,"end_line":510,"raw_tex":"Note that under Assumption \\ref{assum:main}, from the definition of $w_k$ in \\eqref{def:wk}, we obtain $\\EXP{w_k\\mid \\sF_k}=0$ and $\\EXP{\\|w_k\\|^2\\mid \\sF_k}\\fy{\\leq}  \\nu^2$. \\fy{The following result plays a key role in the convergence and rate analysis of the proposed schemes.}\n","sha256_utf8":"317454b9290877d038b3d3639a730cad2bed65f73b8747d21c918d0e6c4d8cd1"},{"start":37741,"end":38366,"start_line":575,"end_line":584,"raw_tex":"\\begin{assumption}[\\fy{Conditions} on sequences for a.s. convergence]\\label{assum:sequences}\nLet the sequences $\\{\\gamma_k\\}$ and $\\{\\mu_k\\}$ be positive and satisfy the following conditions:\n\\begin{flalign*}\n\\begin{array}{lll}\n\\vspace{.2in}(a)\\  \\fy{\\lim_{k \\to \\infty }\\g_k\\mu_k^{2\\alpha-1} =0;}& &(b)\\ \\{\\mu_k\\}\\hbox{ is non-increasing and }\\mu_k \\to 0;\\\\\n\\vspace{.2in}(c)\\  \\lambda_{\\min}\\g_k\\mu_k \\leq 1\\hbox{ for }k \\geq 0; && (d)\\ \\sum_{k=0}^\\infty \\g_k\\mu_k =\\infty;\\\\\n(e)\\ \\sum_{k=0}^\\infty \\mu_k^2\\g_k <\\infty;&&(f)\\  \\sum_{k=0}^\\infty \\g_k^2\\mu_k^{2\\alpha} <\\infty\\fy{.}\n\\end{array}\n\\end{flalign*}\n\\end{assumption}","sha256_utf8":"76c66b0c7f0f6404c98c6ca955aa4bee2446ca88b6d903aef9147a970ee2ae4a"},{"start":41876,"end":42745,"start_line":627,"end_line":637,"raw_tex":"\\begin{assumption}[\\fy{Conditions} on sequences for convergence in mean]\\label{assum:sequences-ms-convergence}\nLet the sequences $\\{\\gamma_k\\}$ and $\\{\\mu_k\\}$ be positive and satisfy the following conditions:\n\\begin{itemize}\n\\item [(a)] $\\lim_{k \\to \\infty }\\g_k\\mu_k^{2\\alpha-1} =0$;\n\\item [(b)] $\\{\\mu_k\\}\\hbox{ is non-increasing and }\\mu_k \\to 0$;\n\\item [(c)] $\\lambda_{\\min}\\g_k\\mu_{k} \\leq 1$ for $k \\geq 0$;\n%\\item [(d)] $\\g_k\\mu_k^{2\\alpha}\\leq \\frac{\\lambda{\\min}}{\\lambda^2(L+\\mu_0)}$ for all $k\\geq 0$;\n\\item [(d)] There exist $K_0$ and $0<\\beta<1$ such that \\begin{align*}\\g_{k-1}\\mu_{k-1}^{2\\alpha-1}\\leq \\g_{k}\\mu_{k}^{2\\alpha-1}(1+\\beta \\lambda_{\\min}\\g_{k}\\mu_{k}),\\quad \\hbox{for all }k \\geq K_0;\\end{align*}\n\\item [(e)] There exists a scalar $\\rho>0$ such that $ \\mu_k^{2-2\\alpha} \\leq \\rho\\g_k$ for all $k \\geq 0$\\fy{.}\n\\end{itemize}\n\\end{assumption}","sha256_utf8":"a2c3d1498e64eefe5e4ab48c5f3225c43b259ecb6f0896dc1b4ccc2d877e3be1"},{"start":52458,"end":52649,"start_line":722,"end_line":727,"raw_tex":"\\begin{align}\\label{eqn:mu-k}\n  \\begin{cases}\n   \\mu_{k}\\fy{:=}\\mu_{k-1},      &   \\text{if } k \\text{ is odd},\\\\\n    \\mu_{k}<\\mu_{k-1},  &    \\text{otherwise}\\fy{.}\n  \\end{cases}\n\\end{align}","sha256_utf8":"96d0fd3a4d6d5d2d744b1fa1505e82b73aecfb6a2c4db276797b1b54a3700638"},{"start":52650,"end":52723,"start_line":728,"end_line":728,"raw_tex":"Note that we allow for updating the stepsize sequence at each iteration.\n","sha256_utf8":"a320a3ee3504215bc55100b3ea2555942079f12e48807db740569935395efa39"},{"start":53252,"end":53469,"start_line":730,"end_line":731,"raw_tex":"\\begin{align}\\label{equ:siyi-LBFGS}&s_{\\lceil k/2\\rceil}:= x_{k}-x_{k-1},\\cr \n&y_{\\lceil k/2\\rceil}:=  \\nabla F(x_{k},\\xi_{k-1}) -\\nabla F(x_{k-1},\\xi_{k-1})  +  \\fys{\\tau}\\mu_k^\\delta s_{\\lceil k/2\\rceil},\\end{align}","sha256_utf8":"0954b6993a778d6cf38c1ef07e6a4a21312a5abe6c874c875ca390a314dfe57c"},{"start":53895,"end":54072,"start_line":732,"end_line":737,"raw_tex":"\\begin{align}\\label{eqn:H-k}H_{k}\\fy{\\triangleq}\n  \\begin{cases}\n    H_{k,m},  &    \\text{if } k \\text{ is odd}, \\\\\n    H_{k-1},  &   \\text{otherwise},\n  \\end{cases}\n\\end{align}","sha256_utf8":"468e68f50f69f38fd281a98a1d67a65d17f393d8ad6c8be360eb6ebe49c2e403"},{"start":54073,"end":54561,"start_line":738,"end_line":741,"raw_tex":"where $m<n$ (in the large scale settings, \\fys{$m\\ll n$}) is \\fy{the memory parameter and represents} the number of pairs \\fy{$(s_i,y_i)$} to be \\fys{stored} to estimate $H_k$. Matrix $H_{k,m}$, for any \\fy{odd $k\\geq 2m-1$}, is updated using the following recursive formula:\n\\begin{align}\\label{eqn:H-k-m}\nH_{k,j}:=\\left(\\mathbf{I}-\\frac{y_is_i^T}{{y_i^Ts_i}}\\right)^TH_{k,j-1}\\left(\\mathbf{I}-\\frac{y_is_i^T}{y_i^Ts_i}\\right)+\\frac{s_is_i^T}{y_i^Ts_i}, \\quad 1 \\leq j\\leq m,\n\\end{align}","sha256_utf8":"5534cb6f0e3a282018836dbcf0805b9ccd9c86051a32f87e3e9a40419e782dd5"},{"start":54562,"end":56079,"start_line":742,"end_line":743,"raw_tex":"where \\fy{$i\\triangleq \\lceil k/2\\rceil-(m-j)$ and we set $H_{k,0}:=\\frac{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}{y_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}\\mathbf{I}$}. Here, at odd iterations, matrix $H_k$ is obtained recursively from $H_{k,0},H_{k,1},\\ldots, H_{k,m-1}$. Note that \\fy{computation of} $H_k$ at an odd $k$ \\fy{needs} $m$ pairs \\fy{of} \\fy{$(s_i,y_i)$}. More precisely, $H_k$ uses the following curvature information: $\\left\\{(s_i,y_i)\\mid i=\\lceil k/2\\rceil-m+1, \\lceil k/2\\rceil-m+2,\\ldots,\\lceil k/2\\rceil\\right\\}.$ \nFor convenience, in the first \\fys{$2m-2$} iterations, we \\fy{let $H_k$ be the identity matrix}. \\fy{This allows for collecting the first set of $m$ pairs $(s_i,y_i)$, where $i=1,2,\\ldots,m$, that is used at iteration $k:=2m-1$ to obtain $H_{2m-1}$}. \\fys{The main differences between update rule \\eqref{eqn:H-k-m} and that of the standard SQN schemes~\\cite{Mokhtari15,nocedal15} are as follows:} (i) The first distinction is with respect to the definition of $y_i$ in \\eqref{equ:siyi-LBFGS}. Here the term $\\fy{\\mu_k^\\delta s_{\\lceil k/2\\rceil}}$ \\vus{compensates} for the lack of strong monotonicity of the gradient mapping and \\vus{aids in} establishing the \\fy{curvature condition}. (ii) Second, instead of obtaining the pair $(s_i,y_i)$ at every iteration, we \\fy{evaluate these terms only at} odd iterations to allow for updating the regularization parameter satisfying \\eqref{eqn:mu-k}. %\\vus{\\bf Farzad -- doesnt the paper by Mokhtari also do odd updates...clarify} \n","sha256_utf8":"de8d22de3fdd9efd0dd655b91a4424a0ac6de61632cbbf4f7d39309139acaf81"},{"start":66295,"end":66887,"start_line":927,"end_line":935,"raw_tex":"\\begin{assumption}\\label{assum:convex2}  \n\\begin{enumerate}\n\\item [(a)]  \\an{The function $F(x,\\xi)$ is convex over $\\Real^n$ for any $\\xi \\in \\Omega$.}  \n%\\an{The function $f(x)$ is convex on $\\mathbb{R}^n$.} \n\\item[(b)] For any $\\xi \\in \\Omega$, $F(\\cdot,\\xi)$ is continuously differentiable \\us{with Lipschitz\n\tcontinuous gradients} over $\\Real^n$ with parameter $L_\\xi>0$. Moreover, $L:=\\sup_{\\xi \\in \\Omega}L_\\xi < \\infty$. %Also, $\\lim_{x\\to \\infty}f(x)< \\infty$.\n\t\\item[(c)] \\ The optimal solution set $X^*$ of problem \\eqref{eqn:problem} is nonempty. \n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"fd3526afa0fff4590b6c945d2a7ff1b61ba123a3f641793e178f537ae689b661"},{"start":72294,"end":74681,"start_line":999,"end_line":1041,"raw_tex":"\\begin{algorithm}\n  \\caption{Iteratively Regularized Stochastic Limited-memory BFGS}\n\\label{algorithm:IR-S-BFGS}\n%\\begin{boxedminipage}{165mm}\n    \\begin{algorithmic}[1]\n    \\State\\textbf{Input:} LBFGS memory parameter $m\\geq 1$, Lipschitzian parameter $L>0$, random initial point $x_0 \\in \\Real^n$, initial stepsize $\\g_0>0$, and initial regularization parameter $\\mu_0>0$ such that $\\g_0\\mu_0 \\leq  (m+n) L$, \\fys{scalars $0<\\epsilon < \\frac{1}{3}$, $\\delta \\in \\left(0,\\frac{1.5\\epsilon}{n+m}\\right)$, and $\\tau>0$}; \n    \\State Set $a:=\\frac{2}{3}-\\epsilon+\\frac{2\\delta(n+m)}{3}$, $b:=\\frac{1}{3}$;\n    \\For {$k=0,1,\\ldots,$}\n    \t\t\\State Compute $\\g_k :=\\frac{\\g_0}{(k+1)^a}$ and $\\mu_k:=\\frac{\\mu_02^b}{\\left(k+1+\\text{mod}(k+1,2)\\right)^b}$;\n        \\State  {Evaluate the stochastic gradient $\\nabla F(x_k,\\xi_k)$;}\n     \\If {mod$(k,2)=1$}\n    \\State  Compute index $i:=\\lceil k/2\\rceil$;\n     \\State   Compute vector $s_i:=x_k-x_{k-1}$; \n     \\State   Compute vector  $y_i:= \\nabla F(x_k,\\xi_{k-1})-\\nabla F(x_{k-1},\\xi_{k-1})+\\fys{\\tau}\\mu_k^\\delta s_i$;\n      \\If {$k> 2m$}\n      \\State Discard the vector pair $\\{s_{i-m},y_{i-m}\\}$ from storage;\n      \\EndIf\n   \\EndIf\n \\If {$k < 2m-1$}\n   \\State Update solution iterate $x_{k+1}:=x_k-\\g_k\\left(\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)\\right)$;\n     \\Else\n     \\State Initialize Hessian inverse $H_{k,0}:=\\frac{s_i^Ty_i}{y_i^Ty_i}\\mathbf{I}$;\\hspace{.4in} \\tikzmark{right}\\tikzmark{top}\n     \\State Initialize $q:=\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)$;\n     \\For {$t=i:i-m+1$}\n     \\State Compute scalar $\\alpha_{i-t+1}:=\\frac{s_t^Tq}{s_t^Ty_t}$;\n     \\State Update vector $q:=q-\\alpha_{i-t+1}y_t$;\n     \\EndFor\n     \\State Initialize vector $r:=H_{k,0}q$;\n     \\For {$t=i-m+1:i$}\n     %\\State Compute scalar $\\beta:=\\frac{y_t^Tr}{s_t^Ty_t}$;\n     \\State  Update vector $r:=r+\\left(\\alpha_{i-t+1}-\\frac{y_t^Tr}{s_t^Ty_t}\\right)s_t$;\n     \\EndFor\\tikzmark{bottom}\n%\\State Given $\\mathcal{M}=\\{(s_t,y_t)\\}_{t=i-m+1}^i$, call Alg. \\ref{algorithm:Hk-update} to compute the following vector\\[G_k \\triangleq H_k\\left(\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)\\right);\\]\n             \\State Update solution iterate $x_{k+1}:=x_k-\\g_kr$;\\Comment{\\footnotesize{LBFGS update}} \n   \n\\EndIf\n    \n    \\EndFor\n   \\end{algorithmic}\n   \\AddNote{top}{bottom}{right}{\\footnotesize{LBFGS two-loop recursion}}\n%\\end{boxedminipage}\n% \\vskip17.5pt\n\\end{algorithm}","sha256_utf8":"50a1bd5bf3f89d068ac150fe333331b38e144935913979873a3e6be8b1ed0b76"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}[\\fy{Rate analysis for Algorithm \\ref{algorithm:IR-S-BFGS}}]\\label{thm:rate}\n\\fy{Consider Algorithm \\ref{algorithm:IR-S-BFGS}. The following statements hold:}\n\\begin{itemize}\n\\item [(a)] Suppose Assumptions \\ref{assum:main} and \\ref{assum:convex2} hold. Then, there exists \\fy{$K\\geq2m-1$} such that \n\\[\\EXP{f(x_{k })}-f^*\\leq \\left(\\frac{\\theta\\g_0}{\\left(\\mu_0\\sqrt[3]{2}\\right)^{1-2\\alpha}}\\right)\\frac{1}{k^{\\frac{1}{3}-\\e}}, \\quad \\hbox{for all } k > K,\\] \nwhere $\\theta$ is given by \\eqref{def:theta}, and $\\lambda_{\\min}$, $\\lambda$, and $\\alpha$ are given by \\eqref{equ:valuesForAssumH_k}. \n\\item [(b)] \\fy{Let the scheme be at the $k$th iteration where $k\\geq 2m-1$. Then, by the end of the LBFGS \\fys{two-loop} recursion, i.e., line $\\#$26 in Algorithm \\ref{algorithm:IR-S-BFGS}, we have \n\\begin{align}\\label{eqn:r_Lemma}\n  r=H_k\\left(\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)\\right),\n\\end{align}\nwhere $H_{k}$ is defined by \\eqref{eqn:H-k}.}\n\\end{itemize}\n\\end{theorem}","units":[{"start":76897,"end":77885,"start_line":1067,"end_line":1079,"raw_tex":"\\begin{theorem}[\\fy{Rate analysis for Algorithm \\ref{algorithm:IR-S-BFGS}}]\\label{thm:rate}\n\\fy{Consider Algorithm \\ref{algorithm:IR-S-BFGS}. The following statements hold:}\n\\begin{itemize}\n\\item [(a)] Suppose Assumptions \\ref{assum:main} and \\ref{assum:convex2} hold. Then, there exists \\fy{$K\\geq2m-1$} such that \n\\[\\EXP{f(x_{k })}-f^*\\leq \\left(\\frac{\\theta\\g_0}{\\left(\\mu_0\\sqrt[3]{2}\\right)^{1-2\\alpha}}\\right)\\frac{1}{k^{\\frac{1}{3}-\\e}}, \\quad \\hbox{for all } k > K,\\] \nwhere $\\theta$ is given by \\eqref{def:theta}, and $\\lambda_{\\min}$, $\\lambda$, and $\\alpha$ are given by \\eqref{equ:valuesForAssumH_k}. \n\\item [(b)] \\fy{Let the scheme be at the $k$th iteration where $k\\geq 2m-1$. Then, by the end of the LBFGS \\fys{two-loop} recursion, i.e., line $\\#$26 in Algorithm \\ref{algorithm:IR-S-BFGS}, we have \n\\begin{align}\\label{eqn:r_Lemma}\n  r=H_k\\left(\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)\\right),\n\\end{align}\nwhere $H_{k}$ is defined by \\eqref{eqn:H-k}.}\n\\end{itemize}\n\\end{theorem}","sha256_utf8":"e234be7e4539db4077577c72ff64049e6f5190ecc38fa0e9da7522b5538ee684"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{property}[{Properties of a regularized function}]\\label{proper:propsfk} We have:\n\\begin{itemize}\n\\item [(a)] $f_k$ is strongly convex with a parameter $\\mu_k$.\n\\item [(b)] $f_k$ has Lipschitzian gradients with parameter $L+\\mu_k$.\n \\item [(c)] $f_k$ has a unique minimizer over $\\Real^n$, denoted by $x^*_k$. Moreover, for any $x \\in \\Real^n$, \n \\[2\\mu_k (f_k(x)-f_k(x^*_k)) \\leq \\|\\nabla f_k(x)\\|^2\\leq 2(L+\\mu_k) (f_k(x)-f_k(x^*_k)).\\]\n\\end{itemize}\n\\end{property}\n\nThe existence and uniqueness of $x^*_k$ in Property~\\ref{proper:propsfk}(c) \nis due to the strong convexity of the function $f_k$ (see, for example, Section 1.3.2 in~\\cite{Polyak87}), \nwhile the relation for the gradient is known to hold for a \nstrongly convex function with a parameter $\\mu$ that also has Lipschitz gradients with a parameter \n$L$ (see Lemma 1 on page 23 in~\\cite{Polyak87}). \n\n\n\\begin{lemma}[A recursive error bound]\\label{lemma:main-ineq}\nConsider the \\eqref{eqn:LM-cyclic-reg-BFGS} method and suppose Assumptions \\ref{assum:convex}, \\ref{assum:main}, and \\ref{assump:Hk} hold. Also, assume $\\mu_k$ is a non-increasing sequence and let \\begin{align}\\label{mainLemmaCond}\\g_k\\mu_k^{2\\alpha} \\leq \\frac{\\lambda_{\\min}}{\\lambda^2(L+\\mu_0)},\\quad \\hbox{for all }k\\geq 0.\n\\end{align}Then, the following inequality holds \\fy{for all $k\\geq 0$}:\n\\begin{align}\\label{ineq:cond-recursive-F-k}\n\\EXP{f_{k+1}(x_{k+1})\\mid\\sF_k}-f^* &\\leq (1-\\lambda_{\\min}\\mu_k\\g_k)(f_k(x_k)-f^*)+\\frac{\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)}{2}\\mu_k^2\\g_k\\notag \\\\ &+\\frac{ (L+\\mu_k)\\lambda^2\\nu^2}{2}\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align}\n\\end{lemma}\n\n\\begin{proof}\nThe Lipschitzian property of $\\nabla f_k$ and  \nthe update rule \\eqref{eqn:LM-cyclic-reg-BFGS} imply that \n\\begin{align*}\nf_k(x_{k+1}) &\\leq f_k(x_k)+\\nabla f_k(x_k)^T(x_{k+1}-x_k)+\\frac{ (L+\\mu_k)}{2}\\|x_{k+1}-x_k\\|^2 \\cr\n&= f_k(x_k)-\\g_k\\nabla f_k(x_k)^TH_k\\left(\\nabla F(x_k,\\xi_k) +\\mu_k(x_k-x_0)\\right)\\\\&+ \\frac{ (L+\\mu_k)}{2}\\g_k^2\\|H_k\\left(\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)\\right)\\|^2.\\end{align*}\n\\fy{Invoking the definition of the stochastic error $w_k$ in} \\eqref{def:wk} \\fy{and} Definition~\\ref{def:regularizedF}, we obtain\n\\begin{align}\\label{ineq:term1-2}\nf_k(x_{k+1}) &\\leq f_k(x_k)-\\g_k\\nabla f_k(x_k)^TH_k(\\nabla f(x_k)+w_k +\\mu_k(x_k-x_0))\\notag\\\\&+ \\frac{ (L+\\mu_k)}{2}\\g_k^2\\|H_k(\\nabla f(x_k)+w_k+\\mu_k(x_k-x_0))\\|^2\\\\ \n\\notag = f_k(x_k)&-\\g_k\\underbrace{\\nabla f_k(x_k)^TH_k(\\nabla f_k(x_k)+w_k)}_{\\tiny\\hbox{Term } 1}+ \\frac{ (L+\\mu_k)}{2}\\g_k^2\\underbrace{\\|H_k(\\nabla f_k(x_k)+w_k)\\|^2}_{\\tiny\\hbox{ Term } 2},\\end{align}\nwhere in the last \\fy{equation,} we used the definition of $f_k$. Next, we estimate the conditional expectation of Term 1 and 2. From Assumption \\ref{assump:Hk}, we have \n\\begin{align*}\n\\hbox{Term }1 &= \\nabla f_k(x_k)^TH_k\\nabla f_k(x_k)+\\nabla f_k(x_k)^TH_kw_k \\\\&\\geq \\lambda_{\\min}\\|\\nabla f_k(x_k)\\|^2+\\nabla f_k(x_k)^TH_kw_k.\n\\end{align*}\nTaking expectations conditioned on $\\sF_k$, from the preceding inequality, we obtain\n\\begin{align}\\label{equ:Term1}\n\\EXP{\\hbox{Term } 1\\mid\\sF_k} &\\geq \\lambda_{\\min}\\|\\nabla f_k(x_k)\\|^2+\\EXP{\\nabla f_k(x_k)^TH_kw_k\\mid\\sF_k}\\\\ \\notag&=\\lambda_{\\min}\\|\\nabla f_k(x_k)\\|^2+\\nabla f_k(x_k)^TH_k\\EXP{w_k\\mid\\sF_k} =\\lambda_{\\min}\\|\\nabla f_k(x_k)\\|^2,\\end{align}\nwhere we recall that $\\EXP{w_k\\mid\\sF_k}=0$ and $\\EXP{H_k\\mid\\sF_k}=H_k$. Similarly, in Term 2, invoking Assumption \\ref{assump:Hk}(b), we may write\n\\begin{align*}\n\\hbox{Term } 2&= (\\nabla f_k(x_k)+w_k)^TH_k^2(\\nabla f_k(x_k)+w_k)\\leq (\\lambda\\mu^{\\alpha}_k)^2\\|\\nabla f_k(x_k)+w_k\\|^2 \\\\ &=\\lambda^2\\mu^{2\\alpha}_k\\left(\\|\\nabla f_k(x_k)\\|^2+\\|w_k\\|^2+2\\nabla f_k(x_k)^Tw_k\\right).\\end{align*}\nTaking conditional expectations from the preceding inequality, and using Assumption \\ref{assum:main}, we obtain\n\\begin{align}\\label{equ:Term2}\n\\EXP{\\hbox{Term } 2\\mid\\sF_k}&\\leq\\lambda^2\\mu^{2\\alpha}_k\\left(\\|\\nabla f_k(x_k)\\|^2+\\EXP{\\|w_k\\|^2\\mid\\sF_k}+2\\nabla f_k(x_k)^T\\EXP{w_k\\mid\\sF_k}\\right)\\notag\\\\ & \\leq \\lambda^2\\mu^{2\\alpha}_k\\left(\\|\\nabla f_k(x_k)\\|^2+\\nu^2\\right).\\end{align}\nNext, taking conditional expectations in \\eqref{ineq:term1-2}, and using \\eqref{equ:Term1} and \\eqref{equ:Term2}, we obtain\n\\begin{align*}%\\label{ineq:lemma2rel7}\n&\\EXP{f_k(x_{k+1})\\mid\\sF_k} \\leq f_k(x_k)-\\g_k\\lambda_{\\min}\\|\\nabla f_k(x_k)\\|^2\\notag \\\\\n&+\\lambda^2\\mu^{2\\alpha}_k\\frac{ (L+\\mu_k)}{2}\\g_k^2\\left(\\|\\nabla f_k(x_k)\\|^2+\\nu^2\\right)\\notag \\\\\n&\\leq f_k(x_k)-\\frac{\\g_k\\lambda_{\\min}}{2}\\|\\nabla f_k(x_k)\\|^2\\left(2-\\frac{\\lambda^2\\mu_k^{2\\alpha}\\g_k(L+\\mu_k)}{\\lambda_{\\min}}\\right)+\\lambda^2\\mu^{2\\alpha}_k\\frac{ (L+\\mu_k)}{2}\\g_k^2\\nu^2.\n\\end{align*}\nFrom the assumption that $\\g_k$ and $\\mu_k$ satisfy $\\g_k\\mu^{2\\alpha}_k \\leq \\frac{\\lambda_{\\min}}{\\lambda^2(L+\\mu_0)}$ for any $k \\geq 0$ and that $\\mu_k$ is non-increasing, we have $\\g_k\\mu^{2\\alpha}_k \\leq \\frac{\\lambda_{\\min}}{\\lambda^2(L+\\mu_k)}$. As a consequence, we get $2-\\frac{\\lambda^2\\mu_k^{2\\alpha}\\g_k(L+\\mu_k)}{\\lambda_{\\min}}\\geq 1$.  Therefore, from \\fy{the} preceding inequality, we obtain\n\\begin{align*}\n\\EXP{f_k(x_{k+1}) \\mid\\sF_k}&\\leq f_k(x_k)-\\frac{\\g_k\\lambda_{\\min}}{2}\\|\\nabla f_k(x_k)\\|^2+\\lambda^2\\mu^{2\\alpha}_k\\frac{ (L+\\mu_k)}{2}\\g_k^2\\nu^2.\n\\end{align*}\nEmploying Property \\ref{proper:propsfk}(c), we have\n\\begin{align*}\n\\EXP{f_k(x_{k+1})\\mid\\sF_k} &\\leq f_k(x_k)-\\lambda_{\\min}\\mu_k\\g_k(f_k(x_k)-f_k(x^*_k))+\\lambda^2\\mu^{2\\alpha}_k\\frac{ (L+\\mu_k)}{2}\\g_k^2\\nu^2.\n\\end{align*}\nNote that, since $\\mu_k$ is a non-increasing sequence, Definition \\ref{def:regularizedF} implies that \\[\\EXP{f_{k+1}(x_{k+1})\\mid \\sF_k} \\leq\\EXP{f_{k}(x_{k+1})\\mid \\sF_k}.\\] Therefore, we obtain \n\\begin{align}\\label{ineq:lemmaLastIneq}\n\\EXP{f_{k+1}(x_{k+1})\\mid\\sF_k} &\\leq f_k(x_k)-\\lambda_{\\min}\\mu_k\\g_k(\\underbrace{f_k(x_k)-f_k(x^*_k)}_{\\tiny{\\hbox{Term } 3}})+\\lambda^2\\mu^{2\\alpha}_k\\frac{ (L+\\mu_k)}{2}\\g_k^2\\nu^2.\n\\end{align}\nNext, we derive a lower bound for Term 3. Since $x_k^*$ is the unique minimizer of $f_k$, we have $f_k(x_k^*) \\leq f_k(x^*)$. Therefore, invoking Definition \\ref{def:regularizedF}, for an arbitrary optimal solution $x^* \\in X^*$, we have \n\\[f_k(x_k)-f_k(x^*_k)\\geq f_k(x_k)-f_k(x^*)=f_k(x_k)-f^*-\\frac{\\mu_k}{2}\\|x^*-x_0\\|^2.\\]\nFrom the preceding relation and \\eqref{ineq:lemmaLastIneq}, we have\n\\begin{align*}\n\\EXP{f_{k+1}(x_{k+1})\\mid\\sF_k} &\\leq f_k(x_k)-\\lambda_{\\min}\\mu_k\\g_k(f_k(x_k)-f^*)+\\frac{\\lambda_{\\min}\\|x^*-x_0\\|^2}{2}\\mu_k^2\\g_k\\\\\n&+\\frac{(L+\\mu_k)\\lambda^2\\nu^2}{2}\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align*}\nSince $x^*$ is an arbitrary optimal solution, taking minimum from the right-hand side of the preceding inequality over $X^*$, we can replace $\\|x^*-x_0\\|$ by $\\mbox{dist}(x_0,X^*)$. Then, subtracting $f^*$ from both sides of the resulting relation yields the desired inequality.\n\\end{proof}\n\n\\begin{proposition}[Convergence in mean]\\label{prop:mean}\nConsider the \\eqref{eqn:LM-cyclic-reg-BFGS} scheme. \nSuppose Assumptions~\\ref{assum:convex},\n\\ref{assum:main}, \\ref{assump:Hk}, and \\ref{assum:sequences-ms-convergence} hold. \nThen, there exists $K\\geq 1$ such that:\\begin{align}\\label{ineq:bound}\n \\EXP{f(x_{k+1})}-f^*\\leq \\theta\\g_k\\mu_k^{2\\alpha-1}, \\quad \\hbox{for all } k \\geq K,\n \\end{align}\n  where $f^*$ is the optimal value of problem \\fy{and}\n\\begin{align}\\label{def:theta}\\theta := \\max \\bigg\\{\\frac{\\EXP{f_{K+1}(x_{K+1})}-f^*}{\\g_K\\mu_K^{2\\alpha-1}},\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2\\fy{\\lambda_{\\min}}(1-\\beta)}\\bigg\\}.\\end{align}\n Moreover, $\\lim_{k\\to \\infty}\\EXP{f(x_{k})}=f^*.$\n\\end{proposition}\n\n\\begin{proof}\nNote that Assumption \\ref{assum:sequences-ms-convergence}(a,b) imply that \\eqref{ineq:cond-recursive-F-k} holds for a large enough $k$, say after $\\hat K$. Then, since the conditions of Lemma \\ref{lemma:main-ineq} are met \\fy{(with an index shift)}, taking expectations on both sides of \\eqref{ineq:cond-recursive-F-k}, we obtain for any $k\\geq \\hat K$:\n\\begin{align*}\n\\EXP{f_{k+1}(x_{k+1})-f^*} &\\leq (1-\\lambda_{\\min}\\mu_k\\g_k)\\EXP{f_k(x_k)-f^*}+\\frac{\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)}{2}\\mu_k^2\\g_k\\\\ &+\\frac{ (L+\\mu_0)\\lambda^2\\nu^2}{2}\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align*}\n%{\\bf Comment: the relation in (9) has different coefficients than originally - edits needed from here onward}\n Using Assumption \\ref{assum:sequences-ms-convergence}(e), we have $\\mu_k^2\\g_k\\leq \\rho \\g_k^2\\mu_k^{2\\alpha}$. Thus, we obtain\n\\begin{align}\\label{ineq:cond-recursive-F-k-expected2}\n\\EXP{f_{k+1}(x_{k+1})-f^*} &\\leq (1-\\lambda_{\\min}\\mu_k\\g_k)\\EXP{f_k(x_k)-f^*}\\notag\\\\ &+\\left(\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2}\\right)\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align}\nLet us define \\fy{$K\\triangleq\\max\\{\\hat K, K_0\\}$, where $K_0$ is from Assumption \\ref{assum:sequences-ms-convergence}(d)}. Using the preceding relation and by induction on $k$, we show the desired result. To show \\eqref{ineq:bound}, we show the following relation first: \n\\begin{align}\\label{ineq:bound_v2}\n \\EXP{f_{k+1}(x_{k+1})}-f^*\\leq \\theta\\g_k\\mu_k^{2\\alpha-1}, \\quad \\hbox{for all } k \\geq K,\n \\end{align}\n Note that \\eqref{ineq:bound_v2} \\fy{implies} the relation \\eqref{ineq:bound} since we have $ \\EXP{f(x_{k+1})}\\leq  \\EXP{f_{k+1}(x_{k+1})}$.\nFirst, we show that \\eqref{ineq:bound_v2} holds for \\fy{$k=K$}.  Consider the term $\\EXP{f_{K+1}(x_{K+1})}-f^*$. Multiplying and dividing by $\\g_K\\mu_K^{2\\alpha-1}$, we obtain\n\\begin{align*}\\EXP{f_{K+1}(x_{K+1})}-f^*=\\left(\\frac{\\EXP{f_{K+1}(x_{K+1})}-f^*}{\\g_k\\mu_K^{2\\alpha-1}}\\right)\\g_K\\mu_K^{2\\alpha-1}\n \\leq \\theta \\g_K\\mu_K^{2\\alpha-1},\\end{align*}\nwhere the last inequality is obtained by invoking the definition of $\\theta$ in \\eqref{def:theta}. This implies that \\eqref{ineq:bound_v2} holds for $k=K$. Now assume that  \\eqref{ineq:bound_v2} holds for some $k \\geq K$. We show that it also holds for $k+1$.  From the induction hypothesis and \\eqref{ineq:cond-recursive-F-k-expected2} we have\n\\begin{align*}\n\\EXP{f_{k+1}(x_{k+1})-f^*} &\\leq (1-\\lambda_{\\min}\\mu_k\\g_k)\\theta\\g_{k-1}\\mu_{k-1}^{2\\alpha-1}\\\\ &+\\left(\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2}\\right)\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align*}\nUsing Assumption \\ref{assum:sequences-ms-convergence}(d) we obtain\n\\begin{align}\\label{ineq:thm2-rel1}\n\\EXP{f_{k+1}(x_{k+1})-f^*} &\\leq \\theta\\g_{k}\\mu_{k}^{2\\alpha-1}(1-\\lambda_{\\min}\\mu_k\\g_k)(1+\\beta \\lambda_{\\min}\\g_{k}\\mu_{k})\\notag \\\\ &+\\left(\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2}\\right)\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align}\nNext we find an upper bound for the term $(1-\\lambda_{\\min}\\mu_k\\g_k)(1+\\beta \\lambda_{\\min}\\g_{k}\\mu_{k}) $ as follows\n\\begin{align*}\n(1-\\lambda_{\\min}\\mu_k\\g_k)(1+\\beta \\lambda_{\\min}\\g_{k}\\mu_{k}) &= 1-\\lambda_{\\min}\\mu_k\\g_k+\\beta\\lambda_{\\min}\\mu_k\\g_k-\\fy{\\beta\\lambda_{\\min}^2\\mu_k^2\\g_k^2}\\\\ & \\leq 1-(1-\\beta)\\lambda_{\\min}\\mu_k\\g_k.\n\\end{align*}\nCombining this relation with \\eqref{ineq:thm2-rel1}, it follows \\fy{that}\n\\begin{align*}%\\label{ineq:thm2-rel1}\n&\\EXP{f_{k+1}(x_{k+1})-f^*} \\leq \\theta\\g_{k}\\mu_{k}^{2\\alpha-1} -\\theta\\fy{\\lambda_{\\min}}(1-\\beta)\\mu_k^{2\\alpha}\\g_k^2  \\\\ &+\\left(\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2}\\right)\\mu^{2\\alpha}_k\\g_k^2\\\\ & =\\theta\\g_{k}\\mu_{k}^{2\\alpha-1}-\\left(\\underbrace{\\theta\\fy{\\lambda_{min}}(1-\\beta)-\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2}}_{\\tiny{\\hbox{Term }1}}\\right)\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align*}\nNote that the definition of $\\theta$ in \\eqref{def:theta} implies that Term 1 is nonnegative. Therefore, \n\\begin{align*}%\\label{ineq:thm2-rel1}\n\\EXP{f_{k+1}(x_{k+1})-f^*} &\\leq \\theta\\g_{k}\\mu_{k}^{2\\alpha-1}.\n\\end{align*}\nHence, the induction statement holds for $k+1$. We conclude that \\eqref{ineq:bound_v2} holds for all $k\\geq K$. As a consequence, \\eqref{ineq:bound} holds for all $k\\geq K$ as well. To complete the proof, we need to show $\\lim_{k\\to \\infty}\\EXP{f(x_{k})}=f^*$. This is an immediate result of \\eqref{ineq:bound} and Assumption \\ref{assum:sequences-ms-convergence}(a).\n\\end{proof}\n\n\\begin{lemma}\\label{sumProductBounds}\nLet $0 < a_1 \\leq a_2 \\leq \\ldots \\leq a_n$, and $P$ and $S$ be positive scalars such that \\fys{$\\sum_{i=1}^n a_i \\leq S$ and $\\prod_{i=1}^na_i \\geq P$. Then, we have \n$a_1 \\geq (n-1)!P/S^{n-1}$.}\n\\end{lemma}\n\n\\begin{proof}\nSee Appendix \\ref{app:sumProductBounds}.\n\\end{proof}\n\n\\begin{lemma}[Properties of  update rule \\eqref{eqn:H-k}-\\eqref{eqn:H-k-m}]\\label{LBFGS-matrix}\nConsider the \\eqref{eqn:LM-cyclic-reg-BFGS} method. Let $H_k$ be given by the update rule \\eqref{eqn:H-k}-\\eqref{eqn:H-k-m}, where $s_i$ and $y_i$ are defined in \\eqref{equ:siyi-LBFGS} and $\\mu_k$ is updated according to the procedure \\eqref{eqn:mu-k}. Let \\fy{Assumption \\ref{assum:convex2}(a,b)} hold. Then, the following results hold:\n\\begin{itemize}\n\\item [(a)]  For any odd \\fy{$k \\geq 2m-1$}, the \\fy{curvature condition} holds, i.e., \\fy{$s_{\\lceil k/2\\rceil}^T{y_{\\lceil k/2\\rceil}} >0$}.\n\\item [(b)]  For any odd \\fy{$k \\geq 2m-1$}, the secant equation holds, i.e., \\fy{$H_{k}{y}_{\\lceil k/2\\rceil}=s_{\\lceil k/2\\rceil}$}. \n\\item [(c)] For any \\fy{$k \\geq 2m-1$}, $H_k$ satisfies Assumption \\ref{assump:Hk} \\fy{with} the following values:\n\\begin{align}\\label{equ:valuesForAssumH_k}&\\lambda_{\\min}=\\frac{1}{(m+n)\\left(L+\\fys{\\tau}\\mu_0^\\delta\\right)}, \\quad \\fy{\\lambda= \\frac{(m+n)^{n+m-1}\\left(L+\\fys{\\tau}\\mu_0^\\delta\\right)^{n+m-1}}{(n-1)!\\fys{\\tau^{(n+m)}}}}, \\notag\\\\ & \\fy{\\hbox{and }\\alpha=-\\delta(n+m).}\n\\end{align}\nMore precisely, $H_k$ is symmetric, $\\EXP{H_k\\mid\\sF_k}=H_k$ and\n\\begin{align}\\label{proof:H_kbounds} \\frac{1}{(m+n)\\left(L+\\fys{\\tau}\\mu_0^\\delta\\right)}\\mathbf{I} \\preceq H_{k} \\preceq \\frac{(m+n)^{n+m-1}\\left(L+\\fys{\\tau}\\mu_0^\\delta\\right)^{n+m-1}}{(n-1)!\\fys{\\left(\\tau\\mu_k^\\delta\\right)^{(n+m)}}}\\ \\mathbf{I}.\n\\end{align}\n\\end{itemize}\n\\end{lemma}\n\n\\begin{proof}\nSee Appendix \\ref{app:lemmaLBFGSmatrix}.\n\\end{proof}\n\n\\begin{lemma}[Feasible tuning sequences for a.s. convergence \\fy{(Proposition \\ref{prop:a.s})}]\\label{lemma:a.s.sequences}\nLet the sequences $\\g_k$ and $\\mu_k$ be given by the following rules:\n\\begin{align}\\label{equ:seq}\n \\g_k=\\frac{\\g_0}{(k+1)^a}, \\quad \\mu_{k}=\\frac{\\fy{2^b\\mu_0}}{\\left(k+\\kappa\\right)^b},\n\\end{align}\nwhere $\\kappa=2$ if $k$ is even and $\\kappa=1$ otherwise, $\\g_0$and $\\mu_0$ are positive scalars such that $\\g_0\\mu_0 \\leq L(m+n)$, \\fy{and $a$, $b$, and $\\delta\\leq 1$ are positive scalars \\fys{satisfying}:\n\\begin{align*}& \\fy{\\frac{a}{b}>1+2\\delta(n+m)}, \\quad a+b \\leq 1,\\quad a+2b > 1, \\quad\\hbox{and}\\quad a-\\delta b(m+n)>0.5.\\end{align*}\nThen, $\\g_k$ and $\\mu_k$ satisfy Assumption \\ref{assum:sequences} with $\\lambda_{\\min}$ and $\\alpha$ in \\eqref{equ:valuesForAssumH_k} and $\\mu_k$ satisfies \\eqref{eqn:mu-k}.}\n\\end{lemma}\n\n\\begin{proof}\nSee Appendix \\ref{app:feasibleSeqASconv}.\n\\end{proof}\n\n\\begin{lemma}[Feasible tuning sequences for convergence in mean \\fy{(Proposition \\ref{prop:mean})}]\\label{lemma:mean-sequences}\nLet the sequences $\\g_k$ and $\\mu_k$ be given by \\eqref{equ:seq},\nwhere $\\g_0$ and $\\mu_0$ are positive scalars such that $\\g_0\\mu_0 \\leq L(m+n)$. \\fy{Let, $0 <\\delta \\leq 1$ $a>0$ and $b>0$ \\fys{satisfying}:\n\\begin{align*}& \\frac{a}{b}>1+2\\delta\\left(m+n\\right), \\quad a+b < 1,\n\\quad \\frac{a}{b} \\leq\n2\\left(1+\\delta\\left(m+n\\right)\\right).\\end{align*}Then, \\vus{$\\mu_k$\nsatisfies \\eqref{eqn:mu-k}} and $\\g_k$ and $\\mu_k$ satisfy Assumption\n\\ref{assum:sequences-ms-convergence} with any arbitrary $0<\\beta<1$,\n$\\rho\\triangleq \\g_0^{-1}\\left(\\mu_02^b\\right)^{2+2\\delta(m+n)}$,\nand with $\\lambda_{\\min}$ and $\\alpha$ given by\n\\eqref{equ:valuesForAssumH_k}.} \\end{lemma}\n\n\\begin{proof}\nSee Appendix \\ref{app:feasibleSeqMeanConv}.\n\\end{proof}\n\n\\begin{proof}\n\\noindent (a) First, we show that the conditions of Proposition~\\ref{prop:mean} are satisfied. Assumption \\ref{assum:convex} holds as a consequence of Assumption \\ref{assum:convex2}. From Lemma \\ref{LBFGS-matrix}(c), Assumption \\ref{assump:Hk} holds for any \\fy{$k \\geq 2m-1$} as well. To show that Assumption \\ref{assum:sequences-ms-convergence} holds, we apply Lemma \\ref{lemma:mean-sequences}. We have \n\\[\\frac{a}{b}=\\frac{\\frac{2}{3}-\\e+\\frac{2\\delta(n+m)}{3}}{1/3}=2-3\\e+2\\delta(n+m)>1+2\\delta\\left(m+n\\right),\\]\nwhere we used $\\e<\\frac{1}{3}$. Moreover, since $\\delta < \\frac{1.5\\e}{n+m}$, we have $a+b=1-\\e +\\frac{2\\delta(n+m)}{3}<1$. Also, from the values of $a$ and $b$ we have $2b\\left(1+\\delta\\left(m+n\\right)\\right)=a+\\e>a.$\nThus, the conditions of Lemma \\ref{lemma:mean-sequences} hold. This implies that there exists $K_0>0$ such that for any $k\\geq K_0$,  the sequences $\\g_k$ and $\\mu_k$ satisfy Assumption \\ref{assum:sequences-ms-convergence} with any arbitrary $0<\\beta<1$ and for $\\rho=\\g_0^{-1}\\left(\\mu_02^b\\right)^{2+2\\delta(m+n)}$, and with $\\lambda_{\\min}$, $\\lambda$ and $\\alpha$ given by \\eqref{equ:valuesForAssumH_k}. Let us define $K:=\\max\\{K_0,2m-1\\}$. Since all conditions in Proposition~\\ref{prop:mean} are met, \\an{from \\eqref{ineq:bound}, \\eqref{equ:seq}, and by substituting values of $a$, $b$, and $\\alpha$, for any $k \\geq K$ we obtain}\n\\begin{align*}\\EXP{f(x_{k+1})}-f^*&\\leq \\theta\\g_{k-1}\\mu_{k-1}^{2\\alpha-1}= \\frac{\\theta\\g_0(k+\\kappa-1)^{(1-2\\alpha)/3}}{\\left(\\mu_0\\sqrt[3]{2}\\right)^{1-2\\alpha}k^{2/3-\\e-\\frac{2}{3}\\alpha}}\\\\ &\\leq  \\frac{\\theta\\g_0(k+1)^{(1-2\\alpha)/3}}{\\left(\\mu_0\\sqrt[3]{2}\\right)^{1-2\\alpha}(k+1)^{2/3-\\e-\\frac{2}{3}\\alpha}}=\\left(\\frac{\\theta\\g_0}{\\left(\\mu_0\\sqrt[3]{2}\\right)^{1-2\\alpha}}\\right)\\frac{1}{(k+1)^{\\frac{1}{3}-\\e}}.\\end{align*}\nThrough a change of variable from $k+1$ to $k$, we conclude the result. \\\\\n\n\\fy{\\noindent (b)  To show \\eqref{eqn:r_Lemma}, it suffices to show that\n \\begin{align*}\n  r=\\begin{cases}\n    H_{k,m}\\left(\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)\\right),  &    \\text{if } k \\text{ is odd}, \\\\\n    H_{k-1,m}\\left(\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)\\right),  &   \\text{otherwise},\n  \\end{cases}\n\\end{align*}\nwhere $H_{k,m}$ is defined by the recursion \\eqref{eqn:H-k-m} for an odd $k$. First, consider the case that $k\\geq 2m-1$ is an odd number. As such, at the $k$th iteration, from line $\\#$7, we have $i:=\\lceil k/2 \\rceil$. For clarity of the presentation, throughout \\vus{this} proof, we use $K$ (instead of $i$), i.e., $K\\triangleq \\lceil k/2 \\rceil$ \\vus{and} $q_{K-t+1}$ \\vus{is used to} denote the value of the vector $q \\in \\mathbb{R}^n$ after being updated at iteration $t$ in line $\\#$21. Similarly, we use $r_{t-K+m}$ to denote the value of the vector $r \\in \\mathbb{R}^n$ after being updated at iteration $t$ in line $\\#$25. Also, we use the following definitions:\n\\begin{align*}\n& q_0\\triangleq \\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0), \\quad r_0\\triangleq H_{k,0}q_{m},\\cr \n&\\rho_j\\triangleq \\frac{1}{y_j^Ts_j}, \\hbox{ and }V_j\\triangleq \\mathbf{I}-\\rho_{j}y_js_j^T, \\quad \\hbox{for all } j=K-(m-1),\\ldots, K.\n\\end{align*}\nConsider relation \\eqref{eqn:H-k-m}. By applying this recursive relation repeatedly, we obtain\n\\begin{align}\\label{equ:H_kClosedForm}\nH_{k,m} &=\\left(\\prod_{j=1}^{m}V_{K-(m-j)}\\right)^TH_{k,0}\\left(\\prod_{j=1}^{m}V_{K-(m-j)}\\right)\\\\\n&+ \\rho_{K-m+1}\\left(\\prod_{j=2}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+1}s^T_{K-m+1}\\left(\\prod_{j=2}^{m}V_{K-(m-j)}\\right) \\notag\\\\\n& + \\rho_{K-m+2}\\left(\\prod_{j=3}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+2}s^T_{K-m+2}\\left(\\prod_{j=3}^{m}V_{K-(m-j)}\\right)\\notag\\\\\n& + \\ldots\\notag\\\\\n& +\\rho_{K-1}V^T_{K}s_{K-1}s_{K-1}^TV_{K}\\notag\\\\\n&+\\rho_{K}s_{K}s_{K}^T.\\notag\n\\end{align}\nNext, we derive a formula for $q_t$. From lines $\\#$20-21 in the algorithm, we have\n\\begin{align*}\nq_{K-t+1}&=q_{K-t} -\\alpha_{K-t+1}y_t =q_{K-t} - \\rho_t\\left(s_t^Tq_{K-t}\\right)y_t = q_{K-t} - \\rho_t\\left(y_ts_t^T\\right)q_{K-t} \\\\\n&= \\left( \\mathbf{I}-\\rho_{t}y_ts_t^T\\right)q_{K-t}=V_tq_{K-t}, \\quad \\hbox{for all } t= K, K-1, \\ldots, K-m+1.\n\\end{align*}\nFrom the preceding relation, we obtain\n\\begin{align}\\label{equ:q_ell}\nq_{\\ell}=\\left(\\prod_{j=m-\\ell+1}^{m}V_{K-(m-j)}\\right)q_0, \\quad \\hbox{for all } \\ell= 1,2, \\ldots, m.\n\\end{align}\nFrom the update rule for $\\alpha_{i-t+1}$ in line $\\#$20, using the definition of $\\rho_t$, and applying the previous relation, we have $\\alpha_1=\\rho_{K}s_K^Tq_0$ and \n\\begin{align}\\label{equ:alpha_ell}\n\\alpha_{\\ell}=\\rho_{K-\\ell+1}s_{K-\\ell+1}^T\\left(\\prod_{j=m-\\ell+2}^{m}V_{K-(m-j)}\\right)q_0, \\quad \\hbox{for all } \\ell= 2,3, \\ldots, m.\n\\end{align}\n\\fys{Multiplying both sides of} \\eqref{equ:H_kClosedForm} by $q_0$ and employing \\eqref{equ:q_ell} and \\eqref{equ:alpha_ell}, we obtain\n\\begin{align}\\label{equ:H_kq_0}\nH_{k,m}q_0 &=\\left(\\prod_{j=1}^{m}V_{K-(m-j)}\\right)^TH_{k,0}q_m + \\left(\\prod_{j=2}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+1}\\alpha_m \\\\\n& + \\left(\\prod_{j=3}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+2}\\alpha_{m-1}  +\\ldots+V^T_{K}s_{K-1}\\alpha_2+s_{K}\\alpha_1.\\notag\n\\end{align}\nNext, we derive a formula for $r_t$. From line $\\#$25 in the algorithm, we have \n\\begin{align*}\nr_{t-K+m}&=r_{t-K+m-1}+\\left(\\alpha_{K-t+1}-\\rho_ty_t^Tr_{t-K+m-1}\\right)s_t \\\\\n&= r_{t-K+m-1}-\\rho_ts_t y_t^Tr_{t-K+m-1}+\\alpha_{K-t+1}s_t\\\\\n& = V_t^Tr_{t-K+m-1}+\\alpha_{K-t+1}s_t, \\quad \\hbox{for all } t= K-m+1,\\ldots,K-1,  K.\n\\end{align*}\nCombining the preceding two relations, we obtain\n\\begin{align*}\nr_{\\ell}&= V_{K-(m-\\ell)}^Tr_{\\ell-1}+\\alpha_{m-\\ell+1}s_{K-(m-\\ell)}, \\quad \\hbox{for all } \\ell= 1,2,\\ldots,m.\n\\end{align*}\nUsing the preceding equation repeatedly, we obtain\n\\begin{align}\\label{equ:r_m}\nr_m &=\\left(\\prod_{j=1}^{m}V_{K-(m-j)}\\right)^Tr_0 +\\alpha_m \\left(\\prod_{j=2}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+1} \\\\\n& + \\alpha_{m-1} \\left(\\prod_{j=3}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+2} +\\ldots+\\alpha_2V^T_{K}s_{K-1}+\\alpha_1s_{K}.\\notag\n\\end{align}\nFrom \\eqref{equ:r_m} and \\eqref{equ:H_kq_0}, and the definition of  $r_0$, we obtain $r_m=H_{k,m}q_0$. Taking to account the definition of $q_0$, the desired result holds for any odd $k\\geq 2m-1$. Now, consider the case where $k\\geq 2m-1$ is an even number. This implies that the ``if'' condition in line $\\#$6 is skipped and as such, the value of $i$ is not updated from the iteration $k-1$, i.e.,  $i= \\lceil (k-1)/2 \\rceil$. Consequently, the LBFGS two-loop recursion at an even $k$ uses the following pairs \n$$\\left\\{(s_\\ell,y_\\ell)\\mid \\ell=\\lceil (k-1)/2\\rceil-m+1, \\lceil (k-1)/2\\rceil-m+2,\\ldots,\\lceil (k-1)/2\\rceil\\right\\}.$$\n\\fys{Now,} considering the definition \\eqref{eqn:H-k-m} for $k-1$, the desired relation can be shown following the same steps discussed for the case the iteration number is odd.}\n%This implies the desired rate statement holds. \n\\end{proof}\n\n\\subsection{Proof of Lemma \\ref{sumProductBounds}}\\label{app:sumProductBounds}\nFrom $0<a_1 \\leq \\ldots \\leq a_n$, we can write \n\\begin{align*}\n(n-(i-1))a_i \\leq \\sum_{j=1}^na_j, \\qquad \\hbox{for all } i \\in \\{1,\\ldots,n\\}.\n\\end{align*}\nInvoking $\\sum_{i=1}^n{a_i} \\leq S$, we obtain $a_i \\leq \\frac{S}{n-(i-1)}$, for all $i \\in \\{1,\\ldots,n\\}$. From the preceding relation and that $\\prod_{j=1}^na_j \\geq P$, we can obtain $a_1 \\geq (n-1)!P/S^{n-1}.$\n}\n\n\n\\begin{proof} \\fys{Throughout, we let $\\lambda_{k,\\min}$, $\\lambda_{k,\\max}$, and $B_k$ denote the minimum eigenvalue, maximum eigenvalue, and inverse of matrix $H_k$ in \\eqref{eqn:H-k-m}, respectively.}\nIt can be seen, by induction on $k$, that \\fy{$H_k$ is symmetric and $\\sF_k$ measurable.} We use induction on odd values of \\fys{$k\\geq 2m-1$ to show that parts (a), (b), and (c) hold.} Suppose \\fy{$k\\geq 2m-1$} is odd and for any \\fys{odd} $t<k$, we have \\fy{$s_{\\lceil t/2\\rceil}^T{y_{\\lceil t/2\\rceil}} >0$, $H_{t}{y}_{\\lceil t/2\\rceil}=s_{\\lceil t/2\\rceil}$}, and \\eqref{proof:H_kbounds} for $t$. We show that these statements also hold for $k$ \\fys{as well}. First, we \\fys{show} that the \\fy{curvature condition} holds. We can write \n\\begin{align*}\n\\fys{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}&=(x_{k}-x_{k-1})^T(\\nabla F(x_k,\\xi_{k-1})- \\nabla F(x_{k-1},\\xi_{k-1})+\\fys{\\tau}\\mu_k^\\delta(x_k-x_{k-1}))\\\\\n&=(x_{k}-x_{k-1})^T(\\nabla F(x_k,\\xi_{k-1})- \\nabla F(x_{k-1},\\xi_{k-1}))+\\fys{\\tau}\\mu_k^\\delta\\|x_k-x_{k-1}\\|^2\\\\\n&\\geq \\fys{\\tau}\\mu_k^\\delta\\|x_k-x_{k-1}\\|^2,\n\\end{align*}\nwhere the inequality follows from the monotonicity of the gradient map $\\nabla F(\\cdot,\\xi)$. \\fys{Next, we show that $\\|x_k-x_{k-1}\\|^2 >0$.} From the induction hypothesis \\fys{and that $k-2$ is odd}, $H_{k-2}$ is positive definite.\n\\fys{Moreover, from the update rule \\eqref{eqn:H-k} and that $k-2$ is odd, we have $H_{k-1}=H_{k-2}$.}  \n%Also, since $B_{k-2}$ is symmetric, the update rule \\eqref{eqn:B-k} implies that $B_{k-1}$ is symmetric.\nTherefore, $H_{k-1}$ is \\fys{also} positive definite. \\fy{Without loss of generality, we assume $\\nabla F(x_{k-1},\\xi_{k-1})+\\mu_{k-1}(x_{k-1}-x_0)\\neq 0$}\\footnote{If $\\nabla F(x_{k},\\xi_k)+\\mu_{k}(x_k-x_0)=0$\\fys{,} then we can draw a new sample of $\\xi_k$ to satisfy the relation.}.\nSince $H_{k-1}$ is positive definite, \nwe have $$H_{k-1}\\left(\\nabla F(x_{k-1},\\xi_{k-1})+\\mu_{k-1}\\fy{(x_{k-1}-x_0)}\\right) \\neq 0,$$ implying that \n$x_{k} \\neq x_{k-1}$. Hence\n$\\fy{s_{\\lceil k/2\\rceil}^T{y_{\\lceil k/2\\rceil}} }\\geq  \\fys{\\tau}\\mu_k^\\delta\\|x_k-x_{k-1}\\|^2 >0,$\nwhere \\us{the second inequality is a consequence of} $\\fys{\\tau},\\mu_k>0$.\nThus, the \\fy{curvature condition} holds.\nNext, we show that \\eqref{proof:H_kbounds} holds for $k$. It is well-known that using the Sherman-Morrison-Woodbury formula, $B_k$ is equal to $B_{k,m}$ given by \n\\begin{align}\\label{equ:B_kLimited}\nB_{k,j}=B_{k,j-1}-\\frac{B_{k,j-1}s_is_i^TB_{k,j-1}}{s_i^TB_{k,j-1}s_i}+\\frac{y_iy_i^T}{y_i^Ts_i}, \\quad i:=\\fy{{\\lceil k/2\\rceil}-(m-j)}, \\quad 1 \\leq j \\leq m,\n\\end{align}\nwhere $s_i$ and $y_i$ are defined by \\eqref{equ:siyi-LBFGS} and \\fy{$B_{k,0}=\\frac{y_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}\\mathbf{I}$}. \\fys{Note that with $j$ varying between $1$ to $m$, the index $i$ takes values in $\\left\\{\\lceil k/2\\rceil-m+1,\\lceil k/2\\rceil-m+2,\\ldots,\\lceil k/2\\rceil\\right\\}$.} First, we show that for any $i$ \\fys{in this range}, \\begin{align}\\label{equ:boundsForB0}\n\\fys{\\tau}\\mu_k^\\delta \\leq \\frac{\\|y_i\\|^2}{y_i^Ts_i} \\leq L+\\fys{\\tau}\\mu_k^\\delta,\n\\end{align}\n where $L$ is the Lipschitzian parameter of the gradient mapping $\\nabla F$ \\fys{given by Assumption \\ref{assum:convex2}(b)}. Let us \\fys{define the function $h(x)\\triangleq F(x,\\xi_{i-1})+\\tau\\frac{\\mu_k^\\delta}{2}\\|x-x_0\\|^2$} for fixed $i$ and $k$. Note that this function is strongly convex and has a gradient mapping of the form $\\nabla F+\\fys{\\tau}\\mu_k^\\delta(\\mathbf{I}-x_0)$ that  is Lipschitz with parameter $L+\\fys{\\tau}\\mu_k^\\delta$. For a convex function $h$ with Lipschitz gradient with parameter $L+\\fys{\\tau}\\mu_k^\\delta$, the following inequality, referred to as co-coercivity  property, holds for any $x_1,x_2 \\in \\Real^n$ \\fys{(see}\\cite{Polyak87}, Pg. 24 , Lemma 2):\n \\[\\|\\nabla h(x_2)-\\nabla h(x_1)\\|^2 \\leq \\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)(x_2-x_1)^T(\\nabla h(x_2)-\\nabla h(x_1)).\\]\nSubstituting $x_2$ by $x_i$, $x_1$ by $x_{i-1}$, and recalling \\eqref{equ:siyi-LBFGS}, the preceding inequality yields\n\\begin{align}\\label{ineq:boundsForB0-1}\\|y_i\\|^2 \\leq \\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)s_i^Ty_i.\\end{align}\nNote that function $h$ is strongly convex \\fy{with} parameter $\\fys{\\tau}\\mu_k^\\delta$. Applying the Cauchy-Schwarz inequality, we can write\n\\[\\frac{\\|y_i\\|^2}{s_i^Ty_i} \\geq \\frac{\\|y_i\\|^2}{\\|s_i\\|\\|y_i\\|} =\\frac{\\|y_i\\|}{\\|s_i\\|}\\geq  \\frac{\\|y_i\\|\\|s_i\\|}{\\|s_i\\|^2} \\geq \\frac{y_i^Ts_i}{\\|s_i\\|^2}\\geq \\fys{\\tau}\\mu_k^\\delta.\\]\nCombining this relation with \\eqref{ineq:boundsForB0-1}, we obtain \\eqref{equ:boundsForB0}. Next, we show that the maximum eigenvalue of $B_k$ is bounded. Let $Trace(\\cdot)$ denote the trace of a matrix. Taking trace from both sides of \\eqref{equ:B_kLimited} and summing up over index $j$, we obtain \\fy{for $i:={\\lceil k/2\\rceil}-(m-j)$,}\n\\begin{align}\\label{ineq:trace}\nTrace(B_{k,m})&=Trace(B_{k,0})-\\sum_{j=1}^m Trace\\left(\\frac{B_{k,j-1}s_is_i^TB_{k,j-1}}{s_i^TB_{k,j-1}s_i}\\right)+\\sum_{j=1}^m Trace\\left(\\frac{y_iy_i^T}{y_i^Ts_i}\\right)\\cr\n& =Trace\\fy{\\left(\\frac{\\|y_{\\lceil k/2\\rceil}\\|^2}{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}\\mathbf{I}\\right)} - \\sum_{j=1}^m \\frac{\\|B_{k,j-1}s_i\\|^2}{s_i^TB_{k,j-1}s_i} + \\sum_{j=1}^m \\frac{\\|y_i\\|^2}{y_i^Ts_i}\\cr \n&\\leq n \\fy{\\frac{\\|y_{\\lceil k/2\\rceil}\\|^2}{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}} +\\sum_{j=1}^m \\left(L+\\fys{\\tau}\\mu_k^\\delta\\right) \\fy{\\leq} (m+n)\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right),\n\\end{align}\nwhere the third relation is obtained by positive-definiteness of $B_k$ (this can be seen by induction on \\fys{$j$}, and using \\eqref{equ:B_kLimited} and $B_{k,0}\\succ 0$). Since $B_k=B_{k,m}$, the maximum eigenvalue of the matrix $B_k$ is bounded \\fys{by $(m+n)\\left(L+\\tau\\mu_k^\\delta\\right)$}. As a result, \n\\begin{align}\\label{proof:lowerbound}\n\\lambda_{k,\\min}\\geq \\frac{1}{(m+n)\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)}.\\end{align}\n In the next part of the proof, we establish the bound for $\\lambda_{k,\\max}$. The following relation can be shown (e.g., see Lemma 3 in \\cite{Mokhtari15})\n\\begin{align*}\ndet(B_{k,m})=det(B_{k,0})\\prod_{j=1}^m\\frac{s_i^Ty_i}{s_i^TB_{k,j-1}s_i}\\fy{, \\quad \\hbox{for }i:={\\lceil k/2\\rceil}-(m-j)}.\n\\end{align*}\nMultiplying and dividing by $s_i^Ts_i$, using the strong convexity of the function $h$, and invoking \\eqref{equ:boundsForB0} and the result of \\eqref{ineq:trace}, we obtain\n\\begin{align}\\label{ineq:detBk}\ndet(B_{k})&=\\fy{det\\left(\\frac{y_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}\\mathbf{I}\\right)}\\prod_{j=1}^m\\left(\\frac{s_i^Ty_i}{s_i^Ts_i}\\right)\\left(\\frac{s_i^Ts_i}{s_i^TB_{k,j-1}s_i}\\right)\\cr  \n& \\geq\\fy{\\left(\\frac{y_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}\\right)^n}\\prod_{j=1}^m\\fys{\\tau}\\mu_k^\\delta\\left(\\frac{s_i^Ts_i}{s_i^TB_{k,j-1}s_i}\\right)\\cr\n& \\geq  \\fys{\\left(\\tau\\mu_k^\\delta\\right)^{(n+m)}} \\prod_{j=1}^m \\frac{1}{(m+n)\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)} = \\frac{\\fys{\\left(\\tau\\mu_k^\\delta\\right)^{(n+m)}}}{(m+n)^{m}\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)^m}.\n\\end{align}\nLet \\fy{$\\alpha_{k,1}\\leq \\alpha_{k,2}\\leq\\ldots\\leq\\alpha_{k,n}$} be the eigenvalues of $B_k$ sorted non-decreasingly. Note that since $B_k\\succ0$, all the eigenvalues are positive. Taking \\eqref{ineq:trace} and \\eqref{ineq:detBk} into account, and employing Lemma \\ref{sumProductBounds}, we obtain\n \\[\\alpha_{1,k} \\fy{\\geq} \\frac{(n-1)!\\fys{\\left(\\tau\\mu_k^\\delta\\right)^{(n+m)}}}{(m+n)^{n+m-1}\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)^{n+m-1}}.\\]\nThis relation and that $\\alpha_{k,1}=\\lambda_{k,\\max}^{-1}$ imply that \n\\begin{align}\\label{proof:upperbound}\n\\lambda_{k,\\max}\\leq \\frac{(m+n)^{n+m-1}\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)^{n+m-1}}{(n-1)!\\fys{\\left(\\tau\\mu_k^\\delta\\right)^{(n+m)}}}.\n\\end{align}\nTherefore, from \\eqref{proof:lowerbound} and \\eqref{proof:upperbound} and that $\\mu_k$ is non-increasing, we conclude that \\eqref{proof:H_kbounds} holds for $k$ as well. Next, we show $H_ky_{\\lceil k/2\\rceil}=s_{\\lceil k/2\\rceil}$. From \\eqref{equ:B_kLimited}, for $j=m$ we obtain\n\\[B_{k,m}=B_{k,m-1}-\\fy{\\frac{B_{k,m-1}s_{\\lceil k/2\\rceil}s_{\\lceil k/2\\rceil}^TB_{k,m-1}}{s_{\\lceil k/2\\rceil}^TB_{k,m-1}s_{\\lceil k/2\\rceil}}+\\frac{y_{\\lceil k/2\\rceil}y_{\\lceil k/2\\rceil}^T}{y_{\\lceil k/2\\rceil}^Ts_{\\lceil k/2\\rceil}}},\\]\nwhere we used \\fy{$i={\\lceil k/2\\rceil}-(m-m)={\\lceil k/2\\rceil}$}. Multiplying both sides of the preceding equation by \\fy{$s_{\\lceil k/2\\rceil}$}, and using $B_k=B_{k,m}$, we have\n\\fy{$B_{k}s_k=B_{k,m-1}s_{\\lceil k/2\\rceil}-B_{k,m-1}s_{\\lceil k/2\\rceil}+y_{\\lceil k/2\\rceil}=y_{\\lceil k/2\\rceil}$}. \nMultiplying both sides of the preceding relation by $H_k$ and invoking $H_k=B_k^{-1}$, we conclude that \\fy{$H_ky_{\\lceil k/2\\rceil}=s_{\\lceil k/2\\rceil}$}. Therefore, we showed that the statements of (a), (b), and (c) hold for \\fys{an odd $k$}, assuming that they hold for any odd \\fy{$t<k$}. In a similar fashion to this analysis, it can be seen that the statements hold for \\fys{$t=2m-1$}. Thus, by induction, we conclude that the statements hold for any odd \\fy{$k\\geq 2m-1$}. To complete the proof, it is enough to show that \\eqref{proof:H_kbounds} holds for any even \\fys{$k\\geq 2m$}. Let $t=k-1$. Since \\fys{$t$} is odd, relation \\eqref{proof:H_kbounds} holds. Writing  \\eqref{proof:H_kbounds} for $k-1$, and taking into account that $H_k=H_{k-1}$, and $\\mu_k<\\mu_{k-1}$, we can conclude that \\eqref{proof:H_kbounds} holds for any even \\fy{$k\\geq 2m-1$} and this completes the proof.\n\\end{proof}\n\n\\begin{proof}\nIn the following, we show that the presented class of sequences satisfy each of the conditions listed in Assumption \\ref{assum:sequences}. Throughout, we let $\\alpha$ denote \\fy{$-(m+n)\\delta$}.\\\\\n(a) Replacing the sequences by their given rules, we obtain \\fy{\\begin{align*} \\g_k\\mu_k^{2\\alpha-1} & =\\frac{\\g_0}{(2^b\\mu_0)^{1-2\\alpha}}(k+1)^{-a}(k+\\kappa)^{(1-2\\alpha) b}\\leq  \\frac{\\g_0}{(2^b\\mu_0)^{1-2\\alpha}}(k+1)^{-a+(1-2\\alpha) b}.\\end{align*}}\n\\fy{From the assumption that $\\frac{a}{b}>1+2\\delta(m+n)$, we obtain$-a+(1-2\\alpha) b<0$}. Thus, the preceding term goes to zero verifying Assumption \\ref{assum:sequences}(a).\\\\\n(b) Let $k$ be an even number. Thus, $\\kappa=2$. From \\eqref{equ:seq} we have $\\mu_{k}=\\mu_{k+1}=\\frac{\\mu_02^b}{\\left(k+2\\right)^b}$. Now, let $k$ be an odd number. Again, according to \\eqref{equ:seq} can write \n\\[\\mu_{k+1}=\\frac{\\mu_02^b}{\\left((k+1)+2\\right)^b }<\\frac{\\mu_02^b}{\\left(k+1\\right)^b}= \\frac{\\mu_02^b}{\\left(k+\\kappa\\right)^b}=\\mu_k.\\]\nTherefore, $\\mu_k$ given by \\eqref{equ:seq} satisfies \\eqref{eqn:mu-k}. Also, from \\eqref{equ:seq} we have $\\mu_k \\to 0$. Thus, Assumption \\ref{assum:sequences}(b) holds.\\\\\n(c) The given rules \\eqref{equ:seq} imply that $\\g_k$ and $\\mu_k$ are both non-increasing sequences. Therefore, we have $\\g_k\\mu_{k} \\leq \\g_0\\mu_{0}$ for any $k\\geq 0$. So, to show that Assumption \\ref{assum:sequences}(c) holds, it is enough to show that $\\lambda_{\\min}\\g_0\\mu_{0} \\leq 1$ where $\\lambda_{\\min}$ is given by \\eqref{equ:valuesForAssumH_k}. Since we assumed that $\\g_0\\mu_0 \\leq L(m+n)$, for any $\\delta \\in (0,1]$, we have $\\g_0\\mu_0 \\leq (m+n)(L+\\mu_0^\\delta)$, implying that $\\lambda_{\\min}\\g_0\\mu_{0} \\leq 1$ and that Assumption \\ref{assum:sequences}(c) holds.\\\\\n(d) From \\eqref{equ:seq}, we can write $$\\sum_{k=0}^\\infty \\g_k\\mu_k =\\g_0\\mu_02^b\\sum_{k=0}^\\infty (k+1)^{-a}(k+\\kappa)^{-b} \\geq \\g_0\\mu_02^b\\sum_{k=0}^\\infty (k+2)^{-(a+b)}= \\infty,$$\nwhere the last relation is due to $a+b\\leq 1$. Therefore, Assumption \\ref{assum:sequences}(d) holds.\\\\\n(e) Using \\eqref{equ:seq}, it follows\n\\fy{\\begin{align*}&\\sum_{k=0}^\\infty \\g_k\\mu_k^2 =  \\g_0\\mu_0^24^b\\sum_{k=0}^\\infty (k+1)^{-a}(k+\\kappa)^{-2b}\\leq   \\g_0\\mu_0^24^b\\sum_{k=0}^\\infty (k+1)^{-(a+2b)}<\\infty,\\end{align*}}\nwhere the last inequality is due to \\an{$a+2b>1$}. Therefore, Assumption \\ref{assum:sequences}(e) holds.\\\\\n(f) From \\eqref{equ:seq}, we have\n\\begin{align*}&\\sum_{k=0}^\\infty \\g_k^2\\mu_k^{2\\alpha} %=  \\g_0^2(\\mu_02^b)^{2\\alpha}\\sum_{k=0}^\\infty \\frac{(k+\\kappa)^{-2\\alpha b}}{(k+1)^{2a}} \n\\leq  \\g_0^2(\\mu_02^b)^{2\\alpha}\\left(\\sum_{k=0}^1 \\frac{(k+\\kappa)^{-2\\alpha b}}{(k+1)^{2a}}+\\sum_{k=2}^\\infty \\frac{(2k)^{-2\\alpha b}}{k^{2a}}\\right)<\\infty \n%&=  \\frac{\\g_0^2}{\\mu_0^24^c}\\left(\\sum_{k=0}^1 \\frac{(k+\\kappa)^{2c}}{(k+1)^{2a}}+4^c\\sum_{k=2}^\\infty k^{2(c-a)}\\right)<\\infty,\n\\end{align*}\nwhere in the first inequality, we use $\\alpha <0$ and in the last inequality, we note that \n$a+\\alpha b=a-\\delta\\left(m+n\\right)b>0.5$. Therefore, Assumption \\ref{assum:sequences}(f) is verified.\n\\end{proof}\n\n\\begin{proof}\n\\fy{Throughout, we let $\\alpha$ denote $-\\delta(m+n)$.} \\fys{Assumption \\ref{assum:sequences-ms-convergence}(a, b, c) and \\eqref{eqn:mu-k} have been already shown in parts (a, b, c) of the proof of Lemma \\ref{lemma:a.s.sequences}.}\\\\\n\\noindent\n%(d) From part (a), since the term $\\g_k\\mu_k^{2\\alpha-1}$ goes to zero, there exist $K_2>0$ such that $\\g_k\\mu_k^{2\\alpha-1}\\leq \\frac{\\lambda_{\\min}}{\\lambda^2(L+\\mu_0)}$ for all $k \\geq K_2$, implying that Assumption \\ref{assum:sequences-ms-convergence}(d) holds for a large enough $k$. \\\\\n%\\noindent\n(d) It suffices to show there exists \\fy{$K_0$} such that for any $k \\geq K_0$ and $0<\\beta<1$, \n\\begin{align}\\label{ineq:partd}\\frac{\\g_{k-1}}{\\g_k}\\frac{\\mu_{k}^{1-2\\alpha}}{\\mu_{k-1}^{1-2\\alpha}}-1\\leq \\beta \\lambda_{\\min}\\g_k\\mu_k.\\end{align}\nFrom \\eqref{equ:seq} \\fy{and the definition of $\\alpha$}, we obtain \n\\begin{align*}&\\frac{\\g_{k-1}}{\\g_k}\\frac{\\mu_{k}^{1-2\\alpha}}{\\mu_{k-1}^{1-2\\alpha}}-1\n\\leq \\frac{\\g_{k-1}}{\\g_k}-1=\\left(1+\\frac{1}{k}\\right)^a-1= 1+\\frac{a}{k}+\\fy{{o}\\left(\\frac{1}{k}\\right)}-1=\\mathcal{O}\\left(\\frac{1}{k}\\right),\n\\end{align*}\nwhere the first inequality is implied due to $\\{\\mu_k\\}$ is non-increasing, and in the second equation, we used the Taylor's expansion of $\\left(1+\\frac{1}{k}\\right)^a$. Therefore, since the right-hand side of \\eqref{ineq:partd} is of the order $\\frac{1}{k^{a+b}}$ and that $a+b<1$, the preceding inequality shows that such $K_0$ exists for which Assumption \\ref{assum:sequences-ms-convergence}(d) holds for all $0<\\beta<1$.\\\\\n\\noindent\n(e) From \\eqref{equ:seq}, we have \n\\begin{align*} \\frac{\\mu_k^{2-2\\alpha}}{\\g_k}& =\\fy{\\g_0}^{-1}\\left(\\mu_02^b\\right)^{2-2\\alpha}(k+\\kappa)^{-b(2-2\\alpha)}(k+1)^{a} \\leq\\frac{\\fy{\\gamma_0}^{-1}\\left(\\mu_02^b\\right)^{2-2\\alpha}}{(k+1)^{-a+(2-2\\alpha)b}}\\\\&\\leq \\fy{\\gamma_0}^{-1}\\left(\\mu_02^b\\right)^{2-2\\alpha}=\\rho, \\end{align*}\nwhere the first inequality is due to \\fy{$\\alpha <0$, and the second inequality follows by the assumption $a\\leq 2b(1+\\delta(m+n))$}. Therefore, Assumption \\ref{assum:sequences-ms-convergence}(e) is satisfied.\n\\end{proof}","units":[{"start":28928,"end":29401,"start_line":478,"end_line":485,"raw_tex":"\\begin{property}[{Properties of a regularized function}]\\label{proper:propsfk} We have:\n\\begin{itemize}\n\\item [(a)] $f_k$ is strongly convex with a parameter $\\mu_k$.\n\\item [(b)] $f_k$ has Lipschitzian gradients with parameter $L+\\mu_k$.\n \\item [(c)] $f_k$ has a unique minimizer over $\\Real^n$, denoted by $x^*_k$. Moreover, for any $x \\in \\Real^n$, \n \\[2\\mu_k (f_k(x)-f_k(x^*_k)) \\leq \\|\\nabla f_k(x)\\|^2\\leq 2(L+\\mu_k) (f_k(x)-f_k(x^*_k)).\\]\n\\end{itemize}\n\\end{property}","sha256_utf8":"1fbe185d25b4d5fbe356eb3fd281d718c700559db02b4f8630e24eae31148b32"},{"start":29402,"end":29797,"start_line":486,"end_line":490,"raw_tex":"The existence and uniqueness of $x^*_k$ in Property~\\ref{proper:propsfk}(c) \nis due to the strong convexity of the function $f_k$ (see, for example, Section 1.3.2 in~\\cite{Polyak87}), \nwhile the relation for the gradient is known to hold for a \nstrongly convex function with a parameter $\\mu$ that also has Lipschitz gradients with a parameter \n$L$ (see Lemma 1 on page 23 in~\\cite{Polyak87}). \n","sha256_utf8":"610458a3c91fc08b7f81b12f898399eda233827ee7d8d873b4f7bbaa61f25acf"},{"start":31037,"end":31777,"start_line":511,"end_line":517,"raw_tex":"\\begin{lemma}[A recursive error bound]\\label{lemma:main-ineq}\nConsider the \\eqref{eqn:LM-cyclic-reg-BFGS} method and suppose Assumptions \\ref{assum:convex}, \\ref{assum:main}, and \\ref{assump:Hk} hold. Also, assume $\\mu_k$ is a non-increasing sequence and let \\begin{align}\\label{mainLemmaCond}\\g_k\\mu_k^{2\\alpha} \\leq \\frac{\\lambda_{\\min}}{\\lambda^2(L+\\mu_0)},\\quad \\hbox{for all }k\\geq 0.\n\\end{align}Then, the following inequality holds \\fy{for all $k\\geq 0$}:\n\\begin{align}\\label{ineq:cond-recursive-F-k}\n\\EXP{f_{k+1}(x_{k+1})\\mid\\sF_k}-f^* &\\leq (1-\\lambda_{\\min}\\mu_k\\g_k)(f_k(x_k)-f^*)+\\frac{\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)}{2}\\mu_k^2\\g_k\\notag \\\\ &+\\frac{ (L+\\mu_k)\\lambda^2\\nu^2}{2}\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align}\n\\end{lemma}","sha256_utf8":"34e885e66031037195a15dcd1b436bcdef0e56b1c74ca223f150fd4454852e62"},{"start":31778,"end":36955,"start_line":518,"end_line":567,"raw_tex":"\\begin{proof}\nThe Lipschitzian property of $\\nabla f_k$ and  \nthe update rule \\eqref{eqn:LM-cyclic-reg-BFGS} imply that \n\\begin{align*}\nf_k(x_{k+1}) &\\leq f_k(x_k)+\\nabla f_k(x_k)^T(x_{k+1}-x_k)+\\frac{ (L+\\mu_k)}{2}\\|x_{k+1}-x_k\\|^2 \\cr\n&= f_k(x_k)-\\g_k\\nabla f_k(x_k)^TH_k\\left(\\nabla F(x_k,\\xi_k) +\\mu_k(x_k-x_0)\\right)\\\\&+ \\frac{ (L+\\mu_k)}{2}\\g_k^2\\|H_k\\left(\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)\\right)\\|^2.\\end{align*}\n\\fy{Invoking the definition of the stochastic error $w_k$ in} \\eqref{def:wk} \\fy{and} Definition~\\ref{def:regularizedF}, we obtain\n\\begin{align}\\label{ineq:term1-2}\nf_k(x_{k+1}) &\\leq f_k(x_k)-\\g_k\\nabla f_k(x_k)^TH_k(\\nabla f(x_k)+w_k +\\mu_k(x_k-x_0))\\notag\\\\&+ \\frac{ (L+\\mu_k)}{2}\\g_k^2\\|H_k(\\nabla f(x_k)+w_k+\\mu_k(x_k-x_0))\\|^2\\\\ \n\\notag = f_k(x_k)&-\\g_k\\underbrace{\\nabla f_k(x_k)^TH_k(\\nabla f_k(x_k)+w_k)}_{\\tiny\\hbox{Term } 1}+ \\frac{ (L+\\mu_k)}{2}\\g_k^2\\underbrace{\\|H_k(\\nabla f_k(x_k)+w_k)\\|^2}_{\\tiny\\hbox{ Term } 2},\\end{align}\nwhere in the last \\fy{equation,} we used the definition of $f_k$. Next, we estimate the conditional expectation of Term 1 and 2. From Assumption \\ref{assump:Hk}, we have \n\\begin{align*}\n\\hbox{Term }1 &= \\nabla f_k(x_k)^TH_k\\nabla f_k(x_k)+\\nabla f_k(x_k)^TH_kw_k \\\\&\\geq \\lambda_{\\min}\\|\\nabla f_k(x_k)\\|^2+\\nabla f_k(x_k)^TH_kw_k.\n\\end{align*}\nTaking expectations conditioned on $\\sF_k$, from the preceding inequality, we obtain\n\\begin{align}\\label{equ:Term1}\n\\EXP{\\hbox{Term } 1\\mid\\sF_k} &\\geq \\lambda_{\\min}\\|\\nabla f_k(x_k)\\|^2+\\EXP{\\nabla f_k(x_k)^TH_kw_k\\mid\\sF_k}\\\\ \\notag&=\\lambda_{\\min}\\|\\nabla f_k(x_k)\\|^2+\\nabla f_k(x_k)^TH_k\\EXP{w_k\\mid\\sF_k} =\\lambda_{\\min}\\|\\nabla f_k(x_k)\\|^2,\\end{align}\nwhere we recall that $\\EXP{w_k\\mid\\sF_k}=0$ and $\\EXP{H_k\\mid\\sF_k}=H_k$. Similarly, in Term 2, invoking Assumption \\ref{assump:Hk}(b), we may write\n\\begin{align*}\n\\hbox{Term } 2&= (\\nabla f_k(x_k)+w_k)^TH_k^2(\\nabla f_k(x_k)+w_k)\\leq (\\lambda\\mu^{\\alpha}_k)^2\\|\\nabla f_k(x_k)+w_k\\|^2 \\\\ &=\\lambda^2\\mu^{2\\alpha}_k\\left(\\|\\nabla f_k(x_k)\\|^2+\\|w_k\\|^2+2\\nabla f_k(x_k)^Tw_k\\right).\\end{align*}\nTaking conditional expectations from the preceding inequality, and using Assumption \\ref{assum:main}, we obtain\n\\begin{align}\\label{equ:Term2}\n\\EXP{\\hbox{Term } 2\\mid\\sF_k}&\\leq\\lambda^2\\mu^{2\\alpha}_k\\left(\\|\\nabla f_k(x_k)\\|^2+\\EXP{\\|w_k\\|^2\\mid\\sF_k}+2\\nabla f_k(x_k)^T\\EXP{w_k\\mid\\sF_k}\\right)\\notag\\\\ & \\leq \\lambda^2\\mu^{2\\alpha}_k\\left(\\|\\nabla f_k(x_k)\\|^2+\\nu^2\\right).\\end{align}\nNext, taking conditional expectations in \\eqref{ineq:term1-2}, and using \\eqref{equ:Term1} and \\eqref{equ:Term2}, we obtain\n\\begin{align*}%\\label{ineq:lemma2rel7}\n&\\EXP{f_k(x_{k+1})\\mid\\sF_k} \\leq f_k(x_k)-\\g_k\\lambda_{\\min}\\|\\nabla f_k(x_k)\\|^2\\notag \\\\\n&+\\lambda^2\\mu^{2\\alpha}_k\\frac{ (L+\\mu_k)}{2}\\g_k^2\\left(\\|\\nabla f_k(x_k)\\|^2+\\nu^2\\right)\\notag \\\\\n&\\leq f_k(x_k)-\\frac{\\g_k\\lambda_{\\min}}{2}\\|\\nabla f_k(x_k)\\|^2\\left(2-\\frac{\\lambda^2\\mu_k^{2\\alpha}\\g_k(L+\\mu_k)}{\\lambda_{\\min}}\\right)+\\lambda^2\\mu^{2\\alpha}_k\\frac{ (L+\\mu_k)}{2}\\g_k^2\\nu^2.\n\\end{align*}\nFrom the assumption that $\\g_k$ and $\\mu_k$ satisfy $\\g_k\\mu^{2\\alpha}_k \\leq \\frac{\\lambda_{\\min}}{\\lambda^2(L+\\mu_0)}$ for any $k \\geq 0$ and that $\\mu_k$ is non-increasing, we have $\\g_k\\mu^{2\\alpha}_k \\leq \\frac{\\lambda_{\\min}}{\\lambda^2(L+\\mu_k)}$. As a consequence, we get $2-\\frac{\\lambda^2\\mu_k^{2\\alpha}\\g_k(L+\\mu_k)}{\\lambda_{\\min}}\\geq 1$.  Therefore, from \\fy{the} preceding inequality, we obtain\n\\begin{align*}\n\\EXP{f_k(x_{k+1}) \\mid\\sF_k}&\\leq f_k(x_k)-\\frac{\\g_k\\lambda_{\\min}}{2}\\|\\nabla f_k(x_k)\\|^2+\\lambda^2\\mu^{2\\alpha}_k\\frac{ (L+\\mu_k)}{2}\\g_k^2\\nu^2.\n\\end{align*}\nEmploying Property \\ref{proper:propsfk}(c), we have\n\\begin{align*}\n\\EXP{f_k(x_{k+1})\\mid\\sF_k} &\\leq f_k(x_k)-\\lambda_{\\min}\\mu_k\\g_k(f_k(x_k)-f_k(x^*_k))+\\lambda^2\\mu^{2\\alpha}_k\\frac{ (L+\\mu_k)}{2}\\g_k^2\\nu^2.\n\\end{align*}\nNote that, since $\\mu_k$ is a non-increasing sequence, Definition \\ref{def:regularizedF} implies that \\[\\EXP{f_{k+1}(x_{k+1})\\mid \\sF_k} \\leq\\EXP{f_{k}(x_{k+1})\\mid \\sF_k}.\\] Therefore, we obtain \n\\begin{align}\\label{ineq:lemmaLastIneq}\n\\EXP{f_{k+1}(x_{k+1})\\mid\\sF_k} &\\leq f_k(x_k)-\\lambda_{\\min}\\mu_k\\g_k(\\underbrace{f_k(x_k)-f_k(x^*_k)}_{\\tiny{\\hbox{Term } 3}})+\\lambda^2\\mu^{2\\alpha}_k\\frac{ (L+\\mu_k)}{2}\\g_k^2\\nu^2.\n\\end{align}\nNext, we derive a lower bound for Term 3. Since $x_k^*$ is the unique minimizer of $f_k$, we have $f_k(x_k^*) \\leq f_k(x^*)$. Therefore, invoking Definition \\ref{def:regularizedF}, for an arbitrary optimal solution $x^* \\in X^*$, we have \n\\[f_k(x_k)-f_k(x^*_k)\\geq f_k(x_k)-f_k(x^*)=f_k(x_k)-f^*-\\frac{\\mu_k}{2}\\|x^*-x_0\\|^2.\\]\nFrom the preceding relation and \\eqref{ineq:lemmaLastIneq}, we have\n\\begin{align*}\n\\EXP{f_{k+1}(x_{k+1})\\mid\\sF_k} &\\leq f_k(x_k)-\\lambda_{\\min}\\mu_k\\g_k(f_k(x_k)-f^*)+\\frac{\\lambda_{\\min}\\|x^*-x_0\\|^2}{2}\\mu_k^2\\g_k\\\\\n&+\\frac{(L+\\mu_k)\\lambda^2\\nu^2}{2}\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align*}\nSince $x^*$ is an arbitrary optimal solution, taking minimum from the right-hand side of the preceding inequality over $X^*$, we can replace $\\|x^*-x_0\\|$ by $\\mbox{dist}(x_0,X^*)$. Then, subtracting $f^*$ from both sides of the resulting relation yields the desired inequality.\n\\end{proof}","sha256_utf8":"f878941038b56b4603488a649a12acd7eac1b9bfddc2b50ca28e4a1ffe336f8b"},{"start":42845,"end":43605,"start_line":639,"end_line":649,"raw_tex":"\\begin{proposition}[Convergence in mean]\\label{prop:mean}\nConsider the \\eqref{eqn:LM-cyclic-reg-BFGS} scheme. \nSuppose Assumptions~\\ref{assum:convex},\n\\ref{assum:main}, \\ref{assump:Hk}, and \\ref{assum:sequences-ms-convergence} hold. \nThen, there exists $K\\geq 1$ such that:\\begin{align}\\label{ineq:bound}\n \\EXP{f(x_{k+1})}-f^*\\leq \\theta\\g_k\\mu_k^{2\\alpha-1}, \\quad \\hbox{for all } k \\geq K,\n \\end{align}\n  where $f^*$ is the optimal value of problem \\fy{and}\n\\begin{align}\\label{def:theta}\\theta := \\max \\bigg\\{\\frac{\\EXP{f_{K+1}(x_{K+1})}-f^*}{\\g_K\\mu_K^{2\\alpha-1}},\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2\\fy{\\lambda_{\\min}}(1-\\beta)}\\bigg\\}.\\end{align}\n Moreover, $\\lim_{k\\to \\infty}\\EXP{f(x_{k})}=f^*.$\n\\end{proposition}","sha256_utf8":"6f219aa48bd2fd4d609c5618b830760c2f1fa0562fa557ae489d87398faa972c"},{"start":43606,"end":48107,"start_line":650,"end_line":689,"raw_tex":"\\begin{proof}\nNote that Assumption \\ref{assum:sequences-ms-convergence}(a,b) imply that \\eqref{ineq:cond-recursive-F-k} holds for a large enough $k$, say after $\\hat K$. Then, since the conditions of Lemma \\ref{lemma:main-ineq} are met \\fy{(with an index shift)}, taking expectations on both sides of \\eqref{ineq:cond-recursive-F-k}, we obtain for any $k\\geq \\hat K$:\n\\begin{align*}\n\\EXP{f_{k+1}(x_{k+1})-f^*} &\\leq (1-\\lambda_{\\min}\\mu_k\\g_k)\\EXP{f_k(x_k)-f^*}+\\frac{\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)}{2}\\mu_k^2\\g_k\\\\ &+\\frac{ (L+\\mu_0)\\lambda^2\\nu^2}{2}\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align*}\n%{\\bf Comment: the relation in (9) has different coefficients than originally - edits needed from here onward}\n Using Assumption \\ref{assum:sequences-ms-convergence}(e), we have $\\mu_k^2\\g_k\\leq \\rho \\g_k^2\\mu_k^{2\\alpha}$. Thus, we obtain\n\\begin{align}\\label{ineq:cond-recursive-F-k-expected2}\n\\EXP{f_{k+1}(x_{k+1})-f^*} &\\leq (1-\\lambda_{\\min}\\mu_k\\g_k)\\EXP{f_k(x_k)-f^*}\\notag\\\\ &+\\left(\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2}\\right)\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align}\nLet us define \\fy{$K\\triangleq\\max\\{\\hat K, K_0\\}$, where $K_0$ is from Assumption \\ref{assum:sequences-ms-convergence}(d)}. Using the preceding relation and by induction on $k$, we show the desired result. To show \\eqref{ineq:bound}, we show the following relation first: \n\\begin{align}\\label{ineq:bound_v2}\n \\EXP{f_{k+1}(x_{k+1})}-f^*\\leq \\theta\\g_k\\mu_k^{2\\alpha-1}, \\quad \\hbox{for all } k \\geq K,\n \\end{align}\n Note that \\eqref{ineq:bound_v2} \\fy{implies} the relation \\eqref{ineq:bound} since we have $ \\EXP{f(x_{k+1})}\\leq  \\EXP{f_{k+1}(x_{k+1})}$.\nFirst, we show that \\eqref{ineq:bound_v2} holds for \\fy{$k=K$}.  Consider the term $\\EXP{f_{K+1}(x_{K+1})}-f^*$. Multiplying and dividing by $\\g_K\\mu_K^{2\\alpha-1}$, we obtain\n\\begin{align*}\\EXP{f_{K+1}(x_{K+1})}-f^*=\\left(\\frac{\\EXP{f_{K+1}(x_{K+1})}-f^*}{\\g_k\\mu_K^{2\\alpha-1}}\\right)\\g_K\\mu_K^{2\\alpha-1}\n \\leq \\theta \\g_K\\mu_K^{2\\alpha-1},\\end{align*}\nwhere the last inequality is obtained by invoking the definition of $\\theta$ in \\eqref{def:theta}. This implies that \\eqref{ineq:bound_v2} holds for $k=K$. Now assume that  \\eqref{ineq:bound_v2} holds for some $k \\geq K$. We show that it also holds for $k+1$.  From the induction hypothesis and \\eqref{ineq:cond-recursive-F-k-expected2} we have\n\\begin{align*}\n\\EXP{f_{k+1}(x_{k+1})-f^*} &\\leq (1-\\lambda_{\\min}\\mu_k\\g_k)\\theta\\g_{k-1}\\mu_{k-1}^{2\\alpha-1}\\\\ &+\\left(\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2}\\right)\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align*}\nUsing Assumption \\ref{assum:sequences-ms-convergence}(d) we obtain\n\\begin{align}\\label{ineq:thm2-rel1}\n\\EXP{f_{k+1}(x_{k+1})-f^*} &\\leq \\theta\\g_{k}\\mu_{k}^{2\\alpha-1}(1-\\lambda_{\\min}\\mu_k\\g_k)(1+\\beta \\lambda_{\\min}\\g_{k}\\mu_{k})\\notag \\\\ &+\\left(\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2}\\right)\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align}\nNext we find an upper bound for the term $(1-\\lambda_{\\min}\\mu_k\\g_k)(1+\\beta \\lambda_{\\min}\\g_{k}\\mu_{k}) $ as follows\n\\begin{align*}\n(1-\\lambda_{\\min}\\mu_k\\g_k)(1+\\beta \\lambda_{\\min}\\g_{k}\\mu_{k}) &= 1-\\lambda_{\\min}\\mu_k\\g_k+\\beta\\lambda_{\\min}\\mu_k\\g_k-\\fy{\\beta\\lambda_{\\min}^2\\mu_k^2\\g_k^2}\\\\ & \\leq 1-(1-\\beta)\\lambda_{\\min}\\mu_k\\g_k.\n\\end{align*}\nCombining this relation with \\eqref{ineq:thm2-rel1}, it follows \\fy{that}\n\\begin{align*}%\\label{ineq:thm2-rel1}\n&\\EXP{f_{k+1}(x_{k+1})-f^*} \\leq \\theta\\g_{k}\\mu_{k}^{2\\alpha-1} -\\theta\\fy{\\lambda_{\\min}}(1-\\beta)\\mu_k^{2\\alpha}\\g_k^2  \\\\ &+\\left(\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2}\\right)\\mu^{2\\alpha}_k\\g_k^2\\\\ & =\\theta\\g_{k}\\mu_{k}^{2\\alpha-1}-\\left(\\underbrace{\\theta\\fy{\\lambda_{min}}(1-\\beta)-\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2}}_{\\tiny{\\hbox{Term }1}}\\right)\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align*}\nNote that the definition of $\\theta$ in \\eqref{def:theta} implies that Term 1 is nonnegative. Therefore, \n\\begin{align*}%\\label{ineq:thm2-rel1}\n\\EXP{f_{k+1}(x_{k+1})-f^*} &\\leq \\theta\\g_{k}\\mu_{k}^{2\\alpha-1}.\n\\end{align*}\nHence, the induction statement holds for $k+1$. We conclude that \\eqref{ineq:bound_v2} holds for all $k\\geq K$. As a consequence, \\eqref{ineq:bound} holds for all $k\\geq K$ as well. To complete the proof, we need to show $\\lim_{k\\to \\infty}\\EXP{f(x_{k})}=f^*$. This is an immediate result of \\eqref{ineq:bound} and Assumption \\ref{assum:sequences-ms-convergence}(a).\n\\end{proof}","sha256_utf8":"89b0046f05ad03b003918e0197f402a01198f6496cabb09fcf37e1bd755cbe44"},{"start":67189,"end":67435,"start_line":937,"end_line":940,"raw_tex":"\\begin{lemma}\\label{sumProductBounds}\nLet $0 < a_1 \\leq a_2 \\leq \\ldots \\leq a_n$, and $P$ and $S$ be positive scalars such that \\fys{$\\sum_{i=1}^n a_i \\leq S$ and $\\prod_{i=1}^na_i \\geq P$. Then, we have \n$a_1 \\geq (n-1)!P/S^{n-1}$.}\n\\end{lemma}","sha256_utf8":"2076c2aeef627d9f2fd63175f1a64814dc1291fa2daafa82a6f1d4c292430eed"},{"start":67437,"end":67503,"start_line":941,"end_line":943,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{app:sumProductBounds}.\n\\end{proof}","sha256_utf8":"ea516e9471b72bdc5e78ce0605c368f9f8ab34c8e88fa06daedbaca7385f6752"},{"start":67505,"end":68988,"start_line":944,"end_line":956,"raw_tex":"\\begin{lemma}[Properties of  update rule \\eqref{eqn:H-k}-\\eqref{eqn:H-k-m}]\\label{LBFGS-matrix}\nConsider the \\eqref{eqn:LM-cyclic-reg-BFGS} method. Let $H_k$ be given by the update rule \\eqref{eqn:H-k}-\\eqref{eqn:H-k-m}, where $s_i$ and $y_i$ are defined in \\eqref{equ:siyi-LBFGS} and $\\mu_k$ is updated according to the procedure \\eqref{eqn:mu-k}. Let \\fy{Assumption \\ref{assum:convex2}(a,b)} hold. Then, the following results hold:\n\\begin{itemize}\n\\item [(a)]  For any odd \\fy{$k \\geq 2m-1$}, the \\fy{curvature condition} holds, i.e., \\fy{$s_{\\lceil k/2\\rceil}^T{y_{\\lceil k/2\\rceil}} >0$}.\n\\item [(b)]  For any odd \\fy{$k \\geq 2m-1$}, the secant equation holds, i.e., \\fy{$H_{k}{y}_{\\lceil k/2\\rceil}=s_{\\lceil k/2\\rceil}$}. \n\\item [(c)] For any \\fy{$k \\geq 2m-1$}, $H_k$ satisfies Assumption \\ref{assump:Hk} \\fy{with} the following values:\n\\begin{align}\\label{equ:valuesForAssumH_k}&\\lambda_{\\min}=\\frac{1}{(m+n)\\left(L+\\fys{\\tau}\\mu_0^\\delta\\right)}, \\quad \\fy{\\lambda= \\frac{(m+n)^{n+m-1}\\left(L+\\fys{\\tau}\\mu_0^\\delta\\right)^{n+m-1}}{(n-1)!\\fys{\\tau^{(n+m)}}}}, \\notag\\\\ & \\fy{\\hbox{and }\\alpha=-\\delta(n+m).}\n\\end{align}\nMore precisely, $H_k$ is symmetric, $\\EXP{H_k\\mid\\sF_k}=H_k$ and\n\\begin{align}\\label{proof:H_kbounds} \\frac{1}{(m+n)\\left(L+\\fys{\\tau}\\mu_0^\\delta\\right)}\\mathbf{I} \\preceq H_{k} \\preceq \\frac{(m+n)^{n+m-1}\\left(L+\\fys{\\tau}\\mu_0^\\delta\\right)^{n+m-1}}{(n-1)!\\fys{\\left(\\tau\\mu_k^\\delta\\right)^{(n+m)}}}\\ \\mathbf{I}.\n\\end{align}\n\\end{itemize}\n\\end{lemma}","sha256_utf8":"986576249719ddb3b3baa02b594dee057ccafe610c927f4d6775159f6a2fc21b"},{"start":68993,"end":69059,"start_line":957,"end_line":959,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{app:lemmaLBFGSmatrix}.\n\\end{proof}","sha256_utf8":"2ced26ab106406954dfdbc677eac8333835297eb18538022586d7342349cfb2b"},{"start":70137,"end":70990,"start_line":965,"end_line":973,"raw_tex":"\\begin{lemma}[Feasible tuning sequences for a.s. convergence \\fy{(Proposition \\ref{prop:a.s})}]\\label{lemma:a.s.sequences}\nLet the sequences $\\g_k$ and $\\mu_k$ be given by the following rules:\n\\begin{align}\\label{equ:seq}\n \\g_k=\\frac{\\g_0}{(k+1)^a}, \\quad \\mu_{k}=\\frac{\\fy{2^b\\mu_0}}{\\left(k+\\kappa\\right)^b},\n\\end{align}\nwhere $\\kappa=2$ if $k$ is even and $\\kappa=1$ otherwise, $\\g_0$and $\\mu_0$ are positive scalars such that $\\g_0\\mu_0 \\leq L(m+n)$, \\fy{and $a$, $b$, and $\\delta\\leq 1$ are positive scalars \\fys{satisfying}:\n\\begin{align*}& \\fy{\\frac{a}{b}>1+2\\delta(n+m)}, \\quad a+b \\leq 1,\\quad a+2b > 1, \\quad\\hbox{and}\\quad a-\\delta b(m+n)>0.5.\\end{align*}\nThen, $\\g_k$ and $\\mu_k$ satisfy Assumption \\ref{assum:sequences} with $\\lambda_{\\min}$ and $\\alpha$ in \\eqref{equ:valuesForAssumH_k} and $\\mu_k$ satisfies \\eqref{eqn:mu-k}.}\n\\end{lemma}","sha256_utf8":"2e737e57e6720005e831c6b3f5bbf9f6034564a0dd213f2ff637616ed2a1dce9"},{"start":70995,"end":71062,"start_line":974,"end_line":976,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{app:feasibleSeqASconv}.\n\\end{proof}","sha256_utf8":"688a220118168d4de3de1bb0920829aabc7c68e439ca510ec849f73f26592065"},{"start":71334,"end":72130,"start_line":983,"end_line":993,"raw_tex":"\\begin{lemma}[Feasible tuning sequences for convergence in mean \\fy{(Proposition \\ref{prop:mean})}]\\label{lemma:mean-sequences}\nLet the sequences $\\g_k$ and $\\mu_k$ be given by \\eqref{equ:seq},\nwhere $\\g_0$ and $\\mu_0$ are positive scalars such that $\\g_0\\mu_0 \\leq L(m+n)$. \\fy{Let, $0 <\\delta \\leq 1$ $a>0$ and $b>0$ \\fys{satisfying}:\n\\begin{align*}& \\frac{a}{b}>1+2\\delta\\left(m+n\\right), \\quad a+b < 1,\n\\quad \\frac{a}{b} \\leq\n2\\left(1+\\delta\\left(m+n\\right)\\right).\\end{align*}Then, \\vus{$\\mu_k$\nsatisfies \\eqref{eqn:mu-k}} and $\\g_k$ and $\\mu_k$ satisfy Assumption\n\\ref{assum:sequences-ms-convergence} with any arbitrary $0<\\beta<1$,\n$\\rho\\triangleq \\g_0^{-1}\\left(\\mu_02^b\\right)^{2+2\\delta(m+n)}$,\nand with $\\lambda_{\\min}$ and $\\alpha$ given by\n\\eqref{equ:valuesForAssumH_k}.} \\end{lemma}","sha256_utf8":"fc4dd64d82e93933630fa82dbdfec0f671be6e57b3a92f84d688dabbb199eea7"},{"start":72135,"end":72204,"start_line":994,"end_line":996,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{app:feasibleSeqMeanConv}.\n\\end{proof}","sha256_utf8":"dd02dee6658ed8b898db306cec12e03b3adaa35417bc42bb70eda9f2c8dd90ee"},{"start":77886,"end":84680,"start_line":1080,"end_line":1146,"raw_tex":"\\begin{proof}\n\\noindent (a) First, we show that the conditions of Proposition~\\ref{prop:mean} are satisfied. Assumption \\ref{assum:convex} holds as a consequence of Assumption \\ref{assum:convex2}. From Lemma \\ref{LBFGS-matrix}(c), Assumption \\ref{assump:Hk} holds for any \\fy{$k \\geq 2m-1$} as well. To show that Assumption \\ref{assum:sequences-ms-convergence} holds, we apply Lemma \\ref{lemma:mean-sequences}. We have \n\\[\\frac{a}{b}=\\frac{\\frac{2}{3}-\\e+\\frac{2\\delta(n+m)}{3}}{1/3}=2-3\\e+2\\delta(n+m)>1+2\\delta\\left(m+n\\right),\\]\nwhere we used $\\e<\\frac{1}{3}$. Moreover, since $\\delta < \\frac{1.5\\e}{n+m}$, we have $a+b=1-\\e +\\frac{2\\delta(n+m)}{3}<1$. Also, from the values of $a$ and $b$ we have $2b\\left(1+\\delta\\left(m+n\\right)\\right)=a+\\e>a.$\nThus, the conditions of Lemma \\ref{lemma:mean-sequences} hold. This implies that there exists $K_0>0$ such that for any $k\\geq K_0$,  the sequences $\\g_k$ and $\\mu_k$ satisfy Assumption \\ref{assum:sequences-ms-convergence} with any arbitrary $0<\\beta<1$ and for $\\rho=\\g_0^{-1}\\left(\\mu_02^b\\right)^{2+2\\delta(m+n)}$, and with $\\lambda_{\\min}$, $\\lambda$ and $\\alpha$ given by \\eqref{equ:valuesForAssumH_k}. Let us define $K:=\\max\\{K_0,2m-1\\}$. Since all conditions in Proposition~\\ref{prop:mean} are met, \\an{from \\eqref{ineq:bound}, \\eqref{equ:seq}, and by substituting values of $a$, $b$, and $\\alpha$, for any $k \\geq K$ we obtain}\n\\begin{align*}\\EXP{f(x_{k+1})}-f^*&\\leq \\theta\\g_{k-1}\\mu_{k-1}^{2\\alpha-1}= \\frac{\\theta\\g_0(k+\\kappa-1)^{(1-2\\alpha)/3}}{\\left(\\mu_0\\sqrt[3]{2}\\right)^{1-2\\alpha}k^{2/3-\\e-\\frac{2}{3}\\alpha}}\\\\ &\\leq  \\frac{\\theta\\g_0(k+1)^{(1-2\\alpha)/3}}{\\left(\\mu_0\\sqrt[3]{2}\\right)^{1-2\\alpha}(k+1)^{2/3-\\e-\\frac{2}{3}\\alpha}}=\\left(\\frac{\\theta\\g_0}{\\left(\\mu_0\\sqrt[3]{2}\\right)^{1-2\\alpha}}\\right)\\frac{1}{(k+1)^{\\frac{1}{3}-\\e}}.\\end{align*}\nThrough a change of variable from $k+1$ to $k$, we conclude the result. \\\\\n\n\\fy{\\noindent (b)  To show \\eqref{eqn:r_Lemma}, it suffices to show that\n \\begin{align*}\n  r=\\begin{cases}\n    H_{k,m}\\left(\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)\\right),  &    \\text{if } k \\text{ is odd}, \\\\\n    H_{k-1,m}\\left(\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)\\right),  &   \\text{otherwise},\n  \\end{cases}\n\\end{align*}\nwhere $H_{k,m}$ is defined by the recursion \\eqref{eqn:H-k-m} for an odd $k$. First, consider the case that $k\\geq 2m-1$ is an odd number. As such, at the $k$th iteration, from line $\\#$7, we have $i:=\\lceil k/2 \\rceil$. For clarity of the presentation, throughout \\vus{this} proof, we use $K$ (instead of $i$), i.e., $K\\triangleq \\lceil k/2 \\rceil$ \\vus{and} $q_{K-t+1}$ \\vus{is used to} denote the value of the vector $q \\in \\mathbb{R}^n$ after being updated at iteration $t$ in line $\\#$21. Similarly, we use $r_{t-K+m}$ to denote the value of the vector $r \\in \\mathbb{R}^n$ after being updated at iteration $t$ in line $\\#$25. Also, we use the following definitions:\n\\begin{align*}\n& q_0\\triangleq \\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0), \\quad r_0\\triangleq H_{k,0}q_{m},\\cr \n&\\rho_j\\triangleq \\frac{1}{y_j^Ts_j}, \\hbox{ and }V_j\\triangleq \\mathbf{I}-\\rho_{j}y_js_j^T, \\quad \\hbox{for all } j=K-(m-1),\\ldots, K.\n\\end{align*}\nConsider relation \\eqref{eqn:H-k-m}. By applying this recursive relation repeatedly, we obtain\n\\begin{align}\\label{equ:H_kClosedForm}\nH_{k,m} &=\\left(\\prod_{j=1}^{m}V_{K-(m-j)}\\right)^TH_{k,0}\\left(\\prod_{j=1}^{m}V_{K-(m-j)}\\right)\\\\\n&+ \\rho_{K-m+1}\\left(\\prod_{j=2}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+1}s^T_{K-m+1}\\left(\\prod_{j=2}^{m}V_{K-(m-j)}\\right) \\notag\\\\\n& + \\rho_{K-m+2}\\left(\\prod_{j=3}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+2}s^T_{K-m+2}\\left(\\prod_{j=3}^{m}V_{K-(m-j)}\\right)\\notag\\\\\n& + \\ldots\\notag\\\\\n& +\\rho_{K-1}V^T_{K}s_{K-1}s_{K-1}^TV_{K}\\notag\\\\\n&+\\rho_{K}s_{K}s_{K}^T.\\notag\n\\end{align}\nNext, we derive a formula for $q_t$. From lines $\\#$20-21 in the algorithm, we have\n\\begin{align*}\nq_{K-t+1}&=q_{K-t} -\\alpha_{K-t+1}y_t =q_{K-t} - \\rho_t\\left(s_t^Tq_{K-t}\\right)y_t = q_{K-t} - \\rho_t\\left(y_ts_t^T\\right)q_{K-t} \\\\\n&= \\left( \\mathbf{I}-\\rho_{t}y_ts_t^T\\right)q_{K-t}=V_tq_{K-t}, \\quad \\hbox{for all } t= K, K-1, \\ldots, K-m+1.\n\\end{align*}\nFrom the preceding relation, we obtain\n\\begin{align}\\label{equ:q_ell}\nq_{\\ell}=\\left(\\prod_{j=m-\\ell+1}^{m}V_{K-(m-j)}\\right)q_0, \\quad \\hbox{for all } \\ell= 1,2, \\ldots, m.\n\\end{align}\nFrom the update rule for $\\alpha_{i-t+1}$ in line $\\#$20, using the definition of $\\rho_t$, and applying the previous relation, we have $\\alpha_1=\\rho_{K}s_K^Tq_0$ and \n\\begin{align}\\label{equ:alpha_ell}\n\\alpha_{\\ell}=\\rho_{K-\\ell+1}s_{K-\\ell+1}^T\\left(\\prod_{j=m-\\ell+2}^{m}V_{K-(m-j)}\\right)q_0, \\quad \\hbox{for all } \\ell= 2,3, \\ldots, m.\n\\end{align}\n\\fys{Multiplying both sides of} \\eqref{equ:H_kClosedForm} by $q_0$ and employing \\eqref{equ:q_ell} and \\eqref{equ:alpha_ell}, we obtain\n\\begin{align}\\label{equ:H_kq_0}\nH_{k,m}q_0 &=\\left(\\prod_{j=1}^{m}V_{K-(m-j)}\\right)^TH_{k,0}q_m + \\left(\\prod_{j=2}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+1}\\alpha_m \\\\\n& + \\left(\\prod_{j=3}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+2}\\alpha_{m-1}  +\\ldots+V^T_{K}s_{K-1}\\alpha_2+s_{K}\\alpha_1.\\notag\n\\end{align}\nNext, we derive a formula for $r_t$. From line $\\#$25 in the algorithm, we have \n\\begin{align*}\nr_{t-K+m}&=r_{t-K+m-1}+\\left(\\alpha_{K-t+1}-\\rho_ty_t^Tr_{t-K+m-1}\\right)s_t \\\\\n&= r_{t-K+m-1}-\\rho_ts_t y_t^Tr_{t-K+m-1}+\\alpha_{K-t+1}s_t\\\\\n& = V_t^Tr_{t-K+m-1}+\\alpha_{K-t+1}s_t, \\quad \\hbox{for all } t= K-m+1,\\ldots,K-1,  K.\n\\end{align*}\nCombining the preceding two relations, we obtain\n\\begin{align*}\nr_{\\ell}&= V_{K-(m-\\ell)}^Tr_{\\ell-1}+\\alpha_{m-\\ell+1}s_{K-(m-\\ell)}, \\quad \\hbox{for all } \\ell= 1,2,\\ldots,m.\n\\end{align*}\nUsing the preceding equation repeatedly, we obtain\n\\begin{align}\\label{equ:r_m}\nr_m &=\\left(\\prod_{j=1}^{m}V_{K-(m-j)}\\right)^Tr_0 +\\alpha_m \\left(\\prod_{j=2}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+1} \\\\\n& + \\alpha_{m-1} \\left(\\prod_{j=3}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+2} +\\ldots+\\alpha_2V^T_{K}s_{K-1}+\\alpha_1s_{K}.\\notag\n\\end{align}\nFrom \\eqref{equ:r_m} and \\eqref{equ:H_kq_0}, and the definition of  $r_0$, we obtain $r_m=H_{k,m}q_0$. Taking to account the definition of $q_0$, the desired result holds for any odd $k\\geq 2m-1$. Now, consider the case where $k\\geq 2m-1$ is an even number. This implies that the ``if'' condition in line $\\#$6 is skipped and as such, the value of $i$ is not updated from the iteration $k-1$, i.e.,  $i= \\lceil (k-1)/2 \\rceil$. Consequently, the LBFGS two-loop recursion at an even $k$ uses the following pairs \n$$\\left\\{(s_\\ell,y_\\ell)\\mid \\ell=\\lceil (k-1)/2\\rceil-m+1, \\lceil (k-1)/2\\rceil-m+2,\\ldots,\\lceil (k-1)/2\\rceil\\right\\}.$$\n\\fys{Now,} considering the definition \\eqref{eqn:H-k-m} for $k-1$, the desired relation can be shown following the same steps discussed for the case the iteration number is odd.}\n%This implies the desired rate statement holds. \n\\end{proof}","sha256_utf8":"ed7c35d155b86fdf8b53375ac6be1fdbd0e365e9b6e4606677a6c8cad58aa6bb"},{"start":114718,"end":115171,"start_line":1472,"end_line":1478,"raw_tex":"\\subsection{Proof of Lemma \\ref{sumProductBounds}}\\label{app:sumProductBounds}\nFrom $0<a_1 \\leq \\ldots \\leq a_n$, we can write \n\\begin{align*}\n(n-(i-1))a_i \\leq \\sum_{j=1}^na_j, \\qquad \\hbox{for all } i \\in \\{1,\\ldots,n\\}.\n\\end{align*}\nInvoking $\\sum_{i=1}^n{a_i} \\leq S$, we obtain $a_i \\leq \\frac{S}{n-(i-1)}$, for all $i \\in \\{1,\\ldots,n\\}$. From the preceding relation and that $\\prod_{j=1}^na_j \\geq P$, we can obtain $a_1 \\geq (n-1)!P/S^{n-1}.$\n}\n","sha256_utf8":"cdb9d6fe57ea2ff58fe9c842e700f502870ac0c7fe699c7c973a008d819a0545"},{"start":115246,"end":124793,"start_line":1480,"end_line":1540,"raw_tex":"\\begin{proof} \\fys{Throughout, we let $\\lambda_{k,\\min}$, $\\lambda_{k,\\max}$, and $B_k$ denote the minimum eigenvalue, maximum eigenvalue, and inverse of matrix $H_k$ in \\eqref{eqn:H-k-m}, respectively.}\nIt can be seen, by induction on $k$, that \\fy{$H_k$ is symmetric and $\\sF_k$ measurable.} We use induction on odd values of \\fys{$k\\geq 2m-1$ to show that parts (a), (b), and (c) hold.} Suppose \\fy{$k\\geq 2m-1$} is odd and for any \\fys{odd} $t<k$, we have \\fy{$s_{\\lceil t/2\\rceil}^T{y_{\\lceil t/2\\rceil}} >0$, $H_{t}{y}_{\\lceil t/2\\rceil}=s_{\\lceil t/2\\rceil}$}, and \\eqref{proof:H_kbounds} for $t$. We show that these statements also hold for $k$ \\fys{as well}. First, we \\fys{show} that the \\fy{curvature condition} holds. We can write \n\\begin{align*}\n\\fys{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}&=(x_{k}-x_{k-1})^T(\\nabla F(x_k,\\xi_{k-1})- \\nabla F(x_{k-1},\\xi_{k-1})+\\fys{\\tau}\\mu_k^\\delta(x_k-x_{k-1}))\\\\\n&=(x_{k}-x_{k-1})^T(\\nabla F(x_k,\\xi_{k-1})- \\nabla F(x_{k-1},\\xi_{k-1}))+\\fys{\\tau}\\mu_k^\\delta\\|x_k-x_{k-1}\\|^2\\\\\n&\\geq \\fys{\\tau}\\mu_k^\\delta\\|x_k-x_{k-1}\\|^2,\n\\end{align*}\nwhere the inequality follows from the monotonicity of the gradient map $\\nabla F(\\cdot,\\xi)$. \\fys{Next, we show that $\\|x_k-x_{k-1}\\|^2 >0$.} From the induction hypothesis \\fys{and that $k-2$ is odd}, $H_{k-2}$ is positive definite.\n\\fys{Moreover, from the update rule \\eqref{eqn:H-k} and that $k-2$ is odd, we have $H_{k-1}=H_{k-2}$.}  \n%Also, since $B_{k-2}$ is symmetric, the update rule \\eqref{eqn:B-k} implies that $B_{k-1}$ is symmetric.\nTherefore, $H_{k-1}$ is \\fys{also} positive definite. \\fy{Without loss of generality, we assume $\\nabla F(x_{k-1},\\xi_{k-1})+\\mu_{k-1}(x_{k-1}-x_0)\\neq 0$}\\footnote{If $\\nabla F(x_{k},\\xi_k)+\\mu_{k}(x_k-x_0)=0$\\fys{,} then we can draw a new sample of $\\xi_k$ to satisfy the relation.}.\nSince $H_{k-1}$ is positive definite, \nwe have $$H_{k-1}\\left(\\nabla F(x_{k-1},\\xi_{k-1})+\\mu_{k-1}\\fy{(x_{k-1}-x_0)}\\right) \\neq 0,$$ implying that \n$x_{k} \\neq x_{k-1}$. Hence\n$\\fy{s_{\\lceil k/2\\rceil}^T{y_{\\lceil k/2\\rceil}} }\\geq  \\fys{\\tau}\\mu_k^\\delta\\|x_k-x_{k-1}\\|^2 >0,$\nwhere \\us{the second inequality is a consequence of} $\\fys{\\tau},\\mu_k>0$.\nThus, the \\fy{curvature condition} holds.\nNext, we show that \\eqref{proof:H_kbounds} holds for $k$. It is well-known that using the Sherman-Morrison-Woodbury formula, $B_k$ is equal to $B_{k,m}$ given by \n\\begin{align}\\label{equ:B_kLimited}\nB_{k,j}=B_{k,j-1}-\\frac{B_{k,j-1}s_is_i^TB_{k,j-1}}{s_i^TB_{k,j-1}s_i}+\\frac{y_iy_i^T}{y_i^Ts_i}, \\quad i:=\\fy{{\\lceil k/2\\rceil}-(m-j)}, \\quad 1 \\leq j \\leq m,\n\\end{align}\nwhere $s_i$ and $y_i$ are defined by \\eqref{equ:siyi-LBFGS} and \\fy{$B_{k,0}=\\frac{y_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}\\mathbf{I}$}. \\fys{Note that with $j$ varying between $1$ to $m$, the index $i$ takes values in $\\left\\{\\lceil k/2\\rceil-m+1,\\lceil k/2\\rceil-m+2,\\ldots,\\lceil k/2\\rceil\\right\\}$.} First, we show that for any $i$ \\fys{in this range}, \\begin{align}\\label{equ:boundsForB0}\n\\fys{\\tau}\\mu_k^\\delta \\leq \\frac{\\|y_i\\|^2}{y_i^Ts_i} \\leq L+\\fys{\\tau}\\mu_k^\\delta,\n\\end{align}\n where $L$ is the Lipschitzian parameter of the gradient mapping $\\nabla F$ \\fys{given by Assumption \\ref{assum:convex2}(b)}. Let us \\fys{define the function $h(x)\\triangleq F(x,\\xi_{i-1})+\\tau\\frac{\\mu_k^\\delta}{2}\\|x-x_0\\|^2$} for fixed $i$ and $k$. Note that this function is strongly convex and has a gradient mapping of the form $\\nabla F+\\fys{\\tau}\\mu_k^\\delta(\\mathbf{I}-x_0)$ that  is Lipschitz with parameter $L+\\fys{\\tau}\\mu_k^\\delta$. For a convex function $h$ with Lipschitz gradient with parameter $L+\\fys{\\tau}\\mu_k^\\delta$, the following inequality, referred to as co-coercivity  property, holds for any $x_1,x_2 \\in \\Real^n$ \\fys{(see}\\cite{Polyak87}, Pg. 24 , Lemma 2):\n \\[\\|\\nabla h(x_2)-\\nabla h(x_1)\\|^2 \\leq \\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)(x_2-x_1)^T(\\nabla h(x_2)-\\nabla h(x_1)).\\]\nSubstituting $x_2$ by $x_i$, $x_1$ by $x_{i-1}$, and recalling \\eqref{equ:siyi-LBFGS}, the preceding inequality yields\n\\begin{align}\\label{ineq:boundsForB0-1}\\|y_i\\|^2 \\leq \\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)s_i^Ty_i.\\end{align}\nNote that function $h$ is strongly convex \\fy{with} parameter $\\fys{\\tau}\\mu_k^\\delta$. Applying the Cauchy-Schwarz inequality, we can write\n\\[\\frac{\\|y_i\\|^2}{s_i^Ty_i} \\geq \\frac{\\|y_i\\|^2}{\\|s_i\\|\\|y_i\\|} =\\frac{\\|y_i\\|}{\\|s_i\\|}\\geq  \\frac{\\|y_i\\|\\|s_i\\|}{\\|s_i\\|^2} \\geq \\frac{y_i^Ts_i}{\\|s_i\\|^2}\\geq \\fys{\\tau}\\mu_k^\\delta.\\]\nCombining this relation with \\eqref{ineq:boundsForB0-1}, we obtain \\eqref{equ:boundsForB0}. Next, we show that the maximum eigenvalue of $B_k$ is bounded. Let $Trace(\\cdot)$ denote the trace of a matrix. Taking trace from both sides of \\eqref{equ:B_kLimited} and summing up over index $j$, we obtain \\fy{for $i:={\\lceil k/2\\rceil}-(m-j)$,}\n\\begin{align}\\label{ineq:trace}\nTrace(B_{k,m})&=Trace(B_{k,0})-\\sum_{j=1}^m Trace\\left(\\frac{B_{k,j-1}s_is_i^TB_{k,j-1}}{s_i^TB_{k,j-1}s_i}\\right)+\\sum_{j=1}^m Trace\\left(\\frac{y_iy_i^T}{y_i^Ts_i}\\right)\\cr\n& =Trace\\fy{\\left(\\frac{\\|y_{\\lceil k/2\\rceil}\\|^2}{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}\\mathbf{I}\\right)} - \\sum_{j=1}^m \\frac{\\|B_{k,j-1}s_i\\|^2}{s_i^TB_{k,j-1}s_i} + \\sum_{j=1}^m \\frac{\\|y_i\\|^2}{y_i^Ts_i}\\cr \n&\\leq n \\fy{\\frac{\\|y_{\\lceil k/2\\rceil}\\|^2}{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}} +\\sum_{j=1}^m \\left(L+\\fys{\\tau}\\mu_k^\\delta\\right) \\fy{\\leq} (m+n)\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right),\n\\end{align}\nwhere the third relation is obtained by positive-definiteness of $B_k$ (this can be seen by induction on \\fys{$j$}, and using \\eqref{equ:B_kLimited} and $B_{k,0}\\succ 0$). Since $B_k=B_{k,m}$, the maximum eigenvalue of the matrix $B_k$ is bounded \\fys{by $(m+n)\\left(L+\\tau\\mu_k^\\delta\\right)$}. As a result, \n\\begin{align}\\label{proof:lowerbound}\n\\lambda_{k,\\min}\\geq \\frac{1}{(m+n)\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)}.\\end{align}\n In the next part of the proof, we establish the bound for $\\lambda_{k,\\max}$. The following relation can be shown (e.g., see Lemma 3 in \\cite{Mokhtari15})\n\\begin{align*}\ndet(B_{k,m})=det(B_{k,0})\\prod_{j=1}^m\\frac{s_i^Ty_i}{s_i^TB_{k,j-1}s_i}\\fy{, \\quad \\hbox{for }i:={\\lceil k/2\\rceil}-(m-j)}.\n\\end{align*}\nMultiplying and dividing by $s_i^Ts_i$, using the strong convexity of the function $h$, and invoking \\eqref{equ:boundsForB0} and the result of \\eqref{ineq:trace}, we obtain\n\\begin{align}\\label{ineq:detBk}\ndet(B_{k})&=\\fy{det\\left(\\frac{y_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}\\mathbf{I}\\right)}\\prod_{j=1}^m\\left(\\frac{s_i^Ty_i}{s_i^Ts_i}\\right)\\left(\\frac{s_i^Ts_i}{s_i^TB_{k,j-1}s_i}\\right)\\cr  \n& \\geq\\fy{\\left(\\frac{y_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}\\right)^n}\\prod_{j=1}^m\\fys{\\tau}\\mu_k^\\delta\\left(\\frac{s_i^Ts_i}{s_i^TB_{k,j-1}s_i}\\right)\\cr\n& \\geq  \\fys{\\left(\\tau\\mu_k^\\delta\\right)^{(n+m)}} \\prod_{j=1}^m \\frac{1}{(m+n)\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)} = \\frac{\\fys{\\left(\\tau\\mu_k^\\delta\\right)^{(n+m)}}}{(m+n)^{m}\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)^m}.\n\\end{align}\nLet \\fy{$\\alpha_{k,1}\\leq \\alpha_{k,2}\\leq\\ldots\\leq\\alpha_{k,n}$} be the eigenvalues of $B_k$ sorted non-decreasingly. Note that since $B_k\\succ0$, all the eigenvalues are positive. Taking \\eqref{ineq:trace} and \\eqref{ineq:detBk} into account, and employing Lemma \\ref{sumProductBounds}, we obtain\n \\[\\alpha_{1,k} \\fy{\\geq} \\frac{(n-1)!\\fys{\\left(\\tau\\mu_k^\\delta\\right)^{(n+m)}}}{(m+n)^{n+m-1}\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)^{n+m-1}}.\\]\nThis relation and that $\\alpha_{k,1}=\\lambda_{k,\\max}^{-1}$ imply that \n\\begin{align}\\label{proof:upperbound}\n\\lambda_{k,\\max}\\leq \\frac{(m+n)^{n+m-1}\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)^{n+m-1}}{(n-1)!\\fys{\\left(\\tau\\mu_k^\\delta\\right)^{(n+m)}}}.\n\\end{align}\nTherefore, from \\eqref{proof:lowerbound} and \\eqref{proof:upperbound} and that $\\mu_k$ is non-increasing, we conclude that \\eqref{proof:H_kbounds} holds for $k$ as well. Next, we show $H_ky_{\\lceil k/2\\rceil}=s_{\\lceil k/2\\rceil}$. From \\eqref{equ:B_kLimited}, for $j=m$ we obtain\n\\[B_{k,m}=B_{k,m-1}-\\fy{\\frac{B_{k,m-1}s_{\\lceil k/2\\rceil}s_{\\lceil k/2\\rceil}^TB_{k,m-1}}{s_{\\lceil k/2\\rceil}^TB_{k,m-1}s_{\\lceil k/2\\rceil}}+\\frac{y_{\\lceil k/2\\rceil}y_{\\lceil k/2\\rceil}^T}{y_{\\lceil k/2\\rceil}^Ts_{\\lceil k/2\\rceil}}},\\]\nwhere we used \\fy{$i={\\lceil k/2\\rceil}-(m-m)={\\lceil k/2\\rceil}$}. Multiplying both sides of the preceding equation by \\fy{$s_{\\lceil k/2\\rceil}$}, and using $B_k=B_{k,m}$, we have\n\\fy{$B_{k}s_k=B_{k,m-1}s_{\\lceil k/2\\rceil}-B_{k,m-1}s_{\\lceil k/2\\rceil}+y_{\\lceil k/2\\rceil}=y_{\\lceil k/2\\rceil}$}. \nMultiplying both sides of the preceding relation by $H_k$ and invoking $H_k=B_k^{-1}$, we conclude that \\fy{$H_ky_{\\lceil k/2\\rceil}=s_{\\lceil k/2\\rceil}$}. Therefore, we showed that the statements of (a), (b), and (c) hold for \\fys{an odd $k$}, assuming that they hold for any odd \\fy{$t<k$}. In a similar fashion to this analysis, it can be seen that the statements hold for \\fys{$t=2m-1$}. Thus, by induction, we conclude that the statements hold for any odd \\fy{$k\\geq 2m-1$}. To complete the proof, it is enough to show that \\eqref{proof:H_kbounds} holds for any even \\fys{$k\\geq 2m$}. Let $t=k-1$. Since \\fys{$t$} is odd, relation \\eqref{proof:H_kbounds} holds. Writing  \\eqref{proof:H_kbounds} for $k-1$, and taking into account that $H_k=H_{k-1}$, and $\\mu_k<\\mu_{k-1}$, we can conclude that \\eqref{proof:H_kbounds} holds for any even \\fy{$k\\geq 2m-1$} and this completes the proof.\n\\end{proof}","sha256_utf8":"f8c59473781c16b5fad6f6ba0177ea46ae037eea247a4c87ffa7b3ec131c04ad"},{"start":124878,"end":127944,"start_line":1543,"end_line":1563,"raw_tex":"\\begin{proof}\nIn the following, we show that the presented class of sequences satisfy each of the conditions listed in Assumption \\ref{assum:sequences}. Throughout, we let $\\alpha$ denote \\fy{$-(m+n)\\delta$}.\\\\\n(a) Replacing the sequences by their given rules, we obtain \\fy{\\begin{align*} \\g_k\\mu_k^{2\\alpha-1} & =\\frac{\\g_0}{(2^b\\mu_0)^{1-2\\alpha}}(k+1)^{-a}(k+\\kappa)^{(1-2\\alpha) b}\\leq  \\frac{\\g_0}{(2^b\\mu_0)^{1-2\\alpha}}(k+1)^{-a+(1-2\\alpha) b}.\\end{align*}}\n\\fy{From the assumption that $\\frac{a}{b}>1+2\\delta(m+n)$, we obtain$-a+(1-2\\alpha) b<0$}. Thus, the preceding term goes to zero verifying Assumption \\ref{assum:sequences}(a).\\\\\n(b) Let $k$ be an even number. Thus, $\\kappa=2$. From \\eqref{equ:seq} we have $\\mu_{k}=\\mu_{k+1}=\\frac{\\mu_02^b}{\\left(k+2\\right)^b}$. Now, let $k$ be an odd number. Again, according to \\eqref{equ:seq} can write \n\\[\\mu_{k+1}=\\frac{\\mu_02^b}{\\left((k+1)+2\\right)^b }<\\frac{\\mu_02^b}{\\left(k+1\\right)^b}= \\frac{\\mu_02^b}{\\left(k+\\kappa\\right)^b}=\\mu_k.\\]\nTherefore, $\\mu_k$ given by \\eqref{equ:seq} satisfies \\eqref{eqn:mu-k}. Also, from \\eqref{equ:seq} we have $\\mu_k \\to 0$. Thus, Assumption \\ref{assum:sequences}(b) holds.\\\\\n(c) The given rules \\eqref{equ:seq} imply that $\\g_k$ and $\\mu_k$ are both non-increasing sequences. Therefore, we have $\\g_k\\mu_{k} \\leq \\g_0\\mu_{0}$ for any $k\\geq 0$. So, to show that Assumption \\ref{assum:sequences}(c) holds, it is enough to show that $\\lambda_{\\min}\\g_0\\mu_{0} \\leq 1$ where $\\lambda_{\\min}$ is given by \\eqref{equ:valuesForAssumH_k}. Since we assumed that $\\g_0\\mu_0 \\leq L(m+n)$, for any $\\delta \\in (0,1]$, we have $\\g_0\\mu_0 \\leq (m+n)(L+\\mu_0^\\delta)$, implying that $\\lambda_{\\min}\\g_0\\mu_{0} \\leq 1$ and that Assumption \\ref{assum:sequences}(c) holds.\\\\\n(d) From \\eqref{equ:seq}, we can write $$\\sum_{k=0}^\\infty \\g_k\\mu_k =\\g_0\\mu_02^b\\sum_{k=0}^\\infty (k+1)^{-a}(k+\\kappa)^{-b} \\geq \\g_0\\mu_02^b\\sum_{k=0}^\\infty (k+2)^{-(a+b)}= \\infty,$$\nwhere the last relation is due to $a+b\\leq 1$. Therefore, Assumption \\ref{assum:sequences}(d) holds.\\\\\n(e) Using \\eqref{equ:seq}, it follows\n\\fy{\\begin{align*}&\\sum_{k=0}^\\infty \\g_k\\mu_k^2 =  \\g_0\\mu_0^24^b\\sum_{k=0}^\\infty (k+1)^{-a}(k+\\kappa)^{-2b}\\leq   \\g_0\\mu_0^24^b\\sum_{k=0}^\\infty (k+1)^{-(a+2b)}<\\infty,\\end{align*}}\nwhere the last inequality is due to \\an{$a+2b>1$}. Therefore, Assumption \\ref{assum:sequences}(e) holds.\\\\\n(f) From \\eqref{equ:seq}, we have\n\\begin{align*}&\\sum_{k=0}^\\infty \\g_k^2\\mu_k^{2\\alpha} %=  \\g_0^2(\\mu_02^b)^{2\\alpha}\\sum_{k=0}^\\infty \\frac{(k+\\kappa)^{-2\\alpha b}}{(k+1)^{2a}} \n\\leq  \\g_0^2(\\mu_02^b)^{2\\alpha}\\left(\\sum_{k=0}^1 \\frac{(k+\\kappa)^{-2\\alpha b}}{(k+1)^{2a}}+\\sum_{k=2}^\\infty \\frac{(2k)^{-2\\alpha b}}{k^{2a}}\\right)<\\infty \n%&=  \\frac{\\g_0^2}{\\mu_0^24^c}\\left(\\sum_{k=0}^1 \\frac{(k+\\kappa)^{2c}}{(k+1)^{2a}}+4^c\\sum_{k=2}^\\infty k^{2(c-a)}\\right)<\\infty,\n\\end{align*}\nwhere in the first inequality, we use $\\alpha <0$ and in the last inequality, we note that \n$a+\\alpha b=a-\\delta\\left(m+n\\right)b>0.5$. Therefore, Assumption \\ref{assum:sequences}(f) is verified.\n\\end{proof}","sha256_utf8":"c8bce1058c16f01dede5636cf017ab5603647051ced3dafbd59b537ac2695f7e"},{"start":128032,"end":130148,"start_line":1566,"end_line":1582,"raw_tex":"\\begin{proof}\n\\fy{Throughout, we let $\\alpha$ denote $-\\delta(m+n)$.} \\fys{Assumption \\ref{assum:sequences-ms-convergence}(a, b, c) and \\eqref{eqn:mu-k} have been already shown in parts (a, b, c) of the proof of Lemma \\ref{lemma:a.s.sequences}.}\\\\\n\\noindent\n%(d) From part (a), since the term $\\g_k\\mu_k^{2\\alpha-1}$ goes to zero, there exist $K_2>0$ such that $\\g_k\\mu_k^{2\\alpha-1}\\leq \\frac{\\lambda_{\\min}}{\\lambda^2(L+\\mu_0)}$ for all $k \\geq K_2$, implying that Assumption \\ref{assum:sequences-ms-convergence}(d) holds for a large enough $k$. \\\\\n%\\noindent\n(d) It suffices to show there exists \\fy{$K_0$} such that for any $k \\geq K_0$ and $0<\\beta<1$, \n\\begin{align}\\label{ineq:partd}\\frac{\\g_{k-1}}{\\g_k}\\frac{\\mu_{k}^{1-2\\alpha}}{\\mu_{k-1}^{1-2\\alpha}}-1\\leq \\beta \\lambda_{\\min}\\g_k\\mu_k.\\end{align}\nFrom \\eqref{equ:seq} \\fy{and the definition of $\\alpha$}, we obtain \n\\begin{align*}&\\frac{\\g_{k-1}}{\\g_k}\\frac{\\mu_{k}^{1-2\\alpha}}{\\mu_{k-1}^{1-2\\alpha}}-1\n\\leq \\frac{\\g_{k-1}}{\\g_k}-1=\\left(1+\\frac{1}{k}\\right)^a-1= 1+\\frac{a}{k}+\\fy{{o}\\left(\\frac{1}{k}\\right)}-1=\\mathcal{O}\\left(\\frac{1}{k}\\right),\n\\end{align*}\nwhere the first inequality is implied due to $\\{\\mu_k\\}$ is non-increasing, and in the second equation, we used the Taylor's expansion of $\\left(1+\\frac{1}{k}\\right)^a$. Therefore, since the right-hand side of \\eqref{ineq:partd} is of the order $\\frac{1}{k^{a+b}}$ and that $a+b<1$, the preceding inequality shows that such $K_0$ exists for which Assumption \\ref{assum:sequences-ms-convergence}(d) holds for all $0<\\beta<1$.\\\\\n\\noindent\n(e) From \\eqref{equ:seq}, we have \n\\begin{align*} \\frac{\\mu_k^{2-2\\alpha}}{\\g_k}& =\\fy{\\g_0}^{-1}\\left(\\mu_02^b\\right)^{2-2\\alpha}(k+\\kappa)^{-b(2-2\\alpha)}(k+1)^{a} \\leq\\frac{\\fy{\\gamma_0}^{-1}\\left(\\mu_02^b\\right)^{2-2\\alpha}}{(k+1)^{-a+(2-2\\alpha)b}}\\\\&\\leq \\fy{\\gamma_0}^{-1}\\left(\\mu_02^b\\right)^{2-2\\alpha}=\\rho, \\end{align*}\nwhere the first inequality is due to \\fy{$\\alpha <0$, and the second inequality follows by the assumption $a\\leq 2b(1+\\delta(m+n))$}. Therefore, Assumption \\ref{assum:sequences-ms-convergence}(e) is satisfied.\n\\end{proof}","sha256_utf8":"c07c1cd0db350bf7f3a7a6dc5650d833ae5052a4a6fdc205a7f6d925e49e6738"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"随机LBFGS速率与双循环等价性","source_units":[{"start":76897,"end":77885,"start_line":1067,"end_line":1079,"raw_tex":"\\begin{theorem}[\\fy{Rate analysis for Algorithm \\ref{algorithm:IR-S-BFGS}}]\\label{thm:rate}\n\\fy{Consider Algorithm \\ref{algorithm:IR-S-BFGS}. The following statements hold:}\n\\begin{itemize}\n\\item [(a)] Suppose Assumptions \\ref{assum:main} and \\ref{assum:convex2} hold. Then, there exists \\fy{$K\\geq2m-1$} such that \n\\[\\EXP{f(x_{k })}-f^*\\leq \\left(\\frac{\\theta\\g_0}{\\left(\\mu_0\\sqrt[3]{2}\\right)^{1-2\\alpha}}\\right)\\frac{1}{k^{\\frac{1}{3}-\\e}}, \\quad \\hbox{for all } k > K,\\] \nwhere $\\theta$ is given by \\eqref{def:theta}, and $\\lambda_{\\min}$, $\\lambda$, and $\\alpha$ are given by \\eqref{equ:valuesForAssumH_k}. \n\\item [(b)] \\fy{Let the scheme be at the $k$th iteration where $k\\geq 2m-1$. Then, by the end of the LBFGS \\fys{two-loop} recursion, i.e., line $\\#$26 in Algorithm \\ref{algorithm:IR-S-BFGS}, we have \n\\begin{align}\\label{eqn:r_Lemma}\n  r=H_k\\left(\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)\\right),\n\\end{align}\nwhere $H_{k}$ is defined by \\eqref{eqn:H-k}.}\n\\end{itemize}\n\\end{theorem}","sha256_utf8":"e234be7e4539db4077577c72ff64049e6f5190ecc38fa0e9da7522b5538ee684"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"主定理完整证明","source_units":[{"start":77886,"end":84680,"start_line":1080,"end_line":1146,"raw_tex":"\\begin{proof}\n\\noindent (a) First, we show that the conditions of Proposition~\\ref{prop:mean} are satisfied. Assumption \\ref{assum:convex} holds as a consequence of Assumption \\ref{assum:convex2}. From Lemma \\ref{LBFGS-matrix}(c), Assumption \\ref{assump:Hk} holds for any \\fy{$k \\geq 2m-1$} as well. To show that Assumption \\ref{assum:sequences-ms-convergence} holds, we apply Lemma \\ref{lemma:mean-sequences}. We have \n\\[\\frac{a}{b}=\\frac{\\frac{2}{3}-\\e+\\frac{2\\delta(n+m)}{3}}{1/3}=2-3\\e+2\\delta(n+m)>1+2\\delta\\left(m+n\\right),\\]\nwhere we used $\\e<\\frac{1}{3}$. Moreover, since $\\delta < \\frac{1.5\\e}{n+m}$, we have $a+b=1-\\e +\\frac{2\\delta(n+m)}{3}<1$. Also, from the values of $a$ and $b$ we have $2b\\left(1+\\delta\\left(m+n\\right)\\right)=a+\\e>a.$\nThus, the conditions of Lemma \\ref{lemma:mean-sequences} hold. This implies that there exists $K_0>0$ such that for any $k\\geq K_0$,  the sequences $\\g_k$ and $\\mu_k$ satisfy Assumption \\ref{assum:sequences-ms-convergence} with any arbitrary $0<\\beta<1$ and for $\\rho=\\g_0^{-1}\\left(\\mu_02^b\\right)^{2+2\\delta(m+n)}$, and with $\\lambda_{\\min}$, $\\lambda$ and $\\alpha$ given by \\eqref{equ:valuesForAssumH_k}. Let us define $K:=\\max\\{K_0,2m-1\\}$. Since all conditions in Proposition~\\ref{prop:mean} are met, \\an{from \\eqref{ineq:bound}, \\eqref{equ:seq}, and by substituting values of $a$, $b$, and $\\alpha$, for any $k \\geq K$ we obtain}\n\\begin{align*}\\EXP{f(x_{k+1})}-f^*&\\leq \\theta\\g_{k-1}\\mu_{k-1}^{2\\alpha-1}= \\frac{\\theta\\g_0(k+\\kappa-1)^{(1-2\\alpha)/3}}{\\left(\\mu_0\\sqrt[3]{2}\\right)^{1-2\\alpha}k^{2/3-\\e-\\frac{2}{3}\\alpha}}\\\\ &\\leq  \\frac{\\theta\\g_0(k+1)^{(1-2\\alpha)/3}}{\\left(\\mu_0\\sqrt[3]{2}\\right)^{1-2\\alpha}(k+1)^{2/3-\\e-\\frac{2}{3}\\alpha}}=\\left(\\frac{\\theta\\g_0}{\\left(\\mu_0\\sqrt[3]{2}\\right)^{1-2\\alpha}}\\right)\\frac{1}{(k+1)^{\\frac{1}{3}-\\e}}.\\end{align*}\nThrough a change of variable from $k+1$ to $k$, we conclude the result. \\\\\n\n\\fy{\\noindent (b)  To show \\eqref{eqn:r_Lemma}, it suffices to show that\n \\begin{align*}\n  r=\\begin{cases}\n    H_{k,m}\\left(\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)\\right),  &    \\text{if } k \\text{ is odd}, \\\\\n    H_{k-1,m}\\left(\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)\\right),  &   \\text{otherwise},\n  \\end{cases}\n\\end{align*}\nwhere $H_{k,m}$ is defined by the recursion \\eqref{eqn:H-k-m} for an odd $k$. First, consider the case that $k\\geq 2m-1$ is an odd number. As such, at the $k$th iteration, from line $\\#$7, we have $i:=\\lceil k/2 \\rceil$. For clarity of the presentation, throughout \\vus{this} proof, we use $K$ (instead of $i$), i.e., $K\\triangleq \\lceil k/2 \\rceil$ \\vus{and} $q_{K-t+1}$ \\vus{is used to} denote the value of the vector $q \\in \\mathbb{R}^n$ after being updated at iteration $t$ in line $\\#$21. Similarly, we use $r_{t-K+m}$ to denote the value of the vector $r \\in \\mathbb{R}^n$ after being updated at iteration $t$ in line $\\#$25. Also, we use the following definitions:\n\\begin{align*}\n& q_0\\triangleq \\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0), \\quad r_0\\triangleq H_{k,0}q_{m},\\cr \n&\\rho_j\\triangleq \\frac{1}{y_j^Ts_j}, \\hbox{ and }V_j\\triangleq \\mathbf{I}-\\rho_{j}y_js_j^T, \\quad \\hbox{for all } j=K-(m-1),\\ldots, K.\n\\end{align*}\nConsider relation \\eqref{eqn:H-k-m}. By applying this recursive relation repeatedly, we obtain\n\\begin{align}\\label{equ:H_kClosedForm}\nH_{k,m} &=\\left(\\prod_{j=1}^{m}V_{K-(m-j)}\\right)^TH_{k,0}\\left(\\prod_{j=1}^{m}V_{K-(m-j)}\\right)\\\\\n&+ \\rho_{K-m+1}\\left(\\prod_{j=2}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+1}s^T_{K-m+1}\\left(\\prod_{j=2}^{m}V_{K-(m-j)}\\right) \\notag\\\\\n& + \\rho_{K-m+2}\\left(\\prod_{j=3}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+2}s^T_{K-m+2}\\left(\\prod_{j=3}^{m}V_{K-(m-j)}\\right)\\notag\\\\\n& + \\ldots\\notag\\\\\n& +\\rho_{K-1}V^T_{K}s_{K-1}s_{K-1}^TV_{K}\\notag\\\\\n&+\\rho_{K}s_{K}s_{K}^T.\\notag\n\\end{align}\nNext, we derive a formula for $q_t$. From lines $\\#$20-21 in the algorithm, we have\n\\begin{align*}\nq_{K-t+1}&=q_{K-t} -\\alpha_{K-t+1}y_t =q_{K-t} - \\rho_t\\left(s_t^Tq_{K-t}\\right)y_t = q_{K-t} - \\rho_t\\left(y_ts_t^T\\right)q_{K-t} \\\\\n&= \\left( \\mathbf{I}-\\rho_{t}y_ts_t^T\\right)q_{K-t}=V_tq_{K-t}, \\quad \\hbox{for all } t= K, K-1, \\ldots, K-m+1.\n\\end{align*}\nFrom the preceding relation, we obtain\n\\begin{align}\\label{equ:q_ell}\nq_{\\ell}=\\left(\\prod_{j=m-\\ell+1}^{m}V_{K-(m-j)}\\right)q_0, \\quad \\hbox{for all } \\ell= 1,2, \\ldots, m.\n\\end{align}\nFrom the update rule for $\\alpha_{i-t+1}$ in line $\\#$20, using the definition of $\\rho_t$, and applying the previous relation, we have $\\alpha_1=\\rho_{K}s_K^Tq_0$ and \n\\begin{align}\\label{equ:alpha_ell}\n\\alpha_{\\ell}=\\rho_{K-\\ell+1}s_{K-\\ell+1}^T\\left(\\prod_{j=m-\\ell+2}^{m}V_{K-(m-j)}\\right)q_0, \\quad \\hbox{for all } \\ell= 2,3, \\ldots, m.\n\\end{align}\n\\fys{Multiplying both sides of} \\eqref{equ:H_kClosedForm} by $q_0$ and employing \\eqref{equ:q_ell} and \\eqref{equ:alpha_ell}, we obtain\n\\begin{align}\\label{equ:H_kq_0}\nH_{k,m}q_0 &=\\left(\\prod_{j=1}^{m}V_{K-(m-j)}\\right)^TH_{k,0}q_m + \\left(\\prod_{j=2}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+1}\\alpha_m \\\\\n& + \\left(\\prod_{j=3}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+2}\\alpha_{m-1}  +\\ldots+V^T_{K}s_{K-1}\\alpha_2+s_{K}\\alpha_1.\\notag\n\\end{align}\nNext, we derive a formula for $r_t$. From line $\\#$25 in the algorithm, we have \n\\begin{align*}\nr_{t-K+m}&=r_{t-K+m-1}+\\left(\\alpha_{K-t+1}-\\rho_ty_t^Tr_{t-K+m-1}\\right)s_t \\\\\n&= r_{t-K+m-1}-\\rho_ts_t y_t^Tr_{t-K+m-1}+\\alpha_{K-t+1}s_t\\\\\n& = V_t^Tr_{t-K+m-1}+\\alpha_{K-t+1}s_t, \\quad \\hbox{for all } t= K-m+1,\\ldots,K-1,  K.\n\\end{align*}\nCombining the preceding two relations, we obtain\n\\begin{align*}\nr_{\\ell}&= V_{K-(m-\\ell)}^Tr_{\\ell-1}+\\alpha_{m-\\ell+1}s_{K-(m-\\ell)}, \\quad \\hbox{for all } \\ell= 1,2,\\ldots,m.\n\\end{align*}\nUsing the preceding equation repeatedly, we obtain\n\\begin{align}\\label{equ:r_m}\nr_m &=\\left(\\prod_{j=1}^{m}V_{K-(m-j)}\\right)^Tr_0 +\\alpha_m \\left(\\prod_{j=2}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+1} \\\\\n& + \\alpha_{m-1} \\left(\\prod_{j=3}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+2} +\\ldots+\\alpha_2V^T_{K}s_{K-1}+\\alpha_1s_{K}.\\notag\n\\end{align}\nFrom \\eqref{equ:r_m} and \\eqref{equ:H_kq_0}, and the definition of  $r_0$, we obtain $r_m=H_{k,m}q_0$. Taking to account the definition of $q_0$, the desired result holds for any odd $k\\geq 2m-1$. Now, consider the case where $k\\geq 2m-1$ is an even number. This implies that the ``if'' condition in line $\\#$6 is skipped and as such, the value of $i$ is not updated from the iteration $k-1$, i.e.,  $i= \\lceil (k-1)/2 \\rceil$. Consequently, the LBFGS two-loop recursion at an even $k$ uses the following pairs \n$$\\left\\{(s_\\ell,y_\\ell)\\mid \\ell=\\lceil (k-1)/2\\rceil-m+1, \\lceil (k-1)/2\\rceil-m+2,\\ldots,\\lceil (k-1)/2\\rceil\\right\\}.$$\n\\fys{Now,} considering the definition \\eqref{eqn:H-k-m} for $k-1$, the desired relation can be shown following the same steps discussed for the case the iteration number is odd.}\n%This implies the desired rate statement holds. \n\\end{proof}","sha256_utf8":"ed7c35d155b86fdf8b53375ac6be1fdbd0e365e9b6e4606677a6c8cad58aa6bb"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"验证均值收敛全部条件","source_units":[{"start":77886,"end":79273,"start_line":1080,"end_line":1084,"raw_tex":"\\begin{proof}\n\\noindent (a) First, we show that the conditions of Proposition~\\ref{prop:mean} are satisfied. Assumption \\ref{assum:convex} holds as a consequence of Assumption \\ref{assum:convex2}. From Lemma \\ref{LBFGS-matrix}(c), Assumption \\ref{assump:Hk} holds for any \\fy{$k \\geq 2m-1$} as well. To show that Assumption \\ref{assum:sequences-ms-convergence} holds, we apply Lemma \\ref{lemma:mean-sequences}. We have \n\\[\\frac{a}{b}=\\frac{\\frac{2}{3}-\\e+\\frac{2\\delta(n+m)}{3}}{1/3}=2-3\\e+2\\delta(n+m)>1+2\\delta\\left(m+n\\right),\\]\nwhere we used $\\e<\\frac{1}{3}$. Moreover, since $\\delta < \\frac{1.5\\e}{n+m}$, we have $a+b=1-\\e +\\frac{2\\delta(n+m)}{3}<1$. Also, from the values of $a$ and $b$ we have $2b\\left(1+\\delta\\left(m+n\\right)\\right)=a+\\e>a.$\nThus, the conditions of Lemma \\ref{lemma:mean-sequences} hold. This implies that there exists $K_0>0$ such that for any $k\\geq K_0$,  the sequences $\\g_k$ and $\\mu_k$ satisfy Assumption \\ref{assum:sequences-ms-convergence} with any arbitrary $0<\\beta<1$ and for $\\rho=\\g_0^{-1}\\left(\\mu_02^b\\right)^{2+2\\delta(m+n)}$, and with $\\lambda_{\\min}$, $\\lambda$ and $\\alpha$ given by \\eqref{equ:valuesForAssumH_k}. Let us define $K:=\\max\\{K_0,2m-1\\}$. Since all conditions in Proposition~\\ref{prop:mean} are met, \\an{from \\eqref{ineq:bound}, \\eqref{equ:seq}, and by substituting values of $a$, $b$, and $\\alpha$, for any $k \\geq K$ we obtain}\n","sha256_utf8":"6abe013b0be0d93d181a3984a922d0463913b09883001ec27c8738bb4cee104b"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"代入参数得到随机收敛率","source_units":[{"start":79273,"end":79784,"start_line":1085,"end_line":1086,"raw_tex":"\\begin{align*}\\EXP{f(x_{k+1})}-f^*&\\leq \\theta\\g_{k-1}\\mu_{k-1}^{2\\alpha-1}= \\frac{\\theta\\g_0(k+\\kappa-1)^{(1-2\\alpha)/3}}{\\left(\\mu_0\\sqrt[3]{2}\\right)^{1-2\\alpha}k^{2/3-\\e-\\frac{2}{3}\\alpha}}\\\\ &\\leq  \\frac{\\theta\\g_0(k+1)^{(1-2\\alpha)/3}}{\\left(\\mu_0\\sqrt[3]{2}\\right)^{1-2\\alpha}(k+1)^{2/3-\\e-\\frac{2}{3}\\alpha}}=\\left(\\frac{\\theta\\g_0}{\\left(\\mu_0\\sqrt[3]{2}\\right)^{1-2\\alpha}}\\right)\\frac{1}{(k+1)^{\\frac{1}{3}-\\e}}.\\end{align*}\nThrough a change of variable from $k+1$ to $k$, we conclude the result. \\\\\n","sha256_utf8":"2bbadc453f07005016603f83404f0f3c639db19832765fbdc5d77e83edfe3ab0"}]},{"id":"S005","semantic_type":"proof_step","kind":"proof_step","title":"区分奇偶迭代并定义递推变量","source_units":[{"start":79785,"end":81028,"start_line":1088,"end_line":1099,"raw_tex":"\\fy{\\noindent (b)  To show \\eqref{eqn:r_Lemma}, it suffices to show that\n \\begin{align*}\n  r=\\begin{cases}\n    H_{k,m}\\left(\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)\\right),  &    \\text{if } k \\text{ is odd}, \\\\\n    H_{k-1,m}\\left(\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)\\right),  &   \\text{otherwise},\n  \\end{cases}\n\\end{align*}\nwhere $H_{k,m}$ is defined by the recursion \\eqref{eqn:H-k-m} for an odd $k$. First, consider the case that $k\\geq 2m-1$ is an odd number. As such, at the $k$th iteration, from line $\\#$7, we have $i:=\\lceil k/2 \\rceil$. For clarity of the presentation, throughout \\vus{this} proof, we use $K$ (instead of $i$), i.e., $K\\triangleq \\lceil k/2 \\rceil$ \\vus{and} $q_{K-t+1}$ \\vus{is used to} denote the value of the vector $q \\in \\mathbb{R}^n$ after being updated at iteration $t$ in line $\\#$21. Similarly, we use $r_{t-K+m}$ to denote the value of the vector $r \\in \\mathbb{R}^n$ after being updated at iteration $t$ in line $\\#$25. Also, we use the following definitions:\n\\begin{align*}\n& q_0\\triangleq \\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0), \\quad r_0\\triangleq H_{k,0}q_{m},\\cr \n&\\rho_j\\triangleq \\frac{1}{y_j^Ts_j}, \\hbox{ and }V_j\\triangleq \\mathbf{I}-\\rho_{j}y_js_j^T, \\quad \\hbox{for all } j=K-(m-1),\\ldots, K.\n\\end{align*}\n","sha256_utf8":"10079b82aef908db615d6996c758b79d2359900fa95e280b99d15b816a946464"}]},{"id":"S006","semantic_type":"proof_step","kind":"proof_step","title":"展开有限记忆逆矩阵","source_units":[{"start":81028,"end":81623,"start_line":1100,"end_line":1108,"raw_tex":"Consider relation \\eqref{eqn:H-k-m}. By applying this recursive relation repeatedly, we obtain\n\\begin{align}\\label{equ:H_kClosedForm}\nH_{k,m} &=\\left(\\prod_{j=1}^{m}V_{K-(m-j)}\\right)^TH_{k,0}\\left(\\prod_{j=1}^{m}V_{K-(m-j)}\\right)\\\\\n&+ \\rho_{K-m+1}\\left(\\prod_{j=2}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+1}s^T_{K-m+1}\\left(\\prod_{j=2}^{m}V_{K-(m-j)}\\right) \\notag\\\\\n& + \\rho_{K-m+2}\\left(\\prod_{j=3}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+2}s^T_{K-m+2}\\left(\\prod_{j=3}^{m}V_{K-(m-j)}\\right)\\notag\\\\\n& + \\ldots\\notag\\\\\n& +\\rho_{K-1}V^T_{K}s_{K-1}s_{K-1}^TV_{K}\\notag\\\\\n&+\\rho_{K}s_{K}s_{K}^T.\\notag\n\\end{align}\n","sha256_utf8":"5cefe7bc8d0c49fe04f73633e79cdeca0d248806f0897165e1bc565cdb8cd8c3"}]},{"id":"S007","semantic_type":"proof_step","kind":"proof_step","title":"推导第一循环向量递推","source_units":[{"start":81623,"end":81981,"start_line":1109,"end_line":1113,"raw_tex":"Next, we derive a formula for $q_t$. From lines $\\#$20-21 in the algorithm, we have\n\\begin{align*}\nq_{K-t+1}&=q_{K-t} -\\alpha_{K-t+1}y_t =q_{K-t} - \\rho_t\\left(s_t^Tq_{K-t}\\right)y_t = q_{K-t} - \\rho_t\\left(y_ts_t^T\\right)q_{K-t} \\\\\n&= \\left( \\mathbf{I}-\\rho_{t}y_ts_t^T\\right)q_{K-t}=V_tq_{K-t}, \\quad \\hbox{for all } t= K, K-1, \\ldots, K-m+1.\n\\end{align*}\n","sha256_utf8":"5ecb8023b7b0eee281267aeb43e93b21e91c79bb8bb5cfdb8b7ce026c70f1e6f"}]},{"id":"S008","semantic_type":"proof_step","kind":"proof_step","title":"写出第一循环乘积表达式","source_units":[{"start":81981,"end":82167,"start_line":1114,"end_line":1117,"raw_tex":"From the preceding relation, we obtain\n\\begin{align}\\label{equ:q_ell}\nq_{\\ell}=\\left(\\prod_{j=m-\\ell+1}^{m}V_{K-(m-j)}\\right)q_0, \\quad \\hbox{for all } \\ell= 1,2, \\ldots, m.\n\\end{align}\n","sha256_utf8":"113320d6867e00081faf776b50a9825d3b02bdb14e5c7d1c6cc78d24f98fe486"}]},{"id":"S009","semantic_type":"proof_step","kind":"proof_step","title":"表达第一循环标量系数","source_units":[{"start":82167,"end":82521,"start_line":1118,"end_line":1121,"raw_tex":"From the update rule for $\\alpha_{i-t+1}$ in line $\\#$20, using the definition of $\\rho_t$, and applying the previous relation, we have $\\alpha_1=\\rho_{K}s_K^Tq_0$ and \n\\begin{align}\\label{equ:alpha_ell}\n\\alpha_{\\ell}=\\rho_{K-\\ell+1}s_{K-\\ell+1}^T\\left(\\prod_{j=m-\\ell+2}^{m}V_{K-(m-j)}\\right)q_0, \\quad \\hbox{for all } \\ell= 2,3, \\ldots, m.\n\\end{align}\n","sha256_utf8":"580d17e5df5f8383549658eced50cfc9ff29550d2652edad6dfc07f09761c30a"}]},{"id":"S010","semantic_type":"proof_step","kind":"proof_step","title":"表达逆矩阵与初始梯度之积","source_units":[{"start":82521,"end":82950,"start_line":1122,"end_line":1126,"raw_tex":"\\fys{Multiplying both sides of} \\eqref{equ:H_kClosedForm} by $q_0$ and employing \\eqref{equ:q_ell} and \\eqref{equ:alpha_ell}, we obtain\n\\begin{align}\\label{equ:H_kq_0}\nH_{k,m}q_0 &=\\left(\\prod_{j=1}^{m}V_{K-(m-j)}\\right)^TH_{k,0}q_m + \\left(\\prod_{j=2}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+1}\\alpha_m \\\\\n& + \\left(\\prod_{j=3}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+2}\\alpha_{m-1}  +\\ldots+V^T_{K}s_{K-1}\\alpha_2+s_{K}\\alpha_1.\\notag\n\\end{align}\n","sha256_utf8":"95f2b62ca2584a357e8ad3f0246623267498021c93ad84e96325ac7cecd7fbfe"}]},{"id":"S011","semantic_type":"proof_step","kind":"proof_step","title":"推导第二循环向量递推","source_units":[{"start":82950,"end":83288,"start_line":1127,"end_line":1132,"raw_tex":"Next, we derive a formula for $r_t$. From line $\\#$25 in the algorithm, we have \n\\begin{align*}\nr_{t-K+m}&=r_{t-K+m-1}+\\left(\\alpha_{K-t+1}-\\rho_ty_t^Tr_{t-K+m-1}\\right)s_t \\\\\n&= r_{t-K+m-1}-\\rho_ts_t y_t^Tr_{t-K+m-1}+\\alpha_{K-t+1}s_t\\\\\n& = V_t^Tr_{t-K+m-1}+\\alpha_{K-t+1}s_t, \\quad \\hbox{for all } t= K-m+1,\\ldots,K-1,  K.\n\\end{align*}\n","sha256_utf8":"326b1299e94bd983e153b13cd69487127de6ba8a2b6b57ca2ee9116b1a9d7131"}]},{"id":"S012","semantic_type":"proof_step","kind":"proof_step","title":"统一第二循环递推指标","source_units":[{"start":83288,"end":83478,"start_line":1133,"end_line":1136,"raw_tex":"Combining the preceding two relations, we obtain\n\\begin{align*}\nr_{\\ell}&= V_{K-(m-\\ell)}^Tr_{\\ell-1}+\\alpha_{m-\\ell+1}s_{K-(m-\\ell)}, \\quad \\hbox{for all } \\ell= 1,2,\\ldots,m.\n\\end{align*}\n","sha256_utf8":"488310ef2d1eb230a240f9ee60919eafc681582cc6e9fbae680e44f72126b739"}]},{"id":"S013","semantic_type":"proof_step","kind":"proof_step","title":"展开第二循环最终输出","source_units":[{"start":83478,"end":83805,"start_line":1137,"end_line":1141,"raw_tex":"Using the preceding equation repeatedly, we obtain\n\\begin{align}\\label{equ:r_m}\nr_m &=\\left(\\prod_{j=1}^{m}V_{K-(m-j)}\\right)^Tr_0 +\\alpha_m \\left(\\prod_{j=2}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+1} \\\\\n& + \\alpha_{m-1} \\left(\\prod_{j=3}^{m}V_{K-(m-j)}\\right)^Ts_{K-m+2} +\\ldots+\\alpha_2V^T_{K}s_{K-1}+\\alpha_1s_{K}.\\notag\n\\end{align}\n","sha256_utf8":"ebe88de47af0e380030d5900b7eb1bfd2a33b7b6092754122e6f9fb82b9eb8ea"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"匹配矩阵乘积并处理偶数迭代","source_units":[{"start":83805,"end":84680,"start_line":1142,"end_line":1146,"raw_tex":"From \\eqref{equ:r_m} and \\eqref{equ:H_kq_0}, and the definition of  $r_0$, we obtain $r_m=H_{k,m}q_0$. Taking to account the definition of $q_0$, the desired result holds for any odd $k\\geq 2m-1$. Now, consider the case where $k\\geq 2m-1$ is an even number. This implies that the ``if'' condition in line $\\#$6 is skipped and as such, the value of $i$ is not updated from the iteration $k-1$, i.e.,  $i= \\lceil (k-1)/2 \\rceil$. Consequently, the LBFGS two-loop recursion at an even $k$ uses the following pairs \n$$\\left\\{(s_\\ell,y_\\ell)\\mid \\ell=\\lceil (k-1)/2\\rceil-m+1, \\lceil (k-1)/2\\rceil-m+2,\\ldots,\\lceil (k-1)/2\\rceil\\right\\}.$$\n\\fys{Now,} considering the definition \\eqref{eqn:H-k-m} for $k-1$, the desired relation can be shown following the same steps discussed for the case the iteration number is odd.}\n%This implies the desired rate statement holds. \n\\end{proof}","sha256_utf8":"2393e433d73b4dcefdc89197a3435a16ba81983199f48515c3d201d70a26ca03"}]},{"id":"S015","semantic_type":"problem_setting","kind":"problem_setting","title":"随机凸优化问题与概率空间","source_units":[{"start":9925,"end":10719,"start_line":253,"end_line":261,"raw_tex":"\\fy{We} consider the following stochastic optimization problem:\n\\begin{equation}\\label{eqn:problem}\n\\min_{x \\in \\Real^n} f(x)\\fy{\\ \\triangleq \\ }\\EXP{F(x,\\xi(\\us{\\omega}))},\\tag{SO}\n\\end{equation}\nwhere $F: \\Real^n\\times\\us{\\Real^d}\\to\\mathbb{R}$ is a \\fy{real-valued} function, the random\nvector $\\xi$ is defined as {$\\xi:\\Omega\n\\rightarrow \\Real^d$}, \\us{$(\\Omega,{\\cal F}, \\mathbb{P})$ denotes the\nassociated probability space\\fy{,} and the expectation \\us{$\\EXP{F(x,\\xi)}$} is\ntaken with respect to $\\mathbb{P}$}. Problem \\eqref{eqn:problem} provides a general framework that can capture a wide range of applications in operations research, machine learning, statistics and control to name a few \\fys{(\\vus{cf.}\\cite{bottou-2010,nocedal15}). Addressing problem \\eqref{eqn:problem}} has led\n","sha256_utf8":"2185a6dedab61e32535871ef890c858b0df1fba026673f468a015f7d89daa884"}]},{"id":"S016","semantic_type":"notation","kind":"notation","title":"范数光滑性与最优解记号","source_units":[{"start":23003,"end":24301,"start_line":409,"end_line":428,"raw_tex":" \\textbf{Notation:} A vector $x$ is assumed to be a column vector and\n$x^T$ denotes its transpose, \\an{while} $\\|x\\|$ {denotes} the Euclidean\nvector norm, i.e., $\\|x\\|=\\sqrt{x^Tx}$.  We write \\textit{a.s.} as the\nabbreviation for ``almost surely''.  \\an{For a symmetric matrix $B$, \n$\\lambda_{\\min}(B)$ and $\\lambda_{\\max}(B)$ \\vus{denotes the\nsmallest and largest eigenvalue of $B$}, respectively.} We use $\\EXP{z}$ to\ndenote the expectation of a random variable~$z$. A function $f:X\n\\subset \\mathbb{R}^n\\rightarrow \\mathbb{R}$ is said to be strongly\nconvex with parameter $\\mu>0$, if  $f(y)\\geq f(x)+\\nabla\nf(x)^T(y-x)+\\frac{\\mu}{2}\\|x-y\\|^2,$ for any $x,y \\in X$.  A mapping\n$F:X \\subset \\mathbb{R}^n\\rightarrow \\mathbb{R}$ is \\vus{said to be} Lipschitz\ncontinuous with parameter $L>0$ if for any $x, y \\in X$, we have\n$\\|F(x)-F(y)\\|\\leq L\\|x-y\\|$. For a continuously differentiable\nfunction $f$ with Lipschitz gradients with parameter $L>0$, we have\n$f(y)\\leq f(x)+\\nabla f(x)^T(y-x)+\\frac{L}{2}\\|x-y\\|^2,$ for any $x,y\n\\in X$. For a vector $x \\in \\Real^n$ and a \\fys{nonempty} set $X\n\\subset \\Real^n$, the Euclidean distance of $x$ from $X$ is denoted by\n$dist(x,X)$. We denote the optimal objective value of problem\n\\eqref{eqn:problem} by $f^*$ and the set of the optimal solutions by\n$X^*$.\n","sha256_utf8":"db37fc7f4f4e719c323e870f227c11b58a6d92a3ab445360f8956111f3c1ce95"}]},{"id":"S017","semantic_type":"algorithm","kind":"algorithm","title":"迭代正则化SQN与历史信息","source_units":[{"start":24356,"end":24761,"start_line":433,"end_line":436,"raw_tex":"In this section, we describe a general SQN scheme for solving problem~\\eqref{eqn:problem} and present the main assumptions. Let $x_0\\in \\Real^n$ be an arbitrary initial point, and $x_k$ be generated by the following recursive rule\n\\begin{align}\\label{eqn:LM-cyclic-reg-BFGS}\\tag{IR-SQN}\nx_{k+1}:=x_k -\\gamma_kH_k\\left(\\nabla F(x_k,\\xi_k)+ \\mu_k(x_k-x_0)\\right), \\quad \\hbox{for all } k \\geq 0.\n\\end{align}","sha256_utf8":"030898d70793c86ac87f42fdd540f71efa652d379f25e44b6cbff9fe5d6eecf0"},{"start":24762,"end":25568,"start_line":437,"end_line":438,"raw_tex":"Here, $\\g_k$ and $\\mu_k$ are the steplength and the regularization parameter, respectively. $H_k \\in \\Real^{n\\times n}$ is a matrix that \\fy{contains} the curvature information of the objective function. The \\eqref{eqn:LM-cyclic-reg-BFGS} scheme can be seen as a regularized variant of \\fy{the} classical stochastic SQN \\fy{method}. Here we regularize the gradient map by the term $\\mu_k(x_k-x_0)$ to \\fy{induce} the strong monotonicity property. In \\fy{the} absence of strong convexity of $f$, unlike the classical schemes where $\\mu_k$ is maintained fixed, we let $\\mu_k$ be updated and \\fy{decreased} to zero. Throughout, we let $\\sF_k$ denote the history of the method up to time $k$, i.e., \n$\\sF_k\\fy{\\ \\triangleq \\ }\\{x_0,\\xi_0,\\xi_1,\\ldots,\\xi_{k-1}\\}$ for $k\\ge 1$, and $\\sF_0\\triangleq\\{x_0\\}$.  \n","sha256_utf8":"e4b85c8f8705bf3f5fd858cfb3d9041185f5b26d3dc08fd999c41c1f3bd96e96"}]},{"id":"S018","semantic_type":"assumption","kind":"assumption","title":"总体目标凸性光滑性与可解性","source_units":[{"start":25568,"end":26084,"start_line":439,"end_line":447,"raw_tex":"\\begin{assumption}\\label{assum:convex}  Consider problem~\\eqref{eqn:problem}. Let the following hold:\n\\begin{enumerate}\n\\item [(a)]  \\an{The function $f(x)$ is convex over $\\Real^n$.}  \n%\\an{The function $f(x)$ is convex on $\\mathbb{R}^n$.} \n\\item[(b)] $f(x)$ is continuously differentiable \\us{with Lipschitz\n\tcontinuous gradients} over $\\Real^n$ with parameter $L>0$. %Also, $\\lim_{x\\to \\infty}f(x)< \\infty$.\n\t\\item[(c)] \\ The optimal solution set of \\fy{the problem} is nonempty. \n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"f3a5f0b7c57eb793f39f23a733a647c4468a34dfb7f6ce3fee8dd0d1644e0e58"}]},{"id":"S019","semantic_type":"assumption","kind":"assumption","title":"独立采样与无偏有界方差","source_units":[{"start":26219,"end":26818,"start_line":449,"end_line":457,"raw_tex":"\\begin{assumption}[Random variable $\\xi$]\\label{assum:main}\n\\begin{enumerate}\n\\item[(a)]  \\fy{Vectors $\\xi_k \\in \\mathbb{R}^d$ are i.i.d.\\ realizations of the random variable $\\xi$} for any $k \\geq 0$; \n\\item[(b)] The stochastic gradient mapping $\\nabla F(x, \\xi)$ is an unbiased estimator of $\\nabla f(x)$, i.e.\n$\\EXP{\\nabla F(x,\\xi)}=\\nabla f(x)$ for all $x$, and has a bounded variance,\n\ti.e., there exists a \\us{scalar} $\\nu>0$ such that $\\EXP{\\|\\nabla F(x,\\xi)-\\nabla f(x)\\|^2} \\leq \\nu^2,$ for all $x \\in \\mathbb{R}^n$.\n%Also, $\\lim_{x\\to \\infty}f(x)< \\infty$.\n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"baf6075aff4c35514acd283bd57e8724454db82e0c737338a58562c3a1868ea5"}]},{"id":"S020","semantic_type":"assumption","kind":"assumption","title":"逆矩阵可测性与谱界条件","source_units":[{"start":26879,"end":27495,"start_line":459,"end_line":463,"raw_tex":"\\begin{assumption}[Conditions on matrix $H_k$]\\label{assump:Hk}\nLet the following hold for all $k\\geq 0$:\\\\ \n(a) \\  The matrix $H_k\\in \\Real^{n\\times n}$ is $\\sF_{k}$-measurable, i.e., $\\EXP{H_k \\mid \\sF_{k}}=H_k$. \\\\\n(b) \\ Matrix $H_k$ is symmetric and positive definite and satisfies the following condition: There exist positive scalars $\\lambda_{\\min},\\lambda$ and scalar $\\alpha\\leq 0$ such that $$\\lambda_{\\min}\\mathbf{I} \\preceq H_k \\preceq \\lambda \\mu_k^\\alpha\\mathbf{I},\\qquad  \\hbox{for all }k\\geq 0,$$ \\fys{where $\\mu_k$ is the regularization parameter in \\eqref{eqn:LM-cyclic-reg-BFGS}.}\n\\end{assumption}","sha256_utf8":"793bb3d1d98413c5d369f6d21f98f1a69ef6849edb6ff4a461ca78399b15ad88"}]},{"id":"S021","semantic_type":"definition","kind":"definition","title":"正则化目标与随机梯度","source_units":[{"start":28281,"end":28777,"start_line":469,"end_line":476,"raw_tex":"\\begin{definition}[Regularized function and gradient mapping]\\label{def:regularizedF}\nConsider the sequence $\\{\\mu_k\\}$ of positive \\an{scalars} and the starting point of the algorithm \\eqref{eqn:LM-cyclic-reg-BFGS}, i.e., $x_0$.\n% for $k \\geq 0$. \nThe regularized function $f_k$ and its gradient are defined as follows \\fys{for all $k\\geq 0$}:\n\\begin{align*}\nf_k(x)\\triangleq  f(x)+\\frac{\\mu_k}{2}{\\|x-x_0\\|^2},\\quad \\nabla f_k(x)\\triangleq\\nabla f(x)+\\mu_k(x-x_0).\n\\end{align*}\n\\end{definition}","sha256_utf8":"f2ea9d08f5de185bcbabdd99de1f4a9e5e7c4c9fc90fe0fb94e46c21d561a87c"},{"start":28778,"end":28928,"start_line":477,"end_line":477,"raw_tex":"In a similar way, we denote the regularized stochastic function $F(x,\\xi)$ and its gradient with $F_k$ and $\\nabla F_k$ for any $\\xi$, respectively. \n","sha256_utf8":"9c15ece068a4275f8baaead7ccc33d7f3691b0666d4c34e5b17a7684029c83bc"}]},{"id":"S022","semantic_type":"definition","kind":"definition","title":"随机梯度误差与条件矩","source_units":[{"start":30561,"end":30755,"start_line":506,"end_line":509,"raw_tex":"Throughout, we denote the stochastic error of the gradient estimator by \n\\begin{align}\n\\label{def:wk}w_k\\triangleq \\nabla F(x_k,\\xi_k)-\\nabla f(x_k), \\an{\\quad\\hbox{for all }k\\ge0.} \n\\end{align}","sha256_utf8":"69200a9c7b1d333ac4d2af465a7ed25104ace918bb1b4fbf574ff25a523a603f"},{"start":30756,"end":31037,"start_line":510,"end_line":510,"raw_tex":"Note that under Assumption \\ref{assum:main}, from the definition of $w_k$ in \\eqref{def:wk}, we obtain $\\EXP{w_k\\mid \\sF_k}=0$ and $\\EXP{\\|w_k\\|^2\\mid \\sF_k}\\fy{\\leq}  \\nu^2$. \\fy{The following result plays a key role in the convergence and rate analysis of the proposed schemes.}\n","sha256_utf8":"317454b9290877d038b3d3639a730cad2bed65f73b8747d21c918d0e6c4d8cd1"}]},{"id":"S023","semantic_type":"assumption","kind":"assumption","title":"几乎必然收敛序列条件","source_units":[{"start":37741,"end":38366,"start_line":575,"end_line":584,"raw_tex":"\\begin{assumption}[\\fy{Conditions} on sequences for a.s. convergence]\\label{assum:sequences}\nLet the sequences $\\{\\gamma_k\\}$ and $\\{\\mu_k\\}$ be positive and satisfy the following conditions:\n\\begin{flalign*}\n\\begin{array}{lll}\n\\vspace{.2in}(a)\\  \\fy{\\lim_{k \\to \\infty }\\g_k\\mu_k^{2\\alpha-1} =0;}& &(b)\\ \\{\\mu_k\\}\\hbox{ is non-increasing and }\\mu_k \\to 0;\\\\\n\\vspace{.2in}(c)\\  \\lambda_{\\min}\\g_k\\mu_k \\leq 1\\hbox{ for }k \\geq 0; && (d)\\ \\sum_{k=0}^\\infty \\g_k\\mu_k =\\infty;\\\\\n(e)\\ \\sum_{k=0}^\\infty \\mu_k^2\\g_k <\\infty;&&(f)\\  \\sum_{k=0}^\\infty \\g_k^2\\mu_k^{2\\alpha} <\\infty\\fy{.}\n\\end{array}\n\\end{flalign*}\n\\end{assumption}","sha256_utf8":"76c66b0c7f0f6404c98c6ca955aa4bee2446ca88b6d903aef9147a970ee2ae4a"}]},{"id":"S024","semantic_type":"assumption","kind":"assumption","title":"均值收敛序列条件","source_units":[{"start":41876,"end":42745,"start_line":627,"end_line":637,"raw_tex":"\\begin{assumption}[\\fy{Conditions} on sequences for convergence in mean]\\label{assum:sequences-ms-convergence}\nLet the sequences $\\{\\gamma_k\\}$ and $\\{\\mu_k\\}$ be positive and satisfy the following conditions:\n\\begin{itemize}\n\\item [(a)] $\\lim_{k \\to \\infty }\\g_k\\mu_k^{2\\alpha-1} =0$;\n\\item [(b)] $\\{\\mu_k\\}\\hbox{ is non-increasing and }\\mu_k \\to 0$;\n\\item [(c)] $\\lambda_{\\min}\\g_k\\mu_{k} \\leq 1$ for $k \\geq 0$;\n%\\item [(d)] $\\g_k\\mu_k^{2\\alpha}\\leq \\frac{\\lambda{\\min}}{\\lambda^2(L+\\mu_0)}$ for all $k\\geq 0$;\n\\item [(d)] There exist $K_0$ and $0<\\beta<1$ such that \\begin{align*}\\g_{k-1}\\mu_{k-1}^{2\\alpha-1}\\leq \\g_{k}\\mu_{k}^{2\\alpha-1}(1+\\beta \\lambda_{\\min}\\g_{k}\\mu_{k}),\\quad \\hbox{for all }k \\geq K_0;\\end{align*}\n\\item [(e)] There exists a scalar $\\rho>0$ such that $ \\mu_k^{2-2\\alpha} \\leq \\rho\\g_k$ for all $k \\geq 0$\\fy{.}\n\\end{itemize}\n\\end{assumption}","sha256_utf8":"a2c3d1498e64eefe5e4ab48c5f3225c43b259ecb6f0896dc1b4ccc2d877e3be1"}]},{"id":"S025","semantic_type":"algorithm","kind":"algorithm","title":"正则化参数交替更新","source_units":[{"start":52458,"end":52649,"start_line":722,"end_line":727,"raw_tex":"\\begin{align}\\label{eqn:mu-k}\n  \\begin{cases}\n   \\mu_{k}\\fy{:=}\\mu_{k-1},      &   \\text{if } k \\text{ is odd},\\\\\n    \\mu_{k}<\\mu_{k-1},  &    \\text{otherwise}\\fy{.}\n  \\end{cases}\n\\end{align}","sha256_utf8":"96d0fd3a4d6d5d2d744b1fa1505e82b73aecfb6a2c4db276797b1b54a3700638"},{"start":52650,"end":52723,"start_line":728,"end_line":728,"raw_tex":"Note that we allow for updating the stepsize sequence at each iteration.\n","sha256_utf8":"a320a3ee3504215bc55100b3ea2555942079f12e48807db740569935395efa39"}]},{"id":"S026","semantic_type":"definition","kind":"definition","title":"位移与正则化曲率向量","source_units":[{"start":53252,"end":53469,"start_line":730,"end_line":731,"raw_tex":"\\begin{align}\\label{equ:siyi-LBFGS}&s_{\\lceil k/2\\rceil}:= x_{k}-x_{k-1},\\cr \n&y_{\\lceil k/2\\rceil}:=  \\nabla F(x_{k},\\xi_{k-1}) -\\nabla F(x_{k-1},\\xi_{k-1})  +  \\fys{\\tau}\\mu_k^\\delta s_{\\lceil k/2\\rceil},\\end{align}","sha256_utf8":"0954b6993a778d6cf38c1ef07e6a4a21312a5abe6c874c875ca390a314dfe57c"}]},{"id":"S027","semantic_type":"algorithm","kind":"algorithm","title":"有限记忆逆矩阵递推与初始化","source_units":[{"start":53895,"end":54072,"start_line":732,"end_line":737,"raw_tex":"\\begin{align}\\label{eqn:H-k}H_{k}\\fy{\\triangleq}\n  \\begin{cases}\n    H_{k,m},  &    \\text{if } k \\text{ is odd}, \\\\\n    H_{k-1},  &   \\text{otherwise},\n  \\end{cases}\n\\end{align}","sha256_utf8":"468e68f50f69f38fd281a98a1d67a65d17f393d8ad6c8be360eb6ebe49c2e403"},{"start":54073,"end":54561,"start_line":738,"end_line":741,"raw_tex":"where $m<n$ (in the large scale settings, \\fys{$m\\ll n$}) is \\fy{the memory parameter and represents} the number of pairs \\fy{$(s_i,y_i)$} to be \\fys{stored} to estimate $H_k$. Matrix $H_{k,m}$, for any \\fy{odd $k\\geq 2m-1$}, is updated using the following recursive formula:\n\\begin{align}\\label{eqn:H-k-m}\nH_{k,j}:=\\left(\\mathbf{I}-\\frac{y_is_i^T}{{y_i^Ts_i}}\\right)^TH_{k,j-1}\\left(\\mathbf{I}-\\frac{y_is_i^T}{y_i^Ts_i}\\right)+\\frac{s_is_i^T}{y_i^Ts_i}, \\quad 1 \\leq j\\leq m,\n\\end{align}","sha256_utf8":"5534cb6f0e3a282018836dbcf0805b9ccd9c86051a32f87e3e9a40419e782dd5"},{"start":54562,"end":56079,"start_line":742,"end_line":743,"raw_tex":"where \\fy{$i\\triangleq \\lceil k/2\\rceil-(m-j)$ and we set $H_{k,0}:=\\frac{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}{y_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}\\mathbf{I}$}. Here, at odd iterations, matrix $H_k$ is obtained recursively from $H_{k,0},H_{k,1},\\ldots, H_{k,m-1}$. Note that \\fy{computation of} $H_k$ at an odd $k$ \\fy{needs} $m$ pairs \\fy{of} \\fy{$(s_i,y_i)$}. More precisely, $H_k$ uses the following curvature information: $\\left\\{(s_i,y_i)\\mid i=\\lceil k/2\\rceil-m+1, \\lceil k/2\\rceil-m+2,\\ldots,\\lceil k/2\\rceil\\right\\}.$ \nFor convenience, in the first \\fys{$2m-2$} iterations, we \\fy{let $H_k$ be the identity matrix}. \\fy{This allows for collecting the first set of $m$ pairs $(s_i,y_i)$, where $i=1,2,\\ldots,m$, that is used at iteration $k:=2m-1$ to obtain $H_{2m-1}$}. \\fys{The main differences between update rule \\eqref{eqn:H-k-m} and that of the standard SQN schemes~\\cite{Mokhtari15,nocedal15} are as follows:} (i) The first distinction is with respect to the definition of $y_i$ in \\eqref{equ:siyi-LBFGS}. Here the term $\\fy{\\mu_k^\\delta s_{\\lceil k/2\\rceil}}$ \\vus{compensates} for the lack of strong monotonicity of the gradient mapping and \\vus{aids in} establishing the \\fy{curvature condition}. (ii) Second, instead of obtaining the pair $(s_i,y_i)$ at every iteration, we \\fy{evaluate these terms only at} odd iterations to allow for updating the regularization parameter satisfying \\eqref{eqn:mu-k}. %\\vus{\\bf Farzad -- doesnt the paper by Mokhtari also do odd updates...clarify} \n","sha256_utf8":"de8d22de3fdd9efd0dd655b91a4424a0ac6de61632cbbf4f7d39309139acaf81"}]},{"id":"S028","semantic_type":"assumption","kind":"assumption","title":"样本目标凸性与一致光滑性","source_units":[{"start":66295,"end":66887,"start_line":927,"end_line":935,"raw_tex":"\\begin{assumption}\\label{assum:convex2}  \n\\begin{enumerate}\n\\item [(a)]  \\an{The function $F(x,\\xi)$ is convex over $\\Real^n$ for any $\\xi \\in \\Omega$.}  \n%\\an{The function $f(x)$ is convex on $\\mathbb{R}^n$.} \n\\item[(b)] For any $\\xi \\in \\Omega$, $F(\\cdot,\\xi)$ is continuously differentiable \\us{with Lipschitz\n\tcontinuous gradients} over $\\Real^n$ with parameter $L_\\xi>0$. Moreover, $L:=\\sup_{\\xi \\in \\Omega}L_\\xi < \\infty$. %Also, $\\lim_{x\\to \\infty}f(x)< \\infty$.\n\t\\item[(c)] \\ The optimal solution set $X^*$ of problem \\eqref{eqn:problem} is nonempty. \n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"fd3526afa0fff4590b6c945d2a7ff1b61ba123a3f641793e178f537ae689b661"}]},{"id":"S029","semantic_type":"algorithm","kind":"algorithm","title":"IRS-LBFGS完整算法与参数","source_units":[{"start":72294,"end":74681,"start_line":999,"end_line":1041,"raw_tex":"\\begin{algorithm}\n  \\caption{Iteratively Regularized Stochastic Limited-memory BFGS}\n\\label{algorithm:IR-S-BFGS}\n%\\begin{boxedminipage}{165mm}\n    \\begin{algorithmic}[1]\n    \\State\\textbf{Input:} LBFGS memory parameter $m\\geq 1$, Lipschitzian parameter $L>0$, random initial point $x_0 \\in \\Real^n$, initial stepsize $\\g_0>0$, and initial regularization parameter $\\mu_0>0$ such that $\\g_0\\mu_0 \\leq  (m+n) L$, \\fys{scalars $0<\\epsilon < \\frac{1}{3}$, $\\delta \\in \\left(0,\\frac{1.5\\epsilon}{n+m}\\right)$, and $\\tau>0$}; \n    \\State Set $a:=\\frac{2}{3}-\\epsilon+\\frac{2\\delta(n+m)}{3}$, $b:=\\frac{1}{3}$;\n    \\For {$k=0,1,\\ldots,$}\n    \t\t\\State Compute $\\g_k :=\\frac{\\g_0}{(k+1)^a}$ and $\\mu_k:=\\frac{\\mu_02^b}{\\left(k+1+\\text{mod}(k+1,2)\\right)^b}$;\n        \\State  {Evaluate the stochastic gradient $\\nabla F(x_k,\\xi_k)$;}\n     \\If {mod$(k,2)=1$}\n    \\State  Compute index $i:=\\lceil k/2\\rceil$;\n     \\State   Compute vector $s_i:=x_k-x_{k-1}$; \n     \\State   Compute vector  $y_i:= \\nabla F(x_k,\\xi_{k-1})-\\nabla F(x_{k-1},\\xi_{k-1})+\\fys{\\tau}\\mu_k^\\delta s_i$;\n      \\If {$k> 2m$}\n      \\State Discard the vector pair $\\{s_{i-m},y_{i-m}\\}$ from storage;\n      \\EndIf\n   \\EndIf\n \\If {$k < 2m-1$}\n   \\State Update solution iterate $x_{k+1}:=x_k-\\g_k\\left(\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)\\right)$;\n     \\Else\n     \\State Initialize Hessian inverse $H_{k,0}:=\\frac{s_i^Ty_i}{y_i^Ty_i}\\mathbf{I}$;\\hspace{.4in} \\tikzmark{right}\\tikzmark{top}\n     \\State Initialize $q:=\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)$;\n     \\For {$t=i:i-m+1$}\n     \\State Compute scalar $\\alpha_{i-t+1}:=\\frac{s_t^Tq}{s_t^Ty_t}$;\n     \\State Update vector $q:=q-\\alpha_{i-t+1}y_t$;\n     \\EndFor\n     \\State Initialize vector $r:=H_{k,0}q$;\n     \\For {$t=i-m+1:i$}\n     %\\State Compute scalar $\\beta:=\\frac{y_t^Tr}{s_t^Ty_t}$;\n     \\State  Update vector $r:=r+\\left(\\alpha_{i-t+1}-\\frac{y_t^Tr}{s_t^Ty_t}\\right)s_t$;\n     \\EndFor\\tikzmark{bottom}\n%\\State Given $\\mathcal{M}=\\{(s_t,y_t)\\}_{t=i-m+1}^i$, call Alg. \\ref{algorithm:Hk-update} to compute the following vector\\[G_k \\triangleq H_k\\left(\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)\\right);\\]\n             \\State Update solution iterate $x_{k+1}:=x_k-\\g_kr$;\\Comment{\\footnotesize{LBFGS update}} \n   \n\\EndIf\n    \n    \\EndFor\n   \\end{algorithmic}\n   \\AddNote{top}{bottom}{right}{\\footnotesize{LBFGS two-loop recursion}}\n%\\end{boxedminipage}\n% \\vskip17.5pt\n\\end{algorithm}","sha256_utf8":"50a1bd5bf3f89d068ac150fe333331b38e144935913979873a3e6be8b1ed0b76"}]},{"id":"S030","semantic_type":"proposition","kind":"proposition","title":"正则化目标的基本性质","source_units":[{"start":28928,"end":29401,"start_line":478,"end_line":485,"raw_tex":"\\begin{property}[{Properties of a regularized function}]\\label{proper:propsfk} We have:\n\\begin{itemize}\n\\item [(a)] $f_k$ is strongly convex with a parameter $\\mu_k$.\n\\item [(b)] $f_k$ has Lipschitzian gradients with parameter $L+\\mu_k$.\n \\item [(c)] $f_k$ has a unique minimizer over $\\Real^n$, denoted by $x^*_k$. Moreover, for any $x \\in \\Real^n$, \n \\[2\\mu_k (f_k(x)-f_k(x^*_k)) \\leq \\|\\nabla f_k(x)\\|^2\\leq 2(L+\\mu_k) (f_k(x)-f_k(x^*_k)).\\]\n\\end{itemize}\n\\end{property}","sha256_utf8":"1fbe185d25b4d5fbe356eb3fd281d718c700559db02b4f8630e24eae31148b32"}]},{"id":"S031","semantic_type":"proof","kind":"proof","title":"正则化性质的作者说明","source_units":[{"start":29402,"end":29797,"start_line":486,"end_line":490,"raw_tex":"The existence and uniqueness of $x^*_k$ in Property~\\ref{proper:propsfk}(c) \nis due to the strong convexity of the function $f_k$ (see, for example, Section 1.3.2 in~\\cite{Polyak87}), \nwhile the relation for the gradient is known to hold for a \nstrongly convex function with a parameter $\\mu$ that also has Lipschitz gradients with a parameter \n$L$ (see Lemma 1 on page 23 in~\\cite{Polyak87}). \n","sha256_utf8":"610458a3c91fc08b7f81b12f898399eda233827ee7d8d873b4f7bbaa61f25acf"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"由强凸性保证唯一极小点","source_units":[{"start":29402,"end":29587,"start_line":486,"end_line":487,"raw_tex":"The existence and uniqueness of $x^*_k$ in Property~\\ref{proper:propsfk}(c) \nis due to the strong convexity of the function $f_k$ (see, for example, Section 1.3.2 in~\\cite{Polyak87}), \n","sha256_utf8":"36866eb8837dda118a0b87a972b4ec316372bcf040f9ad3da376bccceab951d1"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"引用强凸光滑梯度界","source_units":[{"start":29587,"end":29797,"start_line":488,"end_line":490,"raw_tex":"while the relation for the gradient is known to hold for a \nstrongly convex function with a parameter $\\mu$ that also has Lipschitz gradients with a parameter \n$L$ (see Lemma 1 on page 23 in~\\cite{Polyak87}). \n","sha256_utf8":"a5bfb6ed2b15364b710f245278dd2194f4446047b87c26481d3117cebb716b5e"}]},{"id":"S034","semantic_type":"lemma","kind":"lemma","title":"正则化目标误差递推界","source_units":[{"start":31037,"end":31777,"start_line":511,"end_line":517,"raw_tex":"\\begin{lemma}[A recursive error bound]\\label{lemma:main-ineq}\nConsider the \\eqref{eqn:LM-cyclic-reg-BFGS} method and suppose Assumptions \\ref{assum:convex}, \\ref{assum:main}, and \\ref{assump:Hk} hold. Also, assume $\\mu_k$ is a non-increasing sequence and let \\begin{align}\\label{mainLemmaCond}\\g_k\\mu_k^{2\\alpha} \\leq \\frac{\\lambda_{\\min}}{\\lambda^2(L+\\mu_0)},\\quad \\hbox{for all }k\\geq 0.\n\\end{align}Then, the following inequality holds \\fy{for all $k\\geq 0$}:\n\\begin{align}\\label{ineq:cond-recursive-F-k}\n\\EXP{f_{k+1}(x_{k+1})\\mid\\sF_k}-f^* &\\leq (1-\\lambda_{\\min}\\mu_k\\g_k)(f_k(x_k)-f^*)+\\frac{\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)}{2}\\mu_k^2\\g_k\\notag \\\\ &+\\frac{ (L+\\mu_k)\\lambda^2\\nu^2}{2}\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align}\n\\end{lemma}","sha256_utf8":"34e885e66031037195a15dcd1b436bcdef0e56b1c74ca223f150fd4454852e62"}]},{"id":"S035","semantic_type":"proof","kind":"proof","title":"误差递推界完整证明","source_units":[{"start":31778,"end":36955,"start_line":518,"end_line":567,"raw_tex":"\\begin{proof}\nThe Lipschitzian property of $\\nabla f_k$ and  \nthe update rule \\eqref{eqn:LM-cyclic-reg-BFGS} imply that \n\\begin{align*}\nf_k(x_{k+1}) &\\leq f_k(x_k)+\\nabla f_k(x_k)^T(x_{k+1}-x_k)+\\frac{ (L+\\mu_k)}{2}\\|x_{k+1}-x_k\\|^2 \\cr\n&= f_k(x_k)-\\g_k\\nabla f_k(x_k)^TH_k\\left(\\nabla F(x_k,\\xi_k) +\\mu_k(x_k-x_0)\\right)\\\\&+ \\frac{ (L+\\mu_k)}{2}\\g_k^2\\|H_k\\left(\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)\\right)\\|^2.\\end{align*}\n\\fy{Invoking the definition of the stochastic error $w_k$ in} \\eqref{def:wk} \\fy{and} Definition~\\ref{def:regularizedF}, we obtain\n\\begin{align}\\label{ineq:term1-2}\nf_k(x_{k+1}) &\\leq f_k(x_k)-\\g_k\\nabla f_k(x_k)^TH_k(\\nabla f(x_k)+w_k +\\mu_k(x_k-x_0))\\notag\\\\&+ \\frac{ (L+\\mu_k)}{2}\\g_k^2\\|H_k(\\nabla f(x_k)+w_k+\\mu_k(x_k-x_0))\\|^2\\\\ \n\\notag = f_k(x_k)&-\\g_k\\underbrace{\\nabla f_k(x_k)^TH_k(\\nabla f_k(x_k)+w_k)}_{\\tiny\\hbox{Term } 1}+ \\frac{ (L+\\mu_k)}{2}\\g_k^2\\underbrace{\\|H_k(\\nabla f_k(x_k)+w_k)\\|^2}_{\\tiny\\hbox{ Term } 2},\\end{align}\nwhere in the last \\fy{equation,} we used the definition of $f_k$. Next, we estimate the conditional expectation of Term 1 and 2. From Assumption \\ref{assump:Hk}, we have \n\\begin{align*}\n\\hbox{Term }1 &= \\nabla f_k(x_k)^TH_k\\nabla f_k(x_k)+\\nabla f_k(x_k)^TH_kw_k \\\\&\\geq \\lambda_{\\min}\\|\\nabla f_k(x_k)\\|^2+\\nabla f_k(x_k)^TH_kw_k.\n\\end{align*}\nTaking expectations conditioned on $\\sF_k$, from the preceding inequality, we obtain\n\\begin{align}\\label{equ:Term1}\n\\EXP{\\hbox{Term } 1\\mid\\sF_k} &\\geq \\lambda_{\\min}\\|\\nabla f_k(x_k)\\|^2+\\EXP{\\nabla f_k(x_k)^TH_kw_k\\mid\\sF_k}\\\\ \\notag&=\\lambda_{\\min}\\|\\nabla f_k(x_k)\\|^2+\\nabla f_k(x_k)^TH_k\\EXP{w_k\\mid\\sF_k} =\\lambda_{\\min}\\|\\nabla f_k(x_k)\\|^2,\\end{align}\nwhere we recall that $\\EXP{w_k\\mid\\sF_k}=0$ and $\\EXP{H_k\\mid\\sF_k}=H_k$. Similarly, in Term 2, invoking Assumption \\ref{assump:Hk}(b), we may write\n\\begin{align*}\n\\hbox{Term } 2&= (\\nabla f_k(x_k)+w_k)^TH_k^2(\\nabla f_k(x_k)+w_k)\\leq (\\lambda\\mu^{\\alpha}_k)^2\\|\\nabla f_k(x_k)+w_k\\|^2 \\\\ &=\\lambda^2\\mu^{2\\alpha}_k\\left(\\|\\nabla f_k(x_k)\\|^2+\\|w_k\\|^2+2\\nabla f_k(x_k)^Tw_k\\right).\\end{align*}\nTaking conditional expectations from the preceding inequality, and using Assumption \\ref{assum:main}, we obtain\n\\begin{align}\\label{equ:Term2}\n\\EXP{\\hbox{Term } 2\\mid\\sF_k}&\\leq\\lambda^2\\mu^{2\\alpha}_k\\left(\\|\\nabla f_k(x_k)\\|^2+\\EXP{\\|w_k\\|^2\\mid\\sF_k}+2\\nabla f_k(x_k)^T\\EXP{w_k\\mid\\sF_k}\\right)\\notag\\\\ & \\leq \\lambda^2\\mu^{2\\alpha}_k\\left(\\|\\nabla f_k(x_k)\\|^2+\\nu^2\\right).\\end{align}\nNext, taking conditional expectations in \\eqref{ineq:term1-2}, and using \\eqref{equ:Term1} and \\eqref{equ:Term2}, we obtain\n\\begin{align*}%\\label{ineq:lemma2rel7}\n&\\EXP{f_k(x_{k+1})\\mid\\sF_k} \\leq f_k(x_k)-\\g_k\\lambda_{\\min}\\|\\nabla f_k(x_k)\\|^2\\notag \\\\\n&+\\lambda^2\\mu^{2\\alpha}_k\\frac{ (L+\\mu_k)}{2}\\g_k^2\\left(\\|\\nabla f_k(x_k)\\|^2+\\nu^2\\right)\\notag \\\\\n&\\leq f_k(x_k)-\\frac{\\g_k\\lambda_{\\min}}{2}\\|\\nabla f_k(x_k)\\|^2\\left(2-\\frac{\\lambda^2\\mu_k^{2\\alpha}\\g_k(L+\\mu_k)}{\\lambda_{\\min}}\\right)+\\lambda^2\\mu^{2\\alpha}_k\\frac{ (L+\\mu_k)}{2}\\g_k^2\\nu^2.\n\\end{align*}\nFrom the assumption that $\\g_k$ and $\\mu_k$ satisfy $\\g_k\\mu^{2\\alpha}_k \\leq \\frac{\\lambda_{\\min}}{\\lambda^2(L+\\mu_0)}$ for any $k \\geq 0$ and that $\\mu_k$ is non-increasing, we have $\\g_k\\mu^{2\\alpha}_k \\leq \\frac{\\lambda_{\\min}}{\\lambda^2(L+\\mu_k)}$. As a consequence, we get $2-\\frac{\\lambda^2\\mu_k^{2\\alpha}\\g_k(L+\\mu_k)}{\\lambda_{\\min}}\\geq 1$.  Therefore, from \\fy{the} preceding inequality, we obtain\n\\begin{align*}\n\\EXP{f_k(x_{k+1}) \\mid\\sF_k}&\\leq f_k(x_k)-\\frac{\\g_k\\lambda_{\\min}}{2}\\|\\nabla f_k(x_k)\\|^2+\\lambda^2\\mu^{2\\alpha}_k\\frac{ (L+\\mu_k)}{2}\\g_k^2\\nu^2.\n\\end{align*}\nEmploying Property \\ref{proper:propsfk}(c), we have\n\\begin{align*}\n\\EXP{f_k(x_{k+1})\\mid\\sF_k} &\\leq f_k(x_k)-\\lambda_{\\min}\\mu_k\\g_k(f_k(x_k)-f_k(x^*_k))+\\lambda^2\\mu^{2\\alpha}_k\\frac{ (L+\\mu_k)}{2}\\g_k^2\\nu^2.\n\\end{align*}\nNote that, since $\\mu_k$ is a non-increasing sequence, Definition \\ref{def:regularizedF} implies that \\[\\EXP{f_{k+1}(x_{k+1})\\mid \\sF_k} \\leq\\EXP{f_{k}(x_{k+1})\\mid \\sF_k}.\\] Therefore, we obtain \n\\begin{align}\\label{ineq:lemmaLastIneq}\n\\EXP{f_{k+1}(x_{k+1})\\mid\\sF_k} &\\leq f_k(x_k)-\\lambda_{\\min}\\mu_k\\g_k(\\underbrace{f_k(x_k)-f_k(x^*_k)}_{\\tiny{\\hbox{Term } 3}})+\\lambda^2\\mu^{2\\alpha}_k\\frac{ (L+\\mu_k)}{2}\\g_k^2\\nu^2.\n\\end{align}\nNext, we derive a lower bound for Term 3. Since $x_k^*$ is the unique minimizer of $f_k$, we have $f_k(x_k^*) \\leq f_k(x^*)$. Therefore, invoking Definition \\ref{def:regularizedF}, for an arbitrary optimal solution $x^* \\in X^*$, we have \n\\[f_k(x_k)-f_k(x^*_k)\\geq f_k(x_k)-f_k(x^*)=f_k(x_k)-f^*-\\frac{\\mu_k}{2}\\|x^*-x_0\\|^2.\\]\nFrom the preceding relation and \\eqref{ineq:lemmaLastIneq}, we have\n\\begin{align*}\n\\EXP{f_{k+1}(x_{k+1})\\mid\\sF_k} &\\leq f_k(x_k)-\\lambda_{\\min}\\mu_k\\g_k(f_k(x_k)-f^*)+\\frac{\\lambda_{\\min}\\|x^*-x_0\\|^2}{2}\\mu_k^2\\g_k\\\\\n&+\\frac{(L+\\mu_k)\\lambda^2\\nu^2}{2}\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align*}\nSince $x^*$ is an arbitrary optimal solution, taking minimum from the right-hand side of the preceding inequality over $X^*$, we can replace $\\|x^*-x_0\\|$ by $\\mbox{dist}(x_0,X^*)$. Then, subtracting $f^*$ from both sides of the resulting relation yields the desired inequality.\n\\end{proof}","sha256_utf8":"f878941038b56b4603488a649a12acd7eac1b9bfddc2b50ca28e4a1ffe336f8b"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"光滑性给出单步下降界","source_units":[{"start":31778,"end":32200,"start_line":518,"end_line":523,"raw_tex":"\\begin{proof}\nThe Lipschitzian property of $\\nabla f_k$ and  \nthe update rule \\eqref{eqn:LM-cyclic-reg-BFGS} imply that \n\\begin{align*}\nf_k(x_{k+1}) &\\leq f_k(x_k)+\\nabla f_k(x_k)^T(x_{k+1}-x_k)+\\frac{ (L+\\mu_k)}{2}\\|x_{k+1}-x_k\\|^2 \\cr\n&= f_k(x_k)-\\g_k\\nabla f_k(x_k)^TH_k\\left(\\nabla F(x_k,\\xi_k) +\\mu_k(x_k-x_0)\\right)\\\\&+ \\frac{ (L+\\mu_k)}{2}\\g_k^2\\|H_k\\left(\\nabla F(x_k,\\xi_k)+\\mu_k(x_k-x_0)\\right)\\|^2.\\end{align*}\n","sha256_utf8":"bb322f79a425d64497c61dfa4964b5a4bde844ff8a84f321e1368ea7896eb1f7"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"分解梯度项与噪声项","source_units":[{"start":32200,"end":32913,"start_line":524,"end_line":528,"raw_tex":"\\fy{Invoking the definition of the stochastic error $w_k$ in} \\eqref{def:wk} \\fy{and} Definition~\\ref{def:regularizedF}, we obtain\n\\begin{align}\\label{ineq:term1-2}\nf_k(x_{k+1}) &\\leq f_k(x_k)-\\g_k\\nabla f_k(x_k)^TH_k(\\nabla f(x_k)+w_k +\\mu_k(x_k-x_0))\\notag\\\\&+ \\frac{ (L+\\mu_k)}{2}\\g_k^2\\|H_k(\\nabla f(x_k)+w_k+\\mu_k(x_k-x_0))\\|^2\\\\ \n\\notag = f_k(x_k)&-\\g_k\\underbrace{\\nabla f_k(x_k)^TH_k(\\nabla f_k(x_k)+w_k)}_{\\tiny\\hbox{Term } 1}+ \\frac{ (L+\\mu_k)}{2}\\g_k^2\\underbrace{\\|H_k(\\nabla f_k(x_k)+w_k)\\|^2}_{\\tiny\\hbox{ Term } 2},\\end{align}\nwhere in the last \\fy{equation,} we used the definition of $f_k$. Next, we estimate the conditional expectation of Term 1 and 2. From Assumption \\ref{assump:Hk}, we have \n","sha256_utf8":"7b18741a86a3c54d9b19cf0dbebe1d9017a4d96afc10dad1f20d1fa7064aa98e"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"利用最小特征值控制线性项","source_units":[{"start":32913,"end":33087,"start_line":529,"end_line":531,"raw_tex":"\\begin{align*}\n\\hbox{Term }1 &= \\nabla f_k(x_k)^TH_k\\nabla f_k(x_k)+\\nabla f_k(x_k)^TH_kw_k \\\\&\\geq \\lambda_{\\min}\\|\\nabla f_k(x_k)\\|^2+\\nabla f_k(x_k)^TH_kw_k.\n\\end{align*}\n","sha256_utf8":"2577baad1d702056e992db580099dcf86606a4cfdd76f50976fd6e6705d0f0eb"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"条件无偏性消去交叉项","source_units":[{"start":33087,"end":33448,"start_line":532,"end_line":534,"raw_tex":"Taking expectations conditioned on $\\sF_k$, from the preceding inequality, we obtain\n\\begin{align}\\label{equ:Term1}\n\\EXP{\\hbox{Term } 1\\mid\\sF_k} &\\geq \\lambda_{\\min}\\|\\nabla f_k(x_k)\\|^2+\\EXP{\\nabla f_k(x_k)^TH_kw_k\\mid\\sF_k}\\\\ \\notag&=\\lambda_{\\min}\\|\\nabla f_k(x_k)\\|^2+\\nabla f_k(x_k)^TH_k\\EXP{w_k\\mid\\sF_k} =\\lambda_{\\min}\\|\\nabla f_k(x_k)\\|^2,\\end{align}\n","sha256_utf8":"45598626b6a3f01464fe46bba6d47a3085f137437492b49b76b8699dae45a685"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"利用最大特征值控制平方项","source_units":[{"start":33448,"end":33843,"start_line":535,"end_line":537,"raw_tex":"where we recall that $\\EXP{w_k\\mid\\sF_k}=0$ and $\\EXP{H_k\\mid\\sF_k}=H_k$. Similarly, in Term 2, invoking Assumption \\ref{assump:Hk}(b), we may write\n\\begin{align*}\n\\hbox{Term } 2&= (\\nabla f_k(x_k)+w_k)^TH_k^2(\\nabla f_k(x_k)+w_k)\\leq (\\lambda\\mu^{\\alpha}_k)^2\\|\\nabla f_k(x_k)+w_k\\|^2 \\\\ &=\\lambda^2\\mu^{2\\alpha}_k\\left(\\|\\nabla f_k(x_k)\\|^2+\\|w_k\\|^2+2\\nabla f_k(x_k)^Tw_k\\right).\\end{align*}\n","sha256_utf8":"ebbcfd047a68dd10161777ab98c6547dfa99903cc05974496a67a7a13bc4905d"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"有界方差控制平方项期望","source_units":[{"start":33843,"end":34233,"start_line":538,"end_line":540,"raw_tex":"Taking conditional expectations from the preceding inequality, and using Assumption \\ref{assum:main}, we obtain\n\\begin{align}\\label{equ:Term2}\n\\EXP{\\hbox{Term } 2\\mid\\sF_k}&\\leq\\lambda^2\\mu^{2\\alpha}_k\\left(\\|\\nabla f_k(x_k)\\|^2+\\EXP{\\|w_k\\|^2\\mid\\sF_k}+2\\nabla f_k(x_k)^T\\EXP{w_k\\mid\\sF_k}\\right)\\notag\\\\ & \\leq \\lambda^2\\mu^{2\\alpha}_k\\left(\\|\\nabla f_k(x_k)\\|^2+\\nu^2\\right).\\end{align}\n","sha256_utf8":"59dfbc2b55761cf802bf2e8af575d70624e0fe081349ed3fda5fba00b8a5c2d8"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"合并两个条件期望估计","source_units":[{"start":34233,"end":34800,"start_line":541,"end_line":546,"raw_tex":"Next, taking conditional expectations in \\eqref{ineq:term1-2}, and using \\eqref{equ:Term1} and \\eqref{equ:Term2}, we obtain\n\\begin{align*}%\\label{ineq:lemma2rel7}\n&\\EXP{f_k(x_{k+1})\\mid\\sF_k} \\leq f_k(x_k)-\\g_k\\lambda_{\\min}\\|\\nabla f_k(x_k)\\|^2\\notag \\\\\n&+\\lambda^2\\mu^{2\\alpha}_k\\frac{ (L+\\mu_k)}{2}\\g_k^2\\left(\\|\\nabla f_k(x_k)\\|^2+\\nu^2\\right)\\notag \\\\\n&\\leq f_k(x_k)-\\frac{\\g_k\\lambda_{\\min}}{2}\\|\\nabla f_k(x_k)\\|^2\\left(2-\\frac{\\lambda^2\\mu_k^{2\\alpha}\\g_k(L+\\mu_k)}{\\lambda_{\\min}}\\right)+\\lambda^2\\mu^{2\\alpha}_k\\frac{ (L+\\mu_k)}{2}\\g_k^2\\nu^2.\n\\end{align*}\n","sha256_utf8":"f7dac60e347969dfbc9d38804ab8178970dadd6ab999e2731208b85a2261e746"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"步长限制保证有效下降","source_units":[{"start":34800,"end":35387,"start_line":547,"end_line":550,"raw_tex":"From the assumption that $\\g_k$ and $\\mu_k$ satisfy $\\g_k\\mu^{2\\alpha}_k \\leq \\frac{\\lambda_{\\min}}{\\lambda^2(L+\\mu_0)}$ for any $k \\geq 0$ and that $\\mu_k$ is non-increasing, we have $\\g_k\\mu^{2\\alpha}_k \\leq \\frac{\\lambda_{\\min}}{\\lambda^2(L+\\mu_k)}$. As a consequence, we get $2-\\frac{\\lambda^2\\mu_k^{2\\alpha}\\g_k(L+\\mu_k)}{\\lambda_{\\min}}\\geq 1$.  Therefore, from \\fy{the} preceding inequality, we obtain\n\\begin{align*}\n\\EXP{f_k(x_{k+1}) \\mid\\sF_k}&\\leq f_k(x_k)-\\frac{\\g_k\\lambda_{\\min}}{2}\\|\\nabla f_k(x_k)\\|^2+\\lambda^2\\mu^{2\\alpha}_k\\frac{ (L+\\mu_k)}{2}\\g_k^2\\nu^2.\n\\end{align*}\n","sha256_utf8":"e85e42493e96739642775dbf0708ad1c82c95dcd94561101f6593e1a5e49d19e"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"将梯度范数换成正则化误差","source_units":[{"start":35387,"end":35612,"start_line":551,"end_line":554,"raw_tex":"Employing Property \\ref{proper:propsfk}(c), we have\n\\begin{align*}\n\\EXP{f_k(x_{k+1})\\mid\\sF_k} &\\leq f_k(x_k)-\\lambda_{\\min}\\mu_k\\g_k(f_k(x_k)-f_k(x^*_k))+\\lambda^2\\mu^{2\\alpha}_k\\frac{ (L+\\mu_k)}{2}\\g_k^2\\nu^2.\n\\end{align*}\n","sha256_utf8":"658c9b5793a313ee77a9385ec685fdf114a3ac07d2c47baaa5177aca9000258d"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"利用正则化参数单调性","source_units":[{"start":35612,"end":36047,"start_line":555,"end_line":558,"raw_tex":"Note that, since $\\mu_k$ is a non-increasing sequence, Definition \\ref{def:regularizedF} implies that \\[\\EXP{f_{k+1}(x_{k+1})\\mid \\sF_k} \\leq\\EXP{f_{k}(x_{k+1})\\mid \\sF_k}.\\] Therefore, we obtain \n\\begin{align}\\label{ineq:lemmaLastIneq}\n\\EXP{f_{k+1}(x_{k+1})\\mid\\sF_k} &\\leq f_k(x_k)-\\lambda_{\\min}\\mu_k\\g_k(\\underbrace{f_k(x_k)-f_k(x^*_k)}_{\\tiny{\\hbox{Term } 3}})+\\lambda^2\\mu^{2\\alpha}_k\\frac{ (L+\\mu_k)}{2}\\g_k^2\\nu^2.\n\\end{align}\n","sha256_utf8":"2849ec9f1e041541fbb1d9592ec92c8a70618d7b69948f0bd3399086a163cd8e"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"比较原问题与正则化极小点","source_units":[{"start":36047,"end":36375,"start_line":559,"end_line":560,"raw_tex":"Next, we derive a lower bound for Term 3. Since $x_k^*$ is the unique minimizer of $f_k$, we have $f_k(x_k^*) \\leq f_k(x^*)$. Therefore, invoking Definition \\ref{def:regularizedF}, for an arbitrary optimal solution $x^* \\in X^*$, we have \n\\[f_k(x_k)-f_k(x^*_k)\\geq f_k(x_k)-f_k(x^*)=f_k(x_k)-f^*-\\frac{\\mu_k}{2}\\|x^*-x_0\\|^2.\\]\n","sha256_utf8":"a13e0413787ef3ed2bb84bcb4ec4cb4677a3d55f8843b1f47a3ddc56f220bcd8"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"对最优解取最小距离并完成递推","source_units":[{"start":36375,"end":36955,"start_line":561,"end_line":567,"raw_tex":"From the preceding relation and \\eqref{ineq:lemmaLastIneq}, we have\n\\begin{align*}\n\\EXP{f_{k+1}(x_{k+1})\\mid\\sF_k} &\\leq f_k(x_k)-\\lambda_{\\min}\\mu_k\\g_k(f_k(x_k)-f^*)+\\frac{\\lambda_{\\min}\\|x^*-x_0\\|^2}{2}\\mu_k^2\\g_k\\\\\n&+\\frac{(L+\\mu_k)\\lambda^2\\nu^2}{2}\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align*}\nSince $x^*$ is an arbitrary optimal solution, taking minimum from the right-hand side of the preceding inequality over $X^*$, we can replace $\\|x^*-x_0\\|$ by $\\mbox{dist}(x_0,X^*)$. Then, subtracting $f^*$ from both sides of the resulting relation yields the desired inequality.\n\\end{proof}","sha256_utf8":"003d1487f45f7c7486a0abb39def3ec46aa94e94b890111632c755e3280d2045"}]},{"id":"S048","semantic_type":"proposition","kind":"proposition","title":"均值收敛及一般误差率","source_units":[{"start":42845,"end":43605,"start_line":639,"end_line":649,"raw_tex":"\\begin{proposition}[Convergence in mean]\\label{prop:mean}\nConsider the \\eqref{eqn:LM-cyclic-reg-BFGS} scheme. \nSuppose Assumptions~\\ref{assum:convex},\n\\ref{assum:main}, \\ref{assump:Hk}, and \\ref{assum:sequences-ms-convergence} hold. \nThen, there exists $K\\geq 1$ such that:\\begin{align}\\label{ineq:bound}\n \\EXP{f(x_{k+1})}-f^*\\leq \\theta\\g_k\\mu_k^{2\\alpha-1}, \\quad \\hbox{for all } k \\geq K,\n \\end{align}\n  where $f^*$ is the optimal value of problem \\fy{and}\n\\begin{align}\\label{def:theta}\\theta := \\max \\bigg\\{\\frac{\\EXP{f_{K+1}(x_{K+1})}-f^*}{\\g_K\\mu_K^{2\\alpha-1}},\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2\\fy{\\lambda_{\\min}}(1-\\beta)}\\bigg\\}.\\end{align}\n Moreover, $\\lim_{k\\to \\infty}\\EXP{f(x_{k})}=f^*.$\n\\end{proposition}","sha256_utf8":"6f219aa48bd2fd4d609c5618b830760c2f1fa0562fa557ae489d87398faa972c"}]},{"id":"S049","semantic_type":"proof","kind":"proof","title":"均值误差率完整证明","source_units":[{"start":43606,"end":48107,"start_line":650,"end_line":689,"raw_tex":"\\begin{proof}\nNote that Assumption \\ref{assum:sequences-ms-convergence}(a,b) imply that \\eqref{ineq:cond-recursive-F-k} holds for a large enough $k$, say after $\\hat K$. Then, since the conditions of Lemma \\ref{lemma:main-ineq} are met \\fy{(with an index shift)}, taking expectations on both sides of \\eqref{ineq:cond-recursive-F-k}, we obtain for any $k\\geq \\hat K$:\n\\begin{align*}\n\\EXP{f_{k+1}(x_{k+1})-f^*} &\\leq (1-\\lambda_{\\min}\\mu_k\\g_k)\\EXP{f_k(x_k)-f^*}+\\frac{\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)}{2}\\mu_k^2\\g_k\\\\ &+\\frac{ (L+\\mu_0)\\lambda^2\\nu^2}{2}\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align*}\n%{\\bf Comment: the relation in (9) has different coefficients than originally - edits needed from here onward}\n Using Assumption \\ref{assum:sequences-ms-convergence}(e), we have $\\mu_k^2\\g_k\\leq \\rho \\g_k^2\\mu_k^{2\\alpha}$. Thus, we obtain\n\\begin{align}\\label{ineq:cond-recursive-F-k-expected2}\n\\EXP{f_{k+1}(x_{k+1})-f^*} &\\leq (1-\\lambda_{\\min}\\mu_k\\g_k)\\EXP{f_k(x_k)-f^*}\\notag\\\\ &+\\left(\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2}\\right)\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align}\nLet us define \\fy{$K\\triangleq\\max\\{\\hat K, K_0\\}$, where $K_0$ is from Assumption \\ref{assum:sequences-ms-convergence}(d)}. Using the preceding relation and by induction on $k$, we show the desired result. To show \\eqref{ineq:bound}, we show the following relation first: \n\\begin{align}\\label{ineq:bound_v2}\n \\EXP{f_{k+1}(x_{k+1})}-f^*\\leq \\theta\\g_k\\mu_k^{2\\alpha-1}, \\quad \\hbox{for all } k \\geq K,\n \\end{align}\n Note that \\eqref{ineq:bound_v2} \\fy{implies} the relation \\eqref{ineq:bound} since we have $ \\EXP{f(x_{k+1})}\\leq  \\EXP{f_{k+1}(x_{k+1})}$.\nFirst, we show that \\eqref{ineq:bound_v2} holds for \\fy{$k=K$}.  Consider the term $\\EXP{f_{K+1}(x_{K+1})}-f^*$. Multiplying and dividing by $\\g_K\\mu_K^{2\\alpha-1}$, we obtain\n\\begin{align*}\\EXP{f_{K+1}(x_{K+1})}-f^*=\\left(\\frac{\\EXP{f_{K+1}(x_{K+1})}-f^*}{\\g_k\\mu_K^{2\\alpha-1}}\\right)\\g_K\\mu_K^{2\\alpha-1}\n \\leq \\theta \\g_K\\mu_K^{2\\alpha-1},\\end{align*}\nwhere the last inequality is obtained by invoking the definition of $\\theta$ in \\eqref{def:theta}. This implies that \\eqref{ineq:bound_v2} holds for $k=K$. Now assume that  \\eqref{ineq:bound_v2} holds for some $k \\geq K$. We show that it also holds for $k+1$.  From the induction hypothesis and \\eqref{ineq:cond-recursive-F-k-expected2} we have\n\\begin{align*}\n\\EXP{f_{k+1}(x_{k+1})-f^*} &\\leq (1-\\lambda_{\\min}\\mu_k\\g_k)\\theta\\g_{k-1}\\mu_{k-1}^{2\\alpha-1}\\\\ &+\\left(\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2}\\right)\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align*}\nUsing Assumption \\ref{assum:sequences-ms-convergence}(d) we obtain\n\\begin{align}\\label{ineq:thm2-rel1}\n\\EXP{f_{k+1}(x_{k+1})-f^*} &\\leq \\theta\\g_{k}\\mu_{k}^{2\\alpha-1}(1-\\lambda_{\\min}\\mu_k\\g_k)(1+\\beta \\lambda_{\\min}\\g_{k}\\mu_{k})\\notag \\\\ &+\\left(\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2}\\right)\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align}\nNext we find an upper bound for the term $(1-\\lambda_{\\min}\\mu_k\\g_k)(1+\\beta \\lambda_{\\min}\\g_{k}\\mu_{k}) $ as follows\n\\begin{align*}\n(1-\\lambda_{\\min}\\mu_k\\g_k)(1+\\beta \\lambda_{\\min}\\g_{k}\\mu_{k}) &= 1-\\lambda_{\\min}\\mu_k\\g_k+\\beta\\lambda_{\\min}\\mu_k\\g_k-\\fy{\\beta\\lambda_{\\min}^2\\mu_k^2\\g_k^2}\\\\ & \\leq 1-(1-\\beta)\\lambda_{\\min}\\mu_k\\g_k.\n\\end{align*}\nCombining this relation with \\eqref{ineq:thm2-rel1}, it follows \\fy{that}\n\\begin{align*}%\\label{ineq:thm2-rel1}\n&\\EXP{f_{k+1}(x_{k+1})-f^*} \\leq \\theta\\g_{k}\\mu_{k}^{2\\alpha-1} -\\theta\\fy{\\lambda_{\\min}}(1-\\beta)\\mu_k^{2\\alpha}\\g_k^2  \\\\ &+\\left(\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2}\\right)\\mu^{2\\alpha}_k\\g_k^2\\\\ & =\\theta\\g_{k}\\mu_{k}^{2\\alpha-1}-\\left(\\underbrace{\\theta\\fy{\\lambda_{min}}(1-\\beta)-\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2}}_{\\tiny{\\hbox{Term }1}}\\right)\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align*}\nNote that the definition of $\\theta$ in \\eqref{def:theta} implies that Term 1 is nonnegative. Therefore, \n\\begin{align*}%\\label{ineq:thm2-rel1}\n\\EXP{f_{k+1}(x_{k+1})-f^*} &\\leq \\theta\\g_{k}\\mu_{k}^{2\\alpha-1}.\n\\end{align*}\nHence, the induction statement holds for $k+1$. We conclude that \\eqref{ineq:bound_v2} holds for all $k\\geq K$. As a consequence, \\eqref{ineq:bound} holds for all $k\\geq K$ as well. To complete the proof, we need to show $\\lim_{k\\to \\infty}\\EXP{f(x_{k})}=f^*$. This is an immediate result of \\eqref{ineq:bound} and Assumption \\ref{assum:sequences-ms-convergence}(a).\n\\end{proof}","sha256_utf8":"89b0046f05ad03b003918e0197f402a01198f6496cabb09fcf37e1bd755cbe44"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"对条件递推取全期望","source_units":[{"start":43606,"end":44200,"start_line":650,"end_line":654,"raw_tex":"\\begin{proof}\nNote that Assumption \\ref{assum:sequences-ms-convergence}(a,b) imply that \\eqref{ineq:cond-recursive-F-k} holds for a large enough $k$, say after $\\hat K$. Then, since the conditions of Lemma \\ref{lemma:main-ineq} are met \\fy{(with an index shift)}, taking expectations on both sides of \\eqref{ineq:cond-recursive-F-k}, we obtain for any $k\\geq \\hat K$:\n\\begin{align*}\n\\EXP{f_{k+1}(x_{k+1})-f^*} &\\leq (1-\\lambda_{\\min}\\mu_k\\g_k)\\EXP{f_k(x_k)-f^*}+\\frac{\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)}{2}\\mu_k^2\\g_k\\\\ &+\\frac{ (L+\\mu_0)\\lambda^2\\nu^2}{2}\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align*}\n","sha256_utf8":"e58bb28b1fd99226abe02baa910453a7ae039ce37fc6a45288824341362d5f5d"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"合并正则化偏差与方差","source_units":[{"start":44311,"end":44707,"start_line":656,"end_line":659,"raw_tex":" Using Assumption \\ref{assum:sequences-ms-convergence}(e), we have $\\mu_k^2\\g_k\\leq \\rho \\g_k^2\\mu_k^{2\\alpha}$. Thus, we obtain\n\\begin{align}\\label{ineq:cond-recursive-F-k-expected2}\n\\EXP{f_{k+1}(x_{k+1})-f^*} &\\leq (1-\\lambda_{\\min}\\mu_k\\g_k)\\EXP{f_k(x_k)-f^*}\\notag\\\\ &+\\left(\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2}\\right)\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align}\n","sha256_utf8":"66b543e916d1ab5f269810edafdf5ab54cbc25c55440a99777701d5335754822"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"建立正则化误差归纳目标","source_units":[{"start":44707,"end":45263,"start_line":660,"end_line":664,"raw_tex":"Let us define \\fy{$K\\triangleq\\max\\{\\hat K, K_0\\}$, where $K_0$ is from Assumption \\ref{assum:sequences-ms-convergence}(d)}. Using the preceding relation and by induction on $k$, we show the desired result. To show \\eqref{ineq:bound}, we show the following relation first: \n\\begin{align}\\label{ineq:bound_v2}\n \\EXP{f_{k+1}(x_{k+1})}-f^*\\leq \\theta\\g_k\\mu_k^{2\\alpha-1}, \\quad \\hbox{for all } k \\geq K,\n \\end{align}\n Note that \\eqref{ineq:bound_v2} \\fy{implies} the relation \\eqref{ineq:bound} since we have $ \\EXP{f(x_{k+1})}\\leq  \\EXP{f_{k+1}(x_{k+1})}$.\n","sha256_utf8":"781d4573699147900d3c40cad78146051d016a79c36a67c3ba8d2b9ac4fd7bf9"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"验证初始界并设置归纳假设","source_units":[{"start":45263,"end":45964,"start_line":665,"end_line":668,"raw_tex":"First, we show that \\eqref{ineq:bound_v2} holds for \\fy{$k=K$}.  Consider the term $\\EXP{f_{K+1}(x_{K+1})}-f^*$. Multiplying and dividing by $\\g_K\\mu_K^{2\\alpha-1}$, we obtain\n\\begin{align*}\\EXP{f_{K+1}(x_{K+1})}-f^*=\\left(\\frac{\\EXP{f_{K+1}(x_{K+1})}-f^*}{\\g_k\\mu_K^{2\\alpha-1}}\\right)\\g_K\\mu_K^{2\\alpha-1}\n \\leq \\theta \\g_K\\mu_K^{2\\alpha-1},\\end{align*}\nwhere the last inequality is obtained by invoking the definition of $\\theta$ in \\eqref{def:theta}. This implies that \\eqref{ineq:bound_v2} holds for $k=K$. Now assume that  \\eqref{ineq:bound_v2} holds for some $k \\geq K$. We show that it also holds for $k+1$.  From the induction hypothesis and \\eqref{ineq:cond-recursive-F-k-expected2} we have\n","sha256_utf8":"97c0dfe7989c9d53110919f5de9c6124fe7d3d1797d8fb022e818626047b86ef"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"将归纳假设代入误差递推","source_units":[{"start":45964,"end":46203,"start_line":669,"end_line":671,"raw_tex":"\\begin{align*}\n\\EXP{f_{k+1}(x_{k+1})-f^*} &\\leq (1-\\lambda_{\\min}\\mu_k\\g_k)\\theta\\g_{k-1}\\mu_{k-1}^{2\\alpha-1}\\\\ &+\\left(\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2}\\right)\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align*}\n","sha256_utf8":"a55dbaeeb73b6c4bf27d19d34ede2bcbed56f510385fcd3f30edb20861f29de7"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"应用相邻调参序列比较","source_units":[{"start":46203,"end":46569,"start_line":672,"end_line":675,"raw_tex":"Using Assumption \\ref{assum:sequences-ms-convergence}(d) we obtain\n\\begin{align}\\label{ineq:thm2-rel1}\n\\EXP{f_{k+1}(x_{k+1})-f^*} &\\leq \\theta\\g_{k}\\mu_{k}^{2\\alpha-1}(1-\\lambda_{\\min}\\mu_k\\g_k)(1+\\beta \\lambda_{\\min}\\g_{k}\\mu_{k})\\notag \\\\ &+\\left(\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2}\\right)\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align}\n","sha256_utf8":"641e208a805f60f4d3c96369ce0e9897a04cb0280ddf1b56e4ad11d76415b935"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"估计收缩因子的乘积","source_units":[{"start":46569,"end":46925,"start_line":676,"end_line":679,"raw_tex":"Next we find an upper bound for the term $(1-\\lambda_{\\min}\\mu_k\\g_k)(1+\\beta \\lambda_{\\min}\\g_{k}\\mu_{k}) $ as follows\n\\begin{align*}\n(1-\\lambda_{\\min}\\mu_k\\g_k)(1+\\beta \\lambda_{\\min}\\g_{k}\\mu_{k}) &= 1-\\lambda_{\\min}\\mu_k\\g_k+\\beta\\lambda_{\\min}\\mu_k\\g_k-\\fy{\\beta\\lambda_{\\min}^2\\mu_k^2\\g_k^2}\\\\ & \\leq 1-(1-\\beta)\\lambda_{\\min}\\mu_k\\g_k.\n\\end{align*}\n","sha256_utf8":"b349c1f60d5d4535a7760293b0a7018cbc19f7d9dee4d9bc4f638f2aec71aabe"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"整理误差上界与剩余项","source_units":[{"start":46925,"end":47506,"start_line":680,"end_line":683,"raw_tex":"Combining this relation with \\eqref{ineq:thm2-rel1}, it follows \\fy{that}\n\\begin{align*}%\\label{ineq:thm2-rel1}\n&\\EXP{f_{k+1}(x_{k+1})-f^*} \\leq \\theta\\g_{k}\\mu_{k}^{2\\alpha-1} -\\theta\\fy{\\lambda_{\\min}}(1-\\beta)\\mu_k^{2\\alpha}\\g_k^2  \\\\ &+\\left(\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2}\\right)\\mu^{2\\alpha}_k\\g_k^2\\\\ & =\\theta\\g_{k}\\mu_{k}^{2\\alpha-1}-\\left(\\underbrace{\\theta\\fy{\\lambda_{min}}(1-\\beta)-\\frac{\\rho\\lambda_{\\min}\\mbox{dist}^2(x_0,X^*)+ (L+\\mu_0)\\lambda^2\\nu^2}{2}}_{\\tiny{\\hbox{Term }1}}\\right)\\mu^{2\\alpha}_k\\g_k^2.\n\\end{align*}\n","sha256_utf8":"f3fd6a00066816575ce22007b1f3f0be175f89e1727ce614109ec7359c23f768"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"由常数选择闭合归纳并取极限","source_units":[{"start":47506,"end":48107,"start_line":684,"end_line":689,"raw_tex":"Note that the definition of $\\theta$ in \\eqref{def:theta} implies that Term 1 is nonnegative. Therefore, \n\\begin{align*}%\\label{ineq:thm2-rel1}\n\\EXP{f_{k+1}(x_{k+1})-f^*} &\\leq \\theta\\g_{k}\\mu_{k}^{2\\alpha-1}.\n\\end{align*}\nHence, the induction statement holds for $k+1$. We conclude that \\eqref{ineq:bound_v2} holds for all $k\\geq K$. As a consequence, \\eqref{ineq:bound} holds for all $k\\geq K$ as well. To complete the proof, we need to show $\\lim_{k\\to \\infty}\\EXP{f(x_{k})}=f^*$. This is an immediate result of \\eqref{ineq:bound} and Assumption \\ref{assum:sequences-ms-convergence}(a).\n\\end{proof}","sha256_utf8":"781550b5533b9213af72e13bb5a8a2b802a60973140be82885eb4df9112ae5d7"}]},{"id":"S059","semantic_type":"lemma","kind":"lemma","title":"由和与积控制最小正数","source_units":[{"start":67189,"end":67435,"start_line":937,"end_line":940,"raw_tex":"\\begin{lemma}\\label{sumProductBounds}\nLet $0 < a_1 \\leq a_2 \\leq \\ldots \\leq a_n$, and $P$ and $S$ be positive scalars such that \\fys{$\\sum_{i=1}^n a_i \\leq S$ and $\\prod_{i=1}^na_i \\geq P$. Then, we have \n$a_1 \\geq (n-1)!P/S^{n-1}$.}\n\\end{lemma}","sha256_utf8":"2076c2aeef627d9f2fd63175f1a64814dc1291fa2daafa82a6f1d4c292430eed"}]},{"id":"S060","semantic_type":"proof","kind":"proof","title":"和积界完整证明","source_units":[{"start":67437,"end":67503,"start_line":941,"end_line":943,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{app:sumProductBounds}.\n\\end{proof}","sha256_utf8":"ea516e9471b72bdc5e78ce0605c368f9f8ab34c8e88fa06daedbaca7385f6752"},{"start":114718,"end":115171,"start_line":1472,"end_line":1478,"raw_tex":"\\subsection{Proof of Lemma \\ref{sumProductBounds}}\\label{app:sumProductBounds}\nFrom $0<a_1 \\leq \\ldots \\leq a_n$, we can write \n\\begin{align*}\n(n-(i-1))a_i \\leq \\sum_{j=1}^na_j, \\qquad \\hbox{for all } i \\in \\{1,\\ldots,n\\}.\n\\end{align*}\nInvoking $\\sum_{i=1}^n{a_i} \\leq S$, we obtain $a_i \\leq \\frac{S}{n-(i-1)}$, for all $i \\in \\{1,\\ldots,n\\}$. From the preceding relation and that $\\prod_{j=1}^na_j \\geq P$, we can obtain $a_1 \\geq (n-1)!P/S^{n-1}.$\n}\n","sha256_utf8":"cdb9d6fe57ea2ff58fe9c842e700f502870ac0c7fe699c7c973a008d819a0545"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"和积界证明的附录指引","source_units":[{"start":67437,"end":67503,"start_line":941,"end_line":943,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{app:sumProductBounds}.\n\\end{proof}","sha256_utf8":"ea516e9471b72bdc5e78ce0605c368f9f8ab34c8e88fa06daedbaca7385f6752"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"利用排序逐项控制大小","source_units":[{"start":114718,"end":114954,"start_line":1472,"end_line":1476,"raw_tex":"\\subsection{Proof of Lemma \\ref{sumProductBounds}}\\label{app:sumProductBounds}\nFrom $0<a_1 \\leq \\ldots \\leq a_n$, we can write \n\\begin{align*}\n(n-(i-1))a_i \\leq \\sum_{j=1}^na_j, \\qquad \\hbox{for all } i \\in \\{1,\\ldots,n\\}.\n\\end{align*}\n","sha256_utf8":"4010bf2d1484f7fa3164faacc6296c2e5029fc08e1090102f0d068a5d6bc9dd5"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"结合乘积下界得到最小项界","source_units":[{"start":114954,"end":115171,"start_line":1477,"end_line":1478,"raw_tex":"Invoking $\\sum_{i=1}^n{a_i} \\leq S$, we obtain $a_i \\leq \\frac{S}{n-(i-1)}$, for all $i \\in \\{1,\\ldots,n\\}$. From the preceding relation and that $\\prod_{j=1}^na_j \\geq P$, we can obtain $a_1 \\geq (n-1)!P/S^{n-1}.$\n}\n","sha256_utf8":"09841ece319b583ad1f884ae1c794b7f0f1c08f082025f6eb9757ef0ffb386df"}]},{"id":"S064","semantic_type":"lemma","kind":"lemma","title":"LBFGS曲率割线与谱界","source_units":[{"start":67505,"end":68988,"start_line":944,"end_line":956,"raw_tex":"\\begin{lemma}[Properties of  update rule \\eqref{eqn:H-k}-\\eqref{eqn:H-k-m}]\\label{LBFGS-matrix}\nConsider the \\eqref{eqn:LM-cyclic-reg-BFGS} method. Let $H_k$ be given by the update rule \\eqref{eqn:H-k}-\\eqref{eqn:H-k-m}, where $s_i$ and $y_i$ are defined in \\eqref{equ:siyi-LBFGS} and $\\mu_k$ is updated according to the procedure \\eqref{eqn:mu-k}. Let \\fy{Assumption \\ref{assum:convex2}(a,b)} hold. Then, the following results hold:\n\\begin{itemize}\n\\item [(a)]  For any odd \\fy{$k \\geq 2m-1$}, the \\fy{curvature condition} holds, i.e., \\fy{$s_{\\lceil k/2\\rceil}^T{y_{\\lceil k/2\\rceil}} >0$}.\n\\item [(b)]  For any odd \\fy{$k \\geq 2m-1$}, the secant equation holds, i.e., \\fy{$H_{k}{y}_{\\lceil k/2\\rceil}=s_{\\lceil k/2\\rceil}$}. \n\\item [(c)] For any \\fy{$k \\geq 2m-1$}, $H_k$ satisfies Assumption \\ref{assump:Hk} \\fy{with} the following values:\n\\begin{align}\\label{equ:valuesForAssumH_k}&\\lambda_{\\min}=\\frac{1}{(m+n)\\left(L+\\fys{\\tau}\\mu_0^\\delta\\right)}, \\quad \\fy{\\lambda= \\frac{(m+n)^{n+m-1}\\left(L+\\fys{\\tau}\\mu_0^\\delta\\right)^{n+m-1}}{(n-1)!\\fys{\\tau^{(n+m)}}}}, \\notag\\\\ & \\fy{\\hbox{and }\\alpha=-\\delta(n+m).}\n\\end{align}\nMore precisely, $H_k$ is symmetric, $\\EXP{H_k\\mid\\sF_k}=H_k$ and\n\\begin{align}\\label{proof:H_kbounds} \\frac{1}{(m+n)\\left(L+\\fys{\\tau}\\mu_0^\\delta\\right)}\\mathbf{I} \\preceq H_{k} \\preceq \\frac{(m+n)^{n+m-1}\\left(L+\\fys{\\tau}\\mu_0^\\delta\\right)^{n+m-1}}{(n-1)!\\fys{\\left(\\tau\\mu_k^\\delta\\right)^{(n+m)}}}\\ \\mathbf{I}.\n\\end{align}\n\\end{itemize}\n\\end{lemma}","sha256_utf8":"986576249719ddb3b3baa02b594dee057ccafe610c927f4d6775159f6a2fc21b"}]},{"id":"S065","semantic_type":"proof","kind":"proof","title":"LBFGS矩阵性质完整证明","source_units":[{"start":68993,"end":69059,"start_line":957,"end_line":959,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{app:lemmaLBFGSmatrix}.\n\\end{proof}","sha256_utf8":"2ced26ab106406954dfdbc677eac8333835297eb18538022586d7342349cfb2b"},{"start":115246,"end":124793,"start_line":1480,"end_line":1540,"raw_tex":"\\begin{proof} \\fys{Throughout, we let $\\lambda_{k,\\min}$, $\\lambda_{k,\\max}$, and $B_k$ denote the minimum eigenvalue, maximum eigenvalue, and inverse of matrix $H_k$ in \\eqref{eqn:H-k-m}, respectively.}\nIt can be seen, by induction on $k$, that \\fy{$H_k$ is symmetric and $\\sF_k$ measurable.} We use induction on odd values of \\fys{$k\\geq 2m-1$ to show that parts (a), (b), and (c) hold.} Suppose \\fy{$k\\geq 2m-1$} is odd and for any \\fys{odd} $t<k$, we have \\fy{$s_{\\lceil t/2\\rceil}^T{y_{\\lceil t/2\\rceil}} >0$, $H_{t}{y}_{\\lceil t/2\\rceil}=s_{\\lceil t/2\\rceil}$}, and \\eqref{proof:H_kbounds} for $t$. We show that these statements also hold for $k$ \\fys{as well}. First, we \\fys{show} that the \\fy{curvature condition} holds. We can write \n\\begin{align*}\n\\fys{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}&=(x_{k}-x_{k-1})^T(\\nabla F(x_k,\\xi_{k-1})- \\nabla F(x_{k-1},\\xi_{k-1})+\\fys{\\tau}\\mu_k^\\delta(x_k-x_{k-1}))\\\\\n&=(x_{k}-x_{k-1})^T(\\nabla F(x_k,\\xi_{k-1})- \\nabla F(x_{k-1},\\xi_{k-1}))+\\fys{\\tau}\\mu_k^\\delta\\|x_k-x_{k-1}\\|^2\\\\\n&\\geq \\fys{\\tau}\\mu_k^\\delta\\|x_k-x_{k-1}\\|^2,\n\\end{align*}\nwhere the inequality follows from the monotonicity of the gradient map $\\nabla F(\\cdot,\\xi)$. \\fys{Next, we show that $\\|x_k-x_{k-1}\\|^2 >0$.} From the induction hypothesis \\fys{and that $k-2$ is odd}, $H_{k-2}$ is positive definite.\n\\fys{Moreover, from the update rule \\eqref{eqn:H-k} and that $k-2$ is odd, we have $H_{k-1}=H_{k-2}$.}  \n%Also, since $B_{k-2}$ is symmetric, the update rule \\eqref{eqn:B-k} implies that $B_{k-1}$ is symmetric.\nTherefore, $H_{k-1}$ is \\fys{also} positive definite. \\fy{Without loss of generality, we assume $\\nabla F(x_{k-1},\\xi_{k-1})+\\mu_{k-1}(x_{k-1}-x_0)\\neq 0$}\\footnote{If $\\nabla F(x_{k},\\xi_k)+\\mu_{k}(x_k-x_0)=0$\\fys{,} then we can draw a new sample of $\\xi_k$ to satisfy the relation.}.\nSince $H_{k-1}$ is positive definite, \nwe have $$H_{k-1}\\left(\\nabla F(x_{k-1},\\xi_{k-1})+\\mu_{k-1}\\fy{(x_{k-1}-x_0)}\\right) \\neq 0,$$ implying that \n$x_{k} \\neq x_{k-1}$. Hence\n$\\fy{s_{\\lceil k/2\\rceil}^T{y_{\\lceil k/2\\rceil}} }\\geq  \\fys{\\tau}\\mu_k^\\delta\\|x_k-x_{k-1}\\|^2 >0,$\nwhere \\us{the second inequality is a consequence of} $\\fys{\\tau},\\mu_k>0$.\nThus, the \\fy{curvature condition} holds.\nNext, we show that \\eqref{proof:H_kbounds} holds for $k$. It is well-known that using the Sherman-Morrison-Woodbury formula, $B_k$ is equal to $B_{k,m}$ given by \n\\begin{align}\\label{equ:B_kLimited}\nB_{k,j}=B_{k,j-1}-\\frac{B_{k,j-1}s_is_i^TB_{k,j-1}}{s_i^TB_{k,j-1}s_i}+\\frac{y_iy_i^T}{y_i^Ts_i}, \\quad i:=\\fy{{\\lceil k/2\\rceil}-(m-j)}, \\quad 1 \\leq j \\leq m,\n\\end{align}\nwhere $s_i$ and $y_i$ are defined by \\eqref{equ:siyi-LBFGS} and \\fy{$B_{k,0}=\\frac{y_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}\\mathbf{I}$}. \\fys{Note that with $j$ varying between $1$ to $m$, the index $i$ takes values in $\\left\\{\\lceil k/2\\rceil-m+1,\\lceil k/2\\rceil-m+2,\\ldots,\\lceil k/2\\rceil\\right\\}$.} First, we show that for any $i$ \\fys{in this range}, \\begin{align}\\label{equ:boundsForB0}\n\\fys{\\tau}\\mu_k^\\delta \\leq \\frac{\\|y_i\\|^2}{y_i^Ts_i} \\leq L+\\fys{\\tau}\\mu_k^\\delta,\n\\end{align}\n where $L$ is the Lipschitzian parameter of the gradient mapping $\\nabla F$ \\fys{given by Assumption \\ref{assum:convex2}(b)}. Let us \\fys{define the function $h(x)\\triangleq F(x,\\xi_{i-1})+\\tau\\frac{\\mu_k^\\delta}{2}\\|x-x_0\\|^2$} for fixed $i$ and $k$. Note that this function is strongly convex and has a gradient mapping of the form $\\nabla F+\\fys{\\tau}\\mu_k^\\delta(\\mathbf{I}-x_0)$ that  is Lipschitz with parameter $L+\\fys{\\tau}\\mu_k^\\delta$. For a convex function $h$ with Lipschitz gradient with parameter $L+\\fys{\\tau}\\mu_k^\\delta$, the following inequality, referred to as co-coercivity  property, holds for any $x_1,x_2 \\in \\Real^n$ \\fys{(see}\\cite{Polyak87}, Pg. 24 , Lemma 2):\n \\[\\|\\nabla h(x_2)-\\nabla h(x_1)\\|^2 \\leq \\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)(x_2-x_1)^T(\\nabla h(x_2)-\\nabla h(x_1)).\\]\nSubstituting $x_2$ by $x_i$, $x_1$ by $x_{i-1}$, and recalling \\eqref{equ:siyi-LBFGS}, the preceding inequality yields\n\\begin{align}\\label{ineq:boundsForB0-1}\\|y_i\\|^2 \\leq \\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)s_i^Ty_i.\\end{align}\nNote that function $h$ is strongly convex \\fy{with} parameter $\\fys{\\tau}\\mu_k^\\delta$. Applying the Cauchy-Schwarz inequality, we can write\n\\[\\frac{\\|y_i\\|^2}{s_i^Ty_i} \\geq \\frac{\\|y_i\\|^2}{\\|s_i\\|\\|y_i\\|} =\\frac{\\|y_i\\|}{\\|s_i\\|}\\geq  \\frac{\\|y_i\\|\\|s_i\\|}{\\|s_i\\|^2} \\geq \\frac{y_i^Ts_i}{\\|s_i\\|^2}\\geq \\fys{\\tau}\\mu_k^\\delta.\\]\nCombining this relation with \\eqref{ineq:boundsForB0-1}, we obtain \\eqref{equ:boundsForB0}. Next, we show that the maximum eigenvalue of $B_k$ is bounded. Let $Trace(\\cdot)$ denote the trace of a matrix. Taking trace from both sides of \\eqref{equ:B_kLimited} and summing up over index $j$, we obtain \\fy{for $i:={\\lceil k/2\\rceil}-(m-j)$,}\n\\begin{align}\\label{ineq:trace}\nTrace(B_{k,m})&=Trace(B_{k,0})-\\sum_{j=1}^m Trace\\left(\\frac{B_{k,j-1}s_is_i^TB_{k,j-1}}{s_i^TB_{k,j-1}s_i}\\right)+\\sum_{j=1}^m Trace\\left(\\frac{y_iy_i^T}{y_i^Ts_i}\\right)\\cr\n& =Trace\\fy{\\left(\\frac{\\|y_{\\lceil k/2\\rceil}\\|^2}{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}\\mathbf{I}\\right)} - \\sum_{j=1}^m \\frac{\\|B_{k,j-1}s_i\\|^2}{s_i^TB_{k,j-1}s_i} + \\sum_{j=1}^m \\frac{\\|y_i\\|^2}{y_i^Ts_i}\\cr \n&\\leq n \\fy{\\frac{\\|y_{\\lceil k/2\\rceil}\\|^2}{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}} +\\sum_{j=1}^m \\left(L+\\fys{\\tau}\\mu_k^\\delta\\right) \\fy{\\leq} (m+n)\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right),\n\\end{align}\nwhere the third relation is obtained by positive-definiteness of $B_k$ (this can be seen by induction on \\fys{$j$}, and using \\eqref{equ:B_kLimited} and $B_{k,0}\\succ 0$). Since $B_k=B_{k,m}$, the maximum eigenvalue of the matrix $B_k$ is bounded \\fys{by $(m+n)\\left(L+\\tau\\mu_k^\\delta\\right)$}. As a result, \n\\begin{align}\\label{proof:lowerbound}\n\\lambda_{k,\\min}\\geq \\frac{1}{(m+n)\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)}.\\end{align}\n In the next part of the proof, we establish the bound for $\\lambda_{k,\\max}$. The following relation can be shown (e.g., see Lemma 3 in \\cite{Mokhtari15})\n\\begin{align*}\ndet(B_{k,m})=det(B_{k,0})\\prod_{j=1}^m\\frac{s_i^Ty_i}{s_i^TB_{k,j-1}s_i}\\fy{, \\quad \\hbox{for }i:={\\lceil k/2\\rceil}-(m-j)}.\n\\end{align*}\nMultiplying and dividing by $s_i^Ts_i$, using the strong convexity of the function $h$, and invoking \\eqref{equ:boundsForB0} and the result of \\eqref{ineq:trace}, we obtain\n\\begin{align}\\label{ineq:detBk}\ndet(B_{k})&=\\fy{det\\left(\\frac{y_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}\\mathbf{I}\\right)}\\prod_{j=1}^m\\left(\\frac{s_i^Ty_i}{s_i^Ts_i}\\right)\\left(\\frac{s_i^Ts_i}{s_i^TB_{k,j-1}s_i}\\right)\\cr  \n& \\geq\\fy{\\left(\\frac{y_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}\\right)^n}\\prod_{j=1}^m\\fys{\\tau}\\mu_k^\\delta\\left(\\frac{s_i^Ts_i}{s_i^TB_{k,j-1}s_i}\\right)\\cr\n& \\geq  \\fys{\\left(\\tau\\mu_k^\\delta\\right)^{(n+m)}} \\prod_{j=1}^m \\frac{1}{(m+n)\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)} = \\frac{\\fys{\\left(\\tau\\mu_k^\\delta\\right)^{(n+m)}}}{(m+n)^{m}\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)^m}.\n\\end{align}\nLet \\fy{$\\alpha_{k,1}\\leq \\alpha_{k,2}\\leq\\ldots\\leq\\alpha_{k,n}$} be the eigenvalues of $B_k$ sorted non-decreasingly. Note that since $B_k\\succ0$, all the eigenvalues are positive. Taking \\eqref{ineq:trace} and \\eqref{ineq:detBk} into account, and employing Lemma \\ref{sumProductBounds}, we obtain\n \\[\\alpha_{1,k} \\fy{\\geq} \\frac{(n-1)!\\fys{\\left(\\tau\\mu_k^\\delta\\right)^{(n+m)}}}{(m+n)^{n+m-1}\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)^{n+m-1}}.\\]\nThis relation and that $\\alpha_{k,1}=\\lambda_{k,\\max}^{-1}$ imply that \n\\begin{align}\\label{proof:upperbound}\n\\lambda_{k,\\max}\\leq \\frac{(m+n)^{n+m-1}\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)^{n+m-1}}{(n-1)!\\fys{\\left(\\tau\\mu_k^\\delta\\right)^{(n+m)}}}.\n\\end{align}\nTherefore, from \\eqref{proof:lowerbound} and \\eqref{proof:upperbound} and that $\\mu_k$ is non-increasing, we conclude that \\eqref{proof:H_kbounds} holds for $k$ as well. Next, we show $H_ky_{\\lceil k/2\\rceil}=s_{\\lceil k/2\\rceil}$. From \\eqref{equ:B_kLimited}, for $j=m$ we obtain\n\\[B_{k,m}=B_{k,m-1}-\\fy{\\frac{B_{k,m-1}s_{\\lceil k/2\\rceil}s_{\\lceil k/2\\rceil}^TB_{k,m-1}}{s_{\\lceil k/2\\rceil}^TB_{k,m-1}s_{\\lceil k/2\\rceil}}+\\frac{y_{\\lceil k/2\\rceil}y_{\\lceil k/2\\rceil}^T}{y_{\\lceil k/2\\rceil}^Ts_{\\lceil k/2\\rceil}}},\\]\nwhere we used \\fy{$i={\\lceil k/2\\rceil}-(m-m)={\\lceil k/2\\rceil}$}. Multiplying both sides of the preceding equation by \\fy{$s_{\\lceil k/2\\rceil}$}, and using $B_k=B_{k,m}$, we have\n\\fy{$B_{k}s_k=B_{k,m-1}s_{\\lceil k/2\\rceil}-B_{k,m-1}s_{\\lceil k/2\\rceil}+y_{\\lceil k/2\\rceil}=y_{\\lceil k/2\\rceil}$}. \nMultiplying both sides of the preceding relation by $H_k$ and invoking $H_k=B_k^{-1}$, we conclude that \\fy{$H_ky_{\\lceil k/2\\rceil}=s_{\\lceil k/2\\rceil}$}. Therefore, we showed that the statements of (a), (b), and (c) hold for \\fys{an odd $k$}, assuming that they hold for any odd \\fy{$t<k$}. In a similar fashion to this analysis, it can be seen that the statements hold for \\fys{$t=2m-1$}. Thus, by induction, we conclude that the statements hold for any odd \\fy{$k\\geq 2m-1$}. To complete the proof, it is enough to show that \\eqref{proof:H_kbounds} holds for any even \\fys{$k\\geq 2m$}. Let $t=k-1$. Since \\fys{$t$} is odd, relation \\eqref{proof:H_kbounds} holds. Writing  \\eqref{proof:H_kbounds} for $k-1$, and taking into account that $H_k=H_{k-1}$, and $\\mu_k<\\mu_{k-1}$, we can conclude that \\eqref{proof:H_kbounds} holds for any even \\fy{$k\\geq 2m-1$} and this completes the proof.\n\\end{proof}","sha256_utf8":"f8c59473781c16b5fad6f6ba0177ea46ae037eea247a4c87ffa7b3ec131c04ad"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"矩阵性质证明的附录指引","source_units":[{"start":68993,"end":69059,"start_line":957,"end_line":959,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{app:lemmaLBFGSmatrix}.\n\\end{proof}","sha256_utf8":"2ced26ab106406954dfdbc677eac8333835297eb18538022586d7342349cfb2b"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"定义逆矩阵谱量并设置归纳","source_units":[{"start":115246,"end":115990,"start_line":1480,"end_line":1481,"raw_tex":"\\begin{proof} \\fys{Throughout, we let $\\lambda_{k,\\min}$, $\\lambda_{k,\\max}$, and $B_k$ denote the minimum eigenvalue, maximum eigenvalue, and inverse of matrix $H_k$ in \\eqref{eqn:H-k-m}, respectively.}\nIt can be seen, by induction on $k$, that \\fy{$H_k$ is symmetric and $\\sF_k$ measurable.} We use induction on odd values of \\fys{$k\\geq 2m-1$ to show that parts (a), (b), and (c) hold.} Suppose \\fy{$k\\geq 2m-1$} is odd and for any \\fys{odd} $t<k$, we have \\fy{$s_{\\lceil t/2\\rceil}^T{y_{\\lceil t/2\\rceil}} >0$, $H_{t}{y}_{\\lceil t/2\\rceil}=s_{\\lceil t/2\\rceil}$}, and \\eqref{proof:H_kbounds} for $t$. We show that these statements also hold for $k$ \\fys{as well}. First, we \\fys{show} that the \\fy{curvature condition} holds. We can write \n","sha256_utf8":"3df2adcb8cc9371a573811e0d1eab11fd310cd2627eabc68ed028304299b6eda"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"梯度单调性给出曲率下界","source_units":[{"start":115990,"end":116575,"start_line":1482,"end_line":1487,"raw_tex":"\\begin{align*}\n\\fys{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}&=(x_{k}-x_{k-1})^T(\\nabla F(x_k,\\xi_{k-1})- \\nabla F(x_{k-1},\\xi_{k-1})+\\fys{\\tau}\\mu_k^\\delta(x_k-x_{k-1}))\\\\\n&=(x_{k}-x_{k-1})^T(\\nabla F(x_k,\\xi_{k-1})- \\nabla F(x_{k-1},\\xi_{k-1}))+\\fys{\\tau}\\mu_k^\\delta\\|x_k-x_{k-1}\\|^2\\\\\n&\\geq \\fys{\\tau}\\mu_k^\\delta\\|x_k-x_{k-1}\\|^2,\n\\end{align*}\nwhere the inequality follows from the monotonicity of the gradient map $\\nabla F(\\cdot,\\xi)$. \\fys{Next, we show that $\\|x_k-x_{k-1}\\|^2 >0$.} From the induction hypothesis \\fys{and that $k-2$ is odd}, $H_{k-2}$ is positive definite.\n","sha256_utf8":"3ab3d257fec9d5d2c5cbea24395462feeea56248442346595ae72066aea3ad01"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"非零梯度假设保证严格曲率","source_units":[{"start":116575,"end":117469,"start_line":1488,"end_line":1496,"raw_tex":"\\fys{Moreover, from the update rule \\eqref{eqn:H-k} and that $k-2$ is odd, we have $H_{k-1}=H_{k-2}$.}  \n%Also, since $B_{k-2}$ is symmetric, the update rule \\eqref{eqn:B-k} implies that $B_{k-1}$ is symmetric.\nTherefore, $H_{k-1}$ is \\fys{also} positive definite. \\fy{Without loss of generality, we assume $\\nabla F(x_{k-1},\\xi_{k-1})+\\mu_{k-1}(x_{k-1}-x_0)\\neq 0$}\\footnote{If $\\nabla F(x_{k},\\xi_k)+\\mu_{k}(x_k-x_0)=0$\\fys{,} then we can draw a new sample of $\\xi_k$ to satisfy the relation.}.\nSince $H_{k-1}$ is positive definite, \nwe have $$H_{k-1}\\left(\\nabla F(x_{k-1},\\xi_{k-1})+\\mu_{k-1}\\fy{(x_{k-1}-x_0)}\\right) \\neq 0,$$ implying that \n$x_{k} \\neq x_{k-1}$. Hence\n$\\fy{s_{\\lceil k/2\\rceil}^T{y_{\\lceil k/2\\rceil}} }\\geq  \\fys{\\tau}\\mu_k^\\delta\\|x_k-x_{k-1}\\|^2 >0,$\nwhere \\us{the second inequality is a consequence of} $\\fys{\\tau},\\mu_k>0$.\nThus, the \\fy{curvature condition} holds.\n","sha256_utf8":"0ef328d12e4cd72dad77e67b4ea3793e4c4b598f16ba3fb6b11eea2e6c27bb59"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"写出有限记忆矩阵的逆递推","source_units":[{"start":117469,"end":117841,"start_line":1497,"end_line":1500,"raw_tex":"Next, we show that \\eqref{proof:H_kbounds} holds for $k$. It is well-known that using the Sherman-Morrison-Woodbury formula, $B_k$ is equal to $B_{k,m}$ given by \n\\begin{align}\\label{equ:B_kLimited}\nB_{k,j}=B_{k,j-1}-\\frac{B_{k,j-1}s_is_i^TB_{k,j-1}}{s_i^TB_{k,j-1}s_i}+\\frac{y_iy_i^T}{y_i^Ts_i}, \\quad i:=\\fy{{\\lceil k/2\\rceil}-(m-j)}, \\quad 1 \\leq j \\leq m,\n\\end{align}\n","sha256_utf8":"e02684aadb20993d2083f8e76753d5433d47ec2908a787748ef16ef9bc2d64a7"}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","title":"强凸与余强制性控制曲率比","source_units":[{"start":117841,"end":119754,"start_line":1501,"end_line":1509,"raw_tex":"where $s_i$ and $y_i$ are defined by \\eqref{equ:siyi-LBFGS} and \\fy{$B_{k,0}=\\frac{y_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}\\mathbf{I}$}. \\fys{Note that with $j$ varying between $1$ to $m$, the index $i$ takes values in $\\left\\{\\lceil k/2\\rceil-m+1,\\lceil k/2\\rceil-m+2,\\ldots,\\lceil k/2\\rceil\\right\\}$.} First, we show that for any $i$ \\fys{in this range}, \\begin{align}\\label{equ:boundsForB0}\n\\fys{\\tau}\\mu_k^\\delta \\leq \\frac{\\|y_i\\|^2}{y_i^Ts_i} \\leq L+\\fys{\\tau}\\mu_k^\\delta,\n\\end{align}\n where $L$ is the Lipschitzian parameter of the gradient mapping $\\nabla F$ \\fys{given by Assumption \\ref{assum:convex2}(b)}. Let us \\fys{define the function $h(x)\\triangleq F(x,\\xi_{i-1})+\\tau\\frac{\\mu_k^\\delta}{2}\\|x-x_0\\|^2$} for fixed $i$ and $k$. Note that this function is strongly convex and has a gradient mapping of the form $\\nabla F+\\fys{\\tau}\\mu_k^\\delta(\\mathbf{I}-x_0)$ that  is Lipschitz with parameter $L+\\fys{\\tau}\\mu_k^\\delta$. For a convex function $h$ with Lipschitz gradient with parameter $L+\\fys{\\tau}\\mu_k^\\delta$, the following inequality, referred to as co-coercivity  property, holds for any $x_1,x_2 \\in \\Real^n$ \\fys{(see}\\cite{Polyak87}, Pg. 24 , Lemma 2):\n \\[\\|\\nabla h(x_2)-\\nabla h(x_1)\\|^2 \\leq \\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)(x_2-x_1)^T(\\nabla h(x_2)-\\nabla h(x_1)).\\]\nSubstituting $x_2$ by $x_i$, $x_1$ by $x_{i-1}$, and recalling \\eqref{equ:siyi-LBFGS}, the preceding inequality yields\n\\begin{align}\\label{ineq:boundsForB0-1}\\|y_i\\|^2 \\leq \\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)s_i^Ty_i.\\end{align}\nNote that function $h$ is strongly convex \\fy{with} parameter $\\fys{\\tau}\\mu_k^\\delta$. Applying the Cauchy-Schwarz inequality, we can write\n\\[\\frac{\\|y_i\\|^2}{s_i^Ty_i} \\geq \\frac{\\|y_i\\|^2}{\\|s_i\\|\\|y_i\\|} =\\frac{\\|y_i\\|}{\\|s_i\\|}\\geq  \\frac{\\|y_i\\|\\|s_i\\|}{\\|s_i\\|^2} \\geq \\frac{y_i^Ts_i}{\\|s_i\\|^2}\\geq \\fys{\\tau}\\mu_k^\\delta.\\]\n","sha256_utf8":"d1f860508e13b4b4fd0e61bc54dea066adf6cceafeb078a6b239191d714eb607"}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","title":"迹估计给出逆矩阵谱下界","source_units":[{"start":119754,"end":121164,"start_line":1510,"end_line":1518,"raw_tex":"Combining this relation with \\eqref{ineq:boundsForB0-1}, we obtain \\eqref{equ:boundsForB0}. Next, we show that the maximum eigenvalue of $B_k$ is bounded. Let $Trace(\\cdot)$ denote the trace of a matrix. Taking trace from both sides of \\eqref{equ:B_kLimited} and summing up over index $j$, we obtain \\fy{for $i:={\\lceil k/2\\rceil}-(m-j)$,}\n\\begin{align}\\label{ineq:trace}\nTrace(B_{k,m})&=Trace(B_{k,0})-\\sum_{j=1}^m Trace\\left(\\frac{B_{k,j-1}s_is_i^TB_{k,j-1}}{s_i^TB_{k,j-1}s_i}\\right)+\\sum_{j=1}^m Trace\\left(\\frac{y_iy_i^T}{y_i^Ts_i}\\right)\\cr\n& =Trace\\fy{\\left(\\frac{\\|y_{\\lceil k/2\\rceil}\\|^2}{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}\\mathbf{I}\\right)} - \\sum_{j=1}^m \\frac{\\|B_{k,j-1}s_i\\|^2}{s_i^TB_{k,j-1}s_i} + \\sum_{j=1}^m \\frac{\\|y_i\\|^2}{y_i^Ts_i}\\cr \n&\\leq n \\fy{\\frac{\\|y_{\\lceil k/2\\rceil}\\|^2}{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}} +\\sum_{j=1}^m \\left(L+\\fys{\\tau}\\mu_k^\\delta\\right) \\fy{\\leq} (m+n)\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right),\n\\end{align}\nwhere the third relation is obtained by positive-definiteness of $B_k$ (this can be seen by induction on \\fys{$j$}, and using \\eqref{equ:B_kLimited} and $B_{k,0}\\succ 0$). Since $B_k=B_{k,m}$, the maximum eigenvalue of the matrix $B_k$ is bounded \\fys{by $(m+n)\\left(L+\\tau\\mu_k^\\delta\\right)$}. As a result, \n\\begin{align}\\label{proof:lowerbound}\n\\lambda_{k,\\min}\\geq \\frac{1}{(m+n)\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)}.\\end{align}\n","sha256_utf8":"85900d4eb5c729f29a787df7a26dd4b284acbf4472f04de391a46327f1c8deff"}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","title":"行列式递推给出乘积下界","source_units":[{"start":121164,"end":122358,"start_line":1519,"end_line":1528,"raw_tex":" In the next part of the proof, we establish the bound for $\\lambda_{k,\\max}$. The following relation can be shown (e.g., see Lemma 3 in \\cite{Mokhtari15})\n\\begin{align*}\ndet(B_{k,m})=det(B_{k,0})\\prod_{j=1}^m\\frac{s_i^Ty_i}{s_i^TB_{k,j-1}s_i}\\fy{, \\quad \\hbox{for }i:={\\lceil k/2\\rceil}-(m-j)}.\n\\end{align*}\nMultiplying and dividing by $s_i^Ts_i$, using the strong convexity of the function $h$, and invoking \\eqref{equ:boundsForB0} and the result of \\eqref{ineq:trace}, we obtain\n\\begin{align}\\label{ineq:detBk}\ndet(B_{k})&=\\fy{det\\left(\\frac{y_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}\\mathbf{I}\\right)}\\prod_{j=1}^m\\left(\\frac{s_i^Ty_i}{s_i^Ts_i}\\right)\\left(\\frac{s_i^Ts_i}{s_i^TB_{k,j-1}s_i}\\right)\\cr  \n& \\geq\\fy{\\left(\\frac{y_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}{s_{\\lceil k/2\\rceil}^Ty_{\\lceil k/2\\rceil}}\\right)^n}\\prod_{j=1}^m\\fys{\\tau}\\mu_k^\\delta\\left(\\frac{s_i^Ts_i}{s_i^TB_{k,j-1}s_i}\\right)\\cr\n& \\geq  \\fys{\\left(\\tau\\mu_k^\\delta\\right)^{(n+m)}} \\prod_{j=1}^m \\frac{1}{(m+n)\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)} = \\frac{\\fys{\\left(\\tau\\mu_k^\\delta\\right)^{(n+m)}}}{(m+n)^{m}\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)^m}.\n\\end{align}\n","sha256_utf8":"46c8f392d30deecc76daf2809d739f29ac4fbb3ae16ef1e0d7759016c6c60fed"}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","title":"和积引理给出逆矩阵谱上界","source_units":[{"start":122358,"end":123065,"start_line":1529,"end_line":1534,"raw_tex":"Let \\fy{$\\alpha_{k,1}\\leq \\alpha_{k,2}\\leq\\ldots\\leq\\alpha_{k,n}$} be the eigenvalues of $B_k$ sorted non-decreasingly. Note that since $B_k\\succ0$, all the eigenvalues are positive. Taking \\eqref{ineq:trace} and \\eqref{ineq:detBk} into account, and employing Lemma \\ref{sumProductBounds}, we obtain\n \\[\\alpha_{1,k} \\fy{\\geq} \\frac{(n-1)!\\fys{\\left(\\tau\\mu_k^\\delta\\right)^{(n+m)}}}{(m+n)^{n+m-1}\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)^{n+m-1}}.\\]\nThis relation and that $\\alpha_{k,1}=\\lambda_{k,\\max}^{-1}$ imply that \n\\begin{align}\\label{proof:upperbound}\n\\lambda_{k,\\max}\\leq \\frac{(m+n)^{n+m-1}\\left(L+\\fys{\\tau}\\mu_k^\\delta\\right)^{n+m-1}}{(n-1)!\\fys{\\left(\\tau\\mu_k^\\delta\\right)^{(n+m)}}}.\n\\end{align}\n","sha256_utf8":"d2d6a04bc02d1763bf065f1655b5ecd9427e9ff9b41691f0f7c887e2adea01d1"}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","title":"合并谱界并验证割线方程","source_units":[{"start":123065,"end":123891,"start_line":1535,"end_line":1538,"raw_tex":"Therefore, from \\eqref{proof:lowerbound} and \\eqref{proof:upperbound} and that $\\mu_k$ is non-increasing, we conclude that \\eqref{proof:H_kbounds} holds for $k$ as well. Next, we show $H_ky_{\\lceil k/2\\rceil}=s_{\\lceil k/2\\rceil}$. From \\eqref{equ:B_kLimited}, for $j=m$ we obtain\n\\[B_{k,m}=B_{k,m-1}-\\fy{\\frac{B_{k,m-1}s_{\\lceil k/2\\rceil}s_{\\lceil k/2\\rceil}^TB_{k,m-1}}{s_{\\lceil k/2\\rceil}^TB_{k,m-1}s_{\\lceil k/2\\rceil}}+\\frac{y_{\\lceil k/2\\rceil}y_{\\lceil k/2\\rceil}^T}{y_{\\lceil k/2\\rceil}^Ts_{\\lceil k/2\\rceil}}},\\]\nwhere we used \\fy{$i={\\lceil k/2\\rceil}-(m-m)={\\lceil k/2\\rceil}$}. Multiplying both sides of the preceding equation by \\fy{$s_{\\lceil k/2\\rceil}$}, and using $B_k=B_{k,m}$, we have\n\\fy{$B_{k}s_k=B_{k,m-1}s_{\\lceil k/2\\rceil}-B_{k,m-1}s_{\\lceil k/2\\rceil}+y_{\\lceil k/2\\rceil}=y_{\\lceil k/2\\rceil}$}. \n","sha256_utf8":"8c7d21f6cbc3a7923a32b26b3e5add1a62064bd2c83e74dafcdd06cc9532f5bc"}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","title":"完成奇数归纳并推广到偶数","source_units":[{"start":123891,"end":124793,"start_line":1539,"end_line":1540,"raw_tex":"Multiplying both sides of the preceding relation by $H_k$ and invoking $H_k=B_k^{-1}$, we conclude that \\fy{$H_ky_{\\lceil k/2\\rceil}=s_{\\lceil k/2\\rceil}$}. Therefore, we showed that the statements of (a), (b), and (c) hold for \\fys{an odd $k$}, assuming that they hold for any odd \\fy{$t<k$}. In a similar fashion to this analysis, it can be seen that the statements hold for \\fys{$t=2m-1$}. Thus, by induction, we conclude that the statements hold for any odd \\fy{$k\\geq 2m-1$}. To complete the proof, it is enough to show that \\eqref{proof:H_kbounds} holds for any even \\fys{$k\\geq 2m$}. Let $t=k-1$. Since \\fys{$t$} is odd, relation \\eqref{proof:H_kbounds} holds. Writing  \\eqref{proof:H_kbounds} for $k-1$, and taking into account that $H_k=H_{k-1}$, and $\\mu_k<\\mu_{k-1}$, we can conclude that \\eqref{proof:H_kbounds} holds for any even \\fy{$k\\geq 2m-1$} and this completes the proof.\n\\end{proof}","sha256_utf8":"d17069105c4ec45a13666121b7b048ac541f8be33ce2eb7d89f50c632c7f503d"}]},{"id":"S077","semantic_type":"lemma","kind":"lemma","title":"几乎必然收敛的可行调参","source_units":[{"start":70137,"end":70990,"start_line":965,"end_line":973,"raw_tex":"\\begin{lemma}[Feasible tuning sequences for a.s. convergence \\fy{(Proposition \\ref{prop:a.s})}]\\label{lemma:a.s.sequences}\nLet the sequences $\\g_k$ and $\\mu_k$ be given by the following rules:\n\\begin{align}\\label{equ:seq}\n \\g_k=\\frac{\\g_0}{(k+1)^a}, \\quad \\mu_{k}=\\frac{\\fy{2^b\\mu_0}}{\\left(k+\\kappa\\right)^b},\n\\end{align}\nwhere $\\kappa=2$ if $k$ is even and $\\kappa=1$ otherwise, $\\g_0$and $\\mu_0$ are positive scalars such that $\\g_0\\mu_0 \\leq L(m+n)$, \\fy{and $a$, $b$, and $\\delta\\leq 1$ are positive scalars \\fys{satisfying}:\n\\begin{align*}& \\fy{\\frac{a}{b}>1+2\\delta(n+m)}, \\quad a+b \\leq 1,\\quad a+2b > 1, \\quad\\hbox{and}\\quad a-\\delta b(m+n)>0.5.\\end{align*}\nThen, $\\g_k$ and $\\mu_k$ satisfy Assumption \\ref{assum:sequences} with $\\lambda_{\\min}$ and $\\alpha$ in \\eqref{equ:valuesForAssumH_k} and $\\mu_k$ satisfies \\eqref{eqn:mu-k}.}\n\\end{lemma}","sha256_utf8":"2e737e57e6720005e831c6b3f5bbf9f6034564a0dd213f2ff637616ed2a1dce9"}]},{"id":"S078","semantic_type":"proof","kind":"proof","title":"可行调参条件完整证明","source_units":[{"start":70995,"end":71062,"start_line":974,"end_line":976,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{app:feasibleSeqASconv}.\n\\end{proof}","sha256_utf8":"688a220118168d4de3de1bb0920829aabc7c68e439ca510ec849f73f26592065"},{"start":124878,"end":127944,"start_line":1543,"end_line":1563,"raw_tex":"\\begin{proof}\nIn the following, we show that the presented class of sequences satisfy each of the conditions listed in Assumption \\ref{assum:sequences}. Throughout, we let $\\alpha$ denote \\fy{$-(m+n)\\delta$}.\\\\\n(a) Replacing the sequences by their given rules, we obtain \\fy{\\begin{align*} \\g_k\\mu_k^{2\\alpha-1} & =\\frac{\\g_0}{(2^b\\mu_0)^{1-2\\alpha}}(k+1)^{-a}(k+\\kappa)^{(1-2\\alpha) b}\\leq  \\frac{\\g_0}{(2^b\\mu_0)^{1-2\\alpha}}(k+1)^{-a+(1-2\\alpha) b}.\\end{align*}}\n\\fy{From the assumption that $\\frac{a}{b}>1+2\\delta(m+n)$, we obtain$-a+(1-2\\alpha) b<0$}. Thus, the preceding term goes to zero verifying Assumption \\ref{assum:sequences}(a).\\\\\n(b) Let $k$ be an even number. Thus, $\\kappa=2$. From \\eqref{equ:seq} we have $\\mu_{k}=\\mu_{k+1}=\\frac{\\mu_02^b}{\\left(k+2\\right)^b}$. Now, let $k$ be an odd number. Again, according to \\eqref{equ:seq} can write \n\\[\\mu_{k+1}=\\frac{\\mu_02^b}{\\left((k+1)+2\\right)^b }<\\frac{\\mu_02^b}{\\left(k+1\\right)^b}= \\frac{\\mu_02^b}{\\left(k+\\kappa\\right)^b}=\\mu_k.\\]\nTherefore, $\\mu_k$ given by \\eqref{equ:seq} satisfies \\eqref{eqn:mu-k}. Also, from \\eqref{equ:seq} we have $\\mu_k \\to 0$. Thus, Assumption \\ref{assum:sequences}(b) holds.\\\\\n(c) The given rules \\eqref{equ:seq} imply that $\\g_k$ and $\\mu_k$ are both non-increasing sequences. Therefore, we have $\\g_k\\mu_{k} \\leq \\g_0\\mu_{0}$ for any $k\\geq 0$. So, to show that Assumption \\ref{assum:sequences}(c) holds, it is enough to show that $\\lambda_{\\min}\\g_0\\mu_{0} \\leq 1$ where $\\lambda_{\\min}$ is given by \\eqref{equ:valuesForAssumH_k}. Since we assumed that $\\g_0\\mu_0 \\leq L(m+n)$, for any $\\delta \\in (0,1]$, we have $\\g_0\\mu_0 \\leq (m+n)(L+\\mu_0^\\delta)$, implying that $\\lambda_{\\min}\\g_0\\mu_{0} \\leq 1$ and that Assumption \\ref{assum:sequences}(c) holds.\\\\\n(d) From \\eqref{equ:seq}, we can write $$\\sum_{k=0}^\\infty \\g_k\\mu_k =\\g_0\\mu_02^b\\sum_{k=0}^\\infty (k+1)^{-a}(k+\\kappa)^{-b} \\geq \\g_0\\mu_02^b\\sum_{k=0}^\\infty (k+2)^{-(a+b)}= \\infty,$$\nwhere the last relation is due to $a+b\\leq 1$. Therefore, Assumption \\ref{assum:sequences}(d) holds.\\\\\n(e) Using \\eqref{equ:seq}, it follows\n\\fy{\\begin{align*}&\\sum_{k=0}^\\infty \\g_k\\mu_k^2 =  \\g_0\\mu_0^24^b\\sum_{k=0}^\\infty (k+1)^{-a}(k+\\kappa)^{-2b}\\leq   \\g_0\\mu_0^24^b\\sum_{k=0}^\\infty (k+1)^{-(a+2b)}<\\infty,\\end{align*}}\nwhere the last inequality is due to \\an{$a+2b>1$}. Therefore, Assumption \\ref{assum:sequences}(e) holds.\\\\\n(f) From \\eqref{equ:seq}, we have\n\\begin{align*}&\\sum_{k=0}^\\infty \\g_k^2\\mu_k^{2\\alpha} %=  \\g_0^2(\\mu_02^b)^{2\\alpha}\\sum_{k=0}^\\infty \\frac{(k+\\kappa)^{-2\\alpha b}}{(k+1)^{2a}} \n\\leq  \\g_0^2(\\mu_02^b)^{2\\alpha}\\left(\\sum_{k=0}^1 \\frac{(k+\\kappa)^{-2\\alpha b}}{(k+1)^{2a}}+\\sum_{k=2}^\\infty \\frac{(2k)^{-2\\alpha b}}{k^{2a}}\\right)<\\infty \n%&=  \\frac{\\g_0^2}{\\mu_0^24^c}\\left(\\sum_{k=0}^1 \\frac{(k+\\kappa)^{2c}}{(k+1)^{2a}}+4^c\\sum_{k=2}^\\infty k^{2(c-a)}\\right)<\\infty,\n\\end{align*}\nwhere in the first inequality, we use $\\alpha <0$ and in the last inequality, we note that \n$a+\\alpha b=a-\\delta\\left(m+n\\right)b>0.5$. Therefore, Assumption \\ref{assum:sequences}(f) is verified.\n\\end{proof}","sha256_utf8":"c8bce1058c16f01dede5636cf017ab5603647051ced3dafbd59b537ac2695f7e"}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","title":"可行调参证明的附录指引","source_units":[{"start":70995,"end":71062,"start_line":974,"end_line":976,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{app:feasibleSeqASconv}.\n\\end{proof}","sha256_utf8":"688a220118168d4de3de1bb0920829aabc7c68e439ca510ec849f73f26592065"}]},{"id":"S080","semantic_type":"proof_step","kind":"proof_step","title":"逐项验证的目标与谱指数","source_units":[{"start":124878,"end":125089,"start_line":1543,"end_line":1544,"raw_tex":"\\begin{proof}\nIn the following, we show that the presented class of sequences satisfy each of the conditions listed in Assumption \\ref{assum:sequences}. Throughout, we let $\\alpha$ denote \\fy{$-(m+n)\\delta$}.\\\\\n","sha256_utf8":"ed9e05d90b852750ce974cb3d232387da9caf451fddbc44307dc1ddfd8de1a26"}]},{"id":"S081","semantic_type":"proof_step","kind":"proof_step","title":"验证步长正则化比值趋零","source_units":[{"start":125089,"end":125522,"start_line":1545,"end_line":1546,"raw_tex":"(a) Replacing the sequences by their given rules, we obtain \\fy{\\begin{align*} \\g_k\\mu_k^{2\\alpha-1} & =\\frac{\\g_0}{(2^b\\mu_0)^{1-2\\alpha}}(k+1)^{-a}(k+\\kappa)^{(1-2\\alpha) b}\\leq  \\frac{\\g_0}{(2^b\\mu_0)^{1-2\\alpha}}(k+1)^{-a+(1-2\\alpha) b}.\\end{align*}}\n\\fy{From the assumption that $\\frac{a}{b}>1+2\\delta(m+n)$, we obtain$-a+(1-2\\alpha) b<0$}. Thus, the preceding term goes to zero verifying Assumption \\ref{assum:sequences}(a).\\\\\n","sha256_utf8":"67370cf29b9964a56838e87125dd4c0dcae83b94e7f0a382da8e6adc798a2388"}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","title":"验证正则化交替递减并趋零","source_units":[{"start":125522,"end":126048,"start_line":1547,"end_line":1549,"raw_tex":"(b) Let $k$ be an even number. Thus, $\\kappa=2$. From \\eqref{equ:seq} we have $\\mu_{k}=\\mu_{k+1}=\\frac{\\mu_02^b}{\\left(k+2\\right)^b}$. Now, let $k$ be an odd number. Again, according to \\eqref{equ:seq} can write \n\\[\\mu_{k+1}=\\frac{\\mu_02^b}{\\left((k+1)+2\\right)^b }<\\frac{\\mu_02^b}{\\left(k+1\\right)^b}= \\frac{\\mu_02^b}{\\left(k+\\kappa\\right)^b}=\\mu_k.\\]\nTherefore, $\\mu_k$ given by \\eqref{equ:seq} satisfies \\eqref{eqn:mu-k}. Also, from \\eqref{equ:seq} we have $\\mu_k \\to 0$. Thus, Assumption \\ref{assum:sequences}(b) holds.\\\\\n","sha256_utf8":"0affde379b71c84468a8017fb6bc1df972891ae11529c3b765a13b37f9c8b126"}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","title":"验证步长正则化乘积限制","source_units":[{"start":126048,"end":126631,"start_line":1550,"end_line":1550,"raw_tex":"(c) The given rules \\eqref{equ:seq} imply that $\\g_k$ and $\\mu_k$ are both non-increasing sequences. Therefore, we have $\\g_k\\mu_{k} \\leq \\g_0\\mu_{0}$ for any $k\\geq 0$. So, to show that Assumption \\ref{assum:sequences}(c) holds, it is enough to show that $\\lambda_{\\min}\\g_0\\mu_{0} \\leq 1$ where $\\lambda_{\\min}$ is given by \\eqref{equ:valuesForAssumH_k}. Since we assumed that $\\g_0\\mu_0 \\leq L(m+n)$, for any $\\delta \\in (0,1]$, we have $\\g_0\\mu_0 \\leq (m+n)(L+\\mu_0^\\delta)$, implying that $\\lambda_{\\min}\\g_0\\mu_{0} \\leq 1$ and that Assumption \\ref{assum:sequences}(c) holds.\\\\\n","sha256_utf8":"1ab805781664a308fa9112a4ea9f4f0cf94c2472eedc5118ad92d1aa13ccf56f"}]},{"id":"S084","semantic_type":"proof_step","kind":"proof_step","title":"验证收缩系数之和发散","source_units":[{"start":126631,"end":126921,"start_line":1551,"end_line":1552,"raw_tex":"(d) From \\eqref{equ:seq}, we can write $$\\sum_{k=0}^\\infty \\g_k\\mu_k =\\g_0\\mu_02^b\\sum_{k=0}^\\infty (k+1)^{-a}(k+\\kappa)^{-b} \\geq \\g_0\\mu_02^b\\sum_{k=0}^\\infty (k+2)^{-(a+b)}= \\infty,$$\nwhere the last relation is due to $a+b\\leq 1$. Therefore, Assumption \\ref{assum:sequences}(d) holds.\\\\\n","sha256_utf8":"6b326ba69985519cb3fbe7e5decd35da09f63526a228207f41c7d8e32f68d724"}]},{"id":"S085","semantic_type":"proof_step","kind":"proof_step","title":"验证正则化偏差可求和","source_units":[{"start":126921,"end":127252,"start_line":1553,"end_line":1555,"raw_tex":"(e) Using \\eqref{equ:seq}, it follows\n\\fy{\\begin{align*}&\\sum_{k=0}^\\infty \\g_k\\mu_k^2 =  \\g_0\\mu_0^24^b\\sum_{k=0}^\\infty (k+1)^{-a}(k+\\kappa)^{-2b}\\leq   \\g_0\\mu_0^24^b\\sum_{k=0}^\\infty (k+1)^{-(a+2b)}<\\infty,\\end{align*}}\nwhere the last inequality is due to \\an{$a+2b>1$}. Therefore, Assumption \\ref{assum:sequences}(e) holds.\\\\\n","sha256_utf8":"35e6036268f66e7032c88f8143fee9cd261b0f11d3cd010cc122e89bf6832424"}]},{"id":"S086","semantic_type":"proof_step","kind":"proof_step","title":"验证随机方差项可求和","source_units":[{"start":127252,"end":127944,"start_line":1556,"end_line":1563,"raw_tex":"(f) From \\eqref{equ:seq}, we have\n\\begin{align*}&\\sum_{k=0}^\\infty \\g_k^2\\mu_k^{2\\alpha} %=  \\g_0^2(\\mu_02^b)^{2\\alpha}\\sum_{k=0}^\\infty \\frac{(k+\\kappa)^{-2\\alpha b}}{(k+1)^{2a}} \n\\leq  \\g_0^2(\\mu_02^b)^{2\\alpha}\\left(\\sum_{k=0}^1 \\frac{(k+\\kappa)^{-2\\alpha b}}{(k+1)^{2a}}+\\sum_{k=2}^\\infty \\frac{(2k)^{-2\\alpha b}}{k^{2a}}\\right)<\\infty \n%&=  \\frac{\\g_0^2}{\\mu_0^24^c}\\left(\\sum_{k=0}^1 \\frac{(k+\\kappa)^{2c}}{(k+1)^{2a}}+4^c\\sum_{k=2}^\\infty k^{2(c-a)}\\right)<\\infty,\n\\end{align*}\nwhere in the first inequality, we use $\\alpha <0$ and in the last inequality, we note that \n$a+\\alpha b=a-\\delta\\left(m+n\\right)b>0.5$. Therefore, Assumption \\ref{assum:sequences}(f) is verified.\n\\end{proof}","sha256_utf8":"71d387975b376d05244474e991d1f72a61d6f8f76aa98a7cd89524bcbe75af54"}]},{"id":"S087","semantic_type":"lemma","kind":"lemma","title":"均值收敛的可行调参","source_units":[{"start":71334,"end":72130,"start_line":983,"end_line":993,"raw_tex":"\\begin{lemma}[Feasible tuning sequences for convergence in mean \\fy{(Proposition \\ref{prop:mean})}]\\label{lemma:mean-sequences}\nLet the sequences $\\g_k$ and $\\mu_k$ be given by \\eqref{equ:seq},\nwhere $\\g_0$ and $\\mu_0$ are positive scalars such that $\\g_0\\mu_0 \\leq L(m+n)$. \\fy{Let, $0 <\\delta \\leq 1$ $a>0$ and $b>0$ \\fys{satisfying}:\n\\begin{align*}& \\frac{a}{b}>1+2\\delta\\left(m+n\\right), \\quad a+b < 1,\n\\quad \\frac{a}{b} \\leq\n2\\left(1+\\delta\\left(m+n\\right)\\right).\\end{align*}Then, \\vus{$\\mu_k$\nsatisfies \\eqref{eqn:mu-k}} and $\\g_k$ and $\\mu_k$ satisfy Assumption\n\\ref{assum:sequences-ms-convergence} with any arbitrary $0<\\beta<1$,\n$\\rho\\triangleq \\g_0^{-1}\\left(\\mu_02^b\\right)^{2+2\\delta(m+n)}$,\nand with $\\lambda_{\\min}$ and $\\alpha$ given by\n\\eqref{equ:valuesForAssumH_k}.} \\end{lemma}","sha256_utf8":"fc4dd64d82e93933630fa82dbdfec0f671be6e57b3a92f84d688dabbb199eea7"}]},{"id":"S088","semantic_type":"proof","kind":"proof","title":"均值调参条件完整证明","source_units":[{"start":72135,"end":72204,"start_line":994,"end_line":996,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{app:feasibleSeqMeanConv}.\n\\end{proof}","sha256_utf8":"dd02dee6658ed8b898db306cec12e03b3adaa35417bc42bb70eda9f2c8dd90ee"},{"start":128032,"end":130148,"start_line":1566,"end_line":1582,"raw_tex":"\\begin{proof}\n\\fy{Throughout, we let $\\alpha$ denote $-\\delta(m+n)$.} \\fys{Assumption \\ref{assum:sequences-ms-convergence}(a, b, c) and \\eqref{eqn:mu-k} have been already shown in parts (a, b, c) of the proof of Lemma \\ref{lemma:a.s.sequences}.}\\\\\n\\noindent\n%(d) From part (a), since the term $\\g_k\\mu_k^{2\\alpha-1}$ goes to zero, there exist $K_2>0$ such that $\\g_k\\mu_k^{2\\alpha-1}\\leq \\frac{\\lambda_{\\min}}{\\lambda^2(L+\\mu_0)}$ for all $k \\geq K_2$, implying that Assumption \\ref{assum:sequences-ms-convergence}(d) holds for a large enough $k$. \\\\\n%\\noindent\n(d) It suffices to show there exists \\fy{$K_0$} such that for any $k \\geq K_0$ and $0<\\beta<1$, \n\\begin{align}\\label{ineq:partd}\\frac{\\g_{k-1}}{\\g_k}\\frac{\\mu_{k}^{1-2\\alpha}}{\\mu_{k-1}^{1-2\\alpha}}-1\\leq \\beta \\lambda_{\\min}\\g_k\\mu_k.\\end{align}\nFrom \\eqref{equ:seq} \\fy{and the definition of $\\alpha$}, we obtain \n\\begin{align*}&\\frac{\\g_{k-1}}{\\g_k}\\frac{\\mu_{k}^{1-2\\alpha}}{\\mu_{k-1}^{1-2\\alpha}}-1\n\\leq \\frac{\\g_{k-1}}{\\g_k}-1=\\left(1+\\frac{1}{k}\\right)^a-1= 1+\\frac{a}{k}+\\fy{{o}\\left(\\frac{1}{k}\\right)}-1=\\mathcal{O}\\left(\\frac{1}{k}\\right),\n\\end{align*}\nwhere the first inequality is implied due to $\\{\\mu_k\\}$ is non-increasing, and in the second equation, we used the Taylor's expansion of $\\left(1+\\frac{1}{k}\\right)^a$. Therefore, since the right-hand side of \\eqref{ineq:partd} is of the order $\\frac{1}{k^{a+b}}$ and that $a+b<1$, the preceding inequality shows that such $K_0$ exists for which Assumption \\ref{assum:sequences-ms-convergence}(d) holds for all $0<\\beta<1$.\\\\\n\\noindent\n(e) From \\eqref{equ:seq}, we have \n\\begin{align*} \\frac{\\mu_k^{2-2\\alpha}}{\\g_k}& =\\fy{\\g_0}^{-1}\\left(\\mu_02^b\\right)^{2-2\\alpha}(k+\\kappa)^{-b(2-2\\alpha)}(k+1)^{a} \\leq\\frac{\\fy{\\gamma_0}^{-1}\\left(\\mu_02^b\\right)^{2-2\\alpha}}{(k+1)^{-a+(2-2\\alpha)b}}\\\\&\\leq \\fy{\\gamma_0}^{-1}\\left(\\mu_02^b\\right)^{2-2\\alpha}=\\rho, \\end{align*}\nwhere the first inequality is due to \\fy{$\\alpha <0$, and the second inequality follows by the assumption $a\\leq 2b(1+\\delta(m+n))$}. Therefore, Assumption \\ref{assum:sequences-ms-convergence}(e) is satisfied.\n\\end{proof}","sha256_utf8":"c07c1cd0db350bf7f3a7a6dc5650d833ae5052a4a6fdc205a7f6d925e49e6738"}]},{"id":"S089","semantic_type":"proof_step","kind":"proof_step","title":"均值调参证明的附录指引","source_units":[{"start":72135,"end":72204,"start_line":994,"end_line":996,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{app:feasibleSeqMeanConv}.\n\\end{proof}","sha256_utf8":"dd02dee6658ed8b898db306cec12e03b3adaa35417bc42bb70eda9f2c8dd90ee"}]},{"id":"S090","semantic_type":"proof_step","kind":"proof_step","title":"复用此前证明的前三项验证","source_units":[{"start":128032,"end":128290,"start_line":1566,"end_line":1568,"raw_tex":"\\begin{proof}\n\\fy{Throughout, we let $\\alpha$ denote $-\\delta(m+n)$.} \\fys{Assumption \\ref{assum:sequences-ms-convergence}(a, b, c) and \\eqref{eqn:mu-k} have been already shown in parts (a, b, c) of the proof of Lemma \\ref{lemma:a.s.sequences}.}\\\\\n\\noindent\n","sha256_utf8":"aa769fd422e942384a5768ca65bb1667ecca6514f8788ffa1de04c31b87c5534"}]},{"id":"S091","semantic_type":"proof_step","kind":"proof_step","title":"泰勒展开验证相邻序列条件","source_units":[{"start":128594,"end":129585,"start_line":1571,"end_line":1577,"raw_tex":"(d) It suffices to show there exists \\fy{$K_0$} such that for any $k \\geq K_0$ and $0<\\beta<1$, \n\\begin{align}\\label{ineq:partd}\\frac{\\g_{k-1}}{\\g_k}\\frac{\\mu_{k}^{1-2\\alpha}}{\\mu_{k-1}^{1-2\\alpha}}-1\\leq \\beta \\lambda_{\\min}\\g_k\\mu_k.\\end{align}\nFrom \\eqref{equ:seq} \\fy{and the definition of $\\alpha$}, we obtain \n\\begin{align*}&\\frac{\\g_{k-1}}{\\g_k}\\frac{\\mu_{k}^{1-2\\alpha}}{\\mu_{k-1}^{1-2\\alpha}}-1\n\\leq \\frac{\\g_{k-1}}{\\g_k}-1=\\left(1+\\frac{1}{k}\\right)^a-1= 1+\\frac{a}{k}+\\fy{{o}\\left(\\frac{1}{k}\\right)}-1=\\mathcal{O}\\left(\\frac{1}{k}\\right),\n\\end{align*}\nwhere the first inequality is implied due to $\\{\\mu_k\\}$ is non-increasing, and in the second equation, we used the Taylor's expansion of $\\left(1+\\frac{1}{k}\\right)^a$. Therefore, since the right-hand side of \\eqref{ineq:partd} is of the order $\\frac{1}{k^{a+b}}$ and that $a+b<1$, the preceding inequality shows that such $K_0$ exists for which Assumption \\ref{assum:sequences-ms-convergence}(d) holds for all $0<\\beta<1$.\\\\\n","sha256_utf8":"97ad39bd022ec5b780a24e8d9bf046f0db2212e4118832b5b83bfd00a3a8631c"}]},{"id":"S092","semantic_type":"proof_step","kind":"proof_step","title":"验证正则化偏差与步长平衡","source_units":[{"start":129585,"end":130148,"start_line":1578,"end_line":1582,"raw_tex":"\\noindent\n(e) From \\eqref{equ:seq}, we have \n\\begin{align*} \\frac{\\mu_k^{2-2\\alpha}}{\\g_k}& =\\fy{\\g_0}^{-1}\\left(\\mu_02^b\\right)^{2-2\\alpha}(k+\\kappa)^{-b(2-2\\alpha)}(k+1)^{a} \\leq\\frac{\\fy{\\gamma_0}^{-1}\\left(\\mu_02^b\\right)^{2-2\\alpha}}{(k+1)^{-a+(2-2\\alpha)b}}\\\\&\\leq \\fy{\\gamma_0}^{-1}\\left(\\mu_02^b\\right)^{2-2\\alpha}=\\rho, \\end{align*}\nwhere the first inequality is due to \\fy{$\\alpha <0$, and the second inequality follows by the assumption $a\\leq 2b(1+\\delta(m+n))$}. Therefore, Assumption \\ref{assum:sequences-ms-convergence}(e) is satisfied.\n\\end{proof}","sha256_utf8":"a8a41d02b03b8085d6f4116bb4136fdd2277de0aac449d8d021df6ef1a616ab4"}]},{"id":"S093","semantic_type":"external_fact","kind":"external_fact","title":"强凸光滑目标的标准性质","source_units":[],"declared_external":true,"declaration_text":["A differentiable μ-strongly convex function on R^n has a unique minimizer. With an L-Lipschitz gradient it satisfies 2μ(h(x)-min h) ≤ ||∇h(x)||² ≤ 2L(h(x)-min h). Quadratic regularization adds its parameter to the strong-convexity and gradient-Lipschitz bounds."]},{"id":"S094","semantic_type":"external_fact","kind":"external_fact","title":"凸光滑梯度的单调与余强制性","source_units":[],"declared_external":true,"declaration_text":["For differentiable convex L-smooth h, ∇h is monotone and ||∇h(y)-∇h(x)||² ≤ L(y-x)^T(∇h(y)-∇h(x))."]},{"id":"S095","semantic_type":"external_fact","kind":"external_fact","title":"BFGS逆公式与行列式恒等式","source_units":[],"declared_external":true,"declaration_text":["If B is positive definite and s^Ty>0, B+ = B-Bss^TB/(s^TBs)+yy^T/(s^Ty) is positive definite. Sherman–Morrison–Woodbury gives (B+)^{-1}=(I-sy^T/(s^Ty))B^{-1}(I-ys^T/(s^Ty))+ss^T/(s^Ty), and det(B+)=det(B)(s^Ty)/(s^TBs)."]},{"id":"S096","semantic_type":"external_fact","kind":"external_fact","title":"概率线性代数与初等分析事实","source_units":[],"declared_external":true,"declaration_text":["Use the tower property and measurable-factor rule for conditional expectation, Cauchy–Schwarz, positive-definite spectral and trace identities, Taylor expansion (1+t)^a=1+at+o(t), and convergence of Σk^{-p} exactly for p>1. Expectation preserves convexity and uniform gradient Lipschitz bounds under the stated unbiasedness assumptions."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S009","kind":"supports"},{"source_id":"S005","target_id":"S011","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S006","target_id":"S010","kind":"supports"},{"source_id":"S006","target_id":"S002","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S002","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S010","kind":"supports"},{"source_id":"S008","target_id":"S002","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S002","kind":"supports"},{"source_id":"S010","target_id":"S014","kind":"supports"},{"source_id":"S010","target_id":"S002","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S002","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S002","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S002","kind":"supports"},{"source_id":"S014","target_id":"S002","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S015","target_id":"S046","kind":"supports"},{"source_id":"S016","target_id":"S036","kind":"supports"},{"source_id":"S016","target_id":"S047","kind":"supports"},{"source_id":"S017","target_id":"S036","kind":"supports"},{"source_id":"S017","target_id":"S069","kind":"supports"},{"source_id":"S018","target_id":"S030","kind":"supports"},{"source_id":"S018","target_id":"S046","kind":"supports"},{"source_id":"S019","target_id":"S022","kind":"supports"},{"source_id":"S019","target_id":"S039","kind":"supports"},{"source_id":"S019","target_id":"S041","kind":"supports"},{"source_id":"S019","target_id":"S003","kind":"supports"},{"source_id":"S020","target_id":"S038","kind":"supports"},{"source_id":"S020","target_id":"S039","kind":"supports"},{"source_id":"S020","target_id":"S040","kind":"supports"},{"source_id":"S021","target_id":"S030","kind":"supports"},{"source_id":"S021","target_id":"S037","kind":"supports"},{"source_id":"S021","target_id":"S045","kind":"supports"},{"source_id":"S021","target_id":"S046","kind":"supports"},{"source_id":"S021","target_id":"S052","kind":"supports"},{"source_id":"S022","target_id":"S037","kind":"supports"},{"source_id":"S022","target_id":"S039","kind":"supports"},{"source_id":"S022","target_id":"S041","kind":"supports"},{"source_id":"S023","target_id":"S080","kind":"supports"},{"source_id":"S024","target_id":"S050","kind":"supports"},{"source_id":"S024","target_id":"S051","kind":"supports"},{"source_id":"S024","target_id":"S055","kind":"supports"},{"source_id":"S024","target_id":"S058","kind":"supports"},{"source_id":"S024","target_id":"S090","kind":"supports"},{"source_id":"S024","target_id":"S091","kind":"supports"},{"source_id":"S024","target_id":"S092","kind":"supports"},{"source_id":"S025","target_id":"S069","kind":"supports"},{"source_id":"S025","target_id":"S076","kind":"supports"},{"source_id":"S025","target_id":"S082","kind":"supports"},{"source_id":"S026","target_id":"S068","kind":"supports"},{"source_id":"S026","target_id":"S071","kind":"supports"},{"source_id":"S027","target_id":"S005","kind":"supports"},{"source_id":"S027","target_id":"S006","kind":"supports"},{"source_id":"S027","target_id":"S014","kind":"supports"},{"source_id":"S027","target_id":"S067","kind":"supports"},{"source_id":"S027","target_id":"S069","kind":"supports"},{"source_id":"S027","target_id":"S070","kind":"supports"},{"source_id":"S027","target_id":"S075","kind":"supports"},{"source_id":"S027","target_id":"S076","kind":"supports"},{"source_id":"S028","target_id":"S003","kind":"supports"},{"source_id":"S028","target_id":"S068","kind":"supports"},{"source_id":"S028","target_id":"S071","kind":"supports"},{"source_id":"S029","target_id":"S003","kind":"supports"},{"source_id":"S029","target_id":"S004","kind":"supports"},{"source_id":"S029","target_id":"S005","kind":"supports"},{"source_id":"S029","target_id":"S007","kind":"supports"},{"source_id":"S029","target_id":"S009","kind":"supports"},{"source_id":"S029","target_id":"S011","kind":"supports"},{"source_id":"S029","target_id":"S014","kind":"supports"},{"source_id":"S030","target_id":"S036","kind":"supports"},{"source_id":"S030","target_id":"S044","kind":"supports"},{"source_id":"S030","target_id":"S046","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S034","target_id":"S050","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S037","target_id":"S042","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S035","kind":"supports"},{"source_id":"S039","target_id":"S042","kind":"supports"},{"source_id":"S039","target_id":"S035","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S035","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S035","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S035","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S035","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S035","kind":"supports"},{"source_id":"S045","target_id":"S047","kind":"supports"},{"source_id":"S045","target_id":"S035","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S035","kind":"supports"},{"source_id":"S047","target_id":"S035","kind":"supports"},{"source_id":"S048","target_id":"S003","kind":"supports"},{"source_id":"S048","target_id":"S004","kind":"supports"},{"source_id":"S049","target_id":"S048","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S051","target_id":"S054","kind":"supports"},{"source_id":"S051","target_id":"S049","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S052","target_id":"S049","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S053","target_id":"S058","kind":"supports"},{"source_id":"S053","target_id":"S049","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S049","kind":"supports"},{"source_id":"S055","target_id":"S057","kind":"supports"},{"source_id":"S055","target_id":"S049","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S056","target_id":"S049","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S049","kind":"supports"},{"source_id":"S058","target_id":"S049","kind":"supports"},{"source_id":"S059","target_id":"S074","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S062","target_id":"S060","kind":"supports"},{"source_id":"S063","target_id":"S060","kind":"supports"},{"source_id":"S064","target_id":"S003","kind":"supports"},{"source_id":"S064","target_id":"S083","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S069","kind":"supports"},{"source_id":"S067","target_id":"S065","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S065","kind":"supports"},{"source_id":"S069","target_id":"S070","kind":"supports"},{"source_id":"S069","target_id":"S065","kind":"supports"},{"source_id":"S070","target_id":"S072","kind":"supports"},{"source_id":"S070","target_id":"S073","kind":"supports"},{"source_id":"S070","target_id":"S075","kind":"supports"},{"source_id":"S070","target_id":"S065","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S073","kind":"supports"},{"source_id":"S071","target_id":"S065","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S074","kind":"supports"},{"source_id":"S072","target_id":"S075","kind":"supports"},{"source_id":"S072","target_id":"S065","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S065","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S065","kind":"supports"},{"source_id":"S075","target_id":"S076","kind":"supports"},{"source_id":"S075","target_id":"S065","kind":"supports"},{"source_id":"S076","target_id":"S065","kind":"supports"},{"source_id":"S078","target_id":"S077","kind":"supports"},{"source_id":"S078","target_id":"S088","kind":"supports"},{"source_id":"S079","target_id":"S078","kind":"supports"},{"source_id":"S080","target_id":"S081","kind":"supports"},{"source_id":"S080","target_id":"S086","kind":"supports"},{"source_id":"S080","target_id":"S078","kind":"supports"},{"source_id":"S081","target_id":"S090","kind":"supports"},{"source_id":"S081","target_id":"S078","kind":"supports"},{"source_id":"S082","target_id":"S083","kind":"supports"},{"source_id":"S082","target_id":"S090","kind":"supports"},{"source_id":"S082","target_id":"S078","kind":"supports"},{"source_id":"S083","target_id":"S090","kind":"supports"},{"source_id":"S083","target_id":"S078","kind":"supports"},{"source_id":"S084","target_id":"S078","kind":"supports"},{"source_id":"S085","target_id":"S078","kind":"supports"},{"source_id":"S086","target_id":"S078","kind":"supports"},{"source_id":"S087","target_id":"S003","kind":"supports"},{"source_id":"S088","target_id":"S087","kind":"supports"},{"source_id":"S089","target_id":"S088","kind":"supports"},{"source_id":"S090","target_id":"S091","kind":"supports"},{"source_id":"S090","target_id":"S092","kind":"supports"},{"source_id":"S090","target_id":"S088","kind":"supports"},{"source_id":"S091","target_id":"S088","kind":"supports"},{"source_id":"S092","target_id":"S088","kind":"supports"},{"source_id":"S093","target_id":"S030","kind":"supports"},{"source_id":"S093","target_id":"S032","kind":"supports"},{"source_id":"S093","target_id":"S033","kind":"supports"},{"source_id":"S093","target_id":"S071","kind":"supports"},{"source_id":"S094","target_id":"S068","kind":"supports"},{"source_id":"S094","target_id":"S071","kind":"supports"},{"source_id":"S095","target_id":"S070","kind":"supports"},{"source_id":"S095","target_id":"S072","kind":"supports"},{"source_id":"S095","target_id":"S073","kind":"supports"},{"source_id":"S096","target_id":"S003","kind":"supports"},{"source_id":"S096","target_id":"S039","kind":"supports"},{"source_id":"S096","target_id":"S041","kind":"supports"},{"source_id":"S096","target_id":"S050","kind":"supports"},{"source_id":"S096","target_id":"S071","kind":"supports"},{"source_id":"S096","target_id":"S072","kind":"supports"},{"source_id":"S096","target_id":"S074","kind":"supports"},{"source_id":"S096","target_id":"S084","kind":"supports"},{"source_id":"S096","target_id":"S085","kind":"supports"},{"source_id":"S096","target_id":"S086","kind":"supports"},{"source_id":"S096","target_id":"S091","kind":"supports"}]}}
{"record_id":"1711.05812v2","paper":{"title":"Iteration complexity of inexact augmented Lagrangian methods for constrained convex programming","authors":["Yangyang Xu"],"arxiv_id":"1711.05812","version":"v2","journal":"Mathematical Programming","doi":"10.1007/s10107-019-01425-9","abstract_url":"https://arxiv.org/abs/1711.05812v2","pdf_url":"https://arxiv.org/pdf/1711.05812v2"},"context":{"raw_tex":"In this paper, we consider the constrained convex programming\n\\begin{equation}\\label{eq:ccp}\n\\Min_{\\vx\\in\\cX} f_0(\\vx), \\st \\vA\\vx=\\vb, f_i(\\vx)\\le 0, i=1,\\ldots, m,\n\\end{equation}\nwhere $\\cX\\subseteq\\RR^n$ is a closed convex set, $\\vA$ and $\\vb$ are respectively given matrix and vector, and $f_i$ is a convex function for every $i=0,1,\\ldots,m$. Any convex optimization problem can be written in the standard form of \\eqref{eq:ccp}. It appears in many areas including statistics, machine learning, data mining, engineering, signal processing, finance, operations research, and so on. \n\n\nWe will assume composite convex structure on \\eqref{eq:ccp}. More specifically, we assume \\begin{equation}\\label{eq:f0}\nf_0(\\vx)=g(\\vx)+h(\\vx),\n\\end{equation} \nwhere $g$ is a Lipschitz differentiable convex function, and $h$ is a simple\\footnote{By ``simple'', we mean the proximal mapping of $h$ is easy to evaluate, i.e., it is easy to find a solution to $\\min_{\\vx\\in\\cX} h(\\vx) + \\frac{1}{2\\gamma}\\|\\vx-\\hat{\\vx}\\|^2$ for any $\\hat{\\vx}$ and $\\gamma>0$.} (possibly nondifferentiable) convex function. Also, $f_i$ is convex and Lipschitz differentiable for every $i\\in [m]$, namely, there are constants $L_0,L_1,\\ldots, L_m$ such that\n\\begin{subequations}\\label{eq:lip-gf}\n\\begin{align}\n\\|\\nabla g(\\hat{\\vx})-\\nabla g(\\tilde{\\vx})\\|\\le L_0\\|\\hat{\\vx}-\\tilde{\\vx}\\|,\\,&\\forall\\, \\hat{\\vx}, \\tilde{\\vx} \\in \\dom(h)\\cap \\cX,\\label{eq:lip-g}\\\\\n\\|\\nabla f_i(\\hat{\\vx})-\\nabla f_i(\\tilde{\\vx})\\|\\le L_i\\|\\hat{\\vx}-\\tilde{\\vx}\\|,\\,& \\forall\\, \\hat{\\vx}, \\tilde{\\vx} \\in \\dom(h) \\cap \\cX, \\forall i\\in [m].\\label{eq:lip-f}\n\\end{align} \n\\end{subequations}\nIn addition, we assume the boundedness of $\\dom(h)\\cap\\cX$ and denote its diameter as \n$$D=\\Max_{\\hat{\\vx},\\tilde{\\vx}\\in \\dom(h)\\cap\\cX}\\|\\hat{\\vx}-\\tilde{\\vx}\\|.$$\n\n\n\\begin{equation}\\label{eq:def-psi}\n\\psi_\\beta(u,v) = \\left\\{\\begin{array}{ll}uv+\\frac{\\beta}{2}u^2,\\,&\\text{ if }\\beta u +v \\ge0,\\\\[0.2cm]\n-\\frac{v^2}{2\\beta},\\,&\\text{ if }\\beta u +v < 0.\\end{array}\\right.\n\\end{equation}\nLet\n$$\\Psi_\\beta(\\vx,\\vz)=\\sum_{i=1}^m\\psi_\\beta(f_i(\\vx),z_i),$$\nand we obtain the classical AL function of \\eqref{eq:ccp}:\n\\begin{equation}\\label{eq:aug-fun}\n\\cL_\\beta(\\vx,\\vy,\\vz) = f_0(\\vx) + \\vy^\\top(\\vA\\vx-\\vb) + \\frac{\\beta}{2}\\|\\vA\\vx-\\vb\\|^2 + \\Psi_\\beta(\\vx,\\vz).\n\\end{equation}\n\n\n\\begin{algorithm}[h]\n\\caption{Inexact augmented Lagrangian method for \\eqref{eq:ccp}}\\label{alg:ialm}\n\\DontPrintSemicolon\n\\textbf{Initialization:} choose $\\vx^0, \\vy^0, \\vz^0$ and $\\{\\beta_k,\\rho_k\\}$\\;\n\\For{$k=0,1,\\ldots$}{\nFind $\\vx^{k+1}\\in\\cX$ such that \n\\begin{equation}\\label{eq:ialm-x}\n\\cL_{\\beta_k}(\\vx^{k+1}, \\vy^k , \\vz^k  ) \\le \\min_{\\vx\\in\\cX} \\cL_{\\beta_k}(\\vx, \\vy^k , \\vz^k  ) + \\vareps_k.\n\\end{equation}\n\nUpdate $\\vy$ and $\\vz$ by\n\\begin{align}\n\\vy^{k+1} =&~\\vy^k  + \\rho_k (\\vA\\vx^{k+1}-\\vb),\\label{eq:alm-y}\\\\\nz^{k+1} _i=&~z_i^k+\\rho_k \\cdot\\max\\left(-\\frac{z_i^k}{\\beta_k}, f_i(\\vx^{k+1})\\right), i=1,\\ldots, m.\\label{eq:alm-z}\n\\end{align}\n}\n\\end{algorithm}\n\nWe use italic letters $a, c, B, L,\\ldots,$ for scalars, bold lower-case letters $\\vx,\\vy,\\vz,\\ldots$ for vectors, and bold upper-case letters $\\vA,\\vB,\\ldots$ for matrices. $z_i$ denotes the $i$-th entry of a vector $\\vz$. We use $\\vzero$ to denote a vector of all zeros, and its size is clear from the context. $[m]$ denotes the set $\\{1,2,\\ldots,m\\}$ for any positive integer $m$. Given a real number $a$, we let $[a]_+=\\max(0,a)$ and $\\lceil a\\rceil$ be the smallest integer that is no less than $a$. For a vector $\\va$, $[\\va]_+$ takes the positive part of $\\va$ in a component-wise manner. $\\|\\va\\|$ denotes the Euclidean norm of a vector $\\va$ and $\\|\\vA\\|$ the spectral norm of a matrix $\\vA$. %By $p(k)\\lesssim q(k)$, we mean $\\limsup_k\\frac{p(k)}{q(k)} \\le 1$, i.e., $p(k)$ is dominated by $q(k)$. Given a symmetric positive definite (SPD) matrix $\\vP$, we denote $\\|\\cdot\\|_\\vP$ as $\\vP$-weighted norm, i.e., $\\|\\vw\\|_\\vP=\\sqrt{\\vw^\\top \\vP \\vw}$ . If $\\vP=\\vI$, we simply write it as $\\|\\cdot\\|$. \n\nWe denote $\\vell$ as the vector consisting of $L_i, i\\in [m]$, where $L_i$ is the Lipschitz constant of $\\nabla f_i$ in \\eqref{eq:lip-f}. Also we let $\\vf$ be the vector function with $f_i$ as the $i$-th component scalar function. That is \n\\begin{equation}\\label{eq:vec-ell}\n\\vell=[L_1,\\ldots,L_m], \\quad \\vf(\\vx)=[f_1(\\vx),\\ldots, f_m(\\vx)].\n\\end{equation}\nGiven a convex function $f$, $\\tilde{\\nabla}f(\\vx)$ represents one subgradient of $f$ at $\\vx$, namely,\n$$f(\\hat{\\vx})\\ge f(\\vx) + \\langle \\tilde{\\nabla} f(\\vx), \\hat{\\vx}-\\vx\\rangle,\\,\\forall\\, \\hat{\\vx},$$ and $\\partial f(\\vx)$ denotes its subdifferential, i.e., the set of all subgradients. When $f$ is differentiable, we simply write its subgradient as $\\nabla f(\\vx)$. For a convex set $\\cX$, we use $\\iota_\\cX$ as its indicator function, i.e., \n$$\\iota_\\cX(\\vx)=\\left\\{\\begin{array}{ll}0,&\\text{ if }\\vx\\in\\cX,\\\\+\\infty,&\\text{ if }\\vx\\not\\in\\cX,\\end{array}\\right.$$\nand $\\cN_\\cX(\\vx)=\\partial \\iota_\\cX(\\vx)$ as its normal cone at $\\vx\\in \\cX$. \n\n\n\\begin{definition}[$\\vareps$-optimal solution]\\label{def:eps-opt}\nLet $f_0^*$ be the optimal value of \\eqref{eq:ccp}. Given $\\vareps\\ge 0$, a point $\\vx\\in\\cX$ is called an $\\vareps$-optimal solution to \\eqref{eq:ccp} if \n$$|f_0(\\vx)-f_0^*| \\le \\vareps,\\text{ and }\\|\\vA\\vx-\\vb\\|+\\big\\| [\\vf(\\vx)]_+\\big\\| \\le \\vareps.$$\n\\end{definition}\n\nA point $(\\vx,\\vy,\\vz)$ satisfies the Karush-Kuhn-Tucker (KKT) conditions for \\eqref{eq:ccp} if\n\\begin{subequations}\\label{eq:kkt}\n\\begin{align}\n0\\in \\partial f_0(\\vx) +\\cN_\\cX(\\vx) + \\vA^\\top \\vy + \\sum_{i=1}^m z_i \\nabla f_i(\\vx), \\label{eq:kkt-1st}&\\\\\n\\vA\\vx = \\vb, \\quad \\vx\\in \\cX, \\label{eq:kkt-res}&\\\\\nz_i\\ge0, \\quad f_i(\\vx)\\le 0,\\quad z_if_i(\\vx)=0, \\forall i\\in [m]. \\label{eq:kkt-cp}\n\\end{align}\n\\end{subequations}\nFrom the convexity of $f_i$'s, if $(\\vx^*,\\vy^*,\\vz^*)$ is a KKT point, then\n\\begin{equation}\\label{eq:opt}\nf_0(\\vx)-f_0(\\vx^*)+\\langle \\vy^*, \\vA\\vx-\\vb\\rangle + \\sum_{i=1}^m z^*_i f_i(\\vx) \\ge 0,\\, \\forall\\, \\vx\\in\\cX.\n\\end{equation}\n\n\nIn this subsection, we review Nesterov's optimal first-order method for composite convex programs. The method will be used to approximately solve $\\vx$-subproblems in Algorithm \\ref{alg:ialm}. It aims at finding a solution of the following problem\n\\begin{equation}\\label{eq:comp-prob}\n\\Min_\\vx \\phi(\\vx) + \\psi(\\vx),\n\\end{equation}\nwhere $\\phi$ is a Lipschitz differentiable and strongly convex function with gradient Lipschitz constant $L_\\phi$ and strong convexity modulus $\\mu\\ge0$, and $\\psi$ is a simple (possibly nondifferentiable) closed convex function. Algorithm \\ref{alg:apg} summarizes the method. Here, for simplicity, we assume $L_\\phi$ and $\\mu$ are known. The method does not require the value of $L_\\phi$ but can estimate it by backtracking. In addition, it only requires a lower estimate of $\\mu$; see \\cite{nesterov2013gradient} for example.  %For our purpose, we will have $\\mu=1$; see the problem \\eqref{eq:ipalm-x} in Algorithm \\ref{alg:ipalm}.\n\n\n\\begin{algorithm}\\caption{Nesterov's optimal first-order method for \\eqref{eq:comp-prob}}\\label{alg:apg}\n\\textbf{Initialization:} choose $\\hat{\\vx}^0=\\vx^0$, $\\alpha_0\\in (0,1]$, and let $q=\\frac{\\mu}{L_\\phi}$\\; \n\\For{$k=0,1,\\ldots, $}{\nLet \n$$\\vx^{k+1}=\\argmin_{\\vx} \\langle \\nabla \\phi(\\hat{\\vx}^k), \\vx\\rangle + \\frac{L_\\phi}{2}\\|\\vx-\\hat{\\vx}^k\\|^2+\\psi(\\vx).$$\n\nSet $$\\alpha_{k+1}=\\frac{q-\\alpha_k^2+\\sqrt{(q-\\alpha_k^2)^2+4\\alpha_k^2}}{2},$$ and\n$$\\hat{\\vx}^{k+1}=\\vx^{k+1}+\\frac{\\alpha_k(1-\\alpha_k)}{\\alpha_k^2+\\alpha_{k+1}}(\\vx^{k+1}-\\vx^k ).$$\n}\n\\end{algorithm}\n\n\\begin{assumption}\\label{assump-kkt}\nThere exists a point $(\\vx^*,\\vy^*,\\vz^*)$ satisfying the KKT conditions in \\eqref{eq:kkt}.\n\\end{assumption}\n\n\\begin{assumption}\\label{assump-wd-ialm}\nFor every $k$, there is $\\vx^{k+1}$ satisfying \\eqref{eq:ialm-x}.\n\\end{assumption}\n\nLet the smooth part of $\\cL_\\beta$ be denoted as\n$$F_\\beta(\\vx,\\vy,\\vz) = \\cL_\\beta(\\vx,\\vy,\\vz) - h(\\vx).$$\nBased on \\eqref{eq:lip}, we are able to show Lipschitz continuity of $\\nabla_\\vx F_\\beta(\\vx,\\vy,\\vz)$ with respect to $\\vx$ for every $(\\vy,\\vz)$.\n\n\nGiven $\\vareps>0$, we set $\\{\\beta_k\\}$ and $\\{\\rho_k\\}$ according to one of the follows:\n\n\\begin{setting}[constant penalty]\\label{set-penalty} Let $K$ be a positive integer number and $C_\\beta$ a positive real number. Set $$\\rho_k=\\beta_k=\\frac{C_\\beta}{K\\vareps},\\,\\forall \\, 0\\le k < K.$$ \n%where \n%\\begin{equation}\\label{eq:set-penalty}\n%C_*=\\max\\left(4\\|\\vy^*\\|^2+4\\|\\vz^*\\|^2, (1+\\|\\vy^*\\|)^2+\\|\\vone+\\vz^*\\|^2\\right).\n%\\end{equation}\n\\end{setting}\n\n\\begin{setting}[geometrically increasing penalty]\\label{set-ialm} Let $K$ be a positive integer number, $C_\\beta$ a positive real number, and $\\sigma>1$. Set \n\\begin{equation}\\label{eq:def-beta-g}\n\\beta_g = \\frac{C_\\beta}{\\vareps}\\frac{\\sigma-1}{\\sigma^K-1},\n\\end{equation}\nand\n$$ \\rho_k=\\beta_k=\\beta_g\\sigma^k, \\,\\forall\\, 0\\le k < K.$$ \n%\\begin{equation}\\label{eq:set-ialm}\n%\\beta_g = \\frac{C^*}{\\vareps}\\frac{\\sigma-1}{\\sigma^K-1}\n%\\end{equation}\n%where $C_*$ is given in \\eqref{eq:set-penalty}.\n\\end{setting}\n\n\n\\textbf{``Optimal'' subproblem accuracy parameters.} If $t_k$ gradient evaluations are required to produce $\\vx^{k+1}$, then the total number of gradient evaluations is \n$T_K=\\sum_{k=0}^{K-1}t_k$ to generate $\\{\\vx^k\\}_{k=1}^K$. Given $\\vareps>0$, and $\\{\\beta_k\\}$,$\\{\\rho_k\\}$ set according to either Setting \\ref{set-penalty} or Setting \\ref{set-ialm}, we can choose $\\{\\vareps_k\\}$ to minimize $T_K$ subject to the condition in \\eqref{eq:req-eps}. \n\n\n\\begin{equation}\\label{eq:const-eps}\n\\vareps_k=\\frac{\\vareps }{2}\\frac{C_\\vareps}{C_\\beta},\\,\\forall \\, 0\\le k<K,\n\\end{equation}","units":[{"start":11036,"end":11623,"start_line":309,"end_line":313,"raw_tex":"In this paper, we consider the constrained convex programming\n\\begin{equation}\\label{eq:ccp}\n\\Min_{\\vx\\in\\cX} f_0(\\vx), \\st \\vA\\vx=\\vb, f_i(\\vx)\\le 0, i=1,\\ldots, m,\n\\end{equation}\nwhere $\\cX\\subseteq\\RR^n$ is a closed convex set, $\\vA$ and $\\vb$ are respectively given matrix and vector, and $f_i$ is a convex function for every $i=0,1,\\ldots,m$. Any convex optimization problem can be written in the standard form of \\eqref{eq:ccp}. It appears in many areas including statistics, machine learning, data mining, engineering, signal processing, finance, operations research, and so on. \n","sha256_utf8":"ceb6a6b683ed74627406725abc94cbfa024064be7ec1cf097791960236bcc30e"},{"start":13895,"end":15111,"start_line":322,"end_line":333,"raw_tex":"We will assume composite convex structure on \\eqref{eq:ccp}. More specifically, we assume \\begin{equation}\\label{eq:f0}\nf_0(\\vx)=g(\\vx)+h(\\vx),\n\\end{equation} \nwhere $g$ is a Lipschitz differentiable convex function, and $h$ is a simple\\footnote{By ``simple'', we mean the proximal mapping of $h$ is easy to evaluate, i.e., it is easy to find a solution to $\\min_{\\vx\\in\\cX} h(\\vx) + \\frac{1}{2\\gamma}\\|\\vx-\\hat{\\vx}\\|^2$ for any $\\hat{\\vx}$ and $\\gamma>0$.} (possibly nondifferentiable) convex function. Also, $f_i$ is convex and Lipschitz differentiable for every $i\\in [m]$, namely, there are constants $L_0,L_1,\\ldots, L_m$ such that\n\\begin{subequations}\\label{eq:lip-gf}\n\\begin{align}\n\\|\\nabla g(\\hat{\\vx})-\\nabla g(\\tilde{\\vx})\\|\\le L_0\\|\\hat{\\vx}-\\tilde{\\vx}\\|,\\,&\\forall\\, \\hat{\\vx}, \\tilde{\\vx} \\in \\dom(h)\\cap \\cX,\\label{eq:lip-g}\\\\\n\\|\\nabla f_i(\\hat{\\vx})-\\nabla f_i(\\tilde{\\vx})\\|\\le L_i\\|\\hat{\\vx}-\\tilde{\\vx}\\|,\\,& \\forall\\, \\hat{\\vx}, \\tilde{\\vx} \\in \\dom(h) \\cap \\cX, \\forall i\\in [m].\\label{eq:lip-f}\n\\end{align} \n\\end{subequations}\nIn addition, we assume the boundedness of $\\dom(h)\\cap\\cX$ and denote its diameter as \n$$D=\\Max_{\\hat{\\vx},\\tilde{\\vx}\\in \\dom(h)\\cap\\cX}\\|\\hat{\\vx}-\\tilde{\\vx}\\|.$$\n","sha256_utf8":"8caa815bb9ee95c2124a61bc29114ad9be92af5281e8f77ac7b0a79f99434908"},{"start":16794,"end":17305,"start_line":353,"end_line":362,"raw_tex":"\\begin{equation}\\label{eq:def-psi}\n\\psi_\\beta(u,v) = \\left\\{\\begin{array}{ll}uv+\\frac{\\beta}{2}u^2,\\,&\\text{ if }\\beta u +v \\ge0,\\\\[0.2cm]\n-\\frac{v^2}{2\\beta},\\,&\\text{ if }\\beta u +v < 0.\\end{array}\\right.\n\\end{equation}\nLet\n$$\\Psi_\\beta(\\vx,\\vz)=\\sum_{i=1}^m\\psi_\\beta(f_i(\\vx),z_i),$$\nand we obtain the classical AL function of \\eqref{eq:ccp}:\n\\begin{equation}\\label{eq:aug-fun}\n\\cL_\\beta(\\vx,\\vy,\\vz) = f_0(\\vx) + \\vy^\\top(\\vA\\vx-\\vb) + \\frac{\\beta}{2}\\|\\vA\\vx-\\vb\\|^2 + \\Psi_\\beta(\\vx,\\vz).\n\\end{equation}\n","sha256_utf8":"32195e9cca3dc8c920c527179c7d09b957555af8bb626a110f4b5c8c2b3add3c"},{"start":17973,"end":18649,"start_line":366,"end_line":382,"raw_tex":"\\begin{algorithm}[h]\n\\caption{Inexact augmented Lagrangian method for \\eqref{eq:ccp}}\\label{alg:ialm}\n\\DontPrintSemicolon\n\\textbf{Initialization:} choose $\\vx^0, \\vy^0, \\vz^0$ and $\\{\\beta_k,\\rho_k\\}$\\;\n\\For{$k=0,1,\\ldots$}{\nFind $\\vx^{k+1}\\in\\cX$ such that \n\\begin{equation}\\label{eq:ialm-x}\n\\cL_{\\beta_k}(\\vx^{k+1}, \\vy^k , \\vz^k  ) \\le \\min_{\\vx\\in\\cX} \\cL_{\\beta_k}(\\vx, \\vy^k , \\vz^k  ) + \\vareps_k.\n\\end{equation}\n\nUpdate $\\vy$ and $\\vz$ by\n\\begin{align}\n\\vy^{k+1} =&~\\vy^k  + \\rho_k (\\vA\\vx^{k+1}-\\vb),\\label{eq:alm-y}\\\\\nz^{k+1} _i=&~z_i^k+\\rho_k \\cdot\\max\\left(-\\frac{z_i^k}{\\beta_k}, f_i(\\vx^{k+1})\\right), i=1,\\ldots, m.\\label{eq:alm-z}\n\\end{align}\n}\n\\end{algorithm}","sha256_utf8":"389121368824e9bdf1c68af9c7d17bc2e0d8ffa4bab53f169a43cde175065403"},{"start":27237,"end":29258,"start_line":424,"end_line":433,"raw_tex":"We use italic letters $a, c, B, L,\\ldots,$ for scalars, bold lower-case letters $\\vx,\\vy,\\vz,\\ldots$ for vectors, and bold upper-case letters $\\vA,\\vB,\\ldots$ for matrices. $z_i$ denotes the $i$-th entry of a vector $\\vz$. We use $\\vzero$ to denote a vector of all zeros, and its size is clear from the context. $[m]$ denotes the set $\\{1,2,\\ldots,m\\}$ for any positive integer $m$. Given a real number $a$, we let $[a]_+=\\max(0,a)$ and $\\lceil a\\rceil$ be the smallest integer that is no less than $a$. For a vector $\\va$, $[\\va]_+$ takes the positive part of $\\va$ in a component-wise manner. $\\|\\va\\|$ denotes the Euclidean norm of a vector $\\va$ and $\\|\\vA\\|$ the spectral norm of a matrix $\\vA$. %By $p(k)\\lesssim q(k)$, we mean $\\limsup_k\\frac{p(k)}{q(k)} \\le 1$, i.e., $p(k)$ is dominated by $q(k)$. Given a symmetric positive definite (SPD) matrix $\\vP$, we denote $\\|\\cdot\\|_\\vP$ as $\\vP$-weighted norm, i.e., $\\|\\vw\\|_\\vP=\\sqrt{\\vw^\\top \\vP \\vw}$ . If $\\vP=\\vI$, we simply write it as $\\|\\cdot\\|$. \n\nWe denote $\\vell$ as the vector consisting of $L_i, i\\in [m]$, where $L_i$ is the Lipschitz constant of $\\nabla f_i$ in \\eqref{eq:lip-f}. Also we let $\\vf$ be the vector function with $f_i$ as the $i$-th component scalar function. That is \n\\begin{equation}\\label{eq:vec-ell}\n\\vell=[L_1,\\ldots,L_m], \\quad \\vf(\\vx)=[f_1(\\vx),\\ldots, f_m(\\vx)].\n\\end{equation}\nGiven a convex function $f$, $\\tilde{\\nabla}f(\\vx)$ represents one subgradient of $f$ at $\\vx$, namely,\n$$f(\\hat{\\vx})\\ge f(\\vx) + \\langle \\tilde{\\nabla} f(\\vx), \\hat{\\vx}-\\vx\\rangle,\\,\\forall\\, \\hat{\\vx},$$ and $\\partial f(\\vx)$ denotes its subdifferential, i.e., the set of all subgradients. When $f$ is differentiable, we simply write its subgradient as $\\nabla f(\\vx)$. For a convex set $\\cX$, we use $\\iota_\\cX$ as its indicator function, i.e., \n$$\\iota_\\cX(\\vx)=\\left\\{\\begin{array}{ll}0,&\\text{ if }\\vx\\in\\cX,\\\\+\\infty,&\\text{ if }\\vx\\not\\in\\cX,\\end{array}\\right.$$\nand $\\cN_\\cX(\\vx)=\\partial \\iota_\\cX(\\vx)$ as its normal cone at $\\vx\\in \\cX$. \n","sha256_utf8":"e570ae0c5f7c09e12933defd4e4b19ad84b25179d921a99d9bfe93d70a6a56ee"},{"start":29353,"end":29690,"start_line":436,"end_line":439,"raw_tex":"\\begin{definition}[$\\vareps$-optimal solution]\\label{def:eps-opt}\nLet $f_0^*$ be the optimal value of \\eqref{eq:ccp}. Given $\\vareps\\ge 0$, a point $\\vx\\in\\cX$ is called an $\\vareps$-optimal solution to \\eqref{eq:ccp} if \n$$|f_0(\\vx)-f_0^*| \\le \\vareps,\\text{ and }\\|\\vA\\vx-\\vb\\|+\\big\\| [\\vf(\\vx)]_+\\big\\| \\le \\vareps.$$\n\\end{definition}","sha256_utf8":"89f402879dc918cd64acab133dd2e0d441a4a2db394cd5eb34146f9b6fef7e1e"},{"start":31403,"end":32065,"start_line":452,"end_line":463,"raw_tex":"A point $(\\vx,\\vy,\\vz)$ satisfies the Karush-Kuhn-Tucker (KKT) conditions for \\eqref{eq:ccp} if\n\\begin{subequations}\\label{eq:kkt}\n\\begin{align}\n0\\in \\partial f_0(\\vx) +\\cN_\\cX(\\vx) + \\vA^\\top \\vy + \\sum_{i=1}^m z_i \\nabla f_i(\\vx), \\label{eq:kkt-1st}&\\\\\n\\vA\\vx = \\vb, \\quad \\vx\\in \\cX, \\label{eq:kkt-res}&\\\\\nz_i\\ge0, \\quad f_i(\\vx)\\le 0,\\quad z_if_i(\\vx)=0, \\forall i\\in [m]. \\label{eq:kkt-cp}\n\\end{align}\n\\end{subequations}\nFrom the convexity of $f_i$'s, if $(\\vx^*,\\vy^*,\\vz^*)$ is a KKT point, then\n\\begin{equation}\\label{eq:opt}\nf_0(\\vx)-f_0(\\vx^*)+\\langle \\vy^*, \\vA\\vx-\\vb\\rangle + \\sum_{i=1}^m z^*_i f_i(\\vx) \\ge 0,\\, \\forall\\, \\vx\\in\\cX.\n\\end{equation}\n","sha256_utf8":"f840a4a938e3339e8ec978ccff9167288bc58ad115c3af2782d7e09c7d30a939"},{"start":35131,"end":36097,"start_line":509,"end_line":513,"raw_tex":"In this subsection, we review Nesterov's optimal first-order method for composite convex programs. The method will be used to approximately solve $\\vx$-subproblems in Algorithm \\ref{alg:ialm}. It aims at finding a solution of the following problem\n\\begin{equation}\\label{eq:comp-prob}\n\\Min_\\vx \\phi(\\vx) + \\psi(\\vx),\n\\end{equation}\nwhere $\\phi$ is a Lipschitz differentiable and strongly convex function with gradient Lipschitz constant $L_\\phi$ and strong convexity modulus $\\mu\\ge0$, and $\\psi$ is a simple (possibly nondifferentiable) closed convex function. Algorithm \\ref{alg:apg} summarizes the method. Here, for simplicity, we assume $L_\\phi$ and $\\mu$ are known. The method does not require the value of $L_\\phi$ but can estimate it by backtracking. In addition, it only requires a lower estimate of $\\mu$; see \\cite{nesterov2013gradient} for example.  %For our purpose, we will have $\\mu=1$; see the problem \\eqref{eq:ipalm-x} in Algorithm \\ref{alg:ipalm}.\n","sha256_utf8":"bdbc6587a3d9b8e524817b593528a7b560fc3af7235edd427c9d83d3d48baa43"},{"start":36098,"end":36669,"start_line":515,"end_line":524,"raw_tex":"\\begin{algorithm}\\caption{Nesterov's optimal first-order method for \\eqref{eq:comp-prob}}\\label{alg:apg}\n\\textbf{Initialization:} choose $\\hat{\\vx}^0=\\vx^0$, $\\alpha_0\\in (0,1]$, and let $q=\\frac{\\mu}{L_\\phi}$\\; \n\\For{$k=0,1,\\ldots, $}{\nLet \n$$\\vx^{k+1}=\\argmin_{\\vx} \\langle \\nabla \\phi(\\hat{\\vx}^k), \\vx\\rangle + \\frac{L_\\phi}{2}\\|\\vx-\\hat{\\vx}^k\\|^2+\\psi(\\vx).$$\n\nSet $$\\alpha_{k+1}=\\frac{q-\\alpha_k^2+\\sqrt{(q-\\alpha_k^2)^2+4\\alpha_k^2}}{2},$$ and\n$$\\hat{\\vx}^{k+1}=\\vx^{k+1}+\\frac{\\alpha_k(1-\\alpha_k)}{\\alpha_k^2+\\alpha_{k+1}}(\\vx^{k+1}-\\vx^k ).$$\n}\n\\end{algorithm}","sha256_utf8":"e44d81b9313b85fcfa9fff2b281ca1dc5bd5c919197012c9525c27f439d45a50"},{"start":38377,"end":38522,"start_line":547,"end_line":549,"raw_tex":"\\begin{assumption}\\label{assump-kkt}\nThere exists a point $(\\vx^*,\\vy^*,\\vz^*)$ satisfying the KKT conditions in \\eqref{eq:kkt}.\n\\end{assumption}","sha256_utf8":"218387edebc5c4fd6ae3e7bca294e5e8da500f5441001cc94f6a71b4834f9a30"},{"start":38659,"end":38782,"start_line":555,"end_line":557,"raw_tex":"\\begin{assumption}\\label{assump-wd-ialm}\nFor every $k$, there is $\\vx^{k+1}$ satisfying \\eqref{eq:ialm-x}.\n\\end{assumption}","sha256_utf8":"975b0fad2a92a806a17898aeea355b5c0de8592325dec6bb308579778866e6f1"},{"start":64055,"end":64312,"start_line":866,"end_line":868,"raw_tex":"Let the smooth part of $\\cL_\\beta$ be denoted as\n$$F_\\beta(\\vx,\\vy,\\vz) = \\cL_\\beta(\\vx,\\vy,\\vz) - h(\\vx).$$\nBased on \\eqref{eq:lip}, we are able to show Lipschitz continuity of $\\nabla_\\vx F_\\beta(\\vx,\\vy,\\vz)$ with respect to $\\vx$ for every $(\\vy,\\vz)$.\n","sha256_utf8":"35b030a4b143e3c4c7c0826ca00b5021407b6c1000d9836baff59b0625ca97ba"},{"start":68944,"end":69914,"start_line":924,"end_line":943,"raw_tex":"Given $\\vareps>0$, we set $\\{\\beta_k\\}$ and $\\{\\rho_k\\}$ according to one of the follows:\n\n\\begin{setting}[constant penalty]\\label{set-penalty} Let $K$ be a positive integer number and $C_\\beta$ a positive real number. Set $$\\rho_k=\\beta_k=\\frac{C_\\beta}{K\\vareps},\\,\\forall \\, 0\\le k < K.$$ \n%where \n%\\begin{equation}\\label{eq:set-penalty}\n%C_*=\\max\\left(4\\|\\vy^*\\|^2+4\\|\\vz^*\\|^2, (1+\\|\\vy^*\\|)^2+\\|\\vone+\\vz^*\\|^2\\right).\n%\\end{equation}\n\\end{setting}\n\n\\begin{setting}[geometrically increasing penalty]\\label{set-ialm} Let $K$ be a positive integer number, $C_\\beta$ a positive real number, and $\\sigma>1$. Set \n\\begin{equation}\\label{eq:def-beta-g}\n\\beta_g = \\frac{C_\\beta}{\\vareps}\\frac{\\sigma-1}{\\sigma^K-1},\n\\end{equation}\nand\n$$ \\rho_k=\\beta_k=\\beta_g\\sigma^k, \\,\\forall\\, 0\\le k < K.$$ \n%\\begin{equation}\\label{eq:set-ialm}\n%\\beta_g = \\frac{C^*}{\\vareps}\\frac{\\sigma-1}{\\sigma^K-1}\n%\\end{equation}\n%where $C_*$ is given in \\eqref{eq:set-penalty}.\n\\end{setting}\n","sha256_utf8":"44403f888dd88270ffa53f1c60ec852b8001083a68ec211b01e54ed4e9158b1e"},{"start":73492,"end":73945,"start_line":994,"end_line":995,"raw_tex":"\\textbf{``Optimal'' subproblem accuracy parameters.} If $t_k$ gradient evaluations are required to produce $\\vx^{k+1}$, then the total number of gradient evaluations is \n$T_K=\\sum_{k=0}^{K-1}t_k$ to generate $\\{\\vx^k\\}_{k=1}^K$. Given $\\vareps>0$, and $\\{\\beta_k\\}$,$\\{\\rho_k\\}$ set according to either Setting \\ref{set-penalty} or Setting \\ref{set-ialm}, we can choose $\\{\\vareps_k\\}$ to minimize $T_K$ subject to the condition in \\eqref{eq:req-eps}. \n","sha256_utf8":"8904b14b31a3b6d857ba94527fed6a04a922b327ea1792a4e256f47dca567577"},{"start":75677,"end":75805,"start_line":1016,"end_line":1018,"raw_tex":"\\begin{equation}\\label{eq:const-eps}\n\\vareps_k=\\frac{\\vareps }{2}\\frac{C_\\vareps}{C_\\beta},\\,\\forall \\, 0\\le k<K,\n\\end{equation}","sha256_utf8":"cba87c0688bf4ee8f76a206b08355548a7d3e8161479c872de9c196ee27bcb7d"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}[Iteration complexity with constant penalty and constant error]\\label{thm:it-comp-const}\nFor any given $\\vareps>0$, let $K$ be a positive integer number and $C_\\beta, C_\\vareps$ two positive real numbers. Set $\\{\\beta_k\\}$ and $\\{\\rho_k\\}$ according to Setting \\ref{set-penalty} and $\\{\\vareps_k\\}$ by \\eqref{eq:const-eps}. Let $\\bar{\\vx}^K$ be given in \\eqref{eq:barx-K}. Then \n\\begin{subequations}\\label{eq:error-barx-K}\n\\begin{align}\n\\big|f_0(\\bar{\\vx}^{K})-f_0(\\vx^*)\\big| \\le \\frac{\\vareps\\big(2\\|\\vy^*\\|^2+ 2\\|\\vz^*\\|^2\\big)}{C_\\beta}+\\frac{\\vareps}{2}\\frac{C_\\vareps}{C_\\beta},\\label{eq:error-barx-K-obj}\\\\\n\\|\\vA\\bar{\\vx}^{K}-\\vb\\| + \\big\\| [\\vf(\\bar{\\vx}^{K})]_+ \\big\\|\\le  \\frac{\\vareps\\big[(1+\\|\\vy^*\\|)^2+ (1+\\|\\vz^*\\|)^2\\big]}{2C_\\beta}+\\frac{\\vareps}{2}\\frac{C_\\vareps}{C_\\beta}.\\label{eq:error-barx-K-res}\n\\end{align}\n\\end{subequations}\nAssume $\\mu\\le \\frac{L_0}{4}$. Then Algorithm \\ref{alg:ialm} can produce $\\bar{\\vx}^K$ by evaluating gradients of $g, f_i, \\,i\\in [m]$ in at most $T_K$ times, where\n\\begin{equation}\\label{eq:T-penaly-cvx}\nT_K\\le \\left\\lceil2DK\\sqrt{\\frac{C_\\beta}{C_\\vareps}}\\left(\\sqrt{\\frac{L_*}{\\vareps}}+\\frac{1}{\\vareps}\\sqrt{\\frac{C_\\beta H}{K}}\\right)+K\\right\\rceil, \\text{ if }\\mu=0,\n\\end{equation}\nand\n\\begin{equation}\\label{eq:T-penaly-strcvx}\nT_K\\le \\left\\lceil2K\\left(\\sqrt{\\frac{L_*}{\\mu}}+\\sqrt{\\frac{C_\\beta H}{\\mu K\\vareps}}\\right)\\log\\left(\\frac{D^2 C_\\beta}{C_\\vareps}\\big(\\frac{L_*+\\mu}{\\vareps}+\\frac{C_\\beta H}{K\\vareps^2}\\big)\\right) + K\\right\\rceil, \\text{ if }\\mu>0.\n\\end{equation}\n\n\\end{theorem}","units":[{"start":79718,"end":81287,"start_line":1075,"end_line":1092,"raw_tex":"\\begin{theorem}[Iteration complexity with constant penalty and constant error]\\label{thm:it-comp-const}\nFor any given $\\vareps>0$, let $K$ be a positive integer number and $C_\\beta, C_\\vareps$ two positive real numbers. Set $\\{\\beta_k\\}$ and $\\{\\rho_k\\}$ according to Setting \\ref{set-penalty} and $\\{\\vareps_k\\}$ by \\eqref{eq:const-eps}. Let $\\bar{\\vx}^K$ be given in \\eqref{eq:barx-K}. Then \n\\begin{subequations}\\label{eq:error-barx-K}\n\\begin{align}\n\\big|f_0(\\bar{\\vx}^{K})-f_0(\\vx^*)\\big| \\le \\frac{\\vareps\\big(2\\|\\vy^*\\|^2+ 2\\|\\vz^*\\|^2\\big)}{C_\\beta}+\\frac{\\vareps}{2}\\frac{C_\\vareps}{C_\\beta},\\label{eq:error-barx-K-obj}\\\\\n\\|\\vA\\bar{\\vx}^{K}-\\vb\\| + \\big\\| [\\vf(\\bar{\\vx}^{K})]_+ \\big\\|\\le  \\frac{\\vareps\\big[(1+\\|\\vy^*\\|)^2+ (1+\\|\\vz^*\\|)^2\\big]}{2C_\\beta}+\\frac{\\vareps}{2}\\frac{C_\\vareps}{C_\\beta}.\\label{eq:error-barx-K-res}\n\\end{align}\n\\end{subequations}\nAssume $\\mu\\le \\frac{L_0}{4}$. Then Algorithm \\ref{alg:ialm} can produce $\\bar{\\vx}^K$ by evaluating gradients of $g, f_i, \\,i\\in [m]$ in at most $T_K$ times, where\n\\begin{equation}\\label{eq:T-penaly-cvx}\nT_K\\le \\left\\lceil2DK\\sqrt{\\frac{C_\\beta}{C_\\vareps}}\\left(\\sqrt{\\frac{L_*}{\\vareps}}+\\frac{1}{\\vareps}\\sqrt{\\frac{C_\\beta H}{K}}\\right)+K\\right\\rceil, \\text{ if }\\mu=0,\n\\end{equation}\nand\n\\begin{equation}\\label{eq:T-penaly-strcvx}\nT_K\\le \\left\\lceil2K\\left(\\sqrt{\\frac{L_*}{\\mu}}+\\sqrt{\\frac{C_\\beta H}{\\mu K\\vareps}}\\right)\\log\\left(\\frac{D^2 C_\\beta}{C_\\vareps}\\big(\\frac{L_*+\\mu}{\\vareps}+\\frac{C_\\beta H}{K\\vareps^2}\\big)\\right) + K\\right\\rceil, \\text{ if }\\mu>0.\n\\end{equation}\n\n\\end{theorem}","sha256_utf8":"3ac9e348fd89daba6c9c269b2061b0a146d784ecd9fd0893215e1a16f8b5b2a7"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\\label{lem:pre-rate}\nAssume $(\\vx^*,\\vy^*,\\vz^*)$ satisfies the KKT conditions in \\eqref{eq:kkt}. Let $\\bar{\\vx}$ be a point such that for any $\\vy$ and any $\\vz\\ge\\vzero$,\n\\begin{equation}\\label{eq:pre-rate-cond}\nf_0(\\bar{\\vx})-f_0(\\vx^*)+ \\vy^\\top (\\vA\\bar{\\vx}-\\vb) +\\sum_{i=1}^m z_i f_i(\\bar{\\vx}) \\le \\alpha + c_1\\|\\vy\\|^2+c_2\\|\\vz\\|^2,\n\\end{equation}\nwhere $\\alpha$ and $c_1, c_2$ are nonnegative constants independent of $\\vy$ and $\\vz$. Then\n\\begin{align}\n-\\left(\\alpha + 4c_1\\|\\vy^*\\|^2+ 4c_2\\|\\vz^*\\|^2\\right)\\le f_0(\\bar{\\vx})-f_0(\\vx^*)\\le \\alpha,\\label{eq:pre-rate-obj}\\\\\n\\|\\vA\\bar{\\vx}-\\vb\\| + \\big\\|[\\vf(\\bar{\\vx})]_+\\big\\| \\le \\alpha + c_1\\big(1+\\|\\vy^*\\|\\big)^2 + c_2\\big(1+\\|\\vz^*\\|\\big)^2. \\label{eq:pre-rate-res}\n\\end{align}\n\\end{lemma}\n\n\\begin{proof}\nLetting $\\vy=\\vzero$ and $\\vz=\\vzero$ in \\eqref{eq:pre-rate-cond} gives the second inequality in \\eqref{eq:pre-rate-obj}.\nFor any nonnegative $\\gamma_y$ and $\\gamma_z$, we let \n$$\\vy=\\gamma_y\\frac{\\vA\\bar{\\vx}-\\vb}{\\|\\vA\\bar{\\vx}-\\vb\\|},\\quad \\vz = \\gamma_z \\frac{[\\vf(\\bar{\\vx})]_+}{\\big\\|[\\vf(\\bar{\\vx})]_+\\big\\|}$$ %z_i = \\gamma_i \\frac{[f_i(\\bar{\\vx})]_+}{|f_i(\\bar{\\vx})|}, \\forall i\\in [m]$$\nand have from \\eqref{eq:pre-rate-cond} by using the convention $\\frac{0}{0}=0$ that\n\\begin{align}\\label{eq:pre-rate-ineq2}\nf_0(\\bar{\\vx})-f_0(\\vx^*) +\\gamma_y \\|\\vA\\bar{\\vx}-\\vb\\| + \\gamma_z\\big\\|[\\vf(\\bar{\\vx})]_+\\big\\|\n\\le  \\alpha + c_1\\gamma_y^2+ c_2\\gamma_z^2.\n%f_0(\\bar{\\vx})-f_0(\\vx^*) +\\gamma_y \\|\\vA\\bar{\\vx}-\\vb\\| + \\sum_{i=1}^m \\gamma_i [f_i(\\bar{\\vx})]_+\n%\\le  \\alpha + c_1\\gamma_y^2+ c_2\\sum_{i=1}^m\\gamma_i^2.\n\\end{align}\n\nNoting \n\\begin{equation}\\label{eq:pre-rate-ineq3}\n-\\langle \\vy^*, \\vA\\bar{\\vx}-\\vb\\rangle \\ge -\\|\\vy^*\\|\\cdot\\|\\vA\\bar{\\vx}-\\vb\\|,\\quad - \\sum_{i=1}^mz_i^* f_i(\\bar{\\vx}) \\ge - \\|\\vz^*\\|\\cdot \\big\\|[\\vf(\\bar{\\vx})]_+\\big\\|,\n\\end{equation} we have from \\eqref{eq:opt} and \\eqref{eq:pre-rate-ineq2} that\n\\begin{align*}\n(\\gamma_y-\\|\\vy^*\\|) \\|\\vA\\bar{\\vx}-\\vb\\| + (\\gamma_z -\\|\\vz^*\\|)\\|[\\vf(\\bar{\\vx})]_+\\big\\|\n\\le  \\alpha + c_1\\gamma_y^2+ c_2\\gamma_z^2\n\\end{align*}\nIn the above inequality, letting $\\gamma_y=1+\\|\\vy^*\\|$ and $\\gamma_z = 1+\\|\\vz^*\\|$ gives \\eqref{eq:pre-rate-res}, and letting $\\gamma_y=2\\|\\vy^*\\|$ and $\\gamma_z = 2\\|\\vz^*\\|$ gives the first inequality in \\eqref{eq:pre-rate-obj} by \\eqref{eq:opt} and \\eqref{eq:pre-rate-ineq3}.\n\\end{proof}\n\n\\begin{theorem}\\label{thm:rate-apg}\nLet $\\{\\vx^k \\}$ be the sequence generated from Algorithm \\ref{alg:apg}. Assume $\\vx^*$ to be a minimizer of \\eqref{eq:comp-prob}. The following results holds:\n\\begin{enumerate}\n\\item If $\\mu=0$ and $\\alpha_0=1$, then\n\\begin{equation}\\label{eq:optimal-rate-cvx}\n\\phi(\\vx^k) + \\psi(\\vx^k) - \\phi(\\vx^*)-\\psi(\\vx^*) \\le \\frac{2L_\\phi\\|\\vx^0-\\vx^*\\|^2}{k^2},\\,\\forall k\\ge1.\n\\end{equation}\n\\item If $\\mu >0$ and $\\alpha_0=\\sqrt{\\frac{\\mu}{L_\\phi}}$, then\n\\begin{equation}\\label{eq:optimal-rate-scvx}\n\\phi(\\vx^k)-\\phi(\\vx^*)\\le \\frac{(L_\\phi+\\mu)\\|\\vx^0-\\vx^*\\|^2}{2}\\left(1-\\sqrt{\\frac{\\mu}{L_\\phi}}\\right)^k,\\,\\forall k\\ge1.\n\\end{equation}\n\\end{enumerate}\n\\end{theorem}\n\n\\begin{lemma}\\label{lem:multiplier}\nLet $\\vy$ and $\\vz$ be updated by \\eqref{eq:alm-y} and \\eqref{eq:alm-z} respectively. Then for any $k$, it holds\n\\begin{align}\n\\frac{1}{2\\rho_k}\\big[\\|\\vy^{k+1} -\\vy\\|^2-\\|\\vy^k -\\vy\\|^2+\\|\\vy^{k+1} -\\vy^k \\|^2\\big] - \\langle \\vy^{k+1} -\\vy, \\vr^{k+1}\\rangle = &~ 0,\\label{eq:alm-yterm}\\\\\n\\frac{1}{2\\rho_k}\\big[\\|\\vz^{k+1} -\\vz\\|^2-\\|\\vz^k  -\\vz\\|^2+\\|\\vz^{k+1} -\\vz^k  \\|^2\\big]- \\sum_{i=1}^m (z_i^{k+1}-z_i)\\cdot\\max\\big(-\\frac{z_i^k}{\\beta_k}, f_i(\\vx^{k+1})\\big)=& ~0,\\label{eq:alm-zterm}\n\\end{align}\nwhere $\\vr^k =\\vA\\vx^k -\\vb$.\n\\end{lemma}\n\n\\begin{proof}\nUsing the equality $2\\vu^\\top \\vv = \\|\\vu\\|^2 - \\|\\vu-\\vv\\|^2 + \\|\\vv\\|^2$, we have the results from the updates \\eqref{eq:alm-y} and \\eqref{eq:alm-z}.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:ineq-z}\nFor any $\\vz\\ge\\vzero$, we have\n\\begin{align}\\label{eq:alm-ineq-z}\n&\\sum_{i=1}^m\\big([z_i^k+\\beta_k f_i(\\vx^{k+1})]_+-z_i\\big) f_i(\\vx^{k+1})- \\sum_{i=1}^m (z_i^{k+1}-z_i)\\cdot\\max\\big(-\\frac{z_i^k}{\\beta_k}, f_i(\\vx^{k+1})\\big)\\nonumber\\\\\n\\ge&\\frac{1}{\\rho_k^2}(\\beta_k-\\rho_k)\\|\\vz^{k+1} -\\vz^k  \\|^2.\n\\end{align}\n\\end{lemma}\n\n\\begin{proof}\nDenote \n\\begin{equation}\\label{eq:setIk}\nI_+^k=\\{i\\in [m]: z_i^k + \\beta_k f_i(\\vx^{k+1}) \\ge 0\\},\\quad I_-^k=[m]\\backslash I_+^k.\n\\end{equation}\n Then \n\\begin{align*}\n&\\text{the left hand side of \\eqref{eq:alm-ineq-z}}\\\\\n=&\\sum_{i\\in I_+^k}\\left[(z_i^k-z_i)f_i(\\vx^{k+1})+\\beta_k[f_i(\\vx^{k+1})]^2-\\big(z_i^k+\\rho_k f_i(\\vx^{k+1})-z_i\\big)f_i(\\vx^{k+1})\\right]\\\\\n&+\\sum_{i\\in I_-^k}\\left[-z_i f_i(\\vx^{k+1})-\\big(z_i^k-\\frac{\\rho_k z_i^k}{\\beta_k}-z_i\\big)\\big(-\\frac{z_i^k}{\\beta_k}\\big)\\right]\\\\\n=&(\\beta_k-\\rho_k)\\sum_{i\\in I_+^k}[f_i(\\vx^{k+1})]^2+\\sum_{i\\in I_-^k}\\left[-z_i \\big(f_i(\\vx^{k+1})+\\frac{z_i^k}{\\beta_k}\\big)+\\frac{1}{\\beta_k^2}(\\beta_k-\\rho_k)(z_i^k)^2\\right]\\\\\n\\ge& (\\beta_k-\\rho_k)\\sum_{i\\in I_+^k}[f_i(\\vx^{k+1})]^2 +\\frac{1}{\\beta_k^2}(\\beta_k-\\rho_k)\\sum_{i\\in I_-^k}(z_i^k)^2\\\\\n=&\\frac{1}{\\rho_k^2}(\\beta_k-\\rho_k)\\|\\vz^{k+1} -\\vz^k  \\|^2,\n\\end{align*}\nwhere the inequality follows from $z_i\\ge0$ and $f_i(\\vx^{k+1})+\\frac{z_i^k}{\\beta_k}\\le 0,\\,\\forall i\\in I_-^k$, and the last equality holds due to the update \\eqref{eq:alm-z}.\n\\end{proof}\n\n\\begin{theorem}[One-iteration progress of iALM]\\label{thm:1iter-ialm}\nLet $\\{(\\vx^k ,\\vy^k ,\\vz^k  )\\}$ be the sequence generated from Algorithm \\ref{alg:ialm}. Then for any $\\vx\\in\\cX$ such that $\\vA\\vx=\\vb$ and $f_i(\\vx)\\le 0,\\,\\forall i\\in [m]$, any $\\vy$, and any $\\vz\\ge\\vzero$, it holds that\n\\begin{align}\\label{eq:ialm-ineq2}\n&f_0(\\vx^{k+1})-f_0(\\vx) +\\vy^\\top \\vr^{k+1} + \\sum_{i=1}^m z_i f_i(\\vx^{k+1})+\\frac{\\beta_k-\\rho_k}{2}\\|\\vr^{k+1}\\|^2\\cr\n&+\\frac{\\beta_k-\\rho_k}{2\\rho_k^2}\\|\\vz^{k+1} -\\vz^k  \\|^2+\\frac{1}{2\\rho_k}\\|\\vy^{k+1} -\\vy\\|^2 + \\frac{1}{2\\rho_k}\\|\\vz^{k+1} -\\vz\\|^2\\cr\n\\le & \\frac{1}{2\\rho_k}\\|\\vy^k -\\vy\\|^2+ \\frac{1}{2\\rho_k}\\|\\vz^k  -\\vz\\|^2+\\vareps_k.\n\\end{align}\n\\end{theorem}\n\n\\begin{proof}\nFrom \\eqref{eq:ialm-x}, it follows that for any $\\vx$ such that $\\vA\\vx=\\vb$,\n\\begin{equation}\\label{eq:ialm-1st}\nf_0(\\vx^{k+1})+\\langle \\vy^k , \\vr^{k+1}\\rangle +\\frac{\\beta_k}{2}\\|\\vr^{k+1}\\|^2+\\Psi_{\\beta_k}(\\vx^{k+1},\\vz^k  )-f_0(\\vx)-\\Psi_{\\beta_k}(\\vx,\\vz^k  )\\le\\vareps_k.\n\\end{equation}\nSince $\\langle \\vy^k , \\vr^{k+1}\\rangle=\\langle \\vy^{k+1} -\\vy, \\vr^{k+1}\\rangle + \\langle \\vy, \\vr^{k+1}\\rangle -\\rho_k\\|\\vr^{k+1}\\|^2$, by adding \\eqref{eq:alm-yterm} and \\eqref{eq:alm-zterm} to the above inequality, we have \n\\begin{align}\\label{eq:ialm-ineq1}\n&f_0(\\vx^{k+1})-f_0(\\vx) +\\vy^\\top \\vr^{k+1} + \\sum_{i=1}^m z_i f_i(\\vx^{k+1})+\\sum_{i=1}^m\\left([z_i^k+\\beta_k f_i(\\vx^{k+1})]_+-z_i\\right)f_i(\\vx^{k+1})\\cr\n&+\\big(\\frac{\\beta_k}{2}-\\rho_k\\big)\\|\\vr^{k+1}\\|^2+\\Psi_{\\beta_k}(\\vx^{k+1},\\vz^k  )-\\sum_{i=1}^m[z_i^k+\\beta_k f_i(\\vx^{k+1})]_+f_i(\\vx^{k+1})-\\Psi_{\\beta_k}(\\vx,\\vz^k  )\\cr\n&+\\frac{1}{2\\rho_k}\\big[\\|\\vy^{k+1} -\\vy\\|^2-\\|\\vy^k -\\vy\\|^2+\\|\\vy^{k+1} -\\vy^k \\|^2\\big] \\cr\n&+\\frac{1}{2\\rho_k}\\big[\\|\\vz^{k+1} -\\vz\\|^2-\\|\\vz^k  -\\vz\\|^2+\\|\\vz^{k+1} -\\vz^k  \\|^2\\big]- \\sum_{i=1}^m (z_i^{k+1}-z_i)\\cdot\\max\\big(-\\frac{z_i^k}{\\beta_k}, f_i(\\vx^{k+1})\\big)\\cr\n\\le & \\vareps_k.\n\\end{align}\n\nNote that\n\\begin{align}\\label{eq:ialm-ineq3}\n&\\Psi_{\\beta_k}(\\vx^{k+1},\\vz^k  )-\\sum_{i=1}^m[z_i^k+\\beta_k f_i(\\vx^{k+1})]_+f_i(\\vx^{k+1})\\cr\n=&\\sum_{i\\in I_+^k}\\left[z_i^k f_i(\\vx^{k+1}) +\\frac{\\beta_k}{2}[f_i(\\vx^{k+1})]^2-[z_i^k +\\beta_k f_i(\\vx^{k+1})]f_i(\\vx^{k+1})\\right]+\\sum_{i\\in I_-^k}\\left[-\\frac{(z_i^k)^2}{2\\beta_k}\\right]\\cr\n=&-\\sum_{i\\in I_+^k}\\frac{\\beta_k}{2}[f_i(\\vx^{k+1})]^2-\\sum_{i\\in I_-^k}\\frac{(z_i^k)^2}{2\\beta_k}\\cr\n=&-\\frac{\\beta_k}{2\\rho_z^2}\\|\\vz^{k+1} -\\vz^k  \\|^2,\n\\end{align}\nwhere the sets $I_+^k$ and $I_-^k$ are defined in \\eqref{eq:setIk}.\nIn addition, if $f_i(\\vx)\\le 0,\\,\\forall i\\in [m]$, then $\\Psi_{\\beta_k}(\\vx,\\vz^k  )\\le 0.$ Hence, plugging \\eqref{eq:alm-ineq-z} and \\eqref{eq:ialm-ineq3} into \\eqref{eq:ialm-ineq1} yields \\eqref{eq:ialm-ineq2}.\n\\end{proof}\n\n\\begin{theorem}[Ergodic convergence rate of iALM]\\label{thm:ialm-rate}\nUnder Assumptions \\ref{assump-kkt} and \\ref{assump-wd-ialm}, let $\\{(\\vx^k ,\\vy^k ,\\vz^k  )\\}$ be the sequence generated from Algorithm \\ref{alg:ialm} with $\\vy^0=\\vzero, \\vz^0 = \\vzero$ and $0<\\rho_k\\le\\beta_k,\\,\\forall k$. Then\n\\begin{subequations}\\label{eq:ialm-rate}\n\\begin{align}\n\\big|f_0(\\bar{\\vx}^{K})-f_0(\\vx^*)\\big| \\le \\frac{1}{\\sum_{t=0}^{K-1}\\rho_t}\\left(2\\|\\vy^*\\|^2+ 2\\|\\vz^*\\|^2+\\sum_{k=0}^{K-1}\\rho_k\\vareps_k\\right),\\label{eq:ialm-rate-obj}\\\\\n\\|\\vA\\bar{\\vx}^{K}-\\vb\\| + \\big\\| [\\vf(\\bar{\\vx}^{K})]_+\\big\\| \\le  \\frac{1}{\\sum_{t=0}^{K-1}\\rho_t}\\left(\\frac{(1+\\|\\vy^*\\|)^2}{2}+ \\frac{(\\|1+\\|\\vz^*\\|)^2}{2}+\\sum_{k=0}^{K-1}\\rho_k\\vareps_k\\right),\\label{eq:ialm-rate-res}\n\\end{align}\n\\end{subequations}\nwhere \n\\begin{equation}\\label{eq:barx-K}\n\\bar{\\vx}^{K}=\\frac{\\sum_{t=0}^{K-1}\\rho_t\\vx^{t+1} }{\\sum_{t=0}^{K-1}\\rho_t}.\n\\end{equation}\n\\end{theorem}\n\n\\begin{proof}\nSince $\\rho_k\\le \\beta_k$, the two terms $\\frac{\\beta_k-\\rho_k}{2}\\|\\vr^{k+1}\\|^2$ and \n$\\frac{\\beta_k-\\rho_k}{2\\rho_k^2}\\|\\vz^{k+1} -\\vz^k  \\|^2$ are nonnegative. Dropping them and multiplying $\\rho_k$ to both sides of \\eqref{eq:ialm-ineq2} yields \n\\begin{align}\\label{eq:ialm-ineq2-1}\n&\\rho_k\\left[f_0(\\vx^{k+1})-f_0(\\vx) +\\vy^\\top  \\vr^{k+1} + \\sum_{i=1}^m z_i f_i(\\vx^{k+1})\\right]+\\frac{1}{2}\\|\\vy^{k+1} -\\vy\\|^2 + \\frac{1}{2}\\|\\vz^{k+1} -\\vz\\|^2\\cr\n\\le & \\frac{1}{2}\\|\\vy^k -\\vy\\|^2+ \\frac{1}{2}\\|\\vz^k  -\\vz\\|^2+\\rho_k\\vareps_k.\n\\end{align}\n\nSumming up \\eqref{eq:ialm-ineq2} with $\\vx=\\vx^*$ gives\n\\begin{align}\\label{eq:ialm-ineq2-2}\n&\\sum_{k=0}^{K-1}\\rho_k\\left[f_0(\\vx^{k+1})-f_0(\\vx^*) +\\vy^\\top \\vr^{k+1} + \\sum_{i=1}^m z_i f_i(\\vx^{k+1})\\right]+\\frac{1}{2}\\|\\vy^{K}-\\vy\\|^2 + \\frac{1}{2}\\|\\vz^{K}-\\vz\\|^2\\cr\n\\le & \\frac{1}{2}\\|\\vy^0-\\vy\\|^2+ \\frac{1}{2}\\|\\vz^0-\\vz\\|^2+\\sum_{k=0}^{K-1}\\rho_k\\vareps_k.\n\\end{align}\nBy the convexity of $f_i$'s and the nonnegativity of $\\vz$, we have\n\\begin{align*}\n&~f_0(\\bar{\\vx}^{K})-f_0(\\vx^*) +\\vy^\\top  (\\vA\\bar{\\vx}^{K}-\\vb) + \\sum_{i=1}^m z_i f_i(\\bar{\\vx}^{K})\\\\\n\\le &~\\frac{1}{\\sum_{t=0}^{K-1}\\rho_t} \\sum_{k=0}^{K-1}\\rho_k\\left[f_0(\\vx^{k+1})-f_0(\\vx^*) +\\vy^\\top  \\vr^{k+1} + \\sum_{i=1}^m z_i f_i(\\vx^{k+1})\\right],\n\\end{align*}\nwhich together with \\eqref{eq:ialm-ineq2-2} implies\n$$f_0(\\bar{\\vx}^{K})-f_0(\\vx^*) +\\vy^\\top  (\\vA\\bar{\\vx}^{K}-\\vb) + \\sum_{i=1}^m z_i f_i(\\bar{\\vx}^{K})\n\\le  \\frac{1}{\\sum_{t=0}^{K-1}\\rho_t}\\left(\\frac{1}{2}\\|\\vy\\|^2+ \\frac{1}{2}\\|\\vz\\|^2+\\sum_{k=0}^{K-1}\\rho_k\\vareps_k\\right).$$\nThe results thus follow from Lemma \\ref{lem:pre-rate} with \n$$\\alpha=\\frac{\\sum_{k=0}^{K-1}\\rho_k\\vareps_k}{\\sum_{k=0}^{K-1}\\rho_k},\\ c_1=\\frac{1}{2\\sum_{k=0}^{K-1}\\rho_k},\\ c_2=\\frac{1}{2\\sum_{k=0}^{K-1}\\rho_k},$$\nand we complete the proof.\n\\end{proof}\n\nThe following results are easy to show from the Lipschitz differentiability of $g$ and $f_i,\\, i\\in [m]$.\n\\begin{proposition}\\label{prop-lip}\nAssume \\eqref{eq:lip-g}, \\eqref{eq:lip-f}, and the boundedness of $\\dom(h)\\cap \\cX$. Then there exist constants $B_1,\\ldots, B_m$ such that\n\\begin{subequations}\\label{eq:lip}\n\\begin{align}\n\\max\\big(|f_i(\\vx)|, \\|\\nabla f_i(\\vx)\\|\\big) \\le B_i, \\,& \\forall \\vx\\in \\dom(h)\\cap\\cX, \\forall i\\in [m],\\label{eq:bd-f}\\\\\n|f_i(\\hat{\\vx})-f_i(\\tilde{\\vx})| \\le B_i\\|\\hat{\\vx}-\\tilde{\\vx}\\|, \\, & \\forall \\hat{\\vx}, \\tilde{\\vx} \\in \\dom(h)\\cap\\cX, \\forall i\\in [m].\\label{eq:lip-fv}\n\\end{align}\n\\end{subequations}\n\\end{proposition}\n\n\\begin{lemma}\\label{lem:lip}\nAssume \\eqref{eq:lip-g}, \\eqref{eq:lip-f}, and the boundedness of $\\dom(h)\\cap \\cX$. Let $B_i$'s be given in Proposition \\ref{prop-lip}. Then $\\nabla_\\vx F_{\\beta_k}(\\vx,\\vy^k,\\vz^k)$ is Lipschitz continuous on $\\dom(h)\\cap\\cX$ in terms of $\\vx$ with constant\n\\begin{equation}\\label{eq:def-Lz}\nL(\\vz^k) = L_0+\\beta_k\\|\\vA^\\top \\vA\\|+\\sum_{i=1}^m \\big(\\beta_k B_i(B_i+L_i)+L_i|z_i^k|\\big).\n\\end{equation}\n\\end{lemma}\n\n\\begin{proof}\nFor ease of description, we let $\\beta=\\beta_k$ and $(\\vy,\\vz)=(\\vy^k,\\vz^k)$ in the proof. First we notice that $\\frac{\\partial}{\\partial u}\\psi_\\beta(u,v)=[\\beta u+v]_+$, and thus for any $v$,\n$$\\left|\\frac{\\partial}{\\partial u}\\psi_\\beta(\\hat{u},v)-\\frac{\\partial}{\\partial u}\\psi_\\beta(\\tilde{u},v)\\right|\\le \\beta|\\hat{u}-\\tilde{u}|,\\,\\forall \\hat{u},\\tilde{u}.$$\nLet $h_i(x,z_i) = \\psi_\\beta(f_i(\\vx),z_i),\\,i=1,\\ldots,m$. Then\n\\begin{align*}\n&\\|\\nabla_\\vx h_i(\\hat{\\vx},z_i)-\\nabla_\\vx h_i(\\tilde{\\vx},z_i)\\|\\\\\n=&\\big\\|\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\hat{\\vx}),z_i)\\nabla f_i(\\hat{\\vx})-\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\tilde{\\vx}),z_i)\\nabla f_i(\\tilde{\\vx})\\big\\|\\\\\n\\le&\\big\\|\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\hat{\\vx}),z_i)\\nabla f_i(\\hat{\\vx})-\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\tilde{\\vx}),z_i)\\nabla f_i(\\hat{\\vx})\\big\\|\\\\\n&+\\big\\|\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\tilde{\\vx}),z_i)\\nabla f_i(\\hat{\\vx})-\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\tilde{\\vx}),z_i)\\nabla f_i(\\tilde{\\vx})\\big\\|\\\\\n\\le& \\beta|f_i(\\hat{\\vx})-f_i(\\tilde{\\vx})|\\cdot\\|\\nabla f_i(\\hat{\\vx})\\|+\\big|\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\tilde{\\vx}),z_i)\\big|\\cdot\\|\\nabla f_i(\\hat{\\vx})-\\nabla f_i(\\tilde{\\vx})\\|\\\\\n\\le & \\beta B_i^2\\|\\hat{\\vx}-\\tilde{\\vx}\\|+ L_i(\\beta B_i + |z_i|) \\|\\hat{\\vx}-\\tilde{\\vx}\\|.\n\\end{align*}\nHence,\n\\begin{align*}\n&\\|\\nabla_\\vx F_\\beta(\\hat{\\vx},\\vy,\\vz)-\\nabla_\\vx F_\\beta(\\tilde{\\vx},\\vy,\\vz)\\|\\\\\n\\le& \\|\\nabla g(\\hat{\\vx})-\\nabla g(\\tilde{\\vx})\\|+\\beta\\|\\vA^\\top \\vA(\\hat{\\vx}-\\tilde{\\vx})\\| + \\sum_{i=1}^m \\|\\nabla_\\vx h_i(\\hat{\\vx},z_i)-\\nabla_\\vx h_i(\\tilde{\\vx},z_i)\\|\\\\\n\\le & \\left(L_0+\\beta\\|\\vA^\\top \\vA\\|+\\sum_{i=1}^m \\big[\\beta B_i^2+ L_i(\\beta B_i + |z_i|)\\big]\\right)\\|\\hat{\\vx}-\\tilde{\\vx}\\|,\n\\end{align*}\nwhich completes the proof.\n\\end{proof}\n\nTherefore, letting\n$\\phi(\\vx) = F_{\\beta_k}(\\vx,\\vy^k ,\\vz^k  )$ and $\\psi(\\vx) = h(\\vx)+\\iota_\\cX(\\vx),$\nwe can apply Nesterov's optimal first-order method in Algorithm \\ref{alg:apg} to find $\\vx^{k+1}$ in \\eqref{eq:ialm-x}. From Theorem \\ref{thm:rate-apg}, we have the following results. Note that if the strong convexity constant $\\mu=0$, the problem is just convex.\n\n\n\\begin{lemma}\\label{lem:sub-iter}\nAssume that $g$ is strongly convex with modulus $\\mu\\ge0$. Given $\\vareps_k>0$, if we start from $\\vx^k $ and run Algorithm \\ref{alg:apg}, then at most $t_k$ iterations are needed to produce $\\vx^{k+1}$ such that \\eqref{eq:ialm-x} holds,\nwhere \n\\begin{equation}\\label{eq:tk-cvx}\nt_k = \\left\\{\n\\begin{array}{ll}\n\\left\\lceil\\dfrac{\\dist(\\vx^k ,\\cX_k^*)\\sqrt{2L(\\vz^k  )}}{\\sqrt{\\vareps_k}}\\right\\rceil, & \\text{ if }\\mu=0,\\\\[0.5cm]\n\\left\\lceil\\dfrac{\\log\\left(\\frac{L(\\vz^k  )+\\mu}{2\\vareps_k}[\\dist(\\vx^k ,\\cX_k^*)]^2\\right)}{\\log1/\\big(1-\\sqrt{\\frac{\\mu}{L(\\vz^k  )}}\\big)}\\right\\rceil, & \\text{ if }\\mu>0,\n\\end{array}\n\\right.\n\\end{equation}\nand $\\cX_k^*$ denotes the set of optimal solutions to $\\min_{\\vx\\in \\cX}\\cL_{\\beta_k}(\\vx,\\vy^k ,\\vz^k  )$.\n%with\n%$$\\phi(\\vx) = F_\\beta(x,\\vy^k ,\\vz^k  ),\\quad \\psi(\\vx) = h(\\vx)+\\iota_\\cX(\\vx),$$ then we have\n%\\begin{equation}\\label{eq:ialm-xk-apg}\n%\\cL_\\beta(\\vx^{k+1},\\vy^k ,\\vz^k  )-\\min_{x\\in \\cX}\\cL_\\beta(x,\\vy^k ,\\vz^k  ) \\le \\frac{2L(\\vz^k  )[\\dist(\\vx^k ,\\cX_k^*)]^2}{t_k^2},\n%\\end{equation}\n%where $\\cX_k^*$ denotes the set of optimal solutions to $\\min_{x\\in \\cX}\\cL_\\beta(x,\\vy^k ,\\vz^k  )$. Hence, to guarantee a point $\\vx^{k+1}$ satisfying \\eqref{eq:ialm-x}, it suffices to evaluate the gradients of $g$ and $f_i, i\\in [m]$ for $t_k$ times, where \n \n\\end{lemma}\n\n\\begin{lemma}\\label{lem:bd-z-Lz}\nLet $\\{(\\vx^k,\\vy^k,\\vz^k)\\}_{k=0}^K$ be the sequence generated from Algorithm \\ref{alg:ialm} with $\\{\\beta_k\\}$ and $\\{\\rho_k\\}$ set according to either Setting \\ref{set-penalty} or Setting \\ref{set-ialm}. If $\\vy^0=\\vzero, \\vz^0=\\vzero$, and $\\vareps_k$'s are chosen such that \n\\begin{equation}\\label{eq:req-eps}\n\\sum_{k=0}^{K-1}\\rho_k\\vareps_k\\le \\frac{C_\\vareps}{2},\n\\end{equation}\n then\n\\begin{align}\\label{eq:bd-Lzk}\nL(\\vz^k)\\le L_*+\\beta_k H,\\,\\forall \\, 0\\le k\\le K,\n\\end{align}\nwhere \n$$ H=\\|\\vA^\\top \\vA\\|+ \\sum_{i=1}^m  B_i(B_i+L_i),\\ L_*=L_0+ \\|\\vell\\| \\left(\\|\\vy^*\\|+2\\|\\vz^*\\|+\\sqrt{C_\\vareps}\\right)$$\nand $\\vell$ is given in \\eqref{eq:vec-ell}.\n\\end{lemma}\n\n\\begin{proof}\nLetting $(\\vx,\\vy,\\vz)=(\\vx^*,\\vy^*,\\vz^*)$ in \\eqref{eq:ialm-ineq2-1} and using \\eqref{eq:opt}, we have \n$$\\frac{1}{2}\\|\\vy^{k+1} -\\vy^*\\|^2 + \\frac{1}{2}\\|\\vz^{k+1} -\\vz^*\\|^2\n\\le  \\frac{1}{2}\\|\\vy^k -\\vy^*\\|^2+ \\frac{1}{2}\\|\\vz^k  -\\vz^*\\|^2+\\rho_k\\vareps_k.$$\nSumming the above inequality yields \n$$\\frac{1}{2}\\|\\vy^{k}-\\vy^*\\|^2 + \\frac{1}{2}\\|\\vz^{k}-\\vz^*\\|^2\n\\le  \\frac{1}{2}\\|\\vy^0-\\vy^*\\|^2+ \\frac{1}{2}\\|\\vz^0-\\vz^*\\|^2+\\sum_{t=0}^{k-1}\\rho_t\\vareps_t, \\, \\forall \\, 0\\le k \\le K,$$\nwhich implies\n$$\\|\\vz^k  \\| \\le \\|\\vz^*\\|+\\sqrt{\\|\\vy^0-\\vy^*\\|^2+\\|\\vz^0-\\vz^*\\|^2+2\\sum_{t=0}^{k-1}\\rho_t\\vareps_t}.$$\nSince $\\|\\vu\\|\\le \\|\\vu\\|_1$ for any vector $\\vu$, we have from the above inequality that\n\\begin{equation}\\label{eq:bd-zk0}\n\\|\\vz^k  \\| \\le  \\|\\vz^*\\|+\\|\\vy^0-\\vy^*\\|+\\|\\vz^0-\\vz^*\\|+\\sqrt{2\\sum_{t=0}^{K-1}\\rho_t\\vareps_t},\\, \\forall \\, 0\\le k \\le K.\n\\end{equation}\n\nHence, if $\\vy^0=\\vzero$ and $\\vz^0=\\vzero$, and \\eqref{eq:req-eps} holds, it follows from the above inequality that\n\\begin{equation}\\label{eq:bd-zk}\n\\|\\vz^k  \\| \\le  \\|\\vy^*\\|+2\\|\\vz^*\\|+\\sqrt{C_\\vareps},\\, \\forall \\, 0\\le k \\le K,\n\\end{equation}\nBy the Cauchy-Schwartz inequality, we have from \\eqref{eq:def-Lz} that for any $0\\le k\\le K$,\n$$L(\\vz^k  ) \\le L_0+\\beta_k H + \\|\\vz^k  \\|\\cdot\\|\\vell\\|$$\nwhich together with \\eqref{eq:bd-zk} gives the result in \\eqref{eq:bd-Lzk}.\n\\end{proof}\n\n\\begin{proof}\nThe results in \\eqref{eq:error-barx-K} directly follows from Theorem \\ref{thm:ialm-rate} and the settings of $\\{\\beta_k\\}$, $\\{\\rho_k\\}$, and $\\{\\vareps_k\\}$. For the total number of gradient evaluations, we use the inequalities in \\eqref{eq:tk-cvx}. First, for the case of $\\mu=0$, from the first inequality of \\eqref{eq:tk-cvx} and the parameter setting, it follows that the total number of gradient evaluations\n\\begin{equation}\\label{eq:total-grad-cvx-adp1}\nT_K\\le \\sum_{k=0}^{K-1}\\frac{\\dist(\\vx^k ,\\cX_k^*)\\sqrt{2(L_*+\\frac{C_\\beta H}{K\\vareps})}}{\\sqrt{\\vareps/2}\\sqrt{C_\\vareps/C_\\beta}}+K.\n\\end{equation}\nSince $\\sqrt{a+b}\\le \\sqrt{a}+\\sqrt{b}$ for any two nonnegative numbers $a,b$, we have from the above inequality and by noting $\\dist(\\vx^k ,\\cX_k^*)\\le D$ that\n\\begin{align*}\nT_K \\le 2D\\sqrt{\\frac{C_\\beta}{C_\\vareps}}\\sum_{k=0}^{K-1}\\frac{\\sqrt{L_*}+\\sqrt{\\frac{C_\\beta H}{K\\vareps}}}{\\sqrt{\\vareps}}+K=2DK\\sqrt{\\frac{C_\\beta}{C_\\vareps}}\\left(\\sqrt{\\frac{L_*}{\\vareps}}+\\frac{1}{\\vareps}\\sqrt{\\frac{C_\\beta H}{K}}\\right)+K,\n\\end{align*}\nwhich gives \\eqref{eq:T-penaly-cvx}.\n\nFor the case of $\\mu>0$, we first note that for any $0<a\\le1$, it holds $\\log(1+a)\\ge a-\\frac{a^2}{2}\\ge\\frac{a}{2}$. Hence, if $\\mu\\le \\frac{L_0}{4}$, we have $\\mu\\le \\frac{L(\\vz^k)}{4}$ and $\\frac{\\sqrt{\\mu/L(\\vz^k)}}{1-\\sqrt{\\mu/L(\\vz^k)}}\\le 1$. Therefore,\n$$\\log\\frac{1}{1-\\sqrt{\\mu/L(\\vz^k)}}=\\log\\left(1+\\frac{\\sqrt{\\mu/L(\\vz^k)}}{1-\\sqrt{\\mu/L(\\vz^k)}}\\right)\\ge \\frac{1}{2}\\frac{\\sqrt{\\mu/L(\\vz^k)}}{1-\\sqrt{\\mu/L(\\vz^k)}},$$ and thus\n\\begin{equation}\\label{eq:bd-denom}\n\\frac{1}{\\log\\frac{1}{1-\\sqrt{\\mu/L(\\vz^k)}}}\\le 2\\sqrt{\\frac{L(\\vz^k)}{\\mu}}\\left(1-\\sqrt{\\mu/L(\\vz^k)}\\right) \\le 2\\sqrt{\\frac{L(\\vz^k)}{\\mu}}.\n\\end{equation} Using the above inequality and the second inequality of \\eqref{eq:tk-cvx}, we have that the total number of gradient evaluations\n\\begin{equation}\nT_K\\le  \\sum_{k=0}^{K-1}2\\sqrt{\\frac{L_*+\\frac{C_\\beta H}{K\\vareps}}{\\mu}}\\log\\left(\\frac{L_*+\\frac{C_\\beta H}{K\\vareps}+\\mu}{\\vareps C_\\vareps/C_\\beta}[\\dist(\\vx^k ,\\cX_k^*)]^2\\right)+K.\\label{eq:total-grad-scvx-adp1}\n\\end{equation}\nSince $\\sqrt{L_*+\\frac{C_\\beta H}{K\\vareps}}\\le \\sqrt{L_*}+\\sqrt{\\frac{C_\\beta H}{K\\vareps}}$ and $\\dist(\\vx^k ,\\cX_k^*)\\le D$, the above inequality implies \\eqref{eq:T-penaly-strcvx}.\nThis completes the proof.\n\\end{proof}","units":[{"start":32702,"end":33471,"start_line":473,"end_line":483,"raw_tex":"\\begin{lemma}\\label{lem:pre-rate}\nAssume $(\\vx^*,\\vy^*,\\vz^*)$ satisfies the KKT conditions in \\eqref{eq:kkt}. Let $\\bar{\\vx}$ be a point such that for any $\\vy$ and any $\\vz\\ge\\vzero$,\n\\begin{equation}\\label{eq:pre-rate-cond}\nf_0(\\bar{\\vx})-f_0(\\vx^*)+ \\vy^\\top (\\vA\\bar{\\vx}-\\vb) +\\sum_{i=1}^m z_i f_i(\\bar{\\vx}) \\le \\alpha + c_1\\|\\vy\\|^2+c_2\\|\\vz\\|^2,\n\\end{equation}\nwhere $\\alpha$ and $c_1, c_2$ are nonnegative constants independent of $\\vy$ and $\\vz$. Then\n\\begin{align}\n-\\left(\\alpha + 4c_1\\|\\vy^*\\|^2+ 4c_2\\|\\vz^*\\|^2\\right)\\le f_0(\\bar{\\vx})-f_0(\\vx^*)\\le \\alpha,\\label{eq:pre-rate-obj}\\\\\n\\|\\vA\\bar{\\vx}-\\vb\\| + \\big\\|[\\vf(\\bar{\\vx})]_+\\big\\| \\le \\alpha + c_1\\big(1+\\|\\vy^*\\|\\big)^2 + c_2\\big(1+\\|\\vz^*\\|\\big)^2. \\label{eq:pre-rate-res}\n\\end{align}\n\\end{lemma}","sha256_utf8":"42137643fe5232f3595ee78956cefa9291c1e399937c42619cc143f2cc7e86b9"},{"start":33473,"end":35078,"start_line":485,"end_line":506,"raw_tex":"\\begin{proof}\nLetting $\\vy=\\vzero$ and $\\vz=\\vzero$ in \\eqref{eq:pre-rate-cond} gives the second inequality in \\eqref{eq:pre-rate-obj}.\nFor any nonnegative $\\gamma_y$ and $\\gamma_z$, we let \n$$\\vy=\\gamma_y\\frac{\\vA\\bar{\\vx}-\\vb}{\\|\\vA\\bar{\\vx}-\\vb\\|},\\quad \\vz = \\gamma_z \\frac{[\\vf(\\bar{\\vx})]_+}{\\big\\|[\\vf(\\bar{\\vx})]_+\\big\\|}$$ %z_i = \\gamma_i \\frac{[f_i(\\bar{\\vx})]_+}{|f_i(\\bar{\\vx})|}, \\forall i\\in [m]$$\nand have from \\eqref{eq:pre-rate-cond} by using the convention $\\frac{0}{0}=0$ that\n\\begin{align}\\label{eq:pre-rate-ineq2}\nf_0(\\bar{\\vx})-f_0(\\vx^*) +\\gamma_y \\|\\vA\\bar{\\vx}-\\vb\\| + \\gamma_z\\big\\|[\\vf(\\bar{\\vx})]_+\\big\\|\n\\le  \\alpha + c_1\\gamma_y^2+ c_2\\gamma_z^2.\n%f_0(\\bar{\\vx})-f_0(\\vx^*) +\\gamma_y \\|\\vA\\bar{\\vx}-\\vb\\| + \\sum_{i=1}^m \\gamma_i [f_i(\\bar{\\vx})]_+\n%\\le  \\alpha + c_1\\gamma_y^2+ c_2\\sum_{i=1}^m\\gamma_i^2.\n\\end{align}\n\nNoting \n\\begin{equation}\\label{eq:pre-rate-ineq3}\n-\\langle \\vy^*, \\vA\\bar{\\vx}-\\vb\\rangle \\ge -\\|\\vy^*\\|\\cdot\\|\\vA\\bar{\\vx}-\\vb\\|,\\quad - \\sum_{i=1}^mz_i^* f_i(\\bar{\\vx}) \\ge - \\|\\vz^*\\|\\cdot \\big\\|[\\vf(\\bar{\\vx})]_+\\big\\|,\n\\end{equation} we have from \\eqref{eq:opt} and \\eqref{eq:pre-rate-ineq2} that\n\\begin{align*}\n(\\gamma_y-\\|\\vy^*\\|) \\|\\vA\\bar{\\vx}-\\vb\\| + (\\gamma_z -\\|\\vz^*\\|)\\|[\\vf(\\bar{\\vx})]_+\\big\\|\n\\le  \\alpha + c_1\\gamma_y^2+ c_2\\gamma_z^2\n\\end{align*}\nIn the above inequality, letting $\\gamma_y=1+\\|\\vy^*\\|$ and $\\gamma_z = 1+\\|\\vz^*\\|$ gives \\eqref{eq:pre-rate-res}, and letting $\\gamma_y=2\\|\\vy^*\\|$ and $\\gamma_z = 2\\|\\vz^*\\|$ gives the first inequality in \\eqref{eq:pre-rate-obj} by \\eqref{eq:opt} and \\eqref{eq:pre-rate-ineq3}.\n\\end{proof}","sha256_utf8":"4c34aca26b3e6494ca3d165acb71cf3d6e6e0e0c626c448fe8fb70aa590ef3ad"},{"start":36954,"end":37657,"start_line":527,"end_line":539,"raw_tex":"\\begin{theorem}\\label{thm:rate-apg}\nLet $\\{\\vx^k \\}$ be the sequence generated from Algorithm \\ref{alg:apg}. Assume $\\vx^*$ to be a minimizer of \\eqref{eq:comp-prob}. The following results holds:\n\\begin{enumerate}\n\\item If $\\mu=0$ and $\\alpha_0=1$, then\n\\begin{equation}\\label{eq:optimal-rate-cvx}\n\\phi(\\vx^k) + \\psi(\\vx^k) - \\phi(\\vx^*)-\\psi(\\vx^*) \\le \\frac{2L_\\phi\\|\\vx^0-\\vx^*\\|^2}{k^2},\\,\\forall k\\ge1.\n\\end{equation}\n\\item If $\\mu >0$ and $\\alpha_0=\\sqrt{\\frac{\\mu}{L_\\phi}}$, then\n\\begin{equation}\\label{eq:optimal-rate-scvx}\n\\phi(\\vx^k)-\\phi(\\vx^*)\\le \\frac{(L_\\phi+\\mu)\\|\\vx^0-\\vx^*\\|^2}{2}\\left(1-\\sqrt{\\frac{\\mu}{L_\\phi}}\\right)^k,\\,\\forall k\\ge1.\n\\end{equation}\n\\end{enumerate}\n\\end{theorem}","sha256_utf8":"ad5550fc04d08f38fec3f578fdce0509a5ab3b8ea1c82a69f8d5ae51ca3ca313"},{"start":40411,"end":40993,"start_line":583,"end_line":590,"raw_tex":"\\begin{lemma}\\label{lem:multiplier}\nLet $\\vy$ and $\\vz$ be updated by \\eqref{eq:alm-y} and \\eqref{eq:alm-z} respectively. Then for any $k$, it holds\n\\begin{align}\n\\frac{1}{2\\rho_k}\\big[\\|\\vy^{k+1} -\\vy\\|^2-\\|\\vy^k -\\vy\\|^2+\\|\\vy^{k+1} -\\vy^k \\|^2\\big] - \\langle \\vy^{k+1} -\\vy, \\vr^{k+1}\\rangle = &~ 0,\\label{eq:alm-yterm}\\\\\n\\frac{1}{2\\rho_k}\\big[\\|\\vz^{k+1} -\\vz\\|^2-\\|\\vz^k  -\\vz\\|^2+\\|\\vz^{k+1} -\\vz^k  \\|^2\\big]- \\sum_{i=1}^m (z_i^{k+1}-z_i)\\cdot\\max\\big(-\\frac{z_i^k}{\\beta_k}, f_i(\\vx^{k+1})\\big)=& ~0,\\label{eq:alm-zterm}\n\\end{align}\nwhere $\\vr^k =\\vA\\vx^k -\\vb$.\n\\end{lemma}","sha256_utf8":"ef43df6ffaca93fcebe60fd3015c31c243a902c250d66ac4bb3c680504dbe585"},{"start":40994,"end":41171,"start_line":591,"end_line":593,"raw_tex":"\\begin{proof}\nUsing the equality $2\\vu^\\top \\vv = \\|\\vu\\|^2 - \\|\\vu-\\vv\\|^2 + \\|\\vv\\|^2$, we have the results from the updates \\eqref{eq:alm-y} and \\eqref{eq:alm-z}.\n\\end{proof}","sha256_utf8":"d8f4f49dd7fd5604f1eeb1ad81365915fb9bc9bc228d1b89dd6f55ddc1df0c60"},{"start":41172,"end":41531,"start_line":594,"end_line":600,"raw_tex":"\\begin{lemma}\\label{lem:ineq-z}\nFor any $\\vz\\ge\\vzero$, we have\n\\begin{align}\\label{eq:alm-ineq-z}\n&\\sum_{i=1}^m\\big([z_i^k+\\beta_k f_i(\\vx^{k+1})]_+-z_i\\big) f_i(\\vx^{k+1})- \\sum_{i=1}^m (z_i^{k+1}-z_i)\\cdot\\max\\big(-\\frac{z_i^k}{\\beta_k}, f_i(\\vx^{k+1})\\big)\\nonumber\\\\\n\\ge&\\frac{1}{\\rho_k^2}(\\beta_k-\\rho_k)\\|\\vz^{k+1} -\\vz^k  \\|^2.\n\\end{align}\n\\end{lemma}","sha256_utf8":"7d54b371b9751a08ebef1f4ea27ac2f5e355dbaa3adadd462fcf3d6ad6c922b3"},{"start":41533,"end":42615,"start_line":602,"end_line":617,"raw_tex":"\\begin{proof}\nDenote \n\\begin{equation}\\label{eq:setIk}\nI_+^k=\\{i\\in [m]: z_i^k + \\beta_k f_i(\\vx^{k+1}) \\ge 0\\},\\quad I_-^k=[m]\\backslash I_+^k.\n\\end{equation}\n Then \n\\begin{align*}\n&\\text{the left hand side of \\eqref{eq:alm-ineq-z}}\\\\\n=&\\sum_{i\\in I_+^k}\\left[(z_i^k-z_i)f_i(\\vx^{k+1})+\\beta_k[f_i(\\vx^{k+1})]^2-\\big(z_i^k+\\rho_k f_i(\\vx^{k+1})-z_i\\big)f_i(\\vx^{k+1})\\right]\\\\\n&+\\sum_{i\\in I_-^k}\\left[-z_i f_i(\\vx^{k+1})-\\big(z_i^k-\\frac{\\rho_k z_i^k}{\\beta_k}-z_i\\big)\\big(-\\frac{z_i^k}{\\beta_k}\\big)\\right]\\\\\n=&(\\beta_k-\\rho_k)\\sum_{i\\in I_+^k}[f_i(\\vx^{k+1})]^2+\\sum_{i\\in I_-^k}\\left[-z_i \\big(f_i(\\vx^{k+1})+\\frac{z_i^k}{\\beta_k}\\big)+\\frac{1}{\\beta_k^2}(\\beta_k-\\rho_k)(z_i^k)^2\\right]\\\\\n\\ge& (\\beta_k-\\rho_k)\\sum_{i\\in I_+^k}[f_i(\\vx^{k+1})]^2 +\\frac{1}{\\beta_k^2}(\\beta_k-\\rho_k)\\sum_{i\\in I_-^k}(z_i^k)^2\\\\\n=&\\frac{1}{\\rho_k^2}(\\beta_k-\\rho_k)\\|\\vz^{k+1} -\\vz^k  \\|^2,\n\\end{align*}\nwhere the inequality follows from $z_i\\ge0$ and $f_i(\\vx^{k+1})+\\frac{z_i^k}{\\beta_k}\\le 0,\\,\\forall i\\in I_-^k$, and the last equality holds due to the update \\eqref{eq:alm-z}.\n\\end{proof}","sha256_utf8":"7702f704d1d3b7194f0dd720fbd2a7c03deed3abb05c802f064eb2d30d09145c"},{"start":52990,"end":53697,"start_line":739,"end_line":746,"raw_tex":"\\begin{theorem}[One-iteration progress of iALM]\\label{thm:1iter-ialm}\nLet $\\{(\\vx^k ,\\vy^k ,\\vz^k  )\\}$ be the sequence generated from Algorithm \\ref{alg:ialm}. Then for any $\\vx\\in\\cX$ such that $\\vA\\vx=\\vb$ and $f_i(\\vx)\\le 0,\\,\\forall i\\in [m]$, any $\\vy$, and any $\\vz\\ge\\vzero$, it holds that\n\\begin{align}\\label{eq:ialm-ineq2}\n&f_0(\\vx^{k+1})-f_0(\\vx) +\\vy^\\top \\vr^{k+1} + \\sum_{i=1}^m z_i f_i(\\vx^{k+1})+\\frac{\\beta_k-\\rho_k}{2}\\|\\vr^{k+1}\\|^2\\cr\n&+\\frac{\\beta_k-\\rho_k}{2\\rho_k^2}\\|\\vz^{k+1} -\\vz^k  \\|^2+\\frac{1}{2\\rho_k}\\|\\vy^{k+1} -\\vy\\|^2 + \\frac{1}{2\\rho_k}\\|\\vz^{k+1} -\\vz\\|^2\\cr\n\\le & \\frac{1}{2\\rho_k}\\|\\vy^k -\\vy\\|^2+ \\frac{1}{2\\rho_k}\\|\\vz^k  -\\vz\\|^2+\\vareps_k.\n\\end{align}\n\\end{theorem}","sha256_utf8":"ebbf16030cdb2aa8602ef3ae38c6284c0dbad69ccf5edca53becd28b958d8f4b"},{"start":53961,"end":55976,"start_line":750,"end_line":773,"raw_tex":"\\begin{proof}\nFrom \\eqref{eq:ialm-x}, it follows that for any $\\vx$ such that $\\vA\\vx=\\vb$,\n\\begin{equation}\\label{eq:ialm-1st}\nf_0(\\vx^{k+1})+\\langle \\vy^k , \\vr^{k+1}\\rangle +\\frac{\\beta_k}{2}\\|\\vr^{k+1}\\|^2+\\Psi_{\\beta_k}(\\vx^{k+1},\\vz^k  )-f_0(\\vx)-\\Psi_{\\beta_k}(\\vx,\\vz^k  )\\le\\vareps_k.\n\\end{equation}\nSince $\\langle \\vy^k , \\vr^{k+1}\\rangle=\\langle \\vy^{k+1} -\\vy, \\vr^{k+1}\\rangle + \\langle \\vy, \\vr^{k+1}\\rangle -\\rho_k\\|\\vr^{k+1}\\|^2$, by adding \\eqref{eq:alm-yterm} and \\eqref{eq:alm-zterm} to the above inequality, we have \n\\begin{align}\\label{eq:ialm-ineq1}\n&f_0(\\vx^{k+1})-f_0(\\vx) +\\vy^\\top \\vr^{k+1} + \\sum_{i=1}^m z_i f_i(\\vx^{k+1})+\\sum_{i=1}^m\\left([z_i^k+\\beta_k f_i(\\vx^{k+1})]_+-z_i\\right)f_i(\\vx^{k+1})\\cr\n&+\\big(\\frac{\\beta_k}{2}-\\rho_k\\big)\\|\\vr^{k+1}\\|^2+\\Psi_{\\beta_k}(\\vx^{k+1},\\vz^k  )-\\sum_{i=1}^m[z_i^k+\\beta_k f_i(\\vx^{k+1})]_+f_i(\\vx^{k+1})-\\Psi_{\\beta_k}(\\vx,\\vz^k  )\\cr\n&+\\frac{1}{2\\rho_k}\\big[\\|\\vy^{k+1} -\\vy\\|^2-\\|\\vy^k -\\vy\\|^2+\\|\\vy^{k+1} -\\vy^k \\|^2\\big] \\cr\n&+\\frac{1}{2\\rho_k}\\big[\\|\\vz^{k+1} -\\vz\\|^2-\\|\\vz^k  -\\vz\\|^2+\\|\\vz^{k+1} -\\vz^k  \\|^2\\big]- \\sum_{i=1}^m (z_i^{k+1}-z_i)\\cdot\\max\\big(-\\frac{z_i^k}{\\beta_k}, f_i(\\vx^{k+1})\\big)\\cr\n\\le & \\vareps_k.\n\\end{align}\n\nNote that\n\\begin{align}\\label{eq:ialm-ineq3}\n&\\Psi_{\\beta_k}(\\vx^{k+1},\\vz^k  )-\\sum_{i=1}^m[z_i^k+\\beta_k f_i(\\vx^{k+1})]_+f_i(\\vx^{k+1})\\cr\n=&\\sum_{i\\in I_+^k}\\left[z_i^k f_i(\\vx^{k+1}) +\\frac{\\beta_k}{2}[f_i(\\vx^{k+1})]^2-[z_i^k +\\beta_k f_i(\\vx^{k+1})]f_i(\\vx^{k+1})\\right]+\\sum_{i\\in I_-^k}\\left[-\\frac{(z_i^k)^2}{2\\beta_k}\\right]\\cr\n=&-\\sum_{i\\in I_+^k}\\frac{\\beta_k}{2}[f_i(\\vx^{k+1})]^2-\\sum_{i\\in I_-^k}\\frac{(z_i^k)^2}{2\\beta_k}\\cr\n=&-\\frac{\\beta_k}{2\\rho_z^2}\\|\\vz^{k+1} -\\vz^k  \\|^2,\n\\end{align}\nwhere the sets $I_+^k$ and $I_-^k$ are defined in \\eqref{eq:setIk}.\nIn addition, if $f_i(\\vx)\\le 0,\\,\\forall i\\in [m]$, then $\\Psi_{\\beta_k}(\\vx,\\vz^k  )\\le 0.$ Hence, plugging \\eqref{eq:alm-ineq-z} and \\eqref{eq:ialm-ineq3} into \\eqref{eq:ialm-ineq1} yields \\eqref{eq:ialm-ineq2}.\n\\end{proof}","sha256_utf8":"40988c51f17573a213b2e0e3617f7718eab46679d7406fb32ae8de57fdb38626"},{"start":59248,"end":60183,"start_line":799,"end_line":811,"raw_tex":"\\begin{theorem}[Ergodic convergence rate of iALM]\\label{thm:ialm-rate}\nUnder Assumptions \\ref{assump-kkt} and \\ref{assump-wd-ialm}, let $\\{(\\vx^k ,\\vy^k ,\\vz^k  )\\}$ be the sequence generated from Algorithm \\ref{alg:ialm} with $\\vy^0=\\vzero, \\vz^0 = \\vzero$ and $0<\\rho_k\\le\\beta_k,\\,\\forall k$. Then\n\\begin{subequations}\\label{eq:ialm-rate}\n\\begin{align}\n\\big|f_0(\\bar{\\vx}^{K})-f_0(\\vx^*)\\big| \\le \\frac{1}{\\sum_{t=0}^{K-1}\\rho_t}\\left(2\\|\\vy^*\\|^2+ 2\\|\\vz^*\\|^2+\\sum_{k=0}^{K-1}\\rho_k\\vareps_k\\right),\\label{eq:ialm-rate-obj}\\\\\n\\|\\vA\\bar{\\vx}^{K}-\\vb\\| + \\big\\| [\\vf(\\bar{\\vx}^{K})]_+\\big\\| \\le  \\frac{1}{\\sum_{t=0}^{K-1}\\rho_t}\\left(\\frac{(1+\\|\\vy^*\\|)^2}{2}+ \\frac{(\\|1+\\|\\vz^*\\|)^2}{2}+\\sum_{k=0}^{K-1}\\rho_k\\vareps_k\\right),\\label{eq:ialm-rate-res}\n\\end{align}\n\\end{subequations}\nwhere \n\\begin{equation}\\label{eq:barx-K}\n\\bar{\\vx}^{K}=\\frac{\\sum_{t=0}^{K-1}\\rho_t\\vx^{t+1} }{\\sum_{t=0}^{K-1}\\rho_t}.\n\\end{equation}\n\\end{theorem}","sha256_utf8":"89dfc3784575558d71c7f5d2c71fb58009a502d9dbf9256b8b5e2ccfbbd2b3d5"},{"start":60185,"end":62021,"start_line":813,"end_line":837,"raw_tex":"\\begin{proof}\nSince $\\rho_k\\le \\beta_k$, the two terms $\\frac{\\beta_k-\\rho_k}{2}\\|\\vr^{k+1}\\|^2$ and \n$\\frac{\\beta_k-\\rho_k}{2\\rho_k^2}\\|\\vz^{k+1} -\\vz^k  \\|^2$ are nonnegative. Dropping them and multiplying $\\rho_k$ to both sides of \\eqref{eq:ialm-ineq2} yields \n\\begin{align}\\label{eq:ialm-ineq2-1}\n&\\rho_k\\left[f_0(\\vx^{k+1})-f_0(\\vx) +\\vy^\\top  \\vr^{k+1} + \\sum_{i=1}^m z_i f_i(\\vx^{k+1})\\right]+\\frac{1}{2}\\|\\vy^{k+1} -\\vy\\|^2 + \\frac{1}{2}\\|\\vz^{k+1} -\\vz\\|^2\\cr\n\\le & \\frac{1}{2}\\|\\vy^k -\\vy\\|^2+ \\frac{1}{2}\\|\\vz^k  -\\vz\\|^2+\\rho_k\\vareps_k.\n\\end{align}\n\nSumming up \\eqref{eq:ialm-ineq2} with $\\vx=\\vx^*$ gives\n\\begin{align}\\label{eq:ialm-ineq2-2}\n&\\sum_{k=0}^{K-1}\\rho_k\\left[f_0(\\vx^{k+1})-f_0(\\vx^*) +\\vy^\\top \\vr^{k+1} + \\sum_{i=1}^m z_i f_i(\\vx^{k+1})\\right]+\\frac{1}{2}\\|\\vy^{K}-\\vy\\|^2 + \\frac{1}{2}\\|\\vz^{K}-\\vz\\|^2\\cr\n\\le & \\frac{1}{2}\\|\\vy^0-\\vy\\|^2+ \\frac{1}{2}\\|\\vz^0-\\vz\\|^2+\\sum_{k=0}^{K-1}\\rho_k\\vareps_k.\n\\end{align}\nBy the convexity of $f_i$'s and the nonnegativity of $\\vz$, we have\n\\begin{align*}\n&~f_0(\\bar{\\vx}^{K})-f_0(\\vx^*) +\\vy^\\top  (\\vA\\bar{\\vx}^{K}-\\vb) + \\sum_{i=1}^m z_i f_i(\\bar{\\vx}^{K})\\\\\n\\le &~\\frac{1}{\\sum_{t=0}^{K-1}\\rho_t} \\sum_{k=0}^{K-1}\\rho_k\\left[f_0(\\vx^{k+1})-f_0(\\vx^*) +\\vy^\\top  \\vr^{k+1} + \\sum_{i=1}^m z_i f_i(\\vx^{k+1})\\right],\n\\end{align*}\nwhich together with \\eqref{eq:ialm-ineq2-2} implies\n$$f_0(\\bar{\\vx}^{K})-f_0(\\vx^*) +\\vy^\\top  (\\vA\\bar{\\vx}^{K}-\\vb) + \\sum_{i=1}^m z_i f_i(\\bar{\\vx}^{K})\n\\le  \\frac{1}{\\sum_{t=0}^{K-1}\\rho_t}\\left(\\frac{1}{2}\\|\\vy\\|^2+ \\frac{1}{2}\\|\\vz\\|^2+\\sum_{k=0}^{K-1}\\rho_k\\vareps_k\\right).$$\nThe results thus follow from Lemma \\ref{lem:pre-rate} with \n$$\\alpha=\\frac{\\sum_{k=0}^{K-1}\\rho_k\\vareps_k}{\\sum_{k=0}^{K-1}\\rho_k},\\ c_1=\\frac{1}{2\\sum_{k=0}^{K-1}\\rho_k},\\ c_2=\\frac{1}{2\\sum_{k=0}^{K-1}\\rho_k},$$\nand we complete the proof.\n\\end{proof}","sha256_utf8":"7be0d2d9154abfcd09f49b1621e1d7f4cdf89011eb0c82e5d999256b8b84e60e"},{"start":63390,"end":64053,"start_line":855,"end_line":864,"raw_tex":"The following results are easy to show from the Lipschitz differentiability of $g$ and $f_i,\\, i\\in [m]$.\n\\begin{proposition}\\label{prop-lip}\nAssume \\eqref{eq:lip-g}, \\eqref{eq:lip-f}, and the boundedness of $\\dom(h)\\cap \\cX$. Then there exist constants $B_1,\\ldots, B_m$ such that\n\\begin{subequations}\\label{eq:lip}\n\\begin{align}\n\\max\\big(|f_i(\\vx)|, \\|\\nabla f_i(\\vx)\\|\\big) \\le B_i, \\,& \\forall \\vx\\in \\dom(h)\\cap\\cX, \\forall i\\in [m],\\label{eq:bd-f}\\\\\n|f_i(\\hat{\\vx})-f_i(\\tilde{\\vx})| \\le B_i\\|\\hat{\\vx}-\\tilde{\\vx}\\|, \\, & \\forall \\hat{\\vx}, \\tilde{\\vx} \\in \\dom(h)\\cap\\cX, \\forall i\\in [m].\\label{eq:lip-fv}\n\\end{align}\n\\end{subequations}\n\\end{proposition}","sha256_utf8":"f734d0c296a4644e0671bec0a3e9c56f7780cc7b0df017725db3450bd1e09f89"},{"start":64312,"end":64756,"start_line":869,"end_line":874,"raw_tex":"\\begin{lemma}\\label{lem:lip}\nAssume \\eqref{eq:lip-g}, \\eqref{eq:lip-f}, and the boundedness of $\\dom(h)\\cap \\cX$. Let $B_i$'s be given in Proposition \\ref{prop-lip}. Then $\\nabla_\\vx F_{\\beta_k}(\\vx,\\vy^k,\\vz^k)$ is Lipschitz continuous on $\\dom(h)\\cap\\cX$ in terms of $\\vx$ with constant\n\\begin{equation}\\label{eq:def-Lz}\nL(\\vz^k) = L_0+\\beta_k\\|\\vA^\\top \\vA\\|+\\sum_{i=1}^m \\big(\\beta_k B_i(B_i+L_i)+L_i|z_i^k|\\big).\n\\end{equation}\n\\end{lemma}","sha256_utf8":"67cfbc13f857c0d87a63221a86779db377f2013addb2d994b400ea2af0eab9e9"},{"start":64757,"end":66599,"start_line":875,"end_line":894,"raw_tex":"\\begin{proof}\nFor ease of description, we let $\\beta=\\beta_k$ and $(\\vy,\\vz)=(\\vy^k,\\vz^k)$ in the proof. First we notice that $\\frac{\\partial}{\\partial u}\\psi_\\beta(u,v)=[\\beta u+v]_+$, and thus for any $v$,\n$$\\left|\\frac{\\partial}{\\partial u}\\psi_\\beta(\\hat{u},v)-\\frac{\\partial}{\\partial u}\\psi_\\beta(\\tilde{u},v)\\right|\\le \\beta|\\hat{u}-\\tilde{u}|,\\,\\forall \\hat{u},\\tilde{u}.$$\nLet $h_i(x,z_i) = \\psi_\\beta(f_i(\\vx),z_i),\\,i=1,\\ldots,m$. Then\n\\begin{align*}\n&\\|\\nabla_\\vx h_i(\\hat{\\vx},z_i)-\\nabla_\\vx h_i(\\tilde{\\vx},z_i)\\|\\\\\n=&\\big\\|\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\hat{\\vx}),z_i)\\nabla f_i(\\hat{\\vx})-\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\tilde{\\vx}),z_i)\\nabla f_i(\\tilde{\\vx})\\big\\|\\\\\n\\le&\\big\\|\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\hat{\\vx}),z_i)\\nabla f_i(\\hat{\\vx})-\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\tilde{\\vx}),z_i)\\nabla f_i(\\hat{\\vx})\\big\\|\\\\\n&+\\big\\|\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\tilde{\\vx}),z_i)\\nabla f_i(\\hat{\\vx})-\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\tilde{\\vx}),z_i)\\nabla f_i(\\tilde{\\vx})\\big\\|\\\\\n\\le& \\beta|f_i(\\hat{\\vx})-f_i(\\tilde{\\vx})|\\cdot\\|\\nabla f_i(\\hat{\\vx})\\|+\\big|\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\tilde{\\vx}),z_i)\\big|\\cdot\\|\\nabla f_i(\\hat{\\vx})-\\nabla f_i(\\tilde{\\vx})\\|\\\\\n\\le & \\beta B_i^2\\|\\hat{\\vx}-\\tilde{\\vx}\\|+ L_i(\\beta B_i + |z_i|) \\|\\hat{\\vx}-\\tilde{\\vx}\\|.\n\\end{align*}\nHence,\n\\begin{align*}\n&\\|\\nabla_\\vx F_\\beta(\\hat{\\vx},\\vy,\\vz)-\\nabla_\\vx F_\\beta(\\tilde{\\vx},\\vy,\\vz)\\|\\\\\n\\le& \\|\\nabla g(\\hat{\\vx})-\\nabla g(\\tilde{\\vx})\\|+\\beta\\|\\vA^\\top \\vA(\\hat{\\vx}-\\tilde{\\vx})\\| + \\sum_{i=1}^m \\|\\nabla_\\vx h_i(\\hat{\\vx},z_i)-\\nabla_\\vx h_i(\\tilde{\\vx},z_i)\\|\\\\\n\\le & \\left(L_0+\\beta\\|\\vA^\\top \\vA\\|+\\sum_{i=1}^m \\big[\\beta B_i^2+ L_i(\\beta B_i + |z_i|)\\big]\\right)\\|\\hat{\\vx}-\\tilde{\\vx}\\|,\n\\end{align*}\nwhich completes the proof.\n\\end{proof}","sha256_utf8":"924fe6000bf5b3fa219c95d81d674d54acd93d2c04ec48f5b76e009590cec34c"},{"start":66601,"end":66971,"start_line":896,"end_line":898,"raw_tex":"Therefore, letting\n$\\phi(\\vx) = F_{\\beta_k}(\\vx,\\vy^k ,\\vz^k  )$ and $\\psi(\\vx) = h(\\vx)+\\iota_\\cX(\\vx),$\nwe can apply Nesterov's optimal first-order method in Algorithm \\ref{alg:apg} to find $\\vx^{k+1}$ in \\eqref{eq:ialm-x}. From Theorem \\ref{thm:rate-apg}, we have the following results. Note that if the strong convexity constant $\\mu=0$, the problem is just convex.\n","sha256_utf8":"876d71a42ea01d7e28ad33db0cc1b7d80462b0ec76a6cadfa24ef2da470cf68f"},{"start":66972,"end":68326,"start_line":900,"end_line":919,"raw_tex":"\\begin{lemma}\\label{lem:sub-iter}\nAssume that $g$ is strongly convex with modulus $\\mu\\ge0$. Given $\\vareps_k>0$, if we start from $\\vx^k $ and run Algorithm \\ref{alg:apg}, then at most $t_k$ iterations are needed to produce $\\vx^{k+1}$ such that \\eqref{eq:ialm-x} holds,\nwhere \n\\begin{equation}\\label{eq:tk-cvx}\nt_k = \\left\\{\n\\begin{array}{ll}\n\\left\\lceil\\dfrac{\\dist(\\vx^k ,\\cX_k^*)\\sqrt{2L(\\vz^k  )}}{\\sqrt{\\vareps_k}}\\right\\rceil, & \\text{ if }\\mu=0,\\\\[0.5cm]\n\\left\\lceil\\dfrac{\\log\\left(\\frac{L(\\vz^k  )+\\mu}{2\\vareps_k}[\\dist(\\vx^k ,\\cX_k^*)]^2\\right)}{\\log1/\\big(1-\\sqrt{\\frac{\\mu}{L(\\vz^k  )}}\\big)}\\right\\rceil, & \\text{ if }\\mu>0,\n\\end{array}\n\\right.\n\\end{equation}\nand $\\cX_k^*$ denotes the set of optimal solutions to $\\min_{\\vx\\in \\cX}\\cL_{\\beta_k}(\\vx,\\vy^k ,\\vz^k  )$.\n%with\n%$$\\phi(\\vx) = F_\\beta(x,\\vy^k ,\\vz^k  ),\\quad \\psi(\\vx) = h(\\vx)+\\iota_\\cX(\\vx),$$ then we have\n%\\begin{equation}\\label{eq:ialm-xk-apg}\n%\\cL_\\beta(\\vx^{k+1},\\vy^k ,\\vz^k  )-\\min_{x\\in \\cX}\\cL_\\beta(x,\\vy^k ,\\vz^k  ) \\le \\frac{2L(\\vz^k  )[\\dist(\\vx^k ,\\cX_k^*)]^2}{t_k^2},\n%\\end{equation}\n%where $\\cX_k^*$ denotes the set of optimal solutions to $\\min_{x\\in \\cX}\\cL_\\beta(x,\\vy^k ,\\vz^k  )$. Hence, to guarantee a point $\\vx^{k+1}$ satisfying \\eqref{eq:ialm-x}, it suffices to evaluate the gradients of $g$ and $f_i, i\\in [m]$ for $t_k$ times, where \n \n\\end{lemma}","sha256_utf8":"42cdc1fca3a1e95ee09a2e94303516393c3b2a8921239583b60915e61d72b062"},{"start":71396,"end":72102,"start_line":957,"end_line":969,"raw_tex":"\\begin{lemma}\\label{lem:bd-z-Lz}\nLet $\\{(\\vx^k,\\vy^k,\\vz^k)\\}_{k=0}^K$ be the sequence generated from Algorithm \\ref{alg:ialm} with $\\{\\beta_k\\}$ and $\\{\\rho_k\\}$ set according to either Setting \\ref{set-penalty} or Setting \\ref{set-ialm}. If $\\vy^0=\\vzero, \\vz^0=\\vzero$, and $\\vareps_k$'s are chosen such that \n\\begin{equation}\\label{eq:req-eps}\n\\sum_{k=0}^{K-1}\\rho_k\\vareps_k\\le \\frac{C_\\vareps}{2},\n\\end{equation}\n then\n\\begin{align}\\label{eq:bd-Lzk}\nL(\\vz^k)\\le L_*+\\beta_k H,\\,\\forall \\, 0\\le k\\le K,\n\\end{align}\nwhere \n$$ H=\\|\\vA^\\top \\vA\\|+ \\sum_{i=1}^m  B_i(B_i+L_i),\\ L_*=L_0+ \\|\\vell\\| \\left(\\|\\vy^*\\|+2\\|\\vz^*\\|+\\sqrt{C_\\vareps}\\right)$$\nand $\\vell$ is given in \\eqref{eq:vec-ell}.\n\\end{lemma}","sha256_utf8":"df9e39514c89d25bdb9295f47d7de8919010bd54fa3e5b8bdd25f75ce95f4755"},{"start":72104,"end":73490,"start_line":971,"end_line":992,"raw_tex":"\\begin{proof}\nLetting $(\\vx,\\vy,\\vz)=(\\vx^*,\\vy^*,\\vz^*)$ in \\eqref{eq:ialm-ineq2-1} and using \\eqref{eq:opt}, we have \n$$\\frac{1}{2}\\|\\vy^{k+1} -\\vy^*\\|^2 + \\frac{1}{2}\\|\\vz^{k+1} -\\vz^*\\|^2\n\\le  \\frac{1}{2}\\|\\vy^k -\\vy^*\\|^2+ \\frac{1}{2}\\|\\vz^k  -\\vz^*\\|^2+\\rho_k\\vareps_k.$$\nSumming the above inequality yields \n$$\\frac{1}{2}\\|\\vy^{k}-\\vy^*\\|^2 + \\frac{1}{2}\\|\\vz^{k}-\\vz^*\\|^2\n\\le  \\frac{1}{2}\\|\\vy^0-\\vy^*\\|^2+ \\frac{1}{2}\\|\\vz^0-\\vz^*\\|^2+\\sum_{t=0}^{k-1}\\rho_t\\vareps_t, \\, \\forall \\, 0\\le k \\le K,$$\nwhich implies\n$$\\|\\vz^k  \\| \\le \\|\\vz^*\\|+\\sqrt{\\|\\vy^0-\\vy^*\\|^2+\\|\\vz^0-\\vz^*\\|^2+2\\sum_{t=0}^{k-1}\\rho_t\\vareps_t}.$$\nSince $\\|\\vu\\|\\le \\|\\vu\\|_1$ for any vector $\\vu$, we have from the above inequality that\n\\begin{equation}\\label{eq:bd-zk0}\n\\|\\vz^k  \\| \\le  \\|\\vz^*\\|+\\|\\vy^0-\\vy^*\\|+\\|\\vz^0-\\vz^*\\|+\\sqrt{2\\sum_{t=0}^{K-1}\\rho_t\\vareps_t},\\, \\forall \\, 0\\le k \\le K.\n\\end{equation}\n\nHence, if $\\vy^0=\\vzero$ and $\\vz^0=\\vzero$, and \\eqref{eq:req-eps} holds, it follows from the above inequality that\n\\begin{equation}\\label{eq:bd-zk}\n\\|\\vz^k  \\| \\le  \\|\\vy^*\\|+2\\|\\vz^*\\|+\\sqrt{C_\\vareps},\\, \\forall \\, 0\\le k \\le K,\n\\end{equation}\nBy the Cauchy-Schwartz inequality, we have from \\eqref{eq:def-Lz} that for any $0\\le k\\le K$,\n$$L(\\vz^k  ) \\le L_0+\\beta_k H + \\|\\vz^k  \\|\\cdot\\|\\vell\\|$$\nwhich together with \\eqref{eq:bd-zk} gives the result in \\eqref{eq:bd-Lzk}.\n\\end{proof}","sha256_utf8":"606f378c8b100e83f9eb7cf749863dba0f87d555bb2d1028eccda9bd8c580cb6"},{"start":81288,"end":83635,"start_line":1093,"end_line":1114,"raw_tex":"\\begin{proof}\nThe results in \\eqref{eq:error-barx-K} directly follows from Theorem \\ref{thm:ialm-rate} and the settings of $\\{\\beta_k\\}$, $\\{\\rho_k\\}$, and $\\{\\vareps_k\\}$. For the total number of gradient evaluations, we use the inequalities in \\eqref{eq:tk-cvx}. First, for the case of $\\mu=0$, from the first inequality of \\eqref{eq:tk-cvx} and the parameter setting, it follows that the total number of gradient evaluations\n\\begin{equation}\\label{eq:total-grad-cvx-adp1}\nT_K\\le \\sum_{k=0}^{K-1}\\frac{\\dist(\\vx^k ,\\cX_k^*)\\sqrt{2(L_*+\\frac{C_\\beta H}{K\\vareps})}}{\\sqrt{\\vareps/2}\\sqrt{C_\\vareps/C_\\beta}}+K.\n\\end{equation}\nSince $\\sqrt{a+b}\\le \\sqrt{a}+\\sqrt{b}$ for any two nonnegative numbers $a,b$, we have from the above inequality and by noting $\\dist(\\vx^k ,\\cX_k^*)\\le D$ that\n\\begin{align*}\nT_K \\le 2D\\sqrt{\\frac{C_\\beta}{C_\\vareps}}\\sum_{k=0}^{K-1}\\frac{\\sqrt{L_*}+\\sqrt{\\frac{C_\\beta H}{K\\vareps}}}{\\sqrt{\\vareps}}+K=2DK\\sqrt{\\frac{C_\\beta}{C_\\vareps}}\\left(\\sqrt{\\frac{L_*}{\\vareps}}+\\frac{1}{\\vareps}\\sqrt{\\frac{C_\\beta H}{K}}\\right)+K,\n\\end{align*}\nwhich gives \\eqref{eq:T-penaly-cvx}.\n\nFor the case of $\\mu>0$, we first note that for any $0<a\\le1$, it holds $\\log(1+a)\\ge a-\\frac{a^2}{2}\\ge\\frac{a}{2}$. Hence, if $\\mu\\le \\frac{L_0}{4}$, we have $\\mu\\le \\frac{L(\\vz^k)}{4}$ and $\\frac{\\sqrt{\\mu/L(\\vz^k)}}{1-\\sqrt{\\mu/L(\\vz^k)}}\\le 1$. Therefore,\n$$\\log\\frac{1}{1-\\sqrt{\\mu/L(\\vz^k)}}=\\log\\left(1+\\frac{\\sqrt{\\mu/L(\\vz^k)}}{1-\\sqrt{\\mu/L(\\vz^k)}}\\right)\\ge \\frac{1}{2}\\frac{\\sqrt{\\mu/L(\\vz^k)}}{1-\\sqrt{\\mu/L(\\vz^k)}},$$ and thus\n\\begin{equation}\\label{eq:bd-denom}\n\\frac{1}{\\log\\frac{1}{1-\\sqrt{\\mu/L(\\vz^k)}}}\\le 2\\sqrt{\\frac{L(\\vz^k)}{\\mu}}\\left(1-\\sqrt{\\mu/L(\\vz^k)}\\right) \\le 2\\sqrt{\\frac{L(\\vz^k)}{\\mu}}.\n\\end{equation} Using the above inequality and the second inequality of \\eqref{eq:tk-cvx}, we have that the total number of gradient evaluations\n\\begin{equation}\nT_K\\le  \\sum_{k=0}^{K-1}2\\sqrt{\\frac{L_*+\\frac{C_\\beta H}{K\\vareps}}{\\mu}}\\log\\left(\\frac{L_*+\\frac{C_\\beta H}{K\\vareps}+\\mu}{\\vareps C_\\vareps/C_\\beta}[\\dist(\\vx^k ,\\cX_k^*)]^2\\right)+K.\\label{eq:total-grad-scvx-adp1}\n\\end{equation}\nSince $\\sqrt{L_*+\\frac{C_\\beta H}{K\\vareps}}\\le \\sqrt{L_*}+\\sqrt{\\frac{C_\\beta H}{K\\vareps}}$ and $\\dist(\\vx^k ,\\cX_k^*)\\le D$, the above inequality implies \\eqref{eq:T-penaly-strcvx}.\nThis completes the proof.\n\\end{proof}","sha256_utf8":"d65aa39f8f7e24d83c1a2e126a75aac2d730def226cda6f807e52e8df0ef73ee"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"恒定罚参数下的梯度复杂度","source_units":[{"start":79718,"end":81287,"start_line":1075,"end_line":1092,"raw_tex":"\\begin{theorem}[Iteration complexity with constant penalty and constant error]\\label{thm:it-comp-const}\nFor any given $\\vareps>0$, let $K$ be a positive integer number and $C_\\beta, C_\\vareps$ two positive real numbers. Set $\\{\\beta_k\\}$ and $\\{\\rho_k\\}$ according to Setting \\ref{set-penalty} and $\\{\\vareps_k\\}$ by \\eqref{eq:const-eps}. Let $\\bar{\\vx}^K$ be given in \\eqref{eq:barx-K}. Then \n\\begin{subequations}\\label{eq:error-barx-K}\n\\begin{align}\n\\big|f_0(\\bar{\\vx}^{K})-f_0(\\vx^*)\\big| \\le \\frac{\\vareps\\big(2\\|\\vy^*\\|^2+ 2\\|\\vz^*\\|^2\\big)}{C_\\beta}+\\frac{\\vareps}{2}\\frac{C_\\vareps}{C_\\beta},\\label{eq:error-barx-K-obj}\\\\\n\\|\\vA\\bar{\\vx}^{K}-\\vb\\| + \\big\\| [\\vf(\\bar{\\vx}^{K})]_+ \\big\\|\\le  \\frac{\\vareps\\big[(1+\\|\\vy^*\\|)^2+ (1+\\|\\vz^*\\|)^2\\big]}{2C_\\beta}+\\frac{\\vareps}{2}\\frac{C_\\vareps}{C_\\beta}.\\label{eq:error-barx-K-res}\n\\end{align}\n\\end{subequations}\nAssume $\\mu\\le \\frac{L_0}{4}$. Then Algorithm \\ref{alg:ialm} can produce $\\bar{\\vx}^K$ by evaluating gradients of $g, f_i, \\,i\\in [m]$ in at most $T_K$ times, where\n\\begin{equation}\\label{eq:T-penaly-cvx}\nT_K\\le \\left\\lceil2DK\\sqrt{\\frac{C_\\beta}{C_\\vareps}}\\left(\\sqrt{\\frac{L_*}{\\vareps}}+\\frac{1}{\\vareps}\\sqrt{\\frac{C_\\beta H}{K}}\\right)+K\\right\\rceil, \\text{ if }\\mu=0,\n\\end{equation}\nand\n\\begin{equation}\\label{eq:T-penaly-strcvx}\nT_K\\le \\left\\lceil2K\\left(\\sqrt{\\frac{L_*}{\\mu}}+\\sqrt{\\frac{C_\\beta H}{\\mu K\\vareps}}\\right)\\log\\left(\\frac{D^2 C_\\beta}{C_\\vareps}\\big(\\frac{L_*+\\mu}{\\vareps}+\\frac{C_\\beta H}{K\\vareps^2}\\big)\\right) + K\\right\\rceil, \\text{ if }\\mu>0.\n\\end{equation}\n\n\\end{theorem}","sha256_utf8":"3ac9e348fd89daba6c9c269b2061b0a146d784ecd9fd0893215e1a16f8b5b2a7"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"恒定参数复杂度的完整证明","source_units":[{"start":81288,"end":83635,"start_line":1093,"end_line":1114,"raw_tex":"\\begin{proof}\nThe results in \\eqref{eq:error-barx-K} directly follows from Theorem \\ref{thm:ialm-rate} and the settings of $\\{\\beta_k\\}$, $\\{\\rho_k\\}$, and $\\{\\vareps_k\\}$. For the total number of gradient evaluations, we use the inequalities in \\eqref{eq:tk-cvx}. First, for the case of $\\mu=0$, from the first inequality of \\eqref{eq:tk-cvx} and the parameter setting, it follows that the total number of gradient evaluations\n\\begin{equation}\\label{eq:total-grad-cvx-adp1}\nT_K\\le \\sum_{k=0}^{K-1}\\frac{\\dist(\\vx^k ,\\cX_k^*)\\sqrt{2(L_*+\\frac{C_\\beta H}{K\\vareps})}}{\\sqrt{\\vareps/2}\\sqrt{C_\\vareps/C_\\beta}}+K.\n\\end{equation}\nSince $\\sqrt{a+b}\\le \\sqrt{a}+\\sqrt{b}$ for any two nonnegative numbers $a,b$, we have from the above inequality and by noting $\\dist(\\vx^k ,\\cX_k^*)\\le D$ that\n\\begin{align*}\nT_K \\le 2D\\sqrt{\\frac{C_\\beta}{C_\\vareps}}\\sum_{k=0}^{K-1}\\frac{\\sqrt{L_*}+\\sqrt{\\frac{C_\\beta H}{K\\vareps}}}{\\sqrt{\\vareps}}+K=2DK\\sqrt{\\frac{C_\\beta}{C_\\vareps}}\\left(\\sqrt{\\frac{L_*}{\\vareps}}+\\frac{1}{\\vareps}\\sqrt{\\frac{C_\\beta H}{K}}\\right)+K,\n\\end{align*}\nwhich gives \\eqref{eq:T-penaly-cvx}.\n\nFor the case of $\\mu>0$, we first note that for any $0<a\\le1$, it holds $\\log(1+a)\\ge a-\\frac{a^2}{2}\\ge\\frac{a}{2}$. Hence, if $\\mu\\le \\frac{L_0}{4}$, we have $\\mu\\le \\frac{L(\\vz^k)}{4}$ and $\\frac{\\sqrt{\\mu/L(\\vz^k)}}{1-\\sqrt{\\mu/L(\\vz^k)}}\\le 1$. Therefore,\n$$\\log\\frac{1}{1-\\sqrt{\\mu/L(\\vz^k)}}=\\log\\left(1+\\frac{\\sqrt{\\mu/L(\\vz^k)}}{1-\\sqrt{\\mu/L(\\vz^k)}}\\right)\\ge \\frac{1}{2}\\frac{\\sqrt{\\mu/L(\\vz^k)}}{1-\\sqrt{\\mu/L(\\vz^k)}},$$ and thus\n\\begin{equation}\\label{eq:bd-denom}\n\\frac{1}{\\log\\frac{1}{1-\\sqrt{\\mu/L(\\vz^k)}}}\\le 2\\sqrt{\\frac{L(\\vz^k)}{\\mu}}\\left(1-\\sqrt{\\mu/L(\\vz^k)}\\right) \\le 2\\sqrt{\\frac{L(\\vz^k)}{\\mu}}.\n\\end{equation} Using the above inequality and the second inequality of \\eqref{eq:tk-cvx}, we have that the total number of gradient evaluations\n\\begin{equation}\nT_K\\le  \\sum_{k=0}^{K-1}2\\sqrt{\\frac{L_*+\\frac{C_\\beta H}{K\\vareps}}{\\mu}}\\log\\left(\\frac{L_*+\\frac{C_\\beta H}{K\\vareps}+\\mu}{\\vareps C_\\vareps/C_\\beta}[\\dist(\\vx^k ,\\cX_k^*)]^2\\right)+K.\\label{eq:total-grad-scvx-adp1}\n\\end{equation}\nSince $\\sqrt{L_*+\\frac{C_\\beta H}{K\\vareps}}\\le \\sqrt{L_*}+\\sqrt{\\frac{C_\\beta H}{K\\vareps}}$ and $\\dist(\\vx^k ,\\cX_k^*)\\le D$, the above inequality implies \\eqref{eq:T-penaly-strcvx}.\nThis completes the proof.\n\\end{proof}","sha256_utf8":"d65aa39f8f7e24d83c1a2e126a75aac2d730def226cda6f807e52e8df0ef73ee"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"代入参数并汇总凸情形工作量","source_units":[{"start":81288,"end":81915,"start_line":1093,"end_line":1097,"raw_tex":"\\begin{proof}\nThe results in \\eqref{eq:error-barx-K} directly follows from Theorem \\ref{thm:ialm-rate} and the settings of $\\{\\beta_k\\}$, $\\{\\rho_k\\}$, and $\\{\\vareps_k\\}$. For the total number of gradient evaluations, we use the inequalities in \\eqref{eq:tk-cvx}. First, for the case of $\\mu=0$, from the first inequality of \\eqref{eq:tk-cvx} and the parameter setting, it follows that the total number of gradient evaluations\n\\begin{equation}\\label{eq:total-grad-cvx-adp1}\nT_K\\le \\sum_{k=0}^{K-1}\\frac{\\dist(\\vx^k ,\\cX_k^*)\\sqrt{2(L_*+\\frac{C_\\beta H}{K\\vareps})}}{\\sqrt{\\vareps/2}\\sqrt{C_\\vareps/C_\\beta}}+K.\n\\end{equation}\n","sha256_utf8":"ed4966e54c386cc105ea8eb90bd87a2df958956d7f1164b56422155a12c06b03"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"利用直径得到凸情形复杂度","source_units":[{"start":81915,"end":82391,"start_line":1098,"end_line":1102,"raw_tex":"Since $\\sqrt{a+b}\\le \\sqrt{a}+\\sqrt{b}$ for any two nonnegative numbers $a,b$, we have from the above inequality and by noting $\\dist(\\vx^k ,\\cX_k^*)\\le D$ that\n\\begin{align*}\nT_K \\le 2D\\sqrt{\\frac{C_\\beta}{C_\\vareps}}\\sum_{k=0}^{K-1}\\frac{\\sqrt{L_*}+\\sqrt{\\frac{C_\\beta H}{K\\vareps}}}{\\sqrt{\\vareps}}+K=2DK\\sqrt{\\frac{C_\\beta}{C_\\vareps}}\\left(\\sqrt{\\frac{L_*}{\\vareps}}+\\frac{1}{\\vareps}\\sqrt{\\frac{C_\\beta H}{K}}\\right)+K,\n\\end{align*}\nwhich gives \\eqref{eq:T-penaly-cvx}.\n","sha256_utf8":"a08c8909633debb0d34abdf0dc1dd7f5232748c214b4442f64c5b9efaff44a2e"}]},{"id":"S005","semantic_type":"proof_step","kind":"proof_step","title":"估计强凸收敛因子的对数","source_units":[{"start":82392,"end":83162,"start_line":1104,"end_line":1108,"raw_tex":"For the case of $\\mu>0$, we first note that for any $0<a\\le1$, it holds $\\log(1+a)\\ge a-\\frac{a^2}{2}\\ge\\frac{a}{2}$. Hence, if $\\mu\\le \\frac{L_0}{4}$, we have $\\mu\\le \\frac{L(\\vz^k)}{4}$ and $\\frac{\\sqrt{\\mu/L(\\vz^k)}}{1-\\sqrt{\\mu/L(\\vz^k)}}\\le 1$. Therefore,\n$$\\log\\frac{1}{1-\\sqrt{\\mu/L(\\vz^k)}}=\\log\\left(1+\\frac{\\sqrt{\\mu/L(\\vz^k)}}{1-\\sqrt{\\mu/L(\\vz^k)}}\\right)\\ge \\frac{1}{2}\\frac{\\sqrt{\\mu/L(\\vz^k)}}{1-\\sqrt{\\mu/L(\\vz^k)}},$$ and thus\n\\begin{equation}\\label{eq:bd-denom}\n\\frac{1}{\\log\\frac{1}{1-\\sqrt{\\mu/L(\\vz^k)}}}\\le 2\\sqrt{\\frac{L(\\vz^k)}{\\mu}}\\left(1-\\sqrt{\\mu/L(\\vz^k)}\\right) \\le 2\\sqrt{\\frac{L(\\vz^k)}{\\mu}}.\n\\end{equation} Using the above inequality and the second inequality of \\eqref{eq:tk-cvx}, we have that the total number of gradient evaluations\n","sha256_utf8":"408802f8f368c06d8312eceea262550404c07a0a3bd7a7288d4b15d50dbeb7f8"}]},{"id":"S006","semantic_type":"proof_step","kind":"proof_step","title":"汇总并简化强凸情形工作量","source_units":[{"start":83162,"end":83635,"start_line":1109,"end_line":1114,"raw_tex":"\\begin{equation}\nT_K\\le  \\sum_{k=0}^{K-1}2\\sqrt{\\frac{L_*+\\frac{C_\\beta H}{K\\vareps}}{\\mu}}\\log\\left(\\frac{L_*+\\frac{C_\\beta H}{K\\vareps}+\\mu}{\\vareps C_\\vareps/C_\\beta}[\\dist(\\vx^k ,\\cX_k^*)]^2\\right)+K.\\label{eq:total-grad-scvx-adp1}\n\\end{equation}\nSince $\\sqrt{L_*+\\frac{C_\\beta H}{K\\vareps}}\\le \\sqrt{L_*}+\\sqrt{\\frac{C_\\beta H}{K\\vareps}}$ and $\\dist(\\vx^k ,\\cX_k^*)\\le D$, the above inequality implies \\eqref{eq:T-penaly-strcvx}.\nThis completes the proof.\n\\end{proof}","sha256_utf8":"3dffdae587823845235d6ec048da5c0c74a0417a20e2cbd7c811e16c6a291efe"}]},{"id":"S007","semantic_type":"problem_setting","kind":"problem_setting","title":"等式与不等式约束凸规划","source_units":[{"start":11036,"end":11623,"start_line":309,"end_line":313,"raw_tex":"In this paper, we consider the constrained convex programming\n\\begin{equation}\\label{eq:ccp}\n\\Min_{\\vx\\in\\cX} f_0(\\vx), \\st \\vA\\vx=\\vb, f_i(\\vx)\\le 0, i=1,\\ldots, m,\n\\end{equation}\nwhere $\\cX\\subseteq\\RR^n$ is a closed convex set, $\\vA$ and $\\vb$ are respectively given matrix and vector, and $f_i$ is a convex function for every $i=0,1,\\ldots,m$. Any convex optimization problem can be written in the standard form of \\eqref{eq:ccp}. It appears in many areas including statistics, machine learning, data mining, engineering, signal processing, finance, operations research, and so on. \n","sha256_utf8":"ceb6a6b683ed74627406725abc94cbfa024064be7ec1cf097791960236bcc30e"}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","title":"复合结构、光滑性与有界直径","source_units":[{"start":13895,"end":15111,"start_line":322,"end_line":333,"raw_tex":"We will assume composite convex structure on \\eqref{eq:ccp}. More specifically, we assume \\begin{equation}\\label{eq:f0}\nf_0(\\vx)=g(\\vx)+h(\\vx),\n\\end{equation} \nwhere $g$ is a Lipschitz differentiable convex function, and $h$ is a simple\\footnote{By ``simple'', we mean the proximal mapping of $h$ is easy to evaluate, i.e., it is easy to find a solution to $\\min_{\\vx\\in\\cX} h(\\vx) + \\frac{1}{2\\gamma}\\|\\vx-\\hat{\\vx}\\|^2$ for any $\\hat{\\vx}$ and $\\gamma>0$.} (possibly nondifferentiable) convex function. Also, $f_i$ is convex and Lipschitz differentiable for every $i\\in [m]$, namely, there are constants $L_0,L_1,\\ldots, L_m$ such that\n\\begin{subequations}\\label{eq:lip-gf}\n\\begin{align}\n\\|\\nabla g(\\hat{\\vx})-\\nabla g(\\tilde{\\vx})\\|\\le L_0\\|\\hat{\\vx}-\\tilde{\\vx}\\|,\\,&\\forall\\, \\hat{\\vx}, \\tilde{\\vx} \\in \\dom(h)\\cap \\cX,\\label{eq:lip-g}\\\\\n\\|\\nabla f_i(\\hat{\\vx})-\\nabla f_i(\\tilde{\\vx})\\|\\le L_i\\|\\hat{\\vx}-\\tilde{\\vx}\\|,\\,& \\forall\\, \\hat{\\vx}, \\tilde{\\vx} \\in \\dom(h) \\cap \\cX, \\forall i\\in [m].\\label{eq:lip-f}\n\\end{align} \n\\end{subequations}\nIn addition, we assume the boundedness of $\\dom(h)\\cap\\cX$ and denote its diameter as \n$$D=\\Max_{\\hat{\\vx},\\tilde{\\vx}\\in \\dom(h)\\cap\\cX}\\|\\hat{\\vx}-\\tilde{\\vx}\\|.$$\n","sha256_utf8":"8caa815bb9ee95c2124a61bc29114ad9be92af5281e8f77ac7b0a79f99434908"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"增广拉格朗日函数","source_units":[{"start":16794,"end":17305,"start_line":353,"end_line":362,"raw_tex":"\\begin{equation}\\label{eq:def-psi}\n\\psi_\\beta(u,v) = \\left\\{\\begin{array}{ll}uv+\\frac{\\beta}{2}u^2,\\,&\\text{ if }\\beta u +v \\ge0,\\\\[0.2cm]\n-\\frac{v^2}{2\\beta},\\,&\\text{ if }\\beta u +v < 0.\\end{array}\\right.\n\\end{equation}\nLet\n$$\\Psi_\\beta(\\vx,\\vz)=\\sum_{i=1}^m\\psi_\\beta(f_i(\\vx),z_i),$$\nand we obtain the classical AL function of \\eqref{eq:ccp}:\n\\begin{equation}\\label{eq:aug-fun}\n\\cL_\\beta(\\vx,\\vy,\\vz) = f_0(\\vx) + \\vy^\\top(\\vA\\vx-\\vb) + \\frac{\\beta}{2}\\|\\vA\\vx-\\vb\\|^2 + \\Psi_\\beta(\\vx,\\vz).\n\\end{equation}\n","sha256_utf8":"32195e9cca3dc8c920c527179c7d09b957555af8bb626a110f4b5c8c2b3add3c"}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","title":"非精确增广拉格朗日算法","source_units":[{"start":17973,"end":18649,"start_line":366,"end_line":382,"raw_tex":"\\begin{algorithm}[h]\n\\caption{Inexact augmented Lagrangian method for \\eqref{eq:ccp}}\\label{alg:ialm}\n\\DontPrintSemicolon\n\\textbf{Initialization:} choose $\\vx^0, \\vy^0, \\vz^0$ and $\\{\\beta_k,\\rho_k\\}$\\;\n\\For{$k=0,1,\\ldots$}{\nFind $\\vx^{k+1}\\in\\cX$ such that \n\\begin{equation}\\label{eq:ialm-x}\n\\cL_{\\beta_k}(\\vx^{k+1}, \\vy^k , \\vz^k  ) \\le \\min_{\\vx\\in\\cX} \\cL_{\\beta_k}(\\vx, \\vy^k , \\vz^k  ) + \\vareps_k.\n\\end{equation}\n\nUpdate $\\vy$ and $\\vz$ by\n\\begin{align}\n\\vy^{k+1} =&~\\vy^k  + \\rho_k (\\vA\\vx^{k+1}-\\vb),\\label{eq:alm-y}\\\\\nz^{k+1} _i=&~z_i^k+\\rho_k \\cdot\\max\\left(-\\frac{z_i^k}{\\beta_k}, f_i(\\vx^{k+1})\\right), i=1,\\ldots, m.\\label{eq:alm-z}\n\\end{align}\n}\n\\end{algorithm}","sha256_utf8":"389121368824e9bdf1c68af9c7d17bc2e0d8ffa4bab53f169a43cde175065403"}]},{"id":"S011","semantic_type":"notation","kind":"notation","title":"范数、正部与凸分析记号","source_units":[{"start":27237,"end":29258,"start_line":424,"end_line":433,"raw_tex":"We use italic letters $a, c, B, L,\\ldots,$ for scalars, bold lower-case letters $\\vx,\\vy,\\vz,\\ldots$ for vectors, and bold upper-case letters $\\vA,\\vB,\\ldots$ for matrices. $z_i$ denotes the $i$-th entry of a vector $\\vz$. We use $\\vzero$ to denote a vector of all zeros, and its size is clear from the context. $[m]$ denotes the set $\\{1,2,\\ldots,m\\}$ for any positive integer $m$. Given a real number $a$, we let $[a]_+=\\max(0,a)$ and $\\lceil a\\rceil$ be the smallest integer that is no less than $a$. For a vector $\\va$, $[\\va]_+$ takes the positive part of $\\va$ in a component-wise manner. $\\|\\va\\|$ denotes the Euclidean norm of a vector $\\va$ and $\\|\\vA\\|$ the spectral norm of a matrix $\\vA$. %By $p(k)\\lesssim q(k)$, we mean $\\limsup_k\\frac{p(k)}{q(k)} \\le 1$, i.e., $p(k)$ is dominated by $q(k)$. Given a symmetric positive definite (SPD) matrix $\\vP$, we denote $\\|\\cdot\\|_\\vP$ as $\\vP$-weighted norm, i.e., $\\|\\vw\\|_\\vP=\\sqrt{\\vw^\\top \\vP \\vw}$ . If $\\vP=\\vI$, we simply write it as $\\|\\cdot\\|$. \n\nWe denote $\\vell$ as the vector consisting of $L_i, i\\in [m]$, where $L_i$ is the Lipschitz constant of $\\nabla f_i$ in \\eqref{eq:lip-f}. Also we let $\\vf$ be the vector function with $f_i$ as the $i$-th component scalar function. That is \n\\begin{equation}\\label{eq:vec-ell}\n\\vell=[L_1,\\ldots,L_m], \\quad \\vf(\\vx)=[f_1(\\vx),\\ldots, f_m(\\vx)].\n\\end{equation}\nGiven a convex function $f$, $\\tilde{\\nabla}f(\\vx)$ represents one subgradient of $f$ at $\\vx$, namely,\n$$f(\\hat{\\vx})\\ge f(\\vx) + \\langle \\tilde{\\nabla} f(\\vx), \\hat{\\vx}-\\vx\\rangle,\\,\\forall\\, \\hat{\\vx},$$ and $\\partial f(\\vx)$ denotes its subdifferential, i.e., the set of all subgradients. When $f$ is differentiable, we simply write its subgradient as $\\nabla f(\\vx)$. For a convex set $\\cX$, we use $\\iota_\\cX$ as its indicator function, i.e., \n$$\\iota_\\cX(\\vx)=\\left\\{\\begin{array}{ll}0,&\\text{ if }\\vx\\in\\cX,\\\\+\\infty,&\\text{ if }\\vx\\not\\in\\cX,\\end{array}\\right.$$\nand $\\cN_\\cX(\\vx)=\\partial \\iota_\\cX(\\vx)$ as its normal cone at $\\vx\\in \\cX$. \n","sha256_utf8":"e570ae0c5f7c09e12933defd4e4b19ad84b25179d921a99d9bfe93d70a6a56ee"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"目标与可行性的精度标准","source_units":[{"start":29353,"end":29690,"start_line":436,"end_line":439,"raw_tex":"\\begin{definition}[$\\vareps$-optimal solution]\\label{def:eps-opt}\nLet $f_0^*$ be the optimal value of \\eqref{eq:ccp}. Given $\\vareps\\ge 0$, a point $\\vx\\in\\cX$ is called an $\\vareps$-optimal solution to \\eqref{eq:ccp} if \n$$|f_0(\\vx)-f_0^*| \\le \\vareps,\\text{ and }\\|\\vA\\vx-\\vb\\|+\\big\\| [\\vf(\\vx)]_+\\big\\| \\le \\vareps.$$\n\\end{definition}","sha256_utf8":"89f402879dc918cd64acab133dd2e0d441a4a2db394cd5eb34146f9b6fef7e1e"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"最优性条件","source_units":[{"start":31403,"end":31829,"start_line":452,"end_line":459,"raw_tex":"A point $(\\vx,\\vy,\\vz)$ satisfies the Karush-Kuhn-Tucker (KKT) conditions for \\eqref{eq:ccp} if\n\\begin{subequations}\\label{eq:kkt}\n\\begin{align}\n0\\in \\partial f_0(\\vx) +\\cN_\\cX(\\vx) + \\vA^\\top \\vy + \\sum_{i=1}^m z_i \\nabla f_i(\\vx), \\label{eq:kkt-1st}&\\\\\n\\vA\\vx = \\vb, \\quad \\vx\\in \\cX, \\label{eq:kkt-res}&\\\\\nz_i\\ge0, \\quad f_i(\\vx)\\le 0,\\quad z_if_i(\\vx)=0, \\forall i\\in [m]. \\label{eq:kkt-cp}\n\\end{align}\n\\end{subequations}\n","sha256_utf8":"a64271d98cc0cdccd223231a7a6be9890aec5c5355be009f0d07eddd5c62a58d"}]},{"id":"S014","semantic_type":"problem_setting","kind":"problem_setting","title":"加速近端法的复合子问题","source_units":[{"start":35131,"end":36097,"start_line":509,"end_line":513,"raw_tex":"In this subsection, we review Nesterov's optimal first-order method for composite convex programs. The method will be used to approximately solve $\\vx$-subproblems in Algorithm \\ref{alg:ialm}. It aims at finding a solution of the following problem\n\\begin{equation}\\label{eq:comp-prob}\n\\Min_\\vx \\phi(\\vx) + \\psi(\\vx),\n\\end{equation}\nwhere $\\phi$ is a Lipschitz differentiable and strongly convex function with gradient Lipschitz constant $L_\\phi$ and strong convexity modulus $\\mu\\ge0$, and $\\psi$ is a simple (possibly nondifferentiable) closed convex function. Algorithm \\ref{alg:apg} summarizes the method. Here, for simplicity, we assume $L_\\phi$ and $\\mu$ are known. The method does not require the value of $L_\\phi$ but can estimate it by backtracking. In addition, it only requires a lower estimate of $\\mu$; see \\cite{nesterov2013gradient} for example.  %For our purpose, we will have $\\mu=1$; see the problem \\eqref{eq:ipalm-x} in Algorithm \\ref{alg:ipalm}.\n","sha256_utf8":"bdbc6587a3d9b8e524817b593528a7b560fc3af7235edd427c9d83d3d48baa43"}]},{"id":"S015","semantic_type":"algorithm","kind":"algorithm","title":"内层加速近端算法","source_units":[{"start":36098,"end":36669,"start_line":515,"end_line":524,"raw_tex":"\\begin{algorithm}\\caption{Nesterov's optimal first-order method for \\eqref{eq:comp-prob}}\\label{alg:apg}\n\\textbf{Initialization:} choose $\\hat{\\vx}^0=\\vx^0$, $\\alpha_0\\in (0,1]$, and let $q=\\frac{\\mu}{L_\\phi}$\\; \n\\For{$k=0,1,\\ldots, $}{\nLet \n$$\\vx^{k+1}=\\argmin_{\\vx} \\langle \\nabla \\phi(\\hat{\\vx}^k), \\vx\\rangle + \\frac{L_\\phi}{2}\\|\\vx-\\hat{\\vx}^k\\|^2+\\psi(\\vx).$$\n\nSet $$\\alpha_{k+1}=\\frac{q-\\alpha_k^2+\\sqrt{(q-\\alpha_k^2)^2+4\\alpha_k^2}}{2},$$ and\n$$\\hat{\\vx}^{k+1}=\\vx^{k+1}+\\frac{\\alpha_k(1-\\alpha_k)}{\\alpha_k^2+\\alpha_{k+1}}(\\vx^{k+1}-\\vx^k ).$$\n}\n\\end{algorithm}","sha256_utf8":"e44d81b9313b85fcfa9fff2b281ca1dc5bd5c919197012c9525c27f439d45a50"}]},{"id":"S016","semantic_type":"assumption","kind":"assumption","title":"最优性条件解的存在性","source_units":[{"start":38377,"end":38522,"start_line":547,"end_line":549,"raw_tex":"\\begin{assumption}\\label{assump-kkt}\nThere exists a point $(\\vx^*,\\vy^*,\\vz^*)$ satisfying the KKT conditions in \\eqref{eq:kkt}.\n\\end{assumption}","sha256_utf8":"218387edebc5c4fd6ae3e7bca294e5e8da500f5441001cc94f6a71b4834f9a30"}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","title":"非精确子问题的可求解性","source_units":[{"start":38659,"end":38782,"start_line":555,"end_line":557,"raw_tex":"\\begin{assumption}\\label{assump-wd-ialm}\nFor every $k$, there is $\\vx^{k+1}$ satisfying \\eqref{eq:ialm-x}.\n\\end{assumption}","sha256_utf8":"975b0fad2a92a806a17898aeea355b5c0de8592325dec6bb308579778866e6f1"}]},{"id":"S018","semantic_type":"definition","kind":"definition","title":"增广函数的光滑部分","source_units":[{"start":64055,"end":64312,"start_line":866,"end_line":868,"raw_tex":"Let the smooth part of $\\cL_\\beta$ be denoted as\n$$F_\\beta(\\vx,\\vy,\\vz) = \\cL_\\beta(\\vx,\\vy,\\vz) - h(\\vx).$$\nBased on \\eqref{eq:lip}, we are able to show Lipschitz continuity of $\\nabla_\\vx F_\\beta(\\vx,\\vy,\\vz)$ with respect to $\\vx$ for every $(\\vy,\\vz)$.\n","sha256_utf8":"35b030a4b143e3c4c7c0826ca00b5021407b6c1000d9836baff59b0625ca97ba"}]},{"id":"S019","semantic_type":"algorithm","kind":"algorithm","title":"恒定罚参数设置","source_units":[{"start":68944,"end":69399,"start_line":924,"end_line":931,"raw_tex":"Given $\\vareps>0$, we set $\\{\\beta_k\\}$ and $\\{\\rho_k\\}$ according to one of the follows:\n\n\\begin{setting}[constant penalty]\\label{set-penalty} Let $K$ be a positive integer number and $C_\\beta$ a positive real number. Set $$\\rho_k=\\beta_k=\\frac{C_\\beta}{K\\vareps},\\,\\forall \\, 0\\le k < K.$$ \n%where \n%\\begin{equation}\\label{eq:set-penalty}\n%C_*=\\max\\left(4\\|\\vy^*\\|^2+4\\|\\vz^*\\|^2, (1+\\|\\vy^*\\|)^2+\\|\\vone+\\vz^*\\|^2\\right).\n%\\end{equation}\n\\end{setting}\n","sha256_utf8":"84ea093f293bb060105d870bcd02aed317dc6560a465d2f48b0172f672afe75d"}]},{"id":"S020","semantic_type":"algorithm","kind":"algorithm","title":"辅助引理允许的几何参数设置","source_units":[{"start":69400,"end":69914,"start_line":933,"end_line":943,"raw_tex":"\\begin{setting}[geometrically increasing penalty]\\label{set-ialm} Let $K$ be a positive integer number, $C_\\beta$ a positive real number, and $\\sigma>1$. Set \n\\begin{equation}\\label{eq:def-beta-g}\n\\beta_g = \\frac{C_\\beta}{\\vareps}\\frac{\\sigma-1}{\\sigma^K-1},\n\\end{equation}\nand\n$$ \\rho_k=\\beta_k=\\beta_g\\sigma^k, \\,\\forall\\, 0\\le k < K.$$ \n%\\begin{equation}\\label{eq:set-ialm}\n%\\beta_g = \\frac{C^*}{\\vareps}\\frac{\\sigma-1}{\\sigma^K-1}\n%\\end{equation}\n%where $C_*$ is given in \\eqref{eq:set-penalty}.\n\\end{setting}\n","sha256_utf8":"8173a45a2cf1d974eba6597a1cf3ee7fa8041e5d02051b2afcb37d4e125f3904"}]},{"id":"S021","semantic_type":"notation","kind":"notation","title":"总梯度评估次数","source_units":[{"start":73492,"end":73945,"start_line":994,"end_line":995,"raw_tex":"\\textbf{``Optimal'' subproblem accuracy parameters.} If $t_k$ gradient evaluations are required to produce $\\vx^{k+1}$, then the total number of gradient evaluations is \n$T_K=\\sum_{k=0}^{K-1}t_k$ to generate $\\{\\vx^k\\}_{k=1}^K$. Given $\\vareps>0$, and $\\{\\beta_k\\}$,$\\{\\rho_k\\}$ set according to either Setting \\ref{set-penalty} or Setting \\ref{set-ialm}, we can choose $\\{\\vareps_k\\}$ to minimize $T_K$ subject to the condition in \\eqref{eq:req-eps}. \n","sha256_utf8":"8904b14b31a3b6d857ba94527fed6a04a922b327ea1792a4e256f47dca567577"}]},{"id":"S022","semantic_type":"algorithm","kind":"algorithm","title":"恒定子问题误差设置","source_units":[{"start":75677,"end":75805,"start_line":1016,"end_line":1018,"raw_tex":"\\begin{equation}\\label{eq:const-eps}\n\\vareps_k=\\frac{\\vareps }{2}\\frac{C_\\vareps}{C_\\beta},\\,\\forall \\, 0\\le k<K,\n\\end{equation}","sha256_utf8":"cba87c0688bf4ee8f76a206b08355548a7d3e8161479c872de9c196ee27bcb7d"}]},{"id":"S023","semantic_type":"lemma","kind":"lemma","title":"从拉格朗日差距到原始误差","source_units":[{"start":32702,"end":33471,"start_line":473,"end_line":483,"raw_tex":"\\begin{lemma}\\label{lem:pre-rate}\nAssume $(\\vx^*,\\vy^*,\\vz^*)$ satisfies the KKT conditions in \\eqref{eq:kkt}. Let $\\bar{\\vx}$ be a point such that for any $\\vy$ and any $\\vz\\ge\\vzero$,\n\\begin{equation}\\label{eq:pre-rate-cond}\nf_0(\\bar{\\vx})-f_0(\\vx^*)+ \\vy^\\top (\\vA\\bar{\\vx}-\\vb) +\\sum_{i=1}^m z_i f_i(\\bar{\\vx}) \\le \\alpha + c_1\\|\\vy\\|^2+c_2\\|\\vz\\|^2,\n\\end{equation}\nwhere $\\alpha$ and $c_1, c_2$ are nonnegative constants independent of $\\vy$ and $\\vz$. Then\n\\begin{align}\n-\\left(\\alpha + 4c_1\\|\\vy^*\\|^2+ 4c_2\\|\\vz^*\\|^2\\right)\\le f_0(\\bar{\\vx})-f_0(\\vx^*)\\le \\alpha,\\label{eq:pre-rate-obj}\\\\\n\\|\\vA\\bar{\\vx}-\\vb\\| + \\big\\|[\\vf(\\bar{\\vx})]_+\\big\\| \\le \\alpha + c_1\\big(1+\\|\\vy^*\\|\\big)^2 + c_2\\big(1+\\|\\vz^*\\|\\big)^2. \\label{eq:pre-rate-res}\n\\end{align}\n\\end{lemma}","sha256_utf8":"42137643fe5232f3595ee78956cefa9291c1e399937c42619cc143f2cc7e86b9"}]},{"id":"S024","semantic_type":"proof","kind":"proof","title":"原始误差转换的完整证明","source_units":[{"start":33473,"end":35078,"start_line":485,"end_line":506,"raw_tex":"\\begin{proof}\nLetting $\\vy=\\vzero$ and $\\vz=\\vzero$ in \\eqref{eq:pre-rate-cond} gives the second inequality in \\eqref{eq:pre-rate-obj}.\nFor any nonnegative $\\gamma_y$ and $\\gamma_z$, we let \n$$\\vy=\\gamma_y\\frac{\\vA\\bar{\\vx}-\\vb}{\\|\\vA\\bar{\\vx}-\\vb\\|},\\quad \\vz = \\gamma_z \\frac{[\\vf(\\bar{\\vx})]_+}{\\big\\|[\\vf(\\bar{\\vx})]_+\\big\\|}$$ %z_i = \\gamma_i \\frac{[f_i(\\bar{\\vx})]_+}{|f_i(\\bar{\\vx})|}, \\forall i\\in [m]$$\nand have from \\eqref{eq:pre-rate-cond} by using the convention $\\frac{0}{0}=0$ that\n\\begin{align}\\label{eq:pre-rate-ineq2}\nf_0(\\bar{\\vx})-f_0(\\vx^*) +\\gamma_y \\|\\vA\\bar{\\vx}-\\vb\\| + \\gamma_z\\big\\|[\\vf(\\bar{\\vx})]_+\\big\\|\n\\le  \\alpha + c_1\\gamma_y^2+ c_2\\gamma_z^2.\n%f_0(\\bar{\\vx})-f_0(\\vx^*) +\\gamma_y \\|\\vA\\bar{\\vx}-\\vb\\| + \\sum_{i=1}^m \\gamma_i [f_i(\\bar{\\vx})]_+\n%\\le  \\alpha + c_1\\gamma_y^2+ c_2\\sum_{i=1}^m\\gamma_i^2.\n\\end{align}\n\nNoting \n\\begin{equation}\\label{eq:pre-rate-ineq3}\n-\\langle \\vy^*, \\vA\\bar{\\vx}-\\vb\\rangle \\ge -\\|\\vy^*\\|\\cdot\\|\\vA\\bar{\\vx}-\\vb\\|,\\quad - \\sum_{i=1}^mz_i^* f_i(\\bar{\\vx}) \\ge - \\|\\vz^*\\|\\cdot \\big\\|[\\vf(\\bar{\\vx})]_+\\big\\|,\n\\end{equation} we have from \\eqref{eq:opt} and \\eqref{eq:pre-rate-ineq2} that\n\\begin{align*}\n(\\gamma_y-\\|\\vy^*\\|) \\|\\vA\\bar{\\vx}-\\vb\\| + (\\gamma_z -\\|\\vz^*\\|)\\|[\\vf(\\bar{\\vx})]_+\\big\\|\n\\le  \\alpha + c_1\\gamma_y^2+ c_2\\gamma_z^2\n\\end{align*}\nIn the above inequality, letting $\\gamma_y=1+\\|\\vy^*\\|$ and $\\gamma_z = 1+\\|\\vz^*\\|$ gives \\eqref{eq:pre-rate-res}, and letting $\\gamma_y=2\\|\\vy^*\\|$ and $\\gamma_z = 2\\|\\vz^*\\|$ gives the first inequality in \\eqref{eq:pre-rate-obj} by \\eqref{eq:opt} and \\eqref{eq:pre-rate-ineq3}.\n\\end{proof}","sha256_utf8":"4c34aca26b3e6494ca3d165acb71cf3d6e6e0e0c626c448fe8fb70aa590ef3ad"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"零乘子给出目标上界","source_units":[{"start":33473,"end":33609,"start_line":485,"end_line":486,"raw_tex":"\\begin{proof}\nLetting $\\vy=\\vzero$ and $\\vz=\\vzero$ in \\eqref{eq:pre-rate-cond} gives the second inequality in \\eqref{eq:pre-rate-obj}.\n","sha256_utf8":"80ec3e2b8ba389e8e9e9258b85f71adeb2a227e45ba5ea0dea992652c6e41660"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"沿残差方向选择测试乘子","source_units":[{"start":33609,"end":34320,"start_line":487,"end_line":495,"raw_tex":"For any nonnegative $\\gamma_y$ and $\\gamma_z$, we let \n$$\\vy=\\gamma_y\\frac{\\vA\\bar{\\vx}-\\vb}{\\|\\vA\\bar{\\vx}-\\vb\\|},\\quad \\vz = \\gamma_z \\frac{[\\vf(\\bar{\\vx})]_+}{\\big\\|[\\vf(\\bar{\\vx})]_+\\big\\|}$$ %z_i = \\gamma_i \\frac{[f_i(\\bar{\\vx})]_+}{|f_i(\\bar{\\vx})|}, \\forall i\\in [m]$$\nand have from \\eqref{eq:pre-rate-cond} by using the convention $\\frac{0}{0}=0$ that\n\\begin{align}\\label{eq:pre-rate-ineq2}\nf_0(\\bar{\\vx})-f_0(\\vx^*) +\\gamma_y \\|\\vA\\bar{\\vx}-\\vb\\| + \\gamma_z\\big\\|[\\vf(\\bar{\\vx})]_+\\big\\|\n\\le  \\alpha + c_1\\gamma_y^2+ c_2\\gamma_z^2.\n%f_0(\\bar{\\vx})-f_0(\\vx^*) +\\gamma_y \\|\\vA\\bar{\\vx}-\\vb\\| + \\sum_{i=1}^m \\gamma_i [f_i(\\bar{\\vx})]_+\n%\\le  \\alpha + c_1\\gamma_y^2+ c_2\\sum_{i=1}^m\\gamma_i^2.\n\\end{align}\n","sha256_utf8":"0f862a8ffbbf4910bea5f22fc032ff22de864df771a99acad9c58b9461e743bf"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"结合最优乘子控制残差","source_units":[{"start":34321,"end":34786,"start_line":497,"end_line":504,"raw_tex":"Noting \n\\begin{equation}\\label{eq:pre-rate-ineq3}\n-\\langle \\vy^*, \\vA\\bar{\\vx}-\\vb\\rangle \\ge -\\|\\vy^*\\|\\cdot\\|\\vA\\bar{\\vx}-\\vb\\|,\\quad - \\sum_{i=1}^mz_i^* f_i(\\bar{\\vx}) \\ge - \\|\\vz^*\\|\\cdot \\big\\|[\\vf(\\bar{\\vx})]_+\\big\\|,\n\\end{equation} we have from \\eqref{eq:opt} and \\eqref{eq:pre-rate-ineq2} that\n\\begin{align*}\n(\\gamma_y-\\|\\vy^*\\|) \\|\\vA\\bar{\\vx}-\\vb\\| + (\\gamma_z -\\|\\vz^*\\|)\\|[\\vf(\\bar{\\vx})]_+\\big\\|\n\\le  \\alpha + c_1\\gamma_y^2+ c_2\\gamma_z^2\n\\end{align*}\n","sha256_utf8":"61e6e96603c35ad50e66e8509730c16a914d9cc6b99707dcbecded52594adbf2"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"指定乘子尺度完成双侧界","source_units":[{"start":34786,"end":35078,"start_line":505,"end_line":506,"raw_tex":"In the above inequality, letting $\\gamma_y=1+\\|\\vy^*\\|$ and $\\gamma_z = 1+\\|\\vz^*\\|$ gives \\eqref{eq:pre-rate-res}, and letting $\\gamma_y=2\\|\\vy^*\\|$ and $\\gamma_z = 2\\|\\vz^*\\|$ gives the first inequality in \\eqref{eq:pre-rate-obj} by \\eqref{eq:opt} and \\eqref{eq:pre-rate-ineq3}.\n\\end{proof}","sha256_utf8":"250fac59d21e02018f35a364266878e77d552fe3f2545cc155cd975ec7f31bae"}]},{"id":"S029","semantic_type":"lemma","kind":"lemma","title":"乘子更新的平方范数恒等式","source_units":[{"start":40411,"end":40993,"start_line":583,"end_line":590,"raw_tex":"\\begin{lemma}\\label{lem:multiplier}\nLet $\\vy$ and $\\vz$ be updated by \\eqref{eq:alm-y} and \\eqref{eq:alm-z} respectively. Then for any $k$, it holds\n\\begin{align}\n\\frac{1}{2\\rho_k}\\big[\\|\\vy^{k+1} -\\vy\\|^2-\\|\\vy^k -\\vy\\|^2+\\|\\vy^{k+1} -\\vy^k \\|^2\\big] - \\langle \\vy^{k+1} -\\vy, \\vr^{k+1}\\rangle = &~ 0,\\label{eq:alm-yterm}\\\\\n\\frac{1}{2\\rho_k}\\big[\\|\\vz^{k+1} -\\vz\\|^2-\\|\\vz^k  -\\vz\\|^2+\\|\\vz^{k+1} -\\vz^k  \\|^2\\big]- \\sum_{i=1}^m (z_i^{k+1}-z_i)\\cdot\\max\\big(-\\frac{z_i^k}{\\beta_k}, f_i(\\vx^{k+1})\\big)=& ~0,\\label{eq:alm-zterm}\n\\end{align}\nwhere $\\vr^k =\\vA\\vx^k -\\vb$.\n\\end{lemma}","sha256_utf8":"ef43df6ffaca93fcebe60fd3015c31c243a902c250d66ac4bb3c680504dbe585"}]},{"id":"S030","semantic_type":"proof","kind":"proof","title":"乘子恒等式的完整证明","source_units":[{"start":40994,"end":41171,"start_line":591,"end_line":593,"raw_tex":"\\begin{proof}\nUsing the equality $2\\vu^\\top \\vv = \\|\\vu\\|^2 - \\|\\vu-\\vv\\|^2 + \\|\\vv\\|^2$, we have the results from the updates \\eqref{eq:alm-y} and \\eqref{eq:alm-z}.\n\\end{proof}","sha256_utf8":"d8f4f49dd7fd5604f1eeb1ad81365915fb9bc9bc228d1b89dd6f55ddc1df0c60"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"对乘子更新应用极化恒等式","source_units":[{"start":40994,"end":41171,"start_line":591,"end_line":593,"raw_tex":"\\begin{proof}\nUsing the equality $2\\vu^\\top \\vv = \\|\\vu\\|^2 - \\|\\vu-\\vv\\|^2 + \\|\\vv\\|^2$, we have the results from the updates \\eqref{eq:alm-y} and \\eqref{eq:alm-z}.\n\\end{proof}","sha256_utf8":"d8f4f49dd7fd5604f1eeb1ad81365915fb9bc9bc228d1b89dd6f55ddc1df0c60"}]},{"id":"S032","semantic_type":"lemma","kind":"lemma","title":"不等式乘子项的下界","source_units":[{"start":41172,"end":41531,"start_line":594,"end_line":600,"raw_tex":"\\begin{lemma}\\label{lem:ineq-z}\nFor any $\\vz\\ge\\vzero$, we have\n\\begin{align}\\label{eq:alm-ineq-z}\n&\\sum_{i=1}^m\\big([z_i^k+\\beta_k f_i(\\vx^{k+1})]_+-z_i\\big) f_i(\\vx^{k+1})- \\sum_{i=1}^m (z_i^{k+1}-z_i)\\cdot\\max\\big(-\\frac{z_i^k}{\\beta_k}, f_i(\\vx^{k+1})\\big)\\nonumber\\\\\n\\ge&\\frac{1}{\\rho_k^2}(\\beta_k-\\rho_k)\\|\\vz^{k+1} -\\vz^k  \\|^2.\n\\end{align}\n\\end{lemma}","sha256_utf8":"7d54b371b9751a08ebef1f4ea27ac2f5e355dbaa3adadd462fcf3d6ad6c922b3"}]},{"id":"S033","semantic_type":"proof","kind":"proof","title":"乘子项下界的完整证明","source_units":[{"start":41533,"end":42615,"start_line":602,"end_line":617,"raw_tex":"\\begin{proof}\nDenote \n\\begin{equation}\\label{eq:setIk}\nI_+^k=\\{i\\in [m]: z_i^k + \\beta_k f_i(\\vx^{k+1}) \\ge 0\\},\\quad I_-^k=[m]\\backslash I_+^k.\n\\end{equation}\n Then \n\\begin{align*}\n&\\text{the left hand side of \\eqref{eq:alm-ineq-z}}\\\\\n=&\\sum_{i\\in I_+^k}\\left[(z_i^k-z_i)f_i(\\vx^{k+1})+\\beta_k[f_i(\\vx^{k+1})]^2-\\big(z_i^k+\\rho_k f_i(\\vx^{k+1})-z_i\\big)f_i(\\vx^{k+1})\\right]\\\\\n&+\\sum_{i\\in I_-^k}\\left[-z_i f_i(\\vx^{k+1})-\\big(z_i^k-\\frac{\\rho_k z_i^k}{\\beta_k}-z_i\\big)\\big(-\\frac{z_i^k}{\\beta_k}\\big)\\right]\\\\\n=&(\\beta_k-\\rho_k)\\sum_{i\\in I_+^k}[f_i(\\vx^{k+1})]^2+\\sum_{i\\in I_-^k}\\left[-z_i \\big(f_i(\\vx^{k+1})+\\frac{z_i^k}{\\beta_k}\\big)+\\frac{1}{\\beta_k^2}(\\beta_k-\\rho_k)(z_i^k)^2\\right]\\\\\n\\ge& (\\beta_k-\\rho_k)\\sum_{i\\in I_+^k}[f_i(\\vx^{k+1})]^2 +\\frac{1}{\\beta_k^2}(\\beta_k-\\rho_k)\\sum_{i\\in I_-^k}(z_i^k)^2\\\\\n=&\\frac{1}{\\rho_k^2}(\\beta_k-\\rho_k)\\|\\vz^{k+1} -\\vz^k  \\|^2,\n\\end{align*}\nwhere the inequality follows from $z_i\\ge0$ and $f_i(\\vx^{k+1})+\\frac{z_i^k}{\\beta_k}\\le 0,\\,\\forall i\\in I_-^k$, and the last equality holds due to the update \\eqref{eq:alm-z}.\n\\end{proof}","sha256_utf8":"7702f704d1d3b7194f0dd720fbd2a7c03deed3abb05c802f064eb2d30d09145c"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"按截断分支划分约束指标","source_units":[{"start":41533,"end":41693,"start_line":602,"end_line":606,"raw_tex":"\\begin{proof}\nDenote \n\\begin{equation}\\label{eq:setIk}\nI_+^k=\\{i\\in [m]: z_i^k + \\beta_k f_i(\\vx^{k+1}) \\ge 0\\},\\quad I_-^k=[m]\\backslash I_+^k.\n\\end{equation}\n","sha256_utf8":"db558615e79b234ca67fcb0b8975ce8315ac484d33584528f5931df7b1348825"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"展开两类指标并利用非负性","source_units":[{"start":41693,"end":42615,"start_line":607,"end_line":617,"raw_tex":" Then \n\\begin{align*}\n&\\text{the left hand side of \\eqref{eq:alm-ineq-z}}\\\\\n=&\\sum_{i\\in I_+^k}\\left[(z_i^k-z_i)f_i(\\vx^{k+1})+\\beta_k[f_i(\\vx^{k+1})]^2-\\big(z_i^k+\\rho_k f_i(\\vx^{k+1})-z_i\\big)f_i(\\vx^{k+1})\\right]\\\\\n&+\\sum_{i\\in I_-^k}\\left[-z_i f_i(\\vx^{k+1})-\\big(z_i^k-\\frac{\\rho_k z_i^k}{\\beta_k}-z_i\\big)\\big(-\\frac{z_i^k}{\\beta_k}\\big)\\right]\\\\\n=&(\\beta_k-\\rho_k)\\sum_{i\\in I_+^k}[f_i(\\vx^{k+1})]^2+\\sum_{i\\in I_-^k}\\left[-z_i \\big(f_i(\\vx^{k+1})+\\frac{z_i^k}{\\beta_k}\\big)+\\frac{1}{\\beta_k^2}(\\beta_k-\\rho_k)(z_i^k)^2\\right]\\\\\n\\ge& (\\beta_k-\\rho_k)\\sum_{i\\in I_+^k}[f_i(\\vx^{k+1})]^2 +\\frac{1}{\\beta_k^2}(\\beta_k-\\rho_k)\\sum_{i\\in I_-^k}(z_i^k)^2\\\\\n=&\\frac{1}{\\rho_k^2}(\\beta_k-\\rho_k)\\|\\vz^{k+1} -\\vz^k  \\|^2,\n\\end{align*}\nwhere the inequality follows from $z_i\\ge0$ and $f_i(\\vx^{k+1})+\\frac{z_i^k}{\\beta_k}\\le 0,\\,\\forall i\\in I_-^k$, and the last equality holds due to the update \\eqref{eq:alm-z}.\n\\end{proof}","sha256_utf8":"c726b0cc7bb6b5696a00052020179d860b71a1499cfb786880d604fb936262fb"}]},{"id":"S036","semantic_type":"theorem","kind":"theorem","title":"单步原始与对偶进展","source_units":[{"start":52990,"end":53697,"start_line":739,"end_line":746,"raw_tex":"\\begin{theorem}[One-iteration progress of iALM]\\label{thm:1iter-ialm}\nLet $\\{(\\vx^k ,\\vy^k ,\\vz^k  )\\}$ be the sequence generated from Algorithm \\ref{alg:ialm}. Then for any $\\vx\\in\\cX$ such that $\\vA\\vx=\\vb$ and $f_i(\\vx)\\le 0,\\,\\forall i\\in [m]$, any $\\vy$, and any $\\vz\\ge\\vzero$, it holds that\n\\begin{align}\\label{eq:ialm-ineq2}\n&f_0(\\vx^{k+1})-f_0(\\vx) +\\vy^\\top \\vr^{k+1} + \\sum_{i=1}^m z_i f_i(\\vx^{k+1})+\\frac{\\beta_k-\\rho_k}{2}\\|\\vr^{k+1}\\|^2\\cr\n&+\\frac{\\beta_k-\\rho_k}{2\\rho_k^2}\\|\\vz^{k+1} -\\vz^k  \\|^2+\\frac{1}{2\\rho_k}\\|\\vy^{k+1} -\\vy\\|^2 + \\frac{1}{2\\rho_k}\\|\\vz^{k+1} -\\vz\\|^2\\cr\n\\le & \\frac{1}{2\\rho_k}\\|\\vy^k -\\vy\\|^2+ \\frac{1}{2\\rho_k}\\|\\vz^k  -\\vz\\|^2+\\vareps_k.\n\\end{align}\n\\end{theorem}","sha256_utf8":"ebbf16030cdb2aa8602ef3ae38c6284c0dbad69ccf5edca53becd28b958d8f4b"}]},{"id":"S037","semantic_type":"proof","kind":"proof","title":"单步进展的完整证明","source_units":[{"start":53961,"end":55976,"start_line":750,"end_line":773,"raw_tex":"\\begin{proof}\nFrom \\eqref{eq:ialm-x}, it follows that for any $\\vx$ such that $\\vA\\vx=\\vb$,\n\\begin{equation}\\label{eq:ialm-1st}\nf_0(\\vx^{k+1})+\\langle \\vy^k , \\vr^{k+1}\\rangle +\\frac{\\beta_k}{2}\\|\\vr^{k+1}\\|^2+\\Psi_{\\beta_k}(\\vx^{k+1},\\vz^k  )-f_0(\\vx)-\\Psi_{\\beta_k}(\\vx,\\vz^k  )\\le\\vareps_k.\n\\end{equation}\nSince $\\langle \\vy^k , \\vr^{k+1}\\rangle=\\langle \\vy^{k+1} -\\vy, \\vr^{k+1}\\rangle + \\langle \\vy, \\vr^{k+1}\\rangle -\\rho_k\\|\\vr^{k+1}\\|^2$, by adding \\eqref{eq:alm-yterm} and \\eqref{eq:alm-zterm} to the above inequality, we have \n\\begin{align}\\label{eq:ialm-ineq1}\n&f_0(\\vx^{k+1})-f_0(\\vx) +\\vy^\\top \\vr^{k+1} + \\sum_{i=1}^m z_i f_i(\\vx^{k+1})+\\sum_{i=1}^m\\left([z_i^k+\\beta_k f_i(\\vx^{k+1})]_+-z_i\\right)f_i(\\vx^{k+1})\\cr\n&+\\big(\\frac{\\beta_k}{2}-\\rho_k\\big)\\|\\vr^{k+1}\\|^2+\\Psi_{\\beta_k}(\\vx^{k+1},\\vz^k  )-\\sum_{i=1}^m[z_i^k+\\beta_k f_i(\\vx^{k+1})]_+f_i(\\vx^{k+1})-\\Psi_{\\beta_k}(\\vx,\\vz^k  )\\cr\n&+\\frac{1}{2\\rho_k}\\big[\\|\\vy^{k+1} -\\vy\\|^2-\\|\\vy^k -\\vy\\|^2+\\|\\vy^{k+1} -\\vy^k \\|^2\\big] \\cr\n&+\\frac{1}{2\\rho_k}\\big[\\|\\vz^{k+1} -\\vz\\|^2-\\|\\vz^k  -\\vz\\|^2+\\|\\vz^{k+1} -\\vz^k  \\|^2\\big]- \\sum_{i=1}^m (z_i^{k+1}-z_i)\\cdot\\max\\big(-\\frac{z_i^k}{\\beta_k}, f_i(\\vx^{k+1})\\big)\\cr\n\\le & \\vareps_k.\n\\end{align}\n\nNote that\n\\begin{align}\\label{eq:ialm-ineq3}\n&\\Psi_{\\beta_k}(\\vx^{k+1},\\vz^k  )-\\sum_{i=1}^m[z_i^k+\\beta_k f_i(\\vx^{k+1})]_+f_i(\\vx^{k+1})\\cr\n=&\\sum_{i\\in I_+^k}\\left[z_i^k f_i(\\vx^{k+1}) +\\frac{\\beta_k}{2}[f_i(\\vx^{k+1})]^2-[z_i^k +\\beta_k f_i(\\vx^{k+1})]f_i(\\vx^{k+1})\\right]+\\sum_{i\\in I_-^k}\\left[-\\frac{(z_i^k)^2}{2\\beta_k}\\right]\\cr\n=&-\\sum_{i\\in I_+^k}\\frac{\\beta_k}{2}[f_i(\\vx^{k+1})]^2-\\sum_{i\\in I_-^k}\\frac{(z_i^k)^2}{2\\beta_k}\\cr\n=&-\\frac{\\beta_k}{2\\rho_z^2}\\|\\vz^{k+1} -\\vz^k  \\|^2,\n\\end{align}\nwhere the sets $I_+^k$ and $I_-^k$ are defined in \\eqref{eq:setIk}.\nIn addition, if $f_i(\\vx)\\le 0,\\,\\forall i\\in [m]$, then $\\Psi_{\\beta_k}(\\vx,\\vz^k  )\\le 0.$ Hence, plugging \\eqref{eq:alm-ineq-z} and \\eqref{eq:ialm-ineq3} into \\eqref{eq:ialm-ineq1} yields \\eqref{eq:ialm-ineq2}.\n\\end{proof}","sha256_utf8":"40988c51f17573a213b2e0e3617f7718eab46679d7406fb32ae8de57fdb38626"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"由非精确最小化比较可行点","source_units":[{"start":53961,"end":54270,"start_line":750,"end_line":754,"raw_tex":"\\begin{proof}\nFrom \\eqref{eq:ialm-x}, it follows that for any $\\vx$ such that $\\vA\\vx=\\vb$,\n\\begin{equation}\\label{eq:ialm-1st}\nf_0(\\vx^{k+1})+\\langle \\vy^k , \\vr^{k+1}\\rangle +\\frac{\\beta_k}{2}\\|\\vr^{k+1}\\|^2+\\Psi_{\\beta_k}(\\vx^{k+1},\\vz^k  )-f_0(\\vx)-\\Psi_{\\beta_k}(\\vx,\\vz^k  )\\le\\vareps_k.\n\\end{equation}\n","sha256_utf8":"44ae8184a57f81eddf7a36b4477b3faad74ce48ceda60196c19ddd14ca8b61d9"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"加入乘子恒等式整理差距","source_units":[{"start":54270,"end":55174,"start_line":755,"end_line":762,"raw_tex":"Since $\\langle \\vy^k , \\vr^{k+1}\\rangle=\\langle \\vy^{k+1} -\\vy, \\vr^{k+1}\\rangle + \\langle \\vy, \\vr^{k+1}\\rangle -\\rho_k\\|\\vr^{k+1}\\|^2$, by adding \\eqref{eq:alm-yterm} and \\eqref{eq:alm-zterm} to the above inequality, we have \n\\begin{align}\\label{eq:ialm-ineq1}\n&f_0(\\vx^{k+1})-f_0(\\vx) +\\vy^\\top \\vr^{k+1} + \\sum_{i=1}^m z_i f_i(\\vx^{k+1})+\\sum_{i=1}^m\\left([z_i^k+\\beta_k f_i(\\vx^{k+1})]_+-z_i\\right)f_i(\\vx^{k+1})\\cr\n&+\\big(\\frac{\\beta_k}{2}-\\rho_k\\big)\\|\\vr^{k+1}\\|^2+\\Psi_{\\beta_k}(\\vx^{k+1},\\vz^k  )-\\sum_{i=1}^m[z_i^k+\\beta_k f_i(\\vx^{k+1})]_+f_i(\\vx^{k+1})-\\Psi_{\\beta_k}(\\vx,\\vz^k  )\\cr\n&+\\frac{1}{2\\rho_k}\\big[\\|\\vy^{k+1} -\\vy\\|^2-\\|\\vy^k -\\vy\\|^2+\\|\\vy^{k+1} -\\vy^k \\|^2\\big] \\cr\n&+\\frac{1}{2\\rho_k}\\big[\\|\\vz^{k+1} -\\vz\\|^2-\\|\\vz^k  -\\vz\\|^2+\\|\\vz^{k+1} -\\vz^k  \\|^2\\big]- \\sum_{i=1}^m (z_i^{k+1}-z_i)\\cdot\\max\\big(-\\frac{z_i^k}{\\beta_k}, f_i(\\vx^{k+1})\\big)\\cr\n\\le & \\vareps_k.\n\\end{align}\n","sha256_utf8":"6ea25f48aa735d266fafa7a69a5d4eef36f5b5fa1d8e3b74ded6c9bad86e0ecf"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"逐分支计算增广惩罚差项","source_units":[{"start":55175,"end":55751,"start_line":764,"end_line":771,"raw_tex":"Note that\n\\begin{align}\\label{eq:ialm-ineq3}\n&\\Psi_{\\beta_k}(\\vx^{k+1},\\vz^k  )-\\sum_{i=1}^m[z_i^k+\\beta_k f_i(\\vx^{k+1})]_+f_i(\\vx^{k+1})\\cr\n=&\\sum_{i\\in I_+^k}\\left[z_i^k f_i(\\vx^{k+1}) +\\frac{\\beta_k}{2}[f_i(\\vx^{k+1})]^2-[z_i^k +\\beta_k f_i(\\vx^{k+1})]f_i(\\vx^{k+1})\\right]+\\sum_{i\\in I_-^k}\\left[-\\frac{(z_i^k)^2}{2\\beta_k}\\right]\\cr\n=&-\\sum_{i\\in I_+^k}\\frac{\\beta_k}{2}[f_i(\\vx^{k+1})]^2-\\sum_{i\\in I_-^k}\\frac{(z_i^k)^2}{2\\beta_k}\\cr\n=&-\\frac{\\beta_k}{2\\rho_z^2}\\|\\vz^{k+1} -\\vz^k  \\|^2,\n\\end{align}\nwhere the sets $I_+^k$ and $I_-^k$ are defined in \\eqref{eq:setIk}.\n","sha256_utf8":"2873b509a09e03c874abb91b4d6aa65a59a9108c53f1bf1277c42c1336ebac23"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"利用可行性合并单步估计","source_units":[{"start":55751,"end":55976,"start_line":772,"end_line":773,"raw_tex":"In addition, if $f_i(\\vx)\\le 0,\\,\\forall i\\in [m]$, then $\\Psi_{\\beta_k}(\\vx,\\vz^k  )\\le 0.$ Hence, plugging \\eqref{eq:alm-ineq-z} and \\eqref{eq:ialm-ineq3} into \\eqref{eq:ialm-ineq1} yields \\eqref{eq:ialm-ineq2}.\n\\end{proof}","sha256_utf8":"21e49e7fc0eac89ef8ccfee08bad308f3333a1a25d346828972ecd18d1acbd60"}]},{"id":"S042","semantic_type":"theorem","kind":"theorem","title":"加权平均点的遍历收敛界","source_units":[{"start":59248,"end":60183,"start_line":799,"end_line":811,"raw_tex":"\\begin{theorem}[Ergodic convergence rate of iALM]\\label{thm:ialm-rate}\nUnder Assumptions \\ref{assump-kkt} and \\ref{assump-wd-ialm}, let $\\{(\\vx^k ,\\vy^k ,\\vz^k  )\\}$ be the sequence generated from Algorithm \\ref{alg:ialm} with $\\vy^0=\\vzero, \\vz^0 = \\vzero$ and $0<\\rho_k\\le\\beta_k,\\,\\forall k$. Then\n\\begin{subequations}\\label{eq:ialm-rate}\n\\begin{align}\n\\big|f_0(\\bar{\\vx}^{K})-f_0(\\vx^*)\\big| \\le \\frac{1}{\\sum_{t=0}^{K-1}\\rho_t}\\left(2\\|\\vy^*\\|^2+ 2\\|\\vz^*\\|^2+\\sum_{k=0}^{K-1}\\rho_k\\vareps_k\\right),\\label{eq:ialm-rate-obj}\\\\\n\\|\\vA\\bar{\\vx}^{K}-\\vb\\| + \\big\\| [\\vf(\\bar{\\vx}^{K})]_+\\big\\| \\le  \\frac{1}{\\sum_{t=0}^{K-1}\\rho_t}\\left(\\frac{(1+\\|\\vy^*\\|)^2}{2}+ \\frac{(\\|1+\\|\\vz^*\\|)^2}{2}+\\sum_{k=0}^{K-1}\\rho_k\\vareps_k\\right),\\label{eq:ialm-rate-res}\n\\end{align}\n\\end{subequations}\nwhere \n\\begin{equation}\\label{eq:barx-K}\n\\bar{\\vx}^{K}=\\frac{\\sum_{t=0}^{K-1}\\rho_t\\vx^{t+1} }{\\sum_{t=0}^{K-1}\\rho_t}.\n\\end{equation}\n\\end{theorem}","sha256_utf8":"89dfc3784575558d71c7f5d2c71fb58009a502d9dbf9256b8b5e2ccfbbd2b3d5"}]},{"id":"S043","semantic_type":"proof","kind":"proof","title":"遍历收敛界的完整证明","source_units":[{"start":60185,"end":62021,"start_line":813,"end_line":837,"raw_tex":"\\begin{proof}\nSince $\\rho_k\\le \\beta_k$, the two terms $\\frac{\\beta_k-\\rho_k}{2}\\|\\vr^{k+1}\\|^2$ and \n$\\frac{\\beta_k-\\rho_k}{2\\rho_k^2}\\|\\vz^{k+1} -\\vz^k  \\|^2$ are nonnegative. Dropping them and multiplying $\\rho_k$ to both sides of \\eqref{eq:ialm-ineq2} yields \n\\begin{align}\\label{eq:ialm-ineq2-1}\n&\\rho_k\\left[f_0(\\vx^{k+1})-f_0(\\vx) +\\vy^\\top  \\vr^{k+1} + \\sum_{i=1}^m z_i f_i(\\vx^{k+1})\\right]+\\frac{1}{2}\\|\\vy^{k+1} -\\vy\\|^2 + \\frac{1}{2}\\|\\vz^{k+1} -\\vz\\|^2\\cr\n\\le & \\frac{1}{2}\\|\\vy^k -\\vy\\|^2+ \\frac{1}{2}\\|\\vz^k  -\\vz\\|^2+\\rho_k\\vareps_k.\n\\end{align}\n\nSumming up \\eqref{eq:ialm-ineq2} with $\\vx=\\vx^*$ gives\n\\begin{align}\\label{eq:ialm-ineq2-2}\n&\\sum_{k=0}^{K-1}\\rho_k\\left[f_0(\\vx^{k+1})-f_0(\\vx^*) +\\vy^\\top \\vr^{k+1} + \\sum_{i=1}^m z_i f_i(\\vx^{k+1})\\right]+\\frac{1}{2}\\|\\vy^{K}-\\vy\\|^2 + \\frac{1}{2}\\|\\vz^{K}-\\vz\\|^2\\cr\n\\le & \\frac{1}{2}\\|\\vy^0-\\vy\\|^2+ \\frac{1}{2}\\|\\vz^0-\\vz\\|^2+\\sum_{k=0}^{K-1}\\rho_k\\vareps_k.\n\\end{align}\nBy the convexity of $f_i$'s and the nonnegativity of $\\vz$, we have\n\\begin{align*}\n&~f_0(\\bar{\\vx}^{K})-f_0(\\vx^*) +\\vy^\\top  (\\vA\\bar{\\vx}^{K}-\\vb) + \\sum_{i=1}^m z_i f_i(\\bar{\\vx}^{K})\\\\\n\\le &~\\frac{1}{\\sum_{t=0}^{K-1}\\rho_t} \\sum_{k=0}^{K-1}\\rho_k\\left[f_0(\\vx^{k+1})-f_0(\\vx^*) +\\vy^\\top  \\vr^{k+1} + \\sum_{i=1}^m z_i f_i(\\vx^{k+1})\\right],\n\\end{align*}\nwhich together with \\eqref{eq:ialm-ineq2-2} implies\n$$f_0(\\bar{\\vx}^{K})-f_0(\\vx^*) +\\vy^\\top  (\\vA\\bar{\\vx}^{K}-\\vb) + \\sum_{i=1}^m z_i f_i(\\bar{\\vx}^{K})\n\\le  \\frac{1}{\\sum_{t=0}^{K-1}\\rho_t}\\left(\\frac{1}{2}\\|\\vy\\|^2+ \\frac{1}{2}\\|\\vz\\|^2+\\sum_{k=0}^{K-1}\\rho_k\\vareps_k\\right).$$\nThe results thus follow from Lemma \\ref{lem:pre-rate} with \n$$\\alpha=\\frac{\\sum_{k=0}^{K-1}\\rho_k\\vareps_k}{\\sum_{k=0}^{K-1}\\rho_k},\\ c_1=\\frac{1}{2\\sum_{k=0}^{K-1}\\rho_k},\\ c_2=\\frac{1}{2\\sum_{k=0}^{K-1}\\rho_k},$$\nand we complete the proof.\n\\end{proof}","sha256_utf8":"7be0d2d9154abfcd09f49b1621e1d7f4cdf89011eb0c82e5d999256b8b84e60e"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"舍弃非负项并按步长加权","source_units":[{"start":60185,"end":60747,"start_line":813,"end_line":819,"raw_tex":"\\begin{proof}\nSince $\\rho_k\\le \\beta_k$, the two terms $\\frac{\\beta_k-\\rho_k}{2}\\|\\vr^{k+1}\\|^2$ and \n$\\frac{\\beta_k-\\rho_k}{2\\rho_k^2}\\|\\vz^{k+1} -\\vz^k  \\|^2$ are nonnegative. Dropping them and multiplying $\\rho_k$ to both sides of \\eqref{eq:ialm-ineq2} yields \n\\begin{align}\\label{eq:ialm-ineq2-1}\n&\\rho_k\\left[f_0(\\vx^{k+1})-f_0(\\vx) +\\vy^\\top  \\vr^{k+1} + \\sum_{i=1}^m z_i f_i(\\vx^{k+1})\\right]+\\frac{1}{2}\\|\\vy^{k+1} -\\vy\\|^2 + \\frac{1}{2}\\|\\vz^{k+1} -\\vz\\|^2\\cr\n\\le & \\frac{1}{2}\\|\\vy^k -\\vy\\|^2+ \\frac{1}{2}\\|\\vz^k  -\\vz\\|^2+\\rho_k\\vareps_k.\n\\end{align}\n","sha256_utf8":"7965bcca52e2bdee786237ef7175a65c9f38b6c27b9cbe10e5814562075a3002"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"累加得到乘子距离的望远镜界","source_units":[{"start":60748,"end":61126,"start_line":821,"end_line":825,"raw_tex":"Summing up \\eqref{eq:ialm-ineq2} with $\\vx=\\vx^*$ gives\n\\begin{align}\\label{eq:ialm-ineq2-2}\n&\\sum_{k=0}^{K-1}\\rho_k\\left[f_0(\\vx^{k+1})-f_0(\\vx^*) +\\vy^\\top \\vr^{k+1} + \\sum_{i=1}^m z_i f_i(\\vx^{k+1})\\right]+\\frac{1}{2}\\|\\vy^{K}-\\vy\\|^2 + \\frac{1}{2}\\|\\vz^{K}-\\vz\\|^2\\cr\n\\le & \\frac{1}{2}\\|\\vy^0-\\vy\\|^2+ \\frac{1}{2}\\|\\vz^0-\\vz\\|^2+\\sum_{k=0}^{K-1}\\rho_k\\vareps_k.\n\\end{align}\n","sha256_utf8":"501da32a0ec5e60f7ca8c72c6918862a0bb98bdc2b9db2cc4272a7d3e80ba610"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"利用凸性转换为平均点差距","source_units":[{"start":61126,"end":61768,"start_line":826,"end_line":833,"raw_tex":"By the convexity of $f_i$'s and the nonnegativity of $\\vz$, we have\n\\begin{align*}\n&~f_0(\\bar{\\vx}^{K})-f_0(\\vx^*) +\\vy^\\top  (\\vA\\bar{\\vx}^{K}-\\vb) + \\sum_{i=1}^m z_i f_i(\\bar{\\vx}^{K})\\\\\n\\le &~\\frac{1}{\\sum_{t=0}^{K-1}\\rho_t} \\sum_{k=0}^{K-1}\\rho_k\\left[f_0(\\vx^{k+1})-f_0(\\vx^*) +\\vy^\\top  \\vr^{k+1} + \\sum_{i=1}^m z_i f_i(\\vx^{k+1})\\right],\n\\end{align*}\nwhich together with \\eqref{eq:ialm-ineq2-2} implies\n$$f_0(\\bar{\\vx}^{K})-f_0(\\vx^*) +\\vy^\\top  (\\vA\\bar{\\vx}^{K}-\\vb) + \\sum_{i=1}^m z_i f_i(\\bar{\\vx}^{K})\n\\le  \\frac{1}{\\sum_{t=0}^{K-1}\\rho_t}\\left(\\frac{1}{2}\\|\\vy\\|^2+ \\frac{1}{2}\\|\\vz\\|^2+\\sum_{k=0}^{K-1}\\rho_k\\vareps_k\\right).$$\n","sha256_utf8":"809053c9f1ed10fd56f738a22e4c7f7b24e0b6f4803cef9df16361cdfb56dc32"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"应用误差转换引理","source_units":[{"start":61768,"end":62021,"start_line":834,"end_line":837,"raw_tex":"The results thus follow from Lemma \\ref{lem:pre-rate} with \n$$\\alpha=\\frac{\\sum_{k=0}^{K-1}\\rho_k\\vareps_k}{\\sum_{k=0}^{K-1}\\rho_k},\\ c_1=\\frac{1}{2\\sum_{k=0}^{K-1}\\rho_k},\\ c_2=\\frac{1}{2\\sum_{k=0}^{K-1}\\rho_k},$$\nand we complete the proof.\n\\end{proof}","sha256_utf8":"2acc3744e6e7fb1d5388cb772eae72618087eef66627ac4f05c60e052824f59a"}]},{"id":"S048","semantic_type":"proposition","kind":"proposition","title":"约束函数与梯度的统一界","source_units":[{"start":63496,"end":64053,"start_line":856,"end_line":864,"raw_tex":"\\begin{proposition}\\label{prop-lip}\nAssume \\eqref{eq:lip-g}, \\eqref{eq:lip-f}, and the boundedness of $\\dom(h)\\cap \\cX$. Then there exist constants $B_1,\\ldots, B_m$ such that\n\\begin{subequations}\\label{eq:lip}\n\\begin{align}\n\\max\\big(|f_i(\\vx)|, \\|\\nabla f_i(\\vx)\\|\\big) \\le B_i, \\,& \\forall \\vx\\in \\dom(h)\\cap\\cX, \\forall i\\in [m],\\label{eq:bd-f}\\\\\n|f_i(\\hat{\\vx})-f_i(\\tilde{\\vx})| \\le B_i\\|\\hat{\\vx}-\\tilde{\\vx}\\|, \\, & \\forall \\hat{\\vx}, \\tilde{\\vx} \\in \\dom(h)\\cap\\cX, \\forall i\\in [m].\\label{eq:lip-fv}\n\\end{align}\n\\end{subequations}\n\\end{proposition}","sha256_utf8":"557f6642064828d36552dd3c28ba630c21e0814ae1fcad4a80c8040558c0f24b"}]},{"id":"S049","semantic_type":"proof","kind":"proof","title":"有界性命题的作者简述","source_units":[{"start":63390,"end":63496,"start_line":855,"end_line":855,"raw_tex":"The following results are easy to show from the Lipschitz differentiability of $g$ and $f_i,\\, i\\in [m]$.\n","sha256_utf8":"ce8955f2ac724e96df287f784870530677d700c8aceffaca2e3b1feea39d9f2b"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"由梯度光滑性得到统一界","source_units":[{"start":63390,"end":63496,"start_line":855,"end_line":855,"raw_tex":"The following results are easy to show from the Lipschitz differentiability of $g$ and $f_i,\\, i\\in [m]$.\n","sha256_utf8":"ce8955f2ac724e96df287f784870530677d700c8aceffaca2e3b1feea39d9f2b"}]},{"id":"S051","semantic_type":"lemma","kind":"lemma","title":"内层光滑梯度的利普希茨常数","source_units":[{"start":64312,"end":64756,"start_line":869,"end_line":874,"raw_tex":"\\begin{lemma}\\label{lem:lip}\nAssume \\eqref{eq:lip-g}, \\eqref{eq:lip-f}, and the boundedness of $\\dom(h)\\cap \\cX$. Let $B_i$'s be given in Proposition \\ref{prop-lip}. Then $\\nabla_\\vx F_{\\beta_k}(\\vx,\\vy^k,\\vz^k)$ is Lipschitz continuous on $\\dom(h)\\cap\\cX$ in terms of $\\vx$ with constant\n\\begin{equation}\\label{eq:def-Lz}\nL(\\vz^k) = L_0+\\beta_k\\|\\vA^\\top \\vA\\|+\\sum_{i=1}^m \\big(\\beta_k B_i(B_i+L_i)+L_i|z_i^k|\\big).\n\\end{equation}\n\\end{lemma}","sha256_utf8":"67cfbc13f857c0d87a63221a86779db377f2013addb2d994b400ea2af0eab9e9"}]},{"id":"S052","semantic_type":"proof","kind":"proof","title":"内层光滑性界的完整证明","source_units":[{"start":64757,"end":66599,"start_line":875,"end_line":894,"raw_tex":"\\begin{proof}\nFor ease of description, we let $\\beta=\\beta_k$ and $(\\vy,\\vz)=(\\vy^k,\\vz^k)$ in the proof. First we notice that $\\frac{\\partial}{\\partial u}\\psi_\\beta(u,v)=[\\beta u+v]_+$, and thus for any $v$,\n$$\\left|\\frac{\\partial}{\\partial u}\\psi_\\beta(\\hat{u},v)-\\frac{\\partial}{\\partial u}\\psi_\\beta(\\tilde{u},v)\\right|\\le \\beta|\\hat{u}-\\tilde{u}|,\\,\\forall \\hat{u},\\tilde{u}.$$\nLet $h_i(x,z_i) = \\psi_\\beta(f_i(\\vx),z_i),\\,i=1,\\ldots,m$. Then\n\\begin{align*}\n&\\|\\nabla_\\vx h_i(\\hat{\\vx},z_i)-\\nabla_\\vx h_i(\\tilde{\\vx},z_i)\\|\\\\\n=&\\big\\|\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\hat{\\vx}),z_i)\\nabla f_i(\\hat{\\vx})-\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\tilde{\\vx}),z_i)\\nabla f_i(\\tilde{\\vx})\\big\\|\\\\\n\\le&\\big\\|\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\hat{\\vx}),z_i)\\nabla f_i(\\hat{\\vx})-\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\tilde{\\vx}),z_i)\\nabla f_i(\\hat{\\vx})\\big\\|\\\\\n&+\\big\\|\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\tilde{\\vx}),z_i)\\nabla f_i(\\hat{\\vx})-\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\tilde{\\vx}),z_i)\\nabla f_i(\\tilde{\\vx})\\big\\|\\\\\n\\le& \\beta|f_i(\\hat{\\vx})-f_i(\\tilde{\\vx})|\\cdot\\|\\nabla f_i(\\hat{\\vx})\\|+\\big|\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\tilde{\\vx}),z_i)\\big|\\cdot\\|\\nabla f_i(\\hat{\\vx})-\\nabla f_i(\\tilde{\\vx})\\|\\\\\n\\le & \\beta B_i^2\\|\\hat{\\vx}-\\tilde{\\vx}\\|+ L_i(\\beta B_i + |z_i|) \\|\\hat{\\vx}-\\tilde{\\vx}\\|.\n\\end{align*}\nHence,\n\\begin{align*}\n&\\|\\nabla_\\vx F_\\beta(\\hat{\\vx},\\vy,\\vz)-\\nabla_\\vx F_\\beta(\\tilde{\\vx},\\vy,\\vz)\\|\\\\\n\\le& \\|\\nabla g(\\hat{\\vx})-\\nabla g(\\tilde{\\vx})\\|+\\beta\\|\\vA^\\top \\vA(\\hat{\\vx}-\\tilde{\\vx})\\| + \\sum_{i=1}^m \\|\\nabla_\\vx h_i(\\hat{\\vx},z_i)-\\nabla_\\vx h_i(\\tilde{\\vx},z_i)\\|\\\\\n\\le & \\left(L_0+\\beta\\|\\vA^\\top \\vA\\|+\\sum_{i=1}^m \\big[\\beta B_i^2+ L_i(\\beta B_i + |z_i|)\\big]\\right)\\|\\hat{\\vx}-\\tilde{\\vx}\\|,\n\\end{align*}\nwhich completes the proof.\n\\end{proof}","sha256_utf8":"924fe6000bf5b3fa219c95d81d674d54acd93d2c04ec48f5b76e009590cec34c"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"计算标量惩罚函数的导数","source_units":[{"start":64757,"end":65140,"start_line":875,"end_line":877,"raw_tex":"\\begin{proof}\nFor ease of description, we let $\\beta=\\beta_k$ and $(\\vy,\\vz)=(\\vy^k,\\vz^k)$ in the proof. First we notice that $\\frac{\\partial}{\\partial u}\\psi_\\beta(u,v)=[\\beta u+v]_+$, and thus for any $v$,\n$$\\left|\\frac{\\partial}{\\partial u}\\psi_\\beta(\\hat{u},v)-\\frac{\\partial}{\\partial u}\\psi_\\beta(\\tilde{u},v)\\right|\\le \\beta|\\hat{u}-\\tilde{u}|,\\,\\forall \\hat{u},\\tilde{u}.$$\n","sha256_utf8":"f4f548fea2a6e44226ca37959543fa54e1bb733aab00fd6e1ff3b063aa498159"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"估计单个复合惩罚项的梯度差","source_units":[{"start":65140,"end":66132,"start_line":878,"end_line":886,"raw_tex":"Let $h_i(x,z_i) = \\psi_\\beta(f_i(\\vx),z_i),\\,i=1,\\ldots,m$. Then\n\\begin{align*}\n&\\|\\nabla_\\vx h_i(\\hat{\\vx},z_i)-\\nabla_\\vx h_i(\\tilde{\\vx},z_i)\\|\\\\\n=&\\big\\|\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\hat{\\vx}),z_i)\\nabla f_i(\\hat{\\vx})-\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\tilde{\\vx}),z_i)\\nabla f_i(\\tilde{\\vx})\\big\\|\\\\\n\\le&\\big\\|\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\hat{\\vx}),z_i)\\nabla f_i(\\hat{\\vx})-\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\tilde{\\vx}),z_i)\\nabla f_i(\\hat{\\vx})\\big\\|\\\\\n&+\\big\\|\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\tilde{\\vx}),z_i)\\nabla f_i(\\hat{\\vx})-\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\tilde{\\vx}),z_i)\\nabla f_i(\\tilde{\\vx})\\big\\|\\\\\n\\le& \\beta|f_i(\\hat{\\vx})-f_i(\\tilde{\\vx})|\\cdot\\|\\nabla f_i(\\hat{\\vx})\\|+\\big|\\frac{\\partial}{\\partial u}\\psi_\\beta(f_i(\\tilde{\\vx}),z_i)\\big|\\cdot\\|\\nabla f_i(\\hat{\\vx})-\\nabla f_i(\\tilde{\\vx})\\|\\\\\n\\le & \\beta B_i^2\\|\\hat{\\vx}-\\tilde{\\vx}\\|+ L_i(\\beta B_i + |z_i|) \\|\\hat{\\vx}-\\tilde{\\vx}\\|.\n\\end{align*}\n","sha256_utf8":"61af7dcb5103c47c58db000ea6b1a1d5e45387305b0ca2c221869812de7bbe52"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"累加各光滑项得到总常数","source_units":[{"start":66132,"end":66599,"start_line":887,"end_line":894,"raw_tex":"Hence,\n\\begin{align*}\n&\\|\\nabla_\\vx F_\\beta(\\hat{\\vx},\\vy,\\vz)-\\nabla_\\vx F_\\beta(\\tilde{\\vx},\\vy,\\vz)\\|\\\\\n\\le& \\|\\nabla g(\\hat{\\vx})-\\nabla g(\\tilde{\\vx})\\|+\\beta\\|\\vA^\\top \\vA(\\hat{\\vx}-\\tilde{\\vx})\\| + \\sum_{i=1}^m \\|\\nabla_\\vx h_i(\\hat{\\vx},z_i)-\\nabla_\\vx h_i(\\tilde{\\vx},z_i)\\|\\\\\n\\le & \\left(L_0+\\beta\\|\\vA^\\top \\vA\\|+\\sum_{i=1}^m \\big[\\beta B_i^2+ L_i(\\beta B_i + |z_i|)\\big]\\right)\\|\\hat{\\vx}-\\tilde{\\vx}\\|,\n\\end{align*}\nwhich completes the proof.\n\\end{proof}","sha256_utf8":"7e0255e4ae6f54ed1aba3ce83aa669b594e48d3f2a351bde2e2a18d15eb68087"}]},{"id":"S056","semantic_type":"theorem","kind":"theorem","title":"引用的加速近端收敛率","source_units":[{"start":36954,"end":37657,"start_line":527,"end_line":539,"raw_tex":"\\begin{theorem}\\label{thm:rate-apg}\nLet $\\{\\vx^k \\}$ be the sequence generated from Algorithm \\ref{alg:apg}. Assume $\\vx^*$ to be a minimizer of \\eqref{eq:comp-prob}. The following results holds:\n\\begin{enumerate}\n\\item If $\\mu=0$ and $\\alpha_0=1$, then\n\\begin{equation}\\label{eq:optimal-rate-cvx}\n\\phi(\\vx^k) + \\psi(\\vx^k) - \\phi(\\vx^*)-\\psi(\\vx^*) \\le \\frac{2L_\\phi\\|\\vx^0-\\vx^*\\|^2}{k^2},\\,\\forall k\\ge1.\n\\end{equation}\n\\item If $\\mu >0$ and $\\alpha_0=\\sqrt{\\frac{\\mu}{L_\\phi}}$, then\n\\begin{equation}\\label{eq:optimal-rate-scvx}\n\\phi(\\vx^k)-\\phi(\\vx^*)\\le \\frac{(L_\\phi+\\mu)\\|\\vx^0-\\vx^*\\|^2}{2}\\left(1-\\sqrt{\\frac{\\mu}{L_\\phi}}\\right)^k,\\,\\forall k\\ge1.\n\\end{equation}\n\\end{enumerate}\n\\end{theorem}","sha256_utf8":"ad5550fc04d08f38fec3f578fdce0509a5ab3b8ea1c82a69f8d5ae51ca3ca313"}]},{"id":"S057","semantic_type":"external_fact","kind":"external_fact","title":"加速近端法的标准收敛定理","source_units":[],"declared_external":true,"declaration_text":["Use the cited standard accelerated proximal gradient convergence theorem underlying B0045: the convex objective-gap bound is 2L times the squared initial distance divided by k squared; with positive strong convexity modulus, the method has the corresponding geometric contraction with factor 1-sqrt(mu/L). The source formulas and initialization conditions are retained in B0045."]},{"id":"S058","semantic_type":"lemma","kind":"lemma","title":"达到子问题精度的内层迭代数","source_units":[{"start":66972,"end":68326,"start_line":900,"end_line":919,"raw_tex":"\\begin{lemma}\\label{lem:sub-iter}\nAssume that $g$ is strongly convex with modulus $\\mu\\ge0$. Given $\\vareps_k>0$, if we start from $\\vx^k $ and run Algorithm \\ref{alg:apg}, then at most $t_k$ iterations are needed to produce $\\vx^{k+1}$ such that \\eqref{eq:ialm-x} holds,\nwhere \n\\begin{equation}\\label{eq:tk-cvx}\nt_k = \\left\\{\n\\begin{array}{ll}\n\\left\\lceil\\dfrac{\\dist(\\vx^k ,\\cX_k^*)\\sqrt{2L(\\vz^k  )}}{\\sqrt{\\vareps_k}}\\right\\rceil, & \\text{ if }\\mu=0,\\\\[0.5cm]\n\\left\\lceil\\dfrac{\\log\\left(\\frac{L(\\vz^k  )+\\mu}{2\\vareps_k}[\\dist(\\vx^k ,\\cX_k^*)]^2\\right)}{\\log1/\\big(1-\\sqrt{\\frac{\\mu}{L(\\vz^k  )}}\\big)}\\right\\rceil, & \\text{ if }\\mu>0,\n\\end{array}\n\\right.\n\\end{equation}\nand $\\cX_k^*$ denotes the set of optimal solutions to $\\min_{\\vx\\in \\cX}\\cL_{\\beta_k}(\\vx,\\vy^k ,\\vz^k  )$.\n%with\n%$$\\phi(\\vx) = F_\\beta(x,\\vy^k ,\\vz^k  ),\\quad \\psi(\\vx) = h(\\vx)+\\iota_\\cX(\\vx),$$ then we have\n%\\begin{equation}\\label{eq:ialm-xk-apg}\n%\\cL_\\beta(\\vx^{k+1},\\vy^k ,\\vz^k  )-\\min_{x\\in \\cX}\\cL_\\beta(x,\\vy^k ,\\vz^k  ) \\le \\frac{2L(\\vz^k  )[\\dist(\\vx^k ,\\cX_k^*)]^2}{t_k^2},\n%\\end{equation}\n%where $\\cX_k^*$ denotes the set of optimal solutions to $\\min_{x\\in \\cX}\\cL_\\beta(x,\\vy^k ,\\vz^k  )$. Hence, to guarantee a point $\\vx^{k+1}$ satisfying \\eqref{eq:ialm-x}, it suffices to evaluate the gradients of $g$ and $f_i, i\\in [m]$ for $t_k$ times, where \n \n\\end{lemma}","sha256_utf8":"42cdc1fca3a1e95ee09a2e94303516393c3b2a8921239583b60915e61d72b062"}]},{"id":"S059","semantic_type":"proof","kind":"proof","title":"内层迭代界的作者推导","source_units":[{"start":66601,"end":66971,"start_line":896,"end_line":898,"raw_tex":"Therefore, letting\n$\\phi(\\vx) = F_{\\beta_k}(\\vx,\\vy^k ,\\vz^k  )$ and $\\psi(\\vx) = h(\\vx)+\\iota_\\cX(\\vx),$\nwe can apply Nesterov's optimal first-order method in Algorithm \\ref{alg:apg} to find $\\vx^{k+1}$ in \\eqref{eq:ialm-x}. From Theorem \\ref{thm:rate-apg}, we have the following results. Note that if the strong convexity constant $\\mu=0$, the problem is just convex.\n","sha256_utf8":"876d71a42ea01d7e28ad33db0cc1b7d80462b0ec76a6cadfa24ef2da470cf68f"}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","title":"识别复合子问题并应用加速率","source_units":[{"start":66601,"end":66971,"start_line":896,"end_line":898,"raw_tex":"Therefore, letting\n$\\phi(\\vx) = F_{\\beta_k}(\\vx,\\vy^k ,\\vz^k  )$ and $\\psi(\\vx) = h(\\vx)+\\iota_\\cX(\\vx),$\nwe can apply Nesterov's optimal first-order method in Algorithm \\ref{alg:apg} to find $\\vx^{k+1}$ in \\eqref{eq:ialm-x}. From Theorem \\ref{thm:rate-apg}, we have the following results. Note that if the strong convexity constant $\\mu=0$, the problem is just convex.\n","sha256_utf8":"876d71a42ea01d7e28ad33db0cc1b7d80462b0ec76a6cadfa24ef2da470cf68f"}]},{"id":"S061","semantic_type":"lemma","kind":"lemma","title":"误差预算下的乘子与光滑性界","source_units":[{"start":71396,"end":72102,"start_line":957,"end_line":969,"raw_tex":"\\begin{lemma}\\label{lem:bd-z-Lz}\nLet $\\{(\\vx^k,\\vy^k,\\vz^k)\\}_{k=0}^K$ be the sequence generated from Algorithm \\ref{alg:ialm} with $\\{\\beta_k\\}$ and $\\{\\rho_k\\}$ set according to either Setting \\ref{set-penalty} or Setting \\ref{set-ialm}. If $\\vy^0=\\vzero, \\vz^0=\\vzero$, and $\\vareps_k$'s are chosen such that \n\\begin{equation}\\label{eq:req-eps}\n\\sum_{k=0}^{K-1}\\rho_k\\vareps_k\\le \\frac{C_\\vareps}{2},\n\\end{equation}\n then\n\\begin{align}\\label{eq:bd-Lzk}\nL(\\vz^k)\\le L_*+\\beta_k H,\\,\\forall \\, 0\\le k\\le K,\n\\end{align}\nwhere \n$$ H=\\|\\vA^\\top \\vA\\|+ \\sum_{i=1}^m  B_i(B_i+L_i),\\ L_*=L_0+ \\|\\vell\\| \\left(\\|\\vy^*\\|+2\\|\\vz^*\\|+\\sqrt{C_\\vareps}\\right)$$\nand $\\vell$ is given in \\eqref{eq:vec-ell}.\n\\end{lemma}","sha256_utf8":"df9e39514c89d25bdb9295f47d7de8919010bd54fa3e5b8bdd25f75ce95f4755"}]},{"id":"S062","semantic_type":"proof","kind":"proof","title":"乘子与光滑性界的完整证明","source_units":[{"start":72104,"end":73490,"start_line":971,"end_line":992,"raw_tex":"\\begin{proof}\nLetting $(\\vx,\\vy,\\vz)=(\\vx^*,\\vy^*,\\vz^*)$ in \\eqref{eq:ialm-ineq2-1} and using \\eqref{eq:opt}, we have \n$$\\frac{1}{2}\\|\\vy^{k+1} -\\vy^*\\|^2 + \\frac{1}{2}\\|\\vz^{k+1} -\\vz^*\\|^2\n\\le  \\frac{1}{2}\\|\\vy^k -\\vy^*\\|^2+ \\frac{1}{2}\\|\\vz^k  -\\vz^*\\|^2+\\rho_k\\vareps_k.$$\nSumming the above inequality yields \n$$\\frac{1}{2}\\|\\vy^{k}-\\vy^*\\|^2 + \\frac{1}{2}\\|\\vz^{k}-\\vz^*\\|^2\n\\le  \\frac{1}{2}\\|\\vy^0-\\vy^*\\|^2+ \\frac{1}{2}\\|\\vz^0-\\vz^*\\|^2+\\sum_{t=0}^{k-1}\\rho_t\\vareps_t, \\, \\forall \\, 0\\le k \\le K,$$\nwhich implies\n$$\\|\\vz^k  \\| \\le \\|\\vz^*\\|+\\sqrt{\\|\\vy^0-\\vy^*\\|^2+\\|\\vz^0-\\vz^*\\|^2+2\\sum_{t=0}^{k-1}\\rho_t\\vareps_t}.$$\nSince $\\|\\vu\\|\\le \\|\\vu\\|_1$ for any vector $\\vu$, we have from the above inequality that\n\\begin{equation}\\label{eq:bd-zk0}\n\\|\\vz^k  \\| \\le  \\|\\vz^*\\|+\\|\\vy^0-\\vy^*\\|+\\|\\vz^0-\\vz^*\\|+\\sqrt{2\\sum_{t=0}^{K-1}\\rho_t\\vareps_t},\\, \\forall \\, 0\\le k \\le K.\n\\end{equation}\n\nHence, if $\\vy^0=\\vzero$ and $\\vz^0=\\vzero$, and \\eqref{eq:req-eps} holds, it follows from the above inequality that\n\\begin{equation}\\label{eq:bd-zk}\n\\|\\vz^k  \\| \\le  \\|\\vy^*\\|+2\\|\\vz^*\\|+\\sqrt{C_\\vareps},\\, \\forall \\, 0\\le k \\le K,\n\\end{equation}\nBy the Cauchy-Schwartz inequality, we have from \\eqref{eq:def-Lz} that for any $0\\le k\\le K$,\n$$L(\\vz^k  ) \\le L_0+\\beta_k H + \\|\\vz^k  \\|\\cdot\\|\\vell\\|$$\nwhich together with \\eqref{eq:bd-zk} gives the result in \\eqref{eq:bd-Lzk}.\n\\end{proof}","sha256_utf8":"606f378c8b100e83f9eb7cf749863dba0f87d555bb2d1028eccda9bd8c580cb6"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"相对最优乘子累加距离递推","source_units":[{"start":72104,"end":72733,"start_line":971,"end_line":979,"raw_tex":"\\begin{proof}\nLetting $(\\vx,\\vy,\\vz)=(\\vx^*,\\vy^*,\\vz^*)$ in \\eqref{eq:ialm-ineq2-1} and using \\eqref{eq:opt}, we have \n$$\\frac{1}{2}\\|\\vy^{k+1} -\\vy^*\\|^2 + \\frac{1}{2}\\|\\vz^{k+1} -\\vz^*\\|^2\n\\le  \\frac{1}{2}\\|\\vy^k -\\vy^*\\|^2+ \\frac{1}{2}\\|\\vz^k  -\\vz^*\\|^2+\\rho_k\\vareps_k.$$\nSumming the above inequality yields \n$$\\frac{1}{2}\\|\\vy^{k}-\\vy^*\\|^2 + \\frac{1}{2}\\|\\vz^{k}-\\vz^*\\|^2\n\\le  \\frac{1}{2}\\|\\vy^0-\\vy^*\\|^2+ \\frac{1}{2}\\|\\vz^0-\\vz^*\\|^2+\\sum_{t=0}^{k-1}\\rho_t\\vareps_t, \\, \\forall \\, 0\\le k \\le K,$$\nwhich implies\n$$\\|\\vz^k  \\| \\le \\|\\vz^*\\|+\\sqrt{\\|\\vy^0-\\vy^*\\|^2+\\|\\vz^0-\\vz^*\\|^2+2\\sum_{t=0}^{k-1}\\rho_t\\vareps_t}.$$\n","sha256_utf8":"3163a1d5f1c73f1982629f9a1b6f9b1a4c7afa986941eebfe974d5f471c06c96"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"分离初始距离与误差预算","source_units":[{"start":72733,"end":72999,"start_line":980,"end_line":983,"raw_tex":"Since $\\|\\vu\\|\\le \\|\\vu\\|_1$ for any vector $\\vu$, we have from the above inequality that\n\\begin{equation}\\label{eq:bd-zk0}\n\\|\\vz^k  \\| \\le  \\|\\vz^*\\|+\\|\\vy^0-\\vy^*\\|+\\|\\vz^0-\\vz^*\\|+\\sqrt{2\\sum_{t=0}^{K-1}\\rho_t\\vareps_t},\\, \\forall \\, 0\\le k \\le K.\n\\end{equation}\n","sha256_utf8":"3238b5faf3637c0423eda67005930ebd896dc8ddaf4e120d3488f0f5660606c4"}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","title":"零初始乘子给出统一范数界","source_units":[{"start":73000,"end":73248,"start_line":985,"end_line":988,"raw_tex":"Hence, if $\\vy^0=\\vzero$ and $\\vz^0=\\vzero$, and \\eqref{eq:req-eps} holds, it follows from the above inequality that\n\\begin{equation}\\label{eq:bd-zk}\n\\|\\vz^k  \\| \\le  \\|\\vy^*\\|+2\\|\\vz^*\\|+\\sqrt{C_\\vareps},\\, \\forall \\, 0\\le k \\le K,\n\\end{equation}\n","sha256_utf8":"892a97d9a062ce894769b926d69c2d1151a507d37a3fc9eab1eadf9725ab2459"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"由乘子界控制光滑性常数","source_units":[{"start":73248,"end":73490,"start_line":989,"end_line":992,"raw_tex":"By the Cauchy-Schwartz inequality, we have from \\eqref{eq:def-Lz} that for any $0\\le k\\le K$,\n$$L(\\vz^k  ) \\le L_0+\\beta_k H + \\|\\vz^k  \\|\\cdot\\|\\vell\\|$$\nwhich together with \\eqref{eq:bd-zk} gives the result in \\eqref{eq:bd-Lzk}.\n\\end{proof}","sha256_utf8":"69916cf3a6e6cd84401fd17d8bc761dbcd18371066461e39ca7965ca81c4a870"}]},{"id":"S067","semantic_type":"derived_fact","kind":"derived_fact","title":"最优乘子给出的拉格朗日下界","source_units":[{"start":31829,"end":32065,"start_line":460,"end_line":463,"raw_tex":"From the convexity of $f_i$'s, if $(\\vx^*,\\vy^*,\\vz^*)$ is a KKT point, then\n\\begin{equation}\\label{eq:opt}\nf_0(\\vx)-f_0(\\vx^*)+\\langle \\vy^*, \\vA\\vx-\\vb\\rangle + \\sum_{i=1}^m z^*_i f_i(\\vx) \\ge 0,\\, \\forall\\, \\vx\\in\\cX.\n\\end{equation}\n","sha256_utf8":"a5f8f3577f9be8a0ca0b320aa30121b9471b74797efd9187a20469de1b444fb9"}]},{"id":"S068","semantic_type":"external_fact","kind":"external_fact","title":"有界域上光滑函数的标准估计","source_units":[],"declared_external":true,"declaration_text":["On a nonempty bounded convex domain, a Lipschitz continuous gradient is bounded after comparison with any fixed point. Integration along line segments then bounds function differences by the gradient bound and bounds function values after comparison with a fixed value."]},{"id":"S069","semantic_type":"assumption","kind":"assumption","title":"遍历分析的初始化与步长条件","source_units":[{"start":59319,"end":59549,"start_line":800,"end_line":800,"raw_tex":"Under Assumptions \\ref{assump-kkt} and \\ref{assump-wd-ialm}, let $\\{(\\vx^k ,\\vy^k ,\\vz^k  )\\}$ be the sequence generated from Algorithm \\ref{alg:ialm} with $\\vy^0=\\vzero, \\vz^0 = \\vzero$ and $0<\\rho_k\\le\\beta_k,\\,\\forall k$. Then\n","sha256_utf8":"b7ac6baf7bfe3d2d35c8d6c5fc341700f23f7d7f831d78160d2deee6c945ebde"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S006","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S042","target_id":"S003","kind":"supports"},{"source_id":"S058","target_id":"S003","kind":"supports"},{"source_id":"S058","target_id":"S005","kind":"supports"},{"source_id":"S058","target_id":"S006","kind":"supports"},{"source_id":"S061","target_id":"S003","kind":"supports"},{"source_id":"S061","target_id":"S006","kind":"supports"},{"source_id":"S019","target_id":"S003","kind":"supports"},{"source_id":"S019","target_id":"S006","kind":"supports"},{"source_id":"S022","target_id":"S003","kind":"supports"},{"source_id":"S022","target_id":"S006","kind":"supports"},{"source_id":"S008","target_id":"S004","kind":"supports"},{"source_id":"S008","target_id":"S006","kind":"supports"},{"source_id":"S021","target_id":"S003","kind":"supports"},{"source_id":"S012","target_id":"S001","kind":"supports"},{"source_id":"S007","target_id":"S001","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S027","target_id":"S024","kind":"supports"},{"source_id":"S028","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S067","target_id":"S027","kind":"supports"},{"source_id":"S067","target_id":"S028","kind":"supports"},{"source_id":"S011","target_id":"S026","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S010","target_id":"S031","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S035","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S010","target_id":"S035","kind":"supports"},{"source_id":"S011","target_id":"S034","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S037","kind":"supports"},{"source_id":"S040","target_id":"S037","kind":"supports"},{"source_id":"S041","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S010","target_id":"S038","kind":"supports"},{"source_id":"S017","target_id":"S038","kind":"supports"},{"source_id":"S009","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S029","target_id":"S039","kind":"supports"},{"source_id":"S010","target_id":"S039","kind":"supports"},{"source_id":"S009","target_id":"S040","kind":"supports"},{"source_id":"S034","target_id":"S040","kind":"supports"},{"source_id":"S010","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S032","target_id":"S041","kind":"supports"},{"source_id":"S009","target_id":"S041","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"},{"source_id":"S046","target_id":"S043","kind":"supports"},{"source_id":"S047","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S036","target_id":"S044","kind":"supports"},{"source_id":"S069","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S016","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S069","target_id":"S046","kind":"supports"},{"source_id":"S007","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S023","target_id":"S047","kind":"supports"},{"source_id":"S016","target_id":"S047","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S048","kind":"supports"},{"source_id":"S008","target_id":"S050","kind":"supports"},{"source_id":"S068","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S055","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S009","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S048","target_id":"S054","kind":"supports"},{"source_id":"S008","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S008","target_id":"S055","kind":"supports"},{"source_id":"S018","target_id":"S055","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S014","target_id":"S056","kind":"supports"},{"source_id":"S015","target_id":"S056","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S060","kind":"supports"},{"source_id":"S051","target_id":"S060","kind":"supports"},{"source_id":"S018","target_id":"S060","kind":"supports"},{"source_id":"S008","target_id":"S060","kind":"supports"},{"source_id":"S011","target_id":"S060","kind":"supports"},{"source_id":"S010","target_id":"S060","kind":"supports"},{"source_id":"S063","target_id":"S062","kind":"supports"},{"source_id":"S064","target_id":"S062","kind":"supports"},{"source_id":"S065","target_id":"S062","kind":"supports"},{"source_id":"S066","target_id":"S062","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S044","target_id":"S063","kind":"supports"},{"source_id":"S067","target_id":"S063","kind":"supports"},{"source_id":"S016","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S051","target_id":"S066","kind":"supports"},{"source_id":"S011","target_id":"S066","kind":"supports"},{"source_id":"S019","target_id":"S061","kind":"supports"},{"source_id":"S020","target_id":"S061","kind":"supports"},{"source_id":"S013","target_id":"S067","kind":"supports"},{"source_id":"S007","target_id":"S067","kind":"supports"}]}}
{"record_id":"1803.02924v4","paper":{"title":"A Newton-CG Algorithm with Complexity Guarantees for Smooth Unconstrained Optimization","authors":["Clément W. Royer","Michael O'Neill","Stephen J. Wright"],"arxiv_id":"1803.02924","version":"v4","journal":"Mathematical Programming","doi":"10.1007/s10107-019-01362-7","abstract_url":"https://arxiv.org/abs/1803.02924v4","pdf_url":"https://arxiv.org/pdf/1803.02924v4"},"context":{"raw_tex":"We consider the unconstrained optimization problem\n\\begin{equation} \\label{eq:f}\n\\min_{x \\in \\R^n} \\, f(x),\n\\end{equation}\n\nwhere $f:\\R^n \\to \\R$ is a twice Lipschitz continuously differentiable\nfunction that is nonconvex in general. We further assume that $f$ is\nbounded below for all $x$, by some constant $\\flow$. Although the\n\n\nOur algorithm seeks a point that approximately satisfies second-order\nnecessary conditions for optimality, that is,\n\\begin{equation}\\label{eq:2epson}\n\\| \\nabla f (x)\\| \\le \\epsg, \\quad \\lambda_{\\min}(\\nabla^2 f(x)) \\ge\n-\\epsH,\n\\end{equation}\n\nfor specified small positive tolerances $\\epsg$ and $\\epsH$. (Here and\nsubsequently, $\\| \\cdot \\|$ denotes the Euclidean norm, or its induced\nnorms on matrices.)\n\n\n\\begin{assumption} \\label{assum:compactlevelset}\nThe level set $\\mathcal{L}_f(x_0) = \\{x | f(x) \\le f(x_0)\\}$ is\ncompact.\n\\end{assumption}\n\n\\begin{assumption} \\label{assum:fC22}\nThe function $f$ is twice uniformly Lipschitz continuously\ndifferentiable on an open neighborhood of $\\mathcal{L}_f(x_0)$\n\\refer{that includes the trial points generated by the algorithm. We\n  denote by $L_H$ the Lipschitz constant for $\\nabla^2 f$ on this\n  neighborhood}.\n%% \\crcomment{Removed $L_g$ because it is not used anywhere else in\n%% the paper and we already have multiple notations for Hessian norms\n%% or their approximations. We only need Lipschitz continuous\n%% Hessians, and bounded gradients/Hessians (the latter implying that\n%% the gradient is Lipschitz continuous).}\n%\\sjwcomment{I made changes. OK now? CR: Ok.}\n\\end{assumption}\n\nUnder these two assumptions, there exist scalars $\\flow$, $U_g\n> 0$, and $U_H > 0$ such that the following are satisfied for $x \\in\n\\mathcal{L}_f(x_0)$:\n\\begin{equation}\\label{eq:boundscompact}\n\tf(x) \\geq \\flow,\\quad \\|\\nabla f(x)\\| \\leq U_g,\n\t\\quad \\|\\nabla^2 f(x)\\| \\leq U_H.\n\\end{equation}\n\nFor any $x$ and $d$ such that Assumption~\\ref{assum:fC22} is satisfied at $x$ \nand $x+d$, we have\n\\begin{equation} \\label{eq:LH}\nf(x+d) \\le f(x) + \\nabla f(x)^\\top d + \\frac12 d^\\top \\nabla^2 f(x) d\n+ \\frac{L_H}{6} \\| d\\|^3.\n\\end{equation}\n\nNotationally, we use order notation ${\\mathcal O}$ in the usual sense,\nwhereas $\\tcO(\\cdot)$ represents $\\cO$ with logarithmic terms\n\\refer{omitted}.\n\n\n\\begin{algorithm}[ht!]\n\\caption{Capped Conjugate Gradient}\n\\label{alg:ccg}\n\\begin{algorithmic}\n\\STATE \\emph{Inputs:} Symmetric matrix $H \\in \\R^{n \\times n}$; \nvector $g \\ne 0$; damping parameter $\\epsilon \\in (0,1)$; desired relative \naccuracy \\refer{$\\zeta \\in (0,1)$};\n\\STATE \\refer{\\emph{Optional input:} scalar $M$ (set to $0$ if not provided)};\n\\STATE \\refer{\\emph{Outputs:} d\\_type, $d$;}\n\\STATE \\refer{\\emph{Secondary outputs:} final values of $M$, $\\kappa$, \n$\\hat{\\zeta}$, $\\tau$, and $T$;}\n\\STATE Set\n\\[\n\\bH:=H+2\\eps I, \\quad \\kappa := \\frac{M+2\\eps}{\\eps}, \\quad \\hat{\\zeta} := \\frac{\\zeta}{3 \\kappa},\n\\quad\n\\tau := \\frac{\\sqrt{\\kappa}}{\\sqrt{\\kappa}+1}, \\quad\nT:=\\frac{4\\kappa^4}{(1-\\sqrt{\\tau})^2};\n\\]\n\\STATE $y_0 \\leftarrow 0$, $r_0 \\leftarrow g$, $p_0 \\leftarrow -g$, $j \\leftarrow 0$; \n\\IF{$p_0^\\top \\bH p_0 < \\eps \\|p_0\\|^2$}\n\\STATE Set $d=p_0$ and terminate with d\\_type=NC;\n\\ELSIF{\\refer{$\\|H p_0\\| > M \\|p_0\\|$}}\n\\STATE \\refer{Set $M \\leftarrow {\\|H p_0\\|}/{\\|p_0\\|}$ \nand update $\\kappa,\\hat{\\zeta},\\tau,T$ accordingly;}\n\\ENDIF\n\\WHILE{TRUE}\n\\STATE $\\alpha_j \\leftarrow {r_j^\\top r_j}/{p_j^\\top \\bH p_j}$;\n\\COMMENT{Begin Standard CG Operations}\n\\STATE $y_{j+1} \\leftarrow y_j+\\alpha_j p_j$;\n\\STATE $r_{j+1} \\leftarrow r_j + \\alpha_j \\bH p_j$;\n\\STATE $\\beta_{j+1} \\leftarrow {(r_{j+1}^\\top r_{j+1})}/{(r_j^\\top r_j)}$;\n\\STATE $p_{j+1} \\leftarrow  -r_{j+1} + \\beta_{j+1}p_j$;\n\\COMMENT{End Standard CG Operations}\n\\STATE $j \\leftarrow  j+1$;\n\\IF{\\refer{$\\|H p_j\\| > M \\|p_j\\|$}}\n\\STATE \\refer{Set $M \\leftarrow {\\|H p_j\\|}/{\\|p_j\\|}$ \nand update $\\kappa,\\hat{\\zeta},\\tau,T$ accordingly;}\n\\ENDIF\n\\IF{\\refer{$\\|H y_j\\| > M \\|y_j\\|$}}\n\\STATE \\refer{Set $M \\leftarrow {\\|H y_j\\|}/{\\|y_j\\|}$ \nand update $\\kappa,\\hat{\\zeta},\\tau,T$ accordingly;}\n\\ENDIF\n\\IF{\\refer{$\\| H r_j\\| > M \\|r_j\\|$}}\n\\STATE \\refer{Set $M  \\leftarrow {\\| H r_j\\|}/{\\|r_j\\|}$ \nand update $\\kappa,\\hat{\\zeta},\\tau,T$ accordingly;}\n\\ENDIF\n\\IF{$y_j^\\top \\bH y_j < \\epsilon \\|y_j\\|^2$}\n\\STATE Set $d \\leftarrow y_j$ and terminate with d\\_type=NC;\n%\\ELSIF{$\\| r_j \\| \\le \\tfrac{1}{2}\\min\\{\\zeta\\|r_0\\|,\\eps \\|y_j\\|\\}$}\n\\ELSIF{$\\| r_j \\| \\le \\hat{\\zeta} \\|r_0\\|$}\n\\STATE Set $d \\leftarrow y_j$ and terminate with d\\_type=SOL;\n\\ELSIF{$p_j^\\top \\bH p_j < \\eps \\|p_j\\|^2$}\n\\STATE Set $d \\leftarrow p_j$ and terminate with d\\_type=NC;\n\\ELSIF{$\\|r_j\\| >  \\sqrt{T}\\tau^{j/2} \\|r_0\\|$\n%\\crnote{The key here is \n%that when we terminate because of this condition, we are sure that \n%$\\|H v\\| \\le (M+2\\eps)\\|v\\|$ for $v\\in \\{p_0,\\dots,p_j,y_0,\\dots,y_j,r_0,\\dots,r_j\\}$ \n%and this is all we need. As long as the residual norm decreases at the \n%desired rate, things are okay because $\\sqrt{T}\\tau^{j/2}$ is an \n%increasing function of $M$.}\n}\n\\STATE Compute $\\alpha_j,y_{j+1}$ as in the main loop above;\n\\STATE Find $i \\in \\{0,\\dotsc,j-1\\}$ such that\n\\begin{equation} \\label{eq:weakcurvdir}\n\t\\frac{(y_{j+1}-y_i)^\\top \\bH (y_{j+1}-y_i)}{\\|y_{j+1}-y_i\\|^2} \\; < \\; \\eps;\n\\end{equation}\n\\STATE Set $d \\leftarrow y_{j+1}-y_i$ and terminate with d\\_type=NC;\n\\ENDIF\n\\ENDWHILE\n\\end{algorithmic}\n\\end{algorithm}\n\n\\begin{algorithm}[ht!]\n\\caption{Minimum Eigenvalue Oracle}\n\\label{alg:meo}\n\\begin{algorithmic}\n%  \\STATE \\emph{Inputs:} Symmetric matrix $H \\in \\R^{n \\times n}$,\n%  scalar $M$ such that $\\lambdamax(H) \\le M$, and $\\epsilon>0$;\n%  \\STATE Set parameter $\\delta \\in [0,1)$;\n  \\STATE \\emph{Inputs:} Symmetric matrix $H \\in \\R^{n \\times n}$,\n  \\refer{tolerance $\\epsilon>0$};\n  \\STATE \\refer{\\emph{Optional input:} Upper bound on Hessian norm \n  $M$;}\n  \\STATE \\emph{Outputs:} An estimate $\\lambda$ of $\\lambdamin(H)$\n  such that $\\lambda \\le - \\epsilon/2$, and vector $v$\n  with $\\|v\\|=1$ such that $v^\\top H v =\\lambda$ OR\n  a certificate that $\\lambdamin(H) \\ge -\\epsilon$. In the latter case, when\n  the certificate is output, it is false with probability $\\delta$ \n  \\refer{for some $\\delta \\in [0,1)$}.\n\\end{algorithmic}\n\\end{algorithm}\n\n\\begin{algorithm}[ht!]\n\\caption{Damped Newton-CG}\n\\label{alg:dncg}\n\\begin{algorithmic}\n\\STATE \\emph{Inputs:} Tolerances  $\\epsg>0$, $\\epsH>0$; backtracking \nparameter $\\theta \\in (0,1)$; starting point $x_0$; \n\\refer{accuracy parameter $\\zeta \\in (0,1)$;}\n\\STATE \\refer{\\emph{Optional input:} Upper bound  $M > 0$ on Hessian \nnorm};\n%\\STATE \\refer{\\emph{Optional input:} Upper bound  $U_H > 0$ on Hessian \n%norm}; \n\\FOR{$k=0,1,2,\\dotsc$}\n\\IF{$\\|\\nabla f(x_k)\\| > \\epsg$}\n\\STATE Call Algorithm~\\ref{alg:ccg} with $H = \\nabla^2 f(x_k)$, $\\epsilon=\\epsH$, \n$g=\\nabla f(x_k)$, accuracy parameter $\\zeta$ and $M$ if provided, \nto obtain outputs $d$, d\\_type; \n\\IF{d\\_type=NC}\n\\STATE $d_k \\leftarrow -\\mathrm{sgn}(d^\\top g) \\frac{|d^\\top \\nabla^2 f(x_k) d|}{\\|d\\|^2} \n\\frac{d}{\\|d\\|}$;\n\\ELSE[\\refer{d\\_type=SOL}]\n\\STATE $d_k \\leftarrow d$;\n\\ENDIF\n\\STATE Go to \\textbf{Line Search};\n\\ELSE\n\\STATE Call Procedure~\\ref{alg:meo} with $H = \\nabla^2 f(x_k)$,\n$\\epsilon=\\epsH$ and $M$ if provided; \n\\IF{Procedure~\\ref{alg:meo} certifies that $\\lambdamin(\\nabla^2 f(x_k)) \\ge -\\epsH$}\n\\STATE\nTerminate; \n\\ELSE[\\refer{direction of sufficient negative curvature found}]\n\\STATE \n$d_k \\leftarrow -\\mathrm{sgn}(v^\\top g) \\frac{|v^\\top \\nabla^2 f(x_k) v|}{\\|v\\|^2} v$\n% \\sjwnote{Holy smokes, the last $v$ was missing from this formula!}\n(where $v$ is the output from Procedure~\\ref{alg:meo})\nand go to \\textbf{Line Search};\n\\ENDIF\n\\ENDIF\n\\STATE \\textbf{Line Search}: Compute a step length $\\alpha_k=\\theta^{j_k}$, \nwhere $j_k$ is the smallest nonnegative integer such that\n\\begin{equation} \\label{eq:lsdecreasedamped}\n\tf(x_k + \\alpha_k d_k) < f(x_k) - \\frac{\\eta}{6}\\alpha_k^3 \\|d_k\\|^{3};\n\\end{equation}\n\\STATE $x_{k+1} \\leftarrow x_k+\\alpha_k d_k$;\n\\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}\n\n\\begin{assumption} \\label{assum:wccmeo}\nFor every iteration $k$ at which Algorithm~\\ref{alg:dncg} calls\nProcedure~\\ref{alg:meo}, and for a specified failure probability\n$\\delta$ with $0 \\le \\delta \\ll 1$, Procedure~\\ref{alg:meo} either\ncertifies that $\\nabla^2 f(x_k) \\succeq -\\epsH I$ or finds a vector\nof curvature smaller than $-{\\epsH}/{2}$ in at most\n%% \\crcomment{Changed the value to reflect our new bounds on the\n%% number of iterations.}\n\\begin{equation} \\label{eq:wccmeo}\n\t\tN_{\\mathrm{meo}}:=\\min\\left\\{n,\n\t\t1+\\left\\lceil\\Cmeo\\epsH^{-1/2}\\right\\rceil\\right\\}\n\\end{equation}\nHessian-vector products, \\refer{with probability $1-\\delta$,} where\n$\\Cmeo$ depends at most logarithmically on $\\delta$ and $\\epsH$.\n%\\sjwcomment{I changed some more. Is this OK? CR: Yes.}\n%% \\crcomment{Modified the last part of the assumption to cope with \n%% the discussion before Corollary~\\ref{coro:wcc2Hv}, where $\\Cmeo$ depends \n%% on $\\epsH$ through $\\delta$. That also avoids mentioning this logarithmic \n%% dependency in the statement of the corollary.}\n\\end{assumption}\n\n\\begin{algorithm}[ht!]\n\\caption{Conjugate Gradient}\n\\label{alg:cg}\n\\begin{algorithmic}\n  \\STATE \\emph{Inputs:} Symmetric matrix $\\bH$, vector $g$;\n  \\STATE $r_0 \\leftarrow g$, $p_0 \\leftarrow -r_0$, $y_0 \\leftarrow 0$, $j \\leftarrow 0$;\n  \\WHILE{$p_j^\\top\\bH p_j>0$ and $r_j \\ne 0$}\n  \\STATE $\\alpha_j \\leftarrow {r_j^\\top r_j}/{p_j^\\top \\bH p_j}$;\n\\STATE $y_{j+1} \\leftarrow y_j+\\alpha_j p_j$;\n\\STATE $r_{j+1} \\leftarrow r_j + \\alpha_j \\bH p_j$;\n\\STATE $\\beta_{j+1} \\leftarrow {(r_{j+1}^\\top r_{j+1})}/{(r_j^\\top r_j)}$;\n\\STATE $p_{j+1} \\leftarrow -r_{j+1} + \\beta_{j+1}p_j$;\n\\STATE $j \\leftarrow j+1$;\n\\ENDWHILE\n\\end{algorithmic}\n\\end{algorithm}\n\nHere and below, we refer often to the following quadratic function:\n\\begin{equation} \\label{eq:quadraticCG}\n\tq(y) := \\frac{1}{2}y^\\top \\bH y + g^\\top y,\n\\end{equation}\n\nwhere $\\bH$ and $g$ are the matrix and vector parameters of\nAlgorithms~\\ref{alg:ccg} or~\\ref{alg:cg}. When $\\bH$ is positive\ndefinite, the minimizer of $q$ is identical to the unique solution of\n$\\bH y = -g$.  CG can be viewed either as an algorithm to solve $\\bH y\n= -g$ or as an algorithm to minimize $q$.\n","units":[{"start":7042,"end":7164,"start_line":215,"end_line":218,"raw_tex":"We consider the unconstrained optimization problem\n\\begin{equation} \\label{eq:f}\n\\min_{x \\in \\R^n} \\, f(x),\n\\end{equation}","sha256_utf8":"5b65f7ee2a844b7be1d9e235485067d21a64897237b7942efc6c77700e2d397f"},{"start":7165,"end":7371,"start_line":219,"end_line":221,"raw_tex":"where $f:\\R^n \\to \\R$ is a twice Lipschitz continuously differentiable\nfunction that is nonconvex in general. We further assume that $f$ is\nbounded below for all $x$, by some constant $\\flow$. Although the\n","sha256_utf8":"787292d971d208b8d82caab930631ab50c785dde78f53a4e5d353597bb6ed81f"},{"start":12138,"end":12379,"start_line":300,"end_line":305,"raw_tex":"Our algorithm seeks a point that approximately satisfies second-order\nnecessary conditions for optimality, that is,\n\\begin{equation}\\label{eq:2epson}\n\\| \\nabla f (x)\\| \\le \\epsg, \\quad \\lambda_{\\min}(\\nabla^2 f(x)) \\ge\n-\\epsH,\n\\end{equation}","sha256_utf8":"3d19eb2c78393fab5cae74a7b0ed115c8082e9571391f926c40379df3fff5d10"},{"start":12380,"end":12542,"start_line":306,"end_line":308,"raw_tex":"for specified small positive tolerances $\\epsg$ and $\\epsH$. (Here and\nsubsequently, $\\| \\cdot \\|$ denotes the Euclidean norm, or its induced\nnorms on matrices.)\n","sha256_utf8":"10421431718a7dbb43039a023bb4b4a5fe0a6bd5e734fee23c1e3080c5ac6fec"},{"start":12686,"end":12824,"start_line":313,"end_line":316,"raw_tex":"\\begin{assumption} \\label{assum:compactlevelset}\nThe level set $\\mathcal{L}_f(x_0) = \\{x | f(x) \\le f(x_0)\\}$ is\ncompact.\n\\end{assumption}","sha256_utf8":"f49684da37a22ee987182f54e9fca4aab9f4af77c51586342bf07ea730169526"},{"start":12826,"end":13513,"start_line":318,"end_line":330,"raw_tex":"\\begin{assumption} \\label{assum:fC22}\nThe function $f$ is twice uniformly Lipschitz continuously\ndifferentiable on an open neighborhood of $\\mathcal{L}_f(x_0)$\n\\refer{that includes the trial points generated by the algorithm. We\n  denote by $L_H$ the Lipschitz constant for $\\nabla^2 f$ on this\n  neighborhood}.\n%% \\crcomment{Removed $L_g$ because it is not used anywhere else in\n%% the paper and we already have multiple notations for Hessian norms\n%% or their approximations. We only need Lipschitz continuous\n%% Hessians, and bounded gradients/Hessians (the latter implying that\n%% the gradient is Lipschitz continuous).}\n%\\sjwcomment{I made changes. OK now? CR: Ok.}\n\\end{assumption}","sha256_utf8":"bb44a0c307759cc1c3bf6e92abcedc923eca43e79d3279b672174725a3dedce9"},{"start":13696,"end":13988,"start_line":336,"end_line":342,"raw_tex":"Under these two assumptions, there exist scalars $\\flow$, $U_g\n> 0$, and $U_H > 0$ such that the following are satisfied for $x \\in\n\\mathcal{L}_f(x_0)$:\n\\begin{equation}\\label{eq:boundscompact}\n\tf(x) \\geq \\flow,\\quad \\|\\nabla f(x)\\| \\leq U_g,\n\t\\quad \\|\\nabla^2 f(x)\\| \\leq U_H.\n\\end{equation}","sha256_utf8":"108e94175ccac0986892b5dd3bd319461945fe51073324c2912e73c7e27d9555"},{"start":14247,"end":14486,"start_line":348,"end_line":353,"raw_tex":"For any $x$ and $d$ such that Assumption~\\ref{assum:fC22} is satisfied at $x$ \nand $x+d$, we have\n\\begin{equation} \\label{eq:LH}\nf(x+d) \\le f(x) + \\nabla f(x)^\\top d + \\frac12 d^\\top \\nabla^2 f(x) d\n+ \\frac{L_H}{6} \\| d\\|^3.\n\\end{equation}","sha256_utf8":"eda4a236202c7244a7c3f550626de5f3d7c2f99d961181b395cb8331b909b3e1"},{"start":14488,"end":14638,"start_line":355,"end_line":357,"raw_tex":"Notationally, we use order notation ${\\mathcal O}$ in the usual sense,\nwhereas $\\tcO(\\cdot)$ represents $\\cO$ with logarithmic terms\n\\refer{omitted}.\n","sha256_utf8":"9cc933a06214c0e51df80021cafb5f79a83a5b49dfb18417627b0c437d323b6c"},{"start":25078,"end":28150,"start_line":544,"end_line":614,"raw_tex":"\\begin{algorithm}[ht!]\n\\caption{Capped Conjugate Gradient}\n\\label{alg:ccg}\n\\begin{algorithmic}\n\\STATE \\emph{Inputs:} Symmetric matrix $H \\in \\R^{n \\times n}$; \nvector $g \\ne 0$; damping parameter $\\epsilon \\in (0,1)$; desired relative \naccuracy \\refer{$\\zeta \\in (0,1)$};\n\\STATE \\refer{\\emph{Optional input:} scalar $M$ (set to $0$ if not provided)};\n\\STATE \\refer{\\emph{Outputs:} d\\_type, $d$;}\n\\STATE \\refer{\\emph{Secondary outputs:} final values of $M$, $\\kappa$, \n$\\hat{\\zeta}$, $\\tau$, and $T$;}\n\\STATE Set\n\\[\n\\bH:=H+2\\eps I, \\quad \\kappa := \\frac{M+2\\eps}{\\eps}, \\quad \\hat{\\zeta} := \\frac{\\zeta}{3 \\kappa},\n\\quad\n\\tau := \\frac{\\sqrt{\\kappa}}{\\sqrt{\\kappa}+1}, \\quad\nT:=\\frac{4\\kappa^4}{(1-\\sqrt{\\tau})^2};\n\\]\n\\STATE $y_0 \\leftarrow 0$, $r_0 \\leftarrow g$, $p_0 \\leftarrow -g$, $j \\leftarrow 0$; \n\\IF{$p_0^\\top \\bH p_0 < \\eps \\|p_0\\|^2$}\n\\STATE Set $d=p_0$ and terminate with d\\_type=NC;\n\\ELSIF{\\refer{$\\|H p_0\\| > M \\|p_0\\|$}}\n\\STATE \\refer{Set $M \\leftarrow {\\|H p_0\\|}/{\\|p_0\\|}$ \nand update $\\kappa,\\hat{\\zeta},\\tau,T$ accordingly;}\n\\ENDIF\n\\WHILE{TRUE}\n\\STATE $\\alpha_j \\leftarrow {r_j^\\top r_j}/{p_j^\\top \\bH p_j}$;\n\\COMMENT{Begin Standard CG Operations}\n\\STATE $y_{j+1} \\leftarrow y_j+\\alpha_j p_j$;\n\\STATE $r_{j+1} \\leftarrow r_j + \\alpha_j \\bH p_j$;\n\\STATE $\\beta_{j+1} \\leftarrow {(r_{j+1}^\\top r_{j+1})}/{(r_j^\\top r_j)}$;\n\\STATE $p_{j+1} \\leftarrow  -r_{j+1} + \\beta_{j+1}p_j$;\n\\COMMENT{End Standard CG Operations}\n\\STATE $j \\leftarrow  j+1$;\n\\IF{\\refer{$\\|H p_j\\| > M \\|p_j\\|$}}\n\\STATE \\refer{Set $M \\leftarrow {\\|H p_j\\|}/{\\|p_j\\|}$ \nand update $\\kappa,\\hat{\\zeta},\\tau,T$ accordingly;}\n\\ENDIF\n\\IF{\\refer{$\\|H y_j\\| > M \\|y_j\\|$}}\n\\STATE \\refer{Set $M \\leftarrow {\\|H y_j\\|}/{\\|y_j\\|}$ \nand update $\\kappa,\\hat{\\zeta},\\tau,T$ accordingly;}\n\\ENDIF\n\\IF{\\refer{$\\| H r_j\\| > M \\|r_j\\|$}}\n\\STATE \\refer{Set $M  \\leftarrow {\\| H r_j\\|}/{\\|r_j\\|}$ \nand update $\\kappa,\\hat{\\zeta},\\tau,T$ accordingly;}\n\\ENDIF\n\\IF{$y_j^\\top \\bH y_j < \\epsilon \\|y_j\\|^2$}\n\\STATE Set $d \\leftarrow y_j$ and terminate with d\\_type=NC;\n%\\ELSIF{$\\| r_j \\| \\le \\tfrac{1}{2}\\min\\{\\zeta\\|r_0\\|,\\eps \\|y_j\\|\\}$}\n\\ELSIF{$\\| r_j \\| \\le \\hat{\\zeta} \\|r_0\\|$}\n\\STATE Set $d \\leftarrow y_j$ and terminate with d\\_type=SOL;\n\\ELSIF{$p_j^\\top \\bH p_j < \\eps \\|p_j\\|^2$}\n\\STATE Set $d \\leftarrow p_j$ and terminate with d\\_type=NC;\n\\ELSIF{$\\|r_j\\| >  \\sqrt{T}\\tau^{j/2} \\|r_0\\|$\n%\\crnote{The key here is \n%that when we terminate because of this condition, we are sure that \n%$\\|H v\\| \\le (M+2\\eps)\\|v\\|$ for $v\\in \\{p_0,\\dots,p_j,y_0,\\dots,y_j,r_0,\\dots,r_j\\}$ \n%and this is all we need. As long as the residual norm decreases at the \n%desired rate, things are okay because $\\sqrt{T}\\tau^{j/2}$ is an \n%increasing function of $M$.}\n}\n\\STATE Compute $\\alpha_j,y_{j+1}$ as in the main loop above;\n\\STATE Find $i \\in \\{0,\\dotsc,j-1\\}$ such that\n\\begin{equation} \\label{eq:weakcurvdir}\n\t\\frac{(y_{j+1}-y_i)^\\top \\bH (y_{j+1}-y_i)}{\\|y_{j+1}-y_i\\|^2} \\; < \\; \\eps;\n\\end{equation}\n\\STATE Set $d \\leftarrow y_{j+1}-y_i$ and terminate with d\\_type=NC;\n\\ENDIF\n\\ENDWHILE\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"3d966fbb58aaa7d9d655f157cb9c2d1d2ad30b57408f0d7bcd0a6e0c2259e293"},{"start":37931,"end":38768,"start_line":803,"end_line":821,"raw_tex":"\\begin{algorithm}[ht!]\n\\caption{Minimum Eigenvalue Oracle}\n\\label{alg:meo}\n\\begin{algorithmic}\n%  \\STATE \\emph{Inputs:} Symmetric matrix $H \\in \\R^{n \\times n}$,\n%  scalar $M$ such that $\\lambdamax(H) \\le M$, and $\\epsilon>0$;\n%  \\STATE Set parameter $\\delta \\in [0,1)$;\n  \\STATE \\emph{Inputs:} Symmetric matrix $H \\in \\R^{n \\times n}$,\n  \\refer{tolerance $\\epsilon>0$};\n  \\STATE \\refer{\\emph{Optional input:} Upper bound on Hessian norm \n  $M$;}\n  \\STATE \\emph{Outputs:} An estimate $\\lambda$ of $\\lambdamin(H)$\n  such that $\\lambda \\le - \\epsilon/2$, and vector $v$\n  with $\\|v\\|=1$ such that $v^\\top H v =\\lambda$ OR\n  a certificate that $\\lambdamin(H) \\ge -\\epsilon$. In the latter case, when\n  the certificate is output, it is false with probability $\\delta$ \n  \\refer{for some $\\delta \\in [0,1)$}.\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"ecd8559f46df395c10d42d6abc1f049f145c34bfc10835d5572e407c193559e9"},{"start":47561,"end":49332,"start_line":1001,"end_line":1046,"raw_tex":"\\begin{algorithm}[ht!]\n\\caption{Damped Newton-CG}\n\\label{alg:dncg}\n\\begin{algorithmic}\n\\STATE \\emph{Inputs:} Tolerances  $\\epsg>0$, $\\epsH>0$; backtracking \nparameter $\\theta \\in (0,1)$; starting point $x_0$; \n\\refer{accuracy parameter $\\zeta \\in (0,1)$;}\n\\STATE \\refer{\\emph{Optional input:} Upper bound  $M > 0$ on Hessian \nnorm};\n%\\STATE \\refer{\\emph{Optional input:} Upper bound  $U_H > 0$ on Hessian \n%norm}; \n\\FOR{$k=0,1,2,\\dotsc$}\n\\IF{$\\|\\nabla f(x_k)\\| > \\epsg$}\n\\STATE Call Algorithm~\\ref{alg:ccg} with $H = \\nabla^2 f(x_k)$, $\\epsilon=\\epsH$, \n$g=\\nabla f(x_k)$, accuracy parameter $\\zeta$ and $M$ if provided, \nto obtain outputs $d$, d\\_type; \n\\IF{d\\_type=NC}\n\\STATE $d_k \\leftarrow -\\mathrm{sgn}(d^\\top g) \\frac{|d^\\top \\nabla^2 f(x_k) d|}{\\|d\\|^2} \n\\frac{d}{\\|d\\|}$;\n\\ELSE[\\refer{d\\_type=SOL}]\n\\STATE $d_k \\leftarrow d$;\n\\ENDIF\n\\STATE Go to \\textbf{Line Search};\n\\ELSE\n\\STATE Call Procedure~\\ref{alg:meo} with $H = \\nabla^2 f(x_k)$,\n$\\epsilon=\\epsH$ and $M$ if provided; \n\\IF{Procedure~\\ref{alg:meo} certifies that $\\lambdamin(\\nabla^2 f(x_k)) \\ge -\\epsH$}\n\\STATE\nTerminate; \n\\ELSE[\\refer{direction of sufficient negative curvature found}]\n\\STATE \n$d_k \\leftarrow -\\mathrm{sgn}(v^\\top g) \\frac{|v^\\top \\nabla^2 f(x_k) v|}{\\|v\\|^2} v$\n% \\sjwnote{Holy smokes, the last $v$ was missing from this formula!}\n(where $v$ is the output from Procedure~\\ref{alg:meo})\nand go to \\textbf{Line Search};\n\\ENDIF\n\\ENDIF\n\\STATE \\textbf{Line Search}: Compute a step length $\\alpha_k=\\theta^{j_k}$, \nwhere $j_k$ is the smallest nonnegative integer such that\n\\begin{equation} \\label{eq:lsdecreasedamped}\n\tf(x_k + \\alpha_k d_k) < f(x_k) - \\frac{\\eta}{6}\\alpha_k^3 \\|d_k\\|^{3};\n\\end{equation}\n\\STATE $x_{k+1} \\leftarrow x_k+\\alpha_k d_k$;\n\\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"98429eed679a23c4caf32899829f9acaca9dcfed0cd6c94adf9cff69a1c864f6"},{"start":70592,"end":71655,"start_line":1543,"end_line":1562,"raw_tex":"\\begin{assumption} \\label{assum:wccmeo}\nFor every iteration $k$ at which Algorithm~\\ref{alg:dncg} calls\nProcedure~\\ref{alg:meo}, and for a specified failure probability\n$\\delta$ with $0 \\le \\delta \\ll 1$, Procedure~\\ref{alg:meo} either\ncertifies that $\\nabla^2 f(x_k) \\succeq -\\epsH I$ or finds a vector\nof curvature smaller than $-{\\epsH}/{2}$ in at most\n%% \\crcomment{Changed the value to reflect our new bounds on the\n%% number of iterations.}\n\\begin{equation} \\label{eq:wccmeo}\n\t\tN_{\\mathrm{meo}}:=\\min\\left\\{n,\n\t\t1+\\left\\lceil\\Cmeo\\epsH^{-1/2}\\right\\rceil\\right\\}\n\\end{equation}\nHessian-vector products, \\refer{with probability $1-\\delta$,} where\n$\\Cmeo$ depends at most logarithmically on $\\delta$ and $\\epsH$.\n%\\sjwcomment{I changed some more. Is this OK? CR: Yes.}\n%% \\crcomment{Modified the last part of the assumption to cope with \n%% the discussion before Corollary~\\ref{coro:wcc2Hv}, where $\\Cmeo$ depends \n%% on $\\epsH$ through $\\delta$. That also avoids mentioning this logarithmic \n%% dependency in the statement of the corollary.}\n\\end{assumption}","sha256_utf8":"4bb5593da9042da8bc842258bf109167816e55cf4d9d876d1c44d95edd790aa9"},{"start":88305,"end":88952,"start_line":1915,"end_line":1930,"raw_tex":"\\begin{algorithm}[ht!]\n\\caption{Conjugate Gradient}\n\\label{alg:cg}\n\\begin{algorithmic}\n  \\STATE \\emph{Inputs:} Symmetric matrix $\\bH$, vector $g$;\n  \\STATE $r_0 \\leftarrow g$, $p_0 \\leftarrow -r_0$, $y_0 \\leftarrow 0$, $j \\leftarrow 0$;\n  \\WHILE{$p_j^\\top\\bH p_j>0$ and $r_j \\ne 0$}\n  \\STATE $\\alpha_j \\leftarrow {r_j^\\top r_j}/{p_j^\\top \\bH p_j}$;\n\\STATE $y_{j+1} \\leftarrow y_j+\\alpha_j p_j$;\n\\STATE $r_{j+1} \\leftarrow r_j + \\alpha_j \\bH p_j$;\n\\STATE $\\beta_{j+1} \\leftarrow {(r_{j+1}^\\top r_{j+1})}/{(r_j^\\top r_j)}$;\n\\STATE $p_{j+1} \\leftarrow -r_{j+1} + \\beta_{j+1}p_j$;\n\\STATE $j \\leftarrow j+1$;\n\\ENDWHILE\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"e161a46a7e22a03fbfccc89380d66a713df82133d63f45b8f249bbbc6ad226fb"},{"start":88954,"end":89121,"start_line":1932,"end_line":1935,"raw_tex":"Here and below, we refer often to the following quadratic function:\n\\begin{equation} \\label{eq:quadraticCG}\n\tq(y) := \\frac{1}{2}y^\\top \\bH y + g^\\top y,\n\\end{equation}","sha256_utf8":"1381ae9faddbbab8d9a0989ffe6aa1848b0dbe7e167962e73daf87ca234f0754"},{"start":89122,"end":89430,"start_line":1936,"end_line":1940,"raw_tex":"where $\\bH$ and $g$ are the matrix and vector parameters of\nAlgorithms~\\ref{alg:ccg} or~\\ref{alg:cg}. When $\\bH$ is positive\ndefinite, the minimizer of $q$ is identical to the unique solution of\n$\\bH y = -g$.  CG can be viewed either as an algorithm to solve $\\bH y\n= -g$ or as an algorithm to minimize $q$.\n","sha256_utf8":"c9cbc469c2f3fe0ac45c9fea767a0acd1433c040ff3eb95e2cbc733ef35091f5"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{corollary} \\label{coro:wcc2Hv}\n  Suppose that assumptions of Theorem~\\ref{theo:wccits} hold, and let\n  $\\bar{K}_2$ be defined as in \\eqref{eq:wccits}.  \\refer{Suppose that\n    the values of $M$ used or calculated at each instance of\n    Algorithm~\\ref{alg:ccg} satisfy $M \\le U_H$.} Then with\n  probability at least $(1-\\delta)^{\\bar{K}_2}$,\n  Algorithm~\\ref{alg:dncg} terminates\n%  at least $1-\\bar{K}_2 \\delta$, Algorithm~\\ref{alg:dncg} terminates\n  at a point satisfying \\eqref{eq:2epson} after at most\n  \\[\n  \\left(\\max \\{ 2\\min\\{n,J(U_H,\\epsH,\\zeta)\\}+2,N_{\\mathrm{meo}} \\} \\right) \n  \\bar{K}_2\n%   = \\tcO\\left( \\max\\left\\{\\epsg^{-3} \\epsH^{5/2},\\epsH^{-7/2} \n%  \\right\\}\\right)\n  \\]\n  Hessian-vector products and/or gradient evaluations. (With\n  probability at most $1-(1-\\delta)^{\\bar{K}_2}$, it terminates incorrectly\n%%  within this number of Hessian-vector products and/or gradient\n%% evaluations\n  with this complexity at a point for which $\\|\\nabla f(x_k) \\| \\le\n  \\epsg$ but $\\lambda_{\\min}(\\nabla^2 f(x)) < -\\epsH$.)\n  \n  \\refer{For $n$ sufficiently large, and assuming that \n  $\\delta < 1/\\bar{K}_2$, \n%  $\\Cmeo$ depending at\n%    most logarithmically on $\\epsH$ and $\\epsg$, \n    the bound is\\\\\n    $\\tcO\\left( \\max\\left\\{\\epsg^{-3} \\epsH^{5/2},\\epsH^{-7/2}\n    \\right\\}\\right)$, with probability at most $1-\\bar{K}_2 \\delta$.} \n%% while if $J(U_H,\\epsH,\\zeta) \\ge n$ and $N_{\\mathrm{meo}}=n$, the\n%%   bound is in $\\tcO\\left( n\\,\\max\\left\\{\\epsg^{-3}\n%%   \\epsH^{3},\\epsH^{-3}\\right\\} \\right)$.}\n\\end{corollary}","units":[{"start":81996,"end":83531,"start_line":1785,"end_line":1817,"raw_tex":"\\begin{corollary} \\label{coro:wcc2Hv}\n  Suppose that assumptions of Theorem~\\ref{theo:wccits} hold, and let\n  $\\bar{K}_2$ be defined as in \\eqref{eq:wccits}.  \\refer{Suppose that\n    the values of $M$ used or calculated at each instance of\n    Algorithm~\\ref{alg:ccg} satisfy $M \\le U_H$.} Then with\n  probability at least $(1-\\delta)^{\\bar{K}_2}$,\n  Algorithm~\\ref{alg:dncg} terminates\n%  at least $1-\\bar{K}_2 \\delta$, Algorithm~\\ref{alg:dncg} terminates\n  at a point satisfying \\eqref{eq:2epson} after at most\n  \\[\n  \\left(\\max \\{ 2\\min\\{n,J(U_H,\\epsH,\\zeta)\\}+2,N_{\\mathrm{meo}} \\} \\right) \n  \\bar{K}_2\n%   = \\tcO\\left( \\max\\left\\{\\epsg^{-3} \\epsH^{5/2},\\epsH^{-7/2} \n%  \\right\\}\\right)\n  \\]\n  Hessian-vector products and/or gradient evaluations. (With\n  probability at most $1-(1-\\delta)^{\\bar{K}_2}$, it terminates incorrectly\n%%  within this number of Hessian-vector products and/or gradient\n%% evaluations\n  with this complexity at a point for which $\\|\\nabla f(x_k) \\| \\le\n  \\epsg$ but $\\lambda_{\\min}(\\nabla^2 f(x)) < -\\epsH$.)\n  \n  \\refer{For $n$ sufficiently large, and assuming that \n  $\\delta < 1/\\bar{K}_2$, \n%  $\\Cmeo$ depending at\n%    most logarithmically on $\\epsH$ and $\\epsg$, \n    the bound is\\\\\n    $\\tcO\\left( \\max\\left\\{\\epsg^{-3} \\epsH^{5/2},\\epsH^{-7/2}\n    \\right\\}\\right)$, with probability at most $1-\\bar{K}_2 \\delta$.} \n%% while if $J(U_H,\\epsH,\\zeta) \\ge n$ and $N_{\\mathrm{meo}}=n$, the\n%%   bound is in $\\tcO\\left( n\\,\\max\\left\\{\\epsg^{-3}\n%%   \\epsH^{3},\\epsH^{-3}\\right\\} \\right)$.}\n\\end{corollary}","sha256_utf8":"abf9e360059c8b6d3efcad4dcef138c40a1b0a0de9afafb8e75e839f30682889"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma} \\label{lemma:ccgits}\nThe number of iterations of Algorithm~\\ref{alg:ccg} is bounded by\n\\[\n\\min\\{n,J(M,\\eps,\\zeta)\\},\n\\]\nwhere $J=J(M,\\eps,\\zeta)$ is the smallest integer such that\n$\\sqrt{T}\\tau^{J/2} \\le \\hat{\\zeta}$ \\refer{where $M$, $\\hat\\zeta$,\n  $T$, and $\\tau$ are the values returned by the algorithm}. \\refer{If\n  all iterates $y_i$ generated by the algorithm are stored, the number\n  of matrix-vector multiplications required is bounded by\n\\begin{equation} \\label{eq:pv8}\n\t\\min\\{n,J(M,\\eps,\\zeta)\\}+1.\n\\end{equation}\nIf the iterates $y_i$ must be regenerated in order to define the\ndirection $d$ returned after \\eqref{eq:weakcurvdir}, this bound\nbecomes $2\\min\\{n,J(M,\\eps,\\zeta)\\}+1$.}\n\\end{lemma}\n\n\\begin{proof}\n  If the full $n$ iterations are performed, without any of the\n  termination conditions being tripped, the standard properties of CG\n  (see Appendix~\\ref{app:cgresults}) ensure that the final residual\n  $r_n$ is zero, so that the condition $\\|r_n \\| \\le \\hat\\zeta\n  \\|r_0\\|$ is satisfied, and termination occurs.\n\n  \\refer{Since no more than $n$ iterations are performed, the upper\n    bound $M$ is updated at most a finite number of times, so the\n    quantity $J$ is well defined.}\n\n  Supposing that $J<n$, we note from the definition of $J$ that\n  $\\sqrt{T}\\tau^{J/2} \\|r_0 \\| \\le \\hat\\zeta \\|r_0\\|$. Thus at least\n  one of the following two conditions must be satisfied at iteration\n  $J$: $\\|r_{J} \\| \\le \\hat\\zeta \\|r_0\\|$ or $\\|r_{J} \\| >\n  \\sqrt{T}\\tau^{J/2} \\|r_0\\|$. In either case, termination will occur\n  at iteration $J$, unless it has occurred already at a previous\n  iteration.\n\n  To derive \\eqref{eq:pv8}, note that the main workload at each\n  iteration $j$ is computation of a single matrix-vector product\n%  \\refer{$H p_j$ (since matrix-vector products involving the matrices\n%    $H$ and $\\bH$ and the vectors $y_j$ and $r_j$ can be computed in\n%    terms of this vector, in an additional $O(n)$ operations).}\n%  \\refer{(The ``$+1$'' in \\eqref{eq:pv8} is from the additional\n%    matrix-vector multiplication required in the case of $\\|r_{J} \\| >\n%    \\sqrt{T}\\tau^{J/2} \\|r_0\\|$.)}\n  \\refer{$H p_j$ after the increment of $j$ (since \n    matrix-vector products involving the matrices\n    $H$ and $\\bH$ and the vectors $y_j$ and $r_j$ can be computed in\n    terms of this vector, in an additional $O(n)$ operations).}\n  \\refer{(The ``$+1$'' in \\eqref{eq:pv8} accounts for the initial\n    matrix-vector multiplication $H p_0$ performed prior to entering \n    the loop.)}\n%    \\crcomment{I think the new tests based on $M$ changed the \n%    places in which we compute the $H p_j$, so I corrected this \n%    paragraph. The result still holds, but the +1 relates to the \n%    initial step (before the loop). Note that the four termination \n%    tests do not require additional matrix-vector products since $Hp_j$ \n%    has already been computed for the $M$-related tests.\n%    MO: Changed account to accounts.}\n\n  If we do not store additional information, we need to regenerate the\n  information needed to compute the direction $d$ satisfying\n  \\eqref{eq:weakcurvdir} by re-running the iterations of CG, possibly\n  up to the \\refer{second-to-last} iteration. This fact accounts for\n  the additional cost of $\\min\\{n,J(M,\\eps,\\zeta)\\}$ in the no-storage\n  case. \\footnote{\\refer{Interestingly, as we show in\n      Appendix~\\ref{app:cgresults}, the ratios on the left-hand side\n      of \\eqref{eq:weakcurvdir} can be calculated without knowledge of\n      $y_i$ for $i=0,1,\\dotsc,j-1$, provided that we store the scalar\n      quantities $\\alpha_i$ and $\\|r_i\\|^2$ for $i=0,1,\\dotsc,j-1$.}}\n\\end{proof}\n\n\\refer{Note that $J(M,\\epsilon,\\zeta)$ is an increasing function of\n  $M$, since $\\hat{\\zeta}$ is a decreasing function of $M$, while $T$\n  and $\\tau$ (and thus $\\sqrt{T}\\tau^{j/2}$) are increasing in $M$. If\n  $U_H$ is known in advance, we can call Algorithm~\\ref{alg:ccg} with\n  $M = U_H$ and use $J(U_H,\\epsilon,\\zeta)$ as the bound. Alternately,\n  we can call Algorithm~\\ref{alg:ccg} with $M=0$ and let it adjust $M$\n  as needed during the computation. Since the final value of $M$ will\n  be at most $U_H$, and since $J(M,\\epsilon,\\zeta)$ is an increasing\n  function of $M$, the quantity $J(U_H,\\epsilon,\\zeta)$ provides the\n  upper bound on the number of iterations in this case too.}\n\nWe can estimate $J$ by taking logs in its definition, as follows:\n\\[\nJ \\le \\frac{2\\ln(\\hat{\\zeta}/\\sqrt{T})}{\\ln(\\tau)} =\n\\frac{\\ln(\\hat{\\zeta}^2/T)}\n     {\\ln\\left(\\frac{\\sqrt{\\kappa}}{\\sqrt{\\kappa}+1}\\right)} =\n     \\frac{\\ln(T/\\hat{\\zeta}^2)}{\\ln(1+{1}/{\\sqrt{\\kappa}})} \\le\n     \\left(\\sqrt{\\kappa}+\\frac{1}{2}\\right)\n     \\ln\\left(\\frac{T}{\\hat{\\zeta}^2}\\right),\n\\]\nwhere we used $\\ln(1+\\tfrac{1}{t}) \\ge \\tfrac{1}{t+1/2}$ to obtain the\nlatest inequality. By\nreplacing $T,\\tau,\\hat{\\zeta},\\kappa$ by their\ndefinitions in Algorithm~\\ref{alg:ccg}, and using\n$\n\\frac{1}{1-\\sqrt{\\tau}} = \\frac{1+\\sqrt{\\tau}}{1-\\tau} \\le\n\\frac{2}{1-\\tau},\n$\nwe obtain\n\\begin{align} \\nonumber\n%% J(M,\\eps,\\zeta) &  \\le\n%% \\min\\left\\{n,\\left\\lceil \\left(\\sqrt{\\frac{M+2\\eps}{\\eps}}+\\frac{1}{2}\\right)\n%% \\ln\\left(\\frac{144 \\left(\\sqrt{\\frac{M+2\\eps}{\\eps}}+1\\right)^2\n%%   \\left(\\frac{M+2\\eps}{\\eps}\\right)^6} {\\zeta^2}\\right) \\right\\rceil \\right\\} \\\\\nJ(M,\\eps,\\zeta) &  \\le\n\\refer{\\min\\left\\{n,\\left\\lceil \\left(\\sqrt{\\kappa}+\\frac{1}{2}\\right)\n\\ln\\left(\\frac{144 \\left(\\sqrt{\\kappa}+1\\right)^2\n  \\kappa^6} {\\zeta^2}\\right) \\right\\rceil \\right\\}} \\\\\n\\label{eq:HvcountcappedCG}\n & = \\min \\left\\{ n , \\tcO(\\eps^{-1/2}) \\right\\}.\n\\end{align}\n\n\n\\begin{theorem} \\label{theo:cvCGwhilestronglycvx}\nSuppose that the main loop of Algorithm~\\ref{alg:ccg} terminates with\n$j=\\hIcg$, where\n\\[\n\\hIcg \\in \\{1,\\dotsc,\\min\\{n,J(M,\\eps,\\zeta)\\}\\},\n\\]\n(where $J(M,\\eps,\\zeta)$ is defined in Lemma~\\ref{lemma:ccgits} and\n\\eqref{eq:HvcountcappedCG}) \\refer{because the fourth termination test is\nsatisfied and the three earlier conditions do not hold, that is, \n$y_{\\hIcg}^\\top \\bH y_{\\hIcg} \\ge \\eps \\|y_{\\hIcg}\\|^2$, \n$p_{\\hIcg}^\\top \\bH p_{\\hIcg} \\ge \\eps \\|p_{\\hIcg}\\|^2$, and\n\\begin{equation} \\label{eq:Kitsstronglycvx}\n\\|r_{\\hIcg}\\| > \\max\\{\\hat{\\zeta},\\sqrt{T}\\tau^{\\hIcg /2}\\}\\|r_0\\|.\n\\end{equation}\nwhere $M$, $T$, $\\hat{\\zeta}$, and $\\tau$ are the values returned by\nAlgorithm~\\ref{alg:ccg}. Then\n$y_{\\hIcg+1}$ is computed by Algorithm~\\ref{alg:ccg}, and we have}\n\\begin{equation} \\label{eq:yKp1negcurv}\n\\frac{(y_{\\hIcg +1}-y_i)^\\top \\bH (y_{\\hIcg+1}-y_i)}\n{\\|y_{\\hIcg+1}-y_i\\|^2} < \\eps, \\quad \\mbox{for some $i \\in \\{0,\\dotsc,\\hIcg-1\\}$.}\n\\end{equation}\n\\end{theorem}\n\n\\begin{lemma} \\label{lemma:ccgsteps}\nLet Assumptions~\\ref{assum:compactlevelset} and~\\ref{assum:fC22} hold,\nand suppose that Algorithm~\\ref{alg:ccg} is invoked at an iterate\n$x_k$ of Algorithm~\\ref{alg:dncg} (so that $\\|\\nabla f(x_k)\\| >\n\\epsg>0$).\nLet $d_k$ be the vector obtained in Algorithm~\\ref{alg:dncg} from the\noutput $d$ of Algorithm~\\ref{alg:ccg}. Then, one of the two following\nstatements holds:\n\\begin{enumerate}\n\\item {\\em d\\_type=SOL}, and the direction $d_k$ satisfies \n\\begin{subequations} \\label{eq:ccgstepSOL} \n\\begin{align}\n\\label{eq:ccgstepSOLcurv}\nd_k^\\top (\\nabla^2 f(x_k)+2\\epsH I) d_k & \\ge \\epsH \\|d_k\\|^2, \\\\\n\\label{eq:ccgstepSOLnorm}\n% \\|d_k\\| & \\le \\epsH^{-1} \\sqrt{1+\\zeta^2/4} \\|\\nabla f(x_k)\\|,\\\\\n\\|d_k\\| & \\le \\refer{1.1 \\epsH^{-1} \\|\\nabla f(x_k)\\|,}\\\\\n\\label{eq:ccgstepSOLres}\n%% \\|\\hat{r}_k \\| \\le  \n%% \\frac{\\hat{\\zeta}}{1-\\hat{\\zeta}}(U_H+2\\epsH)\\|d_k\\| & \\le \n%% \\frac{1}{2}\\epsH\\zeta \\| d_k \\|,\n\\|\\hat{r}_k \\|  & \\le  \\refer{\\frac{1}{2}\\epsH\\zeta \\| d_k \\|,}\n\\end{align}\n\\end{subequations}\nwhere\n\\begin{equation} \\label{eq:def.rhat}\n\\hat{r}_k := (\\nabla^2 f(x_k)+2\\epsH I) d_k+ \\nabla f(x_k);\n\\end{equation}\n\\item {\\em d\\_type=NC}, and the direction $d_k$ satisfies \n  $d_k^\\top \\nabla f(x_k) \\le 0$ as well as\n  \\refer{\n  \\begin{equation} \\label{eq:ccgstepNC}\n    \\frac{d_k^\\top \\nabla^2 f(x_k) d_k}{\\|d_k\\|^2} = -\\|d_k \\| \\le -\\epsH.\n  \\end{equation}\n  }\n%% \\begin{subequations} \\label{eq:ccgstepNC}\n%% \\begin{align}\n%% \\label{eq:ccgstepNCcurv}\n%% \\frac{d_k^\\top \\nabla^2 f(x_k) d_k}{\\|d_k\\|^2} \\le -\\epsH,\\\\\n%% \\label{eq:ccgstepNCnorm}\n%% \\|d_k\\| \\ge \\epsH.\n%% \\end{align}\n%% \\end{subequations}\n\\end{enumerate}\n\\end{lemma}\n\n\\begin{proof}\n\\refer{For simplicity of notation, we use $H=\\nabla^2 f(x_k)$ and\n  $g=\\nabla f(x_k)$ in the proof.}  Suppose first that d\\_type=SOL. In\nthat case, we have from the termination conditions in\nAlgorithm~\\ref{alg:ccg} and \\eqref{eq:def.rhat} that\n\\begin{subequations} \\label{eq:propSOLstep}\n\\begin{align} \n\\label{eq:propSOLstepCURV}\nd_k^\\top (H+2 \\epsH I)d_k & \\ge \\epsH \\|d_k\\|^2, \\\\\n\\label{eq:propSOLstepRES}\n\\|\\hat{r}_k \\| &  \\le \\hat{\\zeta} \\|g\\|,\n\\end{align}\n\\end{subequations}\n\\refer{where $\\hat\\zeta$ was returned by the algorithm.}\nWe immediately recognize~\\eqref{eq:ccgstepSOLcurv}\nin~\\eqref{eq:propSOLstepCURV}.  We now\nprove~\\eqref{eq:ccgstepSOLnorm}. Observe first \nthat \\eqref{eq:propSOLstepCURV} yields\n\\[\n\\epsH\\|d_k\\|^2 \\le d_k^\\top (H+2 \\epsH I) d_k \\le \\|d_k\\| \\|(H+2\\epsH\nI) d_k\\|,\n\\]\nso from \\eqref{eq:def.rhat} we have\n\\begin{equation*}\n\\|d_k\\| \\le \\epsH^{-1} \\|(H+2\\epsH I)d_k\\| = \\epsH^{-1} \\|-g+\\hat{r}_k\\| \n= \\epsH^{-1} \\sqrt{\\|g\\|^2 + \\| \\hat{r}_k \\|^2} \\le \n\\epsH^{-1} \\sqrt{1+\\hat{\\zeta}^2}  \\|g\\|,\n\\end{equation*}\nwhere we used~\\eqref{eq:propSOLstepRES} to obtain the final bound,\ntogether with the equality $\\|-g+\\hat{r}_k\\|^2=\\|g\\|^2+\\| \\hat{r}_k\n\\|^2$, which follows from $g^\\top \\hat{r}_k = r_0^\\top \\hat{r}_k =\n0$, by orthogonality of the residuals in CG (see\nLemma~\\ref{lemma:propertiesCGoneiter}, Property 2). Since\n\\refer{$\\hat{\\zeta} \\le \\zeta/(3 \\kappa) \\le 1/6$ by construction, we\n  have $\\|d_k \\| \\le \\sqrt{37/36} \\epsH^{-1} \\|g\\| \\le 1.1 \\epsH^{-1}\n  \\|g\\|$, proving \\eqref{eq:ccgstepSOLnorm}.}\n\nThe bound \\eqref{eq:ccgstepSOLres} follows\nfrom~\\eqref{eq:propSOLstepRES} and the logic below:\n\\begin{align*}\n\t\\| \\hat{r}_k \\| &\\le \\hat{\\zeta} \\|g\\| \\le \n\t\\hat{\\zeta}\\left(\\|(H+2 \\epsH I) d_k \\| + \n\t\\| \\hat{r}_k \\|\\right)  \\le \\hat\\zeta \\left( (M+2\\epsH) \\|d_k \\| +  \\| \\hat{r}_k \\| \\right) \\\\\n        \\Rightarrow \\;\\; \\| \\hat{r}_k \\| &\\le \n\t\\frac{\\hat{\\zeta}}{1-\\hat{\\zeta}}(M+2 \\epsH) \\|d_k\\|,\n\\end{align*}\n\\refer{where $M$ is the value returned by the algorithm.}\nWe finally use \\refer{$\\hat\\zeta < 1/6$}\n%% \\[\n%% \\hat\\zeta = \\frac{\\zeta}{3}\\frac{\\epsH}{M+2\\epsH} \\le\n%% \\frac{\\zeta}{3} \\frac12 < \\frac16,\n%% \\]\nto arrive at\n\\[\n\\frac{\\hat\\zeta}{1-\\hat{\\zeta}} (M+2\\epsH) \\le \\frac65 \\hat{\\zeta}\n(M+2\\epsH) = \\frac65 \\frac{\\zeta \\epsH}{3} < \\frac12 \\zeta \\epsH,\n\\]\nyielding  \\eqref{eq:ccgstepSOLres}.\n\nIn the case of d\\_type=NC, we recall that Algorithm~\\ref{alg:dncg}\ndefines\n\\begin{equation} \\label{eq:scaleccgNC}\nd_k = -\\mathrm{sgn}(d^\\top g) \\frac{|d^\\top H\n  d|}{\\|d\\|^2} \\frac{d}{\\|d\\|}\n\\end{equation}\nwhere $d$ denotes the direction obtained by\nAlgorithm~\\ref{alg:ccg}. It follows immediately that $d_k^\\top g \\le\n0$. Since $d_k$ and $d$ are collinear, we also have that\n\\refer{\n\\[\n\\frac{d_k^\\top (H+2\\epsH I)d_k}{\\|d_k\\|^2} = \\frac{d^\\top (H+2\\epsH\n  I)d}{\\|d\\|^2} \\le \\epsH \\; \\Rightarrow \\; \\frac{d_k^\\top H\n  d_k}{\\|d_k\\|^2}\n% = \\frac{d^\\top H d}{\\|d\\|^2}\n\\le -\\epsH.\n\\]\nBy using this bound together with \\eqref{eq:scaleccgNC}, we obtain\n\\[\n\\|d_k\\| = \\frac{|d^\\top H d|}{\\|d\\|^2} = \\frac{|d_k^\\top H d_k|}{\\|d_k\\|^2} =\n-\\frac{d_k^\\top H d_k}{\\|d_k\\|^2} \\ge \\epsH,\n\\]\nproving \\eqref{eq:ccgstepNC}.\n}\n%% establishing \\eqref{eq:ccgstepNCcurv}.  By \\eqref{eq:scaleccgNC}, we\n%% then have\n%% \\[\n%% \\|d_k\\| = \\frac{|d^\\top H d|}{\\|d\\|^2} \\ge \\epsH,\n%% \\]\n%% proving \\eqref{eq:ccgstepNCnorm}.\n\\end{proof}\n\n\\begin{lemma} \\label{lemma:LSccgSOL}\nSuppose that Assumptions~\\ref{assum:compactlevelset}\nand~\\ref{assum:fC22} hold. Suppose that at iteration $k$ of\nAlgorithm~\\ref{alg:dncg}, we have $\\|\\nabla f(x_k)\\| > \\epsg$, so\nthat Algorithm~\\ref{alg:ccg} is called. When Algorithm~\\ref{alg:ccg}\noutputs a direction $d_k$ with d\\_type=SOL, then the backtracking line\nsearch requires at most $j_k \\le \\jsol+1$ iterations, where\n\\begin{equation} \\label{eq:ccglsitsSOL}\n\\jsol \\; = \\; \\left[ \\frac{1}{2}\\log_{\\theta}\\left(\n  \\frac{3(1-\\zeta)}{L_H+\\eta} \\frac{\\epsH^2}{\\refer{1.1} U_g} \\right)\n  \\right]_+,\n\\end{equation}\nand the resulting step $x_{k+1} = x_k + \\alpha_k d_k$ satisfies\n\\begin{equation} \\label{eq:ccglsdecreaseSOL}\nf(x_k) - f(x_{k+1}) \\; \\ge \\; \n\\csol\\min\\left(\\|\\nabla f(x_{k+1})\\|^3 \\epsH^{-3},\\epsH^3\\right),\n\\end{equation}\nwhere \n\\begin{equation*}\n\\csol = \\frac{\\eta}{6}\\min\\left\\{ \\left[ \\frac{4}{\\sqrt{(4+\\zeta)^2+8 L_H}+4+\\zeta}\n  \\right]^3, \\left[ \\frac{3\\theta^2 (1-\\zeta) }{L_H+\\eta}\n  \\right]^3\\right\\}.\n\\end{equation*}\n\\end{lemma}\n\n\\begin{proof}\nThe proof tracks closely that\nof~\\cite[Lemma~13]{CWRoyer_SJWright_2018}. The only significant\ndifference is that equation (65) of~\\cite{CWRoyer_SJWright_2018},\nwhich is instrumental to the proof and requires a probabilistic\nassumption on $\\lambda_{\\min}(\\nabla^2 f(x_k))$, is now ensured\n\\emph{deterministically} by~\\eqref{eq:ccgstepSOLnorm} from\nLemma~\\ref{lemma:ccgsteps}. As a result, both the proof and the result\nare deterministic.\n\\end{proof}\n\n\\begin{lemma} \\label{lemma:LSccgNC}\n  Suppose that Assumptions~\\ref{assum:compactlevelset}\n  and~\\ref{assum:fC22} hold.  Suppose that at iteration $k$ of\n  Algorithm~\\ref{alg:dncg}, we have $\\|\\nabla f(x_k)\\| > \\epsg$, so\n  that Algorithm~\\ref{alg:ccg} is called. When Algorithm~\\ref{alg:ccg}\n  outputs d\\_type=NC, the direction $d_k$ (computed from $d$ in\n  Algorithm~\\ref{alg:dncg}) has the following properties: The\n  backtracking line search terminates with step length $\\alpha_k =\n  \\theta^{j_k}$ with $j_k \\le \\jnc +1$, where\n\\begin{equation} \\label{eq:eiglsits}\t\n\\jnc := \\left[ \\log_{\\theta}\\left( \\frac{3}{L_H+\\eta} \\right)\n  \\right]_+,\n\\end{equation}\nand the resulting step $x_{k+1} = x_k + \\alpha_k d_k$ satisfies\n\\begin{equation} \\label{eq:decreaseNC}\nf(x_k) - f(x_k+\\alpha_k\\,d_k) \\; \\geq \\; \\cnc \\epsH^3,\n\\end{equation}\t\nwith\n\\begin{equation*}\n\\cnc := \\frac{\\eta}{6}\n\\min\\left\\{1,\\frac{27\\theta^3}{(L_H+\\eta)^3}\\right\\}.\n\\end{equation*}\n\\end{lemma}\n\n\\begin{proof}\n  \\refer{\nBy Lemma~\\ref{lemma:ccgsteps}, we have from  \\eqref{eq:ccgstepNC}\nthat\n\\begin{equation} \\label{eq:boundnegcurvNCsteps}\n\t\td_k^\\top \\nabla^2 f(x_k) d_k = -\\|d_k\\|^3 \\le -\\epsH \\|d_k\\|^2\n\\end{equation}\n}\nThe result can thus be obtained exactly as in\n\\cite[Lemma~1]{CWRoyer_SJWright_2018}.\n\\end{proof}\n\n\\begin{lemma} \\label{lemma:LSgeneralNC}\nSuppose that Assumptions~\\ref{assum:compactlevelset}\nand~\\ref{assum:fC22} hold.  Suppose that at iteration $k$ of\nAlgorithm~\\ref{alg:dncg}, the search direction $d_k$ is of negative\ncurvature type, obtained either directly from Procedure~\\ref{alg:meo}\nor as the output of Algorithm~\\ref{alg:ccg} and d\\_type=NC. Then the\nbacktracking line search terminates with step length $\\alpha_k =\n\\theta^{j_k}$ with $j_k \\le \\jnc +1$, where $\\jnc$ is defined as in\nLemma~\\ref{lemma:LSccgNC}, and the decrease in the function value\nresulting from the chosen step length satisfies\n\\begin{equation} \\label{eq:LSgeneralNCdecrease}\nf(x_k) - f(x_k+\\alpha_k\\,d_k) \\; \\geq \\; \\frac{\\cnc}{8} \\epsH^3,\n\\end{equation}\t\nwith $\\cnc$ is defined in Lemma~\\ref{lemma:LSccgNC}.\n\\end{lemma}\n\n\\begin{proof}\n  Lemma~\\ref{lemma:LSccgNC} shows that the claim holds (with a factor\n  of $8$ to spare) when the direction of negative curvature is\n  obtained from Algorithm~\\ref{alg:ccg}.  When the direction is\n  obtained from Procedure~\\ref{alg:meo}, we have \\refer{by the scaling\n    of $d_k$ applied in Algorithm~\\ref{alg:dncg} that}\n\\begin{equation} \\label{eq:boundnegcurvNCproba}\nd_k^\\top \\nabla^2 f(x_k) d_k = -\\|d_k\\|^3 \\le -\\frac12 \\epsH \\|d_k\n\\|^2 < 0,\n\\end{equation}\nfrom which it follows that $\\|d_k \\| \\ge \\tfrac12 \\epsH$. The\nresult can now be obtained by following the proof of\nLemma~\\ref{lemma:LSccgNC}, with $\\tfrac12 \\epsH$ replacing $\\epsH$.\n\\end{proof}\n\n\\begin{theorem} \\label{theo:wccits}\nSuppose that Assumptions~\\ref{assum:compactlevelset},\n\\ref{assum:fC22}, and ~\\ref{assum:wccmeo} hold, and define\n\\begin{equation} \\label{eq:wccits}\n\\bar{K}_2 :=\\left\\lceil \\frac{3(f(x_0)-\\flow)}{\\min\\{\\csol,\\cnc/8\\}}\\max\\{\\epsg^{-3}\n\\epsH^3, \\epsH^{-3}\\} \\right\\rceil + 2,\n\\end{equation}\n where constants $\\csol$ and $\\cnc$\nare defined in Lemmas~\\ref{lemma:LSccgSOL} and~\\ref{lemma:LSccgNC},\nrespectively.  Then with probability at least\n\\refer{$(1-\\delta)^{\\bar{K}_2}$}, Algorithm~\\ref{alg:dncg} terminates\nat a point satisfying \\eqref{eq:2epson} in at most $\\bar{K}_2$\niterations. (With probability at most\n\\refer{$1-(1-\\delta)^{\\bar{K}_2}$}, it terminates incorrectly within\n$\\bar{K}_2$ iterations at a point for which $\\|\\nabla f(x_k) \\| \\le\n\\epsg$ but $\\lambda_{\\min}(\\nabla^2 f(x)) <\n-\\epsH$.)\n\\end{theorem}\n\n\\begin{proof}\nAlgorithm~\\ref{alg:dncg} terminates incorrectly with probability\n$\\delta$ at any iteration at which Procedure~\\ref{alg:meo} is called,\n\\refer{when Procedure~\\ref{alg:meo} certifies erroneously that\n  $\\lambda_{\\min}(\\nabla^2 f(x)) \\ge -\\epsH$.}\n%The overall probability of this happening during the first\n%$\\bar{K}_2$ iterations is \\refer{bounded by} $\\bar{K}_2\n%\\delta$. Therefore, with probability at least $1-\\bar{K}_2 \\delta$,\n%no incorrect termination occurs in the first $\\bar{K}_2$ iterations.\n\\refer{Since an erroneous certificate can only lead to termination, an\n  erroneous certificate at iteration $k$ means that\n  Procedure~\\ref{alg:meo} did not produce an erroneous certificate at\n  iterations $0$ to $k-1$. By a disjunction argument, we have that the\n  overall probability of terminating with an erroneous certificate\n  during the first $\\bar{K}_2$ iterations is bounded by\n  $1-(1-\\delta)^{\\bar{K}_2}$. Therefore, with probability at least\n  $(1-\\delta)^{\\bar{K}_2} $, no incorrect termination occurs in the\n  first $\\bar{K}_2$ iterations.} \n  \nSuppose now for contradiction that Algorithm~\\ref{alg:dncg} runs for\n$\\bar{K}_2$ iterations without terminating. That is, for all \n$l=0,1,\\dotsc,\\bar{K}_2$, we have either $\\| \\nabla f(x_l) \\| > \\epsg$ \nor $\\lambdamin(\\nabla^2 f(x_l)) < -\\epsH$. We perform the following \npartition of the set of iteration indices:\n\\begin{equation} \\label{eq:K123}\n  \\cK_1 \\cup \\cK_2 \\cup \\cK_3 = \\{0,1,\\dotsc,\\bar{K}_2-1\\},\n  \\end{equation}\nwhere $\\cK_1$, $\\cK_2$, and $\\cK_3$ are defined as follows.\n  \t\n\\medskip\n\t\n\\textbf{Case 1:} $\\cK_1 := \\{ l =0,1,\\dotsc,\\bar{K}_2-1 \\, : \\,\n\\|\\nabla f(x_l) \\| \\le \\epsg \\}$.  At each iteration $l \\in \\cK_1$,\nAlgorithm~\\ref{alg:dncg} calls Procedure~\\ref{alg:meo}, which does not\ncertify that $\\lambdamin (\\nabla^2 f(x_l)) \\ge -\\epsH$ (since the\nalgorithm continues to iterate) but rather returns a direction of\nsufficient negative curvature. By Lemma~\\ref{lemma:LSgeneralNC}, the\nstep along this direction leads to an improvement in $f$ that is\nbounded as follows:\n\\begin{equation} \\label{eq:wccitsK1decrease}\nf(x_l)-f(x_{l+1})  \\ge  \\frac{\\cnc}{8} \\epsH^3.\n\\end{equation}\t\t\n\t\n\\medskip\t\n\t\n\\textbf{Case 2:} $\\cK_2 := \\{ l=0,1,\\dotsc,\\bar{K}_2-1 \\, : \\, \\|\n\\nabla f(x_l) \\| > \\epsg \\; \\mbox{and} \\; \\| \\nabla f(x_{l+1}) \\| >\n\\epsg \\}$.  Algorithm~\\ref{alg:dncg} calls Algorithm~\\ref{alg:ccg} at\neach iteration $l \\in \\cK_2$, returning either an approximate damped\nNewton or a negative curvature direction. By combining\nLemmas~\\ref{lemma:LSccgSOL} and \\ref{lemma:LSccgNC}, we obtain a \ndecrease in $f$ satisfying\n\\begin{align}\n  \\nonumber\nf(x_l) - f(x_{l+1})  & \\ge  \\min\\{\\csol,\\cnc\\}\n\\min\\left\\{\\| \\nabla f(x_{l+1}) \\|^3\\epsH^{-3},\\epsH^3\\right\\} \\\\\n  \\label{eq:wccitsK2decrease}\n& \\ge \\min\\{\\csol,\\cnc/8\\}\n\\min\\left\\{\\epsg^3\\epsH^{-3},\\epsH^3\\right\\}.\n\\end{align}\n\n\\medskip\n\t\n\\textbf{Case 3:} $\\cK_3 := \\{ l=0,1,\\dotsc,\\bar{K}_2-1 \\, : \\, \\|\n\\nabla f(x_l) \\| > \\epsg \\ge \\| \\nabla f(x_{l+1}) \\| \\}$.  Because $\\|\n\\nabla f(x_{l+1})\\|$ may be small in this case, we can no longer bound\nthe decrease in $f$ by an expression such as\n\\eqref{eq:wccitsK2decrease}. We can however guarantee at least that\n$f(x_l)-f(x_{l+1}) \\ge 0$. Moreover, provided that $l<\\bar{K}_2-1$, we\nhave from $\\| \\nabla f(x_{l+1}) \\| \\le \\epsg$ that the next iterate\n$l+1$ is in $\\cK_1$. Thus, a significant decrease in $f$ will be\nattained at the {\\em next} iteration, and we have\n\\begin{equation} \\label{eq:K13}\n  | \\cK_3| \\le |\\cK_1|+1.\n\\end{equation}\n\n\\medskip\n\nWe now consider the total decrease in $f$ over the span of $\\bar{K}_2$\niterations, which is bounded by $f(x_0)-\\flow$ as follows:\n\\begin{align} \n  f(x_0) - \\flow &\\ge \\sum_{l=0}^{\\bar{K}_2-1} (f(x_l)-f(x_{l+1})) \\nonumber \\\\\n  \\label{eq:sumdecompwccits}\n& \\ge \\sum_{l \\in \\cK_1} (f(x_l)-f(x_{l+1})) +  \\sum_{l \\in \\cK_2} (f(x_l)-f(x_{l+1})) \n\\end{align}\nwhere both sums in the final expression are nonnegative. Using first\nthe bound \\eqref{eq:wccitsK1decrease} for the sum over $\\cK_1$, we\nobtain\n\\begin{equation} \\label{eq:wccitsboundK1}\nf(x_0) - \\flow \\; \\ge \\; |\\cK_1| \\frac{\\cnc}{8} \\epsH^3 \\; \\Leftrightarrow \\; \n|\\cK_1| \\le \\frac{f(x_0) - \\flow}{\\cnc/8} \\epsH^{-3}.\n\\end{equation}\n%From the sum over $\\cK_2$, we obtain\n%and~\\eqref{eq:wccitsK2decrease} leads to\nApplying~\\eqref{eq:wccitsK2decrease} to the sum over $\\cK_2$ leads to\n\\begin{equation} \\label{eq:wccitsboundK2}\n|\\cK_2| \\le \\frac{f(x_0) -\n  \\flow}{\\min\\{\\csol,\\cnc/8\\}}\\max\\{\\epsg^{-3} \\epsH^3,\\epsH^{-3}\\}.\n\\end{equation}\nUsing these bounds together with \\eqref{eq:K13}, we have\n\\begin{align*}\n  \\bar{K}_2  & = |\\cK_1| + |\\cK_2| + |\\cK_3| \\\\\n  & \\le  2|\\cK_1|\n  + |\\cK_2| + 1 \\\\\n  & \\le 3\\max\\{|\\cK_1|,|\\cK_2|\\} +1  \\\\\n  & \\le\n\\frac{3(f(x_0) - \\flow)}\n     {\\min\\{\\csol,\\cnc/8\\}}\\max\\{\\epsg^{-3}\\epsH^3,\\epsH^{-3}\\} +1 \\\\\n     & \\le \\bar{K}_2-1,\n                \\end{align*}\n                giving the required contradiction.\n\\end{proof}\n\n\\begin{proof}\nThe proof follows by combining Theorem~\\ref{theo:wccits} (which bounds\nthe number of iterations) with Lemma~\\ref{lemma:ccgits} and\nAssumption~\\ref{assum:wccmeo} (which bound the workload per\niteration).\n  \\end{proof}\n\n\\begin{lemma} \\label{lemma:propertiesCGoneiter}\nSuppose that $j$ iterations of the CG loop are performed in\nAlgorithm~\\ref{alg:ccg} or~\\ref{alg:cg}. Then, we have\n\\begin{equation} \\label{eq:suffcurvuptoj}\n\t  \\frac{p_i^\\top \\bH p_i}{\\|p_i\\|^2} > 0 \\quad \\mbox{for\n       all $i=0,1,\\dotsc,j-1$}.\n\\end{equation}\t\nMoreover, the following properties hold.\n\\begin{enumerate}\n\\item $y_i \\in \\lspan\\left\\{p_0,\\dotsc,p_{i-1}\\right\\}$,\n  $i=1,2,\\dotsc,j$.\n\\item $r_i \\in \\lspan\\left\\{p_0,\\dotsc,p_i\\right\\}$ for all $i=1,2,\\dotsc,j$, and \n\\[\nr_i^\\top v = 0, \\quad \\mbox{for all $v \\in\n  \\lspan\\left\\{p_0,\\dotsc,p_{i-1}\\right\\}$ and all $i=1,2,\\dotsc,j$}.\n\\]\n(In particular, $r_i^\\top r_l = 0$ if \\refer{$0 \\le l<i \\le j$. If} $j=n$, \nthen $r_n=0$.)\n\\item $\\|r_i\\| \\le \\|p_i\\|$, $i=0,1,\\dotsc,j$.\n\\item $r_i^\\top p_i = -\\|r_i\\|^2$, $i=0,1,\\dotsc,j$.\n\\item $p_i^\\top \\bH p_k=0$ for all $i,k =0,1,\\dotsc,j$ with $k \\neq i$.\n\\item $p_i = -\\sum_{k=0}^i ({\\|r_i\\|^2}/{\\|r_k\\|^2}) r_k$, $i=0,1,\\dotsc,j$.\n\\item\n  \\[\n  q(y_{i+1}) = q(y_i) - \\frac{\\|r_i\\|^4}{2 p_i^\\top \\bH p_i}, \\quad i=0,1,\\dotsc,j-1.\n  \\]\n\\item $r_i^\\top \\bH r_i \\ge p_i^\\top \\bH p_i$, $i=0,1,\\dotsc,j$.\n\\end{enumerate}\t\t\n\\end{lemma}\n\n\\begin{proof}\nSince CG has not terminated prior to iteration $j$,\n\\eqref{eq:suffcurvuptoj} clearly holds.  All properties then follow\nfrom the definition of the CG process, and most are proved in standard\ntexts (see, for example,\n\\cite[Chapter~5]{JNocedal_SJWright_2006}). Property 8 is less commonly\nused, so we provide a proof here.\n\nThe case $i=0$ is immediate since $r_0=-p_0$ and there is\nequality. When $i \\ge 1$, we have:\n\\begin{equation*}\n\tp_i = -r_i + \\frac{\\|r_i\\|^2}{\\|r_{i-1}\\|^2}p_{i-1} \\; \\Leftrightarrow \\; \n\tr_i = -p_i + \\frac{\\|r_i\\|^2}{\\|r_{i-1}\\|^2}p_{i-1}.\n\\end{equation*}\n(Note that if iteration $i$ is reached, we cannot have\n$\\|r_{i-1}\\|=0$.)  It follows that\n\\begin{eqnarray*}\n\tr_i^\\top \\bH r_i &=   \n\t&p_i^\\top \\bH p_i -2\\frac{\\|r_i\\|^2}{\\|r_{i-1}\\|^2} p_i^\\top \\bH p_{i-1} + \n\t\\frac{\\|r_i\\|^4}{\\|r_{i-1}\\|^4}p_{i-1}^\\top \\bH p_{i-1} \\\\\n\t&= &p_i^\\top \\bH p_i + \\frac{\\|r_i\\|^4}{\\|r_{i-1}\\|^4}p_{i-1}^\\top \\bH p_{i-1},\n\\end{eqnarray*}\nas $p_i^\\top \\bH p_{i-1} = 0$ by Property 5 above. Since iteration $i$\nhas been reached, $p_{i-1}$ is a direction of \\refer{positive\n  curvature},\n% at least $\\eps$,\nand we obtain $r_i^\\top \\bH r_i \\ge p_i^\\top \\bH p_i$, as required.\n\\end{proof}\n\n\\begin{lemma} \\label{lemma:cgcurvaturescalars}\nSuppose that Algorithm~\\ref{alg:ccg} computes iterates up to iteration\n$j+1$. Then, for any $i \\in \\{0,\\dotsc,j\\}$, we can compute\n(\\ref{eq:weakcurvdir}) as\n\\[\n\\frac{(y_{j+1}-y_i)^\\top \\bH (y_{j+1}-y_i)}{\\|y_{j+1}-y_i\\|^2} =\n\\frac{\\sum_{k=i}^{j} \\alpha_k \\|r_k\\|^2}\n{\\sum_{\\ell=0}^j \\left[\\sum_{k=\\max\\{\\ell,i\\}}^j \\alpha_k \\|r_k\\|^2 \\right]^2/\\|r_{\\ell}\\|^2}.\n\\]\n\\end{lemma}\n\n\\begin{proof}\nBy definition, $y_{j+1}-y_i = \\sum_{k=i}^{j} \\alpha_k p_k$. By\nconjugacy of the $p_k$ vectors, we have\n\\begin{equation} \\label{eq:numeratorcomp}\n(y_{j+1}-y_i)^\\top \\bH (y_{j+1}-y_i)\n= \\sum_{k=i}^{j} \\alpha_k^2 p_k^\\top \\bH p_k\n= \\sum_{k=i}^{j} \\alpha_k \\|r_k\\|^2,\n\\end{equation}\nwhere we used the definition of $\\alpha_k$ to obtain the last\nequality.  Now we turn our attention to the denominator. Using\nProperty 6 of Lemma~\\ref{lemma:propertiesCGoneiter}, we have that\n\\[\ny_{j+1}-y_i = \\sum_{k=i}^{j} \\alpha_k p_k =\n\\sum_{k=i}^{j} \\alpha_k \\left(-\\sum_{\\ell=0}^k \\frac{\\|r_k\\|^2}{\\|r_\\ell\\|^2} r_\\ell \\right),\n\\]\nBy rearranging the terms in the sum, we obtain\n\\begin{equation*}\ny_{j+1}-y_i = -\\sum_{k=i}^{j} \\sum_{\\ell=0}^k \\alpha_k\\|r_k\\|^2 \\frac{r_\\ell}{\\|r_{\\ell}\\|^2} \n= -\\sum_{\\ell=0}^j \\left[\\sum_{k=\\max\\{\\ell,i\\}}^j \\alpha_k \\|r_k\\|^2 \\right] \\frac{r_\\ell}{\\|r_\\ell\\|^2}.\n\\end{equation*}\nUsing the fact that the residuals $\\{ r_{\\ell} \\}_{\\ell=0,1,\\dotsc,j}$\nform an orthogonal set (by Property 2 of\nLemma~\\ref{lemma:propertiesCGoneiter}), we have that\n\\begin{equation*}\n  \\|y_{j+1}-y_i\\|^2 = \\sum_{\\ell=0}^j \\frac{1}{\\|r_{\\ell} \\|^2}\n  \\left[\\sum_{k=\\max\\{\\ell,i\\}}^j \\alpha_k \\|r_k\\|^2 \\right]^2.\n\\end{equation*}\nCombining this with~\\eqref{eq:numeratorcomp} gives the desired result.\n\\end{proof}\n\n\\begin{proof}\n  The proof proceeds by contradiction: If we assume that all\n  conditions specified in the statement of the theorem hold, and in\n  addition that\n\\begin{equation} \\label{eq:curvcondyKp1}\n\\frac{(y_{\\hIcg +1}-y_i)^\\top \\bH (y_{\\hIcg+1}-y_i)}\n     {\\|y_{\\hIcg+1}-y_i\\|^2} \\ge \\eps, \\quad \\mbox{for all\n       $i=0,1,\\dotsc,\\hIcg-1$,}\n\\end{equation}\nthen we must have \n\\begin{equation} \\label{eq:cvCGwhilestronglycvx}\n\\|r_{\\hIcg}\\| \\le \\sqrt{T} \\tau^{\\hIcg/2} \\|r_0\\|,\n\\end{equation}\ncontradicting \\eqref{eq:Kitsstronglycvx}. Note that\n\\begin{equation} \\label{eq:casehK}\n\t\t\\frac{(y_{\\hIcg +1}-y_{\\hIcg})^\\top \\bH (y_{\\hIcg+1}-y_{\\hIcg})}\n\t\t{\\|y_{\\hIcg+1}-y_{\\hIcg}\\|^2} = \n\t\t\\frac{\\alpha_{\\hIcg} p_{\\hIcg}^\\top \\bH (\\alpha_{\\hIcg} p_{\\hIcg})}\n\t\t{\\|\\alpha_{\\hIcg} p_{\\hIcg}\\|^2} = \\frac{p_{\\hIcg}^\\top \\bH p_{\\hIcg}}\n\t\t{\\|p_{\\hIcg}\\|^2} \\ge \\eps\n\\end{equation}\nby assumption, therefore we can consider \\eqref{eq:curvcondyKp1} to\nhold for $i=0,1,\\dotsc,\\hIcg$. Moreover, recalling the definition\n\\eqref{eq:quadraticCG} of the quadratic function $q$, we have for any\n$i=0,1,\\dotsc,\\hIcg$ that \n\\[\n\t\tq(y_{\\hIcg+1}) = q(y_i) + \\nabla q(y_i)^\\top (y_{\\hIcg+1}-y_i) + \n\t\t\\frac{1}{2}(y_{\\hIcg+1}-y_i)^\\top \\bH (y_{\\hIcg+1}-y_i).\n                \\]\nThus, \\eqref{eq:curvcondyKp1} can be reformulated as\n\\begin{equation} \\label{eq:stronglycvxyKp1yj}\n% \t\t\\forall i \\in \\{0,\\dotsc,\\hIcg\\}, \\quad \n\t\tq(y_{\\hIcg+1}) \\ge q(y_i) + r_i^\\top (y_{\\hIcg+1}-y_i) \n\t\t+ \\frac{\\eps}{2}\\|y_{\\hIcg+1}-y_i\\|^2, \\quad\n                \\mbox{for all $i=0,1,\\dotsc,\\hIcg$,}\n\\end{equation}\nwhere we used $\\nabla q(y_i)=r_i$ and the definitions\n\\eqref{eq:curvcondyKp1}, \\eqref{eq:casehK} of strong convexity along\nthe directions $y_{\\hIcg+1}-y_i$. In the remainder of the proof, and\nsimilarly to \\cite[Proof of Proposition\n  1]{YCarmon_JCDuchi_OHinder_ASidford_2017b}, we will show that\n\\eqref{eq:stronglycvxyKp1yj} leads to the contradiction\n\\eqref{eq:cvCGwhilestronglycvx}, thus proving that\n\\eqref{eq:yKp1negcurv} holds.\n\nWe define the  sequence of functions $\\Phi_j$, $j=0,1,\\dotsc\\hIcg$ as follows:\n\\[\n    \\Phi_0(z) := q(y_0) + \\frac{\\eps}{2}\\|z-y_0\\|^2,\n    \\]\n    and for $j=0,\\dotsc,\\hIcg-1$:\n\\begin{equation} \n    \\label{eq:formulaPhijplus1}\n\\Phi_{j+1}(z)  := \\tau\\Phi_j(z) + (1-\\tau)\\left( q(y_j) + r_j^\\top\n(z-y_j) +\\frac{\\eps}{2}\\|z-y_j\\|^2 \\right).\n\\end{equation}\nSince each $\\Phi_j$ is a quadratic function with Hessian $\\eps I$, it\ncan be written as follows:\n\\begin{equation} \\label{eq:Phijwithvj}\n\t\t\\Phi_j(z) \\; = \\; \\Phi_j^* + \\frac{\\eps}{2}\\|z-v_j\\|^2,\n\\end{equation}\nwhere $v_j$ is the unique minimizer of $\\Phi_j$, and $\\Phi_j^* =\n\\Phi_j(v_j)$ is the minimum value of $\\Phi_j$. (Note that $v_0=y_0=0$\nand $\\Phi^*_0 = q(y_0) = 0$.)\n\nDefining\n\\begin{equation} \\label{eq:defpsi}\n\\psi(y) := q(y_0) - q(y) + \\frac{\\eps}{2}\\|y-y_0\\|^2 = \\Phi_0(y) - q(y),\n\\end{equation}\nwe give a short inductive argument to establish that \n\\begin{equation} \\label{eq:PhiYJp1}\n\\Phi_j(y_{\\hIcg+1}) \\le q(y_{\\hIcg+1}) + \\tau^j \\psi(y_{\\hIcg+1}),\n\\quad j=0,1,\\dotsc,\\hIcg.\n\\end{equation}\nFor $j=0$, \\eqref{eq:PhiYJp1} holds because $\\Phi_0(y) = q(y) +\n\\psi(y)$ by definition. Assuming that \\eqref{eq:PhiYJp1} holds for\nsome index $j \\ge 0$, we find by first applying\n\\eqref{eq:formulaPhijplus1} (with $z=y_{\\hIcg+1}$) and then\n\\eqref{eq:stronglycvxyKp1yj} (with $i=j$) that\n\\begin{eqnarray*}\n\\Phi_{j+1}(y_{\\hIcg+1}) &= &\\tau\\Phi_j(y_{\\hIcg+1}) + (1-\\tau)\\left( q(y_j) + \nr_j^\\top (y_{\\hIcg+1}-y_j) +\\frac{\\eps}{2}\\|y_{\\hIcg+1}-y_j\\|^2 \\right) \\\\\n&\\le &\\tau\\Phi_j(y_{\\hIcg+1}) + (1-\\tau) q(y_{\\hIcg+1}).\n\\end{eqnarray*}\nThus, we have\n\\begin{align*}\n\\Phi_{j+1}(y_{\\hIcg+1}) &\\le \\tau\\Phi_j(y_{\\hIcg+1}) + (1-\\tau) q(y_{\\hIcg+1}) \\\\\n&\\le \\tau q(y_{\\hIcg+1}) + \\tau^{j+1} \\psi(y_{\\hIcg+1}) + \n(1-\\tau) q(y_{\\hIcg+1}) \\quad \\mbox{from \\eqref{eq:PhiYJp1}} \\\\\n&= q(y_{\\hIcg+1}) + \\tau^{j+1} \\psi(y_{\\hIcg+1}),\n\\end{align*}\nwhich proves \\eqref{eq:PhiYJp1} for $j+1$, and thus completes the\ninductive argument.\n\nWe next prove another technical fact about the relationship between\n$q(y_j)$ and $\\Phi_j^*$, namely,\n\\begin{equation} \\label{eq:468}\n  q(y_j) \\le \\Phi_j^*, \\quad j=0,1,\\dotsc, \\hIcg.\n\\end{equation}\nWe establish this result by an inductive argument that is quite\nlengthy and technical; we note the termination of this phase of the\nproof clearly below.\n\n\nThis result trivially holds (with equality) at $j=0$.  Supposing that\nit holds for some $j =0,1,\\dotsc, \\hIcg-1$, we will prove that it also\nholds for $j+1$.\n\nBy using Properties 7 and 8 of Lemma~\\ref{lemma:propertiesCGoneiter},\n%and also $\\|\\bH \\| \\le (M+2\\eps)$, we have\n\\refer{and also $\\|\\bH r_j\\| \\le (M+2\\eps)\\|r_j\\|$,} we have\n\\begin{equation*} \n\tq(y_{j+1}) = q(y_j) - \\frac{\\|r_j\\|^4}{2\\, p_j^\\top \\bH p_j} \\le \n\tq(y_j) - \\frac{\\|r_j\\|^4}{2\\, r_j^\\top \\bH r_j} \\le \n\tq(y_j) - \\frac{\\|r_j\\|^2}{2(M+2\\eps)}.\n\\end{equation*}\nIt follows from induction hypothesis $q(y_j) \\le \\Phi_j^*$ that \n\\begin{align} \\label{eq:qyjp1Phijstar}\nq(y_{j+1}) \\le q(y_j) - \\frac{\\|r_j\\|^2}{2(M+2\\eps)}\n&= \\tau q(y_j) + (1-\\tau) q(y_j) -\\frac{\\|r_j\\|^2}{2(M+2\\eps)} \\nonumber \\\\\n&\\le \\tau \\Phi_j^* + (1-\\tau) q(y_j) -\\frac{\\|r_j\\|^2}{2(M+2\\eps)}.\n\\end{align}\nBy taking the derivative in \\eqref{eq:formulaPhijplus1}, and using\n\\eqref{eq:Phijwithvj}, we obtain\n\\begin{align*}\n\\nabla \\Phi_{j+1}(z) &= \\tau \\nabla \\Phi_j(z) + (1-\\tau)\\left[r_j + \\eps(z-y_j)\\right] \\\\\n\\Rightarrow \\;\\; \\eps(z-v_{j+1})&=  \\eps \\tau (z-v_j) + (1-\\tau) \\left(r_j + \\eps (z-y_j) \\right).\n\\end{align*}\nBy rearranging the above relation (and noting  that the $z$ terms cancel out), we obtain:\n\\begin{equation} \\label{eq:vjp1minPhijp1}\n\tv_{j+1}  \\; = \\; \\tau v_j - \\frac{1-\\tau}{\\eps}r_j + (1-\\tau) y_j.\n\\end{equation}\nIt follows from this expression together with Properties 1 and 2 of\nLemma~\\ref{lemma:propertiesCGoneiter} that \n\\begin{equation} \\label{eq:vjinKj}\nv_j \\in \\lspan\\left\\{p_0,p_1,\\dotsc,p_{j-1}\\right\\}, \\quad j=1,2,\\dotsc,\\hIcg.\n\\end{equation}\n(The result holds for $j=1$, from \\eqref{eq:vjp1minPhijp1} we have\n$v_1 \\in \\lspan \\{v_0, r_0, y_0\\} = \\lspan \\{r_0\\} = \\lspan \\{p_0 \\}$,\nand an induction based on \\eqref{eq:vjp1minPhijp1} can be used to\nestablish \\eqref{eq:vjinKj} for the other values of $j$.)  By\ncombining the expressions~\\eqref{eq:Phijwithvj} for $\\Phi_j$,\n$\\Phi_{j+1}$ with the recurrence formula~\\eqref{eq:formulaPhijplus1}\nfor $\\Phi_{j+1}$, we obtain\n\\begin{align*} \n\t\t\\Phi_{j+1}^* + \\frac{\\eps}{2}\\|y_j-v_{j+1}\\|^2 &= \\Phi_{j+1}(y_j) \\\\\n%\t\t&= \\tau \\Phi_j(y_j) + (1-\\tau)\\left(q(y_j)+r_j^\\top (y_j-y_j) + \n%\t\t\\frac{\\eps}{2}\\|y_j-y_j\\|^2\\right)  \\\\\n\t\t&= \\tau\\Phi_j(y_j) + (1-\\tau) q(y_j) \\\\\n                &= \\tau\\left(\\Phi_j^* + \n\t\t\\frac{\\eps}{2}\\|y_j-v_j\\|^2\\right) + (1-\\tau) q(y_j)\n\\end{align*}\nand therefore\n\\begin{equation} \\label{eq:doubleexpressionPhijp1}\n  \\Phi_{j+1}^* = \\tau \\left(\\Phi_j^* +\n  \\frac{\\eps}{2}\\|y_j-v_j\\|^2\\right) + (1-\\tau) q(y_j) -\n  \\frac{\\eps}{2}\\|y_j-v_{j+1}\\|^2.\n\\end{equation}\nOn the other hand, we have by~\\eqref{eq:vjp1minPhijp1} that\n\\begin{eqnarray} \\label{eq:normylminusvlp1}\n\t\t\\|y_j-v_{j+1}\\|^2 &= &\\left\\| \\tau (y_j-v_j) \n\t\t+\\frac{1-\\tau}{\\eps}r_j \\right\\|^2 \\nonumber \\\\\n\t\t&= &(\\tau^2 \\|y_j-v_j\\|^2 + \\frac{(1-\\tau)^2}{\\eps^2}\\|r_j\\|^2 \n\t\t+ \\frac{2}{\\eps}(1-\\tau)\\tau r_j^\\top (y_j-v_j) \\nonumber \\\\\n\t\t&= & \\tau^2 \\|y_j-v_j\\|^2 + \\frac{(1-\\tau)^2}{\\eps^2}\\|r_j\\|^2,\n\\end{eqnarray}\nwhere the last relation comes from $r_j \\perp\n\\lspan\\{p_0,\\dotsc,p_{j-1}\\}$ (Property 2 of\nLemma~\\ref{lemma:propertiesCGoneiter}) and \\eqref{eq:vjinKj} in the\ncase $j \\ge 1$, and immediately in the case $j=0$, since $y_0=v_0=0$.\nBy combining \\eqref{eq:doubleexpressionPhijp1}\nand~\\eqref{eq:normylminusvlp1}, we arrive at:\n\\begin{align} \\label{eq:phijp1star}\n\t\t\\Phi_{j+1}^* &= \\tau \\left(\\Phi_j^* + \n\t\t\\frac{\\eps}{2}\\|y_j-v_j\\|^2\\right) + (1-\\tau) q(y_j) \n\t\t- \\frac{\\eps}{2}\\|y_j-v_{j+1}\\|^2 \\nonumber \\\\\n\t\t&= \\tau \\left(\\Phi_j^* + \n\t\t\\frac{\\eps}{2}\\|y_j-v_j\\|^2\\right) + (1-\\tau) q(y_j) \n\t\t-\\frac{\\eps}{2}\\tau^2 \\|y_j-v_j\\|^2 - \\frac{(1-\\tau)^2}{2\\eps} \\|r_j\\|^2 \n\t\t\\nonumber \\\\\n\t\t&= \\tau \\Phi_j^* + \\frac{\\eps}{2}\\left[ \\tau \n\t\t-\\tau^2\\right]\\|y_j-v_j\\|^2 + (1-\\tau) q(y_j) \n\t\t- \\frac{(1-\\tau)^2}{2\\eps} \\|r_j\\|^2 \\nonumber \\\\ \n\t\t&= \\tau \\Phi_j^* + \n\t\t\\frac{\\eps}{2}(1-\\tau)\\tau \\|y_j-v_j\\|^2 + (1-\\tau) q(y_j) \n\t\t- \\frac{(1-\\tau)^2}{2\\eps} \\|r_j\\|^2 \\nonumber \\\\\n\t\t&\\ge \\tau \\Phi_j^* + (1-\\tau) q(y_j) \n\t\t- \\frac{(1-\\tau)^2}{2\\eps} \\|r_j\\|^2 \\nonumber \\\\\n                &\\ge q(y_{j+1}) + \\frac{1}{2(M+2\\eps)}\\|r_j\\|^2 \n\t\t- \\frac{(1-\\tau)^2}{2\\eps} \\|r_j\\|^2.\n\\end{align}\nwhere the last inequality comes from~\\eqref{eq:qyjp1Phijstar}.  By\nusing the definitions of $\\tau$ and $\\kappa$ in\nAlgorithm~\\ref{alg:ccg}, we have\n\\[\n\t\t\\frac{(1-\\tau)^2}{2\\eps} = \\frac{1}{2\\eps(\\sqrt{\\kappa}+1)^2} \\le \n\t\t\\frac{1}{2\\eps\\kappa} = \\frac{1}{2(M+2\\eps)}.\n\\]\nIt therefore follows from \\eqref{eq:phijp1star} that $q(y_{j+1}) \\le\n\\Phi_{j+1}^*$. At this point, we have shown that when $q(y_j) \\le\n\\Phi_j^*$ for $j=0,1,\\dotsc,\\hIcg-1$, it follows that $q(y_{j+1}) \\le\n\\Phi_{j+1}^*$, establishing the inductive step. As a result, our\nproof of \\eqref{eq:468} is complete.\n\n\nBy substituting $j=\\hIcg$ into \\eqref{eq:468}, we obtain\n$q(y_{\\hIcg}) \\le \\Phi_{\\hIcg}^*$,\nwhich in combination with \\eqref{eq:PhiYJp1} with $j=\\hIcg$, and the\ndefinition \\eqref{eq:Phijwithvj}, yields\n\\begin{equation} \\label{eq:desiredecreaseformula}\nq(y_{\\hIcg}) -q(y_{\\hIcg+1}) \\le \\Phi_{\\hIcg}^* - q(y_{\\hIcg+1}) \n\\le \\Phi_{\\hIcg}(y_{\\hIcg+1}) - q(y_{\\hIcg+1})\n\\le \\tau^{\\hIcg} \\psi(y_{\\hIcg+1}).\n\\end{equation}\nBy substitution from \\eqref{eq:defpsi}, we obtain\n\\begin{equation} \\label{eq:funcdecreasecontrad}\nq(y_{\\hIcg}) - q(y_{\\hIcg+1}) \\; \\le \\; \\tau^{\\hIcg}\n\\left(q(y_0)-q(y_{\\hIcg+1}) +\n\\frac{\\eps}{2}\\|y_0-y_{\\hIcg+1}\\|^2\\right).\n\\end{equation}\n\nWe now depart from the analysis\nof~\\cite{YCarmon_JCDuchi_OHinder_ASidford_2017b}, and complete the\nproof of this result by expressing \\eqref{eq:funcdecreasecontrad} in\nterms of residual norms.  On the left-hand side, we have\n\\begin{align*}\n  q(y_{\\hIcg}) - q(y_{\\hIcg+1})\n  % &= &\\nabla q(y_{\\hIcg+1})^\\top (y_{\\hIcg}-y_{\\hIcg+1}) + \n%\t\t\\frac{1}{2}(y_{\\hIcg}-y_{\\hIcg+1})^\\top \\bH (y_{\\hIcg}-y_{\\hIcg+1})  \\\\ \n\t\t&= r_{\\hIcg+1}^\\top (y_{\\hIcg}-y_{\\hIcg+1}) + \n\t\t\\frac{1}{2}(y_{\\hIcg}-y_{\\hIcg+1})^\\top \\bH (y_{\\hIcg}-y_{\\hIcg+1})  \\\\\n\t\t&= \\frac{1}{2}(y_{\\hIcg}-y_{\\hIcg+1})^\\top \\bH (y_{\\hIcg}-y_{\\hIcg+1})\n\\end{align*}\nbecause $r_{\\hIcg+1}^\\top (y_{\\hIcg}-y_{\\hIcg+1}) = r_{\\hIcg+1}^\\top\n(\\alpha_{\\hIcg} p_{\\hIcg}) = 0$ by\nLemma~\\ref{lemma:propertiesCGoneiter}, Property 2.\nWe thus have from \\eqref{eq:casehK} that\n\\begin{alignat}{2}\n  \\nonumber\n  q(y_{\\hIcg}) - q(y_{\\hIcg+1}) &\\ge \\frac{\\eps}{2}\\|y_{\\hIcg}-y_{\\hIcg+1}\\|^2  &&  \\\\\n  \\nonumber\n  &\\refer{= \\frac{\\eps}{2} \\|\\alpha_{\\hIcg}p_{\\hIcg}\\|^2} \\\\\n  \\nonumber\n  &\\refer{\\ge \\frac{\\eps}{2 (M+2\\eps)^2}\\|\\bH(\\alpha_{\\hIcg}p_{\\hIcg})\\|^2} \n  && \\mbox{(since $\\|\\bH p_{\\hIcg}\\| \\le (M+2\\eps)\\|p_{\\hIcg}\\|$)} \\\\\n  \\nonumber\n%  &\\ge \\frac{\\eps}{2 (M+2\\eps)^2} \\|\\bH(y_{\\hIcg}-y_{\\hIcg+1})\\|^2  && \\mbox{(since $\\bH \\preceq (M+2\\eps)I$)} \\\\\n  &= \\frac{\\eps}{2 (M+2\\eps)^2} \\|\\bH(y_{\\hIcg}-y_{\\hIcg+1})\\|^2   \\\\\n  \\nonumber\n  &= \\frac{\\eps}{2 (M+2\\eps)^2} \\|r_{\\hIcg} - r_{\\hIcg+1}\\|^2  && \\mbox{(since $r_j=g+\\bH y_j$)} \\\\\n  \\nonumber\n\t\t&= \\frac{\\eps}{2 (M+2\\eps)^2} (\\|r_{\\hIcg}\\|^2 + \\|r_{\\hIcg+1}\\|^2) & \\quad & \\mbox{(by Lemma~\\ref{lemma:propertiesCGoneiter}, Property 2)} \\\\\n  \\label{eq:funcdeccontradLHS}\n\t\t& \\ge \\frac{\\eps}{2 (M+2\\eps)^2} \\|r_{\\hIcg}\\|^2,\n\\end{alignat}\n\n\nOn the right-hand side of~\\eqref{eq:funcdecreasecontrad}, because of\nthe strong convexity condition \\eqref{eq:stronglycvxyKp1yj} at $i=0$,\nwe have\n\\begin{align*}\nq(y_0)-q(y_{\\hIcg+1}) + \\frac{\\eps}{2}\\|y_0-y_{\\hIcg+1}\\|^2  & \\le \n-\\nabla q(y_0)^\\top (y_{\\hIcg+1}-y_0) \\\\\n&= -r_0^\\top (y_{\\hIcg+1}-y_0)  \\le \\|r_0\\|\\|y_{\\hIcg+1}-y_0\\|.\n\\end{align*}\nMoreover, we have \n\\[\n\t\t\\|y_{\\hIcg+1}-y_0\\| \\; = \\;\n\t\t\\left\\|\\sum_{i=0}^{\\hIcg} \\alpha_i p_i \\right\\| \n\t\t\\; \\le \\; \\sum_{i=0}^{\\hIcg} \\alpha_i \\|p_i\\| \n\t\t\\; = \\; \\sum_{i=0}^{\\hIcg} \\frac{\\|r_i\\|^2}{p_i^\\top \\bH p_i}\\|p_i\\|,\n\\]\nwhere the last relation follows from the definition of $\\alpha_i$. By\ncombining these last two bounds, and using Property 3 of\nLemma~\\ref{lemma:propertiesCGoneiter}, we obtain\n\\begin{equation} \\label{eq:wd12}\n\t\tq(y_0)-q(y_{\\hIcg+1}) + \\frac{\\eps}{2}\\|y_0-y_{\\hIcg+1}\\|^2 \\; \\le \\; \n\t\t\\|r_0\\| \\sum_{i=0}^{\\hIcg} \\|r_i\\| \\frac{\\|p_i\\|^2}{p_i^\\top \\bH p_i} \\;\n\t\t\\le \\; \\|r_0\\| \\frac{1}{\\eps} \\sum_{i=0}^{\\hIcg} \\|r_i\\|,\n\\end{equation}\nwhere the last inequality holds $p_j^\\top \\bH p_j \\ge \\eps \\|p_j\\|^2$\nfor $j=0,1,\\dotsc,\\hIcg$ by assumption.\n\nTo bound the sum in \\eqref{eq:wd12}, we recall that since\nAlgorithm~\\ref{alg:ccg} did not terminate until iteration $\\hIcg$,\n\\refer{the residual norms $\\|r_i \\|$ at all iterations\n  $i=0,1,\\dotsc,\\hIcg-1$ must have decreased at the expected\n  convergence rate. In particularly, we have $\\|r_i\\| \\le \\sqrt{T}\n  \\tau^{i/2} \\|r_0\\|$ for the {\\em possibly smaller} versions of\n  $\\sqrt{T}$ and $\\tau$ that prevailed at iteration $i$, so certainly\n  $\\|r_i\\| \\le \\sqrt{T} \\tau^{i/2} \\|r_0\\|$ for the final values of\n  these parameters.} Thus for $i=0,1,\\dotsc,\\hIcg-1$, we have\n\\[\n\\|r_i\\| \\le \\sqrt{T} \\tau^{i/2} \\|r_0\\| \\le \\tau^{(i-\\hIcg)/2}\\|r_{\\hIcg}\\|,\n\\]\nwhere we used $\\|r_{\\hIcg}\\| \\ge \\sqrt{T} \\tau^{\\hIcg/2} \\|r_0\\|$\n(from \\eqref{eq:Kitsstronglycvx}) for the last inequality. Observing\nthat this bound also holds (trivially) for $i=\\hIcg$, we obtain by\nsubstituting in \\eqref{eq:wd12} that\n\\begin{align}\n  \\nonumber\nq(y_0)-q(y_{\\hIcg+1}) + \\frac{\\eps}{2}\\|y_0-y_{\\hIcg+1}\\|^2\n&\\le \\|r_0\\| \\frac{1}{\\eps} \\sum_{i=0}^{\\hIcg} \\tau^{(i-\\hIcg)/2} \\|r_{\\hIcg}\\| \\\\\n\\nonumber\n&\\le \\|r_0\\| \\frac{\\tau^{-\\hIcg/2}}{\\eps}\\|r_{\\hIcg}\\|\\,\\sum_{i=0}^{\\hIcg} (\\sqrt{\\tau})^{i} \\\\\n\\label{eq:funcdeccontradRHS}\n& \\le \\|r_0\\| \\frac{\\tau^{-\\hIcg/2}}{\\eps}\\|r_{\\hIcg}\\| \\frac{1}{1-\\sqrt{\\tau}}.\n\\end{align}\nApplying successively \\eqref{eq:funcdeccontradLHS},\n\\eqref{eq:funcdecreasecontrad} and \\eqref{eq:funcdeccontradRHS}\nfinally yields:\n\\begin{align*}\n\t\t\\frac{\\eps}{2 (M+2\\eps)^2} \\|r_{\\hIcg}\\|^2 &\\le \n\t\t q(y_{\\hIcg}) - q(y_{\\hIcg+1}) \\\\\n\t\t &\\le \\tau^{\\hIcg} \\left( q(y_0)-q(y_{\\hIcg+1}) + \\frac{\\eps}{2}\\|y_0-y_{\\hIcg+1}\\|^2 \\right) \\\\\n\t\t &\\le \\tau^{\\hIcg} \\|r_0\\|\\|r_{\\hIcg}\\| \n\t\t \\frac{\\tau^{-\\hIcg/2}}{\\eps}\\frac{1}{1-\\sqrt{\\tau}}.\n\\end{align*}\nAfter rearranging this inequality and dividing by $\\|r_{\\hIcg}\\| > 0$,\nwe obtain\n\\begin{equation} \\label{eq:finalcontrad}\n\t\t\\|r_{\\hIcg}\\| \\le \\frac{2 (M+2\\eps)^2}{\\eps^2}\\frac{\\tau^{\\hIcg/2}}{1-\\sqrt{\\tau}} \\|r_0\\|\n\t\t= \\sqrt{T} \\tau^{\\hIcg/2}\\|r_0\\|.\n\\end{equation}\nWe have thus established \\eqref{eq:cvCGwhilestronglycvx} which, as we\nnoted earlier, contradicts \\eqref{eq:Kitsstronglycvx}. Thus\n\\eqref{eq:curvcondyKp1} cannot be true, so we have established\n\\eqref{eq:yKp1negcurv}, as required.\n\\end{proof}","units":[{"start":30735,"end":31455,"start_line":661,"end_line":677,"raw_tex":"\\begin{lemma} \\label{lemma:ccgits}\nThe number of iterations of Algorithm~\\ref{alg:ccg} is bounded by\n\\[\n\\min\\{n,J(M,\\eps,\\zeta)\\},\n\\]\nwhere $J=J(M,\\eps,\\zeta)$ is the smallest integer such that\n$\\sqrt{T}\\tau^{J/2} \\le \\hat{\\zeta}$ \\refer{where $M$, $\\hat\\zeta$,\n  $T$, and $\\tau$ are the values returned by the algorithm}. \\refer{If\n  all iterates $y_i$ generated by the algorithm are stored, the number\n  of matrix-vector multiplications required is bounded by\n\\begin{equation} \\label{eq:pv8}\n\t\\min\\{n,J(M,\\eps,\\zeta)\\}+1.\n\\end{equation}\nIf the iterates $y_i$ must be regenerated in order to define the\ndirection $d$ returned after \\eqref{eq:weakcurvdir}, this bound\nbecomes $2\\min\\{n,J(M,\\eps,\\zeta)\\}+1$.}\n\\end{lemma}","sha256_utf8":"44cb9053c1ec6e2a6ca1f79b4eb507cfd4bc4eb66f249e390fd48574a46f37a5"},{"start":31456,"end":34384,"start_line":678,"end_line":730,"raw_tex":"\\begin{proof}\n  If the full $n$ iterations are performed, without any of the\n  termination conditions being tripped, the standard properties of CG\n  (see Appendix~\\ref{app:cgresults}) ensure that the final residual\n  $r_n$ is zero, so that the condition $\\|r_n \\| \\le \\hat\\zeta\n  \\|r_0\\|$ is satisfied, and termination occurs.\n\n  \\refer{Since no more than $n$ iterations are performed, the upper\n    bound $M$ is updated at most a finite number of times, so the\n    quantity $J$ is well defined.}\n\n  Supposing that $J<n$, we note from the definition of $J$ that\n  $\\sqrt{T}\\tau^{J/2} \\|r_0 \\| \\le \\hat\\zeta \\|r_0\\|$. Thus at least\n  one of the following two conditions must be satisfied at iteration\n  $J$: $\\|r_{J} \\| \\le \\hat\\zeta \\|r_0\\|$ or $\\|r_{J} \\| >\n  \\sqrt{T}\\tau^{J/2} \\|r_0\\|$. In either case, termination will occur\n  at iteration $J$, unless it has occurred already at a previous\n  iteration.\n\n  To derive \\eqref{eq:pv8}, note that the main workload at each\n  iteration $j$ is computation of a single matrix-vector product\n%  \\refer{$H p_j$ (since matrix-vector products involving the matrices\n%    $H$ and $\\bH$ and the vectors $y_j$ and $r_j$ can be computed in\n%    terms of this vector, in an additional $O(n)$ operations).}\n%  \\refer{(The ``$+1$'' in \\eqref{eq:pv8} is from the additional\n%    matrix-vector multiplication required in the case of $\\|r_{J} \\| >\n%    \\sqrt{T}\\tau^{J/2} \\|r_0\\|$.)}\n  \\refer{$H p_j$ after the increment of $j$ (since \n    matrix-vector products involving the matrices\n    $H$ and $\\bH$ and the vectors $y_j$ and $r_j$ can be computed in\n    terms of this vector, in an additional $O(n)$ operations).}\n  \\refer{(The ``$+1$'' in \\eqref{eq:pv8} accounts for the initial\n    matrix-vector multiplication $H p_0$ performed prior to entering \n    the loop.)}\n%    \\crcomment{I think the new tests based on $M$ changed the \n%    places in which we compute the $H p_j$, so I corrected this \n%    paragraph. The result still holds, but the +1 relates to the \n%    initial step (before the loop). Note that the four termination \n%    tests do not require additional matrix-vector products since $Hp_j$ \n%    has already been computed for the $M$-related tests.\n%    MO: Changed account to accounts.}\n\n  If we do not store additional information, we need to regenerate the\n  information needed to compute the direction $d$ satisfying\n  \\eqref{eq:weakcurvdir} by re-running the iterations of CG, possibly\n  up to the \\refer{second-to-last} iteration. This fact accounts for\n  the additional cost of $\\min\\{n,J(M,\\eps,\\zeta)\\}$ in the no-storage\n  case. \\footnote{\\refer{Interestingly, as we show in\n      Appendix~\\ref{app:cgresults}, the ratios on the left-hand side\n      of \\eqref{eq:weakcurvdir} can be calculated without knowledge of\n      $y_i$ for $i=0,1,\\dotsc,j-1$, provided that we store the scalar\n      quantities $\\alpha_i$ and $\\|r_i\\|^2$ for $i=0,1,\\dotsc,j-1$.}}\n\\end{proof}","sha256_utf8":"a466aa1f9b796e1fa6934254c9f24db333882fe7edce51f9f22114ea3c7f5f69"},{"start":34386,"end":36299,"start_line":732,"end_line":772,"raw_tex":"\\refer{Note that $J(M,\\epsilon,\\zeta)$ is an increasing function of\n  $M$, since $\\hat{\\zeta}$ is a decreasing function of $M$, while $T$\n  and $\\tau$ (and thus $\\sqrt{T}\\tau^{j/2}$) are increasing in $M$. If\n  $U_H$ is known in advance, we can call Algorithm~\\ref{alg:ccg} with\n  $M = U_H$ and use $J(U_H,\\epsilon,\\zeta)$ as the bound. Alternately,\n  we can call Algorithm~\\ref{alg:ccg} with $M=0$ and let it adjust $M$\n  as needed during the computation. Since the final value of $M$ will\n  be at most $U_H$, and since $J(M,\\epsilon,\\zeta)$ is an increasing\n  function of $M$, the quantity $J(U_H,\\epsilon,\\zeta)$ provides the\n  upper bound on the number of iterations in this case too.}\n\nWe can estimate $J$ by taking logs in its definition, as follows:\n\\[\nJ \\le \\frac{2\\ln(\\hat{\\zeta}/\\sqrt{T})}{\\ln(\\tau)} =\n\\frac{\\ln(\\hat{\\zeta}^2/T)}\n     {\\ln\\left(\\frac{\\sqrt{\\kappa}}{\\sqrt{\\kappa}+1}\\right)} =\n     \\frac{\\ln(T/\\hat{\\zeta}^2)}{\\ln(1+{1}/{\\sqrt{\\kappa}})} \\le\n     \\left(\\sqrt{\\kappa}+\\frac{1}{2}\\right)\n     \\ln\\left(\\frac{T}{\\hat{\\zeta}^2}\\right),\n\\]\nwhere we used $\\ln(1+\\tfrac{1}{t}) \\ge \\tfrac{1}{t+1/2}$ to obtain the\nlatest inequality. By\nreplacing $T,\\tau,\\hat{\\zeta},\\kappa$ by their\ndefinitions in Algorithm~\\ref{alg:ccg}, and using\n$\n\\frac{1}{1-\\sqrt{\\tau}} = \\frac{1+\\sqrt{\\tau}}{1-\\tau} \\le\n\\frac{2}{1-\\tau},\n$\nwe obtain\n\\begin{align} \\nonumber\n%% J(M,\\eps,\\zeta) &  \\le\n%% \\min\\left\\{n,\\left\\lceil \\left(\\sqrt{\\frac{M+2\\eps}{\\eps}}+\\frac{1}{2}\\right)\n%% \\ln\\left(\\frac{144 \\left(\\sqrt{\\frac{M+2\\eps}{\\eps}}+1\\right)^2\n%%   \\left(\\frac{M+2\\eps}{\\eps}\\right)^6} {\\zeta^2}\\right) \\right\\rceil \\right\\} \\\\\nJ(M,\\eps,\\zeta) &  \\le\n\\refer{\\min\\left\\{n,\\left\\lceil \\left(\\sqrt{\\kappa}+\\frac{1}{2}\\right)\n\\ln\\left(\\frac{144 \\left(\\sqrt{\\kappa}+1\\right)^2\n  \\kappa^6} {\\zeta^2}\\right) \\right\\rceil \\right\\}} \\\\\n\\label{eq:HvcountcappedCG}\n & = \\min \\left\\{ n , \\tcO(\\eps^{-1/2}) \\right\\}.\n\\end{align}\n","sha256_utf8":"f3e558d76d6de5260ef3d3001780932ab4e18563731a1fc32d384ad126fb0334"},{"start":52654,"end":53672,"start_line":1115,"end_line":1136,"raw_tex":"\\begin{theorem} \\label{theo:cvCGwhilestronglycvx}\nSuppose that the main loop of Algorithm~\\ref{alg:ccg} terminates with\n$j=\\hIcg$, where\n\\[\n\\hIcg \\in \\{1,\\dotsc,\\min\\{n,J(M,\\eps,\\zeta)\\}\\},\n\\]\n(where $J(M,\\eps,\\zeta)$ is defined in Lemma~\\ref{lemma:ccgits} and\n\\eqref{eq:HvcountcappedCG}) \\refer{because the fourth termination test is\nsatisfied and the three earlier conditions do not hold, that is, \n$y_{\\hIcg}^\\top \\bH y_{\\hIcg} \\ge \\eps \\|y_{\\hIcg}\\|^2$, \n$p_{\\hIcg}^\\top \\bH p_{\\hIcg} \\ge \\eps \\|p_{\\hIcg}\\|^2$, and\n\\begin{equation} \\label{eq:Kitsstronglycvx}\n\\|r_{\\hIcg}\\| > \\max\\{\\hat{\\zeta},\\sqrt{T}\\tau^{\\hIcg /2}\\}\\|r_0\\|.\n\\end{equation}\nwhere $M$, $T$, $\\hat{\\zeta}$, and $\\tau$ are the values returned by\nAlgorithm~\\ref{alg:ccg}. Then\n$y_{\\hIcg+1}$ is computed by Algorithm~\\ref{alg:ccg}, and we have}\n\\begin{equation} \\label{eq:yKp1negcurv}\n\\frac{(y_{\\hIcg +1}-y_i)^\\top \\bH (y_{\\hIcg+1}-y_i)}\n{\\|y_{\\hIcg+1}-y_i\\|^2} < \\eps, \\quad \\mbox{for some $i \\in \\{0,\\dotsc,\\hIcg-1\\}$.}\n\\end{equation}\n\\end{theorem}","sha256_utf8":"1663aa7eb66d2b31657a693e2220e878ff1d9779d5de3e83195e5fb85cac01a2"},{"start":54799,"end":56463,"start_line":1162,"end_line":1206,"raw_tex":"\\begin{lemma} \\label{lemma:ccgsteps}\nLet Assumptions~\\ref{assum:compactlevelset} and~\\ref{assum:fC22} hold,\nand suppose that Algorithm~\\ref{alg:ccg} is invoked at an iterate\n$x_k$ of Algorithm~\\ref{alg:dncg} (so that $\\|\\nabla f(x_k)\\| >\n\\epsg>0$).\nLet $d_k$ be the vector obtained in Algorithm~\\ref{alg:dncg} from the\noutput $d$ of Algorithm~\\ref{alg:ccg}. Then, one of the two following\nstatements holds:\n\\begin{enumerate}\n\\item {\\em d\\_type=SOL}, and the direction $d_k$ satisfies \n\\begin{subequations} \\label{eq:ccgstepSOL} \n\\begin{align}\n\\label{eq:ccgstepSOLcurv}\nd_k^\\top (\\nabla^2 f(x_k)+2\\epsH I) d_k & \\ge \\epsH \\|d_k\\|^2, \\\\\n\\label{eq:ccgstepSOLnorm}\n% \\|d_k\\| & \\le \\epsH^{-1} \\sqrt{1+\\zeta^2/4} \\|\\nabla f(x_k)\\|,\\\\\n\\|d_k\\| & \\le \\refer{1.1 \\epsH^{-1} \\|\\nabla f(x_k)\\|,}\\\\\n\\label{eq:ccgstepSOLres}\n%% \\|\\hat{r}_k \\| \\le  \n%% \\frac{\\hat{\\zeta}}{1-\\hat{\\zeta}}(U_H+2\\epsH)\\|d_k\\| & \\le \n%% \\frac{1}{2}\\epsH\\zeta \\| d_k \\|,\n\\|\\hat{r}_k \\|  & \\le  \\refer{\\frac{1}{2}\\epsH\\zeta \\| d_k \\|,}\n\\end{align}\n\\end{subequations}\nwhere\n\\begin{equation} \\label{eq:def.rhat}\n\\hat{r}_k := (\\nabla^2 f(x_k)+2\\epsH I) d_k+ \\nabla f(x_k);\n\\end{equation}\n\\item {\\em d\\_type=NC}, and the direction $d_k$ satisfies \n  $d_k^\\top \\nabla f(x_k) \\le 0$ as well as\n  \\refer{\n  \\begin{equation} \\label{eq:ccgstepNC}\n    \\frac{d_k^\\top \\nabla^2 f(x_k) d_k}{\\|d_k\\|^2} = -\\|d_k \\| \\le -\\epsH.\n  \\end{equation}\n  }\n%% \\begin{subequations} \\label{eq:ccgstepNC}\n%% \\begin{align}\n%% \\label{eq:ccgstepNCcurv}\n%% \\frac{d_k^\\top \\nabla^2 f(x_k) d_k}{\\|d_k\\|^2} \\le -\\epsH,\\\\\n%% \\label{eq:ccgstepNCnorm}\n%% \\|d_k\\| \\ge \\epsH.\n%% \\end{align}\n%% \\end{subequations}\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"a62079bb420330fb0c3933b698ba8126ec163e9d4eb32da56832d7b62ac1e4de"},{"start":56466,"end":59826,"start_line":1208,"end_line":1297,"raw_tex":"\\begin{proof}\n\\refer{For simplicity of notation, we use $H=\\nabla^2 f(x_k)$ and\n  $g=\\nabla f(x_k)$ in the proof.}  Suppose first that d\\_type=SOL. In\nthat case, we have from the termination conditions in\nAlgorithm~\\ref{alg:ccg} and \\eqref{eq:def.rhat} that\n\\begin{subequations} \\label{eq:propSOLstep}\n\\begin{align} \n\\label{eq:propSOLstepCURV}\nd_k^\\top (H+2 \\epsH I)d_k & \\ge \\epsH \\|d_k\\|^2, \\\\\n\\label{eq:propSOLstepRES}\n\\|\\hat{r}_k \\| &  \\le \\hat{\\zeta} \\|g\\|,\n\\end{align}\n\\end{subequations}\n\\refer{where $\\hat\\zeta$ was returned by the algorithm.}\nWe immediately recognize~\\eqref{eq:ccgstepSOLcurv}\nin~\\eqref{eq:propSOLstepCURV}.  We now\nprove~\\eqref{eq:ccgstepSOLnorm}. Observe first \nthat \\eqref{eq:propSOLstepCURV} yields\n\\[\n\\epsH\\|d_k\\|^2 \\le d_k^\\top (H+2 \\epsH I) d_k \\le \\|d_k\\| \\|(H+2\\epsH\nI) d_k\\|,\n\\]\nso from \\eqref{eq:def.rhat} we have\n\\begin{equation*}\n\\|d_k\\| \\le \\epsH^{-1} \\|(H+2\\epsH I)d_k\\| = \\epsH^{-1} \\|-g+\\hat{r}_k\\| \n= \\epsH^{-1} \\sqrt{\\|g\\|^2 + \\| \\hat{r}_k \\|^2} \\le \n\\epsH^{-1} \\sqrt{1+\\hat{\\zeta}^2}  \\|g\\|,\n\\end{equation*}\nwhere we used~\\eqref{eq:propSOLstepRES} to obtain the final bound,\ntogether with the equality $\\|-g+\\hat{r}_k\\|^2=\\|g\\|^2+\\| \\hat{r}_k\n\\|^2$, which follows from $g^\\top \\hat{r}_k = r_0^\\top \\hat{r}_k =\n0$, by orthogonality of the residuals in CG (see\nLemma~\\ref{lemma:propertiesCGoneiter}, Property 2). Since\n\\refer{$\\hat{\\zeta} \\le \\zeta/(3 \\kappa) \\le 1/6$ by construction, we\n  have $\\|d_k \\| \\le \\sqrt{37/36} \\epsH^{-1} \\|g\\| \\le 1.1 \\epsH^{-1}\n  \\|g\\|$, proving \\eqref{eq:ccgstepSOLnorm}.}\n\nThe bound \\eqref{eq:ccgstepSOLres} follows\nfrom~\\eqref{eq:propSOLstepRES} and the logic below:\n\\begin{align*}\n\t\\| \\hat{r}_k \\| &\\le \\hat{\\zeta} \\|g\\| \\le \n\t\\hat{\\zeta}\\left(\\|(H+2 \\epsH I) d_k \\| + \n\t\\| \\hat{r}_k \\|\\right)  \\le \\hat\\zeta \\left( (M+2\\epsH) \\|d_k \\| +  \\| \\hat{r}_k \\| \\right) \\\\\n        \\Rightarrow \\;\\; \\| \\hat{r}_k \\| &\\le \n\t\\frac{\\hat{\\zeta}}{1-\\hat{\\zeta}}(M+2 \\epsH) \\|d_k\\|,\n\\end{align*}\n\\refer{where $M$ is the value returned by the algorithm.}\nWe finally use \\refer{$\\hat\\zeta < 1/6$}\n%% \\[\n%% \\hat\\zeta = \\frac{\\zeta}{3}\\frac{\\epsH}{M+2\\epsH} \\le\n%% \\frac{\\zeta}{3} \\frac12 < \\frac16,\n%% \\]\nto arrive at\n\\[\n\\frac{\\hat\\zeta}{1-\\hat{\\zeta}} (M+2\\epsH) \\le \\frac65 \\hat{\\zeta}\n(M+2\\epsH) = \\frac65 \\frac{\\zeta \\epsH}{3} < \\frac12 \\zeta \\epsH,\n\\]\nyielding  \\eqref{eq:ccgstepSOLres}.\n\nIn the case of d\\_type=NC, we recall that Algorithm~\\ref{alg:dncg}\ndefines\n\\begin{equation} \\label{eq:scaleccgNC}\nd_k = -\\mathrm{sgn}(d^\\top g) \\frac{|d^\\top H\n  d|}{\\|d\\|^2} \\frac{d}{\\|d\\|}\n\\end{equation}\nwhere $d$ denotes the direction obtained by\nAlgorithm~\\ref{alg:ccg}. It follows immediately that $d_k^\\top g \\le\n0$. Since $d_k$ and $d$ are collinear, we also have that\n\\refer{\n\\[\n\\frac{d_k^\\top (H+2\\epsH I)d_k}{\\|d_k\\|^2} = \\frac{d^\\top (H+2\\epsH\n  I)d}{\\|d\\|^2} \\le \\epsH \\; \\Rightarrow \\; \\frac{d_k^\\top H\n  d_k}{\\|d_k\\|^2}\n% = \\frac{d^\\top H d}{\\|d\\|^2}\n\\le -\\epsH.\n\\]\nBy using this bound together with \\eqref{eq:scaleccgNC}, we obtain\n\\[\n\\|d_k\\| = \\frac{|d^\\top H d|}{\\|d\\|^2} = \\frac{|d_k^\\top H d_k|}{\\|d_k\\|^2} =\n-\\frac{d_k^\\top H d_k}{\\|d_k\\|^2} \\ge \\epsH,\n\\]\nproving \\eqref{eq:ccgstepNC}.\n}\n%% establishing \\eqref{eq:ccgstepNCcurv}.  By \\eqref{eq:scaleccgNC}, we\n%% then have\n%% \\[\n%% \\|d_k\\| = \\frac{|d^\\top H d|}{\\|d\\|^2} \\ge \\epsH,\n%% \\]\n%% proving \\eqref{eq:ccgstepNCnorm}.\n\\end{proof}","sha256_utf8":"fa278ed40590712146bd5146d33ba8e3225b9f9d88fa61c347fbfdf35de070e3"},{"start":60668,"end":61710,"start_line":1317,"end_line":1340,"raw_tex":"\\begin{lemma} \\label{lemma:LSccgSOL}\nSuppose that Assumptions~\\ref{assum:compactlevelset}\nand~\\ref{assum:fC22} hold. Suppose that at iteration $k$ of\nAlgorithm~\\ref{alg:dncg}, we have $\\|\\nabla f(x_k)\\| > \\epsg$, so\nthat Algorithm~\\ref{alg:ccg} is called. When Algorithm~\\ref{alg:ccg}\noutputs a direction $d_k$ with d\\_type=SOL, then the backtracking line\nsearch requires at most $j_k \\le \\jsol+1$ iterations, where\n\\begin{equation} \\label{eq:ccglsitsSOL}\n\\jsol \\; = \\; \\left[ \\frac{1}{2}\\log_{\\theta}\\left(\n  \\frac{3(1-\\zeta)}{L_H+\\eta} \\frac{\\epsH^2}{\\refer{1.1} U_g} \\right)\n  \\right]_+,\n\\end{equation}\nand the resulting step $x_{k+1} = x_k + \\alpha_k d_k$ satisfies\n\\begin{equation} \\label{eq:ccglsdecreaseSOL}\nf(x_k) - f(x_{k+1}) \\; \\ge \\; \n\\csol\\min\\left(\\|\\nabla f(x_{k+1})\\|^3 \\epsH^{-3},\\epsH^3\\right),\n\\end{equation}\nwhere \n\\begin{equation*}\n\\csol = \\frac{\\eta}{6}\\min\\left\\{ \\left[ \\frac{4}{\\sqrt{(4+\\zeta)^2+8 L_H}+4+\\zeta}\n  \\right]^3, \\left[ \\frac{3\\theta^2 (1-\\zeta) }{L_H+\\eta}\n  \\right]^3\\right\\}.\n\\end{equation*}\n\\end{lemma}","sha256_utf8":"e93473f529045fb629556885b92fc6b294371ed18d61fc3d79a2957f48318f10"},{"start":61713,"end":62175,"start_line":1342,"end_line":1351,"raw_tex":"\\begin{proof}\nThe proof tracks closely that\nof~\\cite[Lemma~13]{CWRoyer_SJWright_2018}. The only significant\ndifference is that equation (65) of~\\cite{CWRoyer_SJWright_2018},\nwhich is instrumental to the proof and requires a probabilistic\nassumption on $\\lambda_{\\min}(\\nabla^2 f(x_k))$, is now ensured\n\\emph{deterministically} by~\\eqref{eq:ccgstepSOLnorm} from\nLemma~\\ref{lemma:ccgsteps}. As a result, both the proof and the result\nare deterministic.\n\\end{proof}","sha256_utf8":"0336cb8c39be4783c7dbf84bf9925f83a3a34cf2e3d3567ed6b358ce348d56be"},{"start":63332,"end":64293,"start_line":1375,"end_line":1397,"raw_tex":"\\begin{lemma} \\label{lemma:LSccgNC}\n  Suppose that Assumptions~\\ref{assum:compactlevelset}\n  and~\\ref{assum:fC22} hold.  Suppose that at iteration $k$ of\n  Algorithm~\\ref{alg:dncg}, we have $\\|\\nabla f(x_k)\\| > \\epsg$, so\n  that Algorithm~\\ref{alg:ccg} is called. When Algorithm~\\ref{alg:ccg}\n  outputs d\\_type=NC, the direction $d_k$ (computed from $d$ in\n  Algorithm~\\ref{alg:dncg}) has the following properties: The\n  backtracking line search terminates with step length $\\alpha_k =\n  \\theta^{j_k}$ with $j_k \\le \\jnc +1$, where\n\\begin{equation} \\label{eq:eiglsits}\t\n\\jnc := \\left[ \\log_{\\theta}\\left( \\frac{3}{L_H+\\eta} \\right)\n  \\right]_+,\n\\end{equation}\nand the resulting step $x_{k+1} = x_k + \\alpha_k d_k$ satisfies\n\\begin{equation} \\label{eq:decreaseNC}\nf(x_k) - f(x_k+\\alpha_k\\,d_k) \\; \\geq \\; \\cnc \\epsH^3,\n\\end{equation}\t\nwith\n\\begin{equation*}\n\\cnc := \\frac{\\eta}{6}\n\\min\\left\\{1,\\frac{27\\theta^3}{(L_H+\\eta)^3}\\right\\}.\n\\end{equation*}\n\\end{lemma}","sha256_utf8":"31d8f98205b61f8c88cbf3cae4958884ed106a356d907edba06a231d79a7265a"},{"start":64296,"end":64617,"start_line":1399,"end_line":1409,"raw_tex":"\\begin{proof}\n  \\refer{\nBy Lemma~\\ref{lemma:ccgsteps}, we have from  \\eqref{eq:ccgstepNC}\nthat\n\\begin{equation} \\label{eq:boundnegcurvNCsteps}\n\t\td_k^\\top \\nabla^2 f(x_k) d_k = -\\|d_k\\|^3 \\le -\\epsH \\|d_k\\|^2\n\\end{equation}\n}\nThe result can thus be obtained exactly as in\n\\cite[Lemma~1]{CWRoyer_SJWright_2018}.\n\\end{proof}","sha256_utf8":"8b3b2541510d8a558b5ab01377fc1baff6ead4b3e0eb050d21c035cd456fc189"},{"start":73247,"end":74048,"start_line":1594,"end_line":1608,"raw_tex":"\\begin{lemma} \\label{lemma:LSgeneralNC}\nSuppose that Assumptions~\\ref{assum:compactlevelset}\nand~\\ref{assum:fC22} hold.  Suppose that at iteration $k$ of\nAlgorithm~\\ref{alg:dncg}, the search direction $d_k$ is of negative\ncurvature type, obtained either directly from Procedure~\\ref{alg:meo}\nor as the output of Algorithm~\\ref{alg:ccg} and d\\_type=NC. Then the\nbacktracking line search terminates with step length $\\alpha_k =\n\\theta^{j_k}$ with $j_k \\le \\jnc +1$, where $\\jnc$ is defined as in\nLemma~\\ref{lemma:LSccgNC}, and the decrease in the function value\nresulting from the chosen step length satisfies\n\\begin{equation} \\label{eq:LSgeneralNCdecrease}\nf(x_k) - f(x_k+\\alpha_k\\,d_k) \\; \\geq \\; \\frac{\\cnc}{8} \\epsH^3,\n\\end{equation}\t\nwith $\\cnc$ is defined in Lemma~\\ref{lemma:LSccgNC}.\n\\end{lemma}","sha256_utf8":"998ad5fb011d128fdf486ec6736bf013339f5f0cda5c6313f9b9c5054f6a792f"},{"start":74049,"end":74720,"start_line":1609,"end_line":1622,"raw_tex":"\\begin{proof}\n  Lemma~\\ref{lemma:LSccgNC} shows that the claim holds (with a factor\n  of $8$ to spare) when the direction of negative curvature is\n  obtained from Algorithm~\\ref{alg:ccg}.  When the direction is\n  obtained from Procedure~\\ref{alg:meo}, we have \\refer{by the scaling\n    of $d_k$ applied in Algorithm~\\ref{alg:dncg} that}\n\\begin{equation} \\label{eq:boundnegcurvNCproba}\nd_k^\\top \\nabla^2 f(x_k) d_k = -\\|d_k\\|^3 \\le -\\frac12 \\epsH \\|d_k\n\\|^2 < 0,\n\\end{equation}\nfrom which it follows that $\\|d_k \\| \\ge \\tfrac12 \\epsH$. The\nresult can now be obtained by following the proof of\nLemma~\\ref{lemma:LSccgNC}, with $\\tfrac12 \\epsH$ replacing $\\epsH$.\n\\end{proof}","sha256_utf8":"610a766d4dcf829ebeacd9a83dec130cd31cf6e8ff5738f069f38e1bae6b4977"},{"start":74820,"end":75669,"start_line":1627,"end_line":1644,"raw_tex":"\\begin{theorem} \\label{theo:wccits}\nSuppose that Assumptions~\\ref{assum:compactlevelset},\n\\ref{assum:fC22}, and ~\\ref{assum:wccmeo} hold, and define\n\\begin{equation} \\label{eq:wccits}\n\\bar{K}_2 :=\\left\\lceil \\frac{3(f(x_0)-\\flow)}{\\min\\{\\csol,\\cnc/8\\}}\\max\\{\\epsg^{-3}\n\\epsH^3, \\epsH^{-3}\\} \\right\\rceil + 2,\n\\end{equation}\n where constants $\\csol$ and $\\cnc$\nare defined in Lemmas~\\ref{lemma:LSccgSOL} and~\\ref{lemma:LSccgNC},\nrespectively.  Then with probability at least\n\\refer{$(1-\\delta)^{\\bar{K}_2}$}, Algorithm~\\ref{alg:dncg} terminates\nat a point satisfying \\eqref{eq:2epson} in at most $\\bar{K}_2$\niterations. (With probability at most\n\\refer{$1-(1-\\delta)^{\\bar{K}_2}$}, it terminates incorrectly within\n$\\bar{K}_2$ iterations at a point for which $\\|\\nabla f(x_k) \\| \\le\n\\epsg$ but $\\lambda_{\\min}(\\nabla^2 f(x)) <\n-\\epsH$.)\n\\end{theorem}","sha256_utf8":"614f7dba742e66aa7fad947630102627c977ebe9292df991b866c26a49d70dac"},{"start":75672,"end":80610,"start_line":1646,"end_line":1757,"raw_tex":"\\begin{proof}\nAlgorithm~\\ref{alg:dncg} terminates incorrectly with probability\n$\\delta$ at any iteration at which Procedure~\\ref{alg:meo} is called,\n\\refer{when Procedure~\\ref{alg:meo} certifies erroneously that\n  $\\lambda_{\\min}(\\nabla^2 f(x)) \\ge -\\epsH$.}\n%The overall probability of this happening during the first\n%$\\bar{K}_2$ iterations is \\refer{bounded by} $\\bar{K}_2\n%\\delta$. Therefore, with probability at least $1-\\bar{K}_2 \\delta$,\n%no incorrect termination occurs in the first $\\bar{K}_2$ iterations.\n\\refer{Since an erroneous certificate can only lead to termination, an\n  erroneous certificate at iteration $k$ means that\n  Procedure~\\ref{alg:meo} did not produce an erroneous certificate at\n  iterations $0$ to $k-1$. By a disjunction argument, we have that the\n  overall probability of terminating with an erroneous certificate\n  during the first $\\bar{K}_2$ iterations is bounded by\n  $1-(1-\\delta)^{\\bar{K}_2}$. Therefore, with probability at least\n  $(1-\\delta)^{\\bar{K}_2} $, no incorrect termination occurs in the\n  first $\\bar{K}_2$ iterations.} \n  \nSuppose now for contradiction that Algorithm~\\ref{alg:dncg} runs for\n$\\bar{K}_2$ iterations without terminating. That is, for all \n$l=0,1,\\dotsc,\\bar{K}_2$, we have either $\\| \\nabla f(x_l) \\| > \\epsg$ \nor $\\lambdamin(\\nabla^2 f(x_l)) < -\\epsH$. We perform the following \npartition of the set of iteration indices:\n\\begin{equation} \\label{eq:K123}\n  \\cK_1 \\cup \\cK_2 \\cup \\cK_3 = \\{0,1,\\dotsc,\\bar{K}_2-1\\},\n  \\end{equation}\nwhere $\\cK_1$, $\\cK_2$, and $\\cK_3$ are defined as follows.\n  \t\n\\medskip\n\t\n\\textbf{Case 1:} $\\cK_1 := \\{ l =0,1,\\dotsc,\\bar{K}_2-1 \\, : \\,\n\\|\\nabla f(x_l) \\| \\le \\epsg \\}$.  At each iteration $l \\in \\cK_1$,\nAlgorithm~\\ref{alg:dncg} calls Procedure~\\ref{alg:meo}, which does not\ncertify that $\\lambdamin (\\nabla^2 f(x_l)) \\ge -\\epsH$ (since the\nalgorithm continues to iterate) but rather returns a direction of\nsufficient negative curvature. By Lemma~\\ref{lemma:LSgeneralNC}, the\nstep along this direction leads to an improvement in $f$ that is\nbounded as follows:\n\\begin{equation} \\label{eq:wccitsK1decrease}\nf(x_l)-f(x_{l+1})  \\ge  \\frac{\\cnc}{8} \\epsH^3.\n\\end{equation}\t\t\n\t\n\\medskip\t\n\t\n\\textbf{Case 2:} $\\cK_2 := \\{ l=0,1,\\dotsc,\\bar{K}_2-1 \\, : \\, \\|\n\\nabla f(x_l) \\| > \\epsg \\; \\mbox{and} \\; \\| \\nabla f(x_{l+1}) \\| >\n\\epsg \\}$.  Algorithm~\\ref{alg:dncg} calls Algorithm~\\ref{alg:ccg} at\neach iteration $l \\in \\cK_2$, returning either an approximate damped\nNewton or a negative curvature direction. By combining\nLemmas~\\ref{lemma:LSccgSOL} and \\ref{lemma:LSccgNC}, we obtain a \ndecrease in $f$ satisfying\n\\begin{align}\n  \\nonumber\nf(x_l) - f(x_{l+1})  & \\ge  \\min\\{\\csol,\\cnc\\}\n\\min\\left\\{\\| \\nabla f(x_{l+1}) \\|^3\\epsH^{-3},\\epsH^3\\right\\} \\\\\n  \\label{eq:wccitsK2decrease}\n& \\ge \\min\\{\\csol,\\cnc/8\\}\n\\min\\left\\{\\epsg^3\\epsH^{-3},\\epsH^3\\right\\}.\n\\end{align}\n\n\\medskip\n\t\n\\textbf{Case 3:} $\\cK_3 := \\{ l=0,1,\\dotsc,\\bar{K}_2-1 \\, : \\, \\|\n\\nabla f(x_l) \\| > \\epsg \\ge \\| \\nabla f(x_{l+1}) \\| \\}$.  Because $\\|\n\\nabla f(x_{l+1})\\|$ may be small in this case, we can no longer bound\nthe decrease in $f$ by an expression such as\n\\eqref{eq:wccitsK2decrease}. We can however guarantee at least that\n$f(x_l)-f(x_{l+1}) \\ge 0$. Moreover, provided that $l<\\bar{K}_2-1$, we\nhave from $\\| \\nabla f(x_{l+1}) \\| \\le \\epsg$ that the next iterate\n$l+1$ is in $\\cK_1$. Thus, a significant decrease in $f$ will be\nattained at the {\\em next} iteration, and we have\n\\begin{equation} \\label{eq:K13}\n  | \\cK_3| \\le |\\cK_1|+1.\n\\end{equation}\n\n\\medskip\n\nWe now consider the total decrease in $f$ over the span of $\\bar{K}_2$\niterations, which is bounded by $f(x_0)-\\flow$ as follows:\n\\begin{align} \n  f(x_0) - \\flow &\\ge \\sum_{l=0}^{\\bar{K}_2-1} (f(x_l)-f(x_{l+1})) \\nonumber \\\\\n  \\label{eq:sumdecompwccits}\n& \\ge \\sum_{l \\in \\cK_1} (f(x_l)-f(x_{l+1})) +  \\sum_{l \\in \\cK_2} (f(x_l)-f(x_{l+1})) \n\\end{align}\nwhere both sums in the final expression are nonnegative. Using first\nthe bound \\eqref{eq:wccitsK1decrease} for the sum over $\\cK_1$, we\nobtain\n\\begin{equation} \\label{eq:wccitsboundK1}\nf(x_0) - \\flow \\; \\ge \\; |\\cK_1| \\frac{\\cnc}{8} \\epsH^3 \\; \\Leftrightarrow \\; \n|\\cK_1| \\le \\frac{f(x_0) - \\flow}{\\cnc/8} \\epsH^{-3}.\n\\end{equation}\n%From the sum over $\\cK_2$, we obtain\n%and~\\eqref{eq:wccitsK2decrease} leads to\nApplying~\\eqref{eq:wccitsK2decrease} to the sum over $\\cK_2$ leads to\n\\begin{equation} \\label{eq:wccitsboundK2}\n|\\cK_2| \\le \\frac{f(x_0) -\n  \\flow}{\\min\\{\\csol,\\cnc/8\\}}\\max\\{\\epsg^{-3} \\epsH^3,\\epsH^{-3}\\}.\n\\end{equation}\nUsing these bounds together with \\eqref{eq:K13}, we have\n\\begin{align*}\n  \\bar{K}_2  & = |\\cK_1| + |\\cK_2| + |\\cK_3| \\\\\n  & \\le  2|\\cK_1|\n  + |\\cK_2| + 1 \\\\\n  & \\le 3\\max\\{|\\cK_1|,|\\cK_2|\\} +1  \\\\\n  & \\le\n\\frac{3(f(x_0) - \\flow)}\n     {\\min\\{\\csol,\\cnc/8\\}}\\max\\{\\epsg^{-3}\\epsH^3,\\epsH^{-3}\\} +1 \\\\\n     & \\le \\bar{K}_2-1,\n                \\end{align*}\n                giving the required contradiction.\n\\end{proof}","sha256_utf8":"9b94fb9f6952d5e120f741727bb90c439697f24ac880ded82c59eb1cebbde5ca"},{"start":83532,"end":83762,"start_line":1818,"end_line":1823,"raw_tex":"\\begin{proof}\nThe proof follows by combining Theorem~\\ref{theo:wccits} (which bounds\nthe number of iterations) with Lemma~\\ref{lemma:ccgits} and\nAssumption~\\ref{assum:wccmeo} (which bound the workload per\niteration).\n  \\end{proof}","sha256_utf8":"56cc21c1a56a5dce7adffb4b0c57ef6308478b4c1f7f4c58e2afcd61a6fb3812"},{"start":89543,"end":90728,"start_line":1945,"end_line":1973,"raw_tex":"\\begin{lemma} \\label{lemma:propertiesCGoneiter}\nSuppose that $j$ iterations of the CG loop are performed in\nAlgorithm~\\ref{alg:ccg} or~\\ref{alg:cg}. Then, we have\n\\begin{equation} \\label{eq:suffcurvuptoj}\n\t  \\frac{p_i^\\top \\bH p_i}{\\|p_i\\|^2} > 0 \\quad \\mbox{for\n       all $i=0,1,\\dotsc,j-1$}.\n\\end{equation}\t\nMoreover, the following properties hold.\n\\begin{enumerate}\n\\item $y_i \\in \\lspan\\left\\{p_0,\\dotsc,p_{i-1}\\right\\}$,\n  $i=1,2,\\dotsc,j$.\n\\item $r_i \\in \\lspan\\left\\{p_0,\\dotsc,p_i\\right\\}$ for all $i=1,2,\\dotsc,j$, and \n\\[\nr_i^\\top v = 0, \\quad \\mbox{for all $v \\in\n  \\lspan\\left\\{p_0,\\dotsc,p_{i-1}\\right\\}$ and all $i=1,2,\\dotsc,j$}.\n\\]\n(In particular, $r_i^\\top r_l = 0$ if \\refer{$0 \\le l<i \\le j$. If} $j=n$, \nthen $r_n=0$.)\n\\item $\\|r_i\\| \\le \\|p_i\\|$, $i=0,1,\\dotsc,j$.\n\\item $r_i^\\top p_i = -\\|r_i\\|^2$, $i=0,1,\\dotsc,j$.\n\\item $p_i^\\top \\bH p_k=0$ for all $i,k =0,1,\\dotsc,j$ with $k \\neq i$.\n\\item $p_i = -\\sum_{k=0}^i ({\\|r_i\\|^2}/{\\|r_k\\|^2}) r_k$, $i=0,1,\\dotsc,j$.\n\\item\n  \\[\n  q(y_{i+1}) = q(y_i) - \\frac{\\|r_i\\|^4}{2 p_i^\\top \\bH p_i}, \\quad i=0,1,\\dotsc,j-1.\n  \\]\n\\item $r_i^\\top \\bH r_i \\ge p_i^\\top \\bH p_i$, $i=0,1,\\dotsc,j$.\n\\end{enumerate}\t\t\n\\end{lemma}","sha256_utf8":"e767a3d15550cb437339b46d7d5d168f88b3122ab4e2d44b839aa48b54c50636"},{"start":90730,"end":91934,"start_line":1974,"end_line":2001,"raw_tex":"\\begin{proof}\nSince CG has not terminated prior to iteration $j$,\n\\eqref{eq:suffcurvuptoj} clearly holds.  All properties then follow\nfrom the definition of the CG process, and most are proved in standard\ntexts (see, for example,\n\\cite[Chapter~5]{JNocedal_SJWright_2006}). Property 8 is less commonly\nused, so we provide a proof here.\n\nThe case $i=0$ is immediate since $r_0=-p_0$ and there is\nequality. When $i \\ge 1$, we have:\n\\begin{equation*}\n\tp_i = -r_i + \\frac{\\|r_i\\|^2}{\\|r_{i-1}\\|^2}p_{i-1} \\; \\Leftrightarrow \\; \n\tr_i = -p_i + \\frac{\\|r_i\\|^2}{\\|r_{i-1}\\|^2}p_{i-1}.\n\\end{equation*}\n(Note that if iteration $i$ is reached, we cannot have\n$\\|r_{i-1}\\|=0$.)  It follows that\n\\begin{eqnarray*}\n\tr_i^\\top \\bH r_i &=   \n\t&p_i^\\top \\bH p_i -2\\frac{\\|r_i\\|^2}{\\|r_{i-1}\\|^2} p_i^\\top \\bH p_{i-1} + \n\t\\frac{\\|r_i\\|^4}{\\|r_{i-1}\\|^4}p_{i-1}^\\top \\bH p_{i-1} \\\\\n\t&= &p_i^\\top \\bH p_i + \\frac{\\|r_i\\|^4}{\\|r_{i-1}\\|^4}p_{i-1}^\\top \\bH p_{i-1},\n\\end{eqnarray*}\nas $p_i^\\top \\bH p_{i-1} = 0$ by Property 5 above. Since iteration $i$\nhas been reached, $p_{i-1}$ is a direction of \\refer{positive\n  curvature},\n% at least $\\eps$,\nand we obtain $r_i^\\top \\bH r_i \\ge p_i^\\top \\bH p_i$, as required.\n\\end{proof}","sha256_utf8":"66c504a39c2979ad58002d344a991c6987f96240a40b45946c85209772c1385f"},{"start":92680,"end":93102,"start_line":2017,"end_line":2026,"raw_tex":"\\begin{lemma} \\label{lemma:cgcurvaturescalars}\nSuppose that Algorithm~\\ref{alg:ccg} computes iterates up to iteration\n$j+1$. Then, for any $i \\in \\{0,\\dotsc,j\\}$, we can compute\n(\\ref{eq:weakcurvdir}) as\n\\[\n\\frac{(y_{j+1}-y_i)^\\top \\bH (y_{j+1}-y_i)}{\\|y_{j+1}-y_i\\|^2} =\n\\frac{\\sum_{k=i}^{j} \\alpha_k \\|r_k\\|^2}\n{\\sum_{\\ell=0}^j \\left[\\sum_{k=\\max\\{\\ell,i\\}}^j \\alpha_k \\|r_k\\|^2 \\right]^2/\\|r_{\\ell}\\|^2}.\n\\]\n\\end{lemma}","sha256_utf8":"39fbe62850b69a4a4371e03a9eaa9e5c2487a711dbbfa2d8345b13baacb08986"},{"start":93103,"end":94423,"start_line":2027,"end_line":2055,"raw_tex":"\\begin{proof}\nBy definition, $y_{j+1}-y_i = \\sum_{k=i}^{j} \\alpha_k p_k$. By\nconjugacy of the $p_k$ vectors, we have\n\\begin{equation} \\label{eq:numeratorcomp}\n(y_{j+1}-y_i)^\\top \\bH (y_{j+1}-y_i)\n= \\sum_{k=i}^{j} \\alpha_k^2 p_k^\\top \\bH p_k\n= \\sum_{k=i}^{j} \\alpha_k \\|r_k\\|^2,\n\\end{equation}\nwhere we used the definition of $\\alpha_k$ to obtain the last\nequality.  Now we turn our attention to the denominator. Using\nProperty 6 of Lemma~\\ref{lemma:propertiesCGoneiter}, we have that\n\\[\ny_{j+1}-y_i = \\sum_{k=i}^{j} \\alpha_k p_k =\n\\sum_{k=i}^{j} \\alpha_k \\left(-\\sum_{\\ell=0}^k \\frac{\\|r_k\\|^2}{\\|r_\\ell\\|^2} r_\\ell \\right),\n\\]\nBy rearranging the terms in the sum, we obtain\n\\begin{equation*}\ny_{j+1}-y_i = -\\sum_{k=i}^{j} \\sum_{\\ell=0}^k \\alpha_k\\|r_k\\|^2 \\frac{r_\\ell}{\\|r_{\\ell}\\|^2} \n= -\\sum_{\\ell=0}^j \\left[\\sum_{k=\\max\\{\\ell,i\\}}^j \\alpha_k \\|r_k\\|^2 \\right] \\frac{r_\\ell}{\\|r_\\ell\\|^2}.\n\\end{equation*}\nUsing the fact that the residuals $\\{ r_{\\ell} \\}_{\\ell=0,1,\\dotsc,j}$\nform an orthogonal set (by Property 2 of\nLemma~\\ref{lemma:propertiesCGoneiter}), we have that\n\\begin{equation*}\n  \\|y_{j+1}-y_i\\|^2 = \\sum_{\\ell=0}^j \\frac{1}{\\|r_{\\ell} \\|^2}\n  \\left[\\sum_{k=\\max\\{\\ell,i\\}}^j \\alpha_k \\|r_k\\|^2 \\right]^2.\n\\end{equation*}\nCombining this with~\\eqref{eq:numeratorcomp} gives the desired result.\n\\end{proof}","sha256_utf8":"1f8b16b93d8b82662c6f812f5ca40ac803fe720e2d6d8f99d09bfa6023369368"},{"start":129473,"end":144313,"start_line":2782,"end_line":3109,"raw_tex":"\\begin{proof}\n  The proof proceeds by contradiction: If we assume that all\n  conditions specified in the statement of the theorem hold, and in\n  addition that\n\\begin{equation} \\label{eq:curvcondyKp1}\n\\frac{(y_{\\hIcg +1}-y_i)^\\top \\bH (y_{\\hIcg+1}-y_i)}\n     {\\|y_{\\hIcg+1}-y_i\\|^2} \\ge \\eps, \\quad \\mbox{for all\n       $i=0,1,\\dotsc,\\hIcg-1$,}\n\\end{equation}\nthen we must have \n\\begin{equation} \\label{eq:cvCGwhilestronglycvx}\n\\|r_{\\hIcg}\\| \\le \\sqrt{T} \\tau^{\\hIcg/2} \\|r_0\\|,\n\\end{equation}\ncontradicting \\eqref{eq:Kitsstronglycvx}. Note that\n\\begin{equation} \\label{eq:casehK}\n\t\t\\frac{(y_{\\hIcg +1}-y_{\\hIcg})^\\top \\bH (y_{\\hIcg+1}-y_{\\hIcg})}\n\t\t{\\|y_{\\hIcg+1}-y_{\\hIcg}\\|^2} = \n\t\t\\frac{\\alpha_{\\hIcg} p_{\\hIcg}^\\top \\bH (\\alpha_{\\hIcg} p_{\\hIcg})}\n\t\t{\\|\\alpha_{\\hIcg} p_{\\hIcg}\\|^2} = \\frac{p_{\\hIcg}^\\top \\bH p_{\\hIcg}}\n\t\t{\\|p_{\\hIcg}\\|^2} \\ge \\eps\n\\end{equation}\nby assumption, therefore we can consider \\eqref{eq:curvcondyKp1} to\nhold for $i=0,1,\\dotsc,\\hIcg$. Moreover, recalling the definition\n\\eqref{eq:quadraticCG} of the quadratic function $q$, we have for any\n$i=0,1,\\dotsc,\\hIcg$ that \n\\[\n\t\tq(y_{\\hIcg+1}) = q(y_i) + \\nabla q(y_i)^\\top (y_{\\hIcg+1}-y_i) + \n\t\t\\frac{1}{2}(y_{\\hIcg+1}-y_i)^\\top \\bH (y_{\\hIcg+1}-y_i).\n                \\]\nThus, \\eqref{eq:curvcondyKp1} can be reformulated as\n\\begin{equation} \\label{eq:stronglycvxyKp1yj}\n% \t\t\\forall i \\in \\{0,\\dotsc,\\hIcg\\}, \\quad \n\t\tq(y_{\\hIcg+1}) \\ge q(y_i) + r_i^\\top (y_{\\hIcg+1}-y_i) \n\t\t+ \\frac{\\eps}{2}\\|y_{\\hIcg+1}-y_i\\|^2, \\quad\n                \\mbox{for all $i=0,1,\\dotsc,\\hIcg$,}\n\\end{equation}\nwhere we used $\\nabla q(y_i)=r_i$ and the definitions\n\\eqref{eq:curvcondyKp1}, \\eqref{eq:casehK} of strong convexity along\nthe directions $y_{\\hIcg+1}-y_i$. In the remainder of the proof, and\nsimilarly to \\cite[Proof of Proposition\n  1]{YCarmon_JCDuchi_OHinder_ASidford_2017b}, we will show that\n\\eqref{eq:stronglycvxyKp1yj} leads to the contradiction\n\\eqref{eq:cvCGwhilestronglycvx}, thus proving that\n\\eqref{eq:yKp1negcurv} holds.\n\nWe define the  sequence of functions $\\Phi_j$, $j=0,1,\\dotsc\\hIcg$ as follows:\n\\[\n    \\Phi_0(z) := q(y_0) + \\frac{\\eps}{2}\\|z-y_0\\|^2,\n    \\]\n    and for $j=0,\\dotsc,\\hIcg-1$:\n\\begin{equation} \n    \\label{eq:formulaPhijplus1}\n\\Phi_{j+1}(z)  := \\tau\\Phi_j(z) + (1-\\tau)\\left( q(y_j) + r_j^\\top\n(z-y_j) +\\frac{\\eps}{2}\\|z-y_j\\|^2 \\right).\n\\end{equation}\nSince each $\\Phi_j$ is a quadratic function with Hessian $\\eps I$, it\ncan be written as follows:\n\\begin{equation} \\label{eq:Phijwithvj}\n\t\t\\Phi_j(z) \\; = \\; \\Phi_j^* + \\frac{\\eps}{2}\\|z-v_j\\|^2,\n\\end{equation}\nwhere $v_j$ is the unique minimizer of $\\Phi_j$, and $\\Phi_j^* =\n\\Phi_j(v_j)$ is the minimum value of $\\Phi_j$. (Note that $v_0=y_0=0$\nand $\\Phi^*_0 = q(y_0) = 0$.)\n\nDefining\n\\begin{equation} \\label{eq:defpsi}\n\\psi(y) := q(y_0) - q(y) + \\frac{\\eps}{2}\\|y-y_0\\|^2 = \\Phi_0(y) - q(y),\n\\end{equation}\nwe give a short inductive argument to establish that \n\\begin{equation} \\label{eq:PhiYJp1}\n\\Phi_j(y_{\\hIcg+1}) \\le q(y_{\\hIcg+1}) + \\tau^j \\psi(y_{\\hIcg+1}),\n\\quad j=0,1,\\dotsc,\\hIcg.\n\\end{equation}\nFor $j=0$, \\eqref{eq:PhiYJp1} holds because $\\Phi_0(y) = q(y) +\n\\psi(y)$ by definition. Assuming that \\eqref{eq:PhiYJp1} holds for\nsome index $j \\ge 0$, we find by first applying\n\\eqref{eq:formulaPhijplus1} (with $z=y_{\\hIcg+1}$) and then\n\\eqref{eq:stronglycvxyKp1yj} (with $i=j$) that\n\\begin{eqnarray*}\n\\Phi_{j+1}(y_{\\hIcg+1}) &= &\\tau\\Phi_j(y_{\\hIcg+1}) + (1-\\tau)\\left( q(y_j) + \nr_j^\\top (y_{\\hIcg+1}-y_j) +\\frac{\\eps}{2}\\|y_{\\hIcg+1}-y_j\\|^2 \\right) \\\\\n&\\le &\\tau\\Phi_j(y_{\\hIcg+1}) + (1-\\tau) q(y_{\\hIcg+1}).\n\\end{eqnarray*}\nThus, we have\n\\begin{align*}\n\\Phi_{j+1}(y_{\\hIcg+1}) &\\le \\tau\\Phi_j(y_{\\hIcg+1}) + (1-\\tau) q(y_{\\hIcg+1}) \\\\\n&\\le \\tau q(y_{\\hIcg+1}) + \\tau^{j+1} \\psi(y_{\\hIcg+1}) + \n(1-\\tau) q(y_{\\hIcg+1}) \\quad \\mbox{from \\eqref{eq:PhiYJp1}} \\\\\n&= q(y_{\\hIcg+1}) + \\tau^{j+1} \\psi(y_{\\hIcg+1}),\n\\end{align*}\nwhich proves \\eqref{eq:PhiYJp1} for $j+1$, and thus completes the\ninductive argument.\n\nWe next prove another technical fact about the relationship between\n$q(y_j)$ and $\\Phi_j^*$, namely,\n\\begin{equation} \\label{eq:468}\n  q(y_j) \\le \\Phi_j^*, \\quad j=0,1,\\dotsc, \\hIcg.\n\\end{equation}\nWe establish this result by an inductive argument that is quite\nlengthy and technical; we note the termination of this phase of the\nproof clearly below.\n\n\nThis result trivially holds (with equality) at $j=0$.  Supposing that\nit holds for some $j =0,1,\\dotsc, \\hIcg-1$, we will prove that it also\nholds for $j+1$.\n\nBy using Properties 7 and 8 of Lemma~\\ref{lemma:propertiesCGoneiter},\n%and also $\\|\\bH \\| \\le (M+2\\eps)$, we have\n\\refer{and also $\\|\\bH r_j\\| \\le (M+2\\eps)\\|r_j\\|$,} we have\n\\begin{equation*} \n\tq(y_{j+1}) = q(y_j) - \\frac{\\|r_j\\|^4}{2\\, p_j^\\top \\bH p_j} \\le \n\tq(y_j) - \\frac{\\|r_j\\|^4}{2\\, r_j^\\top \\bH r_j} \\le \n\tq(y_j) - \\frac{\\|r_j\\|^2}{2(M+2\\eps)}.\n\\end{equation*}\nIt follows from induction hypothesis $q(y_j) \\le \\Phi_j^*$ that \n\\begin{align} \\label{eq:qyjp1Phijstar}\nq(y_{j+1}) \\le q(y_j) - \\frac{\\|r_j\\|^2}{2(M+2\\eps)}\n&= \\tau q(y_j) + (1-\\tau) q(y_j) -\\frac{\\|r_j\\|^2}{2(M+2\\eps)} \\nonumber \\\\\n&\\le \\tau \\Phi_j^* + (1-\\tau) q(y_j) -\\frac{\\|r_j\\|^2}{2(M+2\\eps)}.\n\\end{align}\nBy taking the derivative in \\eqref{eq:formulaPhijplus1}, and using\n\\eqref{eq:Phijwithvj}, we obtain\n\\begin{align*}\n\\nabla \\Phi_{j+1}(z) &= \\tau \\nabla \\Phi_j(z) + (1-\\tau)\\left[r_j + \\eps(z-y_j)\\right] \\\\\n\\Rightarrow \\;\\; \\eps(z-v_{j+1})&=  \\eps \\tau (z-v_j) + (1-\\tau) \\left(r_j + \\eps (z-y_j) \\right).\n\\end{align*}\nBy rearranging the above relation (and noting  that the $z$ terms cancel out), we obtain:\n\\begin{equation} \\label{eq:vjp1minPhijp1}\n\tv_{j+1}  \\; = \\; \\tau v_j - \\frac{1-\\tau}{\\eps}r_j + (1-\\tau) y_j.\n\\end{equation}\nIt follows from this expression together with Properties 1 and 2 of\nLemma~\\ref{lemma:propertiesCGoneiter} that \n\\begin{equation} \\label{eq:vjinKj}\nv_j \\in \\lspan\\left\\{p_0,p_1,\\dotsc,p_{j-1}\\right\\}, \\quad j=1,2,\\dotsc,\\hIcg.\n\\end{equation}\n(The result holds for $j=1$, from \\eqref{eq:vjp1minPhijp1} we have\n$v_1 \\in \\lspan \\{v_0, r_0, y_0\\} = \\lspan \\{r_0\\} = \\lspan \\{p_0 \\}$,\nand an induction based on \\eqref{eq:vjp1minPhijp1} can be used to\nestablish \\eqref{eq:vjinKj} for the other values of $j$.)  By\ncombining the expressions~\\eqref{eq:Phijwithvj} for $\\Phi_j$,\n$\\Phi_{j+1}$ with the recurrence formula~\\eqref{eq:formulaPhijplus1}\nfor $\\Phi_{j+1}$, we obtain\n\\begin{align*} \n\t\t\\Phi_{j+1}^* + \\frac{\\eps}{2}\\|y_j-v_{j+1}\\|^2 &= \\Phi_{j+1}(y_j) \\\\\n%\t\t&= \\tau \\Phi_j(y_j) + (1-\\tau)\\left(q(y_j)+r_j^\\top (y_j-y_j) + \n%\t\t\\frac{\\eps}{2}\\|y_j-y_j\\|^2\\right)  \\\\\n\t\t&= \\tau\\Phi_j(y_j) + (1-\\tau) q(y_j) \\\\\n                &= \\tau\\left(\\Phi_j^* + \n\t\t\\frac{\\eps}{2}\\|y_j-v_j\\|^2\\right) + (1-\\tau) q(y_j)\n\\end{align*}\nand therefore\n\\begin{equation} \\label{eq:doubleexpressionPhijp1}\n  \\Phi_{j+1}^* = \\tau \\left(\\Phi_j^* +\n  \\frac{\\eps}{2}\\|y_j-v_j\\|^2\\right) + (1-\\tau) q(y_j) -\n  \\frac{\\eps}{2}\\|y_j-v_{j+1}\\|^2.\n\\end{equation}\nOn the other hand, we have by~\\eqref{eq:vjp1minPhijp1} that\n\\begin{eqnarray} \\label{eq:normylminusvlp1}\n\t\t\\|y_j-v_{j+1}\\|^2 &= &\\left\\| \\tau (y_j-v_j) \n\t\t+\\frac{1-\\tau}{\\eps}r_j \\right\\|^2 \\nonumber \\\\\n\t\t&= &(\\tau^2 \\|y_j-v_j\\|^2 + \\frac{(1-\\tau)^2}{\\eps^2}\\|r_j\\|^2 \n\t\t+ \\frac{2}{\\eps}(1-\\tau)\\tau r_j^\\top (y_j-v_j) \\nonumber \\\\\n\t\t&= & \\tau^2 \\|y_j-v_j\\|^2 + \\frac{(1-\\tau)^2}{\\eps^2}\\|r_j\\|^2,\n\\end{eqnarray}\nwhere the last relation comes from $r_j \\perp\n\\lspan\\{p_0,\\dotsc,p_{j-1}\\}$ (Property 2 of\nLemma~\\ref{lemma:propertiesCGoneiter}) and \\eqref{eq:vjinKj} in the\ncase $j \\ge 1$, and immediately in the case $j=0$, since $y_0=v_0=0$.\nBy combining \\eqref{eq:doubleexpressionPhijp1}\nand~\\eqref{eq:normylminusvlp1}, we arrive at:\n\\begin{align} \\label{eq:phijp1star}\n\t\t\\Phi_{j+1}^* &= \\tau \\left(\\Phi_j^* + \n\t\t\\frac{\\eps}{2}\\|y_j-v_j\\|^2\\right) + (1-\\tau) q(y_j) \n\t\t- \\frac{\\eps}{2}\\|y_j-v_{j+1}\\|^2 \\nonumber \\\\\n\t\t&= \\tau \\left(\\Phi_j^* + \n\t\t\\frac{\\eps}{2}\\|y_j-v_j\\|^2\\right) + (1-\\tau) q(y_j) \n\t\t-\\frac{\\eps}{2}\\tau^2 \\|y_j-v_j\\|^2 - \\frac{(1-\\tau)^2}{2\\eps} \\|r_j\\|^2 \n\t\t\\nonumber \\\\\n\t\t&= \\tau \\Phi_j^* + \\frac{\\eps}{2}\\left[ \\tau \n\t\t-\\tau^2\\right]\\|y_j-v_j\\|^2 + (1-\\tau) q(y_j) \n\t\t- \\frac{(1-\\tau)^2}{2\\eps} \\|r_j\\|^2 \\nonumber \\\\ \n\t\t&= \\tau \\Phi_j^* + \n\t\t\\frac{\\eps}{2}(1-\\tau)\\tau \\|y_j-v_j\\|^2 + (1-\\tau) q(y_j) \n\t\t- \\frac{(1-\\tau)^2}{2\\eps} \\|r_j\\|^2 \\nonumber \\\\\n\t\t&\\ge \\tau \\Phi_j^* + (1-\\tau) q(y_j) \n\t\t- \\frac{(1-\\tau)^2}{2\\eps} \\|r_j\\|^2 \\nonumber \\\\\n                &\\ge q(y_{j+1}) + \\frac{1}{2(M+2\\eps)}\\|r_j\\|^2 \n\t\t- \\frac{(1-\\tau)^2}{2\\eps} \\|r_j\\|^2.\n\\end{align}\nwhere the last inequality comes from~\\eqref{eq:qyjp1Phijstar}.  By\nusing the definitions of $\\tau$ and $\\kappa$ in\nAlgorithm~\\ref{alg:ccg}, we have\n\\[\n\t\t\\frac{(1-\\tau)^2}{2\\eps} = \\frac{1}{2\\eps(\\sqrt{\\kappa}+1)^2} \\le \n\t\t\\frac{1}{2\\eps\\kappa} = \\frac{1}{2(M+2\\eps)}.\n\\]\nIt therefore follows from \\eqref{eq:phijp1star} that $q(y_{j+1}) \\le\n\\Phi_{j+1}^*$. At this point, we have shown that when $q(y_j) \\le\n\\Phi_j^*$ for $j=0,1,\\dotsc,\\hIcg-1$, it follows that $q(y_{j+1}) \\le\n\\Phi_{j+1}^*$, establishing the inductive step. As a result, our\nproof of \\eqref{eq:468} is complete.\n\n\nBy substituting $j=\\hIcg$ into \\eqref{eq:468}, we obtain\n$q(y_{\\hIcg}) \\le \\Phi_{\\hIcg}^*$,\nwhich in combination with \\eqref{eq:PhiYJp1} with $j=\\hIcg$, and the\ndefinition \\eqref{eq:Phijwithvj}, yields\n\\begin{equation} \\label{eq:desiredecreaseformula}\nq(y_{\\hIcg}) -q(y_{\\hIcg+1}) \\le \\Phi_{\\hIcg}^* - q(y_{\\hIcg+1}) \n\\le \\Phi_{\\hIcg}(y_{\\hIcg+1}) - q(y_{\\hIcg+1})\n\\le \\tau^{\\hIcg} \\psi(y_{\\hIcg+1}).\n\\end{equation}\nBy substitution from \\eqref{eq:defpsi}, we obtain\n\\begin{equation} \\label{eq:funcdecreasecontrad}\nq(y_{\\hIcg}) - q(y_{\\hIcg+1}) \\; \\le \\; \\tau^{\\hIcg}\n\\left(q(y_0)-q(y_{\\hIcg+1}) +\n\\frac{\\eps}{2}\\|y_0-y_{\\hIcg+1}\\|^2\\right).\n\\end{equation}\n\nWe now depart from the analysis\nof~\\cite{YCarmon_JCDuchi_OHinder_ASidford_2017b}, and complete the\nproof of this result by expressing \\eqref{eq:funcdecreasecontrad} in\nterms of residual norms.  On the left-hand side, we have\n\\begin{align*}\n  q(y_{\\hIcg}) - q(y_{\\hIcg+1})\n  % &= &\\nabla q(y_{\\hIcg+1})^\\top (y_{\\hIcg}-y_{\\hIcg+1}) + \n%\t\t\\frac{1}{2}(y_{\\hIcg}-y_{\\hIcg+1})^\\top \\bH (y_{\\hIcg}-y_{\\hIcg+1})  \\\\ \n\t\t&= r_{\\hIcg+1}^\\top (y_{\\hIcg}-y_{\\hIcg+1}) + \n\t\t\\frac{1}{2}(y_{\\hIcg}-y_{\\hIcg+1})^\\top \\bH (y_{\\hIcg}-y_{\\hIcg+1})  \\\\\n\t\t&= \\frac{1}{2}(y_{\\hIcg}-y_{\\hIcg+1})^\\top \\bH (y_{\\hIcg}-y_{\\hIcg+1})\n\\end{align*}\nbecause $r_{\\hIcg+1}^\\top (y_{\\hIcg}-y_{\\hIcg+1}) = r_{\\hIcg+1}^\\top\n(\\alpha_{\\hIcg} p_{\\hIcg}) = 0$ by\nLemma~\\ref{lemma:propertiesCGoneiter}, Property 2.\nWe thus have from \\eqref{eq:casehK} that\n\\begin{alignat}{2}\n  \\nonumber\n  q(y_{\\hIcg}) - q(y_{\\hIcg+1}) &\\ge \\frac{\\eps}{2}\\|y_{\\hIcg}-y_{\\hIcg+1}\\|^2  &&  \\\\\n  \\nonumber\n  &\\refer{= \\frac{\\eps}{2} \\|\\alpha_{\\hIcg}p_{\\hIcg}\\|^2} \\\\\n  \\nonumber\n  &\\refer{\\ge \\frac{\\eps}{2 (M+2\\eps)^2}\\|\\bH(\\alpha_{\\hIcg}p_{\\hIcg})\\|^2} \n  && \\mbox{(since $\\|\\bH p_{\\hIcg}\\| \\le (M+2\\eps)\\|p_{\\hIcg}\\|$)} \\\\\n  \\nonumber\n%  &\\ge \\frac{\\eps}{2 (M+2\\eps)^2} \\|\\bH(y_{\\hIcg}-y_{\\hIcg+1})\\|^2  && \\mbox{(since $\\bH \\preceq (M+2\\eps)I$)} \\\\\n  &= \\frac{\\eps}{2 (M+2\\eps)^2} \\|\\bH(y_{\\hIcg}-y_{\\hIcg+1})\\|^2   \\\\\n  \\nonumber\n  &= \\frac{\\eps}{2 (M+2\\eps)^2} \\|r_{\\hIcg} - r_{\\hIcg+1}\\|^2  && \\mbox{(since $r_j=g+\\bH y_j$)} \\\\\n  \\nonumber\n\t\t&= \\frac{\\eps}{2 (M+2\\eps)^2} (\\|r_{\\hIcg}\\|^2 + \\|r_{\\hIcg+1}\\|^2) & \\quad & \\mbox{(by Lemma~\\ref{lemma:propertiesCGoneiter}, Property 2)} \\\\\n  \\label{eq:funcdeccontradLHS}\n\t\t& \\ge \\frac{\\eps}{2 (M+2\\eps)^2} \\|r_{\\hIcg}\\|^2,\n\\end{alignat}\n\n\nOn the right-hand side of~\\eqref{eq:funcdecreasecontrad}, because of\nthe strong convexity condition \\eqref{eq:stronglycvxyKp1yj} at $i=0$,\nwe have\n\\begin{align*}\nq(y_0)-q(y_{\\hIcg+1}) + \\frac{\\eps}{2}\\|y_0-y_{\\hIcg+1}\\|^2  & \\le \n-\\nabla q(y_0)^\\top (y_{\\hIcg+1}-y_0) \\\\\n&= -r_0^\\top (y_{\\hIcg+1}-y_0)  \\le \\|r_0\\|\\|y_{\\hIcg+1}-y_0\\|.\n\\end{align*}\nMoreover, we have \n\\[\n\t\t\\|y_{\\hIcg+1}-y_0\\| \\; = \\;\n\t\t\\left\\|\\sum_{i=0}^{\\hIcg} \\alpha_i p_i \\right\\| \n\t\t\\; \\le \\; \\sum_{i=0}^{\\hIcg} \\alpha_i \\|p_i\\| \n\t\t\\; = \\; \\sum_{i=0}^{\\hIcg} \\frac{\\|r_i\\|^2}{p_i^\\top \\bH p_i}\\|p_i\\|,\n\\]\nwhere the last relation follows from the definition of $\\alpha_i$. By\ncombining these last two bounds, and using Property 3 of\nLemma~\\ref{lemma:propertiesCGoneiter}, we obtain\n\\begin{equation} \\label{eq:wd12}\n\t\tq(y_0)-q(y_{\\hIcg+1}) + \\frac{\\eps}{2}\\|y_0-y_{\\hIcg+1}\\|^2 \\; \\le \\; \n\t\t\\|r_0\\| \\sum_{i=0}^{\\hIcg} \\|r_i\\| \\frac{\\|p_i\\|^2}{p_i^\\top \\bH p_i} \\;\n\t\t\\le \\; \\|r_0\\| \\frac{1}{\\eps} \\sum_{i=0}^{\\hIcg} \\|r_i\\|,\n\\end{equation}\nwhere the last inequality holds $p_j^\\top \\bH p_j \\ge \\eps \\|p_j\\|^2$\nfor $j=0,1,\\dotsc,\\hIcg$ by assumption.\n\nTo bound the sum in \\eqref{eq:wd12}, we recall that since\nAlgorithm~\\ref{alg:ccg} did not terminate until iteration $\\hIcg$,\n\\refer{the residual norms $\\|r_i \\|$ at all iterations\n  $i=0,1,\\dotsc,\\hIcg-1$ must have decreased at the expected\n  convergence rate. In particularly, we have $\\|r_i\\| \\le \\sqrt{T}\n  \\tau^{i/2} \\|r_0\\|$ for the {\\em possibly smaller} versions of\n  $\\sqrt{T}$ and $\\tau$ that prevailed at iteration $i$, so certainly\n  $\\|r_i\\| \\le \\sqrt{T} \\tau^{i/2} \\|r_0\\|$ for the final values of\n  these parameters.} Thus for $i=0,1,\\dotsc,\\hIcg-1$, we have\n\\[\n\\|r_i\\| \\le \\sqrt{T} \\tau^{i/2} \\|r_0\\| \\le \\tau^{(i-\\hIcg)/2}\\|r_{\\hIcg}\\|,\n\\]\nwhere we used $\\|r_{\\hIcg}\\| \\ge \\sqrt{T} \\tau^{\\hIcg/2} \\|r_0\\|$\n(from \\eqref{eq:Kitsstronglycvx}) for the last inequality. Observing\nthat this bound also holds (trivially) for $i=\\hIcg$, we obtain by\nsubstituting in \\eqref{eq:wd12} that\n\\begin{align}\n  \\nonumber\nq(y_0)-q(y_{\\hIcg+1}) + \\frac{\\eps}{2}\\|y_0-y_{\\hIcg+1}\\|^2\n&\\le \\|r_0\\| \\frac{1}{\\eps} \\sum_{i=0}^{\\hIcg} \\tau^{(i-\\hIcg)/2} \\|r_{\\hIcg}\\| \\\\\n\\nonumber\n&\\le \\|r_0\\| \\frac{\\tau^{-\\hIcg/2}}{\\eps}\\|r_{\\hIcg}\\|\\,\\sum_{i=0}^{\\hIcg} (\\sqrt{\\tau})^{i} \\\\\n\\label{eq:funcdeccontradRHS}\n& \\le \\|r_0\\| \\frac{\\tau^{-\\hIcg/2}}{\\eps}\\|r_{\\hIcg}\\| \\frac{1}{1-\\sqrt{\\tau}}.\n\\end{align}\nApplying successively \\eqref{eq:funcdeccontradLHS},\n\\eqref{eq:funcdecreasecontrad} and \\eqref{eq:funcdeccontradRHS}\nfinally yields:\n\\begin{align*}\n\t\t\\frac{\\eps}{2 (M+2\\eps)^2} \\|r_{\\hIcg}\\|^2 &\\le \n\t\t q(y_{\\hIcg}) - q(y_{\\hIcg+1}) \\\\\n\t\t &\\le \\tau^{\\hIcg} \\left( q(y_0)-q(y_{\\hIcg+1}) + \\frac{\\eps}{2}\\|y_0-y_{\\hIcg+1}\\|^2 \\right) \\\\\n\t\t &\\le \\tau^{\\hIcg} \\|r_0\\|\\|r_{\\hIcg}\\| \n\t\t \\frac{\\tau^{-\\hIcg/2}}{\\eps}\\frac{1}{1-\\sqrt{\\tau}}.\n\\end{align*}\nAfter rearranging this inequality and dividing by $\\|r_{\\hIcg}\\| > 0$,\nwe obtain\n\\begin{equation} \\label{eq:finalcontrad}\n\t\t\\|r_{\\hIcg}\\| \\le \\frac{2 (M+2\\eps)^2}{\\eps^2}\\frac{\\tau^{\\hIcg/2}}{1-\\sqrt{\\tau}} \\|r_0\\|\n\t\t= \\sqrt{T} \\tau^{\\hIcg/2}\\|r_0\\|.\n\\end{equation}\nWe have thus established \\eqref{eq:cvCGwhilestronglycvx} which, as we\nnoted earlier, contradicts \\eqref{eq:Kitsstronglycvx}. Thus\n\\eqref{eq:curvcondyKp1} cannot be true, so we have established\n\\eqref{eq:yKp1negcurv}, as required.\n\\end{proof}","sha256_utf8":"af23d3db3e17ee4f007e6c68fc6958aaa56231366f4ed76ade8cdb84f94e24fe"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"光滑非凸最小化问题","source_units":[{"start":7042,"end":7164,"start_line":215,"end_line":218,"raw_tex":"We consider the unconstrained optimization problem\n\\begin{equation} \\label{eq:f}\n\\min_{x \\in \\R^n} \\, f(x),\n\\end{equation}","sha256_utf8":"5b65f7ee2a844b7be1d9e235485067d21a64897237b7942efc6c77700e2d397f"},{"start":7165,"end":7371,"start_line":219,"end_line":221,"raw_tex":"where $f:\\R^n \\to \\R$ is a twice Lipschitz continuously differentiable\nfunction that is nonconvex in general. We further assume that $f$ is\nbounded below for all $x$, by some constant $\\flow$. Although the\n","sha256_utf8":"787292d971d208b8d82caab930631ab50c785dde78f53a4e5d353597bb6ed81f"}]},{"id":"S002","semantic_type":"definition","kind":"definition","title":"近似二阶最优条件","source_units":[{"start":12138,"end":12379,"start_line":300,"end_line":305,"raw_tex":"Our algorithm seeks a point that approximately satisfies second-order\nnecessary conditions for optimality, that is,\n\\begin{equation}\\label{eq:2epson}\n\\| \\nabla f (x)\\| \\le \\epsg, \\quad \\lambda_{\\min}(\\nabla^2 f(x)) \\ge\n-\\epsH,\n\\end{equation}","sha256_utf8":"3d19eb2c78393fab5cae74a7b0ed115c8082e9571391f926c40379df3fff5d10"},{"start":12380,"end":12542,"start_line":306,"end_line":308,"raw_tex":"for specified small positive tolerances $\\epsg$ and $\\epsH$. (Here and\nsubsequently, $\\| \\cdot \\|$ denotes the Euclidean norm, or its induced\nnorms on matrices.)\n","sha256_utf8":"10421431718a7dbb43039a023bb4b4a5fe0a6bd5e734fee23c1e3080c5ac6fec"}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","title":"初始水平集紧致","source_units":[{"start":12686,"end":12824,"start_line":313,"end_line":316,"raw_tex":"\\begin{assumption} \\label{assum:compactlevelset}\nThe level set $\\mathcal{L}_f(x_0) = \\{x | f(x) \\le f(x_0)\\}$ is\ncompact.\n\\end{assumption}","sha256_utf8":"f49684da37a22ee987182f54e9fca4aab9f4af77c51586342bf07ea730169526"}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","title":"二阶导数利普希茨连续","source_units":[{"start":12826,"end":13513,"start_line":318,"end_line":330,"raw_tex":"\\begin{assumption} \\label{assum:fC22}\nThe function $f$ is twice uniformly Lipschitz continuously\ndifferentiable on an open neighborhood of $\\mathcal{L}_f(x_0)$\n\\refer{that includes the trial points generated by the algorithm. We\n  denote by $L_H$ the Lipschitz constant for $\\nabla^2 f$ on this\n  neighborhood}.\n%% \\crcomment{Removed $L_g$ because it is not used anywhere else in\n%% the paper and we already have multiple notations for Hessian norms\n%% or their approximations. We only need Lipschitz continuous\n%% Hessians, and bounded gradients/Hessians (the latter implying that\n%% the gradient is Lipschitz continuous).}\n%\\sjwcomment{I made changes. OK now? CR: Ok.}\n\\end{assumption}","sha256_utf8":"bb44a0c307759cc1c3bf6e92abcedc923eca43e79d3279b672174725a3dedce9"}]},{"id":"S005","semantic_type":"derived_fact","kind":"derived_fact","title":"函数及导数统一界","source_units":[{"start":13696,"end":13988,"start_line":336,"end_line":342,"raw_tex":"Under these two assumptions, there exist scalars $\\flow$, $U_g\n> 0$, and $U_H > 0$ such that the following are satisfied for $x \\in\n\\mathcal{L}_f(x_0)$:\n\\begin{equation}\\label{eq:boundscompact}\n\tf(x) \\geq \\flow,\\quad \\|\\nabla f(x)\\| \\leq U_g,\n\t\\quad \\|\\nabla^2 f(x)\\| \\leq U_H.\n\\end{equation}","sha256_utf8":"108e94175ccac0986892b5dd3bd319461945fe51073324c2912e73c7e27d9555"}]},{"id":"S006","semantic_type":"derived_fact","kind":"derived_fact","title":"三阶泰勒余项上界","source_units":[{"start":14247,"end":14486,"start_line":348,"end_line":353,"raw_tex":"For any $x$ and $d$ such that Assumption~\\ref{assum:fC22} is satisfied at $x$ \nand $x+d$, we have\n\\begin{equation} \\label{eq:LH}\nf(x+d) \\le f(x) + \\nabla f(x)^\\top d + \\frac12 d^\\top \\nabla^2 f(x) d\n+ \\frac{L_H}{6} \\| d\\|^3.\n\\end{equation}","sha256_utf8":"eda4a236202c7244a7c3f550626de5f3d7c2f99d961181b395cb8331b909b3e1"}]},{"id":"S007","semantic_type":"notation","kind":"notation","title":"复杂度阶记号","source_units":[{"start":14488,"end":14638,"start_line":355,"end_line":357,"raw_tex":"Notationally, we use order notation ${\\mathcal O}$ in the usual sense,\nwhereas $\\tcO(\\cdot)$ represents $\\cO$ with logarithmic terms\n\\refer{omitted}.\n","sha256_utf8":"9cc933a06214c0e51df80021cafb5f79a83a5b49dfb18417627b0c437d323b6c"}]},{"id":"S008","semantic_type":"algorithm","kind":"algorithm","title":"截断共轭梯度算法","source_units":[{"start":25078,"end":28150,"start_line":544,"end_line":614,"raw_tex":"\\begin{algorithm}[ht!]\n\\caption{Capped Conjugate Gradient}\n\\label{alg:ccg}\n\\begin{algorithmic}\n\\STATE \\emph{Inputs:} Symmetric matrix $H \\in \\R^{n \\times n}$; \nvector $g \\ne 0$; damping parameter $\\epsilon \\in (0,1)$; desired relative \naccuracy \\refer{$\\zeta \\in (0,1)$};\n\\STATE \\refer{\\emph{Optional input:} scalar $M$ (set to $0$ if not provided)};\n\\STATE \\refer{\\emph{Outputs:} d\\_type, $d$;}\n\\STATE \\refer{\\emph{Secondary outputs:} final values of $M$, $\\kappa$, \n$\\hat{\\zeta}$, $\\tau$, and $T$;}\n\\STATE Set\n\\[\n\\bH:=H+2\\eps I, \\quad \\kappa := \\frac{M+2\\eps}{\\eps}, \\quad \\hat{\\zeta} := \\frac{\\zeta}{3 \\kappa},\n\\quad\n\\tau := \\frac{\\sqrt{\\kappa}}{\\sqrt{\\kappa}+1}, \\quad\nT:=\\frac{4\\kappa^4}{(1-\\sqrt{\\tau})^2};\n\\]\n\\STATE $y_0 \\leftarrow 0$, $r_0 \\leftarrow g$, $p_0 \\leftarrow -g$, $j \\leftarrow 0$; \n\\IF{$p_0^\\top \\bH p_0 < \\eps \\|p_0\\|^2$}\n\\STATE Set $d=p_0$ and terminate with d\\_type=NC;\n\\ELSIF{\\refer{$\\|H p_0\\| > M \\|p_0\\|$}}\n\\STATE \\refer{Set $M \\leftarrow {\\|H p_0\\|}/{\\|p_0\\|}$ \nand update $\\kappa,\\hat{\\zeta},\\tau,T$ accordingly;}\n\\ENDIF\n\\WHILE{TRUE}\n\\STATE $\\alpha_j \\leftarrow {r_j^\\top r_j}/{p_j^\\top \\bH p_j}$;\n\\COMMENT{Begin Standard CG Operations}\n\\STATE $y_{j+1} \\leftarrow y_j+\\alpha_j p_j$;\n\\STATE $r_{j+1} \\leftarrow r_j + \\alpha_j \\bH p_j$;\n\\STATE $\\beta_{j+1} \\leftarrow {(r_{j+1}^\\top r_{j+1})}/{(r_j^\\top r_j)}$;\n\\STATE $p_{j+1} \\leftarrow  -r_{j+1} + \\beta_{j+1}p_j$;\n\\COMMENT{End Standard CG Operations}\n\\STATE $j \\leftarrow  j+1$;\n\\IF{\\refer{$\\|H p_j\\| > M \\|p_j\\|$}}\n\\STATE \\refer{Set $M \\leftarrow {\\|H p_j\\|}/{\\|p_j\\|}$ \nand update $\\kappa,\\hat{\\zeta},\\tau,T$ accordingly;}\n\\ENDIF\n\\IF{\\refer{$\\|H y_j\\| > M \\|y_j\\|$}}\n\\STATE \\refer{Set $M \\leftarrow {\\|H y_j\\|}/{\\|y_j\\|}$ \nand update $\\kappa,\\hat{\\zeta},\\tau,T$ accordingly;}\n\\ENDIF\n\\IF{\\refer{$\\| H r_j\\| > M \\|r_j\\|$}}\n\\STATE \\refer{Set $M  \\leftarrow {\\| H r_j\\|}/{\\|r_j\\|}$ \nand update $\\kappa,\\hat{\\zeta},\\tau,T$ accordingly;}\n\\ENDIF\n\\IF{$y_j^\\top \\bH y_j < \\epsilon \\|y_j\\|^2$}\n\\STATE Set $d \\leftarrow y_j$ and terminate with d\\_type=NC;\n%\\ELSIF{$\\| r_j \\| \\le \\tfrac{1}{2}\\min\\{\\zeta\\|r_0\\|,\\eps \\|y_j\\|\\}$}\n\\ELSIF{$\\| r_j \\| \\le \\hat{\\zeta} \\|r_0\\|$}\n\\STATE Set $d \\leftarrow y_j$ and terminate with d\\_type=SOL;\n\\ELSIF{$p_j^\\top \\bH p_j < \\eps \\|p_j\\|^2$}\n\\STATE Set $d \\leftarrow p_j$ and terminate with d\\_type=NC;\n\\ELSIF{$\\|r_j\\| >  \\sqrt{T}\\tau^{j/2} \\|r_0\\|$\n%\\crnote{The key here is \n%that when we terminate because of this condition, we are sure that \n%$\\|H v\\| \\le (M+2\\eps)\\|v\\|$ for $v\\in \\{p_0,\\dots,p_j,y_0,\\dots,y_j,r_0,\\dots,r_j\\}$ \n%and this is all we need. As long as the residual norm decreases at the \n%desired rate, things are okay because $\\sqrt{T}\\tau^{j/2}$ is an \n%increasing function of $M$.}\n}\n\\STATE Compute $\\alpha_j,y_{j+1}$ as in the main loop above;\n\\STATE Find $i \\in \\{0,\\dotsc,j-1\\}$ such that\n\\begin{equation} \\label{eq:weakcurvdir}\n\t\\frac{(y_{j+1}-y_i)^\\top \\bH (y_{j+1}-y_i)}{\\|y_{j+1}-y_i\\|^2} \\; < \\; \\eps;\n\\end{equation}\n\\STATE Set $d \\leftarrow y_{j+1}-y_i$ and terminate with d\\_type=NC;\n\\ENDIF\n\\ENDWHILE\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"3d966fbb58aaa7d9d655f157cb9c2d1d2ad30b57408f0d7bcd0a6e0c2259e293"}]},{"id":"S009","semantic_type":"algorithm","kind":"algorithm","title":"最小特征值预言机","source_units":[{"start":37931,"end":38768,"start_line":803,"end_line":821,"raw_tex":"\\begin{algorithm}[ht!]\n\\caption{Minimum Eigenvalue Oracle}\n\\label{alg:meo}\n\\begin{algorithmic}\n%  \\STATE \\emph{Inputs:} Symmetric matrix $H \\in \\R^{n \\times n}$,\n%  scalar $M$ such that $\\lambdamax(H) \\le M$, and $\\epsilon>0$;\n%  \\STATE Set parameter $\\delta \\in [0,1)$;\n  \\STATE \\emph{Inputs:} Symmetric matrix $H \\in \\R^{n \\times n}$,\n  \\refer{tolerance $\\epsilon>0$};\n  \\STATE \\refer{\\emph{Optional input:} Upper bound on Hessian norm \n  $M$;}\n  \\STATE \\emph{Outputs:} An estimate $\\lambda$ of $\\lambdamin(H)$\n  such that $\\lambda \\le - \\epsilon/2$, and vector $v$\n  with $\\|v\\|=1$ such that $v^\\top H v =\\lambda$ OR\n  a certificate that $\\lambdamin(H) \\ge -\\epsilon$. In the latter case, when\n  the certificate is output, it is false with probability $\\delta$ \n  \\refer{for some $\\delta \\in [0,1)$}.\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"ecd8559f46df395c10d42d6abc1f049f145c34bfc10835d5572e407c193559e9"}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","title":"阻尼牛顿共轭梯度算法","source_units":[{"start":47561,"end":49332,"start_line":1001,"end_line":1046,"raw_tex":"\\begin{algorithm}[ht!]\n\\caption{Damped Newton-CG}\n\\label{alg:dncg}\n\\begin{algorithmic}\n\\STATE \\emph{Inputs:} Tolerances  $\\epsg>0$, $\\epsH>0$; backtracking \nparameter $\\theta \\in (0,1)$; starting point $x_0$; \n\\refer{accuracy parameter $\\zeta \\in (0,1)$;}\n\\STATE \\refer{\\emph{Optional input:} Upper bound  $M > 0$ on Hessian \nnorm};\n%\\STATE \\refer{\\emph{Optional input:} Upper bound  $U_H > 0$ on Hessian \n%norm}; \n\\FOR{$k=0,1,2,\\dotsc$}\n\\IF{$\\|\\nabla f(x_k)\\| > \\epsg$}\n\\STATE Call Algorithm~\\ref{alg:ccg} with $H = \\nabla^2 f(x_k)$, $\\epsilon=\\epsH$, \n$g=\\nabla f(x_k)$, accuracy parameter $\\zeta$ and $M$ if provided, \nto obtain outputs $d$, d\\_type; \n\\IF{d\\_type=NC}\n\\STATE $d_k \\leftarrow -\\mathrm{sgn}(d^\\top g) \\frac{|d^\\top \\nabla^2 f(x_k) d|}{\\|d\\|^2} \n\\frac{d}{\\|d\\|}$;\n\\ELSE[\\refer{d\\_type=SOL}]\n\\STATE $d_k \\leftarrow d$;\n\\ENDIF\n\\STATE Go to \\textbf{Line Search};\n\\ELSE\n\\STATE Call Procedure~\\ref{alg:meo} with $H = \\nabla^2 f(x_k)$,\n$\\epsilon=\\epsH$ and $M$ if provided; \n\\IF{Procedure~\\ref{alg:meo} certifies that $\\lambdamin(\\nabla^2 f(x_k)) \\ge -\\epsH$}\n\\STATE\nTerminate; \n\\ELSE[\\refer{direction of sufficient negative curvature found}]\n\\STATE \n$d_k \\leftarrow -\\mathrm{sgn}(v^\\top g) \\frac{|v^\\top \\nabla^2 f(x_k) v|}{\\|v\\|^2} v$\n% \\sjwnote{Holy smokes, the last $v$ was missing from this formula!}\n(where $v$ is the output from Procedure~\\ref{alg:meo})\nand go to \\textbf{Line Search};\n\\ENDIF\n\\ENDIF\n\\STATE \\textbf{Line Search}: Compute a step length $\\alpha_k=\\theta^{j_k}$, \nwhere $j_k$ is the smallest nonnegative integer such that\n\\begin{equation} \\label{eq:lsdecreasedamped}\n\tf(x_k + \\alpha_k d_k) < f(x_k) - \\frac{\\eta}{6}\\alpha_k^3 \\|d_k\\|^{3};\n\\end{equation}\n\\STATE $x_{k+1} \\leftarrow x_k+\\alpha_k d_k$;\n\\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"98429eed679a23c4caf32899829f9acaca9dcfed0cd6c94adf9cff69a1c864f6"}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","title":"预言机概率与工作量假设","source_units":[{"start":70592,"end":71655,"start_line":1543,"end_line":1562,"raw_tex":"\\begin{assumption} \\label{assum:wccmeo}\nFor every iteration $k$ at which Algorithm~\\ref{alg:dncg} calls\nProcedure~\\ref{alg:meo}, and for a specified failure probability\n$\\delta$ with $0 \\le \\delta \\ll 1$, Procedure~\\ref{alg:meo} either\ncertifies that $\\nabla^2 f(x_k) \\succeq -\\epsH I$ or finds a vector\nof curvature smaller than $-{\\epsH}/{2}$ in at most\n%% \\crcomment{Changed the value to reflect our new bounds on the\n%% number of iterations.}\n\\begin{equation} \\label{eq:wccmeo}\n\t\tN_{\\mathrm{meo}}:=\\min\\left\\{n,\n\t\t1+\\left\\lceil\\Cmeo\\epsH^{-1/2}\\right\\rceil\\right\\}\n\\end{equation}\nHessian-vector products, \\refer{with probability $1-\\delta$,} where\n$\\Cmeo$ depends at most logarithmically on $\\delta$ and $\\epsH$.\n%\\sjwcomment{I changed some more. Is this OK? CR: Yes.}\n%% \\crcomment{Modified the last part of the assumption to cope with \n%% the discussion before Corollary~\\ref{coro:wcc2Hv}, where $\\Cmeo$ depends \n%% on $\\epsH$ through $\\delta$. That also avoids mentioning this logarithmic \n%% dependency in the statement of the corollary.}\n\\end{assumption}","sha256_utf8":"4bb5593da9042da8bc842258bf109167816e55cf4d9d876d1c44d95edd790aa9"}]},{"id":"S012","semantic_type":"algorithm","kind":"algorithm","title":"标准共轭梯度迭代","source_units":[{"start":88305,"end":88952,"start_line":1915,"end_line":1930,"raw_tex":"\\begin{algorithm}[ht!]\n\\caption{Conjugate Gradient}\n\\label{alg:cg}\n\\begin{algorithmic}\n  \\STATE \\emph{Inputs:} Symmetric matrix $\\bH$, vector $g$;\n  \\STATE $r_0 \\leftarrow g$, $p_0 \\leftarrow -r_0$, $y_0 \\leftarrow 0$, $j \\leftarrow 0$;\n  \\WHILE{$p_j^\\top\\bH p_j>0$ and $r_j \\ne 0$}\n  \\STATE $\\alpha_j \\leftarrow {r_j^\\top r_j}/{p_j^\\top \\bH p_j}$;\n\\STATE $y_{j+1} \\leftarrow y_j+\\alpha_j p_j$;\n\\STATE $r_{j+1} \\leftarrow r_j + \\alpha_j \\bH p_j$;\n\\STATE $\\beta_{j+1} \\leftarrow {(r_{j+1}^\\top r_{j+1})}/{(r_j^\\top r_j)}$;\n\\STATE $p_{j+1} \\leftarrow -r_{j+1} + \\beta_{j+1}p_j$;\n\\STATE $j \\leftarrow j+1$;\n\\ENDWHILE\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"e161a46a7e22a03fbfccc89380d66a713df82133d63f45b8f249bbbc6ad226fb"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"共轭梯度二次模型","source_units":[{"start":88954,"end":89121,"start_line":1932,"end_line":1935,"raw_tex":"Here and below, we refer often to the following quadratic function:\n\\begin{equation} \\label{eq:quadraticCG}\n\tq(y) := \\frac{1}{2}y^\\top \\bH y + g^\\top y,\n\\end{equation}","sha256_utf8":"1381ae9faddbbab8d9a0989ffe6aa1848b0dbe7e167962e73daf87ca234f0754"},{"start":89122,"end":89430,"start_line":1936,"end_line":1940,"raw_tex":"where $\\bH$ and $g$ are the matrix and vector parameters of\nAlgorithms~\\ref{alg:ccg} or~\\ref{alg:cg}. When $\\bH$ is positive\ndefinite, the minimizer of $q$ is identical to the unique solution of\n$\\bH y = -g$.  CG can be viewed either as an algorithm to solve $\\bH y\n= -g$ or as an algorithm to minimize $q$.\n","sha256_utf8":"c9cbc469c2f3fe0ac45c9fea767a0acd1433c040ff3eb95e2cbc733ef35091f5"}]},{"id":"S014","semantic_type":"corollary","kind":"corollary","title":"近似二阶点的运算复杂度","source_units":[{"start":81996,"end":83531,"start_line":1785,"end_line":1817,"raw_tex":"\\begin{corollary} \\label{coro:wcc2Hv}\n  Suppose that assumptions of Theorem~\\ref{theo:wccits} hold, and let\n  $\\bar{K}_2$ be defined as in \\eqref{eq:wccits}.  \\refer{Suppose that\n    the values of $M$ used or calculated at each instance of\n    Algorithm~\\ref{alg:ccg} satisfy $M \\le U_H$.} Then with\n  probability at least $(1-\\delta)^{\\bar{K}_2}$,\n  Algorithm~\\ref{alg:dncg} terminates\n%  at least $1-\\bar{K}_2 \\delta$, Algorithm~\\ref{alg:dncg} terminates\n  at a point satisfying \\eqref{eq:2epson} after at most\n  \\[\n  \\left(\\max \\{ 2\\min\\{n,J(U_H,\\epsH,\\zeta)\\}+2,N_{\\mathrm{meo}} \\} \\right) \n  \\bar{K}_2\n%   = \\tcO\\left( \\max\\left\\{\\epsg^{-3} \\epsH^{5/2},\\epsH^{-7/2} \n%  \\right\\}\\right)\n  \\]\n  Hessian-vector products and/or gradient evaluations. (With\n  probability at most $1-(1-\\delta)^{\\bar{K}_2}$, it terminates incorrectly\n%%  within this number of Hessian-vector products and/or gradient\n%% evaluations\n  with this complexity at a point for which $\\|\\nabla f(x_k) \\| \\le\n  \\epsg$ but $\\lambda_{\\min}(\\nabla^2 f(x)) < -\\epsH$.)\n  \n  \\refer{For $n$ sufficiently large, and assuming that \n  $\\delta < 1/\\bar{K}_2$, \n%  $\\Cmeo$ depending at\n%    most logarithmically on $\\epsH$ and $\\epsg$, \n    the bound is\\\\\n    $\\tcO\\left( \\max\\left\\{\\epsg^{-3} \\epsH^{5/2},\\epsH^{-7/2}\n    \\right\\}\\right)$, with probability at most $1-\\bar{K}_2 \\delta$.} \n%% while if $J(U_H,\\epsH,\\zeta) \\ge n$ and $N_{\\mathrm{meo}}=n$, the\n%%   bound is in $\\tcO\\left( n\\,\\max\\left\\{\\epsg^{-3}\n%%   \\epsH^{3},\\epsH^{-3}\\right\\} \\right)$.}\n\\end{corollary}","sha256_utf8":"abf9e360059c8b6d3efcad4dcef138c40a1b0a0de9afafb8e75e839f30682889"}]},{"id":"S015","semantic_type":"proof","kind":"proof","title":"运算复杂度完整证明","source_units":[{"start":83532,"end":83762,"start_line":1818,"end_line":1823,"raw_tex":"\\begin{proof}\nThe proof follows by combining Theorem~\\ref{theo:wccits} (which bounds\nthe number of iterations) with Lemma~\\ref{lemma:ccgits} and\nAssumption~\\ref{assum:wccmeo} (which bound the workload per\niteration).\n  \\end{proof}","sha256_utf8":"56cc21c1a56a5dce7adffb4b0c57ef6308478b4c1f7f4c58e2afcd61a6fb3812"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"合并迭代数与单步工作量","source_units":[{"start":83532,"end":83762,"start_line":1818,"end_line":1823,"raw_tex":"\\begin{proof}\nThe proof follows by combining Theorem~\\ref{theo:wccits} (which bounds\nthe number of iterations) with Lemma~\\ref{lemma:ccgits} and\nAssumption~\\ref{assum:wccmeo} (which bound the workload per\niteration).\n  \\end{proof}","sha256_utf8":"56cc21c1a56a5dce7adffb4b0c57ef6308478b4c1f7f4c58e2afcd61a6fb3812"}]},{"id":"S017","semantic_type":"theorem","kind":"theorem","title":"高概率二阶迭代复杂度","source_units":[{"start":74820,"end":75669,"start_line":1627,"end_line":1644,"raw_tex":"\\begin{theorem} \\label{theo:wccits}\nSuppose that Assumptions~\\ref{assum:compactlevelset},\n\\ref{assum:fC22}, and ~\\ref{assum:wccmeo} hold, and define\n\\begin{equation} \\label{eq:wccits}\n\\bar{K}_2 :=\\left\\lceil \\frac{3(f(x_0)-\\flow)}{\\min\\{\\csol,\\cnc/8\\}}\\max\\{\\epsg^{-3}\n\\epsH^3, \\epsH^{-3}\\} \\right\\rceil + 2,\n\\end{equation}\n where constants $\\csol$ and $\\cnc$\nare defined in Lemmas~\\ref{lemma:LSccgSOL} and~\\ref{lemma:LSccgNC},\nrespectively.  Then with probability at least\n\\refer{$(1-\\delta)^{\\bar{K}_2}$}, Algorithm~\\ref{alg:dncg} terminates\nat a point satisfying \\eqref{eq:2epson} in at most $\\bar{K}_2$\niterations. (With probability at most\n\\refer{$1-(1-\\delta)^{\\bar{K}_2}$}, it terminates incorrectly within\n$\\bar{K}_2$ iterations at a point for which $\\|\\nabla f(x_k) \\| \\le\n\\epsg$ but $\\lambda_{\\min}(\\nabla^2 f(x)) <\n-\\epsH$.)\n\\end{theorem}","sha256_utf8":"614f7dba742e66aa7fad947630102627c977ebe9292df991b866c26a49d70dac"}]},{"id":"S018","semantic_type":"proof","kind":"proof","title":"二阶迭代界完整证明","source_units":[{"start":75672,"end":80610,"start_line":1646,"end_line":1757,"raw_tex":"\\begin{proof}\nAlgorithm~\\ref{alg:dncg} terminates incorrectly with probability\n$\\delta$ at any iteration at which Procedure~\\ref{alg:meo} is called,\n\\refer{when Procedure~\\ref{alg:meo} certifies erroneously that\n  $\\lambda_{\\min}(\\nabla^2 f(x)) \\ge -\\epsH$.}\n%The overall probability of this happening during the first\n%$\\bar{K}_2$ iterations is \\refer{bounded by} $\\bar{K}_2\n%\\delta$. Therefore, with probability at least $1-\\bar{K}_2 \\delta$,\n%no incorrect termination occurs in the first $\\bar{K}_2$ iterations.\n\\refer{Since an erroneous certificate can only lead to termination, an\n  erroneous certificate at iteration $k$ means that\n  Procedure~\\ref{alg:meo} did not produce an erroneous certificate at\n  iterations $0$ to $k-1$. By a disjunction argument, we have that the\n  overall probability of terminating with an erroneous certificate\n  during the first $\\bar{K}_2$ iterations is bounded by\n  $1-(1-\\delta)^{\\bar{K}_2}$. Therefore, with probability at least\n  $(1-\\delta)^{\\bar{K}_2} $, no incorrect termination occurs in the\n  first $\\bar{K}_2$ iterations.} \n  \nSuppose now for contradiction that Algorithm~\\ref{alg:dncg} runs for\n$\\bar{K}_2$ iterations without terminating. That is, for all \n$l=0,1,\\dotsc,\\bar{K}_2$, we have either $\\| \\nabla f(x_l) \\| > \\epsg$ \nor $\\lambdamin(\\nabla^2 f(x_l)) < -\\epsH$. We perform the following \npartition of the set of iteration indices:\n\\begin{equation} \\label{eq:K123}\n  \\cK_1 \\cup \\cK_2 \\cup \\cK_3 = \\{0,1,\\dotsc,\\bar{K}_2-1\\},\n  \\end{equation}\nwhere $\\cK_1$, $\\cK_2$, and $\\cK_3$ are defined as follows.\n  \t\n\\medskip\n\t\n\\textbf{Case 1:} $\\cK_1 := \\{ l =0,1,\\dotsc,\\bar{K}_2-1 \\, : \\,\n\\|\\nabla f(x_l) \\| \\le \\epsg \\}$.  At each iteration $l \\in \\cK_1$,\nAlgorithm~\\ref{alg:dncg} calls Procedure~\\ref{alg:meo}, which does not\ncertify that $\\lambdamin (\\nabla^2 f(x_l)) \\ge -\\epsH$ (since the\nalgorithm continues to iterate) but rather returns a direction of\nsufficient negative curvature. By Lemma~\\ref{lemma:LSgeneralNC}, the\nstep along this direction leads to an improvement in $f$ that is\nbounded as follows:\n\\begin{equation} \\label{eq:wccitsK1decrease}\nf(x_l)-f(x_{l+1})  \\ge  \\frac{\\cnc}{8} \\epsH^3.\n\\end{equation}\t\t\n\t\n\\medskip\t\n\t\n\\textbf{Case 2:} $\\cK_2 := \\{ l=0,1,\\dotsc,\\bar{K}_2-1 \\, : \\, \\|\n\\nabla f(x_l) \\| > \\epsg \\; \\mbox{and} \\; \\| \\nabla f(x_{l+1}) \\| >\n\\epsg \\}$.  Algorithm~\\ref{alg:dncg} calls Algorithm~\\ref{alg:ccg} at\neach iteration $l \\in \\cK_2$, returning either an approximate damped\nNewton or a negative curvature direction. By combining\nLemmas~\\ref{lemma:LSccgSOL} and \\ref{lemma:LSccgNC}, we obtain a \ndecrease in $f$ satisfying\n\\begin{align}\n  \\nonumber\nf(x_l) - f(x_{l+1})  & \\ge  \\min\\{\\csol,\\cnc\\}\n\\min\\left\\{\\| \\nabla f(x_{l+1}) \\|^3\\epsH^{-3},\\epsH^3\\right\\} \\\\\n  \\label{eq:wccitsK2decrease}\n& \\ge \\min\\{\\csol,\\cnc/8\\}\n\\min\\left\\{\\epsg^3\\epsH^{-3},\\epsH^3\\right\\}.\n\\end{align}\n\n\\medskip\n\t\n\\textbf{Case 3:} $\\cK_3 := \\{ l=0,1,\\dotsc,\\bar{K}_2-1 \\, : \\, \\|\n\\nabla f(x_l) \\| > \\epsg \\ge \\| \\nabla f(x_{l+1}) \\| \\}$.  Because $\\|\n\\nabla f(x_{l+1})\\|$ may be small in this case, we can no longer bound\nthe decrease in $f$ by an expression such as\n\\eqref{eq:wccitsK2decrease}. We can however guarantee at least that\n$f(x_l)-f(x_{l+1}) \\ge 0$. Moreover, provided that $l<\\bar{K}_2-1$, we\nhave from $\\| \\nabla f(x_{l+1}) \\| \\le \\epsg$ that the next iterate\n$l+1$ is in $\\cK_1$. Thus, a significant decrease in $f$ will be\nattained at the {\\em next} iteration, and we have\n\\begin{equation} \\label{eq:K13}\n  | \\cK_3| \\le |\\cK_1|+1.\n\\end{equation}\n\n\\medskip\n\nWe now consider the total decrease in $f$ over the span of $\\bar{K}_2$\niterations, which is bounded by $f(x_0)-\\flow$ as follows:\n\\begin{align} \n  f(x_0) - \\flow &\\ge \\sum_{l=0}^{\\bar{K}_2-1} (f(x_l)-f(x_{l+1})) \\nonumber \\\\\n  \\label{eq:sumdecompwccits}\n& \\ge \\sum_{l \\in \\cK_1} (f(x_l)-f(x_{l+1})) +  \\sum_{l \\in \\cK_2} (f(x_l)-f(x_{l+1})) \n\\end{align}\nwhere both sums in the final expression are nonnegative. Using first\nthe bound \\eqref{eq:wccitsK1decrease} for the sum over $\\cK_1$, we\nobtain\n\\begin{equation} \\label{eq:wccitsboundK1}\nf(x_0) - \\flow \\; \\ge \\; |\\cK_1| \\frac{\\cnc}{8} \\epsH^3 \\; \\Leftrightarrow \\; \n|\\cK_1| \\le \\frac{f(x_0) - \\flow}{\\cnc/8} \\epsH^{-3}.\n\\end{equation}\n%From the sum over $\\cK_2$, we obtain\n%and~\\eqref{eq:wccitsK2decrease} leads to\nApplying~\\eqref{eq:wccitsK2decrease} to the sum over $\\cK_2$ leads to\n\\begin{equation} \\label{eq:wccitsboundK2}\n|\\cK_2| \\le \\frac{f(x_0) -\n  \\flow}{\\min\\{\\csol,\\cnc/8\\}}\\max\\{\\epsg^{-3} \\epsH^3,\\epsH^{-3}\\}.\n\\end{equation}\nUsing these bounds together with \\eqref{eq:K13}, we have\n\\begin{align*}\n  \\bar{K}_2  & = |\\cK_1| + |\\cK_2| + |\\cK_3| \\\\\n  & \\le  2|\\cK_1|\n  + |\\cK_2| + 1 \\\\\n  & \\le 3\\max\\{|\\cK_1|,|\\cK_2|\\} +1  \\\\\n  & \\le\n\\frac{3(f(x_0) - \\flow)}\n     {\\min\\{\\csol,\\cnc/8\\}}\\max\\{\\epsg^{-3}\\epsH^3,\\epsH^{-3}\\} +1 \\\\\n     & \\le \\bar{K}_2-1,\n                \\end{align*}\n                giving the required contradiction.\n\\end{proof}","sha256_utf8":"9b94fb9f6952d5e120f741727bb90c439697f24ac880ded82c59eb1cebbde5ca"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"累计错误终止概率","source_units":[{"start":75672,"end":76743,"start_line":1646,"end_line":1663,"raw_tex":"\\begin{proof}\nAlgorithm~\\ref{alg:dncg} terminates incorrectly with probability\n$\\delta$ at any iteration at which Procedure~\\ref{alg:meo} is called,\n\\refer{when Procedure~\\ref{alg:meo} certifies erroneously that\n  $\\lambda_{\\min}(\\nabla^2 f(x)) \\ge -\\epsH$.}\n%The overall probability of this happening during the first\n%$\\bar{K}_2$ iterations is \\refer{bounded by} $\\bar{K}_2\n%\\delta$. Therefore, with probability at least $1-\\bar{K}_2 \\delta$,\n%no incorrect termination occurs in the first $\\bar{K}_2$ iterations.\n\\refer{Since an erroneous certificate can only lead to termination, an\n  erroneous certificate at iteration $k$ means that\n  Procedure~\\ref{alg:meo} did not produce an erroneous certificate at\n  iterations $0$ to $k-1$. By a disjunction argument, we have that the\n  overall probability of terminating with an erroneous certificate\n  during the first $\\bar{K}_2$ iterations is bounded by\n  $1-(1-\\delta)^{\\bar{K}_2}$. Therefore, with probability at least\n  $(1-\\delta)^{\\bar{K}_2} $, no incorrect termination occurs in the\n  first $\\bar{K}_2$ iterations.} \n","sha256_utf8":"d282b810935f638b39855b23e4754e430cc5db3764412c18ef46cb72f7bea0c8"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"反设未终止并划分迭代","source_units":[{"start":76746,"end":77246,"start_line":1665,"end_line":1676,"raw_tex":"Suppose now for contradiction that Algorithm~\\ref{alg:dncg} runs for\n$\\bar{K}_2$ iterations without terminating. That is, for all \n$l=0,1,\\dotsc,\\bar{K}_2$, we have either $\\| \\nabla f(x_l) \\| > \\epsg$ \nor $\\lambdamin(\\nabla^2 f(x_l)) < -\\epsH$. We perform the following \npartition of the set of iteration indices:\n\\begin{equation} \\label{eq:K123}\n  \\cK_1 \\cup \\cK_2 \\cup \\cK_3 = \\{0,1,\\dotsc,\\bar{K}_2-1\\},\n  \\end{equation}\nwhere $\\cK_1$, $\\cK_2$, and $\\cK_3$ are defined as follows.\n  \t\n\\medskip\n\t\n","sha256_utf8":"6699ab1d006513b41dcb79306c7dce70ab58c5afecc5f1bfd2dcf9027aab2a12"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"小梯度迭代的下降量","source_units":[{"start":77246,"end":77859,"start_line":1677,"end_line":1690,"raw_tex":"\\textbf{Case 1:} $\\cK_1 := \\{ l =0,1,\\dotsc,\\bar{K}_2-1 \\, : \\,\n\\|\\nabla f(x_l) \\| \\le \\epsg \\}$.  At each iteration $l \\in \\cK_1$,\nAlgorithm~\\ref{alg:dncg} calls Procedure~\\ref{alg:meo}, which does not\ncertify that $\\lambdamin (\\nabla^2 f(x_l)) \\ge -\\epsH$ (since the\nalgorithm continues to iterate) but rather returns a direction of\nsufficient negative curvature. By Lemma~\\ref{lemma:LSgeneralNC}, the\nstep along this direction leads to an improvement in $f$ that is\nbounded as follows:\n\\begin{equation} \\label{eq:wccitsK1decrease}\nf(x_l)-f(x_{l+1})  \\ge  \\frac{\\cnc}{8} \\epsH^3.\n\\end{equation}\t\t\n\t\n\\medskip\t\n\t\n","sha256_utf8":"b2ad8ffc12b295a6fd7889e18c6ed7adf5f7dc16f26a3930d88a3dc6fb5a5bcb"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"连续大梯度迭代的下降量","source_units":[{"start":77859,"end":78546,"start_line":1691,"end_line":1708,"raw_tex":"\\textbf{Case 2:} $\\cK_2 := \\{ l=0,1,\\dotsc,\\bar{K}_2-1 \\, : \\, \\|\n\\nabla f(x_l) \\| > \\epsg \\; \\mbox{and} \\; \\| \\nabla f(x_{l+1}) \\| >\n\\epsg \\}$.  Algorithm~\\ref{alg:dncg} calls Algorithm~\\ref{alg:ccg} at\neach iteration $l \\in \\cK_2$, returning either an approximate damped\nNewton or a negative curvature direction. By combining\nLemmas~\\ref{lemma:LSccgSOL} and \\ref{lemma:LSccgNC}, we obtain a \ndecrease in $f$ satisfying\n\\begin{align}\n  \\nonumber\nf(x_l) - f(x_{l+1})  & \\ge  \\min\\{\\csol,\\cnc\\}\n\\min\\left\\{\\| \\nabla f(x_{l+1}) \\|^3\\epsH^{-3},\\epsH^3\\right\\} \\\\\n  \\label{eq:wccitsK2decrease}\n& \\ge \\min\\{\\csol,\\cnc/8\\}\n\\min\\left\\{\\epsg^3\\epsH^{-3},\\epsH^3\\right\\}.\n\\end{align}\n\n\\medskip\n\t\n","sha256_utf8":"6db878db2dbc6e88e75d6804a77d662386dc3bccb6bca01668eed1e833a12b10"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"梯度跨阈值迭代的配对","source_units":[{"start":78546,"end":79205,"start_line":1709,"end_line":1723,"raw_tex":"\\textbf{Case 3:} $\\cK_3 := \\{ l=0,1,\\dotsc,\\bar{K}_2-1 \\, : \\, \\|\n\\nabla f(x_l) \\| > \\epsg \\ge \\| \\nabla f(x_{l+1}) \\| \\}$.  Because $\\|\n\\nabla f(x_{l+1})\\|$ may be small in this case, we can no longer bound\nthe decrease in $f$ by an expression such as\n\\eqref{eq:wccitsK2decrease}. We can however guarantee at least that\n$f(x_l)-f(x_{l+1}) \\ge 0$. Moreover, provided that $l<\\bar{K}_2-1$, we\nhave from $\\| \\nabla f(x_{l+1}) \\| \\le \\epsg$ that the next iterate\n$l+1$ is in $\\cK_1$. Thus, a significant decrease in $f$ will be\nattained at the {\\em next} iteration, and we have\n\\begin{equation} \\label{eq:K13}\n  | \\cK_3| \\le |\\cK_1|+1.\n\\end{equation}\n\n\\medskip\n\n","sha256_utf8":"d78644ddf220b8564182f2f03e6d6fb376cb04991fe0af93e6cb3942e4476266"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"总下降量约束第一类数量","source_units":[{"start":79205,"end":79892,"start_line":1724,"end_line":1737,"raw_tex":"We now consider the total decrease in $f$ over the span of $\\bar{K}_2$\niterations, which is bounded by $f(x_0)-\\flow$ as follows:\n\\begin{align} \n  f(x_0) - \\flow &\\ge \\sum_{l=0}^{\\bar{K}_2-1} (f(x_l)-f(x_{l+1})) \\nonumber \\\\\n  \\label{eq:sumdecompwccits}\n& \\ge \\sum_{l \\in \\cK_1} (f(x_l)-f(x_{l+1})) +  \\sum_{l \\in \\cK_2} (f(x_l)-f(x_{l+1})) \n\\end{align}\nwhere both sums in the final expression are nonnegative. Using first\nthe bound \\eqref{eq:wccitsK1decrease} for the sum over $\\cK_1$, we\nobtain\n\\begin{equation} \\label{eq:wccitsboundK1}\nf(x_0) - \\flow \\; \\ge \\; |\\cK_1| \\frac{\\cnc}{8} \\epsH^3 \\; \\Leftrightarrow \\; \n|\\cK_1| \\le \\frac{f(x_0) - \\flow}{\\cnc/8} \\epsH^{-3}.\n\\end{equation}\n","sha256_utf8":"9a0d07e10b30d2e1faadfc5c064ddaaaaf6047c446d1003c0e7a525e969a5d04"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"约束第二类迭代数量","source_units":[{"start":79972,"end":80195,"start_line":1740,"end_line":1744,"raw_tex":"Applying~\\eqref{eq:wccitsK2decrease} to the sum over $\\cK_2$ leads to\n\\begin{equation} \\label{eq:wccitsboundK2}\n|\\cK_2| \\le \\frac{f(x_0) -\n  \\flow}{\\min\\{\\csol,\\cnc/8\\}}\\max\\{\\epsg^{-3} \\epsH^3,\\epsH^{-3}\\}.\n\\end{equation}\n","sha256_utf8":"568f9a83e84f09e7420e0357379931f84840aa61e102f8adf27bb152eef328ab"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"合并计数得到矛盾","source_units":[{"start":80195,"end":80610,"start_line":1745,"end_line":1757,"raw_tex":"Using these bounds together with \\eqref{eq:K13}, we have\n\\begin{align*}\n  \\bar{K}_2  & = |\\cK_1| + |\\cK_2| + |\\cK_3| \\\\\n  & \\le  2|\\cK_1|\n  + |\\cK_2| + 1 \\\\\n  & \\le 3\\max\\{|\\cK_1|,|\\cK_2|\\} +1  \\\\\n  & \\le\n\\frac{3(f(x_0) - \\flow)}\n     {\\min\\{\\csol,\\cnc/8\\}}\\max\\{\\epsg^{-3}\\epsH^3,\\epsH^{-3}\\} +1 \\\\\n     & \\le \\bar{K}_2-1,\n                \\end{align*}\n                giving the required contradiction.\n\\end{proof}","sha256_utf8":"fe8bfca046e88f701420147f37ab37c5fac845e6b76da6e8df4505f8be927795"}]},{"id":"S027","semantic_type":"lemma","kind":"lemma","title":"近似牛顿步的线搜索界","source_units":[{"start":60668,"end":61710,"start_line":1317,"end_line":1340,"raw_tex":"\\begin{lemma} \\label{lemma:LSccgSOL}\nSuppose that Assumptions~\\ref{assum:compactlevelset}\nand~\\ref{assum:fC22} hold. Suppose that at iteration $k$ of\nAlgorithm~\\ref{alg:dncg}, we have $\\|\\nabla f(x_k)\\| > \\epsg$, so\nthat Algorithm~\\ref{alg:ccg} is called. When Algorithm~\\ref{alg:ccg}\noutputs a direction $d_k$ with d\\_type=SOL, then the backtracking line\nsearch requires at most $j_k \\le \\jsol+1$ iterations, where\n\\begin{equation} \\label{eq:ccglsitsSOL}\n\\jsol \\; = \\; \\left[ \\frac{1}{2}\\log_{\\theta}\\left(\n  \\frac{3(1-\\zeta)}{L_H+\\eta} \\frac{\\epsH^2}{\\refer{1.1} U_g} \\right)\n  \\right]_+,\n\\end{equation}\nand the resulting step $x_{k+1} = x_k + \\alpha_k d_k$ satisfies\n\\begin{equation} \\label{eq:ccglsdecreaseSOL}\nf(x_k) - f(x_{k+1}) \\; \\ge \\; \n\\csol\\min\\left(\\|\\nabla f(x_{k+1})\\|^3 \\epsH^{-3},\\epsH^3\\right),\n\\end{equation}\nwhere \n\\begin{equation*}\n\\csol = \\frac{\\eta}{6}\\min\\left\\{ \\left[ \\frac{4}{\\sqrt{(4+\\zeta)^2+8 L_H}+4+\\zeta}\n  \\right]^3, \\left[ \\frac{3\\theta^2 (1-\\zeta) }{L_H+\\eta}\n  \\right]^3\\right\\}.\n\\end{equation*}\n\\end{lemma}","sha256_utf8":"e93473f529045fb629556885b92fc6b294371ed18d61fc3d79a2957f48318f10"}]},{"id":"S028","semantic_type":"proof","kind":"proof","title":"牛顿步线搜索完整证明","source_units":[{"start":61713,"end":62175,"start_line":1342,"end_line":1351,"raw_tex":"\\begin{proof}\nThe proof tracks closely that\nof~\\cite[Lemma~13]{CWRoyer_SJWright_2018}. The only significant\ndifference is that equation (65) of~\\cite{CWRoyer_SJWright_2018},\nwhich is instrumental to the proof and requires a probabilistic\nassumption on $\\lambda_{\\min}(\\nabla^2 f(x_k))$, is now ensured\n\\emph{deterministically} by~\\eqref{eq:ccgstepSOLnorm} from\nLemma~\\ref{lemma:ccgsteps}. As a result, both the proof and the result\nare deterministic.\n\\end{proof}","sha256_utf8":"0336cb8c39be4783c7dbf84bf9925f83a3a34cf2e3d3567ed6b358ce348d56be"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"用确定性步长界替代概率条件","source_units":[{"start":61713,"end":62175,"start_line":1342,"end_line":1351,"raw_tex":"\\begin{proof}\nThe proof tracks closely that\nof~\\cite[Lemma~13]{CWRoyer_SJWright_2018}. The only significant\ndifference is that equation (65) of~\\cite{CWRoyer_SJWright_2018},\nwhich is instrumental to the proof and requires a probabilistic\nassumption on $\\lambda_{\\min}(\\nabla^2 f(x_k))$, is now ensured\n\\emph{deterministically} by~\\eqref{eq:ccgstepSOLnorm} from\nLemma~\\ref{lemma:ccgsteps}. As a result, both the proof and the result\nare deterministic.\n\\end{proof}","sha256_utf8":"0336cb8c39be4783c7dbf84bf9925f83a3a34cf2e3d3567ed6b358ce348d56be"}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","title":"截断共轭梯度负曲率下降界","source_units":[{"start":63332,"end":64293,"start_line":1375,"end_line":1397,"raw_tex":"\\begin{lemma} \\label{lemma:LSccgNC}\n  Suppose that Assumptions~\\ref{assum:compactlevelset}\n  and~\\ref{assum:fC22} hold.  Suppose that at iteration $k$ of\n  Algorithm~\\ref{alg:dncg}, we have $\\|\\nabla f(x_k)\\| > \\epsg$, so\n  that Algorithm~\\ref{alg:ccg} is called. When Algorithm~\\ref{alg:ccg}\n  outputs d\\_type=NC, the direction $d_k$ (computed from $d$ in\n  Algorithm~\\ref{alg:dncg}) has the following properties: The\n  backtracking line search terminates with step length $\\alpha_k =\n  \\theta^{j_k}$ with $j_k \\le \\jnc +1$, where\n\\begin{equation} \\label{eq:eiglsits}\t\n\\jnc := \\left[ \\log_{\\theta}\\left( \\frac{3}{L_H+\\eta} \\right)\n  \\right]_+,\n\\end{equation}\nand the resulting step $x_{k+1} = x_k + \\alpha_k d_k$ satisfies\n\\begin{equation} \\label{eq:decreaseNC}\nf(x_k) - f(x_k+\\alpha_k\\,d_k) \\; \\geq \\; \\cnc \\epsH^3,\n\\end{equation}\t\nwith\n\\begin{equation*}\n\\cnc := \\frac{\\eta}{6}\n\\min\\left\\{1,\\frac{27\\theta^3}{(L_H+\\eta)^3}\\right\\}.\n\\end{equation*}\n\\end{lemma}","sha256_utf8":"31d8f98205b61f8c88cbf3cae4958884ed106a356d907edba06a231d79a7265a"}]},{"id":"S031","semantic_type":"proof","kind":"proof","title":"负曲率线搜索完整证明","source_units":[{"start":64296,"end":64617,"start_line":1399,"end_line":1409,"raw_tex":"\\begin{proof}\n  \\refer{\nBy Lemma~\\ref{lemma:ccgsteps}, we have from  \\eqref{eq:ccgstepNC}\nthat\n\\begin{equation} \\label{eq:boundnegcurvNCsteps}\n\t\td_k^\\top \\nabla^2 f(x_k) d_k = -\\|d_k\\|^3 \\le -\\epsH \\|d_k\\|^2\n\\end{equation}\n}\nThe result can thus be obtained exactly as in\n\\cite[Lemma~1]{CWRoyer_SJWright_2018}.\n\\end{proof}","sha256_utf8":"8b3b2541510d8a558b5ab01377fc1baff6ead4b3e0eb050d21c035cd456fc189"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"取得缩放负曲率恒等式","source_units":[{"start":64296,"end":64521,"start_line":1399,"end_line":1406,"raw_tex":"\\begin{proof}\n  \\refer{\nBy Lemma~\\ref{lemma:ccgsteps}, we have from  \\eqref{eq:ccgstepNC}\nthat\n\\begin{equation} \\label{eq:boundnegcurvNCsteps}\n\t\td_k^\\top \\nabla^2 f(x_k) d_k = -\\|d_k\\|^3 \\le -\\epsH \\|d_k\\|^2\n\\end{equation}\n}\n","sha256_utf8":"ed302c6aa5f67919c22c50320abb14847b4ec6240bbeaf9ad7fb8072e6b1906b"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"应用已有负曲率线搜索估计","source_units":[{"start":64521,"end":64617,"start_line":1407,"end_line":1409,"raw_tex":"The result can thus be obtained exactly as in\n\\cite[Lemma~1]{CWRoyer_SJWright_2018}.\n\\end{proof}","sha256_utf8":"d77b20557c5a2394d3a2c0ef9aef35728e346bc900150c121fac9bbef24012f1"}]},{"id":"S034","semantic_type":"lemma","kind":"lemma","title":"两类负曲率步的统一下降界","source_units":[{"start":73247,"end":74048,"start_line":1594,"end_line":1608,"raw_tex":"\\begin{lemma} \\label{lemma:LSgeneralNC}\nSuppose that Assumptions~\\ref{assum:compactlevelset}\nand~\\ref{assum:fC22} hold.  Suppose that at iteration $k$ of\nAlgorithm~\\ref{alg:dncg}, the search direction $d_k$ is of negative\ncurvature type, obtained either directly from Procedure~\\ref{alg:meo}\nor as the output of Algorithm~\\ref{alg:ccg} and d\\_type=NC. Then the\nbacktracking line search terminates with step length $\\alpha_k =\n\\theta^{j_k}$ with $j_k \\le \\jnc +1$, where $\\jnc$ is defined as in\nLemma~\\ref{lemma:LSccgNC}, and the decrease in the function value\nresulting from the chosen step length satisfies\n\\begin{equation} \\label{eq:LSgeneralNCdecrease}\nf(x_k) - f(x_k+\\alpha_k\\,d_k) \\; \\geq \\; \\frac{\\cnc}{8} \\epsH^3,\n\\end{equation}\t\nwith $\\cnc$ is defined in Lemma~\\ref{lemma:LSccgNC}.\n\\end{lemma}","sha256_utf8":"998ad5fb011d128fdf486ec6736bf013339f5f0cda5c6313f9b9c5054f6a792f"}]},{"id":"S035","semantic_type":"proof","kind":"proof","title":"统一负曲率界完整证明","source_units":[{"start":74049,"end":74720,"start_line":1609,"end_line":1622,"raw_tex":"\\begin{proof}\n  Lemma~\\ref{lemma:LSccgNC} shows that the claim holds (with a factor\n  of $8$ to spare) when the direction of negative curvature is\n  obtained from Algorithm~\\ref{alg:ccg}.  When the direction is\n  obtained from Procedure~\\ref{alg:meo}, we have \\refer{by the scaling\n    of $d_k$ applied in Algorithm~\\ref{alg:dncg} that}\n\\begin{equation} \\label{eq:boundnegcurvNCproba}\nd_k^\\top \\nabla^2 f(x_k) d_k = -\\|d_k\\|^3 \\le -\\frac12 \\epsH \\|d_k\n\\|^2 < 0,\n\\end{equation}\nfrom which it follows that $\\|d_k \\| \\ge \\tfrac12 \\epsH$. The\nresult can now be obtained by following the proof of\nLemma~\\ref{lemma:LSccgNC}, with $\\tfrac12 \\epsH$ replacing $\\epsH$.\n\\end{proof}","sha256_utf8":"610a766d4dcf829ebeacd9a83dec130cd31cf6e8ff5738f069f38e1bae6b4977"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"复用共轭梯度负曲率界","source_units":[{"start":74049,"end":74260,"start_line":1609,"end_line":1612,"raw_tex":"\\begin{proof}\n  Lemma~\\ref{lemma:LSccgNC} shows that the claim holds (with a factor\n  of $8$ to spare) when the direction of negative curvature is\n  obtained from Algorithm~\\ref{alg:ccg}.  When the direction is\n","sha256_utf8":"3c7ad7f07b26c2f9835a56c5c30897fb318f1cc92536a454b1c0a565a47101bc"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"预言机步缩放并调整阈值","source_units":[{"start":74260,"end":74720,"start_line":1613,"end_line":1622,"raw_tex":"  obtained from Procedure~\\ref{alg:meo}, we have \\refer{by the scaling\n    of $d_k$ applied in Algorithm~\\ref{alg:dncg} that}\n\\begin{equation} \\label{eq:boundnegcurvNCproba}\nd_k^\\top \\nabla^2 f(x_k) d_k = -\\|d_k\\|^3 \\le -\\frac12 \\epsH \\|d_k\n\\|^2 < 0,\n\\end{equation}\nfrom which it follows that $\\|d_k \\| \\ge \\tfrac12 \\epsH$. The\nresult can now be obtained by following the proof of\nLemma~\\ref{lemma:LSccgNC}, with $\\tfrac12 \\epsH$ replacing $\\epsH$.\n\\end{proof}","sha256_utf8":"b829aa0fdef159e299f463248f3844810ac45fcdfb47e3146635520e1e6119e8"}]},{"id":"S038","semantic_type":"lemma","kind":"lemma","title":"截断共轭梯度输出性质","source_units":[{"start":54799,"end":56463,"start_line":1162,"end_line":1206,"raw_tex":"\\begin{lemma} \\label{lemma:ccgsteps}\nLet Assumptions~\\ref{assum:compactlevelset} and~\\ref{assum:fC22} hold,\nand suppose that Algorithm~\\ref{alg:ccg} is invoked at an iterate\n$x_k$ of Algorithm~\\ref{alg:dncg} (so that $\\|\\nabla f(x_k)\\| >\n\\epsg>0$).\nLet $d_k$ be the vector obtained in Algorithm~\\ref{alg:dncg} from the\noutput $d$ of Algorithm~\\ref{alg:ccg}. Then, one of the two following\nstatements holds:\n\\begin{enumerate}\n\\item {\\em d\\_type=SOL}, and the direction $d_k$ satisfies \n\\begin{subequations} \\label{eq:ccgstepSOL} \n\\begin{align}\n\\label{eq:ccgstepSOLcurv}\nd_k^\\top (\\nabla^2 f(x_k)+2\\epsH I) d_k & \\ge \\epsH \\|d_k\\|^2, \\\\\n\\label{eq:ccgstepSOLnorm}\n% \\|d_k\\| & \\le \\epsH^{-1} \\sqrt{1+\\zeta^2/4} \\|\\nabla f(x_k)\\|,\\\\\n\\|d_k\\| & \\le \\refer{1.1 \\epsH^{-1} \\|\\nabla f(x_k)\\|,}\\\\\n\\label{eq:ccgstepSOLres}\n%% \\|\\hat{r}_k \\| \\le  \n%% \\frac{\\hat{\\zeta}}{1-\\hat{\\zeta}}(U_H+2\\epsH)\\|d_k\\| & \\le \n%% \\frac{1}{2}\\epsH\\zeta \\| d_k \\|,\n\\|\\hat{r}_k \\|  & \\le  \\refer{\\frac{1}{2}\\epsH\\zeta \\| d_k \\|,}\n\\end{align}\n\\end{subequations}\nwhere\n\\begin{equation} \\label{eq:def.rhat}\n\\hat{r}_k := (\\nabla^2 f(x_k)+2\\epsH I) d_k+ \\nabla f(x_k);\n\\end{equation}\n\\item {\\em d\\_type=NC}, and the direction $d_k$ satisfies \n  $d_k^\\top \\nabla f(x_k) \\le 0$ as well as\n  \\refer{\n  \\begin{equation} \\label{eq:ccgstepNC}\n    \\frac{d_k^\\top \\nabla^2 f(x_k) d_k}{\\|d_k\\|^2} = -\\|d_k \\| \\le -\\epsH.\n  \\end{equation}\n  }\n%% \\begin{subequations} \\label{eq:ccgstepNC}\n%% \\begin{align}\n%% \\label{eq:ccgstepNCcurv}\n%% \\frac{d_k^\\top \\nabla^2 f(x_k) d_k}{\\|d_k\\|^2} \\le -\\epsH,\\\\\n%% \\label{eq:ccgstepNCnorm}\n%% \\|d_k\\| \\ge \\epsH.\n%% \\end{align}\n%% \\end{subequations}\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"a62079bb420330fb0c3933b698ba8126ec163e9d4eb32da56832d7b62ac1e4de"}]},{"id":"S039","semantic_type":"proof","kind":"proof","title":"输出性质完整证明","source_units":[{"start":56466,"end":59826,"start_line":1208,"end_line":1297,"raw_tex":"\\begin{proof}\n\\refer{For simplicity of notation, we use $H=\\nabla^2 f(x_k)$ and\n  $g=\\nabla f(x_k)$ in the proof.}  Suppose first that d\\_type=SOL. In\nthat case, we have from the termination conditions in\nAlgorithm~\\ref{alg:ccg} and \\eqref{eq:def.rhat} that\n\\begin{subequations} \\label{eq:propSOLstep}\n\\begin{align} \n\\label{eq:propSOLstepCURV}\nd_k^\\top (H+2 \\epsH I)d_k & \\ge \\epsH \\|d_k\\|^2, \\\\\n\\label{eq:propSOLstepRES}\n\\|\\hat{r}_k \\| &  \\le \\hat{\\zeta} \\|g\\|,\n\\end{align}\n\\end{subequations}\n\\refer{where $\\hat\\zeta$ was returned by the algorithm.}\nWe immediately recognize~\\eqref{eq:ccgstepSOLcurv}\nin~\\eqref{eq:propSOLstepCURV}.  We now\nprove~\\eqref{eq:ccgstepSOLnorm}. Observe first \nthat \\eqref{eq:propSOLstepCURV} yields\n\\[\n\\epsH\\|d_k\\|^2 \\le d_k^\\top (H+2 \\epsH I) d_k \\le \\|d_k\\| \\|(H+2\\epsH\nI) d_k\\|,\n\\]\nso from \\eqref{eq:def.rhat} we have\n\\begin{equation*}\n\\|d_k\\| \\le \\epsH^{-1} \\|(H+2\\epsH I)d_k\\| = \\epsH^{-1} \\|-g+\\hat{r}_k\\| \n= \\epsH^{-1} \\sqrt{\\|g\\|^2 + \\| \\hat{r}_k \\|^2} \\le \n\\epsH^{-1} \\sqrt{1+\\hat{\\zeta}^2}  \\|g\\|,\n\\end{equation*}\nwhere we used~\\eqref{eq:propSOLstepRES} to obtain the final bound,\ntogether with the equality $\\|-g+\\hat{r}_k\\|^2=\\|g\\|^2+\\| \\hat{r}_k\n\\|^2$, which follows from $g^\\top \\hat{r}_k = r_0^\\top \\hat{r}_k =\n0$, by orthogonality of the residuals in CG (see\nLemma~\\ref{lemma:propertiesCGoneiter}, Property 2). Since\n\\refer{$\\hat{\\zeta} \\le \\zeta/(3 \\kappa) \\le 1/6$ by construction, we\n  have $\\|d_k \\| \\le \\sqrt{37/36} \\epsH^{-1} \\|g\\| \\le 1.1 \\epsH^{-1}\n  \\|g\\|$, proving \\eqref{eq:ccgstepSOLnorm}.}\n\nThe bound \\eqref{eq:ccgstepSOLres} follows\nfrom~\\eqref{eq:propSOLstepRES} and the logic below:\n\\begin{align*}\n\t\\| \\hat{r}_k \\| &\\le \\hat{\\zeta} \\|g\\| \\le \n\t\\hat{\\zeta}\\left(\\|(H+2 \\epsH I) d_k \\| + \n\t\\| \\hat{r}_k \\|\\right)  \\le \\hat\\zeta \\left( (M+2\\epsH) \\|d_k \\| +  \\| \\hat{r}_k \\| \\right) \\\\\n        \\Rightarrow \\;\\; \\| \\hat{r}_k \\| &\\le \n\t\\frac{\\hat{\\zeta}}{1-\\hat{\\zeta}}(M+2 \\epsH) \\|d_k\\|,\n\\end{align*}\n\\refer{where $M$ is the value returned by the algorithm.}\nWe finally use \\refer{$\\hat\\zeta < 1/6$}\n%% \\[\n%% \\hat\\zeta = \\frac{\\zeta}{3}\\frac{\\epsH}{M+2\\epsH} \\le\n%% \\frac{\\zeta}{3} \\frac12 < \\frac16,\n%% \\]\nto arrive at\n\\[\n\\frac{\\hat\\zeta}{1-\\hat{\\zeta}} (M+2\\epsH) \\le \\frac65 \\hat{\\zeta}\n(M+2\\epsH) = \\frac65 \\frac{\\zeta \\epsH}{3} < \\frac12 \\zeta \\epsH,\n\\]\nyielding  \\eqref{eq:ccgstepSOLres}.\n\nIn the case of d\\_type=NC, we recall that Algorithm~\\ref{alg:dncg}\ndefines\n\\begin{equation} \\label{eq:scaleccgNC}\nd_k = -\\mathrm{sgn}(d^\\top g) \\frac{|d^\\top H\n  d|}{\\|d\\|^2} \\frac{d}{\\|d\\|}\n\\end{equation}\nwhere $d$ denotes the direction obtained by\nAlgorithm~\\ref{alg:ccg}. It follows immediately that $d_k^\\top g \\le\n0$. Since $d_k$ and $d$ are collinear, we also have that\n\\refer{\n\\[\n\\frac{d_k^\\top (H+2\\epsH I)d_k}{\\|d_k\\|^2} = \\frac{d^\\top (H+2\\epsH\n  I)d}{\\|d\\|^2} \\le \\epsH \\; \\Rightarrow \\; \\frac{d_k^\\top H\n  d_k}{\\|d_k\\|^2}\n% = \\frac{d^\\top H d}{\\|d\\|^2}\n\\le -\\epsH.\n\\]\nBy using this bound together with \\eqref{eq:scaleccgNC}, we obtain\n\\[\n\\|d_k\\| = \\frac{|d^\\top H d|}{\\|d\\|^2} = \\frac{|d_k^\\top H d_k|}{\\|d_k\\|^2} =\n-\\frac{d_k^\\top H d_k}{\\|d_k\\|^2} \\ge \\epsH,\n\\]\nproving \\eqref{eq:ccgstepNC}.\n}\n%% establishing \\eqref{eq:ccgstepNCcurv}.  By \\eqref{eq:scaleccgNC}, we\n%% then have\n%% \\[\n%% \\|d_k\\| = \\frac{|d^\\top H d|}{\\|d\\|^2} \\ge \\epsH,\n%% \\]\n%% proving \\eqref{eq:ccgstepNCnorm}.\n\\end{proof}","sha256_utf8":"fa278ed40590712146bd5146d33ba8e3225b9f9d88fa61c347fbfdf35de070e3"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"读取求解终止条件","source_units":[{"start":56466,"end":57017,"start_line":1208,"end_line":1221,"raw_tex":"\\begin{proof}\n\\refer{For simplicity of notation, we use $H=\\nabla^2 f(x_k)$ and\n  $g=\\nabla f(x_k)$ in the proof.}  Suppose first that d\\_type=SOL. In\nthat case, we have from the termination conditions in\nAlgorithm~\\ref{alg:ccg} and \\eqref{eq:def.rhat} that\n\\begin{subequations} \\label{eq:propSOLstep}\n\\begin{align} \n\\label{eq:propSOLstepCURV}\nd_k^\\top (H+2 \\epsH I)d_k & \\ge \\epsH \\|d_k\\|^2, \\\\\n\\label{eq:propSOLstepRES}\n\\|\\hat{r}_k \\| &  \\le \\hat{\\zeta} \\|g\\|,\n\\end{align}\n\\end{subequations}\n\\refer{where $\\hat\\zeta$ was returned by the algorithm.}\n","sha256_utf8":"4c53621d195d9516ed538add4041c2d96fcc3186b13847b87ffd4877573f79d5"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"残差正交给出牛顿步范数界","source_units":[{"start":57017,"end":58014,"start_line":1222,"end_line":1243,"raw_tex":"We immediately recognize~\\eqref{eq:ccgstepSOLcurv}\nin~\\eqref{eq:propSOLstepCURV}.  We now\nprove~\\eqref{eq:ccgstepSOLnorm}. Observe first \nthat \\eqref{eq:propSOLstepCURV} yields\n\\[\n\\epsH\\|d_k\\|^2 \\le d_k^\\top (H+2 \\epsH I) d_k \\le \\|d_k\\| \\|(H+2\\epsH\nI) d_k\\|,\n\\]\nso from \\eqref{eq:def.rhat} we have\n\\begin{equation*}\n\\|d_k\\| \\le \\epsH^{-1} \\|(H+2\\epsH I)d_k\\| = \\epsH^{-1} \\|-g+\\hat{r}_k\\| \n= \\epsH^{-1} \\sqrt{\\|g\\|^2 + \\| \\hat{r}_k \\|^2} \\le \n\\epsH^{-1} \\sqrt{1+\\hat{\\zeta}^2}  \\|g\\|,\n\\end{equation*}\nwhere we used~\\eqref{eq:propSOLstepRES} to obtain the final bound,\ntogether with the equality $\\|-g+\\hat{r}_k\\|^2=\\|g\\|^2+\\| \\hat{r}_k\n\\|^2$, which follows from $g^\\top \\hat{r}_k = r_0^\\top \\hat{r}_k =\n0$, by orthogonality of the residuals in CG (see\nLemma~\\ref{lemma:propertiesCGoneiter}, Property 2). Since\n\\refer{$\\hat{\\zeta} \\le \\zeta/(3 \\kappa) \\le 1/6$ by construction, we\n  have $\\|d_k \\| \\le \\sqrt{37/36} \\epsH^{-1} \\|g\\| \\le 1.1 \\epsH^{-1}\n  \\|g\\|$, proving \\eqref{eq:ccgstepSOLnorm}.}\n","sha256_utf8":"884f3b64ca2b96f92eb659ed008fa58d5d510dd7dee602995b29eb9c14242f78"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"转换为相对步长残差界","source_units":[{"start":58015,"end":58819,"start_line":1245,"end_line":1265,"raw_tex":"The bound \\eqref{eq:ccgstepSOLres} follows\nfrom~\\eqref{eq:propSOLstepRES} and the logic below:\n\\begin{align*}\n\t\\| \\hat{r}_k \\| &\\le \\hat{\\zeta} \\|g\\| \\le \n\t\\hat{\\zeta}\\left(\\|(H+2 \\epsH I) d_k \\| + \n\t\\| \\hat{r}_k \\|\\right)  \\le \\hat\\zeta \\left( (M+2\\epsH) \\|d_k \\| +  \\| \\hat{r}_k \\| \\right) \\\\\n        \\Rightarrow \\;\\; \\| \\hat{r}_k \\| &\\le \n\t\\frac{\\hat{\\zeta}}{1-\\hat{\\zeta}}(M+2 \\epsH) \\|d_k\\|,\n\\end{align*}\n\\refer{where $M$ is the value returned by the algorithm.}\nWe finally use \\refer{$\\hat\\zeta < 1/6$}\n%% \\[\n%% \\hat\\zeta = \\frac{\\zeta}{3}\\frac{\\epsH}{M+2\\epsH} \\le\n%% \\frac{\\zeta}{3} \\frac12 < \\frac16,\n%% \\]\nto arrive at\n\\[\n\\frac{\\hat\\zeta}{1-\\hat{\\zeta}} (M+2\\epsH) \\le \\frac65 \\hat{\\zeta}\n(M+2\\epsH) = \\frac65 \\frac{\\zeta \\epsH}{3} < \\frac12 \\zeta \\epsH,\n\\]\nyielding  \\eqref{eq:ccgstepSOLres}.\n","sha256_utf8":"a58d5841b57cdb6ed71f7c149947d9717c83df6e599efcb4d7a1be02cda8387c"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"缩放负曲率方向并验证性质","source_units":[{"start":58820,"end":59826,"start_line":1267,"end_line":1297,"raw_tex":"In the case of d\\_type=NC, we recall that Algorithm~\\ref{alg:dncg}\ndefines\n\\begin{equation} \\label{eq:scaleccgNC}\nd_k = -\\mathrm{sgn}(d^\\top g) \\frac{|d^\\top H\n  d|}{\\|d\\|^2} \\frac{d}{\\|d\\|}\n\\end{equation}\nwhere $d$ denotes the direction obtained by\nAlgorithm~\\ref{alg:ccg}. It follows immediately that $d_k^\\top g \\le\n0$. Since $d_k$ and $d$ are collinear, we also have that\n\\refer{\n\\[\n\\frac{d_k^\\top (H+2\\epsH I)d_k}{\\|d_k\\|^2} = \\frac{d^\\top (H+2\\epsH\n  I)d}{\\|d\\|^2} \\le \\epsH \\; \\Rightarrow \\; \\frac{d_k^\\top H\n  d_k}{\\|d_k\\|^2}\n% = \\frac{d^\\top H d}{\\|d\\|^2}\n\\le -\\epsH.\n\\]\nBy using this bound together with \\eqref{eq:scaleccgNC}, we obtain\n\\[\n\\|d_k\\| = \\frac{|d^\\top H d|}{\\|d\\|^2} = \\frac{|d_k^\\top H d_k|}{\\|d_k\\|^2} =\n-\\frac{d_k^\\top H d_k}{\\|d_k\\|^2} \\ge \\epsH,\n\\]\nproving \\eqref{eq:ccgstepNC}.\n}\n%% establishing \\eqref{eq:ccgstepNCcurv}.  By \\eqref{eq:scaleccgNC}, we\n%% then have\n%% \\[\n%% \\|d_k\\| = \\frac{|d^\\top H d|}{\\|d\\|^2} \\ge \\epsH,\n%% \\]\n%% proving \\eqref{eq:ccgstepNCnorm}.\n\\end{proof}","sha256_utf8":"7f689d18c595ab3d21e10ea6ad4bf2ce5acffb99fb4b9020b00ad2d19479059d"}]},{"id":"S044","semantic_type":"lemma","kind":"lemma","title":"共轭梯度基本代数性质","source_units":[{"start":89543,"end":90728,"start_line":1945,"end_line":1973,"raw_tex":"\\begin{lemma} \\label{lemma:propertiesCGoneiter}\nSuppose that $j$ iterations of the CG loop are performed in\nAlgorithm~\\ref{alg:ccg} or~\\ref{alg:cg}. Then, we have\n\\begin{equation} \\label{eq:suffcurvuptoj}\n\t  \\frac{p_i^\\top \\bH p_i}{\\|p_i\\|^2} > 0 \\quad \\mbox{for\n       all $i=0,1,\\dotsc,j-1$}.\n\\end{equation}\t\nMoreover, the following properties hold.\n\\begin{enumerate}\n\\item $y_i \\in \\lspan\\left\\{p_0,\\dotsc,p_{i-1}\\right\\}$,\n  $i=1,2,\\dotsc,j$.\n\\item $r_i \\in \\lspan\\left\\{p_0,\\dotsc,p_i\\right\\}$ for all $i=1,2,\\dotsc,j$, and \n\\[\nr_i^\\top v = 0, \\quad \\mbox{for all $v \\in\n  \\lspan\\left\\{p_0,\\dotsc,p_{i-1}\\right\\}$ and all $i=1,2,\\dotsc,j$}.\n\\]\n(In particular, $r_i^\\top r_l = 0$ if \\refer{$0 \\le l<i \\le j$. If} $j=n$, \nthen $r_n=0$.)\n\\item $\\|r_i\\| \\le \\|p_i\\|$, $i=0,1,\\dotsc,j$.\n\\item $r_i^\\top p_i = -\\|r_i\\|^2$, $i=0,1,\\dotsc,j$.\n\\item $p_i^\\top \\bH p_k=0$ for all $i,k =0,1,\\dotsc,j$ with $k \\neq i$.\n\\item $p_i = -\\sum_{k=0}^i ({\\|r_i\\|^2}/{\\|r_k\\|^2}) r_k$, $i=0,1,\\dotsc,j$.\n\\item\n  \\[\n  q(y_{i+1}) = q(y_i) - \\frac{\\|r_i\\|^4}{2 p_i^\\top \\bH p_i}, \\quad i=0,1,\\dotsc,j-1.\n  \\]\n\\item $r_i^\\top \\bH r_i \\ge p_i^\\top \\bH p_i$, $i=0,1,\\dotsc,j$.\n\\end{enumerate}\t\t\n\\end{lemma}","sha256_utf8":"e767a3d15550cb437339b46d7d5d168f88b3122ab4e2d44b839aa48b54c50636"}]},{"id":"S045","semantic_type":"proof","kind":"proof","title":"共轭梯度性质完整证明","source_units":[{"start":90730,"end":91934,"start_line":1974,"end_line":2001,"raw_tex":"\\begin{proof}\nSince CG has not terminated prior to iteration $j$,\n\\eqref{eq:suffcurvuptoj} clearly holds.  All properties then follow\nfrom the definition of the CG process, and most are proved in standard\ntexts (see, for example,\n\\cite[Chapter~5]{JNocedal_SJWright_2006}). Property 8 is less commonly\nused, so we provide a proof here.\n\nThe case $i=0$ is immediate since $r_0=-p_0$ and there is\nequality. When $i \\ge 1$, we have:\n\\begin{equation*}\n\tp_i = -r_i + \\frac{\\|r_i\\|^2}{\\|r_{i-1}\\|^2}p_{i-1} \\; \\Leftrightarrow \\; \n\tr_i = -p_i + \\frac{\\|r_i\\|^2}{\\|r_{i-1}\\|^2}p_{i-1}.\n\\end{equation*}\n(Note that if iteration $i$ is reached, we cannot have\n$\\|r_{i-1}\\|=0$.)  It follows that\n\\begin{eqnarray*}\n\tr_i^\\top \\bH r_i &=   \n\t&p_i^\\top \\bH p_i -2\\frac{\\|r_i\\|^2}{\\|r_{i-1}\\|^2} p_i^\\top \\bH p_{i-1} + \n\t\\frac{\\|r_i\\|^4}{\\|r_{i-1}\\|^4}p_{i-1}^\\top \\bH p_{i-1} \\\\\n\t&= &p_i^\\top \\bH p_i + \\frac{\\|r_i\\|^4}{\\|r_{i-1}\\|^4}p_{i-1}^\\top \\bH p_{i-1},\n\\end{eqnarray*}\nas $p_i^\\top \\bH p_{i-1} = 0$ by Property 5 above. Since iteration $i$\nhas been reached, $p_{i-1}$ is a direction of \\refer{positive\n  curvature},\n% at least $\\eps$,\nand we obtain $r_i^\\top \\bH r_i \\ge p_i^\\top \\bH p_i$, as required.\n\\end{proof}","sha256_utf8":"66c504a39c2979ad58002d344a991c6987f96240a40b45946c85209772c1385f"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"正曲率条件与前七项标准性质","source_units":[{"start":90730,"end":91065,"start_line":1974,"end_line":1980,"raw_tex":"\\begin{proof}\nSince CG has not terminated prior to iteration $j$,\n\\eqref{eq:suffcurvuptoj} clearly holds.  All properties then follow\nfrom the definition of the CG process, and most are proved in standard\ntexts (see, for example,\n\\cite[Chapter~5]{JNocedal_SJWright_2006}). Property 8 is less commonly\nused, so we provide a proof here.\n","sha256_utf8":"14442b80d06ed5c0cfb86421fba9b928ef161a24a297e261bba3485040bb5f10"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"证明残差曲率支配搜索曲率","source_units":[{"start":91066,"end":91934,"start_line":1982,"end_line":2001,"raw_tex":"The case $i=0$ is immediate since $r_0=-p_0$ and there is\nequality. When $i \\ge 1$, we have:\n\\begin{equation*}\n\tp_i = -r_i + \\frac{\\|r_i\\|^2}{\\|r_{i-1}\\|^2}p_{i-1} \\; \\Leftrightarrow \\; \n\tr_i = -p_i + \\frac{\\|r_i\\|^2}{\\|r_{i-1}\\|^2}p_{i-1}.\n\\end{equation*}\n(Note that if iteration $i$ is reached, we cannot have\n$\\|r_{i-1}\\|=0$.)  It follows that\n\\begin{eqnarray*}\n\tr_i^\\top \\bH r_i &=   \n\t&p_i^\\top \\bH p_i -2\\frac{\\|r_i\\|^2}{\\|r_{i-1}\\|^2} p_i^\\top \\bH p_{i-1} + \n\t\\frac{\\|r_i\\|^4}{\\|r_{i-1}\\|^4}p_{i-1}^\\top \\bH p_{i-1} \\\\\n\t&= &p_i^\\top \\bH p_i + \\frac{\\|r_i\\|^4}{\\|r_{i-1}\\|^4}p_{i-1}^\\top \\bH p_{i-1},\n\\end{eqnarray*}\nas $p_i^\\top \\bH p_{i-1} = 0$ by Property 5 above. Since iteration $i$\nhas been reached, $p_{i-1}$ is a direction of \\refer{positive\n  curvature},\n% at least $\\eps$,\nand we obtain $r_i^\\top \\bH r_i \\ge p_i^\\top \\bH p_i$, as required.\n\\end{proof}","sha256_utf8":"d9417f6ade4bd4b19235643d258133c1824725a286c534cdbae49840de419a40"}]},{"id":"S048","semantic_type":"lemma","kind":"lemma","title":"截断迭代数与矩阵乘积界","source_units":[{"start":30735,"end":31455,"start_line":661,"end_line":677,"raw_tex":"\\begin{lemma} \\label{lemma:ccgits}\nThe number of iterations of Algorithm~\\ref{alg:ccg} is bounded by\n\\[\n\\min\\{n,J(M,\\eps,\\zeta)\\},\n\\]\nwhere $J=J(M,\\eps,\\zeta)$ is the smallest integer such that\n$\\sqrt{T}\\tau^{J/2} \\le \\hat{\\zeta}$ \\refer{where $M$, $\\hat\\zeta$,\n  $T$, and $\\tau$ are the values returned by the algorithm}. \\refer{If\n  all iterates $y_i$ generated by the algorithm are stored, the number\n  of matrix-vector multiplications required is bounded by\n\\begin{equation} \\label{eq:pv8}\n\t\\min\\{n,J(M,\\eps,\\zeta)\\}+1.\n\\end{equation}\nIf the iterates $y_i$ must be regenerated in order to define the\ndirection $d$ returned after \\eqref{eq:weakcurvdir}, this bound\nbecomes $2\\min\\{n,J(M,\\eps,\\zeta)\\}+1$.}\n\\end{lemma}","sha256_utf8":"44cb9053c1ec6e2a6ca1f79b4eb507cfd4bc4eb66f249e390fd48574a46f37a5"}]},{"id":"S049","semantic_type":"proof","kind":"proof","title":"截断工作量完整证明","source_units":[{"start":31456,"end":34384,"start_line":678,"end_line":730,"raw_tex":"\\begin{proof}\n  If the full $n$ iterations are performed, without any of the\n  termination conditions being tripped, the standard properties of CG\n  (see Appendix~\\ref{app:cgresults}) ensure that the final residual\n  $r_n$ is zero, so that the condition $\\|r_n \\| \\le \\hat\\zeta\n  \\|r_0\\|$ is satisfied, and termination occurs.\n\n  \\refer{Since no more than $n$ iterations are performed, the upper\n    bound $M$ is updated at most a finite number of times, so the\n    quantity $J$ is well defined.}\n\n  Supposing that $J<n$, we note from the definition of $J$ that\n  $\\sqrt{T}\\tau^{J/2} \\|r_0 \\| \\le \\hat\\zeta \\|r_0\\|$. Thus at least\n  one of the following two conditions must be satisfied at iteration\n  $J$: $\\|r_{J} \\| \\le \\hat\\zeta \\|r_0\\|$ or $\\|r_{J} \\| >\n  \\sqrt{T}\\tau^{J/2} \\|r_0\\|$. In either case, termination will occur\n  at iteration $J$, unless it has occurred already at a previous\n  iteration.\n\n  To derive \\eqref{eq:pv8}, note that the main workload at each\n  iteration $j$ is computation of a single matrix-vector product\n%  \\refer{$H p_j$ (since matrix-vector products involving the matrices\n%    $H$ and $\\bH$ and the vectors $y_j$ and $r_j$ can be computed in\n%    terms of this vector, in an additional $O(n)$ operations).}\n%  \\refer{(The ``$+1$'' in \\eqref{eq:pv8} is from the additional\n%    matrix-vector multiplication required in the case of $\\|r_{J} \\| >\n%    \\sqrt{T}\\tau^{J/2} \\|r_0\\|$.)}\n  \\refer{$H p_j$ after the increment of $j$ (since \n    matrix-vector products involving the matrices\n    $H$ and $\\bH$ and the vectors $y_j$ and $r_j$ can be computed in\n    terms of this vector, in an additional $O(n)$ operations).}\n  \\refer{(The ``$+1$'' in \\eqref{eq:pv8} accounts for the initial\n    matrix-vector multiplication $H p_0$ performed prior to entering \n    the loop.)}\n%    \\crcomment{I think the new tests based on $M$ changed the \n%    places in which we compute the $H p_j$, so I corrected this \n%    paragraph. The result still holds, but the +1 relates to the \n%    initial step (before the loop). Note that the four termination \n%    tests do not require additional matrix-vector products since $Hp_j$ \n%    has already been computed for the $M$-related tests.\n%    MO: Changed account to accounts.}\n\n  If we do not store additional information, we need to regenerate the\n  information needed to compute the direction $d$ satisfying\n  \\eqref{eq:weakcurvdir} by re-running the iterations of CG, possibly\n  up to the \\refer{second-to-last} iteration. This fact accounts for\n  the additional cost of $\\min\\{n,J(M,\\eps,\\zeta)\\}$ in the no-storage\n  case. \\footnote{\\refer{Interestingly, as we show in\n      Appendix~\\ref{app:cgresults}, the ratios on the left-hand side\n      of \\eqref{eq:weakcurvdir} can be calculated without knowledge of\n      $y_i$ for $i=0,1,\\dotsc,j-1$, provided that we store the scalar\n      quantities $\\alpha_i$ and $\\|r_i\\|^2$ for $i=0,1,\\dotsc,j-1$.}}\n\\end{proof}","sha256_utf8":"a466aa1f9b796e1fa6934254c9f24db333882fe7edce51f9f22114ea3c7f5f69"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"维数给出有限终止界","source_units":[{"start":31456,"end":31783,"start_line":678,"end_line":683,"raw_tex":"\\begin{proof}\n  If the full $n$ iterations are performed, without any of the\n  termination conditions being tripped, the standard properties of CG\n  (see Appendix~\\ref{app:cgresults}) ensure that the final residual\n  $r_n$ is zero, so that the condition $\\|r_n \\| \\le \\hat\\zeta\n  \\|r_0\\|$ is satisfied, and termination occurs.\n","sha256_utf8":"88bea1b556217fcf98ccdb9ccacb93b69f6c64249ef109cc807ac4b57c0c051e"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"残差测试给出隐式截断界","source_units":[{"start":31784,"end":32363,"start_line":685,"end_line":695,"raw_tex":"  \\refer{Since no more than $n$ iterations are performed, the upper\n    bound $M$ is updated at most a finite number of times, so the\n    quantity $J$ is well defined.}\n\n  Supposing that $J<n$, we note from the definition of $J$ that\n  $\\sqrt{T}\\tau^{J/2} \\|r_0 \\| \\le \\hat\\zeta \\|r_0\\|$. Thus at least\n  one of the following two conditions must be satisfied at iteration\n  $J$: $\\|r_{J} \\| \\le \\hat\\zeta \\|r_0\\|$ or $\\|r_{J} \\| >\n  \\sqrt{T}\\tau^{J/2} \\|r_0\\|$. In either case, termination will occur\n  at iteration $J$, unless it has occurred already at a previous\n  iteration.\n","sha256_utf8":"38b0959fed93f698cd322ca17f242f87836b8e0b717a08706c46c6bd06d1d5b4"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"计算主循环矩阵乘积成本","source_units":[{"start":32364,"end":33696,"start_line":697,"end_line":718,"raw_tex":"  To derive \\eqref{eq:pv8}, note that the main workload at each\n  iteration $j$ is computation of a single matrix-vector product\n%  \\refer{$H p_j$ (since matrix-vector products involving the matrices\n%    $H$ and $\\bH$ and the vectors $y_j$ and $r_j$ can be computed in\n%    terms of this vector, in an additional $O(n)$ operations).}\n%  \\refer{(The ``$+1$'' in \\eqref{eq:pv8} is from the additional\n%    matrix-vector multiplication required in the case of $\\|r_{J} \\| >\n%    \\sqrt{T}\\tau^{J/2} \\|r_0\\|$.)}\n  \\refer{$H p_j$ after the increment of $j$ (since \n    matrix-vector products involving the matrices\n    $H$ and $\\bH$ and the vectors $y_j$ and $r_j$ can be computed in\n    terms of this vector, in an additional $O(n)$ operations).}\n  \\refer{(The ``$+1$'' in \\eqref{eq:pv8} accounts for the initial\n    matrix-vector multiplication $H p_0$ performed prior to entering \n    the loop.)}\n%    \\crcomment{I think the new tests based on $M$ changed the \n%    places in which we compute the $H p_j$, so I corrected this \n%    paragraph. The result still holds, but the +1 relates to the \n%    initial step (before the loop). Note that the four termination \n%    tests do not require additional matrix-vector products since $Hp_j$ \n%    has already been computed for the $M$-related tests.\n%    MO: Changed account to accounts.}\n","sha256_utf8":"9633218a9ed8d227742dae7f88dd4f4d30c449790d1eb3cbe7424767baf7ca99"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"计入重建方向与标量计算","source_units":[{"start":33697,"end":34384,"start_line":720,"end_line":730,"raw_tex":"  If we do not store additional information, we need to regenerate the\n  information needed to compute the direction $d$ satisfying\n  \\eqref{eq:weakcurvdir} by re-running the iterations of CG, possibly\n  up to the \\refer{second-to-last} iteration. This fact accounts for\n  the additional cost of $\\min\\{n,J(M,\\eps,\\zeta)\\}$ in the no-storage\n  case. \\footnote{\\refer{Interestingly, as we show in\n      Appendix~\\ref{app:cgresults}, the ratios on the left-hand side\n      of \\eqref{eq:weakcurvdir} can be calculated without knowledge of\n      $y_i$ for $i=0,1,\\dotsc,j-1$, provided that we store the scalar\n      quantities $\\alpha_i$ and $\\|r_i\\|^2$ for $i=0,1,\\dotsc,j-1$.}}\n\\end{proof}","sha256_utf8":"7e99b4213b1b41bf58b046c5410a9fc3404f5493d400081bc7930c5a30477ac1"}]},{"id":"S054","semantic_type":"proof","kind":"proof","title":"截断界渐近估计完整推导","source_units":[{"start":34386,"end":36299,"start_line":732,"end_line":772,"raw_tex":"\\refer{Note that $J(M,\\epsilon,\\zeta)$ is an increasing function of\n  $M$, since $\\hat{\\zeta}$ is a decreasing function of $M$, while $T$\n  and $\\tau$ (and thus $\\sqrt{T}\\tau^{j/2}$) are increasing in $M$. If\n  $U_H$ is known in advance, we can call Algorithm~\\ref{alg:ccg} with\n  $M = U_H$ and use $J(U_H,\\epsilon,\\zeta)$ as the bound. Alternately,\n  we can call Algorithm~\\ref{alg:ccg} with $M=0$ and let it adjust $M$\n  as needed during the computation. Since the final value of $M$ will\n  be at most $U_H$, and since $J(M,\\epsilon,\\zeta)$ is an increasing\n  function of $M$, the quantity $J(U_H,\\epsilon,\\zeta)$ provides the\n  upper bound on the number of iterations in this case too.}\n\nWe can estimate $J$ by taking logs in its definition, as follows:\n\\[\nJ \\le \\frac{2\\ln(\\hat{\\zeta}/\\sqrt{T})}{\\ln(\\tau)} =\n\\frac{\\ln(\\hat{\\zeta}^2/T)}\n     {\\ln\\left(\\frac{\\sqrt{\\kappa}}{\\sqrt{\\kappa}+1}\\right)} =\n     \\frac{\\ln(T/\\hat{\\zeta}^2)}{\\ln(1+{1}/{\\sqrt{\\kappa}})} \\le\n     \\left(\\sqrt{\\kappa}+\\frac{1}{2}\\right)\n     \\ln\\left(\\frac{T}{\\hat{\\zeta}^2}\\right),\n\\]\nwhere we used $\\ln(1+\\tfrac{1}{t}) \\ge \\tfrac{1}{t+1/2}$ to obtain the\nlatest inequality. By\nreplacing $T,\\tau,\\hat{\\zeta},\\kappa$ by their\ndefinitions in Algorithm~\\ref{alg:ccg}, and using\n$\n\\frac{1}{1-\\sqrt{\\tau}} = \\frac{1+\\sqrt{\\tau}}{1-\\tau} \\le\n\\frac{2}{1-\\tau},\n$\nwe obtain\n\\begin{align} \\nonumber\n%% J(M,\\eps,\\zeta) &  \\le\n%% \\min\\left\\{n,\\left\\lceil \\left(\\sqrt{\\frac{M+2\\eps}{\\eps}}+\\frac{1}{2}\\right)\n%% \\ln\\left(\\frac{144 \\left(\\sqrt{\\frac{M+2\\eps}{\\eps}}+1\\right)^2\n%%   \\left(\\frac{M+2\\eps}{\\eps}\\right)^6} {\\zeta^2}\\right) \\right\\rceil \\right\\} \\\\\nJ(M,\\eps,\\zeta) &  \\le\n\\refer{\\min\\left\\{n,\\left\\lceil \\left(\\sqrt{\\kappa}+\\frac{1}{2}\\right)\n\\ln\\left(\\frac{144 \\left(\\sqrt{\\kappa}+1\\right)^2\n  \\kappa^6} {\\zeta^2}\\right) \\right\\rceil \\right\\}} \\\\\n\\label{eq:HvcountcappedCG}\n & = \\min \\left\\{ n , \\tcO(\\eps^{-1/2}) \\right\\}.\n\\end{align}\n","sha256_utf8":"f3e558d76d6de5260ef3d3001780932ab4e18563731a1fc32d384ad126fb0334"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"截断界随曲率上界单调","source_units":[{"start":34386,"end":35076,"start_line":732,"end_line":741,"raw_tex":"\\refer{Note that $J(M,\\epsilon,\\zeta)$ is an increasing function of\n  $M$, since $\\hat{\\zeta}$ is a decreasing function of $M$, while $T$\n  and $\\tau$ (and thus $\\sqrt{T}\\tau^{j/2}$) are increasing in $M$. If\n  $U_H$ is known in advance, we can call Algorithm~\\ref{alg:ccg} with\n  $M = U_H$ and use $J(U_H,\\epsilon,\\zeta)$ as the bound. Alternately,\n  we can call Algorithm~\\ref{alg:ccg} with $M=0$ and let it adjust $M$\n  as needed during the computation. Since the final value of $M$ will\n  be at most $U_H$, and since $J(M,\\epsilon,\\zeta)$ is an increasing\n  function of $M$, the quantity $J(U_H,\\epsilon,\\zeta)$ provides the\n  upper bound on the number of iterations in this case too.}\n","sha256_utf8":"a5a7cc76460e7d846887524f111155dff25f1623bd52dc7506ff52c7cdd78b5d"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"取对数估计截断次数","source_units":[{"start":35077,"end":35541,"start_line":743,"end_line":753,"raw_tex":"We can estimate $J$ by taking logs in its definition, as follows:\n\\[\nJ \\le \\frac{2\\ln(\\hat{\\zeta}/\\sqrt{T})}{\\ln(\\tau)} =\n\\frac{\\ln(\\hat{\\zeta}^2/T)}\n     {\\ln\\left(\\frac{\\sqrt{\\kappa}}{\\sqrt{\\kappa}+1}\\right)} =\n     \\frac{\\ln(T/\\hat{\\zeta}^2)}{\\ln(1+{1}/{\\sqrt{\\kappa}})} \\le\n     \\left(\\sqrt{\\kappa}+\\frac{1}{2}\\right)\n     \\ln\\left(\\frac{T}{\\hat{\\zeta}^2}\\right),\n\\]\nwhere we used $\\ln(1+\\tfrac{1}{t}) \\ge \\tfrac{1}{t+1/2}$ to obtain the\nlatest inequality. By\n","sha256_utf8":"20c613aef407d6fda05b378b44bfe5f32e263e05e43937758ca196bdfbc1b283"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"代入参数得到平方根复杂度","source_units":[{"start":35541,"end":36299,"start_line":754,"end_line":772,"raw_tex":"replacing $T,\\tau,\\hat{\\zeta},\\kappa$ by their\ndefinitions in Algorithm~\\ref{alg:ccg}, and using\n$\n\\frac{1}{1-\\sqrt{\\tau}} = \\frac{1+\\sqrt{\\tau}}{1-\\tau} \\le\n\\frac{2}{1-\\tau},\n$\nwe obtain\n\\begin{align} \\nonumber\n%% J(M,\\eps,\\zeta) &  \\le\n%% \\min\\left\\{n,\\left\\lceil \\left(\\sqrt{\\frac{M+2\\eps}{\\eps}}+\\frac{1}{2}\\right)\n%% \\ln\\left(\\frac{144 \\left(\\sqrt{\\frac{M+2\\eps}{\\eps}}+1\\right)^2\n%%   \\left(\\frac{M+2\\eps}{\\eps}\\right)^6} {\\zeta^2}\\right) \\right\\rceil \\right\\} \\\\\nJ(M,\\eps,\\zeta) &  \\le\n\\refer{\\min\\left\\{n,\\left\\lceil \\left(\\sqrt{\\kappa}+\\frac{1}{2}\\right)\n\\ln\\left(\\frac{144 \\left(\\sqrt{\\kappa}+1\\right)^2\n  \\kappa^6} {\\zeta^2}\\right) \\right\\rceil \\right\\}} \\\\\n\\label{eq:HvcountcappedCG}\n & = \\min \\left\\{ n , \\tcO(\\eps^{-1/2}) \\right\\}.\n\\end{align}\n","sha256_utf8":"4feeb0bba072805cb41b06ecf1f9576d2f79d0040440d5b7b47443d638a7cdee"}]},{"id":"S058","semantic_type":"theorem","kind":"theorem","title":"残差缓降必暴露负曲率","source_units":[{"start":52654,"end":53672,"start_line":1115,"end_line":1136,"raw_tex":"\\begin{theorem} \\label{theo:cvCGwhilestronglycvx}\nSuppose that the main loop of Algorithm~\\ref{alg:ccg} terminates with\n$j=\\hIcg$, where\n\\[\n\\hIcg \\in \\{1,\\dotsc,\\min\\{n,J(M,\\eps,\\zeta)\\}\\},\n\\]\n(where $J(M,\\eps,\\zeta)$ is defined in Lemma~\\ref{lemma:ccgits} and\n\\eqref{eq:HvcountcappedCG}) \\refer{because the fourth termination test is\nsatisfied and the three earlier conditions do not hold, that is, \n$y_{\\hIcg}^\\top \\bH y_{\\hIcg} \\ge \\eps \\|y_{\\hIcg}\\|^2$, \n$p_{\\hIcg}^\\top \\bH p_{\\hIcg} \\ge \\eps \\|p_{\\hIcg}\\|^2$, and\n\\begin{equation} \\label{eq:Kitsstronglycvx}\n\\|r_{\\hIcg}\\| > \\max\\{\\hat{\\zeta},\\sqrt{T}\\tau^{\\hIcg /2}\\}\\|r_0\\|.\n\\end{equation}\nwhere $M$, $T$, $\\hat{\\zeta}$, and $\\tau$ are the values returned by\nAlgorithm~\\ref{alg:ccg}. Then\n$y_{\\hIcg+1}$ is computed by Algorithm~\\ref{alg:ccg}, and we have}\n\\begin{equation} \\label{eq:yKp1negcurv}\n\\frac{(y_{\\hIcg +1}-y_i)^\\top \\bH (y_{\\hIcg+1}-y_i)}\n{\\|y_{\\hIcg+1}-y_i\\|^2} < \\eps, \\quad \\mbox{for some $i \\in \\{0,\\dotsc,\\hIcg-1\\}$.}\n\\end{equation}\n\\end{theorem}","sha256_utf8":"1663aa7eb66d2b31657a693e2220e878ff1d9779d5de3e83195e5fb85cac01a2"}]},{"id":"S059","semantic_type":"proof","kind":"proof","title":"负曲率检测完整证明","source_units":[{"start":129473,"end":144313,"start_line":2782,"end_line":3109,"raw_tex":"\\begin{proof}\n  The proof proceeds by contradiction: If we assume that all\n  conditions specified in the statement of the theorem hold, and in\n  addition that\n\\begin{equation} \\label{eq:curvcondyKp1}\n\\frac{(y_{\\hIcg +1}-y_i)^\\top \\bH (y_{\\hIcg+1}-y_i)}\n     {\\|y_{\\hIcg+1}-y_i\\|^2} \\ge \\eps, \\quad \\mbox{for all\n       $i=0,1,\\dotsc,\\hIcg-1$,}\n\\end{equation}\nthen we must have \n\\begin{equation} \\label{eq:cvCGwhilestronglycvx}\n\\|r_{\\hIcg}\\| \\le \\sqrt{T} \\tau^{\\hIcg/2} \\|r_0\\|,\n\\end{equation}\ncontradicting \\eqref{eq:Kitsstronglycvx}. Note that\n\\begin{equation} \\label{eq:casehK}\n\t\t\\frac{(y_{\\hIcg +1}-y_{\\hIcg})^\\top \\bH (y_{\\hIcg+1}-y_{\\hIcg})}\n\t\t{\\|y_{\\hIcg+1}-y_{\\hIcg}\\|^2} = \n\t\t\\frac{\\alpha_{\\hIcg} p_{\\hIcg}^\\top \\bH (\\alpha_{\\hIcg} p_{\\hIcg})}\n\t\t{\\|\\alpha_{\\hIcg} p_{\\hIcg}\\|^2} = \\frac{p_{\\hIcg}^\\top \\bH p_{\\hIcg}}\n\t\t{\\|p_{\\hIcg}\\|^2} \\ge \\eps\n\\end{equation}\nby assumption, therefore we can consider \\eqref{eq:curvcondyKp1} to\nhold for $i=0,1,\\dotsc,\\hIcg$. Moreover, recalling the definition\n\\eqref{eq:quadraticCG} of the quadratic function $q$, we have for any\n$i=0,1,\\dotsc,\\hIcg$ that \n\\[\n\t\tq(y_{\\hIcg+1}) = q(y_i) + \\nabla q(y_i)^\\top (y_{\\hIcg+1}-y_i) + \n\t\t\\frac{1}{2}(y_{\\hIcg+1}-y_i)^\\top \\bH (y_{\\hIcg+1}-y_i).\n                \\]\nThus, \\eqref{eq:curvcondyKp1} can be reformulated as\n\\begin{equation} \\label{eq:stronglycvxyKp1yj}\n% \t\t\\forall i \\in \\{0,\\dotsc,\\hIcg\\}, \\quad \n\t\tq(y_{\\hIcg+1}) \\ge q(y_i) + r_i^\\top (y_{\\hIcg+1}-y_i) \n\t\t+ \\frac{\\eps}{2}\\|y_{\\hIcg+1}-y_i\\|^2, \\quad\n                \\mbox{for all $i=0,1,\\dotsc,\\hIcg$,}\n\\end{equation}\nwhere we used $\\nabla q(y_i)=r_i$ and the definitions\n\\eqref{eq:curvcondyKp1}, \\eqref{eq:casehK} of strong convexity along\nthe directions $y_{\\hIcg+1}-y_i$. In the remainder of the proof, and\nsimilarly to \\cite[Proof of Proposition\n  1]{YCarmon_JCDuchi_OHinder_ASidford_2017b}, we will show that\n\\eqref{eq:stronglycvxyKp1yj} leads to the contradiction\n\\eqref{eq:cvCGwhilestronglycvx}, thus proving that\n\\eqref{eq:yKp1negcurv} holds.\n\nWe define the  sequence of functions $\\Phi_j$, $j=0,1,\\dotsc\\hIcg$ as follows:\n\\[\n    \\Phi_0(z) := q(y_0) + \\frac{\\eps}{2}\\|z-y_0\\|^2,\n    \\]\n    and for $j=0,\\dotsc,\\hIcg-1$:\n\\begin{equation} \n    \\label{eq:formulaPhijplus1}\n\\Phi_{j+1}(z)  := \\tau\\Phi_j(z) + (1-\\tau)\\left( q(y_j) + r_j^\\top\n(z-y_j) +\\frac{\\eps}{2}\\|z-y_j\\|^2 \\right).\n\\end{equation}\nSince each $\\Phi_j$ is a quadratic function with Hessian $\\eps I$, it\ncan be written as follows:\n\\begin{equation} \\label{eq:Phijwithvj}\n\t\t\\Phi_j(z) \\; = \\; \\Phi_j^* + \\frac{\\eps}{2}\\|z-v_j\\|^2,\n\\end{equation}\nwhere $v_j$ is the unique minimizer of $\\Phi_j$, and $\\Phi_j^* =\n\\Phi_j(v_j)$ is the minimum value of $\\Phi_j$. (Note that $v_0=y_0=0$\nand $\\Phi^*_0 = q(y_0) = 0$.)\n\nDefining\n\\begin{equation} \\label{eq:defpsi}\n\\psi(y) := q(y_0) - q(y) + \\frac{\\eps}{2}\\|y-y_0\\|^2 = \\Phi_0(y) - q(y),\n\\end{equation}\nwe give a short inductive argument to establish that \n\\begin{equation} \\label{eq:PhiYJp1}\n\\Phi_j(y_{\\hIcg+1}) \\le q(y_{\\hIcg+1}) + \\tau^j \\psi(y_{\\hIcg+1}),\n\\quad j=0,1,\\dotsc,\\hIcg.\n\\end{equation}\nFor $j=0$, \\eqref{eq:PhiYJp1} holds because $\\Phi_0(y) = q(y) +\n\\psi(y)$ by definition. Assuming that \\eqref{eq:PhiYJp1} holds for\nsome index $j \\ge 0$, we find by first applying\n\\eqref{eq:formulaPhijplus1} (with $z=y_{\\hIcg+1}$) and then\n\\eqref{eq:stronglycvxyKp1yj} (with $i=j$) that\n\\begin{eqnarray*}\n\\Phi_{j+1}(y_{\\hIcg+1}) &= &\\tau\\Phi_j(y_{\\hIcg+1}) + (1-\\tau)\\left( q(y_j) + \nr_j^\\top (y_{\\hIcg+1}-y_j) +\\frac{\\eps}{2}\\|y_{\\hIcg+1}-y_j\\|^2 \\right) \\\\\n&\\le &\\tau\\Phi_j(y_{\\hIcg+1}) + (1-\\tau) q(y_{\\hIcg+1}).\n\\end{eqnarray*}\nThus, we have\n\\begin{align*}\n\\Phi_{j+1}(y_{\\hIcg+1}) &\\le \\tau\\Phi_j(y_{\\hIcg+1}) + (1-\\tau) q(y_{\\hIcg+1}) \\\\\n&\\le \\tau q(y_{\\hIcg+1}) + \\tau^{j+1} \\psi(y_{\\hIcg+1}) + \n(1-\\tau) q(y_{\\hIcg+1}) \\quad \\mbox{from \\eqref{eq:PhiYJp1}} \\\\\n&= q(y_{\\hIcg+1}) + \\tau^{j+1} \\psi(y_{\\hIcg+1}),\n\\end{align*}\nwhich proves \\eqref{eq:PhiYJp1} for $j+1$, and thus completes the\ninductive argument.\n\nWe next prove another technical fact about the relationship between\n$q(y_j)$ and $\\Phi_j^*$, namely,\n\\begin{equation} \\label{eq:468}\n  q(y_j) \\le \\Phi_j^*, \\quad j=0,1,\\dotsc, \\hIcg.\n\\end{equation}\nWe establish this result by an inductive argument that is quite\nlengthy and technical; we note the termination of this phase of the\nproof clearly below.\n\n\nThis result trivially holds (with equality) at $j=0$.  Supposing that\nit holds for some $j =0,1,\\dotsc, \\hIcg-1$, we will prove that it also\nholds for $j+1$.\n\nBy using Properties 7 and 8 of Lemma~\\ref{lemma:propertiesCGoneiter},\n%and also $\\|\\bH \\| \\le (M+2\\eps)$, we have\n\\refer{and also $\\|\\bH r_j\\| \\le (M+2\\eps)\\|r_j\\|$,} we have\n\\begin{equation*} \n\tq(y_{j+1}) = q(y_j) - \\frac{\\|r_j\\|^4}{2\\, p_j^\\top \\bH p_j} \\le \n\tq(y_j) - \\frac{\\|r_j\\|^4}{2\\, r_j^\\top \\bH r_j} \\le \n\tq(y_j) - \\frac{\\|r_j\\|^2}{2(M+2\\eps)}.\n\\end{equation*}\nIt follows from induction hypothesis $q(y_j) \\le \\Phi_j^*$ that \n\\begin{align} \\label{eq:qyjp1Phijstar}\nq(y_{j+1}) \\le q(y_j) - \\frac{\\|r_j\\|^2}{2(M+2\\eps)}\n&= \\tau q(y_j) + (1-\\tau) q(y_j) -\\frac{\\|r_j\\|^2}{2(M+2\\eps)} \\nonumber \\\\\n&\\le \\tau \\Phi_j^* + (1-\\tau) q(y_j) -\\frac{\\|r_j\\|^2}{2(M+2\\eps)}.\n\\end{align}\nBy taking the derivative in \\eqref{eq:formulaPhijplus1}, and using\n\\eqref{eq:Phijwithvj}, we obtain\n\\begin{align*}\n\\nabla \\Phi_{j+1}(z) &= \\tau \\nabla \\Phi_j(z) + (1-\\tau)\\left[r_j + \\eps(z-y_j)\\right] \\\\\n\\Rightarrow \\;\\; \\eps(z-v_{j+1})&=  \\eps \\tau (z-v_j) + (1-\\tau) \\left(r_j + \\eps (z-y_j) \\right).\n\\end{align*}\nBy rearranging the above relation (and noting  that the $z$ terms cancel out), we obtain:\n\\begin{equation} \\label{eq:vjp1minPhijp1}\n\tv_{j+1}  \\; = \\; \\tau v_j - \\frac{1-\\tau}{\\eps}r_j + (1-\\tau) y_j.\n\\end{equation}\nIt follows from this expression together with Properties 1 and 2 of\nLemma~\\ref{lemma:propertiesCGoneiter} that \n\\begin{equation} \\label{eq:vjinKj}\nv_j \\in \\lspan\\left\\{p_0,p_1,\\dotsc,p_{j-1}\\right\\}, \\quad j=1,2,\\dotsc,\\hIcg.\n\\end{equation}\n(The result holds for $j=1$, from \\eqref{eq:vjp1minPhijp1} we have\n$v_1 \\in \\lspan \\{v_0, r_0, y_0\\} = \\lspan \\{r_0\\} = \\lspan \\{p_0 \\}$,\nand an induction based on \\eqref{eq:vjp1minPhijp1} can be used to\nestablish \\eqref{eq:vjinKj} for the other values of $j$.)  By\ncombining the expressions~\\eqref{eq:Phijwithvj} for $\\Phi_j$,\n$\\Phi_{j+1}$ with the recurrence formula~\\eqref{eq:formulaPhijplus1}\nfor $\\Phi_{j+1}$, we obtain\n\\begin{align*} \n\t\t\\Phi_{j+1}^* + \\frac{\\eps}{2}\\|y_j-v_{j+1}\\|^2 &= \\Phi_{j+1}(y_j) \\\\\n%\t\t&= \\tau \\Phi_j(y_j) + (1-\\tau)\\left(q(y_j)+r_j^\\top (y_j-y_j) + \n%\t\t\\frac{\\eps}{2}\\|y_j-y_j\\|^2\\right)  \\\\\n\t\t&= \\tau\\Phi_j(y_j) + (1-\\tau) q(y_j) \\\\\n                &= \\tau\\left(\\Phi_j^* + \n\t\t\\frac{\\eps}{2}\\|y_j-v_j\\|^2\\right) + (1-\\tau) q(y_j)\n\\end{align*}\nand therefore\n\\begin{equation} \\label{eq:doubleexpressionPhijp1}\n  \\Phi_{j+1}^* = \\tau \\left(\\Phi_j^* +\n  \\frac{\\eps}{2}\\|y_j-v_j\\|^2\\right) + (1-\\tau) q(y_j) -\n  \\frac{\\eps}{2}\\|y_j-v_{j+1}\\|^2.\n\\end{equation}\nOn the other hand, we have by~\\eqref{eq:vjp1minPhijp1} that\n\\begin{eqnarray} \\label{eq:normylminusvlp1}\n\t\t\\|y_j-v_{j+1}\\|^2 &= &\\left\\| \\tau (y_j-v_j) \n\t\t+\\frac{1-\\tau}{\\eps}r_j \\right\\|^2 \\nonumber \\\\\n\t\t&= &(\\tau^2 \\|y_j-v_j\\|^2 + \\frac{(1-\\tau)^2}{\\eps^2}\\|r_j\\|^2 \n\t\t+ \\frac{2}{\\eps}(1-\\tau)\\tau r_j^\\top (y_j-v_j) \\nonumber \\\\\n\t\t&= & \\tau^2 \\|y_j-v_j\\|^2 + \\frac{(1-\\tau)^2}{\\eps^2}\\|r_j\\|^2,\n\\end{eqnarray}\nwhere the last relation comes from $r_j \\perp\n\\lspan\\{p_0,\\dotsc,p_{j-1}\\}$ (Property 2 of\nLemma~\\ref{lemma:propertiesCGoneiter}) and \\eqref{eq:vjinKj} in the\ncase $j \\ge 1$, and immediately in the case $j=0$, since $y_0=v_0=0$.\nBy combining \\eqref{eq:doubleexpressionPhijp1}\nand~\\eqref{eq:normylminusvlp1}, we arrive at:\n\\begin{align} \\label{eq:phijp1star}\n\t\t\\Phi_{j+1}^* &= \\tau \\left(\\Phi_j^* + \n\t\t\\frac{\\eps}{2}\\|y_j-v_j\\|^2\\right) + (1-\\tau) q(y_j) \n\t\t- \\frac{\\eps}{2}\\|y_j-v_{j+1}\\|^2 \\nonumber \\\\\n\t\t&= \\tau \\left(\\Phi_j^* + \n\t\t\\frac{\\eps}{2}\\|y_j-v_j\\|^2\\right) + (1-\\tau) q(y_j) \n\t\t-\\frac{\\eps}{2}\\tau^2 \\|y_j-v_j\\|^2 - \\frac{(1-\\tau)^2}{2\\eps} \\|r_j\\|^2 \n\t\t\\nonumber \\\\\n\t\t&= \\tau \\Phi_j^* + \\frac{\\eps}{2}\\left[ \\tau \n\t\t-\\tau^2\\right]\\|y_j-v_j\\|^2 + (1-\\tau) q(y_j) \n\t\t- \\frac{(1-\\tau)^2}{2\\eps} \\|r_j\\|^2 \\nonumber \\\\ \n\t\t&= \\tau \\Phi_j^* + \n\t\t\\frac{\\eps}{2}(1-\\tau)\\tau \\|y_j-v_j\\|^2 + (1-\\tau) q(y_j) \n\t\t- \\frac{(1-\\tau)^2}{2\\eps} \\|r_j\\|^2 \\nonumber \\\\\n\t\t&\\ge \\tau \\Phi_j^* + (1-\\tau) q(y_j) \n\t\t- \\frac{(1-\\tau)^2}{2\\eps} \\|r_j\\|^2 \\nonumber \\\\\n                &\\ge q(y_{j+1}) + \\frac{1}{2(M+2\\eps)}\\|r_j\\|^2 \n\t\t- \\frac{(1-\\tau)^2}{2\\eps} \\|r_j\\|^2.\n\\end{align}\nwhere the last inequality comes from~\\eqref{eq:qyjp1Phijstar}.  By\nusing the definitions of $\\tau$ and $\\kappa$ in\nAlgorithm~\\ref{alg:ccg}, we have\n\\[\n\t\t\\frac{(1-\\tau)^2}{2\\eps} = \\frac{1}{2\\eps(\\sqrt{\\kappa}+1)^2} \\le \n\t\t\\frac{1}{2\\eps\\kappa} = \\frac{1}{2(M+2\\eps)}.\n\\]\nIt therefore follows from \\eqref{eq:phijp1star} that $q(y_{j+1}) \\le\n\\Phi_{j+1}^*$. At this point, we have shown that when $q(y_j) \\le\n\\Phi_j^*$ for $j=0,1,\\dotsc,\\hIcg-1$, it follows that $q(y_{j+1}) \\le\n\\Phi_{j+1}^*$, establishing the inductive step. As a result, our\nproof of \\eqref{eq:468} is complete.\n\n\nBy substituting $j=\\hIcg$ into \\eqref{eq:468}, we obtain\n$q(y_{\\hIcg}) \\le \\Phi_{\\hIcg}^*$,\nwhich in combination with \\eqref{eq:PhiYJp1} with $j=\\hIcg$, and the\ndefinition \\eqref{eq:Phijwithvj}, yields\n\\begin{equation} \\label{eq:desiredecreaseformula}\nq(y_{\\hIcg}) -q(y_{\\hIcg+1}) \\le \\Phi_{\\hIcg}^* - q(y_{\\hIcg+1}) \n\\le \\Phi_{\\hIcg}(y_{\\hIcg+1}) - q(y_{\\hIcg+1})\n\\le \\tau^{\\hIcg} \\psi(y_{\\hIcg+1}).\n\\end{equation}\nBy substitution from \\eqref{eq:defpsi}, we obtain\n\\begin{equation} \\label{eq:funcdecreasecontrad}\nq(y_{\\hIcg}) - q(y_{\\hIcg+1}) \\; \\le \\; \\tau^{\\hIcg}\n\\left(q(y_0)-q(y_{\\hIcg+1}) +\n\\frac{\\eps}{2}\\|y_0-y_{\\hIcg+1}\\|^2\\right).\n\\end{equation}\n\nWe now depart from the analysis\nof~\\cite{YCarmon_JCDuchi_OHinder_ASidford_2017b}, and complete the\nproof of this result by expressing \\eqref{eq:funcdecreasecontrad} in\nterms of residual norms.  On the left-hand side, we have\n\\begin{align*}\n  q(y_{\\hIcg}) - q(y_{\\hIcg+1})\n  % &= &\\nabla q(y_{\\hIcg+1})^\\top (y_{\\hIcg}-y_{\\hIcg+1}) + \n%\t\t\\frac{1}{2}(y_{\\hIcg}-y_{\\hIcg+1})^\\top \\bH (y_{\\hIcg}-y_{\\hIcg+1})  \\\\ \n\t\t&= r_{\\hIcg+1}^\\top (y_{\\hIcg}-y_{\\hIcg+1}) + \n\t\t\\frac{1}{2}(y_{\\hIcg}-y_{\\hIcg+1})^\\top \\bH (y_{\\hIcg}-y_{\\hIcg+1})  \\\\\n\t\t&= \\frac{1}{2}(y_{\\hIcg}-y_{\\hIcg+1})^\\top \\bH (y_{\\hIcg}-y_{\\hIcg+1})\n\\end{align*}\nbecause $r_{\\hIcg+1}^\\top (y_{\\hIcg}-y_{\\hIcg+1}) = r_{\\hIcg+1}^\\top\n(\\alpha_{\\hIcg} p_{\\hIcg}) = 0$ by\nLemma~\\ref{lemma:propertiesCGoneiter}, Property 2.\nWe thus have from \\eqref{eq:casehK} that\n\\begin{alignat}{2}\n  \\nonumber\n  q(y_{\\hIcg}) - q(y_{\\hIcg+1}) &\\ge \\frac{\\eps}{2}\\|y_{\\hIcg}-y_{\\hIcg+1}\\|^2  &&  \\\\\n  \\nonumber\n  &\\refer{= \\frac{\\eps}{2} \\|\\alpha_{\\hIcg}p_{\\hIcg}\\|^2} \\\\\n  \\nonumber\n  &\\refer{\\ge \\frac{\\eps}{2 (M+2\\eps)^2}\\|\\bH(\\alpha_{\\hIcg}p_{\\hIcg})\\|^2} \n  && \\mbox{(since $\\|\\bH p_{\\hIcg}\\| \\le (M+2\\eps)\\|p_{\\hIcg}\\|$)} \\\\\n  \\nonumber\n%  &\\ge \\frac{\\eps}{2 (M+2\\eps)^2} \\|\\bH(y_{\\hIcg}-y_{\\hIcg+1})\\|^2  && \\mbox{(since $\\bH \\preceq (M+2\\eps)I$)} \\\\\n  &= \\frac{\\eps}{2 (M+2\\eps)^2} \\|\\bH(y_{\\hIcg}-y_{\\hIcg+1})\\|^2   \\\\\n  \\nonumber\n  &= \\frac{\\eps}{2 (M+2\\eps)^2} \\|r_{\\hIcg} - r_{\\hIcg+1}\\|^2  && \\mbox{(since $r_j=g+\\bH y_j$)} \\\\\n  \\nonumber\n\t\t&= \\frac{\\eps}{2 (M+2\\eps)^2} (\\|r_{\\hIcg}\\|^2 + \\|r_{\\hIcg+1}\\|^2) & \\quad & \\mbox{(by Lemma~\\ref{lemma:propertiesCGoneiter}, Property 2)} \\\\\n  \\label{eq:funcdeccontradLHS}\n\t\t& \\ge \\frac{\\eps}{2 (M+2\\eps)^2} \\|r_{\\hIcg}\\|^2,\n\\end{alignat}\n\n\nOn the right-hand side of~\\eqref{eq:funcdecreasecontrad}, because of\nthe strong convexity condition \\eqref{eq:stronglycvxyKp1yj} at $i=0$,\nwe have\n\\begin{align*}\nq(y_0)-q(y_{\\hIcg+1}) + \\frac{\\eps}{2}\\|y_0-y_{\\hIcg+1}\\|^2  & \\le \n-\\nabla q(y_0)^\\top (y_{\\hIcg+1}-y_0) \\\\\n&= -r_0^\\top (y_{\\hIcg+1}-y_0)  \\le \\|r_0\\|\\|y_{\\hIcg+1}-y_0\\|.\n\\end{align*}\nMoreover, we have \n\\[\n\t\t\\|y_{\\hIcg+1}-y_0\\| \\; = \\;\n\t\t\\left\\|\\sum_{i=0}^{\\hIcg} \\alpha_i p_i \\right\\| \n\t\t\\; \\le \\; \\sum_{i=0}^{\\hIcg} \\alpha_i \\|p_i\\| \n\t\t\\; = \\; \\sum_{i=0}^{\\hIcg} \\frac{\\|r_i\\|^2}{p_i^\\top \\bH p_i}\\|p_i\\|,\n\\]\nwhere the last relation follows from the definition of $\\alpha_i$. By\ncombining these last two bounds, and using Property 3 of\nLemma~\\ref{lemma:propertiesCGoneiter}, we obtain\n\\begin{equation} \\label{eq:wd12}\n\t\tq(y_0)-q(y_{\\hIcg+1}) + \\frac{\\eps}{2}\\|y_0-y_{\\hIcg+1}\\|^2 \\; \\le \\; \n\t\t\\|r_0\\| \\sum_{i=0}^{\\hIcg} \\|r_i\\| \\frac{\\|p_i\\|^2}{p_i^\\top \\bH p_i} \\;\n\t\t\\le \\; \\|r_0\\| \\frac{1}{\\eps} \\sum_{i=0}^{\\hIcg} \\|r_i\\|,\n\\end{equation}\nwhere the last inequality holds $p_j^\\top \\bH p_j \\ge \\eps \\|p_j\\|^2$\nfor $j=0,1,\\dotsc,\\hIcg$ by assumption.\n\nTo bound the sum in \\eqref{eq:wd12}, we recall that since\nAlgorithm~\\ref{alg:ccg} did not terminate until iteration $\\hIcg$,\n\\refer{the residual norms $\\|r_i \\|$ at all iterations\n  $i=0,1,\\dotsc,\\hIcg-1$ must have decreased at the expected\n  convergence rate. In particularly, we have $\\|r_i\\| \\le \\sqrt{T}\n  \\tau^{i/2} \\|r_0\\|$ for the {\\em possibly smaller} versions of\n  $\\sqrt{T}$ and $\\tau$ that prevailed at iteration $i$, so certainly\n  $\\|r_i\\| \\le \\sqrt{T} \\tau^{i/2} \\|r_0\\|$ for the final values of\n  these parameters.} Thus for $i=0,1,\\dotsc,\\hIcg-1$, we have\n\\[\n\\|r_i\\| \\le \\sqrt{T} \\tau^{i/2} \\|r_0\\| \\le \\tau^{(i-\\hIcg)/2}\\|r_{\\hIcg}\\|,\n\\]\nwhere we used $\\|r_{\\hIcg}\\| \\ge \\sqrt{T} \\tau^{\\hIcg/2} \\|r_0\\|$\n(from \\eqref{eq:Kitsstronglycvx}) for the last inequality. Observing\nthat this bound also holds (trivially) for $i=\\hIcg$, we obtain by\nsubstituting in \\eqref{eq:wd12} that\n\\begin{align}\n  \\nonumber\nq(y_0)-q(y_{\\hIcg+1}) + \\frac{\\eps}{2}\\|y_0-y_{\\hIcg+1}\\|^2\n&\\le \\|r_0\\| \\frac{1}{\\eps} \\sum_{i=0}^{\\hIcg} \\tau^{(i-\\hIcg)/2} \\|r_{\\hIcg}\\| \\\\\n\\nonumber\n&\\le \\|r_0\\| \\frac{\\tau^{-\\hIcg/2}}{\\eps}\\|r_{\\hIcg}\\|\\,\\sum_{i=0}^{\\hIcg} (\\sqrt{\\tau})^{i} \\\\\n\\label{eq:funcdeccontradRHS}\n& \\le \\|r_0\\| \\frac{\\tau^{-\\hIcg/2}}{\\eps}\\|r_{\\hIcg}\\| \\frac{1}{1-\\sqrt{\\tau}}.\n\\end{align}\nApplying successively \\eqref{eq:funcdeccontradLHS},\n\\eqref{eq:funcdecreasecontrad} and \\eqref{eq:funcdeccontradRHS}\nfinally yields:\n\\begin{align*}\n\t\t\\frac{\\eps}{2 (M+2\\eps)^2} \\|r_{\\hIcg}\\|^2 &\\le \n\t\t q(y_{\\hIcg}) - q(y_{\\hIcg+1}) \\\\\n\t\t &\\le \\tau^{\\hIcg} \\left( q(y_0)-q(y_{\\hIcg+1}) + \\frac{\\eps}{2}\\|y_0-y_{\\hIcg+1}\\|^2 \\right) \\\\\n\t\t &\\le \\tau^{\\hIcg} \\|r_0\\|\\|r_{\\hIcg}\\| \n\t\t \\frac{\\tau^{-\\hIcg/2}}{\\eps}\\frac{1}{1-\\sqrt{\\tau}}.\n\\end{align*}\nAfter rearranging this inequality and dividing by $\\|r_{\\hIcg}\\| > 0$,\nwe obtain\n\\begin{equation} \\label{eq:finalcontrad}\n\t\t\\|r_{\\hIcg}\\| \\le \\frac{2 (M+2\\eps)^2}{\\eps^2}\\frac{\\tau^{\\hIcg/2}}{1-\\sqrt{\\tau}} \\|r_0\\|\n\t\t= \\sqrt{T} \\tau^{\\hIcg/2}\\|r_0\\|.\n\\end{equation}\nWe have thus established \\eqref{eq:cvCGwhilestronglycvx} which, as we\nnoted earlier, contradicts \\eqref{eq:Kitsstronglycvx}. Thus\n\\eqref{eq:curvcondyKp1} cannot be true, so we have established\n\\eqref{eq:yKp1negcurv}, as required.\n\\end{proof}","sha256_utf8":"af23d3db3e17ee4f007e6c68fc6958aaa56231366f4ed76ade8cdb84f94e24fe"}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","title":"反设曲率下界并重写二次模型","source_units":[{"start":129473,"end":131472,"start_line":2782,"end_line":2825,"raw_tex":"\\begin{proof}\n  The proof proceeds by contradiction: If we assume that all\n  conditions specified in the statement of the theorem hold, and in\n  addition that\n\\begin{equation} \\label{eq:curvcondyKp1}\n\\frac{(y_{\\hIcg +1}-y_i)^\\top \\bH (y_{\\hIcg+1}-y_i)}\n     {\\|y_{\\hIcg+1}-y_i\\|^2} \\ge \\eps, \\quad \\mbox{for all\n       $i=0,1,\\dotsc,\\hIcg-1$,}\n\\end{equation}\nthen we must have \n\\begin{equation} \\label{eq:cvCGwhilestronglycvx}\n\\|r_{\\hIcg}\\| \\le \\sqrt{T} \\tau^{\\hIcg/2} \\|r_0\\|,\n\\end{equation}\ncontradicting \\eqref{eq:Kitsstronglycvx}. Note that\n\\begin{equation} \\label{eq:casehK}\n\t\t\\frac{(y_{\\hIcg +1}-y_{\\hIcg})^\\top \\bH (y_{\\hIcg+1}-y_{\\hIcg})}\n\t\t{\\|y_{\\hIcg+1}-y_{\\hIcg}\\|^2} = \n\t\t\\frac{\\alpha_{\\hIcg} p_{\\hIcg}^\\top \\bH (\\alpha_{\\hIcg} p_{\\hIcg})}\n\t\t{\\|\\alpha_{\\hIcg} p_{\\hIcg}\\|^2} = \\frac{p_{\\hIcg}^\\top \\bH p_{\\hIcg}}\n\t\t{\\|p_{\\hIcg}\\|^2} \\ge \\eps\n\\end{equation}\nby assumption, therefore we can consider \\eqref{eq:curvcondyKp1} to\nhold for $i=0,1,\\dotsc,\\hIcg$. Moreover, recalling the definition\n\\eqref{eq:quadraticCG} of the quadratic function $q$, we have for any\n$i=0,1,\\dotsc,\\hIcg$ that \n\\[\n\t\tq(y_{\\hIcg+1}) = q(y_i) + \\nabla q(y_i)^\\top (y_{\\hIcg+1}-y_i) + \n\t\t\\frac{1}{2}(y_{\\hIcg+1}-y_i)^\\top \\bH (y_{\\hIcg+1}-y_i).\n                \\]\nThus, \\eqref{eq:curvcondyKp1} can be reformulated as\n\\begin{equation} \\label{eq:stronglycvxyKp1yj}\n% \t\t\\forall i \\in \\{0,\\dotsc,\\hIcg\\}, \\quad \n\t\tq(y_{\\hIcg+1}) \\ge q(y_i) + r_i^\\top (y_{\\hIcg+1}-y_i) \n\t\t+ \\frac{\\eps}{2}\\|y_{\\hIcg+1}-y_i\\|^2, \\quad\n                \\mbox{for all $i=0,1,\\dotsc,\\hIcg$,}\n\\end{equation}\nwhere we used $\\nabla q(y_i)=r_i$ and the definitions\n\\eqref{eq:curvcondyKp1}, \\eqref{eq:casehK} of strong convexity along\nthe directions $y_{\\hIcg+1}-y_i$. In the remainder of the proof, and\nsimilarly to \\cite[Proof of Proposition\n  1]{YCarmon_JCDuchi_OHinder_ASidford_2017b}, we will show that\n\\eqref{eq:stronglycvxyKp1yj} leads to the contradiction\n\\eqref{eq:cvCGwhilestronglycvx}, thus proving that\n\\eqref{eq:yKp1negcurv} holds.\n","sha256_utf8":"b5ce2d4bab1d2e9c37cac72815ef8b0ecf5b4832f6950520e9a46576aaf22010"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"构造递推二次势函数","source_units":[{"start":131473,"end":132199,"start_line":2827,"end_line":2844,"raw_tex":"We define the  sequence of functions $\\Phi_j$, $j=0,1,\\dotsc\\hIcg$ as follows:\n\\[\n    \\Phi_0(z) := q(y_0) + \\frac{\\eps}{2}\\|z-y_0\\|^2,\n    \\]\n    and for $j=0,\\dotsc,\\hIcg-1$:\n\\begin{equation} \n    \\label{eq:formulaPhijplus1}\n\\Phi_{j+1}(z)  := \\tau\\Phi_j(z) + (1-\\tau)\\left( q(y_j) + r_j^\\top\n(z-y_j) +\\frac{\\eps}{2}\\|z-y_j\\|^2 \\right).\n\\end{equation}\nSince each $\\Phi_j$ is a quadratic function with Hessian $\\eps I$, it\ncan be written as follows:\n\\begin{equation} \\label{eq:Phijwithvj}\n\t\t\\Phi_j(z) \\; = \\; \\Phi_j^* + \\frac{\\eps}{2}\\|z-v_j\\|^2,\n\\end{equation}\nwhere $v_j$ is the unique minimizer of $\\Phi_j$, and $\\Phi_j^* =\n\\Phi_j(v_j)$ is the minimum value of $\\Phi_j$. (Note that $v_0=y_0=0$\nand $\\Phi^*_0 = q(y_0) = 0$.)\n","sha256_utf8":"56acc3c7ac363db6f1da35e64d2104ab37e664ae323513f6581eeed66fd1293b"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"归纳证明势函数上界","source_units":[{"start":132200,"end":133444,"start_line":2846,"end_line":2873,"raw_tex":"Defining\n\\begin{equation} \\label{eq:defpsi}\n\\psi(y) := q(y_0) - q(y) + \\frac{\\eps}{2}\\|y-y_0\\|^2 = \\Phi_0(y) - q(y),\n\\end{equation}\nwe give a short inductive argument to establish that \n\\begin{equation} \\label{eq:PhiYJp1}\n\\Phi_j(y_{\\hIcg+1}) \\le q(y_{\\hIcg+1}) + \\tau^j \\psi(y_{\\hIcg+1}),\n\\quad j=0,1,\\dotsc,\\hIcg.\n\\end{equation}\nFor $j=0$, \\eqref{eq:PhiYJp1} holds because $\\Phi_0(y) = q(y) +\n\\psi(y)$ by definition. Assuming that \\eqref{eq:PhiYJp1} holds for\nsome index $j \\ge 0$, we find by first applying\n\\eqref{eq:formulaPhijplus1} (with $z=y_{\\hIcg+1}$) and then\n\\eqref{eq:stronglycvxyKp1yj} (with $i=j$) that\n\\begin{eqnarray*}\n\\Phi_{j+1}(y_{\\hIcg+1}) &= &\\tau\\Phi_j(y_{\\hIcg+1}) + (1-\\tau)\\left( q(y_j) + \nr_j^\\top (y_{\\hIcg+1}-y_j) +\\frac{\\eps}{2}\\|y_{\\hIcg+1}-y_j\\|^2 \\right) \\\\\n&\\le &\\tau\\Phi_j(y_{\\hIcg+1}) + (1-\\tau) q(y_{\\hIcg+1}).\n\\end{eqnarray*}\nThus, we have\n\\begin{align*}\n\\Phi_{j+1}(y_{\\hIcg+1}) &\\le \\tau\\Phi_j(y_{\\hIcg+1}) + (1-\\tau) q(y_{\\hIcg+1}) \\\\\n&\\le \\tau q(y_{\\hIcg+1}) + \\tau^{j+1} \\psi(y_{\\hIcg+1}) + \n(1-\\tau) q(y_{\\hIcg+1}) \\quad \\mbox{from \\eqref{eq:PhiYJp1}} \\\\\n&= q(y_{\\hIcg+1}) + \\tau^{j+1} \\psi(y_{\\hIcg+1}),\n\\end{align*}\nwhich proves \\eqref{eq:PhiYJp1} for $j+1$, and thus completes the\ninductive argument.\n","sha256_utf8":"91fcf7a96381cd78faf509d3b51be0773017e26a6a1f161d5372fe863e4530c3"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"建立势函数下界归纳目标","source_units":[{"start":133445,"end":133956,"start_line":2875,"end_line":2887,"raw_tex":"We next prove another technical fact about the relationship between\n$q(y_j)$ and $\\Phi_j^*$, namely,\n\\begin{equation} \\label{eq:468}\n  q(y_j) \\le \\Phi_j^*, \\quad j=0,1,\\dotsc, \\hIcg.\n\\end{equation}\nWe establish this result by an inductive argument that is quite\nlengthy and technical; we note the termination of this phase of the\nproof clearly below.\n\n\nThis result trivially holds (with equality) at $j=0$.  Supposing that\nit holds for some $j =0,1,\\dotsc, \\hIcg-1$, we will prove that it also\nholds for $j+1$.\n","sha256_utf8":"6720e6fab50cf87163b0b023b64d2c885da1f99d65d21725a703b3e487519b72"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"共轭梯度单步模型下降界","source_units":[{"start":133957,"end":134641,"start_line":2889,"end_line":2902,"raw_tex":"By using Properties 7 and 8 of Lemma~\\ref{lemma:propertiesCGoneiter},\n%and also $\\|\\bH \\| \\le (M+2\\eps)$, we have\n\\refer{and also $\\|\\bH r_j\\| \\le (M+2\\eps)\\|r_j\\|$,} we have\n\\begin{equation*} \n\tq(y_{j+1}) = q(y_j) - \\frac{\\|r_j\\|^4}{2\\, p_j^\\top \\bH p_j} \\le \n\tq(y_j) - \\frac{\\|r_j\\|^4}{2\\, r_j^\\top \\bH r_j} \\le \n\tq(y_j) - \\frac{\\|r_j\\|^2}{2(M+2\\eps)}.\n\\end{equation*}\nIt follows from induction hypothesis $q(y_j) \\le \\Phi_j^*$ that \n\\begin{align} \\label{eq:qyjp1Phijstar}\nq(y_{j+1}) \\le q(y_j) - \\frac{\\|r_j\\|^2}{2(M+2\\eps)}\n&= \\tau q(y_j) + (1-\\tau) q(y_j) -\\frac{\\|r_j\\|^2}{2(M+2\\eps)} \\nonumber \\\\\n&\\le \\tau \\Phi_j^* + (1-\\tau) q(y_j) -\\frac{\\|r_j\\|^2}{2(M+2\\eps)}.\n\\end{align}\n","sha256_utf8":"08fc281e3edd130f534544742d1947035bff833b6dd2b3e490b4799b823d8053"}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","title":"求势函数极小点递推","source_units":[{"start":134641,"end":135173,"start_line":2903,"end_line":2912,"raw_tex":"By taking the derivative in \\eqref{eq:formulaPhijplus1}, and using\n\\eqref{eq:Phijwithvj}, we obtain\n\\begin{align*}\n\\nabla \\Phi_{j+1}(z) &= \\tau \\nabla \\Phi_j(z) + (1-\\tau)\\left[r_j + \\eps(z-y_j)\\right] \\\\\n\\Rightarrow \\;\\; \\eps(z-v_{j+1})&=  \\eps \\tau (z-v_j) + (1-\\tau) \\left(r_j + \\eps (z-y_j) \\right).\n\\end{align*}\nBy rearranging the above relation (and noting  that the $z$ terms cancel out), we obtain:\n\\begin{equation} \\label{eq:vjp1minPhijp1}\n\tv_{j+1}  \\; = \\; \\tau v_j - \\frac{1-\\tau}{\\eps}r_j + (1-\\tau) y_j.\n\\end{equation}\n","sha256_utf8":"cf1a732a03f8decf560e10b0a6764f22630070165f96c9f7fa2321758f6ce2f8"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"极小点子空间与极小值表达","source_units":[{"start":135173,"end":136398,"start_line":2913,"end_line":2938,"raw_tex":"It follows from this expression together with Properties 1 and 2 of\nLemma~\\ref{lemma:propertiesCGoneiter} that \n\\begin{equation} \\label{eq:vjinKj}\nv_j \\in \\lspan\\left\\{p_0,p_1,\\dotsc,p_{j-1}\\right\\}, \\quad j=1,2,\\dotsc,\\hIcg.\n\\end{equation}\n(The result holds for $j=1$, from \\eqref{eq:vjp1minPhijp1} we have\n$v_1 \\in \\lspan \\{v_0, r_0, y_0\\} = \\lspan \\{r_0\\} = \\lspan \\{p_0 \\}$,\nand an induction based on \\eqref{eq:vjp1minPhijp1} can be used to\nestablish \\eqref{eq:vjinKj} for the other values of $j$.)  By\ncombining the expressions~\\eqref{eq:Phijwithvj} for $\\Phi_j$,\n$\\Phi_{j+1}$ with the recurrence formula~\\eqref{eq:formulaPhijplus1}\nfor $\\Phi_{j+1}$, we obtain\n\\begin{align*} \n\t\t\\Phi_{j+1}^* + \\frac{\\eps}{2}\\|y_j-v_{j+1}\\|^2 &= \\Phi_{j+1}(y_j) \\\\\n%\t\t&= \\tau \\Phi_j(y_j) + (1-\\tau)\\left(q(y_j)+r_j^\\top (y_j-y_j) + \n%\t\t\\frac{\\eps}{2}\\|y_j-y_j\\|^2\\right)  \\\\\n\t\t&= \\tau\\Phi_j(y_j) + (1-\\tau) q(y_j) \\\\\n                &= \\tau\\left(\\Phi_j^* + \n\t\t\\frac{\\eps}{2}\\|y_j-v_j\\|^2\\right) + (1-\\tau) q(y_j)\n\\end{align*}\nand therefore\n\\begin{equation} \\label{eq:doubleexpressionPhijp1}\n  \\Phi_{j+1}^* = \\tau \\left(\\Phi_j^* +\n  \\frac{\\eps}{2}\\|y_j-v_j\\|^2\\right) + (1-\\tau) q(y_j) -\n  \\frac{\\eps}{2}\\|y_j-v_{j+1}\\|^2.\n\\end{equation}\n","sha256_utf8":"6d49a224f3ac50e3c3177eb6722471e331a0a6c3efa62e235876de34752f8b6d"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"利用正交性展开距离","source_units":[{"start":136398,"end":137039,"start_line":2939,"end_line":2950,"raw_tex":"On the other hand, we have by~\\eqref{eq:vjp1minPhijp1} that\n\\begin{eqnarray} \\label{eq:normylminusvlp1}\n\t\t\\|y_j-v_{j+1}\\|^2 &= &\\left\\| \\tau (y_j-v_j) \n\t\t+\\frac{1-\\tau}{\\eps}r_j \\right\\|^2 \\nonumber \\\\\n\t\t&= &(\\tau^2 \\|y_j-v_j\\|^2 + \\frac{(1-\\tau)^2}{\\eps^2}\\|r_j\\|^2 \n\t\t+ \\frac{2}{\\eps}(1-\\tau)\\tau r_j^\\top (y_j-v_j) \\nonumber \\\\\n\t\t&= & \\tau^2 \\|y_j-v_j\\|^2 + \\frac{(1-\\tau)^2}{\\eps^2}\\|r_j\\|^2,\n\\end{eqnarray}\nwhere the last relation comes from $r_j \\perp\n\\lspan\\{p_0,\\dotsc,p_{j-1}\\}$ (Property 2 of\nLemma~\\ref{lemma:propertiesCGoneiter}) and \\eqref{eq:vjinKj} in the\ncase $j \\ge 1$, and immediately in the case $j=0$, since $y_0=v_0=0$.\n","sha256_utf8":"bfb9bc1478a598c6ea7b4a37bed1132a97f34e2ce0b09b75520a513c02ba9436"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"合并估计完成下界归纳","source_units":[{"start":137039,"end":138562,"start_line":2951,"end_line":2983,"raw_tex":"By combining \\eqref{eq:doubleexpressionPhijp1}\nand~\\eqref{eq:normylminusvlp1}, we arrive at:\n\\begin{align} \\label{eq:phijp1star}\n\t\t\\Phi_{j+1}^* &= \\tau \\left(\\Phi_j^* + \n\t\t\\frac{\\eps}{2}\\|y_j-v_j\\|^2\\right) + (1-\\tau) q(y_j) \n\t\t- \\frac{\\eps}{2}\\|y_j-v_{j+1}\\|^2 \\nonumber \\\\\n\t\t&= \\tau \\left(\\Phi_j^* + \n\t\t\\frac{\\eps}{2}\\|y_j-v_j\\|^2\\right) + (1-\\tau) q(y_j) \n\t\t-\\frac{\\eps}{2}\\tau^2 \\|y_j-v_j\\|^2 - \\frac{(1-\\tau)^2}{2\\eps} \\|r_j\\|^2 \n\t\t\\nonumber \\\\\n\t\t&= \\tau \\Phi_j^* + \\frac{\\eps}{2}\\left[ \\tau \n\t\t-\\tau^2\\right]\\|y_j-v_j\\|^2 + (1-\\tau) q(y_j) \n\t\t- \\frac{(1-\\tau)^2}{2\\eps} \\|r_j\\|^2 \\nonumber \\\\ \n\t\t&= \\tau \\Phi_j^* + \n\t\t\\frac{\\eps}{2}(1-\\tau)\\tau \\|y_j-v_j\\|^2 + (1-\\tau) q(y_j) \n\t\t- \\frac{(1-\\tau)^2}{2\\eps} \\|r_j\\|^2 \\nonumber \\\\\n\t\t&\\ge \\tau \\Phi_j^* + (1-\\tau) q(y_j) \n\t\t- \\frac{(1-\\tau)^2}{2\\eps} \\|r_j\\|^2 \\nonumber \\\\\n                &\\ge q(y_{j+1}) + \\frac{1}{2(M+2\\eps)}\\|r_j\\|^2 \n\t\t- \\frac{(1-\\tau)^2}{2\\eps} \\|r_j\\|^2.\n\\end{align}\nwhere the last inequality comes from~\\eqref{eq:qyjp1Phijstar}.  By\nusing the definitions of $\\tau$ and $\\kappa$ in\nAlgorithm~\\ref{alg:ccg}, we have\n\\[\n\t\t\\frac{(1-\\tau)^2}{2\\eps} = \\frac{1}{2\\eps(\\sqrt{\\kappa}+1)^2} \\le \n\t\t\\frac{1}{2\\eps\\kappa} = \\frac{1}{2(M+2\\eps)}.\n\\]\nIt therefore follows from \\eqref{eq:phijp1star} that $q(y_{j+1}) \\le\n\\Phi_{j+1}^*$. At this point, we have shown that when $q(y_j) \\le\n\\Phi_j^*$ for $j=0,1,\\dotsc,\\hIcg-1$, it follows that $q(y_{j+1}) \\le\n\\Phi_{j+1}^*$, establishing the inductive step. As a result, our\nproof of \\eqref{eq:468} is complete.\n","sha256_utf8":"782a392fe399f318a87d35a8a4f266d6c5c17aa7f7209270eb2a484f4e53781e"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"势函数夹逼单步模型差","source_units":[{"start":138564,"end":139220,"start_line":2986,"end_line":3000,"raw_tex":"By substituting $j=\\hIcg$ into \\eqref{eq:468}, we obtain\n$q(y_{\\hIcg}) \\le \\Phi_{\\hIcg}^*$,\nwhich in combination with \\eqref{eq:PhiYJp1} with $j=\\hIcg$, and the\ndefinition \\eqref{eq:Phijwithvj}, yields\n\\begin{equation} \\label{eq:desiredecreaseformula}\nq(y_{\\hIcg}) -q(y_{\\hIcg+1}) \\le \\Phi_{\\hIcg}^* - q(y_{\\hIcg+1}) \n\\le \\Phi_{\\hIcg}(y_{\\hIcg+1}) - q(y_{\\hIcg+1})\n\\le \\tau^{\\hIcg} \\psi(y_{\\hIcg+1}).\n\\end{equation}\nBy substitution from \\eqref{eq:defpsi}, we obtain\n\\begin{equation} \\label{eq:funcdecreasecontrad}\nq(y_{\\hIcg}) - q(y_{\\hIcg+1}) \\; \\le \\; \\tau^{\\hIcg}\n\\left(q(y_0)-q(y_{\\hIcg+1}) +\n\\frac{\\eps}{2}\\|y_0-y_{\\hIcg+1}\\|^2\\right).\n\\end{equation}\n","sha256_utf8":"90ee26ef771752a9ffe5cf0a137fd8c85eb98a40fcddea9bcf48b06476f9ccf8"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"用末次残差下界模型差","source_units":[{"start":139221,"end":140949,"start_line":3002,"end_line":3035,"raw_tex":"We now depart from the analysis\nof~\\cite{YCarmon_JCDuchi_OHinder_ASidford_2017b}, and complete the\nproof of this result by expressing \\eqref{eq:funcdecreasecontrad} in\nterms of residual norms.  On the left-hand side, we have\n\\begin{align*}\n  q(y_{\\hIcg}) - q(y_{\\hIcg+1})\n  % &= &\\nabla q(y_{\\hIcg+1})^\\top (y_{\\hIcg}-y_{\\hIcg+1}) + \n%\t\t\\frac{1}{2}(y_{\\hIcg}-y_{\\hIcg+1})^\\top \\bH (y_{\\hIcg}-y_{\\hIcg+1})  \\\\ \n\t\t&= r_{\\hIcg+1}^\\top (y_{\\hIcg}-y_{\\hIcg+1}) + \n\t\t\\frac{1}{2}(y_{\\hIcg}-y_{\\hIcg+1})^\\top \\bH (y_{\\hIcg}-y_{\\hIcg+1})  \\\\\n\t\t&= \\frac{1}{2}(y_{\\hIcg}-y_{\\hIcg+1})^\\top \\bH (y_{\\hIcg}-y_{\\hIcg+1})\n\\end{align*}\nbecause $r_{\\hIcg+1}^\\top (y_{\\hIcg}-y_{\\hIcg+1}) = r_{\\hIcg+1}^\\top\n(\\alpha_{\\hIcg} p_{\\hIcg}) = 0$ by\nLemma~\\ref{lemma:propertiesCGoneiter}, Property 2.\nWe thus have from \\eqref{eq:casehK} that\n\\begin{alignat}{2}\n  \\nonumber\n  q(y_{\\hIcg}) - q(y_{\\hIcg+1}) &\\ge \\frac{\\eps}{2}\\|y_{\\hIcg}-y_{\\hIcg+1}\\|^2  &&  \\\\\n  \\nonumber\n  &\\refer{= \\frac{\\eps}{2} \\|\\alpha_{\\hIcg}p_{\\hIcg}\\|^2} \\\\\n  \\nonumber\n  &\\refer{\\ge \\frac{\\eps}{2 (M+2\\eps)^2}\\|\\bH(\\alpha_{\\hIcg}p_{\\hIcg})\\|^2} \n  && \\mbox{(since $\\|\\bH p_{\\hIcg}\\| \\le (M+2\\eps)\\|p_{\\hIcg}\\|$)} \\\\\n  \\nonumber\n%  &\\ge \\frac{\\eps}{2 (M+2\\eps)^2} \\|\\bH(y_{\\hIcg}-y_{\\hIcg+1})\\|^2  && \\mbox{(since $\\bH \\preceq (M+2\\eps)I$)} \\\\\n  &= \\frac{\\eps}{2 (M+2\\eps)^2} \\|\\bH(y_{\\hIcg}-y_{\\hIcg+1})\\|^2   \\\\\n  \\nonumber\n  &= \\frac{\\eps}{2 (M+2\\eps)^2} \\|r_{\\hIcg} - r_{\\hIcg+1}\\|^2  && \\mbox{(since $r_j=g+\\bH y_j$)} \\\\\n  \\nonumber\n\t\t&= \\frac{\\eps}{2 (M+2\\eps)^2} (\\|r_{\\hIcg}\\|^2 + \\|r_{\\hIcg+1}\\|^2) & \\quad & \\mbox{(by Lemma~\\ref{lemma:propertiesCGoneiter}, Property 2)} \\\\\n  \\label{eq:funcdeccontradLHS}\n\t\t& \\ge \\frac{\\eps}{2 (M+2\\eps)^2} \\|r_{\\hIcg}\\|^2,\n\\end{alignat}\n","sha256_utf8":"233ba92f09ba014c486e3141aa926385d725ee152714fba841493d1ee05ce1b8"}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","title":"用残差总和上界初始模型差","source_units":[{"start":140951,"end":142068,"start_line":3038,"end_line":3062,"raw_tex":"On the right-hand side of~\\eqref{eq:funcdecreasecontrad}, because of\nthe strong convexity condition \\eqref{eq:stronglycvxyKp1yj} at $i=0$,\nwe have\n\\begin{align*}\nq(y_0)-q(y_{\\hIcg+1}) + \\frac{\\eps}{2}\\|y_0-y_{\\hIcg+1}\\|^2  & \\le \n-\\nabla q(y_0)^\\top (y_{\\hIcg+1}-y_0) \\\\\n&= -r_0^\\top (y_{\\hIcg+1}-y_0)  \\le \\|r_0\\|\\|y_{\\hIcg+1}-y_0\\|.\n\\end{align*}\nMoreover, we have \n\\[\n\t\t\\|y_{\\hIcg+1}-y_0\\| \\; = \\;\n\t\t\\left\\|\\sum_{i=0}^{\\hIcg} \\alpha_i p_i \\right\\| \n\t\t\\; \\le \\; \\sum_{i=0}^{\\hIcg} \\alpha_i \\|p_i\\| \n\t\t\\; = \\; \\sum_{i=0}^{\\hIcg} \\frac{\\|r_i\\|^2}{p_i^\\top \\bH p_i}\\|p_i\\|,\n\\]\nwhere the last relation follows from the definition of $\\alpha_i$. By\ncombining these last two bounds, and using Property 3 of\nLemma~\\ref{lemma:propertiesCGoneiter}, we obtain\n\\begin{equation} \\label{eq:wd12}\n\t\tq(y_0)-q(y_{\\hIcg+1}) + \\frac{\\eps}{2}\\|y_0-y_{\\hIcg+1}\\|^2 \\; \\le \\; \n\t\t\\|r_0\\| \\sum_{i=0}^{\\hIcg} \\|r_i\\| \\frac{\\|p_i\\|^2}{p_i^\\top \\bH p_i} \\;\n\t\t\\le \\; \\|r_0\\| \\frac{1}{\\eps} \\sum_{i=0}^{\\hIcg} \\|r_i\\|,\n\\end{equation}\nwhere the last inequality holds $p_j^\\top \\bH p_j \\ge \\eps \\|p_j\\|^2$\nfor $j=0,1,\\dotsc,\\hIcg$ by assumption.\n","sha256_utf8":"33a1aecdccf07415e6c7cabd22b26cbe13e4d8a6022a7d3c22e6301e553a557e"}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","title":"历史残差的几何级数估计","source_units":[{"start":142069,"end":143361,"start_line":3064,"end_line":3088,"raw_tex":"To bound the sum in \\eqref{eq:wd12}, we recall that since\nAlgorithm~\\ref{alg:ccg} did not terminate until iteration $\\hIcg$,\n\\refer{the residual norms $\\|r_i \\|$ at all iterations\n  $i=0,1,\\dotsc,\\hIcg-1$ must have decreased at the expected\n  convergence rate. In particularly, we have $\\|r_i\\| \\le \\sqrt{T}\n  \\tau^{i/2} \\|r_0\\|$ for the {\\em possibly smaller} versions of\n  $\\sqrt{T}$ and $\\tau$ that prevailed at iteration $i$, so certainly\n  $\\|r_i\\| \\le \\sqrt{T} \\tau^{i/2} \\|r_0\\|$ for the final values of\n  these parameters.} Thus for $i=0,1,\\dotsc,\\hIcg-1$, we have\n\\[\n\\|r_i\\| \\le \\sqrt{T} \\tau^{i/2} \\|r_0\\| \\le \\tau^{(i-\\hIcg)/2}\\|r_{\\hIcg}\\|,\n\\]\nwhere we used $\\|r_{\\hIcg}\\| \\ge \\sqrt{T} \\tau^{\\hIcg/2} \\|r_0\\|$\n(from \\eqref{eq:Kitsstronglycvx}) for the last inequality. Observing\nthat this bound also holds (trivially) for $i=\\hIcg$, we obtain by\nsubstituting in \\eqref{eq:wd12} that\n\\begin{align}\n  \\nonumber\nq(y_0)-q(y_{\\hIcg+1}) + \\frac{\\eps}{2}\\|y_0-y_{\\hIcg+1}\\|^2\n&\\le \\|r_0\\| \\frac{1}{\\eps} \\sum_{i=0}^{\\hIcg} \\tau^{(i-\\hIcg)/2} \\|r_{\\hIcg}\\| \\\\\n\\nonumber\n&\\le \\|r_0\\| \\frac{\\tau^{-\\hIcg/2}}{\\eps}\\|r_{\\hIcg}\\|\\,\\sum_{i=0}^{\\hIcg} (\\sqrt{\\tau})^{i} \\\\\n\\label{eq:funcdeccontradRHS}\n& \\le \\|r_0\\| \\frac{\\tau^{-\\hIcg/2}}{\\eps}\\|r_{\\hIcg}\\| \\frac{1}{1-\\sqrt{\\tau}}.\n\\end{align}\n","sha256_utf8":"205699db9ab7cc8a0354304d5dc9e716db0cc4932af4ebc53c0bd5ed93aa45d6"}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","title":"推出残差速率并得到矛盾","source_units":[{"start":143361,"end":144313,"start_line":3089,"end_line":3109,"raw_tex":"Applying successively \\eqref{eq:funcdeccontradLHS},\n\\eqref{eq:funcdecreasecontrad} and \\eqref{eq:funcdeccontradRHS}\nfinally yields:\n\\begin{align*}\n\t\t\\frac{\\eps}{2 (M+2\\eps)^2} \\|r_{\\hIcg}\\|^2 &\\le \n\t\t q(y_{\\hIcg}) - q(y_{\\hIcg+1}) \\\\\n\t\t &\\le \\tau^{\\hIcg} \\left( q(y_0)-q(y_{\\hIcg+1}) + \\frac{\\eps}{2}\\|y_0-y_{\\hIcg+1}\\|^2 \\right) \\\\\n\t\t &\\le \\tau^{\\hIcg} \\|r_0\\|\\|r_{\\hIcg}\\| \n\t\t \\frac{\\tau^{-\\hIcg/2}}{\\eps}\\frac{1}{1-\\sqrt{\\tau}}.\n\\end{align*}\nAfter rearranging this inequality and dividing by $\\|r_{\\hIcg}\\| > 0$,\nwe obtain\n\\begin{equation} \\label{eq:finalcontrad}\n\t\t\\|r_{\\hIcg}\\| \\le \\frac{2 (M+2\\eps)^2}{\\eps^2}\\frac{\\tau^{\\hIcg/2}}{1-\\sqrt{\\tau}} \\|r_0\\|\n\t\t= \\sqrt{T} \\tau^{\\hIcg/2}\\|r_0\\|.\n\\end{equation}\nWe have thus established \\eqref{eq:cvCGwhilestronglycvx} which, as we\nnoted earlier, contradicts \\eqref{eq:Kitsstronglycvx}. Thus\n\\eqref{eq:curvcondyKp1} cannot be true, so we have established\n\\eqref{eq:yKp1negcurv}, as required.\n\\end{proof}","sha256_utf8":"78505c7819db7878210eceb76266a0d578d9528b250bd5ed754e0634aee4d20e"}]},{"id":"S074","semantic_type":"lemma","kind":"lemma","title":"曲率商的标量计算公式","source_units":[{"start":92680,"end":93102,"start_line":2017,"end_line":2026,"raw_tex":"\\begin{lemma} \\label{lemma:cgcurvaturescalars}\nSuppose that Algorithm~\\ref{alg:ccg} computes iterates up to iteration\n$j+1$. Then, for any $i \\in \\{0,\\dotsc,j\\}$, we can compute\n(\\ref{eq:weakcurvdir}) as\n\\[\n\\frac{(y_{j+1}-y_i)^\\top \\bH (y_{j+1}-y_i)}{\\|y_{j+1}-y_i\\|^2} =\n\\frac{\\sum_{k=i}^{j} \\alpha_k \\|r_k\\|^2}\n{\\sum_{\\ell=0}^j \\left[\\sum_{k=\\max\\{\\ell,i\\}}^j \\alpha_k \\|r_k\\|^2 \\right]^2/\\|r_{\\ell}\\|^2}.\n\\]\n\\end{lemma}","sha256_utf8":"39fbe62850b69a4a4371e03a9eaa9e5c2487a711dbbfa2d8345b13baacb08986"}]},{"id":"S075","semantic_type":"proof","kind":"proof","title":"标量曲率公式完整证明","source_units":[{"start":93103,"end":94423,"start_line":2027,"end_line":2055,"raw_tex":"\\begin{proof}\nBy definition, $y_{j+1}-y_i = \\sum_{k=i}^{j} \\alpha_k p_k$. By\nconjugacy of the $p_k$ vectors, we have\n\\begin{equation} \\label{eq:numeratorcomp}\n(y_{j+1}-y_i)^\\top \\bH (y_{j+1}-y_i)\n= \\sum_{k=i}^{j} \\alpha_k^2 p_k^\\top \\bH p_k\n= \\sum_{k=i}^{j} \\alpha_k \\|r_k\\|^2,\n\\end{equation}\nwhere we used the definition of $\\alpha_k$ to obtain the last\nequality.  Now we turn our attention to the denominator. Using\nProperty 6 of Lemma~\\ref{lemma:propertiesCGoneiter}, we have that\n\\[\ny_{j+1}-y_i = \\sum_{k=i}^{j} \\alpha_k p_k =\n\\sum_{k=i}^{j} \\alpha_k \\left(-\\sum_{\\ell=0}^k \\frac{\\|r_k\\|^2}{\\|r_\\ell\\|^2} r_\\ell \\right),\n\\]\nBy rearranging the terms in the sum, we obtain\n\\begin{equation*}\ny_{j+1}-y_i = -\\sum_{k=i}^{j} \\sum_{\\ell=0}^k \\alpha_k\\|r_k\\|^2 \\frac{r_\\ell}{\\|r_{\\ell}\\|^2} \n= -\\sum_{\\ell=0}^j \\left[\\sum_{k=\\max\\{\\ell,i\\}}^j \\alpha_k \\|r_k\\|^2 \\right] \\frac{r_\\ell}{\\|r_\\ell\\|^2}.\n\\end{equation*}\nUsing the fact that the residuals $\\{ r_{\\ell} \\}_{\\ell=0,1,\\dotsc,j}$\nform an orthogonal set (by Property 2 of\nLemma~\\ref{lemma:propertiesCGoneiter}), we have that\n\\begin{equation*}\n  \\|y_{j+1}-y_i\\|^2 = \\sum_{\\ell=0}^j \\frac{1}{\\|r_{\\ell} \\|^2}\n  \\left[\\sum_{k=\\max\\{\\ell,i\\}}^j \\alpha_k \\|r_k\\|^2 \\right]^2.\n\\end{equation*}\nCombining this with~\\eqref{eq:numeratorcomp} gives the desired result.\n\\end{proof}","sha256_utf8":"1f8b16b93d8b82662c6f812f5ca40ac803fe720e2d6d8f99d09bfa6023369368"}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","title":"展开曲率分子与方向差","source_units":[{"start":93103,"end":93731,"start_line":2027,"end_line":2041,"raw_tex":"\\begin{proof}\nBy definition, $y_{j+1}-y_i = \\sum_{k=i}^{j} \\alpha_k p_k$. By\nconjugacy of the $p_k$ vectors, we have\n\\begin{equation} \\label{eq:numeratorcomp}\n(y_{j+1}-y_i)^\\top \\bH (y_{j+1}-y_i)\n= \\sum_{k=i}^{j} \\alpha_k^2 p_k^\\top \\bH p_k\n= \\sum_{k=i}^{j} \\alpha_k \\|r_k\\|^2,\n\\end{equation}\nwhere we used the definition of $\\alpha_k$ to obtain the last\nequality.  Now we turn our attention to the denominator. Using\nProperty 6 of Lemma~\\ref{lemma:propertiesCGoneiter}, we have that\n\\[\ny_{j+1}-y_i = \\sum_{k=i}^{j} \\alpha_k p_k =\n\\sum_{k=i}^{j} \\alpha_k \\left(-\\sum_{\\ell=0}^k \\frac{\\|r_k\\|^2}{\\|r_\\ell\\|^2} r_\\ell \\right),\n\\]\n","sha256_utf8":"ffefa1b230b127922743626e792b23cbe23993b6efb5b5376bcfba46f95c3cb5"}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","title":"重排方向差的残差展开","source_units":[{"start":93731,"end":94014,"start_line":2042,"end_line":2046,"raw_tex":"By rearranging the terms in the sum, we obtain\n\\begin{equation*}\ny_{j+1}-y_i = -\\sum_{k=i}^{j} \\sum_{\\ell=0}^k \\alpha_k\\|r_k\\|^2 \\frac{r_\\ell}{\\|r_{\\ell}\\|^2} \n= -\\sum_{\\ell=0}^j \\left[\\sum_{k=\\max\\{\\ell,i\\}}^j \\alpha_k \\|r_k\\|^2 \\right] \\frac{r_\\ell}{\\|r_\\ell\\|^2}.\n\\end{equation*}\n","sha256_utf8":"d8af7237c23a3659b128392bcd2aefc33bb98444010c3ba06330f79a1e35406e"}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","title":"残差正交给出曲率分母","source_units":[{"start":94014,"end":94423,"start_line":2047,"end_line":2055,"raw_tex":"Using the fact that the residuals $\\{ r_{\\ell} \\}_{\\ell=0,1,\\dotsc,j}$\nform an orthogonal set (by Property 2 of\nLemma~\\ref{lemma:propertiesCGoneiter}), we have that\n\\begin{equation*}\n  \\|y_{j+1}-y_i\\|^2 = \\sum_{\\ell=0}^j \\frac{1}{\\|r_{\\ell} \\|^2}\n  \\left[\\sum_{k=\\max\\{\\ell,i\\}}^j \\alpha_k \\|r_k\\|^2 \\right]^2.\n\\end{equation*}\nCombining this with~\\eqref{eq:numeratorcomp} gives the desired result.\n\\end{proof}","sha256_utf8":"0eb0dd033e7d80dcd09afd70257f0c9488df8739df89b285061831e3a89da20d"}]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","title":"标准共轭梯度恒等式","source_units":[],"declared_external":true,"declaration_text":["Nocedal and Wright (2006), Chapter 5: in exact arithmetic, while the performed CG search directions have positive curvature, iterates lie in preceding search-direction spans; residuals are mutually orthogonal and orthogonal to preceding search directions; search directions are matrix-conjugate; r_i^T p_i = -||r_i||²; ||r_i|| ≤ ||p_i||; p_i = -Σ_{k=0}^i (||r_i||²/||r_k||²)r_k; and q(y_{i+1}) = q(y_i) - ||r_i||⁴/(2p_i^T H̄p_i). After n performed iterations the residual is zero."]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","title":"已有近似牛顿线搜索估计","source_units":[],"declared_external":true,"declaration_text":["Royer and Wright (2018), Lemma 13 and its proof, including equation (65): for the inexact damped-Newton direction satisfying the shifted-curvature and residual conditions used in B0043, the cubic sufficient-decrease backtracking proof gives the line-search termination and decrease estimates recorded in B0057 once the step-norm bound ||d_k|| ≤ 1.1 ε_H^{-1}||∇f(x_k)|| is available. B0061 explicitly supplies that bound deterministically instead of using the cited proof's probabilistic eigenvalue argument."]},{"id":"S081","semantic_type":"external_fact","kind":"external_fact","title":"已有负曲率线搜索估计","source_units":[],"declared_external":true,"declaration_text":["Royer and Wright (2018), Lemma 1: with Hessian Lipschitz constant L_H, a direction satisfying ∇f(x)^T d ≤ 0, d^T∇²f(x)d = -||d||³, and ||d|| ≥ ε_H yields termination of cubic sufficient-decrease backtracking with j_k ≤ [log_θ(3/(L_H+η))]_+ + 1 and decrease at least (η/6)min{1,27θ³/(L_H+η)³}ε_H³."]},{"id":"S082","semantic_type":"external_fact","kind":"external_fact","title":"对数函数初等下界","source_units":[],"declared_external":true,"declaration_text":["For every t > 0, ln(1 + 1/t) ≥ 1/(t + 1/2)."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S014","edges":[{"source_id":"S001","target_id":"S005","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S002","target_id":"S019","kind":"supports"},{"source_id":"S002","target_id":"S016","kind":"supports"},{"source_id":"S005","target_id":"S029","kind":"supports"},{"source_id":"S005","target_id":"S024","kind":"supports"},{"source_id":"S005","target_id":"S025","kind":"supports"},{"source_id":"S005","target_id":"S026","kind":"supports"},{"source_id":"S005","target_id":"S055","kind":"supports"},{"source_id":"S006","target_id":"S029","kind":"supports"},{"source_id":"S006","target_id":"S033","kind":"supports"},{"source_id":"S007","target_id":"S057","kind":"supports"},{"source_id":"S008","target_id":"S040","kind":"supports"},{"source_id":"S008","target_id":"S041","kind":"supports"},{"source_id":"S008","target_id":"S042","kind":"supports"},{"source_id":"S008","target_id":"S043","kind":"supports"},{"source_id":"S008","target_id":"S050","kind":"supports"},{"source_id":"S008","target_id":"S051","kind":"supports"},{"source_id":"S008","target_id":"S052","kind":"supports"},{"source_id":"S008","target_id":"S053","kind":"supports"},{"source_id":"S008","target_id":"S055","kind":"supports"},{"source_id":"S008","target_id":"S057","kind":"supports"},{"source_id":"S008","target_id":"S060","kind":"supports"},{"source_id":"S008","target_id":"S064","kind":"supports"},{"source_id":"S008","target_id":"S068","kind":"supports"},{"source_id":"S008","target_id":"S070","kind":"supports"},{"source_id":"S008","target_id":"S072","kind":"supports"},{"source_id":"S009","target_id":"S019","kind":"supports"},{"source_id":"S009","target_id":"S021","kind":"supports"},{"source_id":"S009","target_id":"S037","kind":"supports"},{"source_id":"S010","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S020","kind":"supports"},{"source_id":"S010","target_id":"S021","kind":"supports"},{"source_id":"S010","target_id":"S022","kind":"supports"},{"source_id":"S010","target_id":"S023","kind":"supports"},{"source_id":"S010","target_id":"S029","kind":"supports"},{"source_id":"S010","target_id":"S033","kind":"supports"},{"source_id":"S010","target_id":"S037","kind":"supports"},{"source_id":"S010","target_id":"S043","kind":"supports"},{"source_id":"S010","target_id":"S016","kind":"supports"},{"source_id":"S011","target_id":"S019","kind":"supports"},{"source_id":"S011","target_id":"S016","kind":"supports"},{"source_id":"S012","target_id":"S046","kind":"supports"},{"source_id":"S012","target_id":"S047","kind":"supports"},{"source_id":"S013","target_id":"S046","kind":"supports"},{"source_id":"S013","target_id":"S060","kind":"supports"},{"source_id":"S013","target_id":"S064","kind":"supports"},{"source_id":"S013","target_id":"S070","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S022","kind":"supports"},{"source_id":"S020","target_id":"S023","kind":"supports"},{"source_id":"S020","target_id":"S024","kind":"supports"},{"source_id":"S020","target_id":"S018","kind":"supports"},{"source_id":"S021","target_id":"S024","kind":"supports"},{"source_id":"S021","target_id":"S018","kind":"supports"},{"source_id":"S022","target_id":"S025","kind":"supports"},{"source_id":"S022","target_id":"S018","kind":"supports"},{"source_id":"S023","target_id":"S026","kind":"supports"},{"source_id":"S023","target_id":"S018","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S026","kind":"supports"},{"source_id":"S024","target_id":"S018","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S018","kind":"supports"},{"source_id":"S026","target_id":"S018","kind":"supports"},{"source_id":"S027","target_id":"S022","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S022","kind":"supports"},{"source_id":"S030","target_id":"S036","kind":"supports"},{"source_id":"S030","target_id":"S037","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S034","target_id":"S021","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S029","kind":"supports"},{"source_id":"S038","target_id":"S032","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S041","target_id":"S039","kind":"supports"},{"source_id":"S042","target_id":"S039","kind":"supports"},{"source_id":"S043","target_id":"S039","kind":"supports"},{"source_id":"S044","target_id":"S041","kind":"supports"},{"source_id":"S044","target_id":"S050","kind":"supports"},{"source_id":"S044","target_id":"S064","kind":"supports"},{"source_id":"S044","target_id":"S066","kind":"supports"},{"source_id":"S044","target_id":"S067","kind":"supports"},{"source_id":"S044","target_id":"S070","kind":"supports"},{"source_id":"S044","target_id":"S071","kind":"supports"},{"source_id":"S044","target_id":"S076","kind":"supports"},{"source_id":"S044","target_id":"S078","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S048","target_id":"S016","kind":"supports"},{"source_id":"S048","target_id":"S056","kind":"supports"},{"source_id":"S049","target_id":"S048","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S051","target_id":"S053","kind":"supports"},{"source_id":"S051","target_id":"S049","kind":"supports"},{"source_id":"S052","target_id":"S049","kind":"supports"},{"source_id":"S053","target_id":"S049","kind":"supports"},{"source_id":"S054","target_id":"S016","kind":"supports"},{"source_id":"S055","target_id":"S057","kind":"supports"},{"source_id":"S055","target_id":"S072","kind":"supports"},{"source_id":"S055","target_id":"S054","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S056","target_id":"S054","kind":"supports"},{"source_id":"S057","target_id":"S054","kind":"supports"},{"source_id":"S058","target_id":"S043","kind":"supports"},{"source_id":"S058","target_id":"S051","kind":"supports"},{"source_id":"S059","target_id":"S058","kind":"supports"},{"source_id":"S060","target_id":"S062","kind":"supports"},{"source_id":"S060","target_id":"S070","kind":"supports"},{"source_id":"S060","target_id":"S071","kind":"supports"},{"source_id":"S060","target_id":"S073","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S063","kind":"supports"},{"source_id":"S061","target_id":"S065","kind":"supports"},{"source_id":"S061","target_id":"S066","kind":"supports"},{"source_id":"S061","target_id":"S059","kind":"supports"},{"source_id":"S062","target_id":"S069","kind":"supports"},{"source_id":"S062","target_id":"S059","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S063","target_id":"S068","kind":"supports"},{"source_id":"S063","target_id":"S059","kind":"supports"},{"source_id":"S064","target_id":"S068","kind":"supports"},{"source_id":"S064","target_id":"S059","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S067","kind":"supports"},{"source_id":"S065","target_id":"S059","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S068","kind":"supports"},{"source_id":"S066","target_id":"S059","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S059","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S059","kind":"supports"},{"source_id":"S069","target_id":"S073","kind":"supports"},{"source_id":"S069","target_id":"S059","kind":"supports"},{"source_id":"S070","target_id":"S073","kind":"supports"},{"source_id":"S070","target_id":"S059","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S059","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S059","kind":"supports"},{"source_id":"S073","target_id":"S059","kind":"supports"},{"source_id":"S074","target_id":"S053","kind":"supports"},{"source_id":"S075","target_id":"S074","kind":"supports"},{"source_id":"S076","target_id":"S077","kind":"supports"},{"source_id":"S076","target_id":"S078","kind":"supports"},{"source_id":"S076","target_id":"S075","kind":"supports"},{"source_id":"S077","target_id":"S078","kind":"supports"},{"source_id":"S077","target_id":"S075","kind":"supports"},{"source_id":"S078","target_id":"S075","kind":"supports"},{"source_id":"S079","target_id":"S046","kind":"supports"},{"source_id":"S080","target_id":"S029","kind":"supports"},{"source_id":"S081","target_id":"S033","kind":"supports"},{"source_id":"S082","target_id":"S056","kind":"supports"}]}}
{"record_id":"1803.09941v5","paper":{"title":"Iteration complexity of first-order augmented Lagrangian methods for convex conic programming","authors":["Zhaosong Lu","Zirui Zhou"],"arxiv_id":"1803.09941","version":"v5","journal":"SIAM Journal on Optimization","doi":"10.1137/21m1403837","abstract_url":"https://arxiv.org/abs/1803.09941v5","pdf_url":"https://arxiv.org/pdf/1803.09941v5"},"context":{"raw_tex":"In this paper we consider convex conic programming in the form of\r\n\\begin{equation}\\label{eq:conic-cp}\r\n\\begin{array}{rl}\r\nF^*=\\min & \\left\\{F(x) := f(x) + P(x)\\right\\} \\\\\r\n\\mbox{s.t.} &  g(x)\\preceq_\\mcK 0,\r\n\\end{array}\r\n\\end{equation}\r\nwhere $f, P:\\Re^n\\rightarrow(-\\infty,\\infty]$ are proper closed convex functions, $\\mcK$ is a closed convex cone in $\\Re^m$, the symbol $\\preceq_\\mcK$ denotes the partial order induced by $\\mcK$, that is, $y\\preceq_\\mcK z$ if and only if $z-y\\in\\mcK$, and the mapping $g:\\Re^n\\rightarrow\\Re^m$ is convex with respect to $\\mcK$, that is,\r\n\\begin{equation}\r\n\\label{eq:conic-cvx}\r\ng(\\alpha x + (1-\\alpha)y) \\preceq_\\mcK \\alpha g(x) + (1-\\alpha)g(y), \\quad\\forall x,y\\in\\Re^n, \\; \\alpha\\in[0,1].\r\n\\end{equation}\r\nThe associated Lagrangian dual problem of \\eqref{eq:conic-cp} is given by \r\n\\beq \\label{dual-prob}\r\nd^*=\\sup\\limits_{\\lambda\\in\\mcK^{*}} \\inf\\limits_x\\  \\{ f(x) + P(x) + \\Inner{\\lambda}{g(x)}\\},\r\n\\eeq\r\nwhere $\\mcK^*$ is the dual cone of $\\mcK$ (see Section \\ref{sec:preliminaries}). \r\n\n\n\\begin{ass}\r\n\t\\label{ass:function}\r\n\t\\begin{itemize}\r\n\t\t\\item[(a)] The proximal operator associated with $P$ can be exactly evaluated, and the domain of $P$, denoted by $\\dom(P)$, is compact.\\footnote{Some problems with unbounded $\\dom(P)$ can be reformulated as the ones satisfying this assumption. For example, when $f$ is bounded below and $P$ is coercive, the problem generally can be reformulated as the one with the objective $f+\\tilde P$ for some $\\tilde P$ with a compact domain. Such a problem often arises in sparse or low-rank learning, in which $f$ is typically a nonnegative loss function and $P$ is the $\\ell_1$- or nuclear-norm.}\r\n\t\t\\item[(b)] The projection onto $\\mcK$ can be exactly evaluated.\r\n\t\t\\item[(c)] The functions $f$ and $g$ are continuously differentiable on an open set $\\Omega$ containing $\\dom(P)$,  and $\\nabla f$ and $\\nabla g$  are Lipschitz continuous on $\\Omega$ with Lipschitz constants $L_{\\nabla f}$ and $L_{\\nabla g}$, respectively.\\footnote{The symbol $\\nabla g$ denotes the transpose of the Jacobian of $g$.}\r\n\\item[(d)] The strong duality holds for problems \\eqref{eq:conic-cp} and \\eqref{dual-prob}, that is, both problems have optimal solutions and moreover their optimal values $F^*$ and $d^*$ are equal. \r\n\t\\end{itemize}\r\n\\end{ass}\n\n\\begin{equation}\r\n\\label{eq:eps-approx-kkt-point}\r\n\\dist(0, \\nabla f(x) + \\partial P(x) + \\nabla g(x)\\lambda) \\leq \\epsilon, \\quad \\dist(g(x), \\mcN_{\\mcK^*}(\\lambda))\\leq \\epsilon, \\quad (x,\\lambda) \\in \\dom(P) \\times \\mcK^*\r\n\\end{equation}\n\nThe following notations will be used throughout this paper. Let $\\Re^n$ denote the Euclidean space of dimension $n$, $\\langle\\cdot,\\cdot\\rangle$ denote the standard inner product, and $\\|\\cdot\\|$ stand for the Euclidean norm. The symbols $\\Re_+$ and $\\Re_{++}$ stand for the set of nonnegative and positive real numbers, respectively.  \r\n\r\nGiven a closed convex function $h:\\Re^n\\rightarrow(-\\infty,\\infty]$, $\\partial h$ and $\\dom (h)$ denote the subdifferential and domain  of $h$, respectively.  The proximal operator associated with $h$ is denoted by  \r\n$\\prox_h$,  that is,\r\n\\begin{equation*}\r\n\\label{eq:def-prox}\r\n\\prox_h(z) = \\argmin_{x\\in\\Re^n} \\left\\{ \\frac{1}{2}\\|x - z\\|^2 + h(x) \\right\\}, \\quad \\forall z \\in \\Re^n.\r\n\\end{equation*}\r\nGiven a non-empty closed convex set $C\\subseteq\\Re^n$, $\\dist(z,C)$ stands for the Euclidean distance from $z$ to $C$, and $\\Pi_C(z)$ denotes the Euclidean projection of $z$ onto $C$, namely, \r\n\\[\r\n \\Pi_C(z) = \\argmin\\{\\|z - x\\|: x\\in C\\}, \\quad \\dist(z,C)=\\left\\|z - \\Pi_C(z)\\right\\|, \\quad \\forall z \\in \\Re^n.\r\n\\]\r\nThe normal cone of $C$ at any $z\\in C$ is denoted by $\\mcN_C(z)$. For the closed convex cone $\\mcK$, we use $\\mcK^*$ to denote the dual cone of $\\mcK$, that is,  $\\mcK^* = \\{y\\in\\Re^m: \\langle y,x\\rangle\\geq 0, \\; \\forall x\\in\\mcK\\}$.\r\n\r\nThe Lagrangian function $l(x,\\lambda)$ of problem~\\eqref{eq:conic-cp} is defined as\r\n\\begin{equation*}\r\nl(x,\\lambda) = \\left\\{\r\n\\begin{array}{ll}\r\nf(x) + P(x) + \\Inner{\\lambda}{g(x)} & \\mbox{if}\\; x\\in \\dom(P) \\mbox{ and } \\lambda \\in \\mcK^{*}, \\\\\r\n-\\infty & \\mbox{if}\\; x\\in \\dom(P) \\mbox{ and } \\lambda\\notin\\mcK^{*}, \\\\\r\n\\infty & \\mbox{if}\\; x\\notin \\dom(P),\r\n\\end{array}\r\n\\right.\r\n\\end{equation*}\r\nwhich is a closed convex-concave function. The Lagrangian dual function $d:\\Re^m\\rightarrow[-\\infty,\\infty)$ is defined as \r\n\\begin{equation}\r\n\\label{eq:lag-dual-func}\r\nd(\\lambda) = \\inf_x l(x,\\lambda) = \\left\\{\r\n\\begin{array}{ll}\r\n\\inf\\limits_x\\  \\{ f(x) + P(x) + \\Inner{\\lambda}{g(x)}\\} & \\mbox{if}\\; \\lambda\\in\\mcK^{*},\\\\\r\n-\\infty & \\mbox{if}\\; \\lambda\\notin\\mcK^{*},\r\n\\end{array}\r\n\\right.\r\n\\end{equation}\r\nwhich is a closed concave function. Moreover, the augmented Lagrangian function for problem~\\eqref{eq:conic-cp} is defined as (e.g., see~\\cite{Shapiro04})\r\n\\begin{equation}\r\n\\label{eq:al-func}\r\n\\mcL(x,\\lambda;\\rho) = f(x) + P(x) + \\frac{1}{2\\rho}\\Big[\\dist^2\\big(\\lambda + \\rho g(x), -\\mcK\\big) - \\|\\lambda\\|^2\\Big],\r\n\\end{equation}\r\nwhere $\\rho>0$ is  a penalty parameter. For convenience, we let\r\n\\beq \\label{S-fun}\r\n\\mcS(x,\\lambda;\\rho):= f(x) + \\frac{1}{2\\rho} \\dist^2\\big(\\lambda+\\rho g(x), -\\mcK \\big).\r\n\\eeq\r\nIt is clear to see that \r\n\\[\r\n\\mcL(x,\\lambda;\\rho) = \\mcS(x,\\lambda;\\rho) + P(x) - \\frac{\\|\\lambda\\|^2}{2\\rho}.\r\n\\]\r\nThe augmented Lagrangian dual function of~\\eqref{eq:conic-cp} is given by\r\n\\begin{equation}\r\n\\label{eq:aug-dual-func}\r\nd(\\lambda;\\rho)= \\min_{x\\in\\Re^n} \\mcL(x,\\lambda;\\rho).\r\n\\end{equation}\r\nThe Lagrangian dual problem \\eqref{dual-prob} can thus be rewritten as  \r\n\\begin{equation}\\label{eq:dual-conic-cp}\r\nd^{*} = \\max_\\lambda\\; d(\\lambda).\r\n\\end{equation}\r\nLet $\\partial l:\\Re^n\\times\\Re^m\\rightrightarrows\\Re^n\\times \\Re^m$ and $\\partial d:\\Re^m\\rightrightarrows\\Re^m$ be respectively the subdifferential mappings associated with $l$ and $d$ (e.g., see \\cite{R97}). We define two set-valued operators associated with problems \\eqref{eq:conic-cp} and \\eqref{dual-prob} as follows:\r\n\\beqa\r\n&&\\mcT_d: \\lambda \\rightarrow \\{ u\\in\\Re^m: -u\\in \\partial d(\\lambda)\\},  \\quad \\forall \\lambda\\in\\Re^m, \\label{eq:def-two-opt1} \\\\\r\n&& \\mcT_l: (x,\\lambda) \\rightarrow \\{ (v,u)\\in\\Re^n\\times\\Re^m: (v,-u)\\in\\partial l(x,\\lambda)\\}, \\quad \\forall (x,\\lambda) \\in \\Re^n \\times \\Re^m. \\label{eq:def-two-opt2}\r\n\\eeqa\r\nIt is well known that $\\lambda^*$ is an optimal solution of the Lagrangian dual problem~\\eqref{eq:dual-conic-cp} if and only if $0\\in \\partial d(\\lambda^*)$, and $(x^*,\\lambda^*)$ is a saddle point\\footnote{$(x^*,\\lambda^*)$ is called a saddle point of $l$ if it satisfies $ \\sup_{\\lambda}l(x^*,\\lambda) = l(x^*,\\lambda^*) = \\inf_x l(x,\\lambda^*)$.} of $l$ if and only if $(0,0)\\in\\partial l(x^*,\\lambda^*)$. In addition, it can be verified that \r\n\\begin{equation}\r\n\\label{eq:express-Tl}\r\n\\partial l(x,\\lambda) = \\left\\{\r\n\\begin{array}{ll}\r\n\\left(\r\n\\begin{array}{c}\r\n\\nabla f(x) + \\partial P(x) + \\nabla g(x)\\lambda \\\\\r\ng(x) - \\mcN_{\\mcK^*}(\\lambda)\r\n\\end{array}\r\n\\right), & \\mbox{if}\\; x\\in\\dom(P)\\; \\mbox{and}\\; \\lambda\\in\\mcK^*, \\medskip\r\n\\\\ \r\n\\emptyset, & \\mbox{otherwise}.\r\n\\end{array}\\right.\r\n\\end{equation}\r\n\n\n\\begin{defi}\r\n\t\\label{defi:approx-KKT}\r\n\tGiven any $\\epsilon>0$, we say $(x,\\lambda)\\in\\Re^n\\times\\Re^m$ is an $\\epsilon$-KKT solution of~\\eqref{eq:conic-cp}  if there exists $(u,v)\\in\\partial l(x,\\lambda)$ such that $\\|u\\|\\leq \\epsilon$ and $\\|v\\|\\leq \\epsilon$.\r\n\\end{defi}\n\nWe next present the subroutine Postprocessing that is used in Step 4 of Algorithm \\ref{alg:i-al}. Before proceeding, let $L_g$ be the  Lipschitz constant of $g$ on $\\dom(P)$ and $M_g = \\max\\limits_{x\\in\\dom(P)}\\|g(x)\\|$.  \r\n\n\nIn the rest of this section we present our main results for Algorithm \\ref{alg:i-al}, whose proof is deferred  to Subsection \\ref{sec:pf-i-al}. To proceed, we introduce some further notation below. Let $\\Lambda^*$ be the set of optimal solutions of problem \\eqref{dual-prob} and $\\hat\\lambda^*\\in \\Lambda^*$ such that $\\|\\lambda^0-\\hat\\lambda^*\\|=\\dist(\\lambda^0,\\Lambda^*)$. In addition, we define \r\n\\begin{align}\r\n & D_X = \\max\\limits_{x,y\\in\\dom(P)}\\|x - y\\|,  \\  D_\\Lambda = \\|\\lambda^0 - \\hat\\lambda^*\\|, \\ B = L_{\\nabla f} + L_{\\nabla g}(\\|\\hat\\lambda^*\\| + D_\\Lambda), \\label{param} \\\\\r\n & C = L_{\\nabla g}M_g + L_g^2, \\ \\bD=\\max\\{D_\\Lambda,1\\}, \\ \\bB =\\max\\{B,1\\}, \\ \\bC =\\max\\{C,1\\}, \\label{bparam}\r\n\\end{align}\r\nwhere  $L_{\\nabla f}$, $L_{\\nabla g}$, $L_g$ and $M_g$ are defined above.\r\n\n\n\\begin{algo}[An adaptively regularized I-AL method]\r\n\t\\label{alg:mi-al}\r\n\t\\normalfont\r\n\t\\mbox{}\r\n\t\\begin{itemize}\r\n\t\t\\item[0.] Input $\\epsilon>0$, $(x^0,\\lambda^0)\\in\\dom(P)\\times \\mcK^*$,  nondecreasing $\\{\\rho_k\\} \\subset \\Re_{++}$, and $0<\\eta_k \\downarrow 0$. Set $k=0$.\r\n\t\t\\item[1.] Apply Algorithm~\\ref{alg:opt-fom-str} (in Appendix \\ref{app-fom}) to the problem $\\min_x \\varphi_k(x)$ to  find $x^{k+1}\\in\\dom(P)$ satisfying\r\n\t\t\\begin{equation}\r\n\t\t\\label{eq:inext-subprob-m}\r\n\t\t\\dist (0, \\partial \\varphi_k(x^{k+1})) \\leq \\eta_k,\\footnote{In view of Proposition \\ref{prop:cplx-fom-str-cvx},  one can terminate Algorithm~\\ref{alg:opt-fom-str} once a point $x^{k+1}$ satisfying $2L_{\\nabla \\phi}\\|\\tilde{x}^{k+1} - x^{k+1}\\| \\le \\eta_k$ is found, where $\\phi(\\cdot)= \\varphi_k(\\cdot)-P(\\cdot)$, $\\tilde{x}^{k+1}= \\prox_{P/L_{\\nabla \\phi}}\\left( x^{k+1} - \\nabla \\phi(x^{k+1})/L_{\\nabla \\phi} \\right)$, and $L_{\\nabla \\phi}$ is the Lipschitz constant of $\\nabla \\phi$. Such $x^{k+1}$ clearly satisfies \\eqref{eq:inext-subprob-m}.}\r\n\t\t\\end{equation}\r\n\t\twhere \r\n\t\t\\begin{equation}\r\n\t\t\\label{eq:prox-al-func}\r\n\t\t\\varphi_k(x) = \\mcL(x,\\lambda^k;\\rho_k) + \\frac{1}{2\\rho_k}\\|x - x^k\\|^2.\r\n\t\t\\end{equation}\r\n\t\t\\item[2.] Set $\\lambda^{k+1} = \\Pi_{\\mcK^*}\\big(\\lambda^k + \\rho_k g(x^{k+1}) \\big)$. \r\n\t\t\\item[3.] If $(x^{k+1},\\lambda^{k+1})$ satisfies \\eqref{eq:eps-approx-kkt-point} or the following two inequalities are satisfied\r\n\t\t\\begin{equation}\r\n\t\t\\label{eq:check-stop-m}\r\n\t\t\\frac{1}{\\rho_k}\\|(x^{k+1}, \\lambda^{k+1}) - (x^k,\\lambda^k)\\| \\leq \\frac{\\epsilon}{2}, \\quad \\eta_k \\leq \\frac{\\epsilon}{2},\r\n\t\t\\end{equation}\r\n\t\toutput $(x^+, \\lambda^{+}) = (x^{k+1}, \\lambda^{k+1})$ and terminate the algorithm.\r\n\t\t\\item[4.] Set $k \\leftarrow k+1$ and go to Step 1.\r\n\t\\end{itemize}\r\n\t{\\bf End.}\r\n\\end{algo}\n\nFor ease of later reference, we refer to the iterations of Algorithm~\\ref{alg:opt-fom-str} for solving the \r\nAL subproblems as the {\\it inner iterations} of Algorithm \\ref{alg:mi-al}, and call the update from \r\n$(x^k,\\lambda^k)$ to $(x^{k+1},\\lambda^{k+1})$ an {\\it outer iteration} of Algorithm \\ref{alg:mi-al}. \r\nNotice from \\eqref{eq:prox-al-func} that $\\varphi_k$ is strongly convex with modulus $1/\\rho_k$. The AL subproblem $\\min_x \\varphi_k(x)$ arising in Algorithm \\ref{alg:mi-al} can thus be suitably solved by Algorithm~\\ref{alg:opt-fom-str}.\r\n\n\nIn the rest of this section, we present our main results for Algorithm \\ref{alg:mi-al}, whose proof is deferred  to Subsection \\ref{sec:pf-mi-al}. Before proceeding, we introduce some further notation that will be used subsequently. Let $X^*$ be the set of optimal solutions of problem \\eqref{eq:conic-cp} and $\\hat x^*\\in X^*$ such that \r\n$\\|x^0-\\hat x^*\\|=\\dist(x^0,X^*)$. In addition, we define \r\n\\beq \\label{param-m} \r\n{\\bar D}_X =\\max\\{D_X,1\\}, \\ D=  \\dist(x^0,X^*)+D_\\Lambda, \\ \\bar{D}= \\max\\{D,1\\}, \\ \\hat B = L_{\\nabla f} + L_{\\nabla g}\\|\\hat\\lambda^*\\| + L_{\\nabla g}D, \r\n\\eeq","units":[{"start":5367,"end":6399,"start_line":150,"end_line":166,"raw_tex":"In this paper we consider convex conic programming in the form of\r\n\\begin{equation}\\label{eq:conic-cp}\r\n\\begin{array}{rl}\r\nF^*=\\min & \\left\\{F(x) := f(x) + P(x)\\right\\} \\\\\r\n\\mbox{s.t.} &  g(x)\\preceq_\\mcK 0,\r\n\\end{array}\r\n\\end{equation}\r\nwhere $f, P:\\Re^n\\rightarrow(-\\infty,\\infty]$ are proper closed convex functions, $\\mcK$ is a closed convex cone in $\\Re^m$, the symbol $\\preceq_\\mcK$ denotes the partial order induced by $\\mcK$, that is, $y\\preceq_\\mcK z$ if and only if $z-y\\in\\mcK$, and the mapping $g:\\Re^n\\rightarrow\\Re^m$ is convex with respect to $\\mcK$, that is,\r\n\\begin{equation}\r\n\\label{eq:conic-cvx}\r\ng(\\alpha x + (1-\\alpha)y) \\preceq_\\mcK \\alpha g(x) + (1-\\alpha)g(y), \\quad\\forall x,y\\in\\Re^n, \\; \\alpha\\in[0,1].\r\n\\end{equation}\r\nThe associated Lagrangian dual problem of \\eqref{eq:conic-cp} is given by \r\n\\beq \\label{dual-prob}\r\nd^*=\\sup\\limits_{\\lambda\\in\\mcK^{*}} \\inf\\limits_x\\  \\{ f(x) + P(x) + \\Inner{\\lambda}{g(x)}\\},\r\n\\eeq\r\nwhere $\\mcK^*$ is the dual cone of $\\mcK$ (see Section \\ref{sec:preliminaries}). \r\n","sha256_utf8":"e01e13f0b00c8c3a21c38af1d9c1635e743bae74992d1f8d4f5bb5fa718e9540"},{"start":6522,"end":7800,"start_line":168,"end_line":176,"raw_tex":"\\begin{ass}\r\n\t\\label{ass:function}\r\n\t\\begin{itemize}\r\n\t\t\\item[(a)] The proximal operator associated with $P$ can be exactly evaluated, and the domain of $P$, denoted by $\\dom(P)$, is compact.\\footnote{Some problems with unbounded $\\dom(P)$ can be reformulated as the ones satisfying this assumption. For example, when $f$ is bounded below and $P$ is coercive, the problem generally can be reformulated as the one with the objective $f+\\tilde P$ for some $\\tilde P$ with a compact domain. Such a problem often arises in sparse or low-rank learning, in which $f$ is typically a nonnegative loss function and $P$ is the $\\ell_1$- or nuclear-norm.}\r\n\t\t\\item[(b)] The projection onto $\\mcK$ can be exactly evaluated.\r\n\t\t\\item[(c)] The functions $f$ and $g$ are continuously differentiable on an open set $\\Omega$ containing $\\dom(P)$,  and $\\nabla f$ and $\\nabla g$  are Lipschitz continuous on $\\Omega$ with Lipschitz constants $L_{\\nabla f}$ and $L_{\\nabla g}$, respectively.\\footnote{The symbol $\\nabla g$ denotes the transpose of the Jacobian of $g$.}\r\n\\item[(d)] The strong duality holds for problems \\eqref{eq:conic-cp} and \\eqref{dual-prob}, that is, both problems have optimal solutions and moreover their optimal values $F^*$ and $d^*$ are equal. \r\n\t\\end{itemize}\r\n\\end{ass}","sha256_utf8":"9427b3e0ccf40a108133047fcead4e109d57603f250bdbba854f84ee20b3a587"},{"start":10951,"end":11191,"start_line":204,"end_line":207,"raw_tex":"\\begin{equation}\r\n\\label{eq:eps-approx-kkt-point}\r\n\\dist(0, \\nabla f(x) + \\partial P(x) + \\nabla g(x)\\lambda) \\leq \\epsilon, \\quad \\dist(g(x), \\mcN_{\\mcK^*}(\\lambda))\\leq \\epsilon, \\quad (x,\\lambda) \\in \\dom(P) \\times \\mcK^*\r\n\\end{equation}","sha256_utf8":"764f253c4b7d1a6ce128bcd43749f8d98830ef939fe599cdffdde22b7794cca3"},{"start":22603,"end":27170,"start_line":272,"end_line":347,"raw_tex":"The following notations will be used throughout this paper. Let $\\Re^n$ denote the Euclidean space of dimension $n$, $\\langle\\cdot,\\cdot\\rangle$ denote the standard inner product, and $\\|\\cdot\\|$ stand for the Euclidean norm. The symbols $\\Re_+$ and $\\Re_{++}$ stand for the set of nonnegative and positive real numbers, respectively.  \r\n\r\nGiven a closed convex function $h:\\Re^n\\rightarrow(-\\infty,\\infty]$, $\\partial h$ and $\\dom (h)$ denote the subdifferential and domain  of $h$, respectively.  The proximal operator associated with $h$ is denoted by  \r\n$\\prox_h$,  that is,\r\n\\begin{equation*}\r\n\\label{eq:def-prox}\r\n\\prox_h(z) = \\argmin_{x\\in\\Re^n} \\left\\{ \\frac{1}{2}\\|x - z\\|^2 + h(x) \\right\\}, \\quad \\forall z \\in \\Re^n.\r\n\\end{equation*}\r\nGiven a non-empty closed convex set $C\\subseteq\\Re^n$, $\\dist(z,C)$ stands for the Euclidean distance from $z$ to $C$, and $\\Pi_C(z)$ denotes the Euclidean projection of $z$ onto $C$, namely, \r\n\\[\r\n \\Pi_C(z) = \\argmin\\{\\|z - x\\|: x\\in C\\}, \\quad \\dist(z,C)=\\left\\|z - \\Pi_C(z)\\right\\|, \\quad \\forall z \\in \\Re^n.\r\n\\]\r\nThe normal cone of $C$ at any $z\\in C$ is denoted by $\\mcN_C(z)$. For the closed convex cone $\\mcK$, we use $\\mcK^*$ to denote the dual cone of $\\mcK$, that is,  $\\mcK^* = \\{y\\in\\Re^m: \\langle y,x\\rangle\\geq 0, \\; \\forall x\\in\\mcK\\}$.\r\n\r\nThe Lagrangian function $l(x,\\lambda)$ of problem~\\eqref{eq:conic-cp} is defined as\r\n\\begin{equation*}\r\nl(x,\\lambda) = \\left\\{\r\n\\begin{array}{ll}\r\nf(x) + P(x) + \\Inner{\\lambda}{g(x)} & \\mbox{if}\\; x\\in \\dom(P) \\mbox{ and } \\lambda \\in \\mcK^{*}, \\\\\r\n-\\infty & \\mbox{if}\\; x\\in \\dom(P) \\mbox{ and } \\lambda\\notin\\mcK^{*}, \\\\\r\n\\infty & \\mbox{if}\\; x\\notin \\dom(P),\r\n\\end{array}\r\n\\right.\r\n\\end{equation*}\r\nwhich is a closed convex-concave function. The Lagrangian dual function $d:\\Re^m\\rightarrow[-\\infty,\\infty)$ is defined as \r\n\\begin{equation}\r\n\\label{eq:lag-dual-func}\r\nd(\\lambda) = \\inf_x l(x,\\lambda) = \\left\\{\r\n\\begin{array}{ll}\r\n\\inf\\limits_x\\  \\{ f(x) + P(x) + \\Inner{\\lambda}{g(x)}\\} & \\mbox{if}\\; \\lambda\\in\\mcK^{*},\\\\\r\n-\\infty & \\mbox{if}\\; \\lambda\\notin\\mcK^{*},\r\n\\end{array}\r\n\\right.\r\n\\end{equation}\r\nwhich is a closed concave function. Moreover, the augmented Lagrangian function for problem~\\eqref{eq:conic-cp} is defined as (e.g., see~\\cite{Shapiro04})\r\n\\begin{equation}\r\n\\label{eq:al-func}\r\n\\mcL(x,\\lambda;\\rho) = f(x) + P(x) + \\frac{1}{2\\rho}\\Big[\\dist^2\\big(\\lambda + \\rho g(x), -\\mcK\\big) - \\|\\lambda\\|^2\\Big],\r\n\\end{equation}\r\nwhere $\\rho>0$ is  a penalty parameter. For convenience, we let\r\n\\beq \\label{S-fun}\r\n\\mcS(x,\\lambda;\\rho):= f(x) + \\frac{1}{2\\rho} \\dist^2\\big(\\lambda+\\rho g(x), -\\mcK \\big).\r\n\\eeq\r\nIt is clear to see that \r\n\\[\r\n\\mcL(x,\\lambda;\\rho) = \\mcS(x,\\lambda;\\rho) + P(x) - \\frac{\\|\\lambda\\|^2}{2\\rho}.\r\n\\]\r\nThe augmented Lagrangian dual function of~\\eqref{eq:conic-cp} is given by\r\n\\begin{equation}\r\n\\label{eq:aug-dual-func}\r\nd(\\lambda;\\rho)= \\min_{x\\in\\Re^n} \\mcL(x,\\lambda;\\rho).\r\n\\end{equation}\r\nThe Lagrangian dual problem \\eqref{dual-prob} can thus be rewritten as  \r\n\\begin{equation}\\label{eq:dual-conic-cp}\r\nd^{*} = \\max_\\lambda\\; d(\\lambda).\r\n\\end{equation}\r\nLet $\\partial l:\\Re^n\\times\\Re^m\\rightrightarrows\\Re^n\\times \\Re^m$ and $\\partial d:\\Re^m\\rightrightarrows\\Re^m$ be respectively the subdifferential mappings associated with $l$ and $d$ (e.g., see \\cite{R97}). We define two set-valued operators associated with problems \\eqref{eq:conic-cp} and \\eqref{dual-prob} as follows:\r\n\\beqa\r\n&&\\mcT_d: \\lambda \\rightarrow \\{ u\\in\\Re^m: -u\\in \\partial d(\\lambda)\\},  \\quad \\forall \\lambda\\in\\Re^m, \\label{eq:def-two-opt1} \\\\\r\n&& \\mcT_l: (x,\\lambda) \\rightarrow \\{ (v,u)\\in\\Re^n\\times\\Re^m: (v,-u)\\in\\partial l(x,\\lambda)\\}, \\quad \\forall (x,\\lambda) \\in \\Re^n \\times \\Re^m. \\label{eq:def-two-opt2}\r\n\\eeqa\r\nIt is well known that $\\lambda^*$ is an optimal solution of the Lagrangian dual problem~\\eqref{eq:dual-conic-cp} if and only if $0\\in \\partial d(\\lambda^*)$, and $(x^*,\\lambda^*)$ is a saddle point\\footnote{$(x^*,\\lambda^*)$ is called a saddle point of $l$ if it satisfies $ \\sup_{\\lambda}l(x^*,\\lambda) = l(x^*,\\lambda^*) = \\inf_x l(x,\\lambda^*)$.} of $l$ if and only if $(0,0)\\in\\partial l(x^*,\\lambda^*)$. In addition, it can be verified that \r\n\\begin{equation}\r\n\\label{eq:express-Tl}\r\n\\partial l(x,\\lambda) = \\left\\{\r\n\\begin{array}{ll}\r\n\\left(\r\n\\begin{array}{c}\r\n\\nabla f(x) + \\partial P(x) + \\nabla g(x)\\lambda \\\\\r\ng(x) - \\mcN_{\\mcK^*}(\\lambda)\r\n\\end{array}\r\n\\right), & \\mbox{if}\\; x\\in\\dom(P)\\; \\mbox{and}\\; \\lambda\\in\\mcK^*, \\medskip\r\n\\\\ \r\n\\emptyset, & \\mbox{otherwise}.\r\n\\end{array}\\right.\r\n\\end{equation}\r\n","sha256_utf8":"216ca8a944f88512a0a4f9cdd64079a44ccaf0adf948465edeee693c76d8bf61"},{"start":28620,"end":28895,"start_line":367,"end_line":370,"raw_tex":"\\begin{defi}\r\n\t\\label{defi:approx-KKT}\r\n\tGiven any $\\epsilon>0$, we say $(x,\\lambda)\\in\\Re^n\\times\\Re^m$ is an $\\epsilon$-KKT solution of~\\eqref{eq:conic-cp}  if there exists $(u,v)\\in\\partial l(x,\\lambda)$ such that $\\|u\\|\\leq \\epsilon$ and $\\|v\\|\\leq \\epsilon$.\r\n\\end{defi}","sha256_utf8":"50b07a957441d2f800e89b7275fa21cae6ed0f9d8376fa7e301428a211548701"},{"start":33365,"end":33589,"start_line":416,"end_line":416,"raw_tex":"We next present the subroutine Postprocessing that is used in Step 4 of Algorithm \\ref{alg:i-al}. Before proceeding, let $L_g$ be the  Lipschitz constant of $g$ on $\\dom(P)$ and $M_g = \\max\\limits_{x\\in\\dom(P)}\\|g(x)\\|$.  \r\n","sha256_utf8":"7d40651a9a81e6a8d6abde18f309e3e7a328eac05171ebaf848d9c35cdd02b56"},{"start":37153,"end":37950,"start_line":456,"end_line":461,"raw_tex":"In the rest of this section we present our main results for Algorithm \\ref{alg:i-al}, whose proof is deferred  to Subsection \\ref{sec:pf-i-al}. To proceed, we introduce some further notation below. Let $\\Lambda^*$ be the set of optimal solutions of problem \\eqref{dual-prob} and $\\hat\\lambda^*\\in \\Lambda^*$ such that $\\|\\lambda^0-\\hat\\lambda^*\\|=\\dist(\\lambda^0,\\Lambda^*)$. In addition, we define \r\n\\begin{align}\r\n & D_X = \\max\\limits_{x,y\\in\\dom(P)}\\|x - y\\|,  \\  D_\\Lambda = \\|\\lambda^0 - \\hat\\lambda^*\\|, \\ B = L_{\\nabla f} + L_{\\nabla g}(\\|\\hat\\lambda^*\\| + D_\\Lambda), \\label{param} \\\\\r\n & C = L_{\\nabla g}M_g + L_g^2, \\ \\bD=\\max\\{D_\\Lambda,1\\}, \\ \\bB =\\max\\{B,1\\}, \\ \\bC =\\max\\{C,1\\}, \\label{bparam}\r\n\\end{align}\r\nwhere  $L_{\\nabla f}$, $L_{\\nabla g}$, $L_g$ and $M_g$ are defined above.\r\n","sha256_utf8":"4ed29616323492306829627a90706fc310d3e131c4d81c0d29345ebc28192ee7"},{"start":43616,"end":45417,"start_line":532,"end_line":558,"raw_tex":"\\begin{algo}[An adaptively regularized I-AL method]\r\n\t\\label{alg:mi-al}\r\n\t\\normalfont\r\n\t\\mbox{}\r\n\t\\begin{itemize}\r\n\t\t\\item[0.] Input $\\epsilon>0$, $(x^0,\\lambda^0)\\in\\dom(P)\\times \\mcK^*$,  nondecreasing $\\{\\rho_k\\} \\subset \\Re_{++}$, and $0<\\eta_k \\downarrow 0$. Set $k=0$.\r\n\t\t\\item[1.] Apply Algorithm~\\ref{alg:opt-fom-str} (in Appendix \\ref{app-fom}) to the problem $\\min_x \\varphi_k(x)$ to  find $x^{k+1}\\in\\dom(P)$ satisfying\r\n\t\t\\begin{equation}\r\n\t\t\\label{eq:inext-subprob-m}\r\n\t\t\\dist (0, \\partial \\varphi_k(x^{k+1})) \\leq \\eta_k,\\footnote{In view of Proposition \\ref{prop:cplx-fom-str-cvx},  one can terminate Algorithm~\\ref{alg:opt-fom-str} once a point $x^{k+1}$ satisfying $2L_{\\nabla \\phi}\\|\\tilde{x}^{k+1} - x^{k+1}\\| \\le \\eta_k$ is found, where $\\phi(\\cdot)= \\varphi_k(\\cdot)-P(\\cdot)$, $\\tilde{x}^{k+1}= \\prox_{P/L_{\\nabla \\phi}}\\left( x^{k+1} - \\nabla \\phi(x^{k+1})/L_{\\nabla \\phi} \\right)$, and $L_{\\nabla \\phi}$ is the Lipschitz constant of $\\nabla \\phi$. Such $x^{k+1}$ clearly satisfies \\eqref{eq:inext-subprob-m}.}\r\n\t\t\\end{equation}\r\n\t\twhere \r\n\t\t\\begin{equation}\r\n\t\t\\label{eq:prox-al-func}\r\n\t\t\\varphi_k(x) = \\mcL(x,\\lambda^k;\\rho_k) + \\frac{1}{2\\rho_k}\\|x - x^k\\|^2.\r\n\t\t\\end{equation}\r\n\t\t\\item[2.] Set $\\lambda^{k+1} = \\Pi_{\\mcK^*}\\big(\\lambda^k + \\rho_k g(x^{k+1}) \\big)$. \r\n\t\t\\item[3.] If $(x^{k+1},\\lambda^{k+1})$ satisfies \\eqref{eq:eps-approx-kkt-point} or the following two inequalities are satisfied\r\n\t\t\\begin{equation}\r\n\t\t\\label{eq:check-stop-m}\r\n\t\t\\frac{1}{\\rho_k}\\|(x^{k+1}, \\lambda^{k+1}) - (x^k,\\lambda^k)\\| \\leq \\frac{\\epsilon}{2}, \\quad \\eta_k \\leq \\frac{\\epsilon}{2},\r\n\t\t\\end{equation}\r\n\t\toutput $(x^+, \\lambda^{+}) = (x^{k+1}, \\lambda^{k+1})$ and terminate the algorithm.\r\n\t\t\\item[4.] Set $k \\leftarrow k+1$ and go to Step 1.\r\n\t\\end{itemize}\r\n\t{\\bf End.}\r\n\\end{algo}","sha256_utf8":"a48dce36a5c08e0cac594ebe7b843bd465b8f88595aec73712c08db2473ebaf6"},{"start":45423,"end":45977,"start_line":561,"end_line":564,"raw_tex":"For ease of later reference, we refer to the iterations of Algorithm~\\ref{alg:opt-fom-str} for solving the \r\nAL subproblems as the {\\it inner iterations} of Algorithm \\ref{alg:mi-al}, and call the update from \r\n$(x^k,\\lambda^k)$ to $(x^{k+1},\\lambda^{k+1})$ an {\\it outer iteration} of Algorithm \\ref{alg:mi-al}. \r\nNotice from \\eqref{eq:prox-al-func} that $\\varphi_k$ is strongly convex with modulus $1/\\rho_k$. The AL subproblem $\\min_x \\varphi_k(x)$ arising in Algorithm \\ref{alg:mi-al} can thus be suitably solved by Algorithm~\\ref{alg:opt-fom-str}.\r\n","sha256_utf8":"6157a1f0b94efb9f62a5016463c414cc9745b0760601679ab1d1a62468159dba"},{"start":45979,"end":46564,"start_line":566,"end_line":570,"raw_tex":"In the rest of this section, we present our main results for Algorithm \\ref{alg:mi-al}, whose proof is deferred  to Subsection \\ref{sec:pf-mi-al}. Before proceeding, we introduce some further notation that will be used subsequently. Let $X^*$ be the set of optimal solutions of problem \\eqref{eq:conic-cp} and $\\hat x^*\\in X^*$ such that \r\n$\\|x^0-\\hat x^*\\|=\\dist(x^0,X^*)$. In addition, we define \r\n\\beq \\label{param-m} \r\n{\\bar D}_X =\\max\\{D_X,1\\}, \\ D=  \\dist(x^0,X^*)+D_\\Lambda, \\ \\bar{D}= \\max\\{D,1\\}, \\ \\hat B = L_{\\nabla f} + L_{\\nabla g}\\|\\hat\\lambda^*\\| + L_{\\nabla g}D, \r\n\\eeq","sha256_utf8":"2111904263f3d09d422eddf6ffd8c1bce1930871e12bafb895b3e0d5f7ef9c8a"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{thm}\r\n\t\\label{prop:output-gantee-m}\r\n\t\\bi\r\n\t\\item[(i)]\r\n\tIf  Algorithm~\\ref{alg:mi-al} successfully terminates, then the output $(x^+,\\lambda^+)$  is an $\\epsilon$-KKT solution of problem \\eqref{eq:conic-cp}.\r\n\t\\item[(ii)]\r\n\tSuppose that $\\{\\rho_k\\}$  and $\\{\\eta_k\\}$ satisfy that\r\n\t\\beq \\label{rho-eta-m}\r\n\t\\rho_k>0 \\ \\mbox{is nondecreasing}, \\quad 0<\\eta_k \\downarrow 0, \\quad \r\n\t\\frac{\\sum_{i=0}^{2k}\\rho_i\\eta_i}{\\rho_k\\sqrt{k+1}} \\to 0.  \\footnote{For example,  $\\rho_k = \\rho_0\\alpha^k$ and $\\eta_k = \\eta_0\\beta^k$ satisfy \\eqref{rho-eta-m} for any $\\rho_0>0$, $\\eta_0>0$, $\\alpha>1$ and $0<\\beta<1/\\alpha$.}\r\n\t\\eeq \r\n\tThen Algorithm~\\ref{alg:mi-al} terminates in a finite number of outer iterations. \r\n\t\\item[(iii)] Furthermore, if $N$ is a nonnegative integer such that\r\n\t\\begin{equation}\r\n\t\\label{eq:cplx-out-general-m}\r\n\t\\frac{D + \\sum_{k=0}^N \\rho_k\\eta_k}{\\rho_N} \\leq \\frac{\\epsilon}{2}, \\quad \\eta_N \\leq \\frac{\\epsilon}{2},\r\n\t\\end{equation}\r\n\tthen Algorithm~\\ref{alg:mi-al} terminates in at most $N+1$ outer iterations.\r\n\t\\ei\r\n\\end{thm}","units":[{"start":47005,"end":48065,"start_line":574,"end_line":593,"raw_tex":"\\begin{thm}\r\n\t\\label{prop:output-gantee-m}\r\n\t\\bi\r\n\t\\item[(i)]\r\n\tIf  Algorithm~\\ref{alg:mi-al} successfully terminates, then the output $(x^+,\\lambda^+)$  is an $\\epsilon$-KKT solution of problem \\eqref{eq:conic-cp}.\r\n\t\\item[(ii)]\r\n\tSuppose that $\\{\\rho_k\\}$  and $\\{\\eta_k\\}$ satisfy that\r\n\t\\beq \\label{rho-eta-m}\r\n\t\\rho_k>0 \\ \\mbox{is nondecreasing}, \\quad 0<\\eta_k \\downarrow 0, \\quad \r\n\t\\frac{\\sum_{i=0}^{2k}\\rho_i\\eta_i}{\\rho_k\\sqrt{k+1}} \\to 0.  \\footnote{For example,  $\\rho_k = \\rho_0\\alpha^k$ and $\\eta_k = \\eta_0\\beta^k$ satisfy \\eqref{rho-eta-m} for any $\\rho_0>0$, $\\eta_0>0$, $\\alpha>1$ and $0<\\beta<1/\\alpha$.}\r\n\t\\eeq \r\n\tThen Algorithm~\\ref{alg:mi-al} terminates in a finite number of outer iterations. \r\n\t\\item[(iii)] Furthermore, if $N$ is a nonnegative integer such that\r\n\t\\begin{equation}\r\n\t\\label{eq:cplx-out-general-m}\r\n\t\\frac{D + \\sum_{k=0}^N \\rho_k\\eta_k}{\\rho_N} \\leq \\frac{\\epsilon}{2}, \\quad \\eta_N \\leq \\frac{\\epsilon}{2},\r\n\t\\end{equation}\r\n\tthen Algorithm~\\ref{alg:mi-al} terminates in at most $N+1$ outer iterations.\r\n\t\\ei\r\n\\end{thm}","sha256_utf8":"245c2e110296ad427b2c61925ba5b8071067abb67fbab9d9d5d79701dac75cfc"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{prop}\r\nUnder Assumption~\\ref{ass:function}, $x^*\\in\\Re^n$ is an optimal solution of~\\eqref{eq:conic-cp} if and only if there exists $\\lambda^*\\in\\Re^m$ such that\r\n\\begin{equation}\r\n\\label{eq:KKT-condition}\r\n(0,0)\\in \\partial l(x^*,\\lambda^*),\r\n\\end{equation}\r\nor equivalently, $(x^*,\\lambda^*)$ satisfies the KKT conditions for~\\eqref{eq:conic-cp}, that is,\r\n\\[\r\n0\\in\\nabla f(x^*) + \\partial P(x^*) + \\nabla g(x^*) \\lambda^*, \\quad \\lambda^* \\in\\mcK^*, \\quad g(x^*)\\preceq_\\mcK 0, \\quad \\langle \\lambda^*, g(x^*) \\rangle = 0.\r\n\\]\r\n\\end{prop}\n\n\\begin{proof}\r\nThe result \\eqref{eq:KKT-condition} follows from \\cite[Theorem 36.6]{R97}. By \\eqref{eq:express-Tl}, it is not hard to see that \\eqref{eq:KKT-condition} holds if and only if $0\\in \\nabla f(x^*) + \\partial P(x^*) + \\nabla g(x^*)\\lambda^*$, $\\lambda^*\\in\\mcK^*$, and $g(x^*) \\in\\mcN_{\\mcK^*}(\\lambda^*)$. By the definition of $\\mcK^*$ and $\\mcN_{\\mcK^*}$, one can verify that $g(x^*)\\in\\mcN_{\\mcK^*}(\\lambda^*)$ is equivalent to $g(x^*)\\preceq_\\mcK 0$ and $\\langle\\lambda^*, g(x^*) \\rangle = 0$. The proof is then completed.\r\n\\end{proof}\n\n\\begin{lemma}\r\n\t\\label{lemma:al-func-elp}\r\n\tFor any $(x,\\lambda,\\rho)\\in\\Re^n\\times\\Re^m\\times\\Re_{++}$, the following identity holds\r\n\t\\begin{equation*}\r\n\t\\mcL(x,\\lambda;\\rho) = \\max_{\\eta\\in\\Re^m} \\left\\{ l(x,\\eta) - \\frac{1}{2\\rho}\\|\\eta - \\lambda\\|^2\\right\\}.\r\n\t\\end{equation*}\r\n\tIn addition, if $x\\in\\dom(P)$, the maximum is attained uniquely at $\\bar{\\lambda} = \\Pi_{\\mcK^*}(\\lambda + \\rho g(x))$. Consequently, the following statements hold.\r\n\t\\begin{itemize}\r\n\t\t\\item[(i)] For any $(\\lambda,\\rho)\\in\\Re^m\\times\\Re_{++}$, $d(\\lambda;\\rho)$ satisfies that\r\n\t\t\\begin{equation}\r\n\t\t\\label{eq:al-dual-func-elp-cp}\r\n\t\td(\\lambda;\\rho) = \\max_{\\eta\\in\\Re^m}\\left\\{ d(\\eta) - \\frac{1}{2\\rho}\\|\\eta - \\lambda\\|^2\\right\\}. \r\n\t\t\\end{equation}\r\n\t\t\\item[(ii)] $\\mcL(x,\\lambda;\\rho)$ is a convex function in $x$, and for any $x\\in\\dom(P)$, we have\r\n\\[\r\n\t\t\\partial_x\\mcL(x,\\lambda;\\rho) = \\partial_x l(x,\\bar{\\lambda}).\r\n\\]\r\n\t\t\\item[(iii)] $\\mcL(x,\\lambda;\\rho)$ is a concave function in $\\lambda$, and for any $x\\in\\dom(P)$, it is differentiable in $\\lambda$ and\r\n\\[\r\n\t\t\\frac{1}{\\rho}(\\bar{\\lambda} - \\lambda) = \\nabla_\\lambda \\mcL(x,\\lambda;\\rho) \\in \\partial_\\lambda l(x,\\bar{\\lambda}).\r\n\\]\r\n\t\\end{itemize}\r\n\\end{lemma}\n\n\\begin{lemma}\r\n\\label{lemma:cplx-ippa} \r\nLet $\\mcT:\\Re^n\\rightrightarrows\\Re^n$ be a maximally monotone operator and  $z^*\\in\\Re^n$  such that $0\\in\\mcT(z^*)$. Let $\\{z^k\\}$ be a sequence generated by an inexact PPA, starting with some $z^0$ and obtaining $z^{k+1}$ by approximately evaluating $\\mcJ_{\\rho_k}(z^k)$ such that \r\n\\begin{equation}\r\n\\label{eq:iter-PPA}\r\n\\|z^{k+1} - \\mcJ_{\\rho_k}(z^k)\\| \\le  e_k\r\n\\end{equation}\r\nfor some $\\rho_k>0$ and $e_k\\geq 0$, where $\\mcJ_\\rho=(\\mcI + \\rho\\mcT)^{-1}$  and $\\mcI$ is the identify operator. Then we have\r\n\\beqa\r\n\\|z^s - z^{*}\\| \\leq \\|z^t - z^{*}\\| + \\sum_{i=t}^{s-1}e_i, \\quad \\forall s\\geq t\\geq 0, \\label{eq:iter-ppa-bd} \\\\\r\n\\|z^{k+1} - z^k\\| \\leq \\|z^0 - z^*\\| + \\sum_{i=0}^k e_i, \\quad \\forall k\\geq 0. \\label{eq:ippa-iter-cplx}\r\n\\eeqa\r\nMoreover, for any $K\\ge 1$, we have\r\n\\begin{equation}\r\n\\label{eq:ippa-iter-cplx-min}\r\n\\min_{K\\leq k\\leq 2K}\\|z^{k+1} - z^k\\| \\leq \\frac{\\sqrt{2}\\left(\\|z^0 - z^{*}\\| + 2\\sum_{k=0}^{2K}e_k\\right)}{\\sqrt{K+1}}.\r\n\\end{equation}\r\n\\end{lemma}\n\n\\begin{lemma}\r\n\\label{lemma:mi-al=i-ppa}\r\nLet $\\{(x^k,\\lambda^k)\\}$ be generated by Algorithm~\\ref{alg:mi-al}. For any $k\\geq 0$, one has\r\n\\begin{equation}\r\n\\label{eq:mi-al=i-ppa}\r\n\\|(x^{k+1},\\lambda^{k+1}) - \\mcJ_{\\rho_k}(x^k,\\lambda^k)\\| \\leq \\rho_k\\eta_k,\r\n\\end{equation}\r\nwhere $\\mcJ_{\\rho_k}=(\\mcI + \\rho_k\\mcT_l)^{-1}$ and $\\mcT_l$ is defined in \\eqref{eq:def-two-opt2}.\r\n\\end{lemma}\n\n\\begin{proof}\r\nBy Lemma~\\ref{lemma:al-func-elp} and $\\lambda^{k+1} = \\Pi_{\\mcK^*}\\big(\\lambda^k + \\rho_k g(x^{k+1})\\big)$, one has\r\n\\begin{equation}\r\n\\label{eq:diff-x-lambda-m}\r\n\\partial_x \\mcL(x^{k+1},\\lambda^k;\\rho_k) = \\partial_x l(x^{k+1},\\lambda^{k+1}), \\quad \\frac{1}{\\rho_k}(\\lambda^{k+1} - \\lambda^k) \\in \\partial_\\lambda l (x^{k+1},\\lambda^{k+1}).\r\n\\end{equation}\r\nBy~\\eqref{eq:inext-subprob-m}, there exists $\\|v\\|\\leq \\eta_k$ such that\r\n$$ v \\in \\partial_x \\mcL(x^{k+1},\\lambda^k;\\rho_k) + \\frac{1}{\\rho_k}(x^{k+1} - x^k). $$\r\nThis together with~\\eqref{eq:diff-x-lambda-m} implies that\r\n\\begin{equation}\r\n\\label{eq:diff-x-lambda-m-v}\r\nx^k + \\rho_kv \\in \\rho_k \\partial_x l(x^{k+1},\\lambda^{k+1}) +  x^{k+1}, \\quad \\lambda^k \\in -\\rho_k\\partial_\\lambda l(x^{k+1}, \\lambda^{k+1}) + \\lambda^{k+1},\r\n\\end{equation}\r\nwhich, by the definition of $\\mcT_l$, are equivalent to \r\n$$ (x^k + \\rho_kv, \\lambda^k) \\in (\\mcI + \\rho_k\\mcT_l)(x^{k+1},\\lambda^{k+1}). $$\r\nIt follows from this and $\\mcJ_{\\rho_k}=(\\mcI + \\rho_k\\mcT_l)^{-1}$ that  $(x^{k+1},\\lambda^{k+1}) = \\mcJ_{\\rho_k}(x^k+\\rho_kv,\\lambda^k)$. By this and the non-expansion of $\\mcJ_{\\rho_k}$, we obtain\r\n$$ \\|(x^{k+1}, \\lambda^{k+1}) - \\mcJ_{\\rho_k}(x^k,\\lambda^k)\\| = \\|\\mcJ_{\\rho_k}(x^k + \\rho_kv, \\lambda^k) - \\mcJ_{\\rho_k}(x^k,\\lambda^k)\\| \\leq \\|\\rho_kv\\| \\leq \\rho_k\\eta_k, $$\r\nwhich yields \\eqref{eq:mi-al=i-ppa} as desired. \r\n\\end{proof}\n\n\\begin{proof}[Proof of Theorem \\ref{prop:output-gantee-m}]\r\n(i) Suppose that Algorithm~\\ref{alg:mi-al} terminates in Step 3 at some iteration $k$. It then follows \r\nthat $(x^+,\\lambda^+)$  is already an $\\epsilon$-KKT solution of problem \\eqref{eq:conic-cp} \r\nor  the inequalities \\eqref{eq:check-stop-m} hold for such $k$.  We next show that for the latter case, $(x^+, \\lambda^+) = (x^{k+1},\\lambda^{k+1})$ is also an $\\epsilon$-KKT solution of~\\eqref{eq:conic-cp}. To this end, suppose that \\eqref{eq:check-stop-m} holds at the $k$th iteration. Notice that \\eqref{eq:diff-x-lambda-m-v} holds for some $\\|v\\|\\leq \\eta_k$, which yields  \r\n\\[\r\n\\frac{1}{\\rho_k}(x^k-x^{k+1}) + v \\in \\partial_x l(x^{k+1},\\lambda^{k+1}), \\quad \\frac{1}{\\rho_k}(\\lambda^{k+1} - \\lambda^k) \\in \\partial_\\lambda l (x^{k+1},\\lambda^{k+1}).\r\n\\]\r\nBy these, \\eqref{eq:check-stop-m} and $(x^+, \\lambda^+) = (x^{k+1},\\lambda^{k+1})$, we obtain\r\n\\begin{align*}\r\n\\mbox{dist}(0,\\partial_x l(x^+, \\lambda^+)) & \\leq \\frac{1}{\\rho_k}\\|x^{k+1} - x^k - \\rho_kv\\| \\leq \\frac{1}{\\rho_k}\\|x^{k+1} - x^k\\| + \\|v\\| \\leq \\epsilon,\\\\\r\n\\mbox{dist}(0,\\partial_\\lambda l(x^+, \\lambda^+)) & \\leq \\frac{1}{\\rho_k}\\|\\lambda^{k+1} - \\lambda^k\\| \\leq \\epsilon,\r\n\\end{align*}\r\nwhich along with Definition \\ref{defi:approx-KKT} imply that $(x^+,\\lambda^+)$ is an $\\epsilon$-KKT solution of problem \\eqref{eq:conic-cp}.\r\n\r\n(ii) Suppose for contradiction that Algorithm~\\ref{alg:mi-al} does not terminate in a finite number of iterations. It then follows that  \\eqref{eq:check-stop-m} does not hold for any $k$. By \\eqref{eq:ippa-iter-cplx-min} and Lemma \\ref{lemma:mi-al=i-ppa}, one has that\r\n$$ \\min_{k \\leq i\\leq 2k} \\|(x^{i+1},\\lambda^{i+1}) - (x^i,\\lambda^i)\\| \\leq \\frac{\\sqrt{2}\\left(D + 2\\sum_{i=0}^{2k}\\rho_i\\eta_i\\right)}{\\sqrt{k+1}}, $$ \r\nwhere $D$ is defined in \\eqref{param-m}. Since $\\{\\rho_k\\}$ is assumed to be nondecreasing, we further have\r\n\\[\r\n\\min_{k \\leq i\\leq 2k} \\frac{1}{\\rho_i}\\|(x^{i+1},\\lambda^{i+1}) - (x^i,\\lambda^i)\\| \\leq \\frac{\\sqrt{2}\\left(D + 2\\sum_{i=0}^{2k}\\rho_i\\eta_i\\right)}{\\rho_k\\sqrt{k+1}}.\r\n\\]\r\nBy this and \\eqref{rho-eta-m}, one has that $\\min_{k \\leq i\\leq 2k} \\|(x^{i+1},\\lambda^{i+1}) - (x^i,\\lambda^i)\\|/\\rho_i \\to 0$ and $\\eta_k\\to 0$ as $k \\to \\infty$, which implies that \\eqref{eq:check-stop-m} is satisfied for some $k$ and thus leads to a contradiction. \r\n\r\n(iii) Let $\\hat x^*$ and $\\hat\\lambda^*$ be defined right above \\eqref{param} and \\eqref{param-m}, respectively. Suppose for contradiction that Algorithm~\\ref{alg:mi-al} does not terminate within $N+1$ outer iterations. It then follows that \\eqref{eq:check-stop-m} does not hold for all $0\\le k \\le N$. Then, by \\eqref{param-m}, \\eqref{eq:ippa-iter-cplx}, and \\eqref{eq:mi-al=i-ppa}, we have that for all $0\\le k \\le N$,\r\n\\begin{equation*}\\label{eq:mi-ppa-bd}\r\n\\|(x^{k+1},\\lambda^{k+1}) - (x^k,\\lambda^k)\\| \\leq \\|(x^0,\\lambda^0) - (\\hat x^*, \\hat\\lambda^*)\\| + \\sum_{i=0}^k\\rho_i\\eta_i \\leq D + \\sum_{i=0}^k\\rho_i\\eta_i,\r\n\\end{equation*}\r\nwhere $D$ is given in \\eqref{param-m}. \r\nBy this and \\eqref{eq:cplx-out-general-m}, one can see that \\eqref{eq:check-stop-m} is satisfied when $k = N$, which leads to a contradiction.  \r\n\\end{proof}\n\n\\begin{proof}\r\nSince $0\\in\\mcT(z^*)$ and $\\mcT$ is maximally monotone, it follows from \\cite[Proposition 1]{Rock76a} that\r\n\\begin{equation}\r\n\\label{eq:suff-dec-ppa-1}\r\n\\| \\mcJ_\\rho(z) - z^*\\|^2 + \\|\\mcJ_\\rho(z) - z\\|^2 \\leq \\|z - z^*\\|^2,\r\n\\end{equation}\r\nwhich implies that \r\n\\begin{equation}\r\n\\label{eq:suff-dec-ppa-2}\r\n\\|\\mcJ_\\rho(z) - z^*\\| \\leq \\|z - z^*\\|, \\quad \\|\\mcJ_\\rho(z) - z\\| \\leq \\|z - z^*\\|, \\quad \\forall z \\in \\Re^n.\r\n\\end{equation}\r\n\tLet $\\xi^k=z^{k+1} - \\mcJ_{\\rho_k}(z^k)$ for all $k\\ge 0$. By this, \\eqref{eq:suff-dec-ppa-2}, and \\eqref{eq:iter-PPA} with $\\rho = \\rho_k$ and $z = z^k$, one has\r\n\t\\[\r\n\t\\|z^{k+1} - z^{*}\\| \\leq \\|z^{k+1} - \\mcJ_{\\rho_k}(z^k)\\| + \\|\\mcJ_{\\rho_k}(z^k) - z^{*}\\| \r\n\t\\leq \\|\\xi^k\\| + \\|z^k - z^{*}\\|, \\quad \\forall k \\ge 0.\r\n\t\\]\r\n\tSumming up the above inequality from $k=t$ to $k=s-1$ yields \r\n\t\\beq \\label{bdd-1}\r\n\t\\|z^s - z^{*}\\| \\leq \\|z^t - z^{*}\\| + \\sum_{i=t}^{s-1} \\|\\xi^i\\|, \\quad \\forall s\\geq t\\geq 0.\r\n\t\\eeq\r\n\tNotice from \\eqref{eq:iter-PPA} that $\\|\\xi^k\\| \\le e_k$ for all $k\\ge 0$, which along with \\eqref{bdd-1} leads to \\eqref{eq:iter-ppa-bd}. Besides, by~\\eqref{eq:suff-dec-ppa-2} with $\\rho=\\rho_k$ and $z = z^k$, one has $ \\|\\mcJ_{\\rho_k}(z^k) - z^k\\|  \\leq \\|z^k - z^*\\|$. These together with \\eqref{eq:iter-PPA} and \\eqref{eq:iter-ppa-bd} yield\r\n\t$$ \\|z^{k+1} - z^k\\| \\leq \\|z^{k+1} - \\mcJ_{\\rho_k}(z^k)\\| + \\|\\mcJ_{\\rho_k}(z^k) - z^k\\| \\leq e_k + \\|z^k - z^*\\| \\leq \\|z^0 - z^*\\| + \\sum_{i=0}^k e_i.  $$\r\n\tIn addition, by the definition of $\\xi^k$, and \\eqref{eq:suff-dec-ppa-1} \r\n\twith $\\mcJ = \\mcJ_{\\rho_k}$ and $z = z^k$, one has\r\n\t\\begin{equation*}\r\n\t\\begin{split}\r\n\t\\|\\mcJ_{\\rho_k}(z^k) - z^k\\|^2 & \\leq \\|z^k - z^{*}\\|^2 - \\|\\mcJ_{\\rho_k}(z^k) - z^{*}\\|^2 \r\n\t= \\|z^k - z^{*}\\|^2 - \\|\\mcJ_{\\rho_k}(z^k) - z^{k+1} + z^{k+1} - z^{*}\\|^2 \\\\\r\n\t& \\leq \\|z^k - z^{*}\\|^2 - \\|z^{k+1} - z^{*}\\|^2 - \\|\\xi^k\\|^2 + 2\\|\\xi^k\\|\\|z^{k+1} - z^{*}\\|.\r\n\t\\end{split}\r\n\t\\end{equation*}\r\n\tSumming up the above inequality from $k = K$ to $k = 2K$ and using \\eqref{bdd-1}, we obtain that \r\n\t\\begin{align}\r\n\t\\sum_{k=K}^{2K}\\|\\mcJ_{\\rho_k}(z^k) - z^k\\|^2 & \\leq \\|z^K - z^{*}\\|^2 - \\sum_{k=K}^{2K}\\|\\xi^k\\|^2 + 2\\sum_{k=K}^{2K}\\|\\xi^k\\|\\left(\\|z^K - z^{*}\\| + \\sum_{j=K}^k\\|\\xi^j\\|\\right) \\nn\\\\\r\n\t& = \\|z^K - z^{*}\\|^2 - \\sum_{k=K}^{2K}\\|\\xi^k\\|^2 + 2\\|z^K - z^{*}\\|\\cdot\\sum_{k=K}^{2K}\\|\\xi^k\\| + 2\\sum_{k=K}^{2K}\\sum_{j=K}^k\\|\\xi^k\\|\\|\\xi^j\\| \\nn\\\\\r\n\t& = \\|z^K - z^{*}\\|^2 - \\sum_{k=K}^{2K}\\|\\xi^k\\|^2 + 2\\|z^K - z^{*}\\|\\cdot\\sum_{k=K}^{2K}\\|\\xi^k\\| + \\sum_{k=K}^{2K}\\|\\xi^k\\|^2 + \\left(\\sum_{k=K}^{2K}\\|\\xi^k\\|\\right)^2 \\nn\\\\\r\n\t& = \\|z^K - z^{*}\\|^2 + 2\\|z^K - z^{*}\\|\\cdot\\sum_{k=K}^{2K}\\|\\xi^k\\| + \\left(\\sum_{k=K}^{2K}\\|\\xi^k\\|\\right)^2 \\nn\\\\\r\n\t& = \\left( \\|z^K - z^{*}\\| + \\sum_{k=K}^{2K}\\|\\xi^k\\| \\right)^2 \\leq \\left( \\|z^0 - z^{*}\\| + \\sum_{k=0}^{2K}\\|\\xi^k\\| \\right)^2, \\label{eq:bound-sum-jz-z}\r\n\t\\end{align}\r\n\twhere \\eqref{eq:bound-sum-jz-z} follows from \\eqref{bdd-1} with $t=0$ and $s = K$. Again, by the definition of $\\xi^k$, one has\r\n\t$$ \\|z^{k+1} - z^k\\|^2 = \\|\\mcJ_{\\rho_k}(z^k) + \\xi^k- z^k \\|^2 \\leq 2\\left(\\|\\mcJ_{\\rho_k}(z^k) - z^k\\|^2 + \\|\\xi^k\\|^2\\right). $$\r\n\tThis together with~\\eqref{eq:bound-sum-jz-z} yields\r\n\t\\begin{align*}\r\n\t\\sum_{k = K}^{2K}\\|z^{k+1} - z^k\\|^2 & \\leq 2\\sum_{k = K}^{2K}\\|\\mcJ_{\\rho_k}(z^k) - z^k\\|^2 + 2\\sum_{k = K}^{2K}\\|\\xi^k\\|^2 \r\n\t \\leq 2\\left( \\|z^0 - z^{*}\\| + \\sum_{k=0}^{2K}\\|\\xi^k\\| \\right)^2 + 2\\sum_{k=0}^{2K}\\|\\xi^k\\|^2 \\\\\r\n\t& \\leq 2\\left( \\|z^0 - z^{*}\\| + 2\\sum_{k=0}^{2K}\\|\\xi^k\\| \\right)^2,\r\n\t\\end{align*}\r\n\twhich along with $\\|\\xi^k\\| \\le e_k$ leads to \\eqref{eq:ippa-iter-cplx-min}. The proof is then completed.\r\n\\end{proof}","units":[{"start":27269,"end":27817,"start_line":349,"end_line":359,"raw_tex":"\\begin{prop}\r\nUnder Assumption~\\ref{ass:function}, $x^*\\in\\Re^n$ is an optimal solution of~\\eqref{eq:conic-cp} if and only if there exists $\\lambda^*\\in\\Re^m$ such that\r\n\\begin{equation}\r\n\\label{eq:KKT-condition}\r\n(0,0)\\in \\partial l(x^*,\\lambda^*),\r\n\\end{equation}\r\nor equivalently, $(x^*,\\lambda^*)$ satisfies the KKT conditions for~\\eqref{eq:conic-cp}, that is,\r\n\\[\r\n0\\in\\nabla f(x^*) + \\partial P(x^*) + \\nabla g(x^*) \\lambda^*, \\quad \\lambda^* \\in\\mcK^*, \\quad g(x^*)\\preceq_\\mcK 0, \\quad \\langle \\lambda^*, g(x^*) \\rangle = 0.\r\n\\]\r\n\\end{prop}","sha256_utf8":"570b31493f92c425b62c6f21cf428cf576cfd38bda54978148ca444d5c8991b1"},{"start":27819,"end":28369,"start_line":360,"end_line":362,"raw_tex":"\\begin{proof}\r\nThe result \\eqref{eq:KKT-condition} follows from \\cite[Theorem 36.6]{R97}. By \\eqref{eq:express-Tl}, it is not hard to see that \\eqref{eq:KKT-condition} holds if and only if $0\\in \\nabla f(x^*) + \\partial P(x^*) + \\nabla g(x^*)\\lambda^*$, $\\lambda^*\\in\\mcK^*$, and $g(x^*) \\in\\mcN_{\\mcK^*}(\\lambda^*)$. By the definition of $\\mcK^*$ and $\\mcN_{\\mcK^*}$, one can verify that $g(x^*)\\in\\mcN_{\\mcK^*}(\\lambda^*)$ is equivalent to $g(x^*)\\preceq_\\mcK 0$ and $\\langle\\lambda^*, g(x^*) \\rangle = 0$. The proof is then completed.\r\n\\end{proof}","sha256_utf8":"9753c183a3856ca34d5bbda60e53973207c9e803c929f087e63cc4f4965b61ee"},{"start":52000,"end":53213,"start_line":647,"end_line":669,"raw_tex":"\\begin{lemma}\r\n\t\\label{lemma:al-func-elp}\r\n\tFor any $(x,\\lambda,\\rho)\\in\\Re^n\\times\\Re^m\\times\\Re_{++}$, the following identity holds\r\n\t\\begin{equation*}\r\n\t\\mcL(x,\\lambda;\\rho) = \\max_{\\eta\\in\\Re^m} \\left\\{ l(x,\\eta) - \\frac{1}{2\\rho}\\|\\eta - \\lambda\\|^2\\right\\}.\r\n\t\\end{equation*}\r\n\tIn addition, if $x\\in\\dom(P)$, the maximum is attained uniquely at $\\bar{\\lambda} = \\Pi_{\\mcK^*}(\\lambda + \\rho g(x))$. Consequently, the following statements hold.\r\n\t\\begin{itemize}\r\n\t\t\\item[(i)] For any $(\\lambda,\\rho)\\in\\Re^m\\times\\Re_{++}$, $d(\\lambda;\\rho)$ satisfies that\r\n\t\t\\begin{equation}\r\n\t\t\\label{eq:al-dual-func-elp-cp}\r\n\t\td(\\lambda;\\rho) = \\max_{\\eta\\in\\Re^m}\\left\\{ d(\\eta) - \\frac{1}{2\\rho}\\|\\eta - \\lambda\\|^2\\right\\}. \r\n\t\t\\end{equation}\r\n\t\t\\item[(ii)] $\\mcL(x,\\lambda;\\rho)$ is a convex function in $x$, and for any $x\\in\\dom(P)$, we have\r\n\\[\r\n\t\t\\partial_x\\mcL(x,\\lambda;\\rho) = \\partial_x l(x,\\bar{\\lambda}).\r\n\\]\r\n\t\t\\item[(iii)] $\\mcL(x,\\lambda;\\rho)$ is a concave function in $\\lambda$, and for any $x\\in\\dom(P)$, it is differentiable in $\\lambda$ and\r\n\\[\r\n\t\t\\frac{1}{\\rho}(\\bar{\\lambda} - \\lambda) = \\nabla_\\lambda \\mcL(x,\\lambda;\\rho) \\in \\partial_\\lambda l(x,\\bar{\\lambda}).\r\n\\]\r\n\t\\end{itemize}\r\n\\end{lemma}","sha256_utf8":"1a78b4767d75ce1db78b3e449b33c595d127f283f3c854589badfe7c2c6b40c9"},{"start":53347,"end":54376,"start_line":672,"end_line":689,"raw_tex":"\\begin{lemma}\r\n\\label{lemma:cplx-ippa} \r\nLet $\\mcT:\\Re^n\\rightrightarrows\\Re^n$ be a maximally monotone operator and  $z^*\\in\\Re^n$  such that $0\\in\\mcT(z^*)$. Let $\\{z^k\\}$ be a sequence generated by an inexact PPA, starting with some $z^0$ and obtaining $z^{k+1}$ by approximately evaluating $\\mcJ_{\\rho_k}(z^k)$ such that \r\n\\begin{equation}\r\n\\label{eq:iter-PPA}\r\n\\|z^{k+1} - \\mcJ_{\\rho_k}(z^k)\\| \\le  e_k\r\n\\end{equation}\r\nfor some $\\rho_k>0$ and $e_k\\geq 0$, where $\\mcJ_\\rho=(\\mcI + \\rho\\mcT)^{-1}$  and $\\mcI$ is the identify operator. Then we have\r\n\\beqa\r\n\\|z^s - z^{*}\\| \\leq \\|z^t - z^{*}\\| + \\sum_{i=t}^{s-1}e_i, \\quad \\forall s\\geq t\\geq 0, \\label{eq:iter-ppa-bd} \\\\\r\n\\|z^{k+1} - z^k\\| \\leq \\|z^0 - z^*\\| + \\sum_{i=0}^k e_i, \\quad \\forall k\\geq 0. \\label{eq:ippa-iter-cplx}\r\n\\eeqa\r\nMoreover, for any $K\\ge 1$, we have\r\n\\begin{equation}\r\n\\label{eq:ippa-iter-cplx-min}\r\n\\min_{K\\leq k\\leq 2K}\\|z^{k+1} - z^k\\| \\leq \\frac{\\sqrt{2}\\left(\\|z^0 - z^{*}\\| + 2\\sum_{k=0}^{2K}e_k\\right)}{\\sqrt{K+1}}.\r\n\\end{equation}\r\n\\end{lemma}","sha256_utf8":"0e103ca44353e2d79fead744f5bd23e5f60b9614405d3cd3855669800e685e52"},{"start":58478,"end":58867,"start_line":742,"end_line":750,"raw_tex":"\\begin{lemma}\r\n\\label{lemma:mi-al=i-ppa}\r\nLet $\\{(x^k,\\lambda^k)\\}$ be generated by Algorithm~\\ref{alg:mi-al}. For any $k\\geq 0$, one has\r\n\\begin{equation}\r\n\\label{eq:mi-al=i-ppa}\r\n\\|(x^{k+1},\\lambda^{k+1}) - \\mcJ_{\\rho_k}(x^k,\\lambda^k)\\| \\leq \\rho_k\\eta_k,\r\n\\end{equation}\r\nwhere $\\mcJ_{\\rho_k}=(\\mcI + \\rho_k\\mcT_l)^{-1}$ and $\\mcT_l$ is defined in \\eqref{eq:def-two-opt2}.\r\n\\end{lemma}","sha256_utf8":"8f23a52cb2c1568c7deaf987a0c82d268a3430b46bfae5a1aeabf65461b75570"},{"start":58871,"end":60278,"start_line":752,"end_line":770,"raw_tex":"\\begin{proof}\r\nBy Lemma~\\ref{lemma:al-func-elp} and $\\lambda^{k+1} = \\Pi_{\\mcK^*}\\big(\\lambda^k + \\rho_k g(x^{k+1})\\big)$, one has\r\n\\begin{equation}\r\n\\label{eq:diff-x-lambda-m}\r\n\\partial_x \\mcL(x^{k+1},\\lambda^k;\\rho_k) = \\partial_x l(x^{k+1},\\lambda^{k+1}), \\quad \\frac{1}{\\rho_k}(\\lambda^{k+1} - \\lambda^k) \\in \\partial_\\lambda l (x^{k+1},\\lambda^{k+1}).\r\n\\end{equation}\r\nBy~\\eqref{eq:inext-subprob-m}, there exists $\\|v\\|\\leq \\eta_k$ such that\r\n$$ v \\in \\partial_x \\mcL(x^{k+1},\\lambda^k;\\rho_k) + \\frac{1}{\\rho_k}(x^{k+1} - x^k). $$\r\nThis together with~\\eqref{eq:diff-x-lambda-m} implies that\r\n\\begin{equation}\r\n\\label{eq:diff-x-lambda-m-v}\r\nx^k + \\rho_kv \\in \\rho_k \\partial_x l(x^{k+1},\\lambda^{k+1}) +  x^{k+1}, \\quad \\lambda^k \\in -\\rho_k\\partial_\\lambda l(x^{k+1}, \\lambda^{k+1}) + \\lambda^{k+1},\r\n\\end{equation}\r\nwhich, by the definition of $\\mcT_l$, are equivalent to \r\n$$ (x^k + \\rho_kv, \\lambda^k) \\in (\\mcI + \\rho_k\\mcT_l)(x^{k+1},\\lambda^{k+1}). $$\r\nIt follows from this and $\\mcJ_{\\rho_k}=(\\mcI + \\rho_k\\mcT_l)^{-1}$ that  $(x^{k+1},\\lambda^{k+1}) = \\mcJ_{\\rho_k}(x^k+\\rho_kv,\\lambda^k)$. By this and the non-expansion of $\\mcJ_{\\rho_k}$, we obtain\r\n$$ \\|(x^{k+1}, \\lambda^{k+1}) - \\mcJ_{\\rho_k}(x^k,\\lambda^k)\\| = \\|\\mcJ_{\\rho_k}(x^k + \\rho_kv, \\lambda^k) - \\mcJ_{\\rho_k}(x^k,\\lambda^k)\\| \\leq \\|\\rho_kv\\| \\leq \\rho_k\\eta_k, $$\r\nwhich yields \\eqref{eq:mi-al=i-ppa} as desired. \r\n\\end{proof}","sha256_utf8":"fba097800bf79eaaa75ef74f88f4f724e4eabe3f194d7d0262def73fe6e3959d"},{"start":84122,"end":87317,"start_line":1011,"end_line":1039,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{prop:output-gantee-m}]\r\n(i) Suppose that Algorithm~\\ref{alg:mi-al} terminates in Step 3 at some iteration $k$. It then follows \r\nthat $(x^+,\\lambda^+)$  is already an $\\epsilon$-KKT solution of problem \\eqref{eq:conic-cp} \r\nor  the inequalities \\eqref{eq:check-stop-m} hold for such $k$.  We next show that for the latter case, $(x^+, \\lambda^+) = (x^{k+1},\\lambda^{k+1})$ is also an $\\epsilon$-KKT solution of~\\eqref{eq:conic-cp}. To this end, suppose that \\eqref{eq:check-stop-m} holds at the $k$th iteration. Notice that \\eqref{eq:diff-x-lambda-m-v} holds for some $\\|v\\|\\leq \\eta_k$, which yields  \r\n\\[\r\n\\frac{1}{\\rho_k}(x^k-x^{k+1}) + v \\in \\partial_x l(x^{k+1},\\lambda^{k+1}), \\quad \\frac{1}{\\rho_k}(\\lambda^{k+1} - \\lambda^k) \\in \\partial_\\lambda l (x^{k+1},\\lambda^{k+1}).\r\n\\]\r\nBy these, \\eqref{eq:check-stop-m} and $(x^+, \\lambda^+) = (x^{k+1},\\lambda^{k+1})$, we obtain\r\n\\begin{align*}\r\n\\mbox{dist}(0,\\partial_x l(x^+, \\lambda^+)) & \\leq \\frac{1}{\\rho_k}\\|x^{k+1} - x^k - \\rho_kv\\| \\leq \\frac{1}{\\rho_k}\\|x^{k+1} - x^k\\| + \\|v\\| \\leq \\epsilon,\\\\\r\n\\mbox{dist}(0,\\partial_\\lambda l(x^+, \\lambda^+)) & \\leq \\frac{1}{\\rho_k}\\|\\lambda^{k+1} - \\lambda^k\\| \\leq \\epsilon,\r\n\\end{align*}\r\nwhich along with Definition \\ref{defi:approx-KKT} imply that $(x^+,\\lambda^+)$ is an $\\epsilon$-KKT solution of problem \\eqref{eq:conic-cp}.\r\n\r\n(ii) Suppose for contradiction that Algorithm~\\ref{alg:mi-al} does not terminate in a finite number of iterations. It then follows that  \\eqref{eq:check-stop-m} does not hold for any $k$. By \\eqref{eq:ippa-iter-cplx-min} and Lemma \\ref{lemma:mi-al=i-ppa}, one has that\r\n$$ \\min_{k \\leq i\\leq 2k} \\|(x^{i+1},\\lambda^{i+1}) - (x^i,\\lambda^i)\\| \\leq \\frac{\\sqrt{2}\\left(D + 2\\sum_{i=0}^{2k}\\rho_i\\eta_i\\right)}{\\sqrt{k+1}}, $$ \r\nwhere $D$ is defined in \\eqref{param-m}. Since $\\{\\rho_k\\}$ is assumed to be nondecreasing, we further have\r\n\\[\r\n\\min_{k \\leq i\\leq 2k} \\frac{1}{\\rho_i}\\|(x^{i+1},\\lambda^{i+1}) - (x^i,\\lambda^i)\\| \\leq \\frac{\\sqrt{2}\\left(D + 2\\sum_{i=0}^{2k}\\rho_i\\eta_i\\right)}{\\rho_k\\sqrt{k+1}}.\r\n\\]\r\nBy this and \\eqref{rho-eta-m}, one has that $\\min_{k \\leq i\\leq 2k} \\|(x^{i+1},\\lambda^{i+1}) - (x^i,\\lambda^i)\\|/\\rho_i \\to 0$ and $\\eta_k\\to 0$ as $k \\to \\infty$, which implies that \\eqref{eq:check-stop-m} is satisfied for some $k$ and thus leads to a contradiction. \r\n\r\n(iii) Let $\\hat x^*$ and $\\hat\\lambda^*$ be defined right above \\eqref{param} and \\eqref{param-m}, respectively. Suppose for contradiction that Algorithm~\\ref{alg:mi-al} does not terminate within $N+1$ outer iterations. It then follows that \\eqref{eq:check-stop-m} does not hold for all $0\\le k \\le N$. Then, by \\eqref{param-m}, \\eqref{eq:ippa-iter-cplx}, and \\eqref{eq:mi-al=i-ppa}, we have that for all $0\\le k \\le N$,\r\n\\begin{equation*}\\label{eq:mi-ppa-bd}\r\n\\|(x^{k+1},\\lambda^{k+1}) - (x^k,\\lambda^k)\\| \\leq \\|(x^0,\\lambda^0) - (\\hat x^*, \\hat\\lambda^*)\\| + \\sum_{i=0}^k\\rho_i\\eta_i \\leq D + \\sum_{i=0}^k\\rho_i\\eta_i,\r\n\\end{equation*}\r\nwhere $D$ is given in \\eqref{param-m}. \r\nBy this and \\eqref{eq:cplx-out-general-m}, one can see that \\eqref{eq:check-stop-m} is satisfied when $k = N$, which leads to a contradiction.  \r\n\\end{proof}","sha256_utf8":"21d37f65957bf796e05fb870374ff45f4413db43ec6b2b9a51fca82a479f9280"},{"start":114952,"end":118599,"start_line":1372,"end_line":1420,"raw_tex":"\\begin{proof}\r\nSince $0\\in\\mcT(z^*)$ and $\\mcT$ is maximally monotone, it follows from \\cite[Proposition 1]{Rock76a} that\r\n\\begin{equation}\r\n\\label{eq:suff-dec-ppa-1}\r\n\\| \\mcJ_\\rho(z) - z^*\\|^2 + \\|\\mcJ_\\rho(z) - z\\|^2 \\leq \\|z - z^*\\|^2,\r\n\\end{equation}\r\nwhich implies that \r\n\\begin{equation}\r\n\\label{eq:suff-dec-ppa-2}\r\n\\|\\mcJ_\\rho(z) - z^*\\| \\leq \\|z - z^*\\|, \\quad \\|\\mcJ_\\rho(z) - z\\| \\leq \\|z - z^*\\|, \\quad \\forall z \\in \\Re^n.\r\n\\end{equation}\r\n\tLet $\\xi^k=z^{k+1} - \\mcJ_{\\rho_k}(z^k)$ for all $k\\ge 0$. By this, \\eqref{eq:suff-dec-ppa-2}, and \\eqref{eq:iter-PPA} with $\\rho = \\rho_k$ and $z = z^k$, one has\r\n\t\\[\r\n\t\\|z^{k+1} - z^{*}\\| \\leq \\|z^{k+1} - \\mcJ_{\\rho_k}(z^k)\\| + \\|\\mcJ_{\\rho_k}(z^k) - z^{*}\\| \r\n\t\\leq \\|\\xi^k\\| + \\|z^k - z^{*}\\|, \\quad \\forall k \\ge 0.\r\n\t\\]\r\n\tSumming up the above inequality from $k=t$ to $k=s-1$ yields \r\n\t\\beq \\label{bdd-1}\r\n\t\\|z^s - z^{*}\\| \\leq \\|z^t - z^{*}\\| + \\sum_{i=t}^{s-1} \\|\\xi^i\\|, \\quad \\forall s\\geq t\\geq 0.\r\n\t\\eeq\r\n\tNotice from \\eqref{eq:iter-PPA} that $\\|\\xi^k\\| \\le e_k$ for all $k\\ge 0$, which along with \\eqref{bdd-1} leads to \\eqref{eq:iter-ppa-bd}. Besides, by~\\eqref{eq:suff-dec-ppa-2} with $\\rho=\\rho_k$ and $z = z^k$, one has $ \\|\\mcJ_{\\rho_k}(z^k) - z^k\\|  \\leq \\|z^k - z^*\\|$. These together with \\eqref{eq:iter-PPA} and \\eqref{eq:iter-ppa-bd} yield\r\n\t$$ \\|z^{k+1} - z^k\\| \\leq \\|z^{k+1} - \\mcJ_{\\rho_k}(z^k)\\| + \\|\\mcJ_{\\rho_k}(z^k) - z^k\\| \\leq e_k + \\|z^k - z^*\\| \\leq \\|z^0 - z^*\\| + \\sum_{i=0}^k e_i.  $$\r\n\tIn addition, by the definition of $\\xi^k$, and \\eqref{eq:suff-dec-ppa-1} \r\n\twith $\\mcJ = \\mcJ_{\\rho_k}$ and $z = z^k$, one has\r\n\t\\begin{equation*}\r\n\t\\begin{split}\r\n\t\\|\\mcJ_{\\rho_k}(z^k) - z^k\\|^2 & \\leq \\|z^k - z^{*}\\|^2 - \\|\\mcJ_{\\rho_k}(z^k) - z^{*}\\|^2 \r\n\t= \\|z^k - z^{*}\\|^2 - \\|\\mcJ_{\\rho_k}(z^k) - z^{k+1} + z^{k+1} - z^{*}\\|^2 \\\\\r\n\t& \\leq \\|z^k - z^{*}\\|^2 - \\|z^{k+1} - z^{*}\\|^2 - \\|\\xi^k\\|^2 + 2\\|\\xi^k\\|\\|z^{k+1} - z^{*}\\|.\r\n\t\\end{split}\r\n\t\\end{equation*}\r\n\tSumming up the above inequality from $k = K$ to $k = 2K$ and using \\eqref{bdd-1}, we obtain that \r\n\t\\begin{align}\r\n\t\\sum_{k=K}^{2K}\\|\\mcJ_{\\rho_k}(z^k) - z^k\\|^2 & \\leq \\|z^K - z^{*}\\|^2 - \\sum_{k=K}^{2K}\\|\\xi^k\\|^2 + 2\\sum_{k=K}^{2K}\\|\\xi^k\\|\\left(\\|z^K - z^{*}\\| + \\sum_{j=K}^k\\|\\xi^j\\|\\right) \\nn\\\\\r\n\t& = \\|z^K - z^{*}\\|^2 - \\sum_{k=K}^{2K}\\|\\xi^k\\|^2 + 2\\|z^K - z^{*}\\|\\cdot\\sum_{k=K}^{2K}\\|\\xi^k\\| + 2\\sum_{k=K}^{2K}\\sum_{j=K}^k\\|\\xi^k\\|\\|\\xi^j\\| \\nn\\\\\r\n\t& = \\|z^K - z^{*}\\|^2 - \\sum_{k=K}^{2K}\\|\\xi^k\\|^2 + 2\\|z^K - z^{*}\\|\\cdot\\sum_{k=K}^{2K}\\|\\xi^k\\| + \\sum_{k=K}^{2K}\\|\\xi^k\\|^2 + \\left(\\sum_{k=K}^{2K}\\|\\xi^k\\|\\right)^2 \\nn\\\\\r\n\t& = \\|z^K - z^{*}\\|^2 + 2\\|z^K - z^{*}\\|\\cdot\\sum_{k=K}^{2K}\\|\\xi^k\\| + \\left(\\sum_{k=K}^{2K}\\|\\xi^k\\|\\right)^2 \\nn\\\\\r\n\t& = \\left( \\|z^K - z^{*}\\| + \\sum_{k=K}^{2K}\\|\\xi^k\\| \\right)^2 \\leq \\left( \\|z^0 - z^{*}\\| + \\sum_{k=0}^{2K}\\|\\xi^k\\| \\right)^2, \\label{eq:bound-sum-jz-z}\r\n\t\\end{align}\r\n\twhere \\eqref{eq:bound-sum-jz-z} follows from \\eqref{bdd-1} with $t=0$ and $s = K$. Again, by the definition of $\\xi^k$, one has\r\n\t$$ \\|z^{k+1} - z^k\\|^2 = \\|\\mcJ_{\\rho_k}(z^k) + \\xi^k- z^k \\|^2 \\leq 2\\left(\\|\\mcJ_{\\rho_k}(z^k) - z^k\\|^2 + \\|\\xi^k\\|^2\\right). $$\r\n\tThis together with~\\eqref{eq:bound-sum-jz-z} yields\r\n\t\\begin{align*}\r\n\t\\sum_{k = K}^{2K}\\|z^{k+1} - z^k\\|^2 & \\leq 2\\sum_{k = K}^{2K}\\|\\mcJ_{\\rho_k}(z^k) - z^k\\|^2 + 2\\sum_{k = K}^{2K}\\|\\xi^k\\|^2 \r\n\t \\leq 2\\left( \\|z^0 - z^{*}\\| + \\sum_{k=0}^{2K}\\|\\xi^k\\| \\right)^2 + 2\\sum_{k=0}^{2K}\\|\\xi^k\\|^2 \\\\\r\n\t& \\leq 2\\left( \\|z^0 - z^{*}\\| + 2\\sum_{k=0}^{2K}\\|\\xi^k\\| \\right)^2,\r\n\t\\end{align*}\r\n\twhich along with $\\|\\xi^k\\| \\le e_k$ leads to \\eqref{eq:ippa-iter-cplx-min}. The proof is then completed.\r\n\\end{proof}","sha256_utf8":"345bd3bf4df3e8f55fba605bf7d42c842a25af1fa8702b5b6ec0dfd65bc6938b"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"正则增广法的输出与外层复杂度","source_units":[{"start":47005,"end":48065,"start_line":574,"end_line":593,"raw_tex":"\\begin{thm}\r\n\t\\label{prop:output-gantee-m}\r\n\t\\bi\r\n\t\\item[(i)]\r\n\tIf  Algorithm~\\ref{alg:mi-al} successfully terminates, then the output $(x^+,\\lambda^+)$  is an $\\epsilon$-KKT solution of problem \\eqref{eq:conic-cp}.\r\n\t\\item[(ii)]\r\n\tSuppose that $\\{\\rho_k\\}$  and $\\{\\eta_k\\}$ satisfy that\r\n\t\\beq \\label{rho-eta-m}\r\n\t\\rho_k>0 \\ \\mbox{is nondecreasing}, \\quad 0<\\eta_k \\downarrow 0, \\quad \r\n\t\\frac{\\sum_{i=0}^{2k}\\rho_i\\eta_i}{\\rho_k\\sqrt{k+1}} \\to 0.  \\footnote{For example,  $\\rho_k = \\rho_0\\alpha^k$ and $\\eta_k = \\eta_0\\beta^k$ satisfy \\eqref{rho-eta-m} for any $\\rho_0>0$, $\\eta_0>0$, $\\alpha>1$ and $0<\\beta<1/\\alpha$.}\r\n\t\\eeq \r\n\tThen Algorithm~\\ref{alg:mi-al} terminates in a finite number of outer iterations. \r\n\t\\item[(iii)] Furthermore, if $N$ is a nonnegative integer such that\r\n\t\\begin{equation}\r\n\t\\label{eq:cplx-out-general-m}\r\n\t\\frac{D + \\sum_{k=0}^N \\rho_k\\eta_k}{\\rho_N} \\leq \\frac{\\epsilon}{2}, \\quad \\eta_N \\leq \\frac{\\epsilon}{2},\r\n\t\\end{equation}\r\n\tthen Algorithm~\\ref{alg:mi-al} terminates in at most $N+1$ outer iterations.\r\n\t\\ei\r\n\\end{thm}","sha256_utf8":"245c2e110296ad427b2c61925ba5b8071067abb67fbab9d9d5d79701dac75cfc"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"主定理完整证明","source_units":[{"start":84122,"end":87317,"start_line":1011,"end_line":1039,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{prop:output-gantee-m}]\r\n(i) Suppose that Algorithm~\\ref{alg:mi-al} terminates in Step 3 at some iteration $k$. It then follows \r\nthat $(x^+,\\lambda^+)$  is already an $\\epsilon$-KKT solution of problem \\eqref{eq:conic-cp} \r\nor  the inequalities \\eqref{eq:check-stop-m} hold for such $k$.  We next show that for the latter case, $(x^+, \\lambda^+) = (x^{k+1},\\lambda^{k+1})$ is also an $\\epsilon$-KKT solution of~\\eqref{eq:conic-cp}. To this end, suppose that \\eqref{eq:check-stop-m} holds at the $k$th iteration. Notice that \\eqref{eq:diff-x-lambda-m-v} holds for some $\\|v\\|\\leq \\eta_k$, which yields  \r\n\\[\r\n\\frac{1}{\\rho_k}(x^k-x^{k+1}) + v \\in \\partial_x l(x^{k+1},\\lambda^{k+1}), \\quad \\frac{1}{\\rho_k}(\\lambda^{k+1} - \\lambda^k) \\in \\partial_\\lambda l (x^{k+1},\\lambda^{k+1}).\r\n\\]\r\nBy these, \\eqref{eq:check-stop-m} and $(x^+, \\lambda^+) = (x^{k+1},\\lambda^{k+1})$, we obtain\r\n\\begin{align*}\r\n\\mbox{dist}(0,\\partial_x l(x^+, \\lambda^+)) & \\leq \\frac{1}{\\rho_k}\\|x^{k+1} - x^k - \\rho_kv\\| \\leq \\frac{1}{\\rho_k}\\|x^{k+1} - x^k\\| + \\|v\\| \\leq \\epsilon,\\\\\r\n\\mbox{dist}(0,\\partial_\\lambda l(x^+, \\lambda^+)) & \\leq \\frac{1}{\\rho_k}\\|\\lambda^{k+1} - \\lambda^k\\| \\leq \\epsilon,\r\n\\end{align*}\r\nwhich along with Definition \\ref{defi:approx-KKT} imply that $(x^+,\\lambda^+)$ is an $\\epsilon$-KKT solution of problem \\eqref{eq:conic-cp}.\r\n\r\n(ii) Suppose for contradiction that Algorithm~\\ref{alg:mi-al} does not terminate in a finite number of iterations. It then follows that  \\eqref{eq:check-stop-m} does not hold for any $k$. By \\eqref{eq:ippa-iter-cplx-min} and Lemma \\ref{lemma:mi-al=i-ppa}, one has that\r\n$$ \\min_{k \\leq i\\leq 2k} \\|(x^{i+1},\\lambda^{i+1}) - (x^i,\\lambda^i)\\| \\leq \\frac{\\sqrt{2}\\left(D + 2\\sum_{i=0}^{2k}\\rho_i\\eta_i\\right)}{\\sqrt{k+1}}, $$ \r\nwhere $D$ is defined in \\eqref{param-m}. Since $\\{\\rho_k\\}$ is assumed to be nondecreasing, we further have\r\n\\[\r\n\\min_{k \\leq i\\leq 2k} \\frac{1}{\\rho_i}\\|(x^{i+1},\\lambda^{i+1}) - (x^i,\\lambda^i)\\| \\leq \\frac{\\sqrt{2}\\left(D + 2\\sum_{i=0}^{2k}\\rho_i\\eta_i\\right)}{\\rho_k\\sqrt{k+1}}.\r\n\\]\r\nBy this and \\eqref{rho-eta-m}, one has that $\\min_{k \\leq i\\leq 2k} \\|(x^{i+1},\\lambda^{i+1}) - (x^i,\\lambda^i)\\|/\\rho_i \\to 0$ and $\\eta_k\\to 0$ as $k \\to \\infty$, which implies that \\eqref{eq:check-stop-m} is satisfied for some $k$ and thus leads to a contradiction. \r\n\r\n(iii) Let $\\hat x^*$ and $\\hat\\lambda^*$ be defined right above \\eqref{param} and \\eqref{param-m}, respectively. Suppose for contradiction that Algorithm~\\ref{alg:mi-al} does not terminate within $N+1$ outer iterations. It then follows that \\eqref{eq:check-stop-m} does not hold for all $0\\le k \\le N$. Then, by \\eqref{param-m}, \\eqref{eq:ippa-iter-cplx}, and \\eqref{eq:mi-al=i-ppa}, we have that for all $0\\le k \\le N$,\r\n\\begin{equation*}\\label{eq:mi-ppa-bd}\r\n\\|(x^{k+1},\\lambda^{k+1}) - (x^k,\\lambda^k)\\| \\leq \\|(x^0,\\lambda^0) - (\\hat x^*, \\hat\\lambda^*)\\| + \\sum_{i=0}^k\\rho_i\\eta_i \\leq D + \\sum_{i=0}^k\\rho_i\\eta_i,\r\n\\end{equation*}\r\nwhere $D$ is given in \\eqref{param-m}. \r\nBy this and \\eqref{eq:cplx-out-general-m}, one can see that \\eqref{eq:check-stop-m} is satisfied when $k = N$, which leads to a contradiction.  \r\n\\end{proof}","sha256_utf8":"21d37f65957bf796e05fb870374ff45f4413db43ec6b2b9a51fca82a479f9280"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"凸锥规划与拉格朗日对偶","source_units":[{"start":5367,"end":6399,"start_line":150,"end_line":166,"raw_tex":"In this paper we consider convex conic programming in the form of\r\n\\begin{equation}\\label{eq:conic-cp}\r\n\\begin{array}{rl}\r\nF^*=\\min & \\left\\{F(x) := f(x) + P(x)\\right\\} \\\\\r\n\\mbox{s.t.} &  g(x)\\preceq_\\mcK 0,\r\n\\end{array}\r\n\\end{equation}\r\nwhere $f, P:\\Re^n\\rightarrow(-\\infty,\\infty]$ are proper closed convex functions, $\\mcK$ is a closed convex cone in $\\Re^m$, the symbol $\\preceq_\\mcK$ denotes the partial order induced by $\\mcK$, that is, $y\\preceq_\\mcK z$ if and only if $z-y\\in\\mcK$, and the mapping $g:\\Re^n\\rightarrow\\Re^m$ is convex with respect to $\\mcK$, that is,\r\n\\begin{equation}\r\n\\label{eq:conic-cvx}\r\ng(\\alpha x + (1-\\alpha)y) \\preceq_\\mcK \\alpha g(x) + (1-\\alpha)g(y), \\quad\\forall x,y\\in\\Re^n, \\; \\alpha\\in[0,1].\r\n\\end{equation}\r\nThe associated Lagrangian dual problem of \\eqref{eq:conic-cp} is given by \r\n\\beq \\label{dual-prob}\r\nd^*=\\sup\\limits_{\\lambda\\in\\mcK^{*}} \\inf\\limits_x\\  \\{ f(x) + P(x) + \\Inner{\\lambda}{g(x)}\\},\r\n\\eeq\r\nwhere $\\mcK^*$ is the dual cone of $\\mcK$ (see Section \\ref{sec:preliminaries}). \r\n","sha256_utf8":"e01e13f0b00c8c3a21c38af1d9c1635e743bae74992d1f8d4f5bb5fa718e9540"}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","title":"紧域光滑性与强对偶假设","source_units":[{"start":6522,"end":7800,"start_line":168,"end_line":176,"raw_tex":"\\begin{ass}\r\n\t\\label{ass:function}\r\n\t\\begin{itemize}\r\n\t\t\\item[(a)] The proximal operator associated with $P$ can be exactly evaluated, and the domain of $P$, denoted by $\\dom(P)$, is compact.\\footnote{Some problems with unbounded $\\dom(P)$ can be reformulated as the ones satisfying this assumption. For example, when $f$ is bounded below and $P$ is coercive, the problem generally can be reformulated as the one with the objective $f+\\tilde P$ for some $\\tilde P$ with a compact domain. Such a problem often arises in sparse or low-rank learning, in which $f$ is typically a nonnegative loss function and $P$ is the $\\ell_1$- or nuclear-norm.}\r\n\t\t\\item[(b)] The projection onto $\\mcK$ can be exactly evaluated.\r\n\t\t\\item[(c)] The functions $f$ and $g$ are continuously differentiable on an open set $\\Omega$ containing $\\dom(P)$,  and $\\nabla f$ and $\\nabla g$  are Lipschitz continuous on $\\Omega$ with Lipschitz constants $L_{\\nabla f}$ and $L_{\\nabla g}$, respectively.\\footnote{The symbol $\\nabla g$ denotes the transpose of the Jacobian of $g$.}\r\n\\item[(d)] The strong duality holds for problems \\eqref{eq:conic-cp} and \\eqref{dual-prob}, that is, both problems have optimal solutions and moreover their optimal values $F^*$ and $d^*$ are equal. \r\n\t\\end{itemize}\r\n\\end{ass}","sha256_utf8":"9427b3e0ccf40a108133047fcead4e109d57603f250bdbba854f84ee20b3a587"}]},{"id":"S005","semantic_type":"notation","kind":"notation","title":"范数近端投影与法锥","source_units":[{"start":22603,"end":23903,"start_line":272,"end_line":284,"raw_tex":"The following notations will be used throughout this paper. Let $\\Re^n$ denote the Euclidean space of dimension $n$, $\\langle\\cdot,\\cdot\\rangle$ denote the standard inner product, and $\\|\\cdot\\|$ stand for the Euclidean norm. The symbols $\\Re_+$ and $\\Re_{++}$ stand for the set of nonnegative and positive real numbers, respectively.  \r\n\r\nGiven a closed convex function $h:\\Re^n\\rightarrow(-\\infty,\\infty]$, $\\partial h$ and $\\dom (h)$ denote the subdifferential and domain  of $h$, respectively.  The proximal operator associated with $h$ is denoted by  \r\n$\\prox_h$,  that is,\r\n\\begin{equation*}\r\n\\label{eq:def-prox}\r\n\\prox_h(z) = \\argmin_{x\\in\\Re^n} \\left\\{ \\frac{1}{2}\\|x - z\\|^2 + h(x) \\right\\}, \\quad \\forall z \\in \\Re^n.\r\n\\end{equation*}\r\nGiven a non-empty closed convex set $C\\subseteq\\Re^n$, $\\dist(z,C)$ stands for the Euclidean distance from $z$ to $C$, and $\\Pi_C(z)$ denotes the Euclidean projection of $z$ onto $C$, namely, \r\n\\[\r\n \\Pi_C(z) = \\argmin\\{\\|z - x\\|: x\\in C\\}, \\quad \\dist(z,C)=\\left\\|z - \\Pi_C(z)\\right\\|, \\quad \\forall z \\in \\Re^n.\r\n\\]\r\nThe normal cone of $C$ at any $z\\in C$ is denoted by $\\mcN_C(z)$. For the closed convex cone $\\mcK$, we use $\\mcK^*$ to denote the dual cone of $\\mcK$, that is,  $\\mcK^* = \\{y\\in\\Re^m: \\langle y,x\\rangle\\geq 0, \\; \\forall x\\in\\mcK\\}$.\r\n","sha256_utf8":"dd2065fccae7515e30c301c728d95cc0c260b717f53348cfb1887fe5d6b5c6d3"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"拉格朗日函数与对偶函数","source_units":[{"start":23905,"end":24873,"start_line":286,"end_line":306,"raw_tex":"The Lagrangian function $l(x,\\lambda)$ of problem~\\eqref{eq:conic-cp} is defined as\r\n\\begin{equation*}\r\nl(x,\\lambda) = \\left\\{\r\n\\begin{array}{ll}\r\nf(x) + P(x) + \\Inner{\\lambda}{g(x)} & \\mbox{if}\\; x\\in \\dom(P) \\mbox{ and } \\lambda \\in \\mcK^{*}, \\\\\r\n-\\infty & \\mbox{if}\\; x\\in \\dom(P) \\mbox{ and } \\lambda\\notin\\mcK^{*}, \\\\\r\n\\infty & \\mbox{if}\\; x\\notin \\dom(P),\r\n\\end{array}\r\n\\right.\r\n\\end{equation*}\r\nwhich is a closed convex-concave function. The Lagrangian dual function $d:\\Re^m\\rightarrow[-\\infty,\\infty)$ is defined as \r\n\\begin{equation}\r\n\\label{eq:lag-dual-func}\r\nd(\\lambda) = \\inf_x l(x,\\lambda) = \\left\\{\r\n\\begin{array}{ll}\r\n\\inf\\limits_x\\  \\{ f(x) + P(x) + \\Inner{\\lambda}{g(x)}\\} & \\mbox{if}\\; \\lambda\\in\\mcK^{*},\\\\\r\n-\\infty & \\mbox{if}\\; \\lambda\\notin\\mcK^{*},\r\n\\end{array}\r\n\\right.\r\n\\end{equation}\r\nwhich is a closed concave function. Moreover, the augmented Lagrangian function for problem~\\eqref{eq:conic-cp} is defined as (e.g., see~\\cite{Shapiro04})\r\n","sha256_utf8":"80e0e9b8903f074f929cb27f5a037d998e8eae0768949dde1008d328c4d80c3f"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"增广函数及其对偶函数","source_units":[{"start":24873,"end":25542,"start_line":307,"end_line":323,"raw_tex":"\\begin{equation}\r\n\\label{eq:al-func}\r\n\\mcL(x,\\lambda;\\rho) = f(x) + P(x) + \\frac{1}{2\\rho}\\Big[\\dist^2\\big(\\lambda + \\rho g(x), -\\mcK\\big) - \\|\\lambda\\|^2\\Big],\r\n\\end{equation}\r\nwhere $\\rho>0$ is  a penalty parameter. For convenience, we let\r\n\\beq \\label{S-fun}\r\n\\mcS(x,\\lambda;\\rho):= f(x) + \\frac{1}{2\\rho} \\dist^2\\big(\\lambda+\\rho g(x), -\\mcK \\big).\r\n\\eeq\r\nIt is clear to see that \r\n\\[\r\n\\mcL(x,\\lambda;\\rho) = \\mcS(x,\\lambda;\\rho) + P(x) - \\frac{\\|\\lambda\\|^2}{2\\rho}.\r\n\\]\r\nThe augmented Lagrangian dual function of~\\eqref{eq:conic-cp} is given by\r\n\\begin{equation}\r\n\\label{eq:aug-dual-func}\r\nd(\\lambda;\\rho)= \\min_{x\\in\\Re^n} \\mcL(x,\\lambda;\\rho).\r\n\\end{equation}\r\n","sha256_utf8":"e20f9c8bcb8e80f2b53d96b2342564dba4c3c63a6ba5ce25c4130d8ef30833ad"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"鞍点算子及次微分表达式","source_units":[{"start":25542,"end":27170,"start_line":324,"end_line":347,"raw_tex":"The Lagrangian dual problem \\eqref{dual-prob} can thus be rewritten as  \r\n\\begin{equation}\\label{eq:dual-conic-cp}\r\nd^{*} = \\max_\\lambda\\; d(\\lambda).\r\n\\end{equation}\r\nLet $\\partial l:\\Re^n\\times\\Re^m\\rightrightarrows\\Re^n\\times \\Re^m$ and $\\partial d:\\Re^m\\rightrightarrows\\Re^m$ be respectively the subdifferential mappings associated with $l$ and $d$ (e.g., see \\cite{R97}). We define two set-valued operators associated with problems \\eqref{eq:conic-cp} and \\eqref{dual-prob} as follows:\r\n\\beqa\r\n&&\\mcT_d: \\lambda \\rightarrow \\{ u\\in\\Re^m: -u\\in \\partial d(\\lambda)\\},  \\quad \\forall \\lambda\\in\\Re^m, \\label{eq:def-two-opt1} \\\\\r\n&& \\mcT_l: (x,\\lambda) \\rightarrow \\{ (v,u)\\in\\Re^n\\times\\Re^m: (v,-u)\\in\\partial l(x,\\lambda)\\}, \\quad \\forall (x,\\lambda) \\in \\Re^n \\times \\Re^m. \\label{eq:def-two-opt2}\r\n\\eeqa\r\nIt is well known that $\\lambda^*$ is an optimal solution of the Lagrangian dual problem~\\eqref{eq:dual-conic-cp} if and only if $0\\in \\partial d(\\lambda^*)$, and $(x^*,\\lambda^*)$ is a saddle point\\footnote{$(x^*,\\lambda^*)$ is called a saddle point of $l$ if it satisfies $ \\sup_{\\lambda}l(x^*,\\lambda) = l(x^*,\\lambda^*) = \\inf_x l(x,\\lambda^*)$.} of $l$ if and only if $(0,0)\\in\\partial l(x^*,\\lambda^*)$. In addition, it can be verified that \r\n\\begin{equation}\r\n\\label{eq:express-Tl}\r\n\\partial l(x,\\lambda) = \\left\\{\r\n\\begin{array}{ll}\r\n\\left(\r\n\\begin{array}{c}\r\n\\nabla f(x) + \\partial P(x) + \\nabla g(x)\\lambda \\\\\r\ng(x) - \\mcN_{\\mcK^*}(\\lambda)\r\n\\end{array}\r\n\\right), & \\mbox{if}\\; x\\in\\dom(P)\\; \\mbox{and}\\; \\lambda\\in\\mcK^*, \\medskip\r\n\\\\ \r\n\\emptyset, & \\mbox{otherwise}.\r\n\\end{array}\\right.\r\n\\end{equation}\r\n","sha256_utf8":"a0771ee4e15d4e009dd83d10e55ee607a862e4184da5bb60812c86d0796615ea"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"近似KKT解及残差条件","source_units":[{"start":10951,"end":11191,"start_line":204,"end_line":207,"raw_tex":"\\begin{equation}\r\n\\label{eq:eps-approx-kkt-point}\r\n\\dist(0, \\nabla f(x) + \\partial P(x) + \\nabla g(x)\\lambda) \\leq \\epsilon, \\quad \\dist(g(x), \\mcN_{\\mcK^*}(\\lambda))\\leq \\epsilon, \\quad (x,\\lambda) \\in \\dom(P) \\times \\mcK^*\r\n\\end{equation}","sha256_utf8":"764f253c4b7d1a6ce128bcd43749f8d98830ef939fe599cdffdde22b7794cca3"},{"start":28620,"end":28895,"start_line":367,"end_line":370,"raw_tex":"\\begin{defi}\r\n\t\\label{defi:approx-KKT}\r\n\tGiven any $\\epsilon>0$, we say $(x,\\lambda)\\in\\Re^n\\times\\Re^m$ is an $\\epsilon$-KKT solution of~\\eqref{eq:conic-cp}  if there exists $(u,v)\\in\\partial l(x,\\lambda)$ such that $\\|u\\|\\leq \\epsilon$ and $\\|v\\|\\leq \\epsilon$.\r\n\\end{defi}","sha256_utf8":"50b07a957441d2f800e89b7275fa21cae6ed0f9d8376fa7e301428a211548701"}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","title":"自适应正则增广算法","source_units":[{"start":43616,"end":45417,"start_line":532,"end_line":558,"raw_tex":"\\begin{algo}[An adaptively regularized I-AL method]\r\n\t\\label{alg:mi-al}\r\n\t\\normalfont\r\n\t\\mbox{}\r\n\t\\begin{itemize}\r\n\t\t\\item[0.] Input $\\epsilon>0$, $(x^0,\\lambda^0)\\in\\dom(P)\\times \\mcK^*$,  nondecreasing $\\{\\rho_k\\} \\subset \\Re_{++}$, and $0<\\eta_k \\downarrow 0$. Set $k=0$.\r\n\t\t\\item[1.] Apply Algorithm~\\ref{alg:opt-fom-str} (in Appendix \\ref{app-fom}) to the problem $\\min_x \\varphi_k(x)$ to  find $x^{k+1}\\in\\dom(P)$ satisfying\r\n\t\t\\begin{equation}\r\n\t\t\\label{eq:inext-subprob-m}\r\n\t\t\\dist (0, \\partial \\varphi_k(x^{k+1})) \\leq \\eta_k,\\footnote{In view of Proposition \\ref{prop:cplx-fom-str-cvx},  one can terminate Algorithm~\\ref{alg:opt-fom-str} once a point $x^{k+1}$ satisfying $2L_{\\nabla \\phi}\\|\\tilde{x}^{k+1} - x^{k+1}\\| \\le \\eta_k$ is found, where $\\phi(\\cdot)= \\varphi_k(\\cdot)-P(\\cdot)$, $\\tilde{x}^{k+1}= \\prox_{P/L_{\\nabla \\phi}}\\left( x^{k+1} - \\nabla \\phi(x^{k+1})/L_{\\nabla \\phi} \\right)$, and $L_{\\nabla \\phi}$ is the Lipschitz constant of $\\nabla \\phi$. Such $x^{k+1}$ clearly satisfies \\eqref{eq:inext-subprob-m}.}\r\n\t\t\\end{equation}\r\n\t\twhere \r\n\t\t\\begin{equation}\r\n\t\t\\label{eq:prox-al-func}\r\n\t\t\\varphi_k(x) = \\mcL(x,\\lambda^k;\\rho_k) + \\frac{1}{2\\rho_k}\\|x - x^k\\|^2.\r\n\t\t\\end{equation}\r\n\t\t\\item[2.] Set $\\lambda^{k+1} = \\Pi_{\\mcK^*}\\big(\\lambda^k + \\rho_k g(x^{k+1}) \\big)$. \r\n\t\t\\item[3.] If $(x^{k+1},\\lambda^{k+1})$ satisfies \\eqref{eq:eps-approx-kkt-point} or the following two inequalities are satisfied\r\n\t\t\\begin{equation}\r\n\t\t\\label{eq:check-stop-m}\r\n\t\t\\frac{1}{\\rho_k}\\|(x^{k+1}, \\lambda^{k+1}) - (x^k,\\lambda^k)\\| \\leq \\frac{\\epsilon}{2}, \\quad \\eta_k \\leq \\frac{\\epsilon}{2},\r\n\t\t\\end{equation}\r\n\t\toutput $(x^+, \\lambda^{+}) = (x^{k+1}, \\lambda^{k+1})$ and terminate the algorithm.\r\n\t\t\\item[4.] Set $k \\leftarrow k+1$ and go to Step 1.\r\n\t\\end{itemize}\r\n\t{\\bf End.}\r\n\\end{algo}","sha256_utf8":"a48dce36a5c08e0cac594ebe7b843bd465b8f88595aec73712c08db2473ebaf6"}]},{"id":"S011","semantic_type":"notation","kind":"notation","title":"约束映射的有界常数","source_units":[{"start":33365,"end":33589,"start_line":416,"end_line":416,"raw_tex":"We next present the subroutine Postprocessing that is used in Step 4 of Algorithm \\ref{alg:i-al}. Before proceeding, let $L_g$ be the  Lipschitz constant of $g$ on $\\dom(P)$ and $M_g = \\max\\limits_{x\\in\\dom(P)}\\|g(x)\\|$.  \r\n","sha256_utf8":"7d40651a9a81e6a8d6abde18f309e3e7a328eac05171ebaf848d9c35cdd02b56"}]},{"id":"S012","semantic_type":"notation","kind":"notation","title":"对偶最优点与初始距离","source_units":[{"start":37153,"end":37950,"start_line":456,"end_line":461,"raw_tex":"In the rest of this section we present our main results for Algorithm \\ref{alg:i-al}, whose proof is deferred  to Subsection \\ref{sec:pf-i-al}. To proceed, we introduce some further notation below. Let $\\Lambda^*$ be the set of optimal solutions of problem \\eqref{dual-prob} and $\\hat\\lambda^*\\in \\Lambda^*$ such that $\\|\\lambda^0-\\hat\\lambda^*\\|=\\dist(\\lambda^0,\\Lambda^*)$. In addition, we define \r\n\\begin{align}\r\n & D_X = \\max\\limits_{x,y\\in\\dom(P)}\\|x - y\\|,  \\  D_\\Lambda = \\|\\lambda^0 - \\hat\\lambda^*\\|, \\ B = L_{\\nabla f} + L_{\\nabla g}(\\|\\hat\\lambda^*\\| + D_\\Lambda), \\label{param} \\\\\r\n & C = L_{\\nabla g}M_g + L_g^2, \\ \\bD=\\max\\{D_\\Lambda,1\\}, \\ \\bB =\\max\\{B,1\\}, \\ \\bC =\\max\\{C,1\\}, \\label{bparam}\r\n\\end{align}\r\nwhere  $L_{\\nabla f}$, $L_{\\nabla g}$, $L_g$ and $M_g$ are defined above.\r\n","sha256_utf8":"4ed29616323492306829627a90706fc310d3e131c4d81c0d29345ebc28192ee7"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"内外迭代与正则强凸性","source_units":[{"start":45423,"end":45977,"start_line":561,"end_line":564,"raw_tex":"For ease of later reference, we refer to the iterations of Algorithm~\\ref{alg:opt-fom-str} for solving the \r\nAL subproblems as the {\\it inner iterations} of Algorithm \\ref{alg:mi-al}, and call the update from \r\n$(x^k,\\lambda^k)$ to $(x^{k+1},\\lambda^{k+1})$ an {\\it outer iteration} of Algorithm \\ref{alg:mi-al}. \r\nNotice from \\eqref{eq:prox-al-func} that $\\varphi_k$ is strongly convex with modulus $1/\\rho_k$. The AL subproblem $\\min_x \\varphi_k(x)$ arising in Algorithm \\ref{alg:mi-al} can thus be suitably solved by Algorithm~\\ref{alg:opt-fom-str}.\r\n","sha256_utf8":"6157a1f0b94efb9f62a5016463c414cc9745b0760601679ab1d1a62468159dba"}]},{"id":"S014","semantic_type":"notation","kind":"notation","title":"原始最优点与联合初距","source_units":[{"start":45979,"end":46564,"start_line":566,"end_line":570,"raw_tex":"In the rest of this section, we present our main results for Algorithm \\ref{alg:mi-al}, whose proof is deferred  to Subsection \\ref{sec:pf-mi-al}. Before proceeding, we introduce some further notation that will be used subsequently. Let $X^*$ be the set of optimal solutions of problem \\eqref{eq:conic-cp} and $\\hat x^*\\in X^*$ such that \r\n$\\|x^0-\\hat x^*\\|=\\dist(x^0,X^*)$. In addition, we define \r\n\\beq \\label{param-m} \r\n{\\bar D}_X =\\max\\{D_X,1\\}, \\ D=  \\dist(x^0,X^*)+D_\\Lambda, \\ \\bar{D}= \\max\\{D,1\\}, \\ \\hat B = L_{\\nabla f} + L_{\\nabla g}\\|\\hat\\lambda^*\\| + L_{\\nabla g}D, \r\n\\eeq","sha256_utf8":"2111904263f3d09d422eddf6ffd8c1bce1930871e12bafb895b3e0d5f7ef9c8a"}]},{"id":"S015","semantic_type":"proposition","kind":"proposition","title":"最优解与KKT鞍点等价","source_units":[{"start":27269,"end":27817,"start_line":349,"end_line":359,"raw_tex":"\\begin{prop}\r\nUnder Assumption~\\ref{ass:function}, $x^*\\in\\Re^n$ is an optimal solution of~\\eqref{eq:conic-cp} if and only if there exists $\\lambda^*\\in\\Re^m$ such that\r\n\\begin{equation}\r\n\\label{eq:KKT-condition}\r\n(0,0)\\in \\partial l(x^*,\\lambda^*),\r\n\\end{equation}\r\nor equivalently, $(x^*,\\lambda^*)$ satisfies the KKT conditions for~\\eqref{eq:conic-cp}, that is,\r\n\\[\r\n0\\in\\nabla f(x^*) + \\partial P(x^*) + \\nabla g(x^*) \\lambda^*, \\quad \\lambda^* \\in\\mcK^*, \\quad g(x^*)\\preceq_\\mcK 0, \\quad \\langle \\lambda^*, g(x^*) \\rangle = 0.\r\n\\]\r\n\\end{prop}","sha256_utf8":"570b31493f92c425b62c6f21cf428cf576cfd38bda54978148ca444d5c8991b1"}]},{"id":"S016","semantic_type":"proof","kind":"proof","title":"KKT等价完整证明","source_units":[{"start":27819,"end":28369,"start_line":360,"end_line":362,"raw_tex":"\\begin{proof}\r\nThe result \\eqref{eq:KKT-condition} follows from \\cite[Theorem 36.6]{R97}. By \\eqref{eq:express-Tl}, it is not hard to see that \\eqref{eq:KKT-condition} holds if and only if $0\\in \\nabla f(x^*) + \\partial P(x^*) + \\nabla g(x^*)\\lambda^*$, $\\lambda^*\\in\\mcK^*$, and $g(x^*) \\in\\mcN_{\\mcK^*}(\\lambda^*)$. By the definition of $\\mcK^*$ and $\\mcN_{\\mcK^*}$, one can verify that $g(x^*)\\in\\mcN_{\\mcK^*}(\\lambda^*)$ is equivalent to $g(x^*)\\preceq_\\mcK 0$ and $\\langle\\lambda^*, g(x^*) \\rangle = 0$. The proof is then completed.\r\n\\end{proof}","sha256_utf8":"9753c183a3856ca34d5bbda60e53973207c9e803c929f087e63cc4f4965b61ee"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"鞍点条件化为锥互补条件","source_units":[{"start":27819,"end":28369,"start_line":360,"end_line":362,"raw_tex":"\\begin{proof}\r\nThe result \\eqref{eq:KKT-condition} follows from \\cite[Theorem 36.6]{R97}. By \\eqref{eq:express-Tl}, it is not hard to see that \\eqref{eq:KKT-condition} holds if and only if $0\\in \\nabla f(x^*) + \\partial P(x^*) + \\nabla g(x^*)\\lambda^*$, $\\lambda^*\\in\\mcK^*$, and $g(x^*) \\in\\mcN_{\\mcK^*}(\\lambda^*)$. By the definition of $\\mcK^*$ and $\\mcN_{\\mcK^*}$, one can verify that $g(x^*)\\in\\mcN_{\\mcK^*}(\\lambda^*)$ is equivalent to $g(x^*)\\preceq_\\mcK 0$ and $\\langle\\lambda^*, g(x^*) \\rangle = 0$. The proof is then completed.\r\n\\end{proof}","sha256_utf8":"9753c183a3856ca34d5bbda60e53973207c9e803c929f087e63cc4f4965b61ee"}]},{"id":"S018","semantic_type":"lemma","kind":"lemma","title":"增广函数的标准包络恒等式","source_units":[{"start":52000,"end":53213,"start_line":647,"end_line":669,"raw_tex":"\\begin{lemma}\r\n\t\\label{lemma:al-func-elp}\r\n\tFor any $(x,\\lambda,\\rho)\\in\\Re^n\\times\\Re^m\\times\\Re_{++}$, the following identity holds\r\n\t\\begin{equation*}\r\n\t\\mcL(x,\\lambda;\\rho) = \\max_{\\eta\\in\\Re^m} \\left\\{ l(x,\\eta) - \\frac{1}{2\\rho}\\|\\eta - \\lambda\\|^2\\right\\}.\r\n\t\\end{equation*}\r\n\tIn addition, if $x\\in\\dom(P)$, the maximum is attained uniquely at $\\bar{\\lambda} = \\Pi_{\\mcK^*}(\\lambda + \\rho g(x))$. Consequently, the following statements hold.\r\n\t\\begin{itemize}\r\n\t\t\\item[(i)] For any $(\\lambda,\\rho)\\in\\Re^m\\times\\Re_{++}$, $d(\\lambda;\\rho)$ satisfies that\r\n\t\t\\begin{equation}\r\n\t\t\\label{eq:al-dual-func-elp-cp}\r\n\t\td(\\lambda;\\rho) = \\max_{\\eta\\in\\Re^m}\\left\\{ d(\\eta) - \\frac{1}{2\\rho}\\|\\eta - \\lambda\\|^2\\right\\}. \r\n\t\t\\end{equation}\r\n\t\t\\item[(ii)] $\\mcL(x,\\lambda;\\rho)$ is a convex function in $x$, and for any $x\\in\\dom(P)$, we have\r\n\\[\r\n\t\t\\partial_x\\mcL(x,\\lambda;\\rho) = \\partial_x l(x,\\bar{\\lambda}).\r\n\\]\r\n\t\t\\item[(iii)] $\\mcL(x,\\lambda;\\rho)$ is a concave function in $\\lambda$, and for any $x\\in\\dom(P)$, it is differentiable in $\\lambda$ and\r\n\\[\r\n\t\t\\frac{1}{\\rho}(\\bar{\\lambda} - \\lambda) = \\nabla_\\lambda \\mcL(x,\\lambda;\\rho) \\in \\partial_\\lambda l(x,\\bar{\\lambda}).\r\n\\]\r\n\t\\end{itemize}\r\n\\end{lemma}","sha256_utf8":"1a78b4767d75ce1db78b3e449b33c595d127f283f3c854589badfe7c2c6b40c9"}]},{"id":"S019","semantic_type":"lemma","kind":"lemma","title":"非精确近端点迭代界","source_units":[{"start":53347,"end":54376,"start_line":672,"end_line":689,"raw_tex":"\\begin{lemma}\r\n\\label{lemma:cplx-ippa} \r\nLet $\\mcT:\\Re^n\\rightrightarrows\\Re^n$ be a maximally monotone operator and  $z^*\\in\\Re^n$  such that $0\\in\\mcT(z^*)$. Let $\\{z^k\\}$ be a sequence generated by an inexact PPA, starting with some $z^0$ and obtaining $z^{k+1}$ by approximately evaluating $\\mcJ_{\\rho_k}(z^k)$ such that \r\n\\begin{equation}\r\n\\label{eq:iter-PPA}\r\n\\|z^{k+1} - \\mcJ_{\\rho_k}(z^k)\\| \\le  e_k\r\n\\end{equation}\r\nfor some $\\rho_k>0$ and $e_k\\geq 0$, where $\\mcJ_\\rho=(\\mcI + \\rho\\mcT)^{-1}$  and $\\mcI$ is the identify operator. Then we have\r\n\\beqa\r\n\\|z^s - z^{*}\\| \\leq \\|z^t - z^{*}\\| + \\sum_{i=t}^{s-1}e_i, \\quad \\forall s\\geq t\\geq 0, \\label{eq:iter-ppa-bd} \\\\\r\n\\|z^{k+1} - z^k\\| \\leq \\|z^0 - z^*\\| + \\sum_{i=0}^k e_i, \\quad \\forall k\\geq 0. \\label{eq:ippa-iter-cplx}\r\n\\eeqa\r\nMoreover, for any $K\\ge 1$, we have\r\n\\begin{equation}\r\n\\label{eq:ippa-iter-cplx-min}\r\n\\min_{K\\leq k\\leq 2K}\\|z^{k+1} - z^k\\| \\leq \\frac{\\sqrt{2}\\left(\\|z^0 - z^{*}\\| + 2\\sum_{k=0}^{2K}e_k\\right)}{\\sqrt{K+1}}.\r\n\\end{equation}\r\n\\end{lemma}","sha256_utf8":"0e103ca44353e2d79fead744f5bd23e5f60b9614405d3cd3855669800e685e52"}]},{"id":"S020","semantic_type":"proof","kind":"proof","title":"近端点迭代界完整证明","source_units":[{"start":114952,"end":118599,"start_line":1372,"end_line":1420,"raw_tex":"\\begin{proof}\r\nSince $0\\in\\mcT(z^*)$ and $\\mcT$ is maximally monotone, it follows from \\cite[Proposition 1]{Rock76a} that\r\n\\begin{equation}\r\n\\label{eq:suff-dec-ppa-1}\r\n\\| \\mcJ_\\rho(z) - z^*\\|^2 + \\|\\mcJ_\\rho(z) - z\\|^2 \\leq \\|z - z^*\\|^2,\r\n\\end{equation}\r\nwhich implies that \r\n\\begin{equation}\r\n\\label{eq:suff-dec-ppa-2}\r\n\\|\\mcJ_\\rho(z) - z^*\\| \\leq \\|z - z^*\\|, \\quad \\|\\mcJ_\\rho(z) - z\\| \\leq \\|z - z^*\\|, \\quad \\forall z \\in \\Re^n.\r\n\\end{equation}\r\n\tLet $\\xi^k=z^{k+1} - \\mcJ_{\\rho_k}(z^k)$ for all $k\\ge 0$. By this, \\eqref{eq:suff-dec-ppa-2}, and \\eqref{eq:iter-PPA} with $\\rho = \\rho_k$ and $z = z^k$, one has\r\n\t\\[\r\n\t\\|z^{k+1} - z^{*}\\| \\leq \\|z^{k+1} - \\mcJ_{\\rho_k}(z^k)\\| + \\|\\mcJ_{\\rho_k}(z^k) - z^{*}\\| \r\n\t\\leq \\|\\xi^k\\| + \\|z^k - z^{*}\\|, \\quad \\forall k \\ge 0.\r\n\t\\]\r\n\tSumming up the above inequality from $k=t$ to $k=s-1$ yields \r\n\t\\beq \\label{bdd-1}\r\n\t\\|z^s - z^{*}\\| \\leq \\|z^t - z^{*}\\| + \\sum_{i=t}^{s-1} \\|\\xi^i\\|, \\quad \\forall s\\geq t\\geq 0.\r\n\t\\eeq\r\n\tNotice from \\eqref{eq:iter-PPA} that $\\|\\xi^k\\| \\le e_k$ for all $k\\ge 0$, which along with \\eqref{bdd-1} leads to \\eqref{eq:iter-ppa-bd}. Besides, by~\\eqref{eq:suff-dec-ppa-2} with $\\rho=\\rho_k$ and $z = z^k$, one has $ \\|\\mcJ_{\\rho_k}(z^k) - z^k\\|  \\leq \\|z^k - z^*\\|$. These together with \\eqref{eq:iter-PPA} and \\eqref{eq:iter-ppa-bd} yield\r\n\t$$ \\|z^{k+1} - z^k\\| \\leq \\|z^{k+1} - \\mcJ_{\\rho_k}(z^k)\\| + \\|\\mcJ_{\\rho_k}(z^k) - z^k\\| \\leq e_k + \\|z^k - z^*\\| \\leq \\|z^0 - z^*\\| + \\sum_{i=0}^k e_i.  $$\r\n\tIn addition, by the definition of $\\xi^k$, and \\eqref{eq:suff-dec-ppa-1} \r\n\twith $\\mcJ = \\mcJ_{\\rho_k}$ and $z = z^k$, one has\r\n\t\\begin{equation*}\r\n\t\\begin{split}\r\n\t\\|\\mcJ_{\\rho_k}(z^k) - z^k\\|^2 & \\leq \\|z^k - z^{*}\\|^2 - \\|\\mcJ_{\\rho_k}(z^k) - z^{*}\\|^2 \r\n\t= \\|z^k - z^{*}\\|^2 - \\|\\mcJ_{\\rho_k}(z^k) - z^{k+1} + z^{k+1} - z^{*}\\|^2 \\\\\r\n\t& \\leq \\|z^k - z^{*}\\|^2 - \\|z^{k+1} - z^{*}\\|^2 - \\|\\xi^k\\|^2 + 2\\|\\xi^k\\|\\|z^{k+1} - z^{*}\\|.\r\n\t\\end{split}\r\n\t\\end{equation*}\r\n\tSumming up the above inequality from $k = K$ to $k = 2K$ and using \\eqref{bdd-1}, we obtain that \r\n\t\\begin{align}\r\n\t\\sum_{k=K}^{2K}\\|\\mcJ_{\\rho_k}(z^k) - z^k\\|^2 & \\leq \\|z^K - z^{*}\\|^2 - \\sum_{k=K}^{2K}\\|\\xi^k\\|^2 + 2\\sum_{k=K}^{2K}\\|\\xi^k\\|\\left(\\|z^K - z^{*}\\| + \\sum_{j=K}^k\\|\\xi^j\\|\\right) \\nn\\\\\r\n\t& = \\|z^K - z^{*}\\|^2 - \\sum_{k=K}^{2K}\\|\\xi^k\\|^2 + 2\\|z^K - z^{*}\\|\\cdot\\sum_{k=K}^{2K}\\|\\xi^k\\| + 2\\sum_{k=K}^{2K}\\sum_{j=K}^k\\|\\xi^k\\|\\|\\xi^j\\| \\nn\\\\\r\n\t& = \\|z^K - z^{*}\\|^2 - \\sum_{k=K}^{2K}\\|\\xi^k\\|^2 + 2\\|z^K - z^{*}\\|\\cdot\\sum_{k=K}^{2K}\\|\\xi^k\\| + \\sum_{k=K}^{2K}\\|\\xi^k\\|^2 + \\left(\\sum_{k=K}^{2K}\\|\\xi^k\\|\\right)^2 \\nn\\\\\r\n\t& = \\|z^K - z^{*}\\|^2 + 2\\|z^K - z^{*}\\|\\cdot\\sum_{k=K}^{2K}\\|\\xi^k\\| + \\left(\\sum_{k=K}^{2K}\\|\\xi^k\\|\\right)^2 \\nn\\\\\r\n\t& = \\left( \\|z^K - z^{*}\\| + \\sum_{k=K}^{2K}\\|\\xi^k\\| \\right)^2 \\leq \\left( \\|z^0 - z^{*}\\| + \\sum_{k=0}^{2K}\\|\\xi^k\\| \\right)^2, \\label{eq:bound-sum-jz-z}\r\n\t\\end{align}\r\n\twhere \\eqref{eq:bound-sum-jz-z} follows from \\eqref{bdd-1} with $t=0$ and $s = K$. Again, by the definition of $\\xi^k$, one has\r\n\t$$ \\|z^{k+1} - z^k\\|^2 = \\|\\mcJ_{\\rho_k}(z^k) + \\xi^k- z^k \\|^2 \\leq 2\\left(\\|\\mcJ_{\\rho_k}(z^k) - z^k\\|^2 + \\|\\xi^k\\|^2\\right). $$\r\n\tThis together with~\\eqref{eq:bound-sum-jz-z} yields\r\n\t\\begin{align*}\r\n\t\\sum_{k = K}^{2K}\\|z^{k+1} - z^k\\|^2 & \\leq 2\\sum_{k = K}^{2K}\\|\\mcJ_{\\rho_k}(z^k) - z^k\\|^2 + 2\\sum_{k = K}^{2K}\\|\\xi^k\\|^2 \r\n\t \\leq 2\\left( \\|z^0 - z^{*}\\| + \\sum_{k=0}^{2K}\\|\\xi^k\\| \\right)^2 + 2\\sum_{k=0}^{2K}\\|\\xi^k\\|^2 \\\\\r\n\t& \\leq 2\\left( \\|z^0 - z^{*}\\| + 2\\sum_{k=0}^{2K}\\|\\xi^k\\| \\right)^2,\r\n\t\\end{align*}\r\n\twhich along with $\\|\\xi^k\\| \\le e_k$ leads to \\eqref{eq:ippa-iter-cplx-min}. The proof is then completed.\r\n\\end{proof}","sha256_utf8":"345bd3bf4df3e8f55fba605bf7d42c842a25af1fa8702b5b6ec0dfd65bc6938b"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"预解算子的距离下降","source_units":[{"start":114952,"end":115404,"start_line":1372,"end_line":1382,"raw_tex":"\\begin{proof}\r\nSince $0\\in\\mcT(z^*)$ and $\\mcT$ is maximally monotone, it follows from \\cite[Proposition 1]{Rock76a} that\r\n\\begin{equation}\r\n\\label{eq:suff-dec-ppa-1}\r\n\\| \\mcJ_\\rho(z) - z^*\\|^2 + \\|\\mcJ_\\rho(z) - z\\|^2 \\leq \\|z - z^*\\|^2,\r\n\\end{equation}\r\nwhich implies that \r\n\\begin{equation}\r\n\\label{eq:suff-dec-ppa-2}\r\n\\|\\mcJ_\\rho(z) - z^*\\| \\leq \\|z - z^*\\|, \\quad \\|\\mcJ_\\rho(z) - z\\| \\leq \\|z - z^*\\|, \\quad \\forall z \\in \\Re^n.\r\n\\end{equation}\r\n","sha256_utf8":"63621e2e899ca777841c8d387288ccd0722aff98dc58780cdbf722ce5885c753"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"累加非精确迭代误差","source_units":[{"start":115404,"end":115922,"start_line":1383,"end_line":1391,"raw_tex":"\tLet $\\xi^k=z^{k+1} - \\mcJ_{\\rho_k}(z^k)$ for all $k\\ge 0$. By this, \\eqref{eq:suff-dec-ppa-2}, and \\eqref{eq:iter-PPA} with $\\rho = \\rho_k$ and $z = z^k$, one has\r\n\t\\[\r\n\t\\|z^{k+1} - z^{*}\\| \\leq \\|z^{k+1} - \\mcJ_{\\rho_k}(z^k)\\| + \\|\\mcJ_{\\rho_k}(z^k) - z^{*}\\| \r\n\t\\leq \\|\\xi^k\\| + \\|z^k - z^{*}\\|, \\quad \\forall k \\ge 0.\r\n\t\\]\r\n\tSumming up the above inequality from $k=t$ to $k=s-1$ yields \r\n\t\\beq \\label{bdd-1}\r\n\t\\|z^s - z^{*}\\| \\leq \\|z^t - z^{*}\\| + \\sum_{i=t}^{s-1} \\|\\xi^i\\|, \\quad \\forall s\\geq t\\geq 0.\r\n\t\\eeq\r\n","sha256_utf8":"3bf5352a04ab4f464f1a718def89307a9c026821555e10351f1474184aba0d1e"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"导出迭代距离与单步位移界","source_units":[{"start":115922,"end":116429,"start_line":1392,"end_line":1393,"raw_tex":"\tNotice from \\eqref{eq:iter-PPA} that $\\|\\xi^k\\| \\le e_k$ for all $k\\ge 0$, which along with \\eqref{bdd-1} leads to \\eqref{eq:iter-ppa-bd}. Besides, by~\\eqref{eq:suff-dec-ppa-2} with $\\rho=\\rho_k$ and $z = z^k$, one has $ \\|\\mcJ_{\\rho_k}(z^k) - z^k\\|  \\leq \\|z^k - z^*\\|$. These together with \\eqref{eq:iter-PPA} and \\eqref{eq:iter-ppa-bd} yield\r\n\t$$ \\|z^{k+1} - z^k\\| \\leq \\|z^{k+1} - \\mcJ_{\\rho_k}(z^k)\\| + \\|\\mcJ_{\\rho_k}(z^k) - z^k\\| \\leq e_k + \\|z^k - z^*\\| \\leq \\|z^0 - z^*\\| + \\sum_{i=0}^k e_i.  $$\r\n","sha256_utf8":"d20651311b41eafbab218ccdcb3aeb1d9b89a908cc80ac57f420a6e7a8c8634f"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"展开含误差的平方下降","source_units":[{"start":116429,"end":116898,"start_line":1394,"end_line":1402,"raw_tex":"\tIn addition, by the definition of $\\xi^k$, and \\eqref{eq:suff-dec-ppa-1} \r\n\twith $\\mcJ = \\mcJ_{\\rho_k}$ and $z = z^k$, one has\r\n\t\\begin{equation*}\r\n\t\\begin{split}\r\n\t\\|\\mcJ_{\\rho_k}(z^k) - z^k\\|^2 & \\leq \\|z^k - z^{*}\\|^2 - \\|\\mcJ_{\\rho_k}(z^k) - z^{*}\\|^2 \r\n\t= \\|z^k - z^{*}\\|^2 - \\|\\mcJ_{\\rho_k}(z^k) - z^{k+1} + z^{k+1} - z^{*}\\|^2 \\\\\r\n\t& \\leq \\|z^k - z^{*}\\|^2 - \\|z^{k+1} - z^{*}\\|^2 - \\|\\xi^k\\|^2 + 2\\|\\xi^k\\|\\|z^{k+1} - z^{*}\\|.\r\n\t\\end{split}\r\n\t\\end{equation*}\r\n","sha256_utf8":"e0e9466b646c32ca1f02cef6970fed40324167e9202348279e21bef2e594ee86"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"区间求和控制精确位移","source_units":[{"start":116898,"end":118092,"start_line":1403,"end_line":1412,"raw_tex":"\tSumming up the above inequality from $k = K$ to $k = 2K$ and using \\eqref{bdd-1}, we obtain that \r\n\t\\begin{align}\r\n\t\\sum_{k=K}^{2K}\\|\\mcJ_{\\rho_k}(z^k) - z^k\\|^2 & \\leq \\|z^K - z^{*}\\|^2 - \\sum_{k=K}^{2K}\\|\\xi^k\\|^2 + 2\\sum_{k=K}^{2K}\\|\\xi^k\\|\\left(\\|z^K - z^{*}\\| + \\sum_{j=K}^k\\|\\xi^j\\|\\right) \\nn\\\\\r\n\t& = \\|z^K - z^{*}\\|^2 - \\sum_{k=K}^{2K}\\|\\xi^k\\|^2 + 2\\|z^K - z^{*}\\|\\cdot\\sum_{k=K}^{2K}\\|\\xi^k\\| + 2\\sum_{k=K}^{2K}\\sum_{j=K}^k\\|\\xi^k\\|\\|\\xi^j\\| \\nn\\\\\r\n\t& = \\|z^K - z^{*}\\|^2 - \\sum_{k=K}^{2K}\\|\\xi^k\\|^2 + 2\\|z^K - z^{*}\\|\\cdot\\sum_{k=K}^{2K}\\|\\xi^k\\| + \\sum_{k=K}^{2K}\\|\\xi^k\\|^2 + \\left(\\sum_{k=K}^{2K}\\|\\xi^k\\|\\right)^2 \\nn\\\\\r\n\t& = \\|z^K - z^{*}\\|^2 + 2\\|z^K - z^{*}\\|\\cdot\\sum_{k=K}^{2K}\\|\\xi^k\\| + \\left(\\sum_{k=K}^{2K}\\|\\xi^k\\|\\right)^2 \\nn\\\\\r\n\t& = \\left( \\|z^K - z^{*}\\| + \\sum_{k=K}^{2K}\\|\\xi^k\\| \\right)^2 \\leq \\left( \\|z^0 - z^{*}\\| + \\sum_{k=0}^{2K}\\|\\xi^k\\| \\right)^2, \\label{eq:bound-sum-jz-z}\r\n\t\\end{align}\r\n\twhere \\eqref{eq:bound-sum-jz-z} follows from \\eqref{bdd-1} with $t=0$ and $s = K$. Again, by the definition of $\\xi^k$, one has\r\n\t$$ \\|z^{k+1} - z^k\\|^2 = \\|\\mcJ_{\\rho_k}(z^k) + \\xi^k- z^k \\|^2 \\leq 2\\left(\\|\\mcJ_{\\rho_k}(z^k) - z^k\\|^2 + \\|\\xi^k\\|^2\\right). $$\r\n","sha256_utf8":"862549727c06db6ec37203c648d25af1fbe4f6f8939ba8a33964e10b7df1eb48"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"平方和推出最小位移界","source_units":[{"start":118092,"end":118599,"start_line":1413,"end_line":1420,"raw_tex":"\tThis together with~\\eqref{eq:bound-sum-jz-z} yields\r\n\t\\begin{align*}\r\n\t\\sum_{k = K}^{2K}\\|z^{k+1} - z^k\\|^2 & \\leq 2\\sum_{k = K}^{2K}\\|\\mcJ_{\\rho_k}(z^k) - z^k\\|^2 + 2\\sum_{k = K}^{2K}\\|\\xi^k\\|^2 \r\n\t \\leq 2\\left( \\|z^0 - z^{*}\\| + \\sum_{k=0}^{2K}\\|\\xi^k\\| \\right)^2 + 2\\sum_{k=0}^{2K}\\|\\xi^k\\|^2 \\\\\r\n\t& \\leq 2\\left( \\|z^0 - z^{*}\\| + 2\\sum_{k=0}^{2K}\\|\\xi^k\\| \\right)^2,\r\n\t\\end{align*}\r\n\twhich along with $\\|\\xi^k\\| \\le e_k$ leads to \\eqref{eq:ippa-iter-cplx-min}. The proof is then completed.\r\n\\end{proof}","sha256_utf8":"347ce2a672af8b81f59fbf6721467f5af63e09d96d6bcb5baa27d4f301f4acb6"}]},{"id":"S027","semantic_type":"lemma","kind":"lemma","title":"正则增广法的近端误差","source_units":[{"start":58478,"end":58867,"start_line":742,"end_line":750,"raw_tex":"\\begin{lemma}\r\n\\label{lemma:mi-al=i-ppa}\r\nLet $\\{(x^k,\\lambda^k)\\}$ be generated by Algorithm~\\ref{alg:mi-al}. For any $k\\geq 0$, one has\r\n\\begin{equation}\r\n\\label{eq:mi-al=i-ppa}\r\n\\|(x^{k+1},\\lambda^{k+1}) - \\mcJ_{\\rho_k}(x^k,\\lambda^k)\\| \\leq \\rho_k\\eta_k,\r\n\\end{equation}\r\nwhere $\\mcJ_{\\rho_k}=(\\mcI + \\rho_k\\mcT_l)^{-1}$ and $\\mcT_l$ is defined in \\eqref{eq:def-two-opt2}.\r\n\\end{lemma}","sha256_utf8":"8f23a52cb2c1568c7deaf987a0c82d268a3430b46bfae5a1aeabf65461b75570"}]},{"id":"S028","semantic_type":"proof","kind":"proof","title":"近端误差完整证明","source_units":[{"start":58871,"end":60278,"start_line":752,"end_line":770,"raw_tex":"\\begin{proof}\r\nBy Lemma~\\ref{lemma:al-func-elp} and $\\lambda^{k+1} = \\Pi_{\\mcK^*}\\big(\\lambda^k + \\rho_k g(x^{k+1})\\big)$, one has\r\n\\begin{equation}\r\n\\label{eq:diff-x-lambda-m}\r\n\\partial_x \\mcL(x^{k+1},\\lambda^k;\\rho_k) = \\partial_x l(x^{k+1},\\lambda^{k+1}), \\quad \\frac{1}{\\rho_k}(\\lambda^{k+1} - \\lambda^k) \\in \\partial_\\lambda l (x^{k+1},\\lambda^{k+1}).\r\n\\end{equation}\r\nBy~\\eqref{eq:inext-subprob-m}, there exists $\\|v\\|\\leq \\eta_k$ such that\r\n$$ v \\in \\partial_x \\mcL(x^{k+1},\\lambda^k;\\rho_k) + \\frac{1}{\\rho_k}(x^{k+1} - x^k). $$\r\nThis together with~\\eqref{eq:diff-x-lambda-m} implies that\r\n\\begin{equation}\r\n\\label{eq:diff-x-lambda-m-v}\r\nx^k + \\rho_kv \\in \\rho_k \\partial_x l(x^{k+1},\\lambda^{k+1}) +  x^{k+1}, \\quad \\lambda^k \\in -\\rho_k\\partial_\\lambda l(x^{k+1}, \\lambda^{k+1}) + \\lambda^{k+1},\r\n\\end{equation}\r\nwhich, by the definition of $\\mcT_l$, are equivalent to \r\n$$ (x^k + \\rho_kv, \\lambda^k) \\in (\\mcI + \\rho_k\\mcT_l)(x^{k+1},\\lambda^{k+1}). $$\r\nIt follows from this and $\\mcJ_{\\rho_k}=(\\mcI + \\rho_k\\mcT_l)^{-1}$ that  $(x^{k+1},\\lambda^{k+1}) = \\mcJ_{\\rho_k}(x^k+\\rho_kv,\\lambda^k)$. By this and the non-expansion of $\\mcJ_{\\rho_k}$, we obtain\r\n$$ \\|(x^{k+1}, \\lambda^{k+1}) - \\mcJ_{\\rho_k}(x^k,\\lambda^k)\\| = \\|\\mcJ_{\\rho_k}(x^k + \\rho_kv, \\lambda^k) - \\mcJ_{\\rho_k}(x^k,\\lambda^k)\\| \\leq \\|\\rho_kv\\| \\leq \\rho_k\\eta_k, $$\r\nwhich yields \\eqref{eq:mi-al=i-ppa} as desired. \r\n\\end{proof}","sha256_utf8":"fba097800bf79eaaa75ef74f88f4f724e4eabe3f194d7d0262def73fe6e3959d"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"乘子更新的次微分关系","source_units":[{"start":58871,"end":59245,"start_line":752,"end_line":757,"raw_tex":"\\begin{proof}\r\nBy Lemma~\\ref{lemma:al-func-elp} and $\\lambda^{k+1} = \\Pi_{\\mcK^*}\\big(\\lambda^k + \\rho_k g(x^{k+1})\\big)$, one has\r\n\\begin{equation}\r\n\\label{eq:diff-x-lambda-m}\r\n\\partial_x \\mcL(x^{k+1},\\lambda^k;\\rho_k) = \\partial_x l(x^{k+1},\\lambda^{k+1}), \\quad \\frac{1}{\\rho_k}(\\lambda^{k+1} - \\lambda^k) \\in \\partial_\\lambda l (x^{k+1},\\lambda^{k+1}).\r\n\\end{equation}\r\n","sha256_utf8":"53a64fc6667cf98a432897149a2107a113b3406fc256d7f8fc997580595accaf"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"子问题残差进入鞍点包含","source_units":[{"start":59245,"end":59694,"start_line":758,"end_line":764,"raw_tex":"By~\\eqref{eq:inext-subprob-m}, there exists $\\|v\\|\\leq \\eta_k$ such that\r\n$$ v \\in \\partial_x \\mcL(x^{k+1},\\lambda^k;\\rho_k) + \\frac{1}{\\rho_k}(x^{k+1} - x^k). $$\r\nThis together with~\\eqref{eq:diff-x-lambda-m} implies that\r\n\\begin{equation}\r\n\\label{eq:diff-x-lambda-m-v}\r\nx^k + \\rho_kv \\in \\rho_k \\partial_x l(x^{k+1},\\lambda^{k+1}) +  x^{k+1}, \\quad \\lambda^k \\in -\\rho_k\\partial_\\lambda l(x^{k+1}, \\lambda^{k+1}) + \\lambda^{k+1},\r\n\\end{equation}\r\n","sha256_utf8":"0c45b5506b6a46639d101866bb415830e2c486c5a6dd82dd423f5b666915ede8"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"预解算子非扩张控制误差","source_units":[{"start":59694,"end":60278,"start_line":765,"end_line":770,"raw_tex":"which, by the definition of $\\mcT_l$, are equivalent to \r\n$$ (x^k + \\rho_kv, \\lambda^k) \\in (\\mcI + \\rho_k\\mcT_l)(x^{k+1},\\lambda^{k+1}). $$\r\nIt follows from this and $\\mcJ_{\\rho_k}=(\\mcI + \\rho_k\\mcT_l)^{-1}$ that  $(x^{k+1},\\lambda^{k+1}) = \\mcJ_{\\rho_k}(x^k+\\rho_kv,\\lambda^k)$. By this and the non-expansion of $\\mcJ_{\\rho_k}$, we obtain\r\n$$ \\|(x^{k+1}, \\lambda^{k+1}) - \\mcJ_{\\rho_k}(x^k,\\lambda^k)\\| = \\|\\mcJ_{\\rho_k}(x^k + \\rho_kv, \\lambda^k) - \\mcJ_{\\rho_k}(x^k,\\lambda^k)\\| \\leq \\|\\rho_kv\\| \\leq \\rho_k\\eta_k, $$\r\nwhich yields \\eqref{eq:mi-al=i-ppa} as desired. \r\n\\end{proof}","sha256_utf8":"8d1442301fefd29ca5740e4bb3e24687ee3d07ec866fd992b59bf3724f4ad66e"}]},{"id":"S032","semantic_type":"assumption","kind":"assumption","title":"渐近终止的参数条件","source_units":[{"start":47236,"end":47638,"start_line":580,"end_line":584,"raw_tex":"\tSuppose that $\\{\\rho_k\\}$  and $\\{\\eta_k\\}$ satisfy that\r\n\t\\beq \\label{rho-eta-m}\r\n\t\\rho_k>0 \\ \\mbox{is nondecreasing}, \\quad 0<\\eta_k \\downarrow 0, \\quad \r\n\t\\frac{\\sum_{i=0}^{2k}\\rho_i\\eta_i}{\\rho_k\\sqrt{k+1}} \\to 0.  \\footnote{For example,  $\\rho_k = \\rho_0\\alpha^k$ and $\\eta_k = \\eta_0\\beta^k$ satisfy \\eqref{rho-eta-m} for any $\\rho_0>0$, $\\eta_0>0$, $\\alpha>1$ and $0<\\beta<1/\\alpha$.}\r\n\t\\eeq \r\n","sha256_utf8":"b65f74d8dc0a7f651bce6f6a320689a7d21d59972158641c8246e7846c4a0803"}]},{"id":"S033","semantic_type":"assumption","kind":"assumption","title":"有限外层次数的指标条件","source_units":[{"start":47723,"end":47971,"start_line":586,"end_line":590,"raw_tex":"\t\\item[(iii)] Furthermore, if $N$ is a nonnegative integer such that\r\n\t\\begin{equation}\r\n\t\\label{eq:cplx-out-general-m}\r\n\t\\frac{D + \\sum_{k=0}^N \\rho_k\\eta_k}{\\rho_N} \\leq \\frac{\\epsilon}{2}, \\quad \\eta_N \\leq \\frac{\\epsilon}{2},\r\n\t\\end{equation}\r\n","sha256_utf8":"bcd548cfc177c71fd9a64f9f12c0b38fd88d762854c437d99087ee3af6cbae9f"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"停止条件保证近似KKT残差","source_units":[{"start":84122,"end":85490,"start_line":1011,"end_line":1023,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{prop:output-gantee-m}]\r\n(i) Suppose that Algorithm~\\ref{alg:mi-al} terminates in Step 3 at some iteration $k$. It then follows \r\nthat $(x^+,\\lambda^+)$  is already an $\\epsilon$-KKT solution of problem \\eqref{eq:conic-cp} \r\nor  the inequalities \\eqref{eq:check-stop-m} hold for such $k$.  We next show that for the latter case, $(x^+, \\lambda^+) = (x^{k+1},\\lambda^{k+1})$ is also an $\\epsilon$-KKT solution of~\\eqref{eq:conic-cp}. To this end, suppose that \\eqref{eq:check-stop-m} holds at the $k$th iteration. Notice that \\eqref{eq:diff-x-lambda-m-v} holds for some $\\|v\\|\\leq \\eta_k$, which yields  \r\n\\[\r\n\\frac{1}{\\rho_k}(x^k-x^{k+1}) + v \\in \\partial_x l(x^{k+1},\\lambda^{k+1}), \\quad \\frac{1}{\\rho_k}(\\lambda^{k+1} - \\lambda^k) \\in \\partial_\\lambda l (x^{k+1},\\lambda^{k+1}).\r\n\\]\r\nBy these, \\eqref{eq:check-stop-m} and $(x^+, \\lambda^+) = (x^{k+1},\\lambda^{k+1})$, we obtain\r\n\\begin{align*}\r\n\\mbox{dist}(0,\\partial_x l(x^+, \\lambda^+)) & \\leq \\frac{1}{\\rho_k}\\|x^{k+1} - x^k - \\rho_kv\\| \\leq \\frac{1}{\\rho_k}\\|x^{k+1} - x^k\\| + \\|v\\| \\leq \\epsilon,\\\\\r\n\\mbox{dist}(0,\\partial_\\lambda l(x^+, \\lambda^+)) & \\leq \\frac{1}{\\rho_k}\\|\\lambda^{k+1} - \\lambda^k\\| \\leq \\epsilon,\r\n\\end{align*}\r\nwhich along with Definition \\ref{defi:approx-KKT} imply that $(x^+,\\lambda^+)$ is an $\\epsilon$-KKT solution of problem \\eqref{eq:conic-cp}.\r\n","sha256_utf8":"59ae118ed930a4c2e87e997e8deb859c330a07af1ed7979d495c93e255658861"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"区间最小位移推出有限终止","source_units":[{"start":85492,"end":86477,"start_line":1025,"end_line":1031,"raw_tex":"(ii) Suppose for contradiction that Algorithm~\\ref{alg:mi-al} does not terminate in a finite number of iterations. It then follows that  \\eqref{eq:check-stop-m} does not hold for any $k$. By \\eqref{eq:ippa-iter-cplx-min} and Lemma \\ref{lemma:mi-al=i-ppa}, one has that\r\n$$ \\min_{k \\leq i\\leq 2k} \\|(x^{i+1},\\lambda^{i+1}) - (x^i,\\lambda^i)\\| \\leq \\frac{\\sqrt{2}\\left(D + 2\\sum_{i=0}^{2k}\\rho_i\\eta_i\\right)}{\\sqrt{k+1}}, $$ \r\nwhere $D$ is defined in \\eqref{param-m}. Since $\\{\\rho_k\\}$ is assumed to be nondecreasing, we further have\r\n\\[\r\n\\min_{k \\leq i\\leq 2k} \\frac{1}{\\rho_i}\\|(x^{i+1},\\lambda^{i+1}) - (x^i,\\lambda^i)\\| \\leq \\frac{\\sqrt{2}\\left(D + 2\\sum_{i=0}^{2k}\\rho_i\\eta_i\\right)}{\\rho_k\\sqrt{k+1}}.\r\n\\]\r\nBy this and \\eqref{rho-eta-m}, one has that $\\min_{k \\leq i\\leq 2k} \\|(x^{i+1},\\lambda^{i+1}) - (x^i,\\lambda^i)\\|/\\rho_i \\to 0$ and $\\eta_k\\to 0$ as $k \\to \\infty$, which implies that \\eqref{eq:check-stop-m} is satisfied for some $k$ and thus leads to a contradiction. \r\n","sha256_utf8":"b2cd9c8872458b8a4549ae1e05a8f103bba8b8170bdd338db83ed550f15cb22a"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"单步位移界给出外层次数","source_units":[{"start":86479,"end":87317,"start_line":1033,"end_line":1039,"raw_tex":"(iii) Let $\\hat x^*$ and $\\hat\\lambda^*$ be defined right above \\eqref{param} and \\eqref{param-m}, respectively. Suppose for contradiction that Algorithm~\\ref{alg:mi-al} does not terminate within $N+1$ outer iterations. It then follows that \\eqref{eq:check-stop-m} does not hold for all $0\\le k \\le N$. Then, by \\eqref{param-m}, \\eqref{eq:ippa-iter-cplx}, and \\eqref{eq:mi-al=i-ppa}, we have that for all $0\\le k \\le N$,\r\n\\begin{equation*}\\label{eq:mi-ppa-bd}\r\n\\|(x^{k+1},\\lambda^{k+1}) - (x^k,\\lambda^k)\\| \\leq \\|(x^0,\\lambda^0) - (\\hat x^*, \\hat\\lambda^*)\\| + \\sum_{i=0}^k\\rho_i\\eta_i \\leq D + \\sum_{i=0}^k\\rho_i\\eta_i,\r\n\\end{equation*}\r\nwhere $D$ is given in \\eqref{param-m}. \r\nBy this and \\eqref{eq:cplx-out-general-m}, one can see that \\eqref{eq:check-stop-m} is satisfied when $k = N$, which leads to a contradiction.  \r\n\\end{proof}","sha256_utf8":"1e713c5ec0fead2a6ef00b71ece2a166d932aa52e0145a58995fda7bec340b02"}]},{"id":"S037","semantic_type":"external_fact","kind":"external_fact","title":"强对偶下的鞍点刻画","source_units":[],"declared_external":true,"declaration_text":["R97, Theorem 36.6: under primal and dual attainment and strong duality, primal and dual optimal solutions form a Lagrangian saddle pair, and saddle points are characterized by zero belonging to the saddle subdifferential."]},{"id":"S038","semantic_type":"external_fact","kind":"external_fact","title":"鞍点算子的极大单调性","source_units":[],"declared_external":true,"declaration_text":["For a proper closed convex-concave saddle function l, its signed saddle subdifferential T_l(x,lambda)={(v,u):(v,-u) belongs to partial l(x,lambda)} is maximally monotone."]},{"id":"S039","semantic_type":"external_fact","kind":"external_fact","title":"预解算子的非扩张与下降","source_units":[],"declared_external":true,"declaration_text":["For a maximally monotone operator T and rho>0, J_rho=(I+rho T)^(-1) is everywhere defined, single valued and nonexpansive. If 0 belongs to T(z*), then ||J_rho(z)-z*||^2+||J_rho(z)-z||^2 <= ||z-z*||^2; the latter inequality is cited from Rock76a, Proposition 1."]},{"id":"S040","semantic_type":"external_fact","kind":"external_fact","title":"对偶锥法锥的互补刻画","source_units":[],"declared_external":true,"declaration_text":["For a closed convex cone K and lambda in K*, y belongs to N_{K*}(lambda) if and only if y belongs to -K and <lambda,y>=0."]},{"id":"S041","semantic_type":"external_fact","kind":"external_fact","title":"标准增广拉格朗日包络性质","source_units":[],"declared_external":true,"declaration_text":["The standard convex augmented-Lagrangian identities recorded in B0075 and attributed to Rock76b are admitted as a cited convex-analysis boundary fact: L(x,lambda;rho)=max_eta{l(x,eta)-||eta-lambda||^2/(2rho)}, uniquely maximized at bar_lambda=Pi_{K*}(lambda+rho g(x)) when x belongs to dom(P); d(lambda;rho)=max_eta{d(eta)-||eta-lambda||^2/(2rho)}; L is convex in x and concave in lambda; partial_x L(x,lambda;rho)=partial_x l(x,bar_lambda); and gradient_lambda L(x,lambda;rho)=(bar_lambda-lambda)/rho belongs to partial_lambda l(x,bar_lambda)."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S006","kind":"supports"},{"source_id":"S003","target_id":"S007","kind":"supports"},{"source_id":"S004","target_id":"S017","kind":"supports"},{"source_id":"S004","target_id":"S035","kind":"supports"},{"source_id":"S004","target_id":"S036","kind":"supports"},{"source_id":"S005","target_id":"S008","kind":"supports"},{"source_id":"S005","target_id":"S009","kind":"supports"},{"source_id":"S005","target_id":"S010","kind":"supports"},{"source_id":"S006","target_id":"S008","kind":"supports"},{"source_id":"S006","target_id":"S018","kind":"supports"},{"source_id":"S007","target_id":"S010","kind":"supports"},{"source_id":"S007","target_id":"S018","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S017","kind":"supports"},{"source_id":"S008","target_id":"S031","kind":"supports"},{"source_id":"S008","target_id":"S035","kind":"supports"},{"source_id":"S008","target_id":"S036","kind":"supports"},{"source_id":"S009","target_id":"S034","kind":"supports"},{"source_id":"S010","target_id":"S029","kind":"supports"},{"source_id":"S010","target_id":"S030","kind":"supports"},{"source_id":"S010","target_id":"S034","kind":"supports"},{"source_id":"S010","target_id":"S035","kind":"supports"},{"source_id":"S010","target_id":"S036","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S014","kind":"supports"},{"source_id":"S012","target_id":"S036","kind":"supports"},{"source_id":"S013","target_id":"S036","kind":"supports"},{"source_id":"S014","target_id":"S035","kind":"supports"},{"source_id":"S014","target_id":"S036","kind":"supports"},{"source_id":"S015","target_id":"S035","kind":"supports"},{"source_id":"S015","target_id":"S036","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S029","kind":"supports"},{"source_id":"S019","target_id":"S035","kind":"supports"},{"source_id":"S019","target_id":"S036","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S023","kind":"supports"},{"source_id":"S021","target_id":"S024","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S025","kind":"supports"},{"source_id":"S022","target_id":"S020","kind":"supports"},{"source_id":"S023","target_id":"S020","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S020","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S020","kind":"supports"},{"source_id":"S026","target_id":"S020","kind":"supports"},{"source_id":"S027","target_id":"S035","kind":"supports"},{"source_id":"S027","target_id":"S036","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S034","kind":"supports"},{"source_id":"S030","target_id":"S028","kind":"supports"},{"source_id":"S031","target_id":"S028","kind":"supports"},{"source_id":"S032","target_id":"S035","kind":"supports"},{"source_id":"S033","target_id":"S036","kind":"supports"},{"source_id":"S034","target_id":"S002","kind":"supports"},{"source_id":"S035","target_id":"S002","kind":"supports"},{"source_id":"S036","target_id":"S002","kind":"supports"},{"source_id":"S037","target_id":"S017","kind":"supports"},{"source_id":"S038","target_id":"S031","kind":"supports"},{"source_id":"S038","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S039","target_id":"S021","kind":"supports"},{"source_id":"S039","target_id":"S031","kind":"supports"},{"source_id":"S040","target_id":"S017","kind":"supports"},{"source_id":"S041","target_id":"S018","kind":"supports"}]}}
{"record_id":"1807.00387v3","paper":{"title":"Convergence rates for an inertial algorithm of gradient type associated to a smooth nonconvex minimization","authors":["Szilárd Csaba László"],"arxiv_id":"1807.00387","version":"v3","journal":"Mathematical Programming","doi":"10.1007/s10107-020-01534-w","abstract_url":"https://arxiv.org/abs/1807.00387v3","pdf_url":"https://arxiv.org/pdf/1807.00387v3"},"context":{"raw_tex":"Let $g:\\R^m\\To \\R$ be a (not necessarily convex) Fr\\'{e}chet differentiable function with $L_g$-Lipschitz continuous gradient, i.e. there exists $L_g\\ge 0$ such that $\\|\\n g(x)-\\n g(y)\\|\\le L_g\\|x-y\\|$ for all $x,y\\in \\R^m.$ We deal with the optimization problem\r\n\\begin{equation}\\label{opt-pb} (P) \\ \\inf_{x\\in\\R^m}g(x). \\end{equation}\n\nWe associate to \\eqref{opt-pb} the following inertial algorithm of gradient type.\r\nConsider the starting points $x_0=y_0\\in\\R^m,$ and for all $n\\in\\N$\r\n\\begin{equation}\\label{generaldiscrete}\\left\\{\\begin{array}{lll}\r\n\\ds x_{n+1}=y_n-s\\n g(y_n),\\\\\r\n\\\\\r\n\\ds y_n=x_n+\\frac{\\b n}{n+\\a}(x_n-x_{n-1}),\r\n\\end{array}\\right.\r\n\\end{equation}\n\nwhere $\\a>0,\\b\\in(0,1)$ and $0<s<\\frac{2(1-\\b)}{L_g}.$\r\n\n\n\\begin{remark}\\label{r1} Observe that conclusion (iii) in the hypotheses of Theorem \\ref{decreasing} assures that the sequence  $(x_n-x_{n-1})_{n\\in\\N}\\in l^2,$ in particular that\r\n\\begin{equation}\\label{e7}\r\n\\lim_{n\\To+\\infty}(x_n-x_{n-1})=0.\r\n\\end{equation}\r\n\\end{remark}\n\nLet us denote by $\\omega((x_n)_{n\\in\\N})$ the set of cluster points of the sequence $(x_n)_{n\\in\\N},$ and denote by $\\crit(g)=\\{x\\in\\R^m: \\nabla g(x)=0\\}$ the set of critical points of $g$.\r\n\n\nIn the following result we use the distance function to a set, defined for $A\\subseteq\\R^n$ as $\\dist(x,A)=\\inf_{y\\in A}\\|x-y\\|$ for all $x\\in\\R^n$.\r\n\n\nIn order to continue our analysis we need the concept of a KL function. For $\\eta\\in(0,+\\infty]$, we denote by $\\Theta_{\\eta}$\r\nthe class of concave and continuous functions $\\varphi:[0,\\eta)\\rightarrow [0,+\\infty)$ such that $\\varphi(0)=0$, $\\varphi$ is\r\ncontinuously differentiable on $(0,\\eta)$, continuous at $0$ and $\\varphi'(s)>0$ for all\r\n$s\\in(0, \\eta)$.\r\n\n\n\\begin{definition}\\label{KL-property} \\rm({\\it Kurdyka-\\L{}ojasiewicz property}) Let $f:\\R^n\\rightarrow \\R$ be a differentiable function. We say that $f$ satisfies the {\\it Kurdyka-\\L{}ojasiewicz (KL) property} at\r\n$\\ol x\\in \\R^n$\r\nif there exist $\\eta \\in(0,+\\infty]$, a neighborhood $U$ of $\\ol x$ and a function $\\varphi\\in \\Theta_{\\eta}$ such that for all $x$ in the\r\nintersection\r\n$$U\\cap \\{x\\in\\R^n: f(\\ol x)<f(x)<f(\\ol x)+\\eta\\}$$ the following inequality holds\r\n$$\\varphi'(f(x)-f(\\ol x))\\|\\n f(x))\\|\\geq 1.$$\r\nIf $f$ satisfies the KL property at each point in $\\R^n$, then $f$ is called a {\\it KL function}.\r\n\\end{definition}\n\n\\begin{definition}\\label{Lexpo}  Let $f:\\R^n\\To \\R $ be a differentiable function. The function $f$ is said to fulfill the {\\L}ojasiewicz property, if for every $\\ol x\\in\\crit{f}$ there exist $K,\\e>0$ and $\\t\\in(0,1)$ such that\r\n$$|f(x)-f(\\ol x)|^\\t\\le K\\|\\n f(x)\\|\\mbox{ for every }x\\mbox{ fulfilling }\\|x-\\ol x\\|<\\e.$$\r\nThe number $\\t$  is called the {\\L}ojasiewicz exponent of $f$ at the critical  point $\\ol x.$ This corresponds to the case when the desingularizing function $\\varphi$ has the form $\\varphi(t)=\\frac{K}{1-\\t} t^{1-\\t}.$\r\n\\end{definition}","units":[{"start":4408,"end":4744,"start_line":127,"end_line":128,"raw_tex":"Let $g:\\R^m\\To \\R$ be a (not necessarily convex) Fr\\'{e}chet differentiable function with $L_g$-Lipschitz continuous gradient, i.e. there exists $L_g\\ge 0$ such that $\\|\\n g(x)-\\n g(y)\\|\\le L_g\\|x-y\\|$ for all $x,y\\in \\R^m.$ We deal with the optimization problem\r\n\\begin{equation}\\label{opt-pb} (P) \\ \\inf_{x\\in\\R^m}g(x). \\end{equation}","sha256_utf8":"2f2b98905578e0a821800348b06eaa05581b0163e9995d189eb0979bce753623"},{"start":4748,"end":5080,"start_line":130,"end_line":137,"raw_tex":"We associate to \\eqref{opt-pb} the following inertial algorithm of gradient type.\r\nConsider the starting points $x_0=y_0\\in\\R^m,$ and for all $n\\in\\N$\r\n\\begin{equation}\\label{generaldiscrete}\\left\\{\\begin{array}{lll}\r\n\\ds x_{n+1}=y_n-s\\n g(y_n),\\\\\r\n\\\\\r\n\\ds y_n=x_n+\\frac{\\b n}{n+\\a}(x_n-x_{n-1}),\r\n\\end{array}\\right.\r\n\\end{equation}","sha256_utf8":"c30cd3600fdcc9d758b573dd5682d3efeae9b2c7458bd9802c759424d8c6937b"},{"start":5082,"end":5138,"start_line":138,"end_line":138,"raw_tex":"where $\\a>0,\\b\\in(0,1)$ and $0<s<\\frac{2(1-\\b)}{L_g}.$\r\n","sha256_utf8":"ba76c73b9e7536bbc9de14bbf059f20f101d97b0f470c800d0baf25f7eac5044"},{"start":20504,"end":20777,"start_line":351,"end_line":355,"raw_tex":"\\begin{remark}\\label{r1} Observe that conclusion (iii) in the hypotheses of Theorem \\ref{decreasing} assures that the sequence  $(x_n-x_{n-1})_{n\\in\\N}\\in l^2,$ in particular that\r\n\\begin{equation}\\label{e7}\r\n\\lim_{n\\To+\\infty}(x_n-x_{n-1})=0.\r\n\\end{equation}\r\n\\end{remark}","sha256_utf8":"1ddb871a275fb63a6c30773876d36595aab9b86fde25fdd585d24c1584759003"},{"start":20783,"end":20974,"start_line":358,"end_line":358,"raw_tex":"Let us denote by $\\omega((x_n)_{n\\in\\N})$ the set of cluster points of the sequence $(x_n)_{n\\in\\N},$ and denote by $\\crit(g)=\\{x\\in\\R^m: \\nabla g(x)=0\\}$ the set of critical points of $g$.\r\n","sha256_utf8":"a4da268f9ba6654f8b20ff3911c883e9c4f0d6b230ed8cce7fdfc493ad1935b7"},{"start":21043,"end":21193,"start_line":361,"end_line":361,"raw_tex":"In the following result we use the distance function to a set, defined for $A\\subseteq\\R^n$ as $\\dist(x,A)=\\inf_{y\\in A}\\|x-y\\|$ for all $x\\in\\R^n$.\r\n","sha256_utf8":"5b5e7499f4db4e174235d1b1af383035842cc5675313dcf07e58981c5af08cc9"},{"start":28355,"end":28719,"start_line":455,"end_line":458,"raw_tex":"In order to continue our analysis we need the concept of a KL function. For $\\eta\\in(0,+\\infty]$, we denote by $\\Theta_{\\eta}$\r\nthe class of concave and continuous functions $\\varphi:[0,\\eta)\\rightarrow [0,+\\infty)$ such that $\\varphi(0)=0$, $\\varphi$ is\r\ncontinuously differentiable on $(0,\\eta)$, continuous at $0$ and $\\varphi'(s)>0$ for all\r\n$s\\in(0, \\eta)$.\r\n","sha256_utf8":"b2366286e5a0e73d810fbff8dfceda9fcdcaac69e989c7c693f27c8deb0f3462"},{"start":28721,"end":29354,"start_line":460,"end_line":467,"raw_tex":"\\begin{definition}\\label{KL-property} \\rm({\\it Kurdyka-\\L{}ojasiewicz property}) Let $f:\\R^n\\rightarrow \\R$ be a differentiable function. We say that $f$ satisfies the {\\it Kurdyka-\\L{}ojasiewicz (KL) property} at\r\n$\\ol x\\in \\R^n$\r\nif there exist $\\eta \\in(0,+\\infty]$, a neighborhood $U$ of $\\ol x$ and a function $\\varphi\\in \\Theta_{\\eta}$ such that for all $x$ in the\r\nintersection\r\n$$U\\cap \\{x\\in\\R^n: f(\\ol x)<f(x)<f(\\ol x)+\\eta\\}$$ the following inequality holds\r\n$$\\varphi'(f(x)-f(\\ol x))\\|\\n f(x))\\|\\geq 1.$$\r\nIf $f$ satisfies the KL property at each point in $\\R^n$, then $f$ is called a {\\it KL function}.\r\n\\end{definition}","sha256_utf8":"9dc8527e011ea292b99a209df0905d7fadecb2d91114330276a3eddefe0cf293"},{"start":38978,"end":39535,"start_line":610,"end_line":613,"raw_tex":"\\begin{definition}\\label{Lexpo}  Let $f:\\R^n\\To \\R $ be a differentiable function. The function $f$ is said to fulfill the {\\L}ojasiewicz property, if for every $\\ol x\\in\\crit{f}$ there exist $K,\\e>0$ and $\\t\\in(0,1)$ such that\r\n$$|f(x)-f(\\ol x)|^\\t\\le K\\|\\n f(x)\\|\\mbox{ for every }x\\mbox{ fulfilling }\\|x-\\ol x\\|<\\e.$$\r\nThe number $\\t$  is called the {\\L}ojasiewicz exponent of $f$ at the critical  point $\\ol x.$ This corresponds to the case when the desingularizing function $\\varphi$ has the form $\\varphi(t)=\\frac{K}{1-\\t} t^{1-\\t}.$\r\n\\end{definition}","sha256_utf8":"13c773abd73aab21c435567258d3b240df220353cdb3fe9e5990624a2cbd4697"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{th-conv-rate}  In the settings of problem \\eqref{opt-pb} consider the sequences $(x_n)_{n\\in\\N},\\,(y_n)_{n\\in\\N}$ generated by Algorithm \\eqref{generaldiscrete}. Assume that $g$ is bounded from below and that $(x_n)_{n\\in\\N}$ is bounded, let $\\ol x\\in\\crit(g)$ be such that $\\lim_{n\\To+\\infty}x_n=\\ol x$ and suppose that\r\n$$H:\\R^n\\times \\R^n\\To\\R,\\,H(x,y)=g(x)+\\frac{1}{2}\\|x-y\\|^2$$\r\nfulfills the {\\L}ojasiewicz property at $(\\ol x,\\ol x)\\in\\crit H$ with {\\L}ojasiewicz exponent  $\\t\\in\\left(0,\\frac12\\right].$ Then, for every $p>0$ there exist $a_1,a_2,a_3,a_4>0$ and $\\ol k\\in\\N$ such that the following statements hold true:\r\n\\begin{itemize}\r\n\\item[$(\\emph{a}_1)$] $g(y_{n})-g(\\ol x)\\le a_1 \\frac{1}{n^p}$ for  every $n> \\ol k$,\r\n\\item[$(\\emph{a}_2)$] $g(x_{n})-g(\\ol x)\\le a_2\\frac{1}{n^p}$ for  every $n>\\ol k$,\r\n\\item[$(\\emph{a}_3)$] $\\|x_n-\\ol x\\|\\le a_3 \\frac{1}{n^{\\frac{p}{2}}}$ for  every $n>\\ol k$,\r\n\\item[$(\\emph{a}_4)$] $\\|y_n-\\ol x\\|\\le a_4 \\frac{1}{n^{\\frac{p}{2}}}$ for all  $n>\\ol k$.\r\n\\end{itemize}\r\n\\end{theorem}","units":[{"start":39983,"end":41038,"start_line":617,"end_line":626,"raw_tex":"\\begin{theorem}\\label{th-conv-rate}  In the settings of problem \\eqref{opt-pb} consider the sequences $(x_n)_{n\\in\\N},\\,(y_n)_{n\\in\\N}$ generated by Algorithm \\eqref{generaldiscrete}. Assume that $g$ is bounded from below and that $(x_n)_{n\\in\\N}$ is bounded, let $\\ol x\\in\\crit(g)$ be such that $\\lim_{n\\To+\\infty}x_n=\\ol x$ and suppose that\r\n$$H:\\R^n\\times \\R^n\\To\\R,\\,H(x,y)=g(x)+\\frac{1}{2}\\|x-y\\|^2$$\r\nfulfills the {\\L}ojasiewicz property at $(\\ol x,\\ol x)\\in\\crit H$ with {\\L}ojasiewicz exponent  $\\t\\in\\left(0,\\frac12\\right].$ Then, for every $p>0$ there exist $a_1,a_2,a_3,a_4>0$ and $\\ol k\\in\\N$ such that the following statements hold true:\r\n\\begin{itemize}\r\n\\item[$(\\emph{a}_1)$] $g(y_{n})-g(\\ol x)\\le a_1 \\frac{1}{n^p}$ for  every $n> \\ol k$,\r\n\\item[$(\\emph{a}_2)$] $g(x_{n})-g(\\ol x)\\le a_2\\frac{1}{n^p}$ for  every $n>\\ol k$,\r\n\\item[$(\\emph{a}_3)$] $\\|x_n-\\ol x\\|\\le a_3 \\frac{1}{n^{\\frac{p}{2}}}$ for  every $n>\\ol k$,\r\n\\item[$(\\emph{a}_4)$] $\\|y_n-\\ol x\\|\\le a_4 \\frac{1}{n^{\\frac{p}{2}}}$ for all  $n>\\ol k$.\r\n\\end{itemize}\r\n\\end{theorem}","sha256_utf8":"61dd36983907c8351ffead76786d259c1e7c5868bde2b889850ec73eee436d86"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\\label{desc} Let $g:\\R^m\\To\\R$ be Fr\\`echet differentiable with $L_g$ Lipschitz continuous gradient. Then\r\n$$g(y)\\le g(x)+\\<\\n g(x),y-x\\>+\\frac{L_g}{2}\\|y-x\\|^2,\\,\\forall x,y\\in\\R^m.$$\r\n\\end{lemma}\n\n\\begin{theorem}\\label{decreasing} In the settings of problem \\eqref{opt-pb}, for some starting points $x_0=y_0\\in\\R^m$ let $(x_n)_{n\\in\\N},\\,(y_n)_{n\\in\\N}$ be the sequences generated by the numerical scheme \\eqref{generaldiscrete}. Consider the sequences\r\n $$A_{n-1}=\\frac{2-s L_g}{2s}\\left(\\frac{(1+\\b)n+\\a}{n+\\a}\\right)^2-\\frac{\\b n((1+\\b)n+\\a)}{s(n+\\a)^2},$$\r\n$$C_{n-1}=\\frac{2-s L_g}{2s}\\frac{\\b n-\\b}{n+\\a-1}\\frac{(1+\\b)n+\\a}{n+\\a}-\\frac{1}{2s}\\frac{\\b n-\\b}{n+\\a-1}\\frac{\\b n}{n+\\a}$$\r\nand\r\n$$\\d_n=A_{n-1}-C_{n-1}$$\r\nfor all $n\\in\\N,\\,n\\ge 1$.\r\n\r\n Then, there exists $N\\in\\N$ such that %for all $n\\ge N$ one has\r\n\\begin{itemize}\r\n\\item[(i)] The sequence $ \\left(g(y_{n})+\\d_{n}\\|x_{n}-x_{n-1}\\|^2\\right)_{n\\ge N}$ is decreasing and $\\d_n>0$ for all $n\\ge N$.\r\n\\end{itemize}\r\nAssume that $g$ is bounded from below. Then, the following statements hold.\r\n\\begin{itemize}\r\n\\item[(ii)] The sequence $\\left(g(y_{n})+\\d_{n}\\|x_{n}-x_{n-1}\\|^2\\right)_{n\\in \\N}$ is convergent;\r\n\\item[(iii)] $\\sum_{n\\ge 1}\\|x_n-x_{n-1}\\|^2<+\\infty.$\r\n\\end{itemize}\r\n\\end{theorem}\n\n\\begin{proof}\r\n\r\nFrom \\eqref{generaldiscrete} we have $\\n g(y_n)=\\frac{1}{s}(y_n-x_{n+1})$, hence\r\n$$\\<\\n g(y_n),y_{n+1}-y_n\\>=\\frac{1}{s}\\<y_n-x_{n+1},y_{n+1}-y_n\\>.$$\r\nNow, from Lemma \\ref{desc} we obtain\r\n$$g(y_{n+1})\\le g(y_n)+\\<\\n g(y_n),y_{n+1}-y_n\\>+\\frac{L_g}{2}\\|y_{n+1}-y_n\\|^2,$$\r\nconsequently we have\r\n\\begin{equation}\\label{e1}\r\ng(y_{n+1})-\\frac{L_g}{2}\\|y_{n+1}-y_n\\|^2\\le g(y_n)+ \\frac{1}{s}\\<y_n-x_{n+1},y_{n+1}-y_n\\>.\r\n\\end{equation}\r\n\r\nFurther,\r\n$$\\<y_n-x_{n+1},y_{n+1}-y_n\\>=-\\|y_{n+1}-y_n\\|^2+\\<y_{n+1}-x_{n+1},y_{n+1}-y_n\\>,$$\r\n and\r\n$$y_{n+1}-x_{n+1}=\\frac{\\b(n+1)}{n+\\a+1}(x_{n+1}-x_n),$$\r\nhence\r\n\\begin{equation}\\label{e2}\r\ng(y_{n+1})+\\left(\\frac{1}{s}-\\frac{L_g}{2}\\right)\\|y_{n+1}-y_n\\|^2\\le g(y_n)+ \\frac{\\frac{\\b(n+1)}{n+\\a+1}}{s}\\<x_{n+1}-x_n,y_{n+1}-y_n\\>.\r\n\\end{equation}\r\n\r\nSince $$y_{n+1}-y_n=\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}(x_{n+1}-x_n)-\\frac{\\b n}{n+\\a}(x_n-x_{n-1}),$$\r\nwe have,\r\n$$\\|y_{n+1}-y_n\\|^2=\\left\\|\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}(x_{n+1}-x_n)-\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right\\|^2=$$\r\n$$\\left(\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}\\right)^2\\|x_{n+1}-x_n\\|^2+\\left(\\frac{\\b n}{n+\\a}\\right)^2\\|x_n-x_{n-1}\\|^2-$$\r\n$$2\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}\\frac{\\b n}{n+\\a}\\<x_{n+1}-x_n,x_n-x_{n-1}\\>,$$\r\nand\r\n$$\\<x_{n+1}-x_n,y_{n+1}-y_n\\>=\\left\\<x_{n+1}-x_n,\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}(x_{n+1}-x_n)-\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right\\>=$$\r\n$$\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}\\|x_{n+1}-x_n\\|^2-\\frac{\\b n}{n+\\a}\\<x_{n+1}-x_n,x_n-x_{n-1}\\>.$$\r\n\r\nReplacing the above equalities in \\eqref{e2}, we obtain\r\n\r\n$$g(y_{n+1})+\\frac{(2-s L_g)\\left(\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}\\right)^2-2\\frac{\\b(n+1)((1+\\b)n+\\a+\\b+1)}{(n+\\a+1)^2}}{2s}\\|x_{n+1}-x_n\\|^2\\le$$\r\n$$g(y_n)-\\frac{(2-s L_g)\\left(\\frac{\\b n}{n+\\a}\\right)^2}{2s}\\|x_{n}-x_{n-1}\\|^2+$$\r\n$$\\frac{(2-s L_g)\\frac{\\b n}{n+\\a}\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}-\\frac{\\b n}{n+\\a}\\frac{\\b(n+1)}{n+\\a+1}}{s}\\<x_{n+1}-x_n,x_n-x_{n-1}\\>.$$\r\n\r\nFor simplicity let %$$A_{n}=\\frac{(2-s L_g)\\left(\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}\\right)^2-2\\frac{\\b(n+1)((1+\\b)n+\\a+\\b+1)}{(n+\\a+1)^2}}{2s},$$\r\n$$B_n=\\frac{(2-s L_g)\\left(\\frac{\\b n}{n+\\a}\\right)^2}{2s}$$\r\n%and $$C_n=\\frac{(2-s L_g)\\frac{\\b n}{n+\\a}\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}-\\frac{\\b n}{n+\\a}\\frac{\\b(n+1)}{n+\\a+1}}{2s}$$\r\nfor all $n\\in\\N.$\r\n\r\n Hence we have\r\n $$g(y_{n+1})+A_{n}\\|x_{n+1}-x_n\\|^2-2C_n\\<x_{n+1}-x_n,x_n-x_{n-1}\\>\\le g(y_n)-B_n\\|x_{n}-x_{n-1}\\|^2.$$\r\n\r\nBy using the equality\r\n\\begin{equation}\\label{e5}\r\n-2\\<x_{n+1}-x_n,x_n-x_{n-1}\\>=\\|x_{n+1}+x_{n-1}-2x_n\\|^2-\\|x_{n+1}-x_n\\|^2-\\|x_n-x_{n-1}\\|^2\r\n\\end{equation}\r\nwe obtain\r\n$$g(y_{n+1})+(A_n-C_n)\\|x_{n+1}-x_n\\|^2+C_n\\|x_{n+1}+x_{n-1}-2x_n\\|^2\\le g(y_n)+(C_n-B_n)\\|x_n-x_{n-1}\\|^2.$$\r\n\r\nNote that $A_{n}-C_n=\\d_{n+1}$ and let us denote $\\Delta_n=B_n+A_{n-1}-C_{n-1}-C_n$. Consequently the following inequality holds.\r\n\\begin{equation}\\label{e6}\r\nC_n\\|x_{n+1}+x_{n-1}-2x_n\\|^2+\\Delta_n\\|x_n-x_{n-1}\\|^2\\le (g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2)- (g(y_{n+1})+\\d_{n+1}\\|x_{n+1}-x_n\\|^2).\r\n\\end{equation}\r\n\r\n\r\nSince $0<\\b<1$ and $s<\\frac{2(1-\\b)}{L_g},$  we have\r\n$$\\lim_{n\\To+\\infty}A_n=\\frac{(2-s L_g)(\\b+1)^2-2\\b-2\\b^2}{2s}>0,$$\r\n$$\\lim_{n\\To+\\infty}B_n=\\frac{(2-s L_g)\\b^2}{2s}>0,$$\r\n$$\\lim_{n\\To+\\infty}C_n= \\frac{(2-s L_g)(\\b^2+\\b)-\\b^2}{2s} >0,$$\r\n$$\\lim_{n\\To+\\infty}\\Delta_n=\\frac{2-s L_g-2\\b}{2s}>0,$$\r\nand\r\n$$\\lim_{n\\To+\\infty}\\d_n=\\frac{2-\\b^2-s L_g(\\b+1)}{2s}>0.$$\r\n\r\nHence, there exists $N\\in\\N$ and $C>0,\\,D>0$ such that for all $n\\ge N$ one has\r\n$$C_n\\ge C,\\, \\Delta_n\\ge D\\mbox{ and }\\d_n>0$$ which, in the view of \\eqref{e6}, shows (i), that is,  the sequence $g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2$ is decreasing for $n\\ge N.$\r\n\r\nAssume now that $g$ is bounded from below. %Since $\\lim_{n\\To+\\infty}\\Delta_n>0$ there exists $D>0$ and $p\\in\\N$ such that $\\Delta_n\\ge D$ for all $n\\ge p.$\r\n By using \\eqref{e6} again, we obtain\r\n $$0\\le C\\|x_{n+1}+x_{n-1}-2x_n\\|^2+D\\|x_n-x_{n-1}\\|^2\\le (g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2)- (g(y_{n+1})+\\d_{n+1}\\|x_{n+1}-x_n\\|^2),$$\r\nfor all $n\\ge N,$ or more convenient, that\r\n \\begin{equation}\\label{ineq}\r\n0\\le D\\|x_n-x_{n-1}\\|^2\\le (g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2)- (g(y_{n+1})+\\d_{n+1}\\|x_{n+1}-x_n\\|^2),\r\n\\end{equation}\r\nfor all $n\\ge N.$ Let $r>N.$ By summing up the latter relation  we have\r\n$$\r\nD\\sum_{n=N}^r\\|x_n-x_{n-1}\\|^2\\le (g(y_N)+\\d_N\\|x_N-x_{N-1}\\|^2)-(g(y_{r+1})+\\d_{r+1}\\|x_{r+1}-x_r\\|^2)\r\n$$\r\nwhich leads to\r\n\\begin{equation}\\label{forcoercive}\r\ng(y_{r+1})+D\\sum_{n=N}^r\\|x_n-x_{n-1}\\|^2\\le g(y_N)+\\d_N\\|x_N-x_{N-1}\\|^2.\r\n\\end{equation}\r\nNow,  taking into account that $g$ is bounded from below, by letting $r\\To+\\infty$  we obtain\r\n$$\\sum_{n=N}^{\\infty}\\|x_n-x_{n-1}\\|^2\\le+\\infty$$ which proves (iii).\r\n\r\nThe latter relation also shows that\r\n$$\\lim_{n\\To+\\infty}\\|x_n-x_{n-1}\\|^2=0,$$\r\nhence $$\\lim_{n\\To+\\infty}\\d_n\\|x_n-x_{n-1}\\|^2=0.$$\r\nBut then, from the fact that $g$ is bounded from below we obtain that the sequence $g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2$ is bounded from below. On the other hand, from (i) we have that  the sequence $g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2$ is decreasing for $n\\ge N,$ hence there exists\r\n$$\\lim_{n\\To+\\infty} g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2\\in\\R.$$\r\n\\end{proof}\n\n\\begin{lemma}\\label{regularization} In the settings of problem \\eqref{opt-pb}, for some starting points $x_0=y_0\\in\\R^m$ consider the sequences $(x_n)_{n\\in\\N},\\,(y_n)_{n\\in\\N}$ generated by Algorithm \\eqref{generaldiscrete}. Assume that $g$ is bounded from below and consider the function\r\n$$H:\\R^m\\times\\R^m\\To\\R,\\,H(x,y)=g(x)+\\frac12\\|y-x\\|^2.$$\r\nConsider further the sequence\r\n$$u_n=\\sqrt{2\\d_n}(x_{n}-x_{n-1})+y_n\\mbox{, for all }n\\in \\N,$$\r\nwhere $\\d_n$ was defined in Theorem \\ref{decreasing}. Then, the following statements hold true.\r\n\\begin{itemize}\r\n\\item[(i)] $\\omega((u_n)_{n\\in\\N})=\\omega((y_n)_{n\\in\\N})=\\omega((x_n)_{n\\in\\N})\\subseteq\\crit g$;\r\n\\item[(ii)] There exists and is finite the limit $\\lim_{n\\To+\\infty}H(y_n,u_n)$;\r\n\\item[(iii)] $\\omega((y_n,u_n)_{n\\in\\N})\\subseteq\\crit H=\\{(x,x)\\in\\R^m\\times\\R^m:x\\in\\crit g\\}$;\r\n\\item[(iv)] $\\|\\n H(y_n,u_n)\\|\\le \\frac{1}{s}\\|x_{n+1}-x_n\\|+\\left(\\frac{\\b  n}{s(n+\\a)}+2\\sqrt{2\\d_n}\\right)\\|x_{n}-x_{n-1}\\|$ for all $n\\in\\N$;\r\n\\item[(v)]  $\\|\\n H(y_n,u_n)\\|^2\\le \\frac{2}{s^2}\\|x_{n+1}-x_n\\|^2+2\\left(\\left(\\frac{\\b n}{s(n+\\a)}-\\sqrt{2\\d_n}\\right)^2+\\d_n\\right)\\|x_{n}-x_{n-1}\\|^2$ for all $n\\in\\N$;\r\n\\item[(vi)] $H$ is finite and constant on $\\omega((y_n,u_n)_{n\\in\\N}).$\r\n\\end{itemize}\r\n\r\nAssume that $(x_n)_{n\\in\\N}$ is bounded. Then,\r\n\\begin{itemize}\r\n\\item[(vii)] $\\omega((y_n, u_n)_{n\\in\\N})$ is nonempty and compact;\r\n\\item[(viii)] $\\lim_{n\\To+\\infty} \\dist((y_n,u_n),\\omega((y_n,u_n)_{n\\in\\N}))=0.$\r\n\\end{itemize}\r\n\\end{lemma}\n\n\\begin{proof}\r\n(i) Let $\\ol x\\in\\omega((x_n)_{n\\in\\N}).$ Then, there exists a subsequence $(x_{n_k})_{k\\in \\N}$ of $(x_n)_{n\\in\\N}$ such that\r\n$$\\lim_{k\\to+\\infty}x_{n_k}=\\ol x.$$\r\nSince by \\eqref{e7} $\\lim_{n\\To+\\infty}(x_n-x_{n-1})=0$ and the sequences $(\\sqrt{2\\d_n})_{n\\in\\N},\\,\\left(\\frac{\\b n}{n+\\a}\\right)_{n\\in\\N}$ converge, we obtain that\r\n$$\\lim_{k\\to+\\infty}y_{n_k}=\\lim_{k\\to+\\infty}u_{n_k}=\\lim_{k\\to+\\infty}x_{n_k}=\\ol x,$$\r\nwhich shows that $$\\omega((x_n)_{n\\in\\N})\\subseteq \\omega((u_n)_{n\\in\\N})\\mbox{ and }\\omega((x_n)_{n\\in\\N})\\subseteq \\omega((y_n)_{n\\in\\N}).$$\r\nFurther from \\eqref{generaldiscrete}, the continuity of $\\n g$ and \\eqref{e7}, we obtain that\r\n$$\\n g(\\ol x)=\\lim_{k\\To +\\infty} \\n g(y_{n_k})=\\frac{1}{s}\\lim_{k\\To +\\infty}(y_{n_k}-x_{n_k+1})=$$\r\n$$\\frac{1}{s}\\lim_{k\\To +\\infty}\\left[(x_{n_k}-x_{n_k+1})+\\frac{\\b n_k}{n_k+\\a}(x_{n_k}-x_{n_k-1})\\right]=0.$$\r\nHence $\\omega((x_n)_{n\\in\\N})\\subseteq\\crit g.$\r\nConversely, if $\\ol u\\in \\omega((u_n)_{n\\in\\N})$ then, from \\eqref{e7} results that $\\ol u\\in \\omega((y_n)_{n\\in\\N})$ and $\\ol u\\in \\omega((x_n)_{n\\in\\N}).$\r\nHence,\r\n$$\\omega((u_n)_{n\\in\\N})=\\omega((y_n)_{n\\in\\N})=\\omega((x_n)_{n\\in\\N})\\subseteq\\crit g.$$\r\n\r\n(ii) is nothing else than (ii) in Theorem \\ref{decreasing}.\r\n\r\nFor (iii) observe that  $\\n H(x,y)=(\\n g(x)+x-y,y-x)$, hence, $\\n H(x,y)=0$ leads to\r\n$x=y$ and $\\n g(x)=0.$\r\nConsequently\r\n$$\\crit H=\\{(x,x)\\in\\R^m\\times\\R^m: x\\in\\crit g\\}.$$\r\n\r\nFurther, consider $(\\ol y,\\ol u)\\in \\omega((y_n,u_n)_{n\\in\\N}).$ Then, there exists $(y_{n_k},u_{n_k})_{k\\in\\N}\\subseteq (y_n,u_n)_{n\\in\\N}$ such that\r\n$$(\\ol y,\\ol u)=\\lim_{k\\To+\\infty}(y_{n_k},u_{n_k})=\\lim_{k\\To+\\infty}(x_{n_k},x_{n_k})=(\\ol x,\\ol x).$$\r\nHence, $\\ol u=\\ol y=\\ol x\\in\\omega((x_n)_{n\\in\\N})\\subseteq \\crit g$ and $(\\ol x,\\ol x)\\in\\crit H.$\r\n\r\n(iv) By using the 1-norm of $\\R^m\\times\\R^m$ and \\eqref{generaldiscrete}, for every $n\\in \\N$ we have\r\n $$\\|\\n H(y_n,u_n)\\|\\le\\|\\n H(y_n,u_n)\\|_1=\\|(\\n g(y_n)+y_n-u_n,u_n-y_n)\\|_1=\\|\\n g(y_n)+y_n-u_n\\|+\\|u_n-y_n\\|\\le$$\r\n $$\\|\\n g(y_n)\\|+2\\|\\sqrt{2\\d_n}(x_{n}-x_{n-1})\\|=$$\r\n $$\\frac{1}{s}\\left\\|\\left(x_n+\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right)-x_{n+1}\\right\\|+2\\sqrt{2\\d_n}\\|x_{n}-x_{n-1}\\|\\le$$\r\n $$\\frac{1}{s}\\|x_{n+1}-x_n\\|+\\left(\\frac{\\b  n}{s(n+\\a)}+2\\sqrt{2\\d_n}\\right)\\|x_{n}-x_{n-1}\\|.$$\r\n\r\n(v) We use the euclidian norm of $\\R^m\\times\\R^m$, that is $\\|(x,y)\\|=\\sqrt{\\|x\\|^2+\\|y\\|^2}$ for all $(x,y)\\in\\R^m\\times\\R^m.$\r\nWe have:\r\n$$\\|\\n H(y_n,u_n)\\|^2=\\|(\\n g(y_n)+y_n-u_n,u_n-y_n)\\|^2=\\|\\n g(y_n)+y_n-u_n\\|^2+\\|u_n-y_n\\|^2=$$\r\n$$\\left\\|\\frac1s(x_n-x_{n+1})+\\left(\\frac{\\b n}{s(n+\\a)}-\\sqrt{2\\d_n}\\right)(x_n-x_{n-1})\\right\\|^2+2\\d_n\\|x_n-x_{n-1}\\|^2\\le$$\r\n$$\\frac{2}{s^2}\\|x_{n+1}-x_n\\|^2+2\\left(\\left(\\frac{\\b n}{s(n+\\a)}-\\sqrt{2\\d_n}\\right)^2+\\d_n\\right)\\|x_{n}-x_{n-1}\\|^2$$ for all $n\\in\\N.$\r\n\r\n(vi) follows directly from (ii).\r\n\r\n Assume now that $(x_n)_{n\\in\\N}$ is bounded and let us prove (vii), (see also \\cite{BCL-AA}). Obviously $(y_n,u_n)_{n\\in\\N}$ is bounded, hence according to Weierstrass Theorem $\\omega((y_n, u_n)_{n\\in\\N}),$ (and also $\\omega((x_n)_{n\\in\\N})$), is nonempty. It remains to show that $\\omega((y_n, u_n)_{n\\in\\N})$ is closed. From (i) and the proof of (iii) we have\r\n \\begin{equation}\\label{e8}\r\n\\omega((y_n, u_n)_{n\\in\\N})=\\{(\\ol x,\\ol x)\\in\\R^m\\times \\R^m: \\ol x\\in \\omega((x_n)_{n\\in\\N})\\}.\r\n\\end{equation}\r\n Hence, it is enough to show that $\\omega((x_n)_{n\\in\\N})$ is closed.\r\n\r\n\r\n Let be $(\\ol x_p)_{p\\in\\N}\\subseteq \\omega((x_n)_{n\\in\\N})$ and assume that $\\lim_{p\\To+\\infty}\\ol x_p=x^*.$ We show that $x^*\\in\\omega((x_n)_{n\\in\\N}).$ Obviously, for every $p\\in\\N$ there exists a sequence of natural numbers $n_k^p\\To+\\infty,\\,k\\To+\\infty$, such that\r\n$$\\lim_{k\\To+\\infty}x_{n_k^p}=\\ol x_p.$$\r\n\r\nLet be $\\e >0$. Since $\\lim_{p\\To+\\infty}\\ol x_p=x^*,$  there exists $P(\\e) \\in \\N$ such that for every $p\\ge P(\\e)$ it holds\r\n$$\\|\\ol x_p-x^*\\|<\\frac\\e2.$$\r\nLet $p\\in\\N$ be fixed. Since $\\lim_{k\\To+\\infty}x_{n_k^p}=\\ol x_p,$ there exists $k(p,\\e)\\in\\N$ such that for every $k\\ge k(p,\\e)$ it holds $$\\|x_{n_k^p}-\\ol x_p\\|<\\frac\\e2.$$\r\nLet be $k_p \\geq k(p,\\varepsilon)$ such that $n_{k_p}^p >p$. Obviously $n_{k_p}^p \\To \\infty$ as $p \\To +\\infty$ and for every $p \\geq P(\\e)$\r\n$$\\|x_{n_{k_p}^p}-x^*\\|<\\e.$$\r\nHence\r\n$$\\lim_{p\\To+\\infty}x_{n_{k_p}^p}=x^*,$$\r\nthus $x^*\\in \\omega((x_n)_{n\\in\\N}).$\r\n\r\n(viii) By using \\eqref{e8} we have\r\n$$\\lim_{n\\To+\\infty} \\dist((y_n,u_n),\\omega((y_n,u_n)_{n\\in\\N}))=\\lim_{n\\To+\\infty}\\inf_{\\ol x\\in\\omega((x_n)_{n\\in\\N})}\\|(y_n,u_n)-(\\ol x,\\ol x)\\|.$$\r\nSince there exists the subsequences $(y_{n_k})_{k\\in\\N}$ and $(u_{n_k})_{k\\in\\N}$ such that\r\n$\\lim_{k\\To\\infty}y_{n_k}=\\lim_{k\\To\\infty}u_{n_k}=\\ol x_0\\in \\omega((x_n)_{n\\in\\N})$ it is straightforward that\r\n$$\\lim_{n\\To+\\infty} \\dist((y_n,u_n),\\omega((y_n,u_n)_{n\\in\\N}))=0.$$\r\n\\end{proof}\n\n\\begin{lemma}\\label{unif-KL-property} Let $\\Omega\\subseteq \\R^n$ be a compact set and let $f:\\R^n\\rightarrow \\R$ be a differentiable function. Assume that $f$ is constant on $\\Omega$ and $f$ satisfies the KL property at each point of $\\Omega$.\r\nThen there exist $\\varepsilon,\\eta >0$ and $\\varphi\\in \\Theta_{\\eta}$ such that for all $\\ol x\\in\\Omega$ and for all $x$ in the intersection\r\n\\begin{equation}\\label{int} \\{x\\in\\R^n: \\dist(x,\\Omega)<\\varepsilon\\}\\cap \\{x\\in\\R^n: f(\\ol x)<f(x)<f(\\ol x)+\\eta\\}\\end{equation}\r\nthe following inequality holds \\begin{equation}\\label{KL-ineq}\\varphi'(f(x)-f(\\ol x))\\|\\n f(x)\\|\\geq 1.\\end{equation}\r\n\\end{lemma}\n\n\\begin{theorem}\\label{convergence} In the settings of problem \\eqref{opt-pb}, for some starting points $x_0=y_0\\in\\R^m$ consider the sequences $(x_n)_{n\\in\\N},\\,(y_n)_{n\\in\\N}$ generated by Algorithm \\eqref{generaldiscrete}. Assume that $g$ is bounded from below and consider the function\r\n$$H:\\R^m\\times\\R^m\\To\\R,\\,H(x,y)=g(x)+\\frac12\\|y-x\\|^2.$$\r\nAssume that $(x_n)_{n\\in\\N}$ is bounded and $H$ is a KL function. Then the following statements are true\r\n\\begin{itemize}\r\n\\item[(a)] $\\sum_{n\\ge 1}\\|x_n-x_{n-1}\\|<+\\infty;$\r\n\\item[(b)] there exists ${x}\\in\\crit(g)$ such that $\\lim_{n\\To+\\infty}x_n={x}.$\r\n\\end{itemize}\r\n\\end{theorem}\n\n\\begin{proof} Consider the sequence\r\n$$u_n=\\sqrt{2\\d_n}(x_{n}-x_{n-1})+y_n\\mbox{, for all }n\\in \\N,$$\r\nthat  was defined in the hypotheses of Lemma \\ref{regularization}. Furthermore, consider $(\\ol x,\\ol x)\\in\\omega((y_n,u_n)_{n\\in\\N}).$\r\n\r\nThen, according to Lemma \\ref{regularization}, the sequence $H(y_n,u_n)$ is decreasing for all $n\\ge N$,  where $N$ was defined in Theorem \\ref{decreasing}, further\r\n$$\\ol x\\in\\crit g\\mbox{ and  }\\lim_{n\\To+\\infty}H(y_n,u_n)=H(\\ol x,\\ol x).$$\r\nWe divide the proof into two cases.\r\n\r\n{\\bf Case I.} There exists $\\ol n\\ge N,\\,\\ol n\\in\\N,$  such that $H(y_{\\ol n},u_{\\ol n})=H(\\ol x,\\ol x).$ Then, since $H(y_n,u_n)$ is decreasing for all $n\\ge N$ and $\\lim_{n\\To+\\infty}H(y_n,u_n)=H(\\ol x,\\ol x)$ we obtain that\r\n$$H(y_n,u_n)=H(\\ol x,\\ol x)\\mbox{ for all }n\\ge\\ol n.$$\r\n\r\nThe latter relation combined with \\eqref{ineq} leads to\r\n$$0\\le D\\|x_n-x_{n-1}\\|^2\\le H(y_n,u_n)-H(y_{n+1},u_{n+1})=H(\\ol x,\\ol x)-H(\\ol x,\\ol x)=0$$\r\nfor all $n\\ge\\ol n.$\r\n\r\nHence $(x_n)_{n\\ge\\ol n}$ is constant and the conclusion follows.\r\n\r\n\r\n\r\n{\\bf Case II.} For every $n\\ge N$ one has that $H(y_n,u_n)> H(\\ol x,\\ol x).$\r\nLet $\\Omega=\\omega((y_n,u_n)_{n\\in\\N}).$ Then according to Lemma \\ref{regularization}, $\\Omega$ is nonempty and compact and $H$ is constant on $\\Omega.$\r\nSince $H$ is KL, according to Lemma \\ref{unif-KL-property}  there exist $\\varepsilon,\\eta >0$ and $\\varphi\\in \\Theta_{\\eta}$ such that for all  $(z,w)$ belonging to the intersection\r\n$$ \\{(z,w)\\in\\R^n\\times\\R^n: \\dist((z,w),\\Omega)<\\varepsilon\\}\\cap \\{(z,w)\\in\\R^n\\times\\R^n: H(\\ol x,\\ol x)<H(z,w)<H(\\ol x,\\ol x)+\\eta\\}$$\r\none has\r\n$$\\varphi'(H(z,w)-H(\\ol x,\\ol x))\\|\\n H(z,w)\\|\\geq 1.$$\r\n\r\nSince $\\lim_{n\\To+\\infty}\\dist((y_n,u_n),\\Omega)=0$, there exists $n_1\\in\\N$ such that\r\n$$\\dist((y_n,u_n),\\Omega)<\\e,\\,\\forall n\\ge n_1.$$\r\n\r\nSince $$\\lim_{n\\To+\\infty}H(y_n,u_n)=H(\\ol x,\\ol x)$$\r\nand\r\n$$H(y_n,u_n))> H(\\ol{x},\\ol{x})\\mbox{ for all }n\\ge N,$$\r\nthere exists $n_2\\ge N$ such that\r\n$$H(\\ol x,\\ol x)<H(y_n,u_n)< H(\\ol{x},\\ol{x})+\\eta,\\,\\forall n\\ge n_2.$$\r\n\r\nHence, for all $n\\ge \\ol n=\\max(n_1,n_2)$ we have\r\n$$\\varphi'(H(y_n,u_n)-H(\\ol{x},\\ol{x}))\\cdot\\|\\n H(y_n,u_n))\\|\\ge 1.$$\r\nSince $\\varphi$ is concave, for all $n\\in\\N$ we have\r\n$$\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x}))\\ge$$\r\n$$\\varphi'(H(y_n,u_n)-H(\\ol{x},\\ol{x}))\\cdot(H(y_n,u_n)-H(y_{n+1},u_{n+1})),$$\r\nhence,\r\n$$\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x}))\\ge \\frac{H(y_n,u_n)-H(y_{n+1},u_{n+1})}{\\|\\n H(y_n,u_n)\\|}$$\r\nfor all $n\\ge\\ol n.$\r\n\r\n Now, from \\eqref{ineq} and Lemma \\ref{regularization} (iv) we obtain\r\n \\begin{equation}\\label{e9}\r\n \\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x}))\\ge\r\n \\end{equation}\r\n $$\\frac{D\\|x_{n}-x_{n-1}\\|^2}{\\frac{1}{s}\\|x_{n+1}-x_n\\|+\\left(\\frac{\\b  n}{s(n+\\a)}+2\\sqrt{2\\d_n}\\right)\\|x_{n}-x_{n-1}\\|},$$\r\n for all $n\\ge\\ol n.$\r\nSince $\\lim_{n\\To+\\infty}\\d_n=\\frac{2-\\b^2-s L_g(\\b+1)}{2s}>0$ and $\\lim_{n\\To+\\infty}\\frac{\\b  n}{s(n+\\a)}=\\frac{\\b }{s}\\ge 0$ there exists $\\ol N\\in\\N,\\,\\ol N\\ge\\ol n$ and $M>0$ such that\r\n$$\\max\\left(\\frac1s,\\frac{\\b  n}{s(n+\\a)}+2\\sqrt{2\\d_n}\\right)\\le M,$$\r\nfor all $n\\ge \\ol N.$\r\nHence, \\eqref{e9} becomes\r\n \\begin{equation}\\label{e10}\r\n \\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x}))\\ge\r\n \\end{equation}\r\n $$\\frac{D\\|x_{n}-x_{n-1}\\|^2}{M(\\|x_{n+1}-x_n\\|+\\|x_{n}-x_{n-1}\\|)},$$\r\n for all $n\\ge\\ol N.$\r\n\r\nConsequently,\r\n$$\\|x_{n}-x_{n-1}\\|\\le$$\r\n$$\\sqrt{\\frac{M}{D}(\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x})))\\cdot(\\|x_{n+1}-x_n\\|+\\|x_{n}-x_{n-1}\\|)},$$\r\n for all $n\\ge\\ol N.$\r\n\r\nBy using the arithmetical-geometrical mean inequality we have\r\n$$\\sqrt{\\frac{M}{D}(\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x})))\\cdot(\\|x_{n+1}-x_n\\|+\\|x_{n}-x_{n-1}\\|)}\\le$$\r\n$$\\frac{\\|x_{n+1}-x_n\\|+\\|x_{n}-x_{n-1}\\|}{3}+\\frac{3M}{4D}(\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x})))$$\r\n for all $n\\ge\\ol N.$\r\nHence\r\n$$\\|x_{n}-x_{n-1}\\|\\le$$\r\n$$\\frac{\\|x_{n+1}-x_n\\|+\\|x_{n}-x_{n-1}\\|}{3}+\\frac{3M}{4D}(\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x})))$$\r\n for all $n\\ge\\ol N$ which leads to\r\n \\begin{equation}\\label{e11}\r\n 2\\|x_{n}-x_{n-1}\\|-\\|x_{n+1}-x_{n}\\|\\le\\frac{9M}{4D}(\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x})))\r\n \\end{equation}\r\n for all $n\\ge\\ol N.$ Let $P>\\ol N$. By summing up \\eqref{e11} from $\\ol N$ to $P$ we obtain\r\n $$\\sum_{n={\\ol N}}^P \\|x_{n}-x_{n-1}\\|\\le$$\r\n  $$-\\|x_{\\ol N}-x_{\\ol N-1}\\|+\\|x_{P+1}-x_P\\|+\\frac{9M}{4D}(\\varphi(H(y_{\\ol N},u_{\\ol N})-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{P+1},u_{P+1})-H(\\ol{x},\\ol{x}))).$$\r\nNow, by letting $P\\To+\\infty$ and using the fact that $\\varphi(0)=0$ and \\eqref{e7} we obtain that\r\n$$\\sum_{n={\\ol N}}^\\infty \\|x_{n}-x_{n-1}\\|\\le-\\|x_{\\ol N}-x_{\\ol N-1}\\|+\\frac{9M}{4D}\\varphi(H(y_{\\ol N},u_{\\ol N})-H(\\ol{x},\\ol{x}))<+\\infty,$$\r\nhence\r\n$$\\sum_{n\\ge 1}\\|x_n-x_{n-1}\\|<+\\infty$$ which is exactly (a).\r\n\r\nObviously the sequence $S_n=\\sum_{k=1}^n\\|x_k-x_{k-1}\\|$ is Cauchy, hence, for all $\\e>0$ there exists $N_{\\e}\\in \\N$   such that for all $n\\ge N_\\e$ and for all $p\\in \\N$ one has\r\n$$S_{n+p}-S_n\\le \\e.$$\r\nBut\r\n$$S_{n+p}-S_n=\\sum_{k={n+1}}^{n+p}\\|x_k-x_{k-1}\\|\\ge \\left\\|\\sum_{k={n+1}}^{n+p}(x_k-x_{k-1})\\right\\|=\\|x_{n+p}-x_n\\|$$\r\nhence the sequence $(x_n)_{n\\in\\N}$ is Cauchy, consequently is convergent. Let\r\n$$\\lim_{n\\To+\\infty}x_n=x.$$\r\nNow, according to Lemma \\ref{regularization} (i) one has\r\n$$\\{x\\}=\\omega((x_n)_{n\\in\\N})\\subseteq\\crit g$$ which proves (b).\r\n\\end{proof}\n\n\\begin{proof}\r\n As we have seen in the proof of Theorem \\ref{convergence},  if there exists $\\ol n\\ge N,\\,\\ol n\\in\\N$, (where $N$ was defined in Theorem \\ref{decreasing}), such that $H(y_{\\ol n},u_{\\ol n})=H(\\ol x,\\ol x),$ then,\r\n$$H(y_n,u_n)=H(\\ol x,\\ol x)\\mbox{ for all }n\\ge\\ol n$$\r\nand $(x_n)_{n\\ge\\ol n}$ is constant. Consequently $(y_n)_{n\\ge\\ol n}$ is also constant and the conclusion of the theorem is straightforward.\r\n\r\nHence, in what follows we assume  that $H(y_{n},u_{ n})>H(\\ol x,\\ol x),$ for all $n\\ge N.$\r\n\\\\\r\n\r\nLet us fix $p>0$ and let us  prove (a$_1$).\r\n\r\n For simplicity let us denote $r_n=H(y_n,u_n)-H(\\ol x,\\ol x)>0$  for all $n\\in\\N.$\r\n From \\eqref{e6} we have\r\n$$\\Delta_n\\|x_n-x_{n-1}\\|^2\\le r_n-r_{n+1}\\mbox{ for all }n\\ge N.$$\r\n\r\nFrom Lemma \\ref{regularization} (v) we have\r\n $$\\|\\n H(y_n,u_n)\\|^2\\le \\frac{2}{s^2}\\|x_{n+1}-x_n\\|^2+2\\left(\\left(\\frac{\\b n}{s(n+\\a)}-\\sqrt{2\\d_n}\\right)^2+\\d_n\\right)\\|x_{n}-x_{n-1}\\|^2$$ for all $n\\in\\N$.\r\n  Let $S_n=2\\left(\\left(\\frac{\\b n}{s(n+\\a)}-\\sqrt{2\\d_n}\\right)^2+\\d_n\\right),$ for all $n\\in\\N.$\r\n\r\nIt follows that, for all $n\\ge N$ one has\r\n$$\\|x_n-x_{n-1}\\|^2\\ge \\frac{1}{S_n}\\|\\n H(y_n,u_n)\\|^2-\\frac{2}{s^2S_n}\\|x_{n+1}-x_n\\|^2\\ge$$\r\n$$\\frac{1}{S_n}\\|\\n H(y_n,u_n)\\|^2-\\frac{2}{s^2S_n\\Delta_{n+1}}(r_{n+1}-r_{n+2}).$$\r\n\r\n\r\nNow by using the {\\L}ojasiewicz property of $H$ at $(\\ol x,\\ol x)\\in\\crit H$, and the fact that $\\lim_{n\\To+\\infty}(y_n,u_n)=(\\ol x,\\ol x)$,  we obtain that there exists  $K,\\e>0$ and $\\ol N_1\\in\\N,$ such that for all $n\\ge \\ol N_1$ one has\r\n$$\\|(y_n,u_n)-(\\ol x,\\ol x)\\|<\\e,$$\r\nconsequently\r\n\\begin{equation}\\label{e111} r_n-r_{n+1}\\ge \\frac{\\Delta_n}{S_n}\\|\\n H(y_n,u_n)\\|^2-\\frac{2\\Delta_n}{s^2S_n\\Delta_{n+1}}(r_{n+1}-r_{n+2})\\ge\r\n\\end{equation}\r\n$$\\frac{\\Delta_n}{K^2 S_n}r_n^{2\\t}-\\frac{2\\Delta_n}{s^2S_n\\Delta_{n+1}}(r_{n+1}-r_{n+2})\\ge$$\r\n$$\\frac{\\Delta_n}{K^2 S_n}r_{n+1}^{2\\t}-\\frac{2\\Delta_n}{s^2S_n\\Delta_{n+1}}(r_{n+1}-r_{n+2})=\\a_n r_{n+1}^{2\\t}-\\b_n(r_{n+1}-r_{n+2}),$$\r\nwhere $\\a_n= \\frac{\\Delta_n}{K^2 S_n}\\mbox{ and  }\\b_n=\\frac{2\\Delta_n}{s^2S_n\\Delta_{n+1}}.$\r\n\r\nIt is obvious that the sequences $(\\a_n)_{n\\ge\\ol N_1}$ and $(\\b_n)_{n\\ge\\ol N_1}$ are convergent, further\r\n$$\\lim_{n\\To+\\infty} \\a_n>0\\mbox{ and } \\lim_{n\\To+\\infty} \\b_n>0.$$\r\nNow, since $0<2\\t\\le1$ and $r_{n+1}\\To 0$, there exists $\\ol N_2\\in\\N,\\, \\ol N_2\\ge \\ol N_1,$ such that $r_{n+1}^{2\\t}\\ge r_{n+1}$ for all $n\\ge\\ol N_2.$\r\n\r\n$(*)$\\,\\,\\, Note that this implies that $0\\le r_n\\le 1$ for all $n\\ge\\ol N_2.$\r\n\r\nHence,\r\n$$r_n\\ge \\left(\\a_n-\\b_n+1\\right)r_{n+1}+\\b_nr_{n+2},\\mbox{ for all }n\\ge\\ol N_2.$$\r\n\r\nLet us define for every $n>\\ol N_2$ the sequence $\\Xi_n=\\frac{\\b_n n^p}{(n+1)^p-n^p}.$\r\nThen, since $p>0$ one has $\\lim_{n\\To+\\infty}\\Xi_n=+\\infty.$\r\n\r\nSince\r\n$$\\lim_{n\\To+\\infty}\\left(\\a_n-\\b_n+1\\right)-\\left(1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}-\\frac{\\b_{n-1}}{1+\\frac{\\b_{n}}{\\Xi_{n}}}\\right)=\\lim_{n\\To+\\infty}\\a_n>0,$$  there exists $\\ol k\\in\\N,\\,\\ol k\\ge\\ol N_2$ such that for all $n\\ge\\ol k$ one has\r\n$$\\a_n-\\b_n+1\\ge 1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}-\\frac{\\b_{n-1}}{1+\\frac{\\b_{n}}{\\Xi_{n}}}.$$\r\n\r\nConsequently,\r\n$$r_n\\ge \\left(1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}-\\frac{\\b_{n-1}}{1+\\frac{\\b_{n}}{\\Xi_{n}}}\\right)r_{n+1}+\\b_nr_{n+2},\\mbox{ for all }n\\ge\\ol k,$$\r\nor, equivalently\r\n\\begin{equation}\\label{e12}\r\nr_n+\\frac{\\b_{n-1}}{1+\\frac{\\b_{n}}{\\Xi_{n}}}r_{n+1}\\ge \\left(1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}\\right)\\left(r_{n+1}+\\frac{\\b_n}{1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}}r_{n+2}\\right),\r\n\\end{equation}\r\nfor all $n\\ge\\ol k.$\r\nNow \\eqref{e12} leads to\r\n$$\\prod_{k=\\ol k}^n\\left( r_k+\\frac{\\b_{k-1}}{1+\\frac{\\b_{k}}{\\Xi_k}}r_{k+1}\\right)\\ge \\prod_{k=\\ol k}^n\\left(1+\\frac{\\b_{k+1}}{\\Xi_{k+1}}\\right)\\prod_{k=\\ol k}^n\\left(r_{k+1}+\\frac{\\b_k}{1+\\frac{\\b_{k+1}}{\\Xi_{k+1}}}r_{k+2}\\right),$$\r\nhence after simplifying we get\r\n\\begin{equation}\\label{egen}\\left(r_{\\ol k}+\\frac{\\b_{\\ol k-1}}{1+\\frac{\\b_{\\ol k}}{\\Xi_{\\ol k}}}r_{\\ol k+1}\\right)\\prod_{k=\\ol k}^n\\frac{1}{1+\\frac{\\b_{k+1}}{\\Xi_{k+1}}}\\ge\r\nr_{n+1}+\\frac{\\b_n}{1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}}r_{n+2}.\r\n\\end{equation}\r\n\r\nBut, $\\frac{\\b_{n+1}}{\\Xi_{n+1}}=\\frac{(n+2)^p}{(n+1)^p}-1,$ hence\r\n$$\\prod_{k=\\ol k}^n\\frac{1}{1+\\frac{\\b_{k+1}}{\\Xi_{k+1}}}=\\prod_{k=\\ol k}^n\\frac{(k+1)^p}{(k+2)^p}=\\frac{(\\ol k+1)^p}{(n+2)^p}.$$\r\nBy denoting $\\left(r_{\\ol k}+\\frac{\\b_{\\ol k-1}}{1+\\frac{\\b_{\\ol k}}{\\Xi_{\\ol k}}}r_{\\ol k+1}\\right)(\\ol k+1)^p=a_1$,  we have\r\n$$a_1 \\frac{1}{(n+2)^p}\\ge r_{n+1}+\\frac{\\b_n}{1+\\frac{1}{n+1}\\b_{n+1}}r_{n+2}.$$\r\n\r\nHence,\r\n\\begin{equation}\\label{e13}\r\na_1 \\frac{1}{n^p}\\ge a_1 \\frac{1}{(n+1)^p}\\ge r_n= g(y_n)-g(\\ol x)+\\d_n\\|x_n-x_{n-1}\\|^2\\ge g(y_n)-g(\\ol x)\r\n\\end{equation}\r\nwhich is (a$_1$).\r\n\r\nFor (a$_2$) we start from   Lemma \\ref{desc} and \\eqref{generaldiscrete} and we have\r\n$$g(x_n)-g(y_n)\\le\\<\\n g(y_n),x_n-y_n\\>+\\frac{L_g}{2}\\|x_n-y_n\\|^2=$$\r\n$$\\frac1s\\left\\<(x_n-x_{n+1})+\\frac{\\b n}{n+\\a}(x_n-x_{n-1}),-\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right\\>+\\frac{L_g}{2}\\left(\\frac{\\b n}{n+\\a}\\right)^2\\|x_n-x_{n-1}\\|^2= $$\r\n$$-\\left(\\frac{\\b n}{n+\\a}\\right)^2\\frac{2-sL_g}{2s}\\|x_n-x_{n-1}\\|^2+\\frac1s\\left\\<x_{n+1}-x_n,\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right\\>.$$\r\nBy using the inequality $\\<X,Y\\>\\le \\frac12\\left(a^2\\|X\\|^2+\\frac{1}{a^2}\\|Y\\|^2\\right)$ for all $X,Y\\in\\R^m, a\\in \\R\\setminus\\{0\\}$, we obtain\r\n$$\\left\\<x_{n+1}-x_n,\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right\\>\\le \\frac12\\left(\\frac{1}{2-sL_g}\\|x_{n+1}-x_n\\|^2+(2-sL_g)\\left(\\frac{\\b n}{n+\\a}\\right)^2\\|x_n-x_{n-1}\\|^2\\right),$$ consequently\r\n$$g(x_n)-g(y_n)\\le\\frac{1}{2s(2-sL_g)}\\|x_{n+1}-x_n\\|^2.$$\r\n\r\nFrom \\eqref{ineq} we have\r\n$$\\|x_{n}-x_{n-1}\\|^2\\le \\frac{1}{{D}}((g(y_{n})+\\d_{n}\\|x_{n}-x_{n-1}\\|^2)- (g(y_{n+1})+\\d_{n+1}\\|x_{n+1}-x_{n}\\|^2))$$ and since the sequence $(g(y_{n})+\\d_{n}\\|x_{n+1}-x_{n}\\|^2)_{n\\ge \\ol k}$ is decreasing and has the limit $g(\\ol x)$, we obtain that $g(y_{n+1})+\\d_{n+1}\\|x_{n+1}-x_{n}\\|^2\\ge g(\\ol x)$, consequently\r\n\\begin{equation}\\label{ee}\r\n\\|x_{n}-x_{n-1}\\|^2\\le\\frac{1}{{D}}{r_{n}}.\r\n\\end{equation}\r\n\r\nHence, for all $n\\ge\\ol k$ one has\r\n\\begin{equation}\\label{ee1}\r\ng(x_n)-g(y_n)\\le\\frac{1}{2sD(2-sL_g)}r_{n+1}.\r\n\\end{equation}\r\n\r\nNow, the identity $g(x_n)-g(\\ol x)=(g(x_n)-g(y_n))+(g(y_n)-g(\\ol x))$ and (a$_1$) lead to\r\n$$g(x_n)-g(\\ol x)\\le \\frac{1}{2sD(2-sL_g)}r_{n+1}+a_1\\frac{1}{n^p}$$\r\nfor every $n>\\ol k$, which combined with \\eqref{e13} give\r\n$$g(x_n)-g(\\ol x)\\le \\frac{1}{2sD(2-sL_g)}a_1\\frac{1}{(n+2)^p}+a_1\\frac{1}{n^p}\\le a_1\\left(1+\\frac{1}{2sD(2-sL_g)}\\right)\\frac{1}{n^p}=a_2\\frac{1}{n^p},$$\r\nfor every $n>\\ol k$.\r\n\r\nFor (a$_3$) observe, that  by summing up \\eqref{e11} from $n\\ge\\ol k$ to $P>n$ and using the triangle inequality we obtain\r\n $$\\|x_{P}-x_{n-1}\\|\\le \\sum_{k={n}}^P \\|x_{k}-x_{k-1}\\|\\le$$\r\n  $$-\\|x_{n}-x_{n-1}\\|+\\|x_{P+1}-x_P\\|+\\frac{9M}{4D}(\\varphi(H(y_{n},u_{n})-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{P+1},u_{P+1})-H(\\ol{x},\\ol{x}))).$$\r\n\r\nBy  letting $P\\To+\\infty$ we get\r\n$$\\|x_{n-1}-\\ol x\\|\\le -\\|x_{n}-x_{n-1}\\|+\\frac{9M}{4D}\\varphi(H(y_{n},u_{n})-H(\\ol{x},\\ol{x}))\\le \\frac{9M}{4D}\\varphi(H(y_{n},u_{n})-H(\\ol{x},\\ol{x})).$$\r\nBut, $\\varphi(t)=\\frac{K}{1-\\t}t^{1-\\t},$ hence\r\n\\begin{equation}\\label{e1111}\r\n\\|x_{n-1}-\\ol x\\|\\le \\frac{9MK}{4D(1-\\t)}(H(y_{n},u_{n})-H(\\ol{x},\\ol{x}))^{1-\\t}= M_1r_{n}^{1-\\t},\r\n\\end{equation} where $M_1=\\frac{9MK}{4D(1-\\t)}.$\r\n\r\n%From \\eqref{ee} we have\r\n%$$\\|x_{n}-x_{n-1}\\|\\le\\frac{1}{\\sqrt{D}}\\sqrt{r_{n}}.$$\r\n%Hence,\r\n%\\begin{equation}\\label{e1111}\r\n%\\|x_{n}-\\ol x\\|\\le \\frac{1}{\\sqrt{D}}\\sqrt{r_{n}}+M_1r_{n}^{1-\\t}.\r\n%\\end{equation}\r\nBut $(*)$ assures that $0\\le r_{n}\\le 1$ which combined with $\\t\\in\\left(0,\\frac12\\right]$ leads to\r\n$r_{n}^{1-\\t}\\le \\sqrt{r_{n}}$, consequently we have\r\n$$\\|x_{n-1}-\\ol x\\|\\le M_1\\sqrt{r_{n}}.$$\r\n\r\nThe conclusion follow by \\eqref{e13}, since  we have\r\n$$\\|x_{n}-\\ol x\\|\\le M_1\\sqrt{a_1}\\frac{1}{n^{\\frac{p}{2}}}=a_3\\frac{1}{n^{\\frac{p}{2}}}$$\r\nfor every $n>\\ol k.$\r\n\r\nFinally, for $n> \\ol k$ we have\r\n$$\\|y_n-\\ol x\\|=\\left\\|x_n+\\frac{\\b n}{n+\\a}(x_n-x_{n-1})-\\ol x\\right\\|\\le$$\r\n$$\\left(1+\\frac{\\b  n}{n+\\a}\\right)\\|x_n-\\ol x\\|+\\frac{\\b  n}{n+\\a}\\|x_{n-1}-\\ol x\\|\\le$$\r\n$$\\left(1+\\frac{\\b  n}{n+\\a}\\right)a_3\\frac{1}{n^{\\frac{p}{2}}}+\\frac{\\b  n}{n+\\a}a_3\\frac{1}{n^{\\frac{p}{2}}}\\le$$\r\n$$\\left(1+2\\frac{\\b  n}{n+\\a}\\right)a_3\\frac{1}{n^{\\frac{p}{2}}}.$$\r\n\r\n\r\nLet $a_4=(1+2\\b)a_3.$ Then\r\n$$\\|y_n-\\ol x\\|\\le a_4\\frac{1}{n^{\\frac{p}{2}}},$$\r\nfor all $n> \\ol k$, which proves (a$_4$).\r\n\\end{proof}","units":[{"start":14013,"end":14223,"start_line":234,"end_line":236,"raw_tex":"\\begin{lemma}\\label{desc} Let $g:\\R^m\\To\\R$ be Fr\\`echet differentiable with $L_g$ Lipschitz continuous gradient. Then\r\n$$g(y)\\le g(x)+\\<\\n g(x),y-x\\>+\\frac{L_g}{2}\\|y-x\\|^2,\\,\\forall x,y\\in\\R^m.$$\r\n\\end{lemma}","sha256_utf8":"bbb7c8a63ebdcb4f34877de213b189a5e0957fe9efe249547f3879da7455244f"},{"start":14331,"end":15392,"start_line":240,"end_line":256,"raw_tex":"\\begin{theorem}\\label{decreasing} In the settings of problem \\eqref{opt-pb}, for some starting points $x_0=y_0\\in\\R^m$ let $(x_n)_{n\\in\\N},\\,(y_n)_{n\\in\\N}$ be the sequences generated by the numerical scheme \\eqref{generaldiscrete}. Consider the sequences\r\n $$A_{n-1}=\\frac{2-s L_g}{2s}\\left(\\frac{(1+\\b)n+\\a}{n+\\a}\\right)^2-\\frac{\\b n((1+\\b)n+\\a)}{s(n+\\a)^2},$$\r\n$$C_{n-1}=\\frac{2-s L_g}{2s}\\frac{\\b n-\\b}{n+\\a-1}\\frac{(1+\\b)n+\\a}{n+\\a}-\\frac{1}{2s}\\frac{\\b n-\\b}{n+\\a-1}\\frac{\\b n}{n+\\a}$$\r\nand\r\n$$\\d_n=A_{n-1}-C_{n-1}$$\r\nfor all $n\\in\\N,\\,n\\ge 1$.\r\n\r\n Then, there exists $N\\in\\N$ such that %for all $n\\ge N$ one has\r\n\\begin{itemize}\r\n\\item[(i)] The sequence $ \\left(g(y_{n})+\\d_{n}\\|x_{n}-x_{n-1}\\|^2\\right)_{n\\ge N}$ is decreasing and $\\d_n>0$ for all $n\\ge N$.\r\n\\end{itemize}\r\nAssume that $g$ is bounded from below. Then, the following statements hold.\r\n\\begin{itemize}\r\n\\item[(ii)] The sequence $\\left(g(y_{n})+\\d_{n}\\|x_{n}-x_{n-1}\\|^2\\right)_{n\\in \\N}$ is convergent;\r\n\\item[(iii)] $\\sum_{n\\ge 1}\\|x_n-x_{n-1}\\|^2<+\\infty.$\r\n\\end{itemize}\r\n\\end{theorem}","sha256_utf8":"7b2df59d94f8cbd29a1cd4445148247d00b597cf2db341e3ea515f2d88de9184"},{"start":15396,"end":20498,"start_line":258,"end_line":348,"raw_tex":"\\begin{proof}\r\n\r\nFrom \\eqref{generaldiscrete} we have $\\n g(y_n)=\\frac{1}{s}(y_n-x_{n+1})$, hence\r\n$$\\<\\n g(y_n),y_{n+1}-y_n\\>=\\frac{1}{s}\\<y_n-x_{n+1},y_{n+1}-y_n\\>.$$\r\nNow, from Lemma \\ref{desc} we obtain\r\n$$g(y_{n+1})\\le g(y_n)+\\<\\n g(y_n),y_{n+1}-y_n\\>+\\frac{L_g}{2}\\|y_{n+1}-y_n\\|^2,$$\r\nconsequently we have\r\n\\begin{equation}\\label{e1}\r\ng(y_{n+1})-\\frac{L_g}{2}\\|y_{n+1}-y_n\\|^2\\le g(y_n)+ \\frac{1}{s}\\<y_n-x_{n+1},y_{n+1}-y_n\\>.\r\n\\end{equation}\r\n\r\nFurther,\r\n$$\\<y_n-x_{n+1},y_{n+1}-y_n\\>=-\\|y_{n+1}-y_n\\|^2+\\<y_{n+1}-x_{n+1},y_{n+1}-y_n\\>,$$\r\n and\r\n$$y_{n+1}-x_{n+1}=\\frac{\\b(n+1)}{n+\\a+1}(x_{n+1}-x_n),$$\r\nhence\r\n\\begin{equation}\\label{e2}\r\ng(y_{n+1})+\\left(\\frac{1}{s}-\\frac{L_g}{2}\\right)\\|y_{n+1}-y_n\\|^2\\le g(y_n)+ \\frac{\\frac{\\b(n+1)}{n+\\a+1}}{s}\\<x_{n+1}-x_n,y_{n+1}-y_n\\>.\r\n\\end{equation}\r\n\r\nSince $$y_{n+1}-y_n=\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}(x_{n+1}-x_n)-\\frac{\\b n}{n+\\a}(x_n-x_{n-1}),$$\r\nwe have,\r\n$$\\|y_{n+1}-y_n\\|^2=\\left\\|\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}(x_{n+1}-x_n)-\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right\\|^2=$$\r\n$$\\left(\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}\\right)^2\\|x_{n+1}-x_n\\|^2+\\left(\\frac{\\b n}{n+\\a}\\right)^2\\|x_n-x_{n-1}\\|^2-$$\r\n$$2\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}\\frac{\\b n}{n+\\a}\\<x_{n+1}-x_n,x_n-x_{n-1}\\>,$$\r\nand\r\n$$\\<x_{n+1}-x_n,y_{n+1}-y_n\\>=\\left\\<x_{n+1}-x_n,\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}(x_{n+1}-x_n)-\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right\\>=$$\r\n$$\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}\\|x_{n+1}-x_n\\|^2-\\frac{\\b n}{n+\\a}\\<x_{n+1}-x_n,x_n-x_{n-1}\\>.$$\r\n\r\nReplacing the above equalities in \\eqref{e2}, we obtain\r\n\r\n$$g(y_{n+1})+\\frac{(2-s L_g)\\left(\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}\\right)^2-2\\frac{\\b(n+1)((1+\\b)n+\\a+\\b+1)}{(n+\\a+1)^2}}{2s}\\|x_{n+1}-x_n\\|^2\\le$$\r\n$$g(y_n)-\\frac{(2-s L_g)\\left(\\frac{\\b n}{n+\\a}\\right)^2}{2s}\\|x_{n}-x_{n-1}\\|^2+$$\r\n$$\\frac{(2-s L_g)\\frac{\\b n}{n+\\a}\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}-\\frac{\\b n}{n+\\a}\\frac{\\b(n+1)}{n+\\a+1}}{s}\\<x_{n+1}-x_n,x_n-x_{n-1}\\>.$$\r\n\r\nFor simplicity let %$$A_{n}=\\frac{(2-s L_g)\\left(\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}\\right)^2-2\\frac{\\b(n+1)((1+\\b)n+\\a+\\b+1)}{(n+\\a+1)^2}}{2s},$$\r\n$$B_n=\\frac{(2-s L_g)\\left(\\frac{\\b n}{n+\\a}\\right)^2}{2s}$$\r\n%and $$C_n=\\frac{(2-s L_g)\\frac{\\b n}{n+\\a}\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}-\\frac{\\b n}{n+\\a}\\frac{\\b(n+1)}{n+\\a+1}}{2s}$$\r\nfor all $n\\in\\N.$\r\n\r\n Hence we have\r\n $$g(y_{n+1})+A_{n}\\|x_{n+1}-x_n\\|^2-2C_n\\<x_{n+1}-x_n,x_n-x_{n-1}\\>\\le g(y_n)-B_n\\|x_{n}-x_{n-1}\\|^2.$$\r\n\r\nBy using the equality\r\n\\begin{equation}\\label{e5}\r\n-2\\<x_{n+1}-x_n,x_n-x_{n-1}\\>=\\|x_{n+1}+x_{n-1}-2x_n\\|^2-\\|x_{n+1}-x_n\\|^2-\\|x_n-x_{n-1}\\|^2\r\n\\end{equation}\r\nwe obtain\r\n$$g(y_{n+1})+(A_n-C_n)\\|x_{n+1}-x_n\\|^2+C_n\\|x_{n+1}+x_{n-1}-2x_n\\|^2\\le g(y_n)+(C_n-B_n)\\|x_n-x_{n-1}\\|^2.$$\r\n\r\nNote that $A_{n}-C_n=\\d_{n+1}$ and let us denote $\\Delta_n=B_n+A_{n-1}-C_{n-1}-C_n$. Consequently the following inequality holds.\r\n\\begin{equation}\\label{e6}\r\nC_n\\|x_{n+1}+x_{n-1}-2x_n\\|^2+\\Delta_n\\|x_n-x_{n-1}\\|^2\\le (g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2)- (g(y_{n+1})+\\d_{n+1}\\|x_{n+1}-x_n\\|^2).\r\n\\end{equation}\r\n\r\n\r\nSince $0<\\b<1$ and $s<\\frac{2(1-\\b)}{L_g},$  we have\r\n$$\\lim_{n\\To+\\infty}A_n=\\frac{(2-s L_g)(\\b+1)^2-2\\b-2\\b^2}{2s}>0,$$\r\n$$\\lim_{n\\To+\\infty}B_n=\\frac{(2-s L_g)\\b^2}{2s}>0,$$\r\n$$\\lim_{n\\To+\\infty}C_n= \\frac{(2-s L_g)(\\b^2+\\b)-\\b^2}{2s} >0,$$\r\n$$\\lim_{n\\To+\\infty}\\Delta_n=\\frac{2-s L_g-2\\b}{2s}>0,$$\r\nand\r\n$$\\lim_{n\\To+\\infty}\\d_n=\\frac{2-\\b^2-s L_g(\\b+1)}{2s}>0.$$\r\n\r\nHence, there exists $N\\in\\N$ and $C>0,\\,D>0$ such that for all $n\\ge N$ one has\r\n$$C_n\\ge C,\\, \\Delta_n\\ge D\\mbox{ and }\\d_n>0$$ which, in the view of \\eqref{e6}, shows (i), that is,  the sequence $g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2$ is decreasing for $n\\ge N.$\r\n\r\nAssume now that $g$ is bounded from below. %Since $\\lim_{n\\To+\\infty}\\Delta_n>0$ there exists $D>0$ and $p\\in\\N$ such that $\\Delta_n\\ge D$ for all $n\\ge p.$\r\n By using \\eqref{e6} again, we obtain\r\n $$0\\le C\\|x_{n+1}+x_{n-1}-2x_n\\|^2+D\\|x_n-x_{n-1}\\|^2\\le (g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2)- (g(y_{n+1})+\\d_{n+1}\\|x_{n+1}-x_n\\|^2),$$\r\nfor all $n\\ge N,$ or more convenient, that\r\n \\begin{equation}\\label{ineq}\r\n0\\le D\\|x_n-x_{n-1}\\|^2\\le (g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2)- (g(y_{n+1})+\\d_{n+1}\\|x_{n+1}-x_n\\|^2),\r\n\\end{equation}\r\nfor all $n\\ge N.$ Let $r>N.$ By summing up the latter relation  we have\r\n$$\r\nD\\sum_{n=N}^r\\|x_n-x_{n-1}\\|^2\\le (g(y_N)+\\d_N\\|x_N-x_{N-1}\\|^2)-(g(y_{r+1})+\\d_{r+1}\\|x_{r+1}-x_r\\|^2)\r\n$$\r\nwhich leads to\r\n\\begin{equation}\\label{forcoercive}\r\ng(y_{r+1})+D\\sum_{n=N}^r\\|x_n-x_{n-1}\\|^2\\le g(y_N)+\\d_N\\|x_N-x_{N-1}\\|^2.\r\n\\end{equation}\r\nNow,  taking into account that $g$ is bounded from below, by letting $r\\To+\\infty$  we obtain\r\n$$\\sum_{n=N}^{\\infty}\\|x_n-x_{n-1}\\|^2\\le+\\infty$$ which proves (iii).\r\n\r\nThe latter relation also shows that\r\n$$\\lim_{n\\To+\\infty}\\|x_n-x_{n-1}\\|^2=0,$$\r\nhence $$\\lim_{n\\To+\\infty}\\d_n\\|x_n-x_{n-1}\\|^2=0.$$\r\nBut then, from the fact that $g$ is bounded from below we obtain that the sequence $g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2$ is bounded from below. On the other hand, from (i) we have that  the sequence $g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2$ is decreasing for $n\\ge N,$ hence there exists\r\n$$\\lim_{n\\To+\\infty} g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2\\in\\R.$$\r\n\\end{proof}","sha256_utf8":"a762241acfa3d77db6d7f897d8279aa70c7b02509e36d0fc6b64382facd19af4"},{"start":21195,"end":22691,"start_line":363,"end_line":382,"raw_tex":"\\begin{lemma}\\label{regularization} In the settings of problem \\eqref{opt-pb}, for some starting points $x_0=y_0\\in\\R^m$ consider the sequences $(x_n)_{n\\in\\N},\\,(y_n)_{n\\in\\N}$ generated by Algorithm \\eqref{generaldiscrete}. Assume that $g$ is bounded from below and consider the function\r\n$$H:\\R^m\\times\\R^m\\To\\R,\\,H(x,y)=g(x)+\\frac12\\|y-x\\|^2.$$\r\nConsider further the sequence\r\n$$u_n=\\sqrt{2\\d_n}(x_{n}-x_{n-1})+y_n\\mbox{, for all }n\\in \\N,$$\r\nwhere $\\d_n$ was defined in Theorem \\ref{decreasing}. Then, the following statements hold true.\r\n\\begin{itemize}\r\n\\item[(i)] $\\omega((u_n)_{n\\in\\N})=\\omega((y_n)_{n\\in\\N})=\\omega((x_n)_{n\\in\\N})\\subseteq\\crit g$;\r\n\\item[(ii)] There exists and is finite the limit $\\lim_{n\\To+\\infty}H(y_n,u_n)$;\r\n\\item[(iii)] $\\omega((y_n,u_n)_{n\\in\\N})\\subseteq\\crit H=\\{(x,x)\\in\\R^m\\times\\R^m:x\\in\\crit g\\}$;\r\n\\item[(iv)] $\\|\\n H(y_n,u_n)\\|\\le \\frac{1}{s}\\|x_{n+1}-x_n\\|+\\left(\\frac{\\b  n}{s(n+\\a)}+2\\sqrt{2\\d_n}\\right)\\|x_{n}-x_{n-1}\\|$ for all $n\\in\\N$;\r\n\\item[(v)]  $\\|\\n H(y_n,u_n)\\|^2\\le \\frac{2}{s^2}\\|x_{n+1}-x_n\\|^2+2\\left(\\left(\\frac{\\b n}{s(n+\\a)}-\\sqrt{2\\d_n}\\right)^2+\\d_n\\right)\\|x_{n}-x_{n-1}\\|^2$ for all $n\\in\\N$;\r\n\\item[(vi)] $H$ is finite and constant on $\\omega((y_n,u_n)_{n\\in\\N}).$\r\n\\end{itemize}\r\n\r\nAssume that $(x_n)_{n\\in\\N}$ is bounded. Then,\r\n\\begin{itemize}\r\n\\item[(vii)] $\\omega((y_n, u_n)_{n\\in\\N})$ is nonempty and compact;\r\n\\item[(viii)] $\\lim_{n\\To+\\infty} \\dist((y_n,u_n),\\omega((y_n,u_n)_{n\\in\\N}))=0.$\r\n\\end{itemize}\r\n\\end{lemma}","sha256_utf8":"2cbcb4f3f8accb768126370ad94c1f75f8b56c15a796eeeaeaf7ca1d4cc03e7b"},{"start":22693,"end":27515,"start_line":383,"end_line":446,"raw_tex":"\\begin{proof}\r\n(i) Let $\\ol x\\in\\omega((x_n)_{n\\in\\N}).$ Then, there exists a subsequence $(x_{n_k})_{k\\in \\N}$ of $(x_n)_{n\\in\\N}$ such that\r\n$$\\lim_{k\\to+\\infty}x_{n_k}=\\ol x.$$\r\nSince by \\eqref{e7} $\\lim_{n\\To+\\infty}(x_n-x_{n-1})=0$ and the sequences $(\\sqrt{2\\d_n})_{n\\in\\N},\\,\\left(\\frac{\\b n}{n+\\a}\\right)_{n\\in\\N}$ converge, we obtain that\r\n$$\\lim_{k\\to+\\infty}y_{n_k}=\\lim_{k\\to+\\infty}u_{n_k}=\\lim_{k\\to+\\infty}x_{n_k}=\\ol x,$$\r\nwhich shows that $$\\omega((x_n)_{n\\in\\N})\\subseteq \\omega((u_n)_{n\\in\\N})\\mbox{ and }\\omega((x_n)_{n\\in\\N})\\subseteq \\omega((y_n)_{n\\in\\N}).$$\r\nFurther from \\eqref{generaldiscrete}, the continuity of $\\n g$ and \\eqref{e7}, we obtain that\r\n$$\\n g(\\ol x)=\\lim_{k\\To +\\infty} \\n g(y_{n_k})=\\frac{1}{s}\\lim_{k\\To +\\infty}(y_{n_k}-x_{n_k+1})=$$\r\n$$\\frac{1}{s}\\lim_{k\\To +\\infty}\\left[(x_{n_k}-x_{n_k+1})+\\frac{\\b n_k}{n_k+\\a}(x_{n_k}-x_{n_k-1})\\right]=0.$$\r\nHence $\\omega((x_n)_{n\\in\\N})\\subseteq\\crit g.$\r\nConversely, if $\\ol u\\in \\omega((u_n)_{n\\in\\N})$ then, from \\eqref{e7} results that $\\ol u\\in \\omega((y_n)_{n\\in\\N})$ and $\\ol u\\in \\omega((x_n)_{n\\in\\N}).$\r\nHence,\r\n$$\\omega((u_n)_{n\\in\\N})=\\omega((y_n)_{n\\in\\N})=\\omega((x_n)_{n\\in\\N})\\subseteq\\crit g.$$\r\n\r\n(ii) is nothing else than (ii) in Theorem \\ref{decreasing}.\r\n\r\nFor (iii) observe that  $\\n H(x,y)=(\\n g(x)+x-y,y-x)$, hence, $\\n H(x,y)=0$ leads to\r\n$x=y$ and $\\n g(x)=0.$\r\nConsequently\r\n$$\\crit H=\\{(x,x)\\in\\R^m\\times\\R^m: x\\in\\crit g\\}.$$\r\n\r\nFurther, consider $(\\ol y,\\ol u)\\in \\omega((y_n,u_n)_{n\\in\\N}).$ Then, there exists $(y_{n_k},u_{n_k})_{k\\in\\N}\\subseteq (y_n,u_n)_{n\\in\\N}$ such that\r\n$$(\\ol y,\\ol u)=\\lim_{k\\To+\\infty}(y_{n_k},u_{n_k})=\\lim_{k\\To+\\infty}(x_{n_k},x_{n_k})=(\\ol x,\\ol x).$$\r\nHence, $\\ol u=\\ol y=\\ol x\\in\\omega((x_n)_{n\\in\\N})\\subseteq \\crit g$ and $(\\ol x,\\ol x)\\in\\crit H.$\r\n\r\n(iv) By using the 1-norm of $\\R^m\\times\\R^m$ and \\eqref{generaldiscrete}, for every $n\\in \\N$ we have\r\n $$\\|\\n H(y_n,u_n)\\|\\le\\|\\n H(y_n,u_n)\\|_1=\\|(\\n g(y_n)+y_n-u_n,u_n-y_n)\\|_1=\\|\\n g(y_n)+y_n-u_n\\|+\\|u_n-y_n\\|\\le$$\r\n $$\\|\\n g(y_n)\\|+2\\|\\sqrt{2\\d_n}(x_{n}-x_{n-1})\\|=$$\r\n $$\\frac{1}{s}\\left\\|\\left(x_n+\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right)-x_{n+1}\\right\\|+2\\sqrt{2\\d_n}\\|x_{n}-x_{n-1}\\|\\le$$\r\n $$\\frac{1}{s}\\|x_{n+1}-x_n\\|+\\left(\\frac{\\b  n}{s(n+\\a)}+2\\sqrt{2\\d_n}\\right)\\|x_{n}-x_{n-1}\\|.$$\r\n\r\n(v) We use the euclidian norm of $\\R^m\\times\\R^m$, that is $\\|(x,y)\\|=\\sqrt{\\|x\\|^2+\\|y\\|^2}$ for all $(x,y)\\in\\R^m\\times\\R^m.$\r\nWe have:\r\n$$\\|\\n H(y_n,u_n)\\|^2=\\|(\\n g(y_n)+y_n-u_n,u_n-y_n)\\|^2=\\|\\n g(y_n)+y_n-u_n\\|^2+\\|u_n-y_n\\|^2=$$\r\n$$\\left\\|\\frac1s(x_n-x_{n+1})+\\left(\\frac{\\b n}{s(n+\\a)}-\\sqrt{2\\d_n}\\right)(x_n-x_{n-1})\\right\\|^2+2\\d_n\\|x_n-x_{n-1}\\|^2\\le$$\r\n$$\\frac{2}{s^2}\\|x_{n+1}-x_n\\|^2+2\\left(\\left(\\frac{\\b n}{s(n+\\a)}-\\sqrt{2\\d_n}\\right)^2+\\d_n\\right)\\|x_{n}-x_{n-1}\\|^2$$ for all $n\\in\\N.$\r\n\r\n(vi) follows directly from (ii).\r\n\r\n Assume now that $(x_n)_{n\\in\\N}$ is bounded and let us prove (vii), (see also \\cite{BCL-AA}). Obviously $(y_n,u_n)_{n\\in\\N}$ is bounded, hence according to Weierstrass Theorem $\\omega((y_n, u_n)_{n\\in\\N}),$ (and also $\\omega((x_n)_{n\\in\\N})$), is nonempty. It remains to show that $\\omega((y_n, u_n)_{n\\in\\N})$ is closed. From (i) and the proof of (iii) we have\r\n \\begin{equation}\\label{e8}\r\n\\omega((y_n, u_n)_{n\\in\\N})=\\{(\\ol x,\\ol x)\\in\\R^m\\times \\R^m: \\ol x\\in \\omega((x_n)_{n\\in\\N})\\}.\r\n\\end{equation}\r\n Hence, it is enough to show that $\\omega((x_n)_{n\\in\\N})$ is closed.\r\n\r\n\r\n Let be $(\\ol x_p)_{p\\in\\N}\\subseteq \\omega((x_n)_{n\\in\\N})$ and assume that $\\lim_{p\\To+\\infty}\\ol x_p=x^*.$ We show that $x^*\\in\\omega((x_n)_{n\\in\\N}).$ Obviously, for every $p\\in\\N$ there exists a sequence of natural numbers $n_k^p\\To+\\infty,\\,k\\To+\\infty$, such that\r\n$$\\lim_{k\\To+\\infty}x_{n_k^p}=\\ol x_p.$$\r\n\r\nLet be $\\e >0$. Since $\\lim_{p\\To+\\infty}\\ol x_p=x^*,$  there exists $P(\\e) \\in \\N$ such that for every $p\\ge P(\\e)$ it holds\r\n$$\\|\\ol x_p-x^*\\|<\\frac\\e2.$$\r\nLet $p\\in\\N$ be fixed. Since $\\lim_{k\\To+\\infty}x_{n_k^p}=\\ol x_p,$ there exists $k(p,\\e)\\in\\N$ such that for every $k\\ge k(p,\\e)$ it holds $$\\|x_{n_k^p}-\\ol x_p\\|<\\frac\\e2.$$\r\nLet be $k_p \\geq k(p,\\varepsilon)$ such that $n_{k_p}^p >p$. Obviously $n_{k_p}^p \\To \\infty$ as $p \\To +\\infty$ and for every $p \\geq P(\\e)$\r\n$$\\|x_{n_{k_p}^p}-x^*\\|<\\e.$$\r\nHence\r\n$$\\lim_{p\\To+\\infty}x_{n_{k_p}^p}=x^*,$$\r\nthus $x^*\\in \\omega((x_n)_{n\\in\\N}).$\r\n\r\n(viii) By using \\eqref{e8} we have\r\n$$\\lim_{n\\To+\\infty} \\dist((y_n,u_n),\\omega((y_n,u_n)_{n\\in\\N}))=\\lim_{n\\To+\\infty}\\inf_{\\ol x\\in\\omega((x_n)_{n\\in\\N})}\\|(y_n,u_n)-(\\ol x,\\ol x)\\|.$$\r\nSince there exists the subsequences $(y_{n_k})_{k\\in\\N}$ and $(u_{n_k})_{k\\in\\N}$ such that\r\n$\\lim_{k\\To\\infty}y_{n_k}=\\lim_{k\\To\\infty}u_{n_k}=\\ol x_0\\in \\omega((x_n)_{n\\in\\N})$ it is straightforward that\r\n$$\\lim_{n\\To+\\infty} \\dist((y_n,u_n),\\omega((y_n,u_n)_{n\\in\\N}))=0.$$\r\n\\end{proof}","sha256_utf8":"05c27144b70366d523bc0543e918999ff2144c036f4590df4c89d4073d799e26"},{"start":30770,"end":31418,"start_line":482,"end_line":486,"raw_tex":"\\begin{lemma}\\label{unif-KL-property} Let $\\Omega\\subseteq \\R^n$ be a compact set and let $f:\\R^n\\rightarrow \\R$ be a differentiable function. Assume that $f$ is constant on $\\Omega$ and $f$ satisfies the KL property at each point of $\\Omega$.\r\nThen there exist $\\varepsilon,\\eta >0$ and $\\varphi\\in \\Theta_{\\eta}$ such that for all $\\ol x\\in\\Omega$ and for all $x$ in the intersection\r\n\\begin{equation}\\label{int} \\{x\\in\\R^n: \\dist(x,\\Omega)<\\varepsilon\\}\\cap \\{x\\in\\R^n: f(\\ol x)<f(x)<f(\\ol x)+\\eta\\}\\end{equation}\r\nthe following inequality holds \\begin{equation}\\label{KL-ineq}\\varphi'(f(x)-f(\\ol x))\\|\\n f(x)\\|\\geq 1.\\end{equation}\r\n\\end{lemma}","sha256_utf8":"ea708d59fa5a03963d2fc0a5fb9706f3019b19e160cc9db3d09441975604f898"},{"start":31498,"end":32131,"start_line":490,"end_line":497,"raw_tex":"\\begin{theorem}\\label{convergence} In the settings of problem \\eqref{opt-pb}, for some starting points $x_0=y_0\\in\\R^m$ consider the sequences $(x_n)_{n\\in\\N},\\,(y_n)_{n\\in\\N}$ generated by Algorithm \\eqref{generaldiscrete}. Assume that $g$ is bounded from below and consider the function\r\n$$H:\\R^m\\times\\R^m\\To\\R,\\,H(x,y)=g(x)+\\frac12\\|y-x\\|^2.$$\r\nAssume that $(x_n)_{n\\in\\N}$ is bounded and $H$ is a KL function. Then the following statements are true\r\n\\begin{itemize}\r\n\\item[(a)] $\\sum_{n\\ge 1}\\|x_n-x_{n-1}\\|<+\\infty;$\r\n\\item[(b)] there exists ${x}\\in\\crit(g)$ such that $\\lim_{n\\To+\\infty}x_n={x}.$\r\n\\end{itemize}\r\n\\end{theorem}","sha256_utf8":"9c7e0e9bfb9b5f1a8f72795d2364111185ab31ff74408bb0d43dad6ffc2d1bad"},{"start":32151,"end":37825,"start_line":499,"end_line":591,"raw_tex":"\\begin{proof} Consider the sequence\r\n$$u_n=\\sqrt{2\\d_n}(x_{n}-x_{n-1})+y_n\\mbox{, for all }n\\in \\N,$$\r\nthat  was defined in the hypotheses of Lemma \\ref{regularization}. Furthermore, consider $(\\ol x,\\ol x)\\in\\omega((y_n,u_n)_{n\\in\\N}).$\r\n\r\nThen, according to Lemma \\ref{regularization}, the sequence $H(y_n,u_n)$ is decreasing for all $n\\ge N$,  where $N$ was defined in Theorem \\ref{decreasing}, further\r\n$$\\ol x\\in\\crit g\\mbox{ and  }\\lim_{n\\To+\\infty}H(y_n,u_n)=H(\\ol x,\\ol x).$$\r\nWe divide the proof into two cases.\r\n\r\n{\\bf Case I.} There exists $\\ol n\\ge N,\\,\\ol n\\in\\N,$  such that $H(y_{\\ol n},u_{\\ol n})=H(\\ol x,\\ol x).$ Then, since $H(y_n,u_n)$ is decreasing for all $n\\ge N$ and $\\lim_{n\\To+\\infty}H(y_n,u_n)=H(\\ol x,\\ol x)$ we obtain that\r\n$$H(y_n,u_n)=H(\\ol x,\\ol x)\\mbox{ for all }n\\ge\\ol n.$$\r\n\r\nThe latter relation combined with \\eqref{ineq} leads to\r\n$$0\\le D\\|x_n-x_{n-1}\\|^2\\le H(y_n,u_n)-H(y_{n+1},u_{n+1})=H(\\ol x,\\ol x)-H(\\ol x,\\ol x)=0$$\r\nfor all $n\\ge\\ol n.$\r\n\r\nHence $(x_n)_{n\\ge\\ol n}$ is constant and the conclusion follows.\r\n\r\n\r\n\r\n{\\bf Case II.} For every $n\\ge N$ one has that $H(y_n,u_n)> H(\\ol x,\\ol x).$\r\nLet $\\Omega=\\omega((y_n,u_n)_{n\\in\\N}).$ Then according to Lemma \\ref{regularization}, $\\Omega$ is nonempty and compact and $H$ is constant on $\\Omega.$\r\nSince $H$ is KL, according to Lemma \\ref{unif-KL-property}  there exist $\\varepsilon,\\eta >0$ and $\\varphi\\in \\Theta_{\\eta}$ such that for all  $(z,w)$ belonging to the intersection\r\n$$ \\{(z,w)\\in\\R^n\\times\\R^n: \\dist((z,w),\\Omega)<\\varepsilon\\}\\cap \\{(z,w)\\in\\R^n\\times\\R^n: H(\\ol x,\\ol x)<H(z,w)<H(\\ol x,\\ol x)+\\eta\\}$$\r\none has\r\n$$\\varphi'(H(z,w)-H(\\ol x,\\ol x))\\|\\n H(z,w)\\|\\geq 1.$$\r\n\r\nSince $\\lim_{n\\To+\\infty}\\dist((y_n,u_n),\\Omega)=0$, there exists $n_1\\in\\N$ such that\r\n$$\\dist((y_n,u_n),\\Omega)<\\e,\\,\\forall n\\ge n_1.$$\r\n\r\nSince $$\\lim_{n\\To+\\infty}H(y_n,u_n)=H(\\ol x,\\ol x)$$\r\nand\r\n$$H(y_n,u_n))> H(\\ol{x},\\ol{x})\\mbox{ for all }n\\ge N,$$\r\nthere exists $n_2\\ge N$ such that\r\n$$H(\\ol x,\\ol x)<H(y_n,u_n)< H(\\ol{x},\\ol{x})+\\eta,\\,\\forall n\\ge n_2.$$\r\n\r\nHence, for all $n\\ge \\ol n=\\max(n_1,n_2)$ we have\r\n$$\\varphi'(H(y_n,u_n)-H(\\ol{x},\\ol{x}))\\cdot\\|\\n H(y_n,u_n))\\|\\ge 1.$$\r\nSince $\\varphi$ is concave, for all $n\\in\\N$ we have\r\n$$\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x}))\\ge$$\r\n$$\\varphi'(H(y_n,u_n)-H(\\ol{x},\\ol{x}))\\cdot(H(y_n,u_n)-H(y_{n+1},u_{n+1})),$$\r\nhence,\r\n$$\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x}))\\ge \\frac{H(y_n,u_n)-H(y_{n+1},u_{n+1})}{\\|\\n H(y_n,u_n)\\|}$$\r\nfor all $n\\ge\\ol n.$\r\n\r\n Now, from \\eqref{ineq} and Lemma \\ref{regularization} (iv) we obtain\r\n \\begin{equation}\\label{e9}\r\n \\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x}))\\ge\r\n \\end{equation}\r\n $$\\frac{D\\|x_{n}-x_{n-1}\\|^2}{\\frac{1}{s}\\|x_{n+1}-x_n\\|+\\left(\\frac{\\b  n}{s(n+\\a)}+2\\sqrt{2\\d_n}\\right)\\|x_{n}-x_{n-1}\\|},$$\r\n for all $n\\ge\\ol n.$\r\nSince $\\lim_{n\\To+\\infty}\\d_n=\\frac{2-\\b^2-s L_g(\\b+1)}{2s}>0$ and $\\lim_{n\\To+\\infty}\\frac{\\b  n}{s(n+\\a)}=\\frac{\\b }{s}\\ge 0$ there exists $\\ol N\\in\\N,\\,\\ol N\\ge\\ol n$ and $M>0$ such that\r\n$$\\max\\left(\\frac1s,\\frac{\\b  n}{s(n+\\a)}+2\\sqrt{2\\d_n}\\right)\\le M,$$\r\nfor all $n\\ge \\ol N.$\r\nHence, \\eqref{e9} becomes\r\n \\begin{equation}\\label{e10}\r\n \\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x}))\\ge\r\n \\end{equation}\r\n $$\\frac{D\\|x_{n}-x_{n-1}\\|^2}{M(\\|x_{n+1}-x_n\\|+\\|x_{n}-x_{n-1}\\|)},$$\r\n for all $n\\ge\\ol N.$\r\n\r\nConsequently,\r\n$$\\|x_{n}-x_{n-1}\\|\\le$$\r\n$$\\sqrt{\\frac{M}{D}(\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x})))\\cdot(\\|x_{n+1}-x_n\\|+\\|x_{n}-x_{n-1}\\|)},$$\r\n for all $n\\ge\\ol N.$\r\n\r\nBy using the arithmetical-geometrical mean inequality we have\r\n$$\\sqrt{\\frac{M}{D}(\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x})))\\cdot(\\|x_{n+1}-x_n\\|+\\|x_{n}-x_{n-1}\\|)}\\le$$\r\n$$\\frac{\\|x_{n+1}-x_n\\|+\\|x_{n}-x_{n-1}\\|}{3}+\\frac{3M}{4D}(\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x})))$$\r\n for all $n\\ge\\ol N.$\r\nHence\r\n$$\\|x_{n}-x_{n-1}\\|\\le$$\r\n$$\\frac{\\|x_{n+1}-x_n\\|+\\|x_{n}-x_{n-1}\\|}{3}+\\frac{3M}{4D}(\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x})))$$\r\n for all $n\\ge\\ol N$ which leads to\r\n \\begin{equation}\\label{e11}\r\n 2\\|x_{n}-x_{n-1}\\|-\\|x_{n+1}-x_{n}\\|\\le\\frac{9M}{4D}(\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x})))\r\n \\end{equation}\r\n for all $n\\ge\\ol N.$ Let $P>\\ol N$. By summing up \\eqref{e11} from $\\ol N$ to $P$ we obtain\r\n $$\\sum_{n={\\ol N}}^P \\|x_{n}-x_{n-1}\\|\\le$$\r\n  $$-\\|x_{\\ol N}-x_{\\ol N-1}\\|+\\|x_{P+1}-x_P\\|+\\frac{9M}{4D}(\\varphi(H(y_{\\ol N},u_{\\ol N})-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{P+1},u_{P+1})-H(\\ol{x},\\ol{x}))).$$\r\nNow, by letting $P\\To+\\infty$ and using the fact that $\\varphi(0)=0$ and \\eqref{e7} we obtain that\r\n$$\\sum_{n={\\ol N}}^\\infty \\|x_{n}-x_{n-1}\\|\\le-\\|x_{\\ol N}-x_{\\ol N-1}\\|+\\frac{9M}{4D}\\varphi(H(y_{\\ol N},u_{\\ol N})-H(\\ol{x},\\ol{x}))<+\\infty,$$\r\nhence\r\n$$\\sum_{n\\ge 1}\\|x_n-x_{n-1}\\|<+\\infty$$ which is exactly (a).\r\n\r\nObviously the sequence $S_n=\\sum_{k=1}^n\\|x_k-x_{k-1}\\|$ is Cauchy, hence, for all $\\e>0$ there exists $N_{\\e}\\in \\N$   such that for all $n\\ge N_\\e$ and for all $p\\in \\N$ one has\r\n$$S_{n+p}-S_n\\le \\e.$$\r\nBut\r\n$$S_{n+p}-S_n=\\sum_{k={n+1}}^{n+p}\\|x_k-x_{k-1}\\|\\ge \\left\\|\\sum_{k={n+1}}^{n+p}(x_k-x_{k-1})\\right\\|=\\|x_{n+p}-x_n\\|$$\r\nhence the sequence $(x_n)_{n\\in\\N}$ is Cauchy, consequently is convergent. Let\r\n$$\\lim_{n\\To+\\infty}x_n=x.$$\r\nNow, according to Lemma \\ref{regularization} (i) one has\r\n$$\\{x\\}=\\omega((x_n)_{n\\in\\N})\\subseteq\\crit g$$ which proves (b).\r\n\\end{proof}","sha256_utf8":"f3814f0b02a1ab8674ccd75abb67e1c50082c320d3926731cc1c87858c05dad2"},{"start":41040,"end":49377,"start_line":627,"end_line":760,"raw_tex":"\\begin{proof}\r\n As we have seen in the proof of Theorem \\ref{convergence},  if there exists $\\ol n\\ge N,\\,\\ol n\\in\\N$, (where $N$ was defined in Theorem \\ref{decreasing}), such that $H(y_{\\ol n},u_{\\ol n})=H(\\ol x,\\ol x),$ then,\r\n$$H(y_n,u_n)=H(\\ol x,\\ol x)\\mbox{ for all }n\\ge\\ol n$$\r\nand $(x_n)_{n\\ge\\ol n}$ is constant. Consequently $(y_n)_{n\\ge\\ol n}$ is also constant and the conclusion of the theorem is straightforward.\r\n\r\nHence, in what follows we assume  that $H(y_{n},u_{ n})>H(\\ol x,\\ol x),$ for all $n\\ge N.$\r\n\\\\\r\n\r\nLet us fix $p>0$ and let us  prove (a$_1$).\r\n\r\n For simplicity let us denote $r_n=H(y_n,u_n)-H(\\ol x,\\ol x)>0$  for all $n\\in\\N.$\r\n From \\eqref{e6} we have\r\n$$\\Delta_n\\|x_n-x_{n-1}\\|^2\\le r_n-r_{n+1}\\mbox{ for all }n\\ge N.$$\r\n\r\nFrom Lemma \\ref{regularization} (v) we have\r\n $$\\|\\n H(y_n,u_n)\\|^2\\le \\frac{2}{s^2}\\|x_{n+1}-x_n\\|^2+2\\left(\\left(\\frac{\\b n}{s(n+\\a)}-\\sqrt{2\\d_n}\\right)^2+\\d_n\\right)\\|x_{n}-x_{n-1}\\|^2$$ for all $n\\in\\N$.\r\n  Let $S_n=2\\left(\\left(\\frac{\\b n}{s(n+\\a)}-\\sqrt{2\\d_n}\\right)^2+\\d_n\\right),$ for all $n\\in\\N.$\r\n\r\nIt follows that, for all $n\\ge N$ one has\r\n$$\\|x_n-x_{n-1}\\|^2\\ge \\frac{1}{S_n}\\|\\n H(y_n,u_n)\\|^2-\\frac{2}{s^2S_n}\\|x_{n+1}-x_n\\|^2\\ge$$\r\n$$\\frac{1}{S_n}\\|\\n H(y_n,u_n)\\|^2-\\frac{2}{s^2S_n\\Delta_{n+1}}(r_{n+1}-r_{n+2}).$$\r\n\r\n\r\nNow by using the {\\L}ojasiewicz property of $H$ at $(\\ol x,\\ol x)\\in\\crit H$, and the fact that $\\lim_{n\\To+\\infty}(y_n,u_n)=(\\ol x,\\ol x)$,  we obtain that there exists  $K,\\e>0$ and $\\ol N_1\\in\\N,$ such that for all $n\\ge \\ol N_1$ one has\r\n$$\\|(y_n,u_n)-(\\ol x,\\ol x)\\|<\\e,$$\r\nconsequently\r\n\\begin{equation}\\label{e111} r_n-r_{n+1}\\ge \\frac{\\Delta_n}{S_n}\\|\\n H(y_n,u_n)\\|^2-\\frac{2\\Delta_n}{s^2S_n\\Delta_{n+1}}(r_{n+1}-r_{n+2})\\ge\r\n\\end{equation}\r\n$$\\frac{\\Delta_n}{K^2 S_n}r_n^{2\\t}-\\frac{2\\Delta_n}{s^2S_n\\Delta_{n+1}}(r_{n+1}-r_{n+2})\\ge$$\r\n$$\\frac{\\Delta_n}{K^2 S_n}r_{n+1}^{2\\t}-\\frac{2\\Delta_n}{s^2S_n\\Delta_{n+1}}(r_{n+1}-r_{n+2})=\\a_n r_{n+1}^{2\\t}-\\b_n(r_{n+1}-r_{n+2}),$$\r\nwhere $\\a_n= \\frac{\\Delta_n}{K^2 S_n}\\mbox{ and  }\\b_n=\\frac{2\\Delta_n}{s^2S_n\\Delta_{n+1}}.$\r\n\r\nIt is obvious that the sequences $(\\a_n)_{n\\ge\\ol N_1}$ and $(\\b_n)_{n\\ge\\ol N_1}$ are convergent, further\r\n$$\\lim_{n\\To+\\infty} \\a_n>0\\mbox{ and } \\lim_{n\\To+\\infty} \\b_n>0.$$\r\nNow, since $0<2\\t\\le1$ and $r_{n+1}\\To 0$, there exists $\\ol N_2\\in\\N,\\, \\ol N_2\\ge \\ol N_1,$ such that $r_{n+1}^{2\\t}\\ge r_{n+1}$ for all $n\\ge\\ol N_2.$\r\n\r\n$(*)$\\,\\,\\, Note that this implies that $0\\le r_n\\le 1$ for all $n\\ge\\ol N_2.$\r\n\r\nHence,\r\n$$r_n\\ge \\left(\\a_n-\\b_n+1\\right)r_{n+1}+\\b_nr_{n+2},\\mbox{ for all }n\\ge\\ol N_2.$$\r\n\r\nLet us define for every $n>\\ol N_2$ the sequence $\\Xi_n=\\frac{\\b_n n^p}{(n+1)^p-n^p}.$\r\nThen, since $p>0$ one has $\\lim_{n\\To+\\infty}\\Xi_n=+\\infty.$\r\n\r\nSince\r\n$$\\lim_{n\\To+\\infty}\\left(\\a_n-\\b_n+1\\right)-\\left(1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}-\\frac{\\b_{n-1}}{1+\\frac{\\b_{n}}{\\Xi_{n}}}\\right)=\\lim_{n\\To+\\infty}\\a_n>0,$$  there exists $\\ol k\\in\\N,\\,\\ol k\\ge\\ol N_2$ such that for all $n\\ge\\ol k$ one has\r\n$$\\a_n-\\b_n+1\\ge 1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}-\\frac{\\b_{n-1}}{1+\\frac{\\b_{n}}{\\Xi_{n}}}.$$\r\n\r\nConsequently,\r\n$$r_n\\ge \\left(1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}-\\frac{\\b_{n-1}}{1+\\frac{\\b_{n}}{\\Xi_{n}}}\\right)r_{n+1}+\\b_nr_{n+2},\\mbox{ for all }n\\ge\\ol k,$$\r\nor, equivalently\r\n\\begin{equation}\\label{e12}\r\nr_n+\\frac{\\b_{n-1}}{1+\\frac{\\b_{n}}{\\Xi_{n}}}r_{n+1}\\ge \\left(1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}\\right)\\left(r_{n+1}+\\frac{\\b_n}{1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}}r_{n+2}\\right),\r\n\\end{equation}\r\nfor all $n\\ge\\ol k.$\r\nNow \\eqref{e12} leads to\r\n$$\\prod_{k=\\ol k}^n\\left( r_k+\\frac{\\b_{k-1}}{1+\\frac{\\b_{k}}{\\Xi_k}}r_{k+1}\\right)\\ge \\prod_{k=\\ol k}^n\\left(1+\\frac{\\b_{k+1}}{\\Xi_{k+1}}\\right)\\prod_{k=\\ol k}^n\\left(r_{k+1}+\\frac{\\b_k}{1+\\frac{\\b_{k+1}}{\\Xi_{k+1}}}r_{k+2}\\right),$$\r\nhence after simplifying we get\r\n\\begin{equation}\\label{egen}\\left(r_{\\ol k}+\\frac{\\b_{\\ol k-1}}{1+\\frac{\\b_{\\ol k}}{\\Xi_{\\ol k}}}r_{\\ol k+1}\\right)\\prod_{k=\\ol k}^n\\frac{1}{1+\\frac{\\b_{k+1}}{\\Xi_{k+1}}}\\ge\r\nr_{n+1}+\\frac{\\b_n}{1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}}r_{n+2}.\r\n\\end{equation}\r\n\r\nBut, $\\frac{\\b_{n+1}}{\\Xi_{n+1}}=\\frac{(n+2)^p}{(n+1)^p}-1,$ hence\r\n$$\\prod_{k=\\ol k}^n\\frac{1}{1+\\frac{\\b_{k+1}}{\\Xi_{k+1}}}=\\prod_{k=\\ol k}^n\\frac{(k+1)^p}{(k+2)^p}=\\frac{(\\ol k+1)^p}{(n+2)^p}.$$\r\nBy denoting $\\left(r_{\\ol k}+\\frac{\\b_{\\ol k-1}}{1+\\frac{\\b_{\\ol k}}{\\Xi_{\\ol k}}}r_{\\ol k+1}\\right)(\\ol k+1)^p=a_1$,  we have\r\n$$a_1 \\frac{1}{(n+2)^p}\\ge r_{n+1}+\\frac{\\b_n}{1+\\frac{1}{n+1}\\b_{n+1}}r_{n+2}.$$\r\n\r\nHence,\r\n\\begin{equation}\\label{e13}\r\na_1 \\frac{1}{n^p}\\ge a_1 \\frac{1}{(n+1)^p}\\ge r_n= g(y_n)-g(\\ol x)+\\d_n\\|x_n-x_{n-1}\\|^2\\ge g(y_n)-g(\\ol x)\r\n\\end{equation}\r\nwhich is (a$_1$).\r\n\r\nFor (a$_2$) we start from   Lemma \\ref{desc} and \\eqref{generaldiscrete} and we have\r\n$$g(x_n)-g(y_n)\\le\\<\\n g(y_n),x_n-y_n\\>+\\frac{L_g}{2}\\|x_n-y_n\\|^2=$$\r\n$$\\frac1s\\left\\<(x_n-x_{n+1})+\\frac{\\b n}{n+\\a}(x_n-x_{n-1}),-\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right\\>+\\frac{L_g}{2}\\left(\\frac{\\b n}{n+\\a}\\right)^2\\|x_n-x_{n-1}\\|^2= $$\r\n$$-\\left(\\frac{\\b n}{n+\\a}\\right)^2\\frac{2-sL_g}{2s}\\|x_n-x_{n-1}\\|^2+\\frac1s\\left\\<x_{n+1}-x_n,\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right\\>.$$\r\nBy using the inequality $\\<X,Y\\>\\le \\frac12\\left(a^2\\|X\\|^2+\\frac{1}{a^2}\\|Y\\|^2\\right)$ for all $X,Y\\in\\R^m, a\\in \\R\\setminus\\{0\\}$, we obtain\r\n$$\\left\\<x_{n+1}-x_n,\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right\\>\\le \\frac12\\left(\\frac{1}{2-sL_g}\\|x_{n+1}-x_n\\|^2+(2-sL_g)\\left(\\frac{\\b n}{n+\\a}\\right)^2\\|x_n-x_{n-1}\\|^2\\right),$$ consequently\r\n$$g(x_n)-g(y_n)\\le\\frac{1}{2s(2-sL_g)}\\|x_{n+1}-x_n\\|^2.$$\r\n\r\nFrom \\eqref{ineq} we have\r\n$$\\|x_{n}-x_{n-1}\\|^2\\le \\frac{1}{{D}}((g(y_{n})+\\d_{n}\\|x_{n}-x_{n-1}\\|^2)- (g(y_{n+1})+\\d_{n+1}\\|x_{n+1}-x_{n}\\|^2))$$ and since the sequence $(g(y_{n})+\\d_{n}\\|x_{n+1}-x_{n}\\|^2)_{n\\ge \\ol k}$ is decreasing and has the limit $g(\\ol x)$, we obtain that $g(y_{n+1})+\\d_{n+1}\\|x_{n+1}-x_{n}\\|^2\\ge g(\\ol x)$, consequently\r\n\\begin{equation}\\label{ee}\r\n\\|x_{n}-x_{n-1}\\|^2\\le\\frac{1}{{D}}{r_{n}}.\r\n\\end{equation}\r\n\r\nHence, for all $n\\ge\\ol k$ one has\r\n\\begin{equation}\\label{ee1}\r\ng(x_n)-g(y_n)\\le\\frac{1}{2sD(2-sL_g)}r_{n+1}.\r\n\\end{equation}\r\n\r\nNow, the identity $g(x_n)-g(\\ol x)=(g(x_n)-g(y_n))+(g(y_n)-g(\\ol x))$ and (a$_1$) lead to\r\n$$g(x_n)-g(\\ol x)\\le \\frac{1}{2sD(2-sL_g)}r_{n+1}+a_1\\frac{1}{n^p}$$\r\nfor every $n>\\ol k$, which combined with \\eqref{e13} give\r\n$$g(x_n)-g(\\ol x)\\le \\frac{1}{2sD(2-sL_g)}a_1\\frac{1}{(n+2)^p}+a_1\\frac{1}{n^p}\\le a_1\\left(1+\\frac{1}{2sD(2-sL_g)}\\right)\\frac{1}{n^p}=a_2\\frac{1}{n^p},$$\r\nfor every $n>\\ol k$.\r\n\r\nFor (a$_3$) observe, that  by summing up \\eqref{e11} from $n\\ge\\ol k$ to $P>n$ and using the triangle inequality we obtain\r\n $$\\|x_{P}-x_{n-1}\\|\\le \\sum_{k={n}}^P \\|x_{k}-x_{k-1}\\|\\le$$\r\n  $$-\\|x_{n}-x_{n-1}\\|+\\|x_{P+1}-x_P\\|+\\frac{9M}{4D}(\\varphi(H(y_{n},u_{n})-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{P+1},u_{P+1})-H(\\ol{x},\\ol{x}))).$$\r\n\r\nBy  letting $P\\To+\\infty$ we get\r\n$$\\|x_{n-1}-\\ol x\\|\\le -\\|x_{n}-x_{n-1}\\|+\\frac{9M}{4D}\\varphi(H(y_{n},u_{n})-H(\\ol{x},\\ol{x}))\\le \\frac{9M}{4D}\\varphi(H(y_{n},u_{n})-H(\\ol{x},\\ol{x})).$$\r\nBut, $\\varphi(t)=\\frac{K}{1-\\t}t^{1-\\t},$ hence\r\n\\begin{equation}\\label{e1111}\r\n\\|x_{n-1}-\\ol x\\|\\le \\frac{9MK}{4D(1-\\t)}(H(y_{n},u_{n})-H(\\ol{x},\\ol{x}))^{1-\\t}= M_1r_{n}^{1-\\t},\r\n\\end{equation} where $M_1=\\frac{9MK}{4D(1-\\t)}.$\r\n\r\n%From \\eqref{ee} we have\r\n%$$\\|x_{n}-x_{n-1}\\|\\le\\frac{1}{\\sqrt{D}}\\sqrt{r_{n}}.$$\r\n%Hence,\r\n%\\begin{equation}\\label{e1111}\r\n%\\|x_{n}-\\ol x\\|\\le \\frac{1}{\\sqrt{D}}\\sqrt{r_{n}}+M_1r_{n}^{1-\\t}.\r\n%\\end{equation}\r\nBut $(*)$ assures that $0\\le r_{n}\\le 1$ which combined with $\\t\\in\\left(0,\\frac12\\right]$ leads to\r\n$r_{n}^{1-\\t}\\le \\sqrt{r_{n}}$, consequently we have\r\n$$\\|x_{n-1}-\\ol x\\|\\le M_1\\sqrt{r_{n}}.$$\r\n\r\nThe conclusion follow by \\eqref{e13}, since  we have\r\n$$\\|x_{n}-\\ol x\\|\\le M_1\\sqrt{a_1}\\frac{1}{n^{\\frac{p}{2}}}=a_3\\frac{1}{n^{\\frac{p}{2}}}$$\r\nfor every $n>\\ol k.$\r\n\r\nFinally, for $n> \\ol k$ we have\r\n$$\\|y_n-\\ol x\\|=\\left\\|x_n+\\frac{\\b n}{n+\\a}(x_n-x_{n-1})-\\ol x\\right\\|\\le$$\r\n$$\\left(1+\\frac{\\b  n}{n+\\a}\\right)\\|x_n-\\ol x\\|+\\frac{\\b  n}{n+\\a}\\|x_{n-1}-\\ol x\\|\\le$$\r\n$$\\left(1+\\frac{\\b  n}{n+\\a}\\right)a_3\\frac{1}{n^{\\frac{p}{2}}}+\\frac{\\b  n}{n+\\a}a_3\\frac{1}{n^{\\frac{p}{2}}}\\le$$\r\n$$\\left(1+2\\frac{\\b  n}{n+\\a}\\right)a_3\\frac{1}{n^{\\frac{p}{2}}}.$$\r\n\r\n\r\nLet $a_4=(1+2\\b)a_3.$ Then\r\n$$\\|y_n-\\ol x\\|\\le a_4\\frac{1}{n^{\\frac{p}{2}}},$$\r\nfor all $n> \\ol k$, which proves (a$_4$).\r\n\\end{proof}","sha256_utf8":"541d47e2cad7f261832ce09186e42fcf3af92ff03dadd19a913dc88e44c93ad1"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"小指数下的任意多项式收敛率","source_units":[{"start":39983,"end":41038,"start_line":617,"end_line":626,"raw_tex":"\\begin{theorem}\\label{th-conv-rate}  In the settings of problem \\eqref{opt-pb} consider the sequences $(x_n)_{n\\in\\N},\\,(y_n)_{n\\in\\N}$ generated by Algorithm \\eqref{generaldiscrete}. Assume that $g$ is bounded from below and that $(x_n)_{n\\in\\N}$ is bounded, let $\\ol x\\in\\crit(g)$ be such that $\\lim_{n\\To+\\infty}x_n=\\ol x$ and suppose that\r\n$$H:\\R^n\\times \\R^n\\To\\R,\\,H(x,y)=g(x)+\\frac{1}{2}\\|x-y\\|^2$$\r\nfulfills the {\\L}ojasiewicz property at $(\\ol x,\\ol x)\\in\\crit H$ with {\\L}ojasiewicz exponent  $\\t\\in\\left(0,\\frac12\\right].$ Then, for every $p>0$ there exist $a_1,a_2,a_3,a_4>0$ and $\\ol k\\in\\N$ such that the following statements hold true:\r\n\\begin{itemize}\r\n\\item[$(\\emph{a}_1)$] $g(y_{n})-g(\\ol x)\\le a_1 \\frac{1}{n^p}$ for  every $n> \\ol k$,\r\n\\item[$(\\emph{a}_2)$] $g(x_{n})-g(\\ol x)\\le a_2\\frac{1}{n^p}$ for  every $n>\\ol k$,\r\n\\item[$(\\emph{a}_3)$] $\\|x_n-\\ol x\\|\\le a_3 \\frac{1}{n^{\\frac{p}{2}}}$ for  every $n>\\ol k$,\r\n\\item[$(\\emph{a}_4)$] $\\|y_n-\\ol x\\|\\le a_4 \\frac{1}{n^{\\frac{p}{2}}}$ for all  $n>\\ol k$.\r\n\\end{itemize}\r\n\\end{theorem}","sha256_utf8":"61dd36983907c8351ffead76786d259c1e7c5868bde2b889850ec73eee436d86"}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","title":"主定理的有界性与指数假设","source_units":[{"start":39983,"end":40635,"start_line":617,"end_line":619,"raw_tex":"\\begin{theorem}\\label{th-conv-rate}  In the settings of problem \\eqref{opt-pb} consider the sequences $(x_n)_{n\\in\\N},\\,(y_n)_{n\\in\\N}$ generated by Algorithm \\eqref{generaldiscrete}. Assume that $g$ is bounded from below and that $(x_n)_{n\\in\\N}$ is bounded, let $\\ol x\\in\\crit(g)$ be such that $\\lim_{n\\To+\\infty}x_n=\\ol x$ and suppose that\r\n$$H:\\R^n\\times \\R^n\\To\\R,\\,H(x,y)=g(x)+\\frac{1}{2}\\|x-y\\|^2$$\r\nfulfills the {\\L}ojasiewicz property at $(\\ol x,\\ol x)\\in\\crit H$ with {\\L}ojasiewicz exponent  $\\t\\in\\left(0,\\frac12\\right].$ Then, for every $p>0$ there exist $a_1,a_2,a_3,a_4>0$ and $\\ol k\\in\\N$ such that the following statements hold true:\r\n","sha256_utf8":"d69aefedd473230bed1f67391e1c3d6b9b77cbb7d6a42d056d42f30f054e8f43"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"光滑非凸最小化问题","source_units":[{"start":4408,"end":4744,"start_line":127,"end_line":128,"raw_tex":"Let $g:\\R^m\\To \\R$ be a (not necessarily convex) Fr\\'{e}chet differentiable function with $L_g$-Lipschitz continuous gradient, i.e. there exists $L_g\\ge 0$ such that $\\|\\n g(x)-\\n g(y)\\|\\le L_g\\|x-y\\|$ for all $x,y\\in \\R^m.$ We deal with the optimization problem\r\n\\begin{equation}\\label{opt-pb} (P) \\ \\inf_{x\\in\\R^m}g(x). \\end{equation}","sha256_utf8":"2f2b98905578e0a821800348b06eaa05581b0163e9995d189eb0979bce753623"}]},{"id":"S004","semantic_type":"algorithm","kind":"algorithm","title":"惯性梯度迭代与参数","source_units":[{"start":4748,"end":5080,"start_line":130,"end_line":137,"raw_tex":"We associate to \\eqref{opt-pb} the following inertial algorithm of gradient type.\r\nConsider the starting points $x_0=y_0\\in\\R^m,$ and for all $n\\in\\N$\r\n\\begin{equation}\\label{generaldiscrete}\\left\\{\\begin{array}{lll}\r\n\\ds x_{n+1}=y_n-s\\n g(y_n),\\\\\r\n\\\\\r\n\\ds y_n=x_n+\\frac{\\b n}{n+\\a}(x_n-x_{n-1}),\r\n\\end{array}\\right.\r\n\\end{equation}","sha256_utf8":"c30cd3600fdcc9d758b573dd5682d3efeae9b2c7458bd9802c759424d8c6937b"},{"start":5082,"end":5138,"start_line":138,"end_line":138,"raw_tex":"where $\\a>0,\\b\\in(0,1)$ and $0<s<\\frac{2(1-\\b)}{L_g}.$\r\n","sha256_utf8":"ba76c73b9e7536bbc9de14bbf059f20f101d97b0f470c800d0baf25f7eac5044"}]},{"id":"S005","semantic_type":"notation","kind":"notation","title":"聚点集与临界点及距离","source_units":[{"start":20783,"end":20974,"start_line":358,"end_line":358,"raw_tex":"Let us denote by $\\omega((x_n)_{n\\in\\N})$ the set of cluster points of the sequence $(x_n)_{n\\in\\N},$ and denote by $\\crit(g)=\\{x\\in\\R^m: \\nabla g(x)=0\\}$ the set of critical points of $g$.\r\n","sha256_utf8":"a4da268f9ba6654f8b20ff3911c883e9c4f0d6b230ed8cce7fdfc493ad1935b7"},{"start":21043,"end":21193,"start_line":361,"end_line":361,"raw_tex":"In the following result we use the distance function to a set, defined for $A\\subseteq\\R^n$ as $\\dist(x,A)=\\inf_{y\\in A}\\|x-y\\|$ for all $x\\in\\R^n$.\r\n","sha256_utf8":"5b5e7499f4db4e174235d1b1af383035842cc5675313dcf07e58981c5af08cc9"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"凹去奇异函数类","source_units":[{"start":28355,"end":28719,"start_line":455,"end_line":458,"raw_tex":"In order to continue our analysis we need the concept of a KL function. For $\\eta\\in(0,+\\infty]$, we denote by $\\Theta_{\\eta}$\r\nthe class of concave and continuous functions $\\varphi:[0,\\eta)\\rightarrow [0,+\\infty)$ such that $\\varphi(0)=0$, $\\varphi$ is\r\ncontinuously differentiable on $(0,\\eta)$, continuous at $0$ and $\\varphi'(s)>0$ for all\r\n$s\\in(0, \\eta)$.\r\n","sha256_utf8":"b2366286e5a0e73d810fbff8dfceda9fcdcaac69e989c7c693f27c8deb0f3462"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"KL性质","source_units":[{"start":28721,"end":29354,"start_line":460,"end_line":467,"raw_tex":"\\begin{definition}\\label{KL-property} \\rm({\\it Kurdyka-\\L{}ojasiewicz property}) Let $f:\\R^n\\rightarrow \\R$ be a differentiable function. We say that $f$ satisfies the {\\it Kurdyka-\\L{}ojasiewicz (KL) property} at\r\n$\\ol x\\in \\R^n$\r\nif there exist $\\eta \\in(0,+\\infty]$, a neighborhood $U$ of $\\ol x$ and a function $\\varphi\\in \\Theta_{\\eta}$ such that for all $x$ in the\r\nintersection\r\n$$U\\cap \\{x\\in\\R^n: f(\\ol x)<f(x)<f(\\ol x)+\\eta\\}$$ the following inequality holds\r\n$$\\varphi'(f(x)-f(\\ol x))\\|\\n f(x))\\|\\geq 1.$$\r\nIf $f$ satisfies the KL property at each point in $\\R^n$, then $f$ is called a {\\it KL function}.\r\n\\end{definition}","sha256_utf8":"9dc8527e011ea292b99a209df0905d7fadecb2d91114330276a3eddefe0cf293"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"洛雅西维奇指数","source_units":[{"start":38978,"end":39535,"start_line":610,"end_line":613,"raw_tex":"\\begin{definition}\\label{Lexpo}  Let $f:\\R^n\\To \\R $ be a differentiable function. The function $f$ is said to fulfill the {\\L}ojasiewicz property, if for every $\\ol x\\in\\crit{f}$ there exist $K,\\e>0$ and $\\t\\in(0,1)$ such that\r\n$$|f(x)-f(\\ol x)|^\\t\\le K\\|\\n f(x)\\|\\mbox{ for every }x\\mbox{ fulfilling }\\|x-\\ol x\\|<\\e.$$\r\nThe number $\\t$  is called the {\\L}ojasiewicz exponent of $f$ at the critical  point $\\ol x.$ This corresponds to the case when the desingularizing function $\\varphi$ has the form $\\varphi(t)=\\frac{K}{1-\\t} t^{1-\\t}.$\r\n\\end{definition}","sha256_utf8":"13c773abd73aab21c435567258d3b240df220353cdb3fe9e5990624a2cbd4697"}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","title":"相邻迭代差趋零","source_units":[{"start":20504,"end":20777,"start_line":351,"end_line":355,"raw_tex":"\\begin{remark}\\label{r1} Observe that conclusion (iii) in the hypotheses of Theorem \\ref{decreasing} assures that the sequence  $(x_n-x_{n-1})_{n\\in\\N}\\in l^2,$ in particular that\r\n\\begin{equation}\\label{e7}\r\n\\lim_{n\\To+\\infty}(x_n-x_{n-1})=0.\r\n\\end{equation}\r\n\\end{remark}","sha256_utf8":"1ddb871a275fb63a6c30773876d36595aab9b86fde25fdd585d24c1584759003"}]},{"id":"S010","semantic_type":"lemma","kind":"lemma","title":"下降引理","source_units":[{"start":14013,"end":14223,"start_line":234,"end_line":236,"raw_tex":"\\begin{lemma}\\label{desc} Let $g:\\R^m\\To\\R$ be Fr\\`echet differentiable with $L_g$ Lipschitz continuous gradient. Then\r\n$$g(y)\\le g(x)+\\<\\n g(x),y-x\\>+\\frac{L_g}{2}\\|y-x\\|^2,\\,\\forall x,y\\in\\R^m.$$\r\n\\end{lemma}","sha256_utf8":"bbb7c8a63ebdcb4f34877de213b189a5e0957fe9efe249547f3879da7455244f"}]},{"id":"S011","semantic_type":"external_fact","kind":"external_fact","title":"光滑函数的二次上界","source_units":[],"declared_external":true,"declaration_text":["If a differentiable function has L-Lipschitz gradient on Euclidean space, then f(y) ≤ f(x) + ⟨∇f(x), y−x⟩ + (L/2)‖y−x‖² for all x,y. This is the standard descent lemma attributed to Nest."]},{"id":"S012","semantic_type":"theorem","kind":"theorem","title":"能量下降与平方可和性","source_units":[{"start":14331,"end":15392,"start_line":240,"end_line":256,"raw_tex":"\\begin{theorem}\\label{decreasing} In the settings of problem \\eqref{opt-pb}, for some starting points $x_0=y_0\\in\\R^m$ let $(x_n)_{n\\in\\N},\\,(y_n)_{n\\in\\N}$ be the sequences generated by the numerical scheme \\eqref{generaldiscrete}. Consider the sequences\r\n $$A_{n-1}=\\frac{2-s L_g}{2s}\\left(\\frac{(1+\\b)n+\\a}{n+\\a}\\right)^2-\\frac{\\b n((1+\\b)n+\\a)}{s(n+\\a)^2},$$\r\n$$C_{n-1}=\\frac{2-s L_g}{2s}\\frac{\\b n-\\b}{n+\\a-1}\\frac{(1+\\b)n+\\a}{n+\\a}-\\frac{1}{2s}\\frac{\\b n-\\b}{n+\\a-1}\\frac{\\b n}{n+\\a}$$\r\nand\r\n$$\\d_n=A_{n-1}-C_{n-1}$$\r\nfor all $n\\in\\N,\\,n\\ge 1$.\r\n\r\n Then, there exists $N\\in\\N$ such that %for all $n\\ge N$ one has\r\n\\begin{itemize}\r\n\\item[(i)] The sequence $ \\left(g(y_{n})+\\d_{n}\\|x_{n}-x_{n-1}\\|^2\\right)_{n\\ge N}$ is decreasing and $\\d_n>0$ for all $n\\ge N$.\r\n\\end{itemize}\r\nAssume that $g$ is bounded from below. Then, the following statements hold.\r\n\\begin{itemize}\r\n\\item[(ii)] The sequence $\\left(g(y_{n})+\\d_{n}\\|x_{n}-x_{n-1}\\|^2\\right)_{n\\in \\N}$ is convergent;\r\n\\item[(iii)] $\\sum_{n\\ge 1}\\|x_n-x_{n-1}\\|^2<+\\infty.$\r\n\\end{itemize}\r\n\\end{theorem}","sha256_utf8":"7b2df59d94f8cbd29a1cd4445148247d00b597cf2db341e3ea515f2d88de9184"}]},{"id":"S013","semantic_type":"proof","kind":"proof","title":"能量下降定理完整证明","source_units":[{"start":15396,"end":20498,"start_line":258,"end_line":348,"raw_tex":"\\begin{proof}\r\n\r\nFrom \\eqref{generaldiscrete} we have $\\n g(y_n)=\\frac{1}{s}(y_n-x_{n+1})$, hence\r\n$$\\<\\n g(y_n),y_{n+1}-y_n\\>=\\frac{1}{s}\\<y_n-x_{n+1},y_{n+1}-y_n\\>.$$\r\nNow, from Lemma \\ref{desc} we obtain\r\n$$g(y_{n+1})\\le g(y_n)+\\<\\n g(y_n),y_{n+1}-y_n\\>+\\frac{L_g}{2}\\|y_{n+1}-y_n\\|^2,$$\r\nconsequently we have\r\n\\begin{equation}\\label{e1}\r\ng(y_{n+1})-\\frac{L_g}{2}\\|y_{n+1}-y_n\\|^2\\le g(y_n)+ \\frac{1}{s}\\<y_n-x_{n+1},y_{n+1}-y_n\\>.\r\n\\end{equation}\r\n\r\nFurther,\r\n$$\\<y_n-x_{n+1},y_{n+1}-y_n\\>=-\\|y_{n+1}-y_n\\|^2+\\<y_{n+1}-x_{n+1},y_{n+1}-y_n\\>,$$\r\n and\r\n$$y_{n+1}-x_{n+1}=\\frac{\\b(n+1)}{n+\\a+1}(x_{n+1}-x_n),$$\r\nhence\r\n\\begin{equation}\\label{e2}\r\ng(y_{n+1})+\\left(\\frac{1}{s}-\\frac{L_g}{2}\\right)\\|y_{n+1}-y_n\\|^2\\le g(y_n)+ \\frac{\\frac{\\b(n+1)}{n+\\a+1}}{s}\\<x_{n+1}-x_n,y_{n+1}-y_n\\>.\r\n\\end{equation}\r\n\r\nSince $$y_{n+1}-y_n=\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}(x_{n+1}-x_n)-\\frac{\\b n}{n+\\a}(x_n-x_{n-1}),$$\r\nwe have,\r\n$$\\|y_{n+1}-y_n\\|^2=\\left\\|\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}(x_{n+1}-x_n)-\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right\\|^2=$$\r\n$$\\left(\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}\\right)^2\\|x_{n+1}-x_n\\|^2+\\left(\\frac{\\b n}{n+\\a}\\right)^2\\|x_n-x_{n-1}\\|^2-$$\r\n$$2\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}\\frac{\\b n}{n+\\a}\\<x_{n+1}-x_n,x_n-x_{n-1}\\>,$$\r\nand\r\n$$\\<x_{n+1}-x_n,y_{n+1}-y_n\\>=\\left\\<x_{n+1}-x_n,\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}(x_{n+1}-x_n)-\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right\\>=$$\r\n$$\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}\\|x_{n+1}-x_n\\|^2-\\frac{\\b n}{n+\\a}\\<x_{n+1}-x_n,x_n-x_{n-1}\\>.$$\r\n\r\nReplacing the above equalities in \\eqref{e2}, we obtain\r\n\r\n$$g(y_{n+1})+\\frac{(2-s L_g)\\left(\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}\\right)^2-2\\frac{\\b(n+1)((1+\\b)n+\\a+\\b+1)}{(n+\\a+1)^2}}{2s}\\|x_{n+1}-x_n\\|^2\\le$$\r\n$$g(y_n)-\\frac{(2-s L_g)\\left(\\frac{\\b n}{n+\\a}\\right)^2}{2s}\\|x_{n}-x_{n-1}\\|^2+$$\r\n$$\\frac{(2-s L_g)\\frac{\\b n}{n+\\a}\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}-\\frac{\\b n}{n+\\a}\\frac{\\b(n+1)}{n+\\a+1}}{s}\\<x_{n+1}-x_n,x_n-x_{n-1}\\>.$$\r\n\r\nFor simplicity let %$$A_{n}=\\frac{(2-s L_g)\\left(\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}\\right)^2-2\\frac{\\b(n+1)((1+\\b)n+\\a+\\b+1)}{(n+\\a+1)^2}}{2s},$$\r\n$$B_n=\\frac{(2-s L_g)\\left(\\frac{\\b n}{n+\\a}\\right)^2}{2s}$$\r\n%and $$C_n=\\frac{(2-s L_g)\\frac{\\b n}{n+\\a}\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}-\\frac{\\b n}{n+\\a}\\frac{\\b(n+1)}{n+\\a+1}}{2s}$$\r\nfor all $n\\in\\N.$\r\n\r\n Hence we have\r\n $$g(y_{n+1})+A_{n}\\|x_{n+1}-x_n\\|^2-2C_n\\<x_{n+1}-x_n,x_n-x_{n-1}\\>\\le g(y_n)-B_n\\|x_{n}-x_{n-1}\\|^2.$$\r\n\r\nBy using the equality\r\n\\begin{equation}\\label{e5}\r\n-2\\<x_{n+1}-x_n,x_n-x_{n-1}\\>=\\|x_{n+1}+x_{n-1}-2x_n\\|^2-\\|x_{n+1}-x_n\\|^2-\\|x_n-x_{n-1}\\|^2\r\n\\end{equation}\r\nwe obtain\r\n$$g(y_{n+1})+(A_n-C_n)\\|x_{n+1}-x_n\\|^2+C_n\\|x_{n+1}+x_{n-1}-2x_n\\|^2\\le g(y_n)+(C_n-B_n)\\|x_n-x_{n-1}\\|^2.$$\r\n\r\nNote that $A_{n}-C_n=\\d_{n+1}$ and let us denote $\\Delta_n=B_n+A_{n-1}-C_{n-1}-C_n$. Consequently the following inequality holds.\r\n\\begin{equation}\\label{e6}\r\nC_n\\|x_{n+1}+x_{n-1}-2x_n\\|^2+\\Delta_n\\|x_n-x_{n-1}\\|^2\\le (g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2)- (g(y_{n+1})+\\d_{n+1}\\|x_{n+1}-x_n\\|^2).\r\n\\end{equation}\r\n\r\n\r\nSince $0<\\b<1$ and $s<\\frac{2(1-\\b)}{L_g},$  we have\r\n$$\\lim_{n\\To+\\infty}A_n=\\frac{(2-s L_g)(\\b+1)^2-2\\b-2\\b^2}{2s}>0,$$\r\n$$\\lim_{n\\To+\\infty}B_n=\\frac{(2-s L_g)\\b^2}{2s}>0,$$\r\n$$\\lim_{n\\To+\\infty}C_n= \\frac{(2-s L_g)(\\b^2+\\b)-\\b^2}{2s} >0,$$\r\n$$\\lim_{n\\To+\\infty}\\Delta_n=\\frac{2-s L_g-2\\b}{2s}>0,$$\r\nand\r\n$$\\lim_{n\\To+\\infty}\\d_n=\\frac{2-\\b^2-s L_g(\\b+1)}{2s}>0.$$\r\n\r\nHence, there exists $N\\in\\N$ and $C>0,\\,D>0$ such that for all $n\\ge N$ one has\r\n$$C_n\\ge C,\\, \\Delta_n\\ge D\\mbox{ and }\\d_n>0$$ which, in the view of \\eqref{e6}, shows (i), that is,  the sequence $g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2$ is decreasing for $n\\ge N.$\r\n\r\nAssume now that $g$ is bounded from below. %Since $\\lim_{n\\To+\\infty}\\Delta_n>0$ there exists $D>0$ and $p\\in\\N$ such that $\\Delta_n\\ge D$ for all $n\\ge p.$\r\n By using \\eqref{e6} again, we obtain\r\n $$0\\le C\\|x_{n+1}+x_{n-1}-2x_n\\|^2+D\\|x_n-x_{n-1}\\|^2\\le (g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2)- (g(y_{n+1})+\\d_{n+1}\\|x_{n+1}-x_n\\|^2),$$\r\nfor all $n\\ge N,$ or more convenient, that\r\n \\begin{equation}\\label{ineq}\r\n0\\le D\\|x_n-x_{n-1}\\|^2\\le (g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2)- (g(y_{n+1})+\\d_{n+1}\\|x_{n+1}-x_n\\|^2),\r\n\\end{equation}\r\nfor all $n\\ge N.$ Let $r>N.$ By summing up the latter relation  we have\r\n$$\r\nD\\sum_{n=N}^r\\|x_n-x_{n-1}\\|^2\\le (g(y_N)+\\d_N\\|x_N-x_{N-1}\\|^2)-(g(y_{r+1})+\\d_{r+1}\\|x_{r+1}-x_r\\|^2)\r\n$$\r\nwhich leads to\r\n\\begin{equation}\\label{forcoercive}\r\ng(y_{r+1})+D\\sum_{n=N}^r\\|x_n-x_{n-1}\\|^2\\le g(y_N)+\\d_N\\|x_N-x_{N-1}\\|^2.\r\n\\end{equation}\r\nNow,  taking into account that $g$ is bounded from below, by letting $r\\To+\\infty$  we obtain\r\n$$\\sum_{n=N}^{\\infty}\\|x_n-x_{n-1}\\|^2\\le+\\infty$$ which proves (iii).\r\n\r\nThe latter relation also shows that\r\n$$\\lim_{n\\To+\\infty}\\|x_n-x_{n-1}\\|^2=0,$$\r\nhence $$\\lim_{n\\To+\\infty}\\d_n\\|x_n-x_{n-1}\\|^2=0.$$\r\nBut then, from the fact that $g$ is bounded from below we obtain that the sequence $g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2$ is bounded from below. On the other hand, from (i) we have that  the sequence $g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2$ is decreasing for $n\\ge N,$ hence there exists\r\n$$\\lim_{n\\To+\\infty} g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2\\in\\R.$$\r\n\\end{proof}","sha256_utf8":"a762241acfa3d77db6d7f897d8279aa70c7b02509e36d0fc6b64382facd19af4"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"梯度恒等式与下降估计","source_units":[{"start":15396,"end":15848,"start_line":258,"end_line":267,"raw_tex":"\\begin{proof}\r\n\r\nFrom \\eqref{generaldiscrete} we have $\\n g(y_n)=\\frac{1}{s}(y_n-x_{n+1})$, hence\r\n$$\\<\\n g(y_n),y_{n+1}-y_n\\>=\\frac{1}{s}\\<y_n-x_{n+1},y_{n+1}-y_n\\>.$$\r\nNow, from Lemma \\ref{desc} we obtain\r\n$$g(y_{n+1})\\le g(y_n)+\\<\\n g(y_n),y_{n+1}-y_n\\>+\\frac{L_g}{2}\\|y_{n+1}-y_n\\|^2,$$\r\nconsequently we have\r\n\\begin{equation}\\label{e1}\r\ng(y_{n+1})-\\frac{L_g}{2}\\|y_{n+1}-y_n\\|^2\\le g(y_n)+ \\frac{1}{s}\\<y_n-x_{n+1},y_{n+1}-y_n\\>.\r\n\\end{equation}\r\n","sha256_utf8":"d7f0c5d63ca69ccbe0466419bad0c3046791888b86e5f40c5c48adcd2e2ac600"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"分离惯性内积项","source_units":[{"start":15850,"end":16200,"start_line":269,"end_line":276,"raw_tex":"Further,\r\n$$\\<y_n-x_{n+1},y_{n+1}-y_n\\>=-\\|y_{n+1}-y_n\\|^2+\\<y_{n+1}-x_{n+1},y_{n+1}-y_n\\>,$$\r\n and\r\n$$y_{n+1}-x_{n+1}=\\frac{\\b(n+1)}{n+\\a+1}(x_{n+1}-x_n),$$\r\nhence\r\n\\begin{equation}\\label{e2}\r\ng(y_{n+1})+\\left(\\frac{1}{s}-\\frac{L_g}{2}\\right)\\|y_{n+1}-y_n\\|^2\\le g(y_n)+ \\frac{\\frac{\\b(n+1)}{n+\\a+1}}{s}\\<x_{n+1}-x_n,y_{n+1}-y_n\\>.\r\n\\end{equation}\r\n","sha256_utf8":"40cfee39a526664ad3d3f86398397dd9474b111c9f5328c00263de4711efbb14"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"展开外推点差与内积","source_units":[{"start":16202,"end":16868,"start_line":278,"end_line":285,"raw_tex":"Since $$y_{n+1}-y_n=\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}(x_{n+1}-x_n)-\\frac{\\b n}{n+\\a}(x_n-x_{n-1}),$$\r\nwe have,\r\n$$\\|y_{n+1}-y_n\\|^2=\\left\\|\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}(x_{n+1}-x_n)-\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right\\|^2=$$\r\n$$\\left(\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}\\right)^2\\|x_{n+1}-x_n\\|^2+\\left(\\frac{\\b n}{n+\\a}\\right)^2\\|x_n-x_{n-1}\\|^2-$$\r\n$$2\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}\\frac{\\b n}{n+\\a}\\<x_{n+1}-x_n,x_n-x_{n-1}\\>,$$\r\nand\r\n$$\\<x_{n+1}-x_n,y_{n+1}-y_n\\>=\\left\\<x_{n+1}-x_n,\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}(x_{n+1}-x_n)-\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right\\>=$$\r\n$$\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}\\|x_{n+1}-x_n\\|^2-\\frac{\\b n}{n+\\a}\\<x_{n+1}-x_n,x_n-x_{n-1}\\>.$$\r\n","sha256_utf8":"af6f8ec42aa1ae603d9588ab4e4222480ed92ccf2e59d46b1716214a8dd44129"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"整理能量系数","source_units":[{"start":16870,"end":17773,"start_line":287,"end_line":299,"raw_tex":"Replacing the above equalities in \\eqref{e2}, we obtain\r\n\r\n$$g(y_{n+1})+\\frac{(2-s L_g)\\left(\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}\\right)^2-2\\frac{\\b(n+1)((1+\\b)n+\\a+\\b+1)}{(n+\\a+1)^2}}{2s}\\|x_{n+1}-x_n\\|^2\\le$$\r\n$$g(y_n)-\\frac{(2-s L_g)\\left(\\frac{\\b n}{n+\\a}\\right)^2}{2s}\\|x_{n}-x_{n-1}\\|^2+$$\r\n$$\\frac{(2-s L_g)\\frac{\\b n}{n+\\a}\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}-\\frac{\\b n}{n+\\a}\\frac{\\b(n+1)}{n+\\a+1}}{s}\\<x_{n+1}-x_n,x_n-x_{n-1}\\>.$$\r\n\r\nFor simplicity let %$$A_{n}=\\frac{(2-s L_g)\\left(\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}\\right)^2-2\\frac{\\b(n+1)((1+\\b)n+\\a+\\b+1)}{(n+\\a+1)^2}}{2s},$$\r\n$$B_n=\\frac{(2-s L_g)\\left(\\frac{\\b n}{n+\\a}\\right)^2}{2s}$$\r\n%and $$C_n=\\frac{(2-s L_g)\\frac{\\b n}{n+\\a}\\frac{(1+\\b)n+\\a+\\b+1}{n+\\a+1}-\\frac{\\b n}{n+\\a}\\frac{\\b(n+1)}{n+\\a+1}}{2s}$$\r\nfor all $n\\in\\N.$\r\n\r\n Hence we have\r\n $$g(y_{n+1})+A_{n}\\|x_{n+1}-x_n\\|^2-2C_n\\<x_{n+1}-x_n,x_n-x_{n-1}\\>\\le g(y_n)-B_n\\|x_{n}-x_{n-1}\\|^2.$$\r\n","sha256_utf8":"abddb51f8a4d104a9c1b627a484ba63d18d59a760c8ad28d3c9ede9c72a20950"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"建立带耗散项的能量递推","source_units":[{"start":17775,"end":18367,"start_line":301,"end_line":311,"raw_tex":"By using the equality\r\n\\begin{equation}\\label{e5}\r\n-2\\<x_{n+1}-x_n,x_n-x_{n-1}\\>=\\|x_{n+1}+x_{n-1}-2x_n\\|^2-\\|x_{n+1}-x_n\\|^2-\\|x_n-x_{n-1}\\|^2\r\n\\end{equation}\r\nwe obtain\r\n$$g(y_{n+1})+(A_n-C_n)\\|x_{n+1}-x_n\\|^2+C_n\\|x_{n+1}+x_{n-1}-2x_n\\|^2\\le g(y_n)+(C_n-B_n)\\|x_n-x_{n-1}\\|^2.$$\r\n\r\nNote that $A_{n}-C_n=\\d_{n+1}$ and let us denote $\\Delta_n=B_n+A_{n-1}-C_{n-1}-C_n$. Consequently the following inequality holds.\r\n\\begin{equation}\\label{e6}\r\nC_n\\|x_{n+1}+x_{n-1}-2x_n\\|^2+\\Delta_n\\|x_n-x_{n-1}\\|^2\\le (g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2)- (g(y_{n+1})+\\d_{n+1}\\|x_{n+1}-x_n\\|^2).\r\n\\end{equation}\r\n","sha256_utf8":"1b2075ca95e28aaf314aa1859f808b8e1282dfbe98ce0af65d556124f8d13ada"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"系数极限保证最终正性","source_units":[{"start":18371,"end":18999,"start_line":314,"end_line":323,"raw_tex":"Since $0<\\b<1$ and $s<\\frac{2(1-\\b)}{L_g},$  we have\r\n$$\\lim_{n\\To+\\infty}A_n=\\frac{(2-s L_g)(\\b+1)^2-2\\b-2\\b^2}{2s}>0,$$\r\n$$\\lim_{n\\To+\\infty}B_n=\\frac{(2-s L_g)\\b^2}{2s}>0,$$\r\n$$\\lim_{n\\To+\\infty}C_n= \\frac{(2-s L_g)(\\b^2+\\b)-\\b^2}{2s} >0,$$\r\n$$\\lim_{n\\To+\\infty}\\Delta_n=\\frac{2-s L_g-2\\b}{2s}>0,$$\r\nand\r\n$$\\lim_{n\\To+\\infty}\\d_n=\\frac{2-\\b^2-s L_g(\\b+1)}{2s}>0.$$\r\n\r\nHence, there exists $N\\in\\N$ and $C>0,\\,D>0$ such that for all $n\\ge N$ one has\r\n$$C_n\\ge C,\\, \\Delta_n\\ge D\\mbox{ and }\\d_n>0$$ which, in the view of \\eqref{e6}, shows (i), that is,  the sequence $g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2$ is decreasing for $n\\ge N.$\r\n","sha256_utf8":"e4ce512c02649b20b1431ae4b09791c97203dfad5904104e05caece6efdab974"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"累加耗散得到平方可和","source_units":[{"start":19001,"end":20020,"start_line":325,"end_line":341,"raw_tex":"Assume now that $g$ is bounded from below. %Since $\\lim_{n\\To+\\infty}\\Delta_n>0$ there exists $D>0$ and $p\\in\\N$ such that $\\Delta_n\\ge D$ for all $n\\ge p.$\r\n By using \\eqref{e6} again, we obtain\r\n $$0\\le C\\|x_{n+1}+x_{n-1}-2x_n\\|^2+D\\|x_n-x_{n-1}\\|^2\\le (g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2)- (g(y_{n+1})+\\d_{n+1}\\|x_{n+1}-x_n\\|^2),$$\r\nfor all $n\\ge N,$ or more convenient, that\r\n \\begin{equation}\\label{ineq}\r\n0\\le D\\|x_n-x_{n-1}\\|^2\\le (g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2)- (g(y_{n+1})+\\d_{n+1}\\|x_{n+1}-x_n\\|^2),\r\n\\end{equation}\r\nfor all $n\\ge N.$ Let $r>N.$ By summing up the latter relation  we have\r\n$$\r\nD\\sum_{n=N}^r\\|x_n-x_{n-1}\\|^2\\le (g(y_N)+\\d_N\\|x_N-x_{N-1}\\|^2)-(g(y_{r+1})+\\d_{r+1}\\|x_{r+1}-x_r\\|^2)\r\n$$\r\nwhich leads to\r\n\\begin{equation}\\label{forcoercive}\r\ng(y_{r+1})+D\\sum_{n=N}^r\\|x_n-x_{n-1}\\|^2\\le g(y_N)+\\d_N\\|x_N-x_{N-1}\\|^2.\r\n\\end{equation}\r\nNow,  taking into account that $g$ is bounded from below, by letting $r\\To+\\infty$  we obtain\r\n$$\\sum_{n=N}^{\\infty}\\|x_n-x_{n-1}\\|^2\\le+\\infty$$ which proves (iii).\r\n","sha256_utf8":"831ad79dfea90acccd95428f23b947b679c3f5e418e397cd646bde291be41c59"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"证明能量序列收敛","source_units":[{"start":20022,"end":20498,"start_line":343,"end_line":348,"raw_tex":"The latter relation also shows that\r\n$$\\lim_{n\\To+\\infty}\\|x_n-x_{n-1}\\|^2=0,$$\r\nhence $$\\lim_{n\\To+\\infty}\\d_n\\|x_n-x_{n-1}\\|^2=0.$$\r\nBut then, from the fact that $g$ is bounded from below we obtain that the sequence $g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2$ is bounded from below. On the other hand, from (i) we have that  the sequence $g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2$ is decreasing for $n\\ge N,$ hence there exists\r\n$$\\lim_{n\\To+\\infty} g(y_n)+\\d_n\\|x_n-x_{n-1}\\|^2\\in\\R.$$\r\n\\end{proof}","sha256_utf8":"15b8c793b499e7116d9bcc49ec84a79d9de96c7b61ad6a0b6f8594673c4bf1b4"}]},{"id":"S022","semantic_type":"lemma","kind":"lemma","title":"正则化能量与聚点性质","source_units":[{"start":21195,"end":22691,"start_line":363,"end_line":382,"raw_tex":"\\begin{lemma}\\label{regularization} In the settings of problem \\eqref{opt-pb}, for some starting points $x_0=y_0\\in\\R^m$ consider the sequences $(x_n)_{n\\in\\N},\\,(y_n)_{n\\in\\N}$ generated by Algorithm \\eqref{generaldiscrete}. Assume that $g$ is bounded from below and consider the function\r\n$$H:\\R^m\\times\\R^m\\To\\R,\\,H(x,y)=g(x)+\\frac12\\|y-x\\|^2.$$\r\nConsider further the sequence\r\n$$u_n=\\sqrt{2\\d_n}(x_{n}-x_{n-1})+y_n\\mbox{, for all }n\\in \\N,$$\r\nwhere $\\d_n$ was defined in Theorem \\ref{decreasing}. Then, the following statements hold true.\r\n\\begin{itemize}\r\n\\item[(i)] $\\omega((u_n)_{n\\in\\N})=\\omega((y_n)_{n\\in\\N})=\\omega((x_n)_{n\\in\\N})\\subseteq\\crit g$;\r\n\\item[(ii)] There exists and is finite the limit $\\lim_{n\\To+\\infty}H(y_n,u_n)$;\r\n\\item[(iii)] $\\omega((y_n,u_n)_{n\\in\\N})\\subseteq\\crit H=\\{(x,x)\\in\\R^m\\times\\R^m:x\\in\\crit g\\}$;\r\n\\item[(iv)] $\\|\\n H(y_n,u_n)\\|\\le \\frac{1}{s}\\|x_{n+1}-x_n\\|+\\left(\\frac{\\b  n}{s(n+\\a)}+2\\sqrt{2\\d_n}\\right)\\|x_{n}-x_{n-1}\\|$ for all $n\\in\\N$;\r\n\\item[(v)]  $\\|\\n H(y_n,u_n)\\|^2\\le \\frac{2}{s^2}\\|x_{n+1}-x_n\\|^2+2\\left(\\left(\\frac{\\b n}{s(n+\\a)}-\\sqrt{2\\d_n}\\right)^2+\\d_n\\right)\\|x_{n}-x_{n-1}\\|^2$ for all $n\\in\\N$;\r\n\\item[(vi)] $H$ is finite and constant on $\\omega((y_n,u_n)_{n\\in\\N}).$\r\n\\end{itemize}\r\n\r\nAssume that $(x_n)_{n\\in\\N}$ is bounded. Then,\r\n\\begin{itemize}\r\n\\item[(vii)] $\\omega((y_n, u_n)_{n\\in\\N})$ is nonempty and compact;\r\n\\item[(viii)] $\\lim_{n\\To+\\infty} \\dist((y_n,u_n),\\omega((y_n,u_n)_{n\\in\\N}))=0.$\r\n\\end{itemize}\r\n\\end{lemma}","sha256_utf8":"2cbcb4f3f8accb768126370ad94c1f75f8b56c15a796eeeaeaf7ca1d4cc03e7b"}]},{"id":"S023","semantic_type":"proof","kind":"proof","title":"正则化引理完整证明","source_units":[{"start":22693,"end":27515,"start_line":383,"end_line":446,"raw_tex":"\\begin{proof}\r\n(i) Let $\\ol x\\in\\omega((x_n)_{n\\in\\N}).$ Then, there exists a subsequence $(x_{n_k})_{k\\in \\N}$ of $(x_n)_{n\\in\\N}$ such that\r\n$$\\lim_{k\\to+\\infty}x_{n_k}=\\ol x.$$\r\nSince by \\eqref{e7} $\\lim_{n\\To+\\infty}(x_n-x_{n-1})=0$ and the sequences $(\\sqrt{2\\d_n})_{n\\in\\N},\\,\\left(\\frac{\\b n}{n+\\a}\\right)_{n\\in\\N}$ converge, we obtain that\r\n$$\\lim_{k\\to+\\infty}y_{n_k}=\\lim_{k\\to+\\infty}u_{n_k}=\\lim_{k\\to+\\infty}x_{n_k}=\\ol x,$$\r\nwhich shows that $$\\omega((x_n)_{n\\in\\N})\\subseteq \\omega((u_n)_{n\\in\\N})\\mbox{ and }\\omega((x_n)_{n\\in\\N})\\subseteq \\omega((y_n)_{n\\in\\N}).$$\r\nFurther from \\eqref{generaldiscrete}, the continuity of $\\n g$ and \\eqref{e7}, we obtain that\r\n$$\\n g(\\ol x)=\\lim_{k\\To +\\infty} \\n g(y_{n_k})=\\frac{1}{s}\\lim_{k\\To +\\infty}(y_{n_k}-x_{n_k+1})=$$\r\n$$\\frac{1}{s}\\lim_{k\\To +\\infty}\\left[(x_{n_k}-x_{n_k+1})+\\frac{\\b n_k}{n_k+\\a}(x_{n_k}-x_{n_k-1})\\right]=0.$$\r\nHence $\\omega((x_n)_{n\\in\\N})\\subseteq\\crit g.$\r\nConversely, if $\\ol u\\in \\omega((u_n)_{n\\in\\N})$ then, from \\eqref{e7} results that $\\ol u\\in \\omega((y_n)_{n\\in\\N})$ and $\\ol u\\in \\omega((x_n)_{n\\in\\N}).$\r\nHence,\r\n$$\\omega((u_n)_{n\\in\\N})=\\omega((y_n)_{n\\in\\N})=\\omega((x_n)_{n\\in\\N})\\subseteq\\crit g.$$\r\n\r\n(ii) is nothing else than (ii) in Theorem \\ref{decreasing}.\r\n\r\nFor (iii) observe that  $\\n H(x,y)=(\\n g(x)+x-y,y-x)$, hence, $\\n H(x,y)=0$ leads to\r\n$x=y$ and $\\n g(x)=0.$\r\nConsequently\r\n$$\\crit H=\\{(x,x)\\in\\R^m\\times\\R^m: x\\in\\crit g\\}.$$\r\n\r\nFurther, consider $(\\ol y,\\ol u)\\in \\omega((y_n,u_n)_{n\\in\\N}).$ Then, there exists $(y_{n_k},u_{n_k})_{k\\in\\N}\\subseteq (y_n,u_n)_{n\\in\\N}$ such that\r\n$$(\\ol y,\\ol u)=\\lim_{k\\To+\\infty}(y_{n_k},u_{n_k})=\\lim_{k\\To+\\infty}(x_{n_k},x_{n_k})=(\\ol x,\\ol x).$$\r\nHence, $\\ol u=\\ol y=\\ol x\\in\\omega((x_n)_{n\\in\\N})\\subseteq \\crit g$ and $(\\ol x,\\ol x)\\in\\crit H.$\r\n\r\n(iv) By using the 1-norm of $\\R^m\\times\\R^m$ and \\eqref{generaldiscrete}, for every $n\\in \\N$ we have\r\n $$\\|\\n H(y_n,u_n)\\|\\le\\|\\n H(y_n,u_n)\\|_1=\\|(\\n g(y_n)+y_n-u_n,u_n-y_n)\\|_1=\\|\\n g(y_n)+y_n-u_n\\|+\\|u_n-y_n\\|\\le$$\r\n $$\\|\\n g(y_n)\\|+2\\|\\sqrt{2\\d_n}(x_{n}-x_{n-1})\\|=$$\r\n $$\\frac{1}{s}\\left\\|\\left(x_n+\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right)-x_{n+1}\\right\\|+2\\sqrt{2\\d_n}\\|x_{n}-x_{n-1}\\|\\le$$\r\n $$\\frac{1}{s}\\|x_{n+1}-x_n\\|+\\left(\\frac{\\b  n}{s(n+\\a)}+2\\sqrt{2\\d_n}\\right)\\|x_{n}-x_{n-1}\\|.$$\r\n\r\n(v) We use the euclidian norm of $\\R^m\\times\\R^m$, that is $\\|(x,y)\\|=\\sqrt{\\|x\\|^2+\\|y\\|^2}$ for all $(x,y)\\in\\R^m\\times\\R^m.$\r\nWe have:\r\n$$\\|\\n H(y_n,u_n)\\|^2=\\|(\\n g(y_n)+y_n-u_n,u_n-y_n)\\|^2=\\|\\n g(y_n)+y_n-u_n\\|^2+\\|u_n-y_n\\|^2=$$\r\n$$\\left\\|\\frac1s(x_n-x_{n+1})+\\left(\\frac{\\b n}{s(n+\\a)}-\\sqrt{2\\d_n}\\right)(x_n-x_{n-1})\\right\\|^2+2\\d_n\\|x_n-x_{n-1}\\|^2\\le$$\r\n$$\\frac{2}{s^2}\\|x_{n+1}-x_n\\|^2+2\\left(\\left(\\frac{\\b n}{s(n+\\a)}-\\sqrt{2\\d_n}\\right)^2+\\d_n\\right)\\|x_{n}-x_{n-1}\\|^2$$ for all $n\\in\\N.$\r\n\r\n(vi) follows directly from (ii).\r\n\r\n Assume now that $(x_n)_{n\\in\\N}$ is bounded and let us prove (vii), (see also \\cite{BCL-AA}). Obviously $(y_n,u_n)_{n\\in\\N}$ is bounded, hence according to Weierstrass Theorem $\\omega((y_n, u_n)_{n\\in\\N}),$ (and also $\\omega((x_n)_{n\\in\\N})$), is nonempty. It remains to show that $\\omega((y_n, u_n)_{n\\in\\N})$ is closed. From (i) and the proof of (iii) we have\r\n \\begin{equation}\\label{e8}\r\n\\omega((y_n, u_n)_{n\\in\\N})=\\{(\\ol x,\\ol x)\\in\\R^m\\times \\R^m: \\ol x\\in \\omega((x_n)_{n\\in\\N})\\}.\r\n\\end{equation}\r\n Hence, it is enough to show that $\\omega((x_n)_{n\\in\\N})$ is closed.\r\n\r\n\r\n Let be $(\\ol x_p)_{p\\in\\N}\\subseteq \\omega((x_n)_{n\\in\\N})$ and assume that $\\lim_{p\\To+\\infty}\\ol x_p=x^*.$ We show that $x^*\\in\\omega((x_n)_{n\\in\\N}).$ Obviously, for every $p\\in\\N$ there exists a sequence of natural numbers $n_k^p\\To+\\infty,\\,k\\To+\\infty$, such that\r\n$$\\lim_{k\\To+\\infty}x_{n_k^p}=\\ol x_p.$$\r\n\r\nLet be $\\e >0$. Since $\\lim_{p\\To+\\infty}\\ol x_p=x^*,$  there exists $P(\\e) \\in \\N$ such that for every $p\\ge P(\\e)$ it holds\r\n$$\\|\\ol x_p-x^*\\|<\\frac\\e2.$$\r\nLet $p\\in\\N$ be fixed. Since $\\lim_{k\\To+\\infty}x_{n_k^p}=\\ol x_p,$ there exists $k(p,\\e)\\in\\N$ such that for every $k\\ge k(p,\\e)$ it holds $$\\|x_{n_k^p}-\\ol x_p\\|<\\frac\\e2.$$\r\nLet be $k_p \\geq k(p,\\varepsilon)$ such that $n_{k_p}^p >p$. Obviously $n_{k_p}^p \\To \\infty$ as $p \\To +\\infty$ and for every $p \\geq P(\\e)$\r\n$$\\|x_{n_{k_p}^p}-x^*\\|<\\e.$$\r\nHence\r\n$$\\lim_{p\\To+\\infty}x_{n_{k_p}^p}=x^*,$$\r\nthus $x^*\\in \\omega((x_n)_{n\\in\\N}).$\r\n\r\n(viii) By using \\eqref{e8} we have\r\n$$\\lim_{n\\To+\\infty} \\dist((y_n,u_n),\\omega((y_n,u_n)_{n\\in\\N}))=\\lim_{n\\To+\\infty}\\inf_{\\ol x\\in\\omega((x_n)_{n\\in\\N})}\\|(y_n,u_n)-(\\ol x,\\ol x)\\|.$$\r\nSince there exists the subsequences $(y_{n_k})_{k\\in\\N}$ and $(u_{n_k})_{k\\in\\N}$ such that\r\n$\\lim_{k\\To\\infty}y_{n_k}=\\lim_{k\\To\\infty}u_{n_k}=\\ol x_0\\in \\omega((x_n)_{n\\in\\N})$ it is straightforward that\r\n$$\\lim_{n\\To+\\infty} \\dist((y_n,u_n),\\omega((y_n,u_n)_{n\\in\\N}))=0.$$\r\n\\end{proof}","sha256_utf8":"05c27144b70366d523bc0543e918999ff2144c036f4590df4c89d4073d799e26"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"聚点集相同且聚点临界","source_units":[{"start":22693,"end":23891,"start_line":383,"end_line":395,"raw_tex":"\\begin{proof}\r\n(i) Let $\\ol x\\in\\omega((x_n)_{n\\in\\N}).$ Then, there exists a subsequence $(x_{n_k})_{k\\in \\N}$ of $(x_n)_{n\\in\\N}$ such that\r\n$$\\lim_{k\\to+\\infty}x_{n_k}=\\ol x.$$\r\nSince by \\eqref{e7} $\\lim_{n\\To+\\infty}(x_n-x_{n-1})=0$ and the sequences $(\\sqrt{2\\d_n})_{n\\in\\N},\\,\\left(\\frac{\\b n}{n+\\a}\\right)_{n\\in\\N}$ converge, we obtain that\r\n$$\\lim_{k\\to+\\infty}y_{n_k}=\\lim_{k\\to+\\infty}u_{n_k}=\\lim_{k\\to+\\infty}x_{n_k}=\\ol x,$$\r\nwhich shows that $$\\omega((x_n)_{n\\in\\N})\\subseteq \\omega((u_n)_{n\\in\\N})\\mbox{ and }\\omega((x_n)_{n\\in\\N})\\subseteq \\omega((y_n)_{n\\in\\N}).$$\r\nFurther from \\eqref{generaldiscrete}, the continuity of $\\n g$ and \\eqref{e7}, we obtain that\r\n$$\\n g(\\ol x)=\\lim_{k\\To +\\infty} \\n g(y_{n_k})=\\frac{1}{s}\\lim_{k\\To +\\infty}(y_{n_k}-x_{n_k+1})=$$\r\n$$\\frac{1}{s}\\lim_{k\\To +\\infty}\\left[(x_{n_k}-x_{n_k+1})+\\frac{\\b n_k}{n_k+\\a}(x_{n_k}-x_{n_k-1})\\right]=0.$$\r\nHence $\\omega((x_n)_{n\\in\\N})\\subseteq\\crit g.$\r\nConversely, if $\\ol u\\in \\omega((u_n)_{n\\in\\N})$ then, from \\eqref{e7} results that $\\ol u\\in \\omega((y_n)_{n\\in\\N})$ and $\\ol u\\in \\omega((x_n)_{n\\in\\N}).$\r\nHence,\r\n$$\\omega((u_n)_{n\\in\\N})=\\omega((y_n)_{n\\in\\N})=\\omega((x_n)_{n\\in\\N})\\subseteq\\crit g.$$\r\n","sha256_utf8":"aabd13f95bef7505ec126651ed39666c3a7572c9d1f411c794c4aa3a6c0feba4"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"正则化能量极限有限","source_units":[{"start":23893,"end":23954,"start_line":397,"end_line":397,"raw_tex":"(ii) is nothing else than (ii) in Theorem \\ref{decreasing}.\r\n","sha256_utf8":"efc419513168a40c12abd5c0bb99f3066644dba8231176e895710faefc53526c"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"刻画正则化临界点","source_units":[{"start":23956,"end":24495,"start_line":399,"end_line":406,"raw_tex":"For (iii) observe that  $\\n H(x,y)=(\\n g(x)+x-y,y-x)$, hence, $\\n H(x,y)=0$ leads to\r\n$x=y$ and $\\n g(x)=0.$\r\nConsequently\r\n$$\\crit H=\\{(x,x)\\in\\R^m\\times\\R^m: x\\in\\crit g\\}.$$\r\n\r\nFurther, consider $(\\ol y,\\ol u)\\in \\omega((y_n,u_n)_{n\\in\\N}).$ Then, there exists $(y_{n_k},u_{n_k})_{k\\in\\N}\\subseteq (y_n,u_n)_{n\\in\\N}$ such that\r\n$$(\\ol y,\\ol u)=\\lim_{k\\To+\\infty}(y_{n_k},u_{n_k})=\\lim_{k\\To+\\infty}(x_{n_k},x_{n_k})=(\\ol x,\\ol x).$$\r\nHence, $\\ol u=\\ol y=\\ol x\\in\\omega((x_n)_{n\\in\\N})\\subseteq \\crit g$ and $(\\ol x,\\ol x)\\in\\crit H.$\r\n","sha256_utf8":"35e70f8696464786895df2e8ee33a461a8e8fe29dbfb5efa4b98aed5beeaff01"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"梯度范数的一次估计","source_units":[{"start":24497,"end":24993,"start_line":408,"end_line":412,"raw_tex":"(iv) By using the 1-norm of $\\R^m\\times\\R^m$ and \\eqref{generaldiscrete}, for every $n\\in \\N$ we have\r\n $$\\|\\n H(y_n,u_n)\\|\\le\\|\\n H(y_n,u_n)\\|_1=\\|(\\n g(y_n)+y_n-u_n,u_n-y_n)\\|_1=\\|\\n g(y_n)+y_n-u_n\\|+\\|u_n-y_n\\|\\le$$\r\n $$\\|\\n g(y_n)\\|+2\\|\\sqrt{2\\d_n}(x_{n}-x_{n-1})\\|=$$\r\n $$\\frac{1}{s}\\left\\|\\left(x_n+\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right)-x_{n+1}\\right\\|+2\\sqrt{2\\d_n}\\|x_{n}-x_{n-1}\\|\\le$$\r\n $$\\frac{1}{s}\\|x_{n+1}-x_n\\|+\\left(\\frac{\\b  n}{s(n+\\a)}+2\\sqrt{2\\d_n}\\right)\\|x_{n}-x_{n-1}\\|.$$\r\n","sha256_utf8":"bd5f66c77fe8879a9f79f9bf56a2f1b7cfb3d86a22f2a3026c7d5fb04a019d92"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"梯度范数的平方估计","source_units":[{"start":24995,"end":25502,"start_line":414,"end_line":418,"raw_tex":"(v) We use the euclidian norm of $\\R^m\\times\\R^m$, that is $\\|(x,y)\\|=\\sqrt{\\|x\\|^2+\\|y\\|^2}$ for all $(x,y)\\in\\R^m\\times\\R^m.$\r\nWe have:\r\n$$\\|\\n H(y_n,u_n)\\|^2=\\|(\\n g(y_n)+y_n-u_n,u_n-y_n)\\|^2=\\|\\n g(y_n)+y_n-u_n\\|^2+\\|u_n-y_n\\|^2=$$\r\n$$\\left\\|\\frac1s(x_n-x_{n+1})+\\left(\\frac{\\b n}{s(n+\\a)}-\\sqrt{2\\d_n}\\right)(x_n-x_{n-1})\\right\\|^2+2\\d_n\\|x_n-x_{n-1}\\|^2\\le$$\r\n$$\\frac{2}{s^2}\\|x_{n+1}-x_n\\|^2+2\\left(\\left(\\frac{\\b n}{s(n+\\a)}-\\sqrt{2\\d_n}\\right)^2+\\d_n\\right)\\|x_{n}-x_{n-1}\\|^2$$ for all $n\\in\\N.$\r\n","sha256_utf8":"943afc979537cdd6f9476491989907f2eaf76a9b739832955b77dcdbbd8b666f"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"聚点集上能量恒定","source_units":[{"start":25504,"end":25538,"start_line":420,"end_line":420,"raw_tex":"(vi) follows directly from (ii).\r\n","sha256_utf8":"cc52825db80895af97bcd69ac0534a0c97208449252a35c3876e6b891123c354"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"聚点集非空与对角表示","source_units":[{"start":25540,"end":26119,"start_line":422,"end_line":426,"raw_tex":" Assume now that $(x_n)_{n\\in\\N}$ is bounded and let us prove (vii), (see also \\cite{BCL-AA}). Obviously $(y_n,u_n)_{n\\in\\N}$ is bounded, hence according to Weierstrass Theorem $\\omega((y_n, u_n)_{n\\in\\N}),$ (and also $\\omega((x_n)_{n\\in\\N})$), is nonempty. It remains to show that $\\omega((y_n, u_n)_{n\\in\\N})$ is closed. From (i) and the proof of (iii) we have\r\n \\begin{equation}\\label{e8}\r\n\\omega((y_n, u_n)_{n\\in\\N})=\\{(\\ol x,\\ol x)\\in\\R^m\\times \\R^m: \\ol x\\in \\omega((x_n)_{n\\in\\N})\\}.\r\n\\end{equation}\r\n Hence, it is enough to show that $\\omega((x_n)_{n\\in\\N})$ is closed.\r\n","sha256_utf8":"4e33853a9380626b2ec773bdde50524df48f586a1a4f6757615715cbd5615dec"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"对角选取证明聚点集闭性","source_units":[{"start":26123,"end":27036,"start_line":429,"end_line":439,"raw_tex":" Let be $(\\ol x_p)_{p\\in\\N}\\subseteq \\omega((x_n)_{n\\in\\N})$ and assume that $\\lim_{p\\To+\\infty}\\ol x_p=x^*.$ We show that $x^*\\in\\omega((x_n)_{n\\in\\N}).$ Obviously, for every $p\\in\\N$ there exists a sequence of natural numbers $n_k^p\\To+\\infty,\\,k\\To+\\infty$, such that\r\n$$\\lim_{k\\To+\\infty}x_{n_k^p}=\\ol x_p.$$\r\n\r\nLet be $\\e >0$. Since $\\lim_{p\\To+\\infty}\\ol x_p=x^*,$  there exists $P(\\e) \\in \\N$ such that for every $p\\ge P(\\e)$ it holds\r\n$$\\|\\ol x_p-x^*\\|<\\frac\\e2.$$\r\nLet $p\\in\\N$ be fixed. Since $\\lim_{k\\To+\\infty}x_{n_k^p}=\\ol x_p,$ there exists $k(p,\\e)\\in\\N$ such that for every $k\\ge k(p,\\e)$ it holds $$\\|x_{n_k^p}-\\ol x_p\\|<\\frac\\e2.$$\r\nLet be $k_p \\geq k(p,\\varepsilon)$ such that $n_{k_p}^p >p$. Obviously $n_{k_p}^p \\To \\infty$ as $p \\To +\\infty$ and for every $p \\geq P(\\e)$\r\n$$\\|x_{n_{k_p}^p}-x^*\\|<\\e.$$\r\nHence\r\n$$\\lim_{p\\To+\\infty}x_{n_{k_p}^p}=x^*,$$\r\nthus $x^*\\in \\omega((x_n)_{n\\in\\N}).$\r\n","sha256_utf8":"6eb6e9ed51be7579e3c8fb7da61d5524d8cc693e941f2ff5d9ae43140fcd81bd"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"迭代到聚点集的距离趋零","source_units":[{"start":27038,"end":27515,"start_line":441,"end_line":446,"raw_tex":"(viii) By using \\eqref{e8} we have\r\n$$\\lim_{n\\To+\\infty} \\dist((y_n,u_n),\\omega((y_n,u_n)_{n\\in\\N}))=\\lim_{n\\To+\\infty}\\inf_{\\ol x\\in\\omega((x_n)_{n\\in\\N})}\\|(y_n,u_n)-(\\ol x,\\ol x)\\|.$$\r\nSince there exists the subsequences $(y_{n_k})_{k\\in\\N}$ and $(u_{n_k})_{k\\in\\N}$ such that\r\n$\\lim_{k\\To\\infty}y_{n_k}=\\lim_{k\\To\\infty}u_{n_k}=\\ol x_0\\in \\omega((x_n)_{n\\in\\N})$ it is straightforward that\r\n$$\\lim_{n\\To+\\infty} \\dist((y_n,u_n),\\omega((y_n,u_n)_{n\\in\\N}))=0.$$\r\n\\end{proof}","sha256_utf8":"5cb3417ab6803767491a536684ac1043246d43709c98ab6c201fd820c33bb2f3"}]},{"id":"S033","semantic_type":"lemma","kind":"lemma","title":"一致化KL不等式","source_units":[{"start":30770,"end":31418,"start_line":482,"end_line":486,"raw_tex":"\\begin{lemma}\\label{unif-KL-property} Let $\\Omega\\subseteq \\R^n$ be a compact set and let $f:\\R^n\\rightarrow \\R$ be a differentiable function. Assume that $f$ is constant on $\\Omega$ and $f$ satisfies the KL property at each point of $\\Omega$.\r\nThen there exist $\\varepsilon,\\eta >0$ and $\\varphi\\in \\Theta_{\\eta}$ such that for all $\\ol x\\in\\Omega$ and for all $x$ in the intersection\r\n\\begin{equation}\\label{int} \\{x\\in\\R^n: \\dist(x,\\Omega)<\\varepsilon\\}\\cap \\{x\\in\\R^n: f(\\ol x)<f(x)<f(\\ol x)+\\eta\\}\\end{equation}\r\nthe following inequality holds \\begin{equation}\\label{KL-ineq}\\varphi'(f(x)-f(\\ol x))\\|\\n f(x)\\|\\geq 1.\\end{equation}\r\n\\end{lemma}","sha256_utf8":"ea708d59fa5a03963d2fc0a5fb9706f3019b19e160cc9db3d09441975604f898"}]},{"id":"S034","semantic_type":"external_fact","kind":"external_fact","title":"紧集上的KL一致化","source_units":[],"declared_external":true,"declaration_text":["Let Ω be compact and f differentiable, constant on Ω, and KL at every point of Ω. Then there exist ε,η>0 and a common concave desingularizing function φ∈Θη such that φ′(f(x)−f(x̄))‖∇f(x)‖≥1 whenever x̄∈Ω, dist(x,Ω)<ε and f(x̄)<f(x)<f(x̄)+η. This is b-sab-teb, Lemma 6, reproduced as B0098."]},{"id":"S035","semantic_type":"theorem","kind":"theorem","title":"有限长度与临界点收敛","source_units":[{"start":31498,"end":32131,"start_line":490,"end_line":497,"raw_tex":"\\begin{theorem}\\label{convergence} In the settings of problem \\eqref{opt-pb}, for some starting points $x_0=y_0\\in\\R^m$ consider the sequences $(x_n)_{n\\in\\N},\\,(y_n)_{n\\in\\N}$ generated by Algorithm \\eqref{generaldiscrete}. Assume that $g$ is bounded from below and consider the function\r\n$$H:\\R^m\\times\\R^m\\To\\R,\\,H(x,y)=g(x)+\\frac12\\|y-x\\|^2.$$\r\nAssume that $(x_n)_{n\\in\\N}$ is bounded and $H$ is a KL function. Then the following statements are true\r\n\\begin{itemize}\r\n\\item[(a)] $\\sum_{n\\ge 1}\\|x_n-x_{n-1}\\|<+\\infty;$\r\n\\item[(b)] there exists ${x}\\in\\crit(g)$ such that $\\lim_{n\\To+\\infty}x_n={x}.$\r\n\\end{itemize}\r\n\\end{theorem}","sha256_utf8":"9c7e0e9bfb9b5f1a8f72795d2364111185ab31ff74408bb0d43dad6ffc2d1bad"}]},{"id":"S036","semantic_type":"proof","kind":"proof","title":"迭代收敛定理完整证明","source_units":[{"start":32151,"end":37825,"start_line":499,"end_line":591,"raw_tex":"\\begin{proof} Consider the sequence\r\n$$u_n=\\sqrt{2\\d_n}(x_{n}-x_{n-1})+y_n\\mbox{, for all }n\\in \\N,$$\r\nthat  was defined in the hypotheses of Lemma \\ref{regularization}. Furthermore, consider $(\\ol x,\\ol x)\\in\\omega((y_n,u_n)_{n\\in\\N}).$\r\n\r\nThen, according to Lemma \\ref{regularization}, the sequence $H(y_n,u_n)$ is decreasing for all $n\\ge N$,  where $N$ was defined in Theorem \\ref{decreasing}, further\r\n$$\\ol x\\in\\crit g\\mbox{ and  }\\lim_{n\\To+\\infty}H(y_n,u_n)=H(\\ol x,\\ol x).$$\r\nWe divide the proof into two cases.\r\n\r\n{\\bf Case I.} There exists $\\ol n\\ge N,\\,\\ol n\\in\\N,$  such that $H(y_{\\ol n},u_{\\ol n})=H(\\ol x,\\ol x).$ Then, since $H(y_n,u_n)$ is decreasing for all $n\\ge N$ and $\\lim_{n\\To+\\infty}H(y_n,u_n)=H(\\ol x,\\ol x)$ we obtain that\r\n$$H(y_n,u_n)=H(\\ol x,\\ol x)\\mbox{ for all }n\\ge\\ol n.$$\r\n\r\nThe latter relation combined with \\eqref{ineq} leads to\r\n$$0\\le D\\|x_n-x_{n-1}\\|^2\\le H(y_n,u_n)-H(y_{n+1},u_{n+1})=H(\\ol x,\\ol x)-H(\\ol x,\\ol x)=0$$\r\nfor all $n\\ge\\ol n.$\r\n\r\nHence $(x_n)_{n\\ge\\ol n}$ is constant and the conclusion follows.\r\n\r\n\r\n\r\n{\\bf Case II.} For every $n\\ge N$ one has that $H(y_n,u_n)> H(\\ol x,\\ol x).$\r\nLet $\\Omega=\\omega((y_n,u_n)_{n\\in\\N}).$ Then according to Lemma \\ref{regularization}, $\\Omega$ is nonempty and compact and $H$ is constant on $\\Omega.$\r\nSince $H$ is KL, according to Lemma \\ref{unif-KL-property}  there exist $\\varepsilon,\\eta >0$ and $\\varphi\\in \\Theta_{\\eta}$ such that for all  $(z,w)$ belonging to the intersection\r\n$$ \\{(z,w)\\in\\R^n\\times\\R^n: \\dist((z,w),\\Omega)<\\varepsilon\\}\\cap \\{(z,w)\\in\\R^n\\times\\R^n: H(\\ol x,\\ol x)<H(z,w)<H(\\ol x,\\ol x)+\\eta\\}$$\r\none has\r\n$$\\varphi'(H(z,w)-H(\\ol x,\\ol x))\\|\\n H(z,w)\\|\\geq 1.$$\r\n\r\nSince $\\lim_{n\\To+\\infty}\\dist((y_n,u_n),\\Omega)=0$, there exists $n_1\\in\\N$ such that\r\n$$\\dist((y_n,u_n),\\Omega)<\\e,\\,\\forall n\\ge n_1.$$\r\n\r\nSince $$\\lim_{n\\To+\\infty}H(y_n,u_n)=H(\\ol x,\\ol x)$$\r\nand\r\n$$H(y_n,u_n))> H(\\ol{x},\\ol{x})\\mbox{ for all }n\\ge N,$$\r\nthere exists $n_2\\ge N$ such that\r\n$$H(\\ol x,\\ol x)<H(y_n,u_n)< H(\\ol{x},\\ol{x})+\\eta,\\,\\forall n\\ge n_2.$$\r\n\r\nHence, for all $n\\ge \\ol n=\\max(n_1,n_2)$ we have\r\n$$\\varphi'(H(y_n,u_n)-H(\\ol{x},\\ol{x}))\\cdot\\|\\n H(y_n,u_n))\\|\\ge 1.$$\r\nSince $\\varphi$ is concave, for all $n\\in\\N$ we have\r\n$$\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x}))\\ge$$\r\n$$\\varphi'(H(y_n,u_n)-H(\\ol{x},\\ol{x}))\\cdot(H(y_n,u_n)-H(y_{n+1},u_{n+1})),$$\r\nhence,\r\n$$\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x}))\\ge \\frac{H(y_n,u_n)-H(y_{n+1},u_{n+1})}{\\|\\n H(y_n,u_n)\\|}$$\r\nfor all $n\\ge\\ol n.$\r\n\r\n Now, from \\eqref{ineq} and Lemma \\ref{regularization} (iv) we obtain\r\n \\begin{equation}\\label{e9}\r\n \\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x}))\\ge\r\n \\end{equation}\r\n $$\\frac{D\\|x_{n}-x_{n-1}\\|^2}{\\frac{1}{s}\\|x_{n+1}-x_n\\|+\\left(\\frac{\\b  n}{s(n+\\a)}+2\\sqrt{2\\d_n}\\right)\\|x_{n}-x_{n-1}\\|},$$\r\n for all $n\\ge\\ol n.$\r\nSince $\\lim_{n\\To+\\infty}\\d_n=\\frac{2-\\b^2-s L_g(\\b+1)}{2s}>0$ and $\\lim_{n\\To+\\infty}\\frac{\\b  n}{s(n+\\a)}=\\frac{\\b }{s}\\ge 0$ there exists $\\ol N\\in\\N,\\,\\ol N\\ge\\ol n$ and $M>0$ such that\r\n$$\\max\\left(\\frac1s,\\frac{\\b  n}{s(n+\\a)}+2\\sqrt{2\\d_n}\\right)\\le M,$$\r\nfor all $n\\ge \\ol N.$\r\nHence, \\eqref{e9} becomes\r\n \\begin{equation}\\label{e10}\r\n \\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x}))\\ge\r\n \\end{equation}\r\n $$\\frac{D\\|x_{n}-x_{n-1}\\|^2}{M(\\|x_{n+1}-x_n\\|+\\|x_{n}-x_{n-1}\\|)},$$\r\n for all $n\\ge\\ol N.$\r\n\r\nConsequently,\r\n$$\\|x_{n}-x_{n-1}\\|\\le$$\r\n$$\\sqrt{\\frac{M}{D}(\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x})))\\cdot(\\|x_{n+1}-x_n\\|+\\|x_{n}-x_{n-1}\\|)},$$\r\n for all $n\\ge\\ol N.$\r\n\r\nBy using the arithmetical-geometrical mean inequality we have\r\n$$\\sqrt{\\frac{M}{D}(\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x})))\\cdot(\\|x_{n+1}-x_n\\|+\\|x_{n}-x_{n-1}\\|)}\\le$$\r\n$$\\frac{\\|x_{n+1}-x_n\\|+\\|x_{n}-x_{n-1}\\|}{3}+\\frac{3M}{4D}(\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x})))$$\r\n for all $n\\ge\\ol N.$\r\nHence\r\n$$\\|x_{n}-x_{n-1}\\|\\le$$\r\n$$\\frac{\\|x_{n+1}-x_n\\|+\\|x_{n}-x_{n-1}\\|}{3}+\\frac{3M}{4D}(\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x})))$$\r\n for all $n\\ge\\ol N$ which leads to\r\n \\begin{equation}\\label{e11}\r\n 2\\|x_{n}-x_{n-1}\\|-\\|x_{n+1}-x_{n}\\|\\le\\frac{9M}{4D}(\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x})))\r\n \\end{equation}\r\n for all $n\\ge\\ol N.$ Let $P>\\ol N$. By summing up \\eqref{e11} from $\\ol N$ to $P$ we obtain\r\n $$\\sum_{n={\\ol N}}^P \\|x_{n}-x_{n-1}\\|\\le$$\r\n  $$-\\|x_{\\ol N}-x_{\\ol N-1}\\|+\\|x_{P+1}-x_P\\|+\\frac{9M}{4D}(\\varphi(H(y_{\\ol N},u_{\\ol N})-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{P+1},u_{P+1})-H(\\ol{x},\\ol{x}))).$$\r\nNow, by letting $P\\To+\\infty$ and using the fact that $\\varphi(0)=0$ and \\eqref{e7} we obtain that\r\n$$\\sum_{n={\\ol N}}^\\infty \\|x_{n}-x_{n-1}\\|\\le-\\|x_{\\ol N}-x_{\\ol N-1}\\|+\\frac{9M}{4D}\\varphi(H(y_{\\ol N},u_{\\ol N})-H(\\ol{x},\\ol{x}))<+\\infty,$$\r\nhence\r\n$$\\sum_{n\\ge 1}\\|x_n-x_{n-1}\\|<+\\infty$$ which is exactly (a).\r\n\r\nObviously the sequence $S_n=\\sum_{k=1}^n\\|x_k-x_{k-1}\\|$ is Cauchy, hence, for all $\\e>0$ there exists $N_{\\e}\\in \\N$   such that for all $n\\ge N_\\e$ and for all $p\\in \\N$ one has\r\n$$S_{n+p}-S_n\\le \\e.$$\r\nBut\r\n$$S_{n+p}-S_n=\\sum_{k={n+1}}^{n+p}\\|x_k-x_{k-1}\\|\\ge \\left\\|\\sum_{k={n+1}}^{n+p}(x_k-x_{k-1})\\right\\|=\\|x_{n+p}-x_n\\|$$\r\nhence the sequence $(x_n)_{n\\in\\N}$ is Cauchy, consequently is convergent. Let\r\n$$\\lim_{n\\To+\\infty}x_n=x.$$\r\nNow, according to Lemma \\ref{regularization} (i) one has\r\n$$\\{x\\}=\\omega((x_n)_{n\\in\\N})\\subseteq\\crit g$$ which proves (b).\r\n\\end{proof}","sha256_utf8":"f3814f0b02a1ab8674ccd75abb67e1c50082c320d3926731cc1c87858c05dad2"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"选取聚点并确定能量极限","source_units":[{"start":32151,"end":32673,"start_line":499,"end_line":505,"raw_tex":"\\begin{proof} Consider the sequence\r\n$$u_n=\\sqrt{2\\d_n}(x_{n}-x_{n-1})+y_n\\mbox{, for all }n\\in \\N,$$\r\nthat  was defined in the hypotheses of Lemma \\ref{regularization}. Furthermore, consider $(\\ol x,\\ol x)\\in\\omega((y_n,u_n)_{n\\in\\N}).$\r\n\r\nThen, according to Lemma \\ref{regularization}, the sequence $H(y_n,u_n)$ is decreasing for all $n\\ge N$,  where $N$ was defined in Theorem \\ref{decreasing}, further\r\n$$\\ol x\\in\\crit g\\mbox{ and  }\\lim_{n\\To+\\infty}H(y_n,u_n)=H(\\ol x,\\ol x).$$\r\nWe divide the proof into two cases.\r\n","sha256_utf8":"efbc4ec40c668cf91323d2c752a9dd13dfd69760f9fa5b7851f6f33c64f85faf"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"达到极限能量时最终静止","source_units":[{"start":32675,"end":33204,"start_line":507,"end_line":514,"raw_tex":"{\\bf Case I.} There exists $\\ol n\\ge N,\\,\\ol n\\in\\N,$  such that $H(y_{\\ol n},u_{\\ol n})=H(\\ol x,\\ol x).$ Then, since $H(y_n,u_n)$ is decreasing for all $n\\ge N$ and $\\lim_{n\\To+\\infty}H(y_n,u_n)=H(\\ol x,\\ol x)$ we obtain that\r\n$$H(y_n,u_n)=H(\\ol x,\\ol x)\\mbox{ for all }n\\ge\\ol n.$$\r\n\r\nThe latter relation combined with \\eqref{ineq} leads to\r\n$$0\\le D\\|x_n-x_{n-1}\\|^2\\le H(y_n,u_n)-H(y_{n+1},u_{n+1})=H(\\ol x,\\ol x)-H(\\ol x,\\ol x)=0$$\r\nfor all $n\\ge\\ol n.$\r\n\r\nHence $(x_n)_{n\\ge\\ol n}$ is constant and the conclusion follows.\r\n","sha256_utf8":"6ebb8797bbd6f3b3582f354dcb94c081e2a7ddb07241f3bf705d5a802085d766"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"在聚点集邻域一致应用KL","source_units":[{"start":33210,"end":33831,"start_line":518,"end_line":523,"raw_tex":"{\\bf Case II.} For every $n\\ge N$ one has that $H(y_n,u_n)> H(\\ol x,\\ol x).$\r\nLet $\\Omega=\\omega((y_n,u_n)_{n\\in\\N}).$ Then according to Lemma \\ref{regularization}, $\\Omega$ is nonempty and compact and $H$ is constant on $\\Omega.$\r\nSince $H$ is KL, according to Lemma \\ref{unif-KL-property}  there exist $\\varepsilon,\\eta >0$ and $\\varphi\\in \\Theta_{\\eta}$ such that for all  $(z,w)$ belonging to the intersection\r\n$$ \\{(z,w)\\in\\R^n\\times\\R^n: \\dist((z,w),\\Omega)<\\varepsilon\\}\\cap \\{(z,w)\\in\\R^n\\times\\R^n: H(\\ol x,\\ol x)<H(z,w)<H(\\ol x,\\ol x)+\\eta\\}$$\r\none has\r\n$$\\varphi'(H(z,w)-H(\\ol x,\\ol x))\\|\\n H(z,w)\\|\\geq 1.$$\r\n","sha256_utf8":"f99012cfcc00d7e3738cd7a4a8ac680fde3975252b482eb6511a3dfb4a4c9aa7"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"尾序列进入KL适用区域","source_units":[{"start":33833,"end":34202,"start_line":525,"end_line":532,"raw_tex":"Since $\\lim_{n\\To+\\infty}\\dist((y_n,u_n),\\Omega)=0$, there exists $n_1\\in\\N$ such that\r\n$$\\dist((y_n,u_n),\\Omega)<\\e,\\,\\forall n\\ge n_1.$$\r\n\r\nSince $$\\lim_{n\\To+\\infty}H(y_n,u_n)=H(\\ol x,\\ol x)$$\r\nand\r\n$$H(y_n,u_n))> H(\\ol{x},\\ol{x})\\mbox{ for all }n\\ge N,$$\r\nthere exists $n_2\\ge N$ such that\r\n$$H(\\ol x,\\ol x)<H(y_n,u_n)< H(\\ol{x},\\ol{x})+\\eta,\\,\\forall n\\ge n_2.$$\r\n","sha256_utf8":"e114e6e7c6fe735b8d2264416475b1e2b5358af547d0fefdbfa455c22fe6fbff"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"凹性将KL转为能量差估计","source_units":[{"start":34204,"end":34727,"start_line":534,"end_line":541,"raw_tex":"Hence, for all $n\\ge \\ol n=\\max(n_1,n_2)$ we have\r\n$$\\varphi'(H(y_n,u_n)-H(\\ol{x},\\ol{x}))\\cdot\\|\\n H(y_n,u_n))\\|\\ge 1.$$\r\nSince $\\varphi$ is concave, for all $n\\in\\N$ we have\r\n$$\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x}))\\ge$$\r\n$$\\varphi'(H(y_n,u_n)-H(\\ol{x},\\ol{x}))\\cdot(H(y_n,u_n)-H(y_{n+1},u_{n+1})),$$\r\nhence,\r\n$$\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x}))\\ge \\frac{H(y_n,u_n)-H(y_{n+1},u_{n+1})}{\\|\\n H(y_n,u_n)\\|}$$\r\nfor all $n\\ge\\ol n.$\r\n","sha256_utf8":"7571bf556d7337d8d59c4a1ee7b5653263a9eb3d4b6766bc946662f3d80d99ea"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"结合梯度控制与充分下降","source_units":[{"start":34729,"end":35628,"start_line":543,"end_line":557,"raw_tex":" Now, from \\eqref{ineq} and Lemma \\ref{regularization} (iv) we obtain\r\n \\begin{equation}\\label{e9}\r\n \\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x}))\\ge\r\n \\end{equation}\r\n $$\\frac{D\\|x_{n}-x_{n-1}\\|^2}{\\frac{1}{s}\\|x_{n+1}-x_n\\|+\\left(\\frac{\\b  n}{s(n+\\a)}+2\\sqrt{2\\d_n}\\right)\\|x_{n}-x_{n-1}\\|},$$\r\n for all $n\\ge\\ol n.$\r\nSince $\\lim_{n\\To+\\infty}\\d_n=\\frac{2-\\b^2-s L_g(\\b+1)}{2s}>0$ and $\\lim_{n\\To+\\infty}\\frac{\\b  n}{s(n+\\a)}=\\frac{\\b }{s}\\ge 0$ there exists $\\ol N\\in\\N,\\,\\ol N\\ge\\ol n$ and $M>0$ such that\r\n$$\\max\\left(\\frac1s,\\frac{\\b  n}{s(n+\\a)}+2\\sqrt{2\\d_n}\\right)\\le M,$$\r\nfor all $n\\ge \\ol N.$\r\nHence, \\eqref{e9} becomes\r\n \\begin{equation}\\label{e10}\r\n \\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x}))\\ge\r\n \\end{equation}\r\n $$\\frac{D\\|x_{n}-x_{n-1}\\|^2}{M(\\|x_{n+1}-x_n\\|+\\|x_{n}-x_{n-1}\\|)},$$\r\n for all $n\\ge\\ol N.$\r\n","sha256_utf8":"004322976c685e683511f347ffc6a9362c9a3910ca56e1cc9bd838d58ab8aa96"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"均值不等式给出长度递推","source_units":[{"start":35630,"end":36627,"start_line":559,"end_line":574,"raw_tex":"Consequently,\r\n$$\\|x_{n}-x_{n-1}\\|\\le$$\r\n$$\\sqrt{\\frac{M}{D}(\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x})))\\cdot(\\|x_{n+1}-x_n\\|+\\|x_{n}-x_{n-1}\\|)},$$\r\n for all $n\\ge\\ol N.$\r\n\r\nBy using the arithmetical-geometrical mean inequality we have\r\n$$\\sqrt{\\frac{M}{D}(\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x})))\\cdot(\\|x_{n+1}-x_n\\|+\\|x_{n}-x_{n-1}\\|)}\\le$$\r\n$$\\frac{\\|x_{n+1}-x_n\\|+\\|x_{n}-x_{n-1}\\|}{3}+\\frac{3M}{4D}(\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x})))$$\r\n for all $n\\ge\\ol N.$\r\nHence\r\n$$\\|x_{n}-x_{n-1}\\|\\le$$\r\n$$\\frac{\\|x_{n+1}-x_n\\|+\\|x_{n}-x_{n-1}\\|}{3}+\\frac{3M}{4D}(\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x})))$$\r\n for all $n\\ge\\ol N$ which leads to\r\n \\begin{equation}\\label{e11}\r\n 2\\|x_{n}-x_{n-1}\\|-\\|x_{n+1}-x_{n}\\|\\le\\frac{9M}{4D}(\\varphi(H(y_n,u_n)-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{n+1},u_{n+1})-H(\\ol{x},\\ol{x})))\r\n \\end{equation}\r\n","sha256_utf8":"d73768f5f31659310000e872342219f925da106410169f88746e7b41cb789fe3"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"望远镜求和证明有限长度","source_units":[{"start":36627,"end":37245,"start_line":575,"end_line":581,"raw_tex":" for all $n\\ge\\ol N.$ Let $P>\\ol N$. By summing up \\eqref{e11} from $\\ol N$ to $P$ we obtain\r\n $$\\sum_{n={\\ol N}}^P \\|x_{n}-x_{n-1}\\|\\le$$\r\n  $$-\\|x_{\\ol N}-x_{\\ol N-1}\\|+\\|x_{P+1}-x_P\\|+\\frac{9M}{4D}(\\varphi(H(y_{\\ol N},u_{\\ol N})-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{P+1},u_{P+1})-H(\\ol{x},\\ol{x}))).$$\r\nNow, by letting $P\\To+\\infty$ and using the fact that $\\varphi(0)=0$ and \\eqref{e7} we obtain that\r\n$$\\sum_{n={\\ol N}}^\\infty \\|x_{n}-x_{n-1}\\|\\le-\\|x_{\\ol N}-x_{\\ol N-1}\\|+\\frac{9M}{4D}\\varphi(H(y_{\\ol N},u_{\\ol N})-H(\\ol{x},\\ol{x}))<+\\infty,$$\r\nhence\r\n$$\\sum_{n\\ge 1}\\|x_n-x_{n-1}\\|<+\\infty$$ which is exactly (a).\r\n","sha256_utf8":"e59f885a8e95fe865bdd729cb2e5ffd2e2fab2af4d7e3eefb4f04b52a3bffa44"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"柯西性与临界极限","source_units":[{"start":37247,"end":37825,"start_line":583,"end_line":591,"raw_tex":"Obviously the sequence $S_n=\\sum_{k=1}^n\\|x_k-x_{k-1}\\|$ is Cauchy, hence, for all $\\e>0$ there exists $N_{\\e}\\in \\N$   such that for all $n\\ge N_\\e$ and for all $p\\in \\N$ one has\r\n$$S_{n+p}-S_n\\le \\e.$$\r\nBut\r\n$$S_{n+p}-S_n=\\sum_{k={n+1}}^{n+p}\\|x_k-x_{k-1}\\|\\ge \\left\\|\\sum_{k={n+1}}^{n+p}(x_k-x_{k-1})\\right\\|=\\|x_{n+p}-x_n\\|$$\r\nhence the sequence $(x_n)_{n\\in\\N}$ is Cauchy, consequently is convergent. Let\r\n$$\\lim_{n\\To+\\infty}x_n=x.$$\r\nNow, according to Lemma \\ref{regularization} (i) one has\r\n$$\\{x\\}=\\omega((x_n)_{n\\in\\N})\\subseteq\\crit g$$ which proves (b).\r\n\\end{proof}","sha256_utf8":"29d046903ae735c729bc9d51624aa48fd7581c268ece0d8aa98e86e9eae171dd"}]},{"id":"S046","semantic_type":"proof","kind":"proof","title":"任意多项式收敛率完整证明","source_units":[{"start":41040,"end":49377,"start_line":627,"end_line":760,"raw_tex":"\\begin{proof}\r\n As we have seen in the proof of Theorem \\ref{convergence},  if there exists $\\ol n\\ge N,\\,\\ol n\\in\\N$, (where $N$ was defined in Theorem \\ref{decreasing}), such that $H(y_{\\ol n},u_{\\ol n})=H(\\ol x,\\ol x),$ then,\r\n$$H(y_n,u_n)=H(\\ol x,\\ol x)\\mbox{ for all }n\\ge\\ol n$$\r\nand $(x_n)_{n\\ge\\ol n}$ is constant. Consequently $(y_n)_{n\\ge\\ol n}$ is also constant and the conclusion of the theorem is straightforward.\r\n\r\nHence, in what follows we assume  that $H(y_{n},u_{ n})>H(\\ol x,\\ol x),$ for all $n\\ge N.$\r\n\\\\\r\n\r\nLet us fix $p>0$ and let us  prove (a$_1$).\r\n\r\n For simplicity let us denote $r_n=H(y_n,u_n)-H(\\ol x,\\ol x)>0$  for all $n\\in\\N.$\r\n From \\eqref{e6} we have\r\n$$\\Delta_n\\|x_n-x_{n-1}\\|^2\\le r_n-r_{n+1}\\mbox{ for all }n\\ge N.$$\r\n\r\nFrom Lemma \\ref{regularization} (v) we have\r\n $$\\|\\n H(y_n,u_n)\\|^2\\le \\frac{2}{s^2}\\|x_{n+1}-x_n\\|^2+2\\left(\\left(\\frac{\\b n}{s(n+\\a)}-\\sqrt{2\\d_n}\\right)^2+\\d_n\\right)\\|x_{n}-x_{n-1}\\|^2$$ for all $n\\in\\N$.\r\n  Let $S_n=2\\left(\\left(\\frac{\\b n}{s(n+\\a)}-\\sqrt{2\\d_n}\\right)^2+\\d_n\\right),$ for all $n\\in\\N.$\r\n\r\nIt follows that, for all $n\\ge N$ one has\r\n$$\\|x_n-x_{n-1}\\|^2\\ge \\frac{1}{S_n}\\|\\n H(y_n,u_n)\\|^2-\\frac{2}{s^2S_n}\\|x_{n+1}-x_n\\|^2\\ge$$\r\n$$\\frac{1}{S_n}\\|\\n H(y_n,u_n)\\|^2-\\frac{2}{s^2S_n\\Delta_{n+1}}(r_{n+1}-r_{n+2}).$$\r\n\r\n\r\nNow by using the {\\L}ojasiewicz property of $H$ at $(\\ol x,\\ol x)\\in\\crit H$, and the fact that $\\lim_{n\\To+\\infty}(y_n,u_n)=(\\ol x,\\ol x)$,  we obtain that there exists  $K,\\e>0$ and $\\ol N_1\\in\\N,$ such that for all $n\\ge \\ol N_1$ one has\r\n$$\\|(y_n,u_n)-(\\ol x,\\ol x)\\|<\\e,$$\r\nconsequently\r\n\\begin{equation}\\label{e111} r_n-r_{n+1}\\ge \\frac{\\Delta_n}{S_n}\\|\\n H(y_n,u_n)\\|^2-\\frac{2\\Delta_n}{s^2S_n\\Delta_{n+1}}(r_{n+1}-r_{n+2})\\ge\r\n\\end{equation}\r\n$$\\frac{\\Delta_n}{K^2 S_n}r_n^{2\\t}-\\frac{2\\Delta_n}{s^2S_n\\Delta_{n+1}}(r_{n+1}-r_{n+2})\\ge$$\r\n$$\\frac{\\Delta_n}{K^2 S_n}r_{n+1}^{2\\t}-\\frac{2\\Delta_n}{s^2S_n\\Delta_{n+1}}(r_{n+1}-r_{n+2})=\\a_n r_{n+1}^{2\\t}-\\b_n(r_{n+1}-r_{n+2}),$$\r\nwhere $\\a_n= \\frac{\\Delta_n}{K^2 S_n}\\mbox{ and  }\\b_n=\\frac{2\\Delta_n}{s^2S_n\\Delta_{n+1}}.$\r\n\r\nIt is obvious that the sequences $(\\a_n)_{n\\ge\\ol N_1}$ and $(\\b_n)_{n\\ge\\ol N_1}$ are convergent, further\r\n$$\\lim_{n\\To+\\infty} \\a_n>0\\mbox{ and } \\lim_{n\\To+\\infty} \\b_n>0.$$\r\nNow, since $0<2\\t\\le1$ and $r_{n+1}\\To 0$, there exists $\\ol N_2\\in\\N,\\, \\ol N_2\\ge \\ol N_1,$ such that $r_{n+1}^{2\\t}\\ge r_{n+1}$ for all $n\\ge\\ol N_2.$\r\n\r\n$(*)$\\,\\,\\, Note that this implies that $0\\le r_n\\le 1$ for all $n\\ge\\ol N_2.$\r\n\r\nHence,\r\n$$r_n\\ge \\left(\\a_n-\\b_n+1\\right)r_{n+1}+\\b_nr_{n+2},\\mbox{ for all }n\\ge\\ol N_2.$$\r\n\r\nLet us define for every $n>\\ol N_2$ the sequence $\\Xi_n=\\frac{\\b_n n^p}{(n+1)^p-n^p}.$\r\nThen, since $p>0$ one has $\\lim_{n\\To+\\infty}\\Xi_n=+\\infty.$\r\n\r\nSince\r\n$$\\lim_{n\\To+\\infty}\\left(\\a_n-\\b_n+1\\right)-\\left(1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}-\\frac{\\b_{n-1}}{1+\\frac{\\b_{n}}{\\Xi_{n}}}\\right)=\\lim_{n\\To+\\infty}\\a_n>0,$$  there exists $\\ol k\\in\\N,\\,\\ol k\\ge\\ol N_2$ such that for all $n\\ge\\ol k$ one has\r\n$$\\a_n-\\b_n+1\\ge 1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}-\\frac{\\b_{n-1}}{1+\\frac{\\b_{n}}{\\Xi_{n}}}.$$\r\n\r\nConsequently,\r\n$$r_n\\ge \\left(1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}-\\frac{\\b_{n-1}}{1+\\frac{\\b_{n}}{\\Xi_{n}}}\\right)r_{n+1}+\\b_nr_{n+2},\\mbox{ for all }n\\ge\\ol k,$$\r\nor, equivalently\r\n\\begin{equation}\\label{e12}\r\nr_n+\\frac{\\b_{n-1}}{1+\\frac{\\b_{n}}{\\Xi_{n}}}r_{n+1}\\ge \\left(1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}\\right)\\left(r_{n+1}+\\frac{\\b_n}{1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}}r_{n+2}\\right),\r\n\\end{equation}\r\nfor all $n\\ge\\ol k.$\r\nNow \\eqref{e12} leads to\r\n$$\\prod_{k=\\ol k}^n\\left( r_k+\\frac{\\b_{k-1}}{1+\\frac{\\b_{k}}{\\Xi_k}}r_{k+1}\\right)\\ge \\prod_{k=\\ol k}^n\\left(1+\\frac{\\b_{k+1}}{\\Xi_{k+1}}\\right)\\prod_{k=\\ol k}^n\\left(r_{k+1}+\\frac{\\b_k}{1+\\frac{\\b_{k+1}}{\\Xi_{k+1}}}r_{k+2}\\right),$$\r\nhence after simplifying we get\r\n\\begin{equation}\\label{egen}\\left(r_{\\ol k}+\\frac{\\b_{\\ol k-1}}{1+\\frac{\\b_{\\ol k}}{\\Xi_{\\ol k}}}r_{\\ol k+1}\\right)\\prod_{k=\\ol k}^n\\frac{1}{1+\\frac{\\b_{k+1}}{\\Xi_{k+1}}}\\ge\r\nr_{n+1}+\\frac{\\b_n}{1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}}r_{n+2}.\r\n\\end{equation}\r\n\r\nBut, $\\frac{\\b_{n+1}}{\\Xi_{n+1}}=\\frac{(n+2)^p}{(n+1)^p}-1,$ hence\r\n$$\\prod_{k=\\ol k}^n\\frac{1}{1+\\frac{\\b_{k+1}}{\\Xi_{k+1}}}=\\prod_{k=\\ol k}^n\\frac{(k+1)^p}{(k+2)^p}=\\frac{(\\ol k+1)^p}{(n+2)^p}.$$\r\nBy denoting $\\left(r_{\\ol k}+\\frac{\\b_{\\ol k-1}}{1+\\frac{\\b_{\\ol k}}{\\Xi_{\\ol k}}}r_{\\ol k+1}\\right)(\\ol k+1)^p=a_1$,  we have\r\n$$a_1 \\frac{1}{(n+2)^p}\\ge r_{n+1}+\\frac{\\b_n}{1+\\frac{1}{n+1}\\b_{n+1}}r_{n+2}.$$\r\n\r\nHence,\r\n\\begin{equation}\\label{e13}\r\na_1 \\frac{1}{n^p}\\ge a_1 \\frac{1}{(n+1)^p}\\ge r_n= g(y_n)-g(\\ol x)+\\d_n\\|x_n-x_{n-1}\\|^2\\ge g(y_n)-g(\\ol x)\r\n\\end{equation}\r\nwhich is (a$_1$).\r\n\r\nFor (a$_2$) we start from   Lemma \\ref{desc} and \\eqref{generaldiscrete} and we have\r\n$$g(x_n)-g(y_n)\\le\\<\\n g(y_n),x_n-y_n\\>+\\frac{L_g}{2}\\|x_n-y_n\\|^2=$$\r\n$$\\frac1s\\left\\<(x_n-x_{n+1})+\\frac{\\b n}{n+\\a}(x_n-x_{n-1}),-\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right\\>+\\frac{L_g}{2}\\left(\\frac{\\b n}{n+\\a}\\right)^2\\|x_n-x_{n-1}\\|^2= $$\r\n$$-\\left(\\frac{\\b n}{n+\\a}\\right)^2\\frac{2-sL_g}{2s}\\|x_n-x_{n-1}\\|^2+\\frac1s\\left\\<x_{n+1}-x_n,\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right\\>.$$\r\nBy using the inequality $\\<X,Y\\>\\le \\frac12\\left(a^2\\|X\\|^2+\\frac{1}{a^2}\\|Y\\|^2\\right)$ for all $X,Y\\in\\R^m, a\\in \\R\\setminus\\{0\\}$, we obtain\r\n$$\\left\\<x_{n+1}-x_n,\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right\\>\\le \\frac12\\left(\\frac{1}{2-sL_g}\\|x_{n+1}-x_n\\|^2+(2-sL_g)\\left(\\frac{\\b n}{n+\\a}\\right)^2\\|x_n-x_{n-1}\\|^2\\right),$$ consequently\r\n$$g(x_n)-g(y_n)\\le\\frac{1}{2s(2-sL_g)}\\|x_{n+1}-x_n\\|^2.$$\r\n\r\nFrom \\eqref{ineq} we have\r\n$$\\|x_{n}-x_{n-1}\\|^2\\le \\frac{1}{{D}}((g(y_{n})+\\d_{n}\\|x_{n}-x_{n-1}\\|^2)- (g(y_{n+1})+\\d_{n+1}\\|x_{n+1}-x_{n}\\|^2))$$ and since the sequence $(g(y_{n})+\\d_{n}\\|x_{n+1}-x_{n}\\|^2)_{n\\ge \\ol k}$ is decreasing and has the limit $g(\\ol x)$, we obtain that $g(y_{n+1})+\\d_{n+1}\\|x_{n+1}-x_{n}\\|^2\\ge g(\\ol x)$, consequently\r\n\\begin{equation}\\label{ee}\r\n\\|x_{n}-x_{n-1}\\|^2\\le\\frac{1}{{D}}{r_{n}}.\r\n\\end{equation}\r\n\r\nHence, for all $n\\ge\\ol k$ one has\r\n\\begin{equation}\\label{ee1}\r\ng(x_n)-g(y_n)\\le\\frac{1}{2sD(2-sL_g)}r_{n+1}.\r\n\\end{equation}\r\n\r\nNow, the identity $g(x_n)-g(\\ol x)=(g(x_n)-g(y_n))+(g(y_n)-g(\\ol x))$ and (a$_1$) lead to\r\n$$g(x_n)-g(\\ol x)\\le \\frac{1}{2sD(2-sL_g)}r_{n+1}+a_1\\frac{1}{n^p}$$\r\nfor every $n>\\ol k$, which combined with \\eqref{e13} give\r\n$$g(x_n)-g(\\ol x)\\le \\frac{1}{2sD(2-sL_g)}a_1\\frac{1}{(n+2)^p}+a_1\\frac{1}{n^p}\\le a_1\\left(1+\\frac{1}{2sD(2-sL_g)}\\right)\\frac{1}{n^p}=a_2\\frac{1}{n^p},$$\r\nfor every $n>\\ol k$.\r\n\r\nFor (a$_3$) observe, that  by summing up \\eqref{e11} from $n\\ge\\ol k$ to $P>n$ and using the triangle inequality we obtain\r\n $$\\|x_{P}-x_{n-1}\\|\\le \\sum_{k={n}}^P \\|x_{k}-x_{k-1}\\|\\le$$\r\n  $$-\\|x_{n}-x_{n-1}\\|+\\|x_{P+1}-x_P\\|+\\frac{9M}{4D}(\\varphi(H(y_{n},u_{n})-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{P+1},u_{P+1})-H(\\ol{x},\\ol{x}))).$$\r\n\r\nBy  letting $P\\To+\\infty$ we get\r\n$$\\|x_{n-1}-\\ol x\\|\\le -\\|x_{n}-x_{n-1}\\|+\\frac{9M}{4D}\\varphi(H(y_{n},u_{n})-H(\\ol{x},\\ol{x}))\\le \\frac{9M}{4D}\\varphi(H(y_{n},u_{n})-H(\\ol{x},\\ol{x})).$$\r\nBut, $\\varphi(t)=\\frac{K}{1-\\t}t^{1-\\t},$ hence\r\n\\begin{equation}\\label{e1111}\r\n\\|x_{n-1}-\\ol x\\|\\le \\frac{9MK}{4D(1-\\t)}(H(y_{n},u_{n})-H(\\ol{x},\\ol{x}))^{1-\\t}= M_1r_{n}^{1-\\t},\r\n\\end{equation} where $M_1=\\frac{9MK}{4D(1-\\t)}.$\r\n\r\n%From \\eqref{ee} we have\r\n%$$\\|x_{n}-x_{n-1}\\|\\le\\frac{1}{\\sqrt{D}}\\sqrt{r_{n}}.$$\r\n%Hence,\r\n%\\begin{equation}\\label{e1111}\r\n%\\|x_{n}-\\ol x\\|\\le \\frac{1}{\\sqrt{D}}\\sqrt{r_{n}}+M_1r_{n}^{1-\\t}.\r\n%\\end{equation}\r\nBut $(*)$ assures that $0\\le r_{n}\\le 1$ which combined with $\\t\\in\\left(0,\\frac12\\right]$ leads to\r\n$r_{n}^{1-\\t}\\le \\sqrt{r_{n}}$, consequently we have\r\n$$\\|x_{n-1}-\\ol x\\|\\le M_1\\sqrt{r_{n}}.$$\r\n\r\nThe conclusion follow by \\eqref{e13}, since  we have\r\n$$\\|x_{n}-\\ol x\\|\\le M_1\\sqrt{a_1}\\frac{1}{n^{\\frac{p}{2}}}=a_3\\frac{1}{n^{\\frac{p}{2}}}$$\r\nfor every $n>\\ol k.$\r\n\r\nFinally, for $n> \\ol k$ we have\r\n$$\\|y_n-\\ol x\\|=\\left\\|x_n+\\frac{\\b n}{n+\\a}(x_n-x_{n-1})-\\ol x\\right\\|\\le$$\r\n$$\\left(1+\\frac{\\b  n}{n+\\a}\\right)\\|x_n-\\ol x\\|+\\frac{\\b  n}{n+\\a}\\|x_{n-1}-\\ol x\\|\\le$$\r\n$$\\left(1+\\frac{\\b  n}{n+\\a}\\right)a_3\\frac{1}{n^{\\frac{p}{2}}}+\\frac{\\b  n}{n+\\a}a_3\\frac{1}{n^{\\frac{p}{2}}}\\le$$\r\n$$\\left(1+2\\frac{\\b  n}{n+\\a}\\right)a_3\\frac{1}{n^{\\frac{p}{2}}}.$$\r\n\r\n\r\nLet $a_4=(1+2\\b)a_3.$ Then\r\n$$\\|y_n-\\ol x\\|\\le a_4\\frac{1}{n^{\\frac{p}{2}}},$$\r\nfor all $n> \\ol k$, which proves (a$_4$).\r\n\\end{proof}","sha256_utf8":"541d47e2cad7f261832ce09186e42fcf3af92ff03dadd19a913dc88e44c93ad1"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"排除最终静止的平凡情形","source_units":[{"start":41040,"end":41568,"start_line":627,"end_line":634,"raw_tex":"\\begin{proof}\r\n As we have seen in the proof of Theorem \\ref{convergence},  if there exists $\\ol n\\ge N,\\,\\ol n\\in\\N$, (where $N$ was defined in Theorem \\ref{decreasing}), such that $H(y_{\\ol n},u_{\\ol n})=H(\\ol x,\\ol x),$ then,\r\n$$H(y_n,u_n)=H(\\ol x,\\ol x)\\mbox{ for all }n\\ge\\ol n$$\r\nand $(x_n)_{n\\ge\\ol n}$ is constant. Consequently $(y_n)_{n\\ge\\ol n}$ is also constant and the conclusion of the theorem is straightforward.\r\n\r\nHence, in what follows we assume  that $H(y_{n},u_{ n})>H(\\ol x,\\ol x),$ for all $n\\ge N.$\r\n\\\\\r\n\r\n","sha256_utf8":"db150bc11ab19b2399c11bd3db502478befb47dc855f63260a70f2f3a24f9f9a"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"定义能量残差并控制梯度","source_units":[{"start":41568,"end":42332,"start_line":635,"end_line":647,"raw_tex":"Let us fix $p>0$ and let us  prove (a$_1$).\r\n\r\n For simplicity let us denote $r_n=H(y_n,u_n)-H(\\ol x,\\ol x)>0$  for all $n\\in\\N.$\r\n From \\eqref{e6} we have\r\n$$\\Delta_n\\|x_n-x_{n-1}\\|^2\\le r_n-r_{n+1}\\mbox{ for all }n\\ge N.$$\r\n\r\nFrom Lemma \\ref{regularization} (v) we have\r\n $$\\|\\n H(y_n,u_n)\\|^2\\le \\frac{2}{s^2}\\|x_{n+1}-x_n\\|^2+2\\left(\\left(\\frac{\\b n}{s(n+\\a)}-\\sqrt{2\\d_n}\\right)^2+\\d_n\\right)\\|x_{n}-x_{n-1}\\|^2$$ for all $n\\in\\N$.\r\n  Let $S_n=2\\left(\\left(\\frac{\\b n}{s(n+\\a)}-\\sqrt{2\\d_n}\\right)^2+\\d_n\\right),$ for all $n\\in\\N.$\r\n\r\nIt follows that, for all $n\\ge N$ one has\r\n$$\\|x_n-x_{n-1}\\|^2\\ge \\frac{1}{S_n}\\|\\n H(y_n,u_n)\\|^2-\\frac{2}{s^2S_n}\\|x_{n+1}-x_n\\|^2\\ge$$\r\n$$\\frac{1}{S_n}\\|\\n H(y_n,u_n)\\|^2-\\frac{2}{s^2S_n\\Delta_{n+1}}(r_{n+1}-r_{n+2}).$$\r\n","sha256_utf8":"1bb971d2c89ac672f28b3046468624d89cab40acfb8eb4c754693935c306b815"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"指数不等式导出二步递推","source_units":[{"start":42336,"end":43117,"start_line":650,"end_line":657,"raw_tex":"Now by using the {\\L}ojasiewicz property of $H$ at $(\\ol x,\\ol x)\\in\\crit H$, and the fact that $\\lim_{n\\To+\\infty}(y_n,u_n)=(\\ol x,\\ol x)$,  we obtain that there exists  $K,\\e>0$ and $\\ol N_1\\in\\N,$ such that for all $n\\ge \\ol N_1$ one has\r\n$$\\|(y_n,u_n)-(\\ol x,\\ol x)\\|<\\e,$$\r\nconsequently\r\n\\begin{equation}\\label{e111} r_n-r_{n+1}\\ge \\frac{\\Delta_n}{S_n}\\|\\n H(y_n,u_n)\\|^2-\\frac{2\\Delta_n}{s^2S_n\\Delta_{n+1}}(r_{n+1}-r_{n+2})\\ge\r\n\\end{equation}\r\n$$\\frac{\\Delta_n}{K^2 S_n}r_n^{2\\t}-\\frac{2\\Delta_n}{s^2S_n\\Delta_{n+1}}(r_{n+1}-r_{n+2})\\ge$$\r\n$$\\frac{\\Delta_n}{K^2 S_n}r_{n+1}^{2\\t}-\\frac{2\\Delta_n}{s^2S_n\\Delta_{n+1}}(r_{n+1}-r_{n+2})=\\a_n r_{n+1}^{2\\t}-\\b_n(r_{n+1}-r_{n+2}),$$\r\nwhere $\\a_n= \\frac{\\Delta_n}{K^2 S_n}\\mbox{ and  }\\b_n=\\frac{2\\Delta_n}{s^2S_n\\Delta_{n+1}}.$\r\n","sha256_utf8":"754dba6555017c66767e9ba2f83621a4cf01cd7f6e31e39f582faa04530adfe1"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"小指数使残差幂线性化","source_units":[{"start":43119,"end":43629,"start_line":659,"end_line":666,"raw_tex":"It is obvious that the sequences $(\\a_n)_{n\\ge\\ol N_1}$ and $(\\b_n)_{n\\ge\\ol N_1}$ are convergent, further\r\n$$\\lim_{n\\To+\\infty} \\a_n>0\\mbox{ and } \\lim_{n\\To+\\infty} \\b_n>0.$$\r\nNow, since $0<2\\t\\le1$ and $r_{n+1}\\To 0$, there exists $\\ol N_2\\in\\N,\\, \\ol N_2\\ge \\ol N_1,$ such that $r_{n+1}^{2\\t}\\ge r_{n+1}$ for all $n\\ge\\ol N_2.$\r\n\r\n$(*)$\\,\\,\\, Note that this implies that $0\\le r_n\\le 1$ for all $n\\ge\\ol N_2.$\r\n\r\nHence,\r\n$$r_n\\ge \\left(\\a_n-\\b_n+1\\right)r_{n+1}+\\b_nr_{n+2},\\mbox{ for all }n\\ge\\ol N_2.$$\r\n","sha256_utf8":"224de22da109ec2c18c6541e74f03f179a7dfbdab16c69b9f639bc7d180b1fdb"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"构造任意阶多项式比较系数","source_units":[{"start":43631,"end":44536,"start_line":668,"end_line":681,"raw_tex":"Let us define for every $n>\\ol N_2$ the sequence $\\Xi_n=\\frac{\\b_n n^p}{(n+1)^p-n^p}.$\r\nThen, since $p>0$ one has $\\lim_{n\\To+\\infty}\\Xi_n=+\\infty.$\r\n\r\nSince\r\n$$\\lim_{n\\To+\\infty}\\left(\\a_n-\\b_n+1\\right)-\\left(1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}-\\frac{\\b_{n-1}}{1+\\frac{\\b_{n}}{\\Xi_{n}}}\\right)=\\lim_{n\\To+\\infty}\\a_n>0,$$  there exists $\\ol k\\in\\N,\\,\\ol k\\ge\\ol N_2$ such that for all $n\\ge\\ol k$ one has\r\n$$\\a_n-\\b_n+1\\ge 1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}-\\frac{\\b_{n-1}}{1+\\frac{\\b_{n}}{\\Xi_{n}}}.$$\r\n\r\nConsequently,\r\n$$r_n\\ge \\left(1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}-\\frac{\\b_{n-1}}{1+\\frac{\\b_{n}}{\\Xi_{n}}}\\right)r_{n+1}+\\b_nr_{n+2},\\mbox{ for all }n\\ge\\ol k,$$\r\nor, equivalently\r\n\\begin{equation}\\label{e12}\r\nr_n+\\frac{\\b_{n-1}}{1+\\frac{\\b_{n}}{\\Xi_{n}}}r_{n+1}\\ge \\left(1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}\\right)\\left(r_{n+1}+\\frac{\\b_n}{1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}}r_{n+2}\\right),\r\n\\end{equation}\r\nfor all $n\\ge\\ol k.$\r\n","sha256_utf8":"29be3e19c6b4d8f5d58939f236977e85aec59ddc573de589bd833d03399627b2"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"乘积消去二步递推项","source_units":[{"start":44536,"end":45080,"start_line":682,"end_line":687,"raw_tex":"Now \\eqref{e12} leads to\r\n$$\\prod_{k=\\ol k}^n\\left( r_k+\\frac{\\b_{k-1}}{1+\\frac{\\b_{k}}{\\Xi_k}}r_{k+1}\\right)\\ge \\prod_{k=\\ol k}^n\\left(1+\\frac{\\b_{k+1}}{\\Xi_{k+1}}\\right)\\prod_{k=\\ol k}^n\\left(r_{k+1}+\\frac{\\b_k}{1+\\frac{\\b_{k+1}}{\\Xi_{k+1}}}r_{k+2}\\right),$$\r\nhence after simplifying we get\r\n\\begin{equation}\\label{egen}\\left(r_{\\ol k}+\\frac{\\b_{\\ol k-1}}{1+\\frac{\\b_{\\ol k}}{\\Xi_{\\ol k}}}r_{\\ol k+1}\\right)\\prod_{k=\\ol k}^n\\frac{1}{1+\\frac{\\b_{k+1}}{\\Xi_{k+1}}}\\ge\r\nr_{n+1}+\\frac{\\b_n}{1+\\frac{\\b_{n+1}}{\\Xi_{n+1}}}r_{n+2}.\r\n\\end{equation}\r\n","sha256_utf8":"11cebf224f2e3e200f5e644d3a88cf1ea2ad78d4c1ef80a4f65c9bc1f0e8766a"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"计算乘积得到外推点函数值率","source_units":[{"start":45082,"end":45675,"start_line":689,"end_line":698,"raw_tex":"But, $\\frac{\\b_{n+1}}{\\Xi_{n+1}}=\\frac{(n+2)^p}{(n+1)^p}-1,$ hence\r\n$$\\prod_{k=\\ol k}^n\\frac{1}{1+\\frac{\\b_{k+1}}{\\Xi_{k+1}}}=\\prod_{k=\\ol k}^n\\frac{(k+1)^p}{(k+2)^p}=\\frac{(\\ol k+1)^p}{(n+2)^p}.$$\r\nBy denoting $\\left(r_{\\ol k}+\\frac{\\b_{\\ol k-1}}{1+\\frac{\\b_{\\ol k}}{\\Xi_{\\ol k}}}r_{\\ol k+1}\\right)(\\ol k+1)^p=a_1$,  we have\r\n$$a_1 \\frac{1}{(n+2)^p}\\ge r_{n+1}+\\frac{\\b_n}{1+\\frac{1}{n+1}\\b_{n+1}}r_{n+2}.$$\r\n\r\nHence,\r\n\\begin{equation}\\label{e13}\r\na_1 \\frac{1}{n^p}\\ge a_1 \\frac{1}{(n+1)^p}\\ge r_n= g(y_n)-g(\\ol x)+\\d_n\\|x_n-x_{n-1}\\|^2\\ge g(y_n)-g(\\ol x)\r\n\\end{equation}\r\nwhich is (a$_1$).\r\n","sha256_utf8":"f951d2076116ca5548c1e09adc9379892f0640006af7d3aa55e01f30bfd0d9c7"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"估计原迭代与外推点函数值差","source_units":[{"start":45677,"end":46539,"start_line":700,"end_line":706,"raw_tex":"For (a$_2$) we start from   Lemma \\ref{desc} and \\eqref{generaldiscrete} and we have\r\n$$g(x_n)-g(y_n)\\le\\<\\n g(y_n),x_n-y_n\\>+\\frac{L_g}{2}\\|x_n-y_n\\|^2=$$\r\n$$\\frac1s\\left\\<(x_n-x_{n+1})+\\frac{\\b n}{n+\\a}(x_n-x_{n-1}),-\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right\\>+\\frac{L_g}{2}\\left(\\frac{\\b n}{n+\\a}\\right)^2\\|x_n-x_{n-1}\\|^2= $$\r\n$$-\\left(\\frac{\\b n}{n+\\a}\\right)^2\\frac{2-sL_g}{2s}\\|x_n-x_{n-1}\\|^2+\\frac1s\\left\\<x_{n+1}-x_n,\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right\\>.$$\r\nBy using the inequality $\\<X,Y\\>\\le \\frac12\\left(a^2\\|X\\|^2+\\frac{1}{a^2}\\|Y\\|^2\\right)$ for all $X,Y\\in\\R^m, a\\in \\R\\setminus\\{0\\}$, we obtain\r\n$$\\left\\<x_{n+1}-x_n,\\frac{\\b n}{n+\\a}(x_n-x_{n-1})\\right\\>\\le \\frac12\\left(\\frac{1}{2-sL_g}\\|x_{n+1}-x_n\\|^2+(2-sL_g)\\left(\\frac{\\b n}{n+\\a}\\right)^2\\|x_n-x_{n-1}\\|^2\\right),$$ consequently\r\n$$g(x_n)-g(y_n)\\le\\frac{1}{2s(2-sL_g)}\\|x_{n+1}-x_n\\|^2.$$\r\n","sha256_utf8":"b86ea5d1e0e309071f516345660f3990d6681232075f6f489783272f71e05d0f"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"由能量残差控制迭代差","source_units":[{"start":46541,"end":47110,"start_line":708,"end_line":717,"raw_tex":"From \\eqref{ineq} we have\r\n$$\\|x_{n}-x_{n-1}\\|^2\\le \\frac{1}{{D}}((g(y_{n})+\\d_{n}\\|x_{n}-x_{n-1}\\|^2)- (g(y_{n+1})+\\d_{n+1}\\|x_{n+1}-x_{n}\\|^2))$$ and since the sequence $(g(y_{n})+\\d_{n}\\|x_{n+1}-x_{n}\\|^2)_{n\\ge \\ol k}$ is decreasing and has the limit $g(\\ol x)$, we obtain that $g(y_{n+1})+\\d_{n+1}\\|x_{n+1}-x_{n}\\|^2\\ge g(\\ol x)$, consequently\r\n\\begin{equation}\\label{ee}\r\n\\|x_{n}-x_{n-1}\\|^2\\le\\frac{1}{{D}}{r_{n}}.\r\n\\end{equation}\r\n\r\nHence, for all $n\\ge\\ol k$ one has\r\n\\begin{equation}\\label{ee1}\r\ng(x_n)-g(y_n)\\le\\frac{1}{2sD(2-sL_g)}r_{n+1}.\r\n\\end{equation}\r\n","sha256_utf8":"0231473f359710ad2031b2d68fc5c50d52064da3166145190045caefc63394d8"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"推出原迭代函数值收敛率","source_units":[{"start":47112,"end":47511,"start_line":719,"end_line":723,"raw_tex":"Now, the identity $g(x_n)-g(\\ol x)=(g(x_n)-g(y_n))+(g(y_n)-g(\\ol x))$ and (a$_1$) lead to\r\n$$g(x_n)-g(\\ol x)\\le \\frac{1}{2sD(2-sL_g)}r_{n+1}+a_1\\frac{1}{n^p}$$\r\nfor every $n>\\ol k$, which combined with \\eqref{e13} give\r\n$$g(x_n)-g(\\ol x)\\le \\frac{1}{2sD(2-sL_g)}a_1\\frac{1}{(n+2)^p}+a_1\\frac{1}{n^p}\\le a_1\\left(1+\\frac{1}{2sD(2-sL_g)}\\right)\\frac{1}{n^p}=a_2\\frac{1}{n^p},$$\r\nfor every $n>\\ol k$.\r\n","sha256_utf8":"0981887fd57864c4c91c703bea9b753ed5dc21c8da6c0685e9dfad6aa2d01651"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"长度尾和给出距离残差界","source_units":[{"start":47513,"end":48481,"start_line":725,"end_line":741,"raw_tex":"For (a$_3$) observe, that  by summing up \\eqref{e11} from $n\\ge\\ol k$ to $P>n$ and using the triangle inequality we obtain\r\n $$\\|x_{P}-x_{n-1}\\|\\le \\sum_{k={n}}^P \\|x_{k}-x_{k-1}\\|\\le$$\r\n  $$-\\|x_{n}-x_{n-1}\\|+\\|x_{P+1}-x_P\\|+\\frac{9M}{4D}(\\varphi(H(y_{n},u_{n})-H(\\ol{x},\\ol{x}))-\\varphi(H(y_{P+1},u_{P+1})-H(\\ol{x},\\ol{x}))).$$\r\n\r\nBy  letting $P\\To+\\infty$ we get\r\n$$\\|x_{n-1}-\\ol x\\|\\le -\\|x_{n}-x_{n-1}\\|+\\frac{9M}{4D}\\varphi(H(y_{n},u_{n})-H(\\ol{x},\\ol{x}))\\le \\frac{9M}{4D}\\varphi(H(y_{n},u_{n})-H(\\ol{x},\\ol{x})).$$\r\nBut, $\\varphi(t)=\\frac{K}{1-\\t}t^{1-\\t},$ hence\r\n\\begin{equation}\\label{e1111}\r\n\\|x_{n-1}-\\ol x\\|\\le \\frac{9MK}{4D(1-\\t)}(H(y_{n},u_{n})-H(\\ol{x},\\ol{x}))^{1-\\t}= M_1r_{n}^{1-\\t},\r\n\\end{equation} where $M_1=\\frac{9MK}{4D(1-\\t)}.$\r\n\r\n%From \\eqref{ee} we have\r\n%$$\\|x_{n}-x_{n-1}\\|\\le\\frac{1}{\\sqrt{D}}\\sqrt{r_{n}}.$$\r\n%Hence,\r\n%\\begin{equation}\\label{e1111}\r\n%\\|x_{n}-\\ol x\\|\\le \\frac{1}{\\sqrt{D}}\\sqrt{r_{n}}+M_1r_{n}^{1-\\t}.\r\n%\\end{equation}\r\n","sha256_utf8":"87c446e4cb1b1ef411dc1869cef15fe0d1ece7ad203183a0bfaafedb342167f3"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"残差平方根给出迭代距离率","source_units":[{"start":48481,"end":48849,"start_line":742,"end_line":748,"raw_tex":"But $(*)$ assures that $0\\le r_{n}\\le 1$ which combined with $\\t\\in\\left(0,\\frac12\\right]$ leads to\r\n$r_{n}^{1-\\t}\\le \\sqrt{r_{n}}$, consequently we have\r\n$$\\|x_{n-1}-\\ol x\\|\\le M_1\\sqrt{r_{n}}.$$\r\n\r\nThe conclusion follow by \\eqref{e13}, since  we have\r\n$$\\|x_{n}-\\ol x\\|\\le M_1\\sqrt{a_1}\\frac{1}{n^{\\frac{p}{2}}}=a_3\\frac{1}{n^{\\frac{p}{2}}}$$\r\nfor every $n>\\ol k.$\r\n","sha256_utf8":"564c47171f1b2240ed36077775839994460b23ed5b29f0151c5261e0b6cb1c4c"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"惯性关系传递外推点距离率","source_units":[{"start":48851,"end":49377,"start_line":750,"end_line":760,"raw_tex":"Finally, for $n> \\ol k$ we have\r\n$$\\|y_n-\\ol x\\|=\\left\\|x_n+\\frac{\\b n}{n+\\a}(x_n-x_{n-1})-\\ol x\\right\\|\\le$$\r\n$$\\left(1+\\frac{\\b  n}{n+\\a}\\right)\\|x_n-\\ol x\\|+\\frac{\\b  n}{n+\\a}\\|x_{n-1}-\\ol x\\|\\le$$\r\n$$\\left(1+\\frac{\\b  n}{n+\\a}\\right)a_3\\frac{1}{n^{\\frac{p}{2}}}+\\frac{\\b  n}{n+\\a}a_3\\frac{1}{n^{\\frac{p}{2}}}\\le$$\r\n$$\\left(1+2\\frac{\\b  n}{n+\\a}\\right)a_3\\frac{1}{n^{\\frac{p}{2}}}.$$\r\n\r\n\r\nLet $a_4=(1+2\\b)a_3.$ Then\r\n$$\\|y_n-\\ol x\\|\\le a_4\\frac{1}{n^{\\frac{p}{2}}},$$\r\nfor all $n> \\ol k$, which proves (a$_4$).\r\n\\end{proof}","sha256_utf8":"e100284d602e84b47388744c6d535faea984dd1dc54fbe2edc1fe0ef33e85f15"}]},{"id":"S060","semantic_type":"external_fact","kind":"external_fact","title":"有限维分析与基本不等式","source_units":[],"declared_external":true,"declaration_text":["In finite-dimensional Euclidean space, bounded sequences have convergent subsequences, closed bounded sets are compact, and Cauchy sequences converge. Convergent sequences are bounded; monotone real sequences bounded below converge; summable nonnegative sequences tend to zero. Triangle and Cauchy–Schwarz inequalities, weighted Young inequality, telescoping sums and products, and ψ(a)−ψ(b)≥ψ′(a)(a−b) for differentiable concave ψ are available."]},{"id":"S061","semantic_type":"assumption","kind":"assumption","title":"下降定理的迭代与系数设定","source_units":[{"start":14331,"end":14883,"start_line":240,"end_line":245,"raw_tex":"\\begin{theorem}\\label{decreasing} In the settings of problem \\eqref{opt-pb}, for some starting points $x_0=y_0\\in\\R^m$ let $(x_n)_{n\\in\\N},\\,(y_n)_{n\\in\\N}$ be the sequences generated by the numerical scheme \\eqref{generaldiscrete}. Consider the sequences\r\n $$A_{n-1}=\\frac{2-s L_g}{2s}\\left(\\frac{(1+\\b)n+\\a}{n+\\a}\\right)^2-\\frac{\\b n((1+\\b)n+\\a)}{s(n+\\a)^2},$$\r\n$$C_{n-1}=\\frac{2-s L_g}{2s}\\frac{\\b n-\\b}{n+\\a-1}\\frac{(1+\\b)n+\\a}{n+\\a}-\\frac{1}{2s}\\frac{\\b n-\\b}{n+\\a-1}\\frac{\\b n}{n+\\a}$$\r\nand\r\n$$\\d_n=A_{n-1}-C_{n-1}$$\r\nfor all $n\\in\\N,\\,n\\ge 1$.\r\n","sha256_utf8":"a47ba48f33d47e0bc9367584b87d1a017d898d767cb2d7e9e6fc121081cfd186"}]},{"id":"S062","semantic_type":"assumption","kind":"assumption","title":"正则化引理的函数与序列设定","source_units":[{"start":21195,"end":21739,"start_line":363,"end_line":367,"raw_tex":"\\begin{lemma}\\label{regularization} In the settings of problem \\eqref{opt-pb}, for some starting points $x_0=y_0\\in\\R^m$ consider the sequences $(x_n)_{n\\in\\N},\\,(y_n)_{n\\in\\N}$ generated by Algorithm \\eqref{generaldiscrete}. Assume that $g$ is bounded from below and consider the function\r\n$$H:\\R^m\\times\\R^m\\To\\R,\\,H(x,y)=g(x)+\\frac12\\|y-x\\|^2.$$\r\nConsider further the sequence\r\n$$u_n=\\sqrt{2\\d_n}(x_{n}-x_{n-1})+y_n\\mbox{, for all }n\\in \\N,$$\r\nwhere $\\d_n$ was defined in Theorem \\ref{decreasing}. Then, the following statements hold true.\r\n","sha256_utf8":"b76d88df0947929d8a380cd8d08809cf1adb7fedb927c2c8c9a1b77f5952a22d"}]},{"id":"S063","semantic_type":"assumption","kind":"assumption","title":"聚点紧性所需的迭代有界性","source_units":[{"start":22448,"end":22496,"start_line":377,"end_line":377,"raw_tex":"Assume that $(x_n)_{n\\in\\N}$ is bounded. Then,\r\n","sha256_utf8":"e8472015682b70f1b4f3f37551460a5353fb668aeb06e11f8793b8e7c9656e7b"}]},{"id":"S064","semantic_type":"assumption","kind":"assumption","title":"目标函数下有界","source_units":[{"start":15113,"end":15190,"start_line":251,"end_line":251,"raw_tex":"Assume that $g$ is bounded from below. Then, the following statements hold.\r\n","sha256_utf8":"bd0054befffaf5170a8be681a6b8c3f5131e5effc537f9abe8275b7e866b9e9b"}]},{"id":"S065","semantic_type":"assumption","kind":"assumption","title":"收敛定理的有界性与KL假设","source_units":[{"start":31498,"end":31953,"start_line":490,"end_line":492,"raw_tex":"\\begin{theorem}\\label{convergence} In the settings of problem \\eqref{opt-pb}, for some starting points $x_0=y_0\\in\\R^m$ consider the sequences $(x_n)_{n\\in\\N},\\,(y_n)_{n\\in\\N}$ generated by Algorithm \\eqref{generaldiscrete}. Assume that $g$ is bounded from below and consider the function\r\n$$H:\\R^m\\times\\R^m\\To\\R,\\,H(x,y)=g(x)+\\frac12\\|y-x\\|^2.$$\r\nAssume that $(x_n)_{n\\in\\N}$ is bounded and $H$ is a KL function. Then the following statements are true\r\n","sha256_utf8":"ef96971a326f7705235405df41a5130d768b2976123df1c5b2564922daa2e4b6"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S011","target_id":"S010","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S003","target_id":"S014","kind":"supports"},{"source_id":"S004","target_id":"S014","kind":"supports"},{"source_id":"S010","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S004","target_id":"S015","kind":"supports"},{"source_id":"S004","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S061","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S061","target_id":"S018","kind":"supports"},{"source_id":"S004","target_id":"S019","kind":"supports"},{"source_id":"S061","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S064","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S064","target_id":"S021","kind":"supports"},{"source_id":"S060","target_id":"S021","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S015","target_id":"S013","kind":"supports"},{"source_id":"S016","target_id":"S013","kind":"supports"},{"source_id":"S017","target_id":"S013","kind":"supports"},{"source_id":"S018","target_id":"S013","kind":"supports"},{"source_id":"S019","target_id":"S013","kind":"supports"},{"source_id":"S020","target_id":"S013","kind":"supports"},{"source_id":"S021","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S009","kind":"supports"},{"source_id":"S009","target_id":"S024","kind":"supports"},{"source_id":"S019","target_id":"S024","kind":"supports"},{"source_id":"S004","target_id":"S024","kind":"supports"},{"source_id":"S005","target_id":"S024","kind":"supports"},{"source_id":"S062","target_id":"S024","kind":"supports"},{"source_id":"S012","target_id":"S025","kind":"supports"},{"source_id":"S062","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S026","kind":"supports"},{"source_id":"S062","target_id":"S026","kind":"supports"},{"source_id":"S004","target_id":"S027","kind":"supports"},{"source_id":"S062","target_id":"S027","kind":"supports"},{"source_id":"S004","target_id":"S028","kind":"supports"},{"source_id":"S062","target_id":"S028","kind":"supports"},{"source_id":"S060","target_id":"S028","kind":"supports"},{"source_id":"S025","target_id":"S029","kind":"supports"},{"source_id":"S063","target_id":"S030","kind":"supports"},{"source_id":"S024","target_id":"S030","kind":"supports"},{"source_id":"S026","target_id":"S030","kind":"supports"},{"source_id":"S060","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S005","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S063","target_id":"S032","kind":"supports"},{"source_id":"S005","target_id":"S032","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S023","kind":"supports"},{"source_id":"S026","target_id":"S023","kind":"supports"},{"source_id":"S027","target_id":"S023","kind":"supports"},{"source_id":"S028","target_id":"S023","kind":"supports"},{"source_id":"S029","target_id":"S023","kind":"supports"},{"source_id":"S030","target_id":"S023","kind":"supports"},{"source_id":"S031","target_id":"S023","kind":"supports"},{"source_id":"S032","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S037","kind":"supports"},{"source_id":"S012","target_id":"S037","kind":"supports"},{"source_id":"S065","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S020","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S022","target_id":"S039","kind":"supports"},{"source_id":"S033","target_id":"S039","kind":"supports"},{"source_id":"S007","target_id":"S039","kind":"supports"},{"source_id":"S065","target_id":"S039","kind":"supports"},{"source_id":"S022","target_id":"S040","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S006","target_id":"S041","kind":"supports"},{"source_id":"S060","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S020","target_id":"S042","kind":"supports"},{"source_id":"S022","target_id":"S042","kind":"supports"},{"source_id":"S019","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S060","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S009","target_id":"S044","kind":"supports"},{"source_id":"S006","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S022","target_id":"S045","kind":"supports"},{"source_id":"S060","target_id":"S045","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S039","target_id":"S036","kind":"supports"},{"source_id":"S040","target_id":"S036","kind":"supports"},{"source_id":"S041","target_id":"S036","kind":"supports"},{"source_id":"S042","target_id":"S036","kind":"supports"},{"source_id":"S043","target_id":"S036","kind":"supports"},{"source_id":"S044","target_id":"S036","kind":"supports"},{"source_id":"S045","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S047","kind":"supports"},{"source_id":"S002","target_id":"S047","kind":"supports"},{"source_id":"S004","target_id":"S047","kind":"supports"},{"source_id":"S018","target_id":"S048","kind":"supports"},{"source_id":"S022","target_id":"S048","kind":"supports"},{"source_id":"S002","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S002","target_id":"S049","kind":"supports"},{"source_id":"S008","target_id":"S049","kind":"supports"},{"source_id":"S022","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S019","target_id":"S050","kind":"supports"},{"source_id":"S002","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S060","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S019","target_id":"S053","kind":"supports"},{"source_id":"S010","target_id":"S054","kind":"supports"},{"source_id":"S004","target_id":"S054","kind":"supports"},{"source_id":"S060","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S020","target_id":"S055","kind":"supports"},{"source_id":"S037","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S043","target_id":"S057","kind":"supports"},{"source_id":"S035","target_id":"S057","kind":"supports"},{"source_id":"S002","target_id":"S057","kind":"supports"},{"source_id":"S009","target_id":"S057","kind":"supports"},{"source_id":"S008","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S050","target_id":"S058","kind":"supports"},{"source_id":"S053","target_id":"S058","kind":"supports"},{"source_id":"S002","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S004","target_id":"S059","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S049","target_id":"S046","kind":"supports"},{"source_id":"S050","target_id":"S046","kind":"supports"},{"source_id":"S051","target_id":"S046","kind":"supports"},{"source_id":"S052","target_id":"S046","kind":"supports"},{"source_id":"S053","target_id":"S046","kind":"supports"},{"source_id":"S054","target_id":"S046","kind":"supports"},{"source_id":"S055","target_id":"S046","kind":"supports"},{"source_id":"S056","target_id":"S046","kind":"supports"},{"source_id":"S057","target_id":"S046","kind":"supports"},{"source_id":"S058","target_id":"S046","kind":"supports"},{"source_id":"S059","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S001","kind":"supports"}]}}
{"record_id":"1807.07680v5","paper":{"title":"Generalized Stochastic Frank-Wolfe Algorithm with Stochastic \"Substitute\" Gradient for Structured Convex Optimization","authors":["Haihao Lu","Robert M. Freund"],"arxiv_id":"1807.07680","version":"v5","journal":"Mathematical Programming","doi":"10.1007/s10107-020-01480-7","abstract_url":"https://arxiv.org/abs/1807.07680v5","pdf_url":"https://arxiv.org/pdf/1807.07680v5"},"context":{"raw_tex":"\\begin{equation}\\label{eq:primal}\n\\mathrm{(P):} \\ \\ \\ \\ \\ \\ \\ \\min_\\beta P(\\beta):=\\tfrac{1}{n }\\sum_{j=1}^{n}l_{j}(x_j^T\\beta)+R(\\beta) \\ ,\n\\end{equation}\n\nwhere $\\beta \\in \\RR^p$, $l_j(\\cdot) : \\RR \\rightarrow \\RR$, $j=1, \\ldots, n$, is a univariate function (the $j^{\\mathrm{th}}$ loss function), $s_j = x_j^T\\beta$ is the ``predicted value'' of the model $\\beta$ for the data sample $x_j$, and $R(\\cdot)$ is some other function that can be used to model a variety of properties such as a regularizer, an indicator function of a feasible region $Q$, a penalty term, coupling constraints, etc.  Notice that the predicted value $s_j= x_j^T\\beta$ is a linear function of $\\beta$.  Hence in the intentional context of empirical loss minimization we refer to \\eqref{eq:primal} as (empirical) loss minimization with ``linear prediction.''  This structure arises quite generally throughout statistical and machine learning, which we illustrate below via several salient examples.  Throughout this paper we assume the following regarding the functions in our problem setup \\eqref{eq:primal}:\\medskip\n\n\n\\begin{ass}\\label{ass:smooth}  The following hold:\n\\begin{enumerate}\n\\item for $j=1, \\ldots, n$, the univariate function $l_j(\\cdot)$ is $\\gamma$-smooth, namely $| \\dot l_j(a) - \\dot l_j(b)| \\le \\gamma |a-b|$ for all $a$, $b$, and is strictly convex,\n\\item $\\dom R(\\cdot)$ is bounded, and the subproblem\n\\begin{equation}\\label{eq:sub}\n\\min_\\beta c^T\\beta +R(\\beta)\n\\end{equation}\nattains its optimum and can be easily solved for any $c$, and\n\\item $ 0 \\in \\dom R(\\cdot)$.\n\\end{enumerate}\n\\end{ass}\n\nLet $X \\in \\RR^{n \\times p}$ denote the data matrix whose rows are comprised of the vectors $x_1, \\ldots, x_n$, i.e., the $j^{\\mathrm{th}}$ row of $X$ is the vector $x_j$, $j=1, \\ldots, n$.  Let us define $L(s): \\RR^n \\rightarrow \\RR$ by $L(s) :=\\sum_{j=1}^n l_j(s_j)$ which is the total losses associated with $s \\in \\RR^n$.  We refer to $s=X\\beta$ as the ``predicted values'' in the context of empirical loss minimization.\n\n\n\\begin{algorithm}\n\\caption{Stochastic Generalized Frank-Wolfe with Stochastic Substitute Gradient}\\label{al:SFW}\n$ \\ $\n\\begin{algorithmic}\n\\STATE {\\bf Initialize.}  Initialize with $\\bbeta^{-1}=0$, $s^0=0$, and substitute gradient $d^0=\\frac{1}{n}X^T\\nabla L(s^0)$, with\nstep-size sequences $\\{\\alpha_{i}\\} \\in (0,1]$ and $\\{\\eta_{i}\\} \\in (0,1]$.  \\medskip\n\n\\STATE  For iterations $i=0,1, \\ldots $\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf Solve l.o.o. subproblem:} Compute $\\tbeta^{i} \\in \\arg\\min_{\\beta}\\left\\{\\left( d^i\\right)^{T}\\beta+R(\\beta)\\right\\}$\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf Choose random index:}  Choose $j_{i} \\in {\\cal U}[1, \\ldots, n]$\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf Update $s$ value:} $s_{j_i}^{i+1}\\leftarrow(1-\\eta_{i})s_{j_i}^{i}+\\eta_{i}(x_{j_{i}}^{T}\\tbeta^{i})$,  and $s_{j}^{i+1} \\gets s_{j}^{i}$ for $j \\ne j_i$ \\medskip\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf Update substitute gradient:}  $d^{i+1}= \\frac{1}{n}X^T\\nabla L(s^{i+1}) =  d^i+\\frac{1}{n}\\left(\\dot{l}_{j_{i}}(s^{i+1}_{j_i})-\\dot{l}_{j_{i}}(s^i_{j_{i}})\\right)x_{j_{i}}$\\\\ \\medskip\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf Update primal variable:} $\\bbeta^{i} \\leftarrow (1-\\alpha_i) \\bbeta^{i-1} + \\alpha_i \\tbeta^i$.\\medskip\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf (Optional Accounting:)} $w^{i+1} \\gets \\nabla L(s^{i+1})$ \\medskip\n\\end{algorithmic}\n\\end{algorithm}\n\n\\subsection{Notation} We use $e_j$ to denote the $j^{\\mathrm{th}}$ unit coordinate vector in $\\RR^p$.  The $\\ell_p$ norm is denoted $\\|\\cdot\\|_p$.  We use $\\dl_j(\\cdot)$ to denote the first derivative of $l_j(\\cdot)$.  The Bregman distance function associated with a convex function $h(\\cdot)$ is defined as $\\Dh(y,x):= h(y) - h(x) - \\nabla h(x)^T(y-x)$.  We use $\\EE$ to denote expectation and $\\EE_{j_i}$ to denote expectation conditional on the randomly chosen index $j_i$.  For indicator functions, we use $\\bI_Q(\\cdot)$ to denote the indicator function for the set $Q$, namely $\\bI_Q(\\beta):=0$ if $\\beta \\in Q$, and $\\bI_Q(\\beta) :=+\\infty$ otherwise; and we use $\\bI_{\\{\\mathrm constraint\\}}(\\beta)$ to denote the indicator function of a particular constraint (or condition), namely $\\bI_{\\{\\mathrm constraint\\}}(\\beta):=0$ if the constraint is true at $\\beta$, and $\\bI_{\\{\\mathrm constraint\\}}(\\beta) :=+\\infty$ otherwise.  In a slight abuse of terminology we refer to the ``subgradient'' of a concave function when it is perhaps more technically accurate to refer to this as a sup-gradient.   A differentiable function $\\ff$ is $\\mu$-strongly convex with respect to a norm $\\|\\cdot\\|$ if it holds that $f(y) \\ge f(x) + \\nabla f(x)^T(y-x) + \\tfrac{\\mu}{2}\\|y-x\\|^2$ for all $x,y \\in \\dom f(\\cdot)$.  A differentiable function $\\ff$ is $\\mu$-strongly convex with respect to a reference function $h(\\cdot)$ if it holds that $f(y) \\ge f(x) + \\nabla f(x)^T(y-x) + \\mu \\Dh(y,x)$ for all $x,y \\in \\dom f(\\cdot)$.\n\n\n$$\nf^*(y) := \\sup_{x\\in \\dom f(\\cdot)} \\{y^T x - f(x)\\} \\ .\n$$\nWe will also be interested in the following dual problem of \\eqref{eq:primal} that is constructed using the conjugate functions of the component functions of  \\eqref{eq:primal}:\n\\begin{equation}\\label{eq:dual}\n\\mathrm{(D):} \\ \\ \\ \\ \\ \\ \\ \\max_{w} D(w):=-R^{*}\\left(-\\tfrac{1}{n}X^{T}w\\right)-\\tfrac{1}{n}\\sum_{j=1}^{n}l_{j}^{*}(w_{j}) \\ .\n\\end{equation}\nNotice that we can write:\n\\begin{equation}\\label{eq:altered}R^{*}\\left(-\\tfrac{1}{n}X^{T}w\\right)= - \\min_{\\beta}\\left\\{\\tfrac{1}{n} w^{T}X\\beta+R(\\beta)\\right\\}  \\ .\n\\end{equation}\nAlso, defining the convex/concave saddle-function $\\phi(\\cdot,\\cdot)$:\n\\begin{equation}\\label{eq:primal-dual}\n\\phi(\\beta,w):=\\tfrac{1}{n} w^{T}X\\beta-\\tfrac{1}{n}\\sum_{i=1}^{n}l_{i}^{*}(w_{i})+R(\\beta) \\ ,\n\\end{equation} we can write (P) and (D) in saddlepoint minimax format as:\n\n\\begin{equation}\\label{eq:minmax} \\mathrm{(P):} \\ \\ \\min_{\\beta} \\max_{w} \\phi(\\beta,w) \\ \\ \\ \\ \\  \\ \\ \\mathrm{and} \\ \\ \\ \\ \\ \\ \\ \\  \\mathrm{(D):} \\ \\ \\max_{w}\\min_{\\beta} \\phi(\\beta,w) \\ .\n\\end{equation}\n\n\n\\begin{algorithm}\n\\caption{Randomized Coordinate Mirror Descent applied to the dual problem \\eqref{eq:dual}}\\label{al:RCMD}\n$ \\ $\n\\begin{algorithmic}\n\\STATE {\\bf Initialize.}  Define the prox function $h(w):=\\frac{1}{n}\\sum_{i=1}^{n}l_{i}^{*}(w_{i})$.  Initialize with $w^{0}=\\arg\\min_{w} \\tfrac{1}{n}\\sum_{i=1}^{n}l_{i}^{*}(w_{i})$ and\nstep-size sequences $\\{\\alpha_{i}\\} \\in (0,1]$ and $\\{\\eta_{i}\\} \\in (0,1]$. (Optional: set $\\bbeta^{-1}=0$.)\\medskip\n\n\\STATE  For iterations $i=0,1, \\ldots $\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf Compute Randomized Coordinate of Subgradient of $D(\\cdot)$ at $w^i$}\\\\\n\n\\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ Compute $\\tbeta^{i} \\in \\arg\\min_{\\beta}\\left\\{\\left( \\tfrac{1}{n}(w^i)^T X \\beta+R(\\beta) \\right) \\right\\}$ \\\\ \\medskip\n\n\\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\    {\\bf Choose random index.}  Choose $j_{i} \\in {\\cal U}[1, \\ldots, n]$ \\\\\n%Compute $\\bar{\\beta}^{i}:=\\frac{2}{(4n+k)(k+1)}\\sum_{i=0}^{i}(2n+k)\\tilde{\\beta}^{i}=\\frac{(4n+k-1)k}{(4n+k)(k+1)}\\bar{\\beta}^{k-1}+\\frac{2(2n+k)}{(4n+k)(k+1)}\\tbeta^{i}$\n\\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\  {\\bf Compute subgradient coordinate vector:} $\\tilde{g}^i \\gets \\frac{1}{n}\\left(x_{j_{i}}^{T}\\tbeta^{i}-\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})\\right)e_{j_{i}}$ \\\\ \\medskip\n\n\\ \\ \\ \\ \\ \\ \\    {\\bf Update dual variable:} Compute $w^{i+1}=\\arg\\min_w \\left\\{ \\left\\langle -\\eta_{i}\\tilde g^i,w-w^{i}\\right\\rangle +D_{h}(w,w^{i})\\right\\} $ \\\\ \\medskip\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf (Optional Accounting:)} $\\bbeta^{i} \\leftarrow (1-\\alpha_i) \\bbeta^{i-1} + \\alpha_i \\tbeta^i$. \\medskip\n\n\\end{algorithmic}\n\\end{algorithm}\n\nLet $M := \\max_{\\beta \\in \\dom R(\\cdot)} \\max_{j=1, \\ldots, n} \\{| x_j^T\\beta | \\}$, and note that $M < +\\infty$ since $\\dom R(\\cdot)$ is bounded by Assumption \\ref{ass:smooth}.\\medskip\n\nLet ${\\cal W} \\subset \\RR^n$ be the set of ``optimal $w$ responses'' to values $\\beta \\in \\dom R(\\cdot)$ in the saddle-function $\\phi(\\beta, w)$, namely: $${\\cal W} := \\{ \\hw \\in \\RR^n : \\hw \\in \\arg \\max_w \\phi(\\hbeta, w) \\ \\mbox{for~some~} \\hbeta \\in \\dom R(\\cdot) \\} \\ , $$\n\nand let $D_{\\max}$ be any upper bound on $D_h (\\hw,w^0)$ as $\\hw$ ranges over all values in ${\\cal W}$, so that\n$$\nD_h (\\hw ,w^0)\\le D_{\\max} \\ \\  \\mbox{for~all~}  \\hw \\in \\cal W \\ .\n$$\nNote at the moment that there is no guarantee that $D_{\\max} < +\\infty$, but this will be remedied below in Proposition \\ref{prop:Dmax}.\\medskip\n\n\nNotice in Algorithm \\ref{al:SFW} and Algorithm \\ref{al:RCMD} that $j_i$ is a random variable; and that $s^i$, $d^i$, $w^i$, etc., are random variables that depend on all previous random variable values $j_0, j_1, \\ldots, j_{i-1}$, and we denote this string of random variables by \n\\begin{equation}\\label{eq:xi}\n\\xi_i=\\{j_0, j_1, \\ldots, j_{i-1}\\} \\ .\n\\end{equation}\n\n\n\\begin{prop}\\label{prop:gradient_conjugate}(see \\cite{avriel}, \\cite{zalinescu}, \\cite{kakade2012regularization})\nIf $f(\\cdot)$ is a closed convex function, then ${f}^{**}(\\cdot) = f(\\cdot)$.  Furthermore:\n\\begin{enumerate}\n\\item $\\ff$ is $\\gamma$-smooth with domain $\\RR^p$ with respect to the norm $\\|\\cdot\\|$ if and only if  $f^*(\\cdot)$ is $1/\\gamma$-strongly convex with respect to the (dual) norm $\\|\\cdot\\|^*$ .\n\\item If $\\ff$ is differentiable and strictly convex, then the following three conditions are equivalent:\n\\begin{enumerate}\n\\item $y = \\nabla f(x)$\n\\item $x = \\nabla f^*(y)$, and\n\\item $x^T y = f(x)+f^*(y)$ .\n\\end{enumerate}\n\\end{enumerate}\n\\end{prop}\n\n\\begin{lem}\\label{tpp} {\\bf (Three-Point Property \\cite{tseng})} Let $\\phi(x)$ be a convex function, and let $\\Dh(\\cdot, \\cdot)$ be the Bregman distance for $\\hh$. For a given vector $z$, let\n\\begin{equation*}\nz^+ := \\arg\\min_{x\\in Q} \\left\\{ \\phi(x) + \\Dh(x,z) \\right\\} \\ .\n\\end{equation*}\nThen\n\\begin{equation*}\n\\phi(x) + \\Dh(x,z) \\ge \\phi(z^+) + \\Dh(z^+, z) + \\Dh(x,z^+)\\ \\   for\\ all \\ x\\in Q \\ . \\qed\n\\end{equation*}\n\\end{lem}","units":[{"start":6529,"end":6684,"start_line":172,"end_line":174,"raw_tex":"\\begin{equation}\\label{eq:primal}\n\\mathrm{(P):} \\ \\ \\ \\ \\ \\ \\ \\min_\\beta P(\\beta):=\\tfrac{1}{n }\\sum_{j=1}^{n}l_{j}(x_j^T\\beta)+R(\\beta) \\ ,\n\\end{equation}","sha256_utf8":"9223db6c96bace13083b24761a27f63129d7ef157a1af3b96ee669fa757ffff8"},{"start":6685,"end":7623,"start_line":175,"end_line":175,"raw_tex":"where $\\beta \\in \\RR^p$, $l_j(\\cdot) : \\RR \\rightarrow \\RR$, $j=1, \\ldots, n$, is a univariate function (the $j^{\\mathrm{th}}$ loss function), $s_j = x_j^T\\beta$ is the ``predicted value'' of the model $\\beta$ for the data sample $x_j$, and $R(\\cdot)$ is some other function that can be used to model a variety of properties such as a regularizer, an indicator function of a feasible region $Q$, a penalty term, coupling constraints, etc.  Notice that the predicted value $s_j= x_j^T\\beta$ is a linear function of $\\beta$.  Hence in the intentional context of empirical loss minimization we refer to \\eqref{eq:primal} as (empirical) loss minimization with ``linear prediction.''  This structure arises quite generally throughout statistical and machine learning, which we illustrate below via several salient examples.  Throughout this paper we assume the following regarding the functions in our problem setup \\eqref{eq:primal}:\\medskip\n","sha256_utf8":"fcbb70d02c232ecc2599cbc9f3439ca26d5175e03cbccfb1a5efdf50474055a6"},{"start":7624,"end":8121,"start_line":177,"end_line":187,"raw_tex":"\\begin{ass}\\label{ass:smooth}  The following hold:\n\\begin{enumerate}\n\\item for $j=1, \\ldots, n$, the univariate function $l_j(\\cdot)$ is $\\gamma$-smooth, namely $| \\dot l_j(a) - \\dot l_j(b)| \\le \\gamma |a-b|$ for all $a$, $b$, and is strictly convex,\n\\item $\\dom R(\\cdot)$ is bounded, and the subproblem\n\\begin{equation}\\label{eq:sub}\n\\min_\\beta c^T\\beta +R(\\beta)\n\\end{equation}\nattains its optimum and can be easily solved for any $c$, and\n\\item $ 0 \\in \\dom R(\\cdot)$.\n\\end{enumerate}\n\\end{ass}","sha256_utf8":"b164ddf1a7caa4b1737992ae054b6224712c46e8f1e813e9bad530e3a01adc10"},{"start":13284,"end":13709,"start_line":238,"end_line":238,"raw_tex":"Let $X \\in \\RR^{n \\times p}$ denote the data matrix whose rows are comprised of the vectors $x_1, \\ldots, x_n$, i.e., the $j^{\\mathrm{th}}$ row of $X$ is the vector $x_j$, $j=1, \\ldots, n$.  Let us define $L(s): \\RR^n \\rightarrow \\RR$ by $L(s) :=\\sum_{j=1}^n l_j(s_j)$ which is the total losses associated with $s \\in \\RR^n$.  We refer to $s=X\\beta$ as the ``predicted values'' in the context of empirical loss minimization.\n","sha256_utf8":"f5526f89ef08433d6502a87e429fa85378165e8030043d6b974cadb44d3c1a3e"},{"start":14042,"end":15301,"start_line":242,"end_line":263,"raw_tex":"\\begin{algorithm}\n\\caption{Stochastic Generalized Frank-Wolfe with Stochastic Substitute Gradient}\\label{al:SFW}\n$ \\ $\n\\begin{algorithmic}\n\\STATE {\\bf Initialize.}  Initialize with $\\bbeta^{-1}=0$, $s^0=0$, and substitute gradient $d^0=\\frac{1}{n}X^T\\nabla L(s^0)$, with\nstep-size sequences $\\{\\alpha_{i}\\} \\in (0,1]$ and $\\{\\eta_{i}\\} \\in (0,1]$.  \\medskip\n\n\\STATE  For iterations $i=0,1, \\ldots $\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf Solve l.o.o. subproblem:} Compute $\\tbeta^{i} \\in \\arg\\min_{\\beta}\\left\\{\\left( d^i\\right)^{T}\\beta+R(\\beta)\\right\\}$\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf Choose random index:}  Choose $j_{i} \\in {\\cal U}[1, \\ldots, n]$\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf Update $s$ value:} $s_{j_i}^{i+1}\\leftarrow(1-\\eta_{i})s_{j_i}^{i}+\\eta_{i}(x_{j_{i}}^{T}\\tbeta^{i})$,  and $s_{j}^{i+1} \\gets s_{j}^{i}$ for $j \\ne j_i$ \\medskip\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf Update substitute gradient:}  $d^{i+1}= \\frac{1}{n}X^T\\nabla L(s^{i+1}) =  d^i+\\frac{1}{n}\\left(\\dot{l}_{j_{i}}(s^{i+1}_{j_i})-\\dot{l}_{j_{i}}(s^i_{j_{i}})\\right)x_{j_{i}}$\\\\ \\medskip\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf Update primal variable:} $\\bbeta^{i} \\leftarrow (1-\\alpha_i) \\bbeta^{i-1} + \\alpha_i \\tbeta^i$.\\medskip\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf (Optional Accounting:)} $w^{i+1} \\gets \\nabla L(s^{i+1})$ \\medskip\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"fbec6559081106611b5fe7df8d3584fd6ce600f83278608ff6bc401ec567efe6"},{"start":47615,"end":49131,"start_line":399,"end_line":399,"raw_tex":"\\subsection{Notation} We use $e_j$ to denote the $j^{\\mathrm{th}}$ unit coordinate vector in $\\RR^p$.  The $\\ell_p$ norm is denoted $\\|\\cdot\\|_p$.  We use $\\dl_j(\\cdot)$ to denote the first derivative of $l_j(\\cdot)$.  The Bregman distance function associated with a convex function $h(\\cdot)$ is defined as $\\Dh(y,x):= h(y) - h(x) - \\nabla h(x)^T(y-x)$.  We use $\\EE$ to denote expectation and $\\EE_{j_i}$ to denote expectation conditional on the randomly chosen index $j_i$.  For indicator functions, we use $\\bI_Q(\\cdot)$ to denote the indicator function for the set $Q$, namely $\\bI_Q(\\beta):=0$ if $\\beta \\in Q$, and $\\bI_Q(\\beta) :=+\\infty$ otherwise; and we use $\\bI_{\\{\\mathrm constraint\\}}(\\beta)$ to denote the indicator function of a particular constraint (or condition), namely $\\bI_{\\{\\mathrm constraint\\}}(\\beta):=0$ if the constraint is true at $\\beta$, and $\\bI_{\\{\\mathrm constraint\\}}(\\beta) :=+\\infty$ otherwise.  In a slight abuse of terminology we refer to the ``subgradient'' of a concave function when it is perhaps more technically accurate to refer to this as a sup-gradient.   A differentiable function $\\ff$ is $\\mu$-strongly convex with respect to a norm $\\|\\cdot\\|$ if it holds that $f(y) \\ge f(x) + \\nabla f(x)^T(y-x) + \\tfrac{\\mu}{2}\\|y-x\\|^2$ for all $x,y \\in \\dom f(\\cdot)$.  A differentiable function $\\ff$ is $\\mu$-strongly convex with respect to a reference function $h(\\cdot)$ if it holds that $f(y) \\ge f(x) + \\nabla f(x)^T(y-x) + \\mu \\Dh(y,x)$ for all $x,y \\in \\dom f(\\cdot)$.\n","sha256_utf8":"cf7e23c696c7e6a1ecee9e9a67470422ee51db372f938dc81c086c8aa30b5840"},{"start":49270,"end":50355,"start_line":404,"end_line":420,"raw_tex":"$$\nf^*(y) := \\sup_{x\\in \\dom f(\\cdot)} \\{y^T x - f(x)\\} \\ .\n$$\nWe will also be interested in the following dual problem of \\eqref{eq:primal} that is constructed using the conjugate functions of the component functions of  \\eqref{eq:primal}:\n\\begin{equation}\\label{eq:dual}\n\\mathrm{(D):} \\ \\ \\ \\ \\ \\ \\ \\max_{w} D(w):=-R^{*}\\left(-\\tfrac{1}{n}X^{T}w\\right)-\\tfrac{1}{n}\\sum_{j=1}^{n}l_{j}^{*}(w_{j}) \\ .\n\\end{equation}\nNotice that we can write:\n\\begin{equation}\\label{eq:altered}R^{*}\\left(-\\tfrac{1}{n}X^{T}w\\right)= - \\min_{\\beta}\\left\\{\\tfrac{1}{n} w^{T}X\\beta+R(\\beta)\\right\\}  \\ .\n\\end{equation}\nAlso, defining the convex/concave saddle-function $\\phi(\\cdot,\\cdot)$:\n\\begin{equation}\\label{eq:primal-dual}\n\\phi(\\beta,w):=\\tfrac{1}{n} w^{T}X\\beta-\\tfrac{1}{n}\\sum_{i=1}^{n}l_{i}^{*}(w_{i})+R(\\beta) \\ ,\n\\end{equation} we can write (P) and (D) in saddlepoint minimax format as:\n\n\\begin{equation}\\label{eq:minmax} \\mathrm{(P):} \\ \\ \\min_{\\beta} \\max_{w} \\phi(\\beta,w) \\ \\ \\ \\ \\  \\ \\ \\mathrm{and} \\ \\ \\ \\ \\ \\ \\ \\  \\mathrm{(D):} \\ \\ \\max_{w}\\min_{\\beta} \\phi(\\beta,w) \\ .\n\\end{equation}\n","sha256_utf8":"9be759d3cc173e3cb3ed579c908d14797512758c1660c8db20f21c0ec567f8b3"},{"start":52495,"end":54029,"start_line":431,"end_line":453,"raw_tex":"\\begin{algorithm}\n\\caption{Randomized Coordinate Mirror Descent applied to the dual problem \\eqref{eq:dual}}\\label{al:RCMD}\n$ \\ $\n\\begin{algorithmic}\n\\STATE {\\bf Initialize.}  Define the prox function $h(w):=\\frac{1}{n}\\sum_{i=1}^{n}l_{i}^{*}(w_{i})$.  Initialize with $w^{0}=\\arg\\min_{w} \\tfrac{1}{n}\\sum_{i=1}^{n}l_{i}^{*}(w_{i})$ and\nstep-size sequences $\\{\\alpha_{i}\\} \\in (0,1]$ and $\\{\\eta_{i}\\} \\in (0,1]$. (Optional: set $\\bbeta^{-1}=0$.)\\medskip\n\n\\STATE  For iterations $i=0,1, \\ldots $\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf Compute Randomized Coordinate of Subgradient of $D(\\cdot)$ at $w^i$}\\\\\n\n\\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ Compute $\\tbeta^{i} \\in \\arg\\min_{\\beta}\\left\\{\\left( \\tfrac{1}{n}(w^i)^T X \\beta+R(\\beta) \\right) \\right\\}$ \\\\ \\medskip\n\n\\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\    {\\bf Choose random index.}  Choose $j_{i} \\in {\\cal U}[1, \\ldots, n]$ \\\\\n%Compute $\\bar{\\beta}^{i}:=\\frac{2}{(4n+k)(k+1)}\\sum_{i=0}^{i}(2n+k)\\tilde{\\beta}^{i}=\\frac{(4n+k-1)k}{(4n+k)(k+1)}\\bar{\\beta}^{k-1}+\\frac{2(2n+k)}{(4n+k)(k+1)}\\tbeta^{i}$\n\\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\  {\\bf Compute subgradient coordinate vector:} $\\tilde{g}^i \\gets \\frac{1}{n}\\left(x_{j_{i}}^{T}\\tbeta^{i}-\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})\\right)e_{j_{i}}$ \\\\ \\medskip\n\n\\ \\ \\ \\ \\ \\ \\    {\\bf Update dual variable:} Compute $w^{i+1}=\\arg\\min_w \\left\\{ \\left\\langle -\\eta_{i}\\tilde g^i,w-w^{i}\\right\\rangle +D_{h}(w,w^{i})\\right\\} $ \\\\ \\medskip\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf (Optional Accounting:)} $\\bbeta^{i} \\leftarrow (1-\\alpha_i) \\bbeta^{i-1} + \\alpha_i \\tbeta^i$. \\medskip\n\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"fe94b296b697c4dee0e2ddfa6ba80f01a5502456032f30c33963c860c9d2f47e"},{"start":60158,"end":60954,"start_line":526,"end_line":534,"raw_tex":"Let $M := \\max_{\\beta \\in \\dom R(\\cdot)} \\max_{j=1, \\ldots, n} \\{| x_j^T\\beta | \\}$, and note that $M < +\\infty$ since $\\dom R(\\cdot)$ is bounded by Assumption \\ref{ass:smooth}.\\medskip\n\nLet ${\\cal W} \\subset \\RR^n$ be the set of ``optimal $w$ responses'' to values $\\beta \\in \\dom R(\\cdot)$ in the saddle-function $\\phi(\\beta, w)$, namely: $${\\cal W} := \\{ \\hw \\in \\RR^n : \\hw \\in \\arg \\max_w \\phi(\\hbeta, w) \\ \\mbox{for~some~} \\hbeta \\in \\dom R(\\cdot) \\} \\ , $$\n\nand let $D_{\\max}$ be any upper bound on $D_h (\\hw,w^0)$ as $\\hw$ ranges over all values in ${\\cal W}$, so that\n$$\nD_h (\\hw ,w^0)\\le D_{\\max} \\ \\  \\mbox{for~all~}  \\hw \\in \\cal W \\ .\n$$\nNote at the moment that there is no guarantee that $D_{\\max} < +\\infty$, but this will be remedied below in Proposition \\ref{prop:Dmax}.\\medskip\n","sha256_utf8":"aba4003c263bccd854efb6037fdecd1ed44eed552d38fa0e0f7ab7b969ab2915"},{"start":64915,"end":65281,"start_line":585,"end_line":588,"raw_tex":"Notice in Algorithm \\ref{al:SFW} and Algorithm \\ref{al:RCMD} that $j_i$ is a random variable; and that $s^i$, $d^i$, $w^i$, etc., are random variables that depend on all previous random variable values $j_0, j_1, \\ldots, j_{i-1}$, and we denote this string of random variables by \n\\begin{equation}\\label{eq:xi}\n\\xi_i=\\{j_0, j_1, \\ldots, j_{i-1}\\} \\ .\n\\end{equation}\n","sha256_utf8":"5e415e940e6cc2c516c568a25fc5f3be7a09a3c5dfdc37ec86105329915373fb"},{"start":103058,"end":103728,"start_line":1051,"end_line":1062,"raw_tex":"\\begin{prop}\\label{prop:gradient_conjugate}(see \\cite{avriel}, \\cite{zalinescu}, \\cite{kakade2012regularization})\nIf $f(\\cdot)$ is a closed convex function, then ${f}^{**}(\\cdot) = f(\\cdot)$.  Furthermore:\n\\begin{enumerate}\n\\item $\\ff$ is $\\gamma$-smooth with domain $\\RR^p$ with respect to the norm $\\|\\cdot\\|$ if and only if  $f^*(\\cdot)$ is $1/\\gamma$-strongly convex with respect to the (dual) norm $\\|\\cdot\\|^*$ .\n\\item If $\\ff$ is differentiable and strictly convex, then the following three conditions are equivalent:\n\\begin{enumerate}\n\\item $y = \\nabla f(x)$\n\\item $x = \\nabla f^*(y)$, and\n\\item $x^T y = f(x)+f^*(y)$ .\n\\end{enumerate}\n\\end{enumerate}\n\\end{prop}","sha256_utf8":"a0529bc7e17495c292c409c97360d528d686882ae57a3ffa1040a280a5a0f78e"},{"start":103865,"end":104296,"start_line":1069,"end_line":1077,"raw_tex":"\\begin{lem}\\label{tpp} {\\bf (Three-Point Property \\cite{tseng})} Let $\\phi(x)$ be a convex function, and let $\\Dh(\\cdot, \\cdot)$ be the Bregman distance for $\\hh$. For a given vector $z$, let\n\\begin{equation*}\nz^+ := \\arg\\min_{x\\in Q} \\left\\{ \\phi(x) + \\Dh(x,z) \\right\\} \\ .\n\\end{equation*}\nThen\n\\begin{equation*}\n\\phi(x) + \\Dh(x,z) \\ge \\phi(z^+) + \\Dh(z^+, z) + \\Dh(x,z^+)\\ \\   for\\ all \\ x\\in Q \\ . \\qed\n\\end{equation*}\n\\end{lem}","sha256_utf8":"343fd72211540287e891e5c3912ced21f2b952c1a83e67bbc2df94abce3f8a95"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{thm}\\label{thm:non-strong}\nConsider the Stochastic Generalized Frank-Wolfe method (Algorithm \\ref{al:SFW}) or the Randomized Dual Coordinate Mirror Descent method (Algorithm \\ref{al:RCMD}), with step-size sequences $\\alpha_i=\\frac{2(2n+i)}{(i+1)(4n+i)}$ and $\\eta_i=\\frac{2n}{2n+i+1}$ for $i=0, 1, \\ldots$.  Denote $$\\bw^{k} = \\frac{2}{(4n+k)(k+1)}\\sum_{i=0}^k(2n+i) w^i \\ . $$\nUnder Assumption \\ref{ass:smooth}, it holds for all $k \\ge 0$ that\n$$\n\\EE_{\\xi_k}\\left[ P(\\bbeta^k) - D(\\bw^k)\\right] \\le \\frac{8n\\gamma M^{2}}{\\left(4n+k\\right)}+\\frac{2n(2n-1)D_{\\max}}{(4n+k)(k+1)}  \\ \\le \\ \\frac{8n\\gamma M^{2}}{\\left(4n+k\\right)}+\\frac{2n(2n-1)\\gamma M^2}{(4n+k)(k+1)} \\ . \\ \\qed\n$$\n\\end{thm}","units":[{"start":65413,"end":66110,"start_line":592,"end_line":598,"raw_tex":"\\begin{thm}\\label{thm:non-strong}\nConsider the Stochastic Generalized Frank-Wolfe method (Algorithm \\ref{al:SFW}) or the Randomized Dual Coordinate Mirror Descent method (Algorithm \\ref{al:RCMD}), with step-size sequences $\\alpha_i=\\frac{2(2n+i)}{(i+1)(4n+i)}$ and $\\eta_i=\\frac{2n}{2n+i+1}$ for $i=0, 1, \\ldots$.  Denote $$\\bw^{k} = \\frac{2}{(4n+k)(k+1)}\\sum_{i=0}^k(2n+i) w^i \\ . $$\nUnder Assumption \\ref{ass:smooth}, it holds for all $k \\ge 0$ that\n$$\n\\EE_{\\xi_k}\\left[ P(\\bbeta^k) - D(\\bw^k)\\right] \\le \\frac{8n\\gamma M^{2}}{\\left(4n+k\\right)}+\\frac{2n(2n-1)D_{\\max}}{(4n+k)(k+1)}  \\ \\le \\ \\frac{8n\\gamma M^{2}}{\\left(4n+k\\right)}+\\frac{2n(2n-1)\\gamma M^2}{(4n+k)(k+1)} \\ . \\ \\qed\n$$\n\\end{thm}","sha256_utf8":"f52435a81130247c0080406b3d9aad1931ac0f648263d566d406db94a3d8be16"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lem}\\label{lem:equi} {\\bf (Equivalence Lemma)}  Algorithm \\ref{al:SFW} and Algorithm \\ref{al:RCMD} are equivalent as follows:  the iterate sequence of either algorithm exactly corresponds to an iterate sequences of the other. \\qed\n\\end{lem}\n\n\\begin{prop}\\label{prop:wupdate} At iteration $i$ of Algorithm \\ref{al:RCMD} it holds that:\n\n(1.) $\\dlsji(w_{j_{i}}^{i+1})=(1-\\eta_{i})\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i}) + \\eta_{i}x_{j_{i}}^{T}\\tbeta^{i}$, and\n\n(2.) $w_{j_{i}}^{i+1}=\\dot{l}_{j_{i}}\\left((1-\\eta_{i})\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})+ \\eta_{i}x_{j_{i}}^{T}\\tbeta^{i}\\right) $.\n\\end{prop}\n\n{\\bf Proof:}\nBecause $h(w)$ is a coordinate-wise separable function, we can rewrite the update for $w_{j_{i}}^{i+1}$ as\n\\begin{equation*}\n\\begin{array}{ll}\nw_{j_{i}}^{i+1} & =\\arg\\min_{w_{j_{i}}} \\left\\langle -\\frac{\\eta_{i}}{n}\\left(x_{j_{i}}^{T}\\tbeta^{i}-\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})\\right),w_{j_{i}}\\right\\rangle +D_{\\frac{1}{n} l_{j_{i}}^{*}}(w_{j_{i}},w_{j_{i}}^{i})\\\\ \\\\\n\n& =\\arg\\min_{w_{j_{i}}}\\left\\langle -\\eta_{i}\\left(x_{j_{i}}^{T}\\tbeta^{i}-\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})\\right),w_{j_{i}}\\right\\rangle +D_{ l_{j_{i}}^{*}}(w_{j_{i}},w_{j_{i}}^{i})\\\\ \\\\\n\n& =\\arg\\min_{w_{j_{i}}}\\left\\langle -\\eta_{i}x_{j_{i}}^{T}\\tbeta^{i}-(1-\\eta_{i})\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i}),w_{j_{i}}\\right\\rangle +l_{j_{i}}^{*}(w_{j_{i}}) \\ .  \\end{array}\n \\end{equation*}\n\nFrom the first-order optimality condition of the above $1$-dimensional problem we have $\\dls_{j_i}(w_{j_i}^{i+1})=\\eta_{i}x_{j_{i}}^{T}\\tbeta^{i}+(1-\\eta_{i})\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})$, which shows {\\it (1.)}; and {\\it (2.)} follows directly from {\\it (1.)} by the properties of the conjugate function in Proposition \\ref{prop:gradient_conjugate}. \\qed\n\n\n{\\bf Proof of Lemma \\ref{lem:equi}}  We show that the iterate sequence of Algorithm \\ref{al:RCMD} corresponds exactly to an iterate sequence of Algorithm \\ref{al:SFW}.  The $\\{s^i\\}$ sequence is not formally defined in Algorithm \\ref{al:RCMD}, so let us define $s^i := \\nabla L^*(w^i)$ for all $i = 0, 1, \\ldots$, which is consistent through conjugacy with the relationship $w^i = \\nabla L(s^i)$ in the Optional Accounting step of Algorithm \\ref{al:SFW} (see Proposition \\ref{prop:gradient_conjugate}).  In order to show the correspondence we proceed by induction on the iteration counter $i$.  For $i=0$ we have from conjugacy that $s^0 := \\nabla L^*(w^0) = 0 $ from the definition $w^0$ in the  initialization of Algorithm \\ref{al:RCMD}.  We also need to show that $\\tbeta^0$ is a solution to the linear optimization oracle problem in Algorithm \\ref{al:SFW}.  We have for all $i=0, \\ldots$, that:\n$$\\begin{array}{rcl}\n\\tbeta^{i} \\in \\arg\\min_\\beta\\left\\{ \\tfrac{1}{n}\\left(w^{i}\\right)^{T}X\\beta+R(\\beta)\\right\\}    &=& \\arg\\min_\\beta\\left\\{ \\tfrac{1}{n}\\left(\\nabla L(s^{i})\\right)^{T}X\\beta+R(\\beta)\\right\\} \\\\ \\\\\n&=& \\arg\\min_\\beta\\left\\{ \\left(d^{i}\\right)^{T}\\beta+R(\\beta)\\right\\} \\ ,\n\\end{array}$$thus showing that $\\beta^{i}$ corresponds to a linear optimization oracle solution at iteration $i$ in Algorithm \\ref{al:SFW} for all $i=0, \\ldots$.  Now suppose that the correspondence holds for some iteration counter $i$, and let us examine $s^{i+1}:= \\nabla L^*(w^{i+1})$.  We have from Proposition \\ref{prop:wupdate} that:\n\\begin{equation}\\label{robo3}\nw_{j_{i}}^{i+1} =\\dot{l}_{j_{i}}\\left((1-\\eta_{i})\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i}) + \\eta_{i}x_{j_{i}}^{T}\\tbeta^{i} \\right)=\\dot{l}_{j_{i}}\\left((1-\\eta_{i})s_{j_{i}}^{i} + \\eta_{i}x_{j_{i}}^{T}\\tbeta^{i} \\right) \\ ,\n\\end{equation}\nwhere the first equality is from Proposition \\ref{prop:wupdate} and the second equality uses induction.  This then implies that $$ (1-\\eta_{i})s_{j_{i}}^{i} + \\eta_{i}x_{j_{i}}^{T}\\tbeta^{i} = \\dl (w_{j_{i}}^{i+1}) = s^{i+1} \\ . $$\nAnd for all coefficient indices $j \\ne i $ we have\n$$s_j^{i+1} = \\dl^*(w^{i+1}_j) = \\dl(w^i_j)= s^i \\ , $$\nwhere the second equality follows from conjugacy, whereby $s^{i+1}$ satisfies the update rule as stated in Algorithm \\ref{al:SFW}, thus demonstrating that the iterate sequence of Algorithm \\ref{al:RCMD} corresponds exactly to an iterate sequence of Algorithm \\ref{al:SFW}.  The same type of analysis as above can be used to prove that the iterate sequence of Algorithm \\ref{al:SFW} corresponds exactly to an iterate sequence of Algorithm \\ref{al:RCMD}. \\qed\n\n\n\\begin{prop}\\label{prop:Dmax}\n\tUnder Assumption \\ref{ass:smooth} it holds that $D_{\\max} \\le \\gamma M^2$.\n\\end{prop}\n\n\\begin{prop}\\label{prop:well-behavior-of-w}\n\tLet $T:=\\left\\{ w \\in \\RR^n : |\\|w-w^{0}\\|_{\\infty}\\le \\gamma M\\right\\}$.  Then:\n\t\\begin{enumerate}\n\t\t\\item for any $\\hbeta\\in \\dom R(\\cdot)$ it holds that $\\argmax_w \\phi (\\hbeta, w) \\in T$, and\n\t\t\\item for all $w^i$ generated in Algorithm \\ref{al:RCMD}, it holds that  $w^i \\in T$.\n\t\\end{enumerate}\n\\end{prop}\n\n\\textbf{Proof.} We first prove {\\it (1.)}.  Notice that $w_{0}= \\nabla L(0)$ and $\\argmax_w \\phi (\\hbeta, w) = \\nabla L(X\\hbeta)$ (from conjugacy via Proposition \\ref{prop:gradient_conjugate}) , whereby the $\\gamma$-smoothness of $l_{j}(\\cdot)$ implies that\n$$\n\\left\\|\\arg\\max_{w}\\phi(\\hat{\\beta},w)-w_{0}\\right\\|_{\\infty}=\\left\\| \\nabla L(X\\hbeta)-\\nabla L(0)\\right\\|_{\\infty}=\\max_j \\left|\\dl_j(x_j^T\\hat \\beta)-\\dl_j(0)\\right|\\le\\gamma\\max_{j}|x_{j}^{T}\\hat \\beta| \\le \\gamma M \\ ,\n$$\n\n\nwhich proves {\\it (1.)}. It follows from Proposition \\ref{cor:bound} that for any coordinate $j$ and iterate $i$ it holds that $\\left|\\dot{l}_{j}^{*}(w_{j}^{i})\\right|\\le M$. Together with $\\dls_j(w_j^0)=0$, we have\n\n$$\n\\tfrac{1}{\\gamma}\\left|w_{j}^{i}-w_{j}^{0}\\right|\\le \\left|\\dot{l}_{j}^{*}(w_{j}^{i})-\\dot{l}_{j}^{*}(w_{j}^{0}) \\right|\\le M \\ ,\n$$\n(where the first inequality is from the $\\frac{1}{\\gamma}$-strong convexity of $l_j^*(w_j)$), from which it follows that $\\|w^i - w^0\\|_\\infty \\le \\gamma M$, which proves {\\it (2.)}. \\qed\n\n\n{\\bf Proof of Proposition \\ref{prop:Dmax}:}  Let $L(s):=\\sum_{j=1}^n l_j(s_j)$ and $L^*(w):=\\sum_{j=1}^n l^*_j(w_j)$, and note that $L(\\cdot)$ and $L^*(\\cdot)$ are a conjugate pair.  Let $\\hw \\in \\cal W$ and let $\\hbeta$ be such that $\\hw \\in \\arg\\max_w \\phi(\\hbeta, w)$.  Then\n\n%It follows Proposition \\ref{prop:well-behavior-of-w}.1 that $\\hw^k\\in T$, therefore for any $k$ we have\n\n\\begin{equation*}\n\\begin{array}{lcl}\nD_{h}(\\hw, w^0) & = &\\tfrac{1}{n}\\left( L^*(\\hw)-L^*(w^0)\\right)\\\\ \\\\\n& =&\\tfrac{1}{n}\\left( \\left(\\hw \\right)^{T}X\\hbeta-L(X\\hbeta) -L^{*}(w^{0})\\right)\\\\ \\\\\n& \\le&\\tfrac{1}{n}\\left(\\max_{w\\in T,\\beta\\in \\dom R(\\cdot)}\\left\\{ w^{T}X\\beta-L(X\\beta)\\right\\} -L^{*}(w^{0})\\right)\\\\ \\\\\n& =&\\tfrac{1}{n}\\left(\\max_{w\\in T,\\beta\\in  \\dom R(\\cdot)}\\left\\{ (w-w^{0})^{T}X\\beta+\\left(w^{0}\\right)^{T}X\\beta-L(X\\beta)\\right\\} -L^{*}(w^{0})\\right)\\\\ \\\\\n& \\le&\\tfrac{1}{n}\\left(\\max_{w\\in T,\\beta\\in  \\dom R(\\cdot)}\\left\\{ (w-w^{0})^{T}X\\beta\\right\\} +\\max_{\\beta\\in  \\dom R(\\cdot)}\\left\\{ \\left(w^{0}\\right)^{T}X\\beta-L(X\\beta)\\right\\} -L^{*}(w^{0})\\right)\\\\ \\\\\n& \\le &\\tfrac{1}{n}\\left(n\\max_{w\\in T,\\beta\\in  \\dom R(\\cdot)}\\|w-w^{0}\\|_{\\infty}\\|X\\beta\\|_{\\infty}+L^{*}(w^{0})-L^{*}(w^{0})\\right)\\\\ \\\\\n& \\le &\\gamma M^{2} \\ ,\n\\end{array}\n\\end{equation*}\n\nwhere the second equality follows from Proposition \\ref{prop:gradient_conjugate}, the first inequality uses $\\hbeta \\in  \\dom R(\\cdot)$ and $\\hw \\in T$ (from Proposition \\ref{prop:well-behavior-of-w}), and the last inequality uses $\\max_{\\beta\\in  \\dom R(\\cdot)}\\|X\\beta\\|_{\\infty}\\le M$. \\qed \\medskip\n\n\n\\begin{prop}\nFor all iterates $i$ and any $j\\in\\left\\{ 1,\\ldots,n\\right\\} $ it holds that $\\left|\\dot{l}_{j}^{*}(w_{j}^{i})\\right|\\le M$.\n\\end{prop}\n\n{\\bf Proof.} We prove this by induction on $i$. The proposition is true for $i=0$ because $\\dot{l}_{j}^{*}(w_{j}^{0})=0$ for all $j$ by the definition of $w^0$. Next suppose that $\\left|\\dot{l}_{j}^{*}(w_{j}^{i})\\right|\\le M$ for a given iterate $i$ and for all $j=1, \\ldots, n$. Then at iteration $i+1$ and any $j\\not=j_{i}$ we have  $w_j^{i+1}=w_j^i$, whereby $\\left|\\dot{l}_{j}^{*}(w_{j}^{i+1})\\right|=\\left|\\dot{l}_{j}^{*}(w_{j}^{i})\\right|\\le M$.  And it follows from Proposition \\ref{prop:wupdate} that\n\\[\n\\left|\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i+1})\\right|=\\left|\\left(1-\\eta_{i}\\right)\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})+\\eta_{i}x_{j_{i}}^T\\tbeta^{i}\\right|\\le(1-\\eta_{i})M+\\eta_{i}M=M \\ ,\n\\]\nand therefore for any $j=1, \\ldots, n$, we have $\\left|\\dot{l}_{j}^{*}(w_{j}^{i+1})\\right|\\le M$, which completes the proof by induction. \\qed\n\n\n\\begin{cor}\\label{cor:bound}\n$\\|\\tilde{g}^i\\|_{2} =\\frac{1}{n}\\left| x_{j_{i}}^T \\tbeta^i - \\dlsji(\\wiji) \\right| \\le \\frac{2M}{n}$.\n\\end{cor}\n\n\\begin{prop}\\label{prop:strong}\n$h(\\cdot)$ is\n$\\frac{1}{n\\gamma}$-strongly convex with respect to the norm $\\|\\cdot\\|_{2}$.\n\\end{prop}\n\n{\\bf Proof.} Recall that $h(w)=\\frac{1}{n}\\sum_{j=1}^{n}l_{j}^{*}(w_{j})$. It follows from Assumption \\ref{ass:smooth} and Proposition \\ref{prop:gradient_conjugate} that $\\dls_j(\\cdot)$ is $\\frac{1}{\\gamma}$-strongly convex. Therefore for any $w^1, w^2 \\in \\dom h(\\cdot)$ it holds that:\n\\begin{equation*}\n\\begin{array}{lcl}\nh(w^1)& = &\\frac{1}{n}\\sum_{j=1}^{n}l_{j}^{*}(w^1_{j}) \\\\ \\\\\n& \\ge & \\frac{1}{n}\\sum_{j=1}^{n}\\left( l_{j}^{*}(w^2_{j}) + \\dls_j(w_j^2)(w_j^1-w_j^2)+\\frac{1}{2\\gamma}|w^2_j-w^1_j|^2\\right) \\\\ \\\\\n& = & h(w^2) + \\langle\\nabla h(w^2), w^1-w^2\\rangle + \\frac{1}{2n\\gamma}\\|w^2-w^1\\|_2^2 \\ . \\ \\qed\n\\end{array}\n\\end{equation*}\n\n\n\\begin{prop}\\label{prop:equality}\n$\\phi(\\tbeta^{i},w)=D(w^{i})+\\left\\langle \\nabla_{w}\\phi(\\tbeta^{i},w^{i}),w-w^{i}\\right\\rangle -D_{h}(w,w^{i}).$\n\\end{prop}\n\n\\textbf{Proof. } The proof follows from straightforward substitution using $\\phi(\\tbeta^{i},w)=\\frac{1}{n} \\left(w^{T}X\\tbeta^{i}-\\sum_{j=1}^n l_{j}^{*}(w_{j})\\right) + R(\\tbeta^i)$ and noticing from the construction of $\\tbeta^i$ that $D(w^i)=\\phi(\\tbeta^i, w^i)$. \\qed\\medskip\n\n\n\\begin{prop}\\label{robfriday}  Consider the series $\\{ \\alpha_i \\}$ defined by $\\alpha_i = \\frac{2(2n+i)}{(4n+i)(i+1)}$ for $i \\ge 0$ and define the series $\\{\\bbeta^i\\}$ by $\\bbeta^{-1} =0$ and $\\bbeta^i = (1-\\alpha_i)\\bbeta^{i-1} + \\alpha_i \\tbeta^i$ for $i\\ge 0$.  Also define $\\gamma_i = 2n+i$ for $i \\ge 0$.  Then\n$$ \\bbeta^k = \\frac{\\sum_{i=0}^k \\gamma_i \\tilde{\\beta}^i}{\\sum_{i=0}^k \\gamma_i} \\ \\ \\mbox{for~all~} k \\ge 0 \\ . $$\n\\end{prop}\n\n\\noindent {\\bf Proof:} The proof follows easily by induction and using $\\sum_{i=0}^k \\gamma_i = \\frac{(4n+k)(k+1)}{2}$. \\qed\n\n\n{\\bf Proof of Theorem \\ref{thm:non-strong}.}  Denote $g^i:=\\frac{1}{n} \\left( X\\tbeta^i-\\nabla L^*(w^i)\\right)$, whereby $g^i$ is a subgradient of $D(w)$ at $w^i$, and $\\tg^i$ is an unbiased estimator of $g^i$ up to the scalar $n$, namely $\\EE_{j_i} [ \\tg^i ]=\\frac{1}{n}g^i$.  Therefore we have for any $i$ and any $w \\in \\cal W$ that:\n\\begin{equation}\\label{eq:nonsmooth-theonehand}\n\\begin{array}{ll}\n\\left\\langle -g^i,w-w^{i}\\right\\rangle  & =n \\EE_{j_i} \\left[\\left\\langle -\\tg^i,w-w^{i}\\right\\rangle\\right] \\\\ \\\\\n & \\ge n \\EE_{j_i} \\left[\\left\\langle -\\tg^i,w^{i+1}-w^{i}\\right\\rangle +\\frac{1}{\\eta_{i}}D_{h}(w^{i+1},w^{i})+\\frac{1}{\\eta_{i}}D_{h}(w,w^{i+1})-\\frac{1}{\\eta_{i}}D_{h}(w,w^{i})\\right]\\\\ \\\\\n & \\ge n \\EE_{j_i} \\left[\\left\\langle -\\tg^i,w^{i+1}-w^{i}\\right\\rangle +\\frac{1}{2n\\gamma\\eta_{i}}\\|w^{i+1}-w^{i}\\|_{2}^{2}+\\frac{1}{\\eta_{i}}D_{h}(w,w^{i+1})-\\frac{1}{\\eta_{i}}D_{h}(w,w^{i})\\right]\\\\ \\\\\n  & \\ge n \\EE_{j_i} \\left[ -\\frac{1}{2}n\\gamma \\eta_{i} \\|\\tg^i\\|_2^2+\\frac{1}{\\eta_{i}}D_{h}(w,w^{i+1})-\\frac{1}{\\eta_{i}}D_{h}(w,w^{i})\\right]\\\\ \\\\\n & \\ge -2\\gamma M^{2}\\eta_{i}+\\frac{n}{\\eta_{i}}\\EE_{j_i} [D_{h}(w,w^{i+1})]-\\frac{n}{\\eta_{i}}D_{h}(w,w^{i}) \\ ,\n\\end{array}\n\\end{equation}\nwhere the first inequality is from the ``three point property'' of Tseng (Lemma \\ref{tpp} in the Appendix), the second\ninequality is due to the fact that $h(w)$ is $\\frac{1}{n\\gamma}$-strongly convex with respect to the norm $\\|\\cdot\\|_2$ (Proposition \\ref{prop:strong}), and the third inequality is an application of the basic inequality $\\langle x, y\\rangle \\le \\tfrac{1}{2}\\|x\\|_2^2 +  \\tfrac{1}{2}\\|y\\|_2^2$,  and the last inequality uses Corollary \\ref{cor:bound}.\n\nOn the other hand, we have from Proposition \\ref{prop:equality} that\n% by substituting $\\phi(\\tbeta^{i},w)=\\frac{1}{n} \\left(w^{T}X\\tbeta^{i}-\\sum l_{i}^{*}(w_{i})\\right) + R(\\tbeta^i)$  we have\n\\begin{equation}\\label{eq:nonsmooth-theotherhand}\n\\left\\langle -g^i,w-w^{i}\\right\\rangle   =\\left\\langle -\\nabla_{w}\\phi(\\tbeta^{i},w^{i}),w-w^{i}\\right\\rangle  =D(w^{i})-\\phi(\\tbeta^{i},w)-D_{h}(w,w^{i}) \\ .\n\\end{equation}\n\n\nCombining \\eqref{eq:nonsmooth-theonehand} and \\eqref{eq:nonsmooth-theotherhand} and rearranging yields\n$$\n-(\\phi(\\tbeta^{i},w)-D(w^{i}))\\ge-2\\gamma M^{2}\\eta_{i}+\\tfrac{n}{\\eta_{i}}\\EE_{j_i}[D_{h}(w,w^{i+1})]-\\left(\\tfrac{n}{\\eta_{i}}-1\\right)D_{h}(w,w^{i}).\n$$\nSubstituting $\\eta_{i}=\\frac{2n}{2n+i+1}$ and multiplying by $2n+i$ results, we arrive at the following inequality after rearranging terms:\n$$\\begin{array}{l}\n(2n+i)(\\phi(\\tbeta^{i},w)-D(w^{i}))\\\\\n\\le  \\ \\ 4n\\gamma  M^{2}\\left(\\tfrac{2n+i}{2n+i+1}\\right)+\\tfrac{1}{2}\\left((2n+i)(2n+i-1)D_{h}(w,w^{i})-(2n+i)(2n+i+1)\\EE_{j_i}[ D_{h}(w,w^{i+1})] \\right) \\ .\n\\end{array}$$\nSumming the above inequality for $i=0,\\ldots,k$ and recalling from Proposition \\ref{robfriday} that $\\bar{\\beta}^{k}:=\\frac{2}{(4n+k)(k+1)}\\sum_{i=0}^{k}(2n+i)\\tbeta^{i}$,\nand taking the unconditional expectation over $\\xi_k$ (recall the definition of $\\xi_k$ in \\eqref{eq:xi}), we arrive at:\n$$\\begin{array}{rcl}\n\\frac{(4n+k)(k+1)}{2}\\EE_{\\xi_k}[\\phi(\\bar{\\beta}^{k},w)-D(\\bar{w}^{k})] & = & \\left(\\sum_{i=0}^k 2n+i \\right)\\EE_{\\xi_k}[\\phi(\\bar{\\beta}^{k},w)-D(\\bar{w}^{k})] \\\\ \\\\\n& \\le & \\EE_{\\xi_k}\\left[\\sum_{i=0}^{k}(2n+i)(\\phi(\\tbeta^{i},w)-D(w^{i}))\\right]\\\\ \\\\\n& \\le & 4(k+1)n\\gamma M^{2}+\\tfrac{1}{2}(2n)(2n-1)D_{h}(w,w^{0})\\\\ \\\\\n& \\le & 4(k+1)n\\gamma M^{2}+n(2n-1)D_{\\max} \\ ,\n\\end{array}$$\nwhere the first inequality uses the convexity of $\\phi(\\beta,w)$\nover $\\beta$ and the concavity of $D(w)$, the second inequality\nfollows from the summation and canceling terms in the telescoping series, and the third inequality uses $w \\in \\cal W$.  Choosing $\\hw=\\arg\\max_{w}\\phi(\\bar{\\beta}^{k},w)$,\nwe have $P(\\bar{\\beta}^{k})=\\phi(\\bar{\\beta}^{k},\\hw)$, which yields:\n\\[\n\\EE_{\\xi_k}[P(\\bar{\\beta}^{k})-D(\\bar{w}^{k})]\\le\\frac{8n\\gamma M^{2}}{\\left(4n+k\\right)}+\\frac{2n(2n-1)D_{\\max}}{(4n+k)(k+1)} \\ ,\n\\]thus showing the first inequality in the statement of the theorem.\nThe second inequality in the statement of the theorem then follows as a simple application of Proposition \\ref{prop:Dmax}. \\qed\n","units":[{"start":54185,"end":54432,"start_line":457,"end_line":458,"raw_tex":"\\begin{lem}\\label{lem:equi} {\\bf (Equivalence Lemma)}  Algorithm \\ref{al:SFW} and Algorithm \\ref{al:RCMD} are equivalent as follows:  the iterate sequence of either algorithm exactly corresponds to an iterate sequences of the other. \\qed\n\\end{lem}","sha256_utf8":"1bce4ce67a977a9603e8cd148fa56e965152d4ee74952562e27e9fa2c4357fe2"},{"start":54778,"end":55132,"start_line":469,"end_line":474,"raw_tex":"\\begin{prop}\\label{prop:wupdate} At iteration $i$ of Algorithm \\ref{al:RCMD} it holds that:\n\n(1.) $\\dlsji(w_{j_{i}}^{i+1})=(1-\\eta_{i})\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i}) + \\eta_{i}x_{j_{i}}^{T}\\tbeta^{i}$, and\n\n(2.) $w_{j_{i}}^{i+1}=\\dot{l}_{j_{i}}\\left((1-\\eta_{i})\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})+ \\eta_{i}x_{j_{i}}^{T}\\tbeta^{i}\\right) $.\n\\end{prop}","sha256_utf8":"c68efcab92705fbd4a5f4a00602677f627c69962610e3c25fbfc5dbe28cef15a"},{"start":55133,"end":56271,"start_line":475,"end_line":486,"raw_tex":"{\\bf Proof:}\nBecause $h(w)$ is a coordinate-wise separable function, we can rewrite the update for $w_{j_{i}}^{i+1}$ as\n\\begin{equation*}\n\\begin{array}{ll}\nw_{j_{i}}^{i+1} & =\\arg\\min_{w_{j_{i}}} \\left\\langle -\\frac{\\eta_{i}}{n}\\left(x_{j_{i}}^{T}\\tbeta^{i}-\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})\\right),w_{j_{i}}\\right\\rangle +D_{\\frac{1}{n} l_{j_{i}}^{*}}(w_{j_{i}},w_{j_{i}}^{i})\\\\ \\\\\n\n& =\\arg\\min_{w_{j_{i}}}\\left\\langle -\\eta_{i}\\left(x_{j_{i}}^{T}\\tbeta^{i}-\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})\\right),w_{j_{i}}\\right\\rangle +D_{ l_{j_{i}}^{*}}(w_{j_{i}},w_{j_{i}}^{i})\\\\ \\\\\n\n& =\\arg\\min_{w_{j_{i}}}\\left\\langle -\\eta_{i}x_{j_{i}}^{T}\\tbeta^{i}-(1-\\eta_{i})\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i}),w_{j_{i}}\\right\\rangle +l_{j_{i}}^{*}(w_{j_{i}}) \\ .  \\end{array}\n \\end{equation*}\n\nFrom the first-order optimality condition of the above $1$-dimensional problem we have $\\dls_{j_i}(w_{j_i}^{i+1})=\\eta_{i}x_{j_{i}}^{T}\\tbeta^{i}+(1-\\eta_{i})\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})$, which shows {\\it (1.)}; and {\\it (2.)} follows directly from {\\it (1.)} by the properties of the conjugate function in Proposition \\ref{prop:gradient_conjugate}. \\qed\n","sha256_utf8":"3487083c91552224d90ab8c232a9040e2bad87ad47631a0e360ed6098a742f53"},{"start":56824,"end":59419,"start_line":498,"end_line":509,"raw_tex":"{\\bf Proof of Lemma \\ref{lem:equi}}  We show that the iterate sequence of Algorithm \\ref{al:RCMD} corresponds exactly to an iterate sequence of Algorithm \\ref{al:SFW}.  The $\\{s^i\\}$ sequence is not formally defined in Algorithm \\ref{al:RCMD}, so let us define $s^i := \\nabla L^*(w^i)$ for all $i = 0, 1, \\ldots$, which is consistent through conjugacy with the relationship $w^i = \\nabla L(s^i)$ in the Optional Accounting step of Algorithm \\ref{al:SFW} (see Proposition \\ref{prop:gradient_conjugate}).  In order to show the correspondence we proceed by induction on the iteration counter $i$.  For $i=0$ we have from conjugacy that $s^0 := \\nabla L^*(w^0) = 0 $ from the definition $w^0$ in the  initialization of Algorithm \\ref{al:RCMD}.  We also need to show that $\\tbeta^0$ is a solution to the linear optimization oracle problem in Algorithm \\ref{al:SFW}.  We have for all $i=0, \\ldots$, that:\n$$\\begin{array}{rcl}\n\\tbeta^{i} \\in \\arg\\min_\\beta\\left\\{ \\tfrac{1}{n}\\left(w^{i}\\right)^{T}X\\beta+R(\\beta)\\right\\}    &=& \\arg\\min_\\beta\\left\\{ \\tfrac{1}{n}\\left(\\nabla L(s^{i})\\right)^{T}X\\beta+R(\\beta)\\right\\} \\\\ \\\\\n&=& \\arg\\min_\\beta\\left\\{ \\left(d^{i}\\right)^{T}\\beta+R(\\beta)\\right\\} \\ ,\n\\end{array}$$thus showing that $\\beta^{i}$ corresponds to a linear optimization oracle solution at iteration $i$ in Algorithm \\ref{al:SFW} for all $i=0, \\ldots$.  Now suppose that the correspondence holds for some iteration counter $i$, and let us examine $s^{i+1}:= \\nabla L^*(w^{i+1})$.  We have from Proposition \\ref{prop:wupdate} that:\n\\begin{equation}\\label{robo3}\nw_{j_{i}}^{i+1} =\\dot{l}_{j_{i}}\\left((1-\\eta_{i})\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i}) + \\eta_{i}x_{j_{i}}^{T}\\tbeta^{i} \\right)=\\dot{l}_{j_{i}}\\left((1-\\eta_{i})s_{j_{i}}^{i} + \\eta_{i}x_{j_{i}}^{T}\\tbeta^{i} \\right) \\ ,\n\\end{equation}\nwhere the first equality is from Proposition \\ref{prop:wupdate} and the second equality uses induction.  This then implies that $$ (1-\\eta_{i})s_{j_{i}}^{i} + \\eta_{i}x_{j_{i}}^{T}\\tbeta^{i} = \\dl (w_{j_{i}}^{i+1}) = s^{i+1} \\ . $$\nAnd for all coefficient indices $j \\ne i $ we have\n$$s_j^{i+1} = \\dl^*(w^{i+1}_j) = \\dl(w^i_j)= s^i \\ , $$\nwhere the second equality follows from conjugacy, whereby $s^{i+1}$ satisfies the update rule as stated in Algorithm \\ref{al:SFW}, thus demonstrating that the iterate sequence of Algorithm \\ref{al:RCMD} corresponds exactly to an iterate sequence of Algorithm \\ref{al:SFW}.  The same type of analysis as above can be used to prove that the iterate sequence of Algorithm \\ref{al:SFW} corresponds exactly to an iterate sequence of Algorithm \\ref{al:RCMD}. \\qed\n","sha256_utf8":"ec3866e7fcfac7020fa2aa425b547ea63bc3b638e75faf07e82bad5c7e18c77e"},{"start":60955,"end":61071,"start_line":536,"end_line":538,"raw_tex":"\\begin{prop}\\label{prop:Dmax}\n\tUnder Assumption \\ref{ass:smooth} it holds that $D_{\\max} \\le \\gamma M^2$.\n\\end{prop}","sha256_utf8":"cb2c3fc85894df0dc4dfe5bca65ab2a5011c470a42d996f2d6af2a8a0ed4befd"},{"start":61198,"end":61554,"start_line":542,"end_line":548,"raw_tex":"\\begin{prop}\\label{prop:well-behavior-of-w}\n\tLet $T:=\\left\\{ w \\in \\RR^n : |\\|w-w^{0}\\|_{\\infty}\\le \\gamma M\\right\\}$.  Then:\n\t\\begin{enumerate}\n\t\t\\item for any $\\hbeta\\in \\dom R(\\cdot)$ it holds that $\\argmax_w \\phi (\\hbeta, w) \\in T$, and\n\t\t\\item for all $w^i$ generated in Algorithm \\ref{al:RCMD}, it holds that  $w^i \\in T$.\n\t\\end{enumerate}\n\\end{prop}","sha256_utf8":"7dda1e71eb9ce97fdc3888bcd34a9ec2c5962d438764905625659647623f3972"},{"start":61556,"end":62587,"start_line":550,"end_line":561,"raw_tex":"\\textbf{Proof.} We first prove {\\it (1.)}.  Notice that $w_{0}= \\nabla L(0)$ and $\\argmax_w \\phi (\\hbeta, w) = \\nabla L(X\\hbeta)$ (from conjugacy via Proposition \\ref{prop:gradient_conjugate}) , whereby the $\\gamma$-smoothness of $l_{j}(\\cdot)$ implies that\n$$\n\\left\\|\\arg\\max_{w}\\phi(\\hat{\\beta},w)-w_{0}\\right\\|_{\\infty}=\\left\\| \\nabla L(X\\hbeta)-\\nabla L(0)\\right\\|_{\\infty}=\\max_j \\left|\\dl_j(x_j^T\\hat \\beta)-\\dl_j(0)\\right|\\le\\gamma\\max_{j}|x_{j}^{T}\\hat \\beta| \\le \\gamma M \\ ,\n$$\n\n\nwhich proves {\\it (1.)}. It follows from Proposition \\ref{cor:bound} that for any coordinate $j$ and iterate $i$ it holds that $\\left|\\dot{l}_{j}^{*}(w_{j}^{i})\\right|\\le M$. Together with $\\dls_j(w_j^0)=0$, we have\n\n$$\n\\tfrac{1}{\\gamma}\\left|w_{j}^{i}-w_{j}^{0}\\right|\\le \\left|\\dot{l}_{j}^{*}(w_{j}^{i})-\\dot{l}_{j}^{*}(w_{j}^{0}) \\right|\\le M \\ ,\n$$\n(where the first inequality is from the $\\frac{1}{\\gamma}$-strong convexity of $l_j^*(w_j)$), from which it follows that $\\|w^i - w^0\\|_\\infty \\le \\gamma M$, which proves {\\it (2.)}. \\qed\n","sha256_utf8":"f8f8f7dbe8a1ba76f65eb34a4459cf4ff6ff0a6fcdfba7cbbca2911712336781"},{"start":62588,"end":64159,"start_line":563,"end_line":579,"raw_tex":"{\\bf Proof of Proposition \\ref{prop:Dmax}:}  Let $L(s):=\\sum_{j=1}^n l_j(s_j)$ and $L^*(w):=\\sum_{j=1}^n l^*_j(w_j)$, and note that $L(\\cdot)$ and $L^*(\\cdot)$ are a conjugate pair.  Let $\\hw \\in \\cal W$ and let $\\hbeta$ be such that $\\hw \\in \\arg\\max_w \\phi(\\hbeta, w)$.  Then\n\n%It follows Proposition \\ref{prop:well-behavior-of-w}.1 that $\\hw^k\\in T$, therefore for any $k$ we have\n\n\\begin{equation*}\n\\begin{array}{lcl}\nD_{h}(\\hw, w^0) & = &\\tfrac{1}{n}\\left( L^*(\\hw)-L^*(w^0)\\right)\\\\ \\\\\n& =&\\tfrac{1}{n}\\left( \\left(\\hw \\right)^{T}X\\hbeta-L(X\\hbeta) -L^{*}(w^{0})\\right)\\\\ \\\\\n& \\le&\\tfrac{1}{n}\\left(\\max_{w\\in T,\\beta\\in \\dom R(\\cdot)}\\left\\{ w^{T}X\\beta-L(X\\beta)\\right\\} -L^{*}(w^{0})\\right)\\\\ \\\\\n& =&\\tfrac{1}{n}\\left(\\max_{w\\in T,\\beta\\in  \\dom R(\\cdot)}\\left\\{ (w-w^{0})^{T}X\\beta+\\left(w^{0}\\right)^{T}X\\beta-L(X\\beta)\\right\\} -L^{*}(w^{0})\\right)\\\\ \\\\\n& \\le&\\tfrac{1}{n}\\left(\\max_{w\\in T,\\beta\\in  \\dom R(\\cdot)}\\left\\{ (w-w^{0})^{T}X\\beta\\right\\} +\\max_{\\beta\\in  \\dom R(\\cdot)}\\left\\{ \\left(w^{0}\\right)^{T}X\\beta-L(X\\beta)\\right\\} -L^{*}(w^{0})\\right)\\\\ \\\\\n& \\le &\\tfrac{1}{n}\\left(n\\max_{w\\in T,\\beta\\in  \\dom R(\\cdot)}\\|w-w^{0}\\|_{\\infty}\\|X\\beta\\|_{\\infty}+L^{*}(w^{0})-L^{*}(w^{0})\\right)\\\\ \\\\\n& \\le &\\gamma M^{2} \\ ,\n\\end{array}\n\\end{equation*}\n\nwhere the second equality follows from Proposition \\ref{prop:gradient_conjugate}, the first inequality uses $\\hbeta \\in  \\dom R(\\cdot)$ and $\\hw \\in T$ (from Proposition \\ref{prop:well-behavior-of-w}), and the last inequality uses $\\max_{\\beta\\in  \\dom R(\\cdot)}\\|X\\beta\\|_{\\infty}\\le M$. \\qed \\medskip\n","sha256_utf8":"4826b12fce77c261c16b0ae947fefce7ffafd6640a46d70a29e65a58bf85f410"},{"start":67909,"end":68057,"start_line":607,"end_line":609,"raw_tex":"\\begin{prop}\nFor all iterates $i$ and any $j\\in\\left\\{ 1,\\ldots,n\\right\\} $ it holds that $\\left|\\dot{l}_{j}^{*}(w_{j}^{i})\\right|\\le M$.\n\\end{prop}","sha256_utf8":"eb508c07d3bd3b6394710f28ac7aa0549171ac258f0f4d46ed7551d5bfd5036a"},{"start":68059,"end":68900,"start_line":611,"end_line":615,"raw_tex":"{\\bf Proof.} We prove this by induction on $i$. The proposition is true for $i=0$ because $\\dot{l}_{j}^{*}(w_{j}^{0})=0$ for all $j$ by the definition of $w^0$. Next suppose that $\\left|\\dot{l}_{j}^{*}(w_{j}^{i})\\right|\\le M$ for a given iterate $i$ and for all $j=1, \\ldots, n$. Then at iteration $i+1$ and any $j\\not=j_{i}$ we have  $w_j^{i+1}=w_j^i$, whereby $\\left|\\dot{l}_{j}^{*}(w_{j}^{i+1})\\right|=\\left|\\dot{l}_{j}^{*}(w_{j}^{i})\\right|\\le M$.  And it follows from Proposition \\ref{prop:wupdate} that\n\\[\n\\left|\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i+1})\\right|=\\left|\\left(1-\\eta_{i}\\right)\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})+\\eta_{i}x_{j_{i}}^T\\tbeta^{i}\\right|\\le(1-\\eta_{i})M+\\eta_{i}M=M \\ ,\n\\]\nand therefore for any $j=1, \\ldots, n$, we have $\\left|\\dot{l}_{j}^{*}(w_{j}^{i+1})\\right|\\le M$, which completes the proof by induction. \\qed\n","sha256_utf8":"057b3252e1798d3f5e10432f977b3ded73022d146c0ccf9a9498807b482f33af"},{"start":68994,"end":69136,"start_line":619,"end_line":621,"raw_tex":"\\begin{cor}\\label{cor:bound}\n$\\|\\tilde{g}^i\\|_{2} =\\frac{1}{n}\\left| x_{j_{i}}^T \\tbeta^i - \\dlsji(\\wiji) \\right| \\le \\frac{2M}{n}$.\n\\end{cor}","sha256_utf8":"a6c7a9e72bdfc2dea0c278cabac14bac7b838c2a94bab36c954424f943da7ac9"},{"start":69146,"end":69280,"start_line":623,"end_line":626,"raw_tex":"\\begin{prop}\\label{prop:strong}\n$h(\\cdot)$ is\n$\\frac{1}{n\\gamma}$-strongly convex with respect to the norm $\\|\\cdot\\|_{2}$.\n\\end{prop}","sha256_utf8":"a9befa0fd3004b3cf720aab1605664f468c91d0d35d65988a85201a4a1f5e886"},{"start":69281,"end":69927,"start_line":627,"end_line":634,"raw_tex":"{\\bf Proof.} Recall that $h(w)=\\frac{1}{n}\\sum_{j=1}^{n}l_{j}^{*}(w_{j})$. It follows from Assumption \\ref{ass:smooth} and Proposition \\ref{prop:gradient_conjugate} that $\\dls_j(\\cdot)$ is $\\frac{1}{\\gamma}$-strongly convex. Therefore for any $w^1, w^2 \\in \\dom h(\\cdot)$ it holds that:\n\\begin{equation*}\n\\begin{array}{lcl}\nh(w^1)& = &\\frac{1}{n}\\sum_{j=1}^{n}l_{j}^{*}(w^1_{j}) \\\\ \\\\\n& \\ge & \\frac{1}{n}\\sum_{j=1}^{n}\\left( l_{j}^{*}(w^2_{j}) + \\dls_j(w_j^2)(w_j^1-w_j^2)+\\frac{1}{2\\gamma}|w^2_j-w^1_j|^2\\right) \\\\ \\\\\n& = & h(w^2) + \\langle\\nabla h(w^2), w^1-w^2\\rangle + \\frac{1}{2n\\gamma}\\|w^2-w^1\\|_2^2 \\ . \\ \\qed\n\\end{array}\n\\end{equation*}\n","sha256_utf8":"de0ea31bb64d06345e38f0ec89688d7cd54ae7898676e4bf9ce3b2c05ad534d5"},{"start":69938,"end":70096,"start_line":638,"end_line":640,"raw_tex":"\\begin{prop}\\label{prop:equality}\n$\\phi(\\tbeta^{i},w)=D(w^{i})+\\left\\langle \\nabla_{w}\\phi(\\tbeta^{i},w^{i}),w-w^{i}\\right\\rangle -D_{h}(w,w^{i}).$\n\\end{prop}","sha256_utf8":"6d8a4dbee09e5a25bde3754eed56b776650c82541a317dd812d79b6d1281f201"},{"start":70097,"end":70376,"start_line":641,"end_line":641,"raw_tex":"\\textbf{Proof. } The proof follows from straightforward substitution using $\\phi(\\tbeta^{i},w)=\\frac{1}{n} \\left(w^{T}X\\tbeta^{i}-\\sum_{j=1}^n l_{j}^{*}(w_{j})\\right) + R(\\tbeta^i)$ and noticing from the construction of $\\tbeta^i$ that $D(w^i)=\\phi(\\tbeta^i, w^i)$. \\qed\\medskip\n","sha256_utf8":"d16fd0171e99b56649d5c99ac85cf9176f9acaad006506f270622735dfa0ae40"},{"start":70378,"end":70824,"start_line":644,"end_line":646,"raw_tex":"\\begin{prop}\\label{robfriday}  Consider the series $\\{ \\alpha_i \\}$ defined by $\\alpha_i = \\frac{2(2n+i)}{(4n+i)(i+1)}$ for $i \\ge 0$ and define the series $\\{\\bbeta^i\\}$ by $\\bbeta^{-1} =0$ and $\\bbeta^i = (1-\\alpha_i)\\bbeta^{i-1} + \\alpha_i \\tbeta^i$ for $i\\ge 0$.  Also define $\\gamma_i = 2n+i$ for $i \\ge 0$.  Then\n$$ \\bbeta^k = \\frac{\\sum_{i=0}^k \\gamma_i \\tilde{\\beta}^i}{\\sum_{i=0}^k \\gamma_i} \\ \\ \\mbox{for~all~} k \\ge 0 \\ . $$\n\\end{prop}","sha256_utf8":"76338495e852142096060705f9315e84f5b1da5251f14af1e4bb7d3e4c08dab8"},{"start":70825,"end":70950,"start_line":647,"end_line":647,"raw_tex":"\\noindent {\\bf Proof:} The proof follows easily by induction and using $\\sum_{i=0}^k \\gamma_i = \\frac{(4n+k)(k+1)}{2}$. \\qed\n","sha256_utf8":"eabfdd8fca15daa97e159e0a18eb9ff3ba08293f2f5e10720010f37ceed4302a"},{"start":70951,"end":75086,"start_line":649,"end_line":693,"raw_tex":"{\\bf Proof of Theorem \\ref{thm:non-strong}.}  Denote $g^i:=\\frac{1}{n} \\left( X\\tbeta^i-\\nabla L^*(w^i)\\right)$, whereby $g^i$ is a subgradient of $D(w)$ at $w^i$, and $\\tg^i$ is an unbiased estimator of $g^i$ up to the scalar $n$, namely $\\EE_{j_i} [ \\tg^i ]=\\frac{1}{n}g^i$.  Therefore we have for any $i$ and any $w \\in \\cal W$ that:\n\\begin{equation}\\label{eq:nonsmooth-theonehand}\n\\begin{array}{ll}\n\\left\\langle -g^i,w-w^{i}\\right\\rangle  & =n \\EE_{j_i} \\left[\\left\\langle -\\tg^i,w-w^{i}\\right\\rangle\\right] \\\\ \\\\\n & \\ge n \\EE_{j_i} \\left[\\left\\langle -\\tg^i,w^{i+1}-w^{i}\\right\\rangle +\\frac{1}{\\eta_{i}}D_{h}(w^{i+1},w^{i})+\\frac{1}{\\eta_{i}}D_{h}(w,w^{i+1})-\\frac{1}{\\eta_{i}}D_{h}(w,w^{i})\\right]\\\\ \\\\\n & \\ge n \\EE_{j_i} \\left[\\left\\langle -\\tg^i,w^{i+1}-w^{i}\\right\\rangle +\\frac{1}{2n\\gamma\\eta_{i}}\\|w^{i+1}-w^{i}\\|_{2}^{2}+\\frac{1}{\\eta_{i}}D_{h}(w,w^{i+1})-\\frac{1}{\\eta_{i}}D_{h}(w,w^{i})\\right]\\\\ \\\\\n  & \\ge n \\EE_{j_i} \\left[ -\\frac{1}{2}n\\gamma \\eta_{i} \\|\\tg^i\\|_2^2+\\frac{1}{\\eta_{i}}D_{h}(w,w^{i+1})-\\frac{1}{\\eta_{i}}D_{h}(w,w^{i})\\right]\\\\ \\\\\n & \\ge -2\\gamma M^{2}\\eta_{i}+\\frac{n}{\\eta_{i}}\\EE_{j_i} [D_{h}(w,w^{i+1})]-\\frac{n}{\\eta_{i}}D_{h}(w,w^{i}) \\ ,\n\\end{array}\n\\end{equation}\nwhere the first inequality is from the ``three point property'' of Tseng (Lemma \\ref{tpp} in the Appendix), the second\ninequality is due to the fact that $h(w)$ is $\\frac{1}{n\\gamma}$-strongly convex with respect to the norm $\\|\\cdot\\|_2$ (Proposition \\ref{prop:strong}), and the third inequality is an application of the basic inequality $\\langle x, y\\rangle \\le \\tfrac{1}{2}\\|x\\|_2^2 +  \\tfrac{1}{2}\\|y\\|_2^2$,  and the last inequality uses Corollary \\ref{cor:bound}.\n\nOn the other hand, we have from Proposition \\ref{prop:equality} that\n% by substituting $\\phi(\\tbeta^{i},w)=\\frac{1}{n} \\left(w^{T}X\\tbeta^{i}-\\sum l_{i}^{*}(w_{i})\\right) + R(\\tbeta^i)$  we have\n\\begin{equation}\\label{eq:nonsmooth-theotherhand}\n\\left\\langle -g^i,w-w^{i}\\right\\rangle   =\\left\\langle -\\nabla_{w}\\phi(\\tbeta^{i},w^{i}),w-w^{i}\\right\\rangle  =D(w^{i})-\\phi(\\tbeta^{i},w)-D_{h}(w,w^{i}) \\ .\n\\end{equation}\n\n\nCombining \\eqref{eq:nonsmooth-theonehand} and \\eqref{eq:nonsmooth-theotherhand} and rearranging yields\n$$\n-(\\phi(\\tbeta^{i},w)-D(w^{i}))\\ge-2\\gamma M^{2}\\eta_{i}+\\tfrac{n}{\\eta_{i}}\\EE_{j_i}[D_{h}(w,w^{i+1})]-\\left(\\tfrac{n}{\\eta_{i}}-1\\right)D_{h}(w,w^{i}).\n$$\nSubstituting $\\eta_{i}=\\frac{2n}{2n+i+1}$ and multiplying by $2n+i$ results, we arrive at the following inequality after rearranging terms:\n$$\\begin{array}{l}\n(2n+i)(\\phi(\\tbeta^{i},w)-D(w^{i}))\\\\\n\\le  \\ \\ 4n\\gamma  M^{2}\\left(\\tfrac{2n+i}{2n+i+1}\\right)+\\tfrac{1}{2}\\left((2n+i)(2n+i-1)D_{h}(w,w^{i})-(2n+i)(2n+i+1)\\EE_{j_i}[ D_{h}(w,w^{i+1})] \\right) \\ .\n\\end{array}$$\nSumming the above inequality for $i=0,\\ldots,k$ and recalling from Proposition \\ref{robfriday} that $\\bar{\\beta}^{k}:=\\frac{2}{(4n+k)(k+1)}\\sum_{i=0}^{k}(2n+i)\\tbeta^{i}$,\nand taking the unconditional expectation over $\\xi_k$ (recall the definition of $\\xi_k$ in \\eqref{eq:xi}), we arrive at:\n$$\\begin{array}{rcl}\n\\frac{(4n+k)(k+1)}{2}\\EE_{\\xi_k}[\\phi(\\bar{\\beta}^{k},w)-D(\\bar{w}^{k})] & = & \\left(\\sum_{i=0}^k 2n+i \\right)\\EE_{\\xi_k}[\\phi(\\bar{\\beta}^{k},w)-D(\\bar{w}^{k})] \\\\ \\\\\n& \\le & \\EE_{\\xi_k}\\left[\\sum_{i=0}^{k}(2n+i)(\\phi(\\tbeta^{i},w)-D(w^{i}))\\right]\\\\ \\\\\n& \\le & 4(k+1)n\\gamma M^{2}+\\tfrac{1}{2}(2n)(2n-1)D_{h}(w,w^{0})\\\\ \\\\\n& \\le & 4(k+1)n\\gamma M^{2}+n(2n-1)D_{\\max} \\ ,\n\\end{array}$$\nwhere the first inequality uses the convexity of $\\phi(\\beta,w)$\nover $\\beta$ and the concavity of $D(w)$, the second inequality\nfollows from the summation and canceling terms in the telescoping series, and the third inequality uses $w \\in \\cal W$.  Choosing $\\hw=\\arg\\max_{w}\\phi(\\bar{\\beta}^{k},w)$,\nwe have $P(\\bar{\\beta}^{k})=\\phi(\\bar{\\beta}^{k},\\hw)$, which yields:\n\\[\n\\EE_{\\xi_k}[P(\\bar{\\beta}^{k})-D(\\bar{w}^{k})]\\le\\frac{8n\\gamma M^{2}}{\\left(4n+k\\right)}+\\frac{2n(2n-1)D_{\\max}}{(4n+k)(k+1)} \\ ,\n\\]thus showing the first inequality in the statement of the theorem.\nThe second inequality in the statement of the theorem then follows as a simple application of Proposition \\ref{prop:Dmax}. \\qed\n","sha256_utf8":"d2dac9f32ea2cd6cae48607e8ee5194ccdebf33642516afde08d380c6fc44af9"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"随机算法的次线性对偶间隙界","source_units":[{"start":65413,"end":66110,"start_line":592,"end_line":598,"raw_tex":"\\begin{thm}\\label{thm:non-strong}\nConsider the Stochastic Generalized Frank-Wolfe method (Algorithm \\ref{al:SFW}) or the Randomized Dual Coordinate Mirror Descent method (Algorithm \\ref{al:RCMD}), with step-size sequences $\\alpha_i=\\frac{2(2n+i)}{(i+1)(4n+i)}$ and $\\eta_i=\\frac{2n}{2n+i+1}$ for $i=0, 1, \\ldots$.  Denote $$\\bw^{k} = \\frac{2}{(4n+k)(k+1)}\\sum_{i=0}^k(2n+i) w^i \\ . $$\nUnder Assumption \\ref{ass:smooth}, it holds for all $k \\ge 0$ that\n$$\n\\EE_{\\xi_k}\\left[ P(\\bbeta^k) - D(\\bw^k)\\right] \\le \\frac{8n\\gamma M^{2}}{\\left(4n+k\\right)}+\\frac{2n(2n-1)D_{\\max}}{(4n+k)(k+1)}  \\ \\le \\ \\frac{8n\\gamma M^{2}}{\\left(4n+k\\right)}+\\frac{2n(2n-1)\\gamma M^2}{(4n+k)(k+1)} \\ . \\ \\qed\n$$\n\\end{thm}","sha256_utf8":"f52435a81130247c0080406b3d9aad1931ac0f648263d566d406db94a3d8be16"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"主收敛定理完整证明","source_units":[{"start":70951,"end":75086,"start_line":649,"end_line":693,"raw_tex":"{\\bf Proof of Theorem \\ref{thm:non-strong}.}  Denote $g^i:=\\frac{1}{n} \\left( X\\tbeta^i-\\nabla L^*(w^i)\\right)$, whereby $g^i$ is a subgradient of $D(w)$ at $w^i$, and $\\tg^i$ is an unbiased estimator of $g^i$ up to the scalar $n$, namely $\\EE_{j_i} [ \\tg^i ]=\\frac{1}{n}g^i$.  Therefore we have for any $i$ and any $w \\in \\cal W$ that:\n\\begin{equation}\\label{eq:nonsmooth-theonehand}\n\\begin{array}{ll}\n\\left\\langle -g^i,w-w^{i}\\right\\rangle  & =n \\EE_{j_i} \\left[\\left\\langle -\\tg^i,w-w^{i}\\right\\rangle\\right] \\\\ \\\\\n & \\ge n \\EE_{j_i} \\left[\\left\\langle -\\tg^i,w^{i+1}-w^{i}\\right\\rangle +\\frac{1}{\\eta_{i}}D_{h}(w^{i+1},w^{i})+\\frac{1}{\\eta_{i}}D_{h}(w,w^{i+1})-\\frac{1}{\\eta_{i}}D_{h}(w,w^{i})\\right]\\\\ \\\\\n & \\ge n \\EE_{j_i} \\left[\\left\\langle -\\tg^i,w^{i+1}-w^{i}\\right\\rangle +\\frac{1}{2n\\gamma\\eta_{i}}\\|w^{i+1}-w^{i}\\|_{2}^{2}+\\frac{1}{\\eta_{i}}D_{h}(w,w^{i+1})-\\frac{1}{\\eta_{i}}D_{h}(w,w^{i})\\right]\\\\ \\\\\n  & \\ge n \\EE_{j_i} \\left[ -\\frac{1}{2}n\\gamma \\eta_{i} \\|\\tg^i\\|_2^2+\\frac{1}{\\eta_{i}}D_{h}(w,w^{i+1})-\\frac{1}{\\eta_{i}}D_{h}(w,w^{i})\\right]\\\\ \\\\\n & \\ge -2\\gamma M^{2}\\eta_{i}+\\frac{n}{\\eta_{i}}\\EE_{j_i} [D_{h}(w,w^{i+1})]-\\frac{n}{\\eta_{i}}D_{h}(w,w^{i}) \\ ,\n\\end{array}\n\\end{equation}\nwhere the first inequality is from the ``three point property'' of Tseng (Lemma \\ref{tpp} in the Appendix), the second\ninequality is due to the fact that $h(w)$ is $\\frac{1}{n\\gamma}$-strongly convex with respect to the norm $\\|\\cdot\\|_2$ (Proposition \\ref{prop:strong}), and the third inequality is an application of the basic inequality $\\langle x, y\\rangle \\le \\tfrac{1}{2}\\|x\\|_2^2 +  \\tfrac{1}{2}\\|y\\|_2^2$,  and the last inequality uses Corollary \\ref{cor:bound}.\n\nOn the other hand, we have from Proposition \\ref{prop:equality} that\n% by substituting $\\phi(\\tbeta^{i},w)=\\frac{1}{n} \\left(w^{T}X\\tbeta^{i}-\\sum l_{i}^{*}(w_{i})\\right) + R(\\tbeta^i)$  we have\n\\begin{equation}\\label{eq:nonsmooth-theotherhand}\n\\left\\langle -g^i,w-w^{i}\\right\\rangle   =\\left\\langle -\\nabla_{w}\\phi(\\tbeta^{i},w^{i}),w-w^{i}\\right\\rangle  =D(w^{i})-\\phi(\\tbeta^{i},w)-D_{h}(w,w^{i}) \\ .\n\\end{equation}\n\n\nCombining \\eqref{eq:nonsmooth-theonehand} and \\eqref{eq:nonsmooth-theotherhand} and rearranging yields\n$$\n-(\\phi(\\tbeta^{i},w)-D(w^{i}))\\ge-2\\gamma M^{2}\\eta_{i}+\\tfrac{n}{\\eta_{i}}\\EE_{j_i}[D_{h}(w,w^{i+1})]-\\left(\\tfrac{n}{\\eta_{i}}-1\\right)D_{h}(w,w^{i}).\n$$\nSubstituting $\\eta_{i}=\\frac{2n}{2n+i+1}$ and multiplying by $2n+i$ results, we arrive at the following inequality after rearranging terms:\n$$\\begin{array}{l}\n(2n+i)(\\phi(\\tbeta^{i},w)-D(w^{i}))\\\\\n\\le  \\ \\ 4n\\gamma  M^{2}\\left(\\tfrac{2n+i}{2n+i+1}\\right)+\\tfrac{1}{2}\\left((2n+i)(2n+i-1)D_{h}(w,w^{i})-(2n+i)(2n+i+1)\\EE_{j_i}[ D_{h}(w,w^{i+1})] \\right) \\ .\n\\end{array}$$\nSumming the above inequality for $i=0,\\ldots,k$ and recalling from Proposition \\ref{robfriday} that $\\bar{\\beta}^{k}:=\\frac{2}{(4n+k)(k+1)}\\sum_{i=0}^{k}(2n+i)\\tbeta^{i}$,\nand taking the unconditional expectation over $\\xi_k$ (recall the definition of $\\xi_k$ in \\eqref{eq:xi}), we arrive at:\n$$\\begin{array}{rcl}\n\\frac{(4n+k)(k+1)}{2}\\EE_{\\xi_k}[\\phi(\\bar{\\beta}^{k},w)-D(\\bar{w}^{k})] & = & \\left(\\sum_{i=0}^k 2n+i \\right)\\EE_{\\xi_k}[\\phi(\\bar{\\beta}^{k},w)-D(\\bar{w}^{k})] \\\\ \\\\\n& \\le & \\EE_{\\xi_k}\\left[\\sum_{i=0}^{k}(2n+i)(\\phi(\\tbeta^{i},w)-D(w^{i}))\\right]\\\\ \\\\\n& \\le & 4(k+1)n\\gamma M^{2}+\\tfrac{1}{2}(2n)(2n-1)D_{h}(w,w^{0})\\\\ \\\\\n& \\le & 4(k+1)n\\gamma M^{2}+n(2n-1)D_{\\max} \\ ,\n\\end{array}$$\nwhere the first inequality uses the convexity of $\\phi(\\beta,w)$\nover $\\beta$ and the concavity of $D(w)$, the second inequality\nfollows from the summation and canceling terms in the telescoping series, and the third inequality uses $w \\in \\cal W$.  Choosing $\\hw=\\arg\\max_{w}\\phi(\\bar{\\beta}^{k},w)$,\nwe have $P(\\bar{\\beta}^{k})=\\phi(\\bar{\\beta}^{k},\\hw)$, which yields:\n\\[\n\\EE_{\\xi_k}[P(\\bar{\\beta}^{k})-D(\\bar{w}^{k})]\\le\\frac{8n\\gamma M^{2}}{\\left(4n+k\\right)}+\\frac{2n(2n-1)D_{\\max}}{(4n+k)(k+1)} \\ ,\n\\]thus showing the first inequality in the statement of the theorem.\nThe second inequality in the statement of the theorem then follows as a simple application of Proposition \\ref{prop:Dmax}. \\qed\n","sha256_utf8":"d2dac9f32ea2cd6cae48607e8ee5194ccdebf33642516afde08d380c6fc44af9"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"线性预测经验风险问题","source_units":[{"start":6529,"end":6684,"start_line":172,"end_line":174,"raw_tex":"\\begin{equation}\\label{eq:primal}\n\\mathrm{(P):} \\ \\ \\ \\ \\ \\ \\ \\min_\\beta P(\\beta):=\\tfrac{1}{n }\\sum_{j=1}^{n}l_{j}(x_j^T\\beta)+R(\\beta) \\ ,\n\\end{equation}","sha256_utf8":"9223db6c96bace13083b24761a27f63129d7ef157a1af3b96ee669fa757ffff8"},{"start":6685,"end":7623,"start_line":175,"end_line":175,"raw_tex":"where $\\beta \\in \\RR^p$, $l_j(\\cdot) : \\RR \\rightarrow \\RR$, $j=1, \\ldots, n$, is a univariate function (the $j^{\\mathrm{th}}$ loss function), $s_j = x_j^T\\beta$ is the ``predicted value'' of the model $\\beta$ for the data sample $x_j$, and $R(\\cdot)$ is some other function that can be used to model a variety of properties such as a regularizer, an indicator function of a feasible region $Q$, a penalty term, coupling constraints, etc.  Notice that the predicted value $s_j= x_j^T\\beta$ is a linear function of $\\beta$.  Hence in the intentional context of empirical loss minimization we refer to \\eqref{eq:primal} as (empirical) loss minimization with ``linear prediction.''  This structure arises quite generally throughout statistical and machine learning, which we illustrate below via several salient examples.  Throughout this paper we assume the following regarding the functions in our problem setup \\eqref{eq:primal}:\\medskip\n","sha256_utf8":"fcbb70d02c232ecc2599cbc9f3439ca26d5175e03cbccfb1a5efdf50474055a6"}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","title":"损失光滑性与有界优化域","source_units":[{"start":7624,"end":8121,"start_line":177,"end_line":187,"raw_tex":"\\begin{ass}\\label{ass:smooth}  The following hold:\n\\begin{enumerate}\n\\item for $j=1, \\ldots, n$, the univariate function $l_j(\\cdot)$ is $\\gamma$-smooth, namely $| \\dot l_j(a) - \\dot l_j(b)| \\le \\gamma |a-b|$ for all $a$, $b$, and is strictly convex,\n\\item $\\dom R(\\cdot)$ is bounded, and the subproblem\n\\begin{equation}\\label{eq:sub}\n\\min_\\beta c^T\\beta +R(\\beta)\n\\end{equation}\nattains its optimum and can be easily solved for any $c$, and\n\\item $ 0 \\in \\dom R(\\cdot)$.\n\\end{enumerate}\n\\end{ass}","sha256_utf8":"b164ddf1a7caa4b1737992ae054b6224712c46e8f1e813e9bad530e3a01adc10"}]},{"id":"S005","semantic_type":"notation","kind":"notation","title":"数据矩阵与总损失","source_units":[{"start":13284,"end":13709,"start_line":238,"end_line":238,"raw_tex":"Let $X \\in \\RR^{n \\times p}$ denote the data matrix whose rows are comprised of the vectors $x_1, \\ldots, x_n$, i.e., the $j^{\\mathrm{th}}$ row of $X$ is the vector $x_j$, $j=1, \\ldots, n$.  Let us define $L(s): \\RR^n \\rightarrow \\RR$ by $L(s) :=\\sum_{j=1}^n l_j(s_j)$ which is the total losses associated with $s \\in \\RR^n$.  We refer to $s=X\\beta$ as the ``predicted values'' in the context of empirical loss minimization.\n","sha256_utf8":"f5526f89ef08433d6502a87e429fa85378165e8030043d6b974cadb44d3c1a3e"}]},{"id":"S006","semantic_type":"algorithm","kind":"algorithm","title":"随机广义Frank-Wolfe算法","source_units":[{"start":14042,"end":15301,"start_line":242,"end_line":263,"raw_tex":"\\begin{algorithm}\n\\caption{Stochastic Generalized Frank-Wolfe with Stochastic Substitute Gradient}\\label{al:SFW}\n$ \\ $\n\\begin{algorithmic}\n\\STATE {\\bf Initialize.}  Initialize with $\\bbeta^{-1}=0$, $s^0=0$, and substitute gradient $d^0=\\frac{1}{n}X^T\\nabla L(s^0)$, with\nstep-size sequences $\\{\\alpha_{i}\\} \\in (0,1]$ and $\\{\\eta_{i}\\} \\in (0,1]$.  \\medskip\n\n\\STATE  For iterations $i=0,1, \\ldots $\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf Solve l.o.o. subproblem:} Compute $\\tbeta^{i} \\in \\arg\\min_{\\beta}\\left\\{\\left( d^i\\right)^{T}\\beta+R(\\beta)\\right\\}$\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf Choose random index:}  Choose $j_{i} \\in {\\cal U}[1, \\ldots, n]$\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf Update $s$ value:} $s_{j_i}^{i+1}\\leftarrow(1-\\eta_{i})s_{j_i}^{i}+\\eta_{i}(x_{j_{i}}^{T}\\tbeta^{i})$,  and $s_{j}^{i+1} \\gets s_{j}^{i}$ for $j \\ne j_i$ \\medskip\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf Update substitute gradient:}  $d^{i+1}= \\frac{1}{n}X^T\\nabla L(s^{i+1}) =  d^i+\\frac{1}{n}\\left(\\dot{l}_{j_{i}}(s^{i+1}_{j_i})-\\dot{l}_{j_{i}}(s^i_{j_{i}})\\right)x_{j_{i}}$\\\\ \\medskip\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf Update primal variable:} $\\bbeta^{i} \\leftarrow (1-\\alpha_i) \\bbeta^{i-1} + \\alpha_i \\tbeta^i$.\\medskip\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf (Optional Accounting:)} $w^{i+1} \\gets \\nabla L(s^{i+1})$ \\medskip\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"fbec6559081106611b5fe7df8d3584fd6ce600f83278608ff6bc401ec567efe6"}]},{"id":"S007","semantic_type":"notation","kind":"notation","title":"Bregman距离与基本记号","source_units":[{"start":47615,"end":49131,"start_line":399,"end_line":399,"raw_tex":"\\subsection{Notation} We use $e_j$ to denote the $j^{\\mathrm{th}}$ unit coordinate vector in $\\RR^p$.  The $\\ell_p$ norm is denoted $\\|\\cdot\\|_p$.  We use $\\dl_j(\\cdot)$ to denote the first derivative of $l_j(\\cdot)$.  The Bregman distance function associated with a convex function $h(\\cdot)$ is defined as $\\Dh(y,x):= h(y) - h(x) - \\nabla h(x)^T(y-x)$.  We use $\\EE$ to denote expectation and $\\EE_{j_i}$ to denote expectation conditional on the randomly chosen index $j_i$.  For indicator functions, we use $\\bI_Q(\\cdot)$ to denote the indicator function for the set $Q$, namely $\\bI_Q(\\beta):=0$ if $\\beta \\in Q$, and $\\bI_Q(\\beta) :=+\\infty$ otherwise; and we use $\\bI_{\\{\\mathrm constraint\\}}(\\beta)$ to denote the indicator function of a particular constraint (or condition), namely $\\bI_{\\{\\mathrm constraint\\}}(\\beta):=0$ if the constraint is true at $\\beta$, and $\\bI_{\\{\\mathrm constraint\\}}(\\beta) :=+\\infty$ otherwise.  In a slight abuse of terminology we refer to the ``subgradient'' of a concave function when it is perhaps more technically accurate to refer to this as a sup-gradient.   A differentiable function $\\ff$ is $\\mu$-strongly convex with respect to a norm $\\|\\cdot\\|$ if it holds that $f(y) \\ge f(x) + \\nabla f(x)^T(y-x) + \\tfrac{\\mu}{2}\\|y-x\\|^2$ for all $x,y \\in \\dom f(\\cdot)$.  A differentiable function $\\ff$ is $\\mu$-strongly convex with respect to a reference function $h(\\cdot)$ if it holds that $f(y) \\ge f(x) + \\nabla f(x)^T(y-x) + \\mu \\Dh(y,x)$ for all $x,y \\in \\dom f(\\cdot)$.\n","sha256_utf8":"cf7e23c696c7e6a1ecee9e9a67470422ee51db372f938dc81c086c8aa30b5840"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"凸共轭函数","source_units":[{"start":49270,"end":49511,"start_line":404,"end_line":407,"raw_tex":"$$\nf^*(y) := \\sup_{x\\in \\dom f(\\cdot)} \\{y^T x - f(x)\\} \\ .\n$$\nWe will also be interested in the following dual problem of \\eqref{eq:primal} that is constructed using the conjugate functions of the component functions of  \\eqref{eq:primal}:\n","sha256_utf8":"5d2a256a86a49a2b4a6b57183151ac1f2fbb7f6a10dfe4d5774a1116c1fad518"}]},{"id":"S009","semantic_type":"problem_setting","kind":"problem_setting","title":"对偶目标及优化表示","source_units":[{"start":49511,"end":49686,"start_line":408,"end_line":410,"raw_tex":"\\begin{equation}\\label{eq:dual}\n\\mathrm{(D):} \\ \\ \\ \\ \\ \\ \\ \\max_{w} D(w):=-R^{*}\\left(-\\tfrac{1}{n}X^{T}w\\right)-\\tfrac{1}{n}\\sum_{j=1}^{n}l_{j}^{*}(w_{j}) \\ .\n\\end{equation}","sha256_utf8":"4c344d07a31677a5b11f54094e87509b1e3300f3a5c3792ea6672a8ed83852e7"},{"start":49687,"end":49868,"start_line":411,"end_line":413,"raw_tex":"Notice that we can write:\n\\begin{equation}\\label{eq:altered}R^{*}\\left(-\\tfrac{1}{n}X^{T}w\\right)= - \\min_{\\beta}\\left\\{\\tfrac{1}{n} w^{T}X\\beta+R(\\beta)\\right\\}  \\ .\n\\end{equation}","sha256_utf8":"232d5bd252d2e833c81b1ae187e8ca693225da25c5d27a0a501618d3a397d342"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"鞍点函数与极小极大表示","source_units":[{"start":49869,"end":50355,"start_line":414,"end_line":420,"raw_tex":"Also, defining the convex/concave saddle-function $\\phi(\\cdot,\\cdot)$:\n\\begin{equation}\\label{eq:primal-dual}\n\\phi(\\beta,w):=\\tfrac{1}{n} w^{T}X\\beta-\\tfrac{1}{n}\\sum_{i=1}^{n}l_{i}^{*}(w_{i})+R(\\beta) \\ ,\n\\end{equation} we can write (P) and (D) in saddlepoint minimax format as:\n\n\\begin{equation}\\label{eq:minmax} \\mathrm{(P):} \\ \\ \\min_{\\beta} \\max_{w} \\phi(\\beta,w) \\ \\ \\ \\ \\  \\ \\ \\mathrm{and} \\ \\ \\ \\ \\ \\ \\ \\  \\mathrm{(D):} \\ \\ \\max_{w}\\min_{\\beta} \\phi(\\beta,w) \\ .\n\\end{equation}\n","sha256_utf8":"f145b5a5cf42ae026f4bffc3ae5c40e51f36805663a3e590d9282138d6b0e69f"}]},{"id":"S011","semantic_type":"algorithm","kind":"algorithm","title":"随机对偶坐标镜像下降","source_units":[{"start":52495,"end":54029,"start_line":431,"end_line":453,"raw_tex":"\\begin{algorithm}\n\\caption{Randomized Coordinate Mirror Descent applied to the dual problem \\eqref{eq:dual}}\\label{al:RCMD}\n$ \\ $\n\\begin{algorithmic}\n\\STATE {\\bf Initialize.}  Define the prox function $h(w):=\\frac{1}{n}\\sum_{i=1}^{n}l_{i}^{*}(w_{i})$.  Initialize with $w^{0}=\\arg\\min_{w} \\tfrac{1}{n}\\sum_{i=1}^{n}l_{i}^{*}(w_{i})$ and\nstep-size sequences $\\{\\alpha_{i}\\} \\in (0,1]$ and $\\{\\eta_{i}\\} \\in (0,1]$. (Optional: set $\\bbeta^{-1}=0$.)\\medskip\n\n\\STATE  For iterations $i=0,1, \\ldots $\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf Compute Randomized Coordinate of Subgradient of $D(\\cdot)$ at $w^i$}\\\\\n\n\\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ Compute $\\tbeta^{i} \\in \\arg\\min_{\\beta}\\left\\{\\left( \\tfrac{1}{n}(w^i)^T X \\beta+R(\\beta) \\right) \\right\\}$ \\\\ \\medskip\n\n\\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\    {\\bf Choose random index.}  Choose $j_{i} \\in {\\cal U}[1, \\ldots, n]$ \\\\\n%Compute $\\bar{\\beta}^{i}:=\\frac{2}{(4n+k)(k+1)}\\sum_{i=0}^{i}(2n+k)\\tilde{\\beta}^{i}=\\frac{(4n+k-1)k}{(4n+k)(k+1)}\\bar{\\beta}^{k-1}+\\frac{2(2n+k)}{(4n+k)(k+1)}\\tbeta^{i}$\n\\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\  {\\bf Compute subgradient coordinate vector:} $\\tilde{g}^i \\gets \\frac{1}{n}\\left(x_{j_{i}}^{T}\\tbeta^{i}-\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})\\right)e_{j_{i}}$ \\\\ \\medskip\n\n\\ \\ \\ \\ \\ \\ \\    {\\bf Update dual variable:} Compute $w^{i+1}=\\arg\\min_w \\left\\{ \\left\\langle -\\eta_{i}\\tilde g^i,w-w^{i}\\right\\rangle +D_{h}(w,w^{i})\\right\\} $ \\\\ \\medskip\n\n\\ \\ \\ \\ \\ \\ \\  {\\bf (Optional Accounting:)} $\\bbeta^{i} \\leftarrow (1-\\alpha_i) \\bbeta^{i-1} + \\alpha_i \\tbeta^i$. \\medskip\n\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"fe94b296b697c4dee0e2ddfa6ba80f01a5502456032f30c33963c860c9d2f47e"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"预测值幅度常数","source_units":[{"start":60158,"end":60344,"start_line":526,"end_line":526,"raw_tex":"Let $M := \\max_{\\beta \\in \\dom R(\\cdot)} \\max_{j=1, \\ldots, n} \\{| x_j^T\\beta | \\}$, and note that $M < +\\infty$ since $\\dom R(\\cdot)$ is bounded by Assumption \\ref{ass:smooth}.\\medskip\n","sha256_utf8":"2cbc590f162c9d30a95e7d23a0c8cff49bd1b184e6a58533d634a2ba6ac984e2"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"最优对偶响应集合","source_units":[{"start":60345,"end":60622,"start_line":528,"end_line":528,"raw_tex":"Let ${\\cal W} \\subset \\RR^n$ be the set of ``optimal $w$ responses'' to values $\\beta \\in \\dom R(\\cdot)$ in the saddle-function $\\phi(\\beta, w)$, namely: $${\\cal W} := \\{ \\hw \\in \\RR^n : \\hw \\in \\arg \\max_w \\phi(\\hbeta, w) \\ \\mbox{for~some~} \\hbeta \\in \\dom R(\\cdot) \\} \\ , $$\n","sha256_utf8":"2df072139ad9f689020cb600c2940f62814cf8b18721a6bd40919012e920b008"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"响应集合的Bregman半径","source_units":[{"start":60623,"end":60954,"start_line":530,"end_line":534,"raw_tex":"and let $D_{\\max}$ be any upper bound on $D_h (\\hw,w^0)$ as $\\hw$ ranges over all values in ${\\cal W}$, so that\n$$\nD_h (\\hw ,w^0)\\le D_{\\max} \\ \\  \\mbox{for~all~}  \\hw \\in \\cal W \\ .\n$$\nNote at the moment that there is no guarantee that $D_{\\max} < +\\infty$, but this will be remedied below in Proposition \\ref{prop:Dmax}.\\medskip\n","sha256_utf8":"c68827cdfb5a36fa7b9e3589ce4cd0bddabc8337020b3216164dca6595ee7c24"}]},{"id":"S015","semantic_type":"notation","kind":"notation","title":"随机坐标历史","source_units":[{"start":64915,"end":65281,"start_line":585,"end_line":588,"raw_tex":"Notice in Algorithm \\ref{al:SFW} and Algorithm \\ref{al:RCMD} that $j_i$ is a random variable; and that $s^i$, $d^i$, $w^i$, etc., are random variables that depend on all previous random variable values $j_0, j_1, \\ldots, j_{i-1}$, and we denote this string of random variables by \n\\begin{equation}\\label{eq:xi}\n\\xi_i=\\{j_0, j_1, \\ldots, j_{i-1}\\} \\ .\n\\end{equation}\n","sha256_utf8":"5e415e940e6cc2c516c568a25fc5f3be7a09a3c5dfdc37ec86105329915373fb"}]},{"id":"S016","semantic_type":"proposition","kind":"proposition","title":"标准共轭性质","source_units":[{"start":103058,"end":103728,"start_line":1051,"end_line":1062,"raw_tex":"\\begin{prop}\\label{prop:gradient_conjugate}(see \\cite{avriel}, \\cite{zalinescu}, \\cite{kakade2012regularization})\nIf $f(\\cdot)$ is a closed convex function, then ${f}^{**}(\\cdot) = f(\\cdot)$.  Furthermore:\n\\begin{enumerate}\n\\item $\\ff$ is $\\gamma$-smooth with domain $\\RR^p$ with respect to the norm $\\|\\cdot\\|$ if and only if  $f^*(\\cdot)$ is $1/\\gamma$-strongly convex with respect to the (dual) norm $\\|\\cdot\\|^*$ .\n\\item If $\\ff$ is differentiable and strictly convex, then the following three conditions are equivalent:\n\\begin{enumerate}\n\\item $y = \\nabla f(x)$\n\\item $x = \\nabla f^*(y)$, and\n\\item $x^T y = f(x)+f^*(y)$ .\n\\end{enumerate}\n\\end{enumerate}\n\\end{prop}","sha256_utf8":"a0529bc7e17495c292c409c97360d528d686882ae57a3ffa1040a280a5a0f78e"}]},{"id":"S017","semantic_type":"lemma","kind":"lemma","title":"镜像下降三点性质","source_units":[{"start":103865,"end":104296,"start_line":1069,"end_line":1077,"raw_tex":"\\begin{lem}\\label{tpp} {\\bf (Three-Point Property \\cite{tseng})} Let $\\phi(x)$ be a convex function, and let $\\Dh(\\cdot, \\cdot)$ be the Bregman distance for $\\hh$. For a given vector $z$, let\n\\begin{equation*}\nz^+ := \\arg\\min_{x\\in Q} \\left\\{ \\phi(x) + \\Dh(x,z) \\right\\} \\ .\n\\end{equation*}\nThen\n\\begin{equation*}\n\\phi(x) + \\Dh(x,z) \\ge \\phi(z^+) + \\Dh(z^+, z) + \\Dh(x,z^+)\\ \\   for\\ all \\ x\\in Q \\ . \\qed\n\\end{equation*}\n\\end{lem}","sha256_utf8":"343fd72211540287e891e5c3912ced21f2b952c1a83e67bbc2df94abce3f8a95"}]},{"id":"S018","semantic_type":"external_fact","kind":"external_fact","title":"凸共轭标准定理","source_units":[],"declared_external":true,"declaration_text":["闭凸函数满足f**=f；全空间上的γ光滑凸函数的共轭关于对偶范数为1/γ强凸；在文中可微严格凸条件下，y=∇f(x)、x=∇f*(y)和xᵀy=f(x)+f*(y)等价。"]},{"id":"S019","semantic_type":"external_fact","kind":"external_fact","title":"Tseng三点不等式","source_units":[],"declared_external":true,"declaration_text":["若z+∈argmin_{x∈Q}{φ(x)+D_h(x,z)}，其中φ凸，则对所有x∈Q，有φ(x)+D_h(x,z)≥φ(z+)+D_h(z+,z)+D_h(x,z+)。"]},{"id":"S020","semantic_type":"external_fact","kind":"external_fact","title":"凸性不等式与期望法则","source_units":[],"declared_external":true,"declaration_text":["凸函数满足加权Jensen不等式，凹函数满足反向不等式；凸参考函数的Bregman距离非负；⟨x,y⟩≤(||x||₂²+||y||₂²)/2；条件期望具有线性性并满足塔式法则。"]},{"id":"S021","semantic_type":"lemma","kind":"lemma","title":"原始与对偶算法等价","source_units":[{"start":54185,"end":54432,"start_line":457,"end_line":458,"raw_tex":"\\begin{lem}\\label{lem:equi} {\\bf (Equivalence Lemma)}  Algorithm \\ref{al:SFW} and Algorithm \\ref{al:RCMD} are equivalent as follows:  the iterate sequence of either algorithm exactly corresponds to an iterate sequences of the other. \\qed\n\\end{lem}","sha256_utf8":"1bce4ce67a977a9603e8cd148fa56e965152d4ee74952562e27e9fa2c4357fe2"}]},{"id":"S022","semantic_type":"proof","kind":"proof","title":"算法等价性完整证明","source_units":[{"start":56824,"end":59419,"start_line":498,"end_line":509,"raw_tex":"{\\bf Proof of Lemma \\ref{lem:equi}}  We show that the iterate sequence of Algorithm \\ref{al:RCMD} corresponds exactly to an iterate sequence of Algorithm \\ref{al:SFW}.  The $\\{s^i\\}$ sequence is not formally defined in Algorithm \\ref{al:RCMD}, so let us define $s^i := \\nabla L^*(w^i)$ for all $i = 0, 1, \\ldots$, which is consistent through conjugacy with the relationship $w^i = \\nabla L(s^i)$ in the Optional Accounting step of Algorithm \\ref{al:SFW} (see Proposition \\ref{prop:gradient_conjugate}).  In order to show the correspondence we proceed by induction on the iteration counter $i$.  For $i=0$ we have from conjugacy that $s^0 := \\nabla L^*(w^0) = 0 $ from the definition $w^0$ in the  initialization of Algorithm \\ref{al:RCMD}.  We also need to show that $\\tbeta^0$ is a solution to the linear optimization oracle problem in Algorithm \\ref{al:SFW}.  We have for all $i=0, \\ldots$, that:\n$$\\begin{array}{rcl}\n\\tbeta^{i} \\in \\arg\\min_\\beta\\left\\{ \\tfrac{1}{n}\\left(w^{i}\\right)^{T}X\\beta+R(\\beta)\\right\\}    &=& \\arg\\min_\\beta\\left\\{ \\tfrac{1}{n}\\left(\\nabla L(s^{i})\\right)^{T}X\\beta+R(\\beta)\\right\\} \\\\ \\\\\n&=& \\arg\\min_\\beta\\left\\{ \\left(d^{i}\\right)^{T}\\beta+R(\\beta)\\right\\} \\ ,\n\\end{array}$$thus showing that $\\beta^{i}$ corresponds to a linear optimization oracle solution at iteration $i$ in Algorithm \\ref{al:SFW} for all $i=0, \\ldots$.  Now suppose that the correspondence holds for some iteration counter $i$, and let us examine $s^{i+1}:= \\nabla L^*(w^{i+1})$.  We have from Proposition \\ref{prop:wupdate} that:\n\\begin{equation}\\label{robo3}\nw_{j_{i}}^{i+1} =\\dot{l}_{j_{i}}\\left((1-\\eta_{i})\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i}) + \\eta_{i}x_{j_{i}}^{T}\\tbeta^{i} \\right)=\\dot{l}_{j_{i}}\\left((1-\\eta_{i})s_{j_{i}}^{i} + \\eta_{i}x_{j_{i}}^{T}\\tbeta^{i} \\right) \\ ,\n\\end{equation}\nwhere the first equality is from Proposition \\ref{prop:wupdate} and the second equality uses induction.  This then implies that $$ (1-\\eta_{i})s_{j_{i}}^{i} + \\eta_{i}x_{j_{i}}^{T}\\tbeta^{i} = \\dl (w_{j_{i}}^{i+1}) = s^{i+1} \\ . $$\nAnd for all coefficient indices $j \\ne i $ we have\n$$s_j^{i+1} = \\dl^*(w^{i+1}_j) = \\dl(w^i_j)= s^i \\ , $$\nwhere the second equality follows from conjugacy, whereby $s^{i+1}$ satisfies the update rule as stated in Algorithm \\ref{al:SFW}, thus demonstrating that the iterate sequence of Algorithm \\ref{al:RCMD} corresponds exactly to an iterate sequence of Algorithm \\ref{al:SFW}.  The same type of analysis as above can be used to prove that the iterate sequence of Algorithm \\ref{al:SFW} corresponds exactly to an iterate sequence of Algorithm \\ref{al:RCMD}. \\qed\n","sha256_utf8":"ec3866e7fcfac7020fa2aa425b547ea63bc3b638e75faf07e82bad5c7e18c77e"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"共轭初始化与优化子问题对应","source_units":[{"start":56824,"end":58357,"start_line":498,"end_line":502,"raw_tex":"{\\bf Proof of Lemma \\ref{lem:equi}}  We show that the iterate sequence of Algorithm \\ref{al:RCMD} corresponds exactly to an iterate sequence of Algorithm \\ref{al:SFW}.  The $\\{s^i\\}$ sequence is not formally defined in Algorithm \\ref{al:RCMD}, so let us define $s^i := \\nabla L^*(w^i)$ for all $i = 0, 1, \\ldots$, which is consistent through conjugacy with the relationship $w^i = \\nabla L(s^i)$ in the Optional Accounting step of Algorithm \\ref{al:SFW} (see Proposition \\ref{prop:gradient_conjugate}).  In order to show the correspondence we proceed by induction on the iteration counter $i$.  For $i=0$ we have from conjugacy that $s^0 := \\nabla L^*(w^0) = 0 $ from the definition $w^0$ in the  initialization of Algorithm \\ref{al:RCMD}.  We also need to show that $\\tbeta^0$ is a solution to the linear optimization oracle problem in Algorithm \\ref{al:SFW}.  We have for all $i=0, \\ldots$, that:\n$$\\begin{array}{rcl}\n\\tbeta^{i} \\in \\arg\\min_\\beta\\left\\{ \\tfrac{1}{n}\\left(w^{i}\\right)^{T}X\\beta+R(\\beta)\\right\\}    &=& \\arg\\min_\\beta\\left\\{ \\tfrac{1}{n}\\left(\\nabla L(s^{i})\\right)^{T}X\\beta+R(\\beta)\\right\\} \\\\ \\\\\n&=& \\arg\\min_\\beta\\left\\{ \\left(d^{i}\\right)^{T}\\beta+R(\\beta)\\right\\} \\ ,\n\\end{array}$$thus showing that $\\beta^{i}$ corresponds to a linear optimization oracle solution at iteration $i$ in Algorithm \\ref{al:SFW} for all $i=0, \\ldots$.  Now suppose that the correspondence holds for some iteration counter $i$, and let us examine $s^{i+1}:= \\nabla L^*(w^{i+1})$.  We have from Proposition \\ref{prop:wupdate} that:\n","sha256_utf8":"10cd5d182990c450c3e995fad6cc97228781bda43ecfbefef2270ffac39bf5d8"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"所选坐标更新对应","source_units":[{"start":58357,"end":58854,"start_line":503,"end_line":506,"raw_tex":"\\begin{equation}\\label{robo3}\nw_{j_{i}}^{i+1} =\\dot{l}_{j_{i}}\\left((1-\\eta_{i})\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i}) + \\eta_{i}x_{j_{i}}^{T}\\tbeta^{i} \\right)=\\dot{l}_{j_{i}}\\left((1-\\eta_{i})s_{j_{i}}^{i} + \\eta_{i}x_{j_{i}}^{T}\\tbeta^{i} \\right) \\ ,\n\\end{equation}\nwhere the first equality is from Proposition \\ref{prop:wupdate} and the second equality uses induction.  This then implies that $$ (1-\\eta_{i})s_{j_{i}}^{i} + \\eta_{i}x_{j_{i}}^{T}\\tbeta^{i} = \\dl (w_{j_{i}}^{i+1}) = s^{i+1} \\ . $$\n","sha256_utf8":"a92b54cb5a6ac48339453557951b3906f263a4dc765bf21781babfe0becc37ea"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"其余坐标与反向对应","source_units":[{"start":58854,"end":59419,"start_line":507,"end_line":509,"raw_tex":"And for all coefficient indices $j \\ne i $ we have\n$$s_j^{i+1} = \\dl^*(w^{i+1}_j) = \\dl(w^i_j)= s^i \\ , $$\nwhere the second equality follows from conjugacy, whereby $s^{i+1}$ satisfies the update rule as stated in Algorithm \\ref{al:SFW}, thus demonstrating that the iterate sequence of Algorithm \\ref{al:RCMD} corresponds exactly to an iterate sequence of Algorithm \\ref{al:SFW}.  The same type of analysis as above can be used to prove that the iterate sequence of Algorithm \\ref{al:SFW} corresponds exactly to an iterate sequence of Algorithm \\ref{al:RCMD}. \\qed\n","sha256_utf8":"90098a40af000af476ccc1a2e33c6f381e92a37542b2fea37953af4c5d252d8f"}]},{"id":"S026","semantic_type":"proposition","kind":"proposition","title":"对偶坐标的显式更新","source_units":[{"start":54778,"end":55132,"start_line":469,"end_line":474,"raw_tex":"\\begin{prop}\\label{prop:wupdate} At iteration $i$ of Algorithm \\ref{al:RCMD} it holds that:\n\n(1.) $\\dlsji(w_{j_{i}}^{i+1})=(1-\\eta_{i})\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i}) + \\eta_{i}x_{j_{i}}^{T}\\tbeta^{i}$, and\n\n(2.) $w_{j_{i}}^{i+1}=\\dot{l}_{j_{i}}\\left((1-\\eta_{i})\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})+ \\eta_{i}x_{j_{i}}^{T}\\tbeta^{i}\\right) $.\n\\end{prop}","sha256_utf8":"c68efcab92705fbd4a5f4a00602677f627c69962610e3c25fbfc5dbe28cef15a"}]},{"id":"S027","semantic_type":"proof","kind":"proof","title":"坐标更新公式完整证明","source_units":[{"start":55133,"end":56271,"start_line":475,"end_line":486,"raw_tex":"{\\bf Proof:}\nBecause $h(w)$ is a coordinate-wise separable function, we can rewrite the update for $w_{j_{i}}^{i+1}$ as\n\\begin{equation*}\n\\begin{array}{ll}\nw_{j_{i}}^{i+1} & =\\arg\\min_{w_{j_{i}}} \\left\\langle -\\frac{\\eta_{i}}{n}\\left(x_{j_{i}}^{T}\\tbeta^{i}-\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})\\right),w_{j_{i}}\\right\\rangle +D_{\\frac{1}{n} l_{j_{i}}^{*}}(w_{j_{i}},w_{j_{i}}^{i})\\\\ \\\\\n\n& =\\arg\\min_{w_{j_{i}}}\\left\\langle -\\eta_{i}\\left(x_{j_{i}}^{T}\\tbeta^{i}-\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})\\right),w_{j_{i}}\\right\\rangle +D_{ l_{j_{i}}^{*}}(w_{j_{i}},w_{j_{i}}^{i})\\\\ \\\\\n\n& =\\arg\\min_{w_{j_{i}}}\\left\\langle -\\eta_{i}x_{j_{i}}^{T}\\tbeta^{i}-(1-\\eta_{i})\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i}),w_{j_{i}}\\right\\rangle +l_{j_{i}}^{*}(w_{j_{i}}) \\ .  \\end{array}\n \\end{equation*}\n\nFrom the first-order optimality condition of the above $1$-dimensional problem we have $\\dls_{j_i}(w_{j_i}^{i+1})=\\eta_{i}x_{j_{i}}^{T}\\tbeta^{i}+(1-\\eta_{i})\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})$, which shows {\\it (1.)}; and {\\it (2.)} follows directly from {\\it (1.)} by the properties of the conjugate function in Proposition \\ref{prop:gradient_conjugate}. \\qed\n","sha256_utf8":"3487083c91552224d90ab8c232a9040e2bad87ad47631a0e360ed6098a742f53"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"分离镜像子问题","source_units":[{"start":55133,"end":55908,"start_line":475,"end_line":484,"raw_tex":"{\\bf Proof:}\nBecause $h(w)$ is a coordinate-wise separable function, we can rewrite the update for $w_{j_{i}}^{i+1}$ as\n\\begin{equation*}\n\\begin{array}{ll}\nw_{j_{i}}^{i+1} & =\\arg\\min_{w_{j_{i}}} \\left\\langle -\\frac{\\eta_{i}}{n}\\left(x_{j_{i}}^{T}\\tbeta^{i}-\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})\\right),w_{j_{i}}\\right\\rangle +D_{\\frac{1}{n} l_{j_{i}}^{*}}(w_{j_{i}},w_{j_{i}}^{i})\\\\ \\\\\n\n& =\\arg\\min_{w_{j_{i}}}\\left\\langle -\\eta_{i}\\left(x_{j_{i}}^{T}\\tbeta^{i}-\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})\\right),w_{j_{i}}\\right\\rangle +D_{ l_{j_{i}}^{*}}(w_{j_{i}},w_{j_{i}}^{i})\\\\ \\\\\n\n& =\\arg\\min_{w_{j_{i}}}\\left\\langle -\\eta_{i}x_{j_{i}}^{T}\\tbeta^{i}-(1-\\eta_{i})\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i}),w_{j_{i}}\\right\\rangle +l_{j_{i}}^{*}(w_{j_{i}}) \\ .  \\end{array}\n \\end{equation*}\n","sha256_utf8":"9a559dc585916a2a4388b71649cda41705d40ae0e126ea57cbeb373fb70cd847"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"最优性条件与共轭反演","source_units":[{"start":55909,"end":56271,"start_line":486,"end_line":486,"raw_tex":"From the first-order optimality condition of the above $1$-dimensional problem we have $\\dls_{j_i}(w_{j_i}^{i+1})=\\eta_{i}x_{j_{i}}^{T}\\tbeta^{i}+(1-\\eta_{i})\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})$, which shows {\\it (1.)}; and {\\it (2.)} follows directly from {\\it (1.)} by the properties of the conjugate function in Proposition \\ref{prop:gradient_conjugate}. \\qed\n","sha256_utf8":"fb9b5f86ade1f4d2bae993ea6ec0a45c2390d41a6c1ca1bb27ad47a48f7d7966"}]},{"id":"S030","semantic_type":"proposition","kind":"proposition","title":"对偶响应与迭代点有界","source_units":[{"start":61198,"end":61554,"start_line":542,"end_line":548,"raw_tex":"\\begin{prop}\\label{prop:well-behavior-of-w}\n\tLet $T:=\\left\\{ w \\in \\RR^n : |\\|w-w^{0}\\|_{\\infty}\\le \\gamma M\\right\\}$.  Then:\n\t\\begin{enumerate}\n\t\t\\item for any $\\hbeta\\in \\dom R(\\cdot)$ it holds that $\\argmax_w \\phi (\\hbeta, w) \\in T$, and\n\t\t\\item for all $w^i$ generated in Algorithm \\ref{al:RCMD}, it holds that  $w^i \\in T$.\n\t\\end{enumerate}\n\\end{prop}","sha256_utf8":"7dda1e71eb9ce97fdc3888bcd34a9ec2c5962d438764905625659647623f3972"}]},{"id":"S031","semantic_type":"proof","kind":"proof","title":"对偶有界性完整证明","source_units":[{"start":61556,"end":62587,"start_line":550,"end_line":561,"raw_tex":"\\textbf{Proof.} We first prove {\\it (1.)}.  Notice that $w_{0}= \\nabla L(0)$ and $\\argmax_w \\phi (\\hbeta, w) = \\nabla L(X\\hbeta)$ (from conjugacy via Proposition \\ref{prop:gradient_conjugate}) , whereby the $\\gamma$-smoothness of $l_{j}(\\cdot)$ implies that\n$$\n\\left\\|\\arg\\max_{w}\\phi(\\hat{\\beta},w)-w_{0}\\right\\|_{\\infty}=\\left\\| \\nabla L(X\\hbeta)-\\nabla L(0)\\right\\|_{\\infty}=\\max_j \\left|\\dl_j(x_j^T\\hat \\beta)-\\dl_j(0)\\right|\\le\\gamma\\max_{j}|x_{j}^{T}\\hat \\beta| \\le \\gamma M \\ ,\n$$\n\n\nwhich proves {\\it (1.)}. It follows from Proposition \\ref{cor:bound} that for any coordinate $j$ and iterate $i$ it holds that $\\left|\\dot{l}_{j}^{*}(w_{j}^{i})\\right|\\le M$. Together with $\\dls_j(w_j^0)=0$, we have\n\n$$\n\\tfrac{1}{\\gamma}\\left|w_{j}^{i}-w_{j}^{0}\\right|\\le \\left|\\dot{l}_{j}^{*}(w_{j}^{i})-\\dot{l}_{j}^{*}(w_{j}^{0}) \\right|\\le M \\ ,\n$$\n(where the first inequality is from the $\\frac{1}{\\gamma}$-strong convexity of $l_j^*(w_j)$), from which it follows that $\\|w^i - w^0\\|_\\infty \\le \\gamma M$, which proves {\\it (2.)}. \\qed\n","sha256_utf8":"f8f8f7dbe8a1ba76f65eb34a4459cf4ff6ff0a6fcdfba7cbbca2911712336781"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"光滑性控制最优响应","source_units":[{"start":61556,"end":62044,"start_line":550,"end_line":553,"raw_tex":"\\textbf{Proof.} We first prove {\\it (1.)}.  Notice that $w_{0}= \\nabla L(0)$ and $\\argmax_w \\phi (\\hbeta, w) = \\nabla L(X\\hbeta)$ (from conjugacy via Proposition \\ref{prop:gradient_conjugate}) , whereby the $\\gamma$-smoothness of $l_{j}(\\cdot)$ implies that\n$$\n\\left\\|\\arg\\max_{w}\\phi(\\hat{\\beta},w)-w_{0}\\right\\|_{\\infty}=\\left\\| \\nabla L(X\\hbeta)-\\nabla L(0)\\right\\|_{\\infty}=\\max_j \\left|\\dl_j(x_j^T\\hat \\beta)-\\dl_j(0)\\right|\\le\\gamma\\max_{j}|x_{j}^{T}\\hat \\beta| \\le \\gamma M \\ ,\n$$\n","sha256_utf8":"42e410d2fe2c894e6dc40fa591255a1fa231daa35eee643a4f258494d066acd9"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"共轭强凸性控制迭代点","source_units":[{"start":62046,"end":62587,"start_line":556,"end_line":561,"raw_tex":"which proves {\\it (1.)}. It follows from Proposition \\ref{cor:bound} that for any coordinate $j$ and iterate $i$ it holds that $\\left|\\dot{l}_{j}^{*}(w_{j}^{i})\\right|\\le M$. Together with $\\dls_j(w_j^0)=0$, we have\n\n$$\n\\tfrac{1}{\\gamma}\\left|w_{j}^{i}-w_{j}^{0}\\right|\\le \\left|\\dot{l}_{j}^{*}(w_{j}^{i})-\\dot{l}_{j}^{*}(w_{j}^{0}) \\right|\\le M \\ ,\n$$\n(where the first inequality is from the $\\frac{1}{\\gamma}$-strong convexity of $l_j^*(w_j)$), from which it follows that $\\|w^i - w^0\\|_\\infty \\le \\gamma M$, which proves {\\it (2.)}. \\qed\n","sha256_utf8":"08d2d494a3ec4213f2015c50d868ae469b5add63a134273d12f806952e17bbf0"}]},{"id":"S034","semantic_type":"proposition","kind":"proposition","title":"Bregman半径上界","source_units":[{"start":60955,"end":61071,"start_line":536,"end_line":538,"raw_tex":"\\begin{prop}\\label{prop:Dmax}\n\tUnder Assumption \\ref{ass:smooth} it holds that $D_{\\max} \\le \\gamma M^2$.\n\\end{prop}","sha256_utf8":"cb2c3fc85894df0dc4dfe5bca65ab2a5011c470a42d996f2d6af2a8a0ed4befd"}]},{"id":"S035","semantic_type":"proof","kind":"proof","title":"半径上界完整证明","source_units":[{"start":62588,"end":64159,"start_line":563,"end_line":579,"raw_tex":"{\\bf Proof of Proposition \\ref{prop:Dmax}:}  Let $L(s):=\\sum_{j=1}^n l_j(s_j)$ and $L^*(w):=\\sum_{j=1}^n l^*_j(w_j)$, and note that $L(\\cdot)$ and $L^*(\\cdot)$ are a conjugate pair.  Let $\\hw \\in \\cal W$ and let $\\hbeta$ be such that $\\hw \\in \\arg\\max_w \\phi(\\hbeta, w)$.  Then\n\n%It follows Proposition \\ref{prop:well-behavior-of-w}.1 that $\\hw^k\\in T$, therefore for any $k$ we have\n\n\\begin{equation*}\n\\begin{array}{lcl}\nD_{h}(\\hw, w^0) & = &\\tfrac{1}{n}\\left( L^*(\\hw)-L^*(w^0)\\right)\\\\ \\\\\n& =&\\tfrac{1}{n}\\left( \\left(\\hw \\right)^{T}X\\hbeta-L(X\\hbeta) -L^{*}(w^{0})\\right)\\\\ \\\\\n& \\le&\\tfrac{1}{n}\\left(\\max_{w\\in T,\\beta\\in \\dom R(\\cdot)}\\left\\{ w^{T}X\\beta-L(X\\beta)\\right\\} -L^{*}(w^{0})\\right)\\\\ \\\\\n& =&\\tfrac{1}{n}\\left(\\max_{w\\in T,\\beta\\in  \\dom R(\\cdot)}\\left\\{ (w-w^{0})^{T}X\\beta+\\left(w^{0}\\right)^{T}X\\beta-L(X\\beta)\\right\\} -L^{*}(w^{0})\\right)\\\\ \\\\\n& \\le&\\tfrac{1}{n}\\left(\\max_{w\\in T,\\beta\\in  \\dom R(\\cdot)}\\left\\{ (w-w^{0})^{T}X\\beta\\right\\} +\\max_{\\beta\\in  \\dom R(\\cdot)}\\left\\{ \\left(w^{0}\\right)^{T}X\\beta-L(X\\beta)\\right\\} -L^{*}(w^{0})\\right)\\\\ \\\\\n& \\le &\\tfrac{1}{n}\\left(n\\max_{w\\in T,\\beta\\in  \\dom R(\\cdot)}\\|w-w^{0}\\|_{\\infty}\\|X\\beta\\|_{\\infty}+L^{*}(w^{0})-L^{*}(w^{0})\\right)\\\\ \\\\\n& \\le &\\gamma M^{2} \\ ,\n\\end{array}\n\\end{equation*}\n\nwhere the second equality follows from Proposition \\ref{prop:gradient_conjugate}, the first inequality uses $\\hbeta \\in  \\dom R(\\cdot)$ and $\\hw \\in T$ (from Proposition \\ref{prop:well-behavior-of-w}), and the last inequality uses $\\max_{\\beta\\in  \\dom R(\\cdot)}\\|X\\beta\\|_{\\infty}\\le M$. \\qed \\medskip\n","sha256_utf8":"4826b12fce77c261c16b0ae947fefce7ffafd6640a46d70a29e65a58bf85f410"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"选取共轭函数与最优响应","source_units":[{"start":62588,"end":62973,"start_line":563,"end_line":566,"raw_tex":"{\\bf Proof of Proposition \\ref{prop:Dmax}:}  Let $L(s):=\\sum_{j=1}^n l_j(s_j)$ and $L^*(w):=\\sum_{j=1}^n l^*_j(w_j)$, and note that $L(\\cdot)$ and $L^*(\\cdot)$ are a conjugate pair.  Let $\\hw \\in \\cal W$ and let $\\hbeta$ be such that $\\hw \\in \\arg\\max_w \\phi(\\hbeta, w)$.  Then\n\n%It follows Proposition \\ref{prop:well-behavior-of-w}.1 that $\\hw^k\\in T$, therefore for any $k$ we have\n\n","sha256_utf8":"32f3b289187042fe964c23d55c0d749437dbf49d6794dbcac0749444bd1389e6"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"共轭等式导出统一半径界","source_units":[{"start":62973,"end":64159,"start_line":567,"end_line":579,"raw_tex":"\\begin{equation*}\n\\begin{array}{lcl}\nD_{h}(\\hw, w^0) & = &\\tfrac{1}{n}\\left( L^*(\\hw)-L^*(w^0)\\right)\\\\ \\\\\n& =&\\tfrac{1}{n}\\left( \\left(\\hw \\right)^{T}X\\hbeta-L(X\\hbeta) -L^{*}(w^{0})\\right)\\\\ \\\\\n& \\le&\\tfrac{1}{n}\\left(\\max_{w\\in T,\\beta\\in \\dom R(\\cdot)}\\left\\{ w^{T}X\\beta-L(X\\beta)\\right\\} -L^{*}(w^{0})\\right)\\\\ \\\\\n& =&\\tfrac{1}{n}\\left(\\max_{w\\in T,\\beta\\in  \\dom R(\\cdot)}\\left\\{ (w-w^{0})^{T}X\\beta+\\left(w^{0}\\right)^{T}X\\beta-L(X\\beta)\\right\\} -L^{*}(w^{0})\\right)\\\\ \\\\\n& \\le&\\tfrac{1}{n}\\left(\\max_{w\\in T,\\beta\\in  \\dom R(\\cdot)}\\left\\{ (w-w^{0})^{T}X\\beta\\right\\} +\\max_{\\beta\\in  \\dom R(\\cdot)}\\left\\{ \\left(w^{0}\\right)^{T}X\\beta-L(X\\beta)\\right\\} -L^{*}(w^{0})\\right)\\\\ \\\\\n& \\le &\\tfrac{1}{n}\\left(n\\max_{w\\in T,\\beta\\in  \\dom R(\\cdot)}\\|w-w^{0}\\|_{\\infty}\\|X\\beta\\|_{\\infty}+L^{*}(w^{0})-L^{*}(w^{0})\\right)\\\\ \\\\\n& \\le &\\gamma M^{2} \\ ,\n\\end{array}\n\\end{equation*}\n\nwhere the second equality follows from Proposition \\ref{prop:gradient_conjugate}, the first inequality uses $\\hbeta \\in  \\dom R(\\cdot)$ and $\\hw \\in T$ (from Proposition \\ref{prop:well-behavior-of-w}), and the last inequality uses $\\max_{\\beta\\in  \\dom R(\\cdot)}\\|X\\beta\\|_{\\infty}\\le M$. \\qed \\medskip\n","sha256_utf8":"e463a02c7b65e5578edbdb94f0cd1965b5652463828b5f908746162bb5eefc4e"}]},{"id":"S038","semantic_type":"proposition","kind":"proposition","title":"共轭导数沿迭代有界","source_units":[{"start":67909,"end":68057,"start_line":607,"end_line":609,"raw_tex":"\\begin{prop}\nFor all iterates $i$ and any $j\\in\\left\\{ 1,\\ldots,n\\right\\} $ it holds that $\\left|\\dot{l}_{j}^{*}(w_{j}^{i})\\right|\\le M$.\n\\end{prop}","sha256_utf8":"eb508c07d3bd3b6394710f28ac7aa0549171ac258f0f4d46ed7551d5bfd5036a"}]},{"id":"S039","semantic_type":"proof","kind":"proof","title":"共轭导数界完整证明","source_units":[{"start":68059,"end":68900,"start_line":611,"end_line":615,"raw_tex":"{\\bf Proof.} We prove this by induction on $i$. The proposition is true for $i=0$ because $\\dot{l}_{j}^{*}(w_{j}^{0})=0$ for all $j$ by the definition of $w^0$. Next suppose that $\\left|\\dot{l}_{j}^{*}(w_{j}^{i})\\right|\\le M$ for a given iterate $i$ and for all $j=1, \\ldots, n$. Then at iteration $i+1$ and any $j\\not=j_{i}$ we have  $w_j^{i+1}=w_j^i$, whereby $\\left|\\dot{l}_{j}^{*}(w_{j}^{i+1})\\right|=\\left|\\dot{l}_{j}^{*}(w_{j}^{i})\\right|\\le M$.  And it follows from Proposition \\ref{prop:wupdate} that\n\\[\n\\left|\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i+1})\\right|=\\left|\\left(1-\\eta_{i}\\right)\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})+\\eta_{i}x_{j_{i}}^T\\tbeta^{i}\\right|\\le(1-\\eta_{i})M+\\eta_{i}M=M \\ ,\n\\]\nand therefore for any $j=1, \\ldots, n$, we have $\\left|\\dot{l}_{j}^{*}(w_{j}^{i+1})\\right|\\le M$, which completes the proof by induction. \\qed\n","sha256_utf8":"057b3252e1798d3f5e10432f977b3ded73022d146c0ccf9a9498807b482f33af"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"凸组合更新的归纳有界性","source_units":[{"start":68059,"end":68900,"start_line":611,"end_line":615,"raw_tex":"{\\bf Proof.} We prove this by induction on $i$. The proposition is true for $i=0$ because $\\dot{l}_{j}^{*}(w_{j}^{0})=0$ for all $j$ by the definition of $w^0$. Next suppose that $\\left|\\dot{l}_{j}^{*}(w_{j}^{i})\\right|\\le M$ for a given iterate $i$ and for all $j=1, \\ldots, n$. Then at iteration $i+1$ and any $j\\not=j_{i}$ we have  $w_j^{i+1}=w_j^i$, whereby $\\left|\\dot{l}_{j}^{*}(w_{j}^{i+1})\\right|=\\left|\\dot{l}_{j}^{*}(w_{j}^{i})\\right|\\le M$.  And it follows from Proposition \\ref{prop:wupdate} that\n\\[\n\\left|\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i+1})\\right|=\\left|\\left(1-\\eta_{i}\\right)\\dot{l}_{j_{i}}^{*}(w_{j_{i}}^{i})+\\eta_{i}x_{j_{i}}^T\\tbeta^{i}\\right|\\le(1-\\eta_{i})M+\\eta_{i}M=M \\ ,\n\\]\nand therefore for any $j=1, \\ldots, n$, we have $\\left|\\dot{l}_{j}^{*}(w_{j}^{i+1})\\right|\\le M$, which completes the proof by induction. \\qed\n","sha256_utf8":"057b3252e1798d3f5e10432f977b3ded73022d146c0ccf9a9498807b482f33af"}]},{"id":"S041","semantic_type":"corollary","kind":"corollary","title":"随机坐标次梯度范数界","source_units":[{"start":68994,"end":69136,"start_line":619,"end_line":621,"raw_tex":"\\begin{cor}\\label{cor:bound}\n$\\|\\tilde{g}^i\\|_{2} =\\frac{1}{n}\\left| x_{j_{i}}^T \\tbeta^i - \\dlsji(\\wiji) \\right| \\le \\frac{2M}{n}$.\n\\end{cor}","sha256_utf8":"a6c7a9e72bdfc2dea0c278cabac14bac7b838c2a94bab36c954424f943da7ac9"}]},{"id":"S042","semantic_type":"proposition","kind":"proposition","title":"参考函数的强凸性","source_units":[{"start":69146,"end":69280,"start_line":623,"end_line":626,"raw_tex":"\\begin{prop}\\label{prop:strong}\n$h(\\cdot)$ is\n$\\frac{1}{n\\gamma}$-strongly convex with respect to the norm $\\|\\cdot\\|_{2}$.\n\\end{prop}","sha256_utf8":"a9befa0fd3004b3cf720aab1605664f468c91d0d35d65988a85201a4a1f5e886"}]},{"id":"S043","semantic_type":"proof","kind":"proof","title":"参考函数强凸性完整证明","source_units":[{"start":69281,"end":69927,"start_line":627,"end_line":634,"raw_tex":"{\\bf Proof.} Recall that $h(w)=\\frac{1}{n}\\sum_{j=1}^{n}l_{j}^{*}(w_{j})$. It follows from Assumption \\ref{ass:smooth} and Proposition \\ref{prop:gradient_conjugate} that $\\dls_j(\\cdot)$ is $\\frac{1}{\\gamma}$-strongly convex. Therefore for any $w^1, w^2 \\in \\dom h(\\cdot)$ it holds that:\n\\begin{equation*}\n\\begin{array}{lcl}\nh(w^1)& = &\\frac{1}{n}\\sum_{j=1}^{n}l_{j}^{*}(w^1_{j}) \\\\ \\\\\n& \\ge & \\frac{1}{n}\\sum_{j=1}^{n}\\left( l_{j}^{*}(w^2_{j}) + \\dls_j(w_j^2)(w_j^1-w_j^2)+\\frac{1}{2\\gamma}|w^2_j-w^1_j|^2\\right) \\\\ \\\\\n& = & h(w^2) + \\langle\\nabla h(w^2), w^1-w^2\\rangle + \\frac{1}{2n\\gamma}\\|w^2-w^1\\|_2^2 \\ . \\ \\qed\n\\end{array}\n\\end{equation*}\n","sha256_utf8":"de0ea31bb64d06345e38f0ec89688d7cd54ae7898676e4bf9ce3b2c05ad534d5"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"逐坐标累加强凸不等式","source_units":[{"start":69281,"end":69927,"start_line":627,"end_line":634,"raw_tex":"{\\bf Proof.} Recall that $h(w)=\\frac{1}{n}\\sum_{j=1}^{n}l_{j}^{*}(w_{j})$. It follows from Assumption \\ref{ass:smooth} and Proposition \\ref{prop:gradient_conjugate} that $\\dls_j(\\cdot)$ is $\\frac{1}{\\gamma}$-strongly convex. Therefore for any $w^1, w^2 \\in \\dom h(\\cdot)$ it holds that:\n\\begin{equation*}\n\\begin{array}{lcl}\nh(w^1)& = &\\frac{1}{n}\\sum_{j=1}^{n}l_{j}^{*}(w^1_{j}) \\\\ \\\\\n& \\ge & \\frac{1}{n}\\sum_{j=1}^{n}\\left( l_{j}^{*}(w^2_{j}) + \\dls_j(w_j^2)(w_j^1-w_j^2)+\\frac{1}{2\\gamma}|w^2_j-w^1_j|^2\\right) \\\\ \\\\\n& = & h(w^2) + \\langle\\nabla h(w^2), w^1-w^2\\rangle + \\frac{1}{2n\\gamma}\\|w^2-w^1\\|_2^2 \\ . \\ \\qed\n\\end{array}\n\\end{equation*}\n","sha256_utf8":"de0ea31bb64d06345e38f0ec89688d7cd54ae7898676e4bf9ce3b2c05ad534d5"}]},{"id":"S045","semantic_type":"proposition","kind":"proposition","title":"鞍点函数的精确展开","source_units":[{"start":69938,"end":70096,"start_line":638,"end_line":640,"raw_tex":"\\begin{prop}\\label{prop:equality}\n$\\phi(\\tbeta^{i},w)=D(w^{i})+\\left\\langle \\nabla_{w}\\phi(\\tbeta^{i},w^{i}),w-w^{i}\\right\\rangle -D_{h}(w,w^{i}).$\n\\end{prop}","sha256_utf8":"6d8a4dbee09e5a25bde3754eed56b776650c82541a317dd812d79b6d1281f201"}]},{"id":"S046","semantic_type":"proof","kind":"proof","title":"鞍点展开完整证明","source_units":[{"start":70097,"end":70376,"start_line":641,"end_line":641,"raw_tex":"\\textbf{Proof. } The proof follows from straightforward substitution using $\\phi(\\tbeta^{i},w)=\\frac{1}{n} \\left(w^{T}X\\tbeta^{i}-\\sum_{j=1}^n l_{j}^{*}(w_{j})\\right) + R(\\tbeta^i)$ and noticing from the construction of $\\tbeta^i$ that $D(w^i)=\\phi(\\tbeta^i, w^i)$. \\qed\\medskip\n","sha256_utf8":"d16fd0171e99b56649d5c99ac85cf9176f9acaad006506f270622735dfa0ae40"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"代入定义与优化子问题值","source_units":[{"start":70097,"end":70376,"start_line":641,"end_line":641,"raw_tex":"\\textbf{Proof. } The proof follows from straightforward substitution using $\\phi(\\tbeta^{i},w)=\\frac{1}{n} \\left(w^{T}X\\tbeta^{i}-\\sum_{j=1}^n l_{j}^{*}(w_{j})\\right) + R(\\tbeta^i)$ and noticing from the construction of $\\tbeta^i$ that $D(w^i)=\\phi(\\tbeta^i, w^i)$. \\qed\\medskip\n","sha256_utf8":"d16fd0171e99b56649d5c99ac85cf9176f9acaad006506f270622735dfa0ae40"}]},{"id":"S048","semantic_type":"proposition","kind":"proposition","title":"原始迭代的加权平均表示","source_units":[{"start":70378,"end":70824,"start_line":644,"end_line":646,"raw_tex":"\\begin{prop}\\label{robfriday}  Consider the series $\\{ \\alpha_i \\}$ defined by $\\alpha_i = \\frac{2(2n+i)}{(4n+i)(i+1)}$ for $i \\ge 0$ and define the series $\\{\\bbeta^i\\}$ by $\\bbeta^{-1} =0$ and $\\bbeta^i = (1-\\alpha_i)\\bbeta^{i-1} + \\alpha_i \\tbeta^i$ for $i\\ge 0$.  Also define $\\gamma_i = 2n+i$ for $i \\ge 0$.  Then\n$$ \\bbeta^k = \\frac{\\sum_{i=0}^k \\gamma_i \\tilde{\\beta}^i}{\\sum_{i=0}^k \\gamma_i} \\ \\ \\mbox{for~all~} k \\ge 0 \\ . $$\n\\end{prop}","sha256_utf8":"76338495e852142096060705f9315e84f5b1da5251f14af1e4bb7d3e4c08dab8"}]},{"id":"S049","semantic_type":"proof","kind":"proof","title":"加权平均表示完整证明","source_units":[{"start":70825,"end":70950,"start_line":647,"end_line":647,"raw_tex":"\\noindent {\\bf Proof:} The proof follows easily by induction and using $\\sum_{i=0}^k \\gamma_i = \\frac{(4n+k)(k+1)}{2}$. \\qed\n","sha256_utf8":"eabfdd8fca15daa97e159e0a18eb9ff3ba08293f2f5e10720010f37ceed4302a"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"权重求和与归纳验证","source_units":[{"start":70825,"end":70950,"start_line":647,"end_line":647,"raw_tex":"\\noindent {\\bf Proof:} The proof follows easily by induction and using $\\sum_{i=0}^k \\gamma_i = \\frac{(4n+k)(k+1)}{2}$. \\qed\n","sha256_utf8":"eabfdd8fca15daa97e159e0a18eb9ff3ba08293f2f5e10720010f37ceed4302a"}]},{"id":"S051","semantic_type":"assumption","kind":"assumption","title":"主定理步长与对偶平均设定","source_units":[{"start":65447,"end":65798,"start_line":593,"end_line":593,"raw_tex":"Consider the Stochastic Generalized Frank-Wolfe method (Algorithm \\ref{al:SFW}) or the Randomized Dual Coordinate Mirror Descent method (Algorithm \\ref{al:RCMD}), with step-size sequences $\\alpha_i=\\frac{2(2n+i)}{(i+1)(4n+i)}$ and $\\eta_i=\\frac{2n}{2n+i+1}$ for $i=0, 1, \\ldots$.  Denote $$\\bw^{k} = \\frac{2}{(4n+k)(k+1)}\\sum_{i=0}^k(2n+i) w^i \\ . $$\n","sha256_utf8":"2d9681e0a4a430e8267aafb1d1e78cadf5164bd96674b798ed7875132ff0e256"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"次梯度与坐标估计的条件均值","source_units":[{"start":70951,"end":71288,"start_line":649,"end_line":649,"raw_tex":"{\\bf Proof of Theorem \\ref{thm:non-strong}.}  Denote $g^i:=\\frac{1}{n} \\left( X\\tbeta^i-\\nabla L^*(w^i)\\right)$, whereby $g^i$ is a subgradient of $D(w)$ at $w^i$, and $\\tg^i$ is an unbiased estimator of $g^i$ up to the scalar $n$, namely $\\EE_{j_i} [ \\tg^i ]=\\frac{1}{n}g^i$.  Therefore we have for any $i$ and any $w \\in \\cal W$ that:\n","sha256_utf8":"449f9395f4f6d50b6257a4319e9f94f4720ab8e89e61b0f3544a8395d60ba842"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"三点性质导出单步下降界","source_units":[{"start":71288,"end":72627,"start_line":650,"end_line":660,"raw_tex":"\\begin{equation}\\label{eq:nonsmooth-theonehand}\n\\begin{array}{ll}\n\\left\\langle -g^i,w-w^{i}\\right\\rangle  & =n \\EE_{j_i} \\left[\\left\\langle -\\tg^i,w-w^{i}\\right\\rangle\\right] \\\\ \\\\\n & \\ge n \\EE_{j_i} \\left[\\left\\langle -\\tg^i,w^{i+1}-w^{i}\\right\\rangle +\\frac{1}{\\eta_{i}}D_{h}(w^{i+1},w^{i})+\\frac{1}{\\eta_{i}}D_{h}(w,w^{i+1})-\\frac{1}{\\eta_{i}}D_{h}(w,w^{i})\\right]\\\\ \\\\\n & \\ge n \\EE_{j_i} \\left[\\left\\langle -\\tg^i,w^{i+1}-w^{i}\\right\\rangle +\\frac{1}{2n\\gamma\\eta_{i}}\\|w^{i+1}-w^{i}\\|_{2}^{2}+\\frac{1}{\\eta_{i}}D_{h}(w,w^{i+1})-\\frac{1}{\\eta_{i}}D_{h}(w,w^{i})\\right]\\\\ \\\\\n  & \\ge n \\EE_{j_i} \\left[ -\\frac{1}{2}n\\gamma \\eta_{i} \\|\\tg^i\\|_2^2+\\frac{1}{\\eta_{i}}D_{h}(w,w^{i+1})-\\frac{1}{\\eta_{i}}D_{h}(w,w^{i})\\right]\\\\ \\\\\n & \\ge -2\\gamma M^{2}\\eta_{i}+\\frac{n}{\\eta_{i}}\\EE_{j_i} [D_{h}(w,w^{i+1})]-\\frac{n}{\\eta_{i}}D_{h}(w,w^{i}) \\ ,\n\\end{array}\n\\end{equation}\nwhere the first inequality is from the ``three point property'' of Tseng (Lemma \\ref{tpp} in the Appendix), the second\ninequality is due to the fact that $h(w)$ is $\\frac{1}{n\\gamma}$-strongly convex with respect to the norm $\\|\\cdot\\|_2$ (Proposition \\ref{prop:strong}), and the third inequality is an application of the basic inequality $\\langle x, y\\rangle \\le \\tfrac{1}{2}\\|x\\|_2^2 +  \\tfrac{1}{2}\\|y\\|_2^2$,  and the last inequality uses Corollary \\ref{cor:bound}.\n","sha256_utf8":"d022c944a47c10e7db1617474609ba674b5debdac7f66a2212ee25b142c53ffd"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"次梯度内积改写为鞍点间隙","source_units":[{"start":72628,"end":73047,"start_line":662,"end_line":666,"raw_tex":"On the other hand, we have from Proposition \\ref{prop:equality} that\n% by substituting $\\phi(\\tbeta^{i},w)=\\frac{1}{n} \\left(w^{T}X\\tbeta^{i}-\\sum l_{i}^{*}(w_{i})\\right) + R(\\tbeta^i)$  we have\n\\begin{equation}\\label{eq:nonsmooth-theotherhand}\n\\left\\langle -g^i,w-w^{i}\\right\\rangle   =\\left\\langle -\\nabla_{w}\\phi(\\tbeta^{i},w^{i}),w-w^{i}\\right\\rangle  =D(w^{i})-\\phi(\\tbeta^{i},w)-D_{h}(w,w^{i}) \\ .\n\\end{equation}\n","sha256_utf8":"2fe2d0ec8dca848d589b37807a2e5d0f52e552e7f85ae500ef24692b1e455cd9"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"合并得到Bregman递推","source_units":[{"start":73049,"end":73311,"start_line":669,"end_line":672,"raw_tex":"Combining \\eqref{eq:nonsmooth-theonehand} and \\eqref{eq:nonsmooth-theotherhand} and rearranging yields\n$$\n-(\\phi(\\tbeta^{i},w)-D(w^{i}))\\ge-2\\gamma M^{2}\\eta_{i}+\\tfrac{n}{\\eta_{i}}\\EE_{j_i}[D_{h}(w,w^{i+1})]-\\left(\\tfrac{n}{\\eta_{i}}-1\\right)D_{h}(w,w^{i}).\n$$\n","sha256_utf8":"d8a37c2d7a290fed094322cfe2a0346d632ee4f626cd62e2c8f4fc48e0225755"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"代入步长构造望远镜权重","source_units":[{"start":73311,"end":73682,"start_line":673,"end_line":677,"raw_tex":"Substituting $\\eta_{i}=\\frac{2n}{2n+i+1}$ and multiplying by $2n+i$ results, we arrive at the following inequality after rearranging terms:\n$$\\begin{array}{l}\n(2n+i)(\\phi(\\tbeta^{i},w)-D(w^{i}))\\\\\n\\le  \\ \\ 4n\\gamma  M^{2}\\left(\\tfrac{2n+i}{2n+i+1}\\right)+\\tfrac{1}{2}\\left((2n+i)(2n+i-1)D_{h}(w,w^{i})-(2n+i)(2n+i+1)\\EE_{j_i}[ D_{h}(w,w^{i+1})] \\right) \\ .\n\\end{array}$$\n","sha256_utf8":"31d9e204249eb8b1dd29625ea48bdd3ce2ebd29f06be40db6c8020e2fb7335a2"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"加权平均与期望望远镜求和","source_units":[{"start":73682,"end":74685,"start_line":678,"end_line":688,"raw_tex":"Summing the above inequality for $i=0,\\ldots,k$ and recalling from Proposition \\ref{robfriday} that $\\bar{\\beta}^{k}:=\\frac{2}{(4n+k)(k+1)}\\sum_{i=0}^{k}(2n+i)\\tbeta^{i}$,\nand taking the unconditional expectation over $\\xi_k$ (recall the definition of $\\xi_k$ in \\eqref{eq:xi}), we arrive at:\n$$\\begin{array}{rcl}\n\\frac{(4n+k)(k+1)}{2}\\EE_{\\xi_k}[\\phi(\\bar{\\beta}^{k},w)-D(\\bar{w}^{k})] & = & \\left(\\sum_{i=0}^k 2n+i \\right)\\EE_{\\xi_k}[\\phi(\\bar{\\beta}^{k},w)-D(\\bar{w}^{k})] \\\\ \\\\\n& \\le & \\EE_{\\xi_k}\\left[\\sum_{i=0}^{k}(2n+i)(\\phi(\\tbeta^{i},w)-D(w^{i}))\\right]\\\\ \\\\\n& \\le & 4(k+1)n\\gamma M^{2}+\\tfrac{1}{2}(2n)(2n-1)D_{h}(w,w^{0})\\\\ \\\\\n& \\le & 4(k+1)n\\gamma M^{2}+n(2n-1)D_{\\max} \\ ,\n\\end{array}$$\nwhere the first inequality uses the convexity of $\\phi(\\beta,w)$\nover $\\beta$ and the concavity of $D(w)$, the second inequality\nfollows from the summation and canceling terms in the telescoping series, and the third inequality uses $w \\in \\cal W$.  Choosing $\\hw=\\arg\\max_{w}\\phi(\\bar{\\beta}^{k},w)$,\n","sha256_utf8":"8d9a905124cd085dde5f6c572801315ecbb1d68fe3270f5096396aa4605de98b"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"最优响应给出原始对偶间隙","source_units":[{"start":74685,"end":74958,"start_line":689,"end_line":692,"raw_tex":"we have $P(\\bar{\\beta}^{k})=\\phi(\\bar{\\beta}^{k},\\hw)$, which yields:\n\\[\n\\EE_{\\xi_k}[P(\\bar{\\beta}^{k})-D(\\bar{w}^{k})]\\le\\frac{8n\\gamma M^{2}}{\\left(4n+k\\right)}+\\frac{2n(2n-1)D_{\\max}}{(4n+k)(k+1)} \\ ,\n\\]thus showing the first inequality in the statement of the theorem.\n","sha256_utf8":"4df13792efddef9a0805c43f5acd64df1c3c78dcac8613dce97deb611b7e61ca"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"代入半径界完成第二个估计","source_units":[{"start":74958,"end":75086,"start_line":693,"end_line":693,"raw_tex":"The second inequality in the statement of the theorem then follows as a simple application of Proposition \\ref{prop:Dmax}. \\qed\n","sha256_utf8":"3c1598de333a72e97f251974c287a6a0dbb186ac515b14090fd294baad19772d"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S018","target_id":"S016","kind":"supports"},{"source_id":"S019","target_id":"S017","kind":"supports"},{"source_id":"S003","target_id":"S010","kind":"supports"},{"source_id":"S004","target_id":"S012","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S010","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S010","kind":"supports"},{"source_id":"S010","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S011","target_id":"S014","kind":"supports"},{"source_id":"S007","target_id":"S014","kind":"supports"},{"source_id":"S011","target_id":"S028","kind":"supports"},{"source_id":"S007","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S016","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S006","target_id":"S023","kind":"supports"},{"source_id":"S011","target_id":"S023","kind":"supports"},{"source_id":"S016","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S016","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S006","target_id":"S025","kind":"supports"},{"source_id":"S011","target_id":"S025","kind":"supports"},{"source_id":"S016","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S002","kind":"supports"},{"source_id":"S011","target_id":"S040","kind":"supports"},{"source_id":"S026","target_id":"S040","kind":"supports"},{"source_id":"S012","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S041","kind":"supports"},{"source_id":"S012","target_id":"S041","kind":"supports"},{"source_id":"S011","target_id":"S041","kind":"supports"},{"source_id":"S004","target_id":"S032","kind":"supports"},{"source_id":"S016","target_id":"S032","kind":"supports"},{"source_id":"S010","target_id":"S032","kind":"supports"},{"source_id":"S011","target_id":"S032","kind":"supports"},{"source_id":"S012","target_id":"S032","kind":"supports"},{"source_id":"S041","target_id":"S033","kind":"supports"},{"source_id":"S004","target_id":"S033","kind":"supports"},{"source_id":"S016","target_id":"S033","kind":"supports"},{"source_id":"S011","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S005","target_id":"S036","kind":"supports"},{"source_id":"S013","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S016","target_id":"S037","kind":"supports"},{"source_id":"S030","target_id":"S037","kind":"supports"},{"source_id":"S012","target_id":"S037","kind":"supports"},{"source_id":"S014","target_id":"S037","kind":"supports"},{"source_id":"S011","target_id":"S037","kind":"supports"},{"source_id":"S007","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S004","target_id":"S044","kind":"supports"},{"source_id":"S016","target_id":"S044","kind":"supports"},{"source_id":"S011","target_id":"S044","kind":"supports"},{"source_id":"S007","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S009","target_id":"S047","kind":"supports"},{"source_id":"S010","target_id":"S047","kind":"supports"},{"source_id":"S011","target_id":"S047","kind":"supports"},{"source_id":"S007","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S048","kind":"supports"},{"source_id":"S009","target_id":"S052","kind":"supports"},{"source_id":"S010","target_id":"S052","kind":"supports"},{"source_id":"S011","target_id":"S052","kind":"supports"},{"source_id":"S020","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S017","target_id":"S053","kind":"supports"},{"source_id":"S042","target_id":"S053","kind":"supports"},{"source_id":"S041","target_id":"S053","kind":"supports"},{"source_id":"S020","target_id":"S053","kind":"supports"},{"source_id":"S052","target_id":"S054","kind":"supports"},{"source_id":"S045","target_id":"S054","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S051","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S048","target_id":"S057","kind":"supports"},{"source_id":"S051","target_id":"S057","kind":"supports"},{"source_id":"S015","target_id":"S057","kind":"supports"},{"source_id":"S014","target_id":"S057","kind":"supports"},{"source_id":"S020","target_id":"S057","kind":"supports"},{"source_id":"S010","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S010","target_id":"S058","kind":"supports"},{"source_id":"S013","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S034","target_id":"S059","kind":"supports"},{"source_id":"S052","target_id":"S002","kind":"supports"},{"source_id":"S053","target_id":"S002","kind":"supports"},{"source_id":"S054","target_id":"S002","kind":"supports"},{"source_id":"S055","target_id":"S002","kind":"supports"},{"source_id":"S056","target_id":"S002","kind":"supports"},{"source_id":"S057","target_id":"S002","kind":"supports"},{"source_id":"S058","target_id":"S002","kind":"supports"},{"source_id":"S059","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S001","kind":"supports"}]}}
{"record_id":"1807.09146v3","paper":{"title":"Inexact Variable Metric Stochastic Block-Coordinate Descent for Regularized Optimization","authors":["Ching-pei Lee","Stephen J. Wright"],"arxiv_id":"1807.09146","version":"v3","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-020-01639-4","abstract_url":"https://arxiv.org/abs/1807.09146v3","pdf_url":"https://arxiv.org/pdf/1807.09146v3"},"context":{"raw_tex":"We consider the following regularized minimization problem:\n\\begin{equation}\n\t\\min_{x} \\, F\\left(x\\right) \\coloneqq f\\left(x\\right) +\n\t\\psi\\left(x\\right),\n\t\\label{eq:f}\n\\end{equation}\n\nwhere $f$ is blockwise Lipschitz-continuously differentiable (defined\nbelow) but not necessarily convex, and the regularizer $\\psi$ is\nconvex, extended-valued, proper, closed, and block-separable, but\npossibly nondifferentiable.  We assume $F$ is lower-bounded and denote\nthe solution set by $\\Omega$, which is assumed to be nonempty.  For\nsimplicity, we assume $x \\in \\R^n$, but our methods can be applied to\nmatrix variables too. We decompose $x\\in \\R^n$ into $N$ blocks such\nthat\n\\begin{equation*}\nx = (x_1,x_2,\\dotsc,x_N) \\in \\R^n, \\quad\nx_i \\in \\R^{n_i},\\quad n_i \\in \\N, \\quad \\sum_{i=1}^N\nn_i = n,\n\\end{equation*}\n\nand assume throughout that the function $\\psi$ can be decomposed as\n\\[\n\\psi(x) = \\sum_{i=1}^N \\psi_i(x_i),\n\\]\n\nwhere all $\\psi_i$ have the properties claimed for $\\psi$ above.  Many\n\n\nFor the block-separability of $x$, we use the column submatrices of\nthe identity denoted by $U_1,U_2,\\dotsc, U_N$, where {$U_i \\in \\R^{n\n\\times n_i}$} corresponds to\nthe indices in the $i$th block of $x$. Thus, we have\n\\begin{equation*}\n\tx_i = U_i^\\top x, \\quad x = \\sum_{i=1}^N U_i x_i, \\quad \\mbox{and} \\;\n\\nabla_i f = U_i^\\top \\nabla f.\n\\end{equation*}\n\nThe blockwise Lipschitz-continuously differentiable property is that\nthere exist constants $L_i > 0$, $i=1,2,\\dotsc,N$, such\nthat\\footnote{{We use the Euclidean norm throughout the\npaper.}}\n\\begin{equation}\n\\left\\|\\nabla_i f(x + U_i h) - \\nabla_i f(x)\\right\\| \\leq L_i\n\\|h\\|,\\quad \\forall h \\in \\R^{n_i},\\quad \\forall x \\in \\R^n.\n\\label{eq:compL}\n\\end{equation}\n\nThe $k$th iteration of the ``exact'' version of our approach proceeds\nas follows.  Given the current iterate $x^k$, we pick a block $i$,\naccording to some discrete probability distribution over\n$\\{1,\\dotsc,N\\}$, and minimize a quadratic approximation of $f$ plus\nthe function $\\psi_i$ for that block, to obtain the update\ndirection $d^k_i$. That is, we have\n\\begin{equation}\n\\label{eq:quadratic}\nd^{k*}_i \\coloneqq \\arg\\min_{d_i\\in \\R^{n_i}} \\, Q^k_{i}( d_i),\n\\end{equation}\n\nwhere\n\\begin{align}\n\\label{eq:Qdef}\nQ^{k}_{i}( d_i ) \\coloneqq \\nabla_i f\\left( x^k\n\\right)^\\top d_i + \\tfrac12 d_i^\\top H^k_i d_i\n+ \\psi_i\\left( x^k_i + d_i\n\\right) - \\psi_i\\left( x^k_i \\right),\n\\end{align}\n\nand $H^k_i\\in\\R^{n_i\\times n_i}$ is some positive-definite matrix that\ncan change over iterations.  A backtracking line search along $d^k_i$\nis then performed to determine the step.\n\n\nWe assume that $d^k_i$ is an $\\eta$-approximate solution to\n\\eqref{eq:quadratic}, for some $\\eta \\in [0,1)$ fixed over all $k$ and\n  all $i$, satisfying the following condition:\n\\begin{equation}\n{-\\eta Q^{k*}_i = }\\eta \\left(Q^k_i\\left(0\\right) -  Q^{k*}_i \\right) \\geq\nQ^k_i\\left(d^k_i\\right) - Q^{k*}_i,\n\\label{eq:approx}\n\\end{equation}\n\nwhere $Q^{k*}_i \\coloneqq \\inf_{d_i} Q^k_i(d_i) {=\n  Q^k_i(d^{k*}_i)}$. Note that the setting $\\eta=0$ corresponds to the\n\n\nadded to $H^k_{i_k}$. After finding an approximate solution $d^k_{i_k}$ to\n\\eqref{eq:quadratic} that satisfies \\eqref{eq:approx} for some $\\eta\n\\in [0,1)$, we conduct a backtracking line search, as in\n  \\cite{TseY09a}: Given $\\beta, \\gamma \\in (0,1)$, we let\n  $\\alpha^k_{i_k}$ be the largest value in $\\{1,\\beta^1,\\beta^2,\n  \\dotsc\\}$ such that the following sufficient decrease condition is\n  satisfied:\n\\begin{equation}\n\tF\\left(x^k + \\alpha^k_{i_k} U_{i_k} d_{i_k}^k \\right) \\leq F\\left(\n\tx^k \\right) +\n\t\\alpha_{i_k}^k \\gamma \\Delta^k_{i_k},\n\\label{eq:armijo}\n\\end{equation}\n\nwhere\n\\begin{equation}\n\\Delta^k_i \\coloneqq \\nabla_i f\\left( x^k \\right)^\\top d^k_i + \\psi_{i}\\left(\nx^k_{i} + d^k_i \\right) - \\psi_{i}\\left( x^k_i \\right).\n\\label{eq:Delta}\n\\end{equation}\n\nThen the iterate is updated to $x^k + \\alpha^k_{i_k} U_{i_k} d_{i_k}^k$.\n\n\n\\begin{algorithm}[tb]\n\\caption{Inexact variable-metric block-coordinate descent for\n\t\\eqref{eq:f}}\n\\label{alg:icd}\n\\begin{algorithmic}[1]\n\\STATE Given $\\beta, \\gamma \\in (0,1)$, $\\eta \\in [0,1)$, and $x^0\n\t\\in \\R^n$;\n\\FOR{$k=0,1,2,\\dotsc$}\n\t\\STATE Pick a probability distribution\n\t{$p^k_1,\\dotsc,p^k_N > 0$,\n\t\t$\\sum_{i} p^k_i = 1$}, and sample {${i_k}$} accordingly;\n\t\\STATE Compute $\\nabla_{i_k} f(x^k)$ and choose a\n\tpositive-definite $H^k_{i_k}$;\n\t\\STATE Approximately solve \\eqref{eq:quadratic} {for $i=i_k$}  to obtain a\n\tsolution $d^k_{i_k}$ satisfying \\eqref{eq:approx};\n\t\\STATE Compute $\\Delta_{i_k}$ by \\eqref{eq:Delta}, {with\n\t$i=i_k$}; Set {$\\alpha_{i_k}^k \\leftarrow 1$};\n\t\\WHILE{\\eqref{eq:armijo} is not satisfied}\n\t\\STATE $\\alpha^k_{i_k} \\leftarrow \\beta \\alpha^k_{i_k}$;\n\t\\ENDWHILE\n\t\\STATE $x^{k+1} \\leftarrow x^k + \\alpha^k_{i_k} U_{i_k} d^k_{i_k}$;\n\\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}\n\n  results of this section we often focus on a particular iteration\n  $k$, but rather than considering the consequences of updating the\n  chosen block $i_k$ at that iteration, we examine what would happen\n  for {\\em all possible} choices of $i=1,2,\\dotsc,N$, if each of these\n  values happened to be chosen as $i_k$. Since the actual update block\n  $i_k$ is chosen randomly from among these $N$ possibilities, we\n  obtain results about the expected change in $F$ by taking {\\em\n    expectations} over all these hypothetical choices.}\n\n\n\\begin{definition}\nGiven any function $F$ whose minimum value $F^*$ is attainable,\nand for any $x$, define $P_{\\Omega}(x)$ to be the\n{(Euclidean-norm)} projection of $x$ onto the optimal set\n$\\Omega$.\nWe say that $F$ satisfies the {\\em optimal set strong convexity} {(OSSC)}\ncondition with parameter $\\mu \\geq 0$, if for any $x$ and any\n$\\lambda \\in [0,1]$, the following holds.\n\\begin{align}\n  \\nonumber\nF(\\lambda x + & (1 - \\lambda )\nP_{\\Omega}(x)) \\\\\n\\label{eq:strong}\n& \\leq \\lambda F\\left(x\\right) + \\left(1 - \\lambda\\right) F^* -\n\\tfrac12 \\mu \\lambda \\left( 1 - \\lambda \\right)  \\left\\|x -\nP_{\\Omega}\\left(x\\right)\\right\\|^2.\n\\end{align}\n\\end{definition}\n\nGiven any $x^0$, we define\n\\begin{equation}\nR_0 \\coloneqq \\sup_{x: {F( x ) \\leq F( x^0)}}\\quad\\left\\|x - P_\\Omega(x)\\right\\|.\n\\label{eq:r0}\n\\end{equation}\n\nFor the case of general convex problems, we make the assumption that\nfor any $x^0$, the value of $R_0$ defined in \\eqref{eq:r0} is\nfinite.\n","units":[{"start":3568,"end":3751,"start_line":116,"end_line":121,"raw_tex":"We consider the following regularized minimization problem:\n\\begin{equation}\n\t\\min_{x} \\, F\\left(x\\right) \\coloneqq f\\left(x\\right) +\n\t\\psi\\left(x\\right),\n\t\\label{eq:f}\n\\end{equation}","sha256_utf8":"8625915d66d51908bf246990cb9a0d9a9eb7810f1e549fab8e2efa6446c99b09"},{"start":3752,"end":4372,"start_line":122,"end_line":134,"raw_tex":"where $f$ is blockwise Lipschitz-continuously differentiable (defined\nbelow) but not necessarily convex, and the regularizer $\\psi$ is\nconvex, extended-valued, proper, closed, and block-separable, but\npossibly nondifferentiable.  We assume $F$ is lower-bounded and denote\nthe solution set by $\\Omega$, which is assumed to be nonempty.  For\nsimplicity, we assume $x \\in \\R^n$, but our methods can be applied to\nmatrix variables too. We decompose $x\\in \\R^n$ into $N$ blocks such\nthat\n\\begin{equation*}\nx = (x_1,x_2,\\dotsc,x_N) \\in \\R^n, \\quad\nx_i \\in \\R^{n_i},\\quad n_i \\in \\N, \\quad \\sum_{i=1}^N\nn_i = n,\n\\end{equation*}","sha256_utf8":"5b5b6b35a685838cd10111b361122967c009eaaac489848c107bf9e0f62e1078"},{"start":4373,"end":4482,"start_line":135,"end_line":138,"raw_tex":"and assume throughout that the function $\\psi$ can be decomposed as\n\\[\n\\psi(x) = \\sum_{i=1}^N \\psi_i(x_i),\n\\]","sha256_utf8":"ba99058363cc6eec58184573448a27423597e232ed187ba37c6b9194c3c342f7"},{"start":4483,"end":4554,"start_line":139,"end_line":139,"raw_tex":"where all $\\psi_i$ have the properties claimed for $\\psi$ above.  Many\n","sha256_utf8":"fe204f3f0e6aeb4e2635cabe1ec553e838eea25704eedb27bd6120a075ec5421"},{"start":4754,"end":5109,"start_line":143,"end_line":150,"raw_tex":"For the block-separability of $x$, we use the column submatrices of\nthe identity denoted by $U_1,U_2,\\dotsc, U_N$, where {$U_i \\in \\R^{n\n\\times n_i}$} corresponds to\nthe indices in the $i$th block of $x$. Thus, we have\n\\begin{equation*}\n\tx_i = U_i^\\top x, \\quad x = \\sum_{i=1}^N U_i x_i, \\quad \\mbox{and} \\;\n\\nabla_i f = U_i^\\top \\nabla f.\n\\end{equation*}","sha256_utf8":"d747a64894502060357b3116fb1d18cb7bc89346454649451cff8786fe480b43"},{"start":5110,"end":5471,"start_line":151,"end_line":159,"raw_tex":"The blockwise Lipschitz-continuously differentiable property is that\nthere exist constants $L_i > 0$, $i=1,2,\\dotsc,N$, such\nthat\\footnote{{We use the Euclidean norm throughout the\npaper.}}\n\\begin{equation}\n\\left\\|\\nabla_i f(x + U_i h) - \\nabla_i f(x)\\right\\| \\leq L_i\n\\|h\\|,\\quad \\forall h \\in \\R^{n_i},\\quad \\forall x \\in \\R^n.\n\\label{eq:compL}\n\\end{equation}","sha256_utf8":"3208e1cf432e9cea20ae75ae6ba9b62d9195c798a12df72bb1b45dce33f62db4"},{"start":5909,"end":6383,"start_line":169,"end_line":178,"raw_tex":"The $k$th iteration of the ``exact'' version of our approach proceeds\nas follows.  Given the current iterate $x^k$, we pick a block $i$,\naccording to some discrete probability distribution over\n$\\{1,\\dotsc,N\\}$, and minimize a quadratic approximation of $f$ plus\nthe function $\\psi_i$ for that block, to obtain the update\ndirection $d^k_i$. That is, we have\n\\begin{equation}\n\\label{eq:quadratic}\nd^{k*}_i \\coloneqq \\arg\\min_{d_i\\in \\R^{n_i}} \\, Q^k_{i}( d_i),\n\\end{equation}","sha256_utf8":"06a294b317ce9280b832782885e4a3c1026e4113845b474d0683b1efdb7bf649"},{"start":6384,"end":6591,"start_line":179,"end_line":186,"raw_tex":"where\n\\begin{align}\n\\label{eq:Qdef}\nQ^{k}_{i}( d_i ) \\coloneqq \\nabla_i f\\left( x^k\n\\right)^\\top d_i + \\tfrac12 d_i^\\top H^k_i d_i\n+ \\psi_i\\left( x^k_i + d_i\n\\right) - \\psi_i\\left( x^k_i \\right),\n\\end{align}","sha256_utf8":"b9ba5d9cc8bac599c8451201f569ddf103dcb6296a7b53ff58a7bdf33d8d7eda"},{"start":6592,"end":6774,"start_line":187,"end_line":189,"raw_tex":"and $H^k_i\\in\\R^{n_i\\times n_i}$ is some positive-definite matrix that\ncan change over iterations.  A backtracking line search along $d^k_i$\nis then performed to determine the step.\n","sha256_utf8":"36959388ed304be2f25456bb0011b17e7e6a26c2e4dbb45b068c1d8db4c7df44"},{"start":11750,"end":12088,"start_line":281,"end_line":288,"raw_tex":"We assume that $d^k_i$ is an $\\eta$-approximate solution to\n\\eqref{eq:quadratic}, for some $\\eta \\in [0,1)$ fixed over all $k$ and\n  all $i$, satisfying the following condition:\n\\begin{equation}\n{-\\eta Q^{k*}_i = }\\eta \\left(Q^k_i\\left(0\\right) -  Q^{k*}_i \\right) \\geq\nQ^k_i\\left(d^k_i\\right) - Q^{k*}_i,\n\\label{eq:approx}\n\\end{equation}","sha256_utf8":"e2841906f5780517c3ca87826354e0fd3bc34d1ec26896f625063fd0584cef61"},{"start":12089,"end":12211,"start_line":289,"end_line":290,"raw_tex":"where $Q^{k*}_i \\coloneqq \\inf_{d_i} Q^k_i(d_i) {=\n  Q^k_i(d^{k*}_i)}$. Note that the setting $\\eta=0$ corresponds to the\n","sha256_utf8":"aed6d93525c1dfe1b52226701a419f720860467ee5e111afe2d4c993e78b7200"},{"start":13967,"end":14544,"start_line":319,"end_line":331,"raw_tex":"added to $H^k_{i_k}$. After finding an approximate solution $d^k_{i_k}$ to\n\\eqref{eq:quadratic} that satisfies \\eqref{eq:approx} for some $\\eta\n\\in [0,1)$, we conduct a backtracking line search, as in\n  \\cite{TseY09a}: Given $\\beta, \\gamma \\in (0,1)$, we let\n  $\\alpha^k_{i_k}$ be the largest value in $\\{1,\\beta^1,\\beta^2,\n  \\dotsc\\}$ such that the following sufficient decrease condition is\n  satisfied:\n\\begin{equation}\n\tF\\left(x^k + \\alpha^k_{i_k} U_{i_k} d_{i_k}^k \\right) \\leq F\\left(\n\tx^k \\right) +\n\t\\alpha_{i_k}^k \\gamma \\Delta^k_{i_k},\n\\label{eq:armijo}\n\\end{equation}","sha256_utf8":"ed2ac4fc1ab7a199789cb94d19f284c3ae341a5090bf1ca55c69d5915b499e2f"},{"start":14545,"end":14733,"start_line":332,"end_line":337,"raw_tex":"where\n\\begin{equation}\n\\Delta^k_i \\coloneqq \\nabla_i f\\left( x^k \\right)^\\top d^k_i + \\psi_{i}\\left(\nx^k_{i} + d^k_i \\right) - \\psi_{i}\\left( x^k_i \\right).\n\\label{eq:Delta}\n\\end{equation}","sha256_utf8":"55b6042aaf834abf7690713def923cccec99e63426690fdf6bd9ef3c7e594ea7"},{"start":14734,"end":14807,"start_line":338,"end_line":338,"raw_tex":"Then the iterate is updated to $x^k + \\alpha^k_{i_k} U_{i_k} d_{i_k}^k$.\n","sha256_utf8":"2182a84bd74fdad691748a3f2cb394f1ade262404dca317b39fa07943ab5dc11"},{"start":14808,"end":15715,"start_line":340,"end_line":363,"raw_tex":"\\begin{algorithm}[tb]\n\\caption{Inexact variable-metric block-coordinate descent for\n\t\\eqref{eq:f}}\n\\label{alg:icd}\n\\begin{algorithmic}[1]\n\\STATE Given $\\beta, \\gamma \\in (0,1)$, $\\eta \\in [0,1)$, and $x^0\n\t\\in \\R^n$;\n\\FOR{$k=0,1,2,\\dotsc$}\n\t\\STATE Pick a probability distribution\n\t{$p^k_1,\\dotsc,p^k_N > 0$,\n\t\t$\\sum_{i} p^k_i = 1$}, and sample {${i_k}$} accordingly;\n\t\\STATE Compute $\\nabla_{i_k} f(x^k)$ and choose a\n\tpositive-definite $H^k_{i_k}$;\n\t\\STATE Approximately solve \\eqref{eq:quadratic} {for $i=i_k$}  to obtain a\n\tsolution $d^k_{i_k}$ satisfying \\eqref{eq:approx};\n\t\\STATE Compute $\\Delta_{i_k}$ by \\eqref{eq:Delta}, {with\n\t$i=i_k$}; Set {$\\alpha_{i_k}^k \\leftarrow 1$};\n\t\\WHILE{\\eqref{eq:armijo} is not satisfied}\n\t\\STATE $\\alpha^k_{i_k} \\leftarrow \\beta \\alpha^k_{i_k}$;\n\t\\ENDWHILE\n\t\\STATE $x^{k+1} \\leftarrow x^k + \\alpha^k_{i_k} U_{i_k} d^k_{i_k}$;\n\\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"9ce5ffe22d0302e66a63990b20c2d65fe32bf08675f19e94b982c3037a0b081a"},{"start":16032,"end":16565,"start_line":372,"end_line":379,"raw_tex":"  results of this section we often focus on a particular iteration\n  $k$, but rather than considering the consequences of updating the\n  chosen block $i_k$ at that iteration, we examine what would happen\n  for {\\em all possible} choices of $i=1,2,\\dotsc,N$, if each of these\n  values happened to be chosen as $i_k$. Since the actual update block\n  $i_k$ is chosen randomly from among these $N$ possibilities, we\n  obtain results about the expected change in $F$ by taking {\\em\n    expectations} over all these hypothetical choices.}\n","sha256_utf8":"08440d65ee79893163d9c6c473b27247455b0865fcee3a86f288492d13841029"},{"start":18219,"end":18879,"start_line":421,"end_line":438,"raw_tex":"\\begin{definition}\nGiven any function $F$ whose minimum value $F^*$ is attainable,\nand for any $x$, define $P_{\\Omega}(x)$ to be the\n{(Euclidean-norm)} projection of $x$ onto the optimal set\n$\\Omega$.\nWe say that $F$ satisfies the {\\em optimal set strong convexity} {(OSSC)}\ncondition with parameter $\\mu \\geq 0$, if for any $x$ and any\n$\\lambda \\in [0,1]$, the following holds.\n\\begin{align}\n  \\nonumber\nF(\\lambda x + & (1 - \\lambda )\nP_{\\Omega}(x)) \\\\\n\\label{eq:strong}\n& \\leq \\lambda F\\left(x\\right) + \\left(1 - \\lambda\\right) F^* -\n\\tfrac12 \\mu \\lambda \\left( 1 - \\lambda \\right)  \\left\\|x -\nP_{\\Omega}\\left(x\\right)\\right\\|^2.\n\\end{align}\n\\end{definition}","sha256_utf8":"903d25e498024e4e0da0001acd4287001b2ab77bb7a23dc6f7d8ef7760d6bd63"},{"start":24552,"end":24706,"start_line":573,"end_line":577,"raw_tex":"Given any $x^0$, we define\n\\begin{equation}\nR_0 \\coloneqq \\sup_{x: {F( x ) \\leq F( x^0)}}\\quad\\left\\|x - P_\\Omega(x)\\right\\|.\n\\label{eq:r0}\n\\end{equation}","sha256_utf8":"9b93a1e9cbdf94481ed474d2fcf9337586af4201543b33d79a8055f642fdfbd7"},{"start":24707,"end":24846,"start_line":578,"end_line":580,"raw_tex":"For the case of general convex problems, we make the assumption that\nfor any $x^0$, the value of $R_0$ defined in \\eqref{eq:r0} is\nfinite.\n","sha256_utf8":"007b2e5e75ba66a164b90c637261a1b9206bb922be239eb8ad88a0d6de2ab153"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\n\\label{thm:sub}\nAssume that $f$ and $\\psi$ are convex and \\eqref{eq:compL} holds.\nSuppose that at all iterations $k$ of Algorithm~\\ref{alg:icd},\n{and for any choice $i=i_k$ of the update block at iteration\n  $k$, we have that \\eqref{eq:approx} is satisfied with a fixed $\\eta\n  \\in [0,1)$, with $H^k_i$ chosen such that}\n\\begin{equation}\nH^k_i \\succeq m_i I,\\quad k=0,1,\\dotsc,\n\\label{eq:H}\n\\end{equation}\nfor some $ m_i > 0$ for all $i$. Then the following are true.\n\\begin{enumerate}\n\\item At iteration $k$, given any probability distribution\n  $\\{p^k_i\\}_{i=1}^N > 0$ for choosing the update block $i_k$, denote\n  by $\\{\\alpha^k_i\\}_{i=1}^N > 0$ the step sizes generated by the\n  backtracking line search for each possible choice\n  $i=1,2,\\dotsc,N$. (These step sizes are guaranteed to be bounded\n  away from zero, by Lemma~\\ref{lemma:linesearch}.) Define\n\\begin{equation} \\label{eq:pik}\n\t\\pi^k \\coloneqq \\min_{1 \\le i \\le N} \\alpha^k_i p^k_i,\n\\end{equation}\n{and let $\\cP_k$, $\\AA_k$, and $\\HH_k$ be defined as in\nLemma~\\ref{lemma:Q}.} If\n\\[\n  F\\left( x^k \\right) - F^* \\geq \\left(x^k -\n  P_\\Omega\\left(x^k\\right)\\right)^\\top  \\cP^{-1}_k \\AA_k^{-1}\n  \\HH_k\\left(x^k - P_\\Omega\\left(x^k\\right)\\right) \\pi^k,\n  \\]\n  {then the expected improvement in objective optimality gap\n    at this iteration is bounded away from $1$, as follows:}\n\\begin{align}\n\\frac{\\E_{i_k}\\left[F\\left(x^{k+1}\\right) -\nF^*\\right|\\left.x^k\\right]}{\\left(F\\left( x^k \\right) - F^*\\right)}\n\\leq \\left( 1 - \\frac{\\left( 1 - \\eta \\right) \\gamma \\pi^k }{2}\\right) .\n\\label{eq:earlylinear}\n\\end{align}\n\\item {Given $M_i \\geq m_i, i=1,\\dotsc,N,$}\nand define\n\\begin{equation} \\label{eq:def.M}\n\\cM \\coloneqq \\diag(M_1 I_{n_1},\\dotsc, M_N I_{n_N}),\\quad \\bar \\AA\n\\coloneqq \\diag(\\bar{\\alpha}_1 I_{n_1},\\dotsc, \\bar{\\alpha}_N\nI_{n_N}),\n\\end{equation}\n where $\\bar\n\\alpha_i$ are defined in Lemma \\ref{lemma:linesearch}.  Given a\nprobability distribution $\\{p_i\\}_{i=1}^N > 0$, define\n\\begin{equation} \\label{eq:barpi}\n\\cP \\coloneqq \\diag(p_1 I_{n_1},\\dotsc, p_N I_{n_N}),\\quad\n  \\bar\\pi\n  \\coloneqq \\min_{1 \\le i \\le N} \\, \\bar\\alpha_i p_i,\n\\end{equation}\nand let\n\\begin{equation}\n\tk_0 \\coloneqq \\arg \\min\\left\\{ k: F\\left( x^k \\right) - F^* <\n\t\t\\|\\cP^{-1} \\bar{\\AA}^{-1} \\cM\\| \\bar\\pi R_0^2\\right\\}.\n\\label{eq:k0}\n\\end{equation}\n{Suppose that for all $k \\geq k_0$, the sampling of $i_k$\n  follows the distribution $\\{p_i\\}$, which does not depend on $k$,\n  and }\n\\begin{equation}\nM_i I \\succeq H^k_i \\succeq m_i I,\\quad i=1,\\dotsc,N.\n\\label{eq:H2}\n\\end{equation}\nThen for $k \\geq k_0$, the expected objective follows a sublinear\nconvergence rate, as follows:\n\\begin{align}\n\t\\E_{i_{k_0},i_{k_0+1},\\dotsc,i_{k-1}}\\left[F\\left(x^k\\right)\\right|\\left.\nx^{k_0}\\right] - F^* \\leq \\frac{2 \\|\\cP^{-1} \\bar \\AA^{-1} \\cM \\| R_0^2\n}{2N + (1 - \\eta)\\gamma (k - k_0)}.\n\\label{eq:sub}\n\\end{align}\n\n\\item Suppose that a fixed probability distribution $\\{p_i\\}_{i=1}^N >\n  0$ is used throughout to sample the blocks and that \\eqref{eq:H2}\n  holds for all $k$. {Then, defining}\n  \\begin{equation}\n    \\bar k_0 \\coloneqq \\left\\lceil\\max\\left\\{ 0, \\frac{\\log\n      \\frac{F\\left( x^0 \\right) - F^*}{\\left\\| \\cP^{-1}\n\t\\bar{\\AA}^{-1} \\cM\\right\\| \\bar\\pi\n\tR_0^2}}{\\log\\left( \\frac{2}{2 - \\left( 1 - \\eta\\right) \\gamma\n\t\\bar\\pi}\\right)}\\right\\}\\right\\rceil,\n    \\label{eq:k0bar}\n  \\end{equation}\n  (with $\\bar\\pi$ defined in \\eqref{eq:barpi}), {we have for\n    all $k<\\bar k_0$ that the expected objective satisfies}\n  \\begin{equation}\n    \\E_{i_0,\\dotsc,i_{k-1}}\\left[ F\\left( x^k \\right) - F^*\n\t\t\\right] \\leq \\left( 1 - \\frac{(1 - \\eta)\\gamma\n      \\bar\\pi}{2} \\right)^k \\left( F\\left( x^0\n    \\right) - F^* \\right),\n    \\label{eq:upper1}\n  \\end{equation}\n  {while for all $k \\geq \\bar k_0$, we have}\n  \\begin{align}\n\t  \\E_{i_0,\\dotsc,i_{k-1}}\\left[F\\left(x^k\\right) -\n\t\t  F^* \\right] \\leq \\frac{2 \\left\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\n\t\t  \\right\\| R_0^2 }{2N + (1 - \\eta)\\gamma (k - \\bar k_0)}.\n\\label{eq:upper2}\n\\end{align}\n\\end{enumerate}\n\\end{theorem}","units":[{"start":25653,"end":29663,"start_line":594,"end_line":696,"raw_tex":"\\begin{theorem}\n\\label{thm:sub}\nAssume that $f$ and $\\psi$ are convex and \\eqref{eq:compL} holds.\nSuppose that at all iterations $k$ of Algorithm~\\ref{alg:icd},\n{and for any choice $i=i_k$ of the update block at iteration\n  $k$, we have that \\eqref{eq:approx} is satisfied with a fixed $\\eta\n  \\in [0,1)$, with $H^k_i$ chosen such that}\n\\begin{equation}\nH^k_i \\succeq m_i I,\\quad k=0,1,\\dotsc,\n\\label{eq:H}\n\\end{equation}\nfor some $ m_i > 0$ for all $i$. Then the following are true.\n\\begin{enumerate}\n\\item At iteration $k$, given any probability distribution\n  $\\{p^k_i\\}_{i=1}^N > 0$ for choosing the update block $i_k$, denote\n  by $\\{\\alpha^k_i\\}_{i=1}^N > 0$ the step sizes generated by the\n  backtracking line search for each possible choice\n  $i=1,2,\\dotsc,N$. (These step sizes are guaranteed to be bounded\n  away from zero, by Lemma~\\ref{lemma:linesearch}.) Define\n\\begin{equation} \\label{eq:pik}\n\t\\pi^k \\coloneqq \\min_{1 \\le i \\le N} \\alpha^k_i p^k_i,\n\\end{equation}\n{and let $\\cP_k$, $\\AA_k$, and $\\HH_k$ be defined as in\nLemma~\\ref{lemma:Q}.} If\n\\[\n  F\\left( x^k \\right) - F^* \\geq \\left(x^k -\n  P_\\Omega\\left(x^k\\right)\\right)^\\top  \\cP^{-1}_k \\AA_k^{-1}\n  \\HH_k\\left(x^k - P_\\Omega\\left(x^k\\right)\\right) \\pi^k,\n  \\]\n  {then the expected improvement in objective optimality gap\n    at this iteration is bounded away from $1$, as follows:}\n\\begin{align}\n\\frac{\\E_{i_k}\\left[F\\left(x^{k+1}\\right) -\nF^*\\right|\\left.x^k\\right]}{\\left(F\\left( x^k \\right) - F^*\\right)}\n\\leq \\left( 1 - \\frac{\\left( 1 - \\eta \\right) \\gamma \\pi^k }{2}\\right) .\n\\label{eq:earlylinear}\n\\end{align}\n\\item {Given $M_i \\geq m_i, i=1,\\dotsc,N,$}\nand define\n\\begin{equation} \\label{eq:def.M}\n\\cM \\coloneqq \\diag(M_1 I_{n_1},\\dotsc, M_N I_{n_N}),\\quad \\bar \\AA\n\\coloneqq \\diag(\\bar{\\alpha}_1 I_{n_1},\\dotsc, \\bar{\\alpha}_N\nI_{n_N}),\n\\end{equation}\n where $\\bar\n\\alpha_i$ are defined in Lemma \\ref{lemma:linesearch}.  Given a\nprobability distribution $\\{p_i\\}_{i=1}^N > 0$, define\n\\begin{equation} \\label{eq:barpi}\n\\cP \\coloneqq \\diag(p_1 I_{n_1},\\dotsc, p_N I_{n_N}),\\quad\n  \\bar\\pi\n  \\coloneqq \\min_{1 \\le i \\le N} \\, \\bar\\alpha_i p_i,\n\\end{equation}\nand let\n\\begin{equation}\n\tk_0 \\coloneqq \\arg \\min\\left\\{ k: F\\left( x^k \\right) - F^* <\n\t\t\\|\\cP^{-1} \\bar{\\AA}^{-1} \\cM\\| \\bar\\pi R_0^2\\right\\}.\n\\label{eq:k0}\n\\end{equation}\n{Suppose that for all $k \\geq k_0$, the sampling of $i_k$\n  follows the distribution $\\{p_i\\}$, which does not depend on $k$,\n  and }\n\\begin{equation}\nM_i I \\succeq H^k_i \\succeq m_i I,\\quad i=1,\\dotsc,N.\n\\label{eq:H2}\n\\end{equation}\nThen for $k \\geq k_0$, the expected objective follows a sublinear\nconvergence rate, as follows:\n\\begin{align}\n\t\\E_{i_{k_0},i_{k_0+1},\\dotsc,i_{k-1}}\\left[F\\left(x^k\\right)\\right|\\left.\nx^{k_0}\\right] - F^* \\leq \\frac{2 \\|\\cP^{-1} \\bar \\AA^{-1} \\cM \\| R_0^2\n}{2N + (1 - \\eta)\\gamma (k - k_0)}.\n\\label{eq:sub}\n\\end{align}\n\n\\item Suppose that a fixed probability distribution $\\{p_i\\}_{i=1}^N >\n  0$ is used throughout to sample the blocks and that \\eqref{eq:H2}\n  holds for all $k$. {Then, defining}\n  \\begin{equation}\n    \\bar k_0 \\coloneqq \\left\\lceil\\max\\left\\{ 0, \\frac{\\log\n      \\frac{F\\left( x^0 \\right) - F^*}{\\left\\| \\cP^{-1}\n\t\\bar{\\AA}^{-1} \\cM\\right\\| \\bar\\pi\n\tR_0^2}}{\\log\\left( \\frac{2}{2 - \\left( 1 - \\eta\\right) \\gamma\n\t\\bar\\pi}\\right)}\\right\\}\\right\\rceil,\n    \\label{eq:k0bar}\n  \\end{equation}\n  (with $\\bar\\pi$ defined in \\eqref{eq:barpi}), {we have for\n    all $k<\\bar k_0$ that the expected objective satisfies}\n  \\begin{equation}\n    \\E_{i_0,\\dotsc,i_{k-1}}\\left[ F\\left( x^k \\right) - F^*\n\t\t\\right] \\leq \\left( 1 - \\frac{(1 - \\eta)\\gamma\n      \\bar\\pi}{2} \\right)^k \\left( F\\left( x^0\n    \\right) - F^* \\right),\n    \\label{eq:upper1}\n  \\end{equation}\n  {while for all $k \\geq \\bar k_0$, we have}\n  \\begin{align}\n\t  \\E_{i_0,\\dotsc,i_{k-1}}\\left[F\\left(x^k\\right) -\n\t\t  F^* \\right] \\leq \\frac{2 \\left\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\n\t\t  \\right\\| R_0^2 }{2N + (1 - \\eta)\\gamma (k - \\bar k_0)}.\n\\label{eq:upper2}\n\\end{align}\n\\end{enumerate}\n\\end{theorem}","sha256_utf8":"2d98be02851d9bcae68007f0e9999fd2838d96d184cda550afafd676b7554d1f"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\n\\label{lemma:linesearch}\n{At the $k$th iteration, suppose that $H^k_i \\succeq m_i I$ for\n  some $m_i > 0$, for $i=1,2,\\dotsc,N$, and that the subproblem\n    solution $d^k_i$ satisfies \\eqref{eq:approx}, for each\n    $i=1,2,\\dotsc,N$.} Then we have\n\\begin{equation}\n\\Delta^k_i \\leq -\\frac{1}{1 + \\sqrt{\\eta}} \\left(d^k_i\\right)^\\top\nH^k_i d^k_i\n\\leq -\\frac{m_i}{1 + \\sqrt{\\eta}} \\|d_i^k\\|^2.\n\\label{eq:deltabound2}\n\\end{equation}\nMoreover, the backtracking line search procedure in\nAlgorithm~\\ref{alg:icd} terminates finitely, with the step size\n$\\alpha^k_i$ lower bounded by\n\\begin{equation}\n\\alpha^k_i \\geq \\bar\\alpha_i \\coloneqq \\min\\left\\{1, \\frac{2\\beta\\left(1 -\n\\gamma\\right) m_i}{L_i\\left(1 + \\sqrt{\\eta}\\right)}\\right\\}.\n\\label{eq:linesearchbound2}\n\\end{equation}\n\\end{lemma}\n\n\\begin{lemma}\n\\label{lemma:Q}\n{Let $f$ and $\\psi$ be convex with $F$ satisfying\n  \\eqref{eq:strong} for some $\\mu \\geq 0$. At iteration $k$, we\n  consider matrices $H^k_i \\succeq 0$ with $H_i^k \\in \\R^{n_i \\times\n    n_i}$, $i=1,\\dotsc,N$, probability distribution $\\{p_i^k\\}_{i=1}^N\n  > 0$, and step sizes $\\{\\alpha^k_i\\}_{i=1}^N > 0$. We define}\n\\begin{align*}\n\\cP_k &\\coloneqq \\diag(p^k_1 I_{n_1},\\dotsc, p^k_N I_{n_N}),\\quad\n\\AA_k \\coloneqq \\diag(\\alpha_1^k\nI_{n_1},\\dotsc, \\alpha_N^k I_{n_N}),\\\\\n\\HH_k &\\coloneqq \\diag(H_1^k,\\dotsc,H_N^k).\n\\end{align*}\nThen for $Q_i^k$ defined by \\eqref{eq:Qdef}, the following holds\nfor all $\\lambda \\in \\left[0,1\\right]$ and all $\\theta$ such that\n$0\\leq\\theta\\leq \\alpha^k_i p^k_i$, $i=1,\\dotsc,N$:\n\\begin{align}\n\\nonumber\n\\E_i\\left[\\left.\\alpha^k_i Q^{k*}_i \\right| x^k\\right]\n&\\leq \\theta\\lambda \\left(F^* - F\\left(x^k\\right) \\right)- \\tfrac12 \\mu \\theta\n\t\\lambda \\left( 1 - \\lambda \\right) \\left\\|x^k -\nP_\\Omega\\left(x^k\\right)\\right\\|^2 +\\\\\n& \\tfrac12 \\theta^2\\lambda^2 \\left( x^k - P_{\\Omega}\\left( x^k \\right)\n\\right)^\\top  \\cP_k^{-1}\\AA_k^{-1} \\HH_k\\left( x^k - P_{\\Omega}\\left(\nx^k \\right)\n\\right).\n\\label{eq:Q}\n\\end{align}\n\\end{lemma}\n\n\\begin{proof}\nGiven any $d\\in \\R^n$, let $\\tilde d \\coloneqq \\AA_k \\cP_k d \\in\n\\R^n$.  We obtain by change of variables that\n\\begin{align}\n\\nonumber\n&~\\E_i{\\left[\\left.\\alpha_i^k Q_i^{k*} \\right| x^k \\right]} \\\\\n\\nonumber\n=&~ \\min_d \\,\\nabla f \\left( x^k\n\t\\right)^\\top  \\AA_k \\cP_k d + \\tfrac12 d^\\top  \\HH_k \\AA_k \\cP_k d + \\sum_{i=1}^N\n\t\\alpha_i^k p_i^k \\left(\\psi_i\\left(x_i^k + d_i\\right) -\n\t\\psi_i\\left(x_i^k\\right)\\right)\\\\\n\\nonumber\n=&~ \\min_{\\tilde d \\in \\R^n} \\, \\nabla f\\left( x^k \\right)^\\top  \\tilde d + \\tfrac12\n\t\\tilde{d}^\\top  \\cP_k^{-1} \\AA_k^{-1} \\HH_k \\tilde d + \\sum_{i=1}^N\n\t\\alpha_i^k p_i^k \\left( \\psi_i \\left( x^k_i +\n\t\\frac{\\tilde{d}_i}{\\alpha_i^k p_i^k} \\right) - \\psi_i\\left( x_i^k\n\t\\right)\\right)\\\\\n\\nonumber\n\\leq&~ \\min_{\\tilde d \\in \\R^n} \\,\\min_{\\theta\\in[0,1] \\, \\text{s.t.} \\,\n\t\\frac{\\theta}{\\alpha_i^k p_i^k} \\leq 1,\\forall i}\n\t\\nabla f\\left( x^k \\right)^\\top  \\left(\\theta\\tilde d\\right) + \\tfrac12\n\t\\left(\\theta\\tilde{d}\\right)^\\top  \\cP_k^{-1}\\AA_k^{-1}  \\HH_k\n\t\\left(\\theta\\tilde d\\right)\\\\\n\\label{eq:intermediate1}\n\t&\\qquad\\qquad + \\sum_{i=1}^N \\alpha_i^k p_i^k \\left(\n\t\\psi_i \\left( x_i^k + \\frac{\\theta\\tilde{d}_i}{\\alpha_i^k p_i^k} \\right) -\n\t\\psi_i\\left( x_i^k \\right)\\right),\n\\end{align}\n{where each $\\tilde d_i \\in \\R^{n_i}$. In\n  \\eqref{eq:intermediate1}, we used the fact that $\\theta \\tilde {d}$\n  is also a feasible point for the left-hand side, hence its objective\n  value is no smaller than the minimizer.}\n\nNext, from the convexity of $f$, we have\n\\begin{equation*}\n\\nabla f\\left( x^k \\right)^\\top  \\theta \\tilde d = \\theta \\left( \\nabla\nf\\left( x^k \\right)^\\top  \\tilde d\n\\right) \\leq\n\\theta \\left( f\\left( x^k + \\tilde d \\right) - f\\left( x^k \\right)\n\\right),\n\\end{equation*}\nand from  $\\theta / (\\alpha_i^k p_i^k) \\leq 1$ for all $i$ and\nthe convexity of $\\psi$, we obtain\n\\begin{align*}\n\t\\psi_i \\left( x^k_i + \\frac{\\theta \\tilde{d}_i}{\\alpha_i^k p_i^k} \\right)\n\t&\\leq \\left( 1 - \\frac{\\theta }{\\alpha_i^k p_i^k} \\right)\n\t\\psi_i\\left( x_i^k \\right) + \\frac{\\theta}{\\alpha_i^k p_i^k}\n\t\\psi_i\\left( x_i^k + \\tilde{d}_i \\right)\\\\\n\t&= \\frac{\\theta }{\\alpha_i^k p_i^k}\\left( \\psi_i \\left( x_i^k +\n\t\\tilde{d}_i \\right) - \\psi_i\\left( x_i^k \\right) \\right) + \\psi_i \\left(\n\tx_i^k \\right).\n\\end{align*}\nTherefore, we have\n\\begin{align}\n\\nonumber\n&~\\min_{\\tilde d}\\,\\min_{\\theta\\in[0,1] \\, \\text{s.t.} \\,\n\t\\frac{\\theta}{\\alpha_i^k p_i^k} \\leq 1,\\forall i}\n\t\\nabla f\\left( x^k \\right)^\\top  \\left(\\theta\\tilde d\\right) + \\tfrac12\n\t\\left(\\theta\\tilde{d}\\right)^\\top  \\cP_k^{-1}\\AA_k^{-1}  \\HH_k\n\t\\left(\\theta\\tilde d\\right) \\\\\n\t\\nonumber\n\t&\\qquad\\qquad+ \\sum_{i=1}^N \\alpha_i^k p_i^k \\left(\n\t\\psi_i \\left( x^k_i + \\frac{\\theta\\tilde{d}_i}{\\alpha_i^k p^k_i} \\right) -\n\t\\psi_i\\left( x_i^k \\right)\\right)\\\\\n\\nonumber\n\\leq&~ \\min_{\\tilde d} \\, \\min_{\\theta\\in[0,1] \\, \\text{s.t.} \\,\n\t\\frac{\\theta}{\\alpha_i^k p^k_i} \\leq 1,\\forall i}\n\t\\theta\\left(F(x^k+\\tilde d) - F(x^k)\\right)  + \\frac{\\theta^2}{2}\n\t\\tilde{d}^\\top  \\cP_k^{-1}\\AA_k^{-1} \\HH_k \\tilde{d}\\\\\n\t\\nonumber\n\\leq&~ \\min_{\\lambda\\in [0,1]}\\,\\min_{\\theta\\in[0,1] \\, \\text{s.t.} \\,\n\t\\frac{\\theta}{\\alpha^k_i p^k_i} \\leq 1,\\forall i}\n\\theta \\left(F\\left(x^k + \\lambda \\left(\n\tP_{\\Omega}\\left(x^k\\right)  - x^k \\right) \\right) - F\\left( x^k\n\t\\right)\\right) \\\\\n\t&\\qquad\\qquad + \\frac{\\theta^2\n\t\\lambda^2}{2}\\left(P_\\Omega\\left(x^k\\right)  - x^k\\right)^\\top \n\t\\cP_k^{-1}\\AA_k^{-1} \\HH_k \\left(P_\\Omega\\left(x^k\\right) - x^k \\right).\n\\label{eq:intermediate2}\n\\end{align}\nThe result \\eqref{eq:Q} then follows from combining\n\\eqref{eq:intermediate1}, \\eqref{eq:intermediate2}, and\n\\eqref{eq:strong}.  \\qed\n\\end{proof}\n\nBy positive semidefiniteness of $H^k_i$ for all $i$ and all $k$,\n\\eqref{eq:armijo} implies that\n\\begin{align}\n\\nonumber\nF\\left(x^k + \\alpha_{i_k}^k U_{i_k} d_{i_k}^k\\right) - F\\left( x^k \\right)\n&\\leq \\gamma \\alpha_{i_k}^k \\left(\\Delta_{i_k}^k + \\tfrac12\n\\left(d^k_{i_k}\\right)^\\top  H_{i_k}^k d_{i_k}^k\\right) \\\\\n& = \\gamma \\alpha_{i_k}^k Q_{i_k}^k(d_{i_k}^k) \\leq (1 - \\eta)\\gamma \\alpha^k_{i_k}  Q^{k*}_{i_k}.\n\\label{eq:usage}\n\\end{align}\n\n\n\\begin{proof}\n{We first prove Part 1.} Consider Lemma \\ref{lemma:Q}.  For the\n  general convex case, we have $\\mu = 0$ in {the OSSC condition} \\eqref{eq:strong}, so\n  \\eqref{eq:Q} reduces to\n\\begin{align}\n\\label{eq:Q2}\n&~\\E_{i_k}\\left[\\left.\\alpha^k_{i_k} Q^{k*}_{i_k}\\right|\nx^k \\right]\\\\\n\\nonumber\n\\leq&~ \\theta\\lambda \\left(F^* - F\\left(x^k\\right) \\right)+\n\\frac{\\theta^2\\lambda^2}{2} \\left(x^k -\nP_\\Omega\\left(x^k\\right)\\right)^\\top \\cP_k^{-1}\\AA_k^{-1} {\\HH_k} \\left(x^k\n- P_\\Omega\\left(x^k\\right)\\right),\n\\end{align}\nfor all $\\lambda \\in [0,1]$ and all $\\theta \\in [0, \\pi^k]$.\nSetting $\\theta =\n  \\pi^k$, we note that the right-hand side of \\eqref{eq:Q2} is a\n  strongly convex function of $\\lambda$ for $x \\notin \\Omega$, so by\n  minimizing explicitly with respect to $\\lambda$, we obtain\n\\begin{equation}\n\\lambda = \\min\\left\\{ 1, \\frac{F\\left( x^k \\right) - F^* }{ \\left(x^k\n\t- P_\\Omega\\left(x^k\\right)\\right)^\\top \\cP_k^{-1}\\AA_k^{-1}\n\\cH_k \\left(x^k - P_\\Omega\\left(x^k\\right)\\right)\\pi^k} \\right\\}.\n\\label{eq:lambda}\n\\end{equation}\n{With this setting of $\\lambda$, when}\n\\begin{equation*}\n\tF\\left( x^k \\right) - F^* \\geq \\left(x^k -\n\tP_\\Omega\\left(x^k\\right)\\right)^\\top  \\cP_k^{-1}\\AA_k^{-1}\n\t\\cH_k \\left(x^k - P_\\Omega\\left(x^k\\right)\\right) \\pi^k,\n\\end{equation*}\nwe have $\\lambda = 1$ and \\eqref{eq:Q2} becomes\n\\begin{align}\n\t\\E_{i_k}\\left[\\left.\\alpha_{i_k}^k Q^{k*}_{i_k} \\right|\n\tx^k\\right] \\leq \\tfrac12 \\pi^k \\left(F^* - F\\left(x^k\\right)\n\t\\right).\n\\label{eq:Q3}\n\\end{align}\nBy combining \\eqref{eq:Q3} and \\eqref{eq:usage}, we have proved\n\\eqref{eq:earlylinear}.\n\n{Next, we prove Part 2.} Consider \\eqref{eq:Q2} with\n$\\alpha^k_{i_k}$ replaced by $\\bar \\alpha_{i_k}$ (so that $\\AA_k$ is\nreplaced by $\\bar\\AA$) and $p^k_{i_k}$ replaced by $p_{i_k}$ (so that\n$\\cP_k$ is replaced by $\\cP$). For any $k \\geq k_0$, we define\n\\begin{equation*}\n\t\\delta_k \\coloneqq \\E_{i_{k_0},\\dotsc, i_{k-1}} \\left[\\left. F\\left( x^k\n\t\\right) - F^*\\right| x^{k_0}\\right].\n\\end{equation*}\nBy applying the definition \\eqref{eq:r0} and the bound \\eqref{eq:H2}\non the right-hand side of the updated \\eqref{eq:Q2},\nand then taking expectations on both sides over $i_{k_0},\\dotsc,\ni_{k-1}$ conditional on $x^{k_0}$, we have that\n\\begin{align}\n\\label{eq:Q4}\n\\E_{i_{k_0},\\dotsc, i_k}\\left[\\left. \\bar \\alpha_{i_k}\nQ^{k*}_{i_k} \\right| x^{k_0}\\right]\n\\leq -\\theta\\lambda \\delta_k +\n\\frac{\\theta^2\\lambda^2}{2}\\left\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2,\n\\end{align}\nfor all $\\lambda \\in [0,1]$ and all $\\theta \\in [0, \\bar \\pi]$.\nSetting $\\theta = \\bar\\pi$ in \\eqref{eq:Q4}, we have from\n\\eqref{eq:k0} that since Algorithm~\\ref{alg:icd} is a descent method,\n$\\delta_k < \\bar\\pi \\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM \\right\\| R_0^2$,\nfor all $k \\ge k_0$.  Therefore, we can use\n\\begin{equation*}\n\t\\lambda = \\frac{\\delta_k}{\\bar\\pi\n\t\\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\| R_0^2}\n\\end{equation*}\nin \\eqref{eq:Q4} to obtain\n\\begin{align}\n\\nonumber\n\\E_{i_{k_0},\\dotsc,i_k} \\left[ \\left. \\bar \\alpha_{i_k}\nQ^{k*}_{i_k}\\right| x^{k_0}\\right]\n\\leq&~ -\\bar\\pi \\frac{\\delta_k^2\n}{2 \\bar\\pi \\left\\| \\cP^{-1} \\bar\n\\AA^{-1} \\cM \\right\\| R_0^2}\\\\\n=&~ -\\frac{\\delta_k^2}{2 \\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2}.\n\\label{eq:newbound}\n\\end{align}\nTherefore, by taking expectation on \\eqref{eq:usage} over\n$i_{k_0},\\dotsc, i_k$ conditional on $x^{k_0}$, and using\n\\eqref{eq:newbound}, we obtain\n\\begin{equation}\n\t\\delta_{k+1} \\leq \\delta_k - \\frac{\\left( 1 - \\eta\n\t\\right)\\gamma\\delta_k^2}{2\\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2}.\n\\label{eq:deltak}\n\\end{equation}\nBy dividing both sides of \\eqref{eq:deltak} by $\\delta_k \\delta_{k+1}$\nand noting from \\eqref{eq:armijo} and Lemma \\ref{lemma:linesearch}\nthat $\\{F(x_k)\\}$ and therefore $\\{\\delta_k\\}$ is descending, we obtain\n\\begin{equation}\n\\frac{1}{\\delta_{k}}\n\\leq \\frac{1}{\\delta_{k+1}} - \\frac{\\left( 1 - \\eta\n\\right)\\gamma\\delta_k}{2\\delta_{k+1} \\left\\| \\cP^{-1} \\bar \\AA^{-1}\n\\cM\\right\\| R_0^2}\n\\leq \\frac{1}{\\delta_{k+1}} - \\frac{\\left( 1 - \\eta\n\\right)\\gamma}{ 2\\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\| R_0^2}.\n\\label{eq:telescpoe}\n\\end{equation}\nBy summing and telescoping \\eqref{eq:telescpoe}, we obtain\n\\begin{equation}\n\t\\frac{1}{\\delta_k} \\geq \\frac{1}{\\delta_{k_0}} +\n\\left( k - k_0\\right) \\frac{\\left( 1 - \\eta\n\\right)\\gamma }{2\\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\| R_0^2}.\n\\label{eq:summed}\n\\end{equation}\nFinally, note that because $\\bar{\\alpha}_i \\in [0,1]$ for\n$i=1,\\dotsc,N$, \\eqref{eq:k0} implies that\n\\begin{equation}\n\t\\frac{1}{\\delta_{k_0}} \\geq \\frac{1}{\\bar\\pi \\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\| R_0^2}\n\t\\geq\n\t\\frac{1}{\\min_i  p_i \\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2}.\n\\label{eq:deltak0}\n\\end{equation}\nNext, it is straightforward that the solution to\n\t\\[\n\t\t\\min_{p_1,\\dotsc,p_N} \\, \\frac{1}{\\min_{1 \\le i \\le N} p_i}\n\\quad\t\t\\text{subject to}\\;\\; \\sum_{i=1}^N p_i = 1,  \n\t\t\\;\\; p_i \\geq 0, \\;\\; i=1,\\dotsc,N\n\t\\]\nis $p_i \\equiv 1 / N$, and the corresponding objective value is $N$.\nTherefore, \\eqref{eq:deltak0} {further implies} that\n\\begin{equation*}\n\t\\frac{1}{\\delta_{k_0}} \\geq\n\t\\frac{N}{\\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\| R_0^2}.\n\\end{equation*}\nBy combining this inequality with \\eqref{eq:summed}, we obtain\n\\eqref{eq:sub}.\n\n{For Part 3, we again start from \\eqref{eq:Q2} and\n replace $\\alpha^k_{i_k}$ with $\\bar{\\alpha}_{i}$ in\n\\eqref{eq:Q2} to obtain\n\\begin{align}\n\\label{eq:P3Q1}\n\t&~\\E_{i_k}\\left[\\left.\\bar{\\alpha}_{i_k} Q^{k*}_{i_k}\\right|\nx^k \\right]\\\\\n\\nonumber\n\\leq&~ \\theta\\lambda \\left(F^* - F\\left(x^k\\right) \\right)+\n\\frac{\\theta^2\\lambda^2}{2} \\left(x^k -\nP_\\Omega\\left(x^k\\right)\\right)^\\top \\cP^{-1}\\bar\\AA^{-1} \\HH_k \\left(x^k\n- P_\\Omega\\left(x^k\\right)\\right),\n\\end{align}\nfor all $\\lambda \\in [0,1]$ and all $\\theta \\in [0, \\bar \\pi]$.\nBy applying \\eqref{eq:r0} and \\eqref{eq:H2}, we have\n\\begin{align*}\n\t&~\\E_{i_k}\\left[\\left.\\bar{\\alpha}_{i_k} Q^{k*}_{i_k}\\right|\nx^k \\right]\\\\\n\\nonumber\n\\leq&~ \\theta\\lambda \\left(F^* - F\\left(x^k\\right) \\right)+\n\\frac{\\theta^2\\lambda^2}{2} \\left\\|x^k -\nP_\\Omega\\left(x^k\\right)\\right\\| \\|\\cP^{-1}\\bar\\AA^{-1}\n\\cM\\| \\left\\|x^k\n- P_\\Omega\\left(x^k\\right)\\right\\|\\\\\n\\leq&~ \\theta\\lambda \\left(F^* - F\\left(x^k\\right) \\right)+\n\\frac{\\theta^2\\lambda^2}{2} \\|\\cP^{-1}\\bar\\AA^{-1}\n\\cM\\|R_0^2.\n\\end{align*}\nNow we take expectation over $i_0,\\dotsc,i_{k-1}$ on both sides of\nthis inequality (noting that the last term on the right-hand side are\nall constants that do not depend on $i_k$) to obtain\n\\[\n\\E_{i_0,\\dotsc,i_k}\\left[\\bar \\alpha_{i_k}\nQ^{k*}_{i_k}\\right]\n\\leq -\\theta\\lambda \\E_{i_0,\\dotsc,i_{k-1}} \\left[ F^* - F\\left( x^k\n\\right) \\right] +\n\\frac{\\theta^2\\lambda^2}{2}\\left\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2.\n\\]\nBy defining\n\\begin{equation*}\n\t\\hat \\delta_k \\coloneqq \\E_{i_{0},\\dotsc, i_{k-1}} \\left[ F\\left( x^k\n\t\\right) - F^*\\right]\n\\end{equation*}\nand setting $\\theta = \\bar{\\pi}$, we have that\n\\begin{align}\n\\E_{i_0,\\dotsc,i_k}\\left[\\bar \\alpha_{i_k}\nQ^{k*}_{i_k}\\right]\n\\leq -\\bar\\pi\\lambda \\hat \\delta_k +\n\\frac{\\bar \\pi^2\\lambda^2}{2}\\left\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2.\n\\label{eq:P3Q2}\n\\end{align}\nThe minimum of the right-hand side happens when\n\\[\n\t\\lambda = \\min\\left\\{ 1, \\frac{\\hat \\delta_k }{\\bar{\\pi}\\|\\cP^{-1} \\bar\n\t\\AA^{-1} \\cM\\|R_0^2}\\right\\}.\n\\]\nWhen the expected function value satisfies\n\\begin{equation}\n\t\\hat \\delta_k  \\geq \\bar{\\pi}\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\|R_0^2,\n\\label{eq:turning}\n\\end{equation}\nthe minimizer is $\\lambda = 1$, and the bound becomes\n\\begin{align*}\n\\E_{i_0,\\dotsc,i_k}\\left[\\bar \\alpha_{i_k}\nQ^{k*}_{i_k}\\right]\n\\leq -\\bar\\pi\\hat \\delta_k +\n\\frac{\\bar \\pi^2}{2}\\left\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2\n\\leq -\\frac{\\bar \\pi \\hat \\delta_k}{2}.\n\\end{align*}\nNow we consider \\eqref{eq:usage} and take expectation over $i_0,\\dotsc,\ni_k$ on both sides. Note that $Q^{k*}_{i_k} \\leq 0$ so the upper bound is still\nvalid if we replace $\\alpha^k_{i_k}$ with $\\bar \\alpha_{i_k}$.\nThus we obtain\n\\begin{align} \\label{eq:P3Q3}\n\t\\hat \\delta_{k+1} - \\hat{\\delta}_{k} =\n        \\E_{i_0,\\dotsc,i_k}\\left[F\\left( x^{k+1} \\right) - F\\left( x^k\n          \\right)\\right] & \\leq (1 - \\eta) \\gamma \\E_{i_0,\\dotsc,i_k}\n        \\left[ \\bar \\alpha_{i_k} Q^{k*}_{i_k}\\right] \\\\\n        \\nonumber\n        & \\leq -\\frac{ (1 -\n          \\eta) \\gamma\\bar \\pi \\hat \\delta_k}{2}.\n\\end{align}\nBy rearranging the inequality above, we get the linear convergence of\n\\[\n\t\\hat \\delta_{k+1} \\leq \\hat{\\delta}_{k} \\left(1 -\\frac{(1 - \\eta)\n\t\t\\gamma \\bar \\pi}{2}\\right).\n\\]\nTherefore, we always get the bound\n\\[\n\t\\hat \\delta_k \\leq \\left( 1 - \\frac{(1 - \\eta) \\gamma \\bar \\pi\n\t}{2}  \\right)^k \\left( F(x^0) - F^* \\right)\n\\]\nuntil $\\hat \\delta_{k-1} <\\bar{\\pi}\\|\\cP^{-1} \\bar \\AA^{-1}\n\\cM\\|R_0^2$.\nNote that $\\bar k_0$ is obtained as the first value of $k$ such that\n\\[\n\t\\left( 1 - \\frac{(1 - \\eta) \\gamma \\bar{\\pi}}{2} \\right)^k \\left(\n\tF\\left( x^0 \\right) - F^* \\right) \\leq  \\bar{\\pi}\\|\\cP^{-1} \\bar\n\\AA^{-1} \\cM\\|R_0^2.\n\\]\nTherefore, for $k < \\bar k_0$, the upper bound in \\eqref{eq:upper1} is\nlarger than $\\bar{\\pi}\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\|R_0^2$, so the\nrate \\eqref{eq:upper1} remains valid.\nNote that if $\\hat \\delta_k \\leq  \\bar{\\pi}\\|\\cP^{-1} \\bar\n\\AA^{-1} \\cM\\|R_0^2$ has already held true for some $k < \\bar k_0$,\nclearly this bound is still valid.\nOn the other hand,\nafter $\\bar k_0$, we are guaranteed that \\eqref{eq:turning} must stop\nholding.\nThus the minimizer for \\eqref{eq:P3Q2} becomes $\\lambda = \\hat\n\\delta_k / (\\bar{\\pi}\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\|R_0^2)$.\nWe then start from the first inequality of \\eqref{eq:P3Q3}\nand get\n\\[\n\t\\hat{\\delta}_{k+1} - \\hat{\\delta}_k \\leq -\\frac{(1 - \\eta) \\gamma\n\t\t\\hat{\\delta}_k^2}{2 \\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\|R_0^2}.\n\\]\nFollowing the same derivation we had in Part 2, we can get\n\\[\n\t\\frac{1}{ \\hat{\\delta}_k} \\leq \\frac{1}{\\hat{\\delta}_{k+1}} -\n\t\\frac{(1 - \\eta) \\gamma }{2 \\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\|R_0^2}.\n\\]\nBy summing and telescoping the result above, we get\n\\[\n\t\\frac{1}{ \\hat{\\delta}_k} \\geq \\frac{1}{\\hat{\\delta}_{\\bar k_0}} + (k -\n\t\\bar k_0) \\frac{(1 - \\eta) \\gamma }{2 \\|\\cP^{-1} \\bar \\AA^{-1}\n\\cM\\|R_0^2}.\n\\]\nFollowing the same argument in Part 2, we get the final claim in Part\n3.}\n\\qed\n\\end{proof}","units":[{"start":16786,"end":17582,"start_line":385,"end_line":405,"raw_tex":"\\begin{lemma}\n\\label{lemma:linesearch}\n{At the $k$th iteration, suppose that $H^k_i \\succeq m_i I$ for\n  some $m_i > 0$, for $i=1,2,\\dotsc,N$, and that the subproblem\n    solution $d^k_i$ satisfies \\eqref{eq:approx}, for each\n    $i=1,2,\\dotsc,N$.} Then we have\n\\begin{equation}\n\\Delta^k_i \\leq -\\frac{1}{1 + \\sqrt{\\eta}} \\left(d^k_i\\right)^\\top\nH^k_i d^k_i\n\\leq -\\frac{m_i}{1 + \\sqrt{\\eta}} \\|d_i^k\\|^2.\n\\label{eq:deltabound2}\n\\end{equation}\nMoreover, the backtracking line search procedure in\nAlgorithm~\\ref{alg:icd} terminates finitely, with the step size\n$\\alpha^k_i$ lower bounded by\n\\begin{equation}\n\\alpha^k_i \\geq \\bar\\alpha_i \\coloneqq \\min\\left\\{1, \\frac{2\\beta\\left(1 -\n\\gamma\\right) m_i}{L_i\\left(1 + \\sqrt{\\eta}\\right)}\\right\\}.\n\\label{eq:linesearchbound2}\n\\end{equation}\n\\end{lemma}","sha256_utf8":"9b4b0711aa5e9543b512867c2aba6af42623d270edce8726177aaf93b98eb142"},{"start":19171,"end":20356,"start_line":445,"end_line":473,"raw_tex":"\\begin{lemma}\n\\label{lemma:Q}\n{Let $f$ and $\\psi$ be convex with $F$ satisfying\n  \\eqref{eq:strong} for some $\\mu \\geq 0$. At iteration $k$, we\n  consider matrices $H^k_i \\succeq 0$ with $H_i^k \\in \\R^{n_i \\times\n    n_i}$, $i=1,\\dotsc,N$, probability distribution $\\{p_i^k\\}_{i=1}^N\n  > 0$, and step sizes $\\{\\alpha^k_i\\}_{i=1}^N > 0$. We define}\n\\begin{align*}\n\\cP_k &\\coloneqq \\diag(p^k_1 I_{n_1},\\dotsc, p^k_N I_{n_N}),\\quad\n\\AA_k \\coloneqq \\diag(\\alpha_1^k\nI_{n_1},\\dotsc, \\alpha_N^k I_{n_N}),\\\\\n\\HH_k &\\coloneqq \\diag(H_1^k,\\dotsc,H_N^k).\n\\end{align*}\nThen for $Q_i^k$ defined by \\eqref{eq:Qdef}, the following holds\nfor all $\\lambda \\in \\left[0,1\\right]$ and all $\\theta$ such that\n$0\\leq\\theta\\leq \\alpha^k_i p^k_i$, $i=1,\\dotsc,N$:\n\\begin{align}\n\\nonumber\n\\E_i\\left[\\left.\\alpha^k_i Q^{k*}_i \\right| x^k\\right]\n&\\leq \\theta\\lambda \\left(F^* - F\\left(x^k\\right) \\right)- \\tfrac12 \\mu \\theta\n\t\\lambda \\left( 1 - \\lambda \\right) \\left\\|x^k -\nP_\\Omega\\left(x^k\\right)\\right\\|^2 +\\\\\n& \\tfrac12 \\theta^2\\lambda^2 \\left( x^k - P_{\\Omega}\\left( x^k \\right)\n\\right)^\\top  \\cP_k^{-1}\\AA_k^{-1} \\HH_k\\left( x^k - P_{\\Omega}\\left(\nx^k \\right)\n\\right).\n\\label{eq:Q}\n\\end{align}\n\\end{lemma}","sha256_utf8":"d5d23137760b9e97a1824bd449a1ccb1233ea644d6d43566dac6d2e20d0dbe84"},{"start":20357,"end":23964,"start_line":474,"end_line":557,"raw_tex":"\\begin{proof}\nGiven any $d\\in \\R^n$, let $\\tilde d \\coloneqq \\AA_k \\cP_k d \\in\n\\R^n$.  We obtain by change of variables that\n\\begin{align}\n\\nonumber\n&~\\E_i{\\left[\\left.\\alpha_i^k Q_i^{k*} \\right| x^k \\right]} \\\\\n\\nonumber\n=&~ \\min_d \\,\\nabla f \\left( x^k\n\t\\right)^\\top  \\AA_k \\cP_k d + \\tfrac12 d^\\top  \\HH_k \\AA_k \\cP_k d + \\sum_{i=1}^N\n\t\\alpha_i^k p_i^k \\left(\\psi_i\\left(x_i^k + d_i\\right) -\n\t\\psi_i\\left(x_i^k\\right)\\right)\\\\\n\\nonumber\n=&~ \\min_{\\tilde d \\in \\R^n} \\, \\nabla f\\left( x^k \\right)^\\top  \\tilde d + \\tfrac12\n\t\\tilde{d}^\\top  \\cP_k^{-1} \\AA_k^{-1} \\HH_k \\tilde d + \\sum_{i=1}^N\n\t\\alpha_i^k p_i^k \\left( \\psi_i \\left( x^k_i +\n\t\\frac{\\tilde{d}_i}{\\alpha_i^k p_i^k} \\right) - \\psi_i\\left( x_i^k\n\t\\right)\\right)\\\\\n\\nonumber\n\\leq&~ \\min_{\\tilde d \\in \\R^n} \\,\\min_{\\theta\\in[0,1] \\, \\text{s.t.} \\,\n\t\\frac{\\theta}{\\alpha_i^k p_i^k} \\leq 1,\\forall i}\n\t\\nabla f\\left( x^k \\right)^\\top  \\left(\\theta\\tilde d\\right) + \\tfrac12\n\t\\left(\\theta\\tilde{d}\\right)^\\top  \\cP_k^{-1}\\AA_k^{-1}  \\HH_k\n\t\\left(\\theta\\tilde d\\right)\\\\\n\\label{eq:intermediate1}\n\t&\\qquad\\qquad + \\sum_{i=1}^N \\alpha_i^k p_i^k \\left(\n\t\\psi_i \\left( x_i^k + \\frac{\\theta\\tilde{d}_i}{\\alpha_i^k p_i^k} \\right) -\n\t\\psi_i\\left( x_i^k \\right)\\right),\n\\end{align}\n{where each $\\tilde d_i \\in \\R^{n_i}$. In\n  \\eqref{eq:intermediate1}, we used the fact that $\\theta \\tilde {d}$\n  is also a feasible point for the left-hand side, hence its objective\n  value is no smaller than the minimizer.}\n\nNext, from the convexity of $f$, we have\n\\begin{equation*}\n\\nabla f\\left( x^k \\right)^\\top  \\theta \\tilde d = \\theta \\left( \\nabla\nf\\left( x^k \\right)^\\top  \\tilde d\n\\right) \\leq\n\\theta \\left( f\\left( x^k + \\tilde d \\right) - f\\left( x^k \\right)\n\\right),\n\\end{equation*}\nand from  $\\theta / (\\alpha_i^k p_i^k) \\leq 1$ for all $i$ and\nthe convexity of $\\psi$, we obtain\n\\begin{align*}\n\t\\psi_i \\left( x^k_i + \\frac{\\theta \\tilde{d}_i}{\\alpha_i^k p_i^k} \\right)\n\t&\\leq \\left( 1 - \\frac{\\theta }{\\alpha_i^k p_i^k} \\right)\n\t\\psi_i\\left( x_i^k \\right) + \\frac{\\theta}{\\alpha_i^k p_i^k}\n\t\\psi_i\\left( x_i^k + \\tilde{d}_i \\right)\\\\\n\t&= \\frac{\\theta }{\\alpha_i^k p_i^k}\\left( \\psi_i \\left( x_i^k +\n\t\\tilde{d}_i \\right) - \\psi_i\\left( x_i^k \\right) \\right) + \\psi_i \\left(\n\tx_i^k \\right).\n\\end{align*}\nTherefore, we have\n\\begin{align}\n\\nonumber\n&~\\min_{\\tilde d}\\,\\min_{\\theta\\in[0,1] \\, \\text{s.t.} \\,\n\t\\frac{\\theta}{\\alpha_i^k p_i^k} \\leq 1,\\forall i}\n\t\\nabla f\\left( x^k \\right)^\\top  \\left(\\theta\\tilde d\\right) + \\tfrac12\n\t\\left(\\theta\\tilde{d}\\right)^\\top  \\cP_k^{-1}\\AA_k^{-1}  \\HH_k\n\t\\left(\\theta\\tilde d\\right) \\\\\n\t\\nonumber\n\t&\\qquad\\qquad+ \\sum_{i=1}^N \\alpha_i^k p_i^k \\left(\n\t\\psi_i \\left( x^k_i + \\frac{\\theta\\tilde{d}_i}{\\alpha_i^k p^k_i} \\right) -\n\t\\psi_i\\left( x_i^k \\right)\\right)\\\\\n\\nonumber\n\\leq&~ \\min_{\\tilde d} \\, \\min_{\\theta\\in[0,1] \\, \\text{s.t.} \\,\n\t\\frac{\\theta}{\\alpha_i^k p^k_i} \\leq 1,\\forall i}\n\t\\theta\\left(F(x^k+\\tilde d) - F(x^k)\\right)  + \\frac{\\theta^2}{2}\n\t\\tilde{d}^\\top  \\cP_k^{-1}\\AA_k^{-1} \\HH_k \\tilde{d}\\\\\n\t\\nonumber\n\\leq&~ \\min_{\\lambda\\in [0,1]}\\,\\min_{\\theta\\in[0,1] \\, \\text{s.t.} \\,\n\t\\frac{\\theta}{\\alpha^k_i p^k_i} \\leq 1,\\forall i}\n\\theta \\left(F\\left(x^k + \\lambda \\left(\n\tP_{\\Omega}\\left(x^k\\right)  - x^k \\right) \\right) - F\\left( x^k\n\t\\right)\\right) \\\\\n\t&\\qquad\\qquad + \\frac{\\theta^2\n\t\\lambda^2}{2}\\left(P_\\Omega\\left(x^k\\right)  - x^k\\right)^\\top \n\t\\cP_k^{-1}\\AA_k^{-1} \\HH_k \\left(P_\\Omega\\left(x^k\\right) - x^k \\right).\n\\label{eq:intermediate2}\n\\end{align}\nThe result \\eqref{eq:Q} then follows from combining\n\\eqref{eq:intermediate1}, \\eqref{eq:intermediate2}, and\n\\eqref{eq:strong}.  \\qed\n\\end{proof}","sha256_utf8":"7d73e45dbef6cb5df29aa19cfb181395bf4beaa95abd8592a7da92a45ba51cde"},{"start":23966,"end":24408,"start_line":559,"end_line":568,"raw_tex":"By positive semidefiniteness of $H^k_i$ for all $i$ and all $k$,\n\\eqref{eq:armijo} implies that\n\\begin{align}\n\\nonumber\nF\\left(x^k + \\alpha_{i_k}^k U_{i_k} d_{i_k}^k\\right) - F\\left( x^k \\right)\n&\\leq \\gamma \\alpha_{i_k}^k \\left(\\Delta_{i_k}^k + \\tfrac12\n\\left(d^k_{i_k}\\right)^\\top  H_{i_k}^k d_{i_k}^k\\right) \\\\\n& = \\gamma \\alpha_{i_k}^k Q_{i_k}^k(d_{i_k}^k) \\leq (1 - \\eta)\\gamma \\alpha^k_{i_k}  Q^{k*}_{i_k}.\n\\label{eq:usage}\n\\end{align}\n","sha256_utf8":"ac5772f1e18197215cac34d3d25a58f0fde4016284ce69806f42921a157d62c3"},{"start":29664,"end":39843,"start_line":697,"end_line":965,"raw_tex":"\\begin{proof}\n{We first prove Part 1.} Consider Lemma \\ref{lemma:Q}.  For the\n  general convex case, we have $\\mu = 0$ in {the OSSC condition} \\eqref{eq:strong}, so\n  \\eqref{eq:Q} reduces to\n\\begin{align}\n\\label{eq:Q2}\n&~\\E_{i_k}\\left[\\left.\\alpha^k_{i_k} Q^{k*}_{i_k}\\right|\nx^k \\right]\\\\\n\\nonumber\n\\leq&~ \\theta\\lambda \\left(F^* - F\\left(x^k\\right) \\right)+\n\\frac{\\theta^2\\lambda^2}{2} \\left(x^k -\nP_\\Omega\\left(x^k\\right)\\right)^\\top \\cP_k^{-1}\\AA_k^{-1} {\\HH_k} \\left(x^k\n- P_\\Omega\\left(x^k\\right)\\right),\n\\end{align}\nfor all $\\lambda \\in [0,1]$ and all $\\theta \\in [0, \\pi^k]$.\nSetting $\\theta =\n  \\pi^k$, we note that the right-hand side of \\eqref{eq:Q2} is a\n  strongly convex function of $\\lambda$ for $x \\notin \\Omega$, so by\n  minimizing explicitly with respect to $\\lambda$, we obtain\n\\begin{equation}\n\\lambda = \\min\\left\\{ 1, \\frac{F\\left( x^k \\right) - F^* }{ \\left(x^k\n\t- P_\\Omega\\left(x^k\\right)\\right)^\\top \\cP_k^{-1}\\AA_k^{-1}\n\\cH_k \\left(x^k - P_\\Omega\\left(x^k\\right)\\right)\\pi^k} \\right\\}.\n\\label{eq:lambda}\n\\end{equation}\n{With this setting of $\\lambda$, when}\n\\begin{equation*}\n\tF\\left( x^k \\right) - F^* \\geq \\left(x^k -\n\tP_\\Omega\\left(x^k\\right)\\right)^\\top  \\cP_k^{-1}\\AA_k^{-1}\n\t\\cH_k \\left(x^k - P_\\Omega\\left(x^k\\right)\\right) \\pi^k,\n\\end{equation*}\nwe have $\\lambda = 1$ and \\eqref{eq:Q2} becomes\n\\begin{align}\n\t\\E_{i_k}\\left[\\left.\\alpha_{i_k}^k Q^{k*}_{i_k} \\right|\n\tx^k\\right] \\leq \\tfrac12 \\pi^k \\left(F^* - F\\left(x^k\\right)\n\t\\right).\n\\label{eq:Q3}\n\\end{align}\nBy combining \\eqref{eq:Q3} and \\eqref{eq:usage}, we have proved\n\\eqref{eq:earlylinear}.\n\n{Next, we prove Part 2.} Consider \\eqref{eq:Q2} with\n$\\alpha^k_{i_k}$ replaced by $\\bar \\alpha_{i_k}$ (so that $\\AA_k$ is\nreplaced by $\\bar\\AA$) and $p^k_{i_k}$ replaced by $p_{i_k}$ (so that\n$\\cP_k$ is replaced by $\\cP$). For any $k \\geq k_0$, we define\n\\begin{equation*}\n\t\\delta_k \\coloneqq \\E_{i_{k_0},\\dotsc, i_{k-1}} \\left[\\left. F\\left( x^k\n\t\\right) - F^*\\right| x^{k_0}\\right].\n\\end{equation*}\nBy applying the definition \\eqref{eq:r0} and the bound \\eqref{eq:H2}\non the right-hand side of the updated \\eqref{eq:Q2},\nand then taking expectations on both sides over $i_{k_0},\\dotsc,\ni_{k-1}$ conditional on $x^{k_0}$, we have that\n\\begin{align}\n\\label{eq:Q4}\n\\E_{i_{k_0},\\dotsc, i_k}\\left[\\left. \\bar \\alpha_{i_k}\nQ^{k*}_{i_k} \\right| x^{k_0}\\right]\n\\leq -\\theta\\lambda \\delta_k +\n\\frac{\\theta^2\\lambda^2}{2}\\left\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2,\n\\end{align}\nfor all $\\lambda \\in [0,1]$ and all $\\theta \\in [0, \\bar \\pi]$.\nSetting $\\theta = \\bar\\pi$ in \\eqref{eq:Q4}, we have from\n\\eqref{eq:k0} that since Algorithm~\\ref{alg:icd} is a descent method,\n$\\delta_k < \\bar\\pi \\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM \\right\\| R_0^2$,\nfor all $k \\ge k_0$.  Therefore, we can use\n\\begin{equation*}\n\t\\lambda = \\frac{\\delta_k}{\\bar\\pi\n\t\\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\| R_0^2}\n\\end{equation*}\nin \\eqref{eq:Q4} to obtain\n\\begin{align}\n\\nonumber\n\\E_{i_{k_0},\\dotsc,i_k} \\left[ \\left. \\bar \\alpha_{i_k}\nQ^{k*}_{i_k}\\right| x^{k_0}\\right]\n\\leq&~ -\\bar\\pi \\frac{\\delta_k^2\n}{2 \\bar\\pi \\left\\| \\cP^{-1} \\bar\n\\AA^{-1} \\cM \\right\\| R_0^2}\\\\\n=&~ -\\frac{\\delta_k^2}{2 \\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2}.\n\\label{eq:newbound}\n\\end{align}\nTherefore, by taking expectation on \\eqref{eq:usage} over\n$i_{k_0},\\dotsc, i_k$ conditional on $x^{k_0}$, and using\n\\eqref{eq:newbound}, we obtain\n\\begin{equation}\n\t\\delta_{k+1} \\leq \\delta_k - \\frac{\\left( 1 - \\eta\n\t\\right)\\gamma\\delta_k^2}{2\\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2}.\n\\label{eq:deltak}\n\\end{equation}\nBy dividing both sides of \\eqref{eq:deltak} by $\\delta_k \\delta_{k+1}$\nand noting from \\eqref{eq:armijo} and Lemma \\ref{lemma:linesearch}\nthat $\\{F(x_k)\\}$ and therefore $\\{\\delta_k\\}$ is descending, we obtain\n\\begin{equation}\n\\frac{1}{\\delta_{k}}\n\\leq \\frac{1}{\\delta_{k+1}} - \\frac{\\left( 1 - \\eta\n\\right)\\gamma\\delta_k}{2\\delta_{k+1} \\left\\| \\cP^{-1} \\bar \\AA^{-1}\n\\cM\\right\\| R_0^2}\n\\leq \\frac{1}{\\delta_{k+1}} - \\frac{\\left( 1 - \\eta\n\\right)\\gamma}{ 2\\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\| R_0^2}.\n\\label{eq:telescpoe}\n\\end{equation}\nBy summing and telescoping \\eqref{eq:telescpoe}, we obtain\n\\begin{equation}\n\t\\frac{1}{\\delta_k} \\geq \\frac{1}{\\delta_{k_0}} +\n\\left( k - k_0\\right) \\frac{\\left( 1 - \\eta\n\\right)\\gamma }{2\\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\| R_0^2}.\n\\label{eq:summed}\n\\end{equation}\nFinally, note that because $\\bar{\\alpha}_i \\in [0,1]$ for\n$i=1,\\dotsc,N$, \\eqref{eq:k0} implies that\n\\begin{equation}\n\t\\frac{1}{\\delta_{k_0}} \\geq \\frac{1}{\\bar\\pi \\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\| R_0^2}\n\t\\geq\n\t\\frac{1}{\\min_i  p_i \\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2}.\n\\label{eq:deltak0}\n\\end{equation}\nNext, it is straightforward that the solution to\n\t\\[\n\t\t\\min_{p_1,\\dotsc,p_N} \\, \\frac{1}{\\min_{1 \\le i \\le N} p_i}\n\\quad\t\t\\text{subject to}\\;\\; \\sum_{i=1}^N p_i = 1,  \n\t\t\\;\\; p_i \\geq 0, \\;\\; i=1,\\dotsc,N\n\t\\]\nis $p_i \\equiv 1 / N$, and the corresponding objective value is $N$.\nTherefore, \\eqref{eq:deltak0} {further implies} that\n\\begin{equation*}\n\t\\frac{1}{\\delta_{k_0}} \\geq\n\t\\frac{N}{\\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\| R_0^2}.\n\\end{equation*}\nBy combining this inequality with \\eqref{eq:summed}, we obtain\n\\eqref{eq:sub}.\n\n{For Part 3, we again start from \\eqref{eq:Q2} and\n replace $\\alpha^k_{i_k}$ with $\\bar{\\alpha}_{i}$ in\n\\eqref{eq:Q2} to obtain\n\\begin{align}\n\\label{eq:P3Q1}\n\t&~\\E_{i_k}\\left[\\left.\\bar{\\alpha}_{i_k} Q^{k*}_{i_k}\\right|\nx^k \\right]\\\\\n\\nonumber\n\\leq&~ \\theta\\lambda \\left(F^* - F\\left(x^k\\right) \\right)+\n\\frac{\\theta^2\\lambda^2}{2} \\left(x^k -\nP_\\Omega\\left(x^k\\right)\\right)^\\top \\cP^{-1}\\bar\\AA^{-1} \\HH_k \\left(x^k\n- P_\\Omega\\left(x^k\\right)\\right),\n\\end{align}\nfor all $\\lambda \\in [0,1]$ and all $\\theta \\in [0, \\bar \\pi]$.\nBy applying \\eqref{eq:r0} and \\eqref{eq:H2}, we have\n\\begin{align*}\n\t&~\\E_{i_k}\\left[\\left.\\bar{\\alpha}_{i_k} Q^{k*}_{i_k}\\right|\nx^k \\right]\\\\\n\\nonumber\n\\leq&~ \\theta\\lambda \\left(F^* - F\\left(x^k\\right) \\right)+\n\\frac{\\theta^2\\lambda^2}{2} \\left\\|x^k -\nP_\\Omega\\left(x^k\\right)\\right\\| \\|\\cP^{-1}\\bar\\AA^{-1}\n\\cM\\| \\left\\|x^k\n- P_\\Omega\\left(x^k\\right)\\right\\|\\\\\n\\leq&~ \\theta\\lambda \\left(F^* - F\\left(x^k\\right) \\right)+\n\\frac{\\theta^2\\lambda^2}{2} \\|\\cP^{-1}\\bar\\AA^{-1}\n\\cM\\|R_0^2.\n\\end{align*}\nNow we take expectation over $i_0,\\dotsc,i_{k-1}$ on both sides of\nthis inequality (noting that the last term on the right-hand side are\nall constants that do not depend on $i_k$) to obtain\n\\[\n\\E_{i_0,\\dotsc,i_k}\\left[\\bar \\alpha_{i_k}\nQ^{k*}_{i_k}\\right]\n\\leq -\\theta\\lambda \\E_{i_0,\\dotsc,i_{k-1}} \\left[ F^* - F\\left( x^k\n\\right) \\right] +\n\\frac{\\theta^2\\lambda^2}{2}\\left\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2.\n\\]\nBy defining\n\\begin{equation*}\n\t\\hat \\delta_k \\coloneqq \\E_{i_{0},\\dotsc, i_{k-1}} \\left[ F\\left( x^k\n\t\\right) - F^*\\right]\n\\end{equation*}\nand setting $\\theta = \\bar{\\pi}$, we have that\n\\begin{align}\n\\E_{i_0,\\dotsc,i_k}\\left[\\bar \\alpha_{i_k}\nQ^{k*}_{i_k}\\right]\n\\leq -\\bar\\pi\\lambda \\hat \\delta_k +\n\\frac{\\bar \\pi^2\\lambda^2}{2}\\left\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2.\n\\label{eq:P3Q2}\n\\end{align}\nThe minimum of the right-hand side happens when\n\\[\n\t\\lambda = \\min\\left\\{ 1, \\frac{\\hat \\delta_k }{\\bar{\\pi}\\|\\cP^{-1} \\bar\n\t\\AA^{-1} \\cM\\|R_0^2}\\right\\}.\n\\]\nWhen the expected function value satisfies\n\\begin{equation}\n\t\\hat \\delta_k  \\geq \\bar{\\pi}\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\|R_0^2,\n\\label{eq:turning}\n\\end{equation}\nthe minimizer is $\\lambda = 1$, and the bound becomes\n\\begin{align*}\n\\E_{i_0,\\dotsc,i_k}\\left[\\bar \\alpha_{i_k}\nQ^{k*}_{i_k}\\right]\n\\leq -\\bar\\pi\\hat \\delta_k +\n\\frac{\\bar \\pi^2}{2}\\left\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2\n\\leq -\\frac{\\bar \\pi \\hat \\delta_k}{2}.\n\\end{align*}\nNow we consider \\eqref{eq:usage} and take expectation over $i_0,\\dotsc,\ni_k$ on both sides. Note that $Q^{k*}_{i_k} \\leq 0$ so the upper bound is still\nvalid if we replace $\\alpha^k_{i_k}$ with $\\bar \\alpha_{i_k}$.\nThus we obtain\n\\begin{align} \\label{eq:P3Q3}\n\t\\hat \\delta_{k+1} - \\hat{\\delta}_{k} =\n        \\E_{i_0,\\dotsc,i_k}\\left[F\\left( x^{k+1} \\right) - F\\left( x^k\n          \\right)\\right] & \\leq (1 - \\eta) \\gamma \\E_{i_0,\\dotsc,i_k}\n        \\left[ \\bar \\alpha_{i_k} Q^{k*}_{i_k}\\right] \\\\\n        \\nonumber\n        & \\leq -\\frac{ (1 -\n          \\eta) \\gamma\\bar \\pi \\hat \\delta_k}{2}.\n\\end{align}\nBy rearranging the inequality above, we get the linear convergence of\n\\[\n\t\\hat \\delta_{k+1} \\leq \\hat{\\delta}_{k} \\left(1 -\\frac{(1 - \\eta)\n\t\t\\gamma \\bar \\pi}{2}\\right).\n\\]\nTherefore, we always get the bound\n\\[\n\t\\hat \\delta_k \\leq \\left( 1 - \\frac{(1 - \\eta) \\gamma \\bar \\pi\n\t}{2}  \\right)^k \\left( F(x^0) - F^* \\right)\n\\]\nuntil $\\hat \\delta_{k-1} <\\bar{\\pi}\\|\\cP^{-1} \\bar \\AA^{-1}\n\\cM\\|R_0^2$.\nNote that $\\bar k_0$ is obtained as the first value of $k$ such that\n\\[\n\t\\left( 1 - \\frac{(1 - \\eta) \\gamma \\bar{\\pi}}{2} \\right)^k \\left(\n\tF\\left( x^0 \\right) - F^* \\right) \\leq  \\bar{\\pi}\\|\\cP^{-1} \\bar\n\\AA^{-1} \\cM\\|R_0^2.\n\\]\nTherefore, for $k < \\bar k_0$, the upper bound in \\eqref{eq:upper1} is\nlarger than $\\bar{\\pi}\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\|R_0^2$, so the\nrate \\eqref{eq:upper1} remains valid.\nNote that if $\\hat \\delta_k \\leq  \\bar{\\pi}\\|\\cP^{-1} \\bar\n\\AA^{-1} \\cM\\|R_0^2$ has already held true for some $k < \\bar k_0$,\nclearly this bound is still valid.\nOn the other hand,\nafter $\\bar k_0$, we are guaranteed that \\eqref{eq:turning} must stop\nholding.\nThus the minimizer for \\eqref{eq:P3Q2} becomes $\\lambda = \\hat\n\\delta_k / (\\bar{\\pi}\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\|R_0^2)$.\nWe then start from the first inequality of \\eqref{eq:P3Q3}\nand get\n\\[\n\t\\hat{\\delta}_{k+1} - \\hat{\\delta}_k \\leq -\\frac{(1 - \\eta) \\gamma\n\t\t\\hat{\\delta}_k^2}{2 \\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\|R_0^2}.\n\\]\nFollowing the same derivation we had in Part 2, we can get\n\\[\n\t\\frac{1}{ \\hat{\\delta}_k} \\leq \\frac{1}{\\hat{\\delta}_{k+1}} -\n\t\\frac{(1 - \\eta) \\gamma }{2 \\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\|R_0^2}.\n\\]\nBy summing and telescoping the result above, we get\n\\[\n\t\\frac{1}{ \\hat{\\delta}_k} \\geq \\frac{1}{\\hat{\\delta}_{\\bar k_0}} + (k -\n\t\\bar k_0) \\frac{(1 - \\eta) \\gamma }{2 \\|\\cP^{-1} \\bar \\AA^{-1}\n\\cM\\|R_0^2}.\n\\]\nFollowing the same argument in Part 2, we get the final claim in Part\n3.}\n\\qed\n\\end{proof}","sha256_utf8":"3c099671defe94efca451e5aa42ad15285ae792d9acd326a131a619654ec0d66"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"随机变度量法两阶段收敛","source_units":[{"start":25653,"end":29663,"start_line":594,"end_line":696,"raw_tex":"\\begin{theorem}\n\\label{thm:sub}\nAssume that $f$ and $\\psi$ are convex and \\eqref{eq:compL} holds.\nSuppose that at all iterations $k$ of Algorithm~\\ref{alg:icd},\n{and for any choice $i=i_k$ of the update block at iteration\n  $k$, we have that \\eqref{eq:approx} is satisfied with a fixed $\\eta\n  \\in [0,1)$, with $H^k_i$ chosen such that}\n\\begin{equation}\nH^k_i \\succeq m_i I,\\quad k=0,1,\\dotsc,\n\\label{eq:H}\n\\end{equation}\nfor some $ m_i > 0$ for all $i$. Then the following are true.\n\\begin{enumerate}\n\\item At iteration $k$, given any probability distribution\n  $\\{p^k_i\\}_{i=1}^N > 0$ for choosing the update block $i_k$, denote\n  by $\\{\\alpha^k_i\\}_{i=1}^N > 0$ the step sizes generated by the\n  backtracking line search for each possible choice\n  $i=1,2,\\dotsc,N$. (These step sizes are guaranteed to be bounded\n  away from zero, by Lemma~\\ref{lemma:linesearch}.) Define\n\\begin{equation} \\label{eq:pik}\n\t\\pi^k \\coloneqq \\min_{1 \\le i \\le N} \\alpha^k_i p^k_i,\n\\end{equation}\n{and let $\\cP_k$, $\\AA_k$, and $\\HH_k$ be defined as in\nLemma~\\ref{lemma:Q}.} If\n\\[\n  F\\left( x^k \\right) - F^* \\geq \\left(x^k -\n  P_\\Omega\\left(x^k\\right)\\right)^\\top  \\cP^{-1}_k \\AA_k^{-1}\n  \\HH_k\\left(x^k - P_\\Omega\\left(x^k\\right)\\right) \\pi^k,\n  \\]\n  {then the expected improvement in objective optimality gap\n    at this iteration is bounded away from $1$, as follows:}\n\\begin{align}\n\\frac{\\E_{i_k}\\left[F\\left(x^{k+1}\\right) -\nF^*\\right|\\left.x^k\\right]}{\\left(F\\left( x^k \\right) - F^*\\right)}\n\\leq \\left( 1 - \\frac{\\left( 1 - \\eta \\right) \\gamma \\pi^k }{2}\\right) .\n\\label{eq:earlylinear}\n\\end{align}\n\\item {Given $M_i \\geq m_i, i=1,\\dotsc,N,$}\nand define\n\\begin{equation} \\label{eq:def.M}\n\\cM \\coloneqq \\diag(M_1 I_{n_1},\\dotsc, M_N I_{n_N}),\\quad \\bar \\AA\n\\coloneqq \\diag(\\bar{\\alpha}_1 I_{n_1},\\dotsc, \\bar{\\alpha}_N\nI_{n_N}),\n\\end{equation}\n where $\\bar\n\\alpha_i$ are defined in Lemma \\ref{lemma:linesearch}.  Given a\nprobability distribution $\\{p_i\\}_{i=1}^N > 0$, define\n\\begin{equation} \\label{eq:barpi}\n\\cP \\coloneqq \\diag(p_1 I_{n_1},\\dotsc, p_N I_{n_N}),\\quad\n  \\bar\\pi\n  \\coloneqq \\min_{1 \\le i \\le N} \\, \\bar\\alpha_i p_i,\n\\end{equation}\nand let\n\\begin{equation}\n\tk_0 \\coloneqq \\arg \\min\\left\\{ k: F\\left( x^k \\right) - F^* <\n\t\t\\|\\cP^{-1} \\bar{\\AA}^{-1} \\cM\\| \\bar\\pi R_0^2\\right\\}.\n\\label{eq:k0}\n\\end{equation}\n{Suppose that for all $k \\geq k_0$, the sampling of $i_k$\n  follows the distribution $\\{p_i\\}$, which does not depend on $k$,\n  and }\n\\begin{equation}\nM_i I \\succeq H^k_i \\succeq m_i I,\\quad i=1,\\dotsc,N.\n\\label{eq:H2}\n\\end{equation}\nThen for $k \\geq k_0$, the expected objective follows a sublinear\nconvergence rate, as follows:\n\\begin{align}\n\t\\E_{i_{k_0},i_{k_0+1},\\dotsc,i_{k-1}}\\left[F\\left(x^k\\right)\\right|\\left.\nx^{k_0}\\right] - F^* \\leq \\frac{2 \\|\\cP^{-1} \\bar \\AA^{-1} \\cM \\| R_0^2\n}{2N + (1 - \\eta)\\gamma (k - k_0)}.\n\\label{eq:sub}\n\\end{align}\n\n\\item Suppose that a fixed probability distribution $\\{p_i\\}_{i=1}^N >\n  0$ is used throughout to sample the blocks and that \\eqref{eq:H2}\n  holds for all $k$. {Then, defining}\n  \\begin{equation}\n    \\bar k_0 \\coloneqq \\left\\lceil\\max\\left\\{ 0, \\frac{\\log\n      \\frac{F\\left( x^0 \\right) - F^*}{\\left\\| \\cP^{-1}\n\t\\bar{\\AA}^{-1} \\cM\\right\\| \\bar\\pi\n\tR_0^2}}{\\log\\left( \\frac{2}{2 - \\left( 1 - \\eta\\right) \\gamma\n\t\\bar\\pi}\\right)}\\right\\}\\right\\rceil,\n    \\label{eq:k0bar}\n  \\end{equation}\n  (with $\\bar\\pi$ defined in \\eqref{eq:barpi}), {we have for\n    all $k<\\bar k_0$ that the expected objective satisfies}\n  \\begin{equation}\n    \\E_{i_0,\\dotsc,i_{k-1}}\\left[ F\\left( x^k \\right) - F^*\n\t\t\\right] \\leq \\left( 1 - \\frac{(1 - \\eta)\\gamma\n      \\bar\\pi}{2} \\right)^k \\left( F\\left( x^0\n    \\right) - F^* \\right),\n    \\label{eq:upper1}\n  \\end{equation}\n  {while for all $k \\geq \\bar k_0$, we have}\n  \\begin{align}\n\t  \\E_{i_0,\\dotsc,i_{k-1}}\\left[F\\left(x^k\\right) -\n\t\t  F^* \\right] \\leq \\frac{2 \\left\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\n\t\t  \\right\\| R_0^2 }{2N + (1 - \\eta)\\gamma (k - \\bar k_0)}.\n\\label{eq:upper2}\n\\end{align}\n\\end{enumerate}\n\\end{theorem}","sha256_utf8":"2d98be02851d9bcae68007f0e9999fd2838d96d184cda550afafd676b7554d1f"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"两阶段收敛完整证明","source_units":[{"start":29664,"end":39843,"start_line":697,"end_line":965,"raw_tex":"\\begin{proof}\n{We first prove Part 1.} Consider Lemma \\ref{lemma:Q}.  For the\n  general convex case, we have $\\mu = 0$ in {the OSSC condition} \\eqref{eq:strong}, so\n  \\eqref{eq:Q} reduces to\n\\begin{align}\n\\label{eq:Q2}\n&~\\E_{i_k}\\left[\\left.\\alpha^k_{i_k} Q^{k*}_{i_k}\\right|\nx^k \\right]\\\\\n\\nonumber\n\\leq&~ \\theta\\lambda \\left(F^* - F\\left(x^k\\right) \\right)+\n\\frac{\\theta^2\\lambda^2}{2} \\left(x^k -\nP_\\Omega\\left(x^k\\right)\\right)^\\top \\cP_k^{-1}\\AA_k^{-1} {\\HH_k} \\left(x^k\n- P_\\Omega\\left(x^k\\right)\\right),\n\\end{align}\nfor all $\\lambda \\in [0,1]$ and all $\\theta \\in [0, \\pi^k]$.\nSetting $\\theta =\n  \\pi^k$, we note that the right-hand side of \\eqref{eq:Q2} is a\n  strongly convex function of $\\lambda$ for $x \\notin \\Omega$, so by\n  minimizing explicitly with respect to $\\lambda$, we obtain\n\\begin{equation}\n\\lambda = \\min\\left\\{ 1, \\frac{F\\left( x^k \\right) - F^* }{ \\left(x^k\n\t- P_\\Omega\\left(x^k\\right)\\right)^\\top \\cP_k^{-1}\\AA_k^{-1}\n\\cH_k \\left(x^k - P_\\Omega\\left(x^k\\right)\\right)\\pi^k} \\right\\}.\n\\label{eq:lambda}\n\\end{equation}\n{With this setting of $\\lambda$, when}\n\\begin{equation*}\n\tF\\left( x^k \\right) - F^* \\geq \\left(x^k -\n\tP_\\Omega\\left(x^k\\right)\\right)^\\top  \\cP_k^{-1}\\AA_k^{-1}\n\t\\cH_k \\left(x^k - P_\\Omega\\left(x^k\\right)\\right) \\pi^k,\n\\end{equation*}\nwe have $\\lambda = 1$ and \\eqref{eq:Q2} becomes\n\\begin{align}\n\t\\E_{i_k}\\left[\\left.\\alpha_{i_k}^k Q^{k*}_{i_k} \\right|\n\tx^k\\right] \\leq \\tfrac12 \\pi^k \\left(F^* - F\\left(x^k\\right)\n\t\\right).\n\\label{eq:Q3}\n\\end{align}\nBy combining \\eqref{eq:Q3} and \\eqref{eq:usage}, we have proved\n\\eqref{eq:earlylinear}.\n\n{Next, we prove Part 2.} Consider \\eqref{eq:Q2} with\n$\\alpha^k_{i_k}$ replaced by $\\bar \\alpha_{i_k}$ (so that $\\AA_k$ is\nreplaced by $\\bar\\AA$) and $p^k_{i_k}$ replaced by $p_{i_k}$ (so that\n$\\cP_k$ is replaced by $\\cP$). For any $k \\geq k_0$, we define\n\\begin{equation*}\n\t\\delta_k \\coloneqq \\E_{i_{k_0},\\dotsc, i_{k-1}} \\left[\\left. F\\left( x^k\n\t\\right) - F^*\\right| x^{k_0}\\right].\n\\end{equation*}\nBy applying the definition \\eqref{eq:r0} and the bound \\eqref{eq:H2}\non the right-hand side of the updated \\eqref{eq:Q2},\nand then taking expectations on both sides over $i_{k_0},\\dotsc,\ni_{k-1}$ conditional on $x^{k_0}$, we have that\n\\begin{align}\n\\label{eq:Q4}\n\\E_{i_{k_0},\\dotsc, i_k}\\left[\\left. \\bar \\alpha_{i_k}\nQ^{k*}_{i_k} \\right| x^{k_0}\\right]\n\\leq -\\theta\\lambda \\delta_k +\n\\frac{\\theta^2\\lambda^2}{2}\\left\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2,\n\\end{align}\nfor all $\\lambda \\in [0,1]$ and all $\\theta \\in [0, \\bar \\pi]$.\nSetting $\\theta = \\bar\\pi$ in \\eqref{eq:Q4}, we have from\n\\eqref{eq:k0} that since Algorithm~\\ref{alg:icd} is a descent method,\n$\\delta_k < \\bar\\pi \\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM \\right\\| R_0^2$,\nfor all $k \\ge k_0$.  Therefore, we can use\n\\begin{equation*}\n\t\\lambda = \\frac{\\delta_k}{\\bar\\pi\n\t\\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\| R_0^2}\n\\end{equation*}\nin \\eqref{eq:Q4} to obtain\n\\begin{align}\n\\nonumber\n\\E_{i_{k_0},\\dotsc,i_k} \\left[ \\left. \\bar \\alpha_{i_k}\nQ^{k*}_{i_k}\\right| x^{k_0}\\right]\n\\leq&~ -\\bar\\pi \\frac{\\delta_k^2\n}{2 \\bar\\pi \\left\\| \\cP^{-1} \\bar\n\\AA^{-1} \\cM \\right\\| R_0^2}\\\\\n=&~ -\\frac{\\delta_k^2}{2 \\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2}.\n\\label{eq:newbound}\n\\end{align}\nTherefore, by taking expectation on \\eqref{eq:usage} over\n$i_{k_0},\\dotsc, i_k$ conditional on $x^{k_0}$, and using\n\\eqref{eq:newbound}, we obtain\n\\begin{equation}\n\t\\delta_{k+1} \\leq \\delta_k - \\frac{\\left( 1 - \\eta\n\t\\right)\\gamma\\delta_k^2}{2\\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2}.\n\\label{eq:deltak}\n\\end{equation}\nBy dividing both sides of \\eqref{eq:deltak} by $\\delta_k \\delta_{k+1}$\nand noting from \\eqref{eq:armijo} and Lemma \\ref{lemma:linesearch}\nthat $\\{F(x_k)\\}$ and therefore $\\{\\delta_k\\}$ is descending, we obtain\n\\begin{equation}\n\\frac{1}{\\delta_{k}}\n\\leq \\frac{1}{\\delta_{k+1}} - \\frac{\\left( 1 - \\eta\n\\right)\\gamma\\delta_k}{2\\delta_{k+1} \\left\\| \\cP^{-1} \\bar \\AA^{-1}\n\\cM\\right\\| R_0^2}\n\\leq \\frac{1}{\\delta_{k+1}} - \\frac{\\left( 1 - \\eta\n\\right)\\gamma}{ 2\\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\| R_0^2}.\n\\label{eq:telescpoe}\n\\end{equation}\nBy summing and telescoping \\eqref{eq:telescpoe}, we obtain\n\\begin{equation}\n\t\\frac{1}{\\delta_k} \\geq \\frac{1}{\\delta_{k_0}} +\n\\left( k - k_0\\right) \\frac{\\left( 1 - \\eta\n\\right)\\gamma }{2\\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\| R_0^2}.\n\\label{eq:summed}\n\\end{equation}\nFinally, note that because $\\bar{\\alpha}_i \\in [0,1]$ for\n$i=1,\\dotsc,N$, \\eqref{eq:k0} implies that\n\\begin{equation}\n\t\\frac{1}{\\delta_{k_0}} \\geq \\frac{1}{\\bar\\pi \\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\| R_0^2}\n\t\\geq\n\t\\frac{1}{\\min_i  p_i \\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2}.\n\\label{eq:deltak0}\n\\end{equation}\nNext, it is straightforward that the solution to\n\t\\[\n\t\t\\min_{p_1,\\dotsc,p_N} \\, \\frac{1}{\\min_{1 \\le i \\le N} p_i}\n\\quad\t\t\\text{subject to}\\;\\; \\sum_{i=1}^N p_i = 1,  \n\t\t\\;\\; p_i \\geq 0, \\;\\; i=1,\\dotsc,N\n\t\\]\nis $p_i \\equiv 1 / N$, and the corresponding objective value is $N$.\nTherefore, \\eqref{eq:deltak0} {further implies} that\n\\begin{equation*}\n\t\\frac{1}{\\delta_{k_0}} \\geq\n\t\\frac{N}{\\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\| R_0^2}.\n\\end{equation*}\nBy combining this inequality with \\eqref{eq:summed}, we obtain\n\\eqref{eq:sub}.\n\n{For Part 3, we again start from \\eqref{eq:Q2} and\n replace $\\alpha^k_{i_k}$ with $\\bar{\\alpha}_{i}$ in\n\\eqref{eq:Q2} to obtain\n\\begin{align}\n\\label{eq:P3Q1}\n\t&~\\E_{i_k}\\left[\\left.\\bar{\\alpha}_{i_k} Q^{k*}_{i_k}\\right|\nx^k \\right]\\\\\n\\nonumber\n\\leq&~ \\theta\\lambda \\left(F^* - F\\left(x^k\\right) \\right)+\n\\frac{\\theta^2\\lambda^2}{2} \\left(x^k -\nP_\\Omega\\left(x^k\\right)\\right)^\\top \\cP^{-1}\\bar\\AA^{-1} \\HH_k \\left(x^k\n- P_\\Omega\\left(x^k\\right)\\right),\n\\end{align}\nfor all $\\lambda \\in [0,1]$ and all $\\theta \\in [0, \\bar \\pi]$.\nBy applying \\eqref{eq:r0} and \\eqref{eq:H2}, we have\n\\begin{align*}\n\t&~\\E_{i_k}\\left[\\left.\\bar{\\alpha}_{i_k} Q^{k*}_{i_k}\\right|\nx^k \\right]\\\\\n\\nonumber\n\\leq&~ \\theta\\lambda \\left(F^* - F\\left(x^k\\right) \\right)+\n\\frac{\\theta^2\\lambda^2}{2} \\left\\|x^k -\nP_\\Omega\\left(x^k\\right)\\right\\| \\|\\cP^{-1}\\bar\\AA^{-1}\n\\cM\\| \\left\\|x^k\n- P_\\Omega\\left(x^k\\right)\\right\\|\\\\\n\\leq&~ \\theta\\lambda \\left(F^* - F\\left(x^k\\right) \\right)+\n\\frac{\\theta^2\\lambda^2}{2} \\|\\cP^{-1}\\bar\\AA^{-1}\n\\cM\\|R_0^2.\n\\end{align*}\nNow we take expectation over $i_0,\\dotsc,i_{k-1}$ on both sides of\nthis inequality (noting that the last term on the right-hand side are\nall constants that do not depend on $i_k$) to obtain\n\\[\n\\E_{i_0,\\dotsc,i_k}\\left[\\bar \\alpha_{i_k}\nQ^{k*}_{i_k}\\right]\n\\leq -\\theta\\lambda \\E_{i_0,\\dotsc,i_{k-1}} \\left[ F^* - F\\left( x^k\n\\right) \\right] +\n\\frac{\\theta^2\\lambda^2}{2}\\left\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2.\n\\]\nBy defining\n\\begin{equation*}\n\t\\hat \\delta_k \\coloneqq \\E_{i_{0},\\dotsc, i_{k-1}} \\left[ F\\left( x^k\n\t\\right) - F^*\\right]\n\\end{equation*}\nand setting $\\theta = \\bar{\\pi}$, we have that\n\\begin{align}\n\\E_{i_0,\\dotsc,i_k}\\left[\\bar \\alpha_{i_k}\nQ^{k*}_{i_k}\\right]\n\\leq -\\bar\\pi\\lambda \\hat \\delta_k +\n\\frac{\\bar \\pi^2\\lambda^2}{2}\\left\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2.\n\\label{eq:P3Q2}\n\\end{align}\nThe minimum of the right-hand side happens when\n\\[\n\t\\lambda = \\min\\left\\{ 1, \\frac{\\hat \\delta_k }{\\bar{\\pi}\\|\\cP^{-1} \\bar\n\t\\AA^{-1} \\cM\\|R_0^2}\\right\\}.\n\\]\nWhen the expected function value satisfies\n\\begin{equation}\n\t\\hat \\delta_k  \\geq \\bar{\\pi}\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\|R_0^2,\n\\label{eq:turning}\n\\end{equation}\nthe minimizer is $\\lambda = 1$, and the bound becomes\n\\begin{align*}\n\\E_{i_0,\\dotsc,i_k}\\left[\\bar \\alpha_{i_k}\nQ^{k*}_{i_k}\\right]\n\\leq -\\bar\\pi\\hat \\delta_k +\n\\frac{\\bar \\pi^2}{2}\\left\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2\n\\leq -\\frac{\\bar \\pi \\hat \\delta_k}{2}.\n\\end{align*}\nNow we consider \\eqref{eq:usage} and take expectation over $i_0,\\dotsc,\ni_k$ on both sides. Note that $Q^{k*}_{i_k} \\leq 0$ so the upper bound is still\nvalid if we replace $\\alpha^k_{i_k}$ with $\\bar \\alpha_{i_k}$.\nThus we obtain\n\\begin{align} \\label{eq:P3Q3}\n\t\\hat \\delta_{k+1} - \\hat{\\delta}_{k} =\n        \\E_{i_0,\\dotsc,i_k}\\left[F\\left( x^{k+1} \\right) - F\\left( x^k\n          \\right)\\right] & \\leq (1 - \\eta) \\gamma \\E_{i_0,\\dotsc,i_k}\n        \\left[ \\bar \\alpha_{i_k} Q^{k*}_{i_k}\\right] \\\\\n        \\nonumber\n        & \\leq -\\frac{ (1 -\n          \\eta) \\gamma\\bar \\pi \\hat \\delta_k}{2}.\n\\end{align}\nBy rearranging the inequality above, we get the linear convergence of\n\\[\n\t\\hat \\delta_{k+1} \\leq \\hat{\\delta}_{k} \\left(1 -\\frac{(1 - \\eta)\n\t\t\\gamma \\bar \\pi}{2}\\right).\n\\]\nTherefore, we always get the bound\n\\[\n\t\\hat \\delta_k \\leq \\left( 1 - \\frac{(1 - \\eta) \\gamma \\bar \\pi\n\t}{2}  \\right)^k \\left( F(x^0) - F^* \\right)\n\\]\nuntil $\\hat \\delta_{k-1} <\\bar{\\pi}\\|\\cP^{-1} \\bar \\AA^{-1}\n\\cM\\|R_0^2$.\nNote that $\\bar k_0$ is obtained as the first value of $k$ such that\n\\[\n\t\\left( 1 - \\frac{(1 - \\eta) \\gamma \\bar{\\pi}}{2} \\right)^k \\left(\n\tF\\left( x^0 \\right) - F^* \\right) \\leq  \\bar{\\pi}\\|\\cP^{-1} \\bar\n\\AA^{-1} \\cM\\|R_0^2.\n\\]\nTherefore, for $k < \\bar k_0$, the upper bound in \\eqref{eq:upper1} is\nlarger than $\\bar{\\pi}\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\|R_0^2$, so the\nrate \\eqref{eq:upper1} remains valid.\nNote that if $\\hat \\delta_k \\leq  \\bar{\\pi}\\|\\cP^{-1} \\bar\n\\AA^{-1} \\cM\\|R_0^2$ has already held true for some $k < \\bar k_0$,\nclearly this bound is still valid.\nOn the other hand,\nafter $\\bar k_0$, we are guaranteed that \\eqref{eq:turning} must stop\nholding.\nThus the minimizer for \\eqref{eq:P3Q2} becomes $\\lambda = \\hat\n\\delta_k / (\\bar{\\pi}\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\|R_0^2)$.\nWe then start from the first inequality of \\eqref{eq:P3Q3}\nand get\n\\[\n\t\\hat{\\delta}_{k+1} - \\hat{\\delta}_k \\leq -\\frac{(1 - \\eta) \\gamma\n\t\t\\hat{\\delta}_k^2}{2 \\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\|R_0^2}.\n\\]\nFollowing the same derivation we had in Part 2, we can get\n\\[\n\t\\frac{1}{ \\hat{\\delta}_k} \\leq \\frac{1}{\\hat{\\delta}_{k+1}} -\n\t\\frac{(1 - \\eta) \\gamma }{2 \\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\|R_0^2}.\n\\]\nBy summing and telescoping the result above, we get\n\\[\n\t\\frac{1}{ \\hat{\\delta}_k} \\geq \\frac{1}{\\hat{\\delta}_{\\bar k_0}} + (k -\n\t\\bar k_0) \\frac{(1 - \\eta) \\gamma }{2 \\|\\cP^{-1} \\bar \\AA^{-1}\n\\cM\\|R_0^2}.\n\\]\nFollowing the same argument in Part 2, we get the final claim in Part\n3.}\n\\qed\n\\end{proof}","sha256_utf8":"3c099671defe94efca451e5aa42ad15285ae792d9acd326a131a619654ec0d66"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"复合目标与分块正则项","source_units":[{"start":3568,"end":3751,"start_line":116,"end_line":121,"raw_tex":"We consider the following regularized minimization problem:\n\\begin{equation}\n\t\\min_{x} \\, F\\left(x\\right) \\coloneqq f\\left(x\\right) +\n\t\\psi\\left(x\\right),\n\t\\label{eq:f}\n\\end{equation}","sha256_utf8":"8625915d66d51908bf246990cb9a0d9a9eb7810f1e549fab8e2efa6446c99b09"},{"start":3752,"end":4372,"start_line":122,"end_line":134,"raw_tex":"where $f$ is blockwise Lipschitz-continuously differentiable (defined\nbelow) but not necessarily convex, and the regularizer $\\psi$ is\nconvex, extended-valued, proper, closed, and block-separable, but\npossibly nondifferentiable.  We assume $F$ is lower-bounded and denote\nthe solution set by $\\Omega$, which is assumed to be nonempty.  For\nsimplicity, we assume $x \\in \\R^n$, but our methods can be applied to\nmatrix variables too. We decompose $x\\in \\R^n$ into $N$ blocks such\nthat\n\\begin{equation*}\nx = (x_1,x_2,\\dotsc,x_N) \\in \\R^n, \\quad\nx_i \\in \\R^{n_i},\\quad n_i \\in \\N, \\quad \\sum_{i=1}^N\nn_i = n,\n\\end{equation*}","sha256_utf8":"5b5b6b35a685838cd10111b361122967c009eaaac489848c107bf9e0f62e1078"},{"start":4373,"end":4482,"start_line":135,"end_line":138,"raw_tex":"and assume throughout that the function $\\psi$ can be decomposed as\n\\[\n\\psi(x) = \\sum_{i=1}^N \\psi_i(x_i),\n\\]","sha256_utf8":"ba99058363cc6eec58184573448a27423597e232ed187ba37c6b9194c3c342f7"},{"start":4483,"end":4554,"start_line":139,"end_line":139,"raw_tex":"where all $\\psi_i$ have the properties claimed for $\\psi$ above.  Many\n","sha256_utf8":"fe204f3f0e6aeb4e2635cabe1ec553e838eea25704eedb27bd6120a075ec5421"}]},{"id":"S004","semantic_type":"notation","kind":"notation","title":"分块算子与光滑性","source_units":[{"start":4754,"end":5109,"start_line":143,"end_line":150,"raw_tex":"For the block-separability of $x$, we use the column submatrices of\nthe identity denoted by $U_1,U_2,\\dotsc, U_N$, where {$U_i \\in \\R^{n\n\\times n_i}$} corresponds to\nthe indices in the $i$th block of $x$. Thus, we have\n\\begin{equation*}\n\tx_i = U_i^\\top x, \\quad x = \\sum_{i=1}^N U_i x_i, \\quad \\mbox{and} \\;\n\\nabla_i f = U_i^\\top \\nabla f.\n\\end{equation*}","sha256_utf8":"d747a64894502060357b3116fb1d18cb7bc89346454649451cff8786fe480b43"},{"start":5110,"end":5471,"start_line":151,"end_line":159,"raw_tex":"The blockwise Lipschitz-continuously differentiable property is that\nthere exist constants $L_i > 0$, $i=1,2,\\dotsc,N$, such\nthat\\footnote{{We use the Euclidean norm throughout the\npaper.}}\n\\begin{equation}\n\\left\\|\\nabla_i f(x + U_i h) - \\nabla_i f(x)\\right\\| \\leq L_i\n\\|h\\|,\\quad \\forall h \\in \\R^{n_i},\\quad \\forall x \\in \\R^n.\n\\label{eq:compL}\n\\end{equation}","sha256_utf8":"3208e1cf432e9cea20ae75ae6ba9b62d9195c798a12df72bb1b45dce33f62db4"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"变度量子问题","source_units":[{"start":5909,"end":6383,"start_line":169,"end_line":178,"raw_tex":"The $k$th iteration of the ``exact'' version of our approach proceeds\nas follows.  Given the current iterate $x^k$, we pick a block $i$,\naccording to some discrete probability distribution over\n$\\{1,\\dotsc,N\\}$, and minimize a quadratic approximation of $f$ plus\nthe function $\\psi_i$ for that block, to obtain the update\ndirection $d^k_i$. That is, we have\n\\begin{equation}\n\\label{eq:quadratic}\nd^{k*}_i \\coloneqq \\arg\\min_{d_i\\in \\R^{n_i}} \\, Q^k_{i}( d_i),\n\\end{equation}","sha256_utf8":"06a294b317ce9280b832782885e4a3c1026e4113845b474d0683b1efdb7bf649"},{"start":6384,"end":6591,"start_line":179,"end_line":186,"raw_tex":"where\n\\begin{align}\n\\label{eq:Qdef}\nQ^{k}_{i}( d_i ) \\coloneqq \\nabla_i f\\left( x^k\n\\right)^\\top d_i + \\tfrac12 d_i^\\top H^k_i d_i\n+ \\psi_i\\left( x^k_i + d_i\n\\right) - \\psi_i\\left( x^k_i \\right),\n\\end{align}","sha256_utf8":"b9ba5d9cc8bac599c8451201f569ddf103dcb6296a7b53ff58a7bdf33d8d7eda"},{"start":6592,"end":6774,"start_line":187,"end_line":189,"raw_tex":"and $H^k_i\\in\\R^{n_i\\times n_i}$ is some positive-definite matrix that\ncan change over iterations.  A backtracking line search along $d^k_i$\nis then performed to determine the step.\n","sha256_utf8":"36959388ed304be2f25456bb0011b17e7e6a26c2e4dbb45b068c1d8db4c7df44"}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","title":"相对子问题误差条件","source_units":[{"start":11750,"end":12088,"start_line":281,"end_line":288,"raw_tex":"We assume that $d^k_i$ is an $\\eta$-approximate solution to\n\\eqref{eq:quadratic}, for some $\\eta \\in [0,1)$ fixed over all $k$ and\n  all $i$, satisfying the following condition:\n\\begin{equation}\n{-\\eta Q^{k*}_i = }\\eta \\left(Q^k_i\\left(0\\right) -  Q^{k*}_i \\right) \\geq\nQ^k_i\\left(d^k_i\\right) - Q^{k*}_i,\n\\label{eq:approx}\n\\end{equation}","sha256_utf8":"e2841906f5780517c3ca87826354e0fd3bc34d1ec26896f625063fd0584cef61"},{"start":12089,"end":12211,"start_line":289,"end_line":290,"raw_tex":"where $Q^{k*}_i \\coloneqq \\inf_{d_i} Q^k_i(d_i) {=\n  Q^k_i(d^{k*}_i)}$. Note that the setting $\\eta=0$ corresponds to the\n","sha256_utf8":"aed6d93525c1dfe1b52226701a419f720860467ee5e111afe2d4c993e78b7200"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"回溯步长与充分下降","source_units":[{"start":13967,"end":14544,"start_line":319,"end_line":331,"raw_tex":"added to $H^k_{i_k}$. After finding an approximate solution $d^k_{i_k}$ to\n\\eqref{eq:quadratic} that satisfies \\eqref{eq:approx} for some $\\eta\n\\in [0,1)$, we conduct a backtracking line search, as in\n  \\cite{TseY09a}: Given $\\beta, \\gamma \\in (0,1)$, we let\n  $\\alpha^k_{i_k}$ be the largest value in $\\{1,\\beta^1,\\beta^2,\n  \\dotsc\\}$ such that the following sufficient decrease condition is\n  satisfied:\n\\begin{equation}\n\tF\\left(x^k + \\alpha^k_{i_k} U_{i_k} d_{i_k}^k \\right) \\leq F\\left(\n\tx^k \\right) +\n\t\\alpha_{i_k}^k \\gamma \\Delta^k_{i_k},\n\\label{eq:armijo}\n\\end{equation}","sha256_utf8":"ed2ac4fc1ab7a199789cb94d19f284c3ae341a5090bf1ca55c69d5915b499e2f"},{"start":14545,"end":14733,"start_line":332,"end_line":337,"raw_tex":"where\n\\begin{equation}\n\\Delta^k_i \\coloneqq \\nabla_i f\\left( x^k \\right)^\\top d^k_i + \\psi_{i}\\left(\nx^k_{i} + d^k_i \\right) - \\psi_{i}\\left( x^k_i \\right).\n\\label{eq:Delta}\n\\end{equation}","sha256_utf8":"55b6042aaf834abf7690713def923cccec99e63426690fdf6bd9ef3c7e594ea7"},{"start":14734,"end":14807,"start_line":338,"end_line":338,"raw_tex":"Then the iterate is updated to $x^k + \\alpha^k_{i_k} U_{i_k} d_{i_k}^k$.\n","sha256_utf8":"2182a84bd74fdad691748a3f2cb394f1ade262404dca317b39fa07943ab5dc11"}]},{"id":"S008","semantic_type":"algorithm","kind":"algorithm","title":"非精确随机分块算法","source_units":[{"start":14808,"end":15715,"start_line":340,"end_line":363,"raw_tex":"\\begin{algorithm}[tb]\n\\caption{Inexact variable-metric block-coordinate descent for\n\t\\eqref{eq:f}}\n\\label{alg:icd}\n\\begin{algorithmic}[1]\n\\STATE Given $\\beta, \\gamma \\in (0,1)$, $\\eta \\in [0,1)$, and $x^0\n\t\\in \\R^n$;\n\\FOR{$k=0,1,2,\\dotsc$}\n\t\\STATE Pick a probability distribution\n\t{$p^k_1,\\dotsc,p^k_N > 0$,\n\t\t$\\sum_{i} p^k_i = 1$}, and sample {${i_k}$} accordingly;\n\t\\STATE Compute $\\nabla_{i_k} f(x^k)$ and choose a\n\tpositive-definite $H^k_{i_k}$;\n\t\\STATE Approximately solve \\eqref{eq:quadratic} {for $i=i_k$}  to obtain a\n\tsolution $d^k_{i_k}$ satisfying \\eqref{eq:approx};\n\t\\STATE Compute $\\Delta_{i_k}$ by \\eqref{eq:Delta}, {with\n\t$i=i_k$}; Set {$\\alpha_{i_k}^k \\leftarrow 1$};\n\t\\WHILE{\\eqref{eq:armijo} is not satisfied}\n\t\\STATE $\\alpha^k_{i_k} \\leftarrow \\beta \\alpha^k_{i_k}$;\n\t\\ENDWHILE\n\t\\STATE $x^{k+1} \\leftarrow x^k + \\alpha^k_{i_k} U_{i_k} d^k_{i_k}$;\n\\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"9ce5ffe22d0302e66a63990b20c2d65fe32bf08675f19e94b982c3037a0b081a"}]},{"id":"S009","semantic_type":"notation","kind":"notation","title":"所有候选块的条件期望","source_units":[{"start":16032,"end":16565,"start_line":372,"end_line":379,"raw_tex":"  results of this section we often focus on a particular iteration\n  $k$, but rather than considering the consequences of updating the\n  chosen block $i_k$ at that iteration, we examine what would happen\n  for {\\em all possible} choices of $i=1,2,\\dotsc,N$, if each of these\n  values happened to be chosen as $i_k$. Since the actual update block\n  $i_k$ is chosen randomly from among these $N$ possibilities, we\n  obtain results about the expected change in $F$ by taking {\\em\n    expectations} over all these hypothetical choices.}\n","sha256_utf8":"08440d65ee79893163d9c6c473b27247455b0865fcee3a86f288492d13841029"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"最优集投影与强凸条件","source_units":[{"start":18219,"end":18879,"start_line":421,"end_line":438,"raw_tex":"\\begin{definition}\nGiven any function $F$ whose minimum value $F^*$ is attainable,\nand for any $x$, define $P_{\\Omega}(x)$ to be the\n{(Euclidean-norm)} projection of $x$ onto the optimal set\n$\\Omega$.\nWe say that $F$ satisfies the {\\em optimal set strong convexity} {(OSSC)}\ncondition with parameter $\\mu \\geq 0$, if for any $x$ and any\n$\\lambda \\in [0,1]$, the following holds.\n\\begin{align}\n  \\nonumber\nF(\\lambda x + & (1 - \\lambda )\nP_{\\Omega}(x)) \\\\\n\\label{eq:strong}\n& \\leq \\lambda F\\left(x\\right) + \\left(1 - \\lambda\\right) F^* -\n\\tfrac12 \\mu \\lambda \\left( 1 - \\lambda \\right)  \\left\\|x -\nP_{\\Omega}\\left(x\\right)\\right\\|^2.\n\\end{align}\n\\end{definition}","sha256_utf8":"903d25e498024e4e0da0001acd4287001b2ab77bb7a23dc6f7d8ef7760d6bd63"}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","title":"有限水平集距离界","source_units":[{"start":24552,"end":24706,"start_line":573,"end_line":577,"raw_tex":"Given any $x^0$, we define\n\\begin{equation}\nR_0 \\coloneqq \\sup_{x: {F( x ) \\leq F( x^0)}}\\quad\\left\\|x - P_\\Omega(x)\\right\\|.\n\\label{eq:r0}\n\\end{equation}","sha256_utf8":"9b93a1e9cbdf94481ed474d2fcf9337586af4201543b33d79a8055f642fdfbd7"},{"start":24707,"end":24846,"start_line":578,"end_line":580,"raw_tex":"For the case of general convex problems, we make the assumption that\nfor any $x^0$, the value of $R_0$ defined in \\eqref{eq:r0} is\nfinite.\n","sha256_utf8":"007b2e5e75ba66a164b90c637261a1b9206bb922be239eb8ad88a0d6de2ab153"}]},{"id":"S012","semantic_type":"lemma","kind":"lemma","title":"下降方向与步长下界","source_units":[{"start":16786,"end":17582,"start_line":385,"end_line":405,"raw_tex":"\\begin{lemma}\n\\label{lemma:linesearch}\n{At the $k$th iteration, suppose that $H^k_i \\succeq m_i I$ for\n  some $m_i > 0$, for $i=1,2,\\dotsc,N$, and that the subproblem\n    solution $d^k_i$ satisfies \\eqref{eq:approx}, for each\n    $i=1,2,\\dotsc,N$.} Then we have\n\\begin{equation}\n\\Delta^k_i \\leq -\\frac{1}{1 + \\sqrt{\\eta}} \\left(d^k_i\\right)^\\top\nH^k_i d^k_i\n\\leq -\\frac{m_i}{1 + \\sqrt{\\eta}} \\|d_i^k\\|^2.\n\\label{eq:deltabound2}\n\\end{equation}\nMoreover, the backtracking line search procedure in\nAlgorithm~\\ref{alg:icd} terminates finitely, with the step size\n$\\alpha^k_i$ lower bounded by\n\\begin{equation}\n\\alpha^k_i \\geq \\bar\\alpha_i \\coloneqq \\min\\left\\{1, \\frac{2\\beta\\left(1 -\n\\gamma\\right) m_i}{L_i\\left(1 + \\sqrt{\\eta}\\right)}\\right\\}.\n\\label{eq:linesearchbound2}\n\\end{equation}\n\\end{lemma}","sha256_utf8":"9b4b0711aa5e9543b512867c2aba6af42623d270edce8726177aaf93b98eb142"}]},{"id":"S013","semantic_type":"external_fact","kind":"external_fact","title":"既有非精确回溯界","source_units":[],"declared_external":true,"declaration_text":["LeeW18a, Corollary 4, applied to each block: if H_i^k ⪰ m_i I with m_i>0 and d_i^k satisfies the relative model-error condition with η∈[0,1), then Δ_i^k ≤ −(d_i^k)^T H_i^k d_i^k/(1+√η) ≤ −m_i||d_i^k||²/(1+√η). With block Lipschitz constant L_i and backtracking parameters β,γ∈(0,1), backtracking terminates and α_i^k ≥ min{1,2β(1−γ)m_i/[L_i(1+√η)]}. The author explicitly treats B0017 as this cited boundary result and omits its proof at lines 381–384."]},{"id":"S014","semantic_type":"lemma","kind":"lemma","title":"期望最优模型下降界","source_units":[{"start":19171,"end":20356,"start_line":445,"end_line":473,"raw_tex":"\\begin{lemma}\n\\label{lemma:Q}\n{Let $f$ and $\\psi$ be convex with $F$ satisfying\n  \\eqref{eq:strong} for some $\\mu \\geq 0$. At iteration $k$, we\n  consider matrices $H^k_i \\succeq 0$ with $H_i^k \\in \\R^{n_i \\times\n    n_i}$, $i=1,\\dotsc,N$, probability distribution $\\{p_i^k\\}_{i=1}^N\n  > 0$, and step sizes $\\{\\alpha^k_i\\}_{i=1}^N > 0$. We define}\n\\begin{align*}\n\\cP_k &\\coloneqq \\diag(p^k_1 I_{n_1},\\dotsc, p^k_N I_{n_N}),\\quad\n\\AA_k \\coloneqq \\diag(\\alpha_1^k\nI_{n_1},\\dotsc, \\alpha_N^k I_{n_N}),\\\\\n\\HH_k &\\coloneqq \\diag(H_1^k,\\dotsc,H_N^k).\n\\end{align*}\nThen for $Q_i^k$ defined by \\eqref{eq:Qdef}, the following holds\nfor all $\\lambda \\in \\left[0,1\\right]$ and all $\\theta$ such that\n$0\\leq\\theta\\leq \\alpha^k_i p^k_i$, $i=1,\\dotsc,N$:\n\\begin{align}\n\\nonumber\n\\E_i\\left[\\left.\\alpha^k_i Q^{k*}_i \\right| x^k\\right]\n&\\leq \\theta\\lambda \\left(F^* - F\\left(x^k\\right) \\right)- \\tfrac12 \\mu \\theta\n\t\\lambda \\left( 1 - \\lambda \\right) \\left\\|x^k -\nP_\\Omega\\left(x^k\\right)\\right\\|^2 +\\\\\n& \\tfrac12 \\theta^2\\lambda^2 \\left( x^k - P_{\\Omega}\\left( x^k \\right)\n\\right)^\\top  \\cP_k^{-1}\\AA_k^{-1} \\HH_k\\left( x^k - P_{\\Omega}\\left(\nx^k \\right)\n\\right).\n\\label{eq:Q}\n\\end{align}\n\\end{lemma}","sha256_utf8":"d5d23137760b9e97a1824bd449a1ccb1233ea644d6d43566dac6d2e20d0dbe84"}]},{"id":"S015","semantic_type":"proof","kind":"proof","title":"模型下降界完整证明","source_units":[{"start":20357,"end":23964,"start_line":474,"end_line":557,"raw_tex":"\\begin{proof}\nGiven any $d\\in \\R^n$, let $\\tilde d \\coloneqq \\AA_k \\cP_k d \\in\n\\R^n$.  We obtain by change of variables that\n\\begin{align}\n\\nonumber\n&~\\E_i{\\left[\\left.\\alpha_i^k Q_i^{k*} \\right| x^k \\right]} \\\\\n\\nonumber\n=&~ \\min_d \\,\\nabla f \\left( x^k\n\t\\right)^\\top  \\AA_k \\cP_k d + \\tfrac12 d^\\top  \\HH_k \\AA_k \\cP_k d + \\sum_{i=1}^N\n\t\\alpha_i^k p_i^k \\left(\\psi_i\\left(x_i^k + d_i\\right) -\n\t\\psi_i\\left(x_i^k\\right)\\right)\\\\\n\\nonumber\n=&~ \\min_{\\tilde d \\in \\R^n} \\, \\nabla f\\left( x^k \\right)^\\top  \\tilde d + \\tfrac12\n\t\\tilde{d}^\\top  \\cP_k^{-1} \\AA_k^{-1} \\HH_k \\tilde d + \\sum_{i=1}^N\n\t\\alpha_i^k p_i^k \\left( \\psi_i \\left( x^k_i +\n\t\\frac{\\tilde{d}_i}{\\alpha_i^k p_i^k} \\right) - \\psi_i\\left( x_i^k\n\t\\right)\\right)\\\\\n\\nonumber\n\\leq&~ \\min_{\\tilde d \\in \\R^n} \\,\\min_{\\theta\\in[0,1] \\, \\text{s.t.} \\,\n\t\\frac{\\theta}{\\alpha_i^k p_i^k} \\leq 1,\\forall i}\n\t\\nabla f\\left( x^k \\right)^\\top  \\left(\\theta\\tilde d\\right) + \\tfrac12\n\t\\left(\\theta\\tilde{d}\\right)^\\top  \\cP_k^{-1}\\AA_k^{-1}  \\HH_k\n\t\\left(\\theta\\tilde d\\right)\\\\\n\\label{eq:intermediate1}\n\t&\\qquad\\qquad + \\sum_{i=1}^N \\alpha_i^k p_i^k \\left(\n\t\\psi_i \\left( x_i^k + \\frac{\\theta\\tilde{d}_i}{\\alpha_i^k p_i^k} \\right) -\n\t\\psi_i\\left( x_i^k \\right)\\right),\n\\end{align}\n{where each $\\tilde d_i \\in \\R^{n_i}$. In\n  \\eqref{eq:intermediate1}, we used the fact that $\\theta \\tilde {d}$\n  is also a feasible point for the left-hand side, hence its objective\n  value is no smaller than the minimizer.}\n\nNext, from the convexity of $f$, we have\n\\begin{equation*}\n\\nabla f\\left( x^k \\right)^\\top  \\theta \\tilde d = \\theta \\left( \\nabla\nf\\left( x^k \\right)^\\top  \\tilde d\n\\right) \\leq\n\\theta \\left( f\\left( x^k + \\tilde d \\right) - f\\left( x^k \\right)\n\\right),\n\\end{equation*}\nand from  $\\theta / (\\alpha_i^k p_i^k) \\leq 1$ for all $i$ and\nthe convexity of $\\psi$, we obtain\n\\begin{align*}\n\t\\psi_i \\left( x^k_i + \\frac{\\theta \\tilde{d}_i}{\\alpha_i^k p_i^k} \\right)\n\t&\\leq \\left( 1 - \\frac{\\theta }{\\alpha_i^k p_i^k} \\right)\n\t\\psi_i\\left( x_i^k \\right) + \\frac{\\theta}{\\alpha_i^k p_i^k}\n\t\\psi_i\\left( x_i^k + \\tilde{d}_i \\right)\\\\\n\t&= \\frac{\\theta }{\\alpha_i^k p_i^k}\\left( \\psi_i \\left( x_i^k +\n\t\\tilde{d}_i \\right) - \\psi_i\\left( x_i^k \\right) \\right) + \\psi_i \\left(\n\tx_i^k \\right).\n\\end{align*}\nTherefore, we have\n\\begin{align}\n\\nonumber\n&~\\min_{\\tilde d}\\,\\min_{\\theta\\in[0,1] \\, \\text{s.t.} \\,\n\t\\frac{\\theta}{\\alpha_i^k p_i^k} \\leq 1,\\forall i}\n\t\\nabla f\\left( x^k \\right)^\\top  \\left(\\theta\\tilde d\\right) + \\tfrac12\n\t\\left(\\theta\\tilde{d}\\right)^\\top  \\cP_k^{-1}\\AA_k^{-1}  \\HH_k\n\t\\left(\\theta\\tilde d\\right) \\\\\n\t\\nonumber\n\t&\\qquad\\qquad+ \\sum_{i=1}^N \\alpha_i^k p_i^k \\left(\n\t\\psi_i \\left( x^k_i + \\frac{\\theta\\tilde{d}_i}{\\alpha_i^k p^k_i} \\right) -\n\t\\psi_i\\left( x_i^k \\right)\\right)\\\\\n\\nonumber\n\\leq&~ \\min_{\\tilde d} \\, \\min_{\\theta\\in[0,1] \\, \\text{s.t.} \\,\n\t\\frac{\\theta}{\\alpha_i^k p^k_i} \\leq 1,\\forall i}\n\t\\theta\\left(F(x^k+\\tilde d) - F(x^k)\\right)  + \\frac{\\theta^2}{2}\n\t\\tilde{d}^\\top  \\cP_k^{-1}\\AA_k^{-1} \\HH_k \\tilde{d}\\\\\n\t\\nonumber\n\\leq&~ \\min_{\\lambda\\in [0,1]}\\,\\min_{\\theta\\in[0,1] \\, \\text{s.t.} \\,\n\t\\frac{\\theta}{\\alpha^k_i p^k_i} \\leq 1,\\forall i}\n\\theta \\left(F\\left(x^k + \\lambda \\left(\n\tP_{\\Omega}\\left(x^k\\right)  - x^k \\right) \\right) - F\\left( x^k\n\t\\right)\\right) \\\\\n\t&\\qquad\\qquad + \\frac{\\theta^2\n\t\\lambda^2}{2}\\left(P_\\Omega\\left(x^k\\right)  - x^k\\right)^\\top \n\t\\cP_k^{-1}\\AA_k^{-1} \\HH_k \\left(P_\\Omega\\left(x^k\\right) - x^k \\right).\n\\label{eq:intermediate2}\n\\end{align}\nThe result \\eqref{eq:Q} then follows from combining\n\\eqref{eq:intermediate1}, \\eqref{eq:intermediate2}, and\n\\eqref{eq:strong}.  \\qed\n\\end{proof}","sha256_utf8":"7d73e45dbef6cb5df29aa19cfb181395bf4beaa95abd8592a7da92a45ba51cde"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"变量替换与缩放试探点","source_units":[{"start":20357,"end":21814,"start_line":474,"end_line":505,"raw_tex":"\\begin{proof}\nGiven any $d\\in \\R^n$, let $\\tilde d \\coloneqq \\AA_k \\cP_k d \\in\n\\R^n$.  We obtain by change of variables that\n\\begin{align}\n\\nonumber\n&~\\E_i{\\left[\\left.\\alpha_i^k Q_i^{k*} \\right| x^k \\right]} \\\\\n\\nonumber\n=&~ \\min_d \\,\\nabla f \\left( x^k\n\t\\right)^\\top  \\AA_k \\cP_k d + \\tfrac12 d^\\top  \\HH_k \\AA_k \\cP_k d + \\sum_{i=1}^N\n\t\\alpha_i^k p_i^k \\left(\\psi_i\\left(x_i^k + d_i\\right) -\n\t\\psi_i\\left(x_i^k\\right)\\right)\\\\\n\\nonumber\n=&~ \\min_{\\tilde d \\in \\R^n} \\, \\nabla f\\left( x^k \\right)^\\top  \\tilde d + \\tfrac12\n\t\\tilde{d}^\\top  \\cP_k^{-1} \\AA_k^{-1} \\HH_k \\tilde d + \\sum_{i=1}^N\n\t\\alpha_i^k p_i^k \\left( \\psi_i \\left( x^k_i +\n\t\\frac{\\tilde{d}_i}{\\alpha_i^k p_i^k} \\right) - \\psi_i\\left( x_i^k\n\t\\right)\\right)\\\\\n\\nonumber\n\\leq&~ \\min_{\\tilde d \\in \\R^n} \\,\\min_{\\theta\\in[0,1] \\, \\text{s.t.} \\,\n\t\\frac{\\theta}{\\alpha_i^k p_i^k} \\leq 1,\\forall i}\n\t\\nabla f\\left( x^k \\right)^\\top  \\left(\\theta\\tilde d\\right) + \\tfrac12\n\t\\left(\\theta\\tilde{d}\\right)^\\top  \\cP_k^{-1}\\AA_k^{-1}  \\HH_k\n\t\\left(\\theta\\tilde d\\right)\\\\\n\\label{eq:intermediate1}\n\t&\\qquad\\qquad + \\sum_{i=1}^N \\alpha_i^k p_i^k \\left(\n\t\\psi_i \\left( x_i^k + \\frac{\\theta\\tilde{d}_i}{\\alpha_i^k p_i^k} \\right) -\n\t\\psi_i\\left( x_i^k \\right)\\right),\n\\end{align}\n{where each $\\tilde d_i \\in \\R^{n_i}$. In\n  \\eqref{eq:intermediate1}, we used the fact that $\\theta \\tilde {d}$\n  is also a feasible point for the left-hand side, hence its objective\n  value is no smaller than the minimizer.}\n","sha256_utf8":"c185f2840b53130c964e5e80b2b5ad3442fccf3a2e378fcb462df6745d923872"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"光滑项的凸性估计","source_units":[{"start":21815,"end":22086,"start_line":507,"end_line":514,"raw_tex":"Next, from the convexity of $f$, we have\n\\begin{equation*}\n\\nabla f\\left( x^k \\right)^\\top  \\theta \\tilde d = \\theta \\left( \\nabla\nf\\left( x^k \\right)^\\top  \\tilde d\n\\right) \\leq\n\\theta \\left( f\\left( x^k + \\tilde d \\right) - f\\left( x^k \\right)\n\\right),\n\\end{equation*}\n","sha256_utf8":"bb7bd98378ecc3bb9dd7bb35005b2fa9532ae389c96a160476c19d4dba1a1914"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"正则项的凸组合估计","source_units":[{"start":22086,"end":22607,"start_line":515,"end_line":525,"raw_tex":"and from  $\\theta / (\\alpha_i^k p_i^k) \\leq 1$ for all $i$ and\nthe convexity of $\\psi$, we obtain\n\\begin{align*}\n\t\\psi_i \\left( x^k_i + \\frac{\\theta \\tilde{d}_i}{\\alpha_i^k p_i^k} \\right)\n\t&\\leq \\left( 1 - \\frac{\\theta }{\\alpha_i^k p_i^k} \\right)\n\t\\psi_i\\left( x_i^k \\right) + \\frac{\\theta}{\\alpha_i^k p_i^k}\n\t\\psi_i\\left( x_i^k + \\tilde{d}_i \\right)\\\\\n\t&= \\frac{\\theta }{\\alpha_i^k p_i^k}\\left( \\psi_i \\left( x_i^k +\n\t\\tilde{d}_i \\right) - \\psi_i\\left( x_i^k \\right) \\right) + \\psi_i \\left(\n\tx_i^k \\right).\n\\end{align*}\n","sha256_utf8":"74beb3d5ca962a843394164aa85743a284a7e254fa6f45f8096ae6153044ed90"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"沿最优集投影方向比较","source_units":[{"start":22607,"end":23820,"start_line":526,"end_line":553,"raw_tex":"Therefore, we have\n\\begin{align}\n\\nonumber\n&~\\min_{\\tilde d}\\,\\min_{\\theta\\in[0,1] \\, \\text{s.t.} \\,\n\t\\frac{\\theta}{\\alpha_i^k p_i^k} \\leq 1,\\forall i}\n\t\\nabla f\\left( x^k \\right)^\\top  \\left(\\theta\\tilde d\\right) + \\tfrac12\n\t\\left(\\theta\\tilde{d}\\right)^\\top  \\cP_k^{-1}\\AA_k^{-1}  \\HH_k\n\t\\left(\\theta\\tilde d\\right) \\\\\n\t\\nonumber\n\t&\\qquad\\qquad+ \\sum_{i=1}^N \\alpha_i^k p_i^k \\left(\n\t\\psi_i \\left( x^k_i + \\frac{\\theta\\tilde{d}_i}{\\alpha_i^k p^k_i} \\right) -\n\t\\psi_i\\left( x_i^k \\right)\\right)\\\\\n\\nonumber\n\\leq&~ \\min_{\\tilde d} \\, \\min_{\\theta\\in[0,1] \\, \\text{s.t.} \\,\n\t\\frac{\\theta}{\\alpha_i^k p^k_i} \\leq 1,\\forall i}\n\t\\theta\\left(F(x^k+\\tilde d) - F(x^k)\\right)  + \\frac{\\theta^2}{2}\n\t\\tilde{d}^\\top  \\cP_k^{-1}\\AA_k^{-1} \\HH_k \\tilde{d}\\\\\n\t\\nonumber\n\\leq&~ \\min_{\\lambda\\in [0,1]}\\,\\min_{\\theta\\in[0,1] \\, \\text{s.t.} \\,\n\t\\frac{\\theta}{\\alpha^k_i p^k_i} \\leq 1,\\forall i}\n\\theta \\left(F\\left(x^k + \\lambda \\left(\n\tP_{\\Omega}\\left(x^k\\right)  - x^k \\right) \\right) - F\\left( x^k\n\t\\right)\\right) \\\\\n\t&\\qquad\\qquad + \\frac{\\theta^2\n\t\\lambda^2}{2}\\left(P_\\Omega\\left(x^k\\right)  - x^k\\right)^\\top \n\t\\cP_k^{-1}\\AA_k^{-1} \\HH_k \\left(P_\\Omega\\left(x^k\\right) - x^k \\right).\n\\label{eq:intermediate2}\n\\end{align}\n","sha256_utf8":"ab674162969374940458a5a42957c65ade2fc2c76aa7cb6bd99fac2ab768450c"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"应用最优集强凸条件","source_units":[{"start":23820,"end":23964,"start_line":554,"end_line":557,"raw_tex":"The result \\eqref{eq:Q} then follows from combining\n\\eqref{eq:intermediate1}, \\eqref{eq:intermediate2}, and\n\\eqref{eq:strong}.  \\qed\n\\end{proof}","sha256_utf8":"0554765db7f5076f16b98670919cafa8c213b38097b62e4c4919b187970cb81c"}]},{"id":"S021","semantic_type":"proof","kind":"proof","title":"目标下降与模型值联系","source_units":[{"start":23966,"end":24408,"start_line":559,"end_line":568,"raw_tex":"By positive semidefiniteness of $H^k_i$ for all $i$ and all $k$,\n\\eqref{eq:armijo} implies that\n\\begin{align}\n\\nonumber\nF\\left(x^k + \\alpha_{i_k}^k U_{i_k} d_{i_k}^k\\right) - F\\left( x^k \\right)\n&\\leq \\gamma \\alpha_{i_k}^k \\left(\\Delta_{i_k}^k + \\tfrac12\n\\left(d^k_{i_k}\\right)^\\top  H_{i_k}^k d_{i_k}^k\\right) \\\\\n& = \\gamma \\alpha_{i_k}^k Q_{i_k}^k(d_{i_k}^k) \\leq (1 - \\eta)\\gamma \\alpha^k_{i_k}  Q^{k*}_{i_k}.\n\\label{eq:usage}\n\\end{align}\n","sha256_utf8":"ac5772f1e18197215cac34d3d25a58f0fde4016284ce69806f42921a157d62c3"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"由线搜索和误差界得下降","source_units":[{"start":23966,"end":24408,"start_line":559,"end_line":568,"raw_tex":"By positive semidefiniteness of $H^k_i$ for all $i$ and all $k$,\n\\eqref{eq:armijo} implies that\n\\begin{align}\n\\nonumber\nF\\left(x^k + \\alpha_{i_k}^k U_{i_k} d_{i_k}^k\\right) - F\\left( x^k \\right)\n&\\leq \\gamma \\alpha_{i_k}^k \\left(\\Delta_{i_k}^k + \\tfrac12\n\\left(d^k_{i_k}\\right)^\\top  H_{i_k}^k d_{i_k}^k\\right) \\\\\n& = \\gamma \\alpha_{i_k}^k Q_{i_k}^k(d_{i_k}^k) \\leq (1 - \\eta)\\gamma \\alpha^k_{i_k}  Q^{k*}_{i_k}.\n\\label{eq:usage}\n\\end{align}\n","sha256_utf8":"ac5772f1e18197215cac34d3d25a58f0fde4016284ce69806f42921a157d62c3"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"一般凸情形的模型界","source_units":[{"start":29664,"end":30248,"start_line":697,"end_line":711,"raw_tex":"\\begin{proof}\n{We first prove Part 1.} Consider Lemma \\ref{lemma:Q}.  For the\n  general convex case, we have $\\mu = 0$ in {the OSSC condition} \\eqref{eq:strong}, so\n  \\eqref{eq:Q} reduces to\n\\begin{align}\n\\label{eq:Q2}\n&~\\E_{i_k}\\left[\\left.\\alpha^k_{i_k} Q^{k*}_{i_k}\\right|\nx^k \\right]\\\\\n\\nonumber\n\\leq&~ \\theta\\lambda \\left(F^* - F\\left(x^k\\right) \\right)+\n\\frac{\\theta^2\\lambda^2}{2} \\left(x^k -\nP_\\Omega\\left(x^k\\right)\\right)^\\top \\cP_k^{-1}\\AA_k^{-1} {\\HH_k} \\left(x^k\n- P_\\Omega\\left(x^k\\right)\\right),\n\\end{align}\nfor all $\\lambda \\in [0,1]$ and all $\\theta \\in [0, \\pi^k]$.\n","sha256_utf8":"006f635b1443d085b80d4d27dcae1087000cb0274fc9cbde064375767df8cf07"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"显式最小化插值参数","source_units":[{"start":30248,"end":30708,"start_line":712,"end_line":721,"raw_tex":"Setting $\\theta =\n  \\pi^k$, we note that the right-hand side of \\eqref{eq:Q2} is a\n  strongly convex function of $\\lambda$ for $x \\notin \\Omega$, so by\n  minimizing explicitly with respect to $\\lambda$, we obtain\n\\begin{equation}\n\\lambda = \\min\\left\\{ 1, \\frac{F\\left( x^k \\right) - F^* }{ \\left(x^k\n\t- P_\\Omega\\left(x^k\\right)\\right)^\\top \\cP_k^{-1}\\AA_k^{-1}\n\\cH_k \\left(x^k - P_\\Omega\\left(x^k\\right)\\right)\\pi^k} \\right\\}.\n\\label{eq:lambda}\n\\end{equation}\n","sha256_utf8":"5652af4866ac801b7bf44f76eeb491a0edca65e5e060044c7118b72aba2d2a95"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"大误差阶段的线性下降","source_units":[{"start":30708,"end":31248,"start_line":722,"end_line":736,"raw_tex":"{With this setting of $\\lambda$, when}\n\\begin{equation*}\n\tF\\left( x^k \\right) - F^* \\geq \\left(x^k -\n\tP_\\Omega\\left(x^k\\right)\\right)^\\top  \\cP_k^{-1}\\AA_k^{-1}\n\t\\cH_k \\left(x^k - P_\\Omega\\left(x^k\\right)\\right) \\pi^k,\n\\end{equation*}\nwe have $\\lambda = 1$ and \\eqref{eq:Q2} becomes\n\\begin{align}\n\t\\E_{i_k}\\left[\\left.\\alpha_{i_k}^k Q^{k*}_{i_k} \\right|\n\tx^k\\right] \\leq \\tfrac12 \\pi^k \\left(F^* - F\\left(x^k\\right)\n\t\\right).\n\\label{eq:Q3}\n\\end{align}\nBy combining \\eqref{eq:Q3} and \\eqref{eq:usage}, we have proved\n\\eqref{eq:earlylinear}.\n","sha256_utf8":"914188bac7d1aacb237a02958d7b34bc545c86c2223802d2df93c96d132ca3b3"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"固定采样下的条件模型界","source_units":[{"start":31249,"end":32187,"start_line":738,"end_line":758,"raw_tex":"{Next, we prove Part 2.} Consider \\eqref{eq:Q2} with\n$\\alpha^k_{i_k}$ replaced by $\\bar \\alpha_{i_k}$ (so that $\\AA_k$ is\nreplaced by $\\bar\\AA$) and $p^k_{i_k}$ replaced by $p_{i_k}$ (so that\n$\\cP_k$ is replaced by $\\cP$). For any $k \\geq k_0$, we define\n\\begin{equation*}\n\t\\delta_k \\coloneqq \\E_{i_{k_0},\\dotsc, i_{k-1}} \\left[\\left. F\\left( x^k\n\t\\right) - F^*\\right| x^{k_0}\\right].\n\\end{equation*}\nBy applying the definition \\eqref{eq:r0} and the bound \\eqref{eq:H2}\non the right-hand side of the updated \\eqref{eq:Q2},\nand then taking expectations on both sides over $i_{k_0},\\dotsc,\ni_{k-1}$ conditional on $x^{k_0}$, we have that\n\\begin{align}\n\\label{eq:Q4}\n\\E_{i_{k_0},\\dotsc, i_k}\\left[\\left. \\bar \\alpha_{i_k}\nQ^{k*}_{i_k} \\right| x^{k_0}\\right]\n\\leq -\\theta\\lambda \\delta_k +\n\\frac{\\theta^2\\lambda^2}{2}\\left\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2,\n\\end{align}\nfor all $\\lambda \\in [0,1]$ and all $\\theta \\in [0, \\bar \\pi]$.\n","sha256_utf8":"d9b8a1695a0023eeafeab021bcc59c712292e64a9f5754b10ac61edfdfe0defd"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"阈值后选择插值参数","source_units":[{"start":32187,"end":32899,"start_line":759,"end_line":778,"raw_tex":"Setting $\\theta = \\bar\\pi$ in \\eqref{eq:Q4}, we have from\n\\eqref{eq:k0} that since Algorithm~\\ref{alg:icd} is a descent method,\n$\\delta_k < \\bar\\pi \\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM \\right\\| R_0^2$,\nfor all $k \\ge k_0$.  Therefore, we can use\n\\begin{equation*}\n\t\\lambda = \\frac{\\delta_k}{\\bar\\pi\n\t\\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\| R_0^2}\n\\end{equation*}\nin \\eqref{eq:Q4} to obtain\n\\begin{align}\n\\nonumber\n\\E_{i_{k_0},\\dotsc,i_k} \\left[ \\left. \\bar \\alpha_{i_k}\nQ^{k*}_{i_k}\\right| x^{k_0}\\right]\n\\leq&~ -\\bar\\pi \\frac{\\delta_k^2\n}{2 \\bar\\pi \\left\\| \\cP^{-1} \\bar\n\\AA^{-1} \\cM \\right\\| R_0^2}\\\\\n=&~ -\\frac{\\delta_k^2}{2 \\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2}.\n\\label{eq:newbound}\n\\end{align}\n","sha256_utf8":"d6b35e6b256dbb2c4c1cbd21feb074f477c9ef5d8a942ec586b690c8c5885ab3"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"建立误差二次递推","source_units":[{"start":32899,"end":33226,"start_line":779,"end_line":787,"raw_tex":"Therefore, by taking expectation on \\eqref{eq:usage} over\n$i_{k_0},\\dotsc, i_k$ conditional on $x^{k_0}$, and using\n\\eqref{eq:newbound}, we obtain\n\\begin{equation}\n\t\\delta_{k+1} \\leq \\delta_k - \\frac{\\left( 1 - \\eta\n\t\\right)\\gamma\\delta_k^2}{2\\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2}.\n\\label{eq:deltak}\n\\end{equation}\n","sha256_utf8":"3c6739d71e232a6f00ed33590b18226cd46c62a0e8cf96e627f30a6ced593bdb"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"利用单调性转为倒数递推","source_units":[{"start":33226,"end":33769,"start_line":788,"end_line":799,"raw_tex":"By dividing both sides of \\eqref{eq:deltak} by $\\delta_k \\delta_{k+1}$\nand noting from \\eqref{eq:armijo} and Lemma \\ref{lemma:linesearch}\nthat $\\{F(x_k)\\}$ and therefore $\\{\\delta_k\\}$ is descending, we obtain\n\\begin{equation}\n\\frac{1}{\\delta_{k}}\n\\leq \\frac{1}{\\delta_{k+1}} - \\frac{\\left( 1 - \\eta\n\\right)\\gamma\\delta_k}{2\\delta_{k+1} \\left\\| \\cP^{-1} \\bar \\AA^{-1}\n\\cM\\right\\| R_0^2}\n\\leq \\frac{1}{\\delta_{k+1}} - \\frac{\\left( 1 - \\eta\n\\right)\\gamma}{ 2\\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\| R_0^2}.\n\\label{eq:telescpoe}\n\\end{equation}\n","sha256_utf8":"f7fba29878280ac5296a7e9cb28469f665da83da54937b4dbd4d8138dd149d2a"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"倒数递推求和","source_units":[{"start":33769,"end":34040,"start_line":800,"end_line":806,"raw_tex":"By summing and telescoping \\eqref{eq:telescpoe}, we obtain\n\\begin{equation}\n\t\\frac{1}{\\delta_k} \\geq \\frac{1}{\\delta_{k_0}} +\n\\left( k - k_0\\right) \\frac{\\left( 1 - \\eta\n\\right)\\gamma }{2\\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\| R_0^2}.\n\\label{eq:summed}\n\\end{equation}\n","sha256_utf8":"7c6c11ac355a7322b1655afee7e33ee2b2b48ae1698bb87135c56c482c5a8a79"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"控制转折点初始误差","source_units":[{"start":34040,"end":34367,"start_line":807,"end_line":815,"raw_tex":"Finally, note that because $\\bar{\\alpha}_i \\in [0,1]$ for\n$i=1,\\dotsc,N$, \\eqref{eq:k0} implies that\n\\begin{equation}\n\t\\frac{1}{\\delta_{k_0}} \\geq \\frac{1}{\\bar\\pi \\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\| R_0^2}\n\t\\geq\n\t\\frac{1}{\\min_i  p_i \\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2}.\n\\label{eq:deltak0}\n\\end{equation}\n","sha256_utf8":"a8ed200fabb1381307fa0bd280d36d1c4e6cb4d23e55a0494120661a8e53f181"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"由概率归一化得到块数因子","source_units":[{"start":34367,"end":34822,"start_line":816,"end_line":827,"raw_tex":"Next, it is straightforward that the solution to\n\t\\[\n\t\t\\min_{p_1,\\dotsc,p_N} \\, \\frac{1}{\\min_{1 \\le i \\le N} p_i}\n\\quad\t\t\\text{subject to}\\;\\; \\sum_{i=1}^N p_i = 1,  \n\t\t\\;\\; p_i \\geq 0, \\;\\; i=1,\\dotsc,N\n\t\\]\nis $p_i \\equiv 1 / N$, and the corresponding objective value is $N$.\nTherefore, \\eqref{eq:deltak0} {further implies} that\n\\begin{equation*}\n\t\\frac{1}{\\delta_{k_0}} \\geq\n\t\\frac{N}{\\left\\| \\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\| R_0^2}.\n\\end{equation*}\n","sha256_utf8":"3a68c3bcaf6b6e8aea89e86bfadec5d6d49a3225ec1b72cb413fde0892330f8a"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"完成条件次线性界","source_units":[{"start":34822,"end":34901,"start_line":828,"end_line":829,"raw_tex":"By combining this inequality with \\eqref{eq:summed}, we obtain\n\\eqref{eq:sub}.\n","sha256_utf8":"a1403776cd116fcb1dcb93be0dc24b0a7a8958b6981bd5b065299dcd2733bb3f"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"全程固定采样的模型界","source_units":[{"start":34902,"end":35431,"start_line":831,"end_line":844,"raw_tex":"{For Part 3, we again start from \\eqref{eq:Q2} and\n replace $\\alpha^k_{i_k}$ with $\\bar{\\alpha}_{i}$ in\n\\eqref{eq:Q2} to obtain\n\\begin{align}\n\\label{eq:P3Q1}\n\t&~\\E_{i_k}\\left[\\left.\\bar{\\alpha}_{i_k} Q^{k*}_{i_k}\\right|\nx^k \\right]\\\\\n\\nonumber\n\\leq&~ \\theta\\lambda \\left(F^* - F\\left(x^k\\right) \\right)+\n\\frac{\\theta^2\\lambda^2}{2} \\left(x^k -\nP_\\Omega\\left(x^k\\right)\\right)^\\top \\cP^{-1}\\bar\\AA^{-1} \\HH_k \\left(x^k\n- P_\\Omega\\left(x^k\\right)\\right),\n\\end{align}\nfor all $\\lambda \\in [0,1]$ and all $\\theta \\in [0, \\bar \\pi]$.\n","sha256_utf8":"9016c04bc0e8eac15fd749674318e95149b24dac83ff68dc8ab26224aa0f235a"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"统一控制投影距离项","source_units":[{"start":35431,"end":35932,"start_line":845,"end_line":858,"raw_tex":"By applying \\eqref{eq:r0} and \\eqref{eq:H2}, we have\n\\begin{align*}\n\t&~\\E_{i_k}\\left[\\left.\\bar{\\alpha}_{i_k} Q^{k*}_{i_k}\\right|\nx^k \\right]\\\\\n\\nonumber\n\\leq&~ \\theta\\lambda \\left(F^* - F\\left(x^k\\right) \\right)+\n\\frac{\\theta^2\\lambda^2}{2} \\left\\|x^k -\nP_\\Omega\\left(x^k\\right)\\right\\| \\|\\cP^{-1}\\bar\\AA^{-1}\n\\cM\\| \\left\\|x^k\n- P_\\Omega\\left(x^k\\right)\\right\\|\\\\\n\\leq&~ \\theta\\lambda \\left(F^* - F\\left(x^k\\right) \\right)+\n\\frac{\\theta^2\\lambda^2}{2} \\|\\cP^{-1}\\bar\\AA^{-1}\n\\cM\\|R_0^2.\n\\end{align*}\n","sha256_utf8":"3a4c5407127c573f1c1d7de936a021d3b7adf072d4bd69ba16bfe52211d68d0b"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"对全部历史取期望","source_units":[{"start":35932,"end":36760,"start_line":859,"end_line":883,"raw_tex":"Now we take expectation over $i_0,\\dotsc,i_{k-1}$ on both sides of\nthis inequality (noting that the last term on the right-hand side are\nall constants that do not depend on $i_k$) to obtain\n\\[\n\\E_{i_0,\\dotsc,i_k}\\left[\\bar \\alpha_{i_k}\nQ^{k*}_{i_k}\\right]\n\\leq -\\theta\\lambda \\E_{i_0,\\dotsc,i_{k-1}} \\left[ F^* - F\\left( x^k\n\\right) \\right] +\n\\frac{\\theta^2\\lambda^2}{2}\\left\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2.\n\\]\nBy defining\n\\begin{equation*}\n\t\\hat \\delta_k \\coloneqq \\E_{i_{0},\\dotsc, i_{k-1}} \\left[ F\\left( x^k\n\t\\right) - F^*\\right]\n\\end{equation*}\nand setting $\\theta = \\bar{\\pi}$, we have that\n\\begin{align}\n\\E_{i_0,\\dotsc,i_k}\\left[\\bar \\alpha_{i_k}\nQ^{k*}_{i_k}\\right]\n\\leq -\\bar\\pi\\lambda \\hat \\delta_k +\n\\frac{\\bar \\pi^2\\lambda^2}{2}\\left\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2.\n\\label{eq:P3Q2}\n\\end{align}\n","sha256_utf8":"7942ba3d0de5a0d5e6ea46ea8f5ab405ed749f57280b4a692987738d5a688702"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"按期望误差选择最优参数","source_units":[{"start":36760,"end":37361,"start_line":884,"end_line":902,"raw_tex":"The minimum of the right-hand side happens when\n\\[\n\t\\lambda = \\min\\left\\{ 1, \\frac{\\hat \\delta_k }{\\bar{\\pi}\\|\\cP^{-1} \\bar\n\t\\AA^{-1} \\cM\\|R_0^2}\\right\\}.\n\\]\nWhen the expected function value satisfies\n\\begin{equation}\n\t\\hat \\delta_k  \\geq \\bar{\\pi}\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\|R_0^2,\n\\label{eq:turning}\n\\end{equation}\nthe minimizer is $\\lambda = 1$, and the bound becomes\n\\begin{align*}\n\\E_{i_0,\\dotsc,i_k}\\left[\\bar \\alpha_{i_k}\nQ^{k*}_{i_k}\\right]\n\\leq -\\bar\\pi\\hat \\delta_k +\n\\frac{\\bar \\pi^2}{2}\\left\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\right\\|\nR_0^2\n\\leq -\\frac{\\bar \\pi \\hat \\delta_k}{2}.\n\\end{align*}\n","sha256_utf8":"ce1a7a0385d79a798b804bf8df099f82704cffc438bdaca143fcfec6dccfecb5"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"期望误差的线性递推","source_units":[{"start":37361,"end":37966,"start_line":903,"end_line":915,"raw_tex":"Now we consider \\eqref{eq:usage} and take expectation over $i_0,\\dotsc,\ni_k$ on both sides. Note that $Q^{k*}_{i_k} \\leq 0$ so the upper bound is still\nvalid if we replace $\\alpha^k_{i_k}$ with $\\bar \\alpha_{i_k}$.\nThus we obtain\n\\begin{align} \\label{eq:P3Q3}\n\t\\hat \\delta_{k+1} - \\hat{\\delta}_{k} =\n        \\E_{i_0,\\dotsc,i_k}\\left[F\\left( x^{k+1} \\right) - F\\left( x^k\n          \\right)\\right] & \\leq (1 - \\eta) \\gamma \\E_{i_0,\\dotsc,i_k}\n        \\left[ \\bar \\alpha_{i_k} Q^{k*}_{i_k}\\right] \\\\\n        \\nonumber\n        & \\leq -\\frac{ (1 -\n          \\eta) \\gamma\\bar \\pi \\hat \\delta_k}{2}.\n\\end{align}\n","sha256_utf8":"23a06448e154eebbde1f99ac8af39923f76886a70db53b5040bbd1dd20172ace"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"迭代得到早期几何界","source_units":[{"start":37966,"end":38362,"start_line":916,"end_line":927,"raw_tex":"By rearranging the inequality above, we get the linear convergence of\n\\[\n\t\\hat \\delta_{k+1} \\leq \\hat{\\delta}_{k} \\left(1 -\\frac{(1 - \\eta)\n\t\t\\gamma \\bar \\pi}{2}\\right).\n\\]\nTherefore, we always get the bound\n\\[\n\t\\hat \\delta_k \\leq \\left( 1 - \\frac{(1 - \\eta) \\gamma \\bar \\pi\n\t}{2}  \\right)^k \\left( F(x^0) - F^* \\right)\n\\]\nuntil $\\hat \\delta_{k-1} <\\bar{\\pi}\\|\\cP^{-1} \\bar \\AA^{-1}\n\\cM\\|R_0^2$.\n","sha256_utf8":"af2acb27d9ec4da99bb3007ecdaf5779fff6ed92a87d70d249804a6415187d02"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"确定可计算的阶段转折点","source_units":[{"start":38362,"end":38929,"start_line":928,"end_line":939,"raw_tex":"Note that $\\bar k_0$ is obtained as the first value of $k$ such that\n\\[\n\t\\left( 1 - \\frac{(1 - \\eta) \\gamma \\bar{\\pi}}{2} \\right)^k \\left(\n\tF\\left( x^0 \\right) - F^* \\right) \\leq  \\bar{\\pi}\\|\\cP^{-1} \\bar\n\\AA^{-1} \\cM\\|R_0^2.\n\\]\nTherefore, for $k < \\bar k_0$, the upper bound in \\eqref{eq:upper1} is\nlarger than $\\bar{\\pi}\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\|R_0^2$, so the\nrate \\eqref{eq:upper1} remains valid.\nNote that if $\\hat \\delta_k \\leq  \\bar{\\pi}\\|\\cP^{-1} \\bar\n\\AA^{-1} \\cM\\|R_0^2$ has already held true for some $k < \\bar k_0$,\nclearly this bound is still valid.\n","sha256_utf8":"a9f6fb0e002999d21c5034359b266eed6b5746344300969153a68805f54b47c4"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"转折后建立二次误差递推","source_units":[{"start":38929,"end":39350,"start_line":940,"end_line":950,"raw_tex":"On the other hand,\nafter $\\bar k_0$, we are guaranteed that \\eqref{eq:turning} must stop\nholding.\nThus the minimizer for \\eqref{eq:P3Q2} becomes $\\lambda = \\hat\n\\delta_k / (\\bar{\\pi}\\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\|R_0^2)$.\nWe then start from the first inequality of \\eqref{eq:P3Q3}\nand get\n\\[\n\t\\hat{\\delta}_{k+1} - \\hat{\\delta}_k \\leq -\\frac{(1 - \\eta) \\gamma\n\t\t\\hat{\\delta}_k^2}{2 \\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\|R_0^2}.\n\\]\n","sha256_utf8":"8f4c3832182524412b846fafc726bf918845046afa761ca77dffd43daafb51a0"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"倒数求和完成全程收敛界","source_units":[{"start":39350,"end":39843,"start_line":951,"end_line":965,"raw_tex":"Following the same derivation we had in Part 2, we can get\n\\[\n\t\\frac{1}{ \\hat{\\delta}_k} \\leq \\frac{1}{\\hat{\\delta}_{k+1}} -\n\t\\frac{(1 - \\eta) \\gamma }{2 \\|\\cP^{-1} \\bar \\AA^{-1} \\cM\\|R_0^2}.\n\\]\nBy summing and telescoping the result above, we get\n\\[\n\t\\frac{1}{ \\hat{\\delta}_k} \\geq \\frac{1}{\\hat{\\delta}_{\\bar k_0}} + (k -\n\t\\bar k_0) \\frac{(1 - \\eta) \\gamma }{2 \\|\\cP^{-1} \\bar \\AA^{-1}\n\\cM\\|R_0^2}.\n\\]\nFollowing the same argument in Part 2, we get the final claim in Part\n3.}\n\\qed\n\\end{proof}","sha256_utf8":"83bcc42b50b43039fbe448970558196fd6bf519d5d3cedc28307b78d1052486f"}]},{"id":"S043","semantic_type":"assumption","kind":"assumption","title":"目标凸性与统一正定条件","source_units":[{"start":25685,"end":26137,"start_line":596,"end_line":605,"raw_tex":"Assume that $f$ and $\\psi$ are convex and \\eqref{eq:compL} holds.\nSuppose that at all iterations $k$ of Algorithm~\\ref{alg:icd},\n{and for any choice $i=i_k$ of the update block at iteration\n  $k$, we have that \\eqref{eq:approx} is satisfied with a fixed $\\eta\n  \\in [0,1)$, with $H^k_i$ chosen such that}\n\\begin{equation}\nH^k_i \\succeq m_i I,\\quad k=0,1,\\dotsc,\n\\label{eq:H}\n\\end{equation}\nfor some $ m_i > 0$ for all $i$. Then the following are true.\n","sha256_utf8":"01a76cd12769e2df34d0664f82bb08b18025b259f81766ed5909c64eada17242"}]},{"id":"S044","semantic_type":"assumption","kind":"assumption","title":"单步采样与大误差条件","source_units":[{"start":26155,"end":26885,"start_line":607,"end_line":622,"raw_tex":"\\item At iteration $k$, given any probability distribution\n  $\\{p^k_i\\}_{i=1}^N > 0$ for choosing the update block $i_k$, denote\n  by $\\{\\alpha^k_i\\}_{i=1}^N > 0$ the step sizes generated by the\n  backtracking line search for each possible choice\n  $i=1,2,\\dotsc,N$. (These step sizes are guaranteed to be bounded\n  away from zero, by Lemma~\\ref{lemma:linesearch}.) Define\n\\begin{equation} \\label{eq:pik}\n\t\\pi^k \\coloneqq \\min_{1 \\le i \\le N} \\alpha^k_i p^k_i,\n\\end{equation}\n{and let $\\cP_k$, $\\AA_k$, and $\\HH_k$ be defined as in\nLemma~\\ref{lemma:Q}.} If\n\\[\n  F\\left( x^k \\right) - F^* \\geq \\left(x^k -\n  P_\\Omega\\left(x^k\\right)\\right)^\\top  \\cP^{-1}_k \\AA_k^{-1}\n  \\HH_k\\left(x^k - P_\\Omega\\left(x^k\\right)\\right) \\pi^k,\n  \\]\n","sha256_utf8":"b9813c88be8d36b250130a47d7faf7351e5a31a1ce462397b971cbbf701e3955"}]},{"id":"S045","semantic_type":"assumption","kind":"assumption","title":"转折后固定采样与度量界","source_units":[{"start":27241,"end":28197,"start_line":631,"end_line":658,"raw_tex":"\\item {Given $M_i \\geq m_i, i=1,\\dotsc,N,$}\nand define\n\\begin{equation} \\label{eq:def.M}\n\\cM \\coloneqq \\diag(M_1 I_{n_1},\\dotsc, M_N I_{n_N}),\\quad \\bar \\AA\n\\coloneqq \\diag(\\bar{\\alpha}_1 I_{n_1},\\dotsc, \\bar{\\alpha}_N\nI_{n_N}),\n\\end{equation}\n where $\\bar\n\\alpha_i$ are defined in Lemma \\ref{lemma:linesearch}.  Given a\nprobability distribution $\\{p_i\\}_{i=1}^N > 0$, define\n\\begin{equation} \\label{eq:barpi}\n\\cP \\coloneqq \\diag(p_1 I_{n_1},\\dotsc, p_N I_{n_N}),\\quad\n  \\bar\\pi\n  \\coloneqq \\min_{1 \\le i \\le N} \\, \\bar\\alpha_i p_i,\n\\end{equation}\nand let\n\\begin{equation}\n\tk_0 \\coloneqq \\arg \\min\\left\\{ k: F\\left( x^k \\right) - F^* <\n\t\t\\|\\cP^{-1} \\bar{\\AA}^{-1} \\cM\\| \\bar\\pi R_0^2\\right\\}.\n\\label{eq:k0}\n\\end{equation}\n{Suppose that for all $k \\geq k_0$, the sampling of $i_k$\n  follows the distribution $\\{p_i\\}$, which does not depend on $k$,\n  and }\n\\begin{equation}\nM_i I \\succeq H^k_i \\succeq m_i I,\\quad i=1,\\dotsc,N.\n\\label{eq:H2}\n\\end{equation}\n","sha256_utf8":"c23a22b462a8793718e5bd8967c16825f247b1272b6f708f7bc66ddaf136c42f"}]},{"id":"S046","semantic_type":"assumption","kind":"assumption","title":"全程固定采样与转折点定义","source_units":[{"start":28518,"end":29006,"start_line":668,"end_line":678,"raw_tex":"\\item Suppose that a fixed probability distribution $\\{p_i\\}_{i=1}^N >\n  0$ is used throughout to sample the blocks and that \\eqref{eq:H2}\n  holds for all $k$. {Then, defining}\n  \\begin{equation}\n    \\bar k_0 \\coloneqq \\left\\lceil\\max\\left\\{ 0, \\frac{\\log\n      \\frac{F\\left( x^0 \\right) - F^*}{\\left\\| \\cP^{-1}\n\t\\bar{\\AA}^{-1} \\cM\\right\\| \\bar\\pi\n\tR_0^2}}{\\log\\left( \\frac{2}{2 - \\left( 1 - \\eta\\right) \\gamma\n\t\\bar\\pi}\\right)}\\right\\}\\right\\rceil,\n    \\label{eq:k0bar}\n  \\end{equation}\n","sha256_utf8":"8f0a13c2c3098610fee2d70824eb365c428d2a06c893f11bbd4ef2b9e7403207"}]},{"id":"S047","semantic_type":"assumption","kind":"assumption","title":"模型引理的凸性与矩阵条件","source_units":[{"start":19201,"end":19729,"start_line":447,"end_line":457,"raw_tex":"{Let $f$ and $\\psi$ be convex with $F$ satisfying\n  \\eqref{eq:strong} for some $\\mu \\geq 0$. At iteration $k$, we\n  consider matrices $H^k_i \\succeq 0$ with $H_i^k \\in \\R^{n_i \\times\n    n_i}$, $i=1,\\dotsc,N$, probability distribution $\\{p_i^k\\}_{i=1}^N\n  > 0$, and step sizes $\\{\\alpha^k_i\\}_{i=1}^N > 0$. We define}\n\\begin{align*}\n\\cP_k &\\coloneqq \\diag(p^k_1 I_{n_1},\\dotsc, p^k_N I_{n_N}),\\quad\n\\AA_k \\coloneqq \\diag(\\alpha_1^k\nI_{n_1},\\dotsc, \\alpha_N^k I_{n_N}),\\\\\n\\HH_k &\\coloneqq \\diag(H_1^k,\\dotsc,H_N^k).\n\\end{align*}\n","sha256_utf8":"c6b42334266cd79358836f887ccfa42cc79b2b1af02b3cb659b2eacdff8e63cb"},{"start":19794,"end":19912,"start_line":459,"end_line":460,"raw_tex":"for all $\\lambda \\in \\left[0,1\\right]$ and all $\\theta$ such that\n$0\\leq\\theta\\leq \\alpha^k_i p^k_i$, $i=1,\\dotsc,N$:\n","sha256_utf8":"a193a30144b3a45df513149975eb2a16a5e3e2310e71342f84773ea48782620f"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S016","kind":"supports"},{"source_id":"S003","target_id":"S018","kind":"supports"},{"source_id":"S004","target_id":"S016","kind":"supports"},{"source_id":"S004","target_id":"S012","kind":"supports"},{"source_id":"S005","target_id":"S016","kind":"supports"},{"source_id":"S005","target_id":"S022","kind":"supports"},{"source_id":"S006","target_id":"S012","kind":"supports"},{"source_id":"S006","target_id":"S022","kind":"supports"},{"source_id":"S007","target_id":"S012","kind":"supports"},{"source_id":"S007","target_id":"S022","kind":"supports"},{"source_id":"S008","target_id":"S022","kind":"supports"},{"source_id":"S008","target_id":"S027","kind":"supports"},{"source_id":"S009","target_id":"S016","kind":"supports"},{"source_id":"S009","target_id":"S026","kind":"supports"},{"source_id":"S010","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S020","kind":"supports"},{"source_id":"S010","target_id":"S023","kind":"supports"},{"source_id":"S011","target_id":"S026","kind":"supports"},{"source_id":"S011","target_id":"S035","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S026","kind":"supports"},{"source_id":"S012","target_id":"S027","kind":"supports"},{"source_id":"S012","target_id":"S029","kind":"supports"},{"source_id":"S012","target_id":"S034","kind":"supports"},{"source_id":"S012","target_id":"S038","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S023","kind":"supports"},{"source_id":"S047","target_id":"S016","kind":"supports"},{"source_id":"S047","target_id":"S017","kind":"supports"},{"source_id":"S047","target_id":"S018","kind":"supports"},{"source_id":"S047","target_id":"S020","kind":"supports"},{"source_id":"S016","target_id":"S019","kind":"supports"},{"source_id":"S017","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S015","kind":"supports"},{"source_id":"S018","target_id":"S015","kind":"supports"},{"source_id":"S019","target_id":"S015","kind":"supports"},{"source_id":"S020","target_id":"S015","kind":"supports"},{"source_id":"S043","target_id":"S022","kind":"supports"},{"source_id":"S043","target_id":"S023","kind":"supports"},{"source_id":"S044","target_id":"S024","kind":"supports"},{"source_id":"S044","target_id":"S025","kind":"supports"},{"source_id":"S045","target_id":"S026","kind":"supports"},{"source_id":"S045","target_id":"S027","kind":"supports"},{"source_id":"S045","target_id":"S031","kind":"supports"},{"source_id":"S045","target_id":"S035","kind":"supports"},{"source_id":"S046","target_id":"S034","kind":"supports"},{"source_id":"S046","target_id":"S036","kind":"supports"},{"source_id":"S046","target_id":"S040","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S025","kind":"supports"},{"source_id":"S021","target_id":"S028","kind":"supports"},{"source_id":"S021","target_id":"S038","kind":"supports"},{"source_id":"S021","target_id":"S041","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S030","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S023","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S036","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S029","target_id":"S042","kind":"supports"},{"source_id":"S030","target_id":"S042","kind":"supports"},{"source_id":"S032","target_id":"S042","kind":"supports"},{"source_id":"S023","target_id":"S002","kind":"supports"},{"source_id":"S024","target_id":"S002","kind":"supports"},{"source_id":"S025","target_id":"S002","kind":"supports"},{"source_id":"S026","target_id":"S002","kind":"supports"},{"source_id":"S027","target_id":"S002","kind":"supports"},{"source_id":"S028","target_id":"S002","kind":"supports"},{"source_id":"S029","target_id":"S002","kind":"supports"},{"source_id":"S030","target_id":"S002","kind":"supports"},{"source_id":"S031","target_id":"S002","kind":"supports"},{"source_id":"S032","target_id":"S002","kind":"supports"},{"source_id":"S033","target_id":"S002","kind":"supports"},{"source_id":"S034","target_id":"S002","kind":"supports"},{"source_id":"S035","target_id":"S002","kind":"supports"},{"source_id":"S036","target_id":"S002","kind":"supports"},{"source_id":"S037","target_id":"S002","kind":"supports"},{"source_id":"S038","target_id":"S002","kind":"supports"},{"source_id":"S039","target_id":"S002","kind":"supports"},{"source_id":"S040","target_id":"S002","kind":"supports"},{"source_id":"S041","target_id":"S002","kind":"supports"},{"source_id":"S042","target_id":"S002","kind":"supports"}]}}
{"record_id":"1810.02263v4","paper":{"title":"Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimization","authors":["Anas Barakat","Pascal Bianchi"],"arxiv_id":"1810.02263","version":"v4","journal":"SIAM Journal on Optimization","doi":"10.1137/19m1263443","abstract_url":"https://arxiv.org/abs/1810.02263v4","pdf_url":"https://arxiv.org/pdf/1810.02263v4"},"context":{"raw_tex":"\\noindent{\\bf Notations}. If $x$, $y$ are two vectors on $\\bR^d$ for some $d\\geq 1$, we denote by $x \\odot y$, $x^{\\odot 2}$, $x/y$, $|x|$, $\\sqrt{|x|}$ the vectors\non $\\bR^d$ whose $i$-th coordinates are respectively given by $x_iy_i$, $x_i^2$, $x_i/y_i$, $|x_i|$, $\\sqrt{|x_i|}$.\nInequalities of the form $x\\leq y$ are read componentwise.\nDenote by $\\|\\cdot\\|$ the standard Euclidean norm.\nFor any vector $v\\in (0,+\\infty)^d$, write $\\|x\\|^2_v = \\sum_i v_i x_i^2$.\nNotation $A^T$ represents the transpose of a matrix $A$.\nIf  $z\\in \\bR^d$ and $A$ is a non-empty subset of $\\bR^d$,\nwe use the notation $\\mathsf d(z,A) \\eqdef \\inf\\{ \\|z-z'\\| :z'\\in A\\}$.\nIf $A$ is a set, we denote  by $\\1_A$ the function equal to one on that set and to zero elsewhere.\n We denote by $C([0,+\\infty),\\bR^d)$ the space of continuous functions from $[0,+\\infty)$ to $\\bR^d$ endowed with the topology of\nuniform convergence on compact intervals.\n\n\nLet $(\\Omega,\\cF,\\bP)$ be a probability space, and let\n$(\\Xi,\\mathfrak{S})$ denote an other measurable space.  Consider a\nmeasurable map $f:\\bR^d\\times \\Xi\\to \\bR$, where $d$ is an integer.\nFor a fixed value of $\\xi$, the mapping $x\\mapsto f(x,\\xi)$ is\nsupposed to be differentiable, and its gradient w.r.t. $x$ is denoted\nby $\\nabla f(x,\\xi)$.  Define $\\cZ\\eqdef \\bR^d\\times\\bR^d\\times\n\\bR^d$, $\\cZ_{+}\\eqdef \\bR^d\\times\\bR^d\\times [0,+\\infty)^d$ and\n$\\cZ_{+}^*\\eqdef \\bR^d\\times\\bR^d\\times (0,+\\infty)^d$. \\adam\\\ngenerates a sequence $z_n\\eqdef (x_n,m_n,v_n)$ on\n$\\cZ_+$ given by Algorithm~\\ref{alg:adam}.\n\n\n\\begin{assumption} \\label{hyp:model}\nThe mapping $f:\\bR^d\\times\\Xi\\to \\bR$ satisfies the following.\n  \\begin{enumerate}[{\\sl i)}]\n  \\item For every $x\\in \\bR^d$, $f(x,\\,.\\,)$ is $\\mathfrak{S}$-measurable.\n  \\item For almost every $\\xi$, the map $f(\\,.\\,,\\xi)$ is continuously differentiable.  \\label{hyp:dif}\n  \\item There exists $x_*\\in \\bR^d$ s.t. $\\bE(|f(x_*,\\xi)|)<\\infty$ and  $\\bE(\\|\\nabla f(x_*,\\xi)\\|^2)<\\infty$.\n  \\item For every compact subset $K\\subset \\bR^d$, there exists $L_K>~0$ such that\n   for every $(x,y)\\in K^2$, $\\bE(\\|\\nabla f(x,\\xi)-\\nabla f(y,\\xi)\\|^2)\\leq L_K^2\\|x-y\\|^2$.\n  \\end{enumerate}\n\\end{assumption}\n\n\\begin{equation}\n  F(x) \\eqdef \\bE(f(x,\\xi)) \\quad \\text{and} \\quad S(x) \\eqdef \\bE(\\nabla f(x,\\xi)^{\\odot 2})\\label{eq:F_and_S}\n\\end{equation}\n\nare well defined; $F$ is continuously differentiable and by\nLebesgue's dominated convergence theorem, $\\nabla F(x) = \\bE(\\nabla\nf(x,\\xi))$ for all $x$. Moreover, $\\nabla F$ and $S$ are locally Lipschitz continuous.\n\n\n\\begin{assumption}\n  \\label{hyp:S>0}\nFor every $x\\in \\bR^d$, $S(x)>0$.\n\\end{assumption}\n\n$F$, denoted by $$\\cS \\eqdef \\nabla F^{-1}(\\{0\\}),$$ is non-empty.\n\n\n\\begin{assumption}\n  \\label{hyp:iid}\nThe sequence $(\\xi_n:n\\geq 1)$ is iid, with the same distribution as~$\\xi$.\n\\end{assumption}\n\n\\begin{assumption}\n  \\label{hyp:moment-f}\nLet $p>0$. Assume either one of the following conditions.\n\\begin{enumerate}[i)]\n\\item \\label{momentegal} For every compact set $K\\subset \\bR^d$, $\\sup_{x\\in K} \\bE(\\|\\nabla f(x,\\xi)\\|^{p})<\\infty\\,.$\n\\item \\label{momentreinforce} For every compact set $K\\subset \\bR^d$,  $\\exists\\, p_K>p$, $\\sup_{x\\in K} \\bE(\\|\\nabla f(x,\\xi)\\|^{p_K})<\\infty.$\n\\end{enumerate}\n\\end{assumption}\n\n\\begin{algorithm}[tb]\n   \\caption{\\bf \\adam - decreasing stepsize $(((\\gamma_n,\\alpha_n,\\beta_n):n\\in \\bN^*), \\varepsilon)$.}\n   \\label{alg:adam-decreasing}\n\\begin{algorithmic}\n  \\STATE {\\bfseries Initialization:} $x_0\\in \\bR^d, m_0=0$, $v_0=0$, $r_0=\\bar r_0=0$.\n   \\FOR{$n=1$ {\\bfseries to} $n_{\\text{iter}}$}\n   \\STATE $m_n = \\alpha_n m_{n-1} + (1-\\alpha_n)  \\nabla f(x_{n-1},\\xi_n)$\n   \\STATE $v_n = \\beta_n v_{n-1} + (1-\\beta_n) \\nabla f(x_{n-1},\\xi_n)^{\\odot 2}$\n   \\STATE $r_n = \\alpha_n r_{n-1} + (1-\\alpha_n)$\n   \\STATE $\\bar r_n = \\beta_n \\bar r_{n-1} + (1-\\beta_n)$\n  \\STATE $\\hat m_{n} = m_{n}/r_n$\n   \\COMMENT{bias correction step}\n   \\STATE $\\hat v_{n} = v_{n}/\\bar r_n$\n   \\COMMENT{bias correction step}\n   \\STATE $x_{n} = x_{n-1} - \\gamma_n \\hat m_{n} / (\\varepsilon+\\sqrt{\\hat v_{n}}) \\,.$\n   \\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}\n\n  \\item For all $n \\in \\bN$, $0 \\leq \\alpha_n \\leq 1$ and $0 \\leq \\beta_n \\leq 1$,\n  \\item There exist $a,b$ s.t. $0<b<4a$,\\,\n  $\\gamma_n^{-1}(1-\\alpha_n) \\to a$ and $\\gamma_n^{-1}(1-\\beta_n) \\to b$\\,. %as $n \\to \\infty$\\,.\n\n\n\\begin{assumption} %[local regularity of the mean field $h_\\infty$]\n   \\label{hyp:mean_field_tcl}\nLet $x^*\\in \\cS$. There exists a neighborhood $\\cV$ of $x^*$ s.t.\n\\begin{enumerate}[{\\sl i)}]\n\\item $F$ is twice continuously differentiable on $\\cV$,\n  and the Hessian $\\nabla^2 F(x^*)$ of $F$ at $x^*$ is positive\n  definite.\n\\item $S$ is continuously differentiable on $\\cV$.\n\\end{enumerate}\n \\end{assumption}\n\nDefine $\nD \\eqdef \\textrm{diag}\\left((\\varepsilon + \\sqrt{S_1(x^*)})^{-1},\\cdots,(\\varepsilon + \\sqrt{S_d(x^*)})^{-1}\\right)\\,.\n$\nLet $P$ be an orthogonal matrix s.t. the following spectral decomposition holds:\n$$\nD^{1/2}\\nabla^2F(x^*)D^{1/2} = P\\textrm{diag}(\\lambda_1,\\cdots,\\lambda_d)P^{-1}\\,,\n$$\nwhere $\\lambda_1, \\cdots,\\lambda_d$ are the (positive) eigenvalues of\n$D^{1/2} \\nabla^2F(x^*)D^{1/2}$.\nDefine\n\\begin{equation}\nH \\eqdef\n\\begin{pmatrix}\n  0 & -D & 0\\\\ a\\nabla^2F(x^*) & -a I_d & 0 \\\\ b \\nabla S(x^*) & 0 & -b I_d\n\\end{pmatrix}\\label{eq:H}\n\\end{equation}\n\nwhere $I_d$ represents the $d\\times d$ identity matrix and $\\nabla S(x^*)$ is the\nJacobian matrix of~$S$ at $x^*$.\nThe largest real part of the eigenvalues of $H$ coincides with $-L$, where\n\\begin{equation}\nL\\eqdef b\\wedge \\frac a2\\left( 1-\\sqrt{\\left(1-\\frac{4\\lambda_1}a\\right)\\vee 0}\\right) >0\\,.\\label{eq:L}\n\\end{equation}\n\nFinally, define the $3d\\times 3d$ matrix% covariance matrix of the\n% vector $(a\\nabla f(x^*,\\xi),b\\nabla f(x^*,\\xi)^{\\odot 2})$ \\emph{i.e.},\n\\begin{equation}\n  \\resizebox{.9\\hsize}{!}{$\nQ \\eqdef\n\\left( \\begin{array}[h]{cc}\n   0 &\n   \\begin{array}[h]{cc}\n     0 & \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ 0\n   \\end{array}\\\\\n   \\begin{array}[h]{c}\n     0 \\\\ 0\n   \\end{array}\n & \\bE\\left[\n  \\begin{pmatrix}\n    a\\nabla f(x^*,\\xi) \\\\ b(\\nabla f(x^*,\\xi)^{\\odot 2}-S(x^*))\n  \\end{pmatrix}\\begin{pmatrix}\n    a\\nabla f(x^*,\\xi) \\\\ b(\\nabla f(x^*,\\xi)^{\\odot 2}-S(x^*))\n  \\end{pmatrix}^T\\right]\n \\end{array}\\right)\\,.\\label{eq:Q}\n $}\n\\end{equation}\n\n\\begin{assumption}\n\\label{hyp:step-tcl}\nThe following holds.\n\\begin{enumerate}[{\\sl i)}]\n\\item \\label{step-tcl-i} There exist $\\kappa \\in (0,1]$, $\\gamma_0>0$, s.t. the sequence $(\\gamma_n)$\nsatisfies\n$\\gamma_n = {\\gamma_0}/{(n+1)^\\kappa}$ for all $n$.\nIf $\\kappa = 1$, we assume moreover that $\\gamma_0 > \\frac{1}{2L}$.\n\\item The sequences $\\left(\\frac{1}{\\gamma_n}(\\frac{1-\\alpha_n}{\\gamma_n}-a)\\right)$ and $\\left(\\frac{1}{\\gamma_n}(\\frac{1-\\beta_n}{\\gamma_n}-b)\\right)$ are bounded.\n%This is satisfied for instance if $1-\\alpha_n = a\\gamma_n$ and $1-\\beta_n = b\\gamma_n$.\n\\end{enumerate}\n\\end{assumption}\n\n\\noindent For an arbitrary sequence $(X_n)$ of random variables on some Euclidean space, a probability measure\n$\\mu$ on that space and an event $\\Gamma$ s.t. $\\bP(\\Gamma)>0$, we say that $X_n$ converges in distribution\nto $\\mu$ \\emph{given $\\Gamma$} if the measures $\\bP (X_n\\in \\cdot\\,|\\Gamma)$ converge weakly to $\\mu$.\n\n\n\\begin{equation}\n  \\label{eq:h_infty}\nh_{\\infty}((x,m,v)) = \\left(-m / (\\varepsilon+\\sqrt{v})\\,,\\, a (\\nabla F(x)-m) \\,,\\,b (S(x)-v) \\right)\\,.\n\\end{equation}\n\nWe recall that $\\cF_n$ is the $\\sigma$-algebra generated by the r.v. $(\\xi_k:1\\leq k\\leq n)$.\n\n\nIn this section, we denote by $\\bE_n = \\bE(\\cdot|\\cF_n)$ the conditional expectation w.r.t. $\\cF_n$.\nWe also use the notation $\\nabla f_{n+1} \\eqdef \\nabla f(x_n,\\xi_{n+1})$. %to simplify the equation.\n\n\nA similar lemma holds for the sequence $(\\bar{r}_n)$.\n","units":[{"start":8996,"end":9922,"start_line":210,"end_line":220,"raw_tex":"\\noindent{\\bf Notations}. If $x$, $y$ are two vectors on $\\bR^d$ for some $d\\geq 1$, we denote by $x \\odot y$, $x^{\\odot 2}$, $x/y$, $|x|$, $\\sqrt{|x|}$ the vectors\non $\\bR^d$ whose $i$-th coordinates are respectively given by $x_iy_i$, $x_i^2$, $x_i/y_i$, $|x_i|$, $\\sqrt{|x_i|}$.\nInequalities of the form $x\\leq y$ are read componentwise.\nDenote by $\\|\\cdot\\|$ the standard Euclidean norm.\nFor any vector $v\\in (0,+\\infty)^d$, write $\\|x\\|^2_v = \\sum_i v_i x_i^2$.\nNotation $A^T$ represents the transpose of a matrix $A$.\nIf  $z\\in \\bR^d$ and $A$ is a non-empty subset of $\\bR^d$,\nwe use the notation $\\mathsf d(z,A) \\eqdef \\inf\\{ \\|z-z'\\| :z'\\in A\\}$.\nIf $A$ is a set, we denote  by $\\1_A$ the function equal to one on that set and to zero elsewhere.\n We denote by $C([0,+\\infty),\\bR^d)$ the space of continuous functions from $[0,+\\infty)$ to $\\bR^d$ endowed with the topology of\nuniform convergence on compact intervals.\n","sha256_utf8":"b0e4fcd190d5001a636e4a43506c02880a9f0c96123895370d947f1d5221eb53"},{"start":10012,"end":10620,"start_line":227,"end_line":236,"raw_tex":"Let $(\\Omega,\\cF,\\bP)$ be a probability space, and let\n$(\\Xi,\\mathfrak{S})$ denote an other measurable space.  Consider a\nmeasurable map $f:\\bR^d\\times \\Xi\\to \\bR$, where $d$ is an integer.\nFor a fixed value of $\\xi$, the mapping $x\\mapsto f(x,\\xi)$ is\nsupposed to be differentiable, and its gradient w.r.t. $x$ is denoted\nby $\\nabla f(x,\\xi)$.  Define $\\cZ\\eqdef \\bR^d\\times\\bR^d\\times\n\\bR^d$, $\\cZ_{+}\\eqdef \\bR^d\\times\\bR^d\\times [0,+\\infty)^d$ and\n$\\cZ_{+}^*\\eqdef \\bR^d\\times\\bR^d\\times (0,+\\infty)^d$. \\adam\\\ngenerates a sequence $z_n\\eqdef (x_n,m_n,v_n)$ on\n$\\cZ_+$ given by Algorithm~\\ref{alg:adam}.\n","sha256_utf8":"d6729b07f5947a6b30ab579544621976f6d059eec12dcbae5886c0850d4ab49a"},{"start":12850,"end":13482,"start_line":284,"end_line":293,"raw_tex":"\\begin{assumption} \\label{hyp:model}\nThe mapping $f:\\bR^d\\times\\Xi\\to \\bR$ satisfies the following.\n  \\begin{enumerate}[{\\sl i)}]\n  \\item For every $x\\in \\bR^d$, $f(x,\\,.\\,)$ is $\\mathfrak{S}$-measurable.\n  \\item For almost every $\\xi$, the map $f(\\,.\\,,\\xi)$ is continuously differentiable.  \\label{hyp:dif}\n  \\item There exists $x_*\\in \\bR^d$ s.t. $\\bE(|f(x_*,\\xi)|)<\\infty$ and  $\\bE(\\|\\nabla f(x_*,\\xi)\\|^2)<\\infty$.\n  \\item For every compact subset $K\\subset \\bR^d$, there exists $L_K>~0$ such that\n   for every $(x,y)\\in K^2$, $\\bE(\\|\\nabla f(x,\\xi)-\\nabla f(y,\\xi)\\|^2)\\leq L_K^2\\|x-y\\|^2$.\n  \\end{enumerate}\n\\end{assumption}","sha256_utf8":"12ffff4ba701afd15b6a62175b6ca7470fc3ab63d210206e5d1f3fc14ec4a011"},{"start":13616,"end":13759,"start_line":297,"end_line":299,"raw_tex":"\\begin{equation}\n  F(x) \\eqdef \\bE(f(x,\\xi)) \\quad \\text{and} \\quad S(x) \\eqdef \\bE(\\nabla f(x,\\xi)^{\\odot 2})\\label{eq:F_and_S}\n\\end{equation}","sha256_utf8":"c5b7d9b7b07b7a1b0d9b4bc589d9243737307a9218ef020c9167d9c5b83af6fa"},{"start":13896,"end":14111,"start_line":304,"end_line":306,"raw_tex":"are well defined; $F$ is continuously differentiable and by\nLebesgue's dominated convergence theorem, $\\nabla F(x) = \\bE(\\nabla\nf(x,\\xi))$ for all $x$. Moreover, $\\nabla F$ and $S$ are locally Lipschitz continuous.\n","sha256_utf8":"d28b2d63f916afa785e6bd9336e320ab993bda6282ae8ec0717f4df1c0707776"},{"start":14413,"end":14500,"start_line":313,"end_line":316,"raw_tex":"\\begin{assumption}\n  \\label{hyp:S>0}\nFor every $x\\in \\bR^d$, $S(x)>0$.\n\\end{assumption}","sha256_utf8":"593b4e4b655d1693fa4ee8e77ad5b3bcb9a57edbb6609bcf9a7d52cd0c69a1f4"},{"start":14568,"end":14635,"start_line":318,"end_line":318,"raw_tex":"$F$, denoted by $$\\cS \\eqdef \\nabla F^{-1}(\\{0\\}),$$ is non-empty.\n","sha256_utf8":"c283b236ca3421d955ca8790e50a322f25df7ba4ab80e3b9797dddd52c5f9a92"},{"start":28766,"end":28895,"start_line":571,"end_line":574,"raw_tex":"\\begin{assumption}\n  \\label{hyp:iid}\nThe sequence $(\\xi_n:n\\geq 1)$ is iid, with the same distribution as~$\\xi$.\n\\end{assumption}","sha256_utf8":"b94da2b7ed97ac786c6ba4adc298ea6a68d2f3cb748be5f4212fdcf5ebbe3599"},{"start":28897,"end":29316,"start_line":576,"end_line":583,"raw_tex":"\\begin{assumption}\n  \\label{hyp:moment-f}\nLet $p>0$. Assume either one of the following conditions.\n\\begin{enumerate}[i)]\n\\item \\label{momentegal} For every compact set $K\\subset \\bR^d$, $\\sup_{x\\in K} \\bE(\\|\\nabla f(x,\\xi)\\|^{p})<\\infty\\,.$\n\\item \\label{momentreinforce} For every compact set $K\\subset \\bR^d$,  $\\exists\\, p_K>p$, $\\sup_{x\\in K} \\bE(\\|\\nabla f(x,\\xi)\\|^{p_K})<\\infty.$\n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"ac4c912ac33021c99c07c89173a6e88ff59a25244b527e902cfce17932a88170"},{"start":33930,"end":34781,"start_line":669,"end_line":686,"raw_tex":"\\begin{algorithm}[tb]\n   \\caption{\\bf \\adam - decreasing stepsize $(((\\gamma_n,\\alpha_n,\\beta_n):n\\in \\bN^*), \\varepsilon)$.}\n   \\label{alg:adam-decreasing}\n\\begin{algorithmic}\n  \\STATE {\\bfseries Initialization:} $x_0\\in \\bR^d, m_0=0$, $v_0=0$, $r_0=\\bar r_0=0$.\n   \\FOR{$n=1$ {\\bfseries to} $n_{\\text{iter}}$}\n   \\STATE $m_n = \\alpha_n m_{n-1} + (1-\\alpha_n)  \\nabla f(x_{n-1},\\xi_n)$\n   \\STATE $v_n = \\beta_n v_{n-1} + (1-\\beta_n) \\nabla f(x_{n-1},\\xi_n)^{\\odot 2}$\n   \\STATE $r_n = \\alpha_n r_{n-1} + (1-\\alpha_n)$\n   \\STATE $\\bar r_n = \\beta_n \\bar r_{n-1} + (1-\\beta_n)$\n  \\STATE $\\hat m_{n} = m_{n}/r_n$\n   \\COMMENT{bias correction step}\n   \\STATE $\\hat v_{n} = v_{n}/\\bar r_n$\n   \\COMMENT{bias correction step}\n   \\STATE $x_{n} = x_{n-1} - \\gamma_n \\hat m_{n} / (\\varepsilon+\\sqrt{\\hat v_{n}}) \\,.$\n   \\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"f51d66a5cd3ff2cbbf41dbc52529d524503b679ce9af494ac06a8b05e0e6a62c"},{"start":36277,"end":36501,"start_line":709,"end_line":711,"raw_tex":"  \\item For all $n \\in \\bN$, $0 \\leq \\alpha_n \\leq 1$ and $0 \\leq \\beta_n \\leq 1$,\n  \\item There exist $a,b$ s.t. $0<b<4a$,\\,\n  $\\gamma_n^{-1}(1-\\alpha_n) \\to a$ and $\\gamma_n^{-1}(1-\\beta_n) \\to b$\\,. %as $n \\to \\infty$\\,.\n","sha256_utf8":"0a2575ec5c1f0174dfb776d7e64d9ab47214b8159a140d5017168978dbebe3ae"},{"start":40005,"end":40414,"start_line":784,"end_line":793,"raw_tex":"\\begin{assumption} %[local regularity of the mean field $h_\\infty$]\n   \\label{hyp:mean_field_tcl}\nLet $x^*\\in \\cS$. There exists a neighborhood $\\cV$ of $x^*$ s.t.\n\\begin{enumerate}[{\\sl i)}]\n\\item $F$ is twice continuously differentiable on $\\cV$,\n  and the Hessian $\\nabla^2 F(x^*)$ of $F$ at $x^*$ is positive\n  definite.\n\\item $S$ is continuously differentiable on $\\cV$.\n\\end{enumerate}\n \\end{assumption}","sha256_utf8":"d443a5522e05e90ca6da7364488b48ca47d019092431f71373dadbe4762a347f"},{"start":40415,"end":40983,"start_line":794,"end_line":809,"raw_tex":"Define $\nD \\eqdef \\textrm{diag}\\left((\\varepsilon + \\sqrt{S_1(x^*)})^{-1},\\cdots,(\\varepsilon + \\sqrt{S_d(x^*)})^{-1}\\right)\\,.\n$\nLet $P$ be an orthogonal matrix s.t. the following spectral decomposition holds:\n$$\nD^{1/2}\\nabla^2F(x^*)D^{1/2} = P\\textrm{diag}(\\lambda_1,\\cdots,\\lambda_d)P^{-1}\\,,\n$$\nwhere $\\lambda_1, \\cdots,\\lambda_d$ are the (positive) eigenvalues of\n$D^{1/2} \\nabla^2F(x^*)D^{1/2}$.\nDefine\n\\begin{equation}\nH \\eqdef\n\\begin{pmatrix}\n  0 & -D & 0\\\\ a\\nabla^2F(x^*) & -a I_d & 0 \\\\ b \\nabla S(x^*) & 0 & -b I_d\n\\end{pmatrix}\\label{eq:H}\n\\end{equation}","sha256_utf8":"788e6e92f7a0576092000fb4a21c3e719f0852948644d9cd41967466269c5b3e"},{"start":40984,"end":41310,"start_line":810,"end_line":815,"raw_tex":"where $I_d$ represents the $d\\times d$ identity matrix and $\\nabla S(x^*)$ is the\nJacobian matrix of~$S$ at $x^*$.\nThe largest real part of the eigenvalues of $H$ coincides with $-L$, where\n\\begin{equation}\nL\\eqdef b\\wedge \\frac a2\\left( 1-\\sqrt{\\left(1-\\frac{4\\lambda_1}a\\right)\\vee 0}\\right) >0\\,.\\label{eq:L}\n\\end{equation}","sha256_utf8":"375c6666dc3919265a34afb9184e57b8b94412b504bfc463cbaaea6eb23cf3b0"},{"start":41311,"end":41971,"start_line":816,"end_line":837,"raw_tex":"Finally, define the $3d\\times 3d$ matrix% covariance matrix of the\n% vector $(a\\nabla f(x^*,\\xi),b\\nabla f(x^*,\\xi)^{\\odot 2})$ \\emph{i.e.},\n\\begin{equation}\n  \\resizebox{.9\\hsize}{!}{$\nQ \\eqdef\n\\left( \\begin{array}[h]{cc}\n   0 &\n   \\begin{array}[h]{cc}\n     0 & \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ 0\n   \\end{array}\\\\\n   \\begin{array}[h]{c}\n     0 \\\\ 0\n   \\end{array}\n & \\bE\\left[\n  \\begin{pmatrix}\n    a\\nabla f(x^*,\\xi) \\\\ b(\\nabla f(x^*,\\xi)^{\\odot 2}-S(x^*))\n  \\end{pmatrix}\\begin{pmatrix}\n    a\\nabla f(x^*,\\xi) \\\\ b(\\nabla f(x^*,\\xi)^{\\odot 2}-S(x^*))\n  \\end{pmatrix}^T\\right]\n \\end{array}\\right)\\,.\\label{eq:Q}\n $}\n\\end{equation}","sha256_utf8":"d5d6547594afb666aa7dbdbbb69f23bb61e9290181499bdc3ae97954ccc55f62"},{"start":41972,"end":42580,"start_line":838,"end_line":849,"raw_tex":"\\begin{assumption}\n\\label{hyp:step-tcl}\nThe following holds.\n\\begin{enumerate}[{\\sl i)}]\n\\item \\label{step-tcl-i} There exist $\\kappa \\in (0,1]$, $\\gamma_0>0$, s.t. the sequence $(\\gamma_n)$\nsatisfies\n$\\gamma_n = {\\gamma_0}/{(n+1)^\\kappa}$ for all $n$.\nIf $\\kappa = 1$, we assume moreover that $\\gamma_0 > \\frac{1}{2L}$.\n\\item The sequences $\\left(\\frac{1}{\\gamma_n}(\\frac{1-\\alpha_n}{\\gamma_n}-a)\\right)$ and $\\left(\\frac{1}{\\gamma_n}(\\frac{1-\\beta_n}{\\gamma_n}-b)\\right)$ are bounded.\n%This is satisfied for instance if $1-\\alpha_n = a\\gamma_n$ and $1-\\beta_n = b\\gamma_n$.\n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"68ab0ca04bf29a39add5fd967748c8511ec0cb07b7a1398aa5579753b169c249"},{"start":42582,"end":42904,"start_line":851,"end_line":853,"raw_tex":"\\noindent For an arbitrary sequence $(X_n)$ of random variables on some Euclidean space, a probability measure\n$\\mu$ on that space and an event $\\Gamma$ s.t. $\\bP(\\Gamma)>0$, we say that $X_n$ converges in distribution\nto $\\mu$ \\emph{given $\\Gamma$} if the measures $\\bP (X_n\\in \\cdot\\,|\\Gamma)$ converge weakly to $\\mu$.\n","sha256_utf8":"5a7d565b7ca640424e3d0e7e7b50741a480cc8fd72bd0cefddab90fa721b0351"},{"start":64499,"end":64657,"start_line":1168,"end_line":1171,"raw_tex":"\\begin{equation}\n  \\label{eq:h_infty}\nh_{\\infty}((x,m,v)) = \\left(-m / (\\varepsilon+\\sqrt{v})\\,,\\, a (\\nabla F(x)-m) \\,,\\,b (S(x)-v) \\right)\\,.\n\\end{equation}","sha256_utf8":"1aa54b09ee7cc6beacf4fa985bafbdb804982353b508a8d54f1c32632345a237"},{"start":135732,"end":135826,"start_line":2394,"end_line":2394,"raw_tex":"We recall that $\\cF_n$ is the $\\sigma$-algebra generated by the r.v. $(\\xi_k:1\\leq k\\leq n)$.\n","sha256_utf8":"4876543fff08fa04932fc716deb368fd38391531f4a12c083fd67982bba5d4d5"},{"start":156491,"end":156693,"start_line":2688,"end_line":2689,"raw_tex":"In this section, we denote by $\\bE_n = \\bE(\\cdot|\\cF_n)$ the conditional expectation w.r.t. $\\cF_n$.\nWe also use the notation $\\nabla f_{n+1} \\eqdef \\nabla f(x_n,\\xi_{n+1})$. %to simplify the equation.\n","sha256_utf8":"afd49bd2f5ee718766825927de08ba423ddfb616d8b9286aee18d2ce3e6855fc"},{"start":157927,"end":157981,"start_line":2716,"end_line":2716,"raw_tex":"A similar lemma holds for the sequence $(\\bar{r}_n)$.\n","sha256_utf8":"7c41487f57c00b18d641521faec62352e7f6c0507c4a8924ef42df781df3bb63"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\n \\label{thm:clt}\nLet Assumptions~\\ref{hyp:model}, \\ref{hyp:S>0}, \\ref{hyp:iid}, \\ref{hyp:mean_field_tcl} and \\ref{hyp:step-tcl} hold true.\nLet Assumption~\\ref{hyp:moment-f}~\\ref{momentreinforce} hold with $p=4$.\nConsider the iterates $z_n=(x_n,m_n,v_n)$ given by Algorithm~\\ref{alg:adam-decreasing}. Set $z^*=(x^*,0,S(x^*))$.\nSet $\\zeta \\eqdef 0$ if $0<\\kappa<1$ and $\\zeta \\eqdef \\frac{1}{2 \\gamma_0}$ if $\\kappa =1$.\n%Assume $\\bP(x_n \\to x^*)>0$. Then, given the event $\\{x_n\\to x^*\\}$,\nAssume $\\bP(z_n \\to z^*)>0$. Then, given the event $\\{z_n\\to z^*\\}$,\nthe rescaled vector $\\sqrt{\\gamma_n}^{-1}(z_n-z^*)$\nconverges in distribution to a zero mean Gaussian distribution on $\\bR^{3d}$ with a covariance matrix $\\Sigma$\nwhich is solution to the Lyapunov equation: $  \\left(H + \\zeta I_{3d} \\right) \\Sigma + \\Sigma \\left( H^T +  \\zeta I_{3d} \\right) = - Q$.\nIn particular, given $\\{z_n\\to z^*\\}$, the vector $\\sqrt{\\gamma_n}^{-1}(x_n-x^*)$\nconverges in distribution to a zero mean Gaussian distribution with a covariance matrix $\\Sigma_1$ given by:\n\\begin{equation}\n\\Sigma_1 = D^{1/2} P\n\\left(\n\\frac{C_{k,\\ell}}{ (1 - \\frac{2\\zeta}{a})(\\lambda_k+\\lambda_\\ell-2\\zeta + \\frac 2a \\zeta^2) +\\frac 1{2(a-2\\zeta)}(\\lambda_k-\\lambda_\\ell)^2}\n %\\frac{C_{k,\\ell}}{ (1-\\frac{\\zeta}{a})(\\lambda_k+\\lambda_\\ell - 2\\zeta) +\\frac 1{2(a-\\zeta)}(\\lambda_k-\\lambda_\\ell)^2}\n\\right)_{k,\\ell=1\\dots d}\nP^{-1}D^{1/2}\\label{eq:cov}\n\\end{equation}\n%\\begin{equation}\n%\\Sigma_1 = D^{1/2} P\n%\\left(\n%  \\frac{C_{k,\\ell}}{\\lambda_k+\\lambda_\\ell+\\frac 1{2a}(\\lambda_k-\\lambda_\\ell)^2}\n%\\right)_{k,\\ell=1\\dots d}\n%P^{-1}D^{1/2}\\label{eq:cov}\n%\\end{equation}\nwhere $C\\eqdef P^{-1}D^{1/2}\\bE\\left(\\nabla f(x^*,\\xi)\\nabla f(x^*,\\xi)^T\\right)D^{1/2}P$.\n\\end{theorem}","units":[{"start":42904,"end":44653,"start_line":854,"end_line":883,"raw_tex":"\\begin{theorem}\n \\label{thm:clt}\nLet Assumptions~\\ref{hyp:model}, \\ref{hyp:S>0}, \\ref{hyp:iid}, \\ref{hyp:mean_field_tcl} and \\ref{hyp:step-tcl} hold true.\nLet Assumption~\\ref{hyp:moment-f}~\\ref{momentreinforce} hold with $p=4$.\nConsider the iterates $z_n=(x_n,m_n,v_n)$ given by Algorithm~\\ref{alg:adam-decreasing}. Set $z^*=(x^*,0,S(x^*))$.\nSet $\\zeta \\eqdef 0$ if $0<\\kappa<1$ and $\\zeta \\eqdef \\frac{1}{2 \\gamma_0}$ if $\\kappa =1$.\n%Assume $\\bP(x_n \\to x^*)>0$. Then, given the event $\\{x_n\\to x^*\\}$,\nAssume $\\bP(z_n \\to z^*)>0$. Then, given the event $\\{z_n\\to z^*\\}$,\nthe rescaled vector $\\sqrt{\\gamma_n}^{-1}(z_n-z^*)$\nconverges in distribution to a zero mean Gaussian distribution on $\\bR^{3d}$ with a covariance matrix $\\Sigma$\nwhich is solution to the Lyapunov equation: $  \\left(H + \\zeta I_{3d} \\right) \\Sigma + \\Sigma \\left( H^T +  \\zeta I_{3d} \\right) = - Q$.\nIn particular, given $\\{z_n\\to z^*\\}$, the vector $\\sqrt{\\gamma_n}^{-1}(x_n-x^*)$\nconverges in distribution to a zero mean Gaussian distribution with a covariance matrix $\\Sigma_1$ given by:\n\\begin{equation}\n\\Sigma_1 = D^{1/2} P\n\\left(\n\\frac{C_{k,\\ell}}{ (1 - \\frac{2\\zeta}{a})(\\lambda_k+\\lambda_\\ell-2\\zeta + \\frac 2a \\zeta^2) +\\frac 1{2(a-2\\zeta)}(\\lambda_k-\\lambda_\\ell)^2}\n %\\frac{C_{k,\\ell}}{ (1-\\frac{\\zeta}{a})(\\lambda_k+\\lambda_\\ell - 2\\zeta) +\\frac 1{2(a-\\zeta)}(\\lambda_k-\\lambda_\\ell)^2}\n\\right)_{k,\\ell=1\\dots d}\nP^{-1}D^{1/2}\\label{eq:cov}\n\\end{equation}\n%\\begin{equation}\n%\\Sigma_1 = D^{1/2} P\n%\\left(\n%  \\frac{C_{k,\\ell}}{\\lambda_k+\\lambda_\\ell+\\frac 1{2a}(\\lambda_k-\\lambda_\\ell)^2}\n%\\right)_{k,\\ell=1\\dots d}\n%P^{-1}D^{1/2}\\label{eq:cov}\n%\\end{equation}\nwhere $C\\eqdef P^{-1}D^{1/2}\\bE\\left(\\nabla f(x^*,\\xi)\\nabla f(x^*,\\xi)^T\\right)D^{1/2}P$.\n\\end{theorem}","sha256_utf8":"abd832f738d458b86c14ad88e0c2ec516c4eda77cc31ce64193a711e401bcffa"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\n\\label{lemma:r_n}\nLet the sequence $(r_n)$ be defined as in Algorithm~\\ref{alg:adam-decreasing}.\nAssume that $0 \\leq \\alpha_n \\leq 1$ for all $n$ and that $(1-\\alpha_n)/\\gamma_n \\to a > 0$ as $n \\to +\\infty$.\nThen,\n\\begin{enumerate}[{\\sl i)}]\n\\item $\\forall n \\in \\bN, r_n = 1 - \\prod_{i=1}^n \\alpha_i$,\n\\item The sequence $(r_n)$ is nondecreasing and converges to $1$.\n\\item Under Assumption~\\ref{hyp:step-tcl}~\\ref{step-tcl-i}, for every $\\epsilon>0$, for sufficiently large $n$, we have\n$r_n-1 \\leq e^{-\\frac{a\\gamma_0}{2(1-\\kappa)}n^{1-\\kappa}}$ if $\\kappa \\in (0,1)$ and\n$r_n-1 \\leq n^{-a \\gamma_0/(1+\\epsilon)}$ if $\\kappa = 1$.\n% $\n%      r_n-1 \\leq\n%      \\begin{cases}\n%         e^{-\\frac{a\\gamma_0}{2(1-\\kappa)}n^{1-\\kappa}} \\quad \\text{if}\\,\\,\\kappa \\in (0,1)\\\\\n%         n^{-a \\gamma_0/(1+\\epsilon)} \\quad \\text{if}\\,\\,\\kappa=1 \\,.\n%      \\end{cases}\n% $\n%\\item If $\\gamma_n = \\gamma_0 n^{-\\alpha}$ with $\\alpha \\in (0,1)$, then\n%for sufficiently large $n$, $1-r_n \\leq e^{-\\frac{a\\gamma_0}{2(1-\\alpha)}n^{1-\\alpha}}$\\,.\n%\\item If $\\gamma_n = \\gamma_0 n^{-1}$, then for sufficiently large $n$,\n%$1-r_n \\leq n^{-a \\gamma_0/2}$\\,.\n\\end{enumerate}\n\\end{lemma}\n\n\\begin{proof}\ni) stems from observing that $r_{n+1} -1 = \\alpha_{n+1}(r_n -1)$ for every $n \\in \\bN$ and\niterating this relation ($r_0=0$). As a consequence, the sequence $(r_n)$ is nondecreasing.\n%We now show iii) and iv).\nWe can write :\n$\n0 \\leq 1-r_n \\leq \\exp(- \\sum_{i=1}^n (1-\\alpha_i))\\,.\n$\niii) As $\\sum_{n\\geq 1} \\gamma_n = + \\infty$ %for $\\gamma_n = \\gamma_0 n^{-\\kappa}$ with $\\kappa \\in [0,1]$\nand $(1-\\alpha_n) \\sim a\\gamma_n$, we deduce that $\\sum_{i=1}^n (1-\\alpha_i) \\sim \\sum_{i=1}^n a\\gamma_i$.\nThe results follow from the fact that $\\sum_{i=1}^n \\gamma_i \\sim \\frac{\\gamma_0}{1-\\kappa} n^{1-\\kappa}$\nwhen $\\kappa \\in (0,1)$ and $\\sum_{i=1}^n \\gamma_i \\sim \\gamma_0 \\ln n$ for $\\kappa=1$.\n\\end{proof}\n\n\\subsection{Proof of Th.~\\ref{thm:clt}}\n\\label{sec:clt}\n\nWe use~\\cite[Th.~1]{pelletier1998weak}.\nAll the assumptions in the latter can be verified in our case, at the exception of\na positive definiteness condition on the\nlimiting covariance matrix, which corresponds, in our case, to the matrix $Q$\ngiven by Eq.~(\\ref{eq:Q}). As $Q$ is not positive definite, it is strictly speaking not possible\nto just cite and apply \\cite[Th.~1]{pelletier1998weak}. Nevertheless,\na detailed inspection of the proofs of \\cite{pelletier1998weak} shows that only a minor\nadaptation is needed in order to cover the present case.\n%Therefore, it is worthless reprove the convergence result of \\cite{pelletier1998weak} from scratch.\nTherefore, proving the convergence result of \\cite{pelletier1998weak} from scratch is worthless.\nIt is sufficient to verify the assumptions of \\cite[Th.~1]{pelletier1998weak}\n(except the definiteness of $Q$) and then to point out the specific part of the proof of \\cite{pelletier1998weak}\nwhich requires some adaptation.\n% The following lemma is a consequence of \\cite[Part 3.4]{pelletier1998weak}.\n% \\begin{lemma}\n%   Let $z^*\\in \\bR^k$ where $k\\geq 1$. On some probability space equipped with a filtration $\\cF=(\\cF_n)_n$, consider a random sequence on $\\bR^k$ given by\n% $$\n% Z_{n+1} = Z_n + \\gamma_{n+1}\\bar h(Z_n) + \\gamma_{n+1}\\eta_{n+1}+\\gamma_{n+1}\\epsilon_{n+1}\\,,\n% $$\n% where $(\\gamma_n)$ is a deterministic sequence, $\\bar h:\\bR^k\\to\\bR^k$ is measurable,\n% $(\\eta_n)$, $(\\epsilon_n)$ are adapted to $\\cF$. Assume the following conditions.\n% \\begin{itemize}[leftmargin=24pt]\n% \\item[\\emph{(A1)}] It holds that $\\bar h(z^*)=0$. Moreover, $\\bar h$ is continuously differentiable in a neighborhood of $z^*$,\n% and its Jacobian matrix $\\bar H$ at $z^*$ is stable \\emph{i.e.}, the largest real part of its eignevalues\n% is $(-\\bar L)$ with $\\bar L>0$.\n% \\item[\\emph{(A2)}] There exists\n% \\end{itemize}\n% \\end{lemma}\n\nLet $z_n=(x_n,m_n,v_n)$ be the output of Algorithm~\\ref{alg:adam-decreasing}.\nDefine $z^*=(x^*,0,S(x^*))$.\n% Remark that the conditional central limit theorem stated by \\cite{pelletier1998weak} shall be valid\n% only given the event $\\{z_n\\to z^*\\}$, whereas we seek to prove the convergence result stated in Th.~\\ref{thm:clt}\n% given the event $\\{x_n\\to x^*\\}$. As a matter of fact, both events coincide up to a $\\bP$-negligible set,\n% so it is safe to prove the convergence on the event $\\{z_n\\to z^*\\}$.\n% The claim that the two events coincide trivially holds if one lets the assumptions of Th.~\\ref{thm:as_conv_under_stab} be satisfied,\n% because in that case, $m_n\\to 0$ and $v_n- S(x_n)\\to 0$ a.s., so that $z_n\\to z^*$ a.s. on the event $\\{x_n\\to x^*\\}$.\n% But the claim is also satisfied if only the assumptions of Th.~\\ref{thm:clt} are considered.\n% As this point is somewhat technical and maybe non essential in a first reading, we postpone the discussion\n% to the end of this section.\nDefine $\\eta_{n+1} \\eqdef (0, a(\\nabla f_{n+1} - \\nabla F(x_n)), b(\\nabla f_{n+1}^{\\odot 2}- S(x_n)))$.\nWe have\n\\begin{equation}\n  \\label{eq:zn_dec}\n  z_{n+1} = z_n + \\gamma_{n+1} h_\\infty(z_n) + \\gamma_{n+1}\\eta_{n+1} + \\gamma_{n+1} \\epsilon_{n+1}\\,,\n\\end{equation}\nwhere $\\epsilon_{n+1} \\eqdef (\\epsilon_{n+1}^1,\\epsilon_{n+1}^2,\\epsilon^3_{n+1})$, whose components are given by\n\\begin{equation*}\n  \\resizebox{\\hsize}{!}{$\n  \\epsilon_{n+1}^1 =  \\frac{m_n}{\\varepsilon+\\sqrt{v_n}} - \\frac{\\hat m_{n+1}}{\\varepsilon+\\sqrt{\\hat v_{n+1}}};\\,\n  \\epsilon_{n+1}^2 = \\left(\\frac{1-\\alpha_{n+1}}{\\gamma_{n+1}}-a\\right)\\left( \\nabla f_{n+1} - m_n\\right);\\,\n  \\epsilon_{n+1}^3 =  \\left(\\frac{1-\\beta_{n+1}}{\\gamma_{n+1}}-b\\right)\\left( \\nabla f_{n+1}^{\\odot 2} - v_n\\right)\\,.\n  $}\n\\end{equation*}\nHere, $\\eta_{n+1}$ is a martingale increment noise and\n$\\epsilon_{n+1} = (\\epsilon_{n+1}^1,\\epsilon_{n+1}^2,\\epsilon^3_{n+1})$ is a remainder term.\nThe aim is to check the assumptions (A1.1) to (A1.3) of\n\\cite{pelletier1998weak}, where the role of the quantities ($h$,\n$\\varepsilon_n$, $r_n$, $\\sigma_n$, $\\alpha$, $\\rho$, $\\beta$) in\n\\cite{pelletier1998weak} is respectively played by the quantities\n($h_\\infty$, $\\eta_n$, $\\epsilon_n$, $\\gamma_n$, $\\kappa$, $1$, $1$) of\nthe present paper.\n\nLet us first consider Assumption (A1.1) for $h_\\infty$.\nBy construction, $h_\\infty(z^*) = 0$. By Assumptions~\\ref{hyp:mean_field_tcl} and \\ref{hyp:S>0},\n$h_\\infty$ is continuously differentiable in the neighborhood of $z^*$ and its\nJacobian at $z^*$ coincides with the matrix $H$ given by Eq.~(\\ref{eq:H}).\n%As already discussed, after straightforward algebra,\nAs already discussed, after some algebra,\nit can be shown that the largest real part of the eigenvalues of $H$ coincides with $-L$ where $L>0$\nis given by Eq.~(\\ref{eq:L}).\nHence, Assumption (A1.1) of \\cite{pelletier1998weak}\nis satisfied for $h_\\infty$. Assumption (A1.3) is trivially satisfied\nusing Assumption~\\ref{hyp:step-tcl}. The crux is therefore to verify Assumption (A1.2).\nClearly, $\\bE(\\eta_{n+1}|\\cF_n)=0$. Using Assumption~\\ref{hyp:moment-f}\\ref{momentreinforce},\nit follows from straightforward manipulations based on Jensen's inequality\nthat for any $M>0$, %s.t. $B(z^*,M)\\subset \\cV$,\nthere exists $\\delta>0$ s.t.\n$%$$\n\\sup_{n\\geq 0}\\bE_n\\left(\\|\\eta_{n+1}\\|^{2+\\delta}\\right) \\1_{\\{\\|z_n-z^*\\|\\leq M\\}}<\\infty\\,.\n$ %$$\nNext, we verify the condition\n\\begin{equation}\n  \\label{eq:cond-reste}\n  \\lim_{n\\to\\infty} \\bE\\left(\\gamma_{n+1}^{-1}\\|\\epsilon_{n+1}\\|^2\\1_{\\{\\|z_n-z^*\\|\\leq M\\}}\\right) = 0\\,.\n\\end{equation}\nIt is sufficient to verify the latter for $\\epsilon^i_n$ ($i=1,2,3$) in place of $\\epsilon_n$.\nThe map $(m,v)\\mapsto m/(\\varepsilon+\\sqrt{v})$ is Lipschitz continuous in a neighborhood of\n$(0,S(x^*))$ by Assumption~\\ref{hyp:S>0}. Thus, for $M$ small enough, there exists a constant $C$ s.t.\nif $ \\|z_n-z^*\\|\\leq M$, then\n%$$\n%  \\|\\epsilon_{n+1}^1\\| \\leq C\\left\\|\\frac{m_{n+1}}{r_{n+1}}-m_n\\right\\|+C\\left\\|\\frac {v_{n+1}}{\\bar r_{n+1}}-v_n\\right\\| \\,.\n%$$\n$%$$\n  \\|\\epsilon_{n+1}^1\\| \\leq C\\left\\|r_{n+1}^{-1}m_{n+1}-m_n\\right\\|+C\\left\\|\\bar r_{n+1}^{-1} v_{n+1}-v_n\\right\\|\\,.\n$ %$$\nUsing the triangular inequality and the fact that $r_{n+1},\\bar r_{n+1}$ are bounded sequences away from zero, there exists an other constant $C$ s.t.\n\\begin{align*}\n  \\|\\epsilon_{n+1}^1\\| &\\leq C\\left\\|m_{n+1}-m_n\\right\\|+C\\left\\|v_{n+1}-v_{n}\\right\\|\n+C|r_{n+1}-1|+C|\\bar r_{n+1}-1|\\,.\n\\end{align*}\n%Using standard manipulations of series along with Assumption~\\ref{hyp:step-tcl}, it can be shown that\nUsing Lemma~\\ref{lemma:r_n} under Assumption~\\ref{hyp:step-tcl} (note that $\\gamma_0 > 1/2L \\geq 1/a$ when $\\kappa=1$),\nwe obtain that the sequence $|r_{n}-1|/\\gamma_n$ is bounded, thus $|r_{n+1}-1|\\leq C\\gamma_{n+1}$.\n%{\\color{red} voir si on a la place de le mettre dans un lemme.}\n The sequence $(1-\\alpha_{n})/\\gamma_n$ being also bounded, it holds that\n%$\\|m_{n+1}-m_n\\|\\leq C\\gamma_{n+1}\\|\\nabla f_{n+1}-m_n\\|$, and if $ \\|z_n-z^*\\|\\leq M$,\n\\begin{align*}\n  \\|m_{n+1}-m_n\\|^2 \\1_{\\{\\|z_n-z^*\\|\\leq M\\}} \\leq  C\\gamma_{n+1}^2(1+\\|\\nabla f_{n+1}\\|^2 )\\1_{\\{\\|z_n-z^*\\|\\leq M\\}}\\,.\n\\end{align*}\nBy Assumption~\\ref{hyp:moment-f}~\\ref{momentreinforce}, %Assumption~\\ref{hyp:mean_field_tcl},\n$\\bE_n(\\|\\nabla f_{n+1}\\|^2|)$ is bounded by a deterministic constant on $\\{\\|z_n-z^*\\|\\leq M\\}$.\nThus, $\\bE_n(\\|m_{n+1}-m_n\\|^2\\1_{\\{\\|z_n-z^*\\|\\leq M\\}})\\leq C\\gamma_{n+1}^2$.\nA similar result holds for $\\|v_{n+1}-v_n\\|^2$. We have thus shown that\n$ \\bE_n\\left(\\|\\epsilon_{n+1}^1\\|^2\\1_{\\{\\|z_n-z^*\\|\\leq M\\}}\\right)\\leq C\\gamma_{n+1}^2$. Hence,\nEq.~(\\ref{eq:cond-reste}) is proved for $\\epsilon_{n+1}^1$ in place of $\\epsilon_{n+1}$.\n%As far as $\\epsilon_{n+1}^2$, $\\epsilon_{n+1}^3$ are concerned, the proof uses the same kind of arguments\n%under Assumption~\\ref{hyp:step-tcl} and is omitted.\nUnder Assumption~\\ref{hyp:step-tcl}, the proof uses the same kind of arguments for $\\epsilon_{n+1}^2$, $\\epsilon_{n+1}^3$  and is omitted.\nFinally, Eq.~(\\ref{eq:cond-reste}) is proved.\nContinuing the verification of Assumption (A1.2), we establish that\n\\begin{equation}\n  \\label{eq:lim-cov}\n  \\bE_n(\\eta_{n+1}\\eta_{n+1}^T) \\to Q\\textrm{ a.s. on } \\{z_n\\to z^*\\}\\,.\n\\end{equation}\nDenote by $\\bar Q(x)$ the matrix given by the righthand side of Eq.~(\\ref{eq:Q}) when $x^*$\nis replaced by an arbitrary $x\\in \\cV$. It is easily checked that $\\bE_n(\\eta_{n+1}\\eta_{n+1}^T)=\\bar Q(x_n)$\nand by continuity, $\\bar Q(x_n)\\to Q$ a.s. on $\\{z_n\\to z^*\\}$, which proves (\\ref{eq:lim-cov}).\nTherefore, Assumption (A1.2) is fulfilled, except for the point mentioned at the beginning of this section : %paragraph:\n\\cite{pelletier1998weak} puts the additional condition that the limit\nmatrix in Eq.~(\\ref{eq:lim-cov}) is positive definite. This condition is not satisfied in our case,\nbut the proof can still be adapted. The specific part of the proof where the positive definiteness\ncomes into play is Th.~7 in \\cite{pelletier1998weak}. The proof of \\cite[Th.~1]{pelletier1998weak} can therefore\nbe adapted to the case of a positive semidefinite matrix.\n%by substituting the following\n%result to Th.~7 of the same paper.\nIn the proof of \\cite[Th.~7]{pelletier1998weak}, we only substitute the inverse of the square root of $Q$ by the Moore-Penrose inverse.\nFinally, the uniqueness of the stationary distribution $\\mu$ and its expression follow from \\cite[Th.~6.7, p. 357]{karatzas-shreve1991}.\n%{\\color{red} Citer Karatsas-Shreeve, 2d Ed., Th. 6.7.}\n% \\begin{theorem}[adapted from \\cite{pelletier1998weak}, Th.~7]\n% Let $k\\geq 1$. On some probability space equipped with a filtration $\\cF=(\\cF_n)_n$,\n% consider a sequence of r.v. on $\\bR^k$ given by\n% $$\n% Z_{n+1} = (I+\\gamma_{n+1} \\bar H)Z_n+\\gamma_{n+1} b_{n+1}+\\sqrt{\\gamma_{n+1}}\\eta_{n+1}\n% $$\n% and $\\bE(\\|Z_0\\|^2)<0$,\n% where $\\bar H$ is a $k\\times k$ stable matrix \\emph{i.e.}, the largest real part of its eigenvalues\n% is negative, $(b_n)$ and $(\\eta_n)$ are random sequences, $(\\gamma_n)$ satisfies Assumption~\\ref{hyp:step-tcl}.\n% Let $\\Omega_0\\in \\cF_\\infty$ have a positive probability. Assume that the following holds almost surely on $\\Omega_0$:\n% \\begin{enumerate}[{i)}, leftmargin=*]\n% \\item $\\bE_n(\\eta_{n+1})=0$.\n% \\item There exists a constant $\\bar b>2$ s.t. $\\sup_{n\\geq 0}\\bE_n(\\|\\eta_{n+1}\\|^{\\bar b})<\\infty$.\n% \\item $\\bE_n(\\eta_{n+1}\\eta_{n+1}^T) = C + \\Delta_n$ where $\\bE(\\|\\Delta_n\\|\\1_{\\Omega_0})\\to 0$\n% and $C$ is a positive semidefinite matrix.\n% \\item The sequence $(b_n)$ is the sum of two sequences $(b_{n,1})$ and $(b_{n,2})$, adapted to $\\cF$,\n% s.t. $\\sup_{n\\geq 0}\\bE(\\|b_{n,1}\\|^2)<\\infty$, $\\bE(\\|b_{n,1}\\|\\1_{\\Omega_0})\\to 0$ and $b_{n,2}\\to 0$ a.s. on $\\Omega_0$.\n% \\end{enumerate}\n% Then, given $\\Omega_0$, $(Z_n)$ converges in distribution to the unique stationary distribution $\\mu$ of the generalized Ornstein-Uhlenbeck process\n% $$\n% dX_t = \\bar H X_t dt + \\sqrt CdB_t\n% $$\n% where $(B_t)$ is the standard Brownian motion and $\\sqrt{C}$ is the unique positive semidefinite square root of $C$.\n% The distribution $\\mu$ is the zero mean Gaussian distribution with covariance matrix $\\bar \\Sigma$\n% given as the solution to $\\bar H\\bar \\Sigma+\\bar \\Sigma\\bar H^T = -C$.\n% \\end{theorem}\n% \\begin{proof}\n%   The proof is identical to the proof of \\cite[Th.~7]{pelletier1998weak}, only substituting the\n% inverse of the square root of $C$ by the Moore-Penrose inverse. Finally, the uniqueness of the\n% stationary distribution $\\mu$ and its expression follow from \\cite[Th.~6.7, p. 357]{karatzas-shreve1991}.%{\\color{red} Citer Karatsas-Shreeve, 2d Ed., Th. 6.7.}\n% \\end{proof}\n\n%\\subsection{Proof of Cor.~\\ref{cor:x}}\n\\noindent\\textbf{Proof of Eq.~(\\ref{eq:cov})}.\nWe introduce the $d\\times d$ blocks of the $3d\\times 3d$ matrix %$\\Sigma$ by\n$\\Sigma = \\left(\\Sigma_{i,j}\\right)_{i,j=1,2,3}$\nwhere $\\Sigma_{i,j}$ is $d\\times d$. We denote by $\\tilde\\Sigma$ the $2d\\times 2d$ submatrix $\\tilde\\Sigma \\eqdef\\left(\\Sigma_{i,j}\\right)_{i,j=1,2}$.\nBy Th.~\\ref{thm:clt}, we have the subsystem:\n\\begin{equation}\n\\tilde H \\tilde\\Sigma + \\tilde\\Sigma\\tilde H^T =\n\\begin{pmatrix}\n  0 & 0 \\\\\n0 & -a^2 \\tilde Q\n\\end{pmatrix}\\qquad\\text{where }\\tilde H \\eqdef\n\\begin{pmatrix}\n  \\zeta I_d & -D \\\\\na\\nabla^2F(x^*) & (\\zeta-a) I_d\n\\end{pmatrix}\\label{eq:lyap-reduced}\n\\end{equation}\nand where $\\tilde Q \\eqdef \\textrm{Cov}\\left(\\nabla\n  f(x^*,\\xi)\\right)$.  The next step is to triangularize the matrix\n$\\tilde H$ in order to decouple the blocks of $\\tilde\\Sigma$.  For\nevery $k=1,\\dots,d$, set\n$\\nu_k^\\pm \\eqdef -\\frac{a}{2}\\pm\\sqrt{a^2/4-a\\lambda_k}$\nwith the convention that $\\sqrt{-1} =\n\\imath$ (inspecting the characteristic polynomial of $\\tilde H$, these %quantities\nare the eigenvalues of $\\tilde H$). Set\n$M^\\pm\\eqdef\\diag{(\\nu_1^\\pm,\n\\cdots,\\nu_d^\\pm)}$ and $R^\\pm\\eqdef\nD^{-1/2}PM^\\pm P^TD^{-1/2}$.  Using the identities $M^++M^-=-a I_d$ and\n$M^+M^-=a\\Lambda$ where\n$\\Lambda\\eqdef\\diag{(\\lambda_1,\\cdots,\\lambda_d)}$, it can be checked\nthat\n$$\n\\cR\\tilde H =\\begin{pmatrix}\n  D R^+ + \\zeta I_d & -D \\\\ 0 & R^-D + \\zeta I_d\n\\end{pmatrix}\\cR,\\text{ where }\\cR\\eqdef\n\\begin{pmatrix}\n  I_d & 0 \\\\ R^+ & I_d\n\\end{pmatrix}\\,.\n$$\nSet $\\check \\Sigma \\eqdef \\cR\\tilde\\Sigma\\cR^T$. Denote by\n$(\\check \\Sigma_{i,j})_{i,j=1,2}$ the blocks of $\\check\\Sigma$.\nNote that $\\check\\Sigma_{1,1} = \\Sigma_{1,1}$. By left/right multiplication of Eq.~(\\ref{eq:lyap-reduced})\nrespectively with $\\cR$ and $\\cR^T$, we obtain\n%% $$\n%% \\cT\\check\\Sigma + \\check\\Sigma\\cT^T =\n%% \\begin{pmatrix}\n%%   0 & 0 \\\\\n%% 0 & -a^2 \\tilde Q\n%% \\end{pmatrix}\\,.\n%% $$\n%% By expanding the above system, we obtain\n\\begin{align}\n  &(DR^++\\zeta I_d) \\Sigma_{1,1}+\\Sigma_{1,1}(R^+D+\\zeta I_d) = \\check\\Sigma_{1,2} D+ D\\check\\Sigma_{1,2}^T \\label{eq:lapremiere}\\\\\n& (DR^++\\zeta I_d) \\check\\Sigma_{1,2}+\\check\\Sigma_{1,2} (DR^-+\\zeta I_d) = D\\check\\Sigma_{2,2} \\label{eq:ladeuxieme}\\\\\n& (R^-D+\\zeta I_d) \\check\\Sigma_{2,2} + \\check\\Sigma_{2,2} (DR^-+\\zeta I_d) = -a^2\\tilde Q \\label{eq:laderniere}\n\\end{align}\nSet $\\bar \\Sigma_{2,2} = P^{-1}D^{1/2}\\check\\Sigma_{2,2} D^{1/2}P$.\nDefine $C\\eqdef P^{-1}D^{1/2}\\tilde Q D^{1/2}P$.\nEq.~(\\ref{eq:laderniere}) yields $(M^-+\\zeta I_d) \\bar\\Sigma_{2,2} + \\bar\\Sigma_{2,2} (M^-+\\zeta I_d) = -a^2C$.\n%% The component $(k,\\ell)$ is given by\n%% $$\n%% \\bar\\Sigma_{2,2}^{k,\\ell} = -a^2\\frac{C_{k,\\ell}}{\\nu_k^-+\\nu_\\ell^-}\\,.\n%% $$\nSet $\\bar \\Sigma_{1,2} = P^{-1}D^{-1/2}\\check\\Sigma_{1,2} D^{1/2}P$.\nEq.~(\\ref{eq:ladeuxieme}) rewrites $(M^++\\zeta I_d) \\bar \\Sigma_{1,2}+\\bar \\Sigma_{1,2} (M^-+\\zeta I_d) = \\bar \\Sigma_{2,2}$.\nWe obtain that %The component $(k,\\ell)$ is given by\n$%$$\n\\bar\\Sigma_{1,2}^{k,\\ell} = (\\nu_k^++\\nu_\\ell^-+2\\zeta)^{-1}\\bar\\Sigma_{2,2}^{k,\\ell} = \\frac{-a^2C_{k,\\ell}}{(\\nu_k^++\\nu_\\ell^-+2\\zeta)(\\nu_k^-+\\nu_\\ell^-+2\\zeta)}\\,.\n$ %$$\nSet %finally\n$\\bar \\Sigma_{1,1} = P^{-1}D^{-1/2}\\Sigma_{1,1} D^{-1/2}P$.\nEq.~(\\ref{eq:lapremiere}) becomes $(M^++\\zeta I_d)\\bar \\Sigma_{1,1} + \\bar \\Sigma_{1,1}(M^++\\zeta I_d) = \\bar \\Sigma_{1,2} + \\bar \\Sigma_{1,2}^T$. Thus,\n%\\begin{equation*}\n%\\resizebox{.99\\hsize}{!}{$\n\\begin{align*}\n\\bar\\Sigma_{1,1}^{k,\\ell} &\n\\resizebox{.9\\hsize}{!}{$\n= \\frac{\\bar\\Sigma_{1,2}^{k,\\ell}+\\bar\\Sigma_{1,2}^{\\ell,k}}{\\nu_k^++\\nu_\\ell^++2\\zeta}\n=  \\frac{-a^2C_{k,\\ell}}{(\\nu_k^++\\nu_\\ell^++2\\zeta)(\\nu_k^-+\\nu_\\ell^-+2\\zeta)}\\left(\\frac{1}{\\nu_k^++\\nu_\\ell^-+2\\zeta}\n+\\frac{1}{\\nu_k^-+\\nu_\\ell^++2\\zeta}\\right)\n$}\n\\\\ &\n%\\resizebox{.6\\hsize}{!}{$= \\frac{C_{k,\\ell}}{ (1 - \\frac{2\\zeta}{a})(\\lambda_k+\\lambda_\\ell-2\\zeta + \\frac 2a \\zeta^2) +\\frac 1{2(a-2\\zeta)}(\\lambda_k-\\lambda_\\ell)^2}\\,.$}\n= \\frac{C_{k,\\ell}}{ (1 - \\frac{2\\zeta}{a})(\\lambda_k+\\lambda_\\ell-2\\zeta + \\frac 2a \\zeta^2) +\\frac 1{2(a-2\\zeta)}(\\lambda_k-\\lambda_\\ell)^2}\\,,\n%\\,\\,\\text{and  the result is proved.}\n\\end{align*}\nand  the result is proved.\n% \\noindent\\textbf{The events $\\{x_n\\to x^*\\}$ and $\\{z_n\\to z^*\\}$ coincide $\\bP$-a.e.}\n% Set $\\bar \\gamma_n = 1-\\alpha_{n}$. Define the event $\\Gamma_N\\eqdef\\{x_n\\to x^*\\text{ and }\\sup_n\\|x_n-x^*\\|\\leq N\\}$.\n% It is sufficient to show that for all $N>0$, $m_n\\to 0$ and $v_n\\to S(x^*)$ a.s. on $\\Gamma_N$.\n% Considering the iterates $v_n$, we have\n% \\begin{align*}\n% v_{n+1}\\1_{\\Gamma_N}  &= (1-\\bar\\gamma_{n+1})v_{n}\\1_{\\Gamma_N}  +\\bar\\gamma_{n+1} S(x^*)\\1_{\\Gamma_N}\n% + \\bar\\gamma_{n+1} \\tilde \\chi_{n+1} + \\bar\\gamma_{n+1} \\tilde \\varsigma_n^1 + \\bar\\gamma_{n+1} \\tilde \\varsigma_n^2 \\\\\n% \\text{where }\\quad&\n%   \\resizebox{.85\\hsize}{!}{$\n% \\tilde \\chi_{n+1} \\eqdef (\\nabla f_{n+1}^{\\odot\n%     2}-S(x_n))\\1_{\\Gamma_N} - \\bE_n((\\nabla f_{n+1}^{\\odot\n%     2}-S(x_n))\\1_{\\Gamma_N})\n%  + \\bE_n[(\\nabla f_{n+1}^{\\odot{2}} - S(x_n)) \\bE_n(\\1_{\\Gamma_N})]$}\\\\\n% & \\tilde \\varsigma_n^1\\eqdef (S(x_n)-S(x^*))\\1_{\\Gamma_N}\\\\\n% &\\tilde\\varsigma_n^2\\eqdef \\bE_n( (\\nabla f_{n+1}^{\\odot 2}-S(x_n))\\1_{\\|x_n-x^*\\|\\leq N}(\\1_{\\Gamma_N}-\\bE_n(\\1_{\\Gamma_N})) )\\,.\n% \\end{align*}\n% Clearly, $\\tilde \\varsigma_n^1\\to 0$ a.s. The same holds for $\\tilde\\varsigma_n^2$, using H\\\"older's inequality along\n% with Assumption~\\ref{hyp:moment-f}-\\ref{momentreinforce} for $p=2$\n% and the fact that $\\bE_n(\\1_{\\Gamma_N})$ converges a.s. to $\\1_{\\Gamma_N}$.\n% Moreover, $\\bE_n(\\|\\tilde\\chi_{n+1}\\|^2)\\leq C$ for some $C>0$, using again Assumption~\\ref{hyp:moment-f}-\\ref{momentreinforce}\n% and Jensen's inequality. As $\\sum_k\\gamma_k^2<\\infty$, this in turn implies that $\\lim_n\\sum_{k\\leq n}\\gamma_k\\chi_k$ exists.\n% Using the same kind of arguments, one can show that $(v_n)$ is a.s. bounded on $\\Gamma_N$.\n% Thus, as in the proof of Th.~\\ref{thm:as_conv_under_stab},\n% we have shown that, almost surely on $\\Gamma_N$, the piecewise interpolated process of the sequence $(v_n)$ is a bounded APT of\n% $\\dot v=S(x^*)-v$, hence showing that $v_n\\to S(x^*)$ a.s. on that set. The same arguments can be used to show that\n% $m_n\\to 0$ a.s. on $\\Gamma_N$, and the proof is concluded.\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n% \\subsection{Proof of Cor.~\\ref{cor:x}}\n%\n% We introduce the $d\\times d$ blocks of the $3d\\times 3d$ matrix %$\\Sigma$ by\n% $\\Sigma = \\left(\\Sigma_{i,j}\\right)_{i,j=1,2,3}$\n% where $\\Sigma_{i,j}$ is $d\\times d$. We denote by $\\tilde\\Sigma$ the $2d\\times 2d$ submatrix $\\tilde\\Sigma \\eqdef\\left(\\Sigma_{i,j}\\right)_{i,j=1,2}$.\n% By Th.~\\ref{thm:clt}, we have the subsystem:\n% \\begin{equation}\n% \\tilde H \\tilde\\Sigma + \\tilde\\Sigma\\tilde H^T =\n% \\begin{pmatrix}\n%   0 & 0 \\\\\n% 0 & -a^2 \\tilde Q\n% \\end{pmatrix}\\qquad\\text{where }\\tilde H \\eqdef\n% \\begin{pmatrix}\n%   0 & -D \\\\\n% a\\nabla^2F(x^*) & -a I_d\n% \\end{pmatrix}\\label{eq:lyap-reduced}\n% \\end{equation}\n% and where $\\tilde Q \\eqdef \\textrm{Cov}\\left(\\nabla\n%   f(x^*,\\xi)\\right)$.  The next step is to triangularize the matrix\n% $\\tilde H$ in order to decouple the blocks of $\\tilde\\Sigma$.  For\n% every $k=1,\\dots,d$, set $\\nu_k^\\pm \\eqdef -\\frac\n% a2\\pm\\sqrt{a^2/4-a\\lambda_k}$ with the convention that $\\sqrt{-1} =\n% \\imath$ (inspecting the characteristic polynomial of $\\tilde H$, these\n% quantities are the eigenvalues of $\\tilde H$). Set\n% $M^\\pm\\eqdef\\diag{(\\nu_1^\\pm,\\cdots,\\nu_d^\\pm)}$ and $R^\\pm\\eqdef\n% D^{-1/2}PM^\\pm P^TD^{-1/2}$.  Using the identities $M^++M^-=-aI_d$ and\n% $M^+M^-=a\\Lambda$ where\n% $\\Lambda\\eqdef\\diag{(\\lambda_1,\\cdots,\\lambda_d)}$, it can be checked\n% that\n% $$\n% \\cR\\tilde H =\\begin{pmatrix}\n%   D R^+ & -D \\\\ 0 & R^-D\n% \\end{pmatrix}\\cR,\\text{ where }\\cR\\eqdef\n% \\begin{pmatrix}\n%   I_d & 0 \\\\ R^+ & I_d\n% \\end{pmatrix}\\,.\n% $$\n% Set $\\check \\Sigma \\eqdef \\cR\\tilde\\Sigma\\cR^T$. Denote by\n% $(\\check \\Sigma_{i,j})_{i,j=1,2}$ the blocks of $\\check\\Sigma$.\n% Note that $\\check\\Sigma_{1,1} = \\Sigma_{1,1}$. By left/right multiplication of Eq.~(\\ref{eq:lyap-reduced})\n% respectively with $\\cR$ and $\\cR^T$, we obtain\n% %% $$\n% %% \\cT\\check\\Sigma + \\check\\Sigma\\cT^T =\n% %% \\begin{pmatrix}\n% %%   0 & 0 \\\\\n% %% 0 & -a^2 \\tilde Q\n% %% \\end{pmatrix}\\,.\n% %% $$\n% %% By expanding the above system, we obtain\n% \\begin{align}\n%   &DR^+\\Sigma_{1,1}+\\Sigma_{1,1}R^+D = \\check\\Sigma_{1,2} D+ D\\check\\Sigma_{1,2}^T \\label{eq:lapremiere}\\\\\n% & DR^+\\check\\Sigma_{1,2}+\\check\\Sigma_{1,2}DR^- = D\\check\\Sigma_{2,2} \\label{eq:ladeuxieme}\\\\\n% & R^-D\\check\\Sigma_{2,2} + \\check\\Sigma_{2,2} DR^- = -a^2\\tilde Q \\label{eq:laderniere}\n% \\end{align}\n% Set $\\bar \\Sigma_{2,2} = P^{-1}D^{1/2}\\check\\Sigma_{2,2} D^{1/2}P$.\n% Define $C\\eqdef P^{-1}D^{1/2}\\tilde Q D^{1/2}P$.\n% Eq.~(\\ref{eq:laderniere}) yields $M^-\\bar\\Sigma_{2,2} + \\bar\\Sigma_{2,2} M^- = -a^2C$.\n% %% The component $(k,\\ell)$ is given by\n% %% $$\n% %% \\bar\\Sigma_{2,2}^{k,\\ell} = -a^2\\frac{C_{k,\\ell}}{\\nu_k^-+\\nu_\\ell^-}\\,.\n% %% $$\n% Set $\\bar \\Sigma_{1,2} = P^{-1}D^{-1/2}\\check\\Sigma_{1,2} D^{1/2}P$.\n% Eq.~(\\ref{eq:ladeuxieme}) rewrites $M^+\\bar \\Sigma_{1,2}+\\bar \\Sigma_{1,2}M^- = \\bar \\Sigma_{2,2}$.\n% The component $(k,\\ell)$ is given by\n% $$\n% \\bar\\Sigma_{1,2}^{k,\\ell} = (\\nu_k^++\\nu_\\ell^-)^{-1}\\bar\\Sigma_{2,2}^{k,\\ell} = \\frac{-a^2C_{k,\\ell}}{(\\nu_k^++\\nu_\\ell^-)(\\nu_k^-+\\nu_\\ell^-)}\\,.\n% $$\n% Set finally $\\bar \\Sigma_{1,1} = P^{-1}D^{-1/2}\\Sigma_{1,1} D^{-1/2}P$.\n% Eq.~(\\ref{eq:lapremiere}) becomes $M^+\\bar \\Sigma_{1,1} + \\bar \\Sigma_{1,1}M^+ = \\bar \\Sigma_{1,2} + \\bar \\Sigma_{1,2}^T$. Thus,\n% \\begin{equation*}\n% \\resizebox{.99\\hsize}{!}{$\n% \\bar\\Sigma_{1,1}^{k,\\ell} = \\frac{\\bar\\Sigma_{1,2}^{k,\\ell}+\\bar\\Sigma_{1,2}^{\\ell,k}}{\\nu_k^++\\nu_\\ell^+}\n% =  \\frac{-a^2C_{k,\\ell}}{(\\nu_k^++\\nu_\\ell^+)(\\nu_k^-+\\nu_\\ell^-)}\\left(\\frac{1}{\\nu_k^++\\nu_\\ell^-}\n% +\\frac{1}{\\nu_k^-+\\nu_\\ell^+}\\right)\n% = \\frac{C_{k,\\ell}}{\\lambda_k+\\lambda_\\ell+\\frac 1{2a}(\\lambda_k-\\lambda_\\ell)^2}\\,,\n% $}\n% \\end{equation*}\n%% \\begin{align*}\n%%  \\bar\\Sigma_{1,1}^{k,\\ell} &= \\frac{\\bar\\Sigma_{1,2}^{k,\\ell}+\\bar\\Sigma_{1,2}^{\\ell,k}}{\\nu_k^++\\nu_\\ell^+}\n%%   =  \\frac{-a^2C_{k,\\ell}}{(\\nu_k^++\\nu_\\ell^+)(\\nu_k^-+\\nu_\\ell^-)}\\left(\\frac{1}{\\nu_k^++\\nu_\\ell^-}\n%%+\\frac{1}{\\nu_k^-+\\nu_\\ell^+}\\right)\n%% \\\\ &= \\frac{C_{k,\\ell}}{\\lambda_k+\\lambda_\\ell+\\frac 1{2a}(\\lambda_k-\\lambda_\\ell)^2}\\,,\n%% \\end{align*}\n% and  the result is proved.\n\n","units":[{"start":156744,"end":157926,"start_line":2692,"end_line":2715,"raw_tex":"\\begin{lemma}\n\\label{lemma:r_n}\nLet the sequence $(r_n)$ be defined as in Algorithm~\\ref{alg:adam-decreasing}.\nAssume that $0 \\leq \\alpha_n \\leq 1$ for all $n$ and that $(1-\\alpha_n)/\\gamma_n \\to a > 0$ as $n \\to +\\infty$.\nThen,\n\\begin{enumerate}[{\\sl i)}]\n\\item $\\forall n \\in \\bN, r_n = 1 - \\prod_{i=1}^n \\alpha_i$,\n\\item The sequence $(r_n)$ is nondecreasing and converges to $1$.\n\\item Under Assumption~\\ref{hyp:step-tcl}~\\ref{step-tcl-i}, for every $\\epsilon>0$, for sufficiently large $n$, we have\n$r_n-1 \\leq e^{-\\frac{a\\gamma_0}{2(1-\\kappa)}n^{1-\\kappa}}$ if $\\kappa \\in (0,1)$ and\n$r_n-1 \\leq n^{-a \\gamma_0/(1+\\epsilon)}$ if $\\kappa = 1$.\n% $\n%      r_n-1 \\leq\n%      \\begin{cases}\n%         e^{-\\frac{a\\gamma_0}{2(1-\\kappa)}n^{1-\\kappa}} \\quad \\text{if}\\,\\,\\kappa \\in (0,1)\\\\\n%         n^{-a \\gamma_0/(1+\\epsilon)} \\quad \\text{if}\\,\\,\\kappa=1 \\,.\n%      \\end{cases}\n% $\n%\\item If $\\gamma_n = \\gamma_0 n^{-\\alpha}$ with $\\alpha \\in (0,1)$, then\n%for sufficiently large $n$, $1-r_n \\leq e^{-\\frac{a\\gamma_0}{2(1-\\alpha)}n^{1-\\alpha}}$\\,.\n%\\item If $\\gamma_n = \\gamma_0 n^{-1}$, then for sufficiently large $n$,\n%$1-r_n \\leq n^{-a \\gamma_0/2}$\\,.\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"45fd801714445cc2e4107a403d160ddc4fd3968349d67b63637d411f235fe557"},{"start":157981,"end":158699,"start_line":2717,"end_line":2729,"raw_tex":"\\begin{proof}\ni) stems from observing that $r_{n+1} -1 = \\alpha_{n+1}(r_n -1)$ for every $n \\in \\bN$ and\niterating this relation ($r_0=0$). As a consequence, the sequence $(r_n)$ is nondecreasing.\n%We now show iii) and iv).\nWe can write :\n$\n0 \\leq 1-r_n \\leq \\exp(- \\sum_{i=1}^n (1-\\alpha_i))\\,.\n$\niii) As $\\sum_{n\\geq 1} \\gamma_n = + \\infty$ %for $\\gamma_n = \\gamma_0 n^{-\\kappa}$ with $\\kappa \\in [0,1]$\nand $(1-\\alpha_n) \\sim a\\gamma_n$, we deduce that $\\sum_{i=1}^n (1-\\alpha_i) \\sim \\sum_{i=1}^n a\\gamma_i$.\nThe results follow from the fact that $\\sum_{i=1}^n \\gamma_i \\sim \\frac{\\gamma_0}{1-\\kappa} n^{1-\\kappa}$\nwhen $\\kappa \\in (0,1)$ and $\\sum_{i=1}^n \\gamma_i \\sim \\gamma_0 \\ln n$ for $\\kappa=1$.\n\\end{proof}","sha256_utf8":"9a955699a28923acc58c8a3ba098c4ec7789578e043fa34f17e95b59659bc21f"},{"start":172462,"end":194374,"start_line":2950,"end_line":3322,"raw_tex":"\\subsection{Proof of Th.~\\ref{thm:clt}}\n\\label{sec:clt}\n\nWe use~\\cite[Th.~1]{pelletier1998weak}.\nAll the assumptions in the latter can be verified in our case, at the exception of\na positive definiteness condition on the\nlimiting covariance matrix, which corresponds, in our case, to the matrix $Q$\ngiven by Eq.~(\\ref{eq:Q}). As $Q$ is not positive definite, it is strictly speaking not possible\nto just cite and apply \\cite[Th.~1]{pelletier1998weak}. Nevertheless,\na detailed inspection of the proofs of \\cite{pelletier1998weak} shows that only a minor\nadaptation is needed in order to cover the present case.\n%Therefore, it is worthless reprove the convergence result of \\cite{pelletier1998weak} from scratch.\nTherefore, proving the convergence result of \\cite{pelletier1998weak} from scratch is worthless.\nIt is sufficient to verify the assumptions of \\cite[Th.~1]{pelletier1998weak}\n(except the definiteness of $Q$) and then to point out the specific part of the proof of \\cite{pelletier1998weak}\nwhich requires some adaptation.\n% The following lemma is a consequence of \\cite[Part 3.4]{pelletier1998weak}.\n% \\begin{lemma}\n%   Let $z^*\\in \\bR^k$ where $k\\geq 1$. On some probability space equipped with a filtration $\\cF=(\\cF_n)_n$, consider a random sequence on $\\bR^k$ given by\n% $$\n% Z_{n+1} = Z_n + \\gamma_{n+1}\\bar h(Z_n) + \\gamma_{n+1}\\eta_{n+1}+\\gamma_{n+1}\\epsilon_{n+1}\\,,\n% $$\n% where $(\\gamma_n)$ is a deterministic sequence, $\\bar h:\\bR^k\\to\\bR^k$ is measurable,\n% $(\\eta_n)$, $(\\epsilon_n)$ are adapted to $\\cF$. Assume the following conditions.\n% \\begin{itemize}[leftmargin=24pt]\n% \\item[\\emph{(A1)}] It holds that $\\bar h(z^*)=0$. Moreover, $\\bar h$ is continuously differentiable in a neighborhood of $z^*$,\n% and its Jacobian matrix $\\bar H$ at $z^*$ is stable \\emph{i.e.}, the largest real part of its eignevalues\n% is $(-\\bar L)$ with $\\bar L>0$.\n% \\item[\\emph{(A2)}] There exists\n% \\end{itemize}\n% \\end{lemma}\n\nLet $z_n=(x_n,m_n,v_n)$ be the output of Algorithm~\\ref{alg:adam-decreasing}.\nDefine $z^*=(x^*,0,S(x^*))$.\n% Remark that the conditional central limit theorem stated by \\cite{pelletier1998weak} shall be valid\n% only given the event $\\{z_n\\to z^*\\}$, whereas we seek to prove the convergence result stated in Th.~\\ref{thm:clt}\n% given the event $\\{x_n\\to x^*\\}$. As a matter of fact, both events coincide up to a $\\bP$-negligible set,\n% so it is safe to prove the convergence on the event $\\{z_n\\to z^*\\}$.\n% The claim that the two events coincide trivially holds if one lets the assumptions of Th.~\\ref{thm:as_conv_under_stab} be satisfied,\n% because in that case, $m_n\\to 0$ and $v_n- S(x_n)\\to 0$ a.s., so that $z_n\\to z^*$ a.s. on the event $\\{x_n\\to x^*\\}$.\n% But the claim is also satisfied if only the assumptions of Th.~\\ref{thm:clt} are considered.\n% As this point is somewhat technical and maybe non essential in a first reading, we postpone the discussion\n% to the end of this section.\nDefine $\\eta_{n+1} \\eqdef (0, a(\\nabla f_{n+1} - \\nabla F(x_n)), b(\\nabla f_{n+1}^{\\odot 2}- S(x_n)))$.\nWe have\n\\begin{equation}\n  \\label{eq:zn_dec}\n  z_{n+1} = z_n + \\gamma_{n+1} h_\\infty(z_n) + \\gamma_{n+1}\\eta_{n+1} + \\gamma_{n+1} \\epsilon_{n+1}\\,,\n\\end{equation}\nwhere $\\epsilon_{n+1} \\eqdef (\\epsilon_{n+1}^1,\\epsilon_{n+1}^2,\\epsilon^3_{n+1})$, whose components are given by\n\\begin{equation*}\n  \\resizebox{\\hsize}{!}{$\n  \\epsilon_{n+1}^1 =  \\frac{m_n}{\\varepsilon+\\sqrt{v_n}} - \\frac{\\hat m_{n+1}}{\\varepsilon+\\sqrt{\\hat v_{n+1}}};\\,\n  \\epsilon_{n+1}^2 = \\left(\\frac{1-\\alpha_{n+1}}{\\gamma_{n+1}}-a\\right)\\left( \\nabla f_{n+1} - m_n\\right);\\,\n  \\epsilon_{n+1}^3 =  \\left(\\frac{1-\\beta_{n+1}}{\\gamma_{n+1}}-b\\right)\\left( \\nabla f_{n+1}^{\\odot 2} - v_n\\right)\\,.\n  $}\n\\end{equation*}\nHere, $\\eta_{n+1}$ is a martingale increment noise and\n$\\epsilon_{n+1} = (\\epsilon_{n+1}^1,\\epsilon_{n+1}^2,\\epsilon^3_{n+1})$ is a remainder term.\nThe aim is to check the assumptions (A1.1) to (A1.3) of\n\\cite{pelletier1998weak}, where the role of the quantities ($h$,\n$\\varepsilon_n$, $r_n$, $\\sigma_n$, $\\alpha$, $\\rho$, $\\beta$) in\n\\cite{pelletier1998weak} is respectively played by the quantities\n($h_\\infty$, $\\eta_n$, $\\epsilon_n$, $\\gamma_n$, $\\kappa$, $1$, $1$) of\nthe present paper.\n\nLet us first consider Assumption (A1.1) for $h_\\infty$.\nBy construction, $h_\\infty(z^*) = 0$. By Assumptions~\\ref{hyp:mean_field_tcl} and \\ref{hyp:S>0},\n$h_\\infty$ is continuously differentiable in the neighborhood of $z^*$ and its\nJacobian at $z^*$ coincides with the matrix $H$ given by Eq.~(\\ref{eq:H}).\n%As already discussed, after straightforward algebra,\nAs already discussed, after some algebra,\nit can be shown that the largest real part of the eigenvalues of $H$ coincides with $-L$ where $L>0$\nis given by Eq.~(\\ref{eq:L}).\nHence, Assumption (A1.1) of \\cite{pelletier1998weak}\nis satisfied for $h_\\infty$. Assumption (A1.3) is trivially satisfied\nusing Assumption~\\ref{hyp:step-tcl}. The crux is therefore to verify Assumption (A1.2).\nClearly, $\\bE(\\eta_{n+1}|\\cF_n)=0$. Using Assumption~\\ref{hyp:moment-f}\\ref{momentreinforce},\nit follows from straightforward manipulations based on Jensen's inequality\nthat for any $M>0$, %s.t. $B(z^*,M)\\subset \\cV$,\nthere exists $\\delta>0$ s.t.\n$%$$\n\\sup_{n\\geq 0}\\bE_n\\left(\\|\\eta_{n+1}\\|^{2+\\delta}\\right) \\1_{\\{\\|z_n-z^*\\|\\leq M\\}}<\\infty\\,.\n$ %$$\nNext, we verify the condition\n\\begin{equation}\n  \\label{eq:cond-reste}\n  \\lim_{n\\to\\infty} \\bE\\left(\\gamma_{n+1}^{-1}\\|\\epsilon_{n+1}\\|^2\\1_{\\{\\|z_n-z^*\\|\\leq M\\}}\\right) = 0\\,.\n\\end{equation}\nIt is sufficient to verify the latter for $\\epsilon^i_n$ ($i=1,2,3$) in place of $\\epsilon_n$.\nThe map $(m,v)\\mapsto m/(\\varepsilon+\\sqrt{v})$ is Lipschitz continuous in a neighborhood of\n$(0,S(x^*))$ by Assumption~\\ref{hyp:S>0}. Thus, for $M$ small enough, there exists a constant $C$ s.t.\nif $ \\|z_n-z^*\\|\\leq M$, then\n%$$\n%  \\|\\epsilon_{n+1}^1\\| \\leq C\\left\\|\\frac{m_{n+1}}{r_{n+1}}-m_n\\right\\|+C\\left\\|\\frac {v_{n+1}}{\\bar r_{n+1}}-v_n\\right\\| \\,.\n%$$\n$%$$\n  \\|\\epsilon_{n+1}^1\\| \\leq C\\left\\|r_{n+1}^{-1}m_{n+1}-m_n\\right\\|+C\\left\\|\\bar r_{n+1}^{-1} v_{n+1}-v_n\\right\\|\\,.\n$ %$$\nUsing the triangular inequality and the fact that $r_{n+1},\\bar r_{n+1}$ are bounded sequences away from zero, there exists an other constant $C$ s.t.\n\\begin{align*}\n  \\|\\epsilon_{n+1}^1\\| &\\leq C\\left\\|m_{n+1}-m_n\\right\\|+C\\left\\|v_{n+1}-v_{n}\\right\\|\n+C|r_{n+1}-1|+C|\\bar r_{n+1}-1|\\,.\n\\end{align*}\n%Using standard manipulations of series along with Assumption~\\ref{hyp:step-tcl}, it can be shown that\nUsing Lemma~\\ref{lemma:r_n} under Assumption~\\ref{hyp:step-tcl} (note that $\\gamma_0 > 1/2L \\geq 1/a$ when $\\kappa=1$),\nwe obtain that the sequence $|r_{n}-1|/\\gamma_n$ is bounded, thus $|r_{n+1}-1|\\leq C\\gamma_{n+1}$.\n%{\\color{red} voir si on a la place de le mettre dans un lemme.}\n The sequence $(1-\\alpha_{n})/\\gamma_n$ being also bounded, it holds that\n%$\\|m_{n+1}-m_n\\|\\leq C\\gamma_{n+1}\\|\\nabla f_{n+1}-m_n\\|$, and if $ \\|z_n-z^*\\|\\leq M$,\n\\begin{align*}\n  \\|m_{n+1}-m_n\\|^2 \\1_{\\{\\|z_n-z^*\\|\\leq M\\}} \\leq  C\\gamma_{n+1}^2(1+\\|\\nabla f_{n+1}\\|^2 )\\1_{\\{\\|z_n-z^*\\|\\leq M\\}}\\,.\n\\end{align*}\nBy Assumption~\\ref{hyp:moment-f}~\\ref{momentreinforce}, %Assumption~\\ref{hyp:mean_field_tcl},\n$\\bE_n(\\|\\nabla f_{n+1}\\|^2|)$ is bounded by a deterministic constant on $\\{\\|z_n-z^*\\|\\leq M\\}$.\nThus, $\\bE_n(\\|m_{n+1}-m_n\\|^2\\1_{\\{\\|z_n-z^*\\|\\leq M\\}})\\leq C\\gamma_{n+1}^2$.\nA similar result holds for $\\|v_{n+1}-v_n\\|^2$. We have thus shown that\n$ \\bE_n\\left(\\|\\epsilon_{n+1}^1\\|^2\\1_{\\{\\|z_n-z^*\\|\\leq M\\}}\\right)\\leq C\\gamma_{n+1}^2$. Hence,\nEq.~(\\ref{eq:cond-reste}) is proved for $\\epsilon_{n+1}^1$ in place of $\\epsilon_{n+1}$.\n%As far as $\\epsilon_{n+1}^2$, $\\epsilon_{n+1}^3$ are concerned, the proof uses the same kind of arguments\n%under Assumption~\\ref{hyp:step-tcl} and is omitted.\nUnder Assumption~\\ref{hyp:step-tcl}, the proof uses the same kind of arguments for $\\epsilon_{n+1}^2$, $\\epsilon_{n+1}^3$  and is omitted.\nFinally, Eq.~(\\ref{eq:cond-reste}) is proved.\nContinuing the verification of Assumption (A1.2), we establish that\n\\begin{equation}\n  \\label{eq:lim-cov}\n  \\bE_n(\\eta_{n+1}\\eta_{n+1}^T) \\to Q\\textrm{ a.s. on } \\{z_n\\to z^*\\}\\,.\n\\end{equation}\nDenote by $\\bar Q(x)$ the matrix given by the righthand side of Eq.~(\\ref{eq:Q}) when $x^*$\nis replaced by an arbitrary $x\\in \\cV$. It is easily checked that $\\bE_n(\\eta_{n+1}\\eta_{n+1}^T)=\\bar Q(x_n)$\nand by continuity, $\\bar Q(x_n)\\to Q$ a.s. on $\\{z_n\\to z^*\\}$, which proves (\\ref{eq:lim-cov}).\nTherefore, Assumption (A1.2) is fulfilled, except for the point mentioned at the beginning of this section : %paragraph:\n\\cite{pelletier1998weak} puts the additional condition that the limit\nmatrix in Eq.~(\\ref{eq:lim-cov}) is positive definite. This condition is not satisfied in our case,\nbut the proof can still be adapted. The specific part of the proof where the positive definiteness\ncomes into play is Th.~7 in \\cite{pelletier1998weak}. The proof of \\cite[Th.~1]{pelletier1998weak} can therefore\nbe adapted to the case of a positive semidefinite matrix.\n%by substituting the following\n%result to Th.~7 of the same paper.\nIn the proof of \\cite[Th.~7]{pelletier1998weak}, we only substitute the inverse of the square root of $Q$ by the Moore-Penrose inverse.\nFinally, the uniqueness of the stationary distribution $\\mu$ and its expression follow from \\cite[Th.~6.7, p. 357]{karatzas-shreve1991}.\n%{\\color{red} Citer Karatsas-Shreeve, 2d Ed., Th. 6.7.}\n% \\begin{theorem}[adapted from \\cite{pelletier1998weak}, Th.~7]\n% Let $k\\geq 1$. On some probability space equipped with a filtration $\\cF=(\\cF_n)_n$,\n% consider a sequence of r.v. on $\\bR^k$ given by\n% $$\n% Z_{n+1} = (I+\\gamma_{n+1} \\bar H)Z_n+\\gamma_{n+1} b_{n+1}+\\sqrt{\\gamma_{n+1}}\\eta_{n+1}\n% $$\n% and $\\bE(\\|Z_0\\|^2)<0$,\n% where $\\bar H$ is a $k\\times k$ stable matrix \\emph{i.e.}, the largest real part of its eigenvalues\n% is negative, $(b_n)$ and $(\\eta_n)$ are random sequences, $(\\gamma_n)$ satisfies Assumption~\\ref{hyp:step-tcl}.\n% Let $\\Omega_0\\in \\cF_\\infty$ have a positive probability. Assume that the following holds almost surely on $\\Omega_0$:\n% \\begin{enumerate}[{i)}, leftmargin=*]\n% \\item $\\bE_n(\\eta_{n+1})=0$.\n% \\item There exists a constant $\\bar b>2$ s.t. $\\sup_{n\\geq 0}\\bE_n(\\|\\eta_{n+1}\\|^{\\bar b})<\\infty$.\n% \\item $\\bE_n(\\eta_{n+1}\\eta_{n+1}^T) = C + \\Delta_n$ where $\\bE(\\|\\Delta_n\\|\\1_{\\Omega_0})\\to 0$\n% and $C$ is a positive semidefinite matrix.\n% \\item The sequence $(b_n)$ is the sum of two sequences $(b_{n,1})$ and $(b_{n,2})$, adapted to $\\cF$,\n% s.t. $\\sup_{n\\geq 0}\\bE(\\|b_{n,1}\\|^2)<\\infty$, $\\bE(\\|b_{n,1}\\|\\1_{\\Omega_0})\\to 0$ and $b_{n,2}\\to 0$ a.s. on $\\Omega_0$.\n% \\end{enumerate}\n% Then, given $\\Omega_0$, $(Z_n)$ converges in distribution to the unique stationary distribution $\\mu$ of the generalized Ornstein-Uhlenbeck process\n% $$\n% dX_t = \\bar H X_t dt + \\sqrt CdB_t\n% $$\n% where $(B_t)$ is the standard Brownian motion and $\\sqrt{C}$ is the unique positive semidefinite square root of $C$.\n% The distribution $\\mu$ is the zero mean Gaussian distribution with covariance matrix $\\bar \\Sigma$\n% given as the solution to $\\bar H\\bar \\Sigma+\\bar \\Sigma\\bar H^T = -C$.\n% \\end{theorem}\n% \\begin{proof}\n%   The proof is identical to the proof of \\cite[Th.~7]{pelletier1998weak}, only substituting the\n% inverse of the square root of $C$ by the Moore-Penrose inverse. Finally, the uniqueness of the\n% stationary distribution $\\mu$ and its expression follow from \\cite[Th.~6.7, p. 357]{karatzas-shreve1991}.%{\\color{red} Citer Karatsas-Shreeve, 2d Ed., Th. 6.7.}\n% \\end{proof}\n\n%\\subsection{Proof of Cor.~\\ref{cor:x}}\n\\noindent\\textbf{Proof of Eq.~(\\ref{eq:cov})}.\nWe introduce the $d\\times d$ blocks of the $3d\\times 3d$ matrix %$\\Sigma$ by\n$\\Sigma = \\left(\\Sigma_{i,j}\\right)_{i,j=1,2,3}$\nwhere $\\Sigma_{i,j}$ is $d\\times d$. We denote by $\\tilde\\Sigma$ the $2d\\times 2d$ submatrix $\\tilde\\Sigma \\eqdef\\left(\\Sigma_{i,j}\\right)_{i,j=1,2}$.\nBy Th.~\\ref{thm:clt}, we have the subsystem:\n\\begin{equation}\n\\tilde H \\tilde\\Sigma + \\tilde\\Sigma\\tilde H^T =\n\\begin{pmatrix}\n  0 & 0 \\\\\n0 & -a^2 \\tilde Q\n\\end{pmatrix}\\qquad\\text{where }\\tilde H \\eqdef\n\\begin{pmatrix}\n  \\zeta I_d & -D \\\\\na\\nabla^2F(x^*) & (\\zeta-a) I_d\n\\end{pmatrix}\\label{eq:lyap-reduced}\n\\end{equation}\nand where $\\tilde Q \\eqdef \\textrm{Cov}\\left(\\nabla\n  f(x^*,\\xi)\\right)$.  The next step is to triangularize the matrix\n$\\tilde H$ in order to decouple the blocks of $\\tilde\\Sigma$.  For\nevery $k=1,\\dots,d$, set\n$\\nu_k^\\pm \\eqdef -\\frac{a}{2}\\pm\\sqrt{a^2/4-a\\lambda_k}$\nwith the convention that $\\sqrt{-1} =\n\\imath$ (inspecting the characteristic polynomial of $\\tilde H$, these %quantities\nare the eigenvalues of $\\tilde H$). Set\n$M^\\pm\\eqdef\\diag{(\\nu_1^\\pm,\n\\cdots,\\nu_d^\\pm)}$ and $R^\\pm\\eqdef\nD^{-1/2}PM^\\pm P^TD^{-1/2}$.  Using the identities $M^++M^-=-a I_d$ and\n$M^+M^-=a\\Lambda$ where\n$\\Lambda\\eqdef\\diag{(\\lambda_1,\\cdots,\\lambda_d)}$, it can be checked\nthat\n$$\n\\cR\\tilde H =\\begin{pmatrix}\n  D R^+ + \\zeta I_d & -D \\\\ 0 & R^-D + \\zeta I_d\n\\end{pmatrix}\\cR,\\text{ where }\\cR\\eqdef\n\\begin{pmatrix}\n  I_d & 0 \\\\ R^+ & I_d\n\\end{pmatrix}\\,.\n$$\nSet $\\check \\Sigma \\eqdef \\cR\\tilde\\Sigma\\cR^T$. Denote by\n$(\\check \\Sigma_{i,j})_{i,j=1,2}$ the blocks of $\\check\\Sigma$.\nNote that $\\check\\Sigma_{1,1} = \\Sigma_{1,1}$. By left/right multiplication of Eq.~(\\ref{eq:lyap-reduced})\nrespectively with $\\cR$ and $\\cR^T$, we obtain\n%% $$\n%% \\cT\\check\\Sigma + \\check\\Sigma\\cT^T =\n%% \\begin{pmatrix}\n%%   0 & 0 \\\\\n%% 0 & -a^2 \\tilde Q\n%% \\end{pmatrix}\\,.\n%% $$\n%% By expanding the above system, we obtain\n\\begin{align}\n  &(DR^++\\zeta I_d) \\Sigma_{1,1}+\\Sigma_{1,1}(R^+D+\\zeta I_d) = \\check\\Sigma_{1,2} D+ D\\check\\Sigma_{1,2}^T \\label{eq:lapremiere}\\\\\n& (DR^++\\zeta I_d) \\check\\Sigma_{1,2}+\\check\\Sigma_{1,2} (DR^-+\\zeta I_d) = D\\check\\Sigma_{2,2} \\label{eq:ladeuxieme}\\\\\n& (R^-D+\\zeta I_d) \\check\\Sigma_{2,2} + \\check\\Sigma_{2,2} (DR^-+\\zeta I_d) = -a^2\\tilde Q \\label{eq:laderniere}\n\\end{align}\nSet $\\bar \\Sigma_{2,2} = P^{-1}D^{1/2}\\check\\Sigma_{2,2} D^{1/2}P$.\nDefine $C\\eqdef P^{-1}D^{1/2}\\tilde Q D^{1/2}P$.\nEq.~(\\ref{eq:laderniere}) yields $(M^-+\\zeta I_d) \\bar\\Sigma_{2,2} + \\bar\\Sigma_{2,2} (M^-+\\zeta I_d) = -a^2C$.\n%% The component $(k,\\ell)$ is given by\n%% $$\n%% \\bar\\Sigma_{2,2}^{k,\\ell} = -a^2\\frac{C_{k,\\ell}}{\\nu_k^-+\\nu_\\ell^-}\\,.\n%% $$\nSet $\\bar \\Sigma_{1,2} = P^{-1}D^{-1/2}\\check\\Sigma_{1,2} D^{1/2}P$.\nEq.~(\\ref{eq:ladeuxieme}) rewrites $(M^++\\zeta I_d) \\bar \\Sigma_{1,2}+\\bar \\Sigma_{1,2} (M^-+\\zeta I_d) = \\bar \\Sigma_{2,2}$.\nWe obtain that %The component $(k,\\ell)$ is given by\n$%$$\n\\bar\\Sigma_{1,2}^{k,\\ell} = (\\nu_k^++\\nu_\\ell^-+2\\zeta)^{-1}\\bar\\Sigma_{2,2}^{k,\\ell} = \\frac{-a^2C_{k,\\ell}}{(\\nu_k^++\\nu_\\ell^-+2\\zeta)(\\nu_k^-+\\nu_\\ell^-+2\\zeta)}\\,.\n$ %$$\nSet %finally\n$\\bar \\Sigma_{1,1} = P^{-1}D^{-1/2}\\Sigma_{1,1} D^{-1/2}P$.\nEq.~(\\ref{eq:lapremiere}) becomes $(M^++\\zeta I_d)\\bar \\Sigma_{1,1} + \\bar \\Sigma_{1,1}(M^++\\zeta I_d) = \\bar \\Sigma_{1,2} + \\bar \\Sigma_{1,2}^T$. Thus,\n%\\begin{equation*}\n%\\resizebox{.99\\hsize}{!}{$\n\\begin{align*}\n\\bar\\Sigma_{1,1}^{k,\\ell} &\n\\resizebox{.9\\hsize}{!}{$\n= \\frac{\\bar\\Sigma_{1,2}^{k,\\ell}+\\bar\\Sigma_{1,2}^{\\ell,k}}{\\nu_k^++\\nu_\\ell^++2\\zeta}\n=  \\frac{-a^2C_{k,\\ell}}{(\\nu_k^++\\nu_\\ell^++2\\zeta)(\\nu_k^-+\\nu_\\ell^-+2\\zeta)}\\left(\\frac{1}{\\nu_k^++\\nu_\\ell^-+2\\zeta}\n+\\frac{1}{\\nu_k^-+\\nu_\\ell^++2\\zeta}\\right)\n$}\n\\\\ &\n%\\resizebox{.6\\hsize}{!}{$= \\frac{C_{k,\\ell}}{ (1 - \\frac{2\\zeta}{a})(\\lambda_k+\\lambda_\\ell-2\\zeta + \\frac 2a \\zeta^2) +\\frac 1{2(a-2\\zeta)}(\\lambda_k-\\lambda_\\ell)^2}\\,.$}\n= \\frac{C_{k,\\ell}}{ (1 - \\frac{2\\zeta}{a})(\\lambda_k+\\lambda_\\ell-2\\zeta + \\frac 2a \\zeta^2) +\\frac 1{2(a-2\\zeta)}(\\lambda_k-\\lambda_\\ell)^2}\\,,\n%\\,\\,\\text{and  the result is proved.}\n\\end{align*}\nand  the result is proved.\n% \\noindent\\textbf{The events $\\{x_n\\to x^*\\}$ and $\\{z_n\\to z^*\\}$ coincide $\\bP$-a.e.}\n% Set $\\bar \\gamma_n = 1-\\alpha_{n}$. Define the event $\\Gamma_N\\eqdef\\{x_n\\to x^*\\text{ and }\\sup_n\\|x_n-x^*\\|\\leq N\\}$.\n% It is sufficient to show that for all $N>0$, $m_n\\to 0$ and $v_n\\to S(x^*)$ a.s. on $\\Gamma_N$.\n% Considering the iterates $v_n$, we have\n% \\begin{align*}\n% v_{n+1}\\1_{\\Gamma_N}  &= (1-\\bar\\gamma_{n+1})v_{n}\\1_{\\Gamma_N}  +\\bar\\gamma_{n+1} S(x^*)\\1_{\\Gamma_N}\n% + \\bar\\gamma_{n+1} \\tilde \\chi_{n+1} + \\bar\\gamma_{n+1} \\tilde \\varsigma_n^1 + \\bar\\gamma_{n+1} \\tilde \\varsigma_n^2 \\\\\n% \\text{where }\\quad&\n%   \\resizebox{.85\\hsize}{!}{$\n% \\tilde \\chi_{n+1} \\eqdef (\\nabla f_{n+1}^{\\odot\n%     2}-S(x_n))\\1_{\\Gamma_N} - \\bE_n((\\nabla f_{n+1}^{\\odot\n%     2}-S(x_n))\\1_{\\Gamma_N})\n%  + \\bE_n[(\\nabla f_{n+1}^{\\odot{2}} - S(x_n)) \\bE_n(\\1_{\\Gamma_N})]$}\\\\\n% & \\tilde \\varsigma_n^1\\eqdef (S(x_n)-S(x^*))\\1_{\\Gamma_N}\\\\\n% &\\tilde\\varsigma_n^2\\eqdef \\bE_n( (\\nabla f_{n+1}^{\\odot 2}-S(x_n))\\1_{\\|x_n-x^*\\|\\leq N}(\\1_{\\Gamma_N}-\\bE_n(\\1_{\\Gamma_N})) )\\,.\n% \\end{align*}\n% Clearly, $\\tilde \\varsigma_n^1\\to 0$ a.s. The same holds for $\\tilde\\varsigma_n^2$, using H\\\"older's inequality along\n% with Assumption~\\ref{hyp:moment-f}-\\ref{momentreinforce} for $p=2$\n% and the fact that $\\bE_n(\\1_{\\Gamma_N})$ converges a.s. to $\\1_{\\Gamma_N}$.\n% Moreover, $\\bE_n(\\|\\tilde\\chi_{n+1}\\|^2)\\leq C$ for some $C>0$, using again Assumption~\\ref{hyp:moment-f}-\\ref{momentreinforce}\n% and Jensen's inequality. As $\\sum_k\\gamma_k^2<\\infty$, this in turn implies that $\\lim_n\\sum_{k\\leq n}\\gamma_k\\chi_k$ exists.\n% Using the same kind of arguments, one can show that $(v_n)$ is a.s. bounded on $\\Gamma_N$.\n% Thus, as in the proof of Th.~\\ref{thm:as_conv_under_stab},\n% we have shown that, almost surely on $\\Gamma_N$, the piecewise interpolated process of the sequence $(v_n)$ is a bounded APT of\n% $\\dot v=S(x^*)-v$, hence showing that $v_n\\to S(x^*)$ a.s. on that set. The same arguments can be used to show that\n% $m_n\\to 0$ a.s. on $\\Gamma_N$, and the proof is concluded.\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n% \\subsection{Proof of Cor.~\\ref{cor:x}}\n%\n% We introduce the $d\\times d$ blocks of the $3d\\times 3d$ matrix %$\\Sigma$ by\n% $\\Sigma = \\left(\\Sigma_{i,j}\\right)_{i,j=1,2,3}$\n% where $\\Sigma_{i,j}$ is $d\\times d$. We denote by $\\tilde\\Sigma$ the $2d\\times 2d$ submatrix $\\tilde\\Sigma \\eqdef\\left(\\Sigma_{i,j}\\right)_{i,j=1,2}$.\n% By Th.~\\ref{thm:clt}, we have the subsystem:\n% \\begin{equation}\n% \\tilde H \\tilde\\Sigma + \\tilde\\Sigma\\tilde H^T =\n% \\begin{pmatrix}\n%   0 & 0 \\\\\n% 0 & -a^2 \\tilde Q\n% \\end{pmatrix}\\qquad\\text{where }\\tilde H \\eqdef\n% \\begin{pmatrix}\n%   0 & -D \\\\\n% a\\nabla^2F(x^*) & -a I_d\n% \\end{pmatrix}\\label{eq:lyap-reduced}\n% \\end{equation}\n% and where $\\tilde Q \\eqdef \\textrm{Cov}\\left(\\nabla\n%   f(x^*,\\xi)\\right)$.  The next step is to triangularize the matrix\n% $\\tilde H$ in order to decouple the blocks of $\\tilde\\Sigma$.  For\n% every $k=1,\\dots,d$, set $\\nu_k^\\pm \\eqdef -\\frac\n% a2\\pm\\sqrt{a^2/4-a\\lambda_k}$ with the convention that $\\sqrt{-1} =\n% \\imath$ (inspecting the characteristic polynomial of $\\tilde H$, these\n% quantities are the eigenvalues of $\\tilde H$). Set\n% $M^\\pm\\eqdef\\diag{(\\nu_1^\\pm,\\cdots,\\nu_d^\\pm)}$ and $R^\\pm\\eqdef\n% D^{-1/2}PM^\\pm P^TD^{-1/2}$.  Using the identities $M^++M^-=-aI_d$ and\n% $M^+M^-=a\\Lambda$ where\n% $\\Lambda\\eqdef\\diag{(\\lambda_1,\\cdots,\\lambda_d)}$, it can be checked\n% that\n% $$\n% \\cR\\tilde H =\\begin{pmatrix}\n%   D R^+ & -D \\\\ 0 & R^-D\n% \\end{pmatrix}\\cR,\\text{ where }\\cR\\eqdef\n% \\begin{pmatrix}\n%   I_d & 0 \\\\ R^+ & I_d\n% \\end{pmatrix}\\,.\n% $$\n% Set $\\check \\Sigma \\eqdef \\cR\\tilde\\Sigma\\cR^T$. Denote by\n% $(\\check \\Sigma_{i,j})_{i,j=1,2}$ the blocks of $\\check\\Sigma$.\n% Note that $\\check\\Sigma_{1,1} = \\Sigma_{1,1}$. By left/right multiplication of Eq.~(\\ref{eq:lyap-reduced})\n% respectively with $\\cR$ and $\\cR^T$, we obtain\n% %% $$\n% %% \\cT\\check\\Sigma + \\check\\Sigma\\cT^T =\n% %% \\begin{pmatrix}\n% %%   0 & 0 \\\\\n% %% 0 & -a^2 \\tilde Q\n% %% \\end{pmatrix}\\,.\n% %% $$\n% %% By expanding the above system, we obtain\n% \\begin{align}\n%   &DR^+\\Sigma_{1,1}+\\Sigma_{1,1}R^+D = \\check\\Sigma_{1,2} D+ D\\check\\Sigma_{1,2}^T \\label{eq:lapremiere}\\\\\n% & DR^+\\check\\Sigma_{1,2}+\\check\\Sigma_{1,2}DR^- = D\\check\\Sigma_{2,2} \\label{eq:ladeuxieme}\\\\\n% & R^-D\\check\\Sigma_{2,2} + \\check\\Sigma_{2,2} DR^- = -a^2\\tilde Q \\label{eq:laderniere}\n% \\end{align}\n% Set $\\bar \\Sigma_{2,2} = P^{-1}D^{1/2}\\check\\Sigma_{2,2} D^{1/2}P$.\n% Define $C\\eqdef P^{-1}D^{1/2}\\tilde Q D^{1/2}P$.\n% Eq.~(\\ref{eq:laderniere}) yields $M^-\\bar\\Sigma_{2,2} + \\bar\\Sigma_{2,2} M^- = -a^2C$.\n% %% The component $(k,\\ell)$ is given by\n% %% $$\n% %% \\bar\\Sigma_{2,2}^{k,\\ell} = -a^2\\frac{C_{k,\\ell}}{\\nu_k^-+\\nu_\\ell^-}\\,.\n% %% $$\n% Set $\\bar \\Sigma_{1,2} = P^{-1}D^{-1/2}\\check\\Sigma_{1,2} D^{1/2}P$.\n% Eq.~(\\ref{eq:ladeuxieme}) rewrites $M^+\\bar \\Sigma_{1,2}+\\bar \\Sigma_{1,2}M^- = \\bar \\Sigma_{2,2}$.\n% The component $(k,\\ell)$ is given by\n% $$\n% \\bar\\Sigma_{1,2}^{k,\\ell} = (\\nu_k^++\\nu_\\ell^-)^{-1}\\bar\\Sigma_{2,2}^{k,\\ell} = \\frac{-a^2C_{k,\\ell}}{(\\nu_k^++\\nu_\\ell^-)(\\nu_k^-+\\nu_\\ell^-)}\\,.\n% $$\n% Set finally $\\bar \\Sigma_{1,1} = P^{-1}D^{-1/2}\\Sigma_{1,1} D^{-1/2}P$.\n% Eq.~(\\ref{eq:lapremiere}) becomes $M^+\\bar \\Sigma_{1,1} + \\bar \\Sigma_{1,1}M^+ = \\bar \\Sigma_{1,2} + \\bar \\Sigma_{1,2}^T$. Thus,\n% \\begin{equation*}\n% \\resizebox{.99\\hsize}{!}{$\n% \\bar\\Sigma_{1,1}^{k,\\ell} = \\frac{\\bar\\Sigma_{1,2}^{k,\\ell}+\\bar\\Sigma_{1,2}^{\\ell,k}}{\\nu_k^++\\nu_\\ell^+}\n% =  \\frac{-a^2C_{k,\\ell}}{(\\nu_k^++\\nu_\\ell^+)(\\nu_k^-+\\nu_\\ell^-)}\\left(\\frac{1}{\\nu_k^++\\nu_\\ell^-}\n% +\\frac{1}{\\nu_k^-+\\nu_\\ell^+}\\right)\n% = \\frac{C_{k,\\ell}}{\\lambda_k+\\lambda_\\ell+\\frac 1{2a}(\\lambda_k-\\lambda_\\ell)^2}\\,,\n% $}\n% \\end{equation*}\n%% \\begin{align*}\n%%  \\bar\\Sigma_{1,1}^{k,\\ell} &= \\frac{\\bar\\Sigma_{1,2}^{k,\\ell}+\\bar\\Sigma_{1,2}^{\\ell,k}}{\\nu_k^++\\nu_\\ell^+}\n%%   =  \\frac{-a^2C_{k,\\ell}}{(\\nu_k^++\\nu_\\ell^+)(\\nu_k^-+\\nu_\\ell^-)}\\left(\\frac{1}{\\nu_k^++\\nu_\\ell^-}\n%%+\\frac{1}{\\nu_k^-+\\nu_\\ell^+}\\right)\n%% \\\\ &= \\frac{C_{k,\\ell}}{\\lambda_k+\\lambda_\\ell+\\frac 1{2a}(\\lambda_k-\\lambda_\\ell)^2}\\,,\n%% \\end{align*}\n% and  the result is proved.\n\n","sha256_utf8":"58ec68fdff3adf63a0c58c5afbe68902129085dcf74a17d465ea302bcf65533a"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"递减步长随机迭代的条件中心极限定理","source_units":[{"start":42904,"end":44653,"start_line":854,"end_line":883,"raw_tex":"\\begin{theorem}\n \\label{thm:clt}\nLet Assumptions~\\ref{hyp:model}, \\ref{hyp:S>0}, \\ref{hyp:iid}, \\ref{hyp:mean_field_tcl} and \\ref{hyp:step-tcl} hold true.\nLet Assumption~\\ref{hyp:moment-f}~\\ref{momentreinforce} hold with $p=4$.\nConsider the iterates $z_n=(x_n,m_n,v_n)$ given by Algorithm~\\ref{alg:adam-decreasing}. Set $z^*=(x^*,0,S(x^*))$.\nSet $\\zeta \\eqdef 0$ if $0<\\kappa<1$ and $\\zeta \\eqdef \\frac{1}{2 \\gamma_0}$ if $\\kappa =1$.\n%Assume $\\bP(x_n \\to x^*)>0$. Then, given the event $\\{x_n\\to x^*\\}$,\nAssume $\\bP(z_n \\to z^*)>0$. Then, given the event $\\{z_n\\to z^*\\}$,\nthe rescaled vector $\\sqrt{\\gamma_n}^{-1}(z_n-z^*)$\nconverges in distribution to a zero mean Gaussian distribution on $\\bR^{3d}$ with a covariance matrix $\\Sigma$\nwhich is solution to the Lyapunov equation: $  \\left(H + \\zeta I_{3d} \\right) \\Sigma + \\Sigma \\left( H^T +  \\zeta I_{3d} \\right) = - Q$.\nIn particular, given $\\{z_n\\to z^*\\}$, the vector $\\sqrt{\\gamma_n}^{-1}(x_n-x^*)$\nconverges in distribution to a zero mean Gaussian distribution with a covariance matrix $\\Sigma_1$ given by:\n\\begin{equation}\n\\Sigma_1 = D^{1/2} P\n\\left(\n\\frac{C_{k,\\ell}}{ (1 - \\frac{2\\zeta}{a})(\\lambda_k+\\lambda_\\ell-2\\zeta + \\frac 2a \\zeta^2) +\\frac 1{2(a-2\\zeta)}(\\lambda_k-\\lambda_\\ell)^2}\n %\\frac{C_{k,\\ell}}{ (1-\\frac{\\zeta}{a})(\\lambda_k+\\lambda_\\ell - 2\\zeta) +\\frac 1{2(a-\\zeta)}(\\lambda_k-\\lambda_\\ell)^2}\n\\right)_{k,\\ell=1\\dots d}\nP^{-1}D^{1/2}\\label{eq:cov}\n\\end{equation}\n%\\begin{equation}\n%\\Sigma_1 = D^{1/2} P\n%\\left(\n%  \\frac{C_{k,\\ell}}{\\lambda_k+\\lambda_\\ell+\\frac 1{2a}(\\lambda_k-\\lambda_\\ell)^2}\n%\\right)_{k,\\ell=1\\dots d}\n%P^{-1}D^{1/2}\\label{eq:cov}\n%\\end{equation}\nwhere $C\\eqdef P^{-1}D^{1/2}\\bE\\left(\\nabla f(x^*,\\xi)\\nabla f(x^*,\\xi)^T\\right)D^{1/2}P$.\n\\end{theorem}","sha256_utf8":"abd832f738d458b86c14ad88e0c2ec516c4eda77cc31ce64193a711e401bcffa"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"条件极限与显式协方差的完整证明","source_units":[{"start":172462,"end":194374,"start_line":2950,"end_line":3322,"raw_tex":"\\subsection{Proof of Th.~\\ref{thm:clt}}\n\\label{sec:clt}\n\nWe use~\\cite[Th.~1]{pelletier1998weak}.\nAll the assumptions in the latter can be verified in our case, at the exception of\na positive definiteness condition on the\nlimiting covariance matrix, which corresponds, in our case, to the matrix $Q$\ngiven by Eq.~(\\ref{eq:Q}). As $Q$ is not positive definite, it is strictly speaking not possible\nto just cite and apply \\cite[Th.~1]{pelletier1998weak}. Nevertheless,\na detailed inspection of the proofs of \\cite{pelletier1998weak} shows that only a minor\nadaptation is needed in order to cover the present case.\n%Therefore, it is worthless reprove the convergence result of \\cite{pelletier1998weak} from scratch.\nTherefore, proving the convergence result of \\cite{pelletier1998weak} from scratch is worthless.\nIt is sufficient to verify the assumptions of \\cite[Th.~1]{pelletier1998weak}\n(except the definiteness of $Q$) and then to point out the specific part of the proof of \\cite{pelletier1998weak}\nwhich requires some adaptation.\n% The following lemma is a consequence of \\cite[Part 3.4]{pelletier1998weak}.\n% \\begin{lemma}\n%   Let $z^*\\in \\bR^k$ where $k\\geq 1$. On some probability space equipped with a filtration $\\cF=(\\cF_n)_n$, consider a random sequence on $\\bR^k$ given by\n% $$\n% Z_{n+1} = Z_n + \\gamma_{n+1}\\bar h(Z_n) + \\gamma_{n+1}\\eta_{n+1}+\\gamma_{n+1}\\epsilon_{n+1}\\,,\n% $$\n% where $(\\gamma_n)$ is a deterministic sequence, $\\bar h:\\bR^k\\to\\bR^k$ is measurable,\n% $(\\eta_n)$, $(\\epsilon_n)$ are adapted to $\\cF$. Assume the following conditions.\n% \\begin{itemize}[leftmargin=24pt]\n% \\item[\\emph{(A1)}] It holds that $\\bar h(z^*)=0$. Moreover, $\\bar h$ is continuously differentiable in a neighborhood of $z^*$,\n% and its Jacobian matrix $\\bar H$ at $z^*$ is stable \\emph{i.e.}, the largest real part of its eignevalues\n% is $(-\\bar L)$ with $\\bar L>0$.\n% \\item[\\emph{(A2)}] There exists\n% \\end{itemize}\n% \\end{lemma}\n\nLet $z_n=(x_n,m_n,v_n)$ be the output of Algorithm~\\ref{alg:adam-decreasing}.\nDefine $z^*=(x^*,0,S(x^*))$.\n% Remark that the conditional central limit theorem stated by \\cite{pelletier1998weak} shall be valid\n% only given the event $\\{z_n\\to z^*\\}$, whereas we seek to prove the convergence result stated in Th.~\\ref{thm:clt}\n% given the event $\\{x_n\\to x^*\\}$. As a matter of fact, both events coincide up to a $\\bP$-negligible set,\n% so it is safe to prove the convergence on the event $\\{z_n\\to z^*\\}$.\n% The claim that the two events coincide trivially holds if one lets the assumptions of Th.~\\ref{thm:as_conv_under_stab} be satisfied,\n% because in that case, $m_n\\to 0$ and $v_n- S(x_n)\\to 0$ a.s., so that $z_n\\to z^*$ a.s. on the event $\\{x_n\\to x^*\\}$.\n% But the claim is also satisfied if only the assumptions of Th.~\\ref{thm:clt} are considered.\n% As this point is somewhat technical and maybe non essential in a first reading, we postpone the discussion\n% to the end of this section.\nDefine $\\eta_{n+1} \\eqdef (0, a(\\nabla f_{n+1} - \\nabla F(x_n)), b(\\nabla f_{n+1}^{\\odot 2}- S(x_n)))$.\nWe have\n\\begin{equation}\n  \\label{eq:zn_dec}\n  z_{n+1} = z_n + \\gamma_{n+1} h_\\infty(z_n) + \\gamma_{n+1}\\eta_{n+1} + \\gamma_{n+1} \\epsilon_{n+1}\\,,\n\\end{equation}\nwhere $\\epsilon_{n+1} \\eqdef (\\epsilon_{n+1}^1,\\epsilon_{n+1}^2,\\epsilon^3_{n+1})$, whose components are given by\n\\begin{equation*}\n  \\resizebox{\\hsize}{!}{$\n  \\epsilon_{n+1}^1 =  \\frac{m_n}{\\varepsilon+\\sqrt{v_n}} - \\frac{\\hat m_{n+1}}{\\varepsilon+\\sqrt{\\hat v_{n+1}}};\\,\n  \\epsilon_{n+1}^2 = \\left(\\frac{1-\\alpha_{n+1}}{\\gamma_{n+1}}-a\\right)\\left( \\nabla f_{n+1} - m_n\\right);\\,\n  \\epsilon_{n+1}^3 =  \\left(\\frac{1-\\beta_{n+1}}{\\gamma_{n+1}}-b\\right)\\left( \\nabla f_{n+1}^{\\odot 2} - v_n\\right)\\,.\n  $}\n\\end{equation*}\nHere, $\\eta_{n+1}$ is a martingale increment noise and\n$\\epsilon_{n+1} = (\\epsilon_{n+1}^1,\\epsilon_{n+1}^2,\\epsilon^3_{n+1})$ is a remainder term.\nThe aim is to check the assumptions (A1.1) to (A1.3) of\n\\cite{pelletier1998weak}, where the role of the quantities ($h$,\n$\\varepsilon_n$, $r_n$, $\\sigma_n$, $\\alpha$, $\\rho$, $\\beta$) in\n\\cite{pelletier1998weak} is respectively played by the quantities\n($h_\\infty$, $\\eta_n$, $\\epsilon_n$, $\\gamma_n$, $\\kappa$, $1$, $1$) of\nthe present paper.\n\nLet us first consider Assumption (A1.1) for $h_\\infty$.\nBy construction, $h_\\infty(z^*) = 0$. By Assumptions~\\ref{hyp:mean_field_tcl} and \\ref{hyp:S>0},\n$h_\\infty$ is continuously differentiable in the neighborhood of $z^*$ and its\nJacobian at $z^*$ coincides with the matrix $H$ given by Eq.~(\\ref{eq:H}).\n%As already discussed, after straightforward algebra,\nAs already discussed, after some algebra,\nit can be shown that the largest real part of the eigenvalues of $H$ coincides with $-L$ where $L>0$\nis given by Eq.~(\\ref{eq:L}).\nHence, Assumption (A1.1) of \\cite{pelletier1998weak}\nis satisfied for $h_\\infty$. Assumption (A1.3) is trivially satisfied\nusing Assumption~\\ref{hyp:step-tcl}. The crux is therefore to verify Assumption (A1.2).\nClearly, $\\bE(\\eta_{n+1}|\\cF_n)=0$. Using Assumption~\\ref{hyp:moment-f}\\ref{momentreinforce},\nit follows from straightforward manipulations based on Jensen's inequality\nthat for any $M>0$, %s.t. $B(z^*,M)\\subset \\cV$,\nthere exists $\\delta>0$ s.t.\n$%$$\n\\sup_{n\\geq 0}\\bE_n\\left(\\|\\eta_{n+1}\\|^{2+\\delta}\\right) \\1_{\\{\\|z_n-z^*\\|\\leq M\\}}<\\infty\\,.\n$ %$$\nNext, we verify the condition\n\\begin{equation}\n  \\label{eq:cond-reste}\n  \\lim_{n\\to\\infty} \\bE\\left(\\gamma_{n+1}^{-1}\\|\\epsilon_{n+1}\\|^2\\1_{\\{\\|z_n-z^*\\|\\leq M\\}}\\right) = 0\\,.\n\\end{equation}\nIt is sufficient to verify the latter for $\\epsilon^i_n$ ($i=1,2,3$) in place of $\\epsilon_n$.\nThe map $(m,v)\\mapsto m/(\\varepsilon+\\sqrt{v})$ is Lipschitz continuous in a neighborhood of\n$(0,S(x^*))$ by Assumption~\\ref{hyp:S>0}. Thus, for $M$ small enough, there exists a constant $C$ s.t.\nif $ \\|z_n-z^*\\|\\leq M$, then\n%$$\n%  \\|\\epsilon_{n+1}^1\\| \\leq C\\left\\|\\frac{m_{n+1}}{r_{n+1}}-m_n\\right\\|+C\\left\\|\\frac {v_{n+1}}{\\bar r_{n+1}}-v_n\\right\\| \\,.\n%$$\n$%$$\n  \\|\\epsilon_{n+1}^1\\| \\leq C\\left\\|r_{n+1}^{-1}m_{n+1}-m_n\\right\\|+C\\left\\|\\bar r_{n+1}^{-1} v_{n+1}-v_n\\right\\|\\,.\n$ %$$\nUsing the triangular inequality and the fact that $r_{n+1},\\bar r_{n+1}$ are bounded sequences away from zero, there exists an other constant $C$ s.t.\n\\begin{align*}\n  \\|\\epsilon_{n+1}^1\\| &\\leq C\\left\\|m_{n+1}-m_n\\right\\|+C\\left\\|v_{n+1}-v_{n}\\right\\|\n+C|r_{n+1}-1|+C|\\bar r_{n+1}-1|\\,.\n\\end{align*}\n%Using standard manipulations of series along with Assumption~\\ref{hyp:step-tcl}, it can be shown that\nUsing Lemma~\\ref{lemma:r_n} under Assumption~\\ref{hyp:step-tcl} (note that $\\gamma_0 > 1/2L \\geq 1/a$ when $\\kappa=1$),\nwe obtain that the sequence $|r_{n}-1|/\\gamma_n$ is bounded, thus $|r_{n+1}-1|\\leq C\\gamma_{n+1}$.\n%{\\color{red} voir si on a la place de le mettre dans un lemme.}\n The sequence $(1-\\alpha_{n})/\\gamma_n$ being also bounded, it holds that\n%$\\|m_{n+1}-m_n\\|\\leq C\\gamma_{n+1}\\|\\nabla f_{n+1}-m_n\\|$, and if $ \\|z_n-z^*\\|\\leq M$,\n\\begin{align*}\n  \\|m_{n+1}-m_n\\|^2 \\1_{\\{\\|z_n-z^*\\|\\leq M\\}} \\leq  C\\gamma_{n+1}^2(1+\\|\\nabla f_{n+1}\\|^2 )\\1_{\\{\\|z_n-z^*\\|\\leq M\\}}\\,.\n\\end{align*}\nBy Assumption~\\ref{hyp:moment-f}~\\ref{momentreinforce}, %Assumption~\\ref{hyp:mean_field_tcl},\n$\\bE_n(\\|\\nabla f_{n+1}\\|^2|)$ is bounded by a deterministic constant on $\\{\\|z_n-z^*\\|\\leq M\\}$.\nThus, $\\bE_n(\\|m_{n+1}-m_n\\|^2\\1_{\\{\\|z_n-z^*\\|\\leq M\\}})\\leq C\\gamma_{n+1}^2$.\nA similar result holds for $\\|v_{n+1}-v_n\\|^2$. We have thus shown that\n$ \\bE_n\\left(\\|\\epsilon_{n+1}^1\\|^2\\1_{\\{\\|z_n-z^*\\|\\leq M\\}}\\right)\\leq C\\gamma_{n+1}^2$. Hence,\nEq.~(\\ref{eq:cond-reste}) is proved for $\\epsilon_{n+1}^1$ in place of $\\epsilon_{n+1}$.\n%As far as $\\epsilon_{n+1}^2$, $\\epsilon_{n+1}^3$ are concerned, the proof uses the same kind of arguments\n%under Assumption~\\ref{hyp:step-tcl} and is omitted.\nUnder Assumption~\\ref{hyp:step-tcl}, the proof uses the same kind of arguments for $\\epsilon_{n+1}^2$, $\\epsilon_{n+1}^3$  and is omitted.\nFinally, Eq.~(\\ref{eq:cond-reste}) is proved.\nContinuing the verification of Assumption (A1.2), we establish that\n\\begin{equation}\n  \\label{eq:lim-cov}\n  \\bE_n(\\eta_{n+1}\\eta_{n+1}^T) \\to Q\\textrm{ a.s. on } \\{z_n\\to z^*\\}\\,.\n\\end{equation}\nDenote by $\\bar Q(x)$ the matrix given by the righthand side of Eq.~(\\ref{eq:Q}) when $x^*$\nis replaced by an arbitrary $x\\in \\cV$. It is easily checked that $\\bE_n(\\eta_{n+1}\\eta_{n+1}^T)=\\bar Q(x_n)$\nand by continuity, $\\bar Q(x_n)\\to Q$ a.s. on $\\{z_n\\to z^*\\}$, which proves (\\ref{eq:lim-cov}).\nTherefore, Assumption (A1.2) is fulfilled, except for the point mentioned at the beginning of this section : %paragraph:\n\\cite{pelletier1998weak} puts the additional condition that the limit\nmatrix in Eq.~(\\ref{eq:lim-cov}) is positive definite. This condition is not satisfied in our case,\nbut the proof can still be adapted. The specific part of the proof where the positive definiteness\ncomes into play is Th.~7 in \\cite{pelletier1998weak}. The proof of \\cite[Th.~1]{pelletier1998weak} can therefore\nbe adapted to the case of a positive semidefinite matrix.\n%by substituting the following\n%result to Th.~7 of the same paper.\nIn the proof of \\cite[Th.~7]{pelletier1998weak}, we only substitute the inverse of the square root of $Q$ by the Moore-Penrose inverse.\nFinally, the uniqueness of the stationary distribution $\\mu$ and its expression follow from \\cite[Th.~6.7, p. 357]{karatzas-shreve1991}.\n%{\\color{red} Citer Karatsas-Shreeve, 2d Ed., Th. 6.7.}\n% \\begin{theorem}[adapted from \\cite{pelletier1998weak}, Th.~7]\n% Let $k\\geq 1$. On some probability space equipped with a filtration $\\cF=(\\cF_n)_n$,\n% consider a sequence of r.v. on $\\bR^k$ given by\n% $$\n% Z_{n+1} = (I+\\gamma_{n+1} \\bar H)Z_n+\\gamma_{n+1} b_{n+1}+\\sqrt{\\gamma_{n+1}}\\eta_{n+1}\n% $$\n% and $\\bE(\\|Z_0\\|^2)<0$,\n% where $\\bar H$ is a $k\\times k$ stable matrix \\emph{i.e.}, the largest real part of its eigenvalues\n% is negative, $(b_n)$ and $(\\eta_n)$ are random sequences, $(\\gamma_n)$ satisfies Assumption~\\ref{hyp:step-tcl}.\n% Let $\\Omega_0\\in \\cF_\\infty$ have a positive probability. Assume that the following holds almost surely on $\\Omega_0$:\n% \\begin{enumerate}[{i)}, leftmargin=*]\n% \\item $\\bE_n(\\eta_{n+1})=0$.\n% \\item There exists a constant $\\bar b>2$ s.t. $\\sup_{n\\geq 0}\\bE_n(\\|\\eta_{n+1}\\|^{\\bar b})<\\infty$.\n% \\item $\\bE_n(\\eta_{n+1}\\eta_{n+1}^T) = C + \\Delta_n$ where $\\bE(\\|\\Delta_n\\|\\1_{\\Omega_0})\\to 0$\n% and $C$ is a positive semidefinite matrix.\n% \\item The sequence $(b_n)$ is the sum of two sequences $(b_{n,1})$ and $(b_{n,2})$, adapted to $\\cF$,\n% s.t. $\\sup_{n\\geq 0}\\bE(\\|b_{n,1}\\|^2)<\\infty$, $\\bE(\\|b_{n,1}\\|\\1_{\\Omega_0})\\to 0$ and $b_{n,2}\\to 0$ a.s. on $\\Omega_0$.\n% \\end{enumerate}\n% Then, given $\\Omega_0$, $(Z_n)$ converges in distribution to the unique stationary distribution $\\mu$ of the generalized Ornstein-Uhlenbeck process\n% $$\n% dX_t = \\bar H X_t dt + \\sqrt CdB_t\n% $$\n% where $(B_t)$ is the standard Brownian motion and $\\sqrt{C}$ is the unique positive semidefinite square root of $C$.\n% The distribution $\\mu$ is the zero mean Gaussian distribution with covariance matrix $\\bar \\Sigma$\n% given as the solution to $\\bar H\\bar \\Sigma+\\bar \\Sigma\\bar H^T = -C$.\n% \\end{theorem}\n% \\begin{proof}\n%   The proof is identical to the proof of \\cite[Th.~7]{pelletier1998weak}, only substituting the\n% inverse of the square root of $C$ by the Moore-Penrose inverse. Finally, the uniqueness of the\n% stationary distribution $\\mu$ and its expression follow from \\cite[Th.~6.7, p. 357]{karatzas-shreve1991}.%{\\color{red} Citer Karatsas-Shreeve, 2d Ed., Th. 6.7.}\n% \\end{proof}\n\n%\\subsection{Proof of Cor.~\\ref{cor:x}}\n\\noindent\\textbf{Proof of Eq.~(\\ref{eq:cov})}.\nWe introduce the $d\\times d$ blocks of the $3d\\times 3d$ matrix %$\\Sigma$ by\n$\\Sigma = \\left(\\Sigma_{i,j}\\right)_{i,j=1,2,3}$\nwhere $\\Sigma_{i,j}$ is $d\\times d$. We denote by $\\tilde\\Sigma$ the $2d\\times 2d$ submatrix $\\tilde\\Sigma \\eqdef\\left(\\Sigma_{i,j}\\right)_{i,j=1,2}$.\nBy Th.~\\ref{thm:clt}, we have the subsystem:\n\\begin{equation}\n\\tilde H \\tilde\\Sigma + \\tilde\\Sigma\\tilde H^T =\n\\begin{pmatrix}\n  0 & 0 \\\\\n0 & -a^2 \\tilde Q\n\\end{pmatrix}\\qquad\\text{where }\\tilde H \\eqdef\n\\begin{pmatrix}\n  \\zeta I_d & -D \\\\\na\\nabla^2F(x^*) & (\\zeta-a) I_d\n\\end{pmatrix}\\label{eq:lyap-reduced}\n\\end{equation}\nand where $\\tilde Q \\eqdef \\textrm{Cov}\\left(\\nabla\n  f(x^*,\\xi)\\right)$.  The next step is to triangularize the matrix\n$\\tilde H$ in order to decouple the blocks of $\\tilde\\Sigma$.  For\nevery $k=1,\\dots,d$, set\n$\\nu_k^\\pm \\eqdef -\\frac{a}{2}\\pm\\sqrt{a^2/4-a\\lambda_k}$\nwith the convention that $\\sqrt{-1} =\n\\imath$ (inspecting the characteristic polynomial of $\\tilde H$, these %quantities\nare the eigenvalues of $\\tilde H$). Set\n$M^\\pm\\eqdef\\diag{(\\nu_1^\\pm,\n\\cdots,\\nu_d^\\pm)}$ and $R^\\pm\\eqdef\nD^{-1/2}PM^\\pm P^TD^{-1/2}$.  Using the identities $M^++M^-=-a I_d$ and\n$M^+M^-=a\\Lambda$ where\n$\\Lambda\\eqdef\\diag{(\\lambda_1,\\cdots,\\lambda_d)}$, it can be checked\nthat\n$$\n\\cR\\tilde H =\\begin{pmatrix}\n  D R^+ + \\zeta I_d & -D \\\\ 0 & R^-D + \\zeta I_d\n\\end{pmatrix}\\cR,\\text{ where }\\cR\\eqdef\n\\begin{pmatrix}\n  I_d & 0 \\\\ R^+ & I_d\n\\end{pmatrix}\\,.\n$$\nSet $\\check \\Sigma \\eqdef \\cR\\tilde\\Sigma\\cR^T$. Denote by\n$(\\check \\Sigma_{i,j})_{i,j=1,2}$ the blocks of $\\check\\Sigma$.\nNote that $\\check\\Sigma_{1,1} = \\Sigma_{1,1}$. By left/right multiplication of Eq.~(\\ref{eq:lyap-reduced})\nrespectively with $\\cR$ and $\\cR^T$, we obtain\n%% $$\n%% \\cT\\check\\Sigma + \\check\\Sigma\\cT^T =\n%% \\begin{pmatrix}\n%%   0 & 0 \\\\\n%% 0 & -a^2 \\tilde Q\n%% \\end{pmatrix}\\,.\n%% $$\n%% By expanding the above system, we obtain\n\\begin{align}\n  &(DR^++\\zeta I_d) \\Sigma_{1,1}+\\Sigma_{1,1}(R^+D+\\zeta I_d) = \\check\\Sigma_{1,2} D+ D\\check\\Sigma_{1,2}^T \\label{eq:lapremiere}\\\\\n& (DR^++\\zeta I_d) \\check\\Sigma_{1,2}+\\check\\Sigma_{1,2} (DR^-+\\zeta I_d) = D\\check\\Sigma_{2,2} \\label{eq:ladeuxieme}\\\\\n& (R^-D+\\zeta I_d) \\check\\Sigma_{2,2} + \\check\\Sigma_{2,2} (DR^-+\\zeta I_d) = -a^2\\tilde Q \\label{eq:laderniere}\n\\end{align}\nSet $\\bar \\Sigma_{2,2} = P^{-1}D^{1/2}\\check\\Sigma_{2,2} D^{1/2}P$.\nDefine $C\\eqdef P^{-1}D^{1/2}\\tilde Q D^{1/2}P$.\nEq.~(\\ref{eq:laderniere}) yields $(M^-+\\zeta I_d) \\bar\\Sigma_{2,2} + \\bar\\Sigma_{2,2} (M^-+\\zeta I_d) = -a^2C$.\n%% The component $(k,\\ell)$ is given by\n%% $$\n%% \\bar\\Sigma_{2,2}^{k,\\ell} = -a^2\\frac{C_{k,\\ell}}{\\nu_k^-+\\nu_\\ell^-}\\,.\n%% $$\nSet $\\bar \\Sigma_{1,2} = P^{-1}D^{-1/2}\\check\\Sigma_{1,2} D^{1/2}P$.\nEq.~(\\ref{eq:ladeuxieme}) rewrites $(M^++\\zeta I_d) \\bar \\Sigma_{1,2}+\\bar \\Sigma_{1,2} (M^-+\\zeta I_d) = \\bar \\Sigma_{2,2}$.\nWe obtain that %The component $(k,\\ell)$ is given by\n$%$$\n\\bar\\Sigma_{1,2}^{k,\\ell} = (\\nu_k^++\\nu_\\ell^-+2\\zeta)^{-1}\\bar\\Sigma_{2,2}^{k,\\ell} = \\frac{-a^2C_{k,\\ell}}{(\\nu_k^++\\nu_\\ell^-+2\\zeta)(\\nu_k^-+\\nu_\\ell^-+2\\zeta)}\\,.\n$ %$$\nSet %finally\n$\\bar \\Sigma_{1,1} = P^{-1}D^{-1/2}\\Sigma_{1,1} D^{-1/2}P$.\nEq.~(\\ref{eq:lapremiere}) becomes $(M^++\\zeta I_d)\\bar \\Sigma_{1,1} + \\bar \\Sigma_{1,1}(M^++\\zeta I_d) = \\bar \\Sigma_{1,2} + \\bar \\Sigma_{1,2}^T$. Thus,\n%\\begin{equation*}\n%\\resizebox{.99\\hsize}{!}{$\n\\begin{align*}\n\\bar\\Sigma_{1,1}^{k,\\ell} &\n\\resizebox{.9\\hsize}{!}{$\n= \\frac{\\bar\\Sigma_{1,2}^{k,\\ell}+\\bar\\Sigma_{1,2}^{\\ell,k}}{\\nu_k^++\\nu_\\ell^++2\\zeta}\n=  \\frac{-a^2C_{k,\\ell}}{(\\nu_k^++\\nu_\\ell^++2\\zeta)(\\nu_k^-+\\nu_\\ell^-+2\\zeta)}\\left(\\frac{1}{\\nu_k^++\\nu_\\ell^-+2\\zeta}\n+\\frac{1}{\\nu_k^-+\\nu_\\ell^++2\\zeta}\\right)\n$}\n\\\\ &\n%\\resizebox{.6\\hsize}{!}{$= \\frac{C_{k,\\ell}}{ (1 - \\frac{2\\zeta}{a})(\\lambda_k+\\lambda_\\ell-2\\zeta + \\frac 2a \\zeta^2) +\\frac 1{2(a-2\\zeta)}(\\lambda_k-\\lambda_\\ell)^2}\\,.$}\n= \\frac{C_{k,\\ell}}{ (1 - \\frac{2\\zeta}{a})(\\lambda_k+\\lambda_\\ell-2\\zeta + \\frac 2a \\zeta^2) +\\frac 1{2(a-2\\zeta)}(\\lambda_k-\\lambda_\\ell)^2}\\,,\n%\\,\\,\\text{and  the result is proved.}\n\\end{align*}\nand  the result is proved.\n% \\noindent\\textbf{The events $\\{x_n\\to x^*\\}$ and $\\{z_n\\to z^*\\}$ coincide $\\bP$-a.e.}\n% Set $\\bar \\gamma_n = 1-\\alpha_{n}$. Define the event $\\Gamma_N\\eqdef\\{x_n\\to x^*\\text{ and }\\sup_n\\|x_n-x^*\\|\\leq N\\}$.\n% It is sufficient to show that for all $N>0$, $m_n\\to 0$ and $v_n\\to S(x^*)$ a.s. on $\\Gamma_N$.\n% Considering the iterates $v_n$, we have\n% \\begin{align*}\n% v_{n+1}\\1_{\\Gamma_N}  &= (1-\\bar\\gamma_{n+1})v_{n}\\1_{\\Gamma_N}  +\\bar\\gamma_{n+1} S(x^*)\\1_{\\Gamma_N}\n% + \\bar\\gamma_{n+1} \\tilde \\chi_{n+1} + \\bar\\gamma_{n+1} \\tilde \\varsigma_n^1 + \\bar\\gamma_{n+1} \\tilde \\varsigma_n^2 \\\\\n% \\text{where }\\quad&\n%   \\resizebox{.85\\hsize}{!}{$\n% \\tilde \\chi_{n+1} \\eqdef (\\nabla f_{n+1}^{\\odot\n%     2}-S(x_n))\\1_{\\Gamma_N} - \\bE_n((\\nabla f_{n+1}^{\\odot\n%     2}-S(x_n))\\1_{\\Gamma_N})\n%  + \\bE_n[(\\nabla f_{n+1}^{\\odot{2}} - S(x_n)) \\bE_n(\\1_{\\Gamma_N})]$}\\\\\n% & \\tilde \\varsigma_n^1\\eqdef (S(x_n)-S(x^*))\\1_{\\Gamma_N}\\\\\n% &\\tilde\\varsigma_n^2\\eqdef \\bE_n( (\\nabla f_{n+1}^{\\odot 2}-S(x_n))\\1_{\\|x_n-x^*\\|\\leq N}(\\1_{\\Gamma_N}-\\bE_n(\\1_{\\Gamma_N})) )\\,.\n% \\end{align*}\n% Clearly, $\\tilde \\varsigma_n^1\\to 0$ a.s. The same holds for $\\tilde\\varsigma_n^2$, using H\\\"older's inequality along\n% with Assumption~\\ref{hyp:moment-f}-\\ref{momentreinforce} for $p=2$\n% and the fact that $\\bE_n(\\1_{\\Gamma_N})$ converges a.s. to $\\1_{\\Gamma_N}$.\n% Moreover, $\\bE_n(\\|\\tilde\\chi_{n+1}\\|^2)\\leq C$ for some $C>0$, using again Assumption~\\ref{hyp:moment-f}-\\ref{momentreinforce}\n% and Jensen's inequality. As $\\sum_k\\gamma_k^2<\\infty$, this in turn implies that $\\lim_n\\sum_{k\\leq n}\\gamma_k\\chi_k$ exists.\n% Using the same kind of arguments, one can show that $(v_n)$ is a.s. bounded on $\\Gamma_N$.\n% Thus, as in the proof of Th.~\\ref{thm:as_conv_under_stab},\n% we have shown that, almost surely on $\\Gamma_N$, the piecewise interpolated process of the sequence $(v_n)$ is a bounded APT of\n% $\\dot v=S(x^*)-v$, hence showing that $v_n\\to S(x^*)$ a.s. on that set. The same arguments can be used to show that\n% $m_n\\to 0$ a.s. on $\\Gamma_N$, and the proof is concluded.\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n% \\subsection{Proof of Cor.~\\ref{cor:x}}\n%\n% We introduce the $d\\times d$ blocks of the $3d\\times 3d$ matrix %$\\Sigma$ by\n% $\\Sigma = \\left(\\Sigma_{i,j}\\right)_{i,j=1,2,3}$\n% where $\\Sigma_{i,j}$ is $d\\times d$. We denote by $\\tilde\\Sigma$ the $2d\\times 2d$ submatrix $\\tilde\\Sigma \\eqdef\\left(\\Sigma_{i,j}\\right)_{i,j=1,2}$.\n% By Th.~\\ref{thm:clt}, we have the subsystem:\n% \\begin{equation}\n% \\tilde H \\tilde\\Sigma + \\tilde\\Sigma\\tilde H^T =\n% \\begin{pmatrix}\n%   0 & 0 \\\\\n% 0 & -a^2 \\tilde Q\n% \\end{pmatrix}\\qquad\\text{where }\\tilde H \\eqdef\n% \\begin{pmatrix}\n%   0 & -D \\\\\n% a\\nabla^2F(x^*) & -a I_d\n% \\end{pmatrix}\\label{eq:lyap-reduced}\n% \\end{equation}\n% and where $\\tilde Q \\eqdef \\textrm{Cov}\\left(\\nabla\n%   f(x^*,\\xi)\\right)$.  The next step is to triangularize the matrix\n% $\\tilde H$ in order to decouple the blocks of $\\tilde\\Sigma$.  For\n% every $k=1,\\dots,d$, set $\\nu_k^\\pm \\eqdef -\\frac\n% a2\\pm\\sqrt{a^2/4-a\\lambda_k}$ with the convention that $\\sqrt{-1} =\n% \\imath$ (inspecting the characteristic polynomial of $\\tilde H$, these\n% quantities are the eigenvalues of $\\tilde H$). Set\n% $M^\\pm\\eqdef\\diag{(\\nu_1^\\pm,\\cdots,\\nu_d^\\pm)}$ and $R^\\pm\\eqdef\n% D^{-1/2}PM^\\pm P^TD^{-1/2}$.  Using the identities $M^++M^-=-aI_d$ and\n% $M^+M^-=a\\Lambda$ where\n% $\\Lambda\\eqdef\\diag{(\\lambda_1,\\cdots,\\lambda_d)}$, it can be checked\n% that\n% $$\n% \\cR\\tilde H =\\begin{pmatrix}\n%   D R^+ & -D \\\\ 0 & R^-D\n% \\end{pmatrix}\\cR,\\text{ where }\\cR\\eqdef\n% \\begin{pmatrix}\n%   I_d & 0 \\\\ R^+ & I_d\n% \\end{pmatrix}\\,.\n% $$\n% Set $\\check \\Sigma \\eqdef \\cR\\tilde\\Sigma\\cR^T$. Denote by\n% $(\\check \\Sigma_{i,j})_{i,j=1,2}$ the blocks of $\\check\\Sigma$.\n% Note that $\\check\\Sigma_{1,1} = \\Sigma_{1,1}$. By left/right multiplication of Eq.~(\\ref{eq:lyap-reduced})\n% respectively with $\\cR$ and $\\cR^T$, we obtain\n% %% $$\n% %% \\cT\\check\\Sigma + \\check\\Sigma\\cT^T =\n% %% \\begin{pmatrix}\n% %%   0 & 0 \\\\\n% %% 0 & -a^2 \\tilde Q\n% %% \\end{pmatrix}\\,.\n% %% $$\n% %% By expanding the above system, we obtain\n% \\begin{align}\n%   &DR^+\\Sigma_{1,1}+\\Sigma_{1,1}R^+D = \\check\\Sigma_{1,2} D+ D\\check\\Sigma_{1,2}^T \\label{eq:lapremiere}\\\\\n% & DR^+\\check\\Sigma_{1,2}+\\check\\Sigma_{1,2}DR^- = D\\check\\Sigma_{2,2} \\label{eq:ladeuxieme}\\\\\n% & R^-D\\check\\Sigma_{2,2} + \\check\\Sigma_{2,2} DR^- = -a^2\\tilde Q \\label{eq:laderniere}\n% \\end{align}\n% Set $\\bar \\Sigma_{2,2} = P^{-1}D^{1/2}\\check\\Sigma_{2,2} D^{1/2}P$.\n% Define $C\\eqdef P^{-1}D^{1/2}\\tilde Q D^{1/2}P$.\n% Eq.~(\\ref{eq:laderniere}) yields $M^-\\bar\\Sigma_{2,2} + \\bar\\Sigma_{2,2} M^- = -a^2C$.\n% %% The component $(k,\\ell)$ is given by\n% %% $$\n% %% \\bar\\Sigma_{2,2}^{k,\\ell} = -a^2\\frac{C_{k,\\ell}}{\\nu_k^-+\\nu_\\ell^-}\\,.\n% %% $$\n% Set $\\bar \\Sigma_{1,2} = P^{-1}D^{-1/2}\\check\\Sigma_{1,2} D^{1/2}P$.\n% Eq.~(\\ref{eq:ladeuxieme}) rewrites $M^+\\bar \\Sigma_{1,2}+\\bar \\Sigma_{1,2}M^- = \\bar \\Sigma_{2,2}$.\n% The component $(k,\\ell)$ is given by\n% $$\n% \\bar\\Sigma_{1,2}^{k,\\ell} = (\\nu_k^++\\nu_\\ell^-)^{-1}\\bar\\Sigma_{2,2}^{k,\\ell} = \\frac{-a^2C_{k,\\ell}}{(\\nu_k^++\\nu_\\ell^-)(\\nu_k^-+\\nu_\\ell^-)}\\,.\n% $$\n% Set finally $\\bar \\Sigma_{1,1} = P^{-1}D^{-1/2}\\Sigma_{1,1} D^{-1/2}P$.\n% Eq.~(\\ref{eq:lapremiere}) becomes $M^+\\bar \\Sigma_{1,1} + \\bar \\Sigma_{1,1}M^+ = \\bar \\Sigma_{1,2} + \\bar \\Sigma_{1,2}^T$. Thus,\n% \\begin{equation*}\n% \\resizebox{.99\\hsize}{!}{$\n% \\bar\\Sigma_{1,1}^{k,\\ell} = \\frac{\\bar\\Sigma_{1,2}^{k,\\ell}+\\bar\\Sigma_{1,2}^{\\ell,k}}{\\nu_k^++\\nu_\\ell^+}\n% =  \\frac{-a^2C_{k,\\ell}}{(\\nu_k^++\\nu_\\ell^+)(\\nu_k^-+\\nu_\\ell^-)}\\left(\\frac{1}{\\nu_k^++\\nu_\\ell^-}\n% +\\frac{1}{\\nu_k^-+\\nu_\\ell^+}\\right)\n% = \\frac{C_{k,\\ell}}{\\lambda_k+\\lambda_\\ell+\\frac 1{2a}(\\lambda_k-\\lambda_\\ell)^2}\\,,\n% $}\n% \\end{equation*}\n%% \\begin{align*}\n%%  \\bar\\Sigma_{1,1}^{k,\\ell} &= \\frac{\\bar\\Sigma_{1,2}^{k,\\ell}+\\bar\\Sigma_{1,2}^{\\ell,k}}{\\nu_k^++\\nu_\\ell^+}\n%%   =  \\frac{-a^2C_{k,\\ell}}{(\\nu_k^++\\nu_\\ell^+)(\\nu_k^-+\\nu_\\ell^-)}\\left(\\frac{1}{\\nu_k^++\\nu_\\ell^-}\n%%+\\frac{1}{\\nu_k^-+\\nu_\\ell^+}\\right)\n%% \\\\ &= \\frac{C_{k,\\ell}}{\\lambda_k+\\lambda_\\ell+\\frac 1{2a}(\\lambda_k-\\lambda_\\ell)^2}\\,,\n%% \\end{align*}\n% and  the result is proved.\n\n","sha256_utf8":"58ec68fdff3adf63a0c58c5afbe68902129085dcf74a17d465ea302bcf65533a"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"随机目标与状态空间","source_units":[{"start":10012,"end":10620,"start_line":227,"end_line":236,"raw_tex":"Let $(\\Omega,\\cF,\\bP)$ be a probability space, and let\n$(\\Xi,\\mathfrak{S})$ denote an other measurable space.  Consider a\nmeasurable map $f:\\bR^d\\times \\Xi\\to \\bR$, where $d$ is an integer.\nFor a fixed value of $\\xi$, the mapping $x\\mapsto f(x,\\xi)$ is\nsupposed to be differentiable, and its gradient w.r.t. $x$ is denoted\nby $\\nabla f(x,\\xi)$.  Define $\\cZ\\eqdef \\bR^d\\times\\bR^d\\times\n\\bR^d$, $\\cZ_{+}\\eqdef \\bR^d\\times\\bR^d\\times [0,+\\infty)^d$ and\n$\\cZ_{+}^*\\eqdef \\bR^d\\times\\bR^d\\times (0,+\\infty)^d$. \\adam\\\ngenerates a sequence $z_n\\eqdef (x_n,m_n,v_n)$ on\n$\\cZ_+$ given by Algorithm~\\ref{alg:adam}.\n","sha256_utf8":"d6729b07f5947a6b30ab579544621976f6d059eec12dcbae5886c0850d4ab49a"}]},{"id":"S004","semantic_type":"notation","kind":"notation","title":"逐坐标运算与矩阵记号","source_units":[{"start":8996,"end":9922,"start_line":210,"end_line":220,"raw_tex":"\\noindent{\\bf Notations}. If $x$, $y$ are two vectors on $\\bR^d$ for some $d\\geq 1$, we denote by $x \\odot y$, $x^{\\odot 2}$, $x/y$, $|x|$, $\\sqrt{|x|}$ the vectors\non $\\bR^d$ whose $i$-th coordinates are respectively given by $x_iy_i$, $x_i^2$, $x_i/y_i$, $|x_i|$, $\\sqrt{|x_i|}$.\nInequalities of the form $x\\leq y$ are read componentwise.\nDenote by $\\|\\cdot\\|$ the standard Euclidean norm.\nFor any vector $v\\in (0,+\\infty)^d$, write $\\|x\\|^2_v = \\sum_i v_i x_i^2$.\nNotation $A^T$ represents the transpose of a matrix $A$.\nIf  $z\\in \\bR^d$ and $A$ is a non-empty subset of $\\bR^d$,\nwe use the notation $\\mathsf d(z,A) \\eqdef \\inf\\{ \\|z-z'\\| :z'\\in A\\}$.\nIf $A$ is a set, we denote  by $\\1_A$ the function equal to one on that set and to zero elsewhere.\n We denote by $C([0,+\\infty),\\bR^d)$ the space of continuous functions from $[0,+\\infty)$ to $\\bR^d$ endowed with the topology of\nuniform convergence on compact intervals.\n","sha256_utf8":"b0e4fcd190d5001a636e4a43506c02880a9f0c96123895370d947f1d5221eb53"}]},{"id":"S005","semantic_type":"assumption","kind":"assumption","title":"随机目标的正则性","source_units":[{"start":12850,"end":13482,"start_line":284,"end_line":293,"raw_tex":"\\begin{assumption} \\label{hyp:model}\nThe mapping $f:\\bR^d\\times\\Xi\\to \\bR$ satisfies the following.\n  \\begin{enumerate}[{\\sl i)}]\n  \\item For every $x\\in \\bR^d$, $f(x,\\,.\\,)$ is $\\mathfrak{S}$-measurable.\n  \\item For almost every $\\xi$, the map $f(\\,.\\,,\\xi)$ is continuously differentiable.  \\label{hyp:dif}\n  \\item There exists $x_*\\in \\bR^d$ s.t. $\\bE(|f(x_*,\\xi)|)<\\infty$ and  $\\bE(\\|\\nabla f(x_*,\\xi)\\|^2)<\\infty$.\n  \\item For every compact subset $K\\subset \\bR^d$, there exists $L_K>~0$ such that\n   for every $(x,y)\\in K^2$, $\\bE(\\|\\nabla f(x,\\xi)-\\nabla f(y,\\xi)\\|^2)\\leq L_K^2\\|x-y\\|^2$.\n  \\end{enumerate}\n\\end{assumption}","sha256_utf8":"12ffff4ba701afd15b6a62175b6ca7470fc3ab63d210206e5d1f3fc14ec4a011"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"均值目标与梯度二阶矩","source_units":[{"start":13616,"end":13759,"start_line":297,"end_line":299,"raw_tex":"\\begin{equation}\n  F(x) \\eqdef \\bE(f(x,\\xi)) \\quad \\text{and} \\quad S(x) \\eqdef \\bE(\\nabla f(x,\\xi)^{\\odot 2})\\label{eq:F_and_S}\n\\end{equation}","sha256_utf8":"c5b7d9b7b07b7a1b0d9b4bc589d9243737307a9218ef020c9167d9c5b83af6fa"}]},{"id":"S007","semantic_type":"derived_fact","kind":"derived_fact","title":"期望梯度与局部正则性","source_units":[{"start":13896,"end":14111,"start_line":304,"end_line":306,"raw_tex":"are well defined; $F$ is continuously differentiable and by\nLebesgue's dominated convergence theorem, $\\nabla F(x) = \\bE(\\nabla\nf(x,\\xi))$ for all $x$. Moreover, $\\nabla F$ and $S$ are locally Lipschitz continuous.\n","sha256_utf8":"d28b2d63f916afa785e6bd9336e320ab993bda6282ae8ec0717f4df1c0707776"}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","title":"二阶矩逐坐标为正","source_units":[{"start":14413,"end":14500,"start_line":313,"end_line":316,"raw_tex":"\\begin{assumption}\n  \\label{hyp:S>0}\nFor every $x\\in \\bR^d$, $S(x)>0$.\n\\end{assumption}","sha256_utf8":"593b4e4b655d1693fa4ee8e77ad5b3bcb9a57edbb6609bcf9a7d52cd0c69a1f4"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"临界点集合","source_units":[{"start":14568,"end":14635,"start_line":318,"end_line":318,"raw_tex":"$F$, denoted by $$\\cS \\eqdef \\nabla F^{-1}(\\{0\\}),$$ is non-empty.\n","sha256_utf8":"c283b236ca3421d955ca8790e50a322f25df7ba4ab80e3b9797dddd52c5f9a92"}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","title":"递减步长与偏差校正算法","source_units":[{"start":33930,"end":34781,"start_line":669,"end_line":686,"raw_tex":"\\begin{algorithm}[tb]\n   \\caption{\\bf \\adam - decreasing stepsize $(((\\gamma_n,\\alpha_n,\\beta_n):n\\in \\bN^*), \\varepsilon)$.}\n   \\label{alg:adam-decreasing}\n\\begin{algorithmic}\n  \\STATE {\\bfseries Initialization:} $x_0\\in \\bR^d, m_0=0$, $v_0=0$, $r_0=\\bar r_0=0$.\n   \\FOR{$n=1$ {\\bfseries to} $n_{\\text{iter}}$}\n   \\STATE $m_n = \\alpha_n m_{n-1} + (1-\\alpha_n)  \\nabla f(x_{n-1},\\xi_n)$\n   \\STATE $v_n = \\beta_n v_{n-1} + (1-\\beta_n) \\nabla f(x_{n-1},\\xi_n)^{\\odot 2}$\n   \\STATE $r_n = \\alpha_n r_{n-1} + (1-\\alpha_n)$\n   \\STATE $\\bar r_n = \\beta_n \\bar r_{n-1} + (1-\\beta_n)$\n  \\STATE $\\hat m_{n} = m_{n}/r_n$\n   \\COMMENT{bias correction step}\n   \\STATE $\\hat v_{n} = v_{n}/\\bar r_n$\n   \\COMMENT{bias correction step}\n   \\STATE $x_{n} = x_{n-1} - \\gamma_n \\hat m_{n} / (\\varepsilon+\\sqrt{\\hat v_{n}}) \\,.$\n   \\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"f51d66a5cd3ff2cbbf41dbc52529d524503b679ce9af494ac06a8b05e0e6a62c"}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","title":"动量权重与极限参数","source_units":[{"start":36277,"end":36501,"start_line":709,"end_line":711,"raw_tex":"  \\item For all $n \\in \\bN$, $0 \\leq \\alpha_n \\leq 1$ and $0 \\leq \\beta_n \\leq 1$,\n  \\item There exist $a,b$ s.t. $0<b<4a$,\\,\n  $\\gamma_n^{-1}(1-\\alpha_n) \\to a$ and $\\gamma_n^{-1}(1-\\beta_n) \\to b$\\,. %as $n \\to \\infty$\\,.\n","sha256_utf8":"0a2575ec5c1f0174dfb776d7e64d9ab47214b8159a140d5017168978dbebe3ae"}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","title":"局部梯度矩条件","source_units":[{"start":28897,"end":29316,"start_line":576,"end_line":583,"raw_tex":"\\begin{assumption}\n  \\label{hyp:moment-f}\nLet $p>0$. Assume either one of the following conditions.\n\\begin{enumerate}[i)]\n\\item \\label{momentegal} For every compact set $K\\subset \\bR^d$, $\\sup_{x\\in K} \\bE(\\|\\nabla f(x,\\xi)\\|^{p})<\\infty\\,.$\n\\item \\label{momentreinforce} For every compact set $K\\subset \\bR^d$,  $\\exists\\, p_K>p$, $\\sup_{x\\in K} \\bE(\\|\\nabla f(x,\\xi)\\|^{p_K})<\\infty.$\n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"ac4c912ac33021c99c07c89173a6e88ff59a25244b527e902cfce17932a88170"}]},{"id":"S013","semantic_type":"assumption","kind":"assumption","title":"独立同分布采样","source_units":[{"start":28766,"end":28895,"start_line":571,"end_line":574,"raw_tex":"\\begin{assumption}\n  \\label{hyp:iid}\nThe sequence $(\\xi_n:n\\geq 1)$ is iid, with the same distribution as~$\\xi$.\n\\end{assumption}","sha256_utf8":"b94da2b7ed97ac786c6ba4adc298ea6a68d2f3cb748be5f4212fdcf5ebbe3599"}]},{"id":"S014","semantic_type":"assumption","kind":"assumption","title":"极限临界点附近的光滑性","source_units":[{"start":40005,"end":40414,"start_line":784,"end_line":793,"raw_tex":"\\begin{assumption} %[local regularity of the mean field $h_\\infty$]\n   \\label{hyp:mean_field_tcl}\nLet $x^*\\in \\cS$. There exists a neighborhood $\\cV$ of $x^*$ s.t.\n\\begin{enumerate}[{\\sl i)}]\n\\item $F$ is twice continuously differentiable on $\\cV$,\n  and the Hessian $\\nabla^2 F(x^*)$ of $F$ at $x^*$ is positive\n  definite.\n\\item $S$ is continuously differentiable on $\\cV$.\n\\end{enumerate}\n \\end{assumption}","sha256_utf8":"d443a5522e05e90ca6da7364488b48ca47d019092431f71373dadbe4762a347f"}]},{"id":"S015","semantic_type":"notation","kind":"notation","title":"预条件矩阵与谱分解","source_units":[{"start":40415,"end":40818,"start_line":794,"end_line":802,"raw_tex":"Define $\nD \\eqdef \\textrm{diag}\\left((\\varepsilon + \\sqrt{S_1(x^*)})^{-1},\\cdots,(\\varepsilon + \\sqrt{S_d(x^*)})^{-1}\\right)\\,.\n$\nLet $P$ be an orthogonal matrix s.t. the following spectral decomposition holds:\n$$\nD^{1/2}\\nabla^2F(x^*)D^{1/2} = P\\textrm{diag}(\\lambda_1,\\cdots,\\lambda_d)P^{-1}\\,,\n$$\nwhere $\\lambda_1, \\cdots,\\lambda_d$ are the (positive) eigenvalues of\n$D^{1/2} \\nabla^2F(x^*)D^{1/2}$.\n","sha256_utf8":"978fb5c2b67c06bcc6abbf4e094927e8696d864b5d771402cbee8e5b776a00e9"}]},{"id":"S016","semantic_type":"definition","kind":"definition","title":"线性化矩阵","source_units":[{"start":40818,"end":40983,"start_line":803,"end_line":809,"raw_tex":"Define\n\\begin{equation}\nH \\eqdef\n\\begin{pmatrix}\n  0 & -D & 0\\\\ a\\nabla^2F(x^*) & -a I_d & 0 \\\\ b \\nabla S(x^*) & 0 & -b I_d\n\\end{pmatrix}\\label{eq:H}\n\\end{equation}","sha256_utf8":"927a84355713ad09922dbdd8e0b6068b5e6e1579cb6007d6748bfd2ad8cc8def"},{"start":40984,"end":41099,"start_line":810,"end_line":811,"raw_tex":"where $I_d$ represents the $d\\times d$ identity matrix and $\\nabla S(x^*)$ is the\nJacobian matrix of~$S$ at $x^*$.\n","sha256_utf8":"02c449d0d1d69280d4927a7e0b04461c68a2489557898cdfc3103f1d9d1c8e83"}]},{"id":"S017","semantic_type":"derived_fact","kind":"derived_fact","title":"线性化的稳定谱界","source_units":[{"start":41099,"end":41310,"start_line":812,"end_line":815,"raw_tex":"The largest real part of the eigenvalues of $H$ coincides with $-L$, where\n\\begin{equation}\nL\\eqdef b\\wedge \\frac a2\\left( 1-\\sqrt{\\left(1-\\frac{4\\lambda_1}a\\right)\\vee 0}\\right) >0\\,.\\label{eq:L}\n\\end{equation}","sha256_utf8":"564f79596a0594787bfe41512b350369cb41fb3932f89e9775f2d28147dfae5d"}]},{"id":"S018","semantic_type":"definition","kind":"definition","title":"极限噪声协方差","source_units":[{"start":41311,"end":41971,"start_line":816,"end_line":837,"raw_tex":"Finally, define the $3d\\times 3d$ matrix% covariance matrix of the\n% vector $(a\\nabla f(x^*,\\xi),b\\nabla f(x^*,\\xi)^{\\odot 2})$ \\emph{i.e.},\n\\begin{equation}\n  \\resizebox{.9\\hsize}{!}{$\nQ \\eqdef\n\\left( \\begin{array}[h]{cc}\n   0 &\n   \\begin{array}[h]{cc}\n     0 & \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ 0\n   \\end{array}\\\\\n   \\begin{array}[h]{c}\n     0 \\\\ 0\n   \\end{array}\n & \\bE\\left[\n  \\begin{pmatrix}\n    a\\nabla f(x^*,\\xi) \\\\ b(\\nabla f(x^*,\\xi)^{\\odot 2}-S(x^*))\n  \\end{pmatrix}\\begin{pmatrix}\n    a\\nabla f(x^*,\\xi) \\\\ b(\\nabla f(x^*,\\xi)^{\\odot 2}-S(x^*))\n  \\end{pmatrix}^T\\right]\n \\end{array}\\right)\\,.\\label{eq:Q}\n $}\n\\end{equation}","sha256_utf8":"d5d6547594afb666aa7dbdbbb69f23bb61e9290181499bdc3ae97954ccc55f62"}]},{"id":"S019","semantic_type":"assumption","kind":"assumption","title":"中心极限定理的步长条件","source_units":[{"start":41972,"end":42580,"start_line":838,"end_line":849,"raw_tex":"\\begin{assumption}\n\\label{hyp:step-tcl}\nThe following holds.\n\\begin{enumerate}[{\\sl i)}]\n\\item \\label{step-tcl-i} There exist $\\kappa \\in (0,1]$, $\\gamma_0>0$, s.t. the sequence $(\\gamma_n)$\nsatisfies\n$\\gamma_n = {\\gamma_0}/{(n+1)^\\kappa}$ for all $n$.\nIf $\\kappa = 1$, we assume moreover that $\\gamma_0 > \\frac{1}{2L}$.\n\\item The sequences $\\left(\\frac{1}{\\gamma_n}(\\frac{1-\\alpha_n}{\\gamma_n}-a)\\right)$ and $\\left(\\frac{1}{\\gamma_n}(\\frac{1-\\beta_n}{\\gamma_n}-b)\\right)$ are bounded.\n%This is satisfied for instance if $1-\\alpha_n = a\\gamma_n$ and $1-\\beta_n = b\\gamma_n$.\n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"68ab0ca04bf29a39add5fd967748c8511ec0cb07b7a1398aa5579753b169c249"}]},{"id":"S020","semantic_type":"definition","kind":"definition","title":"给定事件的分布收敛","source_units":[{"start":42582,"end":42904,"start_line":851,"end_line":853,"raw_tex":"\\noindent For an arbitrary sequence $(X_n)$ of random variables on some Euclidean space, a probability measure\n$\\mu$ on that space and an event $\\Gamma$ s.t. $\\bP(\\Gamma)>0$, we say that $X_n$ converges in distribution\nto $\\mu$ \\emph{given $\\Gamma$} if the measures $\\bP (X_n\\in \\cdot\\,|\\Gamma)$ converge weakly to $\\mu$.\n","sha256_utf8":"5a7d565b7ca640424e3d0e7e7b50741a480cc8fd72bd0cefddab90fa721b0351"}]},{"id":"S021","semantic_type":"definition","kind":"definition","title":"自治平均场","source_units":[{"start":64499,"end":64657,"start_line":1168,"end_line":1171,"raw_tex":"\\begin{equation}\n  \\label{eq:h_infty}\nh_{\\infty}((x,m,v)) = \\left(-m / (\\varepsilon+\\sqrt{v})\\,,\\, a (\\nabla F(x)-m) \\,,\\,b (S(x)-v) \\right)\\,.\n\\end{equation}","sha256_utf8":"1aa54b09ee7cc6beacf4fa985bafbdb804982353b508a8d54f1c32632345a237"}]},{"id":"S022","semantic_type":"notation","kind":"notation","title":"观测滤过","source_units":[{"start":135732,"end":135826,"start_line":2394,"end_line":2394,"raw_tex":"We recall that $\\cF_n$ is the $\\sigma$-algebra generated by the r.v. $(\\xi_k:1\\leq k\\leq n)$.\n","sha256_utf8":"4876543fff08fa04932fc716deb368fd38391531f4a12c083fd67982bba5d4d5"}]},{"id":"S023","semantic_type":"notation","kind":"notation","title":"条件期望与随机梯度简写","source_units":[{"start":156491,"end":156693,"start_line":2688,"end_line":2689,"raw_tex":"In this section, we denote by $\\bE_n = \\bE(\\cdot|\\cF_n)$ the conditional expectation w.r.t. $\\cF_n$.\nWe also use the notation $\\nabla f_{n+1} \\eqdef \\nabla f(x_n,\\xi_{n+1})$. %to simplify the equation.\n","sha256_utf8":"afd49bd2f5ee718766825927de08ba423ddfb616d8b9286aee18d2ce3e6855fc"}]},{"id":"S024","semantic_type":"derived_fact","kind":"derived_fact","title":"二阶矩校正系数的对应结论","source_units":[{"start":157927,"end":157981,"start_line":2716,"end_line":2716,"raw_tex":"A similar lemma holds for the sequence $(\\bar{r}_n)$.\n","sha256_utf8":"7c41487f57c00b18d641521faec62352e7f6c0507c4a8924ef42df781df3bb63"}]},{"id":"S025","semantic_type":"lemma","kind":"lemma","title":"偏差校正系数及其收敛速度","source_units":[{"start":156744,"end":157926,"start_line":2692,"end_line":2715,"raw_tex":"\\begin{lemma}\n\\label{lemma:r_n}\nLet the sequence $(r_n)$ be defined as in Algorithm~\\ref{alg:adam-decreasing}.\nAssume that $0 \\leq \\alpha_n \\leq 1$ for all $n$ and that $(1-\\alpha_n)/\\gamma_n \\to a > 0$ as $n \\to +\\infty$.\nThen,\n\\begin{enumerate}[{\\sl i)}]\n\\item $\\forall n \\in \\bN, r_n = 1 - \\prod_{i=1}^n \\alpha_i$,\n\\item The sequence $(r_n)$ is nondecreasing and converges to $1$.\n\\item Under Assumption~\\ref{hyp:step-tcl}~\\ref{step-tcl-i}, for every $\\epsilon>0$, for sufficiently large $n$, we have\n$r_n-1 \\leq e^{-\\frac{a\\gamma_0}{2(1-\\kappa)}n^{1-\\kappa}}$ if $\\kappa \\in (0,1)$ and\n$r_n-1 \\leq n^{-a \\gamma_0/(1+\\epsilon)}$ if $\\kappa = 1$.\n% $\n%      r_n-1 \\leq\n%      \\begin{cases}\n%         e^{-\\frac{a\\gamma_0}{2(1-\\kappa)}n^{1-\\kappa}} \\quad \\text{if}\\,\\,\\kappa \\in (0,1)\\\\\n%         n^{-a \\gamma_0/(1+\\epsilon)} \\quad \\text{if}\\,\\,\\kappa=1 \\,.\n%      \\end{cases}\n% $\n%\\item If $\\gamma_n = \\gamma_0 n^{-\\alpha}$ with $\\alpha \\in (0,1)$, then\n%for sufficiently large $n$, $1-r_n \\leq e^{-\\frac{a\\gamma_0}{2(1-\\alpha)}n^{1-\\alpha}}$\\,.\n%\\item If $\\gamma_n = \\gamma_0 n^{-1}$, then for sufficiently large $n$,\n%$1-r_n \\leq n^{-a \\gamma_0/2}$\\,.\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"45fd801714445cc2e4107a403d160ddc4fd3968349d67b63637d411f235fe557"}]},{"id":"S026","semantic_type":"proof","kind":"proof","title":"校正系数引理的完整证明","source_units":[{"start":157981,"end":158699,"start_line":2717,"end_line":2729,"raw_tex":"\\begin{proof}\ni) stems from observing that $r_{n+1} -1 = \\alpha_{n+1}(r_n -1)$ for every $n \\in \\bN$ and\niterating this relation ($r_0=0$). As a consequence, the sequence $(r_n)$ is nondecreasing.\n%We now show iii) and iv).\nWe can write :\n$\n0 \\leq 1-r_n \\leq \\exp(- \\sum_{i=1}^n (1-\\alpha_i))\\,.\n$\niii) As $\\sum_{n\\geq 1} \\gamma_n = + \\infty$ %for $\\gamma_n = \\gamma_0 n^{-\\kappa}$ with $\\kappa \\in [0,1]$\nand $(1-\\alpha_n) \\sim a\\gamma_n$, we deduce that $\\sum_{i=1}^n (1-\\alpha_i) \\sim \\sum_{i=1}^n a\\gamma_i$.\nThe results follow from the fact that $\\sum_{i=1}^n \\gamma_i \\sim \\frac{\\gamma_0}{1-\\kappa} n^{1-\\kappa}$\nwhen $\\kappa \\in (0,1)$ and $\\sum_{i=1}^n \\gamma_i \\sim \\gamma_0 \\ln n$ for $\\kappa=1$.\n\\end{proof}","sha256_utf8":"9a955699a28923acc58c8a3ba098c4ec7789578e043fa34f17e95b59659bc21f"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"递推乘积与指数上界","source_units":[{"start":157995,"end":158279,"start_line":2718,"end_line":2724,"raw_tex":"i) stems from observing that $r_{n+1} -1 = \\alpha_{n+1}(r_n -1)$ for every $n \\in \\bN$ and\niterating this relation ($r_0=0$). As a consequence, the sequence $(r_n)$ is nondecreasing.\n%We now show iii) and iv).\nWe can write :\n$\n0 \\leq 1-r_n \\leq \\exp(- \\sum_{i=1}^n (1-\\alpha_i))\\,.\n$\n","sha256_utf8":"fe91aaa157b3c86954da4133606a7ba1ef567cda5696ec984daab8bd233b3ab3"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"步长和给出校正衰减率","source_units":[{"start":158279,"end":158688,"start_line":2725,"end_line":2728,"raw_tex":"iii) As $\\sum_{n\\geq 1} \\gamma_n = + \\infty$ %for $\\gamma_n = \\gamma_0 n^{-\\kappa}$ with $\\kappa \\in [0,1]$\nand $(1-\\alpha_n) \\sim a\\gamma_n$, we deduce that $\\sum_{i=1}^n (1-\\alpha_i) \\sim \\sum_{i=1}^n a\\gamma_i$.\nThe results follow from the fact that $\\sum_{i=1}^n \\gamma_i \\sim \\frac{\\gamma_0}{1-\\kappa} n^{1-\\kappa}$\nwhen $\\kappa \\in (0,1)$ and $\\sum_{i=1}^n \\gamma_i \\sim \\gamma_0 \\ln n$ for $\\kappa=1$.\n","sha256_utf8":"a5c4842eb358b9b0fdc7eed6d56a0bec9bceaaa5579926fd493598968a9a765d"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"引用极限定理并定位退化协方差问题","source_units":[{"start":172501,"end":173495,"start_line":2950,"end_line":2965,"raw_tex":"\n\\label{sec:clt}\n\nWe use~\\cite[Th.~1]{pelletier1998weak}.\nAll the assumptions in the latter can be verified in our case, at the exception of\na positive definiteness condition on the\nlimiting covariance matrix, which corresponds, in our case, to the matrix $Q$\ngiven by Eq.~(\\ref{eq:Q}). As $Q$ is not positive definite, it is strictly speaking not possible\nto just cite and apply \\cite[Th.~1]{pelletier1998weak}. Nevertheless,\na detailed inspection of the proofs of \\cite{pelletier1998weak} shows that only a minor\nadaptation is needed in order to cover the present case.\n%Therefore, it is worthless reprove the convergence result of \\cite{pelletier1998weak} from scratch.\nTherefore, proving the convergence result of \\cite{pelletier1998weak} from scratch is worthless.\nIt is sufficient to verify the assumptions of \\cite[Th.~1]{pelletier1998weak}\n(except the definiteness of $Q$) and then to point out the specific part of the proof of \\cite{pelletier1998weak}\nwhich requires some adaptation.\n","sha256_utf8":"f638e3656c1572e41ed9b2db1ccb8f6b4fddc41ea1246bbe8571ddc577da2536"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"分解平均场、鞅噪声与余项","source_units":[{"start":174397,"end":176674,"start_line":2982,"end_line":3014,"raw_tex":"Let $z_n=(x_n,m_n,v_n)$ be the output of Algorithm~\\ref{alg:adam-decreasing}.\nDefine $z^*=(x^*,0,S(x^*))$.\n% Remark that the conditional central limit theorem stated by \\cite{pelletier1998weak} shall be valid\n% only given the event $\\{z_n\\to z^*\\}$, whereas we seek to prove the convergence result stated in Th.~\\ref{thm:clt}\n% given the event $\\{x_n\\to x^*\\}$. As a matter of fact, both events coincide up to a $\\bP$-negligible set,\n% so it is safe to prove the convergence on the event $\\{z_n\\to z^*\\}$.\n% The claim that the two events coincide trivially holds if one lets the assumptions of Th.~\\ref{thm:as_conv_under_stab} be satisfied,\n% because in that case, $m_n\\to 0$ and $v_n- S(x_n)\\to 0$ a.s., so that $z_n\\to z^*$ a.s. on the event $\\{x_n\\to x^*\\}$.\n% But the claim is also satisfied if only the assumptions of Th.~\\ref{thm:clt} are considered.\n% As this point is somewhat technical and maybe non essential in a first reading, we postpone the discussion\n% to the end of this section.\nDefine $\\eta_{n+1} \\eqdef (0, a(\\nabla f_{n+1} - \\nabla F(x_n)), b(\\nabla f_{n+1}^{\\odot 2}- S(x_n)))$.\nWe have\n\\begin{equation}\n  \\label{eq:zn_dec}\n  z_{n+1} = z_n + \\gamma_{n+1} h_\\infty(z_n) + \\gamma_{n+1}\\eta_{n+1} + \\gamma_{n+1} \\epsilon_{n+1}\\,,\n\\end{equation}\nwhere $\\epsilon_{n+1} \\eqdef (\\epsilon_{n+1}^1,\\epsilon_{n+1}^2,\\epsilon^3_{n+1})$, whose components are given by\n\\begin{equation*}\n  \\resizebox{\\hsize}{!}{$\n  \\epsilon_{n+1}^1 =  \\frac{m_n}{\\varepsilon+\\sqrt{v_n}} - \\frac{\\hat m_{n+1}}{\\varepsilon+\\sqrt{\\hat v_{n+1}}};\\,\n  \\epsilon_{n+1}^2 = \\left(\\frac{1-\\alpha_{n+1}}{\\gamma_{n+1}}-a\\right)\\left( \\nabla f_{n+1} - m_n\\right);\\,\n  \\epsilon_{n+1}^3 =  \\left(\\frac{1-\\beta_{n+1}}{\\gamma_{n+1}}-b\\right)\\left( \\nabla f_{n+1}^{\\odot 2} - v_n\\right)\\,.\n  $}\n\\end{equation*}\nHere, $\\eta_{n+1}$ is a martingale increment noise and\n$\\epsilon_{n+1} = (\\epsilon_{n+1}^1,\\epsilon_{n+1}^2,\\epsilon^3_{n+1})$ is a remainder term.\nThe aim is to check the assumptions (A1.1) to (A1.3) of\n\\cite{pelletier1998weak}, where the role of the quantities ($h$,\n$\\varepsilon_n$, $r_n$, $\\sigma_n$, $\\alpha$, $\\rho$, $\\beta$) in\n\\cite{pelletier1998weak} is respectively played by the quantities\n($h_\\infty$, $\\eta_n$, $\\epsilon_n$, $\\gamma_n$, $\\kappa$, $1$, $1$) of\nthe present paper.\n","sha256_utf8":"d5d9d75e5938d8b10801a423d438a9e5fa7a1e1129786999aa4751bbf44112b5"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"验证线性化稳定性与步长假设","source_units":[{"start":176675,"end":177420,"start_line":3016,"end_line":3026,"raw_tex":"Let us first consider Assumption (A1.1) for $h_\\infty$.\nBy construction, $h_\\infty(z^*) = 0$. By Assumptions~\\ref{hyp:mean_field_tcl} and \\ref{hyp:S>0},\n$h_\\infty$ is continuously differentiable in the neighborhood of $z^*$ and its\nJacobian at $z^*$ coincides with the matrix $H$ given by Eq.~(\\ref{eq:H}).\n%As already discussed, after straightforward algebra,\nAs already discussed, after some algebra,\nit can be shown that the largest real part of the eigenvalues of $H$ coincides with $-L$ where $L>0$\nis given by Eq.~(\\ref{eq:L}).\nHence, Assumption (A1.1) of \\cite{pelletier1998weak}\nis satisfied for $h_\\infty$. Assumption (A1.3) is trivially satisfied\nusing Assumption~\\ref{hyp:step-tcl}. The crux is therefore to verify Assumption (A1.2).\n","sha256_utf8":"447f1b5e1af9f22da195ef71e71bb2fa807fcaa5694b0245b38babd2e50e9cc8"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"验证鞅差与高阶矩控制","source_units":[{"start":177420,"end":177773,"start_line":3027,"end_line":3033,"raw_tex":"Clearly, $\\bE(\\eta_{n+1}|\\cF_n)=0$. Using Assumption~\\ref{hyp:moment-f}\\ref{momentreinforce},\nit follows from straightforward manipulations based on Jensen's inequality\nthat for any $M>0$, %s.t. $B(z^*,M)\\subset \\cV$,\nthere exists $\\delta>0$ s.t.\n$%$$\n\\sup_{n\\geq 0}\\bE_n\\left(\\|\\eta_{n+1}\\|^{2+\\delta}\\right) \\1_{\\{\\|z_n-z^*\\|\\leq M\\}}<\\infty\\,.\n$ %$$\n","sha256_utf8":"5c4c230862eda01b23aec6afe532ce2b7ffc0f36ece74dbb591fe635a47071f8"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"将余项条件化为增量与校正误差","source_units":[{"start":177773,"end":178851,"start_line":3034,"end_line":3053,"raw_tex":"Next, we verify the condition\n\\begin{equation}\n  \\label{eq:cond-reste}\n  \\lim_{n\\to\\infty} \\bE\\left(\\gamma_{n+1}^{-1}\\|\\epsilon_{n+1}\\|^2\\1_{\\{\\|z_n-z^*\\|\\leq M\\}}\\right) = 0\\,.\n\\end{equation}\nIt is sufficient to verify the latter for $\\epsilon^i_n$ ($i=1,2,3$) in place of $\\epsilon_n$.\nThe map $(m,v)\\mapsto m/(\\varepsilon+\\sqrt{v})$ is Lipschitz continuous in a neighborhood of\n$(0,S(x^*))$ by Assumption~\\ref{hyp:S>0}. Thus, for $M$ small enough, there exists a constant $C$ s.t.\nif $ \\|z_n-z^*\\|\\leq M$, then\n%$$\n%  \\|\\epsilon_{n+1}^1\\| \\leq C\\left\\|\\frac{m_{n+1}}{r_{n+1}}-m_n\\right\\|+C\\left\\|\\frac {v_{n+1}}{\\bar r_{n+1}}-v_n\\right\\| \\,.\n%$$\n$%$$\n  \\|\\epsilon_{n+1}^1\\| \\leq C\\left\\|r_{n+1}^{-1}m_{n+1}-m_n\\right\\|+C\\left\\|\\bar r_{n+1}^{-1} v_{n+1}-v_n\\right\\|\\,.\n$ %$$\nUsing the triangular inequality and the fact that $r_{n+1},\\bar r_{n+1}$ are bounded sequences away from zero, there exists an other constant $C$ s.t.\n\\begin{align*}\n  \\|\\epsilon_{n+1}^1\\| &\\leq C\\left\\|m_{n+1}-m_n\\right\\|+C\\left\\|v_{n+1}-v_{n}\\right\\|\n+C|r_{n+1}-1|+C|\\bar r_{n+1}-1|\\,.\n\\end{align*}\n","sha256_utf8":"f6ce90d4dbbf98b057fc0eb3986e561c53a14fe7104969c2d932f1d77a8f3d9f"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"证明全部余项满足所需可忽略性","source_units":[{"start":178954,"end":180428,"start_line":3055,"end_line":3072,"raw_tex":"Using Lemma~\\ref{lemma:r_n} under Assumption~\\ref{hyp:step-tcl} (note that $\\gamma_0 > 1/2L \\geq 1/a$ when $\\kappa=1$),\nwe obtain that the sequence $|r_{n}-1|/\\gamma_n$ is bounded, thus $|r_{n+1}-1|\\leq C\\gamma_{n+1}$.\n%{\\color{red} voir si on a la place de le mettre dans un lemme.}\n The sequence $(1-\\alpha_{n})/\\gamma_n$ being also bounded, it holds that\n%$\\|m_{n+1}-m_n\\|\\leq C\\gamma_{n+1}\\|\\nabla f_{n+1}-m_n\\|$, and if $ \\|z_n-z^*\\|\\leq M$,\n\\begin{align*}\n  \\|m_{n+1}-m_n\\|^2 \\1_{\\{\\|z_n-z^*\\|\\leq M\\}} \\leq  C\\gamma_{n+1}^2(1+\\|\\nabla f_{n+1}\\|^2 )\\1_{\\{\\|z_n-z^*\\|\\leq M\\}}\\,.\n\\end{align*}\nBy Assumption~\\ref{hyp:moment-f}~\\ref{momentreinforce}, %Assumption~\\ref{hyp:mean_field_tcl},\n$\\bE_n(\\|\\nabla f_{n+1}\\|^2|)$ is bounded by a deterministic constant on $\\{\\|z_n-z^*\\|\\leq M\\}$.\nThus, $\\bE_n(\\|m_{n+1}-m_n\\|^2\\1_{\\{\\|z_n-z^*\\|\\leq M\\}})\\leq C\\gamma_{n+1}^2$.\nA similar result holds for $\\|v_{n+1}-v_n\\|^2$. We have thus shown that\n$ \\bE_n\\left(\\|\\epsilon_{n+1}^1\\|^2\\1_{\\{\\|z_n-z^*\\|\\leq M\\}}\\right)\\leq C\\gamma_{n+1}^2$. Hence,\nEq.~(\\ref{eq:cond-reste}) is proved for $\\epsilon_{n+1}^1$ in place of $\\epsilon_{n+1}$.\n%As far as $\\epsilon_{n+1}^2$, $\\epsilon_{n+1}^3$ are concerned, the proof uses the same kind of arguments\n%under Assumption~\\ref{hyp:step-tcl} and is omitted.\nUnder Assumption~\\ref{hyp:step-tcl}, the proof uses the same kind of arguments for $\\epsilon_{n+1}^2$, $\\epsilon_{n+1}^3$  and is omitted.\nFinally, Eq.~(\\ref{eq:cond-reste}) is proved.\n","sha256_utf8":"f2e37d3dba9c5b6e902bbcf8d4cdf87714d32169e98f18cc1df266156275e41d"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"条件噪声协方差收敛","source_units":[{"start":180428,"end":180922,"start_line":3073,"end_line":3080,"raw_tex":"Continuing the verification of Assumption (A1.2), we establish that\n\\begin{equation}\n  \\label{eq:lim-cov}\n  \\bE_n(\\eta_{n+1}\\eta_{n+1}^T) \\to Q\\textrm{ a.s. on } \\{z_n\\to z^*\\}\\,.\n\\end{equation}\nDenote by $\\bar Q(x)$ the matrix given by the righthand side of Eq.~(\\ref{eq:Q}) when $x^*$\nis replaced by an arbitrary $x\\in \\cV$. It is easily checked that $\\bE_n(\\eta_{n+1}\\eta_{n+1}^T)=\\bar Q(x_n)$\nand by continuity, $\\bar Q(x_n)\\to Q$ a.s. on $\\{z_n\\to z^*\\}$, which proves (\\ref{eq:lim-cov}).\n","sha256_utf8":"8733672048b6fe4e8bff5b5a9a8660d25c195a5b9180654768158dab978d3ab0"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"以广义逆处理退化噪声并确定高斯极限","source_units":[{"start":180922,"end":181823,"start_line":3081,"end_line":3090,"raw_tex":"Therefore, Assumption (A1.2) is fulfilled, except for the point mentioned at the beginning of this section : %paragraph:\n\\cite{pelletier1998weak} puts the additional condition that the limit\nmatrix in Eq.~(\\ref{eq:lim-cov}) is positive definite. This condition is not satisfied in our case,\nbut the proof can still be adapted. The specific part of the proof where the positive definiteness\ncomes into play is Th.~7 in \\cite{pelletier1998weak}. The proof of \\cite[Th.~1]{pelletier1998weak} can therefore\nbe adapted to the case of a positive semidefinite matrix.\n%by substituting the following\n%result to Th.~7 of the same paper.\nIn the proof of \\cite[Th.~7]{pelletier1998weak}, we only substitute the inverse of the square root of $Q$ by the Moore-Penrose inverse.\nFinally, the uniqueness of the stationary distribution $\\mu$ and its expression follow from \\cite[Th.~6.7, p. 357]{karatzas-shreve1991}.\n","sha256_utf8":"c891b9bb14df5c21430e680998845045dd04ecce086d24f8adb66410970181d0"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"提取位置与动量的协方差子系统","source_units":[{"start":184044,"end":184692,"start_line":3125,"end_line":3140,"raw_tex":"\\noindent\\textbf{Proof of Eq.~(\\ref{eq:cov})}.\nWe introduce the $d\\times d$ blocks of the $3d\\times 3d$ matrix %$\\Sigma$ by\n$\\Sigma = \\left(\\Sigma_{i,j}\\right)_{i,j=1,2,3}$\nwhere $\\Sigma_{i,j}$ is $d\\times d$. We denote by $\\tilde\\Sigma$ the $2d\\times 2d$ submatrix $\\tilde\\Sigma \\eqdef\\left(\\Sigma_{i,j}\\right)_{i,j=1,2}$.\nBy Th.~\\ref{thm:clt}, we have the subsystem:\n\\begin{equation}\n\\tilde H \\tilde\\Sigma + \\tilde\\Sigma\\tilde H^T =\n\\begin{pmatrix}\n  0 & 0 \\\\\n0 & -a^2 \\tilde Q\n\\end{pmatrix}\\qquad\\text{where }\\tilde H \\eqdef\n\\begin{pmatrix}\n  \\zeta I_d & -D \\\\\na\\nabla^2F(x^*) & (\\zeta-a) I_d\n\\end{pmatrix}\\label{eq:lyap-reduced}\n\\end{equation}\n","sha256_utf8":"d08efc053feef2610c7bf821e5edba66884227032a712baf190884abf44d9e1c"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"构造谱矩阵并三角化漂移","source_units":[{"start":184692,"end":185542,"start_line":3141,"end_line":3162,"raw_tex":"and where $\\tilde Q \\eqdef \\textrm{Cov}\\left(\\nabla\n  f(x^*,\\xi)\\right)$.  The next step is to triangularize the matrix\n$\\tilde H$ in order to decouple the blocks of $\\tilde\\Sigma$.  For\nevery $k=1,\\dots,d$, set\n$\\nu_k^\\pm \\eqdef -\\frac{a}{2}\\pm\\sqrt{a^2/4-a\\lambda_k}$\nwith the convention that $\\sqrt{-1} =\n\\imath$ (inspecting the characteristic polynomial of $\\tilde H$, these %quantities\nare the eigenvalues of $\\tilde H$). Set\n$M^\\pm\\eqdef\\diag{(\\nu_1^\\pm,\n\\cdots,\\nu_d^\\pm)}$ and $R^\\pm\\eqdef\nD^{-1/2}PM^\\pm P^TD^{-1/2}$.  Using the identities $M^++M^-=-a I_d$ and\n$M^+M^-=a\\Lambda$ where\n$\\Lambda\\eqdef\\diag{(\\lambda_1,\\cdots,\\lambda_d)}$, it can be checked\nthat\n$$\n\\cR\\tilde H =\\begin{pmatrix}\n  D R^+ + \\zeta I_d & -D \\\\ 0 & R^-D + \\zeta I_d\n\\end{pmatrix}\\cR,\\text{ where }\\cR\\eqdef\n\\begin{pmatrix}\n  I_d & 0 \\\\ R^+ & I_d\n\\end{pmatrix}\\,.\n$$\n","sha256_utf8":"d6cd5fee8050886e3d4fc46ffc415b1df4f146dbede7584cedd737acb759aca6"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"变换协方差并建立三个矩阵方程","source_units":[{"start":185542,"end":186610,"start_line":3163,"end_line":3182,"raw_tex":"Set $\\check \\Sigma \\eqdef \\cR\\tilde\\Sigma\\cR^T$. Denote by\n$(\\check \\Sigma_{i,j})_{i,j=1,2}$ the blocks of $\\check\\Sigma$.\nNote that $\\check\\Sigma_{1,1} = \\Sigma_{1,1}$. By left/right multiplication of Eq.~(\\ref{eq:lyap-reduced})\nrespectively with $\\cR$ and $\\cR^T$, we obtain\n%% $$\n%% \\cT\\check\\Sigma + \\check\\Sigma\\cT^T =\n%% \\begin{pmatrix}\n%%   0 & 0 \\\\\n%% 0 & -a^2 \\tilde Q\n%% \\end{pmatrix}\\,.\n%% $$\n%% By expanding the above system, we obtain\n\\begin{align}\n  &(DR^++\\zeta I_d) \\Sigma_{1,1}+\\Sigma_{1,1}(R^+D+\\zeta I_d) = \\check\\Sigma_{1,2} D+ D\\check\\Sigma_{1,2}^T \\label{eq:lapremiere}\\\\\n& (DR^++\\zeta I_d) \\check\\Sigma_{1,2}+\\check\\Sigma_{1,2} (DR^-+\\zeta I_d) = D\\check\\Sigma_{2,2} \\label{eq:ladeuxieme}\\\\\n& (R^-D+\\zeta I_d) \\check\\Sigma_{2,2} + \\check\\Sigma_{2,2} (DR^-+\\zeta I_d) = -a^2\\tilde Q \\label{eq:laderniere}\n\\end{align}\nSet $\\bar \\Sigma_{2,2} = P^{-1}D^{1/2}\\check\\Sigma_{2,2} D^{1/2}P$.\nDefine $C\\eqdef P^{-1}D^{1/2}\\tilde Q D^{1/2}P$.\nEq.~(\\ref{eq:laderniere}) yields $(M^-+\\zeta I_d) \\bar\\Sigma_{2,2} + \\bar\\Sigma_{2,2} (M^-+\\zeta I_d) = -a^2C$.\n","sha256_utf8":"67a8dd3a5eaf3bf024c7fae123eea111286659f5691c5ef2a7dfc4044e6f0141"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"逐坐标求解交叉协方差","source_units":[{"start":186738,"end":187166,"start_line":3187,"end_line":3192,"raw_tex":"Set $\\bar \\Sigma_{1,2} = P^{-1}D^{-1/2}\\check\\Sigma_{1,2} D^{1/2}P$.\nEq.~(\\ref{eq:ladeuxieme}) rewrites $(M^++\\zeta I_d) \\bar \\Sigma_{1,2}+\\bar \\Sigma_{1,2} (M^-+\\zeta I_d) = \\bar \\Sigma_{2,2}$.\nWe obtain that %The component $(k,\\ell)$ is given by\n$%$$\n\\bar\\Sigma_{1,2}^{k,\\ell} = (\\nu_k^++\\nu_\\ell^-+2\\zeta)^{-1}\\bar\\Sigma_{2,2}^{k,\\ell} = \\frac{-a^2C_{k,\\ell}}{(\\nu_k^++\\nu_\\ell^-+2\\zeta)(\\nu_k^-+\\nu_\\ell^-+2\\zeta)}\\,.\n$ %$$\n","sha256_utf8":"2e54cf909318af76552107c0b89e78033fc0c035e060773f9cff23bcec76f41c"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"求解位置协方差并得到显式公式","source_units":[{"start":187166,"end":188169,"start_line":3193,"end_line":3210,"raw_tex":"Set %finally\n$\\bar \\Sigma_{1,1} = P^{-1}D^{-1/2}\\Sigma_{1,1} D^{-1/2}P$.\nEq.~(\\ref{eq:lapremiere}) becomes $(M^++\\zeta I_d)\\bar \\Sigma_{1,1} + \\bar \\Sigma_{1,1}(M^++\\zeta I_d) = \\bar \\Sigma_{1,2} + \\bar \\Sigma_{1,2}^T$. Thus,\n%\\begin{equation*}\n%\\resizebox{.99\\hsize}{!}{$\n\\begin{align*}\n\\bar\\Sigma_{1,1}^{k,\\ell} &\n\\resizebox{.9\\hsize}{!}{$\n= \\frac{\\bar\\Sigma_{1,2}^{k,\\ell}+\\bar\\Sigma_{1,2}^{\\ell,k}}{\\nu_k^++\\nu_\\ell^++2\\zeta}\n=  \\frac{-a^2C_{k,\\ell}}{(\\nu_k^++\\nu_\\ell^++2\\zeta)(\\nu_k^-+\\nu_\\ell^-+2\\zeta)}\\left(\\frac{1}{\\nu_k^++\\nu_\\ell^-+2\\zeta}\n+\\frac{1}{\\nu_k^-+\\nu_\\ell^++2\\zeta}\\right)\n$}\n\\\\ &\n%\\resizebox{.6\\hsize}{!}{$= \\frac{C_{k,\\ell}}{ (1 - \\frac{2\\zeta}{a})(\\lambda_k+\\lambda_\\ell-2\\zeta + \\frac 2a \\zeta^2) +\\frac 1{2(a-2\\zeta)}(\\lambda_k-\\lambda_\\ell)^2}\\,.$}\n= \\frac{C_{k,\\ell}}{ (1 - \\frac{2\\zeta}{a})(\\lambda_k+\\lambda_\\ell-2\\zeta + \\frac 2a \\zeta^2) +\\frac 1{2(a-2\\zeta)}(\\lambda_k-\\lambda_\\ell)^2}\\,,\n%\\,\\,\\text{and  the result is proved.}\n\\end{align*}\nand  the result is proved.\n","sha256_utf8":"60ecb829afce4464570bec2f461fc4f131a1df452dcb472c2eb0a56cd5b8d9cc"}]},{"id":"S042","semantic_type":"external_fact","kind":"external_fact","title":"随机逼近的条件中心极限框架","source_units":[],"declared_external":true,"declaration_text":["Pelletier (1998), Theorem 1 and Theorem 7 proof framework give conditional stochastic-approximation central limit theory under stable differentiable drift, polynomial stepsizes, appropriate local martingale moments, convergent conditional covariance and negligible remainders. The original covariance condition is positive definiteness; the selected author argument supplies its adaptation to the semidefinite case."]},{"id":"S043","semantic_type":"external_fact","kind":"external_fact","title":"退化线性扩散的平稳高斯分布","source_units":[],"declared_external":true,"declaration_text":["For a Hurwitz matrix A and positive semidefinite Q, the Ornstein–Uhlenbeck process dX_t=A X_t dt+sqrt(Q)dB_t has a unique invariant centered Gaussian distribution. Its covariance is the unique solution of A Sigma+Sigma A^T=-Q; this is the cited Karatzas–Shreve stationary-distribution fact."]},{"id":"S044","semantic_type":"external_fact","kind":"external_fact","title":"标准积分、矩与矩阵运算","source_units":[],"declared_external":true,"declaration_text":["Use the standard triangle, Jensen and Cauchy–Schwarz inequalities, conditional-expectation identities, dominated differentiation under expectation under the stated local integrability conditions, and ordinary finite-dimensional block-matrix algebra."]},{"id":"S045","semantic_type":"external_fact","kind":"external_fact","title":"乘积指数界与幂步长求和","source_units":[],"declared_external":true,"declaration_text":["For u in [0,1], 1-u<=exp(-u). For gamma_i=gamma_0/(i+1)^kappa, the partial sums are asymptotic to gamma_0 n^(1-kappa)/(1-kappa) when 0<kappa<1 and to gamma_0 log n when kappa=1."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S044","target_id":"S007","kind":"supports"},{"source_id":"S004","target_id":"S030","kind":"supports"},{"source_id":"S006","target_id":"S030","kind":"supports"},{"source_id":"S007","target_id":"S030","kind":"supports"},{"source_id":"S009","target_id":"S031","kind":"supports"},{"source_id":"S010","target_id":"S027","kind":"supports"},{"source_id":"S011","target_id":"S027","kind":"supports"},{"source_id":"S045","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S011","target_id":"S028","kind":"supports"},{"source_id":"S019","target_id":"S028","kind":"supports"},{"source_id":"S045","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S010","target_id":"S030","kind":"supports"},{"source_id":"S011","target_id":"S030","kind":"supports"},{"source_id":"S021","target_id":"S030","kind":"supports"},{"source_id":"S023","target_id":"S030","kind":"supports"},{"source_id":"S042","target_id":"S029","kind":"supports"},{"source_id":"S018","target_id":"S029","kind":"supports"},{"source_id":"S014","target_id":"S031","kind":"supports"},{"source_id":"S008","target_id":"S031","kind":"supports"},{"source_id":"S015","target_id":"S031","kind":"supports"},{"source_id":"S016","target_id":"S031","kind":"supports"},{"source_id":"S017","target_id":"S031","kind":"supports"},{"source_id":"S019","target_id":"S031","kind":"supports"},{"source_id":"S021","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S007","target_id":"S032","kind":"supports"},{"source_id":"S012","target_id":"S032","kind":"supports"},{"source_id":"S013","target_id":"S032","kind":"supports"},{"source_id":"S022","target_id":"S032","kind":"supports"},{"source_id":"S023","target_id":"S032","kind":"supports"},{"source_id":"S044","target_id":"S032","kind":"supports"},{"source_id":"S030","target_id":"S033","kind":"supports"},{"source_id":"S008","target_id":"S033","kind":"supports"},{"source_id":"S010","target_id":"S033","kind":"supports"},{"source_id":"S044","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S025","target_id":"S034","kind":"supports"},{"source_id":"S024","target_id":"S034","kind":"supports"},{"source_id":"S017","target_id":"S034","kind":"supports"},{"source_id":"S019","target_id":"S034","kind":"supports"},{"source_id":"S010","target_id":"S034","kind":"supports"},{"source_id":"S012","target_id":"S034","kind":"supports"},{"source_id":"S013","target_id":"S034","kind":"supports"},{"source_id":"S023","target_id":"S034","kind":"supports"},{"source_id":"S030","target_id":"S035","kind":"supports"},{"source_id":"S005","target_id":"S035","kind":"supports"},{"source_id":"S012","target_id":"S035","kind":"supports"},{"source_id":"S013","target_id":"S035","kind":"supports"},{"source_id":"S018","target_id":"S035","kind":"supports"},{"source_id":"S023","target_id":"S035","kind":"supports"},{"source_id":"S029","target_id":"S036","kind":"supports"},{"source_id":"S031","target_id":"S036","kind":"supports"},{"source_id":"S032","target_id":"S036","kind":"supports"},{"source_id":"S034","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S020","target_id":"S036","kind":"supports"},{"source_id":"S042","target_id":"S036","kind":"supports"},{"source_id":"S043","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S016","target_id":"S037","kind":"supports"},{"source_id":"S018","target_id":"S037","kind":"supports"},{"source_id":"S044","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S015","target_id":"S038","kind":"supports"},{"source_id":"S044","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S044","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S038","target_id":"S041","kind":"supports"},{"source_id":"S029","target_id":"S002","kind":"supports"},{"source_id":"S030","target_id":"S002","kind":"supports"},{"source_id":"S031","target_id":"S002","kind":"supports"},{"source_id":"S032","target_id":"S002","kind":"supports"},{"source_id":"S033","target_id":"S002","kind":"supports"},{"source_id":"S034","target_id":"S002","kind":"supports"},{"source_id":"S035","target_id":"S002","kind":"supports"},{"source_id":"S036","target_id":"S002","kind":"supports"},{"source_id":"S037","target_id":"S002","kind":"supports"},{"source_id":"S038","target_id":"S002","kind":"supports"},{"source_id":"S039","target_id":"S002","kind":"supports"},{"source_id":"S040","target_id":"S002","kind":"supports"},{"source_id":"S041","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S001","kind":"supports"}]}}
{"record_id":"1902.01644v2","paper":{"title":"On the Convergence of Projected-Gradient Methods with Low-Rank Projections for Smooth Convex Minimization over Trace-Norm Balls and Related Problems","authors":["Dan Garber"],"arxiv_id":"1902.01644","version":"v2","journal":"SIAM Journal on Optimization","doi":"10.1137/18m1233170","abstract_url":"https://arxiv.org/abs/1902.01644v2","pdf_url":"https://arxiv.org/pdf/1902.01644v2"},"context":{"raw_tex":"\\begin{eqnarray}\\label{eq:optProb}\n\\min_{\\Vert{\\X}\\Vert_*\\leq 1}f(\\X),\n\\end{eqnarray}\n\nwhere $f:\\reals^{m\\times n}\\rightarrow\\reals$ is convex and $\\beta$-smooth (i.e., gradient-Lipschitz), and $\\Vert{\\cdot}\\Vert_*$ denotes the trace-norm, i.e., sum of singular values (aka the nuclear norm).\n\n\nWe begin with introducing some notation. For a positive integer $n$ we let $[n]$ denote the set $\\{1,2,\\dots,n\\}$.\nWe let $\\A\\bullet\\B$ denote the standard inner product for matrices, i.e., $\\A\\bullet\\B = \\trace(\\A^{\\top}\\B)$. For a real matrix $\\A$, we let $\\sigma_i(\\A)$ denote its $i$th largest singular value (including multiplicities), and we let $\\#\\sigma_i(\\A)$ denote the multiplicity of the $i$th largest singular value. Similarly, for a real symmetric matrix $\\A$, we let $\\lambda_i(\\A)$ denote its $i$th largest (signed) eigenvalue, and we let $\\#\\lambda_i(\\A)$ denote the multiplicity of the $i$the largest eigenvalue.\nWe denote by $\\mX^*$ the set of optimal solutions to Problem \\eqref{eq:optProb}, and by $f^*$ the corresponding optimal value. %For any feasible $\\X$, we define the distance $\\dist(\\X,\\mX^*) := \\min_{\\X^*\\in\\mX^*}\\Vert{\\X-\\X^*}\\Vert_F$. \n\nFor any $\\X\\in\\reals^{m\\times n}$, step-size $\\eta$ and radius $\\tau$ we denote the projected gradient mapping w.r.t. the trace-norm ball of radius $\\tau$:\n\\begin{eqnarray*}\n\\projgrad_{\\eta,\\tau}(\\X) := \\Pi_{\\Vert\\cdot\\Vert_*\\leq\\tau}\\left[{\\X-\\eta\\nabla{}f(\\X)}\\right].\n\\end{eqnarray*}\nWhen $\\tau=1$, i.e., we consider the unit trace-norm ball, we will omit the $\\tau$ subscript and simply write $\\projgrad_{\\eta}$.\n\nGiven an optimal solution $\\X^*\\in\\mX^*$, a step-size $\\eta >0$, and an integer $r$ in  $\\{\\rank(\\X^*),\\dots\\min\\{m,n\\}\\}$, we let $\\delta(\\X^*,\\eta,r)$ denote the radius of the largest Euclidean ball centered at $\\X^*$, such that for all $\\X$ in the ball it holds that $\\rank\\left({\\projgrad_{\\eta}(\\X)}\\right) \\leq r$. Or equivalently, $\\delta(\\X^*,\\eta,r)$ is the solution to the optimization problem\n\\begin{eqnarray*}\n\\sup\\delta \\geq 0 \\quad \\textrm{s.t.} \\quad \\forall\\X\\in\\ball(\\X^*,\\delta):\\rank\\left({\\projgrad_{\\eta}(\\X)}\\right)\\leq r,\n\\end{eqnarray*}\nwhere $\\ball(\\X,R)$ denotes the Euclidean ball of radius $R$ centered at $\\X$.\n","units":[{"start":4695,"end":4780,"start_line":114,"end_line":116,"raw_tex":"\\begin{eqnarray}\\label{eq:optProb}\n\\min_{\\Vert{\\X}\\Vert_*\\leq 1}f(\\X),\n\\end{eqnarray}","sha256_utf8":"d7f43653e83b6edbb293af2cfd0110ac2f43fce9f1e023ff3a0bdc60816f347b"},{"start":4781,"end":4987,"start_line":117,"end_line":117,"raw_tex":"where $f:\\reals^{m\\times n}\\rightarrow\\reals$ is convex and $\\beta$-smooth (i.e., gradient-Lipschitz), and $\\Vert{\\cdot}\\Vert_*$ denotes the trace-norm, i.e., sum of singular values (aka the nuclear norm).\n","sha256_utf8":"d29dd6c29319151f29763f6f034f81cbc95b4343d8edd0aa0a646ec7f8893cb8"},{"start":18790,"end":20718,"start_line":210,"end_line":224,"raw_tex":"We begin with introducing some notation. For a positive integer $n$ we let $[n]$ denote the set $\\{1,2,\\dots,n\\}$.\nWe let $\\A\\bullet\\B$ denote the standard inner product for matrices, i.e., $\\A\\bullet\\B = \\trace(\\A^{\\top}\\B)$. For a real matrix $\\A$, we let $\\sigma_i(\\A)$ denote its $i$th largest singular value (including multiplicities), and we let $\\#\\sigma_i(\\A)$ denote the multiplicity of the $i$th largest singular value. Similarly, for a real symmetric matrix $\\A$, we let $\\lambda_i(\\A)$ denote its $i$th largest (signed) eigenvalue, and we let $\\#\\lambda_i(\\A)$ denote the multiplicity of the $i$the largest eigenvalue.\nWe denote by $\\mX^*$ the set of optimal solutions to Problem \\eqref{eq:optProb}, and by $f^*$ the corresponding optimal value. %For any feasible $\\X$, we define the distance $\\dist(\\X,\\mX^*) := \\min_{\\X^*\\in\\mX^*}\\Vert{\\X-\\X^*}\\Vert_F$. \n\nFor any $\\X\\in\\reals^{m\\times n}$, step-size $\\eta$ and radius $\\tau$ we denote the projected gradient mapping w.r.t. the trace-norm ball of radius $\\tau$:\n\\begin{eqnarray*}\n\\projgrad_{\\eta,\\tau}(\\X) := \\Pi_{\\Vert\\cdot\\Vert_*\\leq\\tau}\\left[{\\X-\\eta\\nabla{}f(\\X)}\\right].\n\\end{eqnarray*}\nWhen $\\tau=1$, i.e., we consider the unit trace-norm ball, we will omit the $\\tau$ subscript and simply write $\\projgrad_{\\eta}$.\n\nGiven an optimal solution $\\X^*\\in\\mX^*$, a step-size $\\eta >0$, and an integer $r$ in  $\\{\\rank(\\X^*),\\dots\\min\\{m,n\\}\\}$, we let $\\delta(\\X^*,\\eta,r)$ denote the radius of the largest Euclidean ball centered at $\\X^*$, such that for all $\\X$ in the ball it holds that $\\rank\\left({\\projgrad_{\\eta}(\\X)}\\right) \\leq r$. Or equivalently, $\\delta(\\X^*,\\eta,r)$ is the solution to the optimization problem\n\\begin{eqnarray*}\n\\sup\\delta \\geq 0 \\quad \\textrm{s.t.} \\quad \\forall\\X\\in\\ball(\\X^*,\\delta):\\rank\\left({\\projgrad_{\\eta}(\\X)}\\right)\\leq r,\n\\end{eqnarray*}\nwhere $\\ball(\\X,R)$ denotes the Euclidean ball of radius $R$ centered at $\\X$.\n","sha256_utf8":"189d0f7348c04caf9e79050f218bd58218100d64ee478067b96c45982f9b0c68"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{thm:goodProj}\nLet $f:\\reals^{m\\times n}\\rightarrow\\reals$ be $\\beta$-smooth and convex. Assume $\\nabla{}f$ is non-zero over the unit trace-norm ball and fix some $\\X^*\\in\\mX^*$. Let $r$ denote the multiplicity of $\\sigma_1(\\nabla{}f(\\X^*))$, and let $\\mu_1,\\mu_2,\\dots,\\mu_{\\min\\{m,n\\}}$ denote the singular values of $\\nabla{}f(\\X^*)$ (including multiplicities).  Then, for any $\\eta >0$ it holds that\n\\begin{eqnarray}\\label{eq:goodProjRes:1}\n\\delta(\\X^*,\\eta,r) \\geq  \\frac{\\eta{}(\\mu_1-\\mu_{r+1})}{(1+1/\\sqrt{r})(1+\\eta\\beta)}.\n\\end{eqnarray}\n\n%Then, for any $\\eta >0$ and for any $\\X$ (not necessarily feasible) satisfying $\\Vert{\\X-\\X^*}\\Vert_F \\leq \\delta(\\X^*,\\eta,r) :=  \\frac{\\eta{}(\\mu_1-\\mu_{r+1})}{(1+1/\\sqrt{r})(1+\\eta\\beta)}$ it holds that $\\rank\\left({\\Pi_{\\Vert\\cdot\\Vert_*\\leq1}[\\X-\\eta\\nabla{}f(\\X)]}\\right) \\leq r$.\n\nMore generally, for any $\\eta>0$ and $r'\\in\\{r,\\dots,\\min\\{m,n\\}-1\\}$, it holds that \n\\begin{eqnarray}\\label{eq:goodProjRes:2}\n\\delta(\\X^*,\\eta,r') \\geq  \\frac{\\eta(\\mu_1-\\mu_{r'+1})}{(1+1/\\sqrt{r})(1+\\eta\\beta)}.\n\\end{eqnarray}\n\n%for any $\\X$ satisfying $\\Vert{\\X-\\X^*}\\Vert_F  \\leq \\delta(\\X^*,\\eta,r') :=  \\frac{\\eta(\\mu_1-\\mu_{r'+1})}{(1+1/\\sqrt{r})(1+\\eta\\beta)}$ that  $\\rank\\left({\\Pi_{\\Vert\\cdot\\Vert_*\\leq1}[\\X-\\eta\\nabla{}f(\\X)]}\\right) \\leq r'$.\n\nMoreover, for any $\\eta >0$ and $r'\\in\\{r,\\dots,\\min\\{m,n\\}-r\\}$, it holds that \n\\begin{eqnarray}\\label{eq:goodProjRes:3}\n\\delta(\\X^*,\\eta,r'+r-1) \\geq  \\frac{\\sqrt{r}\\eta(\\mu_1-\\mu_{r'+1})}{2(1+\\eta\\beta)}.\n\\end{eqnarray}\n\n%and any $\\eta >0$, it holds for any $\\X$ satisfying $\\Vert{\\X-\\X^*}\\Vert_F  \\leq \\frac{\\sqrt{r}\\eta(\\mu_1-\\mu_{r'+1})}{2(1+\\eta\\beta)}$ that  $\\rank\\left({\\Pi_{\\Vert\\cdot\\Vert_*\\leq1}[\\X-\\eta\\nabla{}f(\\X)]}\\right) \\leq r'' = r'+r-1$.\n\\end{theorem}","units":[{"start":35592,"end":37380,"start_line":424,"end_line":445,"raw_tex":"\\begin{theorem}\\label{thm:goodProj}\nLet $f:\\reals^{m\\times n}\\rightarrow\\reals$ be $\\beta$-smooth and convex. Assume $\\nabla{}f$ is non-zero over the unit trace-norm ball and fix some $\\X^*\\in\\mX^*$. Let $r$ denote the multiplicity of $\\sigma_1(\\nabla{}f(\\X^*))$, and let $\\mu_1,\\mu_2,\\dots,\\mu_{\\min\\{m,n\\}}$ denote the singular values of $\\nabla{}f(\\X^*)$ (including multiplicities).  Then, for any $\\eta >0$ it holds that\n\\begin{eqnarray}\\label{eq:goodProjRes:1}\n\\delta(\\X^*,\\eta,r) \\geq  \\frac{\\eta{}(\\mu_1-\\mu_{r+1})}{(1+1/\\sqrt{r})(1+\\eta\\beta)}.\n\\end{eqnarray}\n\n%Then, for any $\\eta >0$ and for any $\\X$ (not necessarily feasible) satisfying $\\Vert{\\X-\\X^*}\\Vert_F \\leq \\delta(\\X^*,\\eta,r) :=  \\frac{\\eta{}(\\mu_1-\\mu_{r+1})}{(1+1/\\sqrt{r})(1+\\eta\\beta)}$ it holds that $\\rank\\left({\\Pi_{\\Vert\\cdot\\Vert_*\\leq1}[\\X-\\eta\\nabla{}f(\\X)]}\\right) \\leq r$.\n\nMore generally, for any $\\eta>0$ and $r'\\in\\{r,\\dots,\\min\\{m,n\\}-1\\}$, it holds that \n\\begin{eqnarray}\\label{eq:goodProjRes:2}\n\\delta(\\X^*,\\eta,r') \\geq  \\frac{\\eta(\\mu_1-\\mu_{r'+1})}{(1+1/\\sqrt{r})(1+\\eta\\beta)}.\n\\end{eqnarray}\n\n%for any $\\X$ satisfying $\\Vert{\\X-\\X^*}\\Vert_F  \\leq \\delta(\\X^*,\\eta,r') :=  \\frac{\\eta(\\mu_1-\\mu_{r'+1})}{(1+1/\\sqrt{r})(1+\\eta\\beta)}$ that  $\\rank\\left({\\Pi_{\\Vert\\cdot\\Vert_*\\leq1}[\\X-\\eta\\nabla{}f(\\X)]}\\right) \\leq r'$.\n\nMoreover, for any $\\eta >0$ and $r'\\in\\{r,\\dots,\\min\\{m,n\\}-r\\}$, it holds that \n\\begin{eqnarray}\\label{eq:goodProjRes:3}\n\\delta(\\X^*,\\eta,r'+r-1) \\geq  \\frac{\\sqrt{r}\\eta(\\mu_1-\\mu_{r'+1})}{2(1+\\eta\\beta)}.\n\\end{eqnarray}\n\n%and any $\\eta >0$, it holds for any $\\X$ satisfying $\\Vert{\\X-\\X^*}\\Vert_F  \\leq \\frac{\\sqrt{r}\\eta(\\mu_1-\\mu_{r'+1})}{2(1+\\eta\\beta)}$ that  $\\rank\\left({\\Pi_{\\Vert\\cdot\\Vert_*\\leq1}[\\X-\\eta\\nabla{}f(\\X)]}\\right) \\leq r'' = r'+r-1$.\n\\end{theorem}","sha256_utf8":"4bda9aeda6640774648516c494662406cf0481561ba57a14fcf92a9eb7d440a1"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}[projection onto the trace-norm ball]\\label{lem:traceNormProj}\nFix a parameter $\\tau >0$. Let $\\X\\in\\reals^{m\\times n}$ and consider its singular-value decomposition $\\X = \\sum_{i=1}^{\\min\\{m,n]}\\sigma_i\\u_i\\v_i^{\\top}$. If $\\Vert{\\X}\\Vert_* \\geq \\tau$, then the Euclidean projection of $\\X$ onto the trace-norm ball of radius $\\tau$ is given by\n\\begin{eqnarray}\\label{eq:traceBallProj}\n\\Pi_{\\Vert{\\cdot}\\Vert_*\\leq \\tau}[\\X] = \\sum_{i=1}^{\\min\\{m,n\\}}\\max\\{0,\\sigma_i-\\sigma\\}\\u_i\\v_i^{\\top},\n%\\Pi_{\\Vert{\\cdot}\\Vert_*\\leq 1}[\\X] = \\sum_{i=1}^{\\min\\{m,n\\}}\\max\\{0,\\sigma_i-\\sigma\\}\\u_i\\v_i^{\\top},\n\\end{eqnarray}\nwhere $\\sigma \\geq 0$ is the unique solution to the equation $\\sum_{i=1}^{\\min\\{m,n\\}}\\max\\{0,\\sigma_i-\\sigma\\} = \\tau$.\n%where $\\sigma \\geq 0$ is the unique solution to the equation $\\sum_{i=1}^{\\min\\{m,n\\}}\\max\\{0,\\sigma_i-\\sigma\\} = 1$.\n\nMoreover, there exists $r\\in\\{1,\\dots,\\min\\{m,n\\}-1\\}$ such that $\\sum_{i=1}^r\\sigma_i \\geq \\tau+r\\sigma_{r+1}$ if and only if $\\rank\\left({\\Pi_{\\Vert\\cdot\\Vert_*\\leq 1}[\\X]}\\right) \\leq r$. \n\\end{lemma}\n\n\\begin{proof}\nThe first part of the lemma is a well-known fact, see for instance \\cite{Beck17}.\nThe second part of the lemma comes from the simple observation, that if $\\sum_{i=1}^r\\sigma_i \\geq \\tau+r\\sigma_{r+1}$ for some $r$, then $\\sigma$, as defined in the lemma, must satisfy $\\sigma \\geq \\sigma_{r+1}$, in which case Eq. \\eqref{eq:traceBallProj} sets all bottom $(\\min\\{m,n\\}-r)$ components of the SVD of $\\X$ to zero, and hence the projection is of rank at most $r$. It is not hard to show using the same reasoning that the reversed direction also holds.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:eigsOfOptGrad}\n%Assume that $\\nabla{}f$ is non-zero over the unit trace-norm ball.\nLet $\\X^*\\in\\mX^*$ be any optimal solution and write its singular value decomposition as $\\X^* = \\sum_{i=1}^r\\sigma_i\\u_i\\v_i^{\\top}$. Then, the gradient vector $\\nabla{}f(\\X^*)$ admits a singular-value decomposition such that the set of pairs of vectors $\\{(-\\u_i,\\v_i)\\}_{i=1}^r$ is a set of top singular-vector pairs of $\\nabla{}f(\\X^*)$ which corresponds to the largest singular value $\\sigma_1(\\nabla{}f(\\X^*))$.\n\\end{lemma}\n\n\\begin{proof}\nFirst, note that if $\\nabla{}f(\\X^*) = \\mathbf{0}$ then the claim holds trivially. Thus, henceforth we consider the case $\\nabla{}f(\\X^*)\\neq\\mathbf{0}$.\n\nIt suffices to show that for all $i\\in\\{1,\\dots,r\\}$ it holds that $-\\u_i^{\\top}\\nabla{}f(\\X^*)\\v_i = \\sigma_1(\\nabla{}f(\\X^*))$. \n\nAssume by contradiction that for some $i\\in\\{1,\\dots,r\\}$ it holds that  $-\\u_i^{\\top}\\nabla{}f(\\X^*)\\v_i < \\sigma_1(\\nabla{}f(\\X^*))$. Let $\\u,\\v$ denote a singular vector pair corresponding to the top singular value $\\sigma_1(\\nabla{}f(\\X^*))$. Observe that for all $\\alpha\\in(0,\\sigma_i]$, the point $\\X_{\\alpha}:= \\X^* + \\alpha(-\\u\\v^{\\top}-\\u_i\\v_i^{\\top})$ is a feasible solution to Problem \\eqref{eq:optProb}, i.e., $\\Vert{\\X_{\\alpha}}\\Vert_*\\leq 1$. Moreover, it holds that\n\\begin{eqnarray*}\n(\\X_{\\alpha}-\\X^*)\\bullet\\nabla{}f(\\X^*) &=& \\alpha(-\\u\\v^{\\top}-\\u_i\\v_i^{\\top})\\bullet\\nabla{}f(\\X^*) \\\\\n&<& \\alpha(-\\sigma_1(\\nabla{}f(\\X^*)) + \\sigma_1(\\nabla{}f(\\X^*))) = 0,\n\\end{eqnarray*}\nwhich clearly contradicts the optimality of $\\X^*$.\n\\end{proof}\n\n\\begin{proof}\nThroughout the proof we assume without loss of generality that $m \\leq n$.  Fix a step-size $\\eta>0$. \n\nDenote $\\Y^* := \\X^*-\\eta\\nabla{}f(\\X^*)$ and let $\\sigma_1,\\dots,\\sigma_m$ denote the singular values of $\\Y^*$. Let us also denote by $\\mu_1,\\dots\\mu_m$ the singular values of $\\nabla{}f(\\X^*)$, and  $r^* := \\rank(\\X^*)$.\nFrom Lemma \\ref{lem:eigsOfOptGrad} we can deduce that\n\\begin{eqnarray}\\label{eq:goodProj:0}\n\\forall i\\in[r^*]: \\quad \\sigma_i &=& \\sigma_i(\\X^*) + \\eta\\mu_1; \\nonumber \\\\\n \\forall j>r^*: \\quad \\sigma_j &=& \\eta\\mu_j.\n\\end{eqnarray}\n\nFor any integer $r'\\in\\{r,\\dots,m\\}$ let us define \n\\begin{eqnarray*}\n\\xi(r') = \\sum_{i=1}^{r}\\sigma_i - r\\cdot\\sigma_{r'+1} -1.\n\\end{eqnarray*}\n\n\nSince $\\nabla{}f(\\X^*)\\neq\\mathbf{0}$, it follows that $\\sum_{i=1}^{r^*}\\sigma_i(\\X^*) =\\sum_{i=1}^{r}\\sigma_i(\\X^*) = \\Vert{\\X^*}\\Vert_*=1$, we have that\n\n\\begin{eqnarray}\\label{eq:goodProj:1}\n\\xi(r') := \\sum_{i=1}^{r}\\sigma_i - r\\cdot\\sigma_{r'+1} -1 \\underset{(a)}{=} 1+\\eta{}r\\mu_1  - r\\eta\\mu_{r'+1}  - 1= \\eta{}r(\\mu_1-\\mu_{r'+1}),\n\\end{eqnarray}\nwhere (a) follows from \\eqref{eq:goodProj:0}.\n\n\nNow, given some $\\X\\in\\reals^{m\\times n}$, denote $\\Y := \\X-\\eta\\nabla{}f(\\X)$ and let $\\gamma_1,\\dots\\gamma_m$ denote the singular values of $\\Y$. It holds that\n\n\\begin{eqnarray}\\label{eq:goodProj:2}\n\\sum_{i=1}^{r}\\gamma_i &\\underset{(a)}{\\geq} &\\sum_{i=1}^{r}\\sigma_i - \\sum_{i=1}^{r}\\sigma_i(\\Y-\\Y^*) \\geq \\sum_{i=1}^{r}\\sigma_i -\\sqrt{r\\sum_{i=1}^{r}\\sigma_i^2(\\Y-\\Y^*)} \\nonumber \\\\\n&\\geq& \\sum_{i=1}^{r}\\sigma_i -\\sqrt{r\\sum_{i=1}^{m}\\sigma_i^2(\\Y-\\Y^*)} = \\sum_{i=1}^{r}\\sigma_i  -\\sqrt{r}\\Vert{\\Y-\\Y^*}\\Vert_F \\nonumber \\\\\n&=& \\sum_{i=1}^{r}\\sigma_i - \\sqrt{r}\\Vert{\\X-\\eta\\nabla{}f(\\X) - \\X^* + \\eta\\nabla{}f(\\X^*)}\\Vert_F \\nonumber \\\\\n&\\geq& \\sum_{i=1}^{r}\\sigma_i - \\sqrt{r}\\left({\\Vert{\\X-\\X^*}\\Vert_F + \\eta\\Vert{\\nabla{}f(\\X) - \\nabla{}f(\\X^*)}\\Vert_F}\\right) \\nonumber \\\\\n&\\underset{(b)}{\\geq}& \\sum_{i=1}^{r}\\sigma_i - \\sqrt{r}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F,\n\\end{eqnarray}\nwhere (a) follows from Ky Fan's inequality for the singular values, and (b) follows from the $\\beta$-smoothness of $f(\\cdot)$.\n\nAlso, similarly, using Weyl's inequality, it holds that\n\\begin{eqnarray}\\label{eq:goodProj:3}\n\\gamma_{r'+1} \\leq \\sigma_{r'+1} + \\sigma_1(\\Y-\\Y^*) \\leq \\sigma_{r'+1} + (1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F.\n\\end{eqnarray}\n\nCombining Eq. \\eqref{eq:goodProj:1}, \\eqref{eq:goodProj:2}, \\eqref{eq:goodProj:3}, we have that\n\\begin{eqnarray}\\label{eq:goodProj:4}\n\\sum_{i=1}^{r'}\\gamma_i - r'\\gamma_{r'+1} &\\geq & \\sum_{i=1}^r\\gamma_i - r\\gamma_{r'+1} \\nonumber \\\\\n& \\geq &\n\\sum_{i=1}^{r}\\sigma_i - \\sqrt{r}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F - r\\left({\\sigma_{r'+1} + (1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F}\\right) \\nonumber\\\\\n&=& \\sum_{i=1}^{r}(\\sigma_i-\\sigma_{r'+1}) -(r+\\sqrt{r})(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F \\nonumber\\\\\n&=& 1 + \\xi(r') - (r+\\sqrt{r})(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F.\n\\end{eqnarray}\n\n\nThus, it follows that if $\\X$ satisfies:\n\\begin{eqnarray*}%\\label{eq:goodProj:5}\n\\Vert{\\X-\\X^*}\\Vert_F \\leq  \\frac{\\xi(r')}{(r+\\sqrt{r})(1+\\eta\\beta)} = \\frac{\\eta(\\mu_1-\\mu_{r'+1})}{(1+1/\\sqrt{r})(1+\\eta\\beta)},\n\\end{eqnarray*}\nwe have that $\\sum_{i=1}^{r'}\\gamma_i - r'\\gamma_{r'+1} \\geq 1$, which implies via Lemma \\ref{lem:traceNormProj} that $\\rank\\left({\\projgrad_{\\eta}(\\X)}\\right)\\leq r$. This proves \\eqref{eq:goodProjRes:1}, \\eqref{eq:goodProjRes:2}.\n\nAlternatively, for any $r'' \\geq r' + r-1$, using the more general version of Weyl's inequality, we can replace Eq. \\eqref{eq:goodProj:3} with\n\\begin{eqnarray}\\label{eq:goodProj:6}\n\\gamma_{r''+1} &\\leq & \\sigma_{r'+1} + \\sigma_{r''-r'+1}(\\Y-\\Y^*) = \\sigma_{r'+1} + \\sqrt{\\sigma^2_{r''-r'+1}(\\Y-\\Y^*)} \\nonumber \\\\\n& \\leq & \\sigma_{r'+1} + \\sqrt{\\frac{1}{r''-r'+1}\\Vert{\\Y-\\Y^*}\\Vert_F^2} \\nonumber \\\\\n&\\leq &\\sigma_{r'+1} + \\frac{1}{\\sqrt{r''-r'+1}}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F.\n\\end{eqnarray}\n\nThus, similarly to Eq. \\eqref{eq:goodProj:4}, but replacing Eq. \\eqref{eq:goodProj:3} with Eq. \\eqref{eq:goodProj:4}, we obtain\n\\begin{eqnarray*}\n\\sum_{i=1}^{r''}\\gamma_i - r''\\gamma_{r''+1} &\\geq &\\sum_{i=1}^{r}\\gamma_i - r\\gamma_{r''+1}\\geq \\sum_{i=1}^{r}\\sigma_i  - \\sqrt{r}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F \\nonumber \\\\\n&&- r\\left({\\sigma_{r'+1} + \\frac{1}{\\sqrt{r''-r'+1}}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F}\\right) \\nonumber \\\\\n&=& \\sum_{i=1}^{r}(\\sigma_i-\\sigma_{r'+1}) - \\left({\\sqrt{r} + \\frac{r}{\\sqrt{r''-r'+1}}}\\right)(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F \\nonumber \\\\\n&=& 1 + \\xi(r') - \\left({\\sqrt{r} + \\frac{r}{\\sqrt{r''-r'+1}}}\\right)(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F.\n\\end{eqnarray*}\n\nIn particular, for $r'' = r'+r-1$, we obtain\n\\begin{eqnarray*}\n\\sum_{i=1}^{r''}\\gamma_i - r''\\gamma_{r''+1} \\geq 1 + \\xi(r') - 2\\sqrt{r}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F.\n\\end{eqnarray*}\n\nThus, it follows that if $\\X$ satisfies:\n\\begin{eqnarray*}\n\\Vert{\\X-\\X^*}\\Vert_F \\leq \\frac{\\xi(r')}{2\\sqrt{r}(1+\\eta\\beta)} = \\frac{r\\eta(\\mu_1-\\mu_{r'+1})}{2\\sqrt{r}(1+\\eta\\beta)} = \\frac{\\sqrt{r}\\eta(\\mu_1-\\mu_{r'+1})}{2(1+\\eta\\beta)},\n\\end{eqnarray*}\nwe have that  $\\rank\\left({\\projgrad_{\\eta}(\\X)}\\right)\\leq r''$, which proves \\eqref{eq:goodProjRes:3}.\n\n\\end{proof}","units":[{"start":21349,"end":22420,"start_line":235,"end_line":245,"raw_tex":"\\begin{lemma}[projection onto the trace-norm ball]\\label{lem:traceNormProj}\nFix a parameter $\\tau >0$. Let $\\X\\in\\reals^{m\\times n}$ and consider its singular-value decomposition $\\X = \\sum_{i=1}^{\\min\\{m,n]}\\sigma_i\\u_i\\v_i^{\\top}$. If $\\Vert{\\X}\\Vert_* \\geq \\tau$, then the Euclidean projection of $\\X$ onto the trace-norm ball of radius $\\tau$ is given by\n\\begin{eqnarray}\\label{eq:traceBallProj}\n\\Pi_{\\Vert{\\cdot}\\Vert_*\\leq \\tau}[\\X] = \\sum_{i=1}^{\\min\\{m,n\\}}\\max\\{0,\\sigma_i-\\sigma\\}\\u_i\\v_i^{\\top},\n%\\Pi_{\\Vert{\\cdot}\\Vert_*\\leq 1}[\\X] = \\sum_{i=1}^{\\min\\{m,n\\}}\\max\\{0,\\sigma_i-\\sigma\\}\\u_i\\v_i^{\\top},\n\\end{eqnarray}\nwhere $\\sigma \\geq 0$ is the unique solution to the equation $\\sum_{i=1}^{\\min\\{m,n\\}}\\max\\{0,\\sigma_i-\\sigma\\} = \\tau$.\n%where $\\sigma \\geq 0$ is the unique solution to the equation $\\sum_{i=1}^{\\min\\{m,n\\}}\\max\\{0,\\sigma_i-\\sigma\\} = 1$.\n\nMoreover, there exists $r\\in\\{1,\\dots,\\min\\{m,n\\}-1\\}$ such that $\\sum_{i=1}^r\\sigma_i \\geq \\tau+r\\sigma_{r+1}$ if and only if $\\rank\\left({\\Pi_{\\Vert\\cdot\\Vert_*\\leq 1}[\\X]}\\right) \\leq r$. \n\\end{lemma}","sha256_utf8":"864678426bb56eca6bc7034029e623cf5ca816ca9a8cef51e69e7c7365cb0a26"},{"start":22421,"end":22995,"start_line":246,"end_line":249,"raw_tex":"\\begin{proof}\nThe first part of the lemma is a well-known fact, see for instance \\cite{Beck17}.\nThe second part of the lemma comes from the simple observation, that if $\\sum_{i=1}^r\\sigma_i \\geq \\tau+r\\sigma_{r+1}$ for some $r$, then $\\sigma$, as defined in the lemma, must satisfy $\\sigma \\geq \\sigma_{r+1}$, in which case Eq. \\eqref{eq:traceBallProj} sets all bottom $(\\min\\{m,n\\}-r)$ components of the SVD of $\\X$ to zero, and hence the projection is of rank at most $r$. It is not hard to show using the same reasoning that the reversed direction also holds.\n\\end{proof}","sha256_utf8":"9c5ee03d5e187633941076c9180a36f2023b2e5ab6238d82fa532f3898eb9f7b"},{"start":23269,"end":23805,"start_line":253,"end_line":256,"raw_tex":"\\begin{lemma}\\label{lem:eigsOfOptGrad}\n%Assume that $\\nabla{}f$ is non-zero over the unit trace-norm ball.\nLet $\\X^*\\in\\mX^*$ be any optimal solution and write its singular value decomposition as $\\X^* = \\sum_{i=1}^r\\sigma_i\\u_i\\v_i^{\\top}$. Then, the gradient vector $\\nabla{}f(\\X^*)$ admits a singular-value decomposition such that the set of pairs of vectors $\\{(-\\u_i,\\v_i)\\}_{i=1}^r$ is a set of top singular-vector pairs of $\\nabla{}f(\\X^*)$ which corresponds to the largest singular value $\\sigma_1(\\nabla{}f(\\X^*))$.\n\\end{lemma}","sha256_utf8":"9591b4be823ba5d3a480af885fb1072f3a25171e586b327fdb6af74e3240ff38"},{"start":23806,"end":24865,"start_line":257,"end_line":268,"raw_tex":"\\begin{proof}\nFirst, note that if $\\nabla{}f(\\X^*) = \\mathbf{0}$ then the claim holds trivially. Thus, henceforth we consider the case $\\nabla{}f(\\X^*)\\neq\\mathbf{0}$.\n\nIt suffices to show that for all $i\\in\\{1,\\dots,r\\}$ it holds that $-\\u_i^{\\top}\\nabla{}f(\\X^*)\\v_i = \\sigma_1(\\nabla{}f(\\X^*))$. \n\nAssume by contradiction that for some $i\\in\\{1,\\dots,r\\}$ it holds that  $-\\u_i^{\\top}\\nabla{}f(\\X^*)\\v_i < \\sigma_1(\\nabla{}f(\\X^*))$. Let $\\u,\\v$ denote a singular vector pair corresponding to the top singular value $\\sigma_1(\\nabla{}f(\\X^*))$. Observe that for all $\\alpha\\in(0,\\sigma_i]$, the point $\\X_{\\alpha}:= \\X^* + \\alpha(-\\u\\v^{\\top}-\\u_i\\v_i^{\\top})$ is a feasible solution to Problem \\eqref{eq:optProb}, i.e., $\\Vert{\\X_{\\alpha}}\\Vert_*\\leq 1$. Moreover, it holds that\n\\begin{eqnarray*}\n(\\X_{\\alpha}-\\X^*)\\bullet\\nabla{}f(\\X^*) &=& \\alpha(-\\u\\v^{\\top}-\\u_i\\v_i^{\\top})\\bullet\\nabla{}f(\\X^*) \\\\\n&<& \\alpha(-\\sigma_1(\\nabla{}f(\\X^*)) + \\sigma_1(\\nabla{}f(\\X^*))) = 0,\n\\end{eqnarray*}\nwhich clearly contradicts the optimality of $\\X^*$.\n\\end{proof}","sha256_utf8":"a9e6f71990547febb5edc4b68a8e27e892cdc8f1f62d04b1084c3030fa22ec78"},{"start":37382,"end":42550,"start_line":447,"end_line":529,"raw_tex":"\\begin{proof}\nThroughout the proof we assume without loss of generality that $m \\leq n$.  Fix a step-size $\\eta>0$. \n\nDenote $\\Y^* := \\X^*-\\eta\\nabla{}f(\\X^*)$ and let $\\sigma_1,\\dots,\\sigma_m$ denote the singular values of $\\Y^*$. Let us also denote by $\\mu_1,\\dots\\mu_m$ the singular values of $\\nabla{}f(\\X^*)$, and  $r^* := \\rank(\\X^*)$.\nFrom Lemma \\ref{lem:eigsOfOptGrad} we can deduce that\n\\begin{eqnarray}\\label{eq:goodProj:0}\n\\forall i\\in[r^*]: \\quad \\sigma_i &=& \\sigma_i(\\X^*) + \\eta\\mu_1; \\nonumber \\\\\n \\forall j>r^*: \\quad \\sigma_j &=& \\eta\\mu_j.\n\\end{eqnarray}\n\nFor any integer $r'\\in\\{r,\\dots,m\\}$ let us define \n\\begin{eqnarray*}\n\\xi(r') = \\sum_{i=1}^{r}\\sigma_i - r\\cdot\\sigma_{r'+1} -1.\n\\end{eqnarray*}\n\n\nSince $\\nabla{}f(\\X^*)\\neq\\mathbf{0}$, it follows that $\\sum_{i=1}^{r^*}\\sigma_i(\\X^*) =\\sum_{i=1}^{r}\\sigma_i(\\X^*) = \\Vert{\\X^*}\\Vert_*=1$, we have that\n\n\\begin{eqnarray}\\label{eq:goodProj:1}\n\\xi(r') := \\sum_{i=1}^{r}\\sigma_i - r\\cdot\\sigma_{r'+1} -1 \\underset{(a)}{=} 1+\\eta{}r\\mu_1  - r\\eta\\mu_{r'+1}  - 1= \\eta{}r(\\mu_1-\\mu_{r'+1}),\n\\end{eqnarray}\nwhere (a) follows from \\eqref{eq:goodProj:0}.\n\n\nNow, given some $\\X\\in\\reals^{m\\times n}$, denote $\\Y := \\X-\\eta\\nabla{}f(\\X)$ and let $\\gamma_1,\\dots\\gamma_m$ denote the singular values of $\\Y$. It holds that\n\n\\begin{eqnarray}\\label{eq:goodProj:2}\n\\sum_{i=1}^{r}\\gamma_i &\\underset{(a)}{\\geq} &\\sum_{i=1}^{r}\\sigma_i - \\sum_{i=1}^{r}\\sigma_i(\\Y-\\Y^*) \\geq \\sum_{i=1}^{r}\\sigma_i -\\sqrt{r\\sum_{i=1}^{r}\\sigma_i^2(\\Y-\\Y^*)} \\nonumber \\\\\n&\\geq& \\sum_{i=1}^{r}\\sigma_i -\\sqrt{r\\sum_{i=1}^{m}\\sigma_i^2(\\Y-\\Y^*)} = \\sum_{i=1}^{r}\\sigma_i  -\\sqrt{r}\\Vert{\\Y-\\Y^*}\\Vert_F \\nonumber \\\\\n&=& \\sum_{i=1}^{r}\\sigma_i - \\sqrt{r}\\Vert{\\X-\\eta\\nabla{}f(\\X) - \\X^* + \\eta\\nabla{}f(\\X^*)}\\Vert_F \\nonumber \\\\\n&\\geq& \\sum_{i=1}^{r}\\sigma_i - \\sqrt{r}\\left({\\Vert{\\X-\\X^*}\\Vert_F + \\eta\\Vert{\\nabla{}f(\\X) - \\nabla{}f(\\X^*)}\\Vert_F}\\right) \\nonumber \\\\\n&\\underset{(b)}{\\geq}& \\sum_{i=1}^{r}\\sigma_i - \\sqrt{r}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F,\n\\end{eqnarray}\nwhere (a) follows from Ky Fan's inequality for the singular values, and (b) follows from the $\\beta$-smoothness of $f(\\cdot)$.\n\nAlso, similarly, using Weyl's inequality, it holds that\n\\begin{eqnarray}\\label{eq:goodProj:3}\n\\gamma_{r'+1} \\leq \\sigma_{r'+1} + \\sigma_1(\\Y-\\Y^*) \\leq \\sigma_{r'+1} + (1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F.\n\\end{eqnarray}\n\nCombining Eq. \\eqref{eq:goodProj:1}, \\eqref{eq:goodProj:2}, \\eqref{eq:goodProj:3}, we have that\n\\begin{eqnarray}\\label{eq:goodProj:4}\n\\sum_{i=1}^{r'}\\gamma_i - r'\\gamma_{r'+1} &\\geq & \\sum_{i=1}^r\\gamma_i - r\\gamma_{r'+1} \\nonumber \\\\\n& \\geq &\n\\sum_{i=1}^{r}\\sigma_i - \\sqrt{r}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F - r\\left({\\sigma_{r'+1} + (1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F}\\right) \\nonumber\\\\\n&=& \\sum_{i=1}^{r}(\\sigma_i-\\sigma_{r'+1}) -(r+\\sqrt{r})(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F \\nonumber\\\\\n&=& 1 + \\xi(r') - (r+\\sqrt{r})(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F.\n\\end{eqnarray}\n\n\nThus, it follows that if $\\X$ satisfies:\n\\begin{eqnarray*}%\\label{eq:goodProj:5}\n\\Vert{\\X-\\X^*}\\Vert_F \\leq  \\frac{\\xi(r')}{(r+\\sqrt{r})(1+\\eta\\beta)} = \\frac{\\eta(\\mu_1-\\mu_{r'+1})}{(1+1/\\sqrt{r})(1+\\eta\\beta)},\n\\end{eqnarray*}\nwe have that $\\sum_{i=1}^{r'}\\gamma_i - r'\\gamma_{r'+1} \\geq 1$, which implies via Lemma \\ref{lem:traceNormProj} that $\\rank\\left({\\projgrad_{\\eta}(\\X)}\\right)\\leq r$. This proves \\eqref{eq:goodProjRes:1}, \\eqref{eq:goodProjRes:2}.\n\nAlternatively, for any $r'' \\geq r' + r-1$, using the more general version of Weyl's inequality, we can replace Eq. \\eqref{eq:goodProj:3} with\n\\begin{eqnarray}\\label{eq:goodProj:6}\n\\gamma_{r''+1} &\\leq & \\sigma_{r'+1} + \\sigma_{r''-r'+1}(\\Y-\\Y^*) = \\sigma_{r'+1} + \\sqrt{\\sigma^2_{r''-r'+1}(\\Y-\\Y^*)} \\nonumber \\\\\n& \\leq & \\sigma_{r'+1} + \\sqrt{\\frac{1}{r''-r'+1}\\Vert{\\Y-\\Y^*}\\Vert_F^2} \\nonumber \\\\\n&\\leq &\\sigma_{r'+1} + \\frac{1}{\\sqrt{r''-r'+1}}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F.\n\\end{eqnarray}\n\nThus, similarly to Eq. \\eqref{eq:goodProj:4}, but replacing Eq. \\eqref{eq:goodProj:3} with Eq. \\eqref{eq:goodProj:4}, we obtain\n\\begin{eqnarray*}\n\\sum_{i=1}^{r''}\\gamma_i - r''\\gamma_{r''+1} &\\geq &\\sum_{i=1}^{r}\\gamma_i - r\\gamma_{r''+1}\\geq \\sum_{i=1}^{r}\\sigma_i  - \\sqrt{r}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F \\nonumber \\\\\n&&- r\\left({\\sigma_{r'+1} + \\frac{1}{\\sqrt{r''-r'+1}}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F}\\right) \\nonumber \\\\\n&=& \\sum_{i=1}^{r}(\\sigma_i-\\sigma_{r'+1}) - \\left({\\sqrt{r} + \\frac{r}{\\sqrt{r''-r'+1}}}\\right)(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F \\nonumber \\\\\n&=& 1 + \\xi(r') - \\left({\\sqrt{r} + \\frac{r}{\\sqrt{r''-r'+1}}}\\right)(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F.\n\\end{eqnarray*}\n\nIn particular, for $r'' = r'+r-1$, we obtain\n\\begin{eqnarray*}\n\\sum_{i=1}^{r''}\\gamma_i - r''\\gamma_{r''+1} \\geq 1 + \\xi(r') - 2\\sqrt{r}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F.\n\\end{eqnarray*}\n\nThus, it follows that if $\\X$ satisfies:\n\\begin{eqnarray*}\n\\Vert{\\X-\\X^*}\\Vert_F \\leq \\frac{\\xi(r')}{2\\sqrt{r}(1+\\eta\\beta)} = \\frac{r\\eta(\\mu_1-\\mu_{r'+1})}{2\\sqrt{r}(1+\\eta\\beta)} = \\frac{\\sqrt{r}\\eta(\\mu_1-\\mu_{r'+1})}{2(1+\\eta\\beta)},\n\\end{eqnarray*}\nwe have that  $\\rank\\left({\\projgrad_{\\eta}(\\X)}\\right)\\leq r''$, which proves \\eqref{eq:goodProjRes:3}.\n\n\\end{proof}","sha256_utf8":"f5774c5e1d20c8e3107dbdba30211c10e6d069774fdd66c62962d3a9446f21aa"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"迹范数球优化问题","source_units":[{"start":4695,"end":4780,"start_line":114,"end_line":116,"raw_tex":"\\begin{eqnarray}\\label{eq:optProb}\n\\min_{\\Vert{\\X}\\Vert_*\\leq 1}f(\\X),\n\\end{eqnarray}","sha256_utf8":"d7f43653e83b6edbb293af2cfd0110ac2f43fce9f1e023ff3a0bdc60816f347b"},{"start":4781,"end":4987,"start_line":117,"end_line":117,"raw_tex":"where $f:\\reals^{m\\times n}\\rightarrow\\reals$ is convex and $\\beta$-smooth (i.e., gradient-Lipschitz), and $\\Vert{\\cdot}\\Vert_*$ denotes the trace-norm, i.e., sum of singular values (aka the nuclear norm).\n","sha256_utf8":"d29dd6c29319151f29763f6f034f81cbc95b4343d8edd0aa0a646ec7f8893cb8"}]},{"id":"S002","semantic_type":"notation","kind":"notation","title":"矩阵谱与最优解记号","source_units":[{"start":18790,"end":19659,"start_line":210,"end_line":212,"raw_tex":"We begin with introducing some notation. For a positive integer $n$ we let $[n]$ denote the set $\\{1,2,\\dots,n\\}$.\nWe let $\\A\\bullet\\B$ denote the standard inner product for matrices, i.e., $\\A\\bullet\\B = \\trace(\\A^{\\top}\\B)$. For a real matrix $\\A$, we let $\\sigma_i(\\A)$ denote its $i$th largest singular value (including multiplicities), and we let $\\#\\sigma_i(\\A)$ denote the multiplicity of the $i$th largest singular value. Similarly, for a real symmetric matrix $\\A$, we let $\\lambda_i(\\A)$ denote its $i$th largest (signed) eigenvalue, and we let $\\#\\lambda_i(\\A)$ denote the multiplicity of the $i$the largest eigenvalue.\nWe denote by $\\mX^*$ the set of optimal solutions to Problem \\eqref{eq:optProb}, and by $f^*$ the corresponding optimal value. %For any feasible $\\X$, we define the distance $\\dist(\\X,\\mX^*) := \\min_{\\X^*\\in\\mX^*}\\Vert{\\X-\\X^*}\\Vert_F$. \n","sha256_utf8":"08fb2e95ba91f7e5684226355c73034c4aca043fee0412928898f2678a5db14b"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"投影梯度映射","source_units":[{"start":19660,"end":20077,"start_line":214,"end_line":218,"raw_tex":"For any $\\X\\in\\reals^{m\\times n}$, step-size $\\eta$ and radius $\\tau$ we denote the projected gradient mapping w.r.t. the trace-norm ball of radius $\\tau$:\n\\begin{eqnarray*}\n\\projgrad_{\\eta,\\tau}(\\X) := \\Pi_{\\Vert\\cdot\\Vert_*\\leq\\tau}\\left[{\\X-\\eta\\nabla{}f(\\X)}\\right].\n\\end{eqnarray*}\nWhen $\\tau=1$, i.e., we consider the unit trace-norm ball, we will omit the $\\tau$ subscript and simply write $\\projgrad_{\\eta}$.\n","sha256_utf8":"78379bee05e4045a2d1a4fadf6d074a835ace39d094ca3d29f109f7d234d165f"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"低秩邻域半径","source_units":[{"start":20078,"end":20718,"start_line":220,"end_line":224,"raw_tex":"Given an optimal solution $\\X^*\\in\\mX^*$, a step-size $\\eta >0$, and an integer $r$ in  $\\{\\rank(\\X^*),\\dots\\min\\{m,n\\}\\}$, we let $\\delta(\\X^*,\\eta,r)$ denote the radius of the largest Euclidean ball centered at $\\X^*$, such that for all $\\X$ in the ball it holds that $\\rank\\left({\\projgrad_{\\eta}(\\X)}\\right) \\leq r$. Or equivalently, $\\delta(\\X^*,\\eta,r)$ is the solution to the optimization problem\n\\begin{eqnarray*}\n\\sup\\delta \\geq 0 \\quad \\textrm{s.t.} \\quad \\forall\\X\\in\\ball(\\X^*,\\delta):\\rank\\left({\\projgrad_{\\eta}(\\X)}\\right)\\leq r,\n\\end{eqnarray*}\nwhere $\\ball(\\X,R)$ denotes the Euclidean ball of radius $R$ centered at $\\X$.\n","sha256_utf8":"ba11dc4c659c514fd9c24d8add124fd60ed3a0bf19e68eb8032ee290368f5568"}]},{"id":"S005","semantic_type":"theorem","kind":"theorem","title":"谱隙控制低秩邻域","source_units":[{"start":35592,"end":37380,"start_line":424,"end_line":445,"raw_tex":"\\begin{theorem}\\label{thm:goodProj}\nLet $f:\\reals^{m\\times n}\\rightarrow\\reals$ be $\\beta$-smooth and convex. Assume $\\nabla{}f$ is non-zero over the unit trace-norm ball and fix some $\\X^*\\in\\mX^*$. Let $r$ denote the multiplicity of $\\sigma_1(\\nabla{}f(\\X^*))$, and let $\\mu_1,\\mu_2,\\dots,\\mu_{\\min\\{m,n\\}}$ denote the singular values of $\\nabla{}f(\\X^*)$ (including multiplicities).  Then, for any $\\eta >0$ it holds that\n\\begin{eqnarray}\\label{eq:goodProjRes:1}\n\\delta(\\X^*,\\eta,r) \\geq  \\frac{\\eta{}(\\mu_1-\\mu_{r+1})}{(1+1/\\sqrt{r})(1+\\eta\\beta)}.\n\\end{eqnarray}\n\n%Then, for any $\\eta >0$ and for any $\\X$ (not necessarily feasible) satisfying $\\Vert{\\X-\\X^*}\\Vert_F \\leq \\delta(\\X^*,\\eta,r) :=  \\frac{\\eta{}(\\mu_1-\\mu_{r+1})}{(1+1/\\sqrt{r})(1+\\eta\\beta)}$ it holds that $\\rank\\left({\\Pi_{\\Vert\\cdot\\Vert_*\\leq1}[\\X-\\eta\\nabla{}f(\\X)]}\\right) \\leq r$.\n\nMore generally, for any $\\eta>0$ and $r'\\in\\{r,\\dots,\\min\\{m,n\\}-1\\}$, it holds that \n\\begin{eqnarray}\\label{eq:goodProjRes:2}\n\\delta(\\X^*,\\eta,r') \\geq  \\frac{\\eta(\\mu_1-\\mu_{r'+1})}{(1+1/\\sqrt{r})(1+\\eta\\beta)}.\n\\end{eqnarray}\n\n%for any $\\X$ satisfying $\\Vert{\\X-\\X^*}\\Vert_F  \\leq \\delta(\\X^*,\\eta,r') :=  \\frac{\\eta(\\mu_1-\\mu_{r'+1})}{(1+1/\\sqrt{r})(1+\\eta\\beta)}$ that  $\\rank\\left({\\Pi_{\\Vert\\cdot\\Vert_*\\leq1}[\\X-\\eta\\nabla{}f(\\X)]}\\right) \\leq r'$.\n\nMoreover, for any $\\eta >0$ and $r'\\in\\{r,\\dots,\\min\\{m,n\\}-r\\}$, it holds that \n\\begin{eqnarray}\\label{eq:goodProjRes:3}\n\\delta(\\X^*,\\eta,r'+r-1) \\geq  \\frac{\\sqrt{r}\\eta(\\mu_1-\\mu_{r'+1})}{2(1+\\eta\\beta)}.\n\\end{eqnarray}\n\n%and any $\\eta >0$, it holds for any $\\X$ satisfying $\\Vert{\\X-\\X^*}\\Vert_F  \\leq \\frac{\\sqrt{r}\\eta(\\mu_1-\\mu_{r'+1})}{2(1+\\eta\\beta)}$ that  $\\rank\\left({\\Pi_{\\Vert\\cdot\\Vert_*\\leq1}[\\X-\\eta\\nabla{}f(\\X)]}\\right) \\leq r'' = r'+r-1$.\n\\end{theorem}","sha256_utf8":"4bda9aeda6640774648516c494662406cf0481561ba57a14fcf92a9eb7d440a1"}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","title":"光滑凸性与非零梯度","source_units":[{"start":35628,"end":36017,"start_line":425,"end_line":425,"raw_tex":"Let $f:\\reals^{m\\times n}\\rightarrow\\reals$ be $\\beta$-smooth and convex. Assume $\\nabla{}f$ is non-zero over the unit trace-norm ball and fix some $\\X^*\\in\\mX^*$. Let $r$ denote the multiplicity of $\\sigma_1(\\nabla{}f(\\X^*))$, and let $\\mu_1,\\mu_2,\\dots,\\mu_{\\min\\{m,n\\}}$ denote the singular values of $\\nabla{}f(\\X^*)$ (including multiplicities).  Then, for any $\\eta >0$ it holds that\n","sha256_utf8":"ed967b7f37816e0db74552eb889eec091b152ab7a12e638da58df7d3955bcae6"}]},{"id":"S007","semantic_type":"proof","kind":"proof","title":"低秩邻域定理证明","source_units":[{"start":37382,"end":42550,"start_line":447,"end_line":529,"raw_tex":"\\begin{proof}\nThroughout the proof we assume without loss of generality that $m \\leq n$.  Fix a step-size $\\eta>0$. \n\nDenote $\\Y^* := \\X^*-\\eta\\nabla{}f(\\X^*)$ and let $\\sigma_1,\\dots,\\sigma_m$ denote the singular values of $\\Y^*$. Let us also denote by $\\mu_1,\\dots\\mu_m$ the singular values of $\\nabla{}f(\\X^*)$, and  $r^* := \\rank(\\X^*)$.\nFrom Lemma \\ref{lem:eigsOfOptGrad} we can deduce that\n\\begin{eqnarray}\\label{eq:goodProj:0}\n\\forall i\\in[r^*]: \\quad \\sigma_i &=& \\sigma_i(\\X^*) + \\eta\\mu_1; \\nonumber \\\\\n \\forall j>r^*: \\quad \\sigma_j &=& \\eta\\mu_j.\n\\end{eqnarray}\n\nFor any integer $r'\\in\\{r,\\dots,m\\}$ let us define \n\\begin{eqnarray*}\n\\xi(r') = \\sum_{i=1}^{r}\\sigma_i - r\\cdot\\sigma_{r'+1} -1.\n\\end{eqnarray*}\n\n\nSince $\\nabla{}f(\\X^*)\\neq\\mathbf{0}$, it follows that $\\sum_{i=1}^{r^*}\\sigma_i(\\X^*) =\\sum_{i=1}^{r}\\sigma_i(\\X^*) = \\Vert{\\X^*}\\Vert_*=1$, we have that\n\n\\begin{eqnarray}\\label{eq:goodProj:1}\n\\xi(r') := \\sum_{i=1}^{r}\\sigma_i - r\\cdot\\sigma_{r'+1} -1 \\underset{(a)}{=} 1+\\eta{}r\\mu_1  - r\\eta\\mu_{r'+1}  - 1= \\eta{}r(\\mu_1-\\mu_{r'+1}),\n\\end{eqnarray}\nwhere (a) follows from \\eqref{eq:goodProj:0}.\n\n\nNow, given some $\\X\\in\\reals^{m\\times n}$, denote $\\Y := \\X-\\eta\\nabla{}f(\\X)$ and let $\\gamma_1,\\dots\\gamma_m$ denote the singular values of $\\Y$. It holds that\n\n\\begin{eqnarray}\\label{eq:goodProj:2}\n\\sum_{i=1}^{r}\\gamma_i &\\underset{(a)}{\\geq} &\\sum_{i=1}^{r}\\sigma_i - \\sum_{i=1}^{r}\\sigma_i(\\Y-\\Y^*) \\geq \\sum_{i=1}^{r}\\sigma_i -\\sqrt{r\\sum_{i=1}^{r}\\sigma_i^2(\\Y-\\Y^*)} \\nonumber \\\\\n&\\geq& \\sum_{i=1}^{r}\\sigma_i -\\sqrt{r\\sum_{i=1}^{m}\\sigma_i^2(\\Y-\\Y^*)} = \\sum_{i=1}^{r}\\sigma_i  -\\sqrt{r}\\Vert{\\Y-\\Y^*}\\Vert_F \\nonumber \\\\\n&=& \\sum_{i=1}^{r}\\sigma_i - \\sqrt{r}\\Vert{\\X-\\eta\\nabla{}f(\\X) - \\X^* + \\eta\\nabla{}f(\\X^*)}\\Vert_F \\nonumber \\\\\n&\\geq& \\sum_{i=1}^{r}\\sigma_i - \\sqrt{r}\\left({\\Vert{\\X-\\X^*}\\Vert_F + \\eta\\Vert{\\nabla{}f(\\X) - \\nabla{}f(\\X^*)}\\Vert_F}\\right) \\nonumber \\\\\n&\\underset{(b)}{\\geq}& \\sum_{i=1}^{r}\\sigma_i - \\sqrt{r}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F,\n\\end{eqnarray}\nwhere (a) follows from Ky Fan's inequality for the singular values, and (b) follows from the $\\beta$-smoothness of $f(\\cdot)$.\n\nAlso, similarly, using Weyl's inequality, it holds that\n\\begin{eqnarray}\\label{eq:goodProj:3}\n\\gamma_{r'+1} \\leq \\sigma_{r'+1} + \\sigma_1(\\Y-\\Y^*) \\leq \\sigma_{r'+1} + (1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F.\n\\end{eqnarray}\n\nCombining Eq. \\eqref{eq:goodProj:1}, \\eqref{eq:goodProj:2}, \\eqref{eq:goodProj:3}, we have that\n\\begin{eqnarray}\\label{eq:goodProj:4}\n\\sum_{i=1}^{r'}\\gamma_i - r'\\gamma_{r'+1} &\\geq & \\sum_{i=1}^r\\gamma_i - r\\gamma_{r'+1} \\nonumber \\\\\n& \\geq &\n\\sum_{i=1}^{r}\\sigma_i - \\sqrt{r}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F - r\\left({\\sigma_{r'+1} + (1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F}\\right) \\nonumber\\\\\n&=& \\sum_{i=1}^{r}(\\sigma_i-\\sigma_{r'+1}) -(r+\\sqrt{r})(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F \\nonumber\\\\\n&=& 1 + \\xi(r') - (r+\\sqrt{r})(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F.\n\\end{eqnarray}\n\n\nThus, it follows that if $\\X$ satisfies:\n\\begin{eqnarray*}%\\label{eq:goodProj:5}\n\\Vert{\\X-\\X^*}\\Vert_F \\leq  \\frac{\\xi(r')}{(r+\\sqrt{r})(1+\\eta\\beta)} = \\frac{\\eta(\\mu_1-\\mu_{r'+1})}{(1+1/\\sqrt{r})(1+\\eta\\beta)},\n\\end{eqnarray*}\nwe have that $\\sum_{i=1}^{r'}\\gamma_i - r'\\gamma_{r'+1} \\geq 1$, which implies via Lemma \\ref{lem:traceNormProj} that $\\rank\\left({\\projgrad_{\\eta}(\\X)}\\right)\\leq r$. This proves \\eqref{eq:goodProjRes:1}, \\eqref{eq:goodProjRes:2}.\n\nAlternatively, for any $r'' \\geq r' + r-1$, using the more general version of Weyl's inequality, we can replace Eq. \\eqref{eq:goodProj:3} with\n\\begin{eqnarray}\\label{eq:goodProj:6}\n\\gamma_{r''+1} &\\leq & \\sigma_{r'+1} + \\sigma_{r''-r'+1}(\\Y-\\Y^*) = \\sigma_{r'+1} + \\sqrt{\\sigma^2_{r''-r'+1}(\\Y-\\Y^*)} \\nonumber \\\\\n& \\leq & \\sigma_{r'+1} + \\sqrt{\\frac{1}{r''-r'+1}\\Vert{\\Y-\\Y^*}\\Vert_F^2} \\nonumber \\\\\n&\\leq &\\sigma_{r'+1} + \\frac{1}{\\sqrt{r''-r'+1}}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F.\n\\end{eqnarray}\n\nThus, similarly to Eq. \\eqref{eq:goodProj:4}, but replacing Eq. \\eqref{eq:goodProj:3} with Eq. \\eqref{eq:goodProj:4}, we obtain\n\\begin{eqnarray*}\n\\sum_{i=1}^{r''}\\gamma_i - r''\\gamma_{r''+1} &\\geq &\\sum_{i=1}^{r}\\gamma_i - r\\gamma_{r''+1}\\geq \\sum_{i=1}^{r}\\sigma_i  - \\sqrt{r}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F \\nonumber \\\\\n&&- r\\left({\\sigma_{r'+1} + \\frac{1}{\\sqrt{r''-r'+1}}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F}\\right) \\nonumber \\\\\n&=& \\sum_{i=1}^{r}(\\sigma_i-\\sigma_{r'+1}) - \\left({\\sqrt{r} + \\frac{r}{\\sqrt{r''-r'+1}}}\\right)(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F \\nonumber \\\\\n&=& 1 + \\xi(r') - \\left({\\sqrt{r} + \\frac{r}{\\sqrt{r''-r'+1}}}\\right)(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F.\n\\end{eqnarray*}\n\nIn particular, for $r'' = r'+r-1$, we obtain\n\\begin{eqnarray*}\n\\sum_{i=1}^{r''}\\gamma_i - r''\\gamma_{r''+1} \\geq 1 + \\xi(r') - 2\\sqrt{r}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F.\n\\end{eqnarray*}\n\nThus, it follows that if $\\X$ satisfies:\n\\begin{eqnarray*}\n\\Vert{\\X-\\X^*}\\Vert_F \\leq \\frac{\\xi(r')}{2\\sqrt{r}(1+\\eta\\beta)} = \\frac{r\\eta(\\mu_1-\\mu_{r'+1})}{2\\sqrt{r}(1+\\eta\\beta)} = \\frac{\\sqrt{r}\\eta(\\mu_1-\\mu_{r'+1})}{2(1+\\eta\\beta)},\n\\end{eqnarray*}\nwe have that  $\\rank\\left({\\projgrad_{\\eta}(\\X)}\\right)\\leq r''$, which proves \\eqref{eq:goodProjRes:3}.\n\n\\end{proof}","sha256_utf8":"f5774c5e1d20c8e3107dbdba30211c10e6d069774fdd66c62962d3a9446f21aa"}]},{"id":"S008","semantic_type":"proof_step","kind":"proof_step","title":"确定最优点更新的谱","source_units":[{"start":37396,"end":37956,"start_line":448,"end_line":455,"raw_tex":"Throughout the proof we assume without loss of generality that $m \\leq n$.  Fix a step-size $\\eta>0$. \n\nDenote $\\Y^* := \\X^*-\\eta\\nabla{}f(\\X^*)$ and let $\\sigma_1,\\dots,\\sigma_m$ denote the singular values of $\\Y^*$. Let us also denote by $\\mu_1,\\dots\\mu_m$ the singular values of $\\nabla{}f(\\X^*)$, and  $r^* := \\rank(\\X^*)$.\nFrom Lemma \\ref{lem:eigsOfOptGrad} we can deduce that\n\\begin{eqnarray}\\label{eq:goodProj:0}\n\\forall i\\in[r^*]: \\quad \\sigma_i &=& \\sigma_i(\\X^*) + \\eta\\mu_1; \\nonumber \\\\\n \\forall j>r^*: \\quad \\sigma_j &=& \\eta\\mu_j.\n\\end{eqnarray}\n","sha256_utf8":"900373b31d1d2f33bd14645a06847f6fcb11a7d79207670152c9d6687b733b4a"}]},{"id":"S009","semantic_type":"proof_step","kind":"proof_step","title":"计算谱隙余量","source_units":[{"start":37957,"end":38503,"start_line":457,"end_line":468,"raw_tex":"For any integer $r'\\in\\{r,\\dots,m\\}$ let us define \n\\begin{eqnarray*}\n\\xi(r') = \\sum_{i=1}^{r}\\sigma_i - r\\cdot\\sigma_{r'+1} -1.\n\\end{eqnarray*}\n\n\nSince $\\nabla{}f(\\X^*)\\neq\\mathbf{0}$, it follows that $\\sum_{i=1}^{r^*}\\sigma_i(\\X^*) =\\sum_{i=1}^{r}\\sigma_i(\\X^*) = \\Vert{\\X^*}\\Vert_*=1$, we have that\n\n\\begin{eqnarray}\\label{eq:goodProj:1}\n\\xi(r') := \\sum_{i=1}^{r}\\sigma_i - r\\cdot\\sigma_{r'+1} -1 \\underset{(a)}{=} 1+\\eta{}r\\mu_1  - r\\eta\\mu_{r'+1}  - 1= \\eta{}r(\\mu_1-\\mu_{r'+1}),\n\\end{eqnarray}\nwhere (a) follows from \\eqref{eq:goodProj:0}.\n","sha256_utf8":"638f811eff6d0c0360433accb842b1ec81e3b4d0115799f989d3135d1ee6b92f"}]},{"id":"S010","semantic_type":"proof_step","kind":"proof_step","title":"控制顶部奇异值之和","source_units":[{"start":38505,"end":39526,"start_line":471,"end_line":480,"raw_tex":"Now, given some $\\X\\in\\reals^{m\\times n}$, denote $\\Y := \\X-\\eta\\nabla{}f(\\X)$ and let $\\gamma_1,\\dots\\gamma_m$ denote the singular values of $\\Y$. It holds that\n\n\\begin{eqnarray}\\label{eq:goodProj:2}\n\\sum_{i=1}^{r}\\gamma_i &\\underset{(a)}{\\geq} &\\sum_{i=1}^{r}\\sigma_i - \\sum_{i=1}^{r}\\sigma_i(\\Y-\\Y^*) \\geq \\sum_{i=1}^{r}\\sigma_i -\\sqrt{r\\sum_{i=1}^{r}\\sigma_i^2(\\Y-\\Y^*)} \\nonumber \\\\\n&\\geq& \\sum_{i=1}^{r}\\sigma_i -\\sqrt{r\\sum_{i=1}^{m}\\sigma_i^2(\\Y-\\Y^*)} = \\sum_{i=1}^{r}\\sigma_i  -\\sqrt{r}\\Vert{\\Y-\\Y^*}\\Vert_F \\nonumber \\\\\n&=& \\sum_{i=1}^{r}\\sigma_i - \\sqrt{r}\\Vert{\\X-\\eta\\nabla{}f(\\X) - \\X^* + \\eta\\nabla{}f(\\X^*)}\\Vert_F \\nonumber \\\\\n&\\geq& \\sum_{i=1}^{r}\\sigma_i - \\sqrt{r}\\left({\\Vert{\\X-\\X^*}\\Vert_F + \\eta\\Vert{\\nabla{}f(\\X) - \\nabla{}f(\\X^*)}\\Vert_F}\\right) \\nonumber \\\\\n&\\underset{(b)}{\\geq}& \\sum_{i=1}^{r}\\sigma_i - \\sqrt{r}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F,\n\\end{eqnarray}\nwhere (a) follows from Ky Fan's inequality for the singular values, and (b) follows from the $\\beta$-smoothness of $f(\\cdot)$.\n","sha256_utf8":"8f4e817403070aa1078d4c01a4d487b60a4152ad05f994754cd13c2c9e5120d1"}]},{"id":"S011","semantic_type":"proof_step","kind":"proof_step","title":"控制尾部奇异值","source_units":[{"start":39527,"end":39746,"start_line":482,"end_line":485,"raw_tex":"Also, similarly, using Weyl's inequality, it holds that\n\\begin{eqnarray}\\label{eq:goodProj:3}\n\\gamma_{r'+1} \\leq \\sigma_{r'+1} + \\sigma_1(\\Y-\\Y^*) \\leq \\sigma_{r'+1} + (1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F.\n\\end{eqnarray}\n","sha256_utf8":"fcf326299c11dfa9fbe075d154ced42e36f0902a832152a6ad34bcd3366e2742"}]},{"id":"S012","semantic_type":"proof_step","kind":"proof_step","title":"合并投影秩判据","source_units":[{"start":39747,"end":40324,"start_line":487,"end_line":494,"raw_tex":"Combining Eq. \\eqref{eq:goodProj:1}, \\eqref{eq:goodProj:2}, \\eqref{eq:goodProj:3}, we have that\n\\begin{eqnarray}\\label{eq:goodProj:4}\n\\sum_{i=1}^{r'}\\gamma_i - r'\\gamma_{r'+1} &\\geq & \\sum_{i=1}^r\\gamma_i - r\\gamma_{r'+1} \\nonumber \\\\\n& \\geq &\n\\sum_{i=1}^{r}\\sigma_i - \\sqrt{r}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F - r\\left({\\sigma_{r'+1} + (1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F}\\right) \\nonumber\\\\\n&=& \\sum_{i=1}^{r}(\\sigma_i-\\sigma_{r'+1}) -(r+\\sqrt{r})(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F \\nonumber\\\\\n&=& 1 + \\xi(r') - (r+\\sqrt{r})(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F.\n\\end{eqnarray}\n","sha256_utf8":"b427a8b9714a1640b1bc1d04d00e8c6719c5acad7d9113ff6e29abf5cb4b3f76"}]},{"id":"S013","semantic_type":"proof_step","kind":"proof_step","title":"推出前两项半径界","source_units":[{"start":40326,"end":40787,"start_line":497,"end_line":501,"raw_tex":"Thus, it follows that if $\\X$ satisfies:\n\\begin{eqnarray*}%\\label{eq:goodProj:5}\n\\Vert{\\X-\\X^*}\\Vert_F \\leq  \\frac{\\xi(r')}{(r+\\sqrt{r})(1+\\eta\\beta)} = \\frac{\\eta(\\mu_1-\\mu_{r'+1})}{(1+1/\\sqrt{r})(1+\\eta\\beta)},\n\\end{eqnarray*}\nwe have that $\\sum_{i=1}^{r'}\\gamma_i - r'\\gamma_{r'+1} \\geq 1$, which implies via Lemma \\ref{lem:traceNormProj} that $\\rank\\left({\\projgrad_{\\eta}(\\X)}\\right)\\leq r$. This proves \\eqref{eq:goodProjRes:1}, \\eqref{eq:goodProjRes:2}.\n","sha256_utf8":"2898118328f01a0df646a1ec0aecec5a2cf8769c3ee3fb4db4af68123f841cef"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"细化增秩后的谱扰动界","source_units":[{"start":40788,"end":41288,"start_line":503,"end_line":508,"raw_tex":"Alternatively, for any $r'' \\geq r' + r-1$, using the more general version of Weyl's inequality, we can replace Eq. \\eqref{eq:goodProj:3} with\n\\begin{eqnarray}\\label{eq:goodProj:6}\n\\gamma_{r''+1} &\\leq & \\sigma_{r'+1} + \\sigma_{r''-r'+1}(\\Y-\\Y^*) = \\sigma_{r'+1} + \\sqrt{\\sigma^2_{r''-r'+1}(\\Y-\\Y^*)} \\nonumber \\\\\n& \\leq & \\sigma_{r'+1} + \\sqrt{\\frac{1}{r''-r'+1}\\Vert{\\Y-\\Y^*}\\Vert_F^2} \\nonumber \\\\\n&\\leq &\\sigma_{r'+1} + \\frac{1}{\\sqrt{r''-r'+1}}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F.\n\\end{eqnarray}\n","sha256_utf8":"6199ea96b63efddabeeeae23e2b8d055b66c48038ff226b18c3b3ef27b274b0d"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"建立增秩投影判据","source_units":[{"start":41289,"end":41988,"start_line":510,"end_line":516,"raw_tex":"Thus, similarly to Eq. \\eqref{eq:goodProj:4}, but replacing Eq. \\eqref{eq:goodProj:3} with Eq. \\eqref{eq:goodProj:4}, we obtain\n\\begin{eqnarray*}\n\\sum_{i=1}^{r''}\\gamma_i - r''\\gamma_{r''+1} &\\geq &\\sum_{i=1}^{r}\\gamma_i - r\\gamma_{r''+1}\\geq \\sum_{i=1}^{r}\\sigma_i  - \\sqrt{r}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F \\nonumber \\\\\n&&- r\\left({\\sigma_{r'+1} + \\frac{1}{\\sqrt{r''-r'+1}}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F}\\right) \\nonumber \\\\\n&=& \\sum_{i=1}^{r}(\\sigma_i-\\sigma_{r'+1}) - \\left({\\sqrt{r} + \\frac{r}{\\sqrt{r''-r'+1}}}\\right)(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F \\nonumber \\\\\n&=& 1 + \\xi(r') - \\left({\\sqrt{r} + \\frac{r}{\\sqrt{r''-r'+1}}}\\right)(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F.\n\\end{eqnarray*}\n","sha256_utf8":"6bd81c766a1a3651093be65933e288ee4f3d1d846f61cfd82e6b18a949f2504c"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"选取增秩参数","source_units":[{"start":41989,"end":42177,"start_line":518,"end_line":521,"raw_tex":"In particular, for $r'' = r'+r-1$, we obtain\n\\begin{eqnarray*}\n\\sum_{i=1}^{r''}\\gamma_i - r''\\gamma_{r''+1} \\geq 1 + \\xi(r') - 2\\sqrt{r}(1+\\eta\\beta)\\Vert{\\X-\\X^*}\\Vert_F.\n\\end{eqnarray*}\n","sha256_utf8":"5a3bca1ed65a946e6643d02f4543b9217821bf27c612b1b2fa894b552bbb11c5"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"推出第三项半径界","source_units":[{"start":42178,"end":42538,"start_line":523,"end_line":527,"raw_tex":"Thus, it follows that if $\\X$ satisfies:\n\\begin{eqnarray*}\n\\Vert{\\X-\\X^*}\\Vert_F \\leq \\frac{\\xi(r')}{2\\sqrt{r}(1+\\eta\\beta)} = \\frac{r\\eta(\\mu_1-\\mu_{r'+1})}{2\\sqrt{r}(1+\\eta\\beta)} = \\frac{\\sqrt{r}\\eta(\\mu_1-\\mu_{r'+1})}{2(1+\\eta\\beta)},\n\\end{eqnarray*}\nwe have that  $\\rank\\left({\\projgrad_{\\eta}(\\X)}\\right)\\leq r''$, which proves \\eqref{eq:goodProjRes:3}.\n","sha256_utf8":"74538e5028dd24f6637ea9e783a30ee0fde1de3883acd69874ea408c7b5eef38"}]},{"id":"S018","semantic_type":"lemma","kind":"lemma","title":"迹范数球投影秩判据","source_units":[{"start":21349,"end":22420,"start_line":235,"end_line":245,"raw_tex":"\\begin{lemma}[projection onto the trace-norm ball]\\label{lem:traceNormProj}\nFix a parameter $\\tau >0$. Let $\\X\\in\\reals^{m\\times n}$ and consider its singular-value decomposition $\\X = \\sum_{i=1}^{\\min\\{m,n]}\\sigma_i\\u_i\\v_i^{\\top}$. If $\\Vert{\\X}\\Vert_* \\geq \\tau$, then the Euclidean projection of $\\X$ onto the trace-norm ball of radius $\\tau$ is given by\n\\begin{eqnarray}\\label{eq:traceBallProj}\n\\Pi_{\\Vert{\\cdot}\\Vert_*\\leq \\tau}[\\X] = \\sum_{i=1}^{\\min\\{m,n\\}}\\max\\{0,\\sigma_i-\\sigma\\}\\u_i\\v_i^{\\top},\n%\\Pi_{\\Vert{\\cdot}\\Vert_*\\leq 1}[\\X] = \\sum_{i=1}^{\\min\\{m,n\\}}\\max\\{0,\\sigma_i-\\sigma\\}\\u_i\\v_i^{\\top},\n\\end{eqnarray}\nwhere $\\sigma \\geq 0$ is the unique solution to the equation $\\sum_{i=1}^{\\min\\{m,n\\}}\\max\\{0,\\sigma_i-\\sigma\\} = \\tau$.\n%where $\\sigma \\geq 0$ is the unique solution to the equation $\\sum_{i=1}^{\\min\\{m,n\\}}\\max\\{0,\\sigma_i-\\sigma\\} = 1$.\n\nMoreover, there exists $r\\in\\{1,\\dots,\\min\\{m,n\\}-1\\}$ such that $\\sum_{i=1}^r\\sigma_i \\geq \\tau+r\\sigma_{r+1}$ if and only if $\\rank\\left({\\Pi_{\\Vert\\cdot\\Vert_*\\leq 1}[\\X]}\\right) \\leq r$. \n\\end{lemma}","sha256_utf8":"864678426bb56eca6bc7034029e623cf5ca816ca9a8cef51e69e7c7365cb0a26"}]},{"id":"S019","semantic_type":"proof","kind":"proof","title":"投影秩判据证明","source_units":[{"start":22421,"end":22995,"start_line":246,"end_line":249,"raw_tex":"\\begin{proof}\nThe first part of the lemma is a well-known fact, see for instance \\cite{Beck17}.\nThe second part of the lemma comes from the simple observation, that if $\\sum_{i=1}^r\\sigma_i \\geq \\tau+r\\sigma_{r+1}$ for some $r$, then $\\sigma$, as defined in the lemma, must satisfy $\\sigma \\geq \\sigma_{r+1}$, in which case Eq. \\eqref{eq:traceBallProj} sets all bottom $(\\min\\{m,n\\}-r)$ components of the SVD of $\\X$ to zero, and hence the projection is of rank at most $r$. It is not hard to show using the same reasoning that the reversed direction also holds.\n\\end{proof}","sha256_utf8":"9c5ee03d5e187633941076c9180a36f2023b2e5ab6238d82fa532f3898eb9f7b"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"引用标准投影公式","source_units":[{"start":22435,"end":22517,"start_line":247,"end_line":247,"raw_tex":"The first part of the lemma is a well-known fact, see for instance \\cite{Beck17}.\n","sha256_utf8":"e8bfd5ecd56f0537d50119ab2206b0fb41d9d40402149e6fecb493903e44f7a7"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"由阈值刻画投影秩","source_units":[{"start":22517,"end":22984,"start_line":248,"end_line":248,"raw_tex":"The second part of the lemma comes from the simple observation, that if $\\sum_{i=1}^r\\sigma_i \\geq \\tau+r\\sigma_{r+1}$ for some $r$, then $\\sigma$, as defined in the lemma, must satisfy $\\sigma \\geq \\sigma_{r+1}$, in which case Eq. \\eqref{eq:traceBallProj} sets all bottom $(\\min\\{m,n\\}-r)$ components of the SVD of $\\X$ to zero, and hence the projection is of rank at most $r$. It is not hard to show using the same reasoning that the reversed direction also holds.\n","sha256_utf8":"e672b13673e72bc950c08ad875c64f331ec5a87098277cfa9ef478701c6ca97e"}]},{"id":"S022","semantic_type":"lemma","kind":"lemma","title":"最优解与梯度奇异向量","source_units":[{"start":23269,"end":23805,"start_line":253,"end_line":256,"raw_tex":"\\begin{lemma}\\label{lem:eigsOfOptGrad}\n%Assume that $\\nabla{}f$ is non-zero over the unit trace-norm ball.\nLet $\\X^*\\in\\mX^*$ be any optimal solution and write its singular value decomposition as $\\X^* = \\sum_{i=1}^r\\sigma_i\\u_i\\v_i^{\\top}$. Then, the gradient vector $\\nabla{}f(\\X^*)$ admits a singular-value decomposition such that the set of pairs of vectors $\\{(-\\u_i,\\v_i)\\}_{i=1}^r$ is a set of top singular-vector pairs of $\\nabla{}f(\\X^*)$ which corresponds to the largest singular value $\\sigma_1(\\nabla{}f(\\X^*))$.\n\\end{lemma}","sha256_utf8":"9591b4be823ba5d3a480af885fb1072f3a25171e586b327fdb6af74e3240ff38"}]},{"id":"S023","semantic_type":"proof","kind":"proof","title":"最优奇异向量关系证明","source_units":[{"start":23806,"end":24865,"start_line":257,"end_line":268,"raw_tex":"\\begin{proof}\nFirst, note that if $\\nabla{}f(\\X^*) = \\mathbf{0}$ then the claim holds trivially. Thus, henceforth we consider the case $\\nabla{}f(\\X^*)\\neq\\mathbf{0}$.\n\nIt suffices to show that for all $i\\in\\{1,\\dots,r\\}$ it holds that $-\\u_i^{\\top}\\nabla{}f(\\X^*)\\v_i = \\sigma_1(\\nabla{}f(\\X^*))$. \n\nAssume by contradiction that for some $i\\in\\{1,\\dots,r\\}$ it holds that  $-\\u_i^{\\top}\\nabla{}f(\\X^*)\\v_i < \\sigma_1(\\nabla{}f(\\X^*))$. Let $\\u,\\v$ denote a singular vector pair corresponding to the top singular value $\\sigma_1(\\nabla{}f(\\X^*))$. Observe that for all $\\alpha\\in(0,\\sigma_i]$, the point $\\X_{\\alpha}:= \\X^* + \\alpha(-\\u\\v^{\\top}-\\u_i\\v_i^{\\top})$ is a feasible solution to Problem \\eqref{eq:optProb}, i.e., $\\Vert{\\X_{\\alpha}}\\Vert_*\\leq 1$. Moreover, it holds that\n\\begin{eqnarray*}\n(\\X_{\\alpha}-\\X^*)\\bullet\\nabla{}f(\\X^*) &=& \\alpha(-\\u\\v^{\\top}-\\u_i\\v_i^{\\top})\\bullet\\nabla{}f(\\X^*) \\\\\n&<& \\alpha(-\\sigma_1(\\nabla{}f(\\X^*)) + \\sigma_1(\\nabla{}f(\\X^*))) = 0,\n\\end{eqnarray*}\nwhich clearly contradicts the optimality of $\\X^*$.\n\\end{proof}","sha256_utf8":"a9e6f71990547febb5edc4b68a8e27e892cdc8f1f62d04b1084c3030fa22ec78"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"排除零梯度并化归配对","source_units":[{"start":23820,"end":24106,"start_line":258,"end_line":260,"raw_tex":"First, note that if $\\nabla{}f(\\X^*) = \\mathbf{0}$ then the claim holds trivially. Thus, henceforth we consider the case $\\nabla{}f(\\X^*)\\neq\\mathbf{0}$.\n\nIt suffices to show that for all $i\\in\\{1,\\dots,r\\}$ it holds that $-\\u_i^{\\top}\\nabla{}f(\\X^*)\\v_i = \\sigma_1(\\nabla{}f(\\X^*))$. \n","sha256_utf8":"b07ac84fcd88130d01652251b630fa0f07c172c95f2929a4ce4091cc68ef3b14"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"构造可行下降方向反证","source_units":[{"start":24107,"end":24854,"start_line":262,"end_line":267,"raw_tex":"Assume by contradiction that for some $i\\in\\{1,\\dots,r\\}$ it holds that  $-\\u_i^{\\top}\\nabla{}f(\\X^*)\\v_i < \\sigma_1(\\nabla{}f(\\X^*))$. Let $\\u,\\v$ denote a singular vector pair corresponding to the top singular value $\\sigma_1(\\nabla{}f(\\X^*))$. Observe that for all $\\alpha\\in(0,\\sigma_i]$, the point $\\X_{\\alpha}:= \\X^* + \\alpha(-\\u\\v^{\\top}-\\u_i\\v_i^{\\top})$ is a feasible solution to Problem \\eqref{eq:optProb}, i.e., $\\Vert{\\X_{\\alpha}}\\Vert_*\\leq 1$. Moreover, it holds that\n\\begin{eqnarray*}\n(\\X_{\\alpha}-\\X^*)\\bullet\\nabla{}f(\\X^*) &=& \\alpha(-\\u\\v^{\\top}-\\u_i\\v_i^{\\top})\\bullet\\nabla{}f(\\X^*) \\\\\n&<& \\alpha(-\\sigma_1(\\nabla{}f(\\X^*)) + \\sigma_1(\\nabla{}f(\\X^*))) = 0,\n\\end{eqnarray*}\nwhich clearly contradicts the optimality of $\\X^*$.\n","sha256_utf8":"d8166f96ae7b6747631e2a0ea84e7e3e9b80d95b410fa41145f8f99bc5973561"}]},{"id":"S026","semantic_type":"external_fact","kind":"external_fact","title":"标准奇异值阈值投影","source_units":[],"declared_external":true,"declaration_text":["Euclidean projection onto a trace-norm ball preserves singular vectors and applies a common nonnegative threshold to singular values, chosen to attain the prescribed trace norm when the input is outside the ball; this is the standard fact cited to Beck17."]},{"id":"S027","semantic_type":"external_fact","kind":"external_fact","title":"凸优化一阶最优性","source_units":[],"declared_external":true,"declaration_text":["For differentiable convex f on a convex feasible set C, X* minimizes f over C iff <X-X*,∇f(X*)>≥0 for all X∈C. If C is a norm ball and ∇f(X*)≠0, then X* lies on its boundary."]},{"id":"S028","semantic_type":"external_fact","kind":"external_fact","title":"奇异值和扰动不等式","source_units":[],"declared_external":true,"declaration_text":["Ky Fan's singular-value inequality implies |Σ_{i=1}^r σ_i(A)-Σ_{i=1}^r σ_i(B)|≤Σ_{i=1}^r σ_i(A-B)."]},{"id":"S029","semantic_type":"external_fact","kind":"external_fact","title":"广义奇异值扰动界","source_units":[],"declared_external":true,"declaration_text":["Weyl's singular-value inequalities imply σ_{i+j-1}(A+B)≤σ_i(A)+σ_j(B) whenever the indices are defined, including the case j=1."]},{"id":"S030","semantic_type":"external_fact","kind":"external_fact","title":"矩阵范数与奇异值基础","source_units":[],"declared_external":true,"declaration_text":["Matrix norms satisfy their triangle inequalities; Cauchy–Schwarz applies to singular-value vectors; ||A||_F²=Σ_i σ_i(A)² and σ_j(A)≤||A||_F/√j. For unit vectors u,v, |uᵀAv|≤σ_1(A), with attainment characterized by corresponding top singular-vector pairs."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S005","edges":[{"source_id":"S001","target_id":"S025","kind":"supports"},{"source_id":"S002","target_id":"S008","kind":"supports"},{"source_id":"S003","target_id":"S008","kind":"supports"},{"source_id":"S003","target_id":"S013","kind":"supports"},{"source_id":"S003","target_id":"S017","kind":"supports"},{"source_id":"S004","target_id":"S013","kind":"supports"},{"source_id":"S004","target_id":"S017","kind":"supports"},{"source_id":"S006","target_id":"S008","kind":"supports"},{"source_id":"S006","target_id":"S009","kind":"supports"},{"source_id":"S006","target_id":"S010","kind":"supports"},{"source_id":"S006","target_id":"S011","kind":"supports"},{"source_id":"S006","target_id":"S014","kind":"supports"},{"source_id":"S007","target_id":"S005","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S010","kind":"supports"},{"source_id":"S008","target_id":"S011","kind":"supports"},{"source_id":"S008","target_id":"S014","kind":"supports"},{"source_id":"S009","target_id":"S012","kind":"supports"},{"source_id":"S009","target_id":"S015","kind":"supports"},{"source_id":"S009","target_id":"S017","kind":"supports"},{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S010","target_id":"S015","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S008","target_id":"S007","kind":"supports"},{"source_id":"S009","target_id":"S007","kind":"supports"},{"source_id":"S010","target_id":"S007","kind":"supports"},{"source_id":"S011","target_id":"S007","kind":"supports"},{"source_id":"S012","target_id":"S007","kind":"supports"},{"source_id":"S013","target_id":"S007","kind":"supports"},{"source_id":"S014","target_id":"S007","kind":"supports"},{"source_id":"S015","target_id":"S007","kind":"supports"},{"source_id":"S016","target_id":"S007","kind":"supports"},{"source_id":"S017","target_id":"S007","kind":"supports"},{"source_id":"S018","target_id":"S013","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S019","kind":"supports"},{"source_id":"S022","target_id":"S008","kind":"supports"},{"source_id":"S022","target_id":"S009","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S023","kind":"supports"},{"source_id":"S026","target_id":"S020","kind":"supports"},{"source_id":"S027","target_id":"S025","kind":"supports"},{"source_id":"S027","target_id":"S009","kind":"supports"},{"source_id":"S028","target_id":"S010","kind":"supports"},{"source_id":"S029","target_id":"S011","kind":"supports"},{"source_id":"S029","target_id":"S014","kind":"supports"},{"source_id":"S030","target_id":"S024","kind":"supports"},{"source_id":"S030","target_id":"S025","kind":"supports"},{"source_id":"S030","target_id":"S010","kind":"supports"},{"source_id":"S030","target_id":"S011","kind":"supports"},{"source_id":"S030","target_id":"S014","kind":"supports"}]}}
{"record_id":"1905.11957v2","paper":{"title":"Sample Complexity of Sample Average Approximation for Conditional Stochastic Optimization","authors":["Yifan Hu","Xin Chen","Niao He"],"arxiv_id":"1905.11957","version":"v2","journal":"SIAM Journal on Optimization","doi":"10.1137/19m1284865","abstract_url":"https://arxiv.org/abs/1905.11957v2","pdf_url":"https://arxiv.org/pdf/1905.11957v2"},"context":{"raw_tex":"\\begin{equation}\n\\label{pro:ori}\n    \\min_{x \\in \\mathcal{X}}\\; F(x):=\\EE_{\\xi}\\Big[f_\\xi\\Big({\\EE_{\\eta|\\xi}[g_\\eta(x,\\xi)]}\\Big)\\Big].\n\\end{equation}\n\nHere $\\mathcal{X}$ is the domain of the decision variable $x\\in\\RR^d$; $f_\\xi(\\cdot):\\RR^k\\to\\RR$ is a continuous cost function dependent on the random vector $\\xi$, and $g_\\eta(\\cdot,\\xi):\\RR^d\\to\\RR^k$  is a vector-valued continuous cost function dependent on both random vectors $\\xi$ and $\\eta$. The inner expectation is with respect to $\\eta$ given $\\xi$, and the outer expectation is with respect to $\\xi$. Similar to the classical stochastic optimization, we do not assume any knowledge on the underlying distribution of $\\PP(\\xi)$ nor the conditional distribution $\\PP(\\eta|\\xi)$.  Instead, we assume availability of  samples  from the distribution $\\PP(\\xi)$ and samples from the conditional distribution $\\PP(\\eta|\\xi)$ for any given $\\xi$.\n\n\n\\begin{equation}\n\\label{pro:cosaa0}\n   \\min_{x \\in \\mathcal{X}}\\; \\hat F_{nm}(x):= \\frac{1}{n}\\sum_{i=1}^n f_{\\xi_i}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{ij}}(x, \\xi_{i})\\bigg),\n\\end{equation}\n\nwhere  $\\{\\xi_i\\}_{i=1}^n$ are i.i.d. samples generated from $\\PP(\\xi)$ and $\\{\\eta_{ij}\\}_{j=1}^m$ are i.i.d. samples generated from the conditional distribution $\\PP(\\eta | \\xi_i)$ for a given outer sample $\\xi_i$.  We would like to examine the total number of samples $T=nm+n$ required for SAA~(\\ref{pro:cosaa0}) to achieve an $\\epsilon$-optimal solution to the original CSO problem~(\\ref{pro:ori}).\n\n\nFor convenience, we collect here some notations that will be used throughout the paper. We also introduce some  mathematical tools and propositions that are necessary for future discussion. For simplicity, we restrict our attention to $l_2$-norm, denoted as $\\norm{\\cdot}_2$. Similar results on sample complexity with respect to different norms can be obtained with minor modification of the analysis.\n\n\nA function $f: \\mathcal{X} \\rightarrow \\RR$ is said to be $L$-Lipschitz continuous, if there exists a constant $L>0$ such that $|f(x_1) - f(x_2)|\\leq L\\norm{x_1-x_2}_2, \\forall x_1, x_2 \\in \\mathcal{X}$. The function $f: \\mathcal{X} \\rightarrow \\RR$ is said to be $S$-Lipschitz smooth, if it is continuously differentiable and its gradient is $S$-Lipschitz continuous. This also implies that  $\\forall x_1, x_2 \\in \\mathcal{X}:|f(x_1)-f(x_2) -\\nabla f(x_2)^\\top(x_1-x_2)|  \\leq  \\frac{S}{2}\\norm{x_1-x_2}_2^2$. If a continuously differentiable function $f: \\mathcal{X} \\rightarrow \\RR$ satisfies that $\\forall x_1, x_2 \\in \\mathcal{X}$, $f(x_1)-f(x_2)- \\nabla f(x_2)^\\top(x_1-x_2) \\geq \\frac{\\mu}{2}\\norm{x_1-x_2}_2^2,$ \nthen $f$ is called $\\mu$-strongly convex when $\\mu>0$, convex when $\\mu=0$, and $\\mu$-weakly convex when $\\mu<0$.\n\n\n\\begin{defn}[H\\\"olderian error bound  condition]\n\\label{def:lineq}\nLet $f: \\mathcal{X} \\rightarrow \\RR$ be a function with compact domain $\\mathcal{X}$ and the optimal solution set $\\mathcal{X}^*$ is nonempty. $f(\\cdot)$ satisfies the $(\\mu,\\delta)$-H\\\"olderian error bound condition if there exists $\\delta \\geq 0$ and $\\mu>0$ such that \n\\begin{equation*}\n\\forall x\\in \\mathcal{X}, \\quad f(x)-\\min_{x\\in\\mathcal{X}}f(x) \\geq \\mu \\inf_{z\\in\\mathcal{X}^*}\\norm{x-z}_2^{1+\\delta}. \n\\end{equation*}\nIn particular, when $\\delta=1$, we say $f$ satisfies the quadratic growth (QG) condition. \n\\end{defn}\n\nWe denote $x^*$ and $ \\hat{x}_{nm}$ the optimal solutions to the CSO and the SAA problems, respectively. We are interested in estimating the probability of $\\hat x_{nm}$ being an $\\eps$-optimal solution to the CSO problem, namely $\\PP\\left(F(\\hat x_{nm})-F(x^*) \\leq \\eps\\right)$, for an arbitrary accuracy $\\eps>0$.\n\nThroughout the paper, we assume availability of i.i.d. samples generated from distribution $\\PP(\\xi)$ and conditional distribution $\\PP(\\eta\\given \\xi)$ for any given $\\xi$, and we make the following basic assumptions:\n\\begin{assume}\n\\label{as:est}\nWe assume that \n\\begin{enumerate}\n    \\item[(a)] The decision set $\\mathcal{X} \\subseteq \\RR^d$ has a finite diameter $D_\\mathcal{X}>0$.\n    \\item[(b)] $f_\\xi(\\cdot)$ is $L_f$-Lipschitz continuous and $g_\\eta(\\cdot,\\xi)$ is $L_g$-Lipschitz continuous for any given $\\xi$ and $\\eta$.\n    \\item[(c)] For all $x \\in\\mathcal{X}$, $f(x,\\xi)$ is Borel measurable in $\\xi$ and $g_\\eta(x,\\xi)$ is Borel measurable in $\\eta$ for all $\\xi$.\n    \\item[(d)] $\\sigma_f^2:=\\max_{x\\in\\mathcal{X}} \\Var_\\xi\\left(f_\\xi(\\EE_{\\eta|\\xi}[g_\\eta(x,\\xi)])\\right) < \\infty$.\n    \\item[(e)] $\\sigma_g^2:=\\max_{x\\in\\mathcal{X}, \\xi} \\EE_{\\eta|\\xi}\\norm{g_\\eta(x,\\xi)-{\\EE_{\\eta|\\xi}} g_\\eta(x,\\xi)}_2^2 < \\infty$.\n    \\item[(f)] $|f_\\xi(\\cdot)| \\leq M_f$, $\\|g_\\eta(\\cdot,\\xi)\\|_2\\leq M_g$ for any $\\xi$ and $\\eta$.\n\\end{enumerate}\n\\end{assume}\n\n\\begin{assume}\n\\label{as:fpl}\nThe empirical function $\\hat F_{nm}(x)$ satisfies the $(\\mu,\\delta)$-H\\\"olderian error bound condition with $\\mu>0,\\delta\\geq0$, i.e., it holds that \n\\begin{equation*}\n\\forall x\\in \\mathcal{X}, \\; \\hat F_{nm}(x)-\\min_{x\\in\\mathcal{X}}\\hat F_{nm}(x) \\geq \\mu \\inf_{z\\in\\mathcal{X}_{nm}^*}\\norm{x-z}_2^{1+\\delta}, \n\\end{equation*}\nwhere $n,m$ are any positive integers, and $\\cX_{nm}^*$ is the optimal solution set of the empirical objective function $\\hat F_{nm}(x)$ over $\\cX$. \n\\end{assume}\n\n\\begin{assume}\n\\label{as:uniquepro}\nThe empirical function $\\hat F_{nm}$ has a unique minimizer $\\hat x_{nm}$ on $\\mathcal{X}$, for any $n$ and $m$.\n\\end{assume}\n\n\\begin{equation}\n\\label{biasterm}\n    \\Delta(m):=\n\\begin{cases}\n\\frac{L_f\\sigma_{g}}{\\sqrt{m}}, &\\text{ $f_\\xi(\\cdot)$ is $L_f$ Lipschitz continuous},\\\\\n\\frac{S\\sigma_{g}^2}{2m},&\\text{$f_\\xi(\\cdot)$ is additionally $S$ Lipschitz smooth.}\n\\end{cases}\n\\end{equation}","units":[{"start":6970,"end":7121,"start_line":109,"end_line":112,"raw_tex":"\\begin{equation}\n\\label{pro:ori}\n    \\min_{x \\in \\mathcal{X}}\\; F(x):=\\EE_{\\xi}\\Big[f_\\xi\\Big({\\EE_{\\eta|\\xi}[g_\\eta(x,\\xi)]}\\Big)\\Big].\n\\end{equation}","sha256_utf8":"84f151728d7ba01b009d62660df2513090ec8ee12d170f118d49e14636bb6aaa"},{"start":7122,"end":7873,"start_line":113,"end_line":113,"raw_tex":"Here $\\mathcal{X}$ is the domain of the decision variable $x\\in\\RR^d$; $f_\\xi(\\cdot):\\RR^k\\to\\RR$ is a continuous cost function dependent on the random vector $\\xi$, and $g_\\eta(\\cdot,\\xi):\\RR^d\\to\\RR^k$  is a vector-valued continuous cost function dependent on both random vectors $\\xi$ and $\\eta$. The inner expectation is with respect to $\\eta$ given $\\xi$, and the outer expectation is with respect to $\\xi$. Similar to the classical stochastic optimization, we do not assume any knowledge on the underlying distribution of $\\PP(\\xi)$ nor the conditional distribution $\\PP(\\eta|\\xi)$.  Instead, we assume availability of  samples  from the distribution $\\PP(\\xi)$ and samples from the conditional distribution $\\PP(\\eta|\\xi)$ for any given $\\xi$.\n","sha256_utf8":"b2f9db19f6830261f19b7e6e5e6f47e3a0c8f6ed7333c2b1fc3495c5bc3623a2"},{"start":8442,"end":8635,"start_line":118,"end_line":121,"raw_tex":"\\begin{equation}\n\\label{pro:cosaa0}\n   \\min_{x \\in \\mathcal{X}}\\; \\hat F_{nm}(x):= \\frac{1}{n}\\sum_{i=1}^n f_{\\xi_i}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{ij}}(x, \\xi_{i})\\bigg),\n\\end{equation}","sha256_utf8":"a32598d1181da3557bb59eae72b93fc99724971e65d722bb60ca71c438111ac3"},{"start":8636,"end":9039,"start_line":122,"end_line":122,"raw_tex":"where  $\\{\\xi_i\\}_{i=1}^n$ are i.i.d. samples generated from $\\PP(\\xi)$ and $\\{\\eta_{ij}\\}_{j=1}^m$ are i.i.d. samples generated from the conditional distribution $\\PP(\\eta | \\xi_i)$ for a given outer sample $\\xi_i$.  We would like to examine the total number of samples $T=nm+n$ required for SAA~(\\ref{pro:cosaa0}) to achieve an $\\epsilon$-optimal solution to the original CSO problem~(\\ref{pro:ori}).\n","sha256_utf8":"1f164d90b92f5271bd87de4f3733ec9255b06deedd21ce5e8d4333a91c6c818a"},{"start":25347,"end":25749,"start_line":303,"end_line":303,"raw_tex":"For convenience, we collect here some notations that will be used throughout the paper. We also introduce some  mathematical tools and propositions that are necessary for future discussion. For simplicity, we restrict our attention to $l_2$-norm, denoted as $\\norm{\\cdot}_2$. Similar results on sample complexity with respect to different norms can be obtained with minor modification of the analysis.\n","sha256_utf8":"0cc50ef07e23c9de2a649faf3e5df1c13f3e7633b629bd75999591e1560fdf20"},{"start":26477,"end":27312,"start_line":307,"end_line":308,"raw_tex":"A function $f: \\mathcal{X} \\rightarrow \\RR$ is said to be $L$-Lipschitz continuous, if there exists a constant $L>0$ such that $|f(x_1) - f(x_2)|\\leq L\\norm{x_1-x_2}_2, \\forall x_1, x_2 \\in \\mathcal{X}$. The function $f: \\mathcal{X} \\rightarrow \\RR$ is said to be $S$-Lipschitz smooth, if it is continuously differentiable and its gradient is $S$-Lipschitz continuous. This also implies that  $\\forall x_1, x_2 \\in \\mathcal{X}:|f(x_1)-f(x_2) -\\nabla f(x_2)^\\top(x_1-x_2)|  \\leq  \\frac{S}{2}\\norm{x_1-x_2}_2^2$. If a continuously differentiable function $f: \\mathcal{X} \\rightarrow \\RR$ satisfies that $\\forall x_1, x_2 \\in \\mathcal{X}$, $f(x_1)-f(x_2)- \\nabla f(x_2)^\\top(x_1-x_2) \\geq \\frac{\\mu}{2}\\norm{x_1-x_2}_2^2,$ \nthen $f$ is called $\\mu$-strongly convex when $\\mu>0$, convex when $\\mu=0$, and $\\mu$-weakly convex when $\\mu<0$.\n","sha256_utf8":"b92121b287826c943471c2a56bbca27734f5c572fb38a17593db15a49cec3864"},{"start":27314,"end":27911,"start_line":311,"end_line":318,"raw_tex":"\\begin{defn}[H\\\"olderian error bound  condition]\n\\label{def:lineq}\nLet $f: \\mathcal{X} \\rightarrow \\RR$ be a function with compact domain $\\mathcal{X}$ and the optimal solution set $\\mathcal{X}^*$ is nonempty. $f(\\cdot)$ satisfies the $(\\mu,\\delta)$-H\\\"olderian error bound condition if there exists $\\delta \\geq 0$ and $\\mu>0$ such that \n\\begin{equation*}\n\\forall x\\in \\mathcal{X}, \\quad f(x)-\\min_{x\\in\\mathcal{X}}f(x) \\geq \\mu \\inf_{z\\in\\mathcal{X}^*}\\norm{x-z}_2^{1+\\delta}. \n\\end{equation*}\nIn particular, when $\\delta=1$, we say $f$ satisfies the quadratic growth (QG) condition. \n\\end{defn}","sha256_utf8":"8f9417890d7a04dfd4dad83a95c365df41861ff8a27ce470317323975280c74a"},{"start":30862,"end":32247,"start_line":361,"end_line":375,"raw_tex":"We denote $x^*$ and $ \\hat{x}_{nm}$ the optimal solutions to the CSO and the SAA problems, respectively. We are interested in estimating the probability of $\\hat x_{nm}$ being an $\\eps$-optimal solution to the CSO problem, namely $\\PP\\left(F(\\hat x_{nm})-F(x^*) \\leq \\eps\\right)$, for an arbitrary accuracy $\\eps>0$.\n\nThroughout the paper, we assume availability of i.i.d. samples generated from distribution $\\PP(\\xi)$ and conditional distribution $\\PP(\\eta\\given \\xi)$ for any given $\\xi$, and we make the following basic assumptions:\n\\begin{assume}\n\\label{as:est}\nWe assume that \n\\begin{enumerate}\n    \\item[(a)] The decision set $\\mathcal{X} \\subseteq \\RR^d$ has a finite diameter $D_\\mathcal{X}>0$.\n    \\item[(b)] $f_\\xi(\\cdot)$ is $L_f$-Lipschitz continuous and $g_\\eta(\\cdot,\\xi)$ is $L_g$-Lipschitz continuous for any given $\\xi$ and $\\eta$.\n    \\item[(c)] For all $x \\in\\mathcal{X}$, $f(x,\\xi)$ is Borel measurable in $\\xi$ and $g_\\eta(x,\\xi)$ is Borel measurable in $\\eta$ for all $\\xi$.\n    \\item[(d)] $\\sigma_f^2:=\\max_{x\\in\\mathcal{X}} \\Var_\\xi\\left(f_\\xi(\\EE_{\\eta|\\xi}[g_\\eta(x,\\xi)])\\right) < \\infty$.\n    \\item[(e)] $\\sigma_g^2:=\\max_{x\\in\\mathcal{X}, \\xi} \\EE_{\\eta|\\xi}\\norm{g_\\eta(x,\\xi)-{\\EE_{\\eta|\\xi}} g_\\eta(x,\\xi)}_2^2 < \\infty$.\n    \\item[(f)] $|f_\\xi(\\cdot)| \\leq M_f$, $\\|g_\\eta(\\cdot,\\xi)\\|_2\\leq M_g$ for any $\\xi$ and $\\eta$.\n\\end{enumerate}\n\\end{assume}","sha256_utf8":"eb300cab25019edc0a269f6b8b19194bca20a9b50683fb6f0bc24542e85bff8b"},{"start":46441,"end":46962,"start_line":563,"end_line":570,"raw_tex":"\\begin{assume}\n\\label{as:fpl}\nThe empirical function $\\hat F_{nm}(x)$ satisfies the $(\\mu,\\delta)$-H\\\"olderian error bound condition with $\\mu>0,\\delta\\geq0$, i.e., it holds that \n\\begin{equation*}\n\\forall x\\in \\mathcal{X}, \\; \\hat F_{nm}(x)-\\min_{x\\in\\mathcal{X}}\\hat F_{nm}(x) \\geq \\mu \\inf_{z\\in\\mathcal{X}_{nm}^*}\\norm{x-z}_2^{1+\\delta}, \n\\end{equation*}\nwhere $n,m$ are any positive integers, and $\\cX_{nm}^*$ is the optimal solution set of the empirical objective function $\\hat F_{nm}(x)$ over $\\cX$. \n\\end{assume}","sha256_utf8":"0e59b92ff2f3c0ae1b77f7007a6d0f1b8302a8c831da24802adf8ed00610a8ef"},{"start":46964,"end":47125,"start_line":572,"end_line":575,"raw_tex":"\\begin{assume}\n\\label{as:uniquepro}\nThe empirical function $\\hat F_{nm}$ has a unique minimizer $\\hat x_{nm}$ on $\\mathcal{X}$, for any $n$ and $m$.\n\\end{assume}","sha256_utf8":"8e9f1a4e220d72a5ee4fc38462c0b0c3e166d83fc45b633c7f61cc932c50d26c"},{"start":51647,"end":51912,"start_line":630,"end_line":637,"raw_tex":"\\begin{equation}\n\\label{biasterm}\n    \\Delta(m):=\n\\begin{cases}\n\\frac{L_f\\sigma_{g}}{\\sqrt{m}}, &\\text{ $f_\\xi(\\cdot)$ is $L_f$ Lipschitz continuous},\\\\\n\\frac{S\\sigma_{g}^2}{2m},&\\text{$f_\\xi(\\cdot)$ is additionally $S$ Lipschitz smooth.}\n\\end{cases}\n\\end{equation}","sha256_utf8":"77c6881ff40a100ceecb96d88f53ac52dd35038425d5970882910e9df57f609b"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{thm}[SAA under Error Bound Condition]\\label{thm:SAA_error_bound}\nUnder Assumption \\ref{as:est}, \\ref{as:fpl}, \\ref{as:uniquepro}, for any $\\eps > 0$, we have\n\\begin{equation}\n\\PP (F(\\hat x_{nm}) - F(x^*) \\geq \\eps )\n\\leq \n\\frac{1}{\\eps}\\bigg(L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}\n+\n\\frac{2L_{f}\\sigma_{g}}{\\sqrt{m}}\\bigg).\n\\end{equation}\n    \nIf additionally, $f_\\xi(\\cdot)$ is $S$-Lipschitz smooth, then we further have \n\\begin{equation}\n\\PP (F(\\hat x_{nm}) - F(x^*) \\geq \\eps )\n\\leq \n\\frac{1}{\\eps}\\bigg(L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}\n+\n\\frac{S\\sigma_{g}^2}{m}\\bigg).\n\\end{equation}\n\\end{thm}","units":[{"start":50588,"end":51226,"start_line":608,"end_line":626,"raw_tex":"\\begin{thm}[SAA under Error Bound Condition]\\label{thm:SAA_error_bound}\nUnder Assumption \\ref{as:est}, \\ref{as:fpl}, \\ref{as:uniquepro}, for any $\\eps > 0$, we have\n\\begin{equation}\n\\PP (F(\\hat x_{nm}) - F(x^*) \\geq \\eps )\n\\leq \n\\frac{1}{\\eps}\\bigg(L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}\n+\n\\frac{2L_{f}\\sigma_{g}}{\\sqrt{m}}\\bigg).\n\\end{equation}\n    \nIf additionally, $f_\\xi(\\cdot)$ is $S$-Lipschitz smooth, then we further have \n\\begin{equation}\n\\PP (F(\\hat x_{nm}) - F(x^*) \\geq \\eps )\n\\leq \n\\frac{1}{\\eps}\\bigg(L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}\n+\n\\frac{S\\sigma_{g}^2}{m}\\bigg).\n\\end{equation}\n\\end{thm}","sha256_utf8":"15e83d61cfd6d9cb9a6a44cf65f822c43fc37cecdea0d16c5e23be293b2281f6"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lm}\\label{lm:bias} \nLet $\\{\\eta_{j}\\}_{j=1}^m$ be conditional samples from $P(\\eta|\\xi)$ given $\\xi\\sim P(\\xi)$. Under Assumption \\ref{as:est}, for any fixed $x\\in\\cX$ that is independent of $\\xi$ and $\\{\\eta_{j}\\}_{j=1}^m$, it holds that,\n\\begin{equation}\n\\label{eq:lipschitz_bias}\n\\bigg|\\EE_{\\{\\xi,\\{\\eta_{j}\\}_{j=1}^m \\}}\\bigg[f_{\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{j}}(x,\\xi)\\bigg)   -   f_{\\xi}\\big(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\big)\\bigg]\\bigg| \\leq \\frac{ L_f \\sigma_{g}}{\\sqrt{m}}.\n\\end{equation}\nIf additionally, $f_\\xi(\\cdot)$ is $S$-Lipschitz smooth, we have\n\\begin{equation}\n\\label{eq:smooth_bias}\n\\bigg|\\EE_{\\{\\xi,\\{\\eta_{j}\\}_{j=1}^m\\}} \\bigg[f_{\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{j}}(x,\\xi)\\bigg)   -   f_{\\xi}\\big(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\big)\\bigg]\\bigg|  \\leq  \\frac{S\\sigma_{g}^2}{2m}.\n\\end{equation}\n\\end{lm}\n\n\\begin{proof}{}\nDefine $X_{j}: = g_{\\eta_{j}}(x,\\xi)-\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)$ and $\\bar{X}:=\\sum_{j=1}^m X_j/m$. It follows  $\\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi}[\\bar{X}]=0$ by definition, and $\\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi}[\\|\\bar{X}\\|_2^2]\\leq \\sigma_g^2/m$ by Assumption \\ref{as:est}(d).  $\\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi}  \\nabla f_{\\xi} \\big(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\big)^\\top \\big(\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)\\big)=0$ since $x$ is independent of $\\{\\eta_j\\}_{j=1}^m$. \nThe results then follow directly by invoking the Lipschitz continuity and smoothness and taking expectations. \n\\end{proof}\n\n\\begin{proof} \nRecall that $x^*$ and $\\hat x_{nm}$ are the minimizers of $F(x)$ and $\\hat F_{nm}(x)$, respectively. It's clear that $x^*$ has no randomness,  and $\\hat x_{nm}$ is a function of $\\{\\xi_i\\}_{i =1}^n, \\{\\eta_{ij}\\}_{j=1}^m$. \nWe decompose the error $F(\\hat x_{nm}) - F(x^*)$ in three terms, and analyze each term below:\n\\begin{equation*}\n\\begin{split}\nF(\\hat x_{nm}) \\!-\\! F(x^*)\n\t\t\\!=\\!\n\t\t\\underbrace{F(\\hat x_{nm})\\!-\\! \\hat F_{nm}(\\hat x_{nm})}_{:=\\Epsilon_1}+\n\t\t\\underbrace{\\hat F_{nm}(\\hat x_{nm})\\! -\\! \\hat F_{nm}(x^*)}_{:=\\Epsilon_2}+\n\t\t\\underbrace{\\hat F_{nm}(x^*)\\!- \\! F(x^*)}_{:=\\Epsilon_3}.\n\\end{split}\n\\end{equation*}\nFirst, we use a stability argument and Lemma \\ref{lm:bias} to bound $\\EE \\Epsilon_1 = \\EE [F(\\hat x_{nm})- \\hat F_{nm}(\\hat x_{nm})]$. \nDefine \n\\begin{equation}\n\\begin{aligned}\n\\label{eq:Fnmk}\n     \\hat F_{nm}^{(k)}(x) \n     := \\frac{1}{n}\\sum_{i\\not = k}^n f_{\\xi_i}\\bigg(\\frac{1}{m}\\sum_{j =1}^m g_{\\eta_{ij}}(x,\\xi_i)\\bigg) + \\frac{1}{n}f_{\\xi_k^{\\prime}}\\bigg(\\frac{1}{m}\\sum_{j =1}^m g_{\\eta_{kj}^{\\prime}}(x,\\xi_k^{\\prime})\\bigg)\n     \\end{aligned}\n\\end{equation}\nas the empirical function by replacing the $k^{th}$ outer sample $\\xi_k$ with another i.i.d outer sample $\\xi_k^{\\prime}$, and replacing the corresponding  inner samples $\\{\\eta_{kj}\\}_{j=1}^m$  with $\\{\\eta_{kj}^{\\prime}\\}_{j=1}^m$, which are sampled from the conditional distribution of $\\PP(\\eta|\\xi_k^{\\prime})$ for a given sample $\\xi_k^\\prime$. Denote $\\hat x_{nm}^{(k)} := \\argmin_{x \\in \\mathcal{X}} \\hat F_{nm}^{(k)}(x)$. We decompose $\\EE \\Epsilon_1 = \\EE [F(\\hat x_{nm})- \\hat F_{nm}(\\hat x_{nm})]$ into three terms:\n\\begin{equation}\\label{eq:decompose_term_1}\n\\begin{split}\n\\EE \\Epsilon_1 = &\\EE\\bigg[\\frac{1}{n}\\sum_{k=1}^n F(\\hat x_{nm}) -\\frac{1}{n}\\sum_{k =1}^n  f_{\\xi_k} \\bigg( \\EE_{\\eta|\\xi_k} g_\\eta(\\hat x_{nm}^{(k)},\\xi_k) \\bigg)\\bigg]\\\\\n+&\\EE \\bigg[\\frac{1}{n}\\sum_{k =1}^n  f_{\\xi_k} \\bigg( \\EE_{\\eta|\\xi_k} g_\\eta(\\hat x_{nm}^{(k)},\\xi_k) \\bigg) -\\frac{1}{n}\\sum_{k =1}^n f_{\\xi_k}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k)\\bigg)\\bigg]\\\\\n+&\\EE \\bigg[\\frac{1}{n}\\sum_{k =1}^n f_{\\xi_k}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k)\\bigg) - \\hat F_{nm}(\\hat x_{nm})\\bigg].\n\\end{split}\n\\end{equation}\nNote that $\n    \\EE [F(\\hat x_{nm})] = \\EE [F(\\hat x_{nm}^{(k)})]\n$ since $\\xi_k$ and $\\xi_k^{\\prime}$ are i.i.d, which implies that  $\\hat x_{nm}$ and $\\hat x_{nm}^{(k)}$ follow an identical distribution.\nSince $\\hat x_{nm}^{(k)}$ is independent of $\\xi_k$, \n$\n\\EE [F(\\hat x_{nm}^{(k)})] = \\EE [f_{\\xi_k}(\\EE_{\\eta|\\xi_k}g(\\hat x_{nm}^{(k)},\\xi_k))]\n$\nfor any $k$. Then the first term in (\\ref{eq:decompose_term_1}) is $0$.\nAs $\\hat x_{nm}^{(k)}$ is independent of $\\{\\eta_{kj}\\}_{j=1}^m$, the second term in (\\ref{eq:decompose_term_1}) could be bounded by Lemma \\ref{lm:bias},\nit holds\n\\begin{equation}\n\\label{eq:biasboth}\n    \\EE \\bigg[ f_{\\xi_k} \\bigg( \\EE_{\\eta|\\xi_k} g_\\eta(\\hat x_{nm}^{(k)},\\xi_k) \\bigg)  \n    -   \n    f_{\\xi_k}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k) \\bigg)\\bigg]\n    \\leq \n    \\Delta(m).\n\\end{equation}\n\\if 0\nfor $S$-Lipschitz smooth $f_\\xi(\\cdot)$, we get\n\\begin{equation*}\n    \\EE \\bigg[ f_{\\xi_k} \\bigg( \\EE_{\\eta|\\xi_k} g_\\eta(\\hat x_{nm}^{(k)},\\xi_k) \\bigg)  \n    -   \n    f_{\\xi_k}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k) \\bigg)\\bigg] \n    \\leq \n    \\frac{S\\sigma_{g}^2}{2m}.\n\\end{equation*}\n\\fi\nFor the third term in (\\ref{eq:decompose_term_1}), by definition it implies\n\\begin{equation}\n\\label{eq:sca1}\n    \\begin{aligned}\n    \\hat F_{nm}(\\hat x_{nm}^{(k)})  - \\hat F_{nm}(\\hat x_{nm}) = &\\hat F_{nm}^{(k)}(\\hat x_{nm}^{(k)})  - \\hat F_{nm}^{(k)}(\\hat x_{nm}) \\\\\n    + &\n      \\frac{1}{n}f_{\\xi_k}\\! \\bigg(\\! \\frac{1}{m}\\! \\sum_{j =1}^m\\!  g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k)\\! \\bigg) \\! \n    - \\! \\frac{1}{n}f_{\\xi_k}\\! \\bigg(\\! \\frac{1}{m}\\sum_{j =1}^m g_{\\eta_{kj}}(\\hat x_{nm},\\xi_k)\\! \\bigg)\\\\\n     + &\\frac{1}{n}f_{\\xi_k^{\\prime}}\\! \\bigg(\\! \\frac{1}{m}\\sum_{j =1}^m g_{\\eta_{kj}^{\\prime}}(\\hat x_{nm},\\xi_k^{\\prime})\\! \\bigg) \\! \n   \\!  - \\! \n   \\frac{1}{n}f_{\\xi_k^{\\prime}}\\! \\bigg(\\! \\frac{1}{m}\\sum_{j =1}^m g_{\\eta_{kj}^{\\prime}}(\\hat x_{nm}^{(k)},\\xi_k^{\\prime})\\! \\bigg).\n    \\end{aligned}\n\\end{equation}\nBy Lipschitz continuity of $f_\\xi$ and $g_\\eta$ and that $\\hat F_{nm}^{(k)}(\\hat x_{nm}^{(k)})  - \\hat F_{nm}^{(k)}(\\hat x_{nm}) \\leq 0$, it holds\n\\begin{equation}\n\\label{eq:sca2}\n\\hat F_{nm}(\\hat x_{nm}^{(k)})  - \\hat F_{nm}(\\hat x_{nm}) \\leq \\frac{2}{n}L_f L_g \\norm{\\hat x_{nm}^{(k)}-\\hat x_{nm}}_2.\n\\end{equation}\nSince $\\hat x_{nm}$ is the unique minimizer of $\\hat F_{nm}(x)$, and $\\hat F_{nm}(x)$ satisfies QG condition with parameter $\\mu$, we have\n\\begin{equation}\n\\label{eq:strong}\n    \\hat F_{nm}(\\hat x_{nm}^{(k)})  - \\hat F_{nm}(\\hat x_{nm}) \\geq \\mu \\norm{\\hat x_{nm}^{(k)}-\\hat x_{nm}}_2^{1+\\delta}.\n\\end{equation}\nCombining with (\\ref{eq:sca2}), we obtain\n\\begin{equation}\n\\label{eq:strong2}\n\\norm{\\hat x_{nm}^{(k)}-\\hat x_{nm}}_2 \\leq \\bigg(\\frac{2L_f L_g}{\\mu n }\\bigg)^{1/\\delta}.\n\\end{equation}\nBy Lipschitz continuity of $f_\\xi(\\cdot)$ and $g_\\eta(\\cdot,\\xi)$, and definition of $\\hat F_{nm}(\\hat x_{nm})$, we obtain\n\\begin{equation}\n\\label{eq:sca}\n\\begin{split}\n   \\EE \\bigg[\\frac{1}{n}\\sum_{k =1}^n f_{\\xi_k}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k)\\bigg) - \\hat F_{nm}(\\hat x_{nm})\\bigg]\n    \\leq  L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}.\n\\end{split}\n\\end{equation}\n\nCombining (\\ref{eq:decompose_term_1}), (\\ref{eq:sca}), and (\\ref{eq:biasboth}), we obtain\n\\begin{equation}\n\\label{eq:scAl}\n    \\EE \\Epsilon_1\n    \\leq \n    L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}+\\Delta(m).\n\\end{equation}\n\\if 0\nfor $L_f$-Lipschitz continuous $f_\\xi(\\cdot)$,\n\\begin{equation}\n\\label{eq:scAl}\n    \\EE \\Epsilon_1\n    \\leq \n    L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}+\\frac{L_{f}\\sigma_{g}}{\\sqrt{m}};\n\\end{equation}\nfor $S$-Lipschitz smooth $f_\\xi(\\cdot)$,\n\\begin{equation}\n\\label{eq:scAs}\n    \\EE \\Epsilon_1\n    \\leq \n    L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}+\\frac{S\\sigma_{g}^2}{2m}.\n\\end{equation}\n\\fi\n\nSecond, by optimality of $\\hat x_{nm}$ of $\\hat F_{nm}$, we have\n\\begin{equation}\n\\label{eq:scB}\n    \\EE \\Epsilon_2 = \\EE [\\hat F_{nm}(\\hat x_{nm})- \\hat F_{nm}(x^*)] \\leq 0.\n\\end{equation}\n\nNext, we bound $\\EE \\Epsilon_3$.\nDefine\n$\n\\hat{F}_n(x):=\\frac{1}{n}\\sum_{i=1}^nf_{\\xi_i}\\big(\\EE_{\\eta|\\xi_i}[g_\\eta(x,\\xi_i)]\\big).\n$\nNotice that $x^*$ is independent of $\\{\\eta_{ij}\\}_{j=1}^m$ for any $i = \\{1,\\cdots,n\\}$ and $\\EE [\\hat F_n(x^*) - F(x^*)]=0$. By Lemma \\ref{lm:bias}, it holds\n\\begin{equation}\n\\label{eq:scC}\n    \\EE \\Epsilon_3 = \\EE[ \\hat F_{nm}(x^*)-\\hat F_n(x) ]+ \\EE[\\hat F_n(x) - F(x)]\n    \\leq \n    \\Delta(m);\n\\end{equation}\n\\if 0\nfor $L_f$-Lipschitz continuous $f_\\xi(\\cdot)$,\n\\begin{equation}\n\\label{eq:scCL}\n    \\EE \\Epsilon_3 = \\EE[ \\hat F_{nm}(x^*)-\\hat F_n(x) ]+ \\EE[\\hat F_n(x) - F(x)]\n    \\leq \n    \\frac{L_{f}\\sigma_{g}}{\\sqrt{m}};\n\\end{equation}\nfor $S$-Lipschitz smooth $f_\\xi(\\cdot)$, we have\n\\begin{equation}\n\\label{eq:scCS}\n    \\EE \\Epsilon_3 = \\EE[ \\hat F_{nm}(x^*)-\\hat F_n(x) ]+ \\EE[\\hat F_n(x) - F(x)]\n    \\leq \n    \\frac{S\\sigma_{g}^2}{2m}.\n\\end{equation}\n\\fi\nCombining (\\ref{eq:scAl}), (\\ref{eq:scB}), (\\ref{eq:scC}), with Markov inequality, we obtain the desired result.\n\\end{proof}\n\n\\begin{proof}{}\nDenote $X_{j}(x): = g_{\\eta_{j}}(x,\\xi)-\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)$, and $\\{\\eta_{j}\\}_{j=1}^m=\\{\\eta_{1},\\cdots, \\eta_{m}\\}$. By Assumption \\ref{as:est},\n\\begin{equation}\n\\label{eq:mv}\n    \\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)\\bigg)=0, \\quad 0 \\leq \\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi} \\norm{\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)}_2^2 \\leq \\frac{\\sigma_{g}^2}{m}.\n\\end{equation}\nIf $f_\\xi(\\cdot)$ is $L_f$-Lipschitz continuous,\n\\begin{equation*}\n    \\bigg|\\EE f_{\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{j}}(x,\\xi)\\bigg)   -   f_{\\xi}\\bigg(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\bigg)\\bigg|\n    \\leq \n    \\EE L_f \\norm{\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)}_2 \n    \\leq \n    \\frac{ L_f \\sigma_{g}}{\\sqrt{m}}.\n\\end{equation*}\nIf $f_\\xi(\\cdot)$ is $S$-Lipschitz smooth,\n\\begin{equation*}\n\\begin{aligned}\n\t&\\bigg|f_{\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{j}}(x,\\xi)\\bigg) \n    - f_{\\xi}\\bigg(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\bigg)\n    - \\nabla f_{\\xi} \\bigg(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\bigg)^\\top \\frac{1}{m}\\sum_{j=1}^m X_{j}(x)\\bigg| \\\\\n    \\leq &\\frac{S}{2} \\norm{\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)}_2^2.\n    \\end{aligned}\n\\end{equation*}\n$\\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi}  \\nabla f_{\\xi} \\big(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\big)^\\top \\big(\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)\\big)=0$ since $x$ is independent of ${\\eta}_{j=1}^m$, we obtain,\n\\begin{equation*}\n    \\EE   f_{\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{j}}(x,\\xi)\\bigg)   -   f_{\\xi}\\bigg(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\bigg)\n    \\leq \n    \\EE \\bigg[ \\frac{S}{2} \\norm{\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)}_2^2 \\bigg] \\\\\n    \\leq  \n    \\frac{S\\sigma_{g}^2}{2m}.\n\\end{equation*}\nFor the other side, we could derive a similar lower bound. As a result, (\\ref{eq:smooth_bias}) holds.\n\\end{proof}","units":[{"start":33400,"end":34257,"start_line":389,"end_line":400,"raw_tex":"\\begin{lm}\\label{lm:bias} \nLet $\\{\\eta_{j}\\}_{j=1}^m$ be conditional samples from $P(\\eta|\\xi)$ given $\\xi\\sim P(\\xi)$. Under Assumption \\ref{as:est}, for any fixed $x\\in\\cX$ that is independent of $\\xi$ and $\\{\\eta_{j}\\}_{j=1}^m$, it holds that,\n\\begin{equation}\n\\label{eq:lipschitz_bias}\n\\bigg|\\EE_{\\{\\xi,\\{\\eta_{j}\\}_{j=1}^m \\}}\\bigg[f_{\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{j}}(x,\\xi)\\bigg)   -   f_{\\xi}\\big(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\big)\\bigg]\\bigg| \\leq \\frac{ L_f \\sigma_{g}}{\\sqrt{m}}.\n\\end{equation}\nIf additionally, $f_\\xi(\\cdot)$ is $S$-Lipschitz smooth, we have\n\\begin{equation}\n\\label{eq:smooth_bias}\n\\bigg|\\EE_{\\{\\xi,\\{\\eta_{j}\\}_{j=1}^m\\}} \\bigg[f_{\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{j}}(x,\\xi)\\bigg)   -   f_{\\xi}\\big(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\big)\\bigg]\\bigg|  \\leq  \\frac{S\\sigma_{g}^2}{2m}.\n\\end{equation}\n\\end{lm}","sha256_utf8":"00a6b94687bc62bd9b50587c4f9eb849750f198b44be00cec79dabd7ad11dca0"},{"start":34258,"end":34863,"start_line":401,"end_line":404,"raw_tex":"\\begin{proof}{}\nDefine $X_{j}: = g_{\\eta_{j}}(x,\\xi)-\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)$ and $\\bar{X}:=\\sum_{j=1}^m X_j/m$. It follows  $\\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi}[\\bar{X}]=0$ by definition, and $\\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi}[\\|\\bar{X}\\|_2^2]\\leq \\sigma_g^2/m$ by Assumption \\ref{as:est}(d).  $\\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi}  \\nabla f_{\\xi} \\big(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\big)^\\top \\big(\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)\\big)=0$ since $x$ is independent of $\\{\\eta_j\\}_{j=1}^m$. \nThe results then follow directly by invoking the Lipschitz continuity and smoothness and taking expectations. \n\\end{proof}","sha256_utf8":"a6ccee7d32d1aaab9870d97617181dac99bf183370bb275eb9bb91a0d8985c4f"},{"start":51989,"end":59400,"start_line":640,"end_line":793,"raw_tex":"\\begin{proof} \nRecall that $x^*$ and $\\hat x_{nm}$ are the minimizers of $F(x)$ and $\\hat F_{nm}(x)$, respectively. It's clear that $x^*$ has no randomness,  and $\\hat x_{nm}$ is a function of $\\{\\xi_i\\}_{i =1}^n, \\{\\eta_{ij}\\}_{j=1}^m$. \nWe decompose the error $F(\\hat x_{nm}) - F(x^*)$ in three terms, and analyze each term below:\n\\begin{equation*}\n\\begin{split}\nF(\\hat x_{nm}) \\!-\\! F(x^*)\n\t\t\\!=\\!\n\t\t\\underbrace{F(\\hat x_{nm})\\!-\\! \\hat F_{nm}(\\hat x_{nm})}_{:=\\Epsilon_1}+\n\t\t\\underbrace{\\hat F_{nm}(\\hat x_{nm})\\! -\\! \\hat F_{nm}(x^*)}_{:=\\Epsilon_2}+\n\t\t\\underbrace{\\hat F_{nm}(x^*)\\!- \\! F(x^*)}_{:=\\Epsilon_3}.\n\\end{split}\n\\end{equation*}\nFirst, we use a stability argument and Lemma \\ref{lm:bias} to bound $\\EE \\Epsilon_1 = \\EE [F(\\hat x_{nm})- \\hat F_{nm}(\\hat x_{nm})]$. \nDefine \n\\begin{equation}\n\\begin{aligned}\n\\label{eq:Fnmk}\n     \\hat F_{nm}^{(k)}(x) \n     := \\frac{1}{n}\\sum_{i\\not = k}^n f_{\\xi_i}\\bigg(\\frac{1}{m}\\sum_{j =1}^m g_{\\eta_{ij}}(x,\\xi_i)\\bigg) + \\frac{1}{n}f_{\\xi_k^{\\prime}}\\bigg(\\frac{1}{m}\\sum_{j =1}^m g_{\\eta_{kj}^{\\prime}}(x,\\xi_k^{\\prime})\\bigg)\n     \\end{aligned}\n\\end{equation}\nas the empirical function by replacing the $k^{th}$ outer sample $\\xi_k$ with another i.i.d outer sample $\\xi_k^{\\prime}$, and replacing the corresponding  inner samples $\\{\\eta_{kj}\\}_{j=1}^m$  with $\\{\\eta_{kj}^{\\prime}\\}_{j=1}^m$, which are sampled from the conditional distribution of $\\PP(\\eta|\\xi_k^{\\prime})$ for a given sample $\\xi_k^\\prime$. Denote $\\hat x_{nm}^{(k)} := \\argmin_{x \\in \\mathcal{X}} \\hat F_{nm}^{(k)}(x)$. We decompose $\\EE \\Epsilon_1 = \\EE [F(\\hat x_{nm})- \\hat F_{nm}(\\hat x_{nm})]$ into three terms:\n\\begin{equation}\\label{eq:decompose_term_1}\n\\begin{split}\n\\EE \\Epsilon_1 = &\\EE\\bigg[\\frac{1}{n}\\sum_{k=1}^n F(\\hat x_{nm}) -\\frac{1}{n}\\sum_{k =1}^n  f_{\\xi_k} \\bigg( \\EE_{\\eta|\\xi_k} g_\\eta(\\hat x_{nm}^{(k)},\\xi_k) \\bigg)\\bigg]\\\\\n+&\\EE \\bigg[\\frac{1}{n}\\sum_{k =1}^n  f_{\\xi_k} \\bigg( \\EE_{\\eta|\\xi_k} g_\\eta(\\hat x_{nm}^{(k)},\\xi_k) \\bigg) -\\frac{1}{n}\\sum_{k =1}^n f_{\\xi_k}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k)\\bigg)\\bigg]\\\\\n+&\\EE \\bigg[\\frac{1}{n}\\sum_{k =1}^n f_{\\xi_k}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k)\\bigg) - \\hat F_{nm}(\\hat x_{nm})\\bigg].\n\\end{split}\n\\end{equation}\nNote that $\n    \\EE [F(\\hat x_{nm})] = \\EE [F(\\hat x_{nm}^{(k)})]\n$ since $\\xi_k$ and $\\xi_k^{\\prime}$ are i.i.d, which implies that  $\\hat x_{nm}$ and $\\hat x_{nm}^{(k)}$ follow an identical distribution.\nSince $\\hat x_{nm}^{(k)}$ is independent of $\\xi_k$, \n$\n\\EE [F(\\hat x_{nm}^{(k)})] = \\EE [f_{\\xi_k}(\\EE_{\\eta|\\xi_k}g(\\hat x_{nm}^{(k)},\\xi_k))]\n$\nfor any $k$. Then the first term in (\\ref{eq:decompose_term_1}) is $0$.\nAs $\\hat x_{nm}^{(k)}$ is independent of $\\{\\eta_{kj}\\}_{j=1}^m$, the second term in (\\ref{eq:decompose_term_1}) could be bounded by Lemma \\ref{lm:bias},\nit holds\n\\begin{equation}\n\\label{eq:biasboth}\n    \\EE \\bigg[ f_{\\xi_k} \\bigg( \\EE_{\\eta|\\xi_k} g_\\eta(\\hat x_{nm}^{(k)},\\xi_k) \\bigg)  \n    -   \n    f_{\\xi_k}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k) \\bigg)\\bigg]\n    \\leq \n    \\Delta(m).\n\\end{equation}\n\\if 0\nfor $S$-Lipschitz smooth $f_\\xi(\\cdot)$, we get\n\\begin{equation*}\n    \\EE \\bigg[ f_{\\xi_k} \\bigg( \\EE_{\\eta|\\xi_k} g_\\eta(\\hat x_{nm}^{(k)},\\xi_k) \\bigg)  \n    -   \n    f_{\\xi_k}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k) \\bigg)\\bigg] \n    \\leq \n    \\frac{S\\sigma_{g}^2}{2m}.\n\\end{equation*}\n\\fi\nFor the third term in (\\ref{eq:decompose_term_1}), by definition it implies\n\\begin{equation}\n\\label{eq:sca1}\n    \\begin{aligned}\n    \\hat F_{nm}(\\hat x_{nm}^{(k)})  - \\hat F_{nm}(\\hat x_{nm}) = &\\hat F_{nm}^{(k)}(\\hat x_{nm}^{(k)})  - \\hat F_{nm}^{(k)}(\\hat x_{nm}) \\\\\n    + &\n      \\frac{1}{n}f_{\\xi_k}\\! \\bigg(\\! \\frac{1}{m}\\! \\sum_{j =1}^m\\!  g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k)\\! \\bigg) \\! \n    - \\! \\frac{1}{n}f_{\\xi_k}\\! \\bigg(\\! \\frac{1}{m}\\sum_{j =1}^m g_{\\eta_{kj}}(\\hat x_{nm},\\xi_k)\\! \\bigg)\\\\\n     + &\\frac{1}{n}f_{\\xi_k^{\\prime}}\\! \\bigg(\\! \\frac{1}{m}\\sum_{j =1}^m g_{\\eta_{kj}^{\\prime}}(\\hat x_{nm},\\xi_k^{\\prime})\\! \\bigg) \\! \n   \\!  - \\! \n   \\frac{1}{n}f_{\\xi_k^{\\prime}}\\! \\bigg(\\! \\frac{1}{m}\\sum_{j =1}^m g_{\\eta_{kj}^{\\prime}}(\\hat x_{nm}^{(k)},\\xi_k^{\\prime})\\! \\bigg).\n    \\end{aligned}\n\\end{equation}\nBy Lipschitz continuity of $f_\\xi$ and $g_\\eta$ and that $\\hat F_{nm}^{(k)}(\\hat x_{nm}^{(k)})  - \\hat F_{nm}^{(k)}(\\hat x_{nm}) \\leq 0$, it holds\n\\begin{equation}\n\\label{eq:sca2}\n\\hat F_{nm}(\\hat x_{nm}^{(k)})  - \\hat F_{nm}(\\hat x_{nm}) \\leq \\frac{2}{n}L_f L_g \\norm{\\hat x_{nm}^{(k)}-\\hat x_{nm}}_2.\n\\end{equation}\nSince $\\hat x_{nm}$ is the unique minimizer of $\\hat F_{nm}(x)$, and $\\hat F_{nm}(x)$ satisfies QG condition with parameter $\\mu$, we have\n\\begin{equation}\n\\label{eq:strong}\n    \\hat F_{nm}(\\hat x_{nm}^{(k)})  - \\hat F_{nm}(\\hat x_{nm}) \\geq \\mu \\norm{\\hat x_{nm}^{(k)}-\\hat x_{nm}}_2^{1+\\delta}.\n\\end{equation}\nCombining with (\\ref{eq:sca2}), we obtain\n\\begin{equation}\n\\label{eq:strong2}\n\\norm{\\hat x_{nm}^{(k)}-\\hat x_{nm}}_2 \\leq \\bigg(\\frac{2L_f L_g}{\\mu n }\\bigg)^{1/\\delta}.\n\\end{equation}\nBy Lipschitz continuity of $f_\\xi(\\cdot)$ and $g_\\eta(\\cdot,\\xi)$, and definition of $\\hat F_{nm}(\\hat x_{nm})$, we obtain\n\\begin{equation}\n\\label{eq:sca}\n\\begin{split}\n   \\EE \\bigg[\\frac{1}{n}\\sum_{k =1}^n f_{\\xi_k}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k)\\bigg) - \\hat F_{nm}(\\hat x_{nm})\\bigg]\n    \\leq  L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}.\n\\end{split}\n\\end{equation}\n\nCombining (\\ref{eq:decompose_term_1}), (\\ref{eq:sca}), and (\\ref{eq:biasboth}), we obtain\n\\begin{equation}\n\\label{eq:scAl}\n    \\EE \\Epsilon_1\n    \\leq \n    L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}+\\Delta(m).\n\\end{equation}\n\\if 0\nfor $L_f$-Lipschitz continuous $f_\\xi(\\cdot)$,\n\\begin{equation}\n\\label{eq:scAl}\n    \\EE \\Epsilon_1\n    \\leq \n    L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}+\\frac{L_{f}\\sigma_{g}}{\\sqrt{m}};\n\\end{equation}\nfor $S$-Lipschitz smooth $f_\\xi(\\cdot)$,\n\\begin{equation}\n\\label{eq:scAs}\n    \\EE \\Epsilon_1\n    \\leq \n    L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}+\\frac{S\\sigma_{g}^2}{2m}.\n\\end{equation}\n\\fi\n\nSecond, by optimality of $\\hat x_{nm}$ of $\\hat F_{nm}$, we have\n\\begin{equation}\n\\label{eq:scB}\n    \\EE \\Epsilon_2 = \\EE [\\hat F_{nm}(\\hat x_{nm})- \\hat F_{nm}(x^*)] \\leq 0.\n\\end{equation}\n\nNext, we bound $\\EE \\Epsilon_3$.\nDefine\n$\n\\hat{F}_n(x):=\\frac{1}{n}\\sum_{i=1}^nf_{\\xi_i}\\big(\\EE_{\\eta|\\xi_i}[g_\\eta(x,\\xi_i)]\\big).\n$\nNotice that $x^*$ is independent of $\\{\\eta_{ij}\\}_{j=1}^m$ for any $i = \\{1,\\cdots,n\\}$ and $\\EE [\\hat F_n(x^*) - F(x^*)]=0$. By Lemma \\ref{lm:bias}, it holds\n\\begin{equation}\n\\label{eq:scC}\n    \\EE \\Epsilon_3 = \\EE[ \\hat F_{nm}(x^*)-\\hat F_n(x) ]+ \\EE[\\hat F_n(x) - F(x)]\n    \\leq \n    \\Delta(m);\n\\end{equation}\n\\if 0\nfor $L_f$-Lipschitz continuous $f_\\xi(\\cdot)$,\n\\begin{equation}\n\\label{eq:scCL}\n    \\EE \\Epsilon_3 = \\EE[ \\hat F_{nm}(x^*)-\\hat F_n(x) ]+ \\EE[\\hat F_n(x) - F(x)]\n    \\leq \n    \\frac{L_{f}\\sigma_{g}}{\\sqrt{m}};\n\\end{equation}\nfor $S$-Lipschitz smooth $f_\\xi(\\cdot)$, we have\n\\begin{equation}\n\\label{eq:scCS}\n    \\EE \\Epsilon_3 = \\EE[ \\hat F_{nm}(x^*)-\\hat F_n(x) ]+ \\EE[\\hat F_n(x) - F(x)]\n    \\leq \n    \\frac{S\\sigma_{g}^2}{2m}.\n\\end{equation}\n\\fi\nCombining (\\ref{eq:scAl}), (\\ref{eq:scB}), (\\ref{eq:scC}), with Markov inequality, we obtain the desired result.\n\\end{proof}","sha256_utf8":"40405db85924d3af016b693ae7ddb541227dcec3c1dd888f32f991352e087a7c"},{"start":88854,"end":90629,"start_line":1178,"end_line":1210,"raw_tex":"\\begin{proof}{}\nDenote $X_{j}(x): = g_{\\eta_{j}}(x,\\xi)-\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)$, and $\\{\\eta_{j}\\}_{j=1}^m=\\{\\eta_{1},\\cdots, \\eta_{m}\\}$. By Assumption \\ref{as:est},\n\\begin{equation}\n\\label{eq:mv}\n    \\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)\\bigg)=0, \\quad 0 \\leq \\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi} \\norm{\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)}_2^2 \\leq \\frac{\\sigma_{g}^2}{m}.\n\\end{equation}\nIf $f_\\xi(\\cdot)$ is $L_f$-Lipschitz continuous,\n\\begin{equation*}\n    \\bigg|\\EE f_{\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{j}}(x,\\xi)\\bigg)   -   f_{\\xi}\\bigg(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\bigg)\\bigg|\n    \\leq \n    \\EE L_f \\norm{\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)}_2 \n    \\leq \n    \\frac{ L_f \\sigma_{g}}{\\sqrt{m}}.\n\\end{equation*}\nIf $f_\\xi(\\cdot)$ is $S$-Lipschitz smooth,\n\\begin{equation*}\n\\begin{aligned}\n\t&\\bigg|f_{\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{j}}(x,\\xi)\\bigg) \n    - f_{\\xi}\\bigg(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\bigg)\n    - \\nabla f_{\\xi} \\bigg(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\bigg)^\\top \\frac{1}{m}\\sum_{j=1}^m X_{j}(x)\\bigg| \\\\\n    \\leq &\\frac{S}{2} \\norm{\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)}_2^2.\n    \\end{aligned}\n\\end{equation*}\n$\\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi}  \\nabla f_{\\xi} \\big(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\big)^\\top \\big(\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)\\big)=0$ since $x$ is independent of ${\\eta}_{j=1}^m$, we obtain,\n\\begin{equation*}\n    \\EE   f_{\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{j}}(x,\\xi)\\bigg)   -   f_{\\xi}\\bigg(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\bigg)\n    \\leq \n    \\EE \\bigg[ \\frac{S}{2} \\norm{\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)}_2^2 \\bigg] \\\\\n    \\leq  \n    \\frac{S\\sigma_{g}^2}{2m}.\n\\end{equation*}\nFor the other side, we could derive a similar lower bound. As a result, (\\ref{eq:smooth_bias}) holds.\n\\end{proof}","sha256_utf8":"fe1fec07b5e12ff888f8842c5b112d26dc9be14ecb32b8a4815126f09f534bea"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"误差界下的SAA收敛","source_units":[{"start":50588,"end":51226,"start_line":608,"end_line":626,"raw_tex":"\\begin{thm}[SAA under Error Bound Condition]\\label{thm:SAA_error_bound}\nUnder Assumption \\ref{as:est}, \\ref{as:fpl}, \\ref{as:uniquepro}, for any $\\eps > 0$, we have\n\\begin{equation}\n\\PP (F(\\hat x_{nm}) - F(x^*) \\geq \\eps )\n\\leq \n\\frac{1}{\\eps}\\bigg(L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}\n+\n\\frac{2L_{f}\\sigma_{g}}{\\sqrt{m}}\\bigg).\n\\end{equation}\n    \nIf additionally, $f_\\xi(\\cdot)$ is $S$-Lipschitz smooth, then we further have \n\\begin{equation}\n\\PP (F(\\hat x_{nm}) - F(x^*) \\geq \\eps )\n\\leq \n\\frac{1}{\\eps}\\bigg(L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}\n+\n\\frac{S\\sigma_{g}^2}{m}\\bigg).\n\\end{equation}\n\\end{thm}","sha256_utf8":"15e83d61cfd6d9cb9a6a44cf65f822c43fc37cecdea0d16c5e23be293b2281f6"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"稳定性收敛证明","source_units":[{"start":51989,"end":59400,"start_line":640,"end_line":793,"raw_tex":"\\begin{proof} \nRecall that $x^*$ and $\\hat x_{nm}$ are the minimizers of $F(x)$ and $\\hat F_{nm}(x)$, respectively. It's clear that $x^*$ has no randomness,  and $\\hat x_{nm}$ is a function of $\\{\\xi_i\\}_{i =1}^n, \\{\\eta_{ij}\\}_{j=1}^m$. \nWe decompose the error $F(\\hat x_{nm}) - F(x^*)$ in three terms, and analyze each term below:\n\\begin{equation*}\n\\begin{split}\nF(\\hat x_{nm}) \\!-\\! F(x^*)\n\t\t\\!=\\!\n\t\t\\underbrace{F(\\hat x_{nm})\\!-\\! \\hat F_{nm}(\\hat x_{nm})}_{:=\\Epsilon_1}+\n\t\t\\underbrace{\\hat F_{nm}(\\hat x_{nm})\\! -\\! \\hat F_{nm}(x^*)}_{:=\\Epsilon_2}+\n\t\t\\underbrace{\\hat F_{nm}(x^*)\\!- \\! F(x^*)}_{:=\\Epsilon_3}.\n\\end{split}\n\\end{equation*}\nFirst, we use a stability argument and Lemma \\ref{lm:bias} to bound $\\EE \\Epsilon_1 = \\EE [F(\\hat x_{nm})- \\hat F_{nm}(\\hat x_{nm})]$. \nDefine \n\\begin{equation}\n\\begin{aligned}\n\\label{eq:Fnmk}\n     \\hat F_{nm}^{(k)}(x) \n     := \\frac{1}{n}\\sum_{i\\not = k}^n f_{\\xi_i}\\bigg(\\frac{1}{m}\\sum_{j =1}^m g_{\\eta_{ij}}(x,\\xi_i)\\bigg) + \\frac{1}{n}f_{\\xi_k^{\\prime}}\\bigg(\\frac{1}{m}\\sum_{j =1}^m g_{\\eta_{kj}^{\\prime}}(x,\\xi_k^{\\prime})\\bigg)\n     \\end{aligned}\n\\end{equation}\nas the empirical function by replacing the $k^{th}$ outer sample $\\xi_k$ with another i.i.d outer sample $\\xi_k^{\\prime}$, and replacing the corresponding  inner samples $\\{\\eta_{kj}\\}_{j=1}^m$  with $\\{\\eta_{kj}^{\\prime}\\}_{j=1}^m$, which are sampled from the conditional distribution of $\\PP(\\eta|\\xi_k^{\\prime})$ for a given sample $\\xi_k^\\prime$. Denote $\\hat x_{nm}^{(k)} := \\argmin_{x \\in \\mathcal{X}} \\hat F_{nm}^{(k)}(x)$. We decompose $\\EE \\Epsilon_1 = \\EE [F(\\hat x_{nm})- \\hat F_{nm}(\\hat x_{nm})]$ into three terms:\n\\begin{equation}\\label{eq:decompose_term_1}\n\\begin{split}\n\\EE \\Epsilon_1 = &\\EE\\bigg[\\frac{1}{n}\\sum_{k=1}^n F(\\hat x_{nm}) -\\frac{1}{n}\\sum_{k =1}^n  f_{\\xi_k} \\bigg( \\EE_{\\eta|\\xi_k} g_\\eta(\\hat x_{nm}^{(k)},\\xi_k) \\bigg)\\bigg]\\\\\n+&\\EE \\bigg[\\frac{1}{n}\\sum_{k =1}^n  f_{\\xi_k} \\bigg( \\EE_{\\eta|\\xi_k} g_\\eta(\\hat x_{nm}^{(k)},\\xi_k) \\bigg) -\\frac{1}{n}\\sum_{k =1}^n f_{\\xi_k}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k)\\bigg)\\bigg]\\\\\n+&\\EE \\bigg[\\frac{1}{n}\\sum_{k =1}^n f_{\\xi_k}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k)\\bigg) - \\hat F_{nm}(\\hat x_{nm})\\bigg].\n\\end{split}\n\\end{equation}\nNote that $\n    \\EE [F(\\hat x_{nm})] = \\EE [F(\\hat x_{nm}^{(k)})]\n$ since $\\xi_k$ and $\\xi_k^{\\prime}$ are i.i.d, which implies that  $\\hat x_{nm}$ and $\\hat x_{nm}^{(k)}$ follow an identical distribution.\nSince $\\hat x_{nm}^{(k)}$ is independent of $\\xi_k$, \n$\n\\EE [F(\\hat x_{nm}^{(k)})] = \\EE [f_{\\xi_k}(\\EE_{\\eta|\\xi_k}g(\\hat x_{nm}^{(k)},\\xi_k))]\n$\nfor any $k$. Then the first term in (\\ref{eq:decompose_term_1}) is $0$.\nAs $\\hat x_{nm}^{(k)}$ is independent of $\\{\\eta_{kj}\\}_{j=1}^m$, the second term in (\\ref{eq:decompose_term_1}) could be bounded by Lemma \\ref{lm:bias},\nit holds\n\\begin{equation}\n\\label{eq:biasboth}\n    \\EE \\bigg[ f_{\\xi_k} \\bigg( \\EE_{\\eta|\\xi_k} g_\\eta(\\hat x_{nm}^{(k)},\\xi_k) \\bigg)  \n    -   \n    f_{\\xi_k}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k) \\bigg)\\bigg]\n    \\leq \n    \\Delta(m).\n\\end{equation}\n\\if 0\nfor $S$-Lipschitz smooth $f_\\xi(\\cdot)$, we get\n\\begin{equation*}\n    \\EE \\bigg[ f_{\\xi_k} \\bigg( \\EE_{\\eta|\\xi_k} g_\\eta(\\hat x_{nm}^{(k)},\\xi_k) \\bigg)  \n    -   \n    f_{\\xi_k}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k) \\bigg)\\bigg] \n    \\leq \n    \\frac{S\\sigma_{g}^2}{2m}.\n\\end{equation*}\n\\fi\nFor the third term in (\\ref{eq:decompose_term_1}), by definition it implies\n\\begin{equation}\n\\label{eq:sca1}\n    \\begin{aligned}\n    \\hat F_{nm}(\\hat x_{nm}^{(k)})  - \\hat F_{nm}(\\hat x_{nm}) = &\\hat F_{nm}^{(k)}(\\hat x_{nm}^{(k)})  - \\hat F_{nm}^{(k)}(\\hat x_{nm}) \\\\\n    + &\n      \\frac{1}{n}f_{\\xi_k}\\! \\bigg(\\! \\frac{1}{m}\\! \\sum_{j =1}^m\\!  g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k)\\! \\bigg) \\! \n    - \\! \\frac{1}{n}f_{\\xi_k}\\! \\bigg(\\! \\frac{1}{m}\\sum_{j =1}^m g_{\\eta_{kj}}(\\hat x_{nm},\\xi_k)\\! \\bigg)\\\\\n     + &\\frac{1}{n}f_{\\xi_k^{\\prime}}\\! \\bigg(\\! \\frac{1}{m}\\sum_{j =1}^m g_{\\eta_{kj}^{\\prime}}(\\hat x_{nm},\\xi_k^{\\prime})\\! \\bigg) \\! \n   \\!  - \\! \n   \\frac{1}{n}f_{\\xi_k^{\\prime}}\\! \\bigg(\\! \\frac{1}{m}\\sum_{j =1}^m g_{\\eta_{kj}^{\\prime}}(\\hat x_{nm}^{(k)},\\xi_k^{\\prime})\\! \\bigg).\n    \\end{aligned}\n\\end{equation}\nBy Lipschitz continuity of $f_\\xi$ and $g_\\eta$ and that $\\hat F_{nm}^{(k)}(\\hat x_{nm}^{(k)})  - \\hat F_{nm}^{(k)}(\\hat x_{nm}) \\leq 0$, it holds\n\\begin{equation}\n\\label{eq:sca2}\n\\hat F_{nm}(\\hat x_{nm}^{(k)})  - \\hat F_{nm}(\\hat x_{nm}) \\leq \\frac{2}{n}L_f L_g \\norm{\\hat x_{nm}^{(k)}-\\hat x_{nm}}_2.\n\\end{equation}\nSince $\\hat x_{nm}$ is the unique minimizer of $\\hat F_{nm}(x)$, and $\\hat F_{nm}(x)$ satisfies QG condition with parameter $\\mu$, we have\n\\begin{equation}\n\\label{eq:strong}\n    \\hat F_{nm}(\\hat x_{nm}^{(k)})  - \\hat F_{nm}(\\hat x_{nm}) \\geq \\mu \\norm{\\hat x_{nm}^{(k)}-\\hat x_{nm}}_2^{1+\\delta}.\n\\end{equation}\nCombining with (\\ref{eq:sca2}), we obtain\n\\begin{equation}\n\\label{eq:strong2}\n\\norm{\\hat x_{nm}^{(k)}-\\hat x_{nm}}_2 \\leq \\bigg(\\frac{2L_f L_g}{\\mu n }\\bigg)^{1/\\delta}.\n\\end{equation}\nBy Lipschitz continuity of $f_\\xi(\\cdot)$ and $g_\\eta(\\cdot,\\xi)$, and definition of $\\hat F_{nm}(\\hat x_{nm})$, we obtain\n\\begin{equation}\n\\label{eq:sca}\n\\begin{split}\n   \\EE \\bigg[\\frac{1}{n}\\sum_{k =1}^n f_{\\xi_k}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k)\\bigg) - \\hat F_{nm}(\\hat x_{nm})\\bigg]\n    \\leq  L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}.\n\\end{split}\n\\end{equation}\n\nCombining (\\ref{eq:decompose_term_1}), (\\ref{eq:sca}), and (\\ref{eq:biasboth}), we obtain\n\\begin{equation}\n\\label{eq:scAl}\n    \\EE \\Epsilon_1\n    \\leq \n    L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}+\\Delta(m).\n\\end{equation}\n\\if 0\nfor $L_f$-Lipschitz continuous $f_\\xi(\\cdot)$,\n\\begin{equation}\n\\label{eq:scAl}\n    \\EE \\Epsilon_1\n    \\leq \n    L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}+\\frac{L_{f}\\sigma_{g}}{\\sqrt{m}};\n\\end{equation}\nfor $S$-Lipschitz smooth $f_\\xi(\\cdot)$,\n\\begin{equation}\n\\label{eq:scAs}\n    \\EE \\Epsilon_1\n    \\leq \n    L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}+\\frac{S\\sigma_{g}^2}{2m}.\n\\end{equation}\n\\fi\n\nSecond, by optimality of $\\hat x_{nm}$ of $\\hat F_{nm}$, we have\n\\begin{equation}\n\\label{eq:scB}\n    \\EE \\Epsilon_2 = \\EE [\\hat F_{nm}(\\hat x_{nm})- \\hat F_{nm}(x^*)] \\leq 0.\n\\end{equation}\n\nNext, we bound $\\EE \\Epsilon_3$.\nDefine\n$\n\\hat{F}_n(x):=\\frac{1}{n}\\sum_{i=1}^nf_{\\xi_i}\\big(\\EE_{\\eta|\\xi_i}[g_\\eta(x,\\xi_i)]\\big).\n$\nNotice that $x^*$ is independent of $\\{\\eta_{ij}\\}_{j=1}^m$ for any $i = \\{1,\\cdots,n\\}$ and $\\EE [\\hat F_n(x^*) - F(x^*)]=0$. By Lemma \\ref{lm:bias}, it holds\n\\begin{equation}\n\\label{eq:scC}\n    \\EE \\Epsilon_3 = \\EE[ \\hat F_{nm}(x^*)-\\hat F_n(x) ]+ \\EE[\\hat F_n(x) - F(x)]\n    \\leq \n    \\Delta(m);\n\\end{equation}\n\\if 0\nfor $L_f$-Lipschitz continuous $f_\\xi(\\cdot)$,\n\\begin{equation}\n\\label{eq:scCL}\n    \\EE \\Epsilon_3 = \\EE[ \\hat F_{nm}(x^*)-\\hat F_n(x) ]+ \\EE[\\hat F_n(x) - F(x)]\n    \\leq \n    \\frac{L_{f}\\sigma_{g}}{\\sqrt{m}};\n\\end{equation}\nfor $S$-Lipschitz smooth $f_\\xi(\\cdot)$, we have\n\\begin{equation}\n\\label{eq:scCS}\n    \\EE \\Epsilon_3 = \\EE[ \\hat F_{nm}(x^*)-\\hat F_n(x) ]+ \\EE[\\hat F_n(x) - F(x)]\n    \\leq \n    \\frac{S\\sigma_{g}^2}{2m}.\n\\end{equation}\n\\fi\nCombining (\\ref{eq:scAl}), (\\ref{eq:scB}), (\\ref{eq:scC}), with Markov inequality, we obtain the desired result.\n\\end{proof}","sha256_utf8":"40405db85924d3af016b693ae7ddb541227dcec3c1dd888f32f991352e087a7c"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"条件随机优化模型","source_units":[{"start":6970,"end":7121,"start_line":109,"end_line":112,"raw_tex":"\\begin{equation}\n\\label{pro:ori}\n    \\min_{x \\in \\mathcal{X}}\\; F(x):=\\EE_{\\xi}\\Big[f_\\xi\\Big({\\EE_{\\eta|\\xi}[g_\\eta(x,\\xi)]}\\Big)\\Big].\n\\end{equation}","sha256_utf8":"84f151728d7ba01b009d62660df2513090ec8ee12d170f118d49e14636bb6aaa"},{"start":7122,"end":7873,"start_line":113,"end_line":113,"raw_tex":"Here $\\mathcal{X}$ is the domain of the decision variable $x\\in\\RR^d$; $f_\\xi(\\cdot):\\RR^k\\to\\RR$ is a continuous cost function dependent on the random vector $\\xi$, and $g_\\eta(\\cdot,\\xi):\\RR^d\\to\\RR^k$  is a vector-valued continuous cost function dependent on both random vectors $\\xi$ and $\\eta$. The inner expectation is with respect to $\\eta$ given $\\xi$, and the outer expectation is with respect to $\\xi$. Similar to the classical stochastic optimization, we do not assume any knowledge on the underlying distribution of $\\PP(\\xi)$ nor the conditional distribution $\\PP(\\eta|\\xi)$.  Instead, we assume availability of  samples  from the distribution $\\PP(\\xi)$ and samples from the conditional distribution $\\PP(\\eta|\\xi)$ for any given $\\xi$.\n","sha256_utf8":"b2f9db19f6830261f19b7e6e5e6f47e3a0c8f6ed7333c2b1fc3495c5bc3623a2"}]},{"id":"S004","semantic_type":"algorithm","kind":"algorithm","title":"条件采样与经验最优解","source_units":[{"start":8442,"end":8635,"start_line":118,"end_line":121,"raw_tex":"\\begin{equation}\n\\label{pro:cosaa0}\n   \\min_{x \\in \\mathcal{X}}\\; \\hat F_{nm}(x):= \\frac{1}{n}\\sum_{i=1}^n f_{\\xi_i}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{ij}}(x, \\xi_{i})\\bigg),\n\\end{equation}","sha256_utf8":"a32598d1181da3557bb59eae72b93fc99724971e65d722bb60ca71c438111ac3"},{"start":8636,"end":9039,"start_line":122,"end_line":122,"raw_tex":"where  $\\{\\xi_i\\}_{i=1}^n$ are i.i.d. samples generated from $\\PP(\\xi)$ and $\\{\\eta_{ij}\\}_{j=1}^m$ are i.i.d. samples generated from the conditional distribution $\\PP(\\eta | \\xi_i)$ for a given outer sample $\\xi_i$.  We would like to examine the total number of samples $T=nm+n$ required for SAA~(\\ref{pro:cosaa0}) to achieve an $\\epsilon$-optimal solution to the original CSO problem~(\\ref{pro:ori}).\n","sha256_utf8":"1f164d90b92f5271bd87de4f3733ec9255b06deedd21ce5e8d4333a91c6c818a"},{"start":30862,"end":31399,"start_line":361,"end_line":363,"raw_tex":"We denote $x^*$ and $ \\hat{x}_{nm}$ the optimal solutions to the CSO and the SAA problems, respectively. We are interested in estimating the probability of $\\hat x_{nm}$ being an $\\eps$-optimal solution to the CSO problem, namely $\\PP\\left(F(\\hat x_{nm})-F(x^*) \\leq \\eps\\right)$, for an arbitrary accuracy $\\eps>0$.\n\nThroughout the paper, we assume availability of i.i.d. samples generated from distribution $\\PP(\\xi)$ and conditional distribution $\\PP(\\eta\\given \\xi)$ for any given $\\xi$, and we make the following basic assumptions:\n","sha256_utf8":"e33b1a905d420553bbdead0d80ecfb8460f1fd491d3a87c265d8ba7160ae958c"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"范数与连续光滑定义","source_units":[{"start":25347,"end":25749,"start_line":303,"end_line":303,"raw_tex":"For convenience, we collect here some notations that will be used throughout the paper. We also introduce some  mathematical tools and propositions that are necessary for future discussion. For simplicity, we restrict our attention to $l_2$-norm, denoted as $\\norm{\\cdot}_2$. Similar results on sample complexity with respect to different norms can be obtained with minor modification of the analysis.\n","sha256_utf8":"0cc50ef07e23c9de2a649faf3e5df1c13f3e7633b629bd75999591e1560fdf20"},{"start":26477,"end":27312,"start_line":307,"end_line":308,"raw_tex":"A function $f: \\mathcal{X} \\rightarrow \\RR$ is said to be $L$-Lipschitz continuous, if there exists a constant $L>0$ such that $|f(x_1) - f(x_2)|\\leq L\\norm{x_1-x_2}_2, \\forall x_1, x_2 \\in \\mathcal{X}$. The function $f: \\mathcal{X} \\rightarrow \\RR$ is said to be $S$-Lipschitz smooth, if it is continuously differentiable and its gradient is $S$-Lipschitz continuous. This also implies that  $\\forall x_1, x_2 \\in \\mathcal{X}:|f(x_1)-f(x_2) -\\nabla f(x_2)^\\top(x_1-x_2)|  \\leq  \\frac{S}{2}\\norm{x_1-x_2}_2^2$. If a continuously differentiable function $f: \\mathcal{X} \\rightarrow \\RR$ satisfies that $\\forall x_1, x_2 \\in \\mathcal{X}$, $f(x_1)-f(x_2)- \\nabla f(x_2)^\\top(x_1-x_2) \\geq \\frac{\\mu}{2}\\norm{x_1-x_2}_2^2,$ \nthen $f$ is called $\\mu$-strongly convex when $\\mu>0$, convex when $\\mu=0$, and $\\mu$-weakly convex when $\\mu<0$.\n","sha256_utf8":"b92121b287826c943471c2a56bbca27734f5c572fb38a17593db15a49cec3864"}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","title":"基本正则性假设","source_units":[{"start":31399,"end":32247,"start_line":364,"end_line":375,"raw_tex":"\\begin{assume}\n\\label{as:est}\nWe assume that \n\\begin{enumerate}\n    \\item[(a)] The decision set $\\mathcal{X} \\subseteq \\RR^d$ has a finite diameter $D_\\mathcal{X}>0$.\n    \\item[(b)] $f_\\xi(\\cdot)$ is $L_f$-Lipschitz continuous and $g_\\eta(\\cdot,\\xi)$ is $L_g$-Lipschitz continuous for any given $\\xi$ and $\\eta$.\n    \\item[(c)] For all $x \\in\\mathcal{X}$, $f(x,\\xi)$ is Borel measurable in $\\xi$ and $g_\\eta(x,\\xi)$ is Borel measurable in $\\eta$ for all $\\xi$.\n    \\item[(d)] $\\sigma_f^2:=\\max_{x\\in\\mathcal{X}} \\Var_\\xi\\left(f_\\xi(\\EE_{\\eta|\\xi}[g_\\eta(x,\\xi)])\\right) < \\infty$.\n    \\item[(e)] $\\sigma_g^2:=\\max_{x\\in\\mathcal{X}, \\xi} \\EE_{\\eta|\\xi}\\norm{g_\\eta(x,\\xi)-{\\EE_{\\eta|\\xi}} g_\\eta(x,\\xi)}_2^2 < \\infty$.\n    \\item[(f)] $|f_\\xi(\\cdot)| \\leq M_f$, $\\|g_\\eta(\\cdot,\\xi)\\|_2\\leq M_g$ for any $\\xi$ and $\\eta$.\n\\end{enumerate}\n\\end{assume}","sha256_utf8":"9d964db5cbde73f35681270cfcf7709c2ef7bfcdae9b063260e0c26523515c6e"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"霍尔德误差界定义","source_units":[{"start":27314,"end":27911,"start_line":311,"end_line":318,"raw_tex":"\\begin{defn}[H\\\"olderian error bound  condition]\n\\label{def:lineq}\nLet $f: \\mathcal{X} \\rightarrow \\RR$ be a function with compact domain $\\mathcal{X}$ and the optimal solution set $\\mathcal{X}^*$ is nonempty. $f(\\cdot)$ satisfies the $(\\mu,\\delta)$-H\\\"olderian error bound condition if there exists $\\delta \\geq 0$ and $\\mu>0$ such that \n\\begin{equation*}\n\\forall x\\in \\mathcal{X}, \\quad f(x)-\\min_{x\\in\\mathcal{X}}f(x) \\geq \\mu \\inf_{z\\in\\mathcal{X}^*}\\norm{x-z}_2^{1+\\delta}. \n\\end{equation*}\nIn particular, when $\\delta=1$, we say $f$ satisfies the quadratic growth (QG) condition. \n\\end{defn}","sha256_utf8":"8f9417890d7a04dfd4dad83a95c365df41861ff8a27ce470317323975280c74a"}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","title":"经验目标误差界","source_units":[{"start":46441,"end":46962,"start_line":563,"end_line":570,"raw_tex":"\\begin{assume}\n\\label{as:fpl}\nThe empirical function $\\hat F_{nm}(x)$ satisfies the $(\\mu,\\delta)$-H\\\"olderian error bound condition with $\\mu>0,\\delta\\geq0$, i.e., it holds that \n\\begin{equation*}\n\\forall x\\in \\mathcal{X}, \\; \\hat F_{nm}(x)-\\min_{x\\in\\mathcal{X}}\\hat F_{nm}(x) \\geq \\mu \\inf_{z\\in\\mathcal{X}_{nm}^*}\\norm{x-z}_2^{1+\\delta}, \n\\end{equation*}\nwhere $n,m$ are any positive integers, and $\\cX_{nm}^*$ is the optimal solution set of the empirical objective function $\\hat F_{nm}(x)$ over $\\cX$. \n\\end{assume}","sha256_utf8":"0e59b92ff2f3c0ae1b77f7007a6d0f1b8302a8c831da24802adf8ed00610a8ef"}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","title":"经验极小点唯一性","source_units":[{"start":46964,"end":47125,"start_line":572,"end_line":575,"raw_tex":"\\begin{assume}\n\\label{as:uniquepro}\nThe empirical function $\\hat F_{nm}$ has a unique minimizer $\\hat x_{nm}$ on $\\mathcal{X}$, for any $n$ and $m$.\n\\end{assume}","sha256_utf8":"8e9f1a4e220d72a5ee4fc38462c0b0c3e166d83fc45b633c7f61cc932c50d26c"}]},{"id":"S010","semantic_type":"notation","kind":"notation","title":"两类偏差界记号","source_units":[{"start":51647,"end":51912,"start_line":630,"end_line":637,"raw_tex":"\\begin{equation}\n\\label{biasterm}\n    \\Delta(m):=\n\\begin{cases}\n\\frac{L_f\\sigma_{g}}{\\sqrt{m}}, &\\text{ $f_\\xi(\\cdot)$ is $L_f$ Lipschitz continuous},\\\\\n\\frac{S\\sigma_{g}^2}{2m},&\\text{$f_\\xi(\\cdot)$ is additionally $S$ Lipschitz smooth.}\n\\end{cases}\n\\end{equation}","sha256_utf8":"77c6881ff40a100ceecb96d88f53ac52dd35038425d5970882910e9df57f609b"}]},{"id":"S011","semantic_type":"lemma","kind":"lemma","title":"嵌套估计偏差界","source_units":[{"start":33400,"end":34257,"start_line":389,"end_line":400,"raw_tex":"\\begin{lm}\\label{lm:bias} \nLet $\\{\\eta_{j}\\}_{j=1}^m$ be conditional samples from $P(\\eta|\\xi)$ given $\\xi\\sim P(\\xi)$. Under Assumption \\ref{as:est}, for any fixed $x\\in\\cX$ that is independent of $\\xi$ and $\\{\\eta_{j}\\}_{j=1}^m$, it holds that,\n\\begin{equation}\n\\label{eq:lipschitz_bias}\n\\bigg|\\EE_{\\{\\xi,\\{\\eta_{j}\\}_{j=1}^m \\}}\\bigg[f_{\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{j}}(x,\\xi)\\bigg)   -   f_{\\xi}\\big(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\big)\\bigg]\\bigg| \\leq \\frac{ L_f \\sigma_{g}}{\\sqrt{m}}.\n\\end{equation}\nIf additionally, $f_\\xi(\\cdot)$ is $S$-Lipschitz smooth, we have\n\\begin{equation}\n\\label{eq:smooth_bias}\n\\bigg|\\EE_{\\{\\xi,\\{\\eta_{j}\\}_{j=1}^m\\}} \\bigg[f_{\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{j}}(x,\\xi)\\bigg)   -   f_{\\xi}\\big(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\big)\\bigg]\\bigg|  \\leq  \\frac{S\\sigma_{g}^2}{2m}.\n\\end{equation}\n\\end{lm}","sha256_utf8":"00a6b94687bc62bd9b50587c4f9eb849750f198b44be00cec79dabd7ad11dca0"}]},{"id":"S012","semantic_type":"proof","kind":"proof","title":"偏差界正文证明","source_units":[{"start":34258,"end":34863,"start_line":401,"end_line":404,"raw_tex":"\\begin{proof}{}\nDefine $X_{j}: = g_{\\eta_{j}}(x,\\xi)-\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)$ and $\\bar{X}:=\\sum_{j=1}^m X_j/m$. It follows  $\\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi}[\\bar{X}]=0$ by definition, and $\\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi}[\\|\\bar{X}\\|_2^2]\\leq \\sigma_g^2/m$ by Assumption \\ref{as:est}(d).  $\\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi}  \\nabla f_{\\xi} \\big(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\big)^\\top \\big(\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)\\big)=0$ since $x$ is independent of $\\{\\eta_j\\}_{j=1}^m$. \nThe results then follow directly by invoking the Lipschitz continuity and smoothness and taking expectations. \n\\end{proof}","sha256_utf8":"a6ccee7d32d1aaab9870d97617181dac99bf183370bb275eb9bb91a0d8985c4f"}]},{"id":"S013","semantic_type":"proof_step","kind":"proof_step","title":"中心化与条件矩估计","source_units":[{"start":34271,"end":34741,"start_line":401,"end_line":402,"raw_tex":"{}\nDefine $X_{j}: = g_{\\eta_{j}}(x,\\xi)-\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)$ and $\\bar{X}:=\\sum_{j=1}^m X_j/m$. It follows  $\\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi}[\\bar{X}]=0$ by definition, and $\\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi}[\\|\\bar{X}\\|_2^2]\\leq \\sigma_g^2/m$ by Assumption \\ref{as:est}(d).  $\\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi}  \\nabla f_{\\xi} \\big(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\big)^\\top \\big(\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)\\big)=0$ since $x$ is independent of $\\{\\eta_j\\}_{j=1}^m$. \n","sha256_utf8":"8ab56d4bbaae40d84981276d2bc3547b24d47cd6c89d07bad0fae1dab9532044"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"连续性与光滑性给出偏差","source_units":[{"start":34741,"end":34852,"start_line":403,"end_line":403,"raw_tex":"The results then follow directly by invoking the Lipschitz continuity and smoothness and taking expectations. \n","sha256_utf8":"cf62bb5e8b1d9ffd5b7513df84894d622e52bf3d374274eb5c6b1b784fe2ec4f"}]},{"id":"S015","semantic_type":"proof","kind":"proof","title":"偏差界附录完整推导","source_units":[{"start":88854,"end":90629,"start_line":1178,"end_line":1210,"raw_tex":"\\begin{proof}{}\nDenote $X_{j}(x): = g_{\\eta_{j}}(x,\\xi)-\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)$, and $\\{\\eta_{j}\\}_{j=1}^m=\\{\\eta_{1},\\cdots, \\eta_{m}\\}$. By Assumption \\ref{as:est},\n\\begin{equation}\n\\label{eq:mv}\n    \\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)\\bigg)=0, \\quad 0 \\leq \\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi} \\norm{\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)}_2^2 \\leq \\frac{\\sigma_{g}^2}{m}.\n\\end{equation}\nIf $f_\\xi(\\cdot)$ is $L_f$-Lipschitz continuous,\n\\begin{equation*}\n    \\bigg|\\EE f_{\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{j}}(x,\\xi)\\bigg)   -   f_{\\xi}\\bigg(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\bigg)\\bigg|\n    \\leq \n    \\EE L_f \\norm{\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)}_2 \n    \\leq \n    \\frac{ L_f \\sigma_{g}}{\\sqrt{m}}.\n\\end{equation*}\nIf $f_\\xi(\\cdot)$ is $S$-Lipschitz smooth,\n\\begin{equation*}\n\\begin{aligned}\n\t&\\bigg|f_{\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{j}}(x,\\xi)\\bigg) \n    - f_{\\xi}\\bigg(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\bigg)\n    - \\nabla f_{\\xi} \\bigg(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\bigg)^\\top \\frac{1}{m}\\sum_{j=1}^m X_{j}(x)\\bigg| \\\\\n    \\leq &\\frac{S}{2} \\norm{\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)}_2^2.\n    \\end{aligned}\n\\end{equation*}\n$\\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi}  \\nabla f_{\\xi} \\big(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\big)^\\top \\big(\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)\\big)=0$ since $x$ is independent of ${\\eta}_{j=1}^m$, we obtain,\n\\begin{equation*}\n    \\EE   f_{\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{j}}(x,\\xi)\\bigg)   -   f_{\\xi}\\bigg(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\bigg)\n    \\leq \n    \\EE \\bigg[ \\frac{S}{2} \\norm{\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)}_2^2 \\bigg] \\\\\n    \\leq  \n    \\frac{S\\sigma_{g}^2}{2m}.\n\\end{equation*}\nFor the other side, we could derive a similar lower bound. As a result, (\\ref{eq:smooth_bias}) holds.\n\\end{proof}","sha256_utf8":"fe1fec07b5e12ff888f8842c5b112d26dc9be14ecb32b8a4815126f09f534bea"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"条件均值与二阶矩","source_units":[{"start":88867,"end":89274,"start_line":1178,"end_line":1183,"raw_tex":"{}\nDenote $X_{j}(x): = g_{\\eta_{j}}(x,\\xi)-\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)$, and $\\{\\eta_{j}\\}_{j=1}^m=\\{\\eta_{1},\\cdots, \\eta_{m}\\}$. By Assumption \\ref{as:est},\n\\begin{equation}\n\\label{eq:mv}\n    \\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)\\bigg)=0, \\quad 0 \\leq \\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi} \\norm{\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)}_2^2 \\leq \\frac{\\sigma_{g}^2}{m}.\n\\end{equation}\n","sha256_utf8":"8a34169c88f77654cfc15f1154705474a9cf3060fda386c255abee7dd3308472"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"连续情形偏差估计","source_units":[{"start":89274,"end":89609,"start_line":1184,"end_line":1191,"raw_tex":"If $f_\\xi(\\cdot)$ is $L_f$-Lipschitz continuous,\n\\begin{equation*}\n    \\bigg|\\EE f_{\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{j}}(x,\\xi)\\bigg)   -   f_{\\xi}\\bigg(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\bigg)\\bigg|\n    \\leq \n    \\EE L_f \\norm{\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)}_2 \n    \\leq \n    \\frac{ L_f \\sigma_{g}}{\\sqrt{m}}.\n\\end{equation*}\n","sha256_utf8":"21d51bb9cde697782437f0a0e438d990494322ff6c282f1c2739bf79a8727060"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"光滑函数余项控制","source_units":[{"start":89609,"end":90025,"start_line":1192,"end_line":1200,"raw_tex":"If $f_\\xi(\\cdot)$ is $S$-Lipschitz smooth,\n\\begin{equation*}\n\\begin{aligned}\n\t&\\bigg|f_{\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{j}}(x,\\xi)\\bigg) \n    - f_{\\xi}\\bigg(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\bigg)\n    - \\nabla f_{\\xi} \\bigg(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\bigg)^\\top \\frac{1}{m}\\sum_{j=1}^m X_{j}(x)\\bigg| \\\\\n    \\leq &\\frac{S}{2} \\norm{\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)}_2^2.\n    \\end{aligned}\n\\end{equation*}\n","sha256_utf8":"8eb614be8de013165b1da4fd0ebc32fea8139935946e201c5e0a8c27e7fd9d90"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"消去线性项并得双侧界","source_units":[{"start":90025,"end":90618,"start_line":1201,"end_line":1209,"raw_tex":"$\\EE_{\\{\\eta_{j}\\}_{j=1}^m|\\xi}  \\nabla f_{\\xi} \\big(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\big)^\\top \\big(\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)\\big)=0$ since $x$ is independent of ${\\eta}_{j=1}^m$, we obtain,\n\\begin{equation*}\n    \\EE   f_{\\xi} \\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{j}}(x,\\xi)\\bigg)   -   f_{\\xi}\\bigg(\\EE_{\\eta|\\xi}g_{\\eta}(x,\\xi)\\bigg)\n    \\leq \n    \\EE \\bigg[ \\frac{S}{2} \\norm{\\frac{1}{m}\\sum_{j=1}^m X_{j}(x)}_2^2 \\bigg] \\\\\n    \\leq  \n    \\frac{S\\sigma_{g}^2}{2m}.\n\\end{equation*}\nFor the other side, we could derive a similar lower bound. As a result, (\\ref{eq:smooth_bias}) holds.\n","sha256_utf8":"1d8f666ce03694741c86ed2aedb72d3a76a36a3b74bbabc8dddbbce00d4e4193"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"最优性误差三项分解","source_units":[{"start":52004,"end":52634,"start_line":641,"end_line":651,"raw_tex":"Recall that $x^*$ and $\\hat x_{nm}$ are the minimizers of $F(x)$ and $\\hat F_{nm}(x)$, respectively. It's clear that $x^*$ has no randomness,  and $\\hat x_{nm}$ is a function of $\\{\\xi_i\\}_{i =1}^n, \\{\\eta_{ij}\\}_{j=1}^m$. \nWe decompose the error $F(\\hat x_{nm}) - F(x^*)$ in three terms, and analyze each term below:\n\\begin{equation*}\n\\begin{split}\nF(\\hat x_{nm}) \\!-\\! F(x^*)\n\t\t\\!=\\!\n\t\t\\underbrace{F(\\hat x_{nm})\\!-\\! \\hat F_{nm}(\\hat x_{nm})}_{:=\\Epsilon_1}+\n\t\t\\underbrace{\\hat F_{nm}(\\hat x_{nm})\\! -\\! \\hat F_{nm}(x^*)}_{:=\\Epsilon_2}+\n\t\t\\underbrace{\\hat F_{nm}(x^*)\\!- \\! F(x^*)}_{:=\\Epsilon_3}.\n\\end{split}\n\\end{equation*}\n","sha256_utf8":"930a210bf4b85785ab1a9e46359b116e95fbae0b9d4eeda4063f282ac4e60869"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"替换样本并分解泛化差","source_units":[{"start":52634,"end":54275,"start_line":652,"end_line":668,"raw_tex":"First, we use a stability argument and Lemma \\ref{lm:bias} to bound $\\EE \\Epsilon_1 = \\EE [F(\\hat x_{nm})- \\hat F_{nm}(\\hat x_{nm})]$. \nDefine \n\\begin{equation}\n\\begin{aligned}\n\\label{eq:Fnmk}\n     \\hat F_{nm}^{(k)}(x) \n     := \\frac{1}{n}\\sum_{i\\not = k}^n f_{\\xi_i}\\bigg(\\frac{1}{m}\\sum_{j =1}^m g_{\\eta_{ij}}(x,\\xi_i)\\bigg) + \\frac{1}{n}f_{\\xi_k^{\\prime}}\\bigg(\\frac{1}{m}\\sum_{j =1}^m g_{\\eta_{kj}^{\\prime}}(x,\\xi_k^{\\prime})\\bigg)\n     \\end{aligned}\n\\end{equation}\nas the empirical function by replacing the $k^{th}$ outer sample $\\xi_k$ with another i.i.d outer sample $\\xi_k^{\\prime}$, and replacing the corresponding  inner samples $\\{\\eta_{kj}\\}_{j=1}^m$  with $\\{\\eta_{kj}^{\\prime}\\}_{j=1}^m$, which are sampled from the conditional distribution of $\\PP(\\eta|\\xi_k^{\\prime})$ for a given sample $\\xi_k^\\prime$. Denote $\\hat x_{nm}^{(k)} := \\argmin_{x \\in \\mathcal{X}} \\hat F_{nm}^{(k)}(x)$. We decompose $\\EE \\Epsilon_1 = \\EE [F(\\hat x_{nm})- \\hat F_{nm}(\\hat x_{nm})]$ into three terms:\n\\begin{equation}\\label{eq:decompose_term_1}\n\\begin{split}\n\\EE \\Epsilon_1 = &\\EE\\bigg[\\frac{1}{n}\\sum_{k=1}^n F(\\hat x_{nm}) -\\frac{1}{n}\\sum_{k =1}^n  f_{\\xi_k} \\bigg( \\EE_{\\eta|\\xi_k} g_\\eta(\\hat x_{nm}^{(k)},\\xi_k) \\bigg)\\bigg]\\\\\n+&\\EE \\bigg[\\frac{1}{n}\\sum_{k =1}^n  f_{\\xi_k} \\bigg( \\EE_{\\eta|\\xi_k} g_\\eta(\\hat x_{nm}^{(k)},\\xi_k) \\bigg) -\\frac{1}{n}\\sum_{k =1}^n f_{\\xi_k}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k)\\bigg)\\bigg]\\\\\n+&\\EE \\bigg[\\frac{1}{n}\\sum_{k =1}^n f_{\\xi_k}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k)\\bigg) - \\hat F_{nm}(\\hat x_{nm})\\bigg].\n\\end{split}\n\\end{equation}\n","sha256_utf8":"081dda8a4991fb90a6afd1524f8e02921bff6f761301b0dea8d7da75cc9c387c"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"同分布独立性消去首项","source_units":[{"start":54275,"end":54700,"start_line":669,"end_line":676,"raw_tex":"Note that $\n    \\EE [F(\\hat x_{nm})] = \\EE [F(\\hat x_{nm}^{(k)})]\n$ since $\\xi_k$ and $\\xi_k^{\\prime}$ are i.i.d, which implies that  $\\hat x_{nm}$ and $\\hat x_{nm}^{(k)}$ follow an identical distribution.\nSince $\\hat x_{nm}^{(k)}$ is independent of $\\xi_k$, \n$\n\\EE [F(\\hat x_{nm}^{(k)})] = \\EE [f_{\\xi_k}(\\EE_{\\eta|\\xi_k}g(\\hat x_{nm}^{(k)},\\xi_k))]\n$\nfor any $k$. Then the first term in (\\ref{eq:decompose_term_1}) is $0$.\n","sha256_utf8":"d365e0dd15fba96817322e01df5bf39593c46de71c6c309fbff199c6eef05739"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"独立替换点的偏差控制","source_units":[{"start":54700,"end":55461,"start_line":677,"end_line":696,"raw_tex":"As $\\hat x_{nm}^{(k)}$ is independent of $\\{\\eta_{kj}\\}_{j=1}^m$, the second term in (\\ref{eq:decompose_term_1}) could be bounded by Lemma \\ref{lm:bias},\nit holds\n\\begin{equation}\n\\label{eq:biasboth}\n    \\EE \\bigg[ f_{\\xi_k} \\bigg( \\EE_{\\eta|\\xi_k} g_\\eta(\\hat x_{nm}^{(k)},\\xi_k) \\bigg)  \n    -   \n    f_{\\xi_k}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k) \\bigg)\\bigg]\n    \\leq \n    \\Delta(m).\n\\end{equation}\n\\if 0\nfor $S$-Lipschitz smooth $f_\\xi(\\cdot)$, we get\n\\begin{equation*}\n    \\EE \\bigg[ f_{\\xi_k} \\bigg( \\EE_{\\eta|\\xi_k} g_\\eta(\\hat x_{nm}^{(k)},\\xi_k) \\bigg)  \n    -   \n    f_{\\xi_k}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k) \\bigg)\\bigg] \n    \\leq \n    \\frac{S\\sigma_{g}^2}{2m}.\n\\end{equation*}\n\\fi\n","sha256_utf8":"947ef35dbb94d4fdf5e822e58cdea6901b97a43f43523d881c6523dd6635ad1c"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"替换目标差的上界","source_units":[{"start":55461,"end":56607,"start_line":697,"end_line":714,"raw_tex":"For the third term in (\\ref{eq:decompose_term_1}), by definition it implies\n\\begin{equation}\n\\label{eq:sca1}\n    \\begin{aligned}\n    \\hat F_{nm}(\\hat x_{nm}^{(k)})  - \\hat F_{nm}(\\hat x_{nm}) = &\\hat F_{nm}^{(k)}(\\hat x_{nm}^{(k)})  - \\hat F_{nm}^{(k)}(\\hat x_{nm}) \\\\\n    + &\n      \\frac{1}{n}f_{\\xi_k}\\! \\bigg(\\! \\frac{1}{m}\\! \\sum_{j =1}^m\\!  g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k)\\! \\bigg) \\! \n    - \\! \\frac{1}{n}f_{\\xi_k}\\! \\bigg(\\! \\frac{1}{m}\\sum_{j =1}^m g_{\\eta_{kj}}(\\hat x_{nm},\\xi_k)\\! \\bigg)\\\\\n     + &\\frac{1}{n}f_{\\xi_k^{\\prime}}\\! \\bigg(\\! \\frac{1}{m}\\sum_{j =1}^m g_{\\eta_{kj}^{\\prime}}(\\hat x_{nm},\\xi_k^{\\prime})\\! \\bigg) \\! \n   \\!  - \\! \n   \\frac{1}{n}f_{\\xi_k^{\\prime}}\\! \\bigg(\\! \\frac{1}{m}\\sum_{j =1}^m g_{\\eta_{kj}^{\\prime}}(\\hat x_{nm}^{(k)},\\xi_k^{\\prime})\\! \\bigg).\n    \\end{aligned}\n\\end{equation}\nBy Lipschitz continuity of $f_\\xi$ and $g_\\eta$ and that $\\hat F_{nm}^{(k)}(\\hat x_{nm}^{(k)})  - \\hat F_{nm}^{(k)}(\\hat x_{nm}) \\leq 0$, it holds\n\\begin{equation}\n\\label{eq:sca2}\n\\hat F_{nm}(\\hat x_{nm}^{(k)})  - \\hat F_{nm}(\\hat x_{nm}) \\leq \\frac{2}{n}L_f L_g \\norm{\\hat x_{nm}^{(k)}-\\hat x_{nm}}_2.\n\\end{equation}\n","sha256_utf8":"00ecf0b06a1650a01f8ec26edcfbad0f02876da4e836a4838fd0d1db7c69da05"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"误差界推出解的稳定性","source_units":[{"start":56607,"end":57104,"start_line":715,"end_line":724,"raw_tex":"Since $\\hat x_{nm}$ is the unique minimizer of $\\hat F_{nm}(x)$, and $\\hat F_{nm}(x)$ satisfies QG condition with parameter $\\mu$, we have\n\\begin{equation}\n\\label{eq:strong}\n    \\hat F_{nm}(\\hat x_{nm}^{(k)})  - \\hat F_{nm}(\\hat x_{nm}) \\geq \\mu \\norm{\\hat x_{nm}^{(k)}-\\hat x_{nm}}_2^{1+\\delta}.\n\\end{equation}\nCombining with (\\ref{eq:sca2}), we obtain\n\\begin{equation}\n\\label{eq:strong2}\n\\norm{\\hat x_{nm}^{(k)}-\\hat x_{nm}}_2 \\leq \\bigg(\\frac{2L_f L_g}{\\mu n }\\bigg)^{1/\\delta}.\n\\end{equation}\n","sha256_utf8":"ab4dde98192fb88b0baad16054a271124a6a9099a2eac0e4a01e5bc2a75e5e86"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"稳定性控制第三项","source_units":[{"start":57104,"end":57519,"start_line":725,"end_line":732,"raw_tex":"By Lipschitz continuity of $f_\\xi(\\cdot)$ and $g_\\eta(\\cdot,\\xi)$, and definition of $\\hat F_{nm}(\\hat x_{nm})$, we obtain\n\\begin{equation}\n\\label{eq:sca}\n\\begin{split}\n   \\EE \\bigg[\\frac{1}{n}\\sum_{k =1}^n f_{\\xi_k}\\bigg(\\frac{1}{m}\\sum_{j=1}^m g_{\\eta_{kj}}(\\hat x_{nm}^{(k)},\\xi_k)\\bigg) - \\hat F_{nm}(\\hat x_{nm})\\bigg]\n    \\leq  L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}.\n\\end{split}\n\\end{equation}\n","sha256_utf8":"264cd34255d76ffc7544cb2759b69362b6f77011e1ae77b2fa4f2660d3749b95"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"合并经验最优点泛化界","source_units":[{"start":57520,"end":58181,"start_line":734,"end_line":756,"raw_tex":"Combining (\\ref{eq:decompose_term_1}), (\\ref{eq:sca}), and (\\ref{eq:biasboth}), we obtain\n\\begin{equation}\n\\label{eq:scAl}\n    \\EE \\Epsilon_1\n    \\leq \n    L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}+\\Delta(m).\n\\end{equation}\n\\if 0\nfor $L_f$-Lipschitz continuous $f_\\xi(\\cdot)$,\n\\begin{equation}\n\\label{eq:scAl}\n    \\EE \\Epsilon_1\n    \\leq \n    L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}+\\frac{L_{f}\\sigma_{g}}{\\sqrt{m}};\n\\end{equation}\nfor $S$-Lipschitz smooth $f_\\xi(\\cdot)$,\n\\begin{equation}\n\\label{eq:scAs}\n    \\EE \\Epsilon_1\n    \\leq \n    L_f L_g\\bigg(\\frac{2L_f L_g}{\\mu n}\\bigg)^{1/\\delta}+\\frac{S\\sigma_{g}^2}{2m}.\n\\end{equation}\n\\fi\n","sha256_utf8":"534388c2fb0ea79b1b2d05650a4a0bbb270c2b7314de6056f750dbe6112b59a5"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"经验最优性控制第二项","source_units":[{"start":58182,"end":58372,"start_line":758,"end_line":762,"raw_tex":"Second, by optimality of $\\hat x_{nm}$ of $\\hat F_{nm}$, we have\n\\begin{equation}\n\\label{eq:scB}\n    \\EE \\Epsilon_2 = \\EE [\\hat F_{nm}(\\hat x_{nm})- \\hat F_{nm}(x^*)] \\leq 0.\n\\end{equation}\n","sha256_utf8":"18b8bf436a1ca4d75cddb94e9410fcededa94002cb16a2750e238dbf36041fdf"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"固定真最优点的偏差界","source_units":[{"start":58373,"end":59276,"start_line":764,"end_line":791,"raw_tex":"Next, we bound $\\EE \\Epsilon_3$.\nDefine\n$\n\\hat{F}_n(x):=\\frac{1}{n}\\sum_{i=1}^nf_{\\xi_i}\\big(\\EE_{\\eta|\\xi_i}[g_\\eta(x,\\xi_i)]\\big).\n$\nNotice that $x^*$ is independent of $\\{\\eta_{ij}\\}_{j=1}^m$ for any $i = \\{1,\\cdots,n\\}$ and $\\EE [\\hat F_n(x^*) - F(x^*)]=0$. By Lemma \\ref{lm:bias}, it holds\n\\begin{equation}\n\\label{eq:scC}\n    \\EE \\Epsilon_3 = \\EE[ \\hat F_{nm}(x^*)-\\hat F_n(x) ]+ \\EE[\\hat F_n(x) - F(x)]\n    \\leq \n    \\Delta(m);\n\\end{equation}\n\\if 0\nfor $L_f$-Lipschitz continuous $f_\\xi(\\cdot)$,\n\\begin{equation}\n\\label{eq:scCL}\n    \\EE \\Epsilon_3 = \\EE[ \\hat F_{nm}(x^*)-\\hat F_n(x) ]+ \\EE[\\hat F_n(x) - F(x)]\n    \\leq \n    \\frac{L_{f}\\sigma_{g}}{\\sqrt{m}};\n\\end{equation}\nfor $S$-Lipschitz smooth $f_\\xi(\\cdot)$, we have\n\\begin{equation}\n\\label{eq:scCS}\n    \\EE \\Epsilon_3 = \\EE[ \\hat F_{nm}(x^*)-\\hat F_n(x) ]+ \\EE[\\hat F_n(x) - F(x)]\n    \\leq \n    \\frac{S\\sigma_{g}^2}{2m}.\n\\end{equation}\n\\fi\n","sha256_utf8":"74b6f1e71e665f049f1ee1b70617bebdaf9eae8a9fe62a6bbcea7f6024ffabf4"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"马尔可夫不等式得概率界","source_units":[{"start":59276,"end":59389,"start_line":792,"end_line":792,"raw_tex":"Combining (\\ref{eq:scAl}), (\\ref{eq:scB}), (\\ref{eq:scC}), with Markov inequality, we obtain the desired result.\n","sha256_utf8":"875fccac0728a91651a6ad388b2258dd2567a098f0a9a57116949ebd3b8bb170"}]},{"id":"S031","semantic_type":"external_fact","kind":"external_fact","title":"马尔可夫不等式","source_units":[],"declared_external":true,"declaration_text":["For a nonnegative random variable Y and ε>0, P(Y≥ε)≤E[Y]/ε."]},{"id":"S032","semantic_type":"external_fact","kind":"external_fact","title":"条件矩与期望基本性质","source_units":[],"declared_external":true,"declaration_text":["For conditionally independent centered samples X_j, E[||m^{-1}Σ_j X_j||₂²|ξ]=m^{-2}Σ_j E[||X_j||₂²|ξ]. Jensen's inequality implies E||Z||₂≤sqrt(E||Z||₂²). Conditional expectations satisfy the tower property."]},{"id":"S033","semantic_type":"assumption","kind":"assumption","title":"偏差引理光滑情形假设","source_units":[{"start":33920,"end":33985,"start_line":395,"end_line":395,"raw_tex":"If additionally, $f_\\xi(\\cdot)$ is $S$-Lipschitz smooth, we have\n","sha256_utf8":"6de6cc55932983952ef768faf7e972aace62545423fa7627bb34f45529c125c6"}]},{"id":"S034","semantic_type":"assumption","kind":"assumption","title":"主定理光滑情形假设","source_units":[{"start":50953,"end":51032,"start_line":618,"end_line":618,"raw_tex":"If additionally, $f_\\xi(\\cdot)$ is $S$-Lipschitz smooth, then we further have \n","sha256_utf8":"a89b2faa47bb78f0660d0f48bbb4b2ca50ab3e6868b83dcff9df0aff50dbec2d"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S003","target_id":"S020","kind":"supports"},{"source_id":"S004","target_id":"S020","kind":"supports"},{"source_id":"S004","target_id":"S021","kind":"supports"},{"source_id":"S003","target_id":"S022","kind":"supports"},{"source_id":"S004","target_id":"S022","kind":"supports"},{"source_id":"S005","target_id":"S014","kind":"supports"},{"source_id":"S005","target_id":"S017","kind":"supports"},{"source_id":"S005","target_id":"S018","kind":"supports"},{"source_id":"S006","target_id":"S013","kind":"supports"},{"source_id":"S006","target_id":"S014","kind":"supports"},{"source_id":"S006","target_id":"S016","kind":"supports"},{"source_id":"S006","target_id":"S017","kind":"supports"},{"source_id":"S006","target_id":"S024","kind":"supports"},{"source_id":"S006","target_id":"S026","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S025","kind":"supports"},{"source_id":"S009","target_id":"S025","kind":"supports"},{"source_id":"S010","target_id":"S023","kind":"supports"},{"source_id":"S010","target_id":"S027","kind":"supports"},{"source_id":"S010","target_id":"S029","kind":"supports"},{"source_id":"S011","target_id":"S023","kind":"supports"},{"source_id":"S011","target_id":"S029","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S014","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S011","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S015","kind":"supports"},{"source_id":"S018","target_id":"S015","kind":"supports"},{"source_id":"S019","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S011","kind":"supports"},{"source_id":"S020","target_id":"S030","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S023","kind":"supports"},{"source_id":"S021","target_id":"S024","kind":"supports"},{"source_id":"S021","target_id":"S027","kind":"supports"},{"source_id":"S022","target_id":"S027","kind":"supports"},{"source_id":"S023","target_id":"S027","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S004","target_id":"S028","kind":"supports"},{"source_id":"S003","target_id":"S029","kind":"supports"},{"source_id":"S004","target_id":"S029","kind":"supports"},{"source_id":"S027","target_id":"S030","kind":"supports"},{"source_id":"S028","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S013","kind":"supports"},{"source_id":"S032","target_id":"S014","kind":"supports"},{"source_id":"S032","target_id":"S016","kind":"supports"},{"source_id":"S032","target_id":"S017","kind":"supports"},{"source_id":"S032","target_id":"S019","kind":"supports"},{"source_id":"S032","target_id":"S022","kind":"supports"},{"source_id":"S033","target_id":"S014","kind":"supports"},{"source_id":"S033","target_id":"S018","kind":"supports"},{"source_id":"S034","target_id":"S023","kind":"supports"},{"source_id":"S034","target_id":"S029","kind":"supports"},{"source_id":"S020","target_id":"S002","kind":"supports"},{"source_id":"S021","target_id":"S002","kind":"supports"},{"source_id":"S022","target_id":"S002","kind":"supports"},{"source_id":"S023","target_id":"S002","kind":"supports"},{"source_id":"S024","target_id":"S002","kind":"supports"},{"source_id":"S025","target_id":"S002","kind":"supports"},{"source_id":"S026","target_id":"S002","kind":"supports"},{"source_id":"S027","target_id":"S002","kind":"supports"},{"source_id":"S028","target_id":"S002","kind":"supports"},{"source_id":"S029","target_id":"S002","kind":"supports"},{"source_id":"S030","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S001","kind":"supports"}]}}
{"record_id":"1907.03132v1","paper":{"title":"On the Convergence of Stochastic Gradient Descent for Nonlinear Ill-Posed Problems","authors":["Bangti Jin","Zehui Zhou","Jun Zou"],"arxiv_id":"1907.03132","version":"v1","journal":"SIAM Journal on Optimization","doi":"10.1137/19m1271798","abstract_url":"https://arxiv.org/abs/1907.03132v1","pdf_url":"https://arxiv.org/pdf/1907.03132v1"},"context":{"raw_tex":"This work is concerned with the numerical solution of the system of nonlinear ill-posed operator equations\r\n\\begin{equation}\\label{eqn:nonlin}\r\n  F_i(x) = y_i^\\dag,\\quad i =1,\\ldots, n,\r\n\\end{equation}\r\nwhere each $F_i: \\mathcal{D}(F_i)\\to Y$ is a nonlinear mapping\r\nwith its domain $\\mathcal{D}(F_i)\\subset X$, and $X$ and $Y$ are Hilbert spaces with inner\r\nproducts $\\langle\\cdot,\\cdot\\rangle$ and norms $\\|\\cdot\\|$, respectively.\r\nThe number %$n\\in\\mathbb{N}$\r\n$n$ of nonlinear equations in \\eqref{eqn:nonlin} can potentially be very large.\r\nThe notation $y_i^\\dag \\in Y$ denotes the exact data (corresponding to the reference solution\r\n$x^\\dag\\in X$ to be defined below, i.e., $y^\\dag=F(x^\\dag)$). Equivalently, problem \\eqref{eqn:nonlin} can be rewritten as\r\n\\begin{equation}\\label{eqn:nonlin-gen}\r\n  F(x) = y^\\dag,\r\n\\end{equation}\r\nwith the operator $F: X \\to Y^n$ ($Y^n$ denotes the product space $Y\\times \\cdots \\times Y$) and $y^\\dag\\in Y^n$ defined by\r\n\\begin{equation*}\r\n  F(x)=\\frac{1}{\\sqrt{n}}\\left(\\begin{array}{c}F_1(x)\\\\ \\ldots\\\\ F_n(x)\\end{array}\\right)\\quad \\mbox{and}\\quad y^\\dag=\\frac{1}{\\sqrt{n}}\\left(\\begin{array}{c}y_1^\\dag\\\\ \\ldots\\\\ y_n^\\dag\\end{array}\\right),\r\n\\end{equation*}\r\nrespectively. The scaling $n^{-\\frac12}$ above is introduced for the convenience of later discussions. In practice,\r\ninstead of the exact data $y^\\dag$, we have access only to the noisy data $y^\\delta$ of a noise level\r\n$\\delta\\ge 0$, namely\r\n\\begin{equation*}\r\n  \\|y^\\delta - y^\\dag\\| = \\delta\\,.\r\n\\end{equation*}\r\n\n\nindex $i_k$ of the equation  at the $k$th iteration is drawn uniformly from the index set $\\{1,\\ldots,n\\}$, and $\\eta_k>0$ is the\r\ncorresponding step size. This algorithm has demonstrated very encouraging numerical results in \\cite{ChenLiLiu:2018} for diffuse optical\r\n\n\n\\begin{algorithm}\r\n  \\caption{Stochastic gradient method for problem \\eqref{eqn:nonlin}.\\label{alg:sgd}}\r\n  \\begin{algorithmic}[1]\r\n    \\STATE Given initial guess $x_1$.\r\n    \\FOR{$k=1,2,\\ldots$}\r\n      \\STATE Randomly draw an index $i_k$;\r\n      \\STATE Update the iterate $x_k^\\delta$ by\r\n      \\begin{equation}\\label{eqn:sgd}\r\n        x_{k+1}^\\delta = x_k^\\delta - \\eta_k F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta);\r\n      \\end{equation}\r\n      \\STATE Check the stopping criterion.\r\n    \\ENDFOR\r\n  \\end{algorithmic}\r\n\\end{algorithm}\n\nThroughout, we denote the iterate for the exact data $y^\\dag$ by $x_k$. The notation $\\mathcal{F}_k$ denotes the filtration generated\r\nby the random indices $\\{i_1,\\ldots,i_{k-1}\\}$ up to the $(k-1)$th iteration. The notation $c$, with or without a subscript, denotes\r\na generic constant, which may differ at each occurrence, but it is always independent of the noise level $\\delta$ and the iteration number $k$.\r\n\n\nassumptions on the nonlinear operators $F_i$ and the reference solution $x^\\dag$. Since the solution to problem\r\n\\eqref{eqn:nonlin} may be nonunique, the reference solution $x^\\dag$ is taken to be the minimum norm solution (with\r\nrespect to the initial guess $x_1$), which is known to be unique under Assumption \\ref{ass:sol}(ii) below \\cite{HankeNeubauerScherzer:1995}.\r\n\n\n\\begin{assumption}\\label{ass:sol}\r\nThe following conditions hold:\r\n\\begin{itemize}\r\n  \\item[$\\rm(i)$] The operators $F_i$, $i=1,\\ldots,n$, are continuous, with continuous and bounded derivatives on $X$.\r\n  \\item[$\\rm(ii)$] There exists an $\\eta\\in(0,\\frac12)$ such that for any $x,\\tilde x\\in X$,\r\n  \\begin{equation}\\label{eqn:tangential-cone}\r\n    \\|F(x)-F(\\tilde x)-F'(\\tilde x)(x-\\tilde x)\\|\\leq \\eta\\|F(x)-F(\\tilde x)\\|.\r\n  \\end{equation}\r\n  \\item[$\\rm(iii)$] There are a family of uniformly bounded operators $R^i_x$ such that for any $x\\in X$,\r\n  \\begin{equation*}\r\n    F_i'(x) = R_x^iF_i'(x^\\dag)\r\n  \\end{equation*}\r\n  and $R_x=\\mathrm{diag}(R_x^i):Y^n\\to Y^n$, with {\\rm(}with $\\|\\cdot\\|$ denoting the operator norm on $Y^n${\\rm)}\r\n  \\begin{equation*}\r\n    \\|R_x-I\\|\\leq c_R\\|x-x^\\dag\\|.\r\n  \\end{equation*}\r\n\\item[$\\rm(iv)$] The following source condition holds: there exist some $\\nu\\in (0,\\frac12)$ and $w\\in X$ such that\r\n  \\begin{equation*}\r\n    x^\\dag - x_1 = (F'(x^\\dag)^*F'(x^\\dag))^\\nu w.\r\n  \\end{equation*}\r\n\\end{itemize}\r\n\\end{assumption}\n\n\\begin{assumption}\\label{ass:stepsize}\r\nThe step sizes $\\{\\eta_k\\}_{k\\geq 1}$ satisfy one of the following properties.\r\n\\begin{itemize}\r\n\\item[$\\rm(i)$] $\\eta_k\\max_i\\sup_x\\|F_i'(x)\\|^2<1$ and $\\sum_{k=1}^\\infty \\eta_k =\\infty$.\r\n\\item[$\\rm(ii)$] $\\eta_k=\\eta_0k^{-\\alpha}$, with $\\alpha\\in(0,1)$ and $\\eta_0\\leq (\\max_i\\sup_x\\|F_i'(x)\\|^2)^{-1}$.\r\n\\end{itemize}\r\n\\end{assumption}\n\nDue to the random choice of the index $i_k$, the SGD iterate $x_k^\\delta$ is a random variable. There are\r\nseveral different ways to measure the convergence. We shall employ the mean squared norm defined by\r\n$\\E[\\|\\cdot\\|^2]$, where the expectation $\\E[\\cdot]$ is with respect to the filtration $\\mathcal{F}_k$\r\ngenerated by the random indices $i_j$, $j=1,\\ldots,k-1$. Clearly, the iterate $x_k^\\delta$ is measurable\r\nwith respect to $\\mathcal{F}_k$.\r\n\n\nthe notation $\\mathcal{N}(\\cdot)$ denotes the kernel of a linear operator.\r\n","units":[{"start":3824,"end":5346,"start_line":83,"end_line":106,"raw_tex":"This work is concerned with the numerical solution of the system of nonlinear ill-posed operator equations\r\n\\begin{equation}\\label{eqn:nonlin}\r\n  F_i(x) = y_i^\\dag,\\quad i =1,\\ldots, n,\r\n\\end{equation}\r\nwhere each $F_i: \\mathcal{D}(F_i)\\to Y$ is a nonlinear mapping\r\nwith its domain $\\mathcal{D}(F_i)\\subset X$, and $X$ and $Y$ are Hilbert spaces with inner\r\nproducts $\\langle\\cdot,\\cdot\\rangle$ and norms $\\|\\cdot\\|$, respectively.\r\nThe number %$n\\in\\mathbb{N}$\r\n$n$ of nonlinear equations in \\eqref{eqn:nonlin} can potentially be very large.\r\nThe notation $y_i^\\dag \\in Y$ denotes the exact data (corresponding to the reference solution\r\n$x^\\dag\\in X$ to be defined below, i.e., $y^\\dag=F(x^\\dag)$). Equivalently, problem \\eqref{eqn:nonlin} can be rewritten as\r\n\\begin{equation}\\label{eqn:nonlin-gen}\r\n  F(x) = y^\\dag,\r\n\\end{equation}\r\nwith the operator $F: X \\to Y^n$ ($Y^n$ denotes the product space $Y\\times \\cdots \\times Y$) and $y^\\dag\\in Y^n$ defined by\r\n\\begin{equation*}\r\n  F(x)=\\frac{1}{\\sqrt{n}}\\left(\\begin{array}{c}F_1(x)\\\\ \\ldots\\\\ F_n(x)\\end{array}\\right)\\quad \\mbox{and}\\quad y^\\dag=\\frac{1}{\\sqrt{n}}\\left(\\begin{array}{c}y_1^\\dag\\\\ \\ldots\\\\ y_n^\\dag\\end{array}\\right),\r\n\\end{equation*}\r\nrespectively. The scaling $n^{-\\frac12}$ above is introduced for the convenience of later discussions. In practice,\r\ninstead of the exact data $y^\\dag$, we have access only to the noisy data $y^\\delta$ of a noise level\r\n$\\delta\\ge 0$, namely\r\n\\begin{equation*}\r\n  \\|y^\\delta - y^\\dag\\| = \\delta\\,.\r\n\\end{equation*}\r\n","sha256_utf8":"487b6e2db454a33603ee7f1de3927f07ac6ab69e6c58980e707d78b6a9a3a74d"},{"start":6731,"end":7000,"start_line":119,"end_line":120,"raw_tex":"index $i_k$ of the equation  at the $k$th iteration is drawn uniformly from the index set $\\{1,\\ldots,n\\}$, and $\\eta_k>0$ is the\r\ncorresponding step size. This algorithm has demonstrated very encouraging numerical results in \\cite{ChenLiLiu:2018} for diffuse optical\r\n","sha256_utf8":"3d66a4935908843dfb7b5a6c52200690db455035e5d734d14a139ce54653bb22"},{"start":7611,"end":8161,"start_line":127,"end_line":140,"raw_tex":"\\begin{algorithm}\r\n  \\caption{Stochastic gradient method for problem \\eqref{eqn:nonlin}.\\label{alg:sgd}}\r\n  \\begin{algorithmic}[1]\r\n    \\STATE Given initial guess $x_1$.\r\n    \\FOR{$k=1,2,\\ldots$}\r\n      \\STATE Randomly draw an index $i_k$;\r\n      \\STATE Update the iterate $x_k^\\delta$ by\r\n      \\begin{equation}\\label{eqn:sgd}\r\n        x_{k+1}^\\delta = x_k^\\delta - \\eta_k F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta);\r\n      \\end{equation}\r\n      \\STATE Check the stopping criterion.\r\n    \\ENDFOR\r\n  \\end{algorithmic}\r\n\\end{algorithm}","sha256_utf8":"4d242ab2e4a214675c37ba4e193dc325880c02fef484fc5c3bd0229d81af6d2e"},{"start":12647,"end":13061,"start_line":192,"end_line":194,"raw_tex":"Throughout, we denote the iterate for the exact data $y^\\dag$ by $x_k$. The notation $\\mathcal{F}_k$ denotes the filtration generated\r\nby the random indices $\\{i_1,\\ldots,i_{k-1}\\}$ up to the $(k-1)$th iteration. The notation $c$, with or without a subscript, denotes\r\na generic constant, which may differ at each occurrence, but it is always independent of the noise level $\\delta$ and the iteration number $k$.\r\n","sha256_utf8":"5c582dcdf727dc03f3c5c02b25620006a4906173b0fda8cfe36084681aa8c487"},{"start":13929,"end":14301,"start_line":205,"end_line":207,"raw_tex":"assumptions on the nonlinear operators $F_i$ and the reference solution $x^\\dag$. Since the solution to problem\r\n\\eqref{eqn:nonlin} may be nonunique, the reference solution $x^\\dag$ is taken to be the minimum norm solution (with\r\nrespect to the initial guess $x_1$), which is known to be unique under Assumption \\ref{ass:sol}(ii) below \\cite{HankeNeubauerScherzer:1995}.\r\n","sha256_utf8":"806a4dd7769a28a90ffd8d9f2501857d2ca44c79165f019b09eac42807768672"},{"start":14303,"end":15359,"start_line":209,"end_line":230,"raw_tex":"\\begin{assumption}\\label{ass:sol}\r\nThe following conditions hold:\r\n\\begin{itemize}\r\n  \\item[$\\rm(i)$] The operators $F_i$, $i=1,\\ldots,n$, are continuous, with continuous and bounded derivatives on $X$.\r\n  \\item[$\\rm(ii)$] There exists an $\\eta\\in(0,\\frac12)$ such that for any $x,\\tilde x\\in X$,\r\n  \\begin{equation}\\label{eqn:tangential-cone}\r\n    \\|F(x)-F(\\tilde x)-F'(\\tilde x)(x-\\tilde x)\\|\\leq \\eta\\|F(x)-F(\\tilde x)\\|.\r\n  \\end{equation}\r\n  \\item[$\\rm(iii)$] There are a family of uniformly bounded operators $R^i_x$ such that for any $x\\in X$,\r\n  \\begin{equation*}\r\n    F_i'(x) = R_x^iF_i'(x^\\dag)\r\n  \\end{equation*}\r\n  and $R_x=\\mathrm{diag}(R_x^i):Y^n\\to Y^n$, with {\\rm(}with $\\|\\cdot\\|$ denoting the operator norm on $Y^n${\\rm)}\r\n  \\begin{equation*}\r\n    \\|R_x-I\\|\\leq c_R\\|x-x^\\dag\\|.\r\n  \\end{equation*}\r\n\\item[$\\rm(iv)$] The following source condition holds: there exist some $\\nu\\in (0,\\frac12)$ and $w\\in X$ such that\r\n  \\begin{equation*}\r\n    x^\\dag - x_1 = (F'(x^\\dag)^*F'(x^\\dag))^\\nu w.\r\n  \\end{equation*}\r\n\\end{itemize}\r\n\\end{assumption}","sha256_utf8":"ebdfffd8d5d9535ebe0460b7dbdd376845ce25980e088ae64b8691e9fc86b03a"},{"start":17483,"end":17863,"start_line":256,"end_line":262,"raw_tex":"\\begin{assumption}\\label{ass:stepsize}\r\nThe step sizes $\\{\\eta_k\\}_{k\\geq 1}$ satisfy one of the following properties.\r\n\\begin{itemize}\r\n\\item[$\\rm(i)$] $\\eta_k\\max_i\\sup_x\\|F_i'(x)\\|^2<1$ and $\\sum_{k=1}^\\infty \\eta_k =\\infty$.\r\n\\item[$\\rm(ii)$] $\\eta_k=\\eta_0k^{-\\alpha}$, with $\\alpha\\in(0,1)$ and $\\eta_0\\leq (\\max_i\\sup_x\\|F_i'(x)\\|^2)^{-1}$.\r\n\\end{itemize}\r\n\\end{assumption}","sha256_utf8":"5bac8990b519db696792ce9a2265f2a57f9ecd89446138eb76fe6ea6121967cb"},{"start":17867,"end":18319,"start_line":264,"end_line":268,"raw_tex":"Due to the random choice of the index $i_k$, the SGD iterate $x_k^\\delta$ is a random variable. There are\r\nseveral different ways to measure the convergence. We shall employ the mean squared norm defined by\r\n$\\E[\\|\\cdot\\|^2]$, where the expectation $\\E[\\cdot]$ is with respect to the filtration $\\mathcal{F}_k$\r\ngenerated by the random indices $i_j$, $j=1,\\ldots,k-1$. Clearly, the iterate $x_k^\\delta$ is measurable\r\nwith respect to $\\mathcal{F}_k$.\r\n","sha256_utf8":"da947cf6a4f1b38f4fa9627f55eebb94c22bcbc29f563e6429b5d2d093f156bd"},{"start":18558,"end":18634,"start_line":272,"end_line":272,"raw_tex":"the notation $\\mathcal{N}(\\cdot)$ denotes the kernel of a linear operator.\r\n","sha256_utf8":"356878a63ac1c58e7739e3fae8c402108e0e6e36390f0d42db73140533771842"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}[convergence for noisy data]\\label{thm:conv-noisy}\r\nLet Assumptions \\ref{ass:sol}(i)-(ii) and \\ref{ass:stepsize}(i) be fulfilled. If the stopping index $k(\\delta)\\in\\mathbb{N}$ is chosen such that\r\n\\begin{equation*}\r\n  \\lim_{\\delta\\to0^+}k(\\delta)=\\infty \\quad\\mbox{and}\\quad \\lim_{\\delta\\to0^+}\\delta^2\\sum_{i=1}^{k(\\delta)}\\eta_i = 0,\r\n\\end{equation*}\r\nthen there exists a solution $x^*\\in X$ to problem \\eqref{eqn:nonlin} such that\r\n\\begin{equation*}\r\n  \\lim_{\\delta\\to 0^+}\\E[\\|x_{k(\\delta)}^\\delta - x^*\\|^2]=0.\r\n\\end{equation*}\r\nFurther, if $\\mathcal{N}(F'(x^\\dag))\\subset\\mathcal{N}(F'(x))$, then\r\n\\begin{equation*}\r\n  \\lim_{\\delta\\to0^+}\\E[\\|x_{k(\\delta)}^\\delta -x^\\dag\\|^2]=0.\r\n\\end{equation*}\r\n\\end{theorem}","units":[{"start":18634,"end":19367,"start_line":273,"end_line":286,"raw_tex":"\\begin{theorem}[convergence for noisy data]\\label{thm:conv-noisy}\r\nLet Assumptions \\ref{ass:sol}(i)-(ii) and \\ref{ass:stepsize}(i) be fulfilled. If the stopping index $k(\\delta)\\in\\mathbb{N}$ is chosen such that\r\n\\begin{equation*}\r\n  \\lim_{\\delta\\to0^+}k(\\delta)=\\infty \\quad\\mbox{and}\\quad \\lim_{\\delta\\to0^+}\\delta^2\\sum_{i=1}^{k(\\delta)}\\eta_i = 0,\r\n\\end{equation*}\r\nthen there exists a solution $x^*\\in X$ to problem \\eqref{eqn:nonlin} such that\r\n\\begin{equation*}\r\n  \\lim_{\\delta\\to 0^+}\\E[\\|x_{k(\\delta)}^\\delta - x^*\\|^2]=0.\r\n\\end{equation*}\r\nFurther, if $\\mathcal{N}(F'(x^\\dag))\\subset\\mathcal{N}(F'(x))$, then\r\n\\begin{equation*}\r\n  \\lim_{\\delta\\to0^+}\\E[\\|x_{k(\\delta)}^\\delta -x^\\dag\\|^2]=0.\r\n\\end{equation*}\r\n\\end{theorem}","sha256_utf8":"2b036de0d537dbe5cd9b7a65ce26bad425b9fc2a140d7fa78b9131603b5d648d"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\\label{lem:linear}\r\nLet Assumptions \\ref{ass:sol}(i) and (ii) be fulfilled. Then the following statements hold.\r\n\\begin{itemize}\r\n  \\item[$\\rm(i)$] The following inequalities hold:\r\n  \\begin{equation*}\r\n  \\frac{1}{1+\\eta}\\|F'(x)(x-\\tilde x)\\|\\leq \\|F(x)-F(\\tilde x)\\|\\leq \\frac{1}{1-\\eta}\\|F'(x)(x-\\tilde x)\\|.\r\n  \\end{equation*}\r\n  \\item[$\\rm(ii)$]If $x^*$ is a solution of \\eqref{eqn:nonlin}, then any other solution $\\tilde x^*$ satisfies\r\n$x^*-\\tilde x^* \\in \\mathcal{N}(F'(x^*))$, and vice versa.\r\n\\end{itemize}\r\n\\end{lemma}\n\n\\begin{prop}\\label{prop:mono-exact}\r\nLet Assumptions \\ref{ass:sol}(i)-(ii) and \\ref{ass:stepsize}(i) be fulfilled. Then for any solution $x^*$ to problem \\eqref{eqn:nonlin}, there holds\r\n\\begin{equation*}\r\n  \\E[\\|x^*-x_{k+1}^\\delta\\|^2] - \\E[\\|x^*-x_{k}^\\delta\\|^2]\r\n   \\leq  -(1-2\\eta)\\eta_k\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2] + { 2\\eta_k(1+\\eta)\\delta\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2]^\\frac{1}{2}}.\r\n\\end{equation*}\r\n\\end{prop}\n\n\\begin{proof}\r\nCompleting the square gives\r\n\\begin{align*}\r\n    \\|x^*-x_{k+1}^\\delta\\|^2 - \\|x^*-x_{k}^\\delta\\|^2 = & 2\\langle x_k^\\delta - x^*,x_{k+1}^\\delta-x_k^\\delta\\rangle + \\|x_{k+1}^\\delta-x_k^\\delta\\|^2.\r\n\\end{align*}\r\nBy the definition of the SGD iterate $x_{k}^\\delta$ in \\eqref{eqn:sgd}, there holds\r\n\\begin{align*}\r\n   & \\|x^*-x_{k+1}^\\delta\\|^2 - \\|x^*-x_{k}^\\delta\\|^2\\\\\r\n  = & -2\\eta_k\\langle x_k^\\delta-x^*,  F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)\\rangle + \\eta_k^2\\| F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)\\|^2 \\\\\r\n  = & -2\\eta_k\\langle F_{i_k}'(x_k^\\delta)(x_k^\\delta-x^*), F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle + \\eta_k^2\\|F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)\\|^2.\r\n\\end{align*}\r\nNext we split the factor $-F'_{i_k}(x_k^\\delta)(x_k^\\delta-x^*)$ into two terms\r\n\\begin{equation*}\r\n  F_{i_k}'(x_k^\\delta)(x_k^\\delta-x^*) =  (F_{i_k}(x_k^\\delta)- F_{i_k}(x^*)) + (F_{i_k}(x^*)-F_{i_k}(x_k^\\delta)-F_{i_k}'(x_k^\\delta)(x^* - x_k^\\delta)).\r\n\\end{equation*}\r\nCombining the last two identities yields\r\n\\begin{align*}\r\n    &\\|x^*-x_{k+1}^\\delta\\|^2 - \\|x^*-x_{k}^\\delta\\|^2\\\\\r\n  = & -2\\eta_k\\langle F_{i_k}(x_k^\\delta)-F_{i_k}(x^*), F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle +  {\\eta_k^2\\|F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)\\|^2}\\\\\r\n   & - 2\\eta_k\\langle y_{i_k}^\\dag-F_{i_k}(x_k^\\delta)-F_{i_k}'(x_k^\\delta)(x^*-x_k^\\delta),F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle \\\\\r\n  = & -2\\eta_k\\langle F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta, F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle +  {\\eta_k^2\\|F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)\\|^2}\\\\\r\n   & -2\\eta_k\\langle y_{i_k}^\\delta-y_{i_k}^\\dag , F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle\\\\\r\n   & - 2\\eta_k\\langle y_{i_k}^\\dag-F_{i_k}(x_k^\\delta)-F_{i_k}'(x_k^\\delta)(x^*-x_k^\\delta),F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle \\\\\r\n \\leq  & -\\eta_k\\langle F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta, F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle\r\n    -2\\eta_k\\langle y_{i_k}^\\delta-y_{i_k}^\\dag, F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle\\\\\r\n   & - 2\\eta_k\\langle y_{i_k}^\\dag-F_{i_k}(x_k^\\delta)-F_{i_k}'(x_k^\\delta)(x^*-x_k^\\delta),F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle,\r\n\\end{align*}\r\nwhere the inequality follows from the condition $\\eta_k\\|F_{i_k}'(x)\\|^2<1$ in Assumption \\ref{ass:stepsize}(i).\r\nThus, by the measurability of the iterate $x_k$ with respect to the filtration\r\n$\\mathcal{F}_{k}$ and the Cauchy-Schwarz inequality, we have\r\n\\begin{align*}\r\n   &{\\E[\\|x^*-x_{k+1}^\\delta\\|^2 - \\|x^*-x_{k}^\\delta\\|^2|\\mathcal{F}_{k}]} \\\\\r\n    \\leq &-\\frac{\\eta_k}{n}\\sum_{i=1}^n\\langle F_{i}(x_k^\\delta)-y_{i}^\\delta,F_{i}(x_k^\\delta)-y_{i}^\\delta\\rangle-2\\frac{\\eta_k}{n}\\sum_{i=1}^n\\langle y_{i}^\\delta-y_{i}^\\dag, F_{i}(x_k^\\delta)-y_{i}^\\delta\\rangle\\\\\r\n   & - 2\\frac{\\eta_k}{n}\\sum_{i=1}^n\\langle y_{i}^\\dag-F_{i}(x_k^\\delta)-F_{i}'(x_k^\\delta)(x^*-x_k^\\delta),F_{i}(x_k^\\delta)-y_{i}^\\delta\\rangle \\\\\r\n  = & -\\eta_k\\|F(x_k^\\delta)-y^\\delta\\|^2 - 2\\eta_k\\langle y^\\delta-y^\\dag, F(x_k^\\delta)-y^\\delta\\rangle- 2\\eta_k\\langle y^\\dag-F(x_k^\\delta)-F'(x_k^\\delta)(x^*-x_k^\\delta),F(x_k^\\delta)-y^\\delta\\rangle \\\\\r\n  \\leq & -\\eta_k\\|F(x_k^\\delta)-y^\\delta\\|^2 + 2\\eta_k\\delta\\|F(x_k^\\delta)-y^\\delta\\| + 2\\eta_k\\eta\\|F(x_k^\\delta)-y^\\dag\\|\\|F(x_k^\\delta)-y^\\delta\\|\\\\\r\n  \\leq & \\eta_k\\|F(x_k^\\delta)-y^\\delta\\|\\big((2\\eta-1)\\|F(x_k^\\delta)-y^\\delta\\| + 2(1+\\eta)\\delta\\big),\r\n\\end{align*}\r\nwhere the second inequality follows from Assumption \\ref{ass:sol}(i) and the triangle inequality.\r\nLast, by taking full conditional of the inequality yields % and applying Young's inequality yield\r\n\\begin{align*}\r\n   &\\E[\\|x^*-x_{k+1}^\\delta\\|^2] - \\E[\\|x^*-x_{k}^\\delta\\|^2]\\\\\r\n   \\leq &  -(1-2\\eta)\\eta_k\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2]\r\n   +  2\\eta_k(1+\\eta)\\delta\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2]^\\frac{1}{2}.\r\n\\end{align*}\r\nThis completes the proof of the proposition.\r\n\\end{proof}\n\nThe next result is an immediate consequence of Proposition \\ref{prop:mono-exact}.\r\n\n\n\\begin{corollary}\\label{cor:mono}\r\nLet Assumptions \\ref{ass:sol}(i)-(ii) and \\ref{ass:stepsize}(i) be fulfilled. Then for the exact data $y^\\dag$,\r\nany solution $x^*$ to problem \\eqref{eqn:nonlin} satisfies\r\n\\begin{align*}\r\n   \\E[\\|x^*-x_{k+1}\\|^2] - \\E[\\|x^*-x_{k}\\|^2] &\\leq -(1-2\\eta)\\eta_k\\E[\\|F(x_k)-y^\\dag\\|^2],\\\\\r\n   \\sum_{k=1}^\\infty \\eta_k\\E[\\|F(x_k)-y^\\dag\\|^2] &\\leq \\frac{1}{1-2\\eta}\\|x^*-x_1\\|^2.\r\n\\end{align*}\r\n%and\r\n%\\begin{equation*}\r\n%  \\sum_{k=1}^\\infty \\eta_k\\E[\\|F(x_k)-y^\\dag\\|^2] \\leq \\frac{1}{1-2\\eta}\\|x^*-x_1\\|^2.\r\n%\\end{equation*}\r\n\\end{corollary}\n\n\\begin{lemma}\\label{lem:Cauchy}\r\nLet Assumptions \\ref{ass:sol}(i)-(ii) and \\ref{ass:stepsize}(i) be fulfilled. Then for the exact data $y^\\dag$,\r\nthe sequence $\\{x_k\\}_{k\\geq 1}$ generated by Algorithm \\ref{alg:sgd} is a Cauchy sequence.\r\n\\end{lemma}\n\n\\begin{proof}\r\nThe argument below follows closely \\cite[Theorem 2.3]{HankeNeubauerScherzer:1995}, which can be\r\ntraced back to \\cite{McCormickRodrigue:1975}. Let $x^*$ be any solution to problem \\eqref{eqn:nonlin}, and let $e_k:=x_k-x^*$. By\r\nCorollary \\ref{prop:mono-exact}, $\\E[\\|e_k\\|^2]$ is monotonically decreasing to some $\\epsilon\\geq 0$.\r\nNext we show that the sequence $\\{x_k\\}_{k\\ge 1}$ is actually a Cauchy sequence. First we note that\r\n$\\E[\\langle\\cdot,\\cdot\\rangle]$ defines an inner product. For any $j\\geq k$, choose an index $\\ell$\r\nwith $j\\geq \\ell\\geq k$ such that\r\n\\begin{equation}\\label{eqn:minimal-res}\r\n  \\E[\\|y^\\dag - F(x_\\ell)\\|^2]\\leq \\E[\\|y^\\dag - F(x_i)\\|^2], \\quad \\forall k\\leq i \\leq j.\r\n\\end{equation}\r\nIn light of the triangle inequality\r\n\\begin{equation*}\r\n  \\E[\\|e_j-e_k\\|^2]^\\frac{1}{2} \\leq \\E[\\|e_j-e_\\ell\\|^2]^\\frac{1}{2} + \\E[\\|e_\\ell-e_k\\|^2]^\\frac{1}{2},\r\n\\end{equation*}\r\nand the trivial identities\r\n\\begin{equation}\\label{eqn:err-exact}\r\n\\begin{aligned}\r\n  \\E[\\|e_j-e_\\ell\\|^2] & = 2\\E[\\langle e_\\ell-e_j,e_\\ell\\rangle] + \\E[\\|e_j\\|^2] - \\E[\\|e_\\ell\\|^2],\\\\\r\n  \\E[\\|e_\\ell-e_k\\|^2] & = 2\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle] + \\E[\\|e_k\\|^2] - \\E[\\|e_\\ell\\|^2],\r\n\\end{aligned}\r\n\\end{equation}\r\nit suffices to prove that both $\\E[\\|e_j-e_\\ell\\|^2]$ and $\\E[\\|e_\\ell-e_k\\|^2]$ on the left hand side tend to zero as $k\\to \\infty$.\r\nFor $k\\to \\infty$, the last two terms on each of the right hand side of \\eqref{eqn:err-exact} tends to $\\epsilon-\\epsilon=0$, by\r\nthe monotone convergence of $\\E[\\|e_k\\|^2]$ to $\\epsilon$.\r\nNext we show that the term $\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle]$ also tends to zero as $k\\to\\infty$.\r\nActually, by the definition of the SGD iterate $x_k$ in \\eqref{eqn:sgd}, we have\r\n\\begin{equation*}\r\n  e_\\ell - e_k = \\sum_{i=k}^{\\ell-1}(e_{i+1}-e_i) = {\\sum_{i=k}^{\\ell-1}\\eta_iF_{i_i}'(x_i)^*(y_{i_i}^\\dag-F_{i_i}(x_i))}.\r\n\\end{equation*}\r\nBy the triangle inequality, we can bound $\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle]$ by\r\n\\begin{align*}\r\n  |\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle]| & = |\\E[\\sum_{i=k}^{\\ell-1}\\langle \\eta_iF_{i_i}'(x_i)^*(y_{i_i}^\\dag-F_{i_i}(x_i)),e_\\ell\\rangle]|\\\\\r\n  & \\leq \\sum_{i=k}^{\\ell-1}\\eta_i|\\E[\\langle F_{i_i}'(x_i)^*(y_{i_i}^\\dag-F_{i_i}(x_i)),e_\\ell\\rangle]|\\\\\r\n  & = \\sum_{i=k}^{\\ell-1}\\eta_i|\\E[\\langle y_{i_i}^\\dag-F_{i_i}(x_i),F_{i_i}'(x_i)(x^*-x_i+x_i-x_\\ell)\\rangle]|.\r\n\\end{align*}\r\nThen by the Cauchy-Schwarz inequality and triangle inequality, we obtain\r\n\\begin{align*}\r\n |\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle]|\r\n  & \\leq \\sum_{i=k}^{\\ell-1}\\eta_i|\\E[\\langle y^\\dag-F(x_i),F'(x_i)(x^*-x_i+x_i-x_\\ell)\\rangle]|\\\\\r\n  & \\leq \\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F'(x_i)(x^*-x_i+x_i-x_\\ell)\\|^2]^\\frac{1}{2}\\\\\r\n  & \\leq \\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F'(x_i)(x^*-x_i)\\|^2]^\\frac{1}{2}\\\\\r\n   &\\quad +\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F'(x_i)(x_i-x_\\ell)\\|^2]^\\frac{1}{2}\r\n   :={\\rm I} + {\\rm II}.\r\n\\end{align*}\r\nBy Assumption \\ref{ass:sol}(ii) and Lemma \\ref{lem:linear}(i), we have\r\n\\begin{equation*}\r\n\\|F'(x)(x-\\tilde x)\\|\\leq (1+\\eta)\\|F(x)-F(\\tilde x)\\|.\r\n\\end{equation*}\r\nSubstituting this inequality into the term $\\rm I$ leads to\r\n\\begin{align*}\r\n  {\\rm I} & \\leq (1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F(x^*)-F(x_i)\\|^2]^\\frac{1}{2}\r\n           = (1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2].\r\n\\end{align*}\r\nLikewise, by the triangle inequality and the choice of the index $\\ell$ in \\eqref{eqn:minimal-res},\r\n\\begin{align*}\r\n  {\\rm II} & \\leq (1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F(x_\\ell)-F(x_i)\\|^2]^\\frac{1}{2}\\\\\r\n          & \\leq (1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2] + (1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F(x_\\ell)-y^\\dag\\|^2]^\\frac{1}{2}\\\\\r\n          & \\leq 2(1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2].\r\n\\end{align*}\r\nThe last two estimates together imply\r\n\\begin{align*}\r\n  |\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle]| & \\leq 3(1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2].\r\n\\end{align*}\r\nSimilarly, one can deduce\r\n\\begin{equation*}\r\n  |\\E[\\langle e_j-e_\\ell,e_\\ell\\rangle]| \\leq 3(1+\\eta)\\sum_{i=\\ell}^{j-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2].\r\n\\end{equation*}\r\nThese two estimates and Corollary \\ref{prop:mono-exact} imply that the right hand sides of \\eqref{eqn:err-exact}\r\ntend to zero as $k\\to\\infty$. Thus, the sequence $\\{e_k\\}_{k\\geq 1}$ and also $\\{x_k\\}_{k\\geq1}$ are Cauchy sequences.\r\n\\end{proof}\n\n\\begin{lemma}\\label{lemma:res}\r\nLet Assumptions \\ref{ass:sol}(i)-(ii) and \\ref{ass:stepsize}(i) be fulfilled. Then for the exact data $y^\\dag$, there holds\r\n\\begin{equation*}\r\n  \\lim_{k\\to\\infty} \\E[\\|F(x_k)-y^\\dag\\|^2] = 0.\r\n\\end{equation*}\r\n\\end{lemma}\n\n\\begin{proof}\r\nBy Lemma \\ref{lem:Cauchy}, $\\{x_k\\}_{k\\geq 1}$ is a Cauchy sequence. By Assumption \\ref{ass:stepsize}(i),\r\n$\\sup_x\\|F'(x)\\|\\leq \\eta_0^{-\\frac12}$. Further, for any $x,\\tilde x\\in X$, there holds\r\n\\begin{equation*}\r\n  \\|F(x)-F(\\tilde x)\\| \\leq (1-\\eta)^{-1}\\|F'(x)(x-\\tilde x)\\|\\leq (1-\\eta)^{-1}\\eta_0^{-\\frac12}\\|x-\\tilde x\\|.\r\n\\end{equation*}\r\nThus, $\\{F(x_k)-y^\\dag\\}_{k\\geq 1}$ is a Cauchy sequence, and $\\E[\\|F(x_k)-y^\\dag\\|^2]$\r\nconverges. Now we proceed by contradiction, and assume that $\\lim_{k\\to\\infty}\\E[\\|F(x_k)-y^\\dag\\|^2]>0$.\r\nThen there exist some $\\epsilon>0$ and $k^*\\in \\mathbb{N}$, such that $\\E[\\|F(x_k)-y^\\dag\\|^2]\\geq \\epsilon$\r\nfor all $k\\geq k^*$. Hence, by Assumption \\ref{ass:stepsize}(i),\r\n\\begin{equation*}\r\n  \\sum_{k=1}^\\infty \\eta_k\\E[\\|F(x_k)-y^\\dag\\|^2] \\geq \\sum_{k=k^*}^\\infty \\eta_k\\E[\\|F(x_k)-y^\\dag\\|^2] \\geq \\epsilon\\sum_{k=k^*}^\\infty\\eta_k =\\infty,\r\n\\end{equation*}\r\nwhich contradicts the inequality\r\n\\begin{equation*}\r\n\\sum_{k=1}^\\infty \\eta_k\\E[\\|F(x_k)-y^\\dag\\|^2] <\\infty,\r\n\\end{equation*}\r\nfrom Corollary \\ref{cor:mono}. This completes the proof of the lemma.\r\n\\end{proof}\n\n\\begin{theorem}[Convergence for exact data]\\label{thm:conv-exact}\r\nLet Assumptions \\ref{ass:sol}(i)-(ii) and \\ref{ass:stepsize}(i) be fulfilled. Then for the exact data $y^\\dag$, the sequence\r\n$\\{x_k\\}_{k\\geq1}$ by Algorithm \\ref{alg:sgd} converges to a solution $x^*$ of problem \\eqref{eqn:nonlin}:\r\n\\begin{equation*}\r\n  \\lim_{k\\to\\infty}\\E[\\|x_k-x^*\\|^2] = 0.\r\n\\end{equation*}\r\n Further, if\r\n$\\mathcal{N}(F'(x^\\dag))\\subset\\mathcal{N}(F'(x))$, then\r\n\\begin{equation*}\r\n  \\lim_{k\\to\\infty}\\E[\\|x_k -x^\\dag\\|^2]=0.\r\n\\end{equation*}\r\n\\end{theorem}\n\n\\begin{proof}\r\nSince $\\{x_k\\}_{k\\geq1}$ is a Cauchy sequence, there exists a limit of $\\{x_k\\}_{k\\geq 1}$, denoted by $x^*$. Further,\r\n$x^*$ is a solution, since the residual $\\E[\\|y^\\dag-F(x_k)\\|^2]$ converges to zero as $k\\to\\infty$, in view of Lemma \\ref{lemma:res}.\r\n\r\nNote that problem \\eqref{eqn:nonlin} has a unique solution of minimal distance to the initial guess $x_1$ that satisfies\r\n\\begin{equation*}\r\n  x^\\dag - x_1\\in\\mathcal{N}(F'(x^\\dag))^\\perp;\r\n\\end{equation*}\r\nsee Lemma \\ref{lem:linear}. If {$\\mathcal{N}(F'(x^\\dag))\\subset \\mathcal{N}(F'(x_k))$}\r\n for all $k=1,2,\\ldots$, then clearly,\r\n\\begin{equation*}\r\n  x_k-x_1\\in\\mathcal{N}(F'(x^\\dag))^\\perp,\\quad k =1,2,\\ldots.\r\n\\end{equation*}\r\nConsequently,\r\n\\begin{equation*}\r\n  x^\\dag - x^* = x^\\dag - x_1 + x_1-x^*\\in\\mathcal{N}(F'(x^\\dag))^\\perp.\r\n\\end{equation*}\r\nThis and the inequalities from Lemma \\ref{lem:linear}(i) imply $x^*=x^\\dag$, completing the proof.\r\n\\end{proof}\n\n\\begin{lemma}\\label{lem:conti-noise}\r\nLet Assumption \\ref{ass:sol}(i) be fulfilled. For any fixed $k\\in \\mathbb{N}$ and any path\r\n$(i_1,\\ldots,i_{k-1})\\in\\mathcal{F}_k$, let $x_k$ and $x_k^\\delta$ be the SGD iterates along the path for exact\r\ndata $y^\\dag$ and noisy data $y^\\delta$, respectively. Then\r\n\\begin{equation*}\r\n  \\lim_{\\delta\\to 0^+}\\|x_k^\\delta-x_k\\|=0.\r\n\\end{equation*}\r\n\\end{lemma}\n\n\\begin{proof}\r\nWe prove the assertion by mathematical induction. The assertion holds\r\ntrivially for $k=1$, since $x_1^\\delta = x_1$. Now suppose that it\r\nholds for all indices up to $k$ and any path $(i_1,\\ldots,i_{k-1})\\in\\mathcal{F}_k$.\r\nNext, by the definitions of the SGD iterates $x_k$ and $x_k^\\delta$, cf. \\eqref{eqn:sgd}:\r\n\\begin{align*}\r\n   x_{k+1}        &= x_k - \\eta_k F_{i_k}'(x_k)^*(F_{i_k}(x_k)-y_{i_k}),\\\\\r\n   x_{k+1}^\\delta &= x_k^\\delta - \\eta_k F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta).\r\n\\end{align*}\r\nTherefore, for any fixed path $(i_1,\\ldots,i_k)$, we have\r\n\\begin{align*}\r\n   x_{k+1}^\\delta -x_{k+1} &= (x_k^\\delta-x_k) - \\eta_k \\left(F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)-F_{i_k}'(x_k)^*(F_{i_k}(x_k)-y_{i_k})\\right)\\\\\r\n     & = (x_k^\\delta -x_k)-\\eta_k \\big((F_{i_k}'(x_k^\\delta)^* - F_{i_k}'(x_k)^*)(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta) \\\\\r\n     & \\quad +F_{i_k}'(x_k)^*((F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)-(F_{i_k}(x_k)-y_{i_k}))\\big).\r\n\\end{align*}\r\nThus, by triangle inequality,\r\n\\begin{align*}\r\n  \\|x_{k+1}^\\delta-x_{k+1}\\| &\\leq \\|x_k^\\delta -x_k\\|+\\eta_k \\|F_{i_k}'(x_k^\\delta)^* - F_{i_k}'(x_k)^*\\|\\|F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\| \\\\\r\n     & \\quad +\\|F_{i_k}'(x_k)^*\\|\\|((F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)-(F_{i_k}(x_k)-y_{i_k}))\\|.\r\n\\end{align*}\r\nThen the desired assertion follows from the continuity of the operators $F_i$ and $F_i'$ in Assumption \\ref{ass:sol}(i)\r\nand the induction hypothesis.\r\n\\end{proof}\n\n\\begin{proof}[Proof of Theorem \\ref{thm:conv-noisy}]\r\nLet $\\{\\delta_n\\}_{n\\geq 1}\\subset\\mathbb{R}$ be a sequence converging to zero, and let $y_n:=y^{\\delta_n}$\r\nbe a corresponding sequence of noisy data. For each pair $(\\delta_n,y_n)$, we denote by\r\n$k_n=k(\\delta_n)$ the stopping index. Without loss of generality, we may assume that\r\n$k_n$ increases strictly monotonically with $n$.\r\n\r\nBy Proposition \\ref{prop:mono-exact} and Young's inequality $2ab\\leq \\epsilon a^2 + \\epsilon^{-1} b^2$,\r\nwith the choice $a=\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2]^\\frac{1}{2}$, $b=(1+\\eta)\\delta$ and $\\epsilon=1-2\\eta>0$:\r\n\\begin{align*}\r\n    & \\E[\\|x^*-x_{k+1}^\\delta\\|^2] - \\E[\\|x^*-x_{k}^\\delta\\|^2] \\\\\r\n   \\leq & -(1-2\\eta)\\eta_k\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2] +  2\\eta_k(1+\\eta)\\delta\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2]^\\frac{1}{2}\r\n   \\leq \\frac{(1+\\eta)^2}{1-2\\eta}\\eta_k\\delta^2.\r\n\\end{align*}\r\nThen for any $m<n$, summing the above inequality with $\\delta=\\delta_n$ from $k_m$ to $k_n-1$\r\n(since $k_n$ is strictly increasing with $n$ by assumption) and applying the triangle inequality lead to\r\n\\begin{align*}\r\n  \\E[\\|x_{k_n}^{\\delta_n} -x^*\\|^2] &\\leq \\E[\\|x_{k_m}^{\\delta_n}-x^*\\|^2] + \\frac{(1+\\eta)^2}{1-2\\eta}\\delta_n^2\\sum_{j=k_m}^{k_n-1}\\eta_j\\\\\r\n     & \\leq 2\\E[\\|x_{k_m}^{\\delta_n}-x_{k_m}\\|^2] + 2\\E[\\|x_{k_m}-x^*\\|^2] + \\frac{(1+\\eta)^2}{1-2\\eta}\\delta_n^2\\sum_{j=1}^{k_n-1}\\eta_j.\r\n\\end{align*}\r\nBy Theorem \\ref{thm:conv-exact} we can fix $m$ so large that the term $\\E[\\|x_{k_m}-x^*\\|^2]$ is sufficiently small.\r\nSince the index $k_m$ is fixed, we may apply Lemma \\ref{lem:conti-noise} to conclude that the term\r\n$\\E[\\|x_{k_m}^{\\delta_n}-x_{k_m}\\|^2]$ must go to zero as $n\\to\\infty$. The last term also tends to zero under the\r\ncondition on the index $k_n$, i.e., $\\lim_{n\\to\\infty}\\delta_n^2\\sum_{i=1}^{k_n}\\eta_i=0$. This completes\r\nthe proof of the first assertion. The case for $\\mathcal{N}(F'(x^\\dag))\\subset\\mathcal{N}(F'(x))$ follows similarly\r\nas Theorem \\ref{thm:conv-exact}.\r\n\\end{proof}","units":[{"start":25775,"end":26317,"start_line":368,"end_line":378,"raw_tex":"\\begin{lemma}\\label{lem:linear}\r\nLet Assumptions \\ref{ass:sol}(i) and (ii) be fulfilled. Then the following statements hold.\r\n\\begin{itemize}\r\n  \\item[$\\rm(i)$] The following inequalities hold:\r\n  \\begin{equation*}\r\n  \\frac{1}{1+\\eta}\\|F'(x)(x-\\tilde x)\\|\\leq \\|F(x)-F(\\tilde x)\\|\\leq \\frac{1}{1-\\eta}\\|F'(x)(x-\\tilde x)\\|.\r\n  \\end{equation*}\r\n  \\item[$\\rm(ii)$]If $x^*$ is a solution of \\eqref{eqn:nonlin}, then any other solution $\\tilde x^*$ satisfies\r\n$x^*-\\tilde x^* \\in \\mathcal{N}(F'(x^*))$, and vice versa.\r\n\\end{itemize}\r\n\\end{lemma}","sha256_utf8":"fd334fc6b6335dfe674c98ef60dc594cdf9444c91d4bfd20b71269f7d30bad2e"},{"start":26537,"end":26962,"start_line":383,"end_line":389,"raw_tex":"\\begin{prop}\\label{prop:mono-exact}\r\nLet Assumptions \\ref{ass:sol}(i)-(ii) and \\ref{ass:stepsize}(i) be fulfilled. Then for any solution $x^*$ to problem \\eqref{eqn:nonlin}, there holds\r\n\\begin{equation*}\r\n  \\E[\\|x^*-x_{k+1}^\\delta\\|^2] - \\E[\\|x^*-x_{k}^\\delta\\|^2]\r\n   \\leq  -(1-2\\eta)\\eta_k\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2] + { 2\\eta_k(1+\\eta)\\delta\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2]^\\frac{1}{2}}.\r\n\\end{equation*}\r\n\\end{prop}","sha256_utf8":"ac1c1c2c846ad3bcb4433e948a6306a490856e37a49c4c7a0699c278b8510fdc"},{"start":26964,"end":30885,"start_line":390,"end_line":436,"raw_tex":"\\begin{proof}\r\nCompleting the square gives\r\n\\begin{align*}\r\n    \\|x^*-x_{k+1}^\\delta\\|^2 - \\|x^*-x_{k}^\\delta\\|^2 = & 2\\langle x_k^\\delta - x^*,x_{k+1}^\\delta-x_k^\\delta\\rangle + \\|x_{k+1}^\\delta-x_k^\\delta\\|^2.\r\n\\end{align*}\r\nBy the definition of the SGD iterate $x_{k}^\\delta$ in \\eqref{eqn:sgd}, there holds\r\n\\begin{align*}\r\n   & \\|x^*-x_{k+1}^\\delta\\|^2 - \\|x^*-x_{k}^\\delta\\|^2\\\\\r\n  = & -2\\eta_k\\langle x_k^\\delta-x^*,  F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)\\rangle + \\eta_k^2\\| F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)\\|^2 \\\\\r\n  = & -2\\eta_k\\langle F_{i_k}'(x_k^\\delta)(x_k^\\delta-x^*), F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle + \\eta_k^2\\|F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)\\|^2.\r\n\\end{align*}\r\nNext we split the factor $-F'_{i_k}(x_k^\\delta)(x_k^\\delta-x^*)$ into two terms\r\n\\begin{equation*}\r\n  F_{i_k}'(x_k^\\delta)(x_k^\\delta-x^*) =  (F_{i_k}(x_k^\\delta)- F_{i_k}(x^*)) + (F_{i_k}(x^*)-F_{i_k}(x_k^\\delta)-F_{i_k}'(x_k^\\delta)(x^* - x_k^\\delta)).\r\n\\end{equation*}\r\nCombining the last two identities yields\r\n\\begin{align*}\r\n    &\\|x^*-x_{k+1}^\\delta\\|^2 - \\|x^*-x_{k}^\\delta\\|^2\\\\\r\n  = & -2\\eta_k\\langle F_{i_k}(x_k^\\delta)-F_{i_k}(x^*), F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle +  {\\eta_k^2\\|F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)\\|^2}\\\\\r\n   & - 2\\eta_k\\langle y_{i_k}^\\dag-F_{i_k}(x_k^\\delta)-F_{i_k}'(x_k^\\delta)(x^*-x_k^\\delta),F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle \\\\\r\n  = & -2\\eta_k\\langle F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta, F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle +  {\\eta_k^2\\|F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)\\|^2}\\\\\r\n   & -2\\eta_k\\langle y_{i_k}^\\delta-y_{i_k}^\\dag , F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle\\\\\r\n   & - 2\\eta_k\\langle y_{i_k}^\\dag-F_{i_k}(x_k^\\delta)-F_{i_k}'(x_k^\\delta)(x^*-x_k^\\delta),F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle \\\\\r\n \\leq  & -\\eta_k\\langle F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta, F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle\r\n    -2\\eta_k\\langle y_{i_k}^\\delta-y_{i_k}^\\dag, F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle\\\\\r\n   & - 2\\eta_k\\langle y_{i_k}^\\dag-F_{i_k}(x_k^\\delta)-F_{i_k}'(x_k^\\delta)(x^*-x_k^\\delta),F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle,\r\n\\end{align*}\r\nwhere the inequality follows from the condition $\\eta_k\\|F_{i_k}'(x)\\|^2<1$ in Assumption \\ref{ass:stepsize}(i).\r\nThus, by the measurability of the iterate $x_k$ with respect to the filtration\r\n$\\mathcal{F}_{k}$ and the Cauchy-Schwarz inequality, we have\r\n\\begin{align*}\r\n   &{\\E[\\|x^*-x_{k+1}^\\delta\\|^2 - \\|x^*-x_{k}^\\delta\\|^2|\\mathcal{F}_{k}]} \\\\\r\n    \\leq &-\\frac{\\eta_k}{n}\\sum_{i=1}^n\\langle F_{i}(x_k^\\delta)-y_{i}^\\delta,F_{i}(x_k^\\delta)-y_{i}^\\delta\\rangle-2\\frac{\\eta_k}{n}\\sum_{i=1}^n\\langle y_{i}^\\delta-y_{i}^\\dag, F_{i}(x_k^\\delta)-y_{i}^\\delta\\rangle\\\\\r\n   & - 2\\frac{\\eta_k}{n}\\sum_{i=1}^n\\langle y_{i}^\\dag-F_{i}(x_k^\\delta)-F_{i}'(x_k^\\delta)(x^*-x_k^\\delta),F_{i}(x_k^\\delta)-y_{i}^\\delta\\rangle \\\\\r\n  = & -\\eta_k\\|F(x_k^\\delta)-y^\\delta\\|^2 - 2\\eta_k\\langle y^\\delta-y^\\dag, F(x_k^\\delta)-y^\\delta\\rangle- 2\\eta_k\\langle y^\\dag-F(x_k^\\delta)-F'(x_k^\\delta)(x^*-x_k^\\delta),F(x_k^\\delta)-y^\\delta\\rangle \\\\\r\n  \\leq & -\\eta_k\\|F(x_k^\\delta)-y^\\delta\\|^2 + 2\\eta_k\\delta\\|F(x_k^\\delta)-y^\\delta\\| + 2\\eta_k\\eta\\|F(x_k^\\delta)-y^\\dag\\|\\|F(x_k^\\delta)-y^\\delta\\|\\\\\r\n  \\leq & \\eta_k\\|F(x_k^\\delta)-y^\\delta\\|\\big((2\\eta-1)\\|F(x_k^\\delta)-y^\\delta\\| + 2(1+\\eta)\\delta\\big),\r\n\\end{align*}\r\nwhere the second inequality follows from Assumption \\ref{ass:sol}(i) and the triangle inequality.\r\nLast, by taking full conditional of the inequality yields % and applying Young's inequality yield\r\n\\begin{align*}\r\n   &\\E[\\|x^*-x_{k+1}^\\delta\\|^2] - \\E[\\|x^*-x_{k}^\\delta\\|^2]\\\\\r\n   \\leq &  -(1-2\\eta)\\eta_k\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2]\r\n   +  2\\eta_k(1+\\eta)\\delta\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2]^\\frac{1}{2}.\r\n\\end{align*}\r\nThis completes the proof of the proposition.\r\n\\end{proof}","sha256_utf8":"c002c99c525c2a6dafacb5d5b673808dceb8cf02f590e10e6f7db51ba07525fa"},{"start":31024,"end":31107,"start_line":442,"end_line":442,"raw_tex":"The next result is an immediate consequence of Proposition \\ref{prop:mono-exact}.\r\n","sha256_utf8":"6c594955007c0cbb73df4e31c669ea3719eb6688412491aa0a99f63a9b5a1d4b"},{"start":31109,"end":31682,"start_line":444,"end_line":455,"raw_tex":"\\begin{corollary}\\label{cor:mono}\r\nLet Assumptions \\ref{ass:sol}(i)-(ii) and \\ref{ass:stepsize}(i) be fulfilled. Then for the exact data $y^\\dag$,\r\nany solution $x^*$ to problem \\eqref{eqn:nonlin} satisfies\r\n\\begin{align*}\r\n   \\E[\\|x^*-x_{k+1}\\|^2] - \\E[\\|x^*-x_{k}\\|^2] &\\leq -(1-2\\eta)\\eta_k\\E[\\|F(x_k)-y^\\dag\\|^2],\\\\\r\n   \\sum_{k=1}^\\infty \\eta_k\\E[\\|F(x_k)-y^\\dag\\|^2] &\\leq \\frac{1}{1-2\\eta}\\|x^*-x_1\\|^2.\r\n\\end{align*}\r\n%and\r\n%\\begin{equation*}\r\n%  \\sum_{k=1}^\\infty \\eta_k\\E[\\|F(x_k)-y^\\dag\\|^2] \\leq \\frac{1}{1-2\\eta}\\|x^*-x_1\\|^2.\r\n%\\end{equation*}\r\n\\end{corollary}","sha256_utf8":"eafa8b356839159b59ea6961b620a05528b389bdbd0877e62bd52c3b89c73fd6"},{"start":32217,"end":32467,"start_line":465,"end_line":468,"raw_tex":"\\begin{lemma}\\label{lem:Cauchy}\r\nLet Assumptions \\ref{ass:sol}(i)-(ii) and \\ref{ass:stepsize}(i) be fulfilled. Then for the exact data $y^\\dag$,\r\nthe sequence $\\{x_k\\}_{k\\geq 1}$ generated by Algorithm \\ref{alg:sgd} is a Cauchy sequence.\r\n\\end{lemma}","sha256_utf8":"22daf487b6371d0991e564aa24adc49b2f5327603cd071c353c7b3c6d6dd72ff"},{"start":32469,"end":37035,"start_line":469,"end_line":538,"raw_tex":"\\begin{proof}\r\nThe argument below follows closely \\cite[Theorem 2.3]{HankeNeubauerScherzer:1995}, which can be\r\ntraced back to \\cite{McCormickRodrigue:1975}. Let $x^*$ be any solution to problem \\eqref{eqn:nonlin}, and let $e_k:=x_k-x^*$. By\r\nCorollary \\ref{prop:mono-exact}, $\\E[\\|e_k\\|^2]$ is monotonically decreasing to some $\\epsilon\\geq 0$.\r\nNext we show that the sequence $\\{x_k\\}_{k\\ge 1}$ is actually a Cauchy sequence. First we note that\r\n$\\E[\\langle\\cdot,\\cdot\\rangle]$ defines an inner product. For any $j\\geq k$, choose an index $\\ell$\r\nwith $j\\geq \\ell\\geq k$ such that\r\n\\begin{equation}\\label{eqn:minimal-res}\r\n  \\E[\\|y^\\dag - F(x_\\ell)\\|^2]\\leq \\E[\\|y^\\dag - F(x_i)\\|^2], \\quad \\forall k\\leq i \\leq j.\r\n\\end{equation}\r\nIn light of the triangle inequality\r\n\\begin{equation*}\r\n  \\E[\\|e_j-e_k\\|^2]^\\frac{1}{2} \\leq \\E[\\|e_j-e_\\ell\\|^2]^\\frac{1}{2} + \\E[\\|e_\\ell-e_k\\|^2]^\\frac{1}{2},\r\n\\end{equation*}\r\nand the trivial identities\r\n\\begin{equation}\\label{eqn:err-exact}\r\n\\begin{aligned}\r\n  \\E[\\|e_j-e_\\ell\\|^2] & = 2\\E[\\langle e_\\ell-e_j,e_\\ell\\rangle] + \\E[\\|e_j\\|^2] - \\E[\\|e_\\ell\\|^2],\\\\\r\n  \\E[\\|e_\\ell-e_k\\|^2] & = 2\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle] + \\E[\\|e_k\\|^2] - \\E[\\|e_\\ell\\|^2],\r\n\\end{aligned}\r\n\\end{equation}\r\nit suffices to prove that both $\\E[\\|e_j-e_\\ell\\|^2]$ and $\\E[\\|e_\\ell-e_k\\|^2]$ on the left hand side tend to zero as $k\\to \\infty$.\r\nFor $k\\to \\infty$, the last two terms on each of the right hand side of \\eqref{eqn:err-exact} tends to $\\epsilon-\\epsilon=0$, by\r\nthe monotone convergence of $\\E[\\|e_k\\|^2]$ to $\\epsilon$.\r\nNext we show that the term $\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle]$ also tends to zero as $k\\to\\infty$.\r\nActually, by the definition of the SGD iterate $x_k$ in \\eqref{eqn:sgd}, we have\r\n\\begin{equation*}\r\n  e_\\ell - e_k = \\sum_{i=k}^{\\ell-1}(e_{i+1}-e_i) = {\\sum_{i=k}^{\\ell-1}\\eta_iF_{i_i}'(x_i)^*(y_{i_i}^\\dag-F_{i_i}(x_i))}.\r\n\\end{equation*}\r\nBy the triangle inequality, we can bound $\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle]$ by\r\n\\begin{align*}\r\n  |\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle]| & = |\\E[\\sum_{i=k}^{\\ell-1}\\langle \\eta_iF_{i_i}'(x_i)^*(y_{i_i}^\\dag-F_{i_i}(x_i)),e_\\ell\\rangle]|\\\\\r\n  & \\leq \\sum_{i=k}^{\\ell-1}\\eta_i|\\E[\\langle F_{i_i}'(x_i)^*(y_{i_i}^\\dag-F_{i_i}(x_i)),e_\\ell\\rangle]|\\\\\r\n  & = \\sum_{i=k}^{\\ell-1}\\eta_i|\\E[\\langle y_{i_i}^\\dag-F_{i_i}(x_i),F_{i_i}'(x_i)(x^*-x_i+x_i-x_\\ell)\\rangle]|.\r\n\\end{align*}\r\nThen by the Cauchy-Schwarz inequality and triangle inequality, we obtain\r\n\\begin{align*}\r\n |\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle]|\r\n  & \\leq \\sum_{i=k}^{\\ell-1}\\eta_i|\\E[\\langle y^\\dag-F(x_i),F'(x_i)(x^*-x_i+x_i-x_\\ell)\\rangle]|\\\\\r\n  & \\leq \\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F'(x_i)(x^*-x_i+x_i-x_\\ell)\\|^2]^\\frac{1}{2}\\\\\r\n  & \\leq \\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F'(x_i)(x^*-x_i)\\|^2]^\\frac{1}{2}\\\\\r\n   &\\quad +\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F'(x_i)(x_i-x_\\ell)\\|^2]^\\frac{1}{2}\r\n   :={\\rm I} + {\\rm II}.\r\n\\end{align*}\r\nBy Assumption \\ref{ass:sol}(ii) and Lemma \\ref{lem:linear}(i), we have\r\n\\begin{equation*}\r\n\\|F'(x)(x-\\tilde x)\\|\\leq (1+\\eta)\\|F(x)-F(\\tilde x)\\|.\r\n\\end{equation*}\r\nSubstituting this inequality into the term $\\rm I$ leads to\r\n\\begin{align*}\r\n  {\\rm I} & \\leq (1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F(x^*)-F(x_i)\\|^2]^\\frac{1}{2}\r\n           = (1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2].\r\n\\end{align*}\r\nLikewise, by the triangle inequality and the choice of the index $\\ell$ in \\eqref{eqn:minimal-res},\r\n\\begin{align*}\r\n  {\\rm II} & \\leq (1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F(x_\\ell)-F(x_i)\\|^2]^\\frac{1}{2}\\\\\r\n          & \\leq (1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2] + (1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F(x_\\ell)-y^\\dag\\|^2]^\\frac{1}{2}\\\\\r\n          & \\leq 2(1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2].\r\n\\end{align*}\r\nThe last two estimates together imply\r\n\\begin{align*}\r\n  |\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle]| & \\leq 3(1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2].\r\n\\end{align*}\r\nSimilarly, one can deduce\r\n\\begin{equation*}\r\n  |\\E[\\langle e_j-e_\\ell,e_\\ell\\rangle]| \\leq 3(1+\\eta)\\sum_{i=\\ell}^{j-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2].\r\n\\end{equation*}\r\nThese two estimates and Corollary \\ref{prop:mono-exact} imply that the right hand sides of \\eqref{eqn:err-exact}\r\ntend to zero as $k\\to\\infty$. Thus, the sequence $\\{e_k\\}_{k\\geq 1}$ and also $\\{x_k\\}_{k\\geq1}$ are Cauchy sequences.\r\n\\end{proof}","sha256_utf8":"cccc9c2cf9424f7e92b36c82760bcdd7656e004128b41e56b04fb83a62897188"},{"start":37039,"end":37293,"start_line":540,"end_line":545,"raw_tex":"\\begin{lemma}\\label{lemma:res}\r\nLet Assumptions \\ref{ass:sol}(i)-(ii) and \\ref{ass:stepsize}(i) be fulfilled. Then for the exact data $y^\\dag$, there holds\r\n\\begin{equation*}\r\n  \\lim_{k\\to\\infty} \\E[\\|F(x_k)-y^\\dag\\|^2] = 0.\r\n\\end{equation*}\r\n\\end{lemma}","sha256_utf8":"4ab800fe4caf1ea2866a57eb732339baf84ae06723ddb2ef93ed3af2c6918815"},{"start":37295,"end":38429,"start_line":546,"end_line":564,"raw_tex":"\\begin{proof}\r\nBy Lemma \\ref{lem:Cauchy}, $\\{x_k\\}_{k\\geq 1}$ is a Cauchy sequence. By Assumption \\ref{ass:stepsize}(i),\r\n$\\sup_x\\|F'(x)\\|\\leq \\eta_0^{-\\frac12}$. Further, for any $x,\\tilde x\\in X$, there holds\r\n\\begin{equation*}\r\n  \\|F(x)-F(\\tilde x)\\| \\leq (1-\\eta)^{-1}\\|F'(x)(x-\\tilde x)\\|\\leq (1-\\eta)^{-1}\\eta_0^{-\\frac12}\\|x-\\tilde x\\|.\r\n\\end{equation*}\r\nThus, $\\{F(x_k)-y^\\dag\\}_{k\\geq 1}$ is a Cauchy sequence, and $\\E[\\|F(x_k)-y^\\dag\\|^2]$\r\nconverges. Now we proceed by contradiction, and assume that $\\lim_{k\\to\\infty}\\E[\\|F(x_k)-y^\\dag\\|^2]>0$.\r\nThen there exist some $\\epsilon>0$ and $k^*\\in \\mathbb{N}$, such that $\\E[\\|F(x_k)-y^\\dag\\|^2]\\geq \\epsilon$\r\nfor all $k\\geq k^*$. Hence, by Assumption \\ref{ass:stepsize}(i),\r\n\\begin{equation*}\r\n  \\sum_{k=1}^\\infty \\eta_k\\E[\\|F(x_k)-y^\\dag\\|^2] \\geq \\sum_{k=k^*}^\\infty \\eta_k\\E[\\|F(x_k)-y^\\dag\\|^2] \\geq \\epsilon\\sum_{k=k^*}^\\infty\\eta_k =\\infty,\r\n\\end{equation*}\r\nwhich contradicts the inequality\r\n\\begin{equation*}\r\n\\sum_{k=1}^\\infty \\eta_k\\E[\\|F(x_k)-y^\\dag\\|^2] <\\infty,\r\n\\end{equation*}\r\nfrom Corollary \\ref{cor:mono}. This completes the proof of the lemma.\r\n\\end{proof}","sha256_utf8":"9e98e3cd3fd241548e16b5c8dd28662250107aeb813477522d6898564f6a9722"},{"start":38628,"end":39174,"start_line":569,"end_line":580,"raw_tex":"\\begin{theorem}[Convergence for exact data]\\label{thm:conv-exact}\r\nLet Assumptions \\ref{ass:sol}(i)-(ii) and \\ref{ass:stepsize}(i) be fulfilled. Then for the exact data $y^\\dag$, the sequence\r\n$\\{x_k\\}_{k\\geq1}$ by Algorithm \\ref{alg:sgd} converges to a solution $x^*$ of problem \\eqref{eqn:nonlin}:\r\n\\begin{equation*}\r\n  \\lim_{k\\to\\infty}\\E[\\|x_k-x^*\\|^2] = 0.\r\n\\end{equation*}\r\n Further, if\r\n$\\mathcal{N}(F'(x^\\dag))\\subset\\mathcal{N}(F'(x))$, then\r\n\\begin{equation*}\r\n  \\lim_{k\\to\\infty}\\E[\\|x_k -x^\\dag\\|^2]=0.\r\n\\end{equation*}\r\n\\end{theorem}","sha256_utf8":"2c5d2c98226cdf3ed3c53baa333e386e2d70c7dbd9a9703daccef14105d1582d"},{"start":39176,"end":40120,"start_line":581,"end_line":599,"raw_tex":"\\begin{proof}\r\nSince $\\{x_k\\}_{k\\geq1}$ is a Cauchy sequence, there exists a limit of $\\{x_k\\}_{k\\geq 1}$, denoted by $x^*$. Further,\r\n$x^*$ is a solution, since the residual $\\E[\\|y^\\dag-F(x_k)\\|^2]$ converges to zero as $k\\to\\infty$, in view of Lemma \\ref{lemma:res}.\r\n\r\nNote that problem \\eqref{eqn:nonlin} has a unique solution of minimal distance to the initial guess $x_1$ that satisfies\r\n\\begin{equation*}\r\n  x^\\dag - x_1\\in\\mathcal{N}(F'(x^\\dag))^\\perp;\r\n\\end{equation*}\r\nsee Lemma \\ref{lem:linear}. If {$\\mathcal{N}(F'(x^\\dag))\\subset \\mathcal{N}(F'(x_k))$}\r\n for all $k=1,2,\\ldots$, then clearly,\r\n\\begin{equation*}\r\n  x_k-x_1\\in\\mathcal{N}(F'(x^\\dag))^\\perp,\\quad k =1,2,\\ldots.\r\n\\end{equation*}\r\nConsequently,\r\n\\begin{equation*}\r\n  x^\\dag - x^* = x^\\dag - x_1 + x_1-x^*\\in\\mathcal{N}(F'(x^\\dag))^\\perp.\r\n\\end{equation*}\r\nThis and the inequalities from Lemma \\ref{lem:linear}(i) imply $x^*=x^\\dag$, completing the proof.\r\n\\end{proof}","sha256_utf8":"bef39d889b98bf762b1b3d3a983110d73d4c5ffde4f64f132181770bb430bcae"},{"start":41126,"end":41522,"start_line":613,"end_line":620,"raw_tex":"\\begin{lemma}\\label{lem:conti-noise}\r\nLet Assumption \\ref{ass:sol}(i) be fulfilled. For any fixed $k\\in \\mathbb{N}$ and any path\r\n$(i_1,\\ldots,i_{k-1})\\in\\mathcal{F}_k$, let $x_k$ and $x_k^\\delta$ be the SGD iterates along the path for exact\r\ndata $y^\\dag$ and noisy data $y^\\delta$, respectively. Then\r\n\\begin{equation*}\r\n  \\lim_{\\delta\\to 0^+}\\|x_k^\\delta-x_k\\|=0.\r\n\\end{equation*}\r\n\\end{lemma}","sha256_utf8":"dd10595c6545b6ec43d9abdfeb2f938e0d5d9f8ee5a7c5f2471ddcf9b251a36d"},{"start":41524,"end":43015,"start_line":621,"end_line":643,"raw_tex":"\\begin{proof}\r\nWe prove the assertion by mathematical induction. The assertion holds\r\ntrivially for $k=1$, since $x_1^\\delta = x_1$. Now suppose that it\r\nholds for all indices up to $k$ and any path $(i_1,\\ldots,i_{k-1})\\in\\mathcal{F}_k$.\r\nNext, by the definitions of the SGD iterates $x_k$ and $x_k^\\delta$, cf. \\eqref{eqn:sgd}:\r\n\\begin{align*}\r\n   x_{k+1}        &= x_k - \\eta_k F_{i_k}'(x_k)^*(F_{i_k}(x_k)-y_{i_k}),\\\\\r\n   x_{k+1}^\\delta &= x_k^\\delta - \\eta_k F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta).\r\n\\end{align*}\r\nTherefore, for any fixed path $(i_1,\\ldots,i_k)$, we have\r\n\\begin{align*}\r\n   x_{k+1}^\\delta -x_{k+1} &= (x_k^\\delta-x_k) - \\eta_k \\left(F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)-F_{i_k}'(x_k)^*(F_{i_k}(x_k)-y_{i_k})\\right)\\\\\r\n     & = (x_k^\\delta -x_k)-\\eta_k \\big((F_{i_k}'(x_k^\\delta)^* - F_{i_k}'(x_k)^*)(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta) \\\\\r\n     & \\quad +F_{i_k}'(x_k)^*((F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)-(F_{i_k}(x_k)-y_{i_k}))\\big).\r\n\\end{align*}\r\nThus, by triangle inequality,\r\n\\begin{align*}\r\n  \\|x_{k+1}^\\delta-x_{k+1}\\| &\\leq \\|x_k^\\delta -x_k\\|+\\eta_k \\|F_{i_k}'(x_k^\\delta)^* - F_{i_k}'(x_k)^*\\|\\|F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\| \\\\\r\n     & \\quad +\\|F_{i_k}'(x_k)^*\\|\\|((F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)-(F_{i_k}(x_k)-y_{i_k}))\\|.\r\n\\end{align*}\r\nThen the desired assertion follows from the continuity of the operators $F_i$ and $F_i'$ in Assumption \\ref{ass:sol}(i)\r\nand the induction hypothesis.\r\n\\end{proof}","sha256_utf8":"12651afa1f1cf6c55c412dbcbe4908984afd766a1115542735bebd154756d8dc"},{"start":43171,"end":45173,"start_line":647,"end_line":672,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{thm:conv-noisy}]\r\nLet $\\{\\delta_n\\}_{n\\geq 1}\\subset\\mathbb{R}$ be a sequence converging to zero, and let $y_n:=y^{\\delta_n}$\r\nbe a corresponding sequence of noisy data. For each pair $(\\delta_n,y_n)$, we denote by\r\n$k_n=k(\\delta_n)$ the stopping index. Without loss of generality, we may assume that\r\n$k_n$ increases strictly monotonically with $n$.\r\n\r\nBy Proposition \\ref{prop:mono-exact} and Young's inequality $2ab\\leq \\epsilon a^2 + \\epsilon^{-1} b^2$,\r\nwith the choice $a=\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2]^\\frac{1}{2}$, $b=(1+\\eta)\\delta$ and $\\epsilon=1-2\\eta>0$:\r\n\\begin{align*}\r\n    & \\E[\\|x^*-x_{k+1}^\\delta\\|^2] - \\E[\\|x^*-x_{k}^\\delta\\|^2] \\\\\r\n   \\leq & -(1-2\\eta)\\eta_k\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2] +  2\\eta_k(1+\\eta)\\delta\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2]^\\frac{1}{2}\r\n   \\leq \\frac{(1+\\eta)^2}{1-2\\eta}\\eta_k\\delta^2.\r\n\\end{align*}\r\nThen for any $m<n$, summing the above inequality with $\\delta=\\delta_n$ from $k_m$ to $k_n-1$\r\n(since $k_n$ is strictly increasing with $n$ by assumption) and applying the triangle inequality lead to\r\n\\begin{align*}\r\n  \\E[\\|x_{k_n}^{\\delta_n} -x^*\\|^2] &\\leq \\E[\\|x_{k_m}^{\\delta_n}-x^*\\|^2] + \\frac{(1+\\eta)^2}{1-2\\eta}\\delta_n^2\\sum_{j=k_m}^{k_n-1}\\eta_j\\\\\r\n     & \\leq 2\\E[\\|x_{k_m}^{\\delta_n}-x_{k_m}\\|^2] + 2\\E[\\|x_{k_m}-x^*\\|^2] + \\frac{(1+\\eta)^2}{1-2\\eta}\\delta_n^2\\sum_{j=1}^{k_n-1}\\eta_j.\r\n\\end{align*}\r\nBy Theorem \\ref{thm:conv-exact} we can fix $m$ so large that the term $\\E[\\|x_{k_m}-x^*\\|^2]$ is sufficiently small.\r\nSince the index $k_m$ is fixed, we may apply Lemma \\ref{lem:conti-noise} to conclude that the term\r\n$\\E[\\|x_{k_m}^{\\delta_n}-x_{k_m}\\|^2]$ must go to zero as $n\\to\\infty$. The last term also tends to zero under the\r\ncondition on the index $k_n$, i.e., $\\lim_{n\\to\\infty}\\delta_n^2\\sum_{i=1}^{k_n}\\eta_i=0$. This completes\r\nthe proof of the first assertion. The case for $\\mathcal{N}(F'(x^\\dag))\\subset\\mathcal{N}(F'(x))$ follows similarly\r\nas Theorem \\ref{thm:conv-exact}.\r\n\\end{proof}","sha256_utf8":"0b8171af26198965f7186592e93085f283e4e58993964fb7337dde635ff22a34"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"非线性方程与空间","source_units":[{"start":3824,"end":5030,"start_line":83,"end_line":100,"raw_tex":"This work is concerned with the numerical solution of the system of nonlinear ill-posed operator equations\r\n\\begin{equation}\\label{eqn:nonlin}\r\n  F_i(x) = y_i^\\dag,\\quad i =1,\\ldots, n,\r\n\\end{equation}\r\nwhere each $F_i: \\mathcal{D}(F_i)\\to Y$ is a nonlinear mapping\r\nwith its domain $\\mathcal{D}(F_i)\\subset X$, and $X$ and $Y$ are Hilbert spaces with inner\r\nproducts $\\langle\\cdot,\\cdot\\rangle$ and norms $\\|\\cdot\\|$, respectively.\r\nThe number %$n\\in\\mathbb{N}$\r\n$n$ of nonlinear equations in \\eqref{eqn:nonlin} can potentially be very large.\r\nThe notation $y_i^\\dag \\in Y$ denotes the exact data (corresponding to the reference solution\r\n$x^\\dag\\in X$ to be defined below, i.e., $y^\\dag=F(x^\\dag)$). Equivalently, problem \\eqref{eqn:nonlin} can be rewritten as\r\n\\begin{equation}\\label{eqn:nonlin-gen}\r\n  F(x) = y^\\dag,\r\n\\end{equation}\r\nwith the operator $F: X \\to Y^n$ ($Y^n$ denotes the product space $Y\\times \\cdots \\times Y$) and $y^\\dag\\in Y^n$ defined by\r\n\\begin{equation*}\r\n  F(x)=\\frac{1}{\\sqrt{n}}\\left(\\begin{array}{c}F_1(x)\\\\ \\ldots\\\\ F_n(x)\\end{array}\\right)\\quad \\mbox{and}\\quad y^\\dag=\\frac{1}{\\sqrt{n}}\\left(\\begin{array}{c}y_1^\\dag\\\\ \\ldots\\\\ y_n^\\dag\\end{array}\\right),\r\n\\end{equation*}\r\n","sha256_utf8":"a1d5983bdd0387a77198364b5fdbca44cd9ee7443f5f30883d2206e68929db68"}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","title":"数据噪声模型","source_units":[{"start":5030,"end":5346,"start_line":101,"end_line":106,"raw_tex":"respectively. The scaling $n^{-\\frac12}$ above is introduced for the convenience of later discussions. In practice,\r\ninstead of the exact data $y^\\dag$, we have access only to the noisy data $y^\\delta$ of a noise level\r\n$\\delta\\ge 0$, namely\r\n\\begin{equation*}\r\n  \\|y^\\delta - y^\\dag\\| = \\delta\\,.\r\n\\end{equation*}\r\n","sha256_utf8":"0330d400998424a5b8e33cc14b744b52e3938c15bb6a42e930d8c9802daad360"}]},{"id":"S003","semantic_type":"algorithm","kind":"algorithm","title":"随机梯度迭代","source_units":[{"start":7611,"end":8161,"start_line":127,"end_line":140,"raw_tex":"\\begin{algorithm}\r\n  \\caption{Stochastic gradient method for problem \\eqref{eqn:nonlin}.\\label{alg:sgd}}\r\n  \\begin{algorithmic}[1]\r\n    \\STATE Given initial guess $x_1$.\r\n    \\FOR{$k=1,2,\\ldots$}\r\n      \\STATE Randomly draw an index $i_k$;\r\n      \\STATE Update the iterate $x_k^\\delta$ by\r\n      \\begin{equation}\\label{eqn:sgd}\r\n        x_{k+1}^\\delta = x_k^\\delta - \\eta_k F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta);\r\n      \\end{equation}\r\n      \\STATE Check the stopping criterion.\r\n    \\ENDFOR\r\n  \\end{algorithmic}\r\n\\end{algorithm}","sha256_utf8":"4d242ab2e4a214675c37ba4e193dc325880c02fef484fc5c3bd0229d81af6d2e"}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","title":"均匀随机抽样","source_units":[{"start":6731,"end":7000,"start_line":119,"end_line":120,"raw_tex":"index $i_k$ of the equation  at the $k$th iteration is drawn uniformly from the index set $\\{1,\\ldots,n\\}$, and $\\eta_k>0$ is the\r\ncorresponding step size. This algorithm has demonstrated very encouraging numerical results in \\cite{ChenLiLiu:2018} for diffuse optical\r\n","sha256_utf8":"3d66a4935908843dfb7b5a6c52200690db455035e5d734d14a139ce54653bb22"}]},{"id":"S005","semantic_type":"notation","kind":"notation","title":"精确迭代与滤过","source_units":[{"start":12647,"end":13061,"start_line":192,"end_line":194,"raw_tex":"Throughout, we denote the iterate for the exact data $y^\\dag$ by $x_k$. The notation $\\mathcal{F}_k$ denotes the filtration generated\r\nby the random indices $\\{i_1,\\ldots,i_{k-1}\\}$ up to the $(k-1)$th iteration. The notation $c$, with or without a subscript, denotes\r\na generic constant, which may differ at each occurrence, but it is always independent of the noise level $\\delta$ and the iteration number $k$.\r\n","sha256_utf8":"5c582dcdf727dc03f3c5c02b25620006a4906173b0fda8cfe36084681aa8c487"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"最小距离参考解","source_units":[{"start":13929,"end":14301,"start_line":205,"end_line":207,"raw_tex":"assumptions on the nonlinear operators $F_i$ and the reference solution $x^\\dag$. Since the solution to problem\r\n\\eqref{eqn:nonlin} may be nonunique, the reference solution $x^\\dag$ is taken to be the minimum norm solution (with\r\nrespect to the initial guess $x_1$), which is known to be unique under Assumption \\ref{ass:sol}(ii) below \\cite{HankeNeubauerScherzer:1995}.\r\n","sha256_utf8":"806a4dd7769a28a90ffd8d9f2501857d2ca44c79165f019b09eac42807768672"}]},{"id":"S007","semantic_type":"assumption","kind":"assumption","title":"算子与参考解假设","source_units":[{"start":14303,"end":15359,"start_line":209,"end_line":230,"raw_tex":"\\begin{assumption}\\label{ass:sol}\r\nThe following conditions hold:\r\n\\begin{itemize}\r\n  \\item[$\\rm(i)$] The operators $F_i$, $i=1,\\ldots,n$, are continuous, with continuous and bounded derivatives on $X$.\r\n  \\item[$\\rm(ii)$] There exists an $\\eta\\in(0,\\frac12)$ such that for any $x,\\tilde x\\in X$,\r\n  \\begin{equation}\\label{eqn:tangential-cone}\r\n    \\|F(x)-F(\\tilde x)-F'(\\tilde x)(x-\\tilde x)\\|\\leq \\eta\\|F(x)-F(\\tilde x)\\|.\r\n  \\end{equation}\r\n  \\item[$\\rm(iii)$] There are a family of uniformly bounded operators $R^i_x$ such that for any $x\\in X$,\r\n  \\begin{equation*}\r\n    F_i'(x) = R_x^iF_i'(x^\\dag)\r\n  \\end{equation*}\r\n  and $R_x=\\mathrm{diag}(R_x^i):Y^n\\to Y^n$, with {\\rm(}with $\\|\\cdot\\|$ denoting the operator norm on $Y^n${\\rm)}\r\n  \\begin{equation*}\r\n    \\|R_x-I\\|\\leq c_R\\|x-x^\\dag\\|.\r\n  \\end{equation*}\r\n\\item[$\\rm(iv)$] The following source condition holds: there exist some $\\nu\\in (0,\\frac12)$ and $w\\in X$ such that\r\n  \\begin{equation*}\r\n    x^\\dag - x_1 = (F'(x^\\dag)^*F'(x^\\dag))^\\nu w.\r\n  \\end{equation*}\r\n\\end{itemize}\r\n\\end{assumption}","sha256_utf8":"ebdfffd8d5d9535ebe0460b7dbdd376845ce25980e088ae64b8691e9fc86b03a"}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","title":"步长条件与调度","source_units":[{"start":17483,"end":17863,"start_line":256,"end_line":262,"raw_tex":"\\begin{assumption}\\label{ass:stepsize}\r\nThe step sizes $\\{\\eta_k\\}_{k\\geq 1}$ satisfy one of the following properties.\r\n\\begin{itemize}\r\n\\item[$\\rm(i)$] $\\eta_k\\max_i\\sup_x\\|F_i'(x)\\|^2<1$ and $\\sum_{k=1}^\\infty \\eta_k =\\infty$.\r\n\\item[$\\rm(ii)$] $\\eta_k=\\eta_0k^{-\\alpha}$, with $\\alpha\\in(0,1)$ and $\\eta_0\\leq (\\max_i\\sup_x\\|F_i'(x)\\|^2)^{-1}$.\r\n\\end{itemize}\r\n\\end{assumption}","sha256_utf8":"5bac8990b519db696792ce9a2265f2a57f9ecd89446138eb76fe6ea6121967cb"}]},{"id":"S009","semantic_type":"notation","kind":"notation","title":"均方收敛与核记号","source_units":[{"start":17867,"end":18319,"start_line":264,"end_line":268,"raw_tex":"Due to the random choice of the index $i_k$, the SGD iterate $x_k^\\delta$ is a random variable. There are\r\nseveral different ways to measure the convergence. We shall employ the mean squared norm defined by\r\n$\\E[\\|\\cdot\\|^2]$, where the expectation $\\E[\\cdot]$ is with respect to the filtration $\\mathcal{F}_k$\r\ngenerated by the random indices $i_j$, $j=1,\\ldots,k-1$. Clearly, the iterate $x_k^\\delta$ is measurable\r\nwith respect to $\\mathcal{F}_k$.\r\n","sha256_utf8":"da947cf6a4f1b38f4fa9627f55eebb94c22bcbc29f563e6429b5d2d093f156bd"},{"start":18558,"end":18634,"start_line":272,"end_line":272,"raw_tex":"the notation $\\mathcal{N}(\\cdot)$ denotes the kernel of a linear operator.\r\n","sha256_utf8":"356878a63ac1c58e7739e3fae8c402108e0e6e36390f0d42db73140533771842"}]},{"id":"S010","semantic_type":"theorem","kind":"theorem","title":"先验停止下的随机正则化","source_units":[{"start":18634,"end":19367,"start_line":273,"end_line":286,"raw_tex":"\\begin{theorem}[convergence for noisy data]\\label{thm:conv-noisy}\r\nLet Assumptions \\ref{ass:sol}(i)-(ii) and \\ref{ass:stepsize}(i) be fulfilled. If the stopping index $k(\\delta)\\in\\mathbb{N}$ is chosen such that\r\n\\begin{equation*}\r\n  \\lim_{\\delta\\to0^+}k(\\delta)=\\infty \\quad\\mbox{and}\\quad \\lim_{\\delta\\to0^+}\\delta^2\\sum_{i=1}^{k(\\delta)}\\eta_i = 0,\r\n\\end{equation*}\r\nthen there exists a solution $x^*\\in X$ to problem \\eqref{eqn:nonlin} such that\r\n\\begin{equation*}\r\n  \\lim_{\\delta\\to 0^+}\\E[\\|x_{k(\\delta)}^\\delta - x^*\\|^2]=0.\r\n\\end{equation*}\r\nFurther, if $\\mathcal{N}(F'(x^\\dag))\\subset\\mathcal{N}(F'(x))$, then\r\n\\begin{equation*}\r\n  \\lim_{\\delta\\to0^+}\\E[\\|x_{k(\\delta)}^\\delta -x^\\dag\\|^2]=0.\r\n\\end{equation*}\r\n\\end{theorem}","sha256_utf8":"2b036de0d537dbe5cd9b7a65ce26bad425b9fc2a140d7fa78b9131603b5d648d"}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","title":"停止指标条件","source_units":[{"start":18701,"end":19004,"start_line":274,"end_line":277,"raw_tex":"Let Assumptions \\ref{ass:sol}(i)-(ii) and \\ref{ass:stepsize}(i) be fulfilled. If the stopping index $k(\\delta)\\in\\mathbb{N}$ is chosen such that\r\n\\begin{equation*}\r\n  \\lim_{\\delta\\to0^+}k(\\delta)=\\infty \\quad\\mbox{and}\\quad \\lim_{\\delta\\to0^+}\\delta^2\\sum_{i=1}^{k(\\delta)}\\eta_i = 0,\r\n\\end{equation*}\r\n","sha256_utf8":"c181d3476d847be33b8a4e964238768b9931540da42a122166cb58be13c0f6d1"}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","title":"导数核包含条件","source_units":[{"start":19184,"end":19254,"start_line":282,"end_line":282,"raw_tex":"Further, if $\\mathcal{N}(F'(x^\\dag))\\subset\\mathcal{N}(F'(x))$, then\r\n","sha256_utf8":"53eef00cb5ad45e6b719677b4fef49c0bf1603581c7ab3f70bd09d2d4ee4c8cc"}]},{"id":"S013","semantic_type":"proof","kind":"proof","title":"随机正则化完整证明","source_units":[{"start":43171,"end":45173,"start_line":647,"end_line":672,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{thm:conv-noisy}]\r\nLet $\\{\\delta_n\\}_{n\\geq 1}\\subset\\mathbb{R}$ be a sequence converging to zero, and let $y_n:=y^{\\delta_n}$\r\nbe a corresponding sequence of noisy data. For each pair $(\\delta_n,y_n)$, we denote by\r\n$k_n=k(\\delta_n)$ the stopping index. Without loss of generality, we may assume that\r\n$k_n$ increases strictly monotonically with $n$.\r\n\r\nBy Proposition \\ref{prop:mono-exact} and Young's inequality $2ab\\leq \\epsilon a^2 + \\epsilon^{-1} b^2$,\r\nwith the choice $a=\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2]^\\frac{1}{2}$, $b=(1+\\eta)\\delta$ and $\\epsilon=1-2\\eta>0$:\r\n\\begin{align*}\r\n    & \\E[\\|x^*-x_{k+1}^\\delta\\|^2] - \\E[\\|x^*-x_{k}^\\delta\\|^2] \\\\\r\n   \\leq & -(1-2\\eta)\\eta_k\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2] +  2\\eta_k(1+\\eta)\\delta\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2]^\\frac{1}{2}\r\n   \\leq \\frac{(1+\\eta)^2}{1-2\\eta}\\eta_k\\delta^2.\r\n\\end{align*}\r\nThen for any $m<n$, summing the above inequality with $\\delta=\\delta_n$ from $k_m$ to $k_n-1$\r\n(since $k_n$ is strictly increasing with $n$ by assumption) and applying the triangle inequality lead to\r\n\\begin{align*}\r\n  \\E[\\|x_{k_n}^{\\delta_n} -x^*\\|^2] &\\leq \\E[\\|x_{k_m}^{\\delta_n}-x^*\\|^2] + \\frac{(1+\\eta)^2}{1-2\\eta}\\delta_n^2\\sum_{j=k_m}^{k_n-1}\\eta_j\\\\\r\n     & \\leq 2\\E[\\|x_{k_m}^{\\delta_n}-x_{k_m}\\|^2] + 2\\E[\\|x_{k_m}-x^*\\|^2] + \\frac{(1+\\eta)^2}{1-2\\eta}\\delta_n^2\\sum_{j=1}^{k_n-1}\\eta_j.\r\n\\end{align*}\r\nBy Theorem \\ref{thm:conv-exact} we can fix $m$ so large that the term $\\E[\\|x_{k_m}-x^*\\|^2]$ is sufficiently small.\r\nSince the index $k_m$ is fixed, we may apply Lemma \\ref{lem:conti-noise} to conclude that the term\r\n$\\E[\\|x_{k_m}^{\\delta_n}-x_{k_m}\\|^2]$ must go to zero as $n\\to\\infty$. The last term also tends to zero under the\r\ncondition on the index $k_n$, i.e., $\\lim_{n\\to\\infty}\\delta_n^2\\sum_{i=1}^{k_n}\\eta_i=0$. This completes\r\nthe proof of the first assertion. The case for $\\mathcal{N}(F'(x^\\dag))\\subset\\mathcal{N}(F'(x))$ follows similarly\r\nas Theorem \\ref{thm:conv-exact}.\r\n\\end{proof}","sha256_utf8":"0b8171af26198965f7186592e93085f283e4e58993964fb7337dde635ff22a34"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"选取噪声与停止子序列","source_units":[{"start":43225,"end":43559,"start_line":648,"end_line":651,"raw_tex":"Let $\\{\\delta_n\\}_{n\\geq 1}\\subset\\mathbb{R}$ be a sequence converging to zero, and let $y_n:=y^{\\delta_n}$\r\nbe a corresponding sequence of noisy data. For each pair $(\\delta_n,y_n)$, we denote by\r\n$k_n=k(\\delta_n)$ the stopping index. Without loss of generality, we may assume that\r\n$k_n$ increases strictly monotonically with $n$.\r\n","sha256_utf8":"0f8cd98ab4c6bd6d6d363e17822db2117213c470e69392f93c338694de8f308e"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"控制单步噪声增长","source_units":[{"start":43561,"end":44056,"start_line":653,"end_line":659,"raw_tex":"By Proposition \\ref{prop:mono-exact} and Young's inequality $2ab\\leq \\epsilon a^2 + \\epsilon^{-1} b^2$,\r\nwith the choice $a=\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2]^\\frac{1}{2}$, $b=(1+\\eta)\\delta$ and $\\epsilon=1-2\\eta>0$:\r\n\\begin{align*}\r\n    & \\E[\\|x^*-x_{k+1}^\\delta\\|^2] - \\E[\\|x^*-x_{k}^\\delta\\|^2] \\\\\r\n   \\leq & -(1-2\\eta)\\eta_k\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2] +  2\\eta_k(1+\\eta)\\delta\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2]^\\frac{1}{2}\r\n   \\leq \\frac{(1+\\eta)^2}{1-2\\eta}\\eta_k\\delta^2.\r\n\\end{align*}\r\n","sha256_utf8":"585d62b56d1f9fb29063e989c4b8ad4be674be0ea7257370e2998a829430ef47"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"累加并分离三项误差","source_units":[{"start":44056,"end":44570,"start_line":660,"end_line":665,"raw_tex":"Then for any $m<n$, summing the above inequality with $\\delta=\\delta_n$ from $k_m$ to $k_n-1$\r\n(since $k_n$ is strictly increasing with $n$ by assumption) and applying the triangle inequality lead to\r\n\\begin{align*}\r\n  \\E[\\|x_{k_n}^{\\delta_n} -x^*\\|^2] &\\leq \\E[\\|x_{k_m}^{\\delta_n}-x^*\\|^2] + \\frac{(1+\\eta)^2}{1-2\\eta}\\delta_n^2\\sum_{j=k_m}^{k_n-1}\\eta_j\\\\\r\n     & \\leq 2\\E[\\|x_{k_m}^{\\delta_n}-x_{k_m}\\|^2] + 2\\E[\\|x_{k_m}-x^*\\|^2] + \\frac{(1+\\eta)^2}{1-2\\eta}\\delta_n^2\\sum_{j=1}^{k_n-1}\\eta_j.\r\n\\end{align*}\r\n","sha256_utf8":"8f1a20e6c8a25f8ddf9536ec43846031fe3adc09e5ceadb59e0108bb50735838"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"取极限并识别参考解","source_units":[{"start":44570,"end":45162,"start_line":666,"end_line":671,"raw_tex":"By Theorem \\ref{thm:conv-exact} we can fix $m$ so large that the term $\\E[\\|x_{k_m}-x^*\\|^2]$ is sufficiently small.\r\nSince the index $k_m$ is fixed, we may apply Lemma \\ref{lem:conti-noise} to conclude that the term\r\n$\\E[\\|x_{k_m}^{\\delta_n}-x_{k_m}\\|^2]$ must go to zero as $n\\to\\infty$. The last term also tends to zero under the\r\ncondition on the index $k_n$, i.e., $\\lim_{n\\to\\infty}\\delta_n^2\\sum_{i=1}^{k_n}\\eta_i=0$. This completes\r\nthe proof of the first assertion. The case for $\\mathcal{N}(F'(x^\\dag))\\subset\\mathcal{N}(F'(x))$ follows similarly\r\nas Theorem \\ref{thm:conv-exact}.\r\n","sha256_utf8":"91ede40e8ad655e3147d127aa4c7507bf6cbfd6d059fd3241427b1faffd2da88"}]},{"id":"S018","semantic_type":"lemma","kind":"lemma","title":"切向锥范数与解集刻画","source_units":[{"start":25775,"end":26317,"start_line":368,"end_line":378,"raw_tex":"\\begin{lemma}\\label{lem:linear}\r\nLet Assumptions \\ref{ass:sol}(i) and (ii) be fulfilled. Then the following statements hold.\r\n\\begin{itemize}\r\n  \\item[$\\rm(i)$] The following inequalities hold:\r\n  \\begin{equation*}\r\n  \\frac{1}{1+\\eta}\\|F'(x)(x-\\tilde x)\\|\\leq \\|F(x)-F(\\tilde x)\\|\\leq \\frac{1}{1-\\eta}\\|F'(x)(x-\\tilde x)\\|.\r\n  \\end{equation*}\r\n  \\item[$\\rm(ii)$]If $x^*$ is a solution of \\eqref{eqn:nonlin}, then any other solution $\\tilde x^*$ satisfies\r\n$x^*-\\tilde x^* \\in \\mathcal{N}(F'(x^*))$, and vice versa.\r\n\\end{itemize}\r\n\\end{lemma}","sha256_utf8":"fd334fc6b6335dfe674c98ef60dc594cdf9444c91d4bfd20b71269f7d30bad2e"}]},{"id":"S019","semantic_type":"external_fact","kind":"external_fact","title":"已有切向锥解集定理","source_units":[],"declared_external":true,"declaration_text":["Under the continuity and tangential cone assumptions, the derivative-residual norm comparisons and solution-nullspace equivalence stated in B0032 hold; this is the cited Proposition 2.1 of HankeNeubauerScherzer:1995."]},{"id":"S020","semantic_type":"external_fact","kind":"external_fact","title":"希尔伯特空间与概率工具","source_units":[],"declared_external":true,"declaration_text":["L2 of Hilbert-valued random variables is complete and has inner product E[<u,v>]. Triangle, Cauchy–Schwarz and Young inequalities hold; conditional expectation obeys the tower property and uniform finite-index averaging. For a fixed finite number of sampling steps, pathwise convergence implies convergence of the corresponding finite expectation. Orthogonal projection onto an affine closed subspace gives its unique nearest point."]},{"id":"S021","semantic_type":"proposition","kind":"proposition","title":"带噪迭代的近单调性","source_units":[{"start":26537,"end":26962,"start_line":383,"end_line":389,"raw_tex":"\\begin{prop}\\label{prop:mono-exact}\r\nLet Assumptions \\ref{ass:sol}(i)-(ii) and \\ref{ass:stepsize}(i) be fulfilled. Then for any solution $x^*$ to problem \\eqref{eqn:nonlin}, there holds\r\n\\begin{equation*}\r\n  \\E[\\|x^*-x_{k+1}^\\delta\\|^2] - \\E[\\|x^*-x_{k}^\\delta\\|^2]\r\n   \\leq  -(1-2\\eta)\\eta_k\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2] + { 2\\eta_k(1+\\eta)\\delta\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2]^\\frac{1}{2}}.\r\n\\end{equation*}\r\n\\end{prop}","sha256_utf8":"ac1c1c2c846ad3bcb4433e948a6306a490856e37a49c4c7a0699c278b8510fdc"}]},{"id":"S022","semantic_type":"proof","kind":"proof","title":"近单调性完整证明","source_units":[{"start":26964,"end":30885,"start_line":390,"end_line":436,"raw_tex":"\\begin{proof}\r\nCompleting the square gives\r\n\\begin{align*}\r\n    \\|x^*-x_{k+1}^\\delta\\|^2 - \\|x^*-x_{k}^\\delta\\|^2 = & 2\\langle x_k^\\delta - x^*,x_{k+1}^\\delta-x_k^\\delta\\rangle + \\|x_{k+1}^\\delta-x_k^\\delta\\|^2.\r\n\\end{align*}\r\nBy the definition of the SGD iterate $x_{k}^\\delta$ in \\eqref{eqn:sgd}, there holds\r\n\\begin{align*}\r\n   & \\|x^*-x_{k+1}^\\delta\\|^2 - \\|x^*-x_{k}^\\delta\\|^2\\\\\r\n  = & -2\\eta_k\\langle x_k^\\delta-x^*,  F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)\\rangle + \\eta_k^2\\| F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)\\|^2 \\\\\r\n  = & -2\\eta_k\\langle F_{i_k}'(x_k^\\delta)(x_k^\\delta-x^*), F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle + \\eta_k^2\\|F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)\\|^2.\r\n\\end{align*}\r\nNext we split the factor $-F'_{i_k}(x_k^\\delta)(x_k^\\delta-x^*)$ into two terms\r\n\\begin{equation*}\r\n  F_{i_k}'(x_k^\\delta)(x_k^\\delta-x^*) =  (F_{i_k}(x_k^\\delta)- F_{i_k}(x^*)) + (F_{i_k}(x^*)-F_{i_k}(x_k^\\delta)-F_{i_k}'(x_k^\\delta)(x^* - x_k^\\delta)).\r\n\\end{equation*}\r\nCombining the last two identities yields\r\n\\begin{align*}\r\n    &\\|x^*-x_{k+1}^\\delta\\|^2 - \\|x^*-x_{k}^\\delta\\|^2\\\\\r\n  = & -2\\eta_k\\langle F_{i_k}(x_k^\\delta)-F_{i_k}(x^*), F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle +  {\\eta_k^2\\|F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)\\|^2}\\\\\r\n   & - 2\\eta_k\\langle y_{i_k}^\\dag-F_{i_k}(x_k^\\delta)-F_{i_k}'(x_k^\\delta)(x^*-x_k^\\delta),F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle \\\\\r\n  = & -2\\eta_k\\langle F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta, F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle +  {\\eta_k^2\\|F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)\\|^2}\\\\\r\n   & -2\\eta_k\\langle y_{i_k}^\\delta-y_{i_k}^\\dag , F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle\\\\\r\n   & - 2\\eta_k\\langle y_{i_k}^\\dag-F_{i_k}(x_k^\\delta)-F_{i_k}'(x_k^\\delta)(x^*-x_k^\\delta),F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle \\\\\r\n \\leq  & -\\eta_k\\langle F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta, F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle\r\n    -2\\eta_k\\langle y_{i_k}^\\delta-y_{i_k}^\\dag, F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle\\\\\r\n   & - 2\\eta_k\\langle y_{i_k}^\\dag-F_{i_k}(x_k^\\delta)-F_{i_k}'(x_k^\\delta)(x^*-x_k^\\delta),F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle,\r\n\\end{align*}\r\nwhere the inequality follows from the condition $\\eta_k\\|F_{i_k}'(x)\\|^2<1$ in Assumption \\ref{ass:stepsize}(i).\r\nThus, by the measurability of the iterate $x_k$ with respect to the filtration\r\n$\\mathcal{F}_{k}$ and the Cauchy-Schwarz inequality, we have\r\n\\begin{align*}\r\n   &{\\E[\\|x^*-x_{k+1}^\\delta\\|^2 - \\|x^*-x_{k}^\\delta\\|^2|\\mathcal{F}_{k}]} \\\\\r\n    \\leq &-\\frac{\\eta_k}{n}\\sum_{i=1}^n\\langle F_{i}(x_k^\\delta)-y_{i}^\\delta,F_{i}(x_k^\\delta)-y_{i}^\\delta\\rangle-2\\frac{\\eta_k}{n}\\sum_{i=1}^n\\langle y_{i}^\\delta-y_{i}^\\dag, F_{i}(x_k^\\delta)-y_{i}^\\delta\\rangle\\\\\r\n   & - 2\\frac{\\eta_k}{n}\\sum_{i=1}^n\\langle y_{i}^\\dag-F_{i}(x_k^\\delta)-F_{i}'(x_k^\\delta)(x^*-x_k^\\delta),F_{i}(x_k^\\delta)-y_{i}^\\delta\\rangle \\\\\r\n  = & -\\eta_k\\|F(x_k^\\delta)-y^\\delta\\|^2 - 2\\eta_k\\langle y^\\delta-y^\\dag, F(x_k^\\delta)-y^\\delta\\rangle- 2\\eta_k\\langle y^\\dag-F(x_k^\\delta)-F'(x_k^\\delta)(x^*-x_k^\\delta),F(x_k^\\delta)-y^\\delta\\rangle \\\\\r\n  \\leq & -\\eta_k\\|F(x_k^\\delta)-y^\\delta\\|^2 + 2\\eta_k\\delta\\|F(x_k^\\delta)-y^\\delta\\| + 2\\eta_k\\eta\\|F(x_k^\\delta)-y^\\dag\\|\\|F(x_k^\\delta)-y^\\delta\\|\\\\\r\n  \\leq & \\eta_k\\|F(x_k^\\delta)-y^\\delta\\|\\big((2\\eta-1)\\|F(x_k^\\delta)-y^\\delta\\| + 2(1+\\eta)\\delta\\big),\r\n\\end{align*}\r\nwhere the second inequality follows from Assumption \\ref{ass:sol}(i) and the triangle inequality.\r\nLast, by taking full conditional of the inequality yields % and applying Young's inequality yield\r\n\\begin{align*}\r\n   &\\E[\\|x^*-x_{k+1}^\\delta\\|^2] - \\E[\\|x^*-x_{k}^\\delta\\|^2]\\\\\r\n   \\leq &  -(1-2\\eta)\\eta_k\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2]\r\n   +  2\\eta_k(1+\\eta)\\delta\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2]^\\frac{1}{2}.\r\n\\end{align*}\r\nThis completes the proof of the proposition.\r\n\\end{proof}","sha256_utf8":"c002c99c525c2a6dafacb5d5b673808dceb8cf02f590e10e6f7db51ba07525fa"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"展开平方并代入迭代","source_units":[{"start":26979,"end":27728,"start_line":391,"end_line":400,"raw_tex":"Completing the square gives\r\n\\begin{align*}\r\n    \\|x^*-x_{k+1}^\\delta\\|^2 - \\|x^*-x_{k}^\\delta\\|^2 = & 2\\langle x_k^\\delta - x^*,x_{k+1}^\\delta-x_k^\\delta\\rangle + \\|x_{k+1}^\\delta-x_k^\\delta\\|^2.\r\n\\end{align*}\r\nBy the definition of the SGD iterate $x_{k}^\\delta$ in \\eqref{eqn:sgd}, there holds\r\n\\begin{align*}\r\n   & \\|x^*-x_{k+1}^\\delta\\|^2 - \\|x^*-x_{k}^\\delta\\|^2\\\\\r\n  = & -2\\eta_k\\langle x_k^\\delta-x^*,  F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)\\rangle + \\eta_k^2\\| F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)\\|^2 \\\\\r\n  = & -2\\eta_k\\langle F_{i_k}'(x_k^\\delta)(x_k^\\delta-x^*), F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle + \\eta_k^2\\|F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)\\|^2.\r\n\\end{align*}\r\n","sha256_utf8":"0ff8f2d2cc0559ec54c6badcd789475715efee9e73b4041b898f0bb096e3d2b1"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"分解残差并利用步长","source_units":[{"start":27728,"end":29310,"start_line":401,"end_line":417,"raw_tex":"Next we split the factor $-F'_{i_k}(x_k^\\delta)(x_k^\\delta-x^*)$ into two terms\r\n\\begin{equation*}\r\n  F_{i_k}'(x_k^\\delta)(x_k^\\delta-x^*) =  (F_{i_k}(x_k^\\delta)- F_{i_k}(x^*)) + (F_{i_k}(x^*)-F_{i_k}(x_k^\\delta)-F_{i_k}'(x_k^\\delta)(x^* - x_k^\\delta)).\r\n\\end{equation*}\r\nCombining the last two identities yields\r\n\\begin{align*}\r\n    &\\|x^*-x_{k+1}^\\delta\\|^2 - \\|x^*-x_{k}^\\delta\\|^2\\\\\r\n  = & -2\\eta_k\\langle F_{i_k}(x_k^\\delta)-F_{i_k}(x^*), F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle +  {\\eta_k^2\\|F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)\\|^2}\\\\\r\n   & - 2\\eta_k\\langle y_{i_k}^\\dag-F_{i_k}(x_k^\\delta)-F_{i_k}'(x_k^\\delta)(x^*-x_k^\\delta),F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle \\\\\r\n  = & -2\\eta_k\\langle F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta, F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle +  {\\eta_k^2\\|F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)\\|^2}\\\\\r\n   & -2\\eta_k\\langle y_{i_k}^\\delta-y_{i_k}^\\dag , F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle\\\\\r\n   & - 2\\eta_k\\langle y_{i_k}^\\dag-F_{i_k}(x_k^\\delta)-F_{i_k}'(x_k^\\delta)(x^*-x_k^\\delta),F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle \\\\\r\n \\leq  & -\\eta_k\\langle F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta, F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle\r\n    -2\\eta_k\\langle y_{i_k}^\\delta-y_{i_k}^\\dag, F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle\\\\\r\n   & - 2\\eta_k\\langle y_{i_k}^\\dag-F_{i_k}(x_k^\\delta)-F_{i_k}'(x_k^\\delta)(x^*-x_k^\\delta),F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\rangle,\r\n\\end{align*}\r\nwhere the inequality follows from the condition $\\eta_k\\|F_{i_k}'(x)\\|^2<1$ in Assumption \\ref{ass:stepsize}(i).\r\n","sha256_utf8":"7292fb3f738a2f26369569ab1c3816112d62eee32e9985a918df9fad7db6b36f"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"条件期望与切向锥估计","source_units":[{"start":29310,"end":30499,"start_line":418,"end_line":428,"raw_tex":"Thus, by the measurability of the iterate $x_k$ with respect to the filtration\r\n$\\mathcal{F}_{k}$ and the Cauchy-Schwarz inequality, we have\r\n\\begin{align*}\r\n   &{\\E[\\|x^*-x_{k+1}^\\delta\\|^2 - \\|x^*-x_{k}^\\delta\\|^2|\\mathcal{F}_{k}]} \\\\\r\n    \\leq &-\\frac{\\eta_k}{n}\\sum_{i=1}^n\\langle F_{i}(x_k^\\delta)-y_{i}^\\delta,F_{i}(x_k^\\delta)-y_{i}^\\delta\\rangle-2\\frac{\\eta_k}{n}\\sum_{i=1}^n\\langle y_{i}^\\delta-y_{i}^\\dag, F_{i}(x_k^\\delta)-y_{i}^\\delta\\rangle\\\\\r\n   & - 2\\frac{\\eta_k}{n}\\sum_{i=1}^n\\langle y_{i}^\\dag-F_{i}(x_k^\\delta)-F_{i}'(x_k^\\delta)(x^*-x_k^\\delta),F_{i}(x_k^\\delta)-y_{i}^\\delta\\rangle \\\\\r\n  = & -\\eta_k\\|F(x_k^\\delta)-y^\\delta\\|^2 - 2\\eta_k\\langle y^\\delta-y^\\dag, F(x_k^\\delta)-y^\\delta\\rangle- 2\\eta_k\\langle y^\\dag-F(x_k^\\delta)-F'(x_k^\\delta)(x^*-x_k^\\delta),F(x_k^\\delta)-y^\\delta\\rangle \\\\\r\n  \\leq & -\\eta_k\\|F(x_k^\\delta)-y^\\delta\\|^2 + 2\\eta_k\\delta\\|F(x_k^\\delta)-y^\\delta\\| + 2\\eta_k\\eta\\|F(x_k^\\delta)-y^\\dag\\|\\|F(x_k^\\delta)-y^\\delta\\|\\\\\r\n  \\leq & \\eta_k\\|F(x_k^\\delta)-y^\\delta\\|\\big((2\\eta-1)\\|F(x_k^\\delta)-y^\\delta\\| + 2(1+\\eta)\\delta\\big),\r\n\\end{align*}\r\nwhere the second inequality follows from Assumption \\ref{ass:sol}(i) and the triangle inequality.\r\n","sha256_utf8":"a4bf980a01274af55c7c5c10e7280e435bbb3361daacdc46a55a4a0b7563bca3"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"取全期望得到误差界","source_units":[{"start":30499,"end":30874,"start_line":429,"end_line":435,"raw_tex":"Last, by taking full conditional of the inequality yields % and applying Young's inequality yield\r\n\\begin{align*}\r\n   &\\E[\\|x^*-x_{k+1}^\\delta\\|^2] - \\E[\\|x^*-x_{k}^\\delta\\|^2]\\\\\r\n   \\leq &  -(1-2\\eta)\\eta_k\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2]\r\n   +  2\\eta_k(1+\\eta)\\delta\\E[\\|F(x_k^\\delta)-y^\\delta\\|^2]^\\frac{1}{2}.\r\n\\end{align*}\r\nThis completes the proof of the proposition.\r\n","sha256_utf8":"c396b5dbe8611af57e0768245c3a4acc429ce1867c7c5488e925cdf070e4dc0b"}]},{"id":"S027","semantic_type":"corollary","kind":"corollary","title":"精确误差单调与残差可和","source_units":[{"start":31109,"end":31682,"start_line":444,"end_line":455,"raw_tex":"\\begin{corollary}\\label{cor:mono}\r\nLet Assumptions \\ref{ass:sol}(i)-(ii) and \\ref{ass:stepsize}(i) be fulfilled. Then for the exact data $y^\\dag$,\r\nany solution $x^*$ to problem \\eqref{eqn:nonlin} satisfies\r\n\\begin{align*}\r\n   \\E[\\|x^*-x_{k+1}\\|^2] - \\E[\\|x^*-x_{k}\\|^2] &\\leq -(1-2\\eta)\\eta_k\\E[\\|F(x_k)-y^\\dag\\|^2],\\\\\r\n   \\sum_{k=1}^\\infty \\eta_k\\E[\\|F(x_k)-y^\\dag\\|^2] &\\leq \\frac{1}{1-2\\eta}\\|x^*-x_1\\|^2.\r\n\\end{align*}\r\n%and\r\n%\\begin{equation*}\r\n%  \\sum_{k=1}^\\infty \\eta_k\\E[\\|F(x_k)-y^\\dag\\|^2] \\leq \\frac{1}{1-2\\eta}\\|x^*-x_1\\|^2.\r\n%\\end{equation*}\r\n\\end{corollary}","sha256_utf8":"eafa8b356839159b59ea6961b620a05528b389bdbd0877e62bd52c3b89c73fd6"}]},{"id":"S028","semantic_type":"proof","kind":"proof","title":"精确数据推论的作者证明","source_units":[{"start":31024,"end":31107,"start_line":442,"end_line":442,"raw_tex":"The next result is an immediate consequence of Proposition \\ref{prop:mono-exact}.\r\n","sha256_utf8":"6c594955007c0cbb73df4e31c669ea3719eb6688412491aa0a99f63a9b5a1d4b"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"直接应用近单调性命题","source_units":[{"start":31024,"end":31107,"start_line":442,"end_line":442,"raw_tex":"The next result is an immediate consequence of Proposition \\ref{prop:mono-exact}.\r\n","sha256_utf8":"6c594955007c0cbb73df4e31c669ea3719eb6688412491aa0a99f63a9b5a1d4b"}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","title":"精确迭代构成柯西列","source_units":[{"start":32217,"end":32467,"start_line":465,"end_line":468,"raw_tex":"\\begin{lemma}\\label{lem:Cauchy}\r\nLet Assumptions \\ref{ass:sol}(i)-(ii) and \\ref{ass:stepsize}(i) be fulfilled. Then for the exact data $y^\\dag$,\r\nthe sequence $\\{x_k\\}_{k\\geq 1}$ generated by Algorithm \\ref{alg:sgd} is a Cauchy sequence.\r\n\\end{lemma}","sha256_utf8":"22daf487b6371d0991e564aa24adc49b2f5327603cd071c353c7b3c6d6dd72ff"}]},{"id":"S031","semantic_type":"proof","kind":"proof","title":"柯西性质完整证明","source_units":[{"start":32469,"end":37035,"start_line":469,"end_line":538,"raw_tex":"\\begin{proof}\r\nThe argument below follows closely \\cite[Theorem 2.3]{HankeNeubauerScherzer:1995}, which can be\r\ntraced back to \\cite{McCormickRodrigue:1975}. Let $x^*$ be any solution to problem \\eqref{eqn:nonlin}, and let $e_k:=x_k-x^*$. By\r\nCorollary \\ref{prop:mono-exact}, $\\E[\\|e_k\\|^2]$ is monotonically decreasing to some $\\epsilon\\geq 0$.\r\nNext we show that the sequence $\\{x_k\\}_{k\\ge 1}$ is actually a Cauchy sequence. First we note that\r\n$\\E[\\langle\\cdot,\\cdot\\rangle]$ defines an inner product. For any $j\\geq k$, choose an index $\\ell$\r\nwith $j\\geq \\ell\\geq k$ such that\r\n\\begin{equation}\\label{eqn:minimal-res}\r\n  \\E[\\|y^\\dag - F(x_\\ell)\\|^2]\\leq \\E[\\|y^\\dag - F(x_i)\\|^2], \\quad \\forall k\\leq i \\leq j.\r\n\\end{equation}\r\nIn light of the triangle inequality\r\n\\begin{equation*}\r\n  \\E[\\|e_j-e_k\\|^2]^\\frac{1}{2} \\leq \\E[\\|e_j-e_\\ell\\|^2]^\\frac{1}{2} + \\E[\\|e_\\ell-e_k\\|^2]^\\frac{1}{2},\r\n\\end{equation*}\r\nand the trivial identities\r\n\\begin{equation}\\label{eqn:err-exact}\r\n\\begin{aligned}\r\n  \\E[\\|e_j-e_\\ell\\|^2] & = 2\\E[\\langle e_\\ell-e_j,e_\\ell\\rangle] + \\E[\\|e_j\\|^2] - \\E[\\|e_\\ell\\|^2],\\\\\r\n  \\E[\\|e_\\ell-e_k\\|^2] & = 2\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle] + \\E[\\|e_k\\|^2] - \\E[\\|e_\\ell\\|^2],\r\n\\end{aligned}\r\n\\end{equation}\r\nit suffices to prove that both $\\E[\\|e_j-e_\\ell\\|^2]$ and $\\E[\\|e_\\ell-e_k\\|^2]$ on the left hand side tend to zero as $k\\to \\infty$.\r\nFor $k\\to \\infty$, the last two terms on each of the right hand side of \\eqref{eqn:err-exact} tends to $\\epsilon-\\epsilon=0$, by\r\nthe monotone convergence of $\\E[\\|e_k\\|^2]$ to $\\epsilon$.\r\nNext we show that the term $\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle]$ also tends to zero as $k\\to\\infty$.\r\nActually, by the definition of the SGD iterate $x_k$ in \\eqref{eqn:sgd}, we have\r\n\\begin{equation*}\r\n  e_\\ell - e_k = \\sum_{i=k}^{\\ell-1}(e_{i+1}-e_i) = {\\sum_{i=k}^{\\ell-1}\\eta_iF_{i_i}'(x_i)^*(y_{i_i}^\\dag-F_{i_i}(x_i))}.\r\n\\end{equation*}\r\nBy the triangle inequality, we can bound $\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle]$ by\r\n\\begin{align*}\r\n  |\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle]| & = |\\E[\\sum_{i=k}^{\\ell-1}\\langle \\eta_iF_{i_i}'(x_i)^*(y_{i_i}^\\dag-F_{i_i}(x_i)),e_\\ell\\rangle]|\\\\\r\n  & \\leq \\sum_{i=k}^{\\ell-1}\\eta_i|\\E[\\langle F_{i_i}'(x_i)^*(y_{i_i}^\\dag-F_{i_i}(x_i)),e_\\ell\\rangle]|\\\\\r\n  & = \\sum_{i=k}^{\\ell-1}\\eta_i|\\E[\\langle y_{i_i}^\\dag-F_{i_i}(x_i),F_{i_i}'(x_i)(x^*-x_i+x_i-x_\\ell)\\rangle]|.\r\n\\end{align*}\r\nThen by the Cauchy-Schwarz inequality and triangle inequality, we obtain\r\n\\begin{align*}\r\n |\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle]|\r\n  & \\leq \\sum_{i=k}^{\\ell-1}\\eta_i|\\E[\\langle y^\\dag-F(x_i),F'(x_i)(x^*-x_i+x_i-x_\\ell)\\rangle]|\\\\\r\n  & \\leq \\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F'(x_i)(x^*-x_i+x_i-x_\\ell)\\|^2]^\\frac{1}{2}\\\\\r\n  & \\leq \\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F'(x_i)(x^*-x_i)\\|^2]^\\frac{1}{2}\\\\\r\n   &\\quad +\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F'(x_i)(x_i-x_\\ell)\\|^2]^\\frac{1}{2}\r\n   :={\\rm I} + {\\rm II}.\r\n\\end{align*}\r\nBy Assumption \\ref{ass:sol}(ii) and Lemma \\ref{lem:linear}(i), we have\r\n\\begin{equation*}\r\n\\|F'(x)(x-\\tilde x)\\|\\leq (1+\\eta)\\|F(x)-F(\\tilde x)\\|.\r\n\\end{equation*}\r\nSubstituting this inequality into the term $\\rm I$ leads to\r\n\\begin{align*}\r\n  {\\rm I} & \\leq (1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F(x^*)-F(x_i)\\|^2]^\\frac{1}{2}\r\n           = (1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2].\r\n\\end{align*}\r\nLikewise, by the triangle inequality and the choice of the index $\\ell$ in \\eqref{eqn:minimal-res},\r\n\\begin{align*}\r\n  {\\rm II} & \\leq (1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F(x_\\ell)-F(x_i)\\|^2]^\\frac{1}{2}\\\\\r\n          & \\leq (1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2] + (1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F(x_\\ell)-y^\\dag\\|^2]^\\frac{1}{2}\\\\\r\n          & \\leq 2(1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2].\r\n\\end{align*}\r\nThe last two estimates together imply\r\n\\begin{align*}\r\n  |\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle]| & \\leq 3(1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2].\r\n\\end{align*}\r\nSimilarly, one can deduce\r\n\\begin{equation*}\r\n  |\\E[\\langle e_j-e_\\ell,e_\\ell\\rangle]| \\leq 3(1+\\eta)\\sum_{i=\\ell}^{j-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2].\r\n\\end{equation*}\r\nThese two estimates and Corollary \\ref{prop:mono-exact} imply that the right hand sides of \\eqref{eqn:err-exact}\r\ntend to zero as $k\\to\\infty$. Thus, the sequence $\\{e_k\\}_{k\\geq 1}$ and also $\\{x_k\\}_{k\\geq1}$ are Cauchy sequences.\r\n\\end{proof}","sha256_utf8":"cccc9c2cf9424f7e92b36c82760bcdd7656e004128b41e56b04fb83a62897188"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"误差单调与最小残差指标","source_units":[{"start":32484,"end":33203,"start_line":470,"end_line":478,"raw_tex":"The argument below follows closely \\cite[Theorem 2.3]{HankeNeubauerScherzer:1995}, which can be\r\ntraced back to \\cite{McCormickRodrigue:1975}. Let $x^*$ be any solution to problem \\eqref{eqn:nonlin}, and let $e_k:=x_k-x^*$. By\r\nCorollary \\ref{prop:mono-exact}, $\\E[\\|e_k\\|^2]$ is monotonically decreasing to some $\\epsilon\\geq 0$.\r\nNext we show that the sequence $\\{x_k\\}_{k\\ge 1}$ is actually a Cauchy sequence. First we note that\r\n$\\E[\\langle\\cdot,\\cdot\\rangle]$ defines an inner product. For any $j\\geq k$, choose an index $\\ell$\r\nwith $j\\geq \\ell\\geq k$ such that\r\n\\begin{equation}\\label{eqn:minimal-res}\r\n  \\E[\\|y^\\dag - F(x_\\ell)\\|^2]\\leq \\E[\\|y^\\dag - F(x_i)\\|^2], \\quad \\forall k\\leq i \\leq j.\r\n\\end{equation}\r\n","sha256_utf8":"9b8c2ad7f8b52d33243c42559b6ee5d1dbbbc739227def1ca89b7da41d374f2d"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"将柯西性质化为内积估计","source_units":[{"start":33203,"end":34029,"start_line":479,"end_line":492,"raw_tex":"In light of the triangle inequality\r\n\\begin{equation*}\r\n  \\E[\\|e_j-e_k\\|^2]^\\frac{1}{2} \\leq \\E[\\|e_j-e_\\ell\\|^2]^\\frac{1}{2} + \\E[\\|e_\\ell-e_k\\|^2]^\\frac{1}{2},\r\n\\end{equation*}\r\nand the trivial identities\r\n\\begin{equation}\\label{eqn:err-exact}\r\n\\begin{aligned}\r\n  \\E[\\|e_j-e_\\ell\\|^2] & = 2\\E[\\langle e_\\ell-e_j,e_\\ell\\rangle] + \\E[\\|e_j\\|^2] - \\E[\\|e_\\ell\\|^2],\\\\\r\n  \\E[\\|e_\\ell-e_k\\|^2] & = 2\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle] + \\E[\\|e_k\\|^2] - \\E[\\|e_\\ell\\|^2],\r\n\\end{aligned}\r\n\\end{equation}\r\nit suffices to prove that both $\\E[\\|e_j-e_\\ell\\|^2]$ and $\\E[\\|e_\\ell-e_k\\|^2]$ on the left hand side tend to zero as $k\\to \\infty$.\r\nFor $k\\to \\infty$, the last two terms on each of the right hand side of \\eqref{eqn:err-exact} tends to $\\epsilon-\\epsilon=0$, by\r\nthe monotone convergence of $\\E[\\|e_k\\|^2]$ to $\\epsilon$.\r\n","sha256_utf8":"e3696e3ee46eafc6d2c31e24b7bc4330638602209e142b250c92d37e9a799ed2"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"望远镜展开迭代差","source_units":[{"start":34029,"end":34854,"start_line":493,"end_line":503,"raw_tex":"Next we show that the term $\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle]$ also tends to zero as $k\\to\\infty$.\r\nActually, by the definition of the SGD iterate $x_k$ in \\eqref{eqn:sgd}, we have\r\n\\begin{equation*}\r\n  e_\\ell - e_k = \\sum_{i=k}^{\\ell-1}(e_{i+1}-e_i) = {\\sum_{i=k}^{\\ell-1}\\eta_iF_{i_i}'(x_i)^*(y_{i_i}^\\dag-F_{i_i}(x_i))}.\r\n\\end{equation*}\r\nBy the triangle inequality, we can bound $\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle]$ by\r\n\\begin{align*}\r\n  |\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle]| & = |\\E[\\sum_{i=k}^{\\ell-1}\\langle \\eta_iF_{i_i}'(x_i)^*(y_{i_i}^\\dag-F_{i_i}(x_i)),e_\\ell\\rangle]|\\\\\r\n  & \\leq \\sum_{i=k}^{\\ell-1}\\eta_i|\\E[\\langle F_{i_i}'(x_i)^*(y_{i_i}^\\dag-F_{i_i}(x_i)),e_\\ell\\rangle]|\\\\\r\n  & = \\sum_{i=k}^{\\ell-1}\\eta_i|\\E[\\langle y_{i_i}^\\dag-F_{i_i}(x_i),F_{i_i}'(x_i)(x^*-x_i+x_i-x_\\ell)\\rangle]|.\r\n\\end{align*}\r\n","sha256_utf8":"f5c6006a82099d289c795aa551ece8c92a89f18b0e11d6ba24dd1c66aae6d572"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"拆分两项内积上界","source_units":[{"start":34854,"end":35472,"start_line":504,"end_line":512,"raw_tex":"Then by the Cauchy-Schwarz inequality and triangle inequality, we obtain\r\n\\begin{align*}\r\n |\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle]|\r\n  & \\leq \\sum_{i=k}^{\\ell-1}\\eta_i|\\E[\\langle y^\\dag-F(x_i),F'(x_i)(x^*-x_i+x_i-x_\\ell)\\rangle]|\\\\\r\n  & \\leq \\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F'(x_i)(x^*-x_i+x_i-x_\\ell)\\|^2]^\\frac{1}{2}\\\\\r\n  & \\leq \\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F'(x_i)(x^*-x_i)\\|^2]^\\frac{1}{2}\\\\\r\n   &\\quad +\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F'(x_i)(x_i-x_\\ell)\\|^2]^\\frac{1}{2}\r\n   :={\\rm I} + {\\rm II}.\r\n\\end{align*}\r\n","sha256_utf8":"e2b04b607594e87b010afe472a8012dbc8ddf167153cd8b4bc6e589fe8730c78"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"控制第一项残差和","source_units":[{"start":35472,"end":35922,"start_line":513,"end_line":521,"raw_tex":"By Assumption \\ref{ass:sol}(ii) and Lemma \\ref{lem:linear}(i), we have\r\n\\begin{equation*}\r\n\\|F'(x)(x-\\tilde x)\\|\\leq (1+\\eta)\\|F(x)-F(\\tilde x)\\|.\r\n\\end{equation*}\r\nSubstituting this inequality into the term $\\rm I$ leads to\r\n\\begin{align*}\r\n  {\\rm I} & \\leq (1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F(x^*)-F(x_i)\\|^2]^\\frac{1}{2}\r\n           = (1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2].\r\n\\end{align*}\r\n","sha256_utf8":"292e20cfbf8f41f5a980f24846a074b0a48d2492de7338d118bc3b99f033c385"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"利用最小残差控制第二项","source_units":[{"start":35922,"end":36444,"start_line":522,"end_line":527,"raw_tex":"Likewise, by the triangle inequality and the choice of the index $\\ell$ in \\eqref{eqn:minimal-res},\r\n\\begin{align*}\r\n  {\\rm II} & \\leq (1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F(x_\\ell)-F(x_i)\\|^2]^\\frac{1}{2}\\\\\r\n          & \\leq (1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2] + (1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2]^\\frac{1}{2}\\E[\\|F(x_\\ell)-y^\\dag\\|^2]^\\frac{1}{2}\\\\\r\n          & \\leq 2(1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2].\r\n\\end{align*}\r\n","sha256_utf8":"39dac70522d9456395e6bdb1ba7af21a3b4fea717bba434281c89ed24d4f065b"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"残差尾和消失得到柯西性","source_units":[{"start":36444,"end":37024,"start_line":528,"end_line":537,"raw_tex":"The last two estimates together imply\r\n\\begin{align*}\r\n  |\\E[\\langle e_\\ell-e_k,e_\\ell\\rangle]| & \\leq 3(1+\\eta)\\sum_{i=k}^{\\ell-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2].\r\n\\end{align*}\r\nSimilarly, one can deduce\r\n\\begin{equation*}\r\n  |\\E[\\langle e_j-e_\\ell,e_\\ell\\rangle]| \\leq 3(1+\\eta)\\sum_{i=\\ell}^{j-1}\\eta_i\\E[\\|y^\\dag-F(x_i)\\|^2].\r\n\\end{equation*}\r\nThese two estimates and Corollary \\ref{prop:mono-exact} imply that the right hand sides of \\eqref{eqn:err-exact}\r\ntend to zero as $k\\to\\infty$. Thus, the sequence $\\{e_k\\}_{k\\geq 1}$ and also $\\{x_k\\}_{k\\geq1}$ are Cauchy sequences.\r\n","sha256_utf8":"0a262d22659a5d4ed34fb280307311c5db7d68325f28b1c785380b4481bf3995"}]},{"id":"S039","semantic_type":"lemma","kind":"lemma","title":"精确数据残差趋零","source_units":[{"start":37039,"end":37293,"start_line":540,"end_line":545,"raw_tex":"\\begin{lemma}\\label{lemma:res}\r\nLet Assumptions \\ref{ass:sol}(i)-(ii) and \\ref{ass:stepsize}(i) be fulfilled. Then for the exact data $y^\\dag$, there holds\r\n\\begin{equation*}\r\n  \\lim_{k\\to\\infty} \\E[\\|F(x_k)-y^\\dag\\|^2] = 0.\r\n\\end{equation*}\r\n\\end{lemma}","sha256_utf8":"4ab800fe4caf1ea2866a57eb732339baf84ae06723ddb2ef93ed3af2c6918815"}]},{"id":"S040","semantic_type":"proof","kind":"proof","title":"残差趋零完整证明","source_units":[{"start":37295,"end":38429,"start_line":546,"end_line":564,"raw_tex":"\\begin{proof}\r\nBy Lemma \\ref{lem:Cauchy}, $\\{x_k\\}_{k\\geq 1}$ is a Cauchy sequence. By Assumption \\ref{ass:stepsize}(i),\r\n$\\sup_x\\|F'(x)\\|\\leq \\eta_0^{-\\frac12}$. Further, for any $x,\\tilde x\\in X$, there holds\r\n\\begin{equation*}\r\n  \\|F(x)-F(\\tilde x)\\| \\leq (1-\\eta)^{-1}\\|F'(x)(x-\\tilde x)\\|\\leq (1-\\eta)^{-1}\\eta_0^{-\\frac12}\\|x-\\tilde x\\|.\r\n\\end{equation*}\r\nThus, $\\{F(x_k)-y^\\dag\\}_{k\\geq 1}$ is a Cauchy sequence, and $\\E[\\|F(x_k)-y^\\dag\\|^2]$\r\nconverges. Now we proceed by contradiction, and assume that $\\lim_{k\\to\\infty}\\E[\\|F(x_k)-y^\\dag\\|^2]>0$.\r\nThen there exist some $\\epsilon>0$ and $k^*\\in \\mathbb{N}$, such that $\\E[\\|F(x_k)-y^\\dag\\|^2]\\geq \\epsilon$\r\nfor all $k\\geq k^*$. Hence, by Assumption \\ref{ass:stepsize}(i),\r\n\\begin{equation*}\r\n  \\sum_{k=1}^\\infty \\eta_k\\E[\\|F(x_k)-y^\\dag\\|^2] \\geq \\sum_{k=k^*}^\\infty \\eta_k\\E[\\|F(x_k)-y^\\dag\\|^2] \\geq \\epsilon\\sum_{k=k^*}^\\infty\\eta_k =\\infty,\r\n\\end{equation*}\r\nwhich contradicts the inequality\r\n\\begin{equation*}\r\n\\sum_{k=1}^\\infty \\eta_k\\E[\\|F(x_k)-y^\\dag\\|^2] <\\infty,\r\n\\end{equation*}\r\nfrom Corollary \\ref{cor:mono}. This completes the proof of the lemma.\r\n\\end{proof}","sha256_utf8":"9e98e3cd3fd241548e16b5c8dd28662250107aeb813477522d6898564f6a9722"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"残差序列具有极限","source_units":[{"start":37310,"end":37746,"start_line":547,"end_line":552,"raw_tex":"By Lemma \\ref{lem:Cauchy}, $\\{x_k\\}_{k\\geq 1}$ is a Cauchy sequence. By Assumption \\ref{ass:stepsize}(i),\r\n$\\sup_x\\|F'(x)\\|\\leq \\eta_0^{-\\frac12}$. Further, for any $x,\\tilde x\\in X$, there holds\r\n\\begin{equation*}\r\n  \\|F(x)-F(\\tilde x)\\| \\leq (1-\\eta)^{-1}\\|F'(x)(x-\\tilde x)\\|\\leq (1-\\eta)^{-1}\\eta_0^{-\\frac12}\\|x-\\tilde x\\|.\r\n\\end{equation*}\r\nThus, $\\{F(x_k)-y^\\dag\\}_{k\\geq 1}$ is a Cauchy sequence, and $\\E[\\|F(x_k)-y^\\dag\\|^2]$\r\n","sha256_utf8":"e864b917f2bb8a1ff9ef8c12cdbf270fca1934d9bac04893e65c9ddb42ab11ed"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"正极限与残差可和矛盾","source_units":[{"start":37746,"end":38418,"start_line":553,"end_line":563,"raw_tex":"converges. Now we proceed by contradiction, and assume that $\\lim_{k\\to\\infty}\\E[\\|F(x_k)-y^\\dag\\|^2]>0$.\r\nThen there exist some $\\epsilon>0$ and $k^*\\in \\mathbb{N}$, such that $\\E[\\|F(x_k)-y^\\dag\\|^2]\\geq \\epsilon$\r\nfor all $k\\geq k^*$. Hence, by Assumption \\ref{ass:stepsize}(i),\r\n\\begin{equation*}\r\n  \\sum_{k=1}^\\infty \\eta_k\\E[\\|F(x_k)-y^\\dag\\|^2] \\geq \\sum_{k=k^*}^\\infty \\eta_k\\E[\\|F(x_k)-y^\\dag\\|^2] \\geq \\epsilon\\sum_{k=k^*}^\\infty\\eta_k =\\infty,\r\n\\end{equation*}\r\nwhich contradicts the inequality\r\n\\begin{equation*}\r\n\\sum_{k=1}^\\infty \\eta_k\\E[\\|F(x_k)-y^\\dag\\|^2] <\\infty,\r\n\\end{equation*}\r\nfrom Corollary \\ref{cor:mono}. This completes the proof of the lemma.\r\n","sha256_utf8":"be69b9fd18e93de4876c1ba0de1d93032413204651fb3d19651e0959ac1daf58"}]},{"id":"S043","semantic_type":"theorem","kind":"theorem","title":"精确数据随机迭代收敛","source_units":[{"start":38628,"end":39174,"start_line":569,"end_line":580,"raw_tex":"\\begin{theorem}[Convergence for exact data]\\label{thm:conv-exact}\r\nLet Assumptions \\ref{ass:sol}(i)-(ii) and \\ref{ass:stepsize}(i) be fulfilled. Then for the exact data $y^\\dag$, the sequence\r\n$\\{x_k\\}_{k\\geq1}$ by Algorithm \\ref{alg:sgd} converges to a solution $x^*$ of problem \\eqref{eqn:nonlin}:\r\n\\begin{equation*}\r\n  \\lim_{k\\to\\infty}\\E[\\|x_k-x^*\\|^2] = 0.\r\n\\end{equation*}\r\n Further, if\r\n$\\mathcal{N}(F'(x^\\dag))\\subset\\mathcal{N}(F'(x))$, then\r\n\\begin{equation*}\r\n  \\lim_{k\\to\\infty}\\E[\\|x_k -x^\\dag\\|^2]=0.\r\n\\end{equation*}\r\n\\end{theorem}","sha256_utf8":"2c5d2c98226cdf3ed3c53baa333e386e2d70c7dbd9a9703daccef14105d1582d"}]},{"id":"S044","semantic_type":"proof","kind":"proof","title":"精确收敛完整证明","source_units":[{"start":39176,"end":40120,"start_line":581,"end_line":599,"raw_tex":"\\begin{proof}\r\nSince $\\{x_k\\}_{k\\geq1}$ is a Cauchy sequence, there exists a limit of $\\{x_k\\}_{k\\geq 1}$, denoted by $x^*$. Further,\r\n$x^*$ is a solution, since the residual $\\E[\\|y^\\dag-F(x_k)\\|^2]$ converges to zero as $k\\to\\infty$, in view of Lemma \\ref{lemma:res}.\r\n\r\nNote that problem \\eqref{eqn:nonlin} has a unique solution of minimal distance to the initial guess $x_1$ that satisfies\r\n\\begin{equation*}\r\n  x^\\dag - x_1\\in\\mathcal{N}(F'(x^\\dag))^\\perp;\r\n\\end{equation*}\r\nsee Lemma \\ref{lem:linear}. If {$\\mathcal{N}(F'(x^\\dag))\\subset \\mathcal{N}(F'(x_k))$}\r\n for all $k=1,2,\\ldots$, then clearly,\r\n\\begin{equation*}\r\n  x_k-x_1\\in\\mathcal{N}(F'(x^\\dag))^\\perp,\\quad k =1,2,\\ldots.\r\n\\end{equation*}\r\nConsequently,\r\n\\begin{equation*}\r\n  x^\\dag - x^* = x^\\dag - x_1 + x_1-x^*\\in\\mathcal{N}(F'(x^\\dag))^\\perp.\r\n\\end{equation*}\r\nThis and the inequalities from Lemma \\ref{lem:linear}(i) imply $x^*=x^\\dag$, completing the proof.\r\n\\end{proof}","sha256_utf8":"bef39d889b98bf762b1b3d3a983110d73d4c5ffde4f64f132181770bb430bcae"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"柯西极限满足原方程","source_units":[{"start":39191,"end":39447,"start_line":582,"end_line":583,"raw_tex":"Since $\\{x_k\\}_{k\\geq1}$ is a Cauchy sequence, there exists a limit of $\\{x_k\\}_{k\\geq 1}$, denoted by $x^*$. Further,\r\n$x^*$ is a solution, since the residual $\\E[\\|y^\\dag-F(x_k)\\|^2]$ converges to zero as $k\\to\\infty$, in view of Lemma \\ref{lemma:res}.\r\n","sha256_utf8":"48c97cf551a62a18b2b5f4a8bbc26065b9c67a587321e8b9b9e2bc61b57e15eb"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"参考解的正交刻画","source_units":[{"start":39449,"end":39656,"start_line":585,"end_line":588,"raw_tex":"Note that problem \\eqref{eqn:nonlin} has a unique solution of minimal distance to the initial guess $x_1$ that satisfies\r\n\\begin{equation*}\r\n  x^\\dag - x_1\\in\\mathcal{N}(F'(x^\\dag))^\\perp;\r\n\\end{equation*}\r\n","sha256_utf8":"0b15f2a8a29996eb052c561d39a6fc472201dceafc63573a9d152c8ee89e3cb2"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"迭代保持核正交性","source_units":[{"start":39656,"end":39884,"start_line":589,"end_line":593,"raw_tex":"see Lemma \\ref{lem:linear}. If {$\\mathcal{N}(F'(x^\\dag))\\subset \\mathcal{N}(F'(x_k))$}\r\n for all $k=1,2,\\ldots$, then clearly,\r\n\\begin{equation*}\r\n  x_k-x_1\\in\\mathcal{N}(F'(x^\\dag))^\\perp,\\quad k =1,2,\\ldots.\r\n\\end{equation*}\r\n","sha256_utf8":"cf3a7ec616cf3b5b64ec739e18596a39e59dad65bb9f3428bd2c89deb68dd741"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"正交性识别最小距离极限","source_units":[{"start":39884,"end":40109,"start_line":594,"end_line":598,"raw_tex":"Consequently,\r\n\\begin{equation*}\r\n  x^\\dag - x^* = x^\\dag - x_1 + x_1-x^*\\in\\mathcal{N}(F'(x^\\dag))^\\perp.\r\n\\end{equation*}\r\nThis and the inequalities from Lemma \\ref{lem:linear}(i) imply $x^*=x^\\dag$, completing the proof.\r\n","sha256_utf8":"4d065d5a48cd30b9b97e880041431c4015c590f189f7538a78f731675039ce65"}]},{"id":"S049","semantic_type":"lemma","kind":"lemma","title":"固定路径的噪声稳定性","source_units":[{"start":41126,"end":41522,"start_line":613,"end_line":620,"raw_tex":"\\begin{lemma}\\label{lem:conti-noise}\r\nLet Assumption \\ref{ass:sol}(i) be fulfilled. For any fixed $k\\in \\mathbb{N}$ and any path\r\n$(i_1,\\ldots,i_{k-1})\\in\\mathcal{F}_k$, let $x_k$ and $x_k^\\delta$ be the SGD iterates along the path for exact\r\ndata $y^\\dag$ and noisy data $y^\\delta$, respectively. Then\r\n\\begin{equation*}\r\n  \\lim_{\\delta\\to 0^+}\\|x_k^\\delta-x_k\\|=0.\r\n\\end{equation*}\r\n\\end{lemma}","sha256_utf8":"dd10595c6545b6ec43d9abdfeb2f938e0d5d9f8ee5a7c5f2471ddcf9b251a36d"}]},{"id":"S050","semantic_type":"proof","kind":"proof","title":"路径稳定性完整证明","source_units":[{"start":41524,"end":43015,"start_line":621,"end_line":643,"raw_tex":"\\begin{proof}\r\nWe prove the assertion by mathematical induction. The assertion holds\r\ntrivially for $k=1$, since $x_1^\\delta = x_1$. Now suppose that it\r\nholds for all indices up to $k$ and any path $(i_1,\\ldots,i_{k-1})\\in\\mathcal{F}_k$.\r\nNext, by the definitions of the SGD iterates $x_k$ and $x_k^\\delta$, cf. \\eqref{eqn:sgd}:\r\n\\begin{align*}\r\n   x_{k+1}        &= x_k - \\eta_k F_{i_k}'(x_k)^*(F_{i_k}(x_k)-y_{i_k}),\\\\\r\n   x_{k+1}^\\delta &= x_k^\\delta - \\eta_k F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta).\r\n\\end{align*}\r\nTherefore, for any fixed path $(i_1,\\ldots,i_k)$, we have\r\n\\begin{align*}\r\n   x_{k+1}^\\delta -x_{k+1} &= (x_k^\\delta-x_k) - \\eta_k \\left(F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)-F_{i_k}'(x_k)^*(F_{i_k}(x_k)-y_{i_k})\\right)\\\\\r\n     & = (x_k^\\delta -x_k)-\\eta_k \\big((F_{i_k}'(x_k^\\delta)^* - F_{i_k}'(x_k)^*)(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta) \\\\\r\n     & \\quad +F_{i_k}'(x_k)^*((F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)-(F_{i_k}(x_k)-y_{i_k}))\\big).\r\n\\end{align*}\r\nThus, by triangle inequality,\r\n\\begin{align*}\r\n  \\|x_{k+1}^\\delta-x_{k+1}\\| &\\leq \\|x_k^\\delta -x_k\\|+\\eta_k \\|F_{i_k}'(x_k^\\delta)^* - F_{i_k}'(x_k)^*\\|\\|F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\| \\\\\r\n     & \\quad +\\|F_{i_k}'(x_k)^*\\|\\|((F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)-(F_{i_k}(x_k)-y_{i_k}))\\|.\r\n\\end{align*}\r\nThen the desired assertion follows from the continuity of the operators $F_i$ and $F_i'$ in Assumption \\ref{ass:sol}(i)\r\nand the induction hypothesis.\r\n\\end{proof}","sha256_utf8":"12651afa1f1cf6c55c412dbcbe4908984afd766a1115542735bebd154756d8dc"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"归纳起点与迭代公式","source_units":[{"start":41539,"end":42063,"start_line":622,"end_line":629,"raw_tex":"We prove the assertion by mathematical induction. The assertion holds\r\ntrivially for $k=1$, since $x_1^\\delta = x_1$. Now suppose that it\r\nholds for all indices up to $k$ and any path $(i_1,\\ldots,i_{k-1})\\in\\mathcal{F}_k$.\r\nNext, by the definitions of the SGD iterates $x_k$ and $x_k^\\delta$, cf. \\eqref{eqn:sgd}:\r\n\\begin{align*}\r\n   x_{k+1}        &= x_k - \\eta_k F_{i_k}'(x_k)^*(F_{i_k}(x_k)-y_{i_k}),\\\\\r\n   x_{k+1}^\\delta &= x_k^\\delta - \\eta_k F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta).\r\n\\end{align*}\r\n","sha256_utf8":"0c8c41d60d584926db1494a3abddc885160425ee6fb3ba6441b26b4bddcf449f"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"展开带噪与精确迭代差","source_units":[{"start":42063,"end":42541,"start_line":630,"end_line":635,"raw_tex":"Therefore, for any fixed path $(i_1,\\ldots,i_k)$, we have\r\n\\begin{align*}\r\n   x_{k+1}^\\delta -x_{k+1} &= (x_k^\\delta-x_k) - \\eta_k \\left(F_{i_k}'(x_k^\\delta)^*(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)-F_{i_k}'(x_k)^*(F_{i_k}(x_k)-y_{i_k})\\right)\\\\\r\n     & = (x_k^\\delta -x_k)-\\eta_k \\big((F_{i_k}'(x_k^\\delta)^* - F_{i_k}'(x_k)^*)(F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta) \\\\\r\n     & \\quad +F_{i_k}'(x_k)^*((F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)-(F_{i_k}(x_k)-y_{i_k}))\\big).\r\n\\end{align*}\r\n","sha256_utf8":"966e1ae883d14edcc0f04472e89fd9364995bc278a04af37042962b236a820d3"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"连续性完成路径归纳","source_units":[{"start":42541,"end":43004,"start_line":636,"end_line":642,"raw_tex":"Thus, by triangle inequality,\r\n\\begin{align*}\r\n  \\|x_{k+1}^\\delta-x_{k+1}\\| &\\leq \\|x_k^\\delta -x_k\\|+\\eta_k \\|F_{i_k}'(x_k^\\delta)^* - F_{i_k}'(x_k)^*\\|\\|F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta\\| \\\\\r\n     & \\quad +\\|F_{i_k}'(x_k)^*\\|\\|((F_{i_k}(x_k^\\delta)-y_{i_k}^\\delta)-(F_{i_k}(x_k)-y_{i_k}))\\|.\r\n\\end{align*}\r\nThen the desired assertion follows from the continuity of the operators $F_i$ and $F_i'$ in Assumption \\ref{ass:sol}(i)\r\nand the induction hypothesis.\r\n","sha256_utf8":"d6d97baebc09620ba5fda7dece5ab543507a793b4ce9f20593713c1192dd16b1"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S010","edges":[{"source_id":"S001","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S025","kind":"supports"},{"source_id":"S002","target_id":"S053","kind":"supports"},{"source_id":"S003","target_id":"S023","kind":"supports"},{"source_id":"S003","target_id":"S034","kind":"supports"},{"source_id":"S003","target_id":"S047","kind":"supports"},{"source_id":"S003","target_id":"S051","kind":"supports"},{"source_id":"S004","target_id":"S025","kind":"supports"},{"source_id":"S005","target_id":"S025","kind":"supports"},{"source_id":"S005","target_id":"S035","kind":"supports"},{"source_id":"S006","target_id":"S046","kind":"supports"},{"source_id":"S007","target_id":"S018","kind":"supports"},{"source_id":"S007","target_id":"S025","kind":"supports"},{"source_id":"S007","target_id":"S041","kind":"supports"},{"source_id":"S007","target_id":"S045","kind":"supports"},{"source_id":"S007","target_id":"S053","kind":"supports"},{"source_id":"S008","target_id":"S024","kind":"supports"},{"source_id":"S008","target_id":"S041","kind":"supports"},{"source_id":"S008","target_id":"S042","kind":"supports"},{"source_id":"S009","target_id":"S025","kind":"supports"},{"source_id":"S009","target_id":"S033","kind":"supports"},{"source_id":"S011","target_id":"S014","kind":"supports"},{"source_id":"S011","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S047","kind":"supports"},{"source_id":"S014","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S015","target_id":"S013","kind":"supports"},{"source_id":"S016","target_id":"S013","kind":"supports"},{"source_id":"S017","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S010","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S020","target_id":"S015","kind":"supports"},{"source_id":"S020","target_id":"S016","kind":"supports"},{"source_id":"S020","target_id":"S017","kind":"supports"},{"source_id":"S020","target_id":"S023","kind":"supports"},{"source_id":"S020","target_id":"S025","kind":"supports"},{"source_id":"S020","target_id":"S026","kind":"supports"},{"source_id":"S020","target_id":"S033","kind":"supports"},{"source_id":"S020","target_id":"S035","kind":"supports"},{"source_id":"S020","target_id":"S045","kind":"supports"},{"source_id":"S020","target_id":"S046","kind":"supports"},{"source_id":"S020","target_id":"S053","kind":"supports"},{"source_id":"S021","target_id":"S015","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S022","kind":"supports"},{"source_id":"S026","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S038","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S018","target_id":"S036","kind":"supports"},{"source_id":"S018","target_id":"S037","kind":"supports"},{"source_id":"S032","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S027","target_id":"S038","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S034","target_id":"S031","kind":"supports"},{"source_id":"S035","target_id":"S031","kind":"supports"},{"source_id":"S036","target_id":"S031","kind":"supports"},{"source_id":"S037","target_id":"S031","kind":"supports"},{"source_id":"S038","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S041","kind":"supports"},{"source_id":"S018","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S027","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S042","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S030","target_id":"S045","kind":"supports"},{"source_id":"S039","target_id":"S045","kind":"supports"},{"source_id":"S018","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S045","target_id":"S048","kind":"supports"},{"source_id":"S018","target_id":"S048","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S017","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S017","kind":"supports"}]}}
{"record_id":"1907.10300v2","paper":{"title":"Sparse Optimization on Measures with Over-parameterized Gradient Descent","authors":["Lenaic Chizat"],"arxiv_id":"1907.10300","version":"v2","journal":"Mathematical Programming","doi":"10.1007/s10107-021-01636-z","abstract_url":"https://arxiv.org/abs/1907.10300v2","pdf_url":"https://arxiv.org/pdf/1907.10300v2"},"context":{"raw_tex":"Finding parsimonious descriptions of complex observations is an important problem in machine learning and signal processing.  In its simplest form, this task boils down to searching for an element in a Hilbert space $\\Ff$ that is close to a certain $f_0\\in \\Ff$ --- the observations --- and that is a linear combination of a few elements from a parameterized set $\\{\\phi(\\theta)\\}_{\\theta \\in \\Theta}\\subset \\Ff$ --- the parsimonious description. This can be formulated as a minimization problem where the linear combination is expressed through an unknown measure $\\nu$ and the distance to $f_0$ is quantified using a smooth convex loss function $R:\\Ff\\to \\RR$, such as the square loss $R(f)=\\frac12 \\Vert f-f_0\\Vert^2_\\Ff$.  The problem to solve is then\n%\n\\begin{equation}\\label{eq:objective}%\\tag{$\\Pp_\\lambda$}\nJ^*\\coloneqq \\min_{\\nu \\in \\Mm_+(\\Theta)} J(\\nu), \\qquad \\text{}\\qquad J(\\nu) :=R\\left( \\int_\\Theta \\phi(\\theta) \\d \\nu(\\theta)\\right) + \\lambda  \\nu (\\Theta)\n\\end{equation}\n\nwhere $\\Mm_+(\\Theta)$ is the set of nonnegative measures $\\nu$ on the parameter space $\\Theta$ with finite total mass $\\nu(\\Theta)<\\infty$ and $\\lambda> 0$ is the regularization strength. This formulation also covers minimization over signed measures with total variation regularization, by replacing $\\Theta$ with the disjoint union of two copies of $\\Theta$ where $\\phi$ takes opposite values, see Appendix~\\ref{app:signed}. A large body of research has exhibited the favorable properties of minimizers of such problems~\\cite{bach2017breaking, de2012exact, mairal2014sparse} with a statistical or variational viewpoint, showing in particular that $\\lambda$ favors sparser solutions and increases stability as it gets larger, at the expense of introducing a stronger bias. The present paper deals with the optimization aspect: our goal is to design algorithms that return $\\epsilon$-accurate solutions with a guaranteed computational complexity. When the set $\\Theta$ is a finite set, this is a finite dimensional convex optimization problem that is well understood~\\cite{boyd2004convex, bach2012optimization}. However, convex approaches are generally inefficient when $\\Theta$ is a continuous space, such as a $d$-dimensional manifold, where the need to discretize the space leads to a complexity scaling as $\\epsilon^{-d}$ in the accuracy $\\epsilon$. We consider the following setting:\n\\begin{enumerate}\n\\item[] {\\sf (A1)}  $\\Theta$ is a compact $d$-dimensional Riemannian manifold without boundaries. The functions $\\phi:\\Theta\\to \\Ff$ and $R:\\Ff\\to \\RR_+$ are twice Fr\\'echet differentiable, with locally Lipschitz second-order derivatives, and $\\nabla R$ is bounded on sublevel sets.\n\\end{enumerate}\n\n\nThe set of signed (resp.~nonnegative) finite Borel measures on a metric space $(\\Xx,\\dist)$ is denoted by $\\Mm(\\Xx)$ (resp. $\\Mm_+(\\Xx)$). The relative entropy, a.k.a.~Kullback-Leibler divergence, is defined for $\\nu_1,\\nu_2 \\in \\Mm_+(\\Xx)$ as\n$\\Hh (\\nu_1,\\nu_2) = \\int_{\\Xx} \\log(\\d\\nu_1/\\d\\nu_2) \\d\\nu_1 - \\nu_1(\\Xx)+\\nu_2(\\Xx)$ if $\\nu_1$ is absolutely continuous w.r.t.~$\\nu_2$, and $+\\infty$ otherwise. The $p$-Wasserstein distance on the set $\\Pp_p(\\Xx)$ of probability measures with finite $p$-th moment is defined, for $\\mu_1,\\mu_2 \\in \\Pp_p(\\Xx)$ as \n\\[\nW_p(\\mu_1,\\mu_2) = \\left(\\min_{\\gamma \\in \\Pi(\\mu_1,\\mu_2)} \\int \\dist(x_1,x_2)^p\\d\\gamma(x_1,x_2)\\right)^{1/p}\n\\]\nwhere $\\Pi(\\mu_1,\\mu_2)$ is the set of measures on $\\Xx \\times \\Xx$ with marginals $\\mu_1$ and $\\mu_2$. The distance $W_\\infty$ between compactly supported probabilities is defined as the limit of $W_p$ as $p\\to \\infty$ and can be directly defined as $W_\\infty(\\mu_1,\\mu_2) = \\inf_{\\gamma \\in \\Pi(\\mu_1,\\mu_2)} \\max_{(x_1,x_2)\\in \\spt \\gamma} \\dist(x_1,x_2)$~\\cite{santambrogio2015optimal}. We also define the Bounded-Lipschitz norm for a continuous function $\\psi:\\Xx\\to \\RR$ as $\\Vert \\psi \\Vert_{\\BL} = \\Vert \\psi\\Vert_\\infty+\\Lip(\\psi)$ where $\\Lip(\\psi)$ is the Lipschitz constant of $\\psi$ and its dual norm on $\\Mm(\\Xx)$ as \n$\\Vert \\nu \\Vert^*_{\\BL} \\coloneqq \\sup_{\\Vert \\varphi \\Vert_{\\BL}\\leq 1} \\int \\varphi \\, \\d\\nu$. For a Riemannian manifold $\\Theta$, we denote by $T_\\theta\\Theta$ the tangent space of $\\Theta$ at $\\theta$ and by $\\langle \\cdot,\\cdot\\rangle_\\theta : (T_\\theta \\Theta)^2 \\to \\RR_+$ the metric at $\\theta$.\n\n\nConsider a smooth increasing bijection $h:\\RR_+\\to \\RR_+$ (such as a power function $r\\mapsto r^p$) and a number of particles $m \\in \\NN^*$. The idea behind particle gradient-based algorithms is to parameterize the unknown measure $\\nu$ as $\\frac1m \\sum_{i=1}^m h(r_i) \\delta_{\\theta_i}$ and to perform gradient-based optimization on the corresponding objective\n\\begin{equation}\\label{eq:objectivediscrete}%\\tag{$\\widehat \\Pp^m_\\lambda$}\nF_{m}((r_1,\\theta_1),\\dots,(r_{m},\\theta_m)) \\coloneqq  R\\left(\\frac1{m} \\sum_{i=1}^{m} h(r_i)\\phi(\\theta_i)\\right) + \\frac\\lambda{m} \\sum_{i=1}^{m} h(r_i),\n\\end{equation}\n\nwhere the parameters $(r_i,\\theta_i)$ of each particle belong to $\\Omega \\coloneqq \\RR_+ \\times \\Theta$ endowed with a specific choice of metric. Clearly, if $J$ admits a minimizer that is a mixture of $m^\\star$ atoms with $m^\\star\\leq m$, then it is sufficient to minimize $F_m$ from Eq.~\\ref{eq:objectivediscrete} for solving \\eqref{eq:objective}. While~\\eqref{eq:objectivediscrete} is finite dimensional, it is typically non-convex with possibly some strict local minima. Still, when $R$ is convex and for $h(r)=r^p$ for $p\\in \\{1,2\\}$, the message from~\\cite{chizat2018global}  (see Theorem~\\ref{th:consistency}) is that  solving~\\eqref{eq:objectivediscrete} to global optimality with first-order methods is still possible by using over-parameterization, i.e.\\ choosing $m$ much larger than $m^\\star$.  Such a method involve various key hyper-parameters which role is discussed throughout the paper. They include (i) the choice of the function $h$ (ii) the choice of the metric on $\\Omega^m$ and (iii) the choice of the initialization.\n\n\\paragraph{Expression of the gradient.} Under~{\\sf(A1)}, the objective $J$, seen as a function on the space $\\Mm(\\Theta)$ endowed with the total variation norm, is Fr\\'echet-differentiable. Its differential at $\\nu\\in \\Mm(\\Theta)$ can be represented by the function $J'_\\nu : \\Theta \\to \\RR$ given by\n\\begin{equation}\\label{eq:firstvariation}\nJ'_\\nu(\\theta) = \\left\\langle \\phi(\\theta),\\nabla R\\left(\\int_\\Theta \\phi(\\theta) \\d\\nu(\\theta)\\right)\\right\\rangle_{\\Ff} +\\lambda,\n \\end{equation}\n\n in the sense that for any $\\sigma \\in\\Mm(\\Theta)$, it holds\n$\\frac{d}{d\\epsilon} J(\\nu + \\epsilon \\sigma)\\vert_{\\epsilon=0} = \\int_{\\Theta} J'_\\nu(\\theta) \\d\\sigma(\\theta)$. Now, consider a metric on $(\\Omega^*)^m$ that is the average $(1/m)\\sum_{i=1}^m \\langle \\cdot, \\cdot \\rangle_{(r_i,\\theta_i)}$ of metrics on each factor $\\Omega^*\\coloneqq \\RR_+^*\\times \\Theta$, where $\\RR_+^*$ is the set of positive real numbers, of the form\n\\begin{equation}\\label{eq:metric}\n\\langle(\\delta r_1,\\delta \\theta_1),(\\delta r_2,\\delta \\theta_2) \\rangle_{(r,\\theta)} = \\alpha(r)^{-1} \\delta r_1\\delta r_2 +\\beta(r)^{-1}\\langle \\delta \\theta_1,\\delta\\theta_2\\rangle_{\\theta} \n\\end{equation}\n\nwhere $\\alpha$ and $\\beta$ are smooth functions $\\RR_+^*\\to \\RR_+^*$ to be specified\\footnote{Extension of the metric and gradients to the whole of $\\Omega$ can be made on a case by case basis, see Section~\\ref{sec:conic}.}, $ (r,\\theta) \\in \\Omega^*$, $\\delta r_1, \\delta r_2 \\in \\RR$ and $\\delta \\theta_1, \\delta \\theta_2 \\in T_\\theta\\Theta$. Using the fact that gradients are characterized by the relation $\\d F_m(x)(\\delta x) = \\langle \\nabla F_m(x),\\delta x\\rangle$, we get that the gradient of $F_m$ is given, in components, by \n\n\n\\paragraph{Lifted problem in Wasserstein space.} Assume now that $h$ has at most quadratic growth, and that the metric is defined on the whole of $\\Omega$. One can then see the discrete problem~\\eqref{eq:objectivediscrete} as a discretization of a problem on the space $\\Pp_2(\\Omega)$ of probability measures on $\\Omega$ with finite second moment endowed with the Wasserstein-$2$ metric given by\n\\begin{equation}\\label{eq:liftedproblem}%\\tag{$\\Pp_\\lambda$}\nF^\\star = \\min_{\\mu \\in \\Pp_2(\\Omega)} F(\\mu)\\quad \\text{where}\\quad F(\\mu) \\coloneqq \\left( \\int_\\Omega h(r) \\phi(\\theta) \\d\\mu(r,\\theta) \\right) +  \\lambda \\int_\\Omega h(r) \\d\\mu(r,\\theta).\n\\end{equation}\n\nThis point of view leads to insights on the properties of $F_m$ that are independent of $m$, which is crucial for our theoretical analysis. For a measure $\\mu\\in \\Pp_2(\\Omega)$, we define following~\\cite{liero2018optimal} the \\emph{homogeneous projection} operator $\\h:\\Pp_2(\\Omega)\\to \\Mm_+(\\Theta)$ where  $\\h\\mu$ is characterized by\n\\begin{align*}\n\\int_{\\Theta} \\varphi(\\theta) \\d(\\h\\mu)(\\theta) = \\int_{\\Omega} h(r)\\varphi(\\theta)\\d\\mu(r,\\theta)\n\\end{align*}\n\nfor any continuous function $\\varphi:\\Theta\\to \\RR$. With this operator, we simply have $F(\\mu)=J(\\h\\mu)$. \n\n\n\n\\paragraph{Wasserstein gradient flow.} It is also possible to directly study the optimization dynamics in the space $\\Pp_2(\\Omega)$ for the functional $F$ of Eq.~\\eqref{eq:liftedproblem}. For a measure $\\nu \\in \\Mm_+(\\Theta)$, consider the vector field on $\\Omega$ with expression\n\\begin{equation*}\ng_\\nu(r,\\theta) = \\big(\\alpha(r) h'(r) J'_{\\nu}(\\theta),\\beta(r) h(r)\\nabla J'_{\\nu}(\\theta)\\big) \\in \\RR \\times T_\\theta \\Theta.\n\\end{equation*}\n\nWe refer to $g_{\\h \\mu}$ as the Wasserstein gradient of $F$ at $\\mu$ (this notation emphasizes that it only depends on $\\mu$ through $\\h \\mu$). Gradient flows of $F_m$ are particular cases of \\emph{Wasserstein gradient flows} of $F$. The latter are defined as the absolutely continuous curves $(\\mu_t)_{t\\geq 0}$ in $\\Pp_2(\\Omega)$ that satisfy\n\\begin{equation}\\label{eq:WGF}\n\\partial_t \\mu_t =  \\div (\\mu_t g_{\\h\\mu_t})\n\\end{equation}\n\nin the weak sense, which means that for any differentiable function $\\varphi: \\Omega \\to \\RR$, it holds $\\frac{\\d}{\\d t}\\left(\\int \\varphi \\,\\d\\mu_t \\right) = - \\int \\nabla \\varphi \\cdot g_{\\h\\mu_t}\\d\\mu_t$, for almost every $t\\geq 0$, see~\\cite{santambrogio2015optimal}. This is a proper extension of the notion of gradient flow for $F_m$ in the sense that if $x(t)=(r_i(t),\\theta_i(t))_{i=1}^m$ is a gradient flow of $F_m$ then it can be directly checked that  $t\\mapsto \\mu_t = \\frac1m \\sum_{i=1}^m \\delta_{(r_i(t),\\theta_i(t))}$ is a Wasserstein gradient flow of $F$.\n\n\nAs seen in Eq.~\\eqref{eq:gradientgeneric}, the choice of the homogeneity degree and of the metric on $\\Omega$ determine a specific way to combine the \\emph{vertical}  and the \\emph{spatial} components of the gradient (along the variable $r$ and $\\theta$, respectively). From now on, we focus on what we refer to as the \\emph{conic case}, which corresponds to the following assumption:\n\\begin{itemize}\n\\item [\\sf (A2)] The mass parameterization is $h(r)=r^2$ and the metric on $\\Omega^*$ is of the form Eq.~\\eqref{eq:metric} with $(\\alpha(r),\\beta(r))=(\\alpha,\\beta/r^2)$ for some $\\alpha,\\beta>0$.\n\\end{itemize}\nThe corresponding geodesic distance is $\\dist((r_1,\\theta_2),(r_1,\\theta_2))^2 = r_1^2 +r_2^2 -2r_1r_2\\cos_{\\pi}(\\dist(\\theta_1,\\theta_2))$ where $\\cos_\\pi(z) = \\cos(\\min\\{\\pi,z\\})$. This metric can be extended as a proper metric on $\\widetilde \\Omega$, defined as the set $\\Omega$ where the subset $\\{0\\}\\times \\Theta$ is identified to a single point, known as the \\emph{cone metric}, which is the canonical way to define a metric on $\\tilde \\Omega$~\\cite{burago2001course}. In our context, identifying $\\{0\\}\\times \\Theta$ to a single point is desirable because a particle located in this set is a ``dead'' particle carrying no mass.\n\n\nPlugging the metric into Eq.~\\eqref{eq:gradientgeneric} gives the gradient (extended by continuity to $\\{0\\}\\times \\Theta$)\n\\begin{align*}\n\\left\\{\n\\begin{aligned}\n\\nabla_{r_i} F_m((r_i,\\theta_i)_{i=1}^m) &= 2\\alpha r_i J'_{\\nu}(\\theta_i)\\\\\n\\nabla_{\\theta_i} F_m ((r_i,\\theta_i)_{i=1}^m) &=  \\beta \\nabla J'_{\\nu}(\\theta_i)\n\\end{aligned}\n\\right. && \\text{where} && \\nu = \\frac1m \\sum_{i=1}^m r_i^2\\delta_{\\theta_i},\n\\end{align*}\n\nand the Wasserstein gradient is represented by the vector field\n\\begin{equation}\\label{eq:wassersteingradient}\ng_\\nu(r,\\theta) = \\big(2 \\alpha r J'_{\\nu}(\\theta), \\beta \\nabla J'_{\\nu}(\\theta)\\big)  \\in \\RR \\times T_\\theta \\Theta, \\forall (r,\\theta)\\in \\Omega.\n\\end{equation}\n\nIn order to derive global optimality conditions, we assume the following.\n\\begin{itemize}\n\\item [\\sf(A3)] The loss $R$ is convex.\n\\end{itemize}\n\n\n\\begin{itemize}\n\\item [\\sf(A4)] Problem~\\eqref{eq:objective} admits a unique global minimizer on $\\Mm_+(\\Theta)$ which is of the form $\\nu^\\star = \\sum_{i=1}^{m^\\star} r_i^2\\delta_{\\theta_i}$ with $\\nu^\\star(\\Theta)>0$. We denote $f^\\star \\coloneqq \\int \\phi \\d\\nu^\\star = \\sum_{i=1}^{m^\\star}r_i^2\\phi(\\theta_i)$.\n\\end{itemize}\nWithout loss of generality, we assume $r_i>0$ for all $i$ and $\\theta_i \\neq \\theta_{i'}$ whenever $i\\neq i'$, so that $(r_i,\\theta_i)_{i=1}^{m^\\star}$ is uniquely well-defined, up to re-ordering. Let us fix from now on normal coordinates frames on the neighborhood of each $\\theta_i$. This allows to identify tensors at $\\theta_i$ with their expression in coordinates and also induces a set of coordinates on the direct sum of the tangent spaces $T_{\\theta_i}\\Theta$, which is of dimension $m^\\star\\times d$. \n\n\nWe define the \\emph{global kernel} $K\\in \\RR^{(m^\\star\\times (1+d))^2}$ by\n\\[\nK_{(i,j),(i',j')} \\coloneqq \\langle r_i \\bar \\nabla_j \\phi(\\theta_i), r_{i'} \\bar \\nabla_{j'}\\phi(\\theta_{i'})\\rangle_{d^2\\! R_{f^\\star}}\n\\]\n\nwhere $\\bar\\nabla \\phi \\coloneqq (2 \\alpha \\phi, \\beta \\nabla \\phi)$ can be interpreted as the gradient of $\\h\\phi$ at $(1,\\theta)$. Remark that $K$ is defined via the quadratic form associated to the Hessian of $R$ at $f^\\star$. This interaction kernel $K$ appears naturally in the various statistical and optimization analysis of the minimization problem under consideration~\\cite{duval2015exact,traonmilin2018basins}. We also use the notation for the \\emph{local kernels}  for $i\\in 1,\\dots,m^\\star$\n\\[\nH_i\\coloneqq \\beta^2 \\nabla^2 J'_{\\nu^\\star}(\\theta_i)\n\\]\n\nexpressed in local coordinates.  In order to simplify notations, we concatenate these matrices in a large matrix $H\\in \\RR^{(m^\\star\\times (1+d))^2}$ of the same size as $K$ defined as\n\n\n\\[\n H_{(i,j),(i,j')} =\\begin{cases}\n \\beta^2  \\nabla^2_{j,j'} J'_{\\nu^\\star}(\\theta_i) &\\text{if $i=i'$ and $j,j'\\geq 1$},\\\\\n  0 &\\text{ if $j=0$ or $j'=0$}.\n  \\end{cases}\n \\]\n\nwhere here and in the proofs, we use $0$ to label the $r$'s coordinate. The local analysis will be carried under the following non-degeneracy assumptions.\n\\begin{itemize}\n\\item [\\sf(A5)] The minimizer $\\nu^\\star$ is non-degenerate in the sense that $\\nabla^2 R(f^\\star)$ is positive definite and, calling $\\sigma_{\\min}(A)$ the smallest singular value of a linear operator $A$, we have global curvature  $\\sigma_{\\min}(K)>0$, local curvature  $\\sigma_{\\min}(H) = \\min_{i} \\sigma_{\\min}(H_i)>0$, and strict slackness, i.e.\\ the only points where $J'_{\\nu^\\star}$ vanishes are $(\\theta_i)_{i=1}^{m^\\star}$.\n\\end{itemize}\n\n\nA first consequence of these assumptions is that convergence in value implies convergence to minimizers. The distance on $\\Mm_+(\\Theta)$ that naturally appears in the analysis is the Wasserstein-Fisher-Rao, a.k.a.\\ Hellinger-Kantorovich metric $\\widehat W_2$, which is the extension of the Wasserstein $W_2$ metric to unnormalized measures. It admits many equivalent definitions~\\cite{liero2018optimal, kondratyev2016new, chizat2018unbalanced}, the most suitable to our context being~\\cite[\nThm. 7.20]{liero2018optimal}\n\\[\n\\widehat W_2(\\nu_1,\\nu_2) \\coloneqq \\min \\left \\{W_2(\\mu_1,\\mu_2)\\;;\\; (\\mu_1,\\mu_2)\\in \\Pp_2(\\Omega)^2 \\text{ satisfy } (\\h\\mu_1,\\h\\mu_2)=(\\nu_1,\\nu_2)\\right\\}\n\\]\n\nwhere the Wasserstein distance on $\\Omega$ is defined relative to the cone metric (in this paragraph, with $\\alpha=\\beta=1$). The proof of the following result involves the construction of a transport map in the lifted space $\\Pp_2(\\Omega)$ and is postponed to Appendix~\\ref{app:distanceinequality}.\n\n\n Our first main result is a lower bound on the squared norm of the gradient in terms of the sub-optimality gap, an inequality known as sharpness, or Polyak-\\L{}ojasiewicz inequality~\\cite{polyak1963gradient,karimi2016linear}, which is a special case of \\L{}ojasiewicz gradient inequality. It involves the $L^2(\\nu)$ norm of the gradient, which we denote for $\\nu=\\h\\mu$ by\n\n\n\\[\n \\Vert g_{\\nu}\\Vert^2_{L^2(\\nu)} \\coloneqq \\int_{\\Omega} \\left( \\frac1\\alpha \\vert 2\\alpha rJ'_{\\nu}(\\theta)\\vert^2 + \\frac {r^2}{\\beta} \\Vert \\beta \\nabla J'_{\\nu}\\Vert_\\theta^2\\right)\\d\\mu(r,\\theta) = \\int_\\Theta \\left( 4 \\alpha \\vert J'_\\nu(\\theta)\\vert^2 +\\beta \\Vert \\nabla J'_\\nu(\\theta)\\Vert^2_\\theta \\right)\\d\\nu(\\theta).\n \\]\n\nThe proof of Theorem~\\ref{th:gradientinequality}, in Appendix~\\ref{app:sharpness}, is based on a local expansion of $J(\\nu)$ in terms of some local moments of $\\nu$. For a radius $\\tau>0$ (that shall be fixed at some small enough value in the course of the proof), we define the sets  for $i\\in \\{1,\\dots,m^\\star\\}$,\n\\begin{align*}\n\\Theta_i \\coloneqq  \\{ \\theta \\in \\Theta \\;;\\; \\dist(\\theta,\\theta_i) < \\tau \\}.\n\\end{align*}\nWe assume that $\\tau$ is smaller than $1$ and small enough so that these sets together with $\\Theta_0 \\coloneqq \\Theta \\setminus \\cup_{i=1}^{m^\\star} \\Theta_i$ form a partition of $\\Theta$ and that the exponential map at $\\theta_i$ has injectivity radius larger than $\\tau$, for $i\\in \\{1,\\dots,m^\\star\\}$. We then say that $\\tau$ is an \\emph{admissible} radius. \n\\begin{definition}[Local moments]\\label{def:localfeatures}\nGiven an admissible radius $\\tau>0$ and a measure $\\nu\\in \\Mm_+(\\Theta)$, we define for $i\\in \\{0,\\dots,m^\\star\\}$ the local masses $\\bar r_i^2 = \\nu(\\Theta_i)$ and the local means $\\bar \\theta_i \\coloneqq \\frac1{\\bar r_i^2} \\int_{\\Theta_i} \\theta\\d\\nu(\\theta)$ if $\\nu(\\Theta_i)>0$ and $\\bar \\theta_i = \\theta_i$ otherwise. Finally, we define for $i\\in \\{1,\\dots,m^\\star\\}$ the \\emph{weighted biases}\n\\begin{align*}\nb_i^r \\coloneqq \\frac{\\bar r_i^2 -r_i^2}{2\\alpha r_i},&&\nb_i^\\theta \\coloneqq \\frac{\\bar r_i^2}{\\beta r_i}(\\bar \\theta_i-\\theta_i), &&\nb_i = (b_i^r,b_i^\\theta),\n\\end{align*}\nand the \\emph{weighted covariances} $\\Sigma_i \\coloneqq \\frac1{\\bar r_i^2 \\beta^2} \\int_{\\Theta_i} (\\theta - \\bar \\theta_i)(\\theta - \\bar \\theta_i)^\\intercal \\d\\nu(\\theta)$.\n\\end{definition}\n\nIn order to state the condition on the initialization, we first choose a reference measure $\\rho \\in \\Mm_+(\\Theta)$ with a smooth positive density, also denoted by $\\rho$, which represents our prior knowledge about the solution $\\nu^\\star$. We introduce the quantity (analogous to a log-likelihood)\n\\[\n\\bar\\Hh(\\nu^\\star,\\rho) \\coloneqq \\sum_{i=1}^{m^\\star} r_i^2 \\log\\left(\\frac{r_i^2}{\\rho(\\theta_i)}\\right) - \\nu^\\star(\\Theta)+\\rho(\\Theta).\n\\]\n\n The proof of Theorem~\\ref{thm:convergenceflow} mostly relies on the the following general lemma which applies to any type of initialization or any structure of minimizers. It gives an upper bound on the optimality gap during along gradient flows in terms of a \\emph{mirror rate} function $\\Qq_{\\nu_0,\\nu^\\star}: \\RR^*_+ \\to \\RR_+$ defined for $\\nu^\\star, \\nu_0\\in \\Mm_+(\\Theta)$  and $\\tau>0$ as\n\\begin{align}\\label{eq:quantization}\n\\Qq_{\\nu^\\star,\\nu_0}(\\tau) = \n\\inf_{\\nu \\in \\Mm_+(\\Theta)} \\Vert \\nu^\\star- \\nu\\Vert_{\\BL}^* +\\frac{1}{\\tau} \\Hh(\\nu,\\nu_0).\n\\end{align}\n\n In this whole section, we consider without loss of generality $\\alpha=\\beta=1$ (we explain in Section~\\ref{app:conclusionlocal} how to adapt the results to arbitrary $\\alpha,\\beta$). For simplicity, we only track the dependencies in $\\nu$ and $\\tau$. Any quantity that is independent of $\\nu$ and $\\tau$ is treated as a constant and represented by $C,C',C''>0$, and the quantity these symbols refer to can change from line to line.\n\n\nGiven a measure $\\nu\\in \\Mm_+(\\Theta)$, we consider the local centered moments introduced in Definition~\\ref{def:localfeatures} and in addition, for $i\\in \\{1,\\dots,m^\\star\\}$,\n\\begin{align*}\n\\delta\\theta_i = \\bar\\theta_i-\\theta_i && \\tilde b^\\theta_i = \\bar r_i\\delta\\theta_i, && s_i \\coloneqq \\bar r_i(\\tr\\Sigma_i)^{\\frac12}.\n\\end{align*}\n\nFinally, we will quantify errors with the following quantity\n\\begin{align}\\label{eq:error}\nW_\\tau(\\nu)^2 \\coloneqq \\bar r_0^2 + \\Vert  b\\Vert^2 + \\Vert s\\Vert^2 = \\bar r_0^2 + \\sum_{i=1}^{m^\\star} \\left(\\vert  b_i^r\\vert^2 + \\Vert b_i^\\theta\\Vert^2  + s_i^2 \\right)\n\\end{align}\n\n In the next lemma, we globally bound the quantity $W_\\tau(\\nu)$ from Eq.~\\eqref{eq:error} in terms of the function values. It involves the quantity $v^\\star>0$ which is such that for any local minimum $\\theta$ of $J'_{\\nu^\\star}$, either $\\theta = \\theta_i$ for some $i\\in \\{1,\\dots,m^\\star\\}$ or $J'_{\\nu^\\star}(\\theta)\\geq v^*$ (which is non-zero under {\\sf (A5)}). We also recall that $\\tilde b^\\theta_i = \\bar r_i \\delta\\theta_i$, as defined in Section~\\ref{app:bounddistanceA}.\n","units":[{"start":5517,"end":6505,"start_line":169,"end_line":173,"raw_tex":"Finding parsimonious descriptions of complex observations is an important problem in machine learning and signal processing.  In its simplest form, this task boils down to searching for an element in a Hilbert space $\\Ff$ that is close to a certain $f_0\\in \\Ff$ --- the observations --- and that is a linear combination of a few elements from a parameterized set $\\{\\phi(\\theta)\\}_{\\theta \\in \\Theta}\\subset \\Ff$ --- the parsimonious description. This can be formulated as a minimization problem where the linear combination is expressed through an unknown measure $\\nu$ and the distance to $f_0$ is quantified using a smooth convex loss function $R:\\Ff\\to \\RR$, such as the square loss $R(f)=\\frac12 \\Vert f-f_0\\Vert^2_\\Ff$.  The problem to solve is then\n%\n\\begin{equation}\\label{eq:objective}%\\tag{$\\Pp_\\lambda$}\nJ^*\\coloneqq \\min_{\\nu \\in \\Mm_+(\\Theta)} J(\\nu), \\qquad \\text{}\\qquad J(\\nu) :=R\\left( \\int_\\Theta \\phi(\\theta) \\d \\nu(\\theta)\\right) + \\lambda  \\nu (\\Theta)\n\\end{equation}","sha256_utf8":"2537496c6b6c691d8af927c4a3b52480ed12deeee695548a22d18fc58d0f2f29"},{"start":6508,"end":8214,"start_line":175,"end_line":178,"raw_tex":"where $\\Mm_+(\\Theta)$ is the set of nonnegative measures $\\nu$ on the parameter space $\\Theta$ with finite total mass $\\nu(\\Theta)<\\infty$ and $\\lambda> 0$ is the regularization strength. This formulation also covers minimization over signed measures with total variation regularization, by replacing $\\Theta$ with the disjoint union of two copies of $\\Theta$ where $\\phi$ takes opposite values, see Appendix~\\ref{app:signed}. A large body of research has exhibited the favorable properties of minimizers of such problems~\\cite{bach2017breaking, de2012exact, mairal2014sparse} with a statistical or variational viewpoint, showing in particular that $\\lambda$ favors sparser solutions and increases stability as it gets larger, at the expense of introducing a stronger bias. The present paper deals with the optimization aspect: our goal is to design algorithms that return $\\epsilon$-accurate solutions with a guaranteed computational complexity. When the set $\\Theta$ is a finite set, this is a finite dimensional convex optimization problem that is well understood~\\cite{boyd2004convex, bach2012optimization}. However, convex approaches are generally inefficient when $\\Theta$ is a continuous space, such as a $d$-dimensional manifold, where the need to discretize the space leads to a complexity scaling as $\\epsilon^{-d}$ in the accuracy $\\epsilon$. We consider the following setting:\n\\begin{enumerate}\n\\item[] {\\sf (A1)}  $\\Theta$ is a compact $d$-dimensional Riemannian manifold without boundaries. The functions $\\phi:\\Theta\\to \\Ff$ and $R:\\Ff\\to \\RR_+$ are twice Fr\\'echet differentiable, with locally Lipschitz second-order derivatives, and $\\nabla R$ is bounded on sublevel sets.\n\\end{enumerate}\n","sha256_utf8":"33a45e4484feaa9f37790f26f779883720c5991e207c7e3d3338b1f9d26000fd"},{"start":19046,"end":20661,"start_line":249,"end_line":255,"raw_tex":"The set of signed (resp.~nonnegative) finite Borel measures on a metric space $(\\Xx,\\dist)$ is denoted by $\\Mm(\\Xx)$ (resp. $\\Mm_+(\\Xx)$). The relative entropy, a.k.a.~Kullback-Leibler divergence, is defined for $\\nu_1,\\nu_2 \\in \\Mm_+(\\Xx)$ as\n$\\Hh (\\nu_1,\\nu_2) = \\int_{\\Xx} \\log(\\d\\nu_1/\\d\\nu_2) \\d\\nu_1 - \\nu_1(\\Xx)+\\nu_2(\\Xx)$ if $\\nu_1$ is absolutely continuous w.r.t.~$\\nu_2$, and $+\\infty$ otherwise. The $p$-Wasserstein distance on the set $\\Pp_p(\\Xx)$ of probability measures with finite $p$-th moment is defined, for $\\mu_1,\\mu_2 \\in \\Pp_p(\\Xx)$ as \n\\[\nW_p(\\mu_1,\\mu_2) = \\left(\\min_{\\gamma \\in \\Pi(\\mu_1,\\mu_2)} \\int \\dist(x_1,x_2)^p\\d\\gamma(x_1,x_2)\\right)^{1/p}\n\\]\nwhere $\\Pi(\\mu_1,\\mu_2)$ is the set of measures on $\\Xx \\times \\Xx$ with marginals $\\mu_1$ and $\\mu_2$. The distance $W_\\infty$ between compactly supported probabilities is defined as the limit of $W_p$ as $p\\to \\infty$ and can be directly defined as $W_\\infty(\\mu_1,\\mu_2) = \\inf_{\\gamma \\in \\Pi(\\mu_1,\\mu_2)} \\max_{(x_1,x_2)\\in \\spt \\gamma} \\dist(x_1,x_2)$~\\cite{santambrogio2015optimal}. We also define the Bounded-Lipschitz norm for a continuous function $\\psi:\\Xx\\to \\RR$ as $\\Vert \\psi \\Vert_{\\BL} = \\Vert \\psi\\Vert_\\infty+\\Lip(\\psi)$ where $\\Lip(\\psi)$ is the Lipschitz constant of $\\psi$ and its dual norm on $\\Mm(\\Xx)$ as \n$\\Vert \\nu \\Vert^*_{\\BL} \\coloneqq \\sup_{\\Vert \\varphi \\Vert_{\\BL}\\leq 1} \\int \\varphi \\, \\d\\nu$. For a Riemannian manifold $\\Theta$, we denote by $T_\\theta\\Theta$ the tangent space of $\\Theta$ at $\\theta$ and by $\\langle \\cdot,\\cdot\\rangle_\\theta : (T_\\theta \\Theta)^2 \\to \\RR_+$ the metric at $\\theta$.\n","sha256_utf8":"28a655ed4d60106cf67b499fc9379de69d8d8740cced4eccb4659eb2d0fb8e59"},{"start":20748,"end":21357,"start_line":259,"end_line":262,"raw_tex":"Consider a smooth increasing bijection $h:\\RR_+\\to \\RR_+$ (such as a power function $r\\mapsto r^p$) and a number of particles $m \\in \\NN^*$. The idea behind particle gradient-based algorithms is to parameterize the unknown measure $\\nu$ as $\\frac1m \\sum_{i=1}^m h(r_i) \\delta_{\\theta_i}$ and to perform gradient-based optimization on the corresponding objective\n\\begin{equation}\\label{eq:objectivediscrete}%\\tag{$\\widehat \\Pp^m_\\lambda$}\nF_{m}((r_1,\\theta_1),\\dots,(r_{m},\\theta_m)) \\coloneqq  R\\left(\\frac1{m} \\sum_{i=1}^{m} h(r_i)\\phi(\\theta_i)\\right) + \\frac\\lambda{m} \\sum_{i=1}^{m} h(r_i),\n\\end{equation}","sha256_utf8":"bee99d2f14b8bdf1468b0b065717bd4f32c35de18b0bed661a7b92b71c7cb85a"},{"start":21358,"end":22889,"start_line":263,"end_line":268,"raw_tex":"where the parameters $(r_i,\\theta_i)$ of each particle belong to $\\Omega \\coloneqq \\RR_+ \\times \\Theta$ endowed with a specific choice of metric. Clearly, if $J$ admits a minimizer that is a mixture of $m^\\star$ atoms with $m^\\star\\leq m$, then it is sufficient to minimize $F_m$ from Eq.~\\ref{eq:objectivediscrete} for solving \\eqref{eq:objective}. While~\\eqref{eq:objectivediscrete} is finite dimensional, it is typically non-convex with possibly some strict local minima. Still, when $R$ is convex and for $h(r)=r^p$ for $p\\in \\{1,2\\}$, the message from~\\cite{chizat2018global}  (see Theorem~\\ref{th:consistency}) is that  solving~\\eqref{eq:objectivediscrete} to global optimality with first-order methods is still possible by using over-parameterization, i.e.\\ choosing $m$ much larger than $m^\\star$.  Such a method involve various key hyper-parameters which role is discussed throughout the paper. They include (i) the choice of the function $h$ (ii) the choice of the metric on $\\Omega^m$ and (iii) the choice of the initialization.\n\n\\paragraph{Expression of the gradient.} Under~{\\sf(A1)}, the objective $J$, seen as a function on the space $\\Mm(\\Theta)$ endowed with the total variation norm, is Fr\\'echet-differentiable. Its differential at $\\nu\\in \\Mm(\\Theta)$ can be represented by the function $J'_\\nu : \\Theta \\to \\RR$ given by\n\\begin{equation}\\label{eq:firstvariation}\nJ'_\\nu(\\theta) = \\left\\langle \\phi(\\theta),\\nabla R\\left(\\int_\\Theta \\phi(\\theta) \\d\\nu(\\theta)\\right)\\right\\rangle_{\\Ff} +\\lambda,\n \\end{equation}","sha256_utf8":"291acf2279fda7d1cd26930482907222b923c2a6d82a8598ec9bf57ad590ea7c"},{"start":22890,"end":23567,"start_line":269,"end_line":273,"raw_tex":" in the sense that for any $\\sigma \\in\\Mm(\\Theta)$, it holds\n$\\frac{d}{d\\epsilon} J(\\nu + \\epsilon \\sigma)\\vert_{\\epsilon=0} = \\int_{\\Theta} J'_\\nu(\\theta) \\d\\sigma(\\theta)$. Now, consider a metric on $(\\Omega^*)^m$ that is the average $(1/m)\\sum_{i=1}^m \\langle \\cdot, \\cdot \\rangle_{(r_i,\\theta_i)}$ of metrics on each factor $\\Omega^*\\coloneqq \\RR_+^*\\times \\Theta$, where $\\RR_+^*$ is the set of positive real numbers, of the form\n\\begin{equation}\\label{eq:metric}\n\\langle(\\delta r_1,\\delta \\theta_1),(\\delta r_2,\\delta \\theta_2) \\rangle_{(r,\\theta)} = \\alpha(r)^{-1} \\delta r_1\\delta r_2 +\\beta(r)^{-1}\\langle \\delta \\theta_1,\\delta\\theta_2\\rangle_{\\theta} \n\\end{equation}","sha256_utf8":"2e7707065c0c24f993bacd9d0b0b0c6ade1379c8d7a0debda66b4a97ad88a567"},{"start":23568,"end":24103,"start_line":274,"end_line":274,"raw_tex":"where $\\alpha$ and $\\beta$ are smooth functions $\\RR_+^*\\to \\RR_+^*$ to be specified\\footnote{Extension of the metric and gradients to the whole of $\\Omega$ can be made on a case by case basis, see Section~\\ref{sec:conic}.}, $ (r,\\theta) \\in \\Omega^*$, $\\delta r_1, \\delta r_2 \\in \\RR$ and $\\delta \\theta_1, \\delta \\theta_2 \\in T_\\theta\\Theta$. Using the fact that gradients are characterized by the relation $\\d F_m(x)(\\delta x) = \\langle \\nabla F_m(x),\\delta x\\rangle$, we get that the gradient of $F_m$ is given, in components, by \n","sha256_utf8":"5b007dfd4eda2af95b8e3db70a507bffd4e22832e7a325371c71863464cb8f38"},{"start":24451,"end":25114,"start_line":284,"end_line":287,"raw_tex":"\\paragraph{Lifted problem in Wasserstein space.} Assume now that $h$ has at most quadratic growth, and that the metric is defined on the whole of $\\Omega$. One can then see the discrete problem~\\eqref{eq:objectivediscrete} as a discretization of a problem on the space $\\Pp_2(\\Omega)$ of probability measures on $\\Omega$ with finite second moment endowed with the Wasserstein-$2$ metric given by\n\\begin{equation}\\label{eq:liftedproblem}%\\tag{$\\Pp_\\lambda$}\nF^\\star = \\min_{\\mu \\in \\Pp_2(\\Omega)} F(\\mu)\\quad \\text{where}\\quad F(\\mu) \\coloneqq \\left( \\int_\\Omega h(r) \\phi(\\theta) \\d\\mu(r,\\theta) \\right) +  \\lambda \\int_\\Omega h(r) \\d\\mu(r,\\theta).\n\\end{equation}","sha256_utf8":"682089855760c1102be862a6a9236ff8a1e735b1841d82cf108994a42060f9c2"},{"start":25117,"end":25579,"start_line":289,"end_line":292,"raw_tex":"This point of view leads to insights on the properties of $F_m$ that are independent of $m$, which is crucial for our theoretical analysis. For a measure $\\mu\\in \\Pp_2(\\Omega)$, we define following~\\cite{liero2018optimal} the \\emph{homogeneous projection} operator $\\h:\\Pp_2(\\Omega)\\to \\Mm_+(\\Theta)$ where  $\\h\\mu$ is characterized by\n\\begin{align*}\n\\int_{\\Theta} \\varphi(\\theta) \\d(\\h\\mu)(\\theta) = \\int_{\\Omega} h(r)\\varphi(\\theta)\\d\\mu(r,\\theta)\n\\end{align*}","sha256_utf8":"95f78905c295c271013fb811a671a7a9cce6be83587d9eb91b2be79e43576cfe"},{"start":25580,"end":25689,"start_line":293,"end_line":294,"raw_tex":"for any continuous function $\\varphi:\\Theta\\to \\RR$. With this operator, we simply have $F(\\mu)=J(\\h\\mu)$. \n\n","sha256_utf8":"e7559a1a2f49e38bf538a06fb2e390057bbb1086f5b5d0bc782d6a873bf9629b"},{"start":26578,"end":27022,"start_line":302,"end_line":305,"raw_tex":"\\paragraph{Wasserstein gradient flow.} It is also possible to directly study the optimization dynamics in the space $\\Pp_2(\\Omega)$ for the functional $F$ of Eq.~\\eqref{eq:liftedproblem}. For a measure $\\nu \\in \\Mm_+(\\Theta)$, consider the vector field on $\\Omega$ with expression\n\\begin{equation*}\ng_\\nu(r,\\theta) = \\big(\\alpha(r) h'(r) J'_{\\nu}(\\theta),\\beta(r) h(r)\\nabla J'_{\\nu}(\\theta)\\big) \\in \\RR \\times T_\\theta \\Theta.\n\\end{equation*}","sha256_utf8":"3b2bf8bddde06a6b77f365d62aad6aeb153ee9132f80f5bd1404b2669ab7ebce"},{"start":27023,"end":27458,"start_line":306,"end_line":309,"raw_tex":"We refer to $g_{\\h \\mu}$ as the Wasserstein gradient of $F$ at $\\mu$ (this notation emphasizes that it only depends on $\\mu$ through $\\h \\mu$). Gradient flows of $F_m$ are particular cases of \\emph{Wasserstein gradient flows} of $F$. The latter are defined as the absolutely continuous curves $(\\mu_t)_{t\\geq 0}$ in $\\Pp_2(\\Omega)$ that satisfy\n\\begin{equation}\\label{eq:WGF}\n\\partial_t \\mu_t =  \\div (\\mu_t g_{\\h\\mu_t})\n\\end{equation}","sha256_utf8":"dda4dfcbe61fe5664748576237c5b194d6a4c5533d419508c19be2d9e050fecb"},{"start":27459,"end":28031,"start_line":310,"end_line":310,"raw_tex":"in the weak sense, which means that for any differentiable function $\\varphi: \\Omega \\to \\RR$, it holds $\\frac{\\d}{\\d t}\\left(\\int \\varphi \\,\\d\\mu_t \\right) = - \\int \\nabla \\varphi \\cdot g_{\\h\\mu_t}\\d\\mu_t$, for almost every $t\\geq 0$, see~\\cite{santambrogio2015optimal}. This is a proper extension of the notion of gradient flow for $F_m$ in the sense that if $x(t)=(r_i(t),\\theta_i(t))_{i=1}^m$ is a gradient flow of $F_m$ then it can be directly checked that  $t\\mapsto \\mu_t = \\frac1m \\sum_{i=1}^m \\delta_{(r_i(t),\\theta_i(t))}$ is a Wasserstein gradient flow of $F$.\n","sha256_utf8":"13ad65ea5fc392f217bef6e96ec414936dd7062e3716012e3da634c3e95a99b8"},{"start":28077,"end":29325,"start_line":313,"end_line":317,"raw_tex":"As seen in Eq.~\\eqref{eq:gradientgeneric}, the choice of the homogeneity degree and of the metric on $\\Omega$ determine a specific way to combine the \\emph{vertical}  and the \\emph{spatial} components of the gradient (along the variable $r$ and $\\theta$, respectively). From now on, we focus on what we refer to as the \\emph{conic case}, which corresponds to the following assumption:\n\\begin{itemize}\n\\item [\\sf (A2)] The mass parameterization is $h(r)=r^2$ and the metric on $\\Omega^*$ is of the form Eq.~\\eqref{eq:metric} with $(\\alpha(r),\\beta(r))=(\\alpha,\\beta/r^2)$ for some $\\alpha,\\beta>0$.\n\\end{itemize}\nThe corresponding geodesic distance is $\\dist((r_1,\\theta_2),(r_1,\\theta_2))^2 = r_1^2 +r_2^2 -2r_1r_2\\cos_{\\pi}(\\dist(\\theta_1,\\theta_2))$ where $\\cos_\\pi(z) = \\cos(\\min\\{\\pi,z\\})$. This metric can be extended as a proper metric on $\\widetilde \\Omega$, defined as the set $\\Omega$ where the subset $\\{0\\}\\times \\Theta$ is identified to a single point, known as the \\emph{cone metric}, which is the canonical way to define a metric on $\\tilde \\Omega$~\\cite{burago2001course}. In our context, identifying $\\{0\\}\\times \\Theta$ to a single point is desirable because a particle located in this set is a ``dead'' particle carrying no mass.\n","sha256_utf8":"7b143d81ac2c2937b48a318e0b81e32c83aac3ddedd20acb2b57287add856fdc"},{"start":29326,"end":29753,"start_line":319,"end_line":327,"raw_tex":"Plugging the metric into Eq.~\\eqref{eq:gradientgeneric} gives the gradient (extended by continuity to $\\{0\\}\\times \\Theta$)\n\\begin{align*}\n\\left\\{\n\\begin{aligned}\n\\nabla_{r_i} F_m((r_i,\\theta_i)_{i=1}^m) &= 2\\alpha r_i J'_{\\nu}(\\theta_i)\\\\\n\\nabla_{\\theta_i} F_m ((r_i,\\theta_i)_{i=1}^m) &=  \\beta \\nabla J'_{\\nu}(\\theta_i)\n\\end{aligned}\n\\right. && \\text{where} && \\nu = \\frac1m \\sum_{i=1}^m r_i^2\\delta_{\\theta_i},\n\\end{align*}","sha256_utf8":"d4a805c6b103be949ffe88684bd5686e3e2c528e82899e572cc20d9535172a3b"},{"start":29754,"end":30029,"start_line":328,"end_line":331,"raw_tex":"and the Wasserstein gradient is represented by the vector field\n\\begin{equation}\\label{eq:wassersteingradient}\ng_\\nu(r,\\theta) = \\big(2 \\alpha r J'_{\\nu}(\\theta), \\beta \\nabla J'_{\\nu}(\\theta)\\big)  \\in \\RR \\times T_\\theta \\Theta, \\forall (r,\\theta)\\in \\Omega.\n\\end{equation}","sha256_utf8":"ef0b59a4768021e9a29f625499ed65688aecf1a98195cd74731bbbb24db5ae56"},{"start":45873,"end":46017,"start_line":457,"end_line":460,"raw_tex":"In order to derive global optimality conditions, we assume the following.\n\\begin{itemize}\n\\item [\\sf(A3)] The loss $R$ is convex.\n\\end{itemize}\n","sha256_utf8":"af68c94f8e3644914e8d3e61c99354cf25c392c2f30776f884a42a2606a9ab4e"},{"start":47621,"end":48461,"start_line":471,"end_line":474,"raw_tex":"\\begin{itemize}\n\\item [\\sf(A4)] Problem~\\eqref{eq:objective} admits a unique global minimizer on $\\Mm_+(\\Theta)$ which is of the form $\\nu^\\star = \\sum_{i=1}^{m^\\star} r_i^2\\delta_{\\theta_i}$ with $\\nu^\\star(\\Theta)>0$. We denote $f^\\star \\coloneqq \\int \\phi \\d\\nu^\\star = \\sum_{i=1}^{m^\\star}r_i^2\\phi(\\theta_i)$.\n\\end{itemize}\nWithout loss of generality, we assume $r_i>0$ for all $i$ and $\\theta_i \\neq \\theta_{i'}$ whenever $i\\neq i'$, so that $(r_i,\\theta_i)_{i=1}^{m^\\star}$ is uniquely well-defined, up to re-ordering. Let us fix from now on normal coordinates frames on the neighborhood of each $\\theta_i$. This allows to identify tensors at $\\theta_i$ with their expression in coordinates and also induces a set of coordinates on the direct sum of the tangent spaces $T_{\\theta_i}\\Theta$, which is of dimension $m^\\star\\times d$. \n","sha256_utf8":"fe41cf2c23b4b23f4895ca6380bb6262ce3ba65aff2e48a56ec0f0d392ad588f"},{"start":48503,"end":48721,"start_line":477,"end_line":480,"raw_tex":"We define the \\emph{global kernel} $K\\in \\RR^{(m^\\star\\times (1+d))^2}$ by\n\\[\nK_{(i,j),(i',j')} \\coloneqq \\langle r_i \\bar \\nabla_j \\phi(\\theta_i), r_{i'} \\bar \\nabla_{j'}\\phi(\\theta_{i'})\\rangle_{d^2\\! R_{f^\\star}}\n\\]","sha256_utf8":"0e990a31c9860a4708fb3b323cfcf017766c82b9982eb3732a4ef81349d328c1"},{"start":48722,"end":49285,"start_line":481,"end_line":484,"raw_tex":"where $\\bar\\nabla \\phi \\coloneqq (2 \\alpha \\phi, \\beta \\nabla \\phi)$ can be interpreted as the gradient of $\\h\\phi$ at $(1,\\theta)$. Remark that $K$ is defined via the quadratic form associated to the Hessian of $R$ at $f^\\star$. This interaction kernel $K$ appears naturally in the various statistical and optimization analysis of the minimization problem under consideration~\\cite{duval2015exact,traonmilin2018basins}. We also use the notation for the \\emph{local kernels}  for $i\\in 1,\\dots,m^\\star$\n\\[\nH_i\\coloneqq \\beta^2 \\nabla^2 J'_{\\nu^\\star}(\\theta_i)\n\\]","sha256_utf8":"92fb13859bf8f6f9ee3dfdcae8e0fbb74beeebc763b1e184ed5488e5b8065ae1"},{"start":49286,"end":49471,"start_line":485,"end_line":485,"raw_tex":"expressed in local coordinates.  In order to simplify notations, we concatenate these matrices in a large matrix $H\\in \\RR^{(m^\\star\\times (1+d))^2}$ of the same size as $K$ defined as\n","sha256_utf8":"de65c675cd5dc163ea555dab2876167aae9bb3db2923c1199521e52adc175980"},{"start":49472,"end":49647,"start_line":486,"end_line":491,"raw_tex":"\\[\n H_{(i,j),(i,j')} =\\begin{cases}\n \\beta^2  \\nabla^2_{j,j'} J'_{\\nu^\\star}(\\theta_i) &\\text{if $i=i'$ and $j,j'\\geq 1$},\\\\\n  0 &\\text{ if $j=0$ or $j'=0$}.\n  \\end{cases}\n \\]","sha256_utf8":"73dfdd99cdea761fde5d76df42baa8d58ea48adf522959b4e13b03e17a6fb2c0"},{"start":49648,"end":50267,"start_line":492,"end_line":495,"raw_tex":"where here and in the proofs, we use $0$ to label the $r$'s coordinate. The local analysis will be carried under the following non-degeneracy assumptions.\n\\begin{itemize}\n\\item [\\sf(A5)] The minimizer $\\nu^\\star$ is non-degenerate in the sense that $\\nabla^2 R(f^\\star)$ is positive definite and, calling $\\sigma_{\\min}(A)$ the smallest singular value of a linear operator $A$, we have global curvature  $\\sigma_{\\min}(K)>0$, local curvature  $\\sigma_{\\min}(H) = \\min_{i} \\sigma_{\\min}(H_i)>0$, and strict slackness, i.e.\\ the only points where $J'_{\\nu^\\star}$ vanishes are $(\\theta_i)_{i=1}^{m^\\star}$.\n\\end{itemize}\n","sha256_utf8":"6f366ab98e2ea0749b00fbcd42b16a6569e17020bfd4518fe833b53d2f81245e"},{"start":50974,"end":51660,"start_line":500,"end_line":504,"raw_tex":"A first consequence of these assumptions is that convergence in value implies convergence to minimizers. The distance on $\\Mm_+(\\Theta)$ that naturally appears in the analysis is the Wasserstein-Fisher-Rao, a.k.a.\\ Hellinger-Kantorovich metric $\\widehat W_2$, which is the extension of the Wasserstein $W_2$ metric to unnormalized measures. It admits many equivalent definitions~\\cite{liero2018optimal, kondratyev2016new, chizat2018unbalanced}, the most suitable to our context being~\\cite[\nThm. 7.20]{liero2018optimal}\n\\[\n\\widehat W_2(\\nu_1,\\nu_2) \\coloneqq \\min \\left \\{W_2(\\mu_1,\\mu_2)\\;;\\; (\\mu_1,\\mu_2)\\in \\Pp_2(\\Omega)^2 \\text{ satisfy } (\\h\\mu_1,\\h\\mu_2)=(\\nu_1,\\nu_2)\\right\\}\n\\]","sha256_utf8":"4441a86d714d690bf582bb06c661cc1f7c86f0c62d2d2212a02ca38f85ae35c8"},{"start":51661,"end":51961,"start_line":505,"end_line":505,"raw_tex":"where the Wasserstein distance on $\\Omega$ is defined relative to the cone metric (in this paragraph, with $\\alpha=\\beta=1$). The proof of the following result involves the construction of a transport map in the lifted space $\\Pp_2(\\Omega)$ and is postponed to Appendix~\\ref{app:distanceinequality}.\n","sha256_utf8":"43ee6d31719fa6f22eef575e641f1c8e0c6c922a255d55bd0d9b253fd01eabb7"},{"start":52327,"end":52700,"start_line":511,"end_line":511,"raw_tex":" Our first main result is a lower bound on the squared norm of the gradient in terms of the sub-optimality gap, an inequality known as sharpness, or Polyak-\\L{}ojasiewicz inequality~\\cite{polyak1963gradient,karimi2016linear}, which is a special case of \\L{}ojasiewicz gradient inequality. It involves the $L^2(\\nu)$ norm of the gradient, which we denote for $\\nu=\\h\\mu$ by\n","sha256_utf8":"49a66703bca42d911b882cb30c8089742a6a04d4614d71b0f26b55e63467c941"},{"start":52701,"end":53037,"start_line":512,"end_line":514,"raw_tex":"\\[\n \\Vert g_{\\nu}\\Vert^2_{L^2(\\nu)} \\coloneqq \\int_{\\Omega} \\left( \\frac1\\alpha \\vert 2\\alpha rJ'_{\\nu}(\\theta)\\vert^2 + \\frac {r^2}{\\beta} \\Vert \\beta \\nabla J'_{\\nu}\\Vert_\\theta^2\\right)\\d\\mu(r,\\theta) = \\int_\\Theta \\left( 4 \\alpha \\vert J'_\\nu(\\theta)\\vert^2 +\\beta \\Vert \\nabla J'_\\nu(\\theta)\\Vert^2_\\theta \\right)\\d\\nu(\\theta).\n \\]","sha256_utf8":"fffcd1bca7c87a0fabab01ccf7ae4ec36726a22c849e1f1aa491539b7bc6a4c9"},{"start":55983,"end":57613,"start_line":551,"end_line":564,"raw_tex":"The proof of Theorem~\\ref{th:gradientinequality}, in Appendix~\\ref{app:sharpness}, is based on a local expansion of $J(\\nu)$ in terms of some local moments of $\\nu$. For a radius $\\tau>0$ (that shall be fixed at some small enough value in the course of the proof), we define the sets  for $i\\in \\{1,\\dots,m^\\star\\}$,\n\\begin{align*}\n\\Theta_i \\coloneqq  \\{ \\theta \\in \\Theta \\;;\\; \\dist(\\theta,\\theta_i) < \\tau \\}.\n\\end{align*}\nWe assume that $\\tau$ is smaller than $1$ and small enough so that these sets together with $\\Theta_0 \\coloneqq \\Theta \\setminus \\cup_{i=1}^{m^\\star} \\Theta_i$ form a partition of $\\Theta$ and that the exponential map at $\\theta_i$ has injectivity radius larger than $\\tau$, for $i\\in \\{1,\\dots,m^\\star\\}$. We then say that $\\tau$ is an \\emph{admissible} radius. \n\\begin{definition}[Local moments]\\label{def:localfeatures}\nGiven an admissible radius $\\tau>0$ and a measure $\\nu\\in \\Mm_+(\\Theta)$, we define for $i\\in \\{0,\\dots,m^\\star\\}$ the local masses $\\bar r_i^2 = \\nu(\\Theta_i)$ and the local means $\\bar \\theta_i \\coloneqq \\frac1{\\bar r_i^2} \\int_{\\Theta_i} \\theta\\d\\nu(\\theta)$ if $\\nu(\\Theta_i)>0$ and $\\bar \\theta_i = \\theta_i$ otherwise. Finally, we define for $i\\in \\{1,\\dots,m^\\star\\}$ the \\emph{weighted biases}\n\\begin{align*}\nb_i^r \\coloneqq \\frac{\\bar r_i^2 -r_i^2}{2\\alpha r_i},&&\nb_i^\\theta \\coloneqq \\frac{\\bar r_i^2}{\\beta r_i}(\\bar \\theta_i-\\theta_i), &&\nb_i = (b_i^r,b_i^\\theta),\n\\end{align*}\nand the \\emph{weighted covariances} $\\Sigma_i \\coloneqq \\frac1{\\bar r_i^2 \\beta^2} \\int_{\\Theta_i} (\\theta - \\bar \\theta_i)(\\theta - \\bar \\theta_i)^\\intercal \\d\\nu(\\theta)$.\n\\end{definition}","sha256_utf8":"26e88f140c6d0d1bfc28f9ff1c644e7b0c3bec2055a4abed85abea8942179987"},{"start":65412,"end":65857,"start_line":619,"end_line":622,"raw_tex":"In order to state the condition on the initialization, we first choose a reference measure $\\rho \\in \\Mm_+(\\Theta)$ with a smooth positive density, also denoted by $\\rho$, which represents our prior knowledge about the solution $\\nu^\\star$. We introduce the quantity (analogous to a log-likelihood)\n\\[\n\\bar\\Hh(\\nu^\\star,\\rho) \\coloneqq \\sum_{i=1}^{m^\\star} r_i^2 \\log\\left(\\frac{r_i^2}{\\rho(\\theta_i)}\\right) - \\nu^\\star(\\Theta)+\\rho(\\Theta).\n\\]","sha256_utf8":"2f0739846b0119bb9e5563844f753f75276d3af8748e96a8e852e06e69843d02"},{"start":71632,"end":72203,"start_line":658,"end_line":662,"raw_tex":" The proof of Theorem~\\ref{thm:convergenceflow} mostly relies on the the following general lemma which applies to any type of initialization or any structure of minimizers. It gives an upper bound on the optimality gap during along gradient flows in terms of a \\emph{mirror rate} function $\\Qq_{\\nu_0,\\nu^\\star}: \\RR^*_+ \\to \\RR_+$ defined for $\\nu^\\star, \\nu_0\\in \\Mm_+(\\Theta)$  and $\\tau>0$ as\n\\begin{align}\\label{eq:quantization}\n\\Qq_{\\nu^\\star,\\nu_0}(\\tau) = \n\\inf_{\\nu \\in \\Mm_+(\\Theta)} \\Vert \\nu^\\star- \\nu\\Vert_{\\BL}^* +\\frac{1}{\\tau} \\Hh(\\nu,\\nu_0).\n\\end{align}","sha256_utf8":"cbd9dffdc055746045d754e22c8ba9cd03ca2ee20087dd794360df535ed45cb9"},{"start":99686,"end":100119,"start_line":929,"end_line":929,"raw_tex":" In this whole section, we consider without loss of generality $\\alpha=\\beta=1$ (we explain in Section~\\ref{app:conclusionlocal} how to adapt the results to arbitrary $\\alpha,\\beta$). For simplicity, we only track the dependencies in $\\nu$ and $\\tau$. Any quantity that is independent of $\\nu$ and $\\tau$ is treated as a constant and represented by $C,C',C''>0$, and the quantity these symbols refer to can change from line to line.\n","sha256_utf8":"0fe6f4a58e5d625282ebbfe8f5d836f4f50cfcbb9d819b03b75a3a4b5569f042"},{"start":100208,"end":100548,"start_line":932,"end_line":935,"raw_tex":"Given a measure $\\nu\\in \\Mm_+(\\Theta)$, we consider the local centered moments introduced in Definition~\\ref{def:localfeatures} and in addition, for $i\\in \\{1,\\dots,m^\\star\\}$,\n\\begin{align*}\n\\delta\\theta_i = \\bar\\theta_i-\\theta_i && \\tilde b^\\theta_i = \\bar r_i\\delta\\theta_i, && s_i \\coloneqq \\bar r_i(\\tr\\Sigma_i)^{\\frac12}.\n\\end{align*}","sha256_utf8":"2746e13704021b968444867a91edd30ee44ffe9208c6009ed18f94ba41845b8a"},{"start":100549,"end":100826,"start_line":936,"end_line":939,"raw_tex":"Finally, we will quantify errors with the following quantity\n\\begin{align}\\label{eq:error}\nW_\\tau(\\nu)^2 \\coloneqq \\bar r_0^2 + \\Vert  b\\Vert^2 + \\Vert s\\Vert^2 = \\bar r_0^2 + \\sum_{i=1}^{m^\\star} \\left(\\vert  b_i^r\\vert^2 + \\Vert b_i^\\theta\\Vert^2  + s_i^2 \\right)\n\\end{align}","sha256_utf8":"95d13add6820243a360524a3df5d649db1ea839b7f3fb21aca48f9f86fef8647"},{"start":105753,"end":106237,"start_line":1014,"end_line":1014,"raw_tex":" In the next lemma, we globally bound the quantity $W_\\tau(\\nu)$ from Eq.~\\eqref{eq:error} in terms of the function values. It involves the quantity $v^\\star>0$ which is such that for any local minimum $\\theta$ of $J'_{\\nu^\\star}$, either $\\theta = \\theta_i$ for some $i\\in \\{1,\\dots,m^\\star\\}$ or $J'_{\\nu^\\star}(\\theta)\\geq v^*$ (which is non-zero under {\\sf (A5)}). We also recall that $\\tilde b^\\theta_i = \\bar r_i \\delta\\theta_i$, as defined in Section~\\ref{app:bounddistanceA}.\n","sha256_utf8":"a05ebd97be8d2a472d9a1adefc32936dbe263d7a1c53f420c54857821a3d0dde"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}[Global convergence of gradient flow]\\label{thm:convergenceflow}\nUnder {\\sf (A1-5)}, let $J_0$ and $\\kappa_0$ given by Theorem~\\ref{th:gradientinequality}, let $\\rho = \\rho\\!\\vol \\in \\Mm_+(\\Theta)$ an absolutely continuous reference measure with $\\log \\rho$ $L$-Lipschitz and let  $B_{\\nu_0} = \\sup_{J(\\nu)\\leq J(\\nu_0)} \\Vert J'_{\\nu}\\Vert_{\\BL}$, where $\\nu_0\\in \\Mm_+(\\Theta)$ is the initialization. For any $0<\\epsilon\\leq 1/2$, there exists $C_\\epsilon >0$ that only depend on $\\epsilon$ and bounds on the curvature of $\\Theta$ such that if it holds $\\beta/\\alpha \\leq (4B_{\\nu_0}/\\max\\{1,L\\})^2$,\n\\begin{align*}\nW_\\infty(\\nu_0,\\rho)\\leq \\frac{J_0-J^\\star}{2B_{\\nu_0}\\nu^\\star(\\Theta)} &&\\text{and}&&\n\\frac{\\beta}{\\alpha} \\leq\\left(  \\frac{J_0-J^\\star}{2(4B_{\\nu_0})^{\\epsilon}\\left(\\bar \\Hh(\\nu^\\star,\\rho) +B^2_{\\nu_0} +C_\\epsilon \\nu^\\star(\\Theta)d \\right)}\\right)^\\frac{2}{1-\\epsilon}\n\\end{align*}\n then the projected gradient flow $(\\nu_t)_{t\\geq 0}$ initialized with $\\nu_0$ converges to the global minimizer $\\nu^\\star$. Denoting $t_0 = 1/\\sqrt{\\alpha\\beta}$ it satisfies, for $t\\geq t_0$,\n\\begin{align*}\n J(\\nu_t)-J^\\star  \\leq  (J_0 - J^*) \\exp(-2\\kappa_0\\min\\{\\alpha,\\beta\\} (t-t_0)).\n \\end{align*}\n\\end{theorem}","units":[{"start":66643,"end":67885,"start_line":628,"end_line":638,"raw_tex":"\\begin{theorem}[Global convergence of gradient flow]\\label{thm:convergenceflow}\nUnder {\\sf (A1-5)}, let $J_0$ and $\\kappa_0$ given by Theorem~\\ref{th:gradientinequality}, let $\\rho = \\rho\\!\\vol \\in \\Mm_+(\\Theta)$ an absolutely continuous reference measure with $\\log \\rho$ $L$-Lipschitz and let  $B_{\\nu_0} = \\sup_{J(\\nu)\\leq J(\\nu_0)} \\Vert J'_{\\nu}\\Vert_{\\BL}$, where $\\nu_0\\in \\Mm_+(\\Theta)$ is the initialization. For any $0<\\epsilon\\leq 1/2$, there exists $C_\\epsilon >0$ that only depend on $\\epsilon$ and bounds on the curvature of $\\Theta$ such that if it holds $\\beta/\\alpha \\leq (4B_{\\nu_0}/\\max\\{1,L\\})^2$,\n\\begin{align*}\nW_\\infty(\\nu_0,\\rho)\\leq \\frac{J_0-J^\\star}{2B_{\\nu_0}\\nu^\\star(\\Theta)} &&\\text{and}&&\n\\frac{\\beta}{\\alpha} \\leq\\left(  \\frac{J_0-J^\\star}{2(4B_{\\nu_0})^{\\epsilon}\\left(\\bar \\Hh(\\nu^\\star,\\rho) +B^2_{\\nu_0} +C_\\epsilon \\nu^\\star(\\Theta)d \\right)}\\right)^\\frac{2}{1-\\epsilon}\n\\end{align*}\n then the projected gradient flow $(\\nu_t)_{t\\geq 0}$ initialized with $\\nu_0$ converges to the global minimizer $\\nu^\\star$. Denoting $t_0 = 1/\\sqrt{\\alpha\\beta}$ it satisfies, for $t\\geq t_0$,\n\\begin{align*}\n J(\\nu_t)-J^\\star  \\leq  (J_0 - J^*) \\exp(-2\\kappa_0\\min\\{\\alpha,\\beta\\} (t-t_0)).\n \\end{align*}\n\\end{theorem}","sha256_utf8":"4b930333a517afb3fe9fae71318c8191732e66312e0493877ad23a0d077325aa"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{proposition} \\label{prop:projectedGF} Under {\\sf (A1-2)}, let $(\\mu_t)_{t\\geq 0}$ be a Wasserstein gradient flow for $F$, with $\\mu_0\\in \\Pp_2(\\Omega)$. Then $\\nu_t = \\h\\mu_t$ satisfies (in the weak sense)\n\\begin{equation}\\label{eq:projectedGF}\n\\partial_t \\nu_t = - 4 \\alpha \\nu_t J'_{\\nu_t} + \\beta \\div (\\nu_t \\nabla J'_{\\nu_t}).\n\\end{equation}\n\\end{proposition}\n\n\\begin{proof}\nFor any differentiable function $\\varphi: \\Theta\\to \\RR$, since $\\mu_t$ is a Wasserstein gradient flow it holds\n\\begin{equation*}\n\\frac{\\d}{\\d t}\\left(\\int \\varphi \\,\\d\\nu_t \\right) \n=- \\int \\langle \\nabla (\\h^*\\varphi), g_{\\h\\mu_t}\\rangle_{(r,\\theta)} \\d\\mu_t = - \\int (4 \\alpha \\varphi J'_{\\nu_t} + \\beta \\nabla \\varphi \\cdot \\nabla J'_{\\nu_t})\\d\\nu_t,\n\\end{equation*}\nwhich is the definition of weak solutions for~\\eqref{eq:projectedGF}.\n\\end{proof}\n\n\\begin{proposition}[Optimality condition]\\label{prop:optimalitycondition} Under~{\\sf(A1)} and~{\\sf(A3)}, problem~\\eqref{eq:objective} admits minimizers. Moreover, a measure $\\nu^\\star \\in \\Mm_+(\\Theta)$ is a minimizer if and only if it holds $J'_{\\nu^\\star}(\\theta)\\geq 0$ for all $\\theta\\in \\Theta$ and $J'_{\\nu^\\star}(\\theta)=0$ whenever $\\theta$ in the support of $\\nu^\\star$.\n\\end{proposition}\n\n\\begin{proof}\nAs $\\lambda$ is assumed positive, the sublevel sets of $J$ on $\\Mm_+(\\Theta)$ are bounded in total variation, and are thus weakly pre-compact. It follows that any minimizing sequence for $J$ admits at least one weak limit point $\\nu^\\star$, which is a minimizer of~\\eqref{eq:objective} since $J$ is weakly continuous. The stated optimality condition is equivalent to having $\\int J'_{\\nu^\\star}\\d (\\nu -\\nu^\\star) \\geq 0$ for all $\\nu \\in \\Mm_+(\\Theta)$. The latter is a sufficient optimality condition since by convexity of $J$, $J(\\nu)-J(\\nu^\\star) \\geq \\int J'_{\\nu^\\star}\\d (\\nu -\\nu^\\star)$. It is also necessary since  it holds $\\frac{d}{d\\epsilon} J((1-\\epsilon)\\nu^\\star +\\epsilon \\nu)\\vert_{\\epsilon = 0^+} = \\int J'_{\\nu^\\star} \\d(\\nu-\\nu^\\star)$.\n\\end{proof}\n\n\\begin{proposition}\\label{prop:distanceinequality}\nUnder {\\sf (A1-5)}, for all $J_{\\max}\\geq J^\\star$, there exists $C,C'>0$, such that if $\\nu\\in \\Mm_+(\\Theta)$ satisfies $J(\\nu)\\leq J_{\\max}$ then $\\Vert \\nu - \\nu^\\star\\Vert^*_{\\BL} \\leq C \\widehat W_2(\\nu, \\nu^\\star) \\leq C'(J(\\nu)-J^\\star)^{\\frac12}$.\n\\end{proposition}\n\n\\begin{theorem}[Sharpness]\\label{th:gradientinequality} Under {\\sf(A1-5)}, there exists $J_0>J^\\star$ and $\\kappa_0>0$, such that for all $\\nu\\in \\Mm_+(\\Theta)$ satisfying $J(\\nu) \\leq  J_0$ and $\\alpha,\\beta>0$, one has\n\\[\n\\frac12 \\Vert g_\\nu\\Vert^2_{L^2(\\nu)}\\geq \\kappa_0\\, \\min\\{\\alpha,\\beta\\}\\, (J(\\nu)-J^\\star).\n\\]\n\\end{theorem}\n\n\\begin{corollary}[Local convergence of gradient flow]\\label{cor:localGF}\nUnder {\\sf (A1-5)}, let $J_0$ and $\\kappa_0$ be given by Theorem~\\ref{th:gradientinequality}. Consider $(\\nu_t)_{t\\geq 0}$ a projected gradient flow for $J$ as in Eq.~\\eqref{eq:projectedGF}. If $J(\\nu_0)\\leq J_0$ then \n\\begin{align*}\nJ(\\nu_t) - J^* &\\leq  (J(\\nu_0) - J^*) \\exp(-2\\kappa_0 \\min\\{\\alpha,\\beta\\}  t).\n\\end{align*}\n\\end{corollary}\n\n\\begin{proof}\nBy Theorem~\\ref{th:gradientinequality} and direct computations, one has \n\\[\n\\frac{d}{dt} (J(\\nu_t)-J^\\star) = \\int_{\\Omega} J'_{\\nu_t} \\d(\\partial_t \\nu_t) = - \\Vert g_{\\nu_t}\\Vert^2_{L^2(\\nu_t)} \\leq -2\\kappa_0\\min\\{\\alpha,\\beta \\} (J(\\nu_t)-J^\\star)\n\\]\nand the result follows by Gr\\\"onwall's lemma.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:mirrordecrease}\nAssume ${\\sf (A1-3)}$ and that $J$ admits a minimizer $\\nu^\\star\\in \\Mm_+(\\Theta)$. Then for all $\\nu_0\\in \\Mm_+(\\Theta)$, denoting $B_{\\nu_0}\\coloneqq \\sup_{J(\\nu)\\leq J(\\nu_0)} \\Vert J'_{\\nu}\\Vert_{\\BL}$, it holds for $t\\geq 0$,\n\\begin{align*}\nJ(\\nu_t)-J^\\star \\leq \\inf_{s\\in [0,t]}\\left( B_{\\nu_0} \\cdot \\Qq_{\\nu^\\star,\\nu_0}(4\\alpha B_{\\nu_0}s) + \\beta B^2_{\\nu_0} s\\right).\n\\end{align*}\n\\end{lemma}\n\n\\begin{proof}\nLet $\\nu^\\epsilon_0 \\in \\Mm_+(\\Theta)$ be a measure to be specified later that satisfies $\\Hh(\\nu^\\epsilon_0, \\nu_0)<+\\infty$, and let $\\nu^\\epsilon_t$ satisfy $\\partial_t \\nu^\\epsilon_t = \\div(\\beta \\nu^\\epsilon_t\\nabla J'_{\\nu_t})$ for $t\\geq 0$ weakly (this is a continuity equation with a smooth velocity field which admits a unique weak solution). Differentiating the relative entropy with respect to its second argument and using the invariance of the relative entropy under diffeomorphisms, it holds, for $t\\geq 0$,\n\\begin{align*}\n\\frac1{4\\alpha} \\frac{d}{dt} \\Hh(\\nu^\\epsilon_t, \\nu_t) &= \\int_{\\Theta} J'_{\\nu_t} \\d (\\nu^\\epsilon_t-\\nu_t) \\\\\n& = \\int_{\\Theta} J'_{\\nu_t} \\d(\\nu^\\star -\\nu_t) + \\int_{\\Theta} J'_{\\nu_t} \\d(\\nu^\\epsilon_t-\\nu^\\star) \\\\\n& \\leq -(J(\\nu_t)-J^\\star) + \\Vert J'_{\\nu_t}\\Vert_\\BL \\cdot \\Vert \\nu^\\star-\\nu^\\epsilon_t\\Vert_\\BL^*\n\\end{align*}\nwhere the first term comes from the convexity of $J$ and the second from the definition of $\\Vert \\cdot\\Vert_{\\BL}^*$. After integrating in time and rearranging the terms we get\n\\begin{align*}\n\\left(\\frac1t\\int_0^t J(\\nu_s)\\d s\\right)-J^\\star &\\leq \\frac1{4\\alpha t} \\left( \\Hh(\\nu^\\epsilon_0,\\nu_0) - \\Hh(\\nu^\\epsilon_t,\\nu_t) \\right) + \\frac{B_{\\nu_0}}{t} \\int_0^t \\Vert \\nu^\\star-\\nu^\\epsilon_s\\Vert_{\\BL}^* \\d s.\n\\end{align*}\nFor the last integral term, we use the triangular inequality\n\\[\n \\frac{B_{\\nu_0}}{t}  \\int_0^t \\Vert \\nu^\\star-\\nu^\\epsilon_s\\Vert_{\\BL}^* \\d s \\leq B_{\\nu_0}\\Vert \\nu^\\star - \\nu^\\epsilon_0\\Vert_{\\BL}^* + \\frac{B_{\\nu_0}}{t} \\int_0^t \\Vert \\nu_0^\\epsilon -\\nu^\\epsilon_s\\Vert_{\\BL}^*\\d s\\leq B_{\\nu_0} \\Vert \\nu^\\star - \\nu^\\epsilon_0\\Vert_{\\BL}^* +B_{\\nu_0}^2\\beta t\n\\]\nwhere the last term is obtained by bounding the integrated flow of the velocity field $(\\nabla J'_{\\nu_t})_{t\\geq 0}$. Since $\\Hh(\\nu_\\epsilon,\\nu_t)\\geq 0$ and $J(\\nu_s)$ is decreasing, it follows\n\\[\nJ(\\nu_t)-J^\\star  \\leq \\inf_{\\nu^\\epsilon\\in \\Mm_+(\\Theta)} \\left( \\frac1{4\\alpha t} \\Hh(\\nu^\\epsilon,\\nu_0) + B_{\\nu_0}\\Vert  \\nu^\\star -\\nu^\\epsilon\\Vert_{\\BL}^*\\right) + B_{\\nu_0}^2 \\beta t.\\qedhere\n\\]\n\\end{proof}\n\n\\begin{proof}[Proof of Theorem~\\ref{thm:convergenceflow} (gradient flow)]\nBy Lemma~\\ref{lem:qdensity}, we have for $\\tau\\geq L=\\Lip(\\log \\rho)$, by writing $\\bar \\Hh \\coloneqq \\bar \\Hh(\\nu^\\star,\\rho\\vol)$,\n\\[\n\\Qq_{\\nu^\\star,\\hat \\nu_0}(\\tau) \\leq \\frac{\\bar \\Hh+d\\cdot \\nu^\\star(\\Theta)\\cdot (\\log(\\tau)+C_\\Theta)}\\tau + \\nu^\\star(\\Theta)\\cdot W_\\infty(\\nu_0,\\rho\\vol).\n\\]\nCombining this bound with Lemma~\\ref{lem:mirrordecrease}, we get that for $t\\geq L/(4\\alpha B_{\\nu_0})$,\n\\[\nJ(\\nu_t)-J^\\star \\leq \\frac{\\bar \\Hh+d\\cdot\\nu^\\star(\\Theta)\\cdot(\\log(4\\alpha B_{\\nu_0} t)+C_\\Theta)}{4\\alpha B_{\\nu_0} t} + B_{\\nu_0}\\cdot\\nu^\\star(\\Theta)\\cdot W_\\infty(\\nu_0,\\rho\\vol)+\\beta B^2_{\\nu_0} t.\n\\]\nIn particular, for $t=(\\alpha\\beta)^{-\\frac12}$, we get \n\\begin{equation}\\label{eq:lastbeforelast}\nJ(\\nu_t)-J^\\star \\leq \\sqrt{\\frac{\\beta}{\\alpha}}\\left(\\bar \\Hh + d\\cdot \\nu^\\star(\\Theta) (\\log(4B_{\\nu_0}\\sqrt{\\alpha/\\beta})+C_\\Theta) + B^2_{\\nu_0}\\right) + B_{\\nu_0}\\nu^\\star(\\Theta)\\cdot W_\\infty(\\nu_0,\\rho\\vol).\n\\end{equation}\nSince this is valid only when $t\\geq L/(4\\alpha B_{\\nu_0})$, we require $(\\alpha\\beta)^{-\\frac12}\\geq L/(4\\alpha B_{\\nu_0})$ which leads to the first condition on $\\beta/\\alpha$. Now, we want the right-hand side of~\\eqref{eq:lastbeforelast} to be smaller than $\\Delta_0 \\coloneqq J_0 - J^\\star$ so that we can conclude with Corollary~\\ref{cor:localGF}. To this end, we require, on the one hand $W_\\infty(\\nu_0,\\rho\\vol) \\leq \\Delta_0/(2B_{\\nu_0}\\nu^\\star(\\Theta))$. On the other hand, we use the bound $\\log(u)\\leq C_\\epsilon u^\\epsilon$ for $\\epsilon\\in {]0,1/2]}$, require $4B_{\\nu_0}\\sqrt{\\alpha/\\beta}\\geq 1$ and obtain the condition\n\\[\n(4B_{\\nu_0})^\\epsilon(\\beta/\\alpha)^{(1-\\epsilon)/2}\\left( \\bar \\Hh + d\\cdot \\nu^\\star(\\Theta)(C_\\Theta+C_\\epsilon) +B^2_{\\nu_0}\\right)\\leq \\frac12\\Delta_0.\n\\]\nThis leads to the second condition on $\\beta/\\alpha$ is the theorem.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:distanceinequality}\nIt holds $\\widehat W_2(\\nu,\\nu^\\star)\\leq W_\\tau(\\nu)(1+O(\\tau^2)+O(W_\\tau(\\nu)^2))$.\n\\end{lemma}\n\n\\begin{proof}\nNote that for $W_\\tau(\\nu)$ small enough, it holds $\\nu(\\Theta_i)>0$ for $i\\in \\{1,\\dots,m^\\star\\}$. Let $\\mu \\in \\Pp_2(\\Omega)$ be such that $\\h\\mu = \\nu$ and consider the transport map $T:\\Omega\\to\\Omega$ defined as\n \\[\n T(r,\\theta) =\n \\begin{cases}\n(r\\frac{r_i}{\\bar r_i}, \\theta_i) &\\text{if $\\theta \\in \\Theta_i$ and $\\bar r_i>0$, $i\\in \\{1,\\dots,m^\\star\\}$},\\\\\n(0,\\theta) &\\text{otherwise}.\n \\end{cases}\n \\]\nBy construction, it holds $\\h(T_\\#\\mu) =  \\nu^\\star$. Let us estimate the transport cost associated to this map \n\\[\n\\mathcal{T} = \\int_\\Omega \\dist((r,\\theta),T(r,\\theta))^2\\d\\mu(r,\\theta).\n\\]\nThe geodesic distance associated to the cone metric is\n\\begin{align*}\n\\dist((r_1,\\theta_1),(r_2,\\theta_2))^2 &= r_1^2 + r_2^2 - 2r_1r_2\\cos(\\min\\{\\dist(\\theta_1,\\theta_2),\\pi\\})\\\\\n&= (r_1-r_2)^2 +r_1r_2\\dist(\\theta_1,\\theta_2)^2 +O(r_1r_2\\dist(\\theta_1,\\theta_2)^4)\n\\end{align*}\nNow, if we only consider points $\\theta\\in \\Theta_i$ with $\\tilde \\theta$ their coordinates in a normal frame centered at $\\theta_i$ (note that in all other proofs, we do not need to distinguish between $\\theta$ and $\\tilde \\theta$), we have the approximation\n\\begin{align*}\n\\dist((r_1,\\theta_1),(r_2,\\theta_2))^2 = (r_1-r_2)^2 +r_1r_2\\Vert \\tilde \\theta_1 - \\tilde \\theta_2\\Vert^2(1+O(\\tau^2)).\n\\end{align*}\nLet us decompose $T(r,\\theta)$ as $(rT^r(\\theta),T^\\theta(\\theta))$ and estimate the two contributions forming $\\Tt$ separately. On the one hand, we have\n\\begin{align*}\n\\int (rT^r(\\theta)-r)^2\\d\\mu(r,\\theta) &= \\bar r_0^2 + \\sum_{i=1}^{m^\\star} (\\bar r_i-r_i)^2 = \\bar r_0^2 +\\Vert b^r\\Vert^2(1+O(W_\\tau(\\nu)^2)).\n\\end{align*}\nOn the other hand, we have\n\\begin{align*}\n\\int r^2T^r(\\theta)\\Vert T^\\theta(\\theta)-\\theta\\Vert^2\\d\\mu(r,\\theta) &=\\sum_{i=1}^{m^\\star} \\bar r_ir_i (\\tr\\Sigma_i + \\Vert \\delta \\theta_i\\Vert^2)=(\\Vert s\\Vert^2 + \\Vert b^\\theta\\Vert^2)(1+O(W_\\tau(\\nu)^2)).\n\\end{align*}\nAs a consequence, we have $\\Tt = W_\\tau(\\nu)(1 + O(W_\\tau(\\nu)^2)+ O(\\tau^2))$. Remark that this estimate does not depend on the chosen lifting $\\mu$ satisfying $\\h\\mu=\\nu$. We then conclude by using the characterization in~\\cite[\nThm. 7.20]{liero2018optimal} for the distance $\\widehat W_2$:\n\\[\n\\widehat W_2(\\nu_1,\\nu_2) = \\min \\left\\{W_2(\\mu_1,\\mu_2) \\;;\\; (\\h\\mu_1,\\h\\mu_2) = (\\nu_1,\\nu_2)\\right\\}.\n\\]\nThus $\\widehat W_2(\\nu,\\nu^\\star)^2 \\leq W_2(\\mu,T_\\#(\\mu))^2\\leq \\Tt$, and the result follows.\n\\end{proof}\n\n\\begin{lemma}[Expansion around $\\nu^\\star$]\\label{lem:expansion}\n Let $\\psi$ be any (vector or real-valued) smooth function on $\\Theta$ and $\\nu\\in \\Mm_+(\\Theta)$. If $\\tau>0$ is an admissible radius, then the following first and second-order expansions hold\n \\begin{align*}\n \\int \\psi \\d(\\nu - \\nu^\\star) \n &= \\sum_{i=1}^m r_i \\bar \\nabla \\psi(\\theta_i)^\\intercal b_i \n + \\int_{\\Theta_0} \\psi \\d\\nu \n +\\sum_{i=1}^m \\int_{\\Theta_i} M_{2,\\psi}(\\theta_i,\\theta)\\d\\nu(\\theta)\\\\\n &=\\sum_{i=1}^m r_i\\bar \\nabla \\psi(\\theta_i)^\\intercal b_i \n +\\frac12  \\sum_{i=1}^m \\bar r_i^2  \\left( \\tr\\left(\\nabla^2\\psi(\\theta_i) \\Sigma_i \\right) \n + \\delta \\theta_i^\\intercal \\nabla^2\\psi(\\theta_i) \\delta \\theta_i \\right)\\\\\n &\\qquad + \\int_{\\Theta_0}  \\psi \\d\\nu +\\sum_{i=1}^m \\int_{\\Theta_i} M_{3,\\psi}(\\theta_i,\\theta)\\d\\nu(\\theta)\n \\end{align*}\n where $M_{k,\\psi}(\\theta_i,\\theta)$ is the remainder in the $k-1$-th order Taylor expansion of $\\psi$ around $\\theta_i$ in local coordinates (and we recall that $\\bar \\nabla \\psi := (2\\psi,\\nabla \\psi)$).\n \\end{lemma}\n\n\\begin{proof}\nBy a Taylor expansion of $\\psi$ around $\\theta_i$ for $i\\in \\{1,\\dots,m^\\star\\}$, it holds\n\\begin{align*}\n \\int_{\\Theta_i} \\psi \\d\\nu= \\int_{\\Theta_i} \\left( \\psi(\\theta_i) + \\nabla\\psi(\\theta_i)^\\intercal (\\theta-\\theta_i) + (\\theta-\\theta_i) ^\\intercal \\nabla^2\\psi(\\theta_i)(\\theta-\\theta_i) + M_{3,\\psi}(\\theta_i,\\theta)\\right)\\d\\nu(\\theta)\n\\end{align*}\nand substracting $\\int_{\\Theta_i}\\psi\\d\\nu^\\star = r_i^2\\phi(\\theta_i)$ yields\n\\begin{multline*}\n \\int_{\\Theta_i} \\psi \\d(\\nu-\\nu^\\star)= (\\bar r_i^2 -r_i^2)\\psi(\\theta_i) +  \\bar r_i^2 \\nabla \\psi(\\theta_i)^\\intercal \\delta \\theta_i \\\\ \n +\\frac12  \\sum_{i=1}^m \\bar r_i^2  \\left( \\tr\\left(\\nabla^2\\psi(\\theta_i) \\Sigma_i \\right) \n + \\delta \\theta_i^\\intercal \\nabla^2\\psi(\\theta_i) \\delta \\theta_i \\right)  +\\sum_{i=1}^m \\int_{\\Theta_i} M_{3,\\psi}(\\theta_i,\\theta)\\d\\nu(\\theta)\n \\end{multline*}\nwhere  we have used a bias-variance decomposition for the quadratic term. The result follows by summing the integrals over each $\\Theta_i$ and using the expression of $b$.\n \\end{proof}\n\n\\begin{lemma}[Global distance bound]\\label{lem:distancebound}\nUnder {\\sf (A1-5)}, let $\\tau_{\\adm}$ be an admissible radius $\\tau$ as in Definition~\\ref{def:localfeatures}, fix some $J_{\\max}>0$ and let\n\\[\n\\tau_0 = \\min\\left\\{\\tau_{\\adm}, 2\\sqrt{\\frac{v^\\star}{\\sigma_{\\min}(H)}}, \\frac{3\\sigma_{\\min}(H)}{2 \\Lip (\\nabla^2 J'_{\\nu^\\star})} \\right\\}.\n\\]\nThen there exists $C, C'>0$ such that for all $\\tau\\leq \\tau_0$ and  $\\nu \\in \\Mm_+(\\Theta)$ such that $J(\\nu)\\leq J_{\\max}$, it holds \n\\begin{align*}\nW_\\tau(\\nu) \\leq \\frac{C}{\\tau^2}(J(\\nu)-J^\\star)^\\frac12 &&\\text{and}&&\\Vert \\tilde b^\\theta\\Vert^2+\\Vert s\\Vert^2\\leq C'(J(\\nu)-J^\\star). \n\\end{align*}\n\\end{lemma}\n\n\\begin{proof}\nLet us write $f_\\nu \\coloneqq \\int \\phi\\d\\nu$ and $f^\\star = \\int \\phi \\d\\nu^\\star$. By strong convexity of $R$ at $f^\\star$, and optimality of $\\mu^\\star$, there exists $C>0$ such that for all $\\nu\\in \\Mm_+(\\Theta)$ it holds\n\\begin{equation}\\label{eq:strongconvexityinequality}\nJ(\\nu)-J^\\star \\geq \\int_{\\Theta} J'_{\\nu^\\star}\\d\\nu+ C\\min\\{ \\Vert f_\\nu-f^\\star\\Vert^2,\\Vert f_\\nu-f^\\star\\Vert\\} .\n\\end{equation}\nTo prove the first claim, we thus have to bound $W_\\tau(\\nu)$ using the terms in the right-hand side of~\\eqref{eq:strongconvexityinequality}. \n\n\\paragraph{Step 1.} By a Taylor expansion, one has for $\\theta \\in \\Theta_i$ for $i\\in \\{1,\\dots,m^\\star\\}$,\n\\[\n\\vert J'_{\\nu^\\star}(\\theta) - \\tfrac12 (\\theta -\\theta_i)^\\intercal H_i  (\\theta-\\theta_i) \\vert \\leq \\tfrac16 \\Lip( \\nabla^2 J'_{\\nu^\\star}) \\Vert \\theta-\\theta_i\\Vert^3.\n\\]\nThus, if $\\Vert \\theta - \\theta_i\\Vert \\leq 3\\sigma_{\\min}(H)/(2 \\Lip( \\nabla^2 J'_{\\nu^\\star}))$, then $J'_{\\nu^\\star}(\\theta) \\geq \\frac14 (\\theta -\\theta_i)^\\intercal H_i  (\\theta-\\theta_i)$ for $\\theta\\in \\Theta_i$. Decomposing the integral of this quadratic term into bias and variance, we get\n\\begin{align*}\\label{eq:biasvariance}\n\\int_{\\Theta_i} (\\theta -\\theta_i)^\\intercal H_i (\\theta-\\theta_i)\\d\\nu(\\theta) \n&= \\bar r_i^2 \\left(\\delta \\theta_i^\\intercal  H_i \\delta \\theta_i + \\tr(\\Sigma_i H_i )\\right)\n\\end{align*}\nand we deduce a first bound by summing the terms for $i\\in \\{1,\\dots,m^\\star\\}$,\n\\[\n\\int_{\\Theta\\setminus \\Theta_0} J'_{\\nu^\\star} \\d\\nu \\geq \\frac{\\sigma_{\\min}(H)}4(\\Vert \\tilde b^\\theta\\Vert^2 + \\Vert s\\Vert^2).\n\\]\n\n\\paragraph{Step 2.} In order to lower bound the integral over $\\Theta_0$, we first derive a lower bound for $J'_{\\nu^\\star}$ on $\\Theta_0$. This is a continuously differentiable and nonnegative function on a closed domain $\\Theta_0$ so its minimum is attained either at a local minima in the interior of $\\Theta_0$ or on its boundary. Using the quadratic lower bound from the previous paragraph, it follows that for $\\theta\\in \\Theta_0$,\n\\[\nJ'_{\\nu^\\star}(\\theta)\\geq \\min \\{ v^\\star,  \\tau^2 \\sigma_{\\min}(H)/4\\}.\n\\]\nThus, if we also assume that $\\tau\\leq 2\\sqrt{v^\\star/\\sigma_{\\min}(H)}$ then $J'_{\\nu^\\star}(\\theta)\\geq \\tau^2 \\sigma_{\\min}(H)/4$ for $\\theta\\in \\Theta_0$ and it follows that\n\\[\n\\int_{\\Theta_0} J'_{\\nu^\\star} \\d\\nu \\geq \\frac{\\sigma_{\\min}(H)}{4} \\tau^2\\bar r_0^2 .\n\\]\nUsing inequality~\\eqref{eq:strongconvexityinequality}  we have shown so far that\n\\begin{equation}\\label{eq:localcontrol}\n\\tilde W_\\tau(\\nu)^2\\coloneqq  \\bar r_0^2 +\\Vert \\tilde b^\\theta\\Vert^2 +\\Vert s\\Vert^2 \\leq \\frac{C}{\\tau^2} (J(\\mu)-J^\\star).\n\\end{equation}\nNotice that $\\tilde W_\\tau(\\nu)$ is similar to $W_\\tau(\\nu)$ but it does not contain the terms controlling the deviations of mass $\\vert \\bar r_i-r_i\\vert$. These quantities can be controlled by using the coercivity of $R$, i.e. the last term in~\\eqref{eq:strongconvexityinequality}, as we do now.\n\n\\paragraph{Step 3.}\nUsing the first order expansion of Lemma~\\ref{lem:expansion} then squaring gives\n\\begin{align*}\n\\left\\vert \\Vert f_\\nu-f^\\star\\Vert^2 - \\frac12 b^\\intercal K b \\right\\vert\n& \\leq C \\left( \\Vert b\\Vert \\tilde W_\\tau(\\nu)^2 +  \\tilde W_\\tau(\\nu)^4 \\right).\n\\end{align*}\n%\nSince we have assumed that $K$ is positive definite, it follows\n\\[\n\\Vert f_\\nu-f^\\star\\Vert^2 \\geq C \\Vert b\\Vert^2 - C \\tilde W_\\tau(\\nu)^2 \\Vert b\\Vert - C\\tilde W_\\tau(\\nu)^4\n\\]\nand thus, after rearranging the terms\n\\[\n(\\Vert b\\Vert - C\\tilde W_\\tau(\\nu)^2)^2 \\leq  C \\Vert f_\\nu-f^\\star\\Vert^2 + C\\tilde W_\\tau(\\nu)^4.\n\\]\nIt follows that $\\Vert b\\Vert \\leq  C\\Vert f_\\nu-f^\\star\\Vert + C\\tilde W_\\tau(\\nu)^2$. Also, by inequality~\\eqref{eq:strongconvexityinequality}, if $J(\\nu)\\leq J_{\\max}$, then $\\Vert f_\\nu -f^\\star\\Vert^2\\leq C(J(\\nu)-J^\\star)$.  Moreover, by inequality~\\eqref{eq:localcontrol}, we get\n\\[\n\\Vert b\\Vert \\leq \\frac{C}{\\tau^2}(J(\\nu)-J^\\star) +C (J(\\nu)-J^\\star)^\\frac12 \\leq \\frac{C}{\\tau^2} (J(\\nu)-J^\\star)^\\frac12.\n\\]\nWe finally combine with the bound on $\\tilde W_\\tau(\\nu)$ to conclude since $W_\\tau(\\nu)^2\\leq \\tilde W_\\tau(\\nu)^2+\\Vert b\\Vert^2$\n\\end{proof}\n\n\\subsection{Proof of the distance inequality (Proposition~\\ref{prop:distanceinequality})}\\label{app:distanceinequality}\nBy Lemma~\\ref{lem:distanceinequality}, it holds \n\\[\n\\widehat W_2(\\nu,\\nu^\\star)\\leq W_\\tau(\\nu)(1+O(\\tau^2)+O(W_\\tau(\\nu)^2)).\n\\]\nMoreover, by Lemma~\\ref{lem:distancebound}, there exists $\\tau_0>0$ and $C>0$ such that \n\\[\nW_\\tau(\\nu)\\leq \\frac{C}{\\tau_0^2}(J(\\nu)-J^\\star)^\\frac12.\n\\]\nCombining these two lemmas, it follows that for some $C'>0$, we have \n\\[\n\\widehat W_2(\\nu,\\nu^\\star)^2 \\leq C'(J(\\nu)-J^\\star)^\\frac12.\n\\]\n This also implies a control on the Bounded-Lipschitz distance since it holds $(\\Vert \\nu - \\nu^\\star\\Vert_{\\BL}^*)^2\\leq (2+\\pi^2/2)(\\nu(\\Theta)+\\nu^\\star(\\Theta))\\widehat W_2(\\nu,\\nu^\\star)^2$, see~\\cite[Prop. 7.18]{liero2018optimal}.\n \n \n \n\n\\begin{proposition}[Local expansion]\\label{prop:localexpansion}\n It holds\n \\[\n J(\\nu)-J^\\star = \\frac12 b^\\intercal K b + \\frac12 \\sum_{i=1}^m \\bar r_i^2 \\left( \\tr(\\Sigma_i H_i) +\\delta \\theta^\\intercal H_i \\delta \\theta_i \\right) + \\int_{\\Theta_0} J'_{\\nu^\\star} \\d\\nu + \\mathop{\\mathrm{err}}(\\tau,\\nu)\n \\]\n where $\\mathop{\\mathrm{err}}(\\tau,\\nu) =O( \\tau (\\Vert \\tilde b^\\theta \\Vert^2 + \\Vert s\\Vert^2) + W_\\tau(\\nu)^3)$. In particular, if $\\tau$ is fixed small enough,\n \\[\n J(\\nu)-J^\\star \\leq  \\sigma_{\\max}(K) \\Vert b\\Vert^2 + \\sigma_{\\max}(H)(\\Vert b^\\theta\\Vert^2 +\\Vert s\\Vert^2) +\\Vert J'_{\\nu^\\star}\\Vert_\\infty \\bar r_0^2 +O(W_\\tau(\\nu)^3).\n \\]\n \\end{proposition}\n\n\\begin{proof} Let us write $f_\\nu \\coloneqq \\int \\phi\\d\\nu$ and $f^\\star = \\int \\phi \\d\\nu^\\star$. By a second order Taylor expansion of $R$ around $f^\\star$, we have\n \\[\n J(\\nu) - J^\\star = \\int_{\\Theta} J'_{\\nu^\\star} \\d\\nu+ \\frac12 \\Vert f_\\nu-f^\\star\\Vert^2_\\star + O(\\Vert f_\\nu-f^\\star\\Vert^3).\n \\]\n Using the first order expansion of Lemma~\\ref{lem:expansion} for $\\phi$, we get \n $\n \\Vert f_\\nu-f^\\star\\Vert^2_\\star %= b^\\intercal Kb + O\\left( (r_0^2 + \\sum_{i=1}^m s_i^2)(\\Vert b\\Vert + r_0^2 + \\sum_{i=1}^m s_i^2)\\right) \n = b^\\intercal Kb + O(W_{\\tau}(\\nu)^3)\n $. Also, using the second order expansion of Lemma~\\ref{lem:expansion} for $J'_{\\nu^\\star}$ and using the fact that $J'_{\\nu^\\star}$ and its gradient vanish for all $\\theta_i$, we get \n \\[\n \\int_{\\Theta} J'_{\\nu^\\star} \\d\\nu = \\frac12 \\sum_{i=1}^m \\bar r_i^2 \\left( \\tr(\\Sigma_i H_i) +\\delta \\theta^\\intercal H_i \\delta \\theta_i \\right) + \\int_{\\Theta_0}J'_{\\nu^\\star} \\d\\nu  + O(\\tau (\\Vert s\\Vert^2+\\Vert  b^\\theta \\Vert^2))\n \\]\nand the expansion follows. Notice also that in the expression of $J(\\nu)$, $\\bar r_i$ and $r_i$ are interchangeable up to introducing higher order error, since $\\vert r_i - \\bar r_i\\vert = O(\\vert b^r_i\\vert)$ (and also $\\Vert \\tilde b^\\theta\\Vert = \\Vert b^\\theta\\Vert(1+O(W_\\tau(\\nu)))$).\n \\end{proof}\n\n\\begin{proposition}[Gradient estimate]\\label{prop:gradientestimate}\nFor $\\nu \\in \\Pp_2(\\Omega)$, it holds \n\\[\n\\Vert g_\\nu\\Vert^2_{L^2(\\nu)} =  b^\\intercal (K + H)^2b+ \\sum_{i=1}^m \\bar r_i^2 \\tr (\\Sigma_i H_i^2) +\\Vert g_\\nu\\Vert^2_{L^2(\\nu\\vert_{\\Theta_0})}+\\mathop{\\mathrm{err}}(\\tau,\\nu)\n\\]\nwhere $\\mathop{\\mathrm{err}}(\\tau,\\nu)\\lesssim \\tau(\\Vert \\tilde b^\\theta\\Vert^2 +\\Vert s\\Vert^2) +W_\\tau(\\nu)^3$. In particular, if $\\tau$ is fixed small enough\n\\[\n\\Vert g_\\nu\\Vert^2_{L^2(\\nu)}  \\geq \\frac12 (\\sigma_{\\min}(K)+ \\sigma_{\\min}(H))^2\\Vert b\\Vert^2 + \\frac12 \\sigma_{\\min}(H)^2 \\Vert s\\Vert^2 +  \\frac14 \\bar r_0^2 \\sigma_{\\min}(H)^2 \\tau^4 +O(W_\\tau(\\nu)^3).\n\\]\n\\end{proposition}\n\n\\begin{proof}\nFor this proof, we write $f_\\nu - f^\\star = \\delta f_0 +\\delta f_b + \\delta f_{\\mathrm{err}}$ where\n\\begin{align*}\n \\delta f_0 &\\coloneqq  \\int_{\\Theta_0}\\phi(\\theta)\\d\\nu(\\theta),&\n \\delta f_b & \\coloneqq  \\sum_{i=1}^m r_i \\bar\\nabla \\phi(\\theta_i) b_{i},&\n\\delta f_{\\mathrm{err}} &\\coloneqq \\sum_{i=1}^{m} \\int_{\\Theta_i} M_{\\phi,2}(\\theta_i,\\theta)\\d\\nu(\\theta).\n\\end{align*}\nwhere the decomposition follows from Lemma~\\ref{lem:expansion}.\n%\nThe expression for the norm of the gradient is as follows:\n\\[\n\\Vert g_\\nu\\Vert^2_{L^2(\\nu)} = \\int_{\\Theta}\\Vert\\bar \\nabla J'_\\nu(\\theta) \\Vert^2\\d\\nu(\\theta)\n\\]\nwhere $\\bar \\nabla J = (2J,\\nabla J)$. We start with the following decomposition for $\\theta \\in \\Theta_i$ (recall that $J'_\\nu(\\theta) =\\langle \\phi(\\theta),\\nabla R(\\int \\phi \\d\\nu)\\rangle +\\lambda$):\n\\begin{multline*}\nJ'_\\nu(\\theta) = \\lambda + \\left\\langle \\phi(\\theta_i) + (\\theta-\\theta_i)^\\intercal \\nabla \\phi(\\theta_i) + \\frac12  (\\theta-\\theta_i)^\\intercal\\nabla^2\\phi(\\theta_i)(\\theta-\\theta_i)+  M_{\\phi,3}(\\theta_i,\\theta),\\nabla R(f^\\star)\\right\\rangle \\\\\n+\\langle \\phi(\\theta_i)+ (\\theta-\\theta_i)^\\intercal \\nabla \\phi(\\theta_i) + M_{\\phi,2}(\\theta_i,\\theta),f_\\mu-f^\\star\\rangle_{\\star}+ \\langle \\phi(\\theta),M_{\\nabla R,2}(f^\\star,f)\\rangle\n\\end{multline*}\nHere we use the notation $\\langle\\cdot,\\cdot\\rangle_\\star$ to denote the quadratic form associated to $\\nabla^2 R(f^\\star)$.\nThanks to the optimality conditions  $\\bar \\nabla J'_{\\nu^\\star}(\\theta_i)=0$ for $i\\in \\{1,\\dots,m\\}$, we get\n\\begin{align*}\n\\bar \\nabla_j J'_\\nu(\\theta) &=  [H_i(\\theta-\\theta_i)]_j + \\langle \\bar \\nabla_j \\phi(\\theta_i),\\delta f_0+\\delta f_b \\rangle_\\star + [N(\\theta_i,\\theta)]_j\\\\\n &=  [H_i(\\theta-\\bar \\theta_i)]_j+ \\left( [H_i(\\bar \\theta_i-\\theta_i)]_j+ \\langle \\bar \\nabla_j \\phi(\\theta_i),\\delta f_b \\rangle_\\star\\right) + \\langle \\bar \\nabla_j  \\phi(\\theta_i),\\delta f_0\\rangle_\\star + [N(\\theta_i,\\theta)]_j\n\\end{align*}\nwhere $N$ collects the higher order terms and is defined as\n\\begin{multline*}\n [N(\\theta_i,\\theta)]_j =  \\langle \\bar \\nabla_j M_{\\phi,3}(\\theta_i,\\theta),\\nabla R(f^\\star)\\rangle %+ \\langle \\partial_j \\phi(\\theta_i),N_1\\rangle \\\\\n +\\langle \\bar \\nabla_j M_{\\phi,2}(\\theta_i,\\theta),f-f^\\star\\rangle_{\\star} + \\langle \\bar \\nabla_j \\phi(\\theta),M_{\\nabla R,2}(f^\\star,f)\\rangle\n\\end{multline*}\nwhere $\\Vert \\bar \\nabla_j M_{\\phi,3}(\\theta_i,\\theta)\\Vert = O(\\Vert \\theta-\\theta_i\\Vert^2)$ if $j>0$ and $O(\\Vert \\theta -\\theta_i\\Vert^3)$ if $j=0$.\nExpanding the square gives the following ten terms:\n\\begin{align}\n\\int_{\\Omega\\setminus \\Omega_0} \\Vert\\bar \\nabla J'_\\nu \\Vert^2\\d\\nu(\\theta)\n&= \\sum_{i=1}^{m} \\int_{\\Theta_i} \\sum_{j=0}^{d} [H_i(\\theta-\\bar \\theta_i)]_j^2 \\d\\nu \\tag{I} \\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\sum_{j=0}^d \\left( \\langle\\bar \\nabla_j\\phi(\\theta_i),\\delta f_b \\rangle_\\star + [H_i(\\bar \\theta_i-\\theta_i)]_j \\right)^2 \\d\\nu(\\theta) \\tag{II} \\\\\n&+ \\sum_{i=1}^{m} \\int_{\\Theta_i} \\sum_{j=0}^d \\langle \\bar \\nabla_j\\phi(\\theta_i), \\delta f_0 \\rangle_\\star^2\\d\\nu(\\theta)  \\tag{III} \\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d  [H_i(\\theta-\\bar \\theta_i)]_j\\cdot ( \\langle\\bar \\nabla_j \\phi(\\theta_i),\\delta f_b\\rangle_\\star + [H_i(\\bar \\theta_i-\\theta_i)]_j) \\d\\nu(\\theta) \\tag{IV} \\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d [H_i(\\theta-\\bar \\theta_i)]_j\\cdot \\langle \\bar \\nabla_j\\phi(\\theta_i), \\delta f_0\\rangle_\\star\\d\\nu(\\theta) \\tag{V}\\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d (\\langle\\bar \\nabla_j \\phi(\\theta_i),\\delta f_b\\rangle_\\star + [H_i(\\bar \\theta_i-\\theta_i)]_j)\\cdot \\langle \\bar\\nabla_j \\phi(\\theta_i), \\delta f_0\\rangle_\\star\\d\\nu(\\theta)\\!\\! \\tag{VI}\\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d [N(\\theta_i,\\theta)]_j\\cdot  [H_i(\\theta-\\bar \\theta_i)]_j\\d\\nu(\\theta) \\tag{VII}\\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d [N(\\theta_i,\\theta)]_j\\cdot  (\\langle\\bar \\nabla_j \\phi(\\theta_i),\\delta f_b\\rangle_\\star + [H_i(\\bar \\theta_i-\\theta_i)]_j)\\d\\nu(\\theta) \\tag{VIII}\\\\\n&+\\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d [N(\\theta_i,\\theta)]_j\\cdot \\langle \\bar \\nabla_j \\phi(\\theta_i), \\delta f_0 \\rangle_\\star\\d\\nu(\\theta) \\tag{IX}\\\\\n&+ \\sum_{i=1}^{m} \\int_{\\Theta_i} \\sum_{j=0}^d [N(\\theta_i,\\theta)]_j^2 \\d\\nu(\\theta) \\tag{X}\n\\end{align}\nTerms (I) to (II) are the main terms in the expansion, while the other terms are higher order. The term (I) is a local curvature term and can be expressed as $\\mathrm{(I)} = \\sum_{i=1}^m \\bar r_i^2 \\tr \\Sigma_i H_i^2$. \n%\nThe term (II) is a global interaction term that writes\n\\begin{align*}\n\\mathrm{(II)} &=\n\\sum_{i=1}^m \\bar r_i^2 \\sum_{j=0}^d \\vert \\langle\\bar \\nabla_j\\phi(\\theta_i),\\delta f_b\\rangle_\\star + [H_i(\\bar \\theta_i-\\theta_i)]_j \\vert^2 \\\\\n&= \\sum_{i=1}^m\\sum_{j=0}^d \\left\\vert \\sum_{i'=1}^m \\sum_{j'=0}^d (\\langle \\bar r_i \\bar \\nabla_j\\phi(\\theta_i),r_{i'}\\bar \\nabla_{j'}\\phi(\\theta_{i'})\\rangle_\\star + \\bar H_{(i,j),(i,j')})(b_{i',j'}) \\right\\vert^2\\\\\n&= \\Vert (\\bar K + H)(b)\\Vert^2\n\\end{align*}\nwhere the entries of $\\bar K$ and $\\bar H$ differ from those of $K$ and $H$ by a factor $\\bar r_i/r_i$. More precisely,\n\\[\n[\\bar K-K]_{(i,j),(i',j')} = (\\bar r_i/r_i-1)K_{(i,j),(i',j')}\n\\]\nand similarly for $\\bar H- H$. Since $\\vert \\bar r_i/r_i-1\\vert = O(\\vert b^r_i\\vert)$ we have $\\sigma_{\\max}(\\bar K-K )=O(W_\\tau(\\nu))$. It follows, by expanding the square, that \n\\[\n\\Vert (\\bar K + \\bar H)(b)\\Vert^2 = \\Vert (K + H)(b)\\Vert^2 + O(W_\\tau(\\nu)^3).\n\\]\n\nThe remaining terms are error terms, that we estimate directly in terms of $W_\\tau(\\nu)$ and $\\tau$. We use in particular the fact that by H\\\"older's inequality, $\\int_{\\Theta_i} \\Vert \\theta - \\bar \\theta_i\\Vert\\d\\nu(\\theta) = O(\\bar r_i^2 \\tr \\Sigma_i^{\\frac12})$. One has \n\\begin{itemize}\n\\item $\\mathrm{(III)} =O\\left(\\sum_{i=1}^m \\bar r_i^2 \\bar r_0^4 \\right)= O(W_{\\tau}^4(\\nu))$;\n\\item $\\mathrm{(IV)} = \\mathrm{(V)} = 0$ because the integral of the terms $H_i(\\theta-\\bar \\theta_i)$ vanishes;\n%\\item $\\mathrm{(V)} = O(\\sum \\bar r_i^2 \\sigma_i \\bar r_0^2) = O( W_{\\tau}^3)$;\n\\item $\\mathrm{(VI)} = O\\left((\\sum_{i=1}^m \\bar r_i^2 (\\Vert b\\Vert +\\Vert \\delta \\theta_i\\Vert)\\cdot \\bar r_0^2\\right)=O(W_{\\tau}^3(\\nu))$;\n\\item $\\mathrm{(VII)} = O(\\tau (\\Vert \\tilde b^\\theta\\Vert^2 + \\Vert s\\Vert^2)) + O(W_{\\tau}(\\nu)^3)$;\n\\item $\\mathrm{(VIII)} = O(W_{\\tau}^3(\\nu))$;\n\\item $\\mathrm{(IX)} = O(W_{\\tau}^4(\\nu))$;\n\\item $\\mathrm{(X)} = O(\\tau^2(\\Vert \\tilde b^\\theta\\Vert^2+\\Vert s\\Vert^2))+O(W_{\\tau}(\\nu)^4)$.\n\\end{itemize}\nIt follows that overall, the error term is in $O(\\tau (\\Vert \\tilde b^\\theta\\Vert^2+\\Vert s\\Vert^2)+W_{\\tau}(\\nu)^3)$.\nThere remains to lower bound the norm of the gradient over $\\Theta_0$, which can be done as follows. As seen in the proof of Lemma~\\ref{lem:distancebound}, if $\\tau$ is small enough then $J'_{\\nu^\\star}(\\theta)\\geq \\tau^2\\sigma_{\\min}(H)/4$ for $\\theta \\in \\Theta_0$. Considering only the first component of the gradient, it holds\n\\[\n\\int_{\\Theta_0} \\Vert\\bar \\nabla J'_\\nu(\\theta) \\Vert^2\\d\\nu(\\theta) \\geq \\int_{\\Theta_0} 4 \\vert J'_\\nu(\\theta)\\vert^2 \\d\\nu(\\theta).\n\\]\nUsing the expansion $J'_\\nu(\\theta)=J'_{\\nu^\\star}(\\theta)+\\langle \\phi(\\theta),M_{\\nabla R,1}(f^\\star,f_\\nu)\\rangle$, we get\n\\[\n\\int_{\\Theta_0} \\Vert\\bar \\nabla J'_\\nu(\\theta) \\Vert^2\\d\\nu(\\theta) \\geq  C \\bar r_0^2 \\tau^4 + O(W_\\tau(\\nu)^3).\n\\]\nThe result follows by collecting all the estimates above.\n\\end{proof}\n\n\\subsection{Proof of the sharpness inequality (Theorem~\\ref{th:gradientinequality})}\\label{app:conclusionlocal}\nBy Proposition~\\ref{prop:localexpansion} we have that for $\\tau>0$ small enough\n\\[\n J(\\nu)-J^\\star \\leq CW_\\tau(\\nu)^2 + O(W_\\tau(\\nu)^3)\n\\]\nwhere $C = \\sigma_{\\max}(K+H) +\\Vert J'_{\\nu^\\star}\\Vert_\\infty$.\n\nSimilarly, by Proposition~\\ref{prop:gradientestimate}, for $\\tau$ small enough, it holds\n\\[\n\\Vert g_\\nu\\Vert^2_{L^2(\\nu)}  \\geq C'W_\\tau(\\nu)^2 +O(W_\\tau(\\nu)^3)\n\\]\nwhere $C'= \\frac18\\sigma_{\\min}(H)^2\\tau^4$. Now fix $\\tau>0$ satisfying the hypothesis of Lemma~\\ref{lem:distancebound} and the two previous inequalities. By Lemma~\\ref{lem:distancebound},  $W_\\tau(\\nu) = O((J(\\nu)-J^\\star)^\\frac12)$. We deduce that there exists $J_0>J^\\star$ and $\\kappa_0>0$ , such that whenever  $\\nu\\in \\Mm_+(\\Theta)$ satisfies $J(\\nu)<J_0$, one has\n\\[\n\\Vert g_\\nu\\Vert^2_{L^2(\\nu)}\\geq \\kappa_0(J(\\nu)-J^\\star).\n\\]\nFinally, notice that if different metric factors $(\\alpha,\\beta)\\neq (1,1)$ are introduced, one can always lower bound the new gradient squared norm as\n\\[\n\\int_\\Theta \\left( 4 \\alpha \\vert J'_\\nu(\\theta)\\vert^2 +\\beta \\Vert \\nabla J'_\\nu(\\theta)\\Vert^2_\\theta \\right)\\d\\nu(\\theta)\n\\geq \\min\\{\\alpha,\\beta\\} \\int_\\Theta \\left( 4 \\vert J'_\\nu(\\theta)\\vert^2 + \\Vert \\nabla J'_\\nu(\\theta)\\Vert^2_\\theta \\right)\n \\]\n which proves the statement for any $(\\alpha,\\beta)$. Note however that if one wants to make a more quantitative bound, then there are values $(\\alpha_0,\\beta_0)$ that would lead to a better conditioning and potentially higher values for $J_0$. In this case, the factor appearing in the sharpness inequality should rather be $\\min\\{\\alpha/\\alpha_0,\\beta/\\beta_0\\}$.\n\n\n\n\n\n\n\\begin{lemma}\\label{lem:qdensity}\nUnder {\\sf (A1)}, there exists $C_{\\Theta}> 0$ that only depends on the curvature of $\\Theta$, such that for all $\\nu^\\star ,\\nu_0\\in \\Mm_+(\\Theta)$ where $\\nu^\\star=\\sum_{i=1}^{m^\\star} r_i^2\\delta_{\\theta_i}$ and $\\nu_0=\\rho \\vol$ where $\\log \\rho$ is $L$-Lipschitz, then \n\\[\n\\Qq_{\\nu^\\star,\\nu_0}(\\tau)\\leq \\frac1{\\tau} \\left( \\nu^\\star(\\Theta)\\cdot d\\cdot \\big(C_\\Theta +\\log(\\tau)+L/\\tau\\big) +\\nu_0(\\Theta)- \\nu^\\star(\\Theta)+ \\sum_{i=1}^{m^\\star} r_i^2 \\log\\left(\\frac{r_i^2}{\\rho(\\theta_i)}\\right) \\right).\n\\]\nMoreover, for any other $\\hat \\nu_0\\in \\Mm_+(\\Theta)$, it holds\n$\n\\Qq_{\\nu^\\star,\\hat \\nu_0}(\\tau)\\leq \\Qq_{\\nu^\\star,\\nu_0}(\\tau) + \\nu^\\star(\\Theta)\\cdot W_\\infty(\\nu_0,\\hat \\nu_0).\n$\n\\end{lemma}\n\n\\begin{proof}\nLet us build $\\nu_\\epsilon$ in such a way that the quantity defining $\\Qq_{\\nu^\\star,\\nu_0}(\\tau)$ in Eq.~\\eqref{eq:quantization} is small. For this, consider a radius $\\epsilon>0$ and consider the measure $\\nu_\\epsilon$ defined as the normalized volume measure on each geodesic ball of radius $\\tau$ around each $\\theta_i$, with mass $r_i^2$ on this ball, and vanishing everywhere else. Using the transport map that maps these balls to their centers $\\theta_i$, we get if $\\Theta$ is flat,\n\\[\n\\Vert \\nu_\\epsilon -\\nu^\\star\\Vert_\\BL^* \\leq W_1(\\nu_\\epsilon,\\nu^\\star)  \\leq \\sum_{i=1}^{m^\\star} \\frac{r_i^2}{V^{(d)}(\\epsilon)} \\int_0^\\epsilon s \\frac{d}{ds}V^{(d)}(s)\\d s\n\\]\nwhere $V^{(d)}(\\epsilon)$ is the volume of a ball of radius $\\epsilon$ in $\\RR^d$, that scales as $\\epsilon^d$. Using an integration by parts, it follows\n\\[\n\\frac{1}{V^{(d)}(\\epsilon)} \\int_0^\\epsilon s \\frac{d}{ds}V^{(d)}(s)\\d s = \\epsilon - \\int_0^\\epsilon \\frac{V^{(d)}(s)}{V^{(d)}(\\epsilon)}\\d s = \\epsilon -\\int_0^\\epsilon\\left(\\frac{s}{\\epsilon}\\right)^d\\d s = \\frac{\\epsilon d}{d+1},\n\\]\nthus $W_1(\\nu_\\epsilon ,\\nu^\\star)  \\leq \\nu^\\star(\\Theta)\\epsilon$. In the general case where $\\Theta$ is a potentially curved manifold, this upper bound also depends on the curvature of $\\Theta$ around each $\\theta_i$, a dependency that we hide in the multiplicative constant so $W_1(\\nu_\\epsilon ,\\nu^\\star)  \\leq C\\nu^\\star(\\Theta)\\epsilon$. Let us now control the entropy term. Writing $\\rho_\\epsilon =\\d\\nu_\\epsilon/\\d{\\vol}$ and $\\Theta_i$ for the geodesic ball of radius $\\epsilon$ around $\\theta_i$, it holds\n\\begin{align*}\n\\Hh(\\nu_\\epsilon,\\nu_0) & = \\nu_0(\\Theta) -\\nu^\\star(\\Theta) + \\sum_{i=1}^{m^\\star} \\int_{\\Theta_i} \\rho_\\epsilon(\\theta)\\log (\\rho_\\epsilon(\\theta)/\\rho(\\theta))\\d{\\vol}(\\theta).\n\\end{align*}\nThe integral term can be estimated as follows,\n\\begin{align*}\n\\int_{\\Theta_i} \\rho_\\epsilon(\\theta)  \\log (\\rho_\\epsilon(\\theta)/\\rho(\\theta))\\d{\\vol}(\\theta) &= \\int_{\\Theta_i} \\frac{r_i^2}{V^{(d)}(\\epsilon)} \\left(\\log (r_i^2)-\\log V^{(d)}(\\epsilon) -\\log(\\rho(\\theta))\\right)\\d{\\vol}(\\theta)\\\\\n& \\leq r_i^2\\left(\\log(r_i^2)-\\log V^{(d)}(\\epsilon) -\\log \\rho(\\theta_i) +\\Lip(\\log \\rho)\\cdot \\epsilon \\right).\n\\end{align*}\nRecalling that $-\\log V^{(d)}(\\epsilon) \\leq -d\\log(\\epsilon)+C$ for some $C$ that only depends on the curvature of $\\Theta$, we get that the right-hand side of~\\eqref{eq:quantization} is bounded by\n\\begin{align*}\n \\frac1{\\tau} \\left(C\\nu^\\star(\\Theta)+\\nu_0(\\Theta) -\\nu^\\star(\\Theta) d\\log(\\epsilon) + \\sum_{i=1}^{m^\\star} r_i^2 \\log\\left(\\frac{r_i^2}{\\rho(\\theta_i)}\\right) + \\epsilon L\\nu^\\star(\\Theta)\\right) + C\\nu^\\star(\\Theta)\\epsilon.\n\\end{align*}\nLet us fix $\\epsilon>0$ by minimizing $C\\nu^\\star(\\Theta)\\epsilon -\\nu^\\star(\\Theta)d\\log(\\epsilon)/\\tau$, which gives $\\epsilon = d/(C\\tau)$. The first claim follows by plugging this value for $\\epsilon$ in the expression above.\n\nFor the second claim of the statement, let us build a suitable candidate $\\hat \\nu_\\epsilon$ in order to upper bound the infimum that defines $\\Qq_{\\nu^\\star,\\hat \\nu_0}(\\tau)$. Let $T$ be an optimal transport map from $\\nu_0$ to $\\hat \\nu_0$ for $W_\\infty$, i.e.\\ a measurable map $T:\\Theta\\to\\Theta$ satisfying $T_\\# \\nu_0 = \\hat \\nu_0$ and $\\max\\{\\dist(\\theta,T(\\theta))\\;;\\; \\theta\\in \\spt \\nu_0(=\\Theta)\\} = W_\\infty(\\nu_0,\\hat \\nu_0)$ (see~\\cite[Sec. 3.2]{santambrogio2015optimal}, the absolute continuity of $\\nu_0$ is sufficient for such a map to exist). Now we define $\\hat \\nu_\\epsilon = T_\\# \\nu_\\epsilon$ where $\\nu_\\epsilon$ is such that $\\Hh(\\nu_\\epsilon,\\nu_0)<\\infty$. Since the relative entropy is non-increasing under pushforwards, it holds $\\Hh(\\hat \\nu_\\epsilon,\\hat \\nu_0)\\leq \\Hh(\\nu_\\epsilon,\\nu_0)$. Moreover, it holds $\\Vert \\nu_\\epsilon -\\hat \\nu_\\epsilon\\Vert_{\\BL}^*\\leq W_1(\\nu_\\epsilon,\\hat \\nu_\\epsilon)\\leq \\nu^\\star(\\Theta) W_\\infty(\\nu_\\epsilon,\\hat \\nu_\\epsilon)$. Thus we have\n\\begin{align*}\n\\Qq_{\\nu^\\star,\\hat \\nu_0}(\\tau) &\\leq \\Vert \\nu^\\star - \\hat \\nu_\\epsilon\\Vert_{\\BL}^* +\\frac1\\tau \\Hh(\\hat \\nu_\\epsilon,\\hat \\nu_0) \\\\\n&\\leq \\Vert \\nu_\\epsilon -\\hat \\nu_\\epsilon\\Vert_{\\BL}+ \\Vert \\nu^\\star - \\nu_\\epsilon\\Vert_{\\BL} +\\frac1\\tau \\Hh(\\nu_\\epsilon,\\nu_0)\\\\\n&\\leq \\nu^\\star(\\Theta) W_\\infty(\\nu_\\epsilon,\\hat \\nu_\\epsilon) + \\Vert \\nu^\\star - \\nu_\\epsilon\\Vert_{\\BL}^* +\\frac1\\tau \\Hh(\\nu_\\epsilon,\\nu_0) .\n\\end{align*}\nThe claim follows by noticing that,  by construction, $W_\\infty(\\nu_\\epsilon,\\hat \\nu_\\epsilon) \\leq W_\\infty(\\nu_0,\\hat \\nu_0)$ and then by taking the infimum in $\\nu_\\epsilon$.\n\\end{proof}","units":[{"start":31029,"end":31400,"start_line":340,"end_line":344,"raw_tex":"\\begin{proposition} \\label{prop:projectedGF} Under {\\sf (A1-2)}, let $(\\mu_t)_{t\\geq 0}$ be a Wasserstein gradient flow for $F$, with $\\mu_0\\in \\Pp_2(\\Omega)$. Then $\\nu_t = \\h\\mu_t$ satisfies (in the weak sense)\n\\begin{equation}\\label{eq:projectedGF}\n\\partial_t \\nu_t = - 4 \\alpha \\nu_t J'_{\\nu_t} + \\beta \\div (\\nu_t \\nabla J'_{\\nu_t}).\n\\end{equation}\n\\end{proposition}","sha256_utf8":"04f7c6c5fd584fd2ee06c998f1c7de798cb91ea0715bdb77362cfa4d913633cd"},{"start":31401,"end":31867,"start_line":345,"end_line":352,"raw_tex":"\\begin{proof}\nFor any differentiable function $\\varphi: \\Theta\\to \\RR$, since $\\mu_t$ is a Wasserstein gradient flow it holds\n\\begin{equation*}\n\\frac{\\d}{\\d t}\\left(\\int \\varphi \\,\\d\\nu_t \\right) \n=- \\int \\langle \\nabla (\\h^*\\varphi), g_{\\h\\mu_t}\\rangle_{(r,\\theta)} \\d\\mu_t = - \\int (4 \\alpha \\varphi J'_{\\nu_t} + \\beta \\nabla \\varphi \\cdot \\nabla J'_{\\nu_t})\\d\\nu_t,\n\\end{equation*}\nwhich is the definition of weak solutions for~\\eqref{eq:projectedGF}.\n\\end{proof}","sha256_utf8":"ce64cbd769f3ddcb57fb5402963bdc676e431d20092fedefb6fdc4a85293af91"},{"start":46224,"end":46621,"start_line":462,"end_line":463,"raw_tex":"\\begin{proposition}[Optimality condition]\\label{prop:optimalitycondition} Under~{\\sf(A1)} and~{\\sf(A3)}, problem~\\eqref{eq:objective} admits minimizers. Moreover, a measure $\\nu^\\star \\in \\Mm_+(\\Theta)$ is a minimizer if and only if it holds $J'_{\\nu^\\star}(\\theta)\\geq 0$ for all $\\theta\\in \\Theta$ and $J'_{\\nu^\\star}(\\theta)=0$ whenever $\\theta$ in the support of $\\nu^\\star$.\n\\end{proposition}","sha256_utf8":"937b3593aad300a14f6638e3391f3911952df1ea373b9d33b58dd4fe78119d3b"},{"start":46622,"end":47405,"start_line":464,"end_line":466,"raw_tex":"\\begin{proof}\nAs $\\lambda$ is assumed positive, the sublevel sets of $J$ on $\\Mm_+(\\Theta)$ are bounded in total variation, and are thus weakly pre-compact. It follows that any minimizing sequence for $J$ admits at least one weak limit point $\\nu^\\star$, which is a minimizer of~\\eqref{eq:objective} since $J$ is weakly continuous. The stated optimality condition is equivalent to having $\\int J'_{\\nu^\\star}\\d (\\nu -\\nu^\\star) \\geq 0$ for all $\\nu \\in \\Mm_+(\\Theta)$. The latter is a sufficient optimality condition since by convexity of $J$, $J(\\nu)-J(\\nu^\\star) \\geq \\int J'_{\\nu^\\star}\\d (\\nu -\\nu^\\star)$. It is also necessary since  it holds $\\frac{d}{d\\epsilon} J((1-\\epsilon)\\nu^\\star +\\epsilon \\nu)\\vert_{\\epsilon = 0^+} = \\int J'_{\\nu^\\star} \\d(\\nu-\\nu^\\star)$.\n\\end{proof}","sha256_utf8":"f0d9563ee213b2eeefa5527b3cbcfafbe8b05f513781878635d714184cc596bb"},{"start":51961,"end":52285,"start_line":506,"end_line":508,"raw_tex":"\\begin{proposition}\\label{prop:distanceinequality}\nUnder {\\sf (A1-5)}, for all $J_{\\max}\\geq J^\\star$, there exists $C,C'>0$, such that if $\\nu\\in \\Mm_+(\\Theta)$ satisfies $J(\\nu)\\leq J_{\\max}$ then $\\Vert \\nu - \\nu^\\star\\Vert^*_{\\BL} \\leq C \\widehat W_2(\\nu, \\nu^\\star) \\leq C'(J(\\nu)-J^\\star)^{\\frac12}$.\n\\end{proposition}","sha256_utf8":"acd01fb722c73c8d11c6310a0f0ff0640b232c0afae113984a1af542a84ea8fb"},{"start":53039,"end":53373,"start_line":516,"end_line":520,"raw_tex":"\\begin{theorem}[Sharpness]\\label{th:gradientinequality} Under {\\sf(A1-5)}, there exists $J_0>J^\\star$ and $\\kappa_0>0$, such that for all $\\nu\\in \\Mm_+(\\Theta)$ satisfying $J(\\nu) \\leq  J_0$ and $\\alpha,\\beta>0$, one has\n\\[\n\\frac12 \\Vert g_\\nu\\Vert^2_{L^2(\\nu)}\\geq \\kappa_0\\, \\min\\{\\alpha,\\beta\\}\\, (J(\\nu)-J^\\star).\n\\]\n\\end{theorem}","sha256_utf8":"434aca2f219f436576e0d287dc73e2a02e203517e88615d6af61eb2e23f19278"},{"start":54017,"end":54433,"start_line":525,"end_line":530,"raw_tex":"\\begin{corollary}[Local convergence of gradient flow]\\label{cor:localGF}\nUnder {\\sf (A1-5)}, let $J_0$ and $\\kappa_0$ be given by Theorem~\\ref{th:gradientinequality}. Consider $(\\nu_t)_{t\\geq 0}$ a projected gradient flow for $J$ as in Eq.~\\eqref{eq:projectedGF}. If $J(\\nu_0)\\leq J_0$ then \n\\begin{align*}\nJ(\\nu_t) - J^* &\\leq  (J(\\nu_0) - J^*) \\exp(-2\\kappa_0 \\min\\{\\alpha,\\beta\\}  t).\n\\end{align*}\n\\end{corollary}","sha256_utf8":"e2636b9c98ee311be6341e617f2d39146d8d1b2157c24f387d0782e7cb9eb8eb"},{"start":54434,"end":54760,"start_line":531,"end_line":537,"raw_tex":"\\begin{proof}\nBy Theorem~\\ref{th:gradientinequality} and direct computations, one has \n\\[\n\\frac{d}{dt} (J(\\nu_t)-J^\\star) = \\int_{\\Omega} J'_{\\nu_t} \\d(\\partial_t \\nu_t) = - \\Vert g_{\\nu_t}\\Vert^2_{L^2(\\nu_t)} \\leq -2\\kappa_0\\min\\{\\alpha,\\beta \\} (J(\\nu_t)-J^\\star)\n\\]\nand the result follows by Gr\\\"onwall's lemma.\n\\end{proof}","sha256_utf8":"84b3aadf5359d3939f45fbceda5db0c0d282fbf42cffd32e4593c1c18f53f015"},{"start":72618,"end":73062,"start_line":669,"end_line":674,"raw_tex":"\\begin{lemma}\\label{lem:mirrordecrease}\nAssume ${\\sf (A1-3)}$ and that $J$ admits a minimizer $\\nu^\\star\\in \\Mm_+(\\Theta)$. Then for all $\\nu_0\\in \\Mm_+(\\Theta)$, denoting $B_{\\nu_0}\\coloneqq \\sup_{J(\\nu)\\leq J(\\nu_0)} \\Vert J'_{\\nu}\\Vert_{\\BL}$, it holds for $t\\geq 0$,\n\\begin{align*}\nJ(\\nu_t)-J^\\star \\leq \\inf_{s\\in [0,t]}\\left( B_{\\nu_0} \\cdot \\Qq_{\\nu^\\star,\\nu_0}(4\\alpha B_{\\nu_0}s) + \\beta B^2_{\\nu_0} s\\right).\n\\end{align*}\n\\end{lemma}","sha256_utf8":"5d69cb6a0319b0ec9b06fff8de6d3101edf69f55bc49eea19d574995a16d81f9"},{"start":73435,"end":75544,"start_line":676,"end_line":695,"raw_tex":"\\begin{proof}\nLet $\\nu^\\epsilon_0 \\in \\Mm_+(\\Theta)$ be a measure to be specified later that satisfies $\\Hh(\\nu^\\epsilon_0, \\nu_0)<+\\infty$, and let $\\nu^\\epsilon_t$ satisfy $\\partial_t \\nu^\\epsilon_t = \\div(\\beta \\nu^\\epsilon_t\\nabla J'_{\\nu_t})$ for $t\\geq 0$ weakly (this is a continuity equation with a smooth velocity field which admits a unique weak solution). Differentiating the relative entropy with respect to its second argument and using the invariance of the relative entropy under diffeomorphisms, it holds, for $t\\geq 0$,\n\\begin{align*}\n\\frac1{4\\alpha} \\frac{d}{dt} \\Hh(\\nu^\\epsilon_t, \\nu_t) &= \\int_{\\Theta} J'_{\\nu_t} \\d (\\nu^\\epsilon_t-\\nu_t) \\\\\n& = \\int_{\\Theta} J'_{\\nu_t} \\d(\\nu^\\star -\\nu_t) + \\int_{\\Theta} J'_{\\nu_t} \\d(\\nu^\\epsilon_t-\\nu^\\star) \\\\\n& \\leq -(J(\\nu_t)-J^\\star) + \\Vert J'_{\\nu_t}\\Vert_\\BL \\cdot \\Vert \\nu^\\star-\\nu^\\epsilon_t\\Vert_\\BL^*\n\\end{align*}\nwhere the first term comes from the convexity of $J$ and the second from the definition of $\\Vert \\cdot\\Vert_{\\BL}^*$. After integrating in time and rearranging the terms we get\n\\begin{align*}\n\\left(\\frac1t\\int_0^t J(\\nu_s)\\d s\\right)-J^\\star &\\leq \\frac1{4\\alpha t} \\left( \\Hh(\\nu^\\epsilon_0,\\nu_0) - \\Hh(\\nu^\\epsilon_t,\\nu_t) \\right) + \\frac{B_{\\nu_0}}{t} \\int_0^t \\Vert \\nu^\\star-\\nu^\\epsilon_s\\Vert_{\\BL}^* \\d s.\n\\end{align*}\nFor the last integral term, we use the triangular inequality\n\\[\n \\frac{B_{\\nu_0}}{t}  \\int_0^t \\Vert \\nu^\\star-\\nu^\\epsilon_s\\Vert_{\\BL}^* \\d s \\leq B_{\\nu_0}\\Vert \\nu^\\star - \\nu^\\epsilon_0\\Vert_{\\BL}^* + \\frac{B_{\\nu_0}}{t} \\int_0^t \\Vert \\nu_0^\\epsilon -\\nu^\\epsilon_s\\Vert_{\\BL}^*\\d s\\leq B_{\\nu_0} \\Vert \\nu^\\star - \\nu^\\epsilon_0\\Vert_{\\BL}^* +B_{\\nu_0}^2\\beta t\n\\]\nwhere the last term is obtained by bounding the integrated flow of the velocity field $(\\nabla J'_{\\nu_t})_{t\\geq 0}$. Since $\\Hh(\\nu_\\epsilon,\\nu_t)\\geq 0$ and $J(\\nu_s)$ is decreasing, it follows\n\\[\nJ(\\nu_t)-J^\\star  \\leq \\inf_{\\nu^\\epsilon\\in \\Mm_+(\\Theta)} \\left( \\frac1{4\\alpha t} \\Hh(\\nu^\\epsilon,\\nu_0) + B_{\\nu_0}\\Vert  \\nu^\\star -\\nu^\\epsilon\\Vert_{\\BL}^*\\right) + B_{\\nu_0}^2 \\beta t.\\qedhere\n\\]\n\\end{proof}","sha256_utf8":"1d46567219791986ad006a2b53e9211b887de5bba1498cfc6483d45dc783ee40"},{"start":75548,"end":77456,"start_line":699,"end_line":717,"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm:convergenceflow} (gradient flow)]\nBy Lemma~\\ref{lem:qdensity}, we have for $\\tau\\geq L=\\Lip(\\log \\rho)$, by writing $\\bar \\Hh \\coloneqq \\bar \\Hh(\\nu^\\star,\\rho\\vol)$,\n\\[\n\\Qq_{\\nu^\\star,\\hat \\nu_0}(\\tau) \\leq \\frac{\\bar \\Hh+d\\cdot \\nu^\\star(\\Theta)\\cdot (\\log(\\tau)+C_\\Theta)}\\tau + \\nu^\\star(\\Theta)\\cdot W_\\infty(\\nu_0,\\rho\\vol).\n\\]\nCombining this bound with Lemma~\\ref{lem:mirrordecrease}, we get that for $t\\geq L/(4\\alpha B_{\\nu_0})$,\n\\[\nJ(\\nu_t)-J^\\star \\leq \\frac{\\bar \\Hh+d\\cdot\\nu^\\star(\\Theta)\\cdot(\\log(4\\alpha B_{\\nu_0} t)+C_\\Theta)}{4\\alpha B_{\\nu_0} t} + B_{\\nu_0}\\cdot\\nu^\\star(\\Theta)\\cdot W_\\infty(\\nu_0,\\rho\\vol)+\\beta B^2_{\\nu_0} t.\n\\]\nIn particular, for $t=(\\alpha\\beta)^{-\\frac12}$, we get \n\\begin{equation}\\label{eq:lastbeforelast}\nJ(\\nu_t)-J^\\star \\leq \\sqrt{\\frac{\\beta}{\\alpha}}\\left(\\bar \\Hh + d\\cdot \\nu^\\star(\\Theta) (\\log(4B_{\\nu_0}\\sqrt{\\alpha/\\beta})+C_\\Theta) + B^2_{\\nu_0}\\right) + B_{\\nu_0}\\nu^\\star(\\Theta)\\cdot W_\\infty(\\nu_0,\\rho\\vol).\n\\end{equation}\nSince this is valid only when $t\\geq L/(4\\alpha B_{\\nu_0})$, we require $(\\alpha\\beta)^{-\\frac12}\\geq L/(4\\alpha B_{\\nu_0})$ which leads to the first condition on $\\beta/\\alpha$. Now, we want the right-hand side of~\\eqref{eq:lastbeforelast} to be smaller than $\\Delta_0 \\coloneqq J_0 - J^\\star$ so that we can conclude with Corollary~\\ref{cor:localGF}. To this end, we require, on the one hand $W_\\infty(\\nu_0,\\rho\\vol) \\leq \\Delta_0/(2B_{\\nu_0}\\nu^\\star(\\Theta))$. On the other hand, we use the bound $\\log(u)\\leq C_\\epsilon u^\\epsilon$ for $\\epsilon\\in {]0,1/2]}$, require $4B_{\\nu_0}\\sqrt{\\alpha/\\beta}\\geq 1$ and obtain the condition\n\\[\n(4B_{\\nu_0})^\\epsilon(\\beta/\\alpha)^{(1-\\epsilon)/2}\\left( \\bar \\Hh + d\\cdot \\nu^\\star(\\Theta)(C_\\Theta+C_\\epsilon) +B^2_{\\nu_0}\\right)\\leq \\frac12\\Delta_0.\n\\]\nThis leads to the second condition on $\\beta/\\alpha$ is the theorem.\n\\end{proof}","sha256_utf8":"2fcc7fc5e73682a4b2dc0e19717e89589b8acdb1c6f092eb0c8c7d61637d39be"},{"start":100994,"end":101135,"start_line":941,"end_line":943,"raw_tex":"\\begin{lemma}\\label{lem:distanceinequality}\nIt holds $\\widehat W_2(\\nu,\\nu^\\star)\\leq W_\\tau(\\nu)(1+O(\\tau^2)+O(W_\\tau(\\nu)^2))$.\n\\end{lemma}","sha256_utf8":"1677f409bae4bb7011d5f93d2b6e485bc74616e6296c3fbfc67cc7ac0f24a130"},{"start":101136,"end":103552,"start_line":944,"end_line":980,"raw_tex":"\\begin{proof}\nNote that for $W_\\tau(\\nu)$ small enough, it holds $\\nu(\\Theta_i)>0$ for $i\\in \\{1,\\dots,m^\\star\\}$. Let $\\mu \\in \\Pp_2(\\Omega)$ be such that $\\h\\mu = \\nu$ and consider the transport map $T:\\Omega\\to\\Omega$ defined as\n \\[\n T(r,\\theta) =\n \\begin{cases}\n(r\\frac{r_i}{\\bar r_i}, \\theta_i) &\\text{if $\\theta \\in \\Theta_i$ and $\\bar r_i>0$, $i\\in \\{1,\\dots,m^\\star\\}$},\\\\\n(0,\\theta) &\\text{otherwise}.\n \\end{cases}\n \\]\nBy construction, it holds $\\h(T_\\#\\mu) =  \\nu^\\star$. Let us estimate the transport cost associated to this map \n\\[\n\\mathcal{T} = \\int_\\Omega \\dist((r,\\theta),T(r,\\theta))^2\\d\\mu(r,\\theta).\n\\]\nThe geodesic distance associated to the cone metric is\n\\begin{align*}\n\\dist((r_1,\\theta_1),(r_2,\\theta_2))^2 &= r_1^2 + r_2^2 - 2r_1r_2\\cos(\\min\\{\\dist(\\theta_1,\\theta_2),\\pi\\})\\\\\n&= (r_1-r_2)^2 +r_1r_2\\dist(\\theta_1,\\theta_2)^2 +O(r_1r_2\\dist(\\theta_1,\\theta_2)^4)\n\\end{align*}\nNow, if we only consider points $\\theta\\in \\Theta_i$ with $\\tilde \\theta$ their coordinates in a normal frame centered at $\\theta_i$ (note that in all other proofs, we do not need to distinguish between $\\theta$ and $\\tilde \\theta$), we have the approximation\n\\begin{align*}\n\\dist((r_1,\\theta_1),(r_2,\\theta_2))^2 = (r_1-r_2)^2 +r_1r_2\\Vert \\tilde \\theta_1 - \\tilde \\theta_2\\Vert^2(1+O(\\tau^2)).\n\\end{align*}\nLet us decompose $T(r,\\theta)$ as $(rT^r(\\theta),T^\\theta(\\theta))$ and estimate the two contributions forming $\\Tt$ separately. On the one hand, we have\n\\begin{align*}\n\\int (rT^r(\\theta)-r)^2\\d\\mu(r,\\theta) &= \\bar r_0^2 + \\sum_{i=1}^{m^\\star} (\\bar r_i-r_i)^2 = \\bar r_0^2 +\\Vert b^r\\Vert^2(1+O(W_\\tau(\\nu)^2)).\n\\end{align*}\nOn the other hand, we have\n\\begin{align*}\n\\int r^2T^r(\\theta)\\Vert T^\\theta(\\theta)-\\theta\\Vert^2\\d\\mu(r,\\theta) &=\\sum_{i=1}^{m^\\star} \\bar r_ir_i (\\tr\\Sigma_i + \\Vert \\delta \\theta_i\\Vert^2)=(\\Vert s\\Vert^2 + \\Vert b^\\theta\\Vert^2)(1+O(W_\\tau(\\nu)^2)).\n\\end{align*}\nAs a consequence, we have $\\Tt = W_\\tau(\\nu)(1 + O(W_\\tau(\\nu)^2)+ O(\\tau^2))$. Remark that this estimate does not depend on the chosen lifting $\\mu$ satisfying $\\h\\mu=\\nu$. We then conclude by using the characterization in~\\cite[\nThm. 7.20]{liero2018optimal} for the distance $\\widehat W_2$:\n\\[\n\\widehat W_2(\\nu_1,\\nu_2) = \\min \\left\\{W_2(\\mu_1,\\mu_2) \\;;\\; (\\h\\mu_1,\\h\\mu_2) = (\\nu_1,\\nu_2)\\right\\}.\n\\]\nThus $\\widehat W_2(\\nu,\\nu^\\star)^2 \\leq W_2(\\mu,T_\\#(\\mu))^2\\leq \\Tt$, and the result follows.\n\\end{proof}","sha256_utf8":"eadc26c4556ecb6b26a959ea141c16cf77ee065fead37704207cce8d1020d4e1"},{"start":103593,"end":104642,"start_line":984,"end_line":997,"raw_tex":"\\begin{lemma}[Expansion around $\\nu^\\star$]\\label{lem:expansion}\n Let $\\psi$ be any (vector or real-valued) smooth function on $\\Theta$ and $\\nu\\in \\Mm_+(\\Theta)$. If $\\tau>0$ is an admissible radius, then the following first and second-order expansions hold\n \\begin{align*}\n \\int \\psi \\d(\\nu - \\nu^\\star) \n &= \\sum_{i=1}^m r_i \\bar \\nabla \\psi(\\theta_i)^\\intercal b_i \n + \\int_{\\Theta_0} \\psi \\d\\nu \n +\\sum_{i=1}^m \\int_{\\Theta_i} M_{2,\\psi}(\\theta_i,\\theta)\\d\\nu(\\theta)\\\\\n &=\\sum_{i=1}^m r_i\\bar \\nabla \\psi(\\theta_i)^\\intercal b_i \n +\\frac12  \\sum_{i=1}^m \\bar r_i^2  \\left( \\tr\\left(\\nabla^2\\psi(\\theta_i) \\Sigma_i \\right) \n + \\delta \\theta_i^\\intercal \\nabla^2\\psi(\\theta_i) \\delta \\theta_i \\right)\\\\\n &\\qquad + \\int_{\\Theta_0}  \\psi \\d\\nu +\\sum_{i=1}^m \\int_{\\Theta_i} M_{3,\\psi}(\\theta_i,\\theta)\\d\\nu(\\theta)\n \\end{align*}\n where $M_{k,\\psi}(\\theta_i,\\theta)$ is the remainder in the $k-1$-th order Taylor expansion of $\\psi$ around $\\theta_i$ in local coordinates (and we recall that $\\bar \\nabla \\psi := (2\\psi,\\nabla \\psi)$).\n \\end{lemma}","sha256_utf8":"ec32d5d4111f2ed8b148137ebd4e2de8e1a4da1b3a1a74d0785b7031204a05fd"},{"start":104644,"end":105696,"start_line":998,"end_line":1010,"raw_tex":"\\begin{proof}\nBy a Taylor expansion of $\\psi$ around $\\theta_i$ for $i\\in \\{1,\\dots,m^\\star\\}$, it holds\n\\begin{align*}\n \\int_{\\Theta_i} \\psi \\d\\nu= \\int_{\\Theta_i} \\left( \\psi(\\theta_i) + \\nabla\\psi(\\theta_i)^\\intercal (\\theta-\\theta_i) + (\\theta-\\theta_i) ^\\intercal \\nabla^2\\psi(\\theta_i)(\\theta-\\theta_i) + M_{3,\\psi}(\\theta_i,\\theta)\\right)\\d\\nu(\\theta)\n\\end{align*}\nand substracting $\\int_{\\Theta_i}\\psi\\d\\nu^\\star = r_i^2\\phi(\\theta_i)$ yields\n\\begin{multline*}\n \\int_{\\Theta_i} \\psi \\d(\\nu-\\nu^\\star)= (\\bar r_i^2 -r_i^2)\\psi(\\theta_i) +  \\bar r_i^2 \\nabla \\psi(\\theta_i)^\\intercal \\delta \\theta_i \\\\ \n +\\frac12  \\sum_{i=1}^m \\bar r_i^2  \\left( \\tr\\left(\\nabla^2\\psi(\\theta_i) \\Sigma_i \\right) \n + \\delta \\theta_i^\\intercal \\nabla^2\\psi(\\theta_i) \\delta \\theta_i \\right)  +\\sum_{i=1}^m \\int_{\\Theta_i} M_{3,\\psi}(\\theta_i,\\theta)\\d\\nu(\\theta)\n \\end{multline*}\nwhere  we have used a bias-variance decomposition for the quadratic term. The result follows by summing the integrals over each $\\Theta_i$ and using the expression of $b$.\n \\end{proof}","sha256_utf8":"109ffd279caf43ea292d31cb3b41d0682b7dc88ae31a8f336471692f1c5d9cb2"},{"start":106239,"end":106908,"start_line":1016,"end_line":1025,"raw_tex":"\\begin{lemma}[Global distance bound]\\label{lem:distancebound}\nUnder {\\sf (A1-5)}, let $\\tau_{\\adm}$ be an admissible radius $\\tau$ as in Definition~\\ref{def:localfeatures}, fix some $J_{\\max}>0$ and let\n\\[\n\\tau_0 = \\min\\left\\{\\tau_{\\adm}, 2\\sqrt{\\frac{v^\\star}{\\sigma_{\\min}(H)}}, \\frac{3\\sigma_{\\min}(H)}{2 \\Lip (\\nabla^2 J'_{\\nu^\\star})} \\right\\}.\n\\]\nThen there exists $C, C'>0$ such that for all $\\tau\\leq \\tau_0$ and  $\\nu \\in \\Mm_+(\\Theta)$ such that $J(\\nu)\\leq J_{\\max}$, it holds \n\\begin{align*}\nW_\\tau(\\nu) \\leq \\frac{C}{\\tau^2}(J(\\nu)-J^\\star)^\\frac12 &&\\text{and}&&\\Vert \\tilde b^\\theta\\Vert^2+\\Vert s\\Vert^2\\leq C'(J(\\nu)-J^\\star). \n\\end{align*}\n\\end{lemma}","sha256_utf8":"9aa7a7ebed62a47874932d3ac60c05b6df3c7b46b4b82a65327cacda40262776"},{"start":106910,"end":111046,"start_line":1027,"end_line":1082,"raw_tex":"\\begin{proof}\nLet us write $f_\\nu \\coloneqq \\int \\phi\\d\\nu$ and $f^\\star = \\int \\phi \\d\\nu^\\star$. By strong convexity of $R$ at $f^\\star$, and optimality of $\\mu^\\star$, there exists $C>0$ such that for all $\\nu\\in \\Mm_+(\\Theta)$ it holds\n\\begin{equation}\\label{eq:strongconvexityinequality}\nJ(\\nu)-J^\\star \\geq \\int_{\\Theta} J'_{\\nu^\\star}\\d\\nu+ C\\min\\{ \\Vert f_\\nu-f^\\star\\Vert^2,\\Vert f_\\nu-f^\\star\\Vert\\} .\n\\end{equation}\nTo prove the first claim, we thus have to bound $W_\\tau(\\nu)$ using the terms in the right-hand side of~\\eqref{eq:strongconvexityinequality}. \n\n\\paragraph{Step 1.} By a Taylor expansion, one has for $\\theta \\in \\Theta_i$ for $i\\in \\{1,\\dots,m^\\star\\}$,\n\\[\n\\vert J'_{\\nu^\\star}(\\theta) - \\tfrac12 (\\theta -\\theta_i)^\\intercal H_i  (\\theta-\\theta_i) \\vert \\leq \\tfrac16 \\Lip( \\nabla^2 J'_{\\nu^\\star}) \\Vert \\theta-\\theta_i\\Vert^3.\n\\]\nThus, if $\\Vert \\theta - \\theta_i\\Vert \\leq 3\\sigma_{\\min}(H)/(2 \\Lip( \\nabla^2 J'_{\\nu^\\star}))$, then $J'_{\\nu^\\star}(\\theta) \\geq \\frac14 (\\theta -\\theta_i)^\\intercal H_i  (\\theta-\\theta_i)$ for $\\theta\\in \\Theta_i$. Decomposing the integral of this quadratic term into bias and variance, we get\n\\begin{align*}\\label{eq:biasvariance}\n\\int_{\\Theta_i} (\\theta -\\theta_i)^\\intercal H_i (\\theta-\\theta_i)\\d\\nu(\\theta) \n&= \\bar r_i^2 \\left(\\delta \\theta_i^\\intercal  H_i \\delta \\theta_i + \\tr(\\Sigma_i H_i )\\right)\n\\end{align*}\nand we deduce a first bound by summing the terms for $i\\in \\{1,\\dots,m^\\star\\}$,\n\\[\n\\int_{\\Theta\\setminus \\Theta_0} J'_{\\nu^\\star} \\d\\nu \\geq \\frac{\\sigma_{\\min}(H)}4(\\Vert \\tilde b^\\theta\\Vert^2 + \\Vert s\\Vert^2).\n\\]\n\n\\paragraph{Step 2.} In order to lower bound the integral over $\\Theta_0$, we first derive a lower bound for $J'_{\\nu^\\star}$ on $\\Theta_0$. This is a continuously differentiable and nonnegative function on a closed domain $\\Theta_0$ so its minimum is attained either at a local minima in the interior of $\\Theta_0$ or on its boundary. Using the quadratic lower bound from the previous paragraph, it follows that for $\\theta\\in \\Theta_0$,\n\\[\nJ'_{\\nu^\\star}(\\theta)\\geq \\min \\{ v^\\star,  \\tau^2 \\sigma_{\\min}(H)/4\\}.\n\\]\nThus, if we also assume that $\\tau\\leq 2\\sqrt{v^\\star/\\sigma_{\\min}(H)}$ then $J'_{\\nu^\\star}(\\theta)\\geq \\tau^2 \\sigma_{\\min}(H)/4$ for $\\theta\\in \\Theta_0$ and it follows that\n\\[\n\\int_{\\Theta_0} J'_{\\nu^\\star} \\d\\nu \\geq \\frac{\\sigma_{\\min}(H)}{4} \\tau^2\\bar r_0^2 .\n\\]\nUsing inequality~\\eqref{eq:strongconvexityinequality}  we have shown so far that\n\\begin{equation}\\label{eq:localcontrol}\n\\tilde W_\\tau(\\nu)^2\\coloneqq  \\bar r_0^2 +\\Vert \\tilde b^\\theta\\Vert^2 +\\Vert s\\Vert^2 \\leq \\frac{C}{\\tau^2} (J(\\mu)-J^\\star).\n\\end{equation}\nNotice that $\\tilde W_\\tau(\\nu)$ is similar to $W_\\tau(\\nu)$ but it does not contain the terms controlling the deviations of mass $\\vert \\bar r_i-r_i\\vert$. These quantities can be controlled by using the coercivity of $R$, i.e. the last term in~\\eqref{eq:strongconvexityinequality}, as we do now.\n\n\\paragraph{Step 3.}\nUsing the first order expansion of Lemma~\\ref{lem:expansion} then squaring gives\n\\begin{align*}\n\\left\\vert \\Vert f_\\nu-f^\\star\\Vert^2 - \\frac12 b^\\intercal K b \\right\\vert\n& \\leq C \\left( \\Vert b\\Vert \\tilde W_\\tau(\\nu)^2 +  \\tilde W_\\tau(\\nu)^4 \\right).\n\\end{align*}\n%\nSince we have assumed that $K$ is positive definite, it follows\n\\[\n\\Vert f_\\nu-f^\\star\\Vert^2 \\geq C \\Vert b\\Vert^2 - C \\tilde W_\\tau(\\nu)^2 \\Vert b\\Vert - C\\tilde W_\\tau(\\nu)^4\n\\]\nand thus, after rearranging the terms\n\\[\n(\\Vert b\\Vert - C\\tilde W_\\tau(\\nu)^2)^2 \\leq  C \\Vert f_\\nu-f^\\star\\Vert^2 + C\\tilde W_\\tau(\\nu)^4.\n\\]\nIt follows that $\\Vert b\\Vert \\leq  C\\Vert f_\\nu-f^\\star\\Vert + C\\tilde W_\\tau(\\nu)^2$. Also, by inequality~\\eqref{eq:strongconvexityinequality}, if $J(\\nu)\\leq J_{\\max}$, then $\\Vert f_\\nu -f^\\star\\Vert^2\\leq C(J(\\nu)-J^\\star)$.  Moreover, by inequality~\\eqref{eq:localcontrol}, we get\n\\[\n\\Vert b\\Vert \\leq \\frac{C}{\\tau^2}(J(\\nu)-J^\\star) +C (J(\\nu)-J^\\star)^\\frac12 \\leq \\frac{C}{\\tau^2} (J(\\nu)-J^\\star)^\\frac12.\n\\]\nWe finally combine with the bound on $\\tilde W_\\tau(\\nu)$ to conclude since $W_\\tau(\\nu)^2\\leq \\tilde W_\\tau(\\nu)^2+\\Vert b\\Vert^2$\n\\end{proof}","sha256_utf8":"ac902d7a6d2e3bddaa9a6a4851bbc781c96add3ba438bd035d53ab546fd71d06"},{"start":111049,"end":111835,"start_line":1084,"end_line":1100,"raw_tex":"\\subsection{Proof of the distance inequality (Proposition~\\ref{prop:distanceinequality})}\\label{app:distanceinequality}\nBy Lemma~\\ref{lem:distanceinequality}, it holds \n\\[\n\\widehat W_2(\\nu,\\nu^\\star)\\leq W_\\tau(\\nu)(1+O(\\tau^2)+O(W_\\tau(\\nu)^2)).\n\\]\nMoreover, by Lemma~\\ref{lem:distancebound}, there exists $\\tau_0>0$ and $C>0$ such that \n\\[\nW_\\tau(\\nu)\\leq \\frac{C}{\\tau_0^2}(J(\\nu)-J^\\star)^\\frac12.\n\\]\nCombining these two lemmas, it follows that for some $C'>0$, we have \n\\[\n\\widehat W_2(\\nu,\\nu^\\star)^2 \\leq C'(J(\\nu)-J^\\star)^\\frac12.\n\\]\n This also implies a control on the Bounded-Lipschitz distance since it holds $(\\Vert \\nu - \\nu^\\star\\Vert_{\\BL}^*)^2\\leq (2+\\pi^2/2)(\\nu(\\Theta)+\\nu^\\star(\\Theta))\\widehat W_2(\\nu,\\nu^\\star)^2$, see~\\cite[Prop. 7.18]{liero2018optimal}.\n \n \n ","sha256_utf8":"1a6bf4b684bb3b705921ec68836744d58087207d6df99a895e52b5960a9c48be"},{"start":111930,"end":112606,"start_line":1102,"end_line":1111,"raw_tex":"\\begin{proposition}[Local expansion]\\label{prop:localexpansion}\n It holds\n \\[\n J(\\nu)-J^\\star = \\frac12 b^\\intercal K b + \\frac12 \\sum_{i=1}^m \\bar r_i^2 \\left( \\tr(\\Sigma_i H_i) +\\delta \\theta^\\intercal H_i \\delta \\theta_i \\right) + \\int_{\\Theta_0} J'_{\\nu^\\star} \\d\\nu + \\mathop{\\mathrm{err}}(\\tau,\\nu)\n \\]\n where $\\mathop{\\mathrm{err}}(\\tau,\\nu) =O( \\tau (\\Vert \\tilde b^\\theta \\Vert^2 + \\Vert s\\Vert^2) + W_\\tau(\\nu)^3)$. In particular, if $\\tau$ is fixed small enough,\n \\[\n J(\\nu)-J^\\star \\leq  \\sigma_{\\max}(K) \\Vert b\\Vert^2 + \\sigma_{\\max}(H)(\\Vert b^\\theta\\Vert^2 +\\Vert s\\Vert^2) +\\Vert J'_{\\nu^\\star}\\Vert_\\infty \\bar r_0^2 +O(W_\\tau(\\nu)^3).\n \\]\n \\end{proposition}","sha256_utf8":"dfc63f40aa4bd18d68feb37acf73d8cbf73500da36533fa64f66584981379c1f"},{"start":112608,"end":113914,"start_line":1112,"end_line":1125,"raw_tex":"\\begin{proof} Let us write $f_\\nu \\coloneqq \\int \\phi\\d\\nu$ and $f^\\star = \\int \\phi \\d\\nu^\\star$. By a second order Taylor expansion of $R$ around $f^\\star$, we have\n \\[\n J(\\nu) - J^\\star = \\int_{\\Theta} J'_{\\nu^\\star} \\d\\nu+ \\frac12 \\Vert f_\\nu-f^\\star\\Vert^2_\\star + O(\\Vert f_\\nu-f^\\star\\Vert^3).\n \\]\n Using the first order expansion of Lemma~\\ref{lem:expansion} for $\\phi$, we get \n $\n \\Vert f_\\nu-f^\\star\\Vert^2_\\star %= b^\\intercal Kb + O\\left( (r_0^2 + \\sum_{i=1}^m s_i^2)(\\Vert b\\Vert + r_0^2 + \\sum_{i=1}^m s_i^2)\\right) \n = b^\\intercal Kb + O(W_{\\tau}(\\nu)^3)\n $. Also, using the second order expansion of Lemma~\\ref{lem:expansion} for $J'_{\\nu^\\star}$ and using the fact that $J'_{\\nu^\\star}$ and its gradient vanish for all $\\theta_i$, we get \n \\[\n \\int_{\\Theta} J'_{\\nu^\\star} \\d\\nu = \\frac12 \\sum_{i=1}^m \\bar r_i^2 \\left( \\tr(\\Sigma_i H_i) +\\delta \\theta^\\intercal H_i \\delta \\theta_i \\right) + \\int_{\\Theta_0}J'_{\\nu^\\star} \\d\\nu  + O(\\tau (\\Vert s\\Vert^2+\\Vert  b^\\theta \\Vert^2))\n \\]\nand the expansion follows. Notice also that in the expression of $J(\\nu)$, $\\bar r_i$ and $r_i$ are interchangeable up to introducing higher order error, since $\\vert r_i - \\bar r_i\\vert = O(\\vert b^r_i\\vert)$ (and also $\\Vert \\tilde b^\\theta\\Vert = \\Vert b^\\theta\\Vert(1+O(W_\\tau(\\nu)))$).\n \\end{proof}","sha256_utf8":"e877b018984c2192233a321dfd6234b8f1ea2878eb2faf0ac64f5ffb689a463b"},{"start":113996,"end":114683,"start_line":1131,"end_line":1140,"raw_tex":"\\begin{proposition}[Gradient estimate]\\label{prop:gradientestimate}\nFor $\\nu \\in \\Pp_2(\\Omega)$, it holds \n\\[\n\\Vert g_\\nu\\Vert^2_{L^2(\\nu)} =  b^\\intercal (K + H)^2b+ \\sum_{i=1}^m \\bar r_i^2 \\tr (\\Sigma_i H_i^2) +\\Vert g_\\nu\\Vert^2_{L^2(\\nu\\vert_{\\Theta_0})}+\\mathop{\\mathrm{err}}(\\tau,\\nu)\n\\]\nwhere $\\mathop{\\mathrm{err}}(\\tau,\\nu)\\lesssim \\tau(\\Vert \\tilde b^\\theta\\Vert^2 +\\Vert s\\Vert^2) +W_\\tau(\\nu)^3$. In particular, if $\\tau$ is fixed small enough\n\\[\n\\Vert g_\\nu\\Vert^2_{L^2(\\nu)}  \\geq \\frac12 (\\sigma_{\\min}(K)+ \\sigma_{\\min}(H))^2\\Vert b\\Vert^2 + \\frac12 \\sigma_{\\min}(H)^2 \\Vert s\\Vert^2 +  \\frac14 \\bar r_0^2 \\sigma_{\\min}(H)^2 \\tau^4 +O(W_\\tau(\\nu)^3).\n\\]\n\\end{proposition}","sha256_utf8":"d36353e4d37175bcf6aac0ce68c0f82eb20bf55e5b0de1264f09df70f002d2d4"},{"start":114684,"end":122100,"start_line":1141,"end_line":1224,"raw_tex":"\\begin{proof}\nFor this proof, we write $f_\\nu - f^\\star = \\delta f_0 +\\delta f_b + \\delta f_{\\mathrm{err}}$ where\n\\begin{align*}\n \\delta f_0 &\\coloneqq  \\int_{\\Theta_0}\\phi(\\theta)\\d\\nu(\\theta),&\n \\delta f_b & \\coloneqq  \\sum_{i=1}^m r_i \\bar\\nabla \\phi(\\theta_i) b_{i},&\n\\delta f_{\\mathrm{err}} &\\coloneqq \\sum_{i=1}^{m} \\int_{\\Theta_i} M_{\\phi,2}(\\theta_i,\\theta)\\d\\nu(\\theta).\n\\end{align*}\nwhere the decomposition follows from Lemma~\\ref{lem:expansion}.\n%\nThe expression for the norm of the gradient is as follows:\n\\[\n\\Vert g_\\nu\\Vert^2_{L^2(\\nu)} = \\int_{\\Theta}\\Vert\\bar \\nabla J'_\\nu(\\theta) \\Vert^2\\d\\nu(\\theta)\n\\]\nwhere $\\bar \\nabla J = (2J,\\nabla J)$. We start with the following decomposition for $\\theta \\in \\Theta_i$ (recall that $J'_\\nu(\\theta) =\\langle \\phi(\\theta),\\nabla R(\\int \\phi \\d\\nu)\\rangle +\\lambda$):\n\\begin{multline*}\nJ'_\\nu(\\theta) = \\lambda + \\left\\langle \\phi(\\theta_i) + (\\theta-\\theta_i)^\\intercal \\nabla \\phi(\\theta_i) + \\frac12  (\\theta-\\theta_i)^\\intercal\\nabla^2\\phi(\\theta_i)(\\theta-\\theta_i)+  M_{\\phi,3}(\\theta_i,\\theta),\\nabla R(f^\\star)\\right\\rangle \\\\\n+\\langle \\phi(\\theta_i)+ (\\theta-\\theta_i)^\\intercal \\nabla \\phi(\\theta_i) + M_{\\phi,2}(\\theta_i,\\theta),f_\\mu-f^\\star\\rangle_{\\star}+ \\langle \\phi(\\theta),M_{\\nabla R,2}(f^\\star,f)\\rangle\n\\end{multline*}\nHere we use the notation $\\langle\\cdot,\\cdot\\rangle_\\star$ to denote the quadratic form associated to $\\nabla^2 R(f^\\star)$.\nThanks to the optimality conditions  $\\bar \\nabla J'_{\\nu^\\star}(\\theta_i)=0$ for $i\\in \\{1,\\dots,m\\}$, we get\n\\begin{align*}\n\\bar \\nabla_j J'_\\nu(\\theta) &=  [H_i(\\theta-\\theta_i)]_j + \\langle \\bar \\nabla_j \\phi(\\theta_i),\\delta f_0+\\delta f_b \\rangle_\\star + [N(\\theta_i,\\theta)]_j\\\\\n &=  [H_i(\\theta-\\bar \\theta_i)]_j+ \\left( [H_i(\\bar \\theta_i-\\theta_i)]_j+ \\langle \\bar \\nabla_j \\phi(\\theta_i),\\delta f_b \\rangle_\\star\\right) + \\langle \\bar \\nabla_j  \\phi(\\theta_i),\\delta f_0\\rangle_\\star + [N(\\theta_i,\\theta)]_j\n\\end{align*}\nwhere $N$ collects the higher order terms and is defined as\n\\begin{multline*}\n [N(\\theta_i,\\theta)]_j =  \\langle \\bar \\nabla_j M_{\\phi,3}(\\theta_i,\\theta),\\nabla R(f^\\star)\\rangle %+ \\langle \\partial_j \\phi(\\theta_i),N_1\\rangle \\\\\n +\\langle \\bar \\nabla_j M_{\\phi,2}(\\theta_i,\\theta),f-f^\\star\\rangle_{\\star} + \\langle \\bar \\nabla_j \\phi(\\theta),M_{\\nabla R,2}(f^\\star,f)\\rangle\n\\end{multline*}\nwhere $\\Vert \\bar \\nabla_j M_{\\phi,3}(\\theta_i,\\theta)\\Vert = O(\\Vert \\theta-\\theta_i\\Vert^2)$ if $j>0$ and $O(\\Vert \\theta -\\theta_i\\Vert^3)$ if $j=0$.\nExpanding the square gives the following ten terms:\n\\begin{align}\n\\int_{\\Omega\\setminus \\Omega_0} \\Vert\\bar \\nabla J'_\\nu \\Vert^2\\d\\nu(\\theta)\n&= \\sum_{i=1}^{m} \\int_{\\Theta_i} \\sum_{j=0}^{d} [H_i(\\theta-\\bar \\theta_i)]_j^2 \\d\\nu \\tag{I} \\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\sum_{j=0}^d \\left( \\langle\\bar \\nabla_j\\phi(\\theta_i),\\delta f_b \\rangle_\\star + [H_i(\\bar \\theta_i-\\theta_i)]_j \\right)^2 \\d\\nu(\\theta) \\tag{II} \\\\\n&+ \\sum_{i=1}^{m} \\int_{\\Theta_i} \\sum_{j=0}^d \\langle \\bar \\nabla_j\\phi(\\theta_i), \\delta f_0 \\rangle_\\star^2\\d\\nu(\\theta)  \\tag{III} \\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d  [H_i(\\theta-\\bar \\theta_i)]_j\\cdot ( \\langle\\bar \\nabla_j \\phi(\\theta_i),\\delta f_b\\rangle_\\star + [H_i(\\bar \\theta_i-\\theta_i)]_j) \\d\\nu(\\theta) \\tag{IV} \\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d [H_i(\\theta-\\bar \\theta_i)]_j\\cdot \\langle \\bar \\nabla_j\\phi(\\theta_i), \\delta f_0\\rangle_\\star\\d\\nu(\\theta) \\tag{V}\\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d (\\langle\\bar \\nabla_j \\phi(\\theta_i),\\delta f_b\\rangle_\\star + [H_i(\\bar \\theta_i-\\theta_i)]_j)\\cdot \\langle \\bar\\nabla_j \\phi(\\theta_i), \\delta f_0\\rangle_\\star\\d\\nu(\\theta)\\!\\! \\tag{VI}\\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d [N(\\theta_i,\\theta)]_j\\cdot  [H_i(\\theta-\\bar \\theta_i)]_j\\d\\nu(\\theta) \\tag{VII}\\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d [N(\\theta_i,\\theta)]_j\\cdot  (\\langle\\bar \\nabla_j \\phi(\\theta_i),\\delta f_b\\rangle_\\star + [H_i(\\bar \\theta_i-\\theta_i)]_j)\\d\\nu(\\theta) \\tag{VIII}\\\\\n&+\\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d [N(\\theta_i,\\theta)]_j\\cdot \\langle \\bar \\nabla_j \\phi(\\theta_i), \\delta f_0 \\rangle_\\star\\d\\nu(\\theta) \\tag{IX}\\\\\n&+ \\sum_{i=1}^{m} \\int_{\\Theta_i} \\sum_{j=0}^d [N(\\theta_i,\\theta)]_j^2 \\d\\nu(\\theta) \\tag{X}\n\\end{align}\nTerms (I) to (II) are the main terms in the expansion, while the other terms are higher order. The term (I) is a local curvature term and can be expressed as $\\mathrm{(I)} = \\sum_{i=1}^m \\bar r_i^2 \\tr \\Sigma_i H_i^2$. \n%\nThe term (II) is a global interaction term that writes\n\\begin{align*}\n\\mathrm{(II)} &=\n\\sum_{i=1}^m \\bar r_i^2 \\sum_{j=0}^d \\vert \\langle\\bar \\nabla_j\\phi(\\theta_i),\\delta f_b\\rangle_\\star + [H_i(\\bar \\theta_i-\\theta_i)]_j \\vert^2 \\\\\n&= \\sum_{i=1}^m\\sum_{j=0}^d \\left\\vert \\sum_{i'=1}^m \\sum_{j'=0}^d (\\langle \\bar r_i \\bar \\nabla_j\\phi(\\theta_i),r_{i'}\\bar \\nabla_{j'}\\phi(\\theta_{i'})\\rangle_\\star + \\bar H_{(i,j),(i,j')})(b_{i',j'}) \\right\\vert^2\\\\\n&= \\Vert (\\bar K + H)(b)\\Vert^2\n\\end{align*}\nwhere the entries of $\\bar K$ and $\\bar H$ differ from those of $K$ and $H$ by a factor $\\bar r_i/r_i$. More precisely,\n\\[\n[\\bar K-K]_{(i,j),(i',j')} = (\\bar r_i/r_i-1)K_{(i,j),(i',j')}\n\\]\nand similarly for $\\bar H- H$. Since $\\vert \\bar r_i/r_i-1\\vert = O(\\vert b^r_i\\vert)$ we have $\\sigma_{\\max}(\\bar K-K )=O(W_\\tau(\\nu))$. It follows, by expanding the square, that \n\\[\n\\Vert (\\bar K + \\bar H)(b)\\Vert^2 = \\Vert (K + H)(b)\\Vert^2 + O(W_\\tau(\\nu)^3).\n\\]\n\nThe remaining terms are error terms, that we estimate directly in terms of $W_\\tau(\\nu)$ and $\\tau$. We use in particular the fact that by H\\\"older's inequality, $\\int_{\\Theta_i} \\Vert \\theta - \\bar \\theta_i\\Vert\\d\\nu(\\theta) = O(\\bar r_i^2 \\tr \\Sigma_i^{\\frac12})$. One has \n\\begin{itemize}\n\\item $\\mathrm{(III)} =O\\left(\\sum_{i=1}^m \\bar r_i^2 \\bar r_0^4 \\right)= O(W_{\\tau}^4(\\nu))$;\n\\item $\\mathrm{(IV)} = \\mathrm{(V)} = 0$ because the integral of the terms $H_i(\\theta-\\bar \\theta_i)$ vanishes;\n%\\item $\\mathrm{(V)} = O(\\sum \\bar r_i^2 \\sigma_i \\bar r_0^2) = O( W_{\\tau}^3)$;\n\\item $\\mathrm{(VI)} = O\\left((\\sum_{i=1}^m \\bar r_i^2 (\\Vert b\\Vert +\\Vert \\delta \\theta_i\\Vert)\\cdot \\bar r_0^2\\right)=O(W_{\\tau}^3(\\nu))$;\n\\item $\\mathrm{(VII)} = O(\\tau (\\Vert \\tilde b^\\theta\\Vert^2 + \\Vert s\\Vert^2)) + O(W_{\\tau}(\\nu)^3)$;\n\\item $\\mathrm{(VIII)} = O(W_{\\tau}^3(\\nu))$;\n\\item $\\mathrm{(IX)} = O(W_{\\tau}^4(\\nu))$;\n\\item $\\mathrm{(X)} = O(\\tau^2(\\Vert \\tilde b^\\theta\\Vert^2+\\Vert s\\Vert^2))+O(W_{\\tau}(\\nu)^4)$.\n\\end{itemize}\nIt follows that overall, the error term is in $O(\\tau (\\Vert \\tilde b^\\theta\\Vert^2+\\Vert s\\Vert^2)+W_{\\tau}(\\nu)^3)$.\nThere remains to lower bound the norm of the gradient over $\\Theta_0$, which can be done as follows. As seen in the proof of Lemma~\\ref{lem:distancebound}, if $\\tau$ is small enough then $J'_{\\nu^\\star}(\\theta)\\geq \\tau^2\\sigma_{\\min}(H)/4$ for $\\theta \\in \\Theta_0$. Considering only the first component of the gradient, it holds\n\\[\n\\int_{\\Theta_0} \\Vert\\bar \\nabla J'_\\nu(\\theta) \\Vert^2\\d\\nu(\\theta) \\geq \\int_{\\Theta_0} 4 \\vert J'_\\nu(\\theta)\\vert^2 \\d\\nu(\\theta).\n\\]\nUsing the expansion $J'_\\nu(\\theta)=J'_{\\nu^\\star}(\\theta)+\\langle \\phi(\\theta),M_{\\nabla R,1}(f^\\star,f_\\nu)\\rangle$, we get\n\\[\n\\int_{\\Theta_0} \\Vert\\bar \\nabla J'_\\nu(\\theta) \\Vert^2\\d\\nu(\\theta) \\geq  C \\bar r_0^2 \\tau^4 + O(W_\\tau(\\nu)^3).\n\\]\nThe result follows by collecting all the estimates above.\n\\end{proof}","sha256_utf8":"fbba79f1a1e959b4006596437c73894f1c01ab1c96506c55cbe42a9a53087705"},{"start":122102,"end":123807,"start_line":1226,"end_line":1250,"raw_tex":"\\subsection{Proof of the sharpness inequality (Theorem~\\ref{th:gradientinequality})}\\label{app:conclusionlocal}\nBy Proposition~\\ref{prop:localexpansion} we have that for $\\tau>0$ small enough\n\\[\n J(\\nu)-J^\\star \\leq CW_\\tau(\\nu)^2 + O(W_\\tau(\\nu)^3)\n\\]\nwhere $C = \\sigma_{\\max}(K+H) +\\Vert J'_{\\nu^\\star}\\Vert_\\infty$.\n\nSimilarly, by Proposition~\\ref{prop:gradientestimate}, for $\\tau$ small enough, it holds\n\\[\n\\Vert g_\\nu\\Vert^2_{L^2(\\nu)}  \\geq C'W_\\tau(\\nu)^2 +O(W_\\tau(\\nu)^3)\n\\]\nwhere $C'= \\frac18\\sigma_{\\min}(H)^2\\tau^4$. Now fix $\\tau>0$ satisfying the hypothesis of Lemma~\\ref{lem:distancebound} and the two previous inequalities. By Lemma~\\ref{lem:distancebound},  $W_\\tau(\\nu) = O((J(\\nu)-J^\\star)^\\frac12)$. We deduce that there exists $J_0>J^\\star$ and $\\kappa_0>0$ , such that whenever  $\\nu\\in \\Mm_+(\\Theta)$ satisfies $J(\\nu)<J_0$, one has\n\\[\n\\Vert g_\\nu\\Vert^2_{L^2(\\nu)}\\geq \\kappa_0(J(\\nu)-J^\\star).\n\\]\nFinally, notice that if different metric factors $(\\alpha,\\beta)\\neq (1,1)$ are introduced, one can always lower bound the new gradient squared norm as\n\\[\n\\int_\\Theta \\left( 4 \\alpha \\vert J'_\\nu(\\theta)\\vert^2 +\\beta \\Vert \\nabla J'_\\nu(\\theta)\\Vert^2_\\theta \\right)\\d\\nu(\\theta)\n\\geq \\min\\{\\alpha,\\beta\\} \\int_\\Theta \\left( 4 \\vert J'_\\nu(\\theta)\\vert^2 + \\Vert \\nabla J'_\\nu(\\theta)\\Vert^2_\\theta \\right)\n \\]\n which proves the statement for any $(\\alpha,\\beta)$. Note however that if one wants to make a more quantitative bound, then there are values $(\\alpha_0,\\beta_0)$ that would lead to a better conditioning and potentially higher values for $J_0$. In this case, the factor appearing in the sharpness inequality should rather be $\\min\\{\\alpha/\\alpha_0,\\beta/\\beta_0\\}$.\n\n\n\n\n","sha256_utf8":"dc66da56015b5d5a07f848ce6121fc9b40c5cf0bca5b7a355f9a3fa60486b501"},{"start":124083,"end":124832,"start_line":1253,"end_line":1262,"raw_tex":"\\begin{lemma}\\label{lem:qdensity}\nUnder {\\sf (A1)}, there exists $C_{\\Theta}> 0$ that only depends on the curvature of $\\Theta$, such that for all $\\nu^\\star ,\\nu_0\\in \\Mm_+(\\Theta)$ where $\\nu^\\star=\\sum_{i=1}^{m^\\star} r_i^2\\delta_{\\theta_i}$ and $\\nu_0=\\rho \\vol$ where $\\log \\rho$ is $L$-Lipschitz, then \n\\[\n\\Qq_{\\nu^\\star,\\nu_0}(\\tau)\\leq \\frac1{\\tau} \\left( \\nu^\\star(\\Theta)\\cdot d\\cdot \\big(C_\\Theta +\\log(\\tau)+L/\\tau\\big) +\\nu_0(\\Theta)- \\nu^\\star(\\Theta)+ \\sum_{i=1}^{m^\\star} r_i^2 \\log\\left(\\frac{r_i^2}{\\rho(\\theta_i)}\\right) \\right).\n\\]\nMoreover, for any other $\\hat \\nu_0\\in \\Mm_+(\\Theta)$, it holds\n$\n\\Qq_{\\nu^\\star,\\hat \\nu_0}(\\tau)\\leq \\Qq_{\\nu^\\star,\\nu_0}(\\tau) + \\nu^\\star(\\Theta)\\cdot W_\\infty(\\nu_0,\\hat \\nu_0).\n$\n\\end{lemma}","sha256_utf8":"1710c6867e71cfbff0699661e2561819dda46c794c947d0c58c8747149360d13"},{"start":125376,"end":129950,"start_line":1275,"end_line":1306,"raw_tex":"\\begin{proof}\nLet us build $\\nu_\\epsilon$ in such a way that the quantity defining $\\Qq_{\\nu^\\star,\\nu_0}(\\tau)$ in Eq.~\\eqref{eq:quantization} is small. For this, consider a radius $\\epsilon>0$ and consider the measure $\\nu_\\epsilon$ defined as the normalized volume measure on each geodesic ball of radius $\\tau$ around each $\\theta_i$, with mass $r_i^2$ on this ball, and vanishing everywhere else. Using the transport map that maps these balls to their centers $\\theta_i$, we get if $\\Theta$ is flat,\n\\[\n\\Vert \\nu_\\epsilon -\\nu^\\star\\Vert_\\BL^* \\leq W_1(\\nu_\\epsilon,\\nu^\\star)  \\leq \\sum_{i=1}^{m^\\star} \\frac{r_i^2}{V^{(d)}(\\epsilon)} \\int_0^\\epsilon s \\frac{d}{ds}V^{(d)}(s)\\d s\n\\]\nwhere $V^{(d)}(\\epsilon)$ is the volume of a ball of radius $\\epsilon$ in $\\RR^d$, that scales as $\\epsilon^d$. Using an integration by parts, it follows\n\\[\n\\frac{1}{V^{(d)}(\\epsilon)} \\int_0^\\epsilon s \\frac{d}{ds}V^{(d)}(s)\\d s = \\epsilon - \\int_0^\\epsilon \\frac{V^{(d)}(s)}{V^{(d)}(\\epsilon)}\\d s = \\epsilon -\\int_0^\\epsilon\\left(\\frac{s}{\\epsilon}\\right)^d\\d s = \\frac{\\epsilon d}{d+1},\n\\]\nthus $W_1(\\nu_\\epsilon ,\\nu^\\star)  \\leq \\nu^\\star(\\Theta)\\epsilon$. In the general case where $\\Theta$ is a potentially curved manifold, this upper bound also depends on the curvature of $\\Theta$ around each $\\theta_i$, a dependency that we hide in the multiplicative constant so $W_1(\\nu_\\epsilon ,\\nu^\\star)  \\leq C\\nu^\\star(\\Theta)\\epsilon$. Let us now control the entropy term. Writing $\\rho_\\epsilon =\\d\\nu_\\epsilon/\\d{\\vol}$ and $\\Theta_i$ for the geodesic ball of radius $\\epsilon$ around $\\theta_i$, it holds\n\\begin{align*}\n\\Hh(\\nu_\\epsilon,\\nu_0) & = \\nu_0(\\Theta) -\\nu^\\star(\\Theta) + \\sum_{i=1}^{m^\\star} \\int_{\\Theta_i} \\rho_\\epsilon(\\theta)\\log (\\rho_\\epsilon(\\theta)/\\rho(\\theta))\\d{\\vol}(\\theta).\n\\end{align*}\nThe integral term can be estimated as follows,\n\\begin{align*}\n\\int_{\\Theta_i} \\rho_\\epsilon(\\theta)  \\log (\\rho_\\epsilon(\\theta)/\\rho(\\theta))\\d{\\vol}(\\theta) &= \\int_{\\Theta_i} \\frac{r_i^2}{V^{(d)}(\\epsilon)} \\left(\\log (r_i^2)-\\log V^{(d)}(\\epsilon) -\\log(\\rho(\\theta))\\right)\\d{\\vol}(\\theta)\\\\\n& \\leq r_i^2\\left(\\log(r_i^2)-\\log V^{(d)}(\\epsilon) -\\log \\rho(\\theta_i) +\\Lip(\\log \\rho)\\cdot \\epsilon \\right).\n\\end{align*}\nRecalling that $-\\log V^{(d)}(\\epsilon) \\leq -d\\log(\\epsilon)+C$ for some $C$ that only depends on the curvature of $\\Theta$, we get that the right-hand side of~\\eqref{eq:quantization} is bounded by\n\\begin{align*}\n \\frac1{\\tau} \\left(C\\nu^\\star(\\Theta)+\\nu_0(\\Theta) -\\nu^\\star(\\Theta) d\\log(\\epsilon) + \\sum_{i=1}^{m^\\star} r_i^2 \\log\\left(\\frac{r_i^2}{\\rho(\\theta_i)}\\right) + \\epsilon L\\nu^\\star(\\Theta)\\right) + C\\nu^\\star(\\Theta)\\epsilon.\n\\end{align*}\nLet us fix $\\epsilon>0$ by minimizing $C\\nu^\\star(\\Theta)\\epsilon -\\nu^\\star(\\Theta)d\\log(\\epsilon)/\\tau$, which gives $\\epsilon = d/(C\\tau)$. The first claim follows by plugging this value for $\\epsilon$ in the expression above.\n\nFor the second claim of the statement, let us build a suitable candidate $\\hat \\nu_\\epsilon$ in order to upper bound the infimum that defines $\\Qq_{\\nu^\\star,\\hat \\nu_0}(\\tau)$. Let $T$ be an optimal transport map from $\\nu_0$ to $\\hat \\nu_0$ for $W_\\infty$, i.e.\\ a measurable map $T:\\Theta\\to\\Theta$ satisfying $T_\\# \\nu_0 = \\hat \\nu_0$ and $\\max\\{\\dist(\\theta,T(\\theta))\\;;\\; \\theta\\in \\spt \\nu_0(=\\Theta)\\} = W_\\infty(\\nu_0,\\hat \\nu_0)$ (see~\\cite[Sec. 3.2]{santambrogio2015optimal}, the absolute continuity of $\\nu_0$ is sufficient for such a map to exist). Now we define $\\hat \\nu_\\epsilon = T_\\# \\nu_\\epsilon$ where $\\nu_\\epsilon$ is such that $\\Hh(\\nu_\\epsilon,\\nu_0)<\\infty$. Since the relative entropy is non-increasing under pushforwards, it holds $\\Hh(\\hat \\nu_\\epsilon,\\hat \\nu_0)\\leq \\Hh(\\nu_\\epsilon,\\nu_0)$. Moreover, it holds $\\Vert \\nu_\\epsilon -\\hat \\nu_\\epsilon\\Vert_{\\BL}^*\\leq W_1(\\nu_\\epsilon,\\hat \\nu_\\epsilon)\\leq \\nu^\\star(\\Theta) W_\\infty(\\nu_\\epsilon,\\hat \\nu_\\epsilon)$. Thus we have\n\\begin{align*}\n\\Qq_{\\nu^\\star,\\hat \\nu_0}(\\tau) &\\leq \\Vert \\nu^\\star - \\hat \\nu_\\epsilon\\Vert_{\\BL}^* +\\frac1\\tau \\Hh(\\hat \\nu_\\epsilon,\\hat \\nu_0) \\\\\n&\\leq \\Vert \\nu_\\epsilon -\\hat \\nu_\\epsilon\\Vert_{\\BL}+ \\Vert \\nu^\\star - \\nu_\\epsilon\\Vert_{\\BL} +\\frac1\\tau \\Hh(\\nu_\\epsilon,\\nu_0)\\\\\n&\\leq \\nu^\\star(\\Theta) W_\\infty(\\nu_\\epsilon,\\hat \\nu_\\epsilon) + \\Vert \\nu^\\star - \\nu_\\epsilon\\Vert_{\\BL}^* +\\frac1\\tau \\Hh(\\nu_\\epsilon,\\nu_0) .\n\\end{align*}\nThe claim follows by noticing that,  by construction, $W_\\infty(\\nu_\\epsilon,\\hat \\nu_\\epsilon) \\leq W_\\infty(\\nu_0,\\hat \\nu_0)$ and then by taking the infimum in $\\nu_\\epsilon$.\n\\end{proof}","sha256_utf8":"4b2d7d9d1a738943ad091620a2404537f109ecce9e87f0ce2453dd139717dcf5"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"梯度流的定量全局收敛","source_units":[{"start":66643,"end":67885,"start_line":628,"end_line":638,"raw_tex":"\\begin{theorem}[Global convergence of gradient flow]\\label{thm:convergenceflow}\nUnder {\\sf (A1-5)}, let $J_0$ and $\\kappa_0$ given by Theorem~\\ref{th:gradientinequality}, let $\\rho = \\rho\\!\\vol \\in \\Mm_+(\\Theta)$ an absolutely continuous reference measure with $\\log \\rho$ $L$-Lipschitz and let  $B_{\\nu_0} = \\sup_{J(\\nu)\\leq J(\\nu_0)} \\Vert J'_{\\nu}\\Vert_{\\BL}$, where $\\nu_0\\in \\Mm_+(\\Theta)$ is the initialization. For any $0<\\epsilon\\leq 1/2$, there exists $C_\\epsilon >0$ that only depend on $\\epsilon$ and bounds on the curvature of $\\Theta$ such that if it holds $\\beta/\\alpha \\leq (4B_{\\nu_0}/\\max\\{1,L\\})^2$,\n\\begin{align*}\nW_\\infty(\\nu_0,\\rho)\\leq \\frac{J_0-J^\\star}{2B_{\\nu_0}\\nu^\\star(\\Theta)} &&\\text{and}&&\n\\frac{\\beta}{\\alpha} \\leq\\left(  \\frac{J_0-J^\\star}{2(4B_{\\nu_0})^{\\epsilon}\\left(\\bar \\Hh(\\nu^\\star,\\rho) +B^2_{\\nu_0} +C_\\epsilon \\nu^\\star(\\Theta)d \\right)}\\right)^\\frac{2}{1-\\epsilon}\n\\end{align*}\n then the projected gradient flow $(\\nu_t)_{t\\geq 0}$ initialized with $\\nu_0$ converges to the global minimizer $\\nu^\\star$. Denoting $t_0 = 1/\\sqrt{\\alpha\\beta}$ it satisfies, for $t\\geq t_0$,\n\\begin{align*}\n J(\\nu_t)-J^\\star  \\leq  (J_0 - J^*) \\exp(-2\\kappa_0\\min\\{\\alpha,\\beta\\} (t-t_0)).\n \\end{align*}\n\\end{theorem}","sha256_utf8":"4b930333a517afb3fe9fae71318c8191732e66312e0493877ad23a0d077325aa"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"全局收敛完整证明","source_units":[{"start":75548,"end":77456,"start_line":699,"end_line":717,"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm:convergenceflow} (gradient flow)]\nBy Lemma~\\ref{lem:qdensity}, we have for $\\tau\\geq L=\\Lip(\\log \\rho)$, by writing $\\bar \\Hh \\coloneqq \\bar \\Hh(\\nu^\\star,\\rho\\vol)$,\n\\[\n\\Qq_{\\nu^\\star,\\hat \\nu_0}(\\tau) \\leq \\frac{\\bar \\Hh+d\\cdot \\nu^\\star(\\Theta)\\cdot (\\log(\\tau)+C_\\Theta)}\\tau + \\nu^\\star(\\Theta)\\cdot W_\\infty(\\nu_0,\\rho\\vol).\n\\]\nCombining this bound with Lemma~\\ref{lem:mirrordecrease}, we get that for $t\\geq L/(4\\alpha B_{\\nu_0})$,\n\\[\nJ(\\nu_t)-J^\\star \\leq \\frac{\\bar \\Hh+d\\cdot\\nu^\\star(\\Theta)\\cdot(\\log(4\\alpha B_{\\nu_0} t)+C_\\Theta)}{4\\alpha B_{\\nu_0} t} + B_{\\nu_0}\\cdot\\nu^\\star(\\Theta)\\cdot W_\\infty(\\nu_0,\\rho\\vol)+\\beta B^2_{\\nu_0} t.\n\\]\nIn particular, for $t=(\\alpha\\beta)^{-\\frac12}$, we get \n\\begin{equation}\\label{eq:lastbeforelast}\nJ(\\nu_t)-J^\\star \\leq \\sqrt{\\frac{\\beta}{\\alpha}}\\left(\\bar \\Hh + d\\cdot \\nu^\\star(\\Theta) (\\log(4B_{\\nu_0}\\sqrt{\\alpha/\\beta})+C_\\Theta) + B^2_{\\nu_0}\\right) + B_{\\nu_0}\\nu^\\star(\\Theta)\\cdot W_\\infty(\\nu_0,\\rho\\vol).\n\\end{equation}\nSince this is valid only when $t\\geq L/(4\\alpha B_{\\nu_0})$, we require $(\\alpha\\beta)^{-\\frac12}\\geq L/(4\\alpha B_{\\nu_0})$ which leads to the first condition on $\\beta/\\alpha$. Now, we want the right-hand side of~\\eqref{eq:lastbeforelast} to be smaller than $\\Delta_0 \\coloneqq J_0 - J^\\star$ so that we can conclude with Corollary~\\ref{cor:localGF}. To this end, we require, on the one hand $W_\\infty(\\nu_0,\\rho\\vol) \\leq \\Delta_0/(2B_{\\nu_0}\\nu^\\star(\\Theta))$. On the other hand, we use the bound $\\log(u)\\leq C_\\epsilon u^\\epsilon$ for $\\epsilon\\in {]0,1/2]}$, require $4B_{\\nu_0}\\sqrt{\\alpha/\\beta}\\geq 1$ and obtain the condition\n\\[\n(4B_{\\nu_0})^\\epsilon(\\beta/\\alpha)^{(1-\\epsilon)/2}\\left( \\bar \\Hh + d\\cdot \\nu^\\star(\\Theta)(C_\\Theta+C_\\epsilon) +B^2_{\\nu_0}\\right)\\leq \\frac12\\Delta_0.\n\\]\nThis leads to the second condition on $\\beta/\\alpha$ is the theorem.\n\\end{proof}","sha256_utf8":"2fcc7fc5e73682a4b2dc0e19717e89589b8acdb1c6f092eb0c8c7d61637d39be"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"代入镜像速率估计","source_units":[{"start":75622,"end":75922,"start_line":700,"end_line":703,"raw_tex":"By Lemma~\\ref{lem:qdensity}, we have for $\\tau\\geq L=\\Lip(\\log \\rho)$, by writing $\\bar \\Hh \\coloneqq \\bar \\Hh(\\nu^\\star,\\rho\\vol)$,\n\\[\n\\Qq_{\\nu^\\star,\\hat \\nu_0}(\\tau) \\leq \\frac{\\bar \\Hh+d\\cdot \\nu^\\star(\\Theta)\\cdot (\\log(\\tau)+C_\\Theta)}\\tau + \\nu^\\star(\\Theta)\\cdot W_\\infty(\\nu_0,\\rho\\vol).\n\\]\n","sha256_utf8":"4998d5c1a6ea508514a6985925a6ec2c8498a1fa8459be43750be230b6e0dc93"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"得到全局目标差上界","source_units":[{"start":75922,"end":76242,"start_line":704,"end_line":707,"raw_tex":"Combining this bound with Lemma~\\ref{lem:mirrordecrease}, we get that for $t\\geq L/(4\\alpha B_{\\nu_0})$,\n\\[\nJ(\\nu_t)-J^\\star \\leq \\frac{\\bar \\Hh+d\\cdot\\nu^\\star(\\Theta)\\cdot(\\log(4\\alpha B_{\\nu_0} t)+C_\\Theta)}{4\\alpha B_{\\nu_0} t} + B_{\\nu_0}\\cdot\\nu^\\star(\\Theta)\\cdot W_\\infty(\\nu_0,\\rho\\vol)+\\beta B^2_{\\nu_0} t.\n\\]\n","sha256_utf8":"68e0d6f2961b9a174c1625d4e5f5748e6ce5790c85ab960f03f7b3c96c9c0ea2"}]},{"id":"S005","semantic_type":"proof_step","kind":"proof_step","title":"选择进入局部区间的时间","source_units":[{"start":76242,"end":76575,"start_line":708,"end_line":711,"raw_tex":"In particular, for $t=(\\alpha\\beta)^{-\\frac12}$, we get \n\\begin{equation}\\label{eq:lastbeforelast}\nJ(\\nu_t)-J^\\star \\leq \\sqrt{\\frac{\\beta}{\\alpha}}\\left(\\bar \\Hh + d\\cdot \\nu^\\star(\\Theta) (\\log(4B_{\\nu_0}\\sqrt{\\alpha/\\beta})+C_\\Theta) + B^2_{\\nu_0}\\right) + B_{\\nu_0}\\nu^\\star(\\Theta)\\cdot W_\\infty(\\nu_0,\\rho\\vol).\n\\end{equation}\n","sha256_utf8":"fbb471f55b10daa88c26b2ca7336feb2fc3a98fd070cc086b4361820b7354e50"}]},{"id":"S006","semantic_type":"proof_step","kind":"proof_step","title":"核验条件并接续指数收敛","source_units":[{"start":76575,"end":77445,"start_line":712,"end_line":716,"raw_tex":"Since this is valid only when $t\\geq L/(4\\alpha B_{\\nu_0})$, we require $(\\alpha\\beta)^{-\\frac12}\\geq L/(4\\alpha B_{\\nu_0})$ which leads to the first condition on $\\beta/\\alpha$. Now, we want the right-hand side of~\\eqref{eq:lastbeforelast} to be smaller than $\\Delta_0 \\coloneqq J_0 - J^\\star$ so that we can conclude with Corollary~\\ref{cor:localGF}. To this end, we require, on the one hand $W_\\infty(\\nu_0,\\rho\\vol) \\leq \\Delta_0/(2B_{\\nu_0}\\nu^\\star(\\Theta))$. On the other hand, we use the bound $\\log(u)\\leq C_\\epsilon u^\\epsilon$ for $\\epsilon\\in {]0,1/2]}$, require $4B_{\\nu_0}\\sqrt{\\alpha/\\beta}\\geq 1$ and obtain the condition\n\\[\n(4B_{\\nu_0})^\\epsilon(\\beta/\\alpha)^{(1-\\epsilon)/2}\\left( \\bar \\Hh + d\\cdot \\nu^\\star(\\Theta)(C_\\Theta+C_\\epsilon) +B^2_{\\nu_0}\\right)\\leq \\frac12\\Delta_0.\n\\]\nThis leads to the second condition on $\\beta/\\alpha$ is the theorem.\n","sha256_utf8":"78ec9b3b82f03bb12843cd3629b96e7b622da4f0c03ce8da9510bfab96abb31f"}]},{"id":"S007","semantic_type":"problem_setting","kind":"problem_setting","title":"带质量惩罚的测度优化","source_units":[{"start":5517,"end":6505,"start_line":169,"end_line":173,"raw_tex":"Finding parsimonious descriptions of complex observations is an important problem in machine learning and signal processing.  In its simplest form, this task boils down to searching for an element in a Hilbert space $\\Ff$ that is close to a certain $f_0\\in \\Ff$ --- the observations --- and that is a linear combination of a few elements from a parameterized set $\\{\\phi(\\theta)\\}_{\\theta \\in \\Theta}\\subset \\Ff$ --- the parsimonious description. This can be formulated as a minimization problem where the linear combination is expressed through an unknown measure $\\nu$ and the distance to $f_0$ is quantified using a smooth convex loss function $R:\\Ff\\to \\RR$, such as the square loss $R(f)=\\frac12 \\Vert f-f_0\\Vert^2_\\Ff$.  The problem to solve is then\n%\n\\begin{equation}\\label{eq:objective}%\\tag{$\\Pp_\\lambda$}\nJ^*\\coloneqq \\min_{\\nu \\in \\Mm_+(\\Theta)} J(\\nu), \\qquad \\text{}\\qquad J(\\nu) :=R\\left( \\int_\\Theta \\phi(\\theta) \\d \\nu(\\theta)\\right) + \\lambda  \\nu (\\Theta)\n\\end{equation}","sha256_utf8":"2537496c6b6c691d8af927c4a3b52480ed12deeee695548a22d18fc58d0f2f29"},{"start":6508,"end":7897,"start_line":175,"end_line":175,"raw_tex":"where $\\Mm_+(\\Theta)$ is the set of nonnegative measures $\\nu$ on the parameter space $\\Theta$ with finite total mass $\\nu(\\Theta)<\\infty$ and $\\lambda> 0$ is the regularization strength. This formulation also covers minimization over signed measures with total variation regularization, by replacing $\\Theta$ with the disjoint union of two copies of $\\Theta$ where $\\phi$ takes opposite values, see Appendix~\\ref{app:signed}. A large body of research has exhibited the favorable properties of minimizers of such problems~\\cite{bach2017breaking, de2012exact, mairal2014sparse} with a statistical or variational viewpoint, showing in particular that $\\lambda$ favors sparser solutions and increases stability as it gets larger, at the expense of introducing a stronger bias. The present paper deals with the optimization aspect: our goal is to design algorithms that return $\\epsilon$-accurate solutions with a guaranteed computational complexity. When the set $\\Theta$ is a finite set, this is a finite dimensional convex optimization problem that is well understood~\\cite{boyd2004convex, bach2012optimization}. However, convex approaches are generally inefficient when $\\Theta$ is a continuous space, such as a $d$-dimensional manifold, where the need to discretize the space leads to a complexity scaling as $\\epsilon^{-d}$ in the accuracy $\\epsilon$. We consider the following setting:\n","sha256_utf8":"84ba96cb38e010ee52f4b85e13d3ce829fd58277e8822d9c306c29c68e0742de"}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","title":"紧流形与光滑性假设","source_units":[{"start":7897,"end":8214,"start_line":176,"end_line":178,"raw_tex":"\\begin{enumerate}\n\\item[] {\\sf (A1)}  $\\Theta$ is a compact $d$-dimensional Riemannian manifold without boundaries. The functions $\\phi:\\Theta\\to \\Ff$ and $R:\\Ff\\to \\RR_+$ are twice Fr\\'echet differentiable, with locally Lipschitz second-order derivatives, and $\\nabla R$ is bounded on sublevel sets.\n\\end{enumerate}\n","sha256_utf8":"d0c913c73245edaea49162291a97ae1156ab0c307944759ba2fef4744ff2ec7a"}]},{"id":"S009","semantic_type":"notation","kind":"notation","title":"测度熵与距离记号","source_units":[{"start":19046,"end":20661,"start_line":249,"end_line":255,"raw_tex":"The set of signed (resp.~nonnegative) finite Borel measures on a metric space $(\\Xx,\\dist)$ is denoted by $\\Mm(\\Xx)$ (resp. $\\Mm_+(\\Xx)$). The relative entropy, a.k.a.~Kullback-Leibler divergence, is defined for $\\nu_1,\\nu_2 \\in \\Mm_+(\\Xx)$ as\n$\\Hh (\\nu_1,\\nu_2) = \\int_{\\Xx} \\log(\\d\\nu_1/\\d\\nu_2) \\d\\nu_1 - \\nu_1(\\Xx)+\\nu_2(\\Xx)$ if $\\nu_1$ is absolutely continuous w.r.t.~$\\nu_2$, and $+\\infty$ otherwise. The $p$-Wasserstein distance on the set $\\Pp_p(\\Xx)$ of probability measures with finite $p$-th moment is defined, for $\\mu_1,\\mu_2 \\in \\Pp_p(\\Xx)$ as \n\\[\nW_p(\\mu_1,\\mu_2) = \\left(\\min_{\\gamma \\in \\Pi(\\mu_1,\\mu_2)} \\int \\dist(x_1,x_2)^p\\d\\gamma(x_1,x_2)\\right)^{1/p}\n\\]\nwhere $\\Pi(\\mu_1,\\mu_2)$ is the set of measures on $\\Xx \\times \\Xx$ with marginals $\\mu_1$ and $\\mu_2$. The distance $W_\\infty$ between compactly supported probabilities is defined as the limit of $W_p$ as $p\\to \\infty$ and can be directly defined as $W_\\infty(\\mu_1,\\mu_2) = \\inf_{\\gamma \\in \\Pi(\\mu_1,\\mu_2)} \\max_{(x_1,x_2)\\in \\spt \\gamma} \\dist(x_1,x_2)$~\\cite{santambrogio2015optimal}. We also define the Bounded-Lipschitz norm for a continuous function $\\psi:\\Xx\\to \\RR$ as $\\Vert \\psi \\Vert_{\\BL} = \\Vert \\psi\\Vert_\\infty+\\Lip(\\psi)$ where $\\Lip(\\psi)$ is the Lipschitz constant of $\\psi$ and its dual norm on $\\Mm(\\Xx)$ as \n$\\Vert \\nu \\Vert^*_{\\BL} \\coloneqq \\sup_{\\Vert \\varphi \\Vert_{\\BL}\\leq 1} \\int \\varphi \\, \\d\\nu$. For a Riemannian manifold $\\Theta$, we denote by $T_\\theta\\Theta$ the tangent space of $\\Theta$ at $\\theta$ and by $\\langle \\cdot,\\cdot\\rangle_\\theta : (T_\\theta \\Theta)^2 \\to \\RR_+$ the metric at $\\theta$.\n","sha256_utf8":"28a655ed4d60106cf67b499fc9379de69d8d8740cced4eccb4659eb2d0fb8e59"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"粒子参数化与状态空间","source_units":[{"start":20748,"end":21357,"start_line":259,"end_line":262,"raw_tex":"Consider a smooth increasing bijection $h:\\RR_+\\to \\RR_+$ (such as a power function $r\\mapsto r^p$) and a number of particles $m \\in \\NN^*$. The idea behind particle gradient-based algorithms is to parameterize the unknown measure $\\nu$ as $\\frac1m \\sum_{i=1}^m h(r_i) \\delta_{\\theta_i}$ and to perform gradient-based optimization on the corresponding objective\n\\begin{equation}\\label{eq:objectivediscrete}%\\tag{$\\widehat \\Pp^m_\\lambda$}\nF_{m}((r_1,\\theta_1),\\dots,(r_{m},\\theta_m)) \\coloneqq  R\\left(\\frac1{m} \\sum_{i=1}^{m} h(r_i)\\phi(\\theta_i)\\right) + \\frac\\lambda{m} \\sum_{i=1}^{m} h(r_i),\n\\end{equation}","sha256_utf8":"bee99d2f14b8bdf1468b0b065717bd4f32c35de18b0bed661a7b92b71c7cb85a"},{"start":21358,"end":22398,"start_line":263,"end_line":263,"raw_tex":"where the parameters $(r_i,\\theta_i)$ of each particle belong to $\\Omega \\coloneqq \\RR_+ \\times \\Theta$ endowed with a specific choice of metric. Clearly, if $J$ admits a minimizer that is a mixture of $m^\\star$ atoms with $m^\\star\\leq m$, then it is sufficient to minimize $F_m$ from Eq.~\\ref{eq:objectivediscrete} for solving \\eqref{eq:objective}. While~\\eqref{eq:objectivediscrete} is finite dimensional, it is typically non-convex with possibly some strict local minima. Still, when $R$ is convex and for $h(r)=r^p$ for $p\\in \\{1,2\\}$, the message from~\\cite{chizat2018global}  (see Theorem~\\ref{th:consistency}) is that  solving~\\eqref{eq:objectivediscrete} to global optimality with first-order methods is still possible by using over-parameterization, i.e.\\ choosing $m$ much larger than $m^\\star$.  Such a method involve various key hyper-parameters which role is discussed throughout the paper. They include (i) the choice of the function $h$ (ii) the choice of the metric on $\\Omega^m$ and (iii) the choice of the initialization.\n","sha256_utf8":"300426f91fffac8b14e92e14deae53d411045a9e9051bf5f6fd6a935b11d64b9"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"目标泛函的一阶变分","source_units":[{"start":22399,"end":22889,"start_line":265,"end_line":268,"raw_tex":"\\paragraph{Expression of the gradient.} Under~{\\sf(A1)}, the objective $J$, seen as a function on the space $\\Mm(\\Theta)$ endowed with the total variation norm, is Fr\\'echet-differentiable. Its differential at $\\nu\\in \\Mm(\\Theta)$ can be represented by the function $J'_\\nu : \\Theta \\to \\RR$ given by\n\\begin{equation}\\label{eq:firstvariation}\nJ'_\\nu(\\theta) = \\left\\langle \\phi(\\theta),\\nabla R\\left(\\int_\\Theta \\phi(\\theta) \\d\\nu(\\theta)\\right)\\right\\rangle_{\\Ff} +\\lambda,\n \\end{equation}","sha256_utf8":"1ed3614036713ab7a61b58a7f023a658ccedfc90ee72f33c503c42bef7600167"},{"start":22890,"end":23325,"start_line":269,"end_line":270,"raw_tex":" in the sense that for any $\\sigma \\in\\Mm(\\Theta)$, it holds\n$\\frac{d}{d\\epsilon} J(\\nu + \\epsilon \\sigma)\\vert_{\\epsilon=0} = \\int_{\\Theta} J'_\\nu(\\theta) \\d\\sigma(\\theta)$. Now, consider a metric on $(\\Omega^*)^m$ that is the average $(1/m)\\sum_{i=1}^m \\langle \\cdot, \\cdot \\rangle_{(r_i,\\theta_i)}$ of metrics on each factor $\\Omega^*\\coloneqq \\RR_+^*\\times \\Theta$, where $\\RR_+^*$ is the set of positive real numbers, of the form\n","sha256_utf8":"743e545867401173e195a04d3977154fc0b56245b4e9fc1e16a4605daa768daf"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"粒子空间的黎曼度量","source_units":[{"start":23325,"end":23567,"start_line":271,"end_line":273,"raw_tex":"\\begin{equation}\\label{eq:metric}\n\\langle(\\delta r_1,\\delta \\theta_1),(\\delta r_2,\\delta \\theta_2) \\rangle_{(r,\\theta)} = \\alpha(r)^{-1} \\delta r_1\\delta r_2 +\\beta(r)^{-1}\\langle \\delta \\theta_1,\\delta\\theta_2\\rangle_{\\theta} \n\\end{equation}","sha256_utf8":"a9ac298db808147ee293c1be31f3af451affa2d02410615e7e0668da5968810d"},{"start":23568,"end":24103,"start_line":274,"end_line":274,"raw_tex":"where $\\alpha$ and $\\beta$ are smooth functions $\\RR_+^*\\to \\RR_+^*$ to be specified\\footnote{Extension of the metric and gradients to the whole of $\\Omega$ can be made on a case by case basis, see Section~\\ref{sec:conic}.}, $ (r,\\theta) \\in \\Omega^*$, $\\delta r_1, \\delta r_2 \\in \\RR$ and $\\delta \\theta_1, \\delta \\theta_2 \\in T_\\theta\\Theta$. Using the fact that gradients are characterized by the relation $\\d F_m(x)(\\delta x) = \\langle \\nabla F_m(x),\\delta x\\rangle$, we get that the gradient of $F_m$ is given, in components, by \n","sha256_utf8":"5b007dfd4eda2af95b8e3db70a507bffd4e22832e7a325371c71863464cb8f38"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"提升泛函与齐次投影","source_units":[{"start":24451,"end":25114,"start_line":284,"end_line":287,"raw_tex":"\\paragraph{Lifted problem in Wasserstein space.} Assume now that $h$ has at most quadratic growth, and that the metric is defined on the whole of $\\Omega$. One can then see the discrete problem~\\eqref{eq:objectivediscrete} as a discretization of a problem on the space $\\Pp_2(\\Omega)$ of probability measures on $\\Omega$ with finite second moment endowed with the Wasserstein-$2$ metric given by\n\\begin{equation}\\label{eq:liftedproblem}%\\tag{$\\Pp_\\lambda$}\nF^\\star = \\min_{\\mu \\in \\Pp_2(\\Omega)} F(\\mu)\\quad \\text{where}\\quad F(\\mu) \\coloneqq \\left( \\int_\\Omega h(r) \\phi(\\theta) \\d\\mu(r,\\theta) \\right) +  \\lambda \\int_\\Omega h(r) \\d\\mu(r,\\theta).\n\\end{equation}","sha256_utf8":"682089855760c1102be862a6a9236ff8a1e735b1841d82cf108994a42060f9c2"},{"start":25117,"end":25579,"start_line":289,"end_line":292,"raw_tex":"This point of view leads to insights on the properties of $F_m$ that are independent of $m$, which is crucial for our theoretical analysis. For a measure $\\mu\\in \\Pp_2(\\Omega)$, we define following~\\cite{liero2018optimal} the \\emph{homogeneous projection} operator $\\h:\\Pp_2(\\Omega)\\to \\Mm_+(\\Theta)$ where  $\\h\\mu$ is characterized by\n\\begin{align*}\n\\int_{\\Theta} \\varphi(\\theta) \\d(\\h\\mu)(\\theta) = \\int_{\\Omega} h(r)\\varphi(\\theta)\\d\\mu(r,\\theta)\n\\end{align*}","sha256_utf8":"95f78905c295c271013fb811a671a7a9cce6be83587d9eb91b2be79e43576cfe"},{"start":25580,"end":25689,"start_line":293,"end_line":294,"raw_tex":"for any continuous function $\\varphi:\\Theta\\to \\RR$. With this operator, we simply have $F(\\mu)=J(\\h\\mu)$. \n\n","sha256_utf8":"e7559a1a2f49e38bf538a06fb2e390057bbb1086f5b5d0bc782d6a873bf9629b"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"瓦瑟斯坦梯度流定义","source_units":[{"start":26578,"end":27022,"start_line":302,"end_line":305,"raw_tex":"\\paragraph{Wasserstein gradient flow.} It is also possible to directly study the optimization dynamics in the space $\\Pp_2(\\Omega)$ for the functional $F$ of Eq.~\\eqref{eq:liftedproblem}. For a measure $\\nu \\in \\Mm_+(\\Theta)$, consider the vector field on $\\Omega$ with expression\n\\begin{equation*}\ng_\\nu(r,\\theta) = \\big(\\alpha(r) h'(r) J'_{\\nu}(\\theta),\\beta(r) h(r)\\nabla J'_{\\nu}(\\theta)\\big) \\in \\RR \\times T_\\theta \\Theta.\n\\end{equation*}","sha256_utf8":"3b2bf8bddde06a6b77f365d62aad6aeb153ee9132f80f5bd1404b2669ab7ebce"},{"start":27023,"end":27458,"start_line":306,"end_line":309,"raw_tex":"We refer to $g_{\\h \\mu}$ as the Wasserstein gradient of $F$ at $\\mu$ (this notation emphasizes that it only depends on $\\mu$ through $\\h \\mu$). Gradient flows of $F_m$ are particular cases of \\emph{Wasserstein gradient flows} of $F$. The latter are defined as the absolutely continuous curves $(\\mu_t)_{t\\geq 0}$ in $\\Pp_2(\\Omega)$ that satisfy\n\\begin{equation}\\label{eq:WGF}\n\\partial_t \\mu_t =  \\div (\\mu_t g_{\\h\\mu_t})\n\\end{equation}","sha256_utf8":"dda4dfcbe61fe5664748576237c5b194d6a4c5533d419508c19be2d9e050fecb"},{"start":27459,"end":28031,"start_line":310,"end_line":310,"raw_tex":"in the weak sense, which means that for any differentiable function $\\varphi: \\Omega \\to \\RR$, it holds $\\frac{\\d}{\\d t}\\left(\\int \\varphi \\,\\d\\mu_t \\right) = - \\int \\nabla \\varphi \\cdot g_{\\h\\mu_t}\\d\\mu_t$, for almost every $t\\geq 0$, see~\\cite{santambrogio2015optimal}. This is a proper extension of the notion of gradient flow for $F_m$ in the sense that if $x(t)=(r_i(t),\\theta_i(t))_{i=1}^m$ is a gradient flow of $F_m$ then it can be directly checked that  $t\\mapsto \\mu_t = \\frac1m \\sum_{i=1}^m \\delta_{(r_i(t),\\theta_i(t))}$ is a Wasserstein gradient flow of $F$.\n","sha256_utf8":"13ad65ea5fc392f217bef6e96ec414936dd7062e3716012e3da634c3e95a99b8"}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","title":"锥结构假设","source_units":[{"start":28077,"end":28689,"start_line":313,"end_line":316,"raw_tex":"As seen in Eq.~\\eqref{eq:gradientgeneric}, the choice of the homogeneity degree and of the metric on $\\Omega$ determine a specific way to combine the \\emph{vertical}  and the \\emph{spatial} components of the gradient (along the variable $r$ and $\\theta$, respectively). From now on, we focus on what we refer to as the \\emph{conic case}, which corresponds to the following assumption:\n\\begin{itemize}\n\\item [\\sf (A2)] The mass parameterization is $h(r)=r^2$ and the metric on $\\Omega^*$ is of the form Eq.~\\eqref{eq:metric} with $(\\alpha(r),\\beta(r))=(\\alpha,\\beta/r^2)$ for some $\\alpha,\\beta>0$.\n\\end{itemize}\n","sha256_utf8":"65acb80cdcf0edc551b11ae3019523b7175c7819b10ab25b1dc84186026336c4"}]},{"id":"S016","semantic_type":"definition","kind":"definition","title":"锥距离与梯度表达式","source_units":[{"start":28689,"end":29325,"start_line":317,"end_line":317,"raw_tex":"The corresponding geodesic distance is $\\dist((r_1,\\theta_2),(r_1,\\theta_2))^2 = r_1^2 +r_2^2 -2r_1r_2\\cos_{\\pi}(\\dist(\\theta_1,\\theta_2))$ where $\\cos_\\pi(z) = \\cos(\\min\\{\\pi,z\\})$. This metric can be extended as a proper metric on $\\widetilde \\Omega$, defined as the set $\\Omega$ where the subset $\\{0\\}\\times \\Theta$ is identified to a single point, known as the \\emph{cone metric}, which is the canonical way to define a metric on $\\tilde \\Omega$~\\cite{burago2001course}. In our context, identifying $\\{0\\}\\times \\Theta$ to a single point is desirable because a particle located in this set is a ``dead'' particle carrying no mass.\n","sha256_utf8":"96e5b91855b7d54424943248033644c1241f46cf980fc31128b38dbbf04b3b3e"},{"start":29326,"end":29753,"start_line":319,"end_line":327,"raw_tex":"Plugging the metric into Eq.~\\eqref{eq:gradientgeneric} gives the gradient (extended by continuity to $\\{0\\}\\times \\Theta$)\n\\begin{align*}\n\\left\\{\n\\begin{aligned}\n\\nabla_{r_i} F_m((r_i,\\theta_i)_{i=1}^m) &= 2\\alpha r_i J'_{\\nu}(\\theta_i)\\\\\n\\nabla_{\\theta_i} F_m ((r_i,\\theta_i)_{i=1}^m) &=  \\beta \\nabla J'_{\\nu}(\\theta_i)\n\\end{aligned}\n\\right. && \\text{where} && \\nu = \\frac1m \\sum_{i=1}^m r_i^2\\delta_{\\theta_i},\n\\end{align*}","sha256_utf8":"d4a805c6b103be949ffe88684bd5686e3e2c528e82899e572cc20d9535172a3b"},{"start":29754,"end":30029,"start_line":328,"end_line":331,"raw_tex":"and the Wasserstein gradient is represented by the vector field\n\\begin{equation}\\label{eq:wassersteingradient}\ng_\\nu(r,\\theta) = \\big(2 \\alpha r J'_{\\nu}(\\theta), \\beta \\nabla J'_{\\nu}(\\theta)\\big)  \\in \\RR \\times T_\\theta \\Theta, \\forall (r,\\theta)\\in \\Omega.\n\\end{equation}","sha256_utf8":"ef0b59a4768021e9a29f625499ed65688aecf1a98195cd74731bbbb24db5ae56"}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","title":"损失凸性假设","source_units":[{"start":45873,"end":46017,"start_line":457,"end_line":460,"raw_tex":"In order to derive global optimality conditions, we assume the following.\n\\begin{itemize}\n\\item [\\sf(A3)] The loss $R$ is convex.\n\\end{itemize}\n","sha256_utf8":"af68c94f8e3644914e8d3e61c99354cf25c392c2f30776f884a42a2606a9ab4e"}]},{"id":"S018","semantic_type":"assumption","kind":"assumption","title":"唯一稀疏极小解假设","source_units":[{"start":47621,"end":48461,"start_line":471,"end_line":474,"raw_tex":"\\begin{itemize}\n\\item [\\sf(A4)] Problem~\\eqref{eq:objective} admits a unique global minimizer on $\\Mm_+(\\Theta)$ which is of the form $\\nu^\\star = \\sum_{i=1}^{m^\\star} r_i^2\\delta_{\\theta_i}$ with $\\nu^\\star(\\Theta)>0$. We denote $f^\\star \\coloneqq \\int \\phi \\d\\nu^\\star = \\sum_{i=1}^{m^\\star}r_i^2\\phi(\\theta_i)$.\n\\end{itemize}\nWithout loss of generality, we assume $r_i>0$ for all $i$ and $\\theta_i \\neq \\theta_{i'}$ whenever $i\\neq i'$, so that $(r_i,\\theta_i)_{i=1}^{m^\\star}$ is uniquely well-defined, up to re-ordering. Let us fix from now on normal coordinates frames on the neighborhood of each $\\theta_i$. This allows to identify tensors at $\\theta_i$ with their expression in coordinates and also induces a set of coordinates on the direct sum of the tangent spaces $T_{\\theta_i}\\Theta$, which is of dimension $m^\\star\\times d$. \n","sha256_utf8":"fe41cf2c23b4b23f4895ca6380bb6262ce3ba65aff2e48a56ec0f0d392ad588f"}]},{"id":"S019","semantic_type":"definition","kind":"definition","title":"全局核与局部曲率核","source_units":[{"start":48503,"end":48721,"start_line":477,"end_line":480,"raw_tex":"We define the \\emph{global kernel} $K\\in \\RR^{(m^\\star\\times (1+d))^2}$ by\n\\[\nK_{(i,j),(i',j')} \\coloneqq \\langle r_i \\bar \\nabla_j \\phi(\\theta_i), r_{i'} \\bar \\nabla_{j'}\\phi(\\theta_{i'})\\rangle_{d^2\\! R_{f^\\star}}\n\\]","sha256_utf8":"0e990a31c9860a4708fb3b323cfcf017766c82b9982eb3732a4ef81349d328c1"},{"start":48722,"end":49285,"start_line":481,"end_line":484,"raw_tex":"where $\\bar\\nabla \\phi \\coloneqq (2 \\alpha \\phi, \\beta \\nabla \\phi)$ can be interpreted as the gradient of $\\h\\phi$ at $(1,\\theta)$. Remark that $K$ is defined via the quadratic form associated to the Hessian of $R$ at $f^\\star$. This interaction kernel $K$ appears naturally in the various statistical and optimization analysis of the minimization problem under consideration~\\cite{duval2015exact,traonmilin2018basins}. We also use the notation for the \\emph{local kernels}  for $i\\in 1,\\dots,m^\\star$\n\\[\nH_i\\coloneqq \\beta^2 \\nabla^2 J'_{\\nu^\\star}(\\theta_i)\n\\]","sha256_utf8":"92fb13859bf8f6f9ee3dfdcae8e0fbb74beeebc763b1e184ed5488e5b8065ae1"},{"start":49286,"end":49471,"start_line":485,"end_line":485,"raw_tex":"expressed in local coordinates.  In order to simplify notations, we concatenate these matrices in a large matrix $H\\in \\RR^{(m^\\star\\times (1+d))^2}$ of the same size as $K$ defined as\n","sha256_utf8":"de65c675cd5dc163ea555dab2876167aae9bb3db2923c1199521e52adc175980"},{"start":49472,"end":49647,"start_line":486,"end_line":491,"raw_tex":"\\[\n H_{(i,j),(i,j')} =\\begin{cases}\n \\beta^2  \\nabla^2_{j,j'} J'_{\\nu^\\star}(\\theta_i) &\\text{if $i=i'$ and $j,j'\\geq 1$},\\\\\n  0 &\\text{ if $j=0$ or $j'=0$}.\n  \\end{cases}\n \\]","sha256_utf8":"73dfdd99cdea761fde5d76df42baa8d58ea48adf522959b4e13b03e17a6fb2c0"},{"start":49648,"end":49803,"start_line":492,"end_line":492,"raw_tex":"where here and in the proofs, we use $0$ to label the $r$'s coordinate. The local analysis will be carried under the following non-degeneracy assumptions.\n","sha256_utf8":"58fd966bc2e08bf7ba1e9967aa70b7debd9758cff07a2cebbc771f38155f6556"}]},{"id":"S020","semantic_type":"assumption","kind":"assumption","title":"曲率与严格松弛假设","source_units":[{"start":49803,"end":50267,"start_line":493,"end_line":495,"raw_tex":"\\begin{itemize}\n\\item [\\sf(A5)] The minimizer $\\nu^\\star$ is non-degenerate in the sense that $\\nabla^2 R(f^\\star)$ is positive definite and, calling $\\sigma_{\\min}(A)$ the smallest singular value of a linear operator $A$, we have global curvature  $\\sigma_{\\min}(K)>0$, local curvature  $\\sigma_{\\min}(H) = \\min_{i} \\sigma_{\\min}(H_i)>0$, and strict slackness, i.e.\\ the only points where $J'_{\\nu^\\star}$ vanishes are $(\\theta_i)_{i=1}^{m^\\star}$.\n\\end{itemize}\n","sha256_utf8":"b3613b703a792159eaa09dc3305644c3adc9d43a99f022f9eb0182bc01d144ef"}]},{"id":"S021","semantic_type":"definition","kind":"definition","title":"非归一化运输距离","source_units":[{"start":50974,"end":51660,"start_line":500,"end_line":504,"raw_tex":"A first consequence of these assumptions is that convergence in value implies convergence to minimizers. The distance on $\\Mm_+(\\Theta)$ that naturally appears in the analysis is the Wasserstein-Fisher-Rao, a.k.a.\\ Hellinger-Kantorovich metric $\\widehat W_2$, which is the extension of the Wasserstein $W_2$ metric to unnormalized measures. It admits many equivalent definitions~\\cite{liero2018optimal, kondratyev2016new, chizat2018unbalanced}, the most suitable to our context being~\\cite[\nThm. 7.20]{liero2018optimal}\n\\[\n\\widehat W_2(\\nu_1,\\nu_2) \\coloneqq \\min \\left \\{W_2(\\mu_1,\\mu_2)\\;;\\; (\\mu_1,\\mu_2)\\in \\Pp_2(\\Omega)^2 \\text{ satisfy } (\\h\\mu_1,\\h\\mu_2)=(\\nu_1,\\nu_2)\\right\\}\n\\]","sha256_utf8":"4441a86d714d690bf582bb06c661cc1f7c86f0c62d2d2212a02ca38f85ae35c8"},{"start":51661,"end":51961,"start_line":505,"end_line":505,"raw_tex":"where the Wasserstein distance on $\\Omega$ is defined relative to the cone metric (in this paragraph, with $\\alpha=\\beta=1$). The proof of the following result involves the construction of a transport map in the lifted space $\\Pp_2(\\Omega)$ and is postponed to Appendix~\\ref{app:distanceinequality}.\n","sha256_utf8":"43ee6d31719fa6f22eef575e641f1c8e0c6c922a255d55bd0d9b253fd01eabb7"}]},{"id":"S022","semantic_type":"definition","kind":"definition","title":"梯度平方范数","source_units":[{"start":52327,"end":52700,"start_line":511,"end_line":511,"raw_tex":" Our first main result is a lower bound on the squared norm of the gradient in terms of the sub-optimality gap, an inequality known as sharpness, or Polyak-\\L{}ojasiewicz inequality~\\cite{polyak1963gradient,karimi2016linear}, which is a special case of \\L{}ojasiewicz gradient inequality. It involves the $L^2(\\nu)$ norm of the gradient, which we denote for $\\nu=\\h\\mu$ by\n","sha256_utf8":"49a66703bca42d911b882cb30c8089742a6a04d4614d71b0f26b55e63467c941"},{"start":52701,"end":53037,"start_line":512,"end_line":514,"raw_tex":"\\[\n \\Vert g_{\\nu}\\Vert^2_{L^2(\\nu)} \\coloneqq \\int_{\\Omega} \\left( \\frac1\\alpha \\vert 2\\alpha rJ'_{\\nu}(\\theta)\\vert^2 + \\frac {r^2}{\\beta} \\Vert \\beta \\nabla J'_{\\nu}\\Vert_\\theta^2\\right)\\d\\mu(r,\\theta) = \\int_\\Theta \\left( 4 \\alpha \\vert J'_\\nu(\\theta)\\vert^2 +\\beta \\Vert \\nabla J'_\\nu(\\theta)\\Vert^2_\\theta \\right)\\d\\nu(\\theta).\n \\]","sha256_utf8":"fffcd1bca7c87a0fabab01ccf7ae4ec36726a22c849e1f1aa491539b7bc6a4c9"}]},{"id":"S023","semantic_type":"definition","kind":"definition","title":"局部分区与容许半径","source_units":[{"start":55983,"end":56773,"start_line":551,"end_line":555,"raw_tex":"The proof of Theorem~\\ref{th:gradientinequality}, in Appendix~\\ref{app:sharpness}, is based on a local expansion of $J(\\nu)$ in terms of some local moments of $\\nu$. For a radius $\\tau>0$ (that shall be fixed at some small enough value in the course of the proof), we define the sets  for $i\\in \\{1,\\dots,m^\\star\\}$,\n\\begin{align*}\n\\Theta_i \\coloneqq  \\{ \\theta \\in \\Theta \\;;\\; \\dist(\\theta,\\theta_i) < \\tau \\}.\n\\end{align*}\nWe assume that $\\tau$ is smaller than $1$ and small enough so that these sets together with $\\Theta_0 \\coloneqq \\Theta \\setminus \\cup_{i=1}^{m^\\star} \\Theta_i$ form a partition of $\\Theta$ and that the exponential map at $\\theta_i$ has injectivity radius larger than $\\tau$, for $i\\in \\{1,\\dots,m^\\star\\}$. We then say that $\\tau$ is an \\emph{admissible} radius. \n","sha256_utf8":"2caee15eb74da662fd69f6c05c7e244ec09e9f864f7f458ffe49fe83928e56e4"}]},{"id":"S024","semantic_type":"definition","kind":"definition","title":"局部质量偏差与协方差","source_units":[{"start":56773,"end":57613,"start_line":556,"end_line":564,"raw_tex":"\\begin{definition}[Local moments]\\label{def:localfeatures}\nGiven an admissible radius $\\tau>0$ and a measure $\\nu\\in \\Mm_+(\\Theta)$, we define for $i\\in \\{0,\\dots,m^\\star\\}$ the local masses $\\bar r_i^2 = \\nu(\\Theta_i)$ and the local means $\\bar \\theta_i \\coloneqq \\frac1{\\bar r_i^2} \\int_{\\Theta_i} \\theta\\d\\nu(\\theta)$ if $\\nu(\\Theta_i)>0$ and $\\bar \\theta_i = \\theta_i$ otherwise. Finally, we define for $i\\in \\{1,\\dots,m^\\star\\}$ the \\emph{weighted biases}\n\\begin{align*}\nb_i^r \\coloneqq \\frac{\\bar r_i^2 -r_i^2}{2\\alpha r_i},&&\nb_i^\\theta \\coloneqq \\frac{\\bar r_i^2}{\\beta r_i}(\\bar \\theta_i-\\theta_i), &&\nb_i = (b_i^r,b_i^\\theta),\n\\end{align*}\nand the \\emph{weighted covariances} $\\Sigma_i \\coloneqq \\frac1{\\bar r_i^2 \\beta^2} \\int_{\\Theta_i} (\\theta - \\bar \\theta_i)(\\theta - \\bar \\theta_i)^\\intercal \\d\\nu(\\theta)$.\n\\end{definition}","sha256_utf8":"f3e3779a1008dd5389606556a31b5fadc113ef207954869e1c2681d4a986124e"}]},{"id":"S025","semantic_type":"definition","kind":"definition","title":"参考密度的离散熵","source_units":[{"start":65412,"end":65857,"start_line":619,"end_line":622,"raw_tex":"In order to state the condition on the initialization, we first choose a reference measure $\\rho \\in \\Mm_+(\\Theta)$ with a smooth positive density, also denoted by $\\rho$, which represents our prior knowledge about the solution $\\nu^\\star$. We introduce the quantity (analogous to a log-likelihood)\n\\[\n\\bar\\Hh(\\nu^\\star,\\rho) \\coloneqq \\sum_{i=1}^{m^\\star} r_i^2 \\log\\left(\\frac{r_i^2}{\\rho(\\theta_i)}\\right) - \\nu^\\star(\\Theta)+\\rho(\\Theta).\n\\]","sha256_utf8":"2f0739846b0119bb9e5563844f753f75276d3af8748e96a8e852e06e69843d02"}]},{"id":"S026","semantic_type":"definition","kind":"definition","title":"镜像速率函数","source_units":[{"start":71632,"end":72203,"start_line":658,"end_line":662,"raw_tex":" The proof of Theorem~\\ref{thm:convergenceflow} mostly relies on the the following general lemma which applies to any type of initialization or any structure of minimizers. It gives an upper bound on the optimality gap during along gradient flows in terms of a \\emph{mirror rate} function $\\Qq_{\\nu_0,\\nu^\\star}: \\RR^*_+ \\to \\RR_+$ defined for $\\nu^\\star, \\nu_0\\in \\Mm_+(\\Theta)$  and $\\tau>0$ as\n\\begin{align}\\label{eq:quantization}\n\\Qq_{\\nu^\\star,\\nu_0}(\\tau) = \n\\inf_{\\nu \\in \\Mm_+(\\Theta)} \\Vert \\nu^\\star- \\nu\\Vert_{\\BL}^* +\\frac{1}{\\tau} \\Hh(\\nu,\\nu_0).\n\\end{align}","sha256_utf8":"cbd9dffdc055746045d754e22c8ba9cd03ca2ee20087dd794360df535ed45cb9"}]},{"id":"S027","semantic_type":"notation","kind":"notation","title":"局部证明的归一化约定","source_units":[{"start":99686,"end":100119,"start_line":929,"end_line":929,"raw_tex":" In this whole section, we consider without loss of generality $\\alpha=\\beta=1$ (we explain in Section~\\ref{app:conclusionlocal} how to adapt the results to arbitrary $\\alpha,\\beta$). For simplicity, we only track the dependencies in $\\nu$ and $\\tau$. Any quantity that is independent of $\\nu$ and $\\tau$ is treated as a constant and represented by $C,C',C''>0$, and the quantity these symbols refer to can change from line to line.\n","sha256_utf8":"0fe6f4a58e5d625282ebbfe8f5d836f4f50cfcbb9d819b03b75a3a4b5569f042"}]},{"id":"S028","semantic_type":"definition","kind":"definition","title":"局部误差与方差尺度","source_units":[{"start":100208,"end":100548,"start_line":932,"end_line":935,"raw_tex":"Given a measure $\\nu\\in \\Mm_+(\\Theta)$, we consider the local centered moments introduced in Definition~\\ref{def:localfeatures} and in addition, for $i\\in \\{1,\\dots,m^\\star\\}$,\n\\begin{align*}\n\\delta\\theta_i = \\bar\\theta_i-\\theta_i && \\tilde b^\\theta_i = \\bar r_i\\delta\\theta_i, && s_i \\coloneqq \\bar r_i(\\tr\\Sigma_i)^{\\frac12}.\n\\end{align*}","sha256_utf8":"2746e13704021b968444867a91edd30ee44ffe9208c6009ed18f94ba41845b8a"},{"start":100549,"end":100826,"start_line":936,"end_line":939,"raw_tex":"Finally, we will quantify errors with the following quantity\n\\begin{align}\\label{eq:error}\nW_\\tau(\\nu)^2 \\coloneqq \\bar r_0^2 + \\Vert  b\\Vert^2 + \\Vert s\\Vert^2 = \\bar r_0^2 + \\sum_{i=1}^{m^\\star} \\left(\\vert  b_i^r\\vert^2 + \\Vert b_i^\\theta\\Vert^2  + s_i^2 \\right)\n\\end{align}","sha256_utf8":"95d13add6820243a360524a3df5d649db1ea839b7f3fb21aca48f9f86fef8647"}]},{"id":"S029","semantic_type":"definition","kind":"definition","title":"严格松弛的正间隔","source_units":[{"start":105753,"end":106237,"start_line":1014,"end_line":1014,"raw_tex":" In the next lemma, we globally bound the quantity $W_\\tau(\\nu)$ from Eq.~\\eqref{eq:error} in terms of the function values. It involves the quantity $v^\\star>0$ which is such that for any local minimum $\\theta$ of $J'_{\\nu^\\star}$, either $\\theta = \\theta_i$ for some $i\\in \\{1,\\dots,m^\\star\\}$ or $J'_{\\nu^\\star}(\\theta)\\geq v^*$ (which is non-zero under {\\sf (A5)}). We also recall that $\\tilde b^\\theta_i = \\bar r_i \\delta\\theta_i$, as defined in Section~\\ref{app:bounddistanceA}.\n","sha256_utf8":"a05ebd97be8d2a472d9a1adefc32936dbe263d7a1c53f420c54857821a3d0dde"}]},{"id":"S030","semantic_type":"proposition","kind":"proposition","title":"投影后的运输增长方程","source_units":[{"start":31029,"end":31400,"start_line":340,"end_line":344,"raw_tex":"\\begin{proposition} \\label{prop:projectedGF} Under {\\sf (A1-2)}, let $(\\mu_t)_{t\\geq 0}$ be a Wasserstein gradient flow for $F$, with $\\mu_0\\in \\Pp_2(\\Omega)$. Then $\\nu_t = \\h\\mu_t$ satisfies (in the weak sense)\n\\begin{equation}\\label{eq:projectedGF}\n\\partial_t \\nu_t = - 4 \\alpha \\nu_t J'_{\\nu_t} + \\beta \\div (\\nu_t \\nabla J'_{\\nu_t}).\n\\end{equation}\n\\end{proposition}","sha256_utf8":"04f7c6c5fd584fd2ee06c998f1c7de798cb91ea0715bdb77362cfa4d913633cd"}]},{"id":"S031","semantic_type":"proof","kind":"proof","title":"投影方程完整证明","source_units":[{"start":31401,"end":31867,"start_line":345,"end_line":352,"raw_tex":"\\begin{proof}\nFor any differentiable function $\\varphi: \\Theta\\to \\RR$, since $\\mu_t$ is a Wasserstein gradient flow it holds\n\\begin{equation*}\n\\frac{\\d}{\\d t}\\left(\\int \\varphi \\,\\d\\nu_t \\right) \n=- \\int \\langle \\nabla (\\h^*\\varphi), g_{\\h\\mu_t}\\rangle_{(r,\\theta)} \\d\\mu_t = - \\int (4 \\alpha \\varphi J'_{\\nu_t} + \\beta \\nabla \\varphi \\cdot \\nabla J'_{\\nu_t})\\d\\nu_t,\n\\end{equation*}\nwhich is the definition of weak solutions for~\\eqref{eq:projectedGF}.\n\\end{proof}","sha256_utf8":"ce64cbd769f3ddcb57fb5402963bdc676e431d20092fedefb6fdc4a85293af91"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"以测试函数计算投影演化","source_units":[{"start":31415,"end":31856,"start_line":346,"end_line":351,"raw_tex":"For any differentiable function $\\varphi: \\Theta\\to \\RR$, since $\\mu_t$ is a Wasserstein gradient flow it holds\n\\begin{equation*}\n\\frac{\\d}{\\d t}\\left(\\int \\varphi \\,\\d\\nu_t \\right) \n=- \\int \\langle \\nabla (\\h^*\\varphi), g_{\\h\\mu_t}\\rangle_{(r,\\theta)} \\d\\mu_t = - \\int (4 \\alpha \\varphi J'_{\\nu_t} + \\beta \\nabla \\varphi \\cdot \\nabla J'_{\\nu_t})\\d\\nu_t,\n\\end{equation*}\nwhich is the definition of weak solutions for~\\eqref{eq:projectedGF}.\n","sha256_utf8":"7425fee66c9148f52c48032876a46880a2a05efaf28698e26153b13e4af6c5cd"}]},{"id":"S033","semantic_type":"proposition","kind":"proposition","title":"极小解存在与最优性条件","source_units":[{"start":46224,"end":46621,"start_line":462,"end_line":463,"raw_tex":"\\begin{proposition}[Optimality condition]\\label{prop:optimalitycondition} Under~{\\sf(A1)} and~{\\sf(A3)}, problem~\\eqref{eq:objective} admits minimizers. Moreover, a measure $\\nu^\\star \\in \\Mm_+(\\Theta)$ is a minimizer if and only if it holds $J'_{\\nu^\\star}(\\theta)\\geq 0$ for all $\\theta\\in \\Theta$ and $J'_{\\nu^\\star}(\\theta)=0$ whenever $\\theta$ in the support of $\\nu^\\star$.\n\\end{proposition}","sha256_utf8":"937b3593aad300a14f6638e3391f3911952df1ea373b9d33b58dd4fe78119d3b"}]},{"id":"S034","semantic_type":"proof","kind":"proof","title":"最优性条件完整证明","source_units":[{"start":46622,"end":47405,"start_line":464,"end_line":466,"raw_tex":"\\begin{proof}\nAs $\\lambda$ is assumed positive, the sublevel sets of $J$ on $\\Mm_+(\\Theta)$ are bounded in total variation, and are thus weakly pre-compact. It follows that any minimizing sequence for $J$ admits at least one weak limit point $\\nu^\\star$, which is a minimizer of~\\eqref{eq:objective} since $J$ is weakly continuous. The stated optimality condition is equivalent to having $\\int J'_{\\nu^\\star}\\d (\\nu -\\nu^\\star) \\geq 0$ for all $\\nu \\in \\Mm_+(\\Theta)$. The latter is a sufficient optimality condition since by convexity of $J$, $J(\\nu)-J(\\nu^\\star) \\geq \\int J'_{\\nu^\\star}\\d (\\nu -\\nu^\\star)$. It is also necessary since  it holds $\\frac{d}{d\\epsilon} J((1-\\epsilon)\\nu^\\star +\\epsilon \\nu)\\vert_{\\epsilon = 0^+} = \\int J'_{\\nu^\\star} \\d(\\nu-\\nu^\\star)$.\n\\end{proof}","sha256_utf8":"f0d9563ee213b2eeefa5527b3cbcfafbe8b05f513781878635d714184cc596bb"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"弱紧性与凸变分条件","source_units":[{"start":46636,"end":47394,"start_line":465,"end_line":465,"raw_tex":"As $\\lambda$ is assumed positive, the sublevel sets of $J$ on $\\Mm_+(\\Theta)$ are bounded in total variation, and are thus weakly pre-compact. It follows that any minimizing sequence for $J$ admits at least one weak limit point $\\nu^\\star$, which is a minimizer of~\\eqref{eq:objective} since $J$ is weakly continuous. The stated optimality condition is equivalent to having $\\int J'_{\\nu^\\star}\\d (\\nu -\\nu^\\star) \\geq 0$ for all $\\nu \\in \\Mm_+(\\Theta)$. The latter is a sufficient optimality condition since by convexity of $J$, $J(\\nu)-J(\\nu^\\star) \\geq \\int J'_{\\nu^\\star}\\d (\\nu -\\nu^\\star)$. It is also necessary since  it holds $\\frac{d}{d\\epsilon} J((1-\\epsilon)\\nu^\\star +\\epsilon \\nu)\\vert_{\\epsilon = 0^+} = \\int J'_{\\nu^\\star} \\d(\\nu-\\nu^\\star)$.\n","sha256_utf8":"33710c97b05ee985f5b5006da9140bb7f9aed4e2554c97146835a3733569750c"}]},{"id":"S036","semantic_type":"lemma","kind":"lemma","title":"局部误差控制运输距离","source_units":[{"start":100994,"end":101135,"start_line":941,"end_line":943,"raw_tex":"\\begin{lemma}\\label{lem:distanceinequality}\nIt holds $\\widehat W_2(\\nu,\\nu^\\star)\\leq W_\\tau(\\nu)(1+O(\\tau^2)+O(W_\\tau(\\nu)^2))$.\n\\end{lemma}","sha256_utf8":"1677f409bae4bb7011d5f93d2b6e485bc74616e6296c3fbfc67cc7ac0f24a130"}]},{"id":"S037","semantic_type":"proof","kind":"proof","title":"运输距离控制完整证明","source_units":[{"start":101136,"end":103552,"start_line":944,"end_line":980,"raw_tex":"\\begin{proof}\nNote that for $W_\\tau(\\nu)$ small enough, it holds $\\nu(\\Theta_i)>0$ for $i\\in \\{1,\\dots,m^\\star\\}$. Let $\\mu \\in \\Pp_2(\\Omega)$ be such that $\\h\\mu = \\nu$ and consider the transport map $T:\\Omega\\to\\Omega$ defined as\n \\[\n T(r,\\theta) =\n \\begin{cases}\n(r\\frac{r_i}{\\bar r_i}, \\theta_i) &\\text{if $\\theta \\in \\Theta_i$ and $\\bar r_i>0$, $i\\in \\{1,\\dots,m^\\star\\}$},\\\\\n(0,\\theta) &\\text{otherwise}.\n \\end{cases}\n \\]\nBy construction, it holds $\\h(T_\\#\\mu) =  \\nu^\\star$. Let us estimate the transport cost associated to this map \n\\[\n\\mathcal{T} = \\int_\\Omega \\dist((r,\\theta),T(r,\\theta))^2\\d\\mu(r,\\theta).\n\\]\nThe geodesic distance associated to the cone metric is\n\\begin{align*}\n\\dist((r_1,\\theta_1),(r_2,\\theta_2))^2 &= r_1^2 + r_2^2 - 2r_1r_2\\cos(\\min\\{\\dist(\\theta_1,\\theta_2),\\pi\\})\\\\\n&= (r_1-r_2)^2 +r_1r_2\\dist(\\theta_1,\\theta_2)^2 +O(r_1r_2\\dist(\\theta_1,\\theta_2)^4)\n\\end{align*}\nNow, if we only consider points $\\theta\\in \\Theta_i$ with $\\tilde \\theta$ their coordinates in a normal frame centered at $\\theta_i$ (note that in all other proofs, we do not need to distinguish between $\\theta$ and $\\tilde \\theta$), we have the approximation\n\\begin{align*}\n\\dist((r_1,\\theta_1),(r_2,\\theta_2))^2 = (r_1-r_2)^2 +r_1r_2\\Vert \\tilde \\theta_1 - \\tilde \\theta_2\\Vert^2(1+O(\\tau^2)).\n\\end{align*}\nLet us decompose $T(r,\\theta)$ as $(rT^r(\\theta),T^\\theta(\\theta))$ and estimate the two contributions forming $\\Tt$ separately. On the one hand, we have\n\\begin{align*}\n\\int (rT^r(\\theta)-r)^2\\d\\mu(r,\\theta) &= \\bar r_0^2 + \\sum_{i=1}^{m^\\star} (\\bar r_i-r_i)^2 = \\bar r_0^2 +\\Vert b^r\\Vert^2(1+O(W_\\tau(\\nu)^2)).\n\\end{align*}\nOn the other hand, we have\n\\begin{align*}\n\\int r^2T^r(\\theta)\\Vert T^\\theta(\\theta)-\\theta\\Vert^2\\d\\mu(r,\\theta) &=\\sum_{i=1}^{m^\\star} \\bar r_ir_i (\\tr\\Sigma_i + \\Vert \\delta \\theta_i\\Vert^2)=(\\Vert s\\Vert^2 + \\Vert b^\\theta\\Vert^2)(1+O(W_\\tau(\\nu)^2)).\n\\end{align*}\nAs a consequence, we have $\\Tt = W_\\tau(\\nu)(1 + O(W_\\tau(\\nu)^2)+ O(\\tau^2))$. Remark that this estimate does not depend on the chosen lifting $\\mu$ satisfying $\\h\\mu=\\nu$. We then conclude by using the characterization in~\\cite[\nThm. 7.20]{liero2018optimal} for the distance $\\widehat W_2$:\n\\[\n\\widehat W_2(\\nu_1,\\nu_2) = \\min \\left\\{W_2(\\mu_1,\\mu_2) \\;;\\; (\\h\\mu_1,\\h\\mu_2) = (\\nu_1,\\nu_2)\\right\\}.\n\\]\nThus $\\widehat W_2(\\nu,\\nu^\\star)^2 \\leq W_2(\\mu,T_\\#(\\mu))^2\\leq \\Tt$, and the result follows.\n\\end{proof}","sha256_utf8":"eadc26c4556ecb6b26a959ea141c16cf77ee065fead37704207cce8d1020d4e1"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"构造到稀疏解的运输映射","source_units":[{"start":101150,"end":101757,"start_line":945,"end_line":956,"raw_tex":"Note that for $W_\\tau(\\nu)$ small enough, it holds $\\nu(\\Theta_i)>0$ for $i\\in \\{1,\\dots,m^\\star\\}$. Let $\\mu \\in \\Pp_2(\\Omega)$ be such that $\\h\\mu = \\nu$ and consider the transport map $T:\\Omega\\to\\Omega$ defined as\n \\[\n T(r,\\theta) =\n \\begin{cases}\n(r\\frac{r_i}{\\bar r_i}, \\theta_i) &\\text{if $\\theta \\in \\Theta_i$ and $\\bar r_i>0$, $i\\in \\{1,\\dots,m^\\star\\}$},\\\\\n(0,\\theta) &\\text{otherwise}.\n \\end{cases}\n \\]\nBy construction, it holds $\\h(T_\\#\\mu) =  \\nu^\\star$. Let us estimate the transport cost associated to this map \n\\[\n\\mathcal{T} = \\int_\\Omega \\dist((r,\\theta),T(r,\\theta))^2\\d\\mu(r,\\theta).\n\\]\n","sha256_utf8":"edbe426d580c580c2dc247771ba71ac11c12bab31d23c5a1ee5bd77dfe232fb1"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"展开锥距离的局部代价","source_units":[{"start":101757,"end":102445,"start_line":957,"end_line":965,"raw_tex":"The geodesic distance associated to the cone metric is\n\\begin{align*}\n\\dist((r_1,\\theta_1),(r_2,\\theta_2))^2 &= r_1^2 + r_2^2 - 2r_1r_2\\cos(\\min\\{\\dist(\\theta_1,\\theta_2),\\pi\\})\\\\\n&= (r_1-r_2)^2 +r_1r_2\\dist(\\theta_1,\\theta_2)^2 +O(r_1r_2\\dist(\\theta_1,\\theta_2)^4)\n\\end{align*}\nNow, if we only consider points $\\theta\\in \\Theta_i$ with $\\tilde \\theta$ their coordinates in a normal frame centered at $\\theta_i$ (note that in all other proofs, we do not need to distinguish between $\\theta$ and $\\tilde \\theta$), we have the approximation\n\\begin{align*}\n\\dist((r_1,\\theta_1),(r_2,\\theta_2))^2 = (r_1-r_2)^2 +r_1r_2\\Vert \\tilde \\theta_1 - \\tilde \\theta_2\\Vert^2(1+O(\\tau^2)).\n\\end{align*}\n","sha256_utf8":"3e8743e186de1c55221ae55f7613436017acd4aa439908386a322d9a4b29e815"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"分别估计质量与位置代价","source_units":[{"start":102445,"end":103040,"start_line":966,"end_line":973,"raw_tex":"Let us decompose $T(r,\\theta)$ as $(rT^r(\\theta),T^\\theta(\\theta))$ and estimate the two contributions forming $\\Tt$ separately. On the one hand, we have\n\\begin{align*}\n\\int (rT^r(\\theta)-r)^2\\d\\mu(r,\\theta) &= \\bar r_0^2 + \\sum_{i=1}^{m^\\star} (\\bar r_i-r_i)^2 = \\bar r_0^2 +\\Vert b^r\\Vert^2(1+O(W_\\tau(\\nu)^2)).\n\\end{align*}\nOn the other hand, we have\n\\begin{align*}\n\\int r^2T^r(\\theta)\\Vert T^\\theta(\\theta)-\\theta\\Vert^2\\d\\mu(r,\\theta) &=\\sum_{i=1}^{m^\\star} \\bar r_ir_i (\\tr\\Sigma_i + \\Vert \\delta \\theta_i\\Vert^2)=(\\Vert s\\Vert^2 + \\Vert b^\\theta\\Vert^2)(1+O(W_\\tau(\\nu)^2)).\n\\end{align*}\n","sha256_utf8":"3ce93f17ad9977d21cba855fa1aa34d2ce33c3e1a0d503f5bfe6e94e19b015eb"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"利用提升表征完成距离估计","source_units":[{"start":103040,"end":103541,"start_line":974,"end_line":979,"raw_tex":"As a consequence, we have $\\Tt = W_\\tau(\\nu)(1 + O(W_\\tau(\\nu)^2)+ O(\\tau^2))$. Remark that this estimate does not depend on the chosen lifting $\\mu$ satisfying $\\h\\mu=\\nu$. We then conclude by using the characterization in~\\cite[\nThm. 7.20]{liero2018optimal} for the distance $\\widehat W_2$:\n\\[\n\\widehat W_2(\\nu_1,\\nu_2) = \\min \\left\\{W_2(\\mu_1,\\mu_2) \\;;\\; (\\h\\mu_1,\\h\\mu_2) = (\\nu_1,\\nu_2)\\right\\}.\n\\]\nThus $\\widehat W_2(\\nu,\\nu^\\star)^2 \\leq W_2(\\mu,T_\\#(\\mu))^2\\leq \\Tt$, and the result follows.\n","sha256_utf8":"5372a58fbe513452ac6f57e95857215ff3481687d547866cbd60016b446871c1"}]},{"id":"S042","semantic_type":"lemma","kind":"lemma","title":"围绕稀疏解的积分展开","source_units":[{"start":103593,"end":104642,"start_line":984,"end_line":997,"raw_tex":"\\begin{lemma}[Expansion around $\\nu^\\star$]\\label{lem:expansion}\n Let $\\psi$ be any (vector or real-valued) smooth function on $\\Theta$ and $\\nu\\in \\Mm_+(\\Theta)$. If $\\tau>0$ is an admissible radius, then the following first and second-order expansions hold\n \\begin{align*}\n \\int \\psi \\d(\\nu - \\nu^\\star) \n &= \\sum_{i=1}^m r_i \\bar \\nabla \\psi(\\theta_i)^\\intercal b_i \n + \\int_{\\Theta_0} \\psi \\d\\nu \n +\\sum_{i=1}^m \\int_{\\Theta_i} M_{2,\\psi}(\\theta_i,\\theta)\\d\\nu(\\theta)\\\\\n &=\\sum_{i=1}^m r_i\\bar \\nabla \\psi(\\theta_i)^\\intercal b_i \n +\\frac12  \\sum_{i=1}^m \\bar r_i^2  \\left( \\tr\\left(\\nabla^2\\psi(\\theta_i) \\Sigma_i \\right) \n + \\delta \\theta_i^\\intercal \\nabla^2\\psi(\\theta_i) \\delta \\theta_i \\right)\\\\\n &\\qquad + \\int_{\\Theta_0}  \\psi \\d\\nu +\\sum_{i=1}^m \\int_{\\Theta_i} M_{3,\\psi}(\\theta_i,\\theta)\\d\\nu(\\theta)\n \\end{align*}\n where $M_{k,\\psi}(\\theta_i,\\theta)$ is the remainder in the $k-1$-th order Taylor expansion of $\\psi$ around $\\theta_i$ in local coordinates (and we recall that $\\bar \\nabla \\psi := (2\\psi,\\nabla \\psi)$).\n \\end{lemma}","sha256_utf8":"ec32d5d4111f2ed8b148137ebd4e2de8e1a4da1b3a1a74d0785b7031204a05fd"}]},{"id":"S043","semantic_type":"proof","kind":"proof","title":"积分展开完整证明","source_units":[{"start":104644,"end":105696,"start_line":998,"end_line":1010,"raw_tex":"\\begin{proof}\nBy a Taylor expansion of $\\psi$ around $\\theta_i$ for $i\\in \\{1,\\dots,m^\\star\\}$, it holds\n\\begin{align*}\n \\int_{\\Theta_i} \\psi \\d\\nu= \\int_{\\Theta_i} \\left( \\psi(\\theta_i) + \\nabla\\psi(\\theta_i)^\\intercal (\\theta-\\theta_i) + (\\theta-\\theta_i) ^\\intercal \\nabla^2\\psi(\\theta_i)(\\theta-\\theta_i) + M_{3,\\psi}(\\theta_i,\\theta)\\right)\\d\\nu(\\theta)\n\\end{align*}\nand substracting $\\int_{\\Theta_i}\\psi\\d\\nu^\\star = r_i^2\\phi(\\theta_i)$ yields\n\\begin{multline*}\n \\int_{\\Theta_i} \\psi \\d(\\nu-\\nu^\\star)= (\\bar r_i^2 -r_i^2)\\psi(\\theta_i) +  \\bar r_i^2 \\nabla \\psi(\\theta_i)^\\intercal \\delta \\theta_i \\\\ \n +\\frac12  \\sum_{i=1}^m \\bar r_i^2  \\left( \\tr\\left(\\nabla^2\\psi(\\theta_i) \\Sigma_i \\right) \n + \\delta \\theta_i^\\intercal \\nabla^2\\psi(\\theta_i) \\delta \\theta_i \\right)  +\\sum_{i=1}^m \\int_{\\Theta_i} M_{3,\\psi}(\\theta_i,\\theta)\\d\\nu(\\theta)\n \\end{multline*}\nwhere  we have used a bias-variance decomposition for the quadratic term. The result follows by summing the integrals over each $\\Theta_i$ and using the expression of $b$.\n \\end{proof}","sha256_utf8":"109ffd279caf43ea292d31cb3b41d0682b7dc88ae31a8f336471692f1c5d9cb2"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"在各原子附近作泰勒展开","source_units":[{"start":104658,"end":105016,"start_line":999,"end_line":1002,"raw_tex":"By a Taylor expansion of $\\psi$ around $\\theta_i$ for $i\\in \\{1,\\dots,m^\\star\\}$, it holds\n\\begin{align*}\n \\int_{\\Theta_i} \\psi \\d\\nu= \\int_{\\Theta_i} \\left( \\psi(\\theta_i) + \\nabla\\psi(\\theta_i)^\\intercal (\\theta-\\theta_i) + (\\theta-\\theta_i) ^\\intercal \\nabla^2\\psi(\\theta_i)(\\theta-\\theta_i) + M_{3,\\psi}(\\theta_i,\\theta)\\right)\\d\\nu(\\theta)\n\\end{align*}\n","sha256_utf8":"1b0d67623bf4c36e8d2b19f6def2db286a28f7fd6ed67b3dd34fac2ee98b1b54"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"偏差方差分解后求和","source_units":[{"start":105016,"end":105684,"start_line":1003,"end_line":1009,"raw_tex":"and substracting $\\int_{\\Theta_i}\\psi\\d\\nu^\\star = r_i^2\\phi(\\theta_i)$ yields\n\\begin{multline*}\n \\int_{\\Theta_i} \\psi \\d(\\nu-\\nu^\\star)= (\\bar r_i^2 -r_i^2)\\psi(\\theta_i) +  \\bar r_i^2 \\nabla \\psi(\\theta_i)^\\intercal \\delta \\theta_i \\\\ \n +\\frac12  \\sum_{i=1}^m \\bar r_i^2  \\left( \\tr\\left(\\nabla^2\\psi(\\theta_i) \\Sigma_i \\right) \n + \\delta \\theta_i^\\intercal \\nabla^2\\psi(\\theta_i) \\delta \\theta_i \\right)  +\\sum_{i=1}^m \\int_{\\Theta_i} M_{3,\\psi}(\\theta_i,\\theta)\\d\\nu(\\theta)\n \\end{multline*}\nwhere  we have used a bias-variance decomposition for the quadratic term. The result follows by summing the integrals over each $\\Theta_i$ and using the expression of $b$.\n","sha256_utf8":"860d6f8a6ce7b05f02958793381eda6f522f9389483a438d74a1e2f16d8c8b0e"}]},{"id":"S046","semantic_type":"lemma","kind":"lemma","title":"目标差控制全部局部误差","source_units":[{"start":106239,"end":106908,"start_line":1016,"end_line":1025,"raw_tex":"\\begin{lemma}[Global distance bound]\\label{lem:distancebound}\nUnder {\\sf (A1-5)}, let $\\tau_{\\adm}$ be an admissible radius $\\tau$ as in Definition~\\ref{def:localfeatures}, fix some $J_{\\max}>0$ and let\n\\[\n\\tau_0 = \\min\\left\\{\\tau_{\\adm}, 2\\sqrt{\\frac{v^\\star}{\\sigma_{\\min}(H)}}, \\frac{3\\sigma_{\\min}(H)}{2 \\Lip (\\nabla^2 J'_{\\nu^\\star})} \\right\\}.\n\\]\nThen there exists $C, C'>0$ such that for all $\\tau\\leq \\tau_0$ and  $\\nu \\in \\Mm_+(\\Theta)$ such that $J(\\nu)\\leq J_{\\max}$, it holds \n\\begin{align*}\nW_\\tau(\\nu) \\leq \\frac{C}{\\tau^2}(J(\\nu)-J^\\star)^\\frac12 &&\\text{and}&&\\Vert \\tilde b^\\theta\\Vert^2+\\Vert s\\Vert^2\\leq C'(J(\\nu)-J^\\star). \n\\end{align*}\n\\end{lemma}","sha256_utf8":"9aa7a7ebed62a47874932d3ac60c05b6df3c7b46b4b82a65327cacda40262776"}]},{"id":"S047","semantic_type":"proof","kind":"proof","title":"全局误差界完整证明","source_units":[{"start":106910,"end":111046,"start_line":1027,"end_line":1082,"raw_tex":"\\begin{proof}\nLet us write $f_\\nu \\coloneqq \\int \\phi\\d\\nu$ and $f^\\star = \\int \\phi \\d\\nu^\\star$. By strong convexity of $R$ at $f^\\star$, and optimality of $\\mu^\\star$, there exists $C>0$ such that for all $\\nu\\in \\Mm_+(\\Theta)$ it holds\n\\begin{equation}\\label{eq:strongconvexityinequality}\nJ(\\nu)-J^\\star \\geq \\int_{\\Theta} J'_{\\nu^\\star}\\d\\nu+ C\\min\\{ \\Vert f_\\nu-f^\\star\\Vert^2,\\Vert f_\\nu-f^\\star\\Vert\\} .\n\\end{equation}\nTo prove the first claim, we thus have to bound $W_\\tau(\\nu)$ using the terms in the right-hand side of~\\eqref{eq:strongconvexityinequality}. \n\n\\paragraph{Step 1.} By a Taylor expansion, one has for $\\theta \\in \\Theta_i$ for $i\\in \\{1,\\dots,m^\\star\\}$,\n\\[\n\\vert J'_{\\nu^\\star}(\\theta) - \\tfrac12 (\\theta -\\theta_i)^\\intercal H_i  (\\theta-\\theta_i) \\vert \\leq \\tfrac16 \\Lip( \\nabla^2 J'_{\\nu^\\star}) \\Vert \\theta-\\theta_i\\Vert^3.\n\\]\nThus, if $\\Vert \\theta - \\theta_i\\Vert \\leq 3\\sigma_{\\min}(H)/(2 \\Lip( \\nabla^2 J'_{\\nu^\\star}))$, then $J'_{\\nu^\\star}(\\theta) \\geq \\frac14 (\\theta -\\theta_i)^\\intercal H_i  (\\theta-\\theta_i)$ for $\\theta\\in \\Theta_i$. Decomposing the integral of this quadratic term into bias and variance, we get\n\\begin{align*}\\label{eq:biasvariance}\n\\int_{\\Theta_i} (\\theta -\\theta_i)^\\intercal H_i (\\theta-\\theta_i)\\d\\nu(\\theta) \n&= \\bar r_i^2 \\left(\\delta \\theta_i^\\intercal  H_i \\delta \\theta_i + \\tr(\\Sigma_i H_i )\\right)\n\\end{align*}\nand we deduce a first bound by summing the terms for $i\\in \\{1,\\dots,m^\\star\\}$,\n\\[\n\\int_{\\Theta\\setminus \\Theta_0} J'_{\\nu^\\star} \\d\\nu \\geq \\frac{\\sigma_{\\min}(H)}4(\\Vert \\tilde b^\\theta\\Vert^2 + \\Vert s\\Vert^2).\n\\]\n\n\\paragraph{Step 2.} In order to lower bound the integral over $\\Theta_0$, we first derive a lower bound for $J'_{\\nu^\\star}$ on $\\Theta_0$. This is a continuously differentiable and nonnegative function on a closed domain $\\Theta_0$ so its minimum is attained either at a local minima in the interior of $\\Theta_0$ or on its boundary. Using the quadratic lower bound from the previous paragraph, it follows that for $\\theta\\in \\Theta_0$,\n\\[\nJ'_{\\nu^\\star}(\\theta)\\geq \\min \\{ v^\\star,  \\tau^2 \\sigma_{\\min}(H)/4\\}.\n\\]\nThus, if we also assume that $\\tau\\leq 2\\sqrt{v^\\star/\\sigma_{\\min}(H)}$ then $J'_{\\nu^\\star}(\\theta)\\geq \\tau^2 \\sigma_{\\min}(H)/4$ for $\\theta\\in \\Theta_0$ and it follows that\n\\[\n\\int_{\\Theta_0} J'_{\\nu^\\star} \\d\\nu \\geq \\frac{\\sigma_{\\min}(H)}{4} \\tau^2\\bar r_0^2 .\n\\]\nUsing inequality~\\eqref{eq:strongconvexityinequality}  we have shown so far that\n\\begin{equation}\\label{eq:localcontrol}\n\\tilde W_\\tau(\\nu)^2\\coloneqq  \\bar r_0^2 +\\Vert \\tilde b^\\theta\\Vert^2 +\\Vert s\\Vert^2 \\leq \\frac{C}{\\tau^2} (J(\\mu)-J^\\star).\n\\end{equation}\nNotice that $\\tilde W_\\tau(\\nu)$ is similar to $W_\\tau(\\nu)$ but it does not contain the terms controlling the deviations of mass $\\vert \\bar r_i-r_i\\vert$. These quantities can be controlled by using the coercivity of $R$, i.e. the last term in~\\eqref{eq:strongconvexityinequality}, as we do now.\n\n\\paragraph{Step 3.}\nUsing the first order expansion of Lemma~\\ref{lem:expansion} then squaring gives\n\\begin{align*}\n\\left\\vert \\Vert f_\\nu-f^\\star\\Vert^2 - \\frac12 b^\\intercal K b \\right\\vert\n& \\leq C \\left( \\Vert b\\Vert \\tilde W_\\tau(\\nu)^2 +  \\tilde W_\\tau(\\nu)^4 \\right).\n\\end{align*}\n%\nSince we have assumed that $K$ is positive definite, it follows\n\\[\n\\Vert f_\\nu-f^\\star\\Vert^2 \\geq C \\Vert b\\Vert^2 - C \\tilde W_\\tau(\\nu)^2 \\Vert b\\Vert - C\\tilde W_\\tau(\\nu)^4\n\\]\nand thus, after rearranging the terms\n\\[\n(\\Vert b\\Vert - C\\tilde W_\\tau(\\nu)^2)^2 \\leq  C \\Vert f_\\nu-f^\\star\\Vert^2 + C\\tilde W_\\tau(\\nu)^4.\n\\]\nIt follows that $\\Vert b\\Vert \\leq  C\\Vert f_\\nu-f^\\star\\Vert + C\\tilde W_\\tau(\\nu)^2$. Also, by inequality~\\eqref{eq:strongconvexityinequality}, if $J(\\nu)\\leq J_{\\max}$, then $\\Vert f_\\nu -f^\\star\\Vert^2\\leq C(J(\\nu)-J^\\star)$.  Moreover, by inequality~\\eqref{eq:localcontrol}, we get\n\\[\n\\Vert b\\Vert \\leq \\frac{C}{\\tau^2}(J(\\nu)-J^\\star) +C (J(\\nu)-J^\\star)^\\frac12 \\leq \\frac{C}{\\tau^2} (J(\\nu)-J^\\star)^\\frac12.\n\\]\nWe finally combine with the bound on $\\tilde W_\\tau(\\nu)$ to conclude since $W_\\tau(\\nu)^2\\leq \\tilde W_\\tau(\\nu)^2+\\Vert b\\Vert^2$\n\\end{proof}","sha256_utf8":"ac902d7a6d2e3bddaa9a6a4851bbc781c96add3ba438bd035d53ab546fd71d06"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"由凸性取得证书与特征控制","source_units":[{"start":106924,"end":107480,"start_line":1028,"end_line":1032,"raw_tex":"Let us write $f_\\nu \\coloneqq \\int \\phi\\d\\nu$ and $f^\\star = \\int \\phi \\d\\nu^\\star$. By strong convexity of $R$ at $f^\\star$, and optimality of $\\mu^\\star$, there exists $C>0$ such that for all $\\nu\\in \\Mm_+(\\Theta)$ it holds\n\\begin{equation}\\label{eq:strongconvexityinequality}\nJ(\\nu)-J^\\star \\geq \\int_{\\Theta} J'_{\\nu^\\star}\\d\\nu+ C\\min\\{ \\Vert f_\\nu-f^\\star\\Vert^2,\\Vert f_\\nu-f^\\star\\Vert\\} .\n\\end{equation}\nTo prove the first claim, we thus have to bound $W_\\tau(\\nu)$ using the terms in the right-hand side of~\\eqref{eq:strongconvexityinequality}. \n","sha256_utf8":"25edb93f6ec1c609f982a386988428bd85136ed943291cf83e4b317ae43dbfcd"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"局部曲率控制位置与方差","source_units":[{"start":107481,"end":108513,"start_line":1034,"end_line":1046,"raw_tex":"\\paragraph{Step 1.} By a Taylor expansion, one has for $\\theta \\in \\Theta_i$ for $i\\in \\{1,\\dots,m^\\star\\}$,\n\\[\n\\vert J'_{\\nu^\\star}(\\theta) - \\tfrac12 (\\theta -\\theta_i)^\\intercal H_i  (\\theta-\\theta_i) \\vert \\leq \\tfrac16 \\Lip( \\nabla^2 J'_{\\nu^\\star}) \\Vert \\theta-\\theta_i\\Vert^3.\n\\]\nThus, if $\\Vert \\theta - \\theta_i\\Vert \\leq 3\\sigma_{\\min}(H)/(2 \\Lip( \\nabla^2 J'_{\\nu^\\star}))$, then $J'_{\\nu^\\star}(\\theta) \\geq \\frac14 (\\theta -\\theta_i)^\\intercal H_i  (\\theta-\\theta_i)$ for $\\theta\\in \\Theta_i$. Decomposing the integral of this quadratic term into bias and variance, we get\n\\begin{align*}\\label{eq:biasvariance}\n\\int_{\\Theta_i} (\\theta -\\theta_i)^\\intercal H_i (\\theta-\\theta_i)\\d\\nu(\\theta) \n&= \\bar r_i^2 \\left(\\delta \\theta_i^\\intercal  H_i \\delta \\theta_i + \\tr(\\Sigma_i H_i )\\right)\n\\end{align*}\nand we deduce a first bound by summing the terms for $i\\in \\{1,\\dots,m^\\star\\}$,\n\\[\n\\int_{\\Theta\\setminus \\Theta_0} J'_{\\nu^\\star} \\d\\nu \\geq \\frac{\\sigma_{\\min}(H)}4(\\Vert \\tilde b^\\theta\\Vert^2 + \\Vert s\\Vert^2).\n\\]\n","sha256_utf8":"c2efc99703cf6a2ff413e762913152a81ad41d264e191cdf7b6ce40d6df9bf02"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"严格松弛控制分区外质量","source_units":[{"start":108514,"end":109866,"start_line":1048,"end_line":1060,"raw_tex":"\\paragraph{Step 2.} In order to lower bound the integral over $\\Theta_0$, we first derive a lower bound for $J'_{\\nu^\\star}$ on $\\Theta_0$. This is a continuously differentiable and nonnegative function on a closed domain $\\Theta_0$ so its minimum is attained either at a local minima in the interior of $\\Theta_0$ or on its boundary. Using the quadratic lower bound from the previous paragraph, it follows that for $\\theta\\in \\Theta_0$,\n\\[\nJ'_{\\nu^\\star}(\\theta)\\geq \\min \\{ v^\\star,  \\tau^2 \\sigma_{\\min}(H)/4\\}.\n\\]\nThus, if we also assume that $\\tau\\leq 2\\sqrt{v^\\star/\\sigma_{\\min}(H)}$ then $J'_{\\nu^\\star}(\\theta)\\geq \\tau^2 \\sigma_{\\min}(H)/4$ for $\\theta\\in \\Theta_0$ and it follows that\n\\[\n\\int_{\\Theta_0} J'_{\\nu^\\star} \\d\\nu \\geq \\frac{\\sigma_{\\min}(H)}{4} \\tau^2\\bar r_0^2 .\n\\]\nUsing inequality~\\eqref{eq:strongconvexityinequality}  we have shown so far that\n\\begin{equation}\\label{eq:localcontrol}\n\\tilde W_\\tau(\\nu)^2\\coloneqq  \\bar r_0^2 +\\Vert \\tilde b^\\theta\\Vert^2 +\\Vert s\\Vert^2 \\leq \\frac{C}{\\tau^2} (J(\\mu)-J^\\star).\n\\end{equation}\nNotice that $\\tilde W_\\tau(\\nu)$ is similar to $W_\\tau(\\nu)$ but it does not contain the terms controlling the deviations of mass $\\vert \\bar r_i-r_i\\vert$. These quantities can be controlled by using the coercivity of $R$, i.e. the last term in~\\eqref{eq:strongconvexityinequality}, as we do now.\n","sha256_utf8":"377d486fdbc04221bf88ff65fd9a09ddc79d15d3fa9a6d6862d59bd0d756a5f2"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"正定全局核控制质量偏差","source_units":[{"start":109867,"end":111035,"start_line":1062,"end_line":1081,"raw_tex":"\\paragraph{Step 3.}\nUsing the first order expansion of Lemma~\\ref{lem:expansion} then squaring gives\n\\begin{align*}\n\\left\\vert \\Vert f_\\nu-f^\\star\\Vert^2 - \\frac12 b^\\intercal K b \\right\\vert\n& \\leq C \\left( \\Vert b\\Vert \\tilde W_\\tau(\\nu)^2 +  \\tilde W_\\tau(\\nu)^4 \\right).\n\\end{align*}\n%\nSince we have assumed that $K$ is positive definite, it follows\n\\[\n\\Vert f_\\nu-f^\\star\\Vert^2 \\geq C \\Vert b\\Vert^2 - C \\tilde W_\\tau(\\nu)^2 \\Vert b\\Vert - C\\tilde W_\\tau(\\nu)^4\n\\]\nand thus, after rearranging the terms\n\\[\n(\\Vert b\\Vert - C\\tilde W_\\tau(\\nu)^2)^2 \\leq  C \\Vert f_\\nu-f^\\star\\Vert^2 + C\\tilde W_\\tau(\\nu)^4.\n\\]\nIt follows that $\\Vert b\\Vert \\leq  C\\Vert f_\\nu-f^\\star\\Vert + C\\tilde W_\\tau(\\nu)^2$. Also, by inequality~\\eqref{eq:strongconvexityinequality}, if $J(\\nu)\\leq J_{\\max}$, then $\\Vert f_\\nu -f^\\star\\Vert^2\\leq C(J(\\nu)-J^\\star)$.  Moreover, by inequality~\\eqref{eq:localcontrol}, we get\n\\[\n\\Vert b\\Vert \\leq \\frac{C}{\\tau^2}(J(\\nu)-J^\\star) +C (J(\\nu)-J^\\star)^\\frac12 \\leq \\frac{C}{\\tau^2} (J(\\nu)-J^\\star)^\\frac12.\n\\]\nWe finally combine with the bound on $\\tilde W_\\tau(\\nu)$ to conclude since $W_\\tau(\\nu)^2\\leq \\tilde W_\\tau(\\nu)^2+\\Vert b\\Vert^2$\n","sha256_utf8":"961e291acbe2a89c87ae65927f37df2cfadeade249b2a5c4f15739dc5f568661"}]},{"id":"S052","semantic_type":"proposition","kind":"proposition","title":"目标值收敛蕴含测度收敛","source_units":[{"start":51961,"end":52285,"start_line":506,"end_line":508,"raw_tex":"\\begin{proposition}\\label{prop:distanceinequality}\nUnder {\\sf (A1-5)}, for all $J_{\\max}\\geq J^\\star$, there exists $C,C'>0$, such that if $\\nu\\in \\Mm_+(\\Theta)$ satisfies $J(\\nu)\\leq J_{\\max}$ then $\\Vert \\nu - \\nu^\\star\\Vert^*_{\\BL} \\leq C \\widehat W_2(\\nu, \\nu^\\star) \\leq C'(J(\\nu)-J^\\star)^{\\frac12}$.\n\\end{proposition}","sha256_utf8":"acd01fb722c73c8d11c6310a0f0ff0640b232c0afae113984a1af542a84ea8fb"}]},{"id":"S053","semantic_type":"proof","kind":"proof","title":"测度距离不等式完整证明","source_units":[{"start":111049,"end":111835,"start_line":1084,"end_line":1100,"raw_tex":"\\subsection{Proof of the distance inequality (Proposition~\\ref{prop:distanceinequality})}\\label{app:distanceinequality}\nBy Lemma~\\ref{lem:distanceinequality}, it holds \n\\[\n\\widehat W_2(\\nu,\\nu^\\star)\\leq W_\\tau(\\nu)(1+O(\\tau^2)+O(W_\\tau(\\nu)^2)).\n\\]\nMoreover, by Lemma~\\ref{lem:distancebound}, there exists $\\tau_0>0$ and $C>0$ such that \n\\[\nW_\\tau(\\nu)\\leq \\frac{C}{\\tau_0^2}(J(\\nu)-J^\\star)^\\frac12.\n\\]\nCombining these two lemmas, it follows that for some $C'>0$, we have \n\\[\n\\widehat W_2(\\nu,\\nu^\\star)^2 \\leq C'(J(\\nu)-J^\\star)^\\frac12.\n\\]\n This also implies a control on the Bounded-Lipschitz distance since it holds $(\\Vert \\nu - \\nu^\\star\\Vert_{\\BL}^*)^2\\leq (2+\\pi^2/2)(\\nu(\\Theta)+\\nu^\\star(\\Theta))\\widehat W_2(\\nu,\\nu^\\star)^2$, see~\\cite[Prop. 7.18]{liero2018optimal}.\n \n \n ","sha256_utf8":"1a6bf4b684bb3b705921ec68836744d58087207d6df99a895e52b5960a9c48be"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"组合运输界与全局误差界","source_units":[{"start":111138,"end":111454,"start_line":1084,"end_line":1092,"raw_tex":"\\label{app:distanceinequality}\nBy Lemma~\\ref{lem:distanceinequality}, it holds \n\\[\n\\widehat W_2(\\nu,\\nu^\\star)\\leq W_\\tau(\\nu)(1+O(\\tau^2)+O(W_\\tau(\\nu)^2)).\n\\]\nMoreover, by Lemma~\\ref{lem:distancebound}, there exists $\\tau_0>0$ and $C>0$ such that \n\\[\nW_\\tau(\\nu)\\leq \\frac{C}{\\tau_0^2}(J(\\nu)-J^\\star)^\\frac12.\n\\]\n","sha256_utf8":"46b5a54f6348c6ad378655dacc95e6981d78d6d3b0f693766629394ab05a9575"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"得到运输及有界利普希茨控制","source_units":[{"start":111454,"end":111830,"start_line":1093,"end_line":1097,"raw_tex":"Combining these two lemmas, it follows that for some $C'>0$, we have \n\\[\n\\widehat W_2(\\nu,\\nu^\\star)^2 \\leq C'(J(\\nu)-J^\\star)^\\frac12.\n\\]\n This also implies a control on the Bounded-Lipschitz distance since it holds $(\\Vert \\nu - \\nu^\\star\\Vert_{\\BL}^*)^2\\leq (2+\\pi^2/2)(\\nu(\\Theta)+\\nu^\\star(\\Theta))\\widehat W_2(\\nu,\\nu^\\star)^2$, see~\\cite[Prop. 7.18]{liero2018optimal}.\n","sha256_utf8":"b193ad74de3c615e90c3c86460116f22359f18a4929df29646877196d51ee186"}]},{"id":"S056","semantic_type":"proposition","kind":"proposition","title":"目标函数的局部展开","source_units":[{"start":111930,"end":112606,"start_line":1102,"end_line":1111,"raw_tex":"\\begin{proposition}[Local expansion]\\label{prop:localexpansion}\n It holds\n \\[\n J(\\nu)-J^\\star = \\frac12 b^\\intercal K b + \\frac12 \\sum_{i=1}^m \\bar r_i^2 \\left( \\tr(\\Sigma_i H_i) +\\delta \\theta^\\intercal H_i \\delta \\theta_i \\right) + \\int_{\\Theta_0} J'_{\\nu^\\star} \\d\\nu + \\mathop{\\mathrm{err}}(\\tau,\\nu)\n \\]\n where $\\mathop{\\mathrm{err}}(\\tau,\\nu) =O( \\tau (\\Vert \\tilde b^\\theta \\Vert^2 + \\Vert s\\Vert^2) + W_\\tau(\\nu)^3)$. In particular, if $\\tau$ is fixed small enough,\n \\[\n J(\\nu)-J^\\star \\leq  \\sigma_{\\max}(K) \\Vert b\\Vert^2 + \\sigma_{\\max}(H)(\\Vert b^\\theta\\Vert^2 +\\Vert s\\Vert^2) +\\Vert J'_{\\nu^\\star}\\Vert_\\infty \\bar r_0^2 +O(W_\\tau(\\nu)^3).\n \\]\n \\end{proposition}","sha256_utf8":"dfc63f40aa4bd18d68feb37acf73d8cbf73500da36533fa64f66584981379c1f"}]},{"id":"S057","semantic_type":"proof","kind":"proof","title":"目标局部展开完整证明","source_units":[{"start":112608,"end":113914,"start_line":1112,"end_line":1125,"raw_tex":"\\begin{proof} Let us write $f_\\nu \\coloneqq \\int \\phi\\d\\nu$ and $f^\\star = \\int \\phi \\d\\nu^\\star$. By a second order Taylor expansion of $R$ around $f^\\star$, we have\n \\[\n J(\\nu) - J^\\star = \\int_{\\Theta} J'_{\\nu^\\star} \\d\\nu+ \\frac12 \\Vert f_\\nu-f^\\star\\Vert^2_\\star + O(\\Vert f_\\nu-f^\\star\\Vert^3).\n \\]\n Using the first order expansion of Lemma~\\ref{lem:expansion} for $\\phi$, we get \n $\n \\Vert f_\\nu-f^\\star\\Vert^2_\\star %= b^\\intercal Kb + O\\left( (r_0^2 + \\sum_{i=1}^m s_i^2)(\\Vert b\\Vert + r_0^2 + \\sum_{i=1}^m s_i^2)\\right) \n = b^\\intercal Kb + O(W_{\\tau}(\\nu)^3)\n $. Also, using the second order expansion of Lemma~\\ref{lem:expansion} for $J'_{\\nu^\\star}$ and using the fact that $J'_{\\nu^\\star}$ and its gradient vanish for all $\\theta_i$, we get \n \\[\n \\int_{\\Theta} J'_{\\nu^\\star} \\d\\nu = \\frac12 \\sum_{i=1}^m \\bar r_i^2 \\left( \\tr(\\Sigma_i H_i) +\\delta \\theta^\\intercal H_i \\delta \\theta_i \\right) + \\int_{\\Theta_0}J'_{\\nu^\\star} \\d\\nu  + O(\\tau (\\Vert s\\Vert^2+\\Vert  b^\\theta \\Vert^2))\n \\]\nand the expansion follows. Notice also that in the expression of $J(\\nu)$, $\\bar r_i$ and $r_i$ are interchangeable up to introducing higher order error, since $\\vert r_i - \\bar r_i\\vert = O(\\vert b^r_i\\vert)$ (and also $\\Vert \\tilde b^\\theta\\Vert = \\Vert b^\\theta\\Vert(1+O(W_\\tau(\\nu)))$).\n \\end{proof}","sha256_utf8":"e877b018984c2192233a321dfd6234b8f1ea2878eb2faf0ac64f5ffb689a463b"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"在最优特征处展开损失","source_units":[{"start":112608,"end":112913,"start_line":1112,"end_line":1115,"raw_tex":"\\begin{proof} Let us write $f_\\nu \\coloneqq \\int \\phi\\d\\nu$ and $f^\\star = \\int \\phi \\d\\nu^\\star$. By a second order Taylor expansion of $R$ around $f^\\star$, we have\n \\[\n J(\\nu) - J^\\star = \\int_{\\Theta} J'_{\\nu^\\star} \\d\\nu+ \\frac12 \\Vert f_\\nu-f^\\star\\Vert^2_\\star + O(\\Vert f_\\nu-f^\\star\\Vert^3).\n \\]\n","sha256_utf8":"1a96a9882af6c2c25e701a4f61bc3baa14c5c455cd4cfe32d8d8ef335113228c"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"展开特征与证书并归并余项","source_units":[{"start":112913,"end":113902,"start_line":1116,"end_line":1124,"raw_tex":" Using the first order expansion of Lemma~\\ref{lem:expansion} for $\\phi$, we get \n $\n \\Vert f_\\nu-f^\\star\\Vert^2_\\star %= b^\\intercal Kb + O\\left( (r_0^2 + \\sum_{i=1}^m s_i^2)(\\Vert b\\Vert + r_0^2 + \\sum_{i=1}^m s_i^2)\\right) \n = b^\\intercal Kb + O(W_{\\tau}(\\nu)^3)\n $. Also, using the second order expansion of Lemma~\\ref{lem:expansion} for $J'_{\\nu^\\star}$ and using the fact that $J'_{\\nu^\\star}$ and its gradient vanish for all $\\theta_i$, we get \n \\[\n \\int_{\\Theta} J'_{\\nu^\\star} \\d\\nu = \\frac12 \\sum_{i=1}^m \\bar r_i^2 \\left( \\tr(\\Sigma_i H_i) +\\delta \\theta^\\intercal H_i \\delta \\theta_i \\right) + \\int_{\\Theta_0}J'_{\\nu^\\star} \\d\\nu  + O(\\tau (\\Vert s\\Vert^2+\\Vert  b^\\theta \\Vert^2))\n \\]\nand the expansion follows. Notice also that in the expression of $J(\\nu)$, $\\bar r_i$ and $r_i$ are interchangeable up to introducing higher order error, since $\\vert r_i - \\bar r_i\\vert = O(\\vert b^r_i\\vert)$ (and also $\\Vert \\tilde b^\\theta\\Vert = \\Vert b^\\theta\\Vert(1+O(W_\\tau(\\nu)))$).\n","sha256_utf8":"2fda6ab378052c675228efdcb95602375e3f476a56ae56e1d70805a50489369e"}]},{"id":"S060","semantic_type":"external_fact","kind":"external_fact","title":"凸性与泰勒展开","source_units":[],"declared_external":true,"declaration_text":["Differentiable convex functions satisfy the first-order supporting inequality. A locally Lipschitz Hessian gives a second-order Taylor expansion with cubic remainder, and positive definite curvature at the minimizer gives the local coercive bound used in the author proof."]},{"id":"S061","semantic_type":"proposition","kind":"proposition","title":"梯度范数的局部展开与下界","source_units":[{"start":113996,"end":114683,"start_line":1131,"end_line":1140,"raw_tex":"\\begin{proposition}[Gradient estimate]\\label{prop:gradientestimate}\nFor $\\nu \\in \\Pp_2(\\Omega)$, it holds \n\\[\n\\Vert g_\\nu\\Vert^2_{L^2(\\nu)} =  b^\\intercal (K + H)^2b+ \\sum_{i=1}^m \\bar r_i^2 \\tr (\\Sigma_i H_i^2) +\\Vert g_\\nu\\Vert^2_{L^2(\\nu\\vert_{\\Theta_0})}+\\mathop{\\mathrm{err}}(\\tau,\\nu)\n\\]\nwhere $\\mathop{\\mathrm{err}}(\\tau,\\nu)\\lesssim \\tau(\\Vert \\tilde b^\\theta\\Vert^2 +\\Vert s\\Vert^2) +W_\\tau(\\nu)^3$. In particular, if $\\tau$ is fixed small enough\n\\[\n\\Vert g_\\nu\\Vert^2_{L^2(\\nu)}  \\geq \\frac12 (\\sigma_{\\min}(K)+ \\sigma_{\\min}(H))^2\\Vert b\\Vert^2 + \\frac12 \\sigma_{\\min}(H)^2 \\Vert s\\Vert^2 +  \\frac14 \\bar r_0^2 \\sigma_{\\min}(H)^2 \\tau^4 +O(W_\\tau(\\nu)^3).\n\\]\n\\end{proposition}","sha256_utf8":"d36353e4d37175bcf6aac0ce68c0f82eb20bf55e5b0de1264f09df70f002d2d4"}]},{"id":"S062","semantic_type":"proof","kind":"proof","title":"梯度估计完整证明","source_units":[{"start":114684,"end":122100,"start_line":1141,"end_line":1224,"raw_tex":"\\begin{proof}\nFor this proof, we write $f_\\nu - f^\\star = \\delta f_0 +\\delta f_b + \\delta f_{\\mathrm{err}}$ where\n\\begin{align*}\n \\delta f_0 &\\coloneqq  \\int_{\\Theta_0}\\phi(\\theta)\\d\\nu(\\theta),&\n \\delta f_b & \\coloneqq  \\sum_{i=1}^m r_i \\bar\\nabla \\phi(\\theta_i) b_{i},&\n\\delta f_{\\mathrm{err}} &\\coloneqq \\sum_{i=1}^{m} \\int_{\\Theta_i} M_{\\phi,2}(\\theta_i,\\theta)\\d\\nu(\\theta).\n\\end{align*}\nwhere the decomposition follows from Lemma~\\ref{lem:expansion}.\n%\nThe expression for the norm of the gradient is as follows:\n\\[\n\\Vert g_\\nu\\Vert^2_{L^2(\\nu)} = \\int_{\\Theta}\\Vert\\bar \\nabla J'_\\nu(\\theta) \\Vert^2\\d\\nu(\\theta)\n\\]\nwhere $\\bar \\nabla J = (2J,\\nabla J)$. We start with the following decomposition for $\\theta \\in \\Theta_i$ (recall that $J'_\\nu(\\theta) =\\langle \\phi(\\theta),\\nabla R(\\int \\phi \\d\\nu)\\rangle +\\lambda$):\n\\begin{multline*}\nJ'_\\nu(\\theta) = \\lambda + \\left\\langle \\phi(\\theta_i) + (\\theta-\\theta_i)^\\intercal \\nabla \\phi(\\theta_i) + \\frac12  (\\theta-\\theta_i)^\\intercal\\nabla^2\\phi(\\theta_i)(\\theta-\\theta_i)+  M_{\\phi,3}(\\theta_i,\\theta),\\nabla R(f^\\star)\\right\\rangle \\\\\n+\\langle \\phi(\\theta_i)+ (\\theta-\\theta_i)^\\intercal \\nabla \\phi(\\theta_i) + M_{\\phi,2}(\\theta_i,\\theta),f_\\mu-f^\\star\\rangle_{\\star}+ \\langle \\phi(\\theta),M_{\\nabla R,2}(f^\\star,f)\\rangle\n\\end{multline*}\nHere we use the notation $\\langle\\cdot,\\cdot\\rangle_\\star$ to denote the quadratic form associated to $\\nabla^2 R(f^\\star)$.\nThanks to the optimality conditions  $\\bar \\nabla J'_{\\nu^\\star}(\\theta_i)=0$ for $i\\in \\{1,\\dots,m\\}$, we get\n\\begin{align*}\n\\bar \\nabla_j J'_\\nu(\\theta) &=  [H_i(\\theta-\\theta_i)]_j + \\langle \\bar \\nabla_j \\phi(\\theta_i),\\delta f_0+\\delta f_b \\rangle_\\star + [N(\\theta_i,\\theta)]_j\\\\\n &=  [H_i(\\theta-\\bar \\theta_i)]_j+ \\left( [H_i(\\bar \\theta_i-\\theta_i)]_j+ \\langle \\bar \\nabla_j \\phi(\\theta_i),\\delta f_b \\rangle_\\star\\right) + \\langle \\bar \\nabla_j  \\phi(\\theta_i),\\delta f_0\\rangle_\\star + [N(\\theta_i,\\theta)]_j\n\\end{align*}\nwhere $N$ collects the higher order terms and is defined as\n\\begin{multline*}\n [N(\\theta_i,\\theta)]_j =  \\langle \\bar \\nabla_j M_{\\phi,3}(\\theta_i,\\theta),\\nabla R(f^\\star)\\rangle %+ \\langle \\partial_j \\phi(\\theta_i),N_1\\rangle \\\\\n +\\langle \\bar \\nabla_j M_{\\phi,2}(\\theta_i,\\theta),f-f^\\star\\rangle_{\\star} + \\langle \\bar \\nabla_j \\phi(\\theta),M_{\\nabla R,2}(f^\\star,f)\\rangle\n\\end{multline*}\nwhere $\\Vert \\bar \\nabla_j M_{\\phi,3}(\\theta_i,\\theta)\\Vert = O(\\Vert \\theta-\\theta_i\\Vert^2)$ if $j>0$ and $O(\\Vert \\theta -\\theta_i\\Vert^3)$ if $j=0$.\nExpanding the square gives the following ten terms:\n\\begin{align}\n\\int_{\\Omega\\setminus \\Omega_0} \\Vert\\bar \\nabla J'_\\nu \\Vert^2\\d\\nu(\\theta)\n&= \\sum_{i=1}^{m} \\int_{\\Theta_i} \\sum_{j=0}^{d} [H_i(\\theta-\\bar \\theta_i)]_j^2 \\d\\nu \\tag{I} \\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\sum_{j=0}^d \\left( \\langle\\bar \\nabla_j\\phi(\\theta_i),\\delta f_b \\rangle_\\star + [H_i(\\bar \\theta_i-\\theta_i)]_j \\right)^2 \\d\\nu(\\theta) \\tag{II} \\\\\n&+ \\sum_{i=1}^{m} \\int_{\\Theta_i} \\sum_{j=0}^d \\langle \\bar \\nabla_j\\phi(\\theta_i), \\delta f_0 \\rangle_\\star^2\\d\\nu(\\theta)  \\tag{III} \\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d  [H_i(\\theta-\\bar \\theta_i)]_j\\cdot ( \\langle\\bar \\nabla_j \\phi(\\theta_i),\\delta f_b\\rangle_\\star + [H_i(\\bar \\theta_i-\\theta_i)]_j) \\d\\nu(\\theta) \\tag{IV} \\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d [H_i(\\theta-\\bar \\theta_i)]_j\\cdot \\langle \\bar \\nabla_j\\phi(\\theta_i), \\delta f_0\\rangle_\\star\\d\\nu(\\theta) \\tag{V}\\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d (\\langle\\bar \\nabla_j \\phi(\\theta_i),\\delta f_b\\rangle_\\star + [H_i(\\bar \\theta_i-\\theta_i)]_j)\\cdot \\langle \\bar\\nabla_j \\phi(\\theta_i), \\delta f_0\\rangle_\\star\\d\\nu(\\theta)\\!\\! \\tag{VI}\\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d [N(\\theta_i,\\theta)]_j\\cdot  [H_i(\\theta-\\bar \\theta_i)]_j\\d\\nu(\\theta) \\tag{VII}\\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d [N(\\theta_i,\\theta)]_j\\cdot  (\\langle\\bar \\nabla_j \\phi(\\theta_i),\\delta f_b\\rangle_\\star + [H_i(\\bar \\theta_i-\\theta_i)]_j)\\d\\nu(\\theta) \\tag{VIII}\\\\\n&+\\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d [N(\\theta_i,\\theta)]_j\\cdot \\langle \\bar \\nabla_j \\phi(\\theta_i), \\delta f_0 \\rangle_\\star\\d\\nu(\\theta) \\tag{IX}\\\\\n&+ \\sum_{i=1}^{m} \\int_{\\Theta_i} \\sum_{j=0}^d [N(\\theta_i,\\theta)]_j^2 \\d\\nu(\\theta) \\tag{X}\n\\end{align}\nTerms (I) to (II) are the main terms in the expansion, while the other terms are higher order. The term (I) is a local curvature term and can be expressed as $\\mathrm{(I)} = \\sum_{i=1}^m \\bar r_i^2 \\tr \\Sigma_i H_i^2$. \n%\nThe term (II) is a global interaction term that writes\n\\begin{align*}\n\\mathrm{(II)} &=\n\\sum_{i=1}^m \\bar r_i^2 \\sum_{j=0}^d \\vert \\langle\\bar \\nabla_j\\phi(\\theta_i),\\delta f_b\\rangle_\\star + [H_i(\\bar \\theta_i-\\theta_i)]_j \\vert^2 \\\\\n&= \\sum_{i=1}^m\\sum_{j=0}^d \\left\\vert \\sum_{i'=1}^m \\sum_{j'=0}^d (\\langle \\bar r_i \\bar \\nabla_j\\phi(\\theta_i),r_{i'}\\bar \\nabla_{j'}\\phi(\\theta_{i'})\\rangle_\\star + \\bar H_{(i,j),(i,j')})(b_{i',j'}) \\right\\vert^2\\\\\n&= \\Vert (\\bar K + H)(b)\\Vert^2\n\\end{align*}\nwhere the entries of $\\bar K$ and $\\bar H$ differ from those of $K$ and $H$ by a factor $\\bar r_i/r_i$. More precisely,\n\\[\n[\\bar K-K]_{(i,j),(i',j')} = (\\bar r_i/r_i-1)K_{(i,j),(i',j')}\n\\]\nand similarly for $\\bar H- H$. Since $\\vert \\bar r_i/r_i-1\\vert = O(\\vert b^r_i\\vert)$ we have $\\sigma_{\\max}(\\bar K-K )=O(W_\\tau(\\nu))$. It follows, by expanding the square, that \n\\[\n\\Vert (\\bar K + \\bar H)(b)\\Vert^2 = \\Vert (K + H)(b)\\Vert^2 + O(W_\\tau(\\nu)^3).\n\\]\n\nThe remaining terms are error terms, that we estimate directly in terms of $W_\\tau(\\nu)$ and $\\tau$. We use in particular the fact that by H\\\"older's inequality, $\\int_{\\Theta_i} \\Vert \\theta - \\bar \\theta_i\\Vert\\d\\nu(\\theta) = O(\\bar r_i^2 \\tr \\Sigma_i^{\\frac12})$. One has \n\\begin{itemize}\n\\item $\\mathrm{(III)} =O\\left(\\sum_{i=1}^m \\bar r_i^2 \\bar r_0^4 \\right)= O(W_{\\tau}^4(\\nu))$;\n\\item $\\mathrm{(IV)} = \\mathrm{(V)} = 0$ because the integral of the terms $H_i(\\theta-\\bar \\theta_i)$ vanishes;\n%\\item $\\mathrm{(V)} = O(\\sum \\bar r_i^2 \\sigma_i \\bar r_0^2) = O( W_{\\tau}^3)$;\n\\item $\\mathrm{(VI)} = O\\left((\\sum_{i=1}^m \\bar r_i^2 (\\Vert b\\Vert +\\Vert \\delta \\theta_i\\Vert)\\cdot \\bar r_0^2\\right)=O(W_{\\tau}^3(\\nu))$;\n\\item $\\mathrm{(VII)} = O(\\tau (\\Vert \\tilde b^\\theta\\Vert^2 + \\Vert s\\Vert^2)) + O(W_{\\tau}(\\nu)^3)$;\n\\item $\\mathrm{(VIII)} = O(W_{\\tau}^3(\\nu))$;\n\\item $\\mathrm{(IX)} = O(W_{\\tau}^4(\\nu))$;\n\\item $\\mathrm{(X)} = O(\\tau^2(\\Vert \\tilde b^\\theta\\Vert^2+\\Vert s\\Vert^2))+O(W_{\\tau}(\\nu)^4)$.\n\\end{itemize}\nIt follows that overall, the error term is in $O(\\tau (\\Vert \\tilde b^\\theta\\Vert^2+\\Vert s\\Vert^2)+W_{\\tau}(\\nu)^3)$.\nThere remains to lower bound the norm of the gradient over $\\Theta_0$, which can be done as follows. As seen in the proof of Lemma~\\ref{lem:distancebound}, if $\\tau$ is small enough then $J'_{\\nu^\\star}(\\theta)\\geq \\tau^2\\sigma_{\\min}(H)/4$ for $\\theta \\in \\Theta_0$. Considering only the first component of the gradient, it holds\n\\[\n\\int_{\\Theta_0} \\Vert\\bar \\nabla J'_\\nu(\\theta) \\Vert^2\\d\\nu(\\theta) \\geq \\int_{\\Theta_0} 4 \\vert J'_\\nu(\\theta)\\vert^2 \\d\\nu(\\theta).\n\\]\nUsing the expansion $J'_\\nu(\\theta)=J'_{\\nu^\\star}(\\theta)+\\langle \\phi(\\theta),M_{\\nabla R,1}(f^\\star,f_\\nu)\\rangle$, we get\n\\[\n\\int_{\\Theta_0} \\Vert\\bar \\nabla J'_\\nu(\\theta) \\Vert^2\\d\\nu(\\theta) \\geq  C \\bar r_0^2 \\tau^4 + O(W_\\tau(\\nu)^3).\n\\]\nThe result follows by collecting all the estimates above.\n\\end{proof}","sha256_utf8":"fbba79f1a1e959b4006596437c73894f1c01ab1c96506c55cbe42a9a53087705"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"分解特征扰动的三种来源","source_units":[{"start":114698,"end":115141,"start_line":1142,"end_line":1148,"raw_tex":"For this proof, we write $f_\\nu - f^\\star = \\delta f_0 +\\delta f_b + \\delta f_{\\mathrm{err}}$ where\n\\begin{align*}\n \\delta f_0 &\\coloneqq  \\int_{\\Theta_0}\\phi(\\theta)\\d\\nu(\\theta),&\n \\delta f_b & \\coloneqq  \\sum_{i=1}^m r_i \\bar\\nabla \\phi(\\theta_i) b_{i},&\n\\delta f_{\\mathrm{err}} &\\coloneqq \\sum_{i=1}^{m} \\int_{\\Theta_i} M_{\\phi,2}(\\theta_i,\\theta)\\d\\nu(\\theta).\n\\end{align*}\nwhere the decomposition follows from Lemma~\\ref{lem:expansion}.\n","sha256_utf8":"f2ce8ac708a8a95d2823e800f1bd27b6fa578ebad022f9e99884aa80dffc7604"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"写出梯度范数并展开变分","source_units":[{"start":115143,"end":116106,"start_line":1150,"end_line":1159,"raw_tex":"The expression for the norm of the gradient is as follows:\n\\[\n\\Vert g_\\nu\\Vert^2_{L^2(\\nu)} = \\int_{\\Theta}\\Vert\\bar \\nabla J'_\\nu(\\theta) \\Vert^2\\d\\nu(\\theta)\n\\]\nwhere $\\bar \\nabla J = (2J,\\nabla J)$. We start with the following decomposition for $\\theta \\in \\Theta_i$ (recall that $J'_\\nu(\\theta) =\\langle \\phi(\\theta),\\nabla R(\\int \\phi \\d\\nu)\\rangle +\\lambda$):\n\\begin{multline*}\nJ'_\\nu(\\theta) = \\lambda + \\left\\langle \\phi(\\theta_i) + (\\theta-\\theta_i)^\\intercal \\nabla \\phi(\\theta_i) + \\frac12  (\\theta-\\theta_i)^\\intercal\\nabla^2\\phi(\\theta_i)(\\theta-\\theta_i)+  M_{\\phi,3}(\\theta_i,\\theta),\\nabla R(f^\\star)\\right\\rangle \\\\\n+\\langle \\phi(\\theta_i)+ (\\theta-\\theta_i)^\\intercal \\nabla \\phi(\\theta_i) + M_{\\phi,2}(\\theta_i,\\theta),f_\\mu-f^\\star\\rangle_{\\star}+ \\langle \\phi(\\theta),M_{\\nabla R,2}(f^\\star,f)\\rangle\n\\end{multline*}\nHere we use the notation $\\langle\\cdot,\\cdot\\rangle_\\star$ to denote the quadratic form associated to $\\nabla^2 R(f^\\star)$.\n","sha256_utf8":"78b19c775d72f7e8014b2d61821d962d92da9b130714338d3e05d39bdf310990"}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","title":"分离局部波动与全局偏差","source_units":[{"start":116106,"end":117186,"start_line":1160,"end_line":1170,"raw_tex":"Thanks to the optimality conditions  $\\bar \\nabla J'_{\\nu^\\star}(\\theta_i)=0$ for $i\\in \\{1,\\dots,m\\}$, we get\n\\begin{align*}\n\\bar \\nabla_j J'_\\nu(\\theta) &=  [H_i(\\theta-\\theta_i)]_j + \\langle \\bar \\nabla_j \\phi(\\theta_i),\\delta f_0+\\delta f_b \\rangle_\\star + [N(\\theta_i,\\theta)]_j\\\\\n &=  [H_i(\\theta-\\bar \\theta_i)]_j+ \\left( [H_i(\\bar \\theta_i-\\theta_i)]_j+ \\langle \\bar \\nabla_j \\phi(\\theta_i),\\delta f_b \\rangle_\\star\\right) + \\langle \\bar \\nabla_j  \\phi(\\theta_i),\\delta f_0\\rangle_\\star + [N(\\theta_i,\\theta)]_j\n\\end{align*}\nwhere $N$ collects the higher order terms and is defined as\n\\begin{multline*}\n [N(\\theta_i,\\theta)]_j =  \\langle \\bar \\nabla_j M_{\\phi,3}(\\theta_i,\\theta),\\nabla R(f^\\star)\\rangle %+ \\langle \\partial_j \\phi(\\theta_i),N_1\\rangle \\\\\n +\\langle \\bar \\nabla_j M_{\\phi,2}(\\theta_i,\\theta),f-f^\\star\\rangle_{\\star} + \\langle \\bar \\nabla_j \\phi(\\theta),M_{\\nabla R,2}(f^\\star,f)\\rangle\n\\end{multline*}\nwhere $\\Vert \\bar \\nabla_j M_{\\phi,3}(\\theta_i,\\theta)\\Vert = O(\\Vert \\theta-\\theta_i\\Vert^2)$ if $j>0$ and $O(\\Vert \\theta -\\theta_i\\Vert^3)$ if $j=0$.\n","sha256_utf8":"6f461d7e95e6e30a29c27e37941fab40f76973702667d5117ba07cd4f8727250"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"平方展开并识别局部曲率项","source_units":[{"start":117186,"end":119209,"start_line":1171,"end_line":1185,"raw_tex":"Expanding the square gives the following ten terms:\n\\begin{align}\n\\int_{\\Omega\\setminus \\Omega_0} \\Vert\\bar \\nabla J'_\\nu \\Vert^2\\d\\nu(\\theta)\n&= \\sum_{i=1}^{m} \\int_{\\Theta_i} \\sum_{j=0}^{d} [H_i(\\theta-\\bar \\theta_i)]_j^2 \\d\\nu \\tag{I} \\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\sum_{j=0}^d \\left( \\langle\\bar \\nabla_j\\phi(\\theta_i),\\delta f_b \\rangle_\\star + [H_i(\\bar \\theta_i-\\theta_i)]_j \\right)^2 \\d\\nu(\\theta) \\tag{II} \\\\\n&+ \\sum_{i=1}^{m} \\int_{\\Theta_i} \\sum_{j=0}^d \\langle \\bar \\nabla_j\\phi(\\theta_i), \\delta f_0 \\rangle_\\star^2\\d\\nu(\\theta)  \\tag{III} \\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d  [H_i(\\theta-\\bar \\theta_i)]_j\\cdot ( \\langle\\bar \\nabla_j \\phi(\\theta_i),\\delta f_b\\rangle_\\star + [H_i(\\bar \\theta_i-\\theta_i)]_j) \\d\\nu(\\theta) \\tag{IV} \\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d [H_i(\\theta-\\bar \\theta_i)]_j\\cdot \\langle \\bar \\nabla_j\\phi(\\theta_i), \\delta f_0\\rangle_\\star\\d\\nu(\\theta) \\tag{V}\\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d (\\langle\\bar \\nabla_j \\phi(\\theta_i),\\delta f_b\\rangle_\\star + [H_i(\\bar \\theta_i-\\theta_i)]_j)\\cdot \\langle \\bar\\nabla_j \\phi(\\theta_i), \\delta f_0\\rangle_\\star\\d\\nu(\\theta)\\!\\! \\tag{VI}\\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d [N(\\theta_i,\\theta)]_j\\cdot  [H_i(\\theta-\\bar \\theta_i)]_j\\d\\nu(\\theta) \\tag{VII}\\\\\n&+ \\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d [N(\\theta_i,\\theta)]_j\\cdot  (\\langle\\bar \\nabla_j \\phi(\\theta_i),\\delta f_b\\rangle_\\star + [H_i(\\bar \\theta_i-\\theta_i)]_j)\\d\\nu(\\theta) \\tag{VIII}\\\\\n&+\\sum_{i=1}^{m}\\int_{\\Theta_i} \\!\\! 2 \\sum_{j=0}^d [N(\\theta_i,\\theta)]_j\\cdot \\langle \\bar \\nabla_j \\phi(\\theta_i), \\delta f_0 \\rangle_\\star\\d\\nu(\\theta) \\tag{IX}\\\\\n&+ \\sum_{i=1}^{m} \\int_{\\Theta_i} \\sum_{j=0}^d [N(\\theta_i,\\theta)]_j^2 \\d\\nu(\\theta) \\tag{X}\n\\end{align}\nTerms (I) to (II) are the main terms in the expansion, while the other terms are higher order. The term (I) is a local curvature term and can be expressed as $\\mathrm{(I)} = \\sum_{i=1}^m \\bar r_i^2 \\tr \\Sigma_i H_i^2$. \n","sha256_utf8":"9fa24a37f80e2ee78292362c51d46f5e25baffa079411dbdd432c6eea0fb0387"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"识别全局核的主导二次型","source_units":[{"start":119211,"end":120164,"start_line":1187,"end_line":1201,"raw_tex":"The term (II) is a global interaction term that writes\n\\begin{align*}\n\\mathrm{(II)} &=\n\\sum_{i=1}^m \\bar r_i^2 \\sum_{j=0}^d \\vert \\langle\\bar \\nabla_j\\phi(\\theta_i),\\delta f_b\\rangle_\\star + [H_i(\\bar \\theta_i-\\theta_i)]_j \\vert^2 \\\\\n&= \\sum_{i=1}^m\\sum_{j=0}^d \\left\\vert \\sum_{i'=1}^m \\sum_{j'=0}^d (\\langle \\bar r_i \\bar \\nabla_j\\phi(\\theta_i),r_{i'}\\bar \\nabla_{j'}\\phi(\\theta_{i'})\\rangle_\\star + \\bar H_{(i,j),(i,j')})(b_{i',j'}) \\right\\vert^2\\\\\n&= \\Vert (\\bar K + H)(b)\\Vert^2\n\\end{align*}\nwhere the entries of $\\bar K$ and $\\bar H$ differ from those of $K$ and $H$ by a factor $\\bar r_i/r_i$. More precisely,\n\\[\n[\\bar K-K]_{(i,j),(i',j')} = (\\bar r_i/r_i-1)K_{(i,j),(i',j')}\n\\]\nand similarly for $\\bar H- H$. Since $\\vert \\bar r_i/r_i-1\\vert = O(\\vert b^r_i\\vert)$ we have $\\sigma_{\\max}(\\bar K-K )=O(W_\\tau(\\nu))$. It follows, by expanding the square, that \n\\[\n\\Vert (\\bar K + \\bar H)(b)\\Vert^2 = \\Vert (K + H)(b)\\Vert^2 + O(W_\\tau(\\nu)^3).\n\\]\n","sha256_utf8":"4a3c8457ce71fa7f974b9fb64d3aae2b6ab8dcf26fa9cfca55c92558f0450c42"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"逐项估计其余交叉与余项","source_units":[{"start":120165,"end":121312,"start_line":1203,"end_line":1214,"raw_tex":"The remaining terms are error terms, that we estimate directly in terms of $W_\\tau(\\nu)$ and $\\tau$. We use in particular the fact that by H\\\"older's inequality, $\\int_{\\Theta_i} \\Vert \\theta - \\bar \\theta_i\\Vert\\d\\nu(\\theta) = O(\\bar r_i^2 \\tr \\Sigma_i^{\\frac12})$. One has \n\\begin{itemize}\n\\item $\\mathrm{(III)} =O\\left(\\sum_{i=1}^m \\bar r_i^2 \\bar r_0^4 \\right)= O(W_{\\tau}^4(\\nu))$;\n\\item $\\mathrm{(IV)} = \\mathrm{(V)} = 0$ because the integral of the terms $H_i(\\theta-\\bar \\theta_i)$ vanishes;\n%\\item $\\mathrm{(V)} = O(\\sum \\bar r_i^2 \\sigma_i \\bar r_0^2) = O( W_{\\tau}^3)$;\n\\item $\\mathrm{(VI)} = O\\left((\\sum_{i=1}^m \\bar r_i^2 (\\Vert b\\Vert +\\Vert \\delta \\theta_i\\Vert)\\cdot \\bar r_0^2\\right)=O(W_{\\tau}^3(\\nu))$;\n\\item $\\mathrm{(VII)} = O(\\tau (\\Vert \\tilde b^\\theta\\Vert^2 + \\Vert s\\Vert^2)) + O(W_{\\tau}(\\nu)^3)$;\n\\item $\\mathrm{(VIII)} = O(W_{\\tau}^3(\\nu))$;\n\\item $\\mathrm{(IX)} = O(W_{\\tau}^4(\\nu))$;\n\\item $\\mathrm{(X)} = O(\\tau^2(\\Vert \\tilde b^\\theta\\Vert^2+\\Vert s\\Vert^2))+O(W_{\\tau}(\\nu)^4)$.\n\\end{itemize}\nIt follows that overall, the error term is in $O(\\tau (\\Vert \\tilde b^\\theta\\Vert^2+\\Vert s\\Vert^2)+W_{\\tau}(\\nu)^3)$.\n","sha256_utf8":"00cec022f26fc403da59148cf290ef50e15306eabd22ef87ea2de35058119d7b"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"控制分区外梯度并汇总","source_units":[{"start":121312,"end":122089,"start_line":1215,"end_line":1223,"raw_tex":"There remains to lower bound the norm of the gradient over $\\Theta_0$, which can be done as follows. As seen in the proof of Lemma~\\ref{lem:distancebound}, if $\\tau$ is small enough then $J'_{\\nu^\\star}(\\theta)\\geq \\tau^2\\sigma_{\\min}(H)/4$ for $\\theta \\in \\Theta_0$. Considering only the first component of the gradient, it holds\n\\[\n\\int_{\\Theta_0} \\Vert\\bar \\nabla J'_\\nu(\\theta) \\Vert^2\\d\\nu(\\theta) \\geq \\int_{\\Theta_0} 4 \\vert J'_\\nu(\\theta)\\vert^2 \\d\\nu(\\theta).\n\\]\nUsing the expansion $J'_\\nu(\\theta)=J'_{\\nu^\\star}(\\theta)+\\langle \\phi(\\theta),M_{\\nabla R,1}(f^\\star,f_\\nu)\\rangle$, we get\n\\[\n\\int_{\\Theta_0} \\Vert\\bar \\nabla J'_\\nu(\\theta) \\Vert^2\\d\\nu(\\theta) \\geq  C \\bar r_0^2 \\tau^4 + O(W_\\tau(\\nu)^3).\n\\]\nThe result follows by collecting all the estimates above.\n","sha256_utf8":"aa12472db36a24e72ad94605a1f3b962993d623da7fa8bb3fcfec868b0cc5776"}]},{"id":"S070","semantic_type":"theorem","kind":"theorem","title":"局部梯度锐性不等式","source_units":[{"start":53039,"end":53373,"start_line":516,"end_line":520,"raw_tex":"\\begin{theorem}[Sharpness]\\label{th:gradientinequality} Under {\\sf(A1-5)}, there exists $J_0>J^\\star$ and $\\kappa_0>0$, such that for all $\\nu\\in \\Mm_+(\\Theta)$ satisfying $J(\\nu) \\leq  J_0$ and $\\alpha,\\beta>0$, one has\n\\[\n\\frac12 \\Vert g_\\nu\\Vert^2_{L^2(\\nu)}\\geq \\kappa_0\\, \\min\\{\\alpha,\\beta\\}\\, (J(\\nu)-J^\\star).\n\\]\n\\end{theorem}","sha256_utf8":"434aca2f219f436576e0d287dc73e2a02e203517e88615d6af61eb2e23f19278"}]},{"id":"S071","semantic_type":"proof","kind":"proof","title":"锐性不等式完整证明","source_units":[{"start":122102,"end":123807,"start_line":1226,"end_line":1250,"raw_tex":"\\subsection{Proof of the sharpness inequality (Theorem~\\ref{th:gradientinequality})}\\label{app:conclusionlocal}\nBy Proposition~\\ref{prop:localexpansion} we have that for $\\tau>0$ small enough\n\\[\n J(\\nu)-J^\\star \\leq CW_\\tau(\\nu)^2 + O(W_\\tau(\\nu)^3)\n\\]\nwhere $C = \\sigma_{\\max}(K+H) +\\Vert J'_{\\nu^\\star}\\Vert_\\infty$.\n\nSimilarly, by Proposition~\\ref{prop:gradientestimate}, for $\\tau$ small enough, it holds\n\\[\n\\Vert g_\\nu\\Vert^2_{L^2(\\nu)}  \\geq C'W_\\tau(\\nu)^2 +O(W_\\tau(\\nu)^3)\n\\]\nwhere $C'= \\frac18\\sigma_{\\min}(H)^2\\tau^4$. Now fix $\\tau>0$ satisfying the hypothesis of Lemma~\\ref{lem:distancebound} and the two previous inequalities. By Lemma~\\ref{lem:distancebound},  $W_\\tau(\\nu) = O((J(\\nu)-J^\\star)^\\frac12)$. We deduce that there exists $J_0>J^\\star$ and $\\kappa_0>0$ , such that whenever  $\\nu\\in \\Mm_+(\\Theta)$ satisfies $J(\\nu)<J_0$, one has\n\\[\n\\Vert g_\\nu\\Vert^2_{L^2(\\nu)}\\geq \\kappa_0(J(\\nu)-J^\\star).\n\\]\nFinally, notice that if different metric factors $(\\alpha,\\beta)\\neq (1,1)$ are introduced, one can always lower bound the new gradient squared norm as\n\\[\n\\int_\\Theta \\left( 4 \\alpha \\vert J'_\\nu(\\theta)\\vert^2 +\\beta \\Vert \\nabla J'_\\nu(\\theta)\\Vert^2_\\theta \\right)\\d\\nu(\\theta)\n\\geq \\min\\{\\alpha,\\beta\\} \\int_\\Theta \\left( 4 \\vert J'_\\nu(\\theta)\\vert^2 + \\Vert \\nabla J'_\\nu(\\theta)\\Vert^2_\\theta \\right)\n \\]\n which proves the statement for any $(\\alpha,\\beta)$. Note however that if one wants to make a more quantitative bound, then there are values $(\\alpha_0,\\beta_0)$ that would lead to a better conditioning and potentially higher values for $J_0$. In this case, the factor appearing in the sharpness inequality should rather be $\\min\\{\\alpha/\\alpha_0,\\beta/\\beta_0\\}$.\n\n\n\n\n","sha256_utf8":"dc66da56015b5d5a07f848ce6121fc9b40c5cf0bca5b7a355f9a3fa60486b501"}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","title":"以局部误差上界目标差","source_units":[{"start":122186,"end":122421,"start_line":1226,"end_line":1231,"raw_tex":"\\label{app:conclusionlocal}\nBy Proposition~\\ref{prop:localexpansion} we have that for $\\tau>0$ small enough\n\\[\n J(\\nu)-J^\\star \\leq CW_\\tau(\\nu)^2 + O(W_\\tau(\\nu)^3)\n\\]\nwhere $C = \\sigma_{\\max}(K+H) +\\Vert J'_{\\nu^\\star}\\Vert_\\infty$.\n","sha256_utf8":"290a30a98f3f0073431336acb7e980ebe177eee3f860ae89ee3d31d5730dc9cd"}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","title":"比较梯度下界并吸收高阶项","source_units":[{"start":122422,"end":123025,"start_line":1233,"end_line":1240,"raw_tex":"Similarly, by Proposition~\\ref{prop:gradientestimate}, for $\\tau$ small enough, it holds\n\\[\n\\Vert g_\\nu\\Vert^2_{L^2(\\nu)}  \\geq C'W_\\tau(\\nu)^2 +O(W_\\tau(\\nu)^3)\n\\]\nwhere $C'= \\frac18\\sigma_{\\min}(H)^2\\tau^4$. Now fix $\\tau>0$ satisfying the hypothesis of Lemma~\\ref{lem:distancebound} and the two previous inequalities. By Lemma~\\ref{lem:distancebound},  $W_\\tau(\\nu) = O((J(\\nu)-J^\\star)^\\frac12)$. We deduce that there exists $J_0>J^\\star$ and $\\kappa_0>0$ , such that whenever  $\\nu\\in \\Mm_+(\\Theta)$ satisfies $J(\\nu)<J_0$, one has\n\\[\n\\Vert g_\\nu\\Vert^2_{L^2(\\nu)}\\geq \\kappa_0(J(\\nu)-J^\\star).\n\\]\n","sha256_utf8":"6627959989daa938d89af841d3bfc10c16548eec597028bf50d261118dd4562c"}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","title":"推广到任意正度量参数","source_units":[{"start":123025,"end":123803,"start_line":1241,"end_line":1246,"raw_tex":"Finally, notice that if different metric factors $(\\alpha,\\beta)\\neq (1,1)$ are introduced, one can always lower bound the new gradient squared norm as\n\\[\n\\int_\\Theta \\left( 4 \\alpha \\vert J'_\\nu(\\theta)\\vert^2 +\\beta \\Vert \\nabla J'_\\nu(\\theta)\\Vert^2_\\theta \\right)\\d\\nu(\\theta)\n\\geq \\min\\{\\alpha,\\beta\\} \\int_\\Theta \\left( 4 \\vert J'_\\nu(\\theta)\\vert^2 + \\Vert \\nabla J'_\\nu(\\theta)\\Vert^2_\\theta \\right)\n \\]\n which proves the statement for any $(\\alpha,\\beta)$. Note however that if one wants to make a more quantitative bound, then there are values $(\\alpha_0,\\beta_0)$ that would lead to a better conditioning and potentially higher values for $J_0$. In this case, the factor appearing in the sharpness inequality should rather be $\\min\\{\\alpha/\\alpha_0,\\beta/\\beta_0\\}$.\n","sha256_utf8":"4a6e28beb57d3969500e81f1b0b40456690ad4c40c0f209d44ba37de023fd65a"}]},{"id":"S075","semantic_type":"corollary","kind":"corollary","title":"梯度流的局部指数收敛","source_units":[{"start":54017,"end":54433,"start_line":525,"end_line":530,"raw_tex":"\\begin{corollary}[Local convergence of gradient flow]\\label{cor:localGF}\nUnder {\\sf (A1-5)}, let $J_0$ and $\\kappa_0$ be given by Theorem~\\ref{th:gradientinequality}. Consider $(\\nu_t)_{t\\geq 0}$ a projected gradient flow for $J$ as in Eq.~\\eqref{eq:projectedGF}. If $J(\\nu_0)\\leq J_0$ then \n\\begin{align*}\nJ(\\nu_t) - J^* &\\leq  (J(\\nu_0) - J^*) \\exp(-2\\kappa_0 \\min\\{\\alpha,\\beta\\}  t).\n\\end{align*}\n\\end{corollary}","sha256_utf8":"e2636b9c98ee311be6341e617f2d39146d8d1b2157c24f387d0782e7cb9eb8eb"}]},{"id":"S076","semantic_type":"proof","kind":"proof","title":"局部指数收敛完整证明","source_units":[{"start":54434,"end":54760,"start_line":531,"end_line":537,"raw_tex":"\\begin{proof}\nBy Theorem~\\ref{th:gradientinequality} and direct computations, one has \n\\[\n\\frac{d}{dt} (J(\\nu_t)-J^\\star) = \\int_{\\Omega} J'_{\\nu_t} \\d(\\partial_t \\nu_t) = - \\Vert g_{\\nu_t}\\Vert^2_{L^2(\\nu_t)} \\leq -2\\kappa_0\\min\\{\\alpha,\\beta \\} (J(\\nu_t)-J^\\star)\n\\]\nand the result follows by Gr\\\"onwall's lemma.\n\\end{proof}","sha256_utf8":"84b3aadf5359d3939f45fbceda5db0c0d282fbf42cffd32e4593c1c18f53f015"}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","title":"能量耗散结合格朗沃尔引理","source_units":[{"start":54448,"end":54749,"start_line":532,"end_line":536,"raw_tex":"By Theorem~\\ref{th:gradientinequality} and direct computations, one has \n\\[\n\\frac{d}{dt} (J(\\nu_t)-J^\\star) = \\int_{\\Omega} J'_{\\nu_t} \\d(\\partial_t \\nu_t) = - \\Vert g_{\\nu_t}\\Vert^2_{L^2(\\nu_t)} \\leq -2\\kappa_0\\min\\{\\alpha,\\beta \\} (J(\\nu_t)-J^\\star)\n\\]\nand the result follows by Gr\\\"onwall's lemma.\n","sha256_utf8":"3e96ec116ee9845c66ca653606838bb93a4d8d4cba69e3c3ec282743d5ff3047"}]},{"id":"S078","semantic_type":"lemma","kind":"lemma","title":"扰动镜像流的目标差估计","source_units":[{"start":72618,"end":73062,"start_line":669,"end_line":674,"raw_tex":"\\begin{lemma}\\label{lem:mirrordecrease}\nAssume ${\\sf (A1-3)}$ and that $J$ admits a minimizer $\\nu^\\star\\in \\Mm_+(\\Theta)$. Then for all $\\nu_0\\in \\Mm_+(\\Theta)$, denoting $B_{\\nu_0}\\coloneqq \\sup_{J(\\nu)\\leq J(\\nu_0)} \\Vert J'_{\\nu}\\Vert_{\\BL}$, it holds for $t\\geq 0$,\n\\begin{align*}\nJ(\\nu_t)-J^\\star \\leq \\inf_{s\\in [0,t]}\\left( B_{\\nu_0} \\cdot \\Qq_{\\nu^\\star,\\nu_0}(4\\alpha B_{\\nu_0}s) + \\beta B^2_{\\nu_0} s\\right).\n\\end{align*}\n\\end{lemma}","sha256_utf8":"5d69cb6a0319b0ec9b06fff8de6d3101edf69f55bc49eea19d574995a16d81f9"}]},{"id":"S079","semantic_type":"proof","kind":"proof","title":"镜像下降估计完整证明","source_units":[{"start":73435,"end":75544,"start_line":676,"end_line":695,"raw_tex":"\\begin{proof}\nLet $\\nu^\\epsilon_0 \\in \\Mm_+(\\Theta)$ be a measure to be specified later that satisfies $\\Hh(\\nu^\\epsilon_0, \\nu_0)<+\\infty$, and let $\\nu^\\epsilon_t$ satisfy $\\partial_t \\nu^\\epsilon_t = \\div(\\beta \\nu^\\epsilon_t\\nabla J'_{\\nu_t})$ for $t\\geq 0$ weakly (this is a continuity equation with a smooth velocity field which admits a unique weak solution). Differentiating the relative entropy with respect to its second argument and using the invariance of the relative entropy under diffeomorphisms, it holds, for $t\\geq 0$,\n\\begin{align*}\n\\frac1{4\\alpha} \\frac{d}{dt} \\Hh(\\nu^\\epsilon_t, \\nu_t) &= \\int_{\\Theta} J'_{\\nu_t} \\d (\\nu^\\epsilon_t-\\nu_t) \\\\\n& = \\int_{\\Theta} J'_{\\nu_t} \\d(\\nu^\\star -\\nu_t) + \\int_{\\Theta} J'_{\\nu_t} \\d(\\nu^\\epsilon_t-\\nu^\\star) \\\\\n& \\leq -(J(\\nu_t)-J^\\star) + \\Vert J'_{\\nu_t}\\Vert_\\BL \\cdot \\Vert \\nu^\\star-\\nu^\\epsilon_t\\Vert_\\BL^*\n\\end{align*}\nwhere the first term comes from the convexity of $J$ and the second from the definition of $\\Vert \\cdot\\Vert_{\\BL}^*$. After integrating in time and rearranging the terms we get\n\\begin{align*}\n\\left(\\frac1t\\int_0^t J(\\nu_s)\\d s\\right)-J^\\star &\\leq \\frac1{4\\alpha t} \\left( \\Hh(\\nu^\\epsilon_0,\\nu_0) - \\Hh(\\nu^\\epsilon_t,\\nu_t) \\right) + \\frac{B_{\\nu_0}}{t} \\int_0^t \\Vert \\nu^\\star-\\nu^\\epsilon_s\\Vert_{\\BL}^* \\d s.\n\\end{align*}\nFor the last integral term, we use the triangular inequality\n\\[\n \\frac{B_{\\nu_0}}{t}  \\int_0^t \\Vert \\nu^\\star-\\nu^\\epsilon_s\\Vert_{\\BL}^* \\d s \\leq B_{\\nu_0}\\Vert \\nu^\\star - \\nu^\\epsilon_0\\Vert_{\\BL}^* + \\frac{B_{\\nu_0}}{t} \\int_0^t \\Vert \\nu_0^\\epsilon -\\nu^\\epsilon_s\\Vert_{\\BL}^*\\d s\\leq B_{\\nu_0} \\Vert \\nu^\\star - \\nu^\\epsilon_0\\Vert_{\\BL}^* +B_{\\nu_0}^2\\beta t\n\\]\nwhere the last term is obtained by bounding the integrated flow of the velocity field $(\\nabla J'_{\\nu_t})_{t\\geq 0}$. Since $\\Hh(\\nu_\\epsilon,\\nu_t)\\geq 0$ and $J(\\nu_s)$ is decreasing, it follows\n\\[\nJ(\\nu_t)-J^\\star  \\leq \\inf_{\\nu^\\epsilon\\in \\Mm_+(\\Theta)} \\left( \\frac1{4\\alpha t} \\Hh(\\nu^\\epsilon,\\nu_0) + B_{\\nu_0}\\Vert  \\nu^\\star -\\nu^\\epsilon\\Vert_{\\BL}^*\\right) + B_{\\nu_0}^2 \\beta t.\\qedhere\n\\]\n\\end{proof}","sha256_utf8":"1d46567219791986ad006a2b53e9211b887de5bba1498cfc6483d45dc783ee40"}]},{"id":"S080","semantic_type":"proof_step","kind":"proof_step","title":"构造随流比较测度并微分熵","source_units":[{"start":73449,"end":74325,"start_line":677,"end_line":682,"raw_tex":"Let $\\nu^\\epsilon_0 \\in \\Mm_+(\\Theta)$ be a measure to be specified later that satisfies $\\Hh(\\nu^\\epsilon_0, \\nu_0)<+\\infty$, and let $\\nu^\\epsilon_t$ satisfy $\\partial_t \\nu^\\epsilon_t = \\div(\\beta \\nu^\\epsilon_t\\nabla J'_{\\nu_t})$ for $t\\geq 0$ weakly (this is a continuity equation with a smooth velocity field which admits a unique weak solution). Differentiating the relative entropy with respect to its second argument and using the invariance of the relative entropy under diffeomorphisms, it holds, for $t\\geq 0$,\n\\begin{align*}\n\\frac1{4\\alpha} \\frac{d}{dt} \\Hh(\\nu^\\epsilon_t, \\nu_t) &= \\int_{\\Theta} J'_{\\nu_t} \\d (\\nu^\\epsilon_t-\\nu_t) \\\\\n& = \\int_{\\Theta} J'_{\\nu_t} \\d(\\nu^\\star -\\nu_t) + \\int_{\\Theta} J'_{\\nu_t} \\d(\\nu^\\epsilon_t-\\nu^\\star) \\\\\n& \\leq -(J(\\nu_t)-J^\\star) + \\Vert J'_{\\nu_t}\\Vert_\\BL \\cdot \\Vert \\nu^\\star-\\nu^\\epsilon_t\\Vert_\\BL^*\n\\end{align*}\n","sha256_utf8":"f4576b22dba3f9a72d6e431542be58707dd7a4ae54fc250f14acfe8304b2d23b"}]},{"id":"S081","semantic_type":"proof_step","kind":"proof_step","title":"积分熵不等式控制平均目标差","source_units":[{"start":74325,"end":74755,"start_line":683,"end_line":686,"raw_tex":"where the first term comes from the convexity of $J$ and the second from the definition of $\\Vert \\cdot\\Vert_{\\BL}^*$. After integrating in time and rearranging the terms we get\n\\begin{align*}\n\\left(\\frac1t\\int_0^t J(\\nu_s)\\d s\\right)-J^\\star &\\leq \\frac1{4\\alpha t} \\left( \\Hh(\\nu^\\epsilon_0,\\nu_0) - \\Hh(\\nu^\\epsilon_t,\\nu_t) \\right) + \\frac{B_{\\nu_0}}{t} \\int_0^t \\Vert \\nu^\\star-\\nu^\\epsilon_s\\Vert_{\\BL}^* \\d s.\n\\end{align*}\n","sha256_utf8":"fb1a7a4a669ca0e95c66adfc119a5b04514daf6ae38dbbd096e4197a4a0ccce0"}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","title":"估计比较测度的运输漂移","source_units":[{"start":74755,"end":75127,"start_line":687,"end_line":690,"raw_tex":"For the last integral term, we use the triangular inequality\n\\[\n \\frac{B_{\\nu_0}}{t}  \\int_0^t \\Vert \\nu^\\star-\\nu^\\epsilon_s\\Vert_{\\BL}^* \\d s \\leq B_{\\nu_0}\\Vert \\nu^\\star - \\nu^\\epsilon_0\\Vert_{\\BL}^* + \\frac{B_{\\nu_0}}{t} \\int_0^t \\Vert \\nu_0^\\epsilon -\\nu^\\epsilon_s\\Vert_{\\BL}^*\\d s\\leq B_{\\nu_0} \\Vert \\nu^\\star - \\nu^\\epsilon_0\\Vert_{\\BL}^* +B_{\\nu_0}^2\\beta t\n\\]\n","sha256_utf8":"968dcd486a7122e8bf835b79bf989e4f52b56b5bf068ff876d382dcaa48ae2ff"}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","title":"利用单调性并优化比较测度","source_units":[{"start":75127,"end":75533,"start_line":691,"end_line":694,"raw_tex":"where the last term is obtained by bounding the integrated flow of the velocity field $(\\nabla J'_{\\nu_t})_{t\\geq 0}$. Since $\\Hh(\\nu_\\epsilon,\\nu_t)\\geq 0$ and $J(\\nu_s)$ is decreasing, it follows\n\\[\nJ(\\nu_t)-J^\\star  \\leq \\inf_{\\nu^\\epsilon\\in \\Mm_+(\\Theta)} \\left( \\frac1{4\\alpha t} \\Hh(\\nu^\\epsilon,\\nu_0) + B_{\\nu_0}\\Vert  \\nu^\\star -\\nu^\\epsilon\\Vert_{\\BL}^*\\right) + B_{\\nu_0}^2 \\beta t.\\qedhere\n\\]\n","sha256_utf8":"fc709065425361fb637d591a08a9c27e13acc83c6eb45ff1ba97be130af20dcd"}]},{"id":"S084","semantic_type":"lemma","kind":"lemma","title":"稀疏解的镜像速率上界","source_units":[{"start":124083,"end":124832,"start_line":1253,"end_line":1262,"raw_tex":"\\begin{lemma}\\label{lem:qdensity}\nUnder {\\sf (A1)}, there exists $C_{\\Theta}> 0$ that only depends on the curvature of $\\Theta$, such that for all $\\nu^\\star ,\\nu_0\\in \\Mm_+(\\Theta)$ where $\\nu^\\star=\\sum_{i=1}^{m^\\star} r_i^2\\delta_{\\theta_i}$ and $\\nu_0=\\rho \\vol$ where $\\log \\rho$ is $L$-Lipschitz, then \n\\[\n\\Qq_{\\nu^\\star,\\nu_0}(\\tau)\\leq \\frac1{\\tau} \\left( \\nu^\\star(\\Theta)\\cdot d\\cdot \\big(C_\\Theta +\\log(\\tau)+L/\\tau\\big) +\\nu_0(\\Theta)- \\nu^\\star(\\Theta)+ \\sum_{i=1}^{m^\\star} r_i^2 \\log\\left(\\frac{r_i^2}{\\rho(\\theta_i)}\\right) \\right).\n\\]\nMoreover, for any other $\\hat \\nu_0\\in \\Mm_+(\\Theta)$, it holds\n$\n\\Qq_{\\nu^\\star,\\hat \\nu_0}(\\tau)\\leq \\Qq_{\\nu^\\star,\\nu_0}(\\tau) + \\nu^\\star(\\Theta)\\cdot W_\\infty(\\nu_0,\\hat \\nu_0).\n$\n\\end{lemma}","sha256_utf8":"1710c6867e71cfbff0699661e2561819dda46c794c947d0c58c8747149360d13"}]},{"id":"S085","semantic_type":"proof","kind":"proof","title":"镜像速率上界完整证明","source_units":[{"start":125376,"end":129950,"start_line":1275,"end_line":1306,"raw_tex":"\\begin{proof}\nLet us build $\\nu_\\epsilon$ in such a way that the quantity defining $\\Qq_{\\nu^\\star,\\nu_0}(\\tau)$ in Eq.~\\eqref{eq:quantization} is small. For this, consider a radius $\\epsilon>0$ and consider the measure $\\nu_\\epsilon$ defined as the normalized volume measure on each geodesic ball of radius $\\tau$ around each $\\theta_i$, with mass $r_i^2$ on this ball, and vanishing everywhere else. Using the transport map that maps these balls to their centers $\\theta_i$, we get if $\\Theta$ is flat,\n\\[\n\\Vert \\nu_\\epsilon -\\nu^\\star\\Vert_\\BL^* \\leq W_1(\\nu_\\epsilon,\\nu^\\star)  \\leq \\sum_{i=1}^{m^\\star} \\frac{r_i^2}{V^{(d)}(\\epsilon)} \\int_0^\\epsilon s \\frac{d}{ds}V^{(d)}(s)\\d s\n\\]\nwhere $V^{(d)}(\\epsilon)$ is the volume of a ball of radius $\\epsilon$ in $\\RR^d$, that scales as $\\epsilon^d$. Using an integration by parts, it follows\n\\[\n\\frac{1}{V^{(d)}(\\epsilon)} \\int_0^\\epsilon s \\frac{d}{ds}V^{(d)}(s)\\d s = \\epsilon - \\int_0^\\epsilon \\frac{V^{(d)}(s)}{V^{(d)}(\\epsilon)}\\d s = \\epsilon -\\int_0^\\epsilon\\left(\\frac{s}{\\epsilon}\\right)^d\\d s = \\frac{\\epsilon d}{d+1},\n\\]\nthus $W_1(\\nu_\\epsilon ,\\nu^\\star)  \\leq \\nu^\\star(\\Theta)\\epsilon$. In the general case where $\\Theta$ is a potentially curved manifold, this upper bound also depends on the curvature of $\\Theta$ around each $\\theta_i$, a dependency that we hide in the multiplicative constant so $W_1(\\nu_\\epsilon ,\\nu^\\star)  \\leq C\\nu^\\star(\\Theta)\\epsilon$. Let us now control the entropy term. Writing $\\rho_\\epsilon =\\d\\nu_\\epsilon/\\d{\\vol}$ and $\\Theta_i$ for the geodesic ball of radius $\\epsilon$ around $\\theta_i$, it holds\n\\begin{align*}\n\\Hh(\\nu_\\epsilon,\\nu_0) & = \\nu_0(\\Theta) -\\nu^\\star(\\Theta) + \\sum_{i=1}^{m^\\star} \\int_{\\Theta_i} \\rho_\\epsilon(\\theta)\\log (\\rho_\\epsilon(\\theta)/\\rho(\\theta))\\d{\\vol}(\\theta).\n\\end{align*}\nThe integral term can be estimated as follows,\n\\begin{align*}\n\\int_{\\Theta_i} \\rho_\\epsilon(\\theta)  \\log (\\rho_\\epsilon(\\theta)/\\rho(\\theta))\\d{\\vol}(\\theta) &= \\int_{\\Theta_i} \\frac{r_i^2}{V^{(d)}(\\epsilon)} \\left(\\log (r_i^2)-\\log V^{(d)}(\\epsilon) -\\log(\\rho(\\theta))\\right)\\d{\\vol}(\\theta)\\\\\n& \\leq r_i^2\\left(\\log(r_i^2)-\\log V^{(d)}(\\epsilon) -\\log \\rho(\\theta_i) +\\Lip(\\log \\rho)\\cdot \\epsilon \\right).\n\\end{align*}\nRecalling that $-\\log V^{(d)}(\\epsilon) \\leq -d\\log(\\epsilon)+C$ for some $C$ that only depends on the curvature of $\\Theta$, we get that the right-hand side of~\\eqref{eq:quantization} is bounded by\n\\begin{align*}\n \\frac1{\\tau} \\left(C\\nu^\\star(\\Theta)+\\nu_0(\\Theta) -\\nu^\\star(\\Theta) d\\log(\\epsilon) + \\sum_{i=1}^{m^\\star} r_i^2 \\log\\left(\\frac{r_i^2}{\\rho(\\theta_i)}\\right) + \\epsilon L\\nu^\\star(\\Theta)\\right) + C\\nu^\\star(\\Theta)\\epsilon.\n\\end{align*}\nLet us fix $\\epsilon>0$ by minimizing $C\\nu^\\star(\\Theta)\\epsilon -\\nu^\\star(\\Theta)d\\log(\\epsilon)/\\tau$, which gives $\\epsilon = d/(C\\tau)$. The first claim follows by plugging this value for $\\epsilon$ in the expression above.\n\nFor the second claim of the statement, let us build a suitable candidate $\\hat \\nu_\\epsilon$ in order to upper bound the infimum that defines $\\Qq_{\\nu^\\star,\\hat \\nu_0}(\\tau)$. Let $T$ be an optimal transport map from $\\nu_0$ to $\\hat \\nu_0$ for $W_\\infty$, i.e.\\ a measurable map $T:\\Theta\\to\\Theta$ satisfying $T_\\# \\nu_0 = \\hat \\nu_0$ and $\\max\\{\\dist(\\theta,T(\\theta))\\;;\\; \\theta\\in \\spt \\nu_0(=\\Theta)\\} = W_\\infty(\\nu_0,\\hat \\nu_0)$ (see~\\cite[Sec. 3.2]{santambrogio2015optimal}, the absolute continuity of $\\nu_0$ is sufficient for such a map to exist). Now we define $\\hat \\nu_\\epsilon = T_\\# \\nu_\\epsilon$ where $\\nu_\\epsilon$ is such that $\\Hh(\\nu_\\epsilon,\\nu_0)<\\infty$. Since the relative entropy is non-increasing under pushforwards, it holds $\\Hh(\\hat \\nu_\\epsilon,\\hat \\nu_0)\\leq \\Hh(\\nu_\\epsilon,\\nu_0)$. Moreover, it holds $\\Vert \\nu_\\epsilon -\\hat \\nu_\\epsilon\\Vert_{\\BL}^*\\leq W_1(\\nu_\\epsilon,\\hat \\nu_\\epsilon)\\leq \\nu^\\star(\\Theta) W_\\infty(\\nu_\\epsilon,\\hat \\nu_\\epsilon)$. Thus we have\n\\begin{align*}\n\\Qq_{\\nu^\\star,\\hat \\nu_0}(\\tau) &\\leq \\Vert \\nu^\\star - \\hat \\nu_\\epsilon\\Vert_{\\BL}^* +\\frac1\\tau \\Hh(\\hat \\nu_\\epsilon,\\hat \\nu_0) \\\\\n&\\leq \\Vert \\nu_\\epsilon -\\hat \\nu_\\epsilon\\Vert_{\\BL}+ \\Vert \\nu^\\star - \\nu_\\epsilon\\Vert_{\\BL} +\\frac1\\tau \\Hh(\\nu_\\epsilon,\\nu_0)\\\\\n&\\leq \\nu^\\star(\\Theta) W_\\infty(\\nu_\\epsilon,\\hat \\nu_\\epsilon) + \\Vert \\nu^\\star - \\nu_\\epsilon\\Vert_{\\BL}^* +\\frac1\\tau \\Hh(\\nu_\\epsilon,\\nu_0) .\n\\end{align*}\nThe claim follows by noticing that,  by construction, $W_\\infty(\\nu_\\epsilon,\\hat \\nu_\\epsilon) \\leq W_\\infty(\\nu_0,\\hat \\nu_0)$ and then by taking the infimum in $\\nu_\\epsilon$.\n\\end{proof}","sha256_utf8":"4b2d7d9d1a738943ad091620a2404537f109ecce9e87f0ce2453dd139717dcf5"}]},{"id":"S086","semantic_type":"proof_step","kind":"proof_step","title":"以小球平滑原子并估计运输误差","source_units":[{"start":125390,"end":126977,"start_line":1276,"end_line":1284,"raw_tex":"Let us build $\\nu_\\epsilon$ in such a way that the quantity defining $\\Qq_{\\nu^\\star,\\nu_0}(\\tau)$ in Eq.~\\eqref{eq:quantization} is small. For this, consider a radius $\\epsilon>0$ and consider the measure $\\nu_\\epsilon$ defined as the normalized volume measure on each geodesic ball of radius $\\tau$ around each $\\theta_i$, with mass $r_i^2$ on this ball, and vanishing everywhere else. Using the transport map that maps these balls to their centers $\\theta_i$, we get if $\\Theta$ is flat,\n\\[\n\\Vert \\nu_\\epsilon -\\nu^\\star\\Vert_\\BL^* \\leq W_1(\\nu_\\epsilon,\\nu^\\star)  \\leq \\sum_{i=1}^{m^\\star} \\frac{r_i^2}{V^{(d)}(\\epsilon)} \\int_0^\\epsilon s \\frac{d}{ds}V^{(d)}(s)\\d s\n\\]\nwhere $V^{(d)}(\\epsilon)$ is the volume of a ball of radius $\\epsilon$ in $\\RR^d$, that scales as $\\epsilon^d$. Using an integration by parts, it follows\n\\[\n\\frac{1}{V^{(d)}(\\epsilon)} \\int_0^\\epsilon s \\frac{d}{ds}V^{(d)}(s)\\d s = \\epsilon - \\int_0^\\epsilon \\frac{V^{(d)}(s)}{V^{(d)}(\\epsilon)}\\d s = \\epsilon -\\int_0^\\epsilon\\left(\\frac{s}{\\epsilon}\\right)^d\\d s = \\frac{\\epsilon d}{d+1},\n\\]\nthus $W_1(\\nu_\\epsilon ,\\nu^\\star)  \\leq \\nu^\\star(\\Theta)\\epsilon$. In the general case where $\\Theta$ is a potentially curved manifold, this upper bound also depends on the curvature of $\\Theta$ around each $\\theta_i$, a dependency that we hide in the multiplicative constant so $W_1(\\nu_\\epsilon ,\\nu^\\star)  \\leq C\\nu^\\star(\\Theta)\\epsilon$. Let us now control the entropy term. Writing $\\rho_\\epsilon =\\d\\nu_\\epsilon/\\d{\\vol}$ and $\\Theta_i$ for the geodesic ball of radius $\\epsilon$ around $\\theta_i$, it holds\n","sha256_utf8":"d987e8b829548484a1669b0a20c65667f32ac00d2905b55817351c3178428e30"}]},{"id":"S087","semantic_type":"proof_step","kind":"proof_step","title":"估计平滑测度的相对熵","source_units":[{"start":126977,"end":127609,"start_line":1285,"end_line":1292,"raw_tex":"\\begin{align*}\n\\Hh(\\nu_\\epsilon,\\nu_0) & = \\nu_0(\\Theta) -\\nu^\\star(\\Theta) + \\sum_{i=1}^{m^\\star} \\int_{\\Theta_i} \\rho_\\epsilon(\\theta)\\log (\\rho_\\epsilon(\\theta)/\\rho(\\theta))\\d{\\vol}(\\theta).\n\\end{align*}\nThe integral term can be estimated as follows,\n\\begin{align*}\n\\int_{\\Theta_i} \\rho_\\epsilon(\\theta)  \\log (\\rho_\\epsilon(\\theta)/\\rho(\\theta))\\d{\\vol}(\\theta) &= \\int_{\\Theta_i} \\frac{r_i^2}{V^{(d)}(\\epsilon)} \\left(\\log (r_i^2)-\\log V^{(d)}(\\epsilon) -\\log(\\rho(\\theta))\\right)\\d{\\vol}(\\theta)\\\\\n& \\leq r_i^2\\left(\\log(r_i^2)-\\log V^{(d)}(\\epsilon) -\\log \\rho(\\theta_i) +\\Lip(\\log \\rho)\\cdot \\epsilon \\right).\n\\end{align*}\n","sha256_utf8":"7c76fff7c1e37a8666f51510a500a0637de0712d1709ce81a7d9e981a54d4053"}]},{"id":"S088","semantic_type":"proof_step","kind":"proof_step","title":"平衡球半径与熵代价","source_units":[{"start":127609,"end":128296,"start_line":1293,"end_line":1297,"raw_tex":"Recalling that $-\\log V^{(d)}(\\epsilon) \\leq -d\\log(\\epsilon)+C$ for some $C$ that only depends on the curvature of $\\Theta$, we get that the right-hand side of~\\eqref{eq:quantization} is bounded by\n\\begin{align*}\n \\frac1{\\tau} \\left(C\\nu^\\star(\\Theta)+\\nu_0(\\Theta) -\\nu^\\star(\\Theta) d\\log(\\epsilon) + \\sum_{i=1}^{m^\\star} r_i^2 \\log\\left(\\frac{r_i^2}{\\rho(\\theta_i)}\\right) + \\epsilon L\\nu^\\star(\\Theta)\\right) + C\\nu^\\star(\\Theta)\\epsilon.\n\\end{align*}\nLet us fix $\\epsilon>0$ by minimizing $C\\nu^\\star(\\Theta)\\epsilon -\\nu^\\star(\\Theta)d\\log(\\epsilon)/\\tau$, which gives $\\epsilon = d/(C\\tau)$. The first claim follows by plugging this value for $\\epsilon$ in the expression above.\n","sha256_utf8":"7341e822aafb4a9cf3971fb5dc1857703873227553f603ca5263d94c5395425b"}]},{"id":"S089","semantic_type":"proof_step","kind":"proof_step","title":"通过运输推送转移初始化界","source_units":[{"start":128297,"end":129939,"start_line":1299,"end_line":1305,"raw_tex":"For the second claim of the statement, let us build a suitable candidate $\\hat \\nu_\\epsilon$ in order to upper bound the infimum that defines $\\Qq_{\\nu^\\star,\\hat \\nu_0}(\\tau)$. Let $T$ be an optimal transport map from $\\nu_0$ to $\\hat \\nu_0$ for $W_\\infty$, i.e.\\ a measurable map $T:\\Theta\\to\\Theta$ satisfying $T_\\# \\nu_0 = \\hat \\nu_0$ and $\\max\\{\\dist(\\theta,T(\\theta))\\;;\\; \\theta\\in \\spt \\nu_0(=\\Theta)\\} = W_\\infty(\\nu_0,\\hat \\nu_0)$ (see~\\cite[Sec. 3.2]{santambrogio2015optimal}, the absolute continuity of $\\nu_0$ is sufficient for such a map to exist). Now we define $\\hat \\nu_\\epsilon = T_\\# \\nu_\\epsilon$ where $\\nu_\\epsilon$ is such that $\\Hh(\\nu_\\epsilon,\\nu_0)<\\infty$. Since the relative entropy is non-increasing under pushforwards, it holds $\\Hh(\\hat \\nu_\\epsilon,\\hat \\nu_0)\\leq \\Hh(\\nu_\\epsilon,\\nu_0)$. Moreover, it holds $\\Vert \\nu_\\epsilon -\\hat \\nu_\\epsilon\\Vert_{\\BL}^*\\leq W_1(\\nu_\\epsilon,\\hat \\nu_\\epsilon)\\leq \\nu^\\star(\\Theta) W_\\infty(\\nu_\\epsilon,\\hat \\nu_\\epsilon)$. Thus we have\n\\begin{align*}\n\\Qq_{\\nu^\\star,\\hat \\nu_0}(\\tau) &\\leq \\Vert \\nu^\\star - \\hat \\nu_\\epsilon\\Vert_{\\BL}^* +\\frac1\\tau \\Hh(\\hat \\nu_\\epsilon,\\hat \\nu_0) \\\\\n&\\leq \\Vert \\nu_\\epsilon -\\hat \\nu_\\epsilon\\Vert_{\\BL}+ \\Vert \\nu^\\star - \\nu_\\epsilon\\Vert_{\\BL} +\\frac1\\tau \\Hh(\\nu_\\epsilon,\\nu_0)\\\\\n&\\leq \\nu^\\star(\\Theta) W_\\infty(\\nu_\\epsilon,\\hat \\nu_\\epsilon) + \\Vert \\nu^\\star - \\nu_\\epsilon\\Vert_{\\BL}^* +\\frac1\\tau \\Hh(\\nu_\\epsilon,\\nu_0) .\n\\end{align*}\nThe claim follows by noticing that,  by construction, $W_\\infty(\\nu_\\epsilon,\\hat \\nu_\\epsilon) \\leq W_\\infty(\\nu_0,\\hat \\nu_0)$ and then by taking the infimum in $\\nu_\\epsilon$.\n","sha256_utf8":"29def377c66d32085fbb2e3167ebeff8603acd634a7b287afea4043756425646"}]},{"id":"S090","semantic_type":"external_fact","kind":"external_fact","title":"光滑特征流与熵微分","source_units":[],"declared_external":true,"declaration_text":["Smooth time-dependent vector fields on a compact manifold generate diffeomorphic characteristic flows. Relative entropy is invariant under a common diffeomorphic pushforward; combining this invariance with the multiplicative reaction term gives the entropy derivative used at lines 677–682."]},{"id":"S091","semantic_type":"external_fact","kind":"external_fact","title":"有限测度的弱紧性","source_units":[],"declared_external":true,"declaration_text":["A total-variation-bounded family of nonnegative finite measures on a compact space is weakly precompact. Integration of a continuous Hilbert-valued feature map is continuous on such bounded families."]},{"id":"S092","semantic_type":"external_fact","kind":"external_fact","title":"标准范数与谱估计","source_units":[],"declared_external":true,"declaration_text":["Cauchy–Schwarz and Hölder inequalities, the bias–variance identity, triangle inequalities, and the smallest- and largest-eigenvalue bounds for positive definite quadratic forms."]},{"id":"S093","semantic_type":"external_fact","kind":"external_fact","title":"非归一化距离的提升表征","source_units":[],"declared_external":true,"declaration_text":["The Wasserstein–Fisher–Rao distance equals the minimum cone-Wasserstein distance over probability lifts with the prescribed homogeneous projections (Liero et al., Theorem 7.20). Its bounded-Lipschitz comparison is the inequality cited from their Proposition 7.18 at line 1097."]},{"id":"S094","semantic_type":"external_fact","kind":"external_fact","title":"最优运输映射与熵收缩","source_units":[],"declared_external":true,"declaration_text":["For the absolutely continuous source measure in the author argument, use the optimal W_infinity transport map cited from Santambrogio, Section 3.2. Relative entropy does not increase under a common measurable pushforward, and bounded-Lipschitz discrepancy is bounded by the corresponding transport cost."]},{"id":"S095","semantic_type":"external_fact","kind":"external_fact","title":"小测地球的体积尺度","source_units":[],"declared_external":true,"declaration_text":["For sufficiently small radii, the volume of a geodesic ball on the smooth compact d-dimensional manifold is comparable to the radius raised to d, with constants controlled by the manifold geometry; the corresponding local transport estimates differ from Euclidean ones by controlled constants."]},{"id":"S096","semantic_type":"external_fact","kind":"external_fact","title":"格朗沃尔指数衰减","source_units":[],"declared_external":true,"declaration_text":["If a nonnegative absolutely continuous function satisfies h'(t) ≤ -c h(t) almost everywhere for c > 0, then h(t) ≤ h(0) exp(-ct)."]},{"id":"S097","semantic_type":"external_fact","kind":"external_fact","title":"对数的幂函数上界","source_units":[],"declared_external":true,"declaration_text":["For every epsilon > 0 there is C_epsilon > 0 such that log(u) ≤ C_epsilon u^epsilon for all u ≥ 1."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S006","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S007","target_id":"S011","kind":"supports"},{"source_id":"S007","target_id":"S035","kind":"supports"},{"source_id":"S008","target_id":"S035","kind":"supports"},{"source_id":"S008","target_id":"S044","kind":"supports"},{"source_id":"S008","target_id":"S080","kind":"supports"},{"source_id":"S009","target_id":"S080","kind":"supports"},{"source_id":"S009","target_id":"S082","kind":"supports"},{"source_id":"S009","target_id":"S086","kind":"supports"},{"source_id":"S009","target_id":"S087","kind":"supports"},{"source_id":"S009","target_id":"S089","kind":"supports"},{"source_id":"S010","target_id":"S013","kind":"supports"},{"source_id":"S010","target_id":"S014","kind":"supports"},{"source_id":"S011","target_id":"S016","kind":"supports"},{"source_id":"S011","target_id":"S035","kind":"supports"},{"source_id":"S011","target_id":"S058","kind":"supports"},{"source_id":"S011","target_id":"S064","kind":"supports"},{"source_id":"S011","target_id":"S077","kind":"supports"},{"source_id":"S012","target_id":"S016","kind":"supports"},{"source_id":"S013","target_id":"S032","kind":"supports"},{"source_id":"S014","target_id":"S032","kind":"supports"},{"source_id":"S015","target_id":"S032","kind":"supports"},{"source_id":"S016","target_id":"S032","kind":"supports"},{"source_id":"S016","target_id":"S039","kind":"supports"},{"source_id":"S017","target_id":"S035","kind":"supports"},{"source_id":"S017","target_id":"S048","kind":"supports"},{"source_id":"S017","target_id":"S080","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S038","kind":"supports"},{"source_id":"S018","target_id":"S044","kind":"supports"},{"source_id":"S018","target_id":"S086","kind":"supports"},{"source_id":"S019","target_id":"S049","kind":"supports"},{"source_id":"S019","target_id":"S051","kind":"supports"},{"source_id":"S019","target_id":"S059","kind":"supports"},{"source_id":"S019","target_id":"S065","kind":"supports"},{"source_id":"S019","target_id":"S067","kind":"supports"},{"source_id":"S020","target_id":"S048","kind":"supports"},{"source_id":"S020","target_id":"S049","kind":"supports"},{"source_id":"S020","target_id":"S050","kind":"supports"},{"source_id":"S020","target_id":"S051","kind":"supports"},{"source_id":"S020","target_id":"S073","kind":"supports"},{"source_id":"S021","target_id":"S041","kind":"supports"},{"source_id":"S021","target_id":"S055","kind":"supports"},{"source_id":"S022","target_id":"S064","kind":"supports"},{"source_id":"S022","target_id":"S074","kind":"supports"},{"source_id":"S022","target_id":"S077","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S044","kind":"supports"},{"source_id":"S023","target_id":"S049","kind":"supports"},{"source_id":"S024","target_id":"S028","kind":"supports"},{"source_id":"S024","target_id":"S040","kind":"supports"},{"source_id":"S024","target_id":"S045","kind":"supports"},{"source_id":"S024","target_id":"S065","kind":"supports"},{"source_id":"S025","target_id":"S003","kind":"supports"},{"source_id":"S025","target_id":"S088","kind":"supports"},{"source_id":"S026","target_id":"S083","kind":"supports"},{"source_id":"S026","target_id":"S088","kind":"supports"},{"source_id":"S026","target_id":"S089","kind":"supports"},{"source_id":"S027","target_id":"S044","kind":"supports"},{"source_id":"S027","target_id":"S064","kind":"supports"},{"source_id":"S027","target_id":"S074","kind":"supports"},{"source_id":"S028","target_id":"S040","kind":"supports"},{"source_id":"S028","target_id":"S051","kind":"supports"},{"source_id":"S028","target_id":"S068","kind":"supports"},{"source_id":"S029","target_id":"S050","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S077","kind":"supports"},{"source_id":"S030","target_id":"S080","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S048","kind":"supports"},{"source_id":"S033","target_id":"S049","kind":"supports"},{"source_id":"S033","target_id":"S050","kind":"supports"},{"source_id":"S033","target_id":"S058","kind":"supports"},{"source_id":"S033","target_id":"S059","kind":"supports"},{"source_id":"S033","target_id":"S065","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S037","kind":"supports"},{"source_id":"S040","target_id":"S037","kind":"supports"},{"source_id":"S041","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S054","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S051","kind":"supports"},{"source_id":"S042","target_id":"S059","kind":"supports"},{"source_id":"S042","target_id":"S063","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S048","target_id":"S051","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S049","target_id":"S047","kind":"supports"},{"source_id":"S050","target_id":"S047","kind":"supports"},{"source_id":"S051","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S054","kind":"supports"},{"source_id":"S046","target_id":"S069","kind":"supports"},{"source_id":"S046","target_id":"S073","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S055","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S002","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S059","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S072","kind":"supports"},{"source_id":"S063","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S063","target_id":"S062","kind":"supports"},{"source_id":"S064","target_id":"S062","kind":"supports"},{"source_id":"S065","target_id":"S062","kind":"supports"},{"source_id":"S066","target_id":"S062","kind":"supports"},{"source_id":"S067","target_id":"S062","kind":"supports"},{"source_id":"S068","target_id":"S062","kind":"supports"},{"source_id":"S069","target_id":"S062","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S072","target_id":"S071","kind":"supports"},{"source_id":"S073","target_id":"S071","kind":"supports"},{"source_id":"S074","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S070","kind":"supports"},{"source_id":"S070","target_id":"S077","kind":"supports"},{"source_id":"S077","target_id":"S076","kind":"supports"},{"source_id":"S076","target_id":"S075","kind":"supports"},{"source_id":"S075","target_id":"S006","kind":"supports"},{"source_id":"S080","target_id":"S081","kind":"supports"},{"source_id":"S081","target_id":"S083","kind":"supports"},{"source_id":"S082","target_id":"S083","kind":"supports"},{"source_id":"S080","target_id":"S079","kind":"supports"},{"source_id":"S081","target_id":"S079","kind":"supports"},{"source_id":"S082","target_id":"S079","kind":"supports"},{"source_id":"S083","target_id":"S079","kind":"supports"},{"source_id":"S079","target_id":"S078","kind":"supports"},{"source_id":"S078","target_id":"S004","kind":"supports"},{"source_id":"S086","target_id":"S087","kind":"supports"},{"source_id":"S086","target_id":"S088","kind":"supports"},{"source_id":"S087","target_id":"S088","kind":"supports"},{"source_id":"S088","target_id":"S089","kind":"supports"},{"source_id":"S086","target_id":"S085","kind":"supports"},{"source_id":"S087","target_id":"S085","kind":"supports"},{"source_id":"S088","target_id":"S085","kind":"supports"},{"source_id":"S089","target_id":"S085","kind":"supports"},{"source_id":"S085","target_id":"S084","kind":"supports"},{"source_id":"S084","target_id":"S003","kind":"supports"},{"source_id":"S060","target_id":"S035","kind":"supports"},{"source_id":"S060","target_id":"S044","kind":"supports"},{"source_id":"S060","target_id":"S048","kind":"supports"},{"source_id":"S060","target_id":"S049","kind":"supports"},{"source_id":"S060","target_id":"S058","kind":"supports"},{"source_id":"S060","target_id":"S064","kind":"supports"},{"source_id":"S060","target_id":"S080","kind":"supports"},{"source_id":"S090","target_id":"S080","kind":"supports"},{"source_id":"S090","target_id":"S082","kind":"supports"},{"source_id":"S091","target_id":"S035","kind":"supports"},{"source_id":"S092","target_id":"S040","kind":"supports"},{"source_id":"S092","target_id":"S045","kind":"supports"},{"source_id":"S092","target_id":"S051","kind":"supports"},{"source_id":"S092","target_id":"S068","kind":"supports"},{"source_id":"S092","target_id":"S073","kind":"supports"},{"source_id":"S093","target_id":"S038","kind":"supports"},{"source_id":"S093","target_id":"S041","kind":"supports"},{"source_id":"S093","target_id":"S055","kind":"supports"},{"source_id":"S094","target_id":"S089","kind":"supports"},{"source_id":"S095","target_id":"S086","kind":"supports"},{"source_id":"S095","target_id":"S087","kind":"supports"},{"source_id":"S095","target_id":"S088","kind":"supports"},{"source_id":"S096","target_id":"S077","kind":"supports"},{"source_id":"S097","target_id":"S006","kind":"supports"}]}}
{"record_id":"1910.09373v1","paper":{"title":"A Stochastic Extra-Step Quasi-Newton Method for Nonsmooth Nonconvex Optimization","authors":["Minghan Yang","Andre Milzarek","Zaiwen Wen","Tong Zhang"],"arxiv_id":"1910.09373","version":"v1","journal":"Mathematical Programming","doi":"10.1007/s10107-021-01629-y","abstract_url":"https://arxiv.org/abs/1910.09373v1","pdf_url":"https://arxiv.org/pdf/1910.09373v1"},"context":{"raw_tex":"\\be \\label{eq:prob} \\min_{x \\in \\Rn}~\\psi(x) := f(x) + \\vp(x),\\ee\n\nwhere $f : \\Rn \\to \\R$ is a continuously differentiable but not necessarily convex function and $\\vp : \\Rn \\to \\Rex$ is a convex, lower semicontinuous, and proper mapping. We assume that a full evaluation of the function $f$ and its gradient $\\nabla f$ is either too expensive or not  possible and that approximate gradient information can only be accessed via calling \\textit{stochastic oracles}. %which give access to stochastic approximations of gradient $\\nabla f$ and propose a general stochastic two-step framework. \n\n\nBy $\\iprod{\\cdot}{\\cdot}$ and $\\|\\cdot\\| := \\|\\cdot\\|_2$ we denote the standard Euclidean inner product and norm. For matrices, the norm $\\|\\cdot\\|$ is the standard spectral norm. The set of symmetric and positive definite $n \\times n$ matrices is denoted by $\\Spp$. For a given matrix $\\Lambda \\in \\Spp$, we define the inner product $\\iprod{x}{y}_\\Lambda := \\iprod{x}{\\Lambda y} = \\iprod{\\Lambda x}{y}$ and $\\|x\\|_\\Lambda := \\sqrt{\\iprod{x}{x}_\\Lambda}$.\n%$|S|$ represents the cardinality of a finite set $S$. If $S$ is a subset of $\\{1,2,\\dots,n\\},$ then for any vector $x\\in \\R^n$ and matrix $H\\in \\R^{n\\times n}$, $x_{S}$ is the subvector of $x$ whose entries lie in $x$ indexed by $S,$ and $H_{SS}$ represents the submatrix of $H$ whose rows and columns lie in $S$. \nFor any $n \\in \\N$, we set $[n] := \\{1,...,n\\}$ and $[n]_0 = \\{0\\} \\cup [n]$. Let $(\\Omega,\\mathcal F,\\Prob)$ be a probability space. We will use uppercase letters and a sans-serif letterform to describe random variables ${\\sf X} : \\Omega \\to \\Rn$, while lowercase letters or letters with serifs are typically reserved for realizations of a random variable, $x = {\\sf X}(\\omega)$, or deterministic parameters. We use $L^p(\\Omega) := L^p(\\Omega,\\Prob)$, $p \\in [1,\\infty]$, to denote the standard $L^p$ space on $\\Omega$. We write ${\\sf X} \\in \\mathcal F$ for ``${\\sf X}$ is $\\mathcal F$-measurable''. Moreover, $\\sigma({\\sf X}^1,...,{\\sf X}^k)$ denotes the $\\sigma$-algebra generated by the family of random variables ${\\sf X}^1,...,{\\sf X}^k$. For a random variable ${\\sf X} \\in L^1(\\Omega)$ and a sub-$\\sigma$-algebra $\\cH \\subseteq \\mathcal F$, the conditional expectation of ${\\sf X}$ given $\\cH$ is denoted by $\\Exp[{\\sf X} \\mid \\cH]$. We use the abbreviations ``a.e.'' and ``a.s.'' for ``almost everywhere'' and ``almost surely'', respectively.\n\n\n\\subsection{Preliminaries and First Order Optimality} It is well-known that the stationarity conditions of problem \\eqref{eq:prob} can be equivalently rewritten as a proximal fixed-point equation, i.e., a point $x \\in \\dom~\\vp$ is a critical point of \\eqref{eq:prob} if and only if\n%\n\\be \\label{eq:opt} F^\\Lambda(x) := x - \\proxt{\\Lambda}{\\vp}(x-\\Lambda^{-1}\\nabla f(x)) = 0, \\quad \\Lambda \\in \\Spp, \\ee\n\nwhere $\\proxt{\\Lambda}{\\vp} : \\Rn \\to \\Rn$, $\\proxt{\\Lambda}{\\vp}(x) := \\argmin_y \\vp(y) + \\frac{1}{2}\\|x-y\\|^2_\\Lambda$ denotes the proximity operator of the mapping $\\vp$. The proximity operator is a $\\Lambda$-firmly nonexpansive function, i.e., it is globally Lipschitz continuous and satisfies\n%\n\\be \\label{eq:prox-nonexp} \\|\\proxt{\\Lambda}{\\vp}(x) - \\proxt{\\Lambda}{\\vp}(y)\\|_\\Lambda^2 \\leq \\iprod{\\Lambda(x-y)}{\\proxt{\\Lambda}{\\vp}(x) - \\proxt{\\Lambda}{\\vp}(y)}, \\quad \\forall~x,y. \\ee\n\nThe proximity operator $\\proxt{\\Lambda}{\\vp}$ can also be alternatively characterized via the underlying optimality condition\n%\n\\be \\label{eq:prox-opt} \\proxt{\\Lambda}{\\vp}(x) \\in x - \\Lambda^{-1} \\partial \\vp(\\proxt{\\Lambda}{\\vp}(x)), \\ee\n\nGiven a generic (stochastic) oracle represented by a direction $v \\in \\Rn$, we consider inexact variants of the nonsmooth residual \\eqref{eq:opt} of the type\n%\n\\begin{equation}\n\\label{sto_app_optimal}  F^\\Lambda_v(x) := x - \\proxt{\\Lambda}{\\vp}(x- \\Lambda^{-1} v). \n\\end{equation}\n\nWe will also use $u^\\Lambda_v(x) := x - \\Lambda^{-1} v$ and $p^\\Lambda_v := \\proxt{\\Lambda}{\\vp}(u^\\Lambda_v(x))$ to denote the inexact (proximal) gradient steps. In the case $v \\equiv \\nabla f(x)$, the terms $u^\\Lambda$ and $p^\\Lambda$ are used to denote the exact or full gradient and proximal gradient step.\n\n\n\\begin{algorithm2e}[t]\n\\caption{\\textbf{S}tochastic \\textbf{E}xtra-Step \\textbf{Q}uasi-\\textbf{N}ewton Method}    \n\\label{alg:seqn}\n\\lnlset{alg:seqn-1}{1}{Initialization: ~~Select the parameter matrices $(\\Lambda_k)_k, (\\Lambda_{k,+})_k \\in \\Spp$. Choose the step sizes $(\\alpha_k)_k$, $(\\beta_k)_k$ and the initial point  $x^0 \\in \\dom~\\vp$ .} \\\\ \n\\For{$k = 0,1,... $}{\n\\lnlset{alg:seqn-2}{2}{Based on $v^k \\approx \\nabla f(x^k)$, compute a direction $d^k$ and set $z^k = x^k + \\beta_k d^k$.} \\\\\n\\lnlset{alg:seqn-3}{3}{Calculate the new oracle $v^k_+ \\approx \\nabla f(z^k)$ and perform the update \\\\\n\\begin{center} $x^{k+1} = \\proxt{\\Lambda_{k,+}}{\\vp}(x^k + \\alpha_k d^k - \\Lambda_{k,+}^{-1}v^k_+)$ \\vspace{-1ex}.\\end{center}}  \n} \n\\end{algorithm2e}\n\n\\begin{assumption} \\label{ass:lip} Let $f : \\Rn \\to \\R$ be given. We assume: %\\vspace{.5ex}\n\\setlength{\\leftmargini}{7ex}\n\\begin{itemize}\n\\item[{\\rm(A.1)}] The gradient mapping $\\nabla f$ is Lipschitzian on $\\Rn$ with modulus $L_f \\geq 1$. %Moreover, each of the component functions $\\nabla f_i$, $i \\in [N]$, is $L_i$-Lipschitz continuous.\\vspace{.5ex} \n\\item[{\\rm(A.2)}] The objective function $\\psi$ is bounded from below on $\\dom~\\vp$. \\vspace{.5ex}\n%\\item[{\\rm(A.3)}] There exist parameters $\\mu_f \\in \\R$ and $\\mu_\\vp \\geq 0$ with $\\bar\\mu := \\mu_f + \\mu_\\vp > 0$ such that the shifted functions $f - \\frac{\\mu_f}{2}\\|\\cdot\\|^2$ and $\\vp - \\frac{\\mu_\\vp}{2}\\|\\cdot\\|^2$ are convex. \\vspace{.5ex}\n%\\item[{\\rm(A.4)}] There exists a constant $\\bar g_r > 0$ such that for all $x \\in \\dom~r$ there exists $\\lambda \\in \\partial r(x)$ with $\\|\\lambda\\| \\leq \\bar g_r$.\n\\end{itemize}\n\\end{assumption}\n\n\\be \\label{eq:lip-ineq} f(y) \\leq f(x) + \\iprod{\\nabla f(x)}{y-x} + \\frac{L_f}{2}  \\|y-x\\|^2, \\quad \\forall~x,y \\in \\Rn, \\ee\n\nwhich is a direct consequence of assumption {\\rm(A.1)}.\n\n\nThe stochasticity in Algorithm \\ref{alg:seqn} mainly results from the random selection of the oracles $v^k$ and $v^k_+$ in step \\ref{alg:seqn-2} and \\ref{alg:seqn-3}. In this work, we assume that the stochastic approximations $v^k$ and $v^k_+$ correspond to realizations of the random vectors ${\\sf V}^k : \\Omega \\to \\Rn$ and ${\\sf V}^k_+ : \\Omega \\to \\Rn$. Furthermore, we suppose that the underlying probability space $(\\Omega,\\mathcal F,\\Prob)$ is sufficiently rich allowing us to model the involved random processes in a unified way. We now define the filtrations\n%\n\\[ \\mathcal F^k := \\sigma({\\sf V}^0, {\\sf V}^0_+, ..., {\\sf V}^k), \\quad \\mathcal F^k_+ := \\sigma(\\mathcal F^k \\cup \\sigma({\\sf V}^k_+)). \\]\n%\nBy convention, let $({\\sf D}^k)_k$ denote the stochastic process associated with the directions $(d^k)_k$ chosen in step \\ref{alg:seqn-2}. We will work with the following stochastic assumptions. \n\n\n\\begin{assumption} \\label{ass:stoch-1} We assume: \n\\setlength{\\leftmargini}{7ex}\n\\begin{itemize}\n\\item[{\\rm(B.1)}] The mapping ${\\sf D}^k : \\Omega \\to \\R^{n}$ is an $\\mathcal F^k$-measurable function for all $k$. \\vspace{.5ex} \n\\item[{\\rm(B.2)}] There is $\\nu_k > 0$ such that we have $\\Exp[\\|{\\sf D}^k\\|^2 \\mid \\mathcal F^{k-1}_+] \\leq \\nu_k^2 \\cdot \\Exp[\\|F^{\\Lambda_k}_{{\\sf V}^k}({\\sf X}^k)\\|^2 \\mid \\mathcal F^{k-1}_+]$ \\aev and for all $k \\in \\N$.\n\\item[{\\rm(B.3)}] For all $k \\in \\N$, it holds $\\Exp[{\\sf V}^k \\mid \\mathcal F^{k-1}_+] = \\nabla f({\\sf X}^k)$, $\\Exp[{\\sf V}^k_+ \\mid \\mathcal F^{k}] = \\nabla f({\\sf Z}^k)$ \\aev and there exists $\\sigma_k, \\sigma_{k,+} > 0$ such that \\aev \n%\\item[{\\rm(A.3)}] There exist parameters $\\mu_f \\in \\R$ and $\\mu_\\vp \\geq 0$ with $\\bar\\mu := \\mu_f + \\mu_\\vp > 0$ such that the shifted functions $f - \\frac{\\mu_f}{2}\\|\\cdot\\|^2$ and $\\vp - \\frac{\\mu_\\vp}{2}\\|\\cdot\\|^2$ are convex. \\vspace{.5ex}\n%\\item[{\\rm(A.4)}] There exists a constant $\\bar g_r > 0$ such that for all $x \\in \\dom~r$ there exists $\\lambda \\in \\partial r(x)$ with $\\|\\lambda\\| \\leq \\bar g_r$.\n\\end{itemize}\n%\n\\[ \\Exp[\\|\\nabla f({\\sf X}^k) - {\\sf V}^k\\|^2 \\mid \\mathcal F_+^{k-1}] \\leq \\sigma_k^2 \\quad \\text{and} \\quad \\Exp[\\|\\nabla f({\\sf Z}^k) - {\\sf V}_+^k\\|^2 \\mid \\mathcal F^{k}] \\leq \\sigma_{k,+}^2. \\]\n\\end{assumption}\n\n\\be \\label{eq:prop-stoch} {\\sf Z}^k \\in \\mathcal F^k \\quad \\text{and} \\quad {\\sf X}^{k+1} \\in {\\mathcal F}^{k}_+ , \\quad \\forall~k \\geq 0. \\ee","units":[{"start":9578,"end":9643,"start_line":258,"end_line":258,"raw_tex":"\\be \\label{eq:prob} \\min_{x \\in \\Rn}~\\psi(x) := f(x) + \\vp(x),\\ee","sha256_utf8":"19a045ec8effdba1798dfd790cf78fffabf21c1ff436557670e77e05d12d40bc"},{"start":9646,"end":10169,"start_line":260,"end_line":260,"raw_tex":"where $f : \\Rn \\to \\R$ is a continuously differentiable but not necessarily convex function and $\\vp : \\Rn \\to \\Rex$ is a convex, lower semicontinuous, and proper mapping. We assume that a full evaluation of the function $f$ and its gradient $\\nabla f$ is either too expensive or not  possible and that approximate gradient information can only be accessed via calling \\textit{stochastic oracles}. %which give access to stochastic approximations of gradient $\\nabla f$ and propose a general stochastic two-step framework. \n","sha256_utf8":"b72e2c6fd95d167abbce02cb9aa463438f7edc608b547bcd40a775edce4d1512"},{"start":30383,"end":32206,"start_line":378,"end_line":380,"raw_tex":"By $\\iprod{\\cdot}{\\cdot}$ and $\\|\\cdot\\| := \\|\\cdot\\|_2$ we denote the standard Euclidean inner product and norm. For matrices, the norm $\\|\\cdot\\|$ is the standard spectral norm. The set of symmetric and positive definite $n \\times n$ matrices is denoted by $\\Spp$. For a given matrix $\\Lambda \\in \\Spp$, we define the inner product $\\iprod{x}{y}_\\Lambda := \\iprod{x}{\\Lambda y} = \\iprod{\\Lambda x}{y}$ and $\\|x\\|_\\Lambda := \\sqrt{\\iprod{x}{x}_\\Lambda}$.\n%$|S|$ represents the cardinality of a finite set $S$. If $S$ is a subset of $\\{1,2,\\dots,n\\},$ then for any vector $x\\in \\R^n$ and matrix $H\\in \\R^{n\\times n}$, $x_{S}$ is the subvector of $x$ whose entries lie in $x$ indexed by $S,$ and $H_{SS}$ represents the submatrix of $H$ whose rows and columns lie in $S$. \nFor any $n \\in \\N$, we set $[n] := \\{1,...,n\\}$ and $[n]_0 = \\{0\\} \\cup [n]$. Let $(\\Omega,\\mathcal F,\\Prob)$ be a probability space. We will use uppercase letters and a sans-serif letterform to describe random variables ${\\sf X} : \\Omega \\to \\Rn$, while lowercase letters or letters with serifs are typically reserved for realizations of a random variable, $x = {\\sf X}(\\omega)$, or deterministic parameters. We use $L^p(\\Omega) := L^p(\\Omega,\\Prob)$, $p \\in [1,\\infty]$, to denote the standard $L^p$ space on $\\Omega$. We write ${\\sf X} \\in \\mathcal F$ for ``${\\sf X}$ is $\\mathcal F$-measurable''. Moreover, $\\sigma({\\sf X}^1,...,{\\sf X}^k)$ denotes the $\\sigma$-algebra generated by the family of random variables ${\\sf X}^1,...,{\\sf X}^k$. For a random variable ${\\sf X} \\in L^1(\\Omega)$ and a sub-$\\sigma$-algebra $\\cH \\subseteq \\mathcal F$, the conditional expectation of ${\\sf X}$ given $\\cH$ is denoted by $\\Exp[{\\sf X} \\mid \\cH]$. We use the abbreviations ``a.e.'' and ``a.s.'' for ``almost everywhere'' and ``almost surely'', respectively.\n","sha256_utf8":"d21f80d098d7206e9937a665efe6c6a8275a75bb377392e5de7f84f2a3c60fca"},{"start":34910,"end":35313,"start_line":401,"end_line":403,"raw_tex":"\\subsection{Preliminaries and First Order Optimality} It is well-known that the stationarity conditions of problem \\eqref{eq:prob} can be equivalently rewritten as a proximal fixed-point equation, i.e., a point $x \\in \\dom~\\vp$ is a critical point of \\eqref{eq:prob} if and only if\n%\n\\be \\label{eq:opt} F^\\Lambda(x) := x - \\proxt{\\Lambda}{\\vp}(x-\\Lambda^{-1}\\nabla f(x)) = 0, \\quad \\Lambda \\in \\Spp, \\ee","sha256_utf8":"547d16a181e6ecfced30993aa0d1e54494dafd504e1b5249913214ed23fa70ab"},{"start":35316,"end":35807,"start_line":405,"end_line":407,"raw_tex":"where $\\proxt{\\Lambda}{\\vp} : \\Rn \\to \\Rn$, $\\proxt{\\Lambda}{\\vp}(x) := \\argmin_y \\vp(y) + \\frac{1}{2}\\|x-y\\|^2_\\Lambda$ denotes the proximity operator of the mapping $\\vp$. The proximity operator is a $\\Lambda$-firmly nonexpansive function, i.e., it is globally Lipschitz continuous and satisfies\n%\n\\be \\label{eq:prox-nonexp} \\|\\proxt{\\Lambda}{\\vp}(x) - \\proxt{\\Lambda}{\\vp}(y)\\|_\\Lambda^2 \\leq \\iprod{\\Lambda(x-y)}{\\proxt{\\Lambda}{\\vp}(x) - \\proxt{\\Lambda}{\\vp}(y)}, \\quad \\forall~x,y. \\ee","sha256_utf8":"54bde0c5b0ba292196838fced75c268e867e33eac090cfe0c2d6a6c1d8124470"},{"start":35810,"end":36049,"start_line":409,"end_line":411,"raw_tex":"The proximity operator $\\proxt{\\Lambda}{\\vp}$ can also be alternatively characterized via the underlying optimality condition\n%\n\\be \\label{eq:prox-opt} \\proxt{\\Lambda}{\\vp}(x) \\in x - \\Lambda^{-1} \\partial \\vp(\\proxt{\\Lambda}{\\vp}(x)), \\ee","sha256_utf8":"55b17187727e95ad46aae0ba579246476bfd51ea3cae4398ed912345d2ed87ec"},{"start":37490,"end":37770,"start_line":426,"end_line":430,"raw_tex":"Given a generic (stochastic) oracle represented by a direction $v \\in \\Rn$, we consider inexact variants of the nonsmooth residual \\eqref{eq:opt} of the type\n%\n\\begin{equation}\n\\label{sto_app_optimal}  F^\\Lambda_v(x) := x - \\proxt{\\Lambda}{\\vp}(x- \\Lambda^{-1} v). \n\\end{equation}","sha256_utf8":"4ccb9346a913b1df22ae5c2d26299656c5f20be7a61e83b8933a66cff6323aa2"},{"start":37773,"end":38084,"start_line":432,"end_line":432,"raw_tex":"We will also use $u^\\Lambda_v(x) := x - \\Lambda^{-1} v$ and $p^\\Lambda_v := \\proxt{\\Lambda}{\\vp}(u^\\Lambda_v(x))$ to denote the inexact (proximal) gradient steps. In the case $v \\equiv \\nabla f(x)$, the terms $u^\\Lambda$ and $p^\\Lambda$ are used to denote the exact or full gradient and proximal gradient step.\n","sha256_utf8":"858473b0e9bb30140e2e473d0c9446de8873084db9834bbb44cb1748c6709ea5"},{"start":39788,"end":40539,"start_line":449,"end_line":458,"raw_tex":"\\begin{algorithm2e}[t]\n\\caption{\\textbf{S}tochastic \\textbf{E}xtra-Step \\textbf{Q}uasi-\\textbf{N}ewton Method}    \n\\label{alg:seqn}\n\\lnlset{alg:seqn-1}{1}{Initialization: ~~Select the parameter matrices $(\\Lambda_k)_k, (\\Lambda_{k,+})_k \\in \\Spp$. Choose the step sizes $(\\alpha_k)_k$, $(\\beta_k)_k$ and the initial point  $x^0 \\in \\dom~\\vp$ .} \\\\ \n\\For{$k = 0,1,... $}{\n\\lnlset{alg:seqn-2}{2}{Based on $v^k \\approx \\nabla f(x^k)$, compute a direction $d^k$ and set $z^k = x^k + \\beta_k d^k$.} \\\\\n\\lnlset{alg:seqn-3}{3}{Calculate the new oracle $v^k_+ \\approx \\nabla f(z^k)$ and perform the update \\\\\n\\begin{center} $x^{k+1} = \\proxt{\\Lambda_{k,+}}{\\vp}(x^k + \\alpha_k d^k - \\Lambda_{k,+}^{-1}v^k_+)$ \\vspace{-1ex}.\\end{center}}  \n} \n\\end{algorithm2e}","sha256_utf8":"e800588fec90656e1af1d1658e57e7a6cd2e2e57b047e0175f272195ffaab7c0"},{"start":51244,"end":52145,"start_line":509,"end_line":517,"raw_tex":"\\begin{assumption} \\label{ass:lip} Let $f : \\Rn \\to \\R$ be given. We assume: %\\vspace{.5ex}\n\\setlength{\\leftmargini}{7ex}\n\\begin{itemize}\n\\item[{\\rm(A.1)}] The gradient mapping $\\nabla f$ is Lipschitzian on $\\Rn$ with modulus $L_f \\geq 1$. %Moreover, each of the component functions $\\nabla f_i$, $i \\in [N]$, is $L_i$-Lipschitz continuous.\\vspace{.5ex} \n\\item[{\\rm(A.2)}] The objective function $\\psi$ is bounded from below on $\\dom~\\vp$. \\vspace{.5ex}\n%\\item[{\\rm(A.3)}] There exist parameters $\\mu_f \\in \\R$ and $\\mu_\\vp \\geq 0$ with $\\bar\\mu := \\mu_f + \\mu_\\vp > 0$ such that the shifted functions $f - \\frac{\\mu_f}{2}\\|\\cdot\\|^2$ and $\\vp - \\frac{\\mu_\\vp}{2}\\|\\cdot\\|^2$ are convex. \\vspace{.5ex}\n%\\item[{\\rm(A.4)}] There exists a constant $\\bar g_r > 0$ such that for all $x \\in \\dom~r$ there exists $\\lambda \\in \\partial r(x)$ with $\\|\\lambda\\| \\leq \\bar g_r$.\n\\end{itemize}\n\\end{assumption}","sha256_utf8":"401502473b770a2b92d4318b14e416f15816a800e9f899b5baa5c7df8cc06a3d"},{"start":54441,"end":54565,"start_line":544,"end_line":544,"raw_tex":"\\be \\label{eq:lip-ineq} f(y) \\leq f(x) + \\iprod{\\nabla f(x)}{y-x} + \\frac{L_f}{2}  \\|y-x\\|^2, \\quad \\forall~x,y \\in \\Rn, \\ee","sha256_utf8":"7ab269e46123e916e9aed4c12656def89484e97de9db6a7d2db0950a68f14363"},{"start":54570,"end":54626,"start_line":546,"end_line":546,"raw_tex":"which is a direct consequence of assumption {\\rm(A.1)}.\n","sha256_utf8":"728c7ad45383552f249a02ce1790feeb4899fbab9c0360dd3bff10e5c3911546"},{"start":63401,"end":64310,"start_line":612,"end_line":616,"raw_tex":"The stochasticity in Algorithm \\ref{alg:seqn} mainly results from the random selection of the oracles $v^k$ and $v^k_+$ in step \\ref{alg:seqn-2} and \\ref{alg:seqn-3}. In this work, we assume that the stochastic approximations $v^k$ and $v^k_+$ correspond to realizations of the random vectors ${\\sf V}^k : \\Omega \\to \\Rn$ and ${\\sf V}^k_+ : \\Omega \\to \\Rn$. Furthermore, we suppose that the underlying probability space $(\\Omega,\\mathcal F,\\Prob)$ is sufficiently rich allowing us to model the involved random processes in a unified way. We now define the filtrations\n%\n\\[ \\mathcal F^k := \\sigma({\\sf V}^0, {\\sf V}^0_+, ..., {\\sf V}^k), \\quad \\mathcal F^k_+ := \\sigma(\\mathcal F^k \\cup \\sigma({\\sf V}^k_+)). \\]\n%\nBy convention, let $({\\sf D}^k)_k$ denote the stochastic process associated with the directions $(d^k)_k$ chosen in step \\ref{alg:seqn-2}. We will work with the following stochastic assumptions. \n","sha256_utf8":"9547a0e785bfad38f19697fe09546de3c9a6d25dac75ca18e11a446a7ae45b86"},{"start":64311,"end":65652,"start_line":618,"end_line":629,"raw_tex":"\\begin{assumption} \\label{ass:stoch-1} We assume: \n\\setlength{\\leftmargini}{7ex}\n\\begin{itemize}\n\\item[{\\rm(B.1)}] The mapping ${\\sf D}^k : \\Omega \\to \\R^{n}$ is an $\\mathcal F^k$-measurable function for all $k$. \\vspace{.5ex} \n\\item[{\\rm(B.2)}] There is $\\nu_k > 0$ such that we have $\\Exp[\\|{\\sf D}^k\\|^2 \\mid \\mathcal F^{k-1}_+] \\leq \\nu_k^2 \\cdot \\Exp[\\|F^{\\Lambda_k}_{{\\sf V}^k}({\\sf X}^k)\\|^2 \\mid \\mathcal F^{k-1}_+]$ \\aev and for all $k \\in \\N$.\n\\item[{\\rm(B.3)}] For all $k \\in \\N$, it holds $\\Exp[{\\sf V}^k \\mid \\mathcal F^{k-1}_+] = \\nabla f({\\sf X}^k)$, $\\Exp[{\\sf V}^k_+ \\mid \\mathcal F^{k}] = \\nabla f({\\sf Z}^k)$ \\aev and there exists $\\sigma_k, \\sigma_{k,+} > 0$ such that \\aev \n%\\item[{\\rm(A.3)}] There exist parameters $\\mu_f \\in \\R$ and $\\mu_\\vp \\geq 0$ with $\\bar\\mu := \\mu_f + \\mu_\\vp > 0$ such that the shifted functions $f - \\frac{\\mu_f}{2}\\|\\cdot\\|^2$ and $\\vp - \\frac{\\mu_\\vp}{2}\\|\\cdot\\|^2$ are convex. \\vspace{.5ex}\n%\\item[{\\rm(A.4)}] There exists a constant $\\bar g_r > 0$ such that for all $x \\in \\dom~r$ there exists $\\lambda \\in \\partial r(x)$ with $\\|\\lambda\\| \\leq \\bar g_r$.\n\\end{itemize}\n%\n\\[ \\Exp[\\|\\nabla f({\\sf X}^k) - {\\sf V}^k\\|^2 \\mid \\mathcal F_+^{k-1}] \\leq \\sigma_k^2 \\quad \\text{and} \\quad \\Exp[\\|\\nabla f({\\sf Z}^k) - {\\sf V}_+^k\\|^2 \\mid \\mathcal F^{k}] \\leq \\sigma_{k,+}^2. \\]\n\\end{assumption}","sha256_utf8":"ebd05ca1a31559d2f2e57b282dd728e1f0f428c35cb22e7106e82db2e10ad100"},{"start":66636,"end":66778,"start_line":633,"end_line":633,"raw_tex":"\\be \\label{eq:prop-stoch} {\\sf Z}^k \\in \\mathcal F^k \\quad \\text{and} \\quad {\\sf X}^{k+1} \\in {\\mathcal F}^{k}_+ , \\quad \\forall~k \\geq 0. \\ee","sha256_utf8":"c7a9607952be50c59d7fd1fcb4830885e799b4fbc06d630b539e022e4359750c"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem} \\label{theorem:conv-gen} Let the random process $({\\sf X}^k)_k$ be generated by Algorithm \\ref{alg:seqn} using parameter matrices of the form $\\Lambda_k = \\lambda_k^{-1} I$ and $\\Lambda_{k,+} = \\lambda_{k,+}^{-1} I$. Suppose that the assumptions {\\rm(A.1)}--{\\rm(A.2)} and {\\rm(B.1)}--{\\rm(B.3)} are satisfied and let us assume that the step sizes $(\\lambda_k)_k$, $(\\lambda_{k,+})_k$, $(\\alpha_k)_k$, and $(\\beta_k)_k$ are chosen as follows:   \n%\n\\be \\label{eq:choice-step-1} \\lambda_{k,+} \\leq \\frac{1}{L_f}, \\quad \\lambda_k \\leq \\frac{(1-\\bar{\\rho})\\lambda_{k,+}}{1+\\nu_k^2 (\\alpha_k + L_f \\beta_k \\lambda_{k,+})^2}, \\ee\n%\nfor all $k \\in \\N$ and some $\\bar{\\rho} \\in (0,1)$. Then, under the additional conditions \n%\n\\be \\label{eq:choice-step-2} \\sum \\lambda_k = \\infty, \\quad \\sum \\lambda_k \\sigma_k^2 < \\infty, \\quad \\sum \\lambda_{k,+} \\sigma_{k,+}^2 < \\infty \\ee\n%\nit follows $\\liminf_{k \\to \\infty} \\Exp[\\|F^I({\\sf X}^k)\\|^2] = 0$ and $\\liminf_{k \\to \\infty} \\| F^I({\\sf X}^k) \\| = 0$ \\as and $(\\psi({\\sf X}^k))_k$ \\as converges to a random variable $\\sf Y^*$ with $\\lim_{k\\to \\infty} \\Exp[\\psi({\\sf X}^k)] = \\Exp[{\\sf Y}^*]$.  \n\\end{theorem}","units":[{"start":66849,"end":68013,"start_line":637,"end_line":646,"raw_tex":"\\begin{theorem} \\label{theorem:conv-gen} Let the random process $({\\sf X}^k)_k$ be generated by Algorithm \\ref{alg:seqn} using parameter matrices of the form $\\Lambda_k = \\lambda_k^{-1} I$ and $\\Lambda_{k,+} = \\lambda_{k,+}^{-1} I$. Suppose that the assumptions {\\rm(A.1)}--{\\rm(A.2)} and {\\rm(B.1)}--{\\rm(B.3)} are satisfied and let us assume that the step sizes $(\\lambda_k)_k$, $(\\lambda_{k,+})_k$, $(\\alpha_k)_k$, and $(\\beta_k)_k$ are chosen as follows:   \n%\n\\be \\label{eq:choice-step-1} \\lambda_{k,+} \\leq \\frac{1}{L_f}, \\quad \\lambda_k \\leq \\frac{(1-\\bar{\\rho})\\lambda_{k,+}}{1+\\nu_k^2 (\\alpha_k + L_f \\beta_k \\lambda_{k,+})^2}, \\ee\n%\nfor all $k \\in \\N$ and some $\\bar{\\rho} \\in (0,1)$. Then, under the additional conditions \n%\n\\be \\label{eq:choice-step-2} \\sum \\lambda_k = \\infty, \\quad \\sum \\lambda_k \\sigma_k^2 < \\infty, \\quad \\sum \\lambda_{k,+} \\sigma_{k,+}^2 < \\infty \\ee\n%\nit follows $\\liminf_{k \\to \\infty} \\Exp[\\|F^I({\\sf X}^k)\\|^2] = 0$ and $\\liminf_{k \\to \\infty} \\| F^I({\\sf X}^k) \\| = 0$ \\as and $(\\psi({\\sf X}^k))_k$ \\as converges to a random variable $\\sf Y^*$ with $\\lim_{k\\to \\infty} \\Exp[\\psi({\\sf X}^k)] = \\Exp[{\\sf Y}^*]$.  \n\\end{theorem}","sha256_utf8":"9fc3fb7aafa82ec08f07808e05d5c455e1e8d2c95972d4c9cf02a46c9dc2adab"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma} \\label{lemma:prox-descent} Let $x \\in \\dom~\\vp$, $d \\in \\Rn$, $\\Lambda = \\lambda^{-1} I $, $\\Lambda_+ = \\lambda_+^{-1} I$, $\\alpha, \\beta, \\rho > 0$, and the approximations $v, v_+ \\in \\Rn$ be given. Under assumption {\\rm(A.1)} it holds that\n%\n\\begin{align*} 2[\\psi(p_+) - \\psi(x)] & \\\\ & \\hspace{-13.5ex} \\leq \\frac{1}{\\rho} \\|\\nabla f(x) - v\\|^2 +  {\\lambda_+} \\|\\nabla f(x+\\beta d) - v_+\\|^2 +  \\left[ \\frac{1}{\\lambda_+} - \\frac{1}{\\lambda} \\right]  \\|F^\\Lambda_v(x)\\|^2 \\\\ & \\hspace{-11.5ex} + \\left[ L_f - \\frac{1}{\\lambda_+}\\right] \\|p_+ - x\\|^2  +  \\left[ \\rho - \\frac{1}{\\lambda} \\right]  \\|p^\\Lambda_v(x) - p^\\Lambda(x)\\|^2 + {\\ell(\\lambda_+,\\alpha,\\beta)} \\|d\\|^2   \\\\ & \\hspace{-11.5ex} - \\frac{1}{\\lambda} \\|F^\\Lambda(x)\\|^2 +  2\\iprod{\\nabla f(x+\\beta d) - v_+}{\\lambda_+(\\nabla f(x) - \\nabla f(x+\\beta d)) +\\alpha d}, \\end{align*}\n%\nwhere $p_+ := \\proxt{\\Lambda_+}{\\vp}(x+\\alpha d - \\lambda_+v_+)$ and $\\ell(\\lambda_+,\\alpha,\\beta) := \\lambda_+(\\frac{\\alpha}{\\lambda_+} + L_f\\beta)^2$. % $\\ell_\\alpha(z) := \\frac{L}{2}(\\alpha \\|W\\| + \\|I-\\alpha W\\|)^2 + \\left[z - \\frac{L}{2}\\right ] \\|I-\\alpha W\\|^2$.\n\\end{lemma}\n\n\\begin{proof} First, setting $p = \\proxt{\\Gamma}{\\vp}(y+u-\\Gamma^{-1}w)$ and applying the optimality condition of the proximity operator \\eqref{eq:prox-opt}, we have\n%\n\\begin{align*} \\vp(p) - \\vp(z) \\leq \\iprod{\\Gamma u - w}{p-z} + \\half \\|y-z\\|^2_\\Gamma - \\half \\|p-y\\|^2_\\Gamma - \\half \\|p-z\\|_\\Gamma^2 \\end{align*}\n%\\be \\label{eq:vp-prox} \\vp(\\proxt{\\Gamma}{\\vp}(y)) - \\vp(z) \\leq \\iprod{\\Gamma(y - \\proxt{\\Gamma}{\\vp}(y))}{\\proxt{\\Gamma}{\\vp}(y) - z}, \\quad \\Gamma \\in \\Spp, \\ee \n%\nfor all $y,u,w \\in \\Rn$, $z \\in \\dom~\\vp$, and $\\Gamma \\in \\Spp$. Utilizing this estimate three times for different choices of $y$, $z \\in \\dom~\\vp$, and $\\Gamma$ and applying the descent lemma \\eqref{eq:lip-ineq}, Young's inequality, and the Lipschitz continuity of $\\nabla f$, we obtain\n%\n\\begingroup\n\\allowdisplaybreaks\n\\begin{align*} \\psi(p_+) - \\psi(x) & \\\\& \\hspace{-12ex} = f(p_+) - f(x) + \\vp(p_+) - \\vp(p^\\Lambda_v(x)) \\\\ & \\hspace{-9ex}  + \\vp(p^\\Lambda_v(x)) - \\vp(p^\\Lambda(x)) +\\vp(p^\\Lambda(x)) - \\vp(x) \\\\ & \\hspace{-12ex} \\leq  \\iprod{\\nabla f(x)}{p_+-p^\\Lambda_v(x)} + \\frac{L_f}{2}  \\|p_+-x\\|^2 + \\iprod{\\alpha \\Lambda_+ d - v_+}{p_+ - p^\\Lambda_v(x)} \\\\ & \\hspace{-9ex} + \\half \\left[ \\frac{1}{\\lambda_+} - \\frac{1}{\\lambda} \\right] \\|F^\\Lambda_v(x)\\|^2 - \\frac{1}{2\\lambda_+} \\|p_+ - x\\|^2 - \\frac{1}{2\\lambda_+} \\|p_+ - p^\\Lambda_v(x)\\|^2 \\\\ & \\hspace{-9ex} + \\iprod{\\nabla f(x)-v}{p^\\Lambda_v(x) - p^\\Lambda(x)} - \\frac{1}{2\\lambda} \\left[ \\|p^\\Lambda_v(x) - p^\\Lambda(x)\\|^2 + \\|F^\\Lambda(x)\\|^2 \\right] \\\\ & \\hspace{-12ex} \\leq  \\frac{\\| \\nabla f(x) - v\\|^2}{2\\rho}  +  \\left[ \\frac{\\rho}{2} - \\frac{1}{2\\lambda}\\right] \\|p^\\Lambda_v(x) - p^\\Lambda(x) \\|^2 +  \\left[ \\frac{L_f}{2} - \\frac{1}{2\\lambda_+} \\right] \\|p_+-x\\|^2   \\\\  &\\hspace{-9ex}  + \\frac12\\left[\\frac{1}{\\lambda_+} - \\frac{1}{\\lambda}  \\right] \\|F^\\Lambda_v(x)\\|^2 - \\frac{1}{2\\lambda} \\|F^\\Lambda(x)\\|^2  + \\frac{\\lambda_+}{2} \\|\\nabla f(x) + \\lambda_+^{-1}\\alpha d - v_+\\|^2. \\end{align*}\n%\n%\\\\ & \\hspace{-9ex} + \\iprod{\\Lambda F^\\Lambda_v(x) - v}{p^\\Lambda_v(x) - p^\\Lambda(x)} - \\iprod{\\Lambda F^\\Lambda(x) - \\nabla f(x)}{x-p^\\Lambda(x)}  \\\\ & \\hspace{-12ex} = \\iprod{\\nabla f(x) - v}{p^\\Lambda_v(x) - p^\\Lambda(x)} + \\iprod{\\alpha\\lambda_+^{-1}d + \\nabla f(x) - v_+}{p_+ - p^\\Lambda_v(x)} \\\\ & \\hspace{-9ex} +  \\left[ \\frac{L_f}{2} - \\frac{1}{2\\lambda_+} \\right] \\|p_+ - x\\|^2 - \\frac{1}{2\\lambda_+} \\|p_+ - p^\\Lambda_v(x)\\|^2 - \\frac{1}{2\\lambda} \\|F^\\Lambda(x)\\|^2  \\\\ & \\hspace{-9ex}   - \\frac{1}{2\\lambda} \\|p^\\Lambda_v(x) - p^\\Lambda(x)\\|^2 + \\frac12 \\left[ \\frac{1}{\\lambda_+} - \\frac{1}{\\lambda} \\right] \\|F^\\Lambda_v(x)\\|^2  \n\\endgroup\n%\nExtending the penultimate term by adding and subtracting $\\nabla f(x+\\beta d)$ and using\n%\n\\[  \\| \\nabla f(x) - \\nabla f(x+\\beta d) + \\lambda_+^{-1}\\alpha d \\|^2 \\leq \\left( \\frac{\\alpha}{\\lambda_+} + L_f\\beta \\right)^2 \\|d\\|^2 = \\frac{\\ell(\\lambda_+,\\alpha,\\beta)}{\\lambda_+} \\|d\\|^2\n\\]\n%\nwe finally get the estimate stated in Lemma \\ref{lemma:prox-descent}. \n%Since the first term is nonpositive for all $\\alpha \\leq \\overline\\alpha$, this establishes \\eqref{eq:est-prox-step}.\n\\end{proof}\n\n\\begin{theorem} \\label{theorem:conv-superm} Let $({\\sf Y}_k)_k$, $({\\sf U}_k)_k$, $({\\sf A}_k)_k$, $({\\sf B}_k)_k$ be sequences of nonnegative integrable random variables, adapted to the filtration $(\\mathcal U_k)_k$ such that we have $\\Exp[{\\sf Y}_{k+1} \\mid \\mathcal U_k] \\leq (1+{\\sf A}_k) {\\sf Y}_k - {\\sf U}_k + {\\sf B}_k$, $\\sum {\\sf A}_k < \\infty$, and $\\sum {\\sf B}_k < \\infty$ \\as for all $k \\in \\N$. Then, $({\\sf Y}_k)_k$ \\as converges and it holds $\\sum {\\sf U}_k < \\infty$. \\end{theorem}\n\n\\begin{proof} First, due to the law of total expectation and $\\mathcal F^{k-1}_+ \\subset \\mathcal F^k$ and ${\\sf X}^k,{\\sf D}^k, {\\sf Z}^k \\in \\mathcal F^k$, it follows\n%\n\\begin{align*} \\Exp[\\iprod{\\nabla f({\\sf Z}^k) - {\\sf V}^k_+}{\\lambda_{k,+}(\\nabla f({\\sf X}^k) - \\nabla f({\\sf Z}^k)) +\\alpha_k {\\sf D}^k} \\mid \\mathcal F^{k-1}_+] & \\\\ & \\hspace{-55ex} = \\Exp[\\iprod{\\Exp[\\nabla f({\\sf Z}^k) - {\\sf V}^k_+ \\mid \\mathcal F^k]}{\\lambda_{k,+}(\\nabla f({\\sf X}^k) - \\nabla f({\\sf Z}^k)) +\\alpha_k {\\sf D}^k} \\mid \\mathcal F^{k-1}_+] = 0, \\end{align*}\n%\n$\\text{a.e.}$, where we used the fact that ${\\sf V}^k_+$ is an unbiased estimator of the gradient $\\nabla f({\\sf Z}^k)$. Next, taking conditional expectation in Lemma \\ref{lemma:prox-descent}, applying assumptions (B.2) and (B.3), and choosing $\\rho = \\lambda_k^{-1}$, we obtain\n%\n\\begin{align*} \\Exp[\\psi({\\sf X}^{k+1}) \\mid \\mathcal F^{k-1}_+] - \\psi({\\sf X}^k) & \\\\ & \\hspace{-24ex} \\leq  \\frac{\\lambda_{k,+}\\sigma_{k,+}^2}{2} + \\half  \\left[ L_f - \\frac{1}{\\lambda_{k,+}} \\right] \\Exp [\\|{\\sf X}^{k+1} - {\\sf X}^k\\|^2\\mid \\mathcal F^{k-1}_+ ] - \\frac{1}{2\\lambda_k} \\|F^{\\Lambda_k}({\\sf X}^k)\\|^2 \\\\ & \\hspace{-20ex} + \\frac{\\lambda_k \\sigma_k^2}{2}  + \\half \\left[ \\frac{1}{\\lambda_{k,+}} - \\frac{1}{\\lambda_k} + \\nu_k^2 \\cdot \\ell(\\lambda_{k,+},\\alpha_k,\\beta_k) \\right] \\Exp[\\|F^{\\Lambda_k}_{{\\sf V}^k}({\\sf X}^k)\\|^2 \\mid \\mathcal F^{k-1}_+] \n\\end{align*}\n%\nalmost everywhere. Utilizing the bounds stated in \\eqref{eq:choice-step-1}, we have\n%\n\\begin{align} \\nonumber \\Exp[\\psi({\\sf X}^{k+1}) \\mid \\mathcal F^{k-1}_+] - \\psi({\\sf X}^k) & \\\\  \\label{eq:est-martingale} & \\hspace{-20ex} \\leq \\frac{\\lambda_k \\sigma_k^2}{2} + \\frac{\\lambda_{k,+}\\sigma_{k,+}^2}{2} - \\frac{1}{2\\lambda_k} \\|F^{\\Lambda_k}({\\sf X}^k)\\|^2 - \\frac{\\bar{\\rho}}{2\\lambda_k} \\Exp[ \\|F^{\\Lambda_k}_{{\\sf V}^k}({\\sf X}^k)\\|^2 \\mid \\mathcal F^{k-1}_+ ].  \\end{align}\n%\nNow, by assumption (A.2) there exists $\\psi^* \\in \\R$ such that $\\psi(x) \\geq \\psi^*$ for all $x \\in \\Rn$. Taking expectation and summing the latter inequality, we get\n%\n\\begin{align*} \\sum_{k= 0}^R \\frac{\\Exp[\\|F^{\\Lambda_k}({\\sf X}^k)\\|^2 + \\bar{\\rho} \\cdot \\|F^{\\Lambda_k}_{{\\sf V}^k}({\\sf X}^k)\\|^2]}{\\lambda_k} \\leq 2 [\\psi(x^0) - \\psi^*] + \\sum_{k = 0}^R \\lambda_k \\sigma_k^2 + \\lambda_{k,+}\\sigma_{k,+}^2 \\end{align*}\n%\nfor all $R \\in \\N$. Due to $L_f \\geq 1$, it holds $\\lambda_{k,+} \\leq 1$, $\\lambda_k \\leq 1$ and by \\cite[Lemma 2]{Nes13}, we know that the mapping $\\delta \\mapsto {\\delta}^{-1} \\|F^{\\frac{1}{\\delta}I}(x)\\|$ is a decreasing function of $\\delta$. This yields $\\Exp[\\|F^I({\\sf X}^k)\\|^2] \\leq  \\lambda_k^{-2} \\Exp[ \\|F^{\\Lambda_k}({\\sf X}^k)\\|^2]$ for all $k \\in \\N$ and thus, we can infer  $\\sum \\lambda_k \\Exp[\\|F^I({\\sf X}^k)\\|^2] < \\infty$. Hence, the first statement in Theorem \\ref{theorem:conv-gen} follows from the assumption $\\sum \\lambda_k = \\infty$. Furthermore, applying the Borel-Cantelli lemma, it holds that $\\sum \\lambda_k \\|F^I({\\sf X}^k)\\|^2 < \\infty$ a.s., which finally implies $\\liminf_{k\\to \\infty} \\| F^I({\\sf X}^k) \\| = 0$ with probability 1. The third claim in Theorem \\ref{theorem:conv-gen} is a direct consequence of assumption (A.2), \\eqref{eq:est-martingale}, and Theorem \\ref{theorem:conv-superm}. In order to establish convergence of $(\\Exp[\\psi({\\sf X}^k)])_k$, we first notice that we have \n%\n\\begin{align} \\nonumber \\psi({\\sf X}^{k+1}) & \\leq \\psi({\\sf X}^k) + \\frac{\\lambda_k}{2} \\|\\nabla f({\\sf X}^k) - {\\sf V}^k\\|^2  \\\\ \\label{eq:des-later}  & \\hspace{3ex}  +\\lambda_{k,+} \\|\\nabla f({\\sf Z}^k) - {\\sf V}_+^k\\|^2 + (\\nu_k^2\\lambda_k)^{-1}\\|{\\sf D}^k\\|^2, \\end{align}\n%\n(almost) surely and for all $k \\in \\N$, which follows from Lemma \\ref{lemma:prox-descent}, $\\iprod{a}{b} \\leq \\half \\|a\\|^2 + \\half \\|b\\|^2$, $a,b \\in \\Rn$, $\\lambda_{k,+}^{-1} \\leq \\lambda_k^{-1}$, and $\\ell(\\lambda_{k,+},\\alpha_k,\\beta_k) < \\nu_k^{-2}(\\lambda_k^{-1} - \\lambda_{k,+}^{-1}) \\leq (\\nu_k^2 \\lambda_k)^{-1}$. By induction, this shows that the nonnegative random variable $\\psi({\\sf X}^k) - \\psi^*$ is (almost) surely dominated by \n%\n\\[ {\\sf U} := \\psi(x^0) - \\psi^* + \\sum_{k=0}^\\infty  \\frac{\\lambda_k}{2} \\|\\nabla f({\\sf X}^k) - {\\sf V}^k\\|^2 + \\lambda_{k,+} \\|\\nabla f({\\sf Z}^k) - {\\sf V}_+^k\\|^2 + \\frac{\\|{\\sf D}^k\\|^2}{\\nu_k^2\\lambda_k}   \\]\n%\nfor all $k \\in \\N$. Moreover due to Fatou's lemma, we have $\\Exp[{\\sf U}] < \\infty$. Hence, our last claim is a consequence of the dominated convergence theorem. \n\\end{proof}","units":[{"start":54627,"end":55773,"start_line":548,"end_line":553,"raw_tex":"\\begin{lemma} \\label{lemma:prox-descent} Let $x \\in \\dom~\\vp$, $d \\in \\Rn$, $\\Lambda = \\lambda^{-1} I $, $\\Lambda_+ = \\lambda_+^{-1} I$, $\\alpha, \\beta, \\rho > 0$, and the approximations $v, v_+ \\in \\Rn$ be given. Under assumption {\\rm(A.1)} it holds that\n%\n\\begin{align*} 2[\\psi(p_+) - \\psi(x)] & \\\\ & \\hspace{-13.5ex} \\leq \\frac{1}{\\rho} \\|\\nabla f(x) - v\\|^2 +  {\\lambda_+} \\|\\nabla f(x+\\beta d) - v_+\\|^2 +  \\left[ \\frac{1}{\\lambda_+} - \\frac{1}{\\lambda} \\right]  \\|F^\\Lambda_v(x)\\|^2 \\\\ & \\hspace{-11.5ex} + \\left[ L_f - \\frac{1}{\\lambda_+}\\right] \\|p_+ - x\\|^2  +  \\left[ \\rho - \\frac{1}{\\lambda} \\right]  \\|p^\\Lambda_v(x) - p^\\Lambda(x)\\|^2 + {\\ell(\\lambda_+,\\alpha,\\beta)} \\|d\\|^2   \\\\ & \\hspace{-11.5ex} - \\frac{1}{\\lambda} \\|F^\\Lambda(x)\\|^2 +  2\\iprod{\\nabla f(x+\\beta d) - v_+}{\\lambda_+(\\nabla f(x) - \\nabla f(x+\\beta d)) +\\alpha d}, \\end{align*}\n%\nwhere $p_+ := \\proxt{\\Lambda_+}{\\vp}(x+\\alpha d - \\lambda_+v_+)$ and $\\ell(\\lambda_+,\\alpha,\\beta) := \\lambda_+(\\frac{\\alpha}{\\lambda_+} + L_f\\beta)^2$. % $\\ell_\\alpha(z) := \\frac{L}{2}(\\alpha \\|W\\| + \\|I-\\alpha W\\|)^2 + \\left[z - \\frac{L}{2}\\right ] \\|I-\\alpha W\\|^2$.\n\\end{lemma}","sha256_utf8":"07e5a7ec649f1d21b7d9e6b7eb8b2ac195b9ddb6acc64a6d0bcede8a874d5cd1"},{"start":55774,"end":58895,"start_line":554,"end_line":575,"raw_tex":"\\begin{proof} First, setting $p = \\proxt{\\Gamma}{\\vp}(y+u-\\Gamma^{-1}w)$ and applying the optimality condition of the proximity operator \\eqref{eq:prox-opt}, we have\n%\n\\begin{align*} \\vp(p) - \\vp(z) \\leq \\iprod{\\Gamma u - w}{p-z} + \\half \\|y-z\\|^2_\\Gamma - \\half \\|p-y\\|^2_\\Gamma - \\half \\|p-z\\|_\\Gamma^2 \\end{align*}\n%\\be \\label{eq:vp-prox} \\vp(\\proxt{\\Gamma}{\\vp}(y)) - \\vp(z) \\leq \\iprod{\\Gamma(y - \\proxt{\\Gamma}{\\vp}(y))}{\\proxt{\\Gamma}{\\vp}(y) - z}, \\quad \\Gamma \\in \\Spp, \\ee \n%\nfor all $y,u,w \\in \\Rn$, $z \\in \\dom~\\vp$, and $\\Gamma \\in \\Spp$. Utilizing this estimate three times for different choices of $y$, $z \\in \\dom~\\vp$, and $\\Gamma$ and applying the descent lemma \\eqref{eq:lip-ineq}, Young's inequality, and the Lipschitz continuity of $\\nabla f$, we obtain\n%\n\\begingroup\n\\allowdisplaybreaks\n\\begin{align*} \\psi(p_+) - \\psi(x) & \\\\& \\hspace{-12ex} = f(p_+) - f(x) + \\vp(p_+) - \\vp(p^\\Lambda_v(x)) \\\\ & \\hspace{-9ex}  + \\vp(p^\\Lambda_v(x)) - \\vp(p^\\Lambda(x)) +\\vp(p^\\Lambda(x)) - \\vp(x) \\\\ & \\hspace{-12ex} \\leq  \\iprod{\\nabla f(x)}{p_+-p^\\Lambda_v(x)} + \\frac{L_f}{2}  \\|p_+-x\\|^2 + \\iprod{\\alpha \\Lambda_+ d - v_+}{p_+ - p^\\Lambda_v(x)} \\\\ & \\hspace{-9ex} + \\half \\left[ \\frac{1}{\\lambda_+} - \\frac{1}{\\lambda} \\right] \\|F^\\Lambda_v(x)\\|^2 - \\frac{1}{2\\lambda_+} \\|p_+ - x\\|^2 - \\frac{1}{2\\lambda_+} \\|p_+ - p^\\Lambda_v(x)\\|^2 \\\\ & \\hspace{-9ex} + \\iprod{\\nabla f(x)-v}{p^\\Lambda_v(x) - p^\\Lambda(x)} - \\frac{1}{2\\lambda} \\left[ \\|p^\\Lambda_v(x) - p^\\Lambda(x)\\|^2 + \\|F^\\Lambda(x)\\|^2 \\right] \\\\ & \\hspace{-12ex} \\leq  \\frac{\\| \\nabla f(x) - v\\|^2}{2\\rho}  +  \\left[ \\frac{\\rho}{2} - \\frac{1}{2\\lambda}\\right] \\|p^\\Lambda_v(x) - p^\\Lambda(x) \\|^2 +  \\left[ \\frac{L_f}{2} - \\frac{1}{2\\lambda_+} \\right] \\|p_+-x\\|^2   \\\\  &\\hspace{-9ex}  + \\frac12\\left[\\frac{1}{\\lambda_+} - \\frac{1}{\\lambda}  \\right] \\|F^\\Lambda_v(x)\\|^2 - \\frac{1}{2\\lambda} \\|F^\\Lambda(x)\\|^2  + \\frac{\\lambda_+}{2} \\|\\nabla f(x) + \\lambda_+^{-1}\\alpha d - v_+\\|^2. \\end{align*}\n%\n%\\\\ & \\hspace{-9ex} + \\iprod{\\Lambda F^\\Lambda_v(x) - v}{p^\\Lambda_v(x) - p^\\Lambda(x)} - \\iprod{\\Lambda F^\\Lambda(x) - \\nabla f(x)}{x-p^\\Lambda(x)}  \\\\ & \\hspace{-12ex} = \\iprod{\\nabla f(x) - v}{p^\\Lambda_v(x) - p^\\Lambda(x)} + \\iprod{\\alpha\\lambda_+^{-1}d + \\nabla f(x) - v_+}{p_+ - p^\\Lambda_v(x)} \\\\ & \\hspace{-9ex} +  \\left[ \\frac{L_f}{2} - \\frac{1}{2\\lambda_+} \\right] \\|p_+ - x\\|^2 - \\frac{1}{2\\lambda_+} \\|p_+ - p^\\Lambda_v(x)\\|^2 - \\frac{1}{2\\lambda} \\|F^\\Lambda(x)\\|^2  \\\\ & \\hspace{-9ex}   - \\frac{1}{2\\lambda} \\|p^\\Lambda_v(x) - p^\\Lambda(x)\\|^2 + \\frac12 \\left[ \\frac{1}{\\lambda_+} - \\frac{1}{\\lambda} \\right] \\|F^\\Lambda_v(x)\\|^2  \n\\endgroup\n%\nExtending the penultimate term by adding and subtracting $\\nabla f(x+\\beta d)$ and using\n%\n\\[  \\| \\nabla f(x) - \\nabla f(x+\\beta d) + \\lambda_+^{-1}\\alpha d \\|^2 \\leq \\left( \\frac{\\alpha}{\\lambda_+} + L_f\\beta \\right)^2 \\|d\\|^2 = \\frac{\\ell(\\lambda_+,\\alpha,\\beta)}{\\lambda_+} \\|d\\|^2\n\\]\n%\nwe finally get the estimate stated in Lemma \\ref{lemma:prox-descent}. \n%Since the first term is nonpositive for all $\\alpha \\leq \\overline\\alpha$, this establishes \\eqref{eq:est-prox-step}.\n\\end{proof}","sha256_utf8":"5e8453f8b38e4dee8125d83f69c6bff70e649dc82e3bff520b3eb3ab0ce78eb2"},{"start":60880,"end":61379,"start_line":594,"end_line":594,"raw_tex":"\\begin{theorem} \\label{theorem:conv-superm} Let $({\\sf Y}_k)_k$, $({\\sf U}_k)_k$, $({\\sf A}_k)_k$, $({\\sf B}_k)_k$ be sequences of nonnegative integrable random variables, adapted to the filtration $(\\mathcal U_k)_k$ such that we have $\\Exp[{\\sf Y}_{k+1} \\mid \\mathcal U_k] \\leq (1+{\\sf A}_k) {\\sf Y}_k - {\\sf U}_k + {\\sf B}_k$, $\\sum {\\sf A}_k < \\infty$, and $\\sum {\\sf B}_k < \\infty$ \\as for all $k \\in \\N$. Then, $({\\sf Y}_k)_k$ \\as converges and it holds $\\sum {\\sf U}_k < \\infty$. \\end{theorem}","sha256_utf8":"70e255a32b4fc5d2587369ac7c4973710123eb53e5e7b20635e55c8a25447ed1"},{"start":68015,"end":72492,"start_line":648,"end_line":674,"raw_tex":"\\begin{proof} First, due to the law of total expectation and $\\mathcal F^{k-1}_+ \\subset \\mathcal F^k$ and ${\\sf X}^k,{\\sf D}^k, {\\sf Z}^k \\in \\mathcal F^k$, it follows\n%\n\\begin{align*} \\Exp[\\iprod{\\nabla f({\\sf Z}^k) - {\\sf V}^k_+}{\\lambda_{k,+}(\\nabla f({\\sf X}^k) - \\nabla f({\\sf Z}^k)) +\\alpha_k {\\sf D}^k} \\mid \\mathcal F^{k-1}_+] & \\\\ & \\hspace{-55ex} = \\Exp[\\iprod{\\Exp[\\nabla f({\\sf Z}^k) - {\\sf V}^k_+ \\mid \\mathcal F^k]}{\\lambda_{k,+}(\\nabla f({\\sf X}^k) - \\nabla f({\\sf Z}^k)) +\\alpha_k {\\sf D}^k} \\mid \\mathcal F^{k-1}_+] = 0, \\end{align*}\n%\n$\\text{a.e.}$, where we used the fact that ${\\sf V}^k_+$ is an unbiased estimator of the gradient $\\nabla f({\\sf Z}^k)$. Next, taking conditional expectation in Lemma \\ref{lemma:prox-descent}, applying assumptions (B.2) and (B.3), and choosing $\\rho = \\lambda_k^{-1}$, we obtain\n%\n\\begin{align*} \\Exp[\\psi({\\sf X}^{k+1}) \\mid \\mathcal F^{k-1}_+] - \\psi({\\sf X}^k) & \\\\ & \\hspace{-24ex} \\leq  \\frac{\\lambda_{k,+}\\sigma_{k,+}^2}{2} + \\half  \\left[ L_f - \\frac{1}{\\lambda_{k,+}} \\right] \\Exp [\\|{\\sf X}^{k+1} - {\\sf X}^k\\|^2\\mid \\mathcal F^{k-1}_+ ] - \\frac{1}{2\\lambda_k} \\|F^{\\Lambda_k}({\\sf X}^k)\\|^2 \\\\ & \\hspace{-20ex} + \\frac{\\lambda_k \\sigma_k^2}{2}  + \\half \\left[ \\frac{1}{\\lambda_{k,+}} - \\frac{1}{\\lambda_k} + \\nu_k^2 \\cdot \\ell(\\lambda_{k,+},\\alpha_k,\\beta_k) \\right] \\Exp[\\|F^{\\Lambda_k}_{{\\sf V}^k}({\\sf X}^k)\\|^2 \\mid \\mathcal F^{k-1}_+] \n\\end{align*}\n%\nalmost everywhere. Utilizing the bounds stated in \\eqref{eq:choice-step-1}, we have\n%\n\\begin{align} \\nonumber \\Exp[\\psi({\\sf X}^{k+1}) \\mid \\mathcal F^{k-1}_+] - \\psi({\\sf X}^k) & \\\\  \\label{eq:est-martingale} & \\hspace{-20ex} \\leq \\frac{\\lambda_k \\sigma_k^2}{2} + \\frac{\\lambda_{k,+}\\sigma_{k,+}^2}{2} - \\frac{1}{2\\lambda_k} \\|F^{\\Lambda_k}({\\sf X}^k)\\|^2 - \\frac{\\bar{\\rho}}{2\\lambda_k} \\Exp[ \\|F^{\\Lambda_k}_{{\\sf V}^k}({\\sf X}^k)\\|^2 \\mid \\mathcal F^{k-1}_+ ].  \\end{align}\n%\nNow, by assumption (A.2) there exists $\\psi^* \\in \\R$ such that $\\psi(x) \\geq \\psi^*$ for all $x \\in \\Rn$. Taking expectation and summing the latter inequality, we get\n%\n\\begin{align*} \\sum_{k= 0}^R \\frac{\\Exp[\\|F^{\\Lambda_k}({\\sf X}^k)\\|^2 + \\bar{\\rho} \\cdot \\|F^{\\Lambda_k}_{{\\sf V}^k}({\\sf X}^k)\\|^2]}{\\lambda_k} \\leq 2 [\\psi(x^0) - \\psi^*] + \\sum_{k = 0}^R \\lambda_k \\sigma_k^2 + \\lambda_{k,+}\\sigma_{k,+}^2 \\end{align*}\n%\nfor all $R \\in \\N$. Due to $L_f \\geq 1$, it holds $\\lambda_{k,+} \\leq 1$, $\\lambda_k \\leq 1$ and by \\cite[Lemma 2]{Nes13}, we know that the mapping $\\delta \\mapsto {\\delta}^{-1} \\|F^{\\frac{1}{\\delta}I}(x)\\|$ is a decreasing function of $\\delta$. This yields $\\Exp[\\|F^I({\\sf X}^k)\\|^2] \\leq  \\lambda_k^{-2} \\Exp[ \\|F^{\\Lambda_k}({\\sf X}^k)\\|^2]$ for all $k \\in \\N$ and thus, we can infer  $\\sum \\lambda_k \\Exp[\\|F^I({\\sf X}^k)\\|^2] < \\infty$. Hence, the first statement in Theorem \\ref{theorem:conv-gen} follows from the assumption $\\sum \\lambda_k = \\infty$. Furthermore, applying the Borel-Cantelli lemma, it holds that $\\sum \\lambda_k \\|F^I({\\sf X}^k)\\|^2 < \\infty$ a.s., which finally implies $\\liminf_{k\\to \\infty} \\| F^I({\\sf X}^k) \\| = 0$ with probability 1. The third claim in Theorem \\ref{theorem:conv-gen} is a direct consequence of assumption (A.2), \\eqref{eq:est-martingale}, and Theorem \\ref{theorem:conv-superm}. In order to establish convergence of $(\\Exp[\\psi({\\sf X}^k)])_k$, we first notice that we have \n%\n\\begin{align} \\nonumber \\psi({\\sf X}^{k+1}) & \\leq \\psi({\\sf X}^k) + \\frac{\\lambda_k}{2} \\|\\nabla f({\\sf X}^k) - {\\sf V}^k\\|^2  \\\\ \\label{eq:des-later}  & \\hspace{3ex}  +\\lambda_{k,+} \\|\\nabla f({\\sf Z}^k) - {\\sf V}_+^k\\|^2 + (\\nu_k^2\\lambda_k)^{-1}\\|{\\sf D}^k\\|^2, \\end{align}\n%\n(almost) surely and for all $k \\in \\N$, which follows from Lemma \\ref{lemma:prox-descent}, $\\iprod{a}{b} \\leq \\half \\|a\\|^2 + \\half \\|b\\|^2$, $a,b \\in \\Rn$, $\\lambda_{k,+}^{-1} \\leq \\lambda_k^{-1}$, and $\\ell(\\lambda_{k,+},\\alpha_k,\\beta_k) < \\nu_k^{-2}(\\lambda_k^{-1} - \\lambda_{k,+}^{-1}) \\leq (\\nu_k^2 \\lambda_k)^{-1}$. By induction, this shows that the nonnegative random variable $\\psi({\\sf X}^k) - \\psi^*$ is (almost) surely dominated by \n%\n\\[ {\\sf U} := \\psi(x^0) - \\psi^* + \\sum_{k=0}^\\infty  \\frac{\\lambda_k}{2} \\|\\nabla f({\\sf X}^k) - {\\sf V}^k\\|^2 + \\lambda_{k,+} \\|\\nabla f({\\sf Z}^k) - {\\sf V}_+^k\\|^2 + \\frac{\\|{\\sf D}^k\\|^2}{\\nu_k^2\\lambda_k}   \\]\n%\nfor all $k \\in \\N$. Moreover due to Fatou's lemma, we have $\\Exp[{\\sf U}] < \\infty$. Hence, our last claim is a consequence of the dominated convergence theorem. \n\\end{proof}","sha256_utf8":"13ffbbb72fcf21f586155ad9a7b62697425d488358df93f0c13f4e7fd1cf5aab"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"随机额外步全局收敛","source_units":[{"start":66849,"end":68013,"start_line":637,"end_line":646,"raw_tex":"\\begin{theorem} \\label{theorem:conv-gen} Let the random process $({\\sf X}^k)_k$ be generated by Algorithm \\ref{alg:seqn} using parameter matrices of the form $\\Lambda_k = \\lambda_k^{-1} I$ and $\\Lambda_{k,+} = \\lambda_{k,+}^{-1} I$. Suppose that the assumptions {\\rm(A.1)}--{\\rm(A.2)} and {\\rm(B.1)}--{\\rm(B.3)} are satisfied and let us assume that the step sizes $(\\lambda_k)_k$, $(\\lambda_{k,+})_k$, $(\\alpha_k)_k$, and $(\\beta_k)_k$ are chosen as follows:   \n%\n\\be \\label{eq:choice-step-1} \\lambda_{k,+} \\leq \\frac{1}{L_f}, \\quad \\lambda_k \\leq \\frac{(1-\\bar{\\rho})\\lambda_{k,+}}{1+\\nu_k^2 (\\alpha_k + L_f \\beta_k \\lambda_{k,+})^2}, \\ee\n%\nfor all $k \\in \\N$ and some $\\bar{\\rho} \\in (0,1)$. Then, under the additional conditions \n%\n\\be \\label{eq:choice-step-2} \\sum \\lambda_k = \\infty, \\quad \\sum \\lambda_k \\sigma_k^2 < \\infty, \\quad \\sum \\lambda_{k,+} \\sigma_{k,+}^2 < \\infty \\ee\n%\nit follows $\\liminf_{k \\to \\infty} \\Exp[\\|F^I({\\sf X}^k)\\|^2] = 0$ and $\\liminf_{k \\to \\infty} \\| F^I({\\sf X}^k) \\| = 0$ \\as and $(\\psi({\\sf X}^k))_k$ \\as converges to a random variable $\\sf Y^*$ with $\\lim_{k\\to \\infty} \\Exp[\\psi({\\sf X}^k)] = \\Exp[{\\sf Y}^*]$.  \n\\end{theorem}","sha256_utf8":"9fc3fb7aafa82ec08f07808e05d5c455e1e8d2c95972d4c9cf02a46c9dc2adab"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"全局收敛完整证明","source_units":[{"start":68015,"end":72492,"start_line":648,"end_line":674,"raw_tex":"\\begin{proof} First, due to the law of total expectation and $\\mathcal F^{k-1}_+ \\subset \\mathcal F^k$ and ${\\sf X}^k,{\\sf D}^k, {\\sf Z}^k \\in \\mathcal F^k$, it follows\n%\n\\begin{align*} \\Exp[\\iprod{\\nabla f({\\sf Z}^k) - {\\sf V}^k_+}{\\lambda_{k,+}(\\nabla f({\\sf X}^k) - \\nabla f({\\sf Z}^k)) +\\alpha_k {\\sf D}^k} \\mid \\mathcal F^{k-1}_+] & \\\\ & \\hspace{-55ex} = \\Exp[\\iprod{\\Exp[\\nabla f({\\sf Z}^k) - {\\sf V}^k_+ \\mid \\mathcal F^k]}{\\lambda_{k,+}(\\nabla f({\\sf X}^k) - \\nabla f({\\sf Z}^k)) +\\alpha_k {\\sf D}^k} \\mid \\mathcal F^{k-1}_+] = 0, \\end{align*}\n%\n$\\text{a.e.}$, where we used the fact that ${\\sf V}^k_+$ is an unbiased estimator of the gradient $\\nabla f({\\sf Z}^k)$. Next, taking conditional expectation in Lemma \\ref{lemma:prox-descent}, applying assumptions (B.2) and (B.3), and choosing $\\rho = \\lambda_k^{-1}$, we obtain\n%\n\\begin{align*} \\Exp[\\psi({\\sf X}^{k+1}) \\mid \\mathcal F^{k-1}_+] - \\psi({\\sf X}^k) & \\\\ & \\hspace{-24ex} \\leq  \\frac{\\lambda_{k,+}\\sigma_{k,+}^2}{2} + \\half  \\left[ L_f - \\frac{1}{\\lambda_{k,+}} \\right] \\Exp [\\|{\\sf X}^{k+1} - {\\sf X}^k\\|^2\\mid \\mathcal F^{k-1}_+ ] - \\frac{1}{2\\lambda_k} \\|F^{\\Lambda_k}({\\sf X}^k)\\|^2 \\\\ & \\hspace{-20ex} + \\frac{\\lambda_k \\sigma_k^2}{2}  + \\half \\left[ \\frac{1}{\\lambda_{k,+}} - \\frac{1}{\\lambda_k} + \\nu_k^2 \\cdot \\ell(\\lambda_{k,+},\\alpha_k,\\beta_k) \\right] \\Exp[\\|F^{\\Lambda_k}_{{\\sf V}^k}({\\sf X}^k)\\|^2 \\mid \\mathcal F^{k-1}_+] \n\\end{align*}\n%\nalmost everywhere. Utilizing the bounds stated in \\eqref{eq:choice-step-1}, we have\n%\n\\begin{align} \\nonumber \\Exp[\\psi({\\sf X}^{k+1}) \\mid \\mathcal F^{k-1}_+] - \\psi({\\sf X}^k) & \\\\  \\label{eq:est-martingale} & \\hspace{-20ex} \\leq \\frac{\\lambda_k \\sigma_k^2}{2} + \\frac{\\lambda_{k,+}\\sigma_{k,+}^2}{2} - \\frac{1}{2\\lambda_k} \\|F^{\\Lambda_k}({\\sf X}^k)\\|^2 - \\frac{\\bar{\\rho}}{2\\lambda_k} \\Exp[ \\|F^{\\Lambda_k}_{{\\sf V}^k}({\\sf X}^k)\\|^2 \\mid \\mathcal F^{k-1}_+ ].  \\end{align}\n%\nNow, by assumption (A.2) there exists $\\psi^* \\in \\R$ such that $\\psi(x) \\geq \\psi^*$ for all $x \\in \\Rn$. Taking expectation and summing the latter inequality, we get\n%\n\\begin{align*} \\sum_{k= 0}^R \\frac{\\Exp[\\|F^{\\Lambda_k}({\\sf X}^k)\\|^2 + \\bar{\\rho} \\cdot \\|F^{\\Lambda_k}_{{\\sf V}^k}({\\sf X}^k)\\|^2]}{\\lambda_k} \\leq 2 [\\psi(x^0) - \\psi^*] + \\sum_{k = 0}^R \\lambda_k \\sigma_k^2 + \\lambda_{k,+}\\sigma_{k,+}^2 \\end{align*}\n%\nfor all $R \\in \\N$. Due to $L_f \\geq 1$, it holds $\\lambda_{k,+} \\leq 1$, $\\lambda_k \\leq 1$ and by \\cite[Lemma 2]{Nes13}, we know that the mapping $\\delta \\mapsto {\\delta}^{-1} \\|F^{\\frac{1}{\\delta}I}(x)\\|$ is a decreasing function of $\\delta$. This yields $\\Exp[\\|F^I({\\sf X}^k)\\|^2] \\leq  \\lambda_k^{-2} \\Exp[ \\|F^{\\Lambda_k}({\\sf X}^k)\\|^2]$ for all $k \\in \\N$ and thus, we can infer  $\\sum \\lambda_k \\Exp[\\|F^I({\\sf X}^k)\\|^2] < \\infty$. Hence, the first statement in Theorem \\ref{theorem:conv-gen} follows from the assumption $\\sum \\lambda_k = \\infty$. Furthermore, applying the Borel-Cantelli lemma, it holds that $\\sum \\lambda_k \\|F^I({\\sf X}^k)\\|^2 < \\infty$ a.s., which finally implies $\\liminf_{k\\to \\infty} \\| F^I({\\sf X}^k) \\| = 0$ with probability 1. The third claim in Theorem \\ref{theorem:conv-gen} is a direct consequence of assumption (A.2), \\eqref{eq:est-martingale}, and Theorem \\ref{theorem:conv-superm}. In order to establish convergence of $(\\Exp[\\psi({\\sf X}^k)])_k$, we first notice that we have \n%\n\\begin{align} \\nonumber \\psi({\\sf X}^{k+1}) & \\leq \\psi({\\sf X}^k) + \\frac{\\lambda_k}{2} \\|\\nabla f({\\sf X}^k) - {\\sf V}^k\\|^2  \\\\ \\label{eq:des-later}  & \\hspace{3ex}  +\\lambda_{k,+} \\|\\nabla f({\\sf Z}^k) - {\\sf V}_+^k\\|^2 + (\\nu_k^2\\lambda_k)^{-1}\\|{\\sf D}^k\\|^2, \\end{align}\n%\n(almost) surely and for all $k \\in \\N$, which follows from Lemma \\ref{lemma:prox-descent}, $\\iprod{a}{b} \\leq \\half \\|a\\|^2 + \\half \\|b\\|^2$, $a,b \\in \\Rn$, $\\lambda_{k,+}^{-1} \\leq \\lambda_k^{-1}$, and $\\ell(\\lambda_{k,+},\\alpha_k,\\beta_k) < \\nu_k^{-2}(\\lambda_k^{-1} - \\lambda_{k,+}^{-1}) \\leq (\\nu_k^2 \\lambda_k)^{-1}$. By induction, this shows that the nonnegative random variable $\\psi({\\sf X}^k) - \\psi^*$ is (almost) surely dominated by \n%\n\\[ {\\sf U} := \\psi(x^0) - \\psi^* + \\sum_{k=0}^\\infty  \\frac{\\lambda_k}{2} \\|\\nabla f({\\sf X}^k) - {\\sf V}^k\\|^2 + \\lambda_{k,+} \\|\\nabla f({\\sf Z}^k) - {\\sf V}_+^k\\|^2 + \\frac{\\|{\\sf D}^k\\|^2}{\\nu_k^2\\lambda_k}   \\]\n%\nfor all $k \\in \\N$. Moreover due to Fatou's lemma, we have $\\Exp[{\\sf U}] < \\infty$. Hence, our last claim is a consequence of the dominated convergence theorem. \n\\end{proof}","sha256_utf8":"13ffbbb72fcf21f586155ad9a7b62697425d488358df93f0c13f4e7fd1cf5aab"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"非凸复合优化问题","source_units":[{"start":9578,"end":9643,"start_line":258,"end_line":258,"raw_tex":"\\be \\label{eq:prob} \\min_{x \\in \\Rn}~\\psi(x) := f(x) + \\vp(x),\\ee","sha256_utf8":"19a045ec8effdba1798dfd790cf78fffabf21c1ff436557670e77e05d12d40bc"},{"start":9646,"end":10169,"start_line":260,"end_line":260,"raw_tex":"where $f : \\Rn \\to \\R$ is a continuously differentiable but not necessarily convex function and $\\vp : \\Rn \\to \\Rex$ is a convex, lower semicontinuous, and proper mapping. We assume that a full evaluation of the function $f$ and its gradient $\\nabla f$ is either too expensive or not  possible and that approximate gradient information can only be accessed via calling \\textit{stochastic oracles}. %which give access to stochastic approximations of gradient $\\nabla f$ and propose a general stochastic two-step framework. \n","sha256_utf8":"b72e2c6fd95d167abbce02cb9aa463438f7edc608b547bcd40a775edce4d1512"}]},{"id":"S004","semantic_type":"notation","kind":"notation","title":"范数与随机变量记号","source_units":[{"start":30383,"end":32206,"start_line":378,"end_line":380,"raw_tex":"By $\\iprod{\\cdot}{\\cdot}$ and $\\|\\cdot\\| := \\|\\cdot\\|_2$ we denote the standard Euclidean inner product and norm. For matrices, the norm $\\|\\cdot\\|$ is the standard spectral norm. The set of symmetric and positive definite $n \\times n$ matrices is denoted by $\\Spp$. For a given matrix $\\Lambda \\in \\Spp$, we define the inner product $\\iprod{x}{y}_\\Lambda := \\iprod{x}{\\Lambda y} = \\iprod{\\Lambda x}{y}$ and $\\|x\\|_\\Lambda := \\sqrt{\\iprod{x}{x}_\\Lambda}$.\n%$|S|$ represents the cardinality of a finite set $S$. If $S$ is a subset of $\\{1,2,\\dots,n\\},$ then for any vector $x\\in \\R^n$ and matrix $H\\in \\R^{n\\times n}$, $x_{S}$ is the subvector of $x$ whose entries lie in $x$ indexed by $S,$ and $H_{SS}$ represents the submatrix of $H$ whose rows and columns lie in $S$. \nFor any $n \\in \\N$, we set $[n] := \\{1,...,n\\}$ and $[n]_0 = \\{0\\} \\cup [n]$. Let $(\\Omega,\\mathcal F,\\Prob)$ be a probability space. We will use uppercase letters and a sans-serif letterform to describe random variables ${\\sf X} : \\Omega \\to \\Rn$, while lowercase letters or letters with serifs are typically reserved for realizations of a random variable, $x = {\\sf X}(\\omega)$, or deterministic parameters. We use $L^p(\\Omega) := L^p(\\Omega,\\Prob)$, $p \\in [1,\\infty]$, to denote the standard $L^p$ space on $\\Omega$. We write ${\\sf X} \\in \\mathcal F$ for ``${\\sf X}$ is $\\mathcal F$-measurable''. Moreover, $\\sigma({\\sf X}^1,...,{\\sf X}^k)$ denotes the $\\sigma$-algebra generated by the family of random variables ${\\sf X}^1,...,{\\sf X}^k$. For a random variable ${\\sf X} \\in L^1(\\Omega)$ and a sub-$\\sigma$-algebra $\\cH \\subseteq \\mathcal F$, the conditional expectation of ${\\sf X}$ given $\\cH$ is denoted by $\\Exp[{\\sf X} \\mid \\cH]$. We use the abbreviations ``a.e.'' and ``a.s.'' for ``almost everywhere'' and ``almost surely'', respectively.\n","sha256_utf8":"d21f80d098d7206e9937a665efe6c6a8275a75bb377392e5de7f84f2a3c60fca"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"近端算子与驻点残差","source_units":[{"start":34910,"end":35313,"start_line":401,"end_line":403,"raw_tex":"\\subsection{Preliminaries and First Order Optimality} It is well-known that the stationarity conditions of problem \\eqref{eq:prob} can be equivalently rewritten as a proximal fixed-point equation, i.e., a point $x \\in \\dom~\\vp$ is a critical point of \\eqref{eq:prob} if and only if\n%\n\\be \\label{eq:opt} F^\\Lambda(x) := x - \\proxt{\\Lambda}{\\vp}(x-\\Lambda^{-1}\\nabla f(x)) = 0, \\quad \\Lambda \\in \\Spp, \\ee","sha256_utf8":"547d16a181e6ecfced30993aa0d1e54494dafd504e1b5249913214ed23fa70ab"},{"start":35316,"end":35614,"start_line":405,"end_line":405,"raw_tex":"where $\\proxt{\\Lambda}{\\vp} : \\Rn \\to \\Rn$, $\\proxt{\\Lambda}{\\vp}(x) := \\argmin_y \\vp(y) + \\frac{1}{2}\\|x-y\\|^2_\\Lambda$ denotes the proximity operator of the mapping $\\vp$. The proximity operator is a $\\Lambda$-firmly nonexpansive function, i.e., it is globally Lipschitz continuous and satisfies\n","sha256_utf8":"4213ae6402a890cb2646dd2f1ed7a820e12fb9ea81a428c6686eff4a44842463"}]},{"id":"S006","semantic_type":"derived_fact","kind":"derived_fact","title":"近端算子的基本性质","source_units":[{"start":35614,"end":35807,"start_line":406,"end_line":407,"raw_tex":"%\n\\be \\label{eq:prox-nonexp} \\|\\proxt{\\Lambda}{\\vp}(x) - \\proxt{\\Lambda}{\\vp}(y)\\|_\\Lambda^2 \\leq \\iprod{\\Lambda(x-y)}{\\proxt{\\Lambda}{\\vp}(x) - \\proxt{\\Lambda}{\\vp}(y)}, \\quad \\forall~x,y. \\ee","sha256_utf8":"e3334cd98841efab11cf17768ee81f2f15dbf6a7c7a95e2d6782049c3b32cc61"},{"start":35810,"end":36049,"start_line":409,"end_line":411,"raw_tex":"The proximity operator $\\proxt{\\Lambda}{\\vp}$ can also be alternatively characterized via the underlying optimality condition\n%\n\\be \\label{eq:prox-opt} \\proxt{\\Lambda}{\\vp}(x) \\in x - \\Lambda^{-1} \\partial \\vp(\\proxt{\\Lambda}{\\vp}(x)), \\ee","sha256_utf8":"55b17187727e95ad46aae0ba579246476bfd51ea3cae4398ed912345d2ed87ec"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"随机残差与近端步","source_units":[{"start":37490,"end":37770,"start_line":426,"end_line":430,"raw_tex":"Given a generic (stochastic) oracle represented by a direction $v \\in \\Rn$, we consider inexact variants of the nonsmooth residual \\eqref{eq:opt} of the type\n%\n\\begin{equation}\n\\label{sto_app_optimal}  F^\\Lambda_v(x) := x - \\proxt{\\Lambda}{\\vp}(x- \\Lambda^{-1} v). \n\\end{equation}","sha256_utf8":"4ccb9346a913b1df22ae5c2d26299656c5f20be7a61e83b8933a66cff6323aa2"},{"start":37773,"end":38084,"start_line":432,"end_line":432,"raw_tex":"We will also use $u^\\Lambda_v(x) := x - \\Lambda^{-1} v$ and $p^\\Lambda_v := \\proxt{\\Lambda}{\\vp}(u^\\Lambda_v(x))$ to denote the inexact (proximal) gradient steps. In the case $v \\equiv \\nabla f(x)$, the terms $u^\\Lambda$ and $p^\\Lambda$ are used to denote the exact or full gradient and proximal gradient step.\n","sha256_utf8":"858473b0e9bb30140e2e473d0c9446de8873084db9834bbb44cb1748c6709ea5"}]},{"id":"S008","semantic_type":"algorithm","kind":"algorithm","title":"随机额外步算法","source_units":[{"start":39788,"end":40539,"start_line":449,"end_line":458,"raw_tex":"\\begin{algorithm2e}[t]\n\\caption{\\textbf{S}tochastic \\textbf{E}xtra-Step \\textbf{Q}uasi-\\textbf{N}ewton Method}    \n\\label{alg:seqn}\n\\lnlset{alg:seqn-1}{1}{Initialization: ~~Select the parameter matrices $(\\Lambda_k)_k, (\\Lambda_{k,+})_k \\in \\Spp$. Choose the step sizes $(\\alpha_k)_k$, $(\\beta_k)_k$ and the initial point  $x^0 \\in \\dom~\\vp$ .} \\\\ \n\\For{$k = 0,1,... $}{\n\\lnlset{alg:seqn-2}{2}{Based on $v^k \\approx \\nabla f(x^k)$, compute a direction $d^k$ and set $z^k = x^k + \\beta_k d^k$.} \\\\\n\\lnlset{alg:seqn-3}{3}{Calculate the new oracle $v^k_+ \\approx \\nabla f(z^k)$ and perform the update \\\\\n\\begin{center} $x^{k+1} = \\proxt{\\Lambda_{k,+}}{\\vp}(x^k + \\alpha_k d^k - \\Lambda_{k,+}^{-1}v^k_+)$ \\vspace{-1ex}.\\end{center}}  \n} \n\\end{algorithm2e}","sha256_utf8":"e800588fec90656e1af1d1658e57e7a6cd2e2e57b047e0175f272195ffaab7c0"}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","title":"梯度光滑性与目标下界","source_units":[{"start":51244,"end":52145,"start_line":509,"end_line":517,"raw_tex":"\\begin{assumption} \\label{ass:lip} Let $f : \\Rn \\to \\R$ be given. We assume: %\\vspace{.5ex}\n\\setlength{\\leftmargini}{7ex}\n\\begin{itemize}\n\\item[{\\rm(A.1)}] The gradient mapping $\\nabla f$ is Lipschitzian on $\\Rn$ with modulus $L_f \\geq 1$. %Moreover, each of the component functions $\\nabla f_i$, $i \\in [N]$, is $L_i$-Lipschitz continuous.\\vspace{.5ex} \n\\item[{\\rm(A.2)}] The objective function $\\psi$ is bounded from below on $\\dom~\\vp$. \\vspace{.5ex}\n%\\item[{\\rm(A.3)}] There exist parameters $\\mu_f \\in \\R$ and $\\mu_\\vp \\geq 0$ with $\\bar\\mu := \\mu_f + \\mu_\\vp > 0$ such that the shifted functions $f - \\frac{\\mu_f}{2}\\|\\cdot\\|^2$ and $\\vp - \\frac{\\mu_\\vp}{2}\\|\\cdot\\|^2$ are convex. \\vspace{.5ex}\n%\\item[{\\rm(A.4)}] There exists a constant $\\bar g_r > 0$ such that for all $x \\in \\dom~r$ there exists $\\lambda \\in \\partial r(x)$ with $\\|\\lambda\\| \\leq \\bar g_r$.\n\\end{itemize}\n\\end{assumption}","sha256_utf8":"401502473b770a2b92d4318b14e416f15816a800e9f899b5baa5c7df8cc06a3d"}]},{"id":"S010","semantic_type":"derived_fact","kind":"derived_fact","title":"光滑函数下降估计","source_units":[{"start":54441,"end":54565,"start_line":544,"end_line":544,"raw_tex":"\\be \\label{eq:lip-ineq} f(y) \\leq f(x) + \\iprod{\\nabla f(x)}{y-x} + \\frac{L_f}{2}  \\|y-x\\|^2, \\quad \\forall~x,y \\in \\Rn, \\ee","sha256_utf8":"7ab269e46123e916e9aed4c12656def89484e97de9db6a7d2db0950a68f14363"},{"start":54570,"end":54626,"start_line":546,"end_line":546,"raw_tex":"which is a direct consequence of assumption {\\rm(A.1)}.\n","sha256_utf8":"728c7ad45383552f249a02ce1790feeb4899fbab9c0360dd3bff10e5c3911546"}]},{"id":"S011","semantic_type":"notation","kind":"notation","title":"随机预言机与滤过","source_units":[{"start":63401,"end":64310,"start_line":612,"end_line":616,"raw_tex":"The stochasticity in Algorithm \\ref{alg:seqn} mainly results from the random selection of the oracles $v^k$ and $v^k_+$ in step \\ref{alg:seqn-2} and \\ref{alg:seqn-3}. In this work, we assume that the stochastic approximations $v^k$ and $v^k_+$ correspond to realizations of the random vectors ${\\sf V}^k : \\Omega \\to \\Rn$ and ${\\sf V}^k_+ : \\Omega \\to \\Rn$. Furthermore, we suppose that the underlying probability space $(\\Omega,\\mathcal F,\\Prob)$ is sufficiently rich allowing us to model the involved random processes in a unified way. We now define the filtrations\n%\n\\[ \\mathcal F^k := \\sigma({\\sf V}^0, {\\sf V}^0_+, ..., {\\sf V}^k), \\quad \\mathcal F^k_+ := \\sigma(\\mathcal F^k \\cup \\sigma({\\sf V}^k_+)). \\]\n%\nBy convention, let $({\\sf D}^k)_k$ denote the stochastic process associated with the directions $(d^k)_k$ chosen in step \\ref{alg:seqn-2}. We will work with the following stochastic assumptions. \n","sha256_utf8":"9547a0e785bfad38f19697fe09546de3c9a6d25dac75ca18e11a446a7ae45b86"}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","title":"方向控制与无偏方差条件","source_units":[{"start":64311,"end":65652,"start_line":618,"end_line":629,"raw_tex":"\\begin{assumption} \\label{ass:stoch-1} We assume: \n\\setlength{\\leftmargini}{7ex}\n\\begin{itemize}\n\\item[{\\rm(B.1)}] The mapping ${\\sf D}^k : \\Omega \\to \\R^{n}$ is an $\\mathcal F^k$-measurable function for all $k$. \\vspace{.5ex} \n\\item[{\\rm(B.2)}] There is $\\nu_k > 0$ such that we have $\\Exp[\\|{\\sf D}^k\\|^2 \\mid \\mathcal F^{k-1}_+] \\leq \\nu_k^2 \\cdot \\Exp[\\|F^{\\Lambda_k}_{{\\sf V}^k}({\\sf X}^k)\\|^2 \\mid \\mathcal F^{k-1}_+]$ \\aev and for all $k \\in \\N$.\n\\item[{\\rm(B.3)}] For all $k \\in \\N$, it holds $\\Exp[{\\sf V}^k \\mid \\mathcal F^{k-1}_+] = \\nabla f({\\sf X}^k)$, $\\Exp[{\\sf V}^k_+ \\mid \\mathcal F^{k}] = \\nabla f({\\sf Z}^k)$ \\aev and there exists $\\sigma_k, \\sigma_{k,+} > 0$ such that \\aev \n%\\item[{\\rm(A.3)}] There exist parameters $\\mu_f \\in \\R$ and $\\mu_\\vp \\geq 0$ with $\\bar\\mu := \\mu_f + \\mu_\\vp > 0$ such that the shifted functions $f - \\frac{\\mu_f}{2}\\|\\cdot\\|^2$ and $\\vp - \\frac{\\mu_\\vp}{2}\\|\\cdot\\|^2$ are convex. \\vspace{.5ex}\n%\\item[{\\rm(A.4)}] There exists a constant $\\bar g_r > 0$ such that for all $x \\in \\dom~r$ there exists $\\lambda \\in \\partial r(x)$ with $\\|\\lambda\\| \\leq \\bar g_r$.\n\\end{itemize}\n%\n\\[ \\Exp[\\|\\nabla f({\\sf X}^k) - {\\sf V}^k\\|^2 \\mid \\mathcal F_+^{k-1}] \\leq \\sigma_k^2 \\quad \\text{and} \\quad \\Exp[\\|\\nabla f({\\sf Z}^k) - {\\sf V}_+^k\\|^2 \\mid \\mathcal F^{k}] \\leq \\sigma_{k,+}^2. \\]\n\\end{assumption}","sha256_utf8":"ebd05ca1a31559d2f2e57b282dd728e1f0f428c35cb22e7106e82db2e10ad100"}]},{"id":"S013","semantic_type":"derived_fact","kind":"derived_fact","title":"迭代点的可测性","source_units":[{"start":66636,"end":66778,"start_line":633,"end_line":633,"raw_tex":"\\be \\label{eq:prop-stoch} {\\sf Z}^k \\in \\mathcal F^k \\quad \\text{and} \\quad {\\sf X}^{k+1} \\in {\\mathcal F}^{k}_+ , \\quad \\forall~k \\geq 0. \\ee","sha256_utf8":"c7a9607952be50c59d7fd1fcb4830885e799b4fbc06d630b539e022e4359750c"}]},{"id":"S014","semantic_type":"lemma","kind":"lemma","title":"额外步近似下降引理","source_units":[{"start":54627,"end":55773,"start_line":548,"end_line":553,"raw_tex":"\\begin{lemma} \\label{lemma:prox-descent} Let $x \\in \\dom~\\vp$, $d \\in \\Rn$, $\\Lambda = \\lambda^{-1} I $, $\\Lambda_+ = \\lambda_+^{-1} I$, $\\alpha, \\beta, \\rho > 0$, and the approximations $v, v_+ \\in \\Rn$ be given. Under assumption {\\rm(A.1)} it holds that\n%\n\\begin{align*} 2[\\psi(p_+) - \\psi(x)] & \\\\ & \\hspace{-13.5ex} \\leq \\frac{1}{\\rho} \\|\\nabla f(x) - v\\|^2 +  {\\lambda_+} \\|\\nabla f(x+\\beta d) - v_+\\|^2 +  \\left[ \\frac{1}{\\lambda_+} - \\frac{1}{\\lambda} \\right]  \\|F^\\Lambda_v(x)\\|^2 \\\\ & \\hspace{-11.5ex} + \\left[ L_f - \\frac{1}{\\lambda_+}\\right] \\|p_+ - x\\|^2  +  \\left[ \\rho - \\frac{1}{\\lambda} \\right]  \\|p^\\Lambda_v(x) - p^\\Lambda(x)\\|^2 + {\\ell(\\lambda_+,\\alpha,\\beta)} \\|d\\|^2   \\\\ & \\hspace{-11.5ex} - \\frac{1}{\\lambda} \\|F^\\Lambda(x)\\|^2 +  2\\iprod{\\nabla f(x+\\beta d) - v_+}{\\lambda_+(\\nabla f(x) - \\nabla f(x+\\beta d)) +\\alpha d}, \\end{align*}\n%\nwhere $p_+ := \\proxt{\\Lambda_+}{\\vp}(x+\\alpha d - \\lambda_+v_+)$ and $\\ell(\\lambda_+,\\alpha,\\beta) := \\lambda_+(\\frac{\\alpha}{\\lambda_+} + L_f\\beta)^2$. % $\\ell_\\alpha(z) := \\frac{L}{2}(\\alpha \\|W\\| + \\|I-\\alpha W\\|)^2 + \\left[z - \\frac{L}{2}\\right ] \\|I-\\alpha W\\|^2$.\n\\end{lemma}","sha256_utf8":"07e5a7ec649f1d21b7d9e6b7eb8b2ac195b9ddb6acc64a6d0bcede8a874d5cd1"}]},{"id":"S015","semantic_type":"proof","kind":"proof","title":"近似下降引理完整证明","source_units":[{"start":55774,"end":58895,"start_line":554,"end_line":575,"raw_tex":"\\begin{proof} First, setting $p = \\proxt{\\Gamma}{\\vp}(y+u-\\Gamma^{-1}w)$ and applying the optimality condition of the proximity operator \\eqref{eq:prox-opt}, we have\n%\n\\begin{align*} \\vp(p) - \\vp(z) \\leq \\iprod{\\Gamma u - w}{p-z} + \\half \\|y-z\\|^2_\\Gamma - \\half \\|p-y\\|^2_\\Gamma - \\half \\|p-z\\|_\\Gamma^2 \\end{align*}\n%\\be \\label{eq:vp-prox} \\vp(\\proxt{\\Gamma}{\\vp}(y)) - \\vp(z) \\leq \\iprod{\\Gamma(y - \\proxt{\\Gamma}{\\vp}(y))}{\\proxt{\\Gamma}{\\vp}(y) - z}, \\quad \\Gamma \\in \\Spp, \\ee \n%\nfor all $y,u,w \\in \\Rn$, $z \\in \\dom~\\vp$, and $\\Gamma \\in \\Spp$. Utilizing this estimate three times for different choices of $y$, $z \\in \\dom~\\vp$, and $\\Gamma$ and applying the descent lemma \\eqref{eq:lip-ineq}, Young's inequality, and the Lipschitz continuity of $\\nabla f$, we obtain\n%\n\\begingroup\n\\allowdisplaybreaks\n\\begin{align*} \\psi(p_+) - \\psi(x) & \\\\& \\hspace{-12ex} = f(p_+) - f(x) + \\vp(p_+) - \\vp(p^\\Lambda_v(x)) \\\\ & \\hspace{-9ex}  + \\vp(p^\\Lambda_v(x)) - \\vp(p^\\Lambda(x)) +\\vp(p^\\Lambda(x)) - \\vp(x) \\\\ & \\hspace{-12ex} \\leq  \\iprod{\\nabla f(x)}{p_+-p^\\Lambda_v(x)} + \\frac{L_f}{2}  \\|p_+-x\\|^2 + \\iprod{\\alpha \\Lambda_+ d - v_+}{p_+ - p^\\Lambda_v(x)} \\\\ & \\hspace{-9ex} + \\half \\left[ \\frac{1}{\\lambda_+} - \\frac{1}{\\lambda} \\right] \\|F^\\Lambda_v(x)\\|^2 - \\frac{1}{2\\lambda_+} \\|p_+ - x\\|^2 - \\frac{1}{2\\lambda_+} \\|p_+ - p^\\Lambda_v(x)\\|^2 \\\\ & \\hspace{-9ex} + \\iprod{\\nabla f(x)-v}{p^\\Lambda_v(x) - p^\\Lambda(x)} - \\frac{1}{2\\lambda} \\left[ \\|p^\\Lambda_v(x) - p^\\Lambda(x)\\|^2 + \\|F^\\Lambda(x)\\|^2 \\right] \\\\ & \\hspace{-12ex} \\leq  \\frac{\\| \\nabla f(x) - v\\|^2}{2\\rho}  +  \\left[ \\frac{\\rho}{2} - \\frac{1}{2\\lambda}\\right] \\|p^\\Lambda_v(x) - p^\\Lambda(x) \\|^2 +  \\left[ \\frac{L_f}{2} - \\frac{1}{2\\lambda_+} \\right] \\|p_+-x\\|^2   \\\\  &\\hspace{-9ex}  + \\frac12\\left[\\frac{1}{\\lambda_+} - \\frac{1}{\\lambda}  \\right] \\|F^\\Lambda_v(x)\\|^2 - \\frac{1}{2\\lambda} \\|F^\\Lambda(x)\\|^2  + \\frac{\\lambda_+}{2} \\|\\nabla f(x) + \\lambda_+^{-1}\\alpha d - v_+\\|^2. \\end{align*}\n%\n%\\\\ & \\hspace{-9ex} + \\iprod{\\Lambda F^\\Lambda_v(x) - v}{p^\\Lambda_v(x) - p^\\Lambda(x)} - \\iprod{\\Lambda F^\\Lambda(x) - \\nabla f(x)}{x-p^\\Lambda(x)}  \\\\ & \\hspace{-12ex} = \\iprod{\\nabla f(x) - v}{p^\\Lambda_v(x) - p^\\Lambda(x)} + \\iprod{\\alpha\\lambda_+^{-1}d + \\nabla f(x) - v_+}{p_+ - p^\\Lambda_v(x)} \\\\ & \\hspace{-9ex} +  \\left[ \\frac{L_f}{2} - \\frac{1}{2\\lambda_+} \\right] \\|p_+ - x\\|^2 - \\frac{1}{2\\lambda_+} \\|p_+ - p^\\Lambda_v(x)\\|^2 - \\frac{1}{2\\lambda} \\|F^\\Lambda(x)\\|^2  \\\\ & \\hspace{-9ex}   - \\frac{1}{2\\lambda} \\|p^\\Lambda_v(x) - p^\\Lambda(x)\\|^2 + \\frac12 \\left[ \\frac{1}{\\lambda_+} - \\frac{1}{\\lambda} \\right] \\|F^\\Lambda_v(x)\\|^2  \n\\endgroup\n%\nExtending the penultimate term by adding and subtracting $\\nabla f(x+\\beta d)$ and using\n%\n\\[  \\| \\nabla f(x) - \\nabla f(x+\\beta d) + \\lambda_+^{-1}\\alpha d \\|^2 \\leq \\left( \\frac{\\alpha}{\\lambda_+} + L_f\\beta \\right)^2 \\|d\\|^2 = \\frac{\\ell(\\lambda_+,\\alpha,\\beta)}{\\lambda_+} \\|d\\|^2\n\\]\n%\nwe finally get the estimate stated in Lemma \\ref{lemma:prox-descent}. \n%Since the first term is nonpositive for all $\\alpha \\leq \\overline\\alpha$, this establishes \\eqref{eq:est-prox-step}.\n\\end{proof}","sha256_utf8":"5e8453f8b38e4dee8125d83f69c6bff70e649dc82e3bff520b3eb3ab0ce78eb2"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"建立近端三点估计","source_units":[{"start":55774,"end":56549,"start_line":554,"end_line":559,"raw_tex":"\\begin{proof} First, setting $p = \\proxt{\\Gamma}{\\vp}(y+u-\\Gamma^{-1}w)$ and applying the optimality condition of the proximity operator \\eqref{eq:prox-opt}, we have\n%\n\\begin{align*} \\vp(p) - \\vp(z) \\leq \\iprod{\\Gamma u - w}{p-z} + \\half \\|y-z\\|^2_\\Gamma - \\half \\|p-y\\|^2_\\Gamma - \\half \\|p-z\\|_\\Gamma^2 \\end{align*}\n%\\be \\label{eq:vp-prox} \\vp(\\proxt{\\Gamma}{\\vp}(y)) - \\vp(z) \\leq \\iprod{\\Gamma(y - \\proxt{\\Gamma}{\\vp}(y))}{\\proxt{\\Gamma}{\\vp}(y) - z}, \\quad \\Gamma \\in \\Spp, \\ee \n%\nfor all $y,u,w \\in \\Rn$, $z \\in \\dom~\\vp$, and $\\Gamma \\in \\Spp$. Utilizing this estimate three times for different choices of $y$, $z \\in \\dom~\\vp$, and $\\Gamma$ and applying the descent lemma \\eqref{eq:lip-ineq}, Young's inequality, and the Lipschitz continuity of $\\nabla f$, we obtain\n","sha256_utf8":"c6780617cd8a6ebc13af8c42d5e9b112b7d5aabf1d81f38dd6479fb1fbf1e98f"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"合并三次近端估计","source_units":[{"start":56551,"end":58402,"start_line":561,"end_line":566,"raw_tex":"\\begingroup\n\\allowdisplaybreaks\n\\begin{align*} \\psi(p_+) - \\psi(x) & \\\\& \\hspace{-12ex} = f(p_+) - f(x) + \\vp(p_+) - \\vp(p^\\Lambda_v(x)) \\\\ & \\hspace{-9ex}  + \\vp(p^\\Lambda_v(x)) - \\vp(p^\\Lambda(x)) +\\vp(p^\\Lambda(x)) - \\vp(x) \\\\ & \\hspace{-12ex} \\leq  \\iprod{\\nabla f(x)}{p_+-p^\\Lambda_v(x)} + \\frac{L_f}{2}  \\|p_+-x\\|^2 + \\iprod{\\alpha \\Lambda_+ d - v_+}{p_+ - p^\\Lambda_v(x)} \\\\ & \\hspace{-9ex} + \\half \\left[ \\frac{1}{\\lambda_+} - \\frac{1}{\\lambda} \\right] \\|F^\\Lambda_v(x)\\|^2 - \\frac{1}{2\\lambda_+} \\|p_+ - x\\|^2 - \\frac{1}{2\\lambda_+} \\|p_+ - p^\\Lambda_v(x)\\|^2 \\\\ & \\hspace{-9ex} + \\iprod{\\nabla f(x)-v}{p^\\Lambda_v(x) - p^\\Lambda(x)} - \\frac{1}{2\\lambda} \\left[ \\|p^\\Lambda_v(x) - p^\\Lambda(x)\\|^2 + \\|F^\\Lambda(x)\\|^2 \\right] \\\\ & \\hspace{-12ex} \\leq  \\frac{\\| \\nabla f(x) - v\\|^2}{2\\rho}  +  \\left[ \\frac{\\rho}{2} - \\frac{1}{2\\lambda}\\right] \\|p^\\Lambda_v(x) - p^\\Lambda(x) \\|^2 +  \\left[ \\frac{L_f}{2} - \\frac{1}{2\\lambda_+} \\right] \\|p_+-x\\|^2   \\\\  &\\hspace{-9ex}  + \\frac12\\left[\\frac{1}{\\lambda_+} - \\frac{1}{\\lambda}  \\right] \\|F^\\Lambda_v(x)\\|^2 - \\frac{1}{2\\lambda} \\|F^\\Lambda(x)\\|^2  + \\frac{\\lambda_+}{2} \\|\\nabla f(x) + \\lambda_+^{-1}\\alpha d - v_+\\|^2. \\end{align*}\n%\n%\\\\ & \\hspace{-9ex} + \\iprod{\\Lambda F^\\Lambda_v(x) - v}{p^\\Lambda_v(x) - p^\\Lambda(x)} - \\iprod{\\Lambda F^\\Lambda(x) - \\nabla f(x)}{x-p^\\Lambda(x)}  \\\\ & \\hspace{-12ex} = \\iprod{\\nabla f(x) - v}{p^\\Lambda_v(x) - p^\\Lambda(x)} + \\iprod{\\alpha\\lambda_+^{-1}d + \\nabla f(x) - v_+}{p_+ - p^\\Lambda_v(x)} \\\\ & \\hspace{-9ex} +  \\left[ \\frac{L_f}{2} - \\frac{1}{2\\lambda_+} \\right] \\|p_+ - x\\|^2 - \\frac{1}{2\\lambda_+} \\|p_+ - p^\\Lambda_v(x)\\|^2 - \\frac{1}{2\\lambda} \\|F^\\Lambda(x)\\|^2  \\\\ & \\hspace{-9ex}   - \\frac{1}{2\\lambda} \\|p^\\Lambda_v(x) - p^\\Lambda(x)\\|^2 + \\frac12 \\left[ \\frac{1}{\\lambda_+} - \\frac{1}{\\lambda} \\right] \\|F^\\Lambda_v(x)\\|^2  \n\\endgroup\n","sha256_utf8":"0728aa3329793da9beb68eeb1a588dba7f0030073cde0827c9d60f12859e3270"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"展开试探梯度误差","source_units":[{"start":58404,"end":58895,"start_line":568,"end_line":575,"raw_tex":"Extending the penultimate term by adding and subtracting $\\nabla f(x+\\beta d)$ and using\n%\n\\[  \\| \\nabla f(x) - \\nabla f(x+\\beta d) + \\lambda_+^{-1}\\alpha d \\|^2 \\leq \\left( \\frac{\\alpha}{\\lambda_+} + L_f\\beta \\right)^2 \\|d\\|^2 = \\frac{\\ell(\\lambda_+,\\alpha,\\beta)}{\\lambda_+} \\|d\\|^2\n\\]\n%\nwe finally get the estimate stated in Lemma \\ref{lemma:prox-descent}. \n%Since the first term is nonpositive for all $\\alpha \\leq \\overline\\alpha$, this establishes \\eqref{eq:est-prox-step}.\n\\end{proof}","sha256_utf8":"033b08ceeef9b1d9ac6c432d3034a5597fe33b7396107e6bba0bab0799ed57f5"}]},{"id":"S019","semantic_type":"theorem","kind":"theorem","title":"超鞅收敛定理","source_units":[{"start":60880,"end":61379,"start_line":594,"end_line":594,"raw_tex":"\\begin{theorem} \\label{theorem:conv-superm} Let $({\\sf Y}_k)_k$, $({\\sf U}_k)_k$, $({\\sf A}_k)_k$, $({\\sf B}_k)_k$ be sequences of nonnegative integrable random variables, adapted to the filtration $(\\mathcal U_k)_k$ such that we have $\\Exp[{\\sf Y}_{k+1} \\mid \\mathcal U_k] \\leq (1+{\\sf A}_k) {\\sf Y}_k - {\\sf U}_k + {\\sf B}_k$, $\\sum {\\sf A}_k < \\infty$, and $\\sum {\\sf B}_k < \\infty$ \\as for all $k \\in \\N$. Then, $({\\sf Y}_k)_k$ \\as converges and it holds $\\sum {\\sf U}_k < \\infty$. \\end{theorem}","sha256_utf8":"70e255a32b4fc5d2587369ac7c4973710123eb53e5e7b20635e55c8a25447ed1"}]},{"id":"S020","semantic_type":"external_fact","kind":"external_fact","title":"罗宾斯西格蒙德定理","source_units":[],"declared_external":true,"declaration_text":["For nonnegative integrable adapted processes, E[Y_{k+1}|U_k] <= (1+A_k)Y_k-U_k+B_k together with almost sure summability of the nonnegative error sequences A_k and B_k implies almost sure convergence of Y_k and almost sure summability of the nonnegative decrement sequence U_k. This is the Robbins–Siegmund result cited as RobSie71 and restated in B0032."]},{"id":"S021","semantic_type":"external_fact","kind":"external_fact","title":"近端残差的步长单调性","source_units":[],"declared_external":true,"declaration_text":["For fixed x and δ > 0, δ ↦ δ^{-1}||F^{δ^{-1}I}(x)|| is nonincreasing. In particular, when 0 < λ <= 1, ||F^I(x)|| <= λ^{-1}||F^{λ^{-1}I}(x)||. This is the boundary fact cited as Nes13, Lemma 2."]},{"id":"S022","semantic_type":"external_fact","kind":"external_fact","title":"条件期望的塔式性质","source_units":[],"declared_external":true,"declaration_text":["Conditional expectation satisfies the tower property for nested sigma-algebras; a measurable factor can be pulled out of conditional expectation when the relevant quantities are integrable."]},{"id":"S023","semantic_type":"external_fact","kind":"external_fact","title":"内积与范数基本不等式","source_units":[],"declared_external":true,"declaration_text":["For Euclidean vectors and ρ > 0, 2<a,b> <= ρ||a||² + ρ^{-1}||b||²; the Cauchy–Schwarz and triangle inequalities also hold."]},{"id":"S024","semantic_type":"external_fact","kind":"external_fact","title":"随机级数与积分极限定理","source_units":[],"declared_external":true,"declaration_text":["For nonnegative random variables Z_k, sum_k E[Z_k] < infinity implies sum_k Z_k < infinity almost surely. Fatou's lemma controls the expectation of a nonnegative limit, and dominated convergence permits interchange of expectation and an almost sure limit under an integrable dominating variable."]},{"id":"S025","semantic_type":"assumption","kind":"assumption","title":"主定理的步长与可和条件","source_units":[{"start":66849,"end":67733,"start_line":637,"end_line":643,"raw_tex":"\\begin{theorem} \\label{theorem:conv-gen} Let the random process $({\\sf X}^k)_k$ be generated by Algorithm \\ref{alg:seqn} using parameter matrices of the form $\\Lambda_k = \\lambda_k^{-1} I$ and $\\Lambda_{k,+} = \\lambda_{k,+}^{-1} I$. Suppose that the assumptions {\\rm(A.1)}--{\\rm(A.2)} and {\\rm(B.1)}--{\\rm(B.3)} are satisfied and let us assume that the step sizes $(\\lambda_k)_k$, $(\\lambda_{k,+})_k$, $(\\alpha_k)_k$, and $(\\beta_k)_k$ are chosen as follows:   \n%\n\\be \\label{eq:choice-step-1} \\lambda_{k,+} \\leq \\frac{1}{L_f}, \\quad \\lambda_k \\leq \\frac{(1-\\bar{\\rho})\\lambda_{k,+}}{1+\\nu_k^2 (\\alpha_k + L_f \\beta_k \\lambda_{k,+})^2}, \\ee\n%\nfor all $k \\in \\N$ and some $\\bar{\\rho} \\in (0,1)$. Then, under the additional conditions \n%\n\\be \\label{eq:choice-step-2} \\sum \\lambda_k = \\infty, \\quad \\sum \\lambda_k \\sigma_k^2 < \\infty, \\quad \\sum \\lambda_{k,+} \\sigma_{k,+}^2 < \\infty \\ee\n","sha256_utf8":"a6e9910af320eebde6c23be29132454ef9a2bddbd02810ab0357f72338817eeb"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"消去交叉误差并取条件期望","source_units":[{"start":68015,"end":69433,"start_line":648,"end_line":655,"raw_tex":"\\begin{proof} First, due to the law of total expectation and $\\mathcal F^{k-1}_+ \\subset \\mathcal F^k$ and ${\\sf X}^k,{\\sf D}^k, {\\sf Z}^k \\in \\mathcal F^k$, it follows\n%\n\\begin{align*} \\Exp[\\iprod{\\nabla f({\\sf Z}^k) - {\\sf V}^k_+}{\\lambda_{k,+}(\\nabla f({\\sf X}^k) - \\nabla f({\\sf Z}^k)) +\\alpha_k {\\sf D}^k} \\mid \\mathcal F^{k-1}_+] & \\\\ & \\hspace{-55ex} = \\Exp[\\iprod{\\Exp[\\nabla f({\\sf Z}^k) - {\\sf V}^k_+ \\mid \\mathcal F^k]}{\\lambda_{k,+}(\\nabla f({\\sf X}^k) - \\nabla f({\\sf Z}^k)) +\\alpha_k {\\sf D}^k} \\mid \\mathcal F^{k-1}_+] = 0, \\end{align*}\n%\n$\\text{a.e.}$, where we used the fact that ${\\sf V}^k_+$ is an unbiased estimator of the gradient $\\nabla f({\\sf Z}^k)$. Next, taking conditional expectation in Lemma \\ref{lemma:prox-descent}, applying assumptions (B.2) and (B.3), and choosing $\\rho = \\lambda_k^{-1}$, we obtain\n%\n\\begin{align*} \\Exp[\\psi({\\sf X}^{k+1}) \\mid \\mathcal F^{k-1}_+] - \\psi({\\sf X}^k) & \\\\ & \\hspace{-24ex} \\leq  \\frac{\\lambda_{k,+}\\sigma_{k,+}^2}{2} + \\half  \\left[ L_f - \\frac{1}{\\lambda_{k,+}} \\right] \\Exp [\\|{\\sf X}^{k+1} - {\\sf X}^k\\|^2\\mid \\mathcal F^{k-1}_+ ] - \\frac{1}{2\\lambda_k} \\|F^{\\Lambda_k}({\\sf X}^k)\\|^2 \\\\ & \\hspace{-20ex} + \\frac{\\lambda_k \\sigma_k^2}{2}  + \\half \\left[ \\frac{1}{\\lambda_{k,+}} - \\frac{1}{\\lambda_k} + \\nu_k^2 \\cdot \\ell(\\lambda_{k,+},\\alpha_k,\\beta_k) \\right] \\Exp[\\|F^{\\Lambda_k}_{{\\sf V}^k}({\\sf X}^k)\\|^2 \\mid \\mathcal F^{k-1}_+] \n\\end{align*}\n","sha256_utf8":"74a7e9bd16803911d760b0fc1434f16e06ec13ea8d3dcfec67e5964c4620a932"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"获得超鞅型下降","source_units":[{"start":69435,"end":69913,"start_line":657,"end_line":659,"raw_tex":"almost everywhere. Utilizing the bounds stated in \\eqref{eq:choice-step-1}, we have\n%\n\\begin{align} \\nonumber \\Exp[\\psi({\\sf X}^{k+1}) \\mid \\mathcal F^{k-1}_+] - \\psi({\\sf X}^k) & \\\\  \\label{eq:est-martingale} & \\hspace{-20ex} \\leq \\frac{\\lambda_k \\sigma_k^2}{2} + \\frac{\\lambda_{k,+}\\sigma_{k,+}^2}{2} - \\frac{1}{2\\lambda_k} \\|F^{\\Lambda_k}({\\sf X}^k)\\|^2 - \\frac{\\bar{\\rho}}{2\\lambda_k} \\Exp[ \\|F^{\\Lambda_k}_{{\\sf V}^k}({\\sf X}^k)\\|^2 \\mid \\mathcal F^{k-1}_+ ].  \\end{align}\n","sha256_utf8":"dbedfcd23c29bf2d644fa1a97ef25422d46d5af0ae86ac38624e5ca9c4eebda6"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"累加下降并控制残差总和","source_units":[{"start":69915,"end":70340,"start_line":661,"end_line":663,"raw_tex":"Now, by assumption (A.2) there exists $\\psi^* \\in \\R$ such that $\\psi(x) \\geq \\psi^*$ for all $x \\in \\Rn$. Taking expectation and summing the latter inequality, we get\n%\n\\begin{align*} \\sum_{k= 0}^R \\frac{\\Exp[\\|F^{\\Lambda_k}({\\sf X}^k)\\|^2 + \\bar{\\rho} \\cdot \\|F^{\\Lambda_k}_{{\\sf V}^k}({\\sf X}^k)\\|^2]}{\\lambda_k} \\leq 2 [\\psi(x^0) - \\psi^*] + \\sum_{k = 0}^R \\lambda_k \\sigma_k^2 + \\lambda_{k,+}\\sigma_{k,+}^2 \\end{align*}\n","sha256_utf8":"895ecb6a4682d80a2e657041225e937d8c37c51b090398e16118058ebcf136c6"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"推出残差下极限与目标收敛","source_units":[{"start":70342,"end":71364,"start_line":665,"end_line":665,"raw_tex":"for all $R \\in \\N$. Due to $L_f \\geq 1$, it holds $\\lambda_{k,+} \\leq 1$, $\\lambda_k \\leq 1$ and by \\cite[Lemma 2]{Nes13}, we know that the mapping $\\delta \\mapsto {\\delta}^{-1} \\|F^{\\frac{1}{\\delta}I}(x)\\|$ is a decreasing function of $\\delta$. This yields $\\Exp[\\|F^I({\\sf X}^k)\\|^2] \\leq  \\lambda_k^{-2} \\Exp[ \\|F^{\\Lambda_k}({\\sf X}^k)\\|^2]$ for all $k \\in \\N$ and thus, we can infer  $\\sum \\lambda_k \\Exp[\\|F^I({\\sf X}^k)\\|^2] < \\infty$. Hence, the first statement in Theorem \\ref{theorem:conv-gen} follows from the assumption $\\sum \\lambda_k = \\infty$. Furthermore, applying the Borel-Cantelli lemma, it holds that $\\sum \\lambda_k \\|F^I({\\sf X}^k)\\|^2 < \\infty$ a.s., which finally implies $\\liminf_{k\\to \\infty} \\| F^I({\\sf X}^k) \\| = 0$ with probability 1. The third claim in Theorem \\ref{theorem:conv-gen} is a direct consequence of assumption (A.2), \\eqref{eq:est-martingale}, and Theorem \\ref{theorem:conv-superm}. In order to establish convergence of $(\\Exp[\\psi({\\sf X}^k)])_k$, we first notice that we have \n","sha256_utf8":"eb33732864718dff55b5cd722ea0962d9df56e4990794f6d697c7f17be214736"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"建立逐路径目标上界","source_units":[{"start":71366,"end":72098,"start_line":667,"end_line":669,"raw_tex":"\\begin{align} \\nonumber \\psi({\\sf X}^{k+1}) & \\leq \\psi({\\sf X}^k) + \\frac{\\lambda_k}{2} \\|\\nabla f({\\sf X}^k) - {\\sf V}^k\\|^2  \\\\ \\label{eq:des-later}  & \\hspace{3ex}  +\\lambda_{k,+} \\|\\nabla f({\\sf Z}^k) - {\\sf V}_+^k\\|^2 + (\\nu_k^2\\lambda_k)^{-1}\\|{\\sf D}^k\\|^2, \\end{align}\n%\n(almost) surely and for all $k \\in \\N$, which follows from Lemma \\ref{lemma:prox-descent}, $\\iprod{a}{b} \\leq \\half \\|a\\|^2 + \\half \\|b\\|^2$, $a,b \\in \\Rn$, $\\lambda_{k,+}^{-1} \\leq \\lambda_k^{-1}$, and $\\ell(\\lambda_{k,+},\\alpha_k,\\beta_k) < \\nu_k^{-2}(\\lambda_k^{-1} - \\lambda_{k,+}^{-1}) \\leq (\\nu_k^2 \\lambda_k)^{-1}$. By induction, this shows that the nonnegative random variable $\\psi({\\sf X}^k) - \\psi^*$ is (almost) surely dominated by \n","sha256_utf8":"41e58b19ac6fbab215e0af5a71cc281fafff0796c5d4439e59762cffcccc5a79"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"构造可积控制量并交换极限","source_units":[{"start":72100,"end":72492,"start_line":671,"end_line":674,"raw_tex":"\\[ {\\sf U} := \\psi(x^0) - \\psi^* + \\sum_{k=0}^\\infty  \\frac{\\lambda_k}{2} \\|\\nabla f({\\sf X}^k) - {\\sf V}^k\\|^2 + \\lambda_{k,+} \\|\\nabla f({\\sf Z}^k) - {\\sf V}_+^k\\|^2 + \\frac{\\|{\\sf D}^k\\|^2}{\\nu_k^2\\lambda_k}   \\]\n%\nfor all $k \\in \\N$. Moreover due to Fatou's lemma, we have $\\Exp[{\\sf U}] < \\infty$. Hence, our last claim is a consequence of the dominated convergence theorem. \n\\end{proof}","sha256_utf8":"3ceb7468e8bb5fb5a133717ae96e3163658443ba2a71a8a0e133a3e88c21910f"}]},{"id":"S032","semantic_type":"external_fact","kind":"external_fact","title":"凸近端算子标准性质","source_units":[],"declared_external":true,"declaration_text":["For a proper lower semicontinuous convex function φ and a symmetric positive definite matrix Λ, prox^Λ_φ is single-valued and Λ-firmly nonexpansive, and p = prox^Λ_φ(x) is equivalent to Λ(x-p) belonging to the convex subdifferential ∂φ(p)."]},{"id":"S033","semantic_type":"external_fact","kind":"external_fact","title":"光滑下降引理","source_units":[],"declared_external":true,"declaration_text":["If ∇f is globally L_f-Lipschitz, then f(y) <= f(x) + <∇f(x),y-x> + (L_f/2)||y-x||² for every x and y."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S016","kind":"supports"},{"source_id":"S004","target_id":"S026","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S017","kind":"supports"},{"source_id":"S006","target_id":"S016","kind":"supports"},{"source_id":"S007","target_id":"S017","kind":"supports"},{"source_id":"S007","target_id":"S026","kind":"supports"},{"source_id":"S008","target_id":"S013","kind":"supports"},{"source_id":"S008","target_id":"S026","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S018","kind":"supports"},{"source_id":"S009","target_id":"S028","kind":"supports"},{"source_id":"S009","target_id":"S029","kind":"supports"},{"source_id":"S010","target_id":"S017","kind":"supports"},{"source_id":"S011","target_id":"S013","kind":"supports"},{"source_id":"S011","target_id":"S026","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S026","kind":"supports"},{"source_id":"S012","target_id":"S031","kind":"supports"},{"source_id":"S013","target_id":"S026","kind":"supports"},{"source_id":"S014","target_id":"S026","kind":"supports"},{"source_id":"S014","target_id":"S030","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S015","kind":"supports"},{"source_id":"S018","target_id":"S015","kind":"supports"},{"source_id":"S019","target_id":"S029","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S029","kind":"supports"},{"source_id":"S022","target_id":"S026","kind":"supports"},{"source_id":"S023","target_id":"S017","kind":"supports"},{"source_id":"S023","target_id":"S018","kind":"supports"},{"source_id":"S023","target_id":"S030","kind":"supports"},{"source_id":"S024","target_id":"S029","kind":"supports"},{"source_id":"S024","target_id":"S031","kind":"supports"},{"source_id":"S025","target_id":"S027","kind":"supports"},{"source_id":"S025","target_id":"S029","kind":"supports"},{"source_id":"S025","target_id":"S030","kind":"supports"},{"source_id":"S025","target_id":"S031","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S002","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S029","kind":"supports"},{"source_id":"S027","target_id":"S002","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S031","kind":"supports"},{"source_id":"S028","target_id":"S002","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S029","target_id":"S002","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S002","kind":"supports"},{"source_id":"S031","target_id":"S002","kind":"supports"},{"source_id":"S032","target_id":"S006","kind":"supports"},{"source_id":"S033","target_id":"S010","kind":"supports"}]}}
{"record_id":"1912.12004v2","paper":{"title":"Nearly optimal first-order methods for convex optimization under gradient norm measure: An adaptive regularization approach","authors":["Masaru Ito","Mituhiro Fukuda"],"arxiv_id":"1912.12004","version":"v2","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-020-01806-7","abstract_url":"https://arxiv.org/abs/1912.12004v2","pdf_url":"https://arxiv.org/pdf/1912.12004v2"},"context":{"raw_tex":"Throughout this paper, let $\\E$ be a finite dimensional real Hilbert space endowed with an inner product $\\innprod{\\cdot}{\\cdot}$.\nWe denote by $\\norm{x}=\\innprod{x}{x}^{1/2}$ the induced norm on $\\E$.\nThe distance between a point $x \\in \\E$ and a set $S \\subset \\E$ is defined by\n$\\dist(x,S)=\\inf_{z \\in S}\\norm{z-x}.$\n\n\\sloppy % prevents overflow\nWe focus on the the convex composite optimization problem\n\\begin{equation}\\label{main-prob}\n\\varphi^* = \\min_{x \\in \\E}[\\varphi(x) \\equiv f(x)+\\varPsi(x)],\n\\end{equation}\nwhere $f:\\E\\to \\RR$ is an \\emph{$L_f$-smooth} convex function, that is, it is continuously differentiable and its gradient is $L_f$-Lipschitz continuous:\n$$\n\\norm{\\nabla{f}(x)-\\nabla{f}(y)} \\leq L_f\\norm{x-y}, \\quad \\forall x,y \\in \\E,\n$$\nand ${\\varPsi:\\E\\to\\RR\\cup\\{+\\infty\\}}$ is a proper, lower-semicontinuous, and convex function.\nWe denote by ${X^*:=\\Argmin_{x \\in \\E}\\varphi(x)}$ the set of optimal solutions of $\\min_{x \\in \\E}\\varphi(x)$.\n%$X^* = \\Argmin_{x \\in \\E}\\varphi(x).$\nThroughout the paper, we assume that $X^*$ is nonempty.\nThe subdifferential of $\\varphi$ at $x$ is denoted by\n${\\partial\\varphi(x) = \\{g \\in \\E \\mid \\varphi(y)\\geq \\varphi(x)+\\innprod{g}{y-x},~\\forall y \\in \\E\\}}$.\n\nFor the objective function $\\varphi(x)=f(x)+\\varPsi(x)$, we define\n$$\nm_L(y;x):= f(y)+\\innprod{\\nabla{f}(y)}{x-y} + \\frac{L}{2}\\norm{x-y}^2+\\varPsi(x),\\qquad T_L(y):= \\argmin_{x \\in \\E}m_L(y;x), %\\quad(=\\prox_{\\varPsi/L}(y-L^{-1}\\nabla{f}(y))),\n$$\nwhere the minimizer $T_L(y)$ is well-defined since $m_L(y;\\cdot)$ is strongly convex.\nIt is assumed that $\\varPsi(\\cdot)$ has a ``simple'' structure, namely, the proximal mapping $\\prox_{\\varPsi}(y):=\\argmin_{x \\in \\E}\\{\\varPsi(x)+\\frac{1}{2}\\norm{x-y}^2\\}$ as well as the map $T_L(y)\\equiv \\prox_{\\varPsi/L}(y-\\nabla{f}(x)/L)$ is moderately computable (see \\cite{PB14} for examples).\nThe \\emph{gradient mapping} of $\\varphi$ is defined by\n\\[\ng_L(y):=L(y-T_L(y)),\\quad y \\in \\E,~L>0.\n\\]\nFor instance, if $\\varPsi\\equiv 0$, we see that $T_L(y)=y-\\nabla{f}(y)/L$ and $g_L(y)=\\nabla{f}(y)$ hold.\n\n\n\\begin{definition}\nFix $x_0 \\in \\E$.\nWe say that $\\varphi$ satisfies the \\emph{H\\\"olderian error bound condition} with coefficient $\\kappa > 0$ and exponent $\\rho \\geq 1$ if\n\\begin{equation}\\label{HEB}\n\\varphi(x)-\\varphi^* \\geq \\kappa\\dist(x,X^*)^\\rho,\\quad \\forall x\\in \\lev_{\\varphi}(\\varphi(x_0)),\n\\end{equation}\nwhere $\\lev_{\\varphi}(\\gamma)=\\{x \\in \\E \\mid \\varphi(x)\\leq \\gamma\\}$.\n\\end{definition}\n\nWe consider to apply an accelerated proximal gradient method to the regularized problem\n$$\n\\min_{x\\in \\E}\\left[\\varphi_\\sigma(x):=\n\\varphi(x) + \\frac{\\sigma}{2}\\norm{x-x_0}^2\\right],\n$$\nwhere $x_0 \\in \\E$ is a fixed initial point and $\\sigma>0$ is a regularization parameter.\nSince $\\varphi_\\sigma$ is strongly convex, it has a unique minimizer. Let\n$$x^*_\\sigma := \\argmin_{x \\in \\E}\\varphi_\\sigma(x),\\quad \\varphi_\\sigma^* := \\min_{x\\in \\E} \\varphi_\\sigma(x).$$\nWe define the gradient mapping $g_L^\\sigma(x)$ for the regularized function $\\varphi_\\sigma$ in the following manner:\n$$\nf_\\sigma(x):=f(x)+\\frac{\\sigma}{2}\\norm{x-x_0}^2,\\qquad\nm_L^\\sigma(y;x) := f_\\sigma(y)+\\innprod{\\nabla{f}_\\sigma(y)}{x-y}+\\frac{L}{2}\\norm{x-y}^2 + \\varPsi(x),\n$$\n$$\nT_L^\\sigma(y):=\\argmin_{x\\in \\E}m_L^\\sigma(y;x),\\qquad\ng_L^\\sigma(y):=L(y-T_L^\\sigma(y)).\n$$\n\n\nWe employ Nesterov's accelerated proximal gradient method \\cite{Nes13} for solving $\\min_{x \\in E}\\varphi_\\sigma(x)$, by regarding the objective function as\n\\begin{equation}\\label{realloc}\n\\varphi_\\sigma(x) = f(x) + \\varPsi_\\sigma(x),\\quad \\varPsi_\\sigma(x):=\\varPsi(x) + \\frac{\\sigma}{2}\\norm{x-x_0}^2.\n\\end{equation}\nThe analogy of the definition of $T_L(\\cdot)$ for this regularization is \n\\begin{equation*}%\\label{tilde-T}\n\\tilde{T}_L(y) = \\argmin_{x \\in \\E}\\left[f(y) + \\innprod{\\nabla{f}(y)}{x-y} + \\frac{L}{2}\\norm{x-y}^2 + \\varPsi_\\sigma(x)\\right].\n\\end{equation*}\nBy the identity\n$m^\\sigma_{L+\\sigma}(y;x) = f(y) + \\innprod{\\nabla{f}(y)}{x-y} +\\frac{L}{2}\\norm{x-y}^2+ \\varPsi_\\sigma(x)$,\nobserve that $\\tilde{T}_L(y)=T^\\sigma_{L+\\sigma}(y)$ holds.\nTherefore, the accelerated method $\\mathcal{A}(x_0,L_0,\\sigma)$ given in \\cite[Algorithm~(4.9)]{Nes13} applied to the regularized function \\eqref{realloc} can be described as follows:\n{\\it Let $x_0 \\in \\E$, $\\psi_0(x)=\\frac{1}{2}\\norm{x-x_0}^2$, $A_0=0$, $L_0 \\geq L_{\\min}$.\nGenerate the sequence $\\{x_k,\\psi_k,M_k,L_k,A_k\\}$ by the iteration\n$$\\{x_{k+1},\\psi_{k+1},M_k,L_{k+1},A_{k+1}\\} \\leftarrow \\APGIter_\\sigma(x_k,\\psi_k,L_k,A_k),$$\nfor each $k\\geq 0$.}\nThe scheme $\\APGIter$ at each iteration is shown in Algorithm~\\ref{APG-iter}.\nNesterov's method involves the backtracking line-search procedure to adapt the Lipschitz constant $L_f$ with the estimate $M_k$ (and $L_{k+1}$), which is controlled by the parameters $\\gamma_{\\rm inc}>1$ and $\\gamma_{\\rm dec}\\geq 1$.\n\n\n\\begin{algorithm}[t]\n\\caption{\nAccelerated Proximal Gradient Iteration\n\\newline\n$\\{x_{k+1},\\psi_{k+1},M_k,L_{k+1},A_{k+1}\\} \\leftarrow \\APGIter_\\sigma(x_k,\\psi_k,L_k,A_k)$}\n\\label{APG-iter}\n\\textbf{Parameters:} $\\gamma_{\\rm inc}>1$, $\\gamma_{\\rm dec}\\geq 1$, $L_{\\min} \\in  ]0,L_f]$.\\\\\n\\textbf{Input:} $\\sigma>0$, $x_k \\in \\E$, $\\psi_k:\\E\\to\\RR\\cup\\{+\\infty\\}$, $L_k>0$, $A_k\\geq 0$.\n\\begin{algorithmic}[1]\n\\State Compute $v_{k}:=\\argmin_{x\\in \\E} \\psi_{k}(x)$\\quad (cf. \\eqref{APG-iter-v_k-formula}). \\label{APG-iter-vk}\n\\State Set $L:=L_k/\\gamma_{\\rm inc}$.\n\\Repeat \\label{APG-iter-loop-s}\n\t\\State Set $L:=L\\gamma_{\\rm inc}$.\n\t\\State Find the largest root $a>0$ of the equation $\\frac{a^2}{A_k+a} = 2\\frac{1+\\sigma A_k}{L}$. \\label{a-quad-eq}\n\t\\State Set $y = \\frac{A_k x_k +a v_k}{A_k + a}$.\n\t\\State Compute $z=T_{L+\\sigma}^\\sigma(y)=\\argmin_{x}[f(y)+\\innprod{\\nabla{f}(y)}{x-y}+\\frac{L}{2}\\norm{x-y}^2 + \\varPsi_\\sigma(x)]$.\n\t\\State Compute $T_L(z)$ and $T_{L+\\sigma}^\\sigma(z)$ \\label{APG-iter-add-prox}.\n\t\\State Test the conditions\n\t\\begin{subequations}\n\t\\label{APG-iter-test}\n\t\\begin{align}\n\t&\\innprod{\\nabla{f}(y)-\\nabla{f}(T_{L+\\sigma}^\\sigma(y))}{y-T_{L+\\sigma}^\\sigma(y)} \\geq \\frac{1}{L}\\norm{\\nabla{f}(y)-\\nabla{f}(T_{L+\\sigma}^\\sigma(y))}^2, \\label{APG-iter-test-a}\\\\\n\t&\\varphi_\\sigma(T_{L+\\sigma}^\\sigma(z)) \\leq m_{L+\\sigma}^\\sigma(z;T_{L+\\sigma}^\\sigma(z)), \\label{APG-iter-test-b}\\\\\n\t&\\text{[Optional]}~~\\varphi(T_L(z)) \\leq \\varphi(z) \\label{APG-iter-test-c}.\n\t\\end{align}\n\t\\end{subequations}\n\\Until{the conditions in \\eqref{APG-iter-test} hold.} \\label{APG-iter-loop-e}\n\\State Define $y_k:=y$, $M_k:=L$, $x_{k+1}:=z=T_{M_k+\\sigma}^\\sigma(y_k)$, \\\\\n\\quad $a_{k+1}:=a$, $A_{k+1}:=A_k+a_{k+1}$, $L_{k+1}:=\\max\\{L_{\\min},M_k/\\gamma_{\\rm dec}\\}$, \\label{AGP-iter-upadte}\n\\State \\quad $\\psi_{k+1}(x):=\\psi_k(x)+a_{k+1}[f(x_{k+1})+\\innprod{\\nabla{f}(x_{k+1})}{x-x_{k+1}}+\\varPsi_\\sigma(x)]$.\\label{est-seq-formula}\n\\State Output $\\{x_{k+1},\\psi_{k+1},M_k,L_{k+1},A_{k+1}\\}$.\n\\end{algorithmic}\n\\end{algorithm}\n\n\\begin{equation}\\label{APG-iter-v_k-formula}\nv_k = \\prox_{\\gamma_k\\varPsi}(w_k),\\text{ where } \\gamma_k = \\frac{A_k}{1+\\sigma A_k},~~w_k = x_0-\\frac{1}{1+\\sigma A_k}\\sum_{i=1}^ka_i \\nabla{f}(x_i).\n\\end{equation}\n\n\\begin{algorithm}\n\\caption{\nProximal Gradient Iteration\n\\newline\n$\\{T_{M_k}(x_k), M_k, L_{k+1}\\} \\leftarrow \\PGIter(x_k,L_k)$\n}\n\\label{alg:PG-iter}\n\\textbf{Parameters:} $\\gamma_{\\rm inc}>1$, $\\gamma_{\\rm dec}\\geq 1$, $L_{\\min} \\in ]0,L_f]$.\\\\\n\\textbf{Input:} $x_k \\in \\E$, $L_k > 0$.\n\\begin{algorithmic}[1]\n\\State Set $L:=L_k/\\gamma_{\\rm inc}$.\n\\Repeat \\label{PG-iter-loop-s}\n\\State Set $L:=\\gamma_{\\rm inc}L$.\n\\State Compute $T_L(x_k)$.\n\\Until{the following condition holds:} \\label{PG-iter-loop-e}\n\\begin{equation}\\label{PG-iter-test}\n\t\\varphi(T_L(x_k))\\leq m_L(x_k;T_L(x_k)).\n\\end{equation}\n\\State Define $M_k:=L$, $L_{k+1}:=\\max\\{L_{\\min},M_k/\\gamma_{\\rm dec}\\}$.\n\\State Output $\\{T_{M_k}(x_k), M_k, L_{k+1}\\}$.\n\\end{algorithmic}\n\\end{algorithm}\n\n\\begin{algorithm}\n\\caption{\nAdaptive Accelerated Proximal Gradient Method\n\\newline\n$\\AdaAPG(x_0,L_{-1},\\sigma_0,\\ep)$\n}\n\\label{alg:AdaAPG}\n\\textbf{Parameters:} $\\gamma_{\\rm inc}>1$, $\\gamma_{\\rm dec}\\geq 1$, $L_{\\rm min}\\in ]0,L_f]$, $\\gamma_{\\rm reg}>1$, $\\beta\\in]0,1]$.\\\\\n\\textbf{Input:} $x_0 \\in \\E$, $L_{-1} \\in [L_{\\min}, \\gamma_{\\rm inc}L_f]$, $\\sigma_0>0$, $\\ep>0$.\n\\begin{algorithmic}[1]\n\\State $L_{0}:= L_{-1}$.\n\\For{$j=0,1,2,\\ldots$}\n\t\\State $\\sigma_j := \\sigma_0/\\gamma_{\\rm reg}^j$.\n\t\\State $\\psi_{0}(x):=\\frac{1}{2}\\norm{x-x_{0}}^2$, $A_{0}:=0$.\n\t\\Repeat{ \\textbf{for} $k=0,1,2,\\ldots$}\n\t\t\\State $\\{x_{k+1},\\psi_{k+1},M_{k},L_{k+1},A_{k+1}\\} \\leftarrow \\APGIter_{\\sigma_j}(x_{k},\\psi_{k},L_{k},A_{k})$.\n\t\t\\If{$\\norm{g_{M_{k}}(x_{k+1})} \\leq \\ep$}\n\t\t\t\\State output $\\{\\sigma_j, x_{k+1},T_{M_{k}}(x_{k+1}),M_{k},L_{k+1}\\}$ and terminate the algorithm.\n\t\t\\EndIf\n\t\\Until{$A_{k+1} \\geq \\frac{2(M_{k}+\\sigma_j)}{\\beta^2\\sigma_j^2}$}.\\label{A-growth}\n\\State $L_{0} := L_{k+1}$.\n\\EndFor\n\\end{algorithmic}\n\\end{algorithm}\n\nIn this section, given an initial point $\\xinit \\in \\E$, we assume that the objective function $\\varphi$ admits the H\\\"olderian error bound condition\n\\begin{equation}\\label{HEB-review}\n\\varphi(x)-\\varphi^* \\geq \\kappa\\dist(x,X^*)^\\rho,\\quad \\forall x\\in \\lev_{\\varphi}(\\varphi(\\xinit)),\n\\end{equation}\nfor some $\\kappa>0$ and $\\rho\\geq 1$.\n\n\n\\begin{algorithm}\n\\caption{\nRestart Scheme for Adaptive Accelerated Proximal Gradient Method\n\\newline\n$\\rAdaAPG (\\xinit,\\Linit,\\sigmainit,\\ep)$\n}\n\\label{alg:rAdaAPG}\n\\textbf{Parameters:}\\\\\n\\quad $\\gamma_{\\rm inc}>1$, $\\gamma_{\\rm dec}\\geq 1$, $L_{\\rm min}\\in ]0,L_f]$ (for backtracking line-search);\\\\\n\\quad $\\gamma_{\\rm reg}>1$ (the ratio to decrease the regularization parameter);\\\\\n\\quad $\\beta\\in ]0,1]$  (controls the timing when the current regularization parameter must be decreased);\\\\\n\\quad $\\theta \\in ]0,1[$ (the ratio reducing the residue per iteration).\\\\\n\\textbf{Input:} $\\xinit \\in \\E$, $\\Linit \\in [L_{\\min},\\gamma_{\\rm inc}L_f]$, $\\sigmainit>0$, $\\ep>0$.\n\\begin{algorithmic}[1]\n\\State Compute $\\{\\outerXplus{0},\\outerM{0},\\outerL{0}\\} \\leftarrow \\PGIter(\\xinit,\\Linit)$ \\label{rAdaAPG-init}.\n\\For{$t=0,1,2,\\ldots$}\n\t\\State \\textbf{if} $\\norm{g_{\\outerM{t}}(\\outerXinit{t})} \\leq \\ep$ \\textbf{then} output $\\{\\outerXinit{t},\\outerM{t},\\outerXplus{t}\\}$ and terminate the algorithm. \\label{rAdaAPG-terminate}\n\t\\State $\\outerep{t}:= \\theta\\norm{g_{\\outerM{t}}(\\outerXinit{t})}$.\n\t\\State $\\{\\outersigma{t+1},\\outerXinit{t+1},\\outerXplus{t+1},\\outerM{t+1},\\outerL{t+1}\\} \\leftarrow \\AdaAPG(\\outerXplus{t},\\outerL{t},\\outersigma{t},\\ep^{(t)})$\n\t\\State (with testing the optional condition \\eqref{APG-iter-test-c} of $\\APGIter$).\n\\EndFor\n\\end{algorithmic}\n\\end{algorithm}\n\nTo describe the iteration complexity bounds, we define $N(\\ep,\\sigma_*,C)$ and $\\bar\\sigma$ as follows.\nGiven $\\ep,\\sigma_*,C>0$, let $N(\\ep,\\sigma_*,C):=0$ if $\\norm{g_{\\outerM{0}}(\\xinit)} \\leq \\ep$ and otherwise\n\\begin{align}\nN(\\ep,\\sigma_*,C) &\n:= \\left(1+ \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep} + \\log_{\\gamma_{\\rm reg}}\\frac{\\sigmainit}{\\sigma_*}\\right)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}\\right) \\nonumber\\\\\n&\\quad +\n\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\n\\sqrt{\\frac{1}{\\sigma_*}} - \\sqrt{\\frac{1}{\\sigmainit}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*} \n+ C\\sqrt{2\\gamma_{\\rm inc}L_f}\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}. \\label{N-def}\n\\end{align}\n\n\\begin{equation}\\label{bar-sigma-def}\n\\bar\\sigma = \\left\\{\n\\begin{array}{ll}\n\\ds\n\\frac{\\theta}{1+\\sqrt{2}\\beta}\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}}\n\\ep^{\\frac{\\rho-2}{\\rho-1}}\n& (\\text{if } \\rho \\geq 2),\\\\[3truemm]\n\\ds\n\\frac{\\theta}{1+\\sqrt{2}\\beta}\n\\kappa^{\\frac{2}{\\rho}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-1}(\\varphi(\\xinit)-\\varphi^*)^{-\\frac{2-\\rho}{\\rho}}\n& (\\text{if } 1\\leq \\rho<2).\n\\end{array}\n\\right.\n\\end{equation}","units":[{"start":13502,"end":15565,"start_line":264,"end_line":295,"raw_tex":"Throughout this paper, let $\\E$ be a finite dimensional real Hilbert space endowed with an inner product $\\innprod{\\cdot}{\\cdot}$.\nWe denote by $\\norm{x}=\\innprod{x}{x}^{1/2}$ the induced norm on $\\E$.\nThe distance between a point $x \\in \\E$ and a set $S \\subset \\E$ is defined by\n$\\dist(x,S)=\\inf_{z \\in S}\\norm{z-x}.$\n\n\\sloppy % prevents overflow\nWe focus on the the convex composite optimization problem\n\\begin{equation}\\label{main-prob}\n\\varphi^* = \\min_{x \\in \\E}[\\varphi(x) \\equiv f(x)+\\varPsi(x)],\n\\end{equation}\nwhere $f:\\E\\to \\RR$ is an \\emph{$L_f$-smooth} convex function, that is, it is continuously differentiable and its gradient is $L_f$-Lipschitz continuous:\n$$\n\\norm{\\nabla{f}(x)-\\nabla{f}(y)} \\leq L_f\\norm{x-y}, \\quad \\forall x,y \\in \\E,\n$$\nand ${\\varPsi:\\E\\to\\RR\\cup\\{+\\infty\\}}$ is a proper, lower-semicontinuous, and convex function.\nWe denote by ${X^*:=\\Argmin_{x \\in \\E}\\varphi(x)}$ the set of optimal solutions of $\\min_{x \\in \\E}\\varphi(x)$.\n%$X^* = \\Argmin_{x \\in \\E}\\varphi(x).$\nThroughout the paper, we assume that $X^*$ is nonempty.\nThe subdifferential of $\\varphi$ at $x$ is denoted by\n${\\partial\\varphi(x) = \\{g \\in \\E \\mid \\varphi(y)\\geq \\varphi(x)+\\innprod{g}{y-x},~\\forall y \\in \\E\\}}$.\n\nFor the objective function $\\varphi(x)=f(x)+\\varPsi(x)$, we define\n$$\nm_L(y;x):= f(y)+\\innprod{\\nabla{f}(y)}{x-y} + \\frac{L}{2}\\norm{x-y}^2+\\varPsi(x),\\qquad T_L(y):= \\argmin_{x \\in \\E}m_L(y;x), %\\quad(=\\prox_{\\varPsi/L}(y-L^{-1}\\nabla{f}(y))),\n$$\nwhere the minimizer $T_L(y)$ is well-defined since $m_L(y;\\cdot)$ is strongly convex.\nIt is assumed that $\\varPsi(\\cdot)$ has a ``simple'' structure, namely, the proximal mapping $\\prox_{\\varPsi}(y):=\\argmin_{x \\in \\E}\\{\\varPsi(x)+\\frac{1}{2}\\norm{x-y}^2\\}$ as well as the map $T_L(y)\\equiv \\prox_{\\varPsi/L}(y-\\nabla{f}(x)/L)$ is moderately computable (see \\cite{PB14} for examples).\nThe \\emph{gradient mapping} of $\\varphi$ is defined by\n\\[\ng_L(y):=L(y-T_L(y)),\\quad y \\in \\E,~L>0.\n\\]\nFor instance, if $\\varPsi\\equiv 0$, we see that $T_L(y)=y-\\nabla{f}(y)/L$ and $g_L(y)=\\nabla{f}(y)$ hold.\n","sha256_utf8":"d525d6153bf58dee5b0617eb5cc8e277c474db9e6e7f4c6124bdd85dbb812a47"},{"start":19864,"end":20268,"start_line":394,"end_line":401,"raw_tex":"\\begin{definition}\nFix $x_0 \\in \\E$.\nWe say that $\\varphi$ satisfies the \\emph{H\\\"olderian error bound condition} with coefficient $\\kappa > 0$ and exponent $\\rho \\geq 1$ if\n\\begin{equation}\\label{HEB}\n\\varphi(x)-\\varphi^* \\geq \\kappa\\dist(x,X^*)^\\rho,\\quad \\forall x\\in \\lev_{\\varphi}(\\varphi(x_0)),\n\\end{equation}\nwhere $\\lev_{\\varphi}(\\gamma)=\\{x \\in \\E \\mid \\varphi(x)\\leq \\gamma\\}$.\n\\end{definition}","sha256_utf8":"6687fa126d4af7563c58af96e492d4d76d64d42aa0680d5f374d3b577b42ad66"},{"start":29177,"end":30021,"start_line":546,"end_line":562,"raw_tex":"We consider to apply an accelerated proximal gradient method to the regularized problem\n$$\n\\min_{x\\in \\E}\\left[\\varphi_\\sigma(x):=\n\\varphi(x) + \\frac{\\sigma}{2}\\norm{x-x_0}^2\\right],\n$$\nwhere $x_0 \\in \\E$ is a fixed initial point and $\\sigma>0$ is a regularization parameter.\nSince $\\varphi_\\sigma$ is strongly convex, it has a unique minimizer. Let\n$$x^*_\\sigma := \\argmin_{x \\in \\E}\\varphi_\\sigma(x),\\quad \\varphi_\\sigma^* := \\min_{x\\in \\E} \\varphi_\\sigma(x).$$\nWe define the gradient mapping $g_L^\\sigma(x)$ for the regularized function $\\varphi_\\sigma$ in the following manner:\n$$\nf_\\sigma(x):=f(x)+\\frac{\\sigma}{2}\\norm{x-x_0}^2,\\qquad\nm_L^\\sigma(y;x) := f_\\sigma(y)+\\innprod{\\nabla{f}_\\sigma(y)}{x-y}+\\frac{L}{2}\\norm{x-y}^2 + \\varPsi(x),\n$$\n$$\nT_L^\\sigma(y):=\\argmin_{x\\in \\E}m_L^\\sigma(y;x),\\qquad\ng_L^\\sigma(y):=L(y-T_L^\\sigma(y)).\n$$\n","sha256_utf8":"cd99e2c97e34255f8c6be443600b03987603fc9f4584224957f5b7d75dda25cc"},{"start":32086,"end":33616,"start_line":608,"end_line":625,"raw_tex":"We employ Nesterov's accelerated proximal gradient method \\cite{Nes13} for solving $\\min_{x \\in E}\\varphi_\\sigma(x)$, by regarding the objective function as\n\\begin{equation}\\label{realloc}\n\\varphi_\\sigma(x) = f(x) + \\varPsi_\\sigma(x),\\quad \\varPsi_\\sigma(x):=\\varPsi(x) + \\frac{\\sigma}{2}\\norm{x-x_0}^2.\n\\end{equation}\nThe analogy of the definition of $T_L(\\cdot)$ for this regularization is \n\\begin{equation*}%\\label{tilde-T}\n\\tilde{T}_L(y) = \\argmin_{x \\in \\E}\\left[f(y) + \\innprod{\\nabla{f}(y)}{x-y} + \\frac{L}{2}\\norm{x-y}^2 + \\varPsi_\\sigma(x)\\right].\n\\end{equation*}\nBy the identity\n$m^\\sigma_{L+\\sigma}(y;x) = f(y) + \\innprod{\\nabla{f}(y)}{x-y} +\\frac{L}{2}\\norm{x-y}^2+ \\varPsi_\\sigma(x)$,\nobserve that $\\tilde{T}_L(y)=T^\\sigma_{L+\\sigma}(y)$ holds.\nTherefore, the accelerated method $\\mathcal{A}(x_0,L_0,\\sigma)$ given in \\cite[Algorithm~(4.9)]{Nes13} applied to the regularized function \\eqref{realloc} can be described as follows:\n{\\it Let $x_0 \\in \\E$, $\\psi_0(x)=\\frac{1}{2}\\norm{x-x_0}^2$, $A_0=0$, $L_0 \\geq L_{\\min}$.\nGenerate the sequence $\\{x_k,\\psi_k,M_k,L_k,A_k\\}$ by the iteration\n$$\\{x_{k+1},\\psi_{k+1},M_k,L_{k+1},A_{k+1}\\} \\leftarrow \\APGIter_\\sigma(x_k,\\psi_k,L_k,A_k),$$\nfor each $k\\geq 0$.}\nThe scheme $\\APGIter$ at each iteration is shown in Algorithm~\\ref{APG-iter}.\nNesterov's method involves the backtracking line-search procedure to adapt the Lipschitz constant $L_f$ with the estimate $M_k$ (and $L_{k+1}$), which is controlled by the parameters $\\gamma_{\\rm inc}>1$ and $\\gamma_{\\rm dec}\\geq 1$.\n","sha256_utf8":"f061a6192d29e117704b4c88777cf85e1ddceefa4b2174890668aa6e2b8e5e47"},{"start":33617,"end":35638,"start_line":627,"end_line":659,"raw_tex":"\\begin{algorithm}[t]\n\\caption{\nAccelerated Proximal Gradient Iteration\n\\newline\n$\\{x_{k+1},\\psi_{k+1},M_k,L_{k+1},A_{k+1}\\} \\leftarrow \\APGIter_\\sigma(x_k,\\psi_k,L_k,A_k)$}\n\\label{APG-iter}\n\\textbf{Parameters:} $\\gamma_{\\rm inc}>1$, $\\gamma_{\\rm dec}\\geq 1$, $L_{\\min} \\in  ]0,L_f]$.\\\\\n\\textbf{Input:} $\\sigma>0$, $x_k \\in \\E$, $\\psi_k:\\E\\to\\RR\\cup\\{+\\infty\\}$, $L_k>0$, $A_k\\geq 0$.\n\\begin{algorithmic}[1]\n\\State Compute $v_{k}:=\\argmin_{x\\in \\E} \\psi_{k}(x)$\\quad (cf. \\eqref{APG-iter-v_k-formula}). \\label{APG-iter-vk}\n\\State Set $L:=L_k/\\gamma_{\\rm inc}$.\n\\Repeat \\label{APG-iter-loop-s}\n\t\\State Set $L:=L\\gamma_{\\rm inc}$.\n\t\\State Find the largest root $a>0$ of the equation $\\frac{a^2}{A_k+a} = 2\\frac{1+\\sigma A_k}{L}$. \\label{a-quad-eq}\n\t\\State Set $y = \\frac{A_k x_k +a v_k}{A_k + a}$.\n\t\\State Compute $z=T_{L+\\sigma}^\\sigma(y)=\\argmin_{x}[f(y)+\\innprod{\\nabla{f}(y)}{x-y}+\\frac{L}{2}\\norm{x-y}^2 + \\varPsi_\\sigma(x)]$.\n\t\\State Compute $T_L(z)$ and $T_{L+\\sigma}^\\sigma(z)$ \\label{APG-iter-add-prox}.\n\t\\State Test the conditions\n\t\\begin{subequations}\n\t\\label{APG-iter-test}\n\t\\begin{align}\n\t&\\innprod{\\nabla{f}(y)-\\nabla{f}(T_{L+\\sigma}^\\sigma(y))}{y-T_{L+\\sigma}^\\sigma(y)} \\geq \\frac{1}{L}\\norm{\\nabla{f}(y)-\\nabla{f}(T_{L+\\sigma}^\\sigma(y))}^2, \\label{APG-iter-test-a}\\\\\n\t&\\varphi_\\sigma(T_{L+\\sigma}^\\sigma(z)) \\leq m_{L+\\sigma}^\\sigma(z;T_{L+\\sigma}^\\sigma(z)), \\label{APG-iter-test-b}\\\\\n\t&\\text{[Optional]}~~\\varphi(T_L(z)) \\leq \\varphi(z) \\label{APG-iter-test-c}.\n\t\\end{align}\n\t\\end{subequations}\n\\Until{the conditions in \\eqref{APG-iter-test} hold.} \\label{APG-iter-loop-e}\n\\State Define $y_k:=y$, $M_k:=L$, $x_{k+1}:=z=T_{M_k+\\sigma}^\\sigma(y_k)$, \\\\\n\\quad $a_{k+1}:=a$, $A_{k+1}:=A_k+a_{k+1}$, $L_{k+1}:=\\max\\{L_{\\min},M_k/\\gamma_{\\rm dec}\\}$, \\label{AGP-iter-upadte}\n\\State \\quad $\\psi_{k+1}(x):=\\psi_k(x)+a_{k+1}[f(x_{k+1})+\\innprod{\\nabla{f}(x_{k+1})}{x-x_{k+1}}+\\varPsi_\\sigma(x)]$.\\label{est-seq-formula}\n\\State Output $\\{x_{k+1},\\psi_{k+1},M_k,L_{k+1},A_{k+1}\\}$.\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"aa3c0c56f7d9052bd4a4ee593fd9d61be9f56bd2de9f16f614ababf51ada97bf"},{"start":36246,"end":36457,"start_line":664,"end_line":666,"raw_tex":"\\begin{equation}\\label{APG-iter-v_k-formula}\nv_k = \\prox_{\\gamma_k\\varPsi}(w_k),\\text{ where } \\gamma_k = \\frac{A_k}{1+\\sigma A_k},~~w_k = x_0-\\frac{1}{1+\\sigma A_k}\\sum_{i=1}^ka_i \\nabla{f}(x_i).\n\\end{equation}","sha256_utf8":"78bd5c755d13354cca8d0e0a81d41c21a5f5a0d89da30d38f35b3612db5a774a"},{"start":44916,"end":45665,"start_line":820,"end_line":841,"raw_tex":"\\begin{algorithm}\n\\caption{\nProximal Gradient Iteration\n\\newline\n$\\{T_{M_k}(x_k), M_k, L_{k+1}\\} \\leftarrow \\PGIter(x_k,L_k)$\n}\n\\label{alg:PG-iter}\n\\textbf{Parameters:} $\\gamma_{\\rm inc}>1$, $\\gamma_{\\rm dec}\\geq 1$, $L_{\\min} \\in ]0,L_f]$.\\\\\n\\textbf{Input:} $x_k \\in \\E$, $L_k > 0$.\n\\begin{algorithmic}[1]\n\\State Set $L:=L_k/\\gamma_{\\rm inc}$.\n\\Repeat \\label{PG-iter-loop-s}\n\\State Set $L:=\\gamma_{\\rm inc}L$.\n\\State Compute $T_L(x_k)$.\n\\Until{the following condition holds:} \\label{PG-iter-loop-e}\n\\begin{equation}\\label{PG-iter-test}\n\t\\varphi(T_L(x_k))\\leq m_L(x_k;T_L(x_k)).\n\\end{equation}\n\\State Define $M_k:=L$, $L_{k+1}:=\\max\\{L_{\\min},M_k/\\gamma_{\\rm dec}\\}$.\n\\State Output $\\{T_{M_k}(x_k), M_k, L_{k+1}\\}$.\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"4248f51eb1fd62ec761a9ede0f6f8c3a633045c4cf6935c7b102bf5f3202d242"},{"start":47230,"end":48256,"start_line":885,"end_line":908,"raw_tex":"\\begin{algorithm}\n\\caption{\nAdaptive Accelerated Proximal Gradient Method\n\\newline\n$\\AdaAPG(x_0,L_{-1},\\sigma_0,\\ep)$\n}\n\\label{alg:AdaAPG}\n\\textbf{Parameters:} $\\gamma_{\\rm inc}>1$, $\\gamma_{\\rm dec}\\geq 1$, $L_{\\rm min}\\in ]0,L_f]$, $\\gamma_{\\rm reg}>1$, $\\beta\\in]0,1]$.\\\\\n\\textbf{Input:} $x_0 \\in \\E$, $L_{-1} \\in [L_{\\min}, \\gamma_{\\rm inc}L_f]$, $\\sigma_0>0$, $\\ep>0$.\n\\begin{algorithmic}[1]\n\\State $L_{0}:= L_{-1}$.\n\\For{$j=0,1,2,\\ldots$}\n\t\\State $\\sigma_j := \\sigma_0/\\gamma_{\\rm reg}^j$.\n\t\\State $\\psi_{0}(x):=\\frac{1}{2}\\norm{x-x_{0}}^2$, $A_{0}:=0$.\n\t\\Repeat{ \\textbf{for} $k=0,1,2,\\ldots$}\n\t\t\\State $\\{x_{k+1},\\psi_{k+1},M_{k},L_{k+1},A_{k+1}\\} \\leftarrow \\APGIter_{\\sigma_j}(x_{k},\\psi_{k},L_{k},A_{k})$.\n\t\t\\If{$\\norm{g_{M_{k}}(x_{k+1})} \\leq \\ep$}\n\t\t\t\\State output $\\{\\sigma_j, x_{k+1},T_{M_{k}}(x_{k+1}),M_{k},L_{k+1}\\}$ and terminate the algorithm.\n\t\t\\EndIf\n\t\\Until{$A_{k+1} \\geq \\frac{2(M_{k}+\\sigma_j)}{\\beta^2\\sigma_j^2}$}.\\label{A-growth}\n\\State $L_{0} := L_{k+1}$.\n\\EndFor\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"9d1d7a73cb0361ed18892909a804266378fec2baac06aaf88469017992ff93a5"},{"start":60532,"end":60872,"start_line":1126,"end_line":1130,"raw_tex":"In this section, given an initial point $\\xinit \\in \\E$, we assume that the objective function $\\varphi$ admits the H\\\"olderian error bound condition\n\\begin{equation}\\label{HEB-review}\n\\varphi(x)-\\varphi^* \\geq \\kappa\\dist(x,X^*)^\\rho,\\quad \\forall x\\in \\lev_{\\varphi}(\\varphi(\\xinit)),\n\\end{equation}\nfor some $\\kappa>0$ and $\\rho\\geq 1$.\n","sha256_utf8":"f2113a9b9e096ea20f9ba0fdbbe1cb0301041684bab24be512d7551ac0e9211f"},{"start":61920,"end":63301,"start_line":1145,"end_line":1167,"raw_tex":"\\begin{algorithm}\n\\caption{\nRestart Scheme for Adaptive Accelerated Proximal Gradient Method\n\\newline\n$\\rAdaAPG (\\xinit,\\Linit,\\sigmainit,\\ep)$\n}\n\\label{alg:rAdaAPG}\n\\textbf{Parameters:}\\\\\n\\quad $\\gamma_{\\rm inc}>1$, $\\gamma_{\\rm dec}\\geq 1$, $L_{\\rm min}\\in ]0,L_f]$ (for backtracking line-search);\\\\\n\\quad $\\gamma_{\\rm reg}>1$ (the ratio to decrease the regularization parameter);\\\\\n\\quad $\\beta\\in ]0,1]$  (controls the timing when the current regularization parameter must be decreased);\\\\\n\\quad $\\theta \\in ]0,1[$ (the ratio reducing the residue per iteration).\\\\\n\\textbf{Input:} $\\xinit \\in \\E$, $\\Linit \\in [L_{\\min},\\gamma_{\\rm inc}L_f]$, $\\sigmainit>0$, $\\ep>0$.\n\\begin{algorithmic}[1]\n\\State Compute $\\{\\outerXplus{0},\\outerM{0},\\outerL{0}\\} \\leftarrow \\PGIter(\\xinit,\\Linit)$ \\label{rAdaAPG-init}.\n\\For{$t=0,1,2,\\ldots$}\n\t\\State \\textbf{if} $\\norm{g_{\\outerM{t}}(\\outerXinit{t})} \\leq \\ep$ \\textbf{then} output $\\{\\outerXinit{t},\\outerM{t},\\outerXplus{t}\\}$ and terminate the algorithm. \\label{rAdaAPG-terminate}\n\t\\State $\\outerep{t}:= \\theta\\norm{g_{\\outerM{t}}(\\outerXinit{t})}$.\n\t\\State $\\{\\outersigma{t+1},\\outerXinit{t+1},\\outerXplus{t+1},\\outerM{t+1},\\outerL{t+1}\\} \\leftarrow \\AdaAPG(\\outerXplus{t},\\outerL{t},\\outersigma{t},\\ep^{(t)})$\n\t\\State (with testing the optional condition \\eqref{APG-iter-test-c} of $\\APGIter$).\n\\EndFor\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"970e3a362fd188bccc83fab8d05409e4e2c625fe9c7d7baf3e98d882cda5606b"},{"start":63941,"end":64706,"start_line":1181,"end_line":1190,"raw_tex":"To describe the iteration complexity bounds, we define $N(\\ep,\\sigma_*,C)$ and $\\bar\\sigma$ as follows.\nGiven $\\ep,\\sigma_*,C>0$, let $N(\\ep,\\sigma_*,C):=0$ if $\\norm{g_{\\outerM{0}}(\\xinit)} \\leq \\ep$ and otherwise\n\\begin{align}\nN(\\ep,\\sigma_*,C) &\n:= \\left(1+ \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep} + \\log_{\\gamma_{\\rm reg}}\\frac{\\sigmainit}{\\sigma_*}\\right)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}\\right) \\nonumber\\\\\n&\\quad +\n\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\n\\sqrt{\\frac{1}{\\sigma_*}} - \\sqrt{\\frac{1}{\\sigmainit}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*} \n+ C\\sqrt{2\\gamma_{\\rm inc}L_f}\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}. \\label{N-def}\n\\end{align}","sha256_utf8":"596b21831c9037bf0a64d6de1cd1cd8ba3259808170bde713d07eab88801c59b"},{"start":64727,"end":65199,"start_line":1192,"end_line":1206,"raw_tex":"\\begin{equation}\\label{bar-sigma-def}\n\\bar\\sigma = \\left\\{\n\\begin{array}{ll}\n\\ds\n\\frac{\\theta}{1+\\sqrt{2}\\beta}\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}}\n\\ep^{\\frac{\\rho-2}{\\rho-1}}\n& (\\text{if } \\rho \\geq 2),\\\\[3truemm]\n\\ds\n\\frac{\\theta}{1+\\sqrt{2}\\beta}\n\\kappa^{\\frac{2}{\\rho}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-1}(\\varphi(\\xinit)-\\varphi^*)^{-\\frac{2-\\rho}{\\rho}}\n& (\\text{if } 1\\leq \\rho<2).\n\\end{array}\n\\right.\n\\end{equation}","sha256_utf8":"adb6adc19541bcf96e321aedf1e42fbe6387adbc09f0f11dbbb1549d66f7a564"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{thm:rAdaAPG}\nAssume that the H\\\"olderian error bound condition \\eqref{HEB-review} holds.\nIn $\\rAdaAPG$, \nlet $N$ be the total number of executions of $\\APGIter$.\nDenote\n\\begin{equation}\\label{simga*-def}\n\\sigma_* := \\left\\{\n\\begin{array}{ll}\n\\sigmainit & (\\text{if } \\sigmainit \\leq \\bar\\sigma),\\\\\n\\bar\\sigma/\\gamma_{\\rm reg} & (\\text{otherwise}),\n\\end{array}\n\\right.\n\\end{equation}\nfor  $\\bar\\sigma$ defined in \\eqref{bar-sigma-def}.\nAlso, let  $N(\\cdot,\\cdot,\\cdot)$ be defined by    \\eqref{N-def}.\nThen, the following assertions hold.\n\\begin{enumerate}[(i)]\n\\item If $\\rho=2$, then $N\\leq N(\\ep,\\sigma_*,C)$ holds with\n$\nC =\n\\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep}\\right).\n$\n\n\\item Suppose $\\rho>2$.\nIf $\\sigmainit\\geq\\bar\\sigma$, then  $N\\leq N(\\ep,\\sigma_*,C)$ holds with\n\\begin{align*}\nC=&\n\\sqrt{\\frac{1}{\\sigmainit}}\\left(1+\\frac{\\rho-1}{\\rho-2}\\log_{1/\\theta}\\frac{\\sigma_*^{(0)}}{\\min(\\sigma_*^{(0)},\\sigmainit)}\\right)+\n\\frac{\\sqrt{\\gamma_{\\rm reg}}}{1-\\sqrt{\\theta}^{\\frac{\\rho-2}{\\rho-1}}}\n\\left(\n\\sqrt{\\frac{1}{\\bar\\sigma}}\n-\n\\sqrt{\\theta}^{\\frac{\\rho-2}{\\rho-1}}\n\\sqrt{\\frac{1}{\\sigmainit}}\n\\right),\n\\end{align*}\nwhere\n$\n\\sigma_*^{(0)}:=\\frac{\\theta}{1+\\sqrt{2}\\beta}\\cdot\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}}\n\\|g_{\\outerM{0}}(\\xinit)\\|^{\\frac{\\rho-2}{\\rho-1}}.\n$\nOtherwise, it follows $\\outersigma{t}=\\sigmainit$ for all $t\\geq 0$ and $N\\leq N(\\ep,\\sigmainit,C)$ holds with\n$\nC= \n\\sqrt{\\frac{1}{\\sigmainit}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep}\\right).\n$\n\\item Suppose $\\rho\\in ]1,2[$.\nThen, we have\n\\begin{equation}\\label{N-bd-superlinear}\nN \\leq 1+ N(\\max(\\ep,\\ep_*),\\sigma_*,C) + \\left(\\log\\frac{1}{\\rho-1}\\right)^{-1}\\left(\\log\\log \\frac{\\ep_*}{\\theta^{\\frac{\\rho-1}{2-\\rho}}\\min(\\ep,\\ep_*)}-\\log\\log\\frac{1}{\\theta^{\\frac{\\rho-1}{2-\\rho}}}\\right),\n\\end{equation}\nwhere\n$\nC =\n\\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\max(\\ep,\\ep_*)}  \\right)\n$ and\n\\begin{equation}\\label{rho-12-ep*-def}\n\\ep_*=\n\\left[(1+\\sqrt{2})\\theta^{-1}(\\gamma_{\\rm inc}L_f+\\sigmainit)\\right]^{-\\frac{\\rho-1}{2-\\rho}}\n{\\kappa^{\\frac{1}{2-\\rho}}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{2-\\rho}}.\n\\end{equation}\n%\n\\item\n\\sloppy % prevents overflow\nIf $\\rho=1$, then we have\n$\nN \\leq 1 + N(\\max(\\ep,\\ep_*),\\sigma_*,C)\n$,\nwhere\n$\n\\ep_* = \\kappa\\left(\\frac{L_f}{L_{\\min}} + 1\\right)^{-1}\n$ and\n$\n{C=\n\\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\max(\\ep,\\ep_*)}\\right)}\n$.\nMoreover, if $\\ep<\\ep_*$, then $\\outerXplus{t}$ in the output of the algorithm must be an optimal solution to the original problem \\eqref{main-prob}.\n\\end{enumerate}\n\\end{theorem}","units":[{"start":65481,"end":68252,"start_line":1209,"end_line":1288,"raw_tex":"\\begin{theorem}\\label{thm:rAdaAPG}\nAssume that the H\\\"olderian error bound condition \\eqref{HEB-review} holds.\nIn $\\rAdaAPG$, \nlet $N$ be the total number of executions of $\\APGIter$.\nDenote\n\\begin{equation}\\label{simga*-def}\n\\sigma_* := \\left\\{\n\\begin{array}{ll}\n\\sigmainit & (\\text{if } \\sigmainit \\leq \\bar\\sigma),\\\\\n\\bar\\sigma/\\gamma_{\\rm reg} & (\\text{otherwise}),\n\\end{array}\n\\right.\n\\end{equation}\nfor  $\\bar\\sigma$ defined in \\eqref{bar-sigma-def}.\nAlso, let  $N(\\cdot,\\cdot,\\cdot)$ be defined by    \\eqref{N-def}.\nThen, the following assertions hold.\n\\begin{enumerate}[(i)]\n\\item If $\\rho=2$, then $N\\leq N(\\ep,\\sigma_*,C)$ holds with\n$\nC =\n\\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep}\\right).\n$\n\n\\item Suppose $\\rho>2$.\nIf $\\sigmainit\\geq\\bar\\sigma$, then  $N\\leq N(\\ep,\\sigma_*,C)$ holds with\n\\begin{align*}\nC=&\n\\sqrt{\\frac{1}{\\sigmainit}}\\left(1+\\frac{\\rho-1}{\\rho-2}\\log_{1/\\theta}\\frac{\\sigma_*^{(0)}}{\\min(\\sigma_*^{(0)},\\sigmainit)}\\right)+\n\\frac{\\sqrt{\\gamma_{\\rm reg}}}{1-\\sqrt{\\theta}^{\\frac{\\rho-2}{\\rho-1}}}\n\\left(\n\\sqrt{\\frac{1}{\\bar\\sigma}}\n-\n\\sqrt{\\theta}^{\\frac{\\rho-2}{\\rho-1}}\n\\sqrt{\\frac{1}{\\sigmainit}}\n\\right),\n\\end{align*}\nwhere\n$\n\\sigma_*^{(0)}:=\\frac{\\theta}{1+\\sqrt{2}\\beta}\\cdot\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}}\n\\|g_{\\outerM{0}}(\\xinit)\\|^{\\frac{\\rho-2}{\\rho-1}}.\n$\nOtherwise, it follows $\\outersigma{t}=\\sigmainit$ for all $t\\geq 0$ and $N\\leq N(\\ep,\\sigmainit,C)$ holds with\n$\nC= \n\\sqrt{\\frac{1}{\\sigmainit}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep}\\right).\n$\n\\item Suppose $\\rho\\in ]1,2[$.\nThen, we have\n\\begin{equation}\\label{N-bd-superlinear}\nN \\leq 1+ N(\\max(\\ep,\\ep_*),\\sigma_*,C) + \\left(\\log\\frac{1}{\\rho-1}\\right)^{-1}\\left(\\log\\log \\frac{\\ep_*}{\\theta^{\\frac{\\rho-1}{2-\\rho}}\\min(\\ep,\\ep_*)}-\\log\\log\\frac{1}{\\theta^{\\frac{\\rho-1}{2-\\rho}}}\\right),\n\\end{equation}\nwhere\n$\nC =\n\\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\max(\\ep,\\ep_*)}  \\right)\n$ and\n\\begin{equation}\\label{rho-12-ep*-def}\n\\ep_*=\n\\left[(1+\\sqrt{2})\\theta^{-1}(\\gamma_{\\rm inc}L_f+\\sigmainit)\\right]^{-\\frac{\\rho-1}{2-\\rho}}\n{\\kappa^{\\frac{1}{2-\\rho}}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{2-\\rho}}.\n\\end{equation}\n%\n\\item\n\\sloppy % prevents overflow\nIf $\\rho=1$, then we have\n$\nN \\leq 1 + N(\\max(\\ep,\\ep_*),\\sigma_*,C)\n$,\nwhere\n$\n\\ep_* = \\kappa\\left(\\frac{L_f}{L_{\\min}} + 1\\right)^{-1}\n$ and\n$\n{C=\n\\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\max(\\ep,\\ep_*)}\\right)}\n$.\nMoreover, if $\\ep<\\ep_*$, then $\\outerXplus{t}$ in the output of the algorithm must be an optimal solution to the original problem \\eqref{main-prob}.\n\\end{enumerate}\n\\end{theorem}","sha256_utf8":"d6c2e33a94daab73ce7dbcc4ea81a9e41a237601490befb5ed461ac356414c23"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\\label{lem:gmap}\nLet $\\varphi=f+\\varPsi$ be the sum of a continuously differentiable convex function $f:\\E\\to\\RR$ and a proper lower-semicontinuous convex function $\\varPsi:\\E\\to\\RR\\cup\\{+\\infty\\}$.\nThen, the following assertions hold.\n\\begin{enumerate}[(i)]\n\\item For all $y \\in \\E$, the map $L \\mapsto \\norm{g_L(y)}$ is increasing.\n%\n\\item $x^* \\in X^*$ holds if and only if $g_L(x^*)=0$.\n%\n\\item $\\nabla{f}(T_L(y))-\\nabla{f}(y)+g_L(y) \\in \\partial\\varphi(T_L(y))$ for all $y \\in \\E$. Moreover, if $f$ is $L_f$-smooth, then\n$$\n\\norm{\\nabla{f}(T_L(y))-\\nabla{f}(y)+g_L(y)} \\leq \\left(\\frac{L_f}{L}+1\\right)\\norm{g_L(y)}.\n$$\n%\n\\item If $y\\in \\E$ and $L>0$ satisfy $\\varphi(T_L(y))\\leq m_L(y;T_L(y))$, then we have the inequalities\n$$\n\\frac{1}{2L}\\norm{g_L(y)}^2 \\leq \\varphi(y)-\\varphi(T_L(y)) \\leq \\varphi(y)-\\varphi^*,\n\\qquad\n\\frac{1}{2L}\\norm{g_L(y)} \\leq \\dist(y,X^*).\n$$\n%\n\\item If $f$ is $L_f$-smooth, then $\\varphi(T_L(y))\\leq m_L(y;T_L(y))$ holds for all $y\\in \\E$ and $L\\geq L_f$.\n\\end{enumerate}\n\\end{lemma}\n\n\\begin{proof}\n(i) is proved in \\cite[Lemma 2]{Nes13}.\n\n\n(ii) According to the optimality condition of the problem $\\min_{x \\in \\E}m_L(y;x)$, we have the following equivalence for $y,z \\in \\E$:\n\\begin{equation}\\label{optcond-TL}\nz = T_L(y) \\iff 0 \\in \\nabla{f}(y)+L(z-y)+\\partial\\varPsi(z).\n\\end{equation}\nOn the other hand, the optimality of the original problem $\\min_{x}\\varphi(x)$ is characterized as follows. Given $w\\in \\E$, we have\n$w \\in \\Argmin_{x \\in \\E}\\varphi(x)$ if and only if $0 \\in \\nabla{f}(w)+\\partial{\\varPsi}(w)$ holds\n%$$\n%w \\in \\Argmin_{x \\in \\E}\\varphi(x) \\iff 0 \\in \\nabla{f}(w)+\\partial{\\varPsi}(w).\n%$$\n(Remark that we have $\\partial\\varphi=\\nabla{f}+\\partial \\varPsi$ \\cite[Theorem~23.8]{Roc}).\nLet us show the equivalence\n\\begin{equation}\\label{eq:gmap-opt-sol-ch}\nx^* \\in \\Argmin_{x \\in \\E}\\varphi(x) \\iff x^*=T_L(x^*) ~~(\\iff g_L(x^*)=0).\n\\end{equation}\nIf $x^*=T_L(x^*)$, then \\eqref{optcond-TL} implies $0 \\in \\nabla{f}(x^*)+\\partial \\varPsi(x^*)$ and so $x^* \\in \\Argmin_{x\\in \\E}\\varphi(x)$ holds. Conversely, if $x^*\\in \\Argmin_{x \\in \\E}\\varphi(x)$, then we have the condition $0\\in \\nabla{f}(x^*)+\\partial \\varPsi(x^*)$, which coincides with the right hand one of \\eqref{optcond-TL} in the case $y=z=x^*$. Thus, we conclude\n its equivalent counterpart $x^*=T_L(x^*)$. This proves the equivalence \\eqref{eq:gmap-opt-sol-ch} as well as the assertion (ii).\n\n\n(iii) By the condition \\eqref{optcond-TL}, we have\n$\n0 \\in \\nabla{f}(y)+L(T_L(y)-y) + \\partial\\varPsi(T_L(y)) = \\nabla{f}(y)-g_L(y)  + \\partial\\varPsi(T_L(y)).\n$\nHence, we obtain\n$\n\\nabla{f}(T_L(y))-\\nabla{f}(y)+g_L(y) \\in \\nabla{f}(T_L(y))+\\partial\\varPsi(T_L(y)) = \\partial\\varphi(T_L(y))\n$, giving the former assertion of (iii).\nNow if $f$ is $L_f$-smooth, we have\n$\n\\norm{\\nabla{f}(T_L(y))-\\nabla{f}(y)} \\leq L_f\\norm{T_L(y)-y} = \\frac{L_f}{L}\\norm{g_L(y)},\n$\nwhich yields the latter assertion of (iii).\n\n\n(iv)\nIt is shown in \\cite[Lemma~2.3]{BT09} that,\nif $\\varphi(T_L(y))\\leq m_L(y;T_L(y))$ holds, then we have\n$$\n\\varphi(x)  - \\varphi(T_L(y)) \\geq \\frac{1}{2L}\\norm{g_L(y)}^2 + \\innprod{g_L(y)}{x-y},\\quad \\forall x \\in \\E.\n$$\nLetting $x:=y$ shows the first assertion. On the other hand, since $\\varphi(T_L(y))\\geq \\varphi^*$, letting $x:=x^* \\in X^*$ gives\n$\n\\frac{1}{2L}\\norm{g_L(y)}^2 \\leq \\innprod{g_L(y)}{y-x^*} \\leq \\norm{g_L(y)}\\norm{y-x^*}.\n$\nThus, we obtain $\\frac{1}{2L}\\norm{g_L(y)} \\leq \\dist(y,X^*)$.\n\n(v) Note that $f$ is $L$-smooth for any $L\\geq L_f$.\nSince $L$-smooth functions satisfy (e.g., see \\cite{Nes04})\n$$\nf(x) \\leq f(y)+\\innprod{\\nabla{f}(y)}{x-y} + \\frac{L}{2}\\norm{x-y}^2,\\quad \\forall x,y \\in \\E,\n$$\nwe obtain $\\varphi(x) \\leq m_L(y;x)$ for all $x,y \\in \\E$ and $L\\geq L_f$.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:HEB}\nLet $\\varphi:\\E\\to\\RR\\cup\\{+\\infty\\}$ be a proper, lower-semicontinuous, and convex function.\nFor $x_0 \\in \\E$, suppose that $\\varphi$ satisfies the H\\\"olderian error bound condition \\eqref{HEB} with coefficient $\\kappa > 0$ and exponent $\\rho \\geq 1$.\nThen, for every $x \\in \\lev_{\\varphi}(\\varphi(x_0))\\setminus X^*$, we have\nthe inequalities\n\\begin{equation}\\label{Loj-ineq}\n\\kappa \\dist(x,X^*)^{\\rho-1} \\leq \\inf\\{\\norm{g}:g \\in \\partial\\varphi(x)\\},\\quad\n\\kappa^{\\frac{1}{\\rho}} (\\varphi(x)-\\varphi^*)^{\\frac{\\rho-1}{\\rho}} \\leq \\inf\\{\\norm{g}:g \\in \\partial\\varphi(x)\\}.\n\\end{equation}\n\\end{lemma}\n\n\\begin{proof}\nLet $x^*$ be the projection of $x$ onto $X^*$, so that $\\norm{x-x^*}=\\dist(x,X^*)$.\nFor every $g\\in\\partial \\varphi(x)$, we have\n\\begin{align*}\n\\kappa \\dist(x,X^*)^\\rho\n&\\leq \\varphi(x)-\\varphi^*\n\\leq -\\innprod{g}{x^*-x}\n\\leq \\norm{g}\\dist(x,X^*)\\leq \\norm{g}\\frac{1}{\\kappa^{1/\\rho}}(\\varphi(x)-\\varphi^*)^{1/\\rho}.\n\\end{align*}\nIt involves two inequalities\n$\n\\kappa \\dist(x,X^*)^\\rho \\leq \\norm{g}\\dist(x,X^*)$\nand\n$\\varphi(x)-\\varphi^* \\leq \\norm{g}\\frac{1}{\\kappa^{1/\\rho}}(\\varphi(x)-\\varphi^*)^{1/\\rho}.\n$\nSince $x \\not \\in X^*$, the assertion follows dividing them by $\\dist(x,X^*)$ and $\\kappa^{-1/\\rho}(\\varphi(x)-\\varphi^*)^{1/\\rho}$, respectively.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:reg}\n\\begin{enumerate}[(i)]\n\\item $\\varphi_\\sigma(x) \\leq \\varphi_\\sigma(x_0)$ implies $\\varphi(x) \\leq \\varphi(x_0)$.\n\\item $\\norm{x_0-x_\\sigma^*} \\leq \\dist(x_0,X^*)$.\n\\item We have $\\norm{g_L(y)-g_L^\\sigma(y)} \\leq  \\sigma\\norm{y-x_0}$ for any $y \\in \\E$.\n\\end{enumerate}\n\\end{lemma}\n\n\\begin{proof}\n(i) is immediate by $\\varphi(x)\\leq \\varphi_\\sigma(x)$ and $\\varphi_\\sigma(x_0)=\\varphi(x_0)$.\\\\\n(ii) For any $x^* \\in X^*$, the strong convexity of $\\varphi_\\sigma$ implies\n$\n\\varphi_\\sigma(x^*)\\geq \\varphi_\\sigma^* + \\frac{\\sigma}{2}\\norm{x^*-x_\\sigma^*}^2,\n$\nwhich can be rewritten as\n$\\varphi(x^*)-\\varphi(x_\\sigma^*) \\geq \\frac{\\sigma}{2}(-\\norm{x^*-x_0}^2 + \\norm{x_\\sigma^*-x_0}^2 + \\norm{x^*-x_\\sigma^*}^2).$\nAs $\\varphi(x^*)-\\varphi(x_\\sigma^*)\\leq 0$, we conclude $\\norm{x^*-x_0}^2 \\geq \\norm{x_\\sigma^*-x_0}^2 + \\norm{x^*-x_\\sigma^*}^2 \\geq \\norm{x_\\sigma^*-x_0}^2$, which proves the assertion.\\\\\n(iii)\nIn general, if $h$ is a lower-semicontinuous and $\\mu$-strongly convex function, we have for any $a,b \\in \\E$ that\\footnote{\nBy the strong convexity of $\\innprod{a}{x}+h(x)$ and $\\innprod{b}{x}+h(x)$, we have\n$$\n\\frac{\\mu}{2}\\norm{x_a^*-x_b^*}^2 \\leq [\\innprod{a}{x_b^*}+h(x_b^*)] - [\\innprod{a}{x_a^*}+h(x_a^*)]\n~~\\text{and}~~\n\\frac{\\mu}{2}\\norm{x_a^*-x_b^*}^2 \\leq  [\\innprod{b}{x_a^*}+h(x_a^*)]-[\\innprod{b}{x_b^*}+h(x_b^*)],\n$$\nrespectively.\nAdding them implies\n$\n\\mu\\norm{x_a^*-x_b^*}^2 \\leq \\innprod{a-b}{x_b^*-x_a^*} \\leq \\norm{a-b}\\norm{x_b^*-x_a^*}.\n$\n}\n$$\\norm{x_a^*-x_b^*}\\leq \\frac{\\norm{a-b}}{\\mu},~~\\text{where}~~\nx_a^* = \\argmin_{x \\in \\E}\\{\\innprod{a}{x}+h(x)\\},~~ x_b^* = \\argmin_{x \\in \\E}\\{\\innprod{b}{x}+h(x)\\}.$$\nThis fact implies\n$\n\\norm{T_L(y)-T_L^\\sigma(y)} \\leq \\frac{\\norm{\\nabla{f}(y)-\\nabla{f}_\\sigma(y)}}{L} = \\frac{\\sigma\\norm{y-x_0}}{L}.\n$\nTherefore, we conclude (iii) because of $\\norm{g_L(y)-g_L^\\sigma(y)}\n=  L\\norm{T_L(y)-T_L^\\sigma(y)}$.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:APG-iter-L}\nThe condition $\\eqref{APG-iter-test}$ of $\\APGIter$ holds whenever $L\\geq L_f$.\n\\end{lemma}\n\n\\begin{proof}\nThe first condition \\eqref{APG-iter-test-a} is satisfied since $f$ is $L$-smooth (e.g., see \\cite[Theorem 2.1.5]{Nes04}).\nSince $f_\\sigma$ is $(L_f+\\sigma)$-smooth, the second one \\eqref{APG-iter-test-b} holds if $L\\geq L_f$ by Lemma~\\ref{lem:gmap} (v).\nThe third one \\eqref{APG-iter-test-c} can be verified again by Lemma~\\ref{lem:gmap} (v): If $L\\geq L_f$, we have\n$\n\\varphi(T_L(z)) \\leq m_L(z;T_L(z)) \\leq \\varphi(z),\n$\nwhere the second inequality follows from\n\\begin{equation}\\label{mL-leq-phi}\nm_L(z;T_L(z))=\\min_{x \\in \\E}\\left\\{f(z)+\\innprod{\\nabla{f}(z)}{x-z}+\\varPsi(x)+\\frac{L}{2}\\norm{x-z}^2\\right\\} \\stackrel{x=z}{\\leq}\nf(z)+\\varPsi(z)=\\varphi(z).\n\\end{equation}\n\\end{proof}\n\n\\begin{lemma}\\label{lem:APG-iter-LS-compl}\n\\sloppy % prevents overflow\nSuppose that $L_k \\leq \\gamma_{\\rm inc}L_f$ holds.\nThen, $\\APGIter$ ensures the conditions $M_k \\leq \\gamma_{\\rm inc}L_f$ and ${L_{k+1} \\leq \\max\\{L_{\\min},\\frac{\\gamma_{\\rm inc}}{\\gamma_{\\rm dec}}L_f\\}\\leq \\gamma_{\\rm inc}L_f}$.\nThe number of executions of the loop {\\rm (Lines~\\ref{APG-iter-loop-s}--\\ref{APG-iter-loop-e})} is bounded by\n$$\n1 + \\frac{\\log \\gamma_{\\rm dec}}{\\log \\gamma_{\\rm inc}} + \\frac{1}{\\log \\gamma_{\\rm inc}} \\log \\frac{L_{k+1}}{L_k}.\n$$\n\\end{lemma}\n\n\\begin{proof}\n\\sloppy % prevents overflow\nLet $n_k \\geq 1$ be the number of inner loops so that ${M_k = L_k\\gamma_{\\rm inc}^{n_k-1}}$.\nIf $n_k=1$, then we have $M_k = L_k \\leq \\gamma_{\\rm inc}L_f$. If $n_k>1$, then $M_k\\leq \\gamma_{\\rm inc}L_f$ must hold because otherwise ${L_k\\gamma_{\\rm inc}^{n_k-2}=M_k/\\gamma_{\\rm inc} > L_f}$ and the $n_k$-th loop cannot occur (by Lemma~\\ref{lem:APG-iter-L}). Hence, we conclude $M_k \\leq \\gamma_{\\rm inc}L_f$ and also ${L_{k+1} = \\max\\{L_{\\min},M_k/\\gamma_{\\rm dec}\\} \\leq \\max\\{L_{\\min},\\frac{\\gamma_{\\rm inc}}{\\gamma_{\\rm dec}}L_f\\}}$.\n\nNow, the relation $M_k = L_k\\gamma_{\\rm inc}^{n_k-1}$ implies\n$\nL_{k+1} = \\max\\{L_{\\min},M_k/\\gamma_{\\rm dec}\\} \\geq \\frac{1}{\\gamma_{\\rm dec}}L_k\\gamma_{\\rm inc}^{n_k-1}.\n$\nThen, we have $(n_k-1)\\log\\gamma_{\\rm inc} \\leq \\log \\frac{\\gamma_{\\rm dec}L_{k+1}}{L_k} = \\log \\gamma_{\\rm dec} + \\log \\frac{L_{k+1}}{L_k}$ and therefore\n$\nn_k \\leq  1 + \\frac{\\log \\gamma_{\\rm dec}}{\\log \\gamma_{\\rm inc}} + \\frac{1}{\\log \\gamma_{\\rm inc}} \\log \\frac{L_{k+1}}{L_k} .\n$\n\\end{proof}\n\n\\begin{proposition}\\label{prop:APG}\nLet $\\{x_k,\\psi_k,M_k,L_k,A_k\\}$ be generated by the accelerated proximal gradient method applied to the regularized objective function \\eqref{realloc}, that is,\n$$\\{x_{k+1},\\psi_{k+1},M_k,L_{k+1},A_{k+1}\\} \\leftarrow \\APGIter_\\sigma(x_k,\\psi_k,L_k,A_k),\\quad k=0,1,2,\\ldots,$$\nwith the initialization $x_0 \\in \\E$, $\\psi_0(x)=\\frac{1}{2}\\norm{x-x_0}^2$, $A_0=0$, $L_0>0$.\n\\begin{enumerate}[(i)]\n\\item\nIf $L_0 \\in [L_{\\min}, \\gamma_{\\rm inc}L_f]$, then\nwe have $L_{\\min} \\leq L_k \\leq M_k \\leq \\gamma_{\\rm inc}L_f$ for all $k \\geq 0$.\nThe total number of executions of the loop {\\rm (Lines~\\ref{APG-iter-loop-s}--\\ref{APG-iter-loop-e})} until the $k$-th iteration is bounded by \n$$\n\\left[\n1+\\frac{\\log \\gamma_{\\rm dec}}{\\log \\gamma_{\\rm inc}}\\right](k+1) + \\frac{1}{\\log \\gamma_{\\rm inc}}\\log\\frac{L_{k+1}}{L_0}.\n$$\n\\item For each $k\\geq 1$, we have\n$\nA_k \\geq \\frac{2}{\\gamma_{\\rm inc}L_f}\\left[1+\\sqrt{\\frac{\\sigma}{2\\gamma_{\\rm inc}L_f}}\\right]^{2(k-1)}.\n$\n\\item $\\varphi_\\sigma(x_k)-\\varphi_\\sigma^* \\leq \\frac{\\norm{x_0-x_\\sigma^*}^2}{2A_k}$ holds for all $k\\geq 1$.\n\\item $\\varphi(x_k)\\leq \\varphi(x_0)$ holds for all $k\\geq 1$.\n\\item\nFor every $k\\geq 1$, we have\n$$\n\\norm{x_k-x_0} \\leq \\left(1+\\frac{1}{\\sqrt{\\sigma A_k}}\\right)\\dist(x_0,X^*),\n$$\n$$\n\\norm{g_{M_{k-1}}(x_k)} \\leq \\norm{g_{M_{k-1}+\\sigma}^\\sigma(x_k)} + \\sigma\\norm{x_k-x_0}\n\\leq \\left(2\\sqrt{\\frac{M_{k-1}+\\sigma}{A_k}} + \\sigma\\right)\\dist(x_0,X^*).\n$$\n\\end{enumerate}\n\\end{proposition}\n\n\\begin{proof}\n(i) \n$L_{\\min}\\leq L_k\\leq M_k$ is clear by the construction. $M_k\\leq \\gamma_{\\rm inc}L_f$\nis obtained by applying Lemma~\\ref{lem:APG-iter-LS-compl} inductively.\n\nSince $\\varPsi_\\sigma$ is $\\sigma$-strongly convex, \n(ii) follows by \\cite[Lemma~8]{Nes13} applied to the objective function $\\varphi_\\sigma=f+\\varPsi_\\sigma$.\n\nAccording to \\cite[Lemma 7]{Nes13}, the following relations hold for all $k\\geq 0$:\n$$\n\\left\\{\n\\begin{array}{l}\nA_k\\varphi_\\sigma(x_k) \\leq \\min_{x \\in \\E}\\psi_k(x),\\\\\n\\psi_k(x)\\leq A_k\\varphi_\\sigma(x)+\\frac{1}{2}\\norm{x-x_0}^2,\\quad \\forall x \\in \\E.\n\\end{array}\n\\right.\n$$\nCombining them, we obtain\n$$\nA_k\\varphi_\\sigma(x_k) \\leq \\min_{x\\in \\E}\\left\\{ A_k\\varphi_\\sigma(x)+\\frac{1}{2}\\norm{x-x_0}^2 \\right\\}, \\quad \\forall k\\geq 0.\n$$\nTaking $x=x_\\sigma^*$ on the right hand side, we obtain (iii).\nOn the other hand, taking $x=x_0$ yields ${\\varphi_\\sigma(x_k)\\leq \\varphi_\\sigma(x_0)}$. \nThen, Lemma~\\ref{lem:reg} (i) gives the assertion $\\varphi(x_k)\\leq \\varphi(x_0)$.\n\n(v)\nBy the $\\sigma$-strong convexity of $\\varphi_\\sigma$ and using (iii), we have\n$\n\\frac{\\sigma}{2}\\norm{x_k-x_\\sigma^*}^2 \\leq \\varphi_\\sigma(x_k)-\\varphi_\\sigma^* \\leq \\frac{\\norm{x_0 -x_\\sigma^*}^2}{2A_k},\n$\nand thus\n$\n\\norm{x_k-x_\\sigma^*} \\leq \\frac{1}{\\sqrt{\\sigma A_k}}\\norm{x_0-x_\\sigma^*}.\n$\nThis shows the former inequality of (v):\n$$\n\\norm{x_k-x_0}\\leq \\norm{x_k-x_\\sigma^*} + \\norm{x_0-x_\\sigma^*} \\leq \\left(1+\\frac{1}{\\sqrt{\\sigma A_k}}\\right) \\norm{x_0-x_\\sigma^*}\n\\leq\n\\left(1+\\frac{1}{\\sqrt{\\sigma A_k}}\\right) \\dist(x_0,X^*),\n$$\nwhere the last inequality follows by Lemma~\\ref{lem:reg} (ii).\n\nSince $\\varphi_\\sigma(T_{M_{k-1}+\\sigma}^\\sigma(x_k)) \\leq m_{M_{k-1}+\\sigma}^\\sigma(x_k;T_{M_{k-1}+\\sigma}^\\sigma(x_k))$ holds by \\eqref{APG-iter-test-b}, Lemma~\\ref{lem:gmap} (iv) yields\n$$\n\\frac{1}{2(M_{k-1}+\\sigma)}\\norm{g_{M_{k-1}+\\sigma}^\\sigma(x_k)}^2\n\\leq \\varphi_\\sigma(x_k)-\\varphi_\\sigma^* \\leq \\frac{\\norm{x_0-x_\\sigma^*}^2}{2A_k}\n\\leq \\frac{\\dist(x_0,X^*)^{2}}{2A_k}.\n$$\nThus, $\\norm{g_{M_{k-1}+\\sigma}^\\sigma(x_k)} \\leq \\sqrt{\\frac{M_{k-1}+\\sigma}{A_k}}\\dist(x_0,X^*)$ holds.\nConsequently, using Lemma~\\ref{lem:gmap}~(i) and Lemma~\\ref{lem:reg}~(iii), we obtain\n\\begin{align*}\n\\norm{g_{M_{k-1}}(x_k)} &\\leq \\norm{g_{M_{k-1}+\\sigma}(x_k)}\n\\leq \\norm{g_{M_{k-1}+\\sigma}^\\sigma(x_k)} + \\sigma\\norm{x_k - x_0}\n\\\\&\\leq  \\sqrt{\\frac{M_{k-1}+\\sigma}{A_k}}\\dist(x_0,X^*) + \\sigma\\left(1+\\frac{1}{\\sqrt{\\sigma A_k}}\\right)\\dist(x_0,X^*)\n\\\\&\\leq \\left(2\\sqrt{\\frac{M_{k-1}+\\sigma}{A_k}} + \\sigma\\right)\\dist(x_0,X^*).\n\\end{align*}\n\\end{proof}\n\n\\begin{lemma}\\label{lem:PG-iter}\nLet $\\{T_{M_k}(x_k), M_k, L_{k+1}\\}$ be given by $\\PGIter(x_k,L_k)$. Then, the following assertions hold.\n\\begin{enumerate}[(i)]\n\\item $\\varphi(T_{M_k}(x_k))\\leq \\varphi(x_k)$ holds.\n\\item\nIf $L_k \\leq \\gamma_{\\rm inc}L_f$, then we have $M_k \\leq \\gamma_{\\rm inc}L_f$ and $L_{k+1} \\leq \\max\\{L_{\\min},$ $\\frac{\\gamma_{\\rm inc}}{\\gamma_{\\rm dec}}L_f\\} \\leq \\gamma_{\\rm inc}L_f$.\nMoreover, the number of executions of the loop {\\rm (Lines~\\ref{PG-iter-loop-s}--\\ref{PG-iter-loop-e})} is bounded by\n$$\n 1 + \\frac{\\log \\gamma_{\\rm dec}}{\\log \\gamma_{\\rm inc}} + \\frac{1}{\\log \\gamma_{\\rm inc}} \\log \\frac{L_{k+1}}{L_k}.\n$$\n\\end{enumerate}\n\\end{lemma}\n\n\\begin{proof}\nSince the condition \\eqref{PG-iter-test} ensures\n$\\varphi(T_{M_k}(x_k))\\leq m_{M_k}(x_k;T_{M_k}(x_k))\\leq \\varphi(x_k)$\n(recall \\eqref{mL-leq-phi} for the second inequality), we conclude (i).\nThe assertion (ii) can be verified in the same way as Lemma~\\ref{lem:APG-iter-LS-compl}.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:AdaAPG-loop}\nIn $\\AdaAPG$,\nsuppose that a $j$-th outer loop is completed satisfying the criterion $A_{k+1}\\geq \\frac{2(M_{k}+\\sigma_j)}{\\beta^2\\sigma_j^2}$ for some $k\\geq 0$. Then, we have the inequalities\n$$\n\\norm{x_{k+1}-x_0} \\leq \\left(1+\\frac{\\beta}{\\sqrt{2}}\\right)\\dist(x_0,X^*),\n\\qquad\n\\norm{g_{M_{k}}(x_{k+1})} \\leq (1+\\sqrt{2}\\beta)\\sigma_j\\dist(x_0,X^*).\n$$\nMoreover, the number of inner iterations is bounded as follows.\n$$\nk+1 \\leq 2+ \\left(\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_j}}+1\\right)\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_j}{\\beta\\sigma_j}.\n$$\n\\end{lemma}\n\n\\begin{proof}\nThe bounds on $\\norm{x_{k+1}-x_0}$ and $\\norm{g_{M_{k}}(x_{k+1})}$ can be obtained by Proposition~\\ref{prop:APG} (v), applying the assumption $A_{k+1}\\geq \\frac{2(M_{k}+\\sigma_j)}{\\beta^2\\sigma_j^2}$.\nTo show the bound on $k+1$, suppose $k\\geq 1$ (the result is clear when $k=0$).\nBy the assumption on $k$ and Proposition~\\ref{prop:APG}~(i), we have\n$$A_{k} < \\frac{2(M_{k-1}+\\sigma_j)}{\\beta^2\\sigma_j^2} \\leq \\frac{2(\\gamma_{\\rm inc}L_f+\\sigma_j)}{\\beta^2\\sigma_j^2}.$$\nOn the other hand, Proposition~\\ref{prop:APG} (ii) implies\n$$\nA_{k} \\geq  \\frac{2}{\\gamma_{\\rm inc}L_f}\\left[1+\\sqrt{\\frac{\\sigma_j}{2\\gamma_{\\rm inc}L_f}}\\right]^{2(k-1)} \\geq\n \\frac{2}{\\gamma_{\\rm inc}L_f}\\exp\\left(2(k-1) \\frac{1}{\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_j}}+1}\\right),\n$$\nwhere the second inequality is due to the fact\\footnote{\nIn fact, since the derivative $\\log(1+x)$ of the function $h(x):=(1+x)\\log(1+x)-x$ is increasing and vanishes at $x=0$, we have $\\min_{x>-1}h(x)=h(0)=0$.\n}\n$1+x \\geq \\exp\\frac{x}{1+x}~(\\forall x > -1)$.\nTherefore,\n\\begin{align*}\n\\frac{2}{\\gamma_{\\rm inc}L_f}\\exp\\left(2(k-1) \\frac{1}{\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_j}}+1}\\right) \\leq \\frac{2(\\gamma_{\\rm inc}L_f+\\sigma_j)}{\\beta^2\\sigma_j^2}\\\\\n\\Longrightarrow\nk+1 \\leq 2+ \\frac{1}{2}\\left(\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_j}}+1\\right)\\log\\frac{\\gamma_{\\rm inc}L_f(\\gamma_{\\rm inc}L_f+\\sigma_j)}{\\beta^2\\sigma_j^2}.\n\\end{align*}\nThe assertion follows by relaxing $\\gamma_{\\rm inc}L_f \\leq \\gamma_{\\rm inc}L_f + \\sigma_j$.\n\\end{proof}\n\n\\begin{theorem}\\label{thm:AdaAPG}\nIn $\\AdaAPG$, let $N$ be the total number of executions of $\\APGIter$.\nDenote\n\\begin{equation}\\label{sigma-th}\n\\sigma(x_0,\\ep):=\\frac{\\ep}{(1+\\sqrt{2}\\beta)\\dist(x_0,X^*)},\n\\end{equation}\nwhere we let $\\sigma(x_0,\\ep):=+\\infty$ when $x_0 \\in X^*$.\nThen, the following assertions hold.\n\\begin{enumerate}[(i)]\n\\item\nThe algorithm terminates at the $j$-th outer loop whenever\n$\\sigma_j \\leq \\sigma(x_0,\\ep)$.\nMoreover, we have ${\\sigma_j \\geq \\sigma(x_0,\\ep)/\\gamma_{\\rm reg}}$ for all $j>0$.\n%\\begin{equation}\\label{sigma-j-lb}\n%\\sigma_j \\geq \\sigma(x_0,\\ep)/\\gamma_{\\rm reg},\\quad \\forall j>0.\n%\\end{equation}\n\n\\item\nSuppose that the algorithm terminates at $j=\\ell$ for some $\\ell \\geq 0$.\nThen, $N$ is at most\n\\begin{align*}\n&\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\sigma_\\ell}} - \\sqrt{\\frac{1}{\\sigma_0}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell}+ \\left(1+\\log_{\\gamma_{\\rm reg}}\\frac{\\sigma_0}{\\sigma_\\ell}\\right)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell}\\right).\n\\end{align*}\n%\n\\item\nIf $\\sigma_0\\leq \\sigma(x_0,\\ep)$, then\n$\nN \\leq 2+ \\left(\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_0}}+1\\right)\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_0}{\\beta\\sigma_0}.\n$\n\n%\n\\item\nIf $\\sigma_0 \\geq \\sigma(x_0,\\ep)$, then $N$ is at most\n\\begin{align}\n&\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\gamma_{\\rm reg}\\sqrt{\\frac{1}{\\sigma(x_0,\\ep)}} - \\sqrt{\\frac{1}{\\sigma_0}} \\right]\\log\\left(\\frac{\\gamma_{\\rm reg}\\gamma_{\\rm inc}L_f}{\\beta\\sigma(x_0,\\ep)}+\\frac{1}{\\beta}\\right) \\nonumber\\\\\n&+ \\left(2+\\log_{\\gamma_{\\rm reg}}\\frac{ \\sigma_0}{\\sigma(x_0,\\ep)}\\right)\\left[2+\\log\\left(\\frac{\\gamma_{\\rm reg}\\gamma_{\\rm inc}L_f}{\\beta\\sigma(x_0,\\ep)}+\\frac{1}{\\beta}\\right)\\right] \\nonumber\\\\\n=~&\nO\\left(\\sqrt{\\frac{L_f\\dist(x_0,X^*)}{\\ep}}\\log \\frac{L_f\\dist(x_0,X^*)}{\\ep} + \\log \\frac{\\sigma_0 \\dist(x_0,X^*)}{\\ep} \\log \\frac{L_f\\dist(x_0,X^*)}{\\ep}\\right).\n\\label{AdaAPG-compl-O}\n\\end{align}\n\\end{enumerate}\n\\end{theorem}\n\n\\begin{proof}\n(i)\n\\sloppy % prevents overflow\nBy Lemma~\\ref{lem:AdaAPG-loop}, the algorithm must terminate at the $j$-th loop whenever we have\n$\n{(1+\\sqrt{2}\\beta)\\sigma_j\\dist(x_0,X^*) \\leq \\ep}\n$,\nor equivalently,\n$\n\\sigma_j \\leq \\frac{\\ep}{(1+\\sqrt{2}\\beta)\\dist(x_0,X^*)}=\\sigma(x_0,\\ep).\n$\nThus, we obtain the former assertion of (i).\nTo see the latter assertion,\nassume that $\\sigma_j < \\sigma(x_0,\\ep)/\\gamma_{\\rm reg}$ holds for some $j > 0$. Then, the previous $(j-1)$-th loop satisfies $\\sigma_{j-1}< \\sigma(x_0,\\ep)$ so that the $j$-th loop is not executed by the former assertion.\nThis verifies the latter assertion of (i).\n\n(ii) Since $\\sigma_\\ell=\\sigma_0/\\gamma_{\\rm reg}^\\ell$, we have $\\ell = \\log_{\\gamma_{\\rm reg}}\\sigma_0/\\sigma_\\ell$.\nUsing Lemma~\\ref{lem:AdaAPG-loop}, $N$ is bounded as follows.\n\\begin{align*}\nN\n&\\leq\n\\sum_{j=0}^\\ell \\left[\n2+ \\left(\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_j}}+1\\right)\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_j}{\\beta\\sigma_j}\n\\right]\\\\\n&\\leq 2(\\ell+1) + \\sqrt{2\\gamma_{\\rm inc}L_f}\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell}\\sum_{j=0}^\\ell \\sqrt{\\frac{1}{\\sigma_j}}\n+ (\\ell+1)\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell},\n\\end{align*}\nwhere the second inequality is due to $\\sigma_j \\geq \\sigma_\\ell~(\\forall j \\leq \\ell)$.\nNote that\n\\begin{align*}\n\\sum_{j=0}^\\ell \\sqrt{\\frac{1}{\\sigma_j}}\n&=\n\\sum_{j=0}^\\ell \\sqrt{\\frac{1}{\\sigma_0/\\gamma_{\\rm reg}^j}}\n=\n\\sqrt{\\frac{1}{\\sigma_0}} \\sum_{j=0}^\\ell \\sqrt{\\gamma_{\\rm reg}}^{j}\n=\n\\sqrt{\\frac{1}{\\sigma_0}}\\frac{\\sqrt{\\gamma_{\\rm reg}}^{\\ell+1} -1}{\\sqrt{\\gamma_{\\rm reg}} -1}\\\\\n&=\n\\sqrt{\\frac{1}{\\sigma_0}}\\frac{\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\sigma_0/\\sigma_\\ell} -1}{\\sqrt{\\gamma_{\\rm reg}} -1}\n=\n\\frac{1}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\sigma_\\ell}} - \\sqrt{\\frac{1}{\\sigma_0}}\\right].\n\\end{align*}\nTherefore, we see that\n$$\nN\\leq (\\ell+1)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell}\\right)\n+\n\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\sigma_\\ell}} - \\sqrt{\\frac{1}{\\sigma_0}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell}.\n$$\nThe assertion follows by substituting $\\ell = \\log_{\\gamma_{\\rm reg}}{\\sigma_0}/{\\sigma_\\ell}$.\n\nIn view of (i), the assertions (iii) and (iv) follow by applying (ii) with $\\sigma_\\ell = \\sigma_0$ and $\\sigma_\\ell \\geq \\sigma(x_0,\\ep)/\\gamma_{\\rm reg}$, respectively.\nThe big-$O$ expression \\eqref{AdaAPG-compl-O} is obtained by substituting the definition of $\\sigma(x_0,\\ep)$.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:rAdaAPG-base}\nAssume that the H\\\"olderian error bound condition \\eqref{HEB-review} holds.\nIn the execution of $\\rAdaAPG$, the following assertions hold.\n\\begin{enumerate}[(i)]\n\\item $\\outerL{t},\\outerM{t} \\in [L_{\\min},\\gamma_{\\rm inc}L_f]$ for all $t\\geq 0$.\n\\item $\\varphi(\\outerXplus{t+1}) \\leq \\varphi(\\outerXplus{t}) \\leq \\cdots \\leq \\varphi(\\outerXplus{0}) \\leq \\varphi(\\xinit)$ for all $t\\geq 0$.\n\\item $t \\leq \\log_{1/\\theta}\\dfrac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\norm{g_{\\outerM{t}}(\\outerXinit{t})}}$ for each $t\\geq 0$.\n\\item Whenever $\\outerXplus{t} \\not\\in X^*$, we have\n\\begin{equation}\\label{rAdaAPG-dist-bound}\n\\dist(\\outerXplus{t},X^*)^{\\rho-1} \\leq \\frac{1}{\\kappa}\\left(\\frac{L_f}{L_{\\min}}+1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})},\n\\end{equation}\n\\begin{equation}\\label{rAdaAPG-obj-bound}\n(\\varphi(\\outerXplus{t})-\\varphi^*)^{\\rho-1} \\leq \\frac{1}{\\kappa}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^\\rho\\norm{g_{\\outerM{t}}(\\outerXinit{t})}^\\rho.\n\\end{equation}\n\\end{enumerate}\n\\end{lemma}\n\n\\begin{proof}\n(i)\nSince $\\Linit \\in [L_{\\min}, \\gamma_{\\rm inc}L_f]$, it follows $\\outerL{0},\\outerM{0}\\in [L_{\\min}, \\gamma_{\\rm inc}L_f]$ by Lemma~\\ref{lem:PG-iter}~(ii).\nThen, using Proposition~\\ref{prop:APG} (i) inductively, we obtain (i).\n\n(ii)\nFor $t=0$, we have $\\varphi(\\outerXplus{0})=\\varphi(T_{\\outerM{0}}(\\xinit))\\leq \\varphi(\\xinit)$ by Lemma~\\ref{lem:PG-iter} (i).\nMoreover, the criterion \\eqref{APG-iter-test-c} and Proposition~\\ref{prop:APG}~(iv), applied to the subroutine $\\AdaAPG$ in $\\rAdaAPG$, yield\n${\\varphi(\\outerXplus{t+1}) \\leq \\varphi(\\outerXinit{t+1})}$ and $\\varphi(\\outerXinit{t+1}) \\leq \\varphi(\\outerXplus{t})$, respectively. This shows $\\varphi(\\outerXplus{t+1}) \\leq \\varphi(\\outerXplus{t})$.\n\n\n(iii)\nSince the recurrence\n$\n\\ep^{(t)}/\\theta=\\norm{g_{\\outerM{t}}(\\outerXinit{t})} \\leq \\ep^{(t-1)}\n$\nholds for $t\\geq 1$, we have\n$\n\\ep^{(0)} \\geq \\ep^{(t)}/\\theta^t\n$\nfor each $t\\geq 0$.\nTherefore, we conclude\n$\nt \\leq \\log_{1/\\theta}\\frac{\\ep^{(0)}}{\\ep^{(t)}} = \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\norm{g_{\\outerM{t}}(\\outerXinit{t})}}.\n$\n\n\n(iv)\nSince $\\outerXplus{t}=T_{\\outerM{t}}(\\outerXinit{t})$, Lemma~\\ref{lem:gmap} (iii) implies that $g_t:=\\nabla{f}(\\outerXplus{t})-\\nabla{f}(\\outerXinit{t})+g_{\\outerM{t}}(\\outerXinit{t})$ belongs to $\\partial{\\varphi}(\\outerXplus{t})$.\nThen,\nLemma~\\ref{lem:HEB} shows (note that $\\outerXplus{t}$ belongs to $\\lev_\\varphi(\\varphi(\\xinit))\\setminus X^*$ by (ii))\n\\begin{align*}\n\\kappa\\dist(\\outerXplus{t},X^*)^{\\rho-1} &\\leq \\norm{g_t}\n\\leq \\left(\\frac{L_f}{\\outerM{t}}+1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})}\n\\leq \\left(\\frac{L_f}{L_{\\min}}+1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})},\n\\end{align*}\nwhere the second inequality is due to Lemma~\\ref{lem:gmap} (iii) and the last follows by (i). This shows the assertion \\eqref{rAdaAPG-dist-bound}. Similarly, \\eqref{rAdaAPG-obj-bound} can be obtained using the latter inequality of \\eqref{Loj-ineq}.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:rAdaAPG-compl}\nLet $N^{(t)}$ be the number of executions of $\\APGIter$ at the $t$-th outer loop of $\\rAdaAPG$.\nAssume that, for some $T_*\\geq 0$, $\\sigma_*>0$, and $\\ep_*>0$, we have\n$$\n\\outersigma{t+1}\\geq \\sigma_*\\quad (t=0,\\ldots,T_*)\n\\quad\n\\text{and}\n\\quad\n\\norm{g_{\\outerM{T_*}}(\\outerXinit{T_*})} \\geq \\ep_*.\n$$\nThen, under the definition \\eqref{N-def} of $N(\\cdot,\\cdot,\\cdot)$, the following inequality holds.\n$$\n\\sum_{t=0}^{T_*}N^{(t)} \\leq N(\\ep_*,\\sigma_*,C) \\leq N(\\ep_*,\\sigma_*,C_*),\n$$\nwhere\n$$\nC := \\sum_{t=0}^{T_*}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n\\leq C_*:= \\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*}\\right).\n$$\n\\end{lemma}\n\n\\begin{proof}\nSince $\\outersigma{t+1}\\geq \\sigma_*$ for each $t=0,\\ldots,T_*$, Theorem~\\ref{thm:AdaAPG} (ii) gives the following bound for $t=0,\\ldots,T_*$:\n\\begin{align*}\nN^{(t)}\n&\\leq \n\\left(1+\\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{t}}{\\outersigma{t+1}}\\right)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\outersigma{t+1}}{\\beta\\outersigma{t+1}}\\right)\\\\\n&\\quad +\n\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\outersigma{t+1}}} - \\sqrt{\\frac{1}{\\outersigma{t}}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\outersigma{t+1}}{\\beta\\outersigma{t+1}}\\\\\n&\\leq\n\\left(1+\\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{t}}{\\outersigma{t+1}}\\right)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}\\right)\\\\\n&\\quad +\n\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\outersigma{t+1}}} - \\sqrt{\\frac{1}{\\outersigma{t}}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}.\n\\end{align*}\nBy Lemma~\\ref{lem:rAdaAPG-base} (iii), $T_*$ is bounded by\n$$\nT_*\\leq \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\norm{g_{\\outerM{T_*}}(\\outerXinit{T_*})}} \\leq \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*}.\n$$\nTo estimate the sum $\\sum_{t=0}^{T_*}N^{(t)}$, note that\n$$\n\\sum_{t=0}^{T_*}\\left(1+\\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{t}}{\\outersigma{t+1}}\\right) = 1+T_*+\\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{0}}{\\outersigma{T_*+1}} \\leq 1+ \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*} + \\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{0}}{\\sigma_*},\n$$\nand\n\\begin{align*}\n\\sum_{t=0}^{T_*}\n\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\outersigma{t+1}}} - \\sqrt{\\frac{1}{\\outersigma{t}}}\\right]\n&=\n(\\sqrt{\\gamma_{\\rm reg}}-1)\\sum_{t=0}^{T_*}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n+\\sum_{t=0}^{T_*}\n\\left[\\sqrt{\\frac{1}{\\outersigma{t+1}}} - \\sqrt{\\frac{1}{\\outersigma{t}}}\\right]\\\\\n&=\n(\\sqrt{\\gamma_{\\rm reg}}-1)\\sum_{t=0}^{T_*}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n+\n\\sqrt{\\frac{1}{\\outersigma{T_*+1}}} - \\sqrt{\\frac{1}{\\outersigma{0}}}.\n\\end{align*}\nTherefore, $\\sum_{t=0}^{T_*}N^{(t)}$ is bounded by\n\\begin{align*}\n\\sum_{t=0}^{T_*}N^{(t)}\n&\\leq \\left(1+ \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*} + \\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{0}}{\\sigma_*}\\right)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}\\right)\\\\\n&\\quad +\n\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\n\\sqrt{\\frac{1}{\\sigma_*}} - \\sqrt{\\frac{1}{\\outersigma{0}}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*} \\\\\n&\\quad + \\sqrt{2\\gamma_{\\rm inc}L_f} \\sum_{t=0}^{T_*}\\sqrt{\\frac{1}{\\outersigma{t+1}}} \\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}\\\\\n&=N(\\ep_*,\\sigma_*,C).\n\\end{align*}\nFinally, $C$ has the following bound:\n$$\nC = \\sum_{t=0}^{T_*}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n\\leq \\sqrt{\\frac{1}{\\sigma_*}} (1+T_*)\\leq\n\\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*}\\right)=C_*,\n$$\nwhich also concludes $N(\\ep_*,\\sigma_*,C) \\leq N(\\ep_*,\\sigma_*,C_*)$.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:rAdaAPG-sigma}\nAssume that the H\\\"olderian error bound condition \\eqref{HEB-review} holds.\nSuppose that $\\rAdaAPG$ terminated with the stopping criterion at the $(T+1)$-th outer loop for some $T\\geq 0$.\nLet $\\bar\\sigma$ be defined by \\eqref{bar-sigma-def}.\n\\begin{enumerate}[(i)]\n\\item\nFor each $t=0,\\ldots,T$, we have\n$$\n\\outersigma{t+1}\n\\left\\{\n\\begin{array}{ll}\n=\\outersigma{t}&(\\outersigma{t}\\leq \\sigma(\\outerXplus{t},\\ep^{(t)})),\\\\\n\\geq\\sigma(\\outerXplus{t},\\ep^{(t)})/\\gamma_{\\rm reg}&(\\text{otherwise}),\n\\end{array}\n\\right.\n$$\nwhere $\\sigma(\\cdot,\\cdot)$ is defined by \\eqref{sigma-th}. \n\n\\item It follows that $\\sigma(\\outerXplus{t},\\ep^{(t)}) \\geq \\bar\\sigma$ for all $t=0,\\ldots,T$.\n%\\begin{equation}\\label{sigma-th-const}\n%\\sigma(\\outerXplus{t},\\ep^{(t)}) \\geq \\bar\\sigma, \\quad t=0,\\ldots,T.\n%\\end{equation}\nWhen $\\rho \\geq 2$, we further obtain\n\\begin{equation}\\label{sigma-th-t}\n\\sigma(\\outerXplus{t},\\ep^{(t)}) \\geq \\frac{\\theta^{\\frac{1}{\\rho-1}}}{1+\\sqrt{2}\\beta}\\cdot\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}}\n(\\ep^{(t)})^{\\frac{\\rho-2}{\\rho-1}}\n \\geq \\bar\\sigma,\n\\end{equation}\nfor each $t=0,\\ldots,T$.\n\\item\nFor each $t=0,\\ldots,T+1$, we obtain\n$$\n\\outersigma{t} \\left\\{\n\\begin{array}{ll}\n=\\sigmainit & (\\sigmainit \\leq \\bar\\sigma),\\\\\n\\geq\\bar\\sigma/\\gamma_{\\rm reg} & (\\text{otherwise}).\n\\end{array}\n\\right.\n$$\n\\end{enumerate}\n\\end{lemma}\n\n\\begin{proof}\n(i)\nIn the case $\\outersigma{t} \\leq \\sigma(\\outerXplus{t},\\ep^{(t)})$, Theorem~\\ref{thm:AdaAPG} (i) implies that the subroutine $\\AdaAPG$ at the $t$-th outer loop must terminate at the first loop $j=0$ so that $\\outersigma{t+1}$ is defined by $\\outersigma{t+1}=\\outersigma{t}$.\nOn the other hand, consider the case $\\outersigma{t} > \\sigma(\\outerXplus{t},\\ep^{(t)})$.\nWhen the subroutine $\\AdaAPG$ at the $t$-th outer loop terminates at the first loop $j=0$, then it is clear that\n$\\outersigma{t+1}=\\outersigma{t} > \\sigma(\\outerXplus{t},\\ep^{(t)}) \\geq \\sigma(\\outerXplus{t},\\ep^{(t)})/\\gamma_{\\rm reg}.$\nIn the other case, the latter assertion of Theorem~\\ref{thm:AdaAPG}~(i) implies $\\outersigma{t+1}\\geq \\sigma(\\outerXplus{t},\\ep^{(t)})/\\gamma_{\\rm reg}.$\n\n(ii)\nWe may assume $\\outerXplus{t} \\not\\in X^*$, which allows us to apply Lemma~\\ref{lem:rAdaAPG-base} (iv) (Note that the assertion is clear if $\\outerXplus{t} \\in X^*$ since then $\\sigma(\\outerXplus{t},\\ep^{(t)})=+\\infty$).\nIn the case $\\rho\\geq 2$, using \\eqref{rAdaAPG-dist-bound} implies\n$$\n\\sigma(\\outerXplus{t},\\ep^{(t)})\n= \\frac{\\theta\\norm{g_{\\outerM{t}}(\\outerXinit{t})}}{(1+\\sqrt{2}\\beta)\\dist(\\outerXplus{t},X^*)}\n\\geq\n\\frac{\\theta}{1+\\sqrt{2}\\beta}\\cdot\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}}\n\\norm{g_{\\outerM{t}}(\\outerXinit{t})}^{\\frac{\\rho-2}{\\rho-1}}.\n$$\nSince $\\ep^{(t)}\\equiv \\theta\\|g_{\\outerM{t}}(\\outerXinit{t})\\|$ and $\\|g_{\\outerM{t}}(\\outerXinit{t})\\| \\geq \\ep$ (by the definition of $T$), we obtain \\eqref{sigma-th-t}.\n\nIf $\\rho\\in [1,2[$, on the other hand,\nsince $\\varphi(\\outerXplus{t})\\leq \\varphi(\\xinit)$ by Lemma~\\ref{lem:rAdaAPG-base} (ii), the inequalities \\eqref{HEB-review} and \\eqref{rAdaAPG-obj-bound} imply (remark that $\\frac{2-\\rho}{\\rho}>0$)\n\\begin{align*}\n\\dist(\\outerXplus{t},X^*) &\\leq \\frac{1}{\\kappa^{\\frac{1}{\\rho}}}(\\varphi(\\outerXplus{t})-\\varphi^*)^{\\frac{1}{\\rho}}\n=\n\\frac{1}{\\kappa^{\\frac{1}{\\rho}}}(\\varphi(\\outerXplus{t})-\\varphi^*)^{\\frac{\\rho-1}{\\rho}}(\\varphi(\\outerXplus{t})-\\varphi^*)^{\\frac{2-\\rho}{\\rho}}\\\\\n&\\leq\n\\frac{1}{\\kappa^{\\frac{2}{\\rho}}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})}(\\varphi(\\outerXplus{t})-\\varphi^*)^{\\frac{2-\\rho}{\\rho}}\\\\\n&\\leq\n\\frac{1}{\\kappa^{\\frac{2}{\\rho}}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})}(\\varphi(\\xinit)-\\varphi^*)^{\\frac{2-\\rho}{\\rho}}.\n\\end{align*}\nTherefore, we conclude that\n$$\n\\sigma(\\outerXplus{t},\\ep^{(t)})\n= \\frac{\\theta\\norm{g_{\\outerM{t}}(\\outerXinit{t})}}{(1+\\sqrt{2}\\beta)\\dist(\\outerXplus{t},X^*)}\n\\geq\n\\frac{\\theta}{1+\\sqrt{2}\\beta}\n\\kappa^{\\frac{2}{\\rho}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-1}(\\varphi(\\xinit)-\\varphi^*)^{-\\frac{2-\\rho}{\\rho}}\n=\\bar\\sigma.\n$$\nThis proves the assertion (ii).\n\n(iii)\nIn the case $\\sigmainit\\leq \\bar\\sigma$, (ii) implies $\\outersigma{0} \\leq \\sigma(\\outerXplus{0},\\ep^{(0)})$. Then, using (i), we have $\\outersigma{1}=\\outersigma{0}$ and also $\\outersigma{1} \\leq \\bar\\sigma$.\nContinuing this argument inductively, we conclude that $\\outersigma{t}=\\outersigma{t-1}=\\cdots=\\outersigma{0}$ for all $t$.\n\nIn the case $\\sigmainit \\geq \\bar\\sigma$, on the other hand, let us show $\\outersigma{t} \\geq \\bar\\sigma/\\gamma_{\\rm reg}~(t=0,\\ldots,T+1)$ by induction. The assertion for $t=0$ is clear since $\\sigmainit \\geq \\bar\\sigma \\geq \\bar\\sigma/\\gamma_{\\rm reg}$.\nAssume that $\\outersigma{t}\\geq \\bar\\sigma/\\gamma_{\\rm reg}$ holds for some $t\\geq 0$. By (i) and (ii), we have\n$\n\\outersigma{t+1} \\geq \\min\\{\\outersigma{t}, \\sigma(\\outerXplus{t},\\ep^{(t)})/\\gamma_{\\rm reg}\\} \\geq \\min\\{\\bar\\sigma/\\gamma_{\\rm reg}, \\bar\\sigma/\\gamma_{\\rm reg}\\} =\\bar\\sigma/\\gamma_{\\rm reg}.\n$\nThis completes the proof of (iii).\n\\end{proof}\n\n\\begin{lemma}\\label{lem:rAdaAPG-sigma-2}\nAssume that the H\\\"olderian error bound condition \\eqref{HEB-review} holds with $\\rho >2$.\nSuppose that $\\rAdaAPG$ terminated with the stopping criterion at the $(T+1)$-th outer loop for some $T\\geq 0$.\nIf $\\sigmainit \\geq \\bar\\sigma$ holds for $\\bar\\sigma$  defined by \\eqref{bar-sigma-def},\nthen there exists $t_0 \\in \\{0,\\ldots,T+1\\}$ such that the following conditions hold, where\n$\n\\sigma_*^{(0)}:=\\frac{\\theta}{1+\\sqrt{2}\\beta}\\cdot\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}}\n\\|g_{\\outerM{0}}(\\xinit)\\|^{\\frac{\\rho-2}{\\rho-1}}.\n$\n\\begin{align}\n\\text{(i)\\qquad} & \\outersigma{0}=\\cdots=\\outersigma{t_0}. \\label{sigma-bound-rho>2-1}\\\\\n\\text{(ii)\\qquad} & \\outersigma{t+1} \\geq \\theta^{-\\frac{\\rho-2}{\\rho-1}(T-t)} \\bar\\sigma/\\gamma_{\\rm reg},\\quad t=t_0,\\ldots,T.\\label{sigma-bound-rho>2-2}\\\\\n\\text{(iii)\\qquad} & t_0 \\leq 1+\\frac{\\rho-1}{\\rho-2}\\log_{1/\\theta}\\frac{\\sigma_*^{(0)}}{\\min(\\sigmainit,\\sigma_*^{(0)})}.\\label{sigma-bound-rho>2-3}\\\\\n\\text{(iv)\\qquad} & \\theta^{\\frac{\\rho-2}{\\rho-1}(T-t_0)} \\geq {\\bar\\sigma}/{\\outersigma{0}}.\\label{sigma-bound-rho>2-4}\n\\end{align}\n\n\\end{lemma}\n\n\\begin{proof}\nDefine\n$$\n\\sigma_*^{(t)}:=\\frac{\\theta^{\\frac{1}{\\rho-1}}}{1+\\sqrt{2}\\beta}\\cdot\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}}\n(\\ep^{(t)})^{\\frac{\\rho-2}{\\rho-1}},\\quad t \\geq 0.\n$$\nThen, Lemma~\\ref{lem:rAdaAPG-sigma} (ii) can be written as\n\\begin{equation}\\label{simga_*^t-bound}\n\\sigma(\\outerXplus{t},\\ep^{(t)}) \\geq \\sigma_*^{(t)} \\geq \\bar\\sigma,\\quad t=0,\\ldots,T.\n\\end{equation}\nFor simplicity, denote\n$$\n\\omega:=\\frac{\\rho-2}{\\rho-1} \\in ]0,1[,\\quad\nc:=\\frac{\\theta^{\\frac{1}{\\rho-1}}}{1+\\sqrt{2}\\beta}\\cdot\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}},\n$$\nso that we have\n$$\n\\bar\\sigma = c(\\theta\\ep)^{\\omega},\\quad\n\\sigma_*^{(t)}=c(\\ep^{(t)})^{\\omega},\\quad t\\geq 0.\n$$\nNote that $\\{\\sigma_*^{(t)}\\}$ is non-increasing;\nin fact, the relation $\\ep^{(t+1)}\\leq \\theta\\ep^{(t)}$ implies\n\\begin{equation}\\label{sigma-*-mono}\n\\sigma_*^{(t+1)}=c(\\ep^{(t+1)})^\\omega\\leq c\\theta^\\omega(\\ep^{(t)})^\\omega = \\theta^\\omega \\sigma_*^{(t)} \\leq \\sigma_*^{(t)}.\n\\end{equation}\n\nLet $t_0$ be the smallest integer in $\\{0,\\ldots,T+1\\}$ such that $\\sigmainit \\geq \\sigma_*^{(t_0)}$.\nRemark that, by the definition of $T$, we have $\\ep^{(T+1)}=\\theta\\|g_{\\outerM{T+1}}(\\outerXinit{T+1})\\| \\leq \\theta\\ep$.\nThis implies that\n$\n\\sigmainit \\geq \\bar\\sigma= c(\\theta\\ep)^\\omega \\geq c(\\ep^{(T+1)})^\\omega = \\sigma_*^{(T+1)}.\n$\nTherefore, $t_0$ is well-defined.\n\n%%%\n(i)\nBy the definition of $t_0$ and \\eqref{simga_*^t-bound}, we have\n$\\sigmainit < \\sigma_*^{(t)} \\leq \\sigma(\\outerXplus{t},\\ep^{(t)})$ ($0\\leq \\forall t < t_0$).\nTherefore, by induction, we obtain $\\outersigma{0}=\\outersigma{1}=\\cdots=\\outersigma{t_0}$ due to Lemma~\\ref{lem:rAdaAPG-sigma}~(i).\n\n\n(ii)\nLet us show $\\outersigma{t+1}\\geq \\sigma_*^{(t)}/\\gamma_{\\rm reg}~(t_0\\leq t\\leq T)$.\nTo prove this, we verify $\\outersigma{t}\\geq \\sigma_*^{(t)}/\\gamma_{\\rm reg}$ for ${t= t_0,\\ldots,T+1}$ by induction. Note that $\\outersigma{t_0}=\\outersigma{0} \\geq  \\sigma_*^{(t_0)} > \\sigma_*^{(t_0)}/\\gamma_{\\rm reg}$ holds by (i) and the definition of $t_0$. Now under the hypothesis $\\outersigma{t} \\geq  \\sigma_*^{(t)}/\\gamma_{\\rm reg}$ for $t$ with $t_0\\leq t \\leq T$, Lemma~\\ref{lem:rAdaAPG-sigma}~(i) and \\eqref{simga_*^t-bound} imply\n$$\\outersigma{t+1} \\geq \\min\\{\\outersigma{t},\\sigma(\\outerXplus{t},\\ep^{(t)})/\\gamma_{\\rm reg}\\} \\geq \\min\\{\\sigma_*^{(t)}/\\gamma_{\\rm reg},\\sigma_*^{(t)}/\\gamma_{\\rm reg}\\} = \\sigma_*^{(t)}/\\gamma_{\\rm reg} \\geq \\sigma_*^{(t+1)}/\\gamma_{\\rm reg}.$$\nTherefore, this completes the induction; in addition, the above inequality involves the desired inequality $\\outersigma{t+1}\\geq \\sigma_*^{(t)}/\\gamma_{\\rm reg}~(t_0\\leq t\\leq T)$.\nThis yields (ii) combined with \\eqref{simga_*^t-bound} and \\eqref{sigma-*-mono}:\n$$\n\\outersigma{t+1} \\geq \\sigma_*^{(t)}/\\gamma_{\\rm reg} \\geq \\theta^{-\\omega(T-t)}\\sigma_*^{(T)}/\\gamma_{\\rm reg} \\geq \\theta^{-\\omega(T-t)} \\bar\\sigma/\\gamma_{\\rm reg},\\quad t=t_0,\\ldots,T.\n$$\n%%%\n\n(iii)\nIf $t_0=0$, then\n(iii) is trivial since\n$\\sigmainit \\geq\\sigma_*^{(t_0)}=\\sigma_*^{(0)}$. If $t_0 > 0$, then the definition of $t_0$ and using \\eqref{sigma-*-mono} imply\n$\\sigmainit < \\sigma_*^{(t_0-1)} \\leq \\theta^{\\omega(t_0-1)}\\sigma_*^{(0)},$\nwhich yields $t_0 \\leq 1+\\frac{1}{\\omega}\\log_{1/\\theta}\\frac{\\sigma_*^{(0)}}{\\sigmainit}$.\n\n\n(iv)\nBy \\eqref{simga_*^t-bound}, \\eqref{sigma-*-mono}, and the definition of $t_0$, remark that\n$\n\\bar\\sigma \\leq \\sigma_*^{(T)} \\leq \\theta^{\\omega(T-t_0)}\\sigma_*^{(t_0)}\\leq \\theta^{\\omega(T-t_0)}\\sigmainit.\n$\nHence, \n$\n\\theta^{\\omega(T-t_0)} \\geq {\\bar\\sigma}/{\\outersigma{0}}\n$\nholds.\n\\end{proof}\n\n\\begin{proof}[\\underline{Proof of Theorem~\\ref{thm:rAdaAPG}}]\nWe may assume that $\\norm{g_{\\outerM{0}}(\\xinit)} > \\ep$ since $N=0$ on the other case.\nSuppose that $\\rAdaAPG$ terminated with the stopping criterion at the $(T+1)$-th outer loop for some $T\\geq 0$.\nDenote by $N^{(t)}$ the number of executions of $\\APGIter$ at the $t$-th outer loop so that $N=\\sum_{t=0}^T N^{(t)}$.\nLet $\\bar\\sigma$ be defined by \\eqref{bar-sigma-def}.\n\nBy the definition of $T$, we have\n\\begin{equation}\\label{pr:gradmap-lb-ep}\n\\norm{g_{\\outerM{t}}(\\outerXinit{t})} > \\ep,\\quad t=0,\\ldots,T.\n\\end{equation}\nMoreover, by the definition of $\\sigma_*$ in \\eqref{simga*-def},\nusing Lemma~\\ref{lem:rAdaAPG-sigma} (iii) implies\n$$\n\\outersigma{t} \\geq\n\\sigma_*,\\quad t=0,\\ldots,T+1.\n$$\nTherefore, applying Lemma~\\ref{lem:rAdaAPG-compl} with $T_*=T$, we obtain the assertion\n\\begin{equation}\\label{N-ub-basic-HEB}\nN \\leq N(\\ep,\\sigma_*,C) \\quad \\text{with}\\quad\nC=\\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep}\\right).\n\\end{equation}\nFor the case $\\rho=2$, \\eqref{N-ub-basic-HEB} proves the assertion (i).\nWe discuss the other cases to improve \\eqref{N-ub-basic-HEB}.\n\n\\fbox{(ii) Case $\\rho>2$.}\nIf $\\sigmainit < \\bar\\sigma$, then $\\sigma_*$ is defined as $\\sigma_*=\\sigmainit$. \nTherefore, the latter bound of (ii) is obtained by \\eqref{N-ub-basic-HEB}.\n\nNow consider the case $\\sigmainit \\geq \\bar\\sigma$.\nThen, there exists $t_0 \\in \\{0,\\ldots,T+1\\}$ satisfying the conditions in Lemma~\\ref{lem:rAdaAPG-sigma-2}.\nBy \\eqref{sigma-bound-rho>2-4}, remark that\n\\begin{equation}\\label{rAdaAPG-compl-p>2-pf-sum}\n\\sum_{t=t_0}^{T}\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}(T-t)}}\n=\n\\sum_{i=0}^{T-t_0}\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}}^{\\,i}\n=\n\\frac{1-\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}}^{T-t_0+1}}{1-\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}}}\n\\leq\n\\frac{1-\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}\\cdot\\dfrac{\\bar\\sigma}{\\outersigma{0}}}}{1-\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}}}.\n\\end{equation}\nTherefore, we conclude that\n\\begin{align*}\n\\sum_{t=0}^{T}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n&=\n\\sum_{t=0}^{t_0-1} \\sqrt{\\frac{1}{\\outersigma{t+1}}}+\n\\sum_{t=t_0}^{T}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n\\\\\n&\\leq\nt_0 \\sqrt{\\frac{1}{\\outersigma{0}}} +\n\\sqrt{\\frac{\\gamma_{\\rm reg}}{\\bar\\sigma}}\\sum_{t=t_0}^{T}\\sqrt{\\theta^{{\\frac{\\rho-2}{\\rho-1}}(T-t)}} & (\\text{by } \\eqref{sigma-bound-rho>2-1} \\text{ and } \\eqref{sigma-bound-rho>2-2})\\\\\n&\\leq\n\\sqrt{\\frac{1}{\\outersigma{0}}}\\left(1+\\frac{\\rho-1}{\\rho-2}\\log_{1/\\theta}\\frac{\\sigma_*^{(0)}}{\\min(\\sigma_*^{(0)},\\outersigma{0})}\\right)\\\\\n&\\quad~+\n\\frac{\\sqrt{\\gamma_{\\rm reg}}}{1-\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}}}\n\\left(\n\\sqrt{\\frac{1}{\\bar\\sigma}}\n-\n\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}\n}}\n\\sqrt{\\frac{1}{\\outersigma{0}}}\n\\right) & (\\text{by } \\eqref{sigma-bound-rho>2-3} \\text{ and } \\eqref{rAdaAPG-compl-p>2-pf-sum})\\\\\n&=:C.\n\\end{align*}\nWith this definition of $C$, Lemma~\\ref{lem:rAdaAPG-compl} gives $N \\leq N(\\ep,\\sigma_*,C)$.\n\n\n\\fbox{(iii) Case $\\rho \\in ]1,2[$.}\nIf $\\ep \\geq \\ep_*$, then \\eqref{N-ub-basic-HEB} gives our assertion because the second term of \\eqref{N-bd-superlinear} vanishes.\nSuppose, on the other hand, that $\\ep < \\ep_*$. \nDenote\n$\\xi_{t}:=\\norm{g_{\\outerM{t}}(\\outerXinit{t})}$\nand \nlet $T_*\\geq 0$ be the smallest integer such that $\\xi_{T_*} \\leq \\ep_*$.\nThen, since $\\xi_{T_*-1} > \\ep_*$ holds, Lemma~\\ref{lem:rAdaAPG-compl} shows that\n$$\n\\sum_{t=0}^{T_*-1} N^{(t)} \\leq N(\\ep_*,\\sigma_*,C)\\quad \\text{with}\\quad C=\\sqrt{\\frac{1}{\\sigma_*}}\\left( 1 + \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*} \\right).\n$$\nNote that, under the convention $\\sum_{t=0}^{-1}(\\cdot)=0$, this inequality also holds if $T_*=0$ since then $\\xi_0\\leq \\ep_*$ and $N(\\ep_*,\\sigma_*,C)=0$.\n\nIt remains to observe $\\sum_{t=T_*}^T N^{(t)}$.\nTake $t\\in \\{T_*,\\ldots,T\\}$. We shall prove $N^{(t)}=1$.\nIn the $t$-th outer loop, consider the first iteration of the subroutine $\\AdaAPG(\\outerXplus{t},\\outerL{t},\\outersigma{t},\\ep^{(t)})$, which executes\n$$\n\\{x_1,\\psi_1,M_0,L_1,A_1\\} \\leftarrow \\APGIter_{\\outersigma{t}}(\\outerXplus{t},\\psi_0,\\outerL{t},A_0),\n$$\nwhere $\\psi_0(x)=\\frac{1}{2}\\|x-\\outerXplus{t}\\|^2$ and $A_0=0$.\nThen, Proposition~\\ref{prop:APG} (v) implies\n$$\n\\norm{g_{M_{0}}(x_1)} \\leq \\left(2\\sqrt{\\frac{M_{0}+\\outersigma{t}}{A_1}} + \\outersigma{t}\\right)\\dist(\\outerXplus{t},X^*).\n$$\nMoreover, according to the equation at Line~\\ref{a-quad-eq} in Algorithm~\\ref{APG-iter}, $A_1$ can be calculated as $A_1=2/M_0$. Now remark that, using $M_0 \\leq \\gamma_{\\rm inc}L_f$ (Proposition~\\ref{prop:APG}~(i)), we have\n\\begin{align*}\n2\\sqrt{\\frac{M_{0}+\\outersigma{t}}{A_1}} + \\outersigma{t} &= 2\\sqrt{\\frac{M_0(M_0+\\outersigma{t})}{2}} + \\outersigma{t}\n\\leq 2\\sqrt{\\frac{(M_0+\\outersigma{t})^2}{2}} + (M_0+\\outersigma{t}) \\\\&= (1+\\sqrt{2})(M_0+\\outersigma{t}) \\leq (1+\\sqrt{2})(\\gamma_{\\rm inc}L_f + \\outersigma{t}).\n\\end{align*}\nCombining them and using \\eqref{rAdaAPG-dist-bound}\\footnote{Although \\eqref{rAdaAPG-dist-bound} is asserted in the case $\\outerXplus{t} \\not\\in X^*$, it trivially holds if $\\outerXplus{t} \\in X^*$ \\emph{unless} $\\rho = 1$.}, we conclude that\n\\begin{align}\n\\norm{g_{M_{0}}(x_1)}\n&\\leq\n(1+\\sqrt{2})(\\gamma_{\\rm inc}L_f+\\outersigma{0})\\dist(\\outerXplus{t},X^*) \\nonumber\\\\\n&\\leq\n(1+\\sqrt{2})(\\gamma_{\\rm inc}L_f+\\outersigma{0})\n\\left(\\frac{1}{\\kappa}\\right)^{\\frac{1}{\\rho-1}} \\left(\\frac{L_f}{L_{\\min}}+1\\right)^{\\frac{1}{\\rho-1}}\\xi_t^{\\frac{1}{\\rho-1}}\n\\nonumber\\\\\n&= \\theta\\ep_*^{\\frac{\\rho-2}{\\rho-1}}\\xi_t^{\\frac{1}{\\rho-1}}\n\\label{superlinear-rec}\\\\\n&\\leq \\theta\\xi_t^{\\frac{\\rho-2}{\\rho-1}}\\xi_t^{\\frac{1}{\\rho-1}}=\\theta\\xi_t = \\ep^{(t)},\n\\nonumber\n\\end{align}\nwhere the last inequality is due to $\\xi_t \\leq \\ep_*$ for $t \\geq T_*$ (and remark $\\frac{\\rho-2}{\\rho-1} < 0$).\nThis shows $N^{(t)}=1$ and \\eqref{superlinear-rec} yields the recurrence\n$\n\\xi_{t+1} \\leq \\theta\\ep_*^{\\frac{\\rho-2}{\\rho-1}}\\xi_t^{\\frac{1}{\\rho-1}}\n$,\nfor each $t=T_*,\\ldots,T$.\nSince $\\frac{1}{\\rho-1} > 1$, it reduces $\\xi_t$ superlinearly. In particular, solving this recurrence implies\n$$\n\\log \\frac{\\ep_*}{\\theta^{\\frac{\\rho-1}{2-\\rho}}\\xi_T} \\geq \\left(\\frac{1}{\\rho-1}\\right)^{T-T_*}\\log \\frac{\\ep_*}{\\theta^{\\frac{\\rho-1}{2-\\rho}}\\xi_{T_*}}.\n$$\nSince $\\xi_T > \\ep$ and $\\xi_{T_*}\\leq \\ep_*$, we obtain\n$$\n\\sum_{t=T_*}^T N^{(t)} =\nT-T_*+1 \\leq  1+ \\left(\\log\\frac{1}{\\rho-1}\\right)^{-1}\\left(\\log\\log \\frac{\\ep_*}{\\theta^{\\frac{\\rho-1}{2-\\rho}}\\ep}-\\log\\log\\frac{1}{\\theta^{\\frac{\\rho-1}{2-\\rho}}}\\right).\n$$\nConsequently, $N$ is bounded as follows.\n$$\nN = \\sum_{t=0}^{T_*-1} N^{(t)} + \\sum_{t=T_*}^T N^{(t)} \\leq N(\\ep_*,\\sigma_*,C) + 1+\\left(\\log\\frac{1}{\\rho-1}\\right)^{-1}\\left(\\log\\log \\frac{\\ep_*}{\\theta^{\\frac{\\rho-1}{2-\\rho}}\\ep}-\\log\\log\\frac{1}{\\theta^{\\frac{\\rho-1}{2-\\rho}}}\\right).\n$$\n\n\\fbox{(iv) Case $\\rho=1$.}\nWe have\n$\n\\outersigma{t} \\geq\n\\sigma_*\n$ for each $t=0,\\ldots,T+1$, by Lemma~\\ref{lem:rAdaAPG-sigma} (iii).\nMoreover, Lemma~\\ref{lem:rAdaAPG-base}~(vi) shows that, if $\\outerXplus{t} \\not\\in X^*$, then we have\n$$\n1 \\leq \\frac{1}{\\kappa}\\left(\\frac{L_f}{L_{\\min}} + 1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})},\n~~\\text{i.e.,}~~\n\\norm{g_{\\outerM{t}}(\\outerXinit{t})} \\geq \\ep_*:=\\kappa\\left(\\frac{L_f}{L_{\\min}} + 1\\right)^{-1}.\n$$\nIn other words, the condition $\\norm{g_{\\outerM{t}}(\\outerXinit{t})} < \\ep_*$ must imply $\\outerXplus{t}\\in X^*$, which also yields $N^{(t)}=1$ and $\\outerXinit{t+1}=\\outerXplus{t} \\in X^*$ by Proposition~\\ref{prop:APG}~(v) (then the algorithm terminates at the ($t+1$)-th outer loop).\nTherefore, we have\n\\begin{equation}\\label{rho=1:gmap-ep*}\n\\norm{g_{\\outerM{t}}(\\outerXinit{t})} \\geq \\ep_*,\\quad 0\\leq t \\leq T-1.\n\\end{equation}\n\\sloppy\nNow we consider two cases. If $\\norm{g_{\\outerM{T}}(\\outerXinit{T})} \\geq \\ep_*$ holds, then\ncombining with \\eqref{pr:gradmap-lb-ep} yields ${\\norm{g_{\\outerM{T}}(\\outerXinit{T})} \\geq \\max(\\ep,\\ep_*)}$, from which Lemma~\\ref{lem:rAdaAPG-compl} with $T_*=T$ concludes\n$$\nN \\leq N(\\max(\\ep,\\ep_*),\\sigma_*,C)\n\\quad\\text{with}\\quad\nC=\n\\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\max(\\ep,\\ep_*)}\\right).\n$$\nOn the other case $\\norm{g_{\\outerM{T}}(\\outerXinit{T})} < \\ep_*$, we have $N^{(T)}=1$. Since \\eqref{pr:gradmap-lb-ep} and \\eqref{rho=1:gmap-ep*} implies ${\\norm{g_{\\outerM{T-1}}(\\outerXinit{T-1})} \\geq \\max(\\ep,\\ep_*)}$, Lemma~\\ref{lem:rAdaAPG-compl} with $T_*=T-1$ shows $\\sum_{t=0}^{T-1}N^{(t)}\\leq N(\\max(\\ep,\\ep_*),\\sigma_*,C)$. Hence,\n$$\nN = N^{(T)}+\\sum_{t=0}^{T-1}N^{(t)} \\leq 1 + N(\\max(\\ep,\\ep_*),\\sigma_*,C).\n$$\nThis proves the desired bound on $N$.\nTo show the latter assertion of (iv), suppose $\\ep<\\ep_*$. Then, the output $\\{\\outerXinit{T+1},\\outerM{T+1},\\outerXplus{T+1}\\}$ satisfies $\\norm{g_{\\outerM{T+1}}(\\outerXinit{T+1})} \\leq \\ep < \\ep_*$. Therefore, $\\outerXplus{T+1}$ must be optimal.\n\nThe proof of Theorem~\\ref{thm:rAdaAPG} is completed.\n\\end{proof}","units":[{"start":15903,"end":16933,"start_line":305,"end_line":327,"raw_tex":"\\begin{lemma}\\label{lem:gmap}\nLet $\\varphi=f+\\varPsi$ be the sum of a continuously differentiable convex function $f:\\E\\to\\RR$ and a proper lower-semicontinuous convex function $\\varPsi:\\E\\to\\RR\\cup\\{+\\infty\\}$.\nThen, the following assertions hold.\n\\begin{enumerate}[(i)]\n\\item For all $y \\in \\E$, the map $L \\mapsto \\norm{g_L(y)}$ is increasing.\n%\n\\item $x^* \\in X^*$ holds if and only if $g_L(x^*)=0$.\n%\n\\item $\\nabla{f}(T_L(y))-\\nabla{f}(y)+g_L(y) \\in \\partial\\varphi(T_L(y))$ for all $y \\in \\E$. Moreover, if $f$ is $L_f$-smooth, then\n$$\n\\norm{\\nabla{f}(T_L(y))-\\nabla{f}(y)+g_L(y)} \\leq \\left(\\frac{L_f}{L}+1\\right)\\norm{g_L(y)}.\n$$\n%\n\\item If $y\\in \\E$ and $L>0$ satisfy $\\varphi(T_L(y))\\leq m_L(y;T_L(y))$, then we have the inequalities\n$$\n\\frac{1}{2L}\\norm{g_L(y)}^2 \\leq \\varphi(y)-\\varphi(T_L(y)) \\leq \\varphi(y)-\\varphi^*,\n\\qquad\n\\frac{1}{2L}\\norm{g_L(y)} \\leq \\dist(y,X^*).\n$$\n%\n\\item If $f$ is $L_f$-smooth, then $\\varphi(T_L(y))\\leq m_L(y;T_L(y))$ holds for all $y\\in \\E$ and $L\\geq L_f$.\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"0999eb5c4fdbed36cbb29e6c8abde917ae9b5edd60dd59778a4c16558d8a9aa2"},{"start":16934,"end":19649,"start_line":328,"end_line":383,"raw_tex":"\\begin{proof}\n(i) is proved in \\cite[Lemma 2]{Nes13}.\n\n\n(ii) According to the optimality condition of the problem $\\min_{x \\in \\E}m_L(y;x)$, we have the following equivalence for $y,z \\in \\E$:\n\\begin{equation}\\label{optcond-TL}\nz = T_L(y) \\iff 0 \\in \\nabla{f}(y)+L(z-y)+\\partial\\varPsi(z).\n\\end{equation}\nOn the other hand, the optimality of the original problem $\\min_{x}\\varphi(x)$ is characterized as follows. Given $w\\in \\E$, we have\n$w \\in \\Argmin_{x \\in \\E}\\varphi(x)$ if and only if $0 \\in \\nabla{f}(w)+\\partial{\\varPsi}(w)$ holds\n%$$\n%w \\in \\Argmin_{x \\in \\E}\\varphi(x) \\iff 0 \\in \\nabla{f}(w)+\\partial{\\varPsi}(w).\n%$$\n(Remark that we have $\\partial\\varphi=\\nabla{f}+\\partial \\varPsi$ \\cite[Theorem~23.8]{Roc}).\nLet us show the equivalence\n\\begin{equation}\\label{eq:gmap-opt-sol-ch}\nx^* \\in \\Argmin_{x \\in \\E}\\varphi(x) \\iff x^*=T_L(x^*) ~~(\\iff g_L(x^*)=0).\n\\end{equation}\nIf $x^*=T_L(x^*)$, then \\eqref{optcond-TL} implies $0 \\in \\nabla{f}(x^*)+\\partial \\varPsi(x^*)$ and so $x^* \\in \\Argmin_{x\\in \\E}\\varphi(x)$ holds. Conversely, if $x^*\\in \\Argmin_{x \\in \\E}\\varphi(x)$, then we have the condition $0\\in \\nabla{f}(x^*)+\\partial \\varPsi(x^*)$, which coincides with the right hand one of \\eqref{optcond-TL} in the case $y=z=x^*$. Thus, we conclude\n its equivalent counterpart $x^*=T_L(x^*)$. This proves the equivalence \\eqref{eq:gmap-opt-sol-ch} as well as the assertion (ii).\n\n\n(iii) By the condition \\eqref{optcond-TL}, we have\n$\n0 \\in \\nabla{f}(y)+L(T_L(y)-y) + \\partial\\varPsi(T_L(y)) = \\nabla{f}(y)-g_L(y)  + \\partial\\varPsi(T_L(y)).\n$\nHence, we obtain\n$\n\\nabla{f}(T_L(y))-\\nabla{f}(y)+g_L(y) \\in \\nabla{f}(T_L(y))+\\partial\\varPsi(T_L(y)) = \\partial\\varphi(T_L(y))\n$, giving the former assertion of (iii).\nNow if $f$ is $L_f$-smooth, we have\n$\n\\norm{\\nabla{f}(T_L(y))-\\nabla{f}(y)} \\leq L_f\\norm{T_L(y)-y} = \\frac{L_f}{L}\\norm{g_L(y)},\n$\nwhich yields the latter assertion of (iii).\n\n\n(iv)\nIt is shown in \\cite[Lemma~2.3]{BT09} that,\nif $\\varphi(T_L(y))\\leq m_L(y;T_L(y))$ holds, then we have\n$$\n\\varphi(x)  - \\varphi(T_L(y)) \\geq \\frac{1}{2L}\\norm{g_L(y)}^2 + \\innprod{g_L(y)}{x-y},\\quad \\forall x \\in \\E.\n$$\nLetting $x:=y$ shows the first assertion. On the other hand, since $\\varphi(T_L(y))\\geq \\varphi^*$, letting $x:=x^* \\in X^*$ gives\n$\n\\frac{1}{2L}\\norm{g_L(y)}^2 \\leq \\innprod{g_L(y)}{y-x^*} \\leq \\norm{g_L(y)}\\norm{y-x^*}.\n$\nThus, we obtain $\\frac{1}{2L}\\norm{g_L(y)} \\leq \\dist(y,X^*)$.\n\n(v) Note that $f$ is $L$-smooth for any $L\\geq L_f$.\nSince $L$-smooth functions satisfy (e.g., see \\cite{Nes04})\n$$\nf(x) \\leq f(y)+\\innprod{\\nabla{f}(y)}{x-y} + \\frac{L}{2}\\norm{x-y}^2,\\quad \\forall x,y \\in \\E,\n$$\nwe obtain $\\varphi(x) \\leq m_L(y;x)$ for all $x,y \\in \\E$ and $L\\geq L_f$.\n\\end{proof}","sha256_utf8":"e6ef1c4590ada59c4e73a1a0675c541b761583bfdc5507656edcb9dea0e52d5f"},{"start":21390,"end":22022,"start_line":415,"end_line":424,"raw_tex":"\\begin{lemma}\\label{lem:HEB}\nLet $\\varphi:\\E\\to\\RR\\cup\\{+\\infty\\}$ be a proper, lower-semicontinuous, and convex function.\nFor $x_0 \\in \\E$, suppose that $\\varphi$ satisfies the H\\\"olderian error bound condition \\eqref{HEB} with coefficient $\\kappa > 0$ and exponent $\\rho \\geq 1$.\nThen, for every $x \\in \\lev_{\\varphi}(\\varphi(x_0))\\setminus X^*$, we have\nthe inequalities\n\\begin{equation}\\label{Loj-ineq}\n\\kappa \\dist(x,X^*)^{\\rho-1} \\leq \\inf\\{\\norm{g}:g \\in \\partial\\varphi(x)\\},\\quad\n\\kappa^{\\frac{1}{\\rho}} (\\varphi(x)-\\varphi^*)^{\\frac{\\rho-1}{\\rho}} \\leq \\inf\\{\\norm{g}:g \\in \\partial\\varphi(x)\\}.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"5f13fa11531e4c41745b14b909156ec7125b551c61c4c0adb2cfd9d65a2df8bc"},{"start":22023,"end":22707,"start_line":425,"end_line":441,"raw_tex":"\\begin{proof}\nLet $x^*$ be the projection of $x$ onto $X^*$, so that $\\norm{x-x^*}=\\dist(x,X^*)$.\nFor every $g\\in\\partial \\varphi(x)$, we have\n\\begin{align*}\n\\kappa \\dist(x,X^*)^\\rho\n&\\leq \\varphi(x)-\\varphi^*\n\\leq -\\innprod{g}{x^*-x}\n\\leq \\norm{g}\\dist(x,X^*)\\leq \\norm{g}\\frac{1}{\\kappa^{1/\\rho}}(\\varphi(x)-\\varphi^*)^{1/\\rho}.\n\\end{align*}\nIt involves two inequalities\n$\n\\kappa \\dist(x,X^*)^\\rho \\leq \\norm{g}\\dist(x,X^*)$\nand\n$\\varphi(x)-\\varphi^* \\leq \\norm{g}\\frac{1}{\\kappa^{1/\\rho}}(\\varphi(x)-\\varphi^*)^{1/\\rho}.\n$\nSince $x \\not \\in X^*$, the assertion follows dividing them by $\\dist(x,X^*)$ and $\\kappa^{-1/\\rho}(\\varphi(x)-\\varphi^*)^{1/\\rho}$, respectively.\n\\end{proof}","sha256_utf8":"0db1804d3d5744fe55cd709d624e5bca88429adb3d26c5b2c1b658679200a3ac"},{"start":30103,"end":30413,"start_line":565,"end_line":571,"raw_tex":"\\begin{lemma}\\label{lem:reg}\n\\begin{enumerate}[(i)]\n\\item $\\varphi_\\sigma(x) \\leq \\varphi_\\sigma(x_0)$ implies $\\varphi(x) \\leq \\varphi(x_0)$.\n\\item $\\norm{x_0-x_\\sigma^*} \\leq \\dist(x_0,X^*)$.\n\\item We have $\\norm{g_L(y)-g_L^\\sigma(y)} \\leq  \\sigma\\norm{y-x_0}$ for any $y \\in \\E$.\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"458a044b8dd6259f80bc074a1ead8cd5a2faf6c2f25bc2cefe2953d13f83050a"},{"start":30415,"end":32028,"start_line":573,"end_line":604,"raw_tex":"\\begin{proof}\n(i) is immediate by $\\varphi(x)\\leq \\varphi_\\sigma(x)$ and $\\varphi_\\sigma(x_0)=\\varphi(x_0)$.\\\\\n(ii) For any $x^* \\in X^*$, the strong convexity of $\\varphi_\\sigma$ implies\n$\n\\varphi_\\sigma(x^*)\\geq \\varphi_\\sigma^* + \\frac{\\sigma}{2}\\norm{x^*-x_\\sigma^*}^2,\n$\nwhich can be rewritten as\n$\\varphi(x^*)-\\varphi(x_\\sigma^*) \\geq \\frac{\\sigma}{2}(-\\norm{x^*-x_0}^2 + \\norm{x_\\sigma^*-x_0}^2 + \\norm{x^*-x_\\sigma^*}^2).$\nAs $\\varphi(x^*)-\\varphi(x_\\sigma^*)\\leq 0$, we conclude $\\norm{x^*-x_0}^2 \\geq \\norm{x_\\sigma^*-x_0}^2 + \\norm{x^*-x_\\sigma^*}^2 \\geq \\norm{x_\\sigma^*-x_0}^2$, which proves the assertion.\\\\\n(iii)\nIn general, if $h$ is a lower-semicontinuous and $\\mu$-strongly convex function, we have for any $a,b \\in \\E$ that\\footnote{\nBy the strong convexity of $\\innprod{a}{x}+h(x)$ and $\\innprod{b}{x}+h(x)$, we have\n$$\n\\frac{\\mu}{2}\\norm{x_a^*-x_b^*}^2 \\leq [\\innprod{a}{x_b^*}+h(x_b^*)] - [\\innprod{a}{x_a^*}+h(x_a^*)]\n~~\\text{and}~~\n\\frac{\\mu}{2}\\norm{x_a^*-x_b^*}^2 \\leq  [\\innprod{b}{x_a^*}+h(x_a^*)]-[\\innprod{b}{x_b^*}+h(x_b^*)],\n$$\nrespectively.\nAdding them implies\n$\n\\mu\\norm{x_a^*-x_b^*}^2 \\leq \\innprod{a-b}{x_b^*-x_a^*} \\leq \\norm{a-b}\\norm{x_b^*-x_a^*}.\n$\n}\n$$\\norm{x_a^*-x_b^*}\\leq \\frac{\\norm{a-b}}{\\mu},~~\\text{where}~~\nx_a^* = \\argmin_{x \\in \\E}\\{\\innprod{a}{x}+h(x)\\},~~ x_b^* = \\argmin_{x \\in \\E}\\{\\innprod{b}{x}+h(x)\\}.$$\nThis fact implies\n$\n\\norm{T_L(y)-T_L^\\sigma(y)} \\leq \\frac{\\norm{\\nabla{f}(y)-\\nabla{f}_\\sigma(y)}}{L} = \\frac{\\sigma\\norm{y-x_0}}{L}.\n$\nTherefore, we conclude (iii) because of $\\norm{g_L(y)-g_L^\\sigma(y)}\n=  L\\norm{T_L(y)-T_L^\\sigma(y)}$.\n\\end{proof}","sha256_utf8":"f389bcdd37bd22eb5b779b99c3f59108b27f5faeedec1465355cb03e1d101e5a"},{"start":37695,"end":37822,"start_line":679,"end_line":681,"raw_tex":"\\begin{lemma}\\label{lem:APG-iter-L}\nThe condition $\\eqref{APG-iter-test}$ of $\\APGIter$ holds whenever $L\\geq L_f$.\n\\end{lemma}","sha256_utf8":"6ae5df3ca9a2d1ae7416b911ebb91a35d86d047061eb738903c1107480ea355f"},{"start":37823,"end":38523,"start_line":682,"end_line":694,"raw_tex":"\\begin{proof}\nThe first condition \\eqref{APG-iter-test-a} is satisfied since $f$ is $L$-smooth (e.g., see \\cite[Theorem 2.1.5]{Nes04}).\nSince $f_\\sigma$ is $(L_f+\\sigma)$-smooth, the second one \\eqref{APG-iter-test-b} holds if $L\\geq L_f$ by Lemma~\\ref{lem:gmap} (v).\nThe third one \\eqref{APG-iter-test-c} can be verified again by Lemma~\\ref{lem:gmap} (v): If $L\\geq L_f$, we have\n$\n\\varphi(T_L(z)) \\leq m_L(z;T_L(z)) \\leq \\varphi(z),\n$\nwhere the second inequality follows from\n\\begin{equation}\\label{mL-leq-phi}\nm_L(z;T_L(z))=\\min_{x \\in \\E}\\left\\{f(z)+\\innprod{\\nabla{f}(z)}{x-z}+\\varPsi(x)+\\frac{L}{2}\\norm{x-z}^2\\right\\} \\stackrel{x=z}{\\leq}\nf(z)+\\varPsi(z)=\\varphi(z).\n\\end{equation}\n\\end{proof}","sha256_utf8":"0ff5272d993234996546c933a721f93e183b4105da321ce6285d361d7fb4a6e1"},{"start":38657,"end":39201,"start_line":697,"end_line":705,"raw_tex":"\\begin{lemma}\\label{lem:APG-iter-LS-compl}\n\\sloppy % prevents overflow\nSuppose that $L_k \\leq \\gamma_{\\rm inc}L_f$ holds.\nThen, $\\APGIter$ ensures the conditions $M_k \\leq \\gamma_{\\rm inc}L_f$ and ${L_{k+1} \\leq \\max\\{L_{\\min},\\frac{\\gamma_{\\rm inc}}{\\gamma_{\\rm dec}}L_f\\}\\leq \\gamma_{\\rm inc}L_f}$.\nThe number of executions of the loop {\\rm (Lines~\\ref{APG-iter-loop-s}--\\ref{APG-iter-loop-e})} is bounded by\n$$\n1 + \\frac{\\log \\gamma_{\\rm dec}}{\\log \\gamma_{\\rm inc}} + \\frac{1}{\\log \\gamma_{\\rm inc}} \\log \\frac{L_{k+1}}{L_k}.\n$$\n\\end{lemma}","sha256_utf8":"ea0941d3144fb0b34c7b624b26ed2c8cac59d29cb93c3faaf67af23c703b491e"},{"start":39202,"end":40253,"start_line":706,"end_line":719,"raw_tex":"\\begin{proof}\n\\sloppy % prevents overflow\nLet $n_k \\geq 1$ be the number of inner loops so that ${M_k = L_k\\gamma_{\\rm inc}^{n_k-1}}$.\nIf $n_k=1$, then we have $M_k = L_k \\leq \\gamma_{\\rm inc}L_f$. If $n_k>1$, then $M_k\\leq \\gamma_{\\rm inc}L_f$ must hold because otherwise ${L_k\\gamma_{\\rm inc}^{n_k-2}=M_k/\\gamma_{\\rm inc} > L_f}$ and the $n_k$-th loop cannot occur (by Lemma~\\ref{lem:APG-iter-L}). Hence, we conclude $M_k \\leq \\gamma_{\\rm inc}L_f$ and also ${L_{k+1} = \\max\\{L_{\\min},M_k/\\gamma_{\\rm dec}\\} \\leq \\max\\{L_{\\min},\\frac{\\gamma_{\\rm inc}}{\\gamma_{\\rm dec}}L_f\\}}$.\n\nNow, the relation $M_k = L_k\\gamma_{\\rm inc}^{n_k-1}$ implies\n$\nL_{k+1} = \\max\\{L_{\\min},M_k/\\gamma_{\\rm dec}\\} \\geq \\frac{1}{\\gamma_{\\rm dec}}L_k\\gamma_{\\rm inc}^{n_k-1}.\n$\nThen, we have $(n_k-1)\\log\\gamma_{\\rm inc} \\leq \\log \\frac{\\gamma_{\\rm dec}L_{k+1}}{L_k} = \\log \\gamma_{\\rm dec} + \\log \\frac{L_{k+1}}{L_k}$ and therefore\n$\nn_k \\leq  1 + \\frac{\\log \\gamma_{\\rm dec}}{\\log \\gamma_{\\rm inc}} + \\frac{1}{\\log \\gamma_{\\rm inc}} \\log \\frac{L_{k+1}}{L_k} .\n$\n\\end{proof}","sha256_utf8":"d034a4e2e910358072f2c9687bdeac8348b4df7bd235eec36cd5fb86dc96413b"},{"start":40323,"end":41819,"start_line":724,"end_line":753,"raw_tex":"\\begin{proposition}\\label{prop:APG}\nLet $\\{x_k,\\psi_k,M_k,L_k,A_k\\}$ be generated by the accelerated proximal gradient method applied to the regularized objective function \\eqref{realloc}, that is,\n$$\\{x_{k+1},\\psi_{k+1},M_k,L_{k+1},A_{k+1}\\} \\leftarrow \\APGIter_\\sigma(x_k,\\psi_k,L_k,A_k),\\quad k=0,1,2,\\ldots,$$\nwith the initialization $x_0 \\in \\E$, $\\psi_0(x)=\\frac{1}{2}\\norm{x-x_0}^2$, $A_0=0$, $L_0>0$.\n\\begin{enumerate}[(i)]\n\\item\nIf $L_0 \\in [L_{\\min}, \\gamma_{\\rm inc}L_f]$, then\nwe have $L_{\\min} \\leq L_k \\leq M_k \\leq \\gamma_{\\rm inc}L_f$ for all $k \\geq 0$.\nThe total number of executions of the loop {\\rm (Lines~\\ref{APG-iter-loop-s}--\\ref{APG-iter-loop-e})} until the $k$-th iteration is bounded by \n$$\n\\left[\n1+\\frac{\\log \\gamma_{\\rm dec}}{\\log \\gamma_{\\rm inc}}\\right](k+1) + \\frac{1}{\\log \\gamma_{\\rm inc}}\\log\\frac{L_{k+1}}{L_0}.\n$$\n\\item For each $k\\geq 1$, we have\n$\nA_k \\geq \\frac{2}{\\gamma_{\\rm inc}L_f}\\left[1+\\sqrt{\\frac{\\sigma}{2\\gamma_{\\rm inc}L_f}}\\right]^{2(k-1)}.\n$\n\\item $\\varphi_\\sigma(x_k)-\\varphi_\\sigma^* \\leq \\frac{\\norm{x_0-x_\\sigma^*}^2}{2A_k}$ holds for all $k\\geq 1$.\n\\item $\\varphi(x_k)\\leq \\varphi(x_0)$ holds for all $k\\geq 1$.\n\\item\nFor every $k\\geq 1$, we have\n$$\n\\norm{x_k-x_0} \\leq \\left(1+\\frac{1}{\\sqrt{\\sigma A_k}}\\right)\\dist(x_0,X^*),\n$$\n$$\n\\norm{g_{M_{k-1}}(x_k)} \\leq \\norm{g_{M_{k-1}+\\sigma}^\\sigma(x_k)} + \\sigma\\norm{x_k-x_0}\n\\leq \\left(2\\sqrt{\\frac{M_{k-1}+\\sigma}{A_k}} + \\sigma\\right)\\dist(x_0,X^*).\n$$\n\\end{enumerate}\n\\end{proposition}","sha256_utf8":"bc25cb6ddf7182778f82aaab94d84b86ea8213c9b303f92c39f20f59c75a4429"},{"start":41820,"end":44393,"start_line":754,"end_line":810,"raw_tex":"\\begin{proof}\n(i) \n$L_{\\min}\\leq L_k\\leq M_k$ is clear by the construction. $M_k\\leq \\gamma_{\\rm inc}L_f$\nis obtained by applying Lemma~\\ref{lem:APG-iter-LS-compl} inductively.\n\nSince $\\varPsi_\\sigma$ is $\\sigma$-strongly convex, \n(ii) follows by \\cite[Lemma~8]{Nes13} applied to the objective function $\\varphi_\\sigma=f+\\varPsi_\\sigma$.\n\nAccording to \\cite[Lemma 7]{Nes13}, the following relations hold for all $k\\geq 0$:\n$$\n\\left\\{\n\\begin{array}{l}\nA_k\\varphi_\\sigma(x_k) \\leq \\min_{x \\in \\E}\\psi_k(x),\\\\\n\\psi_k(x)\\leq A_k\\varphi_\\sigma(x)+\\frac{1}{2}\\norm{x-x_0}^2,\\quad \\forall x \\in \\E.\n\\end{array}\n\\right.\n$$\nCombining them, we obtain\n$$\nA_k\\varphi_\\sigma(x_k) \\leq \\min_{x\\in \\E}\\left\\{ A_k\\varphi_\\sigma(x)+\\frac{1}{2}\\norm{x-x_0}^2 \\right\\}, \\quad \\forall k\\geq 0.\n$$\nTaking $x=x_\\sigma^*$ on the right hand side, we obtain (iii).\nOn the other hand, taking $x=x_0$ yields ${\\varphi_\\sigma(x_k)\\leq \\varphi_\\sigma(x_0)}$. \nThen, Lemma~\\ref{lem:reg} (i) gives the assertion $\\varphi(x_k)\\leq \\varphi(x_0)$.\n\n(v)\nBy the $\\sigma$-strong convexity of $\\varphi_\\sigma$ and using (iii), we have\n$\n\\frac{\\sigma}{2}\\norm{x_k-x_\\sigma^*}^2 \\leq \\varphi_\\sigma(x_k)-\\varphi_\\sigma^* \\leq \\frac{\\norm{x_0 -x_\\sigma^*}^2}{2A_k},\n$\nand thus\n$\n\\norm{x_k-x_\\sigma^*} \\leq \\frac{1}{\\sqrt{\\sigma A_k}}\\norm{x_0-x_\\sigma^*}.\n$\nThis shows the former inequality of (v):\n$$\n\\norm{x_k-x_0}\\leq \\norm{x_k-x_\\sigma^*} + \\norm{x_0-x_\\sigma^*} \\leq \\left(1+\\frac{1}{\\sqrt{\\sigma A_k}}\\right) \\norm{x_0-x_\\sigma^*}\n\\leq\n\\left(1+\\frac{1}{\\sqrt{\\sigma A_k}}\\right) \\dist(x_0,X^*),\n$$\nwhere the last inequality follows by Lemma~\\ref{lem:reg} (ii).\n\nSince $\\varphi_\\sigma(T_{M_{k-1}+\\sigma}^\\sigma(x_k)) \\leq m_{M_{k-1}+\\sigma}^\\sigma(x_k;T_{M_{k-1}+\\sigma}^\\sigma(x_k))$ holds by \\eqref{APG-iter-test-b}, Lemma~\\ref{lem:gmap} (iv) yields\n$$\n\\frac{1}{2(M_{k-1}+\\sigma)}\\norm{g_{M_{k-1}+\\sigma}^\\sigma(x_k)}^2\n\\leq \\varphi_\\sigma(x_k)-\\varphi_\\sigma^* \\leq \\frac{\\norm{x_0-x_\\sigma^*}^2}{2A_k}\n\\leq \\frac{\\dist(x_0,X^*)^{2}}{2A_k}.\n$$\nThus, $\\norm{g_{M_{k-1}+\\sigma}^\\sigma(x_k)} \\leq \\sqrt{\\frac{M_{k-1}+\\sigma}{A_k}}\\dist(x_0,X^*)$ holds.\nConsequently, using Lemma~\\ref{lem:gmap}~(i) and Lemma~\\ref{lem:reg}~(iii), we obtain\n\\begin{align*}\n\\norm{g_{M_{k-1}}(x_k)} &\\leq \\norm{g_{M_{k-1}+\\sigma}(x_k)}\n\\leq \\norm{g_{M_{k-1}+\\sigma}^\\sigma(x_k)} + \\sigma\\norm{x_k - x_0}\n\\\\&\\leq  \\sqrt{\\frac{M_{k-1}+\\sigma}{A_k}}\\dist(x_0,X^*) + \\sigma\\left(1+\\frac{1}{\\sqrt{\\sigma A_k}}\\right)\\dist(x_0,X^*)\n\\\\&\\leq \\left(2\\sqrt{\\frac{M_{k-1}+\\sigma}{A_k}} + \\sigma\\right)\\dist(x_0,X^*).\n\\end{align*}\n\\end{proof}","sha256_utf8":"d68d6e7e8fc850fc7c58926610acaa348501df7a31366145852296618043953b"},{"start":45668,"end":46347,"start_line":844,"end_line":855,"raw_tex":"\\begin{lemma}\\label{lem:PG-iter}\nLet $\\{T_{M_k}(x_k), M_k, L_{k+1}\\}$ be given by $\\PGIter(x_k,L_k)$. Then, the following assertions hold.\n\\begin{enumerate}[(i)]\n\\item $\\varphi(T_{M_k}(x_k))\\leq \\varphi(x_k)$ holds.\n\\item\nIf $L_k \\leq \\gamma_{\\rm inc}L_f$, then we have $M_k \\leq \\gamma_{\\rm inc}L_f$ and $L_{k+1} \\leq \\max\\{L_{\\min},$ $\\frac{\\gamma_{\\rm inc}}{\\gamma_{\\rm dec}}L_f\\} \\leq \\gamma_{\\rm inc}L_f$.\nMoreover, the number of executions of the loop {\\rm (Lines~\\ref{PG-iter-loop-s}--\\ref{PG-iter-loop-e})} is bounded by\n$$\n 1 + \\frac{\\log \\gamma_{\\rm dec}}{\\log \\gamma_{\\rm inc}} + \\frac{1}{\\log \\gamma_{\\rm inc}} \\log \\frac{L_{k+1}}{L_k}.\n$$\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"d5ab750bb5482b9e4a468eb0f35a1abd5cc76c9ed76aed61a6d8140c33564ad9"},{"start":46348,"end":46654,"start_line":856,"end_line":861,"raw_tex":"\\begin{proof}\nSince the condition \\eqref{PG-iter-test} ensures\n$\\varphi(T_{M_k}(x_k))\\leq m_{M_k}(x_k;T_{M_k}(x_k))\\leq \\varphi(x_k)$\n(recall \\eqref{mL-leq-phi} for the second inequality), we conclude (i).\nThe assertion (ii) can be verified in the same way as Lemma~\\ref{lem:APG-iter-LS-compl}.\n\\end{proof}","sha256_utf8":"0aed7ff569a2ea7e7277f2b0f8288b65e36989559003cc6f51c9633e8605fc9a"},{"start":49636,"end":50237,"start_line":918,"end_line":930,"raw_tex":"\\begin{lemma}\\label{lem:AdaAPG-loop}\nIn $\\AdaAPG$,\nsuppose that a $j$-th outer loop is completed satisfying the criterion $A_{k+1}\\geq \\frac{2(M_{k}+\\sigma_j)}{\\beta^2\\sigma_j^2}$ for some $k\\geq 0$. Then, we have the inequalities\n$$\n\\norm{x_{k+1}-x_0} \\leq \\left(1+\\frac{\\beta}{\\sqrt{2}}\\right)\\dist(x_0,X^*),\n\\qquad\n\\norm{g_{M_{k}}(x_{k+1})} \\leq (1+\\sqrt{2}\\beta)\\sigma_j\\dist(x_0,X^*).\n$$\nMoreover, the number of inner iterations is bounded as follows.\n$$\nk+1 \\leq 2+ \\left(\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_j}}+1\\right)\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_j}{\\beta\\sigma_j}.\n$$\n\\end{lemma}","sha256_utf8":"1d0d3483880e724983351a1a170800060e1d40fa8d26d9c5d07533fcc1b016ae"},{"start":50238,"end":51773,"start_line":931,"end_line":952,"raw_tex":"\\begin{proof}\nThe bounds on $\\norm{x_{k+1}-x_0}$ and $\\norm{g_{M_{k}}(x_{k+1})}$ can be obtained by Proposition~\\ref{prop:APG} (v), applying the assumption $A_{k+1}\\geq \\frac{2(M_{k}+\\sigma_j)}{\\beta^2\\sigma_j^2}$.\nTo show the bound on $k+1$, suppose $k\\geq 1$ (the result is clear when $k=0$).\nBy the assumption on $k$ and Proposition~\\ref{prop:APG}~(i), we have\n$$A_{k} < \\frac{2(M_{k-1}+\\sigma_j)}{\\beta^2\\sigma_j^2} \\leq \\frac{2(\\gamma_{\\rm inc}L_f+\\sigma_j)}{\\beta^2\\sigma_j^2}.$$\nOn the other hand, Proposition~\\ref{prop:APG} (ii) implies\n$$\nA_{k} \\geq  \\frac{2}{\\gamma_{\\rm inc}L_f}\\left[1+\\sqrt{\\frac{\\sigma_j}{2\\gamma_{\\rm inc}L_f}}\\right]^{2(k-1)} \\geq\n \\frac{2}{\\gamma_{\\rm inc}L_f}\\exp\\left(2(k-1) \\frac{1}{\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_j}}+1}\\right),\n$$\nwhere the second inequality is due to the fact\\footnote{\nIn fact, since the derivative $\\log(1+x)$ of the function $h(x):=(1+x)\\log(1+x)-x$ is increasing and vanishes at $x=0$, we have $\\min_{x>-1}h(x)=h(0)=0$.\n}\n$1+x \\geq \\exp\\frac{x}{1+x}~(\\forall x > -1)$.\nTherefore,\n\\begin{align*}\n\\frac{2}{\\gamma_{\\rm inc}L_f}\\exp\\left(2(k-1) \\frac{1}{\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_j}}+1}\\right) \\leq \\frac{2(\\gamma_{\\rm inc}L_f+\\sigma_j)}{\\beta^2\\sigma_j^2}\\\\\n\\Longrightarrow\nk+1 \\leq 2+ \\frac{1}{2}\\left(\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_j}}+1\\right)\\log\\frac{\\gamma_{\\rm inc}L_f(\\gamma_{\\rm inc}L_f+\\sigma_j)}{\\beta^2\\sigma_j^2}.\n\\end{align*}\nThe assertion follows by relaxing $\\gamma_{\\rm inc}L_f \\leq \\gamma_{\\rm inc}L_f + \\sigma_j$.\n\\end{proof}","sha256_utf8":"230c63b328385c8782c1a78e479a4de21d56d04f170d946ef9df73d187f729fd"},{"start":52203,"end":54284,"start_line":960,"end_line":1001,"raw_tex":"\\begin{theorem}\\label{thm:AdaAPG}\nIn $\\AdaAPG$, let $N$ be the total number of executions of $\\APGIter$.\nDenote\n\\begin{equation}\\label{sigma-th}\n\\sigma(x_0,\\ep):=\\frac{\\ep}{(1+\\sqrt{2}\\beta)\\dist(x_0,X^*)},\n\\end{equation}\nwhere we let $\\sigma(x_0,\\ep):=+\\infty$ when $x_0 \\in X^*$.\nThen, the following assertions hold.\n\\begin{enumerate}[(i)]\n\\item\nThe algorithm terminates at the $j$-th outer loop whenever\n$\\sigma_j \\leq \\sigma(x_0,\\ep)$.\nMoreover, we have ${\\sigma_j \\geq \\sigma(x_0,\\ep)/\\gamma_{\\rm reg}}$ for all $j>0$.\n%\\begin{equation}\\label{sigma-j-lb}\n%\\sigma_j \\geq \\sigma(x_0,\\ep)/\\gamma_{\\rm reg},\\quad \\forall j>0.\n%\\end{equation}\n\n\\item\nSuppose that the algorithm terminates at $j=\\ell$ for some $\\ell \\geq 0$.\nThen, $N$ is at most\n\\begin{align*}\n&\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\sigma_\\ell}} - \\sqrt{\\frac{1}{\\sigma_0}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell}+ \\left(1+\\log_{\\gamma_{\\rm reg}}\\frac{\\sigma_0}{\\sigma_\\ell}\\right)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell}\\right).\n\\end{align*}\n%\n\\item\nIf $\\sigma_0\\leq \\sigma(x_0,\\ep)$, then\n$\nN \\leq 2+ \\left(\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_0}}+1\\right)\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_0}{\\beta\\sigma_0}.\n$\n\n%\n\\item\nIf $\\sigma_0 \\geq \\sigma(x_0,\\ep)$, then $N$ is at most\n\\begin{align}\n&\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\gamma_{\\rm reg}\\sqrt{\\frac{1}{\\sigma(x_0,\\ep)}} - \\sqrt{\\frac{1}{\\sigma_0}} \\right]\\log\\left(\\frac{\\gamma_{\\rm reg}\\gamma_{\\rm inc}L_f}{\\beta\\sigma(x_0,\\ep)}+\\frac{1}{\\beta}\\right) \\nonumber\\\\\n&+ \\left(2+\\log_{\\gamma_{\\rm reg}}\\frac{ \\sigma_0}{\\sigma(x_0,\\ep)}\\right)\\left[2+\\log\\left(\\frac{\\gamma_{\\rm reg}\\gamma_{\\rm inc}L_f}{\\beta\\sigma(x_0,\\ep)}+\\frac{1}{\\beta}\\right)\\right] \\nonumber\\\\\n=~&\nO\\left(\\sqrt{\\frac{L_f\\dist(x_0,X^*)}{\\ep}}\\log \\frac{L_f\\dist(x_0,X^*)}{\\ep} + \\log \\frac{\\sigma_0 \\dist(x_0,X^*)}{\\ep} \\log \\frac{L_f\\dist(x_0,X^*)}{\\ep}\\right).\n\\label{AdaAPG-compl-O}\n\\end{align}\n\\end{enumerate}\n\\end{theorem}","sha256_utf8":"bc393412150ef464fe2a5853799bf7f6df4eb486bc084dbbf864e0b54857eb5d"},{"start":54286,"end":56888,"start_line":1003,"end_line":1055,"raw_tex":"\\begin{proof}\n(i)\n\\sloppy % prevents overflow\nBy Lemma~\\ref{lem:AdaAPG-loop}, the algorithm must terminate at the $j$-th loop whenever we have\n$\n{(1+\\sqrt{2}\\beta)\\sigma_j\\dist(x_0,X^*) \\leq \\ep}\n$,\nor equivalently,\n$\n\\sigma_j \\leq \\frac{\\ep}{(1+\\sqrt{2}\\beta)\\dist(x_0,X^*)}=\\sigma(x_0,\\ep).\n$\nThus, we obtain the former assertion of (i).\nTo see the latter assertion,\nassume that $\\sigma_j < \\sigma(x_0,\\ep)/\\gamma_{\\rm reg}$ holds for some $j > 0$. Then, the previous $(j-1)$-th loop satisfies $\\sigma_{j-1}< \\sigma(x_0,\\ep)$ so that the $j$-th loop is not executed by the former assertion.\nThis verifies the latter assertion of (i).\n\n(ii) Since $\\sigma_\\ell=\\sigma_0/\\gamma_{\\rm reg}^\\ell$, we have $\\ell = \\log_{\\gamma_{\\rm reg}}\\sigma_0/\\sigma_\\ell$.\nUsing Lemma~\\ref{lem:AdaAPG-loop}, $N$ is bounded as follows.\n\\begin{align*}\nN\n&\\leq\n\\sum_{j=0}^\\ell \\left[\n2+ \\left(\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_j}}+1\\right)\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_j}{\\beta\\sigma_j}\n\\right]\\\\\n&\\leq 2(\\ell+1) + \\sqrt{2\\gamma_{\\rm inc}L_f}\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell}\\sum_{j=0}^\\ell \\sqrt{\\frac{1}{\\sigma_j}}\n+ (\\ell+1)\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell},\n\\end{align*}\nwhere the second inequality is due to $\\sigma_j \\geq \\sigma_\\ell~(\\forall j \\leq \\ell)$.\nNote that\n\\begin{align*}\n\\sum_{j=0}^\\ell \\sqrt{\\frac{1}{\\sigma_j}}\n&=\n\\sum_{j=0}^\\ell \\sqrt{\\frac{1}{\\sigma_0/\\gamma_{\\rm reg}^j}}\n=\n\\sqrt{\\frac{1}{\\sigma_0}} \\sum_{j=0}^\\ell \\sqrt{\\gamma_{\\rm reg}}^{j}\n=\n\\sqrt{\\frac{1}{\\sigma_0}}\\frac{\\sqrt{\\gamma_{\\rm reg}}^{\\ell+1} -1}{\\sqrt{\\gamma_{\\rm reg}} -1}\\\\\n&=\n\\sqrt{\\frac{1}{\\sigma_0}}\\frac{\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\sigma_0/\\sigma_\\ell} -1}{\\sqrt{\\gamma_{\\rm reg}} -1}\n=\n\\frac{1}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\sigma_\\ell}} - \\sqrt{\\frac{1}{\\sigma_0}}\\right].\n\\end{align*}\nTherefore, we see that\n$$\nN\\leq (\\ell+1)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell}\\right)\n+\n\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\sigma_\\ell}} - \\sqrt{\\frac{1}{\\sigma_0}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell}.\n$$\nThe assertion follows by substituting $\\ell = \\log_{\\gamma_{\\rm reg}}{\\sigma_0}/{\\sigma_\\ell}$.\n\nIn view of (i), the assertions (iii) and (iv) follow by applying (ii) with $\\sigma_\\ell = \\sigma_0$ and $\\sigma_\\ell \\geq \\sigma(x_0,\\ep)/\\gamma_{\\rm reg}$, respectively.\nThe big-$O$ expression \\eqref{AdaAPG-compl-O} is obtained by substituting the definition of $\\sigma(x_0,\\ep)$.\n\\end{proof}","sha256_utf8":"d42107834263c41936376ba77fcb9d9f5c8746d899cb7fda74845945069af274"},{"start":73430,"end":74460,"start_line":1369,"end_line":1384,"raw_tex":"\\begin{lemma}\\label{lem:rAdaAPG-base}\nAssume that the H\\\"olderian error bound condition \\eqref{HEB-review} holds.\nIn the execution of $\\rAdaAPG$, the following assertions hold.\n\\begin{enumerate}[(i)]\n\\item $\\outerL{t},\\outerM{t} \\in [L_{\\min},\\gamma_{\\rm inc}L_f]$ for all $t\\geq 0$.\n\\item $\\varphi(\\outerXplus{t+1}) \\leq \\varphi(\\outerXplus{t}) \\leq \\cdots \\leq \\varphi(\\outerXplus{0}) \\leq \\varphi(\\xinit)$ for all $t\\geq 0$.\n\\item $t \\leq \\log_{1/\\theta}\\dfrac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\norm{g_{\\outerM{t}}(\\outerXinit{t})}}$ for each $t\\geq 0$.\n\\item Whenever $\\outerXplus{t} \\not\\in X^*$, we have\n\\begin{equation}\\label{rAdaAPG-dist-bound}\n\\dist(\\outerXplus{t},X^*)^{\\rho-1} \\leq \\frac{1}{\\kappa}\\left(\\frac{L_f}{L_{\\min}}+1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})},\n\\end{equation}\n\\begin{equation}\\label{rAdaAPG-obj-bound}\n(\\varphi(\\outerXplus{t})-\\varphi^*)^{\\rho-1} \\leq \\frac{1}{\\kappa}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^\\rho\\norm{g_{\\outerM{t}}(\\outerXinit{t})}^\\rho.\n\\end{equation}\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"8e0962380ffac61e9186ef181af899b035124d047d11e94123fb7633cd8dfffe"},{"start":74461,"end":76421,"start_line":1385,"end_line":1422,"raw_tex":"\\begin{proof}\n(i)\nSince $\\Linit \\in [L_{\\min}, \\gamma_{\\rm inc}L_f]$, it follows $\\outerL{0},\\outerM{0}\\in [L_{\\min}, \\gamma_{\\rm inc}L_f]$ by Lemma~\\ref{lem:PG-iter}~(ii).\nThen, using Proposition~\\ref{prop:APG} (i) inductively, we obtain (i).\n\n(ii)\nFor $t=0$, we have $\\varphi(\\outerXplus{0})=\\varphi(T_{\\outerM{0}}(\\xinit))\\leq \\varphi(\\xinit)$ by Lemma~\\ref{lem:PG-iter} (i).\nMoreover, the criterion \\eqref{APG-iter-test-c} and Proposition~\\ref{prop:APG}~(iv), applied to the subroutine $\\AdaAPG$ in $\\rAdaAPG$, yield\n${\\varphi(\\outerXplus{t+1}) \\leq \\varphi(\\outerXinit{t+1})}$ and $\\varphi(\\outerXinit{t+1}) \\leq \\varphi(\\outerXplus{t})$, respectively. This shows $\\varphi(\\outerXplus{t+1}) \\leq \\varphi(\\outerXplus{t})$.\n\n\n(iii)\nSince the recurrence\n$\n\\ep^{(t)}/\\theta=\\norm{g_{\\outerM{t}}(\\outerXinit{t})} \\leq \\ep^{(t-1)}\n$\nholds for $t\\geq 1$, we have\n$\n\\ep^{(0)} \\geq \\ep^{(t)}/\\theta^t\n$\nfor each $t\\geq 0$.\nTherefore, we conclude\n$\nt \\leq \\log_{1/\\theta}\\frac{\\ep^{(0)}}{\\ep^{(t)}} = \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\norm{g_{\\outerM{t}}(\\outerXinit{t})}}.\n$\n\n\n(iv)\nSince $\\outerXplus{t}=T_{\\outerM{t}}(\\outerXinit{t})$, Lemma~\\ref{lem:gmap} (iii) implies that $g_t:=\\nabla{f}(\\outerXplus{t})-\\nabla{f}(\\outerXinit{t})+g_{\\outerM{t}}(\\outerXinit{t})$ belongs to $\\partial{\\varphi}(\\outerXplus{t})$.\nThen,\nLemma~\\ref{lem:HEB} shows (note that $\\outerXplus{t}$ belongs to $\\lev_\\varphi(\\varphi(\\xinit))\\setminus X^*$ by (ii))\n\\begin{align*}\n\\kappa\\dist(\\outerXplus{t},X^*)^{\\rho-1} &\\leq \\norm{g_t}\n\\leq \\left(\\frac{L_f}{\\outerM{t}}+1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})}\n\\leq \\left(\\frac{L_f}{L_{\\min}}+1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})},\n\\end{align*}\nwhere the second inequality is due to Lemma~\\ref{lem:gmap} (iii) and the last follows by (i). This shows the assertion \\eqref{rAdaAPG-dist-bound}. Similarly, \\eqref{rAdaAPG-obj-bound} can be obtained using the latter inequality of \\eqref{Loj-ineq}.\n\\end{proof}","sha256_utf8":"ca560d103b7e5d95e26da3bd452b690e0b4037fcefab1ff8f9ac85e5fee5cd24"},{"start":76512,"end":77226,"start_line":1426,"end_line":1445,"raw_tex":"\\begin{lemma}\\label{lem:rAdaAPG-compl}\nLet $N^{(t)}$ be the number of executions of $\\APGIter$ at the $t$-th outer loop of $\\rAdaAPG$.\nAssume that, for some $T_*\\geq 0$, $\\sigma_*>0$, and $\\ep_*>0$, we have\n$$\n\\outersigma{t+1}\\geq \\sigma_*\\quad (t=0,\\ldots,T_*)\n\\quad\n\\text{and}\n\\quad\n\\norm{g_{\\outerM{T_*}}(\\outerXinit{T_*})} \\geq \\ep_*.\n$$\nThen, under the definition \\eqref{N-def} of $N(\\cdot,\\cdot,\\cdot)$, the following inequality holds.\n$$\n\\sum_{t=0}^{T_*}N^{(t)} \\leq N(\\ep_*,\\sigma_*,C) \\leq N(\\ep_*,\\sigma_*,C_*),\n$$\nwhere\n$$\nC := \\sum_{t=0}^{T_*}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n\\leq C_*:= \\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*}\\right).\n$$\n\\end{lemma}","sha256_utf8":"72be9b0d4bd77ce8d8472f319fc8c41feae7be3144d330ec907ed8d7635e1e19"},{"start":77227,"end":80304,"start_line":1446,"end_line":1497,"raw_tex":"\\begin{proof}\nSince $\\outersigma{t+1}\\geq \\sigma_*$ for each $t=0,\\ldots,T_*$, Theorem~\\ref{thm:AdaAPG} (ii) gives the following bound for $t=0,\\ldots,T_*$:\n\\begin{align*}\nN^{(t)}\n&\\leq \n\\left(1+\\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{t}}{\\outersigma{t+1}}\\right)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\outersigma{t+1}}{\\beta\\outersigma{t+1}}\\right)\\\\\n&\\quad +\n\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\outersigma{t+1}}} - \\sqrt{\\frac{1}{\\outersigma{t}}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\outersigma{t+1}}{\\beta\\outersigma{t+1}}\\\\\n&\\leq\n\\left(1+\\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{t}}{\\outersigma{t+1}}\\right)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}\\right)\\\\\n&\\quad +\n\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\outersigma{t+1}}} - \\sqrt{\\frac{1}{\\outersigma{t}}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}.\n\\end{align*}\nBy Lemma~\\ref{lem:rAdaAPG-base} (iii), $T_*$ is bounded by\n$$\nT_*\\leq \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\norm{g_{\\outerM{T_*}}(\\outerXinit{T_*})}} \\leq \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*}.\n$$\nTo estimate the sum $\\sum_{t=0}^{T_*}N^{(t)}$, note that\n$$\n\\sum_{t=0}^{T_*}\\left(1+\\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{t}}{\\outersigma{t+1}}\\right) = 1+T_*+\\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{0}}{\\outersigma{T_*+1}} \\leq 1+ \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*} + \\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{0}}{\\sigma_*},\n$$\nand\n\\begin{align*}\n\\sum_{t=0}^{T_*}\n\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\outersigma{t+1}}} - \\sqrt{\\frac{1}{\\outersigma{t}}}\\right]\n&=\n(\\sqrt{\\gamma_{\\rm reg}}-1)\\sum_{t=0}^{T_*}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n+\\sum_{t=0}^{T_*}\n\\left[\\sqrt{\\frac{1}{\\outersigma{t+1}}} - \\sqrt{\\frac{1}{\\outersigma{t}}}\\right]\\\\\n&=\n(\\sqrt{\\gamma_{\\rm reg}}-1)\\sum_{t=0}^{T_*}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n+\n\\sqrt{\\frac{1}{\\outersigma{T_*+1}}} - \\sqrt{\\frac{1}{\\outersigma{0}}}.\n\\end{align*}\nTherefore, $\\sum_{t=0}^{T_*}N^{(t)}$ is bounded by\n\\begin{align*}\n\\sum_{t=0}^{T_*}N^{(t)}\n&\\leq \\left(1+ \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*} + \\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{0}}{\\sigma_*}\\right)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}\\right)\\\\\n&\\quad +\n\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\n\\sqrt{\\frac{1}{\\sigma_*}} - \\sqrt{\\frac{1}{\\outersigma{0}}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*} \\\\\n&\\quad + \\sqrt{2\\gamma_{\\rm inc}L_f} \\sum_{t=0}^{T_*}\\sqrt{\\frac{1}{\\outersigma{t+1}}} \\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}\\\\\n&=N(\\ep_*,\\sigma_*,C).\n\\end{align*}\nFinally, $C$ has the following bound:\n$$\nC = \\sum_{t=0}^{T_*}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n\\leq \\sqrt{\\frac{1}{\\sigma_*}} (1+T_*)\\leq\n\\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*}\\right)=C_*,\n$$\nwhich also concludes $N(\\ep_*,\\sigma_*,C) \\leq N(\\ep_*,\\sigma_*,C_*)$.\n\\end{proof}","sha256_utf8":"5ee41c48d7e97bb135a5840143f9db557a083e734e77b06eeeaae26884ae16f7"},{"start":80429,"end":81852,"start_line":1502,"end_line":1543,"raw_tex":"\\begin{lemma}\\label{lem:rAdaAPG-sigma}\nAssume that the H\\\"olderian error bound condition \\eqref{HEB-review} holds.\nSuppose that $\\rAdaAPG$ terminated with the stopping criterion at the $(T+1)$-th outer loop for some $T\\geq 0$.\nLet $\\bar\\sigma$ be defined by \\eqref{bar-sigma-def}.\n\\begin{enumerate}[(i)]\n\\item\nFor each $t=0,\\ldots,T$, we have\n$$\n\\outersigma{t+1}\n\\left\\{\n\\begin{array}{ll}\n=\\outersigma{t}&(\\outersigma{t}\\leq \\sigma(\\outerXplus{t},\\ep^{(t)})),\\\\\n\\geq\\sigma(\\outerXplus{t},\\ep^{(t)})/\\gamma_{\\rm reg}&(\\text{otherwise}),\n\\end{array}\n\\right.\n$$\nwhere $\\sigma(\\cdot,\\cdot)$ is defined by \\eqref{sigma-th}. \n\n\\item It follows that $\\sigma(\\outerXplus{t},\\ep^{(t)}) \\geq \\bar\\sigma$ for all $t=0,\\ldots,T$.\n%\\begin{equation}\\label{sigma-th-const}\n%\\sigma(\\outerXplus{t},\\ep^{(t)}) \\geq \\bar\\sigma, \\quad t=0,\\ldots,T.\n%\\end{equation}\nWhen $\\rho \\geq 2$, we further obtain\n\\begin{equation}\\label{sigma-th-t}\n\\sigma(\\outerXplus{t},\\ep^{(t)}) \\geq \\frac{\\theta^{\\frac{1}{\\rho-1}}}{1+\\sqrt{2}\\beta}\\cdot\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}}\n(\\ep^{(t)})^{\\frac{\\rho-2}{\\rho-1}}\n \\geq \\bar\\sigma,\n\\end{equation}\nfor each $t=0,\\ldots,T$.\n\\item\nFor each $t=0,\\ldots,T+1$, we obtain\n$$\n\\outersigma{t} \\left\\{\n\\begin{array}{ll}\n=\\sigmainit & (\\sigmainit \\leq \\bar\\sigma),\\\\\n\\geq\\bar\\sigma/\\gamma_{\\rm reg} & (\\text{otherwise}).\n\\end{array}\n\\right.\n$$\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"6ab60fbff256977bc00ff44516593c8827fe6d8ee241805720a460070261d520"},{"start":81854,"end":85603,"start_line":1545,"end_line":1598,"raw_tex":"\\begin{proof}\n(i)\nIn the case $\\outersigma{t} \\leq \\sigma(\\outerXplus{t},\\ep^{(t)})$, Theorem~\\ref{thm:AdaAPG} (i) implies that the subroutine $\\AdaAPG$ at the $t$-th outer loop must terminate at the first loop $j=0$ so that $\\outersigma{t+1}$ is defined by $\\outersigma{t+1}=\\outersigma{t}$.\nOn the other hand, consider the case $\\outersigma{t} > \\sigma(\\outerXplus{t},\\ep^{(t)})$.\nWhen the subroutine $\\AdaAPG$ at the $t$-th outer loop terminates at the first loop $j=0$, then it is clear that\n$\\outersigma{t+1}=\\outersigma{t} > \\sigma(\\outerXplus{t},\\ep^{(t)}) \\geq \\sigma(\\outerXplus{t},\\ep^{(t)})/\\gamma_{\\rm reg}.$\nIn the other case, the latter assertion of Theorem~\\ref{thm:AdaAPG}~(i) implies $\\outersigma{t+1}\\geq \\sigma(\\outerXplus{t},\\ep^{(t)})/\\gamma_{\\rm reg}.$\n\n(ii)\nWe may assume $\\outerXplus{t} \\not\\in X^*$, which allows us to apply Lemma~\\ref{lem:rAdaAPG-base} (iv) (Note that the assertion is clear if $\\outerXplus{t} \\in X^*$ since then $\\sigma(\\outerXplus{t},\\ep^{(t)})=+\\infty$).\nIn the case $\\rho\\geq 2$, using \\eqref{rAdaAPG-dist-bound} implies\n$$\n\\sigma(\\outerXplus{t},\\ep^{(t)})\n= \\frac{\\theta\\norm{g_{\\outerM{t}}(\\outerXinit{t})}}{(1+\\sqrt{2}\\beta)\\dist(\\outerXplus{t},X^*)}\n\\geq\n\\frac{\\theta}{1+\\sqrt{2}\\beta}\\cdot\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}}\n\\norm{g_{\\outerM{t}}(\\outerXinit{t})}^{\\frac{\\rho-2}{\\rho-1}}.\n$$\nSince $\\ep^{(t)}\\equiv \\theta\\|g_{\\outerM{t}}(\\outerXinit{t})\\|$ and $\\|g_{\\outerM{t}}(\\outerXinit{t})\\| \\geq \\ep$ (by the definition of $T$), we obtain \\eqref{sigma-th-t}.\n\nIf $\\rho\\in [1,2[$, on the other hand,\nsince $\\varphi(\\outerXplus{t})\\leq \\varphi(\\xinit)$ by Lemma~\\ref{lem:rAdaAPG-base} (ii), the inequalities \\eqref{HEB-review} and \\eqref{rAdaAPG-obj-bound} imply (remark that $\\frac{2-\\rho}{\\rho}>0$)\n\\begin{align*}\n\\dist(\\outerXplus{t},X^*) &\\leq \\frac{1}{\\kappa^{\\frac{1}{\\rho}}}(\\varphi(\\outerXplus{t})-\\varphi^*)^{\\frac{1}{\\rho}}\n=\n\\frac{1}{\\kappa^{\\frac{1}{\\rho}}}(\\varphi(\\outerXplus{t})-\\varphi^*)^{\\frac{\\rho-1}{\\rho}}(\\varphi(\\outerXplus{t})-\\varphi^*)^{\\frac{2-\\rho}{\\rho}}\\\\\n&\\leq\n\\frac{1}{\\kappa^{\\frac{2}{\\rho}}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})}(\\varphi(\\outerXplus{t})-\\varphi^*)^{\\frac{2-\\rho}{\\rho}}\\\\\n&\\leq\n\\frac{1}{\\kappa^{\\frac{2}{\\rho}}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})}(\\varphi(\\xinit)-\\varphi^*)^{\\frac{2-\\rho}{\\rho}}.\n\\end{align*}\nTherefore, we conclude that\n$$\n\\sigma(\\outerXplus{t},\\ep^{(t)})\n= \\frac{\\theta\\norm{g_{\\outerM{t}}(\\outerXinit{t})}}{(1+\\sqrt{2}\\beta)\\dist(\\outerXplus{t},X^*)}\n\\geq\n\\frac{\\theta}{1+\\sqrt{2}\\beta}\n\\kappa^{\\frac{2}{\\rho}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-1}(\\varphi(\\xinit)-\\varphi^*)^{-\\frac{2-\\rho}{\\rho}}\n=\\bar\\sigma.\n$$\nThis proves the assertion (ii).\n\n(iii)\nIn the case $\\sigmainit\\leq \\bar\\sigma$, (ii) implies $\\outersigma{0} \\leq \\sigma(\\outerXplus{0},\\ep^{(0)})$. Then, using (i), we have $\\outersigma{1}=\\outersigma{0}$ and also $\\outersigma{1} \\leq \\bar\\sigma$.\nContinuing this argument inductively, we conclude that $\\outersigma{t}=\\outersigma{t-1}=\\cdots=\\outersigma{0}$ for all $t$.\n\nIn the case $\\sigmainit \\geq \\bar\\sigma$, on the other hand, let us show $\\outersigma{t} \\geq \\bar\\sigma/\\gamma_{\\rm reg}~(t=0,\\ldots,T+1)$ by induction. The assertion for $t=0$ is clear since $\\sigmainit \\geq \\bar\\sigma \\geq \\bar\\sigma/\\gamma_{\\rm reg}$.\nAssume that $\\outersigma{t}\\geq \\bar\\sigma/\\gamma_{\\rm reg}$ holds for some $t\\geq 0$. By (i) and (ii), we have\n$\n\\outersigma{t+1} \\geq \\min\\{\\outersigma{t}, \\sigma(\\outerXplus{t},\\ep^{(t)})/\\gamma_{\\rm reg}\\} \\geq \\min\\{\\bar\\sigma/\\gamma_{\\rm reg}, \\bar\\sigma/\\gamma_{\\rm reg}\\} =\\bar\\sigma/\\gamma_{\\rm reg}.\n$\nThis completes the proof of (iii).\n\\end{proof}","sha256_utf8":"4e6ddb1f31fbcdf2d80d20a9ee5a036e2c42c3a8fd631aeade3875dfaf1b6f56"},{"start":85733,"end":86907,"start_line":1602,"end_line":1619,"raw_tex":"\\begin{lemma}\\label{lem:rAdaAPG-sigma-2}\nAssume that the H\\\"olderian error bound condition \\eqref{HEB-review} holds with $\\rho >2$.\nSuppose that $\\rAdaAPG$ terminated with the stopping criterion at the $(T+1)$-th outer loop for some $T\\geq 0$.\nIf $\\sigmainit \\geq \\bar\\sigma$ holds for $\\bar\\sigma$  defined by \\eqref{bar-sigma-def},\nthen there exists $t_0 \\in \\{0,\\ldots,T+1\\}$ such that the following conditions hold, where\n$\n\\sigma_*^{(0)}:=\\frac{\\theta}{1+\\sqrt{2}\\beta}\\cdot\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}}\n\\|g_{\\outerM{0}}(\\xinit)\\|^{\\frac{\\rho-2}{\\rho-1}}.\n$\n\\begin{align}\n\\text{(i)\\qquad} & \\outersigma{0}=\\cdots=\\outersigma{t_0}. \\label{sigma-bound-rho>2-1}\\\\\n\\text{(ii)\\qquad} & \\outersigma{t+1} \\geq \\theta^{-\\frac{\\rho-2}{\\rho-1}(T-t)} \\bar\\sigma/\\gamma_{\\rm reg},\\quad t=t_0,\\ldots,T.\\label{sigma-bound-rho>2-2}\\\\\n\\text{(iii)\\qquad} & t_0 \\leq 1+\\frac{\\rho-1}{\\rho-2}\\log_{1/\\theta}\\frac{\\sigma_*^{(0)}}{\\min(\\sigmainit,\\sigma_*^{(0)})}.\\label{sigma-bound-rho>2-3}\\\\\n\\text{(iv)\\qquad} & \\theta^{\\frac{\\rho-2}{\\rho-1}(T-t_0)} \\geq {\\bar\\sigma}/{\\outersigma{0}}.\\label{sigma-bound-rho>2-4}\n\\end{align}\n\n\\end{lemma}","sha256_utf8":"c75db0f27fcd5c8cf1065970691258ca91a2c49654a24acf7d244bd0e73e39b6"},{"start":86908,"end":90540,"start_line":1620,"end_line":1692,"raw_tex":"\\begin{proof}\nDefine\n$$\n\\sigma_*^{(t)}:=\\frac{\\theta^{\\frac{1}{\\rho-1}}}{1+\\sqrt{2}\\beta}\\cdot\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}}\n(\\ep^{(t)})^{\\frac{\\rho-2}{\\rho-1}},\\quad t \\geq 0.\n$$\nThen, Lemma~\\ref{lem:rAdaAPG-sigma} (ii) can be written as\n\\begin{equation}\\label{simga_*^t-bound}\n\\sigma(\\outerXplus{t},\\ep^{(t)}) \\geq \\sigma_*^{(t)} \\geq \\bar\\sigma,\\quad t=0,\\ldots,T.\n\\end{equation}\nFor simplicity, denote\n$$\n\\omega:=\\frac{\\rho-2}{\\rho-1} \\in ]0,1[,\\quad\nc:=\\frac{\\theta^{\\frac{1}{\\rho-1}}}{1+\\sqrt{2}\\beta}\\cdot\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}},\n$$\nso that we have\n$$\n\\bar\\sigma = c(\\theta\\ep)^{\\omega},\\quad\n\\sigma_*^{(t)}=c(\\ep^{(t)})^{\\omega},\\quad t\\geq 0.\n$$\nNote that $\\{\\sigma_*^{(t)}\\}$ is non-increasing;\nin fact, the relation $\\ep^{(t+1)}\\leq \\theta\\ep^{(t)}$ implies\n\\begin{equation}\\label{sigma-*-mono}\n\\sigma_*^{(t+1)}=c(\\ep^{(t+1)})^\\omega\\leq c\\theta^\\omega(\\ep^{(t)})^\\omega = \\theta^\\omega \\sigma_*^{(t)} \\leq \\sigma_*^{(t)}.\n\\end{equation}\n\nLet $t_0$ be the smallest integer in $\\{0,\\ldots,T+1\\}$ such that $\\sigmainit \\geq \\sigma_*^{(t_0)}$.\nRemark that, by the definition of $T$, we have $\\ep^{(T+1)}=\\theta\\|g_{\\outerM{T+1}}(\\outerXinit{T+1})\\| \\leq \\theta\\ep$.\nThis implies that\n$\n\\sigmainit \\geq \\bar\\sigma= c(\\theta\\ep)^\\omega \\geq c(\\ep^{(T+1)})^\\omega = \\sigma_*^{(T+1)}.\n$\nTherefore, $t_0$ is well-defined.\n\n%%%\n(i)\nBy the definition of $t_0$ and \\eqref{simga_*^t-bound}, we have\n$\\sigmainit < \\sigma_*^{(t)} \\leq \\sigma(\\outerXplus{t},\\ep^{(t)})$ ($0\\leq \\forall t < t_0$).\nTherefore, by induction, we obtain $\\outersigma{0}=\\outersigma{1}=\\cdots=\\outersigma{t_0}$ due to Lemma~\\ref{lem:rAdaAPG-sigma}~(i).\n\n\n(ii)\nLet us show $\\outersigma{t+1}\\geq \\sigma_*^{(t)}/\\gamma_{\\rm reg}~(t_0\\leq t\\leq T)$.\nTo prove this, we verify $\\outersigma{t}\\geq \\sigma_*^{(t)}/\\gamma_{\\rm reg}$ for ${t= t_0,\\ldots,T+1}$ by induction. Note that $\\outersigma{t_0}=\\outersigma{0} \\geq  \\sigma_*^{(t_0)} > \\sigma_*^{(t_0)}/\\gamma_{\\rm reg}$ holds by (i) and the definition of $t_0$. Now under the hypothesis $\\outersigma{t} \\geq  \\sigma_*^{(t)}/\\gamma_{\\rm reg}$ for $t$ with $t_0\\leq t \\leq T$, Lemma~\\ref{lem:rAdaAPG-sigma}~(i) and \\eqref{simga_*^t-bound} imply\n$$\\outersigma{t+1} \\geq \\min\\{\\outersigma{t},\\sigma(\\outerXplus{t},\\ep^{(t)})/\\gamma_{\\rm reg}\\} \\geq \\min\\{\\sigma_*^{(t)}/\\gamma_{\\rm reg},\\sigma_*^{(t)}/\\gamma_{\\rm reg}\\} = \\sigma_*^{(t)}/\\gamma_{\\rm reg} \\geq \\sigma_*^{(t+1)}/\\gamma_{\\rm reg}.$$\nTherefore, this completes the induction; in addition, the above inequality involves the desired inequality $\\outersigma{t+1}\\geq \\sigma_*^{(t)}/\\gamma_{\\rm reg}~(t_0\\leq t\\leq T)$.\nThis yields (ii) combined with \\eqref{simga_*^t-bound} and \\eqref{sigma-*-mono}:\n$$\n\\outersigma{t+1} \\geq \\sigma_*^{(t)}/\\gamma_{\\rm reg} \\geq \\theta^{-\\omega(T-t)}\\sigma_*^{(T)}/\\gamma_{\\rm reg} \\geq \\theta^{-\\omega(T-t)} \\bar\\sigma/\\gamma_{\\rm reg},\\quad t=t_0,\\ldots,T.\n$$\n%%%\n\n(iii)\nIf $t_0=0$, then\n(iii) is trivial since\n$\\sigmainit \\geq\\sigma_*^{(t_0)}=\\sigma_*^{(0)}$. If $t_0 > 0$, then the definition of $t_0$ and using \\eqref{sigma-*-mono} imply\n$\\sigmainit < \\sigma_*^{(t_0-1)} \\leq \\theta^{\\omega(t_0-1)}\\sigma_*^{(0)},$\nwhich yields $t_0 \\leq 1+\\frac{1}{\\omega}\\log_{1/\\theta}\\frac{\\sigma_*^{(0)}}{\\sigmainit}$.\n\n\n(iv)\nBy \\eqref{simga_*^t-bound}, \\eqref{sigma-*-mono}, and the definition of $t_0$, remark that\n$\n\\bar\\sigma \\leq \\sigma_*^{(T)} \\leq \\theta^{\\omega(T-t_0)}\\sigma_*^{(t_0)}\\leq \\theta^{\\omega(T-t_0)}\\sigmainit.\n$\nHence, \n$\n\\theta^{\\omega(T-t_0)} \\geq {\\bar\\sigma}/{\\outersigma{0}}\n$\nholds.\n\\end{proof}","sha256_utf8":"637300a1c2a31fba47d7adabcbd352b42c2354a228c512db9e1e852d47462f5b"},{"start":90637,"end":99547,"start_line":1697,"end_line":1860,"raw_tex":"\\begin{proof}[\\underline{Proof of Theorem~\\ref{thm:rAdaAPG}}]\nWe may assume that $\\norm{g_{\\outerM{0}}(\\xinit)} > \\ep$ since $N=0$ on the other case.\nSuppose that $\\rAdaAPG$ terminated with the stopping criterion at the $(T+1)$-th outer loop for some $T\\geq 0$.\nDenote by $N^{(t)}$ the number of executions of $\\APGIter$ at the $t$-th outer loop so that $N=\\sum_{t=0}^T N^{(t)}$.\nLet $\\bar\\sigma$ be defined by \\eqref{bar-sigma-def}.\n\nBy the definition of $T$, we have\n\\begin{equation}\\label{pr:gradmap-lb-ep}\n\\norm{g_{\\outerM{t}}(\\outerXinit{t})} > \\ep,\\quad t=0,\\ldots,T.\n\\end{equation}\nMoreover, by the definition of $\\sigma_*$ in \\eqref{simga*-def},\nusing Lemma~\\ref{lem:rAdaAPG-sigma} (iii) implies\n$$\n\\outersigma{t} \\geq\n\\sigma_*,\\quad t=0,\\ldots,T+1.\n$$\nTherefore, applying Lemma~\\ref{lem:rAdaAPG-compl} with $T_*=T$, we obtain the assertion\n\\begin{equation}\\label{N-ub-basic-HEB}\nN \\leq N(\\ep,\\sigma_*,C) \\quad \\text{with}\\quad\nC=\\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep}\\right).\n\\end{equation}\nFor the case $\\rho=2$, \\eqref{N-ub-basic-HEB} proves the assertion (i).\nWe discuss the other cases to improve \\eqref{N-ub-basic-HEB}.\n\n\\fbox{(ii) Case $\\rho>2$.}\nIf $\\sigmainit < \\bar\\sigma$, then $\\sigma_*$ is defined as $\\sigma_*=\\sigmainit$. \nTherefore, the latter bound of (ii) is obtained by \\eqref{N-ub-basic-HEB}.\n\nNow consider the case $\\sigmainit \\geq \\bar\\sigma$.\nThen, there exists $t_0 \\in \\{0,\\ldots,T+1\\}$ satisfying the conditions in Lemma~\\ref{lem:rAdaAPG-sigma-2}.\nBy \\eqref{sigma-bound-rho>2-4}, remark that\n\\begin{equation}\\label{rAdaAPG-compl-p>2-pf-sum}\n\\sum_{t=t_0}^{T}\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}(T-t)}}\n=\n\\sum_{i=0}^{T-t_0}\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}}^{\\,i}\n=\n\\frac{1-\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}}^{T-t_0+1}}{1-\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}}}\n\\leq\n\\frac{1-\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}\\cdot\\dfrac{\\bar\\sigma}{\\outersigma{0}}}}{1-\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}}}.\n\\end{equation}\nTherefore, we conclude that\n\\begin{align*}\n\\sum_{t=0}^{T}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n&=\n\\sum_{t=0}^{t_0-1} \\sqrt{\\frac{1}{\\outersigma{t+1}}}+\n\\sum_{t=t_0}^{T}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n\\\\\n&\\leq\nt_0 \\sqrt{\\frac{1}{\\outersigma{0}}} +\n\\sqrt{\\frac{\\gamma_{\\rm reg}}{\\bar\\sigma}}\\sum_{t=t_0}^{T}\\sqrt{\\theta^{{\\frac{\\rho-2}{\\rho-1}}(T-t)}} & (\\text{by } \\eqref{sigma-bound-rho>2-1} \\text{ and } \\eqref{sigma-bound-rho>2-2})\\\\\n&\\leq\n\\sqrt{\\frac{1}{\\outersigma{0}}}\\left(1+\\frac{\\rho-1}{\\rho-2}\\log_{1/\\theta}\\frac{\\sigma_*^{(0)}}{\\min(\\sigma_*^{(0)},\\outersigma{0})}\\right)\\\\\n&\\quad~+\n\\frac{\\sqrt{\\gamma_{\\rm reg}}}{1-\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}}}\n\\left(\n\\sqrt{\\frac{1}{\\bar\\sigma}}\n-\n\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}\n}}\n\\sqrt{\\frac{1}{\\outersigma{0}}}\n\\right) & (\\text{by } \\eqref{sigma-bound-rho>2-3} \\text{ and } \\eqref{rAdaAPG-compl-p>2-pf-sum})\\\\\n&=:C.\n\\end{align*}\nWith this definition of $C$, Lemma~\\ref{lem:rAdaAPG-compl} gives $N \\leq N(\\ep,\\sigma_*,C)$.\n\n\n\\fbox{(iii) Case $\\rho \\in ]1,2[$.}\nIf $\\ep \\geq \\ep_*$, then \\eqref{N-ub-basic-HEB} gives our assertion because the second term of \\eqref{N-bd-superlinear} vanishes.\nSuppose, on the other hand, that $\\ep < \\ep_*$. \nDenote\n$\\xi_{t}:=\\norm{g_{\\outerM{t}}(\\outerXinit{t})}$\nand \nlet $T_*\\geq 0$ be the smallest integer such that $\\xi_{T_*} \\leq \\ep_*$.\nThen, since $\\xi_{T_*-1} > \\ep_*$ holds, Lemma~\\ref{lem:rAdaAPG-compl} shows that\n$$\n\\sum_{t=0}^{T_*-1} N^{(t)} \\leq N(\\ep_*,\\sigma_*,C)\\quad \\text{with}\\quad C=\\sqrt{\\frac{1}{\\sigma_*}}\\left( 1 + \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*} \\right).\n$$\nNote that, under the convention $\\sum_{t=0}^{-1}(\\cdot)=0$, this inequality also holds if $T_*=0$ since then $\\xi_0\\leq \\ep_*$ and $N(\\ep_*,\\sigma_*,C)=0$.\n\nIt remains to observe $\\sum_{t=T_*}^T N^{(t)}$.\nTake $t\\in \\{T_*,\\ldots,T\\}$. We shall prove $N^{(t)}=1$.\nIn the $t$-th outer loop, consider the first iteration of the subroutine $\\AdaAPG(\\outerXplus{t},\\outerL{t},\\outersigma{t},\\ep^{(t)})$, which executes\n$$\n\\{x_1,\\psi_1,M_0,L_1,A_1\\} \\leftarrow \\APGIter_{\\outersigma{t}}(\\outerXplus{t},\\psi_0,\\outerL{t},A_0),\n$$\nwhere $\\psi_0(x)=\\frac{1}{2}\\|x-\\outerXplus{t}\\|^2$ and $A_0=0$.\nThen, Proposition~\\ref{prop:APG} (v) implies\n$$\n\\norm{g_{M_{0}}(x_1)} \\leq \\left(2\\sqrt{\\frac{M_{0}+\\outersigma{t}}{A_1}} + \\outersigma{t}\\right)\\dist(\\outerXplus{t},X^*).\n$$\nMoreover, according to the equation at Line~\\ref{a-quad-eq} in Algorithm~\\ref{APG-iter}, $A_1$ can be calculated as $A_1=2/M_0$. Now remark that, using $M_0 \\leq \\gamma_{\\rm inc}L_f$ (Proposition~\\ref{prop:APG}~(i)), we have\n\\begin{align*}\n2\\sqrt{\\frac{M_{0}+\\outersigma{t}}{A_1}} + \\outersigma{t} &= 2\\sqrt{\\frac{M_0(M_0+\\outersigma{t})}{2}} + \\outersigma{t}\n\\leq 2\\sqrt{\\frac{(M_0+\\outersigma{t})^2}{2}} + (M_0+\\outersigma{t}) \\\\&= (1+\\sqrt{2})(M_0+\\outersigma{t}) \\leq (1+\\sqrt{2})(\\gamma_{\\rm inc}L_f + \\outersigma{t}).\n\\end{align*}\nCombining them and using \\eqref{rAdaAPG-dist-bound}\\footnote{Although \\eqref{rAdaAPG-dist-bound} is asserted in the case $\\outerXplus{t} \\not\\in X^*$, it trivially holds if $\\outerXplus{t} \\in X^*$ \\emph{unless} $\\rho = 1$.}, we conclude that\n\\begin{align}\n\\norm{g_{M_{0}}(x_1)}\n&\\leq\n(1+\\sqrt{2})(\\gamma_{\\rm inc}L_f+\\outersigma{0})\\dist(\\outerXplus{t},X^*) \\nonumber\\\\\n&\\leq\n(1+\\sqrt{2})(\\gamma_{\\rm inc}L_f+\\outersigma{0})\n\\left(\\frac{1}{\\kappa}\\right)^{\\frac{1}{\\rho-1}} \\left(\\frac{L_f}{L_{\\min}}+1\\right)^{\\frac{1}{\\rho-1}}\\xi_t^{\\frac{1}{\\rho-1}}\n\\nonumber\\\\\n&= \\theta\\ep_*^{\\frac{\\rho-2}{\\rho-1}}\\xi_t^{\\frac{1}{\\rho-1}}\n\\label{superlinear-rec}\\\\\n&\\leq \\theta\\xi_t^{\\frac{\\rho-2}{\\rho-1}}\\xi_t^{\\frac{1}{\\rho-1}}=\\theta\\xi_t = \\ep^{(t)},\n\\nonumber\n\\end{align}\nwhere the last inequality is due to $\\xi_t \\leq \\ep_*$ for $t \\geq T_*$ (and remark $\\frac{\\rho-2}{\\rho-1} < 0$).\nThis shows $N^{(t)}=1$ and \\eqref{superlinear-rec} yields the recurrence\n$\n\\xi_{t+1} \\leq \\theta\\ep_*^{\\frac{\\rho-2}{\\rho-1}}\\xi_t^{\\frac{1}{\\rho-1}}\n$,\nfor each $t=T_*,\\ldots,T$.\nSince $\\frac{1}{\\rho-1} > 1$, it reduces $\\xi_t$ superlinearly. In particular, solving this recurrence implies\n$$\n\\log \\frac{\\ep_*}{\\theta^{\\frac{\\rho-1}{2-\\rho}}\\xi_T} \\geq \\left(\\frac{1}{\\rho-1}\\right)^{T-T_*}\\log \\frac{\\ep_*}{\\theta^{\\frac{\\rho-1}{2-\\rho}}\\xi_{T_*}}.\n$$\nSince $\\xi_T > \\ep$ and $\\xi_{T_*}\\leq \\ep_*$, we obtain\n$$\n\\sum_{t=T_*}^T N^{(t)} =\nT-T_*+1 \\leq  1+ \\left(\\log\\frac{1}{\\rho-1}\\right)^{-1}\\left(\\log\\log \\frac{\\ep_*}{\\theta^{\\frac{\\rho-1}{2-\\rho}}\\ep}-\\log\\log\\frac{1}{\\theta^{\\frac{\\rho-1}{2-\\rho}}}\\right).\n$$\nConsequently, $N$ is bounded as follows.\n$$\nN = \\sum_{t=0}^{T_*-1} N^{(t)} + \\sum_{t=T_*}^T N^{(t)} \\leq N(\\ep_*,\\sigma_*,C) + 1+\\left(\\log\\frac{1}{\\rho-1}\\right)^{-1}\\left(\\log\\log \\frac{\\ep_*}{\\theta^{\\frac{\\rho-1}{2-\\rho}}\\ep}-\\log\\log\\frac{1}{\\theta^{\\frac{\\rho-1}{2-\\rho}}}\\right).\n$$\n\n\\fbox{(iv) Case $\\rho=1$.}\nWe have\n$\n\\outersigma{t} \\geq\n\\sigma_*\n$ for each $t=0,\\ldots,T+1$, by Lemma~\\ref{lem:rAdaAPG-sigma} (iii).\nMoreover, Lemma~\\ref{lem:rAdaAPG-base}~(vi) shows that, if $\\outerXplus{t} \\not\\in X^*$, then we have\n$$\n1 \\leq \\frac{1}{\\kappa}\\left(\\frac{L_f}{L_{\\min}} + 1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})},\n~~\\text{i.e.,}~~\n\\norm{g_{\\outerM{t}}(\\outerXinit{t})} \\geq \\ep_*:=\\kappa\\left(\\frac{L_f}{L_{\\min}} + 1\\right)^{-1}.\n$$\nIn other words, the condition $\\norm{g_{\\outerM{t}}(\\outerXinit{t})} < \\ep_*$ must imply $\\outerXplus{t}\\in X^*$, which also yields $N^{(t)}=1$ and $\\outerXinit{t+1}=\\outerXplus{t} \\in X^*$ by Proposition~\\ref{prop:APG}~(v) (then the algorithm terminates at the ($t+1$)-th outer loop).\nTherefore, we have\n\\begin{equation}\\label{rho=1:gmap-ep*}\n\\norm{g_{\\outerM{t}}(\\outerXinit{t})} \\geq \\ep_*,\\quad 0\\leq t \\leq T-1.\n\\end{equation}\n\\sloppy\nNow we consider two cases. If $\\norm{g_{\\outerM{T}}(\\outerXinit{T})} \\geq \\ep_*$ holds, then\ncombining with \\eqref{pr:gradmap-lb-ep} yields ${\\norm{g_{\\outerM{T}}(\\outerXinit{T})} \\geq \\max(\\ep,\\ep_*)}$, from which Lemma~\\ref{lem:rAdaAPG-compl} with $T_*=T$ concludes\n$$\nN \\leq N(\\max(\\ep,\\ep_*),\\sigma_*,C)\n\\quad\\text{with}\\quad\nC=\n\\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\max(\\ep,\\ep_*)}\\right).\n$$\nOn the other case $\\norm{g_{\\outerM{T}}(\\outerXinit{T})} < \\ep_*$, we have $N^{(T)}=1$. Since \\eqref{pr:gradmap-lb-ep} and \\eqref{rho=1:gmap-ep*} implies ${\\norm{g_{\\outerM{T-1}}(\\outerXinit{T-1})} \\geq \\max(\\ep,\\ep_*)}$, Lemma~\\ref{lem:rAdaAPG-compl} with $T_*=T-1$ shows $\\sum_{t=0}^{T-1}N^{(t)}\\leq N(\\max(\\ep,\\ep_*),\\sigma_*,C)$. Hence,\n$$\nN = N^{(T)}+\\sum_{t=0}^{T-1}N^{(t)} \\leq 1 + N(\\max(\\ep,\\ep_*),\\sigma_*,C).\n$$\nThis proves the desired bound on $N$.\nTo show the latter assertion of (iv), suppose $\\ep<\\ep_*$. Then, the output $\\{\\outerXinit{T+1},\\outerM{T+1},\\outerXplus{T+1}\\}$ satisfies $\\norm{g_{\\outerM{T+1}}(\\outerXinit{T+1})} \\leq \\ep < \\ep_*$. Therefore, $\\outerXplus{T+1}$ must be optimal.\n\nThe proof of Theorem~\\ref{thm:rAdaAPG} is completed.\n\\end{proof}","sha256_utf8":"af4e9b0e0f501dea26471b395705fa1d256c907508a0fbd61fe0d028b6fb972e"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"复合凸优化问题与基本假设","source_units":[{"start":13502,"end":14724,"start_line":264,"end_line":284,"raw_tex":"Throughout this paper, let $\\E$ be a finite dimensional real Hilbert space endowed with an inner product $\\innprod{\\cdot}{\\cdot}$.\nWe denote by $\\norm{x}=\\innprod{x}{x}^{1/2}$ the induced norm on $\\E$.\nThe distance between a point $x \\in \\E$ and a set $S \\subset \\E$ is defined by\n$\\dist(x,S)=\\inf_{z \\in S}\\norm{z-x}.$\n\n\\sloppy % prevents overflow\nWe focus on the the convex composite optimization problem\n\\begin{equation}\\label{main-prob}\n\\varphi^* = \\min_{x \\in \\E}[\\varphi(x) \\equiv f(x)+\\varPsi(x)],\n\\end{equation}\nwhere $f:\\E\\to \\RR$ is an \\emph{$L_f$-smooth} convex function, that is, it is continuously differentiable and its gradient is $L_f$-Lipschitz continuous:\n$$\n\\norm{\\nabla{f}(x)-\\nabla{f}(y)} \\leq L_f\\norm{x-y}, \\quad \\forall x,y \\in \\E,\n$$\nand ${\\varPsi:\\E\\to\\RR\\cup\\{+\\infty\\}}$ is a proper, lower-semicontinuous, and convex function.\nWe denote by ${X^*:=\\Argmin_{x \\in \\E}\\varphi(x)}$ the set of optimal solutions of $\\min_{x \\in \\E}\\varphi(x)$.\n%$X^* = \\Argmin_{x \\in \\E}\\varphi(x).$\nThroughout the paper, we assume that $X^*$ is nonempty.\nThe subdifferential of $\\varphi$ at $x$ is denoted by\n${\\partial\\varphi(x) = \\{g \\in \\E \\mid \\varphi(y)\\geq \\varphi(x)+\\innprod{g}{y-x},~\\forall y \\in \\E\\}}$.\n\n","sha256_utf8":"046b2e75110c425f2614c29343a429bfd0054e2e2d1f26dae27561065cd78548"}]},{"id":"S002","semantic_type":"definition","kind":"definition","title":"近端模型与梯度映射","source_units":[{"start":14724,"end":15565,"start_line":285,"end_line":295,"raw_tex":"For the objective function $\\varphi(x)=f(x)+\\varPsi(x)$, we define\n$$\nm_L(y;x):= f(y)+\\innprod{\\nabla{f}(y)}{x-y} + \\frac{L}{2}\\norm{x-y}^2+\\varPsi(x),\\qquad T_L(y):= \\argmin_{x \\in \\E}m_L(y;x), %\\quad(=\\prox_{\\varPsi/L}(y-L^{-1}\\nabla{f}(y))),\n$$\nwhere the minimizer $T_L(y)$ is well-defined since $m_L(y;\\cdot)$ is strongly convex.\nIt is assumed that $\\varPsi(\\cdot)$ has a ``simple'' structure, namely, the proximal mapping $\\prox_{\\varPsi}(y):=\\argmin_{x \\in \\E}\\{\\varPsi(x)+\\frac{1}{2}\\norm{x-y}^2\\}$ as well as the map $T_L(y)\\equiv \\prox_{\\varPsi/L}(y-\\nabla{f}(x)/L)$ is moderately computable (see \\cite{PB14} for examples).\nThe \\emph{gradient mapping} of $\\varphi$ is defined by\n\\[\ng_L(y):=L(y-T_L(y)),\\quad y \\in \\E,~L>0.\n\\]\nFor instance, if $\\varPsi\\equiv 0$, we see that $T_L(y)=y-\\nabla{f}(y)/L$ and $g_L(y)=\\nabla{f}(y)$ hold.\n","sha256_utf8":"83bc277517368415d7b5e621ad8efad0c35d65c08d81be7855518791f3ec6217"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"霍尔德误差界定义","source_units":[{"start":19864,"end":20268,"start_line":394,"end_line":401,"raw_tex":"\\begin{definition}\nFix $x_0 \\in \\E$.\nWe say that $\\varphi$ satisfies the \\emph{H\\\"olderian error bound condition} with coefficient $\\kappa > 0$ and exponent $\\rho \\geq 1$ if\n\\begin{equation}\\label{HEB}\n\\varphi(x)-\\varphi^* \\geq \\kappa\\dist(x,X^*)^\\rho,\\quad \\forall x\\in \\lev_{\\varphi}(\\varphi(x_0)),\n\\end{equation}\nwhere $\\lev_{\\varphi}(\\gamma)=\\{x \\in \\E \\mid \\varphi(x)\\leq \\gamma\\}$.\n\\end{definition}","sha256_utf8":"6687fa126d4af7563c58af96e492d4d76d64d42aa0680d5f374d3b577b42ad66"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"正则化问题及其梯度映射","source_units":[{"start":29177,"end":30021,"start_line":546,"end_line":562,"raw_tex":"We consider to apply an accelerated proximal gradient method to the regularized problem\n$$\n\\min_{x\\in \\E}\\left[\\varphi_\\sigma(x):=\n\\varphi(x) + \\frac{\\sigma}{2}\\norm{x-x_0}^2\\right],\n$$\nwhere $x_0 \\in \\E$ is a fixed initial point and $\\sigma>0$ is a regularization parameter.\nSince $\\varphi_\\sigma$ is strongly convex, it has a unique minimizer. Let\n$$x^*_\\sigma := \\argmin_{x \\in \\E}\\varphi_\\sigma(x),\\quad \\varphi_\\sigma^* := \\min_{x\\in \\E} \\varphi_\\sigma(x).$$\nWe define the gradient mapping $g_L^\\sigma(x)$ for the regularized function $\\varphi_\\sigma$ in the following manner:\n$$\nf_\\sigma(x):=f(x)+\\frac{\\sigma}{2}\\norm{x-x_0}^2,\\qquad\nm_L^\\sigma(y;x) := f_\\sigma(y)+\\innprod{\\nabla{f}_\\sigma(y)}{x-y}+\\frac{L}{2}\\norm{x-y}^2 + \\varPsi(x),\n$$\n$$\nT_L^\\sigma(y):=\\argmin_{x\\in \\E}m_L^\\sigma(y;x),\\qquad\ng_L^\\sigma(y):=L(y-T_L^\\sigma(y)).\n$$\n","sha256_utf8":"cd99e2c97e34255f8c6be443600b03987603fc9f4584224957f5b7d75dda25cc"}]},{"id":"S005","semantic_type":"algorithm","kind":"algorithm","title":"正则项重分配与加速迭代","source_units":[{"start":32086,"end":33616,"start_line":608,"end_line":625,"raw_tex":"We employ Nesterov's accelerated proximal gradient method \\cite{Nes13} for solving $\\min_{x \\in E}\\varphi_\\sigma(x)$, by regarding the objective function as\n\\begin{equation}\\label{realloc}\n\\varphi_\\sigma(x) = f(x) + \\varPsi_\\sigma(x),\\quad \\varPsi_\\sigma(x):=\\varPsi(x) + \\frac{\\sigma}{2}\\norm{x-x_0}^2.\n\\end{equation}\nThe analogy of the definition of $T_L(\\cdot)$ for this regularization is \n\\begin{equation*}%\\label{tilde-T}\n\\tilde{T}_L(y) = \\argmin_{x \\in \\E}\\left[f(y) + \\innprod{\\nabla{f}(y)}{x-y} + \\frac{L}{2}\\norm{x-y}^2 + \\varPsi_\\sigma(x)\\right].\n\\end{equation*}\nBy the identity\n$m^\\sigma_{L+\\sigma}(y;x) = f(y) + \\innprod{\\nabla{f}(y)}{x-y} +\\frac{L}{2}\\norm{x-y}^2+ \\varPsi_\\sigma(x)$,\nobserve that $\\tilde{T}_L(y)=T^\\sigma_{L+\\sigma}(y)$ holds.\nTherefore, the accelerated method $\\mathcal{A}(x_0,L_0,\\sigma)$ given in \\cite[Algorithm~(4.9)]{Nes13} applied to the regularized function \\eqref{realloc} can be described as follows:\n{\\it Let $x_0 \\in \\E$, $\\psi_0(x)=\\frac{1}{2}\\norm{x-x_0}^2$, $A_0=0$, $L_0 \\geq L_{\\min}$.\nGenerate the sequence $\\{x_k,\\psi_k,M_k,L_k,A_k\\}$ by the iteration\n$$\\{x_{k+1},\\psi_{k+1},M_k,L_{k+1},A_{k+1}\\} \\leftarrow \\APGIter_\\sigma(x_k,\\psi_k,L_k,A_k),$$\nfor each $k\\geq 0$.}\nThe scheme $\\APGIter$ at each iteration is shown in Algorithm~\\ref{APG-iter}.\nNesterov's method involves the backtracking line-search procedure to adapt the Lipschitz constant $L_f$ with the estimate $M_k$ (and $L_{k+1}$), which is controlled by the parameters $\\gamma_{\\rm inc}>1$ and $\\gamma_{\\rm dec}\\geq 1$.\n","sha256_utf8":"f061a6192d29e117704b4c88777cf85e1ddceefa4b2174890668aa6e2b8e5e47"}]},{"id":"S006","semantic_type":"algorithm","kind":"algorithm","title":"加速近端梯度单步算法","source_units":[{"start":33617,"end":35638,"start_line":627,"end_line":659,"raw_tex":"\\begin{algorithm}[t]\n\\caption{\nAccelerated Proximal Gradient Iteration\n\\newline\n$\\{x_{k+1},\\psi_{k+1},M_k,L_{k+1},A_{k+1}\\} \\leftarrow \\APGIter_\\sigma(x_k,\\psi_k,L_k,A_k)$}\n\\label{APG-iter}\n\\textbf{Parameters:} $\\gamma_{\\rm inc}>1$, $\\gamma_{\\rm dec}\\geq 1$, $L_{\\min} \\in  ]0,L_f]$.\\\\\n\\textbf{Input:} $\\sigma>0$, $x_k \\in \\E$, $\\psi_k:\\E\\to\\RR\\cup\\{+\\infty\\}$, $L_k>0$, $A_k\\geq 0$.\n\\begin{algorithmic}[1]\n\\State Compute $v_{k}:=\\argmin_{x\\in \\E} \\psi_{k}(x)$\\quad (cf. \\eqref{APG-iter-v_k-formula}). \\label{APG-iter-vk}\n\\State Set $L:=L_k/\\gamma_{\\rm inc}$.\n\\Repeat \\label{APG-iter-loop-s}\n\t\\State Set $L:=L\\gamma_{\\rm inc}$.\n\t\\State Find the largest root $a>0$ of the equation $\\frac{a^2}{A_k+a} = 2\\frac{1+\\sigma A_k}{L}$. \\label{a-quad-eq}\n\t\\State Set $y = \\frac{A_k x_k +a v_k}{A_k + a}$.\n\t\\State Compute $z=T_{L+\\sigma}^\\sigma(y)=\\argmin_{x}[f(y)+\\innprod{\\nabla{f}(y)}{x-y}+\\frac{L}{2}\\norm{x-y}^2 + \\varPsi_\\sigma(x)]$.\n\t\\State Compute $T_L(z)$ and $T_{L+\\sigma}^\\sigma(z)$ \\label{APG-iter-add-prox}.\n\t\\State Test the conditions\n\t\\begin{subequations}\n\t\\label{APG-iter-test}\n\t\\begin{align}\n\t&\\innprod{\\nabla{f}(y)-\\nabla{f}(T_{L+\\sigma}^\\sigma(y))}{y-T_{L+\\sigma}^\\sigma(y)} \\geq \\frac{1}{L}\\norm{\\nabla{f}(y)-\\nabla{f}(T_{L+\\sigma}^\\sigma(y))}^2, \\label{APG-iter-test-a}\\\\\n\t&\\varphi_\\sigma(T_{L+\\sigma}^\\sigma(z)) \\leq m_{L+\\sigma}^\\sigma(z;T_{L+\\sigma}^\\sigma(z)), \\label{APG-iter-test-b}\\\\\n\t&\\text{[Optional]}~~\\varphi(T_L(z)) \\leq \\varphi(z) \\label{APG-iter-test-c}.\n\t\\end{align}\n\t\\end{subequations}\n\\Until{the conditions in \\eqref{APG-iter-test} hold.} \\label{APG-iter-loop-e}\n\\State Define $y_k:=y$, $M_k:=L$, $x_{k+1}:=z=T_{M_k+\\sigma}^\\sigma(y_k)$, \\\\\n\\quad $a_{k+1}:=a$, $A_{k+1}:=A_k+a_{k+1}$, $L_{k+1}:=\\max\\{L_{\\min},M_k/\\gamma_{\\rm dec}\\}$, \\label{AGP-iter-upadte}\n\\State \\quad $\\psi_{k+1}(x):=\\psi_k(x)+a_{k+1}[f(x_{k+1})+\\innprod{\\nabla{f}(x_{k+1})}{x-x_{k+1}}+\\varPsi_\\sigma(x)]$.\\label{est-seq-formula}\n\\State Output $\\{x_{k+1},\\psi_{k+1},M_k,L_{k+1},A_{k+1}\\}$.\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"aa3c0c56f7d9052bd4a4ee593fd9d61be9f56bd2de9f16f614ababf51ada97bf"}]},{"id":"S007","semantic_type":"derived_fact","kind":"derived_fact","title":"估计序列极小点计算式","source_units":[{"start":36246,"end":36457,"start_line":664,"end_line":666,"raw_tex":"\\begin{equation}\\label{APG-iter-v_k-formula}\nv_k = \\prox_{\\gamma_k\\varPsi}(w_k),\\text{ where } \\gamma_k = \\frac{A_k}{1+\\sigma A_k},~~w_k = x_0-\\frac{1}{1+\\sigma A_k}\\sum_{i=1}^ka_i \\nabla{f}(x_i).\n\\end{equation}","sha256_utf8":"78bd5c755d13354cca8d0e0a81d41c21a5f5a0d89da30d38f35b3612db5a774a"}]},{"id":"S008","semantic_type":"algorithm","kind":"algorithm","title":"带回溯的近端梯度单步","source_units":[{"start":44916,"end":45665,"start_line":820,"end_line":841,"raw_tex":"\\begin{algorithm}\n\\caption{\nProximal Gradient Iteration\n\\newline\n$\\{T_{M_k}(x_k), M_k, L_{k+1}\\} \\leftarrow \\PGIter(x_k,L_k)$\n}\n\\label{alg:PG-iter}\n\\textbf{Parameters:} $\\gamma_{\\rm inc}>1$, $\\gamma_{\\rm dec}\\geq 1$, $L_{\\min} \\in ]0,L_f]$.\\\\\n\\textbf{Input:} $x_k \\in \\E$, $L_k > 0$.\n\\begin{algorithmic}[1]\n\\State Set $L:=L_k/\\gamma_{\\rm inc}$.\n\\Repeat \\label{PG-iter-loop-s}\n\\State Set $L:=\\gamma_{\\rm inc}L$.\n\\State Compute $T_L(x_k)$.\n\\Until{the following condition holds:} \\label{PG-iter-loop-e}\n\\begin{equation}\\label{PG-iter-test}\n\t\\varphi(T_L(x_k))\\leq m_L(x_k;T_L(x_k)).\n\\end{equation}\n\\State Define $M_k:=L$, $L_{k+1}:=\\max\\{L_{\\min},M_k/\\gamma_{\\rm dec}\\}$.\n\\State Output $\\{T_{M_k}(x_k), M_k, L_{k+1}\\}$.\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"4248f51eb1fd62ec761a9ede0f6f8c3a633045c4cf6935c7b102bf5f3202d242"}]},{"id":"S009","semantic_type":"algorithm","kind":"algorithm","title":"自适应正则化加速算法","source_units":[{"start":47230,"end":48256,"start_line":885,"end_line":908,"raw_tex":"\\begin{algorithm}\n\\caption{\nAdaptive Accelerated Proximal Gradient Method\n\\newline\n$\\AdaAPG(x_0,L_{-1},\\sigma_0,\\ep)$\n}\n\\label{alg:AdaAPG}\n\\textbf{Parameters:} $\\gamma_{\\rm inc}>1$, $\\gamma_{\\rm dec}\\geq 1$, $L_{\\rm min}\\in ]0,L_f]$, $\\gamma_{\\rm reg}>1$, $\\beta\\in]0,1]$.\\\\\n\\textbf{Input:} $x_0 \\in \\E$, $L_{-1} \\in [L_{\\min}, \\gamma_{\\rm inc}L_f]$, $\\sigma_0>0$, $\\ep>0$.\n\\begin{algorithmic}[1]\n\\State $L_{0}:= L_{-1}$.\n\\For{$j=0,1,2,\\ldots$}\n\t\\State $\\sigma_j := \\sigma_0/\\gamma_{\\rm reg}^j$.\n\t\\State $\\psi_{0}(x):=\\frac{1}{2}\\norm{x-x_{0}}^2$, $A_{0}:=0$.\n\t\\Repeat{ \\textbf{for} $k=0,1,2,\\ldots$}\n\t\t\\State $\\{x_{k+1},\\psi_{k+1},M_{k},L_{k+1},A_{k+1}\\} \\leftarrow \\APGIter_{\\sigma_j}(x_{k},\\psi_{k},L_{k},A_{k})$.\n\t\t\\If{$\\norm{g_{M_{k}}(x_{k+1})} \\leq \\ep$}\n\t\t\t\\State output $\\{\\sigma_j, x_{k+1},T_{M_{k}}(x_{k+1}),M_{k},L_{k+1}\\}$ and terminate the algorithm.\n\t\t\\EndIf\n\t\\Until{$A_{k+1} \\geq \\frac{2(M_{k}+\\sigma_j)}{\\beta^2\\sigma_j^2}$}.\\label{A-growth}\n\\State $L_{0} := L_{k+1}$.\n\\EndFor\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"9d1d7a73cb0361ed18892909a804266378fec2baac06aaf88469017992ff93a5"}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","title":"初始水平集上的误差界假设","source_units":[{"start":60532,"end":60872,"start_line":1126,"end_line":1130,"raw_tex":"In this section, given an initial point $\\xinit \\in \\E$, we assume that the objective function $\\varphi$ admits the H\\\"olderian error bound condition\n\\begin{equation}\\label{HEB-review}\n\\varphi(x)-\\varphi^* \\geq \\kappa\\dist(x,X^*)^\\rho,\\quad \\forall x\\in \\lev_{\\varphi}(\\varphi(\\xinit)),\n\\end{equation}\nfor some $\\kappa>0$ and $\\rho\\geq 1$.\n","sha256_utf8":"f2113a9b9e096ea20f9ba0fdbbe1cb0301041684bab24be512d7551ac0e9211f"}]},{"id":"S011","semantic_type":"algorithm","kind":"algorithm","title":"自适应加速算法的重启方案","source_units":[{"start":61920,"end":63301,"start_line":1145,"end_line":1167,"raw_tex":"\\begin{algorithm}\n\\caption{\nRestart Scheme for Adaptive Accelerated Proximal Gradient Method\n\\newline\n$\\rAdaAPG (\\xinit,\\Linit,\\sigmainit,\\ep)$\n}\n\\label{alg:rAdaAPG}\n\\textbf{Parameters:}\\\\\n\\quad $\\gamma_{\\rm inc}>1$, $\\gamma_{\\rm dec}\\geq 1$, $L_{\\rm min}\\in ]0,L_f]$ (for backtracking line-search);\\\\\n\\quad $\\gamma_{\\rm reg}>1$ (the ratio to decrease the regularization parameter);\\\\\n\\quad $\\beta\\in ]0,1]$  (controls the timing when the current regularization parameter must be decreased);\\\\\n\\quad $\\theta \\in ]0,1[$ (the ratio reducing the residue per iteration).\\\\\n\\textbf{Input:} $\\xinit \\in \\E$, $\\Linit \\in [L_{\\min},\\gamma_{\\rm inc}L_f]$, $\\sigmainit>0$, $\\ep>0$.\n\\begin{algorithmic}[1]\n\\State Compute $\\{\\outerXplus{0},\\outerM{0},\\outerL{0}\\} \\leftarrow \\PGIter(\\xinit,\\Linit)$ \\label{rAdaAPG-init}.\n\\For{$t=0,1,2,\\ldots$}\n\t\\State \\textbf{if} $\\norm{g_{\\outerM{t}}(\\outerXinit{t})} \\leq \\ep$ \\textbf{then} output $\\{\\outerXinit{t},\\outerM{t},\\outerXplus{t}\\}$ and terminate the algorithm. \\label{rAdaAPG-terminate}\n\t\\State $\\outerep{t}:= \\theta\\norm{g_{\\outerM{t}}(\\outerXinit{t})}$.\n\t\\State $\\{\\outersigma{t+1},\\outerXinit{t+1},\\outerXplus{t+1},\\outerM{t+1},\\outerL{t+1}\\} \\leftarrow \\AdaAPG(\\outerXplus{t},\\outerL{t},\\outersigma{t},\\ep^{(t)})$\n\t\\State (with testing the optional condition \\eqref{APG-iter-test-c} of $\\APGIter$).\n\\EndFor\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"970e3a362fd188bccc83fab8d05409e4e2c625fe9c7d7baf3e98d882cda5606b"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"累计迭代复杂度函数","source_units":[{"start":63941,"end":64706,"start_line":1181,"end_line":1190,"raw_tex":"To describe the iteration complexity bounds, we define $N(\\ep,\\sigma_*,C)$ and $\\bar\\sigma$ as follows.\nGiven $\\ep,\\sigma_*,C>0$, let $N(\\ep,\\sigma_*,C):=0$ if $\\norm{g_{\\outerM{0}}(\\xinit)} \\leq \\ep$ and otherwise\n\\begin{align}\nN(\\ep,\\sigma_*,C) &\n:= \\left(1+ \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep} + \\log_{\\gamma_{\\rm reg}}\\frac{\\sigmainit}{\\sigma_*}\\right)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}\\right) \\nonumber\\\\\n&\\quad +\n\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\n\\sqrt{\\frac{1}{\\sigma_*}} - \\sqrt{\\frac{1}{\\sigmainit}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*} \n+ C\\sqrt{2\\gamma_{\\rm inc}L_f}\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}. \\label{N-def}\n\\end{align}","sha256_utf8":"596b21831c9037bf0a64d6de1cd1cd8ba3259808170bde713d07eab88801c59b"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"正则化参数统一阈值","source_units":[{"start":64727,"end":65199,"start_line":1192,"end_line":1206,"raw_tex":"\\begin{equation}\\label{bar-sigma-def}\n\\bar\\sigma = \\left\\{\n\\begin{array}{ll}\n\\ds\n\\frac{\\theta}{1+\\sqrt{2}\\beta}\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}}\n\\ep^{\\frac{\\rho-2}{\\rho-1}}\n& (\\text{if } \\rho \\geq 2),\\\\[3truemm]\n\\ds\n\\frac{\\theta}{1+\\sqrt{2}\\beta}\n\\kappa^{\\frac{2}{\\rho}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-1}(\\varphi(\\xinit)-\\varphi^*)^{-\\frac{2-\\rho}{\\rho}}\n& (\\text{if } 1\\leq \\rho<2).\n\\end{array}\n\\right.\n\\end{equation}","sha256_utf8":"adb6adc19541bcf96e321aedf1e42fbe6387adbc09f0f11dbbb1549d66f7a564"}]},{"id":"S014","semantic_type":"theorem","kind":"theorem","title":"误差界下自适应重启复杂度","source_units":[{"start":65481,"end":68252,"start_line":1209,"end_line":1288,"raw_tex":"\\begin{theorem}\\label{thm:rAdaAPG}\nAssume that the H\\\"olderian error bound condition \\eqref{HEB-review} holds.\nIn $\\rAdaAPG$, \nlet $N$ be the total number of executions of $\\APGIter$.\nDenote\n\\begin{equation}\\label{simga*-def}\n\\sigma_* := \\left\\{\n\\begin{array}{ll}\n\\sigmainit & (\\text{if } \\sigmainit \\leq \\bar\\sigma),\\\\\n\\bar\\sigma/\\gamma_{\\rm reg} & (\\text{otherwise}),\n\\end{array}\n\\right.\n\\end{equation}\nfor  $\\bar\\sigma$ defined in \\eqref{bar-sigma-def}.\nAlso, let  $N(\\cdot,\\cdot,\\cdot)$ be defined by    \\eqref{N-def}.\nThen, the following assertions hold.\n\\begin{enumerate}[(i)]\n\\item If $\\rho=2$, then $N\\leq N(\\ep,\\sigma_*,C)$ holds with\n$\nC =\n\\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep}\\right).\n$\n\n\\item Suppose $\\rho>2$.\nIf $\\sigmainit\\geq\\bar\\sigma$, then  $N\\leq N(\\ep,\\sigma_*,C)$ holds with\n\\begin{align*}\nC=&\n\\sqrt{\\frac{1}{\\sigmainit}}\\left(1+\\frac{\\rho-1}{\\rho-2}\\log_{1/\\theta}\\frac{\\sigma_*^{(0)}}{\\min(\\sigma_*^{(0)},\\sigmainit)}\\right)+\n\\frac{\\sqrt{\\gamma_{\\rm reg}}}{1-\\sqrt{\\theta}^{\\frac{\\rho-2}{\\rho-1}}}\n\\left(\n\\sqrt{\\frac{1}{\\bar\\sigma}}\n-\n\\sqrt{\\theta}^{\\frac{\\rho-2}{\\rho-1}}\n\\sqrt{\\frac{1}{\\sigmainit}}\n\\right),\n\\end{align*}\nwhere\n$\n\\sigma_*^{(0)}:=\\frac{\\theta}{1+\\sqrt{2}\\beta}\\cdot\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}}\n\\|g_{\\outerM{0}}(\\xinit)\\|^{\\frac{\\rho-2}{\\rho-1}}.\n$\nOtherwise, it follows $\\outersigma{t}=\\sigmainit$ for all $t\\geq 0$ and $N\\leq N(\\ep,\\sigmainit,C)$ holds with\n$\nC= \n\\sqrt{\\frac{1}{\\sigmainit}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep}\\right).\n$\n\\item Suppose $\\rho\\in ]1,2[$.\nThen, we have\n\\begin{equation}\\label{N-bd-superlinear}\nN \\leq 1+ N(\\max(\\ep,\\ep_*),\\sigma_*,C) + \\left(\\log\\frac{1}{\\rho-1}\\right)^{-1}\\left(\\log\\log \\frac{\\ep_*}{\\theta^{\\frac{\\rho-1}{2-\\rho}}\\min(\\ep,\\ep_*)}-\\log\\log\\frac{1}{\\theta^{\\frac{\\rho-1}{2-\\rho}}}\\right),\n\\end{equation}\nwhere\n$\nC =\n\\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\max(\\ep,\\ep_*)}  \\right)\n$ and\n\\begin{equation}\\label{rho-12-ep*-def}\n\\ep_*=\n\\left[(1+\\sqrt{2})\\theta^{-1}(\\gamma_{\\rm inc}L_f+\\sigmainit)\\right]^{-\\frac{\\rho-1}{2-\\rho}}\n{\\kappa^{\\frac{1}{2-\\rho}}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{2-\\rho}}.\n\\end{equation}\n%\n\\item\n\\sloppy % prevents overflow\nIf $\\rho=1$, then we have\n$\nN \\leq 1 + N(\\max(\\ep,\\ep_*),\\sigma_*,C)\n$,\nwhere\n$\n\\ep_* = \\kappa\\left(\\frac{L_f}{L_{\\min}} + 1\\right)^{-1}\n$ and\n$\n{C=\n\\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\max(\\ep,\\ep_*)}\\right)}\n$.\nMoreover, if $\\ep<\\ep_*$, then $\\outerXplus{t}$ in the output of the algorithm must be an optimal solution to the original problem \\eqref{main-prob}.\n\\end{enumerate}\n\\end{theorem}","sha256_utf8":"d6c2e33a94daab73ce7dbcc4ea81a9e41a237601490befb5ed461ac356414c23"}]},{"id":"S015","semantic_type":"proof","kind":"proof","title":"重启复杂度主定理完整证明","source_units":[{"start":90637,"end":99547,"start_line":1697,"end_line":1860,"raw_tex":"\\begin{proof}[\\underline{Proof of Theorem~\\ref{thm:rAdaAPG}}]\nWe may assume that $\\norm{g_{\\outerM{0}}(\\xinit)} > \\ep$ since $N=0$ on the other case.\nSuppose that $\\rAdaAPG$ terminated with the stopping criterion at the $(T+1)$-th outer loop for some $T\\geq 0$.\nDenote by $N^{(t)}$ the number of executions of $\\APGIter$ at the $t$-th outer loop so that $N=\\sum_{t=0}^T N^{(t)}$.\nLet $\\bar\\sigma$ be defined by \\eqref{bar-sigma-def}.\n\nBy the definition of $T$, we have\n\\begin{equation}\\label{pr:gradmap-lb-ep}\n\\norm{g_{\\outerM{t}}(\\outerXinit{t})} > \\ep,\\quad t=0,\\ldots,T.\n\\end{equation}\nMoreover, by the definition of $\\sigma_*$ in \\eqref{simga*-def},\nusing Lemma~\\ref{lem:rAdaAPG-sigma} (iii) implies\n$$\n\\outersigma{t} \\geq\n\\sigma_*,\\quad t=0,\\ldots,T+1.\n$$\nTherefore, applying Lemma~\\ref{lem:rAdaAPG-compl} with $T_*=T$, we obtain the assertion\n\\begin{equation}\\label{N-ub-basic-HEB}\nN \\leq N(\\ep,\\sigma_*,C) \\quad \\text{with}\\quad\nC=\\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep}\\right).\n\\end{equation}\nFor the case $\\rho=2$, \\eqref{N-ub-basic-HEB} proves the assertion (i).\nWe discuss the other cases to improve \\eqref{N-ub-basic-HEB}.\n\n\\fbox{(ii) Case $\\rho>2$.}\nIf $\\sigmainit < \\bar\\sigma$, then $\\sigma_*$ is defined as $\\sigma_*=\\sigmainit$. \nTherefore, the latter bound of (ii) is obtained by \\eqref{N-ub-basic-HEB}.\n\nNow consider the case $\\sigmainit \\geq \\bar\\sigma$.\nThen, there exists $t_0 \\in \\{0,\\ldots,T+1\\}$ satisfying the conditions in Lemma~\\ref{lem:rAdaAPG-sigma-2}.\nBy \\eqref{sigma-bound-rho>2-4}, remark that\n\\begin{equation}\\label{rAdaAPG-compl-p>2-pf-sum}\n\\sum_{t=t_0}^{T}\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}(T-t)}}\n=\n\\sum_{i=0}^{T-t_0}\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}}^{\\,i}\n=\n\\frac{1-\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}}^{T-t_0+1}}{1-\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}}}\n\\leq\n\\frac{1-\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}\\cdot\\dfrac{\\bar\\sigma}{\\outersigma{0}}}}{1-\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}}}.\n\\end{equation}\nTherefore, we conclude that\n\\begin{align*}\n\\sum_{t=0}^{T}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n&=\n\\sum_{t=0}^{t_0-1} \\sqrt{\\frac{1}{\\outersigma{t+1}}}+\n\\sum_{t=t_0}^{T}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n\\\\\n&\\leq\nt_0 \\sqrt{\\frac{1}{\\outersigma{0}}} +\n\\sqrt{\\frac{\\gamma_{\\rm reg}}{\\bar\\sigma}}\\sum_{t=t_0}^{T}\\sqrt{\\theta^{{\\frac{\\rho-2}{\\rho-1}}(T-t)}} & (\\text{by } \\eqref{sigma-bound-rho>2-1} \\text{ and } \\eqref{sigma-bound-rho>2-2})\\\\\n&\\leq\n\\sqrt{\\frac{1}{\\outersigma{0}}}\\left(1+\\frac{\\rho-1}{\\rho-2}\\log_{1/\\theta}\\frac{\\sigma_*^{(0)}}{\\min(\\sigma_*^{(0)},\\outersigma{0})}\\right)\\\\\n&\\quad~+\n\\frac{\\sqrt{\\gamma_{\\rm reg}}}{1-\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}}}\n\\left(\n\\sqrt{\\frac{1}{\\bar\\sigma}}\n-\n\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}\n}}\n\\sqrt{\\frac{1}{\\outersigma{0}}}\n\\right) & (\\text{by } \\eqref{sigma-bound-rho>2-3} \\text{ and } \\eqref{rAdaAPG-compl-p>2-pf-sum})\\\\\n&=:C.\n\\end{align*}\nWith this definition of $C$, Lemma~\\ref{lem:rAdaAPG-compl} gives $N \\leq N(\\ep,\\sigma_*,C)$.\n\n\n\\fbox{(iii) Case $\\rho \\in ]1,2[$.}\nIf $\\ep \\geq \\ep_*$, then \\eqref{N-ub-basic-HEB} gives our assertion because the second term of \\eqref{N-bd-superlinear} vanishes.\nSuppose, on the other hand, that $\\ep < \\ep_*$. \nDenote\n$\\xi_{t}:=\\norm{g_{\\outerM{t}}(\\outerXinit{t})}$\nand \nlet $T_*\\geq 0$ be the smallest integer such that $\\xi_{T_*} \\leq \\ep_*$.\nThen, since $\\xi_{T_*-1} > \\ep_*$ holds, Lemma~\\ref{lem:rAdaAPG-compl} shows that\n$$\n\\sum_{t=0}^{T_*-1} N^{(t)} \\leq N(\\ep_*,\\sigma_*,C)\\quad \\text{with}\\quad C=\\sqrt{\\frac{1}{\\sigma_*}}\\left( 1 + \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*} \\right).\n$$\nNote that, under the convention $\\sum_{t=0}^{-1}(\\cdot)=0$, this inequality also holds if $T_*=0$ since then $\\xi_0\\leq \\ep_*$ and $N(\\ep_*,\\sigma_*,C)=0$.\n\nIt remains to observe $\\sum_{t=T_*}^T N^{(t)}$.\nTake $t\\in \\{T_*,\\ldots,T\\}$. We shall prove $N^{(t)}=1$.\nIn the $t$-th outer loop, consider the first iteration of the subroutine $\\AdaAPG(\\outerXplus{t},\\outerL{t},\\outersigma{t},\\ep^{(t)})$, which executes\n$$\n\\{x_1,\\psi_1,M_0,L_1,A_1\\} \\leftarrow \\APGIter_{\\outersigma{t}}(\\outerXplus{t},\\psi_0,\\outerL{t},A_0),\n$$\nwhere $\\psi_0(x)=\\frac{1}{2}\\|x-\\outerXplus{t}\\|^2$ and $A_0=0$.\nThen, Proposition~\\ref{prop:APG} (v) implies\n$$\n\\norm{g_{M_{0}}(x_1)} \\leq \\left(2\\sqrt{\\frac{M_{0}+\\outersigma{t}}{A_1}} + \\outersigma{t}\\right)\\dist(\\outerXplus{t},X^*).\n$$\nMoreover, according to the equation at Line~\\ref{a-quad-eq} in Algorithm~\\ref{APG-iter}, $A_1$ can be calculated as $A_1=2/M_0$. Now remark that, using $M_0 \\leq \\gamma_{\\rm inc}L_f$ (Proposition~\\ref{prop:APG}~(i)), we have\n\\begin{align*}\n2\\sqrt{\\frac{M_{0}+\\outersigma{t}}{A_1}} + \\outersigma{t} &= 2\\sqrt{\\frac{M_0(M_0+\\outersigma{t})}{2}} + \\outersigma{t}\n\\leq 2\\sqrt{\\frac{(M_0+\\outersigma{t})^2}{2}} + (M_0+\\outersigma{t}) \\\\&= (1+\\sqrt{2})(M_0+\\outersigma{t}) \\leq (1+\\sqrt{2})(\\gamma_{\\rm inc}L_f + \\outersigma{t}).\n\\end{align*}\nCombining them and using \\eqref{rAdaAPG-dist-bound}\\footnote{Although \\eqref{rAdaAPG-dist-bound} is asserted in the case $\\outerXplus{t} \\not\\in X^*$, it trivially holds if $\\outerXplus{t} \\in X^*$ \\emph{unless} $\\rho = 1$.}, we conclude that\n\\begin{align}\n\\norm{g_{M_{0}}(x_1)}\n&\\leq\n(1+\\sqrt{2})(\\gamma_{\\rm inc}L_f+\\outersigma{0})\\dist(\\outerXplus{t},X^*) \\nonumber\\\\\n&\\leq\n(1+\\sqrt{2})(\\gamma_{\\rm inc}L_f+\\outersigma{0})\n\\left(\\frac{1}{\\kappa}\\right)^{\\frac{1}{\\rho-1}} \\left(\\frac{L_f}{L_{\\min}}+1\\right)^{\\frac{1}{\\rho-1}}\\xi_t^{\\frac{1}{\\rho-1}}\n\\nonumber\\\\\n&= \\theta\\ep_*^{\\frac{\\rho-2}{\\rho-1}}\\xi_t^{\\frac{1}{\\rho-1}}\n\\label{superlinear-rec}\\\\\n&\\leq \\theta\\xi_t^{\\frac{\\rho-2}{\\rho-1}}\\xi_t^{\\frac{1}{\\rho-1}}=\\theta\\xi_t = \\ep^{(t)},\n\\nonumber\n\\end{align}\nwhere the last inequality is due to $\\xi_t \\leq \\ep_*$ for $t \\geq T_*$ (and remark $\\frac{\\rho-2}{\\rho-1} < 0$).\nThis shows $N^{(t)}=1$ and \\eqref{superlinear-rec} yields the recurrence\n$\n\\xi_{t+1} \\leq \\theta\\ep_*^{\\frac{\\rho-2}{\\rho-1}}\\xi_t^{\\frac{1}{\\rho-1}}\n$,\nfor each $t=T_*,\\ldots,T$.\nSince $\\frac{1}{\\rho-1} > 1$, it reduces $\\xi_t$ superlinearly. In particular, solving this recurrence implies\n$$\n\\log \\frac{\\ep_*}{\\theta^{\\frac{\\rho-1}{2-\\rho}}\\xi_T} \\geq \\left(\\frac{1}{\\rho-1}\\right)^{T-T_*}\\log \\frac{\\ep_*}{\\theta^{\\frac{\\rho-1}{2-\\rho}}\\xi_{T_*}}.\n$$\nSince $\\xi_T > \\ep$ and $\\xi_{T_*}\\leq \\ep_*$, we obtain\n$$\n\\sum_{t=T_*}^T N^{(t)} =\nT-T_*+1 \\leq  1+ \\left(\\log\\frac{1}{\\rho-1}\\right)^{-1}\\left(\\log\\log \\frac{\\ep_*}{\\theta^{\\frac{\\rho-1}{2-\\rho}}\\ep}-\\log\\log\\frac{1}{\\theta^{\\frac{\\rho-1}{2-\\rho}}}\\right).\n$$\nConsequently, $N$ is bounded as follows.\n$$\nN = \\sum_{t=0}^{T_*-1} N^{(t)} + \\sum_{t=T_*}^T N^{(t)} \\leq N(\\ep_*,\\sigma_*,C) + 1+\\left(\\log\\frac{1}{\\rho-1}\\right)^{-1}\\left(\\log\\log \\frac{\\ep_*}{\\theta^{\\frac{\\rho-1}{2-\\rho}}\\ep}-\\log\\log\\frac{1}{\\theta^{\\frac{\\rho-1}{2-\\rho}}}\\right).\n$$\n\n\\fbox{(iv) Case $\\rho=1$.}\nWe have\n$\n\\outersigma{t} \\geq\n\\sigma_*\n$ for each $t=0,\\ldots,T+1$, by Lemma~\\ref{lem:rAdaAPG-sigma} (iii).\nMoreover, Lemma~\\ref{lem:rAdaAPG-base}~(vi) shows that, if $\\outerXplus{t} \\not\\in X^*$, then we have\n$$\n1 \\leq \\frac{1}{\\kappa}\\left(\\frac{L_f}{L_{\\min}} + 1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})},\n~~\\text{i.e.,}~~\n\\norm{g_{\\outerM{t}}(\\outerXinit{t})} \\geq \\ep_*:=\\kappa\\left(\\frac{L_f}{L_{\\min}} + 1\\right)^{-1}.\n$$\nIn other words, the condition $\\norm{g_{\\outerM{t}}(\\outerXinit{t})} < \\ep_*$ must imply $\\outerXplus{t}\\in X^*$, which also yields $N^{(t)}=1$ and $\\outerXinit{t+1}=\\outerXplus{t} \\in X^*$ by Proposition~\\ref{prop:APG}~(v) (then the algorithm terminates at the ($t+1$)-th outer loop).\nTherefore, we have\n\\begin{equation}\\label{rho=1:gmap-ep*}\n\\norm{g_{\\outerM{t}}(\\outerXinit{t})} \\geq \\ep_*,\\quad 0\\leq t \\leq T-1.\n\\end{equation}\n\\sloppy\nNow we consider two cases. If $\\norm{g_{\\outerM{T}}(\\outerXinit{T})} \\geq \\ep_*$ holds, then\ncombining with \\eqref{pr:gradmap-lb-ep} yields ${\\norm{g_{\\outerM{T}}(\\outerXinit{T})} \\geq \\max(\\ep,\\ep_*)}$, from which Lemma~\\ref{lem:rAdaAPG-compl} with $T_*=T$ concludes\n$$\nN \\leq N(\\max(\\ep,\\ep_*),\\sigma_*,C)\n\\quad\\text{with}\\quad\nC=\n\\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\max(\\ep,\\ep_*)}\\right).\n$$\nOn the other case $\\norm{g_{\\outerM{T}}(\\outerXinit{T})} < \\ep_*$, we have $N^{(T)}=1$. Since \\eqref{pr:gradmap-lb-ep} and \\eqref{rho=1:gmap-ep*} implies ${\\norm{g_{\\outerM{T-1}}(\\outerXinit{T-1})} \\geq \\max(\\ep,\\ep_*)}$, Lemma~\\ref{lem:rAdaAPG-compl} with $T_*=T-1$ shows $\\sum_{t=0}^{T-1}N^{(t)}\\leq N(\\max(\\ep,\\ep_*),\\sigma_*,C)$. Hence,\n$$\nN = N^{(T)}+\\sum_{t=0}^{T-1}N^{(t)} \\leq 1 + N(\\max(\\ep,\\ep_*),\\sigma_*,C).\n$$\nThis proves the desired bound on $N$.\nTo show the latter assertion of (iv), suppose $\\ep<\\ep_*$. Then, the output $\\{\\outerXinit{T+1},\\outerM{T+1},\\outerXplus{T+1}\\}$ satisfies $\\norm{g_{\\outerM{T+1}}(\\outerXinit{T+1})} \\leq \\ep < \\ep_*$. Therefore, $\\outerXplus{T+1}$ must be optimal.\n\nThe proof of Theorem~\\ref{thm:rAdaAPG} is completed.\n\\end{proof}","sha256_utf8":"af4e9b0e0f501dea26471b395705fa1d256c907508a0fbd61fe0d028b6fb972e"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"排除即时终止并分解迭代次数","source_units":[{"start":90637,"end":91072,"start_line":1697,"end_line":1702,"raw_tex":"\\begin{proof}[\\underline{Proof of Theorem~\\ref{thm:rAdaAPG}}]\nWe may assume that $\\norm{g_{\\outerM{0}}(\\xinit)} > \\ep$ since $N=0$ on the other case.\nSuppose that $\\rAdaAPG$ terminated with the stopping criterion at the $(T+1)$-th outer loop for some $T\\geq 0$.\nDenote by $N^{(t)}$ the number of executions of $\\APGIter$ at the $t$-th outer loop so that $N=\\sum_{t=0}^T N^{(t)}$.\nLet $\\bar\\sigma$ be defined by \\eqref{bar-sigma-def}.\n\n","sha256_utf8":"6b0cfe56ebec54f766c12e0d22d2fb373f7b4f5bd17e78efbfee604c5ea16a4c"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"统一参数下界与二次误差界复杂度","source_units":[{"start":91072,"end":91823,"start_line":1703,"end_line":1720,"raw_tex":"By the definition of $T$, we have\n\\begin{equation}\\label{pr:gradmap-lb-ep}\n\\norm{g_{\\outerM{t}}(\\outerXinit{t})} > \\ep,\\quad t=0,\\ldots,T.\n\\end{equation}\nMoreover, by the definition of $\\sigma_*$ in \\eqref{simga*-def},\nusing Lemma~\\ref{lem:rAdaAPG-sigma} (iii) implies\n$$\n\\outersigma{t} \\geq\n\\sigma_*,\\quad t=0,\\ldots,T+1.\n$$\nTherefore, applying Lemma~\\ref{lem:rAdaAPG-compl} with $T_*=T$, we obtain the assertion\n\\begin{equation}\\label{N-ub-basic-HEB}\nN \\leq N(\\ep,\\sigma_*,C) \\quad \\text{with}\\quad\nC=\\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep}\\right).\n\\end{equation}\nFor the case $\\rho=2$, \\eqref{N-ub-basic-HEB} proves the assertion (i).\nWe discuss the other cases to improve \\eqref{N-ub-basic-HEB}.\n\n","sha256_utf8":"6c953fbd5050847340b18c6f51a0c270b17880d757da37db12b81610de573cf7"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"高阶误差界的小初始参数情形","source_units":[{"start":91823,"end":92010,"start_line":1721,"end_line":1724,"raw_tex":"\\fbox{(ii) Case $\\rho>2$.}\nIf $\\sigmainit < \\bar\\sigma$, then $\\sigma_*$ is defined as $\\sigma_*=\\sigmainit$. \nTherefore, the latter bound of (ii) is obtained by \\eqref{N-ub-basic-HEB}.\n\n","sha256_utf8":"25200ee5e0bb799bdb6473b0bdc64d52762636a28d675a182435cd65b6a01984"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"高阶误差界的几何级数估计","source_units":[{"start":92010,"end":92634,"start_line":1725,"end_line":1736,"raw_tex":"Now consider the case $\\sigmainit \\geq \\bar\\sigma$.\nThen, there exists $t_0 \\in \\{0,\\ldots,T+1\\}$ satisfying the conditions in Lemma~\\ref{lem:rAdaAPG-sigma-2}.\nBy \\eqref{sigma-bound-rho>2-4}, remark that\n\\begin{equation}\\label{rAdaAPG-compl-p>2-pf-sum}\n\\sum_{t=t_0}^{T}\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}(T-t)}}\n=\n\\sum_{i=0}^{T-t_0}\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}}^{\\,i}\n=\n\\frac{1-\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}}^{T-t_0+1}}{1-\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}}}\n\\leq\n\\frac{1-\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}\\cdot\\dfrac{\\bar\\sigma}{\\outersigma{0}}}}{1-\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}}}.\n\\end{equation}\n","sha256_utf8":"8234bad0e95f540d1f596c37e62c5079dc9e48972e29e39b23b747c261e22dc9"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"精化倒平方根参数和","source_units":[{"start":92634,"end":93619,"start_line":1737,"end_line":1762,"raw_tex":"Therefore, we conclude that\n\\begin{align*}\n\\sum_{t=0}^{T}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n&=\n\\sum_{t=0}^{t_0-1} \\sqrt{\\frac{1}{\\outersigma{t+1}}}+\n\\sum_{t=t_0}^{T}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n\\\\\n&\\leq\nt_0 \\sqrt{\\frac{1}{\\outersigma{0}}} +\n\\sqrt{\\frac{\\gamma_{\\rm reg}}{\\bar\\sigma}}\\sum_{t=t_0}^{T}\\sqrt{\\theta^{{\\frac{\\rho-2}{\\rho-1}}(T-t)}} & (\\text{by } \\eqref{sigma-bound-rho>2-1} \\text{ and } \\eqref{sigma-bound-rho>2-2})\\\\\n&\\leq\n\\sqrt{\\frac{1}{\\outersigma{0}}}\\left(1+\\frac{\\rho-1}{\\rho-2}\\log_{1/\\theta}\\frac{\\sigma_*^{(0)}}{\\min(\\sigma_*^{(0)},\\outersigma{0})}\\right)\\\\\n&\\quad~+\n\\frac{\\sqrt{\\gamma_{\\rm reg}}}{1-\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}}}}\n\\left(\n\\sqrt{\\frac{1}{\\bar\\sigma}}\n-\n\\sqrt{\\theta^{\\frac{\\rho-2}{\\rho-1}\n}}\n\\sqrt{\\frac{1}{\\outersigma{0}}}\n\\right) & (\\text{by } \\eqref{sigma-bound-rho>2-3} \\text{ and } \\eqref{rAdaAPG-compl-p>2-pf-sum})\\\\\n&=:C.\n\\end{align*}\nWith this definition of $C$, Lemma~\\ref{lem:rAdaAPG-compl} gives $N \\leq N(\\ep,\\sigma_*,C)$.\n\n\n","sha256_utf8":"ad26e8bf10a0c58a4ebfe42c6625bc45e96b204ebc4a26f196b36bc5aaaf579f"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"超线性阶段前的累计复杂度","source_units":[{"start":93619,"end":94395,"start_line":1763,"end_line":1775,"raw_tex":"\\fbox{(iii) Case $\\rho \\in ]1,2[$.}\nIf $\\ep \\geq \\ep_*$, then \\eqref{N-ub-basic-HEB} gives our assertion because the second term of \\eqref{N-bd-superlinear} vanishes.\nSuppose, on the other hand, that $\\ep < \\ep_*$. \nDenote\n$\\xi_{t}:=\\norm{g_{\\outerM{t}}(\\outerXinit{t})}$\nand \nlet $T_*\\geq 0$ be the smallest integer such that $\\xi_{T_*} \\leq \\ep_*$.\nThen, since $\\xi_{T_*-1} > \\ep_*$ holds, Lemma~\\ref{lem:rAdaAPG-compl} shows that\n$$\n\\sum_{t=0}^{T_*-1} N^{(t)} \\leq N(\\ep_*,\\sigma_*,C)\\quad \\text{with}\\quad C=\\sqrt{\\frac{1}{\\sigma_*}}\\left( 1 + \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*} \\right).\n$$\nNote that, under the convention $\\sum_{t=0}^{-1}(\\cdot)=0$, this inequality also holds if $T_*=0$ since then $\\xi_0\\leq \\ep_*$ and $N(\\ep_*,\\sigma_*,C)=0$.\n\n","sha256_utf8":"37cd09351844de5509555d1ad45f64ba683911718ae11c346ecb63e4453fbb07"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"构造阈值后的首个内迭代","source_units":[{"start":94395,"end":94826,"start_line":1776,"end_line":1782,"raw_tex":"It remains to observe $\\sum_{t=T_*}^T N^{(t)}$.\nTake $t\\in \\{T_*,\\ldots,T\\}$. We shall prove $N^{(t)}=1$.\nIn the $t$-th outer loop, consider the first iteration of the subroutine $\\AdaAPG(\\outerXplus{t},\\outerL{t},\\outersigma{t},\\ep^{(t)})$, which executes\n$$\n\\{x_1,\\psi_1,M_0,L_1,A_1\\} \\leftarrow \\APGIter_{\\outersigma{t}}(\\outerXplus{t},\\psi_0,\\outerL{t},A_0),\n$$\nwhere $\\psi_0(x)=\\frac{1}{2}\\|x-\\outerXplus{t}\\|^2$ and $A_0=0$.\n","sha256_utf8":"bda9f8056d6502875b17a10aa59ab697582a4bd0f1c2aa7740e317ea41a3d43b"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"控制首步梯度映射系数","source_units":[{"start":94826,"end":95538,"start_line":1783,"end_line":1791,"raw_tex":"Then, Proposition~\\ref{prop:APG} (v) implies\n$$\n\\norm{g_{M_{0}}(x_1)} \\leq \\left(2\\sqrt{\\frac{M_{0}+\\outersigma{t}}{A_1}} + \\outersigma{t}\\right)\\dist(\\outerXplus{t},X^*).\n$$\nMoreover, according to the equation at Line~\\ref{a-quad-eq} in Algorithm~\\ref{APG-iter}, $A_1$ can be calculated as $A_1=2/M_0$. Now remark that, using $M_0 \\leq \\gamma_{\\rm inc}L_f$ (Proposition~\\ref{prop:APG}~(i)), we have\n\\begin{align*}\n2\\sqrt{\\frac{M_{0}+\\outersigma{t}}{A_1}} + \\outersigma{t} &= 2\\sqrt{\\frac{M_0(M_0+\\outersigma{t})}{2}} + \\outersigma{t}\n\\leq 2\\sqrt{\\frac{(M_0+\\outersigma{t})^2}{2}} + (M_0+\\outersigma{t}) \\\\&= (1+\\sqrt{2})(M_0+\\outersigma{t}) \\leq (1+\\sqrt{2})(\\gamma_{\\rm inc}L_f + \\outersigma{t}).\n\\end{align*}\n","sha256_utf8":"ce0a85da37310669b106f5f66c312c82d8859ace9b1082b2595f1178d592197a"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"误差界导出单步超线性递推","source_units":[{"start":95538,"end":96420,"start_line":1792,"end_line":1806,"raw_tex":"Combining them and using \\eqref{rAdaAPG-dist-bound}\\footnote{Although \\eqref{rAdaAPG-dist-bound} is asserted in the case $\\outerXplus{t} \\not\\in X^*$, it trivially holds if $\\outerXplus{t} \\in X^*$ \\emph{unless} $\\rho = 1$.}, we conclude that\n\\begin{align}\n\\norm{g_{M_{0}}(x_1)}\n&\\leq\n(1+\\sqrt{2})(\\gamma_{\\rm inc}L_f+\\outersigma{0})\\dist(\\outerXplus{t},X^*) \\nonumber\\\\\n&\\leq\n(1+\\sqrt{2})(\\gamma_{\\rm inc}L_f+\\outersigma{0})\n\\left(\\frac{1}{\\kappa}\\right)^{\\frac{1}{\\rho-1}} \\left(\\frac{L_f}{L_{\\min}}+1\\right)^{\\frac{1}{\\rho-1}}\\xi_t^{\\frac{1}{\\rho-1}}\n\\nonumber\\\\\n&= \\theta\\ep_*^{\\frac{\\rho-2}{\\rho-1}}\\xi_t^{\\frac{1}{\\rho-1}}\n\\label{superlinear-rec}\\\\\n&\\leq \\theta\\xi_t^{\\frac{\\rho-2}{\\rho-1}}\\xi_t^{\\frac{1}{\\rho-1}}=\\theta\\xi_t = \\ep^{(t)},\n\\nonumber\n\\end{align}\nwhere the last inequality is due to $\\xi_t \\leq \\ep_*$ for $t \\geq T_*$ (and remark $\\frac{\\rho-2}{\\rho-1} < 0$).\n","sha256_utf8":"543e18818fc9f5692ec0155731c289f9f9b8d4e837d0b72d64ecf5b8aad00675"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"求解超线性残差递推","source_units":[{"start":96420,"end":96874,"start_line":1807,"end_line":1815,"raw_tex":"This shows $N^{(t)}=1$ and \\eqref{superlinear-rec} yields the recurrence\n$\n\\xi_{t+1} \\leq \\theta\\ep_*^{\\frac{\\rho-2}{\\rho-1}}\\xi_t^{\\frac{1}{\\rho-1}}\n$,\nfor each $t=T_*,\\ldots,T$.\nSince $\\frac{1}{\\rho-1} > 1$, it reduces $\\xi_t$ superlinearly. In particular, solving this recurrence implies\n$$\n\\log \\frac{\\ep_*}{\\theta^{\\frac{\\rho-1}{2-\\rho}}\\xi_T} \\geq \\left(\\frac{1}{\\rho-1}\\right)^{T-T_*}\\log \\frac{\\ep_*}{\\theta^{\\frac{\\rho-1}{2-\\rho}}\\xi_{T_*}}.\n$$\n","sha256_utf8":"e0e2ff10e2cb2f42ebdb38d426768fe50fbb7609fdd0b20937abf83d3ee78435"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"合并得到双对数复杂度","source_units":[{"start":96874,"end":97428,"start_line":1816,"end_line":1825,"raw_tex":"Since $\\xi_T > \\ep$ and $\\xi_{T_*}\\leq \\ep_*$, we obtain\n$$\n\\sum_{t=T_*}^T N^{(t)} =\nT-T_*+1 \\leq  1+ \\left(\\log\\frac{1}{\\rho-1}\\right)^{-1}\\left(\\log\\log \\frac{\\ep_*}{\\theta^{\\frac{\\rho-1}{2-\\rho}}\\ep}-\\log\\log\\frac{1}{\\theta^{\\frac{\\rho-1}{2-\\rho}}}\\right).\n$$\nConsequently, $N$ is bounded as follows.\n$$\nN = \\sum_{t=0}^{T_*-1} N^{(t)} + \\sum_{t=T_*}^T N^{(t)} \\leq N(\\ep_*,\\sigma_*,C) + 1+\\left(\\log\\frac{1}{\\rho-1}\\right)^{-1}\\left(\\log\\log \\frac{\\ep_*}{\\theta^{\\frac{\\rho-1}{2-\\rho}}\\ep}-\\log\\log\\frac{1}{\\theta^{\\frac{\\rho-1}{2-\\rho}}}\\right).\n$$\n\n","sha256_utf8":"9e45a60e9dc26c49f0e143eae500d818d46bb8bc4ae775d5ba70aa1c6b34a9b9"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"尖锐误差界给出非最优残差阈值","source_units":[{"start":97428,"end":97887,"start_line":1826,"end_line":1837,"raw_tex":"\\fbox{(iv) Case $\\rho=1$.}\nWe have\n$\n\\outersigma{t} \\geq\n\\sigma_*\n$ for each $t=0,\\ldots,T+1$, by Lemma~\\ref{lem:rAdaAPG-sigma} (iii).\nMoreover, Lemma~\\ref{lem:rAdaAPG-base}~(vi) shows that, if $\\outerXplus{t} \\not\\in X^*$, then we have\n$$\n1 \\leq \\frac{1}{\\kappa}\\left(\\frac{L_f}{L_{\\min}} + 1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})},\n~~\\text{i.e.,}~~\n\\norm{g_{\\outerM{t}}(\\outerXinit{t})} \\geq \\ep_*:=\\kappa\\left(\\frac{L_f}{L_{\\min}} + 1\\right)^{-1}.\n$$\n","sha256_utf8":"6da50d24774f682c211adb899bdad8c5e7675a65b4ea7d8dff93b1ec57af05d5"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"低于阈值后一步达到最优","source_units":[{"start":97887,"end":98319,"start_line":1838,"end_line":1842,"raw_tex":"In other words, the condition $\\norm{g_{\\outerM{t}}(\\outerXinit{t})} < \\ep_*$ must imply $\\outerXplus{t}\\in X^*$, which also yields $N^{(t)}=1$ and $\\outerXinit{t+1}=\\outerXplus{t} \\in X^*$ by Proposition~\\ref{prop:APG}~(v) (then the algorithm terminates at the ($t+1$)-th outer loop).\nTherefore, we have\n\\begin{equation}\\label{rho=1:gmap-ep*}\n\\norm{g_{\\outerM{t}}(\\outerXinit{t})} \\geq \\ep_*,\\quad 0\\leq t \\leq T-1.\n\\end{equation}\n","sha256_utf8":"ba9e1bbdff042b6a9da33f800c7577e6a798bd16a6d49396ea7239855d9f7e78"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"末次残差较大时的复杂度","source_units":[{"start":98319,"end":98773,"start_line":1843,"end_line":1851,"raw_tex":"\\sloppy\nNow we consider two cases. If $\\norm{g_{\\outerM{T}}(\\outerXinit{T})} \\geq \\ep_*$ holds, then\ncombining with \\eqref{pr:gradmap-lb-ep} yields ${\\norm{g_{\\outerM{T}}(\\outerXinit{T})} \\geq \\max(\\ep,\\ep_*)}$, from which Lemma~\\ref{lem:rAdaAPG-compl} with $T_*=T$ concludes\n$$\nN \\leq N(\\max(\\ep,\\ep_*),\\sigma_*,C)\n\\quad\\text{with}\\quad\nC=\n\\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\max(\\ep,\\ep_*)}\\right).\n$$\n","sha256_utf8":"9d98880fa25e15b7be35e5ffc43da7150c14654fac1863abe821c3411226c3f9"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"末次残差较小时的有限步界","source_units":[{"start":98773,"end":99234,"start_line":1852,"end_line":1856,"raw_tex":"On the other case $\\norm{g_{\\outerM{T}}(\\outerXinit{T})} < \\ep_*$, we have $N^{(T)}=1$. Since \\eqref{pr:gradmap-lb-ep} and \\eqref{rho=1:gmap-ep*} implies ${\\norm{g_{\\outerM{T-1}}(\\outerXinit{T-1})} \\geq \\max(\\ep,\\ep_*)}$, Lemma~\\ref{lem:rAdaAPG-compl} with $T_*=T-1$ shows $\\sum_{t=0}^{T-1}N^{(t)}\\leq N(\\max(\\ep,\\ep_*),\\sigma_*,C)$. Hence,\n$$\nN = N^{(T)}+\\sum_{t=0}^{T-1}N^{(t)} \\leq 1 + N(\\max(\\ep,\\ep_*),\\sigma_*,C).\n$$\nThis proves the desired bound on $N$.\n","sha256_utf8":"f3202aeaa772e09c4519f95f967ed1aeb46d7dee9c54b9b82d19e45b8b047615"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"验证输出近端点的精确最优性","source_units":[{"start":99234,"end":99547,"start_line":1857,"end_line":1860,"raw_tex":"To show the latter assertion of (iv), suppose $\\ep<\\ep_*$. Then, the output $\\{\\outerXinit{T+1},\\outerM{T+1},\\outerXplus{T+1}\\}$ satisfies $\\norm{g_{\\outerM{T+1}}(\\outerXinit{T+1})} \\leq \\ep < \\ep_*$. Therefore, $\\outerXplus{T+1}$ must be optimal.\n\nThe proof of Theorem~\\ref{thm:rAdaAPG} is completed.\n\\end{proof}","sha256_utf8":"9f100f080201d207ca6815031e6dc3e6abbe040b967d3ffd1ba5b36864fdf5a7"}]},{"id":"S032","semantic_type":"lemma","kind":"lemma","title":"梯度映射基本性质","source_units":[{"start":15903,"end":16933,"start_line":305,"end_line":327,"raw_tex":"\\begin{lemma}\\label{lem:gmap}\nLet $\\varphi=f+\\varPsi$ be the sum of a continuously differentiable convex function $f:\\E\\to\\RR$ and a proper lower-semicontinuous convex function $\\varPsi:\\E\\to\\RR\\cup\\{+\\infty\\}$.\nThen, the following assertions hold.\n\\begin{enumerate}[(i)]\n\\item For all $y \\in \\E$, the map $L \\mapsto \\norm{g_L(y)}$ is increasing.\n%\n\\item $x^* \\in X^*$ holds if and only if $g_L(x^*)=0$.\n%\n\\item $\\nabla{f}(T_L(y))-\\nabla{f}(y)+g_L(y) \\in \\partial\\varphi(T_L(y))$ for all $y \\in \\E$. Moreover, if $f$ is $L_f$-smooth, then\n$$\n\\norm{\\nabla{f}(T_L(y))-\\nabla{f}(y)+g_L(y)} \\leq \\left(\\frac{L_f}{L}+1\\right)\\norm{g_L(y)}.\n$$\n%\n\\item If $y\\in \\E$ and $L>0$ satisfy $\\varphi(T_L(y))\\leq m_L(y;T_L(y))$, then we have the inequalities\n$$\n\\frac{1}{2L}\\norm{g_L(y)}^2 \\leq \\varphi(y)-\\varphi(T_L(y)) \\leq \\varphi(y)-\\varphi^*,\n\\qquad\n\\frac{1}{2L}\\norm{g_L(y)} \\leq \\dist(y,X^*).\n$$\n%\n\\item If $f$ is $L_f$-smooth, then $\\varphi(T_L(y))\\leq m_L(y;T_L(y))$ holds for all $y\\in \\E$ and $L\\geq L_f$.\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"0999eb5c4fdbed36cbb29e6c8abde917ae9b5edd60dd59778a4c16558d8a9aa2"}]},{"id":"S033","semantic_type":"proof","kind":"proof","title":"梯度映射性质完整证明","source_units":[{"start":16934,"end":19649,"start_line":328,"end_line":383,"raw_tex":"\\begin{proof}\n(i) is proved in \\cite[Lemma 2]{Nes13}.\n\n\n(ii) According to the optimality condition of the problem $\\min_{x \\in \\E}m_L(y;x)$, we have the following equivalence for $y,z \\in \\E$:\n\\begin{equation}\\label{optcond-TL}\nz = T_L(y) \\iff 0 \\in \\nabla{f}(y)+L(z-y)+\\partial\\varPsi(z).\n\\end{equation}\nOn the other hand, the optimality of the original problem $\\min_{x}\\varphi(x)$ is characterized as follows. Given $w\\in \\E$, we have\n$w \\in \\Argmin_{x \\in \\E}\\varphi(x)$ if and only if $0 \\in \\nabla{f}(w)+\\partial{\\varPsi}(w)$ holds\n%$$\n%w \\in \\Argmin_{x \\in \\E}\\varphi(x) \\iff 0 \\in \\nabla{f}(w)+\\partial{\\varPsi}(w).\n%$$\n(Remark that we have $\\partial\\varphi=\\nabla{f}+\\partial \\varPsi$ \\cite[Theorem~23.8]{Roc}).\nLet us show the equivalence\n\\begin{equation}\\label{eq:gmap-opt-sol-ch}\nx^* \\in \\Argmin_{x \\in \\E}\\varphi(x) \\iff x^*=T_L(x^*) ~~(\\iff g_L(x^*)=0).\n\\end{equation}\nIf $x^*=T_L(x^*)$, then \\eqref{optcond-TL} implies $0 \\in \\nabla{f}(x^*)+\\partial \\varPsi(x^*)$ and so $x^* \\in \\Argmin_{x\\in \\E}\\varphi(x)$ holds. Conversely, if $x^*\\in \\Argmin_{x \\in \\E}\\varphi(x)$, then we have the condition $0\\in \\nabla{f}(x^*)+\\partial \\varPsi(x^*)$, which coincides with the right hand one of \\eqref{optcond-TL} in the case $y=z=x^*$. Thus, we conclude\n its equivalent counterpart $x^*=T_L(x^*)$. This proves the equivalence \\eqref{eq:gmap-opt-sol-ch} as well as the assertion (ii).\n\n\n(iii) By the condition \\eqref{optcond-TL}, we have\n$\n0 \\in \\nabla{f}(y)+L(T_L(y)-y) + \\partial\\varPsi(T_L(y)) = \\nabla{f}(y)-g_L(y)  + \\partial\\varPsi(T_L(y)).\n$\nHence, we obtain\n$\n\\nabla{f}(T_L(y))-\\nabla{f}(y)+g_L(y) \\in \\nabla{f}(T_L(y))+\\partial\\varPsi(T_L(y)) = \\partial\\varphi(T_L(y))\n$, giving the former assertion of (iii).\nNow if $f$ is $L_f$-smooth, we have\n$\n\\norm{\\nabla{f}(T_L(y))-\\nabla{f}(y)} \\leq L_f\\norm{T_L(y)-y} = \\frac{L_f}{L}\\norm{g_L(y)},\n$\nwhich yields the latter assertion of (iii).\n\n\n(iv)\nIt is shown in \\cite[Lemma~2.3]{BT09} that,\nif $\\varphi(T_L(y))\\leq m_L(y;T_L(y))$ holds, then we have\n$$\n\\varphi(x)  - \\varphi(T_L(y)) \\geq \\frac{1}{2L}\\norm{g_L(y)}^2 + \\innprod{g_L(y)}{x-y},\\quad \\forall x \\in \\E.\n$$\nLetting $x:=y$ shows the first assertion. On the other hand, since $\\varphi(T_L(y))\\geq \\varphi^*$, letting $x:=x^* \\in X^*$ gives\n$\n\\frac{1}{2L}\\norm{g_L(y)}^2 \\leq \\innprod{g_L(y)}{y-x^*} \\leq \\norm{g_L(y)}\\norm{y-x^*}.\n$\nThus, we obtain $\\frac{1}{2L}\\norm{g_L(y)} \\leq \\dist(y,X^*)$.\n\n(v) Note that $f$ is $L$-smooth for any $L\\geq L_f$.\nSince $L$-smooth functions satisfy (e.g., see \\cite{Nes04})\n$$\nf(x) \\leq f(y)+\\innprod{\\nabla{f}(y)}{x-y} + \\frac{L}{2}\\norm{x-y}^2,\\quad \\forall x,y \\in \\E,\n$$\nwe obtain $\\varphi(x) \\leq m_L(y;x)$ for all $x,y \\in \\E$ and $L\\geq L_f$.\n\\end{proof}","sha256_utf8":"e6ef1c4590ada59c4e73a1a0675c541b761583bfdc5507656edcb9dea0e52d5f"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"引用梯度映射范数单调性","source_units":[{"start":16934,"end":16989,"start_line":328,"end_line":330,"raw_tex":"\\begin{proof}\n(i) is proved in \\cite[Lemma 2]{Nes13}.\n\n","sha256_utf8":"70a8388008522c625b7b55990d397c758e56fe0c8e514a9d3baf7429a2d4fa6b"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"近端最优性条件刻画最优解","source_units":[{"start":16989,"end":18325,"start_line":331,"end_line":348,"raw_tex":"\n(ii) According to the optimality condition of the problem $\\min_{x \\in \\E}m_L(y;x)$, we have the following equivalence for $y,z \\in \\E$:\n\\begin{equation}\\label{optcond-TL}\nz = T_L(y) \\iff 0 \\in \\nabla{f}(y)+L(z-y)+\\partial\\varPsi(z).\n\\end{equation}\nOn the other hand, the optimality of the original problem $\\min_{x}\\varphi(x)$ is characterized as follows. Given $w\\in \\E$, we have\n$w \\in \\Argmin_{x \\in \\E}\\varphi(x)$ if and only if $0 \\in \\nabla{f}(w)+\\partial{\\varPsi}(w)$ holds\n%$$\n%w \\in \\Argmin_{x \\in \\E}\\varphi(x) \\iff 0 \\in \\nabla{f}(w)+\\partial{\\varPsi}(w).\n%$$\n(Remark that we have $\\partial\\varphi=\\nabla{f}+\\partial \\varPsi$ \\cite[Theorem~23.8]{Roc}).\nLet us show the equivalence\n\\begin{equation}\\label{eq:gmap-opt-sol-ch}\nx^* \\in \\Argmin_{x \\in \\E}\\varphi(x) \\iff x^*=T_L(x^*) ~~(\\iff g_L(x^*)=0).\n\\end{equation}\nIf $x^*=T_L(x^*)$, then \\eqref{optcond-TL} implies $0 \\in \\nabla{f}(x^*)+\\partial \\varPsi(x^*)$ and so $x^* \\in \\Argmin_{x\\in \\E}\\varphi(x)$ holds. Conversely, if $x^*\\in \\Argmin_{x \\in \\E}\\varphi(x)$, then we have the condition $0\\in \\nabla{f}(x^*)+\\partial \\varPsi(x^*)$, which coincides with the right hand one of \\eqref{optcond-TL} in the case $y=z=x^*$. Thus, we conclude\n its equivalent counterpart $x^*=T_L(x^*)$. This proves the equivalence \\eqref{eq:gmap-opt-sol-ch} as well as the assertion (ii).\n\n","sha256_utf8":"cdb9ff2bcc433281dd86350408fb49a3479fa6a978be314cf5271711397dd434"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"构造近端点次梯度并控制范数","source_units":[{"start":18325,"end":18836,"start_line":349,"end_line":364,"raw_tex":"\n(iii) By the condition \\eqref{optcond-TL}, we have\n$\n0 \\in \\nabla{f}(y)+L(T_L(y)-y) + \\partial\\varPsi(T_L(y)) = \\nabla{f}(y)-g_L(y)  + \\partial\\varPsi(T_L(y)).\n$\nHence, we obtain\n$\n\\nabla{f}(T_L(y))-\\nabla{f}(y)+g_L(y) \\in \\nabla{f}(T_L(y))+\\partial\\varPsi(T_L(y)) = \\partial\\varphi(T_L(y))\n$, giving the former assertion of (iii).\nNow if $f$ is $L_f$-smooth, we have\n$\n\\norm{\\nabla{f}(T_L(y))-\\nabla{f}(y)} \\leq L_f\\norm{T_L(y)-y} = \\frac{L_f}{L}\\norm{g_L(y)},\n$\nwhich yields the latter assertion of (iii).\n\n\n","sha256_utf8":"5ef287dd26c787be494d6fdc50a895c43cc42e2a6147dc9374870a9eda0ff874"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"下降不等式控制残差与距离","source_units":[{"start":18836,"end":19349,"start_line":365,"end_line":376,"raw_tex":"(iv)\nIt is shown in \\cite[Lemma~2.3]{BT09} that,\nif $\\varphi(T_L(y))\\leq m_L(y;T_L(y))$ holds, then we have\n$$\n\\varphi(x)  - \\varphi(T_L(y)) \\geq \\frac{1}{2L}\\norm{g_L(y)}^2 + \\innprod{g_L(y)}{x-y},\\quad \\forall x \\in \\E.\n$$\nLetting $x:=y$ shows the first assertion. On the other hand, since $\\varphi(T_L(y))\\geq \\varphi^*$, letting $x:=x^* \\in X^*$ gives\n$\n\\frac{1}{2L}\\norm{g_L(y)}^2 \\leq \\innprod{g_L(y)}{y-x^*} \\leq \\norm{g_L(y)}\\norm{y-x^*}.\n$\nThus, we obtain $\\frac{1}{2L}\\norm{g_L(y)} \\leq \\dist(y,X^*)$.\n\n","sha256_utf8":"41d559fa6adbb837550893717b06eff7f091682228272998a7793f6f219ded7a"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"光滑性保证模型上界","source_units":[{"start":19349,"end":19649,"start_line":377,"end_line":383,"raw_tex":"(v) Note that $f$ is $L$-smooth for any $L\\geq L_f$.\nSince $L$-smooth functions satisfy (e.g., see \\cite{Nes04})\n$$\nf(x) \\leq f(y)+\\innprod{\\nabla{f}(y)}{x-y} + \\frac{L}{2}\\norm{x-y}^2,\\quad \\forall x,y \\in \\E,\n$$\nwe obtain $\\varphi(x) \\leq m_L(y;x)$ for all $x,y \\in \\E$ and $L\\geq L_f$.\n\\end{proof}","sha256_utf8":"8dfd8fc5c81c5bcad386d1197a98845e1dd3a14284c9a9ac3d6e4ccf457e32d5"}]},{"id":"S039","semantic_type":"lemma","kind":"lemma","title":"误差界蕴含次梯度下界","source_units":[{"start":21390,"end":22022,"start_line":415,"end_line":424,"raw_tex":"\\begin{lemma}\\label{lem:HEB}\nLet $\\varphi:\\E\\to\\RR\\cup\\{+\\infty\\}$ be a proper, lower-semicontinuous, and convex function.\nFor $x_0 \\in \\E$, suppose that $\\varphi$ satisfies the H\\\"olderian error bound condition \\eqref{HEB} with coefficient $\\kappa > 0$ and exponent $\\rho \\geq 1$.\nThen, for every $x \\in \\lev_{\\varphi}(\\varphi(x_0))\\setminus X^*$, we have\nthe inequalities\n\\begin{equation}\\label{Loj-ineq}\n\\kappa \\dist(x,X^*)^{\\rho-1} \\leq \\inf\\{\\norm{g}:g \\in \\partial\\varphi(x)\\},\\quad\n\\kappa^{\\frac{1}{\\rho}} (\\varphi(x)-\\varphi^*)^{\\frac{\\rho-1}{\\rho}} \\leq \\inf\\{\\norm{g}:g \\in \\partial\\varphi(x)\\}.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"5f13fa11531e4c41745b14b909156ec7125b551c61c4c0adb2cfd9d65a2df8bc"}]},{"id":"S040","semantic_type":"proof","kind":"proof","title":"次梯度下界完整证明","source_units":[{"start":22023,"end":22707,"start_line":425,"end_line":441,"raw_tex":"\\begin{proof}\nLet $x^*$ be the projection of $x$ onto $X^*$, so that $\\norm{x-x^*}=\\dist(x,X^*)$.\nFor every $g\\in\\partial \\varphi(x)$, we have\n\\begin{align*}\n\\kappa \\dist(x,X^*)^\\rho\n&\\leq \\varphi(x)-\\varphi^*\n\\leq -\\innprod{g}{x^*-x}\n\\leq \\norm{g}\\dist(x,X^*)\\leq \\norm{g}\\frac{1}{\\kappa^{1/\\rho}}(\\varphi(x)-\\varphi^*)^{1/\\rho}.\n\\end{align*}\nIt involves two inequalities\n$\n\\kappa \\dist(x,X^*)^\\rho \\leq \\norm{g}\\dist(x,X^*)$\nand\n$\\varphi(x)-\\varphi^* \\leq \\norm{g}\\frac{1}{\\kappa^{1/\\rho}}(\\varphi(x)-\\varphi^*)^{1/\\rho}.\n$\nSince $x \\not \\in X^*$, the assertion follows dividing them by $\\dist(x,X^*)$ and $\\kappa^{-1/\\rho}(\\varphi(x)-\\varphi^*)^{1/\\rho}$, respectively.\n\\end{proof}","sha256_utf8":"0db1804d3d5744fe55cd709d624e5bca88429adb3d26c5b2c1b658679200a3ac"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"投影点与次梯度不等式","source_units":[{"start":22023,"end":22367,"start_line":425,"end_line":433,"raw_tex":"\\begin{proof}\nLet $x^*$ be the projection of $x$ onto $X^*$, so that $\\norm{x-x^*}=\\dist(x,X^*)$.\nFor every $g\\in\\partial \\varphi(x)$, we have\n\\begin{align*}\n\\kappa \\dist(x,X^*)^\\rho\n&\\leq \\varphi(x)-\\varphi^*\n\\leq -\\innprod{g}{x^*-x}\n\\leq \\norm{g}\\dist(x,X^*)\\leq \\norm{g}\\frac{1}{\\kappa^{1/\\rho}}(\\varphi(x)-\\varphi^*)^{1/\\rho}.\n\\end{align*}\n","sha256_utf8":"b3f37b68bc2e2ce9f64f94905193c2ee62aa6dd5b87a89b5aef81c709b733884"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"除以正距离和正残差","source_units":[{"start":22367,"end":22707,"start_line":434,"end_line":441,"raw_tex":"It involves two inequalities\n$\n\\kappa \\dist(x,X^*)^\\rho \\leq \\norm{g}\\dist(x,X^*)$\nand\n$\\varphi(x)-\\varphi^* \\leq \\norm{g}\\frac{1}{\\kappa^{1/\\rho}}(\\varphi(x)-\\varphi^*)^{1/\\rho}.\n$\nSince $x \\not \\in X^*$, the assertion follows dividing them by $\\dist(x,X^*)$ and $\\kappa^{-1/\\rho}(\\varphi(x)-\\varphi^*)^{1/\\rho}$, respectively.\n\\end{proof}","sha256_utf8":"cc0fe8924cce67aa6f4d0ba7a554a11f90ab49a51696e7bdaa733d7c1fd33ae9"}]},{"id":"S043","semantic_type":"lemma","kind":"lemma","title":"正则化问题与原问题的关系","source_units":[{"start":30103,"end":30413,"start_line":565,"end_line":571,"raw_tex":"\\begin{lemma}\\label{lem:reg}\n\\begin{enumerate}[(i)]\n\\item $\\varphi_\\sigma(x) \\leq \\varphi_\\sigma(x_0)$ implies $\\varphi(x) \\leq \\varphi(x_0)$.\n\\item $\\norm{x_0-x_\\sigma^*} \\leq \\dist(x_0,X^*)$.\n\\item We have $\\norm{g_L(y)-g_L^\\sigma(y)} \\leq  \\sigma\\norm{y-x_0}$ for any $y \\in \\E$.\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"458a044b8dd6259f80bc074a1ead8cd5a2faf6c2f25bc2cefe2953d13f83050a"}]},{"id":"S044","semantic_type":"proof","kind":"proof","title":"正则化关系完整证明","source_units":[{"start":30415,"end":32028,"start_line":573,"end_line":604,"raw_tex":"\\begin{proof}\n(i) is immediate by $\\varphi(x)\\leq \\varphi_\\sigma(x)$ and $\\varphi_\\sigma(x_0)=\\varphi(x_0)$.\\\\\n(ii) For any $x^* \\in X^*$, the strong convexity of $\\varphi_\\sigma$ implies\n$\n\\varphi_\\sigma(x^*)\\geq \\varphi_\\sigma^* + \\frac{\\sigma}{2}\\norm{x^*-x_\\sigma^*}^2,\n$\nwhich can be rewritten as\n$\\varphi(x^*)-\\varphi(x_\\sigma^*) \\geq \\frac{\\sigma}{2}(-\\norm{x^*-x_0}^2 + \\norm{x_\\sigma^*-x_0}^2 + \\norm{x^*-x_\\sigma^*}^2).$\nAs $\\varphi(x^*)-\\varphi(x_\\sigma^*)\\leq 0$, we conclude $\\norm{x^*-x_0}^2 \\geq \\norm{x_\\sigma^*-x_0}^2 + \\norm{x^*-x_\\sigma^*}^2 \\geq \\norm{x_\\sigma^*-x_0}^2$, which proves the assertion.\\\\\n(iii)\nIn general, if $h$ is a lower-semicontinuous and $\\mu$-strongly convex function, we have for any $a,b \\in \\E$ that\\footnote{\nBy the strong convexity of $\\innprod{a}{x}+h(x)$ and $\\innprod{b}{x}+h(x)$, we have\n$$\n\\frac{\\mu}{2}\\norm{x_a^*-x_b^*}^2 \\leq [\\innprod{a}{x_b^*}+h(x_b^*)] - [\\innprod{a}{x_a^*}+h(x_a^*)]\n~~\\text{and}~~\n\\frac{\\mu}{2}\\norm{x_a^*-x_b^*}^2 \\leq  [\\innprod{b}{x_a^*}+h(x_a^*)]-[\\innprod{b}{x_b^*}+h(x_b^*)],\n$$\nrespectively.\nAdding them implies\n$\n\\mu\\norm{x_a^*-x_b^*}^2 \\leq \\innprod{a-b}{x_b^*-x_a^*} \\leq \\norm{a-b}\\norm{x_b^*-x_a^*}.\n$\n}\n$$\\norm{x_a^*-x_b^*}\\leq \\frac{\\norm{a-b}}{\\mu},~~\\text{where}~~\nx_a^* = \\argmin_{x \\in \\E}\\{\\innprod{a}{x}+h(x)\\},~~ x_b^* = \\argmin_{x \\in \\E}\\{\\innprod{b}{x}+h(x)\\}.$$\nThis fact implies\n$\n\\norm{T_L(y)-T_L^\\sigma(y)} \\leq \\frac{\\norm{\\nabla{f}(y)-\\nabla{f}_\\sigma(y)}}{L} = \\frac{\\sigma\\norm{y-x_0}}{L}.\n$\nTherefore, we conclude (iii) because of $\\norm{g_L(y)-g_L^\\sigma(y)}\n=  L\\norm{T_L(y)-T_L^\\sigma(y)}$.\n\\end{proof}","sha256_utf8":"f389bcdd37bd22eb5b779b99c3f59108b27f5faeedec1465355cb03e1d101e5a"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"正则化下降保证原目标下降","source_units":[{"start":30415,"end":30526,"start_line":573,"end_line":574,"raw_tex":"\\begin{proof}\n(i) is immediate by $\\varphi(x)\\leq \\varphi_\\sigma(x)$ and $\\varphi_\\sigma(x_0)=\\varphi(x_0)$.\\\\\n","sha256_utf8":"9a02a14c3c748532a359528062a546e202b5aeaa9ced88c2191413e06f548347"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"强凸性控制正则化最优点距离","source_units":[{"start":30526,"end":31037,"start_line":575,"end_line":581,"raw_tex":"(ii) For any $x^* \\in X^*$, the strong convexity of $\\varphi_\\sigma$ implies\n$\n\\varphi_\\sigma(x^*)\\geq \\varphi_\\sigma^* + \\frac{\\sigma}{2}\\norm{x^*-x_\\sigma^*}^2,\n$\nwhich can be rewritten as\n$\\varphi(x^*)-\\varphi(x_\\sigma^*) \\geq \\frac{\\sigma}{2}(-\\norm{x^*-x_0}^2 + \\norm{x_\\sigma^*-x_0}^2 + \\norm{x^*-x_\\sigma^*}^2).$\nAs $\\varphi(x^*)-\\varphi(x_\\sigma^*)\\leq 0$, we conclude $\\norm{x^*-x_0}^2 \\geq \\norm{x_\\sigma^*-x_0}^2 + \\norm{x^*-x_\\sigma^*}^2 \\geq \\norm{x_\\sigma^*-x_0}^2$, which proves the assertion.\\\\\n","sha256_utf8":"c830bf537e753f8813fda6964ef148c3a1d8a57f9dbfde49e85553b43713c5be"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"强凸极小点对线性扰动的稳定性","source_units":[{"start":31037,"end":31168,"start_line":582,"end_line":583,"raw_tex":"(iii)\nIn general, if $h$ is a lower-semicontinuous and $\\mu$-strongly convex function, we have for any $a,b \\in \\E$ that\\footnote{\n","sha256_utf8":"ee2c525e234614188ba45d83c39e145e798d69387a464d8e320ff11215100a35"},{"start":31606,"end":31777,"start_line":596,"end_line":597,"raw_tex":"$$\\norm{x_a^*-x_b^*}\\leq \\frac{\\norm{a-b}}{\\mu},~~\\text{where}~~\nx_a^* = \\argmin_{x \\in \\E}\\{\\innprod{a}{x}+h(x)\\},~~ x_b^* = \\argmin_{x \\in \\E}\\{\\innprod{b}{x}+h(x)\\}.$$\n","sha256_utf8":"d639963c65048a1e0bf4aed371a11a91b5269e47902fcfdbe0be13cdb29d7016"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"脚注中稳定性不等式的证明","source_units":[{"start":31168,"end":31606,"start_line":584,"end_line":595,"raw_tex":"By the strong convexity of $\\innprod{a}{x}+h(x)$ and $\\innprod{b}{x}+h(x)$, we have\n$$\n\\frac{\\mu}{2}\\norm{x_a^*-x_b^*}^2 \\leq [\\innprod{a}{x_b^*}+h(x_b^*)] - [\\innprod{a}{x_a^*}+h(x_a^*)]\n~~\\text{and}~~\n\\frac{\\mu}{2}\\norm{x_a^*-x_b^*}^2 \\leq  [\\innprod{b}{x_a^*}+h(x_a^*)]-[\\innprod{b}{x_b^*}+h(x_b^*)],\n$$\nrespectively.\nAdding them implies\n$\n\\mu\\norm{x_a^*-x_b^*}^2 \\leq \\innprod{a-b}{x_b^*-x_a^*} \\leq \\norm{a-b}\\norm{x_b^*-x_a^*}.\n$\n}\n","sha256_utf8":"2b1707a7532e021f6fc0d7fc6d476397e7a51f4e880d1366f3cc8d59647737e3"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"应用稳定性控制梯度映射扰动","source_units":[{"start":31777,"end":32028,"start_line":598,"end_line":604,"raw_tex":"This fact implies\n$\n\\norm{T_L(y)-T_L^\\sigma(y)} \\leq \\frac{\\norm{\\nabla{f}(y)-\\nabla{f}_\\sigma(y)}}{L} = \\frac{\\sigma\\norm{y-x_0}}{L}.\n$\nTherefore, we conclude (iii) because of $\\norm{g_L(y)-g_L^\\sigma(y)}\n=  L\\norm{T_L(y)-T_L^\\sigma(y)}$.\n\\end{proof}","sha256_utf8":"315d300deebaaff6b75d237e333bda2e891a32bdc1396b1f4f79e651b213ea48"}]},{"id":"S050","semantic_type":"lemma","kind":"lemma","title":"充分大的回溯参数保证验收","source_units":[{"start":37695,"end":37822,"start_line":679,"end_line":681,"raw_tex":"\\begin{lemma}\\label{lem:APG-iter-L}\nThe condition $\\eqref{APG-iter-test}$ of $\\APGIter$ holds whenever $L\\geq L_f$.\n\\end{lemma}","sha256_utf8":"6ae5df3ca9a2d1ae7416b911ebb91a35d86d047061eb738903c1107480ea355f"}]},{"id":"S051","semantic_type":"proof","kind":"proof","title":"回溯验收条件完整证明","source_units":[{"start":37823,"end":38523,"start_line":682,"end_line":694,"raw_tex":"\\begin{proof}\nThe first condition \\eqref{APG-iter-test-a} is satisfied since $f$ is $L$-smooth (e.g., see \\cite[Theorem 2.1.5]{Nes04}).\nSince $f_\\sigma$ is $(L_f+\\sigma)$-smooth, the second one \\eqref{APG-iter-test-b} holds if $L\\geq L_f$ by Lemma~\\ref{lem:gmap} (v).\nThe third one \\eqref{APG-iter-test-c} can be verified again by Lemma~\\ref{lem:gmap} (v): If $L\\geq L_f$, we have\n$\n\\varphi(T_L(z)) \\leq m_L(z;T_L(z)) \\leq \\varphi(z),\n$\nwhere the second inequality follows from\n\\begin{equation}\\label{mL-leq-phi}\nm_L(z;T_L(z))=\\min_{x \\in \\E}\\left\\{f(z)+\\innprod{\\nabla{f}(z)}{x-z}+\\varPsi(x)+\\frac{L}{2}\\norm{x-z}^2\\right\\} \\stackrel{x=z}{\\leq}\nf(z)+\\varPsi(z)=\\varphi(z).\n\\end{equation}\n\\end{proof}","sha256_utf8":"0ff5272d993234996546c933a721f93e183b4105da321ce6285d361d7fb4a6e1"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"验证共胁性与正则化模型条件","source_units":[{"start":37823,"end":38091,"start_line":682,"end_line":684,"raw_tex":"\\begin{proof}\nThe first condition \\eqref{APG-iter-test-a} is satisfied since $f$ is $L$-smooth (e.g., see \\cite[Theorem 2.1.5]{Nes04}).\nSince $f_\\sigma$ is $(L_f+\\sigma)$-smooth, the second one \\eqref{APG-iter-test-b} holds if $L\\geq L_f$ by Lemma~\\ref{lem:gmap} (v).\n","sha256_utf8":"0193f5dc0ca9dfe7313452e5ae14a6b2a5fa59ba2f898ee6e81ed007a8478a59"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"模型极小性验证原目标下降","source_units":[{"start":38091,"end":38523,"start_line":685,"end_line":694,"raw_tex":"The third one \\eqref{APG-iter-test-c} can be verified again by Lemma~\\ref{lem:gmap} (v): If $L\\geq L_f$, we have\n$\n\\varphi(T_L(z)) \\leq m_L(z;T_L(z)) \\leq \\varphi(z),\n$\nwhere the second inequality follows from\n\\begin{equation}\\label{mL-leq-phi}\nm_L(z;T_L(z))=\\min_{x \\in \\E}\\left\\{f(z)+\\innprod{\\nabla{f}(z)}{x-z}+\\varPsi(x)+\\frac{L}{2}\\norm{x-z}^2\\right\\} \\stackrel{x=z}{\\leq}\nf(z)+\\varPsi(z)=\\varphi(z).\n\\end{equation}\n\\end{proof}","sha256_utf8":"79504ff3f31cb76577d5a65eddf5099afd81044913fbf2a3f81665e44a5716c7"}]},{"id":"S054","semantic_type":"lemma","kind":"lemma","title":"回溯参数及搜索次数界","source_units":[{"start":38657,"end":39201,"start_line":697,"end_line":705,"raw_tex":"\\begin{lemma}\\label{lem:APG-iter-LS-compl}\n\\sloppy % prevents overflow\nSuppose that $L_k \\leq \\gamma_{\\rm inc}L_f$ holds.\nThen, $\\APGIter$ ensures the conditions $M_k \\leq \\gamma_{\\rm inc}L_f$ and ${L_{k+1} \\leq \\max\\{L_{\\min},\\frac{\\gamma_{\\rm inc}}{\\gamma_{\\rm dec}}L_f\\}\\leq \\gamma_{\\rm inc}L_f}$.\nThe number of executions of the loop {\\rm (Lines~\\ref{APG-iter-loop-s}--\\ref{APG-iter-loop-e})} is bounded by\n$$\n1 + \\frac{\\log \\gamma_{\\rm dec}}{\\log \\gamma_{\\rm inc}} + \\frac{1}{\\log \\gamma_{\\rm inc}} \\log \\frac{L_{k+1}}{L_k}.\n$$\n\\end{lemma}","sha256_utf8":"ea0941d3144fb0b34c7b624b26ed2c8cac59d29cb93c3faaf67af23c703b491e"}]},{"id":"S055","semantic_type":"proof","kind":"proof","title":"回溯搜索复杂度完整证明","source_units":[{"start":39202,"end":40253,"start_line":706,"end_line":719,"raw_tex":"\\begin{proof}\n\\sloppy % prevents overflow\nLet $n_k \\geq 1$ be the number of inner loops so that ${M_k = L_k\\gamma_{\\rm inc}^{n_k-1}}$.\nIf $n_k=1$, then we have $M_k = L_k \\leq \\gamma_{\\rm inc}L_f$. If $n_k>1$, then $M_k\\leq \\gamma_{\\rm inc}L_f$ must hold because otherwise ${L_k\\gamma_{\\rm inc}^{n_k-2}=M_k/\\gamma_{\\rm inc} > L_f}$ and the $n_k$-th loop cannot occur (by Lemma~\\ref{lem:APG-iter-L}). Hence, we conclude $M_k \\leq \\gamma_{\\rm inc}L_f$ and also ${L_{k+1} = \\max\\{L_{\\min},M_k/\\gamma_{\\rm dec}\\} \\leq \\max\\{L_{\\min},\\frac{\\gamma_{\\rm inc}}{\\gamma_{\\rm dec}}L_f\\}}$.\n\nNow, the relation $M_k = L_k\\gamma_{\\rm inc}^{n_k-1}$ implies\n$\nL_{k+1} = \\max\\{L_{\\min},M_k/\\gamma_{\\rm dec}\\} \\geq \\frac{1}{\\gamma_{\\rm dec}}L_k\\gamma_{\\rm inc}^{n_k-1}.\n$\nThen, we have $(n_k-1)\\log\\gamma_{\\rm inc} \\leq \\log \\frac{\\gamma_{\\rm dec}L_{k+1}}{L_k} = \\log \\gamma_{\\rm dec} + \\log \\frac{L_{k+1}}{L_k}$ and therefore\n$\nn_k \\leq  1 + \\frac{\\log \\gamma_{\\rm dec}}{\\log \\gamma_{\\rm inc}} + \\frac{1}{\\log \\gamma_{\\rm inc}} \\log \\frac{L_{k+1}}{L_k} .\n$\n\\end{proof}","sha256_utf8":"d034a4e2e910358072f2c9687bdeac8348b4df7bd235eec36cd5fb86dc96413b"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"由验收阈值控制接受参数","source_units":[{"start":39202,"end":39782,"start_line":706,"end_line":710,"raw_tex":"\\begin{proof}\n\\sloppy % prevents overflow\nLet $n_k \\geq 1$ be the number of inner loops so that ${M_k = L_k\\gamma_{\\rm inc}^{n_k-1}}$.\nIf $n_k=1$, then we have $M_k = L_k \\leq \\gamma_{\\rm inc}L_f$. If $n_k>1$, then $M_k\\leq \\gamma_{\\rm inc}L_f$ must hold because otherwise ${L_k\\gamma_{\\rm inc}^{n_k-2}=M_k/\\gamma_{\\rm inc} > L_f}$ and the $n_k$-th loop cannot occur (by Lemma~\\ref{lem:APG-iter-L}). Hence, we conclude $M_k \\leq \\gamma_{\\rm inc}L_f$ and also ${L_{k+1} = \\max\\{L_{\\min},M_k/\\gamma_{\\rm dec}\\} \\leq \\max\\{L_{\\min},\\frac{\\gamma_{\\rm inc}}{\\gamma_{\\rm dec}}L_f\\}}$.\n\n","sha256_utf8":"15d5b4eeabb871d0be5d67d1995a5a1faf0aff5aaf93994101cbbb67019b997b"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"参数更新导出搜索次数界","source_units":[{"start":39782,"end":40253,"start_line":711,"end_line":719,"raw_tex":"Now, the relation $M_k = L_k\\gamma_{\\rm inc}^{n_k-1}$ implies\n$\nL_{k+1} = \\max\\{L_{\\min},M_k/\\gamma_{\\rm dec}\\} \\geq \\frac{1}{\\gamma_{\\rm dec}}L_k\\gamma_{\\rm inc}^{n_k-1}.\n$\nThen, we have $(n_k-1)\\log\\gamma_{\\rm inc} \\leq \\log \\frac{\\gamma_{\\rm dec}L_{k+1}}{L_k} = \\log \\gamma_{\\rm dec} + \\log \\frac{L_{k+1}}{L_k}$ and therefore\n$\nn_k \\leq  1 + \\frac{\\log \\gamma_{\\rm dec}}{\\log \\gamma_{\\rm inc}} + \\frac{1}{\\log \\gamma_{\\rm inc}} \\log \\frac{L_{k+1}}{L_k} .\n$\n\\end{proof}","sha256_utf8":"6893dcfb69ecdbe9ec34c84896f82f4aba5613a35562034da751902bd220b5e6"}]},{"id":"S058","semantic_type":"proposition","kind":"proposition","title":"正则化加速方法的基本估计","source_units":[{"start":40323,"end":41819,"start_line":724,"end_line":753,"raw_tex":"\\begin{proposition}\\label{prop:APG}\nLet $\\{x_k,\\psi_k,M_k,L_k,A_k\\}$ be generated by the accelerated proximal gradient method applied to the regularized objective function \\eqref{realloc}, that is,\n$$\\{x_{k+1},\\psi_{k+1},M_k,L_{k+1},A_{k+1}\\} \\leftarrow \\APGIter_\\sigma(x_k,\\psi_k,L_k,A_k),\\quad k=0,1,2,\\ldots,$$\nwith the initialization $x_0 \\in \\E$, $\\psi_0(x)=\\frac{1}{2}\\norm{x-x_0}^2$, $A_0=0$, $L_0>0$.\n\\begin{enumerate}[(i)]\n\\item\nIf $L_0 \\in [L_{\\min}, \\gamma_{\\rm inc}L_f]$, then\nwe have $L_{\\min} \\leq L_k \\leq M_k \\leq \\gamma_{\\rm inc}L_f$ for all $k \\geq 0$.\nThe total number of executions of the loop {\\rm (Lines~\\ref{APG-iter-loop-s}--\\ref{APG-iter-loop-e})} until the $k$-th iteration is bounded by \n$$\n\\left[\n1+\\frac{\\log \\gamma_{\\rm dec}}{\\log \\gamma_{\\rm inc}}\\right](k+1) + \\frac{1}{\\log \\gamma_{\\rm inc}}\\log\\frac{L_{k+1}}{L_0}.\n$$\n\\item For each $k\\geq 1$, we have\n$\nA_k \\geq \\frac{2}{\\gamma_{\\rm inc}L_f}\\left[1+\\sqrt{\\frac{\\sigma}{2\\gamma_{\\rm inc}L_f}}\\right]^{2(k-1)}.\n$\n\\item $\\varphi_\\sigma(x_k)-\\varphi_\\sigma^* \\leq \\frac{\\norm{x_0-x_\\sigma^*}^2}{2A_k}$ holds for all $k\\geq 1$.\n\\item $\\varphi(x_k)\\leq \\varphi(x_0)$ holds for all $k\\geq 1$.\n\\item\nFor every $k\\geq 1$, we have\n$$\n\\norm{x_k-x_0} \\leq \\left(1+\\frac{1}{\\sqrt{\\sigma A_k}}\\right)\\dist(x_0,X^*),\n$$\n$$\n\\norm{g_{M_{k-1}}(x_k)} \\leq \\norm{g_{M_{k-1}+\\sigma}^\\sigma(x_k)} + \\sigma\\norm{x_k-x_0}\n\\leq \\left(2\\sqrt{\\frac{M_{k-1}+\\sigma}{A_k}} + \\sigma\\right)\\dist(x_0,X^*).\n$$\n\\end{enumerate}\n\\end{proposition}","sha256_utf8":"bc25cb6ddf7182778f82aaab94d84b86ea8213c9b303f92c39f20f59c75a4429"}]},{"id":"S059","semantic_type":"proof","kind":"proof","title":"正则化加速估计完整证明","source_units":[{"start":41820,"end":44393,"start_line":754,"end_line":810,"raw_tex":"\\begin{proof}\n(i) \n$L_{\\min}\\leq L_k\\leq M_k$ is clear by the construction. $M_k\\leq \\gamma_{\\rm inc}L_f$\nis obtained by applying Lemma~\\ref{lem:APG-iter-LS-compl} inductively.\n\nSince $\\varPsi_\\sigma$ is $\\sigma$-strongly convex, \n(ii) follows by \\cite[Lemma~8]{Nes13} applied to the objective function $\\varphi_\\sigma=f+\\varPsi_\\sigma$.\n\nAccording to \\cite[Lemma 7]{Nes13}, the following relations hold for all $k\\geq 0$:\n$$\n\\left\\{\n\\begin{array}{l}\nA_k\\varphi_\\sigma(x_k) \\leq \\min_{x \\in \\E}\\psi_k(x),\\\\\n\\psi_k(x)\\leq A_k\\varphi_\\sigma(x)+\\frac{1}{2}\\norm{x-x_0}^2,\\quad \\forall x \\in \\E.\n\\end{array}\n\\right.\n$$\nCombining them, we obtain\n$$\nA_k\\varphi_\\sigma(x_k) \\leq \\min_{x\\in \\E}\\left\\{ A_k\\varphi_\\sigma(x)+\\frac{1}{2}\\norm{x-x_0}^2 \\right\\}, \\quad \\forall k\\geq 0.\n$$\nTaking $x=x_\\sigma^*$ on the right hand side, we obtain (iii).\nOn the other hand, taking $x=x_0$ yields ${\\varphi_\\sigma(x_k)\\leq \\varphi_\\sigma(x_0)}$. \nThen, Lemma~\\ref{lem:reg} (i) gives the assertion $\\varphi(x_k)\\leq \\varphi(x_0)$.\n\n(v)\nBy the $\\sigma$-strong convexity of $\\varphi_\\sigma$ and using (iii), we have\n$\n\\frac{\\sigma}{2}\\norm{x_k-x_\\sigma^*}^2 \\leq \\varphi_\\sigma(x_k)-\\varphi_\\sigma^* \\leq \\frac{\\norm{x_0 -x_\\sigma^*}^2}{2A_k},\n$\nand thus\n$\n\\norm{x_k-x_\\sigma^*} \\leq \\frac{1}{\\sqrt{\\sigma A_k}}\\norm{x_0-x_\\sigma^*}.\n$\nThis shows the former inequality of (v):\n$$\n\\norm{x_k-x_0}\\leq \\norm{x_k-x_\\sigma^*} + \\norm{x_0-x_\\sigma^*} \\leq \\left(1+\\frac{1}{\\sqrt{\\sigma A_k}}\\right) \\norm{x_0-x_\\sigma^*}\n\\leq\n\\left(1+\\frac{1}{\\sqrt{\\sigma A_k}}\\right) \\dist(x_0,X^*),\n$$\nwhere the last inequality follows by Lemma~\\ref{lem:reg} (ii).\n\nSince $\\varphi_\\sigma(T_{M_{k-1}+\\sigma}^\\sigma(x_k)) \\leq m_{M_{k-1}+\\sigma}^\\sigma(x_k;T_{M_{k-1}+\\sigma}^\\sigma(x_k))$ holds by \\eqref{APG-iter-test-b}, Lemma~\\ref{lem:gmap} (iv) yields\n$$\n\\frac{1}{2(M_{k-1}+\\sigma)}\\norm{g_{M_{k-1}+\\sigma}^\\sigma(x_k)}^2\n\\leq \\varphi_\\sigma(x_k)-\\varphi_\\sigma^* \\leq \\frac{\\norm{x_0-x_\\sigma^*}^2}{2A_k}\n\\leq \\frac{\\dist(x_0,X^*)^{2}}{2A_k}.\n$$\nThus, $\\norm{g_{M_{k-1}+\\sigma}^\\sigma(x_k)} \\leq \\sqrt{\\frac{M_{k-1}+\\sigma}{A_k}}\\dist(x_0,X^*)$ holds.\nConsequently, using Lemma~\\ref{lem:gmap}~(i) and Lemma~\\ref{lem:reg}~(iii), we obtain\n\\begin{align*}\n\\norm{g_{M_{k-1}}(x_k)} &\\leq \\norm{g_{M_{k-1}+\\sigma}(x_k)}\n\\leq \\norm{g_{M_{k-1}+\\sigma}^\\sigma(x_k)} + \\sigma\\norm{x_k - x_0}\n\\\\&\\leq  \\sqrt{\\frac{M_{k-1}+\\sigma}{A_k}}\\dist(x_0,X^*) + \\sigma\\left(1+\\frac{1}{\\sqrt{\\sigma A_k}}\\right)\\dist(x_0,X^*)\n\\\\&\\leq \\left(2\\sqrt{\\frac{M_{k-1}+\\sigma}{A_k}} + \\sigma\\right)\\dist(x_0,X^*).\n\\end{align*}\n\\end{proof}","sha256_utf8":"d68d6e7e8fc850fc7c58926610acaa348501df7a31366145852296618043953b"}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","title":"归纳控制加速回溯参数","source_units":[{"start":41820,"end":41998,"start_line":754,"end_line":758,"raw_tex":"\\begin{proof}\n(i) \n$L_{\\min}\\leq L_k\\leq M_k$ is clear by the construction. $M_k\\leq \\gamma_{\\rm inc}L_f$\nis obtained by applying Lemma~\\ref{lem:APG-iter-LS-compl} inductively.\n\n","sha256_utf8":"021422e7988855ceb395e433368e73f42468ed1ab3bc23e51b7db795db25982a"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"引用加速权重增长界","source_units":[{"start":41998,"end":42158,"start_line":759,"end_line":760,"raw_tex":"Since $\\varPsi_\\sigma$ is $\\sigma$-strongly convex, \n(ii) follows by \\cite[Lemma~8]{Nes13} applied to the objective function $\\varphi_\\sigma=f+\\varPsi_\\sigma$.\n","sha256_utf8":"adb1f7d1d53fc9368cd7fc2d6a8e568e389083fe2aac3538e43c814aa8b26c58"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"估计序列导出目标残差与下降","source_units":[{"start":42158,"end":42835,"start_line":761,"end_line":778,"raw_tex":"\nAccording to \\cite[Lemma 7]{Nes13}, the following relations hold for all $k\\geq 0$:\n$$\n\\left\\{\n\\begin{array}{l}\nA_k\\varphi_\\sigma(x_k) \\leq \\min_{x \\in \\E}\\psi_k(x),\\\\\n\\psi_k(x)\\leq A_k\\varphi_\\sigma(x)+\\frac{1}{2}\\norm{x-x_0}^2,\\quad \\forall x \\in \\E.\n\\end{array}\n\\right.\n$$\nCombining them, we obtain\n$$\nA_k\\varphi_\\sigma(x_k) \\leq \\min_{x\\in \\E}\\left\\{ A_k\\varphi_\\sigma(x)+\\frac{1}{2}\\norm{x-x_0}^2 \\right\\}, \\quad \\forall k\\geq 0.\n$$\nTaking $x=x_\\sigma^*$ on the right hand side, we obtain (iii).\nOn the other hand, taking $x=x_0$ yields ${\\varphi_\\sigma(x_k)\\leq \\varphi_\\sigma(x_0)}$. \nThen, Lemma~\\ref{lem:reg} (i) gives the assertion $\\varphi(x_k)\\leq \\varphi(x_0)$.\n\n","sha256_utf8":"98295c91e234db188daa27578e813422483bc498a184c96177503086c41512f2"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"强凸性导出迭代点距离界","source_units":[{"start":42835,"end":43446,"start_line":779,"end_line":794,"raw_tex":"(v)\nBy the $\\sigma$-strong convexity of $\\varphi_\\sigma$ and using (iii), we have\n$\n\\frac{\\sigma}{2}\\norm{x_k-x_\\sigma^*}^2 \\leq \\varphi_\\sigma(x_k)-\\varphi_\\sigma^* \\leq \\frac{\\norm{x_0 -x_\\sigma^*}^2}{2A_k},\n$\nand thus\n$\n\\norm{x_k-x_\\sigma^*} \\leq \\frac{1}{\\sqrt{\\sigma A_k}}\\norm{x_0-x_\\sigma^*}.\n$\nThis shows the former inequality of (v):\n$$\n\\norm{x_k-x_0}\\leq \\norm{x_k-x_\\sigma^*} + \\norm{x_0-x_\\sigma^*} \\leq \\left(1+\\frac{1}{\\sqrt{\\sigma A_k}}\\right) \\norm{x_0-x_\\sigma^*}\n\\leq\n\\left(1+\\frac{1}{\\sqrt{\\sigma A_k}}\\right) \\dist(x_0,X^*),\n$$\nwhere the last inequality follows by Lemma~\\ref{lem:reg} (ii).\n","sha256_utf8":"9afb03ad7ec158e2affe3e50a95dd2844c4d8a71b0da049182d437410c2f3caa"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"控制正则化梯度映射范数","source_units":[{"start":43446,"end":43937,"start_line":795,"end_line":802,"raw_tex":"\nSince $\\varphi_\\sigma(T_{M_{k-1}+\\sigma}^\\sigma(x_k)) \\leq m_{M_{k-1}+\\sigma}^\\sigma(x_k;T_{M_{k-1}+\\sigma}^\\sigma(x_k))$ holds by \\eqref{APG-iter-test-b}, Lemma~\\ref{lem:gmap} (iv) yields\n$$\n\\frac{1}{2(M_{k-1}+\\sigma)}\\norm{g_{M_{k-1}+\\sigma}^\\sigma(x_k)}^2\n\\leq \\varphi_\\sigma(x_k)-\\varphi_\\sigma^* \\leq \\frac{\\norm{x_0-x_\\sigma^*}^2}{2A_k}\n\\leq \\frac{\\dist(x_0,X^*)^{2}}{2A_k}.\n$$\nThus, $\\norm{g_{M_{k-1}+\\sigma}^\\sigma(x_k)} \\leq \\sqrt{\\frac{M_{k-1}+\\sigma}{A_k}}\\dist(x_0,X^*)$ holds.\n","sha256_utf8":"46cd86749e19c38158e00562ab08ce9662cbcc03e0ca7bce483d632c7991e80f"}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","title":"转移为原问题梯度映射界","source_units":[{"start":43937,"end":44393,"start_line":803,"end_line":810,"raw_tex":"Consequently, using Lemma~\\ref{lem:gmap}~(i) and Lemma~\\ref{lem:reg}~(iii), we obtain\n\\begin{align*}\n\\norm{g_{M_{k-1}}(x_k)} &\\leq \\norm{g_{M_{k-1}+\\sigma}(x_k)}\n\\leq \\norm{g_{M_{k-1}+\\sigma}^\\sigma(x_k)} + \\sigma\\norm{x_k - x_0}\n\\\\&\\leq  \\sqrt{\\frac{M_{k-1}+\\sigma}{A_k}}\\dist(x_0,X^*) + \\sigma\\left(1+\\frac{1}{\\sqrt{\\sigma A_k}}\\right)\\dist(x_0,X^*)\n\\\\&\\leq \\left(2\\sqrt{\\frac{M_{k-1}+\\sigma}{A_k}} + \\sigma\\right)\\dist(x_0,X^*).\n\\end{align*}\n\\end{proof}","sha256_utf8":"1e29d853eac547423c5a0495de63be216c0a8609ed4ae075ed88348700ab1d3f"}]},{"id":"S066","semantic_type":"lemma","kind":"lemma","title":"近端梯度单步下降与回溯界","source_units":[{"start":45668,"end":46347,"start_line":844,"end_line":855,"raw_tex":"\\begin{lemma}\\label{lem:PG-iter}\nLet $\\{T_{M_k}(x_k), M_k, L_{k+1}\\}$ be given by $\\PGIter(x_k,L_k)$. Then, the following assertions hold.\n\\begin{enumerate}[(i)]\n\\item $\\varphi(T_{M_k}(x_k))\\leq \\varphi(x_k)$ holds.\n\\item\nIf $L_k \\leq \\gamma_{\\rm inc}L_f$, then we have $M_k \\leq \\gamma_{\\rm inc}L_f$ and $L_{k+1} \\leq \\max\\{L_{\\min},$ $\\frac{\\gamma_{\\rm inc}}{\\gamma_{\\rm dec}}L_f\\} \\leq \\gamma_{\\rm inc}L_f$.\nMoreover, the number of executions of the loop {\\rm (Lines~\\ref{PG-iter-loop-s}--\\ref{PG-iter-loop-e})} is bounded by\n$$\n 1 + \\frac{\\log \\gamma_{\\rm dec}}{\\log \\gamma_{\\rm inc}} + \\frac{1}{\\log \\gamma_{\\rm inc}} \\log \\frac{L_{k+1}}{L_k}.\n$$\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"d5ab750bb5482b9e4a468eb0f35a1abd5cc76c9ed76aed61a6d8140c33564ad9"}]},{"id":"S067","semantic_type":"proof","kind":"proof","title":"近端梯度单步性质完整证明","source_units":[{"start":46348,"end":46654,"start_line":856,"end_line":861,"raw_tex":"\\begin{proof}\nSince the condition \\eqref{PG-iter-test} ensures\n$\\varphi(T_{M_k}(x_k))\\leq m_{M_k}(x_k;T_{M_k}(x_k))\\leq \\varphi(x_k)$\n(recall \\eqref{mL-leq-phi} for the second inequality), we conclude (i).\nThe assertion (ii) can be verified in the same way as Lemma~\\ref{lem:APG-iter-LS-compl}.\n\\end{proof}","sha256_utf8":"0aed7ff569a2ea7e7277f2b0f8288b65e36989559003cc6f51c9633e8605fc9a"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"验收条件与模型极小性保证下降","source_units":[{"start":46348,"end":46554,"start_line":856,"end_line":859,"raw_tex":"\\begin{proof}\nSince the condition \\eqref{PG-iter-test} ensures\n$\\varphi(T_{M_k}(x_k))\\leq m_{M_k}(x_k;T_{M_k}(x_k))\\leq \\varphi(x_k)$\n(recall \\eqref{mL-leq-phi} for the second inequality), we conclude (i).\n","sha256_utf8":"57e7462d89709774938c9b19d361217fef203a1ab1c7f774689b1584f8335c00"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"沿用回溯搜索次数论证","source_units":[{"start":46554,"end":46654,"start_line":860,"end_line":861,"raw_tex":"The assertion (ii) can be verified in the same way as Lemma~\\ref{lem:APG-iter-LS-compl}.\n\\end{proof}","sha256_utf8":"75150df6c4ed9ce44980cf323903ddd6f5c97f2897a153e7b83659ff48a85e60"}]},{"id":"S070","semantic_type":"lemma","kind":"lemma","title":"正则化外循环的终点估计","source_units":[{"start":49636,"end":50237,"start_line":918,"end_line":930,"raw_tex":"\\begin{lemma}\\label{lem:AdaAPG-loop}\nIn $\\AdaAPG$,\nsuppose that a $j$-th outer loop is completed satisfying the criterion $A_{k+1}\\geq \\frac{2(M_{k}+\\sigma_j)}{\\beta^2\\sigma_j^2}$ for some $k\\geq 0$. Then, we have the inequalities\n$$\n\\norm{x_{k+1}-x_0} \\leq \\left(1+\\frac{\\beta}{\\sqrt{2}}\\right)\\dist(x_0,X^*),\n\\qquad\n\\norm{g_{M_{k}}(x_{k+1})} \\leq (1+\\sqrt{2}\\beta)\\sigma_j\\dist(x_0,X^*).\n$$\nMoreover, the number of inner iterations is bounded as follows.\n$$\nk+1 \\leq 2+ \\left(\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_j}}+1\\right)\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_j}{\\beta\\sigma_j}.\n$$\n\\end{lemma}","sha256_utf8":"1d0d3483880e724983351a1a170800060e1d40fa8d26d9c5d07533fcc1b016ae"}]},{"id":"S071","semantic_type":"proof","kind":"proof","title":"正则化外循环估计完整证明","source_units":[{"start":50238,"end":51773,"start_line":931,"end_line":952,"raw_tex":"\\begin{proof}\nThe bounds on $\\norm{x_{k+1}-x_0}$ and $\\norm{g_{M_{k}}(x_{k+1})}$ can be obtained by Proposition~\\ref{prop:APG} (v), applying the assumption $A_{k+1}\\geq \\frac{2(M_{k}+\\sigma_j)}{\\beta^2\\sigma_j^2}$.\nTo show the bound on $k+1$, suppose $k\\geq 1$ (the result is clear when $k=0$).\nBy the assumption on $k$ and Proposition~\\ref{prop:APG}~(i), we have\n$$A_{k} < \\frac{2(M_{k-1}+\\sigma_j)}{\\beta^2\\sigma_j^2} \\leq \\frac{2(\\gamma_{\\rm inc}L_f+\\sigma_j)}{\\beta^2\\sigma_j^2}.$$\nOn the other hand, Proposition~\\ref{prop:APG} (ii) implies\n$$\nA_{k} \\geq  \\frac{2}{\\gamma_{\\rm inc}L_f}\\left[1+\\sqrt{\\frac{\\sigma_j}{2\\gamma_{\\rm inc}L_f}}\\right]^{2(k-1)} \\geq\n \\frac{2}{\\gamma_{\\rm inc}L_f}\\exp\\left(2(k-1) \\frac{1}{\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_j}}+1}\\right),\n$$\nwhere the second inequality is due to the fact\\footnote{\nIn fact, since the derivative $\\log(1+x)$ of the function $h(x):=(1+x)\\log(1+x)-x$ is increasing and vanishes at $x=0$, we have $\\min_{x>-1}h(x)=h(0)=0$.\n}\n$1+x \\geq \\exp\\frac{x}{1+x}~(\\forall x > -1)$.\nTherefore,\n\\begin{align*}\n\\frac{2}{\\gamma_{\\rm inc}L_f}\\exp\\left(2(k-1) \\frac{1}{\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_j}}+1}\\right) \\leq \\frac{2(\\gamma_{\\rm inc}L_f+\\sigma_j)}{\\beta^2\\sigma_j^2}\\\\\n\\Longrightarrow\nk+1 \\leq 2+ \\frac{1}{2}\\left(\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_j}}+1\\right)\\log\\frac{\\gamma_{\\rm inc}L_f(\\gamma_{\\rm inc}L_f+\\sigma_j)}{\\beta^2\\sigma_j^2}.\n\\end{align*}\nThe assertion follows by relaxing $\\gamma_{\\rm inc}L_f \\leq \\gamma_{\\rm inc}L_f + \\sigma_j$.\n\\end{proof}","sha256_utf8":"230c63b328385c8782c1a78e479a4de21d56d04f170d946ef9df73d187f729fd"}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","title":"增长停止条件控制终点距离与梯度","source_units":[{"start":50238,"end":50453,"start_line":931,"end_line":932,"raw_tex":"\\begin{proof}\nThe bounds on $\\norm{x_{k+1}-x_0}$ and $\\norm{g_{M_{k}}(x_{k+1})}$ can be obtained by Proposition~\\ref{prop:APG} (v), applying the assumption $A_{k+1}\\geq \\frac{2(M_{k}+\\sigma_j)}{\\beta^2\\sigma_j^2}$.\n","sha256_utf8":"35f15b9978a8f3d9e092388e05fabd4212d1dbd6e3ac6c841954bd14eb968fbb"}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","title":"夹逼停止前的加速权重","source_units":[{"start":50453,"end":51016,"start_line":933,"end_line":940,"raw_tex":"To show the bound on $k+1$, suppose $k\\geq 1$ (the result is clear when $k=0$).\nBy the assumption on $k$ and Proposition~\\ref{prop:APG}~(i), we have\n$$A_{k} < \\frac{2(M_{k-1}+\\sigma_j)}{\\beta^2\\sigma_j^2} \\leq \\frac{2(\\gamma_{\\rm inc}L_f+\\sigma_j)}{\\beta^2\\sigma_j^2}.$$\nOn the other hand, Proposition~\\ref{prop:APG} (ii) implies\n$$\nA_{k} \\geq  \\frac{2}{\\gamma_{\\rm inc}L_f}\\left[1+\\sqrt{\\frac{\\sigma_j}{2\\gamma_{\\rm inc}L_f}}\\right]^{2(k-1)} \\geq\n \\frac{2}{\\gamma_{\\rm inc}L_f}\\exp\\left(2(k-1) \\frac{1}{\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_j}}+1}\\right),\n$$\n","sha256_utf8":"fa2ad6c7e0e8fa3efda23a4e44e200097f81cf2a2e22516df3887d65246adde5"}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","title":"证明权重增长所用指数不等式","source_units":[{"start":51016,"end":51276,"start_line":941,"end_line":944,"raw_tex":"where the second inequality is due to the fact\\footnote{\nIn fact, since the derivative $\\log(1+x)$ of the function $h(x):=(1+x)\\log(1+x)-x$ is increasing and vanishes at $x=0$, we have $\\min_{x>-1}h(x)=h(0)=0$.\n}\n$1+x \\geq \\exp\\frac{x}{1+x}~(\\forall x > -1)$.\n","sha256_utf8":"25d94f34d8f39271a4182a4841fccfe22a186cbc38e98372cdf2d12bf393b4d4"}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","title":"解出外循环内迭代次数界","source_units":[{"start":51276,"end":51773,"start_line":945,"end_line":952,"raw_tex":"Therefore,\n\\begin{align*}\n\\frac{2}{\\gamma_{\\rm inc}L_f}\\exp\\left(2(k-1) \\frac{1}{\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_j}}+1}\\right) \\leq \\frac{2(\\gamma_{\\rm inc}L_f+\\sigma_j)}{\\beta^2\\sigma_j^2}\\\\\n\\Longrightarrow\nk+1 \\leq 2+ \\frac{1}{2}\\left(\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_j}}+1\\right)\\log\\frac{\\gamma_{\\rm inc}L_f(\\gamma_{\\rm inc}L_f+\\sigma_j)}{\\beta^2\\sigma_j^2}.\n\\end{align*}\nThe assertion follows by relaxing $\\gamma_{\\rm inc}L_f \\leq \\gamma_{\\rm inc}L_f + \\sigma_j$.\n\\end{proof}","sha256_utf8":"7c2ef44b88b315c98144d8295a8a5e438524c533c366230cbaa5cb52c45f0ce6"}]},{"id":"S076","semantic_type":"theorem","kind":"theorem","title":"自适应正则化终止与复杂度","source_units":[{"start":52203,"end":54284,"start_line":960,"end_line":1001,"raw_tex":"\\begin{theorem}\\label{thm:AdaAPG}\nIn $\\AdaAPG$, let $N$ be the total number of executions of $\\APGIter$.\nDenote\n\\begin{equation}\\label{sigma-th}\n\\sigma(x_0,\\ep):=\\frac{\\ep}{(1+\\sqrt{2}\\beta)\\dist(x_0,X^*)},\n\\end{equation}\nwhere we let $\\sigma(x_0,\\ep):=+\\infty$ when $x_0 \\in X^*$.\nThen, the following assertions hold.\n\\begin{enumerate}[(i)]\n\\item\nThe algorithm terminates at the $j$-th outer loop whenever\n$\\sigma_j \\leq \\sigma(x_0,\\ep)$.\nMoreover, we have ${\\sigma_j \\geq \\sigma(x_0,\\ep)/\\gamma_{\\rm reg}}$ for all $j>0$.\n%\\begin{equation}\\label{sigma-j-lb}\n%\\sigma_j \\geq \\sigma(x_0,\\ep)/\\gamma_{\\rm reg},\\quad \\forall j>0.\n%\\end{equation}\n\n\\item\nSuppose that the algorithm terminates at $j=\\ell$ for some $\\ell \\geq 0$.\nThen, $N$ is at most\n\\begin{align*}\n&\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\sigma_\\ell}} - \\sqrt{\\frac{1}{\\sigma_0}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell}+ \\left(1+\\log_{\\gamma_{\\rm reg}}\\frac{\\sigma_0}{\\sigma_\\ell}\\right)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell}\\right).\n\\end{align*}\n%\n\\item\nIf $\\sigma_0\\leq \\sigma(x_0,\\ep)$, then\n$\nN \\leq 2+ \\left(\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_0}}+1\\right)\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_0}{\\beta\\sigma_0}.\n$\n\n%\n\\item\nIf $\\sigma_0 \\geq \\sigma(x_0,\\ep)$, then $N$ is at most\n\\begin{align}\n&\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\gamma_{\\rm reg}\\sqrt{\\frac{1}{\\sigma(x_0,\\ep)}} - \\sqrt{\\frac{1}{\\sigma_0}} \\right]\\log\\left(\\frac{\\gamma_{\\rm reg}\\gamma_{\\rm inc}L_f}{\\beta\\sigma(x_0,\\ep)}+\\frac{1}{\\beta}\\right) \\nonumber\\\\\n&+ \\left(2+\\log_{\\gamma_{\\rm reg}}\\frac{ \\sigma_0}{\\sigma(x_0,\\ep)}\\right)\\left[2+\\log\\left(\\frac{\\gamma_{\\rm reg}\\gamma_{\\rm inc}L_f}{\\beta\\sigma(x_0,\\ep)}+\\frac{1}{\\beta}\\right)\\right] \\nonumber\\\\\n=~&\nO\\left(\\sqrt{\\frac{L_f\\dist(x_0,X^*)}{\\ep}}\\log \\frac{L_f\\dist(x_0,X^*)}{\\ep} + \\log \\frac{\\sigma_0 \\dist(x_0,X^*)}{\\ep} \\log \\frac{L_f\\dist(x_0,X^*)}{\\ep}\\right).\n\\label{AdaAPG-compl-O}\n\\end{align}\n\\end{enumerate}\n\\end{theorem}","sha256_utf8":"bc393412150ef464fe2a5853799bf7f6df4eb486bc084dbbf864e0b54857eb5d"}]},{"id":"S077","semantic_type":"proof","kind":"proof","title":"自适应正则化定理完整证明","source_units":[{"start":54286,"end":56888,"start_line":1003,"end_line":1055,"raw_tex":"\\begin{proof}\n(i)\n\\sloppy % prevents overflow\nBy Lemma~\\ref{lem:AdaAPG-loop}, the algorithm must terminate at the $j$-th loop whenever we have\n$\n{(1+\\sqrt{2}\\beta)\\sigma_j\\dist(x_0,X^*) \\leq \\ep}\n$,\nor equivalently,\n$\n\\sigma_j \\leq \\frac{\\ep}{(1+\\sqrt{2}\\beta)\\dist(x_0,X^*)}=\\sigma(x_0,\\ep).\n$\nThus, we obtain the former assertion of (i).\nTo see the latter assertion,\nassume that $\\sigma_j < \\sigma(x_0,\\ep)/\\gamma_{\\rm reg}$ holds for some $j > 0$. Then, the previous $(j-1)$-th loop satisfies $\\sigma_{j-1}< \\sigma(x_0,\\ep)$ so that the $j$-th loop is not executed by the former assertion.\nThis verifies the latter assertion of (i).\n\n(ii) Since $\\sigma_\\ell=\\sigma_0/\\gamma_{\\rm reg}^\\ell$, we have $\\ell = \\log_{\\gamma_{\\rm reg}}\\sigma_0/\\sigma_\\ell$.\nUsing Lemma~\\ref{lem:AdaAPG-loop}, $N$ is bounded as follows.\n\\begin{align*}\nN\n&\\leq\n\\sum_{j=0}^\\ell \\left[\n2+ \\left(\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_j}}+1\\right)\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_j}{\\beta\\sigma_j}\n\\right]\\\\\n&\\leq 2(\\ell+1) + \\sqrt{2\\gamma_{\\rm inc}L_f}\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell}\\sum_{j=0}^\\ell \\sqrt{\\frac{1}{\\sigma_j}}\n+ (\\ell+1)\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell},\n\\end{align*}\nwhere the second inequality is due to $\\sigma_j \\geq \\sigma_\\ell~(\\forall j \\leq \\ell)$.\nNote that\n\\begin{align*}\n\\sum_{j=0}^\\ell \\sqrt{\\frac{1}{\\sigma_j}}\n&=\n\\sum_{j=0}^\\ell \\sqrt{\\frac{1}{\\sigma_0/\\gamma_{\\rm reg}^j}}\n=\n\\sqrt{\\frac{1}{\\sigma_0}} \\sum_{j=0}^\\ell \\sqrt{\\gamma_{\\rm reg}}^{j}\n=\n\\sqrt{\\frac{1}{\\sigma_0}}\\frac{\\sqrt{\\gamma_{\\rm reg}}^{\\ell+1} -1}{\\sqrt{\\gamma_{\\rm reg}} -1}\\\\\n&=\n\\sqrt{\\frac{1}{\\sigma_0}}\\frac{\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\sigma_0/\\sigma_\\ell} -1}{\\sqrt{\\gamma_{\\rm reg}} -1}\n=\n\\frac{1}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\sigma_\\ell}} - \\sqrt{\\frac{1}{\\sigma_0}}\\right].\n\\end{align*}\nTherefore, we see that\n$$\nN\\leq (\\ell+1)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell}\\right)\n+\n\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\sigma_\\ell}} - \\sqrt{\\frac{1}{\\sigma_0}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell}.\n$$\nThe assertion follows by substituting $\\ell = \\log_{\\gamma_{\\rm reg}}{\\sigma_0}/{\\sigma_\\ell}$.\n\nIn view of (i), the assertions (iii) and (iv) follow by applying (ii) with $\\sigma_\\ell = \\sigma_0$ and $\\sigma_\\ell \\geq \\sigma(x_0,\\ep)/\\gamma_{\\rm reg}$, respectively.\nThe big-$O$ expression \\eqref{AdaAPG-compl-O} is obtained by substituting the definition of $\\sigma(x_0,\\ep)$.\n\\end{proof}","sha256_utf8":"d42107834263c41936376ba77fcb9d9f5c8746d899cb7fda74845945069af274"}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","title":"阈值保证终止并限制参数下降","source_units":[{"start":54286,"end":54923,"start_line":1003,"end_line":1018,"raw_tex":"\\begin{proof}\n(i)\n\\sloppy % prevents overflow\nBy Lemma~\\ref{lem:AdaAPG-loop}, the algorithm must terminate at the $j$-th loop whenever we have\n$\n{(1+\\sqrt{2}\\beta)\\sigma_j\\dist(x_0,X^*) \\leq \\ep}\n$,\nor equivalently,\n$\n\\sigma_j \\leq \\frac{\\ep}{(1+\\sqrt{2}\\beta)\\dist(x_0,X^*)}=\\sigma(x_0,\\ep).\n$\nThus, we obtain the former assertion of (i).\nTo see the latter assertion,\nassume that $\\sigma_j < \\sigma(x_0,\\ep)/\\gamma_{\\rm reg}$ holds for some $j > 0$. Then, the previous $(j-1)$-th loop satisfies $\\sigma_{j-1}< \\sigma(x_0,\\ep)$ so that the $j$-th loop is not executed by the former assertion.\nThis verifies the latter assertion of (i).\n\n","sha256_utf8":"71a452c4461cd99dc35eeb679129e1c55a4004d5cccdafc5d9223961f6e1b781"}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","title":"累加各正则化外循环代价","source_units":[{"start":54923,"end":55598,"start_line":1019,"end_line":1030,"raw_tex":"(ii) Since $\\sigma_\\ell=\\sigma_0/\\gamma_{\\rm reg}^\\ell$, we have $\\ell = \\log_{\\gamma_{\\rm reg}}\\sigma_0/\\sigma_\\ell$.\nUsing Lemma~\\ref{lem:AdaAPG-loop}, $N$ is bounded as follows.\n\\begin{align*}\nN\n&\\leq\n\\sum_{j=0}^\\ell \\left[\n2+ \\left(\\sqrt{\\frac{2\\gamma_{\\rm inc}L_f}{\\sigma_j}}+1\\right)\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_j}{\\beta\\sigma_j}\n\\right]\\\\\n&\\leq 2(\\ell+1) + \\sqrt{2\\gamma_{\\rm inc}L_f}\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell}\\sum_{j=0}^\\ell \\sqrt{\\frac{1}{\\sigma_j}}\n+ (\\ell+1)\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell},\n\\end{align*}\nwhere the second inequality is due to $\\sigma_j \\geq \\sigma_\\ell~(\\forall j \\leq \\ell)$.\n","sha256_utf8":"d2ba265a0aabf5f25c81caafdada50a205305cd06a1c0ac6e700cbc3284cb513"}]},{"id":"S080","semantic_type":"proof_step","kind":"proof_step","title":"计算倒平方根参数几何和","source_units":[{"start":55598,"end":56162,"start_line":1031,"end_line":1044,"raw_tex":"Note that\n\\begin{align*}\n\\sum_{j=0}^\\ell \\sqrt{\\frac{1}{\\sigma_j}}\n&=\n\\sum_{j=0}^\\ell \\sqrt{\\frac{1}{\\sigma_0/\\gamma_{\\rm reg}^j}}\n=\n\\sqrt{\\frac{1}{\\sigma_0}} \\sum_{j=0}^\\ell \\sqrt{\\gamma_{\\rm reg}}^{j}\n=\n\\sqrt{\\frac{1}{\\sigma_0}}\\frac{\\sqrt{\\gamma_{\\rm reg}}^{\\ell+1} -1}{\\sqrt{\\gamma_{\\rm reg}} -1}\\\\\n&=\n\\sqrt{\\frac{1}{\\sigma_0}}\\frac{\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\sigma_0/\\sigma_\\ell} -1}{\\sqrt{\\gamma_{\\rm reg}} -1}\n=\n\\frac{1}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\sigma_\\ell}} - \\sqrt{\\frac{1}{\\sigma_0}}\\right].\n\\end{align*}\n","sha256_utf8":"7370581b9ef13c5ec23c3e3428ea29c219f47793960c752f1a3e4ab622ca480e"}]},{"id":"S081","semantic_type":"proof_step","kind":"proof_step","title":"得到依赖终止参数的累计界","source_units":[{"start":56162,"end":56594,"start_line":1045,"end_line":1051,"raw_tex":"Therefore, we see that\n$$\nN\\leq (\\ell+1)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell}\\right)\n+\n\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\sigma_\\ell}} - \\sqrt{\\frac{1}{\\sigma_0}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_\\ell}{\\beta\\sigma_\\ell}.\n$$\nThe assertion follows by substituting $\\ell = \\log_{\\gamma_{\\rm reg}}{\\sigma_0}/{\\sigma_\\ell}$.\n","sha256_utf8":"81618380e2c0f5652e2bff37bb330e7084dd8e913bb20d27f8d593df20956501"}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","title":"代入终止阈值获得显式复杂度","source_units":[{"start":56594,"end":56888,"start_line":1052,"end_line":1055,"raw_tex":"\nIn view of (i), the assertions (iii) and (iv) follow by applying (ii) with $\\sigma_\\ell = \\sigma_0$ and $\\sigma_\\ell \\geq \\sigma(x_0,\\ep)/\\gamma_{\\rm reg}$, respectively.\nThe big-$O$ expression \\eqref{AdaAPG-compl-O} is obtained by substituting the definition of $\\sigma(x_0,\\ep)$.\n\\end{proof}","sha256_utf8":"0d4a998934b50f78e0ee20af58e529943a2b204a9db7984b611aaa772fae7202"}]},{"id":"S083","semantic_type":"lemma","kind":"lemma","title":"重启序列下降与误差界估计","source_units":[{"start":73430,"end":74460,"start_line":1369,"end_line":1384,"raw_tex":"\\begin{lemma}\\label{lem:rAdaAPG-base}\nAssume that the H\\\"olderian error bound condition \\eqref{HEB-review} holds.\nIn the execution of $\\rAdaAPG$, the following assertions hold.\n\\begin{enumerate}[(i)]\n\\item $\\outerL{t},\\outerM{t} \\in [L_{\\min},\\gamma_{\\rm inc}L_f]$ for all $t\\geq 0$.\n\\item $\\varphi(\\outerXplus{t+1}) \\leq \\varphi(\\outerXplus{t}) \\leq \\cdots \\leq \\varphi(\\outerXplus{0}) \\leq \\varphi(\\xinit)$ for all $t\\geq 0$.\n\\item $t \\leq \\log_{1/\\theta}\\dfrac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\norm{g_{\\outerM{t}}(\\outerXinit{t})}}$ for each $t\\geq 0$.\n\\item Whenever $\\outerXplus{t} \\not\\in X^*$, we have\n\\begin{equation}\\label{rAdaAPG-dist-bound}\n\\dist(\\outerXplus{t},X^*)^{\\rho-1} \\leq \\frac{1}{\\kappa}\\left(\\frac{L_f}{L_{\\min}}+1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})},\n\\end{equation}\n\\begin{equation}\\label{rAdaAPG-obj-bound}\n(\\varphi(\\outerXplus{t})-\\varphi^*)^{\\rho-1} \\leq \\frac{1}{\\kappa}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^\\rho\\norm{g_{\\outerM{t}}(\\outerXinit{t})}^\\rho.\n\\end{equation}\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"8e0962380ffac61e9186ef181af899b035124d047d11e94123fb7633cd8dfffe"}]},{"id":"S084","semantic_type":"proof","kind":"proof","title":"重启序列基本性质完整证明","source_units":[{"start":74461,"end":76421,"start_line":1385,"end_line":1422,"raw_tex":"\\begin{proof}\n(i)\nSince $\\Linit \\in [L_{\\min}, \\gamma_{\\rm inc}L_f]$, it follows $\\outerL{0},\\outerM{0}\\in [L_{\\min}, \\gamma_{\\rm inc}L_f]$ by Lemma~\\ref{lem:PG-iter}~(ii).\nThen, using Proposition~\\ref{prop:APG} (i) inductively, we obtain (i).\n\n(ii)\nFor $t=0$, we have $\\varphi(\\outerXplus{0})=\\varphi(T_{\\outerM{0}}(\\xinit))\\leq \\varphi(\\xinit)$ by Lemma~\\ref{lem:PG-iter} (i).\nMoreover, the criterion \\eqref{APG-iter-test-c} and Proposition~\\ref{prop:APG}~(iv), applied to the subroutine $\\AdaAPG$ in $\\rAdaAPG$, yield\n${\\varphi(\\outerXplus{t+1}) \\leq \\varphi(\\outerXinit{t+1})}$ and $\\varphi(\\outerXinit{t+1}) \\leq \\varphi(\\outerXplus{t})$, respectively. This shows $\\varphi(\\outerXplus{t+1}) \\leq \\varphi(\\outerXplus{t})$.\n\n\n(iii)\nSince the recurrence\n$\n\\ep^{(t)}/\\theta=\\norm{g_{\\outerM{t}}(\\outerXinit{t})} \\leq \\ep^{(t-1)}\n$\nholds for $t\\geq 1$, we have\n$\n\\ep^{(0)} \\geq \\ep^{(t)}/\\theta^t\n$\nfor each $t\\geq 0$.\nTherefore, we conclude\n$\nt \\leq \\log_{1/\\theta}\\frac{\\ep^{(0)}}{\\ep^{(t)}} = \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\norm{g_{\\outerM{t}}(\\outerXinit{t})}}.\n$\n\n\n(iv)\nSince $\\outerXplus{t}=T_{\\outerM{t}}(\\outerXinit{t})$, Lemma~\\ref{lem:gmap} (iii) implies that $g_t:=\\nabla{f}(\\outerXplus{t})-\\nabla{f}(\\outerXinit{t})+g_{\\outerM{t}}(\\outerXinit{t})$ belongs to $\\partial{\\varphi}(\\outerXplus{t})$.\nThen,\nLemma~\\ref{lem:HEB} shows (note that $\\outerXplus{t}$ belongs to $\\lev_\\varphi(\\varphi(\\xinit))\\setminus X^*$ by (ii))\n\\begin{align*}\n\\kappa\\dist(\\outerXplus{t},X^*)^{\\rho-1} &\\leq \\norm{g_t}\n\\leq \\left(\\frac{L_f}{\\outerM{t}}+1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})}\n\\leq \\left(\\frac{L_f}{L_{\\min}}+1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})},\n\\end{align*}\nwhere the second inequality is due to Lemma~\\ref{lem:gmap} (iii) and the last follows by (i). This shows the assertion \\eqref{rAdaAPG-dist-bound}. Similarly, \\eqref{rAdaAPG-obj-bound} can be obtained using the latter inequality of \\eqref{Loj-ineq}.\n\\end{proof}","sha256_utf8":"ca560d103b7e5d95e26da3bd452b690e0b4037fcefab1ff8f9ac85e5fee5cd24"}]},{"id":"S085","semantic_type":"proof_step","kind":"proof_step","title":"归纳控制重启回溯参数","source_units":[{"start":74461,"end":74706,"start_line":1385,"end_line":1389,"raw_tex":"\\begin{proof}\n(i)\nSince $\\Linit \\in [L_{\\min}, \\gamma_{\\rm inc}L_f]$, it follows $\\outerL{0},\\outerM{0}\\in [L_{\\min}, \\gamma_{\\rm inc}L_f]$ by Lemma~\\ref{lem:PG-iter}~(ii).\nThen, using Proposition~\\ref{prop:APG} (i) inductively, we obtain (i).\n\n","sha256_utf8":"1bee3171da2562ebe49d88468ed2a0e857926e076cc265b42ac4c9105c24e2d3"}]},{"id":"S086","semantic_type":"proof_step","kind":"proof_step","title":"保持重启近端点位于初始水平集","source_units":[{"start":74706,"end":75189,"start_line":1390,"end_line":1394,"raw_tex":"(ii)\nFor $t=0$, we have $\\varphi(\\outerXplus{0})=\\varphi(T_{\\outerM{0}}(\\xinit))\\leq \\varphi(\\xinit)$ by Lemma~\\ref{lem:PG-iter} (i).\nMoreover, the criterion \\eqref{APG-iter-test-c} and Proposition~\\ref{prop:APG}~(iv), applied to the subroutine $\\AdaAPG$ in $\\rAdaAPG$, yield\n${\\varphi(\\outerXplus{t+1}) \\leq \\varphi(\\outerXinit{t+1})}$ and $\\varphi(\\outerXinit{t+1}) \\leq \\varphi(\\outerXplus{t})$, respectively. This shows $\\varphi(\\outerXplus{t+1}) \\leq \\varphi(\\outerXplus{t})$.\n\n","sha256_utf8":"05cb5db492f84b7f90e4e4b6b28206a59778f2436065af1622ee1c7d54364ac5"}]},{"id":"S087","semantic_type":"proof_step","kind":"proof_step","title":"残差收缩控制重启次数","source_units":[{"start":75189,"end":75553,"start_line":1395,"end_line":1411,"raw_tex":"\n(iii)\nSince the recurrence\n$\n\\ep^{(t)}/\\theta=\\norm{g_{\\outerM{t}}(\\outerXinit{t})} \\leq \\ep^{(t-1)}\n$\nholds for $t\\geq 1$, we have\n$\n\\ep^{(0)} \\geq \\ep^{(t)}/\\theta^t\n$\nfor each $t\\geq 0$.\nTherefore, we conclude\n$\nt \\leq \\log_{1/\\theta}\\frac{\\ep^{(0)}}{\\ep^{(t)}} = \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\norm{g_{\\outerM{t}}(\\outerXinit{t})}}.\n$\n\n\n","sha256_utf8":"6cc8e005366e89ed22171386d55f327b5720949bb4facb6e2a2462c9eac478d0"}]},{"id":"S088","semantic_type":"proof_step","kind":"proof_step","title":"次梯度下界控制距离与目标残差","source_units":[{"start":75553,"end":76421,"start_line":1412,"end_line":1422,"raw_tex":"(iv)\nSince $\\outerXplus{t}=T_{\\outerM{t}}(\\outerXinit{t})$, Lemma~\\ref{lem:gmap} (iii) implies that $g_t:=\\nabla{f}(\\outerXplus{t})-\\nabla{f}(\\outerXinit{t})+g_{\\outerM{t}}(\\outerXinit{t})$ belongs to $\\partial{\\varphi}(\\outerXplus{t})$.\nThen,\nLemma~\\ref{lem:HEB} shows (note that $\\outerXplus{t}$ belongs to $\\lev_\\varphi(\\varphi(\\xinit))\\setminus X^*$ by (ii))\n\\begin{align*}\n\\kappa\\dist(\\outerXplus{t},X^*)^{\\rho-1} &\\leq \\norm{g_t}\n\\leq \\left(\\frac{L_f}{\\outerM{t}}+1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})}\n\\leq \\left(\\frac{L_f}{L_{\\min}}+1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})},\n\\end{align*}\nwhere the second inequality is due to Lemma~\\ref{lem:gmap} (iii) and the last follows by (i). This shows the assertion \\eqref{rAdaAPG-dist-bound}. Similarly, \\eqref{rAdaAPG-obj-bound} can be obtained using the latter inequality of \\eqref{Loj-ineq}.\n\\end{proof}","sha256_utf8":"980f902a79c1bb156cc22618cdc39e8a66edd98e7bd439a9b671557044a36bfc"}]},{"id":"S089","semantic_type":"lemma","kind":"lemma","title":"重启累计复杂度归约","source_units":[{"start":76512,"end":77226,"start_line":1426,"end_line":1445,"raw_tex":"\\begin{lemma}\\label{lem:rAdaAPG-compl}\nLet $N^{(t)}$ be the number of executions of $\\APGIter$ at the $t$-th outer loop of $\\rAdaAPG$.\nAssume that, for some $T_*\\geq 0$, $\\sigma_*>0$, and $\\ep_*>0$, we have\n$$\n\\outersigma{t+1}\\geq \\sigma_*\\quad (t=0,\\ldots,T_*)\n\\quad\n\\text{and}\n\\quad\n\\norm{g_{\\outerM{T_*}}(\\outerXinit{T_*})} \\geq \\ep_*.\n$$\nThen, under the definition \\eqref{N-def} of $N(\\cdot,\\cdot,\\cdot)$, the following inequality holds.\n$$\n\\sum_{t=0}^{T_*}N^{(t)} \\leq N(\\ep_*,\\sigma_*,C) \\leq N(\\ep_*,\\sigma_*,C_*),\n$$\nwhere\n$$\nC := \\sum_{t=0}^{T_*}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n\\leq C_*:= \\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*}\\right).\n$$\n\\end{lemma}","sha256_utf8":"72be9b0d4bd77ce8d8472f319fc8c41feae7be3144d330ec907ed8d7635e1e19"}]},{"id":"S090","semantic_type":"proof","kind":"proof","title":"累计复杂度归约完整证明","source_units":[{"start":77227,"end":80304,"start_line":1446,"end_line":1497,"raw_tex":"\\begin{proof}\nSince $\\outersigma{t+1}\\geq \\sigma_*$ for each $t=0,\\ldots,T_*$, Theorem~\\ref{thm:AdaAPG} (ii) gives the following bound for $t=0,\\ldots,T_*$:\n\\begin{align*}\nN^{(t)}\n&\\leq \n\\left(1+\\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{t}}{\\outersigma{t+1}}\\right)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\outersigma{t+1}}{\\beta\\outersigma{t+1}}\\right)\\\\\n&\\quad +\n\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\outersigma{t+1}}} - \\sqrt{\\frac{1}{\\outersigma{t}}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\outersigma{t+1}}{\\beta\\outersigma{t+1}}\\\\\n&\\leq\n\\left(1+\\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{t}}{\\outersigma{t+1}}\\right)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}\\right)\\\\\n&\\quad +\n\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\outersigma{t+1}}} - \\sqrt{\\frac{1}{\\outersigma{t}}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}.\n\\end{align*}\nBy Lemma~\\ref{lem:rAdaAPG-base} (iii), $T_*$ is bounded by\n$$\nT_*\\leq \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\norm{g_{\\outerM{T_*}}(\\outerXinit{T_*})}} \\leq \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*}.\n$$\nTo estimate the sum $\\sum_{t=0}^{T_*}N^{(t)}$, note that\n$$\n\\sum_{t=0}^{T_*}\\left(1+\\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{t}}{\\outersigma{t+1}}\\right) = 1+T_*+\\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{0}}{\\outersigma{T_*+1}} \\leq 1+ \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*} + \\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{0}}{\\sigma_*},\n$$\nand\n\\begin{align*}\n\\sum_{t=0}^{T_*}\n\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\outersigma{t+1}}} - \\sqrt{\\frac{1}{\\outersigma{t}}}\\right]\n&=\n(\\sqrt{\\gamma_{\\rm reg}}-1)\\sum_{t=0}^{T_*}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n+\\sum_{t=0}^{T_*}\n\\left[\\sqrt{\\frac{1}{\\outersigma{t+1}}} - \\sqrt{\\frac{1}{\\outersigma{t}}}\\right]\\\\\n&=\n(\\sqrt{\\gamma_{\\rm reg}}-1)\\sum_{t=0}^{T_*}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n+\n\\sqrt{\\frac{1}{\\outersigma{T_*+1}}} - \\sqrt{\\frac{1}{\\outersigma{0}}}.\n\\end{align*}\nTherefore, $\\sum_{t=0}^{T_*}N^{(t)}$ is bounded by\n\\begin{align*}\n\\sum_{t=0}^{T_*}N^{(t)}\n&\\leq \\left(1+ \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*} + \\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{0}}{\\sigma_*}\\right)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}\\right)\\\\\n&\\quad +\n\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\n\\sqrt{\\frac{1}{\\sigma_*}} - \\sqrt{\\frac{1}{\\outersigma{0}}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*} \\\\\n&\\quad + \\sqrt{2\\gamma_{\\rm inc}L_f} \\sum_{t=0}^{T_*}\\sqrt{\\frac{1}{\\outersigma{t+1}}} \\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}\\\\\n&=N(\\ep_*,\\sigma_*,C).\n\\end{align*}\nFinally, $C$ has the following bound:\n$$\nC = \\sum_{t=0}^{T_*}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n\\leq \\sqrt{\\frac{1}{\\sigma_*}} (1+T_*)\\leq\n\\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*}\\right)=C_*,\n$$\nwhich also concludes $N(\\ep_*,\\sigma_*,C) \\leq N(\\ep_*,\\sigma_*,C_*)$.\n\\end{proof}","sha256_utf8":"5ee41c48d7e97bb135a5840143f9db557a083e734e77b06eeeaae26884ae16f7"}]},{"id":"S091","semantic_type":"proof_step","kind":"proof_step","title":"统一上界每次重启的内迭代代价","source_units":[{"start":77227,"end":78222,"start_line":1446,"end_line":1458,"raw_tex":"\\begin{proof}\nSince $\\outersigma{t+1}\\geq \\sigma_*$ for each $t=0,\\ldots,T_*$, Theorem~\\ref{thm:AdaAPG} (ii) gives the following bound for $t=0,\\ldots,T_*$:\n\\begin{align*}\nN^{(t)}\n&\\leq \n\\left(1+\\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{t}}{\\outersigma{t+1}}\\right)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\outersigma{t+1}}{\\beta\\outersigma{t+1}}\\right)\\\\\n&\\quad +\n\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\outersigma{t+1}}} - \\sqrt{\\frac{1}{\\outersigma{t}}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\outersigma{t+1}}{\\beta\\outersigma{t+1}}\\\\\n&\\leq\n\\left(1+\\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{t}}{\\outersigma{t+1}}\\right)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}\\right)\\\\\n&\\quad +\n\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\outersigma{t+1}}} - \\sqrt{\\frac{1}{\\outersigma{t}}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}.\n\\end{align*}\n","sha256_utf8":"8283890f33a924232e0ede212609e2bdcafba5da69a225121afb2dbc76d836d7"}]},{"id":"S092","semantic_type":"proof_step","kind":"proof_step","title":"由残差下界限制重启次数","source_units":[{"start":78222,"end":78455,"start_line":1459,"end_line":1462,"raw_tex":"By Lemma~\\ref{lem:rAdaAPG-base} (iii), $T_*$ is bounded by\n$$\nT_*\\leq \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\norm{g_{\\outerM{T_*}}(\\outerXinit{T_*})}} \\leq \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*}.\n$$\n","sha256_utf8":"13d57b8eb5dd092ae12dc473d486ea3af43524044eeb61c487dbeae227d3e4e6"}]},{"id":"S093","semantic_type":"proof_step","kind":"proof_step","title":"望远镜求和分离参数变化代价","source_units":[{"start":78455,"end":79297,"start_line":1463,"end_line":1479,"raw_tex":"To estimate the sum $\\sum_{t=0}^{T_*}N^{(t)}$, note that\n$$\n\\sum_{t=0}^{T_*}\\left(1+\\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{t}}{\\outersigma{t+1}}\\right) = 1+T_*+\\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{0}}{\\outersigma{T_*+1}} \\leq 1+ \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*} + \\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{0}}{\\sigma_*},\n$$\nand\n\\begin{align*}\n\\sum_{t=0}^{T_*}\n\\left[\\sqrt{\\gamma_{\\rm reg}}\\sqrt{\\frac{1}{\\outersigma{t+1}}} - \\sqrt{\\frac{1}{\\outersigma{t}}}\\right]\n&=\n(\\sqrt{\\gamma_{\\rm reg}}-1)\\sum_{t=0}^{T_*}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n+\\sum_{t=0}^{T_*}\n\\left[\\sqrt{\\frac{1}{\\outersigma{t+1}}} - \\sqrt{\\frac{1}{\\outersigma{t}}}\\right]\\\\\n&=\n(\\sqrt{\\gamma_{\\rm reg}}-1)\\sum_{t=0}^{T_*}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n+\n\\sqrt{\\frac{1}{\\outersigma{T_*+1}}} - \\sqrt{\\frac{1}{\\outersigma{0}}}.\n\\end{align*}\n","sha256_utf8":"c8b54ec4b8daca3a573af539c4668bc331b6f197cea2111dd6183cc4a417c40b"}]},{"id":"S094","semantic_type":"proof_step","kind":"proof_step","title":"合并为统一复杂度函数","source_units":[{"start":79297,"end":79977,"start_line":1480,"end_line":1489,"raw_tex":"Therefore, $\\sum_{t=0}^{T_*}N^{(t)}$ is bounded by\n\\begin{align*}\n\\sum_{t=0}^{T_*}N^{(t)}\n&\\leq \\left(1+ \\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*} + \\log_{\\gamma_{\\rm reg}}\\frac{\\outersigma{0}}{\\sigma_*}\\right)\\left(2+\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}\\right)\\\\\n&\\quad +\n\\frac{\\sqrt{2\\gamma_{\\rm inc}L_f}}{\\sqrt{\\gamma_{\\rm reg}}-1}\\left[\n\\sqrt{\\frac{1}{\\sigma_*}} - \\sqrt{\\frac{1}{\\outersigma{0}}}\\right]\\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*} \\\\\n&\\quad + \\sqrt{2\\gamma_{\\rm inc}L_f} \\sum_{t=0}^{T_*}\\sqrt{\\frac{1}{\\outersigma{t+1}}} \\log\\frac{\\gamma_{\\rm inc}L_f+\\sigma_*}{\\beta\\sigma_*}\\\\\n&=N(\\ep_*,\\sigma_*,C).\n\\end{align*}\n","sha256_utf8":"dc1a8a744ceb3c08ab5ce855c3e8c06d6e08440ebc852b3ff1a0c406840ba63f"}]},{"id":"S095","semantic_type":"proof_step","kind":"proof_step","title":"以统一参数下界控制剩余和","source_units":[{"start":79977,"end":80304,"start_line":1490,"end_line":1497,"raw_tex":"Finally, $C$ has the following bound:\n$$\nC = \\sum_{t=0}^{T_*}\\sqrt{\\frac{1}{\\outersigma{t+1}}}\n\\leq \\sqrt{\\frac{1}{\\sigma_*}} (1+T_*)\\leq\n\\sqrt{\\frac{1}{\\sigma_*}}\\left(1+\\log_{1/\\theta}\\frac{\\norm{g_{\\outerM{0}}(\\xinit)}}{\\ep_*}\\right)=C_*,\n$$\nwhich also concludes $N(\\ep_*,\\sigma_*,C) \\leq N(\\ep_*,\\sigma_*,C_*)$.\n\\end{proof}","sha256_utf8":"ca8c6bc7d5f4e08e44e5ece62149e2aa94dc4e31e1a917de02cfea97598d8339"}]},{"id":"S096","semantic_type":"lemma","kind":"lemma","title":"重启正则化参数统一下界","source_units":[{"start":80429,"end":81852,"start_line":1502,"end_line":1543,"raw_tex":"\\begin{lemma}\\label{lem:rAdaAPG-sigma}\nAssume that the H\\\"olderian error bound condition \\eqref{HEB-review} holds.\nSuppose that $\\rAdaAPG$ terminated with the stopping criterion at the $(T+1)$-th outer loop for some $T\\geq 0$.\nLet $\\bar\\sigma$ be defined by \\eqref{bar-sigma-def}.\n\\begin{enumerate}[(i)]\n\\item\nFor each $t=0,\\ldots,T$, we have\n$$\n\\outersigma{t+1}\n\\left\\{\n\\begin{array}{ll}\n=\\outersigma{t}&(\\outersigma{t}\\leq \\sigma(\\outerXplus{t},\\ep^{(t)})),\\\\\n\\geq\\sigma(\\outerXplus{t},\\ep^{(t)})/\\gamma_{\\rm reg}&(\\text{otherwise}),\n\\end{array}\n\\right.\n$$\nwhere $\\sigma(\\cdot,\\cdot)$ is defined by \\eqref{sigma-th}. \n\n\\item It follows that $\\sigma(\\outerXplus{t},\\ep^{(t)}) \\geq \\bar\\sigma$ for all $t=0,\\ldots,T$.\n%\\begin{equation}\\label{sigma-th-const}\n%\\sigma(\\outerXplus{t},\\ep^{(t)}) \\geq \\bar\\sigma, \\quad t=0,\\ldots,T.\n%\\end{equation}\nWhen $\\rho \\geq 2$, we further obtain\n\\begin{equation}\\label{sigma-th-t}\n\\sigma(\\outerXplus{t},\\ep^{(t)}) \\geq \\frac{\\theta^{\\frac{1}{\\rho-1}}}{1+\\sqrt{2}\\beta}\\cdot\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}}\n(\\ep^{(t)})^{\\frac{\\rho-2}{\\rho-1}}\n \\geq \\bar\\sigma,\n\\end{equation}\nfor each $t=0,\\ldots,T$.\n\\item\nFor each $t=0,\\ldots,T+1$, we obtain\n$$\n\\outersigma{t} \\left\\{\n\\begin{array}{ll}\n=\\sigmainit & (\\sigmainit \\leq \\bar\\sigma),\\\\\n\\geq\\bar\\sigma/\\gamma_{\\rm reg} & (\\text{otherwise}).\n\\end{array}\n\\right.\n$$\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"6ab60fbff256977bc00ff44516593c8827fe6d8ee241805720a460070261d520"}]},{"id":"S097","semantic_type":"proof","kind":"proof","title":"正则化参数统一下界完整证明","source_units":[{"start":81854,"end":85603,"start_line":1545,"end_line":1598,"raw_tex":"\\begin{proof}\n(i)\nIn the case $\\outersigma{t} \\leq \\sigma(\\outerXplus{t},\\ep^{(t)})$, Theorem~\\ref{thm:AdaAPG} (i) implies that the subroutine $\\AdaAPG$ at the $t$-th outer loop must terminate at the first loop $j=0$ so that $\\outersigma{t+1}$ is defined by $\\outersigma{t+1}=\\outersigma{t}$.\nOn the other hand, consider the case $\\outersigma{t} > \\sigma(\\outerXplus{t},\\ep^{(t)})$.\nWhen the subroutine $\\AdaAPG$ at the $t$-th outer loop terminates at the first loop $j=0$, then it is clear that\n$\\outersigma{t+1}=\\outersigma{t} > \\sigma(\\outerXplus{t},\\ep^{(t)}) \\geq \\sigma(\\outerXplus{t},\\ep^{(t)})/\\gamma_{\\rm reg}.$\nIn the other case, the latter assertion of Theorem~\\ref{thm:AdaAPG}~(i) implies $\\outersigma{t+1}\\geq \\sigma(\\outerXplus{t},\\ep^{(t)})/\\gamma_{\\rm reg}.$\n\n(ii)\nWe may assume $\\outerXplus{t} \\not\\in X^*$, which allows us to apply Lemma~\\ref{lem:rAdaAPG-base} (iv) (Note that the assertion is clear if $\\outerXplus{t} \\in X^*$ since then $\\sigma(\\outerXplus{t},\\ep^{(t)})=+\\infty$).\nIn the case $\\rho\\geq 2$, using \\eqref{rAdaAPG-dist-bound} implies\n$$\n\\sigma(\\outerXplus{t},\\ep^{(t)})\n= \\frac{\\theta\\norm{g_{\\outerM{t}}(\\outerXinit{t})}}{(1+\\sqrt{2}\\beta)\\dist(\\outerXplus{t},X^*)}\n\\geq\n\\frac{\\theta}{1+\\sqrt{2}\\beta}\\cdot\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}}\n\\norm{g_{\\outerM{t}}(\\outerXinit{t})}^{\\frac{\\rho-2}{\\rho-1}}.\n$$\nSince $\\ep^{(t)}\\equiv \\theta\\|g_{\\outerM{t}}(\\outerXinit{t})\\|$ and $\\|g_{\\outerM{t}}(\\outerXinit{t})\\| \\geq \\ep$ (by the definition of $T$), we obtain \\eqref{sigma-th-t}.\n\nIf $\\rho\\in [1,2[$, on the other hand,\nsince $\\varphi(\\outerXplus{t})\\leq \\varphi(\\xinit)$ by Lemma~\\ref{lem:rAdaAPG-base} (ii), the inequalities \\eqref{HEB-review} and \\eqref{rAdaAPG-obj-bound} imply (remark that $\\frac{2-\\rho}{\\rho}>0$)\n\\begin{align*}\n\\dist(\\outerXplus{t},X^*) &\\leq \\frac{1}{\\kappa^{\\frac{1}{\\rho}}}(\\varphi(\\outerXplus{t})-\\varphi^*)^{\\frac{1}{\\rho}}\n=\n\\frac{1}{\\kappa^{\\frac{1}{\\rho}}}(\\varphi(\\outerXplus{t})-\\varphi^*)^{\\frac{\\rho-1}{\\rho}}(\\varphi(\\outerXplus{t})-\\varphi^*)^{\\frac{2-\\rho}{\\rho}}\\\\\n&\\leq\n\\frac{1}{\\kappa^{\\frac{2}{\\rho}}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})}(\\varphi(\\outerXplus{t})-\\varphi^*)^{\\frac{2-\\rho}{\\rho}}\\\\\n&\\leq\n\\frac{1}{\\kappa^{\\frac{2}{\\rho}}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})}(\\varphi(\\xinit)-\\varphi^*)^{\\frac{2-\\rho}{\\rho}}.\n\\end{align*}\nTherefore, we conclude that\n$$\n\\sigma(\\outerXplus{t},\\ep^{(t)})\n= \\frac{\\theta\\norm{g_{\\outerM{t}}(\\outerXinit{t})}}{(1+\\sqrt{2}\\beta)\\dist(\\outerXplus{t},X^*)}\n\\geq\n\\frac{\\theta}{1+\\sqrt{2}\\beta}\n\\kappa^{\\frac{2}{\\rho}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-1}(\\varphi(\\xinit)-\\varphi^*)^{-\\frac{2-\\rho}{\\rho}}\n=\\bar\\sigma.\n$$\nThis proves the assertion (ii).\n\n(iii)\nIn the case $\\sigmainit\\leq \\bar\\sigma$, (ii) implies $\\outersigma{0} \\leq \\sigma(\\outerXplus{0},\\ep^{(0)})$. Then, using (i), we have $\\outersigma{1}=\\outersigma{0}$ and also $\\outersigma{1} \\leq \\bar\\sigma$.\nContinuing this argument inductively, we conclude that $\\outersigma{t}=\\outersigma{t-1}=\\cdots=\\outersigma{0}$ for all $t$.\n\nIn the case $\\sigmainit \\geq \\bar\\sigma$, on the other hand, let us show $\\outersigma{t} \\geq \\bar\\sigma/\\gamma_{\\rm reg}~(t=0,\\ldots,T+1)$ by induction. The assertion for $t=0$ is clear since $\\sigmainit \\geq \\bar\\sigma \\geq \\bar\\sigma/\\gamma_{\\rm reg}$.\nAssume that $\\outersigma{t}\\geq \\bar\\sigma/\\gamma_{\\rm reg}$ holds for some $t\\geq 0$. By (i) and (ii), we have\n$\n\\outersigma{t+1} \\geq \\min\\{\\outersigma{t}, \\sigma(\\outerXplus{t},\\ep^{(t)})/\\gamma_{\\rm reg}\\} \\geq \\min\\{\\bar\\sigma/\\gamma_{\\rm reg}, \\bar\\sigma/\\gamma_{\\rm reg}\\} =\\bar\\sigma/\\gamma_{\\rm reg}.\n$\nThis completes the proof of (iii).\n\\end{proof}","sha256_utf8":"4e6ddb1f31fbcdf2d80d20a9ee5a036e2c42c3a8fd631aeade3875dfaf1b6f56"}]},{"id":"S098","semantic_type":"proof_step","kind":"proof_step","title":"子程序终止阈值限制参数下降","source_units":[{"start":81854,"end":82630,"start_line":1545,"end_line":1552,"raw_tex":"\\begin{proof}\n(i)\nIn the case $\\outersigma{t} \\leq \\sigma(\\outerXplus{t},\\ep^{(t)})$, Theorem~\\ref{thm:AdaAPG} (i) implies that the subroutine $\\AdaAPG$ at the $t$-th outer loop must terminate at the first loop $j=0$ so that $\\outersigma{t+1}$ is defined by $\\outersigma{t+1}=\\outersigma{t}$.\nOn the other hand, consider the case $\\outersigma{t} > \\sigma(\\outerXplus{t},\\ep^{(t)})$.\nWhen the subroutine $\\AdaAPG$ at the $t$-th outer loop terminates at the first loop $j=0$, then it is clear that\n$\\outersigma{t+1}=\\outersigma{t} > \\sigma(\\outerXplus{t},\\ep^{(t)}) \\geq \\sigma(\\outerXplus{t},\\ep^{(t)})/\\gamma_{\\rm reg}.$\nIn the other case, the latter assertion of Theorem~\\ref{thm:AdaAPG}~(i) implies $\\outersigma{t+1}\\geq \\sigma(\\outerXplus{t},\\ep^{(t)})/\\gamma_{\\rm reg}.$\n\n","sha256_utf8":"e47e3577c063d23e9739862c6456f78622e5f78e6fef955b46f09548b602e838"}]},{"id":"S099","semantic_type":"proof_step","kind":"proof_step","title":"高阶误差界导出残差相关阈值","source_units":[{"start":82630,"end":83417,"start_line":1553,"end_line":1564,"raw_tex":"(ii)\nWe may assume $\\outerXplus{t} \\not\\in X^*$, which allows us to apply Lemma~\\ref{lem:rAdaAPG-base} (iv) (Note that the assertion is clear if $\\outerXplus{t} \\in X^*$ since then $\\sigma(\\outerXplus{t},\\ep^{(t)})=+\\infty$).\nIn the case $\\rho\\geq 2$, using \\eqref{rAdaAPG-dist-bound} implies\n$$\n\\sigma(\\outerXplus{t},\\ep^{(t)})\n= \\frac{\\theta\\norm{g_{\\outerM{t}}(\\outerXinit{t})}}{(1+\\sqrt{2}\\beta)\\dist(\\outerXplus{t},X^*)}\n\\geq\n\\frac{\\theta}{1+\\sqrt{2}\\beta}\\cdot\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}}\n\\norm{g_{\\outerM{t}}(\\outerXinit{t})}^{\\frac{\\rho-2}{\\rho-1}}.\n$$\nSince $\\ep^{(t)}\\equiv \\theta\\|g_{\\outerM{t}}(\\outerXinit{t})\\|$ and $\\|g_{\\outerM{t}}(\\outerXinit{t})\\| \\geq \\ep$ (by the definition of $T$), we obtain \\eqref{sigma-th-t}.\n","sha256_utf8":"1ea67a6fbb44ae767d89f5d31ddf622402ccb45800b4a21cdedef3d586a76083"}]},{"id":"S100","semantic_type":"proof_step","kind":"proof_step","title":"低阶误差界导出统一阈值","source_units":[{"start":83417,"end":84647,"start_line":1565,"end_line":1586,"raw_tex":"\nIf $\\rho\\in [1,2[$, on the other hand,\nsince $\\varphi(\\outerXplus{t})\\leq \\varphi(\\xinit)$ by Lemma~\\ref{lem:rAdaAPG-base} (ii), the inequalities \\eqref{HEB-review} and \\eqref{rAdaAPG-obj-bound} imply (remark that $\\frac{2-\\rho}{\\rho}>0$)\n\\begin{align*}\n\\dist(\\outerXplus{t},X^*) &\\leq \\frac{1}{\\kappa^{\\frac{1}{\\rho}}}(\\varphi(\\outerXplus{t})-\\varphi^*)^{\\frac{1}{\\rho}}\n=\n\\frac{1}{\\kappa^{\\frac{1}{\\rho}}}(\\varphi(\\outerXplus{t})-\\varphi^*)^{\\frac{\\rho-1}{\\rho}}(\\varphi(\\outerXplus{t})-\\varphi^*)^{\\frac{2-\\rho}{\\rho}}\\\\\n&\\leq\n\\frac{1}{\\kappa^{\\frac{2}{\\rho}}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})}(\\varphi(\\outerXplus{t})-\\varphi^*)^{\\frac{2-\\rho}{\\rho}}\\\\\n&\\leq\n\\frac{1}{\\kappa^{\\frac{2}{\\rho}}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)\\norm{g_{\\outerM{t}}(\\outerXinit{t})}(\\varphi(\\xinit)-\\varphi^*)^{\\frac{2-\\rho}{\\rho}}.\n\\end{align*}\nTherefore, we conclude that\n$$\n\\sigma(\\outerXplus{t},\\ep^{(t)})\n= \\frac{\\theta\\norm{g_{\\outerM{t}}(\\outerXinit{t})}}{(1+\\sqrt{2}\\beta)\\dist(\\outerXplus{t},X^*)}\n\\geq\n\\frac{\\theta}{1+\\sqrt{2}\\beta}\n\\kappa^{\\frac{2}{\\rho}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-1}(\\varphi(\\xinit)-\\varphi^*)^{-\\frac{2-\\rho}{\\rho}}\n=\\bar\\sigma.\n$$\nThis proves the assertion (ii).\n","sha256_utf8":"820538becb223c07910c5bb2e822cab3e9cbae8c65f979478b0a5f970fa5e86f"}]},{"id":"S101","semantic_type":"proof_step","kind":"proof_step","title":"小初始参数始终保持不变","source_units":[{"start":84647,"end":84989,"start_line":1587,"end_line":1591,"raw_tex":"\n(iii)\nIn the case $\\sigmainit\\leq \\bar\\sigma$, (ii) implies $\\outersigma{0} \\leq \\sigma(\\outerXplus{0},\\ep^{(0)})$. Then, using (i), we have $\\outersigma{1}=\\outersigma{0}$ and also $\\outersigma{1} \\leq \\bar\\sigma$.\nContinuing this argument inductively, we conclude that $\\outersigma{t}=\\outersigma{t-1}=\\cdots=\\outersigma{0}$ for all $t$.\n\n","sha256_utf8":"e79f4e5156d41cb0d6b7aaa43ef7e0ca52fffd8b5695853cc45fad931e429536"}]},{"id":"S102","semantic_type":"proof_step","kind":"proof_step","title":"归纳保持大初始参数的统一下界","source_units":[{"start":84989,"end":85603,"start_line":1592,"end_line":1598,"raw_tex":"In the case $\\sigmainit \\geq \\bar\\sigma$, on the other hand, let us show $\\outersigma{t} \\geq \\bar\\sigma/\\gamma_{\\rm reg}~(t=0,\\ldots,T+1)$ by induction. The assertion for $t=0$ is clear since $\\sigmainit \\geq \\bar\\sigma \\geq \\bar\\sigma/\\gamma_{\\rm reg}$.\nAssume that $\\outersigma{t}\\geq \\bar\\sigma/\\gamma_{\\rm reg}$ holds for some $t\\geq 0$. By (i) and (ii), we have\n$\n\\outersigma{t+1} \\geq \\min\\{\\outersigma{t}, \\sigma(\\outerXplus{t},\\ep^{(t)})/\\gamma_{\\rm reg}\\} \\geq \\min\\{\\bar\\sigma/\\gamma_{\\rm reg}, \\bar\\sigma/\\gamma_{\\rm reg}\\} =\\bar\\sigma/\\gamma_{\\rm reg}.\n$\nThis completes the proof of (iii).\n\\end{proof}","sha256_utf8":"eaf3ba0edcc27cbdfcb786baf2f54935badd249c441fe0d96b551bcee628d095"}]},{"id":"S103","semantic_type":"lemma","kind":"lemma","title":"高阶误差界下的精细参数估计","source_units":[{"start":85733,"end":86907,"start_line":1602,"end_line":1619,"raw_tex":"\\begin{lemma}\\label{lem:rAdaAPG-sigma-2}\nAssume that the H\\\"olderian error bound condition \\eqref{HEB-review} holds with $\\rho >2$.\nSuppose that $\\rAdaAPG$ terminated with the stopping criterion at the $(T+1)$-th outer loop for some $T\\geq 0$.\nIf $\\sigmainit \\geq \\bar\\sigma$ holds for $\\bar\\sigma$  defined by \\eqref{bar-sigma-def},\nthen there exists $t_0 \\in \\{0,\\ldots,T+1\\}$ such that the following conditions hold, where\n$\n\\sigma_*^{(0)}:=\\frac{\\theta}{1+\\sqrt{2}\\beta}\\cdot\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}}\n\\|g_{\\outerM{0}}(\\xinit)\\|^{\\frac{\\rho-2}{\\rho-1}}.\n$\n\\begin{align}\n\\text{(i)\\qquad} & \\outersigma{0}=\\cdots=\\outersigma{t_0}. \\label{sigma-bound-rho>2-1}\\\\\n\\text{(ii)\\qquad} & \\outersigma{t+1} \\geq \\theta^{-\\frac{\\rho-2}{\\rho-1}(T-t)} \\bar\\sigma/\\gamma_{\\rm reg},\\quad t=t_0,\\ldots,T.\\label{sigma-bound-rho>2-2}\\\\\n\\text{(iii)\\qquad} & t_0 \\leq 1+\\frac{\\rho-1}{\\rho-2}\\log_{1/\\theta}\\frac{\\sigma_*^{(0)}}{\\min(\\sigmainit,\\sigma_*^{(0)})}.\\label{sigma-bound-rho>2-3}\\\\\n\\text{(iv)\\qquad} & \\theta^{\\frac{\\rho-2}{\\rho-1}(T-t_0)} \\geq {\\bar\\sigma}/{\\outersigma{0}}.\\label{sigma-bound-rho>2-4}\n\\end{align}\n\n\\end{lemma}","sha256_utf8":"c75db0f27fcd5c8cf1065970691258ca91a2c49654a24acf7d244bd0e73e39b6"}]},{"id":"S104","semantic_type":"proof","kind":"proof","title":"精细参数估计完整证明","source_units":[{"start":86908,"end":90540,"start_line":1620,"end_line":1692,"raw_tex":"\\begin{proof}\nDefine\n$$\n\\sigma_*^{(t)}:=\\frac{\\theta^{\\frac{1}{\\rho-1}}}{1+\\sqrt{2}\\beta}\\cdot\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}}\n(\\ep^{(t)})^{\\frac{\\rho-2}{\\rho-1}},\\quad t \\geq 0.\n$$\nThen, Lemma~\\ref{lem:rAdaAPG-sigma} (ii) can be written as\n\\begin{equation}\\label{simga_*^t-bound}\n\\sigma(\\outerXplus{t},\\ep^{(t)}) \\geq \\sigma_*^{(t)} \\geq \\bar\\sigma,\\quad t=0,\\ldots,T.\n\\end{equation}\nFor simplicity, denote\n$$\n\\omega:=\\frac{\\rho-2}{\\rho-1} \\in ]0,1[,\\quad\nc:=\\frac{\\theta^{\\frac{1}{\\rho-1}}}{1+\\sqrt{2}\\beta}\\cdot\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}},\n$$\nso that we have\n$$\n\\bar\\sigma = c(\\theta\\ep)^{\\omega},\\quad\n\\sigma_*^{(t)}=c(\\ep^{(t)})^{\\omega},\\quad t\\geq 0.\n$$\nNote that $\\{\\sigma_*^{(t)}\\}$ is non-increasing;\nin fact, the relation $\\ep^{(t+1)}\\leq \\theta\\ep^{(t)}$ implies\n\\begin{equation}\\label{sigma-*-mono}\n\\sigma_*^{(t+1)}=c(\\ep^{(t+1)})^\\omega\\leq c\\theta^\\omega(\\ep^{(t)})^\\omega = \\theta^\\omega \\sigma_*^{(t)} \\leq \\sigma_*^{(t)}.\n\\end{equation}\n\nLet $t_0$ be the smallest integer in $\\{0,\\ldots,T+1\\}$ such that $\\sigmainit \\geq \\sigma_*^{(t_0)}$.\nRemark that, by the definition of $T$, we have $\\ep^{(T+1)}=\\theta\\|g_{\\outerM{T+1}}(\\outerXinit{T+1})\\| \\leq \\theta\\ep$.\nThis implies that\n$\n\\sigmainit \\geq \\bar\\sigma= c(\\theta\\ep)^\\omega \\geq c(\\ep^{(T+1)})^\\omega = \\sigma_*^{(T+1)}.\n$\nTherefore, $t_0$ is well-defined.\n\n%%%\n(i)\nBy the definition of $t_0$ and \\eqref{simga_*^t-bound}, we have\n$\\sigmainit < \\sigma_*^{(t)} \\leq \\sigma(\\outerXplus{t},\\ep^{(t)})$ ($0\\leq \\forall t < t_0$).\nTherefore, by induction, we obtain $\\outersigma{0}=\\outersigma{1}=\\cdots=\\outersigma{t_0}$ due to Lemma~\\ref{lem:rAdaAPG-sigma}~(i).\n\n\n(ii)\nLet us show $\\outersigma{t+1}\\geq \\sigma_*^{(t)}/\\gamma_{\\rm reg}~(t_0\\leq t\\leq T)$.\nTo prove this, we verify $\\outersigma{t}\\geq \\sigma_*^{(t)}/\\gamma_{\\rm reg}$ for ${t= t_0,\\ldots,T+1}$ by induction. Note that $\\outersigma{t_0}=\\outersigma{0} \\geq  \\sigma_*^{(t_0)} > \\sigma_*^{(t_0)}/\\gamma_{\\rm reg}$ holds by (i) and the definition of $t_0$. Now under the hypothesis $\\outersigma{t} \\geq  \\sigma_*^{(t)}/\\gamma_{\\rm reg}$ for $t$ with $t_0\\leq t \\leq T$, Lemma~\\ref{lem:rAdaAPG-sigma}~(i) and \\eqref{simga_*^t-bound} imply\n$$\\outersigma{t+1} \\geq \\min\\{\\outersigma{t},\\sigma(\\outerXplus{t},\\ep^{(t)})/\\gamma_{\\rm reg}\\} \\geq \\min\\{\\sigma_*^{(t)}/\\gamma_{\\rm reg},\\sigma_*^{(t)}/\\gamma_{\\rm reg}\\} = \\sigma_*^{(t)}/\\gamma_{\\rm reg} \\geq \\sigma_*^{(t+1)}/\\gamma_{\\rm reg}.$$\nTherefore, this completes the induction; in addition, the above inequality involves the desired inequality $\\outersigma{t+1}\\geq \\sigma_*^{(t)}/\\gamma_{\\rm reg}~(t_0\\leq t\\leq T)$.\nThis yields (ii) combined with \\eqref{simga_*^t-bound} and \\eqref{sigma-*-mono}:\n$$\n\\outersigma{t+1} \\geq \\sigma_*^{(t)}/\\gamma_{\\rm reg} \\geq \\theta^{-\\omega(T-t)}\\sigma_*^{(T)}/\\gamma_{\\rm reg} \\geq \\theta^{-\\omega(T-t)} \\bar\\sigma/\\gamma_{\\rm reg},\\quad t=t_0,\\ldots,T.\n$$\n%%%\n\n(iii)\nIf $t_0=0$, then\n(iii) is trivial since\n$\\sigmainit \\geq\\sigma_*^{(t_0)}=\\sigma_*^{(0)}$. If $t_0 > 0$, then the definition of $t_0$ and using \\eqref{sigma-*-mono} imply\n$\\sigmainit < \\sigma_*^{(t_0-1)} \\leq \\theta^{\\omega(t_0-1)}\\sigma_*^{(0)},$\nwhich yields $t_0 \\leq 1+\\frac{1}{\\omega}\\log_{1/\\theta}\\frac{\\sigma_*^{(0)}}{\\sigmainit}$.\n\n\n(iv)\nBy \\eqref{simga_*^t-bound}, \\eqref{sigma-*-mono}, and the definition of $t_0$, remark that\n$\n\\bar\\sigma \\leq \\sigma_*^{(T)} \\leq \\theta^{\\omega(T-t_0)}\\sigma_*^{(t_0)}\\leq \\theta^{\\omega(T-t_0)}\\sigmainit.\n$\nHence, \n$\n\\theta^{\\omega(T-t_0)} \\geq {\\bar\\sigma}/{\\outersigma{0}}\n$\nholds.\n\\end{proof}","sha256_utf8":"637300a1c2a31fba47d7adabcbd352b42c2354a228c512db9e1e852d47462f5b"}]},{"id":"S105","semantic_type":"proof_step","kind":"proof_step","title":"定义残差相关阈值及简化常数","source_units":[{"start":86908,"end":87672,"start_line":1620,"end_line":1641,"raw_tex":"\\begin{proof}\nDefine\n$$\n\\sigma_*^{(t)}:=\\frac{\\theta^{\\frac{1}{\\rho-1}}}{1+\\sqrt{2}\\beta}\\cdot\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}}\n(\\ep^{(t)})^{\\frac{\\rho-2}{\\rho-1}},\\quad t \\geq 0.\n$$\nThen, Lemma~\\ref{lem:rAdaAPG-sigma} (ii) can be written as\n\\begin{equation}\\label{simga_*^t-bound}\n\\sigma(\\outerXplus{t},\\ep^{(t)}) \\geq \\sigma_*^{(t)} \\geq \\bar\\sigma,\\quad t=0,\\ldots,T.\n\\end{equation}\nFor simplicity, denote\n$$\n\\omega:=\\frac{\\rho-2}{\\rho-1} \\in ]0,1[,\\quad\nc:=\\frac{\\theta^{\\frac{1}{\\rho-1}}}{1+\\sqrt{2}\\beta}\\cdot\n\\kappa^{\\frac{1}{\\rho-1}}\\left(\\frac{L_f}{L_{\\min}}+1\\right)^{-\\frac{1}{\\rho-1}},\n$$\nso that we have\n$$\n\\bar\\sigma = c(\\theta\\ep)^{\\omega},\\quad\n\\sigma_*^{(t)}=c(\\ep^{(t)})^{\\omega},\\quad t\\geq 0.\n$$\n","sha256_utf8":"afc115bb9e6c0eaf4952434cecfea34444663da80de90d7e4a9566cba0be0186"}]},{"id":"S106","semantic_type":"proof_step","kind":"proof_step","title":"确定参数变化阶段的分界点","source_units":[{"start":87966,"end":88343,"start_line":1647,"end_line":1655,"raw_tex":"\nLet $t_0$ be the smallest integer in $\\{0,\\ldots,T+1\\}$ such that $\\sigmainit \\geq \\sigma_*^{(t_0)}$.\nRemark that, by the definition of $T$, we have $\\ep^{(T+1)}=\\theta\\|g_{\\outerM{T+1}}(\\outerXinit{T+1})\\| \\leq \\theta\\ep$.\nThis implies that\n$\n\\sigmainit \\geq \\bar\\sigma= c(\\theta\\ep)^\\omega \\geq c(\\ep^{(T+1)})^\\omega = \\sigma_*^{(T+1)}.\n$\nTherefore, $t_0$ is well-defined.\n\n","sha256_utf8":"7d8528e0013a854dc32aa0d949592ae279b8653942c95c736db378da9184d87b"}]},{"id":"S107","semantic_type":"proof_step","kind":"proof_step","title":"分界点之前参数保持不变","source_units":[{"start":88343,"end":88645,"start_line":1656,"end_line":1662,"raw_tex":"%%%\n(i)\nBy the definition of $t_0$ and \\eqref{simga_*^t-bound}, we have\n$\\sigmainit < \\sigma_*^{(t)} \\leq \\sigma(\\outerXplus{t},\\ep^{(t)})$ ($0\\leq \\forall t < t_0$).\nTherefore, by induction, we obtain $\\outersigma{0}=\\outersigma{1}=\\cdots=\\outersigma{t_0}$ due to Lemma~\\ref{lem:rAdaAPG-sigma}~(i).\n\n\n","sha256_utf8":"85e58f87a09fead7bb8c184d62056c991944a7bc9f3f3663213705db04e5421f"}]},{"id":"S108","semantic_type":"proof_step","kind":"proof_step","title":"分界点之后归纳控制参数下界","source_units":[{"start":88645,"end":89892,"start_line":1663,"end_line":1673,"raw_tex":"(ii)\nLet us show $\\outersigma{t+1}\\geq \\sigma_*^{(t)}/\\gamma_{\\rm reg}~(t_0\\leq t\\leq T)$.\nTo prove this, we verify $\\outersigma{t}\\geq \\sigma_*^{(t)}/\\gamma_{\\rm reg}$ for ${t= t_0,\\ldots,T+1}$ by induction. Note that $\\outersigma{t_0}=\\outersigma{0} \\geq  \\sigma_*^{(t_0)} > \\sigma_*^{(t_0)}/\\gamma_{\\rm reg}$ holds by (i) and the definition of $t_0$. Now under the hypothesis $\\outersigma{t} \\geq  \\sigma_*^{(t)}/\\gamma_{\\rm reg}$ for $t$ with $t_0\\leq t \\leq T$, Lemma~\\ref{lem:rAdaAPG-sigma}~(i) and \\eqref{simga_*^t-bound} imply\n$$\\outersigma{t+1} \\geq \\min\\{\\outersigma{t},\\sigma(\\outerXplus{t},\\ep^{(t)})/\\gamma_{\\rm reg}\\} \\geq \\min\\{\\sigma_*^{(t)}/\\gamma_{\\rm reg},\\sigma_*^{(t)}/\\gamma_{\\rm reg}\\} = \\sigma_*^{(t)}/\\gamma_{\\rm reg} \\geq \\sigma_*^{(t+1)}/\\gamma_{\\rm reg}.$$\nTherefore, this completes the induction; in addition, the above inequality involves the desired inequality $\\outersigma{t+1}\\geq \\sigma_*^{(t)}/\\gamma_{\\rm reg}~(t_0\\leq t\\leq T)$.\nThis yields (ii) combined with \\eqref{simga_*^t-bound} and \\eqref{sigma-*-mono}:\n$$\n\\outersigma{t+1} \\geq \\sigma_*^{(t)}/\\gamma_{\\rm reg} \\geq \\theta^{-\\omega(T-t)}\\sigma_*^{(T)}/\\gamma_{\\rm reg} \\geq \\theta^{-\\omega(T-t)} \\bar\\sigma/\\gamma_{\\rm reg},\\quad t=t_0,\\ldots,T.\n$$\n%%%\n\n","sha256_utf8":"d87059f74f24fca41c220a927df37cebb63bf384681f153106ae75c97b7559f5"}]},{"id":"S109","semantic_type":"proof_step","kind":"proof_step","title":"估计恒定参数阶段长度","source_units":[{"start":89892,"end":90239,"start_line":1674,"end_line":1681,"raw_tex":"(iii)\nIf $t_0=0$, then\n(iii) is trivial since\n$\\sigmainit \\geq\\sigma_*^{(t_0)}=\\sigma_*^{(0)}$. If $t_0 > 0$, then the definition of $t_0$ and using \\eqref{sigma-*-mono} imply\n$\\sigmainit < \\sigma_*^{(t_0-1)} \\leq \\theta^{\\omega(t_0-1)}\\sigma_*^{(0)},$\nwhich yields $t_0 \\leq 1+\\frac{1}{\\omega}\\log_{1/\\theta}\\frac{\\sigma_*^{(0)}}{\\sigmainit}$.\n\n\n","sha256_utf8":"6354fee2554071828a97badbfba3b88b490fa774f10b248159f230c046c665ff"}]},{"id":"S110","semantic_type":"proof_step","kind":"proof_step","title":"联系尾段长度与初始参数","source_units":[{"start":90239,"end":90540,"start_line":1682,"end_line":1692,"raw_tex":"(iv)\nBy \\eqref{simga_*^t-bound}, \\eqref{sigma-*-mono}, and the definition of $t_0$, remark that\n$\n\\bar\\sigma \\leq \\sigma_*^{(T)} \\leq \\theta^{\\omega(T-t_0)}\\sigma_*^{(t_0)}\\leq \\theta^{\\omega(T-t_0)}\\sigmainit.\n$\nHence, \n$\n\\theta^{\\omega(T-t_0)} \\geq {\\bar\\sigma}/{\\outersigma{0}}\n$\nholds.\n\\end{proof}","sha256_utf8":"1d75abb69888cbd53807036400d33911568ae9ecae7ba23515f936f18d42e71e"}]},{"id":"S111","semantic_type":"external_fact","kind":"external_fact","title":"梯度映射范数关于参数单调","source_units":[],"declared_external":true,"declaration_text":["For a convex composite objective and every fixed y, L↦||g_L(y)|| is nondecreasing on L>0 (Nes13, Lemma 2)."]},{"id":"S112","semantic_type":"external_fact","kind":"external_fact","title":"凸次微分求和与最优性条件","source_units":[],"declared_external":true,"declaration_text":["For finite continuously differentiable convex f and proper lower-semicontinuous convex Ψ, ∂(f+Ψ)(x)=∇f(x)+∂Ψ(x). A point minimizes a proper convex function exactly when zero belongs to its subdifferential (Roc, Theorem 23.8, and convex Fermat optimality)."]},{"id":"S113","semantic_type":"external_fact","kind":"external_fact","title":"近端梯度三点下降不等式","source_units":[],"declared_external":true,"declaration_text":["If φ(T_L(y))≤m_L(y;T_L(y)), then φ(x)−φ(T_L(y))≥||g_L(y)||²/(2L)+〈g_L(y),x−y〉 for every x (BT09, Lemma 2.3)."]},{"id":"S114","semantic_type":"external_fact","kind":"external_fact","title":"光滑函数的二次上界","source_units":[],"declared_external":true,"declaration_text":["An L-smooth function satisfies f(x)≤f(y)+〈∇f(y),x−y〉+(L/2)||x−y||² for every x,y (Nes04)."]},{"id":"S115","semantic_type":"external_fact","kind":"external_fact","title":"凸光滑梯度的共胁性","source_units":[],"declared_external":true,"declaration_text":["For convex L-smooth f, 〈∇f(y)−∇f(z),y−z〉≥||∇f(y)−∇f(z)||²/L for every y,z (Nes04, Theorem 2.1.5)."]},{"id":"S116","semantic_type":"external_fact","kind":"external_fact","title":"强凸加速方法的权重增长","source_units":[],"declared_external":true,"declaration_text":["For Nesterov's accelerated proximal gradient method applied to φ_σ=f+Ψ_σ with σ-strongly convex Ψ_σ and the accepted line-search bounds, A_k≥[2/(γ_inc L_f)]·[1+sqrt(σ/(2γ_inc L_f))]^{2(k−1)} for k≥1 (Nes13, Lemma 8)."]},{"id":"S117","semantic_type":"external_fact","kind":"external_fact","title":"加速估计序列的上下界","source_units":[],"declared_external":true,"declaration_text":["The accelerated proximal gradient estimate sequence satisfies A_k φ_σ(x_k)≤min_x ψ_k(x) and ψ_k(x)≤A_k φ_σ(x)+(1/2)||x−x_0||² for every x and k≥0 (Nes13, Lemma 7)."]},{"id":"S118","semantic_type":"external_fact","kind":"external_fact","title":"强凸性给出的最优点二次增长","source_units":[],"declared_external":true,"declaration_text":["If h is μ-strongly convex and x* minimizes h, then h(x)≥h(x*)+(μ/2)||x−x*||² for every x."]},{"id":"S119","semantic_type":"proof_step","kind":"proof_step","title":"残差阈值按几何比例递减","source_units":[{"start":87672,"end":87966,"start_line":1642,"end_line":1646,"raw_tex":"Note that $\\{\\sigma_*^{(t)}\\}$ is non-increasing;\nin fact, the relation $\\ep^{(t+1)}\\leq \\theta\\ep^{(t)}$ implies\n\\begin{equation}\\label{sigma-*-mono}\n\\sigma_*^{(t+1)}=c(\\ep^{(t+1)})^\\omega\\leq c\\theta^\\omega(\\ep^{(t)})^\\omega = \\theta^\\omega \\sigma_*^{(t)} \\leq \\sigma_*^{(t)}.\n\\end{equation}\n","sha256_utf8":"5ca9a24ed0bea674bddeec694b8f738ad33748d890a96cc263399dff12c56327"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S014","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S001","target_id":"S003","kind":"supports"},{"source_id":"S001","target_id":"S004","kind":"supports"},{"source_id":"S002","target_id":"S006","kind":"supports"},{"source_id":"S002","target_id":"S008","kind":"supports"},{"source_id":"S002","target_id":"S035","kind":"supports"},{"source_id":"S002","target_id":"S036","kind":"supports"},{"source_id":"S003","target_id":"S010","kind":"supports"},{"source_id":"S003","target_id":"S041","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S045","kind":"supports"},{"source_id":"S004","target_id":"S046","kind":"supports"},{"source_id":"S004","target_id":"S049","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S061","kind":"supports"},{"source_id":"S005","target_id":"S062","kind":"supports"},{"source_id":"S007","target_id":"S006","kind":"supports"},{"source_id":"S006","target_id":"S009","kind":"supports"},{"source_id":"S006","target_id":"S052","kind":"supports"},{"source_id":"S006","target_id":"S053","kind":"supports"},{"source_id":"S006","target_id":"S056","kind":"supports"},{"source_id":"S006","target_id":"S057","kind":"supports"},{"source_id":"S006","target_id":"S060","kind":"supports"},{"source_id":"S006","target_id":"S064","kind":"supports"},{"source_id":"S006","target_id":"S023","kind":"supports"},{"source_id":"S008","target_id":"S011","kind":"supports"},{"source_id":"S008","target_id":"S068","kind":"supports"},{"source_id":"S008","target_id":"S069","kind":"supports"},{"source_id":"S009","target_id":"S011","kind":"supports"},{"source_id":"S009","target_id":"S072","kind":"supports"},{"source_id":"S009","target_id":"S073","kind":"supports"},{"source_id":"S009","target_id":"S078","kind":"supports"},{"source_id":"S009","target_id":"S079","kind":"supports"},{"source_id":"S009","target_id":"S080","kind":"supports"},{"source_id":"S010","target_id":"S088","kind":"supports"},{"source_id":"S010","target_id":"S100","kind":"supports"},{"source_id":"S011","target_id":"S016","kind":"supports"},{"source_id":"S011","target_id":"S017","kind":"supports"},{"source_id":"S011","target_id":"S021","kind":"supports"},{"source_id":"S011","target_id":"S022","kind":"supports"},{"source_id":"S011","target_id":"S024","kind":"supports"},{"source_id":"S011","target_id":"S031","kind":"supports"},{"source_id":"S011","target_id":"S085","kind":"supports"},{"source_id":"S011","target_id":"S086","kind":"supports"},{"source_id":"S011","target_id":"S087","kind":"supports"},{"source_id":"S011","target_id":"S098","kind":"supports"},{"source_id":"S011","target_id":"S105","kind":"supports"},{"source_id":"S011","target_id":"S106","kind":"supports"},{"source_id":"S011","target_id":"S119","kind":"supports"},{"source_id":"S012","target_id":"S094","kind":"supports"},{"source_id":"S012","target_id":"S095","kind":"supports"},{"source_id":"S012","target_id":"S017","kind":"supports"},{"source_id":"S013","target_id":"S099","kind":"supports"},{"source_id":"S013","target_id":"S100","kind":"supports"},{"source_id":"S013","target_id":"S105","kind":"supports"},{"source_id":"S111","target_id":"S034","kind":"supports"},{"source_id":"S112","target_id":"S035","kind":"supports"},{"source_id":"S112","target_id":"S036","kind":"supports"},{"source_id":"S113","target_id":"S037","kind":"supports"},{"source_id":"S114","target_id":"S038","kind":"supports"},{"source_id":"S115","target_id":"S052","kind":"supports"},{"source_id":"S116","target_id":"S061","kind":"supports"},{"source_id":"S117","target_id":"S062","kind":"supports"},{"source_id":"S118","target_id":"S046","kind":"supports"},{"source_id":"S118","target_id":"S048","kind":"supports"},{"source_id":"S118","target_id":"S063","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S035","target_id":"S033","kind":"supports"},{"source_id":"S036","target_id":"S033","kind":"supports"},{"source_id":"S037","target_id":"S033","kind":"supports"},{"source_id":"S038","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S042","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S044","kind":"supports"},{"source_id":"S049","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S032","target_id":"S052","kind":"supports"},{"source_id":"S032","target_id":"S053","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S055","kind":"supports"},{"source_id":"S057","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S060","kind":"supports"},{"source_id":"S043","target_id":"S062","kind":"supports"},{"source_id":"S043","target_id":"S063","kind":"supports"},{"source_id":"S043","target_id":"S064","kind":"supports"},{"source_id":"S043","target_id":"S065","kind":"supports"},{"source_id":"S032","target_id":"S064","kind":"supports"},{"source_id":"S032","target_id":"S065","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S062","target_id":"S064","kind":"supports"},{"source_id":"S063","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S061","target_id":"S059","kind":"supports"},{"source_id":"S062","target_id":"S059","kind":"supports"},{"source_id":"S063","target_id":"S059","kind":"supports"},{"source_id":"S064","target_id":"S059","kind":"supports"},{"source_id":"S065","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S058","kind":"supports"},{"source_id":"S053","target_id":"S068","kind":"supports"},{"source_id":"S054","target_id":"S069","kind":"supports"},{"source_id":"S032","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S067","kind":"supports"},{"source_id":"S069","target_id":"S067","kind":"supports"},{"source_id":"S067","target_id":"S066","kind":"supports"},{"source_id":"S058","target_id":"S072","kind":"supports"},{"source_id":"S058","target_id":"S073","kind":"supports"},{"source_id":"S074","target_id":"S073","kind":"supports"},{"source_id":"S073","target_id":"S075","kind":"supports"},{"source_id":"S072","target_id":"S071","kind":"supports"},{"source_id":"S073","target_id":"S071","kind":"supports"},{"source_id":"S074","target_id":"S071","kind":"supports"},{"source_id":"S075","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S070","kind":"supports"},{"source_id":"S070","target_id":"S078","kind":"supports"},{"source_id":"S070","target_id":"S079","kind":"supports"},{"source_id":"S079","target_id":"S081","kind":"supports"},{"source_id":"S080","target_id":"S081","kind":"supports"},{"source_id":"S078","target_id":"S082","kind":"supports"},{"source_id":"S081","target_id":"S082","kind":"supports"},{"source_id":"S078","target_id":"S077","kind":"supports"},{"source_id":"S079","target_id":"S077","kind":"supports"},{"source_id":"S080","target_id":"S077","kind":"supports"},{"source_id":"S081","target_id":"S077","kind":"supports"},{"source_id":"S082","target_id":"S077","kind":"supports"},{"source_id":"S077","target_id":"S076","kind":"supports"},{"source_id":"S066","target_id":"S085","kind":"supports"},{"source_id":"S058","target_id":"S085","kind":"supports"},{"source_id":"S066","target_id":"S086","kind":"supports"},{"source_id":"S058","target_id":"S086","kind":"supports"},{"source_id":"S006","target_id":"S086","kind":"supports"},{"source_id":"S032","target_id":"S088","kind":"supports"},{"source_id":"S039","target_id":"S088","kind":"supports"},{"source_id":"S085","target_id":"S088","kind":"supports"},{"source_id":"S086","target_id":"S088","kind":"supports"},{"source_id":"S085","target_id":"S084","kind":"supports"},{"source_id":"S086","target_id":"S084","kind":"supports"},{"source_id":"S087","target_id":"S084","kind":"supports"},{"source_id":"S088","target_id":"S084","kind":"supports"},{"source_id":"S084","target_id":"S083","kind":"supports"},{"source_id":"S076","target_id":"S091","kind":"supports"},{"source_id":"S083","target_id":"S092","kind":"supports"},{"source_id":"S092","target_id":"S093","kind":"supports"},{"source_id":"S091","target_id":"S094","kind":"supports"},{"source_id":"S093","target_id":"S094","kind":"supports"},{"source_id":"S092","target_id":"S095","kind":"supports"},{"source_id":"S094","target_id":"S095","kind":"supports"},{"source_id":"S091","target_id":"S090","kind":"supports"},{"source_id":"S092","target_id":"S090","kind":"supports"},{"source_id":"S093","target_id":"S090","kind":"supports"},{"source_id":"S094","target_id":"S090","kind":"supports"},{"source_id":"S095","target_id":"S090","kind":"supports"},{"source_id":"S090","target_id":"S089","kind":"supports"},{"source_id":"S076","target_id":"S098","kind":"supports"},{"source_id":"S076","target_id":"S099","kind":"supports"},{"source_id":"S076","target_id":"S100","kind":"supports"},{"source_id":"S083","target_id":"S099","kind":"supports"},{"source_id":"S083","target_id":"S100","kind":"supports"},{"source_id":"S098","target_id":"S101","kind":"supports"},{"source_id":"S099","target_id":"S101","kind":"supports"},{"source_id":"S100","target_id":"S101","kind":"supports"},{"source_id":"S098","target_id":"S102","kind":"supports"},{"source_id":"S099","target_id":"S102","kind":"supports"},{"source_id":"S100","target_id":"S102","kind":"supports"},{"source_id":"S098","target_id":"S097","kind":"supports"},{"source_id":"S099","target_id":"S097","kind":"supports"},{"source_id":"S100","target_id":"S097","kind":"supports"},{"source_id":"S101","target_id":"S097","kind":"supports"},{"source_id":"S102","target_id":"S097","kind":"supports"},{"source_id":"S097","target_id":"S096","kind":"supports"},{"source_id":"S096","target_id":"S105","kind":"supports"},{"source_id":"S105","target_id":"S119","kind":"supports"},{"source_id":"S105","target_id":"S106","kind":"supports"},{"source_id":"S106","target_id":"S107","kind":"supports"},{"source_id":"S096","target_id":"S107","kind":"supports"},{"source_id":"S105","target_id":"S107","kind":"supports"},{"source_id":"S107","target_id":"S108","kind":"supports"},{"source_id":"S106","target_id":"S108","kind":"supports"},{"source_id":"S096","target_id":"S108","kind":"supports"},{"source_id":"S105","target_id":"S108","kind":"supports"},{"source_id":"S119","target_id":"S108","kind":"supports"},{"source_id":"S106","target_id":"S109","kind":"supports"},{"source_id":"S119","target_id":"S109","kind":"supports"},{"source_id":"S105","target_id":"S110","kind":"supports"},{"source_id":"S106","target_id":"S110","kind":"supports"},{"source_id":"S119","target_id":"S110","kind":"supports"},{"source_id":"S105","target_id":"S104","kind":"supports"},{"source_id":"S119","target_id":"S104","kind":"supports"},{"source_id":"S106","target_id":"S104","kind":"supports"},{"source_id":"S107","target_id":"S104","kind":"supports"},{"source_id":"S108","target_id":"S104","kind":"supports"},{"source_id":"S109","target_id":"S104","kind":"supports"},{"source_id":"S110","target_id":"S104","kind":"supports"},{"source_id":"S104","target_id":"S103","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S096","target_id":"S017","kind":"supports"},{"source_id":"S089","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S096","target_id":"S018","kind":"supports"},{"source_id":"S103","target_id":"S019","kind":"supports"},{"source_id":"S103","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S089","target_id":"S020","kind":"supports"},{"source_id":"S017","target_id":"S021","kind":"supports"},{"source_id":"S089","target_id":"S021","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S058","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S083","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S021","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S096","target_id":"S027","kind":"supports"},{"source_id":"S083","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S058","target_id":"S028","kind":"supports"},{"source_id":"S017","target_id":"S029","kind":"supports"},{"source_id":"S027","target_id":"S029","kind":"supports"},{"source_id":"S089","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S030","kind":"supports"},{"source_id":"S017","target_id":"S030","kind":"supports"},{"source_id":"S089","target_id":"S030","kind":"supports"},{"source_id":"S027","target_id":"S031","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S015","kind":"supports"},{"source_id":"S018","target_id":"S015","kind":"supports"},{"source_id":"S019","target_id":"S015","kind":"supports"},{"source_id":"S020","target_id":"S015","kind":"supports"},{"source_id":"S021","target_id":"S015","kind":"supports"},{"source_id":"S022","target_id":"S015","kind":"supports"},{"source_id":"S023","target_id":"S015","kind":"supports"},{"source_id":"S024","target_id":"S015","kind":"supports"},{"source_id":"S025","target_id":"S015","kind":"supports"},{"source_id":"S026","target_id":"S015","kind":"supports"},{"source_id":"S027","target_id":"S015","kind":"supports"},{"source_id":"S028","target_id":"S015","kind":"supports"},{"source_id":"S029","target_id":"S015","kind":"supports"},{"source_id":"S030","target_id":"S015","kind":"supports"},{"source_id":"S031","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"}]}}
{"record_id":"2002.05545v3","paper":{"title":"Sampling and Update Frequencies in Proximal Variance-Reduced Stochastic Gradient Methods","authors":["Martin Morin","Pontus Giselsson"],"arxiv_id":"2002.05545","version":"v3","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-025-02671-y","abstract_url":"https://arxiv.org/abs/2002.05545v3","pdf_url":"https://arxiv.org/pdf/2002.05545v3"},"context":{"raw_tex":"Let $\\R$ be the set of real numbers.\nWe will work in finite dimensional real spaces $\\R^N$.\nLet $\\inprod{\\cdot}{\\cdot}$ denote the standard Euclidean inner product and let $\\norm{\\cdot}$ be the norm induced by the inner product.\nThe expected value conditioned on the filtration $\\mathcal{F}$ is $\\Exp[\\cdot | \\mathcal{F}]$.\nThe probability of a discrete random variable taking value $i$ is $P(\\cdot = i)$.\nWe define $\\mathbf{1}_{X} = 1$ if the predicate $X$ is true, otherwise $\\mathbf{1}_X = 0$.\n\n\nA convex function $f: \\R^d \\to \\R$ is \\emph{$L$-smooth} with $L > 0$ if it is differentiable and its gradient is $\\tfrac{1}{L}$-cocoercive, i.e.,\n\\begin{align*}\n\t\\inprod{\\nabla f(x) - \\nabla f(y)}{x-y} \\geq \\tfrac{1}{L} \\norm{\\nabla f(x) - \\nabla f(y)}^2, \\quad \\forall x,y \\in \\R^d.\n\\end{align*}\nNote that the definition of smoothness implies \\emph{$L$-Lipschitz continuity} of the gradient $\\nabla f$.\nIn fact, for convex $f$, Lipschitz continuity and cocoercivity of $\\nabla f$ are equivalent \\cite[Corollary 18.17]{bauschke_convex_2017}.\nA proper function $f: \\R^d \\to \\R \\cup \\{\\infty\\}$ is \\emph{$\\mu$-strongly convex} with $\\mu > 0$ if $f - \\frac{\\mu}{2}\\norm{\\cdot}^2$ is convex.\n\nThe subdifferential of a $\\mu$-strongly convex function is $\\mu$-strongly monotone \\cite[Example 22.4]{bauschke_convex_2017}, i.e.,\n\\begin{align*}\n\t\\inprod{u - v}{x-y} \\geq \\mu \\norm{x-y}^2\n\\end{align*}\nholds $\\forall x,y \\in \\mathop{\\mathrm{dom}} \\partial f$ and $\\forall u\\in \\partial f(x), \\forall v \\in \\partial f(y)$.\nA closed, proper and strongly convex function has a unique minimum \\cite[Corollary 11.17]{bauschke_convex_2017}.\n\nThe \\emph{proximal operator} of a closed, convex and proper function $g: \\R^d \\to \\R \\cup \\{\\infty\\}$ is defined as\n\\begin{align*}\n\t\\prox_g(z) = \\argmin_x g(x) + \\tfrac{1}{2}\\norm{x-z}^2.\n\\end{align*}\nDue to strong convexity of $g + \\frac{1}{2}\\norm{\\cdot - z}^2$, the minimum exist and is unique.\nFurthermore, the operator $\\prox_g$ is \\emph{non-expansive}, i.e., Lipschitz continuous with constant $1$ \\cite[Proposition 12.28]{bauschke_convex_2017}.\n\n\nWe consider the regularized finite sum problem\n\\begin{align}\\label{eq:problem}\n\t\\min_{x \\in \\R^N} \\quad& g(x) + F(x),\n\\end{align}\n\nwhere $g: \\R^N \\rightarrow \\R \\cup \\{\\infty\\}$ and $F$ is of finite sum form $F(x) = \\frac{1}{n} \\sum_{i=1}^n f_i(x)$ with $f_i: \\R^N \\rightarrow \\R$ for all $i\\in\\{1,\\dots,n\\}$.\n\n\n\\begin{ass}[Problem Properties]\\label{ass:strong-ass}%\n\tThe function $g: \\R^N \\rightarrow \\R \\cup \\{\\infty\\}$ is closed, convex and proper.\n\tFor all $i\\in\\{1,\\dots,n\\}$, the function $f_i: \\R^N \\rightarrow \\R$ is convex, differentiable and $L_i$-smooth.\n\tThe function $F: \\R^N \\rightarrow \\R$ is $\\mu$-strongly convex, differentiable and $L$-smooth with $L \\leq \\frac{1}{n}\\sum_{i=1}^n L_i$.\n\\end{ass}\n\nAs a consequence of \\Cref{ass:strong-ass}, $g + F$ is closed, proper and $\\mu$-strongly convex and  hence there exists a unique solution to \\Cref{eq:problem}, which we denote $x^\\star$.\n\n\n\\begin{algorithm}[H]%\n\t\\caption{PVRSG - Proximal Variance-Reduced Stochastic Gradient}\\label{alg:pvrsg}\n\tGiven the function $g$, the functions $f_1,\\dots,f_n$, initial primal and dual points, $x^0$ and $y_1^0,\\dots y_n^0$, iteratively perform the following for $k\\in\\{0,1,\\dots\\}$.\n\t\\begin{align*}\n\t\t&\\textbf{Sampling:}\n\t\t&&\\text{Randomly sample } (I^k,U_1^k,\\dots,U_n^k) \\text{ from } \\{1,\\dots,n\\}\\times\\{0,1\\}^n.\n\t\t\\\\\n\t\t&\\textbf{Primal Update:}\n\t\t&&\\begin{aligned}[t]\n\t\t\t&\\begin{aligned}\n\t\t\t\tz^{k+1} &=x^k - \\tfrac{\\lambda}{n} \\big(\\tfrac{1}{p_{I^k}}(\\nabla f_{I^k} (x^k) - y_{I^k}^k) + {\\textstyle\\sum_{i=1}^n} y_i^k\\big), \\\\\n\t\t\t\tx^{k+1} &= \\prox_{\\lambda g}(z^{k+1}).\n\t\t\t\\end{aligned}\n\t\t\\end{aligned}\\\\\n\t\t\\\\\n\t\t&\\textbf{Dual Update:}\n\t\t&&\\begin{aligned}[t]\n\t\t\t&\\begin{aligned}\n\t\t\t\ty_i^{k+1} &= y_i^k + U_i^k(\\nabla f_i (x^k) - y_i^k), \\quad \\forall i \\in \\{1,\\dots,n\\}.\n\t\t\t\\end{aligned}\n\t\t\\end{aligned}\n\t\\end{align*}\n\tThe sampling distributions of $(I^k,U_1^k,\\dots,U_n^k)$ for all $k\\in\\{0,1,\\dots\\}$ are the same and independent.\n\tFurthermore, the distribution is such that $P(I^k = i) = p_i > 0$ and the expected update frequency $\\eta_i > 0$, see \\Cref{def:update-freq}, for all $i\\in\\{1,\\dots,n\\}$.\n\tThe step-size satisfies $\\lambda > 0$.\n\\end{algorithm}\n\n\\begin{defin}[Expected Update Frequency]\\label{def:update-freq}%\n\tLet $U_1^k,\\dots,U_n^k$ be given by the sampling in \\Cref{alg:pvrsg}.\n\tThe expected update frequency of the $i$th dual variable is\n\t\\begin{align*}\n\t\t\\eta_i = \\Exp[U_i^k|\\mathcal{F}^k],\n\t\\end{align*}\n\twhere $\\mathcal{F}^k = \\cup_{i=1}^k \\mathcal{X}^i$ and $\\mathcal{X}^k = \\{x^k,y^k,I^{k-1},U_1^{k-1},\\dots,U_n^{k-1}\\}$.\n\\end{defin}\n\nNote, the expected update frequency does not depend on the iteration number $k$ since $(I^k,U_1^k,\\dots,U_n^k)$ is independently sampled and its distribution does not depend on $k$.\n\n\n\\begin{defin}\\label{def:lyap-terms}%\n\tLet $x^\\star$ be the solution to \\Cref{eq:problem} and $y_i^\\star = \\nabla f_i(x^\\star)$ for all $i\\in\\{1,\\dots,n\\}$.\n\tWith $y = (y_1,\\dots,y_n)$, where $y_i \\in \\R^n$ for $i \\in \\{1,\\dots,n\\}$, we define\n\t\\begin{align*}\n\t\t\\mathcal{P}(x) = \\norm{x - x^\\star}^2 - 2\\lambda\\inprod{\\nabla F(x) - \\nabla F (x^\\star)}{x - x^\\star} + \\lambda^2\\mathcal{V}(x)\n\t\\end{align*}\n\tand\n\t\\begin{align*}\n\t\t\\mathcal{D}(y)\n\t\t&= \\sum_{i=1}^n {\\textstyle(1 - \\eta_i  + \\frac{1}{\\gamma_i})\\widehat{\\gamma}_i} \\norm{y_i - y_i^\\star}^2\n\t\t-{\\textstyle (1+\\delta^{-1})\\lambda^2}\\norm{{{\\textstyle\\frac{1}{n}}\\sum_{i=1}^n} y_i -  y_i^\\star }^2,\n\t\\end{align*}\n\twhere\n\t\\begin{align*}\n\t\t\\mathcal{V}(x)\n\t\t= \\sum_{i=1}^n {\\textstyle \\frac{(1+\\delta)}{n^2 p_i}(\\frac{\\eta_i\\gamma_i}{\\delta}  + 1) }\\norm{\\nabla f_i(x) - \\nabla f_i(x^\\star)}^2 - \\delta\\norm{\\nabla F(x) - \\nabla F(x^\\star)}^2\n\t\\end{align*}\n\twith $\\gamma_i \\geq 0$, $\\delta > 0$ and $\\widehat{\\gamma}_i = \\gamma_i \\frac{(1+\\delta^{-1}) \\lambda^2}{n^2p_i}$.\n\tIf $\\gamma_i = 0$ we define $\\tfrac{\\gamma_i}{\\gamma_i} := 1$.\n\\end{defin}","units":[{"start":11186,"end":11683,"start_line":247,"end_line":252,"raw_tex":"Let $\\R$ be the set of real numbers.\nWe will work in finite dimensional real spaces $\\R^N$.\nLet $\\inprod{\\cdot}{\\cdot}$ denote the standard Euclidean inner product and let $\\norm{\\cdot}$ be the norm induced by the inner product.\nThe expected value conditioned on the filtration $\\mathcal{F}$ is $\\Exp[\\cdot | \\mathcal{F}]$.\nThe probability of a discrete random variable taking value $i$ is $P(\\cdot = i)$.\nWe define $\\mathbf{1}_{X} = 1$ if the predicate $X$ is true, otherwise $\\mathbf{1}_X = 0$.\n","sha256_utf8":"815df576a053dbf70c6b5f8cb1f3e15cbea847d727bd403c9811ba88b865ee76"},{"start":11684,"end":13262,"start_line":254,"end_line":274,"raw_tex":"A convex function $f: \\R^d \\to \\R$ is \\emph{$L$-smooth} with $L > 0$ if it is differentiable and its gradient is $\\tfrac{1}{L}$-cocoercive, i.e.,\n\\begin{align*}\n\t\\inprod{\\nabla f(x) - \\nabla f(y)}{x-y} \\geq \\tfrac{1}{L} \\norm{\\nabla f(x) - \\nabla f(y)}^2, \\quad \\forall x,y \\in \\R^d.\n\\end{align*}\nNote that the definition of smoothness implies \\emph{$L$-Lipschitz continuity} of the gradient $\\nabla f$.\nIn fact, for convex $f$, Lipschitz continuity and cocoercivity of $\\nabla f$ are equivalent \\cite[Corollary 18.17]{bauschke_convex_2017}.\nA proper function $f: \\R^d \\to \\R \\cup \\{\\infty\\}$ is \\emph{$\\mu$-strongly convex} with $\\mu > 0$ if $f - \\frac{\\mu}{2}\\norm{\\cdot}^2$ is convex.\n\nThe subdifferential of a $\\mu$-strongly convex function is $\\mu$-strongly monotone \\cite[Example 22.4]{bauschke_convex_2017}, i.e.,\n\\begin{align*}\n\t\\inprod{u - v}{x-y} \\geq \\mu \\norm{x-y}^2\n\\end{align*}\nholds $\\forall x,y \\in \\mathop{\\mathrm{dom}} \\partial f$ and $\\forall u\\in \\partial f(x), \\forall v \\in \\partial f(y)$.\nA closed, proper and strongly convex function has a unique minimum \\cite[Corollary 11.17]{bauschke_convex_2017}.\n\nThe \\emph{proximal operator} of a closed, convex and proper function $g: \\R^d \\to \\R \\cup \\{\\infty\\}$ is defined as\n\\begin{align*}\n\t\\prox_g(z) = \\argmin_x g(x) + \\tfrac{1}{2}\\norm{x-z}^2.\n\\end{align*}\nDue to strong convexity of $g + \\frac{1}{2}\\norm{\\cdot - z}^2$, the minimum exist and is unique.\nFurthermore, the operator $\\prox_g$ is \\emph{non-expansive}, i.e., Lipschitz continuous with constant $1$ \\cite[Proposition 12.28]{bauschke_convex_2017}.\n","sha256_utf8":"5b8989141a4e6698b3834d24a2ecdb0c079f45b6e57e5b5f2dddfe0f7f50e31c"},{"start":13514,"end":13643,"start_line":282,"end_line":285,"raw_tex":"We consider the regularized finite sum problem\n\\begin{align}\\label{eq:problem}\n\t\\min_{x \\in \\R^N} \\quad& g(x) + F(x),\n\\end{align}","sha256_utf8":"073f790b25eb0e7aed7a0985fe60bcbf070e5b661de88cf27f739936990d326d"},{"start":13644,"end":13823,"start_line":286,"end_line":286,"raw_tex":"where $g: \\R^N \\rightarrow \\R \\cup \\{\\infty\\}$ and $F$ is of finite sum form $F(x) = \\frac{1}{n} \\sum_{i=1}^n f_i(x)$ with $f_i: \\R^N \\rightarrow \\R$ for all $i\\in\\{1,\\dots,n\\}$.\n","sha256_utf8":"fbc69c7e50981c7620e7c36dd8fc69e86e0ddb46d7ca6b95714a8aeeba81967d"},{"start":13891,"end":14292,"start_line":288,"end_line":292,"raw_tex":"\\begin{ass}[Problem Properties]\\label{ass:strong-ass}%\n\tThe function $g: \\R^N \\rightarrow \\R \\cup \\{\\infty\\}$ is closed, convex and proper.\n\tFor all $i\\in\\{1,\\dots,n\\}$, the function $f_i: \\R^N \\rightarrow \\R$ is convex, differentiable and $L_i$-smooth.\n\tThe function $F: \\R^N \\rightarrow \\R$ is $\\mu$-strongly convex, differentiable and $L$-smooth with $L \\leq \\frac{1}{n}\\sum_{i=1}^n L_i$.\n\\end{ass}","sha256_utf8":"5c943c8b64d8cb1675e8255a33af53170578da3868d89501bda2b13c1b358cf6"},{"start":14293,"end":14479,"start_line":293,"end_line":293,"raw_tex":"As a consequence of \\Cref{ass:strong-ass}, $g + F$ is closed, proper and $\\mu$-strongly convex and  hence there exists a unique solution to \\Cref{eq:problem}, which we denote $x^\\star$.\n","sha256_utf8":"c894c9a07f29e7db0b06bd09fab4614e65ba2ca3cfde7ffe75924f51d0485c75"},{"start":14596,"end":15859,"start_line":295,"end_line":320,"raw_tex":"\\begin{algorithm}[H]%\n\t\\caption{PVRSG - Proximal Variance-Reduced Stochastic Gradient}\\label{alg:pvrsg}\n\tGiven the function $g$, the functions $f_1,\\dots,f_n$, initial primal and dual points, $x^0$ and $y_1^0,\\dots y_n^0$, iteratively perform the following for $k\\in\\{0,1,\\dots\\}$.\n\t\\begin{align*}\n\t\t&\\textbf{Sampling:}\n\t\t&&\\text{Randomly sample } (I^k,U_1^k,\\dots,U_n^k) \\text{ from } \\{1,\\dots,n\\}\\times\\{0,1\\}^n.\n\t\t\\\\\n\t\t&\\textbf{Primal Update:}\n\t\t&&\\begin{aligned}[t]\n\t\t\t&\\begin{aligned}\n\t\t\t\tz^{k+1} &=x^k - \\tfrac{\\lambda}{n} \\big(\\tfrac{1}{p_{I^k}}(\\nabla f_{I^k} (x^k) - y_{I^k}^k) + {\\textstyle\\sum_{i=1}^n} y_i^k\\big), \\\\\n\t\t\t\tx^{k+1} &= \\prox_{\\lambda g}(z^{k+1}).\n\t\t\t\\end{aligned}\n\t\t\\end{aligned}\\\\\n\t\t\\\\\n\t\t&\\textbf{Dual Update:}\n\t\t&&\\begin{aligned}[t]\n\t\t\t&\\begin{aligned}\n\t\t\t\ty_i^{k+1} &= y_i^k + U_i^k(\\nabla f_i (x^k) - y_i^k), \\quad \\forall i \\in \\{1,\\dots,n\\}.\n\t\t\t\\end{aligned}\n\t\t\\end{aligned}\n\t\\end{align*}\n\tThe sampling distributions of $(I^k,U_1^k,\\dots,U_n^k)$ for all $k\\in\\{0,1,\\dots\\}$ are the same and independent.\n\tFurthermore, the distribution is such that $P(I^k = i) = p_i > 0$ and the expected update frequency $\\eta_i > 0$, see \\Cref{def:update-freq}, for all $i\\in\\{1,\\dots,n\\}$.\n\tThe step-size satisfies $\\lambda > 0$.\n\\end{algorithm}","sha256_utf8":"f930789d44290a61ba0141ddccec3633bd0d03233a3c10917f8bfe2f20b1c72a"},{"start":16554,"end":16951,"start_line":326,"end_line":333,"raw_tex":"\\begin{defin}[Expected Update Frequency]\\label{def:update-freq}%\n\tLet $U_1^k,\\dots,U_n^k$ be given by the sampling in \\Cref{alg:pvrsg}.\n\tThe expected update frequency of the $i$th dual variable is\n\t\\begin{align*}\n\t\t\\eta_i = \\Exp[U_i^k|\\mathcal{F}^k],\n\t\\end{align*}\n\twhere $\\mathcal{F}^k = \\cup_{i=1}^k \\mathcal{X}^i$ and $\\mathcal{X}^k = \\{x^k,y^k,I^{k-1},U_1^{k-1},\\dots,U_n^{k-1}\\}$.\n\\end{defin}","sha256_utf8":"ca0a524f050fe11042d563fbbfb905bbbfb62dae8d64c6d89e9ac1a36a71c1ca"},{"start":16952,"end":17134,"start_line":334,"end_line":334,"raw_tex":"Note, the expected update frequency does not depend on the iteration number $k$ since $(I^k,U_1^k,\\dots,U_n^k)$ is independently sampled and its distribution does not depend on $k$.\n","sha256_utf8":"874fe6d97b030abefd51687f25200143eca8db322807298de8204e3170783881"},{"start":20280,"end":21382,"start_line":376,"end_line":395,"raw_tex":"\\begin{defin}\\label{def:lyap-terms}%\n\tLet $x^\\star$ be the solution to \\Cref{eq:problem} and $y_i^\\star = \\nabla f_i(x^\\star)$ for all $i\\in\\{1,\\dots,n\\}$.\n\tWith $y = (y_1,\\dots,y_n)$, where $y_i \\in \\R^n$ for $i \\in \\{1,\\dots,n\\}$, we define\n\t\\begin{align*}\n\t\t\\mathcal{P}(x) = \\norm{x - x^\\star}^2 - 2\\lambda\\inprod{\\nabla F(x) - \\nabla F (x^\\star)}{x - x^\\star} + \\lambda^2\\mathcal{V}(x)\n\t\\end{align*}\n\tand\n\t\\begin{align*}\n\t\t\\mathcal{D}(y)\n\t\t&= \\sum_{i=1}^n {\\textstyle(1 - \\eta_i  + \\frac{1}{\\gamma_i})\\widehat{\\gamma}_i} \\norm{y_i - y_i^\\star}^2\n\t\t-{\\textstyle (1+\\delta^{-1})\\lambda^2}\\norm{{{\\textstyle\\frac{1}{n}}\\sum_{i=1}^n} y_i -  y_i^\\star }^2,\n\t\\end{align*}\n\twhere\n\t\\begin{align*}\n\t\t\\mathcal{V}(x)\n\t\t= \\sum_{i=1}^n {\\textstyle \\frac{(1+\\delta)}{n^2 p_i}(\\frac{\\eta_i\\gamma_i}{\\delta}  + 1) }\\norm{\\nabla f_i(x) - \\nabla f_i(x^\\star)}^2 - \\delta\\norm{\\nabla F(x) - \\nabla F(x^\\star)}^2\n\t\\end{align*}\n\twith $\\gamma_i \\geq 0$, $\\delta > 0$ and $\\widehat{\\gamma}_i = \\gamma_i \\frac{(1+\\delta^{-1}) \\lambda^2}{n^2p_i}$.\n\tIf $\\gamma_i = 0$ we define $\\tfrac{\\gamma_i}{\\gamma_i} := 1$.\n\\end{defin}","sha256_utf8":"fa2fb5d7d4aff623bdee12fa5c6d173a68af604fb178bcbd8431bfd8a128ab8c"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{thm}[PVRSG Convergence]\\label{thm:pvrsg-conv}%\n\tGiven $\\max_i \\frac{L_i}{np_i} > \\mu$ and \\Cref{ass:strong-ass}, if there exists $\\rho \\in (0, \\min_i \\eta_i)$ such that\n\t\\begin{align*}\n\t\t\\rho &= \\mu\\lambda(2 - \\nu\\lambda )\\\\\n\t\t\\nu &= \\min_{\\delta > 0} \\max_i {\\textstyle(1+\\delta^{-1})\\frac{L_i}{n p_i}\\frac{\\eta_i}{\\eta_i - \\rho}  +  (1+\\delta)\\frac{L_i}{n p_i} - \\delta\\mu},\n\t\\end{align*}\n\tthen the iterates of \\Cref{alg:pvrsg} converge according to\n\t\\begin{align*}\n\t\t\\Exp \\Big[ \\norm{x^{k} - x^\\star}^2 + \\sum_{i=1}^n \\widehat{\\gamma}_i\\norm{y_i^{k} - y_i^\\star}^2 \\Big] \\in \\mathcal{O}((1 - \\rho)^k)\n\t\\end{align*}\n\twhere $\\widehat{\\gamma}_1,\\dots,\\widehat{\\gamma}_n$ are given by $\\widehat{\\gamma}_i ={\\textstyle \\frac{ \\lambda^2}{n^2p_i} \\frac{1}{\\eta_i - \\rho}(1 + \\frac{1}{\\delta^\\star})}$ and $\\delta^\\star$ is the unique minimizer of the minimization problem defining $\\nu$.\n\n\tIf instead $\\max_i \\frac{L_i}{np_i} = \\mu$ then $\\nu = \\mu + \\mu\\max_i \\tfrac{\\eta_i}{\\eta_i-\\rho}$ and the convergence is such that\n\t\\begin{align*}\n\t\t\\Exp \\Big[ \\norm{x^{k} - x^\\star}^2 + \\sum_{i=1}^n{\\textstyle \\frac{ \\lambda^2}{n^2p_i} \\frac{1}{\\eta_i - \\tilde{\\rho}}} \\norm{y_i^{k} - y_i^\\star}^2 \\Big] \\in \\mathcal{O}((1 - \\tilde{\\rho})^k)\n\t\\end{align*}\n\tholds for all $\\tilde{\\rho} \\in (0, \\rho)$.\n\t\n\n\n\\end{thm}","units":[{"start":25644,"end":26942,"start_line":468,"end_line":485,"raw_tex":"\\begin{thm}[PVRSG Convergence]\\label{thm:pvrsg-conv}%\n\tGiven $\\max_i \\frac{L_i}{np_i} > \\mu$ and \\Cref{ass:strong-ass}, if there exists $\\rho \\in (0, \\min_i \\eta_i)$ such that\n\t\\begin{align*}\n\t\t\\rho &= \\mu\\lambda(2 - \\nu\\lambda )\\\\\n\t\t\\nu &= \\min_{\\delta > 0} \\max_i {\\textstyle(1+\\delta^{-1})\\frac{L_i}{n p_i}\\frac{\\eta_i}{\\eta_i - \\rho}  +  (1+\\delta)\\frac{L_i}{n p_i} - \\delta\\mu},\n\t\\end{align*}\n\tthen the iterates of \\Cref{alg:pvrsg} converge according to\n\t\\begin{align*}\n\t\t\\Exp \\Big[ \\norm{x^{k} - x^\\star}^2 + \\sum_{i=1}^n \\widehat{\\gamma}_i\\norm{y_i^{k} - y_i^\\star}^2 \\Big] \\in \\mathcal{O}((1 - \\rho)^k)\n\t\\end{align*}\n\twhere $\\widehat{\\gamma}_1,\\dots,\\widehat{\\gamma}_n$ are given by $\\widehat{\\gamma}_i ={\\textstyle \\frac{ \\lambda^2}{n^2p_i} \\frac{1}{\\eta_i - \\rho}(1 + \\frac{1}{\\delta^\\star})}$ and $\\delta^\\star$ is the unique minimizer of the minimization problem defining $\\nu$.\n\n\tIf instead $\\max_i \\frac{L_i}{np_i} = \\mu$ then $\\nu = \\mu + \\mu\\max_i \\tfrac{\\eta_i}{\\eta_i-\\rho}$ and the convergence is such that\n\t\\begin{align*}\n\t\t\\Exp \\Big[ \\norm{x^{k} - x^\\star}^2 + \\sum_{i=1}^n{\\textstyle \\frac{ \\lambda^2}{n^2p_i} \\frac{1}{\\eta_i - \\tilde{\\rho}}} \\norm{y_i^{k} - y_i^\\star}^2 \\Big] \\in \\mathcal{O}((1 - \\tilde{\\rho})^k)\n\t\\end{align*}\n\tholds for all $\\tilde{\\rho} \\in (0, \\rho)$.\n\t","sha256_utf8":"8a8b78a9f31b0938f849396161a5d149d5a1d30d8cc4b6d483ca24fb54d1c9d7"},{"start":27001,"end":27011,"start_line":487,"end_line":488,"raw_tex":"\n\\end{thm}","sha256_utf8":"27661e277ceb78e4dd676b4ea4b5c8d9a4d3c607a1a758eac095282a4f311f77"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{prop}\\label{prop:main-conv}%\n\tLet the filtration $\\mathcal{F}^k = \\cup_{i=1}^k \\mathcal{X}^i$ be given by the state $\\mathcal{X}^k = \\{x^k,y^k,I^{k-1},U_1^{k-1},\\dots,U_n^{k-1}\\}$.\n\tIf \\Cref{ass:strong-ass} holds, the iterates of \\Cref{alg:pvrsg} satisfy\n\t\\begin{equation}\\label{eq:prop-lyapunov-bound}\n\t\t\\Exp \\big[ \\norm{x^{k+1} - x^\\star}^2 + \\sum_{i=1}^n \\widehat{\\gamma}_i\\norm{y_i^{k+1} - y_i^\\star}^2 \\big| \\mathcal{F}^k \\big] \\leq  \\mathcal{P}(x^k) + \\mathcal{D}(y^k),\n\t\\end{equation}\n\twhere $x^\\star$ is the unique solution of \\Cref{eq:problem} and $y_i^\\star = \\nabla f_i(x^\\star)$.\n\tSee \\Cref{def:lyap-terms} for $\\mathcal{P}$, $\\mathcal{D}$, and $\\mathcal{V}$.\n\n\tIf the primal updates satisfy\n\t\\begin{align}\\label{eq:prim-contract}\n\t\t\\mathcal{P}(x^k) \\leq (1-\\rho_P)\\norm{x^k - x^\\star}^2\n\t\\end{align}\n\tand the dual updates satisfy\n\t\\begin{align}\\label{eq:dual-contract}\n\t\t\\mathcal{D}(y^k) \\leq (1-\\rho_D)\\sum_{i=1}^n \\widehat{\\gamma}_i\\norm{y_i^k - y_i^\\star}^2\n\t\\end{align}\n\twith $\\rho_P, \\rho_D \\in (0,1]$ then \\Cref{alg:pvrsg} converges linearly according to\n\t\\begin{align*}\n\t\t\\Exp \\big[ \\norm{x^{k} - x^\\star}^2 + \\sum_{i=1}^n \\widehat{\\gamma}_i\\norm{y_i^{k} - y_i^\\star}^2 \\big] \\in \\mathcal{O}((1 - \\min(\\rho_P, \\rho_D))^k).\n\t\\end{align*}\n\t\\begin{proof}\n\t\tSee \\Cref{sec:proof-prop-lemma}.\n\t\\end{proof}\n\\end{prop}\n\n\\begin{lem}[Primal Contraction]\\label{lem:prim-contract}%\n\tLet \\Cref{ass:strong-ass} hold, the primal iterates of \\Cref{alg:pvrsg} satisfy the primal contraction \\Cref{eq:prim-contract} with\n\t\\begin{align*}\n\t\t\\rho_P = \\mu\\lambda(2 - \\nu\\lambda)\n\t\\end{align*}\n\twhere $\\nu = \\max_i {\\textstyle  (1+\\delta^{-1})\\frac{L_i\\eta_i\\gamma_i}{n p_i}  +  (1+\\delta)\\frac{L_i}{n p_i} - \\delta\\mu}$.\n\t\\begin{proof}\n\t\tSee \\Cref{sec:proof-prop-lemma}.\n\t\\end{proof}\n\\end{lem}\n\n\\begin{lem}[Dual Contraction]\\label{lem:dual-contract}%\n\tLet \\Cref{ass:strong-ass} and $\\gamma_i > 0$ hold for all $i \\in\\{1,\\dots,n\\}$, the dual iterates of \\Cref{alg:pvrsg} satisfy the dual contraction \\Cref{eq:dual-contract} with\n\t\\begin{align*}\n\t\t\\rho_D = \\min_i {\\textstyle \\eta_i - \\tfrac{1}{\\gamma_i}}.\n\t\\end{align*}\n\t\\begin{proof}\n\t\tSee \\Cref{sec:proof-prop-lemma}.\n\t\\end{proof}\n\\end{lem}\n\n\\begin{lem}\\label{lem:sampled-lipschitz}%\n\tLet $L_i$ and $\\mu$ be from \\Cref{ass:strong-ass} and $p_i$ from \\Cref{alg:pvrsg}, then $\\max_i \\frac{L_i}{np_i} \\geq \\mu$.\n\tFurthermore, if $\\max_i \\frac{L_i}{np_i} = \\mu$ then $\\frac{L_i}{np_i} = \\mu$ for all $i \\in \\{1,\\dots,n\\}$.\n\t\\begin{proof}\n\t\tSee \\Cref{sec:proof-prop-lemma}.\n\t\\end{proof}\n\\end{lem}\n\n\\begin{proof}\n\t\tSee \\Cref{sec:proof-theorems}.\n\t\\end{proof}\n\n\\begin{proof}[\\textbf{Proof of \\Cref{prop:main-conv}}]%\n\tLet $x^\\star$ be the unique solution to \\Cref{eq:problem}.\n\tWith $g$ being proper, closed and convex, the primal updates satisfy\n\t\\begin{equation}\\label{eq:prim-update-lyap}\n\t\t\\begin{aligned}\n\t\t\t&\\Exp[\\norm{x^{k+1} - x^\\star}^2|\\mathcal{F}^k] \\\\\n\t\t\t&\\quad= \\Exp[\\norm{\\prox_{\\lambda g}(z^{k+1}) - \\prox_{\\lambda g}(x^\\star - \\lambda \\nabla F(x^\\star))}^2|\\mathcal{F}^k] \\\\\n\t\t\t&\\quad\\leq\n\t\t\t%\\Exp[\\norm{z^{k+1} - (x^\\star - \\lambda \\nabla F(x^\\star))}^2|\\mathcal{F}^k] \\\\\n\t\t\t%&\\quad=\n\t\t\t\\Exp[\\norm{x^k - \\tfrac{\\lambda}{n}\\big(\\tfrac{1}{p_{I^k}}(\\nabla f_{I^k}(x^k) - y_{I^k}^k) + \\textstyle\\sum_{i=1}^n y_i^k\\big) - x^\\star + \\lambda \\nabla F(x^\\star)}^2|\\mathcal{F}^k] \\\\\n\t\t\t&\\quad= \\norm{(x^k - \\lambda \\nabla F(x^k)) - (x^\\star - \\lambda \\nabla F(x^\\star))}^2\\\\\n\t\t&\\quad\\quad + \\lambda^2 \\Exp\\big[\\norm[\\big]{\\big(\\tfrac{1}{n p_{I^k}} \\nabla f_{I^k}(x^k) - \\nabla F(x^k)\\big) - \\big(\\tfrac{1}{n p_{I^k}}y_{I^k}^k - {\\textstyle\\frac{1}{n}\\sum_{i=1}^n} y_i^k \\big)}^2|\\mathcal{F}^k\\big] \\\\\n\t\t&\\quad= \\norm{x^k - x^\\star}^2 - 2\\lambda\\inprod{\\nabla F(x^k) - \\nabla F(x^\\star)}{x^k-x^\\star} + \\lambda^2\\norm{\\nabla F(x^k) -\\nabla F(x^\\star)}^2\\\\\n\t&\\quad\\quad + \\lambda^2 \\Exp\\big[\\norm[\\big]{\\big(\\tfrac{1}{n p_{I^k}} \\nabla f_{I^k}(x^k) - \\nabla F(x^k)\\big) - \\big(\\tfrac{1}{n p_{I^k}}y_{I^k}^k - {\\textstyle\\frac{1}{n}\\sum_{i=1}^n} y_i^k \\big)}^2|\\mathcal{F}^k\\big].\n\\end{aligned}\n\\end{equation}\nThe first equality is given by the solution being a fixed point to the proximal-gradient update $x^\\star = \\prox_{\\lambda g}(x^\\star - \\lambda \\nabla F(x^\\star))$ \\cite[Corollary 28.9]{bauschke_convex_2017}.\nThe first inequality is due to the non-expansiveness of $\\prox_{\\lambda g}$.\nThe second to last equality is given by $\\Exp\\norm{X}^2 =\\norm{\\Exp X}^2 + \\Exp\\norm{X - \\Exp X}^2$ where $X$ is a random variable.\n\nThe last term in \\Cref{eq:prim-update-lyap} satisfies the following upper bound for all $\\delta > 0$:\n\\begin{equation}\\label{eq:update-variance-bound}\n\t\\begin{aligned}\n\t\t&\\Exp[\\norm{(\\tfrac{1}{n p_{I^k}} \\nabla f_{I^k}(x^k) - \\nabla F(x^k)) - (\\tfrac{1}{n p_{I^k}}y_{I^k}^k - {\\textstyle\\frac{1}{n}\\sum_{i=1}^n} y_i^k )}^2|\\mathcal{F}^k]\\\\\n\t\t&\\quad\\begin{aligned}\n\t\t\t= \\Exp[\\| &\\tfrac{1}{np_{I^k}}(\\nabla f_{I^k}(x^k) - \\nabla f_{I^k}(x^\\star)) - (\\nabla F(x^k) - \\nabla F(x^\\star))\\\\\n\t\t\t&- \\tfrac{1}{np_{I^k}}(y_{I^k}^k - y_{I^k}^\\star) + {\\textstyle\\frac{1}{n}\\sum_{i=1}^n}(y_i^k - y_i^\\star) \\|^2 |\\mathcal{F}^k]\n\t\t\\end{aligned} \\\\\n\t\t&\\quad\\leq (1+\\delta)\\Exp[\\norm{ \\tfrac{1}{np_{I^k}}(\\nabla f_{I^k}(x^k) - \\nabla f_{I^k}(x^\\star)) - (\\nabla F(x^k) - \\nabla F(x^\\star)) }^2 |\\mathcal{F}^k]\\\\\n\t\t&\\quad\\quad + (1+\\delta^{-1})\\Exp[\\norm{\\tfrac{1}{np_{I^k}}(y_{I^k}^k - y_{I^k}^\\star) - {\\textstyle\\frac{1}{n}\\sum_{i=1}^n}(y_i^k - y_i^\\star)}^2 |\\mathcal{F}^k]\\\\\n\t\t&\\quad= (1+\\delta)\\big(\\Exp[\\norm{\\tfrac{1}{np_{I^k}}(\\nabla f_{I^k}(x^k) - \\nabla f_{I^k}(x^\\star))}^2|\\mathcal{F}^k] - \\norm{\\nabla F(x^k) - \\nabla F(x^\\star)}^2\\big) \\\\\n\t\t&\\quad\\quad + (1+\\delta^{-1})\\big(\\Exp[\\norm{\\tfrac{1}{np_{I^k}}(y_{I^k}^k - y_{I^k}^\\star)}^2|\\mathcal{F}^k] - \\norm{{\\textstyle\\frac{1}{n}\\sum_{i=1}^n}(y_i^k - y_i^\\star)}^2\\big) \\\\\n\t\t&\\quad= (1+\\delta){\\textstyle\\sum\\frac{1}{n^2 p_i}}\\norm{\\nabla f_i(x^k) - \\nabla f_i(x^\\star)}^2 - (1+\\delta)\\norm{\\nabla F(x^k) - \\nabla F(x^\\star)}^2 \\\\\n\t\t&\\quad\\quad + (1+\\delta^{-1}){\\textstyle\\sum\\frac{1}{n^2 p_i}}\\norm{y_i^k - y_i^\\star}^2 - (1+\\delta^{-1})\\norm{{\\textstyle\\frac{1}{n}\\sum_{i=1}^n}(y_i^k - y_i^\\star)}^2.\n\t\\end{aligned}\n\\end{equation}\nThe inequality is given by Young's inequality, the second to last equality is given by $\\Exp\\norm{X - \\Exp X}^2 = \\Exp\\norm{X}^2-\\norm{\\Exp X}^2$ where $X$ is a random variable.\n\nThe dual updates satisfy\n\\begin{align*}\n\t\\norm{y_i^{k+1} - y_i^\\star}^2\n\t&= \\norm{y_i^k + U_i^k(\\nabla f_i(x^k) - y_i^k) - y_i^\\star}^2 \\\\\n\t&= (1- U_i^k)\\norm{y_i^k - y_i^\\star}^2 + U_i^k\\norm{\\nabla f_i(x^k) - \\nabla f_i(x^\\star)}^2\n\\end{align*}\nsince $U_i^k \\in \\{0,1\\}$.\nSumming over all terms, taking expected value and using linearity of the expected value give\n\\begin{equation}\\label{eq:dual-update-lyap}\n\t\\begin{aligned}\n\t\t\\Exp[{\\textstyle\\sum_{i=1}^n} \\widehat{\\gamma}_i\\norm{y_i^{k+1} - y_i^\\star}^2|\\mathcal{F}^k]\n\t\t&= {\\textstyle\\sum_{i=1}^n} (1-\\eta_i)\\widehat{\\gamma}_i\\norm{y^k - y^\\star}^2 \\\\\n\t\t&\\quad + {\\textstyle\\sum_{i=1}^n}{\\eta_i \\widehat{\\gamma}_i}\\norm{\\nabla f_i(x^k) - \\nabla f_i(x^\\star)}^2.\n\t\\end{aligned}\n\\end{equation}\nAdding \\Cref{eq:dual-update-lyap} to \\Cref{eq:prim-update-lyap} and substituting in \\Cref{eq:update-variance-bound} and using the definition $\\frac{\\gamma_i}{\\gamma_i} = 1$ when $\\gamma_i = 0$ then yield \\Cref{eq:prop-lyapunov-bound}.\nApplying \\Cref{eq:prim-contract} and \\Cref{eq:dual-contract}, using the law of total expectation and telescoping the inequalities give the stated rate.\n\\end{proof}\n\n\\begin{proof}[\\textbf{Proof of \\Cref{lem:prim-contract}}]%\n\tFirst note that $\\mu$-strong monotonicity and the Cauchy--Schwarz inequality imply\n\t\\begin{equation}\\label{eq:reverse-lipschitz}\n\t\t\\norm{\\nabla F(x^k) - \\nabla F(x^\\star)} \\geq \\mu \\norm{x^k - x^\\star}.\n\t\\end{equation}\n\tConsider the terms of $\\mathcal{V}(x^k)$.\n\tUsing \\Cref{eq:reverse-lipschitz} and Cauchy--Schwarz in the last term yield\n\t\\begin{align*}\n\t\t\\norm{\\nabla F(x^k) - \\nabla F(x^\\star)}^2\n\t\t&\\geq \\mu\\norm{\\nabla F(x^k) - \\nabla F(x^\\star)}\\norm{x^k - x^\\star} \\\\\n\t\t&\\geq \\mu\\inprod{\\nabla F(x^k) - \\nabla F(x^\\star)}{x^k - x^\\star}.\n\t\\end{align*}\n\tUsing $\\frac{1}{L_i}$-cocoercivity of $\\nabla f_i$ in the first term of $\\mathcal{V}(x^k)$ yields\n\t\\begin{align*}\n\t\t\\sum_{i=1}^n & {\\textstyle (\\frac{\\eta_i\\gamma_i}{\\delta}  + 1)\\frac{1}{n^2 p_i} }\\norm{\\nabla f_i(x^k) - \\nabla f_i(x^\\star)}^2 \\\\\n\t\t&\\leq \\sum_{i=1}^n {\\textstyle (\\frac{\\eta_i\\gamma_i}{\\delta}  + 1)\\frac{L_i}{n^2 p_i} } \\inprod{\\nabla f_i(x^k) - \\nabla f_i(x^\\star)}{x^k - x^\\star} \\\\\n\t\t&\\leq \\max_i({\\textstyle(\\frac{\\eta_i\\gamma_i}{\\delta}  + 1)\\frac{L_i}{n p_i}}) \\inprod{\\nabla F(x^k) - \\nabla F(x^\\star)}{x^k - x^\\star}.\n\t\\end{align*}\n\tAdding the terms back together yields\n\t\\begin{align*}\n\t\t\\mathcal{V}(x^k)\n\t\t&\\leq \\lambda^2 \\nu \\inprod{\\nabla F(x^k) - \\nabla F(x^\\star)}{x^k - x^\\star}\n\t\\end{align*}\n\twhere $\\nu = \\max_i(  (1+\\delta^{-1})\\frac{L_i\\eta_i\\gamma_i}{n p_i}  +  (1+\\delta)\\frac{L_i}{n p_i} - \\delta\\mu)$.\n\tThis can now be summarized as\n\t\\begin{align*}\n\t\t\\mathcal{P}(x^k)\n\t\t&\\leq \\norm{x^k - x^\\star}^2  - \\lambda\\big(2 - \\nu\\lambda \\big) \\inprod{\\nabla F(x^k) - \\nabla F (x^\\star)}{x^k - x^\\star} \\\\\n\t\t&\\leq \\norm{x^k - x^\\star}^2  - \\mu\\lambda\\big(2 - \\nu\\lambda \\big) \\norm{x^k - x^\\star}^2 \\\\\n\t\t&= (1 - \\rho_P)\\norm{x^k - x^\\star}^2,\n\t\\end{align*}\n\twhere $\\rho_P = \\mu\\lambda(2 - \\nu\\lambda)$ and the last inequality is given by the strong monotonicity of $\\nabla F$.\n\\end{proof}\n\n\\begin{proof}[\\textbf{Proof of \\Cref{lem:dual-contract}}]%\n\tSince norms are non-negative, we have\n\t\\begin{align*}\n\t\t\\mathcal{D}(y^k)\n\t\t\\leq \\sum_{i=1}^n {\\textstyle(1 - \\eta_i  + \\frac{1}{\\gamma_i})\\widehat{\\gamma}_i} \\norm{y_i^k - y_i^\\star}^2\n\t\t\\leq (1 - \\rho_D)\\sum_{i=1}^n \\widehat{\\gamma}_i \\norm{y_i^k - y_i^\\star}^2,\n\t\\end{align*}\n\twhere $\\rho_D = \\min_i(\\eta_i - \\tfrac{1}{\\gamma_i})$.\n\\end{proof}\n\n\\begin{proof}[\\textbf{Proof of \\Cref{lem:sampled-lipschitz}}]%\n\tAn $L$-smooth and $\\mu$-strongly convex function must satisfy $L \\geq \\mu$.\n\tAssuming $\\max_i(\\frac{L_i}{np_i}) < \\mu$ yields the following contradiction\n\t\\begin{align*}\n\t\t\\mu > \\max_i \\tfrac{L_i}{np_i} = \\sum_{j=1}^n  p_j \\max_i \\tfrac{L_i}{np_i} \\geq \\sum_{j=1}^n  p_j \\tfrac{L_j}{np_j} = \\sum_{i=1}^n \\tfrac{L_i}{n} \\geq L.\n\t\\end{align*}\n\tIf $\\max_i(\\frac{L_i}{np_i}) = \\mu$, equality must hold everywhere and we have\n\t\\begin{align*}\n\t\t0 = \\sum_{j=1}^n  p_j \\max_i \\tfrac{L_i}{np_i} - \\sum_{j=1}^n  p_j \\tfrac{L_j}{np_j} = \\sum_{j=1}^n  p_j (\\max_i \\tfrac{L_i}{np_i} - \\tfrac{L_j}{np_j}).\n\t\\end{align*}\n\tSince $p_j > 0$ and $\\max_i \\tfrac{L_i}{np_i} - \\tfrac{L_j}{np_j} \\geq 0$, we have $\\max_i \\tfrac{L_i}{np_i} = \\tfrac{L_j}{np_j}$ for all $j\\in\\{1,\\dots,n\\}$.\n\\end{proof}\n\n\\begin{proof}[\\textbf{Proof of \\Cref{thm:pvrsg-conv}}]%\n\tApplication of \\Cref{lem:prim-contract} and \\ref{lem:dual-contract} in \\Cref{prop:main-conv} yields the convergence rate\n\t\\begin{align*}\n\t\t\\Exp \\Big[ \\norm{x^{k} - x^\\star}^2 + \\sum_{i=1}^n \\widehat{\\gamma}_i\\norm{y_i^{k} - y_i^\\star}^2 \\Big] \\in \\mathcal{O}((1 - \\min(\\rho_P, \\rho_D))^k)\n\t\\end{align*}\n\twith\n\t\\begin{align*}\n\t\t\\rho_P &= \\mu\\lambda(2- \\nu\\lambda) \\\\\n\t\t\\rho_D &= \\min_i \\eta_i - \\tfrac{1}{\\gamma_i} \\\\\n\t\t\\nu &= \\max_i  {\\textstyle(1+\\delta^{-1})\\frac{L_i\\eta_i\\gamma_i}{n p_i}  +  (1+\\delta)\\frac{L_i}{n p_i} - \\delta\\mu},\n\t\\end{align*}\n\twhich hold for all choices of $\\delta > 0$ and $\\gamma_i > 0$ for all $i\\in\\{1,\\dots,n\\}$.\n\tIf there exists $\\delta$ and $\\gamma_1,\\dots,\\gamma_n$ such that $\\min(\\rho_P,\\rho_D) \\in (0,1]$ we have convergence.\n\tWe restrict ourselves to only search for $\\delta$ and $\\gamma_1,\\dots,\\gamma_n$ such that $\\rho_P = \\rho_D = \\rho$ for some $\\rho \\in (0,1]$.\n\tFor all $i\\in\\{1,\\dots,n\\}$, select $\\gamma_i = \\frac{1}{\\eta_i - \\rho}$, which is positive when $\\rho < \\eta_i$, and convergence is then proved if there exists $\\rho \\in (0,\\min_i \\eta_i)$ and $\\delta > 0$ such that\n\t\\begin{align*}\n\t\t\\rho &= \\mu\\lambda(2 - \\nu\\lambda )\\\\\n\t\t\\nu &= \\max_i {\\textstyle(1+\\delta^{-1})\\frac{L_i}{n p_i}\\frac{\\eta_i}{\\eta_i - \\rho}  +  (1+\\delta)\\frac{L_i}{n p_i} - \\delta\\mu}.\n\t\\end{align*}\n\tThe variable $\\nu$ can be minimized w.r.t. $\\delta$ if $\\max_i \\frac{L_i}{np_i} > \\mu$.\n\tThe minimum then exists and is unique since $\\nu$ as a function of $\\delta$ is continuous, strictly convex, and $\\nu\\to\\infty$ both when $\\delta \\to 0^+$ and $\\delta \\to \\infty$.\n\tCalling the minimum point $\\delta^\\star$, noting that $\\delta^\\star > 0$, and inserting it and the choice of $\\gamma_i$ in the expression for $\\widehat{\\gamma}_i$ from \\Cref{def:lyap-terms} yield the first statement of the theorem.\n\n\tWhen $\\max_i \\frac{L_i}{np_i} \\not> \\mu$, \\Cref{lem:sampled-lipschitz} gives $\\frac{L_i}{n p_i} = \\mu$ for all $i\\in\\{1,\\dots,n\\}$ and\n\t\\begin{align*}\n\t\t\\nu = \\mu + \\mu(1+\\delta^{-1})\\max_i \\tfrac{\\eta_i}{\\eta_i-\\rho}.\n\t\\end{align*}\n\tThis can not be minimized w.r.t. $\\delta$ since the $\\inf$ is not attained.\n\tHowever, any $\\delta>0$ will yield a valid $\\rho$ and $\\widehat{\\gamma}_i$, giving the rate\n\t\\begin{align*}\n\t\t&\\Exp \\Big[ \\norm{x^{k} - x^\\star}^2 + \\sum_{i=1}^n{\\textstyle \\frac{ \\lambda^2}{n^2p_i} \\frac{1}{\\eta_i - \\rho}} \\norm{y_i^{k} - y_i^\\star}^2 \\Big] \\\\\n\t\t&\\quad \\leq \\Exp \\Big[ \\norm{x^{k} - x^\\star}^2 + \\sum_{i=1}^n \\widehat{\\gamma}_i\\norm{y_i^{k} - y_i^\\star}^2 \\Big]\n\t\t\\in \\mathcal{O}((1 - \\rho)^k).\n\t\\end{align*}\n\tTaking the limit as $\\delta \\to \\infty$ results in the stated interval.\n\\end{proof}","units":[{"start":21589,"end":22926,"start_line":398,"end_line":422,"raw_tex":"\\begin{prop}\\label{prop:main-conv}%\n\tLet the filtration $\\mathcal{F}^k = \\cup_{i=1}^k \\mathcal{X}^i$ be given by the state $\\mathcal{X}^k = \\{x^k,y^k,I^{k-1},U_1^{k-1},\\dots,U_n^{k-1}\\}$.\n\tIf \\Cref{ass:strong-ass} holds, the iterates of \\Cref{alg:pvrsg} satisfy\n\t\\begin{equation}\\label{eq:prop-lyapunov-bound}\n\t\t\\Exp \\big[ \\norm{x^{k+1} - x^\\star}^2 + \\sum_{i=1}^n \\widehat{\\gamma}_i\\norm{y_i^{k+1} - y_i^\\star}^2 \\big| \\mathcal{F}^k \\big] \\leq  \\mathcal{P}(x^k) + \\mathcal{D}(y^k),\n\t\\end{equation}\n\twhere $x^\\star$ is the unique solution of \\Cref{eq:problem} and $y_i^\\star = \\nabla f_i(x^\\star)$.\n\tSee \\Cref{def:lyap-terms} for $\\mathcal{P}$, $\\mathcal{D}$, and $\\mathcal{V}$.\n\n\tIf the primal updates satisfy\n\t\\begin{align}\\label{eq:prim-contract}\n\t\t\\mathcal{P}(x^k) \\leq (1-\\rho_P)\\norm{x^k - x^\\star}^2\n\t\\end{align}\n\tand the dual updates satisfy\n\t\\begin{align}\\label{eq:dual-contract}\n\t\t\\mathcal{D}(y^k) \\leq (1-\\rho_D)\\sum_{i=1}^n \\widehat{\\gamma}_i\\norm{y_i^k - y_i^\\star}^2\n\t\\end{align}\n\twith $\\rho_P, \\rho_D \\in (0,1]$ then \\Cref{alg:pvrsg} converges linearly according to\n\t\\begin{align*}\n\t\t\\Exp \\big[ \\norm{x^{k} - x^\\star}^2 + \\sum_{i=1}^n \\widehat{\\gamma}_i\\norm{y_i^{k} - y_i^\\star}^2 \\big] \\in \\mathcal{O}((1 - \\min(\\rho_P, \\rho_D))^k).\n\t\\end{align*}\n\t\\begin{proof}\n\t\tSee \\Cref{sec:proof-prop-lemma}.\n\t\\end{proof}\n\\end{prop}","sha256_utf8":"51b274f9603e228298fd797d1f0ff43c139f959d95f3457abc4b02edbc03b922"},{"start":23257,"end":23716,"start_line":425,"end_line":434,"raw_tex":"\\begin{lem}[Primal Contraction]\\label{lem:prim-contract}%\n\tLet \\Cref{ass:strong-ass} hold, the primal iterates of \\Cref{alg:pvrsg} satisfy the primal contraction \\Cref{eq:prim-contract} with\n\t\\begin{align*}\n\t\t\\rho_P = \\mu\\lambda(2 - \\nu\\lambda)\n\t\\end{align*}\n\twhere $\\nu = \\max_i {\\textstyle  (1+\\delta^{-1})\\frac{L_i\\eta_i\\gamma_i}{n p_i}  +  (1+\\delta)\\frac{L_i}{n p_i} - \\delta\\mu}$.\n\t\\begin{proof}\n\t\tSee \\Cref{sec:proof-prop-lemma}.\n\t\\end{proof}\n\\end{lem}","sha256_utf8":"9fbbdc3a5597879392fe98f6966b1a878f252734c00d0c63aeef3b9a2f568a83"},{"start":23717,"end":24113,"start_line":435,"end_line":443,"raw_tex":"\\begin{lem}[Dual Contraction]\\label{lem:dual-contract}%\n\tLet \\Cref{ass:strong-ass} and $\\gamma_i > 0$ hold for all $i \\in\\{1,\\dots,n\\}$, the dual iterates of \\Cref{alg:pvrsg} satisfy the dual contraction \\Cref{eq:dual-contract} with\n\t\\begin{align*}\n\t\t\\rho_D = \\min_i {\\textstyle \\eta_i - \\tfrac{1}{\\gamma_i}}.\n\t\\end{align*}\n\t\\begin{proof}\n\t\tSee \\Cref{sec:proof-prop-lemma}.\n\t\\end{proof}\n\\end{lem}","sha256_utf8":"fd3d1921d5ee423488e4cb7c1e28bede227ec0512f7450ad8c89e2ae0a60bcea"},{"start":25245,"end":25594,"start_line":460,"end_line":466,"raw_tex":"\\begin{lem}\\label{lem:sampled-lipschitz}%\n\tLet $L_i$ and $\\mu$ be from \\Cref{ass:strong-ass} and $p_i$ from \\Cref{alg:pvrsg}, then $\\max_i \\frac{L_i}{np_i} \\geq \\mu$.\n\tFurthermore, if $\\max_i \\frac{L_i}{np_i} = \\mu$ then $\\frac{L_i}{np_i} = \\mu$ for all $i \\in \\{1,\\dots,n\\}$.\n\t\\begin{proof}\n\t\tSee \\Cref{sec:proof-prop-lemma}.\n\t\\end{proof}\n\\end{lem}","sha256_utf8":"80c2bfc12f8cfde5f1e2d809a2b601804836dec5e7dc4592d03eb1eff3d8c59a"},{"start":26942,"end":27001,"start_line":485,"end_line":487,"raw_tex":"\\begin{proof}\n\t\tSee \\Cref{sec:proof-theorems}.\n\t\\end{proof}","sha256_utf8":"56f47e9dead4585c044ff7375a027a6a2bc39d2500de4d7b3fb97d73a47cfdae"},{"start":45971,"end":50853,"start_line":777,"end_line":833,"raw_tex":"\\begin{proof}[\\textbf{Proof of \\Cref{prop:main-conv}}]%\n\tLet $x^\\star$ be the unique solution to \\Cref{eq:problem}.\n\tWith $g$ being proper, closed and convex, the primal updates satisfy\n\t\\begin{equation}\\label{eq:prim-update-lyap}\n\t\t\\begin{aligned}\n\t\t\t&\\Exp[\\norm{x^{k+1} - x^\\star}^2|\\mathcal{F}^k] \\\\\n\t\t\t&\\quad= \\Exp[\\norm{\\prox_{\\lambda g}(z^{k+1}) - \\prox_{\\lambda g}(x^\\star - \\lambda \\nabla F(x^\\star))}^2|\\mathcal{F}^k] \\\\\n\t\t\t&\\quad\\leq\n\t\t\t%\\Exp[\\norm{z^{k+1} - (x^\\star - \\lambda \\nabla F(x^\\star))}^2|\\mathcal{F}^k] \\\\\n\t\t\t%&\\quad=\n\t\t\t\\Exp[\\norm{x^k - \\tfrac{\\lambda}{n}\\big(\\tfrac{1}{p_{I^k}}(\\nabla f_{I^k}(x^k) - y_{I^k}^k) + \\textstyle\\sum_{i=1}^n y_i^k\\big) - x^\\star + \\lambda \\nabla F(x^\\star)}^2|\\mathcal{F}^k] \\\\\n\t\t\t&\\quad= \\norm{(x^k - \\lambda \\nabla F(x^k)) - (x^\\star - \\lambda \\nabla F(x^\\star))}^2\\\\\n\t\t&\\quad\\quad + \\lambda^2 \\Exp\\big[\\norm[\\big]{\\big(\\tfrac{1}{n p_{I^k}} \\nabla f_{I^k}(x^k) - \\nabla F(x^k)\\big) - \\big(\\tfrac{1}{n p_{I^k}}y_{I^k}^k - {\\textstyle\\frac{1}{n}\\sum_{i=1}^n} y_i^k \\big)}^2|\\mathcal{F}^k\\big] \\\\\n\t\t&\\quad= \\norm{x^k - x^\\star}^2 - 2\\lambda\\inprod{\\nabla F(x^k) - \\nabla F(x^\\star)}{x^k-x^\\star} + \\lambda^2\\norm{\\nabla F(x^k) -\\nabla F(x^\\star)}^2\\\\\n\t&\\quad\\quad + \\lambda^2 \\Exp\\big[\\norm[\\big]{\\big(\\tfrac{1}{n p_{I^k}} \\nabla f_{I^k}(x^k) - \\nabla F(x^k)\\big) - \\big(\\tfrac{1}{n p_{I^k}}y_{I^k}^k - {\\textstyle\\frac{1}{n}\\sum_{i=1}^n} y_i^k \\big)}^2|\\mathcal{F}^k\\big].\n\\end{aligned}\n\\end{equation}\nThe first equality is given by the solution being a fixed point to the proximal-gradient update $x^\\star = \\prox_{\\lambda g}(x^\\star - \\lambda \\nabla F(x^\\star))$ \\cite[Corollary 28.9]{bauschke_convex_2017}.\nThe first inequality is due to the non-expansiveness of $\\prox_{\\lambda g}$.\nThe second to last equality is given by $\\Exp\\norm{X}^2 =\\norm{\\Exp X}^2 + \\Exp\\norm{X - \\Exp X}^2$ where $X$ is a random variable.\n\nThe last term in \\Cref{eq:prim-update-lyap} satisfies the following upper bound for all $\\delta > 0$:\n\\begin{equation}\\label{eq:update-variance-bound}\n\t\\begin{aligned}\n\t\t&\\Exp[\\norm{(\\tfrac{1}{n p_{I^k}} \\nabla f_{I^k}(x^k) - \\nabla F(x^k)) - (\\tfrac{1}{n p_{I^k}}y_{I^k}^k - {\\textstyle\\frac{1}{n}\\sum_{i=1}^n} y_i^k )}^2|\\mathcal{F}^k]\\\\\n\t\t&\\quad\\begin{aligned}\n\t\t\t= \\Exp[\\| &\\tfrac{1}{np_{I^k}}(\\nabla f_{I^k}(x^k) - \\nabla f_{I^k}(x^\\star)) - (\\nabla F(x^k) - \\nabla F(x^\\star))\\\\\n\t\t\t&- \\tfrac{1}{np_{I^k}}(y_{I^k}^k - y_{I^k}^\\star) + {\\textstyle\\frac{1}{n}\\sum_{i=1}^n}(y_i^k - y_i^\\star) \\|^2 |\\mathcal{F}^k]\n\t\t\\end{aligned} \\\\\n\t\t&\\quad\\leq (1+\\delta)\\Exp[\\norm{ \\tfrac{1}{np_{I^k}}(\\nabla f_{I^k}(x^k) - \\nabla f_{I^k}(x^\\star)) - (\\nabla F(x^k) - \\nabla F(x^\\star)) }^2 |\\mathcal{F}^k]\\\\\n\t\t&\\quad\\quad + (1+\\delta^{-1})\\Exp[\\norm{\\tfrac{1}{np_{I^k}}(y_{I^k}^k - y_{I^k}^\\star) - {\\textstyle\\frac{1}{n}\\sum_{i=1}^n}(y_i^k - y_i^\\star)}^2 |\\mathcal{F}^k]\\\\\n\t\t&\\quad= (1+\\delta)\\big(\\Exp[\\norm{\\tfrac{1}{np_{I^k}}(\\nabla f_{I^k}(x^k) - \\nabla f_{I^k}(x^\\star))}^2|\\mathcal{F}^k] - \\norm{\\nabla F(x^k) - \\nabla F(x^\\star)}^2\\big) \\\\\n\t\t&\\quad\\quad + (1+\\delta^{-1})\\big(\\Exp[\\norm{\\tfrac{1}{np_{I^k}}(y_{I^k}^k - y_{I^k}^\\star)}^2|\\mathcal{F}^k] - \\norm{{\\textstyle\\frac{1}{n}\\sum_{i=1}^n}(y_i^k - y_i^\\star)}^2\\big) \\\\\n\t\t&\\quad= (1+\\delta){\\textstyle\\sum\\frac{1}{n^2 p_i}}\\norm{\\nabla f_i(x^k) - \\nabla f_i(x^\\star)}^2 - (1+\\delta)\\norm{\\nabla F(x^k) - \\nabla F(x^\\star)}^2 \\\\\n\t\t&\\quad\\quad + (1+\\delta^{-1}){\\textstyle\\sum\\frac{1}{n^2 p_i}}\\norm{y_i^k - y_i^\\star}^2 - (1+\\delta^{-1})\\norm{{\\textstyle\\frac{1}{n}\\sum_{i=1}^n}(y_i^k - y_i^\\star)}^2.\n\t\\end{aligned}\n\\end{equation}\nThe inequality is given by Young's inequality, the second to last equality is given by $\\Exp\\norm{X - \\Exp X}^2 = \\Exp\\norm{X}^2-\\norm{\\Exp X}^2$ where $X$ is a random variable.\n\nThe dual updates satisfy\n\\begin{align*}\n\t\\norm{y_i^{k+1} - y_i^\\star}^2\n\t&= \\norm{y_i^k + U_i^k(\\nabla f_i(x^k) - y_i^k) - y_i^\\star}^2 \\\\\n\t&= (1- U_i^k)\\norm{y_i^k - y_i^\\star}^2 + U_i^k\\norm{\\nabla f_i(x^k) - \\nabla f_i(x^\\star)}^2\n\\end{align*}\nsince $U_i^k \\in \\{0,1\\}$.\nSumming over all terms, taking expected value and using linearity of the expected value give\n\\begin{equation}\\label{eq:dual-update-lyap}\n\t\\begin{aligned}\n\t\t\\Exp[{\\textstyle\\sum_{i=1}^n} \\widehat{\\gamma}_i\\norm{y_i^{k+1} - y_i^\\star}^2|\\mathcal{F}^k]\n\t\t&= {\\textstyle\\sum_{i=1}^n} (1-\\eta_i)\\widehat{\\gamma}_i\\norm{y^k - y^\\star}^2 \\\\\n\t\t&\\quad + {\\textstyle\\sum_{i=1}^n}{\\eta_i \\widehat{\\gamma}_i}\\norm{\\nabla f_i(x^k) - \\nabla f_i(x^\\star)}^2.\n\t\\end{aligned}\n\\end{equation}\nAdding \\Cref{eq:dual-update-lyap} to \\Cref{eq:prim-update-lyap} and substituting in \\Cref{eq:update-variance-bound} and using the definition $\\frac{\\gamma_i}{\\gamma_i} = 1$ when $\\gamma_i = 0$ then yield \\Cref{eq:prop-lyapunov-bound}.\nApplying \\Cref{eq:prim-contract} and \\Cref{eq:dual-contract}, using the law of total expectation and telescoping the inequalities give the stated rate.\n\\end{proof}","sha256_utf8":"3892e4184cec40821b1f0783f0d3f24e62db5b72f7333cc0f9a39537523e9aea"},{"start":50856,"end":52798,"start_line":836,"end_line":868,"raw_tex":"\\begin{proof}[\\textbf{Proof of \\Cref{lem:prim-contract}}]%\n\tFirst note that $\\mu$-strong monotonicity and the Cauchy--Schwarz inequality imply\n\t\\begin{equation}\\label{eq:reverse-lipschitz}\n\t\t\\norm{\\nabla F(x^k) - \\nabla F(x^\\star)} \\geq \\mu \\norm{x^k - x^\\star}.\n\t\\end{equation}\n\tConsider the terms of $\\mathcal{V}(x^k)$.\n\tUsing \\Cref{eq:reverse-lipschitz} and Cauchy--Schwarz in the last term yield\n\t\\begin{align*}\n\t\t\\norm{\\nabla F(x^k) - \\nabla F(x^\\star)}^2\n\t\t&\\geq \\mu\\norm{\\nabla F(x^k) - \\nabla F(x^\\star)}\\norm{x^k - x^\\star} \\\\\n\t\t&\\geq \\mu\\inprod{\\nabla F(x^k) - \\nabla F(x^\\star)}{x^k - x^\\star}.\n\t\\end{align*}\n\tUsing $\\frac{1}{L_i}$-cocoercivity of $\\nabla f_i$ in the first term of $\\mathcal{V}(x^k)$ yields\n\t\\begin{align*}\n\t\t\\sum_{i=1}^n & {\\textstyle (\\frac{\\eta_i\\gamma_i}{\\delta}  + 1)\\frac{1}{n^2 p_i} }\\norm{\\nabla f_i(x^k) - \\nabla f_i(x^\\star)}^2 \\\\\n\t\t&\\leq \\sum_{i=1}^n {\\textstyle (\\frac{\\eta_i\\gamma_i}{\\delta}  + 1)\\frac{L_i}{n^2 p_i} } \\inprod{\\nabla f_i(x^k) - \\nabla f_i(x^\\star)}{x^k - x^\\star} \\\\\n\t\t&\\leq \\max_i({\\textstyle(\\frac{\\eta_i\\gamma_i}{\\delta}  + 1)\\frac{L_i}{n p_i}}) \\inprod{\\nabla F(x^k) - \\nabla F(x^\\star)}{x^k - x^\\star}.\n\t\\end{align*}\n\tAdding the terms back together yields\n\t\\begin{align*}\n\t\t\\mathcal{V}(x^k)\n\t\t&\\leq \\lambda^2 \\nu \\inprod{\\nabla F(x^k) - \\nabla F(x^\\star)}{x^k - x^\\star}\n\t\\end{align*}\n\twhere $\\nu = \\max_i(  (1+\\delta^{-1})\\frac{L_i\\eta_i\\gamma_i}{n p_i}  +  (1+\\delta)\\frac{L_i}{n p_i} - \\delta\\mu)$.\n\tThis can now be summarized as\n\t\\begin{align*}\n\t\t\\mathcal{P}(x^k)\n\t\t&\\leq \\norm{x^k - x^\\star}^2  - \\lambda\\big(2 - \\nu\\lambda \\big) \\inprod{\\nabla F(x^k) - \\nabla F (x^\\star)}{x^k - x^\\star} \\\\\n\t\t&\\leq \\norm{x^k - x^\\star}^2  - \\mu\\lambda\\big(2 - \\nu\\lambda \\big) \\norm{x^k - x^\\star}^2 \\\\\n\t\t&= (1 - \\rho_P)\\norm{x^k - x^\\star}^2,\n\t\\end{align*}\n\twhere $\\rho_P = \\mu\\lambda(2 - \\nu\\lambda)$ and the last inequality is given by the strong monotonicity of $\\nabla F$.\n\\end{proof}","sha256_utf8":"a33b8551f4f6c2d7a8b1c9f4d713e0e7c304ac478612df4ac078fd34d2c06f5b"},{"start":52801,"end":53206,"start_line":871,"end_line":879,"raw_tex":"\\begin{proof}[\\textbf{Proof of \\Cref{lem:dual-contract}}]%\n\tSince norms are non-negative, we have\n\t\\begin{align*}\n\t\t\\mathcal{D}(y^k)\n\t\t\\leq \\sum_{i=1}^n {\\textstyle(1 - \\eta_i  + \\frac{1}{\\gamma_i})\\widehat{\\gamma}_i} \\norm{y_i^k - y_i^\\star}^2\n\t\t\\leq (1 - \\rho_D)\\sum_{i=1}^n \\widehat{\\gamma}_i \\norm{y_i^k - y_i^\\star}^2,\n\t\\end{align*}\n\twhere $\\rho_D = \\min_i(\\eta_i - \\tfrac{1}{\\gamma_i})$.\n\\end{proof}","sha256_utf8":"c1c2a10cbbe424a4ef1666283c6cf492f4feafcca2cdd975995471acf0df4a1d"},{"start":55674,"end":56515,"start_line":926,"end_line":937,"raw_tex":"\\begin{proof}[\\textbf{Proof of \\Cref{lem:sampled-lipschitz}}]%\n\tAn $L$-smooth and $\\mu$-strongly convex function must satisfy $L \\geq \\mu$.\n\tAssuming $\\max_i(\\frac{L_i}{np_i}) < \\mu$ yields the following contradiction\n\t\\begin{align*}\n\t\t\\mu > \\max_i \\tfrac{L_i}{np_i} = \\sum_{j=1}^n  p_j \\max_i \\tfrac{L_i}{np_i} \\geq \\sum_{j=1}^n  p_j \\tfrac{L_j}{np_j} = \\sum_{i=1}^n \\tfrac{L_i}{n} \\geq L.\n\t\\end{align*}\n\tIf $\\max_i(\\frac{L_i}{np_i}) = \\mu$, equality must hold everywhere and we have\n\t\\begin{align*}\n\t\t0 = \\sum_{j=1}^n  p_j \\max_i \\tfrac{L_i}{np_i} - \\sum_{j=1}^n  p_j \\tfrac{L_j}{np_j} = \\sum_{j=1}^n  p_j (\\max_i \\tfrac{L_i}{np_i} - \\tfrac{L_j}{np_j}).\n\t\\end{align*}\n\tSince $p_j > 0$ and $\\max_i \\tfrac{L_i}{np_i} - \\tfrac{L_j}{np_j} \\geq 0$, we have $\\max_i \\tfrac{L_i}{np_i} = \\tfrac{L_j}{np_j}$ for all $j\\in\\{1,\\dots,n\\}$.\n\\end{proof}","sha256_utf8":"3146d4c2a359f15209232d4d9e37ccc0ecb451925cba79bd9737281a5b70b181"},{"start":56575,"end":59286,"start_line":943,"end_line":978,"raw_tex":"\\begin{proof}[\\textbf{Proof of \\Cref{thm:pvrsg-conv}}]%\n\tApplication of \\Cref{lem:prim-contract} and \\ref{lem:dual-contract} in \\Cref{prop:main-conv} yields the convergence rate\n\t\\begin{align*}\n\t\t\\Exp \\Big[ \\norm{x^{k} - x^\\star}^2 + \\sum_{i=1}^n \\widehat{\\gamma}_i\\norm{y_i^{k} - y_i^\\star}^2 \\Big] \\in \\mathcal{O}((1 - \\min(\\rho_P, \\rho_D))^k)\n\t\\end{align*}\n\twith\n\t\\begin{align*}\n\t\t\\rho_P &= \\mu\\lambda(2- \\nu\\lambda) \\\\\n\t\t\\rho_D &= \\min_i \\eta_i - \\tfrac{1}{\\gamma_i} \\\\\n\t\t\\nu &= \\max_i  {\\textstyle(1+\\delta^{-1})\\frac{L_i\\eta_i\\gamma_i}{n p_i}  +  (1+\\delta)\\frac{L_i}{n p_i} - \\delta\\mu},\n\t\\end{align*}\n\twhich hold for all choices of $\\delta > 0$ and $\\gamma_i > 0$ for all $i\\in\\{1,\\dots,n\\}$.\n\tIf there exists $\\delta$ and $\\gamma_1,\\dots,\\gamma_n$ such that $\\min(\\rho_P,\\rho_D) \\in (0,1]$ we have convergence.\n\tWe restrict ourselves to only search for $\\delta$ and $\\gamma_1,\\dots,\\gamma_n$ such that $\\rho_P = \\rho_D = \\rho$ for some $\\rho \\in (0,1]$.\n\tFor all $i\\in\\{1,\\dots,n\\}$, select $\\gamma_i = \\frac{1}{\\eta_i - \\rho}$, which is positive when $\\rho < \\eta_i$, and convergence is then proved if there exists $\\rho \\in (0,\\min_i \\eta_i)$ and $\\delta > 0$ such that\n\t\\begin{align*}\n\t\t\\rho &= \\mu\\lambda(2 - \\nu\\lambda )\\\\\n\t\t\\nu &= \\max_i {\\textstyle(1+\\delta^{-1})\\frac{L_i}{n p_i}\\frac{\\eta_i}{\\eta_i - \\rho}  +  (1+\\delta)\\frac{L_i}{n p_i} - \\delta\\mu}.\n\t\\end{align*}\n\tThe variable $\\nu$ can be minimized w.r.t. $\\delta$ if $\\max_i \\frac{L_i}{np_i} > \\mu$.\n\tThe minimum then exists and is unique since $\\nu$ as a function of $\\delta$ is continuous, strictly convex, and $\\nu\\to\\infty$ both when $\\delta \\to 0^+$ and $\\delta \\to \\infty$.\n\tCalling the minimum point $\\delta^\\star$, noting that $\\delta^\\star > 0$, and inserting it and the choice of $\\gamma_i$ in the expression for $\\widehat{\\gamma}_i$ from \\Cref{def:lyap-terms} yield the first statement of the theorem.\n\n\tWhen $\\max_i \\frac{L_i}{np_i} \\not> \\mu$, \\Cref{lem:sampled-lipschitz} gives $\\frac{L_i}{n p_i} = \\mu$ for all $i\\in\\{1,\\dots,n\\}$ and\n\t\\begin{align*}\n\t\t\\nu = \\mu + \\mu(1+\\delta^{-1})\\max_i \\tfrac{\\eta_i}{\\eta_i-\\rho}.\n\t\\end{align*}\n\tThis can not be minimized w.r.t. $\\delta$ since the $\\inf$ is not attained.\n\tHowever, any $\\delta>0$ will yield a valid $\\rho$ and $\\widehat{\\gamma}_i$, giving the rate\n\t\\begin{align*}\n\t\t&\\Exp \\Big[ \\norm{x^{k} - x^\\star}^2 + \\sum_{i=1}^n{\\textstyle \\frac{ \\lambda^2}{n^2p_i} \\frac{1}{\\eta_i - \\rho}} \\norm{y_i^{k} - y_i^\\star}^2 \\Big] \\\\\n\t\t&\\quad \\leq \\Exp \\Big[ \\norm{x^{k} - x^\\star}^2 + \\sum_{i=1}^n \\widehat{\\gamma}_i\\norm{y_i^{k} - y_i^\\star}^2 \\Big]\n\t\t\\in \\mathcal{O}((1 - \\rho)^k).\n\t\\end{align*}\n\tTaking the limit as $\\delta \\to \\infty$ results in the stated interval.\n\\end{proof}","sha256_utf8":"1afc4777c9781ef26558d9540bde722b37bd07a18e9e9cb50200f5ae01765cdc"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"复合有限和问题","source_units":[{"start":13514,"end":13643,"start_line":282,"end_line":285,"raw_tex":"We consider the regularized finite sum problem\n\\begin{align}\\label{eq:problem}\n\t\\min_{x \\in \\R^N} \\quad& g(x) + F(x),\n\\end{align}","sha256_utf8":"073f790b25eb0e7aed7a0985fe60bcbf070e5b661de88cf27f739936990d326d"},{"start":13644,"end":13823,"start_line":286,"end_line":286,"raw_tex":"where $g: \\R^N \\rightarrow \\R \\cup \\{\\infty\\}$ and $F$ is of finite sum form $F(x) = \\frac{1}{n} \\sum_{i=1}^n f_i(x)$ with $f_i: \\R^N \\rightarrow \\R$ for all $i\\in\\{1,\\dots,n\\}$.\n","sha256_utf8":"fbc69c7e50981c7620e7c36dd8fc69e86e0ddb46d7ca6b95714a8aeeba81967d"}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","title":"凸性与光滑性假设","source_units":[{"start":13891,"end":14292,"start_line":288,"end_line":292,"raw_tex":"\\begin{ass}[Problem Properties]\\label{ass:strong-ass}%\n\tThe function $g: \\R^N \\rightarrow \\R \\cup \\{\\infty\\}$ is closed, convex and proper.\n\tFor all $i\\in\\{1,\\dots,n\\}$, the function $f_i: \\R^N \\rightarrow \\R$ is convex, differentiable and $L_i$-smooth.\n\tThe function $F: \\R^N \\rightarrow \\R$ is $\\mu$-strongly convex, differentiable and $L$-smooth with $L \\leq \\frac{1}{n}\\sum_{i=1}^n L_i$.\n\\end{ass}","sha256_utf8":"5c943c8b64d8cb1675e8255a33af53170578da3868d89501bda2b13c1b358cf6"}]},{"id":"S003","semantic_type":"derived_fact","kind":"derived_fact","title":"唯一最优解","source_units":[{"start":14293,"end":14479,"start_line":293,"end_line":293,"raw_tex":"As a consequence of \\Cref{ass:strong-ass}, $g + F$ is closed, proper and $\\mu$-strongly convex and  hence there exists a unique solution to \\Cref{eq:problem}, which we denote $x^\\star$.\n","sha256_utf8":"c894c9a07f29e7db0b06bd09fab4614e65ba2ca3cfde7ffe75924f51d0485c75"}]},{"id":"S004","semantic_type":"algorithm","kind":"algorithm","title":"随机近端方差缩减迭代","source_units":[{"start":14596,"end":15859,"start_line":295,"end_line":320,"raw_tex":"\\begin{algorithm}[H]%\n\t\\caption{PVRSG - Proximal Variance-Reduced Stochastic Gradient}\\label{alg:pvrsg}\n\tGiven the function $g$, the functions $f_1,\\dots,f_n$, initial primal and dual points, $x^0$ and $y_1^0,\\dots y_n^0$, iteratively perform the following for $k\\in\\{0,1,\\dots\\}$.\n\t\\begin{align*}\n\t\t&\\textbf{Sampling:}\n\t\t&&\\text{Randomly sample } (I^k,U_1^k,\\dots,U_n^k) \\text{ from } \\{1,\\dots,n\\}\\times\\{0,1\\}^n.\n\t\t\\\\\n\t\t&\\textbf{Primal Update:}\n\t\t&&\\begin{aligned}[t]\n\t\t\t&\\begin{aligned}\n\t\t\t\tz^{k+1} &=x^k - \\tfrac{\\lambda}{n} \\big(\\tfrac{1}{p_{I^k}}(\\nabla f_{I^k} (x^k) - y_{I^k}^k) + {\\textstyle\\sum_{i=1}^n} y_i^k\\big), \\\\\n\t\t\t\tx^{k+1} &= \\prox_{\\lambda g}(z^{k+1}).\n\t\t\t\\end{aligned}\n\t\t\\end{aligned}\\\\\n\t\t\\\\\n\t\t&\\textbf{Dual Update:}\n\t\t&&\\begin{aligned}[t]\n\t\t\t&\\begin{aligned}\n\t\t\t\ty_i^{k+1} &= y_i^k + U_i^k(\\nabla f_i (x^k) - y_i^k), \\quad \\forall i \\in \\{1,\\dots,n\\}.\n\t\t\t\\end{aligned}\n\t\t\\end{aligned}\n\t\\end{align*}\n\tThe sampling distributions of $(I^k,U_1^k,\\dots,U_n^k)$ for all $k\\in\\{0,1,\\dots\\}$ are the same and independent.\n\tFurthermore, the distribution is such that $P(I^k = i) = p_i > 0$ and the expected update frequency $\\eta_i > 0$, see \\Cref{def:update-freq}, for all $i\\in\\{1,\\dots,n\\}$.\n\tThe step-size satisfies $\\lambda > 0$.\n\\end{algorithm}","sha256_utf8":"f930789d44290a61ba0141ddccec3633bd0d03233a3c10917f8bfe2f20b1c72a"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"更新频率与滤过","source_units":[{"start":16554,"end":16951,"start_line":326,"end_line":333,"raw_tex":"\\begin{defin}[Expected Update Frequency]\\label{def:update-freq}%\n\tLet $U_1^k,\\dots,U_n^k$ be given by the sampling in \\Cref{alg:pvrsg}.\n\tThe expected update frequency of the $i$th dual variable is\n\t\\begin{align*}\n\t\t\\eta_i = \\Exp[U_i^k|\\mathcal{F}^k],\n\t\\end{align*}\n\twhere $\\mathcal{F}^k = \\cup_{i=1}^k \\mathcal{X}^i$ and $\\mathcal{X}^k = \\{x^k,y^k,I^{k-1},U_1^{k-1},\\dots,U_n^{k-1}\\}$.\n\\end{defin}","sha256_utf8":"ca0a524f050fe11042d563fbbfb905bbbfb62dae8d64c6d89e9ac1a36a71c1ca"},{"start":16952,"end":17134,"start_line":334,"end_line":334,"raw_tex":"Note, the expected update frequency does not depend on the iteration number $k$ since $(I^k,U_1^k,\\dots,U_n^k)$ is independently sampled and its distribution does not depend on $k$.\n","sha256_utf8":"874fe6d97b030abefd51687f25200143eca8db322807298de8204e3170783881"}]},{"id":"S006","semantic_type":"notation","kind":"notation","title":"空间与概率记号","source_units":[{"start":11186,"end":11683,"start_line":247,"end_line":252,"raw_tex":"Let $\\R$ be the set of real numbers.\nWe will work in finite dimensional real spaces $\\R^N$.\nLet $\\inprod{\\cdot}{\\cdot}$ denote the standard Euclidean inner product and let $\\norm{\\cdot}$ be the norm induced by the inner product.\nThe expected value conditioned on the filtration $\\mathcal{F}$ is $\\Exp[\\cdot | \\mathcal{F}]$.\nThe probability of a discrete random variable taking value $i$ is $P(\\cdot = i)$.\nWe define $\\mathbf{1}_{X} = 1$ if the predicate $X$ is true, otherwise $\\mathbf{1}_X = 0$.\n","sha256_utf8":"815df576a053dbf70c6b5f8cb1f3e15cbea847d727bd403c9811ba88b865ee76"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"光滑性与强凸性","source_units":[{"start":11684,"end":12372,"start_line":254,"end_line":260,"raw_tex":"A convex function $f: \\R^d \\to \\R$ is \\emph{$L$-smooth} with $L > 0$ if it is differentiable and its gradient is $\\tfrac{1}{L}$-cocoercive, i.e.,\n\\begin{align*}\n\t\\inprod{\\nabla f(x) - \\nabla f(y)}{x-y} \\geq \\tfrac{1}{L} \\norm{\\nabla f(x) - \\nabla f(y)}^2, \\quad \\forall x,y \\in \\R^d.\n\\end{align*}\nNote that the definition of smoothness implies \\emph{$L$-Lipschitz continuity} of the gradient $\\nabla f$.\nIn fact, for convex $f$, Lipschitz continuity and cocoercivity of $\\nabla f$ are equivalent \\cite[Corollary 18.17]{bauschke_convex_2017}.\nA proper function $f: \\R^d \\to \\R \\cup \\{\\infty\\}$ is \\emph{$\\mu$-strongly convex} with $\\mu > 0$ if $f - \\frac{\\mu}{2}\\norm{\\cdot}^2$ is convex.\n","sha256_utf8":"23cf7bc9f11b7e1dc760a7210b3170c4599f50977cc33175acda4cf5af05f7e7"}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","title":"强单调性与极小值","source_units":[{"start":12373,"end":12809,"start_line":262,"end_line":267,"raw_tex":"The subdifferential of a $\\mu$-strongly convex function is $\\mu$-strongly monotone \\cite[Example 22.4]{bauschke_convex_2017}, i.e.,\n\\begin{align*}\n\t\\inprod{u - v}{x-y} \\geq \\mu \\norm{x-y}^2\n\\end{align*}\nholds $\\forall x,y \\in \\mathop{\\mathrm{dom}} \\partial f$ and $\\forall u\\in \\partial f(x), \\forall v \\in \\partial f(y)$.\nA closed, proper and strongly convex function has a unique minimum \\cite[Corollary 11.17]{bauschke_convex_2017}.\n","sha256_utf8":"c6e7d3b31b2cdf37d2d487ad93e2ea9572ba82fa99353ee6ccb85a9791c0d7ba"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"近端算子及非扩张性","source_units":[{"start":12810,"end":13262,"start_line":269,"end_line":274,"raw_tex":"The \\emph{proximal operator} of a closed, convex and proper function $g: \\R^d \\to \\R \\cup \\{\\infty\\}$ is defined as\n\\begin{align*}\n\t\\prox_g(z) = \\argmin_x g(x) + \\tfrac{1}{2}\\norm{x-z}^2.\n\\end{align*}\nDue to strong convexity of $g + \\frac{1}{2}\\norm{\\cdot - z}^2$, the minimum exist and is unique.\nFurthermore, the operator $\\prox_g$ is \\emph{non-expansive}, i.e., Lipschitz continuous with constant $1$ \\cite[Proposition 12.28]{bauschke_convex_2017}.\n","sha256_utf8":"d25d405520cb8f228584ae149b47606a09acfee6e5d727294059d5a714960613"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"李雅普诺夫分量与权重","source_units":[{"start":20280,"end":21382,"start_line":376,"end_line":395,"raw_tex":"\\begin{defin}\\label{def:lyap-terms}%\n\tLet $x^\\star$ be the solution to \\Cref{eq:problem} and $y_i^\\star = \\nabla f_i(x^\\star)$ for all $i\\in\\{1,\\dots,n\\}$.\n\tWith $y = (y_1,\\dots,y_n)$, where $y_i \\in \\R^n$ for $i \\in \\{1,\\dots,n\\}$, we define\n\t\\begin{align*}\n\t\t\\mathcal{P}(x) = \\norm{x - x^\\star}^2 - 2\\lambda\\inprod{\\nabla F(x) - \\nabla F (x^\\star)}{x - x^\\star} + \\lambda^2\\mathcal{V}(x)\n\t\\end{align*}\n\tand\n\t\\begin{align*}\n\t\t\\mathcal{D}(y)\n\t\t&= \\sum_{i=1}^n {\\textstyle(1 - \\eta_i  + \\frac{1}{\\gamma_i})\\widehat{\\gamma}_i} \\norm{y_i - y_i^\\star}^2\n\t\t-{\\textstyle (1+\\delta^{-1})\\lambda^2}\\norm{{{\\textstyle\\frac{1}{n}}\\sum_{i=1}^n} y_i -  y_i^\\star }^2,\n\t\\end{align*}\n\twhere\n\t\\begin{align*}\n\t\t\\mathcal{V}(x)\n\t\t= \\sum_{i=1}^n {\\textstyle \\frac{(1+\\delta)}{n^2 p_i}(\\frac{\\eta_i\\gamma_i}{\\delta}  + 1) }\\norm{\\nabla f_i(x) - \\nabla f_i(x^\\star)}^2 - \\delta\\norm{\\nabla F(x) - \\nabla F(x^\\star)}^2\n\t\\end{align*}\n\twith $\\gamma_i \\geq 0$, $\\delta > 0$ and $\\widehat{\\gamma}_i = \\gamma_i \\frac{(1+\\delta^{-1}) \\lambda^2}{n^2p_i}$.\n\tIf $\\gamma_i = 0$ we define $\\tfrac{\\gamma_i}{\\gamma_i} := 1$.\n\\end{defin}","sha256_utf8":"fa2fb5d7d4aff623bdee12fa5c6d173a68af604fb178bcbd8431bfd8a128ab8c"}]},{"id":"S011","semantic_type":"theorem","kind":"theorem","title":"一般随机迭代的线性收敛","source_units":[{"start":25644,"end":26942,"start_line":468,"end_line":485,"raw_tex":"\\begin{thm}[PVRSG Convergence]\\label{thm:pvrsg-conv}%\n\tGiven $\\max_i \\frac{L_i}{np_i} > \\mu$ and \\Cref{ass:strong-ass}, if there exists $\\rho \\in (0, \\min_i \\eta_i)$ such that\n\t\\begin{align*}\n\t\t\\rho &= \\mu\\lambda(2 - \\nu\\lambda )\\\\\n\t\t\\nu &= \\min_{\\delta > 0} \\max_i {\\textstyle(1+\\delta^{-1})\\frac{L_i}{n p_i}\\frac{\\eta_i}{\\eta_i - \\rho}  +  (1+\\delta)\\frac{L_i}{n p_i} - \\delta\\mu},\n\t\\end{align*}\n\tthen the iterates of \\Cref{alg:pvrsg} converge according to\n\t\\begin{align*}\n\t\t\\Exp \\Big[ \\norm{x^{k} - x^\\star}^2 + \\sum_{i=1}^n \\widehat{\\gamma}_i\\norm{y_i^{k} - y_i^\\star}^2 \\Big] \\in \\mathcal{O}((1 - \\rho)^k)\n\t\\end{align*}\n\twhere $\\widehat{\\gamma}_1,\\dots,\\widehat{\\gamma}_n$ are given by $\\widehat{\\gamma}_i ={\\textstyle \\frac{ \\lambda^2}{n^2p_i} \\frac{1}{\\eta_i - \\rho}(1 + \\frac{1}{\\delta^\\star})}$ and $\\delta^\\star$ is the unique minimizer of the minimization problem defining $\\nu$.\n\n\tIf instead $\\max_i \\frac{L_i}{np_i} = \\mu$ then $\\nu = \\mu + \\mu\\max_i \\tfrac{\\eta_i}{\\eta_i-\\rho}$ and the convergence is such that\n\t\\begin{align*}\n\t\t\\Exp \\Big[ \\norm{x^{k} - x^\\star}^2 + \\sum_{i=1}^n{\\textstyle \\frac{ \\lambda^2}{n^2p_i} \\frac{1}{\\eta_i - \\tilde{\\rho}}} \\norm{y_i^{k} - y_i^\\star}^2 \\Big] \\in \\mathcal{O}((1 - \\tilde{\\rho})^k)\n\t\\end{align*}\n\tholds for all $\\tilde{\\rho} \\in (0, \\rho)$.\n\t","sha256_utf8":"8a8b78a9f31b0938f849396161a5d149d5a1d30d8cc4b6d483ca24fb54d1c9d7"},{"start":27001,"end":27011,"start_line":487,"end_line":488,"raw_tex":"\n\\end{thm}","sha256_utf8":"27661e277ceb78e4dd676b4ea4b5c8d9a4d3c607a1a758eac095282a4f311f77"}]},{"id":"S012","semantic_type":"proof","kind":"proof","title":"主收敛定理完整证明","source_units":[{"start":26942,"end":27001,"start_line":485,"end_line":487,"raw_tex":"\\begin{proof}\n\t\tSee \\Cref{sec:proof-theorems}.\n\t\\end{proof}","sha256_utf8":"56f47e9dead4585c044ff7375a027a6a2bc39d2500de4d7b3fb97d73a47cfdae"},{"start":56575,"end":59286,"start_line":943,"end_line":978,"raw_tex":"\\begin{proof}[\\textbf{Proof of \\Cref{thm:pvrsg-conv}}]%\n\tApplication of \\Cref{lem:prim-contract} and \\ref{lem:dual-contract} in \\Cref{prop:main-conv} yields the convergence rate\n\t\\begin{align*}\n\t\t\\Exp \\Big[ \\norm{x^{k} - x^\\star}^2 + \\sum_{i=1}^n \\widehat{\\gamma}_i\\norm{y_i^{k} - y_i^\\star}^2 \\Big] \\in \\mathcal{O}((1 - \\min(\\rho_P, \\rho_D))^k)\n\t\\end{align*}\n\twith\n\t\\begin{align*}\n\t\t\\rho_P &= \\mu\\lambda(2- \\nu\\lambda) \\\\\n\t\t\\rho_D &= \\min_i \\eta_i - \\tfrac{1}{\\gamma_i} \\\\\n\t\t\\nu &= \\max_i  {\\textstyle(1+\\delta^{-1})\\frac{L_i\\eta_i\\gamma_i}{n p_i}  +  (1+\\delta)\\frac{L_i}{n p_i} - \\delta\\mu},\n\t\\end{align*}\n\twhich hold for all choices of $\\delta > 0$ and $\\gamma_i > 0$ for all $i\\in\\{1,\\dots,n\\}$.\n\tIf there exists $\\delta$ and $\\gamma_1,\\dots,\\gamma_n$ such that $\\min(\\rho_P,\\rho_D) \\in (0,1]$ we have convergence.\n\tWe restrict ourselves to only search for $\\delta$ and $\\gamma_1,\\dots,\\gamma_n$ such that $\\rho_P = \\rho_D = \\rho$ for some $\\rho \\in (0,1]$.\n\tFor all $i\\in\\{1,\\dots,n\\}$, select $\\gamma_i = \\frac{1}{\\eta_i - \\rho}$, which is positive when $\\rho < \\eta_i$, and convergence is then proved if there exists $\\rho \\in (0,\\min_i \\eta_i)$ and $\\delta > 0$ such that\n\t\\begin{align*}\n\t\t\\rho &= \\mu\\lambda(2 - \\nu\\lambda )\\\\\n\t\t\\nu &= \\max_i {\\textstyle(1+\\delta^{-1})\\frac{L_i}{n p_i}\\frac{\\eta_i}{\\eta_i - \\rho}  +  (1+\\delta)\\frac{L_i}{n p_i} - \\delta\\mu}.\n\t\\end{align*}\n\tThe variable $\\nu$ can be minimized w.r.t. $\\delta$ if $\\max_i \\frac{L_i}{np_i} > \\mu$.\n\tThe minimum then exists and is unique since $\\nu$ as a function of $\\delta$ is continuous, strictly convex, and $\\nu\\to\\infty$ both when $\\delta \\to 0^+$ and $\\delta \\to \\infty$.\n\tCalling the minimum point $\\delta^\\star$, noting that $\\delta^\\star > 0$, and inserting it and the choice of $\\gamma_i$ in the expression for $\\widehat{\\gamma}_i$ from \\Cref{def:lyap-terms} yield the first statement of the theorem.\n\n\tWhen $\\max_i \\frac{L_i}{np_i} \\not> \\mu$, \\Cref{lem:sampled-lipschitz} gives $\\frac{L_i}{n p_i} = \\mu$ for all $i\\in\\{1,\\dots,n\\}$ and\n\t\\begin{align*}\n\t\t\\nu = \\mu + \\mu(1+\\delta^{-1})\\max_i \\tfrac{\\eta_i}{\\eta_i-\\rho}.\n\t\\end{align*}\n\tThis can not be minimized w.r.t. $\\delta$ since the $\\inf$ is not attained.\n\tHowever, any $\\delta>0$ will yield a valid $\\rho$ and $\\widehat{\\gamma}_i$, giving the rate\n\t\\begin{align*}\n\t\t&\\Exp \\Big[ \\norm{x^{k} - x^\\star}^2 + \\sum_{i=1}^n{\\textstyle \\frac{ \\lambda^2}{n^2p_i} \\frac{1}{\\eta_i - \\rho}} \\norm{y_i^{k} - y_i^\\star}^2 \\Big] \\\\\n\t\t&\\quad \\leq \\Exp \\Big[ \\norm{x^{k} - x^\\star}^2 + \\sum_{i=1}^n \\widehat{\\gamma}_i\\norm{y_i^{k} - y_i^\\star}^2 \\Big]\n\t\t\\in \\mathcal{O}((1 - \\rho)^k).\n\t\\end{align*}\n\tTaking the limit as $\\delta \\to \\infty$ results in the stated interval.\n\\end{proof}","sha256_utf8":"1afc4777c9781ef26558d9540bde722b37bd07a18e9e9cb50200f5ae01765cdc"}]},{"id":"S013","semantic_type":"proof_step","kind":"proof_step","title":"主证明位置说明","source_units":[{"start":26956,"end":26989,"start_line":486,"end_line":486,"raw_tex":"\t\tSee \\Cref{sec:proof-theorems}.\n","sha256_utf8":"f64996e509efee1dead96ebc573c09078cc5b82ede3a5db622625282ac25d92c"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"组合原始与对偶收缩","source_units":[{"start":56631,"end":57395,"start_line":944,"end_line":955,"raw_tex":"\tApplication of \\Cref{lem:prim-contract} and \\ref{lem:dual-contract} in \\Cref{prop:main-conv} yields the convergence rate\n\t\\begin{align*}\n\t\t\\Exp \\Big[ \\norm{x^{k} - x^\\star}^2 + \\sum_{i=1}^n \\widehat{\\gamma}_i\\norm{y_i^{k} - y_i^\\star}^2 \\Big] \\in \\mathcal{O}((1 - \\min(\\rho_P, \\rho_D))^k)\n\t\\end{align*}\n\twith\n\t\\begin{align*}\n\t\t\\rho_P &= \\mu\\lambda(2- \\nu\\lambda) \\\\\n\t\t\\rho_D &= \\min_i \\eta_i - \\tfrac{1}{\\gamma_i} \\\\\n\t\t\\nu &= \\max_i  {\\textstyle(1+\\delta^{-1})\\frac{L_i\\eta_i\\gamma_i}{n p_i}  +  (1+\\delta)\\frac{L_i}{n p_i} - \\delta\\mu},\n\t\\end{align*}\n\twhich hold for all choices of $\\delta > 0$ and $\\gamma_i > 0$ for all $i\\in\\{1,\\dots,n\\}$.\n\tIf there exists $\\delta$ and $\\gamma_1,\\dots,\\gamma_n$ such that $\\min(\\rho_P,\\rho_D) \\in (0,1]$ we have convergence.\n","sha256_utf8":"0392a6eac368864a03a802de30582abfd41875128ce3056d4e23c29ac9b076a6"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"选择权重平衡收缩率","source_units":[{"start":57395,"end":57960,"start_line":956,"end_line":961,"raw_tex":"\tWe restrict ourselves to only search for $\\delta$ and $\\gamma_1,\\dots,\\gamma_n$ such that $\\rho_P = \\rho_D = \\rho$ for some $\\rho \\in (0,1]$.\n\tFor all $i\\in\\{1,\\dots,n\\}$, select $\\gamma_i = \\frac{1}{\\eta_i - \\rho}$, which is positive when $\\rho < \\eta_i$, and convergence is then proved if there exists $\\rho \\in (0,\\min_i \\eta_i)$ and $\\delta > 0$ such that\n\t\\begin{align*}\n\t\t\\rho &= \\mu\\lambda(2 - \\nu\\lambda )\\\\\n\t\t\\nu &= \\max_i {\\textstyle(1+\\delta^{-1})\\frac{L_i}{n p_i}\\frac{\\eta_i}{\\eta_i - \\rho}  +  (1+\\delta)\\frac{L_i}{n p_i} - \\delta\\mu}.\n\t\\end{align*}\n","sha256_utf8":"fe13809031dc01103c4866c224869c6832a164ac61b3a0e0758027f2b67a1eff"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"优化参数得到一般情形","source_units":[{"start":57960,"end":58462,"start_line":962,"end_line":964,"raw_tex":"\tThe variable $\\nu$ can be minimized w.r.t. $\\delta$ if $\\max_i \\frac{L_i}{np_i} > \\mu$.\n\tThe minimum then exists and is unique since $\\nu$ as a function of $\\delta$ is continuous, strictly convex, and $\\nu\\to\\infty$ both when $\\delta \\to 0^+$ and $\\delta \\to \\infty$.\n\tCalling the minimum point $\\delta^\\star$, noting that $\\delta^\\star > 0$, and inserting it and the choice of $\\gamma_i$ in the expression for $\\widehat{\\gamma}_i$ from \\Cref{def:lyap-terms} yield the first statement of the theorem.\n","sha256_utf8":"10b29b4f58cf9ff4a4c1679539f8a06e178789e57872947b9d39a8c3db81db81"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"等号情形化简参数","source_units":[{"start":58463,"end":58774,"start_line":966,"end_line":970,"raw_tex":"\tWhen $\\max_i \\frac{L_i}{np_i} \\not> \\mu$, \\Cref{lem:sampled-lipschitz} gives $\\frac{L_i}{n p_i} = \\mu$ for all $i\\in\\{1,\\dots,n\\}$ and\n\t\\begin{align*}\n\t\t\\nu = \\mu + \\mu(1+\\delta^{-1})\\max_i \\tfrac{\\eta_i}{\\eta_i-\\rho}.\n\t\\end{align*}\n\tThis can not be minimized w.r.t. $\\delta$ since the $\\inf$ is not attained.\n","sha256_utf8":"f0137dce6cb451eb5fb58aedf1c9cce6fc500b64273dcc9269c2de72d76b06e7"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"极限给出较小收敛率","source_units":[{"start":58774,"end":59275,"start_line":971,"end_line":977,"raw_tex":"\tHowever, any $\\delta>0$ will yield a valid $\\rho$ and $\\widehat{\\gamma}_i$, giving the rate\n\t\\begin{align*}\n\t\t&\\Exp \\Big[ \\norm{x^{k} - x^\\star}^2 + \\sum_{i=1}^n{\\textstyle \\frac{ \\lambda^2}{n^2p_i} \\frac{1}{\\eta_i - \\rho}} \\norm{y_i^{k} - y_i^\\star}^2 \\Big] \\\\\n\t\t&\\quad \\leq \\Exp \\Big[ \\norm{x^{k} - x^\\star}^2 + \\sum_{i=1}^n \\widehat{\\gamma}_i\\norm{y_i^{k} - y_i^\\star}^2 \\Big]\n\t\t\\in \\mathcal{O}((1 - \\rho)^k).\n\t\\end{align*}\n\tTaking the limit as $\\delta \\to \\infty$ results in the stated interval.\n","sha256_utf8":"ba0b58feea9a4bb00a90c42be91db9b1f12ccb3f7928783f476384c8027073d2"}]},{"id":"S019","semantic_type":"proposition","kind":"proposition","title":"单步误差界与收缩准则","source_units":[{"start":21589,"end":22854,"start_line":398,"end_line":419,"raw_tex":"\\begin{prop}\\label{prop:main-conv}%\n\tLet the filtration $\\mathcal{F}^k = \\cup_{i=1}^k \\mathcal{X}^i$ be given by the state $\\mathcal{X}^k = \\{x^k,y^k,I^{k-1},U_1^{k-1},\\dots,U_n^{k-1}\\}$.\n\tIf \\Cref{ass:strong-ass} holds, the iterates of \\Cref{alg:pvrsg} satisfy\n\t\\begin{equation}\\label{eq:prop-lyapunov-bound}\n\t\t\\Exp \\big[ \\norm{x^{k+1} - x^\\star}^2 + \\sum_{i=1}^n \\widehat{\\gamma}_i\\norm{y_i^{k+1} - y_i^\\star}^2 \\big| \\mathcal{F}^k \\big] \\leq  \\mathcal{P}(x^k) + \\mathcal{D}(y^k),\n\t\\end{equation}\n\twhere $x^\\star$ is the unique solution of \\Cref{eq:problem} and $y_i^\\star = \\nabla f_i(x^\\star)$.\n\tSee \\Cref{def:lyap-terms} for $\\mathcal{P}$, $\\mathcal{D}$, and $\\mathcal{V}$.\n\n\tIf the primal updates satisfy\n\t\\begin{align}\\label{eq:prim-contract}\n\t\t\\mathcal{P}(x^k) \\leq (1-\\rho_P)\\norm{x^k - x^\\star}^2\n\t\\end{align}\n\tand the dual updates satisfy\n\t\\begin{align}\\label{eq:dual-contract}\n\t\t\\mathcal{D}(y^k) \\leq (1-\\rho_D)\\sum_{i=1}^n \\widehat{\\gamma}_i\\norm{y_i^k - y_i^\\star}^2\n\t\\end{align}\n\twith $\\rho_P, \\rho_D \\in (0,1]$ then \\Cref{alg:pvrsg} converges linearly according to\n\t\\begin{align*}\n\t\t\\Exp \\big[ \\norm{x^{k} - x^\\star}^2 + \\sum_{i=1}^n \\widehat{\\gamma}_i\\norm{y_i^{k} - y_i^\\star}^2 \\big] \\in \\mathcal{O}((1 - \\min(\\rho_P, \\rho_D))^k).\n\t\\end{align*}\n\t","sha256_utf8":"dfaa42a86e11e28863f58a3423d4baa8452568b19c7ebc46e0c7242dd7d12c71"},{"start":22915,"end":22926,"start_line":421,"end_line":422,"raw_tex":"\n\\end{prop}","sha256_utf8":"05d52a2387e348b9b8c6add2d64ac5b19b3c01cd0607238154a9f309962fa7aa"}]},{"id":"S020","semantic_type":"proof","kind":"proof","title":"误差界命题完整证明","source_units":[{"start":22854,"end":22915,"start_line":419,"end_line":421,"raw_tex":"\\begin{proof}\n\t\tSee \\Cref{sec:proof-prop-lemma}.\n\t\\end{proof}","sha256_utf8":"348bf3f69d77e9410af220b3a95bf45811f3f3b3b4be8c1ccdc6d3a6d23ff838"},{"start":45971,"end":50853,"start_line":777,"end_line":833,"raw_tex":"\\begin{proof}[\\textbf{Proof of \\Cref{prop:main-conv}}]%\n\tLet $x^\\star$ be the unique solution to \\Cref{eq:problem}.\n\tWith $g$ being proper, closed and convex, the primal updates satisfy\n\t\\begin{equation}\\label{eq:prim-update-lyap}\n\t\t\\begin{aligned}\n\t\t\t&\\Exp[\\norm{x^{k+1} - x^\\star}^2|\\mathcal{F}^k] \\\\\n\t\t\t&\\quad= \\Exp[\\norm{\\prox_{\\lambda g}(z^{k+1}) - \\prox_{\\lambda g}(x^\\star - \\lambda \\nabla F(x^\\star))}^2|\\mathcal{F}^k] \\\\\n\t\t\t&\\quad\\leq\n\t\t\t%\\Exp[\\norm{z^{k+1} - (x^\\star - \\lambda \\nabla F(x^\\star))}^2|\\mathcal{F}^k] \\\\\n\t\t\t%&\\quad=\n\t\t\t\\Exp[\\norm{x^k - \\tfrac{\\lambda}{n}\\big(\\tfrac{1}{p_{I^k}}(\\nabla f_{I^k}(x^k) - y_{I^k}^k) + \\textstyle\\sum_{i=1}^n y_i^k\\big) - x^\\star + \\lambda \\nabla F(x^\\star)}^2|\\mathcal{F}^k] \\\\\n\t\t\t&\\quad= \\norm{(x^k - \\lambda \\nabla F(x^k)) - (x^\\star - \\lambda \\nabla F(x^\\star))}^2\\\\\n\t\t&\\quad\\quad + \\lambda^2 \\Exp\\big[\\norm[\\big]{\\big(\\tfrac{1}{n p_{I^k}} \\nabla f_{I^k}(x^k) - \\nabla F(x^k)\\big) - \\big(\\tfrac{1}{n p_{I^k}}y_{I^k}^k - {\\textstyle\\frac{1}{n}\\sum_{i=1}^n} y_i^k \\big)}^2|\\mathcal{F}^k\\big] \\\\\n\t\t&\\quad= \\norm{x^k - x^\\star}^2 - 2\\lambda\\inprod{\\nabla F(x^k) - \\nabla F(x^\\star)}{x^k-x^\\star} + \\lambda^2\\norm{\\nabla F(x^k) -\\nabla F(x^\\star)}^2\\\\\n\t&\\quad\\quad + \\lambda^2 \\Exp\\big[\\norm[\\big]{\\big(\\tfrac{1}{n p_{I^k}} \\nabla f_{I^k}(x^k) - \\nabla F(x^k)\\big) - \\big(\\tfrac{1}{n p_{I^k}}y_{I^k}^k - {\\textstyle\\frac{1}{n}\\sum_{i=1}^n} y_i^k \\big)}^2|\\mathcal{F}^k\\big].\n\\end{aligned}\n\\end{equation}\nThe first equality is given by the solution being a fixed point to the proximal-gradient update $x^\\star = \\prox_{\\lambda g}(x^\\star - \\lambda \\nabla F(x^\\star))$ \\cite[Corollary 28.9]{bauschke_convex_2017}.\nThe first inequality is due to the non-expansiveness of $\\prox_{\\lambda g}$.\nThe second to last equality is given by $\\Exp\\norm{X}^2 =\\norm{\\Exp X}^2 + \\Exp\\norm{X - \\Exp X}^2$ where $X$ is a random variable.\n\nThe last term in \\Cref{eq:prim-update-lyap} satisfies the following upper bound for all $\\delta > 0$:\n\\begin{equation}\\label{eq:update-variance-bound}\n\t\\begin{aligned}\n\t\t&\\Exp[\\norm{(\\tfrac{1}{n p_{I^k}} \\nabla f_{I^k}(x^k) - \\nabla F(x^k)) - (\\tfrac{1}{n p_{I^k}}y_{I^k}^k - {\\textstyle\\frac{1}{n}\\sum_{i=1}^n} y_i^k )}^2|\\mathcal{F}^k]\\\\\n\t\t&\\quad\\begin{aligned}\n\t\t\t= \\Exp[\\| &\\tfrac{1}{np_{I^k}}(\\nabla f_{I^k}(x^k) - \\nabla f_{I^k}(x^\\star)) - (\\nabla F(x^k) - \\nabla F(x^\\star))\\\\\n\t\t\t&- \\tfrac{1}{np_{I^k}}(y_{I^k}^k - y_{I^k}^\\star) + {\\textstyle\\frac{1}{n}\\sum_{i=1}^n}(y_i^k - y_i^\\star) \\|^2 |\\mathcal{F}^k]\n\t\t\\end{aligned} \\\\\n\t\t&\\quad\\leq (1+\\delta)\\Exp[\\norm{ \\tfrac{1}{np_{I^k}}(\\nabla f_{I^k}(x^k) - \\nabla f_{I^k}(x^\\star)) - (\\nabla F(x^k) - \\nabla F(x^\\star)) }^2 |\\mathcal{F}^k]\\\\\n\t\t&\\quad\\quad + (1+\\delta^{-1})\\Exp[\\norm{\\tfrac{1}{np_{I^k}}(y_{I^k}^k - y_{I^k}^\\star) - {\\textstyle\\frac{1}{n}\\sum_{i=1}^n}(y_i^k - y_i^\\star)}^2 |\\mathcal{F}^k]\\\\\n\t\t&\\quad= (1+\\delta)\\big(\\Exp[\\norm{\\tfrac{1}{np_{I^k}}(\\nabla f_{I^k}(x^k) - \\nabla f_{I^k}(x^\\star))}^2|\\mathcal{F}^k] - \\norm{\\nabla F(x^k) - \\nabla F(x^\\star)}^2\\big) \\\\\n\t\t&\\quad\\quad + (1+\\delta^{-1})\\big(\\Exp[\\norm{\\tfrac{1}{np_{I^k}}(y_{I^k}^k - y_{I^k}^\\star)}^2|\\mathcal{F}^k] - \\norm{{\\textstyle\\frac{1}{n}\\sum_{i=1}^n}(y_i^k - y_i^\\star)}^2\\big) \\\\\n\t\t&\\quad= (1+\\delta){\\textstyle\\sum\\frac{1}{n^2 p_i}}\\norm{\\nabla f_i(x^k) - \\nabla f_i(x^\\star)}^2 - (1+\\delta)\\norm{\\nabla F(x^k) - \\nabla F(x^\\star)}^2 \\\\\n\t\t&\\quad\\quad + (1+\\delta^{-1}){\\textstyle\\sum\\frac{1}{n^2 p_i}}\\norm{y_i^k - y_i^\\star}^2 - (1+\\delta^{-1})\\norm{{\\textstyle\\frac{1}{n}\\sum_{i=1}^n}(y_i^k - y_i^\\star)}^2.\n\t\\end{aligned}\n\\end{equation}\nThe inequality is given by Young's inequality, the second to last equality is given by $\\Exp\\norm{X - \\Exp X}^2 = \\Exp\\norm{X}^2-\\norm{\\Exp X}^2$ where $X$ is a random variable.\n\nThe dual updates satisfy\n\\begin{align*}\n\t\\norm{y_i^{k+1} - y_i^\\star}^2\n\t&= \\norm{y_i^k + U_i^k(\\nabla f_i(x^k) - y_i^k) - y_i^\\star}^2 \\\\\n\t&= (1- U_i^k)\\norm{y_i^k - y_i^\\star}^2 + U_i^k\\norm{\\nabla f_i(x^k) - \\nabla f_i(x^\\star)}^2\n\\end{align*}\nsince $U_i^k \\in \\{0,1\\}$.\nSumming over all terms, taking expected value and using linearity of the expected value give\n\\begin{equation}\\label{eq:dual-update-lyap}\n\t\\begin{aligned}\n\t\t\\Exp[{\\textstyle\\sum_{i=1}^n} \\widehat{\\gamma}_i\\norm{y_i^{k+1} - y_i^\\star}^2|\\mathcal{F}^k]\n\t\t&= {\\textstyle\\sum_{i=1}^n} (1-\\eta_i)\\widehat{\\gamma}_i\\norm{y^k - y^\\star}^2 \\\\\n\t\t&\\quad + {\\textstyle\\sum_{i=1}^n}{\\eta_i \\widehat{\\gamma}_i}\\norm{\\nabla f_i(x^k) - \\nabla f_i(x^\\star)}^2.\n\t\\end{aligned}\n\\end{equation}\nAdding \\Cref{eq:dual-update-lyap} to \\Cref{eq:prim-update-lyap} and substituting in \\Cref{eq:update-variance-bound} and using the definition $\\frac{\\gamma_i}{\\gamma_i} = 1$ when $\\gamma_i = 0$ then yield \\Cref{eq:prop-lyapunov-bound}.\nApplying \\Cref{eq:prim-contract} and \\Cref{eq:dual-contract}, using the law of total expectation and telescoping the inequalities give the stated rate.\n\\end{proof}","sha256_utf8":"3892e4184cec40821b1f0783f0d3f24e62db5b72f7333cc0f9a39537523e9aea"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"命题证明位置说明","source_units":[{"start":22868,"end":22903,"start_line":420,"end_line":420,"raw_tex":"\t\tSee \\Cref{sec:proof-prop-lemma}.\n","sha256_utf8":"7932c28351c7f0d3375c661cebf6e7f8a405a960fe79c83e365609deecd00ed5"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"近端非扩张与方差分解","source_units":[{"start":46027,"end":47842,"start_line":778,"end_line":796,"raw_tex":"\tLet $x^\\star$ be the unique solution to \\Cref{eq:problem}.\n\tWith $g$ being proper, closed and convex, the primal updates satisfy\n\t\\begin{equation}\\label{eq:prim-update-lyap}\n\t\t\\begin{aligned}\n\t\t\t&\\Exp[\\norm{x^{k+1} - x^\\star}^2|\\mathcal{F}^k] \\\\\n\t\t\t&\\quad= \\Exp[\\norm{\\prox_{\\lambda g}(z^{k+1}) - \\prox_{\\lambda g}(x^\\star - \\lambda \\nabla F(x^\\star))}^2|\\mathcal{F}^k] \\\\\n\t\t\t&\\quad\\leq\n\t\t\t%\\Exp[\\norm{z^{k+1} - (x^\\star - \\lambda \\nabla F(x^\\star))}^2|\\mathcal{F}^k] \\\\\n\t\t\t%&\\quad=\n\t\t\t\\Exp[\\norm{x^k - \\tfrac{\\lambda}{n}\\big(\\tfrac{1}{p_{I^k}}(\\nabla f_{I^k}(x^k) - y_{I^k}^k) + \\textstyle\\sum_{i=1}^n y_i^k\\big) - x^\\star + \\lambda \\nabla F(x^\\star)}^2|\\mathcal{F}^k] \\\\\n\t\t\t&\\quad= \\norm{(x^k - \\lambda \\nabla F(x^k)) - (x^\\star - \\lambda \\nabla F(x^\\star))}^2\\\\\n\t\t&\\quad\\quad + \\lambda^2 \\Exp\\big[\\norm[\\big]{\\big(\\tfrac{1}{n p_{I^k}} \\nabla f_{I^k}(x^k) - \\nabla F(x^k)\\big) - \\big(\\tfrac{1}{n p_{I^k}}y_{I^k}^k - {\\textstyle\\frac{1}{n}\\sum_{i=1}^n} y_i^k \\big)}^2|\\mathcal{F}^k\\big] \\\\\n\t\t&\\quad= \\norm{x^k - x^\\star}^2 - 2\\lambda\\inprod{\\nabla F(x^k) - \\nabla F(x^\\star)}{x^k-x^\\star} + \\lambda^2\\norm{\\nabla F(x^k) -\\nabla F(x^\\star)}^2\\\\\n\t&\\quad\\quad + \\lambda^2 \\Exp\\big[\\norm[\\big]{\\big(\\tfrac{1}{n p_{I^k}} \\nabla f_{I^k}(x^k) - \\nabla F(x^k)\\big) - \\big(\\tfrac{1}{n p_{I^k}}y_{I^k}^k - {\\textstyle\\frac{1}{n}\\sum_{i=1}^n} y_i^k \\big)}^2|\\mathcal{F}^k\\big].\n\\end{aligned}\n\\end{equation}\nThe first equality is given by the solution being a fixed point to the proximal-gradient update $x^\\star = \\prox_{\\lambda g}(x^\\star - \\lambda \\nabla F(x^\\star))$ \\cite[Corollary 28.9]{bauschke_convex_2017}.\nThe first inequality is due to the non-expansiveness of $\\prox_{\\lambda g}$.\nThe second to last equality is given by $\\Exp\\norm{X}^2 =\\norm{\\Exp X}^2 + \\Exp\\norm{X - \\Exp X}^2$ where $X$ is a random variable.\n","sha256_utf8":"47da5c12d5a9642a589fa384584ebb0bc32667d43ef2b91509b2505a4106ef0a"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"随机梯度方差上界","source_units":[{"start":47843,"end":49706,"start_line":798,"end_line":814,"raw_tex":"The last term in \\Cref{eq:prim-update-lyap} satisfies the following upper bound for all $\\delta > 0$:\n\\begin{equation}\\label{eq:update-variance-bound}\n\t\\begin{aligned}\n\t\t&\\Exp[\\norm{(\\tfrac{1}{n p_{I^k}} \\nabla f_{I^k}(x^k) - \\nabla F(x^k)) - (\\tfrac{1}{n p_{I^k}}y_{I^k}^k - {\\textstyle\\frac{1}{n}\\sum_{i=1}^n} y_i^k )}^2|\\mathcal{F}^k]\\\\\n\t\t&\\quad\\begin{aligned}\n\t\t\t= \\Exp[\\| &\\tfrac{1}{np_{I^k}}(\\nabla f_{I^k}(x^k) - \\nabla f_{I^k}(x^\\star)) - (\\nabla F(x^k) - \\nabla F(x^\\star))\\\\\n\t\t\t&- \\tfrac{1}{np_{I^k}}(y_{I^k}^k - y_{I^k}^\\star) + {\\textstyle\\frac{1}{n}\\sum_{i=1}^n}(y_i^k - y_i^\\star) \\|^2 |\\mathcal{F}^k]\n\t\t\\end{aligned} \\\\\n\t\t&\\quad\\leq (1+\\delta)\\Exp[\\norm{ \\tfrac{1}{np_{I^k}}(\\nabla f_{I^k}(x^k) - \\nabla f_{I^k}(x^\\star)) - (\\nabla F(x^k) - \\nabla F(x^\\star)) }^2 |\\mathcal{F}^k]\\\\\n\t\t&\\quad\\quad + (1+\\delta^{-1})\\Exp[\\norm{\\tfrac{1}{np_{I^k}}(y_{I^k}^k - y_{I^k}^\\star) - {\\textstyle\\frac{1}{n}\\sum_{i=1}^n}(y_i^k - y_i^\\star)}^2 |\\mathcal{F}^k]\\\\\n\t\t&\\quad= (1+\\delta)\\big(\\Exp[\\norm{\\tfrac{1}{np_{I^k}}(\\nabla f_{I^k}(x^k) - \\nabla f_{I^k}(x^\\star))}^2|\\mathcal{F}^k] - \\norm{\\nabla F(x^k) - \\nabla F(x^\\star)}^2\\big) \\\\\n\t\t&\\quad\\quad + (1+\\delta^{-1})\\big(\\Exp[\\norm{\\tfrac{1}{np_{I^k}}(y_{I^k}^k - y_{I^k}^\\star)}^2|\\mathcal{F}^k] - \\norm{{\\textstyle\\frac{1}{n}\\sum_{i=1}^n}(y_i^k - y_i^\\star)}^2\\big) \\\\\n\t\t&\\quad= (1+\\delta){\\textstyle\\sum\\frac{1}{n^2 p_i}}\\norm{\\nabla f_i(x^k) - \\nabla f_i(x^\\star)}^2 - (1+\\delta)\\norm{\\nabla F(x^k) - \\nabla F(x^\\star)}^2 \\\\\n\t\t&\\quad\\quad + (1+\\delta^{-1}){\\textstyle\\sum\\frac{1}{n^2 p_i}}\\norm{y_i^k - y_i^\\star}^2 - (1+\\delta^{-1})\\norm{{\\textstyle\\frac{1}{n}\\sum_{i=1}^n}(y_i^k - y_i^\\star)}^2.\n\t\\end{aligned}\n\\end{equation}\nThe inequality is given by Young's inequality, the second to last equality is given by $\\Exp\\norm{X - \\Exp X}^2 = \\Exp\\norm{X}^2-\\norm{\\Exp X}^2$ where $X$ is a random variable.\n","sha256_utf8":"36440c7d8ea30d5a88d3f4cd95a7da1f16b8c346c2daf55cb7420d1325bde843"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"对偶更新条件期望","source_units":[{"start":49707,"end":50455,"start_line":816,"end_line":830,"raw_tex":"The dual updates satisfy\n\\begin{align*}\n\t\\norm{y_i^{k+1} - y_i^\\star}^2\n\t&= \\norm{y_i^k + U_i^k(\\nabla f_i(x^k) - y_i^k) - y_i^\\star}^2 \\\\\n\t&= (1- U_i^k)\\norm{y_i^k - y_i^\\star}^2 + U_i^k\\norm{\\nabla f_i(x^k) - \\nabla f_i(x^\\star)}^2\n\\end{align*}\nsince $U_i^k \\in \\{0,1\\}$.\nSumming over all terms, taking expected value and using linearity of the expected value give\n\\begin{equation}\\label{eq:dual-update-lyap}\n\t\\begin{aligned}\n\t\t\\Exp[{\\textstyle\\sum_{i=1}^n} \\widehat{\\gamma}_i\\norm{y_i^{k+1} - y_i^\\star}^2|\\mathcal{F}^k]\n\t\t&= {\\textstyle\\sum_{i=1}^n} (1-\\eta_i)\\widehat{\\gamma}_i\\norm{y^k - y^\\star}^2 \\\\\n\t\t&\\quad + {\\textstyle\\sum_{i=1}^n}{\\eta_i \\widehat{\\gamma}_i}\\norm{\\nabla f_i(x^k) - \\nabla f_i(x^\\star)}^2.\n\t\\end{aligned}\n\\end{equation}\n","sha256_utf8":"fbdee3f1a4ab46525c4cfe9833939a6b746fd1d0288af865ada39016c9346bd1"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"合并误差并递推收敛","source_units":[{"start":50455,"end":50842,"start_line":831,"end_line":832,"raw_tex":"Adding \\Cref{eq:dual-update-lyap} to \\Cref{eq:prim-update-lyap} and substituting in \\Cref{eq:update-variance-bound} and using the definition $\\frac{\\gamma_i}{\\gamma_i} = 1$ when $\\gamma_i = 0$ then yield \\Cref{eq:prop-lyapunov-bound}.\nApplying \\Cref{eq:prim-contract} and \\Cref{eq:dual-contract}, using the law of total expectation and telescoping the inequalities give the stated rate.\n","sha256_utf8":"462ab7f0e1357be1e8d6dac6586e9348d08fedbe6614cb18b6b5d6bd4e6e2356"}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","title":"原始变量收缩引理","source_units":[{"start":23257,"end":23645,"start_line":425,"end_line":431,"raw_tex":"\\begin{lem}[Primal Contraction]\\label{lem:prim-contract}%\n\tLet \\Cref{ass:strong-ass} hold, the primal iterates of \\Cref{alg:pvrsg} satisfy the primal contraction \\Cref{eq:prim-contract} with\n\t\\begin{align*}\n\t\t\\rho_P = \\mu\\lambda(2 - \\nu\\lambda)\n\t\\end{align*}\n\twhere $\\nu = \\max_i {\\textstyle  (1+\\delta^{-1})\\frac{L_i\\eta_i\\gamma_i}{n p_i}  +  (1+\\delta)\\frac{L_i}{n p_i} - \\delta\\mu}$.\n\t","sha256_utf8":"07759d776b262cacb46c2f9bf9076ee6ea15f70736b549a3382c73ca73692da4"},{"start":23706,"end":23716,"start_line":433,"end_line":434,"raw_tex":"\n\\end{lem}","sha256_utf8":"401658723a1e057444124cf77a6be177a3c9cb2efee9af44c2830fc4a15ddaba"}]},{"id":"S027","semantic_type":"proof","kind":"proof","title":"原始收缩完整证明","source_units":[{"start":23645,"end":23706,"start_line":431,"end_line":433,"raw_tex":"\\begin{proof}\n\t\tSee \\Cref{sec:proof-prop-lemma}.\n\t\\end{proof}","sha256_utf8":"348bf3f69d77e9410af220b3a95bf45811f3f3b3b4be8c1ccdc6d3a6d23ff838"},{"start":50856,"end":52798,"start_line":836,"end_line":868,"raw_tex":"\\begin{proof}[\\textbf{Proof of \\Cref{lem:prim-contract}}]%\n\tFirst note that $\\mu$-strong monotonicity and the Cauchy--Schwarz inequality imply\n\t\\begin{equation}\\label{eq:reverse-lipschitz}\n\t\t\\norm{\\nabla F(x^k) - \\nabla F(x^\\star)} \\geq \\mu \\norm{x^k - x^\\star}.\n\t\\end{equation}\n\tConsider the terms of $\\mathcal{V}(x^k)$.\n\tUsing \\Cref{eq:reverse-lipschitz} and Cauchy--Schwarz in the last term yield\n\t\\begin{align*}\n\t\t\\norm{\\nabla F(x^k) - \\nabla F(x^\\star)}^2\n\t\t&\\geq \\mu\\norm{\\nabla F(x^k) - \\nabla F(x^\\star)}\\norm{x^k - x^\\star} \\\\\n\t\t&\\geq \\mu\\inprod{\\nabla F(x^k) - \\nabla F(x^\\star)}{x^k - x^\\star}.\n\t\\end{align*}\n\tUsing $\\frac{1}{L_i}$-cocoercivity of $\\nabla f_i$ in the first term of $\\mathcal{V}(x^k)$ yields\n\t\\begin{align*}\n\t\t\\sum_{i=1}^n & {\\textstyle (\\frac{\\eta_i\\gamma_i}{\\delta}  + 1)\\frac{1}{n^2 p_i} }\\norm{\\nabla f_i(x^k) - \\nabla f_i(x^\\star)}^2 \\\\\n\t\t&\\leq \\sum_{i=1}^n {\\textstyle (\\frac{\\eta_i\\gamma_i}{\\delta}  + 1)\\frac{L_i}{n^2 p_i} } \\inprod{\\nabla f_i(x^k) - \\nabla f_i(x^\\star)}{x^k - x^\\star} \\\\\n\t\t&\\leq \\max_i({\\textstyle(\\frac{\\eta_i\\gamma_i}{\\delta}  + 1)\\frac{L_i}{n p_i}}) \\inprod{\\nabla F(x^k) - \\nabla F(x^\\star)}{x^k - x^\\star}.\n\t\\end{align*}\n\tAdding the terms back together yields\n\t\\begin{align*}\n\t\t\\mathcal{V}(x^k)\n\t\t&\\leq \\lambda^2 \\nu \\inprod{\\nabla F(x^k) - \\nabla F(x^\\star)}{x^k - x^\\star}\n\t\\end{align*}\n\twhere $\\nu = \\max_i(  (1+\\delta^{-1})\\frac{L_i\\eta_i\\gamma_i}{n p_i}  +  (1+\\delta)\\frac{L_i}{n p_i} - \\delta\\mu)$.\n\tThis can now be summarized as\n\t\\begin{align*}\n\t\t\\mathcal{P}(x^k)\n\t\t&\\leq \\norm{x^k - x^\\star}^2  - \\lambda\\big(2 - \\nu\\lambda \\big) \\inprod{\\nabla F(x^k) - \\nabla F (x^\\star)}{x^k - x^\\star} \\\\\n\t\t&\\leq \\norm{x^k - x^\\star}^2  - \\mu\\lambda\\big(2 - \\nu\\lambda \\big) \\norm{x^k - x^\\star}^2 \\\\\n\t\t&= (1 - \\rho_P)\\norm{x^k - x^\\star}^2,\n\t\\end{align*}\n\twhere $\\rho_P = \\mu\\lambda(2 - \\nu\\lambda)$ and the last inequality is given by the strong monotonicity of $\\nabla F$.\n\\end{proof}","sha256_utf8":"a33b8551f4f6c2d7a8b1c9f4d713e0e7c304ac478612df4ac078fd34d2c06f5b"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"原始收缩证明位置","source_units":[{"start":23659,"end":23694,"start_line":432,"end_line":432,"raw_tex":"\t\tSee \\Cref{sec:proof-prop-lemma}.\n","sha256_utf8":"7932c28351c7f0d3375c661cebf6e7f8a405a960fe79c83e365609deecd00ed5"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"梯度差的反向范数界","source_units":[{"start":50915,"end":51135,"start_line":837,"end_line":840,"raw_tex":"\tFirst note that $\\mu$-strong monotonicity and the Cauchy--Schwarz inequality imply\n\t\\begin{equation}\\label{eq:reverse-lipschitz}\n\t\t\\norm{\\nabla F(x^k) - \\nabla F(x^\\star)} \\geq \\mu \\norm{x^k - x^\\star}.\n\t\\end{equation}\n","sha256_utf8":"66f0317b7483eae7d92f36e4269fde821de9879f4b3c2da52ab5a54744531dff"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"控制总梯度平方项","source_units":[{"start":51135,"end":51476,"start_line":841,"end_line":847,"raw_tex":"\tConsider the terms of $\\mathcal{V}(x^k)$.\n\tUsing \\Cref{eq:reverse-lipschitz} and Cauchy--Schwarz in the last term yield\n\t\\begin{align*}\n\t\t\\norm{\\nabla F(x^k) - \\nabla F(x^\\star)}^2\n\t\t&\\geq \\mu\\norm{\\nabla F(x^k) - \\nabla F(x^\\star)}\\norm{x^k - x^\\star} \\\\\n\t\t&\\geq \\mu\\inprod{\\nabla F(x^k) - \\nabla F(x^\\star)}{x^k - x^\\star}.\n\t\\end{align*}\n","sha256_utf8":"aa21da30142021eff1a91214d8ff450e7e31fde565defe47f16bb90d51a2d446"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"余强制性控制分量项","source_units":[{"start":51476,"end":52036,"start_line":848,"end_line":853,"raw_tex":"\tUsing $\\frac{1}{L_i}$-cocoercivity of $\\nabla f_i$ in the first term of $\\mathcal{V}(x^k)$ yields\n\t\\begin{align*}\n\t\t\\sum_{i=1}^n & {\\textstyle (\\frac{\\eta_i\\gamma_i}{\\delta}  + 1)\\frac{1}{n^2 p_i} }\\norm{\\nabla f_i(x^k) - \\nabla f_i(x^\\star)}^2 \\\\\n\t\t&\\leq \\sum_{i=1}^n {\\textstyle (\\frac{\\eta_i\\gamma_i}{\\delta}  + 1)\\frac{L_i}{n^2 p_i} } \\inprod{\\nabla f_i(x^k) - \\nabla f_i(x^\\star)}{x^k - x^\\star} \\\\\n\t\t&\\leq \\max_i({\\textstyle(\\frac{\\eta_i\\gamma_i}{\\delta}  + 1)\\frac{L_i}{n p_i}}) \\inprod{\\nabla F(x^k) - \\nabla F(x^\\star)}{x^k - x^\\star}.\n\t\\end{align*}\n","sha256_utf8":"7fa672f377df128b9bd0c52fb84f14ad6bf23ed5ccf30ed70b3d9dab9d241c46"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"合并得到方差分量界","source_units":[{"start":52036,"end":52321,"start_line":854,"end_line":859,"raw_tex":"\tAdding the terms back together yields\n\t\\begin{align*}\n\t\t\\mathcal{V}(x^k)\n\t\t&\\leq \\lambda^2 \\nu \\inprod{\\nabla F(x^k) - \\nabla F(x^\\star)}{x^k - x^\\star}\n\t\\end{align*}\n\twhere $\\nu = \\max_i(  (1+\\delta^{-1})\\frac{L_i\\eta_i\\gamma_i}{n p_i}  +  (1+\\delta)\\frac{L_i}{n p_i} - \\delta\\mu)$.\n","sha256_utf8":"9bf4620c05e02b289cb93f41a076ab1dfbd360c667b99081f42503e08aded5ef"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"强单调性推出原始收缩","source_units":[{"start":52321,"end":52787,"start_line":860,"end_line":867,"raw_tex":"\tThis can now be summarized as\n\t\\begin{align*}\n\t\t\\mathcal{P}(x^k)\n\t\t&\\leq \\norm{x^k - x^\\star}^2  - \\lambda\\big(2 - \\nu\\lambda \\big) \\inprod{\\nabla F(x^k) - \\nabla F (x^\\star)}{x^k - x^\\star} \\\\\n\t\t&\\leq \\norm{x^k - x^\\star}^2  - \\mu\\lambda\\big(2 - \\nu\\lambda \\big) \\norm{x^k - x^\\star}^2 \\\\\n\t\t&= (1 - \\rho_P)\\norm{x^k - x^\\star}^2,\n\t\\end{align*}\n\twhere $\\rho_P = \\mu\\lambda(2 - \\nu\\lambda)$ and the last inequality is given by the strong monotonicity of $\\nabla F$.\n","sha256_utf8":"69cdd202d8c9367009ae9bef56896215ecb0dc7bd3f39b561f6d877b1143cb0a"}]},{"id":"S034","semantic_type":"lemma","kind":"lemma","title":"对偶变量收缩引理","source_units":[{"start":23717,"end":24042,"start_line":435,"end_line":440,"raw_tex":"\\begin{lem}[Dual Contraction]\\label{lem:dual-contract}%\n\tLet \\Cref{ass:strong-ass} and $\\gamma_i > 0$ hold for all $i \\in\\{1,\\dots,n\\}$, the dual iterates of \\Cref{alg:pvrsg} satisfy the dual contraction \\Cref{eq:dual-contract} with\n\t\\begin{align*}\n\t\t\\rho_D = \\min_i {\\textstyle \\eta_i - \\tfrac{1}{\\gamma_i}}.\n\t\\end{align*}\n\t","sha256_utf8":"5ea93aba0cd32c214f0acda11574c6c66e22372c7f586983bf39cbe8eff5d4df"},{"start":24103,"end":24113,"start_line":442,"end_line":443,"raw_tex":"\n\\end{lem}","sha256_utf8":"401658723a1e057444124cf77a6be177a3c9cb2efee9af44c2830fc4a15ddaba"}]},{"id":"S035","semantic_type":"proof","kind":"proof","title":"对偶收缩完整证明","source_units":[{"start":24042,"end":24103,"start_line":440,"end_line":442,"raw_tex":"\\begin{proof}\n\t\tSee \\Cref{sec:proof-prop-lemma}.\n\t\\end{proof}","sha256_utf8":"348bf3f69d77e9410af220b3a95bf45811f3f3b3b4be8c1ccdc6d3a6d23ff838"},{"start":52801,"end":53206,"start_line":871,"end_line":879,"raw_tex":"\\begin{proof}[\\textbf{Proof of \\Cref{lem:dual-contract}}]%\n\tSince norms are non-negative, we have\n\t\\begin{align*}\n\t\t\\mathcal{D}(y^k)\n\t\t\\leq \\sum_{i=1}^n {\\textstyle(1 - \\eta_i  + \\frac{1}{\\gamma_i})\\widehat{\\gamma}_i} \\norm{y_i^k - y_i^\\star}^2\n\t\t\\leq (1 - \\rho_D)\\sum_{i=1}^n \\widehat{\\gamma}_i \\norm{y_i^k - y_i^\\star}^2,\n\t\\end{align*}\n\twhere $\\rho_D = \\min_i(\\eta_i - \\tfrac{1}{\\gamma_i})$.\n\\end{proof}","sha256_utf8":"c1c2a10cbbe424a4ef1666283c6cf492f4feafcca2cdd975995471acf0df4a1d"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"对偶收缩证明位置","source_units":[{"start":24056,"end":24091,"start_line":441,"end_line":441,"raw_tex":"\t\tSee \\Cref{sec:proof-prop-lemma}.\n","sha256_utf8":"7932c28351c7f0d3375c661cebf6e7f8a405a960fe79c83e365609deecd00ed5"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"舍去负项得到对偶收缩","source_units":[{"start":52860,"end":53195,"start_line":872,"end_line":878,"raw_tex":"\tSince norms are non-negative, we have\n\t\\begin{align*}\n\t\t\\mathcal{D}(y^k)\n\t\t\\leq \\sum_{i=1}^n {\\textstyle(1 - \\eta_i  + \\frac{1}{\\gamma_i})\\widehat{\\gamma}_i} \\norm{y_i^k - y_i^\\star}^2\n\t\t\\leq (1 - \\rho_D)\\sum_{i=1}^n \\widehat{\\gamma}_i \\norm{y_i^k - y_i^\\star}^2,\n\t\\end{align*}\n\twhere $\\rho_D = \\min_i(\\eta_i - \\tfrac{1}{\\gamma_i})$.\n","sha256_utf8":"caa88298c80927e6238fafeb43ea28fb782355a7368e1123648b12d5b8cea15d"}]},{"id":"S038","semantic_type":"lemma","kind":"lemma","title":"采样光滑常数下界","source_units":[{"start":25245,"end":25523,"start_line":460,"end_line":463,"raw_tex":"\\begin{lem}\\label{lem:sampled-lipschitz}%\n\tLet $L_i$ and $\\mu$ be from \\Cref{ass:strong-ass} and $p_i$ from \\Cref{alg:pvrsg}, then $\\max_i \\frac{L_i}{np_i} \\geq \\mu$.\n\tFurthermore, if $\\max_i \\frac{L_i}{np_i} = \\mu$ then $\\frac{L_i}{np_i} = \\mu$ for all $i \\in \\{1,\\dots,n\\}$.\n\t","sha256_utf8":"05c412a975b624a8c826c62a494804440b1cea02c5922f2a91e29e159c97fd15"},{"start":25584,"end":25594,"start_line":465,"end_line":466,"raw_tex":"\n\\end{lem}","sha256_utf8":"401658723a1e057444124cf77a6be177a3c9cb2efee9af44c2830fc4a15ddaba"}]},{"id":"S039","semantic_type":"proof","kind":"proof","title":"采样常数引理完整证明","source_units":[{"start":25523,"end":25584,"start_line":463,"end_line":465,"raw_tex":"\\begin{proof}\n\t\tSee \\Cref{sec:proof-prop-lemma}.\n\t\\end{proof}","sha256_utf8":"348bf3f69d77e9410af220b3a95bf45811f3f3b3b4be8c1ccdc6d3a6d23ff838"},{"start":55674,"end":56515,"start_line":926,"end_line":937,"raw_tex":"\\begin{proof}[\\textbf{Proof of \\Cref{lem:sampled-lipschitz}}]%\n\tAn $L$-smooth and $\\mu$-strongly convex function must satisfy $L \\geq \\mu$.\n\tAssuming $\\max_i(\\frac{L_i}{np_i}) < \\mu$ yields the following contradiction\n\t\\begin{align*}\n\t\t\\mu > \\max_i \\tfrac{L_i}{np_i} = \\sum_{j=1}^n  p_j \\max_i \\tfrac{L_i}{np_i} \\geq \\sum_{j=1}^n  p_j \\tfrac{L_j}{np_j} = \\sum_{i=1}^n \\tfrac{L_i}{n} \\geq L.\n\t\\end{align*}\n\tIf $\\max_i(\\frac{L_i}{np_i}) = \\mu$, equality must hold everywhere and we have\n\t\\begin{align*}\n\t\t0 = \\sum_{j=1}^n  p_j \\max_i \\tfrac{L_i}{np_i} - \\sum_{j=1}^n  p_j \\tfrac{L_j}{np_j} = \\sum_{j=1}^n  p_j (\\max_i \\tfrac{L_i}{np_i} - \\tfrac{L_j}{np_j}).\n\t\\end{align*}\n\tSince $p_j > 0$ and $\\max_i \\tfrac{L_i}{np_i} - \\tfrac{L_j}{np_j} \\geq 0$, we have $\\max_i \\tfrac{L_i}{np_i} = \\tfrac{L_j}{np_j}$ for all $j\\in\\{1,\\dots,n\\}$.\n\\end{proof}","sha256_utf8":"3146d4c2a359f15209232d4d9e37ccc0ecb451925cba79bd9737281a5b70b181"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"采样引理证明位置","source_units":[{"start":25537,"end":25572,"start_line":464,"end_line":464,"raw_tex":"\t\tSee \\Cref{sec:proof-prop-lemma}.\n","sha256_utf8":"7932c28351c7f0d3375c661cebf6e7f8a405a960fe79c83e365609deecd00ed5"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"加权平均反证下界","source_units":[{"start":55737,"end":56079,"start_line":927,"end_line":931,"raw_tex":"\tAn $L$-smooth and $\\mu$-strongly convex function must satisfy $L \\geq \\mu$.\n\tAssuming $\\max_i(\\frac{L_i}{np_i}) < \\mu$ yields the following contradiction\n\t\\begin{align*}\n\t\t\\mu > \\max_i \\tfrac{L_i}{np_i} = \\sum_{j=1}^n  p_j \\max_i \\tfrac{L_i}{np_i} \\geq \\sum_{j=1}^n  p_j \\tfrac{L_j}{np_j} = \\sum_{i=1}^n \\tfrac{L_i}{n} \\geq L.\n\t\\end{align*}\n","sha256_utf8":"d3c8379b3d3b8f7a9ebc087c19f141192f60bcb41b4ad67f0ca97e0f53b3abb6"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"等号迫使所有常数相同","source_units":[{"start":56079,"end":56504,"start_line":932,"end_line":936,"raw_tex":"\tIf $\\max_i(\\frac{L_i}{np_i}) = \\mu$, equality must hold everywhere and we have\n\t\\begin{align*}\n\t\t0 = \\sum_{j=1}^n  p_j \\max_i \\tfrac{L_i}{np_i} - \\sum_{j=1}^n  p_j \\tfrac{L_j}{np_j} = \\sum_{j=1}^n  p_j (\\max_i \\tfrac{L_i}{np_i} - \\tfrac{L_j}{np_j}).\n\t\\end{align*}\n\tSince $p_j > 0$ and $\\max_i \\tfrac{L_i}{np_i} - \\tfrac{L_j}{np_j} \\geq 0$, we have $\\max_i \\tfrac{L_i}{np_i} = \\tfrac{L_j}{np_j}$ for all $j\\in\\{1,\\dots,n\\}$.\n","sha256_utf8":"385f5470d1811abfd7d27fa8efc47c5f821a08a3d7bba18a1947ba5951f439f8"}]},{"id":"S043","semantic_type":"external_fact","kind":"external_fact","title":"近端梯度最优性条件","source_units":[],"declared_external":true,"declaration_text":["For the stated convex composite problem, a minimizer x* satisfies x* = prox_{lambda g}(x* - lambda grad F(x*)) for every lambda > 0, as cited in bauschke_convex_2017, Corollary 28.9."]},{"id":"S044","semantic_type":"external_fact","kind":"external_fact","title":"近端算子非扩张定理","source_units":[],"declared_external":true,"declaration_text":["For closed proper convex g, prox_g is single-valued and satisfies ||prox_g(u)-prox_g(v)|| <= ||u-v||; cited Proposition 12.28 of bauschke_convex_2017."]},{"id":"S045","semantic_type":"external_fact","kind":"external_fact","title":"强凸函数基本性质","source_units":[],"declared_external":true,"declaration_text":["The subdifferential of a mu-strongly convex function is mu-strongly monotone; a closed proper strongly convex function has a unique minimizer. These are the cited Example 22.4 and Corollary 11.17 of bauschke_convex_2017."]},{"id":"S046","semantic_type":"external_fact","kind":"external_fact","title":"条件期望与方差恒等式","source_units":[],"declared_external":true,"declaration_text":["Conditional expectation is linear and obeys the tower property. For a square-integrable random vector X, E[||X||^2|F] = ||E[X|F]||^2 + E[||X-E[X|F]||^2|F]."]},{"id":"S047","semantic_type":"external_fact","kind":"external_fact","title":"柯西与杨氏不等式","source_units":[],"declared_external":true,"declaration_text":["Cauchy–Schwarz gives |<a,b>| <= ||a||||b||. For delta > 0, Young's inequality gives ||a+b||^2 <= (1+delta)||a||^2 + (1+delta^{-1})||b||^2."]},{"id":"S048","semantic_type":"external_fact","kind":"external_fact","title":"严格凸强制函数极小值","source_units":[],"declared_external":true,"declaration_text":["A continuous strictly convex function on (0,infinity) that tends to infinity as its argument approaches either endpoint attains its minimum at exactly one positive point."]},{"id":"S049","semantic_type":"external_fact","kind":"external_fact","title":"光滑性与余强制性等价","source_units":[],"declared_external":true,"declaration_text":["For convex differentiable f, L-Lipschitz continuity of grad f is equivalent to 1/L-cocoercivity; cited Corollary 18.17 of bauschke_convex_2017."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S011","edges":[{"source_id":"S001","target_id":"S003","kind":"supports"},{"source_id":"S001","target_id":"S004","kind":"supports"},{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S022","kind":"supports"},{"source_id":"S002","target_id":"S029","kind":"supports"},{"source_id":"S002","target_id":"S031","kind":"supports"},{"source_id":"S002","target_id":"S041","kind":"supports"},{"source_id":"S003","target_id":"S010","kind":"supports"},{"source_id":"S003","target_id":"S022","kind":"supports"},{"source_id":"S004","target_id":"S022","kind":"supports"},{"source_id":"S004","target_id":"S023","kind":"supports"},{"source_id":"S004","target_id":"S024","kind":"supports"},{"source_id":"S004","target_id":"S041","kind":"supports"},{"source_id":"S004","target_id":"S042","kind":"supports"},{"source_id":"S005","target_id":"S024","kind":"supports"},{"source_id":"S005","target_id":"S015","kind":"supports"},{"source_id":"S006","target_id":"S022","kind":"supports"},{"source_id":"S006","target_id":"S023","kind":"supports"},{"source_id":"S007","target_id":"S031","kind":"supports"},{"source_id":"S007","target_id":"S041","kind":"supports"},{"source_id":"S008","target_id":"S003","kind":"supports"},{"source_id":"S008","target_id":"S029","kind":"supports"},{"source_id":"S008","target_id":"S033","kind":"supports"},{"source_id":"S008","target_id":"S041","kind":"supports"},{"source_id":"S009","target_id":"S022","kind":"supports"},{"source_id":"S010","target_id":"S023","kind":"supports"},{"source_id":"S010","target_id":"S024","kind":"supports"},{"source_id":"S010","target_id":"S025","kind":"supports"},{"source_id":"S010","target_id":"S030","kind":"supports"},{"source_id":"S010","target_id":"S031","kind":"supports"},{"source_id":"S010","target_id":"S032","kind":"supports"},{"source_id":"S010","target_id":"S033","kind":"supports"},{"source_id":"S010","target_id":"S037","kind":"supports"},{"source_id":"S010","target_id":"S016","kind":"supports"},{"source_id":"S010","target_id":"S018","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S014","target_id":"S012","kind":"supports"},{"source_id":"S015","target_id":"S012","kind":"supports"},{"source_id":"S016","target_id":"S012","kind":"supports"},{"source_id":"S017","target_id":"S012","kind":"supports"},{"source_id":"S018","target_id":"S012","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S014","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S012","target_id":"S011","kind":"supports"},{"source_id":"S019","target_id":"S014","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S020","kind":"supports"},{"source_id":"S023","target_id":"S020","kind":"supports"},{"source_id":"S024","target_id":"S020","kind":"supports"},{"source_id":"S025","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S026","target_id":"S014","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S027","kind":"supports"},{"source_id":"S031","target_id":"S027","kind":"supports"},{"source_id":"S032","target_id":"S027","kind":"supports"},{"source_id":"S033","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S034","target_id":"S014","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S017","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S041","target_id":"S039","kind":"supports"},{"source_id":"S042","target_id":"S039","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S043","target_id":"S022","kind":"supports"},{"source_id":"S044","target_id":"S009","kind":"supports"},{"source_id":"S045","target_id":"S008","kind":"supports"},{"source_id":"S046","target_id":"S022","kind":"supports"},{"source_id":"S046","target_id":"S023","kind":"supports"},{"source_id":"S046","target_id":"S024","kind":"supports"},{"source_id":"S046","target_id":"S025","kind":"supports"},{"source_id":"S047","target_id":"S023","kind":"supports"},{"source_id":"S047","target_id":"S029","kind":"supports"},{"source_id":"S047","target_id":"S030","kind":"supports"},{"source_id":"S048","target_id":"S016","kind":"supports"},{"source_id":"S049","target_id":"S007","kind":"supports"}]}}
{"record_id":"2002.06309v1","paper":{"title":"Stochastic optimization over proximally smooth sets","authors":["Damek Davis","Dmitriy Drusvyatskiy","Zhan Shi"],"arxiv_id":"2002.06309","version":"v1","journal":"SIAM Journal on Optimization","doi":"10.1137/20m1320225","abstract_url":"https://arxiv.org/abs/2002.06309v1","pdf_url":"https://arxiv.org/pdf/2002.06309v1"},"context":{"raw_tex":"Throughout the work, \nwe \nconsider \nthe Euclidean space $\\mathbb{R}^d$, \nwhich we \nequip \nwith \na fixed inner product $\\langle\\cdot,\\cdot\\rangle$ \nand \nits induced norm $\\|\\cdot\\|=\\sqrt{\\dotp{\\cdot,\\cdot}}$.  \nWe \nlet \nthe symbol $\\BB(x,r)$ \ndenote \nthe closed ball \nof \nradius $r>0$ \naround \na point $x$.  \nWe \ndefine  \nthe \ndistance \nand \nthe nearest-point projection \nof \na point $y\\in\\R^d$ \nonto \na set $\\cX\\subset\\R^d$ \nby\n\\[\n\\dist(y,\\cX):=\\inf_{x\\in \\cX}~\\|x-y\\|\\qquad \\textrm{and}\\qquad\\proj_{\\cX}(y):=\\argmin_{x\\in \\cX}~\\|x-y\\|,\n\\]\nrespectively. \nFor any real number $r>0$, \nwe\ndefine \nthe {\\em $r$-tube} \naround \n$\\cX$ \nto be the set\n$\nU_{\\cX}(r):=\\{y\\in\\R^d:\\dist(y,\\cX)<r\\}.\n$\n\n\nConsider a function $f\\colon\\R^d\\to\\R\\cup\\{\\infty\\}$ and a point $\\bar x$, with $f(\\bar x)$ finite. \nThe {\\em subdifferential} of $f$ at $\\bar x$, denoted by $\\partial f(\\bar x)$, consists of all vectors $v\\in\\R^d$ satisfying\n$$f(x)\\geq f(\\bar x)+\\langle v,x-\\bar x\\rangle+o(\\|x-\\bar x\\|)\\qquad \\textrm{as } x\\to \\bar x.$$ \nWe will often use the following basic variational construction. The \\textit{normal cone} to a set $\\cX$ at a point $\\bar x\\in \\cX$, denoted $N_{\\cX}(\\bar x)$, consists of all vectors $v\\in\\R^d$ satisfying \n\\begin{equation}\\label{eqn:frech_norm}\n\\langle v,x-\\bar x\\rangle\\leq o(\\|x-\\bar x\\|) \\qquad \\textrm{as }x\\to \\bar x \\textrm{ in }\\cX.\n\\end{equation}\n\n\nFirst, we say that a function $f\\colon\\R^d\\to\\R\\cup\\{\\infty\\}$ is {\\em $\\rho$-weakly convex}, for some $\\rho>0$, if the perturbed function $x\\mapsto f(x)+\\frac{\\rho}{2}\\|x\\|^2$ is convex. Second, following \\cite{clarkelower}, we say that a closed set $\\cX$ is {\\em $R$-proximally smooth} if the projection $\\proj_{\\cX}(x)$ is a singleton whenever $\\dist(x,\\cX)<R$. Although these conditions appear at first unrelated, they are deeply connected in the sense that weak convexity of a function $f$ is essentially equivalent to proximal smoothness of its epigraph. For a precise statement, we refer the reader to \n\\cite[Section 5]{clarkelower}.\n\n\nFor example, \nif $f$ \nis \na $\\rho$-weakly convex function, \nthen the inequality\n\\begin{equation}\\label{eqn:subgrad_ineq}\nf(y)\\ge f(x)+\\dotp{v,y-x}-\\frac{\\rho}{2}\\|y-x\\|^2,\\qquad \\textrm{holds for all }x,y \\in \\R^d,~v\\in \\partial f(x).\n\\end{equation}\n\nHenceforth, we consider the optimization problem\n\\begin{equation}\\label{eqn:target_prob}\n\\min ~ f(x)\\quad \\textrm{subject to}\\quad x\\in \\cX,\n\\end{equation}\nfor some closed function $f\\colon\\R^d\\to\\R$ that is bounded from below on a closed set $\\cX\\subseteq\\R^d$. We will place further assumptions on $f$ and $\\cX$ shortly.\n\n\nNamely, \ndefine \nthe {\\em Moreau envelope} \nand \nthe {\\em proximal map}, respectively, by\n$$\\cM_{\\lambda}(x)= \\min_{y\\in \\cX}~ \\left\\{f(y)+\\frac{1}{2\\lambda}\\|y-x\\|^2\\right\\},\\qquad\\cP_{\\lambda}(x):=\\argmin_{y\\in \\cX}~ \\left\\{f(y)+\\frac{1}{2\\lambda}\\|y-x\\|^2\\right\\},$$\nand define the {\\em stationarity measure}:\n$$\\mathcal{C}_{\\lambda}(x):=\\lambda^{-1}\\cdot\\dist(x,\\cP_{\\lambda}(x)).$$\n\n\n\\begin{definition}[Set-approximation]\n\t{\\rm Consider a set $\\cX\\subset\\R^d$ and a collection of sets $\\cX_x\\subset\\R^d$ and functions $\\cR_x\\colon\\cX_x\\to\\cX$, indexed by points $x\\in \\R^d$. We say that the collection $\\{(\\cX_x,\\cR_{x})\\}_{x\\in\\cX}$ is a {\\em set approximation of $\\cX$ with parameters $(R,\\accD,r_1,\\accR ,r_2)$}  if the following conditions hold:\n\t\t\\begin{enumerate}\n\t\t\t\\item [$(i)$] {\\bf (Accuracy)} For every $x \\in \\cX$, the set $\\cX_x$ is $R$-proximally smooth and satisfies\n\t\t\t\\begin{equation}\\label{eqn:need_this_eqn}\n\t\t\t\\dist(y, \\cX_{x})  \\leq \\frac{\\accD}{2} \\| x- y\\|^2 \\qquad \\forall y \\in \\cX\\cap \\BB(x,r_1).\n\t\t\t\\end{equation}\n\t\t\t%\t\t\t\\textcolor{red}{Why need this???}\n\t\t\t\n\t\t\t\\item[$(ii)$] {\\bf (Retraction)} For every $x \\in \\cX$, the mapping $\\cR_x$ satisfies\n\t\t\t$$\n\t\t\t\\|y - \\cR_x(y) \\| \\leq \\frac{\\accR }{2} \\|x-y\\|^2 \\qquad \\forall y \\in \\cX_x \\cap \\BB(x,r_2).\n\t\t\t$$\n\t\t\\end{enumerate}\n\t}\n\\end{definition}\n\n\tNamely, \n\tthe algorithm we propose (Algorithm~\\ref{alg:stoc_prox}) \n\twill use \n\tthree building blocks: \n\ta family \n\tof \n\tstochastic models $f_{x}(\\cdot,\\xi)$ \n\tof \n\tthe objective function, \n\ta family \n\tof \n\tproximally smooth sets $\\cX_x$ \n\tthat \n\tapproximate $\\cX$ near $x$, \n\tand \n\ta retraction operation $\\cR_x\\colon\\cX_x\\to\\cX$ \n\tthat \n\trestores feasibility. \n\tThus in each iteration $t$, \n\tAlgorithm~\\ref{alg:stoc_prox} \n\tforms \n\tboth \n\ta proximally smooth approximation $\\cX_{x_t}$ \n\tof \n\tthe original constraint \n\tand  \n\ta stochastic model $f_{x_t}(\\cdot,\\xi_t)$ \n\tof \n\tthe objective function, \n\tcentered at the current iterate $x_t$. \n\tThe procedure then \n\tcomputes \n\ta minimizer ${\\tilde x}_t$ \n\tof \n\tthe function  $f_{x_t}(\\cdot,\\xi_t)+\\frac{\\beta}{2}\\|\\cdot-x_t\\|^2$ \n\tover\n\t$\\cX_{x_t}$ \n\tand \n\tretracts \n\tit back to $\\cX$, thereby defining the next iterate $x_{t+1}=\\cR_{x_t}(\\tilde x_t)$. \n\n\n\\begin{algorithm}[H]\n\t\t{\\bf Input:} initialization $x_0\\in \\R^d$,   a sequence $\\beta_t>0$,  and an iteration count $T\\in \\mathbb{N}$.\n\t\t\\break{\\bf Step } $t=0,\\ldots,T$:\t\n\t\t\\begin{equation*}\n\t\t\\begin{aligned}\n\t\t&\\textrm{Sample } \\xi_t \\sim P\\\\\n\t\t& \\textrm{Choose } \\tilde x_{t} \\in \\argmin_{x \\in \\cX_x}~ \\left\\{f_{x_t}(x,\\xi_t) + \\frac{\\beta_t}{2} \\|x - x_t\\|^2\\right\\}\\\\\n\t\t& \\textrm{Set }  x_{t+1} = \\cR_{x_t}(\\tilde x_t)\n\t\t\\end{aligned}\n\t\t\\end{equation*}\n\t\t\\caption{Retracted Stochastic Model Based Algorithm\n\t\t\t%: PSSM($x_0,T,\\{\\alpha_t\\}$)\n\t\t}\n\t\t\\label{alg:stoc_prox}\n\t\\end{algorithm}\n\n\\begin{assumption}\n\t\t{\\rm\n\t\t\tFix a probability space $(\\Omega,\\mathcal{F},P)$ and equip $\\R^d$ with the Borel $\\sigma$-algebra. \n\t\t\tWe assume that there exist real $\\eta,\\osa, L, R, \\accD, \\accR, r_1, r_2 \\in \\R$ satisfying the following properties. \n\t\t\t\\begin{enumerate}\n\t\t\t\t\\item[(B1)] {\\bf (Sampling)} It is possible to generate i.i.d.\\ realizations $\\xi_1,\\xi_2, \\ldots \\sim P$.\n\t\t\t\t\\item[(B2)]\\label{it:B2} {\\bf (One-sided accuracy)} There is an open set $U$ containing $\\cX $ and a measurable function $(x,y,\\xi)\\mapsto f_x(y,\\xi)$, defined on $U\\times U\\times\\Omega$, satisfying  $$\\EE_{\\xi}\\left[f_x(x,\\xi)\\right]=f(x)\n\t\t\t\t\\quad \\textrm{and} \n\t\t\t\t\\quad\\EE_{\\xi}\\left[f_x(y,\\xi)-f(y)\\right]\n\t\t\t\t\\leq\\frac{\\osa}{2}\\|y-x\\|^2\\qquad \\forall x,y\\in U.$$\n\t\t\t\t\\item[(B3)] {\\bf (Weak-convexity)} The function $f_x(\\cdot,\\xi)$ is $\\eta$-weakly convex $\\forall x\\in U$, a.e. $\\xi\\sim P$.\n\t\t\t\t\\item [(B4)] {\\bf (Set approximation)} There exists a collection $\\{(\\cX_x,\\cR_{x})\\}_{x\\in\\cX}$ that is a set approximation of $\\cX$ with parameters $(R,\\accD,r_1,\\accR ,r_2)$.\n\t\t\t\t\\item[(B6)] {\\bf (Lipschitz property)} $f$ is $L$-Lipschitz continuous on $\\cX$; and for all $x\\in \\cX$ and a.e. $\\xi\\sim P$, the function $f_x(\\cdot, \\xi)$ is $L$-Lipschitz continuous on some neighborhood of $\\cX_x$.\n\t\t\\end{enumerate}}\n\t\\end{assumption}\n\n\tIn what follows, we let   $x_0,\\ldots, x_t$ and $\\xi_0,\\ldots, \\xi_{t}$ be the iterates and random elements generated by Algorithm~\\ref{alg:stoc_prox}. We will use the shorthand $\\EE_t$ to denote the expectation conditioned on $\\xi_0,\\ldots, \\xi_{t-1}$.  We begin with a key one-step improvement lemma. \n","units":[{"start":31301,"end":31989,"start_line":910,"end_line":953,"raw_tex":"Throughout the work, \nwe \nconsider \nthe Euclidean space $\\mathbb{R}^d$, \nwhich we \nequip \nwith \na fixed inner product $\\langle\\cdot,\\cdot\\rangle$ \nand \nits induced norm $\\|\\cdot\\|=\\sqrt{\\dotp{\\cdot,\\cdot}}$.  \nWe \nlet \nthe symbol $\\BB(x,r)$ \ndenote \nthe closed ball \nof \nradius $r>0$ \naround \na point $x$.  \nWe \ndefine  \nthe \ndistance \nand \nthe nearest-point projection \nof \na point $y\\in\\R^d$ \nonto \na set $\\cX\\subset\\R^d$ \nby\n\\[\n\\dist(y,\\cX):=\\inf_{x\\in \\cX}~\\|x-y\\|\\qquad \\textrm{and}\\qquad\\proj_{\\cX}(y):=\\argmin_{x\\in \\cX}~\\|x-y\\|,\n\\]\nrespectively. \nFor any real number $r>0$, \nwe\ndefine \nthe {\\em $r$-tube} \naround \n$\\cX$ \nto be the set\n$\nU_{\\cX}(r):=\\{y\\in\\R^d:\\dist(y,\\cX)<r\\}.\n$\n","sha256_utf8":"78da2b5a82239ece44140ff2241d69aad4dce0c3d23931ba26d91ec76c6b26e3"},{"start":32251,"end":32930,"start_line":962,"end_line":968,"raw_tex":"Consider a function $f\\colon\\R^d\\to\\R\\cup\\{\\infty\\}$ and a point $\\bar x$, with $f(\\bar x)$ finite. \nThe {\\em subdifferential} of $f$ at $\\bar x$, denoted by $\\partial f(\\bar x)$, consists of all vectors $v\\in\\R^d$ satisfying\n$$f(x)\\geq f(\\bar x)+\\langle v,x-\\bar x\\rangle+o(\\|x-\\bar x\\|)\\qquad \\textrm{as } x\\to \\bar x.$$ \nWe will often use the following basic variational construction. The \\textit{normal cone} to a set $\\cX$ at a point $\\bar x\\in \\cX$, denoted $N_{\\cX}(\\bar x)$, consists of all vectors $v\\in\\R^d$ satisfying \n\\begin{equation}\\label{eqn:frech_norm}\n\\langle v,x-\\bar x\\rangle\\leq o(\\|x-\\bar x\\|) \\qquad \\textrm{as }x\\to \\bar x \\textrm{ in }\\cX.\n\\end{equation}\n","sha256_utf8":"8f122061e6f869bc86e0a93ea149fa940bd487894501cade070513d77d93610f"},{"start":33245,"end":33886,"start_line":985,"end_line":986,"raw_tex":"First, we say that a function $f\\colon\\R^d\\to\\R\\cup\\{\\infty\\}$ is {\\em $\\rho$-weakly convex}, for some $\\rho>0$, if the perturbed function $x\\mapsto f(x)+\\frac{\\rho}{2}\\|x\\|^2$ is convex. Second, following \\cite{clarkelower}, we say that a closed set $\\cX$ is {\\em $R$-proximally smooth} if the projection $\\proj_{\\cX}(x)$ is a singleton whenever $\\dist(x,\\cX)<R$. Although these conditions appear at first unrelated, they are deeply connected in the sense that weak convexity of a function $f$ is essentially equivalent to proximal smoothness of its epigraph. For a precise statement, we refer the reader to \n\\cite[Section 5]{clarkelower}.\n","sha256_utf8":"05a0216165bdbd0c8dbd9f92316781182c78a435a844d0d4f0e2ca25063ec5f4"},{"start":34129,"end":34378,"start_line":1001,"end_line":1008,"raw_tex":"For example, \nif $f$ \nis \na $\\rho$-weakly convex function, \nthen the inequality\n\\begin{equation}\\label{eqn:subgrad_ineq}\nf(y)\\ge f(x)+\\dotp{v,y-x}-\\frac{\\rho}{2}\\|y-x\\|^2,\\qquad \\textrm{holds for all }x,y \\in \\R^d,~v\\in \\partial f(x).\n\\end{equation}","sha256_utf8":"6310115899de1eb4877038ec845310944e60ee8cd6ca01329121fd55017346b5"},{"start":40237,"end":40560,"start_line":1101,"end_line":1105,"raw_tex":"Henceforth, we consider the optimization problem\n\\begin{equation}\\label{eqn:target_prob}\n\\min ~ f(x)\\quad \\textrm{subject to}\\quad x\\in \\cX,\n\\end{equation}\nfor some closed function $f\\colon\\R^d\\to\\R$ that is bounded from below on a closed set $\\cX\\subseteq\\R^d$. We will place further assumptions on $f$ and $\\cX$ shortly.\n","sha256_utf8":"58fbc53caca475050d03bf1f54bb2a85214d00236f08c80322f08cc965d31195"},{"start":40738,"end":41125,"start_line":1117,"end_line":1124,"raw_tex":"Namely, \ndefine \nthe {\\em Moreau envelope} \nand \nthe {\\em proximal map}, respectively, by\n$$\\cM_{\\lambda}(x)= \\min_{y\\in \\cX}~ \\left\\{f(y)+\\frac{1}{2\\lambda}\\|y-x\\|^2\\right\\},\\qquad\\cP_{\\lambda}(x):=\\argmin_{y\\in \\cX}~ \\left\\{f(y)+\\frac{1}{2\\lambda}\\|y-x\\|^2\\right\\},$$\nand define the {\\em stationarity measure}:\n$$\\mathcal{C}_{\\lambda}(x):=\\lambda^{-1}\\cdot\\dist(x,\\cP_{\\lambda}(x)).$$\n","sha256_utf8":"62f087e46672815cb7c8d7194dcac66a643b28f08f44f343863bf3ebf9c4c3cd"},{"start":52788,"end":53725,"start_line":1423,"end_line":1438,"raw_tex":"\\begin{definition}[Set-approximation]\n\t{\\rm Consider a set $\\cX\\subset\\R^d$ and a collection of sets $\\cX_x\\subset\\R^d$ and functions $\\cR_x\\colon\\cX_x\\to\\cX$, indexed by points $x\\in \\R^d$. We say that the collection $\\{(\\cX_x,\\cR_{x})\\}_{x\\in\\cX}$ is a {\\em set approximation of $\\cX$ with parameters $(R,\\accD,r_1,\\accR ,r_2)$}  if the following conditions hold:\n\t\t\\begin{enumerate}\n\t\t\t\\item [$(i)$] {\\bf (Accuracy)} For every $x \\in \\cX$, the set $\\cX_x$ is $R$-proximally smooth and satisfies\n\t\t\t\\begin{equation}\\label{eqn:need_this_eqn}\n\t\t\t\\dist(y, \\cX_{x})  \\leq \\frac{\\accD}{2} \\| x- y\\|^2 \\qquad \\forall y \\in \\cX\\cap \\BB(x,r_1).\n\t\t\t\\end{equation}\n\t\t\t%\t\t\t\\textcolor{red}{Why need this???}\n\t\t\t\n\t\t\t\\item[$(ii)$] {\\bf (Retraction)} For every $x \\in \\cX$, the mapping $\\cR_x$ satisfies\n\t\t\t$$\n\t\t\t\\|y - \\cR_x(y) \\| \\leq \\frac{\\accR }{2} \\|x-y\\|^2 \\qquad \\forall y \\in \\cX_x \\cap \\BB(x,r_2).\n\t\t\t$$\n\t\t\\end{enumerate}\n\t}\n\\end{definition}","sha256_utf8":"406dcd7a077005c845ab16998666aebe8a8d53bff47ed2bf26082be289f05028"},{"start":80260,"end":81163,"start_line":2038,"end_line":2077,"raw_tex":"\tNamely, \n\tthe algorithm we propose (Algorithm~\\ref{alg:stoc_prox}) \n\twill use \n\tthree building blocks: \n\ta family \n\tof \n\tstochastic models $f_{x}(\\cdot,\\xi)$ \n\tof \n\tthe objective function, \n\ta family \n\tof \n\tproximally smooth sets $\\cX_x$ \n\tthat \n\tapproximate $\\cX$ near $x$, \n\tand \n\ta retraction operation $\\cR_x\\colon\\cX_x\\to\\cX$ \n\tthat \n\trestores feasibility. \n\tThus in each iteration $t$, \n\tAlgorithm~\\ref{alg:stoc_prox} \n\tforms \n\tboth \n\ta proximally smooth approximation $\\cX_{x_t}$ \n\tof \n\tthe original constraint \n\tand  \n\ta stochastic model $f_{x_t}(\\cdot,\\xi_t)$ \n\tof \n\tthe objective function, \n\tcentered at the current iterate $x_t$. \n\tThe procedure then \n\tcomputes \n\ta minimizer ${\\tilde x}_t$ \n\tof \n\tthe function  $f_{x_t}(\\cdot,\\xi_t)+\\frac{\\beta}{2}\\|\\cdot-x_t\\|^2$ \n\tover\n\t$\\cX_{x_t}$ \n\tand \n\tretracts \n\tit back to $\\cX$, thereby defining the next iterate $x_{t+1}=\\cR_{x_t}(\\tilde x_t)$. \n","sha256_utf8":"48a3a6260d84ab9f395c94e6099441c6a288f81caf0ccc281af1b714f6fe2fa4"},{"start":81181,"end":81771,"start_line":2082,"end_line":2096,"raw_tex":"\\begin{algorithm}[H]\n\t\t{\\bf Input:} initialization $x_0\\in \\R^d$,   a sequence $\\beta_t>0$,  and an iteration count $T\\in \\mathbb{N}$.\n\t\t\\break{\\bf Step } $t=0,\\ldots,T$:\t\n\t\t\\begin{equation*}\n\t\t\\begin{aligned}\n\t\t&\\textrm{Sample } \\xi_t \\sim P\\\\\n\t\t& \\textrm{Choose } \\tilde x_{t} \\in \\argmin_{x \\in \\cX_x}~ \\left\\{f_{x_t}(x,\\xi_t) + \\frac{\\beta_t}{2} \\|x - x_t\\|^2\\right\\}\\\\\n\t\t& \\textrm{Set }  x_{t+1} = \\cR_{x_t}(\\tilde x_t)\n\t\t\\end{aligned}\n\t\t\\end{equation*}\n\t\t\\caption{Retracted Stochastic Model Based Algorithm\n\t\t\t%: PSSM($x_0,T,\\{\\alpha_t\\}$)\n\t\t}\n\t\t\\label{alg:stoc_prox}\n\t\\end{algorithm}","sha256_utf8":"f419b15c7236dc29132f169420008caa529c40462109c48e1aa275f304233d6b"},{"start":82023,"end":83348,"start_line":2104,"end_line":2118,"raw_tex":"\\begin{assumption}\n\t\t{\\rm\n\t\t\tFix a probability space $(\\Omega,\\mathcal{F},P)$ and equip $\\R^d$ with the Borel $\\sigma$-algebra. \n\t\t\tWe assume that there exist real $\\eta,\\osa, L, R, \\accD, \\accR, r_1, r_2 \\in \\R$ satisfying the following properties. \n\t\t\t\\begin{enumerate}\n\t\t\t\t\\item[(B1)] {\\bf (Sampling)} It is possible to generate i.i.d.\\ realizations $\\xi_1,\\xi_2, \\ldots \\sim P$.\n\t\t\t\t\\item[(B2)]\\label{it:B2} {\\bf (One-sided accuracy)} There is an open set $U$ containing $\\cX $ and a measurable function $(x,y,\\xi)\\mapsto f_x(y,\\xi)$, defined on $U\\times U\\times\\Omega$, satisfying  $$\\EE_{\\xi}\\left[f_x(x,\\xi)\\right]=f(x)\n\t\t\t\t\\quad \\textrm{and} \n\t\t\t\t\\quad\\EE_{\\xi}\\left[f_x(y,\\xi)-f(y)\\right]\n\t\t\t\t\\leq\\frac{\\osa}{2}\\|y-x\\|^2\\qquad \\forall x,y\\in U.$$\n\t\t\t\t\\item[(B3)] {\\bf (Weak-convexity)} The function $f_x(\\cdot,\\xi)$ is $\\eta$-weakly convex $\\forall x\\in U$, a.e. $\\xi\\sim P$.\n\t\t\t\t\\item [(B4)] {\\bf (Set approximation)} There exists a collection $\\{(\\cX_x,\\cR_{x})\\}_{x\\in\\cX}$ that is a set approximation of $\\cX$ with parameters $(R,\\accD,r_1,\\accR ,r_2)$.\n\t\t\t\t\\item[(B6)] {\\bf (Lipschitz property)} $f$ is $L$-Lipschitz continuous on $\\cX$; and for all $x\\in \\cX$ and a.e. $\\xi\\sim P$, the function $f_x(\\cdot, \\xi)$ is $L$-Lipschitz continuous on some neighborhood of $\\cX_x$.\n\t\t\\end{enumerate}}\n\t\\end{assumption}","sha256_utf8":"0ac1c33ab80af57a5ba34b88fef5d51e1f222a51af5ab4e579aee7f493e68290"},{"start":85440,"end":85745,"start_line":2155,"end_line":2155,"raw_tex":"\tIn what follows, we let   $x_0,\\ldots, x_t$ and $\\xi_0,\\ldots, \\xi_{t}$ be the iterates and random elements generated by Algorithm~\\ref{alg:stoc_prox}. We will use the shorthand $\\EE_t$ to denote the expectation conditioned on $\\xi_0,\\ldots, \\xi_{t-1}$.  We begin with a key one-step improvement lemma. \n","sha256_utf8":"1dbf56a1b9fe0dc33234257cb8e4e011e6f5963d110d9a494002fb83896bfbe8"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}[Convergence guarantees]\\label{thm:gen_conv_guarant}\n\t\tWithout loss of generality, suppose $r_1<R$ and define the constants $\\nu := \\frac{R}{2(R-r_1)^2} $ and $\\gamma:=\\eta+3L\\nu$. Fix a real $\\bar{\\rho} > \\max\\{\\frac{2L}{r_1}, \\gamma+\\osa+3\\accD L\\}$ and a sequence $\\beta_t > \\max\\{\\frac{2L}{r_2},\\gamma\\}$. Let $\\{x_t\\}_{t=0}^T$ be the iterates generated by  Algorithm~\\ref{alg:stoc_prox} and let  $t^*\\in \\{0,\\ldots,T\\}$ be sampled according to the discrete probability distribution\n\t\t$\\mathbb{P}(t^*=t)\\propto\\frac{1}{\\beta_t - \\eta-\\frac{3LR}{2(R-r_1)^2}}$.\n\t\tThen $x_{t^*}$ satisfies the estimate:\t\n\t\t\\begin{align*}\n\t\t\\EE_t\\left[\\cC_{1/\\bar \\rho}(x_{t^\\ast})^2\\right] \\leq \\frac{2\\bar\\rho(f(x_0) - \\min_{\\cX} f) + \\bar\\rho^2L^2\\sum_{t=0}^T a_t}{\\sum_{t=0}^T \\frac{\\bar \\rho - \\gamma - \\osa -  3\\accD L}{\\beta_t - \\gamma }},\n\t\t\\end{align*}\n\t\twhere we define\n\t\t$$\n\t\ta_t :=  \\frac{1}{\\beta_t(\\beta_t - \\gamma)} + \\frac{8\\accR L(\\bar \\rho^{-1} + \\beta_t^{-1})}{\\beta_t^2} + \\frac{4\\accR ^2 L^2}{\\beta_t^4}.\n\t\t$$\n\t\tIn particular, if we set $\\beta_t=\\gamma+\\frac{\\sqrt{T+1}}{\\alpha}$ for some positive  $\\alpha < \\frac{r_2}{2L-\\gamma r_2}$, then it holds:\n\t\t\\begin{align*}\n\t\t\\EE_t\\left[\\cC_{1/\\bar \\rho}(x_{t^\\ast})^2\\right] \\leq & \\frac{2\\bar\\rho(f(x_0) - \\min_{\\cX} f)}{\\alpha(\\bar \\rho - \\gamma - \\osa -  3\\accD L)\\sqrt{T+1}}\\\\\n\t\t&\\quad+\\frac{\\alpha\\bar \\rho^2L^2}{\\bar \\rho - \\gamma - \\osa -  3\\accD L}\\left[\\frac{1+\\accR L(\\bar \\rho^{-1}+\\frac{\\alpha}{\\sqrt{T+1}})}{\\sqrt{T+1}}+\\frac{4\\alpha^2\\accR ^2L^2}{(T+1)^{3/2}}\\right].\n\t\t\\end{align*}\n\t\t\n\t\\end{theorem}","units":[{"start":83494,"end":85074,"start_line":2129,"end_line":2146,"raw_tex":"\\begin{theorem}[Convergence guarantees]\\label{thm:gen_conv_guarant}\n\t\tWithout loss of generality, suppose $r_1<R$ and define the constants $\\nu := \\frac{R}{2(R-r_1)^2} $ and $\\gamma:=\\eta+3L\\nu$. Fix a real $\\bar{\\rho} > \\max\\{\\frac{2L}{r_1}, \\gamma+\\osa+3\\accD L\\}$ and a sequence $\\beta_t > \\max\\{\\frac{2L}{r_2},\\gamma\\}$. Let $\\{x_t\\}_{t=0}^T$ be the iterates generated by  Algorithm~\\ref{alg:stoc_prox} and let  $t^*\\in \\{0,\\ldots,T\\}$ be sampled according to the discrete probability distribution\n\t\t$\\mathbb{P}(t^*=t)\\propto\\frac{1}{\\beta_t - \\eta-\\frac{3LR}{2(R-r_1)^2}}$.\n\t\tThen $x_{t^*}$ satisfies the estimate:\t\n\t\t\\begin{align*}\n\t\t\\EE_t\\left[\\cC_{1/\\bar \\rho}(x_{t^\\ast})^2\\right] \\leq \\frac{2\\bar\\rho(f(x_0) - \\min_{\\cX} f) + \\bar\\rho^2L^2\\sum_{t=0}^T a_t}{\\sum_{t=0}^T \\frac{\\bar \\rho - \\gamma - \\osa -  3\\accD L}{\\beta_t - \\gamma }},\n\t\t\\end{align*}\n\t\twhere we define\n\t\t$$\n\t\ta_t :=  \\frac{1}{\\beta_t(\\beta_t - \\gamma)} + \\frac{8\\accR L(\\bar \\rho^{-1} + \\beta_t^{-1})}{\\beta_t^2} + \\frac{4\\accR ^2 L^2}{\\beta_t^4}.\n\t\t$$\n\t\tIn particular, if we set $\\beta_t=\\gamma+\\frac{\\sqrt{T+1}}{\\alpha}$ for some positive  $\\alpha < \\frac{r_2}{2L-\\gamma r_2}$, then it holds:\n\t\t\\begin{align*}\n\t\t\\EE_t\\left[\\cC_{1/\\bar \\rho}(x_{t^\\ast})^2\\right] \\leq & \\frac{2\\bar\\rho(f(x_0) - \\min_{\\cX} f)}{\\alpha(\\bar \\rho - \\gamma - \\osa -  3\\accD L)\\sqrt{T+1}}\\\\\n\t\t&\\quad+\\frac{\\alpha\\bar \\rho^2L^2}{\\bar \\rho - \\gamma - \\osa -  3\\accD L}\\left[\\frac{1+\\accR L(\\bar \\rho^{-1}+\\frac{\\alpha}{\\sqrt{T+1}})}{\\sqrt{T+1}}+\\frac{4\\alpha^2\\accR ^2L^2}{(T+1)^{3/2}}\\right].\n\t\t\\end{align*}\n\t\t\n\t\\end{theorem}","sha256_utf8":"159269083289ed02d909e0de75d2b4de72bcd5ed30130812f6800b922900ebea"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}[Proximally smooth sets]\\label{lem_basic_prop_man}\n\tConsider an $R$-proximally smooth set $\\cX$. The following are true.\n\t\\begin{enumerate}\n\t\t\\item {\\bf (Lipschitz projector)} For any real $r\\in (0, R)$ the estimate holds:\n\t\t$$\\|\\proj_{\\cX}(x)-\\proj_{\\cX}(y)\\|\\le \\frac{R}{R-r}\\|x-y\\|\\qquad \\forall x,y\\in U_{\\cX}(r).$$\n\t\t%\t\t\\item {\\bf (Projections \\& normals)} For any points $x\\in\\cX$, $y\\in U_{\\cX}(R)$, the equivalence holds:\n\t\t%\t\t$$x=\\proj_{\\cX}(y)\\qquad\\Longleftrightarrow\\qquad y-x\\in N_{\\cX}(x).$$ \n\t\t\\item\\label{it3:lem} {\\bf (Uniform normal inequality)} For any  point $x\\in \\cX$ and a normal $v\\in N_{\\cX}(x)$,  the inequality\n\t\t\\begin{equation}\\label{ineq:proximal_normal}\n\t\t\\left\\langle v,y-x\\right\\rangle\\le \\frac{\\|v\\|}{2R}\\cdot \\|y-x\\|^2 \\qquad\\textrm{holds for all } y\\in \\cX.\n\t\t\\end{equation}\n\t\\end{enumerate}\n\\end{lemma}\n\n\\begin{lemma}\\label{lem:prox_point_bound}\n\tConsider a function $g\\colon \\RR^d \\to \\RR$ that is $L$-Lipschitz continuous on a set $\\cX\\subseteq \\RR^d$. Then for any point $x \\in \\cX$,  every point $\\displaystyle \\hat x \\in \\argmin_{y \\in \\cX} \\left\\{g(y) + \\frac{\\beta}{2} \\|x - y\\|^2\\right\\}$  satisfies the estimate $\n\t\\|\\beta(x - \\hat x)\\| \\leq 2L.\n\t$\n\\end{lemma}\n\n\\begin{lemma}[One-step improvement]\\label{lem:stoc_prox_descent}\n\t\tFix an index $t$ and choose  $\\hat x_t\\in \\cP_{1/\\bar \\rho}(x)$. Then the estimate holds:\n\t\t\\begin{equation}\n\t\t\\EE_t\\left[\\|\\hat x_t - x_{t+1}\\|^2\\right] \\leq  \\frac{\\beta +\\osa+ 3\\accD L - \\bar \\rho }{\\beta- \\gamma}\\|x_t - {\\hat x}_t \\|^2 + L^2 a_t.\\label{ineq:stoc_prox_descent}\n\t\t\\end{equation}\n\t\t%\t\\end{align}\n\t\\end{lemma}\n\n\\begin{proof}\n\t\tSince the statement of the theorem is independent of $x_0,\\ldots, x_{t-1}$, we will simplify the notation by dropping the index $t$ in $x_t$, ${\\tilde x}_t$, ${\\hat x}_t$, $\\xi_t$, $\\EE_t$, $a_t$ and setting $x^+:=x_{t+1}$.\n\t\tBy the definition of $\\tilde x$, there exist vectors $v \\in \\partial f_{x}(\\tilde x, \\xi)$ and $w \\in N_{\\cX_{x}}(\\tilde x)$ satisfying  $\\beta(x - \\tilde x) = v + w.$\n\t\tUsing Lemma~\\ref{lem:prox_point_bound} and Lipschitz continuity of $f_x(\\cdot,\\xi)$ yields the estimate $\\|w\\| \\leq 3L$. Fix any  $y \\in \\cX\\cap\\BB(x,r_1)$. We then deduce\n\t\t\\begin{align}\n\t\t\\dotp{w, y- \\tilde x} &= \\dotp{w, (y - \\proj_{\\cX_x}(y))} + \\dotp{w, \\proj_{\\cX_x}(y) - \\tilde x}\\notag\\\\\n\t\t&\\leq \\|w\\|\\cdot \\dist(y, \\cX_x) + \\frac{\\|w\\|}{2R} \\cdot\\|\\proj_{\\cX_x}(y) - \\tilde x\\|^2\\label{eqn:part2it1}\\\\\n\t\t&\\leq \\frac{\\|w\\|\\accD}{2} \\|y - x\\|^2 + \\frac{\\|w\\| \\nu}{2} \\|y - \\tilde x\\|^2,\\label{eqn:part2it2}\n\t\t\\end{align}\n\t\twhere \\eqref{eqn:part2it1} follows from the Cauchy-Schwarz inequality and from Lemma~\\ref{lem_basic_prop_man}, while \\eqref{eqn:part2it2} follows from (B4) and Lemma~\\ref{lem_basic_prop_man}.\n\t\tAdding this estimate to the subgradient inequality for $f_x(\\cdot,\\xi)$ and completing the square, we conclude  for all $y \\in \\cX\\cap\\BB(x,r_1)$ the  bound:\n\t\t\\begin{align}\n\t\tf_{x}(y, \\xi) &\\geq f_{x}(\\tilde x, \\xi) + \\dotp{v + w, y- \\tilde x} - \\frac{\\eta + \\|w\\| \\nu}{2} \\|y - \\tilde x\\|^2 - \\frac{\\|w\\|\\accD}{2}\\|y - x\\|^2\\notag\\\\\n\t\t&\\geq f_{x}(\\tilde x, \\xi) + \\dotp{\\beta(x - \\tilde x) , y- \\tilde x} - \\frac{\\eta + \\|w\\| \\nu}{2} \\|y - \\tilde x\\|^2  - \\frac{\\|w\\|\\accD}{2}\\|y - x\\|^2 \\notag\\\\\n\t\t&= f_{x}(\\tilde x, \\xi) + \\frac{\\beta}{2}\\left[ \\|x - \\tilde x\\|^2 + \\| y - \\tilde x\\|^2 - \\|x - y\\|^2\\right]\\label{eqn:perturb_three_point} \\\\\n\t\t&\\qquad- \\frac{\\eta + \\|w\\| \\nu}{2} \\|y - \\tilde x\\|^2  - \\frac{\\|w\\|\\accD}{2}\\|y - x\\|^2.\\notag\n\t\t\\end{align}\n\t\tLemma~\\ref{lem:prox_point_bound} implies\n\t\t$\n\t\t\\|\\hat x - x\\| \\leq \\frac{2L}{\\bar \\rho} \\leq r_1.\n\t\t$\n\t\tHence, setting $y = \\hat x$ in \\eqref{eqn:perturb_three_point} yields: \n\t\t\\begin{align}\n\t\t&\\EE\\left[\\frac{\\beta - \\gamma}{2} \\|\\hat x - \\tilde x\\|^2 + \\frac{\\beta}{2}\\| x - \\tilde x\\|^2 - \\frac{\\beta + \\|w\\|\\accD }{2}\\|x - \\hat x\\|^2\\right]\\notag\\\\\n\t\t&\\leq \\EE\\left[f_x(\\hat x, \\xi) - f_x(\\tilde x, \\xi)\\right]\\notag \\\\\n\t\t&\\leq \\EE\\left[f_x(\\hat x, \\xi) - f_x(x,\\xi)\\right]  + L \\EE\\left[\\|x - \\tilde x\\|\\right]\\label{eqn:blah1}\\\\\t\n\t\t&\\leq f(\\hat x) - f(x)  + L \\EE\\left[\\|x - \\tilde x\\|\\right] + \\frac{\\osa}{2}\\|x - \\hat x \\|^2\\label{eqn:blah2}\\\\\n\t\t&\\leq \\frac{\\osa - \\bar \\rho }{2}\\|x - \\hat x \\|^2 + L\\sqrt{\\EE\\left[ \\|x - \\tilde x\\|^2\\right]},\\label{eqn:blah3}\n\t\t%&\\leq \\frac{\\tau - \\bar \\rho }{2}\\|x - \\hat x \\|^2 + \\frac{2L^2}{\\beta}.\n\t\t\\end{align}\n\t\twhere \\eqref{eqn:blah1} uses (B6), the estimate \\eqref{eqn:blah2} uses (B2), and \\eqref{eqn:blah3} uses the definition of $\\hat x$ as the proximal point.\n\t\tRearranging and setting $\\delta :=\\sqrt{ \\EE\\left[\\|x - \\tilde x\\|^2\\right]}$ yields the bound:\n\t\t\\begin{equation}\\label{eqn:intermed}\n\t\t\\begin{aligned}\n\t\t\\frac{\\beta - \\gamma}{2}\\EE\\left[\\|\\hat x - \\tilde x\\|^2\\right] &\\leq \\frac{\\beta + \\accD\\|w\\|+ \\osa - \\bar \\rho  }{2}\\|x - \\hat x \\|^2 + L \\delta - \\frac{\\beta }{2}\\delta^2 \\\\\n\t\t&\\leq \\frac{\\beta + \\accD\\|w\\|+ \\osa - \\bar \\rho  }{2}\\|x - \\hat x \\|^2 + \\frac{L^2}{2\\beta},\n\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\twhere the last inequality follows by maximizing the expression  $L\\delta - \\frac{\\beta }{2}\\delta^2$ over $\\delta\\in \\R$.\n\t\tTo complete the proof, we will use the following estimate.\n\t\t\\begin{claim}\\label{claim1} The estimate holds:\n\t\t\t$\n\t\t\t\\|\\hat x -  x_+\\|^2 \\leq \\|\\hat x - \\tilde x\\|^2 + \\frac{8\\accR  L^3}{\\beta^2}(\\bar \\rho^{-1} + \\beta^{-1}) + \\frac{4\\accR ^2 L^4}{\\beta^4}\n\t\t\t$.\n\t\t\\end{claim}\n\t\t\\begin{proof}\n\t\t\tUsing Lemma~\\ref{lem:prox_point_bound}, we deduce\n\t\t\t$\\|\\tilde x - x\\| \\leq 2L/\\beta  \\leq r_2$.\n\t\t\tIn addition, we have $$\\|\\tilde x - x_+ \\| = \\|\\tilde x - \\cR_x(\\tilde x)\\| \\leq \\frac{\\accR }{2}\\|\\tilde x - x\\|^2\\leq \\frac{2\\accR  L^2}{\\beta^2}.$$ \n\t\t\tUsing the triangle inequality, we  deduce \n\t\t\t$\n\t\t\t\\|\\hat x - x_+\\|  \\leq \\|\\hat x - \\tilde x \\| + \\frac{2\\accR  L^2}{\\beta^2}.\n\t\t\t$\n\t\t\tSquaring both sides, we get\n\t\t\t$$\n\t\t\t\\|\\hat x - x_+\\|^2  \\leq \\|\\hat x - \\tilde x \\|^2 +  \\frac{4\\accR  L^2}{\\beta^2}\\|\\hat x - \\tilde x \\| + \\frac{4\\accR ^2 L^4}{\\beta^4}.\n\t\t\t$$\n\t\t\tTaking into account the estimate $\n\t\t\t\\|\\hat x - \\tilde x \\| \\leq \\|\\hat x - x \\| + \\|x - \\tilde x\\| \\leq 2L/\\bar \\rho + 2L/\\beta,\n\t\t\t$\n\t\t\tcompletes the proof.\n\t\t\\end{proof}\n\t\tCombining Claim~\\ref{claim1} with the estimate \\eqref{eqn:intermed}, we compute\n\t\t\\begin{align*}\n\t\t\\EE\\left[\\|\\hat x -  x_+\\|^2\\right] &\\leq \\EE\\left[\\|\\hat x - \\tilde x\\|^2\\right] + \\frac{8\\accR  L^3(\\bar \\rho^{-1} + \\beta^{-1})}{\\beta^2} + \\frac{4\\accR ^2 L^4}{\\beta^4} \\\\\n\t\t&\\leq \\frac{\\beta + \\accD\\|w\\|+ \\osa - \\bar \\rho }{\\beta- \\gamma}\\|x - \\hat x \\|^2 + L^2 a\n\t\t%\\frac{L^2}{\\beta(\\beta - \\gamma)} + \\frac{8\\accR  L^3(\\bar \\rho^{-1} + \\beta^{-1})}{\\beta^2} + \\frac{4\\accR ^2 L^4}{\\beta^4}\n\t\t%\t&=  \\|x - \\hat x \\|^2  - \\frac{\\bar \\rho - \\gamma - \\osa  -  \\accD\\|w\\|}{\\beta - \\gamma}\\|x - \\hat x \\|^2  + \\frac{L^2}{\\beta(\\beta - \\gamma)} + \\frac{8\\accR  L^3(\\bar \\rho^{-1} + \\beta^{-1})}{\\beta^2} + \\frac{4\\accR ^2 L^4}{\\beta^4}.\n\t\t\\end{align*}\n\t\tBounding $\\|w \\| \\leq 3L$ completes the proof of \\eqref{ineq:stoc_prox_descent}.\n\t\\end{proof}\n\n\\begin{proof}[Proof of Theorem~\\ref{thm:gen_conv_guarant}]\n\t\tFix an iteration $t$ and a  point  $\\hat x_t\\in\\cP_{1/\\bar \\rho}(x_t)$. Then\n\t\tusing the definition of the Moreau envelope and appealing to  Lemma~\\ref{lem:stoc_prox_descent}, we deduce  \n\t\t\\begin{align*}\n\t\t\\EE_t[ \\cM_{1/\\bar \\rho}(x_{t+1})] &\\leq  \\EE_t\\left[f( \\hat x_{t})  +\\frac{\\bar \\rho}{2}\\cdot \\| x_{t+1} - \\hat x_t \\|^2\\right],\\\\\n\t\t&\\leq f( \\hat x_{t}) +\\frac{\\bar \\rho}{2}\\left[\\|\\hat x_t-x_{t}\\|^2+\\left( \\tfrac{\\beta +\\osa+ 3\\accD L - \\bar \\rho }{\\beta- \\gamma}-1\\right)\\|\\hat x_t-x_{t}\\|^2  +L^2 a_t\\right]\\\\\n\t\t&= \\cM_{1/\\bar \\rho}(x_{t}) - \\frac{\\bar \\rho- \\gamma-\\osa-3\\accD L}{2\\bar \\rho(\\beta_t-\\gamma)}\\cC_{1/\\bar \\rho}(x_t)^2  + \\frac{\\bar\\rho L^2a_t}{2}.\n\t\t\\end{align*}\n\t\tTaking expectations, iterating the inequality, and using the tower rule  yields:\n\t\t$$\\sum_{t=0}^T \\frac{\\bar \\rho - \\gamma - \\osa-3\\accD L}{\\beta_t-\\gamma}\\EE[\\cC_{1/\\bar{\\rho}}(x_t)^2]\\leq 2\\bar{\\rho}(\\cM_{1/\\bar \\rho}(x_{0})-\\min \\cM_{1/\\bar\\rho})+  \\bar \\rho^2 L^2\\cdot\\sum_{t=0}^T a_t.$$\n\t\tDividing through by $\\sum_{t=0}^T \\frac{\\bar \\rho - \\gamma - \\osa-3\\accD L}{\\beta_t-\\gamma}$ and recognizing the left side as\n\t\t$\\EE[\\cC_{1/\\bar{\\rho}}(x_{t^*})^2]$ completes the proof.\n\t\\end{proof}","units":[{"start":34549,"end":35401,"start_line":1015,"end_line":1027,"raw_tex":"\\begin{lemma}[Proximally smooth sets]\\label{lem_basic_prop_man}\n\tConsider an $R$-proximally smooth set $\\cX$. The following are true.\n\t\\begin{enumerate}\n\t\t\\item {\\bf (Lipschitz projector)} For any real $r\\in (0, R)$ the estimate holds:\n\t\t$$\\|\\proj_{\\cX}(x)-\\proj_{\\cX}(y)\\|\\le \\frac{R}{R-r}\\|x-y\\|\\qquad \\forall x,y\\in U_{\\cX}(r).$$\n\t\t%\t\t\\item {\\bf (Projections \\& normals)} For any points $x\\in\\cX$, $y\\in U_{\\cX}(R)$, the equivalence holds:\n\t\t%\t\t$$x=\\proj_{\\cX}(y)\\qquad\\Longleftrightarrow\\qquad y-x\\in N_{\\cX}(x).$$ \n\t\t\\item\\label{it3:lem} {\\bf (Uniform normal inequality)} For any  point $x\\in \\cX$ and a normal $v\\in N_{\\cX}(x)$,  the inequality\n\t\t\\begin{equation}\\label{ineq:proximal_normal}\n\t\t\\left\\langle v,y-x\\right\\rangle\\le \\frac{\\|v\\|}{2R}\\cdot \\|y-x\\|^2 \\qquad\\textrm{holds for all } y\\in \\cX.\n\t\t\\end{equation}\n\t\\end{enumerate}\n\\end{lemma}","sha256_utf8":"a493692bc1e8023d137df18efc6fb087ddaa10ef485dd303252d3cb783e71094"},{"start":36588,"end":36953,"start_line":1036,"end_line":1040,"raw_tex":"\\begin{lemma}\\label{lem:prox_point_bound}\n\tConsider a function $g\\colon \\RR^d \\to \\RR$ that is $L$-Lipschitz continuous on a set $\\cX\\subseteq \\RR^d$. Then for any point $x \\in \\cX$,  every point $\\displaystyle \\hat x \\in \\argmin_{y \\in \\cX} \\left\\{g(y) + \\frac{\\beta}{2} \\|x - y\\|^2\\right\\}$  satisfies the estimate $\n\t\\|\\beta(x - \\hat x)\\| \\leq 2L.\n\t$\n\\end{lemma}","sha256_utf8":"e7c08308526ab029294f1b0c687264b2a43d14cc36160265b960a8ca64895087"},{"start":85959,"end":86352,"start_line":2162,"end_line":2168,"raw_tex":"\\begin{lemma}[One-step improvement]\\label{lem:stoc_prox_descent}\n\t\tFix an index $t$ and choose  $\\hat x_t\\in \\cP_{1/\\bar \\rho}(x)$. Then the estimate holds:\n\t\t\\begin{equation}\n\t\t\\EE_t\\left[\\|\\hat x_t - x_{t+1}\\|^2\\right] \\leq  \\frac{\\beta +\\osa+ 3\\accD L - \\bar \\rho }{\\beta- \\gamma}\\|x_t - {\\hat x}_t \\|^2 + L^2 a_t.\\label{ineq:stoc_prox_descent}\n\t\t\\end{equation}\n\t\t%\t\\end{align}\n\t\\end{lemma}","sha256_utf8":"193575bbf03bff390da0b5bc2edcb78795d2a8723389b8be5374e6a3b6c05208"},{"start":86354,"end":91744,"start_line":2169,"end_line":2239,"raw_tex":"\\begin{proof}\n\t\tSince the statement of the theorem is independent of $x_0,\\ldots, x_{t-1}$, we will simplify the notation by dropping the index $t$ in $x_t$, ${\\tilde x}_t$, ${\\hat x}_t$, $\\xi_t$, $\\EE_t$, $a_t$ and setting $x^+:=x_{t+1}$.\n\t\tBy the definition of $\\tilde x$, there exist vectors $v \\in \\partial f_{x}(\\tilde x, \\xi)$ and $w \\in N_{\\cX_{x}}(\\tilde x)$ satisfying  $\\beta(x - \\tilde x) = v + w.$\n\t\tUsing Lemma~\\ref{lem:prox_point_bound} and Lipschitz continuity of $f_x(\\cdot,\\xi)$ yields the estimate $\\|w\\| \\leq 3L$. Fix any  $y \\in \\cX\\cap\\BB(x,r_1)$. We then deduce\n\t\t\\begin{align}\n\t\t\\dotp{w, y- \\tilde x} &= \\dotp{w, (y - \\proj_{\\cX_x}(y))} + \\dotp{w, \\proj_{\\cX_x}(y) - \\tilde x}\\notag\\\\\n\t\t&\\leq \\|w\\|\\cdot \\dist(y, \\cX_x) + \\frac{\\|w\\|}{2R} \\cdot\\|\\proj_{\\cX_x}(y) - \\tilde x\\|^2\\label{eqn:part2it1}\\\\\n\t\t&\\leq \\frac{\\|w\\|\\accD}{2} \\|y - x\\|^2 + \\frac{\\|w\\| \\nu}{2} \\|y - \\tilde x\\|^2,\\label{eqn:part2it2}\n\t\t\\end{align}\n\t\twhere \\eqref{eqn:part2it1} follows from the Cauchy-Schwarz inequality and from Lemma~\\ref{lem_basic_prop_man}, while \\eqref{eqn:part2it2} follows from (B4) and Lemma~\\ref{lem_basic_prop_man}.\n\t\tAdding this estimate to the subgradient inequality for $f_x(\\cdot,\\xi)$ and completing the square, we conclude  for all $y \\in \\cX\\cap\\BB(x,r_1)$ the  bound:\n\t\t\\begin{align}\n\t\tf_{x}(y, \\xi) &\\geq f_{x}(\\tilde x, \\xi) + \\dotp{v + w, y- \\tilde x} - \\frac{\\eta + \\|w\\| \\nu}{2} \\|y - \\tilde x\\|^2 - \\frac{\\|w\\|\\accD}{2}\\|y - x\\|^2\\notag\\\\\n\t\t&\\geq f_{x}(\\tilde x, \\xi) + \\dotp{\\beta(x - \\tilde x) , y- \\tilde x} - \\frac{\\eta + \\|w\\| \\nu}{2} \\|y - \\tilde x\\|^2  - \\frac{\\|w\\|\\accD}{2}\\|y - x\\|^2 \\notag\\\\\n\t\t&= f_{x}(\\tilde x, \\xi) + \\frac{\\beta}{2}\\left[ \\|x - \\tilde x\\|^2 + \\| y - \\tilde x\\|^2 - \\|x - y\\|^2\\right]\\label{eqn:perturb_three_point} \\\\\n\t\t&\\qquad- \\frac{\\eta + \\|w\\| \\nu}{2} \\|y - \\tilde x\\|^2  - \\frac{\\|w\\|\\accD}{2}\\|y - x\\|^2.\\notag\n\t\t\\end{align}\n\t\tLemma~\\ref{lem:prox_point_bound} implies\n\t\t$\n\t\t\\|\\hat x - x\\| \\leq \\frac{2L}{\\bar \\rho} \\leq r_1.\n\t\t$\n\t\tHence, setting $y = \\hat x$ in \\eqref{eqn:perturb_three_point} yields: \n\t\t\\begin{align}\n\t\t&\\EE\\left[\\frac{\\beta - \\gamma}{2} \\|\\hat x - \\tilde x\\|^2 + \\frac{\\beta}{2}\\| x - \\tilde x\\|^2 - \\frac{\\beta + \\|w\\|\\accD }{2}\\|x - \\hat x\\|^2\\right]\\notag\\\\\n\t\t&\\leq \\EE\\left[f_x(\\hat x, \\xi) - f_x(\\tilde x, \\xi)\\right]\\notag \\\\\n\t\t&\\leq \\EE\\left[f_x(\\hat x, \\xi) - f_x(x,\\xi)\\right]  + L \\EE\\left[\\|x - \\tilde x\\|\\right]\\label{eqn:blah1}\\\\\t\n\t\t&\\leq f(\\hat x) - f(x)  + L \\EE\\left[\\|x - \\tilde x\\|\\right] + \\frac{\\osa}{2}\\|x - \\hat x \\|^2\\label{eqn:blah2}\\\\\n\t\t&\\leq \\frac{\\osa - \\bar \\rho }{2}\\|x - \\hat x \\|^2 + L\\sqrt{\\EE\\left[ \\|x - \\tilde x\\|^2\\right]},\\label{eqn:blah3}\n\t\t%&\\leq \\frac{\\tau - \\bar \\rho }{2}\\|x - \\hat x \\|^2 + \\frac{2L^2}{\\beta}.\n\t\t\\end{align}\n\t\twhere \\eqref{eqn:blah1} uses (B6), the estimate \\eqref{eqn:blah2} uses (B2), and \\eqref{eqn:blah3} uses the definition of $\\hat x$ as the proximal point.\n\t\tRearranging and setting $\\delta :=\\sqrt{ \\EE\\left[\\|x - \\tilde x\\|^2\\right]}$ yields the bound:\n\t\t\\begin{equation}\\label{eqn:intermed}\n\t\t\\begin{aligned}\n\t\t\\frac{\\beta - \\gamma}{2}\\EE\\left[\\|\\hat x - \\tilde x\\|^2\\right] &\\leq \\frac{\\beta + \\accD\\|w\\|+ \\osa - \\bar \\rho  }{2}\\|x - \\hat x \\|^2 + L \\delta - \\frac{\\beta }{2}\\delta^2 \\\\\n\t\t&\\leq \\frac{\\beta + \\accD\\|w\\|+ \\osa - \\bar \\rho  }{2}\\|x - \\hat x \\|^2 + \\frac{L^2}{2\\beta},\n\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\twhere the last inequality follows by maximizing the expression  $L\\delta - \\frac{\\beta }{2}\\delta^2$ over $\\delta\\in \\R$.\n\t\tTo complete the proof, we will use the following estimate.\n\t\t\\begin{claim}\\label{claim1} The estimate holds:\n\t\t\t$\n\t\t\t\\|\\hat x -  x_+\\|^2 \\leq \\|\\hat x - \\tilde x\\|^2 + \\frac{8\\accR  L^3}{\\beta^2}(\\bar \\rho^{-1} + \\beta^{-1}) + \\frac{4\\accR ^2 L^4}{\\beta^4}\n\t\t\t$.\n\t\t\\end{claim}\n\t\t\\begin{proof}\n\t\t\tUsing Lemma~\\ref{lem:prox_point_bound}, we deduce\n\t\t\t$\\|\\tilde x - x\\| \\leq 2L/\\beta  \\leq r_2$.\n\t\t\tIn addition, we have $$\\|\\tilde x - x_+ \\| = \\|\\tilde x - \\cR_x(\\tilde x)\\| \\leq \\frac{\\accR }{2}\\|\\tilde x - x\\|^2\\leq \\frac{2\\accR  L^2}{\\beta^2}.$$ \n\t\t\tUsing the triangle inequality, we  deduce \n\t\t\t$\n\t\t\t\\|\\hat x - x_+\\|  \\leq \\|\\hat x - \\tilde x \\| + \\frac{2\\accR  L^2}{\\beta^2}.\n\t\t\t$\n\t\t\tSquaring both sides, we get\n\t\t\t$$\n\t\t\t\\|\\hat x - x_+\\|^2  \\leq \\|\\hat x - \\tilde x \\|^2 +  \\frac{4\\accR  L^2}{\\beta^2}\\|\\hat x - \\tilde x \\| + \\frac{4\\accR ^2 L^4}{\\beta^4}.\n\t\t\t$$\n\t\t\tTaking into account the estimate $\n\t\t\t\\|\\hat x - \\tilde x \\| \\leq \\|\\hat x - x \\| + \\|x - \\tilde x\\| \\leq 2L/\\bar \\rho + 2L/\\beta,\n\t\t\t$\n\t\t\tcompletes the proof.\n\t\t\\end{proof}\n\t\tCombining Claim~\\ref{claim1} with the estimate \\eqref{eqn:intermed}, we compute\n\t\t\\begin{align*}\n\t\t\\EE\\left[\\|\\hat x -  x_+\\|^2\\right] &\\leq \\EE\\left[\\|\\hat x - \\tilde x\\|^2\\right] + \\frac{8\\accR  L^3(\\bar \\rho^{-1} + \\beta^{-1})}{\\beta^2} + \\frac{4\\accR ^2 L^4}{\\beta^4} \\\\\n\t\t&\\leq \\frac{\\beta + \\accD\\|w\\|+ \\osa - \\bar \\rho }{\\beta- \\gamma}\\|x - \\hat x \\|^2 + L^2 a\n\t\t%\\frac{L^2}{\\beta(\\beta - \\gamma)} + \\frac{8\\accR  L^3(\\bar \\rho^{-1} + \\beta^{-1})}{\\beta^2} + \\frac{4\\accR ^2 L^4}{\\beta^4}\n\t\t%\t&=  \\|x - \\hat x \\|^2  - \\frac{\\bar \\rho - \\gamma - \\osa  -  \\accD\\|w\\|}{\\beta - \\gamma}\\|x - \\hat x \\|^2  + \\frac{L^2}{\\beta(\\beta - \\gamma)} + \\frac{8\\accR  L^3(\\bar \\rho^{-1} + \\beta^{-1})}{\\beta^2} + \\frac{4\\accR ^2 L^4}{\\beta^4}.\n\t\t\\end{align*}\n\t\tBounding $\\|w \\| \\leq 3L$ completes the proof of \\eqref{ineq:stoc_prox_descent}.\n\t\\end{proof}","sha256_utf8":"ac2c6a13278a5ced02eefe43b25cf6ddb35b9c3b3901d7e05957155fa5866f95"},{"start":91802,"end":93047,"start_line":2245,"end_line":2257,"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm:gen_conv_guarant}]\n\t\tFix an iteration $t$ and a  point  $\\hat x_t\\in\\cP_{1/\\bar \\rho}(x_t)$. Then\n\t\tusing the definition of the Moreau envelope and appealing to  Lemma~\\ref{lem:stoc_prox_descent}, we deduce  \n\t\t\\begin{align*}\n\t\t\\EE_t[ \\cM_{1/\\bar \\rho}(x_{t+1})] &\\leq  \\EE_t\\left[f( \\hat x_{t})  +\\frac{\\bar \\rho}{2}\\cdot \\| x_{t+1} - \\hat x_t \\|^2\\right],\\\\\n\t\t&\\leq f( \\hat x_{t}) +\\frac{\\bar \\rho}{2}\\left[\\|\\hat x_t-x_{t}\\|^2+\\left( \\tfrac{\\beta +\\osa+ 3\\accD L - \\bar \\rho }{\\beta- \\gamma}-1\\right)\\|\\hat x_t-x_{t}\\|^2  +L^2 a_t\\right]\\\\\n\t\t&= \\cM_{1/\\bar \\rho}(x_{t}) - \\frac{\\bar \\rho- \\gamma-\\osa-3\\accD L}{2\\bar \\rho(\\beta_t-\\gamma)}\\cC_{1/\\bar \\rho}(x_t)^2  + \\frac{\\bar\\rho L^2a_t}{2}.\n\t\t\\end{align*}\n\t\tTaking expectations, iterating the inequality, and using the tower rule  yields:\n\t\t$$\\sum_{t=0}^T \\frac{\\bar \\rho - \\gamma - \\osa-3\\accD L}{\\beta_t-\\gamma}\\EE[\\cC_{1/\\bar{\\rho}}(x_t)^2]\\leq 2\\bar{\\rho}(\\cM_{1/\\bar \\rho}(x_{0})-\\min \\cM_{1/\\bar\\rho})+  \\bar \\rho^2 L^2\\cdot\\sum_{t=0}^T a_t.$$\n\t\tDividing through by $\\sum_{t=0}^T \\frac{\\bar \\rho - \\gamma - \\osa-3\\accD L}{\\beta_t-\\gamma}$ and recognizing the left side as\n\t\t$\\EE[\\cC_{1/\\bar{\\rho}}(x_{t^*})^2]$ completes the proof.\n\t\\end{proof}","sha256_utf8":"433df90dcc64c2e6e922d7938a757bbd4bf7ac62e31fe0d43d8d00eda68ad929"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"约束优化问题","source_units":[{"start":40237,"end":40560,"start_line":1101,"end_line":1105,"raw_tex":"Henceforth, we consider the optimization problem\n\\begin{equation}\\label{eqn:target_prob}\n\\min ~ f(x)\\quad \\textrm{subject to}\\quad x\\in \\cX,\n\\end{equation}\nfor some closed function $f\\colon\\R^d\\to\\R$ that is bounded from below on a closed set $\\cX\\subseteq\\R^d$. We will place further assumptions on $f$ and $\\cX$ shortly.\n","sha256_utf8":"58fbc53caca475050d03bf1f54bb2a85214d00236f08c80322f08cc965d31195"}]},{"id":"S002","semantic_type":"notation","kind":"notation","title":"欧氏空间与投影记号","source_units":[{"start":31301,"end":31989,"start_line":910,"end_line":953,"raw_tex":"Throughout the work, \nwe \nconsider \nthe Euclidean space $\\mathbb{R}^d$, \nwhich we \nequip \nwith \na fixed inner product $\\langle\\cdot,\\cdot\\rangle$ \nand \nits induced norm $\\|\\cdot\\|=\\sqrt{\\dotp{\\cdot,\\cdot}}$.  \nWe \nlet \nthe symbol $\\BB(x,r)$ \ndenote \nthe closed ball \nof \nradius $r>0$ \naround \na point $x$.  \nWe \ndefine  \nthe \ndistance \nand \nthe nearest-point projection \nof \na point $y\\in\\R^d$ \nonto \na set $\\cX\\subset\\R^d$ \nby\n\\[\n\\dist(y,\\cX):=\\inf_{x\\in \\cX}~\\|x-y\\|\\qquad \\textrm{and}\\qquad\\proj_{\\cX}(y):=\\argmin_{x\\in \\cX}~\\|x-y\\|,\n\\]\nrespectively. \nFor any real number $r>0$, \nwe\ndefine \nthe {\\em $r$-tube} \naround \n$\\cX$ \nto be the set\n$\nU_{\\cX}(r):=\\{y\\in\\R^d:\\dist(y,\\cX)<r\\}.\n$\n","sha256_utf8":"78da2b5a82239ece44140ff2241d69aad4dce0c3d23931ba26d91ec76c6b26e3"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"次微分与法锥","source_units":[{"start":32251,"end":32930,"start_line":962,"end_line":968,"raw_tex":"Consider a function $f\\colon\\R^d\\to\\R\\cup\\{\\infty\\}$ and a point $\\bar x$, with $f(\\bar x)$ finite. \nThe {\\em subdifferential} of $f$ at $\\bar x$, denoted by $\\partial f(\\bar x)$, consists of all vectors $v\\in\\R^d$ satisfying\n$$f(x)\\geq f(\\bar x)+\\langle v,x-\\bar x\\rangle+o(\\|x-\\bar x\\|)\\qquad \\textrm{as } x\\to \\bar x.$$ \nWe will often use the following basic variational construction. The \\textit{normal cone} to a set $\\cX$ at a point $\\bar x\\in \\cX$, denoted $N_{\\cX}(\\bar x)$, consists of all vectors $v\\in\\R^d$ satisfying \n\\begin{equation}\\label{eqn:frech_norm}\n\\langle v,x-\\bar x\\rangle\\leq o(\\|x-\\bar x\\|) \\qquad \\textrm{as }x\\to \\bar x \\textrm{ in }\\cX.\n\\end{equation}\n","sha256_utf8":"8f122061e6f869bc86e0a93ea149fa940bd487894501cade070513d77d93610f"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"弱凸与近端光滑","source_units":[{"start":33245,"end":33886,"start_line":985,"end_line":986,"raw_tex":"First, we say that a function $f\\colon\\R^d\\to\\R\\cup\\{\\infty\\}$ is {\\em $\\rho$-weakly convex}, for some $\\rho>0$, if the perturbed function $x\\mapsto f(x)+\\frac{\\rho}{2}\\|x\\|^2$ is convex. Second, following \\cite{clarkelower}, we say that a closed set $\\cX$ is {\\em $R$-proximally smooth} if the projection $\\proj_{\\cX}(x)$ is a singleton whenever $\\dist(x,\\cX)<R$. Although these conditions appear at first unrelated, they are deeply connected in the sense that weak convexity of a function $f$ is essentially equivalent to proximal smoothness of its epigraph. For a precise statement, we refer the reader to \n\\cite[Section 5]{clarkelower}.\n","sha256_utf8":"05a0216165bdbd0c8dbd9f92316781182c78a435a844d0d4f0e2ca25063ec5f4"}]},{"id":"S005","semantic_type":"derived_fact","kind":"derived_fact","title":"弱凸次梯度不等式","source_units":[{"start":34129,"end":34378,"start_line":1001,"end_line":1008,"raw_tex":"For example, \nif $f$ \nis \na $\\rho$-weakly convex function, \nthen the inequality\n\\begin{equation}\\label{eqn:subgrad_ineq}\nf(y)\\ge f(x)+\\dotp{v,y-x}-\\frac{\\rho}{2}\\|y-x\\|^2,\\qquad \\textrm{holds for all }x,y \\in \\R^d,~v\\in \\partial f(x).\n\\end{equation}","sha256_utf8":"6310115899de1eb4877038ec845310944e60ee8cd6ca01329121fd55017346b5"}]},{"id":"S006","semantic_type":"lemma","kind":"lemma","title":"投影与法向统一估计","source_units":[{"start":34549,"end":35401,"start_line":1015,"end_line":1027,"raw_tex":"\\begin{lemma}[Proximally smooth sets]\\label{lem_basic_prop_man}\n\tConsider an $R$-proximally smooth set $\\cX$. The following are true.\n\t\\begin{enumerate}\n\t\t\\item {\\bf (Lipschitz projector)} For any real $r\\in (0, R)$ the estimate holds:\n\t\t$$\\|\\proj_{\\cX}(x)-\\proj_{\\cX}(y)\\|\\le \\frac{R}{R-r}\\|x-y\\|\\qquad \\forall x,y\\in U_{\\cX}(r).$$\n\t\t%\t\t\\item {\\bf (Projections \\& normals)} For any points $x\\in\\cX$, $y\\in U_{\\cX}(R)$, the equivalence holds:\n\t\t%\t\t$$x=\\proj_{\\cX}(y)\\qquad\\Longleftrightarrow\\qquad y-x\\in N_{\\cX}(x).$$ \n\t\t\\item\\label{it3:lem} {\\bf (Uniform normal inequality)} For any  point $x\\in \\cX$ and a normal $v\\in N_{\\cX}(x)$,  the inequality\n\t\t\\begin{equation}\\label{ineq:proximal_normal}\n\t\t\\left\\langle v,y-x\\right\\rangle\\le \\frac{\\|v\\|}{2R}\\cdot \\|y-x\\|^2 \\qquad\\textrm{holds for all } y\\in \\cX.\n\t\t\\end{equation}\n\t\\end{enumerate}\n\\end{lemma}","sha256_utf8":"a493692bc1e8023d137df18efc6fb087ddaa10ef485dd303252d3cb783e71094"}]},{"id":"S007","semantic_type":"lemma","kind":"lemma","title":"近端点位移界","source_units":[{"start":36588,"end":36953,"start_line":1036,"end_line":1040,"raw_tex":"\\begin{lemma}\\label{lem:prox_point_bound}\n\tConsider a function $g\\colon \\RR^d \\to \\RR$ that is $L$-Lipschitz continuous on a set $\\cX\\subseteq \\RR^d$. Then for any point $x \\in \\cX$,  every point $\\displaystyle \\hat x \\in \\argmin_{y \\in \\cX} \\left\\{g(y) + \\frac{\\beta}{2} \\|x - y\\|^2\\right\\}$  satisfies the estimate $\n\t\\|\\beta(x - \\hat x)\\| \\leq 2L.\n\t$\n\\end{lemma}","sha256_utf8":"e7c08308526ab029294f1b0c687264b2a43d14cc36160265b960a8ca64895087"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"包络与平稳性度量","source_units":[{"start":40738,"end":41125,"start_line":1117,"end_line":1124,"raw_tex":"Namely, \ndefine \nthe {\\em Moreau envelope} \nand \nthe {\\em proximal map}, respectively, by\n$$\\cM_{\\lambda}(x)= \\min_{y\\in \\cX}~ \\left\\{f(y)+\\frac{1}{2\\lambda}\\|y-x\\|^2\\right\\},\\qquad\\cP_{\\lambda}(x):=\\argmin_{y\\in \\cX}~ \\left\\{f(y)+\\frac{1}{2\\lambda}\\|y-x\\|^2\\right\\},$$\nand define the {\\em stationarity measure}:\n$$\\mathcal{C}_{\\lambda}(x):=\\lambda^{-1}\\cdot\\dist(x,\\cP_{\\lambda}(x)).$$\n","sha256_utf8":"62f087e46672815cb7c8d7194dcac66a643b28f08f44f343863bf3ebf9c4c3cd"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"集合近似与回缩","source_units":[{"start":52788,"end":53725,"start_line":1423,"end_line":1438,"raw_tex":"\\begin{definition}[Set-approximation]\n\t{\\rm Consider a set $\\cX\\subset\\R^d$ and a collection of sets $\\cX_x\\subset\\R^d$ and functions $\\cR_x\\colon\\cX_x\\to\\cX$, indexed by points $x\\in \\R^d$. We say that the collection $\\{(\\cX_x,\\cR_{x})\\}_{x\\in\\cX}$ is a {\\em set approximation of $\\cX$ with parameters $(R,\\accD,r_1,\\accR ,r_2)$}  if the following conditions hold:\n\t\t\\begin{enumerate}\n\t\t\t\\item [$(i)$] {\\bf (Accuracy)} For every $x \\in \\cX$, the set $\\cX_x$ is $R$-proximally smooth and satisfies\n\t\t\t\\begin{equation}\\label{eqn:need_this_eqn}\n\t\t\t\\dist(y, \\cX_{x})  \\leq \\frac{\\accD}{2} \\| x- y\\|^2 \\qquad \\forall y \\in \\cX\\cap \\BB(x,r_1).\n\t\t\t\\end{equation}\n\t\t\t%\t\t\t\\textcolor{red}{Why need this???}\n\t\t\t\n\t\t\t\\item[$(ii)$] {\\bf (Retraction)} For every $x \\in \\cX$, the mapping $\\cR_x$ satisfies\n\t\t\t$$\n\t\t\t\\|y - \\cR_x(y) \\| \\leq \\frac{\\accR }{2} \\|x-y\\|^2 \\qquad \\forall y \\in \\cX_x \\cap \\BB(x,r_2).\n\t\t\t$$\n\t\t\\end{enumerate}\n\t}\n\\end{definition}","sha256_utf8":"406dcd7a077005c845ab16998666aebe8a8d53bff47ed2bf26082be289f05028"}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","title":"模型迭代的构成","source_units":[{"start":80260,"end":81163,"start_line":2038,"end_line":2077,"raw_tex":"\tNamely, \n\tthe algorithm we propose (Algorithm~\\ref{alg:stoc_prox}) \n\twill use \n\tthree building blocks: \n\ta family \n\tof \n\tstochastic models $f_{x}(\\cdot,\\xi)$ \n\tof \n\tthe objective function, \n\ta family \n\tof \n\tproximally smooth sets $\\cX_x$ \n\tthat \n\tapproximate $\\cX$ near $x$, \n\tand \n\ta retraction operation $\\cR_x\\colon\\cX_x\\to\\cX$ \n\tthat \n\trestores feasibility. \n\tThus in each iteration $t$, \n\tAlgorithm~\\ref{alg:stoc_prox} \n\tforms \n\tboth \n\ta proximally smooth approximation $\\cX_{x_t}$ \n\tof \n\tthe original constraint \n\tand  \n\ta stochastic model $f_{x_t}(\\cdot,\\xi_t)$ \n\tof \n\tthe objective function, \n\tcentered at the current iterate $x_t$. \n\tThe procedure then \n\tcomputes \n\ta minimizer ${\\tilde x}_t$ \n\tof \n\tthe function  $f_{x_t}(\\cdot,\\xi_t)+\\frac{\\beta}{2}\\|\\cdot-x_t\\|^2$ \n\tover\n\t$\\cX_{x_t}$ \n\tand \n\tretracts \n\tit back to $\\cX$, thereby defining the next iterate $x_{t+1}=\\cR_{x_t}(\\tilde x_t)$. \n","sha256_utf8":"48a3a6260d84ab9f395c94e6099441c6a288f81caf0ccc281af1b714f6fe2fa4"}]},{"id":"S011","semantic_type":"algorithm","kind":"algorithm","title":"回缩随机模型算法","source_units":[{"start":81181,"end":81771,"start_line":2082,"end_line":2096,"raw_tex":"\\begin{algorithm}[H]\n\t\t{\\bf Input:} initialization $x_0\\in \\R^d$,   a sequence $\\beta_t>0$,  and an iteration count $T\\in \\mathbb{N}$.\n\t\t\\break{\\bf Step } $t=0,\\ldots,T$:\t\n\t\t\\begin{equation*}\n\t\t\\begin{aligned}\n\t\t&\\textrm{Sample } \\xi_t \\sim P\\\\\n\t\t& \\textrm{Choose } \\tilde x_{t} \\in \\argmin_{x \\in \\cX_x}~ \\left\\{f_{x_t}(x,\\xi_t) + \\frac{\\beta_t}{2} \\|x - x_t\\|^2\\right\\}\\\\\n\t\t& \\textrm{Set }  x_{t+1} = \\cR_{x_t}(\\tilde x_t)\n\t\t\\end{aligned}\n\t\t\\end{equation*}\n\t\t\\caption{Retracted Stochastic Model Based Algorithm\n\t\t\t%: PSSM($x_0,T,\\{\\alpha_t\\}$)\n\t\t}\n\t\t\\label{alg:stoc_prox}\n\t\\end{algorithm}","sha256_utf8":"f419b15c7236dc29132f169420008caa529c40462109c48e1aa275f304233d6b"}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","title":"随机模型与集合假设","source_units":[{"start":82023,"end":83348,"start_line":2104,"end_line":2118,"raw_tex":"\\begin{assumption}\n\t\t{\\rm\n\t\t\tFix a probability space $(\\Omega,\\mathcal{F},P)$ and equip $\\R^d$ with the Borel $\\sigma$-algebra. \n\t\t\tWe assume that there exist real $\\eta,\\osa, L, R, \\accD, \\accR, r_1, r_2 \\in \\R$ satisfying the following properties. \n\t\t\t\\begin{enumerate}\n\t\t\t\t\\item[(B1)] {\\bf (Sampling)} It is possible to generate i.i.d.\\ realizations $\\xi_1,\\xi_2, \\ldots \\sim P$.\n\t\t\t\t\\item[(B2)]\\label{it:B2} {\\bf (One-sided accuracy)} There is an open set $U$ containing $\\cX $ and a measurable function $(x,y,\\xi)\\mapsto f_x(y,\\xi)$, defined on $U\\times U\\times\\Omega$, satisfying  $$\\EE_{\\xi}\\left[f_x(x,\\xi)\\right]=f(x)\n\t\t\t\t\\quad \\textrm{and} \n\t\t\t\t\\quad\\EE_{\\xi}\\left[f_x(y,\\xi)-f(y)\\right]\n\t\t\t\t\\leq\\frac{\\osa}{2}\\|y-x\\|^2\\qquad \\forall x,y\\in U.$$\n\t\t\t\t\\item[(B3)] {\\bf (Weak-convexity)} The function $f_x(\\cdot,\\xi)$ is $\\eta$-weakly convex $\\forall x\\in U$, a.e. $\\xi\\sim P$.\n\t\t\t\t\\item [(B4)] {\\bf (Set approximation)} There exists a collection $\\{(\\cX_x,\\cR_{x})\\}_{x\\in\\cX}$ that is a set approximation of $\\cX$ with parameters $(R,\\accD,r_1,\\accR ,r_2)$.\n\t\t\t\t\\item[(B6)] {\\bf (Lipschitz property)} $f$ is $L$-Lipschitz continuous on $\\cX$; and for all $x\\in \\cX$ and a.e. $\\xi\\sim P$, the function $f_x(\\cdot, \\xi)$ is $L$-Lipschitz continuous on some neighborhood of $\\cX_x$.\n\t\t\\end{enumerate}}\n\t\\end{assumption}","sha256_utf8":"0ac1c33ab80af57a5ba34b88fef5d51e1f222a51af5ab4e579aee7f493e68290"}]},{"id":"S013","semantic_type":"notation","kind":"notation","title":"条件期望记号","source_units":[{"start":85440,"end":85745,"start_line":2155,"end_line":2155,"raw_tex":"\tIn what follows, we let   $x_0,\\ldots, x_t$ and $\\xi_0,\\ldots, \\xi_{t}$ be the iterates and random elements generated by Algorithm~\\ref{alg:stoc_prox}. We will use the shorthand $\\EE_t$ to denote the expectation conditioned on $\\xi_0,\\ldots, \\xi_{t-1}$.  We begin with a key one-step improvement lemma. \n","sha256_utf8":"1dbf56a1b9fe0dc33234257cb8e4e011e6f5963d110d9a494002fb83896bfbe8"}]},{"id":"S014","semantic_type":"theorem","kind":"theorem","title":"回缩随机算法收敛率","source_units":[{"start":83494,"end":85074,"start_line":2129,"end_line":2146,"raw_tex":"\\begin{theorem}[Convergence guarantees]\\label{thm:gen_conv_guarant}\n\t\tWithout loss of generality, suppose $r_1<R$ and define the constants $\\nu := \\frac{R}{2(R-r_1)^2} $ and $\\gamma:=\\eta+3L\\nu$. Fix a real $\\bar{\\rho} > \\max\\{\\frac{2L}{r_1}, \\gamma+\\osa+3\\accD L\\}$ and a sequence $\\beta_t > \\max\\{\\frac{2L}{r_2},\\gamma\\}$. Let $\\{x_t\\}_{t=0}^T$ be the iterates generated by  Algorithm~\\ref{alg:stoc_prox} and let  $t^*\\in \\{0,\\ldots,T\\}$ be sampled according to the discrete probability distribution\n\t\t$\\mathbb{P}(t^*=t)\\propto\\frac{1}{\\beta_t - \\eta-\\frac{3LR}{2(R-r_1)^2}}$.\n\t\tThen $x_{t^*}$ satisfies the estimate:\t\n\t\t\\begin{align*}\n\t\t\\EE_t\\left[\\cC_{1/\\bar \\rho}(x_{t^\\ast})^2\\right] \\leq \\frac{2\\bar\\rho(f(x_0) - \\min_{\\cX} f) + \\bar\\rho^2L^2\\sum_{t=0}^T a_t}{\\sum_{t=0}^T \\frac{\\bar \\rho - \\gamma - \\osa -  3\\accD L}{\\beta_t - \\gamma }},\n\t\t\\end{align*}\n\t\twhere we define\n\t\t$$\n\t\ta_t :=  \\frac{1}{\\beta_t(\\beta_t - \\gamma)} + \\frac{8\\accR L(\\bar \\rho^{-1} + \\beta_t^{-1})}{\\beta_t^2} + \\frac{4\\accR ^2 L^2}{\\beta_t^4}.\n\t\t$$\n\t\tIn particular, if we set $\\beta_t=\\gamma+\\frac{\\sqrt{T+1}}{\\alpha}$ for some positive  $\\alpha < \\frac{r_2}{2L-\\gamma r_2}$, then it holds:\n\t\t\\begin{align*}\n\t\t\\EE_t\\left[\\cC_{1/\\bar \\rho}(x_{t^\\ast})^2\\right] \\leq & \\frac{2\\bar\\rho(f(x_0) - \\min_{\\cX} f)}{\\alpha(\\bar \\rho - \\gamma - \\osa -  3\\accD L)\\sqrt{T+1}}\\\\\n\t\t&\\quad+\\frac{\\alpha\\bar \\rho^2L^2}{\\bar \\rho - \\gamma - \\osa -  3\\accD L}\\left[\\frac{1+\\accR L(\\bar \\rho^{-1}+\\frac{\\alpha}{\\sqrt{T+1}})}{\\sqrt{T+1}}+\\frac{4\\alpha^2\\accR ^2L^2}{(T+1)^{3/2}}\\right].\n\t\t\\end{align*}\n\t\t\n\t\\end{theorem}","sha256_utf8":"159269083289ed02d909e0de75d2b4de72bcd5ed30130812f6800b922900ebea"}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","title":"参数与随机输出条件","source_units":[{"start":83562,"end":84073,"start_line":2130,"end_line":2131,"raw_tex":"\t\tWithout loss of generality, suppose $r_1<R$ and define the constants $\\nu := \\frac{R}{2(R-r_1)^2} $ and $\\gamma:=\\eta+3L\\nu$. Fix a real $\\bar{\\rho} > \\max\\{\\frac{2L}{r_1}, \\gamma+\\osa+3\\accD L\\}$ and a sequence $\\beta_t > \\max\\{\\frac{2L}{r_2},\\gamma\\}$. Let $\\{x_t\\}_{t=0}^T$ be the iterates generated by  Algorithm~\\ref{alg:stoc_prox} and let  $t^*\\in \\{0,\\ldots,T\\}$ be sampled according to the discrete probability distribution\n\t\t$\\mathbb{P}(t^*=t)\\propto\\frac{1}{\\beta_t - \\eta-\\frac{3LR}{2(R-r_1)^2}}$.\n","sha256_utf8":"e7d3dc56fd6d71826bd210e518372b319ee4b764bda733612a6c784423f79d6a"}]},{"id":"S016","semantic_type":"lemma","kind":"lemma","title":"单步近端距离改进","source_units":[{"start":85959,"end":86352,"start_line":2162,"end_line":2168,"raw_tex":"\\begin{lemma}[One-step improvement]\\label{lem:stoc_prox_descent}\n\t\tFix an index $t$ and choose  $\\hat x_t\\in \\cP_{1/\\bar \\rho}(x)$. Then the estimate holds:\n\t\t\\begin{equation}\n\t\t\\EE_t\\left[\\|\\hat x_t - x_{t+1}\\|^2\\right] \\leq  \\frac{\\beta +\\osa+ 3\\accD L - \\bar \\rho }{\\beta- \\gamma}\\|x_t - {\\hat x}_t \\|^2 + L^2 a_t.\\label{ineq:stoc_prox_descent}\n\t\t\\end{equation}\n\t\t%\t\\end{align}\n\t\\end{lemma}","sha256_utf8":"193575bbf03bff390da0b5bc2edcb78795d2a8723389b8be5374e6a3b6c05208"}]},{"id":"S017","semantic_type":"proof","kind":"proof","title":"单步改进完整证明","source_units":[{"start":86354,"end":91744,"start_line":2169,"end_line":2239,"raw_tex":"\\begin{proof}\n\t\tSince the statement of the theorem is independent of $x_0,\\ldots, x_{t-1}$, we will simplify the notation by dropping the index $t$ in $x_t$, ${\\tilde x}_t$, ${\\hat x}_t$, $\\xi_t$, $\\EE_t$, $a_t$ and setting $x^+:=x_{t+1}$.\n\t\tBy the definition of $\\tilde x$, there exist vectors $v \\in \\partial f_{x}(\\tilde x, \\xi)$ and $w \\in N_{\\cX_{x}}(\\tilde x)$ satisfying  $\\beta(x - \\tilde x) = v + w.$\n\t\tUsing Lemma~\\ref{lem:prox_point_bound} and Lipschitz continuity of $f_x(\\cdot,\\xi)$ yields the estimate $\\|w\\| \\leq 3L$. Fix any  $y \\in \\cX\\cap\\BB(x,r_1)$. We then deduce\n\t\t\\begin{align}\n\t\t\\dotp{w, y- \\tilde x} &= \\dotp{w, (y - \\proj_{\\cX_x}(y))} + \\dotp{w, \\proj_{\\cX_x}(y) - \\tilde x}\\notag\\\\\n\t\t&\\leq \\|w\\|\\cdot \\dist(y, \\cX_x) + \\frac{\\|w\\|}{2R} \\cdot\\|\\proj_{\\cX_x}(y) - \\tilde x\\|^2\\label{eqn:part2it1}\\\\\n\t\t&\\leq \\frac{\\|w\\|\\accD}{2} \\|y - x\\|^2 + \\frac{\\|w\\| \\nu}{2} \\|y - \\tilde x\\|^2,\\label{eqn:part2it2}\n\t\t\\end{align}\n\t\twhere \\eqref{eqn:part2it1} follows from the Cauchy-Schwarz inequality and from Lemma~\\ref{lem_basic_prop_man}, while \\eqref{eqn:part2it2} follows from (B4) and Lemma~\\ref{lem_basic_prop_man}.\n\t\tAdding this estimate to the subgradient inequality for $f_x(\\cdot,\\xi)$ and completing the square, we conclude  for all $y \\in \\cX\\cap\\BB(x,r_1)$ the  bound:\n\t\t\\begin{align}\n\t\tf_{x}(y, \\xi) &\\geq f_{x}(\\tilde x, \\xi) + \\dotp{v + w, y- \\tilde x} - \\frac{\\eta + \\|w\\| \\nu}{2} \\|y - \\tilde x\\|^2 - \\frac{\\|w\\|\\accD}{2}\\|y - x\\|^2\\notag\\\\\n\t\t&\\geq f_{x}(\\tilde x, \\xi) + \\dotp{\\beta(x - \\tilde x) , y- \\tilde x} - \\frac{\\eta + \\|w\\| \\nu}{2} \\|y - \\tilde x\\|^2  - \\frac{\\|w\\|\\accD}{2}\\|y - x\\|^2 \\notag\\\\\n\t\t&= f_{x}(\\tilde x, \\xi) + \\frac{\\beta}{2}\\left[ \\|x - \\tilde x\\|^2 + \\| y - \\tilde x\\|^2 - \\|x - y\\|^2\\right]\\label{eqn:perturb_three_point} \\\\\n\t\t&\\qquad- \\frac{\\eta + \\|w\\| \\nu}{2} \\|y - \\tilde x\\|^2  - \\frac{\\|w\\|\\accD}{2}\\|y - x\\|^2.\\notag\n\t\t\\end{align}\n\t\tLemma~\\ref{lem:prox_point_bound} implies\n\t\t$\n\t\t\\|\\hat x - x\\| \\leq \\frac{2L}{\\bar \\rho} \\leq r_1.\n\t\t$\n\t\tHence, setting $y = \\hat x$ in \\eqref{eqn:perturb_three_point} yields: \n\t\t\\begin{align}\n\t\t&\\EE\\left[\\frac{\\beta - \\gamma}{2} \\|\\hat x - \\tilde x\\|^2 + \\frac{\\beta}{2}\\| x - \\tilde x\\|^2 - \\frac{\\beta + \\|w\\|\\accD }{2}\\|x - \\hat x\\|^2\\right]\\notag\\\\\n\t\t&\\leq \\EE\\left[f_x(\\hat x, \\xi) - f_x(\\tilde x, \\xi)\\right]\\notag \\\\\n\t\t&\\leq \\EE\\left[f_x(\\hat x, \\xi) - f_x(x,\\xi)\\right]  + L \\EE\\left[\\|x - \\tilde x\\|\\right]\\label{eqn:blah1}\\\\\t\n\t\t&\\leq f(\\hat x) - f(x)  + L \\EE\\left[\\|x - \\tilde x\\|\\right] + \\frac{\\osa}{2}\\|x - \\hat x \\|^2\\label{eqn:blah2}\\\\\n\t\t&\\leq \\frac{\\osa - \\bar \\rho }{2}\\|x - \\hat x \\|^2 + L\\sqrt{\\EE\\left[ \\|x - \\tilde x\\|^2\\right]},\\label{eqn:blah3}\n\t\t%&\\leq \\frac{\\tau - \\bar \\rho }{2}\\|x - \\hat x \\|^2 + \\frac{2L^2}{\\beta}.\n\t\t\\end{align}\n\t\twhere \\eqref{eqn:blah1} uses (B6), the estimate \\eqref{eqn:blah2} uses (B2), and \\eqref{eqn:blah3} uses the definition of $\\hat x$ as the proximal point.\n\t\tRearranging and setting $\\delta :=\\sqrt{ \\EE\\left[\\|x - \\tilde x\\|^2\\right]}$ yields the bound:\n\t\t\\begin{equation}\\label{eqn:intermed}\n\t\t\\begin{aligned}\n\t\t\\frac{\\beta - \\gamma}{2}\\EE\\left[\\|\\hat x - \\tilde x\\|^2\\right] &\\leq \\frac{\\beta + \\accD\\|w\\|+ \\osa - \\bar \\rho  }{2}\\|x - \\hat x \\|^2 + L \\delta - \\frac{\\beta }{2}\\delta^2 \\\\\n\t\t&\\leq \\frac{\\beta + \\accD\\|w\\|+ \\osa - \\bar \\rho  }{2}\\|x - \\hat x \\|^2 + \\frac{L^2}{2\\beta},\n\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\twhere the last inequality follows by maximizing the expression  $L\\delta - \\frac{\\beta }{2}\\delta^2$ over $\\delta\\in \\R$.\n\t\tTo complete the proof, we will use the following estimate.\n\t\t\\begin{claim}\\label{claim1} The estimate holds:\n\t\t\t$\n\t\t\t\\|\\hat x -  x_+\\|^2 \\leq \\|\\hat x - \\tilde x\\|^2 + \\frac{8\\accR  L^3}{\\beta^2}(\\bar \\rho^{-1} + \\beta^{-1}) + \\frac{4\\accR ^2 L^4}{\\beta^4}\n\t\t\t$.\n\t\t\\end{claim}\n\t\t\\begin{proof}\n\t\t\tUsing Lemma~\\ref{lem:prox_point_bound}, we deduce\n\t\t\t$\\|\\tilde x - x\\| \\leq 2L/\\beta  \\leq r_2$.\n\t\t\tIn addition, we have $$\\|\\tilde x - x_+ \\| = \\|\\tilde x - \\cR_x(\\tilde x)\\| \\leq \\frac{\\accR }{2}\\|\\tilde x - x\\|^2\\leq \\frac{2\\accR  L^2}{\\beta^2}.$$ \n\t\t\tUsing the triangle inequality, we  deduce \n\t\t\t$\n\t\t\t\\|\\hat x - x_+\\|  \\leq \\|\\hat x - \\tilde x \\| + \\frac{2\\accR  L^2}{\\beta^2}.\n\t\t\t$\n\t\t\tSquaring both sides, we get\n\t\t\t$$\n\t\t\t\\|\\hat x - x_+\\|^2  \\leq \\|\\hat x - \\tilde x \\|^2 +  \\frac{4\\accR  L^2}{\\beta^2}\\|\\hat x - \\tilde x \\| + \\frac{4\\accR ^2 L^4}{\\beta^4}.\n\t\t\t$$\n\t\t\tTaking into account the estimate $\n\t\t\t\\|\\hat x - \\tilde x \\| \\leq \\|\\hat x - x \\| + \\|x - \\tilde x\\| \\leq 2L/\\bar \\rho + 2L/\\beta,\n\t\t\t$\n\t\t\tcompletes the proof.\n\t\t\\end{proof}\n\t\tCombining Claim~\\ref{claim1} with the estimate \\eqref{eqn:intermed}, we compute\n\t\t\\begin{align*}\n\t\t\\EE\\left[\\|\\hat x -  x_+\\|^2\\right] &\\leq \\EE\\left[\\|\\hat x - \\tilde x\\|^2\\right] + \\frac{8\\accR  L^3(\\bar \\rho^{-1} + \\beta^{-1})}{\\beta^2} + \\frac{4\\accR ^2 L^4}{\\beta^4} \\\\\n\t\t&\\leq \\frac{\\beta + \\accD\\|w\\|+ \\osa - \\bar \\rho }{\\beta- \\gamma}\\|x - \\hat x \\|^2 + L^2 a\n\t\t%\\frac{L^2}{\\beta(\\beta - \\gamma)} + \\frac{8\\accR  L^3(\\bar \\rho^{-1} + \\beta^{-1})}{\\beta^2} + \\frac{4\\accR ^2 L^4}{\\beta^4}\n\t\t%\t&=  \\|x - \\hat x \\|^2  - \\frac{\\bar \\rho - \\gamma - \\osa  -  \\accD\\|w\\|}{\\beta - \\gamma}\\|x - \\hat x \\|^2  + \\frac{L^2}{\\beta(\\beta - \\gamma)} + \\frac{8\\accR  L^3(\\bar \\rho^{-1} + \\beta^{-1})}{\\beta^2} + \\frac{4\\accR ^2 L^4}{\\beta^4}.\n\t\t\\end{align*}\n\t\tBounding $\\|w \\| \\leq 3L$ completes the proof of \\eqref{ineq:stoc_prox_descent}.\n\t\\end{proof}","sha256_utf8":"ac2c6a13278a5ced02eefe43b25cf6ddb35b9c3b3901d7e05957155fa5866f95"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"最优性分解与法向界","source_units":[{"start":86354,"end":86938,"start_line":2169,"end_line":2172,"raw_tex":"\\begin{proof}\n\t\tSince the statement of the theorem is independent of $x_0,\\ldots, x_{t-1}$, we will simplify the notation by dropping the index $t$ in $x_t$, ${\\tilde x}_t$, ${\\hat x}_t$, $\\xi_t$, $\\EE_t$, $a_t$ and setting $x^+:=x_{t+1}$.\n\t\tBy the definition of $\\tilde x$, there exist vectors $v \\in \\partial f_{x}(\\tilde x, \\xi)$ and $w \\in N_{\\cX_{x}}(\\tilde x)$ satisfying  $\\beta(x - \\tilde x) = v + w.$\n\t\tUsing Lemma~\\ref{lem:prox_point_bound} and Lipschitz continuity of $f_x(\\cdot,\\xi)$ yields the estimate $\\|w\\| \\leq 3L$. Fix any  $y \\in \\cX\\cap\\BB(x,r_1)$. We then deduce\n","sha256_utf8":"e0de3d2479c9ac1a80bd2448d2299f4383f530062d11f88989b5009138b0232c"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"集合近似修正法向界","source_units":[{"start":86938,"end":87488,"start_line":2173,"end_line":2178,"raw_tex":"\t\t\\begin{align}\n\t\t\\dotp{w, y- \\tilde x} &= \\dotp{w, (y - \\proj_{\\cX_x}(y))} + \\dotp{w, \\proj_{\\cX_x}(y) - \\tilde x}\\notag\\\\\n\t\t&\\leq \\|w\\|\\cdot \\dist(y, \\cX_x) + \\frac{\\|w\\|}{2R} \\cdot\\|\\proj_{\\cX_x}(y) - \\tilde x\\|^2\\label{eqn:part2it1}\\\\\n\t\t&\\leq \\frac{\\|w\\|\\accD}{2} \\|y - x\\|^2 + \\frac{\\|w\\| \\nu}{2} \\|y - \\tilde x\\|^2,\\label{eqn:part2it2}\n\t\t\\end{align}\n\t\twhere \\eqref{eqn:part2it1} follows from the Cauchy-Schwarz inequality and from Lemma~\\ref{lem_basic_prop_man}, while \\eqref{eqn:part2it2} follows from (B4) and Lemma~\\ref{lem_basic_prop_man}.\n","sha256_utf8":"25b3b1ad5e5c8fda50ba47e5b845c00d33c64dcdbd3641da5e84451e2dcd3fd0"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"推导扰动三点不等式","source_units":[{"start":87488,"end":88248,"start_line":2179,"end_line":2185,"raw_tex":"\t\tAdding this estimate to the subgradient inequality for $f_x(\\cdot,\\xi)$ and completing the square, we conclude  for all $y \\in \\cX\\cap\\BB(x,r_1)$ the  bound:\n\t\t\\begin{align}\n\t\tf_{x}(y, \\xi) &\\geq f_{x}(\\tilde x, \\xi) + \\dotp{v + w, y- \\tilde x} - \\frac{\\eta + \\|w\\| \\nu}{2} \\|y - \\tilde x\\|^2 - \\frac{\\|w\\|\\accD}{2}\\|y - x\\|^2\\notag\\\\\n\t\t&\\geq f_{x}(\\tilde x, \\xi) + \\dotp{\\beta(x - \\tilde x) , y- \\tilde x} - \\frac{\\eta + \\|w\\| \\nu}{2} \\|y - \\tilde x\\|^2  - \\frac{\\|w\\|\\accD}{2}\\|y - x\\|^2 \\notag\\\\\n\t\t&= f_{x}(\\tilde x, \\xi) + \\frac{\\beta}{2}\\left[ \\|x - \\tilde x\\|^2 + \\| y - \\tilde x\\|^2 - \\|x - y\\|^2\\right]\\label{eqn:perturb_three_point} \\\\\n\t\t&\\qquad- \\frac{\\eta + \\|w\\| \\nu}{2} \\|y - \\tilde x\\|^2  - \\frac{\\|w\\|\\accD}{2}\\|y - x\\|^2.\\notag\n\t\t\\end{align}\n","sha256_utf8":"9869b9c700a49bf65d71dd1a80835165213b396474ceba928c40f6514ed8c223"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"代入近端点并取期望","source_units":[{"start":88248,"end":89265,"start_line":2186,"end_line":2199,"raw_tex":"\t\tLemma~\\ref{lem:prox_point_bound} implies\n\t\t$\n\t\t\\|\\hat x - x\\| \\leq \\frac{2L}{\\bar \\rho} \\leq r_1.\n\t\t$\n\t\tHence, setting $y = \\hat x$ in \\eqref{eqn:perturb_three_point} yields: \n\t\t\\begin{align}\n\t\t&\\EE\\left[\\frac{\\beta - \\gamma}{2} \\|\\hat x - \\tilde x\\|^2 + \\frac{\\beta}{2}\\| x - \\tilde x\\|^2 - \\frac{\\beta + \\|w\\|\\accD }{2}\\|x - \\hat x\\|^2\\right]\\notag\\\\\n\t\t&\\leq \\EE\\left[f_x(\\hat x, \\xi) - f_x(\\tilde x, \\xi)\\right]\\notag \\\\\n\t\t&\\leq \\EE\\left[f_x(\\hat x, \\xi) - f_x(x,\\xi)\\right]  + L \\EE\\left[\\|x - \\tilde x\\|\\right]\\label{eqn:blah1}\\\\\t\n\t\t&\\leq f(\\hat x) - f(x)  + L \\EE\\left[\\|x - \\tilde x\\|\\right] + \\frac{\\osa}{2}\\|x - \\hat x \\|^2\\label{eqn:blah2}\\\\\n\t\t&\\leq \\frac{\\osa - \\bar \\rho }{2}\\|x - \\hat x \\|^2 + L\\sqrt{\\EE\\left[ \\|x - \\tilde x\\|^2\\right]},\\label{eqn:blah3}\n\t\t%&\\leq \\frac{\\tau - \\bar \\rho }{2}\\|x - \\hat x \\|^2 + \\frac{2L^2}{\\beta}.\n\t\t\\end{align}\n\t\twhere \\eqref{eqn:blah1} uses (B6), the estimate \\eqref{eqn:blah2} uses (B2), and \\eqref{eqn:blah3} uses the definition of $\\hat x$ as the proximal point.\n","sha256_utf8":"e1a343451ae6830fbbbf0b2e41e0221e0c86775673e28549f7d220362321ac33"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"优化标量二次项","source_units":[{"start":89265,"end":89913,"start_line":2200,"end_line":2208,"raw_tex":"\t\tRearranging and setting $\\delta :=\\sqrt{ \\EE\\left[\\|x - \\tilde x\\|^2\\right]}$ yields the bound:\n\t\t\\begin{equation}\\label{eqn:intermed}\n\t\t\\begin{aligned}\n\t\t\\frac{\\beta - \\gamma}{2}\\EE\\left[\\|\\hat x - \\tilde x\\|^2\\right] &\\leq \\frac{\\beta + \\accD\\|w\\|+ \\osa - \\bar \\rho  }{2}\\|x - \\hat x \\|^2 + L \\delta - \\frac{\\beta }{2}\\delta^2 \\\\\n\t\t&\\leq \\frac{\\beta + \\accD\\|w\\|+ \\osa - \\bar \\rho  }{2}\\|x - \\hat x \\|^2 + \\frac{L^2}{2\\beta},\n\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\twhere the last inequality follows by maximizing the expression  $L\\delta - \\frac{\\beta }{2}\\delta^2$ over $\\delta\\in \\R$.\n\t\tTo complete the proof, we will use the following estimate.\n","sha256_utf8":"54be4e930165b4e2d8e2fd5c97fc4f63fb47d9af74a47a1150ec44a0261ae0b3"}]},{"id":"S023","semantic_type":"derived_fact","kind":"derived_fact","title":"回缩引起的距离误差","source_units":[{"start":89915,"end":90130,"start_line":2209,"end_line":2213,"raw_tex":"\\begin{claim}\\label{claim1} The estimate holds:\n\t\t\t$\n\t\t\t\\|\\hat x -  x_+\\|^2 \\leq \\|\\hat x - \\tilde x\\|^2 + \\frac{8\\accR  L^3}{\\beta^2}(\\bar \\rho^{-1} + \\beta^{-1}) + \\frac{4\\accR ^2 L^4}{\\beta^4}\n\t\t\t$.\n\t\t\\end{claim}","sha256_utf8":"142394408ec6db7ffba56adedcdbc9bed7bf7ced3d693ac6c20d9da1263e602b"}]},{"id":"S024","semantic_type":"proof","kind":"proof","title":"回缩误差完整证明","source_units":[{"start":90133,"end":90896,"start_line":2214,"end_line":2230,"raw_tex":"\\begin{proof}\n\t\t\tUsing Lemma~\\ref{lem:prox_point_bound}, we deduce\n\t\t\t$\\|\\tilde x - x\\| \\leq 2L/\\beta  \\leq r_2$.\n\t\t\tIn addition, we have $$\\|\\tilde x - x_+ \\| = \\|\\tilde x - \\cR_x(\\tilde x)\\| \\leq \\frac{\\accR }{2}\\|\\tilde x - x\\|^2\\leq \\frac{2\\accR  L^2}{\\beta^2}.$$ \n\t\t\tUsing the triangle inequality, we  deduce \n\t\t\t$\n\t\t\t\\|\\hat x - x_+\\|  \\leq \\|\\hat x - \\tilde x \\| + \\frac{2\\accR  L^2}{\\beta^2}.\n\t\t\t$\n\t\t\tSquaring both sides, we get\n\t\t\t$$\n\t\t\t\\|\\hat x - x_+\\|^2  \\leq \\|\\hat x - \\tilde x \\|^2 +  \\frac{4\\accR  L^2}{\\beta^2}\\|\\hat x - \\tilde x \\| + \\frac{4\\accR ^2 L^4}{\\beta^4}.\n\t\t\t$$\n\t\t\tTaking into account the estimate $\n\t\t\t\\|\\hat x - \\tilde x \\| \\leq \\|\\hat x - x \\| + \\|x - \\tilde x\\| \\leq 2L/\\bar \\rho + 2L/\\beta,\n\t\t\t$\n\t\t\tcompletes the proof.\n\t\t\\end{proof}","sha256_utf8":"d7c0b4f81e5cc0de8d01afb23639218960eb15cee1d5fcf3f1a501408bdb64e9"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"控制模型步与回缩位移","source_units":[{"start":90131,"end":90402,"start_line":2214,"end_line":2217,"raw_tex":"\t\t\\begin{proof}\n\t\t\tUsing Lemma~\\ref{lem:prox_point_bound}, we deduce\n\t\t\t$\\|\\tilde x - x\\| \\leq 2L/\\beta  \\leq r_2$.\n\t\t\tIn addition, we have $$\\|\\tilde x - x_+ \\| = \\|\\tilde x - \\cR_x(\\tilde x)\\| \\leq \\frac{\\accR }{2}\\|\\tilde x - x\\|^2\\leq \\frac{2\\accR  L^2}{\\beta^2}.$$ \n","sha256_utf8":"d713b2a0d84a9284572d36973f0b14798d66bcbd2de20200c003af6c1225b734"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"平方展开回缩距离","source_units":[{"start":90402,"end":90720,"start_line":2218,"end_line":2225,"raw_tex":"\t\t\tUsing the triangle inequality, we  deduce \n\t\t\t$\n\t\t\t\\|\\hat x - x_+\\|  \\leq \\|\\hat x - \\tilde x \\| + \\frac{2\\accR  L^2}{\\beta^2}.\n\t\t\t$\n\t\t\tSquaring both sides, we get\n\t\t\t$$\n\t\t\t\\|\\hat x - x_+\\|^2  \\leq \\|\\hat x - \\tilde x \\|^2 +  \\frac{4\\accR  L^2}{\\beta^2}\\|\\hat x - \\tilde x \\| + \\frac{4\\accR ^2 L^4}{\\beta^4}.\n\t\t\t$$\n","sha256_utf8":"5625b5a10b44bb65c886a72fcb8ed1c8605fc44d47ddad320128779d7bcae173"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"用近端位移界收尾","source_units":[{"start":90720,"end":90897,"start_line":2226,"end_line":2230,"raw_tex":"\t\t\tTaking into account the estimate $\n\t\t\t\\|\\hat x - \\tilde x \\| \\leq \\|\\hat x - x \\| + \\|x - \\tilde x\\| \\leq 2L/\\bar \\rho + 2L/\\beta,\n\t\t\t$\n\t\t\tcompletes the proof.\n\t\t\\end{proof}\n","sha256_utf8":"0162800a3f5152b13445108dd3bb893284f8ef4a5fec2c0faddbb2c4819a2c4f"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"合并回缩误差与单步界","source_units":[{"start":90897,"end":91744,"start_line":2231,"end_line":2239,"raw_tex":"\t\tCombining Claim~\\ref{claim1} with the estimate \\eqref{eqn:intermed}, we compute\n\t\t\\begin{align*}\n\t\t\\EE\\left[\\|\\hat x -  x_+\\|^2\\right] &\\leq \\EE\\left[\\|\\hat x - \\tilde x\\|^2\\right] + \\frac{8\\accR  L^3(\\bar \\rho^{-1} + \\beta^{-1})}{\\beta^2} + \\frac{4\\accR ^2 L^4}{\\beta^4} \\\\\n\t\t&\\leq \\frac{\\beta + \\accD\\|w\\|+ \\osa - \\bar \\rho }{\\beta- \\gamma}\\|x - \\hat x \\|^2 + L^2 a\n\t\t%\\frac{L^2}{\\beta(\\beta - \\gamma)} + \\frac{8\\accR  L^3(\\bar \\rho^{-1} + \\beta^{-1})}{\\beta^2} + \\frac{4\\accR ^2 L^4}{\\beta^4}\n\t\t%\t&=  \\|x - \\hat x \\|^2  - \\frac{\\bar \\rho - \\gamma - \\osa  -  \\accD\\|w\\|}{\\beta - \\gamma}\\|x - \\hat x \\|^2  + \\frac{L^2}{\\beta(\\beta - \\gamma)} + \\frac{8\\accR  L^3(\\bar \\rho^{-1} + \\beta^{-1})}{\\beta^2} + \\frac{4\\accR ^2 L^4}{\\beta^4}.\n\t\t\\end{align*}\n\t\tBounding $\\|w \\| \\leq 3L$ completes the proof of \\eqref{ineq:stoc_prox_descent}.\n\t\\end{proof}","sha256_utf8":"b967ce32ff727b15a2f5ed758b4d7ff7fcb54f7878e64280869656e4a3f8e41c"}]},{"id":"S029","semantic_type":"proof","kind":"proof","title":"主收敛定理完整证明","source_units":[{"start":91802,"end":93047,"start_line":2245,"end_line":2257,"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm:gen_conv_guarant}]\n\t\tFix an iteration $t$ and a  point  $\\hat x_t\\in\\cP_{1/\\bar \\rho}(x_t)$. Then\n\t\tusing the definition of the Moreau envelope and appealing to  Lemma~\\ref{lem:stoc_prox_descent}, we deduce  \n\t\t\\begin{align*}\n\t\t\\EE_t[ \\cM_{1/\\bar \\rho}(x_{t+1})] &\\leq  \\EE_t\\left[f( \\hat x_{t})  +\\frac{\\bar \\rho}{2}\\cdot \\| x_{t+1} - \\hat x_t \\|^2\\right],\\\\\n\t\t&\\leq f( \\hat x_{t}) +\\frac{\\bar \\rho}{2}\\left[\\|\\hat x_t-x_{t}\\|^2+\\left( \\tfrac{\\beta +\\osa+ 3\\accD L - \\bar \\rho }{\\beta- \\gamma}-1\\right)\\|\\hat x_t-x_{t}\\|^2  +L^2 a_t\\right]\\\\\n\t\t&= \\cM_{1/\\bar \\rho}(x_{t}) - \\frac{\\bar \\rho- \\gamma-\\osa-3\\accD L}{2\\bar \\rho(\\beta_t-\\gamma)}\\cC_{1/\\bar \\rho}(x_t)^2  + \\frac{\\bar\\rho L^2a_t}{2}.\n\t\t\\end{align*}\n\t\tTaking expectations, iterating the inequality, and using the tower rule  yields:\n\t\t$$\\sum_{t=0}^T \\frac{\\bar \\rho - \\gamma - \\osa-3\\accD L}{\\beta_t-\\gamma}\\EE[\\cC_{1/\\bar{\\rho}}(x_t)^2]\\leq 2\\bar{\\rho}(\\cM_{1/\\bar \\rho}(x_{0})-\\min \\cM_{1/\\bar\\rho})+  \\bar \\rho^2 L^2\\cdot\\sum_{t=0}^T a_t.$$\n\t\tDividing through by $\\sum_{t=0}^T \\frac{\\bar \\rho - \\gamma - \\osa-3\\accD L}{\\beta_t-\\gamma}$ and recognizing the left side as\n\t\t$\\EE[\\cC_{1/\\bar{\\rho}}(x_{t^*})^2]$ completes the proof.\n\t\\end{proof}","sha256_utf8":"433df90dcc64c2e6e922d7938a757bbd4bf7ac62e31fe0d43d8d00eda68ad929"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"建立包络期望下降","source_units":[{"start":91802,"end":92553,"start_line":2245,"end_line":2252,"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm:gen_conv_guarant}]\n\t\tFix an iteration $t$ and a  point  $\\hat x_t\\in\\cP_{1/\\bar \\rho}(x_t)$. Then\n\t\tusing the definition of the Moreau envelope and appealing to  Lemma~\\ref{lem:stoc_prox_descent}, we deduce  \n\t\t\\begin{align*}\n\t\t\\EE_t[ \\cM_{1/\\bar \\rho}(x_{t+1})] &\\leq  \\EE_t\\left[f( \\hat x_{t})  +\\frac{\\bar \\rho}{2}\\cdot \\| x_{t+1} - \\hat x_t \\|^2\\right],\\\\\n\t\t&\\leq f( \\hat x_{t}) +\\frac{\\bar \\rho}{2}\\left[\\|\\hat x_t-x_{t}\\|^2+\\left( \\tfrac{\\beta +\\osa+ 3\\accD L - \\bar \\rho }{\\beta- \\gamma}-1\\right)\\|\\hat x_t-x_{t}\\|^2  +L^2 a_t\\right]\\\\\n\t\t&= \\cM_{1/\\bar \\rho}(x_{t}) - \\frac{\\bar \\rho- \\gamma-\\osa-3\\accD L}{2\\bar \\rho(\\beta_t-\\gamma)}\\cC_{1/\\bar \\rho}(x_t)^2  + \\frac{\\bar\\rho L^2a_t}{2}.\n\t\t\\end{align*}\n","sha256_utf8":"e4b03aea16a3eed6685c701783bf26a61227ea2ff4c9e8d717fd7df27be74251"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"塔式法则与累加","source_units":[{"start":92553,"end":92847,"start_line":2253,"end_line":2254,"raw_tex":"\t\tTaking expectations, iterating the inequality, and using the tower rule  yields:\n\t\t$$\\sum_{t=0}^T \\frac{\\bar \\rho - \\gamma - \\osa-3\\accD L}{\\beta_t-\\gamma}\\EE[\\cC_{1/\\bar{\\rho}}(x_t)^2]\\leq 2\\bar{\\rho}(\\cM_{1/\\bar \\rho}(x_{0})-\\min \\cM_{1/\\bar\\rho})+  \\bar \\rho^2 L^2\\cdot\\sum_{t=0}^T a_t.$$\n","sha256_utf8":"d3139ef72dcbff1d349ac54f049b9e1c6262e70353ec75e2dc32a453d31aafaa"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"归一化得到随机输出界","source_units":[{"start":92847,"end":93047,"start_line":2255,"end_line":2257,"raw_tex":"\t\tDividing through by $\\sum_{t=0}^T \\frac{\\bar \\rho - \\gamma - \\osa-3\\accD L}{\\beta_t-\\gamma}$ and recognizing the left side as\n\t\t$\\EE[\\cC_{1/\\bar{\\rho}}(x_{t^*})^2]$ completes the proof.\n\t\\end{proof}","sha256_utf8":"3975df28ec278c457544d8b9a739554799c519cc930ae20e67199b19279a1749"}]},{"id":"S033","semantic_type":"external_fact","kind":"external_fact","title":"近端光滑集合标准性质","source_units":[],"declared_external":true,"declaration_text":["对R-近端光滑集合Q及0<r<R，投影在U_Q(r)上满足||proj_Q(x)-proj_Q(y)||≤R/(R-r)||x-y||；对z,y∈Q及w∈N_Q(z)，有〈w,y-z〉≤||w||||y-z||²/(2R)。这是原文归于clarkelower的标准结果。"]},{"id":"S034","semantic_type":"external_fact","kind":"external_fact","title":"标准近端位移估计","source_units":[],"declared_external":true,"declaration_text":["若g在Q上为L-Lipschitz，x∈Q，β>0，且z∈argmin_{y∈Q}{g(y)+β||x-y||²/2}，则β||x-z||≤2L。原文B0041记录这一初等标准估计，并在1034–1035行明确省略其证明。"]},{"id":"S035","semantic_type":"external_fact","kind":"external_fact","title":"标准变分最优性条件","source_units":[],"declared_external":true,"declaration_text":["对在闭近端光滑集合Q的邻域内局部Lipschitz且弱凸的函数q，若z局部极小化q(y)+β||y-x||²/2于Q，则存在v∈∂q(z)、w∈N_Q(z)使β(x-z)=v+w；若q在该邻域为L-Lipschitz，则||v||≤L。"]},{"id":"S036","semantic_type":"external_fact","kind":"external_fact","title":"欧氏与概率标准工具","source_units":[],"declared_external":true,"declaration_text":["使用Cauchy–Schwarz不等式、E[Z]≤sqrt(E[Z²])、条件期望塔式法则，以及对离散随机指标I的恒等式E[h(X_I)]=Σ_t P(I=t)E[h(X_t)]，其中输出指标按给定权重独立抽取。"]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S014","edges":[{"source_id":"S001","target_id":"S008","kind":"supports"},{"source_id":"S001","target_id":"S031","kind":"supports"},{"source_id":"S002","target_id":"S006","kind":"supports"},{"source_id":"S002","target_id":"S009","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S003","target_id":"S006","kind":"supports"},{"source_id":"S003","target_id":"S018","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S004","target_id":"S012","kind":"supports"},{"source_id":"S005","target_id":"S020","kind":"supports"},{"source_id":"S006","target_id":"S019","kind":"supports"},{"source_id":"S007","target_id":"S018","kind":"supports"},{"source_id":"S007","target_id":"S021","kind":"supports"},{"source_id":"S007","target_id":"S025","kind":"supports"},{"source_id":"S007","target_id":"S027","kind":"supports"},{"source_id":"S008","target_id":"S021","kind":"supports"},{"source_id":"S008","target_id":"S027","kind":"supports"},{"source_id":"S008","target_id":"S030","kind":"supports"},{"source_id":"S008","target_id":"S031","kind":"supports"},{"source_id":"S009","target_id":"S012","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S011","target_id":"S018","kind":"supports"},{"source_id":"S011","target_id":"S025","kind":"supports"},{"source_id":"S012","target_id":"S018","kind":"supports"},{"source_id":"S012","target_id":"S019","kind":"supports"},{"source_id":"S012","target_id":"S020","kind":"supports"},{"source_id":"S012","target_id":"S021","kind":"supports"},{"source_id":"S012","target_id":"S025","kind":"supports"},{"source_id":"S012","target_id":"S027","kind":"supports"},{"source_id":"S013","target_id":"S021","kind":"supports"},{"source_id":"S013","target_id":"S030","kind":"supports"},{"source_id":"S013","target_id":"S031","kind":"supports"},{"source_id":"S015","target_id":"S019","kind":"supports"},{"source_id":"S015","target_id":"S021","kind":"supports"},{"source_id":"S015","target_id":"S025","kind":"supports"},{"source_id":"S015","target_id":"S028","kind":"supports"},{"source_id":"S015","target_id":"S030","kind":"supports"},{"source_id":"S015","target_id":"S032","kind":"supports"},{"source_id":"S016","target_id":"S030","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S018","target_id":"S028","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S017","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S017","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S017","kind":"supports"},{"source_id":"S022","target_id":"S028","kind":"supports"},{"source_id":"S022","target_id":"S017","kind":"supports"},{"source_id":"S023","target_id":"S028","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S027","target_id":"S024","kind":"supports"},{"source_id":"S028","target_id":"S017","kind":"supports"},{"source_id":"S029","target_id":"S014","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S032","target_id":"S029","kind":"supports"},{"source_id":"S033","target_id":"S006","kind":"supports"},{"source_id":"S034","target_id":"S007","kind":"supports"},{"source_id":"S035","target_id":"S018","kind":"supports"},{"source_id":"S036","target_id":"S019","kind":"supports"},{"source_id":"S036","target_id":"S021","kind":"supports"},{"source_id":"S036","target_id":"S031","kind":"supports"},{"source_id":"S036","target_id":"S032","kind":"supports"}]}}
{"record_id":"2003.01666v2","paper":{"title":"General convergence analysis of stochastic first order methods for composite optimization","authors":["Ion Necoara"],"arxiv_id":"2003.01666","version":"v2","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-021-01821-2","abstract_url":"https://arxiv.org/abs/2003.01666v2","pdf_url":"https://arxiv.org/pdf/2003.01666v2"},"context":{"raw_tex":"\\noindent Let   $f, g \\colon \\rset^m \\times \\Omega \\to \\bar{\\rset}$ be  two proper convex\n functions in the first argument, where $\\Omega$ is endowed with a\nprobability distribution $\\Prob$.  Then, we consider the following\ngeneral composite stochastic optimization problem:\n\\begin{align}\n\\label{eq_problem}\nF^* & = \\min_{x \\in \\rset^m} F(x) \\quad \\left( \\eqdef f(x) + g(x) \\right),\n\\end{align}\nwhere the two functions have stochastic representations in the form of expectation, i.e.  $f(x) = \\Exp{f(x, \\xi)}$ and $ g(x) = \\Exp{g(x, \\xi)}$. We   assume that the expectation taken with respect to the random variable $\\xi \\in \\Omega$ is finite\nfor all $x \\in \\textnormal{dom} \\ F$. Hence we consider a flexible splitting of our stochastic objective function $F(x) =   \\Exp{F(x, \\xi)}$ in  composite form  as a sum of two  terms: \n$$ F(x,\\xi) = f(x,\\xi) + g(x,\\xi). $$   In this paper we assume $f(\\cdot,\\xi)$ either differentiable or nondifferentiable function and we use,  with some abuse of notation, the same notation for the gradient or subgradient of $f(\\cdot,\\xi)$ at $x$, that is $\\nabla f(x,\\xi) \\in \\partial f(x, \\xi)$, where $\\partial f(x,\\xi)$ is either a singleton or a nonempty convex set. The other term, $g(x,\\xi)$,  is assumed to be simple and its structure known.  We usually refer to $g$ as the regularizer and use the same notation as before  $\\nabla g(x,\\xi) \\in \\partial g(x, \\xi)$ for a (sub)gradient of $g(x, \\xi)$ at $x$.  We  also assume in the sequel that to have access to either  an unbiased\nstochastic estimates of the (sub)gradients of function $f$, i.e.:\n\\[ \\mathbb{E}[\\nabla f(x,\\xi)] \\in \\partial f(x), \\;\\; \\text{where}  \\;\\; \\nabla f(x,\\xi) \\in \\partial f(x,\\xi),  \\] or to\nstochastic estimates of the proximal operator of $f$, i.e.:\n\\[   \\prox_{\\gamma f(\\cdot,\\xi)}(x) = \\arg\n\\min\\limits_{y \\in \\rset^m} f(y,\\xi) + \\frac{1}{2 \\gamma} \\|y\n-x\\|^2.\\]\nWe refer to $\\prox_{\\gamma f (\\cdot,\\xi)}(x)$ as the proximal operator of the\nfunction $f(\\cdot,\\xi)$ at $x$ with stepsize $\\gamma$.  We  always assume to have access  to  stochastic estimates of the proximal operator of $g$, i.e. $ \\prox_{\\gamma g(\\cdot,\\xi)}(x) $. These are  key assumptions in our stochastic optimization to scale up the numerical algorithms. Template \\eqref{eq_problem}\n\n\n\\noindent Let us denote by $X^*$ the optimal set of the convex problem \\eqref{eq_problem} and for any $x \\in \\rset^m$ we denote its projection onto $X^*$ by $\\overline{x}$, that is $\\overline{x} = \\Pi_{X^*}(x)$. In this paper we consider additionally the following assumptions:\n\\begin{assumption}\n\\label{assumption1}\nThe stochastic (sub)gradients\nof $F$ satisfy a stochastic bounded gradient condition restricted on\nany segment $[x, \\; \\overline{x}]$, that is there exist nonnegative\nconstants  $L \\geq 0$ and $B \\geq 0$ such that:\n\\begin{equation}\n\\label{as:main1_spg} B^2 +  L(F(x) - F(\\overline{x})) \\geq\n\\E_{\\xi}[\\| \\nabla F(x,\\xi)  \\|^2 ] \\quad  \\forall x \\in \\textnormal{dom} \\ F.\n\\end{equation}\n\\end{assumption}\n\n\\begin{assumption}\n\\label{assumption2}\nThe function  $F$ satisfies a quadratic functional growth condition restricted on\nany segment $[x, \\; \\overline{x}]$, that is there exists  $\\mu \\geq 0$ such that:\n\\begin{equation}\n\\label{as:strong_spg} F(x) - F(\\overline{x}) \\geq  \\frac{\\mu}{2} \\|x\n- \\overline{x}\\|^2 \\quad \\forall x \\in \\textnormal{dom} \\ F.\n\\end{equation}\n\\end{assumption}\n\n\\begin{algorithm}\nLet $(\\xi_t)_{t \\in \\nset}$ be an i.i.d sequence and $x_0 \\in\n\\rset^m$. Iterate:\n\\begin{equation}\n\\label{eq_spgm}  \\nabla f(x_t,\\xi_t) \\in\\partial f(x_t,\\xi_t) \\quad  \\text{and} \\quad x_{t+1} = \\prox_{\\gamma_t g(\\cdot,\\xi_t)}(x_t - \\gamma_t \\nabla f(x_t,\\xi_t) ),\n\\end{equation}\nwhere $(\\gamma_t)_{t \\in \\nset}$ is a strictly positive sequence of\nstepsizes.\n\\end{algorithm}\n\ngradient mapping (for simplicity we omit its dependence on stepsize~$\\gamma$):\n\\[  \\mathcal{G}(x;\\xi) = \\gamma^{-1} \\left( x - \\prox_{\\gamma g(\\cdot,\\xi)}(x - \\gamma {\\nabla} f(x,\\xi)) \\right). \\]\nThen, it follows immediately that  the previous stochastic proximal\ngradient iteration can  be written as:\n\\[ x_{t+1} = x_t - \\gamma_t \\mathcal{G}(x_t;\\xi_t).  \\]\nMoreover, from  optimality condition of prox operator there exists\n$\\nabla g(x_{t+1},\\xi_t) \\in \\partial g(x_{t+1},\\xi_t)$ such that:\n\\[  \\mathcal{G}(x_t;\\xi_t) =  \\nabla f(x_t,\\xi_t)  + \\nabla g(x_{t+1},\\xi_t).  \\]\n\n\n\\noindent Denote  also $\\xi_{[t]} = \\{\\xi_0,\\ldots, \\xi_{t}\\} $.  The next theorem provides a descent property for  SPG and for the  proof we use as main tool  stochastic\ngradient mapping~$\\mathcal{G}(\\cdot)$.\n\n\n\\begin{algorithm}\nLet $(\\xi_t)_{t \\in \\nset}$ be an i.i.d sequence, and $x_0 \\in\n\\rset^m$. Iterate:\n\\begin{equation}\n\\label{eq_sppm}  x_{t+1/2} = \\prox_{\\gamma_t\nf(\\cdot,\\xi_t)}(x_t) \\quad \\text{and} \\quad x_{t+1}  =\n\\prox_{\\gamma_t g}(x_{t+1/2}),\n\\end{equation}\nwhere $(\\gamma_t)_{t \\in \\nset}$ is a strictly positive sequence of\nstepsizes.\n\\end{algorithm}\n\n\\noindent Let us denote $R_t^2 = \\E_{\\xi_{[t-1]}} [\\|x_t -\n\\overline{x}_{t}\\|^2]$. It follows that $R_0= \\|x_0 -\n\\overline{x}_{0}\\|^2$.  From Theorem \\ref{th:spg_basic} for SPG and Theorem \\ref{th:spp_basic} for SPP, we can derive various convergence rates for the stochastic first order (SFO) methods (i.e. SPG, SPP) depending on  the values taken by the constants $B, L$ and $\\mu$ and also depending on the choice of the stepsize: constant or variable. Our convergence results recover, complement or extend the previous convergence rates for stochastic gradient descent (SGD) to more general  functions $f(\\cdot,\\xi)$  and  $g(\\cdot,\\xi)$.  To derive the convergence rates for the two SFO algorithms derived in the previous sections,  first we can notice that the descent relations from  Theorem \\ref{th:spg_basic}  and Theorem \\ref{th:spp_basic} can be written compactly as:\n\\begin{align}\n\\label{sfo_basic} \n& \\Exp{\\|x_{t+1} - \\overline{x}_{t+1} \\|^2} \\leq   \\Exp{\\| x_t - \\overline{x}_{t} \\|^2}   - \\gamma_t (2 - \\gamma_t L) \\Exp{ F(x_t) - F(\\overline{x}_{t})}  + \\gamma_t^2 {\\cal B}^2, \n\\end{align}\nwhere \n\\begin{equation*}\n{\\cal B}^2  \\leq \\left\\{ \\begin{array}{lr}\nB^2 & \\text{if}  \\quad \\;  \\text{SPG} \\\\\nB^2+B_g^2  & \\;\\; \\text{if}  \\quad \\text{SPP}.\n\\end{array}\\right.\n\\end{equation*}\nBased on this common descent property  of our stochastic first order methods, for  the convex case (i.e. $\\mu=0$) and constant stepsize we get the following convergence result (we make the convention $2/0 =\\infty$):\n","units":[{"start":9349,"end":11623,"start_line":133,"end_line":151,"raw_tex":"\\noindent Let   $f, g \\colon \\rset^m \\times \\Omega \\to \\bar{\\rset}$ be  two proper convex\n functions in the first argument, where $\\Omega$ is endowed with a\nprobability distribution $\\Prob$.  Then, we consider the following\ngeneral composite stochastic optimization problem:\n\\begin{align}\n\\label{eq_problem}\nF^* & = \\min_{x \\in \\rset^m} F(x) \\quad \\left( \\eqdef f(x) + g(x) \\right),\n\\end{align}\nwhere the two functions have stochastic representations in the form of expectation, i.e.  $f(x) = \\Exp{f(x, \\xi)}$ and $ g(x) = \\Exp{g(x, \\xi)}$. We   assume that the expectation taken with respect to the random variable $\\xi \\in \\Omega$ is finite\nfor all $x \\in \\textnormal{dom} \\ F$. Hence we consider a flexible splitting of our stochastic objective function $F(x) =   \\Exp{F(x, \\xi)}$ in  composite form  as a sum of two  terms: \n$$ F(x,\\xi) = f(x,\\xi) + g(x,\\xi). $$   In this paper we assume $f(\\cdot,\\xi)$ either differentiable or nondifferentiable function and we use,  with some abuse of notation, the same notation for the gradient or subgradient of $f(\\cdot,\\xi)$ at $x$, that is $\\nabla f(x,\\xi) \\in \\partial f(x, \\xi)$, where $\\partial f(x,\\xi)$ is either a singleton or a nonempty convex set. The other term, $g(x,\\xi)$,  is assumed to be simple and its structure known.  We usually refer to $g$ as the regularizer and use the same notation as before  $\\nabla g(x,\\xi) \\in \\partial g(x, \\xi)$ for a (sub)gradient of $g(x, \\xi)$ at $x$.  We  also assume in the sequel that to have access to either  an unbiased\nstochastic estimates of the (sub)gradients of function $f$, i.e.:\n\\[ \\mathbb{E}[\\nabla f(x,\\xi)] \\in \\partial f(x), \\;\\; \\text{where}  \\;\\; \\nabla f(x,\\xi) \\in \\partial f(x,\\xi),  \\] or to\nstochastic estimates of the proximal operator of $f$, i.e.:\n\\[   \\prox_{\\gamma f(\\cdot,\\xi)}(x) = \\arg\n\\min\\limits_{y \\in \\rset^m} f(y,\\xi) + \\frac{1}{2 \\gamma} \\|y\n-x\\|^2.\\]\nWe refer to $\\prox_{\\gamma f (\\cdot,\\xi)}(x)$ as the proximal operator of the\nfunction $f(\\cdot,\\xi)$ at $x$ with stepsize $\\gamma$.  We  always assume to have access  to  stochastic estimates of the proximal operator of $g$, i.e. $ \\prox_{\\gamma g(\\cdot,\\xi)}(x) $. These are  key assumptions in our stochastic optimization to scale up the numerical algorithms. Template \\eqref{eq_problem}\n","sha256_utf8":"bd3f13b9e8c1fc51d0f4d55709fce93ef627ca73e7ab0441b16ed1a0b06162d3"},{"start":11896,"end":12614,"start_line":162,"end_line":173,"raw_tex":"\\noindent Let us denote by $X^*$ the optimal set of the convex problem \\eqref{eq_problem} and for any $x \\in \\rset^m$ we denote its projection onto $X^*$ by $\\overline{x}$, that is $\\overline{x} = \\Pi_{X^*}(x)$. In this paper we consider additionally the following assumptions:\n\\begin{assumption}\n\\label{assumption1}\nThe stochastic (sub)gradients\nof $F$ satisfy a stochastic bounded gradient condition restricted on\nany segment $[x, \\; \\overline{x}]$, that is there exist nonnegative\nconstants  $L \\geq 0$ and $B \\geq 0$ such that:\n\\begin{equation}\n\\label{as:main1_spg} B^2 +  L(F(x) - F(\\overline{x})) \\geq\n\\E_{\\xi}[\\| \\nabla F(x,\\xi)  \\|^2 ] \\quad  \\forall x \\in \\textnormal{dom} \\ F.\n\\end{equation}\n\\end{assumption}","sha256_utf8":"a5f1290082ef3a9d0c6f5aac1e5394fa2f76099c6caee63ba4c23caf21ab7b0b"},{"start":12616,"end":12997,"start_line":175,"end_line":183,"raw_tex":"\\begin{assumption}\n\\label{assumption2}\nThe function  $F$ satisfies a quadratic functional growth condition restricted on\nany segment $[x, \\; \\overline{x}]$, that is there exists  $\\mu \\geq 0$ such that:\n\\begin{equation}\n\\label{as:strong_spg} F(x) - F(\\overline{x}) \\geq  \\frac{\\mu}{2} \\|x\n- \\overline{x}\\|^2 \\quad \\forall x \\in \\textnormal{dom} \\ F.\n\\end{equation}\n\\end{assumption}","sha256_utf8":"4608b77f4f69e960098b71991bed363f9b45a09205558f92f4e045cdbd8e3f6e"},{"start":24019,"end":24410,"start_line":293,"end_line":301,"raw_tex":"\\begin{algorithm}\nLet $(\\xi_t)_{t \\in \\nset}$ be an i.i.d sequence and $x_0 \\in\n\\rset^m$. Iterate:\n\\begin{equation}\n\\label{eq_spgm}  \\nabla f(x_t,\\xi_t) \\in\\partial f(x_t,\\xi_t) \\quad  \\text{and} \\quad x_{t+1} = \\prox_{\\gamma_t g(\\cdot,\\xi_t)}(x_t - \\gamma_t \\nabla f(x_t,\\xi_t) ),\n\\end{equation}\nwhere $(\\gamma_t)_{t \\in \\nset}$ is a strictly positive sequence of\nstepsizes.\n\\end{algorithm}","sha256_utf8":"1b80c71d74faf5252dbc6a23f357a6e10f2bfd386360b9234b91f186d8e7ae8b"},{"start":24798,"end":25374,"start_line":308,"end_line":315,"raw_tex":"gradient mapping (for simplicity we omit its dependence on stepsize~$\\gamma$):\n\\[  \\mathcal{G}(x;\\xi) = \\gamma^{-1} \\left( x - \\prox_{\\gamma g(\\cdot,\\xi)}(x - \\gamma {\\nabla} f(x,\\xi)) \\right). \\]\nThen, it follows immediately that  the previous stochastic proximal\ngradient iteration can  be written as:\n\\[ x_{t+1} = x_t - \\gamma_t \\mathcal{G}(x_t;\\xi_t).  \\]\nMoreover, from  optimality condition of prox operator there exists\n$\\nabla g(x_{t+1},\\xi_t) \\in \\partial g(x_{t+1},\\xi_t)$ such that:\n\\[  \\mathcal{G}(x_t;\\xi_t) =  \\nabla f(x_t,\\xi_t)  + \\nabla g(x_{t+1},\\xi_t).  \\]\n","sha256_utf8":"5a592a3e3392098e152ff6629d90ebb7b62244985638df67756ffd61b530d4bc"},{"start":25376,"end":25586,"start_line":318,"end_line":319,"raw_tex":"\\noindent Denote  also $\\xi_{[t]} = \\{\\xi_0,\\ldots, \\xi_{t}\\} $.  The next theorem provides a descent property for  SPG and for the  proof we use as main tool  stochastic\ngradient mapping~$\\mathcal{G}(\\cdot)$.\n","sha256_utf8":"cf75351f6a67d125c52a9a431edd29db8a3775937fdc26457a6fd7bbcc2e98b7"},{"start":31618,"end":31975,"start_line":414,"end_line":424,"raw_tex":"\\begin{algorithm}\nLet $(\\xi_t)_{t \\in \\nset}$ be an i.i.d sequence, and $x_0 \\in\n\\rset^m$. Iterate:\n\\begin{equation}\n\\label{eq_sppm}  x_{t+1/2} = \\prox_{\\gamma_t\nf(\\cdot,\\xi_t)}(x_t) \\quad \\text{and} \\quad x_{t+1}  =\n\\prox_{\\gamma_t g}(x_{t+1/2}),\n\\end{equation}\nwhere $(\\gamma_t)_{t \\in \\nset}$ is a strictly positive sequence of\nstepsizes.\n\\end{algorithm}","sha256_utf8":"6bee7346941dbc627c834167655a721a2a5776168f05975e3128a4226c65a6bd"},{"start":36657,"end":38168,"start_line":504,"end_line":518,"raw_tex":"\\noindent Let us denote $R_t^2 = \\E_{\\xi_{[t-1]}} [\\|x_t -\n\\overline{x}_{t}\\|^2]$. It follows that $R_0= \\|x_0 -\n\\overline{x}_{0}\\|^2$.  From Theorem \\ref{th:spg_basic} for SPG and Theorem \\ref{th:spp_basic} for SPP, we can derive various convergence rates for the stochastic first order (SFO) methods (i.e. SPG, SPP) depending on  the values taken by the constants $B, L$ and $\\mu$ and also depending on the choice of the stepsize: constant or variable. Our convergence results recover, complement or extend the previous convergence rates for stochastic gradient descent (SGD) to more general  functions $f(\\cdot,\\xi)$  and  $g(\\cdot,\\xi)$.  To derive the convergence rates for the two SFO algorithms derived in the previous sections,  first we can notice that the descent relations from  Theorem \\ref{th:spg_basic}  and Theorem \\ref{th:spp_basic} can be written compactly as:\n\\begin{align}\n\\label{sfo_basic} \n& \\Exp{\\|x_{t+1} - \\overline{x}_{t+1} \\|^2} \\leq   \\Exp{\\| x_t - \\overline{x}_{t} \\|^2}   - \\gamma_t (2 - \\gamma_t L) \\Exp{ F(x_t) - F(\\overline{x}_{t})}  + \\gamma_t^2 {\\cal B}^2, \n\\end{align}\nwhere \n\\begin{equation*}\n{\\cal B}^2  \\leq \\left\\{ \\begin{array}{lr}\nB^2 & \\text{if}  \\quad \\;  \\text{SPG} \\\\\nB^2+B_g^2  & \\;\\; \\text{if}  \\quad \\text{SPP}.\n\\end{array}\\right.\n\\end{equation*}\nBased on this common descent property  of our stochastic first order methods, for  the convex case (i.e. $\\mu=0$) and constant stepsize we get the following convergence result (we make the convention $2/0 =\\infty$):\n","sha256_utf8":"dabac4d7836045f069f2dd3eb03a54b45f720d37789984817a047c004aa09665"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem} [Sufficient conditions for linear convergence of SFO]\n\\label{th:spg_muL} Let assumptions of Theorem \\ref{th:spg_basic} for SPG and  Theorem \\ref{th:spp_basic} for SPP hold. Additionally, we  assume that the quadratic functional growth condition from  Assumption \\ref{assumption2} holds  and we choose constant stepsize  $\\gamma_t = \\gamma \\in (0,2/L)$. Then, we get the following upper bounds for the square distance of the SFO iterates to the solution set:\n\\begin{equation*}\n\\Exp{\\|x_{t} - \\overline{x}_{t} \\|^2} \\!\\leq\\!\\! \\left\\{ \\!\\begin{array}{lr}\n\\left| 1 - \\mu \\gamma + \\frac{\\mu L \\gamma^2}{2} \\right |^t  \\!\\!R_0^2 +\\!\n\\frac{{\\cal B}^2 \\gamma^2}{1- |1-\\mu \\gamma  + \\mu L \\gamma^2/2|}  \\!\\!\\!\\!\\!\\!\\!\\! & \\text{if}  \\; \n1 \\!-\\! \\mu \\gamma + \\frac{\\mu L \\gamma^2}{2} \\!>\\! -1 \\\\\n{\\cal B}^2 \\gamma^2  & \\text{if}  \\quad  1 - \\mu \\gamma + \\frac{\\mu L\n\\gamma^2}{2} \\leq -1.\n\\end{array}\\right.\n\\end{equation*}\nMoreover, if the condition $\\mu < 4L$ holds or  the constant stepsize satisfies $\\gamma < \\min\\left({2}/{L}, {2}/{\\mu} \\right)$, then  $|  1 - \\mu \\gamma + \\frac{\\mu L\\gamma^2}{2}| < 1$ and therefore we get linear converge to a noise dominated region with radius  proportional to the square of the stepsize $\\gamma$:\n\\[ \\Exp{\\|x_{t} - \\overline{x}_{t} \\|^2} \\leq  \\left| 1 - \\mu \\gamma + \\frac{\\mu L \\gamma^2}{2} \\right|^t  R_0^2 +\n\\frac{{\\cal B}^2 \\gamma^2}{1- |1-\\mu \\gamma  + \\mu L \\gamma^2/2|}. \\]\nConsequently, if ${\\cal B}=0$, then we get pure linear convergence:\n\\begin{equation*}\n\\Exp{\\|x_{t} - \\overline{x}_{t} \\|^2} \\leq \\left| 1 - \\mu \\gamma + \\frac{\\mu L \\gamma^2}{2} \\right |^t  R_0^2.\n\\end{equation*}\n\\end{theorem}","units":[{"start":43863,"end":45521,"start_line":599,"end_line":617,"raw_tex":"\\begin{theorem} [Sufficient conditions for linear convergence of SFO]\n\\label{th:spg_muL} Let assumptions of Theorem \\ref{th:spg_basic} for SPG and  Theorem \\ref{th:spp_basic} for SPP hold. Additionally, we  assume that the quadratic functional growth condition from  Assumption \\ref{assumption2} holds  and we choose constant stepsize  $\\gamma_t = \\gamma \\in (0,2/L)$. Then, we get the following upper bounds for the square distance of the SFO iterates to the solution set:\n\\begin{equation*}\n\\Exp{\\|x_{t} - \\overline{x}_{t} \\|^2} \\!\\leq\\!\\! \\left\\{ \\!\\begin{array}{lr}\n\\left| 1 - \\mu \\gamma + \\frac{\\mu L \\gamma^2}{2} \\right |^t  \\!\\!R_0^2 +\\!\n\\frac{{\\cal B}^2 \\gamma^2}{1- |1-\\mu \\gamma  + \\mu L \\gamma^2/2|}  \\!\\!\\!\\!\\!\\!\\!\\! & \\text{if}  \\; \n1 \\!-\\! \\mu \\gamma + \\frac{\\mu L \\gamma^2}{2} \\!>\\! -1 \\\\\n{\\cal B}^2 \\gamma^2  & \\text{if}  \\quad  1 - \\mu \\gamma + \\frac{\\mu L\n\\gamma^2}{2} \\leq -1.\n\\end{array}\\right.\n\\end{equation*}\nMoreover, if the condition $\\mu < 4L$ holds or  the constant stepsize satisfies $\\gamma < \\min\\left({2}/{L}, {2}/{\\mu} \\right)$, then  $|  1 - \\mu \\gamma + \\frac{\\mu L\\gamma^2}{2}| < 1$ and therefore we get linear converge to a noise dominated region with radius  proportional to the square of the stepsize $\\gamma$:\n\\[ \\Exp{\\|x_{t} - \\overline{x}_{t} \\|^2} \\leq  \\left| 1 - \\mu \\gamma + \\frac{\\mu L \\gamma^2}{2} \\right|^t  R_0^2 +\n\\frac{{\\cal B}^2 \\gamma^2}{1- |1-\\mu \\gamma  + \\mu L \\gamma^2/2|}. \\]\nConsequently, if ${\\cal B}=0$, then we get pure linear convergence:\n\\begin{equation*}\n\\Exp{\\|x_{t} - \\overline{x}_{t} \\|^2} \\leq \\left| 1 - \\mu \\gamma + \\frac{\\mu L \\gamma^2}{2} \\right |^t  R_0^2.\n\\end{equation*}\n\\end{theorem}","sha256_utf8":"855a7a60458b49302bb8f09134c246acc7367bd1ad94d74a0f1117b47858836d"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{theorem}\n\\label{th:spg_basic} Let  $f(\\cdot, \\xi)$ and $g(\\cdot, \\xi)$ be  convex functions such that $g$ admits an easily computable proximal operator. Additionally,  assume that the stochastic bounded gradient condition from   Assumption \\ref{assumption1} holds. Then, for any $t \\geq 0$ and  stepsize $\\gamma_t >0$, we have the following recursion  for the   SPG iteration:\n\\begin{align}\n\\label{spg_basic} & \\Exp{\\|x_{t+1} - \\overline{x}_{t+1} \\|^2}  \\leq\n\\Exp{\\| x_t - \\overline{x}_{t} \\|^2} -  \\gamma_t  (2 - \\gamma_t L)\n\\Exp{ F(x_t) - F(\\overline{x}_{t})} + \\gamma_t^2 B^2.\n\\end{align}\n\\end{theorem}\n\n\\begin{proof}\nFrom the definition of SPG iteration we\nhave:\n\\begin{align}\n\\label{der_spg_basic}\n& \\| x_{t+1} - \\overline{x}_{t+1} \\|^2  \\leq   \\| x_{t+1} - \\overline{x}_{t} \\|^2 =  \\| x_{t} - \\overline{x}_{t} - \\gamma_t \\mathcal{G}(x_t;\\xi_t)\\|^2 \\\\\n& =  \\| x_{t} - \\overline{x}_{t} \\|^2  - 2 \\gamma_t \\langle \\mathcal{G}(x_t;\\xi_t), x_t  - \\overline{x}_{t} \\rangle + \\gamma_t^2 \\| \\mathcal{G}(x_t;\\xi_t) \\|^2  \\nonumber  \\\\\n& = \\| x_{t} - \\overline{x}_{t} \\|^2  - 2 \\gamma_t \\langle \\nabla\nf(x_t,\\xi_t) + \\nabla g(x_{t+1},\\xi_t), x_t  - \\overline{x}_{t} \\rangle +\n\\gamma_t^2 \\| \\mathcal{G}(x_t;\\xi_t) \\|^2.\\nonumber \n\\end{align}\nNow, we refine the second term. First, from convexity of $f$ we\nhave:\n\\[ \\langle \\nabla f(x_t,\\xi_t), x_t  - \\overline{x}_{t} \\rangle \\geq f(x_t,\\xi_t) - f(\\overline{x}_{t},\\xi_t). \\]\nThen, from  convexity of $g$ and the definition of  stochastic\ngradient mapping $\\mathcal{G}(\\cdot)$, we have:\n\\begin{align*}\n&\\langle \\nabla g(x_{t+1},\\xi_t), x_t  - \\overline{x}_{t} \\rangle  = \\langle  \\nabla g(x_{t+1},\\xi_t), x_t  - x_{t+1} \\rangle + \\langle  \\nabla g(x_{t+1},\\xi_t), x_{t+1}  - \\overline{x}_{t} \\rangle \\\\\n& \\geq \\gamma_t \\| \\mathcal{G}(x_t;\\xi_t) \\|^2 - \\gamma_t \\langle \\nabla f(x_t,\\xi_t), \\mathcal{G}(x_t;\\xi_t) \\rangle + g(x_{t+1}, \\xi_t) - g(\\overline{x}_t,\\xi_t)\\\\\n& \\geq \\gamma_t \\| \\mathcal{G}(x_t;\\xi_t) \\|^2 - \\gamma_t \\langle\n\\nabla f(x_t,\\xi_t) + \\nabla g(x_t,\\xi_t), \\mathcal{G}(x_t;\\xi_t) \\rangle +\ng(x_{t}, \\xi_t) - g(\\overline{x}_t, \\xi_t).\n\\end{align*}\nReplacing the previous two inequalities in \\eqref{der_spg_basic}, we\nobtain:\n\\begin{align*}\n\\| x_{t+1} - \\overline{x}_{t+1} \\|^2 & \\leq  \\| x_{t} -\n\\overline{x}_{t} \\|^2\n - 2 \\gamma_t  \\left( f(x_t,\\xi_t) + g(x_t, \\xi_t) - f(\\overline{x}_{t},\\xi_t) - g(\\overline{x}_{t}, \\xi_t)\\right)  \\\\\n& \\quad + 2 \\gamma_t^2 \\langle \\nabla f(x_t,\\xi_t) + \\nabla g(x_t, \\xi_t),\n\\mathcal{G}(x_t;\\xi_t) \\rangle - \\gamma_t^2 \\|\n\\mathcal{G}(x_t;\\xi_t) \\|^2.\n\\end{align*}\nSince $2\\langle u , v \\rangle - \\|v\\|^2 \\leq \\|u\\|^2$ for\nall $v \\in \\rset^n$ and using that $ F(x_t,\\xi_t)  =  f(x_t,\\xi_t) +  g(x_t, \\xi_t)$ and $\\nabla F(x_t,\\xi_t)  = \\nabla f(x_t,\\xi_t) + \\nabla g(x_t, \\xi_t)$, we further get:\n\\begin{align*}\n\\| x_{t+1} - \\overline{x}_{t+1} \\|^2 & \\leq \\| x_{t} -\n\\overline{x}_{t} \\|^2\n - 2 \\gamma_t \\left(  F(x_t,\\xi_t)  - F(\\overline{x}_{t},\\xi_t) \\right)  +  \\gamma_t^2 \\| \\nabla F(x_t,\\xi_t)\\|^2.\n\\end{align*}\nNote that $x_t$ depends only on history $\\xi_{[t-1]} =\n\\{\\xi_0,\\ldots, \\xi_{t-1}\\} $, not $\\xi_t$. It follows from the\nbasic property of  conditional expectation that:\n\\begin{align*}\n& \\E_{\\xi_{[t]}} [\\|x_{t+1}  - \\overline{x}_{t+1}\\|^2] \\\\\n& \\leq \\E_{\\xi_{[t-1]}}[\\|x_{t} - \\overline{x}_{t}\\|^2]  - 2 \\gamma_t  \\E_{\\xi_{[t-1]}}\n\\left[ \\E_{\\xi_t} [ F(x_t,\\xi_t) - F(\\overline{x}_{t},\\xi_t)  | \\xi_{[t-1]}\n]\\right] \\\\\n& \\qquad +  \\gamma_t^2 \\E_{\\xi_{[t]}}  \\left[\\| \\nabla F(x_t,\\xi_t) \\|^2 ] \\right]\\\\\n& =  \\E_{\\xi_{[t-1]}}[\\|x_{t} \\!- \\overline{x}_{t}\\|^2]  - 2\n\\gamma_t \\E_{\\xi_{[t-1]}} \\left[ F(x_t) - F(\\overline{x}_{t})\\right]\n+ \\gamma_t^2 \\E_{\\xi_{[t]}}  \\left[\\| \\nabla F(x_t,\\xi_t) \\|^2 \\right].\n\\end{align*}\nFurther, making use of the stochastic bounded gradient condition given in  Assumption\n\\ref{assumption1} in the previous relation we get:\n\\begin{align*}\n& \\E_{\\xi_{[t]}} [\\|x_{t+1}  - \\overline{x}_{t+1}\\|^2] \\\\\n& \\leq \\E_{\\xi_{[t-1]}}[\\|x_{t} \\!- \\overline{x}_{t}\\|^2]  - 2\n\\gamma_t \\E_{\\xi_{[t-1]}} \\left[ F(x_t) - F(\\overline{x}_{t})\\right]\n+ \\gamma_t^2 \\E_{\\xi_{[t]}}  \\left[\\| \\nabla F(x_t,\\xi_t) \\|^2\n\\right] \\\\\n& \\overset{\\eqref{as:main1_spg}}{\\leq} \\! \\E_{\\xi_{[t-1]}}[\\|x_{t}\n- \\overline{x}_{t}\\|^2]  - 2 \\gamma_t  \\E_{\\xi_{[t-1]}} \\left[\nF(x_t) - F(\\overline{x}_{t})\\right] \\\\\n& \\qquad + \\gamma_t^2 \\left( B^2 + L\n\\E_{\\xi_{[t-1]}} \\!\\left[F(x_t) - F(\\overline{x}_{t})\\right] \\right)\n\\\\\n& = \\E_{\\xi_{[t-1]}}[\\|x_{t} \\!- \\overline{x}_{t}\\|^2]  - \\gamma_t\n(2 - \\gamma_t L) \\E_{\\xi_{[t-1]}} \\left[ F(x_t) -\nF(\\overline{x}_{t})\\right] + \\gamma_t^2 B^2,\n\\end{align*}\nwhich, omitting the dependence  of expectation on $\\xi_{[t]}$,  proves the  statement of the theorem. \\qed\n\\end{proof}\n\nWe first recall that  any strongly convex function $h$ with\nconvexity constant $\\sigma_h$ and having the optimal point $x^*$\nsatisfies the following inequality~\\cite{Nes:04}: $h(x) \\geq  h(x^*) + \\frac{\\sigma_h}{2}\\|x - x^*\\|^2 \\quad \\forall x$. Since for any convex function $h$ and fixed  point $x$, the Moreau smoothing  function  $h_\\gamma(y;x) = h(y) +  \\frac{1}{2 \\gamma} \\|y -x\\|^2$ is\nstrongly convex in the first argument $y$ with strong convexity constant\n$1/\\gamma$, the following holds for all $x, y \\in \\rset^n$:\n\\begin{align}\n\\label{rel1_prox} \n& h(y)  \\!+\\!  \\frac{1}{2 \\gamma} \\|y - x\\|^2   \\!\\geq\\!  h(\\prox_{\\gamma h}(x)) \\!+\\! \\frac{1}{2 \\gamma} \\|\\prox_{\\gamma h}(x)\n- x\\|^2 \\!+\\! \\frac{1}{2 \\gamma} \\|\\prox_{\\gamma h}(x) - y\\|^2. \n\\end{align}\n\n\n\\begin{theorem}\n\\label{th:spp_basic} Let $f(\\cdot, \\xi)$ and $g(\\cdot, \\xi)$ be convex functions  that  admit easily computable proximal operators. Additionally, assume that the  stochastic bounded gradient condition from Assumption \\ref{assumption1} holds and $g(\\cdot, \\xi)$ have bounded subgradients, that is there exists $B_{g}>0$ such that $ \\|\\nabla g(x, \\xi)\\| \\leq B_g$ for all $x$ and $\\xi$. Then, for any stepsize $\\gamma_t>0$ we have the following recursive inequality for the SPP iteration:\n\\begin{align}\n\\label{spp_basic} & \\Exp{\\|x_{t+1} - \\overline{x}_{t+1} \\|^2} \\leq \\\\\n&  \\Exp{\\| x_t - \\overline{x}_{t} \\|^2}   - \\gamma_t (2 - \\gamma_t L) \\Exp{ F(x_t) -\nF(\\overline{x}_{t})}  + \\gamma_t^2 \\left( B^2 + B_{g}^2 \\right). \\nonumber\n\\end{align}\n\\end{theorem}\n\n\\begin{proof}\nUsing \\eqref{rel1_prox} for the function $h=f(\\cdot,\\xi_t)$ with $y\n= \\overline{x}_{t}$ and $x = x_t$ we get:\n\\[  f(\\overline{x}_{t},\\xi_t) + \\frac{1}{2 \\gamma_t} \\|x_t - \\overline{x}_{t}\\|^2 \\geq\nf(x_{t+1/2},\\xi_t) + \\frac{1}{2 \\gamma_t} \\|x_{t+1/2} - x_t\\|^2 +\n\\frac{1}{2 \\gamma_t} \\|x_{t+1/2} - \\overline{x}_{t}\\|^2. \\] Using\nagain \\eqref{rel1_prox} for  function $h=g(\\cdot,\\xi_t)$ with $y = \\overline{x}_{t}$\nand $x = x_{t+1/2}$ we get:\n\\[  g(\\overline{x}_{t},\\xi_t) + \\frac{1}{2 \\gamma_t} \\|x_{t+1/2} - \\overline{x}_{t}\\|^2 \\geq\ng(x_{t+1},\\xi_t) + \\frac{1}{2 \\gamma_t} \\|x_{t+1} - x_{t+1/2}\\|^2 +\n\\frac{1}{2 \\gamma_t} \\|x_{t+1} - \\overline{x}_{t}\\|^2. \\] Adding the\nprevious two inequalities  and using the convexity of $f(\\cdot,\\xi_t)$\nand $g(\\cdot,\\xi_t)$, we get:\n\\begin{align*}\n& f(\\overline{x}_{t},\\xi_t) +  g(\\overline{x}_{t},\\xi_t) + \\frac{1}{2\n\\gamma_t} \\|x_t - \\overline{x}_{t}\\|^2 -\n\\frac{1}{2 \\gamma_t} \\|x_{t+1} - \\overline{x}_{t}\\|^2\\\\\n& \\geq f(x_{t+1/2},\\xi_t) + g(x_{t+1},\\xi_t) + \\frac{1}{2 \\gamma_t}\n\\|x_{t+1/2} - x_t\\|^2 + \\frac{1}{2 \\gamma_t} \\|x_{t+1} -\nx_{t+1/2}\\|^2\n\\\\\n& \\geq  f(x_t,\\xi_t) + g(x_t,\\xi_t)  +  \\langle \\nabla f(x_t,\\xi_t) +\n\\nabla g(x_t,\\xi_t) , x_{t+1/2} - x_t \\rangle \n\\\\\n&\\qquad + \\frac{1}{2 \\gamma_t} \\|x_{t+1/2} -\nx_t\\|^2 + \\langle \\nabla g(x_t,\\xi_t), x_{t+1}\n- x_{t+1/2}  \\rangle + \\frac{1}{2 \\gamma_t} \\|x_{t+1} -\nx_{t+1/2}\\|^2\\\\\n& \\geq f(x_t,\\xi_t) + g(x_t,\\xi_t)  -\\frac{\\gamma_t}{2} \\left( \\| \\nabla\nf(x_t,\\xi_t) + \\nabla g(x_t,\\xi_t) \\|^2 + \\|\\nabla g(x_t,\\xi_t)\\|^2 \\right),\n\\end{align*}\nwhere in the last inequality we used that $\\langle \\alpha,z \\rangle\n + \\frac{1}{2\\gamma} \\|z\\|^2 \\geq - \\frac{\\gamma}{2} \\|\\alpha\\|^2$ for all\n$z$. Taking now  expectation and using that $\\|x_{t+1} -\n\\overline{x}_{t}\\|^2 \\geq \\|x_{t+1} - \\overline{x}_{t+1}\\|^2$ we get:\n\\begin{align*}\n& \\E_{\\xi_{[t]}} [\\|x_{t+1} - \\overline{x}_{t+1} \\|^2] \\leq  \\E_{\\xi_{[t-1]}} [\\| x_t - \\overline{x}_{t} \\|^2   - 2 \\gamma_t  \\E_{\\xi_{[t-1]}}[ F(x_t) - F(\\overline{x}_{t})]  \\\\ \n& \\quad + \\gamma_t^2 \\E_{\\xi_{[t]}} [ \\| \\nabla\nF(x_t,\\xi_t)   \\|^2 + \\| \\nabla g(x_t,\\xi_t) \\|^2 ],\n\\end{align*}\nwhere we used basic properties of the conditional expectation as in the proof of Theorem \\ref{th:spg_basic}. Further, using the stochastic bounded gradient condition \\eqref{as:main1_spg}  and  omitting the dependence  of expectation on $\\xi_{[t]}$, we get  the statement of the theorem.  \\qed\n\\end{proof}\n\n\\begin{proof}\nFrom \\eqref{sfo_basic}  and Assumption \\ref{assumption2}  it follows that:\n\\begin{align*}\n& \\E_{\\xi_{[t]}} [\\|x_{t+1}  - \\overline{x}_{t+1}\\|^2] \\leq\n\\E_{\\xi_{[t-1]}}[\\|x_{t} - \\overline{x}_{t}\\|^2]  -  \\gamma_t (2 -\n\\gamma_t L) \\E_{\\xi_{[t-1]}}\n\\left[F(x_t) - F(\\overline{x}_{t})\\right]  \\\\\n& + {\\cal B}^2 \\gamma_{t}^2  \\overset{\\eqref{as:strong_spg}}{\\leq} \\E_{\\xi_{[t-1]}}[\\|x_{t} -\n\\overline{x}_{t}\\|^2]  - \\frac{\\mu \\gamma_t (2 - \\gamma_t L)}{2}\n\\E_{\\xi_{[t-1]}} [\\|x_{t} - \\overline{x}_{t}\\|^2] +  {\\cal B}^2\n\\gamma_{t}^2,\n\\end{align*}\nprovided that $\\gamma_t \\leq 2/L$.  In conclusion, we obtain the\nfollowing recursion:\n\\begin{align}\n\\label{recursive1_spg} \\E_{\\xi_{[t]}} [\\|x_{t+1} -\n\\overline{x}_{t+1}\\|^2] & \\leq   \\left(1 - \\mu \\gamma_t + \\frac{\\mu\nL \\gamma_t^2}{2} \\right) \\E_{\\xi_{[t-1]}} [\\|x_{t} -\n\\overline{x}_{t}\\|^2] +  {\\cal B}^2 \\gamma_{t}^2.\n\\end{align}\nFor constant stepsize $\\gamma_t = \\gamma < 2/L$ it follows that $ 1 - \\mu \\gamma\n+ \\mu L \\gamma^2/2 < 1$.  Then, combining inequality \\eqref{recursive1_spg} with\ncondition $1 - \\mu \\gamma + \\mu L\n\\gamma^2/2 > -1$, we get:\n\\begin{align*}\n\\E [\\|x_{t} - \\overline{x}_{t}\\|^2] & \\leq   |1 - \\mu \\gamma +\n\\frac{\\mu L \\gamma^2}{2} |^t \\|x_{0} - \\overline{x}_{0}\\|^2 +\n{\\cal B}^2 \\gamma^2 \\sum_{j=0}^{t-1} |1 - \\mu \\gamma +\n\\frac{\\mu L \\gamma^2}{2}|^j \\\\\n& \\leq |1 - \\mu \\gamma + \\frac{\\mu L \\gamma^2}{2} |^t\n\\|x_{0} - \\overline{x}_{0}\\|^2 +    \\frac{{\\cal B}^2 \\gamma^2}{ 1- |1-\\mu \\gamma\n+ \\mu L \\gamma^2/2|},\n\\end{align*}\nwhere in the second inequality we used that $\\sum_{j=0}^{t-1} c^j\n\\leq \\sum_{j=0}^{\\infty} c^j = \\frac{1}{1-c}$ for any $|c|<1$.\nOtherwise, if $1 - \\mu \\gamma + \\mu L \\gamma^2/2 \\leq -1$, then from\n\\eqref{recursive1_spg} it follows that:\n\\begin{align*} \\E [\\|x_{t} - \\overline{x}_{t}\\|^2] &\n\\leq {\\cal B}^2 \\gamma^2.\n\\end{align*}\nMoreover, if $\\mu < 4L$, then the relation $1 - \\mu \\gamma + \\mu L\n\\gamma^2/2 > -1$ always holds, regardless of the value of $\\gamma$. Thus, if $\\mu < 4L$ and  the constant stepsize is chosen such $\\gamma < \\frac{2}{L}$,  then $|1 - \\mu \\gamma + \\mu L \\gamma^2| < 1$. Similarly, if  the  constant stepsize satisfies  $\\gamma < \\min(2/L, 2/\\mu)$, then again we have $0< 1 - \\mu \\gamma + \\mu L \\gamma^2 < 1$. Therefore, these two previous conditions guarantee linear convergence  to a noise dominated region whose radius is proportional to the square of the stepsize $\\gamma$.    Finally, the inequality \\eqref{recursive1_spg} with ${\\cal B}=0$ and $|1 - \\mu \\gamma + \\mu L \\gamma^2| < 1$  proves  linear convergence for $\\Exp{\\|x_{t} - \\overline{x}_{t} \\|^2}$, with $x_t$ generated by SFO methods.  \\qed\n\\end{proof}","units":[{"start":25587,"end":26199,"start_line":321,"end_line":328,"raw_tex":"\\begin{theorem}\n\\label{th:spg_basic} Let  $f(\\cdot, \\xi)$ and $g(\\cdot, \\xi)$ be  convex functions such that $g$ admits an easily computable proximal operator. Additionally,  assume that the stochastic bounded gradient condition from   Assumption \\ref{assumption1} holds. Then, for any $t \\geq 0$ and  stepsize $\\gamma_t >0$, we have the following recursion  for the   SPG iteration:\n\\begin{align}\n\\label{spg_basic} & \\Exp{\\|x_{t+1} - \\overline{x}_{t+1} \\|^2}  \\leq\n\\Exp{\\| x_t - \\overline{x}_{t} \\|^2} -  \\gamma_t  (2 - \\gamma_t L)\n\\Exp{ F(x_t) - F(\\overline{x}_{t})} + \\gamma_t^2 B^2.\n\\end{align}\n\\end{theorem}","sha256_utf8":"30f718d3ba227601740b986edfab99ab940ad1b4c8cbcd4466f01f431e86216a"},{"start":26201,"end":30316,"start_line":330,"end_line":402,"raw_tex":"\\begin{proof}\nFrom the definition of SPG iteration we\nhave:\n\\begin{align}\n\\label{der_spg_basic}\n& \\| x_{t+1} - \\overline{x}_{t+1} \\|^2  \\leq   \\| x_{t+1} - \\overline{x}_{t} \\|^2 =  \\| x_{t} - \\overline{x}_{t} - \\gamma_t \\mathcal{G}(x_t;\\xi_t)\\|^2 \\\\\n& =  \\| x_{t} - \\overline{x}_{t} \\|^2  - 2 \\gamma_t \\langle \\mathcal{G}(x_t;\\xi_t), x_t  - \\overline{x}_{t} \\rangle + \\gamma_t^2 \\| \\mathcal{G}(x_t;\\xi_t) \\|^2  \\nonumber  \\\\\n& = \\| x_{t} - \\overline{x}_{t} \\|^2  - 2 \\gamma_t \\langle \\nabla\nf(x_t,\\xi_t) + \\nabla g(x_{t+1},\\xi_t), x_t  - \\overline{x}_{t} \\rangle +\n\\gamma_t^2 \\| \\mathcal{G}(x_t;\\xi_t) \\|^2.\\nonumber \n\\end{align}\nNow, we refine the second term. First, from convexity of $f$ we\nhave:\n\\[ \\langle \\nabla f(x_t,\\xi_t), x_t  - \\overline{x}_{t} \\rangle \\geq f(x_t,\\xi_t) - f(\\overline{x}_{t},\\xi_t). \\]\nThen, from  convexity of $g$ and the definition of  stochastic\ngradient mapping $\\mathcal{G}(\\cdot)$, we have:\n\\begin{align*}\n&\\langle \\nabla g(x_{t+1},\\xi_t), x_t  - \\overline{x}_{t} \\rangle  = \\langle  \\nabla g(x_{t+1},\\xi_t), x_t  - x_{t+1} \\rangle + \\langle  \\nabla g(x_{t+1},\\xi_t), x_{t+1}  - \\overline{x}_{t} \\rangle \\\\\n& \\geq \\gamma_t \\| \\mathcal{G}(x_t;\\xi_t) \\|^2 - \\gamma_t \\langle \\nabla f(x_t,\\xi_t), \\mathcal{G}(x_t;\\xi_t) \\rangle + g(x_{t+1}, \\xi_t) - g(\\overline{x}_t,\\xi_t)\\\\\n& \\geq \\gamma_t \\| \\mathcal{G}(x_t;\\xi_t) \\|^2 - \\gamma_t \\langle\n\\nabla f(x_t,\\xi_t) + \\nabla g(x_t,\\xi_t), \\mathcal{G}(x_t;\\xi_t) \\rangle +\ng(x_{t}, \\xi_t) - g(\\overline{x}_t, \\xi_t).\n\\end{align*}\nReplacing the previous two inequalities in \\eqref{der_spg_basic}, we\nobtain:\n\\begin{align*}\n\\| x_{t+1} - \\overline{x}_{t+1} \\|^2 & \\leq  \\| x_{t} -\n\\overline{x}_{t} \\|^2\n - 2 \\gamma_t  \\left( f(x_t,\\xi_t) + g(x_t, \\xi_t) - f(\\overline{x}_{t},\\xi_t) - g(\\overline{x}_{t}, \\xi_t)\\right)  \\\\\n& \\quad + 2 \\gamma_t^2 \\langle \\nabla f(x_t,\\xi_t) + \\nabla g(x_t, \\xi_t),\n\\mathcal{G}(x_t;\\xi_t) \\rangle - \\gamma_t^2 \\|\n\\mathcal{G}(x_t;\\xi_t) \\|^2.\n\\end{align*}\nSince $2\\langle u , v \\rangle - \\|v\\|^2 \\leq \\|u\\|^2$ for\nall $v \\in \\rset^n$ and using that $ F(x_t,\\xi_t)  =  f(x_t,\\xi_t) +  g(x_t, \\xi_t)$ and $\\nabla F(x_t,\\xi_t)  = \\nabla f(x_t,\\xi_t) + \\nabla g(x_t, \\xi_t)$, we further get:\n\\begin{align*}\n\\| x_{t+1} - \\overline{x}_{t+1} \\|^2 & \\leq \\| x_{t} -\n\\overline{x}_{t} \\|^2\n - 2 \\gamma_t \\left(  F(x_t,\\xi_t)  - F(\\overline{x}_{t},\\xi_t) \\right)  +  \\gamma_t^2 \\| \\nabla F(x_t,\\xi_t)\\|^2.\n\\end{align*}\nNote that $x_t$ depends only on history $\\xi_{[t-1]} =\n\\{\\xi_0,\\ldots, \\xi_{t-1}\\} $, not $\\xi_t$. It follows from the\nbasic property of  conditional expectation that:\n\\begin{align*}\n& \\E_{\\xi_{[t]}} [\\|x_{t+1}  - \\overline{x}_{t+1}\\|^2] \\\\\n& \\leq \\E_{\\xi_{[t-1]}}[\\|x_{t} - \\overline{x}_{t}\\|^2]  - 2 \\gamma_t  \\E_{\\xi_{[t-1]}}\n\\left[ \\E_{\\xi_t} [ F(x_t,\\xi_t) - F(\\overline{x}_{t},\\xi_t)  | \\xi_{[t-1]}\n]\\right] \\\\\n& \\qquad +  \\gamma_t^2 \\E_{\\xi_{[t]}}  \\left[\\| \\nabla F(x_t,\\xi_t) \\|^2 ] \\right]\\\\\n& =  \\E_{\\xi_{[t-1]}}[\\|x_{t} \\!- \\overline{x}_{t}\\|^2]  - 2\n\\gamma_t \\E_{\\xi_{[t-1]}} \\left[ F(x_t) - F(\\overline{x}_{t})\\right]\n+ \\gamma_t^2 \\E_{\\xi_{[t]}}  \\left[\\| \\nabla F(x_t,\\xi_t) \\|^2 \\right].\n\\end{align*}\nFurther, making use of the stochastic bounded gradient condition given in  Assumption\n\\ref{assumption1} in the previous relation we get:\n\\begin{align*}\n& \\E_{\\xi_{[t]}} [\\|x_{t+1}  - \\overline{x}_{t+1}\\|^2] \\\\\n& \\leq \\E_{\\xi_{[t-1]}}[\\|x_{t} \\!- \\overline{x}_{t}\\|^2]  - 2\n\\gamma_t \\E_{\\xi_{[t-1]}} \\left[ F(x_t) - F(\\overline{x}_{t})\\right]\n+ \\gamma_t^2 \\E_{\\xi_{[t]}}  \\left[\\| \\nabla F(x_t,\\xi_t) \\|^2\n\\right] \\\\\n& \\overset{\\eqref{as:main1_spg}}{\\leq} \\! \\E_{\\xi_{[t-1]}}[\\|x_{t}\n- \\overline{x}_{t}\\|^2]  - 2 \\gamma_t  \\E_{\\xi_{[t-1]}} \\left[\nF(x_t) - F(\\overline{x}_{t})\\right] \\\\\n& \\qquad + \\gamma_t^2 \\left( B^2 + L\n\\E_{\\xi_{[t-1]}} \\!\\left[F(x_t) - F(\\overline{x}_{t})\\right] \\right)\n\\\\\n& = \\E_{\\xi_{[t-1]}}[\\|x_{t} \\!- \\overline{x}_{t}\\|^2]  - \\gamma_t\n(2 - \\gamma_t L) \\E_{\\xi_{[t-1]}} \\left[ F(x_t) -\nF(\\overline{x}_{t})\\right] + \\gamma_t^2 B^2,\n\\end{align*}\nwhich, omitting the dependence  of expectation on $\\xi_{[t]}$,  proves the  statement of the theorem. \\qed\n\\end{proof}","sha256_utf8":"f84f57429c63da7ceef719cae5bed0ccfd17bdebf35318ab96e7998d28558430"},{"start":32384,"end":33148,"start_line":432,"end_line":441,"raw_tex":"We first recall that  any strongly convex function $h$ with\nconvexity constant $\\sigma_h$ and having the optimal point $x^*$\nsatisfies the following inequality~\\cite{Nes:04}: $h(x) \\geq  h(x^*) + \\frac{\\sigma_h}{2}\\|x - x^*\\|^2 \\quad \\forall x$. Since for any convex function $h$ and fixed  point $x$, the Moreau smoothing  function  $h_\\gamma(y;x) = h(y) +  \\frac{1}{2 \\gamma} \\|y -x\\|^2$ is\nstrongly convex in the first argument $y$ with strong convexity constant\n$1/\\gamma$, the following holds for all $x, y \\in \\rset^n$:\n\\begin{align}\n\\label{rel1_prox} \n& h(y)  \\!+\\!  \\frac{1}{2 \\gamma} \\|y - x\\|^2   \\!\\geq\\!  h(\\prox_{\\gamma h}(x)) \\!+\\! \\frac{1}{2 \\gamma} \\|\\prox_{\\gamma h}(x)\n- x\\|^2 \\!+\\! \\frac{1}{2 \\gamma} \\|\\prox_{\\gamma h}(x) - y\\|^2. \n\\end{align}\n","sha256_utf8":"f7d16a4867c7631764bbad84fdf6cce39fc459e154f1564ccef284236f4927f4"},{"start":33297,"end":34069,"start_line":447,"end_line":454,"raw_tex":"\\begin{theorem}\n\\label{th:spp_basic} Let $f(\\cdot, \\xi)$ and $g(\\cdot, \\xi)$ be convex functions  that  admit easily computable proximal operators. Additionally, assume that the  stochastic bounded gradient condition from Assumption \\ref{assumption1} holds and $g(\\cdot, \\xi)$ have bounded subgradients, that is there exists $B_{g}>0$ such that $ \\|\\nabla g(x, \\xi)\\| \\leq B_g$ for all $x$ and $\\xi$. Then, for any stepsize $\\gamma_t>0$ we have the following recursive inequality for the SPP iteration:\n\\begin{align}\n\\label{spp_basic} & \\Exp{\\|x_{t+1} - \\overline{x}_{t+1} \\|^2} \\leq \\\\\n&  \\Exp{\\| x_t - \\overline{x}_{t} \\|^2}   - \\gamma_t (2 - \\gamma_t L) \\Exp{ F(x_t) -\nF(\\overline{x}_{t})}  + \\gamma_t^2 \\left( B^2 + B_{g}^2 \\right). \\nonumber\n\\end{align}\n\\end{theorem}","sha256_utf8":"7e10de33c998e0d34bc42be98d9ca22a52c76efffc16ca6b24b8688521a19edc"},{"start":34071,"end":36511,"start_line":456,"end_line":497,"raw_tex":"\\begin{proof}\nUsing \\eqref{rel1_prox} for the function $h=f(\\cdot,\\xi_t)$ with $y\n= \\overline{x}_{t}$ and $x = x_t$ we get:\n\\[  f(\\overline{x}_{t},\\xi_t) + \\frac{1}{2 \\gamma_t} \\|x_t - \\overline{x}_{t}\\|^2 \\geq\nf(x_{t+1/2},\\xi_t) + \\frac{1}{2 \\gamma_t} \\|x_{t+1/2} - x_t\\|^2 +\n\\frac{1}{2 \\gamma_t} \\|x_{t+1/2} - \\overline{x}_{t}\\|^2. \\] Using\nagain \\eqref{rel1_prox} for  function $h=g(\\cdot,\\xi_t)$ with $y = \\overline{x}_{t}$\nand $x = x_{t+1/2}$ we get:\n\\[  g(\\overline{x}_{t},\\xi_t) + \\frac{1}{2 \\gamma_t} \\|x_{t+1/2} - \\overline{x}_{t}\\|^2 \\geq\ng(x_{t+1},\\xi_t) + \\frac{1}{2 \\gamma_t} \\|x_{t+1} - x_{t+1/2}\\|^2 +\n\\frac{1}{2 \\gamma_t} \\|x_{t+1} - \\overline{x}_{t}\\|^2. \\] Adding the\nprevious two inequalities  and using the convexity of $f(\\cdot,\\xi_t)$\nand $g(\\cdot,\\xi_t)$, we get:\n\\begin{align*}\n& f(\\overline{x}_{t},\\xi_t) +  g(\\overline{x}_{t},\\xi_t) + \\frac{1}{2\n\\gamma_t} \\|x_t - \\overline{x}_{t}\\|^2 -\n\\frac{1}{2 \\gamma_t} \\|x_{t+1} - \\overline{x}_{t}\\|^2\\\\\n& \\geq f(x_{t+1/2},\\xi_t) + g(x_{t+1},\\xi_t) + \\frac{1}{2 \\gamma_t}\n\\|x_{t+1/2} - x_t\\|^2 + \\frac{1}{2 \\gamma_t} \\|x_{t+1} -\nx_{t+1/2}\\|^2\n\\\\\n& \\geq  f(x_t,\\xi_t) + g(x_t,\\xi_t)  +  \\langle \\nabla f(x_t,\\xi_t) +\n\\nabla g(x_t,\\xi_t) , x_{t+1/2} - x_t \\rangle \n\\\\\n&\\qquad + \\frac{1}{2 \\gamma_t} \\|x_{t+1/2} -\nx_t\\|^2 + \\langle \\nabla g(x_t,\\xi_t), x_{t+1}\n- x_{t+1/2}  \\rangle + \\frac{1}{2 \\gamma_t} \\|x_{t+1} -\nx_{t+1/2}\\|^2\\\\\n& \\geq f(x_t,\\xi_t) + g(x_t,\\xi_t)  -\\frac{\\gamma_t}{2} \\left( \\| \\nabla\nf(x_t,\\xi_t) + \\nabla g(x_t,\\xi_t) \\|^2 + \\|\\nabla g(x_t,\\xi_t)\\|^2 \\right),\n\\end{align*}\nwhere in the last inequality we used that $\\langle \\alpha,z \\rangle\n + \\frac{1}{2\\gamma} \\|z\\|^2 \\geq - \\frac{\\gamma}{2} \\|\\alpha\\|^2$ for all\n$z$. Taking now  expectation and using that $\\|x_{t+1} -\n\\overline{x}_{t}\\|^2 \\geq \\|x_{t+1} - \\overline{x}_{t+1}\\|^2$ we get:\n\\begin{align*}\n& \\E_{\\xi_{[t]}} [\\|x_{t+1} - \\overline{x}_{t+1} \\|^2] \\leq  \\E_{\\xi_{[t-1]}} [\\| x_t - \\overline{x}_{t} \\|^2   - 2 \\gamma_t  \\E_{\\xi_{[t-1]}}[ F(x_t) - F(\\overline{x}_{t})]  \\\\ \n& \\quad + \\gamma_t^2 \\E_{\\xi_{[t]}} [ \\| \\nabla\nF(x_t,\\xi_t)   \\|^2 + \\| \\nabla g(x_t,\\xi_t) \\|^2 ],\n\\end{align*}\nwhere we used basic properties of the conditional expectation as in the proof of Theorem \\ref{th:spg_basic}. Further, using the stochastic bounded gradient condition \\eqref{as:main1_spg}  and  omitting the dependence  of expectation on $\\xi_{[t]}$, we get  the statement of the theorem.  \\qed\n\\end{proof}","sha256_utf8":"3e3eec4055520858068764cfbd10d7f513147a7ffbff4e8110bdb0d1766210b5"},{"start":45523,"end":48175,"start_line":619,"end_line":661,"raw_tex":"\\begin{proof}\nFrom \\eqref{sfo_basic}  and Assumption \\ref{assumption2}  it follows that:\n\\begin{align*}\n& \\E_{\\xi_{[t]}} [\\|x_{t+1}  - \\overline{x}_{t+1}\\|^2] \\leq\n\\E_{\\xi_{[t-1]}}[\\|x_{t} - \\overline{x}_{t}\\|^2]  -  \\gamma_t (2 -\n\\gamma_t L) \\E_{\\xi_{[t-1]}}\n\\left[F(x_t) - F(\\overline{x}_{t})\\right]  \\\\\n& + {\\cal B}^2 \\gamma_{t}^2  \\overset{\\eqref{as:strong_spg}}{\\leq} \\E_{\\xi_{[t-1]}}[\\|x_{t} -\n\\overline{x}_{t}\\|^2]  - \\frac{\\mu \\gamma_t (2 - \\gamma_t L)}{2}\n\\E_{\\xi_{[t-1]}} [\\|x_{t} - \\overline{x}_{t}\\|^2] +  {\\cal B}^2\n\\gamma_{t}^2,\n\\end{align*}\nprovided that $\\gamma_t \\leq 2/L$.  In conclusion, we obtain the\nfollowing recursion:\n\\begin{align}\n\\label{recursive1_spg} \\E_{\\xi_{[t]}} [\\|x_{t+1} -\n\\overline{x}_{t+1}\\|^2] & \\leq   \\left(1 - \\mu \\gamma_t + \\frac{\\mu\nL \\gamma_t^2}{2} \\right) \\E_{\\xi_{[t-1]}} [\\|x_{t} -\n\\overline{x}_{t}\\|^2] +  {\\cal B}^2 \\gamma_{t}^2.\n\\end{align}\nFor constant stepsize $\\gamma_t = \\gamma < 2/L$ it follows that $ 1 - \\mu \\gamma\n+ \\mu L \\gamma^2/2 < 1$.  Then, combining inequality \\eqref{recursive1_spg} with\ncondition $1 - \\mu \\gamma + \\mu L\n\\gamma^2/2 > -1$, we get:\n\\begin{align*}\n\\E [\\|x_{t} - \\overline{x}_{t}\\|^2] & \\leq   |1 - \\mu \\gamma +\n\\frac{\\mu L \\gamma^2}{2} |^t \\|x_{0} - \\overline{x}_{0}\\|^2 +\n{\\cal B}^2 \\gamma^2 \\sum_{j=0}^{t-1} |1 - \\mu \\gamma +\n\\frac{\\mu L \\gamma^2}{2}|^j \\\\\n& \\leq |1 - \\mu \\gamma + \\frac{\\mu L \\gamma^2}{2} |^t\n\\|x_{0} - \\overline{x}_{0}\\|^2 +    \\frac{{\\cal B}^2 \\gamma^2}{ 1- |1-\\mu \\gamma\n+ \\mu L \\gamma^2/2|},\n\\end{align*}\nwhere in the second inequality we used that $\\sum_{j=0}^{t-1} c^j\n\\leq \\sum_{j=0}^{\\infty} c^j = \\frac{1}{1-c}$ for any $|c|<1$.\nOtherwise, if $1 - \\mu \\gamma + \\mu L \\gamma^2/2 \\leq -1$, then from\n\\eqref{recursive1_spg} it follows that:\n\\begin{align*} \\E [\\|x_{t} - \\overline{x}_{t}\\|^2] &\n\\leq {\\cal B}^2 \\gamma^2.\n\\end{align*}\nMoreover, if $\\mu < 4L$, then the relation $1 - \\mu \\gamma + \\mu L\n\\gamma^2/2 > -1$ always holds, regardless of the value of $\\gamma$. Thus, if $\\mu < 4L$ and  the constant stepsize is chosen such $\\gamma < \\frac{2}{L}$,  then $|1 - \\mu \\gamma + \\mu L \\gamma^2| < 1$. Similarly, if  the  constant stepsize satisfies  $\\gamma < \\min(2/L, 2/\\mu)$, then again we have $0< 1 - \\mu \\gamma + \\mu L \\gamma^2 < 1$. Therefore, these two previous conditions guarantee linear convergence  to a noise dominated region whose radius is proportional to the square of the stepsize $\\gamma$.    Finally, the inequality \\eqref{recursive1_spg} with ${\\cal B}=0$ and $|1 - \\mu \\gamma + \\mu L \\gamma^2| < 1$  proves  linear convergence for $\\Exp{\\|x_{t} - \\overline{x}_{t} \\|^2}$, with $x_t$ generated by SFO methods.  \\qed\n\\end{proof}","sha256_utf8":"0285608c6e7be3516dbeb3972d7303c01d0e3715268f58f310a6ba168568a4fd"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"随机一阶方法的线性收敛","source_units":[{"start":43863,"end":45521,"start_line":599,"end_line":617,"raw_tex":"\\begin{theorem} [Sufficient conditions for linear convergence of SFO]\n\\label{th:spg_muL} Let assumptions of Theorem \\ref{th:spg_basic} for SPG and  Theorem \\ref{th:spp_basic} for SPP hold. Additionally, we  assume that the quadratic functional growth condition from  Assumption \\ref{assumption2} holds  and we choose constant stepsize  $\\gamma_t = \\gamma \\in (0,2/L)$. Then, we get the following upper bounds for the square distance of the SFO iterates to the solution set:\n\\begin{equation*}\n\\Exp{\\|x_{t} - \\overline{x}_{t} \\|^2} \\!\\leq\\!\\! \\left\\{ \\!\\begin{array}{lr}\n\\left| 1 - \\mu \\gamma + \\frac{\\mu L \\gamma^2}{2} \\right |^t  \\!\\!R_0^2 +\\!\n\\frac{{\\cal B}^2 \\gamma^2}{1- |1-\\mu \\gamma  + \\mu L \\gamma^2/2|}  \\!\\!\\!\\!\\!\\!\\!\\! & \\text{if}  \\; \n1 \\!-\\! \\mu \\gamma + \\frac{\\mu L \\gamma^2}{2} \\!>\\! -1 \\\\\n{\\cal B}^2 \\gamma^2  & \\text{if}  \\quad  1 - \\mu \\gamma + \\frac{\\mu L\n\\gamma^2}{2} \\leq -1.\n\\end{array}\\right.\n\\end{equation*}\nMoreover, if the condition $\\mu < 4L$ holds or  the constant stepsize satisfies $\\gamma < \\min\\left({2}/{L}, {2}/{\\mu} \\right)$, then  $|  1 - \\mu \\gamma + \\frac{\\mu L\\gamma^2}{2}| < 1$ and therefore we get linear converge to a noise dominated region with radius  proportional to the square of the stepsize $\\gamma$:\n\\[ \\Exp{\\|x_{t} - \\overline{x}_{t} \\|^2} \\leq  \\left| 1 - \\mu \\gamma + \\frac{\\mu L \\gamma^2}{2} \\right|^t  R_0^2 +\n\\frac{{\\cal B}^2 \\gamma^2}{1- |1-\\mu \\gamma  + \\mu L \\gamma^2/2|}. \\]\nConsequently, if ${\\cal B}=0$, then we get pure linear convergence:\n\\begin{equation*}\n\\Exp{\\|x_{t} - \\overline{x}_{t} \\|^2} \\leq \\left| 1 - \\mu \\gamma + \\frac{\\mu L \\gamma^2}{2} \\right |^t  R_0^2.\n\\end{equation*}\n\\end{theorem}","sha256_utf8":"855a7a60458b49302bb8f09134c246acc7367bd1ad94d74a0f1117b47858836d"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"线性收敛完整证明","source_units":[{"start":45523,"end":48175,"start_line":619,"end_line":661,"raw_tex":"\\begin{proof}\nFrom \\eqref{sfo_basic}  and Assumption \\ref{assumption2}  it follows that:\n\\begin{align*}\n& \\E_{\\xi_{[t]}} [\\|x_{t+1}  - \\overline{x}_{t+1}\\|^2] \\leq\n\\E_{\\xi_{[t-1]}}[\\|x_{t} - \\overline{x}_{t}\\|^2]  -  \\gamma_t (2 -\n\\gamma_t L) \\E_{\\xi_{[t-1]}}\n\\left[F(x_t) - F(\\overline{x}_{t})\\right]  \\\\\n& + {\\cal B}^2 \\gamma_{t}^2  \\overset{\\eqref{as:strong_spg}}{\\leq} \\E_{\\xi_{[t-1]}}[\\|x_{t} -\n\\overline{x}_{t}\\|^2]  - \\frac{\\mu \\gamma_t (2 - \\gamma_t L)}{2}\n\\E_{\\xi_{[t-1]}} [\\|x_{t} - \\overline{x}_{t}\\|^2] +  {\\cal B}^2\n\\gamma_{t}^2,\n\\end{align*}\nprovided that $\\gamma_t \\leq 2/L$.  In conclusion, we obtain the\nfollowing recursion:\n\\begin{align}\n\\label{recursive1_spg} \\E_{\\xi_{[t]}} [\\|x_{t+1} -\n\\overline{x}_{t+1}\\|^2] & \\leq   \\left(1 - \\mu \\gamma_t + \\frac{\\mu\nL \\gamma_t^2}{2} \\right) \\E_{\\xi_{[t-1]}} [\\|x_{t} -\n\\overline{x}_{t}\\|^2] +  {\\cal B}^2 \\gamma_{t}^2.\n\\end{align}\nFor constant stepsize $\\gamma_t = \\gamma < 2/L$ it follows that $ 1 - \\mu \\gamma\n+ \\mu L \\gamma^2/2 < 1$.  Then, combining inequality \\eqref{recursive1_spg} with\ncondition $1 - \\mu \\gamma + \\mu L\n\\gamma^2/2 > -1$, we get:\n\\begin{align*}\n\\E [\\|x_{t} - \\overline{x}_{t}\\|^2] & \\leq   |1 - \\mu \\gamma +\n\\frac{\\mu L \\gamma^2}{2} |^t \\|x_{0} - \\overline{x}_{0}\\|^2 +\n{\\cal B}^2 \\gamma^2 \\sum_{j=0}^{t-1} |1 - \\mu \\gamma +\n\\frac{\\mu L \\gamma^2}{2}|^j \\\\\n& \\leq |1 - \\mu \\gamma + \\frac{\\mu L \\gamma^2}{2} |^t\n\\|x_{0} - \\overline{x}_{0}\\|^2 +    \\frac{{\\cal B}^2 \\gamma^2}{ 1- |1-\\mu \\gamma\n+ \\mu L \\gamma^2/2|},\n\\end{align*}\nwhere in the second inequality we used that $\\sum_{j=0}^{t-1} c^j\n\\leq \\sum_{j=0}^{\\infty} c^j = \\frac{1}{1-c}$ for any $|c|<1$.\nOtherwise, if $1 - \\mu \\gamma + \\mu L \\gamma^2/2 \\leq -1$, then from\n\\eqref{recursive1_spg} it follows that:\n\\begin{align*} \\E [\\|x_{t} - \\overline{x}_{t}\\|^2] &\n\\leq {\\cal B}^2 \\gamma^2.\n\\end{align*}\nMoreover, if $\\mu < 4L$, then the relation $1 - \\mu \\gamma + \\mu L\n\\gamma^2/2 > -1$ always holds, regardless of the value of $\\gamma$. Thus, if $\\mu < 4L$ and  the constant stepsize is chosen such $\\gamma < \\frac{2}{L}$,  then $|1 - \\mu \\gamma + \\mu L \\gamma^2| < 1$. Similarly, if  the  constant stepsize satisfies  $\\gamma < \\min(2/L, 2/\\mu)$, then again we have $0< 1 - \\mu \\gamma + \\mu L \\gamma^2 < 1$. Therefore, these two previous conditions guarantee linear convergence  to a noise dominated region whose radius is proportional to the square of the stepsize $\\gamma$.    Finally, the inequality \\eqref{recursive1_spg} with ${\\cal B}=0$ and $|1 - \\mu \\gamma + \\mu L \\gamma^2| < 1$  proves  linear convergence for $\\Exp{\\|x_{t} - \\overline{x}_{t} \\|^2}$, with $x_t$ generated by SFO methods.  \\qed\n\\end{proof}","sha256_utf8":"0285608c6e7be3516dbeb3972d7303c01d0e3715268f58f310a6ba168568a4fd"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"随机复合凸优化问题","source_units":[{"start":9349,"end":10868,"start_line":133,"end_line":143,"raw_tex":"\\noindent Let   $f, g \\colon \\rset^m \\times \\Omega \\to \\bar{\\rset}$ be  two proper convex\n functions in the first argument, where $\\Omega$ is endowed with a\nprobability distribution $\\Prob$.  Then, we consider the following\ngeneral composite stochastic optimization problem:\n\\begin{align}\n\\label{eq_problem}\nF^* & = \\min_{x \\in \\rset^m} F(x) \\quad \\left( \\eqdef f(x) + g(x) \\right),\n\\end{align}\nwhere the two functions have stochastic representations in the form of expectation, i.e.  $f(x) = \\Exp{f(x, \\xi)}$ and $ g(x) = \\Exp{g(x, \\xi)}$. We   assume that the expectation taken with respect to the random variable $\\xi \\in \\Omega$ is finite\nfor all $x \\in \\textnormal{dom} \\ F$. Hence we consider a flexible splitting of our stochastic objective function $F(x) =   \\Exp{F(x, \\xi)}$ in  composite form  as a sum of two  terms: \n$$ F(x,\\xi) = f(x,\\xi) + g(x,\\xi). $$   In this paper we assume $f(\\cdot,\\xi)$ either differentiable or nondifferentiable function and we use,  with some abuse of notation, the same notation for the gradient or subgradient of $f(\\cdot,\\xi)$ at $x$, that is $\\nabla f(x,\\xi) \\in \\partial f(x, \\xi)$, where $\\partial f(x,\\xi)$ is either a singleton or a nonempty convex set. The other term, $g(x,\\xi)$,  is assumed to be simple and its structure known.  We usually refer to $g$ as the regularizer and use the same notation as before  $\\nabla g(x,\\xi) \\in \\partial g(x, \\xi)$ for a (sub)gradient of $g(x, \\xi)$ at $x$.  We  also assume in the sequel that to have access to either  an unbiased\n","sha256_utf8":"c0d87fbcaf70711ac253cb1ed00a3cbc3c9f8cf726a119c28c1ad7bd5e6d4bc9"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"随机预言机与近端算子","source_units":[{"start":10868,"end":11623,"start_line":144,"end_line":151,"raw_tex":"stochastic estimates of the (sub)gradients of function $f$, i.e.:\n\\[ \\mathbb{E}[\\nabla f(x,\\xi)] \\in \\partial f(x), \\;\\; \\text{where}  \\;\\; \\nabla f(x,\\xi) \\in \\partial f(x,\\xi),  \\] or to\nstochastic estimates of the proximal operator of $f$, i.e.:\n\\[   \\prox_{\\gamma f(\\cdot,\\xi)}(x) = \\arg\n\\min\\limits_{y \\in \\rset^m} f(y,\\xi) + \\frac{1}{2 \\gamma} \\|y\n-x\\|^2.\\]\nWe refer to $\\prox_{\\gamma f (\\cdot,\\xi)}(x)$ as the proximal operator of the\nfunction $f(\\cdot,\\xi)$ at $x$ with stepsize $\\gamma$.  We  always assume to have access  to  stochastic estimates of the proximal operator of $g$, i.e. $ \\prox_{\\gamma g(\\cdot,\\xi)}(x) $. These are  key assumptions in our stochastic optimization to scale up the numerical algorithms. Template \\eqref{eq_problem}\n","sha256_utf8":"774b0d7cabe24eb223f927cbe50cdcd11376a9cc3f93f01958338ee28b2ab6b2"}]},{"id":"S005","semantic_type":"notation","kind":"notation","title":"最优集及投影记号","source_units":[{"start":11896,"end":12174,"start_line":162,"end_line":162,"raw_tex":"\\noindent Let us denote by $X^*$ the optimal set of the convex problem \\eqref{eq_problem} and for any $x \\in \\rset^m$ we denote its projection onto $X^*$ by $\\overline{x}$, that is $\\overline{x} = \\Pi_{X^*}(x)$. In this paper we consider additionally the following assumptions:\n","sha256_utf8":"4fa58e13007064a84ba794c60c3ab13e13187cd8064d0f71b746b697f10e4836"}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","title":"随机梯度有界条件","source_units":[{"start":12174,"end":12614,"start_line":163,"end_line":173,"raw_tex":"\\begin{assumption}\n\\label{assumption1}\nThe stochastic (sub)gradients\nof $F$ satisfy a stochastic bounded gradient condition restricted on\nany segment $[x, \\; \\overline{x}]$, that is there exist nonnegative\nconstants  $L \\geq 0$ and $B \\geq 0$ such that:\n\\begin{equation}\n\\label{as:main1_spg} B^2 +  L(F(x) - F(\\overline{x})) \\geq\n\\E_{\\xi}[\\| \\nabla F(x,\\xi)  \\|^2 ] \\quad  \\forall x \\in \\textnormal{dom} \\ F.\n\\end{equation}\n\\end{assumption}","sha256_utf8":"5dbf4854b9f03214a605379e4f2ab8a0dd9ad0837d244ff08fa1e0b42a326dd4"}]},{"id":"S007","semantic_type":"assumption","kind":"assumption","title":"二次函数增长条件","source_units":[{"start":12616,"end":12997,"start_line":175,"end_line":183,"raw_tex":"\\begin{assumption}\n\\label{assumption2}\nThe function  $F$ satisfies a quadratic functional growth condition restricted on\nany segment $[x, \\; \\overline{x}]$, that is there exists  $\\mu \\geq 0$ such that:\n\\begin{equation}\n\\label{as:strong_spg} F(x) - F(\\overline{x}) \\geq  \\frac{\\mu}{2} \\|x\n- \\overline{x}\\|^2 \\quad \\forall x \\in \\textnormal{dom} \\ F.\n\\end{equation}\n\\end{assumption}","sha256_utf8":"4608b77f4f69e960098b71991bed363f9b45a09205558f92f4e045cdbd8e3f6e"}]},{"id":"S008","semantic_type":"algorithm","kind":"algorithm","title":"随机近端梯度迭代","source_units":[{"start":24019,"end":24410,"start_line":293,"end_line":301,"raw_tex":"\\begin{algorithm}\nLet $(\\xi_t)_{t \\in \\nset}$ be an i.i.d sequence and $x_0 \\in\n\\rset^m$. Iterate:\n\\begin{equation}\n\\label{eq_spgm}  \\nabla f(x_t,\\xi_t) \\in\\partial f(x_t,\\xi_t) \\quad  \\text{and} \\quad x_{t+1} = \\prox_{\\gamma_t g(\\cdot,\\xi_t)}(x_t - \\gamma_t \\nabla f(x_t,\\xi_t) ),\n\\end{equation}\nwhere $(\\gamma_t)_{t \\in \\nset}$ is a strictly positive sequence of\nstepsizes.\n\\end{algorithm}","sha256_utf8":"1b80c71d74faf5252dbc6a23f357a6e10f2bfd386360b9234b91f186d8e7ae8b"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"随机梯度映射","source_units":[{"start":24798,"end":25158,"start_line":308,"end_line":312,"raw_tex":"gradient mapping (for simplicity we omit its dependence on stepsize~$\\gamma$):\n\\[  \\mathcal{G}(x;\\xi) = \\gamma^{-1} \\left( x - \\prox_{\\gamma g(\\cdot,\\xi)}(x - \\gamma {\\nabla} f(x,\\xi)) \\right). \\]\nThen, it follows immediately that  the previous stochastic proximal\ngradient iteration can  be written as:\n\\[ x_{t+1} = x_t - \\gamma_t \\mathcal{G}(x_t;\\xi_t).  \\]\n","sha256_utf8":"3734d124a2b4590b5bd7953f67fa64d618269eb726b90a7e95a04080aaa9ccae"}]},{"id":"S010","semantic_type":"derived_fact","kind":"derived_fact","title":"梯度映射的次梯度表示","source_units":[{"start":25158,"end":25374,"start_line":313,"end_line":315,"raw_tex":"Moreover, from  optimality condition of prox operator there exists\n$\\nabla g(x_{t+1},\\xi_t) \\in \\partial g(x_{t+1},\\xi_t)$ such that:\n\\[  \\mathcal{G}(x_t;\\xi_t) =  \\nabla f(x_t,\\xi_t)  + \\nabla g(x_{t+1},\\xi_t).  \\]\n","sha256_utf8":"9ef6afac33dfe39db5781d73daac5e849ccea6bb59ba21feb087f0a2f113267d"}]},{"id":"S011","semantic_type":"notation","kind":"notation","title":"样本历史记号","source_units":[{"start":25376,"end":25586,"start_line":318,"end_line":319,"raw_tex":"\\noindent Denote  also $\\xi_{[t]} = \\{\\xi_0,\\ldots, \\xi_{t}\\} $.  The next theorem provides a descent property for  SPG and for the  proof we use as main tool  stochastic\ngradient mapping~$\\mathcal{G}(\\cdot)$.\n","sha256_utf8":"cf75351f6a67d125c52a9a431edd29db8a3775937fdc26457a6fd7bbcc2e98b7"}]},{"id":"S012","semantic_type":"algorithm","kind":"algorithm","title":"随机近端点迭代","source_units":[{"start":31618,"end":31975,"start_line":414,"end_line":424,"raw_tex":"\\begin{algorithm}\nLet $(\\xi_t)_{t \\in \\nset}$ be an i.i.d sequence, and $x_0 \\in\n\\rset^m$. Iterate:\n\\begin{equation}\n\\label{eq_sppm}  x_{t+1/2} = \\prox_{\\gamma_t\nf(\\cdot,\\xi_t)}(x_t) \\quad \\text{and} \\quad x_{t+1}  =\n\\prox_{\\gamma_t g}(x_{t+1/2}),\n\\end{equation}\nwhere $(\\gamma_t)_{t \\in \\nset}$ is a strictly positive sequence of\nstepsizes.\n\\end{algorithm}","sha256_utf8":"6bee7346941dbc627c834167655a721a2a5776168f05975e3128a4226c65a6bd"}]},{"id":"S013","semantic_type":"notation","kind":"notation","title":"期望距离记号","source_units":[{"start":36657,"end":37535,"start_line":504,"end_line":506,"raw_tex":"\\noindent Let us denote $R_t^2 = \\E_{\\xi_{[t-1]}} [\\|x_t -\n\\overline{x}_{t}\\|^2]$. It follows that $R_0= \\|x_0 -\n\\overline{x}_{0}\\|^2$.  From Theorem \\ref{th:spg_basic} for SPG and Theorem \\ref{th:spp_basic} for SPP, we can derive various convergence rates for the stochastic first order (SFO) methods (i.e. SPG, SPP) depending on  the values taken by the constants $B, L$ and $\\mu$ and also depending on the choice of the stepsize: constant or variable. Our convergence results recover, complement or extend the previous convergence rates for stochastic gradient descent (SGD) to more general  functions $f(\\cdot,\\xi)$  and  $g(\\cdot,\\xi)$.  To derive the convergence rates for the two SFO algorithms derived in the previous sections,  first we can notice that the descent relations from  Theorem \\ref{th:spg_basic}  and Theorem \\ref{th:spp_basic} can be written compactly as:\n","sha256_utf8":"ff131f3f6b6dda98dba10448726423c1e1936e84b24427d5eece54ea0634639b"}]},{"id":"S014","semantic_type":"derived_fact","kind":"derived_fact","title":"统一下降递推与噪声常数","source_units":[{"start":37535,"end":37952,"start_line":507,"end_line":517,"raw_tex":"\\begin{align}\n\\label{sfo_basic} \n& \\Exp{\\|x_{t+1} - \\overline{x}_{t+1} \\|^2} \\leq   \\Exp{\\| x_t - \\overline{x}_{t} \\|^2}   - \\gamma_t (2 - \\gamma_t L) \\Exp{ F(x_t) - F(\\overline{x}_{t})}  + \\gamma_t^2 {\\cal B}^2, \n\\end{align}\nwhere \n\\begin{equation*}\n{\\cal B}^2  \\leq \\left\\{ \\begin{array}{lr}\nB^2 & \\text{if}  \\quad \\;  \\text{SPG} \\\\\nB^2+B_g^2  & \\;\\; \\text{if}  \\quad \\text{SPP}.\n\\end{array}\\right.\n\\end{equation*}\n","sha256_utf8":"b0e433b91f82cf2ef8f95aa17e32f0dcb522b9e2da8f68b72ef3d4ecb6be27cb"}]},{"id":"S015","semantic_type":"notation","kind":"notation","title":"零分母约定","source_units":[{"start":37952,"end":38168,"start_line":518,"end_line":518,"raw_tex":"Based on this common descent property  of our stochastic first order methods, for  the convex case (i.e. $\\mu=0$) and constant stepsize we get the following convergence result (we make the convention $2/0 =\\infty$):\n","sha256_utf8":"dddab5803f0575807c56a5e2385d7efbbb5b027c1b87c9a25fbc784bb70f9e0e"}]},{"id":"S016","semantic_type":"theorem","kind":"theorem","title":"近端梯度下降递推","source_units":[{"start":25587,"end":26199,"start_line":321,"end_line":328,"raw_tex":"\\begin{theorem}\n\\label{th:spg_basic} Let  $f(\\cdot, \\xi)$ and $g(\\cdot, \\xi)$ be  convex functions such that $g$ admits an easily computable proximal operator. Additionally,  assume that the stochastic bounded gradient condition from   Assumption \\ref{assumption1} holds. Then, for any $t \\geq 0$ and  stepsize $\\gamma_t >0$, we have the following recursion  for the   SPG iteration:\n\\begin{align}\n\\label{spg_basic} & \\Exp{\\|x_{t+1} - \\overline{x}_{t+1} \\|^2}  \\leq\n\\Exp{\\| x_t - \\overline{x}_{t} \\|^2} -  \\gamma_t  (2 - \\gamma_t L)\n\\Exp{ F(x_t) - F(\\overline{x}_{t})} + \\gamma_t^2 B^2.\n\\end{align}\n\\end{theorem}","sha256_utf8":"30f718d3ba227601740b986edfab99ab940ad1b4c8cbcd4466f01f431e86216a"}]},{"id":"S017","semantic_type":"proof","kind":"proof","title":"近端梯度递推完整证明","source_units":[{"start":26201,"end":30316,"start_line":330,"end_line":402,"raw_tex":"\\begin{proof}\nFrom the definition of SPG iteration we\nhave:\n\\begin{align}\n\\label{der_spg_basic}\n& \\| x_{t+1} - \\overline{x}_{t+1} \\|^2  \\leq   \\| x_{t+1} - \\overline{x}_{t} \\|^2 =  \\| x_{t} - \\overline{x}_{t} - \\gamma_t \\mathcal{G}(x_t;\\xi_t)\\|^2 \\\\\n& =  \\| x_{t} - \\overline{x}_{t} \\|^2  - 2 \\gamma_t \\langle \\mathcal{G}(x_t;\\xi_t), x_t  - \\overline{x}_{t} \\rangle + \\gamma_t^2 \\| \\mathcal{G}(x_t;\\xi_t) \\|^2  \\nonumber  \\\\\n& = \\| x_{t} - \\overline{x}_{t} \\|^2  - 2 \\gamma_t \\langle \\nabla\nf(x_t,\\xi_t) + \\nabla g(x_{t+1},\\xi_t), x_t  - \\overline{x}_{t} \\rangle +\n\\gamma_t^2 \\| \\mathcal{G}(x_t;\\xi_t) \\|^2.\\nonumber \n\\end{align}\nNow, we refine the second term. First, from convexity of $f$ we\nhave:\n\\[ \\langle \\nabla f(x_t,\\xi_t), x_t  - \\overline{x}_{t} \\rangle \\geq f(x_t,\\xi_t) - f(\\overline{x}_{t},\\xi_t). \\]\nThen, from  convexity of $g$ and the definition of  stochastic\ngradient mapping $\\mathcal{G}(\\cdot)$, we have:\n\\begin{align*}\n&\\langle \\nabla g(x_{t+1},\\xi_t), x_t  - \\overline{x}_{t} \\rangle  = \\langle  \\nabla g(x_{t+1},\\xi_t), x_t  - x_{t+1} \\rangle + \\langle  \\nabla g(x_{t+1},\\xi_t), x_{t+1}  - \\overline{x}_{t} \\rangle \\\\\n& \\geq \\gamma_t \\| \\mathcal{G}(x_t;\\xi_t) \\|^2 - \\gamma_t \\langle \\nabla f(x_t,\\xi_t), \\mathcal{G}(x_t;\\xi_t) \\rangle + g(x_{t+1}, \\xi_t) - g(\\overline{x}_t,\\xi_t)\\\\\n& \\geq \\gamma_t \\| \\mathcal{G}(x_t;\\xi_t) \\|^2 - \\gamma_t \\langle\n\\nabla f(x_t,\\xi_t) + \\nabla g(x_t,\\xi_t), \\mathcal{G}(x_t;\\xi_t) \\rangle +\ng(x_{t}, \\xi_t) - g(\\overline{x}_t, \\xi_t).\n\\end{align*}\nReplacing the previous two inequalities in \\eqref{der_spg_basic}, we\nobtain:\n\\begin{align*}\n\\| x_{t+1} - \\overline{x}_{t+1} \\|^2 & \\leq  \\| x_{t} -\n\\overline{x}_{t} \\|^2\n - 2 \\gamma_t  \\left( f(x_t,\\xi_t) + g(x_t, \\xi_t) - f(\\overline{x}_{t},\\xi_t) - g(\\overline{x}_{t}, \\xi_t)\\right)  \\\\\n& \\quad + 2 \\gamma_t^2 \\langle \\nabla f(x_t,\\xi_t) + \\nabla g(x_t, \\xi_t),\n\\mathcal{G}(x_t;\\xi_t) \\rangle - \\gamma_t^2 \\|\n\\mathcal{G}(x_t;\\xi_t) \\|^2.\n\\end{align*}\nSince $2\\langle u , v \\rangle - \\|v\\|^2 \\leq \\|u\\|^2$ for\nall $v \\in \\rset^n$ and using that $ F(x_t,\\xi_t)  =  f(x_t,\\xi_t) +  g(x_t, \\xi_t)$ and $\\nabla F(x_t,\\xi_t)  = \\nabla f(x_t,\\xi_t) + \\nabla g(x_t, \\xi_t)$, we further get:\n\\begin{align*}\n\\| x_{t+1} - \\overline{x}_{t+1} \\|^2 & \\leq \\| x_{t} -\n\\overline{x}_{t} \\|^2\n - 2 \\gamma_t \\left(  F(x_t,\\xi_t)  - F(\\overline{x}_{t},\\xi_t) \\right)  +  \\gamma_t^2 \\| \\nabla F(x_t,\\xi_t)\\|^2.\n\\end{align*}\nNote that $x_t$ depends only on history $\\xi_{[t-1]} =\n\\{\\xi_0,\\ldots, \\xi_{t-1}\\} $, not $\\xi_t$. It follows from the\nbasic property of  conditional expectation that:\n\\begin{align*}\n& \\E_{\\xi_{[t]}} [\\|x_{t+1}  - \\overline{x}_{t+1}\\|^2] \\\\\n& \\leq \\E_{\\xi_{[t-1]}}[\\|x_{t} - \\overline{x}_{t}\\|^2]  - 2 \\gamma_t  \\E_{\\xi_{[t-1]}}\n\\left[ \\E_{\\xi_t} [ F(x_t,\\xi_t) - F(\\overline{x}_{t},\\xi_t)  | \\xi_{[t-1]}\n]\\right] \\\\\n& \\qquad +  \\gamma_t^2 \\E_{\\xi_{[t]}}  \\left[\\| \\nabla F(x_t,\\xi_t) \\|^2 ] \\right]\\\\\n& =  \\E_{\\xi_{[t-1]}}[\\|x_{t} \\!- \\overline{x}_{t}\\|^2]  - 2\n\\gamma_t \\E_{\\xi_{[t-1]}} \\left[ F(x_t) - F(\\overline{x}_{t})\\right]\n+ \\gamma_t^2 \\E_{\\xi_{[t]}}  \\left[\\| \\nabla F(x_t,\\xi_t) \\|^2 \\right].\n\\end{align*}\nFurther, making use of the stochastic bounded gradient condition given in  Assumption\n\\ref{assumption1} in the previous relation we get:\n\\begin{align*}\n& \\E_{\\xi_{[t]}} [\\|x_{t+1}  - \\overline{x}_{t+1}\\|^2] \\\\\n& \\leq \\E_{\\xi_{[t-1]}}[\\|x_{t} \\!- \\overline{x}_{t}\\|^2]  - 2\n\\gamma_t \\E_{\\xi_{[t-1]}} \\left[ F(x_t) - F(\\overline{x}_{t})\\right]\n+ \\gamma_t^2 \\E_{\\xi_{[t]}}  \\left[\\| \\nabla F(x_t,\\xi_t) \\|^2\n\\right] \\\\\n& \\overset{\\eqref{as:main1_spg}}{\\leq} \\! \\E_{\\xi_{[t-1]}}[\\|x_{t}\n- \\overline{x}_{t}\\|^2]  - 2 \\gamma_t  \\E_{\\xi_{[t-1]}} \\left[\nF(x_t) - F(\\overline{x}_{t})\\right] \\\\\n& \\qquad + \\gamma_t^2 \\left( B^2 + L\n\\E_{\\xi_{[t-1]}} \\!\\left[F(x_t) - F(\\overline{x}_{t})\\right] \\right)\n\\\\\n& = \\E_{\\xi_{[t-1]}}[\\|x_{t} \\!- \\overline{x}_{t}\\|^2]  - \\gamma_t\n(2 - \\gamma_t L) \\E_{\\xi_{[t-1]}} \\left[ F(x_t) -\nF(\\overline{x}_{t})\\right] + \\gamma_t^2 B^2,\n\\end{align*}\nwhich, omitting the dependence  of expectation on $\\xi_{[t]}$,  proves the  statement of the theorem. \\qed\n\\end{proof}","sha256_utf8":"f84f57429c63da7ceef719cae5bed0ccfd17bdebf35318ab96e7998d28558430"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"展开投影距离平方","source_units":[{"start":26201,"end":26831,"start_line":330,"end_line":340,"raw_tex":"\\begin{proof}\nFrom the definition of SPG iteration we\nhave:\n\\begin{align}\n\\label{der_spg_basic}\n& \\| x_{t+1} - \\overline{x}_{t+1} \\|^2  \\leq   \\| x_{t+1} - \\overline{x}_{t} \\|^2 =  \\| x_{t} - \\overline{x}_{t} - \\gamma_t \\mathcal{G}(x_t;\\xi_t)\\|^2 \\\\\n& =  \\| x_{t} - \\overline{x}_{t} \\|^2  - 2 \\gamma_t \\langle \\mathcal{G}(x_t;\\xi_t), x_t  - \\overline{x}_{t} \\rangle + \\gamma_t^2 \\| \\mathcal{G}(x_t;\\xi_t) \\|^2  \\nonumber  \\\\\n& = \\| x_{t} - \\overline{x}_{t} \\|^2  - 2 \\gamma_t \\langle \\nabla\nf(x_t,\\xi_t) + \\nabla g(x_{t+1},\\xi_t), x_t  - \\overline{x}_{t} \\rangle +\n\\gamma_t^2 \\| \\mathcal{G}(x_t;\\xi_t) \\|^2.\\nonumber \n\\end{align}\n","sha256_utf8":"fb5f20fd33c08a34efba3184d386d26aea39305f4f36405b3f200d2b22b92a6a"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"应用首项凸性","source_units":[{"start":26831,"end":27015,"start_line":341,"end_line":343,"raw_tex":"Now, we refine the second term. First, from convexity of $f$ we\nhave:\n\\[ \\langle \\nabla f(x_t,\\xi_t), x_t  - \\overline{x}_{t} \\rangle \\geq f(x_t,\\xi_t) - f(\\overline{x}_{t},\\xi_t). \\]\n","sha256_utf8":"1eac2a73a416f920627f28fe3189f9c53f276c0969af5073829de730acbcefa9"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"估计正则项内积","source_units":[{"start":27015,"end":27707,"start_line":344,"end_line":352,"raw_tex":"Then, from  convexity of $g$ and the definition of  stochastic\ngradient mapping $\\mathcal{G}(\\cdot)$, we have:\n\\begin{align*}\n&\\langle \\nabla g(x_{t+1},\\xi_t), x_t  - \\overline{x}_{t} \\rangle  = \\langle  \\nabla g(x_{t+1},\\xi_t), x_t  - x_{t+1} \\rangle + \\langle  \\nabla g(x_{t+1},\\xi_t), x_{t+1}  - \\overline{x}_{t} \\rangle \\\\\n& \\geq \\gamma_t \\| \\mathcal{G}(x_t;\\xi_t) \\|^2 - \\gamma_t \\langle \\nabla f(x_t,\\xi_t), \\mathcal{G}(x_t;\\xi_t) \\rangle + g(x_{t+1}, \\xi_t) - g(\\overline{x}_t,\\xi_t)\\\\\n& \\geq \\gamma_t \\| \\mathcal{G}(x_t;\\xi_t) \\|^2 - \\gamma_t \\langle\n\\nabla f(x_t,\\xi_t) + \\nabla g(x_t,\\xi_t), \\mathcal{G}(x_t;\\xi_t) \\rangle +\ng(x_{t}, \\xi_t) - g(\\overline{x}_t, \\xi_t).\n\\end{align*}\n","sha256_utf8":"31adf5d2aac4307a5f43249ce288df75f9f026f6acd73a286a8237fcef638424"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"合并两个凸性估计","source_units":[{"start":27707,"end":28160,"start_line":353,"end_line":362,"raw_tex":"Replacing the previous two inequalities in \\eqref{der_spg_basic}, we\nobtain:\n\\begin{align*}\n\\| x_{t+1} - \\overline{x}_{t+1} \\|^2 & \\leq  \\| x_{t} -\n\\overline{x}_{t} \\|^2\n - 2 \\gamma_t  \\left( f(x_t,\\xi_t) + g(x_t, \\xi_t) - f(\\overline{x}_{t},\\xi_t) - g(\\overline{x}_{t}, \\xi_t)\\right)  \\\\\n& \\quad + 2 \\gamma_t^2 \\langle \\nabla f(x_t,\\xi_t) + \\nabla g(x_t, \\xi_t),\n\\mathcal{G}(x_t;\\xi_t) \\rangle - \\gamma_t^2 \\|\n\\mathcal{G}(x_t;\\xi_t) \\|^2.\n\\end{align*}\n","sha256_utf8":"015d4214c0839d1745710418efd3cb258c52d936773b5bf16516cfd6b845ac31"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"配方消去梯度映射","source_units":[{"start":28160,"end":28612,"start_line":363,"end_line":369,"raw_tex":"Since $2\\langle u , v \\rangle - \\|v\\|^2 \\leq \\|u\\|^2$ for\nall $v \\in \\rset^n$ and using that $ F(x_t,\\xi_t)  =  f(x_t,\\xi_t) +  g(x_t, \\xi_t)$ and $\\nabla F(x_t,\\xi_t)  = \\nabla f(x_t,\\xi_t) + \\nabla g(x_t, \\xi_t)$, we further get:\n\\begin{align*}\n\\| x_{t+1} - \\overline{x}_{t+1} \\|^2 & \\leq \\| x_{t} -\n\\overline{x}_{t} \\|^2\n - 2 \\gamma_t \\left(  F(x_t,\\xi_t)  - F(\\overline{x}_{t},\\xi_t) \\right)  +  \\gamma_t^2 \\| \\nabla F(x_t,\\xi_t)\\|^2.\n\\end{align*}\n","sha256_utf8":"6f5186cbf2593ff38b7a983ae633570e13525a52bb81c60eb336ec7337e5fef8"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"按样本历史取条件期望","source_units":[{"start":28612,"end":29329,"start_line":370,"end_line":382,"raw_tex":"Note that $x_t$ depends only on history $\\xi_{[t-1]} =\n\\{\\xi_0,\\ldots, \\xi_{t-1}\\} $, not $\\xi_t$. It follows from the\nbasic property of  conditional expectation that:\n\\begin{align*}\n& \\E_{\\xi_{[t]}} [\\|x_{t+1}  - \\overline{x}_{t+1}\\|^2] \\\\\n& \\leq \\E_{\\xi_{[t-1]}}[\\|x_{t} - \\overline{x}_{t}\\|^2]  - 2 \\gamma_t  \\E_{\\xi_{[t-1]}}\n\\left[ \\E_{\\xi_t} [ F(x_t,\\xi_t) - F(\\overline{x}_{t},\\xi_t)  | \\xi_{[t-1]}\n]\\right] \\\\\n& \\qquad +  \\gamma_t^2 \\E_{\\xi_{[t]}}  \\left[\\| \\nabla F(x_t,\\xi_t) \\|^2 ] \\right]\\\\\n& =  \\E_{\\xi_{[t-1]}}[\\|x_{t} \\!- \\overline{x}_{t}\\|^2]  - 2\n\\gamma_t \\E_{\\xi_{[t-1]}} \\left[ F(x_t) - F(\\overline{x}_{t})\\right]\n+ \\gamma_t^2 \\E_{\\xi_{[t]}}  \\left[\\| \\nabla F(x_t,\\xi_t) \\|^2 \\right].\n\\end{align*}\n","sha256_utf8":"93692389d968e9e31c93280c471dc80e692612d9c4f5db54a7ef6bf404c18908"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"代入随机梯度界","source_units":[{"start":29329,"end":30316,"start_line":383,"end_line":402,"raw_tex":"Further, making use of the stochastic bounded gradient condition given in  Assumption\n\\ref{assumption1} in the previous relation we get:\n\\begin{align*}\n& \\E_{\\xi_{[t]}} [\\|x_{t+1}  - \\overline{x}_{t+1}\\|^2] \\\\\n& \\leq \\E_{\\xi_{[t-1]}}[\\|x_{t} \\!- \\overline{x}_{t}\\|^2]  - 2\n\\gamma_t \\E_{\\xi_{[t-1]}} \\left[ F(x_t) - F(\\overline{x}_{t})\\right]\n+ \\gamma_t^2 \\E_{\\xi_{[t]}}  \\left[\\| \\nabla F(x_t,\\xi_t) \\|^2\n\\right] \\\\\n& \\overset{\\eqref{as:main1_spg}}{\\leq} \\! \\E_{\\xi_{[t-1]}}[\\|x_{t}\n- \\overline{x}_{t}\\|^2]  - 2 \\gamma_t  \\E_{\\xi_{[t-1]}} \\left[\nF(x_t) - F(\\overline{x}_{t})\\right] \\\\\n& \\qquad + \\gamma_t^2 \\left( B^2 + L\n\\E_{\\xi_{[t-1]}} \\!\\left[F(x_t) - F(\\overline{x}_{t})\\right] \\right)\n\\\\\n& = \\E_{\\xi_{[t-1]}}[\\|x_{t} \\!- \\overline{x}_{t}\\|^2]  - \\gamma_t\n(2 - \\gamma_t L) \\E_{\\xi_{[t-1]}} \\left[ F(x_t) -\nF(\\overline{x}_{t})\\right] + \\gamma_t^2 B^2,\n\\end{align*}\nwhich, omitting the dependence  of expectation on $\\xi_{[t]}$,  proves the  statement of the theorem. \\qed\n\\end{proof}","sha256_utf8":"3698224ea02a2566f72608d15f418d26a1c7ac076c4bb07bcbec0adb518378f4"}]},{"id":"S025","semantic_type":"proof","kind":"proof","title":"近端三点不等式推导","source_units":[{"start":32384,"end":33148,"start_line":432,"end_line":441,"raw_tex":"We first recall that  any strongly convex function $h$ with\nconvexity constant $\\sigma_h$ and having the optimal point $x^*$\nsatisfies the following inequality~\\cite{Nes:04}: $h(x) \\geq  h(x^*) + \\frac{\\sigma_h}{2}\\|x - x^*\\|^2 \\quad \\forall x$. Since for any convex function $h$ and fixed  point $x$, the Moreau smoothing  function  $h_\\gamma(y;x) = h(y) +  \\frac{1}{2 \\gamma} \\|y -x\\|^2$ is\nstrongly convex in the first argument $y$ with strong convexity constant\n$1/\\gamma$, the following holds for all $x, y \\in \\rset^n$:\n\\begin{align}\n\\label{rel1_prox} \n& h(y)  \\!+\\!  \\frac{1}{2 \\gamma} \\|y - x\\|^2   \\!\\geq\\!  h(\\prox_{\\gamma h}(x)) \\!+\\! \\frac{1}{2 \\gamma} \\|\\prox_{\\gamma h}(x)\n- x\\|^2 \\!+\\! \\frac{1}{2 \\gamma} \\|\\prox_{\\gamma h}(x) - y\\|^2. \n\\end{align}\n","sha256_utf8":"f7d16a4867c7631764bbad84fdf6cce39fc459e154f1564ccef284236f4927f4"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"利用近端目标强凸性","source_units":[{"start":32384,"end":33148,"start_line":432,"end_line":441,"raw_tex":"We first recall that  any strongly convex function $h$ with\nconvexity constant $\\sigma_h$ and having the optimal point $x^*$\nsatisfies the following inequality~\\cite{Nes:04}: $h(x) \\geq  h(x^*) + \\frac{\\sigma_h}{2}\\|x - x^*\\|^2 \\quad \\forall x$. Since for any convex function $h$ and fixed  point $x$, the Moreau smoothing  function  $h_\\gamma(y;x) = h(y) +  \\frac{1}{2 \\gamma} \\|y -x\\|^2$ is\nstrongly convex in the first argument $y$ with strong convexity constant\n$1/\\gamma$, the following holds for all $x, y \\in \\rset^n$:\n\\begin{align}\n\\label{rel1_prox} \n& h(y)  \\!+\\!  \\frac{1}{2 \\gamma} \\|y - x\\|^2   \\!\\geq\\!  h(\\prox_{\\gamma h}(x)) \\!+\\! \\frac{1}{2 \\gamma} \\|\\prox_{\\gamma h}(x)\n- x\\|^2 \\!+\\! \\frac{1}{2 \\gamma} \\|\\prox_{\\gamma h}(x) - y\\|^2. \n\\end{align}\n","sha256_utf8":"f7d16a4867c7631764bbad84fdf6cce39fc459e154f1564ccef284236f4927f4"}]},{"id":"S027","semantic_type":"theorem","kind":"theorem","title":"近端点下降递推","source_units":[{"start":33297,"end":34069,"start_line":447,"end_line":454,"raw_tex":"\\begin{theorem}\n\\label{th:spp_basic} Let $f(\\cdot, \\xi)$ and $g(\\cdot, \\xi)$ be convex functions  that  admit easily computable proximal operators. Additionally, assume that the  stochastic bounded gradient condition from Assumption \\ref{assumption1} holds and $g(\\cdot, \\xi)$ have bounded subgradients, that is there exists $B_{g}>0$ such that $ \\|\\nabla g(x, \\xi)\\| \\leq B_g$ for all $x$ and $\\xi$. Then, for any stepsize $\\gamma_t>0$ we have the following recursive inequality for the SPP iteration:\n\\begin{align}\n\\label{spp_basic} & \\Exp{\\|x_{t+1} - \\overline{x}_{t+1} \\|^2} \\leq \\\\\n&  \\Exp{\\| x_t - \\overline{x}_{t} \\|^2}   - \\gamma_t (2 - \\gamma_t L) \\Exp{ F(x_t) -\nF(\\overline{x}_{t})}  + \\gamma_t^2 \\left( B^2 + B_{g}^2 \\right). \\nonumber\n\\end{align}\n\\end{theorem}","sha256_utf8":"7e10de33c998e0d34bc42be98d9ca22a52c76efffc16ca6b24b8688521a19edc"}]},{"id":"S028","semantic_type":"proof","kind":"proof","title":"近端点递推完整证明","source_units":[{"start":34071,"end":36511,"start_line":456,"end_line":497,"raw_tex":"\\begin{proof}\nUsing \\eqref{rel1_prox} for the function $h=f(\\cdot,\\xi_t)$ with $y\n= \\overline{x}_{t}$ and $x = x_t$ we get:\n\\[  f(\\overline{x}_{t},\\xi_t) + \\frac{1}{2 \\gamma_t} \\|x_t - \\overline{x}_{t}\\|^2 \\geq\nf(x_{t+1/2},\\xi_t) + \\frac{1}{2 \\gamma_t} \\|x_{t+1/2} - x_t\\|^2 +\n\\frac{1}{2 \\gamma_t} \\|x_{t+1/2} - \\overline{x}_{t}\\|^2. \\] Using\nagain \\eqref{rel1_prox} for  function $h=g(\\cdot,\\xi_t)$ with $y = \\overline{x}_{t}$\nand $x = x_{t+1/2}$ we get:\n\\[  g(\\overline{x}_{t},\\xi_t) + \\frac{1}{2 \\gamma_t} \\|x_{t+1/2} - \\overline{x}_{t}\\|^2 \\geq\ng(x_{t+1},\\xi_t) + \\frac{1}{2 \\gamma_t} \\|x_{t+1} - x_{t+1/2}\\|^2 +\n\\frac{1}{2 \\gamma_t} \\|x_{t+1} - \\overline{x}_{t}\\|^2. \\] Adding the\nprevious two inequalities  and using the convexity of $f(\\cdot,\\xi_t)$\nand $g(\\cdot,\\xi_t)$, we get:\n\\begin{align*}\n& f(\\overline{x}_{t},\\xi_t) +  g(\\overline{x}_{t},\\xi_t) + \\frac{1}{2\n\\gamma_t} \\|x_t - \\overline{x}_{t}\\|^2 -\n\\frac{1}{2 \\gamma_t} \\|x_{t+1} - \\overline{x}_{t}\\|^2\\\\\n& \\geq f(x_{t+1/2},\\xi_t) + g(x_{t+1},\\xi_t) + \\frac{1}{2 \\gamma_t}\n\\|x_{t+1/2} - x_t\\|^2 + \\frac{1}{2 \\gamma_t} \\|x_{t+1} -\nx_{t+1/2}\\|^2\n\\\\\n& \\geq  f(x_t,\\xi_t) + g(x_t,\\xi_t)  +  \\langle \\nabla f(x_t,\\xi_t) +\n\\nabla g(x_t,\\xi_t) , x_{t+1/2} - x_t \\rangle \n\\\\\n&\\qquad + \\frac{1}{2 \\gamma_t} \\|x_{t+1/2} -\nx_t\\|^2 + \\langle \\nabla g(x_t,\\xi_t), x_{t+1}\n- x_{t+1/2}  \\rangle + \\frac{1}{2 \\gamma_t} \\|x_{t+1} -\nx_{t+1/2}\\|^2\\\\\n& \\geq f(x_t,\\xi_t) + g(x_t,\\xi_t)  -\\frac{\\gamma_t}{2} \\left( \\| \\nabla\nf(x_t,\\xi_t) + \\nabla g(x_t,\\xi_t) \\|^2 + \\|\\nabla g(x_t,\\xi_t)\\|^2 \\right),\n\\end{align*}\nwhere in the last inequality we used that $\\langle \\alpha,z \\rangle\n + \\frac{1}{2\\gamma} \\|z\\|^2 \\geq - \\frac{\\gamma}{2} \\|\\alpha\\|^2$ for all\n$z$. Taking now  expectation and using that $\\|x_{t+1} -\n\\overline{x}_{t}\\|^2 \\geq \\|x_{t+1} - \\overline{x}_{t+1}\\|^2$ we get:\n\\begin{align*}\n& \\E_{\\xi_{[t]}} [\\|x_{t+1} - \\overline{x}_{t+1} \\|^2] \\leq  \\E_{\\xi_{[t-1]}} [\\| x_t - \\overline{x}_{t} \\|^2   - 2 \\gamma_t  \\E_{\\xi_{[t-1]}}[ F(x_t) - F(\\overline{x}_{t})]  \\\\ \n& \\quad + \\gamma_t^2 \\E_{\\xi_{[t]}} [ \\| \\nabla\nF(x_t,\\xi_t)   \\|^2 + \\| \\nabla g(x_t,\\xi_t) \\|^2 ],\n\\end{align*}\nwhere we used basic properties of the conditional expectation as in the proof of Theorem \\ref{th:spg_basic}. Further, using the stochastic bounded gradient condition \\eqref{as:main1_spg}  and  omitting the dependence  of expectation on $\\xi_{[t]}$, we get  the statement of the theorem.  \\qed\n\\end{proof}","sha256_utf8":"3e3eec4055520858068764cfbd10d7f513147a7ffbff4e8110bdb0d1766210b5"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"分别应用两次近端不等式","source_units":[{"start":34071,"end":34757,"start_line":456,"end_line":466,"raw_tex":"\\begin{proof}\nUsing \\eqref{rel1_prox} for the function $h=f(\\cdot,\\xi_t)$ with $y\n= \\overline{x}_{t}$ and $x = x_t$ we get:\n\\[  f(\\overline{x}_{t},\\xi_t) + \\frac{1}{2 \\gamma_t} \\|x_t - \\overline{x}_{t}\\|^2 \\geq\nf(x_{t+1/2},\\xi_t) + \\frac{1}{2 \\gamma_t} \\|x_{t+1/2} - x_t\\|^2 +\n\\frac{1}{2 \\gamma_t} \\|x_{t+1/2} - \\overline{x}_{t}\\|^2. \\] Using\nagain \\eqref{rel1_prox} for  function $h=g(\\cdot,\\xi_t)$ with $y = \\overline{x}_{t}$\nand $x = x_{t+1/2}$ we get:\n\\[  g(\\overline{x}_{t},\\xi_t) + \\frac{1}{2 \\gamma_t} \\|x_{t+1/2} - \\overline{x}_{t}\\|^2 \\geq\ng(x_{t+1},\\xi_t) + \\frac{1}{2 \\gamma_t} \\|x_{t+1} - x_{t+1/2}\\|^2 +\n\\frac{1}{2 \\gamma_t} \\|x_{t+1} - \\overline{x}_{t}\\|^2. \\] Adding the\n","sha256_utf8":"42cdd4c41103000b67739c1870701c0990500a594d966f22d860666a252f28f6"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"合并凸性与平方项","source_units":[{"start":34757,"end":35629,"start_line":467,"end_line":486,"raw_tex":"previous two inequalities  and using the convexity of $f(\\cdot,\\xi_t)$\nand $g(\\cdot,\\xi_t)$, we get:\n\\begin{align*}\n& f(\\overline{x}_{t},\\xi_t) +  g(\\overline{x}_{t},\\xi_t) + \\frac{1}{2\n\\gamma_t} \\|x_t - \\overline{x}_{t}\\|^2 -\n\\frac{1}{2 \\gamma_t} \\|x_{t+1} - \\overline{x}_{t}\\|^2\\\\\n& \\geq f(x_{t+1/2},\\xi_t) + g(x_{t+1},\\xi_t) + \\frac{1}{2 \\gamma_t}\n\\|x_{t+1/2} - x_t\\|^2 + \\frac{1}{2 \\gamma_t} \\|x_{t+1} -\nx_{t+1/2}\\|^2\n\\\\\n& \\geq  f(x_t,\\xi_t) + g(x_t,\\xi_t)  +  \\langle \\nabla f(x_t,\\xi_t) +\n\\nabla g(x_t,\\xi_t) , x_{t+1/2} - x_t \\rangle \n\\\\\n&\\qquad + \\frac{1}{2 \\gamma_t} \\|x_{t+1/2} -\nx_t\\|^2 + \\langle \\nabla g(x_t,\\xi_t), x_{t+1}\n- x_{t+1/2}  \\rangle + \\frac{1}{2 \\gamma_t} \\|x_{t+1} -\nx_{t+1/2}\\|^2\\\\\n& \\geq f(x_t,\\xi_t) + g(x_t,\\xi_t)  -\\frac{\\gamma_t}{2} \\left( \\| \\nabla\nf(x_t,\\xi_t) + \\nabla g(x_t,\\xi_t) \\|^2 + \\|\\nabla g(x_t,\\xi_t)\\|^2 \\right),\n\\end{align*}\n","sha256_utf8":"d4f2f7e9b1d931ec4e321e077dcbe92ad155e82d2dae319d40fd99fac3ffc9de"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"配方说明并取期望完成递推","source_units":[{"start":35629,"end":36511,"start_line":487,"end_line":497,"raw_tex":"where in the last inequality we used that $\\langle \\alpha,z \\rangle\n + \\frac{1}{2\\gamma} \\|z\\|^2 \\geq - \\frac{\\gamma}{2} \\|\\alpha\\|^2$ for all\n$z$. Taking now  expectation and using that $\\|x_{t+1} -\n\\overline{x}_{t}\\|^2 \\geq \\|x_{t+1} - \\overline{x}_{t+1}\\|^2$ we get:\n\\begin{align*}\n& \\E_{\\xi_{[t]}} [\\|x_{t+1} - \\overline{x}_{t+1} \\|^2] \\leq  \\E_{\\xi_{[t-1]}} [\\| x_t - \\overline{x}_{t} \\|^2   - 2 \\gamma_t  \\E_{\\xi_{[t-1]}}[ F(x_t) - F(\\overline{x}_{t})]  \\\\ \n& \\quad + \\gamma_t^2 \\E_{\\xi_{[t]}} [ \\| \\nabla\nF(x_t,\\xi_t)   \\|^2 + \\| \\nabla g(x_t,\\xi_t) \\|^2 ],\n\\end{align*}\nwhere we used basic properties of the conditional expectation as in the proof of Theorem \\ref{th:spg_basic}. Further, using the stochastic bounded gradient condition \\eqref{as:main1_spg}  and  omitting the dependence  of expectation on $\\xi_{[t]}$, we get  the statement of the theorem.  \\qed\n\\end{proof}","sha256_utf8":"4bcacb2c540cf547d0fb61d89b4d6f5e04f13e3199c9eb1990e2b5aaccddc62b"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"增长条件控制目标差","source_units":[{"start":45523,"end":46079,"start_line":619,"end_line":630,"raw_tex":"\\begin{proof}\nFrom \\eqref{sfo_basic}  and Assumption \\ref{assumption2}  it follows that:\n\\begin{align*}\n& \\E_{\\xi_{[t]}} [\\|x_{t+1}  - \\overline{x}_{t+1}\\|^2] \\leq\n\\E_{\\xi_{[t-1]}}[\\|x_{t} - \\overline{x}_{t}\\|^2]  -  \\gamma_t (2 -\n\\gamma_t L) \\E_{\\xi_{[t-1]}}\n\\left[F(x_t) - F(\\overline{x}_{t})\\right]  \\\\\n& + {\\cal B}^2 \\gamma_{t}^2  \\overset{\\eqref{as:strong_spg}}{\\leq} \\E_{\\xi_{[t-1]}}[\\|x_{t} -\n\\overline{x}_{t}\\|^2]  - \\frac{\\mu \\gamma_t (2 - \\gamma_t L)}{2}\n\\E_{\\xi_{[t-1]}} [\\|x_{t} - \\overline{x}_{t}\\|^2] +  {\\cal B}^2\n\\gamma_{t}^2,\n\\end{align*}\n","sha256_utf8":"0759904578e9915c671a359b2964bb442f5700dcb566c09f398ec3f29f9a8ee1"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"得到距离仿射递推","source_units":[{"start":46079,"end":46413,"start_line":631,"end_line":638,"raw_tex":"provided that $\\gamma_t \\leq 2/L$.  In conclusion, we obtain the\nfollowing recursion:\n\\begin{align}\n\\label{recursive1_spg} \\E_{\\xi_{[t]}} [\\|x_{t+1} -\n\\overline{x}_{t+1}\\|^2] & \\leq   \\left(1 - \\mu \\gamma_t + \\frac{\\mu\nL \\gamma_t^2}{2} \\right) \\E_{\\xi_{[t-1]}} [\\|x_{t} -\n\\overline{x}_{t}\\|^2] +  {\\cal B}^2 \\gamma_{t}^2.\n\\end{align}\n","sha256_utf8":"8bb8c4a8691b80e9cd21429d32ce18b00b0815491a4ed201bdaa2c4486897c70"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"收缩情形求和几何级数","source_units":[{"start":46413,"end":47160,"start_line":639,"end_line":653,"raw_tex":"For constant stepsize $\\gamma_t = \\gamma < 2/L$ it follows that $ 1 - \\mu \\gamma\n+ \\mu L \\gamma^2/2 < 1$.  Then, combining inequality \\eqref{recursive1_spg} with\ncondition $1 - \\mu \\gamma + \\mu L\n\\gamma^2/2 > -1$, we get:\n\\begin{align*}\n\\E [\\|x_{t} - \\overline{x}_{t}\\|^2] & \\leq   |1 - \\mu \\gamma +\n\\frac{\\mu L \\gamma^2}{2} |^t \\|x_{0} - \\overline{x}_{0}\\|^2 +\n{\\cal B}^2 \\gamma^2 \\sum_{j=0}^{t-1} |1 - \\mu \\gamma +\n\\frac{\\mu L \\gamma^2}{2}|^j \\\\\n& \\leq |1 - \\mu \\gamma + \\frac{\\mu L \\gamma^2}{2} |^t\n\\|x_{0} - \\overline{x}_{0}\\|^2 +    \\frac{{\\cal B}^2 \\gamma^2}{ 1- |1-\\mu \\gamma\n+ \\mu L \\gamma^2/2|},\n\\end{align*}\nwhere in the second inequality we used that $\\sum_{j=0}^{t-1} c^j\n\\leq \\sum_{j=0}^{\\infty} c^j = \\frac{1}{1-c}$ for any $|c|<1$.\n","sha256_utf8":"0cb99e645527a43aeb45265b343fd348a20e2c749396d2f1d60154865051ebe8"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"处理非正递推系数","source_units":[{"start":47160,"end":47361,"start_line":654,"end_line":658,"raw_tex":"Otherwise, if $1 - \\mu \\gamma + \\mu L \\gamma^2/2 \\leq -1$, then from\n\\eqref{recursive1_spg} it follows that:\n\\begin{align*} \\E [\\|x_{t} - \\overline{x}_{t}\\|^2] &\n\\leq {\\cal B}^2 \\gamma^2.\n\\end{align*}\n","sha256_utf8":"a28ccddaad1ce1637fc7c1c12588a42e68f96707504ed535fd83c1530c61b82b"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"验证收缩条件与无噪声结论","source_units":[{"start":47361,"end":48175,"start_line":659,"end_line":661,"raw_tex":"Moreover, if $\\mu < 4L$, then the relation $1 - \\mu \\gamma + \\mu L\n\\gamma^2/2 > -1$ always holds, regardless of the value of $\\gamma$. Thus, if $\\mu < 4L$ and  the constant stepsize is chosen such $\\gamma < \\frac{2}{L}$,  then $|1 - \\mu \\gamma + \\mu L \\gamma^2| < 1$. Similarly, if  the  constant stepsize satisfies  $\\gamma < \\min(2/L, 2/\\mu)$, then again we have $0< 1 - \\mu \\gamma + \\mu L \\gamma^2 < 1$. Therefore, these two previous conditions guarantee linear convergence  to a noise dominated region whose radius is proportional to the square of the stepsize $\\gamma$.    Finally, the inequality \\eqref{recursive1_spg} with ${\\cal B}=0$ and $|1 - \\mu \\gamma + \\mu L \\gamma^2| < 1$  proves  linear convergence for $\\Exp{\\|x_{t} - \\overline{x}_{t} \\|^2}$, with $x_t$ generated by SFO methods.  \\qed\n\\end{proof}","sha256_utf8":"4ec3c9fe5022e129c78549ac2f57b240472998c9e98acf7794f739430012421c"}]},{"id":"S037","semantic_type":"external_fact","kind":"external_fact","title":"强凸极小点下界","source_units":[],"declared_external":true,"declaration_text":["A strongly convex function h with modulus sigma and minimizer x* satisfies h(x) >= h(x*) + (sigma/2)||x-x*||^2. For convex h and gamma>0, y -> h(y)+(1/(2gamma))||y-x||^2 is 1/gamma-strongly convex."]},{"id":"S038","semantic_type":"external_fact","kind":"external_fact","title":"凸次梯度与投影性质","source_units":[],"declared_external":true,"declaration_text":["For convex h and v in its subdifferential at x, h(y) >= h(x)+<v,y-x>. A proximal minimizer p satisfies (x-p)/gamma in the subdifferential of h at p. Projection onto the optimal set minimizes distance to that set."]},{"id":"S039","semantic_type":"external_fact","kind":"external_fact","title":"条件期望与独立采样","source_units":[],"declared_external":true,"declaration_text":["The tower property of conditional expectation and independence of a fresh i.i.d. sample from the iterate history permit replacement of conditional sampled function values by their expected objective values."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S003","target_id":"S019","kind":"supports"},{"source_id":"S003","target_id":"S020","kind":"supports"},{"source_id":"S003","target_id":"S022","kind":"supports"},{"source_id":"S003","target_id":"S023","kind":"supports"},{"source_id":"S003","target_id":"S030","kind":"supports"},{"source_id":"S004","target_id":"S008","kind":"supports"},{"source_id":"S004","target_id":"S012","kind":"supports"},{"source_id":"S004","target_id":"S026","kind":"supports"},{"source_id":"S005","target_id":"S018","kind":"supports"},{"source_id":"S005","target_id":"S029","kind":"supports"},{"source_id":"S005","target_id":"S031","kind":"supports"},{"source_id":"S006","target_id":"S024","kind":"supports"},{"source_id":"S006","target_id":"S031","kind":"supports"},{"source_id":"S007","target_id":"S032","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S018","kind":"supports"},{"source_id":"S008","target_id":"S023","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S018","kind":"supports"},{"source_id":"S009","target_id":"S020","kind":"supports"},{"source_id":"S010","target_id":"S018","kind":"supports"},{"source_id":"S010","target_id":"S020","kind":"supports"},{"source_id":"S011","target_id":"S023","kind":"supports"},{"source_id":"S011","target_id":"S031","kind":"supports"},{"source_id":"S012","target_id":"S029","kind":"supports"},{"source_id":"S012","target_id":"S031","kind":"supports"},{"source_id":"S013","target_id":"S034","kind":"supports"},{"source_id":"S014","target_id":"S032","kind":"supports"},{"source_id":"S015","target_id":"S033","kind":"supports"},{"source_id":"S016","target_id":"S014","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S021","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S031","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S017","kind":"supports"},{"source_id":"S020","target_id":"S017","kind":"supports"},{"source_id":"S021","target_id":"S017","kind":"supports"},{"source_id":"S022","target_id":"S017","kind":"supports"},{"source_id":"S023","target_id":"S017","kind":"supports"},{"source_id":"S024","target_id":"S017","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S029","kind":"supports"},{"source_id":"S027","target_id":"S014","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S027","target_id":"S001","kind":"supports"},{"source_id":"S016","target_id":"S001","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S028","kind":"supports"},{"source_id":"S031","target_id":"S028","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S033","target_id":"S036","kind":"supports"},{"source_id":"S034","target_id":"S036","kind":"supports"},{"source_id":"S032","target_id":"S002","kind":"supports"},{"source_id":"S033","target_id":"S002","kind":"supports"},{"source_id":"S034","target_id":"S002","kind":"supports"},{"source_id":"S035","target_id":"S002","kind":"supports"},{"source_id":"S036","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S037","target_id":"S026","kind":"supports"},{"source_id":"S038","target_id":"S010","kind":"supports"},{"source_id":"S038","target_id":"S018","kind":"supports"},{"source_id":"S038","target_id":"S019","kind":"supports"},{"source_id":"S038","target_id":"S020","kind":"supports"},{"source_id":"S038","target_id":"S030","kind":"supports"},{"source_id":"S038","target_id":"S031","kind":"supports"},{"source_id":"S039","target_id":"S023","kind":"supports"},{"source_id":"S039","target_id":"S031","kind":"supports"}]}}
{"record_id":"2004.00475v2","paper":{"title":"Stopping Criteria for, and Strong Convergence of, Stochastic Gradient Descent on Bottou-Curtis-Nocedal Functions","authors":["Vivak Patel"],"arxiv_id":"2004.00475","version":"v2","journal":"Mathematical Programming","doi":"10.1007/s10107-021-01710-6","abstract_url":"https://arxiv.org/abs/2004.00475v2","pdf_url":"https://arxiv.org/pdf/2004.00475v2"},"context":{"raw_tex":"In data-driven and simulation-based disciplines, the optimization problem\n\\begin{equation} \\label{problem}\n\\min_{\\theta} F(\\theta)\n\\end{equation}\n\nis frequently solved, \nwhere $F(\\theta) = \\E{ f(\\theta,X)}$; $f: \\mathbb{R}^p \\times \\mathbb{R}^d \\to \\mathbb{R}$; $X$ is a random variable with arbitrary support; and $\\mathbb{E}$ is the expectation operator. Depending on $X$, the optimization problem's objective function and its gradient may be impractical or impossible to evaluate directly \\cite{bertsekas2011}. Fortunately, when certain regularity conditions hold on $f$ and $X$ (e.g., \\cite{bottou2018,wu2009}), the optimization problem's structure is exploited to generate solvers that use the gradient of $f$ with respect to the argument $\\theta$ for independent copies of $X$ \\cite{robbins1951,chung1954}. Moreover, when the gradient of $f$ is significantly cheaper to compute than the gradient of $F$, the optimization problem can be efficiently solved using these so-called Stochastic Gradient Descent (SGD) methods \\cite{bertsekas2011}.\n\n\n\\nomenclature[A, 1]{$F$}{Objective function that maps $\\mathbb{R}^p \\to \\mathbb{R}$. Also referred to as ``Deterministic Objective.''}\n\\nomenclature[A, 2]{$\\dot F$}{Gradient function of the deterministic objective that maps $\\mathbb{R}^p \\to \\mathbb{R}^p$. Also referred to as ``Deterministic Gradient.''}\n\\nomenclature[A, 3]{$f$}{A function that maps $\\mathbb{R}^p \\times \\mathbb{R}^d \\to \\mathbb{R}$. Used to construct the deterministic objective, and is referred to as the ``Stochastic Objective.''}\n\\nomenclature[A, 4]{$\\dot f$}{Gradient function of the stochastic objective with respect to the first argument, and maps $\\mathbb{R}^p \\times \\mathbb{R}^d \\to \\mathbb{R}^p$. Also referred to as the ``Stochastic Gradient.''}\n\\nomenclature[A, 5]{$\\theta$}{The parameter to be optimized in the objective function. Takes value in $\\mathbb{R}^p$.}\n\\nomenclature[A, 6]{$\\norm{\\cdot}_2$}{If the argument is a vector, this refers to the Euclidean norm of the vector. If the argument is a matrix, this refers to the Operator 2-norm.}\n\\nomenclature[A, 7]{$\\lambda_{\\max}$}{The largest eigenvalue corresponding to a given symmetric, positive definite matrix argument.}\n\\nomenclature[A, 8]{$\\lambda_{\\min}$}{The smallest eigenvalue corresponding to the given symmetric, positive definite matrix argument.}\n\\nomenclature[A, 9]{$\\kappa $}{The 2-norm condition number of the given matrix.}\n\n\n\\nomenclature[B, 1]{$\\mathbb{P}$}{The probability operator with respect to the underlying Probability Space.}\n\\nomenclature[B, 2]{$\\mathbb{E}$}{The expectation operator corresponding to the underlying Probability Space.}\n\\nomenclature[B, 3]{$X, X_k$}{Independent, identically distributed random variables that map from the underlying Probability Space to $\\mathbb{R}^d$.}\n\n\n\\nomenclature[B, 7]{$i.o.$}{Infinitely often.}\n\\nomenclature[B, 8]{$\\mathbf{1}$}{The indicator function over a specified event.}\n\n\n\\condition{condition-noise-grad}{There exists a $C_1,C_2 \\geq 0$ such that, $\\forall \\theta \\geq 0$,\n$$ \\E{ \\norm{ \\dot{f}(\\theta,X) - \\dot{F}(\\theta)}_2^2} \\leq C_1 + C_2 \\norm{ \\dot{F}(\\theta)}_2^2.$$}{NM-4}\n\n\n\\condition{condition-dg-lipschitz}{There exista a $C > 0$ such that for all $\\theta_1,\\theta_2$, \n$$ \\norm{ \\dot{F}(\\theta_1) - \\dot{F}(\\theta_2)}_2 \\leq C \\norm{\\theta_1 - \\theta_2}_2.$$}{DG-2}\n\n\n\\condition{bcn}{The BCN conditions are\n\n\\begin{enumerate}\n\\item $F$ is bounded from below.\n\\item $\\dot{F}$ is Lipschitz continuous (\\ref{condition-dg-lipschitz}).\n\\item For all $\\theta \\in \\mathbb{R}^p$, $\\dot{F}(\\theta) = \\mathbb{E}[ \\dot{f}(\\theta,X)]$.\n\\item The stochastic gradients satisfy \\ref{condition-noise-grad}.\n\\end{enumerate}}{BCN}\n\n\n\\begin{definition}\nA function that satisfies the BCN conditions is referred to as a BCN function.\n\\end{definition}\n\nLet $\\lbrace X_k : k \\in \\mathbb{N} \\rbrace$ be independent and identically distributed random variables that have the same distribution as $X$. Let $\\beta_0$ be either a fixed or random quantity in $\\mathbb{R}^p$. Let $\\mathcal{F}_0 = \\sigma(\\beta_0)$ and $\\mathcal{F}_k = \\sigma(\\beta_0, X_1,\\ldots,X_k)$ denote the corresponding elements of the usual filtration. Define the Stochastic Gradient Descent iterates $\\lbrace \\beta_k : k \\in \\mathbb{N} \\rbrace$ recursively by\n\\begin{equation}\n\\beta_{k+1} = \\beta_k - M_k \\dot f (\\beta_k, X_{k+1}),\n\\end{equation} \nwhere $\\lbrace M_k \\rbrace$ are matrices that satisfy:\n\\condition{prop-sympd}{The matrices $\\lbrace M_k \\rbrace$ are symmetric and positive definite.}{P1}\n\\condition{prop-maxeig}{There exists an $S > 0$ such that\n$$\\sum_{k=0}^\\infty \\lambda_{\\max}(M_k)^2 < S,$$\nwhere $\\lambda_{\\max}(\\cdot)$ denotes the largest eigenvalue of the given matrix.}{P2}\n\\condition{prop-mineig}{The sum,\n$$\\sum_{k=0}^\\infty \\lambda_{\\min}(M_k),$$\ndiverges, where $\\lambda_{\\min}(\\cdot)$ is the smallest eigenvalue of the given matrix.}{P3}\n\\condition{prop-condnum}{The condition numbers of $\\lbrace M_k \\rbrace$ with respect to operator 2-norm, $\\lbrace \\kappa(M_k) \\rbrace$, satisfy\n$$ \\lim_{k \\to \\infty} \\lambda_{\\max}(M_k) \\kappa(M_k) = 0.$$\n}{P4}\n\n\n\\begin{remark}\nWe also note that all of the inequalities and equalities below hold with probability one, even if this is not explicitly stated.\n\\end{remark}\n\\begin{remark}\nWe also point out that the probabilities and expectations below should be conditional on $\\mathcal{F}_0$. However, to avoid the additional cumbersome notation, we will not explicitly state this.\n\\end{remark}\n\n\nOur first task will be to set the stage to the analogue of strong Markov property that will be relevant in analyzing the SGD method. Let $\\tau$ be a finite stopping time with respect to $\\lbrace \\mathcal{F}_k \\rbrace$; that is, $\\Prb{ \\tau < \\infty} = 1$. Moreover, as all of our arguments will be asymptotic in this section, we will assume also that $\\Prb{ \\tau \\geq K} = 1$ where $K \\in \\mathbb{N}$ is defined in the following lemma to satisfy the following property, which will be used later. \n\n\nNow, using $\\tau$, we will define a sequence of iterates $\\lbrace \\psi_k \\rbrace$ that we will eventually couple with $\\lbrace \\beta_k \\rbrace$. To define $\\lbrace \\psi_k \\rbrace$, let\n\\begin{enumerate}\n\\item $Z_k := X_{\\tau+1+k}$ for all $k \\in \\mathbb{N}$.\n\\item $\\psi_0 := \\beta_{\\tau + 1}$.\n\\item For all $k \\in \\lbrace 0 \\rbrace \\cup \\mathbb{N}$, $P_k := M_{\\tau+1+k}$ and\n\\begin{equation} \\label{eqn-psi-iter}\n\\psi_{k+1} := \\psi_k - P_k \\dot f( \\psi_k, Z_{k+1}) \\1{ \\norm{\\dot F(\\psi_k)}_2^2 > \\delta},\n\\end{equation}\nwhere $\\1{ \\cdot} $ is the indicator function for the given event.\n\\end{enumerate} \n\nThere are two properties of these quantities that are worth noting: $\\lbrace Z_k \\rbrace$ are independent and identically distributed; $\\lbrace P_k \\rbrace$ inherit the properties of $\\lbrace M_k \\rbrace$ with probability one. The former is verified by Theorem 4.1.3 of \\cite{durrett2010}, which states that $\\lbrace Z_k \\rbrace$ are mutually independent and independent of $\\mathcal{F}_{\\tau+1}$, and have the same distribution as $X_1$. The latter is verified by the following lemma.\n","units":[{"start":4951,"end":5096,"start_line":139,"end_line":142,"raw_tex":"In data-driven and simulation-based disciplines, the optimization problem\n\\begin{equation} \\label{problem}\n\\min_{\\theta} F(\\theta)\n\\end{equation}","sha256_utf8":"bc44877bae2066e61119d1eef600cfde3252e8d69d362bade444f1cf65dae8a8"},{"start":5097,"end":5997,"start_line":143,"end_line":144,"raw_tex":"is frequently solved, \nwhere $F(\\theta) = \\E{ f(\\theta,X)}$; $f: \\mathbb{R}^p \\times \\mathbb{R}^d \\to \\mathbb{R}$; $X$ is a random variable with arbitrary support; and $\\mathbb{E}$ is the expectation operator. Depending on $X$, the optimization problem's objective function and its gradient may be impractical or impossible to evaluate directly \\cite{bertsekas2011}. Fortunately, when certain regularity conditions hold on $f$ and $X$ (e.g., \\cite{bottou2018,wu2009}), the optimization problem's structure is exploited to generate solvers that use the gradient of $f$ with respect to the argument $\\theta$ for independent copies of $X$ \\cite{robbins1951,chung1954}. Moreover, when the gradient of $f$ is significantly cheaper to compute than the gradient of $F$, the optimization problem can be efficiently solved using these so-called Stochastic Gradient Descent (SGD) methods \\cite{bertsekas2011}.\n","sha256_utf8":"c7ea4f3a9c3c25b68fecf8b2f9ce4f14f09840a1dce3760ab00e585f11713cbc"},{"start":12433,"end":13811,"start_line":168,"end_line":176,"raw_tex":"\\nomenclature[A, 1]{$F$}{Objective function that maps $\\mathbb{R}^p \\to \\mathbb{R}$. Also referred to as ``Deterministic Objective.''}\n\\nomenclature[A, 2]{$\\dot F$}{Gradient function of the deterministic objective that maps $\\mathbb{R}^p \\to \\mathbb{R}^p$. Also referred to as ``Deterministic Gradient.''}\n\\nomenclature[A, 3]{$f$}{A function that maps $\\mathbb{R}^p \\times \\mathbb{R}^d \\to \\mathbb{R}$. Used to construct the deterministic objective, and is referred to as the ``Stochastic Objective.''}\n\\nomenclature[A, 4]{$\\dot f$}{Gradient function of the stochastic objective with respect to the first argument, and maps $\\mathbb{R}^p \\times \\mathbb{R}^d \\to \\mathbb{R}^p$. Also referred to as the ``Stochastic Gradient.''}\n\\nomenclature[A, 5]{$\\theta$}{The parameter to be optimized in the objective function. Takes value in $\\mathbb{R}^p$.}\n\\nomenclature[A, 6]{$\\norm{\\cdot}_2$}{If the argument is a vector, this refers to the Euclidean norm of the vector. If the argument is a matrix, this refers to the Operator 2-norm.}\n\\nomenclature[A, 7]{$\\lambda_{\\max}$}{The largest eigenvalue corresponding to a given symmetric, positive definite matrix argument.}\n\\nomenclature[A, 8]{$\\lambda_{\\min}$}{The smallest eigenvalue corresponding to the given symmetric, positive definite matrix argument.}\n\\nomenclature[A, 9]{$\\kappa $}{The 2-norm condition number of the given matrix.}\n","sha256_utf8":"952e0c96152edb0e09210c82c84da2ea1da49a3712e3bd381672f872a089c1b5"},{"start":13812,"end":14184,"start_line":178,"end_line":180,"raw_tex":"\\nomenclature[B, 1]{$\\mathbb{P}$}{The probability operator with respect to the underlying Probability Space.}\n\\nomenclature[B, 2]{$\\mathbb{E}$}{The expectation operator corresponding to the underlying Probability Space.}\n\\nomenclature[B, 3]{$X, X_k$}{Independent, identically distributed random variables that map from the underlying Probability Space to $\\mathbb{R}^d$.}\n","sha256_utf8":"7daa64f25255994a43a29db9ab85fe2147205f8c45facd4d38221488d5748b39"},{"start":14498,"end":14627,"start_line":184,"end_line":185,"raw_tex":"\\nomenclature[B, 7]{$i.o.$}{Infinitely often.}\n\\nomenclature[B, 8]{$\\mathbf{1}$}{The indicator function over a specified event.}\n","sha256_utf8":"0da2161f1074d0940a05a428d49044990b7036dddac6e7949e3d2b4b92eb09a7"},{"start":41287,"end":41497,"start_line":397,"end_line":398,"raw_tex":"\\condition{condition-noise-grad}{There exists a $C_1,C_2 \\geq 0$ such that, $\\forall \\theta \\geq 0$,\n$$ \\E{ \\norm{ \\dot{f}(\\theta,X) - \\dot{F}(\\theta)}_2^2} \\leq C_1 + C_2 \\norm{ \\dot{F}(\\theta)}_2^2.$$}{NM-4}\n","sha256_utf8":"bb5c951a2d3dcfb0840c999c29951e9e966a7db3294c2572610c664bc0249fbe"},{"start":43330,"end":43525,"start_line":418,"end_line":419,"raw_tex":"\\condition{condition-dg-lipschitz}{There exista a $C > 0$ such that for all $\\theta_1,\\theta_2$, \n$$ \\norm{ \\dot{F}(\\theta_1) - \\dot{F}(\\theta_2)}_2 \\leq C \\norm{\\theta_1 - \\theta_2}_2.$$}{DG-2}\n","sha256_utf8":"7ace40d26979dabf314984d1cb9d1aa1d1a3aaf48f31b409d749c269940a974c"},{"start":43865,"end":44210,"start_line":425,"end_line":432,"raw_tex":"\\condition{bcn}{The BCN conditions are\n\n\\begin{enumerate}\n\\item $F$ is bounded from below.\n\\item $\\dot{F}$ is Lipschitz continuous (\\ref{condition-dg-lipschitz}).\n\\item For all $\\theta \\in \\mathbb{R}^p$, $\\dot{F}(\\theta) = \\mathbb{E}[ \\dot{f}(\\theta,X)]$.\n\\item The stochastic gradients satisfy \\ref{condition-noise-grad}.\n\\end{enumerate}}{BCN}\n","sha256_utf8":"b31e92963479a929712a7c4469fa52e8dfac2eda14bf96c576d8b0bb1dfedcf4"},{"start":44211,"end":44325,"start_line":434,"end_line":436,"raw_tex":"\\begin{definition}\nA function that satisfies the BCN conditions is referred to as a BCN function.\n\\end{definition}","sha256_utf8":"d88f293af82c7fd0313ad38d4cf9cbe52edf929def9b55a8e5544a83c71fb2b8"},{"start":45476,"end":46768,"start_line":446,"end_line":460,"raw_tex":"Let $\\lbrace X_k : k \\in \\mathbb{N} \\rbrace$ be independent and identically distributed random variables that have the same distribution as $X$. Let $\\beta_0$ be either a fixed or random quantity in $\\mathbb{R}^p$. Let $\\mathcal{F}_0 = \\sigma(\\beta_0)$ and $\\mathcal{F}_k = \\sigma(\\beta_0, X_1,\\ldots,X_k)$ denote the corresponding elements of the usual filtration. Define the Stochastic Gradient Descent iterates $\\lbrace \\beta_k : k \\in \\mathbb{N} \\rbrace$ recursively by\n\\begin{equation}\n\\beta_{k+1} = \\beta_k - M_k \\dot f (\\beta_k, X_{k+1}),\n\\end{equation} \nwhere $\\lbrace M_k \\rbrace$ are matrices that satisfy:\n\\condition{prop-sympd}{The matrices $\\lbrace M_k \\rbrace$ are symmetric and positive definite.}{P1}\n\\condition{prop-maxeig}{There exists an $S > 0$ such that\n$$\\sum_{k=0}^\\infty \\lambda_{\\max}(M_k)^2 < S,$$\nwhere $\\lambda_{\\max}(\\cdot)$ denotes the largest eigenvalue of the given matrix.}{P2}\n\\condition{prop-mineig}{The sum,\n$$\\sum_{k=0}^\\infty \\lambda_{\\min}(M_k),$$\ndiverges, where $\\lambda_{\\min}(\\cdot)$ is the smallest eigenvalue of the given matrix.}{P3}\n\\condition{prop-condnum}{The condition numbers of $\\lbrace M_k \\rbrace$ with respect to operator 2-norm, $\\lbrace \\kappa(M_k) \\rbrace$, satisfy\n$$ \\lim_{k \\to \\infty} \\lambda_{\\max}(M_k) \\kappa(M_k) = 0.$$\n}{P4}\n","sha256_utf8":"3a7a62901f92e6757807f0f1a934ceeee4949ee390ea094f657a600eab226a0f"},{"start":51766,"end":52146,"start_line":490,"end_line":495,"raw_tex":"\\begin{remark}\nWe also note that all of the inequalities and equalities below hold with probability one, even if this is not explicitly stated.\n\\end{remark}\n\\begin{remark}\nWe also point out that the probabilities and expectations below should be conditional on $\\mathcal{F}_0$. However, to avoid the additional cumbersome notation, we will not explicitly state this.\n\\end{remark}\n","sha256_utf8":"640c6af40e0864dea5901e2a9fd3e85e518d8b474b5a1c1f28b0f967332187a6"},{"start":52200,"end":52697,"start_line":499,"end_line":499,"raw_tex":"Our first task will be to set the stage to the analogue of strong Markov property that will be relevant in analyzing the SGD method. Let $\\tau$ be a finite stopping time with respect to $\\lbrace \\mathcal{F}_k \\rbrace$; that is, $\\Prb{ \\tau < \\infty} = 1$. Moreover, as all of our arguments will be asymptotic in this section, we will assume also that $\\Prb{ \\tau \\geq K} = 1$ where $K \\in \\mathbb{N}$ is defined in the following lemma to satisfy the following property, which will be used later. \n","sha256_utf8":"1e2aa4e984015ac2c6415187295f4c95de8bf7e4f78a2bc49305dc0c6a246d42"},{"start":55446,"end":56541,"start_line":542,"end_line":553,"raw_tex":"Now, using $\\tau$, we will define a sequence of iterates $\\lbrace \\psi_k \\rbrace$ that we will eventually couple with $\\lbrace \\beta_k \\rbrace$. To define $\\lbrace \\psi_k \\rbrace$, let\n\\begin{enumerate}\n\\item $Z_k := X_{\\tau+1+k}$ for all $k \\in \\mathbb{N}$.\n\\item $\\psi_0 := \\beta_{\\tau + 1}$.\n\\item For all $k \\in \\lbrace 0 \\rbrace \\cup \\mathbb{N}$, $P_k := M_{\\tau+1+k}$ and\n\\begin{equation} \\label{eqn-psi-iter}\n\\psi_{k+1} := \\psi_k - P_k \\dot f( \\psi_k, Z_{k+1}) \\1{ \\norm{\\dot F(\\psi_k)}_2^2 > \\delta},\n\\end{equation}\nwhere $\\1{ \\cdot} $ is the indicator function for the given event.\n\\end{enumerate} \n\nThere are two properties of these quantities that are worth noting: $\\lbrace Z_k \\rbrace$ are independent and identically distributed; $\\lbrace P_k \\rbrace$ inherit the properties of $\\lbrace M_k \\rbrace$ with probability one. The former is verified by Theorem 4.1.3 of \\cite{durrett2010}, which states that $\\lbrace Z_k \\rbrace$ are mutually independent and independent of $\\mathcal{F}_{\\tau+1}$, and have the same distribution as $X_1$. The latter is verified by the following lemma.\n","sha256_utf8":"5f5b58a3c9dd15d1dd97d135bfeb0dfc358eee0051986ea415a35191974bb940"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{corollary} \\label{corollary-convergence}\nLet $F$ be a Bottou-Curtis-Nocedal function (\\ref{bcn}) and let $\\lbrace \\beta_k \\rbrace$ be the iterates generated by Stochastic Gradient Descent satisfying \\ref{prop-sympd} to \\ref{prop-condnum} (\\S \\ref{subsection-sgd}), then\n\\begin{equation}\n\\condPrb{ \\lim_{k \\to \\infty} \\norm{ \\dot{F} (\\beta_k)}_2 = 0}{\\mathcal{F}_0} = 1, ~\\text{with probability 1},\n\\end{equation}\nwhere $\\mathcal{F}_0 = \\sigma(\\beta_0)$.\n\\end{corollary}","units":[{"start":73868,"end":74344,"start_line":789,"end_line":795,"raw_tex":"\\begin{corollary} \\label{corollary-convergence}\nLet $F$ be a Bottou-Curtis-Nocedal function (\\ref{bcn}) and let $\\lbrace \\beta_k \\rbrace$ be the iterates generated by Stochastic Gradient Descent satisfying \\ref{prop-sympd} to \\ref{prop-condnum} (\\S \\ref{subsection-sgd}), then\n\\begin{equation}\n\\condPrb{ \\lim_{k \\to \\infty} \\norm{ \\dot{F} (\\beta_k)}_2 = 0}{\\mathcal{F}_0} = 1, ~\\text{with probability 1},\n\\end{equation}\nwhere $\\mathcal{F}_0 = \\sigma(\\beta_0)$.\n\\end{corollary}","sha256_utf8":"b9f42f4471f1ae2be0280d181ae92d7a20ebba1becb7f29317a32d3010bbdb29"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma} \\label{lemma-minimality-smallest-eigenvalue}\nSuppose $\\lbrace M_k \\rbrace$ satisfy \\ref{prop-sympd} to \\ref{prop-condnum}, and recall that $C$ is the Lipschitz constant in \\ref{condition-dg-lipschitz} and $C_2$ is the scaling constant in \\ref{condition-noise-grad}. There exists a $K \\in \\mathbb{N}$ such that, for all $k \\geq K$ and for any eigenvalue, $\\lambda$ of $M_k$,\n\\begin{equation} \\label{eqn-grad-norm-lb}\n\\frac{1}{2} \\lambda_{\\min}(M_k) \\leq \\lambda - \\frac{C}{2}\\lambda^2 - \\frac{C C_2}{2} \\lambda_{\\max}(M_k)^2.\n\\end{equation}\n\\end{lemma}\n\n\\begin{proof}\nWe begin by showing that right hand side of \\eqref{eqn-grad-norm-lb} is minimized when $\\lambda = \\lambda_{\\min}(M_k)$ for all $k$ sufficiently large and then establish \\eqref{eqn-grad-norm-lb} for this choice of $\\lambda$. \nWe first establish\n\\begin{equation} \\label{eqn-def-K}\n\\lambda_{\\max}(M_k) + \\lambda_{\\min}(M_k) \\leq \\frac{2}{C}.\n\\end{equation}\nBy \\ref{prop-maxeig}, $\\lambda_{\\max}(M_k) \\to 0$. Therefore, there exists a $k_0 \\in \\mathbb{N}$ such that for all $k \\geq k_0$, $\\lambda_{\\max}(M_k) \\leq 1/C$. Therefore, (\\ref{eqn-def-K}) follows for all $k \\geq k_0$. Moreover, for every $k \\geq k_0$, we see that (\\ref{eqn-def-K}) holds for any other eigenvalue of $M_k$ in place of $\\lambda_{\\max}(M_k)$. Letting $\\lambda$ denote any eigenvalue of $M_k$ and noting that $\\lambda - \\lambda_{\\min}(M_k) \\geq 0$,\n\\begin{equation}\n(\\lambda - \\lambda_{\\min}(M_k) ) (\\lambda + \\lambda_{\\min}(M_k) ) \\leq \\frac{2}{C}(\\lambda - \\lambda_{\\min}(M_k) ),\n\\end{equation}\nwhich can be rearranged to conclude that, for any $k \\geq k_0$ and for any eigenvalue $\\lambda$ of $M_k$,\n\\begin{equation}\n\\begin{aligned}\n\\lambda_{\\min}(M_k) - \\frac{C}{2}\\lambda_{\\min}(M_k)^2 \\leq \\lambda - \\frac{C}{2}\\lambda^2.\n\\end{aligned}\n\\end{equation} \nIt follows that the right hand side of \\eqref{eqn-grad-norm-lb} is minimized by $\\lambda_{\\min}(M_k)$.\n\nWe now establish \\eqref{eqn-grad-norm-lb} with $\\lambda = \\lambda_{\\min}(M_k)$ by contradiction. Suppose that there exists a subsequence $\\lbrace k_j \\rbrace$ for which \\eqref{eqn-grad-norm-lb} does not hold; that is, by rearranging \\eqref{eqn-grad-norm-lb},\n\\begin{equation}\nC \\lambda_{\\min}(M_{k_j})^2 + CC_2 \\lambda_{\\max}(M_{k_j})^2 > \\lambda_{\\min}(M_{k_j}).\n\\end{equation}\nTaking an upper bound on the left hand side,\n\\begin{align}\n2C(C_2 + 1) \\lambda_{\\max}(M_{k_j})^2 > \\lambda_{\\min}(M_{k_j}).\n\\end{align}\nHence,\n\\begin{equation} \\label{eqn-relaxed-condnum-control}\n\\lambda_{\\max}(M_{k_j}) \\kappa( M_{k_j} ) > \\frac{1}{2 C(C_2 + 1)},\n\\end{equation}\nwhich contradicts \\ref{prop-condnum} for $j$ sufficiently large. Hence, there is a $K \\geq k_0$ such that, for all $k \\geq K$, \\eqref{eqn-grad-norm-lb} holds.\n\\qed\n\\end{proof}\n\n\\begin{lemma} \\label{lemma-pk-satisfy-mk-properties}\nWith probability one, $\\lbrace P_k \\rbrace$ satisfy \\ref{prop-sympd} to \\ref{prop-condnum}. Moreover, with probability one, for all $k \\geq 0$ and any eigenvalue $\\lambda$ of $P_k$,\n\\begin{equation}\n\\frac{1}{2}\\lambda_{\\min}(P_k) \\leq \\lambda - \\frac{C}{2} \\lambda^2 - \\frac{C C_2}{2} \\lambda_{\\max}(P_k)^2.\n\\end{equation}\n\\end{lemma}\n\n\\begin{proof}\nThe result follows from a standard divide and conquer argument. For \\ref{prop-sympd},\n\\begin{align}\n&\\Prb{ P_k = P_k', P_k \\succ 0 } = \\Prb{ M_{\\tau+1+k}' = M_{\\tau+1+k}, M_{\\tau+1+k} \\succ 0} \\\\\n&= \\sum_{j=0}^\\infty \\condPrb{ M_{j + 1 +k}' = M_{j + 1 +k}, M_{j+1+k} \\succ 0}{\\tau = j} \\Prb{ \\tau = j} \\\\\n&= \\sum_{j=0}^\\infty \\underbrace{\\Prb{ M_{j + 1 +k}' = M_{j + 1 +k}, M_{j+1+k} \\succ 0 }}_{=1~\\text{by \\ref{prop-sympd} on } \\lbrace M_k \\rbrace } \\Prb{ \\tau=j} \\\\\n&= \\sum_{j=0}^\\infty \\Prb{ \\tau = j} \\\\\n&= \\Prb{ \\tau < \\infty } = 1.\n\\end{align}\nSimilarly, for \\ref{prop-maxeig},\n\\begin{align}\n&\\Prb{ \\sum_{k=0}^\\infty \\lambda_{\\max}(P_k)^2 < S} \\\\\n&= \\sum_{j=0}^\\infty \\underbrace{\\condPrb{ \\sum_{k=0}^\\infty \\lambda_{\\max}(M_{j+1+k})^2 < S}{\\tau=j}}_{=1~\\text{by \\ref{prop-maxeig} on } \\lbrace M_k \\rbrace} \\Prb{ \\tau=j} \\\\\n&= \\Prb{ \\tau < \\infty} =1.\n\\end{align}\nThe analogous arguments will show that the remaining conclusions of the lemma hold.\n\\qed\n\\end{proof}\n\nFrom these properties, we see that if (\\ref{eqn-psi-iter}) did not have the indicator term, then the only difference between $\\lbrace \\psi_k \\rbrace$ and $\\lbrace \\beta_k \\rbrace$ is the initialization---the fact that $\\lbrace P_k \\rbrace$ and $\\lbrace M_k \\rbrace$ are distinct is of little importance for our purposes as long as \\ref{prop-sympd} to \\ref{prop-condnum} are satisfied. Thus, we see that $\\lbrace \\beta_k \\rbrace$ exhibit an analogue of the strong Markov property.\n\nNow, to couple these two iterate sequences, let $\\mathcal{G}_0 = \\mathcal{F}_{\\tau+1}$ and $\\mathcal{G}_k = \\sigma(\\mathcal{F}_{\\tau+1},Z_1,\\ldots,Z_k)$, and, for $\\delta >0$, define $\\tau_\\delta$ to be a stopping time with respect to $\\lbrace \\mathcal{G}_k \\rbrace$ such that\n\\begin{equation}\n\\tau_\\delta = \\min\\left\\lbrace k \\geq 0: \\norm{ \\dot{F}(\\psi_k)}_2^2 \\leq \\delta \\right\\rbrace.\n\\end{equation} \nThen, on the event $\\lbrace k \\geq \\tau_\\delta \\rbrace$, $\\psi_{k+1} = \\psi_k$. Moreover, on the event $\\lbrace k < \\tau_\\delta \\rbrace$,\n\\begin{equation}\n\\begin{aligned}\n\\psi_{k+1} &= \\psi_{k} - P_k \\dot{f}( \\psi_k, Z_{k+1}) \\\\\n&= \\beta_{\\tau+1+k} - M_{\\tau+1 + k} \\dot{f}( \\beta_{\\tau+1+k}, X_{\\tau+2+k}) = \\beta_{\\tau+2+k}\n\\end{aligned}\n\\end{equation}\nfollows by induction. Therefore, for all $0 \\leq k \\leq \\tau_\\delta$, $\\psi_k = \\beta_{\\tau+1+k}$; that is, the sequences are coupled in this interval. Owing to this coupling, we see that $\\tau_\\delta + 1$ is the number of iterates for $\\beta_k$ to be within a ``$\\delta$-region of zero gradient'' after iterate $\\tau$. We now apply Zoutendijk's global convergence approach to conclude that $\\tau_\\delta$ is finite with probability one.\n\n\nWe now apply Zoutendijk's global convergence approach \\cite{zoutendijk1970} to $\\lbrace \\psi_k \\rbrace$ to conclude that $\\Prb{ \\tau_\\delta < \\infty} = 1$. First, by the fundamental theorem of calculus and \\ref{condition-dg-lipschitz} (recall, with constant $C$),\n\\begin{align}\nF(\\psi_{k+1}) &\\leq F(\\psi_k) + \\dot{F}(\\psi_k)'(\\psi_{k+1} - \\psi_k) + \\frac{C}{2}\\norm{ \\psi_{k+1} - \\psi_k}_2^2 \\\\\n\t\t\t  & \\begin{aligned}\n\t\t\t  &=F(\\psi_k) - \\dot{F}(\\psi_k)'P_k \\dot f (\\psi_k, Z_{k+1}) \\1{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta} \\\\\n\t\t\t  &+ \\frac{C}{2} \\norm{ P_k \\dot f(\\psi_k, Z_{k+1})}_2^2 \\1{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta}. \n\t\t\t  \\end{aligned} \\label{eqn-ftc}\n\\end{align}\nWe now take the conditional expectation of the resulting inequality with respect to $\\mathcal{G}_k$. Note, since $\\psi_k, P_k$ are measurable with respect to $ \\mathcal{G}_k$ and $Z_{k+1}$ is independent of $\\mathcal{G}_k$, then $\\cond{ F(\\psi_k)}{\\mathcal G_k} = F(\\psi_k)$ and \n\\begin{equation}\n\\cond{ \\dot{F}(\\psi_k)'P_k \\dot f(\\psi_k, Z_{k+1})}{\\mathcal{G}_k} = \\dot{F}(\\psi_k)'P_k \\dot{F}(\\psi_k).\n\\end{equation}\nFor the third term in (\\ref{eqn-ftc}), we will need to make use of \\ref{condition-noise-grad} (with parameters $C_1,C_2 \\geq 0$).\n\\begin{align}\n&\\cond{ \\norm{ P_k \\left[ \\dot f(\\psi_k, Z_{k+1}) - \\dot{F}(\\psi_k) + \\dot{F}(\\psi_k) \\right]}_2^2}{\\mathcal G_k} \\nonumber \\\\\n&= \\cond{ \\norm{P_k \\left[ \\dot f(\\psi_k,Z_{k+1}) - \\dot{F}(\\psi_k) \\right]}_2^2}{\\mathcal{G}_k} + \\dot F(\\psi_k)'P_k^2 \\dot F(\\psi_k) \\\\\n&\\leq C_1\\lambda_{\\max}(P_k)^2 + C_2 \\lambda_{\\max}(P_k)^2 \\norm{ \\dot{F}(\\psi_k)}_2^2 + \\dot F(\\psi_k)'P_k^2 \\dot F(\\psi_k)\n\\end{align}\nPutting the calculation for these three terms together in (\\ref{eqn-ftc}), we conclude\n\\begin{equation} \\label{eqn-ftc-cond}\n\\begin{aligned}\n&\\cond{F(\\psi_{k+1})}{\\mathcal{G}_k} \\leq F(\\psi_k) + \\frac{CC_1}{2} \\lambda_{\\max}(P_k)^2 -  \\1{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta} \\\\\n&\\times \\left[\\dot{F}(\\psi_k)'P_k \\dot F(\\psi_k) - \\frac{C}{2} \\dot{F}(\\psi_k)'P_k^2 \\dot F(\\psi_k) - \\frac{C C_2}{2} \\lambda_{\\max}(P_k)^2 \\norm{ \\dot F(\\psi_k)}_2^2 \\right]. \n\\end{aligned}\n\\end{equation}\nWe can find an upper bound for (\\ref{eqn-ftc-cond}) by finding a lower bound for the third term in the right hand side of the inequality. In particular, we will lower bound\n\\begin{equation} \\label{eqn-ftc-cond-min}\n\\min_{v \\in \\mathbb{S}^{p-1}} v'P_k v - \\frac{C}{2} v' P_k^2 v - \\frac{C C_2}{2} \\lambda_{\\max}(P_k)^2 \\norm{v}_2^2,\n\\end{equation}\nwhere $\\mathbb S^{p-1}$ is the unit sphere in $\\mathbb {R}^p$.\\footnote{We could drop the last term in the optimization problem as it is a constant.}\n\nUsing the Schur decomposition of $P_k$,\\footnote{Since $P_k$ is random, its Schur decomposition is random.} we can transform (\\ref{eqn-ftc-cond-min}) into the equivalent problem\n\\begin{equation} \\label{eqn-ftc-cond-min-2}\n\\min_{v \\in \\mathbb{S}^{p-1}} \\sum_{i=1}^p \\left[\\lambda_i  - \\frac{C}{2} \\lambda_i^2 - \\frac{C C_2}{2} \\lambda_1^2\\right] v_i^2,\n\\end{equation}\nwhere $\\lambda_{\\max}(P_k) = \\lambda_1 \\geq \\lambda_2 \\geq \\cdots \\geq \\lambda_p = \\lambda_{\\min}(P_k)$; and $v_i$ are the components of $v$. Applying Lemma \\ref{lemma-pk-satisfy-mk-properties}, the solution to (\\ref{eqn-ftc-cond-min-2}) is lower bounded by $\\lambda_{\\min}(P_k)/2$.\n\n\nPlugging this lower bound into (\\ref{eqn-ftc-cond}), we have that\n\\begin{equation}\n\\begin{aligned}\n\\cond{F(\\psi_{k+1})}{\\mathcal{G}_k} &\\leq F(\\psi_k) + \\frac{CC_1}{2} \\lambda_{\\max}(P_k)^2 \\\\\n&\\quad - \\frac{1}{2}\\lambda_{\\min}(P_k)\\norm{ \\dot{F} (\\psi_k)}_2^2 \\1{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta}.\n\\end{aligned}\n\\end{equation}\nRearranging and applying the condition in the indicator,\n\\begin{equation}\n\\begin{aligned}\n&\\frac{\\delta}{2}\\lambda_{\\min}(P_k) \\1{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta} \\\\\n&\\quad \\leq F(\\psi_k) - \\cond{ F(\\psi_{k+1})}{ \\mathcal{G}_k} + \\frac{C C_1}{2} \\lambda_{\\max}(P_k)^2.\n\\end{aligned}\n\\end{equation}\nNow, recall that $P_k$ are measurable with respect to $\\mathcal{F}_{\\tau+1}$ and recall that $\\mathcal{F}_{\\tau+1} \\subset \\mathcal{G}_k$ for all $k$. Therefore,\n\\begin{equation}\n\\begin{aligned}\n&\\frac{\\delta}{2}\\lambda_{\\min}(P_k) \\condPrb{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta }{ \\mathcal{F}_{\\tau+1}} \\\\\n&\\quad \\leq \\cond{ F(\\psi_k) - F(\\psi_{k+1})}{\\mathcal{F}_{\\tau+1}} + \\frac{C C_1}{2} \\lambda_{\\max}(P_k)^2.\n\\end{aligned}\n\\end{equation}\nMoreover, by (a) summing both sides from $k=0$ to $n \\in \\mathbb{N}$, (b) recalling that $F(\\psi_0)$ is finite with probability one given $\\mathcal{F}_{\\tau+1}$, and (c) applying \\ref{prop-maxeig} from Lemma \\ref{lemma-pk-satisfy-mk-properties}, we conclude\n\\begin{equation}\n\\begin{aligned}\n&\\frac{\\delta}{2} \\sum_{k=0}^n \\lambda_{\\min}(P_k) \\condPrb{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta }{ \\mathcal{F}_{\\tau+1}} \\\\\n&\\quad \\leq F(\\psi_0) - \\cond{ F(\\psi_{n+1})}{\\mathcal{F}_{\\tau+1}} + \\frac{SCC_1}{2}.\n\\end{aligned}\n\\end{equation}\n\nRecall that, we have assumed that $F(\\theta)$ is bounded from below by some constant $F_{l.b.}$ as a core assumption. Using this, we see that\n\\begin{equation}\n\\begin{aligned}\n\\frac{\\delta}{2} \\sum_{k=0}^n \\lambda_{\\min}(P_k) \\condPrb{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta }{ \\mathcal{F}_{\\tau+1}} \n\\leq F(\\psi_0) - F_{l.b.} + \\frac{SCC_1}{2}.\n\\end{aligned}\n\\end{equation}\n\nMoreover, note that, for all $k \\geq 0$,\n\\begin{equation}\n\\condPrb{ \\bigcap_{j=0}^\\infty \\left\\lbrace \\norm{ \\dot{F}(\\psi_j)}_2^2 > \\delta \\right\\rbrace }{ \\mathcal{F}_{\\tau+1}} \\leq \\condPrb{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta }{ \\mathcal{F}_{\\tau+1}}. \n\\end{equation}\nTherefore, for arbitrary $n$,\n\\begin{equation}\n\\condPrb{ \\bigcap_{j=0}^\\infty \\left\\lbrace \\norm{ \\dot{F}(\\psi_j)}_2^2 > \\delta \\right\\rbrace }{ \\mathcal{F}_{\\tau+1}} \\leq \\frac{F(\\psi_0) - F_{l.b.} + S C C_1 /2 }{(\\delta/2)\\sum_{k=0}^n \\lambda_{\\min}(P_k)}\n\\end{equation} \nBy \\ref{prop-mineig} from Lemma \\ref{lemma-pk-satisfy-mk-properties}, the right hand side of this inequality can be made arbitrarily small, which implies that the conditional probability on the left hand side is zero.\n\nThat is,\n\\begin{equation}\n0 = \\condPrb{ \\bigcap_{j=0}^\\infty  \\left\\lbrace \\norm{ \\dot{F}(\\psi_j)}_2^2 > \\delta \\right\\rbrace }{ \\mathcal{F}_{\\tau+1}} = \\condPrb{ \\tau_{\\delta} = \\infty}{\\mathcal{F}_{\\tau+1}}.\n\\end{equation}\nIn other words, we have concluded that $\\condPrb{ \\tau_\\delta < \\infty}{\\mathcal{F}_{\\tau+1}} = 1$ with probability one for any finite stopping time $\\tau$. With this result, our last step is to use induction. \n\nFor now, we will say that the iterates are in a $\\delta$-region of zero gradient if the squared-norm of the gradient of the iterate is no greater than $\\delta$. If we let $\\tau = -1$, then $\\tau_\\delta$ is the first time the iterates enter a $\\delta$-region of zero gradient. Let $T_1(\\delta) = \\tau_\\delta$ when $\\tau = -1$. Then, from the above argument, we have shown that $T_1(\\delta)$ is a finite stopping time. Now, define $T_j(\\delta)$ to be the $j^{th}$ time that the iterates enter a $\\delta$-region of zero gradient. Suppose that $T_j(\\delta)$ is finite. Then, define $\\tau = T_j(\\delta)$. Then, $\\tau_\\delta$ for this $\\tau$ is the next time that the iterates enter a $\\delta$-region of zero gradient. That is, $T_{j+1}(\\delta) = \\tau_\\delta + T_j(\\delta)$. Since we have assumed that $\\tau = T_{j}(\\delta)$ is finite, we conclude that $\\tau_\\delta$ is finite, which implies that $T_{j+1}(\\delta)$ is finite. Therefore, by induction we have proven the following result.\n\n\n\\begin{theorem} \\label{theorem-revisit}\nLet $F$ be a Bottou-Curtis-Nocedal function (\\ref{bcn}) and let $\\lbrace \\beta_k \\rbrace$ be the iterates generated by Stochastic Gradient Descent satisfying \\ref{prop-sympd} to \\ref{prop-condnum} (\\S \\ref{subsection-sgd}), then, for any $\\delta > 0$,\n\\begin{equation}\n\\condPrb{ \\norm{ \\dot F( \\beta_k)}_2^2 \\leq \\delta, ~i.o.}{\\mathcal{F}_0} = 1 ~\\text{with probability 1},\n\\end{equation}\nwhere $\\mathcal{F}_0 = \\sigma(\\beta_0)$.\n\\end{theorem}\n\nAgain, we will temporarily drop the conditioning on $\\mathcal{F}_0$ for simplicity of the notation. By Theorem \\ref{theorem-revisit} and the inclusion-exclusion principle, \n\\begin{align}\n1 &= \\Prb{\\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace \\cup \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace } \\\\\n  &\\begin{aligned}\n  &= \\Prb{ \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o.} + \\Prb{\\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o.} \\\\\n  &- \\Prb{ \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace \\cap \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace }.\n  \\end{aligned}\n\\end{align}\nApplying Theorem \\ref{theorem-revisit} again, we conclude that\n\\begin{equation} \\label{eqn-inc-exc}\n\\begin{aligned}\n&\\Prb{\\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o.} \\\\\n&\\quad = \\Prb{ \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace \\cap \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace }.\n\\end{aligned}\n\\end{equation}\n\nWe will now show that the probability of the right hand side is zero. Note, for any outcome\n\\begin{equation}\n\\omega \\in \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace \\cap \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace,\n\\end{equation}\nthere must be an infinite subsequence of $\\mathbb{N}$ such that $\\beta_k$ is in a $\\delta^2$-region of zero gradient and then $\\beta_{k+1}$ exits this $\\delta^2$-region of zero gradient. Suppose this were not true. Then, there are two cases. In the first case, $\\beta_k$ enters a $\\delta^2$-region of zero gradient and then never leaves, in which case\n\\begin{equation}\n\\omega \\not\\in \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace.\n\\end{equation}\nIn the second case, we have that $\\beta_k$ exits a $\\delta^2$-region of zero gradient, and never enters again, which implies \n\\begin{equation}\n\\omega \\not\\in \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace.\n\\end{equation}\nIn both cases, we have a contradiction. Therefore, using just one of the cases, we conclude that\n\\begin{equation} \\label{eqn-set-inc-1}\n\\begin{aligned}\n&\\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace \\cap \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace \\\\\n&\\quad \\subset \\left\\lbrace \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta,~ \\norm{\\dot{F}(\\beta_{k+1})}_2 > \\delta, ~i.o. \\right\\rbrace.\n\\end{aligned}\n\\end{equation}\n\nWe can write this latter event as\n\\begin{equation} \\label{eqn-set-inc-2}\n\\begin{aligned}\n&\\left\\lbrace \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta,~ \\norm{\\dot{F}(\\beta_{k+1})}_2 > \\delta, ~i.o. \\right\\rbrace \\\\\n&\\quad = \n\\left\\lbrace \\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} > \\delta, ~i.o. \\right\\rbrace.\n\\end{aligned}\n\\end{equation}\n\nWe will now show that this ultimate event occurs with probability zero using Markov's inequality and the Borel-Cantelli lemma. Let $\\epsilon > 0$ and recall that $C > 0$ is the parameter in \\ref{condition-dg-lipschitz}. Then,\n\\begin{align}\n&\\condPrb{ \\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq \\delta + C \\epsilon }{\\mathcal{F}_k} \\\\\n&\\begin{aligned}\n&\\leq \\mathbb{P}\\bigg{[} \\left( \\norm{ \\dot{F}(\\beta_{k+1}) - \\dot{F}(\\beta_k) }_2 + \\norm{ \\dot{F}(\\beta_k)}_2 \\right) \\1{ \\norm{\\dot{F}(\\beta_k)}_2\\leq \\delta}  \\\\\n&\\quad  \\geq \\delta + C \\epsilon \\bigg{\\vert} \\mathcal{F}_k \\bigg{]}\n\\end{aligned} \\\\\n&\\leq \\condPrb{ \\norm{ \\dot{F}(\\beta_{k+1}) - \\dot{F}(\\beta_k)}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} + \\delta \\geq \\delta + C \\epsilon}{\\mathcal{F}_k}\\\\\n& \\leq \\condPrb{ C \\norm{\\beta_{k+1} - \\beta_k}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq  C \\epsilon}{\\mathcal{F}_k} \\\\\n& \\leq \\condPrb{ \\norm{M_k \\dot{f}(\\beta_k, X_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq   \\epsilon}{\\mathcal{F}_k}\n\\end{align}\n\nApplying Markov's inequality to the last conditional probability and using \\ref{condition-noise-grad},\n\\begin{align}\n&\\condPrb{ \\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq \\delta + C \\epsilon }{\\mathcal{F}_k} \\\\\n& \\leq \\frac{\\lambda_{\\max}(M_k)^2}{\\epsilon^2}\\left[ C_1 + (C_2+1) \\norm{ \\dot{F}(\\beta_k)}_2^2 \\right] \\1{ \\norm{ \\dot{F}(\\beta_k)}_2 \\leq \\delta}  \\\\\n& \\leq \\frac{\\lambda_{\\max}(M_k)^2}{\\epsilon^2}\\left[ C_1 + (C_2+1)\\delta^2 \\right]\n\\end{align}\nBy \\ref{prop-maxeig}, the sum of the right hand side is bounded with probability one. Therefore, by the conditional second Borel-Cantelli lemma (Theorem 5.3.2 of \\cite{durrett2010}),\n\\begin{equation}\n\\Prb{\\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq \\delta + C \\epsilon, ~i.o.} = 0. \n\\end{equation}\nSince $\\epsilon > 0$ is arbitrary, then this conclusion holds for each element in the sequence $\\lbrace \\epsilon_m \\rbrace$ where $\\epsilon_m \\downarrow 0$. Therefore,\n\\begin{align}\n&\\Prb{\\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} > \\delta, ~i.o.} \\\\\n&=\\Prb{\\bigcup_{m \\in \\mathbb{N}} \\left\\lbrace \\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq \\delta + C \\epsilon_m, ~i.o. \\right\\rbrace} \\\\\n&\\leq \\sum_{m=1}^\\infty \\Prb{\\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq \\delta + C \\epsilon_m, ~i.o.} \\\\\n&= 0.  \n\\end{align}\n\nTherefore, by using this result with (\\ref{eqn-inc-exc}), (\\ref{eqn-set-inc-1}) and (\\ref{eqn-set-inc-2}), we conclude the following result.\n\\begin{theorem} \\label{theorem-exit}\nLet $F$ be a Bottou-Curtis-Nocedal function (\\ref{bcn}) and let $\\lbrace \\beta_k \\rbrace$ be the iterates generated by Stochastic Gradient Descent satisfying \\ref{prop-sympd} to \\ref{prop-condnum} (\\S \\ref{subsection-sgd}), then, for any $\\delta > 0$,\n\\begin{equation}\n\\condPrb{ \\norm{ \\dot F( \\beta_k)}_2 > \\delta, ~i.o.}{\\mathcal{F}_0} = 0 ~\\text{with probability 1},\n\\end{equation}\nwhere $\\mathcal{F}_0 = \\sigma(\\beta_0)$.\n\\end{theorem}\n\n\\begin{proof}\nFor any $\\delta > 0$, by Theorem \\ref{theorem-exit},\n\\begin{align}\n1 &= \\condPrb{ \\left\\lbrace \\norm{ \\dot F( \\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace^c}{\\mathcal{F}_0} \\\\\n  &= \\condPrb{ \\limsup_{k \\to \\infty} \\norm{ \\dot{F}(\\beta_k)}_2 \\leq \\delta}{\\mathcal{F}_0}.\n\\end{align}\nSince $\\delta > 0$ is arbitrary, the preceding result applies to each element in the sequence $\\lbrace \\delta_m \\rbrace$ where $\\delta_m \\downarrow 0$. Since the countable intersection of probability one events has probability one,\n\\begin{align}\n1 &= \\condPrb{ \\bigcap_{m \\in \\mathbb{N}} \\left\\lbrace \\limsup_{k \\to \\infty} \\norm{ \\dot{F}(\\beta_k)}_2 \\leq \\delta_m \\right\\rbrace }{\\mathcal{F}_0} \\\\\n  &= \\condPrb{ \\limsup_{k \\to \\infty} \\norm{ \\dot{F}(\\beta_k)}_2 = 0}{\\mathcal{F}_0},\n\\end{align}\nwhich is the desired result. \\qed\n\\end{proof}","units":[{"start":52698,"end":53263,"start_line":501,"end_line":506,"raw_tex":"\\begin{lemma} \\label{lemma-minimality-smallest-eigenvalue}\nSuppose $\\lbrace M_k \\rbrace$ satisfy \\ref{prop-sympd} to \\ref{prop-condnum}, and recall that $C$ is the Lipschitz constant in \\ref{condition-dg-lipschitz} and $C_2$ is the scaling constant in \\ref{condition-noise-grad}. There exists a $K \\in \\mathbb{N}$ such that, for all $k \\geq K$ and for any eigenvalue, $\\lambda$ of $M_k$,\n\\begin{equation} \\label{eqn-grad-norm-lb}\n\\frac{1}{2} \\lambda_{\\min}(M_k) \\leq \\lambda - \\frac{C}{2}\\lambda^2 - \\frac{C C_2}{2} \\lambda_{\\max}(M_k)^2.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"84c4dc79ae9d562d3b051292246ae4e78674f2e2234ea0014f17629bfe86d641"},{"start":53265,"end":55444,"start_line":508,"end_line":540,"raw_tex":"\\begin{proof}\nWe begin by showing that right hand side of \\eqref{eqn-grad-norm-lb} is minimized when $\\lambda = \\lambda_{\\min}(M_k)$ for all $k$ sufficiently large and then establish \\eqref{eqn-grad-norm-lb} for this choice of $\\lambda$. \nWe first establish\n\\begin{equation} \\label{eqn-def-K}\n\\lambda_{\\max}(M_k) + \\lambda_{\\min}(M_k) \\leq \\frac{2}{C}.\n\\end{equation}\nBy \\ref{prop-maxeig}, $\\lambda_{\\max}(M_k) \\to 0$. Therefore, there exists a $k_0 \\in \\mathbb{N}$ such that for all $k \\geq k_0$, $\\lambda_{\\max}(M_k) \\leq 1/C$. Therefore, (\\ref{eqn-def-K}) follows for all $k \\geq k_0$. Moreover, for every $k \\geq k_0$, we see that (\\ref{eqn-def-K}) holds for any other eigenvalue of $M_k$ in place of $\\lambda_{\\max}(M_k)$. Letting $\\lambda$ denote any eigenvalue of $M_k$ and noting that $\\lambda - \\lambda_{\\min}(M_k) \\geq 0$,\n\\begin{equation}\n(\\lambda - \\lambda_{\\min}(M_k) ) (\\lambda + \\lambda_{\\min}(M_k) ) \\leq \\frac{2}{C}(\\lambda - \\lambda_{\\min}(M_k) ),\n\\end{equation}\nwhich can be rearranged to conclude that, for any $k \\geq k_0$ and for any eigenvalue $\\lambda$ of $M_k$,\n\\begin{equation}\n\\begin{aligned}\n\\lambda_{\\min}(M_k) - \\frac{C}{2}\\lambda_{\\min}(M_k)^2 \\leq \\lambda - \\frac{C}{2}\\lambda^2.\n\\end{aligned}\n\\end{equation} \nIt follows that the right hand side of \\eqref{eqn-grad-norm-lb} is minimized by $\\lambda_{\\min}(M_k)$.\n\nWe now establish \\eqref{eqn-grad-norm-lb} with $\\lambda = \\lambda_{\\min}(M_k)$ by contradiction. Suppose that there exists a subsequence $\\lbrace k_j \\rbrace$ for which \\eqref{eqn-grad-norm-lb} does not hold; that is, by rearranging \\eqref{eqn-grad-norm-lb},\n\\begin{equation}\nC \\lambda_{\\min}(M_{k_j})^2 + CC_2 \\lambda_{\\max}(M_{k_j})^2 > \\lambda_{\\min}(M_{k_j}).\n\\end{equation}\nTaking an upper bound on the left hand side,\n\\begin{align}\n2C(C_2 + 1) \\lambda_{\\max}(M_{k_j})^2 > \\lambda_{\\min}(M_{k_j}).\n\\end{align}\nHence,\n\\begin{equation} \\label{eqn-relaxed-condnum-control}\n\\lambda_{\\max}(M_{k_j}) \\kappa( M_{k_j} ) > \\frac{1}{2 C(C_2 + 1)},\n\\end{equation}\nwhich contradicts \\ref{prop-condnum} for $j$ sufficiently large. Hence, there is a $K \\geq k_0$ such that, for all $k \\geq K$, \\eqref{eqn-grad-norm-lb} holds.\n\\qed\n\\end{proof}","sha256_utf8":"0cecee5390e6240a281b6b311e252815eb980eaa7994f6746b863bc18711658b"},{"start":56541,"end":56928,"start_line":554,"end_line":559,"raw_tex":"\\begin{lemma} \\label{lemma-pk-satisfy-mk-properties}\nWith probability one, $\\lbrace P_k \\rbrace$ satisfy \\ref{prop-sympd} to \\ref{prop-condnum}. Moreover, with probability one, for all $k \\geq 0$ and any eigenvalue $\\lambda$ of $P_k$,\n\\begin{equation}\n\\frac{1}{2}\\lambda_{\\min}(P_k) \\leq \\lambda - \\frac{C}{2} \\lambda^2 - \\frac{C C_2}{2} \\lambda_{\\max}(P_k)^2.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"cf8b0af9e5820f8c92191bcff6fa2fe4ef0dcb39006900cb926e2f1df2a51ff5"},{"start":56929,"end":57914,"start_line":560,"end_line":577,"raw_tex":"\\begin{proof}\nThe result follows from a standard divide and conquer argument. For \\ref{prop-sympd},\n\\begin{align}\n&\\Prb{ P_k = P_k', P_k \\succ 0 } = \\Prb{ M_{\\tau+1+k}' = M_{\\tau+1+k}, M_{\\tau+1+k} \\succ 0} \\\\\n&= \\sum_{j=0}^\\infty \\condPrb{ M_{j + 1 +k}' = M_{j + 1 +k}, M_{j+1+k} \\succ 0}{\\tau = j} \\Prb{ \\tau = j} \\\\\n&= \\sum_{j=0}^\\infty \\underbrace{\\Prb{ M_{j + 1 +k}' = M_{j + 1 +k}, M_{j+1+k} \\succ 0 }}_{=1~\\text{by \\ref{prop-sympd} on } \\lbrace M_k \\rbrace } \\Prb{ \\tau=j} \\\\\n&= \\sum_{j=0}^\\infty \\Prb{ \\tau = j} \\\\\n&= \\Prb{ \\tau < \\infty } = 1.\n\\end{align}\nSimilarly, for \\ref{prop-maxeig},\n\\begin{align}\n&\\Prb{ \\sum_{k=0}^\\infty \\lambda_{\\max}(P_k)^2 < S} \\\\\n&= \\sum_{j=0}^\\infty \\underbrace{\\condPrb{ \\sum_{k=0}^\\infty \\lambda_{\\max}(M_{j+1+k})^2 < S}{\\tau=j}}_{=1~\\text{by \\ref{prop-maxeig} on } \\lbrace M_k \\rbrace} \\Prb{ \\tau=j} \\\\\n&= \\Prb{ \\tau < \\infty} =1.\n\\end{align}\nThe analogous arguments will show that the remaining conclusions of the lemma hold.\n\\qed\n\\end{proof}","sha256_utf8":"1b1334f523a02c037f3d05bec9be45198185f8afbc04644b2de5fb77e5aa8b2d"},{"start":57916,"end":59595,"start_line":579,"end_line":592,"raw_tex":"From these properties, we see that if (\\ref{eqn-psi-iter}) did not have the indicator term, then the only difference between $\\lbrace \\psi_k \\rbrace$ and $\\lbrace \\beta_k \\rbrace$ is the initialization---the fact that $\\lbrace P_k \\rbrace$ and $\\lbrace M_k \\rbrace$ are distinct is of little importance for our purposes as long as \\ref{prop-sympd} to \\ref{prop-condnum} are satisfied. Thus, we see that $\\lbrace \\beta_k \\rbrace$ exhibit an analogue of the strong Markov property.\n\nNow, to couple these two iterate sequences, let $\\mathcal{G}_0 = \\mathcal{F}_{\\tau+1}$ and $\\mathcal{G}_k = \\sigma(\\mathcal{F}_{\\tau+1},Z_1,\\ldots,Z_k)$, and, for $\\delta >0$, define $\\tau_\\delta$ to be a stopping time with respect to $\\lbrace \\mathcal{G}_k \\rbrace$ such that\n\\begin{equation}\n\\tau_\\delta = \\min\\left\\lbrace k \\geq 0: \\norm{ \\dot{F}(\\psi_k)}_2^2 \\leq \\delta \\right\\rbrace.\n\\end{equation} \nThen, on the event $\\lbrace k \\geq \\tau_\\delta \\rbrace$, $\\psi_{k+1} = \\psi_k$. Moreover, on the event $\\lbrace k < \\tau_\\delta \\rbrace$,\n\\begin{equation}\n\\begin{aligned}\n\\psi_{k+1} &= \\psi_{k} - P_k \\dot{f}( \\psi_k, Z_{k+1}) \\\\\n&= \\beta_{\\tau+1+k} - M_{\\tau+1 + k} \\dot{f}( \\beta_{\\tau+1+k}, X_{\\tau+2+k}) = \\beta_{\\tau+2+k}\n\\end{aligned}\n\\end{equation}\nfollows by induction. Therefore, for all $0 \\leq k \\leq \\tau_\\delta$, $\\psi_k = \\beta_{\\tau+1+k}$; that is, the sequences are coupled in this interval. Owing to this coupling, we see that $\\tau_\\delta + 1$ is the number of iterates for $\\beta_k$ to be within a ``$\\delta$-region of zero gradient'' after iterate $\\tau$. We now apply Zoutendijk's global convergence approach to conclude that $\\tau_\\delta$ is finite with probability one.\n","sha256_utf8":"adfae6fc9a0d50d902306c982f6d5459dfef8889dd1ae812b7402f61ffcee752"},{"start":59655,"end":67106,"start_line":596,"end_line":687,"raw_tex":"We now apply Zoutendijk's global convergence approach \\cite{zoutendijk1970} to $\\lbrace \\psi_k \\rbrace$ to conclude that $\\Prb{ \\tau_\\delta < \\infty} = 1$. First, by the fundamental theorem of calculus and \\ref{condition-dg-lipschitz} (recall, with constant $C$),\n\\begin{align}\nF(\\psi_{k+1}) &\\leq F(\\psi_k) + \\dot{F}(\\psi_k)'(\\psi_{k+1} - \\psi_k) + \\frac{C}{2}\\norm{ \\psi_{k+1} - \\psi_k}_2^2 \\\\\n\t\t\t  & \\begin{aligned}\n\t\t\t  &=F(\\psi_k) - \\dot{F}(\\psi_k)'P_k \\dot f (\\psi_k, Z_{k+1}) \\1{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta} \\\\\n\t\t\t  &+ \\frac{C}{2} \\norm{ P_k \\dot f(\\psi_k, Z_{k+1})}_2^2 \\1{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta}. \n\t\t\t  \\end{aligned} \\label{eqn-ftc}\n\\end{align}\nWe now take the conditional expectation of the resulting inequality with respect to $\\mathcal{G}_k$. Note, since $\\psi_k, P_k$ are measurable with respect to $ \\mathcal{G}_k$ and $Z_{k+1}$ is independent of $\\mathcal{G}_k$, then $\\cond{ F(\\psi_k)}{\\mathcal G_k} = F(\\psi_k)$ and \n\\begin{equation}\n\\cond{ \\dot{F}(\\psi_k)'P_k \\dot f(\\psi_k, Z_{k+1})}{\\mathcal{G}_k} = \\dot{F}(\\psi_k)'P_k \\dot{F}(\\psi_k).\n\\end{equation}\nFor the third term in (\\ref{eqn-ftc}), we will need to make use of \\ref{condition-noise-grad} (with parameters $C_1,C_2 \\geq 0$).\n\\begin{align}\n&\\cond{ \\norm{ P_k \\left[ \\dot f(\\psi_k, Z_{k+1}) - \\dot{F}(\\psi_k) + \\dot{F}(\\psi_k) \\right]}_2^2}{\\mathcal G_k} \\nonumber \\\\\n&= \\cond{ \\norm{P_k \\left[ \\dot f(\\psi_k,Z_{k+1}) - \\dot{F}(\\psi_k) \\right]}_2^2}{\\mathcal{G}_k} + \\dot F(\\psi_k)'P_k^2 \\dot F(\\psi_k) \\\\\n&\\leq C_1\\lambda_{\\max}(P_k)^2 + C_2 \\lambda_{\\max}(P_k)^2 \\norm{ \\dot{F}(\\psi_k)}_2^2 + \\dot F(\\psi_k)'P_k^2 \\dot F(\\psi_k)\n\\end{align}\nPutting the calculation for these three terms together in (\\ref{eqn-ftc}), we conclude\n\\begin{equation} \\label{eqn-ftc-cond}\n\\begin{aligned}\n&\\cond{F(\\psi_{k+1})}{\\mathcal{G}_k} \\leq F(\\psi_k) + \\frac{CC_1}{2} \\lambda_{\\max}(P_k)^2 -  \\1{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta} \\\\\n&\\times \\left[\\dot{F}(\\psi_k)'P_k \\dot F(\\psi_k) - \\frac{C}{2} \\dot{F}(\\psi_k)'P_k^2 \\dot F(\\psi_k) - \\frac{C C_2}{2} \\lambda_{\\max}(P_k)^2 \\norm{ \\dot F(\\psi_k)}_2^2 \\right]. \n\\end{aligned}\n\\end{equation}\nWe can find an upper bound for (\\ref{eqn-ftc-cond}) by finding a lower bound for the third term in the right hand side of the inequality. In particular, we will lower bound\n\\begin{equation} \\label{eqn-ftc-cond-min}\n\\min_{v \\in \\mathbb{S}^{p-1}} v'P_k v - \\frac{C}{2} v' P_k^2 v - \\frac{C C_2}{2} \\lambda_{\\max}(P_k)^2 \\norm{v}_2^2,\n\\end{equation}\nwhere $\\mathbb S^{p-1}$ is the unit sphere in $\\mathbb {R}^p$.\\footnote{We could drop the last term in the optimization problem as it is a constant.}\n\nUsing the Schur decomposition of $P_k$,\\footnote{Since $P_k$ is random, its Schur decomposition is random.} we can transform (\\ref{eqn-ftc-cond-min}) into the equivalent problem\n\\begin{equation} \\label{eqn-ftc-cond-min-2}\n\\min_{v \\in \\mathbb{S}^{p-1}} \\sum_{i=1}^p \\left[\\lambda_i  - \\frac{C}{2} \\lambda_i^2 - \\frac{C C_2}{2} \\lambda_1^2\\right] v_i^2,\n\\end{equation}\nwhere $\\lambda_{\\max}(P_k) = \\lambda_1 \\geq \\lambda_2 \\geq \\cdots \\geq \\lambda_p = \\lambda_{\\min}(P_k)$; and $v_i$ are the components of $v$. Applying Lemma \\ref{lemma-pk-satisfy-mk-properties}, the solution to (\\ref{eqn-ftc-cond-min-2}) is lower bounded by $\\lambda_{\\min}(P_k)/2$.\n\n\nPlugging this lower bound into (\\ref{eqn-ftc-cond}), we have that\n\\begin{equation}\n\\begin{aligned}\n\\cond{F(\\psi_{k+1})}{\\mathcal{G}_k} &\\leq F(\\psi_k) + \\frac{CC_1}{2} \\lambda_{\\max}(P_k)^2 \\\\\n&\\quad - \\frac{1}{2}\\lambda_{\\min}(P_k)\\norm{ \\dot{F} (\\psi_k)}_2^2 \\1{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta}.\n\\end{aligned}\n\\end{equation}\nRearranging and applying the condition in the indicator,\n\\begin{equation}\n\\begin{aligned}\n&\\frac{\\delta}{2}\\lambda_{\\min}(P_k) \\1{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta} \\\\\n&\\quad \\leq F(\\psi_k) - \\cond{ F(\\psi_{k+1})}{ \\mathcal{G}_k} + \\frac{C C_1}{2} \\lambda_{\\max}(P_k)^2.\n\\end{aligned}\n\\end{equation}\nNow, recall that $P_k$ are measurable with respect to $\\mathcal{F}_{\\tau+1}$ and recall that $\\mathcal{F}_{\\tau+1} \\subset \\mathcal{G}_k$ for all $k$. Therefore,\n\\begin{equation}\n\\begin{aligned}\n&\\frac{\\delta}{2}\\lambda_{\\min}(P_k) \\condPrb{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta }{ \\mathcal{F}_{\\tau+1}} \\\\\n&\\quad \\leq \\cond{ F(\\psi_k) - F(\\psi_{k+1})}{\\mathcal{F}_{\\tau+1}} + \\frac{C C_1}{2} \\lambda_{\\max}(P_k)^2.\n\\end{aligned}\n\\end{equation}\nMoreover, by (a) summing both sides from $k=0$ to $n \\in \\mathbb{N}$, (b) recalling that $F(\\psi_0)$ is finite with probability one given $\\mathcal{F}_{\\tau+1}$, and (c) applying \\ref{prop-maxeig} from Lemma \\ref{lemma-pk-satisfy-mk-properties}, we conclude\n\\begin{equation}\n\\begin{aligned}\n&\\frac{\\delta}{2} \\sum_{k=0}^n \\lambda_{\\min}(P_k) \\condPrb{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta }{ \\mathcal{F}_{\\tau+1}} \\\\\n&\\quad \\leq F(\\psi_0) - \\cond{ F(\\psi_{n+1})}{\\mathcal{F}_{\\tau+1}} + \\frac{SCC_1}{2}.\n\\end{aligned}\n\\end{equation}\n\nRecall that, we have assumed that $F(\\theta)$ is bounded from below by some constant $F_{l.b.}$ as a core assumption. Using this, we see that\n\\begin{equation}\n\\begin{aligned}\n\\frac{\\delta}{2} \\sum_{k=0}^n \\lambda_{\\min}(P_k) \\condPrb{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta }{ \\mathcal{F}_{\\tau+1}} \n\\leq F(\\psi_0) - F_{l.b.} + \\frac{SCC_1}{2}.\n\\end{aligned}\n\\end{equation}\n\nMoreover, note that, for all $k \\geq 0$,\n\\begin{equation}\n\\condPrb{ \\bigcap_{j=0}^\\infty \\left\\lbrace \\norm{ \\dot{F}(\\psi_j)}_2^2 > \\delta \\right\\rbrace }{ \\mathcal{F}_{\\tau+1}} \\leq \\condPrb{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta }{ \\mathcal{F}_{\\tau+1}}. \n\\end{equation}\nTherefore, for arbitrary $n$,\n\\begin{equation}\n\\condPrb{ \\bigcap_{j=0}^\\infty \\left\\lbrace \\norm{ \\dot{F}(\\psi_j)}_2^2 > \\delta \\right\\rbrace }{ \\mathcal{F}_{\\tau+1}} \\leq \\frac{F(\\psi_0) - F_{l.b.} + S C C_1 /2 }{(\\delta/2)\\sum_{k=0}^n \\lambda_{\\min}(P_k)}\n\\end{equation} \nBy \\ref{prop-mineig} from Lemma \\ref{lemma-pk-satisfy-mk-properties}, the right hand side of this inequality can be made arbitrarily small, which implies that the conditional probability on the left hand side is zero.\n\nThat is,\n\\begin{equation}\n0 = \\condPrb{ \\bigcap_{j=0}^\\infty  \\left\\lbrace \\norm{ \\dot{F}(\\psi_j)}_2^2 > \\delta \\right\\rbrace }{ \\mathcal{F}_{\\tau+1}} = \\condPrb{ \\tau_{\\delta} = \\infty}{\\mathcal{F}_{\\tau+1}}.\n\\end{equation}\nIn other words, we have concluded that $\\condPrb{ \\tau_\\delta < \\infty}{\\mathcal{F}_{\\tau+1}} = 1$ with probability one for any finite stopping time $\\tau$. With this result, our last step is to use induction. \n\nFor now, we will say that the iterates are in a $\\delta$-region of zero gradient if the squared-norm of the gradient of the iterate is no greater than $\\delta$. If we let $\\tau = -1$, then $\\tau_\\delta$ is the first time the iterates enter a $\\delta$-region of zero gradient. Let $T_1(\\delta) = \\tau_\\delta$ when $\\tau = -1$. Then, from the above argument, we have shown that $T_1(\\delta)$ is a finite stopping time. Now, define $T_j(\\delta)$ to be the $j^{th}$ time that the iterates enter a $\\delta$-region of zero gradient. Suppose that $T_j(\\delta)$ is finite. Then, define $\\tau = T_j(\\delta)$. Then, $\\tau_\\delta$ for this $\\tau$ is the next time that the iterates enter a $\\delta$-region of zero gradient. That is, $T_{j+1}(\\delta) = \\tau_\\delta + T_j(\\delta)$. Since we have assumed that $\\tau = T_{j}(\\delta)$ is finite, we conclude that $\\tau_\\delta$ is finite, which implies that $T_{j+1}(\\delta)$ is finite. Therefore, by induction we have proven the following result.\n","sha256_utf8":"c2a32b3708a3723f504e6728f6e28345cc2de2fc8a7f8e6eb11f24d06571513f"},{"start":67107,"end":67591,"start_line":689,"end_line":695,"raw_tex":"\\begin{theorem} \\label{theorem-revisit}\nLet $F$ be a Bottou-Curtis-Nocedal function (\\ref{bcn}) and let $\\lbrace \\beta_k \\rbrace$ be the iterates generated by Stochastic Gradient Descent satisfying \\ref{prop-sympd} to \\ref{prop-condnum} (\\S \\ref{subsection-sgd}), then, for any $\\delta > 0$,\n\\begin{equation}\n\\condPrb{ \\norm{ \\dot F( \\beta_k)}_2^2 \\leq \\delta, ~i.o.}{\\mathcal{F}_0} = 1 ~\\text{with probability 1},\n\\end{equation}\nwhere $\\mathcal{F}_0 = \\sigma(\\beta_0)$.\n\\end{theorem}","sha256_utf8":"8d82ebe9d8b43e797639d509d83b4ec4ed623f10c9e5ce474f04a126556d116d"},{"start":67819,"end":73804,"start_line":704,"end_line":786,"raw_tex":"Again, we will temporarily drop the conditioning on $\\mathcal{F}_0$ for simplicity of the notation. By Theorem \\ref{theorem-revisit} and the inclusion-exclusion principle, \n\\begin{align}\n1 &= \\Prb{\\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace \\cup \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace } \\\\\n  &\\begin{aligned}\n  &= \\Prb{ \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o.} + \\Prb{\\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o.} \\\\\n  &- \\Prb{ \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace \\cap \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace }.\n  \\end{aligned}\n\\end{align}\nApplying Theorem \\ref{theorem-revisit} again, we conclude that\n\\begin{equation} \\label{eqn-inc-exc}\n\\begin{aligned}\n&\\Prb{\\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o.} \\\\\n&\\quad = \\Prb{ \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace \\cap \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace }.\n\\end{aligned}\n\\end{equation}\n\nWe will now show that the probability of the right hand side is zero. Note, for any outcome\n\\begin{equation}\n\\omega \\in \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace \\cap \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace,\n\\end{equation}\nthere must be an infinite subsequence of $\\mathbb{N}$ such that $\\beta_k$ is in a $\\delta^2$-region of zero gradient and then $\\beta_{k+1}$ exits this $\\delta^2$-region of zero gradient. Suppose this were not true. Then, there are two cases. In the first case, $\\beta_k$ enters a $\\delta^2$-region of zero gradient and then never leaves, in which case\n\\begin{equation}\n\\omega \\not\\in \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace.\n\\end{equation}\nIn the second case, we have that $\\beta_k$ exits a $\\delta^2$-region of zero gradient, and never enters again, which implies \n\\begin{equation}\n\\omega \\not\\in \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace.\n\\end{equation}\nIn both cases, we have a contradiction. Therefore, using just one of the cases, we conclude that\n\\begin{equation} \\label{eqn-set-inc-1}\n\\begin{aligned}\n&\\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace \\cap \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace \\\\\n&\\quad \\subset \\left\\lbrace \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta,~ \\norm{\\dot{F}(\\beta_{k+1})}_2 > \\delta, ~i.o. \\right\\rbrace.\n\\end{aligned}\n\\end{equation}\n\nWe can write this latter event as\n\\begin{equation} \\label{eqn-set-inc-2}\n\\begin{aligned}\n&\\left\\lbrace \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta,~ \\norm{\\dot{F}(\\beta_{k+1})}_2 > \\delta, ~i.o. \\right\\rbrace \\\\\n&\\quad = \n\\left\\lbrace \\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} > \\delta, ~i.o. \\right\\rbrace.\n\\end{aligned}\n\\end{equation}\n\nWe will now show that this ultimate event occurs with probability zero using Markov's inequality and the Borel-Cantelli lemma. Let $\\epsilon > 0$ and recall that $C > 0$ is the parameter in \\ref{condition-dg-lipschitz}. Then,\n\\begin{align}\n&\\condPrb{ \\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq \\delta + C \\epsilon }{\\mathcal{F}_k} \\\\\n&\\begin{aligned}\n&\\leq \\mathbb{P}\\bigg{[} \\left( \\norm{ \\dot{F}(\\beta_{k+1}) - \\dot{F}(\\beta_k) }_2 + \\norm{ \\dot{F}(\\beta_k)}_2 \\right) \\1{ \\norm{\\dot{F}(\\beta_k)}_2\\leq \\delta}  \\\\\n&\\quad  \\geq \\delta + C \\epsilon \\bigg{\\vert} \\mathcal{F}_k \\bigg{]}\n\\end{aligned} \\\\\n&\\leq \\condPrb{ \\norm{ \\dot{F}(\\beta_{k+1}) - \\dot{F}(\\beta_k)}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} + \\delta \\geq \\delta + C \\epsilon}{\\mathcal{F}_k}\\\\\n& \\leq \\condPrb{ C \\norm{\\beta_{k+1} - \\beta_k}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq  C \\epsilon}{\\mathcal{F}_k} \\\\\n& \\leq \\condPrb{ \\norm{M_k \\dot{f}(\\beta_k, X_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq   \\epsilon}{\\mathcal{F}_k}\n\\end{align}\n\nApplying Markov's inequality to the last conditional probability and using \\ref{condition-noise-grad},\n\\begin{align}\n&\\condPrb{ \\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq \\delta + C \\epsilon }{\\mathcal{F}_k} \\\\\n& \\leq \\frac{\\lambda_{\\max}(M_k)^2}{\\epsilon^2}\\left[ C_1 + (C_2+1) \\norm{ \\dot{F}(\\beta_k)}_2^2 \\right] \\1{ \\norm{ \\dot{F}(\\beta_k)}_2 \\leq \\delta}  \\\\\n& \\leq \\frac{\\lambda_{\\max}(M_k)^2}{\\epsilon^2}\\left[ C_1 + (C_2+1)\\delta^2 \\right]\n\\end{align}\nBy \\ref{prop-maxeig}, the sum of the right hand side is bounded with probability one. Therefore, by the conditional second Borel-Cantelli lemma (Theorem 5.3.2 of \\cite{durrett2010}),\n\\begin{equation}\n\\Prb{\\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq \\delta + C \\epsilon, ~i.o.} = 0. \n\\end{equation}\nSince $\\epsilon > 0$ is arbitrary, then this conclusion holds for each element in the sequence $\\lbrace \\epsilon_m \\rbrace$ where $\\epsilon_m \\downarrow 0$. Therefore,\n\\begin{align}\n&\\Prb{\\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} > \\delta, ~i.o.} \\\\\n&=\\Prb{\\bigcup_{m \\in \\mathbb{N}} \\left\\lbrace \\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq \\delta + C \\epsilon_m, ~i.o. \\right\\rbrace} \\\\\n&\\leq \\sum_{m=1}^\\infty \\Prb{\\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq \\delta + C \\epsilon_m, ~i.o.} \\\\\n&= 0.  \n\\end{align}\n\nTherefore, by using this result with (\\ref{eqn-inc-exc}), (\\ref{eqn-set-inc-1}) and (\\ref{eqn-set-inc-2}), we conclude the following result.\n\\begin{theorem} \\label{theorem-exit}\nLet $F$ be a Bottou-Curtis-Nocedal function (\\ref{bcn}) and let $\\lbrace \\beta_k \\rbrace$ be the iterates generated by Stochastic Gradient Descent satisfying \\ref{prop-sympd} to \\ref{prop-condnum} (\\S \\ref{subsection-sgd}), then, for any $\\delta > 0$,\n\\begin{equation}\n\\condPrb{ \\norm{ \\dot F( \\beta_k)}_2 > \\delta, ~i.o.}{\\mathcal{F}_0} = 0 ~\\text{with probability 1},\n\\end{equation}\nwhere $\\mathcal{F}_0 = \\sigma(\\beta_0)$.\n\\end{theorem}","sha256_utf8":"f1079e515535efe6c8325be830175ad1fd236b23a97890170141463cdba3e865"},{"start":74345,"end":75180,"start_line":796,"end_line":808,"raw_tex":"\\begin{proof}\nFor any $\\delta > 0$, by Theorem \\ref{theorem-exit},\n\\begin{align}\n1 &= \\condPrb{ \\left\\lbrace \\norm{ \\dot F( \\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace^c}{\\mathcal{F}_0} \\\\\n  &= \\condPrb{ \\limsup_{k \\to \\infty} \\norm{ \\dot{F}(\\beta_k)}_2 \\leq \\delta}{\\mathcal{F}_0}.\n\\end{align}\nSince $\\delta > 0$ is arbitrary, the preceding result applies to each element in the sequence $\\lbrace \\delta_m \\rbrace$ where $\\delta_m \\downarrow 0$. Since the countable intersection of probability one events has probability one,\n\\begin{align}\n1 &= \\condPrb{ \\bigcap_{m \\in \\mathbb{N}} \\left\\lbrace \\limsup_{k \\to \\infty} \\norm{ \\dot{F}(\\beta_k)}_2 \\leq \\delta_m \\right\\rbrace }{\\mathcal{F}_0} \\\\\n  &= \\condPrb{ \\limsup_{k \\to \\infty} \\norm{ \\dot{F}(\\beta_k)}_2 = 0}{\\mathcal{F}_0},\n\\end{align}\nwhich is the desired result. \\qed\n\\end{proof}","sha256_utf8":"fa4b4085adfde270cfc485afd0a8e434705994b20f23934574b21710b7a3de98"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"corollary","kind":"corollary","title":"随机梯度的强收敛","source_units":[{"start":73868,"end":74344,"start_line":789,"end_line":795,"raw_tex":"\\begin{corollary} \\label{corollary-convergence}\nLet $F$ be a Bottou-Curtis-Nocedal function (\\ref{bcn}) and let $\\lbrace \\beta_k \\rbrace$ be the iterates generated by Stochastic Gradient Descent satisfying \\ref{prop-sympd} to \\ref{prop-condnum} (\\S \\ref{subsection-sgd}), then\n\\begin{equation}\n\\condPrb{ \\lim_{k \\to \\infty} \\norm{ \\dot{F} (\\beta_k)}_2 = 0}{\\mathcal{F}_0} = 1, ~\\text{with probability 1},\n\\end{equation}\nwhere $\\mathcal{F}_0 = \\sigma(\\beta_0)$.\n\\end{corollary}","sha256_utf8":"b9f42f4471f1ae2be0280d181ae92d7a20ebba1becb7f29317a32d3010bbdb29"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"强收敛的完整证明","source_units":[{"start":74345,"end":75180,"start_line":796,"end_line":808,"raw_tex":"\\begin{proof}\nFor any $\\delta > 0$, by Theorem \\ref{theorem-exit},\n\\begin{align}\n1 &= \\condPrb{ \\left\\lbrace \\norm{ \\dot F( \\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace^c}{\\mathcal{F}_0} \\\\\n  &= \\condPrb{ \\limsup_{k \\to \\infty} \\norm{ \\dot{F}(\\beta_k)}_2 \\leq \\delta}{\\mathcal{F}_0}.\n\\end{align}\nSince $\\delta > 0$ is arbitrary, the preceding result applies to each element in the sequence $\\lbrace \\delta_m \\rbrace$ where $\\delta_m \\downarrow 0$. Since the countable intersection of probability one events has probability one,\n\\begin{align}\n1 &= \\condPrb{ \\bigcap_{m \\in \\mathbb{N}} \\left\\lbrace \\limsup_{k \\to \\infty} \\norm{ \\dot{F}(\\beta_k)}_2 \\leq \\delta_m \\right\\rbrace }{\\mathcal{F}_0} \\\\\n  &= \\condPrb{ \\limsup_{k \\to \\infty} \\norm{ \\dot{F}(\\beta_k)}_2 = 0}{\\mathcal{F}_0},\n\\end{align}\nwhich is the desired result. \\qed\n\\end{proof}","sha256_utf8":"fa4b4085adfde270cfc485afd0a8e434705994b20f23934574b21710b7a3de98"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"有限超阈推出上极限界","source_units":[{"start":74359,"end":74638,"start_line":797,"end_line":801,"raw_tex":"For any $\\delta > 0$, by Theorem \\ref{theorem-exit},\n\\begin{align}\n1 &= \\condPrb{ \\left\\lbrace \\norm{ \\dot F( \\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace^c}{\\mathcal{F}_0} \\\\\n  &= \\condPrb{ \\limsup_{k \\to \\infty} \\norm{ \\dot{F}(\\beta_k)}_2 \\leq \\delta}{\\mathcal{F}_0}.\n\\end{align}\n","sha256_utf8":"e4a27e540c375836733b253ba54140fa541dbce1c7c7efc99259390fa6676a4b"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"可数阈值趋零","source_units":[{"start":74638,"end":75180,"start_line":802,"end_line":808,"raw_tex":"Since $\\delta > 0$ is arbitrary, the preceding result applies to each element in the sequence $\\lbrace \\delta_m \\rbrace$ where $\\delta_m \\downarrow 0$. Since the countable intersection of probability one events has probability one,\n\\begin{align}\n1 &= \\condPrb{ \\bigcap_{m \\in \\mathbb{N}} \\left\\lbrace \\limsup_{k \\to \\infty} \\norm{ \\dot{F}(\\beta_k)}_2 \\leq \\delta_m \\right\\rbrace }{\\mathcal{F}_0} \\\\\n  &= \\condPrb{ \\limsup_{k \\to \\infty} \\norm{ \\dot{F}(\\beta_k)}_2 = 0}{\\mathcal{F}_0},\n\\end{align}\nwhich is the desired result. \\qed\n\\end{proof}","sha256_utf8":"ec10c248f1d51e21d2cc90a00e09ab323fa524e5454ace95529ee0d6cc42c03b"}]},{"id":"S005","semantic_type":"theorem","kind":"theorem","title":"超阈事件仅有限次发生","source_units":[{"start":73328,"end":73804,"start_line":780,"end_line":786,"raw_tex":"\\begin{theorem} \\label{theorem-exit}\nLet $F$ be a Bottou-Curtis-Nocedal function (\\ref{bcn}) and let $\\lbrace \\beta_k \\rbrace$ be the iterates generated by Stochastic Gradient Descent satisfying \\ref{prop-sympd} to \\ref{prop-condnum} (\\S \\ref{subsection-sgd}), then, for any $\\delta > 0$,\n\\begin{equation}\n\\condPrb{ \\norm{ \\dot F( \\beta_k)}_2 > \\delta, ~i.o.}{\\mathcal{F}_0} = 0 ~\\text{with probability 1},\n\\end{equation}\nwhere $\\mathcal{F}_0 = \\sigma(\\beta_0)$.\n\\end{theorem}","sha256_utf8":"b39e03c0bc38235874c019bde636b6be1d46f5d508252cb1eae41d115b1e3099"}]},{"id":"S006","semantic_type":"proof","kind":"proof","title":"有限超阈的完整证明","source_units":[{"start":67819,"end":73328,"start_line":704,"end_line":779,"raw_tex":"Again, we will temporarily drop the conditioning on $\\mathcal{F}_0$ for simplicity of the notation. By Theorem \\ref{theorem-revisit} and the inclusion-exclusion principle, \n\\begin{align}\n1 &= \\Prb{\\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace \\cup \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace } \\\\\n  &\\begin{aligned}\n  &= \\Prb{ \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o.} + \\Prb{\\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o.} \\\\\n  &- \\Prb{ \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace \\cap \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace }.\n  \\end{aligned}\n\\end{align}\nApplying Theorem \\ref{theorem-revisit} again, we conclude that\n\\begin{equation} \\label{eqn-inc-exc}\n\\begin{aligned}\n&\\Prb{\\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o.} \\\\\n&\\quad = \\Prb{ \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace \\cap \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace }.\n\\end{aligned}\n\\end{equation}\n\nWe will now show that the probability of the right hand side is zero. Note, for any outcome\n\\begin{equation}\n\\omega \\in \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace \\cap \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace,\n\\end{equation}\nthere must be an infinite subsequence of $\\mathbb{N}$ such that $\\beta_k$ is in a $\\delta^2$-region of zero gradient and then $\\beta_{k+1}$ exits this $\\delta^2$-region of zero gradient. Suppose this were not true. Then, there are two cases. In the first case, $\\beta_k$ enters a $\\delta^2$-region of zero gradient and then never leaves, in which case\n\\begin{equation}\n\\omega \\not\\in \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace.\n\\end{equation}\nIn the second case, we have that $\\beta_k$ exits a $\\delta^2$-region of zero gradient, and never enters again, which implies \n\\begin{equation}\n\\omega \\not\\in \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace.\n\\end{equation}\nIn both cases, we have a contradiction. Therefore, using just one of the cases, we conclude that\n\\begin{equation} \\label{eqn-set-inc-1}\n\\begin{aligned}\n&\\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace \\cap \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace \\\\\n&\\quad \\subset \\left\\lbrace \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta,~ \\norm{\\dot{F}(\\beta_{k+1})}_2 > \\delta, ~i.o. \\right\\rbrace.\n\\end{aligned}\n\\end{equation}\n\nWe can write this latter event as\n\\begin{equation} \\label{eqn-set-inc-2}\n\\begin{aligned}\n&\\left\\lbrace \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta,~ \\norm{\\dot{F}(\\beta_{k+1})}_2 > \\delta, ~i.o. \\right\\rbrace \\\\\n&\\quad = \n\\left\\lbrace \\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} > \\delta, ~i.o. \\right\\rbrace.\n\\end{aligned}\n\\end{equation}\n\nWe will now show that this ultimate event occurs with probability zero using Markov's inequality and the Borel-Cantelli lemma. Let $\\epsilon > 0$ and recall that $C > 0$ is the parameter in \\ref{condition-dg-lipschitz}. Then,\n\\begin{align}\n&\\condPrb{ \\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq \\delta + C \\epsilon }{\\mathcal{F}_k} \\\\\n&\\begin{aligned}\n&\\leq \\mathbb{P}\\bigg{[} \\left( \\norm{ \\dot{F}(\\beta_{k+1}) - \\dot{F}(\\beta_k) }_2 + \\norm{ \\dot{F}(\\beta_k)}_2 \\right) \\1{ \\norm{\\dot{F}(\\beta_k)}_2\\leq \\delta}  \\\\\n&\\quad  \\geq \\delta + C \\epsilon \\bigg{\\vert} \\mathcal{F}_k \\bigg{]}\n\\end{aligned} \\\\\n&\\leq \\condPrb{ \\norm{ \\dot{F}(\\beta_{k+1}) - \\dot{F}(\\beta_k)}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} + \\delta \\geq \\delta + C \\epsilon}{\\mathcal{F}_k}\\\\\n& \\leq \\condPrb{ C \\norm{\\beta_{k+1} - \\beta_k}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq  C \\epsilon}{\\mathcal{F}_k} \\\\\n& \\leq \\condPrb{ \\norm{M_k \\dot{f}(\\beta_k, X_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq   \\epsilon}{\\mathcal{F}_k}\n\\end{align}\n\nApplying Markov's inequality to the last conditional probability and using \\ref{condition-noise-grad},\n\\begin{align}\n&\\condPrb{ \\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq \\delta + C \\epsilon }{\\mathcal{F}_k} \\\\\n& \\leq \\frac{\\lambda_{\\max}(M_k)^2}{\\epsilon^2}\\left[ C_1 + (C_2+1) \\norm{ \\dot{F}(\\beta_k)}_2^2 \\right] \\1{ \\norm{ \\dot{F}(\\beta_k)}_2 \\leq \\delta}  \\\\\n& \\leq \\frac{\\lambda_{\\max}(M_k)^2}{\\epsilon^2}\\left[ C_1 + (C_2+1)\\delta^2 \\right]\n\\end{align}\nBy \\ref{prop-maxeig}, the sum of the right hand side is bounded with probability one. Therefore, by the conditional second Borel-Cantelli lemma (Theorem 5.3.2 of \\cite{durrett2010}),\n\\begin{equation}\n\\Prb{\\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq \\delta + C \\epsilon, ~i.o.} = 0. \n\\end{equation}\nSince $\\epsilon > 0$ is arbitrary, then this conclusion holds for each element in the sequence $\\lbrace \\epsilon_m \\rbrace$ where $\\epsilon_m \\downarrow 0$. Therefore,\n\\begin{align}\n&\\Prb{\\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} > \\delta, ~i.o.} \\\\\n&=\\Prb{\\bigcup_{m \\in \\mathbb{N}} \\left\\lbrace \\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq \\delta + C \\epsilon_m, ~i.o. \\right\\rbrace} \\\\\n&\\leq \\sum_{m=1}^\\infty \\Prb{\\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq \\delta + C \\epsilon_m, ~i.o.} \\\\\n&= 0.  \n\\end{align}\n\nTherefore, by using this result with (\\ref{eqn-inc-exc}), (\\ref{eqn-set-inc-1}) and (\\ref{eqn-set-inc-2}), we conclude the following result.\n","sha256_utf8":"f1ce25c0a90b3d6b21327769e4f27f7cecc55e1fc897320774ca5949d7a2964e"}]},{"id":"S007","semantic_type":"proof_step","kind":"proof_step","title":"容斥归约为反复进出","source_units":[{"start":67819,"end":68845,"start_line":704,"end_line":718,"raw_tex":"Again, we will temporarily drop the conditioning on $\\mathcal{F}_0$ for simplicity of the notation. By Theorem \\ref{theorem-revisit} and the inclusion-exclusion principle, \n\\begin{align}\n1 &= \\Prb{\\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace \\cup \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace } \\\\\n  &\\begin{aligned}\n  &= \\Prb{ \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o.} + \\Prb{\\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o.} \\\\\n  &- \\Prb{ \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace \\cap \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace }.\n  \\end{aligned}\n\\end{align}\nApplying Theorem \\ref{theorem-revisit} again, we conclude that\n\\begin{equation} \\label{eqn-inc-exc}\n\\begin{aligned}\n&\\Prb{\\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o.} \\\\\n&\\quad = \\Prb{ \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace \\cap \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace }.\n\\end{aligned}\n\\end{equation}\n","sha256_utf8":"082d14e0c1fba1fd6adf7c10fee054cc5664b3c18328038635b770962d73fe2a"}]},{"id":"S008","semantic_type":"proof_step","kind":"proof_step","title":"反复进出蕴含无限次越界","source_units":[{"start":68846,"end":70303,"start_line":720,"end_line":738,"raw_tex":"We will now show that the probability of the right hand side is zero. Note, for any outcome\n\\begin{equation}\n\\omega \\in \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace \\cap \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace,\n\\end{equation}\nthere must be an infinite subsequence of $\\mathbb{N}$ such that $\\beta_k$ is in a $\\delta^2$-region of zero gradient and then $\\beta_{k+1}$ exits this $\\delta^2$-region of zero gradient. Suppose this were not true. Then, there are two cases. In the first case, $\\beta_k$ enters a $\\delta^2$-region of zero gradient and then never leaves, in which case\n\\begin{equation}\n\\omega \\not\\in \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace.\n\\end{equation}\nIn the second case, we have that $\\beta_k$ exits a $\\delta^2$-region of zero gradient, and never enters again, which implies \n\\begin{equation}\n\\omega \\not\\in \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace.\n\\end{equation}\nIn both cases, we have a contradiction. Therefore, using just one of the cases, we conclude that\n\\begin{equation} \\label{eqn-set-inc-1}\n\\begin{aligned}\n&\\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 \\leq \\delta, ~i.o. \\right\\rbrace \\cap \\left\\lbrace \\norm{ \\dot F(\\beta_k)}_2 > \\delta, ~i.o. \\right\\rbrace \\\\\n&\\quad \\subset \\left\\lbrace \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta,~ \\norm{\\dot{F}(\\beta_{k+1})}_2 > \\delta, ~i.o. \\right\\rbrace.\n\\end{aligned}\n\\end{equation}\n","sha256_utf8":"78e24c3d1e0925275dc57c573b24fc91905fe3b1eb56d509974f66a9d615ee20"}]},{"id":"S009","semantic_type":"proof_step","kind":"proof_step","title":"用指示变量表示越界","source_units":[{"start":70304,"end":70667,"start_line":740,"end_line":747,"raw_tex":"We can write this latter event as\n\\begin{equation} \\label{eqn-set-inc-2}\n\\begin{aligned}\n&\\left\\lbrace \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta,~ \\norm{\\dot{F}(\\beta_{k+1})}_2 > \\delta, ~i.o. \\right\\rbrace \\\\\n&\\quad = \n\\left\\lbrace \\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} > \\delta, ~i.o. \\right\\rbrace.\n\\end{aligned}\n\\end{equation}\n","sha256_utf8":"fe8d1cd3ba0ee4f8f7b26cf344c06d744f1253a0c4ef2d73114bf4eb5a297131"}]},{"id":"S010","semantic_type":"proof_step","kind":"proof_step","title":"梯度越界归约为迭代步长","source_units":[{"start":70668,"end":71740,"start_line":749,"end_line":759,"raw_tex":"We will now show that this ultimate event occurs with probability zero using Markov's inequality and the Borel-Cantelli lemma. Let $\\epsilon > 0$ and recall that $C > 0$ is the parameter in \\ref{condition-dg-lipschitz}. Then,\n\\begin{align}\n&\\condPrb{ \\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq \\delta + C \\epsilon }{\\mathcal{F}_k} \\\\\n&\\begin{aligned}\n&\\leq \\mathbb{P}\\bigg{[} \\left( \\norm{ \\dot{F}(\\beta_{k+1}) - \\dot{F}(\\beta_k) }_2 + \\norm{ \\dot{F}(\\beta_k)}_2 \\right) \\1{ \\norm{\\dot{F}(\\beta_k)}_2\\leq \\delta}  \\\\\n&\\quad  \\geq \\delta + C \\epsilon \\bigg{\\vert} \\mathcal{F}_k \\bigg{]}\n\\end{aligned} \\\\\n&\\leq \\condPrb{ \\norm{ \\dot{F}(\\beta_{k+1}) - \\dot{F}(\\beta_k)}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} + \\delta \\geq \\delta + C \\epsilon}{\\mathcal{F}_k}\\\\\n& \\leq \\condPrb{ C \\norm{\\beta_{k+1} - \\beta_k}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq  C \\epsilon}{\\mathcal{F}_k} \\\\\n& \\leq \\condPrb{ \\norm{M_k \\dot{f}(\\beta_k, X_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq   \\epsilon}{\\mathcal{F}_k}\n\\end{align}\n","sha256_utf8":"a2ce84e9a9b313ebdb59561aa996a890b2d82f35252f527e4d266f2fa2250080"}]},{"id":"S011","semantic_type":"proof_step","kind":"proof_step","title":"条件越界概率的二阶界","source_units":[{"start":71741,"end":72237,"start_line":761,"end_line":766,"raw_tex":"Applying Markov's inequality to the last conditional probability and using \\ref{condition-noise-grad},\n\\begin{align}\n&\\condPrb{ \\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq \\delta + C \\epsilon }{\\mathcal{F}_k} \\\\\n& \\leq \\frac{\\lambda_{\\max}(M_k)^2}{\\epsilon^2}\\left[ C_1 + (C_2+1) \\norm{ \\dot{F}(\\beta_k)}_2^2 \\right] \\1{ \\norm{ \\dot{F}(\\beta_k)}_2 \\leq \\delta}  \\\\\n& \\leq \\frac{\\lambda_{\\max}(M_k)^2}{\\epsilon^2}\\left[ C_1 + (C_2+1)\\delta^2 \\right]\n\\end{align}\n","sha256_utf8":"604328cb2e484e58624a06f1b4f659d4d4137fbd3a0629402b350a95c9442c37"}]},{"id":"S012","semantic_type":"proof_step","kind":"proof_step","title":"可求和性排除固定幅度越界","source_units":[{"start":72237,"end":72570,"start_line":767,"end_line":770,"raw_tex":"By \\ref{prop-maxeig}, the sum of the right hand side is bounded with probability one. Therefore, by the conditional second Borel-Cantelli lemma (Theorem 5.3.2 of \\cite{durrett2010}),\n\\begin{equation}\n\\Prb{\\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq \\delta + C \\epsilon, ~i.o.} = 0. \n\\end{equation}\n","sha256_utf8":"b8ba14f2c21af1838891211d62aecd9f873203d83a5a2eec085b570dcd50480e"}]},{"id":"S013","semantic_type":"proof_step","kind":"proof_step","title":"作者的递减幅度论证","source_units":[{"start":72570,"end":73186,"start_line":771,"end_line":777,"raw_tex":"Since $\\epsilon > 0$ is arbitrary, then this conclusion holds for each element in the sequence $\\lbrace \\epsilon_m \\rbrace$ where $\\epsilon_m \\downarrow 0$. Therefore,\n\\begin{align}\n&\\Prb{\\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} > \\delta, ~i.o.} \\\\\n&=\\Prb{\\bigcup_{m \\in \\mathbb{N}} \\left\\lbrace \\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq \\delta + C \\epsilon_m, ~i.o. \\right\\rbrace} \\\\\n&\\leq \\sum_{m=1}^\\infty \\Prb{\\norm{ \\dot{F}(\\beta_{k+1})}_2 \\1{ \\norm{\\dot{F}(\\beta_k)}_2 \\leq \\delta} \\geq \\delta + C \\epsilon_m, ~i.o.} \\\\\n&= 0.  \n\\end{align}\n","sha256_utf8":"64938f0d8dade6e07edfb092f80a07cf44433ed45b4da8de983a1a33452d51a0"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"合并事件包含关系","source_units":[{"start":73187,"end":73328,"start_line":779,"end_line":779,"raw_tex":"Therefore, by using this result with (\\ref{eqn-inc-exc}), (\\ref{eqn-set-inc-1}) and (\\ref{eqn-set-inc-2}), we conclude the following result.\n","sha256_utf8":"44fe6b2076bb2ffea806b716373ef2cd98fa41672947ad7517d49e514f4154a4"}]},{"id":"S015","semantic_type":"theorem","kind":"theorem","title":"无限次访问小梯度区域","source_units":[{"start":67107,"end":67591,"start_line":689,"end_line":695,"raw_tex":"\\begin{theorem} \\label{theorem-revisit}\nLet $F$ be a Bottou-Curtis-Nocedal function (\\ref{bcn}) and let $\\lbrace \\beta_k \\rbrace$ be the iterates generated by Stochastic Gradient Descent satisfying \\ref{prop-sympd} to \\ref{prop-condnum} (\\S \\ref{subsection-sgd}), then, for any $\\delta > 0$,\n\\begin{equation}\n\\condPrb{ \\norm{ \\dot F( \\beta_k)}_2^2 \\leq \\delta, ~i.o.}{\\mathcal{F}_0} = 1 ~\\text{with probability 1},\n\\end{equation}\nwhere $\\mathcal{F}_0 = \\sigma(\\beta_0)$.\n\\end{theorem}","sha256_utf8":"8d82ebe9d8b43e797639d509d83b4ec4ed623f10c9e5ce474f04a126556d116d"}]},{"id":"S016","semantic_type":"proof","kind":"proof","title":"反复访问的完整证明","source_units":[{"start":57916,"end":59595,"start_line":579,"end_line":592,"raw_tex":"From these properties, we see that if (\\ref{eqn-psi-iter}) did not have the indicator term, then the only difference between $\\lbrace \\psi_k \\rbrace$ and $\\lbrace \\beta_k \\rbrace$ is the initialization---the fact that $\\lbrace P_k \\rbrace$ and $\\lbrace M_k \\rbrace$ are distinct is of little importance for our purposes as long as \\ref{prop-sympd} to \\ref{prop-condnum} are satisfied. Thus, we see that $\\lbrace \\beta_k \\rbrace$ exhibit an analogue of the strong Markov property.\n\nNow, to couple these two iterate sequences, let $\\mathcal{G}_0 = \\mathcal{F}_{\\tau+1}$ and $\\mathcal{G}_k = \\sigma(\\mathcal{F}_{\\tau+1},Z_1,\\ldots,Z_k)$, and, for $\\delta >0$, define $\\tau_\\delta$ to be a stopping time with respect to $\\lbrace \\mathcal{G}_k \\rbrace$ such that\n\\begin{equation}\n\\tau_\\delta = \\min\\left\\lbrace k \\geq 0: \\norm{ \\dot{F}(\\psi_k)}_2^2 \\leq \\delta \\right\\rbrace.\n\\end{equation} \nThen, on the event $\\lbrace k \\geq \\tau_\\delta \\rbrace$, $\\psi_{k+1} = \\psi_k$. Moreover, on the event $\\lbrace k < \\tau_\\delta \\rbrace$,\n\\begin{equation}\n\\begin{aligned}\n\\psi_{k+1} &= \\psi_{k} - P_k \\dot{f}( \\psi_k, Z_{k+1}) \\\\\n&= \\beta_{\\tau+1+k} - M_{\\tau+1 + k} \\dot{f}( \\beta_{\\tau+1+k}, X_{\\tau+2+k}) = \\beta_{\\tau+2+k}\n\\end{aligned}\n\\end{equation}\nfollows by induction. Therefore, for all $0 \\leq k \\leq \\tau_\\delta$, $\\psi_k = \\beta_{\\tau+1+k}$; that is, the sequences are coupled in this interval. Owing to this coupling, we see that $\\tau_\\delta + 1$ is the number of iterates for $\\beta_k$ to be within a ``$\\delta$-region of zero gradient'' after iterate $\\tau$. We now apply Zoutendijk's global convergence approach to conclude that $\\tau_\\delta$ is finite with probability one.\n","sha256_utf8":"adfae6fc9a0d50d902306c982f6d5459dfef8889dd1ae812b7402f61ffcee752"},{"start":59655,"end":67106,"start_line":596,"end_line":687,"raw_tex":"We now apply Zoutendijk's global convergence approach \\cite{zoutendijk1970} to $\\lbrace \\psi_k \\rbrace$ to conclude that $\\Prb{ \\tau_\\delta < \\infty} = 1$. First, by the fundamental theorem of calculus and \\ref{condition-dg-lipschitz} (recall, with constant $C$),\n\\begin{align}\nF(\\psi_{k+1}) &\\leq F(\\psi_k) + \\dot{F}(\\psi_k)'(\\psi_{k+1} - \\psi_k) + \\frac{C}{2}\\norm{ \\psi_{k+1} - \\psi_k}_2^2 \\\\\n\t\t\t  & \\begin{aligned}\n\t\t\t  &=F(\\psi_k) - \\dot{F}(\\psi_k)'P_k \\dot f (\\psi_k, Z_{k+1}) \\1{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta} \\\\\n\t\t\t  &+ \\frac{C}{2} \\norm{ P_k \\dot f(\\psi_k, Z_{k+1})}_2^2 \\1{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta}. \n\t\t\t  \\end{aligned} \\label{eqn-ftc}\n\\end{align}\nWe now take the conditional expectation of the resulting inequality with respect to $\\mathcal{G}_k$. Note, since $\\psi_k, P_k$ are measurable with respect to $ \\mathcal{G}_k$ and $Z_{k+1}$ is independent of $\\mathcal{G}_k$, then $\\cond{ F(\\psi_k)}{\\mathcal G_k} = F(\\psi_k)$ and \n\\begin{equation}\n\\cond{ \\dot{F}(\\psi_k)'P_k \\dot f(\\psi_k, Z_{k+1})}{\\mathcal{G}_k} = \\dot{F}(\\psi_k)'P_k \\dot{F}(\\psi_k).\n\\end{equation}\nFor the third term in (\\ref{eqn-ftc}), we will need to make use of \\ref{condition-noise-grad} (with parameters $C_1,C_2 \\geq 0$).\n\\begin{align}\n&\\cond{ \\norm{ P_k \\left[ \\dot f(\\psi_k, Z_{k+1}) - \\dot{F}(\\psi_k) + \\dot{F}(\\psi_k) \\right]}_2^2}{\\mathcal G_k} \\nonumber \\\\\n&= \\cond{ \\norm{P_k \\left[ \\dot f(\\psi_k,Z_{k+1}) - \\dot{F}(\\psi_k) \\right]}_2^2}{\\mathcal{G}_k} + \\dot F(\\psi_k)'P_k^2 \\dot F(\\psi_k) \\\\\n&\\leq C_1\\lambda_{\\max}(P_k)^2 + C_2 \\lambda_{\\max}(P_k)^2 \\norm{ \\dot{F}(\\psi_k)}_2^2 + \\dot F(\\psi_k)'P_k^2 \\dot F(\\psi_k)\n\\end{align}\nPutting the calculation for these three terms together in (\\ref{eqn-ftc}), we conclude\n\\begin{equation} \\label{eqn-ftc-cond}\n\\begin{aligned}\n&\\cond{F(\\psi_{k+1})}{\\mathcal{G}_k} \\leq F(\\psi_k) + \\frac{CC_1}{2} \\lambda_{\\max}(P_k)^2 -  \\1{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta} \\\\\n&\\times \\left[\\dot{F}(\\psi_k)'P_k \\dot F(\\psi_k) - \\frac{C}{2} \\dot{F}(\\psi_k)'P_k^2 \\dot F(\\psi_k) - \\frac{C C_2}{2} \\lambda_{\\max}(P_k)^2 \\norm{ \\dot F(\\psi_k)}_2^2 \\right]. \n\\end{aligned}\n\\end{equation}\nWe can find an upper bound for (\\ref{eqn-ftc-cond}) by finding a lower bound for the third term in the right hand side of the inequality. In particular, we will lower bound\n\\begin{equation} \\label{eqn-ftc-cond-min}\n\\min_{v \\in \\mathbb{S}^{p-1}} v'P_k v - \\frac{C}{2} v' P_k^2 v - \\frac{C C_2}{2} \\lambda_{\\max}(P_k)^2 \\norm{v}_2^2,\n\\end{equation}\nwhere $\\mathbb S^{p-1}$ is the unit sphere in $\\mathbb {R}^p$.\\footnote{We could drop the last term in the optimization problem as it is a constant.}\n\nUsing the Schur decomposition of $P_k$,\\footnote{Since $P_k$ is random, its Schur decomposition is random.} we can transform (\\ref{eqn-ftc-cond-min}) into the equivalent problem\n\\begin{equation} \\label{eqn-ftc-cond-min-2}\n\\min_{v \\in \\mathbb{S}^{p-1}} \\sum_{i=1}^p \\left[\\lambda_i  - \\frac{C}{2} \\lambda_i^2 - \\frac{C C_2}{2} \\lambda_1^2\\right] v_i^2,\n\\end{equation}\nwhere $\\lambda_{\\max}(P_k) = \\lambda_1 \\geq \\lambda_2 \\geq \\cdots \\geq \\lambda_p = \\lambda_{\\min}(P_k)$; and $v_i$ are the components of $v$. Applying Lemma \\ref{lemma-pk-satisfy-mk-properties}, the solution to (\\ref{eqn-ftc-cond-min-2}) is lower bounded by $\\lambda_{\\min}(P_k)/2$.\n\n\nPlugging this lower bound into (\\ref{eqn-ftc-cond}), we have that\n\\begin{equation}\n\\begin{aligned}\n\\cond{F(\\psi_{k+1})}{\\mathcal{G}_k} &\\leq F(\\psi_k) + \\frac{CC_1}{2} \\lambda_{\\max}(P_k)^2 \\\\\n&\\quad - \\frac{1}{2}\\lambda_{\\min}(P_k)\\norm{ \\dot{F} (\\psi_k)}_2^2 \\1{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta}.\n\\end{aligned}\n\\end{equation}\nRearranging and applying the condition in the indicator,\n\\begin{equation}\n\\begin{aligned}\n&\\frac{\\delta}{2}\\lambda_{\\min}(P_k) \\1{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta} \\\\\n&\\quad \\leq F(\\psi_k) - \\cond{ F(\\psi_{k+1})}{ \\mathcal{G}_k} + \\frac{C C_1}{2} \\lambda_{\\max}(P_k)^2.\n\\end{aligned}\n\\end{equation}\nNow, recall that $P_k$ are measurable with respect to $\\mathcal{F}_{\\tau+1}$ and recall that $\\mathcal{F}_{\\tau+1} \\subset \\mathcal{G}_k$ for all $k$. Therefore,\n\\begin{equation}\n\\begin{aligned}\n&\\frac{\\delta}{2}\\lambda_{\\min}(P_k) \\condPrb{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta }{ \\mathcal{F}_{\\tau+1}} \\\\\n&\\quad \\leq \\cond{ F(\\psi_k) - F(\\psi_{k+1})}{\\mathcal{F}_{\\tau+1}} + \\frac{C C_1}{2} \\lambda_{\\max}(P_k)^2.\n\\end{aligned}\n\\end{equation}\nMoreover, by (a) summing both sides from $k=0$ to $n \\in \\mathbb{N}$, (b) recalling that $F(\\psi_0)$ is finite with probability one given $\\mathcal{F}_{\\tau+1}$, and (c) applying \\ref{prop-maxeig} from Lemma \\ref{lemma-pk-satisfy-mk-properties}, we conclude\n\\begin{equation}\n\\begin{aligned}\n&\\frac{\\delta}{2} \\sum_{k=0}^n \\lambda_{\\min}(P_k) \\condPrb{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta }{ \\mathcal{F}_{\\tau+1}} \\\\\n&\\quad \\leq F(\\psi_0) - \\cond{ F(\\psi_{n+1})}{\\mathcal{F}_{\\tau+1}} + \\frac{SCC_1}{2}.\n\\end{aligned}\n\\end{equation}\n\nRecall that, we have assumed that $F(\\theta)$ is bounded from below by some constant $F_{l.b.}$ as a core assumption. Using this, we see that\n\\begin{equation}\n\\begin{aligned}\n\\frac{\\delta}{2} \\sum_{k=0}^n \\lambda_{\\min}(P_k) \\condPrb{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta }{ \\mathcal{F}_{\\tau+1}} \n\\leq F(\\psi_0) - F_{l.b.} + \\frac{SCC_1}{2}.\n\\end{aligned}\n\\end{equation}\n\nMoreover, note that, for all $k \\geq 0$,\n\\begin{equation}\n\\condPrb{ \\bigcap_{j=0}^\\infty \\left\\lbrace \\norm{ \\dot{F}(\\psi_j)}_2^2 > \\delta \\right\\rbrace }{ \\mathcal{F}_{\\tau+1}} \\leq \\condPrb{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta }{ \\mathcal{F}_{\\tau+1}}. \n\\end{equation}\nTherefore, for arbitrary $n$,\n\\begin{equation}\n\\condPrb{ \\bigcap_{j=0}^\\infty \\left\\lbrace \\norm{ \\dot{F}(\\psi_j)}_2^2 > \\delta \\right\\rbrace }{ \\mathcal{F}_{\\tau+1}} \\leq \\frac{F(\\psi_0) - F_{l.b.} + S C C_1 /2 }{(\\delta/2)\\sum_{k=0}^n \\lambda_{\\min}(P_k)}\n\\end{equation} \nBy \\ref{prop-mineig} from Lemma \\ref{lemma-pk-satisfy-mk-properties}, the right hand side of this inequality can be made arbitrarily small, which implies that the conditional probability on the left hand side is zero.\n\nThat is,\n\\begin{equation}\n0 = \\condPrb{ \\bigcap_{j=0}^\\infty  \\left\\lbrace \\norm{ \\dot{F}(\\psi_j)}_2^2 > \\delta \\right\\rbrace }{ \\mathcal{F}_{\\tau+1}} = \\condPrb{ \\tau_{\\delta} = \\infty}{\\mathcal{F}_{\\tau+1}}.\n\\end{equation}\nIn other words, we have concluded that $\\condPrb{ \\tau_\\delta < \\infty}{\\mathcal{F}_{\\tau+1}} = 1$ with probability one for any finite stopping time $\\tau$. With this result, our last step is to use induction. \n\nFor now, we will say that the iterates are in a $\\delta$-region of zero gradient if the squared-norm of the gradient of the iterate is no greater than $\\delta$. If we let $\\tau = -1$, then $\\tau_\\delta$ is the first time the iterates enter a $\\delta$-region of zero gradient. Let $T_1(\\delta) = \\tau_\\delta$ when $\\tau = -1$. Then, from the above argument, we have shown that $T_1(\\delta)$ is a finite stopping time. Now, define $T_j(\\delta)$ to be the $j^{th}$ time that the iterates enter a $\\delta$-region of zero gradient. Suppose that $T_j(\\delta)$ is finite. Then, define $\\tau = T_j(\\delta)$. Then, $\\tau_\\delta$ for this $\\tau$ is the next time that the iterates enter a $\\delta$-region of zero gradient. That is, $T_{j+1}(\\delta) = \\tau_\\delta + T_j(\\delta)$. Since we have assumed that $\\tau = T_{j}(\\delta)$ is finite, we conclude that $\\tau_\\delta$ is finite, which implies that $T_{j+1}(\\delta)$ is finite. Therefore, by induction we have proven the following result.\n","sha256_utf8":"c2a32b3708a3723f504e6728f6e28345cc2de2fc8a7f8e6eb11f24d06571513f"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"重启性质与首次进入时刻","source_units":[{"start":57916,"end":58803,"start_line":579,"end_line":584,"raw_tex":"From these properties, we see that if (\\ref{eqn-psi-iter}) did not have the indicator term, then the only difference between $\\lbrace \\psi_k \\rbrace$ and $\\lbrace \\beta_k \\rbrace$ is the initialization---the fact that $\\lbrace P_k \\rbrace$ and $\\lbrace M_k \\rbrace$ are distinct is of little importance for our purposes as long as \\ref{prop-sympd} to \\ref{prop-condnum} are satisfied. Thus, we see that $\\lbrace \\beta_k \\rbrace$ exhibit an analogue of the strong Markov property.\n\nNow, to couple these two iterate sequences, let $\\mathcal{G}_0 = \\mathcal{F}_{\\tau+1}$ and $\\mathcal{G}_k = \\sigma(\\mathcal{F}_{\\tau+1},Z_1,\\ldots,Z_k)$, and, for $\\delta >0$, define $\\tau_\\delta$ to be a stopping time with respect to $\\lbrace \\mathcal{G}_k \\rbrace$ such that\n\\begin{equation}\n\\tau_\\delta = \\min\\left\\lbrace k \\geq 0: \\norm{ \\dot{F}(\\psi_k)}_2^2 \\leq \\delta \\right\\rbrace.\n\\end{equation} \n","sha256_utf8":"20f64a9e31570d70a2e982deb173218527f134b87403838575b02804f00e3ec8"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"停止前两序列的耦合","source_units":[{"start":58803,"end":59595,"start_line":585,"end_line":592,"raw_tex":"Then, on the event $\\lbrace k \\geq \\tau_\\delta \\rbrace$, $\\psi_{k+1} = \\psi_k$. Moreover, on the event $\\lbrace k < \\tau_\\delta \\rbrace$,\n\\begin{equation}\n\\begin{aligned}\n\\psi_{k+1} &= \\psi_{k} - P_k \\dot{f}( \\psi_k, Z_{k+1}) \\\\\n&= \\beta_{\\tau+1+k} - M_{\\tau+1 + k} \\dot{f}( \\beta_{\\tau+1+k}, X_{\\tau+2+k}) = \\beta_{\\tau+2+k}\n\\end{aligned}\n\\end{equation}\nfollows by induction. Therefore, for all $0 \\leq k \\leq \\tau_\\delta$, $\\psi_k = \\beta_{\\tau+1+k}$; that is, the sequences are coupled in this interval. Owing to this coupling, we see that $\\tau_\\delta + 1$ is the number of iterates for $\\beta_k$ to be within a ``$\\delta$-region of zero gradient'' after iterate $\\tau$. We now apply Zoutendijk's global convergence approach to conclude that $\\tau_\\delta$ is finite with probability one.\n","sha256_utf8":"4a4f3bfe98d8b10e78191ff77a86f69c5dc96792cdc0bdc4594222c4762f39d9"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"目标函数的一步下降界","source_units":[{"start":59655,"end":60334,"start_line":596,"end_line":603,"raw_tex":"We now apply Zoutendijk's global convergence approach \\cite{zoutendijk1970} to $\\lbrace \\psi_k \\rbrace$ to conclude that $\\Prb{ \\tau_\\delta < \\infty} = 1$. First, by the fundamental theorem of calculus and \\ref{condition-dg-lipschitz} (recall, with constant $C$),\n\\begin{align}\nF(\\psi_{k+1}) &\\leq F(\\psi_k) + \\dot{F}(\\psi_k)'(\\psi_{k+1} - \\psi_k) + \\frac{C}{2}\\norm{ \\psi_{k+1} - \\psi_k}_2^2 \\\\\n\t\t\t  & \\begin{aligned}\n\t\t\t  &=F(\\psi_k) - \\dot{F}(\\psi_k)'P_k \\dot f (\\psi_k, Z_{k+1}) \\1{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta} \\\\\n\t\t\t  &+ \\frac{C}{2} \\norm{ P_k \\dot f(\\psi_k, Z_{k+1})}_2^2 \\1{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta}. \n\t\t\t  \\end{aligned} \\label{eqn-ftc}\n\\end{align}\n","sha256_utf8":"6a04c6202f095148f6ff7aaf649b0dc13ff60ac849036841480474123b8bcd83"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"条件均值与噪声二阶矩","source_units":[{"start":60334,"end":61298,"start_line":604,"end_line":613,"raw_tex":"We now take the conditional expectation of the resulting inequality with respect to $\\mathcal{G}_k$. Note, since $\\psi_k, P_k$ are measurable with respect to $ \\mathcal{G}_k$ and $Z_{k+1}$ is independent of $\\mathcal{G}_k$, then $\\cond{ F(\\psi_k)}{\\mathcal G_k} = F(\\psi_k)$ and \n\\begin{equation}\n\\cond{ \\dot{F}(\\psi_k)'P_k \\dot f(\\psi_k, Z_{k+1})}{\\mathcal{G}_k} = \\dot{F}(\\psi_k)'P_k \\dot{F}(\\psi_k).\n\\end{equation}\nFor the third term in (\\ref{eqn-ftc}), we will need to make use of \\ref{condition-noise-grad} (with parameters $C_1,C_2 \\geq 0$).\n\\begin{align}\n&\\cond{ \\norm{ P_k \\left[ \\dot f(\\psi_k, Z_{k+1}) - \\dot{F}(\\psi_k) + \\dot{F}(\\psi_k) \\right]}_2^2}{\\mathcal G_k} \\nonumber \\\\\n&= \\cond{ \\norm{P_k \\left[ \\dot f(\\psi_k,Z_{k+1}) - \\dot{F}(\\psi_k) \\right]}_2^2}{\\mathcal{G}_k} + \\dot F(\\psi_k)'P_k^2 \\dot F(\\psi_k) \\\\\n&\\leq C_1\\lambda_{\\max}(P_k)^2 + C_2 \\lambda_{\\max}(P_k)^2 \\norm{ \\dot{F}(\\psi_k)}_2^2 + \\dot F(\\psi_k)'P_k^2 \\dot F(\\psi_k)\n\\end{align}\n","sha256_utf8":"a3234451b38a0cb18899d10d0238f598ddba794a258d225589a4d2e98c043bec"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"条件下降归约为二次型","source_units":[{"start":61298,"end":62281,"start_line":614,"end_line":625,"raw_tex":"Putting the calculation for these three terms together in (\\ref{eqn-ftc}), we conclude\n\\begin{equation} \\label{eqn-ftc-cond}\n\\begin{aligned}\n&\\cond{F(\\psi_{k+1})}{\\mathcal{G}_k} \\leq F(\\psi_k) + \\frac{CC_1}{2} \\lambda_{\\max}(P_k)^2 -  \\1{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta} \\\\\n&\\times \\left[\\dot{F}(\\psi_k)'P_k \\dot F(\\psi_k) - \\frac{C}{2} \\dot{F}(\\psi_k)'P_k^2 \\dot F(\\psi_k) - \\frac{C C_2}{2} \\lambda_{\\max}(P_k)^2 \\norm{ \\dot F(\\psi_k)}_2^2 \\right]. \n\\end{aligned}\n\\end{equation}\nWe can find an upper bound for (\\ref{eqn-ftc-cond}) by finding a lower bound for the third term in the right hand side of the inequality. In particular, we will lower bound\n\\begin{equation} \\label{eqn-ftc-cond-min}\n\\min_{v \\in \\mathbb{S}^{p-1}} v'P_k v - \\frac{C}{2} v' P_k^2 v - \\frac{C C_2}{2} \\lambda_{\\max}(P_k)^2 \\norm{v}_2^2,\n\\end{equation}\nwhere $\\mathbb S^{p-1}$ is the unit sphere in $\\mathbb {R}^p$.\\footnote{We could drop the last term in the optimization problem as it is a constant.}\n","sha256_utf8":"db065c929dc5d72da89afdacf414bebc6ccb1f9a5424f3758dcc1a8cc039de90"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"谱分解给出统一下界","source_units":[{"start":62282,"end":62932,"start_line":627,"end_line":631,"raw_tex":"Using the Schur decomposition of $P_k$,\\footnote{Since $P_k$ is random, its Schur decomposition is random.} we can transform (\\ref{eqn-ftc-cond-min}) into the equivalent problem\n\\begin{equation} \\label{eqn-ftc-cond-min-2}\n\\min_{v \\in \\mathbb{S}^{p-1}} \\sum_{i=1}^p \\left[\\lambda_i  - \\frac{C}{2} \\lambda_i^2 - \\frac{C C_2}{2} \\lambda_1^2\\right] v_i^2,\n\\end{equation}\nwhere $\\lambda_{\\max}(P_k) = \\lambda_1 \\geq \\lambda_2 \\geq \\cdots \\geq \\lambda_p = \\lambda_{\\min}(P_k)$; and $v_i$ are the components of $v$. Applying Lemma \\ref{lemma-pk-satisfy-mk-properties}, the solution to (\\ref{eqn-ftc-cond-min-2}) is lower bounded by $\\lambda_{\\min}(P_k)/2$.\n","sha256_utf8":"887093c85af14b4027dd1bf48955e7d2ae77e9df267fe0950925ace9b8824afa"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"得到带指示项的漂移界","source_units":[{"start":62934,"end":63571,"start_line":634,"end_line":647,"raw_tex":"Plugging this lower bound into (\\ref{eqn-ftc-cond}), we have that\n\\begin{equation}\n\\begin{aligned}\n\\cond{F(\\psi_{k+1})}{\\mathcal{G}_k} &\\leq F(\\psi_k) + \\frac{CC_1}{2} \\lambda_{\\max}(P_k)^2 \\\\\n&\\quad - \\frac{1}{2}\\lambda_{\\min}(P_k)\\norm{ \\dot{F} (\\psi_k)}_2^2 \\1{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta}.\n\\end{aligned}\n\\end{equation}\nRearranging and applying the condition in the indicator,\n\\begin{equation}\n\\begin{aligned}\n&\\frac{\\delta}{2}\\lambda_{\\min}(P_k) \\1{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta} \\\\\n&\\quad \\leq F(\\psi_k) - \\cond{ F(\\psi_{k+1})}{ \\mathcal{G}_k} + \\frac{C C_1}{2} \\lambda_{\\max}(P_k)^2.\n\\end{aligned}\n\\end{equation}\n","sha256_utf8":"af40e1f0cd0d3b105a32fb73a6685527c0de8096eafc2feac9f84af5e3d93532"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"条件化并望远镜求和","source_units":[{"start":63571,"end":64549,"start_line":648,"end_line":661,"raw_tex":"Now, recall that $P_k$ are measurable with respect to $\\mathcal{F}_{\\tau+1}$ and recall that $\\mathcal{F}_{\\tau+1} \\subset \\mathcal{G}_k$ for all $k$. Therefore,\n\\begin{equation}\n\\begin{aligned}\n&\\frac{\\delta}{2}\\lambda_{\\min}(P_k) \\condPrb{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta }{ \\mathcal{F}_{\\tau+1}} \\\\\n&\\quad \\leq \\cond{ F(\\psi_k) - F(\\psi_{k+1})}{\\mathcal{F}_{\\tau+1}} + \\frac{C C_1}{2} \\lambda_{\\max}(P_k)^2.\n\\end{aligned}\n\\end{equation}\nMoreover, by (a) summing both sides from $k=0$ to $n \\in \\mathbb{N}$, (b) recalling that $F(\\psi_0)$ is finite with probability one given $\\mathcal{F}_{\\tau+1}$, and (c) applying \\ref{prop-maxeig} from Lemma \\ref{lemma-pk-satisfy-mk-properties}, we conclude\n\\begin{equation}\n\\begin{aligned}\n&\\frac{\\delta}{2} \\sum_{k=0}^n \\lambda_{\\min}(P_k) \\condPrb{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta }{ \\mathcal{F}_{\\tau+1}} \\\\\n&\\quad \\leq F(\\psi_0) - \\cond{ F(\\psi_{n+1})}{\\mathcal{F}_{\\tau+1}} + \\frac{SCC_1}{2}.\n\\end{aligned}\n\\end{equation}\n","sha256_utf8":"f2eb5b7b322799551517d68914f24169bb9b5ff198d68b9e7f908643fc7c0dda"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"目标下界控制累计概率","source_units":[{"start":64550,"end":64922,"start_line":663,"end_line":669,"raw_tex":"Recall that, we have assumed that $F(\\theta)$ is bounded from below by some constant $F_{l.b.}$ as a core assumption. Using this, we see that\n\\begin{equation}\n\\begin{aligned}\n\\frac{\\delta}{2} \\sum_{k=0}^n \\lambda_{\\min}(P_k) \\condPrb{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta }{ \\mathcal{F}_{\\tau+1}} \n\\leq F(\\psi_0) - F_{l.b.} + \\frac{SCC_1}{2}.\n\\end{aligned}\n\\end{equation}\n","sha256_utf8":"b5e15df9e25a8bde80be2949233ddec7310b102e6f59223f8a34ed1d0bb0e567"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"发散权重排除永不进入","source_units":[{"start":64923,"end":65687,"start_line":671,"end_line":679,"raw_tex":"Moreover, note that, for all $k \\geq 0$,\n\\begin{equation}\n\\condPrb{ \\bigcap_{j=0}^\\infty \\left\\lbrace \\norm{ \\dot{F}(\\psi_j)}_2^2 > \\delta \\right\\rbrace }{ \\mathcal{F}_{\\tau+1}} \\leq \\condPrb{ \\norm{ \\dot{F}(\\psi_k)}_2^2 > \\delta }{ \\mathcal{F}_{\\tau+1}}. \n\\end{equation}\nTherefore, for arbitrary $n$,\n\\begin{equation}\n\\condPrb{ \\bigcap_{j=0}^\\infty \\left\\lbrace \\norm{ \\dot{F}(\\psi_j)}_2^2 > \\delta \\right\\rbrace }{ \\mathcal{F}_{\\tau+1}} \\leq \\frac{F(\\psi_0) - F_{l.b.} + S C C_1 /2 }{(\\delta/2)\\sum_{k=0}^n \\lambda_{\\min}(P_k)}\n\\end{equation} \nBy \\ref{prop-mineig} from Lemma \\ref{lemma-pk-satisfy-mk-properties}, the right hand side of this inequality can be made arbitrarily small, which implies that the conditional probability on the left hand side is zero.\n","sha256_utf8":"a6ddfe80dd3355ddae4d1ad7c5db68edb656c3f37e31c6250a3b5d87499e63f0"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"任意有限重启后的进入性","source_units":[{"start":65688,"end":66124,"start_line":681,"end_line":685,"raw_tex":"That is,\n\\begin{equation}\n0 = \\condPrb{ \\bigcap_{j=0}^\\infty  \\left\\lbrace \\norm{ \\dot{F}(\\psi_j)}_2^2 > \\delta \\right\\rbrace }{ \\mathcal{F}_{\\tau+1}} = \\condPrb{ \\tau_{\\delta} = \\infty}{\\mathcal{F}_{\\tau+1}}.\n\\end{equation}\nIn other words, we have concluded that $\\condPrb{ \\tau_\\delta < \\infty}{\\mathcal{F}_{\\tau+1}} = 1$ with probability one for any finite stopping time $\\tau$. With this result, our last step is to use induction. \n","sha256_utf8":"f9cba381196cc01d5eb7f8dc63038cbe08b4af35fc9e779749d3e338b91d4c6c"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"归纳得到无限次访问","source_units":[{"start":66125,"end":67106,"start_line":687,"end_line":687,"raw_tex":"For now, we will say that the iterates are in a $\\delta$-region of zero gradient if the squared-norm of the gradient of the iterate is no greater than $\\delta$. If we let $\\tau = -1$, then $\\tau_\\delta$ is the first time the iterates enter a $\\delta$-region of zero gradient. Let $T_1(\\delta) = \\tau_\\delta$ when $\\tau = -1$. Then, from the above argument, we have shown that $T_1(\\delta)$ is a finite stopping time. Now, define $T_j(\\delta)$ to be the $j^{th}$ time that the iterates enter a $\\delta$-region of zero gradient. Suppose that $T_j(\\delta)$ is finite. Then, define $\\tau = T_j(\\delta)$. Then, $\\tau_\\delta$ for this $\\tau$ is the next time that the iterates enter a $\\delta$-region of zero gradient. That is, $T_{j+1}(\\delta) = \\tau_\\delta + T_j(\\delta)$. Since we have assumed that $\\tau = T_{j}(\\delta)$ is finite, we conclude that $\\tau_\\delta$ is finite, which implies that $T_{j+1}(\\delta)$ is finite. Therefore, by induction we have proven the following result.\n","sha256_utf8":"983d157e3756158441ab706ec1199c9887a6b0e5f2a885657f8ccfe89f2680e7"}]},{"id":"S029","semantic_type":"lemma","kind":"lemma","title":"最终成立的特征值下界","source_units":[{"start":52698,"end":53263,"start_line":501,"end_line":506,"raw_tex":"\\begin{lemma} \\label{lemma-minimality-smallest-eigenvalue}\nSuppose $\\lbrace M_k \\rbrace$ satisfy \\ref{prop-sympd} to \\ref{prop-condnum}, and recall that $C$ is the Lipschitz constant in \\ref{condition-dg-lipschitz} and $C_2$ is the scaling constant in \\ref{condition-noise-grad}. There exists a $K \\in \\mathbb{N}$ such that, for all $k \\geq K$ and for any eigenvalue, $\\lambda$ of $M_k$,\n\\begin{equation} \\label{eqn-grad-norm-lb}\n\\frac{1}{2} \\lambda_{\\min}(M_k) \\leq \\lambda - \\frac{C}{2}\\lambda^2 - \\frac{C C_2}{2} \\lambda_{\\max}(M_k)^2.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"84c4dc79ae9d562d3b051292246ae4e78674f2e2234ea0014f17629bfe86d641"}]},{"id":"S030","semantic_type":"proof","kind":"proof","title":"特征值下界的完整证明","source_units":[{"start":53265,"end":55444,"start_line":508,"end_line":540,"raw_tex":"\\begin{proof}\nWe begin by showing that right hand side of \\eqref{eqn-grad-norm-lb} is minimized when $\\lambda = \\lambda_{\\min}(M_k)$ for all $k$ sufficiently large and then establish \\eqref{eqn-grad-norm-lb} for this choice of $\\lambda$. \nWe first establish\n\\begin{equation} \\label{eqn-def-K}\n\\lambda_{\\max}(M_k) + \\lambda_{\\min}(M_k) \\leq \\frac{2}{C}.\n\\end{equation}\nBy \\ref{prop-maxeig}, $\\lambda_{\\max}(M_k) \\to 0$. Therefore, there exists a $k_0 \\in \\mathbb{N}$ such that for all $k \\geq k_0$, $\\lambda_{\\max}(M_k) \\leq 1/C$. Therefore, (\\ref{eqn-def-K}) follows for all $k \\geq k_0$. Moreover, for every $k \\geq k_0$, we see that (\\ref{eqn-def-K}) holds for any other eigenvalue of $M_k$ in place of $\\lambda_{\\max}(M_k)$. Letting $\\lambda$ denote any eigenvalue of $M_k$ and noting that $\\lambda - \\lambda_{\\min}(M_k) \\geq 0$,\n\\begin{equation}\n(\\lambda - \\lambda_{\\min}(M_k) ) (\\lambda + \\lambda_{\\min}(M_k) ) \\leq \\frac{2}{C}(\\lambda - \\lambda_{\\min}(M_k) ),\n\\end{equation}\nwhich can be rearranged to conclude that, for any $k \\geq k_0$ and for any eigenvalue $\\lambda$ of $M_k$,\n\\begin{equation}\n\\begin{aligned}\n\\lambda_{\\min}(M_k) - \\frac{C}{2}\\lambda_{\\min}(M_k)^2 \\leq \\lambda - \\frac{C}{2}\\lambda^2.\n\\end{aligned}\n\\end{equation} \nIt follows that the right hand side of \\eqref{eqn-grad-norm-lb} is minimized by $\\lambda_{\\min}(M_k)$.\n\nWe now establish \\eqref{eqn-grad-norm-lb} with $\\lambda = \\lambda_{\\min}(M_k)$ by contradiction. Suppose that there exists a subsequence $\\lbrace k_j \\rbrace$ for which \\eqref{eqn-grad-norm-lb} does not hold; that is, by rearranging \\eqref{eqn-grad-norm-lb},\n\\begin{equation}\nC \\lambda_{\\min}(M_{k_j})^2 + CC_2 \\lambda_{\\max}(M_{k_j})^2 > \\lambda_{\\min}(M_{k_j}).\n\\end{equation}\nTaking an upper bound on the left hand side,\n\\begin{align}\n2C(C_2 + 1) \\lambda_{\\max}(M_{k_j})^2 > \\lambda_{\\min}(M_{k_j}).\n\\end{align}\nHence,\n\\begin{equation} \\label{eqn-relaxed-condnum-control}\n\\lambda_{\\max}(M_{k_j}) \\kappa( M_{k_j} ) > \\frac{1}{2 C(C_2 + 1)},\n\\end{equation}\nwhich contradicts \\ref{prop-condnum} for $j$ sufficiently large. Hence, there is a $K \\geq k_0$ such that, for all $k \\geq K$, \\eqref{eqn-grad-norm-lb} holds.\n\\qed\n\\end{proof}","sha256_utf8":"0cecee5390e6240a281b6b311e252815eb980eaa7994f6746b863bc18711658b"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"最大特征值最终充分小","source_units":[{"start":53279,"end":54098,"start_line":509,"end_line":514,"raw_tex":"We begin by showing that right hand side of \\eqref{eqn-grad-norm-lb} is minimized when $\\lambda = \\lambda_{\\min}(M_k)$ for all $k$ sufficiently large and then establish \\eqref{eqn-grad-norm-lb} for this choice of $\\lambda$. \nWe first establish\n\\begin{equation} \\label{eqn-def-K}\n\\lambda_{\\max}(M_k) + \\lambda_{\\min}(M_k) \\leq \\frac{2}{C}.\n\\end{equation}\nBy \\ref{prop-maxeig}, $\\lambda_{\\max}(M_k) \\to 0$. Therefore, there exists a $k_0 \\in \\mathbb{N}$ such that for all $k \\geq k_0$, $\\lambda_{\\max}(M_k) \\leq 1/C$. Therefore, (\\ref{eqn-def-K}) follows for all $k \\geq k_0$. Moreover, for every $k \\geq k_0$, we see that (\\ref{eqn-def-K}) holds for any other eigenvalue of $M_k$ in place of $\\lambda_{\\max}(M_k)$. Letting $\\lambda$ denote any eigenvalue of $M_k$ and noting that $\\lambda - \\lambda_{\\min}(M_k) \\geq 0$,\n","sha256_utf8":"989364ad661ca494294cc854ffbdc35469c050e16e727006b74ab5c796067e57"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"最小特征值实现最小值","source_units":[{"start":54098,"end":54610,"start_line":515,"end_line":524,"raw_tex":"\\begin{equation}\n(\\lambda - \\lambda_{\\min}(M_k) ) (\\lambda + \\lambda_{\\min}(M_k) ) \\leq \\frac{2}{C}(\\lambda - \\lambda_{\\min}(M_k) ),\n\\end{equation}\nwhich can be rearranged to conclude that, for any $k \\geq k_0$ and for any eigenvalue $\\lambda$ of $M_k$,\n\\begin{equation}\n\\begin{aligned}\n\\lambda_{\\min}(M_k) - \\frac{C}{2}\\lambda_{\\min}(M_k)^2 \\leq \\lambda - \\frac{C}{2}\\lambda^2.\n\\end{aligned}\n\\end{equation} \nIt follows that the right hand side of \\eqref{eqn-grad-norm-lb} is minimized by $\\lambda_{\\min}(M_k)$.\n","sha256_utf8":"c768bae099a066956cbb7e21bf5f1f1891d7b158d859c4e2d7e9d89733b079fd"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"条件数控制排除反例子列","source_units":[{"start":54611,"end":55444,"start_line":526,"end_line":540,"raw_tex":"We now establish \\eqref{eqn-grad-norm-lb} with $\\lambda = \\lambda_{\\min}(M_k)$ by contradiction. Suppose that there exists a subsequence $\\lbrace k_j \\rbrace$ for which \\eqref{eqn-grad-norm-lb} does not hold; that is, by rearranging \\eqref{eqn-grad-norm-lb},\n\\begin{equation}\nC \\lambda_{\\min}(M_{k_j})^2 + CC_2 \\lambda_{\\max}(M_{k_j})^2 > \\lambda_{\\min}(M_{k_j}).\n\\end{equation}\nTaking an upper bound on the left hand side,\n\\begin{align}\n2C(C_2 + 1) \\lambda_{\\max}(M_{k_j})^2 > \\lambda_{\\min}(M_{k_j}).\n\\end{align}\nHence,\n\\begin{equation} \\label{eqn-relaxed-condnum-control}\n\\lambda_{\\max}(M_{k_j}) \\kappa( M_{k_j} ) > \\frac{1}{2 C(C_2 + 1)},\n\\end{equation}\nwhich contradicts \\ref{prop-condnum} for $j$ sufficiently large. Hence, there is a $K \\geq k_0$ such that, for all $k \\geq K$, \\eqref{eqn-grad-norm-lb} holds.\n\\qed\n\\end{proof}","sha256_utf8":"ea66f8c5ec4262f2c66093e057e8a3bbe707bbcdcc4fc8196656d7052eb74e9e"}]},{"id":"S034","semantic_type":"lemma","kind":"lemma","title":"重启矩阵继承全部条件","source_units":[{"start":56541,"end":56928,"start_line":554,"end_line":559,"raw_tex":"\\begin{lemma} \\label{lemma-pk-satisfy-mk-properties}\nWith probability one, $\\lbrace P_k \\rbrace$ satisfy \\ref{prop-sympd} to \\ref{prop-condnum}. Moreover, with probability one, for all $k \\geq 0$ and any eigenvalue $\\lambda$ of $P_k$,\n\\begin{equation}\n\\frac{1}{2}\\lambda_{\\min}(P_k) \\leq \\lambda - \\frac{C}{2} \\lambda^2 - \\frac{C C_2}{2} \\lambda_{\\max}(P_k)^2.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"cf8b0af9e5820f8c92191bcff6fa2fe4ef0dcb39006900cb926e2f1df2a51ff5"}]},{"id":"S035","semantic_type":"proof","kind":"proof","title":"重启条件的完整证明","source_units":[{"start":56929,"end":57914,"start_line":560,"end_line":577,"raw_tex":"\\begin{proof}\nThe result follows from a standard divide and conquer argument. For \\ref{prop-sympd},\n\\begin{align}\n&\\Prb{ P_k = P_k', P_k \\succ 0 } = \\Prb{ M_{\\tau+1+k}' = M_{\\tau+1+k}, M_{\\tau+1+k} \\succ 0} \\\\\n&= \\sum_{j=0}^\\infty \\condPrb{ M_{j + 1 +k}' = M_{j + 1 +k}, M_{j+1+k} \\succ 0}{\\tau = j} \\Prb{ \\tau = j} \\\\\n&= \\sum_{j=0}^\\infty \\underbrace{\\Prb{ M_{j + 1 +k}' = M_{j + 1 +k}, M_{j+1+k} \\succ 0 }}_{=1~\\text{by \\ref{prop-sympd} on } \\lbrace M_k \\rbrace } \\Prb{ \\tau=j} \\\\\n&= \\sum_{j=0}^\\infty \\Prb{ \\tau = j} \\\\\n&= \\Prb{ \\tau < \\infty } = 1.\n\\end{align}\nSimilarly, for \\ref{prop-maxeig},\n\\begin{align}\n&\\Prb{ \\sum_{k=0}^\\infty \\lambda_{\\max}(P_k)^2 < S} \\\\\n&= \\sum_{j=0}^\\infty \\underbrace{\\condPrb{ \\sum_{k=0}^\\infty \\lambda_{\\max}(M_{j+1+k})^2 < S}{\\tau=j}}_{=1~\\text{by \\ref{prop-maxeig} on } \\lbrace M_k \\rbrace} \\Prb{ \\tau=j} \\\\\n&= \\Prb{ \\tau < \\infty} =1.\n\\end{align}\nThe analogous arguments will show that the remaining conclusions of the lemma hold.\n\\qed\n\\end{proof}","sha256_utf8":"1b1334f523a02c037f3d05bec9be45198185f8afbc04644b2de5fb77e5aa8b2d"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"按停止时刻分解正定事件","source_units":[{"start":56943,"end":57494,"start_line":561,"end_line":568,"raw_tex":"The result follows from a standard divide and conquer argument. For \\ref{prop-sympd},\n\\begin{align}\n&\\Prb{ P_k = P_k', P_k \\succ 0 } = \\Prb{ M_{\\tau+1+k}' = M_{\\tau+1+k}, M_{\\tau+1+k} \\succ 0} \\\\\n&= \\sum_{j=0}^\\infty \\condPrb{ M_{j + 1 +k}' = M_{j + 1 +k}, M_{j+1+k} \\succ 0}{\\tau = j} \\Prb{ \\tau = j} \\\\\n&= \\sum_{j=0}^\\infty \\underbrace{\\Prb{ M_{j + 1 +k}' = M_{j + 1 +k}, M_{j+1+k} \\succ 0 }}_{=1~\\text{by \\ref{prop-sympd} on } \\lbrace M_k \\rbrace } \\Prb{ \\tau=j} \\\\\n&= \\sum_{j=0}^\\infty \\Prb{ \\tau = j} \\\\\n&= \\Prb{ \\tau < \\infty } = 1.\n\\end{align}\n","sha256_utf8":"9e7479dc19f6a7aa284a3c335e02896f852e602f571744896a1bd21b8d08c9f1"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"平方可求和性的继承","source_units":[{"start":57494,"end":57814,"start_line":569,"end_line":574,"raw_tex":"Similarly, for \\ref{prop-maxeig},\n\\begin{align}\n&\\Prb{ \\sum_{k=0}^\\infty \\lambda_{\\max}(P_k)^2 < S} \\\\\n&= \\sum_{j=0}^\\infty \\underbrace{\\condPrb{ \\sum_{k=0}^\\infty \\lambda_{\\max}(M_{j+1+k})^2 < S}{\\tau=j}}_{=1~\\text{by \\ref{prop-maxeig} on } \\lbrace M_k \\rbrace} \\Prb{ \\tau=j} \\\\\n&= \\Prb{ \\tau < \\infty} =1.\n\\end{align}\n","sha256_utf8":"ed9d59f6a8140fd9723ec530c02a66a8059664cd66497faf41e9912a004ac278"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"其余条件的同类论证","source_units":[{"start":57814,"end":57914,"start_line":575,"end_line":577,"raw_tex":"The analogous arguments will show that the remaining conclusions of the lemma hold.\n\\qed\n\\end{proof}","sha256_utf8":"3631b3588bb6a365e276175d94fead6a8648a63e830ab78af1c33ad372d563a0"}]},{"id":"S039","semantic_type":"problem_setting","kind":"problem_setting","title":"期望目标优化问题","source_units":[{"start":4951,"end":5096,"start_line":139,"end_line":142,"raw_tex":"In data-driven and simulation-based disciplines, the optimization problem\n\\begin{equation} \\label{problem}\n\\min_{\\theta} F(\\theta)\n\\end{equation}","sha256_utf8":"bc44877bae2066e61119d1eef600cfde3252e8d69d362bade444f1cf65dae8a8"},{"start":5097,"end":5997,"start_line":143,"end_line":144,"raw_tex":"is frequently solved, \nwhere $F(\\theta) = \\E{ f(\\theta,X)}$; $f: \\mathbb{R}^p \\times \\mathbb{R}^d \\to \\mathbb{R}$; $X$ is a random variable with arbitrary support; and $\\mathbb{E}$ is the expectation operator. Depending on $X$, the optimization problem's objective function and its gradient may be impractical or impossible to evaluate directly \\cite{bertsekas2011}. Fortunately, when certain regularity conditions hold on $f$ and $X$ (e.g., \\cite{bottou2018,wu2009}), the optimization problem's structure is exploited to generate solvers that use the gradient of $f$ with respect to the argument $\\theta$ for independent copies of $X$ \\cite{robbins1951,chung1954}. Moreover, when the gradient of $f$ is significantly cheaper to compute than the gradient of $F$, the optimization problem can be efficiently solved using these so-called Stochastic Gradient Descent (SGD) methods \\cite{bertsekas2011}.\n","sha256_utf8":"c7ea4f3a9c3c25b68fecf8b2f9ce4f14f09840a1dce3760ab00e585f11713cbc"}]},{"id":"S040","semantic_type":"notation","kind":"notation","title":"函数矩阵与概率记号","source_units":[{"start":12433,"end":13811,"start_line":168,"end_line":176,"raw_tex":"\\nomenclature[A, 1]{$F$}{Objective function that maps $\\mathbb{R}^p \\to \\mathbb{R}$. Also referred to as ``Deterministic Objective.''}\n\\nomenclature[A, 2]{$\\dot F$}{Gradient function of the deterministic objective that maps $\\mathbb{R}^p \\to \\mathbb{R}^p$. Also referred to as ``Deterministic Gradient.''}\n\\nomenclature[A, 3]{$f$}{A function that maps $\\mathbb{R}^p \\times \\mathbb{R}^d \\to \\mathbb{R}$. Used to construct the deterministic objective, and is referred to as the ``Stochastic Objective.''}\n\\nomenclature[A, 4]{$\\dot f$}{Gradient function of the stochastic objective with respect to the first argument, and maps $\\mathbb{R}^p \\times \\mathbb{R}^d \\to \\mathbb{R}^p$. Also referred to as the ``Stochastic Gradient.''}\n\\nomenclature[A, 5]{$\\theta$}{The parameter to be optimized in the objective function. Takes value in $\\mathbb{R}^p$.}\n\\nomenclature[A, 6]{$\\norm{\\cdot}_2$}{If the argument is a vector, this refers to the Euclidean norm of the vector. If the argument is a matrix, this refers to the Operator 2-norm.}\n\\nomenclature[A, 7]{$\\lambda_{\\max}$}{The largest eigenvalue corresponding to a given symmetric, positive definite matrix argument.}\n\\nomenclature[A, 8]{$\\lambda_{\\min}$}{The smallest eigenvalue corresponding to the given symmetric, positive definite matrix argument.}\n\\nomenclature[A, 9]{$\\kappa $}{The 2-norm condition number of the given matrix.}\n","sha256_utf8":"952e0c96152edb0e09210c82c84da2ea1da49a3712e3bd381672f872a089c1b5"},{"start":13812,"end":14184,"start_line":178,"end_line":180,"raw_tex":"\\nomenclature[B, 1]{$\\mathbb{P}$}{The probability operator with respect to the underlying Probability Space.}\n\\nomenclature[B, 2]{$\\mathbb{E}$}{The expectation operator corresponding to the underlying Probability Space.}\n\\nomenclature[B, 3]{$X, X_k$}{Independent, identically distributed random variables that map from the underlying Probability Space to $\\mathbb{R}^d$.}\n","sha256_utf8":"7daa64f25255994a43a29db9ab85fe2147205f8c45facd4d38221488d5748b39"},{"start":14498,"end":14627,"start_line":184,"end_line":185,"raw_tex":"\\nomenclature[B, 7]{$i.o.$}{Infinitely often.}\n\\nomenclature[B, 8]{$\\mathbf{1}$}{The indicator function over a specified event.}\n","sha256_utf8":"0da2161f1074d0940a05a428d49044990b7036dddac6e7949e3d2b4b92eb09a7"}]},{"id":"S041","semantic_type":"assumption","kind":"assumption","title":"梯度相关噪声模型","source_units":[{"start":41287,"end":41497,"start_line":397,"end_line":398,"raw_tex":"\\condition{condition-noise-grad}{There exists a $C_1,C_2 \\geq 0$ such that, $\\forall \\theta \\geq 0$,\n$$ \\E{ \\norm{ \\dot{f}(\\theta,X) - \\dot{F}(\\theta)}_2^2} \\leq C_1 + C_2 \\norm{ \\dot{F}(\\theta)}_2^2.$$}{NM-4}\n","sha256_utf8":"bb5c951a2d3dcfb0840c999c29951e9e966a7db3294c2572610c664bc0249fbe"}]},{"id":"S042","semantic_type":"assumption","kind":"assumption","title":"确定性梯度的连续性","source_units":[{"start":43330,"end":43525,"start_line":418,"end_line":419,"raw_tex":"\\condition{condition-dg-lipschitz}{There exista a $C > 0$ such that for all $\\theta_1,\\theta_2$, \n$$ \\norm{ \\dot{F}(\\theta_1) - \\dot{F}(\\theta_2)}_2 \\leq C \\norm{\\theta_1 - \\theta_2}_2.$$}{DG-2}\n","sha256_utf8":"7ace40d26979dabf314984d1cb9d1aa1d1a3aaf48f31b409d749c269940a974c"}]},{"id":"S043","semantic_type":"definition","kind":"definition","title":"BCN函数及四项条件","source_units":[{"start":43865,"end":44210,"start_line":425,"end_line":432,"raw_tex":"\\condition{bcn}{The BCN conditions are\n\n\\begin{enumerate}\n\\item $F$ is bounded from below.\n\\item $\\dot{F}$ is Lipschitz continuous (\\ref{condition-dg-lipschitz}).\n\\item For all $\\theta \\in \\mathbb{R}^p$, $\\dot{F}(\\theta) = \\mathbb{E}[ \\dot{f}(\\theta,X)]$.\n\\item The stochastic gradients satisfy \\ref{condition-noise-grad}.\n\\end{enumerate}}{BCN}\n","sha256_utf8":"b31e92963479a929712a7c4469fa52e8dfac2eda14bf96c576d8b0bb1dfedcf4"},{"start":44211,"end":44325,"start_line":434,"end_line":436,"raw_tex":"\\begin{definition}\nA function that satisfies the BCN conditions is referred to as a BCN function.\n\\end{definition}","sha256_utf8":"d88f293af82c7fd0313ad38d4cf9cbe52edf929def9b55a8e5544a83c71fb2b8"}]},{"id":"S044","semantic_type":"algorithm","kind":"algorithm","title":"矩阵步长随机梯度迭代","source_units":[{"start":45476,"end":46038,"start_line":446,"end_line":449,"raw_tex":"Let $\\lbrace X_k : k \\in \\mathbb{N} \\rbrace$ be independent and identically distributed random variables that have the same distribution as $X$. Let $\\beta_0$ be either a fixed or random quantity in $\\mathbb{R}^p$. Let $\\mathcal{F}_0 = \\sigma(\\beta_0)$ and $\\mathcal{F}_k = \\sigma(\\beta_0, X_1,\\ldots,X_k)$ denote the corresponding elements of the usual filtration. Define the Stochastic Gradient Descent iterates $\\lbrace \\beta_k : k \\in \\mathbb{N} \\rbrace$ recursively by\n\\begin{equation}\n\\beta_{k+1} = \\beta_k - M_k \\dot f (\\beta_k, X_{k+1}),\n\\end{equation} \n","sha256_utf8":"3706fc987cf0253cb52f4a8bb0ef1f708f1b278310f00ccef254fee5d589bef0"}]},{"id":"S045","semantic_type":"assumption","kind":"assumption","title":"矩阵步长的四项条件","source_units":[{"start":46038,"end":46768,"start_line":450,"end_line":460,"raw_tex":"where $\\lbrace M_k \\rbrace$ are matrices that satisfy:\n\\condition{prop-sympd}{The matrices $\\lbrace M_k \\rbrace$ are symmetric and positive definite.}{P1}\n\\condition{prop-maxeig}{There exists an $S > 0$ such that\n$$\\sum_{k=0}^\\infty \\lambda_{\\max}(M_k)^2 < S,$$\nwhere $\\lambda_{\\max}(\\cdot)$ denotes the largest eigenvalue of the given matrix.}{P2}\n\\condition{prop-mineig}{The sum,\n$$\\sum_{k=0}^\\infty \\lambda_{\\min}(M_k),$$\ndiverges, where $\\lambda_{\\min}(\\cdot)$ is the smallest eigenvalue of the given matrix.}{P3}\n\\condition{prop-condnum}{The condition numbers of $\\lbrace M_k \\rbrace$ with respect to operator 2-norm, $\\lbrace \\kappa(M_k) \\rbrace$, satisfy\n$$ \\lim_{k \\to \\infty} \\lambda_{\\max}(M_k) \\kappa(M_k) = 0.$$\n}{P4}\n","sha256_utf8":"ce1a599417aeba68f60c0352cbe2677579b4a19ff75ad459ab77c652efaa8d23"}]},{"id":"S046","semantic_type":"notation","kind":"notation","title":"几乎处处与初始条件约定","source_units":[{"start":51766,"end":52146,"start_line":490,"end_line":495,"raw_tex":"\\begin{remark}\nWe also note that all of the inequalities and equalities below hold with probability one, even if this is not explicitly stated.\n\\end{remark}\n\\begin{remark}\nWe also point out that the probabilities and expectations below should be conditional on $\\mathcal{F}_0$. However, to avoid the additional cumbersome notation, we will not explicitly state this.\n\\end{remark}\n","sha256_utf8":"640c6af40e0864dea5901e2a9fd3e85e518d8b474b5a1c1f28b0f967332187a6"}]},{"id":"S047","semantic_type":"assumption","kind":"assumption","title":"有限且足够晚的停止时刻","source_units":[{"start":52200,"end":52697,"start_line":499,"end_line":499,"raw_tex":"Our first task will be to set the stage to the analogue of strong Markov property that will be relevant in analyzing the SGD method. Let $\\tau$ be a finite stopping time with respect to $\\lbrace \\mathcal{F}_k \\rbrace$; that is, $\\Prb{ \\tau < \\infty} = 1$. Moreover, as all of our arguments will be asymptotic in this section, we will assume also that $\\Prb{ \\tau \\geq K} = 1$ where $K \\in \\mathbb{N}$ is defined in the following lemma to satisfy the following property, which will be used later. \n","sha256_utf8":"1e2aa4e984015ac2c6415187295f4c95de8bf7e4f78a2bc49305dc0c6a246d42"}]},{"id":"S048","semantic_type":"definition","kind":"definition","title":"重启样本矩阵与冻结迭代","source_units":[{"start":55446,"end":56541,"start_line":542,"end_line":553,"raw_tex":"Now, using $\\tau$, we will define a sequence of iterates $\\lbrace \\psi_k \\rbrace$ that we will eventually couple with $\\lbrace \\beta_k \\rbrace$. To define $\\lbrace \\psi_k \\rbrace$, let\n\\begin{enumerate}\n\\item $Z_k := X_{\\tau+1+k}$ for all $k \\in \\mathbb{N}$.\n\\item $\\psi_0 := \\beta_{\\tau + 1}$.\n\\item For all $k \\in \\lbrace 0 \\rbrace \\cup \\mathbb{N}$, $P_k := M_{\\tau+1+k}$ and\n\\begin{equation} \\label{eqn-psi-iter}\n\\psi_{k+1} := \\psi_k - P_k \\dot f( \\psi_k, Z_{k+1}) \\1{ \\norm{\\dot F(\\psi_k)}_2^2 > \\delta},\n\\end{equation}\nwhere $\\1{ \\cdot} $ is the indicator function for the given event.\n\\end{enumerate} \n\nThere are two properties of these quantities that are worth noting: $\\lbrace Z_k \\rbrace$ are independent and identically distributed; $\\lbrace P_k \\rbrace$ inherit the properties of $\\lbrace M_k \\rbrace$ with probability one. The former is verified by Theorem 4.1.3 of \\cite{durrett2010}, which states that $\\lbrace Z_k \\rbrace$ are mutually independent and independent of $\\mathcal{F}_{\\tau+1}$, and have the same distribution as $X_1$. The latter is verified by the following lemma.\n","sha256_utf8":"5f5b58a3c9dd15d1dd97d135bfeb0dfc358eee0051986ea415a35191974bb940"}]},{"id":"S049","semantic_type":"external_fact","kind":"external_fact","title":"停止后的独立样本性质","source_units":[],"declared_external":true,"declaration_text":["对独立同分布序列及其自然过滤，几乎处处有限停止时刻之后的未来样本保持独立同分布，并独立于停止时刻的信息。文中引用 Durrett Theorem 4.1.3。"]},{"id":"S050","semantic_type":"external_fact","kind":"external_fact","title":"条件Borel–Cantelli引理","source_units":[],"declared_external":true,"declaration_text":["若事件 A_k 属于 F_{k+1}，且条件概率之和 Σ_k P(A_k | F_k) 几乎处处有限，则 P(A_k 无限次发生)=0。文中引用 Durrett Theorem 5.3.2。"]},{"id":"S051","semantic_type":"external_fact","kind":"external_fact","title":"条件Markov不等式","source_units":[],"declared_external":true,"declaration_text":["对非负随机变量 Y、正数 a 和 σ-代数 H，有 P(Y≥a | H)≤E(Y | H)/a；可将此式应用于随机向量范数的平方。"]},{"id":"S052","semantic_type":"external_fact","kind":"external_fact","title":"对称矩阵的谱性质","source_units":[],"declared_external":true,"declaration_text":["实对称矩阵可正交对角化，其单位球面二次型由极端特征值控制；对称正定矩阵的算子二范数为最大特征值，二范数条件数为最大与最小特征值之比。"]},{"id":"S053","semantic_type":"external_fact","kind":"external_fact","title":"光滑函数的下降上界","source_units":[],"declared_external":true,"declaration_text":["由微积分基本定理，若梯度为 C-Lipschitz，则 F(y)≤F(x)+∇F(x)'(y−x)+(C/2)‖y−x‖²。"]},{"id":"S054","semantic_type":"external_fact","kind":"external_fact","title":"条件期望运算法则","source_units":[],"declared_external":true,"declaration_text":["条件期望满足塔式法则和可测因子提出法则；独立于条件 σ-代数的样本可用于计算给定可测参数下的条件均值与条件二阶矩。"]},{"id":"S055","semantic_type":"external_fact","kind":"external_fact","title":"基本概率集合运算","source_units":[],"declared_external":true,"declaration_text":["概率满足容斥公式与可数次可加性，且可数个概率一事件的交仍具有概率一。"]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S003","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S006","target_id":"S005","kind":"supports"},{"source_id":"S007","target_id":"S006","kind":"supports"},{"source_id":"S008","target_id":"S006","kind":"supports"},{"source_id":"S009","target_id":"S006","kind":"supports"},{"source_id":"S010","target_id":"S006","kind":"supports"},{"source_id":"S011","target_id":"S006","kind":"supports"},{"source_id":"S012","target_id":"S006","kind":"supports"},{"source_id":"S013","target_id":"S006","kind":"supports"},{"source_id":"S014","target_id":"S006","kind":"supports"},{"source_id":"S015","target_id":"S007","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S007","target_id":"S014","kind":"supports"},{"source_id":"S009","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S016","kind":"supports"},{"source_id":"S019","target_id":"S016","kind":"supports"},{"source_id":"S020","target_id":"S016","kind":"supports"},{"source_id":"S021","target_id":"S016","kind":"supports"},{"source_id":"S022","target_id":"S016","kind":"supports"},{"source_id":"S023","target_id":"S016","kind":"supports"},{"source_id":"S024","target_id":"S016","kind":"supports"},{"source_id":"S025","target_id":"S016","kind":"supports"},{"source_id":"S026","target_id":"S016","kind":"supports"},{"source_id":"S027","target_id":"S016","kind":"supports"},{"source_id":"S028","target_id":"S016","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S018","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S030","kind":"supports"},{"source_id":"S033","target_id":"S030","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S029","target_id":"S038","kind":"supports"},{"source_id":"S029","target_id":"S047","kind":"supports"},{"source_id":"S034","target_id":"S017","kind":"supports"},{"source_id":"S034","target_id":"S022","kind":"supports"},{"source_id":"S034","target_id":"S024","kind":"supports"},{"source_id":"S034","target_id":"S026","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S035","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S039","target_id":"S044","kind":"supports"},{"source_id":"S040","target_id":"S044","kind":"supports"},{"source_id":"S041","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S041","target_id":"S020","kind":"supports"},{"source_id":"S041","target_id":"S011","kind":"supports"},{"source_id":"S041","target_id":"S033","kind":"supports"},{"source_id":"S042","target_id":"S019","kind":"supports"},{"source_id":"S042","target_id":"S010","kind":"supports"},{"source_id":"S042","target_id":"S031","kind":"supports"},{"source_id":"S042","target_id":"S032","kind":"supports"},{"source_id":"S042","target_id":"S033","kind":"supports"},{"source_id":"S043","target_id":"S020","kind":"supports"},{"source_id":"S043","target_id":"S025","kind":"supports"},{"source_id":"S044","target_id":"S048","kind":"supports"},{"source_id":"S044","target_id":"S018","kind":"supports"},{"source_id":"S044","target_id":"S010","kind":"supports"},{"source_id":"S044","target_id":"S011","kind":"supports"},{"source_id":"S045","target_id":"S031","kind":"supports"},{"source_id":"S045","target_id":"S033","kind":"supports"},{"source_id":"S045","target_id":"S036","kind":"supports"},{"source_id":"S045","target_id":"S037","kind":"supports"},{"source_id":"S045","target_id":"S038","kind":"supports"},{"source_id":"S045","target_id":"S011","kind":"supports"},{"source_id":"S045","target_id":"S012","kind":"supports"},{"source_id":"S046","target_id":"S020","kind":"supports"},{"source_id":"S046","target_id":"S007","kind":"supports"},{"source_id":"S046","target_id":"S003","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S036","kind":"supports"},{"source_id":"S047","target_id":"S037","kind":"supports"},{"source_id":"S047","target_id":"S038","kind":"supports"},{"source_id":"S048","target_id":"S017","kind":"supports"},{"source_id":"S048","target_id":"S018","kind":"supports"},{"source_id":"S048","target_id":"S019","kind":"supports"},{"source_id":"S048","target_id":"S020","kind":"supports"},{"source_id":"S048","target_id":"S036","kind":"supports"},{"source_id":"S048","target_id":"S037","kind":"supports"},{"source_id":"S048","target_id":"S038","kind":"supports"},{"source_id":"S049","target_id":"S048","kind":"supports"},{"source_id":"S049","target_id":"S020","kind":"supports"},{"source_id":"S050","target_id":"S012","kind":"supports"},{"source_id":"S051","target_id":"S011","kind":"supports"},{"source_id":"S052","target_id":"S022","kind":"supports"},{"source_id":"S052","target_id":"S033","kind":"supports"},{"source_id":"S053","target_id":"S019","kind":"supports"},{"source_id":"S054","target_id":"S020","kind":"supports"},{"source_id":"S054","target_id":"S024","kind":"supports"},{"source_id":"S055","target_id":"S007","kind":"supports"},{"source_id":"S055","target_id":"S013","kind":"supports"},{"source_id":"S055","target_id":"S004","kind":"supports"}]}}
{"record_id":"2004.04357v2","paper":{"title":"Stochastic Variance-Reduced Prox-Linear Algorithms for Nonconvex Composite Optimization","authors":["Junyu Zhang","Lin Xiao"],"arxiv_id":"2004.04357","version":"v2","journal":"Mathematical Programming","doi":"10.1007/s10107-021-01709-z","abstract_url":"https://arxiv.org/abs/2004.04357v2","pdf_url":"https://arxiv.org/pdf/2004.04357v2"},"context":{"raw_tex":"\\begin{equation}\n\\label{eqn:composite-opt}\n\\minimize_{x\\in\\R^n} \\quad f(g(x)) + h(x), \n\\end{equation}\n\nwhere $f:\\R^m\\rightarrow\\R$ is a convex and possibly nonsmooth function, $g:\\R^n\\rightarrow\\R^m$ is a smooth mapping (vector-valued function), and $h:\\R^n\\rightarrow\\R$ is a convex and lower-semicontinuous function. \nAlthough both $f$ and $h$ are convex, the problem is in general nonconvex due to the composition of~$f$ and~$g$. \nIn addition, we assume that~$g$ is either the average of finite number of component mappings, i.e., $g(x) = \\frac{1}{N}\\sum_{i=1}^N g_i(x)$, \nor the expectation of a family of random component mappings, i.e.,\n$g(x) = \\E_\\xi [g_\\xi(x)]$ where~$\\xi$ is a random variable.\nMore explicitly, we consider the problems\n\\begin{equation}\n\\label{eqn:composite-finite}\n\\minimize_{x\\in\\R^n} \\quad f\\biggl(\\frac{1}{N}\\sum_{i=1}^N g_i(x)\\biggr) + h(x)\n\\end{equation}\n\n\\begin{equation}\\label{eqn:Phi-def}\n\\Phi(x) \\triangleq f(g(x)) + h(x),\n\\end{equation}\n\n\\begin{assumption}\\label{assumption:f-cvx} \n    The function $f:\\R^m\\to\\R\\cup\\{+\\infty\\}$ is convex and $\\ell_f$-Lipschitz continuous, i.e.,\n    \\[\n        |f(u)-f(v)|\\leq\\ell_f\\|u-v\\|, \\qquad \\forall\\, u, v\\in \\R^m.\n    \\]\nThe function $h:\\R^n\\to\\R\\cup\\{+\\infty\\}$ is convex and lower semi-continuous. \n\\end{assumption}\n\n\\begin{assumption}\\label{assumption:g-lip}\n    The vector mapping $g:\\R^n\\to\\R^m$ is $\\ell_g$-Lipschitz continuous and its Jacobian $g':\\R^n\\to\\R^{m\\times n}$ is $L_g$-Lipschitz continuous, i.e., \n    \\begin{align*}\n        \\|g(x) - g(y)\\| &\\leq \\ell_g \\|x-y\\|, \\\\\n        \\|g'(x) - g'(y)\\| &\\leq L_g \\|x-y\\|, \n    \\end{align*}\nfor all $x,y\\in\\dom h$, where $\\|\\cdot\\|$ for matrices denotes the spectral norm.\n\\end{assumption}\n\nA direct consequence of the Lipschitz condition on $g'$ in Assumption~\\ref{assumption:g-lip} is \n\\begin{equation}\\label{eqn:g-quadratic-bd}\n    \\bigl\\|g(x)-g(y)-g(y)'(x-y)\\bigr\\| \\leq \\frac{L_g}{2}\\|x-y\\|^2.\n\\end{equation}\n\n(See, e.g., \\cite[Theorem~3.2.12]{OrtegaRheinboldt1970}.) \n\n\n\\begin{assumption}\\label{assumption:phi-lowerbound}\n\tThere exists $\\Phi_*$ such that  $\\Phi_* = \\inf_x\\Phi(x)>-\\infty$.\n\\end{assumption}\n\n\\begin{algorithm2e}[t]\n    \\caption{Stochastic variance-reduced prox-linear algorithm} %(framework)}\n\t\\label{algo:SVR-PL} \n\t{\\bf input:} initial point $x^1_0$, $M>0$, number of outer and inner iterations $K$ and $\\tau$.\\\\\n\t\\For{$k = 1,\\ldots,K$}{ \n\t\t\\For{$i = 0,\\ldots,\\tau-1$}{\n            \\eIf{$i==0$}{\n                compute $\\tgk_0$ and $\\tJk_0$ using \\emph{large} batches $\\Bkz$ and $\\Skz$ respectively. \\label{alg-line:gk0-Jk0}\n            }{\n                compute $\\tgki$ and $\\tJki$ using \\emph{small} batches $\\Bki$ and $\\Ski$ respectively. \\label{alg-line:gki-Jki}\n            }\n            $\\xkie = \\displaystyle\\argmin_x ~\\textstyle\\Bigl\\{f\\bigl(\\tgki + \\tJki(x-\\xki)\\bigr) + h(x) + \\frac{M}{2}\\|x-\\xki\\|^2\\Bigr\\}$. \\label{alg-line:argmin}\n\t\t}\n\t\tSet $x_0^{k+1} = x^k_{\\tau}.$\n\t}  \n    {\\bf output:} choose $x^{k^*}_{i^*}$ from $\\{\\xki\\}_{i = 0,\\ldots,\\tau-1}^{k = 1,\\ldots,K}$ uniformly at random.\n\\end{algorithm2e}\n\nIn order to characterize the sample complexity of different algorithms, we first define what is an $\\epsilon$-stationary point.\nFor any $x\\in\\dom h$, we define the proximal point\n\\[\nx_+\\triangleq\n\\argmin_y \\left\\{ f\\bigl(g(x)+g'(x)(y-x))+h(y) + \\frac{M}{2}\\|y-x\\|^2\\right\\}\n\\]\n\nand the composite gradient mapping at~$x$,\n\\begin{equation}\n\\label{eqn:grad-mapping}\n\\cG_M(x)\\triangleq M(x-x_+).\n\\end{equation}\n\nGiven any $\\epsilon>0$, we call $\\bar{x}$ an $\\epsilon$-stationary point of~$\\Phi$ defined in~\\eqref{eqn:Phi-def} if $\\|\\cG_M(\\bar{x})\\|^2\\leq\\epsilon$.\n\n\nTo simplify notation, we will omit the subscript~$M$ (which is a constant throughout this paper) and denote the composite gradient mapping as $\\cG(x)$.\n\nThe sample complexity of a randomized algorithm, such as Algorithm~\\ref{algo:SVR-PL}, is the total number of evaluations of the component mappings $g_i$ or $g_\\xi$ and their Jacobians required in order to output some $\\bar{x}$ satisfying\n\\begin{equation}\n\\label{eqn:eps-solu}\n\\E\\bigl[\\|\\cG(\\bar{x})\\|^2\\bigr]\\leq \\epsilon,\n\\end{equation}\n\nwhere the expectation is taken over all the random samplings during the iterations of the algorithm.\n\nNotice that the proximal point $x_+$ used in the definition of $\\cG(x)$ is computed with~$g(x)$ and~$g'(x)$, which can be very costly if not impossible to evaluate. \nIn Algorithm~\\ref{algo:SVR-PL}, the proximal point $\\xkie$ is computed using the estimates $\\tgki$ and $\\tJki$, i.e.,\n\\begin{equation}\\label{eqn:xkie-def}\n    \\xkie = \\argmin_x \\left\\{f\\bigl(\\tgki + \\tJki(x-\\xki)\\bigr) + h(x) + \\frac{M}{2}\\|x-\\xki\\|^2\\right\\}.\n\\end{equation}\n\nThis leads to a convenient approximation, \n\\begin{equation}\\label{eqn:approx-gm}\n\\widetilde{\\cG}(\\xki)\\triangleq M(\\xki-\\xkie),\n\\end{equation}\n\nof the true gradient mapping $\\cGki= M(\\xki-\\hat{x}^k_{i+1})$, where \n\\begin{equation}\\label{eqn:x-hat-def}\n    \\hat{x}^k_{i+1} %= (\\xki)_+\n    = \\argmin_x \\left\\{f\\bigl(g(\\xki) + g'(\\xki)(x-\\xki)\\bigr) + h(x) + \\frac{M}{2}\\|x-\\xki\\|^2\\right\\}.\n\\end{equation}\n\nIn this section, we consider the composite finite-average problem~\\eqref{eqn:composite-finite} with nonsmooth~$f$ and smooth~$g_i$'s.\nIn particular, we replace Assumption~\\ref{assumption:g-lip} with the following more structured one, which implies Assumption~\\ref{assumption:g-lip}.\n\\begin{assumption}\n\t\\label{assumption:g-lip-finite}\n    For each $i = 1,\\ldots,N$, the mapping $g_i:\\R^n\\to\\R^m$, is $\\ell_{g,i}$-Lipschitz continuous and its Jacobian matrix $g'_i:\\R^n\\to\\R^{m\\times n}$ is $L_{g,i}$-Lipschitz continuous. Namely, \n\\begin{align*}\n\\|g_i(x) - g_i(x)\\| & \\leq\\ell_{g,i}\\|x-y\\|, \\\\\n\\|g'_i(x) - g'_i(x)\\| & \\leq L_{g,i}\\|x-y\\|,\n\\end{align*}\nfor all $x,y\\in\\dom h$ and $i = 1,\\ldots,N$. \n\\end{assumption}\n\nA direct consequence of this assumption is that $g$ is $\\big(\\frac{1}{N}\\sum_{i}^{N}\\ell_{g,i}\\big)$-Lipschitz continuous and $g'$ is $\\big(\\frac{1}{N}\\sum_{i}^{N}L_{g,i}\\big)$-Lipschitz continuous. Due to the root-mean square inequality $\\frac{z_1+...+z_N}{N}\\leq \\sqrt{\\frac{z_1^2+...+z_N^2}{N}}$, \nWe define \n\\begin{equation}\\label{eqn:ms-Lip-constants}\n\\ell_g = \\sqrt{\\frac{1}{N}\\sum_{i}^{N}\\ell_{g,i}^2}, \\qquad\nL_g = \\sqrt{\\frac{1}{N}\\sum_{i}^{N}L_{g,i}^2},\n\\end{equation}\n\nwhich can serve as the Lipschitz constants of $g$ and $g'$ respectively\nas in Assumption~\\ref{assumption:g-lip}.\n\n\nIn this case, we construct the estimates $\\tgkz$ and $\\tJkz$ using the full batch. In other words, we let $\\Bkz=\\Skz=\\{1,2,\\ldots,N\\}$ and replace Line~\\ref{alg-line:gk0-Jk0} in Algorithm~\\ref{algo:SVR-PL} with\n\\begin{eqnarray}\n\\tgkz &=& g(\\xkz) = \\frac{1}{N}\\sum_{i=1}^N g_i(\\xkz), \n    \\label{eqn:finite-gk0} \\\\\n    \\tJkz &=& g'(\\xkz) = \\frac{1}{N}\\sum_{i=1}^N g'_i(\\xkz).\n    \\label{eqn:finite-Jk0}\n\\end{eqnarray}\n\nFor $i>0$, we sample with replacement from $\\{1,2,\\ldots,N\\}$ to obtain smaller sets $\\Bki$ and $\\Ski$\n(whose cardinalities will be determined later), and\napply the following construction:\n\\begin{eqnarray}\n    \\tgki &=& \\frac{1}{|\\cB_i^k|}\\sum_{j\\in\\cB_i^k} \\Bigl(g_j(\\xki) - g_j(x_0^k) - g_j'(x_0^k)(\\xki-x_0^k)\\Bigr) + g(x_0^k) + g'(x_0^k)(\\xki-x_0^k),\n          \\label{eqn:g-svrg-correction} \\\\\n    \\tJki &=& \\frac{1}{|\\cS_i^k|}\\sum_{j\\in\\cS_i^k} \\Bigl(g'_j(\\xki) - g'_j(x_0^k)\\Bigr) + g'(x_0^k). \n\\label{eqn:J-svrg} \n\\end{eqnarray}\n\nRecall the definition that $\\tcGki: = M(\\xki-\\xkie)$. \nIn order to complete the convergence analysis, we define a stochastic Lyapunov function\n\\begin{equation}\\label{eqn:lyapunov-def}\nR_i^k = \\E\\left[\\Phi(\\xki) + c_i\\biggl\\|\\sum_{t=0}^{i-1}\\tcGkt\\biggr\\|^2\\right],\n\\quad k=1,\\ldots,K, \\quad i=0,\\ldots,\\tau,\n\\end{equation}\n\nwhere the coefficients $c_i$ for $i=0,1,\\ldots,\\tau$ are obtained through the recursion:\n\\begin{eqnarray}\nc_{\\tau} &=& 0, \\nonumber \\\\\nc_i &=& c_{i+1}\\left(1+\\frac{1}{\\tau}\\right) + \\frac{1}{3M\\sqrt{|\\Bki|}} + \\frac{1}{5M|\\Ski|}, \\quad i=\\tau-1,\\ldots,0.\n\\label{eqn:ct-finite-nonsmooth}\n\\end{eqnarray}\n\n(Our choices or $|\\Bki|$ and $|\\Ski|$ will not depend on~$k$.)\nIn addition, we define the following constant\n\\[\n   \\gamma \\triangleq \\min_{0\\leq i\\leq \\tau-1} \\frac{1}{3}\\left(\\frac{1}{4M} - c_{i+1}(1+\\tau)\\right).\n\\]\n\nWe use $\\lceil\\cdot\\rceil$ to denote the nearest integer from above.\n","units":[{"start":4649,"end":4750,"start_line":120,"end_line":123,"raw_tex":"\\begin{equation}\n\\label{eqn:composite-opt}\n\\minimize_{x\\in\\R^n} \\quad f(g(x)) + h(x), \n\\end{equation}","sha256_utf8":"42005a3c99e4419462c4e6133b20abf0613240db98f5b2fa9c9d27deb3f77fc8"},{"start":4751,"end":5534,"start_line":124,"end_line":133,"raw_tex":"where $f:\\R^m\\rightarrow\\R$ is a convex and possibly nonsmooth function, $g:\\R^n\\rightarrow\\R^m$ is a smooth mapping (vector-valued function), and $h:\\R^n\\rightarrow\\R$ is a convex and lower-semicontinuous function. \nAlthough both $f$ and $h$ are convex, the problem is in general nonconvex due to the composition of~$f$ and~$g$. \nIn addition, we assume that~$g$ is either the average of finite number of component mappings, i.e., $g(x) = \\frac{1}{N}\\sum_{i=1}^N g_i(x)$, \nor the expectation of a family of random component mappings, i.e.,\n$g(x) = \\E_\\xi [g_\\xi(x)]$ where~$\\xi$ is a random variable.\nMore explicitly, we consider the problems\n\\begin{equation}\n\\label{eqn:composite-finite}\n\\minimize_{x\\in\\R^n} \\quad f\\biggl(\\frac{1}{N}\\sum_{i=1}^N g_i(x)\\biggr) + h(x)\n\\end{equation}","sha256_utf8":"f68b01a076e28fa37948ffde1c4cf7da721b5f9ff3dea64df31d6c82f1341904"},{"start":25642,"end":25727,"start_line":345,"end_line":347,"raw_tex":"\\begin{equation}\\label{eqn:Phi-def}\n\\Phi(x) \\triangleq f(g(x)) + h(x),\n\\end{equation}","sha256_utf8":"96316e676f566396a1b55cfa98c6417e77b92d230f45c4c0e7f9a1d734ed5ca3"},{"start":25996,"end":26316,"start_line":351,"end_line":357,"raw_tex":"\\begin{assumption}\\label{assumption:f-cvx} \n    The function $f:\\R^m\\to\\R\\cup\\{+\\infty\\}$ is convex and $\\ell_f$-Lipschitz continuous, i.e.,\n    \\[\n        |f(u)-f(v)|\\leq\\ell_f\\|u-v\\|, \\qquad \\forall\\, u, v\\in \\R^m.\n    \\]\nThe function $h:\\R^n\\to\\R\\cup\\{+\\infty\\}$ is convex and lower semi-continuous. \n\\end{assumption}","sha256_utf8":"2717eac3588454fc3854c69ee35e9a3b61654bfd501510b0590f1232234c016a"},{"start":26318,"end":26744,"start_line":358,"end_line":365,"raw_tex":"\\begin{assumption}\\label{assumption:g-lip}\n    The vector mapping $g:\\R^n\\to\\R^m$ is $\\ell_g$-Lipschitz continuous and its Jacobian $g':\\R^n\\to\\R^{m\\times n}$ is $L_g$-Lipschitz continuous, i.e., \n    \\begin{align*}\n        \\|g(x) - g(y)\\| &\\leq \\ell_g \\|x-y\\|, \\\\\n        \\|g'(x) - g'(y)\\| &\\leq L_g \\|x-y\\|, \n    \\end{align*}\nfor all $x,y\\in\\dom h$, where $\\|\\cdot\\|$ for matrices denotes the spectral norm.\n\\end{assumption}","sha256_utf8":"e09057ca41e932c40bf3ed83b262bcaca25a2b65ab31807865ba5dd3daca8d34"},{"start":26745,"end":26967,"start_line":366,"end_line":369,"raw_tex":"A direct consequence of the Lipschitz condition on $g'$ in Assumption~\\ref{assumption:g-lip} is \n\\begin{equation}\\label{eqn:g-quadratic-bd}\n    \\bigl\\|g(x)-g(y)-g(y)'(x-y)\\bigr\\| \\leq \\frac{L_g}{2}\\|x-y\\|^2.\n\\end{equation}","sha256_utf8":"a6a6f7fdf7dfb9e1f2c09bd92491b8b2ba099bdc6c10c033876ea85e9e75fbac"},{"start":26968,"end":27027,"start_line":370,"end_line":370,"raw_tex":"(See, e.g., \\cite[Theorem~3.2.12]{OrtegaRheinboldt1970}.) \n","sha256_utf8":"16bcaada096e87eea80772610f12443737cda83d2da685e88a604883873331a0"},{"start":27151,"end":27287,"start_line":372,"end_line":374,"raw_tex":"\\begin{assumption}\\label{assumption:phi-lowerbound}\n\tThere exists $\\Phi_*$ such that  $\\Phi_* = \\inf_x\\Phi(x)>-\\infty$.\n\\end{assumption}","sha256_utf8":"266930316ba3a8f3a29d6750ca4f334f7bffe08732cbe48a7a14704a0905bf3f"},{"start":29252,"end":30182,"start_line":403,"end_line":419,"raw_tex":"\\begin{algorithm2e}[t]\n    \\caption{Stochastic variance-reduced prox-linear algorithm} %(framework)}\n\t\\label{algo:SVR-PL} \n\t{\\bf input:} initial point $x^1_0$, $M>0$, number of outer and inner iterations $K$ and $\\tau$.\\\\\n\t\\For{$k = 1,\\ldots,K$}{ \n\t\t\\For{$i = 0,\\ldots,\\tau-1$}{\n            \\eIf{$i==0$}{\n                compute $\\tgk_0$ and $\\tJk_0$ using \\emph{large} batches $\\Bkz$ and $\\Skz$ respectively. \\label{alg-line:gk0-Jk0}\n            }{\n                compute $\\tgki$ and $\\tJki$ using \\emph{small} batches $\\Bki$ and $\\Ski$ respectively. \\label{alg-line:gki-Jki}\n            }\n            $\\xkie = \\displaystyle\\argmin_x ~\\textstyle\\Bigl\\{f\\bigl(\\tgki + \\tJki(x-\\xki)\\bigr) + h(x) + \\frac{M}{2}\\|x-\\xki\\|^2\\Bigr\\}$. \\label{alg-line:argmin}\n\t\t}\n\t\tSet $x_0^{k+1} = x^k_{\\tau}.$\n\t}  \n    {\\bf output:} choose $x^{k^*}_{i^*}$ from $\\{\\xki\\}_{i = 0,\\ldots,\\tau-1}^{k = 1,\\ldots,K}$ uniformly at random.\n\\end{algorithm2e}","sha256_utf8":"3457498daccf2115e13eff2616988e82e479b1b40f0b810840d9aa5a72508ead"},{"start":31117,"end":31393,"start_line":429,"end_line":434,"raw_tex":"In order to characterize the sample complexity of different algorithms, we first define what is an $\\epsilon$-stationary point.\nFor any $x\\in\\dom h$, we define the proximal point\n\\[\nx_+\\triangleq\n\\argmin_y \\left\\{ f\\bigl(g(x)+g'(x)(y-x))+h(y) + \\frac{M}{2}\\|y-x\\|^2\\right\\}\n\\]","sha256_utf8":"fb94b814fd1879148d9827fe296e92fb0b24f2aeb594f284f47f8336b40590bd"},{"start":31394,"end":31522,"start_line":435,"end_line":439,"raw_tex":"and the composite gradient mapping at~$x$,\n\\begin{equation}\n\\label{eqn:grad-mapping}\n\\cG_M(x)\\triangleq M(x-x_+).\n\\end{equation}","sha256_utf8":"31a24091476a417eaf237066f728518198185e11576461f24a3c43eebb08588f"},{"start":31523,"end":31676,"start_line":440,"end_line":440,"raw_tex":"Given any $\\epsilon>0$, we call $\\bar{x}$ an $\\epsilon$-stationary point of~$\\Phi$ defined in~\\eqref{eqn:Phi-def} if $\\|\\cG_M(\\bar{x})\\|^2\\leq\\epsilon$.\n","sha256_utf8":"e36eb590fa6b906b6e88fdc1ef094473f305514f8845204302252ce4fa7e7749"},{"start":32098,"end":32588,"start_line":443,"end_line":449,"raw_tex":"To simplify notation, we will omit the subscript~$M$ (which is a constant throughout this paper) and denote the composite gradient mapping as $\\cG(x)$.\n\nThe sample complexity of a randomized algorithm, such as Algorithm~\\ref{algo:SVR-PL}, is the total number of evaluations of the component mappings $g_i$ or $g_\\xi$ and their Jacobians required in order to output some $\\bar{x}$ satisfying\n\\begin{equation}\n\\label{eqn:eps-solu}\n\\E\\bigl[\\|\\cG(\\bar{x})\\|^2\\bigr]\\leq \\epsilon,\n\\end{equation}","sha256_utf8":"287773655228a64b9ca207b5d26cd8a58009d9baa451b5ebf539e404ff74a796"},{"start":32589,"end":33132,"start_line":450,"end_line":456,"raw_tex":"where the expectation is taken over all the random samplings during the iterations of the algorithm.\n\nNotice that the proximal point $x_+$ used in the definition of $\\cG(x)$ is computed with~$g(x)$ and~$g'(x)$, which can be very costly if not impossible to evaluate. \nIn Algorithm~\\ref{algo:SVR-PL}, the proximal point $\\xkie$ is computed using the estimates $\\tgki$ and $\\tJki$, i.e.,\n\\begin{equation}\\label{eqn:xkie-def}\n    \\xkie = \\argmin_x \\left\\{f\\bigl(\\tgki + \\tJki(x-\\xki)\\bigr) + h(x) + \\frac{M}{2}\\|x-\\xki\\|^2\\right\\}.\n\\end{equation}","sha256_utf8":"bad7f9cf3504db18e35e74ba17106624d6c7066f8e27d91bcefd40438749eba7"},{"start":33133,"end":33275,"start_line":457,"end_line":460,"raw_tex":"This leads to a convenient approximation, \n\\begin{equation}\\label{eqn:approx-gm}\n\\widetilde{\\cG}(\\xki)\\triangleq M(\\xki-\\xkie),\n\\end{equation}","sha256_utf8":"74cf06153a1f8a7f1fc08dc54d4b6777bdbc43b18c5f28d4e17d0c07553139a4"},{"start":33276,"end":33535,"start_line":461,"end_line":465,"raw_tex":"of the true gradient mapping $\\cGki= M(\\xki-\\hat{x}^k_{i+1})$, where \n\\begin{equation}\\label{eqn:x-hat-def}\n    \\hat{x}^k_{i+1} %= (\\xki)_+\n    = \\argmin_x \\left\\{f\\bigl(g(\\xki) + g'(\\xki)(x-\\xki)\\bigr) + h(x) + \\frac{M}{2}\\|x-\\xki\\|^2\\right\\}.\n\\end{equation}","sha256_utf8":"ff3349cd64fc4f75f1bd4f58429117f4b4431408255699fe562e7807747708d5"},{"start":38583,"end":39297,"start_line":557,"end_line":567,"raw_tex":"In this section, we consider the composite finite-average problem~\\eqref{eqn:composite-finite} with nonsmooth~$f$ and smooth~$g_i$'s.\nIn particular, we replace Assumption~\\ref{assumption:g-lip} with the following more structured one, which implies Assumption~\\ref{assumption:g-lip}.\n\\begin{assumption}\n\t\\label{assumption:g-lip-finite}\n    For each $i = 1,\\ldots,N$, the mapping $g_i:\\R^n\\to\\R^m$, is $\\ell_{g,i}$-Lipschitz continuous and its Jacobian matrix $g'_i:\\R^n\\to\\R^{m\\times n}$ is $L_{g,i}$-Lipschitz continuous. Namely, \n\\begin{align*}\n\\|g_i(x) - g_i(x)\\| & \\leq\\ell_{g,i}\\|x-y\\|, \\\\\n\\|g'_i(x) - g'_i(x)\\| & \\leq L_{g,i}\\|x-y\\|,\n\\end{align*}\nfor all $x,y\\in\\dom h$ and $i = 1,\\ldots,N$. \n\\end{assumption}","sha256_utf8":"df2c8485978161c52ada1135fc83c09779bcb1259f2e5279825b3cc02e7b39bd"},{"start":39298,"end":39776,"start_line":568,"end_line":573,"raw_tex":"A direct consequence of this assumption is that $g$ is $\\big(\\frac{1}{N}\\sum_{i}^{N}\\ell_{g,i}\\big)$-Lipschitz continuous and $g'$ is $\\big(\\frac{1}{N}\\sum_{i}^{N}L_{g,i}\\big)$-Lipschitz continuous. Due to the root-mean square inequality $\\frac{z_1+...+z_N}{N}\\leq \\sqrt{\\frac{z_1^2+...+z_N^2}{N}}$, \nWe define \n\\begin{equation}\\label{eqn:ms-Lip-constants}\n\\ell_g = \\sqrt{\\frac{1}{N}\\sum_{i}^{N}\\ell_{g,i}^2}, \\qquad\nL_g = \\sqrt{\\frac{1}{N}\\sum_{i}^{N}L_{g,i}^2},\n\\end{equation}","sha256_utf8":"bbc7b328c13b8d54cdbd099ed6690c7e9de04b77958af70b67389d75f58a4d38"},{"start":39777,"end":39890,"start_line":574,"end_line":575,"raw_tex":"which can serve as the Lipschitz constants of $g$ and $g'$ respectively\nas in Assumption~\\ref{assumption:g-lip}.\n","sha256_utf8":"cf08840fb6c5b782e98582727fe490dcd245408776ff22ab31f8a8e7a3d048cc"},{"start":39892,"end":40308,"start_line":578,"end_line":584,"raw_tex":"In this case, we construct the estimates $\\tgkz$ and $\\tJkz$ using the full batch. In other words, we let $\\Bkz=\\Skz=\\{1,2,\\ldots,N\\}$ and replace Line~\\ref{alg-line:gk0-Jk0} in Algorithm~\\ref{algo:SVR-PL} with\n\\begin{eqnarray}\n\\tgkz &=& g(\\xkz) = \\frac{1}{N}\\sum_{i=1}^N g_i(\\xkz), \n    \\label{eqn:finite-gk0} \\\\\n    \\tJkz &=& g'(\\xkz) = \\frac{1}{N}\\sum_{i=1}^N g'_i(\\xkz).\n    \\label{eqn:finite-Jk0}\n\\end{eqnarray}","sha256_utf8":"5ec6f13b6fc713ccdd738dbe1dead01d926b5aad42b7fd8be67c16a95ea13616"},{"start":40309,"end":40844,"start_line":585,"end_line":593,"raw_tex":"For $i>0$, we sample with replacement from $\\{1,2,\\ldots,N\\}$ to obtain smaller sets $\\Bki$ and $\\Ski$\n(whose cardinalities will be determined later), and\napply the following construction:\n\\begin{eqnarray}\n    \\tgki &=& \\frac{1}{|\\cB_i^k|}\\sum_{j\\in\\cB_i^k} \\Bigl(g_j(\\xki) - g_j(x_0^k) - g_j'(x_0^k)(\\xki-x_0^k)\\Bigr) + g(x_0^k) + g'(x_0^k)(\\xki-x_0^k),\n          \\label{eqn:g-svrg-correction} \\\\\n    \\tJki &=& \\frac{1}{|\\cS_i^k|}\\sum_{j\\in\\cS_i^k} \\Bigl(g'_j(\\xki) - g'_j(x_0^k)\\Bigr) + g'(x_0^k). \n\\label{eqn:J-svrg} \n\\end{eqnarray}","sha256_utf8":"605bc2ba74bd07b84db21a0a7bc2ae6e34fcef15899d366c5064a2b41722aacd"},{"start":47614,"end":47936,"start_line":700,"end_line":705,"raw_tex":"Recall the definition that $\\tcGki: = M(\\xki-\\xkie)$. \nIn order to complete the convergence analysis, we define a stochastic Lyapunov function\n\\begin{equation}\\label{eqn:lyapunov-def}\nR_i^k = \\E\\left[\\Phi(\\xki) + c_i\\biggl\\|\\sum_{t=0}^{i-1}\\tcGkt\\biggr\\|^2\\right],\n\\quad k=1,\\ldots,K, \\quad i=0,\\ldots,\\tau,\n\\end{equation}","sha256_utf8":"ae265862a2bc955305c72d62240f503b11f1244f30439c6f0151351a269e4f3f"},{"start":47937,"end":48238,"start_line":706,"end_line":711,"raw_tex":"where the coefficients $c_i$ for $i=0,1,\\ldots,\\tau$ are obtained through the recursion:\n\\begin{eqnarray}\nc_{\\tau} &=& 0, \\nonumber \\\\\nc_i &=& c_{i+1}\\left(1+\\frac{1}{\\tau}\\right) + \\frac{1}{3M\\sqrt{|\\Bki|}} + \\frac{1}{5M|\\Ski|}, \\quad i=\\tau-1,\\ldots,0.\n\\label{eqn:ct-finite-nonsmooth}\n\\end{eqnarray}","sha256_utf8":"32f287cb6aa8ea59180524b52739157584ea5f04390dc5ab3aed5bdf8e25e0ce"},{"start":48239,"end":48456,"start_line":712,"end_line":716,"raw_tex":"(Our choices or $|\\Bki|$ and $|\\Ski|$ will not depend on~$k$.)\nIn addition, we define the following constant\n\\[\n   \\gamma \\triangleq \\min_{0\\leq i\\leq \\tau-1} \\frac{1}{3}\\left(\\frac{1}{4M} - c_{i+1}(1+\\tau)\\right).\n\\]","sha256_utf8":"038e9224f8c1f176d6990b96f9a633cd9a779619d3c1cf397047175ee61d7c57"},{"start":53561,"end":53630,"start_line":802,"end_line":802,"raw_tex":"We use $\\lceil\\cdot\\rceil$ to denote the nearest integer from above.\n","sha256_utf8":"0c1f09076152242b4527a265df234fedbc642f2959b5efb4bb8e26beb19c01ac"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\n\t\\label{theorem:svrg}\n    Suppose Assumptions~\\ref{assumption:f-cvx}, \\ref{assumption:phi-lowerbound}  and~\\ref{assumption:g-lip-finite} hold for problem~\\eqref{eqn:composite-finite}. Let the estimates $\\tgkz$, $\\tJkz$, $\\tgki$ and $\\tJki$ in Algorithm~\\ref{algo:SVR-PL} be given in~\\eqref{eqn:finite-gk0}-\\eqref{eqn:J-svrg} respectively. If we choose $M\\geq 4\\ell_f L_g$ and $\\tau = \\bigl\\lceil\\frac{1}{2}N^{1/5}-1\\bigr\\rceil$, and\n\\[\n    |\\cB_i^k| = \\bigr\\lceil 4N^{4/5}\\bigr\\rceil, \\qquad\n    |\\cS_i^k| = \\bigr\\lceil N^{2/5}\\bigr\\rceil, \\qquad \n    i=1,\\ldots,\\tau-1,  \\qquad k=1,\\ldots,K,\n\\]\nthen the output of Algorithm~\\ref{algo:SVR-PL} satisfies\n\\begin{equation}\\label{eqn:cvg-svr-pl-nonsmooth-1}\n    \\E\\bigl[\\|\\cG(x_{i^*}^{k^*})\\|^2\\bigr] ~\\leq~ \\frac{15M\\bigl(\\Phi(x_0^1) - \\Phi_*\\bigr)}{K\\tau}.\n\\end{equation}\nTo get an $\\epsilon$-stationary point in expectation, the total sample complexity for the component mappings $g_j$ and their Jacobians are both $\\cO(N+N^{4/5}\\epsilon^{-1})$.\n\\end{theorem}","units":[{"start":55357,"end":56381,"start_line":838,"end_line":851,"raw_tex":"\\begin{theorem}\n\t\\label{theorem:svrg}\n    Suppose Assumptions~\\ref{assumption:f-cvx}, \\ref{assumption:phi-lowerbound}  and~\\ref{assumption:g-lip-finite} hold for problem~\\eqref{eqn:composite-finite}. Let the estimates $\\tgkz$, $\\tJkz$, $\\tgki$ and $\\tJki$ in Algorithm~\\ref{algo:SVR-PL} be given in~\\eqref{eqn:finite-gk0}-\\eqref{eqn:J-svrg} respectively. If we choose $M\\geq 4\\ell_f L_g$ and $\\tau = \\bigl\\lceil\\frac{1}{2}N^{1/5}-1\\bigr\\rceil$, and\n\\[\n    |\\cB_i^k| = \\bigr\\lceil 4N^{4/5}\\bigr\\rceil, \\qquad\n    |\\cS_i^k| = \\bigr\\lceil N^{2/5}\\bigr\\rceil, \\qquad \n    i=1,\\ldots,\\tau-1,  \\qquad k=1,\\ldots,K,\n\\]\nthen the output of Algorithm~\\ref{algo:SVR-PL} satisfies\n\\begin{equation}\\label{eqn:cvg-svr-pl-nonsmooth-1}\n    \\E\\bigl[\\|\\cG(x_{i^*}^{k^*})\\|^2\\bigr] ~\\leq~ \\frac{15M\\bigl(\\Phi(x_0^1) - \\Phi_*\\bigr)}{K\\tau}.\n\\end{equation}\nTo get an $\\epsilon$-stationary point in expectation, the total sample complexity for the component mappings $g_j$ and their Jacobians are both $\\cO(N+N^{4/5}\\epsilon^{-1})$.\n\\end{theorem}","sha256_utf8":"11b3060c20099f1e4f46a105b48fdbb826ca8842f202ccc267fcc511ca0e8344"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\\label{lemma:upper-bound} \nSuppose Assumption \\ref{assumption:f-cvx} and \\ref{assumption:g-lip} hold, then \nfor any $x,y\\in\\dom h$,\n\\begin{equation}\n    f(g(x)) \\leq f(g(y) + g'(y)(x-y)) + \\frac{\\ell_f L_g}{2}\\|x-y\\|^2.\n\\end{equation}\n\\end{lemma}\n\n\\begin{proof} \n    By the Lipschitz continuity of $f$ and $g'$, we have\n\t\\begin{eqnarray*}\n    f(g(x)) \n    &=&  f\\bigl(g(y) + g'(y)(x-y)\\bigr) + f(g(x)) - f\\bigl(g(y) + g'(y)(x-y)\\bigr)\\\\\n    & \\leq &  f\\bigl(g(y) + g'(y)(x-y)\\bigr) + \\bigl|f(g(x)) - f\\bigl(g(y) + g'(y)(x-y)\\bigr)\\bigr|\\\\\n\t&\\leq &f\\bigl(g(y) + g'(y)(x-y)\\bigr) + \\ell_f\\bigl\\|g(x) - g(y) - g'(y)(x-y)\\bigr\\|\\\\\n\t& \\leq & f\\bigl(g(y) + g'(y)(x-y)\\bigr) + \\frac{\\ell_f L_g}{2}\\|x-y\\|^2,\n\t\\end{eqnarray*}\n    where the last inequality is due to~\\eqref{eqn:g-quadratic-bd}.\n\\end{proof}\n\n\\begin{lemma}\n\\label{lemma:PG-finite-sum-nonsmooth}\nUnder Assumptions \\ref{assumption:f-cvx} and \\ref{assumption:g-lip}, \nthe iterates generated by Algorithm~\\ref{algo:SVR-PL} satisfy\n\\begin{eqnarray*}\n    \\frac{M - \\ell_f L_g}{M^2}\\bigl\\|\\cG(\\xki)\\bigr\\|^2 \n    &\\leq& \\frac{2M+\\ell_f L_g}{M^2}\\bigl\\|\\widetilde{\\cG}(\\xki)\\bigr\\|^2  \\\\\n    && \\,+\\, 4\\ell_f\\bigl\\|\\tgki-g(\\xki)\\bigr\\| + \\frac{2\\ell_f}{L_g}\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|^2.\n\\end{eqnarray*}\n\\end{lemma}\n\n\\begin{proof}\nFor the ease of notation,  we denote \n\\begin{eqnarray}\n    F(x;\\xki) &=& f\\bigl(g(\\xki) + g'(\\xki)(x-\\xki)\\bigr), \\label{eqn:F-def}\\\\\n    \\widetilde{F}(x;\\xki) &=& f\\bigl(\\tgki + \\tJki(x-\\xki)\\bigr).\n    \\label{eqn:tF-def}\n\\end{eqnarray}\nSince both~$f$ and~$h$ are convex (Assumption~\\ref{assumption:f-cvx}),\nthe following two functions are $M$-strongly convex:\n\\begin{eqnarray}\n    && F(x;\\xki) + h(x) + \\frac{M}{2}\\|x-\\xki\\|^2, \\label{eqn:F+h+M} \\\\\n    && \\widetilde{F}(x;\\xki) + h(x) + \\frac{M}{2}\\|x-\\xki\\|^2.\n    \\label{eqn:tF+h+M} \n\\end{eqnarray} \nAccording to~\\eqref{eqn:x-hat-def} and~\\eqref{eqn:xkie-def}, $\\hat{x}_{i+1}^k$ and $\\xkie$ are the minimizers of these two functions respectively. \nTherefore\n\\begin{eqnarray*}\n    F(\\hat{x}^k_{i+1};\\xki) + h(\\hat{x}^k_{i+1}) + \\frac{M}{2}\\|\\hat{x}^k_{i+1}-\\xki\\|^2 \n\t& \\leq & F(\\xkie;\\xki) + h(\\xkie) + \\frac{M}{2}\\|\\xkie-\\xki\\|^2  \\\\\n    &&  - \\frac{M}{2}\\|\\hat{x}^k_{i+1}-\\xkie\\|^2,  \n\\end{eqnarray*} \nand\n\\begin{eqnarray*}\n    \\widetilde{F}(x^k_{i+1};\\xki) + h(x^k_{i+1}) + \\frac{M}{2}\\|x^k_{i+1}-\\xki\\|^2 \n& \\leq & \\widetilde{F}(\\hat{x}^k_{i+1};\\xki) + h(\\hat{x}^k_{i+1}) + \\frac{M}{2}\\|\\hat{x}^k_{i+1}-\\xki\\|^2 \\\\\n&& - \\frac{M}{2}\\|\\hat{x}^k_{i+1}-\\xkie\\|^2. \n\\end{eqnarray*} \nSumming the two inequalities above and rearranging the terms, we obtain\n\\begin{eqnarray}\n\t\\label{lm:PG-finite-sum-3}\n\tM\\|\\hat{x}^k_{i+1}-\\xkie\\|^2 \n    &\\leq& F(\\xkie;\\xki)-\\widetilde{F}(\\xkie;\\xki) \n     + \\widetilde{F}(\\hat{x}^k_{i+1};\\xki)-F(\\hat{x}^k_{i+1};\\xki).\n\\end{eqnarray}\nUsing the Lipschitz property of~$f$, we have \n\\begin{eqnarray*}\n\\bigl| F(\\xkie;\\xki)-\\widetilde{F}(\\xkie;\\xki)\\bigr| \n    &=& \\left| f\\bigl(g(\\xki)+g'(\\xki)(\\xkie-\\xki)\\bigr) - f\\bigl(\\tgki+\\tJki(\\xkie-\\xki)\\bigr)\\right| \\\\\n    & \\leq & \\ell_f\\left\\|\\bigl(g(\\xki) - \\tgki\\bigr) + \\bigl(g'(\\xki) - \\tJki\\bigr)\\bigl(\\xkie - \\xki\\bigr)\\right\\|\\\\\n\t& \\leq & \\ell_f\\left(\\bigl\\|\\tgki - g(\\xki)\\bigr\\| + \\bigl\\|\\tJki - g'(\\xki)\\bigr\\|\\bigl\\|\\xkie - \\xki\\bigr\\|\\right)\\\\\n    & \\leq & \\ell_f\\left(\\bigl\\|\\tgki - g(\\xki)\\bigr\\| + \\frac{1}{2L_g}\\bigl\\|\\tJki - g'(\\xki)\\bigr\\|^2 + \\frac{L_g}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2 \\right).\n\\end{eqnarray*}\nReplacing $\\xkie$ in the above inequality with $\\hxkie$, we get  \n\\begin{eqnarray*}\n\\bigl|F(\\hxkie;\\xki)-\\widetilde{F}(\\hxkie;\\xki)\\bigr| \n\t& \\leq & \\ell_f \\left(\\bigl\\|\\tgki - g(\\xki)\\bigr\\| + \\frac{1}{2L_g}\\bigl\\|\\tJki - g'(\\xki)\\bigr\\|^2 + \\frac{L_g}{2}\\bigl\\|\\hxkie-\\xki\\bigr\\|^2\\right).\n\\end{eqnarray*}\nCombining the two bounds above with~\\eqref{lm:PG-finite-sum-3} gives\n\\begin{eqnarray*}\n\tM\\bigl\\|\\hxkie-\\xkie\\bigr\\|^2 \n    & \\leq & 2\\ell_f\\bigl\\|\\tgki - g(\\xki)\\bigr\\| + \\frac{\\ell_f}{L_g}\\bigl\\|\\tJki - g'(\\xki)\\bigr\\|^2 \\\\\n    && +\\, \\frac{\\ell_f L_g}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2+\\frac{\\ell_f L_g}{2}\\bigl\\|\\hxkie-\\xki\\bigr\\|^2 .\n\\end{eqnarray*}\nNext, using the fact that $\\|a+b\\|^2\\leq 2\\|a\\|^2+2\\|b\\|^2$ and the above inequality, we have\n\\begin{eqnarray*} \n\tM\\bigl\\|\\hxkie-\\xki\\bigr\\|^2] \n    & \\leq & 2M\\bigl\\|\\xkie-\\xki\\bigr\\|^2 + 2M\\bigl\\|\\hxkie -\\xkie\\bigr\\|^2 \\\\\n\t& \\leq & 2M\\bigl\\|\\xkie-\\xki\\bigr\\|^2 + 4\\ell_f\\bigl\\|\\tgki - g(\\xki)\\bigr\\| + \\frac{2\\ell_f}{L_g}\\bigl\\|\\tJki - g'(\\xki)\\bigr\\|^2 \\\\\n\t& & + \\ell_f L_g\\bigl\\|\\xkie-\\xki\\bigr\\|^2 + \\ell_f L_g\\bigl\\|\\hxkie-\\xki\\bigr\\|^2.\n\\end{eqnarray*}\nRearranging the terms yields \n\\begin{eqnarray*}\n    (M - \\ell_f L_g)\\bigl\\|\\hxkie-\\xki\\bigr\\|^2 \n    &\\leq& (2M+\\ell_f L_g)\\bigl\\|\\xkie-\\xki\\bigr\\|^2  \\\\\n    && +\\, 4\\ell_f\\bigl\\|\\tgki-g(\\xki)\\bigr\\| \n     \\,+\\, \\frac{2\\ell_f}{L_g}\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|^2.\n\\end{eqnarray*}\nFinally, using the definitions $\\cG(\\xki)=M(\\xki-\\hxkie)$ and\n$\\widetilde{\\cG}(\\xki)=M(\\xki-\\xkie)$, we obtain the desired result.\n\\end{proof}\n\n\\begin{lemma}\n\t\\label{lemma:svr-f-new}\n    Suppose Assumption~\\ref{assumption:g-lip-finite} holds and $\\tgki$ and $\\tJki$ are constructed according to~\\eqref{eqn:g-svrg-correction} and~\\eqref{eqn:J-svrg} respectively, then \n\\begin{eqnarray*}\n\t\\E\\Bigl[\\bigl\\|\\tgki - g(\\xki)\\bigr\\| \\,\\big|\\, \\xki\\Bigr] \n        &\\leq& \\frac{L_g}{2\\sqrt{|\\cB^k_i|}}\\bigl\\|\\xki - x_0^k\\bigr\\|^2,\\\\ \n\t\\E\\Bigl[\\bigl\\|\\tJki - g'(\\xki)\\bigr\\|^2 \\,\\big|\\, \\xki\\Bigr] \n    &\\leq& \\frac{L_g^2}{|\\cS_i^k|}\\bigl\\|\\xki - x_0^k\\bigr\\|^2,\n\\end{eqnarray*}\nwhere $\\E[\\cdot|\\xki]$ denotes conditional expectation given~$\\xki$, i.e., expectation with respect to the random indices in $\\Bki$ and $\\Ski$.\n\\end{lemma}\n\n\\begin{proof}\n    To prove the first inequality, we start with~\\eqref{eqn:g-svrg-correction} and write\n\\begin{equation}\\label{eqn:tgki-Z}\n    \\tgki - g(\\xki) = \\frac{1}{|\\Bki|}\\sum_{j\\in\\Bki}Z_j,\n\\end{equation}\n    where\n    $$Z_j = g_j(\\xki) - g_j(x_0^k) - g_j'(x_0^k)(\\xki-x_0^k) \\,+\\, g(x_0^k) + g'(x_0^k)(\\xki-x_0^k) \\,-\\, g(\\xki).$$\n    Since~$j$ is randomly sampled from $\\{1,2,\\ldots,N\\}$, we have $\\E[g_j(\\xki)]=g(\\xki)$ and $\\E[g'_j(\\xki)]=g'(\\xki)$, which implies $\\E[Z_j|\\xki] = 0$. That is, $\\tgki$ is an unbiased estimate of $g(\\xki)$.\n    In addition, we have\n\t$$\\E\\bigl[g_j(\\xki) - g_j(x_0^k) - g_j'(x_0^k)(\\xki-x_0^k)\\,\\bigl|\\,\\xki\\bigl] =   g(\\xki) - g(x_0^k) - g'(x_0^k)(\\xki-x_0^k).$$ \nThis allows us to bound the variance of $Z_j$ as follows:\n\\begin{eqnarray*}\n\t\\E\\bigl[\\|Z_j\\|^2\\,\\big|\\,\\xki\\bigr] \n    & = & \\E\\Bigl[\\bigl\\|g_j(\\xki) - g_j(x_0^k) - g_j'(x_0^k)(\\xki-x_0^k) \\bigr\\|^2\\,\\big|\\,\\xki\\Bigr] \\\\\n    && -\\, \\bigl\\|g(\\xki) - g(x_0^k) - g'(x_0^k)(\\xki-x_0^k)\\bigr\\|^2 \\\\\n\t& \\leq & \\E\\Bigl[\\bigl\\|g_j(\\xki) - g_j(x_0^k) - g_j'(x_0^k)(\\xki-x_0^k) \\bigr\\|^2 \\,\\big|\\,\\xki\\Bigr]\\\\\n\t&\\leq & \\frac{1}{N}\\sum_{j=1}^N\\left(\\frac{L_{g,j}}{2}\\|\\xki-x_0^k\\|^2\\right)^2\\\\\n\t& = & \\frac{L_{g}^2}{4}\\|\\xki-x_0^k\\|^4,\n\\end{eqnarray*}\nwhere the last inequality is due to~\\eqref{eqn:g-quadratic-bd} and Assumption \\ref{assumption:g-lip-finite} respectively.\nIn the last equality, we used the definition of~$L_g$ in~\\eqref{eqn:ms-Lip-constants}.\n\nCombining the above inequality with~\\eqref{eqn:tgki-Z} yields\n\t$$\\E\\Bigl[\\bigl\\|\\tgki - g(\\xki)\\bigr\\|^2 \\,\\big|\\, \\xki\\Bigr] \\leq \\frac{L_g^2}{4|\\cB^k_i|}\\bigl\\|\\xki - x_0^k\\bigr\\|^4.$$ \n    Next, using the concavity of~$\\sqrt{\\cdot}$ and Jensen's inequality, we obtain the desired result:\n\\[\n\t\\E\\Bigl[\\bigl\\|\\tgki - g(\\xki)\\bigr\\| \\,\\big|\\, \\xki\\Bigr] \n\t\\leq \\sqrt{\\E\\Bigl[\\bigl\\|\\tgki - g(\\xki)\\bigr\\|^2 \\,\\big|\\, \\xki\\Bigr] }\n\t\\leq \\frac{L_g}{2\\sqrt{|\\cB^k_i|}}\\bigl\\|\\xki - x_0^k\\bigr\\|^2.\n\\]\nTo prove the second inequality, we define $Z_j = g'_j(\\xki) - g'_j(x_0^k) + g'(x_0^k) - g'(\\xki)$ and follow a similar line of arguments. \n\\end{proof}\n\n\\begin{lemma}\n\\label{lemma: descent-Finite-nonsmooth}\nSuppose Assumptions~\\ref{assumption:f-cvx} and~\\ref{assumption:g-lip-finite} hold and the estimates $\\tgkz$, $\\tJkz$, $\\tgki$ and $\\tJki$ in Algorithm~\\ref{algo:SVR-PL} are constructed as in~\\eqref{eqn:finite-gk0}-\\eqref{eqn:J-svrg} respectively.\nThen for $k=1,\\ldots,K$ and $i=0,\\ldots,\\tau-1$, \n\\begin{eqnarray}\n\\Phi(\\xkie) \n&\\leq& \\Phi(\\xki) - \\frac{M - 2\\ell_f L_g}{2M^2}\\bigl\\|\\tcGki\\bigr\\|^2 \\nonumber \\\\\n&& +\\, 2\\ell_f \\bigl\\|\\tgki-g(\\xki)\\bigr\\| + \\frac{\\ell_f}{2L_g}\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|^2. \n\\label{eqn:descent-lemma}\n\\end{eqnarray}\n\\end{lemma}\n\n\\begin{proof}\nBy the definition of $\\Phi$ in~\\eqref{eqn:Phi-def} and Lemma~\\ref{lemma:upper-bound}, we have \n\\begin{eqnarray}\n\\Phi(\\xkie) & \\leq & f(g(\\xki) + g'(\\xki)(\\xkie-\\xki)) + \\frac{\\ell_f L_g}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2 + h(\\xkie) \\nonumber \\\\\n& = &  f\\bigl(\\tgki + \\tJki(\\xkie-\\xki)\\bigr) + h(\\xkie) + \\frac{M}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2\n -\\, \\frac{M - \\ell_f L_g}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2 \\nonumber \\\\\n& & + \\underbrace{f\\bigl(g(\\xki) + g'(\\xki)(\\xkie-\\xki)\\bigr)-f\\bigl(\\tgki + g'(\\xki)(\\xkie-\\xki)\\bigr)}_{T_1}\\nonumber\\\\\n& & + \\underbrace{f\\bigl(\\tgki + g'(\\xki)(\\xkie-\\xki)\\bigr)-f\\bigl(\\tgki + \\tJki(\\xkie-\\xki)\\bigr)}_{T_2} .\n\\label{eqn:descent-finite-0}\n\\end{eqnarray}\nAccording to~\\eqref{eqn:xkie-def}, we have\n\\[\n    f\\bigl(\\tgki + \\tJki(\\xkie-\\xki)\\bigr) + h(\\xkie) + \\frac{M}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2 ~\\leq~ f(\\tgki) + h(\\xki).\n\\]\nTherefore,\n\\begin{eqnarray}\n\\Phi(\\xkie) \n&\\leq & f(\\tgki) + h(\\xki) - \\frac{M-\\ell_f L_g}{2} \\|\\xkie-\\xki\\|^2 + T_1+ T_2 \\nonumber\\\\\n&\\leq & f(g(\\xki)) + h(\\xki) - \\frac{M-\\ell_f L_g}{2} \\|\\xkie-\\xki\\|^2 + T_1+ T_2  +  \\underbrace{f(\\tgki)-f(g(\\xki))}_{T_3}\\nonumber\\\\\n& = & \\Phi(\\xki)  - \\frac{M-\\ell_f L_g}{2} \\|\\xkie-\\xki\\|^2 + T_1+ T_2 +T_3. \n\\label{eqn:Phi-descent}\n\\end{eqnarray}\nBy the Lipschitz property of~$f$, we have\n\\begin{eqnarray*}\n    T_1 \\leq \\ell_f \\bigl\\|\\tgki-g(\\xki)\\bigr\\|, \\qquad\n    T_3 \\leq \\ell_f \\bigl\\|\\tgki-g(\\xki)\\bigr\\|,\n\\end{eqnarray*}\nand\n\\begin{eqnarray}\nT_2 &\\leq& \\ell_f\\bigl\\|\\bigl(\\tJki-g'(\\xki)\\bigr)(\\xkie-\\xki)\\bigr\\|\n~\\leq~ \\ell_f\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|\\cdot\\bigl\\|\\xkie-\\xki\\bigr\\| \\nonumber\\\\\n&\\leq& \\frac{\\ell_f}{2L_g}\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|^2 + \\frac{\\ell_f L_g}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2.\n\\label{eqn:descent-lemma-T2}\n\\end{eqnarray}\nCombining the bounds on $T_1$, $T_2$ and $T_3$ and the inequality~\\eqref{eqn:Phi-descent} yields\n\\begin{eqnarray*}\n    \\Phi(\\xkie) \n    &\\leq& \\Phi(\\xki) - \\left(\\frac{M}{2} - \\ell_f L_g\\right)\\bigl\\|\\xkie-\\xki\\bigr\\|^2 \\\\\n    && +\\, 2\\ell_f \\bigl\\|\\tgki-g(\\xki)\\bigr\\| + \\frac{\\ell_f}{2L_g}\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|^2, \n\\end{eqnarray*}\nwhich, upon noticing $\\tcGki=-M(\\xkie-\\xki)$, is equivalent to the\ndesired result. \n\\end{proof}\n\n\\begin{lemma}\n\t\\label{lemma:Lyapunov-finite-nonsmooth}\nSuppose Assumptions~\\ref{assumption:f-cvx} and~\\ref{assumption:g-lip-finite} hold and the estimates $\\tgkz$, $\\tJkz$, $\\tgki$ and $\\tJki$ in Algorithm~\\ref{algo:SVR-PL} are constructed as in~\\eqref{eqn:finite-gk0}-\\eqref{eqn:J-svrg} respectively.\nIn addition, we assume $M\\geq 4\\ell_f L_g$ and $\\gamma>0$.\nThen for each $k=1,\\ldots,K$,\n\\begin{equation}\\label{lm:Lyapunov-finite-nonsmooth-1}\n\t\\sum_{i=0}^{\\tau-1}\\E\\Bigl[\\bigl\\|\\cGki\\bigr\\|^2\\Bigr] \n    ~\\leq~  \\frac{R^k_0-R^k_\\tau}{\\gamma}  \n    ~=~ \\frac{\\E[\\Phi(x_0^k)]-\\E[\\Phi(x_0^{k+1})]}{\\gamma}.\n\\end{equation}\n\\end{lemma}\n\n\\begin{proof}\nFor the ease of notation, we write the stochastic Lyapunov function as\n$$R^k_i = \\E\\bigl[\\Phi(\\xki) + c_i\\|G_i^k\\|^2\\bigr],$$\nwhere\n\\begin{equation}\\label{eqn:Gki-def}\n    \\Gki = \\sum_{t=0}^{i-1}\\tcGkt = -M(\\xki - x_0^k).\n\\end{equation}\nIn particular, we have $G_{0}^k = -M(x_0^k-x_0^k) = 0$.\nMoreover, we have \n\\begin{eqnarray}\n\t\\E\\bigl[\\|G_{i+1}^k\\|^2\\bigr] \n    &=& \\E\\bigl[\\|G_{i}^k + \\tcGki\\|^2\\bigr]   \\nonumber \\\\\n    &\\leq& \\left(1+\\frac{1}{\\tau}\\right)\\E\\bigl[\\|G_{i}^k\\|^2\\bigr] + (1+\\tau)\\E\\bigl[\\|\\tcGki\\|^2\\bigr]. \n\\label{eqn:Gki-tcG}\n\\end{eqnarray}\nCombining Lemmas~\\ref{lemma:PG-finite-sum-nonsmooth} \nand~\\ref{lemma: descent-Finite-nonsmooth} yields\n\\begin{eqnarray*}\n\\E\\bigl[\\Phi(\\xkie)\\bigr] \n    &\\leq& \\E\\bigl[\\Phi(\\xki)\\bigr] - \\frac{M - 2\\ell_f L_g}{2M^2}\\E\\bigl[\\bigl\\|\\tcGki\\bigr\\|^2\\bigr] + \\left(\\frac{\\ell_f L_g}{\\sqrt{|\\cB_i^k|}}+\\frac{\\ell_f L_g}{2|\\cS_i^k|}\\right)\\E\\bigl[\\|\\xki-x_0^k\\|^2\\bigr] \\\\\n    &=& \\E\\bigl[\\Phi(\\xki)\\bigr] - \\frac{M - 2\\ell_f L_g}{2M^2}\\E\\bigl[\\bigl\\|\\tcGki\\bigr\\|^2\\bigr] + \\frac{1}{M^2}\\left(\\frac{\\ell_f L_g}{\\sqrt{|\\cB_i^k|}}+\\frac{\\ell_f L_g}{2|\\cS_i^k|}\\right)\\E\\bigl[\\|G^k_i\\|^2\\bigr],\n\\end{eqnarray*}\nwhere in the last equality we used~\\eqref{eqn:Gki-def}.\nAdding both sides of~\\eqref{eqn:Gki-tcG} to that of the above inequality \nand using the assumption $M\\geq 4\\ell_f L_g$, we obtain\n\\begin{eqnarray}\n\\E\\bigl[\\Phi(\\xkie)+c_{i+1}\\|G_{i+1}^k\\|^2\\bigr] \n    & \\leq & \\E\\bigl[\\Phi(\\xki)\\bigr] - \\left(\\frac{M-2\\ell_f L_g}{2M^2} - c_{i+1}(1+\\tau)\\right)\\E\\bigl[\\|\\tcGki\\|^2\\bigr] \\nonumber \\\\ \n    & & +\\left(\\frac{1}{M^2}\\left(\\frac{\\ell_f L_g}{\\sqrt{|\\cB_i^k|}}+\\frac{\\ell_f L_g}{2|\\cS_i^k|}\\right) + c_{i+1}\\left(1+\\frac{1}{\\tau}\\right)\\right)\\E\\bigl[\\|G_i^k\\|^2\\bigr] \\nonumber \\\\\n    & \\leq & \\E\\bigl[\\Phi(\\xki)\\bigr] - \\left(\\frac{1}{4M} - c_{i+1}(1+\\tau)\\right)\\E\\bigl[\\|\\tcGki\\|^2\\bigr] \\label{eqn:lyapunov-descent}\\\\ \n    & & +\\left(\\frac{1}{4M}\\left(\\frac{1}{\\sqrt{|\\cB_i^k|}}+\\frac{1}{2|\\cS_i^k|}\\right) + c_{i+1}\\left(1+\\frac{1}{\\tau}\\right)\\right)\\E\\bigl[\\|G_i^k\\|^2\\bigr]. \\nonumber\n\\end{eqnarray}\nNext, combining Lemma~\\ref{lemma:PG-finite-sum-nonsmooth} with Lemmas~\\ref{lemma:svr-f-new} yields\n\\begin{eqnarray*}\n    \\frac{M - \\ell_f L_g}{M^2}\\E\\bigl[\\|\\cGki\\|^2\\bigr]\n    &\\leq&\n    \\frac{2M+\\ell_f L_g}{M^2}\\E\\bigl[\\|\\tcGki\\|^2\\bigr] \n     + \\left( \\frac{2\\ell_f L_g}{\\sqrt{|\\cB_i^k|}} + \\frac{2\\ell_f L_g}{|\\cS_i^k|} \\right) \\E\\bigl[\\|\\xki-\\xkz\\|^2\\bigr].\n\\end{eqnarray*}\nUsing the equality $G^k_i=-M(\\xki-\\xkz)$ and the assumption $M\\geq 4\\ell_f L_g$, the above inequality implies\n\\begin{eqnarray}\n    \\frac{3}{4M}\\E\\bigl[\\|\\cGki\\|^2\\bigr]\n    &\\leq&\n    \\frac{9}{4M}\\E\\bigl[\\|\\tcGki\\|^2\\bigr] \n     +\\, \\frac{1}{2M} \\left( \\frac{1}{\\sqrt{|\\cB_i^k|}} + \\frac{1}{|\\cS_i^k|} \\right) \\E\\bigl[\\|G^k_i\\|^2\\bigr].\n    \\label{eqn:cG-bound}\n\\end{eqnarray}\nMultiplying both sides of~\\eqref{eqn:cG-bound} by $\\left(\\frac{1}{4M}-c_{i+1}(1+\\tau)\\right)\\big/\\frac{9}{4M}$, which is positive by the assumption $\\gamma>0$, and adding the resulting inequality to~\\eqref{eqn:lyapunov-descent}, we get\n\\begin{eqnarray}\n%\\label{lm:Lyapunov-finite-nonsmooth-2}\n\\E\\bigl[\\Phi(\\xkie)+c_{i+1}\\|G_{i+1}^k\\|^2\\bigr] \n&\\leq& \\E\\left[\\Phi(\\xki)+\\left( c_{i+1}\\left(1+\\frac{1}{\\tau}\\right)+ \\frac{1}{3M\\sqrt{|\\cB_i^k|}} + \\frac{1}{5M|\\cS_i^k|}\\right)\\|G_i^k\\|^2\\right] \\nonumber\\\\\n\t&& -\\frac{1}{3}\\left(\\frac{1}{4M} - c_{i+1}(1+\\tau)\\right)\\E[\\|\\cGki\\|^2].\\nonumber\n\\end{eqnarray}\nNow, using the definitions in~\\eqref{eqn:lyapunov-def} and~\\eqref{eqn:ct-finite-nonsmooth}, the above inequality is the same as\n\\[\n\t\\frac{1}{3}\\left(\\frac{1}{4M} - c_{i+1}(1+\\tau)\\right)\\E\\bigl[\\|\\cGki\\|^2\\bigr] ~\\leq~  R_i^k  - R_{i+1}^k.\n\\]\nRecalling the definition of $\\gamma$ and summing up the above inequality over~$i$ from~$0$ to~$\\tau-1$, we get\n\\[\n\t\\gamma\\sum_{i=0}^{\\tau-1}\\E\\bigl[\\|\\cGki\\|^2\\bigr]\n    ~\\leq~ R_0^k-R_\\tau^k\n    ~=~\\E\\bigl[\\Phi(x_0^k)\\bigr] - \\E\\bigl[\\Phi(x_\\tau^k)\\bigr] ,\n\\]\nwhere the last equality is due to the observations that \n$c_\\tau=0$ and $G^k_0=0$. \nFinally, dividing both sides by $\\gamma$ and using $x^{k+1}_0 = x^k_\\tau$\ngive the desired result.\n\\end{proof}\n\n\\begin{lemma}\n\\label{lemma:gamma-nonsmooth}\n    If we choose \n    $\\tau = \\bigl\\lceil\\frac{1}{2}N^{1/5}-1\\bigr\\rceil$,\n    $|\\cB_i^k| = \\bigr\\lceil 4N^{4/5}\\bigr\\rceil$ and\n    $|\\cS_i^k| = \\bigr\\lceil N^{2/5}\\bigr\\rceil$ for $i=1,\\ldots,\\tau-1$,\n    then $\\gamma  \\geq \\frac{1}{15M}$.\n\\end{lemma}\n\n\\begin{proof}\nTo simplify notation, let $B=|\\Bki|$ and $S=|\\Ski|$ for $i=1,\\ldots,\\tau-1$.\nFrom \\eqref{eqn:ct-finite-nonsmooth}, we deduce\n\\[\n    (c_i+C) = (c_{i+1}+C)\\left(1+\\frac{1}{\\tau}\\right), \\qquad\n        \\mbox{where}\\quad C = \\frac{\\tau}{3M\\sqrt{B}} + \\frac{\\tau}{5MS}.\n\\] \nConsequently, with $c_\\tau=0$, we have for all $i=1,\\ldots,\\tau$, \n\\[\n    c_i ~=~ (c_\\tau + C)\\left(1+\\frac{1}{\\tau}\\right)^{\\tau-i}\\!\\! -C\n    ~\\leq~  C\\left(1+\\frac{1}{\\tau}\\right)^{\\tau} - C\n    ~\\leq~ Ce -C ~=~ C(e-1),\n\\] \nwhere the last inequality is due to the fact that \n$(1+1/\\tau)^\\tau\\leq e$ with $e$ is Euler's number (the basis of natural logarithm). Therefore, \n\\begin{eqnarray*}\n\t\\gamma  \n    & =  & \\min_{0\\leq i\\leq \\tau-1}\\frac{1}{3}\\left(\\frac{1}{4M} - c_{i+1}(1+\\tau)\\right)\\\\\n    & \\geq & \\frac{1}{3}\\left(\\frac{1}{4M} - C(e-1)(1+\\tau)\\right)\\\\\n    & = & \\frac{1}{3M}\\left(\\frac{1}{4} - \\left( \\frac{1}{3\\sqrt{B}} + \\frac{1}{5S}\\right)(e-1)\\tau(1+\\tau)\\right) \\\\\n    & \\geq & \\frac{1}{3M}\\left(\\frac{1}{4} - \\left( \\frac{1}{3\\sqrt{B}} + \\frac{1}{5S}\\right)2(1+\\tau)^2\\right).\n\\end{eqnarray*}\nFinally, setting $\\tau=\\frac{1}{2}N^{1/5}-1$, $B=4N^{4/5}$ and $S=N^{2/5}$ yields $\\gamma\\geq\\frac{1}{15M}$.\n\\end{proof}\n\n\\begin{proof}\n\tSumming up the inequality \\eqref{lm:Lyapunov-finite-nonsmooth-1} \n    over~$k$ from~$1$ to~$K$ and using the fact $\\Phi(x^K_\\tau)>\\Phi_*$, we get\n\t\\begin{eqnarray*}\n\t\t\\sum_{k=1}^K\\sum_{i=0}^{\\tau-1}\\E\\bigl[\\|\\cGki\\|^2\\bigr] ~\\leq~ \\frac{\\Phi(x_0^1) - \\Phi(x^*)}{\\gamma}.\n\t\\end{eqnarray*}\n\tBy the random choice of the output $x_{i^*}^{k^*}$, we can get the inequality \\eqref{eqn:cvg-svr-pl-nonsmooth-1}.\n\t\n    To get an $\\epsilon$-stationary point in expectation, we need to set $K\\tau = \\cO(\\epsilon^{-1})$, which implies \n\\[\n    K=\\cO(\\tau^{-1}\\epsilon^{-1})=\\cO(N^{-1/5}\\epsilon^{-1}).\n\\] \nConsequently, the sample complexity of the component mappings (the $g_i$'s) is \n\\[\n    KN + K\\tau B \n    ~=~ \\cO(N^{-1/5}\\epsilon^{-1})\\cdot N + \\cO(\\epsilon^{-1}) \\cdot 4 N^{4/5} \n    ~=~ \\cO(N+N^{4/5}\\epsilon^{-1}).\n\\]\nand the sample complexity for the component Jacobians is\n\\[\n    KN + K\\tau S \n    ~=~ \\cO(N^{-1/5}\\epsilon^{-1})\\cdot N + \\cO(\\epsilon^{-1}) \\cdot N^{2/5} \n    ~=~ \\cO(N+N^{4/5}\\epsilon^{-1}).\n\\]\nThis completes the proof.\n\\end{proof}","units":[{"start":27345,"end":27604,"start_line":376,"end_line":382,"raw_tex":"\\begin{lemma}\\label{lemma:upper-bound} \nSuppose Assumption \\ref{assumption:f-cvx} and \\ref{assumption:g-lip} hold, then \nfor any $x,y\\in\\dom h$,\n\\begin{equation}\n    f(g(x)) \\leq f(g(y) + g'(y)(x-y)) + \\frac{\\ell_f L_g}{2}\\|x-y\\|^2.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"ba953ab0e40f1ecf686b76eb009d0e258340a25879f14f1ff67a5b8a4ae2b304"},{"start":27605,"end":28154,"start_line":383,"end_line":393,"raw_tex":"\\begin{proof} \n    By the Lipschitz continuity of $f$ and $g'$, we have\n\t\\begin{eqnarray*}\n    f(g(x)) \n    &=&  f\\bigl(g(y) + g'(y)(x-y)\\bigr) + f(g(x)) - f\\bigl(g(y) + g'(y)(x-y)\\bigr)\\\\\n    & \\leq &  f\\bigl(g(y) + g'(y)(x-y)\\bigr) + \\bigl|f(g(x)) - f\\bigl(g(y) + g'(y)(x-y)\\bigr)\\bigr|\\\\\n\t&\\leq &f\\bigl(g(y) + g'(y)(x-y)\\bigr) + \\ell_f\\bigl\\|g(x) - g(y) - g'(y)(x-y)\\bigr\\|\\\\\n\t& \\leq & f\\bigl(g(y) + g'(y)(x-y)\\bigr) + \\frac{\\ell_f L_g}{2}\\|x-y\\|^2,\n\t\\end{eqnarray*}\n    where the last inequality is due to~\\eqref{eqn:g-quadratic-bd}.\n\\end{proof}","sha256_utf8":"5a9506f597d73e66f9aea0c3c4cc9a869e226761982517c10c8d000e6350d04a"},{"start":33956,"end":34421,"start_line":469,"end_line":478,"raw_tex":"\\begin{lemma}\n\\label{lemma:PG-finite-sum-nonsmooth}\nUnder Assumptions \\ref{assumption:f-cvx} and \\ref{assumption:g-lip}, \nthe iterates generated by Algorithm~\\ref{algo:SVR-PL} satisfy\n\\begin{eqnarray*}\n    \\frac{M - \\ell_f L_g}{M^2}\\bigl\\|\\cG(\\xki)\\bigr\\|^2 \n    &\\leq& \\frac{2M+\\ell_f L_g}{M^2}\\bigl\\|\\widetilde{\\cG}(\\xki)\\bigr\\|^2  \\\\\n    && \\,+\\, 4\\ell_f\\bigl\\|\\tgki-g(\\xki)\\bigr\\| + \\frac{2\\ell_f}{L_g}\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|^2.\n\\end{eqnarray*}\n\\end{lemma}","sha256_utf8":"6d1b523aad7e7ae8c813b92c5271d53547dd4df898db4774cf2eac6c5af5a721"},{"start":34422,"end":38127,"start_line":479,"end_line":548,"raw_tex":"\\begin{proof}\nFor the ease of notation,  we denote \n\\begin{eqnarray}\n    F(x;\\xki) &=& f\\bigl(g(\\xki) + g'(\\xki)(x-\\xki)\\bigr), \\label{eqn:F-def}\\\\\n    \\widetilde{F}(x;\\xki) &=& f\\bigl(\\tgki + \\tJki(x-\\xki)\\bigr).\n    \\label{eqn:tF-def}\n\\end{eqnarray}\nSince both~$f$ and~$h$ are convex (Assumption~\\ref{assumption:f-cvx}),\nthe following two functions are $M$-strongly convex:\n\\begin{eqnarray}\n    && F(x;\\xki) + h(x) + \\frac{M}{2}\\|x-\\xki\\|^2, \\label{eqn:F+h+M} \\\\\n    && \\widetilde{F}(x;\\xki) + h(x) + \\frac{M}{2}\\|x-\\xki\\|^2.\n    \\label{eqn:tF+h+M} \n\\end{eqnarray} \nAccording to~\\eqref{eqn:x-hat-def} and~\\eqref{eqn:xkie-def}, $\\hat{x}_{i+1}^k$ and $\\xkie$ are the minimizers of these two functions respectively. \nTherefore\n\\begin{eqnarray*}\n    F(\\hat{x}^k_{i+1};\\xki) + h(\\hat{x}^k_{i+1}) + \\frac{M}{2}\\|\\hat{x}^k_{i+1}-\\xki\\|^2 \n\t& \\leq & F(\\xkie;\\xki) + h(\\xkie) + \\frac{M}{2}\\|\\xkie-\\xki\\|^2  \\\\\n    &&  - \\frac{M}{2}\\|\\hat{x}^k_{i+1}-\\xkie\\|^2,  \n\\end{eqnarray*} \nand\n\\begin{eqnarray*}\n    \\widetilde{F}(x^k_{i+1};\\xki) + h(x^k_{i+1}) + \\frac{M}{2}\\|x^k_{i+1}-\\xki\\|^2 \n& \\leq & \\widetilde{F}(\\hat{x}^k_{i+1};\\xki) + h(\\hat{x}^k_{i+1}) + \\frac{M}{2}\\|\\hat{x}^k_{i+1}-\\xki\\|^2 \\\\\n&& - \\frac{M}{2}\\|\\hat{x}^k_{i+1}-\\xkie\\|^2. \n\\end{eqnarray*} \nSumming the two inequalities above and rearranging the terms, we obtain\n\\begin{eqnarray}\n\t\\label{lm:PG-finite-sum-3}\n\tM\\|\\hat{x}^k_{i+1}-\\xkie\\|^2 \n    &\\leq& F(\\xkie;\\xki)-\\widetilde{F}(\\xkie;\\xki) \n     + \\widetilde{F}(\\hat{x}^k_{i+1};\\xki)-F(\\hat{x}^k_{i+1};\\xki).\n\\end{eqnarray}\nUsing the Lipschitz property of~$f$, we have \n\\begin{eqnarray*}\n\\bigl| F(\\xkie;\\xki)-\\widetilde{F}(\\xkie;\\xki)\\bigr| \n    &=& \\left| f\\bigl(g(\\xki)+g'(\\xki)(\\xkie-\\xki)\\bigr) - f\\bigl(\\tgki+\\tJki(\\xkie-\\xki)\\bigr)\\right| \\\\\n    & \\leq & \\ell_f\\left\\|\\bigl(g(\\xki) - \\tgki\\bigr) + \\bigl(g'(\\xki) - \\tJki\\bigr)\\bigl(\\xkie - \\xki\\bigr)\\right\\|\\\\\n\t& \\leq & \\ell_f\\left(\\bigl\\|\\tgki - g(\\xki)\\bigr\\| + \\bigl\\|\\tJki - g'(\\xki)\\bigr\\|\\bigl\\|\\xkie - \\xki\\bigr\\|\\right)\\\\\n    & \\leq & \\ell_f\\left(\\bigl\\|\\tgki - g(\\xki)\\bigr\\| + \\frac{1}{2L_g}\\bigl\\|\\tJki - g'(\\xki)\\bigr\\|^2 + \\frac{L_g}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2 \\right).\n\\end{eqnarray*}\nReplacing $\\xkie$ in the above inequality with $\\hxkie$, we get  \n\\begin{eqnarray*}\n\\bigl|F(\\hxkie;\\xki)-\\widetilde{F}(\\hxkie;\\xki)\\bigr| \n\t& \\leq & \\ell_f \\left(\\bigl\\|\\tgki - g(\\xki)\\bigr\\| + \\frac{1}{2L_g}\\bigl\\|\\tJki - g'(\\xki)\\bigr\\|^2 + \\frac{L_g}{2}\\bigl\\|\\hxkie-\\xki\\bigr\\|^2\\right).\n\\end{eqnarray*}\nCombining the two bounds above with~\\eqref{lm:PG-finite-sum-3} gives\n\\begin{eqnarray*}\n\tM\\bigl\\|\\hxkie-\\xkie\\bigr\\|^2 \n    & \\leq & 2\\ell_f\\bigl\\|\\tgki - g(\\xki)\\bigr\\| + \\frac{\\ell_f}{L_g}\\bigl\\|\\tJki - g'(\\xki)\\bigr\\|^2 \\\\\n    && +\\, \\frac{\\ell_f L_g}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2+\\frac{\\ell_f L_g}{2}\\bigl\\|\\hxkie-\\xki\\bigr\\|^2 .\n\\end{eqnarray*}\nNext, using the fact that $\\|a+b\\|^2\\leq 2\\|a\\|^2+2\\|b\\|^2$ and the above inequality, we have\n\\begin{eqnarray*} \n\tM\\bigl\\|\\hxkie-\\xki\\bigr\\|^2] \n    & \\leq & 2M\\bigl\\|\\xkie-\\xki\\bigr\\|^2 + 2M\\bigl\\|\\hxkie -\\xkie\\bigr\\|^2 \\\\\n\t& \\leq & 2M\\bigl\\|\\xkie-\\xki\\bigr\\|^2 + 4\\ell_f\\bigl\\|\\tgki - g(\\xki)\\bigr\\| + \\frac{2\\ell_f}{L_g}\\bigl\\|\\tJki - g'(\\xki)\\bigr\\|^2 \\\\\n\t& & + \\ell_f L_g\\bigl\\|\\xkie-\\xki\\bigr\\|^2 + \\ell_f L_g\\bigl\\|\\hxkie-\\xki\\bigr\\|^2.\n\\end{eqnarray*}\nRearranging the terms yields \n\\begin{eqnarray*}\n    (M - \\ell_f L_g)\\bigl\\|\\hxkie-\\xki\\bigr\\|^2 \n    &\\leq& (2M+\\ell_f L_g)\\bigl\\|\\xkie-\\xki\\bigr\\|^2  \\\\\n    && +\\, 4\\ell_f\\bigl\\|\\tgki-g(\\xki)\\bigr\\| \n     \\,+\\, \\frac{2\\ell_f}{L_g}\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|^2.\n\\end{eqnarray*}\nFinally, using the definitions $\\cG(\\xki)=M(\\xki-\\hxkie)$ and\n$\\widetilde{\\cG}(\\xki)=M(\\xki-\\xkie)$, we obtain the desired result.\n\\end{proof}","sha256_utf8":"1e14e01f85067b363c2d6e8148300913ef1dbbe60293c96a12b3949dc476dc0e"},{"start":41913,"end":42592,"start_line":600,"end_line":610,"raw_tex":"\\begin{lemma}\n\t\\label{lemma:svr-f-new}\n    Suppose Assumption~\\ref{assumption:g-lip-finite} holds and $\\tgki$ and $\\tJki$ are constructed according to~\\eqref{eqn:g-svrg-correction} and~\\eqref{eqn:J-svrg} respectively, then \n\\begin{eqnarray*}\n\t\\E\\Bigl[\\bigl\\|\\tgki - g(\\xki)\\bigr\\| \\,\\big|\\, \\xki\\Bigr] \n        &\\leq& \\frac{L_g}{2\\sqrt{|\\cB^k_i|}}\\bigl\\|\\xki - x_0^k\\bigr\\|^2,\\\\ \n\t\\E\\Bigl[\\bigl\\|\\tJki - g'(\\xki)\\bigr\\|^2 \\,\\big|\\, \\xki\\Bigr] \n    &\\leq& \\frac{L_g^2}{|\\cS_i^k|}\\bigl\\|\\xki - x_0^k\\bigr\\|^2,\n\\end{eqnarray*}\nwhere $\\E[\\cdot|\\xki]$ denotes conditional expectation given~$\\xki$, i.e., expectation with respect to the random indices in $\\Bki$ and $\\Ski$.\n\\end{lemma}","sha256_utf8":"aa0b2dca0941bd710e08bb6d0915865808bdc124e6d77f46a13f1530bac0d64d"},{"start":42593,"end":44697,"start_line":611,"end_line":642,"raw_tex":"\\begin{proof}\n    To prove the first inequality, we start with~\\eqref{eqn:g-svrg-correction} and write\n\\begin{equation}\\label{eqn:tgki-Z}\n    \\tgki - g(\\xki) = \\frac{1}{|\\Bki|}\\sum_{j\\in\\Bki}Z_j,\n\\end{equation}\n    where\n    $$Z_j = g_j(\\xki) - g_j(x_0^k) - g_j'(x_0^k)(\\xki-x_0^k) \\,+\\, g(x_0^k) + g'(x_0^k)(\\xki-x_0^k) \\,-\\, g(\\xki).$$\n    Since~$j$ is randomly sampled from $\\{1,2,\\ldots,N\\}$, we have $\\E[g_j(\\xki)]=g(\\xki)$ and $\\E[g'_j(\\xki)]=g'(\\xki)$, which implies $\\E[Z_j|\\xki] = 0$. That is, $\\tgki$ is an unbiased estimate of $g(\\xki)$.\n    In addition, we have\n\t$$\\E\\bigl[g_j(\\xki) - g_j(x_0^k) - g_j'(x_0^k)(\\xki-x_0^k)\\,\\bigl|\\,\\xki\\bigl] =   g(\\xki) - g(x_0^k) - g'(x_0^k)(\\xki-x_0^k).$$ \nThis allows us to bound the variance of $Z_j$ as follows:\n\\begin{eqnarray*}\n\t\\E\\bigl[\\|Z_j\\|^2\\,\\big|\\,\\xki\\bigr] \n    & = & \\E\\Bigl[\\bigl\\|g_j(\\xki) - g_j(x_0^k) - g_j'(x_0^k)(\\xki-x_0^k) \\bigr\\|^2\\,\\big|\\,\\xki\\Bigr] \\\\\n    && -\\, \\bigl\\|g(\\xki) - g(x_0^k) - g'(x_0^k)(\\xki-x_0^k)\\bigr\\|^2 \\\\\n\t& \\leq & \\E\\Bigl[\\bigl\\|g_j(\\xki) - g_j(x_0^k) - g_j'(x_0^k)(\\xki-x_0^k) \\bigr\\|^2 \\,\\big|\\,\\xki\\Bigr]\\\\\n\t&\\leq & \\frac{1}{N}\\sum_{j=1}^N\\left(\\frac{L_{g,j}}{2}\\|\\xki-x_0^k\\|^2\\right)^2\\\\\n\t& = & \\frac{L_{g}^2}{4}\\|\\xki-x_0^k\\|^4,\n\\end{eqnarray*}\nwhere the last inequality is due to~\\eqref{eqn:g-quadratic-bd} and Assumption \\ref{assumption:g-lip-finite} respectively.\nIn the last equality, we used the definition of~$L_g$ in~\\eqref{eqn:ms-Lip-constants}.\n\nCombining the above inequality with~\\eqref{eqn:tgki-Z} yields\n\t$$\\E\\Bigl[\\bigl\\|\\tgki - g(\\xki)\\bigr\\|^2 \\,\\big|\\, \\xki\\Bigr] \\leq \\frac{L_g^2}{4|\\cB^k_i|}\\bigl\\|\\xki - x_0^k\\bigr\\|^4.$$ \n    Next, using the concavity of~$\\sqrt{\\cdot}$ and Jensen's inequality, we obtain the desired result:\n\\[\n\t\\E\\Bigl[\\bigl\\|\\tgki - g(\\xki)\\bigr\\| \\,\\big|\\, \\xki\\Bigr] \n\t\\leq \\sqrt{\\E\\Bigl[\\bigl\\|\\tgki - g(\\xki)\\bigr\\|^2 \\,\\big|\\, \\xki\\Bigr] }\n\t\\leq \\frac{L_g}{2\\sqrt{|\\cB^k_i|}}\\bigl\\|\\xki - x_0^k\\bigr\\|^2.\n\\]\nTo prove the second inequality, we define $Z_j = g'_j(\\xki) - g'_j(x_0^k) + g'(x_0^k) - g'(\\xki)$ and follow a similar line of arguments. \n\\end{proof}","sha256_utf8":"0700c1015fee7f0ccef9252d589a91ef6bef2a498e92ab3abfe4fb7ba952a47e"},{"start":44805,"end":45419,"start_line":645,"end_line":655,"raw_tex":"\\begin{lemma}\n\\label{lemma: descent-Finite-nonsmooth}\nSuppose Assumptions~\\ref{assumption:f-cvx} and~\\ref{assumption:g-lip-finite} hold and the estimates $\\tgkz$, $\\tJkz$, $\\tgki$ and $\\tJki$ in Algorithm~\\ref{algo:SVR-PL} are constructed as in~\\eqref{eqn:finite-gk0}-\\eqref{eqn:J-svrg} respectively.\nThen for $k=1,\\ldots,K$ and $i=0,\\ldots,\\tau-1$, \n\\begin{eqnarray}\n\\Phi(\\xkie) \n&\\leq& \\Phi(\\xki) - \\frac{M - 2\\ell_f L_g}{2M^2}\\bigl\\|\\tcGki\\bigr\\|^2 \\nonumber \\\\\n&& +\\, 2\\ell_f \\bigl\\|\\tgki-g(\\xki)\\bigr\\| + \\frac{\\ell_f}{2L_g}\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|^2. \n\\label{eqn:descent-lemma}\n\\end{eqnarray}\n\\end{lemma}","sha256_utf8":"b1a7378f8a56a338f4070a6af87b277e590e74346f9db4140e5129603e4a7ffd"},{"start":45420,"end":47611,"start_line":656,"end_line":698,"raw_tex":"\\begin{proof}\nBy the definition of $\\Phi$ in~\\eqref{eqn:Phi-def} and Lemma~\\ref{lemma:upper-bound}, we have \n\\begin{eqnarray}\n\\Phi(\\xkie) & \\leq & f(g(\\xki) + g'(\\xki)(\\xkie-\\xki)) + \\frac{\\ell_f L_g}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2 + h(\\xkie) \\nonumber \\\\\n& = &  f\\bigl(\\tgki + \\tJki(\\xkie-\\xki)\\bigr) + h(\\xkie) + \\frac{M}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2\n -\\, \\frac{M - \\ell_f L_g}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2 \\nonumber \\\\\n& & + \\underbrace{f\\bigl(g(\\xki) + g'(\\xki)(\\xkie-\\xki)\\bigr)-f\\bigl(\\tgki + g'(\\xki)(\\xkie-\\xki)\\bigr)}_{T_1}\\nonumber\\\\\n& & + \\underbrace{f\\bigl(\\tgki + g'(\\xki)(\\xkie-\\xki)\\bigr)-f\\bigl(\\tgki + \\tJki(\\xkie-\\xki)\\bigr)}_{T_2} .\n\\label{eqn:descent-finite-0}\n\\end{eqnarray}\nAccording to~\\eqref{eqn:xkie-def}, we have\n\\[\n    f\\bigl(\\tgki + \\tJki(\\xkie-\\xki)\\bigr) + h(\\xkie) + \\frac{M}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2 ~\\leq~ f(\\tgki) + h(\\xki).\n\\]\nTherefore,\n\\begin{eqnarray}\n\\Phi(\\xkie) \n&\\leq & f(\\tgki) + h(\\xki) - \\frac{M-\\ell_f L_g}{2} \\|\\xkie-\\xki\\|^2 + T_1+ T_2 \\nonumber\\\\\n&\\leq & f(g(\\xki)) + h(\\xki) - \\frac{M-\\ell_f L_g}{2} \\|\\xkie-\\xki\\|^2 + T_1+ T_2  +  \\underbrace{f(\\tgki)-f(g(\\xki))}_{T_3}\\nonumber\\\\\n& = & \\Phi(\\xki)  - \\frac{M-\\ell_f L_g}{2} \\|\\xkie-\\xki\\|^2 + T_1+ T_2 +T_3. \n\\label{eqn:Phi-descent}\n\\end{eqnarray}\nBy the Lipschitz property of~$f$, we have\n\\begin{eqnarray*}\n    T_1 \\leq \\ell_f \\bigl\\|\\tgki-g(\\xki)\\bigr\\|, \\qquad\n    T_3 \\leq \\ell_f \\bigl\\|\\tgki-g(\\xki)\\bigr\\|,\n\\end{eqnarray*}\nand\n\\begin{eqnarray}\nT_2 &\\leq& \\ell_f\\bigl\\|\\bigl(\\tJki-g'(\\xki)\\bigr)(\\xkie-\\xki)\\bigr\\|\n~\\leq~ \\ell_f\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|\\cdot\\bigl\\|\\xkie-\\xki\\bigr\\| \\nonumber\\\\\n&\\leq& \\frac{\\ell_f}{2L_g}\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|^2 + \\frac{\\ell_f L_g}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2.\n\\label{eqn:descent-lemma-T2}\n\\end{eqnarray}\nCombining the bounds on $T_1$, $T_2$ and $T_3$ and the inequality~\\eqref{eqn:Phi-descent} yields\n\\begin{eqnarray*}\n    \\Phi(\\xkie) \n    &\\leq& \\Phi(\\xki) - \\left(\\frac{M}{2} - \\ell_f L_g\\right)\\bigl\\|\\xkie-\\xki\\bigr\\|^2 \\\\\n    && +\\, 2\\ell_f \\bigl\\|\\tgki-g(\\xki)\\bigr\\| + \\frac{\\ell_f}{2L_g}\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|^2, \n\\end{eqnarray*}\nwhich, upon noticing $\\tcGki=-M(\\xkie-\\xki)$, is equivalent to the\ndesired result. \n\\end{proof}","sha256_utf8":"dd83b907e22e8e702e22b32685d2351f3d058e6201b679db6c40554c9393be6d"},{"start":48643,"end":49276,"start_line":720,"end_line":730,"raw_tex":"\\begin{lemma}\n\t\\label{lemma:Lyapunov-finite-nonsmooth}\nSuppose Assumptions~\\ref{assumption:f-cvx} and~\\ref{assumption:g-lip-finite} hold and the estimates $\\tgkz$, $\\tJkz$, $\\tgki$ and $\\tJki$ in Algorithm~\\ref{algo:SVR-PL} are constructed as in~\\eqref{eqn:finite-gk0}-\\eqref{eqn:J-svrg} respectively.\nIn addition, we assume $M\\geq 4\\ell_f L_g$ and $\\gamma>0$.\nThen for each $k=1,\\ldots,K$,\n\\begin{equation}\\label{lm:Lyapunov-finite-nonsmooth-1}\n\t\\sum_{i=0}^{\\tau-1}\\E\\Bigl[\\bigl\\|\\cGki\\bigr\\|^2\\Bigr] \n    ~\\leq~  \\frac{R^k_0-R^k_\\tau}{\\gamma}  \n    ~=~ \\frac{\\E[\\Phi(x_0^k)]-\\E[\\Phi(x_0^{k+1})]}{\\gamma}.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"ba180d62d52f73f30e00ca7440f0f7186c78125145953cf0ddee7f4751bc83c7"},{"start":49277,"end":53422,"start_line":731,"end_line":799,"raw_tex":"\\begin{proof}\nFor the ease of notation, we write the stochastic Lyapunov function as\n$$R^k_i = \\E\\bigl[\\Phi(\\xki) + c_i\\|G_i^k\\|^2\\bigr],$$\nwhere\n\\begin{equation}\\label{eqn:Gki-def}\n    \\Gki = \\sum_{t=0}^{i-1}\\tcGkt = -M(\\xki - x_0^k).\n\\end{equation}\nIn particular, we have $G_{0}^k = -M(x_0^k-x_0^k) = 0$.\nMoreover, we have \n\\begin{eqnarray}\n\t\\E\\bigl[\\|G_{i+1}^k\\|^2\\bigr] \n    &=& \\E\\bigl[\\|G_{i}^k + \\tcGki\\|^2\\bigr]   \\nonumber \\\\\n    &\\leq& \\left(1+\\frac{1}{\\tau}\\right)\\E\\bigl[\\|G_{i}^k\\|^2\\bigr] + (1+\\tau)\\E\\bigl[\\|\\tcGki\\|^2\\bigr]. \n\\label{eqn:Gki-tcG}\n\\end{eqnarray}\nCombining Lemmas~\\ref{lemma:PG-finite-sum-nonsmooth} \nand~\\ref{lemma: descent-Finite-nonsmooth} yields\n\\begin{eqnarray*}\n\\E\\bigl[\\Phi(\\xkie)\\bigr] \n    &\\leq& \\E\\bigl[\\Phi(\\xki)\\bigr] - \\frac{M - 2\\ell_f L_g}{2M^2}\\E\\bigl[\\bigl\\|\\tcGki\\bigr\\|^2\\bigr] + \\left(\\frac{\\ell_f L_g}{\\sqrt{|\\cB_i^k|}}+\\frac{\\ell_f L_g}{2|\\cS_i^k|}\\right)\\E\\bigl[\\|\\xki-x_0^k\\|^2\\bigr] \\\\\n    &=& \\E\\bigl[\\Phi(\\xki)\\bigr] - \\frac{M - 2\\ell_f L_g}{2M^2}\\E\\bigl[\\bigl\\|\\tcGki\\bigr\\|^2\\bigr] + \\frac{1}{M^2}\\left(\\frac{\\ell_f L_g}{\\sqrt{|\\cB_i^k|}}+\\frac{\\ell_f L_g}{2|\\cS_i^k|}\\right)\\E\\bigl[\\|G^k_i\\|^2\\bigr],\n\\end{eqnarray*}\nwhere in the last equality we used~\\eqref{eqn:Gki-def}.\nAdding both sides of~\\eqref{eqn:Gki-tcG} to that of the above inequality \nand using the assumption $M\\geq 4\\ell_f L_g$, we obtain\n\\begin{eqnarray}\n\\E\\bigl[\\Phi(\\xkie)+c_{i+1}\\|G_{i+1}^k\\|^2\\bigr] \n    & \\leq & \\E\\bigl[\\Phi(\\xki)\\bigr] - \\left(\\frac{M-2\\ell_f L_g}{2M^2} - c_{i+1}(1+\\tau)\\right)\\E\\bigl[\\|\\tcGki\\|^2\\bigr] \\nonumber \\\\ \n    & & +\\left(\\frac{1}{M^2}\\left(\\frac{\\ell_f L_g}{\\sqrt{|\\cB_i^k|}}+\\frac{\\ell_f L_g}{2|\\cS_i^k|}\\right) + c_{i+1}\\left(1+\\frac{1}{\\tau}\\right)\\right)\\E\\bigl[\\|G_i^k\\|^2\\bigr] \\nonumber \\\\\n    & \\leq & \\E\\bigl[\\Phi(\\xki)\\bigr] - \\left(\\frac{1}{4M} - c_{i+1}(1+\\tau)\\right)\\E\\bigl[\\|\\tcGki\\|^2\\bigr] \\label{eqn:lyapunov-descent}\\\\ \n    & & +\\left(\\frac{1}{4M}\\left(\\frac{1}{\\sqrt{|\\cB_i^k|}}+\\frac{1}{2|\\cS_i^k|}\\right) + c_{i+1}\\left(1+\\frac{1}{\\tau}\\right)\\right)\\E\\bigl[\\|G_i^k\\|^2\\bigr]. \\nonumber\n\\end{eqnarray}\nNext, combining Lemma~\\ref{lemma:PG-finite-sum-nonsmooth} with Lemmas~\\ref{lemma:svr-f-new} yields\n\\begin{eqnarray*}\n    \\frac{M - \\ell_f L_g}{M^2}\\E\\bigl[\\|\\cGki\\|^2\\bigr]\n    &\\leq&\n    \\frac{2M+\\ell_f L_g}{M^2}\\E\\bigl[\\|\\tcGki\\|^2\\bigr] \n     + \\left( \\frac{2\\ell_f L_g}{\\sqrt{|\\cB_i^k|}} + \\frac{2\\ell_f L_g}{|\\cS_i^k|} \\right) \\E\\bigl[\\|\\xki-\\xkz\\|^2\\bigr].\n\\end{eqnarray*}\nUsing the equality $G^k_i=-M(\\xki-\\xkz)$ and the assumption $M\\geq 4\\ell_f L_g$, the above inequality implies\n\\begin{eqnarray}\n    \\frac{3}{4M}\\E\\bigl[\\|\\cGki\\|^2\\bigr]\n    &\\leq&\n    \\frac{9}{4M}\\E\\bigl[\\|\\tcGki\\|^2\\bigr] \n     +\\, \\frac{1}{2M} \\left( \\frac{1}{\\sqrt{|\\cB_i^k|}} + \\frac{1}{|\\cS_i^k|} \\right) \\E\\bigl[\\|G^k_i\\|^2\\bigr].\n    \\label{eqn:cG-bound}\n\\end{eqnarray}\nMultiplying both sides of~\\eqref{eqn:cG-bound} by $\\left(\\frac{1}{4M}-c_{i+1}(1+\\tau)\\right)\\big/\\frac{9}{4M}$, which is positive by the assumption $\\gamma>0$, and adding the resulting inequality to~\\eqref{eqn:lyapunov-descent}, we get\n\\begin{eqnarray}\n%\\label{lm:Lyapunov-finite-nonsmooth-2}\n\\E\\bigl[\\Phi(\\xkie)+c_{i+1}\\|G_{i+1}^k\\|^2\\bigr] \n&\\leq& \\E\\left[\\Phi(\\xki)+\\left( c_{i+1}\\left(1+\\frac{1}{\\tau}\\right)+ \\frac{1}{3M\\sqrt{|\\cB_i^k|}} + \\frac{1}{5M|\\cS_i^k|}\\right)\\|G_i^k\\|^2\\right] \\nonumber\\\\\n\t&& -\\frac{1}{3}\\left(\\frac{1}{4M} - c_{i+1}(1+\\tau)\\right)\\E[\\|\\cGki\\|^2].\\nonumber\n\\end{eqnarray}\nNow, using the definitions in~\\eqref{eqn:lyapunov-def} and~\\eqref{eqn:ct-finite-nonsmooth}, the above inequality is the same as\n\\[\n\t\\frac{1}{3}\\left(\\frac{1}{4M} - c_{i+1}(1+\\tau)\\right)\\E\\bigl[\\|\\cGki\\|^2\\bigr] ~\\leq~  R_i^k  - R_{i+1}^k.\n\\]\nRecalling the definition of $\\gamma$ and summing up the above inequality over~$i$ from~$0$ to~$\\tau-1$, we get\n\\[\n\t\\gamma\\sum_{i=0}^{\\tau-1}\\E\\bigl[\\|\\cGki\\|^2\\bigr]\n    ~\\leq~ R_0^k-R_\\tau^k\n    ~=~\\E\\bigl[\\Phi(x_0^k)\\bigr] - \\E\\bigl[\\Phi(x_\\tau^k)\\bigr] ,\n\\]\nwhere the last equality is due to the observations that \n$c_\\tau=0$ and $G^k_0=0$. \nFinally, dividing both sides by $\\gamma$ and using $x^{k+1}_0 = x^k_\\tau$\ngive the desired result.\n\\end{proof}","sha256_utf8":"6a9b406476689a8b9fdd00c5034abed67182b65f1fb190927c065a81ba7249a1"},{"start":53630,"end":53927,"start_line":803,"end_line":810,"raw_tex":"\\begin{lemma}\n\\label{lemma:gamma-nonsmooth}\n    If we choose \n    $\\tau = \\bigl\\lceil\\frac{1}{2}N^{1/5}-1\\bigr\\rceil$,\n    $|\\cB_i^k| = \\bigr\\lceil 4N^{4/5}\\bigr\\rceil$ and\n    $|\\cS_i^k| = \\bigr\\lceil N^{2/5}\\bigr\\rceil$ for $i=1,\\ldots,\\tau-1$,\n    then $\\gamma  \\geq \\frac{1}{15M}$.\n\\end{lemma}","sha256_utf8":"a7addcdd88b446ce79dde6dfb7afeafaf7b503398ba0c84556b6194bcd511590"},{"start":53928,"end":55143,"start_line":811,"end_line":834,"raw_tex":"\\begin{proof}\nTo simplify notation, let $B=|\\Bki|$ and $S=|\\Ski|$ for $i=1,\\ldots,\\tau-1$.\nFrom \\eqref{eqn:ct-finite-nonsmooth}, we deduce\n\\[\n    (c_i+C) = (c_{i+1}+C)\\left(1+\\frac{1}{\\tau}\\right), \\qquad\n        \\mbox{where}\\quad C = \\frac{\\tau}{3M\\sqrt{B}} + \\frac{\\tau}{5MS}.\n\\] \nConsequently, with $c_\\tau=0$, we have for all $i=1,\\ldots,\\tau$, \n\\[\n    c_i ~=~ (c_\\tau + C)\\left(1+\\frac{1}{\\tau}\\right)^{\\tau-i}\\!\\! -C\n    ~\\leq~  C\\left(1+\\frac{1}{\\tau}\\right)^{\\tau} - C\n    ~\\leq~ Ce -C ~=~ C(e-1),\n\\] \nwhere the last inequality is due to the fact that \n$(1+1/\\tau)^\\tau\\leq e$ with $e$ is Euler's number (the basis of natural logarithm). Therefore, \n\\begin{eqnarray*}\n\t\\gamma  \n    & =  & \\min_{0\\leq i\\leq \\tau-1}\\frac{1}{3}\\left(\\frac{1}{4M} - c_{i+1}(1+\\tau)\\right)\\\\\n    & \\geq & \\frac{1}{3}\\left(\\frac{1}{4M} - C(e-1)(1+\\tau)\\right)\\\\\n    & = & \\frac{1}{3M}\\left(\\frac{1}{4} - \\left( \\frac{1}{3\\sqrt{B}} + \\frac{1}{5S}\\right)(e-1)\\tau(1+\\tau)\\right) \\\\\n    & \\geq & \\frac{1}{3M}\\left(\\frac{1}{4} - \\left( \\frac{1}{3\\sqrt{B}} + \\frac{1}{5S}\\right)2(1+\\tau)^2\\right).\n\\end{eqnarray*}\nFinally, setting $\\tau=\\frac{1}{2}N^{1/5}-1$, $B=4N^{4/5}$ and $S=N^{2/5}$ yields $\\gamma\\geq\\frac{1}{15M}$.\n\\end{proof}","sha256_utf8":"ed4ade629af30980d769eaea12bddf03e2d616765306d73fb94f6fb9c0868a2b"},{"start":56382,"end":57443,"start_line":852,"end_line":877,"raw_tex":"\\begin{proof}\n\tSumming up the inequality \\eqref{lm:Lyapunov-finite-nonsmooth-1} \n    over~$k$ from~$1$ to~$K$ and using the fact $\\Phi(x^K_\\tau)>\\Phi_*$, we get\n\t\\begin{eqnarray*}\n\t\t\\sum_{k=1}^K\\sum_{i=0}^{\\tau-1}\\E\\bigl[\\|\\cGki\\|^2\\bigr] ~\\leq~ \\frac{\\Phi(x_0^1) - \\Phi(x^*)}{\\gamma}.\n\t\\end{eqnarray*}\n\tBy the random choice of the output $x_{i^*}^{k^*}$, we can get the inequality \\eqref{eqn:cvg-svr-pl-nonsmooth-1}.\n\t\n    To get an $\\epsilon$-stationary point in expectation, we need to set $K\\tau = \\cO(\\epsilon^{-1})$, which implies \n\\[\n    K=\\cO(\\tau^{-1}\\epsilon^{-1})=\\cO(N^{-1/5}\\epsilon^{-1}).\n\\] \nConsequently, the sample complexity of the component mappings (the $g_i$'s) is \n\\[\n    KN + K\\tau B \n    ~=~ \\cO(N^{-1/5}\\epsilon^{-1})\\cdot N + \\cO(\\epsilon^{-1}) \\cdot 4 N^{4/5} \n    ~=~ \\cO(N+N^{4/5}\\epsilon^{-1}).\n\\]\nand the sample complexity for the component Jacobians is\n\\[\n    KN + K\\tau S \n    ~=~ \\cO(N^{-1/5}\\epsilon^{-1})\\cdot N + \\cO(\\epsilon^{-1}) \\cdot N^{2/5} \n    ~=~ \\cO(N+N^{4/5}\\epsilon^{-1}).\n\\]\nThis completes the proof.\n\\end{proof}","sha256_utf8":"6d371675506c70c86eae9494785e8620d0e4fb79d308ff35c4baefa84aa5173a"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"非光滑有限平均算法复杂度","source_units":[{"start":55357,"end":56381,"start_line":838,"end_line":851,"raw_tex":"\\begin{theorem}\n\t\\label{theorem:svrg}\n    Suppose Assumptions~\\ref{assumption:f-cvx}, \\ref{assumption:phi-lowerbound}  and~\\ref{assumption:g-lip-finite} hold for problem~\\eqref{eqn:composite-finite}. Let the estimates $\\tgkz$, $\\tJkz$, $\\tgki$ and $\\tJki$ in Algorithm~\\ref{algo:SVR-PL} be given in~\\eqref{eqn:finite-gk0}-\\eqref{eqn:J-svrg} respectively. If we choose $M\\geq 4\\ell_f L_g$ and $\\tau = \\bigl\\lceil\\frac{1}{2}N^{1/5}-1\\bigr\\rceil$, and\n\\[\n    |\\cB_i^k| = \\bigr\\lceil 4N^{4/5}\\bigr\\rceil, \\qquad\n    |\\cS_i^k| = \\bigr\\lceil N^{2/5}\\bigr\\rceil, \\qquad \n    i=1,\\ldots,\\tau-1,  \\qquad k=1,\\ldots,K,\n\\]\nthen the output of Algorithm~\\ref{algo:SVR-PL} satisfies\n\\begin{equation}\\label{eqn:cvg-svr-pl-nonsmooth-1}\n    \\E\\bigl[\\|\\cG(x_{i^*}^{k^*})\\|^2\\bigr] ~\\leq~ \\frac{15M\\bigl(\\Phi(x_0^1) - \\Phi_*\\bigr)}{K\\tau}.\n\\end{equation}\nTo get an $\\epsilon$-stationary point in expectation, the total sample complexity for the component mappings $g_j$ and their Jacobians are both $\\cO(N+N^{4/5}\\epsilon^{-1})$.\n\\end{theorem}","sha256_utf8":"11b3060c20099f1e4f46a105b48fdbb826ca8842f202ccc267fcc511ca0e8344"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"主定理完整证明","source_units":[{"start":56382,"end":57443,"start_line":852,"end_line":877,"raw_tex":"\\begin{proof}\n\tSumming up the inequality \\eqref{lm:Lyapunov-finite-nonsmooth-1} \n    over~$k$ from~$1$ to~$K$ and using the fact $\\Phi(x^K_\\tau)>\\Phi_*$, we get\n\t\\begin{eqnarray*}\n\t\t\\sum_{k=1}^K\\sum_{i=0}^{\\tau-1}\\E\\bigl[\\|\\cGki\\|^2\\bigr] ~\\leq~ \\frac{\\Phi(x_0^1) - \\Phi(x^*)}{\\gamma}.\n\t\\end{eqnarray*}\n\tBy the random choice of the output $x_{i^*}^{k^*}$, we can get the inequality \\eqref{eqn:cvg-svr-pl-nonsmooth-1}.\n\t\n    To get an $\\epsilon$-stationary point in expectation, we need to set $K\\tau = \\cO(\\epsilon^{-1})$, which implies \n\\[\n    K=\\cO(\\tau^{-1}\\epsilon^{-1})=\\cO(N^{-1/5}\\epsilon^{-1}).\n\\] \nConsequently, the sample complexity of the component mappings (the $g_i$'s) is \n\\[\n    KN + K\\tau B \n    ~=~ \\cO(N^{-1/5}\\epsilon^{-1})\\cdot N + \\cO(\\epsilon^{-1}) \\cdot 4 N^{4/5} \n    ~=~ \\cO(N+N^{4/5}\\epsilon^{-1}).\n\\]\nand the sample complexity for the component Jacobians is\n\\[\n    KN + K\\tau S \n    ~=~ \\cO(N^{-1/5}\\epsilon^{-1})\\cdot N + \\cO(\\epsilon^{-1}) \\cdot N^{2/5} \n    ~=~ \\cO(N+N^{4/5}\\epsilon^{-1}).\n\\]\nThis completes the proof.\n\\end{proof}","sha256_utf8":"6d371675506c70c86eae9494785e8620d0e4fb79d308ff35c4baefa84aa5173a"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"有限平均复合优化问题","source_units":[{"start":4649,"end":4750,"start_line":120,"end_line":123,"raw_tex":"\\begin{equation}\n\\label{eqn:composite-opt}\n\\minimize_{x\\in\\R^n} \\quad f(g(x)) + h(x), \n\\end{equation}","sha256_utf8":"42005a3c99e4419462c4e6133b20abf0613240db98f5b2fa9c9d27deb3f77fc8"},{"start":4751,"end":5534,"start_line":124,"end_line":133,"raw_tex":"where $f:\\R^m\\rightarrow\\R$ is a convex and possibly nonsmooth function, $g:\\R^n\\rightarrow\\R^m$ is a smooth mapping (vector-valued function), and $h:\\R^n\\rightarrow\\R$ is a convex and lower-semicontinuous function. \nAlthough both $f$ and $h$ are convex, the problem is in general nonconvex due to the composition of~$f$ and~$g$. \nIn addition, we assume that~$g$ is either the average of finite number of component mappings, i.e., $g(x) = \\frac{1}{N}\\sum_{i=1}^N g_i(x)$, \nor the expectation of a family of random component mappings, i.e.,\n$g(x) = \\E_\\xi [g_\\xi(x)]$ where~$\\xi$ is a random variable.\nMore explicitly, we consider the problems\n\\begin{equation}\n\\label{eqn:composite-finite}\n\\minimize_{x\\in\\R^n} \\quad f\\biggl(\\frac{1}{N}\\sum_{i=1}^N g_i(x)\\biggr) + h(x)\n\\end{equation}","sha256_utf8":"f68b01a076e28fa37948ffde1c4cf7da721b5f9ff3dea64df31d6c82f1341904"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"复合目标函数","source_units":[{"start":25642,"end":25727,"start_line":345,"end_line":347,"raw_tex":"\\begin{equation}\\label{eqn:Phi-def}\n\\Phi(x) \\triangleq f(g(x)) + h(x),\n\\end{equation}","sha256_utf8":"96316e676f566396a1b55cfa98c6417e77b92d230f45c4c0e7f9a1d734ed5ca3"}]},{"id":"S005","semantic_type":"assumption","kind":"assumption","title":"外层函数与正则项假设","source_units":[{"start":25996,"end":26316,"start_line":351,"end_line":357,"raw_tex":"\\begin{assumption}\\label{assumption:f-cvx} \n    The function $f:\\R^m\\to\\R\\cup\\{+\\infty\\}$ is convex and $\\ell_f$-Lipschitz continuous, i.e.,\n    \\[\n        |f(u)-f(v)|\\leq\\ell_f\\|u-v\\|, \\qquad \\forall\\, u, v\\in \\R^m.\n    \\]\nThe function $h:\\R^n\\to\\R\\cup\\{+\\infty\\}$ is convex and lower semi-continuous. \n\\end{assumption}","sha256_utf8":"2717eac3588454fc3854c69ee35e9a3b61654bfd501510b0590f1232234c016a"}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","title":"映射及雅可比光滑性","source_units":[{"start":26318,"end":26744,"start_line":358,"end_line":365,"raw_tex":"\\begin{assumption}\\label{assumption:g-lip}\n    The vector mapping $g:\\R^n\\to\\R^m$ is $\\ell_g$-Lipschitz continuous and its Jacobian $g':\\R^n\\to\\R^{m\\times n}$ is $L_g$-Lipschitz continuous, i.e., \n    \\begin{align*}\n        \\|g(x) - g(y)\\| &\\leq \\ell_g \\|x-y\\|, \\\\\n        \\|g'(x) - g'(y)\\| &\\leq L_g \\|x-y\\|, \n    \\end{align*}\nfor all $x,y\\in\\dom h$, where $\\|\\cdot\\|$ for matrices denotes the spectral norm.\n\\end{assumption}","sha256_utf8":"e09057ca41e932c40bf3ed83b262bcaca25a2b65ab31807865ba5dd3daca8d34"}]},{"id":"S007","semantic_type":"derived_fact","kind":"derived_fact","title":"一阶线性化余项界","source_units":[{"start":26745,"end":26967,"start_line":366,"end_line":369,"raw_tex":"A direct consequence of the Lipschitz condition on $g'$ in Assumption~\\ref{assumption:g-lip} is \n\\begin{equation}\\label{eqn:g-quadratic-bd}\n    \\bigl\\|g(x)-g(y)-g(y)'(x-y)\\bigr\\| \\leq \\frac{L_g}{2}\\|x-y\\|^2.\n\\end{equation}","sha256_utf8":"a6a6f7fdf7dfb9e1f2c09bd92491b8b2ba099bdc6c10c033876ea85e9e75fbac"},{"start":26968,"end":27027,"start_line":370,"end_line":370,"raw_tex":"(See, e.g., \\cite[Theorem~3.2.12]{OrtegaRheinboldt1970}.) \n","sha256_utf8":"16bcaada096e87eea80772610f12443737cda83d2da685e88a604883873331a0"}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","title":"目标函数下界","source_units":[{"start":27151,"end":27287,"start_line":372,"end_line":374,"raw_tex":"\\begin{assumption}\\label{assumption:phi-lowerbound}\n\tThere exists $\\Phi_*$ such that  $\\Phi_* = \\inf_x\\Phi(x)>-\\infty$.\n\\end{assumption}","sha256_utf8":"266930316ba3a8f3a29d6750ca4f334f7bffe08732cbe48a7a14704a0905bf3f"}]},{"id":"S009","semantic_type":"algorithm","kind":"algorithm","title":"随机方差缩减近端线性框架","source_units":[{"start":29252,"end":30182,"start_line":403,"end_line":419,"raw_tex":"\\begin{algorithm2e}[t]\n    \\caption{Stochastic variance-reduced prox-linear algorithm} %(framework)}\n\t\\label{algo:SVR-PL} \n\t{\\bf input:} initial point $x^1_0$, $M>0$, number of outer and inner iterations $K$ and $\\tau$.\\\\\n\t\\For{$k = 1,\\ldots,K$}{ \n\t\t\\For{$i = 0,\\ldots,\\tau-1$}{\n            \\eIf{$i==0$}{\n                compute $\\tgk_0$ and $\\tJk_0$ using \\emph{large} batches $\\Bkz$ and $\\Skz$ respectively. \\label{alg-line:gk0-Jk0}\n            }{\n                compute $\\tgki$ and $\\tJki$ using \\emph{small} batches $\\Bki$ and $\\Ski$ respectively. \\label{alg-line:gki-Jki}\n            }\n            $\\xkie = \\displaystyle\\argmin_x ~\\textstyle\\Bigl\\{f\\bigl(\\tgki + \\tJki(x-\\xki)\\bigr) + h(x) + \\frac{M}{2}\\|x-\\xki\\|^2\\Bigr\\}$. \\label{alg-line:argmin}\n\t\t}\n\t\tSet $x_0^{k+1} = x^k_{\\tau}.$\n\t}  \n    {\\bf output:} choose $x^{k^*}_{i^*}$ from $\\{\\xki\\}_{i = 0,\\ldots,\\tau-1}^{k = 1,\\ldots,K}$ uniformly at random.\n\\end{algorithm2e}","sha256_utf8":"3457498daccf2115e13eff2616988e82e479b1b40f0b810840d9aa5a72508ead"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"真实梯度映射与驻点","source_units":[{"start":31117,"end":31393,"start_line":429,"end_line":434,"raw_tex":"In order to characterize the sample complexity of different algorithms, we first define what is an $\\epsilon$-stationary point.\nFor any $x\\in\\dom h$, we define the proximal point\n\\[\nx_+\\triangleq\n\\argmin_y \\left\\{ f\\bigl(g(x)+g'(x)(y-x))+h(y) + \\frac{M}{2}\\|y-x\\|^2\\right\\}\n\\]","sha256_utf8":"fb94b814fd1879148d9827fe296e92fb0b24f2aeb594f284f47f8336b40590bd"},{"start":31394,"end":31522,"start_line":435,"end_line":439,"raw_tex":"and the composite gradient mapping at~$x$,\n\\begin{equation}\n\\label{eqn:grad-mapping}\n\\cG_M(x)\\triangleq M(x-x_+).\n\\end{equation}","sha256_utf8":"31a24091476a417eaf237066f728518198185e11576461f24a3c43eebb08588f"},{"start":31523,"end":31676,"start_line":440,"end_line":440,"raw_tex":"Given any $\\epsilon>0$, we call $\\bar{x}$ an $\\epsilon$-stationary point of~$\\Phi$ defined in~\\eqref{eqn:Phi-def} if $\\|\\cG_M(\\bar{x})\\|^2\\leq\\epsilon$.\n","sha256_utf8":"e36eb590fa6b906b6e88fdc1ef094473f305514f8845204302252ce4fa7e7749"},{"start":32098,"end":32250,"start_line":443,"end_line":443,"raw_tex":"To simplify notation, we will omit the subscript~$M$ (which is a constant throughout this paper) and denote the composite gradient mapping as $\\cG(x)$.\n","sha256_utf8":"7ae655c7ec57db8650c7c97227b77a46d808ffb2c879224a06849c2d116e161f"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"期望驻点与采样复杂度","source_units":[{"start":32251,"end":32588,"start_line":445,"end_line":449,"raw_tex":"The sample complexity of a randomized algorithm, such as Algorithm~\\ref{algo:SVR-PL}, is the total number of evaluations of the component mappings $g_i$ or $g_\\xi$ and their Jacobians required in order to output some $\\bar{x}$ satisfying\n\\begin{equation}\n\\label{eqn:eps-solu}\n\\E\\bigl[\\|\\cG(\\bar{x})\\|^2\\bigr]\\leq \\epsilon,\n\\end{equation}","sha256_utf8":"75a58b3fa43139100f9d4ba66caa9bacade1803fec12ad186316fecc5b59dd42"},{"start":32589,"end":32690,"start_line":450,"end_line":450,"raw_tex":"where the expectation is taken over all the random samplings during the iterations of the algorithm.\n","sha256_utf8":"90845f2f1b497fa711349a2764393aff204510e8ec5ab33408ca6088c938d166"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"近似更新与精确近端点","source_units":[{"start":32691,"end":33132,"start_line":452,"end_line":456,"raw_tex":"Notice that the proximal point $x_+$ used in the definition of $\\cG(x)$ is computed with~$g(x)$ and~$g'(x)$, which can be very costly if not impossible to evaluate. \nIn Algorithm~\\ref{algo:SVR-PL}, the proximal point $\\xkie$ is computed using the estimates $\\tgki$ and $\\tJki$, i.e.,\n\\begin{equation}\\label{eqn:xkie-def}\n    \\xkie = \\argmin_x \\left\\{f\\bigl(\\tgki + \\tJki(x-\\xki)\\bigr) + h(x) + \\frac{M}{2}\\|x-\\xki\\|^2\\right\\}.\n\\end{equation}","sha256_utf8":"81ca8a65108a609323817def6c6a22da97a53fd758411d63368bd6ffe84933cd"},{"start":33133,"end":33275,"start_line":457,"end_line":460,"raw_tex":"This leads to a convenient approximation, \n\\begin{equation}\\label{eqn:approx-gm}\n\\widetilde{\\cG}(\\xki)\\triangleq M(\\xki-\\xkie),\n\\end{equation}","sha256_utf8":"74cf06153a1f8a7f1fc08dc54d4b6777bdbc43b18c5f28d4e17d0c07553139a4"},{"start":33276,"end":33535,"start_line":461,"end_line":465,"raw_tex":"of the true gradient mapping $\\cGki= M(\\xki-\\hat{x}^k_{i+1})$, where \n\\begin{equation}\\label{eqn:x-hat-def}\n    \\hat{x}^k_{i+1} %= (\\xki)_+\n    = \\argmin_x \\left\\{f\\bigl(g(\\xki) + g'(\\xki)(x-\\xki)\\bigr) + h(x) + \\frac{M}{2}\\|x-\\xki\\|^2\\right\\}.\n\\end{equation}","sha256_utf8":"ff3349cd64fc4f75f1bd4f58429117f4b4431408255699fe562e7807747708d5"}]},{"id":"S013","semantic_type":"assumption","kind":"assumption","title":"有限分量光滑性假设","source_units":[{"start":38583,"end":39297,"start_line":557,"end_line":567,"raw_tex":"In this section, we consider the composite finite-average problem~\\eqref{eqn:composite-finite} with nonsmooth~$f$ and smooth~$g_i$'s.\nIn particular, we replace Assumption~\\ref{assumption:g-lip} with the following more structured one, which implies Assumption~\\ref{assumption:g-lip}.\n\\begin{assumption}\n\t\\label{assumption:g-lip-finite}\n    For each $i = 1,\\ldots,N$, the mapping $g_i:\\R^n\\to\\R^m$, is $\\ell_{g,i}$-Lipschitz continuous and its Jacobian matrix $g'_i:\\R^n\\to\\R^{m\\times n}$ is $L_{g,i}$-Lipschitz continuous. Namely, \n\\begin{align*}\n\\|g_i(x) - g_i(x)\\| & \\leq\\ell_{g,i}\\|x-y\\|, \\\\\n\\|g'_i(x) - g'_i(x)\\| & \\leq L_{g,i}\\|x-y\\|,\n\\end{align*}\nfor all $x,y\\in\\dom h$ and $i = 1,\\ldots,N$. \n\\end{assumption}","sha256_utf8":"df2c8485978161c52ada1135fc83c09779bcb1259f2e5279825b3cc02e7b39bd"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"均方光滑常数及其作用","source_units":[{"start":39298,"end":39776,"start_line":568,"end_line":573,"raw_tex":"A direct consequence of this assumption is that $g$ is $\\big(\\frac{1}{N}\\sum_{i}^{N}\\ell_{g,i}\\big)$-Lipschitz continuous and $g'$ is $\\big(\\frac{1}{N}\\sum_{i}^{N}L_{g,i}\\big)$-Lipschitz continuous. Due to the root-mean square inequality $\\frac{z_1+...+z_N}{N}\\leq \\sqrt{\\frac{z_1^2+...+z_N^2}{N}}$, \nWe define \n\\begin{equation}\\label{eqn:ms-Lip-constants}\n\\ell_g = \\sqrt{\\frac{1}{N}\\sum_{i}^{N}\\ell_{g,i}^2}, \\qquad\nL_g = \\sqrt{\\frac{1}{N}\\sum_{i}^{N}L_{g,i}^2},\n\\end{equation}","sha256_utf8":"bbc7b328c13b8d54cdbd099ed6690c7e9de04b77958af70b67389d75f58a4d38"},{"start":39777,"end":39890,"start_line":574,"end_line":575,"raw_tex":"which can serve as the Lipschitz constants of $g$ and $g'$ respectively\nas in Assumption~\\ref{assumption:g-lip}.\n","sha256_utf8":"cf08840fb6c5b782e98582727fe490dcd245408776ff22ab31f8a8e7a3d048cc"}]},{"id":"S015","semantic_type":"algorithm","kind":"algorithm","title":"阶段起点全批量估计","source_units":[{"start":39892,"end":40308,"start_line":578,"end_line":584,"raw_tex":"In this case, we construct the estimates $\\tgkz$ and $\\tJkz$ using the full batch. In other words, we let $\\Bkz=\\Skz=\\{1,2,\\ldots,N\\}$ and replace Line~\\ref{alg-line:gk0-Jk0} in Algorithm~\\ref{algo:SVR-PL} with\n\\begin{eqnarray}\n\\tgkz &=& g(\\xkz) = \\frac{1}{N}\\sum_{i=1}^N g_i(\\xkz), \n    \\label{eqn:finite-gk0} \\\\\n    \\tJkz &=& g'(\\xkz) = \\frac{1}{N}\\sum_{i=1}^N g'_i(\\xkz).\n    \\label{eqn:finite-Jk0}\n\\end{eqnarray}","sha256_utf8":"5ec6f13b6fc713ccdd738dbe1dead01d926b5aad42b7fd8be67c16a95ea13616"}]},{"id":"S016","semantic_type":"algorithm","kind":"algorithm","title":"一阶修正方差缩减估计","source_units":[{"start":40309,"end":40844,"start_line":585,"end_line":593,"raw_tex":"For $i>0$, we sample with replacement from $\\{1,2,\\ldots,N\\}$ to obtain smaller sets $\\Bki$ and $\\Ski$\n(whose cardinalities will be determined later), and\napply the following construction:\n\\begin{eqnarray}\n    \\tgki &=& \\frac{1}{|\\cB_i^k|}\\sum_{j\\in\\cB_i^k} \\Bigl(g_j(\\xki) - g_j(x_0^k) - g_j'(x_0^k)(\\xki-x_0^k)\\Bigr) + g(x_0^k) + g'(x_0^k)(\\xki-x_0^k),\n          \\label{eqn:g-svrg-correction} \\\\\n    \\tJki &=& \\frac{1}{|\\cS_i^k|}\\sum_{j\\in\\cS_i^k} \\Bigl(g'_j(\\xki) - g'_j(x_0^k)\\Bigr) + g'(x_0^k). \n\\label{eqn:J-svrg} \n\\end{eqnarray}","sha256_utf8":"605bc2ba74bd07b84db21a0a7bc2ae6e34fcef15899d366c5064a2b41722aacd"}]},{"id":"S017","semantic_type":"definition","kind":"definition","title":"随机李雅普诺夫函数","source_units":[{"start":47614,"end":47936,"start_line":700,"end_line":705,"raw_tex":"Recall the definition that $\\tcGki: = M(\\xki-\\xkie)$. \nIn order to complete the convergence analysis, we define a stochastic Lyapunov function\n\\begin{equation}\\label{eqn:lyapunov-def}\nR_i^k = \\E\\left[\\Phi(\\xki) + c_i\\biggl\\|\\sum_{t=0}^{i-1}\\tcGkt\\biggr\\|^2\\right],\n\\quad k=1,\\ldots,K, \\quad i=0,\\ldots,\\tau,\n\\end{equation}","sha256_utf8":"ae265862a2bc955305c72d62240f503b11f1244f30439c6f0151351a269e4f3f"}]},{"id":"S018","semantic_type":"definition","kind":"definition","title":"李雅普诺夫系数递推","source_units":[{"start":47937,"end":48238,"start_line":706,"end_line":711,"raw_tex":"where the coefficients $c_i$ for $i=0,1,\\ldots,\\tau$ are obtained through the recursion:\n\\begin{eqnarray}\nc_{\\tau} &=& 0, \\nonumber \\\\\nc_i &=& c_{i+1}\\left(1+\\frac{1}{\\tau}\\right) + \\frac{1}{3M\\sqrt{|\\Bki|}} + \\frac{1}{5M|\\Ski|}, \\quad i=\\tau-1,\\ldots,0.\n\\label{eqn:ct-finite-nonsmooth}\n\\end{eqnarray}","sha256_utf8":"32f287cb6aa8ea59180524b52739157584ea5f04390dc5ab3aed5bdf8e25e0ce"},{"start":48239,"end":48302,"start_line":712,"end_line":712,"raw_tex":"(Our choices or $|\\Bki|$ and $|\\Ski|$ will not depend on~$k$.)\n","sha256_utf8":"dc6e4606c83ed14bc32ac4afc9e1bf760be338eaafc8f020a758dfb1a3db7b8b"}]},{"id":"S019","semantic_type":"definition","kind":"definition","title":"统一下降系数","source_units":[{"start":48302,"end":48456,"start_line":713,"end_line":716,"raw_tex":"In addition, we define the following constant\n\\[\n   \\gamma \\triangleq \\min_{0\\leq i\\leq \\tau-1} \\frac{1}{3}\\left(\\frac{1}{4M} - c_{i+1}(1+\\tau)\\right).\n\\]","sha256_utf8":"5f46fb9bf3143fc43ef001539b193caaf6b1b28b27b757c381a0b269b8236210"}]},{"id":"S020","semantic_type":"notation","kind":"notation","title":"向上取整记号","source_units":[{"start":53561,"end":53630,"start_line":802,"end_line":802,"raw_tex":"We use $\\lceil\\cdot\\rceil$ to denote the nearest integer from above.\n","sha256_utf8":"0c1f09076152242b4527a265df234fedbc642f2959b5efb4bb8e26beb19c01ac"}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","title":"复合函数线性模型上界","source_units":[{"start":27345,"end":27604,"start_line":376,"end_line":382,"raw_tex":"\\begin{lemma}\\label{lemma:upper-bound} \nSuppose Assumption \\ref{assumption:f-cvx} and \\ref{assumption:g-lip} hold, then \nfor any $x,y\\in\\dom h$,\n\\begin{equation}\n    f(g(x)) \\leq f(g(y) + g'(y)(x-y)) + \\frac{\\ell_f L_g}{2}\\|x-y\\|^2.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"ba953ab0e40f1ecf686b76eb009d0e258340a25879f14f1ff67a5b8a4ae2b304"}]},{"id":"S022","semantic_type":"proof","kind":"proof","title":"线性模型上界完整证明","source_units":[{"start":27605,"end":28154,"start_line":383,"end_line":393,"raw_tex":"\\begin{proof} \n    By the Lipschitz continuity of $f$ and $g'$, we have\n\t\\begin{eqnarray*}\n    f(g(x)) \n    &=&  f\\bigl(g(y) + g'(y)(x-y)\\bigr) + f(g(x)) - f\\bigl(g(y) + g'(y)(x-y)\\bigr)\\\\\n    & \\leq &  f\\bigl(g(y) + g'(y)(x-y)\\bigr) + \\bigl|f(g(x)) - f\\bigl(g(y) + g'(y)(x-y)\\bigr)\\bigr|\\\\\n\t&\\leq &f\\bigl(g(y) + g'(y)(x-y)\\bigr) + \\ell_f\\bigl\\|g(x) - g(y) - g'(y)(x-y)\\bigr\\|\\\\\n\t& \\leq & f\\bigl(g(y) + g'(y)(x-y)\\bigr) + \\frac{\\ell_f L_g}{2}\\|x-y\\|^2,\n\t\\end{eqnarray*}\n    where the last inequality is due to~\\eqref{eqn:g-quadratic-bd}.\n\\end{proof}","sha256_utf8":"5a9506f597d73e66f9aea0c3c4cc9a869e226761982517c10c8d000e6350d04a"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"以外层利普希茨界控制余项","source_units":[{"start":27620,"end":28143,"start_line":384,"end_line":392,"raw_tex":"    By the Lipschitz continuity of $f$ and $g'$, we have\n\t\\begin{eqnarray*}\n    f(g(x)) \n    &=&  f\\bigl(g(y) + g'(y)(x-y)\\bigr) + f(g(x)) - f\\bigl(g(y) + g'(y)(x-y)\\bigr)\\\\\n    & \\leq &  f\\bigl(g(y) + g'(y)(x-y)\\bigr) + \\bigl|f(g(x)) - f\\bigl(g(y) + g'(y)(x-y)\\bigr)\\bigr|\\\\\n\t&\\leq &f\\bigl(g(y) + g'(y)(x-y)\\bigr) + \\ell_f\\bigl\\|g(x) - g(y) - g'(y)(x-y)\\bigr\\|\\\\\n\t& \\leq & f\\bigl(g(y) + g'(y)(x-y)\\bigr) + \\frac{\\ell_f L_g}{2}\\|x-y\\|^2,\n\t\\end{eqnarray*}\n    where the last inequality is due to~\\eqref{eqn:g-quadratic-bd}.\n","sha256_utf8":"843d37a47870ee542d427edfd78ead3cf170bcd94cde6596cbf6b66044ef3de5"}]},{"id":"S024","semantic_type":"lemma","kind":"lemma","title":"真实与近似梯度映射比较","source_units":[{"start":33956,"end":34421,"start_line":469,"end_line":478,"raw_tex":"\\begin{lemma}\n\\label{lemma:PG-finite-sum-nonsmooth}\nUnder Assumptions \\ref{assumption:f-cvx} and \\ref{assumption:g-lip}, \nthe iterates generated by Algorithm~\\ref{algo:SVR-PL} satisfy\n\\begin{eqnarray*}\n    \\frac{M - \\ell_f L_g}{M^2}\\bigl\\|\\cG(\\xki)\\bigr\\|^2 \n    &\\leq& \\frac{2M+\\ell_f L_g}{M^2}\\bigl\\|\\widetilde{\\cG}(\\xki)\\bigr\\|^2  \\\\\n    && \\,+\\, 4\\ell_f\\bigl\\|\\tgki-g(\\xki)\\bigr\\| + \\frac{2\\ell_f}{L_g}\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|^2.\n\\end{eqnarray*}\n\\end{lemma}","sha256_utf8":"6d1b523aad7e7ae8c813b92c5271d53547dd4df898db4774cf2eac6c5af5a721"}]},{"id":"S025","semantic_type":"proof","kind":"proof","title":"梯度映射比较完整证明","source_units":[{"start":34422,"end":38127,"start_line":479,"end_line":548,"raw_tex":"\\begin{proof}\nFor the ease of notation,  we denote \n\\begin{eqnarray}\n    F(x;\\xki) &=& f\\bigl(g(\\xki) + g'(\\xki)(x-\\xki)\\bigr), \\label{eqn:F-def}\\\\\n    \\widetilde{F}(x;\\xki) &=& f\\bigl(\\tgki + \\tJki(x-\\xki)\\bigr).\n    \\label{eqn:tF-def}\n\\end{eqnarray}\nSince both~$f$ and~$h$ are convex (Assumption~\\ref{assumption:f-cvx}),\nthe following two functions are $M$-strongly convex:\n\\begin{eqnarray}\n    && F(x;\\xki) + h(x) + \\frac{M}{2}\\|x-\\xki\\|^2, \\label{eqn:F+h+M} \\\\\n    && \\widetilde{F}(x;\\xki) + h(x) + \\frac{M}{2}\\|x-\\xki\\|^2.\n    \\label{eqn:tF+h+M} \n\\end{eqnarray} \nAccording to~\\eqref{eqn:x-hat-def} and~\\eqref{eqn:xkie-def}, $\\hat{x}_{i+1}^k$ and $\\xkie$ are the minimizers of these two functions respectively. \nTherefore\n\\begin{eqnarray*}\n    F(\\hat{x}^k_{i+1};\\xki) + h(\\hat{x}^k_{i+1}) + \\frac{M}{2}\\|\\hat{x}^k_{i+1}-\\xki\\|^2 \n\t& \\leq & F(\\xkie;\\xki) + h(\\xkie) + \\frac{M}{2}\\|\\xkie-\\xki\\|^2  \\\\\n    &&  - \\frac{M}{2}\\|\\hat{x}^k_{i+1}-\\xkie\\|^2,  \n\\end{eqnarray*} \nand\n\\begin{eqnarray*}\n    \\widetilde{F}(x^k_{i+1};\\xki) + h(x^k_{i+1}) + \\frac{M}{2}\\|x^k_{i+1}-\\xki\\|^2 \n& \\leq & \\widetilde{F}(\\hat{x}^k_{i+1};\\xki) + h(\\hat{x}^k_{i+1}) + \\frac{M}{2}\\|\\hat{x}^k_{i+1}-\\xki\\|^2 \\\\\n&& - \\frac{M}{2}\\|\\hat{x}^k_{i+1}-\\xkie\\|^2. \n\\end{eqnarray*} \nSumming the two inequalities above and rearranging the terms, we obtain\n\\begin{eqnarray}\n\t\\label{lm:PG-finite-sum-3}\n\tM\\|\\hat{x}^k_{i+1}-\\xkie\\|^2 \n    &\\leq& F(\\xkie;\\xki)-\\widetilde{F}(\\xkie;\\xki) \n     + \\widetilde{F}(\\hat{x}^k_{i+1};\\xki)-F(\\hat{x}^k_{i+1};\\xki).\n\\end{eqnarray}\nUsing the Lipschitz property of~$f$, we have \n\\begin{eqnarray*}\n\\bigl| F(\\xkie;\\xki)-\\widetilde{F}(\\xkie;\\xki)\\bigr| \n    &=& \\left| f\\bigl(g(\\xki)+g'(\\xki)(\\xkie-\\xki)\\bigr) - f\\bigl(\\tgki+\\tJki(\\xkie-\\xki)\\bigr)\\right| \\\\\n    & \\leq & \\ell_f\\left\\|\\bigl(g(\\xki) - \\tgki\\bigr) + \\bigl(g'(\\xki) - \\tJki\\bigr)\\bigl(\\xkie - \\xki\\bigr)\\right\\|\\\\\n\t& \\leq & \\ell_f\\left(\\bigl\\|\\tgki - g(\\xki)\\bigr\\| + \\bigl\\|\\tJki - g'(\\xki)\\bigr\\|\\bigl\\|\\xkie - \\xki\\bigr\\|\\right)\\\\\n    & \\leq & \\ell_f\\left(\\bigl\\|\\tgki - g(\\xki)\\bigr\\| + \\frac{1}{2L_g}\\bigl\\|\\tJki - g'(\\xki)\\bigr\\|^2 + \\frac{L_g}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2 \\right).\n\\end{eqnarray*}\nReplacing $\\xkie$ in the above inequality with $\\hxkie$, we get  \n\\begin{eqnarray*}\n\\bigl|F(\\hxkie;\\xki)-\\widetilde{F}(\\hxkie;\\xki)\\bigr| \n\t& \\leq & \\ell_f \\left(\\bigl\\|\\tgki - g(\\xki)\\bigr\\| + \\frac{1}{2L_g}\\bigl\\|\\tJki - g'(\\xki)\\bigr\\|^2 + \\frac{L_g}{2}\\bigl\\|\\hxkie-\\xki\\bigr\\|^2\\right).\n\\end{eqnarray*}\nCombining the two bounds above with~\\eqref{lm:PG-finite-sum-3} gives\n\\begin{eqnarray*}\n\tM\\bigl\\|\\hxkie-\\xkie\\bigr\\|^2 \n    & \\leq & 2\\ell_f\\bigl\\|\\tgki - g(\\xki)\\bigr\\| + \\frac{\\ell_f}{L_g}\\bigl\\|\\tJki - g'(\\xki)\\bigr\\|^2 \\\\\n    && +\\, \\frac{\\ell_f L_g}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2+\\frac{\\ell_f L_g}{2}\\bigl\\|\\hxkie-\\xki\\bigr\\|^2 .\n\\end{eqnarray*}\nNext, using the fact that $\\|a+b\\|^2\\leq 2\\|a\\|^2+2\\|b\\|^2$ and the above inequality, we have\n\\begin{eqnarray*} \n\tM\\bigl\\|\\hxkie-\\xki\\bigr\\|^2] \n    & \\leq & 2M\\bigl\\|\\xkie-\\xki\\bigr\\|^2 + 2M\\bigl\\|\\hxkie -\\xkie\\bigr\\|^2 \\\\\n\t& \\leq & 2M\\bigl\\|\\xkie-\\xki\\bigr\\|^2 + 4\\ell_f\\bigl\\|\\tgki - g(\\xki)\\bigr\\| + \\frac{2\\ell_f}{L_g}\\bigl\\|\\tJki - g'(\\xki)\\bigr\\|^2 \\\\\n\t& & + \\ell_f L_g\\bigl\\|\\xkie-\\xki\\bigr\\|^2 + \\ell_f L_g\\bigl\\|\\hxkie-\\xki\\bigr\\|^2.\n\\end{eqnarray*}\nRearranging the terms yields \n\\begin{eqnarray*}\n    (M - \\ell_f L_g)\\bigl\\|\\hxkie-\\xki\\bigr\\|^2 \n    &\\leq& (2M+\\ell_f L_g)\\bigl\\|\\xkie-\\xki\\bigr\\|^2  \\\\\n    && +\\, 4\\ell_f\\bigl\\|\\tgki-g(\\xki)\\bigr\\| \n     \\,+\\, \\frac{2\\ell_f}{L_g}\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|^2.\n\\end{eqnarray*}\nFinally, using the definitions $\\cG(\\xki)=M(\\xki-\\hxkie)$ and\n$\\widetilde{\\cG}(\\xki)=M(\\xki-\\xkie)$, we obtain the desired result.\n\\end{proof}","sha256_utf8":"1e14e01f85067b363c2d6e8148300913ef1dbbe60293c96a12b3949dc476dc0e"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"定义双模型并确认强凸性","source_units":[{"start":34436,"end":35138,"start_line":480,"end_line":493,"raw_tex":"For the ease of notation,  we denote \n\\begin{eqnarray}\n    F(x;\\xki) &=& f\\bigl(g(\\xki) + g'(\\xki)(x-\\xki)\\bigr), \\label{eqn:F-def}\\\\\n    \\widetilde{F}(x;\\xki) &=& f\\bigl(\\tgki + \\tJki(x-\\xki)\\bigr).\n    \\label{eqn:tF-def}\n\\end{eqnarray}\nSince both~$f$ and~$h$ are convex (Assumption~\\ref{assumption:f-cvx}),\nthe following two functions are $M$-strongly convex:\n\\begin{eqnarray}\n    && F(x;\\xki) + h(x) + \\frac{M}{2}\\|x-\\xki\\|^2, \\label{eqn:F+h+M} \\\\\n    && \\widetilde{F}(x;\\xki) + h(x) + \\frac{M}{2}\\|x-\\xki\\|^2.\n    \\label{eqn:tF+h+M} \n\\end{eqnarray} \nAccording to~\\eqref{eqn:x-hat-def} and~\\eqref{eqn:xkie-def}, $\\hat{x}_{i+1}^k$ and $\\xkie$ are the minimizers of these two functions respectively. \n","sha256_utf8":"d37ce284fb1692e1044fecfd56d30b866826aa4727d63a3b0935afb344b37e36"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"分别应用强凸最优性","source_units":[{"start":35138,"end":35672,"start_line":494,"end_line":505,"raw_tex":"Therefore\n\\begin{eqnarray*}\n    F(\\hat{x}^k_{i+1};\\xki) + h(\\hat{x}^k_{i+1}) + \\frac{M}{2}\\|\\hat{x}^k_{i+1}-\\xki\\|^2 \n\t& \\leq & F(\\xkie;\\xki) + h(\\xkie) + \\frac{M}{2}\\|\\xkie-\\xki\\|^2  \\\\\n    &&  - \\frac{M}{2}\\|\\hat{x}^k_{i+1}-\\xkie\\|^2,  \n\\end{eqnarray*} \nand\n\\begin{eqnarray*}\n    \\widetilde{F}(x^k_{i+1};\\xki) + h(x^k_{i+1}) + \\frac{M}{2}\\|x^k_{i+1}-\\xki\\|^2 \n& \\leq & \\widetilde{F}(\\hat{x}^k_{i+1};\\xki) + h(\\hat{x}^k_{i+1}) + \\frac{M}{2}\\|\\hat{x}^k_{i+1}-\\xki\\|^2 \\\\\n&& - \\frac{M}{2}\\|\\hat{x}^k_{i+1}-\\xkie\\|^2. \n\\end{eqnarray*} \n","sha256_utf8":"4f362d53c6e4d9786be9f90608723b14c78bd72516aeb885e47abc1e5cef0356"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"相加得到近端点距离界","source_units":[{"start":35672,"end":35955,"start_line":506,"end_line":512,"raw_tex":"Summing the two inequalities above and rearranging the terms, we obtain\n\\begin{eqnarray}\n\t\\label{lm:PG-finite-sum-3}\n\tM\\|\\hat{x}^k_{i+1}-\\xkie\\|^2 \n    &\\leq& F(\\xkie;\\xki)-\\widetilde{F}(\\xkie;\\xki) \n     + \\widetilde{F}(\\hat{x}^k_{i+1};\\xki)-F(\\hat{x}^k_{i+1};\\xki).\n\\end{eqnarray}\n","sha256_utf8":"4726370753fe984a64eca396d62466cf7a581d8bd5c034224d7238479dd76a23"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"控制两个近端点的模型误差","source_units":[{"start":35955,"end":36897,"start_line":513,"end_line":525,"raw_tex":"Using the Lipschitz property of~$f$, we have \n\\begin{eqnarray*}\n\\bigl| F(\\xkie;\\xki)-\\widetilde{F}(\\xkie;\\xki)\\bigr| \n    &=& \\left| f\\bigl(g(\\xki)+g'(\\xki)(\\xkie-\\xki)\\bigr) - f\\bigl(\\tgki+\\tJki(\\xkie-\\xki)\\bigr)\\right| \\\\\n    & \\leq & \\ell_f\\left\\|\\bigl(g(\\xki) - \\tgki\\bigr) + \\bigl(g'(\\xki) - \\tJki\\bigr)\\bigl(\\xkie - \\xki\\bigr)\\right\\|\\\\\n\t& \\leq & \\ell_f\\left(\\bigl\\|\\tgki - g(\\xki)\\bigr\\| + \\bigl\\|\\tJki - g'(\\xki)\\bigr\\|\\bigl\\|\\xkie - \\xki\\bigr\\|\\right)\\\\\n    & \\leq & \\ell_f\\left(\\bigl\\|\\tgki - g(\\xki)\\bigr\\| + \\frac{1}{2L_g}\\bigl\\|\\tJki - g'(\\xki)\\bigr\\|^2 + \\frac{L_g}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2 \\right).\n\\end{eqnarray*}\nReplacing $\\xkie$ in the above inequality with $\\hxkie$, we get  \n\\begin{eqnarray*}\n\\bigl|F(\\hxkie;\\xki)-\\widetilde{F}(\\hxkie;\\xki)\\bigr| \n\t& \\leq & \\ell_f \\left(\\bigl\\|\\tgki - g(\\xki)\\bigr\\| + \\frac{1}{2L_g}\\bigl\\|\\tJki - g'(\\xki)\\bigr\\|^2 + \\frac{L_g}{2}\\bigl\\|\\hxkie-\\xki\\bigr\\|^2\\right).\n\\end{eqnarray*}\n","sha256_utf8":"cd9ffc4b08d76f5b1a6c15d04f7b5e2e89f9476867e6ae5ae5c02b66537ea2d4"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"合并模型误差与距离界","source_units":[{"start":36897,"end":37246,"start_line":526,"end_line":531,"raw_tex":"Combining the two bounds above with~\\eqref{lm:PG-finite-sum-3} gives\n\\begin{eqnarray*}\n\tM\\bigl\\|\\hxkie-\\xkie\\bigr\\|^2 \n    & \\leq & 2\\ell_f\\bigl\\|\\tgki - g(\\xki)\\bigr\\| + \\frac{\\ell_f}{L_g}\\bigl\\|\\tJki - g'(\\xki)\\bigr\\|^2 \\\\\n    && +\\, \\frac{\\ell_f L_g}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2+\\frac{\\ell_f L_g}{2}\\bigl\\|\\hxkie-\\xki\\bigr\\|^2 .\n\\end{eqnarray*}\n","sha256_utf8":"6919174407cbe4a1d1ce056234da9719e72c255928ad2635ebbbe7b79403b6e9"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"用三角界比较两种步长","source_units":[{"start":37246,"end":37706,"start_line":532,"end_line":538,"raw_tex":"Next, using the fact that $\\|a+b\\|^2\\leq 2\\|a\\|^2+2\\|b\\|^2$ and the above inequality, we have\n\\begin{eqnarray*} \n\tM\\bigl\\|\\hxkie-\\xki\\bigr\\|^2] \n    & \\leq & 2M\\bigl\\|\\xkie-\\xki\\bigr\\|^2 + 2M\\bigl\\|\\hxkie -\\xkie\\bigr\\|^2 \\\\\n\t& \\leq & 2M\\bigl\\|\\xkie-\\xki\\bigr\\|^2 + 4\\ell_f\\bigl\\|\\tgki - g(\\xki)\\bigr\\| + \\frac{2\\ell_f}{L_g}\\bigl\\|\\tJki - g'(\\xki)\\bigr\\|^2 \\\\\n\t& & + \\ell_f L_g\\bigl\\|\\xkie-\\xki\\bigr\\|^2 + \\ell_f L_g\\bigl\\|\\hxkie-\\xki\\bigr\\|^2.\n\\end{eqnarray*}\n","sha256_utf8":"f94deccbc4c9ca02dc3eb3df6af56712434e93afd18ac273d2225585b7efb9cb"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"整理并代入梯度映射定义","source_units":[{"start":37706,"end":38116,"start_line":539,"end_line":547,"raw_tex":"Rearranging the terms yields \n\\begin{eqnarray*}\n    (M - \\ell_f L_g)\\bigl\\|\\hxkie-\\xki\\bigr\\|^2 \n    &\\leq& (2M+\\ell_f L_g)\\bigl\\|\\xkie-\\xki\\bigr\\|^2  \\\\\n    && +\\, 4\\ell_f\\bigl\\|\\tgki-g(\\xki)\\bigr\\| \n     \\,+\\, \\frac{2\\ell_f}{L_g}\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|^2.\n\\end{eqnarray*}\nFinally, using the definitions $\\cG(\\xki)=M(\\xki-\\hxkie)$ and\n$\\widetilde{\\cG}(\\xki)=M(\\xki-\\xkie)$, we obtain the desired result.\n","sha256_utf8":"3d506e3b04eab44ce1cb15d3ce48096e8566a73f34e45ee17f6bdf4d61efc2d9"}]},{"id":"S033","semantic_type":"lemma","kind":"lemma","title":"修正估计器误差界","source_units":[{"start":41913,"end":42592,"start_line":600,"end_line":610,"raw_tex":"\\begin{lemma}\n\t\\label{lemma:svr-f-new}\n    Suppose Assumption~\\ref{assumption:g-lip-finite} holds and $\\tgki$ and $\\tJki$ are constructed according to~\\eqref{eqn:g-svrg-correction} and~\\eqref{eqn:J-svrg} respectively, then \n\\begin{eqnarray*}\n\t\\E\\Bigl[\\bigl\\|\\tgki - g(\\xki)\\bigr\\| \\,\\big|\\, \\xki\\Bigr] \n        &\\leq& \\frac{L_g}{2\\sqrt{|\\cB^k_i|}}\\bigl\\|\\xki - x_0^k\\bigr\\|^2,\\\\ \n\t\\E\\Bigl[\\bigl\\|\\tJki - g'(\\xki)\\bigr\\|^2 \\,\\big|\\, \\xki\\Bigr] \n    &\\leq& \\frac{L_g^2}{|\\cS_i^k|}\\bigl\\|\\xki - x_0^k\\bigr\\|^2,\n\\end{eqnarray*}\nwhere $\\E[\\cdot|\\xki]$ denotes conditional expectation given~$\\xki$, i.e., expectation with respect to the random indices in $\\Bki$ and $\\Ski$.\n\\end{lemma}","sha256_utf8":"aa0b2dca0941bd710e08bb6d0915865808bdc124e6d77f46a13f1530bac0d64d"}]},{"id":"S034","semantic_type":"proof","kind":"proof","title":"估计器误差界完整证明","source_units":[{"start":42593,"end":44697,"start_line":611,"end_line":642,"raw_tex":"\\begin{proof}\n    To prove the first inequality, we start with~\\eqref{eqn:g-svrg-correction} and write\n\\begin{equation}\\label{eqn:tgki-Z}\n    \\tgki - g(\\xki) = \\frac{1}{|\\Bki|}\\sum_{j\\in\\Bki}Z_j,\n\\end{equation}\n    where\n    $$Z_j = g_j(\\xki) - g_j(x_0^k) - g_j'(x_0^k)(\\xki-x_0^k) \\,+\\, g(x_0^k) + g'(x_0^k)(\\xki-x_0^k) \\,-\\, g(\\xki).$$\n    Since~$j$ is randomly sampled from $\\{1,2,\\ldots,N\\}$, we have $\\E[g_j(\\xki)]=g(\\xki)$ and $\\E[g'_j(\\xki)]=g'(\\xki)$, which implies $\\E[Z_j|\\xki] = 0$. That is, $\\tgki$ is an unbiased estimate of $g(\\xki)$.\n    In addition, we have\n\t$$\\E\\bigl[g_j(\\xki) - g_j(x_0^k) - g_j'(x_0^k)(\\xki-x_0^k)\\,\\bigl|\\,\\xki\\bigl] =   g(\\xki) - g(x_0^k) - g'(x_0^k)(\\xki-x_0^k).$$ \nThis allows us to bound the variance of $Z_j$ as follows:\n\\begin{eqnarray*}\n\t\\E\\bigl[\\|Z_j\\|^2\\,\\big|\\,\\xki\\bigr] \n    & = & \\E\\Bigl[\\bigl\\|g_j(\\xki) - g_j(x_0^k) - g_j'(x_0^k)(\\xki-x_0^k) \\bigr\\|^2\\,\\big|\\,\\xki\\Bigr] \\\\\n    && -\\, \\bigl\\|g(\\xki) - g(x_0^k) - g'(x_0^k)(\\xki-x_0^k)\\bigr\\|^2 \\\\\n\t& \\leq & \\E\\Bigl[\\bigl\\|g_j(\\xki) - g_j(x_0^k) - g_j'(x_0^k)(\\xki-x_0^k) \\bigr\\|^2 \\,\\big|\\,\\xki\\Bigr]\\\\\n\t&\\leq & \\frac{1}{N}\\sum_{j=1}^N\\left(\\frac{L_{g,j}}{2}\\|\\xki-x_0^k\\|^2\\right)^2\\\\\n\t& = & \\frac{L_{g}^2}{4}\\|\\xki-x_0^k\\|^4,\n\\end{eqnarray*}\nwhere the last inequality is due to~\\eqref{eqn:g-quadratic-bd} and Assumption \\ref{assumption:g-lip-finite} respectively.\nIn the last equality, we used the definition of~$L_g$ in~\\eqref{eqn:ms-Lip-constants}.\n\nCombining the above inequality with~\\eqref{eqn:tgki-Z} yields\n\t$$\\E\\Bigl[\\bigl\\|\\tgki - g(\\xki)\\bigr\\|^2 \\,\\big|\\, \\xki\\Bigr] \\leq \\frac{L_g^2}{4|\\cB^k_i|}\\bigl\\|\\xki - x_0^k\\bigr\\|^4.$$ \n    Next, using the concavity of~$\\sqrt{\\cdot}$ and Jensen's inequality, we obtain the desired result:\n\\[\n\t\\E\\Bigl[\\bigl\\|\\tgki - g(\\xki)\\bigr\\| \\,\\big|\\, \\xki\\Bigr] \n\t\\leq \\sqrt{\\E\\Bigl[\\bigl\\|\\tgki - g(\\xki)\\bigr\\|^2 \\,\\big|\\, \\xki\\Bigr] }\n\t\\leq \\frac{L_g}{2\\sqrt{|\\cB^k_i|}}\\bigl\\|\\xki - x_0^k\\bigr\\|^2.\n\\]\nTo prove the second inequality, we define $Z_j = g'_j(\\xki) - g'_j(x_0^k) + g'(x_0^k) - g'(\\xki)$ and follow a similar line of arguments. \n\\end{proof}","sha256_utf8":"0700c1015fee7f0ccef9252d589a91ef6bef2a498e92ab3abfe4fb7ba952a47e"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"中心化分解并确认条件无偏","source_units":[{"start":42607,"end":43298,"start_line":612,"end_line":620,"raw_tex":"    To prove the first inequality, we start with~\\eqref{eqn:g-svrg-correction} and write\n\\begin{equation}\\label{eqn:tgki-Z}\n    \\tgki - g(\\xki) = \\frac{1}{|\\Bki|}\\sum_{j\\in\\Bki}Z_j,\n\\end{equation}\n    where\n    $$Z_j = g_j(\\xki) - g_j(x_0^k) - g_j'(x_0^k)(\\xki-x_0^k) \\,+\\, g(x_0^k) + g'(x_0^k)(\\xki-x_0^k) \\,-\\, g(\\xki).$$\n    Since~$j$ is randomly sampled from $\\{1,2,\\ldots,N\\}$, we have $\\E[g_j(\\xki)]=g(\\xki)$ and $\\E[g'_j(\\xki)]=g'(\\xki)$, which implies $\\E[Z_j|\\xki] = 0$. That is, $\\tgki$ is an unbiased estimate of $g(\\xki)$.\n    In addition, we have\n\t$$\\E\\bigl[g_j(\\xki) - g_j(x_0^k) - g_j'(x_0^k)(\\xki-x_0^k)\\,\\bigl|\\,\\xki\\bigl] =   g(\\xki) - g(x_0^k) - g'(x_0^k)(\\xki-x_0^k).$$ \n","sha256_utf8":"165594af6e5074c07327ab6cc3da5532441c0f05898ec92bff9bb819d866794f"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"以二阶余项控制单样本方差","source_units":[{"start":43298,"end":44048,"start_line":621,"end_line":631,"raw_tex":"This allows us to bound the variance of $Z_j$ as follows:\n\\begin{eqnarray*}\n\t\\E\\bigl[\\|Z_j\\|^2\\,\\big|\\,\\xki\\bigr] \n    & = & \\E\\Bigl[\\bigl\\|g_j(\\xki) - g_j(x_0^k) - g_j'(x_0^k)(\\xki-x_0^k) \\bigr\\|^2\\,\\big|\\,\\xki\\Bigr] \\\\\n    && -\\, \\bigl\\|g(\\xki) - g(x_0^k) - g'(x_0^k)(\\xki-x_0^k)\\bigr\\|^2 \\\\\n\t& \\leq & \\E\\Bigl[\\bigl\\|g_j(\\xki) - g_j(x_0^k) - g_j'(x_0^k)(\\xki-x_0^k) \\bigr\\|^2 \\,\\big|\\,\\xki\\Bigr]\\\\\n\t&\\leq & \\frac{1}{N}\\sum_{j=1}^N\\left(\\frac{L_{g,j}}{2}\\|\\xki-x_0^k\\|^2\\right)^2\\\\\n\t& = & \\frac{L_{g}^2}{4}\\|\\xki-x_0^k\\|^4,\n\\end{eqnarray*}\nwhere the last inequality is due to~\\eqref{eqn:g-quadratic-bd} and Assumption \\ref{assumption:g-lip-finite} respectively.\nIn the last equality, we used the definition of~$L_g$ in~\\eqref{eqn:ms-Lip-constants}.\n","sha256_utf8":"5cd0f53cb8963a2bf57f192a5ad3215b263d4bb67d2db359eb36f7625dc0d631"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"批量平均及詹森不等式","source_units":[{"start":44049,"end":44547,"start_line":633,"end_line":640,"raw_tex":"Combining the above inequality with~\\eqref{eqn:tgki-Z} yields\n\t$$\\E\\Bigl[\\bigl\\|\\tgki - g(\\xki)\\bigr\\|^2 \\,\\big|\\, \\xki\\Bigr] \\leq \\frac{L_g^2}{4|\\cB^k_i|}\\bigl\\|\\xki - x_0^k\\bigr\\|^4.$$ \n    Next, using the concavity of~$\\sqrt{\\cdot}$ and Jensen's inequality, we obtain the desired result:\n\\[\n\t\\E\\Bigl[\\bigl\\|\\tgki - g(\\xki)\\bigr\\| \\,\\big|\\, \\xki\\Bigr] \n\t\\leq \\sqrt{\\E\\Bigl[\\bigl\\|\\tgki - g(\\xki)\\bigr\\|^2 \\,\\big|\\, \\xki\\Bigr] }\n\t\\leq \\frac{L_g}{2\\sqrt{|\\cB^k_i|}}\\bigl\\|\\xki - x_0^k\\bigr\\|^2.\n\\]\n","sha256_utf8":"4b05df2c1e20360f820fdbfa351bb8890661a7bfde37b2073c76f355aebab573"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"雅可比误差的同类论证","source_units":[{"start":44547,"end":44686,"start_line":641,"end_line":641,"raw_tex":"To prove the second inequality, we define $Z_j = g'_j(\\xki) - g'_j(x_0^k) + g'(x_0^k) - g'(\\xki)$ and follow a similar line of arguments. \n","sha256_utf8":"0e189177d491d67943f93495f54d07c178ed59683870392e3cd590dc9761299a"}]},{"id":"S039","semantic_type":"lemma","kind":"lemma","title":"含估计误差的单步下降","source_units":[{"start":44805,"end":45419,"start_line":645,"end_line":655,"raw_tex":"\\begin{lemma}\n\\label{lemma: descent-Finite-nonsmooth}\nSuppose Assumptions~\\ref{assumption:f-cvx} and~\\ref{assumption:g-lip-finite} hold and the estimates $\\tgkz$, $\\tJkz$, $\\tgki$ and $\\tJki$ in Algorithm~\\ref{algo:SVR-PL} are constructed as in~\\eqref{eqn:finite-gk0}-\\eqref{eqn:J-svrg} respectively.\nThen for $k=1,\\ldots,K$ and $i=0,\\ldots,\\tau-1$, \n\\begin{eqnarray}\n\\Phi(\\xkie) \n&\\leq& \\Phi(\\xki) - \\frac{M - 2\\ell_f L_g}{2M^2}\\bigl\\|\\tcGki\\bigr\\|^2 \\nonumber \\\\\n&& +\\, 2\\ell_f \\bigl\\|\\tgki-g(\\xki)\\bigr\\| + \\frac{\\ell_f}{2L_g}\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|^2. \n\\label{eqn:descent-lemma}\n\\end{eqnarray}\n\\end{lemma}","sha256_utf8":"b1a7378f8a56a338f4070a6af87b277e590e74346f9db4140e5129603e4a7ffd"}]},{"id":"S040","semantic_type":"proof","kind":"proof","title":"单步下降完整证明","source_units":[{"start":45420,"end":47611,"start_line":656,"end_line":698,"raw_tex":"\\begin{proof}\nBy the definition of $\\Phi$ in~\\eqref{eqn:Phi-def} and Lemma~\\ref{lemma:upper-bound}, we have \n\\begin{eqnarray}\n\\Phi(\\xkie) & \\leq & f(g(\\xki) + g'(\\xki)(\\xkie-\\xki)) + \\frac{\\ell_f L_g}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2 + h(\\xkie) \\nonumber \\\\\n& = &  f\\bigl(\\tgki + \\tJki(\\xkie-\\xki)\\bigr) + h(\\xkie) + \\frac{M}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2\n -\\, \\frac{M - \\ell_f L_g}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2 \\nonumber \\\\\n& & + \\underbrace{f\\bigl(g(\\xki) + g'(\\xki)(\\xkie-\\xki)\\bigr)-f\\bigl(\\tgki + g'(\\xki)(\\xkie-\\xki)\\bigr)}_{T_1}\\nonumber\\\\\n& & + \\underbrace{f\\bigl(\\tgki + g'(\\xki)(\\xkie-\\xki)\\bigr)-f\\bigl(\\tgki + \\tJki(\\xkie-\\xki)\\bigr)}_{T_2} .\n\\label{eqn:descent-finite-0}\n\\end{eqnarray}\nAccording to~\\eqref{eqn:xkie-def}, we have\n\\[\n    f\\bigl(\\tgki + \\tJki(\\xkie-\\xki)\\bigr) + h(\\xkie) + \\frac{M}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2 ~\\leq~ f(\\tgki) + h(\\xki).\n\\]\nTherefore,\n\\begin{eqnarray}\n\\Phi(\\xkie) \n&\\leq & f(\\tgki) + h(\\xki) - \\frac{M-\\ell_f L_g}{2} \\|\\xkie-\\xki\\|^2 + T_1+ T_2 \\nonumber\\\\\n&\\leq & f(g(\\xki)) + h(\\xki) - \\frac{M-\\ell_f L_g}{2} \\|\\xkie-\\xki\\|^2 + T_1+ T_2  +  \\underbrace{f(\\tgki)-f(g(\\xki))}_{T_3}\\nonumber\\\\\n& = & \\Phi(\\xki)  - \\frac{M-\\ell_f L_g}{2} \\|\\xkie-\\xki\\|^2 + T_1+ T_2 +T_3. \n\\label{eqn:Phi-descent}\n\\end{eqnarray}\nBy the Lipschitz property of~$f$, we have\n\\begin{eqnarray*}\n    T_1 \\leq \\ell_f \\bigl\\|\\tgki-g(\\xki)\\bigr\\|, \\qquad\n    T_3 \\leq \\ell_f \\bigl\\|\\tgki-g(\\xki)\\bigr\\|,\n\\end{eqnarray*}\nand\n\\begin{eqnarray}\nT_2 &\\leq& \\ell_f\\bigl\\|\\bigl(\\tJki-g'(\\xki)\\bigr)(\\xkie-\\xki)\\bigr\\|\n~\\leq~ \\ell_f\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|\\cdot\\bigl\\|\\xkie-\\xki\\bigr\\| \\nonumber\\\\\n&\\leq& \\frac{\\ell_f}{2L_g}\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|^2 + \\frac{\\ell_f L_g}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2.\n\\label{eqn:descent-lemma-T2}\n\\end{eqnarray}\nCombining the bounds on $T_1$, $T_2$ and $T_3$ and the inequality~\\eqref{eqn:Phi-descent} yields\n\\begin{eqnarray*}\n    \\Phi(\\xkie) \n    &\\leq& \\Phi(\\xki) - \\left(\\frac{M}{2} - \\ell_f L_g\\right)\\bigl\\|\\xkie-\\xki\\bigr\\|^2 \\\\\n    && +\\, 2\\ell_f \\bigl\\|\\tgki-g(\\xki)\\bigr\\| + \\frac{\\ell_f}{2L_g}\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|^2, \n\\end{eqnarray*}\nwhich, upon noticing $\\tcGki=-M(\\xkie-\\xki)$, is equivalent to the\ndesired result. \n\\end{proof}","sha256_utf8":"dd83b907e22e8e702e22b32685d2351f3d058e6201b679db6c40554c9393be6d"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"线性模型上界及误差分解","source_units":[{"start":45434,"end":46114,"start_line":657,"end_line":665,"raw_tex":"By the definition of $\\Phi$ in~\\eqref{eqn:Phi-def} and Lemma~\\ref{lemma:upper-bound}, we have \n\\begin{eqnarray}\n\\Phi(\\xkie) & \\leq & f(g(\\xki) + g'(\\xki)(\\xkie-\\xki)) + \\frac{\\ell_f L_g}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2 + h(\\xkie) \\nonumber \\\\\n& = &  f\\bigl(\\tgki + \\tJki(\\xkie-\\xki)\\bigr) + h(\\xkie) + \\frac{M}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2\n -\\, \\frac{M - \\ell_f L_g}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2 \\nonumber \\\\\n& & + \\underbrace{f\\bigl(g(\\xki) + g'(\\xki)(\\xkie-\\xki)\\bigr)-f\\bigl(\\tgki + g'(\\xki)(\\xkie-\\xki)\\bigr)}_{T_1}\\nonumber\\\\\n& & + \\underbrace{f\\bigl(\\tgki + g'(\\xki)(\\xkie-\\xki)\\bigr)-f\\bigl(\\tgki + \\tJki(\\xkie-\\xki)\\bigr)}_{T_2} .\n\\label{eqn:descent-finite-0}\n\\end{eqnarray}\n","sha256_utf8":"6ca4344d47222b2d61445364f71b9730e7418d15e930689c7ac2ea4fd2cc49a4"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"利用子问题最优性展开下降","source_units":[{"start":46114,"end":46670,"start_line":666,"end_line":677,"raw_tex":"According to~\\eqref{eqn:xkie-def}, we have\n\\[\n    f\\bigl(\\tgki + \\tJki(\\xkie-\\xki)\\bigr) + h(\\xkie) + \\frac{M}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2 ~\\leq~ f(\\tgki) + h(\\xki).\n\\]\nTherefore,\n\\begin{eqnarray}\n\\Phi(\\xkie) \n&\\leq & f(\\tgki) + h(\\xki) - \\frac{M-\\ell_f L_g}{2} \\|\\xkie-\\xki\\|^2 + T_1+ T_2 \\nonumber\\\\\n&\\leq & f(g(\\xki)) + h(\\xki) - \\frac{M-\\ell_f L_g}{2} \\|\\xkie-\\xki\\|^2 + T_1+ T_2  +  \\underbrace{f(\\tgki)-f(g(\\xki))}_{T_3}\\nonumber\\\\\n& = & \\Phi(\\xki)  - \\frac{M-\\ell_f L_g}{2} \\|\\xkie-\\xki\\|^2 + T_1+ T_2 +T_3. \n\\label{eqn:Phi-descent}\n\\end{eqnarray}\n","sha256_utf8":"53ed6389832049355bfe31c66351a953360f2a275ab2ac60e65dbfdd549844f5"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"逐项控制三类误差","source_units":[{"start":46670,"end":47176,"start_line":678,"end_line":689,"raw_tex":"By the Lipschitz property of~$f$, we have\n\\begin{eqnarray*}\n    T_1 \\leq \\ell_f \\bigl\\|\\tgki-g(\\xki)\\bigr\\|, \\qquad\n    T_3 \\leq \\ell_f \\bigl\\|\\tgki-g(\\xki)\\bigr\\|,\n\\end{eqnarray*}\nand\n\\begin{eqnarray}\nT_2 &\\leq& \\ell_f\\bigl\\|\\bigl(\\tJki-g'(\\xki)\\bigr)(\\xkie-\\xki)\\bigr\\|\n~\\leq~ \\ell_f\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|\\cdot\\bigl\\|\\xkie-\\xki\\bigr\\| \\nonumber\\\\\n&\\leq& \\frac{\\ell_f}{2L_g}\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|^2 + \\frac{\\ell_f L_g}{2}\\bigl\\|\\xkie-\\xki\\bigr\\|^2.\n\\label{eqn:descent-lemma-T2}\n\\end{eqnarray}\n","sha256_utf8":"cec0f2995d4d5bd199d66448bbef96cc3b18577c34551a3c2986e546b939dbb4"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"合并误差得到梯度映射下降","source_units":[{"start":47176,"end":47600,"start_line":690,"end_line":697,"raw_tex":"Combining the bounds on $T_1$, $T_2$ and $T_3$ and the inequality~\\eqref{eqn:Phi-descent} yields\n\\begin{eqnarray*}\n    \\Phi(\\xkie) \n    &\\leq& \\Phi(\\xki) - \\left(\\frac{M}{2} - \\ell_f L_g\\right)\\bigl\\|\\xkie-\\xki\\bigr\\|^2 \\\\\n    && +\\, 2\\ell_f \\bigl\\|\\tgki-g(\\xki)\\bigr\\| + \\frac{\\ell_f}{2L_g}\\bigl\\|\\tJki-g'(\\xki)\\bigr\\|^2, \n\\end{eqnarray*}\nwhich, upon noticing $\\tcGki=-M(\\xkie-\\xki)$, is equivalent to the\ndesired result. \n","sha256_utf8":"f666f05c3fc80d0ca7f2dab0ae71f079ef572bdf65c10d96a0d95828f2575a0a"}]},{"id":"S045","semantic_type":"lemma","kind":"lemma","title":"阶段梯度总量的势函数界","source_units":[{"start":48643,"end":49276,"start_line":720,"end_line":730,"raw_tex":"\\begin{lemma}\n\t\\label{lemma:Lyapunov-finite-nonsmooth}\nSuppose Assumptions~\\ref{assumption:f-cvx} and~\\ref{assumption:g-lip-finite} hold and the estimates $\\tgkz$, $\\tJkz$, $\\tgki$ and $\\tJki$ in Algorithm~\\ref{algo:SVR-PL} are constructed as in~\\eqref{eqn:finite-gk0}-\\eqref{eqn:J-svrg} respectively.\nIn addition, we assume $M\\geq 4\\ell_f L_g$ and $\\gamma>0$.\nThen for each $k=1,\\ldots,K$,\n\\begin{equation}\\label{lm:Lyapunov-finite-nonsmooth-1}\n\t\\sum_{i=0}^{\\tau-1}\\E\\Bigl[\\bigl\\|\\cGki\\bigr\\|^2\\Bigr] \n    ~\\leq~  \\frac{R^k_0-R^k_\\tau}{\\gamma}  \n    ~=~ \\frac{\\E[\\Phi(x_0^k)]-\\E[\\Phi(x_0^{k+1})]}{\\gamma}.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"ba180d62d52f73f30e00ca7440f0f7186c78125145953cf0ddee7f4751bc83c7"}]},{"id":"S046","semantic_type":"proof","kind":"proof","title":"阶段势函数界完整证明","source_units":[{"start":49277,"end":53422,"start_line":731,"end_line":799,"raw_tex":"\\begin{proof}\nFor the ease of notation, we write the stochastic Lyapunov function as\n$$R^k_i = \\E\\bigl[\\Phi(\\xki) + c_i\\|G_i^k\\|^2\\bigr],$$\nwhere\n\\begin{equation}\\label{eqn:Gki-def}\n    \\Gki = \\sum_{t=0}^{i-1}\\tcGkt = -M(\\xki - x_0^k).\n\\end{equation}\nIn particular, we have $G_{0}^k = -M(x_0^k-x_0^k) = 0$.\nMoreover, we have \n\\begin{eqnarray}\n\t\\E\\bigl[\\|G_{i+1}^k\\|^2\\bigr] \n    &=& \\E\\bigl[\\|G_{i}^k + \\tcGki\\|^2\\bigr]   \\nonumber \\\\\n    &\\leq& \\left(1+\\frac{1}{\\tau}\\right)\\E\\bigl[\\|G_{i}^k\\|^2\\bigr] + (1+\\tau)\\E\\bigl[\\|\\tcGki\\|^2\\bigr]. \n\\label{eqn:Gki-tcG}\n\\end{eqnarray}\nCombining Lemmas~\\ref{lemma:PG-finite-sum-nonsmooth} \nand~\\ref{lemma: descent-Finite-nonsmooth} yields\n\\begin{eqnarray*}\n\\E\\bigl[\\Phi(\\xkie)\\bigr] \n    &\\leq& \\E\\bigl[\\Phi(\\xki)\\bigr] - \\frac{M - 2\\ell_f L_g}{2M^2}\\E\\bigl[\\bigl\\|\\tcGki\\bigr\\|^2\\bigr] + \\left(\\frac{\\ell_f L_g}{\\sqrt{|\\cB_i^k|}}+\\frac{\\ell_f L_g}{2|\\cS_i^k|}\\right)\\E\\bigl[\\|\\xki-x_0^k\\|^2\\bigr] \\\\\n    &=& \\E\\bigl[\\Phi(\\xki)\\bigr] - \\frac{M - 2\\ell_f L_g}{2M^2}\\E\\bigl[\\bigl\\|\\tcGki\\bigr\\|^2\\bigr] + \\frac{1}{M^2}\\left(\\frac{\\ell_f L_g}{\\sqrt{|\\cB_i^k|}}+\\frac{\\ell_f L_g}{2|\\cS_i^k|}\\right)\\E\\bigl[\\|G^k_i\\|^2\\bigr],\n\\end{eqnarray*}\nwhere in the last equality we used~\\eqref{eqn:Gki-def}.\nAdding both sides of~\\eqref{eqn:Gki-tcG} to that of the above inequality \nand using the assumption $M\\geq 4\\ell_f L_g$, we obtain\n\\begin{eqnarray}\n\\E\\bigl[\\Phi(\\xkie)+c_{i+1}\\|G_{i+1}^k\\|^2\\bigr] \n    & \\leq & \\E\\bigl[\\Phi(\\xki)\\bigr] - \\left(\\frac{M-2\\ell_f L_g}{2M^2} - c_{i+1}(1+\\tau)\\right)\\E\\bigl[\\|\\tcGki\\|^2\\bigr] \\nonumber \\\\ \n    & & +\\left(\\frac{1}{M^2}\\left(\\frac{\\ell_f L_g}{\\sqrt{|\\cB_i^k|}}+\\frac{\\ell_f L_g}{2|\\cS_i^k|}\\right) + c_{i+1}\\left(1+\\frac{1}{\\tau}\\right)\\right)\\E\\bigl[\\|G_i^k\\|^2\\bigr] \\nonumber \\\\\n    & \\leq & \\E\\bigl[\\Phi(\\xki)\\bigr] - \\left(\\frac{1}{4M} - c_{i+1}(1+\\tau)\\right)\\E\\bigl[\\|\\tcGki\\|^2\\bigr] \\label{eqn:lyapunov-descent}\\\\ \n    & & +\\left(\\frac{1}{4M}\\left(\\frac{1}{\\sqrt{|\\cB_i^k|}}+\\frac{1}{2|\\cS_i^k|}\\right) + c_{i+1}\\left(1+\\frac{1}{\\tau}\\right)\\right)\\E\\bigl[\\|G_i^k\\|^2\\bigr]. \\nonumber\n\\end{eqnarray}\nNext, combining Lemma~\\ref{lemma:PG-finite-sum-nonsmooth} with Lemmas~\\ref{lemma:svr-f-new} yields\n\\begin{eqnarray*}\n    \\frac{M - \\ell_f L_g}{M^2}\\E\\bigl[\\|\\cGki\\|^2\\bigr]\n    &\\leq&\n    \\frac{2M+\\ell_f L_g}{M^2}\\E\\bigl[\\|\\tcGki\\|^2\\bigr] \n     + \\left( \\frac{2\\ell_f L_g}{\\sqrt{|\\cB_i^k|}} + \\frac{2\\ell_f L_g}{|\\cS_i^k|} \\right) \\E\\bigl[\\|\\xki-\\xkz\\|^2\\bigr].\n\\end{eqnarray*}\nUsing the equality $G^k_i=-M(\\xki-\\xkz)$ and the assumption $M\\geq 4\\ell_f L_g$, the above inequality implies\n\\begin{eqnarray}\n    \\frac{3}{4M}\\E\\bigl[\\|\\cGki\\|^2\\bigr]\n    &\\leq&\n    \\frac{9}{4M}\\E\\bigl[\\|\\tcGki\\|^2\\bigr] \n     +\\, \\frac{1}{2M} \\left( \\frac{1}{\\sqrt{|\\cB_i^k|}} + \\frac{1}{|\\cS_i^k|} \\right) \\E\\bigl[\\|G^k_i\\|^2\\bigr].\n    \\label{eqn:cG-bound}\n\\end{eqnarray}\nMultiplying both sides of~\\eqref{eqn:cG-bound} by $\\left(\\frac{1}{4M}-c_{i+1}(1+\\tau)\\right)\\big/\\frac{9}{4M}$, which is positive by the assumption $\\gamma>0$, and adding the resulting inequality to~\\eqref{eqn:lyapunov-descent}, we get\n\\begin{eqnarray}\n%\\label{lm:Lyapunov-finite-nonsmooth-2}\n\\E\\bigl[\\Phi(\\xkie)+c_{i+1}\\|G_{i+1}^k\\|^2\\bigr] \n&\\leq& \\E\\left[\\Phi(\\xki)+\\left( c_{i+1}\\left(1+\\frac{1}{\\tau}\\right)+ \\frac{1}{3M\\sqrt{|\\cB_i^k|}} + \\frac{1}{5M|\\cS_i^k|}\\right)\\|G_i^k\\|^2\\right] \\nonumber\\\\\n\t&& -\\frac{1}{3}\\left(\\frac{1}{4M} - c_{i+1}(1+\\tau)\\right)\\E[\\|\\cGki\\|^2].\\nonumber\n\\end{eqnarray}\nNow, using the definitions in~\\eqref{eqn:lyapunov-def} and~\\eqref{eqn:ct-finite-nonsmooth}, the above inequality is the same as\n\\[\n\t\\frac{1}{3}\\left(\\frac{1}{4M} - c_{i+1}(1+\\tau)\\right)\\E\\bigl[\\|\\cGki\\|^2\\bigr] ~\\leq~  R_i^k  - R_{i+1}^k.\n\\]\nRecalling the definition of $\\gamma$ and summing up the above inequality over~$i$ from~$0$ to~$\\tau-1$, we get\n\\[\n\t\\gamma\\sum_{i=0}^{\\tau-1}\\E\\bigl[\\|\\cGki\\|^2\\bigr]\n    ~\\leq~ R_0^k-R_\\tau^k\n    ~=~\\E\\bigl[\\Phi(x_0^k)\\bigr] - \\E\\bigl[\\Phi(x_\\tau^k)\\bigr] ,\n\\]\nwhere the last equality is due to the observations that \n$c_\\tau=0$ and $G^k_0=0$. \nFinally, dividing both sides by $\\gamma$ and using $x^{k+1}_0 = x^k_\\tau$\ngive the desired result.\n\\end{proof}","sha256_utf8":"6a9b406476689a8b9fdd00c5034abed67182b65f1fb190927c065a81ba7249a1"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"累积梯度与锚点距离递推","source_units":[{"start":49291,"end":49854,"start_line":732,"end_line":745,"raw_tex":"For the ease of notation, we write the stochastic Lyapunov function as\n$$R^k_i = \\E\\bigl[\\Phi(\\xki) + c_i\\|G_i^k\\|^2\\bigr],$$\nwhere\n\\begin{equation}\\label{eqn:Gki-def}\n    \\Gki = \\sum_{t=0}^{i-1}\\tcGkt = -M(\\xki - x_0^k).\n\\end{equation}\nIn particular, we have $G_{0}^k = -M(x_0^k-x_0^k) = 0$.\nMoreover, we have \n\\begin{eqnarray}\n\t\\E\\bigl[\\|G_{i+1}^k\\|^2\\bigr] \n    &=& \\E\\bigl[\\|G_{i}^k + \\tcGki\\|^2\\bigr]   \\nonumber \\\\\n    &\\leq& \\left(1+\\frac{1}{\\tau}\\right)\\E\\bigl[\\|G_{i}^k\\|^2\\bigr] + (1+\\tau)\\E\\bigl[\\|\\tcGki\\|^2\\bigr]. \n\\label{eqn:Gki-tcG}\n\\end{eqnarray}\n","sha256_utf8":"ad99c3c35539433e33a4502b7b090c0f38125d43c5d7ab2ec2116086de5ccfc4"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"将估计误差代入期望下降","source_units":[{"start":49854,"end":50511,"start_line":746,"end_line":753,"raw_tex":"Combining Lemmas~\\ref{lemma:PG-finite-sum-nonsmooth} \nand~\\ref{lemma: descent-Finite-nonsmooth} yields\n\\begin{eqnarray*}\n\\E\\bigl[\\Phi(\\xkie)\\bigr] \n    &\\leq& \\E\\bigl[\\Phi(\\xki)\\bigr] - \\frac{M - 2\\ell_f L_g}{2M^2}\\E\\bigl[\\bigl\\|\\tcGki\\bigr\\|^2\\bigr] + \\left(\\frac{\\ell_f L_g}{\\sqrt{|\\cB_i^k|}}+\\frac{\\ell_f L_g}{2|\\cS_i^k|}\\right)\\E\\bigl[\\|\\xki-x_0^k\\|^2\\bigr] \\\\\n    &=& \\E\\bigl[\\Phi(\\xki)\\bigr] - \\frac{M - 2\\ell_f L_g}{2M^2}\\E\\bigl[\\bigl\\|\\tcGki\\bigr\\|^2\\bigr] + \\frac{1}{M^2}\\left(\\frac{\\ell_f L_g}{\\sqrt{|\\cB_i^k|}}+\\frac{\\ell_f L_g}{2|\\cS_i^k|}\\right)\\E\\bigl[\\|G^k_i\\|^2\\bigr],\n\\end{eqnarray*}\nwhere in the last equality we used~\\eqref{eqn:Gki-def}.\n","sha256_utf8":"cc2f58cfadf635e9dd1e98a5a509a2973c3068fd476bc0e143779d54399c9c16"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"构造含累积梯度的势下降","source_units":[{"start":50511,"end":51364,"start_line":754,"end_line":762,"raw_tex":"Adding both sides of~\\eqref{eqn:Gki-tcG} to that of the above inequality \nand using the assumption $M\\geq 4\\ell_f L_g$, we obtain\n\\begin{eqnarray}\n\\E\\bigl[\\Phi(\\xkie)+c_{i+1}\\|G_{i+1}^k\\|^2\\bigr] \n    & \\leq & \\E\\bigl[\\Phi(\\xki)\\bigr] - \\left(\\frac{M-2\\ell_f L_g}{2M^2} - c_{i+1}(1+\\tau)\\right)\\E\\bigl[\\|\\tcGki\\|^2\\bigr] \\nonumber \\\\ \n    & & +\\left(\\frac{1}{M^2}\\left(\\frac{\\ell_f L_g}{\\sqrt{|\\cB_i^k|}}+\\frac{\\ell_f L_g}{2|\\cS_i^k|}\\right) + c_{i+1}\\left(1+\\frac{1}{\\tau}\\right)\\right)\\E\\bigl[\\|G_i^k\\|^2\\bigr] \\nonumber \\\\\n    & \\leq & \\E\\bigl[\\Phi(\\xki)\\bigr] - \\left(\\frac{1}{4M} - c_{i+1}(1+\\tau)\\right)\\E\\bigl[\\|\\tcGki\\|^2\\bigr] \\label{eqn:lyapunov-descent}\\\\ \n    & & +\\left(\\frac{1}{4M}\\left(\\frac{1}{\\sqrt{|\\cB_i^k|}}+\\frac{1}{2|\\cS_i^k|}\\right) + c_{i+1}\\left(1+\\frac{1}{\\tau}\\right)\\right)\\E\\bigl[\\|G_i^k\\|^2\\bigr]. \\nonumber\n\\end{eqnarray}\n","sha256_utf8":"624b6f570f89d808d9135838ea88c6a1d8cbc4e27a46dc46ad5359f152c3142b"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"真实梯度的期望比较界","source_units":[{"start":51364,"end":52120,"start_line":763,"end_line":777,"raw_tex":"Next, combining Lemma~\\ref{lemma:PG-finite-sum-nonsmooth} with Lemmas~\\ref{lemma:svr-f-new} yields\n\\begin{eqnarray*}\n    \\frac{M - \\ell_f L_g}{M^2}\\E\\bigl[\\|\\cGki\\|^2\\bigr]\n    &\\leq&\n    \\frac{2M+\\ell_f L_g}{M^2}\\E\\bigl[\\|\\tcGki\\|^2\\bigr] \n     + \\left( \\frac{2\\ell_f L_g}{\\sqrt{|\\cB_i^k|}} + \\frac{2\\ell_f L_g}{|\\cS_i^k|} \\right) \\E\\bigl[\\|\\xki-\\xkz\\|^2\\bigr].\n\\end{eqnarray*}\nUsing the equality $G^k_i=-M(\\xki-\\xkz)$ and the assumption $M\\geq 4\\ell_f L_g$, the above inequality implies\n\\begin{eqnarray}\n    \\frac{3}{4M}\\E\\bigl[\\|\\cGki\\|^2\\bigr]\n    &\\leq&\n    \\frac{9}{4M}\\E\\bigl[\\|\\tcGki\\|^2\\bigr] \n     +\\, \\frac{1}{2M} \\left( \\frac{1}{\\sqrt{|\\cB_i^k|}} + \\frac{1}{|\\cS_i^k|} \\right) \\E\\bigl[\\|G^k_i\\|^2\\bigr].\n    \\label{eqn:cG-bound}\n\\end{eqnarray}\n","sha256_utf8":"91ae8dde80a74d50418e96bfd13b584d044367732d3039caad5a138055dba8ef"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"消去近似梯度项","source_units":[{"start":52120,"end":52724,"start_line":778,"end_line":784,"raw_tex":"Multiplying both sides of~\\eqref{eqn:cG-bound} by $\\left(\\frac{1}{4M}-c_{i+1}(1+\\tau)\\right)\\big/\\frac{9}{4M}$, which is positive by the assumption $\\gamma>0$, and adding the resulting inequality to~\\eqref{eqn:lyapunov-descent}, we get\n\\begin{eqnarray}\n%\\label{lm:Lyapunov-finite-nonsmooth-2}\n\\E\\bigl[\\Phi(\\xkie)+c_{i+1}\\|G_{i+1}^k\\|^2\\bigr] \n&\\leq& \\E\\left[\\Phi(\\xki)+\\left( c_{i+1}\\left(1+\\frac{1}{\\tau}\\right)+ \\frac{1}{3M\\sqrt{|\\cB_i^k|}} + \\frac{1}{5M|\\cS_i^k|}\\right)\\|G_i^k\\|^2\\right] \\nonumber\\\\\n\t&& -\\frac{1}{3}\\left(\\frac{1}{4M} - c_{i+1}(1+\\tau)\\right)\\E[\\|\\cGki\\|^2].\\nonumber\n\\end{eqnarray}\n","sha256_utf8":"a858f2a581fc9f0665ddcad0b17000c4fa828f501b4b87fe6b2756670ef283c3"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"识别递推系数与势函数差","source_units":[{"start":52724,"end":52967,"start_line":785,"end_line":788,"raw_tex":"Now, using the definitions in~\\eqref{eqn:lyapunov-def} and~\\eqref{eqn:ct-finite-nonsmooth}, the above inequality is the same as\n\\[\n\t\\frac{1}{3}\\left(\\frac{1}{4M} - c_{i+1}(1+\\tau)\\right)\\E\\bigl[\\|\\cGki\\|^2\\bigr] ~\\leq~  R_i^k  - R_{i+1}^k.\n\\]\n","sha256_utf8":"f1c04ed0b2581d2c3d276a565baec412d3d1732e18ccee07adc6a12df3ec3b92"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"阶段内求和及端点消去","source_units":[{"start":52967,"end":53411,"start_line":789,"end_line":798,"raw_tex":"Recalling the definition of $\\gamma$ and summing up the above inequality over~$i$ from~$0$ to~$\\tau-1$, we get\n\\[\n\t\\gamma\\sum_{i=0}^{\\tau-1}\\E\\bigl[\\|\\cGki\\|^2\\bigr]\n    ~\\leq~ R_0^k-R_\\tau^k\n    ~=~\\E\\bigl[\\Phi(x_0^k)\\bigr] - \\E\\bigl[\\Phi(x_\\tau^k)\\bigr] ,\n\\]\nwhere the last equality is due to the observations that \n$c_\\tau=0$ and $G^k_0=0$. \nFinally, dividing both sides by $\\gamma$ and using $x^{k+1}_0 = x^k_\\tau$\ngive the desired result.\n","sha256_utf8":"2267a5e3ce79a2aaaf25cad3bbc97b50f597ad133158769aed9446cbba9be690"}]},{"id":"S054","semantic_type":"lemma","kind":"lemma","title":"批量选择保证正下降系数","source_units":[{"start":53630,"end":53927,"start_line":803,"end_line":810,"raw_tex":"\\begin{lemma}\n\\label{lemma:gamma-nonsmooth}\n    If we choose \n    $\\tau = \\bigl\\lceil\\frac{1}{2}N^{1/5}-1\\bigr\\rceil$,\n    $|\\cB_i^k| = \\bigr\\lceil 4N^{4/5}\\bigr\\rceil$ and\n    $|\\cS_i^k| = \\bigr\\lceil N^{2/5}\\bigr\\rceil$ for $i=1,\\ldots,\\tau-1$,\n    then $\\gamma  \\geq \\frac{1}{15M}$.\n\\end{lemma}","sha256_utf8":"a7addcdd88b446ce79dde6dfb7afeafaf7b503398ba0c84556b6194bcd511590"}]},{"id":"S055","semantic_type":"proof","kind":"proof","title":"下降系数下界完整证明","source_units":[{"start":53928,"end":55143,"start_line":811,"end_line":834,"raw_tex":"\\begin{proof}\nTo simplify notation, let $B=|\\Bki|$ and $S=|\\Ski|$ for $i=1,\\ldots,\\tau-1$.\nFrom \\eqref{eqn:ct-finite-nonsmooth}, we deduce\n\\[\n    (c_i+C) = (c_{i+1}+C)\\left(1+\\frac{1}{\\tau}\\right), \\qquad\n        \\mbox{where}\\quad C = \\frac{\\tau}{3M\\sqrt{B}} + \\frac{\\tau}{5MS}.\n\\] \nConsequently, with $c_\\tau=0$, we have for all $i=1,\\ldots,\\tau$, \n\\[\n    c_i ~=~ (c_\\tau + C)\\left(1+\\frac{1}{\\tau}\\right)^{\\tau-i}\\!\\! -C\n    ~\\leq~  C\\left(1+\\frac{1}{\\tau}\\right)^{\\tau} - C\n    ~\\leq~ Ce -C ~=~ C(e-1),\n\\] \nwhere the last inequality is due to the fact that \n$(1+1/\\tau)^\\tau\\leq e$ with $e$ is Euler's number (the basis of natural logarithm). Therefore, \n\\begin{eqnarray*}\n\t\\gamma  \n    & =  & \\min_{0\\leq i\\leq \\tau-1}\\frac{1}{3}\\left(\\frac{1}{4M} - c_{i+1}(1+\\tau)\\right)\\\\\n    & \\geq & \\frac{1}{3}\\left(\\frac{1}{4M} - C(e-1)(1+\\tau)\\right)\\\\\n    & = & \\frac{1}{3M}\\left(\\frac{1}{4} - \\left( \\frac{1}{3\\sqrt{B}} + \\frac{1}{5S}\\right)(e-1)\\tau(1+\\tau)\\right) \\\\\n    & \\geq & \\frac{1}{3M}\\left(\\frac{1}{4} - \\left( \\frac{1}{3\\sqrt{B}} + \\frac{1}{5S}\\right)2(1+\\tau)^2\\right).\n\\end{eqnarray*}\nFinally, setting $\\tau=\\frac{1}{2}N^{1/5}-1$, $B=4N^{4/5}$ and $S=N^{2/5}$ yields $\\gamma\\geq\\frac{1}{15M}$.\n\\end{proof}","sha256_utf8":"ed4ade629af30980d769eaea12bddf03e2d616765306d73fb94f6fb9c0868a2b"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"平移系数化为几何递推","source_units":[{"start":53942,"end":54211,"start_line":812,"end_line":817,"raw_tex":"To simplify notation, let $B=|\\Bki|$ and $S=|\\Ski|$ for $i=1,\\ldots,\\tau-1$.\nFrom \\eqref{eqn:ct-finite-nonsmooth}, we deduce\n\\[\n    (c_i+C) = (c_{i+1}+C)\\left(1+\\frac{1}{\\tau}\\right), \\qquad\n        \\mbox{where}\\quad C = \\frac{\\tau}{3M\\sqrt{B}} + \\frac{\\tau}{5MS}.\n\\] \n","sha256_utf8":"d0e8cd94967d1603bfda3cddc0fc422f8c87610ce565c71f31f9eb392eb497b3"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"求解递推并控制系数增长","source_units":[{"start":54211,"end":54586,"start_line":818,"end_line":825,"raw_tex":"Consequently, with $c_\\tau=0$, we have for all $i=1,\\ldots,\\tau$, \n\\[\n    c_i ~=~ (c_\\tau + C)\\left(1+\\frac{1}{\\tau}\\right)^{\\tau-i}\\!\\! -C\n    ~\\leq~  C\\left(1+\\frac{1}{\\tau}\\right)^{\\tau} - C\n    ~\\leq~ Ce -C ~=~ C(e-1),\n\\] \nwhere the last inequality is due to the fact that \n$(1+1/\\tau)^\\tau\\leq e$ with $e$ is Euler's number (the basis of natural logarithm). Therefore, \n","sha256_utf8":"577c362c3030cf2f8a274cacf65ab011bc8224bd873d68ad5d5e6d07706e3010"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"代入批量与阶段长度求下界","source_units":[{"start":54586,"end":55132,"start_line":826,"end_line":833,"raw_tex":"\\begin{eqnarray*}\n\t\\gamma  \n    & =  & \\min_{0\\leq i\\leq \\tau-1}\\frac{1}{3}\\left(\\frac{1}{4M} - c_{i+1}(1+\\tau)\\right)\\\\\n    & \\geq & \\frac{1}{3}\\left(\\frac{1}{4M} - C(e-1)(1+\\tau)\\right)\\\\\n    & = & \\frac{1}{3M}\\left(\\frac{1}{4} - \\left( \\frac{1}{3\\sqrt{B}} + \\frac{1}{5S}\\right)(e-1)\\tau(1+\\tau)\\right) \\\\\n    & \\geq & \\frac{1}{3M}\\left(\\frac{1}{4} - \\left( \\frac{1}{3\\sqrt{B}} + \\frac{1}{5S}\\right)2(1+\\tau)^2\\right).\n\\end{eqnarray*}\nFinally, setting $\\tau=\\frac{1}{2}N^{1/5}-1$, $B=4N^{4/5}$ and $S=N^{2/5}$ yields $\\gamma\\geq\\frac{1}{15M}$.\n","sha256_utf8":"8d5ed53cc6478bbb4827750f6e8eb260a36d7d5941c04de11a1d57ac166e658b"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"跨阶段求和与随机输出界","source_units":[{"start":56396,"end":56800,"start_line":853,"end_line":858,"raw_tex":"\tSumming up the inequality \\eqref{lm:Lyapunov-finite-nonsmooth-1} \n    over~$k$ from~$1$ to~$K$ and using the fact $\\Phi(x^K_\\tau)>\\Phi_*$, we get\n\t\\begin{eqnarray*}\n\t\t\\sum_{k=1}^K\\sum_{i=0}^{\\tau-1}\\E\\bigl[\\|\\cGki\\|^2\\bigr] ~\\leq~ \\frac{\\Phi(x_0^1) - \\Phi(x^*)}{\\gamma}.\n\t\\end{eqnarray*}\n\tBy the random choice of the output $x_{i^*}^{k^*}$, we can get the inequality \\eqref{eqn:cvg-svr-pl-nonsmooth-1}.\n","sha256_utf8":"7ef5dc836da850f9976d3437ad3e855693bd6dec4498e62ab401648344250e7f"}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","title":"按精度确定阶段数量","source_units":[{"start":56802,"end":56989,"start_line":860,"end_line":863,"raw_tex":"    To get an $\\epsilon$-stationary point in expectation, we need to set $K\\tau = \\cO(\\epsilon^{-1})$, which implies \n\\[\n    K=\\cO(\\tau^{-1}\\epsilon^{-1})=\\cO(N^{-1/5}\\epsilon^{-1}).\n\\] \n","sha256_utf8":"d4c1e216e2259218f556f0118d6caa1f7c9c297cc710ac832a335207735aae67"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"统计分量映射采样复杂度","source_units":[{"start":56989,"end":57210,"start_line":864,"end_line":869,"raw_tex":"Consequently, the sample complexity of the component mappings (the $g_i$'s) is \n\\[\n    KN + K\\tau B \n    ~=~ \\cO(N^{-1/5}\\epsilon^{-1})\\cdot N + \\cO(\\epsilon^{-1}) \\cdot 4 N^{4/5} \n    ~=~ \\cO(N+N^{4/5}\\epsilon^{-1}).\n\\]\n","sha256_utf8":"20cb9cc09a5c9a9e8153441ed794089dfa3f386de4fb396874574e27977e992a"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"统计雅可比采样复杂度","source_units":[{"start":57210,"end":57432,"start_line":870,"end_line":876,"raw_tex":"and the sample complexity for the component Jacobians is\n\\[\n    KN + K\\tau S \n    ~=~ \\cO(N^{-1/5}\\epsilon^{-1})\\cdot N + \\cO(\\epsilon^{-1}) \\cdot N^{2/5} \n    ~=~ \\cO(N+N^{4/5}\\epsilon^{-1}).\n\\]\nThis completes the proof.\n","sha256_utf8":"6ab2156345994208a1d057a5a4b491b548520b8e5348eb0614d56e7200598e25"}]},{"id":"S063","semantic_type":"assumption","kind":"assumption","title":"主定理参数与估计器条件","source_units":[{"start":55395,"end":55969,"start_line":840,"end_line":845,"raw_tex":"    Suppose Assumptions~\\ref{assumption:f-cvx}, \\ref{assumption:phi-lowerbound}  and~\\ref{assumption:g-lip-finite} hold for problem~\\eqref{eqn:composite-finite}. Let the estimates $\\tgkz$, $\\tJkz$, $\\tgki$ and $\\tJki$ in Algorithm~\\ref{algo:SVR-PL} be given in~\\eqref{eqn:finite-gk0}-\\eqref{eqn:J-svrg} respectively. If we choose $M\\geq 4\\ell_f L_g$ and $\\tau = \\bigl\\lceil\\frac{1}{2}N^{1/5}-1\\bigr\\rceil$, and\n\\[\n    |\\cB_i^k| = \\bigr\\lceil 4N^{4/5}\\bigr\\rceil, \\qquad\n    |\\cS_i^k| = \\bigr\\lceil N^{2/5}\\bigr\\rceil, \\qquad \n    i=1,\\ldots,\\tau-1,  \\qquad k=1,\\ldots,K,\n\\]\n","sha256_utf8":"4d81ad9e3107e24e0f6632110e404437d44803ad6ca02ae92935abe76f48c4bf"}]},{"id":"S064","semantic_type":"assumption","kind":"assumption","title":"阶段引理的正下降条件","source_units":[{"start":48945,"end":49004,"start_line":723,"end_line":723,"raw_tex":"In addition, we assume $M\\geq 4\\ell_f L_g$ and $\\gamma>0$.\n","sha256_utf8":"396175bdbdaca84f4bcd410e203acf1bd3b5ab491586d2d7688c5f9eb1c9ea50"}]},{"id":"S065","semantic_type":"assumption","kind":"assumption","title":"系数引理的参数选择","source_units":[{"start":53674,"end":53877,"start_line":805,"end_line":808,"raw_tex":"    If we choose \n    $\\tau = \\bigl\\lceil\\frac{1}{2}N^{1/5}-1\\bigr\\rceil$,\n    $|\\cB_i^k| = \\bigr\\lceil 4N^{4/5}\\bigr\\rceil$ and\n    $|\\cS_i^k| = \\bigr\\lceil N^{2/5}\\bigr\\rceil$ for $i=1,\\ldots,\\tau-1$,\n","sha256_utf8":"d7b80da8856f64ce372ee2c695e1eb43bf702858013b9dc1565439db339685ad"}]},{"id":"S066","semantic_type":"external_fact","kind":"external_fact","title":"光滑映射泰勒余项事实","source_units":[],"declared_external":true,"declaration_text":["If a differentiable mapping has an L-Lipschitz Jacobian along the segment joining x and y, then ||g(x)-g(y)-g'(y)(x-y)|| <= (L/2)||x-y||^2. This is the standard fact cited as OrtegaRheinboldt1970, Theorem 3.2.12."]},{"id":"S067","semantic_type":"external_fact","kind":"external_fact","title":"强凸最优性与范数不等式","source_units":[],"declared_external":true,"declaration_text":["If q is M-strongly convex and z minimizes q, then q(z)+(M/2)||y-z||^2 <= q(y). The proofs also use the triangle inequality, Cauchy–Schwarz, 2ab <= a^2/eta + eta b^2 for eta>0, and ||u+v||^2 <= (1+1/t)||u||^2+(1+t)||v||^2 for t>0."]},{"id":"S068","semantic_type":"external_fact","kind":"external_fact","title":"条件期望与独立平均事实","source_units":[],"declared_external":true,"declaration_text":["For square-integrable Euclidean random vectors, E||Z-EZ||^2=E||Z||^2-||EZ||^2. Conditionally independent centered vectors satisfy E||b^{-1} sum_j Z_j||^2=b^{-2} sum_j E||Z_j||^2. Jensen gives E||Z|| <= sqrt(E||Z||^2), and iterated conditional expectations equal the unconditional expectation."]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","title":"几何增长的指数上界","source_units":[],"declared_external":true,"declaration_text":["For t>0, (1+1/t)^t <= e, where e is Euler's number and e-1<2."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S014","kind":"supports"},{"source_id":"S003","target_id":"S015","kind":"supports"},{"source_id":"S003","target_id":"S035","kind":"supports"},{"source_id":"S004","target_id":"S041","kind":"supports"},{"source_id":"S004","target_id":"S047","kind":"supports"},{"source_id":"S004","target_id":"S059","kind":"supports"},{"source_id":"S005","target_id":"S023","kind":"supports"},{"source_id":"S005","target_id":"S026","kind":"supports"},{"source_id":"S005","target_id":"S029","kind":"supports"},{"source_id":"S005","target_id":"S043","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S007","target_id":"S023","kind":"supports"},{"source_id":"S007","target_id":"S036","kind":"supports"},{"source_id":"S008","target_id":"S059","kind":"supports"},{"source_id":"S009","target_id":"S012","kind":"supports"},{"source_id":"S009","target_id":"S053","kind":"supports"},{"source_id":"S009","target_id":"S059","kind":"supports"},{"source_id":"S009","target_id":"S061","kind":"supports"},{"source_id":"S009","target_id":"S062","kind":"supports"},{"source_id":"S010","target_id":"S032","kind":"supports"},{"source_id":"S010","target_id":"S059","kind":"supports"},{"source_id":"S011","target_id":"S060","kind":"supports"},{"source_id":"S011","target_id":"S061","kind":"supports"},{"source_id":"S011","target_id":"S062","kind":"supports"},{"source_id":"S012","target_id":"S026","kind":"supports"},{"source_id":"S012","target_id":"S032","kind":"supports"},{"source_id":"S012","target_id":"S042","kind":"supports"},{"source_id":"S012","target_id":"S044","kind":"supports"},{"source_id":"S012","target_id":"S047","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S036","kind":"supports"},{"source_id":"S013","target_id":"S038","kind":"supports"},{"source_id":"S014","target_id":"S006","kind":"supports"},{"source_id":"S014","target_id":"S036","kind":"supports"},{"source_id":"S014","target_id":"S038","kind":"supports"},{"source_id":"S015","target_id":"S035","kind":"supports"},{"source_id":"S015","target_id":"S048","kind":"supports"},{"source_id":"S015","target_id":"S061","kind":"supports"},{"source_id":"S015","target_id":"S062","kind":"supports"},{"source_id":"S016","target_id":"S035","kind":"supports"},{"source_id":"S016","target_id":"S038","kind":"supports"},{"source_id":"S016","target_id":"S061","kind":"supports"},{"source_id":"S016","target_id":"S062","kind":"supports"},{"source_id":"S017","target_id":"S047","kind":"supports"},{"source_id":"S017","target_id":"S052","kind":"supports"},{"source_id":"S018","target_id":"S052","kind":"supports"},{"source_id":"S018","target_id":"S053","kind":"supports"},{"source_id":"S018","target_id":"S056","kind":"supports"},{"source_id":"S019","target_id":"S051","kind":"supports"},{"source_id":"S019","target_id":"S053","kind":"supports"},{"source_id":"S019","target_id":"S058","kind":"supports"},{"source_id":"S020","target_id":"S063","kind":"supports"},{"source_id":"S020","target_id":"S065","kind":"supports"},{"source_id":"S021","target_id":"S041","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S050","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S029","kind":"supports"},{"source_id":"S027","target_id":"S025","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S025","kind":"supports"},{"source_id":"S028","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S025","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S025","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S025","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S025","kind":"supports"},{"source_id":"S033","target_id":"S048","kind":"supports"},{"source_id":"S033","target_id":"S050","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S034","kind":"supports"},{"source_id":"S035","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S034","kind":"supports"},{"source_id":"S039","target_id":"S048","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S040","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S040","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S040","kind":"supports"},{"source_id":"S045","target_id":"S059","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S047","target_id":"S050","kind":"supports"},{"source_id":"S047","target_id":"S053","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S046","kind":"supports"},{"source_id":"S049","target_id":"S051","kind":"supports"},{"source_id":"S050","target_id":"S046","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S046","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S046","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S046","kind":"supports"},{"source_id":"S054","target_id":"S059","kind":"supports"},{"source_id":"S055","target_id":"S054","kind":"supports"},{"source_id":"S056","target_id":"S055","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S055","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S055","kind":"supports"},{"source_id":"S059","target_id":"S002","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S002","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S060","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S002","kind":"supports"},{"source_id":"S062","target_id":"S002","kind":"supports"},{"source_id":"S063","target_id":"S059","kind":"supports"},{"source_id":"S063","target_id":"S060","kind":"supports"},{"source_id":"S063","target_id":"S061","kind":"supports"},{"source_id":"S063","target_id":"S062","kind":"supports"},{"source_id":"S064","target_id":"S049","kind":"supports"},{"source_id":"S064","target_id":"S050","kind":"supports"},{"source_id":"S064","target_id":"S051","kind":"supports"},{"source_id":"S065","target_id":"S056","kind":"supports"},{"source_id":"S065","target_id":"S058","kind":"supports"},{"source_id":"S066","target_id":"S007","kind":"supports"},{"source_id":"S067","target_id":"S026","kind":"supports"},{"source_id":"S067","target_id":"S027","kind":"supports"},{"source_id":"S067","target_id":"S029","kind":"supports"},{"source_id":"S067","target_id":"S031","kind":"supports"},{"source_id":"S067","target_id":"S043","kind":"supports"},{"source_id":"S067","target_id":"S047","kind":"supports"},{"source_id":"S068","target_id":"S035","kind":"supports"},{"source_id":"S068","target_id":"S036","kind":"supports"},{"source_id":"S068","target_id":"S037","kind":"supports"},{"source_id":"S068","target_id":"S038","kind":"supports"},{"source_id":"S068","target_id":"S048","kind":"supports"},{"source_id":"S068","target_id":"S050","kind":"supports"},{"source_id":"S069","target_id":"S057","kind":"supports"},{"source_id":"S069","target_id":"S058","kind":"supports"}]}}
{"record_id":"2006.00425v3","paper":{"title":"Momentum-based variance-reduced proximal stochastic gradient method for composite nonconvex stochastic optimization","authors":["Yangyang Xu","Yibo Xu"],"arxiv_id":"2006.00425","version":"v3","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-022-02132-w","abstract_url":"https://arxiv.org/abs/2006.00425v3","pdf_url":"https://arxiv.org/pdf/2006.00425v3"},"context":{"raw_tex":"In this paper, we consider the regularized nonconvex stochastic programming\n\\begin{equation}\\label{eq:stoc-prob}\n\\Phi^*= \\Min_{\\vx\\in\\RR^n} ~\\Phi(\\vx):= \\big\\{F(\\vx)\\equiv\\EE_\\xi [f(\\vx;\\xi)]\\big\\} + r(\\vx), \n\\end{equation}\n\nwhere $f(\\,\\cdot\\,; \\xi)$ is a smooth nonconvex function almost surely for $\\xi$, and $r$ is a closed convex function on $\\RR^n$. Examples of \\eqref{eq:stoc-prob} include the sparse online matrix factorization \\cite{mairal2010online}, the online nonnegative matrix factorization \\cite{zhao2016online}, and the streaming PCA (by a unit-ball constraint) \\cite{mitliagkas2013memory}. In addition, as $\\xi$ follows a uniform distribution on a finite set $\\Xi=\\{\\xi_1,\\ldots,\\xi_N\\}$, \\eqref{eq:stoc-prob} recovers the so-called finite-sum structured problem. It includes most regularized machine learning problems such as the sparse bilinear logistic regression \\cite{shi2014sparse}, the sparse convolutional neural network \\cite{liu2015sparse}, and the group sparse regularized deep neural networks \\cite{scardapane2017group}. \n\n\nLet $w$ be a continuously differentiable and 1-strongly convex function on $\\dom(r)$, i.e., \n$$w(\\vy) \\ge w(\\vx) + \\langle \\nabla w(\\vx), \\vy-\\vx\\rangle + \\frac{1}{2}\\|\\vy-\\vx\\|^2,\\, \\forall\\, \\vx,\\vy\\in \\dom(r).$$\nThe Bregman divergence induced by $w$ is defined as\n\\begin{equation}\nV(\\vx,\\vz)=w(\\vx)-w(\\vz) - \\langle \\nabla w(\\vz), \\vx-\\vz\\rangle.\n\\end{equation}\n\n\n\\begin{algorithm}[h]\n\\caption{Momentum-based variance-reduced proximal stochastic gradient method for \\eqref{eq:stoc-prob}}\\label{alg:prox-storm}\n\\DontPrintSemicolon\n\\textbf{Input:} max iteration numer $K$, minibatch size $m$, and positive sequences $\\{\\beta_k\\}\\subseteq(0,1)$ and $\\{\\eta_k\\}$.\\;\n\\textbf{Initialization:} choose $\\vx^0\\in\\dom(r)$ and let $\\vd^0 = \\frac{1}{m_0}\\sum_{\\xi\\in B_0}\\nabla f(\\vx^0;\\xi)$ with $m_0$ i.i.d. samples $B_0=\\{\\xi_1^0,\\ldots,\\xi_{m_0}^0\\}$\\;\n\\For{$k=0, 1,\\ldots, K-1$}{\nUpdate $\\vx$ by\n\\begin{equation}\\label{eq:update-x}\n\\vx^{k+1}=\\argmin_{\\vx} \\left\\{\\langle \\vd^k, \\vx\\rangle + \\frac{1}{\\eta_k}V(\\vx,\\vx^k) + r(\\vx)\\right\\}.\\vspace{-0.3cm}\n\\end{equation}\n\\;\nObtain $m$ i.i.d. samples $B_{k+1}=\\{\\xi_1^{k+1},\\ldots,\\xi_m^{k+1}\\}$ and let \n\\begin{equation}\\label{eq:def-v-u-k}\n\\textstyle \\vv^{k+1} = \\frac{1}{m}\\sum_{\\xi\\in B_{k+1}}\\nabla f(\\vx^{k+1}; \\xi),\\quad \\vu^{k+1} = \\frac{1}{m}\\sum_{\\xi\\in B_{k+1}}\\nabla f(\\vx^{k}; \\xi).\\vspace{-0.3cm}\n\\end{equation}\\;\nLet $\\vd^{k+1} = \\vv^{k+1} + (1-\\beta_{k})(\\vd^{k}-\\vu^{k+1})$.\\;\n}\nReturn $\\vx^\\tau$ with $\\tau$ selected from $\\{0,1,\\ldots,K-1\\}$ uniformly at random or by the distribution\n\\begin{equation}\\label{eq:select-tau}\n\\textstyle \\Prob(\\tau = k) = \\frac{\\frac{\\eta_k}{4}(1-\\eta_k L)-\\frac{\\eta_k^2}{5m\\eta_{k+1}}(1-\\beta_k)^2}{\\sum_{j=0}^{K-1}\\left(\\frac{\\eta_j}{4}(1-\\eta_j L)-\\frac{\\eta_j^2}{5m\\eta_{j+1}}(1-\\beta_j)^2\\right)},\\, k = 0,1,\\ldots,K-1.\n\\end{equation}\n\\vspace{-0.2cm}\n\\end{algorithm}\n\nWe use bold lowercase letters $\\vx,\\vy,\\vg,\\ldots$ for vectors. $\\EE_{B_{k}}$ denotes the expectation about a mini-batch set $B_{k}$ conditionally on the all previous history, and $\\EE$ denotes the full expectation. $|B_k|$ counts the number of elements in the set $B_k$. We use $\\|\\cdot\\|$ for the Euclidean norm. A differentiable function $F$ is called $L$-smooth, if $\\|\\nabla F(\\vx)-\\nabla F(\\vy)\\|\\le L\\|\\vx-\\vy\\|$ for all $\\vx$ and $\\vy$.\n\n\n\\begin{definition}[proximal gradient mapping]\\label{def:prox-map}\nGiven $\\vd$, $\\vx\\in\\dom(r)$, and $\\eta>0$, we define $P(\\vx,\\vd,\\eta)=\\frac{1}{\\eta}(\\vx-\\vx^+)$, where \n%\\begin{equation*}\n$\\vx^+ =\\argmin_\\vy \\textstyle \\left\\{\\langle \\vd, \\vy\\rangle  +\\frac{1}{\\eta}V(\\vy,\\vx) + r(\\vy)\\right\\}.$\n%\\end{equation*}\n\\end{definition}\n\nFrom \\cite[Lemma 1]{ghadimi2016mini}, it holds\n\\begin{equation}\\label{eq:lem1-lan}\n\\big\\langle \\vd, P(\\vx,\\vd,\\eta)\\big\\rangle \\ge \\|P(\\vx,\\vd,\\eta)\\|^2 + \\frac{1}{\\eta}\\big(r(\\vx^+)-r(\\vx)\\big).\n\\end{equation}\n\nIn addition, the proximal gradient mapping is nonexpansive from \\cite[Proposition 1]{ghadimi2016mini}, i.e.,\n\\begin{equation}\\label{eq:nonexp-P}\n\\|P(\\vx, \\vd_1, \\eta) - P(\\vx, \\vd_2, \\eta)\\| \\le \\|\\vd_1-\\vd_2\\|, \\ \\forall\\, \\vd_1,\\vd_2, \\, \\forall\\, \\vx\\in \\dom(r), \\, \\forall\\, \\eta > 0.\n\\end{equation}\n\nFor each $k\\ge0$, we denote\n\\begin{equation}\\label{eq:vg-vgbar}\n\\vg^k = P(\\vx^k, \\vd^k, \\eta_k),\\quad \\bar\\vg^k = P(\\vx^k, \\nabla F(\\vx^k), \\eta_k).\n\\end{equation}\n\nNotice that $\\|\\bar\\vg^k\\|$ measures the violation of stationarity of $\\vx^k$. The gradient error is represented by\n\\begin{equation}\\label{eq:error-grad}\n\\ve^k = \\vd^k - \\nabla F(\\vx^k).\n\\end{equation}\n\n\\begin{assumption}[finite optimal objective]\\label{assump-obj}\nThe optimal objective value $\\Phi^*$ of \\eqref{eq:stoc-prob} is finite.\n\\end{assumption}\n\n\\begin{assumption}[mean-squared smoothness]\\label{assump-smooth}\nThe function $f(\\,\\cdot\\,;\\xi)$ satisfies the mean-squared smoothness condition:  \n$\\EE_\\xi \\big[\\|\\nabla f(\\vx;\\xi) - \\nabla f(\\vy;\\xi)\\|^2\\big] \\le L^2 \\|\\vx-\\vy\\|^2,\\, \\forall\\, \\vx,\\vy\\in \\dom(r).$\n\\end{assumption}\n\n\\begin{assumption}[unbiasedness and variance boundedness]\\label{assump-sgd}\nThere is $\\sigma>0$ such that %for each $\\vx\\in \\dom(r)$, \n\\begin{align}\n\\EE_\\xi [\\nabla f(\\vx; \\xi)] = \\nabla F(\\vx),\\quad \\EE[\\|\\nabla f(\\vx; \\xi) - \\nabla F(\\vx)\\|^2] \\le \\sigma^2, \\ \\forall\\, \\vx\\in \\dom(r).\n%&\\EE_{B_k}[\\vv^k] = \\nabla F(\\vx^k),\\quad \\EE_{B_k}[\\vu^k] = \\nabla F(\\vx^{k-1}),\\\\\n%&\\EE[\\|\\vv^k-\\nabla F(\\vx^k)\\|^2] \\le \\sigma^2.\n\\end{align}\n\\end{assumption}\n\nIt is easy to show that under Assumptions~\\ref{assump-smooth} and \\ref{assump-sgd}, the function $F(\\vx)=\\EE_\\xi [f(\\vx;\\xi)]$ is $L$-smooth; see the arguments at the end of section~2.2 of \\cite{tran2021hybrid}. \n\n\n\\begin{equation}\\label{eq:gamma}\n\t\\Gamma_k = \\left\\{\n\t\\begin{array}{ll}\n\t\t\\prod_{i=0}^{k-1}(1-\\beta_i)^2, & \\text{ if } k\\ge 1, \\\\[0.2cm]\n\t\t1, & \\text{ if } k=0.\n\t\\end{array}\n\t\\right.\n\\end{equation}","units":[{"start":10873,"end":11096,"start_line":305,"end_line":308,"raw_tex":"In this paper, we consider the regularized nonconvex stochastic programming\n\\begin{equation}\\label{eq:stoc-prob}\n\\Phi^*= \\Min_{\\vx\\in\\RR^n} ~\\Phi(\\vx):= \\big\\{F(\\vx)\\equiv\\EE_\\xi [f(\\vx;\\xi)]\\big\\} + r(\\vx), \n\\end{equation}","sha256_utf8":"9850027c6cac9ee5e877a64c58f4f396c97b3bc85b9c8e2545170028858ab48e"},{"start":11097,"end":11922,"start_line":309,"end_line":309,"raw_tex":"where $f(\\,\\cdot\\,; \\xi)$ is a smooth nonconvex function almost surely for $\\xi$, and $r$ is a closed convex function on $\\RR^n$. Examples of \\eqref{eq:stoc-prob} include the sparse online matrix factorization \\cite{mairal2010online}, the online nonnegative matrix factorization \\cite{zhao2016online}, and the streaming PCA (by a unit-ball constraint) \\cite{mitliagkas2013memory}. In addition, as $\\xi$ follows a uniform distribution on a finite set $\\Xi=\\{\\xi_1,\\ldots,\\xi_N\\}$, \\eqref{eq:stoc-prob} recovers the so-called finite-sum structured problem. It includes most regularized machine learning problems such as the sparse bilinear logistic regression \\cite{shi2014sparse}, the sparse convolutional neural network \\cite{liu2015sparse}, and the group sparse regularized deep neural networks \\cite{scardapane2017group}. \n","sha256_utf8":"d5d55de741b4d54139476875fed5151856b51f916e087a238cbf1ea171a90c8f"},{"start":14454,"end":14819,"start_line":316,"end_line":321,"raw_tex":"Let $w$ be a continuously differentiable and 1-strongly convex function on $\\dom(r)$, i.e., \n$$w(\\vy) \\ge w(\\vx) + \\langle \\nabla w(\\vx), \\vy-\\vx\\rangle + \\frac{1}{2}\\|\\vy-\\vx\\|^2,\\, \\forall\\, \\vx,\\vy\\in \\dom(r).$$\nThe Bregman divergence induced by $w$ is defined as\n\\begin{equation}\nV(\\vx,\\vz)=w(\\vx)-w(\\vz) - \\langle \\nabla w(\\vz), \\vx-\\vz\\rangle.\n\\end{equation}\n","sha256_utf8":"82c517a83e2194642a8ae0de059fae47305a89fae88ceaae90e6079d2b92b7b6"},{"start":15485,"end":16981,"start_line":324,"end_line":346,"raw_tex":"\\begin{algorithm}[h]\n\\caption{Momentum-based variance-reduced proximal stochastic gradient method for \\eqref{eq:stoc-prob}}\\label{alg:prox-storm}\n\\DontPrintSemicolon\n\\textbf{Input:} max iteration numer $K$, minibatch size $m$, and positive sequences $\\{\\beta_k\\}\\subseteq(0,1)$ and $\\{\\eta_k\\}$.\\;\n\\textbf{Initialization:} choose $\\vx^0\\in\\dom(r)$ and let $\\vd^0 = \\frac{1}{m_0}\\sum_{\\xi\\in B_0}\\nabla f(\\vx^0;\\xi)$ with $m_0$ i.i.d. samples $B_0=\\{\\xi_1^0,\\ldots,\\xi_{m_0}^0\\}$\\;\n\\For{$k=0, 1,\\ldots, K-1$}{\nUpdate $\\vx$ by\n\\begin{equation}\\label{eq:update-x}\n\\vx^{k+1}=\\argmin_{\\vx} \\left\\{\\langle \\vd^k, \\vx\\rangle + \\frac{1}{\\eta_k}V(\\vx,\\vx^k) + r(\\vx)\\right\\}.\\vspace{-0.3cm}\n\\end{equation}\n\\;\nObtain $m$ i.i.d. samples $B_{k+1}=\\{\\xi_1^{k+1},\\ldots,\\xi_m^{k+1}\\}$ and let \n\\begin{equation}\\label{eq:def-v-u-k}\n\\textstyle \\vv^{k+1} = \\frac{1}{m}\\sum_{\\xi\\in B_{k+1}}\\nabla f(\\vx^{k+1}; \\xi),\\quad \\vu^{k+1} = \\frac{1}{m}\\sum_{\\xi\\in B_{k+1}}\\nabla f(\\vx^{k}; \\xi).\\vspace{-0.3cm}\n\\end{equation}\\;\nLet $\\vd^{k+1} = \\vv^{k+1} + (1-\\beta_{k})(\\vd^{k}-\\vu^{k+1})$.\\;\n}\nReturn $\\vx^\\tau$ with $\\tau$ selected from $\\{0,1,\\ldots,K-1\\}$ uniformly at random or by the distribution\n\\begin{equation}\\label{eq:select-tau}\n\\textstyle \\Prob(\\tau = k) = \\frac{\\frac{\\eta_k}{4}(1-\\eta_k L)-\\frac{\\eta_k^2}{5m\\eta_{k+1}}(1-\\beta_k)^2}{\\sum_{j=0}^{K-1}\\left(\\frac{\\eta_j}{4}(1-\\eta_j L)-\\frac{\\eta_j^2}{5m\\eta_{j+1}}(1-\\beta_j)^2\\right)},\\, k = 0,1,\\ldots,K-1.\n\\end{equation}\n\\vspace{-0.2cm}\n\\end{algorithm}","sha256_utf8":"abbc025bf73abbcfc9fc9016619e6aaae1821ff97570dad751993e28673b00ea"},{"start":30356,"end":30801,"start_line":435,"end_line":435,"raw_tex":"We use bold lowercase letters $\\vx,\\vy,\\vg,\\ldots$ for vectors. $\\EE_{B_{k}}$ denotes the expectation about a mini-batch set $B_{k}$ conditionally on the all previous history, and $\\EE$ denotes the full expectation. $|B_k|$ counts the number of elements in the set $B_k$. We use $\\|\\cdot\\|$ for the Euclidean norm. A differentiable function $F$ is called $L$-smooth, if $\\|\\nabla F(\\vx)-\\nabla F(\\vy)\\|\\le L\\|\\vx-\\vy\\|$ for all $\\vx$ and $\\vy$.\n","sha256_utf8":"c01a82dd29becaba8ad6ec8d527f9631ad46afd6c62825b140fbb93faa866d92"},{"start":30802,"end":31134,"start_line":437,"end_line":442,"raw_tex":"\\begin{definition}[proximal gradient mapping]\\label{def:prox-map}\nGiven $\\vd$, $\\vx\\in\\dom(r)$, and $\\eta>0$, we define $P(\\vx,\\vd,\\eta)=\\frac{1}{\\eta}(\\vx-\\vx^+)$, where \n%\\begin{equation*}\n$\\vx^+ =\\argmin_\\vy \\textstyle \\left\\{\\langle \\vd, \\vy\\rangle  +\\frac{1}{\\eta}V(\\vy,\\vx) + r(\\vy)\\right\\}.$\n%\\end{equation*}\n\\end{definition}","sha256_utf8":"fcf48b267a8e7e584bb3a743724f9ab3566f7e5100eef267465051e1c2cf857f"},{"start":31819,"end":32029,"start_line":450,"end_line":453,"raw_tex":"From \\cite[Lemma 1]{ghadimi2016mini}, it holds\n\\begin{equation}\\label{eq:lem1-lan}\n\\big\\langle \\vd, P(\\vx,\\vd,\\eta)\\big\\rangle \\ge \\|P(\\vx,\\vd,\\eta)\\|^2 + \\frac{1}{\\eta}\\big(r(\\vx^+)-r(\\vx)\\big).\n\\end{equation}","sha256_utf8":"e403ca54fc5e30d5540810d498d897828b9cc2f6f051d470cf9a9df8c5670064"},{"start":32030,"end":32333,"start_line":454,"end_line":457,"raw_tex":"In addition, the proximal gradient mapping is nonexpansive from \\cite[Proposition 1]{ghadimi2016mini}, i.e.,\n\\begin{equation}\\label{eq:nonexp-P}\n\\|P(\\vx, \\vd_1, \\eta) - P(\\vx, \\vd_2, \\eta)\\| \\le \\|\\vd_1-\\vd_2\\|, \\ \\forall\\, \\vd_1,\\vd_2, \\, \\forall\\, \\vx\\in \\dom(r), \\, \\forall\\, \\eta > 0.\n\\end{equation}","sha256_utf8":"18a066a07814365d6a3c121e316e46df9dc40f6de478d4c1b1752399633f0f00"},{"start":32334,"end":32497,"start_line":458,"end_line":461,"raw_tex":"For each $k\\ge0$, we denote\n\\begin{equation}\\label{eq:vg-vgbar}\n\\vg^k = P(\\vx^k, \\vd^k, \\eta_k),\\quad \\bar\\vg^k = P(\\vx^k, \\nabla F(\\vx^k), \\eta_k).\n\\end{equation}","sha256_utf8":"cd4bf5446086a2fe2906fd7df08625b5b336f98f3124775a0077c09b536f6713"},{"start":32498,"end":32699,"start_line":462,"end_line":465,"raw_tex":"Notice that $\\|\\bar\\vg^k\\|$ measures the violation of stationarity of $\\vx^k$. The gradient error is represented by\n\\begin{equation}\\label{eq:error-grad}\n\\ve^k = \\vd^k - \\nabla F(\\vx^k).\n\\end{equation}","sha256_utf8":"6722ee61072d533539b12647317637a2b00473ee579799a391d6d9f8d3e25a66"},{"start":33459,"end":33610,"start_line":474,"end_line":476,"raw_tex":"\\begin{assumption}[finite optimal objective]\\label{assump-obj}\nThe optimal objective value $\\Phi^*$ of \\eqref{eq:stoc-prob} is finite.\n\\end{assumption}","sha256_utf8":"63aec75d3374131ff1d4b0a87f5304856df844708ba0754e7b05405a6ee89c89"},{"start":33612,"end":33895,"start_line":478,"end_line":481,"raw_tex":"\\begin{assumption}[mean-squared smoothness]\\label{assump-smooth}\nThe function $f(\\,\\cdot\\,;\\xi)$ satisfies the mean-squared smoothness condition:  \n$\\EE_\\xi \\big[\\|\\nabla f(\\vx;\\xi) - \\nabla f(\\vy;\\xi)\\|^2\\big] \\le L^2 \\|\\vx-\\vy\\|^2,\\, \\forall\\, \\vx,\\vy\\in \\dom(r).$\n\\end{assumption}","sha256_utf8":"05b62f4a26d2d85a50001b929e54e26caa69a79a38485ea09aed50a1ea0f3b5a"},{"start":33897,"end":34347,"start_line":483,"end_line":490,"raw_tex":"\\begin{assumption}[unbiasedness and variance boundedness]\\label{assump-sgd}\nThere is $\\sigma>0$ such that %for each $\\vx\\in \\dom(r)$, \n\\begin{align}\n\\EE_\\xi [\\nabla f(\\vx; \\xi)] = \\nabla F(\\vx),\\quad \\EE[\\|\\nabla f(\\vx; \\xi) - \\nabla F(\\vx)\\|^2] \\le \\sigma^2, \\ \\forall\\, \\vx\\in \\dom(r).\n%&\\EE_{B_k}[\\vv^k] = \\nabla F(\\vx^k),\\quad \\EE_{B_k}[\\vu^k] = \\nabla F(\\vx^{k-1}),\\\\\n%&\\EE[\\|\\vv^k-\\nabla F(\\vx^k)\\|^2] \\le \\sigma^2.\n\\end{align}\n\\end{assumption}","sha256_utf8":"b5d2c9bfe92c6612460139ff88cbb0a90f0a3b9bbb96709a2fd985aa8311fc41"},{"start":34349,"end":34562,"start_line":492,"end_line":492,"raw_tex":"It is easy to show that under Assumptions~\\ref{assump-smooth} and \\ref{assump-sgd}, the function $F(\\vx)=\\EE_\\xi [f(\\vx;\\xi)]$ is $L$-smooth; see the arguments at the end of section~2.2 of \\cite{tran2021hybrid}. \n","sha256_utf8":"51edad21ff1b7847d55097515dbdefd502b29248985412476d200a2fc2ecec64"},{"start":57915,"end":58113,"start_line":735,"end_line":742,"raw_tex":"\\begin{equation}\\label{eq:gamma}\n\t\\Gamma_k = \\left\\{\n\t\\begin{array}{ll}\n\t\t\\prod_{i=0}^{k-1}(1-\\beta_i)^2, & \\text{ if } k\\ge 1, \\\\[0.2cm]\n\t\t1, & \\text{ if } k=0.\n\t\\end{array}\n\t\\right.\n\\end{equation}","sha256_utf8":"25dfd7b13baa5e2be01090b6d89c6c7f926b4e95903696fb53fe763ffd8cecaf"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}[convergence rate II with constant stepsizes]\\label{thm:rate-const2}\n\tUnder Assumptions~\\ref{assump-obj} through \\ref{assump-sgd}, let $\\{\\vx^k\\}$ be the iterate sequence from Algorithm~\\ref{alg:prox-storm} with $ \\eta_k =\\frac{\\eta}{L \\sqrt[3]{K}}$ and $\\{\\beta_k\\}$ set by \\eqref{eq:beta-def}, \n\twhere $\\eta \\le \\frac{1}{4}$ is a positive number. If $\\tau$ is selected from $\\{0,1,\\ldots,K-1\\}$ uniformly at random, then\n\t\\begin{equation}\\label{eq:rate-grad-const2}\n\t\t\\EE[\\|\\bar\\vg^\\tau\\|^2] \\le \\frac{1}{K^{\\frac{2}{3}}} \n\t\t\\left(\t\\frac{12 L}{\\eta}\\big[\\Phi(\\vx^0)-\\Phi^*\\big] + \\Big( 2^{-1/3}+\\frac{1}{6}2^{1/3}+\\frac{1}{36}\\Big) \\frac{8}{\\sqrt[3]{K}} \\frac{\\sigma^2}{ m_0} +  \\frac{12\\sigma^2 L}{ \\eta m_0}\n\t\t+ \\frac{ 32}{(1-2^{-2/3})^2} \\frac{\\sigma^2}{m}\\right).\n\t\\end{equation}\n\\end{theorem}","units":[{"start":67086,"end":67900,"start_line":863,"end_line":871,"raw_tex":"\\begin{theorem}[convergence rate II with constant stepsizes]\\label{thm:rate-const2}\n\tUnder Assumptions~\\ref{assump-obj} through \\ref{assump-sgd}, let $\\{\\vx^k\\}$ be the iterate sequence from Algorithm~\\ref{alg:prox-storm} with $ \\eta_k =\\frac{\\eta}{L \\sqrt[3]{K}}$ and $\\{\\beta_k\\}$ set by \\eqref{eq:beta-def}, \n\twhere $\\eta \\le \\frac{1}{4}$ is a positive number. If $\\tau$ is selected from $\\{0,1,\\ldots,K-1\\}$ uniformly at random, then\n\t\\begin{equation}\\label{eq:rate-grad-const2}\n\t\t\\EE[\\|\\bar\\vg^\\tau\\|^2] \\le \\frac{1}{K^{\\frac{2}{3}}} \n\t\t\\left(\t\\frac{12 L}{\\eta}\\big[\\Phi(\\vx^0)-\\Phi^*\\big] + \\Big( 2^{-1/3}+\\frac{1}{6}2^{1/3}+\\frac{1}{36}\\Big) \\frac{8}{\\sqrt[3]{K}} \\frac{\\sigma^2}{ m_0} +  \\frac{12\\sigma^2 L}{ \\eta m_0}\n\t\t+ \\frac{ 32}{(1-2^{-2/3})^2} \\frac{\\sigma^2}{m}\\right).\n\t\\end{equation}\n\\end{theorem}","sha256_utf8":"96951cdf0a01d788ade908f623e93e084743a510bdbba6ae83c39cd12b3e84a4"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}[one-iteration progress]\\label{lem:obj-diff}\nLet $\\{\\vx^k\\}$ be generated from Algorithm~\\ref{alg:prox-storm}. If $F$ is $L$-smooth, then\n$$\\Phi(\\vx^{k+1})-\\Phi(\\vx^k) \\le \\frac{\\eta_k}{2}(2-\\eta_k L)\\|\\ve^k\\|^2 - \\frac{\\eta_k}{4}(1-\\eta_k L)\\|\\bar\\vg^k\\|^2,\\, \\forall\\, k\\ge 1,$$\nwhere $\\bar\\vg^k$ is defined in \\eqref{eq:vg-vgbar}.\n\\end{lemma}\n\n\\begin{proof}\nBy the $L$-smoothness of $F$ and the definition of $\\vg^k$ in \\eqref{eq:vg-vgbar}, we have\n\\begin{equation}\\label{eq:lip-F-ineq}\nF(\\vx^{k+1})-F(\\vx^k)\\le \\langle \\nabla F(\\vx^k), \\vx^{k+1}-\\vx^k\\rangle + \\frac{L}{2}\\|\\vx^{k+1}-\\vx^k\\|^2 = -\\eta_k\\langle \\nabla F(\\vx^k), \\vg^k\\rangle + \\frac{\\eta_k^2L}{2}\\|\\vg^k\\|^2.\n\\end{equation}\nUsing the definition of $\\ve^k$ in \\eqref{eq:error-grad} and the inequality in \\eqref{eq:lem1-lan}, we have \n$$-\\langle \\nabla F(\\vx^k), \\vg^k\\rangle = \\langle \\ve^k, \\vg^k\\rangle - \\langle \\vd^k, \\vg^k\\rangle \\le \\langle \\ve^k, \\vg^k\\rangle - \\|\\vg^k\\|^2 + \\frac{1}{\\eta_k}\\big(r(\\vx^k)-r(\\vx^{k+1})\\big).$$\nPlugging the above inequality into \\eqref{eq:lip-F-ineq} and rearranging terms give\n$$\\Phi(\\vx^{k+1})-\\Phi(\\vx^k) \\le \\eta_k\\langle \\ve^k, \\vg^k\\rangle - \\eta_k \\|\\vg^k\\|^2+ \\frac{\\eta_k^2L}{2}\\|\\vg^k\\|^2.$$\nBy the Cauchy-Schwartz inequality, it holds $\\eta_k\\langle \\ve^k, \\vg^k\\rangle \\le \\frac{\\eta_k}{2}\\|\\ve^k\\|^2 + \\frac{\\eta_k}{2}\\|\\vg^k\\|^2$, which together with the above inequality implies\n\\begin{equation}\\label{eq:chg-obj-0}\n\\Phi(\\vx^{k+1})-\\Phi(\\vx^k) \\le \\frac{\\eta_k}{2}\\|\\ve^k\\|^2 - \\frac{\\eta_k}{2}(1-\\eta_k L)\\|\\vg^k\\|^2.\n\\end{equation}\nFrom \\eqref{eq:nonexp-P} and the definitions of $\\vg^k$ and $\\bar\\vg^k$ in \\eqref{eq:vg-vgbar}, it follows\n\\begin{equation}\\label{eq:tri-g}\n-\\|\\vg^k\\|^2 \\le -\\frac{1}{2}\\|\\bar\\vg^k\\|^2 + \\|\\vg^k-\\bar\\vg^k\\|^2\\le -\\frac{1}{2}\\|\\bar\\vg^k\\|^2 + \\|\\vd^k -\\nabla F(\\vx^k)\\|^2 = -\\frac{1}{2}\\|\\bar\\vg^k\\|^2 + \\|\\ve^k\\|^2.\n\\end{equation}\nNow plug the above inequality into \\eqref{eq:chg-obj-0} to give the desired result.\n\\end{proof}\n\n\\begin{lemma}[recursive bound on gradient error]\\label{lem:bd-ek}\nUnder Assumptions~\\ref{assump-smooth} and \\ref{assump-sgd}, it holds %for each $k\\ge 0$ that\n$$\\EE\\big[\\|\\ve^{k+1}\\|^2\\big] \\le \\frac{2\\beta_k^2\\sigma^2}{m} + \\frac{4(1-\\beta_k)^2\\eta_{k}^2 L^2}{m} \\EE\\big[\\|\\bar\\vg^{k}\\|^2\\big] + (1-\\beta_k)^2\\left(\\textstyle 1+\\frac{4\\eta_{k}^2 L^2}{m}\\right)\\EE\\big[\\|\\ve^{k}\\|^2\\big], \\forall\\, k\\ge0,$$\nwhere $\\bar\\vg^k$ and $\\ve^k$ are defined in \\eqref{eq:vg-vgbar} and \\eqref{eq:error-grad}.\n\\end{lemma}\n\n\\begin{proof}\nFirst, notice $\\EE_{B_{k+1}}[\\langle \\vv^{k+1}, \\ve^{k}\\rangle] = \\langle\\nabla F(\\vx^{k+1}), \\ve^{k}\\rangle$ and $\\EE_{B_{k+1}}[\\langle \\vu^{k+1}, \\ve^{k}\\rangle] = \\langle\\nabla F(\\vx^{k}), \\ve^{k}\\rangle$, and thus\n\\begin{equation}\\label{eq:crs-term0}\n\\EE_{B_{k+1}}[\\langle \\vv^{k+1} - \\nabla F(\\vx^{k+1}), \\ve^{k}\\rangle] = 0,\\quad \\EE_{B_{k+1}}[\\langle \\vu^{k+1} - \\nabla F(\\vx^{k}), \\ve^{k}\\rangle] = 0.\n\\end{equation}\nHence, by writing $\\ve^{k+1} = \\vv^{k+1} - \\nabla F(\\vx^{k+1}) + (1-\\beta_k)(\\nabla F(\\vx^{k})- \\vu^{k+1})+(1-\\beta_k)\\ve^{k}$, we have\n\\begin{equation}\\label{eq:one-step-e}\n\\EE_{B_{k+1}}\\big[\\|\\ve^{k+1}\\|^2\\big] = \\EE_{B_{k+1}}\\big[\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1}) + (1-\\beta_k)(\\nabla F(\\vx^{k})- \\vu^{k+1})\\|^2\\big] + (1-\\beta_k)^2\\|\\ve^{k}\\|^2.\n\\end{equation}\nBy the Young's inequality, it holds\n\\begin{align}\\label{eq:step2}\n&~\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1}) + (1-\\beta_k)(\\nabla F(\\vx^{k})- \\vu^{k+1})\\|^2 \\cr\n= &~ \\big\\|\\beta_k\\big(\\vv^{k+1} - \\nabla F(\\vx^{k+1})\\big) + (1-\\beta_k)\\big(\\vv^{k+1} - \\nabla F(\\vx^{k+1}) + \\nabla F(\\vx^{k})- \\vu^{k+1}\\big)\\big\\|^2\\cr\n\\le &~ 2\\beta_k^2\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1})\\|^2 + 2(1-\\beta_k)^2\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1}) + \\nabla F(\\vx^{k})- \\vu^{k+1}\\|^2.\n\\end{align}\nFrom the definition of $\\vv^{k+1}$ and $ \\vu^{k+1}$ in \\eqref{eq:def-v-u-k}, we have \n\\begin{align}\\label{eq:step23}\n \t&~\\EE_{B_{k+1}}\\big[\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1}) + \\nabla F(\\vx^{k})- \\vu^{k+1}\\|^2\\big] \\cr \n \t=&~\\frac{1}{m^2}\\EE_{B_{k+1}}\\left\\| \\sum_{\\xi\\in B_{k+1}}\\left(\\nabla f(\\vx^{k+1};\\xi)-\\nabla f(\\vx^{k};\\xi)  - \\nabla F(\\vx^{k+1})+\\nabla F(\\vx^{k})\\right) \\right\\|^2 \t \\cr\n \t=&~\\frac{1}{m^2}\\sum_{j=1}^m\\EE_{\\xi_j^{k+1}}\\left\\| \\nabla f(\\vx^{k+1};\\xi_j^{k+1})-\\nabla f(\\vx^{k};\\xi_j^{k+1})  - \\nabla F(\\vx^{k+1})+\\nabla F(\\vx^{k})  \\right\\|^2 \t \\cr\n \t\\le&~\\frac{1}{m^2}\\sum_{j=1}^m\\EE_{\\xi_j^{k+1}}\\left\\| \\nabla f(\\vx^{k+1};\\xi_j^{k+1})-\\nabla f(\\vx^{k};\\xi_j^{k+1}) \\right\\|^2 \t \\cr\t\n \t\\le&~\\frac{L^2}{m}\\|\\vx^{k+1}-\\vx^{k}\\|^2,\n\\end{align}\nwhere the second equality holds because of the i.i.d. samples in $B_{k+1}$ and the zero mean of the random vector $\\nabla f(\\vx^{k+1};\\xi_j^{k+1})-\\nabla f(\\vx^{k};\\xi_j^{k+1})  - \\nabla F(\\vx^{k+1})+\\nabla F(\\vx^{k}) $ resulted from unbiasedness in Assumption~\\ref{assump-sgd}, \nthe first inequality is due to the fact that the variance of a random vector is upper-bounded by its second moment, and the last inequality follows from Assumption~\\ref{assump-smooth}.\n\nNow, take conditional expectation on both sides of \\eqref{eq:step2}, use \\eqref{eq:step23}, and substitute it into \\eqref{eq:one-step-e}. We have\n$$\\EE_{B_{k+1}}\\big[\\|\\ve^{k+1}\\|^2\\big] \\le (1-\\beta_k)^2\\|\\ve^{k}\\|^2 +2\\beta_k^2\\EE_{B_{k+1}} \\big[\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1})\\|^2\\big] + \\frac{2(1-\\beta_k)^2 L^2}{m} \\EE_{B_{k+1}}\\big[\\|\\vx^{k+1}-\\vx^{k}\\|^2\\big] .$$\nTaking a full expectation over the above inequality and using  Assumption~\\ref{assump-sgd}, we have\n\t\\begin{align}\\label{eq:step32}\n\t\t\\EE\\big[\\|\\ve^{k+1}\\|^2\\big] \\le &~ (1-\\beta_k)^2\\EE\\big[\\|\\ve^{k}\\|^2\\big]+ \\frac{2\\beta_k^2\\sigma^2}{m} + \\frac{2(1-\\beta_k)^2 L^2}{m} \\EE\\big[\\|\\vx^{k+1}-\\vx^{k}\\|^2\\big]\\cr\n\t\t= & ~ (1-\\beta_k)^2\\EE\\big[\\|\\ve^{k}\\|^2\\big]+ \\frac{2\\beta_k^2\\sigma^2}{m} + \\frac{2(1-\\beta_k)^2 \\eta_k^2 L^2}{m} \\EE\\big[\\|\\vg^{k}\\|^2\\big],\n\t\\end{align}\nwhere we have used\t$\\vx^{k+1}-\\vx^{k} = -\\eta_{k} \\vg^{k}$ in the equality.\n\t\n\nBy similar arguments as those in \\eqref{eq:tri-g}, it holds\n$$\\|\\vg^{k}\\|^2 \\le 2\\|\\bar\\vg^{k}\\|^2 + 2\\|\\vg^{k}-\\bar\\vg^{k}\\|^2\\le 2\\|\\bar\\vg^{k}\\|^2  + 2\\|\\ve^{k}\\|^2.$$\n%Now notice $\\vx^{k+1}-\\vx^{k} = -\\eta_{k} \\vg^{k}$ and \nPlugging the above inequality into \\eqref{eq:step32}, we obtain the desired result.\n\\end{proof}\n\n\\begin{theorem}\\label{thm:main}\n\tUnder Assumptions~\\ref{assump-obj} through \\ref{assump-sgd}, let $\\{\\vx^k\\}$ be the iterate sequence from Algorithm~\\ref{alg:prox-storm}. Suppose there are constants $A$ and $B$ such that the parameters $\\{\\eta_k\\}$ and $\\{\\beta_k\\}$ satisfying the conditions:\n\t\\begin{equation}\\label{eq:cond-eta-beta1}\n\t\t2\\eta_k L+ \\frac{4 L^2}{m} \\frac{\\eta_{k}}{\\Gamma_{k} } \\sum_{j=k+1}^{K-1}\\eta_j\\Gamma_{j} \\le 1, \\,\\forall\\, k=0,\\ldots,K-1,\n\t\\end{equation}\n\t\\begin{equation}\\label{eq:cond-eta-beta2}\n\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\le A,~\\text{and}~\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} }\\le B,\n\\end{equation}\nwhere $K$ is the maximum number of iterations in Algorithm~\\ref{alg:prox-storm}. \n\tLet $\\{\\bar\\vg^k\\}$ be defined in \\eqref{eq:vg-vgbar}. Then\t\n\t\\begin{equation}\\label{eq:mainthm}\n\t\t\\sum_{k=0}^{K-1} \\eta_k \\EE\\big[\\|\\bar\\vg^{k}\\|^2\\big] \n\t\t\\le\t12\\big[\\Phi(\\vx^0)-\\Phi^*\\big] +4(2A+3)\\frac{\\sigma^2}{ m_0}+ 16B\\frac{\\sigma^2}{m} .\n\t\\end{equation}\t\n\\end{theorem}\n\n\\begin{proof}\nWe begin by taking the total expectation and \ntelescoping the inequality in \\eqref{eq:chg-obj-0} over $k=0,\\ldots,K-1$ to obtain\n\\begin{align*}\n\t\\EE\\big[\\Phi(\\vx^{K})\\big]-\\Phi(\\vx^0) \n\t\\le&~ \\sum_{k=0}^{K-1}\\frac{\\eta_k}{2} \\EE\\big[\\|\\ve^k\\|^2\\big] -\\sum_{k=0}^{K-1} \\frac{\\eta_k}{2}(1-\\eta_k L)\\EE\\big[\\|\\vg^k\\|^2\\big]\\\\\n\t\\le&~ \\frac{\\sigma^2}{m_0}+ \\sum_{k=1}^{K-1}\\frac{\\eta_k}{2} \\EE\\big[\\|\\ve^k\\|^2\\big] -\\sum_{k=0}^{K-1} \\frac{\\eta_k}{2}(1-\\eta_k L)\\EE\\big[\\|\\vg^k\\|^2\\big],\n\\end{align*}\nwhere we have used $\\EE\\big[\\|\\ve^{0}\\|^2\\big]\\le \\frac{\\sigma^2}{m_0}$ by Assumption~\\ref{assump-sgd}. Since $\\Phi(\\vx^{K}) \\ge \\Phi^*$ from Assumption~\\ref{assump-obj}, the above inequality implies \n\\begin{equation}\\label{eq:main1}\n\t\\sum_{k=0}^{K-1} \\frac{\\eta_k}{2}(1-\\eta_k L)\\EE\\big[\\|\\vg^k\\|^2\\big] \n\t\\le \\Phi(\\vx^0)-\\Phi^* +\\frac{\\sigma^2}{m_0}+ \\sum_{k=1}^{K-1}\\frac{\\eta_k}{2} \\EE\\big[\\|\\ve^k\\|^2\\big].\n\\end{equation}\t\n\nIn addition, we divide both sides of \\eqref{eq:step32} by $\\Gamma_{k+1}$ and obtain from the definition of $\\Gamma_{k+1}$ in \\eqref{eq:gamma} that\n\t$$\\frac{1}{\\Gamma_{k+1} }\\EE\\big[\\|\\ve^{k+1}\\|^2\\big] \\le \\frac{1}{\\Gamma_{k} }\\EE\\big[\\|\\ve^{k}\\|^2\\big] +\\frac{1}{\\Gamma_{k+1} }\\frac{2\\beta_k^2\\sigma^2}{m} + \\frac{1}{\\Gamma_{k} }\\frac{2\\eta_{k}^2 L^2}{m} \\EE\\big[\\|\\vg^{k}\\|^2\\big],\\, \\forall\\, k\\ge 0.$$\nLet $j=0,\\ldots,k-1$ be another index on which the above inequality is telescoped. We obtain\n\t$$\\frac{1}{\\Gamma_{k} }\\EE\\big[\\|\\ve^{k}\\|^2\\big] \\le \\EE\\big[\\|\\ve^{0}\\|^2\\big] +\\sum_{j=0}^{k-1}\\frac{1}{\\Gamma_{j+1} }\\frac{2\\beta_j^2\\sigma^2}{m} + \\sum_{j=0}^{k-1}\\frac{1}{\\Gamma_{j} }\\frac{2\\eta_{j}^2 L^2}{m} \\EE\\big[\\|\\vg^{j}\\|^2\\big],\\, \\forall\\, k\\ge 1.$$\nMultiplying $\\Gamma_{k}$ to both sides of the above inequality and rearranging it gives\n%which further expands and rearranges into \n$$ \t\\EE\\big[\\|\\ve^{k}\\|^2\\big] \\le \\Gamma_{k}\\bigg(\\frac{\\sigma^2}{m_0} +\\frac{2\\sigma^2}{m}\\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \\bigg)+ \\frac{2 L^2}{m}\\sum_{j=0}^{k-1}\\frac{\\Gamma_{k}}{\\Gamma_{j} } \\eta_{j}^2  \\EE\\big[\\|\\vg^{j}\\|^2\\big],\\, \\forall\\, k\\ge 1,$$\nwhere we have used $\\EE\\big[\\|\\ve^{0}\\|^2\\big]\\le \\frac{\\sigma^2}{m_0}$ again. \n Now multiply $\\eta_k$ to the above inequality and sum it up over $k=1,\\ldots,K-1$ to have\n\\begin{align}\\label{eq:main2}\n \t\\sum_{k=1}^{K-1}\\eta_k\\EE\\big[\\|\\ve^{k}\\|^2\\big] \n \t\\le&~ \\sigma^2\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\bigg(\\frac{1}{m_0} +\\frac{2}{m} \\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \\bigg)+\\frac{2 L^2}{m}\\sum_{k=1}^{K-1} \\sum_{j=0}^{k-1}\\frac{\\eta_k\\Gamma_{k}}{\\Gamma_{j} } \\eta_{j}^2  \\EE\\big[\\|\\vg^{j}\\|^2\\big] \\cr\n \t=&~ \\sigma^2\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\bigg(\\frac{1}{m_0} +\\frac{2}{m} \\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \\bigg)+\\frac{2 L^2}{m}\\sum_{j=0}^{K-2}\\frac{\\eta_{j}^2}{\\Gamma_{j} }\\bigg(\\sum_{k=j+1}^{K-1}\\eta_k\\Gamma_{k}\\bigg)    \\EE\\big[\\|\\vg^{j}\\|^2\\big] \\cr\n \t=&~ \\sigma^2\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\bigg(\\frac{1}{m_0} +\\frac{2}{m} \\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \\bigg)+\\frac{2 L^2}{m}\\sum_{k=0}^{K-1}\\frac{\\eta_{k}^2}{\\Gamma_{k} }\\bigg(\\sum_{j=k+1}^{K-1}\\eta_j\\Gamma_{j}\\bigg)    \\EE\\big[\\|\\vg^{k}\\|^2\\big],\n\\end{align}\nwhere the first equality follows by swapping summation, and the second equality is obtained by swapping indices and realizing that the coefficient for $\\EE\\big[\\|\\vg^{K-1}\\|^2\\big]$ is null.\n\nNow we have by substituting \\eqref{eq:main2} into \\eqref{eq:main1} and rearranging terms that\n$$\n\t\\sum_{k=0}^{K-1} \\frac{\\eta_k}{2}\\bigg(1-\\eta_k L- \\frac{2 L^2}{m} \\frac{\\eta_{k}}{\\Gamma_{k} } \\sum_{j=k+1}^{K-1}\\eta_j\\Gamma_{j}\\bigg)\\EE\\big[\\|\\vg^k\\|^2\\big] \n\t\\le \\Phi(\\vx^0)-\\Phi^* +\\frac{\\sigma^2}{m_0}+ \\frac{\\sigma^2}{2}\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\bigg(\\frac{1}{m_0} +\\frac{2}{m} \\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \\bigg),\n$$\nwhich together with the conditions in \\eqref{eq:cond-eta-beta1} and \\eqref{eq:cond-eta-beta2} gives the bound for $\\vg^k$:\n\\begin{equation}\\label{eq:main3}\n\t\\sum_{k=0}^{K-1}  \\eta_k \\EE\\big[\\|\\vg^k\\|^2\\big] \n\t\\le 4\\big[\\Phi(\\vx^0)-\\Phi^*\\big]  + 2(A+2)\\frac{\\sigma^2}{ m_0}+ 4B\\frac{\\sigma^2}{m} .\n\\end{equation}\t\nUse \\eqref{eq:cond-eta-beta1} again and substitute \\eqref{eq:main3} into \\eqref{eq:main2}. We obtain the bound for $\\ve^k$:\n\\begin{equation}\\label{eq:main4}\n\t\\sum_{k=0}^{K-1} \\eta_k \\EE\\big[\\|\\ve^k\\|^2\\big] \n\t\\le A\\frac{\\sigma^2}{ m_0}+ 2B\\frac{\\sigma^2}{m} +\\sum_{k=0}^{K-1}\\frac{\\eta_{k}}{2} \\EE\\big[\\|\\vg^{k}\\|^2\\big]\n\t\\le 2\\big[\\Phi(\\vx^0)-\\Phi^*\\big] +2(A+1)\\frac{\\sigma^2}{ m_0}+ 4B\\frac{\\sigma^2}{m} .\n\\end{equation}\t\n\nFinally, we have from \\eqref{eq:tri-g} that  %from \\eqref{eq:nonexp-P} and the definitions of $\\vg^k$ and $\\bar\\vg^k$ in \\eqref{eq:vg-vgbar}, it follows \n$\\|\\bar\\vg^{k}\\|^2 \\le 2\\|\\vg^{k}\\|^2  + 2\\|\\ve^{k}\\|^2.$ Sum up this inequality over $k=0,\\ldots,K-1$ and\nsubstitute \\eqref{eq:main3} and  \\eqref{eq:main4} into the summation. We obtain the result in \\eqref{eq:mainthm}.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:keylem}\n\tLet \n\t\\begin{equation}\\label{eq:beta-def}\n\t\t\\beta_{k}=3\\big[(k+3)^{1/3}-(k+2)^{1/3}\\big],\\quad k\\ge 0,\n\t\\end{equation}\nThen we  have %the following bounds,\n\\begin{equation}\\label{eq:keylem1}\n\t\\sum_{j=k+1}^{K-1}\\frac{\\Gamma_{j}}{\\Gamma_{k}} \\le \\frac{1}{2}(k+2)^{2/3}+\\frac{1}{6}(k+2)^{1/3}+\\frac{1}{36}. \n\\end{equation}\n%and \n%\\begin{equation}\\label{eq:keylem2}\n%\t\\sum_{j=k+1}^{K-1}(j+2)^{-1/3}\\frac{\\Gamma_{j}}{\\Gamma_{k}} \n%\t\\le \\frac{1}{2}  (k+2)^{1/3}+\\frac{1}{12}.\n%\\end{equation}\n\\end{lemma}\n\n\\begin{proof}\t\n\tBy the fact $a^3-b^3 = (a-b)(a^2+ab+b^2)$, we have %We begin by realizing that \n\t\\begin{equation}\\label{eq:beta-bound}\n\t\t\\beta_{k}=3\\big[(k+3)^{1/3}-(k+2)^{1/3}\\big]=\n\t\t\\frac{3}{(k+3)^{2/3}+(k+3)^{1/3}(k+2)^{1/3}+(k+2)^{2/3}}.\n\t\\end{equation}\n\tHence, $\\beta_{k}\\in \\big[(k+3)^{-2/3},(k+2)^{-2/3}\\big]$ for all $k\\ge 0$, and it is a decreasing sequence. In addition,\n\t%We then establish the trick for bounding $\\Gamma_{j}/\\Gamma_{k}$: for all $j>k\\ge 0,$ \nby the definition of $\\Gamma_k$ and $\\beta_k$, it holds for all $j>k\\ge 0$ that\n\\begin{equation}\\label{eq:keystring}\n\t\\frac{\\Gamma_{j}}{\\Gamma_{k}} \n\t= \\frac{\\prod_{l=0}^{j-1}(1-\\beta_l)^2}{\\prod_{l=0}^{k-1}(1-\\beta_l)^2}  = \\prod_{l=k}^{j-1}(1-\\beta_l)^2\n\t\\le  e^{-2\\sum_{l=k}^{j-1}\\beta_l}\n\t=  e^{-6\\big[(j+2)^{1/3}-(k+2)^{1/3}\\big]},\n\\end{equation}\t\nwhere the inequality holds because $0\\le 1+x\\le e^x, \\forall\\, x\\ge -1$. Therefore we have that for any $k\\ge 0,$\n\\begin{equation}\\label{eq:key-beta-ineq1}\n\\sum_{j=k+1}^{K-1}\\frac{\\Gamma_{j}}{\\Gamma_{k}} \n%=\t\\sum_{j=k+1}^{K-1} \\prod_{l=k}^{j-1}(1-\\beta_l)^2\n\\le %\\sum_{j=k+1}^{K-1} e^{-2\\sum_{l=k}^{j-1}\\beta_l}\n \\sum_{j=k+1}^{K-1} e^{-6\\big[(j+2)^{1/3}-(k+2)^{1/3}\\big]}\n= e^{6(k+2)^{1/3}}\\sum_{j=k+1}^{K-1} e^{-6 (j+2)^{1/3} }.\n\\end{equation}\n\tSince $e^{-6x^{1/3}}$ is a decreasing function and has an anti-derivative $-\\frac{1}{36} e^{-6x^{1/3}} (18x^{2/3}+6x^{1/3}+1)$, we have %$\\int e^{-6x^{1/3}}dx=-\\frac{1}{36} e^{-6x^{1/3}} (18x^{2/3}+6x^{1/3}+1)+C,$ \n\t\\begin{equation}\\label{eq:key-beta-ineq2}\n\\sum_{j=k+1}^{K-1} e^{-6 (j+2)^{1/3} }\n\\le \\int_{k+2}^{K+1} e^{-6x^{1/3}}dx\n\\le \\frac{1}{36}  e^{-6(k+2)^{1/3}} (18(k+2)^{2/3}+6(k+2)^{1/3}+1) .\n\t\\end{equation}\n\tSubstituting \\eqref{eq:key-beta-ineq2} into \\eqref{eq:key-beta-ineq1} gives \\eqref{eq:keylem1} and completes the proof.\n\\end{proof}\n\n\\begin{proof}\nWe show the desired result by verifying the conditions in Theorem~\\ref{thm:main}. First, with $ \\eta_k =\\frac{\\eta}{L \\sqrt[3]{K}}$, the condition in\n \\eqref{eq:cond-eta-beta1} becomes %is satisfied if and only if\n$$\\frac{2 \\eta }{\\sqrt[3]{K}} + \\frac{4 }{m} \\frac{\\eta^2}{K^{2/3}}  \\sum_{j=k+1}^{K-1}\\frac{\\Gamma_{j}}{\\Gamma_{k} }  \\le 1, \\quad k=0,\\ldots,K-1.$$\nNotice that when $k=K-1$ the summation above is null. Hence by \\eqref{eq:keylem1}, it suffices to require \n\t$$\\frac{2\\eta }{\\sqrt[3]{K}} + \\frac{4 }{m} \\frac{\\eta^2 }{K^{2/3}}  \\bigg(\\frac{1}{2}K^{2/3}+\\frac{1}{6}K^{1/3}+\\frac{1}{36}\\bigg)  \\le 1, $$ which is guaranteed when $\\eta\\le\\frac{1}{4}$ and $K\\ge1$. Therefore, the condition in\n \\eqref{eq:cond-eta-beta1} holds.\n\t\nSecondly, by letting $k=0$ in \\eqref{eq:keylem1} and recalling $\\Gamma_0=1$, we have \n\t$\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\le \\big(\\frac{1}{2}2^{2/3}+\\frac{1}{6}2^{1/3}+\\frac{1}{36}\\big) \\frac{ \\eta}{L\\sqrt[3]{K}}.$\n Hence, the first condition in \\eqref{eq:cond-eta-beta2} holds with\n$A=\\big(2^{-1/3}+\\frac{1}{6}2^{1/3}+\\frac{1}{36}\\big) \\frac{ \\eta}{L\\sqrt[3]{K}}.$\t \nFinally, notice %utilize the below bound and obtain $B$ as in \\eqref{eq:cond-eta-beta2},\n\t\\begin{align}\\label{eq:last1}\n\t\t \\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \n\t\t=&~\\sum_{j=0}^{K-2}\\frac{\\beta_j^2}{(1-\\beta_j)^2 }  \\sum_{k=j+1}^{K-1}\\eta_k\\frac{\\Gamma_{k}}{\\Gamma_{j}} \\cr\n\t\t\\le&~ \\frac{ \\eta}{L\\sqrt[3]{K}} \\sum_{j=0}^{K-2}\\frac{\\beta_j^2}{(1-\\beta_0)^2 }  \n\t\t\\bigg(\\frac{1}{2}(j+2)^{2/3}+\\frac{1}{6}(j+2)^{1/3}+\\frac{1}{36}\\bigg) \\cr\n\t\t\\le&~ \\frac{ \\eta}{(1-\\beta_0)^2 L \\sqrt[3]{K}} \\sum_{j=0}^{K-2} \n\t\t\\bigg(\\frac{1}{2}(j+2)^{-2/3}+\\frac{1}{6}(j+2)^{-1}+\\frac{1}{36} (j+2)^{-4/3}\\bigg) \\cr\n\t\t\\le&~\\frac{ \\eta}{(1-\\beta_0)^2 L \\sqrt[3]{K}}   \n\t\t\\bigg(\\frac{3}{2}(K^{1/3}-1)+\\frac{1}{6}\\log K+\\frac{1}{12} (1-K^{-1/3})\\bigg) \t\\cr\n\t\t\\le&~ \\frac{ 2\\eta}{(1-2^{-2/3})^2 L}, \n\t\\end{align}\nwhere the first inequality follows from \\eqref{eq:keylem1}, the decreasing monotonicity of $\\beta_k$, and the setting of $\\eta_k$, the second inequality holds by $\\beta_j \\le (j+2)^{-2/3}$, and the last inequality is obtained by $\\beta_0\\le 2^{-2/3}$ and using the fact $3x^{1/3} > \\log x, \\forall\\, x>0$.\t\n%for sufficiently large $K$ so that $K^{1/3}\\ge \\ln K.$ \nThus the second condition in \\eqref{eq:cond-eta-beta2} holds with $B = \\frac{ 2\\eta}{(1-2^{-2/3})^2 L}$. Therefore, \\eqref{eq:rate-grad-const2} follows from \\eqref{eq:mainthm} and the choice of $\\tau$ by uniformly random selection. %Apply Theorem~\\ref{thm:rate-const2}, the proof is complete. \n\\end{proof}","units":[{"start":34692,"end":35050,"start_line":495,"end_line":499,"raw_tex":"\\begin{lemma}[one-iteration progress]\\label{lem:obj-diff}\nLet $\\{\\vx^k\\}$ be generated from Algorithm~\\ref{alg:prox-storm}. If $F$ is $L$-smooth, then\n$$\\Phi(\\vx^{k+1})-\\Phi(\\vx^k) \\le \\frac{\\eta_k}{2}(2-\\eta_k L)\\|\\ve^k\\|^2 - \\frac{\\eta_k}{4}(1-\\eta_k L)\\|\\bar\\vg^k\\|^2,\\, \\forall\\, k\\ge 1,$$\nwhere $\\bar\\vg^k$ is defined in \\eqref{eq:vg-vgbar}.\n\\end{lemma}","sha256_utf8":"2979367c6fbf0cdc6d2727d6a04522f261bbefc760eb84defd39faadd0446d6e"},{"start":35052,"end":36689,"start_line":501,"end_line":519,"raw_tex":"\\begin{proof}\nBy the $L$-smoothness of $F$ and the definition of $\\vg^k$ in \\eqref{eq:vg-vgbar}, we have\n\\begin{equation}\\label{eq:lip-F-ineq}\nF(\\vx^{k+1})-F(\\vx^k)\\le \\langle \\nabla F(\\vx^k), \\vx^{k+1}-\\vx^k\\rangle + \\frac{L}{2}\\|\\vx^{k+1}-\\vx^k\\|^2 = -\\eta_k\\langle \\nabla F(\\vx^k), \\vg^k\\rangle + \\frac{\\eta_k^2L}{2}\\|\\vg^k\\|^2.\n\\end{equation}\nUsing the definition of $\\ve^k$ in \\eqref{eq:error-grad} and the inequality in \\eqref{eq:lem1-lan}, we have \n$$-\\langle \\nabla F(\\vx^k), \\vg^k\\rangle = \\langle \\ve^k, \\vg^k\\rangle - \\langle \\vd^k, \\vg^k\\rangle \\le \\langle \\ve^k, \\vg^k\\rangle - \\|\\vg^k\\|^2 + \\frac{1}{\\eta_k}\\big(r(\\vx^k)-r(\\vx^{k+1})\\big).$$\nPlugging the above inequality into \\eqref{eq:lip-F-ineq} and rearranging terms give\n$$\\Phi(\\vx^{k+1})-\\Phi(\\vx^k) \\le \\eta_k\\langle \\ve^k, \\vg^k\\rangle - \\eta_k \\|\\vg^k\\|^2+ \\frac{\\eta_k^2L}{2}\\|\\vg^k\\|^2.$$\nBy the Cauchy-Schwartz inequality, it holds $\\eta_k\\langle \\ve^k, \\vg^k\\rangle \\le \\frac{\\eta_k}{2}\\|\\ve^k\\|^2 + \\frac{\\eta_k}{2}\\|\\vg^k\\|^2$, which together with the above inequality implies\n\\begin{equation}\\label{eq:chg-obj-0}\n\\Phi(\\vx^{k+1})-\\Phi(\\vx^k) \\le \\frac{\\eta_k}{2}\\|\\ve^k\\|^2 - \\frac{\\eta_k}{2}(1-\\eta_k L)\\|\\vg^k\\|^2.\n\\end{equation}\nFrom \\eqref{eq:nonexp-P} and the definitions of $\\vg^k$ and $\\bar\\vg^k$ in \\eqref{eq:vg-vgbar}, it follows\n\\begin{equation}\\label{eq:tri-g}\n-\\|\\vg^k\\|^2 \\le -\\frac{1}{2}\\|\\bar\\vg^k\\|^2 + \\|\\vg^k-\\bar\\vg^k\\|^2\\le -\\frac{1}{2}\\|\\bar\\vg^k\\|^2 + \\|\\vd^k -\\nabla F(\\vx^k)\\|^2 = -\\frac{1}{2}\\|\\bar\\vg^k\\|^2 + \\|\\ve^k\\|^2.\n\\end{equation}\nNow plug the above inequality into \\eqref{eq:chg-obj-0} to give the desired result.\n\\end{proof}","sha256_utf8":"22ecb55b253b60951ddebc0909416cf052dd0dbc66ff8b1c3d03a174d5b0dfa0"},{"start":36845,"end":37356,"start_line":522,"end_line":526,"raw_tex":"\\begin{lemma}[recursive bound on gradient error]\\label{lem:bd-ek}\nUnder Assumptions~\\ref{assump-smooth} and \\ref{assump-sgd}, it holds %for each $k\\ge 0$ that\n$$\\EE\\big[\\|\\ve^{k+1}\\|^2\\big] \\le \\frac{2\\beta_k^2\\sigma^2}{m} + \\frac{4(1-\\beta_k)^2\\eta_{k}^2 L^2}{m} \\EE\\big[\\|\\bar\\vg^{k}\\|^2\\big] + (1-\\beta_k)^2\\left(\\textstyle 1+\\frac{4\\eta_{k}^2 L^2}{m}\\right)\\EE\\big[\\|\\ve^{k}\\|^2\\big], \\forall\\, k\\ge0,$$\nwhere $\\bar\\vg^k$ and $\\ve^k$ are defined in \\eqref{eq:vg-vgbar} and \\eqref{eq:error-grad}.\n\\end{lemma}","sha256_utf8":"2e6b2e2b67934d25c1dcc899bf5a389069d8b9d2603d429b67edf3ce02cb6ed4"},{"start":37358,"end":41102,"start_line":528,"end_line":568,"raw_tex":"\\begin{proof}\nFirst, notice $\\EE_{B_{k+1}}[\\langle \\vv^{k+1}, \\ve^{k}\\rangle] = \\langle\\nabla F(\\vx^{k+1}), \\ve^{k}\\rangle$ and $\\EE_{B_{k+1}}[\\langle \\vu^{k+1}, \\ve^{k}\\rangle] = \\langle\\nabla F(\\vx^{k}), \\ve^{k}\\rangle$, and thus\n\\begin{equation}\\label{eq:crs-term0}\n\\EE_{B_{k+1}}[\\langle \\vv^{k+1} - \\nabla F(\\vx^{k+1}), \\ve^{k}\\rangle] = 0,\\quad \\EE_{B_{k+1}}[\\langle \\vu^{k+1} - \\nabla F(\\vx^{k}), \\ve^{k}\\rangle] = 0.\n\\end{equation}\nHence, by writing $\\ve^{k+1} = \\vv^{k+1} - \\nabla F(\\vx^{k+1}) + (1-\\beta_k)(\\nabla F(\\vx^{k})- \\vu^{k+1})+(1-\\beta_k)\\ve^{k}$, we have\n\\begin{equation}\\label{eq:one-step-e}\n\\EE_{B_{k+1}}\\big[\\|\\ve^{k+1}\\|^2\\big] = \\EE_{B_{k+1}}\\big[\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1}) + (1-\\beta_k)(\\nabla F(\\vx^{k})- \\vu^{k+1})\\|^2\\big] + (1-\\beta_k)^2\\|\\ve^{k}\\|^2.\n\\end{equation}\nBy the Young's inequality, it holds\n\\begin{align}\\label{eq:step2}\n&~\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1}) + (1-\\beta_k)(\\nabla F(\\vx^{k})- \\vu^{k+1})\\|^2 \\cr\n= &~ \\big\\|\\beta_k\\big(\\vv^{k+1} - \\nabla F(\\vx^{k+1})\\big) + (1-\\beta_k)\\big(\\vv^{k+1} - \\nabla F(\\vx^{k+1}) + \\nabla F(\\vx^{k})- \\vu^{k+1}\\big)\\big\\|^2\\cr\n\\le &~ 2\\beta_k^2\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1})\\|^2 + 2(1-\\beta_k)^2\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1}) + \\nabla F(\\vx^{k})- \\vu^{k+1}\\|^2.\n\\end{align}\nFrom the definition of $\\vv^{k+1}$ and $ \\vu^{k+1}$ in \\eqref{eq:def-v-u-k}, we have \n\\begin{align}\\label{eq:step23}\n \t&~\\EE_{B_{k+1}}\\big[\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1}) + \\nabla F(\\vx^{k})- \\vu^{k+1}\\|^2\\big] \\cr \n \t=&~\\frac{1}{m^2}\\EE_{B_{k+1}}\\left\\| \\sum_{\\xi\\in B_{k+1}}\\left(\\nabla f(\\vx^{k+1};\\xi)-\\nabla f(\\vx^{k};\\xi)  - \\nabla F(\\vx^{k+1})+\\nabla F(\\vx^{k})\\right) \\right\\|^2 \t \\cr\n \t=&~\\frac{1}{m^2}\\sum_{j=1}^m\\EE_{\\xi_j^{k+1}}\\left\\| \\nabla f(\\vx^{k+1};\\xi_j^{k+1})-\\nabla f(\\vx^{k};\\xi_j^{k+1})  - \\nabla F(\\vx^{k+1})+\\nabla F(\\vx^{k})  \\right\\|^2 \t \\cr\n \t\\le&~\\frac{1}{m^2}\\sum_{j=1}^m\\EE_{\\xi_j^{k+1}}\\left\\| \\nabla f(\\vx^{k+1};\\xi_j^{k+1})-\\nabla f(\\vx^{k};\\xi_j^{k+1}) \\right\\|^2 \t \\cr\t\n \t\\le&~\\frac{L^2}{m}\\|\\vx^{k+1}-\\vx^{k}\\|^2,\n\\end{align}\nwhere the second equality holds because of the i.i.d. samples in $B_{k+1}$ and the zero mean of the random vector $\\nabla f(\\vx^{k+1};\\xi_j^{k+1})-\\nabla f(\\vx^{k};\\xi_j^{k+1})  - \\nabla F(\\vx^{k+1})+\\nabla F(\\vx^{k}) $ resulted from unbiasedness in Assumption~\\ref{assump-sgd}, \nthe first inequality is due to the fact that the variance of a random vector is upper-bounded by its second moment, and the last inequality follows from Assumption~\\ref{assump-smooth}.\n\nNow, take conditional expectation on both sides of \\eqref{eq:step2}, use \\eqref{eq:step23}, and substitute it into \\eqref{eq:one-step-e}. We have\n$$\\EE_{B_{k+1}}\\big[\\|\\ve^{k+1}\\|^2\\big] \\le (1-\\beta_k)^2\\|\\ve^{k}\\|^2 +2\\beta_k^2\\EE_{B_{k+1}} \\big[\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1})\\|^2\\big] + \\frac{2(1-\\beta_k)^2 L^2}{m} \\EE_{B_{k+1}}\\big[\\|\\vx^{k+1}-\\vx^{k}\\|^2\\big] .$$\nTaking a full expectation over the above inequality and using  Assumption~\\ref{assump-sgd}, we have\n\t\\begin{align}\\label{eq:step32}\n\t\t\\EE\\big[\\|\\ve^{k+1}\\|^2\\big] \\le &~ (1-\\beta_k)^2\\EE\\big[\\|\\ve^{k}\\|^2\\big]+ \\frac{2\\beta_k^2\\sigma^2}{m} + \\frac{2(1-\\beta_k)^2 L^2}{m} \\EE\\big[\\|\\vx^{k+1}-\\vx^{k}\\|^2\\big]\\cr\n\t\t= & ~ (1-\\beta_k)^2\\EE\\big[\\|\\ve^{k}\\|^2\\big]+ \\frac{2\\beta_k^2\\sigma^2}{m} + \\frac{2(1-\\beta_k)^2 \\eta_k^2 L^2}{m} \\EE\\big[\\|\\vg^{k}\\|^2\\big],\n\t\\end{align}\nwhere we have used\t$\\vx^{k+1}-\\vx^{k} = -\\eta_{k} \\vg^{k}$ in the equality.\n\t\n\nBy similar arguments as those in \\eqref{eq:tri-g}, it holds\n$$\\|\\vg^{k}\\|^2 \\le 2\\|\\bar\\vg^{k}\\|^2 + 2\\|\\vg^{k}-\\bar\\vg^{k}\\|^2\\le 2\\|\\bar\\vg^{k}\\|^2  + 2\\|\\ve^{k}\\|^2.$$\n%Now notice $\\vx^{k+1}-\\vx^{k} = -\\eta_{k} \\vg^{k}$ and \nPlugging the above inequality into \\eqref{eq:step32}, we obtain the desired result.\n\\end{proof}","sha256_utf8":"2b8abe18b38d3fd6c5c89b2afb3531ff83466de7b72c543093878dd0d693dfa4"},{"start":58271,"end":59306,"start_line":745,"end_line":759,"raw_tex":"\\begin{theorem}\\label{thm:main}\n\tUnder Assumptions~\\ref{assump-obj} through \\ref{assump-sgd}, let $\\{\\vx^k\\}$ be the iterate sequence from Algorithm~\\ref{alg:prox-storm}. Suppose there are constants $A$ and $B$ such that the parameters $\\{\\eta_k\\}$ and $\\{\\beta_k\\}$ satisfying the conditions:\n\t\\begin{equation}\\label{eq:cond-eta-beta1}\n\t\t2\\eta_k L+ \\frac{4 L^2}{m} \\frac{\\eta_{k}}{\\Gamma_{k} } \\sum_{j=k+1}^{K-1}\\eta_j\\Gamma_{j} \\le 1, \\,\\forall\\, k=0,\\ldots,K-1,\n\t\\end{equation}\n\t\\begin{equation}\\label{eq:cond-eta-beta2}\n\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\le A,~\\text{and}~\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} }\\le B,\n\\end{equation}\nwhere $K$ is the maximum number of iterations in Algorithm~\\ref{alg:prox-storm}. \n\tLet $\\{\\bar\\vg^k\\}$ be defined in \\eqref{eq:vg-vgbar}. Then\t\n\t\\begin{equation}\\label{eq:mainthm}\n\t\t\\sum_{k=0}^{K-1} \\eta_k \\EE\\big[\\|\\bar\\vg^{k}\\|^2\\big] \n\t\t\\le\t12\\big[\\Phi(\\vx^0)-\\Phi^*\\big] +4(2A+3)\\frac{\\sigma^2}{ m_0}+ 16B\\frac{\\sigma^2}{m} .\n\t\\end{equation}\t\n\\end{theorem}","sha256_utf8":"faf186b14f667534232dd2a8fdfc750cd1d5bb62c4204be2c637b3597dbe32b1"},{"start":59307,"end":64259,"start_line":760,"end_line":811,"raw_tex":"\\begin{proof}\nWe begin by taking the total expectation and \ntelescoping the inequality in \\eqref{eq:chg-obj-0} over $k=0,\\ldots,K-1$ to obtain\n\\begin{align*}\n\t\\EE\\big[\\Phi(\\vx^{K})\\big]-\\Phi(\\vx^0) \n\t\\le&~ \\sum_{k=0}^{K-1}\\frac{\\eta_k}{2} \\EE\\big[\\|\\ve^k\\|^2\\big] -\\sum_{k=0}^{K-1} \\frac{\\eta_k}{2}(1-\\eta_k L)\\EE\\big[\\|\\vg^k\\|^2\\big]\\\\\n\t\\le&~ \\frac{\\sigma^2}{m_0}+ \\sum_{k=1}^{K-1}\\frac{\\eta_k}{2} \\EE\\big[\\|\\ve^k\\|^2\\big] -\\sum_{k=0}^{K-1} \\frac{\\eta_k}{2}(1-\\eta_k L)\\EE\\big[\\|\\vg^k\\|^2\\big],\n\\end{align*}\nwhere we have used $\\EE\\big[\\|\\ve^{0}\\|^2\\big]\\le \\frac{\\sigma^2}{m_0}$ by Assumption~\\ref{assump-sgd}. Since $\\Phi(\\vx^{K}) \\ge \\Phi^*$ from Assumption~\\ref{assump-obj}, the above inequality implies \n\\begin{equation}\\label{eq:main1}\n\t\\sum_{k=0}^{K-1} \\frac{\\eta_k}{2}(1-\\eta_k L)\\EE\\big[\\|\\vg^k\\|^2\\big] \n\t\\le \\Phi(\\vx^0)-\\Phi^* +\\frac{\\sigma^2}{m_0}+ \\sum_{k=1}^{K-1}\\frac{\\eta_k}{2} \\EE\\big[\\|\\ve^k\\|^2\\big].\n\\end{equation}\t\n\nIn addition, we divide both sides of \\eqref{eq:step32} by $\\Gamma_{k+1}$ and obtain from the definition of $\\Gamma_{k+1}$ in \\eqref{eq:gamma} that\n\t$$\\frac{1}{\\Gamma_{k+1} }\\EE\\big[\\|\\ve^{k+1}\\|^2\\big] \\le \\frac{1}{\\Gamma_{k} }\\EE\\big[\\|\\ve^{k}\\|^2\\big] +\\frac{1}{\\Gamma_{k+1} }\\frac{2\\beta_k^2\\sigma^2}{m} + \\frac{1}{\\Gamma_{k} }\\frac{2\\eta_{k}^2 L^2}{m} \\EE\\big[\\|\\vg^{k}\\|^2\\big],\\, \\forall\\, k\\ge 0.$$\nLet $j=0,\\ldots,k-1$ be another index on which the above inequality is telescoped. We obtain\n\t$$\\frac{1}{\\Gamma_{k} }\\EE\\big[\\|\\ve^{k}\\|^2\\big] \\le \\EE\\big[\\|\\ve^{0}\\|^2\\big] +\\sum_{j=0}^{k-1}\\frac{1}{\\Gamma_{j+1} }\\frac{2\\beta_j^2\\sigma^2}{m} + \\sum_{j=0}^{k-1}\\frac{1}{\\Gamma_{j} }\\frac{2\\eta_{j}^2 L^2}{m} \\EE\\big[\\|\\vg^{j}\\|^2\\big],\\, \\forall\\, k\\ge 1.$$\nMultiplying $\\Gamma_{k}$ to both sides of the above inequality and rearranging it gives\n%which further expands and rearranges into \n$$ \t\\EE\\big[\\|\\ve^{k}\\|^2\\big] \\le \\Gamma_{k}\\bigg(\\frac{\\sigma^2}{m_0} +\\frac{2\\sigma^2}{m}\\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \\bigg)+ \\frac{2 L^2}{m}\\sum_{j=0}^{k-1}\\frac{\\Gamma_{k}}{\\Gamma_{j} } \\eta_{j}^2  \\EE\\big[\\|\\vg^{j}\\|^2\\big],\\, \\forall\\, k\\ge 1,$$\nwhere we have used $\\EE\\big[\\|\\ve^{0}\\|^2\\big]\\le \\frac{\\sigma^2}{m_0}$ again. \n Now multiply $\\eta_k$ to the above inequality and sum it up over $k=1,\\ldots,K-1$ to have\n\\begin{align}\\label{eq:main2}\n \t\\sum_{k=1}^{K-1}\\eta_k\\EE\\big[\\|\\ve^{k}\\|^2\\big] \n \t\\le&~ \\sigma^2\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\bigg(\\frac{1}{m_0} +\\frac{2}{m} \\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \\bigg)+\\frac{2 L^2}{m}\\sum_{k=1}^{K-1} \\sum_{j=0}^{k-1}\\frac{\\eta_k\\Gamma_{k}}{\\Gamma_{j} } \\eta_{j}^2  \\EE\\big[\\|\\vg^{j}\\|^2\\big] \\cr\n \t=&~ \\sigma^2\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\bigg(\\frac{1}{m_0} +\\frac{2}{m} \\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \\bigg)+\\frac{2 L^2}{m}\\sum_{j=0}^{K-2}\\frac{\\eta_{j}^2}{\\Gamma_{j} }\\bigg(\\sum_{k=j+1}^{K-1}\\eta_k\\Gamma_{k}\\bigg)    \\EE\\big[\\|\\vg^{j}\\|^2\\big] \\cr\n \t=&~ \\sigma^2\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\bigg(\\frac{1}{m_0} +\\frac{2}{m} \\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \\bigg)+\\frac{2 L^2}{m}\\sum_{k=0}^{K-1}\\frac{\\eta_{k}^2}{\\Gamma_{k} }\\bigg(\\sum_{j=k+1}^{K-1}\\eta_j\\Gamma_{j}\\bigg)    \\EE\\big[\\|\\vg^{k}\\|^2\\big],\n\\end{align}\nwhere the first equality follows by swapping summation, and the second equality is obtained by swapping indices and realizing that the coefficient for $\\EE\\big[\\|\\vg^{K-1}\\|^2\\big]$ is null.\n\nNow we have by substituting \\eqref{eq:main2} into \\eqref{eq:main1} and rearranging terms that\n$$\n\t\\sum_{k=0}^{K-1} \\frac{\\eta_k}{2}\\bigg(1-\\eta_k L- \\frac{2 L^2}{m} \\frac{\\eta_{k}}{\\Gamma_{k} } \\sum_{j=k+1}^{K-1}\\eta_j\\Gamma_{j}\\bigg)\\EE\\big[\\|\\vg^k\\|^2\\big] \n\t\\le \\Phi(\\vx^0)-\\Phi^* +\\frac{\\sigma^2}{m_0}+ \\frac{\\sigma^2}{2}\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\bigg(\\frac{1}{m_0} +\\frac{2}{m} \\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \\bigg),\n$$\nwhich together with the conditions in \\eqref{eq:cond-eta-beta1} and \\eqref{eq:cond-eta-beta2} gives the bound for $\\vg^k$:\n\\begin{equation}\\label{eq:main3}\n\t\\sum_{k=0}^{K-1}  \\eta_k \\EE\\big[\\|\\vg^k\\|^2\\big] \n\t\\le 4\\big[\\Phi(\\vx^0)-\\Phi^*\\big]  + 2(A+2)\\frac{\\sigma^2}{ m_0}+ 4B\\frac{\\sigma^2}{m} .\n\\end{equation}\t\nUse \\eqref{eq:cond-eta-beta1} again and substitute \\eqref{eq:main3} into \\eqref{eq:main2}. We obtain the bound for $\\ve^k$:\n\\begin{equation}\\label{eq:main4}\n\t\\sum_{k=0}^{K-1} \\eta_k \\EE\\big[\\|\\ve^k\\|^2\\big] \n\t\\le A\\frac{\\sigma^2}{ m_0}+ 2B\\frac{\\sigma^2}{m} +\\sum_{k=0}^{K-1}\\frac{\\eta_{k}}{2} \\EE\\big[\\|\\vg^{k}\\|^2\\big]\n\t\\le 2\\big[\\Phi(\\vx^0)-\\Phi^*\\big] +2(A+1)\\frac{\\sigma^2}{ m_0}+ 4B\\frac{\\sigma^2}{m} .\n\\end{equation}\t\n\nFinally, we have from \\eqref{eq:tri-g} that  %from \\eqref{eq:nonexp-P} and the definitions of $\\vg^k$ and $\\bar\\vg^k$ in \\eqref{eq:vg-vgbar}, it follows \n$\\|\\bar\\vg^{k}\\|^2 \\le 2\\|\\vg^{k}\\|^2  + 2\\|\\ve^{k}\\|^2.$ Sum up this inequality over $k=0,\\ldots,K-1$ and\nsubstitute \\eqref{eq:main3} and  \\eqref{eq:main4} into the summation. We obtain the result in \\eqref{eq:mainthm}.\n\\end{proof}","sha256_utf8":"f8c1462a1d1b488c8a98b843879fae089fb6bce7b67cb5560cb35c8d6cd57cad"},{"start":64647,"end":65177,"start_line":815,"end_line":829,"raw_tex":"\\begin{lemma}\\label{lem:keylem}\n\tLet \n\t\\begin{equation}\\label{eq:beta-def}\n\t\t\\beta_{k}=3\\big[(k+3)^{1/3}-(k+2)^{1/3}\\big],\\quad k\\ge 0,\n\t\\end{equation}\nThen we  have %the following bounds,\n\\begin{equation}\\label{eq:keylem1}\n\t\\sum_{j=k+1}^{K-1}\\frac{\\Gamma_{j}}{\\Gamma_{k}} \\le \\frac{1}{2}(k+2)^{2/3}+\\frac{1}{6}(k+2)^{1/3}+\\frac{1}{36}. \n\\end{equation}\n%and \n%\\begin{equation}\\label{eq:keylem2}\n%\t\\sum_{j=k+1}^{K-1}(j+2)^{-1/3}\\frac{\\Gamma_{j}}{\\Gamma_{k}} \n%\t\\le \\frac{1}{2}  (k+2)^{1/3}+\\frac{1}{12}.\n%\\end{equation}\n\\end{lemma}","sha256_utf8":"175e50d671f3319bfbb2c0150db137c46b310756521e967ca5b1e352fb6805c5"},{"start":65178,"end":66999,"start_line":830,"end_line":860,"raw_tex":"\\begin{proof}\t\n\tBy the fact $a^3-b^3 = (a-b)(a^2+ab+b^2)$, we have %We begin by realizing that \n\t\\begin{equation}\\label{eq:beta-bound}\n\t\t\\beta_{k}=3\\big[(k+3)^{1/3}-(k+2)^{1/3}\\big]=\n\t\t\\frac{3}{(k+3)^{2/3}+(k+3)^{1/3}(k+2)^{1/3}+(k+2)^{2/3}}.\n\t\\end{equation}\n\tHence, $\\beta_{k}\\in \\big[(k+3)^{-2/3},(k+2)^{-2/3}\\big]$ for all $k\\ge 0$, and it is a decreasing sequence. In addition,\n\t%We then establish the trick for bounding $\\Gamma_{j}/\\Gamma_{k}$: for all $j>k\\ge 0,$ \nby the definition of $\\Gamma_k$ and $\\beta_k$, it holds for all $j>k\\ge 0$ that\n\\begin{equation}\\label{eq:keystring}\n\t\\frac{\\Gamma_{j}}{\\Gamma_{k}} \n\t= \\frac{\\prod_{l=0}^{j-1}(1-\\beta_l)^2}{\\prod_{l=0}^{k-1}(1-\\beta_l)^2}  = \\prod_{l=k}^{j-1}(1-\\beta_l)^2\n\t\\le  e^{-2\\sum_{l=k}^{j-1}\\beta_l}\n\t=  e^{-6\\big[(j+2)^{1/3}-(k+2)^{1/3}\\big]},\n\\end{equation}\t\nwhere the inequality holds because $0\\le 1+x\\le e^x, \\forall\\, x\\ge -1$. Therefore we have that for any $k\\ge 0,$\n\\begin{equation}\\label{eq:key-beta-ineq1}\n\\sum_{j=k+1}^{K-1}\\frac{\\Gamma_{j}}{\\Gamma_{k}} \n%=\t\\sum_{j=k+1}^{K-1} \\prod_{l=k}^{j-1}(1-\\beta_l)^2\n\\le %\\sum_{j=k+1}^{K-1} e^{-2\\sum_{l=k}^{j-1}\\beta_l}\n \\sum_{j=k+1}^{K-1} e^{-6\\big[(j+2)^{1/3}-(k+2)^{1/3}\\big]}\n= e^{6(k+2)^{1/3}}\\sum_{j=k+1}^{K-1} e^{-6 (j+2)^{1/3} }.\n\\end{equation}\n\tSince $e^{-6x^{1/3}}$ is a decreasing function and has an anti-derivative $-\\frac{1}{36} e^{-6x^{1/3}} (18x^{2/3}+6x^{1/3}+1)$, we have %$\\int e^{-6x^{1/3}}dx=-\\frac{1}{36} e^{-6x^{1/3}} (18x^{2/3}+6x^{1/3}+1)+C,$ \n\t\\begin{equation}\\label{eq:key-beta-ineq2}\n\\sum_{j=k+1}^{K-1} e^{-6 (j+2)^{1/3} }\n\\le \\int_{k+2}^{K+1} e^{-6x^{1/3}}dx\n\\le \\frac{1}{36}  e^{-6(k+2)^{1/3}} (18(k+2)^{2/3}+6(k+2)^{1/3}+1) .\n\t\\end{equation}\n\tSubstituting \\eqref{eq:key-beta-ineq2} into \\eqref{eq:key-beta-ineq1} gives \\eqref{eq:keylem1} and completes the proof.\n\\end{proof}","sha256_utf8":"5b7191ae5c5ef02a908d71a80970d48682ca2bb5782675c4df062e0909e6f1ba"},{"start":67901,"end":70520,"start_line":872,"end_line":899,"raw_tex":"\\begin{proof}\nWe show the desired result by verifying the conditions in Theorem~\\ref{thm:main}. First, with $ \\eta_k =\\frac{\\eta}{L \\sqrt[3]{K}}$, the condition in\n \\eqref{eq:cond-eta-beta1} becomes %is satisfied if and only if\n$$\\frac{2 \\eta }{\\sqrt[3]{K}} + \\frac{4 }{m} \\frac{\\eta^2}{K^{2/3}}  \\sum_{j=k+1}^{K-1}\\frac{\\Gamma_{j}}{\\Gamma_{k} }  \\le 1, \\quad k=0,\\ldots,K-1.$$\nNotice that when $k=K-1$ the summation above is null. Hence by \\eqref{eq:keylem1}, it suffices to require \n\t$$\\frac{2\\eta }{\\sqrt[3]{K}} + \\frac{4 }{m} \\frac{\\eta^2 }{K^{2/3}}  \\bigg(\\frac{1}{2}K^{2/3}+\\frac{1}{6}K^{1/3}+\\frac{1}{36}\\bigg)  \\le 1, $$ which is guaranteed when $\\eta\\le\\frac{1}{4}$ and $K\\ge1$. Therefore, the condition in\n \\eqref{eq:cond-eta-beta1} holds.\n\t\nSecondly, by letting $k=0$ in \\eqref{eq:keylem1} and recalling $\\Gamma_0=1$, we have \n\t$\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\le \\big(\\frac{1}{2}2^{2/3}+\\frac{1}{6}2^{1/3}+\\frac{1}{36}\\big) \\frac{ \\eta}{L\\sqrt[3]{K}}.$\n Hence, the first condition in \\eqref{eq:cond-eta-beta2} holds with\n$A=\\big(2^{-1/3}+\\frac{1}{6}2^{1/3}+\\frac{1}{36}\\big) \\frac{ \\eta}{L\\sqrt[3]{K}}.$\t \nFinally, notice %utilize the below bound and obtain $B$ as in \\eqref{eq:cond-eta-beta2},\n\t\\begin{align}\\label{eq:last1}\n\t\t \\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \n\t\t=&~\\sum_{j=0}^{K-2}\\frac{\\beta_j^2}{(1-\\beta_j)^2 }  \\sum_{k=j+1}^{K-1}\\eta_k\\frac{\\Gamma_{k}}{\\Gamma_{j}} \\cr\n\t\t\\le&~ \\frac{ \\eta}{L\\sqrt[3]{K}} \\sum_{j=0}^{K-2}\\frac{\\beta_j^2}{(1-\\beta_0)^2 }  \n\t\t\\bigg(\\frac{1}{2}(j+2)^{2/3}+\\frac{1}{6}(j+2)^{1/3}+\\frac{1}{36}\\bigg) \\cr\n\t\t\\le&~ \\frac{ \\eta}{(1-\\beta_0)^2 L \\sqrt[3]{K}} \\sum_{j=0}^{K-2} \n\t\t\\bigg(\\frac{1}{2}(j+2)^{-2/3}+\\frac{1}{6}(j+2)^{-1}+\\frac{1}{36} (j+2)^{-4/3}\\bigg) \\cr\n\t\t\\le&~\\frac{ \\eta}{(1-\\beta_0)^2 L \\sqrt[3]{K}}   \n\t\t\\bigg(\\frac{3}{2}(K^{1/3}-1)+\\frac{1}{6}\\log K+\\frac{1}{12} (1-K^{-1/3})\\bigg) \t\\cr\n\t\t\\le&~ \\frac{ 2\\eta}{(1-2^{-2/3})^2 L}, \n\t\\end{align}\nwhere the first inequality follows from \\eqref{eq:keylem1}, the decreasing monotonicity of $\\beta_k$, and the setting of $\\eta_k$, the second inequality holds by $\\beta_j \\le (j+2)^{-2/3}$, and the last inequality is obtained by $\\beta_0\\le 2^{-2/3}$ and using the fact $3x^{1/3} > \\log x, \\forall\\, x>0$.\t\n%for sufficiently large $K$ so that $K^{1/3}\\ge \\ln K.$ \nThus the second condition in \\eqref{eq:cond-eta-beta2} holds with $B = \\frac{ 2\\eta}{(1-2^{-2/3})^2 L}$. Therefore, \\eqref{eq:rate-grad-const2} follows from \\eqref{eq:mainthm} and the choice of $\\tau$ by uniformly random selection. %Apply Theorem~\\ref{thm:rate-const2}, the proof is complete. \n\\end{proof}","sha256_utf8":"79c3fc45c0b7d9d9908f4fffde0b8ac1ffd93a6c2c7536a2b19f28d2e191ed90"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"复合随机非凸问题","source_units":[{"start":10873,"end":11096,"start_line":305,"end_line":308,"raw_tex":"In this paper, we consider the regularized nonconvex stochastic programming\n\\begin{equation}\\label{eq:stoc-prob}\n\\Phi^*= \\Min_{\\vx\\in\\RR^n} ~\\Phi(\\vx):= \\big\\{F(\\vx)\\equiv\\EE_\\xi [f(\\vx;\\xi)]\\big\\} + r(\\vx), \n\\end{equation}","sha256_utf8":"9850027c6cac9ee5e877a64c58f4f396c97b3bc85b9c8e2545170028858ab48e"},{"start":11097,"end":11922,"start_line":309,"end_line":309,"raw_tex":"where $f(\\,\\cdot\\,; \\xi)$ is a smooth nonconvex function almost surely for $\\xi$, and $r$ is a closed convex function on $\\RR^n$. Examples of \\eqref{eq:stoc-prob} include the sparse online matrix factorization \\cite{mairal2010online}, the online nonnegative matrix factorization \\cite{zhao2016online}, and the streaming PCA (by a unit-ball constraint) \\cite{mitliagkas2013memory}. In addition, as $\\xi$ follows a uniform distribution on a finite set $\\Xi=\\{\\xi_1,\\ldots,\\xi_N\\}$, \\eqref{eq:stoc-prob} recovers the so-called finite-sum structured problem. It includes most regularized machine learning problems such as the sparse bilinear logistic regression \\cite{shi2014sparse}, the sparse convolutional neural network \\cite{liu2015sparse}, and the group sparse regularized deep neural networks \\cite{scardapane2017group}. \n","sha256_utf8":"d5d55de741b4d54139476875fed5151856b51f916e087a238cbf1ea171a90c8f"}]},{"id":"S002","semantic_type":"definition","kind":"definition","title":"强凸函数与布雷格曼散度","source_units":[{"start":14454,"end":14819,"start_line":316,"end_line":321,"raw_tex":"Let $w$ be a continuously differentiable and 1-strongly convex function on $\\dom(r)$, i.e., \n$$w(\\vy) \\ge w(\\vx) + \\langle \\nabla w(\\vx), \\vy-\\vx\\rangle + \\frac{1}{2}\\|\\vy-\\vx\\|^2,\\, \\forall\\, \\vx,\\vy\\in \\dom(r).$$\nThe Bregman divergence induced by $w$ is defined as\n\\begin{equation}\nV(\\vx,\\vz)=w(\\vx)-w(\\vz) - \\langle \\nabla w(\\vz), \\vx-\\vz\\rangle.\n\\end{equation}\n","sha256_utf8":"82c517a83e2194642a8ae0de059fae47305a89fae88ceaae90e6079d2b92b7b6"}]},{"id":"S003","semantic_type":"algorithm","kind":"algorithm","title":"PStorm迭代与抽样","source_units":[{"start":15485,"end":16981,"start_line":324,"end_line":346,"raw_tex":"\\begin{algorithm}[h]\n\\caption{Momentum-based variance-reduced proximal stochastic gradient method for \\eqref{eq:stoc-prob}}\\label{alg:prox-storm}\n\\DontPrintSemicolon\n\\textbf{Input:} max iteration numer $K$, minibatch size $m$, and positive sequences $\\{\\beta_k\\}\\subseteq(0,1)$ and $\\{\\eta_k\\}$.\\;\n\\textbf{Initialization:} choose $\\vx^0\\in\\dom(r)$ and let $\\vd^0 = \\frac{1}{m_0}\\sum_{\\xi\\in B_0}\\nabla f(\\vx^0;\\xi)$ with $m_0$ i.i.d. samples $B_0=\\{\\xi_1^0,\\ldots,\\xi_{m_0}^0\\}$\\;\n\\For{$k=0, 1,\\ldots, K-1$}{\nUpdate $\\vx$ by\n\\begin{equation}\\label{eq:update-x}\n\\vx^{k+1}=\\argmin_{\\vx} \\left\\{\\langle \\vd^k, \\vx\\rangle + \\frac{1}{\\eta_k}V(\\vx,\\vx^k) + r(\\vx)\\right\\}.\\vspace{-0.3cm}\n\\end{equation}\n\\;\nObtain $m$ i.i.d. samples $B_{k+1}=\\{\\xi_1^{k+1},\\ldots,\\xi_m^{k+1}\\}$ and let \n\\begin{equation}\\label{eq:def-v-u-k}\n\\textstyle \\vv^{k+1} = \\frac{1}{m}\\sum_{\\xi\\in B_{k+1}}\\nabla f(\\vx^{k+1}; \\xi),\\quad \\vu^{k+1} = \\frac{1}{m}\\sum_{\\xi\\in B_{k+1}}\\nabla f(\\vx^{k}; \\xi).\\vspace{-0.3cm}\n\\end{equation}\\;\nLet $\\vd^{k+1} = \\vv^{k+1} + (1-\\beta_{k})(\\vd^{k}-\\vu^{k+1})$.\\;\n}\nReturn $\\vx^\\tau$ with $\\tau$ selected from $\\{0,1,\\ldots,K-1\\}$ uniformly at random or by the distribution\n\\begin{equation}\\label{eq:select-tau}\n\\textstyle \\Prob(\\tau = k) = \\frac{\\frac{\\eta_k}{4}(1-\\eta_k L)-\\frac{\\eta_k^2}{5m\\eta_{k+1}}(1-\\beta_k)^2}{\\sum_{j=0}^{K-1}\\left(\\frac{\\eta_j}{4}(1-\\eta_j L)-\\frac{\\eta_j^2}{5m\\eta_{j+1}}(1-\\beta_j)^2\\right)},\\, k = 0,1,\\ldots,K-1.\n\\end{equation}\n\\vspace{-0.2cm}\n\\end{algorithm}","sha256_utf8":"abbc025bf73abbcfc9fc9016619e6aaae1821ff97570dad751993e28673b00ea"}]},{"id":"S004","semantic_type":"notation","kind":"notation","title":"期望与光滑性记号","source_units":[{"start":30356,"end":30801,"start_line":435,"end_line":435,"raw_tex":"We use bold lowercase letters $\\vx,\\vy,\\vg,\\ldots$ for vectors. $\\EE_{B_{k}}$ denotes the expectation about a mini-batch set $B_{k}$ conditionally on the all previous history, and $\\EE$ denotes the full expectation. $|B_k|$ counts the number of elements in the set $B_k$. We use $\\|\\cdot\\|$ for the Euclidean norm. A differentiable function $F$ is called $L$-smooth, if $\\|\\nabla F(\\vx)-\\nabla F(\\vy)\\|\\le L\\|\\vx-\\vy\\|$ for all $\\vx$ and $\\vy$.\n","sha256_utf8":"c01a82dd29becaba8ad6ec8d527f9631ad46afd6c62825b140fbb93faa866d92"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"近端梯度映射","source_units":[{"start":30802,"end":31134,"start_line":437,"end_line":442,"raw_tex":"\\begin{definition}[proximal gradient mapping]\\label{def:prox-map}\nGiven $\\vd$, $\\vx\\in\\dom(r)$, and $\\eta>0$, we define $P(\\vx,\\vd,\\eta)=\\frac{1}{\\eta}(\\vx-\\vx^+)$, where \n%\\begin{equation*}\n$\\vx^+ =\\argmin_\\vy \\textstyle \\left\\{\\langle \\vd, \\vy\\rangle  +\\frac{1}{\\eta}V(\\vy,\\vx) + r(\\vy)\\right\\}.$\n%\\end{equation*}\n\\end{definition}","sha256_utf8":"fcf48b267a8e7e584bb3a743724f9ab3566f7e5100eef267465051e1c2cf857f"}]},{"id":"S006","semantic_type":"derived_fact","kind":"derived_fact","title":"近端映射下降不等式","source_units":[{"start":31819,"end":32029,"start_line":450,"end_line":453,"raw_tex":"From \\cite[Lemma 1]{ghadimi2016mini}, it holds\n\\begin{equation}\\label{eq:lem1-lan}\n\\big\\langle \\vd, P(\\vx,\\vd,\\eta)\\big\\rangle \\ge \\|P(\\vx,\\vd,\\eta)\\|^2 + \\frac{1}{\\eta}\\big(r(\\vx^+)-r(\\vx)\\big).\n\\end{equation}","sha256_utf8":"e403ca54fc5e30d5540810d498d897828b9cc2f6f051d470cf9a9df8c5670064"}]},{"id":"S007","semantic_type":"derived_fact","kind":"derived_fact","title":"近端映射非扩张性","source_units":[{"start":32030,"end":32333,"start_line":454,"end_line":457,"raw_tex":"In addition, the proximal gradient mapping is nonexpansive from \\cite[Proposition 1]{ghadimi2016mini}, i.e.,\n\\begin{equation}\\label{eq:nonexp-P}\n\\|P(\\vx, \\vd_1, \\eta) - P(\\vx, \\vd_2, \\eta)\\| \\le \\|\\vd_1-\\vd_2\\|, \\ \\forall\\, \\vd_1,\\vd_2, \\, \\forall\\, \\vx\\in \\dom(r), \\, \\forall\\, \\eta > 0.\n\\end{equation}","sha256_utf8":"18a066a07814365d6a3c121e316e46df9dc40f6de478d4c1b1752399633f0f00"}]},{"id":"S008","semantic_type":"notation","kind":"notation","title":"估计与真实梯度映射","source_units":[{"start":32334,"end":32497,"start_line":458,"end_line":461,"raw_tex":"For each $k\\ge0$, we denote\n\\begin{equation}\\label{eq:vg-vgbar}\n\\vg^k = P(\\vx^k, \\vd^k, \\eta_k),\\quad \\bar\\vg^k = P(\\vx^k, \\nabla F(\\vx^k), \\eta_k).\n\\end{equation}","sha256_utf8":"cd4bf5446086a2fe2906fd7df08625b5b336f98f3124775a0077c09b536f6713"}]},{"id":"S009","semantic_type":"notation","kind":"notation","title":"梯度估计误差","source_units":[{"start":32498,"end":32699,"start_line":462,"end_line":465,"raw_tex":"Notice that $\\|\\bar\\vg^k\\|$ measures the violation of stationarity of $\\vx^k$. The gradient error is represented by\n\\begin{equation}\\label{eq:error-grad}\n\\ve^k = \\vd^k - \\nabla F(\\vx^k).\n\\end{equation}","sha256_utf8":"6722ee61072d533539b12647317637a2b00473ee579799a391d6d9f8d3e25a66"}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","title":"有限最优目标值","source_units":[{"start":33459,"end":33610,"start_line":474,"end_line":476,"raw_tex":"\\begin{assumption}[finite optimal objective]\\label{assump-obj}\nThe optimal objective value $\\Phi^*$ of \\eqref{eq:stoc-prob} is finite.\n\\end{assumption}","sha256_utf8":"63aec75d3374131ff1d4b0a87f5304856df844708ba0754e7b05405a6ee89c89"}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","title":"均方光滑性","source_units":[{"start":33612,"end":33895,"start_line":478,"end_line":481,"raw_tex":"\\begin{assumption}[mean-squared smoothness]\\label{assump-smooth}\nThe function $f(\\,\\cdot\\,;\\xi)$ satisfies the mean-squared smoothness condition:  \n$\\EE_\\xi \\big[\\|\\nabla f(\\vx;\\xi) - \\nabla f(\\vy;\\xi)\\|^2\\big] \\le L^2 \\|\\vx-\\vy\\|^2,\\, \\forall\\, \\vx,\\vy\\in \\dom(r).$\n\\end{assumption}","sha256_utf8":"05b62f4a26d2d85a50001b929e54e26caa69a79a38485ea09aed50a1ea0f3b5a"}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","title":"无偏性与有界方差","source_units":[{"start":33897,"end":34347,"start_line":483,"end_line":490,"raw_tex":"\\begin{assumption}[unbiasedness and variance boundedness]\\label{assump-sgd}\nThere is $\\sigma>0$ such that %for each $\\vx\\in \\dom(r)$, \n\\begin{align}\n\\EE_\\xi [\\nabla f(\\vx; \\xi)] = \\nabla F(\\vx),\\quad \\EE[\\|\\nabla f(\\vx; \\xi) - \\nabla F(\\vx)\\|^2] \\le \\sigma^2, \\ \\forall\\, \\vx\\in \\dom(r).\n%&\\EE_{B_k}[\\vv^k] = \\nabla F(\\vx^k),\\quad \\EE_{B_k}[\\vu^k] = \\nabla F(\\vx^{k-1}),\\\\\n%&\\EE[\\|\\vv^k-\\nabla F(\\vx^k)\\|^2] \\le \\sigma^2.\n\\end{align}\n\\end{assumption}","sha256_utf8":"b5d2c9bfe92c6612460139ff88cbb0a90f0a3b9bbb96709a2fd985aa8311fc41"}]},{"id":"S013","semantic_type":"derived_fact","kind":"derived_fact","title":"期望目标的光滑性","source_units":[{"start":34349,"end":34562,"start_line":492,"end_line":492,"raw_tex":"It is easy to show that under Assumptions~\\ref{assump-smooth} and \\ref{assump-sgd}, the function $F(\\vx)=\\EE_\\xi [f(\\vx;\\xi)]$ is $L$-smooth; see the arguments at the end of section~2.2 of \\cite{tran2021hybrid}. \n","sha256_utf8":"51edad21ff1b7847d55097515dbdefd502b29248985412476d200a2fc2ecec64"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"动量累积乘积","source_units":[{"start":57915,"end":58113,"start_line":735,"end_line":742,"raw_tex":"\\begin{equation}\\label{eq:gamma}\n\t\\Gamma_k = \\left\\{\n\t\\begin{array}{ll}\n\t\t\\prod_{i=0}^{k-1}(1-\\beta_i)^2, & \\text{ if } k\\ge 1, \\\\[0.2cm]\n\t\t1, & \\text{ if } k=0.\n\t\\end{array}\n\t\\right.\n\\end{equation}","sha256_utf8":"25dfd7b13baa5e2be01090b6d89c6c7f926b4e95903696fb53fe763ffd8cecaf"}]},{"id":"S015","semantic_type":"lemma","kind":"lemma","title":"单步目标下降","source_units":[{"start":34692,"end":35050,"start_line":495,"end_line":499,"raw_tex":"\\begin{lemma}[one-iteration progress]\\label{lem:obj-diff}\nLet $\\{\\vx^k\\}$ be generated from Algorithm~\\ref{alg:prox-storm}. If $F$ is $L$-smooth, then\n$$\\Phi(\\vx^{k+1})-\\Phi(\\vx^k) \\le \\frac{\\eta_k}{2}(2-\\eta_k L)\\|\\ve^k\\|^2 - \\frac{\\eta_k}{4}(1-\\eta_k L)\\|\\bar\\vg^k\\|^2,\\, \\forall\\, k\\ge 1,$$\nwhere $\\bar\\vg^k$ is defined in \\eqref{eq:vg-vgbar}.\n\\end{lemma}","sha256_utf8":"2979367c6fbf0cdc6d2727d6a04522f261bbefc760eb84defd39faadd0446d6e"}]},{"id":"S016","semantic_type":"proof","kind":"proof","title":"单步下降完整证明","source_units":[{"start":35052,"end":36689,"start_line":501,"end_line":519,"raw_tex":"\\begin{proof}\nBy the $L$-smoothness of $F$ and the definition of $\\vg^k$ in \\eqref{eq:vg-vgbar}, we have\n\\begin{equation}\\label{eq:lip-F-ineq}\nF(\\vx^{k+1})-F(\\vx^k)\\le \\langle \\nabla F(\\vx^k), \\vx^{k+1}-\\vx^k\\rangle + \\frac{L}{2}\\|\\vx^{k+1}-\\vx^k\\|^2 = -\\eta_k\\langle \\nabla F(\\vx^k), \\vg^k\\rangle + \\frac{\\eta_k^2L}{2}\\|\\vg^k\\|^2.\n\\end{equation}\nUsing the definition of $\\ve^k$ in \\eqref{eq:error-grad} and the inequality in \\eqref{eq:lem1-lan}, we have \n$$-\\langle \\nabla F(\\vx^k), \\vg^k\\rangle = \\langle \\ve^k, \\vg^k\\rangle - \\langle \\vd^k, \\vg^k\\rangle \\le \\langle \\ve^k, \\vg^k\\rangle - \\|\\vg^k\\|^2 + \\frac{1}{\\eta_k}\\big(r(\\vx^k)-r(\\vx^{k+1})\\big).$$\nPlugging the above inequality into \\eqref{eq:lip-F-ineq} and rearranging terms give\n$$\\Phi(\\vx^{k+1})-\\Phi(\\vx^k) \\le \\eta_k\\langle \\ve^k, \\vg^k\\rangle - \\eta_k \\|\\vg^k\\|^2+ \\frac{\\eta_k^2L}{2}\\|\\vg^k\\|^2.$$\nBy the Cauchy-Schwartz inequality, it holds $\\eta_k\\langle \\ve^k, \\vg^k\\rangle \\le \\frac{\\eta_k}{2}\\|\\ve^k\\|^2 + \\frac{\\eta_k}{2}\\|\\vg^k\\|^2$, which together with the above inequality implies\n\\begin{equation}\\label{eq:chg-obj-0}\n\\Phi(\\vx^{k+1})-\\Phi(\\vx^k) \\le \\frac{\\eta_k}{2}\\|\\ve^k\\|^2 - \\frac{\\eta_k}{2}(1-\\eta_k L)\\|\\vg^k\\|^2.\n\\end{equation}\nFrom \\eqref{eq:nonexp-P} and the definitions of $\\vg^k$ and $\\bar\\vg^k$ in \\eqref{eq:vg-vgbar}, it follows\n\\begin{equation}\\label{eq:tri-g}\n-\\|\\vg^k\\|^2 \\le -\\frac{1}{2}\\|\\bar\\vg^k\\|^2 + \\|\\vg^k-\\bar\\vg^k\\|^2\\le -\\frac{1}{2}\\|\\bar\\vg^k\\|^2 + \\|\\vd^k -\\nabla F(\\vx^k)\\|^2 = -\\frac{1}{2}\\|\\bar\\vg^k\\|^2 + \\|\\ve^k\\|^2.\n\\end{equation}\nNow plug the above inequality into \\eqref{eq:chg-obj-0} to give the desired result.\n\\end{proof}","sha256_utf8":"22ecb55b253b60951ddebc0909416cf052dd0dbc66ff8b1c3d03a174d5b0dfa0"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"光滑目标变化界","source_units":[{"start":35052,"end":35399,"start_line":501,"end_line":505,"raw_tex":"\\begin{proof}\nBy the $L$-smoothness of $F$ and the definition of $\\vg^k$ in \\eqref{eq:vg-vgbar}, we have\n\\begin{equation}\\label{eq:lip-F-ineq}\nF(\\vx^{k+1})-F(\\vx^k)\\le \\langle \\nabla F(\\vx^k), \\vx^{k+1}-\\vx^k\\rangle + \\frac{L}{2}\\|\\vx^{k+1}-\\vx^k\\|^2 = -\\eta_k\\langle \\nabla F(\\vx^k), \\vg^k\\rangle + \\frac{\\eta_k^2L}{2}\\|\\vg^k\\|^2.\n\\end{equation}\n","sha256_utf8":"fc50a2f7670d2722a85facf0cb98759889da4082860c26519b386193bd3719dc"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"加入近端正则项","source_units":[{"start":35399,"end":35916,"start_line":506,"end_line":509,"raw_tex":"Using the definition of $\\ve^k$ in \\eqref{eq:error-grad} and the inequality in \\eqref{eq:lem1-lan}, we have \n$$-\\langle \\nabla F(\\vx^k), \\vg^k\\rangle = \\langle \\ve^k, \\vg^k\\rangle - \\langle \\vd^k, \\vg^k\\rangle \\le \\langle \\ve^k, \\vg^k\\rangle - \\|\\vg^k\\|^2 + \\frac{1}{\\eta_k}\\big(r(\\vx^k)-r(\\vx^{k+1})\\big).$$\nPlugging the above inequality into \\eqref{eq:lip-F-ineq} and rearranging terms give\n$$\\Phi(\\vx^{k+1})-\\Phi(\\vx^k) \\le \\eta_k\\langle \\ve^k, \\vg^k\\rangle - \\eta_k \\|\\vg^k\\|^2+ \\frac{\\eta_k^2L}{2}\\|\\vg^k\\|^2.$$\n","sha256_utf8":"9945be4cbd42ff511255059a2bde615549ec0de2965b382fd08a201f8dfddab6"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"分离误差与下降量","source_units":[{"start":35916,"end":36263,"start_line":510,"end_line":513,"raw_tex":"By the Cauchy-Schwartz inequality, it holds $\\eta_k\\langle \\ve^k, \\vg^k\\rangle \\le \\frac{\\eta_k}{2}\\|\\ve^k\\|^2 + \\frac{\\eta_k}{2}\\|\\vg^k\\|^2$, which together with the above inequality implies\n\\begin{equation}\\label{eq:chg-obj-0}\n\\Phi(\\vx^{k+1})-\\Phi(\\vx^k) \\le \\frac{\\eta_k}{2}\\|\\ve^k\\|^2 - \\frac{\\eta_k}{2}(1-\\eta_k L)\\|\\vg^k\\|^2.\n\\end{equation}\n","sha256_utf8":"028734b44a3ac2815ab69e713896c2d9457092f7193390da4d5a5455ae9058ae"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"比较两种梯度映射","source_units":[{"start":36263,"end":36594,"start_line":514,"end_line":517,"raw_tex":"From \\eqref{eq:nonexp-P} and the definitions of $\\vg^k$ and $\\bar\\vg^k$ in \\eqref{eq:vg-vgbar}, it follows\n\\begin{equation}\\label{eq:tri-g}\n-\\|\\vg^k\\|^2 \\le -\\frac{1}{2}\\|\\bar\\vg^k\\|^2 + \\|\\vg^k-\\bar\\vg^k\\|^2\\le -\\frac{1}{2}\\|\\bar\\vg^k\\|^2 + \\|\\vd^k -\\nabla F(\\vx^k)\\|^2 = -\\frac{1}{2}\\|\\bar\\vg^k\\|^2 + \\|\\ve^k\\|^2.\n\\end{equation}\n","sha256_utf8":"b9c98bd68b25bfdfb256f391ee9d48498f5c1ce541149e31ab691c4ff8de9098"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"得到单步进展结论","source_units":[{"start":36594,"end":36689,"start_line":518,"end_line":519,"raw_tex":"Now plug the above inequality into \\eqref{eq:chg-obj-0} to give the desired result.\n\\end{proof}","sha256_utf8":"09e2c407b8a5c04170a7b351b84d8fa5760fb32703a65bb467371cc573a7f032"}]},{"id":"S022","semantic_type":"lemma","kind":"lemma","title":"梯度误差递推界","source_units":[{"start":36845,"end":37356,"start_line":522,"end_line":526,"raw_tex":"\\begin{lemma}[recursive bound on gradient error]\\label{lem:bd-ek}\nUnder Assumptions~\\ref{assump-smooth} and \\ref{assump-sgd}, it holds %for each $k\\ge 0$ that\n$$\\EE\\big[\\|\\ve^{k+1}\\|^2\\big] \\le \\frac{2\\beta_k^2\\sigma^2}{m} + \\frac{4(1-\\beta_k)^2\\eta_{k}^2 L^2}{m} \\EE\\big[\\|\\bar\\vg^{k}\\|^2\\big] + (1-\\beta_k)^2\\left(\\textstyle 1+\\frac{4\\eta_{k}^2 L^2}{m}\\right)\\EE\\big[\\|\\ve^{k}\\|^2\\big], \\forall\\, k\\ge0,$$\nwhere $\\bar\\vg^k$ and $\\ve^k$ are defined in \\eqref{eq:vg-vgbar} and \\eqref{eq:error-grad}.\n\\end{lemma}","sha256_utf8":"2e6b2e2b67934d25c1dcc899bf5a389069d8b9d2603d429b67edf3ce02cb6ed4"}]},{"id":"S023","semantic_type":"proof","kind":"proof","title":"误差递推完整证明","source_units":[{"start":37358,"end":41102,"start_line":528,"end_line":568,"raw_tex":"\\begin{proof}\nFirst, notice $\\EE_{B_{k+1}}[\\langle \\vv^{k+1}, \\ve^{k}\\rangle] = \\langle\\nabla F(\\vx^{k+1}), \\ve^{k}\\rangle$ and $\\EE_{B_{k+1}}[\\langle \\vu^{k+1}, \\ve^{k}\\rangle] = \\langle\\nabla F(\\vx^{k}), \\ve^{k}\\rangle$, and thus\n\\begin{equation}\\label{eq:crs-term0}\n\\EE_{B_{k+1}}[\\langle \\vv^{k+1} - \\nabla F(\\vx^{k+1}), \\ve^{k}\\rangle] = 0,\\quad \\EE_{B_{k+1}}[\\langle \\vu^{k+1} - \\nabla F(\\vx^{k}), \\ve^{k}\\rangle] = 0.\n\\end{equation}\nHence, by writing $\\ve^{k+1} = \\vv^{k+1} - \\nabla F(\\vx^{k+1}) + (1-\\beta_k)(\\nabla F(\\vx^{k})- \\vu^{k+1})+(1-\\beta_k)\\ve^{k}$, we have\n\\begin{equation}\\label{eq:one-step-e}\n\\EE_{B_{k+1}}\\big[\\|\\ve^{k+1}\\|^2\\big] = \\EE_{B_{k+1}}\\big[\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1}) + (1-\\beta_k)(\\nabla F(\\vx^{k})- \\vu^{k+1})\\|^2\\big] + (1-\\beta_k)^2\\|\\ve^{k}\\|^2.\n\\end{equation}\nBy the Young's inequality, it holds\n\\begin{align}\\label{eq:step2}\n&~\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1}) + (1-\\beta_k)(\\nabla F(\\vx^{k})- \\vu^{k+1})\\|^2 \\cr\n= &~ \\big\\|\\beta_k\\big(\\vv^{k+1} - \\nabla F(\\vx^{k+1})\\big) + (1-\\beta_k)\\big(\\vv^{k+1} - \\nabla F(\\vx^{k+1}) + \\nabla F(\\vx^{k})- \\vu^{k+1}\\big)\\big\\|^2\\cr\n\\le &~ 2\\beta_k^2\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1})\\|^2 + 2(1-\\beta_k)^2\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1}) + \\nabla F(\\vx^{k})- \\vu^{k+1}\\|^2.\n\\end{align}\nFrom the definition of $\\vv^{k+1}$ and $ \\vu^{k+1}$ in \\eqref{eq:def-v-u-k}, we have \n\\begin{align}\\label{eq:step23}\n \t&~\\EE_{B_{k+1}}\\big[\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1}) + \\nabla F(\\vx^{k})- \\vu^{k+1}\\|^2\\big] \\cr \n \t=&~\\frac{1}{m^2}\\EE_{B_{k+1}}\\left\\| \\sum_{\\xi\\in B_{k+1}}\\left(\\nabla f(\\vx^{k+1};\\xi)-\\nabla f(\\vx^{k};\\xi)  - \\nabla F(\\vx^{k+1})+\\nabla F(\\vx^{k})\\right) \\right\\|^2 \t \\cr\n \t=&~\\frac{1}{m^2}\\sum_{j=1}^m\\EE_{\\xi_j^{k+1}}\\left\\| \\nabla f(\\vx^{k+1};\\xi_j^{k+1})-\\nabla f(\\vx^{k};\\xi_j^{k+1})  - \\nabla F(\\vx^{k+1})+\\nabla F(\\vx^{k})  \\right\\|^2 \t \\cr\n \t\\le&~\\frac{1}{m^2}\\sum_{j=1}^m\\EE_{\\xi_j^{k+1}}\\left\\| \\nabla f(\\vx^{k+1};\\xi_j^{k+1})-\\nabla f(\\vx^{k};\\xi_j^{k+1}) \\right\\|^2 \t \\cr\t\n \t\\le&~\\frac{L^2}{m}\\|\\vx^{k+1}-\\vx^{k}\\|^2,\n\\end{align}\nwhere the second equality holds because of the i.i.d. samples in $B_{k+1}$ and the zero mean of the random vector $\\nabla f(\\vx^{k+1};\\xi_j^{k+1})-\\nabla f(\\vx^{k};\\xi_j^{k+1})  - \\nabla F(\\vx^{k+1})+\\nabla F(\\vx^{k}) $ resulted from unbiasedness in Assumption~\\ref{assump-sgd}, \nthe first inequality is due to the fact that the variance of a random vector is upper-bounded by its second moment, and the last inequality follows from Assumption~\\ref{assump-smooth}.\n\nNow, take conditional expectation on both sides of \\eqref{eq:step2}, use \\eqref{eq:step23}, and substitute it into \\eqref{eq:one-step-e}. We have\n$$\\EE_{B_{k+1}}\\big[\\|\\ve^{k+1}\\|^2\\big] \\le (1-\\beta_k)^2\\|\\ve^{k}\\|^2 +2\\beta_k^2\\EE_{B_{k+1}} \\big[\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1})\\|^2\\big] + \\frac{2(1-\\beta_k)^2 L^2}{m} \\EE_{B_{k+1}}\\big[\\|\\vx^{k+1}-\\vx^{k}\\|^2\\big] .$$\nTaking a full expectation over the above inequality and using  Assumption~\\ref{assump-sgd}, we have\n\t\\begin{align}\\label{eq:step32}\n\t\t\\EE\\big[\\|\\ve^{k+1}\\|^2\\big] \\le &~ (1-\\beta_k)^2\\EE\\big[\\|\\ve^{k}\\|^2\\big]+ \\frac{2\\beta_k^2\\sigma^2}{m} + \\frac{2(1-\\beta_k)^2 L^2}{m} \\EE\\big[\\|\\vx^{k+1}-\\vx^{k}\\|^2\\big]\\cr\n\t\t= & ~ (1-\\beta_k)^2\\EE\\big[\\|\\ve^{k}\\|^2\\big]+ \\frac{2\\beta_k^2\\sigma^2}{m} + \\frac{2(1-\\beta_k)^2 \\eta_k^2 L^2}{m} \\EE\\big[\\|\\vg^{k}\\|^2\\big],\n\t\\end{align}\nwhere we have used\t$\\vx^{k+1}-\\vx^{k} = -\\eta_{k} \\vg^{k}$ in the equality.\n\t\n\nBy similar arguments as those in \\eqref{eq:tri-g}, it holds\n$$\\|\\vg^{k}\\|^2 \\le 2\\|\\bar\\vg^{k}\\|^2 + 2\\|\\vg^{k}-\\bar\\vg^{k}\\|^2\\le 2\\|\\bar\\vg^{k}\\|^2  + 2\\|\\ve^{k}\\|^2.$$\n%Now notice $\\vx^{k+1}-\\vx^{k} = -\\eta_{k} \\vg^{k}$ and \nPlugging the above inequality into \\eqref{eq:step32}, we obtain the desired result.\n\\end{proof}","sha256_utf8":"2b8abe18b38d3fd6c5c89b2afb3531ff83466de7b72c543093878dd0d693dfa4"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"条件交叉项为零","source_units":[{"start":37358,"end":37797,"start_line":528,"end_line":532,"raw_tex":"\\begin{proof}\nFirst, notice $\\EE_{B_{k+1}}[\\langle \\vv^{k+1}, \\ve^{k}\\rangle] = \\langle\\nabla F(\\vx^{k+1}), \\ve^{k}\\rangle$ and $\\EE_{B_{k+1}}[\\langle \\vu^{k+1}, \\ve^{k}\\rangle] = \\langle\\nabla F(\\vx^{k}), \\ve^{k}\\rangle$, and thus\n\\begin{equation}\\label{eq:crs-term0}\n\\EE_{B_{k+1}}[\\langle \\vv^{k+1} - \\nabla F(\\vx^{k+1}), \\ve^{k}\\rangle] = 0,\\quad \\EE_{B_{k+1}}[\\langle \\vu^{k+1} - \\nabla F(\\vx^{k}), \\ve^{k}\\rangle] = 0.\n\\end{equation}\n","sha256_utf8":"7be3794b3dd281ef4a331caa5329dc0cba9aac274aba4a78dde2d4b6382a439e"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"展开下一步误差","source_units":[{"start":37797,"end":38162,"start_line":533,"end_line":536,"raw_tex":"Hence, by writing $\\ve^{k+1} = \\vv^{k+1} - \\nabla F(\\vx^{k+1}) + (1-\\beta_k)(\\nabla F(\\vx^{k})- \\vu^{k+1})+(1-\\beta_k)\\ve^{k}$, we have\n\\begin{equation}\\label{eq:one-step-e}\n\\EE_{B_{k+1}}\\big[\\|\\ve^{k+1}\\|^2\\big] = \\EE_{B_{k+1}}\\big[\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1}) + (1-\\beta_k)(\\nabla F(\\vx^{k})- \\vu^{k+1})\\|^2\\big] + (1-\\beta_k)^2\\|\\ve^{k}\\|^2.\n\\end{equation}\n","sha256_utf8":"9a5886d98187e92d765fb0f0123579770c27ea9af8486c5fd2a3a6e3b0b752f3"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"杨氏不等式拆分噪声","source_units":[{"start":38162,"end":38626,"start_line":537,"end_line":542,"raw_tex":"By the Young's inequality, it holds\n\\begin{align}\\label{eq:step2}\n&~\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1}) + (1-\\beta_k)(\\nabla F(\\vx^{k})- \\vu^{k+1})\\|^2 \\cr\n= &~ \\big\\|\\beta_k\\big(\\vv^{k+1} - \\nabla F(\\vx^{k+1})\\big) + (1-\\beta_k)\\big(\\vv^{k+1} - \\nabla F(\\vx^{k+1}) + \\nabla F(\\vx^{k})- \\vu^{k+1}\\big)\\big\\|^2\\cr\n\\le &~ 2\\beta_k^2\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1})\\|^2 + 2(1-\\beta_k)^2\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1}) + \\nabla F(\\vx^{k})- \\vu^{k+1}\\|^2.\n\\end{align}\n","sha256_utf8":"013611cd707f03f196c6162400e99940894563f8545bc6a468f7508540aec294"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"同批梯度增量方差界","source_units":[{"start":38626,"end":39856,"start_line":543,"end_line":552,"raw_tex":"From the definition of $\\vv^{k+1}$ and $ \\vu^{k+1}$ in \\eqref{eq:def-v-u-k}, we have \n\\begin{align}\\label{eq:step23}\n \t&~\\EE_{B_{k+1}}\\big[\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1}) + \\nabla F(\\vx^{k})- \\vu^{k+1}\\|^2\\big] \\cr \n \t=&~\\frac{1}{m^2}\\EE_{B_{k+1}}\\left\\| \\sum_{\\xi\\in B_{k+1}}\\left(\\nabla f(\\vx^{k+1};\\xi)-\\nabla f(\\vx^{k};\\xi)  - \\nabla F(\\vx^{k+1})+\\nabla F(\\vx^{k})\\right) \\right\\|^2 \t \\cr\n \t=&~\\frac{1}{m^2}\\sum_{j=1}^m\\EE_{\\xi_j^{k+1}}\\left\\| \\nabla f(\\vx^{k+1};\\xi_j^{k+1})-\\nabla f(\\vx^{k};\\xi_j^{k+1})  - \\nabla F(\\vx^{k+1})+\\nabla F(\\vx^{k})  \\right\\|^2 \t \\cr\n \t\\le&~\\frac{1}{m^2}\\sum_{j=1}^m\\EE_{\\xi_j^{k+1}}\\left\\| \\nabla f(\\vx^{k+1};\\xi_j^{k+1})-\\nabla f(\\vx^{k};\\xi_j^{k+1}) \\right\\|^2 \t \\cr\t\n \t\\le&~\\frac{L^2}{m}\\|\\vx^{k+1}-\\vx^{k}\\|^2,\n\\end{align}\nwhere the second equality holds because of the i.i.d. samples in $B_{k+1}$ and the zero mean of the random vector $\\nabla f(\\vx^{k+1};\\xi_j^{k+1})-\\nabla f(\\vx^{k};\\xi_j^{k+1})  - \\nabla F(\\vx^{k+1})+\\nabla F(\\vx^{k}) $ resulted from unbiasedness in Assumption~\\ref{assump-sgd}, \nthe first inequality is due to the fact that the variance of a random vector is upper-bounded by its second moment, and the last inequality follows from Assumption~\\ref{assump-smooth}.\n","sha256_utf8":"4e646c666622bff331f706529de16caf119d3a7a2f6134644ed9f7bf5b9ac0d8"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"得到映射形式误差递推","source_units":[{"start":39857,"end":40776,"start_line":554,"end_line":561,"raw_tex":"Now, take conditional expectation on both sides of \\eqref{eq:step2}, use \\eqref{eq:step23}, and substitute it into \\eqref{eq:one-step-e}. We have\n$$\\EE_{B_{k+1}}\\big[\\|\\ve^{k+1}\\|^2\\big] \\le (1-\\beta_k)^2\\|\\ve^{k}\\|^2 +2\\beta_k^2\\EE_{B_{k+1}} \\big[\\|\\vv^{k+1} - \\nabla F(\\vx^{k+1})\\|^2\\big] + \\frac{2(1-\\beta_k)^2 L^2}{m} \\EE_{B_{k+1}}\\big[\\|\\vx^{k+1}-\\vx^{k}\\|^2\\big] .$$\nTaking a full expectation over the above inequality and using  Assumption~\\ref{assump-sgd}, we have\n\t\\begin{align}\\label{eq:step32}\n\t\t\\EE\\big[\\|\\ve^{k+1}\\|^2\\big] \\le &~ (1-\\beta_k)^2\\EE\\big[\\|\\ve^{k}\\|^2\\big]+ \\frac{2\\beta_k^2\\sigma^2}{m} + \\frac{2(1-\\beta_k)^2 L^2}{m} \\EE\\big[\\|\\vx^{k+1}-\\vx^{k}\\|^2\\big]\\cr\n\t\t= & ~ (1-\\beta_k)^2\\EE\\big[\\|\\ve^{k}\\|^2\\big]+ \\frac{2\\beta_k^2\\sigma^2}{m} + \\frac{2(1-\\beta_k)^2 \\eta_k^2 L^2}{m} \\EE\\big[\\|\\vg^{k}\\|^2\\big],\n\t\\end{align}\nwhere we have used\t$\\vx^{k+1}-\\vx^{k} = -\\eta_{k} \\vg^{k}$ in the equality.\n","sha256_utf8":"5fc9b049a10d5c43d674cf95df589b919964df7460259c7051176b2ecf3ed7b8"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"转换为真实映射递推","source_units":[{"start":40779,"end":41102,"start_line":564,"end_line":568,"raw_tex":"By similar arguments as those in \\eqref{eq:tri-g}, it holds\n$$\\|\\vg^{k}\\|^2 \\le 2\\|\\bar\\vg^{k}\\|^2 + 2\\|\\vg^{k}-\\bar\\vg^{k}\\|^2\\le 2\\|\\bar\\vg^{k}\\|^2  + 2\\|\\ve^{k}\\|^2.$$\n%Now notice $\\vx^{k+1}-\\vx^{k} = -\\eta_{k} \\vg^{k}$ and \nPlugging the above inequality into \\eqref{eq:step32}, we obtain the desired result.\n\\end{proof}","sha256_utf8":"dea59f9b7d7f91f1e0ca8c055d8b8e7fdf3b1981f84908f88e4fc74ebd7e3624"}]},{"id":"S030","semantic_type":"theorem","kind":"theorem","title":"动量累积通用收敛界","source_units":[{"start":58271,"end":59306,"start_line":745,"end_line":759,"raw_tex":"\\begin{theorem}\\label{thm:main}\n\tUnder Assumptions~\\ref{assump-obj} through \\ref{assump-sgd}, let $\\{\\vx^k\\}$ be the iterate sequence from Algorithm~\\ref{alg:prox-storm}. Suppose there are constants $A$ and $B$ such that the parameters $\\{\\eta_k\\}$ and $\\{\\beta_k\\}$ satisfying the conditions:\n\t\\begin{equation}\\label{eq:cond-eta-beta1}\n\t\t2\\eta_k L+ \\frac{4 L^2}{m} \\frac{\\eta_{k}}{\\Gamma_{k} } \\sum_{j=k+1}^{K-1}\\eta_j\\Gamma_{j} \\le 1, \\,\\forall\\, k=0,\\ldots,K-1,\n\t\\end{equation}\n\t\\begin{equation}\\label{eq:cond-eta-beta2}\n\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\le A,~\\text{and}~\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} }\\le B,\n\\end{equation}\nwhere $K$ is the maximum number of iterations in Algorithm~\\ref{alg:prox-storm}. \n\tLet $\\{\\bar\\vg^k\\}$ be defined in \\eqref{eq:vg-vgbar}. Then\t\n\t\\begin{equation}\\label{eq:mainthm}\n\t\t\\sum_{k=0}^{K-1} \\eta_k \\EE\\big[\\|\\bar\\vg^{k}\\|^2\\big] \n\t\t\\le\t12\\big[\\Phi(\\vx^0)-\\Phi^*\\big] +4(2A+3)\\frac{\\sigma^2}{ m_0}+ 16B\\frac{\\sigma^2}{m} .\n\t\\end{equation}\t\n\\end{theorem}","sha256_utf8":"faf186b14f667534232dd2a8fdfc750cd1d5bb62c4204be2c637b3597dbe32b1"}]},{"id":"S031","semantic_type":"assumption","kind":"assumption","title":"通用收敛参数条件","source_units":[{"start":58303,"end":59028,"start_line":746,"end_line":753,"raw_tex":"\tUnder Assumptions~\\ref{assump-obj} through \\ref{assump-sgd}, let $\\{\\vx^k\\}$ be the iterate sequence from Algorithm~\\ref{alg:prox-storm}. Suppose there are constants $A$ and $B$ such that the parameters $\\{\\eta_k\\}$ and $\\{\\beta_k\\}$ satisfying the conditions:\n\t\\begin{equation}\\label{eq:cond-eta-beta1}\n\t\t2\\eta_k L+ \\frac{4 L^2}{m} \\frac{\\eta_{k}}{\\Gamma_{k} } \\sum_{j=k+1}^{K-1}\\eta_j\\Gamma_{j} \\le 1, \\,\\forall\\, k=0,\\ldots,K-1,\n\t\\end{equation}\n\t\\begin{equation}\\label{eq:cond-eta-beta2}\n\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\le A,~\\text{and}~\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} }\\le B,\n\\end{equation}\nwhere $K$ is the maximum number of iterations in Algorithm~\\ref{alg:prox-storm}. \n","sha256_utf8":"4d3dcdecf8b4630ff1b8e57d6b6bccd1c0997b156136d8a69767ff75e672c572"}]},{"id":"S032","semantic_type":"proof","kind":"proof","title":"通用收敛界完整证明","source_units":[{"start":59307,"end":64259,"start_line":760,"end_line":811,"raw_tex":"\\begin{proof}\nWe begin by taking the total expectation and \ntelescoping the inequality in \\eqref{eq:chg-obj-0} over $k=0,\\ldots,K-1$ to obtain\n\\begin{align*}\n\t\\EE\\big[\\Phi(\\vx^{K})\\big]-\\Phi(\\vx^0) \n\t\\le&~ \\sum_{k=0}^{K-1}\\frac{\\eta_k}{2} \\EE\\big[\\|\\ve^k\\|^2\\big] -\\sum_{k=0}^{K-1} \\frac{\\eta_k}{2}(1-\\eta_k L)\\EE\\big[\\|\\vg^k\\|^2\\big]\\\\\n\t\\le&~ \\frac{\\sigma^2}{m_0}+ \\sum_{k=1}^{K-1}\\frac{\\eta_k}{2} \\EE\\big[\\|\\ve^k\\|^2\\big] -\\sum_{k=0}^{K-1} \\frac{\\eta_k}{2}(1-\\eta_k L)\\EE\\big[\\|\\vg^k\\|^2\\big],\n\\end{align*}\nwhere we have used $\\EE\\big[\\|\\ve^{0}\\|^2\\big]\\le \\frac{\\sigma^2}{m_0}$ by Assumption~\\ref{assump-sgd}. Since $\\Phi(\\vx^{K}) \\ge \\Phi^*$ from Assumption~\\ref{assump-obj}, the above inequality implies \n\\begin{equation}\\label{eq:main1}\n\t\\sum_{k=0}^{K-1} \\frac{\\eta_k}{2}(1-\\eta_k L)\\EE\\big[\\|\\vg^k\\|^2\\big] \n\t\\le \\Phi(\\vx^0)-\\Phi^* +\\frac{\\sigma^2}{m_0}+ \\sum_{k=1}^{K-1}\\frac{\\eta_k}{2} \\EE\\big[\\|\\ve^k\\|^2\\big].\n\\end{equation}\t\n\nIn addition, we divide both sides of \\eqref{eq:step32} by $\\Gamma_{k+1}$ and obtain from the definition of $\\Gamma_{k+1}$ in \\eqref{eq:gamma} that\n\t$$\\frac{1}{\\Gamma_{k+1} }\\EE\\big[\\|\\ve^{k+1}\\|^2\\big] \\le \\frac{1}{\\Gamma_{k} }\\EE\\big[\\|\\ve^{k}\\|^2\\big] +\\frac{1}{\\Gamma_{k+1} }\\frac{2\\beta_k^2\\sigma^2}{m} + \\frac{1}{\\Gamma_{k} }\\frac{2\\eta_{k}^2 L^2}{m} \\EE\\big[\\|\\vg^{k}\\|^2\\big],\\, \\forall\\, k\\ge 0.$$\nLet $j=0,\\ldots,k-1$ be another index on which the above inequality is telescoped. We obtain\n\t$$\\frac{1}{\\Gamma_{k} }\\EE\\big[\\|\\ve^{k}\\|^2\\big] \\le \\EE\\big[\\|\\ve^{0}\\|^2\\big] +\\sum_{j=0}^{k-1}\\frac{1}{\\Gamma_{j+1} }\\frac{2\\beta_j^2\\sigma^2}{m} + \\sum_{j=0}^{k-1}\\frac{1}{\\Gamma_{j} }\\frac{2\\eta_{j}^2 L^2}{m} \\EE\\big[\\|\\vg^{j}\\|^2\\big],\\, \\forall\\, k\\ge 1.$$\nMultiplying $\\Gamma_{k}$ to both sides of the above inequality and rearranging it gives\n%which further expands and rearranges into \n$$ \t\\EE\\big[\\|\\ve^{k}\\|^2\\big] \\le \\Gamma_{k}\\bigg(\\frac{\\sigma^2}{m_0} +\\frac{2\\sigma^2}{m}\\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \\bigg)+ \\frac{2 L^2}{m}\\sum_{j=0}^{k-1}\\frac{\\Gamma_{k}}{\\Gamma_{j} } \\eta_{j}^2  \\EE\\big[\\|\\vg^{j}\\|^2\\big],\\, \\forall\\, k\\ge 1,$$\nwhere we have used $\\EE\\big[\\|\\ve^{0}\\|^2\\big]\\le \\frac{\\sigma^2}{m_0}$ again. \n Now multiply $\\eta_k$ to the above inequality and sum it up over $k=1,\\ldots,K-1$ to have\n\\begin{align}\\label{eq:main2}\n \t\\sum_{k=1}^{K-1}\\eta_k\\EE\\big[\\|\\ve^{k}\\|^2\\big] \n \t\\le&~ \\sigma^2\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\bigg(\\frac{1}{m_0} +\\frac{2}{m} \\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \\bigg)+\\frac{2 L^2}{m}\\sum_{k=1}^{K-1} \\sum_{j=0}^{k-1}\\frac{\\eta_k\\Gamma_{k}}{\\Gamma_{j} } \\eta_{j}^2  \\EE\\big[\\|\\vg^{j}\\|^2\\big] \\cr\n \t=&~ \\sigma^2\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\bigg(\\frac{1}{m_0} +\\frac{2}{m} \\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \\bigg)+\\frac{2 L^2}{m}\\sum_{j=0}^{K-2}\\frac{\\eta_{j}^2}{\\Gamma_{j} }\\bigg(\\sum_{k=j+1}^{K-1}\\eta_k\\Gamma_{k}\\bigg)    \\EE\\big[\\|\\vg^{j}\\|^2\\big] \\cr\n \t=&~ \\sigma^2\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\bigg(\\frac{1}{m_0} +\\frac{2}{m} \\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \\bigg)+\\frac{2 L^2}{m}\\sum_{k=0}^{K-1}\\frac{\\eta_{k}^2}{\\Gamma_{k} }\\bigg(\\sum_{j=k+1}^{K-1}\\eta_j\\Gamma_{j}\\bigg)    \\EE\\big[\\|\\vg^{k}\\|^2\\big],\n\\end{align}\nwhere the first equality follows by swapping summation, and the second equality is obtained by swapping indices and realizing that the coefficient for $\\EE\\big[\\|\\vg^{K-1}\\|^2\\big]$ is null.\n\nNow we have by substituting \\eqref{eq:main2} into \\eqref{eq:main1} and rearranging terms that\n$$\n\t\\sum_{k=0}^{K-1} \\frac{\\eta_k}{2}\\bigg(1-\\eta_k L- \\frac{2 L^2}{m} \\frac{\\eta_{k}}{\\Gamma_{k} } \\sum_{j=k+1}^{K-1}\\eta_j\\Gamma_{j}\\bigg)\\EE\\big[\\|\\vg^k\\|^2\\big] \n\t\\le \\Phi(\\vx^0)-\\Phi^* +\\frac{\\sigma^2}{m_0}+ \\frac{\\sigma^2}{2}\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\bigg(\\frac{1}{m_0} +\\frac{2}{m} \\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \\bigg),\n$$\nwhich together with the conditions in \\eqref{eq:cond-eta-beta1} and \\eqref{eq:cond-eta-beta2} gives the bound for $\\vg^k$:\n\\begin{equation}\\label{eq:main3}\n\t\\sum_{k=0}^{K-1}  \\eta_k \\EE\\big[\\|\\vg^k\\|^2\\big] \n\t\\le 4\\big[\\Phi(\\vx^0)-\\Phi^*\\big]  + 2(A+2)\\frac{\\sigma^2}{ m_0}+ 4B\\frac{\\sigma^2}{m} .\n\\end{equation}\t\nUse \\eqref{eq:cond-eta-beta1} again and substitute \\eqref{eq:main3} into \\eqref{eq:main2}. We obtain the bound for $\\ve^k$:\n\\begin{equation}\\label{eq:main4}\n\t\\sum_{k=0}^{K-1} \\eta_k \\EE\\big[\\|\\ve^k\\|^2\\big] \n\t\\le A\\frac{\\sigma^2}{ m_0}+ 2B\\frac{\\sigma^2}{m} +\\sum_{k=0}^{K-1}\\frac{\\eta_{k}}{2} \\EE\\big[\\|\\vg^{k}\\|^2\\big]\n\t\\le 2\\big[\\Phi(\\vx^0)-\\Phi^*\\big] +2(A+1)\\frac{\\sigma^2}{ m_0}+ 4B\\frac{\\sigma^2}{m} .\n\\end{equation}\t\n\nFinally, we have from \\eqref{eq:tri-g} that  %from \\eqref{eq:nonexp-P} and the definitions of $\\vg^k$ and $\\bar\\vg^k$ in \\eqref{eq:vg-vgbar}, it follows \n$\\|\\bar\\vg^{k}\\|^2 \\le 2\\|\\vg^{k}\\|^2  + 2\\|\\ve^{k}\\|^2.$ Sum up this inequality over $k=0,\\ldots,K-1$ and\nsubstitute \\eqref{eq:main3} and  \\eqref{eq:main4} into the summation. We obtain the result in \\eqref{eq:mainthm}.\n\\end{proof}","sha256_utf8":"f8c1462a1d1b488c8a98b843879fae089fb6bce7b67cb5560cb35c8d6cd57cad"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"累加目标下降不等式","source_units":[{"start":59307,"end":60244,"start_line":760,"end_line":772,"raw_tex":"\\begin{proof}\nWe begin by taking the total expectation and \ntelescoping the inequality in \\eqref{eq:chg-obj-0} over $k=0,\\ldots,K-1$ to obtain\n\\begin{align*}\n\t\\EE\\big[\\Phi(\\vx^{K})\\big]-\\Phi(\\vx^0) \n\t\\le&~ \\sum_{k=0}^{K-1}\\frac{\\eta_k}{2} \\EE\\big[\\|\\ve^k\\|^2\\big] -\\sum_{k=0}^{K-1} \\frac{\\eta_k}{2}(1-\\eta_k L)\\EE\\big[\\|\\vg^k\\|^2\\big]\\\\\n\t\\le&~ \\frac{\\sigma^2}{m_0}+ \\sum_{k=1}^{K-1}\\frac{\\eta_k}{2} \\EE\\big[\\|\\ve^k\\|^2\\big] -\\sum_{k=0}^{K-1} \\frac{\\eta_k}{2}(1-\\eta_k L)\\EE\\big[\\|\\vg^k\\|^2\\big],\n\\end{align*}\nwhere we have used $\\EE\\big[\\|\\ve^{0}\\|^2\\big]\\le \\frac{\\sigma^2}{m_0}$ by Assumption~\\ref{assump-sgd}. Since $\\Phi(\\vx^{K}) \\ge \\Phi^*$ from Assumption~\\ref{assump-obj}, the above inequality implies \n\\begin{equation}\\label{eq:main1}\n\t\\sum_{k=0}^{K-1} \\frac{\\eta_k}{2}(1-\\eta_k L)\\EE\\big[\\|\\vg^k\\|^2\\big] \n\t\\le \\Phi(\\vx^0)-\\Phi^* +\\frac{\\sigma^2}{m_0}+ \\sum_{k=1}^{K-1}\\frac{\\eta_k}{2} \\EE\\big[\\|\\ve^k\\|^2\\big].\n\\end{equation}\t\n","sha256_utf8":"742d9bf8f89340ebcf605d718834123120e120a95a081f0193e22dd285209269"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"按动量乘积展开误差","source_units":[{"start":60245,"end":61494,"start_line":774,"end_line":781,"raw_tex":"In addition, we divide both sides of \\eqref{eq:step32} by $\\Gamma_{k+1}$ and obtain from the definition of $\\Gamma_{k+1}$ in \\eqref{eq:gamma} that\n\t$$\\frac{1}{\\Gamma_{k+1} }\\EE\\big[\\|\\ve^{k+1}\\|^2\\big] \\le \\frac{1}{\\Gamma_{k} }\\EE\\big[\\|\\ve^{k}\\|^2\\big] +\\frac{1}{\\Gamma_{k+1} }\\frac{2\\beta_k^2\\sigma^2}{m} + \\frac{1}{\\Gamma_{k} }\\frac{2\\eta_{k}^2 L^2}{m} \\EE\\big[\\|\\vg^{k}\\|^2\\big],\\, \\forall\\, k\\ge 0.$$\nLet $j=0,\\ldots,k-1$ be another index on which the above inequality is telescoped. We obtain\n\t$$\\frac{1}{\\Gamma_{k} }\\EE\\big[\\|\\ve^{k}\\|^2\\big] \\le \\EE\\big[\\|\\ve^{0}\\|^2\\big] +\\sum_{j=0}^{k-1}\\frac{1}{\\Gamma_{j+1} }\\frac{2\\beta_j^2\\sigma^2}{m} + \\sum_{j=0}^{k-1}\\frac{1}{\\Gamma_{j} }\\frac{2\\eta_{j}^2 L^2}{m} \\EE\\big[\\|\\vg^{j}\\|^2\\big],\\, \\forall\\, k\\ge 1.$$\nMultiplying $\\Gamma_{k}$ to both sides of the above inequality and rearranging it gives\n%which further expands and rearranges into \n$$ \t\\EE\\big[\\|\\ve^{k}\\|^2\\big] \\le \\Gamma_{k}\\bigg(\\frac{\\sigma^2}{m_0} +\\frac{2\\sigma^2}{m}\\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \\bigg)+ \\frac{2 L^2}{m}\\sum_{j=0}^{k-1}\\frac{\\Gamma_{k}}{\\Gamma_{j} } \\eta_{j}^2  \\EE\\big[\\|\\vg^{j}\\|^2\\big],\\, \\forall\\, k\\ge 1,$$\nwhere we have used $\\EE\\big[\\|\\ve^{0}\\|^2\\big]\\le \\frac{\\sigma^2}{m_0}$ again. \n","sha256_utf8":"e1a7147b33b5ead73c76d8822889713ad6a7513700b194506c7c712f99d784b5"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"加权累加并交换求和","source_units":[{"start":61494,"end":62683,"start_line":782,"end_line":789,"raw_tex":" Now multiply $\\eta_k$ to the above inequality and sum it up over $k=1,\\ldots,K-1$ to have\n\\begin{align}\\label{eq:main2}\n \t\\sum_{k=1}^{K-1}\\eta_k\\EE\\big[\\|\\ve^{k}\\|^2\\big] \n \t\\le&~ \\sigma^2\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\bigg(\\frac{1}{m_0} +\\frac{2}{m} \\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \\bigg)+\\frac{2 L^2}{m}\\sum_{k=1}^{K-1} \\sum_{j=0}^{k-1}\\frac{\\eta_k\\Gamma_{k}}{\\Gamma_{j} } \\eta_{j}^2  \\EE\\big[\\|\\vg^{j}\\|^2\\big] \\cr\n \t=&~ \\sigma^2\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\bigg(\\frac{1}{m_0} +\\frac{2}{m} \\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \\bigg)+\\frac{2 L^2}{m}\\sum_{j=0}^{K-2}\\frac{\\eta_{j}^2}{\\Gamma_{j} }\\bigg(\\sum_{k=j+1}^{K-1}\\eta_k\\Gamma_{k}\\bigg)    \\EE\\big[\\|\\vg^{j}\\|^2\\big] \\cr\n \t=&~ \\sigma^2\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\bigg(\\frac{1}{m_0} +\\frac{2}{m} \\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \\bigg)+\\frac{2 L^2}{m}\\sum_{k=0}^{K-1}\\frac{\\eta_{k}^2}{\\Gamma_{k} }\\bigg(\\sum_{j=k+1}^{K-1}\\eta_j\\Gamma_{j}\\bigg)    \\EE\\big[\\|\\vg^{k}\\|^2\\big],\n\\end{align}\nwhere the first equality follows by swapping summation, and the second equality is obtained by swapping indices and realizing that the coefficient for $\\EE\\big[\\|\\vg^{K-1}\\|^2\\big]$ is null.\n","sha256_utf8":"a6a8bd506acc8584256eac48e63f6690d2be78d711f8dd304f7c38b6b8ce3e16"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"吸收误差获得映射界","source_units":[{"start":62684,"end":63447,"start_line":791,"end_line":800,"raw_tex":"Now we have by substituting \\eqref{eq:main2} into \\eqref{eq:main1} and rearranging terms that\n$$\n\t\\sum_{k=0}^{K-1} \\frac{\\eta_k}{2}\\bigg(1-\\eta_k L- \\frac{2 L^2}{m} \\frac{\\eta_{k}}{\\Gamma_{k} } \\sum_{j=k+1}^{K-1}\\eta_j\\Gamma_{j}\\bigg)\\EE\\big[\\|\\vg^k\\|^2\\big] \n\t\\le \\Phi(\\vx^0)-\\Phi^* +\\frac{\\sigma^2}{m_0}+ \\frac{\\sigma^2}{2}\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\bigg(\\frac{1}{m_0} +\\frac{2}{m} \\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \\bigg),\n$$\nwhich together with the conditions in \\eqref{eq:cond-eta-beta1} and \\eqref{eq:cond-eta-beta2} gives the bound for $\\vg^k$:\n\\begin{equation}\\label{eq:main3}\n\t\\sum_{k=0}^{K-1}  \\eta_k \\EE\\big[\\|\\vg^k\\|^2\\big] \n\t\\le 4\\big[\\Phi(\\vx^0)-\\Phi^*\\big]  + 2(A+2)\\frac{\\sigma^2}{ m_0}+ 4B\\frac{\\sigma^2}{m} .\n\\end{equation}\t\n","sha256_utf8":"68a0eeaf85b0ac244be9641c4466d1da5b7d3c0f4182238c44048567dc2d37ea"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"控制加权误差总和","source_units":[{"start":63447,"end":63872,"start_line":801,"end_line":806,"raw_tex":"Use \\eqref{eq:cond-eta-beta1} again and substitute \\eqref{eq:main3} into \\eqref{eq:main2}. We obtain the bound for $\\ve^k$:\n\\begin{equation}\\label{eq:main4}\n\t\\sum_{k=0}^{K-1} \\eta_k \\EE\\big[\\|\\ve^k\\|^2\\big] \n\t\\le A\\frac{\\sigma^2}{ m_0}+ 2B\\frac{\\sigma^2}{m} +\\sum_{k=0}^{K-1}\\frac{\\eta_{k}}{2} \\EE\\big[\\|\\vg^{k}\\|^2\\big]\n\t\\le 2\\big[\\Phi(\\vx^0)-\\Phi^*\\big] +2(A+1)\\frac{\\sigma^2}{ m_0}+ 4B\\frac{\\sigma^2}{m} .\n\\end{equation}\t\n","sha256_utf8":"882c180104140092c0fa5685201e861656f1fa578df3eda07df6124591617626"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"合成真实驻点残差界","source_units":[{"start":63873,"end":64259,"start_line":808,"end_line":811,"raw_tex":"Finally, we have from \\eqref{eq:tri-g} that  %from \\eqref{eq:nonexp-P} and the definitions of $\\vg^k$ and $\\bar\\vg^k$ in \\eqref{eq:vg-vgbar}, it follows \n$\\|\\bar\\vg^{k}\\|^2 \\le 2\\|\\vg^{k}\\|^2  + 2\\|\\ve^{k}\\|^2.$ Sum up this inequality over $k=0,\\ldots,K-1$ and\nsubstitute \\eqref{eq:main3} and  \\eqref{eq:main4} into the summation. We obtain the result in \\eqref{eq:mainthm}.\n\\end{proof}","sha256_utf8":"ec541461f8a732f7d01282d6d15c4a1e0c6dfdf9442d4af07c8ad7eb91cff77c"}]},{"id":"S039","semantic_type":"lemma","kind":"lemma","title":"动量乘积尾和估计","source_units":[{"start":64647,"end":65177,"start_line":815,"end_line":829,"raw_tex":"\\begin{lemma}\\label{lem:keylem}\n\tLet \n\t\\begin{equation}\\label{eq:beta-def}\n\t\t\\beta_{k}=3\\big[(k+3)^{1/3}-(k+2)^{1/3}\\big],\\quad k\\ge 0,\n\t\\end{equation}\nThen we  have %the following bounds,\n\\begin{equation}\\label{eq:keylem1}\n\t\\sum_{j=k+1}^{K-1}\\frac{\\Gamma_{j}}{\\Gamma_{k}} \\le \\frac{1}{2}(k+2)^{2/3}+\\frac{1}{6}(k+2)^{1/3}+\\frac{1}{36}. \n\\end{equation}\n%and \n%\\begin{equation}\\label{eq:keylem2}\n%\t\\sum_{j=k+1}^{K-1}(j+2)^{-1/3}\\frac{\\Gamma_{j}}{\\Gamma_{k}} \n%\t\\le \\frac{1}{2}  (k+2)^{1/3}+\\frac{1}{12}.\n%\\end{equation}\n\\end{lemma}","sha256_utf8":"175e50d671f3319bfbb2c0150db137c46b310756521e967ca5b1e352fb6805c5"}]},{"id":"S040","semantic_type":"assumption","kind":"assumption","title":"立方根差动量参数","source_units":[{"start":64679,"end":64799,"start_line":816,"end_line":819,"raw_tex":"\tLet \n\t\\begin{equation}\\label{eq:beta-def}\n\t\t\\beta_{k}=3\\big[(k+3)^{1/3}-(k+2)^{1/3}\\big],\\quad k\\ge 0,\n\t\\end{equation}\n","sha256_utf8":"41db6fdda6198fb7cdf05bdc29c6c66a39f7c3bc268b28249cebf0b99fe05f24"}]},{"id":"S041","semantic_type":"proof","kind":"proof","title":"动量尾和完整证明","source_units":[{"start":65178,"end":66999,"start_line":830,"end_line":860,"raw_tex":"\\begin{proof}\t\n\tBy the fact $a^3-b^3 = (a-b)(a^2+ab+b^2)$, we have %We begin by realizing that \n\t\\begin{equation}\\label{eq:beta-bound}\n\t\t\\beta_{k}=3\\big[(k+3)^{1/3}-(k+2)^{1/3}\\big]=\n\t\t\\frac{3}{(k+3)^{2/3}+(k+3)^{1/3}(k+2)^{1/3}+(k+2)^{2/3}}.\n\t\\end{equation}\n\tHence, $\\beta_{k}\\in \\big[(k+3)^{-2/3},(k+2)^{-2/3}\\big]$ for all $k\\ge 0$, and it is a decreasing sequence. In addition,\n\t%We then establish the trick for bounding $\\Gamma_{j}/\\Gamma_{k}$: for all $j>k\\ge 0,$ \nby the definition of $\\Gamma_k$ and $\\beta_k$, it holds for all $j>k\\ge 0$ that\n\\begin{equation}\\label{eq:keystring}\n\t\\frac{\\Gamma_{j}}{\\Gamma_{k}} \n\t= \\frac{\\prod_{l=0}^{j-1}(1-\\beta_l)^2}{\\prod_{l=0}^{k-1}(1-\\beta_l)^2}  = \\prod_{l=k}^{j-1}(1-\\beta_l)^2\n\t\\le  e^{-2\\sum_{l=k}^{j-1}\\beta_l}\n\t=  e^{-6\\big[(j+2)^{1/3}-(k+2)^{1/3}\\big]},\n\\end{equation}\t\nwhere the inequality holds because $0\\le 1+x\\le e^x, \\forall\\, x\\ge -1$. Therefore we have that for any $k\\ge 0,$\n\\begin{equation}\\label{eq:key-beta-ineq1}\n\\sum_{j=k+1}^{K-1}\\frac{\\Gamma_{j}}{\\Gamma_{k}} \n%=\t\\sum_{j=k+1}^{K-1} \\prod_{l=k}^{j-1}(1-\\beta_l)^2\n\\le %\\sum_{j=k+1}^{K-1} e^{-2\\sum_{l=k}^{j-1}\\beta_l}\n \\sum_{j=k+1}^{K-1} e^{-6\\big[(j+2)^{1/3}-(k+2)^{1/3}\\big]}\n= e^{6(k+2)^{1/3}}\\sum_{j=k+1}^{K-1} e^{-6 (j+2)^{1/3} }.\n\\end{equation}\n\tSince $e^{-6x^{1/3}}$ is a decreasing function and has an anti-derivative $-\\frac{1}{36} e^{-6x^{1/3}} (18x^{2/3}+6x^{1/3}+1)$, we have %$\\int e^{-6x^{1/3}}dx=-\\frac{1}{36} e^{-6x^{1/3}} (18x^{2/3}+6x^{1/3}+1)+C,$ \n\t\\begin{equation}\\label{eq:key-beta-ineq2}\n\\sum_{j=k+1}^{K-1} e^{-6 (j+2)^{1/3} }\n\\le \\int_{k+2}^{K+1} e^{-6x^{1/3}}dx\n\\le \\frac{1}{36}  e^{-6(k+2)^{1/3}} (18(k+2)^{2/3}+6(k+2)^{1/3}+1) .\n\t\\end{equation}\n\tSubstituting \\eqref{eq:key-beta-ineq2} into \\eqref{eq:key-beta-ineq1} gives \\eqref{eq:keylem1} and completes the proof.\n\\end{proof}","sha256_utf8":"5b7191ae5c5ef02a908d71a80970d48682ca2bb5782675c4df062e0909e6f1ba"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"动量参数界与单调性","source_units":[{"start":65178,"end":65560,"start_line":830,"end_line":836,"raw_tex":"\\begin{proof}\t\n\tBy the fact $a^3-b^3 = (a-b)(a^2+ab+b^2)$, we have %We begin by realizing that \n\t\\begin{equation}\\label{eq:beta-bound}\n\t\t\\beta_{k}=3\\big[(k+3)^{1/3}-(k+2)^{1/3}\\big]=\n\t\t\\frac{3}{(k+3)^{2/3}+(k+3)^{1/3}(k+2)^{1/3}+(k+2)^{2/3}}.\n\t\\end{equation}\n\tHence, $\\beta_{k}\\in \\big[(k+3)^{-2/3},(k+2)^{-2/3}\\big]$ for all $k\\ge 0$, and it is a decreasing sequence. In addition,\n","sha256_utf8":"1850bb6460a5019008771fc3b4a1a7d9dc46e945d44616757d19d69f3c3cf431"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"乘积比的指数衰减","source_units":[{"start":65649,"end":66116,"start_line":838,"end_line":845,"raw_tex":"by the definition of $\\Gamma_k$ and $\\beta_k$, it holds for all $j>k\\ge 0$ that\n\\begin{equation}\\label{eq:keystring}\n\t\\frac{\\Gamma_{j}}{\\Gamma_{k}} \n\t= \\frac{\\prod_{l=0}^{j-1}(1-\\beta_l)^2}{\\prod_{l=0}^{k-1}(1-\\beta_l)^2}  = \\prod_{l=k}^{j-1}(1-\\beta_l)^2\n\t\\le  e^{-2\\sum_{l=k}^{j-1}\\beta_l}\n\t=  e^{-6\\big[(j+2)^{1/3}-(k+2)^{1/3}\\big]},\n\\end{equation}\t\nwhere the inequality holds because $0\\le 1+x\\le e^x, \\forall\\, x\\ge -1$. Therefore we have that for any $k\\ge 0,$\n","sha256_utf8":"cebad4e71bbc85344d0ea4401545387e4b4e364d34f2a223f291d001ffe5c63d"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"尾和转为指数级数","source_units":[{"start":66116,"end":66447,"start_line":846,"end_line":852,"raw_tex":"\\begin{equation}\\label{eq:key-beta-ineq1}\n\\sum_{j=k+1}^{K-1}\\frac{\\Gamma_{j}}{\\Gamma_{k}} \n%=\t\\sum_{j=k+1}^{K-1} \\prod_{l=k}^{j-1}(1-\\beta_l)^2\n\\le %\\sum_{j=k+1}^{K-1} e^{-2\\sum_{l=k}^{j-1}\\beta_l}\n \\sum_{j=k+1}^{K-1} e^{-6\\big[(j+2)^{1/3}-(k+2)^{1/3}\\big]}\n= e^{6(k+2)^{1/3}}\\sum_{j=k+1}^{K-1} e^{-6 (j+2)^{1/3} }.\n\\end{equation}\n","sha256_utf8":"54dd79bea4464a2e2afd573f4c77137fe0c378d019503f16a29578189c3362b1"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"积分估计指数级数","source_units":[{"start":66447,"end":66867,"start_line":853,"end_line":858,"raw_tex":"\tSince $e^{-6x^{1/3}}$ is a decreasing function and has an anti-derivative $-\\frac{1}{36} e^{-6x^{1/3}} (18x^{2/3}+6x^{1/3}+1)$, we have %$\\int e^{-6x^{1/3}}dx=-\\frac{1}{36} e^{-6x^{1/3}} (18x^{2/3}+6x^{1/3}+1)+C,$ \n\t\\begin{equation}\\label{eq:key-beta-ineq2}\n\\sum_{j=k+1}^{K-1} e^{-6 (j+2)^{1/3} }\n\\le \\int_{k+2}^{K+1} e^{-6x^{1/3}}dx\n\\le \\frac{1}{36}  e^{-6(k+2)^{1/3}} (18(k+2)^{2/3}+6(k+2)^{1/3}+1) .\n\t\\end{equation}\n","sha256_utf8":"9d41750c6c8c47bd8c882bacd4a440f346e5da2fa039708b0cf0278d170d9d92"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"代入得到尾和上界","source_units":[{"start":66867,"end":66999,"start_line":859,"end_line":860,"raw_tex":"\tSubstituting \\eqref{eq:key-beta-ineq2} into \\eqref{eq:key-beta-ineq1} gives \\eqref{eq:keylem1} and completes the proof.\n\\end{proof}","sha256_utf8":"13ac2eb640235620fbb69b4ae959dd843413bdb8607e9969418d06d5dee251c2"}]},{"id":"S047","semantic_type":"theorem","kind":"theorem","title":"常数步长小批量收敛率","source_units":[{"start":67086,"end":67900,"start_line":863,"end_line":871,"raw_tex":"\\begin{theorem}[convergence rate II with constant stepsizes]\\label{thm:rate-const2}\n\tUnder Assumptions~\\ref{assump-obj} through \\ref{assump-sgd}, let $\\{\\vx^k\\}$ be the iterate sequence from Algorithm~\\ref{alg:prox-storm} with $ \\eta_k =\\frac{\\eta}{L \\sqrt[3]{K}}$ and $\\{\\beta_k\\}$ set by \\eqref{eq:beta-def}, \n\twhere $\\eta \\le \\frac{1}{4}$ is a positive number. If $\\tau$ is selected from $\\{0,1,\\ldots,K-1\\}$ uniformly at random, then\n\t\\begin{equation}\\label{eq:rate-grad-const2}\n\t\t\\EE[\\|\\bar\\vg^\\tau\\|^2] \\le \\frac{1}{K^{\\frac{2}{3}}} \n\t\t\\left(\t\\frac{12 L}{\\eta}\\big[\\Phi(\\vx^0)-\\Phi^*\\big] + \\Big( 2^{-1/3}+\\frac{1}{6}2^{1/3}+\\frac{1}{36}\\Big) \\frac{8}{\\sqrt[3]{K}} \\frac{\\sigma^2}{ m_0} +  \\frac{12\\sigma^2 L}{ \\eta m_0}\n\t\t+ \\frac{ 32}{(1-2^{-2/3})^2} \\frac{\\sigma^2}{m}\\right).\n\t\\end{equation}\n\\end{theorem}","sha256_utf8":"96951cdf0a01d788ade908f623e93e084743a510bdbba6ae83c39cd12b3e84a4"}]},{"id":"S048","semantic_type":"assumption","kind":"assumption","title":"目标定理的参数与输出","source_units":[{"start":67170,"end":67524,"start_line":864,"end_line":865,"raw_tex":"\tUnder Assumptions~\\ref{assump-obj} through \\ref{assump-sgd}, let $\\{\\vx^k\\}$ be the iterate sequence from Algorithm~\\ref{alg:prox-storm} with $ \\eta_k =\\frac{\\eta}{L \\sqrt[3]{K}}$ and $\\{\\beta_k\\}$ set by \\eqref{eq:beta-def}, \n\twhere $\\eta \\le \\frac{1}{4}$ is a positive number. If $\\tau$ is selected from $\\{0,1,\\ldots,K-1\\}$ uniformly at random, then\n","sha256_utf8":"7a22eaeedd4f848d0b6d173571b042a52fb56252cf49282fea29dcedd0cb5790"}]},{"id":"S049","semantic_type":"proof","kind":"proof","title":"常数步长收敛完整证明","source_units":[{"start":67901,"end":70520,"start_line":872,"end_line":899,"raw_tex":"\\begin{proof}\nWe show the desired result by verifying the conditions in Theorem~\\ref{thm:main}. First, with $ \\eta_k =\\frac{\\eta}{L \\sqrt[3]{K}}$, the condition in\n \\eqref{eq:cond-eta-beta1} becomes %is satisfied if and only if\n$$\\frac{2 \\eta }{\\sqrt[3]{K}} + \\frac{4 }{m} \\frac{\\eta^2}{K^{2/3}}  \\sum_{j=k+1}^{K-1}\\frac{\\Gamma_{j}}{\\Gamma_{k} }  \\le 1, \\quad k=0,\\ldots,K-1.$$\nNotice that when $k=K-1$ the summation above is null. Hence by \\eqref{eq:keylem1}, it suffices to require \n\t$$\\frac{2\\eta }{\\sqrt[3]{K}} + \\frac{4 }{m} \\frac{\\eta^2 }{K^{2/3}}  \\bigg(\\frac{1}{2}K^{2/3}+\\frac{1}{6}K^{1/3}+\\frac{1}{36}\\bigg)  \\le 1, $$ which is guaranteed when $\\eta\\le\\frac{1}{4}$ and $K\\ge1$. Therefore, the condition in\n \\eqref{eq:cond-eta-beta1} holds.\n\t\nSecondly, by letting $k=0$ in \\eqref{eq:keylem1} and recalling $\\Gamma_0=1$, we have \n\t$\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\le \\big(\\frac{1}{2}2^{2/3}+\\frac{1}{6}2^{1/3}+\\frac{1}{36}\\big) \\frac{ \\eta}{L\\sqrt[3]{K}}.$\n Hence, the first condition in \\eqref{eq:cond-eta-beta2} holds with\n$A=\\big(2^{-1/3}+\\frac{1}{6}2^{1/3}+\\frac{1}{36}\\big) \\frac{ \\eta}{L\\sqrt[3]{K}}.$\t \nFinally, notice %utilize the below bound and obtain $B$ as in \\eqref{eq:cond-eta-beta2},\n\t\\begin{align}\\label{eq:last1}\n\t\t \\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \n\t\t=&~\\sum_{j=0}^{K-2}\\frac{\\beta_j^2}{(1-\\beta_j)^2 }  \\sum_{k=j+1}^{K-1}\\eta_k\\frac{\\Gamma_{k}}{\\Gamma_{j}} \\cr\n\t\t\\le&~ \\frac{ \\eta}{L\\sqrt[3]{K}} \\sum_{j=0}^{K-2}\\frac{\\beta_j^2}{(1-\\beta_0)^2 }  \n\t\t\\bigg(\\frac{1}{2}(j+2)^{2/3}+\\frac{1}{6}(j+2)^{1/3}+\\frac{1}{36}\\bigg) \\cr\n\t\t\\le&~ \\frac{ \\eta}{(1-\\beta_0)^2 L \\sqrt[3]{K}} \\sum_{j=0}^{K-2} \n\t\t\\bigg(\\frac{1}{2}(j+2)^{-2/3}+\\frac{1}{6}(j+2)^{-1}+\\frac{1}{36} (j+2)^{-4/3}\\bigg) \\cr\n\t\t\\le&~\\frac{ \\eta}{(1-\\beta_0)^2 L \\sqrt[3]{K}}   \n\t\t\\bigg(\\frac{3}{2}(K^{1/3}-1)+\\frac{1}{6}\\log K+\\frac{1}{12} (1-K^{-1/3})\\bigg) \t\\cr\n\t\t\\le&~ \\frac{ 2\\eta}{(1-2^{-2/3})^2 L}, \n\t\\end{align}\nwhere the first inequality follows from \\eqref{eq:keylem1}, the decreasing monotonicity of $\\beta_k$, and the setting of $\\eta_k$, the second inequality holds by $\\beta_j \\le (j+2)^{-2/3}$, and the last inequality is obtained by $\\beta_0\\le 2^{-2/3}$ and using the fact $3x^{1/3} > \\log x, \\forall\\, x>0$.\t\n%for sufficiently large $K$ so that $K^{1/3}\\ge \\ln K.$ \nThus the second condition in \\eqref{eq:cond-eta-beta2} holds with $B = \\frac{ 2\\eta}{(1-2^{-2/3})^2 L}$. Therefore, \\eqref{eq:rate-grad-const2} follows from \\eqref{eq:mainthm} and the choice of $\\tau$ by uniformly random selection. %Apply Theorem~\\ref{thm:rate-const2}, the proof is complete. \n\\end{proof}","sha256_utf8":"79c3fc45c0b7d9d9908f4fffde0b8ac1ffd93a6c2c7536a2b19f28d2e191ed90"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"验证下降吸收条件","source_units":[{"start":67901,"end":68651,"start_line":872,"end_line":878,"raw_tex":"\\begin{proof}\nWe show the desired result by verifying the conditions in Theorem~\\ref{thm:main}. First, with $ \\eta_k =\\frac{\\eta}{L \\sqrt[3]{K}}$, the condition in\n \\eqref{eq:cond-eta-beta1} becomes %is satisfied if and only if\n$$\\frac{2 \\eta }{\\sqrt[3]{K}} + \\frac{4 }{m} \\frac{\\eta^2}{K^{2/3}}  \\sum_{j=k+1}^{K-1}\\frac{\\Gamma_{j}}{\\Gamma_{k} }  \\le 1, \\quad k=0,\\ldots,K-1.$$\nNotice that when $k=K-1$ the summation above is null. Hence by \\eqref{eq:keylem1}, it suffices to require \n\t$$\\frac{2\\eta }{\\sqrt[3]{K}} + \\frac{4 }{m} \\frac{\\eta^2 }{K^{2/3}}  \\bigg(\\frac{1}{2}K^{2/3}+\\frac{1}{6}K^{1/3}+\\frac{1}{36}\\bigg)  \\le 1, $$ which is guaranteed when $\\eta\\le\\frac{1}{4}$ and $K\\ge1$. Therefore, the condition in\n \\eqref{eq:cond-eta-beta1} holds.\n","sha256_utf8":"a4f96c0ab80691f9df63a3f861f9ad7d3047820c579ba1604426786473cb50ea"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"确定初始误差系数","source_units":[{"start":68653,"end":69020,"start_line":880,"end_line":883,"raw_tex":"Secondly, by letting $k=0$ in \\eqref{eq:keylem1} and recalling $\\Gamma_0=1$, we have \n\t$\\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\le \\big(\\frac{1}{2}2^{2/3}+\\frac{1}{6}2^{1/3}+\\frac{1}{36}\\big) \\frac{ \\eta}{L\\sqrt[3]{K}}.$\n Hence, the first condition in \\eqref{eq:cond-eta-beta2} holds with\n$A=\\big(2^{-1/3}+\\frac{1}{6}2^{1/3}+\\frac{1}{36}\\big) \\frac{ \\eta}{L\\sqrt[3]{K}}.$\t \n","sha256_utf8":"341aee00037deabe2e5fe846c6a67bcf5d66dc4c2fc5ab278c1348e2f0b85b64"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"控制累积噪声系数","source_units":[{"start":69020,"end":70158,"start_line":884,"end_line":896,"raw_tex":"Finally, notice %utilize the below bound and obtain $B$ as in \\eqref{eq:cond-eta-beta2},\n\t\\begin{align}\\label{eq:last1}\n\t\t \\sum_{k=1}^{K-1}\\eta_k\\Gamma_{k}\\sum_{j=0}^{k-1}\\frac{\\beta_j^2}{\\Gamma_{j+1} } \n\t\t=&~\\sum_{j=0}^{K-2}\\frac{\\beta_j^2}{(1-\\beta_j)^2 }  \\sum_{k=j+1}^{K-1}\\eta_k\\frac{\\Gamma_{k}}{\\Gamma_{j}} \\cr\n\t\t\\le&~ \\frac{ \\eta}{L\\sqrt[3]{K}} \\sum_{j=0}^{K-2}\\frac{\\beta_j^2}{(1-\\beta_0)^2 }  \n\t\t\\bigg(\\frac{1}{2}(j+2)^{2/3}+\\frac{1}{6}(j+2)^{1/3}+\\frac{1}{36}\\bigg) \\cr\n\t\t\\le&~ \\frac{ \\eta}{(1-\\beta_0)^2 L \\sqrt[3]{K}} \\sum_{j=0}^{K-2} \n\t\t\\bigg(\\frac{1}{2}(j+2)^{-2/3}+\\frac{1}{6}(j+2)^{-1}+\\frac{1}{36} (j+2)^{-4/3}\\bigg) \\cr\n\t\t\\le&~\\frac{ \\eta}{(1-\\beta_0)^2 L \\sqrt[3]{K}}   \n\t\t\\bigg(\\frac{3}{2}(K^{1/3}-1)+\\frac{1}{6}\\log K+\\frac{1}{12} (1-K^{-1/3})\\bigg) \t\\cr\n\t\t\\le&~ \\frac{ 2\\eta}{(1-2^{-2/3})^2 L}, \n\t\\end{align}\nwhere the first inequality follows from \\eqref{eq:keylem1}, the decreasing monotonicity of $\\beta_k$, and the setting of $\\eta_k$, the second inequality holds by $\\beta_j \\le (j+2)^{-2/3}$, and the last inequality is obtained by $\\beta_0\\le 2^{-2/3}$ and using the fact $3x^{1/3} > \\log x, \\forall\\, x>0$.\t\n","sha256_utf8":"4435afa076db66cfd60c996017978faf75ba6a575cc680aa4e3dacb40aca32a1"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"应用通用界与均匀输出","source_units":[{"start":70215,"end":70520,"start_line":898,"end_line":899,"raw_tex":"Thus the second condition in \\eqref{eq:cond-eta-beta2} holds with $B = \\frac{ 2\\eta}{(1-2^{-2/3})^2 L}$. Therefore, \\eqref{eq:rate-grad-const2} follows from \\eqref{eq:mainthm} and the choice of $\\tau$ by uniformly random selection. %Apply Theorem~\\ref{thm:rate-const2}, the proof is complete. \n\\end{proof}","sha256_utf8":"65d67055c1f66d8583f68e547061345564bda0de5c7e0120161c6bc51f16947a"}]},{"id":"S054","semantic_type":"external_fact","kind":"external_fact","title":"标准近端下降性质","source_units":[],"declared_external":true,"declaration_text":["For the proximal gradient mapping defined using a differentiable 1-strongly convex Bregman generator and closed convex r, ⟨d,P(x,d,η)⟩ ≥ ‖P(x,d,η)‖² + (r(x⁺)−r(x))/η; cited as ghadimi2016mini, Lemma 1."]},{"id":"S055","semantic_type":"external_fact","kind":"external_fact","title":"标准近端非扩张性质","source_units":[],"declared_external":true,"declaration_text":["For fixed x and η>0, ‖P(x,d₁,η)−P(x,d₂,η)‖ ≤ ‖d₁−d₂‖; cited as ghadimi2016mini, Proposition 1."]},{"id":"S056","semantic_type":"external_fact","kind":"external_fact","title":"均方光滑蕴含期望光滑","source_units":[],"declared_external":true,"declaration_text":["Unbiasedness and E‖∇f(x;ξ)−∇f(y;ξ)‖²≤L²‖x−y‖² imply ‖∇F(x)−∇F(y)‖≤L‖x−y‖ by Jensen's inequality; the source cites the end of section 2.2 of tran2021hybrid."]},{"id":"S057","semantic_type":"external_fact","kind":"external_fact","title":"标准概率与初等分析工具","source_units":[],"declared_external":true,"declaration_text":["An L-smooth function satisfies F(y)≤F(x)+⟨∇F(x),y−x⟩+(L/2)‖y−x‖². Also 2⟨a,b⟩≤‖a‖²+‖b‖² and ‖a+b‖²≤2‖a‖²+2‖b‖². Conditional expectation has the tower property; conditionally independent centered vectors have zero expected cross products, giving minibatch variance reduction by the batch size, and variance is bounded by the second moment. For t≥−1, 1+t≤eᵗ. Decreasing positive sums admit the integral comparisons used in the proofs; a³−b³=(a−b)(a²+ab+b²), and log x<3x^(1/3) for x>0."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S047","edges":[{"source_id":"S001","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S005","kind":"supports"},{"source_id":"S054","target_id":"S006","kind":"supports"},{"source_id":"S055","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S008","kind":"supports"},{"source_id":"S011","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S056","target_id":"S013","kind":"supports"},{"source_id":"S003","target_id":"S017","kind":"supports"},{"source_id":"S008","target_id":"S017","kind":"supports"},{"source_id":"S013","target_id":"S017","kind":"supports"},{"source_id":"S057","target_id":"S017","kind":"supports"},{"source_id":"S006","target_id":"S018","kind":"supports"},{"source_id":"S009","target_id":"S018","kind":"supports"},{"source_id":"S001","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S057","target_id":"S019","kind":"supports"},{"source_id":"S007","target_id":"S020","kind":"supports"},{"source_id":"S008","target_id":"S020","kind":"supports"},{"source_id":"S009","target_id":"S020","kind":"supports"},{"source_id":"S057","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S016","kind":"supports"},{"source_id":"S019","target_id":"S016","kind":"supports"},{"source_id":"S020","target_id":"S016","kind":"supports"},{"source_id":"S021","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S003","target_id":"S024","kind":"supports"},{"source_id":"S004","target_id":"S024","kind":"supports"},{"source_id":"S009","target_id":"S024","kind":"supports"},{"source_id":"S012","target_id":"S024","kind":"supports"},{"source_id":"S003","target_id":"S025","kind":"supports"},{"source_id":"S009","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S057","target_id":"S026","kind":"supports"},{"source_id":"S003","target_id":"S027","kind":"supports"},{"source_id":"S011","target_id":"S027","kind":"supports"},{"source_id":"S012","target_id":"S027","kind":"supports"},{"source_id":"S057","target_id":"S027","kind":"supports"},{"source_id":"S025","target_id":"S028","kind":"supports"},{"source_id":"S026","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S012","target_id":"S028","kind":"supports"},{"source_id":"S008","target_id":"S028","kind":"supports"},{"source_id":"S003","target_id":"S028","kind":"supports"},{"source_id":"S057","target_id":"S028","kind":"supports"},{"source_id":"S020","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S023","kind":"supports"},{"source_id":"S026","target_id":"S023","kind":"supports"},{"source_id":"S027","target_id":"S023","kind":"supports"},{"source_id":"S028","target_id":"S023","kind":"supports"},{"source_id":"S029","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S019","target_id":"S033","kind":"supports"},{"source_id":"S016","target_id":"S033","kind":"supports"},{"source_id":"S003","target_id":"S033","kind":"supports"},{"source_id":"S010","target_id":"S033","kind":"supports"},{"source_id":"S012","target_id":"S033","kind":"supports"},{"source_id":"S028","target_id":"S034","kind":"supports"},{"source_id":"S023","target_id":"S034","kind":"supports"},{"source_id":"S014","target_id":"S034","kind":"supports"},{"source_id":"S012","target_id":"S034","kind":"supports"},{"source_id":"S003","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S033","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S031","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S031","target_id":"S037","kind":"supports"},{"source_id":"S020","target_id":"S038","kind":"supports"},{"source_id":"S016","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S032","kind":"supports"},{"source_id":"S035","target_id":"S032","kind":"supports"},{"source_id":"S036","target_id":"S032","kind":"supports"},{"source_id":"S037","target_id":"S032","kind":"supports"},{"source_id":"S038","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S030","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S057","target_id":"S042","kind":"supports"},{"source_id":"S014","target_id":"S043","kind":"supports"},{"source_id":"S040","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S057","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S057","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S043","target_id":"S041","kind":"supports"},{"source_id":"S044","target_id":"S041","kind":"supports"},{"source_id":"S045","target_id":"S041","kind":"supports"},{"source_id":"S046","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S039","kind":"supports"},{"source_id":"S031","target_id":"S050","kind":"supports"},{"source_id":"S039","target_id":"S050","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S003","target_id":"S050","kind":"supports"},{"source_id":"S014","target_id":"S051","kind":"supports"},{"source_id":"S031","target_id":"S051","kind":"supports"},{"source_id":"S039","target_id":"S051","kind":"supports"},{"source_id":"S048","target_id":"S051","kind":"supports"},{"source_id":"S014","target_id":"S052","kind":"supports"},{"source_id":"S031","target_id":"S052","kind":"supports"},{"source_id":"S039","target_id":"S052","kind":"supports"},{"source_id":"S042","target_id":"S052","kind":"supports"},{"source_id":"S048","target_id":"S052","kind":"supports"},{"source_id":"S057","target_id":"S052","kind":"supports"},{"source_id":"S030","target_id":"S053","kind":"supports"},{"source_id":"S050","target_id":"S053","kind":"supports"},{"source_id":"S051","target_id":"S053","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S048","target_id":"S053","kind":"supports"},{"source_id":"S008","target_id":"S053","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S051","target_id":"S049","kind":"supports"},{"source_id":"S052","target_id":"S049","kind":"supports"},{"source_id":"S053","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S047","kind":"supports"}]}}
{"record_id":"2006.08667v3","paper":{"title":"The Landscape of the Proximal Point Method for Nonconvex-Nonconcave Minimax Optimization","authors":["Benjamin Grimmer","Haihao Lu","Pratik Worah","Vahab Mirrokni"],"arxiv_id":"2006.08667","version":"v3","journal":"Mathematical Programming","doi":"10.1007/s10107-022-01910-8","abstract_url":"https://arxiv.org/abs/2006.08667v3","pdf_url":"https://arxiv.org/pdf/2006.08667v3"},"context":{"raw_tex":"Minimax optimization has become a central tool for modern machine learning, recently receiving increasing attention in optimization and machine learning communities. We consider the following saddle point optimization problem\n\\begin{equation}\\label{eq:main-problem}\n\\min_{x\\in\\RR^n}\\max_{y\\in\\RR^m} L(x,y) \\ ,\n\\end{equation}\n\nwhere $L(x,y)$ is a differentiable function in $x$ and $y$. \n\n\n\\begin{equation}\\label{eq:prox-definition}\n(x_{k+1},y_{k+1})=\\prox_\\eta(x_k,y_k):= \\mathrm{arg}\\min_{u\\in\\mathbb{R}^n}\\max_{v\\in\\mathbb{R}^m} L(u,v) +  \\frac{\\eta}{2}\\|u-x_k\\|^2 - \\frac{\\eta}{2}\\|v-y_k\\|^2.\n\\end{equation}\n\n\\begin{equation}\\label{eq:saddle-envelope}\nL_{\\eta}(x,y) := \\min_{u\\in\\mathbb{R}^n}\\max_{v\\in\\mathbb{R}^m} L(u,v) +  \\frac{\\eta}{2}\\|u-x\\|^2 - \\frac{\\eta}{2}\\|v-y\\|^2.\n\\end{equation}\n\nWe say a  function $M(x,y)$ is $\\beta$-smooth if its gradient is uniformly $\\beta$-Lipschitz\n$ \\|\\nabla M(z) - \\nabla M(z')\\| \\leq \\beta\\|z-z'\\| $\nor equivalently for twice differentiable functions, if $\\|\\nabla^2 M(z)\\| \\leq \\beta$.\nFurther, we say a twice differentiable $M(x,y)$ is $\\mu$-strongly convex-strongly concave for some $\\mu\\geq 0$ if\n$ \\nabla^2_{xx} M(z) \\succeq \\mu I $ and $-\\nabla^2_{yy}M(z) \\succeq \\mu I.$\nWhen $\\mu=0$, this corresponds to $M$ being convex with respect to $x$ and concave with respect to $y$.\n\n\nThroughout this paper, we are primarily interested in the weakening of this convexity condition to allow negative curvature given by $\\rho$-weak convexity and $\\rho$-weak concavity: we assume that $L$ is twice differentiable, and for any $z=(x,y)\\in\\RR^n\\times\\RR^m$ that\n\\begin{equation} \\label{eq:weak}\n\\nabla^2_{xx} L(z)\\succeq -\\rho I\\ ,\\ \\ \\  -\\nabla^2_{yy} L(z)\\succeq -\\rho I \\ .\n\\end{equation}\n\nNotice that the objective $L(x,y)$ is convex-concave when $\\rho=0$, and strongly convex-strongly concave when $\\rho<0$. Here our primary interest is in the regime where $\\rho>0$ is positive, quantifying how nonconvex-nonconcave $L$ is.\n\n\nBesides PPM, Gradient Descent Ascent (GDA) is another classic algorithm for minimax problems~\\eqref{eq:main-problem} given by\n\\begin{equation}\\label{eq:GDA}\n\\begin{bmatrix} x_{k+1} \\\\ y_{k+1} \\end{bmatrix} = \\begin{bmatrix} x_{k} \\\\ y_{k} \\end{bmatrix} - s\\begin{bmatrix} \\nabla_x L(x_k,y_k) \\\\ -\\nabla_y L(x_k,y_k) \\end{bmatrix} \\ ,\n\\end{equation}\n\nIn this paper, we study a more generalized algorithm, damped PPM, with damping parameter $\\lambda\\in(0,1]$ and proximal parameter $\\eta>0$ given by\n\\begin{equation}\\label{eq:saddle-PPM}\n\\begin{bmatrix} x_{k+1} \\\\ y_{k+1} \\end{bmatrix} = (1-\\lambda)\\begin{bmatrix} x_{k} \\\\ y_{k} \\end{bmatrix} +\\lambda\\ \\prox_\\eta(x_k,y_k) \\ .\n\\end{equation}\n\nIn particular, when $\\lambda=1$, we recover the traditional PPM \\eqref{eq:prox-definition}.\n\n\nWe denote the Moreau envelope of a function $f$ with parameter $\\eta>0$ by\n\\begin{equation}\ne_{\\eta}\\left\\{f\\right\\}(x) = \\min_{u}\\ f(u) + \\frac{\\eta}{2}\\|u - x\\|^2 \\ . \\label{eq:moreau-def}\n\\end{equation}\n\nFor $\\rho$-weakly convex functions and $\\eta>\\rho$, there is a nice calculus for the Moreau envelope.\nIts gradient at some $x\\in\\RR^n$ is determined by the proximal step $x_+=\\mathrm{argmin}_u\\  f(u) + \\frac{\\eta}{2}\\|u - x\\|^2$ having\n\\begin{equation}\n\\nabla e_{\\eta}\\left\\{f\\right\\}(x) = \\eta(x-x_+) = \\nabla f(x_+) \\ . \\label{eq:moreau-grad}\n\\end{equation}\n\nWe require that the parameter $\\eta$ is selected with $\\eta > \\rho$, which ensures the minimax problem in~\\eqref{eq:saddle-envelope} is strongly convex-strongly concave. As a result, the saddle envelope is well-defined (as its subproblem has a unique minimax point) and often can be efficiently approximated.\n\n\n\\begin{definition}\\label{def:interaction-dominance}\n\tA function $L$ is $\\alpha$-interaction dominant with respect to $x$ if\n\t\\begin{align}\n\t\\nabla^2_{xx} L(z) + \\nabla^2_{xy} L(z)(\\eta I -\\nabla^2_{yy} L(z))^{-1}\\nabla^2_{yx} L(z) &\\succeq \\alpha I \\label{eq:x-dominance}\n\t\\end{align}\t   \n\tand $\\alpha$-interaction dominant with respect to $y$ if\n\t\\begin{align}\n\t-\\nabla^2_{yy} L(z) + \\nabla^2_{yx} L(z)(\\eta I +\\nabla^2_{xx} L(z))^{-1}\\nabla^2_{xy} L(z) &\\succeq \\alpha I\\ . \\label{eq:y-dominance}   \n\t\\end{align}\t  \n\\end{definition}","units":[{"start":3087,"end":3411,"start_line":55,"end_line":58,"raw_tex":"Minimax optimization has become a central tool for modern machine learning, recently receiving increasing attention in optimization and machine learning communities. We consider the following saddle point optimization problem\n\\begin{equation}\\label{eq:main-problem}\n\\min_{x\\in\\RR^n}\\max_{y\\in\\RR^m} L(x,y) \\ ,\n\\end{equation}","sha256_utf8":"bbb5903e4ccddfb4718b7f1ba7997a2da69f258fab14eb254b2f1ec07845a072"},{"start":3412,"end":3473,"start_line":59,"end_line":59,"raw_tex":"where $L(x,y)$ is a differentiable function in $x$ and $y$. \n","sha256_utf8":"7868d384914f8c2c9c931c8fe59878bcac56c9e932e35d94a9c5419da8edc1f3"},{"start":5669,"end":5890,"start_line":76,"end_line":78,"raw_tex":"\\begin{equation}\\label{eq:prox-definition}\n(x_{k+1},y_{k+1})=\\prox_\\eta(x_k,y_k):= \\mathrm{arg}\\min_{u\\in\\mathbb{R}^n}\\max_{v\\in\\mathbb{R}^m} L(u,v) +  \\frac{\\eta}{2}\\|u-x_k\\|^2 - \\frac{\\eta}{2}\\|v-y_k\\|^2.\n\\end{equation}","sha256_utf8":"5a39dbc02e5e40afdb3a8ddb3338f763c796d39638538c3a6792c7f42731b3dc"},{"start":8508,"end":8690,"start_line":103,"end_line":105,"raw_tex":"\\begin{equation}\\label{eq:saddle-envelope}\nL_{\\eta}(x,y) := \\min_{u\\in\\mathbb{R}^n}\\max_{v\\in\\mathbb{R}^m} L(u,v) +  \\frac{\\eta}{2}\\|u-x\\|^2 - \\frac{\\eta}{2}\\|v-y\\|^2.\n\\end{equation}","sha256_utf8":"c20e9efdd6827dbd2fbcd3091db61636b3ea9bcc8cf39576bcdea3893cbab3b9"},{"start":12977,"end":13506,"start_line":124,"end_line":129,"raw_tex":"We say a  function $M(x,y)$ is $\\beta$-smooth if its gradient is uniformly $\\beta$-Lipschitz\n$ \\|\\nabla M(z) - \\nabla M(z')\\| \\leq \\beta\\|z-z'\\| $\nor equivalently for twice differentiable functions, if $\\|\\nabla^2 M(z)\\| \\leq \\beta$.\nFurther, we say a twice differentiable $M(x,y)$ is $\\mu$-strongly convex-strongly concave for some $\\mu\\geq 0$ if\n$ \\nabla^2_{xx} M(z) \\succeq \\mu I $ and $-\\nabla^2_{yy}M(z) \\succeq \\mu I.$\nWhen $\\mu=0$, this corresponds to $M$ being convex with respect to $x$ and concave with respect to $y$.\n","sha256_utf8":"2a3a0b75ba47f39d8be7aef2b49026ca37b7a06d8206654ca54e9e35c9a9b3be"},{"start":13507,"end":13908,"start_line":131,"end_line":134,"raw_tex":"Throughout this paper, we are primarily interested in the weakening of this convexity condition to allow negative curvature given by $\\rho$-weak convexity and $\\rho$-weak concavity: we assume that $L$ is twice differentiable, and for any $z=(x,y)\\in\\RR^n\\times\\RR^m$ that\n\\begin{equation} \\label{eq:weak}\n\\nabla^2_{xx} L(z)\\succeq -\\rho I\\ ,\\ \\ \\  -\\nabla^2_{yy} L(z)\\succeq -\\rho I \\ .\n\\end{equation}","sha256_utf8":"ad27e8566ad2b15676a9f67bd5b9b37ca644178e18b9402942d6836ce98c0b65"},{"start":13909,"end":14145,"start_line":135,"end_line":135,"raw_tex":"Notice that the objective $L(x,y)$ is convex-concave when $\\rho=0$, and strongly convex-strongly concave when $\\rho<0$. Here our primary interest is in the regime where $\\rho>0$ is positive, quantifying how nonconvex-nonconcave $L$ is.\n","sha256_utf8":"60abeddb46dc075a1ee887890878f75b7a51ab11b0be541c6bb68415cd6ac9be"},{"start":14146,"end":14494,"start_line":137,"end_line":140,"raw_tex":"Besides PPM, Gradient Descent Ascent (GDA) is another classic algorithm for minimax problems~\\eqref{eq:main-problem} given by\n\\begin{equation}\\label{eq:GDA}\n\\begin{bmatrix} x_{k+1} \\\\ y_{k+1} \\end{bmatrix} = \\begin{bmatrix} x_{k} \\\\ y_{k} \\end{bmatrix} - s\\begin{bmatrix} \\nabla_x L(x_k,y_k) \\\\ -\\nabla_y L(x_k,y_k) \\end{bmatrix} \\ ,\n\\end{equation}","sha256_utf8":"6f74fef25c6f890cc1c7db5b6a2b9b459c814a57e8a0fb09791e24e046c0c547"},{"start":14711,"end":15052,"start_line":143,"end_line":146,"raw_tex":"In this paper, we study a more generalized algorithm, damped PPM, with damping parameter $\\lambda\\in(0,1]$ and proximal parameter $\\eta>0$ given by\n\\begin{equation}\\label{eq:saddle-PPM}\n\\begin{bmatrix} x_{k+1} \\\\ y_{k+1} \\end{bmatrix} = (1-\\lambda)\\begin{bmatrix} x_{k} \\\\ y_{k} \\end{bmatrix} +\\lambda\\ \\prox_\\eta(x_k,y_k) \\ .\n\\end{equation}","sha256_utf8":"0b1b25b0a6af4fcf82444dd730e7e356552140b2eff167f3c7bf40b4af73040b"},{"start":15053,"end":15145,"start_line":147,"end_line":147,"raw_tex":"In particular, when $\\lambda=1$, we recover the traditional PPM \\eqref{eq:prox-definition}.\n","sha256_utf8":"452006a548c72f44ded05501e811edd7c43e9a21600d7f051999735b17536f54"},{"start":22429,"end":22634,"start_line":189,"end_line":192,"raw_tex":"We denote the Moreau envelope of a function $f$ with parameter $\\eta>0$ by\n\\begin{equation}\ne_{\\eta}\\left\\{f\\right\\}(x) = \\min_{u}\\ f(u) + \\frac{\\eta}{2}\\|u - x\\|^2 \\ . \\label{eq:moreau-def}\n\\end{equation}","sha256_utf8":"6830a6648cb7e7342168642174fde3fd174afdca076a09f772679b65db3b4373"},{"start":22804,"end":23163,"start_line":197,"end_line":201,"raw_tex":"For $\\rho$-weakly convex functions and $\\eta>\\rho$, there is a nice calculus for the Moreau envelope.\nIts gradient at some $x\\in\\RR^n$ is determined by the proximal step $x_+=\\mathrm{argmin}_u\\  f(u) + \\frac{\\eta}{2}\\|u - x\\|^2$ having\n\\begin{equation}\n\\nabla e_{\\eta}\\left\\{f\\right\\}(x) = \\eta(x-x_+) = \\nabla f(x_+) \\ . \\label{eq:moreau-grad}\n\\end{equation}","sha256_utf8":"c368b940a499d9dada5122028c242def08f3d90d627611f667ab08dcf6b47931"},{"start":24524,"end":24833,"start_line":220,"end_line":220,"raw_tex":"We require that the parameter $\\eta$ is selected with $\\eta > \\rho$, which ensures the minimax problem in~\\eqref{eq:saddle-envelope} is strongly convex-strongly concave. As a result, the saddle envelope is well-defined (as its subproblem has a unique minimax point) and often can be efficiently approximated.\n","sha256_utf8":"779b49987ae2db504e8210f35f40e8304676264cd1a6cafb0759198d4428dc36"},{"start":37566,"end":38100,"start_line":322,"end_line":331,"raw_tex":"\\begin{definition}\\label{def:interaction-dominance}\n\tA function $L$ is $\\alpha$-interaction dominant with respect to $x$ if\n\t\\begin{align}\n\t\\nabla^2_{xx} L(z) + \\nabla^2_{xy} L(z)(\\eta I -\\nabla^2_{yy} L(z))^{-1}\\nabla^2_{yx} L(z) &\\succeq \\alpha I \\label{eq:x-dominance}\n\t\\end{align}\t   \n\tand $\\alpha$-interaction dominant with respect to $y$ if\n\t\\begin{align}\n\t-\\nabla^2_{yy} L(z) + \\nabla^2_{yx} L(z)(\\eta I +\\nabla^2_{xx} L(z))^{-1}\\nabla^2_{xy} L(z) &\\succeq \\alpha I\\ . \\label{eq:y-dominance}   \n\t\\end{align}\t  \n\\end{definition}","sha256_utf8":"6195fc7f8590e2741675de81b5c4e93037d791c07fb57f180cf954e72cda27ed"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{thm:two-sided-dominance}\n\tFor any objective $L$ that is $\\rho$-weakly convex-weakly concave and $\\alpha > 0$-interaction dominant in both $x$ and $y$, the damped PPM~\\eqref{eq:saddle-PPM} with $\\eta$ and $\\lambda$ satisfying\n\t$$ \\lambda \\leq 2\\frac{\\min\\left\\{1,(\\eta/\\rho-1)^2\\right\\}}{\\eta/\\alpha+1} $$\n\tlinearly converges to the unique stationary point $(x^*,y^*)$ of~\\eqref{eq:main-problem} with\n\t$$\\left\\|\\begin{bmatrix} x_k-x^* \\\\ y_k-y^* \\end{bmatrix}\\right\\|^2 \\leq \\left(1 - \\frac{2\\lambda}{\\eta/\\alpha+1} + \\frac{\\lambda^2}{\\min\\left\\{1,(\\eta/\\rho-1)^2\\right\\}}\\right)^{k}\\left\\|\\begin{bmatrix} x_0-x^* \\\\ y_0-y^* \\end{bmatrix}\\right\\|^2.$$\n\tFor example, setting $\\eta=2\\rho$ and $\\lambda=\\frac{1}{1+\\eta/\\alpha} $, our convergence rate simplifies to\n\t$$\\left\\|\\begin{bmatrix} x_k-x^* \\\\ y_k-y^* \\end{bmatrix}\\right\\|^2 \\leq \\left(1-\\frac{1}{(2\\rho/\\alpha+1)^2} \\right)^{k}\\left\\|\\begin{bmatrix} x_0-x^* \\\\ y_0-y^* \\end{bmatrix}\\right\\|^2.$$\n\\end{theorem}","units":[{"start":47503,"end":48490,"start_line":409,"end_line":416,"raw_tex":"\\begin{theorem}\\label{thm:two-sided-dominance}\n\tFor any objective $L$ that is $\\rho$-weakly convex-weakly concave and $\\alpha > 0$-interaction dominant in both $x$ and $y$, the damped PPM~\\eqref{eq:saddle-PPM} with $\\eta$ and $\\lambda$ satisfying\n\t$$ \\lambda \\leq 2\\frac{\\min\\left\\{1,(\\eta/\\rho-1)^2\\right\\}}{\\eta/\\alpha+1} $$\n\tlinearly converges to the unique stationary point $(x^*,y^*)$ of~\\eqref{eq:main-problem} with\n\t$$\\left\\|\\begin{bmatrix} x_k-x^* \\\\ y_k-y^* \\end{bmatrix}\\right\\|^2 \\leq \\left(1 - \\frac{2\\lambda}{\\eta/\\alpha+1} + \\frac{\\lambda^2}{\\min\\left\\{1,(\\eta/\\rho-1)^2\\right\\}}\\right)^{k}\\left\\|\\begin{bmatrix} x_0-x^* \\\\ y_0-y^* \\end{bmatrix}\\right\\|^2.$$\n\tFor example, setting $\\eta=2\\rho$ and $\\lambda=\\frac{1}{1+\\eta/\\alpha} $, our convergence rate simplifies to\n\t$$\\left\\|\\begin{bmatrix} x_k-x^* \\\\ y_k-y^* \\end{bmatrix}\\right\\|^2 \\leq \\left(1-\\frac{1}{(2\\rho/\\alpha+1)^2} \\right)^{k}\\left\\|\\begin{bmatrix} x_0-x^* \\\\ y_0-y^* \\end{bmatrix}\\right\\|^2.$$\n\\end{theorem}","sha256_utf8":"af98423c166d20840d37117cfa47a8889466012590417c5b478fe8cbb32be043"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma} \\label{lem:helper0}\n\tSuppose $M(x,y)$ is $\\mu$-strongly convex-strongly concave on a convex set $S=S_x\\times S_y$, then it holds for any $(x,y),(x',y') \\in S$ that\n\t$$\\mu\\left\\|\\begin{bmatrix} x-x' \\\\ y-y' \\end{bmatrix}\\right\\|^2 \\leq \\left(\\begin{bmatrix} \\nabla_x M(x,y)  \\\\-\\nabla_y M(x,y) \\end{bmatrix} -\\begin{bmatrix} \\nabla_x M(x',y')  \\\\-\\nabla_y M(x',y') \\end{bmatrix}\\right)^T\\begin{bmatrix} x-x' \\\\ y-y' \\end{bmatrix}.$$\n\tWhen $\\nabla M(x', y')=0$, the distance to this stationary point is bounded by\n\t$$\n\t\\left\\|\\begin{bmatrix} x-x' \\\\ y-y' \\end{bmatrix}\\right\\| \\leq \\frac{\\|\\nabla M(x,y)\\|}{\\mu} \\ . \n\t$$\n\\end{lemma}\n\n\\begin{theorem} \\label{thm:standard-contraction}\n\tConsider any minimax problem $\\min_{x\\in\\RR^n}\\max_{y\\in\\RR^m} M(x,y)$ where $M(x,y)$ is $\\beta$-smooth and $\\mu$-strongly convex-strongly concave on a set $B(x_0,r)\\times B(y_0,r)$ with $r\\geq 2\\|\\nabla M(x_0,y_0)\\|/\\mu$. Then GDA~\\eqref{eq:GDA} with step-size $s\\in(0, 2\\mu/\\beta^2) $ linearly converges to a stationary point $(x^*,y^*)\\in B((x_0,y_0),r/2)$ with\n\t$$ \\left\\|\\begin{bmatrix} x_k-x^* \\\\ y_k-y^* \\end{bmatrix}\\right\\|^2 \\leq \\left(1 - 2\\mu s+\\beta^2s^2\\right)^{k}\\left\\|\\begin{bmatrix} x_0-x^* \\\\ y_0-y^* \\end{bmatrix}\\right\\|^2.$$\n\\end{theorem}\n\n\\begin{lemma} \\label{lem:grad-formula}\n\tThe gradient of the saddle envelope $L_\\eta(x,y)$ at $z=(x,y)$ is\n\t$$ \\begin{bmatrix} \\nabla_x L_\\eta(z)\\\\  \\nabla_y L_\\eta(z)\\end{bmatrix} = \\begin{bmatrix} \\eta(x-x_+)\\\\ \\eta(y_+-y) \\end{bmatrix} = \\begin{bmatrix} \\nabla_x L(z_+)\\\\  \\nabla_y L(z_+)\\end{bmatrix}$$\n\twhere $z_+ = (x_+,y_+) = \\prox_\\eta(z)$ is given by the proximal operator.\n\\end{lemma}\n\n\\begin{proof}\n\tNotice that the saddle envelope is a composition of Moreau envelopes\n\t\\begin{align*}\n\tL_{\\eta}(x,y) &= \\min_{u} \\left(\\max_{v} L(u,v) - \\frac{\\eta}{2}\\|v-y\\|^2\\right) + \\frac{\\eta}{2}\\|u-x\\|^2\\\\\n\t&= \\min_{u}\\ -e_\\eta\\{-L(u,\\cdot)\\}(y) + \\frac{\\eta}{2}\\|u-x\\|^2 = e_{\\eta}\\{g(\\cdot,y)\\}(x)\n\t\\end{align*}\n\twhere $g(u,y) = -e_{\\eta}\\{-L(u,\\cdot)\\}(y)$. Applying the gradient formula~\\eqref{eq:moreau-grad} gives our first claimed gradient formula in $x$ of\n\t$\\nabla_x L_\\eta(x,y) = \\eta (x-x_+)$\n\tsince $x_+$ is the unique minimizer of $u \\mapsto g(u,y) + \\frac{\\eta}{2}\\|u-x\\|^2$. Symmetric reasoning gives our first claimed formula $\\nabla_y L_\\eta(x,y) = \\eta (y_+-y) $ in $y$.\n\tThe second claimed equality is precisely the first-order optimality condition for~\\eqref{eq:prox-definition}. \n\\end{proof}\n\n\\begin{corollary} \\label{cor:stationarity}\n\tThe stationary points of $L_\\eta$ are exactly the same as those of $L$.\n\\end{corollary}\n\n\\begin{proof}\n\tFirst consider any stationary point $z=(x,y)$ of $L$. Denote $z_+=\\prox_\\eta(z)$ and the objective function defining the proximal operator~\\eqref{eq:prox-definition} as $M(u,v) = L(u,v) + \\frac{\\eta}{2}\\|u-x\\|^2-\\frac{\\eta}{2}\\|v-y\\|^2$. Then observing that $\\nabla M(z)=0$, $z$ must be the unique minimax point of $M$ (that is, $z=z_+=\\prox_\\eta(z)$). Hence $z$ must be a stationary point of $L_\\eta$ as well since $\\nabla L_\\eta(z) =\\nabla L(z_+)=\\nabla L(z)= 0$.\n\t\n\tConversely consider a stationary point $z=(x,y)$ of $L_\\eta$. Then $\\eta(x-x_+) = \\nabla_x L_\\eta(z)=0$ and $\\eta(y_+-y) = \\nabla_y L_\\eta(z)=0$. Hence we again find that $z=z_+=\\prox_\\eta(z)$ and consequently, this point must be a stationary point of $L$ as well since $\\nabla L(z)=\\nabla L(z_+)=\\nabla L_\\eta(z)=0$.\n\\end{proof}\n\n\\begin{corollary} \\label{cor:PPM-to-GDA}\n\tOne step of the (damped) PPM~\\eqref{eq:saddle-PPM} on the original objective $L$ is equivalent to one step of GDA~\\eqref{eq:GDA} on the saddle envelope $L_\\eta$ with $s = \\lambda/\\eta$. \n\\end{corollary}\n\n\\begin{proof}\n\tLet $(x^+_k,y^+_k) = \\prox_\\eta(x_k,y_k)$ and let $(x_{k+1}, y_{k+1})$ be a step of GDA on $L_\\eta(x,y)$ from $(x_k,y_k)$ with step-size $s=\\lambda/\\eta$. Then\n\t\\begin{align*}\n\t\\begin{bmatrix} x_{k+1} \\\\ y_{k+1} \\end{bmatrix} = \\begin{bmatrix} x_{k} \\\\ y_{k}\\end{bmatrix} - s\\begin{bmatrix} \\nabla_xL_\\eta(z_k) \\\\ -\\nabla_yL_\\eta(z_k)\\end{bmatrix}\n\t&= \\begin{bmatrix} x_{k} \\\\ y_{k}\\end{bmatrix} - \\frac{\\lambda}{\\eta}\\begin{bmatrix} \\eta(x_k-x^+_k) \\\\ -\\eta(y^+_k-y_k) \\end{bmatrix}\n\t= (1-\\lambda)\\begin{bmatrix} x_{k} \\\\ y_{k}\\end{bmatrix} + \\lambda\\begin{bmatrix} x^+_k \\\\ y^{+}_k \\end{bmatrix}\n\t\\end{align*}\n\tfollows from Lemma~\\ref{lem:grad-formula}.\n\\end{proof}\n\n\\begin{lemma} \\label{lem:hess-formula}\n\tThe Hessian of the saddle envelope $L_\\eta(z)$ is\n\t$$ \\begin{bmatrix} \\nabla^2_{xx} L_\\eta(z) & \\nabla^2_{xy} L_\\eta(z)\\\\  -\\nabla^2_{yx}L_\\eta(z) & -\\nabla^2_{yy} L_\\eta(z)\\end{bmatrix} = \\eta I-\\eta^2\\left(\\eta I+ \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)^{-1} $$\n\twhere $z_+ = \\prox_\\eta(z)$. Since $\\eta>\\rho$, we have\n\t$$ \\nabla^2_{xx} L_\\eta(z) = \\eta I -\\eta^2\\left(\\eta I +\\nabla_{xx} L(z_+) + \\nabla^2_{xy} L(z_+)(\\eta I -\\nabla^2_{yy} L(z_+))^{-1}\\nabla^2_{yx} L(z_+)\\right)^{-1},$$\n\t$$ \\nabla^2_{yy} L_\\eta(z) = -\\eta I +\\eta^2\\left(\\eta I +\\nabla_{yy} L(z_+) + \\nabla^2_{yx} L(z_+)(\\eta I +\\nabla^2_{xx} L(z_+))^{-1}\\nabla^2_{xy} L(z_+)\\right)^{-1}.$$\n\\end{lemma}\n\n\\begin{proof}\n\tConsider some $z=(x,y)$ and a nearby point $z^\\Delta=z+\\Delta$. Denote one proximal step from each of these points by $z_+=(x_+,y_+)= \\prox_\\eta(z)$ and $z^\\Delta_+=(x^\\Delta_+,y^\\Delta_+)= \\prox_\\eta(z^\\Delta)$. \n\tThen our claimed Hessian formula amounts to showing\n\t\\begin{align*}\n\t\t\\begin{bmatrix} \\nabla_x L_\\eta(z^\\Delta) \\\\ -\\nabla_y L_\\eta(z^\\Delta) \\end{bmatrix} - \\begin{bmatrix} \\nabla_x L_\\eta(z) \\\\ -\\nabla_y L_\\eta(z) \\end{bmatrix}  &= \\left(\\eta I - \\eta^2\\left(\\eta I + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)^{-1}\\right)\\begin{bmatrix} \\Delta_x\\\\ \\Delta_y  \\end{bmatrix} +o(\\|\\Delta\\|).\n\t\\end{align*}\n\tRecall Lemma~\\ref{lem:grad-formula} showed the gradient of the saddle envelope is given by $\\nabla_x L_\\eta(z) = \\eta(x-x_+)$ and $\\nabla_y L_\\eta(z) = \\eta(y_+-y)$. Applying this at $z$ and $z_+$ and dividing by $\\eta$, our claimed Hessian formula becomes\n\t\\begin{equation} \\label{eq:target-hessian-condition}\n\t\\begin{bmatrix} x^\\Delta_+-x_+\\\\ y^\\Delta_+-y_+ \\end{bmatrix} = \\eta\\left(\\eta I + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)^{-1}\\begin{bmatrix} \\Delta_x\\\\ \\Delta_y \\end{bmatrix} +o(\\|\\Delta\\|) \\ .\n\t\\end{equation}\n\tOur proof shows this in two steps: first considering a proximal step on the second-order Taylor approximation of $L$ at $z_+$ and then showing this closely matches the result of a proximal step on $L$.\n\t\n\tFirst, consider the following quadratic model of the objective around $z_+$:\n\t$$ \\widetilde L(z) = L(z_+) + \\nabla L(z_+)^T(z-z_+) + \\frac{1}{2}(z-z_+)^T\\nabla^2 L(z_+)(z-z_+) \\ . $$\n\tDenote the result of one proximal step on $\\widetilde L$ from $z^\\Delta$ by $\\widetilde z^\\Delta_+ = (\\widetilde x_+^\\Delta, \\widetilde y_+^\\Delta)$. Since the proximal subproblem is strongly convex-strongly concave, this solution is uniquely determined by\n\t$$\n\t\\begin{bmatrix} \\nabla_x \\widetilde L(\\widetilde x^\\Delta_+,\\widetilde y^\\Delta_+)\\\\  -\\nabla_y \\widetilde L(\\widetilde x^\\Delta_+,\\widetilde y^\\Delta_+)\\end{bmatrix} + \\begin{bmatrix} \\eta(\\widetilde x^\\Delta_+-x^\\Delta)\\\\ \\eta(\\widetilde y^\\Delta_+-y^\\Delta) \\end{bmatrix} = 0 \\ .\n\t$$\n\tPlugging in the definition of our quadratic model $\\widetilde L$ yields\n\t\\begin{align*}\n\t&\\begin{bmatrix} \\nabla_x L(z_+)\\\\ -\\nabla_y L(z_+)\\end{bmatrix} + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\begin{bmatrix} \\widetilde x^\\Delta_+-x_+\\\\ \\widetilde y^\\Delta_+-y_+ \\end{bmatrix} + \\begin{bmatrix} \\eta(\\widetilde x^\\Delta_+-x^\\Delta)\\\\ \\eta(\\widetilde y^\\Delta_+-y^\\Delta) \\end{bmatrix} = 0 \\ .\n\t\\end{align*}\n\tHence\n\t\\begin{align*}\n\t& \\left(\\eta I + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)\\begin{bmatrix} \\widetilde x^\\Delta_+-x_+\\\\  \\widetilde y^\\Delta_+-y_+ \\end{bmatrix} = \\eta \\begin{bmatrix} x^\\Delta - x_+ -\\eta^{-1}\\nabla_x L(z_+) \\\\ y^\\Delta - y_+ +\\eta^{-1}\\nabla_y L(z_+) \\end{bmatrix}\\\\\n\t&\\implies  \\left(\\eta I + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)\\begin{bmatrix} \\widetilde x^\\Delta_+-x_+\\\\  \\widetilde y^\\Delta_+-y_+ \\end{bmatrix} = \\eta \\begin{bmatrix} \\Delta_x\\\\ \\Delta_y \\end{bmatrix}\\\\\n\t&\\implies  \\begin{bmatrix} \\widetilde x^\\Delta_+-x_+\\\\ \\widetilde y^\\Delta_+-y_+ \\end{bmatrix} = \\eta\\left(\\eta I + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)^{-1}\\begin{bmatrix} \\Delta_x\\\\ \\Delta_y \\end{bmatrix}.\n\t\\end{align*}\n\t\n\tThis is nearly our target condition~\\eqref{eq:target-hessian-condition}. All that remains is to show our second-order approximation satisfies $\\| z^\\Delta_+ - \\widetilde z^\\Delta_+\\| = o(\\|\\Delta\\|)$.\n\tDenote the proximal subproblem objective by $ M^\\Delta (u,v) = L(u,v) + \\frac{\\eta}{2}\\|u-x^\\Delta\\|^2 - \\frac{\\eta}{2}\\|v-y^\\Delta\\|^2$ and its approximation by $\\widetilde M^\\Delta (u,v) = \\widetilde L(u,v) + \\frac{\\eta}{2}\\|u-x^\\Delta\\|^2 - \\frac{\\eta}{2}\\|v-y^\\Delta\\|^2$.\n\tNoting that $\\|\\nabla \\widetilde M^\\Delta (x_+,y_+)\\| = \\eta\\|\\Delta\\|$, we can apply Lemma~\\ref{lem:helper0} to the $(\\eta-\\rho)$-strongly convex-strongly concave function $\\widetilde M^\\Delta$ to bound the distance to its minimax point as\n\t$$ \\|z_+-\\widetilde z_+^\\Delta\\|\\leq \\frac{\\eta}{\\eta-\\rho}\\|\\Delta\\| \\ .$$\n\tConsequently, we can bound difference in gradients between $L$ and its quadratic model $\\widetilde L$ at $\\widetilde z^\\Delta_+$ by $ \\|\\nabla L(\\widetilde z^\\Delta_+) - \\nabla \\widetilde L(\\widetilde z^\\Delta_+)\\| = o(\\|\\Delta\\|)$. Therefore $\\|\\nabla M^\\Delta(\\widetilde z^\\Delta_+)\\| = o(\\|\\Delta\\|)$ and so applying Lemma~\\ref{lem:helper0} to the strongly convex-strongly concave function $M^\\Delta$ bounds the distance to its minimax point as $\\|z^\\Delta_+ - \\widetilde z^\\Delta_+\\| = o(\\|\\Delta\\|)$, which completes our proof.\n\\end{proof}\n\n\\begin{proposition} \\label{prop:hessians}\n\tIf the $x$-interaction dominance~\\eqref{eq:x-dominance} holds with $\\alpha\\in\\RR$, the saddle envelope is smooth and weakly convex with respect to $x$ \n\t$$  (\\eta^{-1} + \\alpha^{-1})^{-1}I \\preceq \\nabla^2_{xx}L_\\eta(z) \\preceq \\eta I\\ ,$$\n\tand if the $y$-interaction dominance condition~\\eqref{eq:y-dominance} holds with $\\alpha\\in\\RR$, the saddle envelope is smooth and weakly concave with respect to $y$\n\t$$  (\\eta^{-1} + \\alpha^{-1})^{-1}I \\preceq -\\nabla^2_{yy}L_\\eta(z) \\preceq \\eta I\\ .$$\n\\end{proposition}\n\n\\begin{proof}\n\tRecall the formula for the $x$ component of the Hessian given by Lemma~\\ref{lem:hess-formula}.\n\tThen the interaction dominance condition~\\eqref{eq:x-dominance} can lower bound this Hessian by\n\t\\begin{align*}\n\t\\nabla^2_{xx} L_\\eta(z) &=\\eta I -\\eta^2\\left(\\eta I +\\nabla_{xx} L(z_+) + \\nabla^2_{xy} L(z_+)(\\eta I -\\nabla^2_{yy} L(z_+))^{-1}\\nabla^2_{yx} L(z_+)\\right)^{-1}\\\\\n\t&\\succeq \\eta I -\\eta^2\\left(\\eta I +\\alpha I\\right)^{-1}\\\\\n\t&=(\\eta - \\eta^2/(\\eta+\\alpha))I\\\\\n\t&=(\\eta^{-1} + \\alpha^{-1})^{-1}I \\ .\n\t\\end{align*}\n\tNote that $\\eta I +\\nabla^2_{xx} L(z_+)$ is positive definite (since $\\eta>\\rho$) and $\\nabla^2_{xy} L(z_+)(\\eta I +\\nabla^2_{yy} L(z_+))^{-1}\\nabla^2_{yx} L(z_+)$ is positive semidefinite (since its written as a square). Then the inverse of their sum must also be positive definite and consequently $\\nabla^2_{xx} L_\\eta(z)$ is upper bounded by\n\t\\begin{align*}\n\t\\eta I -\\eta^2\\left(\\eta I +\\nabla_{xx} L(z_+) + \\nabla^2_{xy} L(z_+)(\\eta I +\\nabla^2_{yy} L(z_+))^{-1}\\nabla^2_{yx} L(z_+)\\right)^{-1} &\\preceq \\eta I \\ .\n\t\\end{align*}\n\tSymmetric reasoning applies to give bounds on $\\nabla^2_{yy} L_\\eta(z)$.\n\\end{proof}\n\n\\begin{proposition}\\label{prop:smoothness}\n\t$L_\\eta$ has $\\max\\{\\eta, |\\eta^{-1}-\\rho^{-1}|^{-1}\\}$-Lipschitz gradient.\n\\end{proposition}\n\n\\begin{proof}\n\tConsider two points $z=(x,y)$ and $\\bar z = (\\bar x, \\bar y)$ and denote one proximal step from each of them by $z_+=(x_+,y_+) = \\prox_\\eta(z)$ and $\\bar z_+ = (\\bar x_+,\\bar y_+) = \\prox_\\eta(\\bar z)$.\n\tDefine the  $(\\eta-\\rho)$-strongly convex-strongly concave function underlying the computation of the saddle envelope at $z$ as\n\t$$ M(u,v) = L(u,v) + \\frac{\\eta}{2}\\|u-x\\|^2 -\\frac{\\eta}{2}\\|v-y\\|^2. $$\n\tFirst we compute the gradient of $M$ at $\\bar z_+$ which is given by\n\t\\begin{align*}\n\t\\begin{bmatrix} \\nabla_x M(\\bar z_+) \\\\ \\nabla_y M(\\bar z_+)\\end{bmatrix} =  \\begin{bmatrix} \\nabla_x L(\\bar z_+) + \\eta(\\bar x_+-x) \\\\ \\nabla_y L(\\bar z_+) - \\eta(\\bar y_+-y)\\end{bmatrix}\n\t%=  \\begin{bmatrix} \\nabla_x L(\\bar z_+) + \\eta(\\bar x_+-\\bar x+\\bar x-x) \\\\ \\nabla_y L(\\bar z_+) - \\eta(\\bar y_+-\\bar y+\\bar y-y)\\end{bmatrix}\\\n\t= \\eta \\begin{bmatrix} \\bar x-x \\\\ y-\\bar y\\end{bmatrix}.\n\t\\end{align*}\n\tApplying Lemma~\\ref{lem:helper0}, and noting that $z_+=\\prox_\\eta(z)$ has $\\nabla M(z_+)=0$ yields\n\t$$ \\left\\|\\begin{bmatrix} \\bar x_+-x_+ \\\\ \\bar y_+-y_+ \\end{bmatrix}\\right\\|^2 \\leq \\frac{\\eta}{\\eta-\\rho} \\begin{bmatrix} \\bar x- x \\\\ \\bar y- y \\end{bmatrix}^T\\begin{bmatrix} \\bar x_+-x_+ \\\\ \\bar y_+-y_+ \\end{bmatrix}.$$\n\tRecalling the saddle envelope's gradient formula from Lemma~\\ref{lem:grad-formula}, we can upper bound the difference between its gradients at $z$ and $\\bar z$ by\n\t\\begin{align*}\n\t\\frac{1}{\\eta^{2}}\\|\\nabla L_\\eta(z) - \\nabla L_\\eta(\\bar z)\\|^2\n\t&= \\left\\|\\begin{bmatrix} x-x_+ \\\\ y_+-y \\end{bmatrix} - \\begin{bmatrix} \\bar x-\\bar x_+ \\\\ \\bar y_+-\\bar y \\end{bmatrix}\\right\\|^2\\\\\n\t&=\\left\\|\\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}\\right\\|^2 + 2 \\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}^T\\begin{bmatrix} \\bar x_+- x_+ \\\\ \\bar y_+- y_+ \\end{bmatrix} +\\left\\|\\begin{bmatrix} \\bar x_+- x_+ \\\\ \\bar y_+- y_+ \\end{bmatrix}\\right\\|^2\\\\\n\t&\\leq \\left\\|\\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}\\right\\|^2 + \\left(\\frac{\\eta}{\\eta-\\rho}-2\\right)\\begin{bmatrix} \\bar x- x \\\\ \\bar y- y \\end{bmatrix}^T\\begin{bmatrix} \\bar x_+- x_+ \\\\ \\bar y_+- y_+ \\end{bmatrix}.\n\t\\end{align*}\n\tNotice that $\\begin{bmatrix} \\bar x- x \\\\ \\bar y- y \\end{bmatrix}^T\\begin{bmatrix} \\bar x_+- x_+ \\\\ \\bar y_+- y_+ \\end{bmatrix}$ is non-negative but the sign of $\\left(\\frac{\\eta}{\\eta-\\rho}-2\\right)$ may be positive or negative. If this coefficient is negative, we can upperbound the second term above by zero, giving $\t\\|\\nabla L_\\eta(z) - \\nabla L_\\eta(\\bar z)\\|^2 \\leq \\eta^2\\left\\|z-\\bar z\\right\\|^2 .$\n\tIf instead $\\left(\\frac{\\eta}{\\eta-\\rho}-2\\right)\\geq 0$, then we have smoothness constant $|\\eta^{-1}-\\rho^{-1}|^{-1}$ as\n\t\\begin{align*}\n\t\\|\\nabla L_\\eta(z) - \\nabla L_\\eta(\\bar z)\\|^2 &\\leq \\eta^2\\left(1 + \\left(\\frac{\\eta}{\\eta-\\rho}-2\\right)\\frac{\\eta}{\\eta-\\rho}\\right)\\left\\|\\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}\\right\\|^2 \\\\\n\t&= \\eta^2\\left(\\frac{\\eta}{\\eta-\\rho}-1\\right)^2\\left\\|\\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}\\right\\|^2 \\\\\n\t&= \\left(\\frac{\\eta\\rho}{\\eta-\\rho}\\right)^2\\left\\|\\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}\\right\\|^2 \\ .\n\t\\end{align*}\n\tby Cauchy–Schwarz and using that $\\left\\|\\begin{bmatrix} \\bar x_+- x_+ \\\\ \\bar y_+- y_+ \\end{bmatrix}\\right\\| \\leq \\frac{\\eta}{\\eta-\\rho}\\left\\|\\begin{bmatrix} \\bar x- x \\\\ \\bar y- y \\end{bmatrix}\\right\\|$.\n\\end{proof}\n\n\\subsection{Proof of Theorem~\\ref{thm:two-sided-dominance}}\nPropositions~\\ref{prop:hessians} and~\\ref{prop:smoothness} show that $L_\\eta$ is $\\mu=(\\eta^{-1}+\\alpha^{-1})^{-1}$-strongly convex-strongly concave and has a $\\beta=\\max\\{\\eta, |\\eta^{-1}-\\rho^{-1}|^{-1}\\}$-Lipschitz gradient. Having strong convexity and strong concavity ensures $L_\\eta$ has a unique stationary point $(x^*,y^*)$, which in turn must be the unique stationary point of $L$ by Corollary~\\ref{cor:stationarity}. \nRecall Corollary~\\ref{cor:PPM-to-GDA} showed that the damped PPM~\\eqref{eq:saddle-PPM} on $L$ is equivalent to GDA~\\eqref{eq:GDA} with $s=\\lambda/\\eta$ on $L_\\eta$. Then provided \n$$ \\lambda \\leq 2\\frac{\\min\\left\\{1,(\\eta/\\rho-1)^2\\right\\}}{\\eta/\\alpha+1} = \\frac{2(\\eta^{-1}+\\alpha^{-1})^{-1}}{\\max\\{\\eta^2, (\\eta^{-1}-\\rho^{-1})^{-2}\\}}\\ , $$\nwe have $s=\\lambda/\\eta\\in(0,2\\mu/\\beta^2)$. Hence applying Theorem~\\ref{thm:standard-contraction} shows the iterations of GDA (and consequently PPM) linearly converge to this unique stationary point as\n\\begin{align*}\n\\left\\|\\begin{bmatrix} x_k-x^* \\\\ y_k-y^* \\end{bmatrix}\\right\\|^2 &\\leq \\left(1 - \\frac{2\\lambda}{\\eta/\\alpha+1} + \\frac{\\lambda^2}{\\min\\left\\{1,(\\eta/\\rho-1)^2\\right\\}}\\right)^{k}\\left\\|\\begin{bmatrix} x_0-x^* \\\\ y_0-y^* \\end{bmatrix}\\right\\|^2.\n\\end{align*}\n\n\n\n\n\\subsection{Proof of Lemma~\\ref{lem:helper0}}\nObserve that the assumed strong convexity and strong concavity ensures\n\\begin{align*}\nM(x',y') \\leq &M(x,y') -\\nabla_x M(x',y')^T(x-x')- \\frac{\\mu}{2}\\|x-x'\\|^2\\\\\n&\\leq M(x,y) + \\nabla_y M(x,y)^T(y'-y) -\\nabla_x M(x',y')^T(x-x') - \\frac{\\mu}{2}\\|y-y'\\|^2 - \\frac{\\mu}{2}\\|x-x'\\|^2\n\\end{align*}\nand symmetrically,\n\\begin{align*}\nM(x',y') &\\geq M(x',y) - \\nabla_y M(x',y')^T(y-y') + \\frac{\\mu}{2}\\|y-y^*\\|^2\\\\\n&\\geq M(x,y) + \\nabla_x M(x,y)^T(x'-x) - \\nabla_y M(x',y')^T(y-y') + \\frac{\\mu}{2}\\|x-x'\\|^2 + \\frac{\\mu}{2}\\|y-y'\\|^2.\n\\end{align*}\nCombining the above two inequalities gives the first claimed inequality\n\\begin{equation*}\n\\mu\\left\\|\\begin{bmatrix} x-x' \\\\ y-y' \\end{bmatrix}\\right\\|^2 \\leq \\left(\\begin{bmatrix} \\nabla_x M(x,y)  \\\\-\\nabla_y M(x,y) \\end{bmatrix} -\\begin{bmatrix} \\nabla_x M(x',y')  \\\\-\\nabla_y M(x',y') \\end{bmatrix}\\right)^T\\begin{bmatrix} x-x' \\\\ y-y' \\end{bmatrix}.\n\\end{equation*}\nFurthermore, when $\\nabla M(x',y')=0$, Cauchy–Schwarz gives the second inequality.\n\n\\subsection{Proof of Theorem~\\ref{thm:standard-contraction}}\nFirst we use Lemma~\\ref{lem:helper0} to conclude that if the set $S$ is large enough, $M$ must have a stationary point in $S$. Define $B(z,r)=\\{z'|\\|z-z'\\|\\le r\\}$ as the ball centered as $a$ with radius $r$. \n\n\n\\begin{lemma} \\label{lem:helper1}\n\tSuppose $M$ is $\\mu$-strongly convex-strongly concave in a set $B(x,r)\\times B(y,r)$ for some fixed $(x,y)$ and $r\\geq 2\\|\\nabla M(x,y)\\|/\\mu$, then there exists a stationary point of $M$ in $B((x,y),r/2)$.\n\\end{lemma}\n\n\\begin{proof}\n\tConsider the following constrained minimax problem $ \\min_{x'\\in B(x,r)} \\max_{y'\\in B(y,r)} M(x,y).$\n\tSince $M(x,y)$ is strongly convex-strongly concave, it must have a unique solution $(x^*,y^*)$.\n\tThe first-order optimality condition for $(x^*,y^*)$ ensures\n\t$$ \\nabla_x M(x^*,y^*) =-\\lambda(x^*-x) \\ \\ \\ \\text{ and } \\ \\ \\  -\\nabla_y M(x^*,y^*) =-\\gamma(y^*-y)$$\n\tfor some constants $\\lambda,\\gamma\\geq 0$ that are nonzero only if $x^*$ or $y^*$ are on the boundary of $B(x,r)$ and $B(y,r)$ respectively. Taking an inner product with $(x^*-x,y^*-y)$ gives\n\t\\begin{equation}\\label{eq:lemA2.1}\n\t\\begin{bmatrix} \\nabla_x M(x^*,y^*)  \\\\-\\nabla_y M(x^*,y^*) \\end{bmatrix}^T\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix} =-\\left\\|\\begin{bmatrix}\\sqrt{\\lambda}(x^*-x)  \\\\ \\sqrt{\\gamma}(y^*-y) \\end{bmatrix}\\right\\|^2\\leq0.\n\t\\end{equation}\n\tApplying Lemma~\\ref{lem:helper0} and utilizing \\eqref{eq:lemA2.1}, we conclude that\n\t\\begin{equation}\\label{eq:lemA2.2}\n\t\\mu\\left\\|\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix}\\right\\|^2 + \\begin{bmatrix} \\nabla_x M(x,y)  \\\\-\\nabla_y M(x,y) \\end{bmatrix}^T\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix}\\leq 0. \n\t\\end{equation}\n\tHence\n\t$$ \\left\\|\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix}\\right\\|^2 \\leq \\frac{1}{\\mu}\\left\\|\\begin{bmatrix} \\nabla_x M(x,y)  \\\\-\\nabla_y M(x,y) \\end{bmatrix}\\right\\|\\left\\|\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix}\\right\\|\\ ,$$\n\twhereby\n\t$$ \\left\\|\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix}\\right\\| \\leq \\frac{1}{\\mu}\\left\\|\\begin{bmatrix} \\nabla_x M(x,y)  \\\\-\\nabla_y M(x,y) \\end{bmatrix}\\right\\| \\leq r/2\\ ,$$\n\tutilizing our assumed condition on $r$.\n\tSince $(x^*,y^*)$ lies strictly inside $B((x,y),r)$, the first-order optimality condition implies $(x^*,y^*)$ is a stationary point of $M$.\n\\end{proof}\n\nLemma~\\ref{lem:helper1} ensures the existence of a nearby stationary point $(x^*,y^*)$. Then the standard proof for strongly monotone (from Lemma~\\ref{lem:helper0}) and Lipschitz operators gives a contraction whenever $s\\in(0, 2\\mu/\\beta^2) $:\n\\begin{align*}\n\\left\\|\\begin{bmatrix} x_{k+1}-x^* \\\\ y_{k+1}-y^* \\end{bmatrix}\\right\\|^2\n&= \\left\\|\\begin{bmatrix} x_{k}-x^* \\\\ y_{k}-y^* \\end{bmatrix}\\right\\|^2 - 2s\\begin{bmatrix} \\nabla_x M(x_k,y_k) \\\\ -\\nabla_y M(x_k,y_k) \\end{bmatrix}^T\\begin{bmatrix} x_k-x^* \\\\ y_k-y^* \\end{bmatrix} + s^2\\left\\|\\begin{bmatrix} \\nabla_x M(x_k,y_k) \\\\ -\\nabla_y M(x_k,y_k) \\end{bmatrix}\\right\\|^2\\\\\n&\\leq \\left\\|\\begin{bmatrix} x_{k}-x^* \\\\ y_{k}-y^* \\end{bmatrix}\\right\\|^2 - 2\\mu s\\left\\|\\begin{bmatrix} x_{k}-x^* \\\\ y_{k}-y^* \\end{bmatrix}\\right\\|^2 + \\beta^2s^2\\left\\|\\begin{bmatrix} x_{k}-x^* \\\\ y_{k}-y^* \\end{bmatrix}\\right\\|^2\\\\\n&= \\left(1 - 2\\mu s+\\beta^2s^2\\right)\\left\\|\\begin{bmatrix} x_{k}-x^* \\\\ y_{k}-y^* \\end{bmatrix}\\right\\|^2 \\ , \n\\end{align*}\nwhere the inequality utilizes \\eqref{eq:lemA2.2} at $(x,y)=(x_k,y_k)$ and the smoothness of $M(x,y)$. \n","units":[{"start":21172,"end":21816,"start_line":176,"end_line":183,"raw_tex":"\\begin{lemma} \\label{lem:helper0}\n\tSuppose $M(x,y)$ is $\\mu$-strongly convex-strongly concave on a convex set $S=S_x\\times S_y$, then it holds for any $(x,y),(x',y') \\in S$ that\n\t$$\\mu\\left\\|\\begin{bmatrix} x-x' \\\\ y-y' \\end{bmatrix}\\right\\|^2 \\leq \\left(\\begin{bmatrix} \\nabla_x M(x,y)  \\\\-\\nabla_y M(x,y) \\end{bmatrix} -\\begin{bmatrix} \\nabla_x M(x',y')  \\\\-\\nabla_y M(x',y') \\end{bmatrix}\\right)^T\\begin{bmatrix} x-x' \\\\ y-y' \\end{bmatrix}.$$\n\tWhen $\\nabla M(x', y')=0$, the distance to this stationary point is bounded by\n\t$$\n\t\\left\\|\\begin{bmatrix} x-x' \\\\ y-y' \\end{bmatrix}\\right\\| \\leq \\frac{\\|\\nabla M(x,y)\\|}{\\mu} \\ . \n\t$$\n\\end{lemma}","sha256_utf8":"f99310ccda8c0be5e3b62da3b0bd43461992de4813216f44e8cf16749be67464"},{"start":21817,"end":22427,"start_line":184,"end_line":187,"raw_tex":"\\begin{theorem} \\label{thm:standard-contraction}\n\tConsider any minimax problem $\\min_{x\\in\\RR^n}\\max_{y\\in\\RR^m} M(x,y)$ where $M(x,y)$ is $\\beta$-smooth and $\\mu$-strongly convex-strongly concave on a set $B(x_0,r)\\times B(y_0,r)$ with $r\\geq 2\\|\\nabla M(x_0,y_0)\\|/\\mu$. Then GDA~\\eqref{eq:GDA} with step-size $s\\in(0, 2\\mu/\\beta^2) $ linearly converges to a stationary point $(x^*,y^*)\\in B((x_0,y_0),r/2)$ with\n\t$$ \\left\\|\\begin{bmatrix} x_k-x^* \\\\ y_k-y^* \\end{bmatrix}\\right\\|^2 \\leq \\left(1 - 2\\mu s+\\beta^2s^2\\right)^{k}\\left\\|\\begin{bmatrix} x_0-x^* \\\\ y_0-y^* \\end{bmatrix}\\right\\|^2.$$\n\\end{theorem}","sha256_utf8":"78f02cad031cbe56e06e92b0900742c90951ee162f42ff173c18ff20d8fb8a7c"},{"start":26998,"end":27391,"start_line":235,"end_line":239,"raw_tex":"\\begin{lemma} \\label{lem:grad-formula}\n\tThe gradient of the saddle envelope $L_\\eta(x,y)$ at $z=(x,y)$ is\n\t$$ \\begin{bmatrix} \\nabla_x L_\\eta(z)\\\\  \\nabla_y L_\\eta(z)\\end{bmatrix} = \\begin{bmatrix} \\eta(x-x_+)\\\\ \\eta(y_+-y) \\end{bmatrix} = \\begin{bmatrix} \\nabla_x L(z_+)\\\\  \\nabla_y L(z_+)\\end{bmatrix}$$\n\twhere $z_+ = (x_+,y_+) = \\prox_\\eta(z)$ is given by the proximal operator.\n\\end{lemma}","sha256_utf8":"3f2af75cf3db4b9dfc0cae13f5c93b2c0d061ba4bc275b1f7935aef71780d10e"},{"start":27392,"end":28208,"start_line":240,"end_line":250,"raw_tex":"\\begin{proof}\n\tNotice that the saddle envelope is a composition of Moreau envelopes\n\t\\begin{align*}\n\tL_{\\eta}(x,y) &= \\min_{u} \\left(\\max_{v} L(u,v) - \\frac{\\eta}{2}\\|v-y\\|^2\\right) + \\frac{\\eta}{2}\\|u-x\\|^2\\\\\n\t&= \\min_{u}\\ -e_\\eta\\{-L(u,\\cdot)\\}(y) + \\frac{\\eta}{2}\\|u-x\\|^2 = e_{\\eta}\\{g(\\cdot,y)\\}(x)\n\t\\end{align*}\n\twhere $g(u,y) = -e_{\\eta}\\{-L(u,\\cdot)\\}(y)$. Applying the gradient formula~\\eqref{eq:moreau-grad} gives our first claimed gradient formula in $x$ of\n\t$\\nabla_x L_\\eta(x,y) = \\eta (x-x_+)$\n\tsince $x_+$ is the unique minimizer of $u \\mapsto g(u,y) + \\frac{\\eta}{2}\\|u-x\\|^2$. Symmetric reasoning gives our first claimed formula $\\nabla_y L_\\eta(x,y) = \\eta (y_+-y) $ in $y$.\n\tThe second claimed equality is precisely the first-order optimality condition for~\\eqref{eq:prox-definition}. \n\\end{proof}","sha256_utf8":"e355d72fdee48531c9496004fcf327e7d7723878ff0853750037eb7199bc7955"},{"start":28209,"end":28340,"start_line":251,"end_line":253,"raw_tex":"\\begin{corollary} \\label{cor:stationarity}\n\tThe stationary points of $L_\\eta$ are exactly the same as those of $L$.\n\\end{corollary}","sha256_utf8":"235b1efa236c8626ca00c9eb16ff82a3a299aab7bffb1d369ef2c88f20cec845"},{"start":28341,"end":29154,"start_line":254,"end_line":258,"raw_tex":"\\begin{proof}\n\tFirst consider any stationary point $z=(x,y)$ of $L$. Denote $z_+=\\prox_\\eta(z)$ and the objective function defining the proximal operator~\\eqref{eq:prox-definition} as $M(u,v) = L(u,v) + \\frac{\\eta}{2}\\|u-x\\|^2-\\frac{\\eta}{2}\\|v-y\\|^2$. Then observing that $\\nabla M(z)=0$, $z$ must be the unique minimax point of $M$ (that is, $z=z_+=\\prox_\\eta(z)$). Hence $z$ must be a stationary point of $L_\\eta$ as well since $\\nabla L_\\eta(z) =\\nabla L(z_+)=\\nabla L(z)= 0$.\n\t\n\tConversely consider a stationary point $z=(x,y)$ of $L_\\eta$. Then $\\eta(x-x_+) = \\nabla_x L_\\eta(z)=0$ and $\\eta(y_+-y) = \\nabla_y L_\\eta(z)=0$. Hence we again find that $z=z_+=\\prox_\\eta(z)$ and consequently, this point must be a stationary point of $L$ as well since $\\nabla L(z)=\\nabla L(z_+)=\\nabla L_\\eta(z)=0$.\n\\end{proof}","sha256_utf8":"668d4e8b99a78b02b04329fb30fdbc130531ea97c6439c89e328ee638281a2d4"},{"start":29155,"end":29399,"start_line":259,"end_line":261,"raw_tex":"\\begin{corollary} \\label{cor:PPM-to-GDA}\n\tOne step of the (damped) PPM~\\eqref{eq:saddle-PPM} on the original objective $L$ is equivalent to one step of GDA~\\eqref{eq:GDA} on the saddle envelope $L_\\eta$ with $s = \\lambda/\\eta$. \n\\end{corollary}","sha256_utf8":"1c286635ba373812d088f40e3e4e0d49b9e40d151b232a157abbc852ec737fc9"},{"start":29400,"end":30082,"start_line":262,"end_line":270,"raw_tex":"\\begin{proof}\n\tLet $(x^+_k,y^+_k) = \\prox_\\eta(x_k,y_k)$ and let $(x_{k+1}, y_{k+1})$ be a step of GDA on $L_\\eta(x,y)$ from $(x_k,y_k)$ with step-size $s=\\lambda/\\eta$. Then\n\t\\begin{align*}\n\t\\begin{bmatrix} x_{k+1} \\\\ y_{k+1} \\end{bmatrix} = \\begin{bmatrix} x_{k} \\\\ y_{k}\\end{bmatrix} - s\\begin{bmatrix} \\nabla_xL_\\eta(z_k) \\\\ -\\nabla_yL_\\eta(z_k)\\end{bmatrix}\n\t&= \\begin{bmatrix} x_{k} \\\\ y_{k}\\end{bmatrix} - \\frac{\\lambda}{\\eta}\\begin{bmatrix} \\eta(x_k-x^+_k) \\\\ -\\eta(y^+_k-y_k) \\end{bmatrix}\n\t= (1-\\lambda)\\begin{bmatrix} x_{k} \\\\ y_{k}\\end{bmatrix} + \\lambda\\begin{bmatrix} x^+_k \\\\ y^{+}_k \\end{bmatrix}\n\t\\end{align*}\n\tfollows from Lemma~\\ref{lem:grad-formula}.\n\\end{proof}","sha256_utf8":"ec72f3bcee7ac56f02c110a3990479ac5d92f3c6683d096b45f2081c2aedbb45"},{"start":30346,"end":31148,"start_line":273,"end_line":279,"raw_tex":"\\begin{lemma} \\label{lem:hess-formula}\n\tThe Hessian of the saddle envelope $L_\\eta(z)$ is\n\t$$ \\begin{bmatrix} \\nabla^2_{xx} L_\\eta(z) & \\nabla^2_{xy} L_\\eta(z)\\\\  -\\nabla^2_{yx}L_\\eta(z) & -\\nabla^2_{yy} L_\\eta(z)\\end{bmatrix} = \\eta I-\\eta^2\\left(\\eta I+ \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)^{-1} $$\n\twhere $z_+ = \\prox_\\eta(z)$. Since $\\eta>\\rho$, we have\n\t$$ \\nabla^2_{xx} L_\\eta(z) = \\eta I -\\eta^2\\left(\\eta I +\\nabla_{xx} L(z_+) + \\nabla^2_{xy} L(z_+)(\\eta I -\\nabla^2_{yy} L(z_+))^{-1}\\nabla^2_{yx} L(z_+)\\right)^{-1},$$\n\t$$ \\nabla^2_{yy} L_\\eta(z) = -\\eta I +\\eta^2\\left(\\eta I +\\nabla_{yy} L(z_+) + \\nabla^2_{yx} L(z_+)(\\eta I +\\nabla^2_{xx} L(z_+))^{-1}\\nabla^2_{xy} L(z_+)\\right)^{-1}.$$\n\\end{lemma}","sha256_utf8":"0e091ae7103b8cbe47634bc70f29d90f44ed6342fb0e366f06bb4e801fbc393b"},{"start":31149,"end":36253,"start_line":280,"end_line":314,"raw_tex":"\\begin{proof}\n\tConsider some $z=(x,y)$ and a nearby point $z^\\Delta=z+\\Delta$. Denote one proximal step from each of these points by $z_+=(x_+,y_+)= \\prox_\\eta(z)$ and $z^\\Delta_+=(x^\\Delta_+,y^\\Delta_+)= \\prox_\\eta(z^\\Delta)$. \n\tThen our claimed Hessian formula amounts to showing\n\t\\begin{align*}\n\t\t\\begin{bmatrix} \\nabla_x L_\\eta(z^\\Delta) \\\\ -\\nabla_y L_\\eta(z^\\Delta) \\end{bmatrix} - \\begin{bmatrix} \\nabla_x L_\\eta(z) \\\\ -\\nabla_y L_\\eta(z) \\end{bmatrix}  &= \\left(\\eta I - \\eta^2\\left(\\eta I + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)^{-1}\\right)\\begin{bmatrix} \\Delta_x\\\\ \\Delta_y  \\end{bmatrix} +o(\\|\\Delta\\|).\n\t\\end{align*}\n\tRecall Lemma~\\ref{lem:grad-formula} showed the gradient of the saddle envelope is given by $\\nabla_x L_\\eta(z) = \\eta(x-x_+)$ and $\\nabla_y L_\\eta(z) = \\eta(y_+-y)$. Applying this at $z$ and $z_+$ and dividing by $\\eta$, our claimed Hessian formula becomes\n\t\\begin{equation} \\label{eq:target-hessian-condition}\n\t\\begin{bmatrix} x^\\Delta_+-x_+\\\\ y^\\Delta_+-y_+ \\end{bmatrix} = \\eta\\left(\\eta I + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)^{-1}\\begin{bmatrix} \\Delta_x\\\\ \\Delta_y \\end{bmatrix} +o(\\|\\Delta\\|) \\ .\n\t\\end{equation}\n\tOur proof shows this in two steps: first considering a proximal step on the second-order Taylor approximation of $L$ at $z_+$ and then showing this closely matches the result of a proximal step on $L$.\n\t\n\tFirst, consider the following quadratic model of the objective around $z_+$:\n\t$$ \\widetilde L(z) = L(z_+) + \\nabla L(z_+)^T(z-z_+) + \\frac{1}{2}(z-z_+)^T\\nabla^2 L(z_+)(z-z_+) \\ . $$\n\tDenote the result of one proximal step on $\\widetilde L$ from $z^\\Delta$ by $\\widetilde z^\\Delta_+ = (\\widetilde x_+^\\Delta, \\widetilde y_+^\\Delta)$. Since the proximal subproblem is strongly convex-strongly concave, this solution is uniquely determined by\n\t$$\n\t\\begin{bmatrix} \\nabla_x \\widetilde L(\\widetilde x^\\Delta_+,\\widetilde y^\\Delta_+)\\\\  -\\nabla_y \\widetilde L(\\widetilde x^\\Delta_+,\\widetilde y^\\Delta_+)\\end{bmatrix} + \\begin{bmatrix} \\eta(\\widetilde x^\\Delta_+-x^\\Delta)\\\\ \\eta(\\widetilde y^\\Delta_+-y^\\Delta) \\end{bmatrix} = 0 \\ .\n\t$$\n\tPlugging in the definition of our quadratic model $\\widetilde L$ yields\n\t\\begin{align*}\n\t&\\begin{bmatrix} \\nabla_x L(z_+)\\\\ -\\nabla_y L(z_+)\\end{bmatrix} + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\begin{bmatrix} \\widetilde x^\\Delta_+-x_+\\\\ \\widetilde y^\\Delta_+-y_+ \\end{bmatrix} + \\begin{bmatrix} \\eta(\\widetilde x^\\Delta_+-x^\\Delta)\\\\ \\eta(\\widetilde y^\\Delta_+-y^\\Delta) \\end{bmatrix} = 0 \\ .\n\t\\end{align*}\n\tHence\n\t\\begin{align*}\n\t& \\left(\\eta I + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)\\begin{bmatrix} \\widetilde x^\\Delta_+-x_+\\\\  \\widetilde y^\\Delta_+-y_+ \\end{bmatrix} = \\eta \\begin{bmatrix} x^\\Delta - x_+ -\\eta^{-1}\\nabla_x L(z_+) \\\\ y^\\Delta - y_+ +\\eta^{-1}\\nabla_y L(z_+) \\end{bmatrix}\\\\\n\t&\\implies  \\left(\\eta I + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)\\begin{bmatrix} \\widetilde x^\\Delta_+-x_+\\\\  \\widetilde y^\\Delta_+-y_+ \\end{bmatrix} = \\eta \\begin{bmatrix} \\Delta_x\\\\ \\Delta_y \\end{bmatrix}\\\\\n\t&\\implies  \\begin{bmatrix} \\widetilde x^\\Delta_+-x_+\\\\ \\widetilde y^\\Delta_+-y_+ \\end{bmatrix} = \\eta\\left(\\eta I + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)^{-1}\\begin{bmatrix} \\Delta_x\\\\ \\Delta_y \\end{bmatrix}.\n\t\\end{align*}\n\t\n\tThis is nearly our target condition~\\eqref{eq:target-hessian-condition}. All that remains is to show our second-order approximation satisfies $\\| z^\\Delta_+ - \\widetilde z^\\Delta_+\\| = o(\\|\\Delta\\|)$.\n\tDenote the proximal subproblem objective by $ M^\\Delta (u,v) = L(u,v) + \\frac{\\eta}{2}\\|u-x^\\Delta\\|^2 - \\frac{\\eta}{2}\\|v-y^\\Delta\\|^2$ and its approximation by $\\widetilde M^\\Delta (u,v) = \\widetilde L(u,v) + \\frac{\\eta}{2}\\|u-x^\\Delta\\|^2 - \\frac{\\eta}{2}\\|v-y^\\Delta\\|^2$.\n\tNoting that $\\|\\nabla \\widetilde M^\\Delta (x_+,y_+)\\| = \\eta\\|\\Delta\\|$, we can apply Lemma~\\ref{lem:helper0} to the $(\\eta-\\rho)$-strongly convex-strongly concave function $\\widetilde M^\\Delta$ to bound the distance to its minimax point as\n\t$$ \\|z_+-\\widetilde z_+^\\Delta\\|\\leq \\frac{\\eta}{\\eta-\\rho}\\|\\Delta\\| \\ .$$\n\tConsequently, we can bound difference in gradients between $L$ and its quadratic model $\\widetilde L$ at $\\widetilde z^\\Delta_+$ by $ \\|\\nabla L(\\widetilde z^\\Delta_+) - \\nabla \\widetilde L(\\widetilde z^\\Delta_+)\\| = o(\\|\\Delta\\|)$. Therefore $\\|\\nabla M^\\Delta(\\widetilde z^\\Delta_+)\\| = o(\\|\\Delta\\|)$ and so applying Lemma~\\ref{lem:helper0} to the strongly convex-strongly concave function $M^\\Delta$ bounds the distance to its minimax point as $\\|z^\\Delta_+ - \\widetilde z^\\Delta_+\\| = o(\\|\\Delta\\|)$, which completes our proof.\n\\end{proof}","sha256_utf8":"de033d7fdde60281c83df02761202eccf32ee31f56537b38449e63ea0bf98b27"},{"start":39886,"end":40442,"start_line":342,"end_line":347,"raw_tex":"\\begin{proposition} \\label{prop:hessians}\n\tIf the $x$-interaction dominance~\\eqref{eq:x-dominance} holds with $\\alpha\\in\\RR$, the saddle envelope is smooth and weakly convex with respect to $x$ \n\t$$  (\\eta^{-1} + \\alpha^{-1})^{-1}I \\preceq \\nabla^2_{xx}L_\\eta(z) \\preceq \\eta I\\ ,$$\n\tand if the $y$-interaction dominance condition~\\eqref{eq:y-dominance} holds with $\\alpha\\in\\RR$, the saddle envelope is smooth and weakly concave with respect to $y$\n\t$$  (\\eta^{-1} + \\alpha^{-1})^{-1}I \\preceq -\\nabla^2_{yy}L_\\eta(z) \\preceq \\eta I\\ .$$\n\\end{proposition}","sha256_utf8":"f2d88316e9bbbb39a1af32588d3e11f27f5f63020dcd8d246c889ac5f296e8e7"},{"start":40443,"end":41602,"start_line":348,"end_line":362,"raw_tex":"\\begin{proof}\n\tRecall the formula for the $x$ component of the Hessian given by Lemma~\\ref{lem:hess-formula}.\n\tThen the interaction dominance condition~\\eqref{eq:x-dominance} can lower bound this Hessian by\n\t\\begin{align*}\n\t\\nabla^2_{xx} L_\\eta(z) &=\\eta I -\\eta^2\\left(\\eta I +\\nabla_{xx} L(z_+) + \\nabla^2_{xy} L(z_+)(\\eta I -\\nabla^2_{yy} L(z_+))^{-1}\\nabla^2_{yx} L(z_+)\\right)^{-1}\\\\\n\t&\\succeq \\eta I -\\eta^2\\left(\\eta I +\\alpha I\\right)^{-1}\\\\\n\t&=(\\eta - \\eta^2/(\\eta+\\alpha))I\\\\\n\t&=(\\eta^{-1} + \\alpha^{-1})^{-1}I \\ .\n\t\\end{align*}\n\tNote that $\\eta I +\\nabla^2_{xx} L(z_+)$ is positive definite (since $\\eta>\\rho$) and $\\nabla^2_{xy} L(z_+)(\\eta I +\\nabla^2_{yy} L(z_+))^{-1}\\nabla^2_{yx} L(z_+)$ is positive semidefinite (since its written as a square). Then the inverse of their sum must also be positive definite and consequently $\\nabla^2_{xx} L_\\eta(z)$ is upper bounded by\n\t\\begin{align*}\n\t\\eta I -\\eta^2\\left(\\eta I +\\nabla_{xx} L(z_+) + \\nabla^2_{xy} L(z_+)(\\eta I +\\nabla^2_{yy} L(z_+))^{-1}\\nabla^2_{yx} L(z_+)\\right)^{-1} &\\preceq \\eta I \\ .\n\t\\end{align*}\n\tSymmetric reasoning applies to give bounds on $\\nabla^2_{yy} L_\\eta(z)$.\n\\end{proof}","sha256_utf8":"0db845bd937190d043b58f1fe3b6d273a0962df09b860ed75719225550f864fa"},{"start":42721,"end":42858,"start_line":372,"end_line":374,"raw_tex":"\\begin{proposition}\\label{prop:smoothness}\n\t$L_\\eta$ has $\\max\\{\\eta, |\\eta^{-1}-\\rho^{-1}|^{-1}\\}$-Lipschitz gradient.\n\\end{proposition}","sha256_utf8":"ee22690bbe1172ce9ccf0022be0acdc3d6372efcfab8977a78586575f675e1c9"},{"start":42859,"end":46220,"start_line":375,"end_line":402,"raw_tex":"\\begin{proof}\n\tConsider two points $z=(x,y)$ and $\\bar z = (\\bar x, \\bar y)$ and denote one proximal step from each of them by $z_+=(x_+,y_+) = \\prox_\\eta(z)$ and $\\bar z_+ = (\\bar x_+,\\bar y_+) = \\prox_\\eta(\\bar z)$.\n\tDefine the  $(\\eta-\\rho)$-strongly convex-strongly concave function underlying the computation of the saddle envelope at $z$ as\n\t$$ M(u,v) = L(u,v) + \\frac{\\eta}{2}\\|u-x\\|^2 -\\frac{\\eta}{2}\\|v-y\\|^2. $$\n\tFirst we compute the gradient of $M$ at $\\bar z_+$ which is given by\n\t\\begin{align*}\n\t\\begin{bmatrix} \\nabla_x M(\\bar z_+) \\\\ \\nabla_y M(\\bar z_+)\\end{bmatrix} =  \\begin{bmatrix} \\nabla_x L(\\bar z_+) + \\eta(\\bar x_+-x) \\\\ \\nabla_y L(\\bar z_+) - \\eta(\\bar y_+-y)\\end{bmatrix}\n\t%=  \\begin{bmatrix} \\nabla_x L(\\bar z_+) + \\eta(\\bar x_+-\\bar x+\\bar x-x) \\\\ \\nabla_y L(\\bar z_+) - \\eta(\\bar y_+-\\bar y+\\bar y-y)\\end{bmatrix}\\\n\t= \\eta \\begin{bmatrix} \\bar x-x \\\\ y-\\bar y\\end{bmatrix}.\n\t\\end{align*}\n\tApplying Lemma~\\ref{lem:helper0}, and noting that $z_+=\\prox_\\eta(z)$ has $\\nabla M(z_+)=0$ yields\n\t$$ \\left\\|\\begin{bmatrix} \\bar x_+-x_+ \\\\ \\bar y_+-y_+ \\end{bmatrix}\\right\\|^2 \\leq \\frac{\\eta}{\\eta-\\rho} \\begin{bmatrix} \\bar x- x \\\\ \\bar y- y \\end{bmatrix}^T\\begin{bmatrix} \\bar x_+-x_+ \\\\ \\bar y_+-y_+ \\end{bmatrix}.$$\n\tRecalling the saddle envelope's gradient formula from Lemma~\\ref{lem:grad-formula}, we can upper bound the difference between its gradients at $z$ and $\\bar z$ by\n\t\\begin{align*}\n\t\\frac{1}{\\eta^{2}}\\|\\nabla L_\\eta(z) - \\nabla L_\\eta(\\bar z)\\|^2\n\t&= \\left\\|\\begin{bmatrix} x-x_+ \\\\ y_+-y \\end{bmatrix} - \\begin{bmatrix} \\bar x-\\bar x_+ \\\\ \\bar y_+-\\bar y \\end{bmatrix}\\right\\|^2\\\\\n\t&=\\left\\|\\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}\\right\\|^2 + 2 \\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}^T\\begin{bmatrix} \\bar x_+- x_+ \\\\ \\bar y_+- y_+ \\end{bmatrix} +\\left\\|\\begin{bmatrix} \\bar x_+- x_+ \\\\ \\bar y_+- y_+ \\end{bmatrix}\\right\\|^2\\\\\n\t&\\leq \\left\\|\\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}\\right\\|^2 + \\left(\\frac{\\eta}{\\eta-\\rho}-2\\right)\\begin{bmatrix} \\bar x- x \\\\ \\bar y- y \\end{bmatrix}^T\\begin{bmatrix} \\bar x_+- x_+ \\\\ \\bar y_+- y_+ \\end{bmatrix}.\n\t\\end{align*}\n\tNotice that $\\begin{bmatrix} \\bar x- x \\\\ \\bar y- y \\end{bmatrix}^T\\begin{bmatrix} \\bar x_+- x_+ \\\\ \\bar y_+- y_+ \\end{bmatrix}$ is non-negative but the sign of $\\left(\\frac{\\eta}{\\eta-\\rho}-2\\right)$ may be positive or negative. If this coefficient is negative, we can upperbound the second term above by zero, giving $\t\\|\\nabla L_\\eta(z) - \\nabla L_\\eta(\\bar z)\\|^2 \\leq \\eta^2\\left\\|z-\\bar z\\right\\|^2 .$\n\tIf instead $\\left(\\frac{\\eta}{\\eta-\\rho}-2\\right)\\geq 0$, then we have smoothness constant $|\\eta^{-1}-\\rho^{-1}|^{-1}$ as\n\t\\begin{align*}\n\t\\|\\nabla L_\\eta(z) - \\nabla L_\\eta(\\bar z)\\|^2 &\\leq \\eta^2\\left(1 + \\left(\\frac{\\eta}{\\eta-\\rho}-2\\right)\\frac{\\eta}{\\eta-\\rho}\\right)\\left\\|\\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}\\right\\|^2 \\\\\n\t&= \\eta^2\\left(\\frac{\\eta}{\\eta-\\rho}-1\\right)^2\\left\\|\\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}\\right\\|^2 \\\\\n\t&= \\left(\\frac{\\eta\\rho}{\\eta-\\rho}\\right)^2\\left\\|\\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}\\right\\|^2 \\ .\n\t\\end{align*}\n\tby Cauchy–Schwarz and using that $\\left\\|\\begin{bmatrix} \\bar x_+- x_+ \\\\ \\bar y_+- y_+ \\end{bmatrix}\\right\\| \\leq \\frac{\\eta}{\\eta-\\rho}\\left\\|\\begin{bmatrix} \\bar x- x \\\\ \\bar y- y \\end{bmatrix}\\right\\|$.\n\\end{proof}","sha256_utf8":"512a2da1fbabc84387c55173820015ca65d201389dd4d5bd0e8072fd1f303e47"},{"start":52282,"end":53595,"start_line":441,"end_line":450,"raw_tex":"\\subsection{Proof of Theorem~\\ref{thm:two-sided-dominance}}\nPropositions~\\ref{prop:hessians} and~\\ref{prop:smoothness} show that $L_\\eta$ is $\\mu=(\\eta^{-1}+\\alpha^{-1})^{-1}$-strongly convex-strongly concave and has a $\\beta=\\max\\{\\eta, |\\eta^{-1}-\\rho^{-1}|^{-1}\\}$-Lipschitz gradient. Having strong convexity and strong concavity ensures $L_\\eta$ has a unique stationary point $(x^*,y^*)$, which in turn must be the unique stationary point of $L$ by Corollary~\\ref{cor:stationarity}. \nRecall Corollary~\\ref{cor:PPM-to-GDA} showed that the damped PPM~\\eqref{eq:saddle-PPM} on $L$ is equivalent to GDA~\\eqref{eq:GDA} with $s=\\lambda/\\eta$ on $L_\\eta$. Then provided \n$$ \\lambda \\leq 2\\frac{\\min\\left\\{1,(\\eta/\\rho-1)^2\\right\\}}{\\eta/\\alpha+1} = \\frac{2(\\eta^{-1}+\\alpha^{-1})^{-1}}{\\max\\{\\eta^2, (\\eta^{-1}-\\rho^{-1})^{-2}\\}}\\ , $$\nwe have $s=\\lambda/\\eta\\in(0,2\\mu/\\beta^2)$. Hence applying Theorem~\\ref{thm:standard-contraction} shows the iterations of GDA (and consequently PPM) linearly converge to this unique stationary point as\n\\begin{align*}\n\\left\\|\\begin{bmatrix} x_k-x^* \\\\ y_k-y^* \\end{bmatrix}\\right\\|^2 &\\leq \\left(1 - \\frac{2\\lambda}{\\eta/\\alpha+1} + \\frac{\\lambda^2}{\\min\\left\\{1,(\\eta/\\rho-1)^2\\right\\}}\\right)^{k}\\left\\|\\begin{bmatrix} x_0-x^* \\\\ y_0-y^* \\end{bmatrix}\\right\\|^2.\n\\end{align*}\n\n\n","sha256_utf8":"7aa67100e8641e9e661608fbf84a80ca69474237da273665b53f1fd74b7d5fac"},{"start":96330,"end":97641,"start_line":895,"end_line":913,"raw_tex":"\\subsection{Proof of Lemma~\\ref{lem:helper0}}\nObserve that the assumed strong convexity and strong concavity ensures\n\\begin{align*}\nM(x',y') \\leq &M(x,y') -\\nabla_x M(x',y')^T(x-x')- \\frac{\\mu}{2}\\|x-x'\\|^2\\\\\n&\\leq M(x,y) + \\nabla_y M(x,y)^T(y'-y) -\\nabla_x M(x',y')^T(x-x') - \\frac{\\mu}{2}\\|y-y'\\|^2 - \\frac{\\mu}{2}\\|x-x'\\|^2\n\\end{align*}\nand symmetrically,\n\\begin{align*}\nM(x',y') &\\geq M(x',y) - \\nabla_y M(x',y')^T(y-y') + \\frac{\\mu}{2}\\|y-y^*\\|^2\\\\\n&\\geq M(x,y) + \\nabla_x M(x,y)^T(x'-x) - \\nabla_y M(x',y')^T(y-y') + \\frac{\\mu}{2}\\|x-x'\\|^2 + \\frac{\\mu}{2}\\|y-y'\\|^2.\n\\end{align*}\nCombining the above two inequalities gives the first claimed inequality\n\\begin{equation*}\n\\mu\\left\\|\\begin{bmatrix} x-x' \\\\ y-y' \\end{bmatrix}\\right\\|^2 \\leq \\left(\\begin{bmatrix} \\nabla_x M(x,y)  \\\\-\\nabla_y M(x,y) \\end{bmatrix} -\\begin{bmatrix} \\nabla_x M(x',y')  \\\\-\\nabla_y M(x',y') \\end{bmatrix}\\right)^T\\begin{bmatrix} x-x' \\\\ y-y' \\end{bmatrix}.\n\\end{equation*}\nFurthermore, when $\\nabla M(x',y')=0$, Cauchy–Schwarz gives the second inequality.\n\n\\subsection{Proof of Theorem~\\ref{thm:standard-contraction}}\nFirst we use Lemma~\\ref{lem:helper0} to conclude that if the set $S$ is large enough, $M$ must have a stationary point in $S$. Define $B(z,r)=\\{z'|\\|z-z'\\|\\le r\\}$ as the ball centered as $a$ with radius $r$. \n","sha256_utf8":"bff0c4ca8a203372f109b45d6737141588c372bcc3605a021b8c91899f6851a2"},{"start":97642,"end":97895,"start_line":915,"end_line":917,"raw_tex":"\\begin{lemma} \\label{lem:helper1}\n\tSuppose $M$ is $\\mu$-strongly convex-strongly concave in a set $B(x,r)\\times B(y,r)$ for some fixed $(x,y)$ and $r\\geq 2\\|\\nabla M(x,y)\\|/\\mu$, then there exists a stationary point of $M$ in $B((x,y),r/2)$.\n\\end{lemma}","sha256_utf8":"d18f87f500b6e951cc46049c79b1279e70474860686c72871941b6301f6d8002"},{"start":97896,"end":99691,"start_line":918,"end_line":937,"raw_tex":"\\begin{proof}\n\tConsider the following constrained minimax problem $ \\min_{x'\\in B(x,r)} \\max_{y'\\in B(y,r)} M(x,y).$\n\tSince $M(x,y)$ is strongly convex-strongly concave, it must have a unique solution $(x^*,y^*)$.\n\tThe first-order optimality condition for $(x^*,y^*)$ ensures\n\t$$ \\nabla_x M(x^*,y^*) =-\\lambda(x^*-x) \\ \\ \\ \\text{ and } \\ \\ \\  -\\nabla_y M(x^*,y^*) =-\\gamma(y^*-y)$$\n\tfor some constants $\\lambda,\\gamma\\geq 0$ that are nonzero only if $x^*$ or $y^*$ are on the boundary of $B(x,r)$ and $B(y,r)$ respectively. Taking an inner product with $(x^*-x,y^*-y)$ gives\n\t\\begin{equation}\\label{eq:lemA2.1}\n\t\\begin{bmatrix} \\nabla_x M(x^*,y^*)  \\\\-\\nabla_y M(x^*,y^*) \\end{bmatrix}^T\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix} =-\\left\\|\\begin{bmatrix}\\sqrt{\\lambda}(x^*-x)  \\\\ \\sqrt{\\gamma}(y^*-y) \\end{bmatrix}\\right\\|^2\\leq0.\n\t\\end{equation}\n\tApplying Lemma~\\ref{lem:helper0} and utilizing \\eqref{eq:lemA2.1}, we conclude that\n\t\\begin{equation}\\label{eq:lemA2.2}\n\t\\mu\\left\\|\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix}\\right\\|^2 + \\begin{bmatrix} \\nabla_x M(x,y)  \\\\-\\nabla_y M(x,y) \\end{bmatrix}^T\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix}\\leq 0. \n\t\\end{equation}\n\tHence\n\t$$ \\left\\|\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix}\\right\\|^2 \\leq \\frac{1}{\\mu}\\left\\|\\begin{bmatrix} \\nabla_x M(x,y)  \\\\-\\nabla_y M(x,y) \\end{bmatrix}\\right\\|\\left\\|\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix}\\right\\|\\ ,$$\n\twhereby\n\t$$ \\left\\|\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix}\\right\\| \\leq \\frac{1}{\\mu}\\left\\|\\begin{bmatrix} \\nabla_x M(x,y)  \\\\-\\nabla_y M(x,y) \\end{bmatrix}\\right\\| \\leq r/2\\ ,$$\n\tutilizing our assumed condition on $r$.\n\tSince $(x^*,y^*)$ lies strictly inside $B((x,y),r)$, the first-order optimality condition implies $(x^*,y^*)$ is a stationary point of $M$.\n\\end{proof}","sha256_utf8":"0bf33ab68efa66f24971db7658c46e12544de3159c6f74e4396e0e692601dec7"},{"start":99693,"end":100791,"start_line":939,"end_line":946,"raw_tex":"Lemma~\\ref{lem:helper1} ensures the existence of a nearby stationary point $(x^*,y^*)$. Then the standard proof for strongly monotone (from Lemma~\\ref{lem:helper0}) and Lipschitz operators gives a contraction whenever $s\\in(0, 2\\mu/\\beta^2) $:\n\\begin{align*}\n\\left\\|\\begin{bmatrix} x_{k+1}-x^* \\\\ y_{k+1}-y^* \\end{bmatrix}\\right\\|^2\n&= \\left\\|\\begin{bmatrix} x_{k}-x^* \\\\ y_{k}-y^* \\end{bmatrix}\\right\\|^2 - 2s\\begin{bmatrix} \\nabla_x M(x_k,y_k) \\\\ -\\nabla_y M(x_k,y_k) \\end{bmatrix}^T\\begin{bmatrix} x_k-x^* \\\\ y_k-y^* \\end{bmatrix} + s^2\\left\\|\\begin{bmatrix} \\nabla_x M(x_k,y_k) \\\\ -\\nabla_y M(x_k,y_k) \\end{bmatrix}\\right\\|^2\\\\\n&\\leq \\left\\|\\begin{bmatrix} x_{k}-x^* \\\\ y_{k}-y^* \\end{bmatrix}\\right\\|^2 - 2\\mu s\\left\\|\\begin{bmatrix} x_{k}-x^* \\\\ y_{k}-y^* \\end{bmatrix}\\right\\|^2 + \\beta^2s^2\\left\\|\\begin{bmatrix} x_{k}-x^* \\\\ y_{k}-y^* \\end{bmatrix}\\right\\|^2\\\\\n&= \\left(1 - 2\\mu s+\\beta^2s^2\\right)\\left\\|\\begin{bmatrix} x_{k}-x^* \\\\ y_{k}-y^* \\end{bmatrix}\\right\\|^2 \\ , \n\\end{align*}\nwhere the inequality utilizes \\eqref{eq:lemA2.2} at $(x,y)=(x_k,y_k)$ and the smoothness of $M(x,y)$. \n","sha256_utf8":"25da88de233013469288ee796d84180986c08367b71669dc34b3341ab0800320"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"强交互下的全局线性收敛","source_units":[{"start":47503,"end":48490,"start_line":409,"end_line":416,"raw_tex":"\\begin{theorem}\\label{thm:two-sided-dominance}\n\tFor any objective $L$ that is $\\rho$-weakly convex-weakly concave and $\\alpha > 0$-interaction dominant in both $x$ and $y$, the damped PPM~\\eqref{eq:saddle-PPM} with $\\eta$ and $\\lambda$ satisfying\n\t$$ \\lambda \\leq 2\\frac{\\min\\left\\{1,(\\eta/\\rho-1)^2\\right\\}}{\\eta/\\alpha+1} $$\n\tlinearly converges to the unique stationary point $(x^*,y^*)$ of~\\eqref{eq:main-problem} with\n\t$$\\left\\|\\begin{bmatrix} x_k-x^* \\\\ y_k-y^* \\end{bmatrix}\\right\\|^2 \\leq \\left(1 - \\frac{2\\lambda}{\\eta/\\alpha+1} + \\frac{\\lambda^2}{\\min\\left\\{1,(\\eta/\\rho-1)^2\\right\\}}\\right)^{k}\\left\\|\\begin{bmatrix} x_0-x^* \\\\ y_0-y^* \\end{bmatrix}\\right\\|^2.$$\n\tFor example, setting $\\eta=2\\rho$ and $\\lambda=\\frac{1}{1+\\eta/\\alpha} $, our convergence rate simplifies to\n\t$$\\left\\|\\begin{bmatrix} x_k-x^* \\\\ y_k-y^* \\end{bmatrix}\\right\\|^2 \\leq \\left(1-\\frac{1}{(2\\rho/\\alpha+1)^2} \\right)^{k}\\left\\|\\begin{bmatrix} x_0-x^* \\\\ y_0-y^* \\end{bmatrix}\\right\\|^2.$$\n\\end{theorem}","sha256_utf8":"af98423c166d20840d37117cfa47a8889466012590417c5b478fe8cbb32be043"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"全局收敛定理证明","source_units":[{"start":52282,"end":53595,"start_line":441,"end_line":450,"raw_tex":"\\subsection{Proof of Theorem~\\ref{thm:two-sided-dominance}}\nPropositions~\\ref{prop:hessians} and~\\ref{prop:smoothness} show that $L_\\eta$ is $\\mu=(\\eta^{-1}+\\alpha^{-1})^{-1}$-strongly convex-strongly concave and has a $\\beta=\\max\\{\\eta, |\\eta^{-1}-\\rho^{-1}|^{-1}\\}$-Lipschitz gradient. Having strong convexity and strong concavity ensures $L_\\eta$ has a unique stationary point $(x^*,y^*)$, which in turn must be the unique stationary point of $L$ by Corollary~\\ref{cor:stationarity}. \nRecall Corollary~\\ref{cor:PPM-to-GDA} showed that the damped PPM~\\eqref{eq:saddle-PPM} on $L$ is equivalent to GDA~\\eqref{eq:GDA} with $s=\\lambda/\\eta$ on $L_\\eta$. Then provided \n$$ \\lambda \\leq 2\\frac{\\min\\left\\{1,(\\eta/\\rho-1)^2\\right\\}}{\\eta/\\alpha+1} = \\frac{2(\\eta^{-1}+\\alpha^{-1})^{-1}}{\\max\\{\\eta^2, (\\eta^{-1}-\\rho^{-1})^{-2}\\}}\\ , $$\nwe have $s=\\lambda/\\eta\\in(0,2\\mu/\\beta^2)$. Hence applying Theorem~\\ref{thm:standard-contraction} shows the iterations of GDA (and consequently PPM) linearly converge to this unique stationary point as\n\\begin{align*}\n\\left\\|\\begin{bmatrix} x_k-x^* \\\\ y_k-y^* \\end{bmatrix}\\right\\|^2 &\\leq \\left(1 - \\frac{2\\lambda}{\\eta/\\alpha+1} + \\frac{\\lambda^2}{\\min\\left\\{1,(\\eta/\\rho-1)^2\\right\\}}\\right)^{k}\\left\\|\\begin{bmatrix} x_0-x^* \\\\ y_0-y^* \\end{bmatrix}\\right\\|^2.\n\\end{align*}\n\n\n","sha256_utf8":"7aa67100e8641e9e661608fbf84a80ca69474237da273665b53f1fd74b7d5fac"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"包络正则性与唯一驻点","source_units":[{"start":52342,"end":52770,"start_line":442,"end_line":442,"raw_tex":"Propositions~\\ref{prop:hessians} and~\\ref{prop:smoothness} show that $L_\\eta$ is $\\mu=(\\eta^{-1}+\\alpha^{-1})^{-1}$-strongly convex-strongly concave and has a $\\beta=\\max\\{\\eta, |\\eta^{-1}-\\rho^{-1}|^{-1}\\}$-Lipschitz gradient. Having strong convexity and strong concavity ensures $L_\\eta$ has a unique stationary point $(x^*,y^*)$, which in turn must be the unique stationary point of $L$ by Corollary~\\ref{cor:stationarity}. \n","sha256_utf8":"0d20424cfa75580cefc7067312d827264f6502625e4d13d7bb4bd1561c16f253"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"迭代等价与阻尼条件","source_units":[{"start":52770,"end":53115,"start_line":443,"end_line":444,"raw_tex":"Recall Corollary~\\ref{cor:PPM-to-GDA} showed that the damped PPM~\\eqref{eq:saddle-PPM} on $L$ is equivalent to GDA~\\eqref{eq:GDA} with $s=\\lambda/\\eta$ on $L_\\eta$. Then provided \n$$ \\lambda \\leq 2\\frac{\\min\\left\\{1,(\\eta/\\rho-1)^2\\right\\}}{\\eta/\\alpha+1} = \\frac{2(\\eta^{-1}+\\alpha^{-1})^{-1}}{\\max\\{\\eta^2, (\\eta^{-1}-\\rho^{-1})^{-2}\\}}\\ , $$\n","sha256_utf8":"db6390c9824df3124da57328bfd5581c296a724daa1342eb880c0fc760b25b20"}]},{"id":"S005","semantic_type":"proof_step","kind":"proof_step","title":"应用收缩定理得到速率","source_units":[{"start":53115,"end":53593,"start_line":445,"end_line":448,"raw_tex":"we have $s=\\lambda/\\eta\\in(0,2\\mu/\\beta^2)$. Hence applying Theorem~\\ref{thm:standard-contraction} shows the iterations of GDA (and consequently PPM) linearly converge to this unique stationary point as\n\\begin{align*}\n\\left\\|\\begin{bmatrix} x_k-x^* \\\\ y_k-y^* \\end{bmatrix}\\right\\|^2 &\\leq \\left(1 - \\frac{2\\lambda}{\\eta/\\alpha+1} + \\frac{\\lambda^2}{\\min\\left\\{1,(\\eta/\\rho-1)^2\\right\\}}\\right)^{k}\\left\\|\\begin{bmatrix} x_0-x^* \\\\ y_0-y^* \\end{bmatrix}\\right\\|^2.\n\\end{align*}\n","sha256_utf8":"8aaa42666766ac433af8030c0ca40759369ac7f2942ed7dff99646717059dcf6"}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","title":"主定理的交互与参数假设","source_units":[{"start":47550,"end":47830,"start_line":410,"end_line":411,"raw_tex":"\tFor any objective $L$ that is $\\rho$-weakly convex-weakly concave and $\\alpha > 0$-interaction dominant in both $x$ and $y$, the damped PPM~\\eqref{eq:saddle-PPM} with $\\eta$ and $\\lambda$ satisfying\n\t$$ \\lambda \\leq 2\\frac{\\min\\left\\{1,(\\eta/\\rho-1)^2\\right\\}}{\\eta/\\alpha+1} $$\n","sha256_utf8":"5b072e43a11992681e32b82d6bced62ae9818969b45e0a663047be0d8a266188"}]},{"id":"S007","semantic_type":"lemma","kind":"lemma","title":"鞍点梯度的强单调性","source_units":[{"start":21172,"end":21816,"start_line":176,"end_line":183,"raw_tex":"\\begin{lemma} \\label{lem:helper0}\n\tSuppose $M(x,y)$ is $\\mu$-strongly convex-strongly concave on a convex set $S=S_x\\times S_y$, then it holds for any $(x,y),(x',y') \\in S$ that\n\t$$\\mu\\left\\|\\begin{bmatrix} x-x' \\\\ y-y' \\end{bmatrix}\\right\\|^2 \\leq \\left(\\begin{bmatrix} \\nabla_x M(x,y)  \\\\-\\nabla_y M(x,y) \\end{bmatrix} -\\begin{bmatrix} \\nabla_x M(x',y')  \\\\-\\nabla_y M(x',y') \\end{bmatrix}\\right)^T\\begin{bmatrix} x-x' \\\\ y-y' \\end{bmatrix}.$$\n\tWhen $\\nabla M(x', y')=0$, the distance to this stationary point is bounded by\n\t$$\n\t\\left\\|\\begin{bmatrix} x-x' \\\\ y-y' \\end{bmatrix}\\right\\| \\leq \\frac{\\|\\nabla M(x,y)\\|}{\\mu} \\ . \n\t$$\n\\end{lemma}","sha256_utf8":"f99310ccda8c0be5e3b62da3b0bd43461992de4813216f44e8cf16749be67464"}]},{"id":"S008","semantic_type":"proof","kind":"proof","title":"强单调性引理证明","source_units":[{"start":96330,"end":97370,"start_line":895,"end_line":911,"raw_tex":"\\subsection{Proof of Lemma~\\ref{lem:helper0}}\nObserve that the assumed strong convexity and strong concavity ensures\n\\begin{align*}\nM(x',y') \\leq &M(x,y') -\\nabla_x M(x',y')^T(x-x')- \\frac{\\mu}{2}\\|x-x'\\|^2\\\\\n&\\leq M(x,y) + \\nabla_y M(x,y)^T(y'-y) -\\nabla_x M(x',y')^T(x-x') - \\frac{\\mu}{2}\\|y-y'\\|^2 - \\frac{\\mu}{2}\\|x-x'\\|^2\n\\end{align*}\nand symmetrically,\n\\begin{align*}\nM(x',y') &\\geq M(x',y) - \\nabla_y M(x',y')^T(y-y') + \\frac{\\mu}{2}\\|y-y^*\\|^2\\\\\n&\\geq M(x,y) + \\nabla_x M(x,y)^T(x'-x) - \\nabla_y M(x',y')^T(y-y') + \\frac{\\mu}{2}\\|x-x'\\|^2 + \\frac{\\mu}{2}\\|y-y'\\|^2.\n\\end{align*}\nCombining the above two inequalities gives the first claimed inequality\n\\begin{equation*}\n\\mu\\left\\|\\begin{bmatrix} x-x' \\\\ y-y' \\end{bmatrix}\\right\\|^2 \\leq \\left(\\begin{bmatrix} \\nabla_x M(x,y)  \\\\-\\nabla_y M(x,y) \\end{bmatrix} -\\begin{bmatrix} \\nabla_x M(x',y')  \\\\-\\nabla_y M(x',y') \\end{bmatrix}\\right)^T\\begin{bmatrix} x-x' \\\\ y-y' \\end{bmatrix}.\n\\end{equation*}\nFurthermore, when $\\nabla M(x',y')=0$, Cauchy–Schwarz gives the second inequality.\n\n","sha256_utf8":"2176134fd800b9974a522999bff1c16b0c591f087701f300d7fd3b793eaf500f"}]},{"id":"S009","semantic_type":"proof_step","kind":"proof_step","title":"强凸强凹给出上界","source_units":[{"start":96376,"end":96670,"start_line":896,"end_line":900,"raw_tex":"Observe that the assumed strong convexity and strong concavity ensures\n\\begin{align*}\nM(x',y') \\leq &M(x,y') -\\nabla_x M(x',y')^T(x-x')- \\frac{\\mu}{2}\\|x-x'\\|^2\\\\\n&\\leq M(x,y) + \\nabla_y M(x,y)^T(y'-y) -\\nabla_x M(x',y')^T(x-x') - \\frac{\\mu}{2}\\|y-y'\\|^2 - \\frac{\\mu}{2}\\|x-x'\\|^2\n\\end{align*}\n","sha256_utf8":"271c5d2628b646277908b5fead75c881d1668090bb6a392809ce36fccaea56a2"}]},{"id":"S010","semantic_type":"proof_step","kind":"proof_step","title":"对称地建立下界","source_units":[{"start":96670,"end":96917,"start_line":901,"end_line":905,"raw_tex":"and symmetrically,\n\\begin{align*}\nM(x',y') &\\geq M(x',y) - \\nabla_y M(x',y')^T(y-y') + \\frac{\\mu}{2}\\|y-y^*\\|^2\\\\\n&\\geq M(x,y) + \\nabla_x M(x,y)^T(x'-x) - \\nabla_y M(x',y')^T(y-y') + \\frac{\\mu}{2}\\|x-x'\\|^2 + \\frac{\\mu}{2}\\|y-y'\\|^2.\n\\end{align*}\n","sha256_utf8":"796927ef49a4bfde6bc80f8d4b5b60ab020ea9c4d5595560a3dc83d43f83e1b0"}]},{"id":"S011","semantic_type":"proof_step","kind":"proof_step","title":"合并两界得到强单调性","source_units":[{"start":96917,"end":97286,"start_line":906,"end_line":909,"raw_tex":"Combining the above two inequalities gives the first claimed inequality\n\\begin{equation*}\n\\mu\\left\\|\\begin{bmatrix} x-x' \\\\ y-y' \\end{bmatrix}\\right\\|^2 \\leq \\left(\\begin{bmatrix} \\nabla_x M(x,y)  \\\\-\\nabla_y M(x,y) \\end{bmatrix} -\\begin{bmatrix} \\nabla_x M(x',y')  \\\\-\\nabla_y M(x',y') \\end{bmatrix}\\right)^T\\begin{bmatrix} x-x' \\\\ y-y' \\end{bmatrix}.\n\\end{equation*}\n","sha256_utf8":"5e22401db7154e270299fbc4f124964c8ca1112ea1bb72aded2a84bace4a314f"}]},{"id":"S012","semantic_type":"proof_step","kind":"proof_step","title":"推出驻点距离界","source_units":[{"start":97286,"end":97369,"start_line":910,"end_line":910,"raw_tex":"Furthermore, when $\\nabla M(x',y')=0$, Cauchy–Schwarz gives the second inequality.\n","sha256_utf8":"c1d8d73a17ee7033e9c3131c7079a0a4587333181314c6e02a25d572ff902dbe"}]},{"id":"S013","semantic_type":"theorem","kind":"theorem","title":"局部强凸强凹问题的收缩","source_units":[{"start":21817,"end":22427,"start_line":184,"end_line":187,"raw_tex":"\\begin{theorem} \\label{thm:standard-contraction}\n\tConsider any minimax problem $\\min_{x\\in\\RR^n}\\max_{y\\in\\RR^m} M(x,y)$ where $M(x,y)$ is $\\beta$-smooth and $\\mu$-strongly convex-strongly concave on a set $B(x_0,r)\\times B(y_0,r)$ with $r\\geq 2\\|\\nabla M(x_0,y_0)\\|/\\mu$. Then GDA~\\eqref{eq:GDA} with step-size $s\\in(0, 2\\mu/\\beta^2) $ linearly converges to a stationary point $(x^*,y^*)\\in B((x_0,y_0),r/2)$ with\n\t$$ \\left\\|\\begin{bmatrix} x_k-x^* \\\\ y_k-y^* \\end{bmatrix}\\right\\|^2 \\leq \\left(1 - 2\\mu s+\\beta^2s^2\\right)^{k}\\left\\|\\begin{bmatrix} x_0-x^* \\\\ y_0-y^* \\end{bmatrix}\\right\\|^2.$$\n\\end{theorem}","sha256_utf8":"78f02cad031cbe56e06e92b0900742c90951ee162f42ff173c18ff20d8fb8a7c"}]},{"id":"S014","semantic_type":"proof","kind":"proof","title":"标准收缩定理完整证明","source_units":[{"start":97370,"end":97641,"start_line":912,"end_line":913,"raw_tex":"\\subsection{Proof of Theorem~\\ref{thm:standard-contraction}}\nFirst we use Lemma~\\ref{lem:helper0} to conclude that if the set $S$ is large enough, $M$ must have a stationary point in $S$. Define $B(z,r)=\\{z'|\\|z-z'\\|\\le r\\}$ as the ball centered as $a$ with radius $r$. \n","sha256_utf8":"bbfa3eeeb85092ac00a459faa78207237cf32e7a8a162cc103d49e2a688ee158"},{"start":99693,"end":100791,"start_line":939,"end_line":946,"raw_tex":"Lemma~\\ref{lem:helper1} ensures the existence of a nearby stationary point $(x^*,y^*)$. Then the standard proof for strongly monotone (from Lemma~\\ref{lem:helper0}) and Lipschitz operators gives a contraction whenever $s\\in(0, 2\\mu/\\beta^2) $:\n\\begin{align*}\n\\left\\|\\begin{bmatrix} x_{k+1}-x^* \\\\ y_{k+1}-y^* \\end{bmatrix}\\right\\|^2\n&= \\left\\|\\begin{bmatrix} x_{k}-x^* \\\\ y_{k}-y^* \\end{bmatrix}\\right\\|^2 - 2s\\begin{bmatrix} \\nabla_x M(x_k,y_k) \\\\ -\\nabla_y M(x_k,y_k) \\end{bmatrix}^T\\begin{bmatrix} x_k-x^* \\\\ y_k-y^* \\end{bmatrix} + s^2\\left\\|\\begin{bmatrix} \\nabla_x M(x_k,y_k) \\\\ -\\nabla_y M(x_k,y_k) \\end{bmatrix}\\right\\|^2\\\\\n&\\leq \\left\\|\\begin{bmatrix} x_{k}-x^* \\\\ y_{k}-y^* \\end{bmatrix}\\right\\|^2 - 2\\mu s\\left\\|\\begin{bmatrix} x_{k}-x^* \\\\ y_{k}-y^* \\end{bmatrix}\\right\\|^2 + \\beta^2s^2\\left\\|\\begin{bmatrix} x_{k}-x^* \\\\ y_{k}-y^* \\end{bmatrix}\\right\\|^2\\\\\n&= \\left(1 - 2\\mu s+\\beta^2s^2\\right)\\left\\|\\begin{bmatrix} x_{k}-x^* \\\\ y_{k}-y^* \\end{bmatrix}\\right\\|^2 \\ , \n\\end{align*}\nwhere the inequality utilizes \\eqref{eq:lemA2.2} at $(x,y)=(x_k,y_k)$ and the smoothness of $M(x,y)$. \n","sha256_utf8":"25da88de233013469288ee796d84180986c08367b71669dc34b3341ab0800320"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"驻点存在性的准备","source_units":[{"start":97431,"end":97641,"start_line":913,"end_line":913,"raw_tex":"First we use Lemma~\\ref{lem:helper0} to conclude that if the set $S$ is large enough, $M$ must have a stationary point in $S$. Define $B(z,r)=\\{z'|\\|z-z'\\|\\le r\\}$ as the ball centered as $a$ with radius $r$. \n","sha256_utf8":"2f35787b0516948801d0d1b1da42c152aa8fc7a959f5fb1a1d0d0a4784df645d"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"展开迭代误差并收缩","source_units":[{"start":99693,"end":100688,"start_line":939,"end_line":945,"raw_tex":"Lemma~\\ref{lem:helper1} ensures the existence of a nearby stationary point $(x^*,y^*)$. Then the standard proof for strongly monotone (from Lemma~\\ref{lem:helper0}) and Lipschitz operators gives a contraction whenever $s\\in(0, 2\\mu/\\beta^2) $:\n\\begin{align*}\n\\left\\|\\begin{bmatrix} x_{k+1}-x^* \\\\ y_{k+1}-y^* \\end{bmatrix}\\right\\|^2\n&= \\left\\|\\begin{bmatrix} x_{k}-x^* \\\\ y_{k}-y^* \\end{bmatrix}\\right\\|^2 - 2s\\begin{bmatrix} \\nabla_x M(x_k,y_k) \\\\ -\\nabla_y M(x_k,y_k) \\end{bmatrix}^T\\begin{bmatrix} x_k-x^* \\\\ y_k-y^* \\end{bmatrix} + s^2\\left\\|\\begin{bmatrix} \\nabla_x M(x_k,y_k) \\\\ -\\nabla_y M(x_k,y_k) \\end{bmatrix}\\right\\|^2\\\\\n&\\leq \\left\\|\\begin{bmatrix} x_{k}-x^* \\\\ y_{k}-y^* \\end{bmatrix}\\right\\|^2 - 2\\mu s\\left\\|\\begin{bmatrix} x_{k}-x^* \\\\ y_{k}-y^* \\end{bmatrix}\\right\\|^2 + \\beta^2s^2\\left\\|\\begin{bmatrix} x_{k}-x^* \\\\ y_{k}-y^* \\end{bmatrix}\\right\\|^2\\\\\n&= \\left(1 - 2\\mu s+\\beta^2s^2\\right)\\left\\|\\begin{bmatrix} x_{k}-x^* \\\\ y_{k}-y^* \\end{bmatrix}\\right\\|^2 \\ , \n\\end{align*}\n","sha256_utf8":"8126df62342fea337f44ac92d06111e1736e1e3ce29d1102e528b363cdc120e4"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"说明误差不等式的依据","source_units":[{"start":100688,"end":100791,"start_line":946,"end_line":946,"raw_tex":"where the inequality utilizes \\eqref{eq:lemA2.2} at $(x,y)=(x_k,y_k)$ and the smoothness of $M(x,y)$. \n","sha256_utf8":"ed6e6e793c8ce471d03d7bb37ceb21f996d8526ebdd4db64683c0c8c715d1544"}]},{"id":"S018","semantic_type":"lemma","kind":"lemma","title":"局部驻点存在性","source_units":[{"start":97642,"end":97895,"start_line":915,"end_line":917,"raw_tex":"\\begin{lemma} \\label{lem:helper1}\n\tSuppose $M$ is $\\mu$-strongly convex-strongly concave in a set $B(x,r)\\times B(y,r)$ for some fixed $(x,y)$ and $r\\geq 2\\|\\nabla M(x,y)\\|/\\mu$, then there exists a stationary point of $M$ in $B((x,y),r/2)$.\n\\end{lemma}","sha256_utf8":"d18f87f500b6e951cc46049c79b1279e70474860686c72871941b6301f6d8002"}]},{"id":"S019","semantic_type":"proof","kind":"proof","title":"局部驻点存在性证明","source_units":[{"start":97896,"end":99691,"start_line":918,"end_line":937,"raw_tex":"\\begin{proof}\n\tConsider the following constrained minimax problem $ \\min_{x'\\in B(x,r)} \\max_{y'\\in B(y,r)} M(x,y).$\n\tSince $M(x,y)$ is strongly convex-strongly concave, it must have a unique solution $(x^*,y^*)$.\n\tThe first-order optimality condition for $(x^*,y^*)$ ensures\n\t$$ \\nabla_x M(x^*,y^*) =-\\lambda(x^*-x) \\ \\ \\ \\text{ and } \\ \\ \\  -\\nabla_y M(x^*,y^*) =-\\gamma(y^*-y)$$\n\tfor some constants $\\lambda,\\gamma\\geq 0$ that are nonzero only if $x^*$ or $y^*$ are on the boundary of $B(x,r)$ and $B(y,r)$ respectively. Taking an inner product with $(x^*-x,y^*-y)$ gives\n\t\\begin{equation}\\label{eq:lemA2.1}\n\t\\begin{bmatrix} \\nabla_x M(x^*,y^*)  \\\\-\\nabla_y M(x^*,y^*) \\end{bmatrix}^T\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix} =-\\left\\|\\begin{bmatrix}\\sqrt{\\lambda}(x^*-x)  \\\\ \\sqrt{\\gamma}(y^*-y) \\end{bmatrix}\\right\\|^2\\leq0.\n\t\\end{equation}\n\tApplying Lemma~\\ref{lem:helper0} and utilizing \\eqref{eq:lemA2.1}, we conclude that\n\t\\begin{equation}\\label{eq:lemA2.2}\n\t\\mu\\left\\|\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix}\\right\\|^2 + \\begin{bmatrix} \\nabla_x M(x,y)  \\\\-\\nabla_y M(x,y) \\end{bmatrix}^T\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix}\\leq 0. \n\t\\end{equation}\n\tHence\n\t$$ \\left\\|\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix}\\right\\|^2 \\leq \\frac{1}{\\mu}\\left\\|\\begin{bmatrix} \\nabla_x M(x,y)  \\\\-\\nabla_y M(x,y) \\end{bmatrix}\\right\\|\\left\\|\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix}\\right\\|\\ ,$$\n\twhereby\n\t$$ \\left\\|\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix}\\right\\| \\leq \\frac{1}{\\mu}\\left\\|\\begin{bmatrix} \\nabla_x M(x,y)  \\\\-\\nabla_y M(x,y) \\end{bmatrix}\\right\\| \\leq r/2\\ ,$$\n\tutilizing our assumed condition on $r$.\n\tSince $(x^*,y^*)$ lies strictly inside $B((x,y),r)$, the first-order optimality condition implies $(x^*,y^*)$ is a stationary point of $M$.\n\\end{proof}","sha256_utf8":"0bf33ab68efa66f24971db7658c46e12544de3159c6f74e4396e0e692601dec7"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"约束鞍点与法锥条件","source_units":[{"start":97910,"end":98471,"start_line":919,"end_line":923,"raw_tex":"\tConsider the following constrained minimax problem $ \\min_{x'\\in B(x,r)} \\max_{y'\\in B(y,r)} M(x,y).$\n\tSince $M(x,y)$ is strongly convex-strongly concave, it must have a unique solution $(x^*,y^*)$.\n\tThe first-order optimality condition for $(x^*,y^*)$ ensures\n\t$$ \\nabla_x M(x^*,y^*) =-\\lambda(x^*-x) \\ \\ \\ \\text{ and } \\ \\ \\  -\\nabla_y M(x^*,y^*) =-\\gamma(y^*-y)$$\n\tfor some constants $\\lambda,\\gamma\\geq 0$ that are nonzero only if $x^*$ or $y^*$ are on the boundary of $B(x,r)$ and $B(y,r)$ respectively. Taking an inner product with $(x^*-x,y^*-y)$ gives\n","sha256_utf8":"ac26cf602dd49e7aa5f9efa1b37ecf1eb82e9545a026ded90b4ac028c30499d7"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"最优性条件的内积符号","source_units":[{"start":98471,"end":98746,"start_line":924,"end_line":926,"raw_tex":"\t\\begin{equation}\\label{eq:lemA2.1}\n\t\\begin{bmatrix} \\nabla_x M(x^*,y^*)  \\\\-\\nabla_y M(x^*,y^*) \\end{bmatrix}^T\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix} =-\\left\\|\\begin{bmatrix}\\sqrt{\\lambda}(x^*-x)  \\\\ \\sqrt{\\gamma}(y^*-y) \\end{bmatrix}\\right\\|^2\\leq0.\n\t\\end{equation}\n","sha256_utf8":"f59e944a1e4f457ff95fa8f6fe16f6802438a674bc4885dcddc5728c9e297576"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"用强单调性控制位移","source_units":[{"start":98746,"end":99073,"start_line":927,"end_line":930,"raw_tex":"\tApplying Lemma~\\ref{lem:helper0} and utilizing \\eqref{eq:lemA2.1}, we conclude that\n\t\\begin{equation}\\label{eq:lemA2.2}\n\t\\mu\\left\\|\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix}\\right\\|^2 + \\begin{bmatrix} \\nabla_x M(x,y)  \\\\-\\nabla_y M(x,y) \\end{bmatrix}^T\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix}\\leq 0. \n\t\\end{equation}\n","sha256_utf8":"b1327753506b77a58c83169002bf2113cdd611aef088ae7b383701788a5c919c"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"位移落入半径一半的球","source_units":[{"start":99073,"end":99539,"start_line":931,"end_line":935,"raw_tex":"\tHence\n\t$$ \\left\\|\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix}\\right\\|^2 \\leq \\frac{1}{\\mu}\\left\\|\\begin{bmatrix} \\nabla_x M(x,y)  \\\\-\\nabla_y M(x,y) \\end{bmatrix}\\right\\|\\left\\|\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix}\\right\\|\\ ,$$\n\twhereby\n\t$$ \\left\\|\\begin{bmatrix} x^*-x  \\\\ y^*-y \\end{bmatrix}\\right\\| \\leq \\frac{1}{\\mu}\\left\\|\\begin{bmatrix} \\nabla_x M(x,y)  \\\\-\\nabla_y M(x,y) \\end{bmatrix}\\right\\| \\leq r/2\\ ,$$\n\tutilizing our assumed condition on $r$.\n","sha256_utf8":"302b53549657b614865e8a5f2dee1d93001335c5af1be5683c0c459badf0677d"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"内点最优性得到驻点","source_units":[{"start":99539,"end":99680,"start_line":936,"end_line":936,"raw_tex":"\tSince $(x^*,y^*)$ lies strictly inside $B((x,y),r)$, the first-order optimality condition implies $(x^*,y^*)$ is a stationary point of $M$.\n","sha256_utf8":"f4915e6233fa4e2947637141bff140582655ac41a3c1217b4e7d26b7d91c7b60"}]},{"id":"S025","semantic_type":"lemma","kind":"lemma","title":"鞍点包络梯度公式","source_units":[{"start":26998,"end":27391,"start_line":235,"end_line":239,"raw_tex":"\\begin{lemma} \\label{lem:grad-formula}\n\tThe gradient of the saddle envelope $L_\\eta(x,y)$ at $z=(x,y)$ is\n\t$$ \\begin{bmatrix} \\nabla_x L_\\eta(z)\\\\  \\nabla_y L_\\eta(z)\\end{bmatrix} = \\begin{bmatrix} \\eta(x-x_+)\\\\ \\eta(y_+-y) \\end{bmatrix} = \\begin{bmatrix} \\nabla_x L(z_+)\\\\  \\nabla_y L(z_+)\\end{bmatrix}$$\n\twhere $z_+ = (x_+,y_+) = \\prox_\\eta(z)$ is given by the proximal operator.\n\\end{lemma}","sha256_utf8":"3f2af75cf3db4b9dfc0cae13f5c93b2c0d061ba4bc275b1f7935aef71780d10e"}]},{"id":"S026","semantic_type":"proof","kind":"proof","title":"包络梯度公式证明","source_units":[{"start":27392,"end":28208,"start_line":240,"end_line":250,"raw_tex":"\\begin{proof}\n\tNotice that the saddle envelope is a composition of Moreau envelopes\n\t\\begin{align*}\n\tL_{\\eta}(x,y) &= \\min_{u} \\left(\\max_{v} L(u,v) - \\frac{\\eta}{2}\\|v-y\\|^2\\right) + \\frac{\\eta}{2}\\|u-x\\|^2\\\\\n\t&= \\min_{u}\\ -e_\\eta\\{-L(u,\\cdot)\\}(y) + \\frac{\\eta}{2}\\|u-x\\|^2 = e_{\\eta}\\{g(\\cdot,y)\\}(x)\n\t\\end{align*}\n\twhere $g(u,y) = -e_{\\eta}\\{-L(u,\\cdot)\\}(y)$. Applying the gradient formula~\\eqref{eq:moreau-grad} gives our first claimed gradient formula in $x$ of\n\t$\\nabla_x L_\\eta(x,y) = \\eta (x-x_+)$\n\tsince $x_+$ is the unique minimizer of $u \\mapsto g(u,y) + \\frac{\\eta}{2}\\|u-x\\|^2$. Symmetric reasoning gives our first claimed formula $\\nabla_y L_\\eta(x,y) = \\eta (y_+-y) $ in $y$.\n\tThe second claimed equality is precisely the first-order optimality condition for~\\eqref{eq:prox-definition}. \n\\end{proof}","sha256_utf8":"e355d72fdee48531c9496004fcf327e7d7723878ff0853750037eb7199bc7955"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"复合包络给出横向梯度","source_units":[{"start":27406,"end":27900,"start_line":241,"end_line":247,"raw_tex":"\tNotice that the saddle envelope is a composition of Moreau envelopes\n\t\\begin{align*}\n\tL_{\\eta}(x,y) &= \\min_{u} \\left(\\max_{v} L(u,v) - \\frac{\\eta}{2}\\|v-y\\|^2\\right) + \\frac{\\eta}{2}\\|u-x\\|^2\\\\\n\t&= \\min_{u}\\ -e_\\eta\\{-L(u,\\cdot)\\}(y) + \\frac{\\eta}{2}\\|u-x\\|^2 = e_{\\eta}\\{g(\\cdot,y)\\}(x)\n\t\\end{align*}\n\twhere $g(u,y) = -e_{\\eta}\\{-L(u,\\cdot)\\}(y)$. Applying the gradient formula~\\eqref{eq:moreau-grad} gives our first claimed gradient formula in $x$ of\n\t$\\nabla_x L_\\eta(x,y) = \\eta (x-x_+)$\n","sha256_utf8":"75e63b4dbb35905cf4d647c68e05f15ceef50904a24c062f80ada92c2a799eae"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"唯一极小点与纵向对称公式","source_units":[{"start":27900,"end":28085,"start_line":248,"end_line":248,"raw_tex":"\tsince $x_+$ is the unique minimizer of $u \\mapsto g(u,y) + \\frac{\\eta}{2}\\|u-x\\|^2$. Symmetric reasoning gives our first claimed formula $\\nabla_y L_\\eta(x,y) = \\eta (y_+-y) $ in $y$.\n","sha256_utf8":"e7684f308ca079c70aaa5a4849f1f16cc74e950214abf382b6fa107ec98c5a74"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"近端最优性给出梯度相等","source_units":[{"start":28085,"end":28197,"start_line":249,"end_line":249,"raw_tex":"\tThe second claimed equality is precisely the first-order optimality condition for~\\eqref{eq:prox-definition}. \n","sha256_utf8":"8cd74d8e991d8c9dc0b27fadc5d81e95fd509b9ecc811254dcbca4a3c7390666"}]},{"id":"S030","semantic_type":"corollary","kind":"corollary","title":"原目标与包络驻点一致","source_units":[{"start":28209,"end":28340,"start_line":251,"end_line":253,"raw_tex":"\\begin{corollary} \\label{cor:stationarity}\n\tThe stationary points of $L_\\eta$ are exactly the same as those of $L$.\n\\end{corollary}","sha256_utf8":"235b1efa236c8626ca00c9eb16ff82a3a299aab7bffb1d369ef2c88f20cec845"}]},{"id":"S031","semantic_type":"proof","kind":"proof","title":"驻点一致性证明","source_units":[{"start":28341,"end":29154,"start_line":254,"end_line":258,"raw_tex":"\\begin{proof}\n\tFirst consider any stationary point $z=(x,y)$ of $L$. Denote $z_+=\\prox_\\eta(z)$ and the objective function defining the proximal operator~\\eqref{eq:prox-definition} as $M(u,v) = L(u,v) + \\frac{\\eta}{2}\\|u-x\\|^2-\\frac{\\eta}{2}\\|v-y\\|^2$. Then observing that $\\nabla M(z)=0$, $z$ must be the unique minimax point of $M$ (that is, $z=z_+=\\prox_\\eta(z)$). Hence $z$ must be a stationary point of $L_\\eta$ as well since $\\nabla L_\\eta(z) =\\nabla L(z_+)=\\nabla L(z)= 0$.\n\t\n\tConversely consider a stationary point $z=(x,y)$ of $L_\\eta$. Then $\\eta(x-x_+) = \\nabla_x L_\\eta(z)=0$ and $\\eta(y_+-y) = \\nabla_y L_\\eta(z)=0$. Hence we again find that $z=z_+=\\prox_\\eta(z)$ and consequently, this point must be a stationary point of $L$ as well since $\\nabla L(z)=\\nabla L(z_+)=\\nabla L_\\eta(z)=0$.\n\\end{proof}","sha256_utf8":"668d4e8b99a78b02b04329fb30fdbc130531ea97c6439c89e328ee638281a2d4"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"原目标驻点也是包络驻点","source_units":[{"start":28355,"end":28822,"start_line":255,"end_line":255,"raw_tex":"\tFirst consider any stationary point $z=(x,y)$ of $L$. Denote $z_+=\\prox_\\eta(z)$ and the objective function defining the proximal operator~\\eqref{eq:prox-definition} as $M(u,v) = L(u,v) + \\frac{\\eta}{2}\\|u-x\\|^2-\\frac{\\eta}{2}\\|v-y\\|^2$. Then observing that $\\nabla M(z)=0$, $z$ must be the unique minimax point of $M$ (that is, $z=z_+=\\prox_\\eta(z)$). Hence $z$ must be a stationary point of $L_\\eta$ as well since $\\nabla L_\\eta(z) =\\nabla L(z_+)=\\nabla L(z)= 0$.\n","sha256_utf8":"70a89ee6a9f0060d3033d34fcff5f06df6e70510486f86c620eb9e36073ea8fe"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"包络驻点也是原目标驻点","source_units":[{"start":28824,"end":29143,"start_line":257,"end_line":257,"raw_tex":"\tConversely consider a stationary point $z=(x,y)$ of $L_\\eta$. Then $\\eta(x-x_+) = \\nabla_x L_\\eta(z)=0$ and $\\eta(y_+-y) = \\nabla_y L_\\eta(z)=0$. Hence we again find that $z=z_+=\\prox_\\eta(z)$ and consequently, this point must be a stationary point of $L$ as well since $\\nabla L(z)=\\nabla L(z_+)=\\nabla L_\\eta(z)=0$.\n","sha256_utf8":"51b1d96c0dca679dd2b68e077e61b558eb5b01a24a33ef7c374954f64b74fc0b"}]},{"id":"S034","semantic_type":"corollary","kind":"corollary","title":"阻尼近端迭代等价于梯度迭代","source_units":[{"start":29155,"end":29399,"start_line":259,"end_line":261,"raw_tex":"\\begin{corollary} \\label{cor:PPM-to-GDA}\n\tOne step of the (damped) PPM~\\eqref{eq:saddle-PPM} on the original objective $L$ is equivalent to one step of GDA~\\eqref{eq:GDA} on the saddle envelope $L_\\eta$ with $s = \\lambda/\\eta$. \n\\end{corollary}","sha256_utf8":"1c286635ba373812d088f40e3e4e0d49b9e40d151b232a157abbc852ec737fc9"}]},{"id":"S035","semantic_type":"proof","kind":"proof","title":"两种迭代等价性证明","source_units":[{"start":29400,"end":30082,"start_line":262,"end_line":270,"raw_tex":"\\begin{proof}\n\tLet $(x^+_k,y^+_k) = \\prox_\\eta(x_k,y_k)$ and let $(x_{k+1}, y_{k+1})$ be a step of GDA on $L_\\eta(x,y)$ from $(x_k,y_k)$ with step-size $s=\\lambda/\\eta$. Then\n\t\\begin{align*}\n\t\\begin{bmatrix} x_{k+1} \\\\ y_{k+1} \\end{bmatrix} = \\begin{bmatrix} x_{k} \\\\ y_{k}\\end{bmatrix} - s\\begin{bmatrix} \\nabla_xL_\\eta(z_k) \\\\ -\\nabla_yL_\\eta(z_k)\\end{bmatrix}\n\t&= \\begin{bmatrix} x_{k} \\\\ y_{k}\\end{bmatrix} - \\frac{\\lambda}{\\eta}\\begin{bmatrix} \\eta(x_k-x^+_k) \\\\ -\\eta(y^+_k-y_k) \\end{bmatrix}\n\t= (1-\\lambda)\\begin{bmatrix} x_{k} \\\\ y_{k}\\end{bmatrix} + \\lambda\\begin{bmatrix} x^+_k \\\\ y^{+}_k \\end{bmatrix}\n\t\\end{align*}\n\tfollows from Lemma~\\ref{lem:grad-formula}.\n\\end{proof}","sha256_utf8":"ec72f3bcee7ac56f02c110a3990479ac5d92f3c6683d096b45f2081c2aedbb45"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"代入包络梯度得到阻尼更新","source_units":[{"start":29414,"end":30027,"start_line":263,"end_line":268,"raw_tex":"\tLet $(x^+_k,y^+_k) = \\prox_\\eta(x_k,y_k)$ and let $(x_{k+1}, y_{k+1})$ be a step of GDA on $L_\\eta(x,y)$ from $(x_k,y_k)$ with step-size $s=\\lambda/\\eta$. Then\n\t\\begin{align*}\n\t\\begin{bmatrix} x_{k+1} \\\\ y_{k+1} \\end{bmatrix} = \\begin{bmatrix} x_{k} \\\\ y_{k}\\end{bmatrix} - s\\begin{bmatrix} \\nabla_xL_\\eta(z_k) \\\\ -\\nabla_yL_\\eta(z_k)\\end{bmatrix}\n\t&= \\begin{bmatrix} x_{k} \\\\ y_{k}\\end{bmatrix} - \\frac{\\lambda}{\\eta}\\begin{bmatrix} \\eta(x_k-x^+_k) \\\\ -\\eta(y^+_k-y_k) \\end{bmatrix}\n\t= (1-\\lambda)\\begin{bmatrix} x_{k} \\\\ y_{k}\\end{bmatrix} + \\lambda\\begin{bmatrix} x^+_k \\\\ y^{+}_k \\end{bmatrix}\n\t\\end{align*}\n","sha256_utf8":"a61be1000264489a7f0d6edb6a5e6a6efd6cd0a6bc5bf89e0d0dfeec9899fb2b"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"确认代入使用的梯度公式","source_units":[{"start":30027,"end":30071,"start_line":269,"end_line":269,"raw_tex":"\tfollows from Lemma~\\ref{lem:grad-formula}.\n","sha256_utf8":"0133e0b87999ea409a8e6ac2429e9c06e9dc5b551a97ad9b2095e0b0de1fd1b8"}]},{"id":"S038","semantic_type":"lemma","kind":"lemma","title":"鞍点包络海森矩阵公式","source_units":[{"start":30346,"end":31148,"start_line":273,"end_line":279,"raw_tex":"\\begin{lemma} \\label{lem:hess-formula}\n\tThe Hessian of the saddle envelope $L_\\eta(z)$ is\n\t$$ \\begin{bmatrix} \\nabla^2_{xx} L_\\eta(z) & \\nabla^2_{xy} L_\\eta(z)\\\\  -\\nabla^2_{yx}L_\\eta(z) & -\\nabla^2_{yy} L_\\eta(z)\\end{bmatrix} = \\eta I-\\eta^2\\left(\\eta I+ \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)^{-1} $$\n\twhere $z_+ = \\prox_\\eta(z)$. Since $\\eta>\\rho$, we have\n\t$$ \\nabla^2_{xx} L_\\eta(z) = \\eta I -\\eta^2\\left(\\eta I +\\nabla_{xx} L(z_+) + \\nabla^2_{xy} L(z_+)(\\eta I -\\nabla^2_{yy} L(z_+))^{-1}\\nabla^2_{yx} L(z_+)\\right)^{-1},$$\n\t$$ \\nabla^2_{yy} L_\\eta(z) = -\\eta I +\\eta^2\\left(\\eta I +\\nabla_{yy} L(z_+) + \\nabla^2_{yx} L(z_+)(\\eta I +\\nabla^2_{xx} L(z_+))^{-1}\\nabla^2_{xy} L(z_+)\\right)^{-1}.$$\n\\end{lemma}","sha256_utf8":"0e091ae7103b8cbe47634bc70f29d90f44ed6342fb0e366f06bb4e801fbc393b"}]},{"id":"S039","semantic_type":"proof","kind":"proof","title":"包络海森矩阵公式证明","source_units":[{"start":31149,"end":36253,"start_line":280,"end_line":314,"raw_tex":"\\begin{proof}\n\tConsider some $z=(x,y)$ and a nearby point $z^\\Delta=z+\\Delta$. Denote one proximal step from each of these points by $z_+=(x_+,y_+)= \\prox_\\eta(z)$ and $z^\\Delta_+=(x^\\Delta_+,y^\\Delta_+)= \\prox_\\eta(z^\\Delta)$. \n\tThen our claimed Hessian formula amounts to showing\n\t\\begin{align*}\n\t\t\\begin{bmatrix} \\nabla_x L_\\eta(z^\\Delta) \\\\ -\\nabla_y L_\\eta(z^\\Delta) \\end{bmatrix} - \\begin{bmatrix} \\nabla_x L_\\eta(z) \\\\ -\\nabla_y L_\\eta(z) \\end{bmatrix}  &= \\left(\\eta I - \\eta^2\\left(\\eta I + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)^{-1}\\right)\\begin{bmatrix} \\Delta_x\\\\ \\Delta_y  \\end{bmatrix} +o(\\|\\Delta\\|).\n\t\\end{align*}\n\tRecall Lemma~\\ref{lem:grad-formula} showed the gradient of the saddle envelope is given by $\\nabla_x L_\\eta(z) = \\eta(x-x_+)$ and $\\nabla_y L_\\eta(z) = \\eta(y_+-y)$. Applying this at $z$ and $z_+$ and dividing by $\\eta$, our claimed Hessian formula becomes\n\t\\begin{equation} \\label{eq:target-hessian-condition}\n\t\\begin{bmatrix} x^\\Delta_+-x_+\\\\ y^\\Delta_+-y_+ \\end{bmatrix} = \\eta\\left(\\eta I + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)^{-1}\\begin{bmatrix} \\Delta_x\\\\ \\Delta_y \\end{bmatrix} +o(\\|\\Delta\\|) \\ .\n\t\\end{equation}\n\tOur proof shows this in two steps: first considering a proximal step on the second-order Taylor approximation of $L$ at $z_+$ and then showing this closely matches the result of a proximal step on $L$.\n\t\n\tFirst, consider the following quadratic model of the objective around $z_+$:\n\t$$ \\widetilde L(z) = L(z_+) + \\nabla L(z_+)^T(z-z_+) + \\frac{1}{2}(z-z_+)^T\\nabla^2 L(z_+)(z-z_+) \\ . $$\n\tDenote the result of one proximal step on $\\widetilde L$ from $z^\\Delta$ by $\\widetilde z^\\Delta_+ = (\\widetilde x_+^\\Delta, \\widetilde y_+^\\Delta)$. Since the proximal subproblem is strongly convex-strongly concave, this solution is uniquely determined by\n\t$$\n\t\\begin{bmatrix} \\nabla_x \\widetilde L(\\widetilde x^\\Delta_+,\\widetilde y^\\Delta_+)\\\\  -\\nabla_y \\widetilde L(\\widetilde x^\\Delta_+,\\widetilde y^\\Delta_+)\\end{bmatrix} + \\begin{bmatrix} \\eta(\\widetilde x^\\Delta_+-x^\\Delta)\\\\ \\eta(\\widetilde y^\\Delta_+-y^\\Delta) \\end{bmatrix} = 0 \\ .\n\t$$\n\tPlugging in the definition of our quadratic model $\\widetilde L$ yields\n\t\\begin{align*}\n\t&\\begin{bmatrix} \\nabla_x L(z_+)\\\\ -\\nabla_y L(z_+)\\end{bmatrix} + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\begin{bmatrix} \\widetilde x^\\Delta_+-x_+\\\\ \\widetilde y^\\Delta_+-y_+ \\end{bmatrix} + \\begin{bmatrix} \\eta(\\widetilde x^\\Delta_+-x^\\Delta)\\\\ \\eta(\\widetilde y^\\Delta_+-y^\\Delta) \\end{bmatrix} = 0 \\ .\n\t\\end{align*}\n\tHence\n\t\\begin{align*}\n\t& \\left(\\eta I + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)\\begin{bmatrix} \\widetilde x^\\Delta_+-x_+\\\\  \\widetilde y^\\Delta_+-y_+ \\end{bmatrix} = \\eta \\begin{bmatrix} x^\\Delta - x_+ -\\eta^{-1}\\nabla_x L(z_+) \\\\ y^\\Delta - y_+ +\\eta^{-1}\\nabla_y L(z_+) \\end{bmatrix}\\\\\n\t&\\implies  \\left(\\eta I + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)\\begin{bmatrix} \\widetilde x^\\Delta_+-x_+\\\\  \\widetilde y^\\Delta_+-y_+ \\end{bmatrix} = \\eta \\begin{bmatrix} \\Delta_x\\\\ \\Delta_y \\end{bmatrix}\\\\\n\t&\\implies  \\begin{bmatrix} \\widetilde x^\\Delta_+-x_+\\\\ \\widetilde y^\\Delta_+-y_+ \\end{bmatrix} = \\eta\\left(\\eta I + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)^{-1}\\begin{bmatrix} \\Delta_x\\\\ \\Delta_y \\end{bmatrix}.\n\t\\end{align*}\n\t\n\tThis is nearly our target condition~\\eqref{eq:target-hessian-condition}. All that remains is to show our second-order approximation satisfies $\\| z^\\Delta_+ - \\widetilde z^\\Delta_+\\| = o(\\|\\Delta\\|)$.\n\tDenote the proximal subproblem objective by $ M^\\Delta (u,v) = L(u,v) + \\frac{\\eta}{2}\\|u-x^\\Delta\\|^2 - \\frac{\\eta}{2}\\|v-y^\\Delta\\|^2$ and its approximation by $\\widetilde M^\\Delta (u,v) = \\widetilde L(u,v) + \\frac{\\eta}{2}\\|u-x^\\Delta\\|^2 - \\frac{\\eta}{2}\\|v-y^\\Delta\\|^2$.\n\tNoting that $\\|\\nabla \\widetilde M^\\Delta (x_+,y_+)\\| = \\eta\\|\\Delta\\|$, we can apply Lemma~\\ref{lem:helper0} to the $(\\eta-\\rho)$-strongly convex-strongly concave function $\\widetilde M^\\Delta$ to bound the distance to its minimax point as\n\t$$ \\|z_+-\\widetilde z_+^\\Delta\\|\\leq \\frac{\\eta}{\\eta-\\rho}\\|\\Delta\\| \\ .$$\n\tConsequently, we can bound difference in gradients between $L$ and its quadratic model $\\widetilde L$ at $\\widetilde z^\\Delta_+$ by $ \\|\\nabla L(\\widetilde z^\\Delta_+) - \\nabla \\widetilde L(\\widetilde z^\\Delta_+)\\| = o(\\|\\Delta\\|)$. Therefore $\\|\\nabla M^\\Delta(\\widetilde z^\\Delta_+)\\| = o(\\|\\Delta\\|)$ and so applying Lemma~\\ref{lem:helper0} to the strongly convex-strongly concave function $M^\\Delta$ bounds the distance to its minimax point as $\\|z^\\Delta_+ - \\widetilde z^\\Delta_+\\| = o(\\|\\Delta\\|)$, which completes our proof.\n\\end{proof}","sha256_utf8":"de033d7fdde60281c83df02761202eccf32ee31f56537b38449e63ea0bf98b27"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"将海森公式化为近端扰动公式","source_units":[{"start":31163,"end":32687,"start_line":281,"end_line":290,"raw_tex":"\tConsider some $z=(x,y)$ and a nearby point $z^\\Delta=z+\\Delta$. Denote one proximal step from each of these points by $z_+=(x_+,y_+)= \\prox_\\eta(z)$ and $z^\\Delta_+=(x^\\Delta_+,y^\\Delta_+)= \\prox_\\eta(z^\\Delta)$. \n\tThen our claimed Hessian formula amounts to showing\n\t\\begin{align*}\n\t\t\\begin{bmatrix} \\nabla_x L_\\eta(z^\\Delta) \\\\ -\\nabla_y L_\\eta(z^\\Delta) \\end{bmatrix} - \\begin{bmatrix} \\nabla_x L_\\eta(z) \\\\ -\\nabla_y L_\\eta(z) \\end{bmatrix}  &= \\left(\\eta I - \\eta^2\\left(\\eta I + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)^{-1}\\right)\\begin{bmatrix} \\Delta_x\\\\ \\Delta_y  \\end{bmatrix} +o(\\|\\Delta\\|).\n\t\\end{align*}\n\tRecall Lemma~\\ref{lem:grad-formula} showed the gradient of the saddle envelope is given by $\\nabla_x L_\\eta(z) = \\eta(x-x_+)$ and $\\nabla_y L_\\eta(z) = \\eta(y_+-y)$. Applying this at $z$ and $z_+$ and dividing by $\\eta$, our claimed Hessian formula becomes\n\t\\begin{equation} \\label{eq:target-hessian-condition}\n\t\\begin{bmatrix} x^\\Delta_+-x_+\\\\ y^\\Delta_+-y_+ \\end{bmatrix} = \\eta\\left(\\eta I + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)^{-1}\\begin{bmatrix} \\Delta_x\\\\ \\Delta_y \\end{bmatrix} +o(\\|\\Delta\\|) \\ .\n\t\\end{equation}\n\tOur proof shows this in two steps: first considering a proximal step on the second-order Taylor approximation of $L$ at $z_+$ and then showing this closely matches the result of a proximal step on $L$.\n","sha256_utf8":"0a4a181de22bd8d9334b8f5a2b0445dd3ed9c0c20fa8c11f56c222996b7480b9"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"建立二次模型及其最优性方程","source_units":[{"start":32689,"end":33915,"start_line":292,"end_line":301,"raw_tex":"\tFirst, consider the following quadratic model of the objective around $z_+$:\n\t$$ \\widetilde L(z) = L(z_+) + \\nabla L(z_+)^T(z-z_+) + \\frac{1}{2}(z-z_+)^T\\nabla^2 L(z_+)(z-z_+) \\ . $$\n\tDenote the result of one proximal step on $\\widetilde L$ from $z^\\Delta$ by $\\widetilde z^\\Delta_+ = (\\widetilde x_+^\\Delta, \\widetilde y_+^\\Delta)$. Since the proximal subproblem is strongly convex-strongly concave, this solution is uniquely determined by\n\t$$\n\t\\begin{bmatrix} \\nabla_x \\widetilde L(\\widetilde x^\\Delta_+,\\widetilde y^\\Delta_+)\\\\  -\\nabla_y \\widetilde L(\\widetilde x^\\Delta_+,\\widetilde y^\\Delta_+)\\end{bmatrix} + \\begin{bmatrix} \\eta(\\widetilde x^\\Delta_+-x^\\Delta)\\\\ \\eta(\\widetilde y^\\Delta_+-y^\\Delta) \\end{bmatrix} = 0 \\ .\n\t$$\n\tPlugging in the definition of our quadratic model $\\widetilde L$ yields\n\t\\begin{align*}\n\t&\\begin{bmatrix} \\nabla_x L(z_+)\\\\ -\\nabla_y L(z_+)\\end{bmatrix} + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\begin{bmatrix} \\widetilde x^\\Delta_+-x_+\\\\ \\widetilde y^\\Delta_+-y_+ \\end{bmatrix} + \\begin{bmatrix} \\eta(\\widetilde x^\\Delta_+-x^\\Delta)\\\\ \\eta(\\widetilde y^\\Delta_+-y^\\Delta) \\end{bmatrix} = 0 \\ .\n\t\\end{align*}\n","sha256_utf8":"356668dc8a39e467d6357286181c48b1b9f3e5437e863d05036bff6c8095119e"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"求解二次模型的近端扰动","source_units":[{"start":33915,"end":34907,"start_line":302,"end_line":307,"raw_tex":"\tHence\n\t\\begin{align*}\n\t& \\left(\\eta I + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)\\begin{bmatrix} \\widetilde x^\\Delta_+-x_+\\\\  \\widetilde y^\\Delta_+-y_+ \\end{bmatrix} = \\eta \\begin{bmatrix} x^\\Delta - x_+ -\\eta^{-1}\\nabla_x L(z_+) \\\\ y^\\Delta - y_+ +\\eta^{-1}\\nabla_y L(z_+) \\end{bmatrix}\\\\\n\t&\\implies  \\left(\\eta I + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)\\begin{bmatrix} \\widetilde x^\\Delta_+-x_+\\\\  \\widetilde y^\\Delta_+-y_+ \\end{bmatrix} = \\eta \\begin{bmatrix} \\Delta_x\\\\ \\Delta_y \\end{bmatrix}\\\\\n\t&\\implies  \\begin{bmatrix} \\widetilde x^\\Delta_+-x_+\\\\ \\widetilde y^\\Delta_+-y_+ \\end{bmatrix} = \\eta\\left(\\eta I + \\begin{bmatrix} \\nabla^2_{xx} L(z_+) & \\nabla^2_{xy} L(z_+)\\\\  -\\nabla^2_{yx} L(z_+) & -\\nabla^2_{yy} L(z_+)\\end{bmatrix}\\right)^{-1}\\begin{bmatrix} \\Delta_x\\\\ \\Delta_y \\end{bmatrix}.\n\t\\end{align*}\n","sha256_utf8":"c7e1497539d5138c08d284300a0b9978235928ed1b53bb9a3fe2d09418e6c766"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"控制模型近端点的距离","source_units":[{"start":34909,"end":35708,"start_line":309,"end_line":312,"raw_tex":"\tThis is nearly our target condition~\\eqref{eq:target-hessian-condition}. All that remains is to show our second-order approximation satisfies $\\| z^\\Delta_+ - \\widetilde z^\\Delta_+\\| = o(\\|\\Delta\\|)$.\n\tDenote the proximal subproblem objective by $ M^\\Delta (u,v) = L(u,v) + \\frac{\\eta}{2}\\|u-x^\\Delta\\|^2 - \\frac{\\eta}{2}\\|v-y^\\Delta\\|^2$ and its approximation by $\\widetilde M^\\Delta (u,v) = \\widetilde L(u,v) + \\frac{\\eta}{2}\\|u-x^\\Delta\\|^2 - \\frac{\\eta}{2}\\|v-y^\\Delta\\|^2$.\n\tNoting that $\\|\\nabla \\widetilde M^\\Delta (x_+,y_+)\\| = \\eta\\|\\Delta\\|$, we can apply Lemma~\\ref{lem:helper0} to the $(\\eta-\\rho)$-strongly convex-strongly concave function $\\widetilde M^\\Delta$ to bound the distance to its minimax point as\n\t$$ \\|z_+-\\widetilde z_+^\\Delta\\|\\leq \\frac{\\eta}{\\eta-\\rho}\\|\\Delta\\| \\ .$$\n","sha256_utf8":"d72e5ba345f54e71c809fee0b17d22c5e14f0b925d477612db7150f7bf9214fe"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"控制模型余项并完成扰动公式","source_units":[{"start":35708,"end":36242,"start_line":313,"end_line":313,"raw_tex":"\tConsequently, we can bound difference in gradients between $L$ and its quadratic model $\\widetilde L$ at $\\widetilde z^\\Delta_+$ by $ \\|\\nabla L(\\widetilde z^\\Delta_+) - \\nabla \\widetilde L(\\widetilde z^\\Delta_+)\\| = o(\\|\\Delta\\|)$. Therefore $\\|\\nabla M^\\Delta(\\widetilde z^\\Delta_+)\\| = o(\\|\\Delta\\|)$ and so applying Lemma~\\ref{lem:helper0} to the strongly convex-strongly concave function $M^\\Delta$ bounds the distance to its minimax point as $\\|z^\\Delta_+ - \\widetilde z^\\Delta_+\\| = o(\\|\\Delta\\|)$, which completes our proof.\n","sha256_utf8":"c9afb462c0109ca3c0ab7a2a5c695045d073fd80d8a3bcecb4ffcb03123235f8"}]},{"id":"S045","semantic_type":"proposition","kind":"proposition","title":"交互支配产生包络曲率界","source_units":[{"start":39886,"end":40442,"start_line":342,"end_line":347,"raw_tex":"\\begin{proposition} \\label{prop:hessians}\n\tIf the $x$-interaction dominance~\\eqref{eq:x-dominance} holds with $\\alpha\\in\\RR$, the saddle envelope is smooth and weakly convex with respect to $x$ \n\t$$  (\\eta^{-1} + \\alpha^{-1})^{-1}I \\preceq \\nabla^2_{xx}L_\\eta(z) \\preceq \\eta I\\ ,$$\n\tand if the $y$-interaction dominance condition~\\eqref{eq:y-dominance} holds with $\\alpha\\in\\RR$, the saddle envelope is smooth and weakly concave with respect to $y$\n\t$$  (\\eta^{-1} + \\alpha^{-1})^{-1}I \\preceq -\\nabla^2_{yy}L_\\eta(z) \\preceq \\eta I\\ .$$\n\\end{proposition}","sha256_utf8":"f2d88316e9bbbb39a1af32588d3e11f27f5f63020dcd8d246c889ac5f296e8e7"}]},{"id":"S046","semantic_type":"proof","kind":"proof","title":"包络曲率界证明","source_units":[{"start":40443,"end":41602,"start_line":348,"end_line":362,"raw_tex":"\\begin{proof}\n\tRecall the formula for the $x$ component of the Hessian given by Lemma~\\ref{lem:hess-formula}.\n\tThen the interaction dominance condition~\\eqref{eq:x-dominance} can lower bound this Hessian by\n\t\\begin{align*}\n\t\\nabla^2_{xx} L_\\eta(z) &=\\eta I -\\eta^2\\left(\\eta I +\\nabla_{xx} L(z_+) + \\nabla^2_{xy} L(z_+)(\\eta I -\\nabla^2_{yy} L(z_+))^{-1}\\nabla^2_{yx} L(z_+)\\right)^{-1}\\\\\n\t&\\succeq \\eta I -\\eta^2\\left(\\eta I +\\alpha I\\right)^{-1}\\\\\n\t&=(\\eta - \\eta^2/(\\eta+\\alpha))I\\\\\n\t&=(\\eta^{-1} + \\alpha^{-1})^{-1}I \\ .\n\t\\end{align*}\n\tNote that $\\eta I +\\nabla^2_{xx} L(z_+)$ is positive definite (since $\\eta>\\rho$) and $\\nabla^2_{xy} L(z_+)(\\eta I +\\nabla^2_{yy} L(z_+))^{-1}\\nabla^2_{yx} L(z_+)$ is positive semidefinite (since its written as a square). Then the inverse of their sum must also be positive definite and consequently $\\nabla^2_{xx} L_\\eta(z)$ is upper bounded by\n\t\\begin{align*}\n\t\\eta I -\\eta^2\\left(\\eta I +\\nabla_{xx} L(z_+) + \\nabla^2_{xy} L(z_+)(\\eta I +\\nabla^2_{yy} L(z_+))^{-1}\\nabla^2_{yx} L(z_+)\\right)^{-1} &\\preceq \\eta I \\ .\n\t\\end{align*}\n\tSymmetric reasoning applies to give bounds on $\\nabla^2_{yy} L_\\eta(z)$.\n\\end{proof}","sha256_utf8":"0db845bd937190d043b58f1fe3b6d273a0962df09b860ed75719225550f864fa"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"交互条件给出横向曲率下界","source_units":[{"start":40457,"end":40982,"start_line":349,"end_line":356,"raw_tex":"\tRecall the formula for the $x$ component of the Hessian given by Lemma~\\ref{lem:hess-formula}.\n\tThen the interaction dominance condition~\\eqref{eq:x-dominance} can lower bound this Hessian by\n\t\\begin{align*}\n\t\\nabla^2_{xx} L_\\eta(z) &=\\eta I -\\eta^2\\left(\\eta I +\\nabla_{xx} L(z_+) + \\nabla^2_{xy} L(z_+)(\\eta I -\\nabla^2_{yy} L(z_+))^{-1}\\nabla^2_{yx} L(z_+)\\right)^{-1}\\\\\n\t&\\succeq \\eta I -\\eta^2\\left(\\eta I +\\alpha I\\right)^{-1}\\\\\n\t&=(\\eta - \\eta^2/(\\eta+\\alpha))I\\\\\n\t&=(\\eta^{-1} + \\alpha^{-1})^{-1}I \\ .\n\t\\end{align*}\n","sha256_utf8":"f3e2d89f86ca8cc957358aeb95c48700520c978277c4986fcc14746b71d3ed3f"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"正定求逆给出横向曲率上界","source_units":[{"start":40982,"end":41517,"start_line":357,"end_line":360,"raw_tex":"\tNote that $\\eta I +\\nabla^2_{xx} L(z_+)$ is positive definite (since $\\eta>\\rho$) and $\\nabla^2_{xy} L(z_+)(\\eta I +\\nabla^2_{yy} L(z_+))^{-1}\\nabla^2_{yx} L(z_+)$ is positive semidefinite (since its written as a square). Then the inverse of their sum must also be positive definite and consequently $\\nabla^2_{xx} L_\\eta(z)$ is upper bounded by\n\t\\begin{align*}\n\t\\eta I -\\eta^2\\left(\\eta I +\\nabla_{xx} L(z_+) + \\nabla^2_{xy} L(z_+)(\\eta I +\\nabla^2_{yy} L(z_+))^{-1}\\nabla^2_{yx} L(z_+)\\right)^{-1} &\\preceq \\eta I \\ .\n\t\\end{align*}\n","sha256_utf8":"09b87e6b9888eb0f70b759c18bec5f7268d51e1118f5498d5272fc2bd47f328e"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"对称地得到纵向曲率界","source_units":[{"start":41517,"end":41591,"start_line":361,"end_line":361,"raw_tex":"\tSymmetric reasoning applies to give bounds on $\\nabla^2_{yy} L_\\eta(z)$.\n","sha256_utf8":"902f872f8e913add2c6767836f4a571c1306bae21e4ce697396f2a5580cfbe09"}]},{"id":"S050","semantic_type":"proposition","kind":"proposition","title":"包络整体梯度光滑性","source_units":[{"start":42721,"end":42858,"start_line":372,"end_line":374,"raw_tex":"\\begin{proposition}\\label{prop:smoothness}\n\t$L_\\eta$ has $\\max\\{\\eta, |\\eta^{-1}-\\rho^{-1}|^{-1}\\}$-Lipschitz gradient.\n\\end{proposition}","sha256_utf8":"ee22690bbe1172ce9ccf0022be0acdc3d6372efcfab8977a78586575f675e1c9"}]},{"id":"S051","semantic_type":"proof","kind":"proof","title":"包络整体光滑性证明","source_units":[{"start":42859,"end":46220,"start_line":375,"end_line":402,"raw_tex":"\\begin{proof}\n\tConsider two points $z=(x,y)$ and $\\bar z = (\\bar x, \\bar y)$ and denote one proximal step from each of them by $z_+=(x_+,y_+) = \\prox_\\eta(z)$ and $\\bar z_+ = (\\bar x_+,\\bar y_+) = \\prox_\\eta(\\bar z)$.\n\tDefine the  $(\\eta-\\rho)$-strongly convex-strongly concave function underlying the computation of the saddle envelope at $z$ as\n\t$$ M(u,v) = L(u,v) + \\frac{\\eta}{2}\\|u-x\\|^2 -\\frac{\\eta}{2}\\|v-y\\|^2. $$\n\tFirst we compute the gradient of $M$ at $\\bar z_+$ which is given by\n\t\\begin{align*}\n\t\\begin{bmatrix} \\nabla_x M(\\bar z_+) \\\\ \\nabla_y M(\\bar z_+)\\end{bmatrix} =  \\begin{bmatrix} \\nabla_x L(\\bar z_+) + \\eta(\\bar x_+-x) \\\\ \\nabla_y L(\\bar z_+) - \\eta(\\bar y_+-y)\\end{bmatrix}\n\t%=  \\begin{bmatrix} \\nabla_x L(\\bar z_+) + \\eta(\\bar x_+-\\bar x+\\bar x-x) \\\\ \\nabla_y L(\\bar z_+) - \\eta(\\bar y_+-\\bar y+\\bar y-y)\\end{bmatrix}\\\n\t= \\eta \\begin{bmatrix} \\bar x-x \\\\ y-\\bar y\\end{bmatrix}.\n\t\\end{align*}\n\tApplying Lemma~\\ref{lem:helper0}, and noting that $z_+=\\prox_\\eta(z)$ has $\\nabla M(z_+)=0$ yields\n\t$$ \\left\\|\\begin{bmatrix} \\bar x_+-x_+ \\\\ \\bar y_+-y_+ \\end{bmatrix}\\right\\|^2 \\leq \\frac{\\eta}{\\eta-\\rho} \\begin{bmatrix} \\bar x- x \\\\ \\bar y- y \\end{bmatrix}^T\\begin{bmatrix} \\bar x_+-x_+ \\\\ \\bar y_+-y_+ \\end{bmatrix}.$$\n\tRecalling the saddle envelope's gradient formula from Lemma~\\ref{lem:grad-formula}, we can upper bound the difference between its gradients at $z$ and $\\bar z$ by\n\t\\begin{align*}\n\t\\frac{1}{\\eta^{2}}\\|\\nabla L_\\eta(z) - \\nabla L_\\eta(\\bar z)\\|^2\n\t&= \\left\\|\\begin{bmatrix} x-x_+ \\\\ y_+-y \\end{bmatrix} - \\begin{bmatrix} \\bar x-\\bar x_+ \\\\ \\bar y_+-\\bar y \\end{bmatrix}\\right\\|^2\\\\\n\t&=\\left\\|\\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}\\right\\|^2 + 2 \\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}^T\\begin{bmatrix} \\bar x_+- x_+ \\\\ \\bar y_+- y_+ \\end{bmatrix} +\\left\\|\\begin{bmatrix} \\bar x_+- x_+ \\\\ \\bar y_+- y_+ \\end{bmatrix}\\right\\|^2\\\\\n\t&\\leq \\left\\|\\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}\\right\\|^2 + \\left(\\frac{\\eta}{\\eta-\\rho}-2\\right)\\begin{bmatrix} \\bar x- x \\\\ \\bar y- y \\end{bmatrix}^T\\begin{bmatrix} \\bar x_+- x_+ \\\\ \\bar y_+- y_+ \\end{bmatrix}.\n\t\\end{align*}\n\tNotice that $\\begin{bmatrix} \\bar x- x \\\\ \\bar y- y \\end{bmatrix}^T\\begin{bmatrix} \\bar x_+- x_+ \\\\ \\bar y_+- y_+ \\end{bmatrix}$ is non-negative but the sign of $\\left(\\frac{\\eta}{\\eta-\\rho}-2\\right)$ may be positive or negative. If this coefficient is negative, we can upperbound the second term above by zero, giving $\t\\|\\nabla L_\\eta(z) - \\nabla L_\\eta(\\bar z)\\|^2 \\leq \\eta^2\\left\\|z-\\bar z\\right\\|^2 .$\n\tIf instead $\\left(\\frac{\\eta}{\\eta-\\rho}-2\\right)\\geq 0$, then we have smoothness constant $|\\eta^{-1}-\\rho^{-1}|^{-1}$ as\n\t\\begin{align*}\n\t\\|\\nabla L_\\eta(z) - \\nabla L_\\eta(\\bar z)\\|^2 &\\leq \\eta^2\\left(1 + \\left(\\frac{\\eta}{\\eta-\\rho}-2\\right)\\frac{\\eta}{\\eta-\\rho}\\right)\\left\\|\\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}\\right\\|^2 \\\\\n\t&= \\eta^2\\left(\\frac{\\eta}{\\eta-\\rho}-1\\right)^2\\left\\|\\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}\\right\\|^2 \\\\\n\t&= \\left(\\frac{\\eta\\rho}{\\eta-\\rho}\\right)^2\\left\\|\\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}\\right\\|^2 \\ .\n\t\\end{align*}\n\tby Cauchy–Schwarz and using that $\\left\\|\\begin{bmatrix} \\bar x_+- x_+ \\\\ \\bar y_+- y_+ \\end{bmatrix}\\right\\| \\leq \\frac{\\eta}{\\eta-\\rho}\\left\\|\\begin{bmatrix} \\bar x- x \\\\ \\bar y- y \\end{bmatrix}\\right\\|$.\n\\end{proof}","sha256_utf8":"512a2da1fbabc84387c55173820015ca65d201389dd4d5bd0e8072fd1f303e47"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"比较两近端点的子问题梯度","source_units":[{"start":42873,"end":43776,"start_line":376,"end_line":384,"raw_tex":"\tConsider two points $z=(x,y)$ and $\\bar z = (\\bar x, \\bar y)$ and denote one proximal step from each of them by $z_+=(x_+,y_+) = \\prox_\\eta(z)$ and $\\bar z_+ = (\\bar x_+,\\bar y_+) = \\prox_\\eta(\\bar z)$.\n\tDefine the  $(\\eta-\\rho)$-strongly convex-strongly concave function underlying the computation of the saddle envelope at $z$ as\n\t$$ M(u,v) = L(u,v) + \\frac{\\eta}{2}\\|u-x\\|^2 -\\frac{\\eta}{2}\\|v-y\\|^2. $$\n\tFirst we compute the gradient of $M$ at $\\bar z_+$ which is given by\n\t\\begin{align*}\n\t\\begin{bmatrix} \\nabla_x M(\\bar z_+) \\\\ \\nabla_y M(\\bar z_+)\\end{bmatrix} =  \\begin{bmatrix} \\nabla_x L(\\bar z_+) + \\eta(\\bar x_+-x) \\\\ \\nabla_y L(\\bar z_+) - \\eta(\\bar y_+-y)\\end{bmatrix}\n\t%=  \\begin{bmatrix} \\nabla_x L(\\bar z_+) + \\eta(\\bar x_+-\\bar x+\\bar x-x) \\\\ \\nabla_y L(\\bar z_+) - \\eta(\\bar y_+-\\bar y+\\bar y-y)\\end{bmatrix}\\\n\t= \\eta \\begin{bmatrix} \\bar x-x \\\\ y-\\bar y\\end{bmatrix}.\n\t\\end{align*}\n","sha256_utf8":"b56824134d2e2a8fc8312c89524de7f7b4871af32c7b2586019115c563127a7e"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"强单调性控制近端点差","source_units":[{"start":43776,"end":44100,"start_line":385,"end_line":386,"raw_tex":"\tApplying Lemma~\\ref{lem:helper0}, and noting that $z_+=\\prox_\\eta(z)$ has $\\nabla M(z_+)=0$ yields\n\t$$ \\left\\|\\begin{bmatrix} \\bar x_+-x_+ \\\\ \\bar y_+-y_+ \\end{bmatrix}\\right\\|^2 \\leq \\frac{\\eta}{\\eta-\\rho} \\begin{bmatrix} \\bar x- x \\\\ \\bar y- y \\end{bmatrix}^T\\begin{bmatrix} \\bar x_+-x_+ \\\\ \\bar y_+-y_+ \\end{bmatrix}.$$\n","sha256_utf8":"5ac5414c098f6cd27eaffc1ba63048fac4668ae1828c0e0d6c3d599c6afea97c"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"展开包络梯度差的平方","source_units":[{"start":44100,"end":44994,"start_line":387,"end_line":393,"raw_tex":"\tRecalling the saddle envelope's gradient formula from Lemma~\\ref{lem:grad-formula}, we can upper bound the difference between its gradients at $z$ and $\\bar z$ by\n\t\\begin{align*}\n\t\\frac{1}{\\eta^{2}}\\|\\nabla L_\\eta(z) - \\nabla L_\\eta(\\bar z)\\|^2\n\t&= \\left\\|\\begin{bmatrix} x-x_+ \\\\ y_+-y \\end{bmatrix} - \\begin{bmatrix} \\bar x-\\bar x_+ \\\\ \\bar y_+-\\bar y \\end{bmatrix}\\right\\|^2\\\\\n\t&=\\left\\|\\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}\\right\\|^2 + 2 \\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}^T\\begin{bmatrix} \\bar x_+- x_+ \\\\ \\bar y_+- y_+ \\end{bmatrix} +\\left\\|\\begin{bmatrix} \\bar x_+- x_+ \\\\ \\bar y_+- y_+ \\end{bmatrix}\\right\\|^2\\\\\n\t&\\leq \\left\\|\\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}\\right\\|^2 + \\left(\\frac{\\eta}{\\eta-\\rho}-2\\right)\\begin{bmatrix} \\bar x- x \\\\ \\bar y- y \\end{bmatrix}^T\\begin{bmatrix} \\bar x_+- x_+ \\\\ \\bar y_+- y_+ \\end{bmatrix}.\n\t\\end{align*}\n","sha256_utf8":"f8836d96f79672650132f87ec2214e5a31663725bb3e1f42275fdb9dd74c91a4"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"非正系数情形的光滑界","source_units":[{"start":44994,"end":45403,"start_line":394,"end_line":394,"raw_tex":"\tNotice that $\\begin{bmatrix} \\bar x- x \\\\ \\bar y- y \\end{bmatrix}^T\\begin{bmatrix} \\bar x_+- x_+ \\\\ \\bar y_+- y_+ \\end{bmatrix}$ is non-negative but the sign of $\\left(\\frac{\\eta}{\\eta-\\rho}-2\\right)$ may be positive or negative. If this coefficient is negative, we can upperbound the second term above by zero, giving $\t\\|\\nabla L_\\eta(z) - \\nabla L_\\eta(\\bar z)\\|^2 \\leq \\eta^2\\left\\|z-\\bar z\\right\\|^2 .$\n","sha256_utf8":"e824483df10cabdacf75a31d3e06de23d11572bb9d7cf835e78039d72b0eaf9f"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"非负系数情形的光滑界","source_units":[{"start":45403,"end":46209,"start_line":395,"end_line":401,"raw_tex":"\tIf instead $\\left(\\frac{\\eta}{\\eta-\\rho}-2\\right)\\geq 0$, then we have smoothness constant $|\\eta^{-1}-\\rho^{-1}|^{-1}$ as\n\t\\begin{align*}\n\t\\|\\nabla L_\\eta(z) - \\nabla L_\\eta(\\bar z)\\|^2 &\\leq \\eta^2\\left(1 + \\left(\\frac{\\eta}{\\eta-\\rho}-2\\right)\\frac{\\eta}{\\eta-\\rho}\\right)\\left\\|\\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}\\right\\|^2 \\\\\n\t&= \\eta^2\\left(\\frac{\\eta}{\\eta-\\rho}-1\\right)^2\\left\\|\\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}\\right\\|^2 \\\\\n\t&= \\left(\\frac{\\eta\\rho}{\\eta-\\rho}\\right)^2\\left\\|\\begin{bmatrix} x-\\bar x \\\\ y-\\bar y \\end{bmatrix}\\right\\|^2 \\ .\n\t\\end{align*}\n\tby Cauchy–Schwarz and using that $\\left\\|\\begin{bmatrix} \\bar x_+- x_+ \\\\ \\bar y_+- y_+ \\end{bmatrix}\\right\\| \\leq \\frac{\\eta}{\\eta-\\rho}\\left\\|\\begin{bmatrix} \\bar x- x \\\\ \\bar y- y \\end{bmatrix}\\right\\|$.\n","sha256_utf8":"0274ec859585e5a18c44bf77f6904f8f40d01e7b3015aecb8054c675f513fa48"}]},{"id":"S057","semantic_type":"problem_setting","kind":"problem_setting","title":"无约束极小极大问题","source_units":[{"start":3087,"end":3411,"start_line":55,"end_line":58,"raw_tex":"Minimax optimization has become a central tool for modern machine learning, recently receiving increasing attention in optimization and machine learning communities. We consider the following saddle point optimization problem\n\\begin{equation}\\label{eq:main-problem}\n\\min_{x\\in\\RR^n}\\max_{y\\in\\RR^m} L(x,y) \\ ,\n\\end{equation}","sha256_utf8":"bbb5903e4ccddfb4718b7f1ba7997a2da69f258fab14eb254b2f1ec07845a072"},{"start":3412,"end":3473,"start_line":59,"end_line":59,"raw_tex":"where $L(x,y)$ is a differentiable function in $x$ and $y$. \n","sha256_utf8":"7868d384914f8c2c9c931c8fe59878bcac56c9e932e35d94a9c5419da8edc1f3"}]},{"id":"S058","semantic_type":"definition","kind":"definition","title":"近端鞍点算子","source_units":[{"start":5669,"end":5890,"start_line":76,"end_line":78,"raw_tex":"\\begin{equation}\\label{eq:prox-definition}\n(x_{k+1},y_{k+1})=\\prox_\\eta(x_k,y_k):= \\mathrm{arg}\\min_{u\\in\\mathbb{R}^n}\\max_{v\\in\\mathbb{R}^m} L(u,v) +  \\frac{\\eta}{2}\\|u-x_k\\|^2 - \\frac{\\eta}{2}\\|v-y_k\\|^2.\n\\end{equation}","sha256_utf8":"5a39dbc02e5e40afdb3a8ddb3338f763c796d39638538c3a6792c7f42731b3dc"}]},{"id":"S059","semantic_type":"definition","kind":"definition","title":"光滑性与强凸强凹定义","source_units":[{"start":12977,"end":13506,"start_line":124,"end_line":129,"raw_tex":"We say a  function $M(x,y)$ is $\\beta$-smooth if its gradient is uniformly $\\beta$-Lipschitz\n$ \\|\\nabla M(z) - \\nabla M(z')\\| \\leq \\beta\\|z-z'\\| $\nor equivalently for twice differentiable functions, if $\\|\\nabla^2 M(z)\\| \\leq \\beta$.\nFurther, we say a twice differentiable $M(x,y)$ is $\\mu$-strongly convex-strongly concave for some $\\mu\\geq 0$ if\n$ \\nabla^2_{xx} M(z) \\succeq \\mu I $ and $-\\nabla^2_{yy}M(z) \\succeq \\mu I.$\nWhen $\\mu=0$, this corresponds to $M$ being convex with respect to $x$ and concave with respect to $y$.\n","sha256_utf8":"2a3a0b75ba47f39d8be7aef2b49026ca37b7a06d8206654ca54e9e35c9a9b3be"}]},{"id":"S060","semantic_type":"assumption","kind":"assumption","title":"二阶可微与弱凸弱凹假设","source_units":[{"start":13507,"end":13908,"start_line":131,"end_line":134,"raw_tex":"Throughout this paper, we are primarily interested in the weakening of this convexity condition to allow negative curvature given by $\\rho$-weak convexity and $\\rho$-weak concavity: we assume that $L$ is twice differentiable, and for any $z=(x,y)\\in\\RR^n\\times\\RR^m$ that\n\\begin{equation} \\label{eq:weak}\n\\nabla^2_{xx} L(z)\\succeq -\\rho I\\ ,\\ \\ \\  -\\nabla^2_{yy} L(z)\\succeq -\\rho I \\ .\n\\end{equation}","sha256_utf8":"ad27e8566ad2b15676a9f67bd5b9b37ca644178e18b9402942d6836ce98c0b65"},{"start":13909,"end":14145,"start_line":135,"end_line":135,"raw_tex":"Notice that the objective $L(x,y)$ is convex-concave when $\\rho=0$, and strongly convex-strongly concave when $\\rho<0$. Here our primary interest is in the regime where $\\rho>0$ is positive, quantifying how nonconvex-nonconcave $L$ is.\n","sha256_utf8":"60abeddb46dc075a1ee887890878f75b7a51ab11b0be541c6bb68415cd6ac9be"}]},{"id":"S061","semantic_type":"algorithm","kind":"algorithm","title":"梯度下降上升迭代","source_units":[{"start":14146,"end":14494,"start_line":137,"end_line":140,"raw_tex":"Besides PPM, Gradient Descent Ascent (GDA) is another classic algorithm for minimax problems~\\eqref{eq:main-problem} given by\n\\begin{equation}\\label{eq:GDA}\n\\begin{bmatrix} x_{k+1} \\\\ y_{k+1} \\end{bmatrix} = \\begin{bmatrix} x_{k} \\\\ y_{k} \\end{bmatrix} - s\\begin{bmatrix} \\nabla_x L(x_k,y_k) \\\\ -\\nabla_y L(x_k,y_k) \\end{bmatrix} \\ ,\n\\end{equation}","sha256_utf8":"6f74fef25c6f890cc1c7db5b6a2b9b459c814a57e8a0fb09791e24e046c0c547"}]},{"id":"S062","semantic_type":"algorithm","kind":"algorithm","title":"阻尼近端点迭代","source_units":[{"start":14711,"end":15052,"start_line":143,"end_line":146,"raw_tex":"In this paper, we study a more generalized algorithm, damped PPM, with damping parameter $\\lambda\\in(0,1]$ and proximal parameter $\\eta>0$ given by\n\\begin{equation}\\label{eq:saddle-PPM}\n\\begin{bmatrix} x_{k+1} \\\\ y_{k+1} \\end{bmatrix} = (1-\\lambda)\\begin{bmatrix} x_{k} \\\\ y_{k} \\end{bmatrix} +\\lambda\\ \\prox_\\eta(x_k,y_k) \\ .\n\\end{equation}","sha256_utf8":"0b1b25b0a6af4fcf82444dd730e7e356552140b2eff167f3c7bf40b4af73040b"},{"start":15053,"end":15145,"start_line":147,"end_line":147,"raw_tex":"In particular, when $\\lambda=1$, we recover the traditional PPM \\eqref{eq:prox-definition}.\n","sha256_utf8":"452006a548c72f44ded05501e811edd7c43e9a21600d7f051999735b17536f54"}]},{"id":"S063","semantic_type":"definition","kind":"definition","title":"Moreau包络定义","source_units":[{"start":22429,"end":22634,"start_line":189,"end_line":192,"raw_tex":"We denote the Moreau envelope of a function $f$ with parameter $\\eta>0$ by\n\\begin{equation}\ne_{\\eta}\\left\\{f\\right\\}(x) = \\min_{u}\\ f(u) + \\frac{\\eta}{2}\\|u - x\\|^2 \\ . \\label{eq:moreau-def}\n\\end{equation}","sha256_utf8":"6830a6648cb7e7342168642174fde3fd174afdca076a09f772679b65db3b4373"}]},{"id":"S064","semantic_type":"derived_fact","kind":"derived_fact","title":"Moreau包络梯度公式","source_units":[{"start":22804,"end":23163,"start_line":197,"end_line":201,"raw_tex":"For $\\rho$-weakly convex functions and $\\eta>\\rho$, there is a nice calculus for the Moreau envelope.\nIts gradient at some $x\\in\\RR^n$ is determined by the proximal step $x_+=\\mathrm{argmin}_u\\  f(u) + \\frac{\\eta}{2}\\|u - x\\|^2$ having\n\\begin{equation}\n\\nabla e_{\\eta}\\left\\{f\\right\\}(x) = \\eta(x-x_+) = \\nabla f(x_+) \\ . \\label{eq:moreau-grad}\n\\end{equation}","sha256_utf8":"c368b940a499d9dada5122028c242def08f3d90d627611f667ab08dcf6b47931"}]},{"id":"S065","semantic_type":"definition","kind":"definition","title":"鞍点包络及近端参数条件","source_units":[{"start":8508,"end":8690,"start_line":103,"end_line":105,"raw_tex":"\\begin{equation}\\label{eq:saddle-envelope}\nL_{\\eta}(x,y) := \\min_{u\\in\\mathbb{R}^n}\\max_{v\\in\\mathbb{R}^m} L(u,v) +  \\frac{\\eta}{2}\\|u-x\\|^2 - \\frac{\\eta}{2}\\|v-y\\|^2.\n\\end{equation}","sha256_utf8":"c20e9efdd6827dbd2fbcd3091db61636b3ea9bcc8cf39576bcdea3893cbab3b9"},{"start":24524,"end":24833,"start_line":220,"end_line":220,"raw_tex":"We require that the parameter $\\eta$ is selected with $\\eta > \\rho$, which ensures the minimax problem in~\\eqref{eq:saddle-envelope} is strongly convex-strongly concave. As a result, the saddle envelope is well-defined (as its subproblem has a unique minimax point) and often can be efficiently approximated.\n","sha256_utf8":"779b49987ae2db504e8210f35f40e8304676264cd1a6cafb0759198d4428dc36"}]},{"id":"S066","semantic_type":"definition","kind":"definition","title":"双侧交互支配条件","source_units":[{"start":37566,"end":38100,"start_line":322,"end_line":331,"raw_tex":"\\begin{definition}\\label{def:interaction-dominance}\n\tA function $L$ is $\\alpha$-interaction dominant with respect to $x$ if\n\t\\begin{align}\n\t\\nabla^2_{xx} L(z) + \\nabla^2_{xy} L(z)(\\eta I -\\nabla^2_{yy} L(z))^{-1}\\nabla^2_{yx} L(z) &\\succeq \\alpha I \\label{eq:x-dominance}\n\t\\end{align}\t   \n\tand $\\alpha$-interaction dominant with respect to $y$ if\n\t\\begin{align}\n\t-\\nabla^2_{yy} L(z) + \\nabla^2_{yx} L(z)(\\eta I +\\nabla^2_{xx} L(z))^{-1}\\nabla^2_{xy} L(z) &\\succeq \\alpha I\\ . \\label{eq:y-dominance}   \n\t\\end{align}\t  \n\\end{definition}","sha256_utf8":"6195fc7f8590e2741675de81b5c4e93037d791c07fb57f180cf954e72cda27ed"}]},{"id":"S067","semantic_type":"external_fact","kind":"external_fact","title":"标准Moreau可微性定理","source_units":[],"declared_external":true,"declaration_text":["For a differentiable rho-weakly convex function f on Euclidean space and eta > rho, the proximal minimizer p(x) is unique, the Moreau envelope is differentiable, and its gradient is eta(x-p(x)) = gradient f(p(x))."]},{"id":"S068","semantic_type":"external_fact","kind":"external_fact","title":"欧氏内积与范数不等式","source_units":[],"declared_external":true,"declaration_text":["Euclidean inner products satisfy Cauchy–Schwarz; vector norms satisfy the triangle inequality; induced matrix norms are submultiplicative. Changing the sign of one coordinate block preserves the Euclidean norm."]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","title":"二阶可微的Taylor余项","source_units":[],"declared_external":true,"declaration_text":["At a point of second differentiability, the gradient equals its affine Taylor model plus a remainder o(||h||). Consequently, evaluating the gradient error of the quadratic model at points displaced by O(||Delta||) gives o(||Delta||)."]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","title":"标准鞍点最优性与球约束条件","source_units":[],"declared_external":true,"declaration_text":["A differentiable strongly convex-strongly concave function has at most one saddle point, and any unconstrained stationary point is that saddle point. On a compact convex product set a continuous convex-concave function has a saddle point. For Euclidean ball constraints, differentiable saddle optimality gives the normal-cone multiplier conditions used in lines 922–923; an interior constrained saddle point is stationary."]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","title":"正定求逆与Schur补公式","source_units":[],"declared_external":true,"declaration_text":["For symmetric positive definite matrices, inversion reverses the Loewner order and preserves positive definiteness. The diagonal blocks of an invertible block matrix are expressed through the inverses of the corresponding Schur complements when these inverses exist."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S006","target_id":"S003","kind":"supports"},{"source_id":"S006","target_id":"S004","kind":"supports"},{"source_id":"S006","target_id":"S005","kind":"supports"},{"source_id":"S045","target_id":"S003","kind":"supports"},{"source_id":"S050","target_id":"S003","kind":"supports"},{"source_id":"S030","target_id":"S003","kind":"supports"},{"source_id":"S013","target_id":"S003","kind":"supports"},{"source_id":"S034","target_id":"S004","kind":"supports"},{"source_id":"S061","target_id":"S004","kind":"supports"},{"source_id":"S062","target_id":"S004","kind":"supports"},{"source_id":"S013","target_id":"S005","kind":"supports"},{"source_id":"S008","target_id":"S007","kind":"supports"},{"source_id":"S009","target_id":"S008","kind":"supports"},{"source_id":"S010","target_id":"S008","kind":"supports"},{"source_id":"S011","target_id":"S008","kind":"supports"},{"source_id":"S012","target_id":"S008","kind":"supports"},{"source_id":"S009","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S059","target_id":"S009","kind":"supports"},{"source_id":"S059","target_id":"S010","kind":"supports"},{"source_id":"S068","target_id":"S012","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S016","target_id":"S014","kind":"supports"},{"source_id":"S017","target_id":"S014","kind":"supports"},{"source_id":"S018","target_id":"S016","kind":"supports"},{"source_id":"S007","target_id":"S016","kind":"supports"},{"source_id":"S061","target_id":"S016","kind":"supports"},{"source_id":"S059","target_id":"S016","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S022","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S019","kind":"supports"},{"source_id":"S022","target_id":"S019","kind":"supports"},{"source_id":"S023","target_id":"S019","kind":"supports"},{"source_id":"S024","target_id":"S019","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S007","target_id":"S022","kind":"supports"},{"source_id":"S068","target_id":"S023","kind":"supports"},{"source_id":"S070","target_id":"S020","kind":"supports"},{"source_id":"S070","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S029","target_id":"S026","kind":"supports"},{"source_id":"S063","target_id":"S027","kind":"supports"},{"source_id":"S064","target_id":"S027","kind":"supports"},{"source_id":"S065","target_id":"S027","kind":"supports"},{"source_id":"S064","target_id":"S028","kind":"supports"},{"source_id":"S065","target_id":"S028","kind":"supports"},{"source_id":"S058","target_id":"S029","kind":"supports"},{"source_id":"S070","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S025","target_id":"S032","kind":"supports"},{"source_id":"S025","target_id":"S033","kind":"supports"},{"source_id":"S058","target_id":"S032","kind":"supports"},{"source_id":"S065","target_id":"S032","kind":"supports"},{"source_id":"S070","target_id":"S032","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S025","target_id":"S037","kind":"supports"},{"source_id":"S058","target_id":"S036","kind":"supports"},{"source_id":"S061","target_id":"S036","kind":"supports"},{"source_id":"S062","target_id":"S036","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S041","target_id":"S039","kind":"supports"},{"source_id":"S042","target_id":"S039","kind":"supports"},{"source_id":"S043","target_id":"S039","kind":"supports"},{"source_id":"S044","target_id":"S039","kind":"supports"},{"source_id":"S025","target_id":"S040","kind":"supports"},{"source_id":"S058","target_id":"S040","kind":"supports"},{"source_id":"S065","target_id":"S040","kind":"supports"},{"source_id":"S060","target_id":"S041","kind":"supports"},{"source_id":"S069","target_id":"S041","kind":"supports"},{"source_id":"S070","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S058","target_id":"S042","kind":"supports"},{"source_id":"S071","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S007","target_id":"S043","kind":"supports"},{"source_id":"S060","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S069","target_id":"S044","kind":"supports"},{"source_id":"S007","target_id":"S044","kind":"supports"},{"source_id":"S071","target_id":"S039","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S049","target_id":"S046","kind":"supports"},{"source_id":"S038","target_id":"S047","kind":"supports"},{"source_id":"S066","target_id":"S047","kind":"supports"},{"source_id":"S071","target_id":"S047","kind":"supports"},{"source_id":"S038","target_id":"S048","kind":"supports"},{"source_id":"S060","target_id":"S048","kind":"supports"},{"source_id":"S071","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S066","target_id":"S049","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S051","kind":"supports"},{"source_id":"S055","target_id":"S051","kind":"supports"},{"source_id":"S056","target_id":"S051","kind":"supports"},{"source_id":"S058","target_id":"S052","kind":"supports"},{"source_id":"S060","target_id":"S052","kind":"supports"},{"source_id":"S065","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S007","target_id":"S053","kind":"supports"},{"source_id":"S025","target_id":"S054","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S056","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S056","kind":"supports"},{"source_id":"S068","target_id":"S056","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S065","kind":"supports"},{"source_id":"S058","target_id":"S062","kind":"supports"},{"source_id":"S060","target_id":"S065","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S067","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S006","kind":"supports"}]}}
{"record_id":"2006.11573v1","paper":{"title":"Unified Analysis of Stochastic Gradient Methods for Composite Convex and Smooth Optimization","authors":["Ahmed Khaled","Othmane Sebbouh","Nicolas Loizou","Robert M. Gower","Peter Richtárik"],"arxiv_id":"2006.11573","version":"v1","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-023-02297-y","abstract_url":"https://arxiv.org/abs/2006.11573v1","pdf_url":"https://arxiv.org/pdf/2006.11573v1"},"context":{"raw_tex":"\\begin{eqnarray}\\label{eq:optimization_problem}\n\\min_{x\\in \\R^d} \\pbr{  F(x) \\equiv f(x) + R(x) },\n\\end{eqnarray}\n\n\\begin{equation}\n    \\label{eq:sgd-proximal-iterates}\n    x_0 \\in \\R^d, \\quad x_{k+1} = \\prox_{\\gamma_k R} \\br{ x_k - \\gamma_k g_k },\n\\end{equation}\n\nwhere $\\br{\\gamma_k}_k$ is a sequence of step sizes.  This estimate $g_k$\ncan take on many different forms depending on the problem of interest. Here we list a few.\n\n\n\\paragraph{Notation.}   The Bregman divergence associated with $f$ is the mapping $$D_{f} (x, y) \\; \\eqdef \\; f(x) - f(y) - \\ev{ \\nabla f(y), x - y }, \\quad x,y\\in R^d$$ and the proximal operator of $\\gamma R$ is the function $$\\prox_{\\gamma R} \\br{x} \\; \\eqdef \\; \\argmin_u \\left\\{ \\gamma R(x) + \\frac{1}{2} \\norm{x-u}^2\\right\\}.$$ Let $[n] \\eqdef \\left\\{1,\\dots,n\\right\\}$. \n\n\n\\begin{assumption} \\label{asm:function-class}\nThe function $f$ is  $L$--smooth and convex:\n\\begin{align}\nf(y) & \\leq f(x) + \\langle \\nabla f(x), y - x \\rangle + \\frac{L}{2} \\norm{y-x}^2,  \\quad \\mbox{for all }x,y \\in \\R^d, \\label{eq:Lsmooth}\\\\\nf(y )& \\geq f(x) + \\langle \\nabla f(x), y - x \\rangle, \\quad \\mbox{for all }x,y \\in \\R^d. \\label{eq:convex}\n\\end{align}\nThe function $R$ is convex:\n\\begin{equation}\\label{eq:R-convex}\nR(\\alpha x + (1- \\alpha) y) \\geq \\alpha R(x) + (1-\\alpha) R(y), \\quad \\mbox{for all }x,y \\in \\R^d, \\alpha \\in [0,\\,1].\n\\end{equation}\n\n\\end{assumption}\n\n\\begin{assumption}[Assumption 4.1 in \\cite{Gorbunov19}]\\label{asm:main_assumption}\nConsider the iterates $\\br{x_k}_k$ and gradients $\\br{g_k}_k$ in \\eqref{eq:sgd-proximal-iterates}.\n    \\begin{enumerate}\n        \\item The gradient estimates are unbiased:\n        \\begin{equation}\n            \\label{eq:asm-gradient-unbiased}\n            \\ec{g_k \\mid x_k} = \\nabla f(x_k).\n        \\end{equation}\n        \\item There exist constants $A,B,C,D_1, D_2, \\rho \\geq 0$, and a sequence of random variables $\\sigma^2_k \\geq 0 $  such that:\n        \\begin{align}\n            \\label{eq:asm-gradient-opt-distance}\n            \\ec{ \\norm{g_k - \\nabla f(x_\\ast)}^2 \\mid x_k } &\\leq 2 A D_{f} (x_k, x_\\ast) + B \\sigma_k^2 + D_1, \\\\\n            \\label{eq:asm-decreasing-noise}\n            \\ec{ \\sigma_{k+1}^2 \\mid x_k } &\\leq \\br{ 1 - \\rho } \\sigma_k^2 + 2 C D_{f} (x_k, x_\\ast) + D_2.\n        \\end{align}\n    \\end{enumerate} \n\\end{assumption}\n\n\\begin{definition}\n    Given a stepsize $\\gamma > 0$, the prox-grad mapping is defined as:\n    \\begin{equation}\n        T_{\\gamma} (x) \\eqdef \\prox_{\\gamma R} \\br{x - \\gamma \\nabla f(x)}.\n    \\end{equation}\n\\end{definition}","units":[{"start":11983,"end":12096,"start_line":310,"end_line":312,"raw_tex":"\\begin{eqnarray}\\label{eq:optimization_problem}\n\\min_{x\\in \\R^d} \\pbr{  F(x) \\equiv f(x) + R(x) },\n\\end{eqnarray}","sha256_utf8":"ceecc42030fd6df92e03bfae1a753c292db43b91f986e84c222f2d5b29f2e280"},{"start":13175,"end":13323,"start_line":318,"end_line":321,"raw_tex":"\\begin{equation}\n    \\label{eq:sgd-proximal-iterates}\n    x_0 \\in \\R^d, \\quad x_{k+1} = \\prox_{\\gamma_k R} \\br{ x_k - \\gamma_k g_k },\n\\end{equation}","sha256_utf8":"6ee0a54975598732eba4de5605c9b11746de268926b89d88c622ad7e30238ce6"},{"start":13324,"end":13489,"start_line":322,"end_line":323,"raw_tex":"where $\\br{\\gamma_k}_k$ is a sequence of step sizes.  This estimate $g_k$\ncan take on many different forms depending on the problem of interest. Here we list a few.\n","sha256_utf8":"1e37900b3f956da340aed2c0c674588c28cf54fd87e29ab683c62628a8f0b87b"},{"start":23702,"end":24079,"start_line":372,"end_line":372,"raw_tex":"\\paragraph{Notation.}   The Bregman divergence associated with $f$ is the mapping $$D_{f} (x, y) \\; \\eqdef \\; f(x) - f(y) - \\ev{ \\nabla f(y), x - y }, \\quad x,y\\in R^d$$ and the proximal operator of $\\gamma R$ is the function $$\\prox_{\\gamma R} \\br{x} \\; \\eqdef \\; \\argmin_u \\left\\{ \\gamma R(x) + \\frac{1}{2} \\norm{x-u}^2\\right\\}.$$ Let $[n] \\eqdef \\left\\{1,\\dots,n\\right\\}$. \n","sha256_utf8":"f4e32b6661abb19988dd3bbd9e69fa50f9ee2c886455fe65fec077a094dccc32"},{"start":24478,"end":25057,"start_line":376,"end_line":387,"raw_tex":"\\begin{assumption} \\label{asm:function-class}\nThe function $f$ is  $L$--smooth and convex:\n\\begin{align}\nf(y) & \\leq f(x) + \\langle \\nabla f(x), y - x \\rangle + \\frac{L}{2} \\norm{y-x}^2,  \\quad \\mbox{for all }x,y \\in \\R^d, \\label{eq:Lsmooth}\\\\\nf(y )& \\geq f(x) + \\langle \\nabla f(x), y - x \\rangle, \\quad \\mbox{for all }x,y \\in \\R^d. \\label{eq:convex}\n\\end{align}\nThe function $R$ is convex:\n\\begin{equation}\\label{eq:R-convex}\nR(\\alpha x + (1- \\alpha) y) \\geq \\alpha R(x) + (1-\\alpha) R(y), \\quad \\mbox{for all }x,y \\in \\R^d, \\alpha \\in [0,\\,1].\n\\end{equation}\n\n\\end{assumption}","sha256_utf8":"2e060b22b0fb6193becb1ee6d95002687dc4a5f62151b2090f0c542e2fca3398"},{"start":25566,"end":26492,"start_line":393,"end_line":409,"raw_tex":"\\begin{assumption}[Assumption 4.1 in \\cite{Gorbunov19}]\\label{asm:main_assumption}\nConsider the iterates $\\br{x_k}_k$ and gradients $\\br{g_k}_k$ in \\eqref{eq:sgd-proximal-iterates}.\n    \\begin{enumerate}\n        \\item The gradient estimates are unbiased:\n        \\begin{equation}\n            \\label{eq:asm-gradient-unbiased}\n            \\ec{g_k \\mid x_k} = \\nabla f(x_k).\n        \\end{equation}\n        \\item There exist constants $A,B,C,D_1, D_2, \\rho \\geq 0$, and a sequence of random variables $\\sigma^2_k \\geq 0 $  such that:\n        \\begin{align}\n            \\label{eq:asm-gradient-opt-distance}\n            \\ec{ \\norm{g_k - \\nabla f(x_\\ast)}^2 \\mid x_k } &\\leq 2 A D_{f} (x_k, x_\\ast) + B \\sigma_k^2 + D_1, \\\\\n            \\label{eq:asm-decreasing-noise}\n            \\ec{ \\sigma_{k+1}^2 \\mid x_k } &\\leq \\br{ 1 - \\rho } \\sigma_k^2 + 2 C D_{f} (x_k, x_\\ast) + D_2.\n        \\end{align}\n    \\end{enumerate} \n\\end{assumption}","sha256_utf8":"2b517a71c5398f10131265c3c19c6ae9458ad9f5f14293ad78ca487d3224d5f4"},{"start":84903,"end":85126,"start_line":1301,"end_line":1306,"raw_tex":"\\begin{definition}\n    Given a stepsize $\\gamma > 0$, the prox-grad mapping is defined as:\n    \\begin{equation}\n        T_{\\gamma} (x) \\eqdef \\prox_{\\gamma R} \\br{x - \\gamma \\nabla f(x)}.\n    \\end{equation}\n\\end{definition}","sha256_utf8":"91044e86089d078d1c5134c7961a24ec0697c0e2afe6ca4b7042237419c05e7c"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\n    \\label{thm:main-prox-dec}\n    Suppose that Assumptions~\\ref{asm:function-class} and~\\ref{asm:main_assumption} hold. Let $M \\eqdef B/\\rho$ and let $(\\gamma_k)_{k\\geq 0}$ be a decreasing, strictly positive sequence of step sizes chosen such that\n    \\[ 0 < \\gamma_0 < \\min \\left\\{ \\frac{1}{2 (A + MC)}, \\frac{1}{L} \\right\\}. \\]\nThe iterates given by \\eqref{eq:sgd-proximal-iterates} satisfy\n    \\begin{align}\n        \\label{eq:main-thm-convergence-bound}\n        \\ec{F(\\bar{x}_t) - F(x_\\ast)} \\leq \\frac{\\sqn{x_0 - x_*} + 2\\gamma_0\\br{\\delta_0 + \\gamma_0M\\sigma_0^2} + 2\\br{D_1 + 2 M D_2}\\sum \\limits_{k=0}^{t-1}\\gamma_k^2}{2\\sum \\limits_{i=0}^{t-1}\\br{1 - 2\\gamma_i\\br{A+MC}} \\gamma_i},\n    \\end{align}\n    where $\\bar{x}_t \\eqdef \\sum\\limits_{k=0}^{t-1} \\frac{\\br{ 1 - 2\\gamma_k(A+MC)}  \\gamma_k}{\\sum \\limits_{i=0}^{t-1}\\br{1 - 2\\gamma_i(A+MC)} \\gamma_i}x_k$ and ${\\delta_0 \\eqdef F(x_0) - F(x_\\ast)}$.\n\\end{theorem}","units":[{"start":27468,"end":28405,"start_line":415,"end_line":425,"raw_tex":"\\begin{theorem}\n    \\label{thm:main-prox-dec}\n    Suppose that Assumptions~\\ref{asm:function-class} and~\\ref{asm:main_assumption} hold. Let $M \\eqdef B/\\rho$ and let $(\\gamma_k)_{k\\geq 0}$ be a decreasing, strictly positive sequence of step sizes chosen such that\n    \\[ 0 < \\gamma_0 < \\min \\left\\{ \\frac{1}{2 (A + MC)}, \\frac{1}{L} \\right\\}. \\]\nThe iterates given by \\eqref{eq:sgd-proximal-iterates} satisfy\n    \\begin{align}\n        \\label{eq:main-thm-convergence-bound}\n        \\ec{F(\\bar{x}_t) - F(x_\\ast)} \\leq \\frac{\\sqn{x_0 - x_*} + 2\\gamma_0\\br{\\delta_0 + \\gamma_0M\\sigma_0^2} + 2\\br{D_1 + 2 M D_2}\\sum \\limits_{k=0}^{t-1}\\gamma_k^2}{2\\sum \\limits_{i=0}^{t-1}\\br{1 - 2\\gamma_i\\br{A+MC}} \\gamma_i},\n    \\end{align}\n    where $\\bar{x}_t \\eqdef \\sum\\limits_{k=0}^{t-1} \\frac{\\br{ 1 - 2\\gamma_k(A+MC)}  \\gamma_k}{\\sum \\limits_{i=0}^{t-1}\\br{1 - 2\\gamma_i(A+MC)} \\gamma_i}x_k$ and ${\\delta_0 \\eqdef F(x_0) - F(x_\\ast)}$.\n\\end{theorem}","sha256_utf8":"d75fb32f9e4085edbd080f4688e2bbfb3a603016f7c9c6e32bdc8f5c4dbd8c9e"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}[Bounding the gradient variance] Assuming that the $g_k$ are unbiased and that Assumption~\\ref{asm:main_assumption} holds, we have\n    \\begin{equation}\n        \\label{eq:lma-gradient-variance}\n        \\ecn{g_k - \\nabla f(x_k)}  \\leq 2 A D_{f} (x_k, x_\\ast) + B \\sigma_k^2 + D_1\n    \\end{equation}\n\\end{lemma}\n\n\\begin{proof}\n    Starting from the left hand side of \\eqref{eq:lma-gradient-variance}, we have\n    \\begin{align*}\n        \\ecn{g_k - \\nabla f(x_k)} &= \\ecn{ g_k - \\nabla f(x_\\ast) - \\br{ \\nabla f(x_k) - \\nabla f(x_\\ast) } } \\\\\n        &= \\ecn{ g_k - \\nabla f(x_\\ast) - \\ec{ g_k - \\nabla f(x_\\ast) } } \\\\\n        &\\leq \\ecn{g_k - \\nabla f(x_\\ast)} \\leq 2 A D_{f} (x_k, x_\\ast) + B \\sigma_k^2 + D_1,\n    \\end{align*}\n    where we used that $\\ecn{ X - \\ec{X} } = \\ecn{X} - \\norm{\\ec{X}}^2 \\leq \\ecn{X}$ for any random variable $X$.\n\\end{proof}\n\n\\begin{lemma}\n    \\label{lemma:atchade-functional-value-bound}\n    Suppose that Assumption~\\ref{asm:function-class} holds and let $\\gamma \\in \\left (0, \\frac{1}{L} \\right ]$, then for all $x, y \\in \\R^d$ and $p = \\prox_{\\gamma g}(y)$ we have,\n    \\begin{equation}\n        \\label{eq:atchade-lemma}\n        - 2 \\gamma \\br{ F(p) - F(x_\\ast) } \\geq \\sqn{p - z } + 2  \\ev{p - x_\\ast, x - \\gamma \\nabla f(x) - y} - \\sqn{x_\\ast - x}.\n    \\end{equation}\n\\end{lemma}\n\n\\begin{proof}\n    We leave the proof to Section~\\ref{app:sec:prooflemma}.\n\\end{proof}\n\n\\begin{lemma}\n    For any $x \\in \\R^d$ and minimizer $x_\\ast$ of $F$ we have,\n    \\begin{equation}\n        \\label{eq:bregman-divergence-bound}\n        D_{f} (x, x_\\ast) \\leq F(x) - F(x_\\ast).\n    \\end{equation}\n\\end{lemma}\n\n\\begin{proof}\n    Because $x_\\ast$ is a minimizer of $F$ we have that $- \\nabla f(x_\\ast)  \\in \\partial R(x_\\ast)$. By the definition of subgradients we have\n    \\[ R(x_\\ast) + \\ev{ - \\nabla f(x_\\ast), x - x_\\ast } \\leq R(x). \\]\n    Rearranging gives\n    \\[ -\\ev{ \\nabla f(x_\\ast), x - x_\\ast} \\leq R(x) - R(x_\\ast) \\]\n    Adding $f(x) - f(x_\\ast)$ to both sides we have,\n    \\[ f(x) - f(x_\\ast) - \\ev{ \\nabla f(x_\\ast), x - x_\\ast } \\leq f(x) + R(x) - \\br{ f(x_\\ast) + R(x_\\ast)} = F(x) - F(x_\\ast). \\]\n   Now note that the on the left hand side we have the Bregman divergence $D_{f} (x, x_\\ast)$.\n\\end{proof}\n\n\\begin{proof}\n    Let $x_\\ast$ be a minimizer of $F$. Using \\eqref{eq:atchade-lemma} from Lemma~\\ref{lemma:atchade-functional-value-bound} with $y = x_k - \\gamma_k g_k$, $x = x_k$ and $\\gamma = \\gamma_k$ gives\n    \\begin{align*}\n        -2 \\gamma_k \\br{ F(x_{k+1}) - F(x_\\ast) } &\\geq \\norm{x_{k+1} - x_\\ast}^2 - \\norm{x_k - x_\\ast}^2 + 2 \\gamma_k \\ev{ x_{k+1} - x_\\ast, g_k - \\nabla f(x_k) }.\n    \\end{align*}\n    Multiplying both sides by $-1$ results in\n    \\begin{align}        \n        \\label{eq:gen-prox-dec-proof-init}\n        2 \\gamma_k \\br{F(x_{k+1}) - F(x_\\ast)} &\\leq \\sqn{x_k - x_\\ast} - \\sqn{x_{k+1} - x_\\ast} + 2 \\gamma_k \\ev{x_{k+1} - x_\\ast, \\nabla f(x_k) - g_k}.\n    \\end{align}\n    Now focusing on the last term in the above and consider the straightforward decomposition\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-0}\n        \\ev{x_{k+1} - x_\\ast, \\nabla f(x_k) - g_k} &= \\ev{x_{k+1} - T_{\\gamma_k} (x_k), \\nabla f(x_k) - g_k } + \\ev{T_{\\gamma_k} (x_k) - x_\\ast, \\nabla f(x_k) - g_k}.\n     \\end{align}\n    By Cauchy Schwartz we have that\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-1}\n        \\ev{x_{k+1} - T_{\\gamma_k} (x_k), \\nabla f(x_k) - g_k} \\leq \\norm{ x_{k+1} - T_{\\gamma_k} (x_k) } \\norm{g_k - \\nabla f(x_k)}.\n    \\end{align}\n    Now using the nonexpansivity of the proximal operator\n    \\begin{align*}\n        \\norm{x_{k+1} - T_{\\gamma_k} (x_k)} &= \\norm{ \\prox_{\\gamma_k R} \\br{ x_k - \\gamma_k g_k } - \\prox_{\\gamma_k R} \\br{x_k - \\gamma_k \\nabla f(x_k)} } \\\\\n        &\\leq \\norm{ \\br{x_k - \\gamma_k g_k} - \\br{x_k - \\gamma_k \\nabla f(x_k)} } = \\gamma_k \\norm{g_k - \\nabla f(x_k)}.\n    \\end{align*}\n    Using this in \\eqref{eq:gen-prox-dec-proof-1}, we have\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-2}\n        \\ev{x_{k+1} - T_{\\gamma_k} (x_k), \\nabla f(x_k) - g_k} \\leq \\gamma_k \\norm{g_k - \\nabla f(x_k)}^2.\n    \\end{align}\n    Using \\eqref{eq:gen-prox-dec-proof-2} in \\eqref{eq:gen-prox-dec-proof-0} and taking expectation conditioned on $x_k$, and using $\\ec[k]{\\cdot} \\eqdef \\ec{\\cdot \\; | \\; x_k}$ for shorthand, we have\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-2-1}\n        \\ec[k]{\\ev{x_{k+1} - x_\\ast, g_k - \\nabla f(x_k)}} &\\leq \\gamma_k \\cdot \\ecn[k]{g_k - \\nabla f(x_k)} + \\ev{ T_{\\gamma_k} (x_k) - x_\\ast, \\underbrace{\\ec[k]{\\nabla f(x_k) - g_k}}_{= 0} } \\nonumber \\\\\n        &= \\gamma_k \\cdot \\ecn[k]{g_k - \\nabla f(x_k)}.\n    \\end{align}\n\tLet $r_k \\eqdef x_k-x_\\ast.$    Taking expectation conditioned on $x_k$ in \\eqref{eq:gen-prox-dec-proof-init} and using \\eqref{eq:gen-prox-dec-proof-2-1}, we have\n    \\begin{align*}\n        2 \\gamma_k \\ec[k]{ F(x_{k+1} - F(x_\\ast)) } &\\leq \\sqn{r_k} - \\ecn[k]{r_{k+1}} + 2 \\gamma_k^2 \\ecn[k]{g_k - \\nabla f(x_k)}.\n    \\end{align*}\n    Using \\eqref{eq:asm-gradient-opt-distance} from Assumption~\\ref{asm:main_assumption} we have\n    \\begin{align*}\n        2 \\gamma_k \\ec[k]{ F(x_{k+1}) - F(x_\\ast) } &\\leq \\sqn{r_k} - \\ecn[k]{r_{k+1}} + 2 \\gamma_k^2 \\br{ 2 A D_{f} (x_k, x_\\ast) + B \\sigma_k^2 + D_1}.\n    \\end{align*}\n    Let $V_k \\eqdef \\sqn{r_k} + 2 M \\gamma_k^2 \\sigma_{k}^2$ where $M = \\frac{B}{\\rho}$, then\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-3}\n        \\begin{split}\n            2 \\gamma_k \\ec[k]{ F(x_{k+1}) - F(x_\\ast) } \\leq V_k &- \\ec[k]{V_{k+1}} + 4 \\gamma_k^2 A D_{f} (x_k, x_\\ast) + 2 \\gamma_k^2 D_1 \\\\\n            &+ \\gamma_k^2 \\br{ 2B - 2 M } \\sigma_{k}^2 + 2 M \\gamma_{k+1}^2 \\ec{\\sigma_{k+1}^2}.\n        \\end{split}\n    \\end{align}\nSince $\\gamma_{k+1} \\leq \\gamma_k$ we have that\n\t\\begin{align}\n        \\label{eq:gen-prox-dec-proof-3.1}\n        \\begin{split}\n            2 \\gamma_{k+1} \\ec[k]{ F(x_{k+1}) - F(x_\\ast) } \\leq V_k &- \\ec[k]{V_{k+1}} + 4 \\gamma_k^2 A D_{f} (x_k, x_\\ast) + 2 \\gamma_k^2 D_1 \\\\\n            &+ \\gamma_k^2 \\br{ 2B - 2 M } \\sigma_{k}^2 + 2 M \\gamma_k^2 \\ec{\\sigma_{k+1}^2}.\n        \\end{split}\n    \\end{align}\n    \n    Using \\eqref{eq:asm-decreasing-noise} from Assumption~\\ref{asm:main_assumption}, we have\n        \\begin{align}\n        \\label{eq:gen-prox-dec-proof-4}\n            2 \\gamma_k^2 \\br{B - M} \\sigma_k^2 + 2 M \\gamma_k^2 \\ec[k]{\\sigma_{k+1}^2} & \\leq 2 \\gamma_k^2 \\br{ B - M + M (1 - \\rho) } \\sigma_k^2 + 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast)\\nonumber  \\\\\n            &\\quad + 2 M \\gamma_k^2 D_2\n \\nonumber \\\\\n       & = 2 \\gamma_k^2 \\underbrace{\\br{ B - \\rho M}}_{= 0} \\sigma_k^2 + 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast) +2 M \\gamma_k^2 D_2 \\nonumber \\\\\n       & \\leq 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast) +2 M \\gamma_k^2 D_2.\n    \\end{align}\n%    \\begin{align}\n%        \\label{eq:gen-prox-dec-proof-4}\n%        \\begin{split}\n%            2 \\gamma_k^2 \\br{B - M} \\sigma_k^2 + 2 M \\gamma_k^2 \\ec[k]{\\sigma_{k+1}^2} \\leq 2 \\gamma_k^2 \\br{ B - M + M (1 - \\rho) } \\sigma_k^2 &+ 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast) \\\\\n%            &+ 2 M \\gamma_k^2 D_2\n%        \\end{split} \\nonumber \\\\\n%        = 2 \\gamma_k^2 \\underbrace{\\br{ B - \\rho M}}_{= 0} \\sigma_k^2 + 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast) &+2 M \\gamma_k^2 D_2 \\nonumber \\\\\n%        \\leq 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast) &+2 M \\gamma_k^2 D_2.\n%    \\end{align}\n    Using \\eqref{eq:gen-prox-dec-proof-4} in \\eqref{eq:gen-prox-dec-proof-3} gives\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-5}\n        2 \\gamma_{k+1} \\ec[k]{ F(x_{k+1}) - F(x_\\ast) } &\\leq V_k - \\ec[k]{V_{k+1}} + 2 \\gamma_k^2 \\br{ 2 A + 2 M C } D_{f} (x_k, x_\\ast) + 2 \\gamma_k^2 \\br{ D_1 + M D_2 }.\n    \\end{align}\n    Let $\\eta \\eqdef 2A + 2 M C$. Using \\eqref{eq:bregman-divergence-bound} in \\eqref{eq:gen-prox-dec-proof-5} we have,\n    \\begin{align*}\n        2 \\gamma_{k+1} \\ec[k]{ F(x_{k+1}) - F(x_\\ast) } &\\leq V_k - \\ec[k]{V_{k+1}} + 2 \\gamma_k^2 \\eta \\br{ F(x_k) - F(x_\\ast) } + 2 \\gamma_k^2 \\br{ D_1 + M D_2 }.\n    \\end{align*}\n    Using the abbreviation $\\delta_k = F(x_k) - F(x_\\ast)$ gives\n    \\begin{align*}\n        2 \\gamma_{k+1} \\ec[k]{\\delta_{k+1}} &\\leq V_{k} - \\ec[k]{V_{k+1}} + 2 \\gamma_k^2 \\eta \\delta_k + 2 \\gamma_k^2 \\br{D_1 + M D_2}.\n    \\end{align*}\n    Taking expectation,\n   \\begin{align*}\n        2 \\gamma_{k+1} \\ec{\\delta_{k+1}} &\\leq \\ec{V_{k}} - \\ec{V_{k+1}} + 2 \\gamma_k^2 \\eta \\ec{\\delta_k} + 2 \\gamma_k^2 \\br{D_1 + M D_2},\n    \\end{align*}\n summing over $k =0,\\ldots, t-1$ and using telescopic cancellation gives\n    \\begin{align*}\n        2  \\sum_{k=1}^{t}\\gamma_k\\ec{\\delta_{k}} &\\leq V_{0} - \\ec{V_{t}} + 2  \\eta \\sum_{k=0}^{t-1}\\gamma_k^2 \\ec{\\delta_k} + 2 \\br{ D_1 + M D_2 } \\sum_{k=0}^{t-1}\\gamma_k^2.\n    \\end{align*}\nAdding $2\\gamma_0\\delta_0$ to both sides of the above inequality and rearranging,\n\\begin{align*}\n2  \\sum_{k=0}^{t-1}\\gamma_k(1-\\eta\\gamma_k)\\ec{\\delta_{k}} &\\leq V_{0} - \\ec{V_{t}} + 2 \\gamma_0 \\delta_0 + 2 \\br{ D_1 + M D_2 } \\sum_{k=0}^{t-1}\\gamma_k^2\n\\end{align*}\n\n    where we also used that $V_t \\geq 0$ and $\\delta_t \\geq 0.$\n   \n    By the choice of $\\gamma_0$ we have $1 - \\gamma_0 \\eta > 0$, and since $(\\gamma_i)_i$ is a decreasing sequence, we have $1 - \\gamma_i \\eta > 0$ for all $i$. Hence dividing both sides by $2\\sum\\limits_{i=0}^{t-1}\\br{1 - \\gamma_i\\eta} \\gamma_i$, we have\n    \\begin{align*}\n        \\sum_{k=0}^{t-1} w_k \\ec{\\delta_k} \\leq \\frac{V_0 + 2 \\gamma_0 \\delta_0}{2\\sum_{i=0}^{t-1}\\br{1 - \\gamma_i\\eta} \\gamma_i} + \\br{D_1 + 2 M D_2} \\frac{\\sum_{k=0}^{t-1}\\gamma_k^2}{\\sum_{i=1}^{t}\\br{1-\\gamma_i\\eta}\\gamma_i},\n    \\end{align*}\n    where $w_k \\eqdef \\frac{\\br{ 1 - \\gamma_k \\eta }  \\gamma_k}{\\sum_{i=0}^{t-1}\\br{1 - \\gamma_i\\eta} \\gamma_i}$ for all $k \\in \\left\\{0,\\dots,t-1\\right\\}$. Note that $\\sum_{k=0}^{t-1} w_k = 1$ and $w_k \\geq 0$ for all $k \\in \\left\\{0,\\dots,t-1\\right\\}$. Hence, since $F$ is convex, we can use Jensen's inequality to conclude\n    \\begin{align*}\n        \\ec{F(\\bar{x}^k) - F(x_\\ast)} &= \\ec{F\\br{ \\sum\\limits_{k=0}^{t-1} w_k x_k} - F(x_\\ast)} \\\\\n        &\\leq \\sum_{k=0}^{t-1}w_k \\ec{\\delta_k} \\leq \\frac{V_0 + 2\\gamma_0 \\delta_0}{2\\sum_{i=0}^{t-1}\\br{1 - \\gamma_i\\eta} \\gamma_i} + \\frac{\\br{D_1 + 2 M D_2} \\sum_{k=0}^{t-1}\\gamma_k^2}{\\sum_{i=0}^{t-1}\\br{1-\\gamma_i\\eta}\\gamma_i}.\n    \\end{align*}\n    Writing out the definition of $\\delta_0$ yields the theorem's statement.\n\\end{proof}\n\n\\begin{lemma} Let $R: \\R^d \\mapsto \\R$ be a convex lower semi-continuous function. For $z,y \\in \\R^d$ and $\\gamma>0$. With $p = \\prox_{\\gamma g} (y)$ we have that for\n\\begin{equation} \\label{eq:2ndproxtheo}\ng(p) - g(z) \\leq -\\frac{1}{\\gamma} \\dotprod{p - y, p -z}.\n\\end{equation}\n\\end{lemma}\n\n\\begin{proof}\nThis is classic result, see for example the ``Second Prox Theorem'' in Section 6.5 in~\\cite{beck2017first}.\n%See Propositions 4.2., 12.26 and 12.27 in~\\cite{combetteeBook2011}\n\\end{proof}\n\n\\subsection{Proof of Lemma~\\ref{lemma:atchade-functional-value-bound}}\n\\label{app:sec:prooflemma}\n%Here\n%\\begin{lemma}\n% \\[  F(p) - F(z)  \\leq -\\frac{1}{2\\gamma}\\sqn{p - z } - \\frac{1}{\\gamma}  \\ev{x - \\gamma \\nabla f(x) - y, p-z} + \\frac{1}{2\\gamma}\\sqn{z - x}. \\]\n%\\end{lemma}\n\nThis proof is based on the proof of  Lemma~8 in \\cite{Atchade17}. The only difference is that in  \\cite{Atchade17} the authors assume that $f$ is convex.\nIndeed, using the convexity of $f$\n \\[f(x) -f(x_\\ast) \\geq  - \\langle \\nabla f(x), x_\\ast - x \\rangle\\]\n%\\[f(x)- f(z) \\leq  - \\dotprod{\\nabla f(x), z-x} \\]\nin combination with~\\eqref{eq:2ndproxtheo} where $z = x_\\ast$ gives\n\\begin{align*}\nf(x)+g(p)- F(x_\\ast) & \\leq -\\frac{1}{\\gamma} \\dotprod{p - y, p -x_\\ast}- \\dotprod{\\nabla f(x), x_\\ast-x}. \n\\end{align*}\nNow using smoothness\n\\[ f(p) -f(x) \\leq \\dotprod{\\nabla f(x),p -x} + \\frac{1}{2\\gamma} \\norm{p-x}^2,\\]\ngives\n%& \\leq \\frac{1}{\\gamma} \\dotprod{ y-p +\\gamma \\nabla f(x) , p -z}- \\dotprod{\\nabla f(x), p-x}+\\dotprod{\\nabla f(x),p -x} + \\frac{1}{2\\gamma} \\norm{p-x}^2\\\\\n\\begin{align}\nF(p)- F(x_\\ast) \n&\\leq-\\frac{1}{\\gamma} \\dotprod{p - y, p -x_\\ast}- \\dotprod{\\nabla f(x), x_\\ast-x} +\\dotprod{\\nabla f(x),p -x} + \\frac{1}{2\\gamma} \\norm{p-x}^2 \\nonumber \\\\\n&= -\\frac{1}{\\gamma} \\dotprod{p - y, p -x_\\ast}+\\dotprod{\\nabla f(x),p -x_\\ast} + \\frac{1}{2\\gamma} \\norm{p-x}^2 \\nonumber \\\\\n&= -\\frac{1}{\\gamma} \\dotprod{p -\\gamma \\nabla f(x)- y, p -x_\\ast} + \\frac{1}{2\\gamma} \\norm{p-x}^2 \\nonumber  \\\\\n&= -\\frac{1}{\\gamma} \\dotprod{p-x+x -\\gamma \\nabla f(x)- y, p -x_\\ast} + \\frac{1}{2\\gamma} \\norm{p-x}^2 \\nonumber \\\\\n&= -\\frac{1}{\\gamma} \\dotprod{p-x, p -x_\\ast} -\\frac{1}{\\gamma} \\dotprod{x -\\gamma \\nabla f(x)- y, p -x_\\ast} + \\frac{1}{2\\gamma} \\norm{p-x}^2 . \\label{eq:tenaoineria}\n\\end{align}\nUsing that\n\\begin{equation}\n-2 \\dotprod{p-x, p -x_\\ast}  + \\norm{p-x}^2 = -\\norm{p-x_\\ast}^2 + \\norm{z-x}^2,\n\\end{equation}\nin combination with~\\eqref{eq:tenaoineria} gives \n \\[  F(p) - F(x_\\ast)  \\leq -\\frac{1}{2\\gamma}\\sqn{p - x_\\ast} - \\frac{1}{\\gamma}  \\ev{x - \\gamma \\nabla f(x) - y, p-x_\\ast} + \\frac{1}{2\\gamma}\\sqn{x_\\ast - x}. \\]\n Now it remains to multiply both sides by $-2\\gamma$ to arrive at~\\eqref{eq:atchade-lemma}. \n \n \n%\\section{Maple Code for Section \\ref{sec:optimal-b-SAGA}}\\label{sec:maple-optimal-b-SAGA}\n\n%\\begin{figure*}[h]\n%\\begin{subfigure}[b]{\\textwidth}\n%\\includegraphics[width=\\textwidth]{maple_def.png}\n%\\end{subfigure}\\\\\n%\\begin{subfigure}[b]{\\textwidth}\n%\\includegraphics[width=\\textwidth]{maple_diff_1.png}\n%\\end{subfigure}\\\\\n%\\begin{subfigure}[b]{\\textwidth}\n%\\includegraphics[width=\\textwidth]{maple_disc.png}\n%\\end{subfigure}\\\\\n%\\begin{subfigure}[b]{\\textwidth}\n%\\includegraphics[width=\\textwidth]{maple_diff_2.png}\n%\\end{subfigure}\\\\\n%\\begin{subfigure}[b]{\\textwidth}\n%\\includegraphics[width=\\textwidth]{maple_solutions.png}\n%\\end{subfigure}\n%\\end{figure*}\n\n\\end{appendices}\n\n","units":[{"start":82658,"end":82979,"start_line":1256,"end_line":1261,"raw_tex":"\\begin{lemma}[Bounding the gradient variance] Assuming that the $g_k$ are unbiased and that Assumption~\\ref{asm:main_assumption} holds, we have\n    \\begin{equation}\n        \\label{eq:lma-gradient-variance}\n        \\ecn{g_k - \\nabla f(x_k)}  \\leq 2 A D_{f} (x_k, x_\\ast) + B \\sigma_k^2 + D_1\n    \\end{equation}\n\\end{lemma}","sha256_utf8":"128cc5eb88bb9791f4f8105c2247097b6916a54e3196a9dbc2865bb27090ef91"},{"start":82980,"end":83521,"start_line":1262,"end_line":1270,"raw_tex":"\\begin{proof}\n    Starting from the left hand side of \\eqref{eq:lma-gradient-variance}, we have\n    \\begin{align*}\n        \\ecn{g_k - \\nabla f(x_k)} &= \\ecn{ g_k - \\nabla f(x_\\ast) - \\br{ \\nabla f(x_k) - \\nabla f(x_\\ast) } } \\\\\n        &= \\ecn{ g_k - \\nabla f(x_\\ast) - \\ec{ g_k - \\nabla f(x_\\ast) } } \\\\\n        &\\leq \\ecn{g_k - \\nabla f(x_\\ast)} \\leq 2 A D_{f} (x_k, x_\\ast) + B \\sigma_k^2 + D_1,\n    \\end{align*}\n    where we used that $\\ecn{ X - \\ec{X} } = \\ecn{X} - \\norm{\\ec{X}}^2 \\leq \\ecn{X}$ for any random variable $X$.\n\\end{proof}","sha256_utf8":"4951fd5e93dcc697f8781e748c6692fa8fb03541822f026d9b580e0a57fdb733"},{"start":83523,"end":83980,"start_line":1272,"end_line":1279,"raw_tex":"\\begin{lemma}\n    \\label{lemma:atchade-functional-value-bound}\n    Suppose that Assumption~\\ref{asm:function-class} holds and let $\\gamma \\in \\left (0, \\frac{1}{L} \\right ]$, then for all $x, y \\in \\R^d$ and $p = \\prox_{\\gamma g}(y)$ we have,\n    \\begin{equation}\n        \\label{eq:atchade-lemma}\n        - 2 \\gamma \\br{ F(p) - F(x_\\ast) } \\geq \\sqn{p - z } + 2  \\ev{p - x_\\ast, x - \\gamma \\nabla f(x) - y} - \\sqn{x_\\ast - x}.\n    \\end{equation}\n\\end{lemma}","sha256_utf8":"a3e0ab470d909a483dc748a3ebb6c725da06ce289a0af374a33a080ca6dceaff"},{"start":83981,"end":84066,"start_line":1280,"end_line":1282,"raw_tex":"\\begin{proof}\n    We leave the proof to Section~\\ref{app:sec:prooflemma}.\n\\end{proof}","sha256_utf8":"f35d91a5c7138a80201388bc0967c76b56554a44ad53e3bb118f99ad60be8397"},{"start":84068,"end":84290,"start_line":1284,"end_line":1290,"raw_tex":"\\begin{lemma}\n    For any $x \\in \\R^d$ and minimizer $x_\\ast$ of $F$ we have,\n    \\begin{equation}\n        \\label{eq:bregman-divergence-bound}\n        D_{f} (x, x_\\ast) \\leq F(x) - F(x_\\ast).\n    \\end{equation}\n\\end{lemma}","sha256_utf8":"bd2644791b2b63beca42894b77bbeb40a2bb535dded04ed9467494090a26450a"},{"start":84291,"end":84901,"start_line":1291,"end_line":1299,"raw_tex":"\\begin{proof}\n    Because $x_\\ast$ is a minimizer of $F$ we have that $- \\nabla f(x_\\ast)  \\in \\partial R(x_\\ast)$. By the definition of subgradients we have\n    \\[ R(x_\\ast) + \\ev{ - \\nabla f(x_\\ast), x - x_\\ast } \\leq R(x). \\]\n    Rearranging gives\n    \\[ -\\ev{ \\nabla f(x_\\ast), x - x_\\ast} \\leq R(x) - R(x_\\ast) \\]\n    Adding $f(x) - f(x_\\ast)$ to both sides we have,\n    \\[ f(x) - f(x_\\ast) - \\ev{ \\nabla f(x_\\ast), x - x_\\ast } \\leq f(x) + R(x) - \\br{ f(x_\\ast) + R(x_\\ast)} = F(x) - F(x_\\ast). \\]\n   Now note that the on the left hand side we have the Bregman divergence $D_{f} (x, x_\\ast)$.\n\\end{proof}","sha256_utf8":"a3bac6a8fa3473418588ff93256152ca5c644a781693b3abbe0088509c528458"},{"start":86130,"end":94263,"start_line":1322,"end_line":1439,"raw_tex":"\\begin{proof}\n    Let $x_\\ast$ be a minimizer of $F$. Using \\eqref{eq:atchade-lemma} from Lemma~\\ref{lemma:atchade-functional-value-bound} with $y = x_k - \\gamma_k g_k$, $x = x_k$ and $\\gamma = \\gamma_k$ gives\n    \\begin{align*}\n        -2 \\gamma_k \\br{ F(x_{k+1}) - F(x_\\ast) } &\\geq \\norm{x_{k+1} - x_\\ast}^2 - \\norm{x_k - x_\\ast}^2 + 2 \\gamma_k \\ev{ x_{k+1} - x_\\ast, g_k - \\nabla f(x_k) }.\n    \\end{align*}\n    Multiplying both sides by $-1$ results in\n    \\begin{align}        \n        \\label{eq:gen-prox-dec-proof-init}\n        2 \\gamma_k \\br{F(x_{k+1}) - F(x_\\ast)} &\\leq \\sqn{x_k - x_\\ast} - \\sqn{x_{k+1} - x_\\ast} + 2 \\gamma_k \\ev{x_{k+1} - x_\\ast, \\nabla f(x_k) - g_k}.\n    \\end{align}\n    Now focusing on the last term in the above and consider the straightforward decomposition\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-0}\n        \\ev{x_{k+1} - x_\\ast, \\nabla f(x_k) - g_k} &= \\ev{x_{k+1} - T_{\\gamma_k} (x_k), \\nabla f(x_k) - g_k } + \\ev{T_{\\gamma_k} (x_k) - x_\\ast, \\nabla f(x_k) - g_k}.\n     \\end{align}\n    By Cauchy Schwartz we have that\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-1}\n        \\ev{x_{k+1} - T_{\\gamma_k} (x_k), \\nabla f(x_k) - g_k} \\leq \\norm{ x_{k+1} - T_{\\gamma_k} (x_k) } \\norm{g_k - \\nabla f(x_k)}.\n    \\end{align}\n    Now using the nonexpansivity of the proximal operator\n    \\begin{align*}\n        \\norm{x_{k+1} - T_{\\gamma_k} (x_k)} &= \\norm{ \\prox_{\\gamma_k R} \\br{ x_k - \\gamma_k g_k } - \\prox_{\\gamma_k R} \\br{x_k - \\gamma_k \\nabla f(x_k)} } \\\\\n        &\\leq \\norm{ \\br{x_k - \\gamma_k g_k} - \\br{x_k - \\gamma_k \\nabla f(x_k)} } = \\gamma_k \\norm{g_k - \\nabla f(x_k)}.\n    \\end{align*}\n    Using this in \\eqref{eq:gen-prox-dec-proof-1}, we have\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-2}\n        \\ev{x_{k+1} - T_{\\gamma_k} (x_k), \\nabla f(x_k) - g_k} \\leq \\gamma_k \\norm{g_k - \\nabla f(x_k)}^2.\n    \\end{align}\n    Using \\eqref{eq:gen-prox-dec-proof-2} in \\eqref{eq:gen-prox-dec-proof-0} and taking expectation conditioned on $x_k$, and using $\\ec[k]{\\cdot} \\eqdef \\ec{\\cdot \\; | \\; x_k}$ for shorthand, we have\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-2-1}\n        \\ec[k]{\\ev{x_{k+1} - x_\\ast, g_k - \\nabla f(x_k)}} &\\leq \\gamma_k \\cdot \\ecn[k]{g_k - \\nabla f(x_k)} + \\ev{ T_{\\gamma_k} (x_k) - x_\\ast, \\underbrace{\\ec[k]{\\nabla f(x_k) - g_k}}_{= 0} } \\nonumber \\\\\n        &= \\gamma_k \\cdot \\ecn[k]{g_k - \\nabla f(x_k)}.\n    \\end{align}\n\tLet $r_k \\eqdef x_k-x_\\ast.$    Taking expectation conditioned on $x_k$ in \\eqref{eq:gen-prox-dec-proof-init} and using \\eqref{eq:gen-prox-dec-proof-2-1}, we have\n    \\begin{align*}\n        2 \\gamma_k \\ec[k]{ F(x_{k+1} - F(x_\\ast)) } &\\leq \\sqn{r_k} - \\ecn[k]{r_{k+1}} + 2 \\gamma_k^2 \\ecn[k]{g_k - \\nabla f(x_k)}.\n    \\end{align*}\n    Using \\eqref{eq:asm-gradient-opt-distance} from Assumption~\\ref{asm:main_assumption} we have\n    \\begin{align*}\n        2 \\gamma_k \\ec[k]{ F(x_{k+1}) - F(x_\\ast) } &\\leq \\sqn{r_k} - \\ecn[k]{r_{k+1}} + 2 \\gamma_k^2 \\br{ 2 A D_{f} (x_k, x_\\ast) + B \\sigma_k^2 + D_1}.\n    \\end{align*}\n    Let $V_k \\eqdef \\sqn{r_k} + 2 M \\gamma_k^2 \\sigma_{k}^2$ where $M = \\frac{B}{\\rho}$, then\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-3}\n        \\begin{split}\n            2 \\gamma_k \\ec[k]{ F(x_{k+1}) - F(x_\\ast) } \\leq V_k &- \\ec[k]{V_{k+1}} + 4 \\gamma_k^2 A D_{f} (x_k, x_\\ast) + 2 \\gamma_k^2 D_1 \\\\\n            &+ \\gamma_k^2 \\br{ 2B - 2 M } \\sigma_{k}^2 + 2 M \\gamma_{k+1}^2 \\ec{\\sigma_{k+1}^2}.\n        \\end{split}\n    \\end{align}\nSince $\\gamma_{k+1} \\leq \\gamma_k$ we have that\n\t\\begin{align}\n        \\label{eq:gen-prox-dec-proof-3.1}\n        \\begin{split}\n            2 \\gamma_{k+1} \\ec[k]{ F(x_{k+1}) - F(x_\\ast) } \\leq V_k &- \\ec[k]{V_{k+1}} + 4 \\gamma_k^2 A D_{f} (x_k, x_\\ast) + 2 \\gamma_k^2 D_1 \\\\\n            &+ \\gamma_k^2 \\br{ 2B - 2 M } \\sigma_{k}^2 + 2 M \\gamma_k^2 \\ec{\\sigma_{k+1}^2}.\n        \\end{split}\n    \\end{align}\n    \n    Using \\eqref{eq:asm-decreasing-noise} from Assumption~\\ref{asm:main_assumption}, we have\n        \\begin{align}\n        \\label{eq:gen-prox-dec-proof-4}\n            2 \\gamma_k^2 \\br{B - M} \\sigma_k^2 + 2 M \\gamma_k^2 \\ec[k]{\\sigma_{k+1}^2} & \\leq 2 \\gamma_k^2 \\br{ B - M + M (1 - \\rho) } \\sigma_k^2 + 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast)\\nonumber  \\\\\n            &\\quad + 2 M \\gamma_k^2 D_2\n \\nonumber \\\\\n       & = 2 \\gamma_k^2 \\underbrace{\\br{ B - \\rho M}}_{= 0} \\sigma_k^2 + 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast) +2 M \\gamma_k^2 D_2 \\nonumber \\\\\n       & \\leq 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast) +2 M \\gamma_k^2 D_2.\n    \\end{align}\n%    \\begin{align}\n%        \\label{eq:gen-prox-dec-proof-4}\n%        \\begin{split}\n%            2 \\gamma_k^2 \\br{B - M} \\sigma_k^2 + 2 M \\gamma_k^2 \\ec[k]{\\sigma_{k+1}^2} \\leq 2 \\gamma_k^2 \\br{ B - M + M (1 - \\rho) } \\sigma_k^2 &+ 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast) \\\\\n%            &+ 2 M \\gamma_k^2 D_2\n%        \\end{split} \\nonumber \\\\\n%        = 2 \\gamma_k^2 \\underbrace{\\br{ B - \\rho M}}_{= 0} \\sigma_k^2 + 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast) &+2 M \\gamma_k^2 D_2 \\nonumber \\\\\n%        \\leq 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast) &+2 M \\gamma_k^2 D_2.\n%    \\end{align}\n    Using \\eqref{eq:gen-prox-dec-proof-4} in \\eqref{eq:gen-prox-dec-proof-3} gives\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-5}\n        2 \\gamma_{k+1} \\ec[k]{ F(x_{k+1}) - F(x_\\ast) } &\\leq V_k - \\ec[k]{V_{k+1}} + 2 \\gamma_k^2 \\br{ 2 A + 2 M C } D_{f} (x_k, x_\\ast) + 2 \\gamma_k^2 \\br{ D_1 + M D_2 }.\n    \\end{align}\n    Let $\\eta \\eqdef 2A + 2 M C$. Using \\eqref{eq:bregman-divergence-bound} in \\eqref{eq:gen-prox-dec-proof-5} we have,\n    \\begin{align*}\n        2 \\gamma_{k+1} \\ec[k]{ F(x_{k+1}) - F(x_\\ast) } &\\leq V_k - \\ec[k]{V_{k+1}} + 2 \\gamma_k^2 \\eta \\br{ F(x_k) - F(x_\\ast) } + 2 \\gamma_k^2 \\br{ D_1 + M D_2 }.\n    \\end{align*}\n    Using the abbreviation $\\delta_k = F(x_k) - F(x_\\ast)$ gives\n    \\begin{align*}\n        2 \\gamma_{k+1} \\ec[k]{\\delta_{k+1}} &\\leq V_{k} - \\ec[k]{V_{k+1}} + 2 \\gamma_k^2 \\eta \\delta_k + 2 \\gamma_k^2 \\br{D_1 + M D_2}.\n    \\end{align*}\n    Taking expectation,\n   \\begin{align*}\n        2 \\gamma_{k+1} \\ec{\\delta_{k+1}} &\\leq \\ec{V_{k}} - \\ec{V_{k+1}} + 2 \\gamma_k^2 \\eta \\ec{\\delta_k} + 2 \\gamma_k^2 \\br{D_1 + M D_2},\n    \\end{align*}\n summing over $k =0,\\ldots, t-1$ and using telescopic cancellation gives\n    \\begin{align*}\n        2  \\sum_{k=1}^{t}\\gamma_k\\ec{\\delta_{k}} &\\leq V_{0} - \\ec{V_{t}} + 2  \\eta \\sum_{k=0}^{t-1}\\gamma_k^2 \\ec{\\delta_k} + 2 \\br{ D_1 + M D_2 } \\sum_{k=0}^{t-1}\\gamma_k^2.\n    \\end{align*}\nAdding $2\\gamma_0\\delta_0$ to both sides of the above inequality and rearranging,\n\\begin{align*}\n2  \\sum_{k=0}^{t-1}\\gamma_k(1-\\eta\\gamma_k)\\ec{\\delta_{k}} &\\leq V_{0} - \\ec{V_{t}} + 2 \\gamma_0 \\delta_0 + 2 \\br{ D_1 + M D_2 } \\sum_{k=0}^{t-1}\\gamma_k^2\n\\end{align*}\n\n    where we also used that $V_t \\geq 0$ and $\\delta_t \\geq 0.$\n   \n    By the choice of $\\gamma_0$ we have $1 - \\gamma_0 \\eta > 0$, and since $(\\gamma_i)_i$ is a decreasing sequence, we have $1 - \\gamma_i \\eta > 0$ for all $i$. Hence dividing both sides by $2\\sum\\limits_{i=0}^{t-1}\\br{1 - \\gamma_i\\eta} \\gamma_i$, we have\n    \\begin{align*}\n        \\sum_{k=0}^{t-1} w_k \\ec{\\delta_k} \\leq \\frac{V_0 + 2 \\gamma_0 \\delta_0}{2\\sum_{i=0}^{t-1}\\br{1 - \\gamma_i\\eta} \\gamma_i} + \\br{D_1 + 2 M D_2} \\frac{\\sum_{k=0}^{t-1}\\gamma_k^2}{\\sum_{i=1}^{t}\\br{1-\\gamma_i\\eta}\\gamma_i},\n    \\end{align*}\n    where $w_k \\eqdef \\frac{\\br{ 1 - \\gamma_k \\eta }  \\gamma_k}{\\sum_{i=0}^{t-1}\\br{1 - \\gamma_i\\eta} \\gamma_i}$ for all $k \\in \\left\\{0,\\dots,t-1\\right\\}$. Note that $\\sum_{k=0}^{t-1} w_k = 1$ and $w_k \\geq 0$ for all $k \\in \\left\\{0,\\dots,t-1\\right\\}$. Hence, since $F$ is convex, we can use Jensen's inequality to conclude\n    \\begin{align*}\n        \\ec{F(\\bar{x}^k) - F(x_\\ast)} &= \\ec{F\\br{ \\sum\\limits_{k=0}^{t-1} w_k x_k} - F(x_\\ast)} \\\\\n        &\\leq \\sum_{k=0}^{t-1}w_k \\ec{\\delta_k} \\leq \\frac{V_0 + 2\\gamma_0 \\delta_0}{2\\sum_{i=0}^{t-1}\\br{1 - \\gamma_i\\eta} \\gamma_i} + \\frac{\\br{D_1 + 2 M D_2} \\sum_{k=0}^{t-1}\\gamma_k^2}{\\sum_{i=0}^{t-1}\\br{1-\\gamma_i\\eta}\\gamma_i}.\n    \\end{align*}\n    Writing out the definition of $\\delta_0$ yields the theorem's statement.\n\\end{proof}","sha256_utf8":"cf8bbef3661ce0b2b7bdbb1d051159fe6be8965877eeca3a2a75000620b0bfa3"},{"start":110661,"end":110952,"start_line":1765,"end_line":1769,"raw_tex":"\\begin{lemma} Let $R: \\R^d \\mapsto \\R$ be a convex lower semi-continuous function. For $z,y \\in \\R^d$ and $\\gamma>0$. With $p = \\prox_{\\gamma g} (y)$ we have that for\n\\begin{equation} \\label{eq:2ndproxtheo}\ng(p) - g(z) \\leq -\\frac{1}{\\gamma} \\dotprod{p - y, p -z}.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"eeeb8374a40dce9bf45c4eb851bcb2f6d3a518368340aebe668b72851bb86181"},{"start":110953,"end":111154,"start_line":1770,"end_line":1773,"raw_tex":"\\begin{proof}\nThis is classic result, see for example the ``Second Prox Theorem'' in Section 6.5 in~\\cite{beck2017first}.\n%See Propositions 4.2., 12.26 and 12.27 in~\\cite{combetteeBook2011}\n\\end{proof}","sha256_utf8":"ae043053c5991c2af28a69e0759744417546b0f6267d1f92ead1ca948c940ec0"},{"start":111157,"end":114055,"start_line":1776,"end_line":1833,"raw_tex":"\\subsection{Proof of Lemma~\\ref{lemma:atchade-functional-value-bound}}\n\\label{app:sec:prooflemma}\n%Here\n%\\begin{lemma}\n% \\[  F(p) - F(z)  \\leq -\\frac{1}{2\\gamma}\\sqn{p - z } - \\frac{1}{\\gamma}  \\ev{x - \\gamma \\nabla f(x) - y, p-z} + \\frac{1}{2\\gamma}\\sqn{z - x}. \\]\n%\\end{lemma}\n\nThis proof is based on the proof of  Lemma~8 in \\cite{Atchade17}. The only difference is that in  \\cite{Atchade17} the authors assume that $f$ is convex.\nIndeed, using the convexity of $f$\n \\[f(x) -f(x_\\ast) \\geq  - \\langle \\nabla f(x), x_\\ast - x \\rangle\\]\n%\\[f(x)- f(z) \\leq  - \\dotprod{\\nabla f(x), z-x} \\]\nin combination with~\\eqref{eq:2ndproxtheo} where $z = x_\\ast$ gives\n\\begin{align*}\nf(x)+g(p)- F(x_\\ast) & \\leq -\\frac{1}{\\gamma} \\dotprod{p - y, p -x_\\ast}- \\dotprod{\\nabla f(x), x_\\ast-x}. \n\\end{align*}\nNow using smoothness\n\\[ f(p) -f(x) \\leq \\dotprod{\\nabla f(x),p -x} + \\frac{1}{2\\gamma} \\norm{p-x}^2,\\]\ngives\n%& \\leq \\frac{1}{\\gamma} \\dotprod{ y-p +\\gamma \\nabla f(x) , p -z}- \\dotprod{\\nabla f(x), p-x}+\\dotprod{\\nabla f(x),p -x} + \\frac{1}{2\\gamma} \\norm{p-x}^2\\\\\n\\begin{align}\nF(p)- F(x_\\ast) \n&\\leq-\\frac{1}{\\gamma} \\dotprod{p - y, p -x_\\ast}- \\dotprod{\\nabla f(x), x_\\ast-x} +\\dotprod{\\nabla f(x),p -x} + \\frac{1}{2\\gamma} \\norm{p-x}^2 \\nonumber \\\\\n&= -\\frac{1}{\\gamma} \\dotprod{p - y, p -x_\\ast}+\\dotprod{\\nabla f(x),p -x_\\ast} + \\frac{1}{2\\gamma} \\norm{p-x}^2 \\nonumber \\\\\n&= -\\frac{1}{\\gamma} \\dotprod{p -\\gamma \\nabla f(x)- y, p -x_\\ast} + \\frac{1}{2\\gamma} \\norm{p-x}^2 \\nonumber  \\\\\n&= -\\frac{1}{\\gamma} \\dotprod{p-x+x -\\gamma \\nabla f(x)- y, p -x_\\ast} + \\frac{1}{2\\gamma} \\norm{p-x}^2 \\nonumber \\\\\n&= -\\frac{1}{\\gamma} \\dotprod{p-x, p -x_\\ast} -\\frac{1}{\\gamma} \\dotprod{x -\\gamma \\nabla f(x)- y, p -x_\\ast} + \\frac{1}{2\\gamma} \\norm{p-x}^2 . \\label{eq:tenaoineria}\n\\end{align}\nUsing that\n\\begin{equation}\n-2 \\dotprod{p-x, p -x_\\ast}  + \\norm{p-x}^2 = -\\norm{p-x_\\ast}^2 + \\norm{z-x}^2,\n\\end{equation}\nin combination with~\\eqref{eq:tenaoineria} gives \n \\[  F(p) - F(x_\\ast)  \\leq -\\frac{1}{2\\gamma}\\sqn{p - x_\\ast} - \\frac{1}{\\gamma}  \\ev{x - \\gamma \\nabla f(x) - y, p-x_\\ast} + \\frac{1}{2\\gamma}\\sqn{x_\\ast - x}. \\]\n Now it remains to multiply both sides by $-2\\gamma$ to arrive at~\\eqref{eq:atchade-lemma}. \n \n \n%\\section{Maple Code for Section \\ref{sec:optimal-b-SAGA}}\\label{sec:maple-optimal-b-SAGA}\n\n%\\begin{figure*}[h]\n%\\begin{subfigure}[b]{\\textwidth}\n%\\includegraphics[width=\\textwidth]{maple_def.png}\n%\\end{subfigure}\\\\\n%\\begin{subfigure}[b]{\\textwidth}\n%\\includegraphics[width=\\textwidth]{maple_diff_1.png}\n%\\end{subfigure}\\\\\n%\\begin{subfigure}[b]{\\textwidth}\n%\\includegraphics[width=\\textwidth]{maple_disc.png}\n%\\end{subfigure}\\\\\n%\\begin{subfigure}[b]{\\textwidth}\n%\\includegraphics[width=\\textwidth]{maple_diff_2.png}\n%\\end{subfigure}\\\\\n%\\begin{subfigure}[b]{\\textwidth}\n%\\includegraphics[width=\\textwidth]{maple_solutions.png}\n%\\end{subfigure}\n%\\end{figure*}\n\n\\end{appendices}\n\n","sha256_utf8":"5cd995655c0a14ad5ab26904119d5920d36739e4db458c568e3925d8ea7311f7"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"随机近端梯度统一收敛界","source_units":[{"start":27468,"end":28405,"start_line":415,"end_line":425,"raw_tex":"\\begin{theorem}\n    \\label{thm:main-prox-dec}\n    Suppose that Assumptions~\\ref{asm:function-class} and~\\ref{asm:main_assumption} hold. Let $M \\eqdef B/\\rho$ and let $(\\gamma_k)_{k\\geq 0}$ be a decreasing, strictly positive sequence of step sizes chosen such that\n    \\[ 0 < \\gamma_0 < \\min \\left\\{ \\frac{1}{2 (A + MC)}, \\frac{1}{L} \\right\\}. \\]\nThe iterates given by \\eqref{eq:sgd-proximal-iterates} satisfy\n    \\begin{align}\n        \\label{eq:main-thm-convergence-bound}\n        \\ec{F(\\bar{x}_t) - F(x_\\ast)} \\leq \\frac{\\sqn{x_0 - x_*} + 2\\gamma_0\\br{\\delta_0 + \\gamma_0M\\sigma_0^2} + 2\\br{D_1 + 2 M D_2}\\sum \\limits_{k=0}^{t-1}\\gamma_k^2}{2\\sum \\limits_{i=0}^{t-1}\\br{1 - 2\\gamma_i\\br{A+MC}} \\gamma_i},\n    \\end{align}\n    where $\\bar{x}_t \\eqdef \\sum\\limits_{k=0}^{t-1} \\frac{\\br{ 1 - 2\\gamma_k(A+MC)}  \\gamma_k}{\\sum \\limits_{i=0}^{t-1}\\br{1 - 2\\gamma_i(A+MC)} \\gamma_i}x_k$ and ${\\delta_0 \\eqdef F(x_0) - F(x_\\ast)}$.\n\\end{theorem}","sha256_utf8":"d75fb32f9e4085edbd080f4688e2bbfb3a603016f7c9c6e32bdc8f5c4dbd8c9e"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"统一收敛定理完整证明","source_units":[{"start":86130,"end":94263,"start_line":1322,"end_line":1439,"raw_tex":"\\begin{proof}\n    Let $x_\\ast$ be a minimizer of $F$. Using \\eqref{eq:atchade-lemma} from Lemma~\\ref{lemma:atchade-functional-value-bound} with $y = x_k - \\gamma_k g_k$, $x = x_k$ and $\\gamma = \\gamma_k$ gives\n    \\begin{align*}\n        -2 \\gamma_k \\br{ F(x_{k+1}) - F(x_\\ast) } &\\geq \\norm{x_{k+1} - x_\\ast}^2 - \\norm{x_k - x_\\ast}^2 + 2 \\gamma_k \\ev{ x_{k+1} - x_\\ast, g_k - \\nabla f(x_k) }.\n    \\end{align*}\n    Multiplying both sides by $-1$ results in\n    \\begin{align}        \n        \\label{eq:gen-prox-dec-proof-init}\n        2 \\gamma_k \\br{F(x_{k+1}) - F(x_\\ast)} &\\leq \\sqn{x_k - x_\\ast} - \\sqn{x_{k+1} - x_\\ast} + 2 \\gamma_k \\ev{x_{k+1} - x_\\ast, \\nabla f(x_k) - g_k}.\n    \\end{align}\n    Now focusing on the last term in the above and consider the straightforward decomposition\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-0}\n        \\ev{x_{k+1} - x_\\ast, \\nabla f(x_k) - g_k} &= \\ev{x_{k+1} - T_{\\gamma_k} (x_k), \\nabla f(x_k) - g_k } + \\ev{T_{\\gamma_k} (x_k) - x_\\ast, \\nabla f(x_k) - g_k}.\n     \\end{align}\n    By Cauchy Schwartz we have that\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-1}\n        \\ev{x_{k+1} - T_{\\gamma_k} (x_k), \\nabla f(x_k) - g_k} \\leq \\norm{ x_{k+1} - T_{\\gamma_k} (x_k) } \\norm{g_k - \\nabla f(x_k)}.\n    \\end{align}\n    Now using the nonexpansivity of the proximal operator\n    \\begin{align*}\n        \\norm{x_{k+1} - T_{\\gamma_k} (x_k)} &= \\norm{ \\prox_{\\gamma_k R} \\br{ x_k - \\gamma_k g_k } - \\prox_{\\gamma_k R} \\br{x_k - \\gamma_k \\nabla f(x_k)} } \\\\\n        &\\leq \\norm{ \\br{x_k - \\gamma_k g_k} - \\br{x_k - \\gamma_k \\nabla f(x_k)} } = \\gamma_k \\norm{g_k - \\nabla f(x_k)}.\n    \\end{align*}\n    Using this in \\eqref{eq:gen-prox-dec-proof-1}, we have\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-2}\n        \\ev{x_{k+1} - T_{\\gamma_k} (x_k), \\nabla f(x_k) - g_k} \\leq \\gamma_k \\norm{g_k - \\nabla f(x_k)}^2.\n    \\end{align}\n    Using \\eqref{eq:gen-prox-dec-proof-2} in \\eqref{eq:gen-prox-dec-proof-0} and taking expectation conditioned on $x_k$, and using $\\ec[k]{\\cdot} \\eqdef \\ec{\\cdot \\; | \\; x_k}$ for shorthand, we have\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-2-1}\n        \\ec[k]{\\ev{x_{k+1} - x_\\ast, g_k - \\nabla f(x_k)}} &\\leq \\gamma_k \\cdot \\ecn[k]{g_k - \\nabla f(x_k)} + \\ev{ T_{\\gamma_k} (x_k) - x_\\ast, \\underbrace{\\ec[k]{\\nabla f(x_k) - g_k}}_{= 0} } \\nonumber \\\\\n        &= \\gamma_k \\cdot \\ecn[k]{g_k - \\nabla f(x_k)}.\n    \\end{align}\n\tLet $r_k \\eqdef x_k-x_\\ast.$    Taking expectation conditioned on $x_k$ in \\eqref{eq:gen-prox-dec-proof-init} and using \\eqref{eq:gen-prox-dec-proof-2-1}, we have\n    \\begin{align*}\n        2 \\gamma_k \\ec[k]{ F(x_{k+1} - F(x_\\ast)) } &\\leq \\sqn{r_k} - \\ecn[k]{r_{k+1}} + 2 \\gamma_k^2 \\ecn[k]{g_k - \\nabla f(x_k)}.\n    \\end{align*}\n    Using \\eqref{eq:asm-gradient-opt-distance} from Assumption~\\ref{asm:main_assumption} we have\n    \\begin{align*}\n        2 \\gamma_k \\ec[k]{ F(x_{k+1}) - F(x_\\ast) } &\\leq \\sqn{r_k} - \\ecn[k]{r_{k+1}} + 2 \\gamma_k^2 \\br{ 2 A D_{f} (x_k, x_\\ast) + B \\sigma_k^2 + D_1}.\n    \\end{align*}\n    Let $V_k \\eqdef \\sqn{r_k} + 2 M \\gamma_k^2 \\sigma_{k}^2$ where $M = \\frac{B}{\\rho}$, then\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-3}\n        \\begin{split}\n            2 \\gamma_k \\ec[k]{ F(x_{k+1}) - F(x_\\ast) } \\leq V_k &- \\ec[k]{V_{k+1}} + 4 \\gamma_k^2 A D_{f} (x_k, x_\\ast) + 2 \\gamma_k^2 D_1 \\\\\n            &+ \\gamma_k^2 \\br{ 2B - 2 M } \\sigma_{k}^2 + 2 M \\gamma_{k+1}^2 \\ec{\\sigma_{k+1}^2}.\n        \\end{split}\n    \\end{align}\nSince $\\gamma_{k+1} \\leq \\gamma_k$ we have that\n\t\\begin{align}\n        \\label{eq:gen-prox-dec-proof-3.1}\n        \\begin{split}\n            2 \\gamma_{k+1} \\ec[k]{ F(x_{k+1}) - F(x_\\ast) } \\leq V_k &- \\ec[k]{V_{k+1}} + 4 \\gamma_k^2 A D_{f} (x_k, x_\\ast) + 2 \\gamma_k^2 D_1 \\\\\n            &+ \\gamma_k^2 \\br{ 2B - 2 M } \\sigma_{k}^2 + 2 M \\gamma_k^2 \\ec{\\sigma_{k+1}^2}.\n        \\end{split}\n    \\end{align}\n    \n    Using \\eqref{eq:asm-decreasing-noise} from Assumption~\\ref{asm:main_assumption}, we have\n        \\begin{align}\n        \\label{eq:gen-prox-dec-proof-4}\n            2 \\gamma_k^2 \\br{B - M} \\sigma_k^2 + 2 M \\gamma_k^2 \\ec[k]{\\sigma_{k+1}^2} & \\leq 2 \\gamma_k^2 \\br{ B - M + M (1 - \\rho) } \\sigma_k^2 + 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast)\\nonumber  \\\\\n            &\\quad + 2 M \\gamma_k^2 D_2\n \\nonumber \\\\\n       & = 2 \\gamma_k^2 \\underbrace{\\br{ B - \\rho M}}_{= 0} \\sigma_k^2 + 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast) +2 M \\gamma_k^2 D_2 \\nonumber \\\\\n       & \\leq 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast) +2 M \\gamma_k^2 D_2.\n    \\end{align}\n%    \\begin{align}\n%        \\label{eq:gen-prox-dec-proof-4}\n%        \\begin{split}\n%            2 \\gamma_k^2 \\br{B - M} \\sigma_k^2 + 2 M \\gamma_k^2 \\ec[k]{\\sigma_{k+1}^2} \\leq 2 \\gamma_k^2 \\br{ B - M + M (1 - \\rho) } \\sigma_k^2 &+ 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast) \\\\\n%            &+ 2 M \\gamma_k^2 D_2\n%        \\end{split} \\nonumber \\\\\n%        = 2 \\gamma_k^2 \\underbrace{\\br{ B - \\rho M}}_{= 0} \\sigma_k^2 + 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast) &+2 M \\gamma_k^2 D_2 \\nonumber \\\\\n%        \\leq 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast) &+2 M \\gamma_k^2 D_2.\n%    \\end{align}\n    Using \\eqref{eq:gen-prox-dec-proof-4} in \\eqref{eq:gen-prox-dec-proof-3} gives\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-5}\n        2 \\gamma_{k+1} \\ec[k]{ F(x_{k+1}) - F(x_\\ast) } &\\leq V_k - \\ec[k]{V_{k+1}} + 2 \\gamma_k^2 \\br{ 2 A + 2 M C } D_{f} (x_k, x_\\ast) + 2 \\gamma_k^2 \\br{ D_1 + M D_2 }.\n    \\end{align}\n    Let $\\eta \\eqdef 2A + 2 M C$. Using \\eqref{eq:bregman-divergence-bound} in \\eqref{eq:gen-prox-dec-proof-5} we have,\n    \\begin{align*}\n        2 \\gamma_{k+1} \\ec[k]{ F(x_{k+1}) - F(x_\\ast) } &\\leq V_k - \\ec[k]{V_{k+1}} + 2 \\gamma_k^2 \\eta \\br{ F(x_k) - F(x_\\ast) } + 2 \\gamma_k^2 \\br{ D_1 + M D_2 }.\n    \\end{align*}\n    Using the abbreviation $\\delta_k = F(x_k) - F(x_\\ast)$ gives\n    \\begin{align*}\n        2 \\gamma_{k+1} \\ec[k]{\\delta_{k+1}} &\\leq V_{k} - \\ec[k]{V_{k+1}} + 2 \\gamma_k^2 \\eta \\delta_k + 2 \\gamma_k^2 \\br{D_1 + M D_2}.\n    \\end{align*}\n    Taking expectation,\n   \\begin{align*}\n        2 \\gamma_{k+1} \\ec{\\delta_{k+1}} &\\leq \\ec{V_{k}} - \\ec{V_{k+1}} + 2 \\gamma_k^2 \\eta \\ec{\\delta_k} + 2 \\gamma_k^2 \\br{D_1 + M D_2},\n    \\end{align*}\n summing over $k =0,\\ldots, t-1$ and using telescopic cancellation gives\n    \\begin{align*}\n        2  \\sum_{k=1}^{t}\\gamma_k\\ec{\\delta_{k}} &\\leq V_{0} - \\ec{V_{t}} + 2  \\eta \\sum_{k=0}^{t-1}\\gamma_k^2 \\ec{\\delta_k} + 2 \\br{ D_1 + M D_2 } \\sum_{k=0}^{t-1}\\gamma_k^2.\n    \\end{align*}\nAdding $2\\gamma_0\\delta_0$ to both sides of the above inequality and rearranging,\n\\begin{align*}\n2  \\sum_{k=0}^{t-1}\\gamma_k(1-\\eta\\gamma_k)\\ec{\\delta_{k}} &\\leq V_{0} - \\ec{V_{t}} + 2 \\gamma_0 \\delta_0 + 2 \\br{ D_1 + M D_2 } \\sum_{k=0}^{t-1}\\gamma_k^2\n\\end{align*}\n\n    where we also used that $V_t \\geq 0$ and $\\delta_t \\geq 0.$\n   \n    By the choice of $\\gamma_0$ we have $1 - \\gamma_0 \\eta > 0$, and since $(\\gamma_i)_i$ is a decreasing sequence, we have $1 - \\gamma_i \\eta > 0$ for all $i$. Hence dividing both sides by $2\\sum\\limits_{i=0}^{t-1}\\br{1 - \\gamma_i\\eta} \\gamma_i$, we have\n    \\begin{align*}\n        \\sum_{k=0}^{t-1} w_k \\ec{\\delta_k} \\leq \\frac{V_0 + 2 \\gamma_0 \\delta_0}{2\\sum_{i=0}^{t-1}\\br{1 - \\gamma_i\\eta} \\gamma_i} + \\br{D_1 + 2 M D_2} \\frac{\\sum_{k=0}^{t-1}\\gamma_k^2}{\\sum_{i=1}^{t}\\br{1-\\gamma_i\\eta}\\gamma_i},\n    \\end{align*}\n    where $w_k \\eqdef \\frac{\\br{ 1 - \\gamma_k \\eta }  \\gamma_k}{\\sum_{i=0}^{t-1}\\br{1 - \\gamma_i\\eta} \\gamma_i}$ for all $k \\in \\left\\{0,\\dots,t-1\\right\\}$. Note that $\\sum_{k=0}^{t-1} w_k = 1$ and $w_k \\geq 0$ for all $k \\in \\left\\{0,\\dots,t-1\\right\\}$. Hence, since $F$ is convex, we can use Jensen's inequality to conclude\n    \\begin{align*}\n        \\ec{F(\\bar{x}^k) - F(x_\\ast)} &= \\ec{F\\br{ \\sum\\limits_{k=0}^{t-1} w_k x_k} - F(x_\\ast)} \\\\\n        &\\leq \\sum_{k=0}^{t-1}w_k \\ec{\\delta_k} \\leq \\frac{V_0 + 2\\gamma_0 \\delta_0}{2\\sum_{i=0}^{t-1}\\br{1 - \\gamma_i\\eta} \\gamma_i} + \\frac{\\br{D_1 + 2 M D_2} \\sum_{k=0}^{t-1}\\gamma_k^2}{\\sum_{i=0}^{t-1}\\br{1-\\gamma_i\\eta}\\gamma_i}.\n    \\end{align*}\n    Writing out the definition of $\\delta_0$ yields the theorem's statement.\n\\end{proof}","sha256_utf8":"cf8bbef3661ce0b2b7bdbb1d051159fe6be8965877eeca3a2a75000620b0bfa3"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"复合凸优化目标","source_units":[{"start":11983,"end":12096,"start_line":310,"end_line":312,"raw_tex":"\\begin{eqnarray}\\label{eq:optimization_problem}\n\\min_{x\\in \\R^d} \\pbr{  F(x) \\equiv f(x) + R(x) },\n\\end{eqnarray}","sha256_utf8":"ceecc42030fd6df92e03bfae1a753c292db43b91f986e84c222f2d5b29f2e280"}]},{"id":"S004","semantic_type":"algorithm","kind":"algorithm","title":"随机近端梯度迭代","source_units":[{"start":13175,"end":13323,"start_line":318,"end_line":321,"raw_tex":"\\begin{equation}\n    \\label{eq:sgd-proximal-iterates}\n    x_0 \\in \\R^d, \\quad x_{k+1} = \\prox_{\\gamma_k R} \\br{ x_k - \\gamma_k g_k },\n\\end{equation}","sha256_utf8":"6ee0a54975598732eba4de5605c9b11746de268926b89d88c622ad7e30238ce6"},{"start":13324,"end":13489,"start_line":322,"end_line":323,"raw_tex":"where $\\br{\\gamma_k}_k$ is a sequence of step sizes.  This estimate $g_k$\ncan take on many different forms depending on the problem of interest. Here we list a few.\n","sha256_utf8":"1e37900b3f956da340aed2c0c674588c28cf54fd87e29ab683c62628a8f0b87b"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"散度与近端算子记号","source_units":[{"start":23702,"end":24079,"start_line":372,"end_line":372,"raw_tex":"\\paragraph{Notation.}   The Bregman divergence associated with $f$ is the mapping $$D_{f} (x, y) \\; \\eqdef \\; f(x) - f(y) - \\ev{ \\nabla f(y), x - y }, \\quad x,y\\in R^d$$ and the proximal operator of $\\gamma R$ is the function $$\\prox_{\\gamma R} \\br{x} \\; \\eqdef \\; \\argmin_u \\left\\{ \\gamma R(x) + \\frac{1}{2} \\norm{x-u}^2\\right\\}.$$ Let $[n] \\eqdef \\left\\{1,\\dots,n\\right\\}$. \n","sha256_utf8":"f4e32b6661abb19988dd3bbd9e69fa50f9ee2c886455fe65fec077a094dccc32"}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","title":"光滑性与凸性假设","source_units":[{"start":24478,"end":25057,"start_line":376,"end_line":387,"raw_tex":"\\begin{assumption} \\label{asm:function-class}\nThe function $f$ is  $L$--smooth and convex:\n\\begin{align}\nf(y) & \\leq f(x) + \\langle \\nabla f(x), y - x \\rangle + \\frac{L}{2} \\norm{y-x}^2,  \\quad \\mbox{for all }x,y \\in \\R^d, \\label{eq:Lsmooth}\\\\\nf(y )& \\geq f(x) + \\langle \\nabla f(x), y - x \\rangle, \\quad \\mbox{for all }x,y \\in \\R^d. \\label{eq:convex}\n\\end{align}\nThe function $R$ is convex:\n\\begin{equation}\\label{eq:R-convex}\nR(\\alpha x + (1- \\alpha) y) \\geq \\alpha R(x) + (1-\\alpha) R(y), \\quad \\mbox{for all }x,y \\in \\R^d, \\alpha \\in [0,\\,1].\n\\end{equation}\n\n\\end{assumption}","sha256_utf8":"2e060b22b0fb6193becb1ee6d95002687dc4a5f62151b2090f0c542e2fca3398"}]},{"id":"S007","semantic_type":"assumption","kind":"assumption","title":"无偏梯度与噪声递推","source_units":[{"start":25566,"end":26492,"start_line":393,"end_line":409,"raw_tex":"\\begin{assumption}[Assumption 4.1 in \\cite{Gorbunov19}]\\label{asm:main_assumption}\nConsider the iterates $\\br{x_k}_k$ and gradients $\\br{g_k}_k$ in \\eqref{eq:sgd-proximal-iterates}.\n    \\begin{enumerate}\n        \\item The gradient estimates are unbiased:\n        \\begin{equation}\n            \\label{eq:asm-gradient-unbiased}\n            \\ec{g_k \\mid x_k} = \\nabla f(x_k).\n        \\end{equation}\n        \\item There exist constants $A,B,C,D_1, D_2, \\rho \\geq 0$, and a sequence of random variables $\\sigma^2_k \\geq 0 $  such that:\n        \\begin{align}\n            \\label{eq:asm-gradient-opt-distance}\n            \\ec{ \\norm{g_k - \\nabla f(x_\\ast)}^2 \\mid x_k } &\\leq 2 A D_{f} (x_k, x_\\ast) + B \\sigma_k^2 + D_1, \\\\\n            \\label{eq:asm-decreasing-noise}\n            \\ec{ \\sigma_{k+1}^2 \\mid x_k } &\\leq \\br{ 1 - \\rho } \\sigma_k^2 + 2 C D_{f} (x_k, x_\\ast) + D_2.\n        \\end{align}\n    \\end{enumerate} \n\\end{assumption}","sha256_utf8":"2b517a71c5398f10131265c3c19c6ae9458ad9f5f14293ad78ca487d3224d5f4"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"确定性近端梯度映射","source_units":[{"start":84903,"end":85126,"start_line":1301,"end_line":1306,"raw_tex":"\\begin{definition}\n    Given a stepsize $\\gamma > 0$, the prox-grad mapping is defined as:\n    \\begin{equation}\n        T_{\\gamma} (x) \\eqdef \\prox_{\\gamma R} \\br{x - \\gamma \\nabla f(x)}.\n    \\end{equation}\n\\end{definition}","sha256_utf8":"91044e86089d078d1c5134c7961a24ec0697c0e2afe6ca4b7042237419c05e7c"}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","title":"主定理步长条件","source_units":[{"start":27514,"end":27814,"start_line":417,"end_line":418,"raw_tex":"    Suppose that Assumptions~\\ref{asm:function-class} and~\\ref{asm:main_assumption} hold. Let $M \\eqdef B/\\rho$ and let $(\\gamma_k)_{k\\geq 0}$ be a decreasing, strictly positive sequence of step sizes chosen such that\n    \\[ 0 < \\gamma_0 < \\min \\left\\{ \\frac{1}{2 (A + MC)}, \\frac{1}{L} \\right\\}. \\]\n","sha256_utf8":"93a49aa6d1101f41925c6149b2368c3b2b7fa554f289f295f8dc0a7829d47c8d"}]},{"id":"S010","semantic_type":"lemma","kind":"lemma","title":"随机梯度方差上界","source_units":[{"start":82658,"end":82979,"start_line":1256,"end_line":1261,"raw_tex":"\\begin{lemma}[Bounding the gradient variance] Assuming that the $g_k$ are unbiased and that Assumption~\\ref{asm:main_assumption} holds, we have\n    \\begin{equation}\n        \\label{eq:lma-gradient-variance}\n        \\ecn{g_k - \\nabla f(x_k)}  \\leq 2 A D_{f} (x_k, x_\\ast) + B \\sigma_k^2 + D_1\n    \\end{equation}\n\\end{lemma}","sha256_utf8":"128cc5eb88bb9791f4f8105c2247097b6916a54e3196a9dbc2865bb27090ef91"}]},{"id":"S011","semantic_type":"proof","kind":"proof","title":"梯度方差引理证明","source_units":[{"start":82980,"end":83521,"start_line":1262,"end_line":1270,"raw_tex":"\\begin{proof}\n    Starting from the left hand side of \\eqref{eq:lma-gradient-variance}, we have\n    \\begin{align*}\n        \\ecn{g_k - \\nabla f(x_k)} &= \\ecn{ g_k - \\nabla f(x_\\ast) - \\br{ \\nabla f(x_k) - \\nabla f(x_\\ast) } } \\\\\n        &= \\ecn{ g_k - \\nabla f(x_\\ast) - \\ec{ g_k - \\nabla f(x_\\ast) } } \\\\\n        &\\leq \\ecn{g_k - \\nabla f(x_\\ast)} \\leq 2 A D_{f} (x_k, x_\\ast) + B \\sigma_k^2 + D_1,\n    \\end{align*}\n    where we used that $\\ecn{ X - \\ec{X} } = \\ecn{X} - \\norm{\\ec{X}}^2 \\leq \\ecn{X}$ for any random variable $X$.\n\\end{proof}","sha256_utf8":"4951fd5e93dcc697f8781e748c6692fa8fb03541822f026d9b580e0a57fdb733"}]},{"id":"S012","semantic_type":"proof_step","kind":"proof_step","title":"中心化方差与二阶矩界","source_units":[{"start":82980,"end":83521,"start_line":1262,"end_line":1270,"raw_tex":"\\begin{proof}\n    Starting from the left hand side of \\eqref{eq:lma-gradient-variance}, we have\n    \\begin{align*}\n        \\ecn{g_k - \\nabla f(x_k)} &= \\ecn{ g_k - \\nabla f(x_\\ast) - \\br{ \\nabla f(x_k) - \\nabla f(x_\\ast) } } \\\\\n        &= \\ecn{ g_k - \\nabla f(x_\\ast) - \\ec{ g_k - \\nabla f(x_\\ast) } } \\\\\n        &\\leq \\ecn{g_k - \\nabla f(x_\\ast)} \\leq 2 A D_{f} (x_k, x_\\ast) + B \\sigma_k^2 + D_1,\n    \\end{align*}\n    where we used that $\\ecn{ X - \\ec{X} } = \\ecn{X} - \\norm{\\ec{X}}^2 \\leq \\ecn{X}$ for any random variable $X$.\n\\end{proof}","sha256_utf8":"4951fd5e93dcc697f8781e748c6692fa8fb03541822f026d9b580e0a57fdb733"}]},{"id":"S013","semantic_type":"lemma","kind":"lemma","title":"近端步函数值不等式","source_units":[{"start":83523,"end":83980,"start_line":1272,"end_line":1279,"raw_tex":"\\begin{lemma}\n    \\label{lemma:atchade-functional-value-bound}\n    Suppose that Assumption~\\ref{asm:function-class} holds and let $\\gamma \\in \\left (0, \\frac{1}{L} \\right ]$, then for all $x, y \\in \\R^d$ and $p = \\prox_{\\gamma g}(y)$ we have,\n    \\begin{equation}\n        \\label{eq:atchade-lemma}\n        - 2 \\gamma \\br{ F(p) - F(x_\\ast) } \\geq \\sqn{p - z } + 2  \\ev{p - x_\\ast, x - \\gamma \\nabla f(x) - y} - \\sqn{x_\\ast - x}.\n    \\end{equation}\n\\end{lemma}","sha256_utf8":"a3e0ab470d909a483dc748a3ebb6c725da06ce289a0af374a33a080ca6dceaff"}]},{"id":"S014","semantic_type":"proof","kind":"proof","title":"近端函数值引理完整证明","source_units":[{"start":83981,"end":84066,"start_line":1280,"end_line":1282,"raw_tex":"\\begin{proof}\n    We leave the proof to Section~\\ref{app:sec:prooflemma}.\n\\end{proof}","sha256_utf8":"f35d91a5c7138a80201388bc0967c76b56554a44ad53e3bb118f99ad60be8397"},{"start":111157,"end":114055,"start_line":1776,"end_line":1833,"raw_tex":"\\subsection{Proof of Lemma~\\ref{lemma:atchade-functional-value-bound}}\n\\label{app:sec:prooflemma}\n%Here\n%\\begin{lemma}\n% \\[  F(p) - F(z)  \\leq -\\frac{1}{2\\gamma}\\sqn{p - z } - \\frac{1}{\\gamma}  \\ev{x - \\gamma \\nabla f(x) - y, p-z} + \\frac{1}{2\\gamma}\\sqn{z - x}. \\]\n%\\end{lemma}\n\nThis proof is based on the proof of  Lemma~8 in \\cite{Atchade17}. The only difference is that in  \\cite{Atchade17} the authors assume that $f$ is convex.\nIndeed, using the convexity of $f$\n \\[f(x) -f(x_\\ast) \\geq  - \\langle \\nabla f(x), x_\\ast - x \\rangle\\]\n%\\[f(x)- f(z) \\leq  - \\dotprod{\\nabla f(x), z-x} \\]\nin combination with~\\eqref{eq:2ndproxtheo} where $z = x_\\ast$ gives\n\\begin{align*}\nf(x)+g(p)- F(x_\\ast) & \\leq -\\frac{1}{\\gamma} \\dotprod{p - y, p -x_\\ast}- \\dotprod{\\nabla f(x), x_\\ast-x}. \n\\end{align*}\nNow using smoothness\n\\[ f(p) -f(x) \\leq \\dotprod{\\nabla f(x),p -x} + \\frac{1}{2\\gamma} \\norm{p-x}^2,\\]\ngives\n%& \\leq \\frac{1}{\\gamma} \\dotprod{ y-p +\\gamma \\nabla f(x) , p -z}- \\dotprod{\\nabla f(x), p-x}+\\dotprod{\\nabla f(x),p -x} + \\frac{1}{2\\gamma} \\norm{p-x}^2\\\\\n\\begin{align}\nF(p)- F(x_\\ast) \n&\\leq-\\frac{1}{\\gamma} \\dotprod{p - y, p -x_\\ast}- \\dotprod{\\nabla f(x), x_\\ast-x} +\\dotprod{\\nabla f(x),p -x} + \\frac{1}{2\\gamma} \\norm{p-x}^2 \\nonumber \\\\\n&= -\\frac{1}{\\gamma} \\dotprod{p - y, p -x_\\ast}+\\dotprod{\\nabla f(x),p -x_\\ast} + \\frac{1}{2\\gamma} \\norm{p-x}^2 \\nonumber \\\\\n&= -\\frac{1}{\\gamma} \\dotprod{p -\\gamma \\nabla f(x)- y, p -x_\\ast} + \\frac{1}{2\\gamma} \\norm{p-x}^2 \\nonumber  \\\\\n&= -\\frac{1}{\\gamma} \\dotprod{p-x+x -\\gamma \\nabla f(x)- y, p -x_\\ast} + \\frac{1}{2\\gamma} \\norm{p-x}^2 \\nonumber \\\\\n&= -\\frac{1}{\\gamma} \\dotprod{p-x, p -x_\\ast} -\\frac{1}{\\gamma} \\dotprod{x -\\gamma \\nabla f(x)- y, p -x_\\ast} + \\frac{1}{2\\gamma} \\norm{p-x}^2 . \\label{eq:tenaoineria}\n\\end{align}\nUsing that\n\\begin{equation}\n-2 \\dotprod{p-x, p -x_\\ast}  + \\norm{p-x}^2 = -\\norm{p-x_\\ast}^2 + \\norm{z-x}^2,\n\\end{equation}\nin combination with~\\eqref{eq:tenaoineria} gives \n \\[  F(p) - F(x_\\ast)  \\leq -\\frac{1}{2\\gamma}\\sqn{p - x_\\ast} - \\frac{1}{\\gamma}  \\ev{x - \\gamma \\nabla f(x) - y, p-x_\\ast} + \\frac{1}{2\\gamma}\\sqn{x_\\ast - x}. \\]\n Now it remains to multiply both sides by $-2\\gamma$ to arrive at~\\eqref{eq:atchade-lemma}. \n \n \n%\\section{Maple Code for Section \\ref{sec:optimal-b-SAGA}}\\label{sec:maple-optimal-b-SAGA}\n\n%\\begin{figure*}[h]\n%\\begin{subfigure}[b]{\\textwidth}\n%\\includegraphics[width=\\textwidth]{maple_def.png}\n%\\end{subfigure}\\\\\n%\\begin{subfigure}[b]{\\textwidth}\n%\\includegraphics[width=\\textwidth]{maple_diff_1.png}\n%\\end{subfigure}\\\\\n%\\begin{subfigure}[b]{\\textwidth}\n%\\includegraphics[width=\\textwidth]{maple_disc.png}\n%\\end{subfigure}\\\\\n%\\begin{subfigure}[b]{\\textwidth}\n%\\includegraphics[width=\\textwidth]{maple_diff_2.png}\n%\\end{subfigure}\\\\\n%\\begin{subfigure}[b]{\\textwidth}\n%\\includegraphics[width=\\textwidth]{maple_solutions.png}\n%\\end{subfigure}\n%\\end{figure*}\n\n\\end{appendices}\n\n","sha256_utf8":"5cd995655c0a14ad5ab26904119d5920d36739e4db458c568e3925d8ea7311f7"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"指向附录详细证明","source_units":[{"start":83981,"end":84066,"start_line":1280,"end_line":1282,"raw_tex":"\\begin{proof}\n    We leave the proof to Section~\\ref{app:sec:prooflemma}.\n\\end{proof}","sha256_utf8":"f35d91a5c7138a80201388bc0967c76b56554a44ad53e3bb118f99ad60be8397"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"引入附录证明并结合凸性与近端界","source_units":[{"start":111157,"end":111951,"start_line":1776,"end_line":1790,"raw_tex":"\\subsection{Proof of Lemma~\\ref{lemma:atchade-functional-value-bound}}\n\\label{app:sec:prooflemma}\n%Here\n%\\begin{lemma}\n% \\[  F(p) - F(z)  \\leq -\\frac{1}{2\\gamma}\\sqn{p - z } - \\frac{1}{\\gamma}  \\ev{x - \\gamma \\nabla f(x) - y, p-z} + \\frac{1}{2\\gamma}\\sqn{z - x}. \\]\n%\\end{lemma}\n\nThis proof is based on the proof of  Lemma~8 in \\cite{Atchade17}. The only difference is that in  \\cite{Atchade17} the authors assume that $f$ is convex.\nIndeed, using the convexity of $f$\n \\[f(x) -f(x_\\ast) \\geq  - \\langle \\nabla f(x), x_\\ast - x \\rangle\\]\n%\\[f(x)- f(z) \\leq  - \\dotprod{\\nabla f(x), z-x} \\]\nin combination with~\\eqref{eq:2ndproxtheo} where $z = x_\\ast$ gives\n\\begin{align*}\nf(x)+g(p)- F(x_\\ast) & \\leq -\\frac{1}{\\gamma} \\dotprod{p - y, p -x_\\ast}- \\dotprod{\\nabla f(x), x_\\ast-x}. \n\\end{align*}\n","sha256_utf8":"b441f888c3c6f36ac584f8121ea17030ed74f7ed75066cc8c1195b3e7374daec"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"加入光滑性并展开内积","source_units":[{"start":111951,"end":112942,"start_line":1791,"end_line":1802,"raw_tex":"Now using smoothness\n\\[ f(p) -f(x) \\leq \\dotprod{\\nabla f(x),p -x} + \\frac{1}{2\\gamma} \\norm{p-x}^2,\\]\ngives\n%& \\leq \\frac{1}{\\gamma} \\dotprod{ y-p +\\gamma \\nabla f(x) , p -z}- \\dotprod{\\nabla f(x), p-x}+\\dotprod{\\nabla f(x),p -x} + \\frac{1}{2\\gamma} \\norm{p-x}^2\\\\\n\\begin{align}\nF(p)- F(x_\\ast) \n&\\leq-\\frac{1}{\\gamma} \\dotprod{p - y, p -x_\\ast}- \\dotprod{\\nabla f(x), x_\\ast-x} +\\dotprod{\\nabla f(x),p -x} + \\frac{1}{2\\gamma} \\norm{p-x}^2 \\nonumber \\\\\n&= -\\frac{1}{\\gamma} \\dotprod{p - y, p -x_\\ast}+\\dotprod{\\nabla f(x),p -x_\\ast} + \\frac{1}{2\\gamma} \\norm{p-x}^2 \\nonumber \\\\\n&= -\\frac{1}{\\gamma} \\dotprod{p -\\gamma \\nabla f(x)- y, p -x_\\ast} + \\frac{1}{2\\gamma} \\norm{p-x}^2 \\nonumber  \\\\\n&= -\\frac{1}{\\gamma} \\dotprod{p-x+x -\\gamma \\nabla f(x)- y, p -x_\\ast} + \\frac{1}{2\\gamma} \\norm{p-x}^2 \\nonumber \\\\\n&= -\\frac{1}{\\gamma} \\dotprod{p-x, p -x_\\ast} -\\frac{1}{\\gamma} \\dotprod{x -\\gamma \\nabla f(x)- y, p -x_\\ast} + \\frac{1}{2\\gamma} \\norm{p-x}^2 . \\label{eq:tenaoineria}\n\\end{align}\n","sha256_utf8":"ad1bbbc68129803eb4806f5697fa0efd9c8b6ae477a3c9d694d1af0a70ba869a"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"平方范数恒等式收尾","source_units":[{"start":112942,"end":114055,"start_line":1803,"end_line":1833,"raw_tex":"Using that\n\\begin{equation}\n-2 \\dotprod{p-x, p -x_\\ast}  + \\norm{p-x}^2 = -\\norm{p-x_\\ast}^2 + \\norm{z-x}^2,\n\\end{equation}\nin combination with~\\eqref{eq:tenaoineria} gives \n \\[  F(p) - F(x_\\ast)  \\leq -\\frac{1}{2\\gamma}\\sqn{p - x_\\ast} - \\frac{1}{\\gamma}  \\ev{x - \\gamma \\nabla f(x) - y, p-x_\\ast} + \\frac{1}{2\\gamma}\\sqn{x_\\ast - x}. \\]\n Now it remains to multiply both sides by $-2\\gamma$ to arrive at~\\eqref{eq:atchade-lemma}. \n \n \n%\\section{Maple Code for Section \\ref{sec:optimal-b-SAGA}}\\label{sec:maple-optimal-b-SAGA}\n\n%\\begin{figure*}[h]\n%\\begin{subfigure}[b]{\\textwidth}\n%\\includegraphics[width=\\textwidth]{maple_def.png}\n%\\end{subfigure}\\\\\n%\\begin{subfigure}[b]{\\textwidth}\n%\\includegraphics[width=\\textwidth]{maple_diff_1.png}\n%\\end{subfigure}\\\\\n%\\begin{subfigure}[b]{\\textwidth}\n%\\includegraphics[width=\\textwidth]{maple_disc.png}\n%\\end{subfigure}\\\\\n%\\begin{subfigure}[b]{\\textwidth}\n%\\includegraphics[width=\\textwidth]{maple_diff_2.png}\n%\\end{subfigure}\\\\\n%\\begin{subfigure}[b]{\\textwidth}\n%\\includegraphics[width=\\textwidth]{maple_solutions.png}\n%\\end{subfigure}\n%\\end{figure*}\n\n\\end{appendices}\n\n","sha256_utf8":"3f10d7fff8f6b3cb8076000029afaef7a7e231fca1c719d0f3bdc11ef22335bc"}]},{"id":"S019","semantic_type":"lemma","kind":"lemma","title":"散度由目标差控制","source_units":[{"start":84068,"end":84290,"start_line":1284,"end_line":1290,"raw_tex":"\\begin{lemma}\n    For any $x \\in \\R^d$ and minimizer $x_\\ast$ of $F$ we have,\n    \\begin{equation}\n        \\label{eq:bregman-divergence-bound}\n        D_{f} (x, x_\\ast) \\leq F(x) - F(x_\\ast).\n    \\end{equation}\n\\end{lemma}","sha256_utf8":"bd2644791b2b63beca42894b77bbeb40a2bb535dded04ed9467494090a26450a"}]},{"id":"S020","semantic_type":"proof","kind":"proof","title":"散度控制引理证明","source_units":[{"start":84291,"end":84901,"start_line":1291,"end_line":1299,"raw_tex":"\\begin{proof}\n    Because $x_\\ast$ is a minimizer of $F$ we have that $- \\nabla f(x_\\ast)  \\in \\partial R(x_\\ast)$. By the definition of subgradients we have\n    \\[ R(x_\\ast) + \\ev{ - \\nabla f(x_\\ast), x - x_\\ast } \\leq R(x). \\]\n    Rearranging gives\n    \\[ -\\ev{ \\nabla f(x_\\ast), x - x_\\ast} \\leq R(x) - R(x_\\ast) \\]\n    Adding $f(x) - f(x_\\ast)$ to both sides we have,\n    \\[ f(x) - f(x_\\ast) - \\ev{ \\nabla f(x_\\ast), x - x_\\ast } \\leq f(x) + R(x) - \\br{ f(x_\\ast) + R(x_\\ast)} = F(x) - F(x_\\ast). \\]\n   Now note that the on the left hand side we have the Bregman divergence $D_{f} (x, x_\\ast)$.\n\\end{proof}","sha256_utf8":"a3bac6a8fa3473418588ff93256152ca5c644a781693b3abbe0088509c528458"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"最优性导出次梯度界","source_units":[{"start":84291,"end":84610,"start_line":1291,"end_line":1295,"raw_tex":"\\begin{proof}\n    Because $x_\\ast$ is a minimizer of $F$ we have that $- \\nabla f(x_\\ast)  \\in \\partial R(x_\\ast)$. By the definition of subgradients we have\n    \\[ R(x_\\ast) + \\ev{ - \\nabla f(x_\\ast), x - x_\\ast } \\leq R(x). \\]\n    Rearranging gives\n    \\[ -\\ev{ \\nabla f(x_\\ast), x - x_\\ast} \\leq R(x) - R(x_\\ast) \\]\n","sha256_utf8":"6b530349494172bcf1f71c889c1effed27de45df98d55879fd7c4ab8134f6b60"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"相加识别散度与目标差","source_units":[{"start":84610,"end":84901,"start_line":1296,"end_line":1299,"raw_tex":"    Adding $f(x) - f(x_\\ast)$ to both sides we have,\n    \\[ f(x) - f(x_\\ast) - \\ev{ \\nabla f(x_\\ast), x - x_\\ast } \\leq f(x) + R(x) - \\br{ f(x_\\ast) + R(x_\\ast)} = F(x) - F(x_\\ast). \\]\n   Now note that the on the left hand side we have the Bregman divergence $D_{f} (x, x_\\ast)$.\n\\end{proof}","sha256_utf8":"8393df0c03b081249d1b36daedf518569f14a7aebfba29c1d18d42a9394a3de1"}]},{"id":"S023","semantic_type":"lemma","kind":"lemma","title":"第二近端定理","source_units":[{"start":110661,"end":110952,"start_line":1765,"end_line":1769,"raw_tex":"\\begin{lemma} Let $R: \\R^d \\mapsto \\R$ be a convex lower semi-continuous function. For $z,y \\in \\R^d$ and $\\gamma>0$. With $p = \\prox_{\\gamma g} (y)$ we have that for\n\\begin{equation} \\label{eq:2ndproxtheo}\ng(p) - g(z) \\leq -\\frac{1}{\\gamma} \\dotprod{p - y, p -z}.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"eeeb8374a40dce9bf45c4eb851bcb2f6d3a518368340aebe668b72851bb86181"}]},{"id":"S024","semantic_type":"proof","kind":"proof","title":"近端引理引用证明","source_units":[{"start":110953,"end":111154,"start_line":1770,"end_line":1773,"raw_tex":"\\begin{proof}\nThis is classic result, see for example the ``Second Prox Theorem'' in Section 6.5 in~\\cite{beck2017first}.\n%See Propositions 4.2., 12.26 and 12.27 in~\\cite{combetteeBook2011}\n\\end{proof}","sha256_utf8":"ae043053c5991c2af28a69e0759744417546b0f6267d1f92ead1ca948c940ec0"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"引用经典第二近端定理","source_units":[{"start":110953,"end":111154,"start_line":1770,"end_line":1773,"raw_tex":"\\begin{proof}\nThis is classic result, see for example the ``Second Prox Theorem'' in Section 6.5 in~\\cite{beck2017first}.\n%See Propositions 4.2., 12.26 and 12.27 in~\\cite{combetteeBook2011}\n\\end{proof}","sha256_utf8":"ae043053c5991c2af28a69e0759744417546b0f6267d1f92ead1ca948c940ec0"}]},{"id":"S026","semantic_type":"external_fact","kind":"external_fact","title":"经典近端变分不等式","source_units":[],"declared_external":true,"declaration_text":["Second Prox Theorem, cited from Beck (2017), Section 6.5: if h is proper, lower semicontinuous and convex, gamma > 0, and p = prox_{gamma h}(y), then h(p)-h(z) <= -(1/gamma)<p-y,p-z> for every z in dom(h)."]},{"id":"S027","semantic_type":"external_fact","kind":"external_fact","title":"近端算子非扩张性","source_units":[],"declared_external":true,"declaration_text":["For a proper lower semicontinuous convex h and gamma > 0, ||prox_{gamma h}(u)-prox_{gamma h}(v)|| <= ||u-v||."]},{"id":"S028","semantic_type":"external_fact","kind":"external_fact","title":"柯西施瓦茨不等式","source_units":[],"declared_external":true,"declaration_text":["For Euclidean vectors u and v, <u,v> <= ||u|| ||v||."]},{"id":"S029","semantic_type":"external_fact","kind":"external_fact","title":"条件方差恒等式","source_units":[],"declared_external":true,"declaration_text":["For a square-integrable random vector X and sigma-algebra G, E[||X-E[X|G]||^2|G] = E[||X||^2|G]-||E[X|G]||^2 <= E[||X||^2|G]."]},{"id":"S030","semantic_type":"external_fact","kind":"external_fact","title":"凸最优性与次梯度定义","source_units":[],"declared_external":true,"declaration_text":["For differentiable convex f and convex R, a minimizer x_* of f+R satisfies -grad f(x_*) in partial R(x_*). If s in partial R(x_*), then R(x) >= R(x_*)+<s,x-x_*> for every x."]},{"id":"S031","semantic_type":"external_fact","kind":"external_fact","title":"凸函数詹森不等式","source_units":[],"declared_external":true,"declaration_text":["If F is convex, w_k >= 0 and sum_k w_k = 1, then F(sum_k w_k x_k) <= sum_k w_k F(x_k)."]},{"id":"S032","semantic_type":"external_fact","kind":"external_fact","title":"条件期望塔式法则","source_units":[],"declared_external":true,"declaration_text":["For integrable X and a sigma-algebra G, E[E[X|G]] = E[X]; expectation is linear over finite integrable sums."]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"应用近端函数值界","source_units":[{"start":86130,"end":86826,"start_line":1322,"end_line":1331,"raw_tex":"\\begin{proof}\n    Let $x_\\ast$ be a minimizer of $F$. Using \\eqref{eq:atchade-lemma} from Lemma~\\ref{lemma:atchade-functional-value-bound} with $y = x_k - \\gamma_k g_k$, $x = x_k$ and $\\gamma = \\gamma_k$ gives\n    \\begin{align*}\n        -2 \\gamma_k \\br{ F(x_{k+1}) - F(x_\\ast) } &\\geq \\norm{x_{k+1} - x_\\ast}^2 - \\norm{x_k - x_\\ast}^2 + 2 \\gamma_k \\ev{ x_{k+1} - x_\\ast, g_k - \\nabla f(x_k) }.\n    \\end{align*}\n    Multiplying both sides by $-1$ results in\n    \\begin{align}        \n        \\label{eq:gen-prox-dec-proof-init}\n        2 \\gamma_k \\br{F(x_{k+1}) - F(x_\\ast)} &\\leq \\sqn{x_k - x_\\ast} - \\sqn{x_{k+1} - x_\\ast} + 2 \\gamma_k \\ev{x_{k+1} - x_\\ast, \\nabla f(x_k) - g_k}.\n    \\end{align}\n","sha256_utf8":"10b20f5755d5c7f0c5ab3a2204641d334be68ecb910091d05e2fc17ef58d7818"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"分解噪声交叉项","source_units":[{"start":86826,"end":87406,"start_line":1332,"end_line":1341,"raw_tex":"    Now focusing on the last term in the above and consider the straightforward decomposition\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-0}\n        \\ev{x_{k+1} - x_\\ast, \\nabla f(x_k) - g_k} &= \\ev{x_{k+1} - T_{\\gamma_k} (x_k), \\nabla f(x_k) - g_k } + \\ev{T_{\\gamma_k} (x_k) - x_\\ast, \\nabla f(x_k) - g_k}.\n     \\end{align}\n    By Cauchy Schwartz we have that\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-1}\n        \\ev{x_{k+1} - T_{\\gamma_k} (x_k), \\nabla f(x_k) - g_k} \\leq \\norm{ x_{k+1} - T_{\\gamma_k} (x_k) } \\norm{g_k - \\nabla f(x_k)}.\n    \\end{align}\n","sha256_utf8":"c0336ee79da362a924690c5c2e673009b49877ba001577fec193be84ed3256bb"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"非扩张性控制迭代偏差","source_units":[{"start":87406,"end":88021,"start_line":1342,"end_line":1351,"raw_tex":"    Now using the nonexpansivity of the proximal operator\n    \\begin{align*}\n        \\norm{x_{k+1} - T_{\\gamma_k} (x_k)} &= \\norm{ \\prox_{\\gamma_k R} \\br{ x_k - \\gamma_k g_k } - \\prox_{\\gamma_k R} \\br{x_k - \\gamma_k \\nabla f(x_k)} } \\\\\n        &\\leq \\norm{ \\br{x_k - \\gamma_k g_k} - \\br{x_k - \\gamma_k \\nabla f(x_k)} } = \\gamma_k \\norm{g_k - \\nabla f(x_k)}.\n    \\end{align*}\n    Using this in \\eqref{eq:gen-prox-dec-proof-1}, we have\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-2}\n        \\ev{x_{k+1} - T_{\\gamma_k} (x_k), \\nabla f(x_k) - g_k} \\leq \\gamma_k \\norm{g_k - \\nabla f(x_k)}^2.\n    \\end{align}\n","sha256_utf8":"617851de02971f59de7c5fd56f016e872d5b18645f912d249cfd289b08f4a0c6"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"无偏性消去条件均值项","source_units":[{"start":88021,"end":88561,"start_line":1352,"end_line":1357,"raw_tex":"    Using \\eqref{eq:gen-prox-dec-proof-2} in \\eqref{eq:gen-prox-dec-proof-0} and taking expectation conditioned on $x_k$, and using $\\ec[k]{\\cdot} \\eqdef \\ec{\\cdot \\; | \\; x_k}$ for shorthand, we have\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-2-1}\n        \\ec[k]{\\ev{x_{k+1} - x_\\ast, g_k - \\nabla f(x_k)}} &\\leq \\gamma_k \\cdot \\ecn[k]{g_k - \\nabla f(x_k)} + \\ev{ T_{\\gamma_k} (x_k) - x_\\ast, \\underbrace{\\ec[k]{\\nabla f(x_k) - g_k}}_{= 0} } \\nonumber \\\\\n        &= \\gamma_k \\cdot \\ecn[k]{g_k - \\nabla f(x_k)}.\n    \\end{align}\n","sha256_utf8":"bfc094c3220b16896b669fbd50dfb76b59d1b6a93351c547dbbee4bcc7ffaa00"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"建立含方差的期望下降界","source_units":[{"start":88561,"end":89180,"start_line":1358,"end_line":1365,"raw_tex":"\tLet $r_k \\eqdef x_k-x_\\ast.$    Taking expectation conditioned on $x_k$ in \\eqref{eq:gen-prox-dec-proof-init} and using \\eqref{eq:gen-prox-dec-proof-2-1}, we have\n    \\begin{align*}\n        2 \\gamma_k \\ec[k]{ F(x_{k+1} - F(x_\\ast)) } &\\leq \\sqn{r_k} - \\ecn[k]{r_{k+1}} + 2 \\gamma_k^2 \\ecn[k]{g_k - \\nabla f(x_k)}.\n    \\end{align*}\n    Using \\eqref{eq:asm-gradient-opt-distance} from Assumption~\\ref{asm:main_assumption} we have\n    \\begin{align*}\n        2 \\gamma_k \\ec[k]{ F(x_{k+1}) - F(x_\\ast) } &\\leq \\sqn{r_k} - \\ecn[k]{r_{k+1}} + 2 \\gamma_k^2 \\br{ 2 A D_{f} (x_k, x_\\ast) + B \\sigma_k^2 + D_1}.\n    \\end{align*}\n","sha256_utf8":"df3ade774b9b4164aa21d933bd529378013ac29649792e61d82ba15ba8953147"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"引入李雅普诺夫量","source_units":[{"start":89180,"end":89630,"start_line":1366,"end_line":1373,"raw_tex":"    Let $V_k \\eqdef \\sqn{r_k} + 2 M \\gamma_k^2 \\sigma_{k}^2$ where $M = \\frac{B}{\\rho}$, then\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-3}\n        \\begin{split}\n            2 \\gamma_k \\ec[k]{ F(x_{k+1}) - F(x_\\ast) } \\leq V_k &- \\ec[k]{V_{k+1}} + 4 \\gamma_k^2 A D_{f} (x_k, x_\\ast) + 2 \\gamma_k^2 D_1 \\\\\n            &+ \\gamma_k^2 \\br{ 2B - 2 M } \\sigma_{k}^2 + 2 M \\gamma_{k+1}^2 \\ec{\\sigma_{k+1}^2}.\n        \\end{split}\n    \\end{align}\n","sha256_utf8":"b432e5931f33daac18767e4ac8f10c021b7ecf2038432ce64e7ea8074bc702f4"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"利用步长递减性","source_units":[{"start":89630,"end":90033,"start_line":1374,"end_line":1381,"raw_tex":"Since $\\gamma_{k+1} \\leq \\gamma_k$ we have that\n\t\\begin{align}\n        \\label{eq:gen-prox-dec-proof-3.1}\n        \\begin{split}\n            2 \\gamma_{k+1} \\ec[k]{ F(x_{k+1}) - F(x_\\ast) } \\leq V_k &- \\ec[k]{V_{k+1}} + 4 \\gamma_k^2 A D_{f} (x_k, x_\\ast) + 2 \\gamma_k^2 D_1 \\\\\n            &+ \\gamma_k^2 \\br{ 2B - 2 M } \\sigma_{k}^2 + 2 M \\gamma_k^2 \\ec{\\sigma_{k+1}^2}.\n        \\end{split}\n    \\end{align}\n","sha256_utf8":"4ae0b2b823fba072aaad007d00cb6ba0e4cfdd101c2e4f6e5e946cc10b080a3c"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"噪声递推抵消方差项","source_units":[{"start":90038,"end":91250,"start_line":1383,"end_line":1400,"raw_tex":"    Using \\eqref{eq:asm-decreasing-noise} from Assumption~\\ref{asm:main_assumption}, we have\n        \\begin{align}\n        \\label{eq:gen-prox-dec-proof-4}\n            2 \\gamma_k^2 \\br{B - M} \\sigma_k^2 + 2 M \\gamma_k^2 \\ec[k]{\\sigma_{k+1}^2} & \\leq 2 \\gamma_k^2 \\br{ B - M + M (1 - \\rho) } \\sigma_k^2 + 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast)\\nonumber  \\\\\n            &\\quad + 2 M \\gamma_k^2 D_2\n \\nonumber \\\\\n       & = 2 \\gamma_k^2 \\underbrace{\\br{ B - \\rho M}}_{= 0} \\sigma_k^2 + 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast) +2 M \\gamma_k^2 D_2 \\nonumber \\\\\n       & \\leq 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast) +2 M \\gamma_k^2 D_2.\n    \\end{align}\n%    \\begin{align}\n%        \\label{eq:gen-prox-dec-proof-4}\n%        \\begin{split}\n%            2 \\gamma_k^2 \\br{B - M} \\sigma_k^2 + 2 M \\gamma_k^2 \\ec[k]{\\sigma_{k+1}^2} \\leq 2 \\gamma_k^2 \\br{ B - M + M (1 - \\rho) } \\sigma_k^2 &+ 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast) \\\\\n%            &+ 2 M \\gamma_k^2 D_2\n%        \\end{split} \\nonumber \\\\\n%        = 2 \\gamma_k^2 \\underbrace{\\br{ B - \\rho M}}_{= 0} \\sigma_k^2 + 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast) &+2 M \\gamma_k^2 D_2 \\nonumber \\\\\n%        \\leq 4 M \\gamma_k^2 C D_{f} (x_k, x_\\ast) &+2 M \\gamma_k^2 D_2.\n%    \\end{align}\n","sha256_utf8":"fb16e9e96a798abfbd96d7c81c80303f51d6f13df7ded385e067673399b11cf1"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"合并得到势函数递推","source_units":[{"start":91250,"end":91580,"start_line":1401,"end_line":1405,"raw_tex":"    Using \\eqref{eq:gen-prox-dec-proof-4} in \\eqref{eq:gen-prox-dec-proof-3} gives\n    \\begin{align}\n        \\label{eq:gen-prox-dec-proof-5}\n        2 \\gamma_{k+1} \\ec[k]{ F(x_{k+1}) - F(x_\\ast) } &\\leq V_k - \\ec[k]{V_{k+1}} + 2 \\gamma_k^2 \\br{ 2 A + 2 M C } D_{f} (x_k, x_\\ast) + 2 \\gamma_k^2 \\br{ D_1 + M D_2 }.\n    \\end{align}\n","sha256_utf8":"c114ad6feae079b9649cb8075e85282ab253f808d49299bac975c961def741a4"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"以目标差替换散度","source_units":[{"start":91580,"end":92138,"start_line":1406,"end_line":1413,"raw_tex":"    Let $\\eta \\eqdef 2A + 2 M C$. Using \\eqref{eq:bregman-divergence-bound} in \\eqref{eq:gen-prox-dec-proof-5} we have,\n    \\begin{align*}\n        2 \\gamma_{k+1} \\ec[k]{ F(x_{k+1}) - F(x_\\ast) } &\\leq V_k - \\ec[k]{V_{k+1}} + 2 \\gamma_k^2 \\eta \\br{ F(x_k) - F(x_\\ast) } + 2 \\gamma_k^2 \\br{ D_1 + M D_2 }.\n    \\end{align*}\n    Using the abbreviation $\\delta_k = F(x_k) - F(x_\\ast)$ gives\n    \\begin{align*}\n        2 \\gamma_{k+1} \\ec[k]{\\delta_{k+1}} &\\leq V_{k} - \\ec[k]{V_{k+1}} + 2 \\gamma_k^2 \\eta \\delta_k + 2 \\gamma_k^2 \\br{D_1 + M D_2}.\n    \\end{align*}\n","sha256_utf8":"d32152eaaa3a9a79c5d02c94e5b6dd3f2cb2bf2bc304dfc7b47b961bd7ca2cd5"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"全期望与望远镜求和","source_units":[{"start":92138,"end":92622,"start_line":1414,"end_line":1421,"raw_tex":"    Taking expectation,\n   \\begin{align*}\n        2 \\gamma_{k+1} \\ec{\\delta_{k+1}} &\\leq \\ec{V_{k}} - \\ec{V_{k+1}} + 2 \\gamma_k^2 \\eta \\ec{\\delta_k} + 2 \\gamma_k^2 \\br{D_1 + M D_2},\n    \\end{align*}\n summing over $k =0,\\ldots, t-1$ and using telescopic cancellation gives\n    \\begin{align*}\n        2  \\sum_{k=1}^{t}\\gamma_k\\ec{\\delta_{k}} &\\leq V_{0} - \\ec{V_{t}} + 2  \\eta \\sum_{k=0}^{t-1}\\gamma_k^2 \\ec{\\delta_k} + 2 \\br{ D_1 + M D_2 } \\sum_{k=0}^{t-1}\\gamma_k^2.\n    \\end{align*}\n","sha256_utf8":"658ba5d7105107715adc3506fe6ad21e76da969b8ba543f7516dc783a14d9d85"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"重排加权目标差","source_units":[{"start":92622,"end":92953,"start_line":1422,"end_line":1427,"raw_tex":"Adding $2\\gamma_0\\delta_0$ to both sides of the above inequality and rearranging,\n\\begin{align*}\n2  \\sum_{k=0}^{t-1}\\gamma_k(1-\\eta\\gamma_k)\\ec{\\delta_{k}} &\\leq V_{0} - \\ec{V_{t}} + 2 \\gamma_0 \\delta_0 + 2 \\br{ D_1 + M D_2 } \\sum_{k=0}^{t-1}\\gamma_k^2\n\\end{align*}\n\n    where we also used that $V_t \\geq 0$ and $\\delta_t \\geq 0.$\n","sha256_utf8":"8f0fe9424b38aa29620bd5371cc7941607d4e29fb12687899a37a0585423a583"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"正权重归一化","source_units":[{"start":92957,"end":93478,"start_line":1429,"end_line":1432,"raw_tex":"    By the choice of $\\gamma_0$ we have $1 - \\gamma_0 \\eta > 0$, and since $(\\gamma_i)_i$ is a decreasing sequence, we have $1 - \\gamma_i \\eta > 0$ for all $i$. Hence dividing both sides by $2\\sum\\limits_{i=0}^{t-1}\\br{1 - \\gamma_i\\eta} \\gamma_i$, we have\n    \\begin{align*}\n        \\sum_{k=0}^{t-1} w_k \\ec{\\delta_k} \\leq \\frac{V_0 + 2 \\gamma_0 \\delta_0}{2\\sum_{i=0}^{t-1}\\br{1 - \\gamma_i\\eta} \\gamma_i} + \\br{D_1 + 2 M D_2} \\frac{\\sum_{k=0}^{t-1}\\gamma_k^2}{\\sum_{i=1}^{t}\\br{1-\\gamma_i\\eta}\\gamma_i},\n    \\end{align*}\n","sha256_utf8":"2473aa92b275cae008759d9d9b33982e529bf4e28077adda2c4516e08da89820"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"詹森不等式得到平均迭代界","source_units":[{"start":93478,"end":94263,"start_line":1433,"end_line":1439,"raw_tex":"    where $w_k \\eqdef \\frac{\\br{ 1 - \\gamma_k \\eta }  \\gamma_k}{\\sum_{i=0}^{t-1}\\br{1 - \\gamma_i\\eta} \\gamma_i}$ for all $k \\in \\left\\{0,\\dots,t-1\\right\\}$. Note that $\\sum_{k=0}^{t-1} w_k = 1$ and $w_k \\geq 0$ for all $k \\in \\left\\{0,\\dots,t-1\\right\\}$. Hence, since $F$ is convex, we can use Jensen's inequality to conclude\n    \\begin{align*}\n        \\ec{F(\\bar{x}^k) - F(x_\\ast)} &= \\ec{F\\br{ \\sum\\limits_{k=0}^{t-1} w_k x_k} - F(x_\\ast)} \\\\\n        &\\leq \\sum_{k=0}^{t-1}w_k \\ec{\\delta_k} \\leq \\frac{V_0 + 2\\gamma_0 \\delta_0}{2\\sum_{i=0}^{t-1}\\br{1 - \\gamma_i\\eta} \\gamma_i} + \\frac{\\br{D_1 + 2 M D_2} \\sum_{k=0}^{t-1}\\gamma_k^2}{\\sum_{i=0}^{t-1}\\br{1-\\gamma_i\\eta}\\gamma_i}.\n    \\end{align*}\n    Writing out the definition of $\\delta_0$ yields the theorem's statement.\n\\end{proof}","sha256_utf8":"ef359052d7d8e565ef397b6bd72bb47d180bde7156a81c7e32cc5c855f4ce502"}]},{"id":"S047","semantic_type":"assumption","kind":"assumption","title":"函数值引理局部条件","source_units":[{"start":83586,"end":83766,"start_line":1274,"end_line":1274,"raw_tex":"    Suppose that Assumption~\\ref{asm:function-class} holds and let $\\gamma \\in \\left (0, \\frac{1}{L} \\right ]$, then for all $x, y \\in \\R^d$ and $p = \\prox_{\\gamma g}(y)$ we have,\n","sha256_utf8":"c47fcb39581ce9631dafd810f2b7462f7203ba9e18647bf4bc885841c55210f3"}]},{"id":"S048","semantic_type":"assumption","kind":"assumption","title":"近端引理局部条件","source_units":[{"start":110661,"end":110828,"start_line":1765,"end_line":1765,"raw_tex":"\\begin{lemma} Let $R: \\R^d \\mapsto \\R$ be a convex lower semi-continuous function. For $z,y \\in \\R^d$ and $\\gamma>0$. With $p = \\prox_{\\gamma g} (y)$ we have that for\n","sha256_utf8":"c95020b8a72163568abc341e816a5405cf57fa01c2e86cd28be15905cf5aff51"}]},{"id":"S049","semantic_type":"assumption","kind":"assumption","title":"散度引理最优点条件","source_units":[{"start":84082,"end":84146,"start_line":1285,"end_line":1285,"raw_tex":"    For any $x \\in \\R^d$ and minimizer $x_\\ast$ of $F$ we have,\n","sha256_utf8":"546a219d2bad3b05f14534a3dcebdda471ff99cef86ef8a97704066059bcdd16"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S016","kind":"supports"},{"source_id":"S003","target_id":"S021","kind":"supports"},{"source_id":"S003","target_id":"S033","kind":"supports"},{"source_id":"S004","target_id":"S033","kind":"supports"},{"source_id":"S004","target_id":"S035","kind":"supports"},{"source_id":"S005","target_id":"S004","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S008","kind":"supports"},{"source_id":"S005","target_id":"S012","kind":"supports"},{"source_id":"S005","target_id":"S022","kind":"supports"},{"source_id":"S006","target_id":"S016","kind":"supports"},{"source_id":"S006","target_id":"S017","kind":"supports"},{"source_id":"S006","target_id":"S021","kind":"supports"},{"source_id":"S006","target_id":"S035","kind":"supports"},{"source_id":"S006","target_id":"S046","kind":"supports"},{"source_id":"S007","target_id":"S012","kind":"supports"},{"source_id":"S007","target_id":"S036","kind":"supports"},{"source_id":"S007","target_id":"S037","kind":"supports"},{"source_id":"S007","target_id":"S040","kind":"supports"},{"source_id":"S008","target_id":"S034","kind":"supports"},{"source_id":"S008","target_id":"S035","kind":"supports"},{"source_id":"S009","target_id":"S033","kind":"supports"},{"source_id":"S009","target_id":"S038","kind":"supports"},{"source_id":"S009","target_id":"S039","kind":"supports"},{"source_id":"S009","target_id":"S040","kind":"supports"},{"source_id":"S009","target_id":"S045","kind":"supports"},{"source_id":"S010","target_id":"S037","kind":"supports"},{"source_id":"S011","target_id":"S010","kind":"supports"},{"source_id":"S012","target_id":"S011","kind":"supports"},{"source_id":"S013","target_id":"S033","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S014","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S014","kind":"supports"},{"source_id":"S018","target_id":"S014","kind":"supports"},{"source_id":"S019","target_id":"S042","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S020","kind":"supports"},{"source_id":"S023","target_id":"S016","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S027","target_id":"S035","kind":"supports"},{"source_id":"S028","target_id":"S034","kind":"supports"},{"source_id":"S029","target_id":"S012","kind":"supports"},{"source_id":"S030","target_id":"S021","kind":"supports"},{"source_id":"S031","target_id":"S046","kind":"supports"},{"source_id":"S032","target_id":"S043","kind":"supports"},{"source_id":"S033","target_id":"S037","kind":"supports"},{"source_id":"S033","target_id":"S002","kind":"supports"},{"source_id":"S034","target_id":"S036","kind":"supports"},{"source_id":"S034","target_id":"S002","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S002","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S002","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S002","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S002","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S002","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S002","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S002","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S002","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S002","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S002","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S002","kind":"supports"},{"source_id":"S046","target_id":"S002","kind":"supports"},{"source_id":"S047","target_id":"S016","kind":"supports"},{"source_id":"S047","target_id":"S017","kind":"supports"},{"source_id":"S048","target_id":"S025","kind":"supports"},{"source_id":"S049","target_id":"S021","kind":"supports"}]}}
{"record_id":"2007.13605v4","paper":{"title":"Alternating proximal-gradient steps for (stochastic) nonconvex-concave minimax problems","authors":["Radu Ioan Boţ","Axel Böhm"],"arxiv_id":"2007.13605","version":"v4","journal":"SIAM Journal on Optimization","doi":"10.1137/21m1465470","abstract_url":"https://arxiv.org/abs/2007.13605v4","pdf_url":"https://arxiv.org/pdf/2007.13605v4"},"context":{"raw_tex":"\\begin{equation}\n  \\label{eq:minimax}\n  \\min_{x\\in\\R^d} \\max_{y\\in\\R^n}\\, \\Big\\{\\Psi(x,y):=f(x) + \\Phi(x,y) - h(y)\\Big\\}\n  % \\min_{x\\in\\R^d} \\Big\\{f(x) + \\max_{y\\in\\R^n}\\, \\{\\psi(x,y):= \\Phi(x,y) - h(y)\\}\\Big\\}\n\\end{equation}\n\n\\begin{equation}\n  \\label{eq:max-function}\n  \\phi(x) := \\max_{y\\in\\R^n}\\, \\Phi(x,y) - h(y),\\quad \\text{where $\\phi: \\R^d \\to \\R$}.\n\\end{equation}\n\n\\begin{equation*}\n  \\psi := \\phi+f, \\quad \\text{where $\\psi:\\R^d \\to \\R \\cup\\{+\\infty\\}$}.\n\\end{equation*}\n\n\nFor some $\\rho \\ge 0$, we say that\n\\begin{equation*}\n  \\label{eq:weakly_convex}\n  \\mbox{$\\psi:\\R^d \\to \\R\\cup\\{+\\infty\\}$ is \\emph{$\\rho$-weakly convex} if $\\psi + (\\rho/2) \\Vert \\cdot \\Vert^2$ is convex.}\n\\end{equation*}\nAn example of a weakly convex function is one which is differentiable and the gradient is uniformly Lipschitz continuous with constant $L$ (we call such a function $L$-smooth). In this case, the weak convexity parameter $\\rho$ is given by the Lipschitz constant.\n\n\nFor a proper, $\\rho$-weakly convex and l.s.c.\\ function $\\psi: \\R^d \\rightarrow \\R\\cup\\{+\\infty\\}$, the Moreau envelope of $\\psi$ with the parameter $\\lambda \\in (0,\\rho^{-1})$ is the function from $\\R^d$ to $\\R$ defined by\n\\begin{equation*}\n  \\label{eq:moreau}\n  \\psi_\\lambda(x) := \\inf_{z \\in \\R^d} \\Big\\{ \\psi(z) + \\frac{1}{2\\lambda}\\Vert z-x \\Vert^2\\Big\\}.\n\\end{equation*}\nThe proximal operator of the function $\\lambda \\psi$ is the $\\arg\\min$ of the right-hand side in this definition, that is,\n\\begin{equation}\n  \\label{eq:prox}\n  \\prox{\\lambda \\psi}{x} := \\argmin_{z \\in \\R^d}\\Big\\{ \\psi(z) + \\frac{1}{2\\lambda}\\Vert z-x \\Vert^2\\Big\\}.\n\\end{equation}\nNote that $\\prox{\\lambda \\psi}{x}$ is uniquely defined by~\\eqref{eq:prox} because the function being minimized is proper, l.s.c.\\ and strongly convex.\n% TODO: consider moving this paragraph past the definition of \\epsilon-Stationarity??!!\nFor a function $\\psi:\\R^d \\to \\R\\cup\\{+\\infty\\}$ and a point $\\bar{x}$ such that $\\psi(\\bar{x})$ is finite, the \\emph{Fr\\'echet subdifferential} of $\\psi$ at $\\bar{x}$, denoted by $\\partial \\psi(\\bar{x})$, is the set of all vectors $v \\in \\R^d$ such that\n\\begin{equation*}\n  \\label{eq:subgradient}\n  \\psi(x) \\ge \\psi(\\bar{x}) + \\langle v, x- \\bar{x}\\rangle + o(\\Vert x-\\bar{x} \\Vert) \\quad \\text{as $x \\to \\bar{x}$}.\n\\end{equation*}\nFor weakly convex function the Fr\\'echet subdifferential can simply be expressed in terms of the convex subdifferential of the (convex) function $\\psi+ (\\rho/2)\\Vert \\cdot \\Vert^2$.\n\n\nThe Moreau envelope allows us to naturally define a notion of \\emph{near stationarity} even for nonsmooth and $\\rho$-weakly convex functions.\n% We say that a point $\\bar{x}$ is stationary for a function if the Fr\\'echet subdifferential of the function contains $0$ at $\\bar{x}$. The concept of \\emph{nearly stationary} is not quite as straightforward. We will use the norm of the gradient of the Moreau envelope as a proxy.\nWe say that for an $\\epsilon>0$ and a $\\lambda\\in(0,\\rho^{-1})$\n\\begin{equation}\n  \\label{eq:epsilon-stationary}\n  \\mbox{a point $x$ is $\\epsilon$-\\emph{stationary} for $\\psi$ if $\\Vert \\nabla \\psi_\\lambda(x) \\Vert \\le \\epsilon$}.\n\\end{equation}\nCanonically, we call a point stationary if the above holds for $\\epsilon=0$.\n\n\nWe discuss the stochastic version of problem~\\eqref{eq:minimax} where the coupling function $\\Phi$ is actually given as an expectation,\n\\begin{equation*}\n  \\Phi(x,y) = \\E_{\\xi\\sim\\mathcal{D}}\\left[ \\Phi(x,y;\\xi)\\right] \\quad \\forall (x,y)\\in\\R^d\\times\\R^n\n\\end{equation*}\nand we can only access independent samples of the gradient $\\nabla_x\\Phi(x,y;\\xi)$ (or subgradient) and $\\nabla_y\\Phi(x,y;\\zeta)$, where $\\xi$ and $\\zeta$ are drawn from the (in general unknown) distribution $\\mathcal{D}$.\n\n\n\\begin{assumption}[unbiased]%\n  \\label{ass:unbiased}\n  The stochastic gradient estimator is unbiased, i.e.\n  \\begin{equation*}\n    \\Exp{\\nabla\\Phi(x,y;\\xi)} = \\nabla\\Phi(x,y) \\quad \\forall (x,y)\\in \\R^d\\times \\R^n,\n  \\end{equation*}\n  or in the case of subgradients\n  \\begin{equation*}\n    \\Exp{g^\\xi} \\in \\partial [\\Phi(\\cdot,y)](x), \\quad\\text{where $g^\\xi \\in \\partial [\\Phi(\\cdot,y;\\xi)](x)$}.\n  \\end{equation*}\n\\end{assumption}\n\n\\begin{assumption}[bounded variance]%\n  \\label{ass:bounded-variance}\n  The variance of the estimator is uniformly bounded, i.e.\\ for all $(x,y)\\in\\R^d\\times\\R^n$ and a variance $\\sigma^2\\ge0$ we have\n  \\begin{equation}\n    \\label{eq:bounded-variance}\n    \\Exp{\\Vert \\nabla\\Phi(x,y;\\xi) - \\nabla\\Phi(x,y) \\Vert^2} \\le \\sigma^2.\n  \\end{equation}\n  In the setting of Section~\\ref{sec:non-strongly-concave} where $\\Phi$ is not necessarily smooth in the first component, we make the analogous assumption for subgradients, i.e.\\\n  \\begin{equation}\n    \\label{eq:bounded-variance-subgradient}\n    \\Exp{\\left\\Vert g^\\xi-\\Exp{g^\\xi}  \\right\\Vert^2} \\le \\sigma^2,\n  \\end{equation}\n  for a stochastic subgradient $g^\\xi\\in \\partial[\\Phi(\\cdot,y;\\xi)](x)$.\n\\end{assumption}\n\n    Coupling function & $\\Phi(x,y)$ \\\\\n    Objective function & $\\Psi(x,y):= f(x) + \\Phi(x,y) - h(y)$ \\\\\n    Regularized coupling function & $\\Gamma(x,y):=\\Phi(x,y) - h(y)$ \\\\\n    Max function & $\\phi(x):= \\max_{y} \\Phi(x,y) - h(y)$ \\\\\n    Regularized max function & $\\psi(x):= f(x)+\\phi(x)$ \\\\\n\n\n\\begin{assumption}%\n  \\label{ass:weakly-convex-gradient-lipschitz}\n  The coupling function $\\Phi$ is\n  \\begin{enumerate}\n    \\item[(i)]\n      concave and $L_{\\nabla\\Phi}$-smooth in the second component uniformly in $x$,\n      \\begin{equation*}\n        \\Vert \\nabla_y\\Phi (x,y) - \\nabla_y\\Phi(x, y') \\Vert \\le L_{\\nabla\\Phi} \\Vert y - y' \\Vert \\quad \\forall x \\in \\R^d \\, \\forall y,y' \\in \\R^n.\n      \\end{equation*}\n      \\item[(ii)]\n        $\\rho$-weakly convex in the first component uniformly in the second one, i.e.\n        \\begin{equation*}\n          \\Phi(\\cdot,y) + \\frac{\\rho}{2}\\Vert \\cdot \\Vert^2 \\quad \\text{is convex for all $y\\in\\R^n$}.\n        \\end{equation*}\n  \\end{enumerate}\n\\end{assumption}\n\n\\begin{assumption}%\n  \\label{ass:max-function-lower-bounded}\n  The function $\\psi$ is lower bounded, i.e.\\ $\\inf_{x\\in\\R^d}\\, \\psi(x) > -\\infty$.\n\\end{assumption}\n\nIn Section~\\ref{sec:non-strongly-concave} we will actually need to bound the Moreau envelope $\\psi_\\lambda$, but these two conditions are in fact equivalent as for all $x\\in\\R^d$ and any $\\lambda\\in (0,\\rho^{-1})$\n\\begin{equation*}\n  \\psi(x) \\ge \\psi_\\lambda(x) \\ge \\inf_{u\\in\\R^d}\\, \\psi(u).\n\\end{equation*}\n\n\n\\begin{assumption}%\n  \\label{ass:lipschitz}\n  $\\Phi$ is $L$-Lipschitz in the first component uniformly over $\\dom h$ in the second one, i.e.\n  \\begin{equation*}\n    \\Vert \\Phi(x, y) - \\Phi(x', y) \\Vert \\le L \\Vert x - x' \\Vert \\quad \\forall x,x' \\in \\R^d\\; \\forall y \\in \\dom h.\n  \\end{equation*}\n\\end{assumption}\n\n\\begin{assumption}%\n  \\label{ass:regularizers}\n  The regularizers $f$ and $h$ are proper, l.s.c.\\ and convex\n  \\begin{enumerate}\n    \\item[(i)]\n      Additionally, $f$ is either $L_f$-Lipschitz continuous on its domain, which is assumed to be open, or the indicator of a nonempty, convex and closed set.\n      Either of those assumptions guarantees for any $\\gamma>0$ the bound\n      \\begin{equation}\n        \\label{eq:bound-prox-f}\n        \\Vert \\prox{\\gamma f}{x}-x \\Vert \\le \\gamma L_f \\quad \\forall x \\in \\dom f.\n      \\end{equation}\n    \\item[(ii)]\n    Furthermore, $h$ has a bounded domain $\\dom h$ such that the diameter of $\\dom h$ is bounded by $D_h$.\n  \\end{enumerate}\n\\end{assumption}\n\nThis implies that for every $x\\in\\R^d$ the set\n\\begin{equation}\n  \\label{eq:solution-set-mapping}\n  Y(x) := \\Big\\{y^* \\in \\R^n : \\phi(x) = \\Phi(x,y^*) -h(y^*) = \\max_{y\\in\\R^n} \\{\\Phi(x,y)-h(y)\\} \\Big\\}\n\\end{equation}\n\nis nonempty. For brevity we denote arbitrary elements of $Y(x_k)$ by $y_k^*$ for all $k\\ge0$.\n\n\nFor initial values $(x_0,y_0)\\in \\dom f \\times \\dom h$ the deterministic version of alternating GDA, for $g_k \\in \\partial [\\Phi(\\cdot,y_k)](x_k)$, reads as\n\\begin{equation}%\n  \\label{eq:alt-gda-non-strongly}\n  (\\forall k \\ge 0)\n  \\left\\lfloor\n    \\begin{array}{l}\n      x_{k+1} = \\prox{\\eta_x f}{x_k - \\eta_x g_k} \\\\\n      y_{k+1} = \\prox{\\eta_y h}{y_k + \\eta_y \\nabla_y\\Phi(x_{k+1},y_k)}.\n    \\end{array}\\right.\n\\end{equation}\n\n  For algorithm~\\eqref{eq:alt-gda-non-strongly} with the step sizes\n  \\begin{equation*}\n  \\eta_x=\\min \\left\\{\\frac{\\epsilon^4}{L_{\\nabla\\Phi}\\rho^2 D_h^2{(L+L_f)}^2}, \\frac{\\epsilon^2}{\\rho L^2}\\right\\}, \\eta_y=\\frac{1}{L_{\\nabla\\Phi}} \\,\\, \\text{and}\\,\\, \\lambda = \\frac{1}{2\\rho}\n  \\end{equation*}\n\n\nIn order to estimate the summation of $\\Delta_k$ we will use a trick to sum over it in blocks, where the size $B$ of these blocks will depend on the total number of iterations $K$. Note that w.l.o.g.\\ we assume that the block size $B \\le K$ divides $K$ without remainder.\n\n\nFor initial values $(x_0,y_0)\\in \\dom f \\times \\dom h$ the stochastic version of alternating GDA is given by\n\\begin{equation}%\n  \\label{eq:gda-non-strongly-stoch}\n  (\\forall k\\ge 0)\n  \\left\\lfloor\n    \\begin{array}{l}\n      x_{k+1} = \\prox{\\eta_x f}{x_k - \\eta_x g_k^\\xi} \\\\\n      y_{k+1} = \\prox{\\eta_y h}{y_k + \\eta_y \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k)},\n    \\end{array}\\right.\n\\end{equation}\n\nfor $g_k^\\xi \\in \\partial[\\Phi(\\cdot,y_k;\\xi_k)](x_k)$ for $\\xi_k,\\zeta_k\\sim\\mathcal{D}$ independent from all previous iterates.\\\\\n\n\nIn order to estimate the summation of $\\hat{\\Delta}_k$ we will use the same trick as in the deterministic setting and sum over it in blocks, where the size $B$ of these blocks will divide the total number of iterations $K$.\n","units":[{"start":4238,"end":4463,"start_line":134,"end_line":138,"raw_tex":"\\begin{equation}\n  \\label{eq:minimax}\n  \\min_{x\\in\\R^d} \\max_{y\\in\\R^n}\\, \\Big\\{\\Psi(x,y):=f(x) + \\Phi(x,y) - h(y)\\Big\\}\n  % \\min_{x\\in\\R^d} \\Big\\{f(x) + \\max_{y\\in\\R^n}\\, \\{\\psi(x,y):= \\Phi(x,y) - h(y)\\}\\Big\\}\n\\end{equation}","sha256_utf8":"f6c28f2b0313ada42cfabd0c8a79ada240ef5c4bfd47ddda35ed7a571bd1897e"},{"start":10046,"end":10191,"start_line":196,"end_line":199,"raw_tex":"\\begin{equation}\n  \\label{eq:max-function}\n  \\phi(x) := \\max_{y\\in\\R^n}\\, \\Phi(x,y) - h(y),\\quad \\text{where $\\phi: \\R^d \\to \\R$}.\n\\end{equation}","sha256_utf8":"7b60d8ef29af71c2ef92b29692b4a4fb05880a100911b46e69e144a10343ab6d"},{"start":23907,"end":24014,"start_line":318,"end_line":320,"raw_tex":"\\begin{equation*}\n  \\psi := \\phi+f, \\quad \\text{where $\\psi:\\R^d \\to \\R \\cup\\{+\\infty\\}$}.\n\\end{equation*}\n","sha256_utf8":"1b69bc0a7004ff85d88728848048d46e6919db3f37b046ca7ce8cba424fd76b9"},{"start":24779,"end":25264,"start_line":330,"end_line":335,"raw_tex":"For some $\\rho \\ge 0$, we say that\n\\begin{equation*}\n  \\label{eq:weakly_convex}\n  \\mbox{$\\psi:\\R^d \\to \\R\\cup\\{+\\infty\\}$ is \\emph{$\\rho$-weakly convex} if $\\psi + (\\rho/2) \\Vert \\cdot \\Vert^2$ is convex.}\n\\end{equation*}\nAn example of a weakly convex function is one which is differentiable and the gradient is uniformly Lipschitz continuous with constant $L$ (we call such a function $L$-smooth). In this case, the weak convexity parameter $\\rho$ is given by the Lipschitz constant.\n","sha256_utf8":"63a0c164321048453ba65922aeebb3f1e3ddc21b8e21984f6ccd780869610c62"},{"start":25590,"end":27102,"start_line":339,"end_line":356,"raw_tex":"For a proper, $\\rho$-weakly convex and l.s.c.\\ function $\\psi: \\R^d \\rightarrow \\R\\cup\\{+\\infty\\}$, the Moreau envelope of $\\psi$ with the parameter $\\lambda \\in (0,\\rho^{-1})$ is the function from $\\R^d$ to $\\R$ defined by\n\\begin{equation*}\n  \\label{eq:moreau}\n  \\psi_\\lambda(x) := \\inf_{z \\in \\R^d} \\Big\\{ \\psi(z) + \\frac{1}{2\\lambda}\\Vert z-x \\Vert^2\\Big\\}.\n\\end{equation*}\nThe proximal operator of the function $\\lambda \\psi$ is the $\\arg\\min$ of the right-hand side in this definition, that is,\n\\begin{equation}\n  \\label{eq:prox}\n  \\prox{\\lambda \\psi}{x} := \\argmin_{z \\in \\R^d}\\Big\\{ \\psi(z) + \\frac{1}{2\\lambda}\\Vert z-x \\Vert^2\\Big\\}.\n\\end{equation}\nNote that $\\prox{\\lambda \\psi}{x}$ is uniquely defined by~\\eqref{eq:prox} because the function being minimized is proper, l.s.c.\\ and strongly convex.\n% TODO: consider moving this paragraph past the definition of \\epsilon-Stationarity??!!\nFor a function $\\psi:\\R^d \\to \\R\\cup\\{+\\infty\\}$ and a point $\\bar{x}$ such that $\\psi(\\bar{x})$ is finite, the \\emph{Fr\\'echet subdifferential} of $\\psi$ at $\\bar{x}$, denoted by $\\partial \\psi(\\bar{x})$, is the set of all vectors $v \\in \\R^d$ such that\n\\begin{equation*}\n  \\label{eq:subgradient}\n  \\psi(x) \\ge \\psi(\\bar{x}) + \\langle v, x- \\bar{x}\\rangle + o(\\Vert x-\\bar{x} \\Vert) \\quad \\text{as $x \\to \\bar{x}$}.\n\\end{equation*}\nFor weakly convex function the Fr\\'echet subdifferential can simply be expressed in terms of the convex subdifferential of the (convex) function $\\psi+ (\\rho/2)\\Vert \\cdot \\Vert^2$.\n","sha256_utf8":"6edd79849b09ac572dbe44a5828a477b49fe97ff57b3e061945acc7e0b3a0d7f"},{"start":28832,"end":29579,"start_line":385,"end_line":392,"raw_tex":"The Moreau envelope allows us to naturally define a notion of \\emph{near stationarity} even for nonsmooth and $\\rho$-weakly convex functions.\n% We say that a point $\\bar{x}$ is stationary for a function if the Fr\\'echet subdifferential of the function contains $0$ at $\\bar{x}$. The concept of \\emph{nearly stationary} is not quite as straightforward. We will use the norm of the gradient of the Moreau envelope as a proxy.\nWe say that for an $\\epsilon>0$ and a $\\lambda\\in(0,\\rho^{-1})$\n\\begin{equation}\n  \\label{eq:epsilon-stationary}\n  \\mbox{a point $x$ is $\\epsilon$-\\emph{stationary} for $\\psi$ if $\\Vert \\nabla \\psi_\\lambda(x) \\Vert \\le \\epsilon$}.\n\\end{equation}\nCanonically, we call a point stationary if the above holds for $\\epsilon=0$.\n","sha256_utf8":"5cdbd888a12d75fb46ef174908d6d05e1704111904a59b75cc58612fe40b6c00"},{"start":30547,"end":31042,"start_line":411,"end_line":415,"raw_tex":"We discuss the stochastic version of problem~\\eqref{eq:minimax} where the coupling function $\\Phi$ is actually given as an expectation,\n\\begin{equation*}\n  \\Phi(x,y) = \\E_{\\xi\\sim\\mathcal{D}}\\left[ \\Phi(x,y;\\xi)\\right] \\quad \\forall (x,y)\\in\\R^d\\times\\R^n\n\\end{equation*}\nand we can only access independent samples of the gradient $\\nabla_x\\Phi(x,y;\\xi)$ (or subgradient) and $\\nabla_y\\Phi(x,y;\\zeta)$, where $\\xi$ and $\\zeta$ are drawn from the (in general unknown) distribution $\\mathcal{D}$.\n","sha256_utf8":"4e8f602ab196959b0cf150a12dbcd795893675fae918ae1cfaab644347fd3fef"},{"start":31144,"end":31576,"start_line":418,"end_line":428,"raw_tex":"\\begin{assumption}[unbiased]%\n  \\label{ass:unbiased}\n  The stochastic gradient estimator is unbiased, i.e.\n  \\begin{equation*}\n    \\Exp{\\nabla\\Phi(x,y;\\xi)} = \\nabla\\Phi(x,y) \\quad \\forall (x,y)\\in \\R^d\\times \\R^n,\n  \\end{equation*}\n  or in the case of subgradients\n  \\begin{equation*}\n    \\Exp{g^\\xi} \\in \\partial [\\Phi(\\cdot,y)](x), \\quad\\text{where $g^\\xi \\in \\partial [\\Phi(\\cdot,y;\\xi)](x)$}.\n  \\end{equation*}\n\\end{assumption}","sha256_utf8":"a9ef4b60495ca37352d20354abc70d0ad93a06b86b20a973d96a067b8eec8b7e"},{"start":31578,"end":32339,"start_line":430,"end_line":443,"raw_tex":"\\begin{assumption}[bounded variance]%\n  \\label{ass:bounded-variance}\n  The variance of the estimator is uniformly bounded, i.e.\\ for all $(x,y)\\in\\R^d\\times\\R^n$ and a variance $\\sigma^2\\ge0$ we have\n  \\begin{equation}\n    \\label{eq:bounded-variance}\n    \\Exp{\\Vert \\nabla\\Phi(x,y;\\xi) - \\nabla\\Phi(x,y) \\Vert^2} \\le \\sigma^2.\n  \\end{equation}\n  In the setting of Section~\\ref{sec:non-strongly-concave} where $\\Phi$ is not necessarily smooth in the first component, we make the analogous assumption for subgradients, i.e.\\\n  \\begin{equation}\n    \\label{eq:bounded-variance-subgradient}\n    \\Exp{\\left\\Vert g^\\xi-\\Exp{g^\\xi}  \\right\\Vert^2} \\le \\sigma^2,\n  \\end{equation}\n  for a stochastic subgradient $g^\\xi\\in \\partial[\\Phi(\\cdot,y;\\xi)](x)$.\n\\end{assumption}","sha256_utf8":"b5b82745109a2e997cbb2bdbc296e8a249e116079b67fb1c1af3874bacf27555"},{"start":33399,"end":33694,"start_line":476,"end_line":480,"raw_tex":"    Coupling function & $\\Phi(x,y)$ \\\\\n    Objective function & $\\Psi(x,y):= f(x) + \\Phi(x,y) - h(y)$ \\\\\n    Regularized coupling function & $\\Gamma(x,y):=\\Phi(x,y) - h(y)$ \\\\\n    Max function & $\\phi(x):= \\max_{y} \\Phi(x,y) - h(y)$ \\\\\n    Regularized max function & $\\psi(x):= f(x)+\\phi(x)$ \\\\\n","sha256_utf8":"04034e17235fb76680495142f90a62181133b9cab2ff1706fe922dba391ac6bf"},{"start":34765,"end":35472,"start_line":501,"end_line":516,"raw_tex":"\\begin{assumption}%\n  \\label{ass:weakly-convex-gradient-lipschitz}\n  The coupling function $\\Phi$ is\n  \\begin{enumerate}\n    \\item[(i)]\n      concave and $L_{\\nabla\\Phi}$-smooth in the second component uniformly in $x$,\n      \\begin{equation*}\n        \\Vert \\nabla_y\\Phi (x,y) - \\nabla_y\\Phi(x, y') \\Vert \\le L_{\\nabla\\Phi} \\Vert y - y' \\Vert \\quad \\forall x \\in \\R^d \\, \\forall y,y' \\in \\R^n.\n      \\end{equation*}\n      \\item[(ii)]\n        $\\rho$-weakly convex in the first component uniformly in the second one, i.e.\n        \\begin{equation*}\n          \\Phi(\\cdot,y) + \\frac{\\rho}{2}\\Vert \\cdot \\Vert^2 \\quad \\text{is convex for all $y\\in\\R^n$}.\n        \\end{equation*}\n  \\end{enumerate}\n\\end{assumption}","sha256_utf8":"2c17a7f1ae57724433b36ad2a999be6393226cc519827f6b9ab1e7df2fb850f5"},{"start":35925,"end":36087,"start_line":525,"end_line":528,"raw_tex":"\\begin{assumption}%\n  \\label{ass:max-function-lower-bounded}\n  The function $\\psi$ is lower bounded, i.e.\\ $\\inf_{x\\in\\R^d}\\, \\psi(x) > -\\infty$.\n\\end{assumption}","sha256_utf8":"a60aa33303a01f00f540652d4f39fce96da596b060f362eff34905afc3f45087"},{"start":36088,"end":36397,"start_line":529,"end_line":532,"raw_tex":"In Section~\\ref{sec:non-strongly-concave} we will actually need to bound the Moreau envelope $\\psi_\\lambda$, but these two conditions are in fact equivalent as for all $x\\in\\R^d$ and any $\\lambda\\in (0,\\rho^{-1})$\n\\begin{equation*}\n  \\psi(x) \\ge \\psi_\\lambda(x) \\ge \\inf_{u\\in\\R^d}\\, \\psi(u).\n\\end{equation*}\n","sha256_utf8":"dccac1ca050b07e3415925355030c1e6d60f2f74b40f66e6054858f28a4ae345"},{"start":36947,"end":37260,"start_line":542,"end_line":548,"raw_tex":"\\begin{assumption}%\n  \\label{ass:lipschitz}\n  $\\Phi$ is $L$-Lipschitz in the first component uniformly over $\\dom h$ in the second one, i.e.\n  \\begin{equation*}\n    \\Vert \\Phi(x, y) - \\Phi(x', y) \\Vert \\le L \\Vert x - x' \\Vert \\quad \\forall x,x' \\in \\R^d\\; \\forall y \\in \\dom h.\n  \\end{equation*}\n\\end{assumption}","sha256_utf8":"5d9c82e2a9f9f7715b96cbdcd243098331bde8ebb48a2b750d60eb1017248223"},{"start":37333,"end":38028,"start_line":551,"end_line":565,"raw_tex":"\\begin{assumption}%\n  \\label{ass:regularizers}\n  The regularizers $f$ and $h$ are proper, l.s.c.\\ and convex\n  \\begin{enumerate}\n    \\item[(i)]\n      Additionally, $f$ is either $L_f$-Lipschitz continuous on its domain, which is assumed to be open, or the indicator of a nonempty, convex and closed set.\n      Either of those assumptions guarantees for any $\\gamma>0$ the bound\n      \\begin{equation}\n        \\label{eq:bound-prox-f}\n        \\Vert \\prox{\\gamma f}{x}-x \\Vert \\le \\gamma L_f \\quad \\forall x \\in \\dom f.\n      \\end{equation}\n    \\item[(ii)]\n    Furthermore, $h$ has a bounded domain $\\dom h$ such that the diameter of $\\dom h$ is bounded by $D_h$.\n  \\end{enumerate}\n\\end{assumption}","sha256_utf8":"9c1757d7904b44f0c05b40f131a740952b778b662f5d5d93d37ca765f5e9630e"},{"start":38926,"end":39143,"start_line":574,"end_line":578,"raw_tex":"This implies that for every $x\\in\\R^d$ the set\n\\begin{equation}\n  \\label{eq:solution-set-mapping}\n  Y(x) := \\Big\\{y^* \\in \\R^n : \\phi(x) = \\Phi(x,y^*) -h(y^*) = \\max_{y\\in\\R^n} \\{\\Phi(x,y)-h(y)\\} \\Big\\}\n\\end{equation}","sha256_utf8":"8c0dc675ffbd96b97dfa0769b4731f58aa6dadeb206881551d8b6531d3a7c483"},{"start":39144,"end":39238,"start_line":579,"end_line":579,"raw_tex":"is nonempty. For brevity we denote arbitrary elements of $Y(x_k)$ by $y_k^*$ for all $k\\ge0$.\n","sha256_utf8":"45175b5a284f1791892092c5ab98ae8da269649fbce59b06f9108a82dc3c016d"},{"start":42336,"end":42764,"start_line":639,"end_line":648,"raw_tex":"For initial values $(x_0,y_0)\\in \\dom f \\times \\dom h$ the deterministic version of alternating GDA, for $g_k \\in \\partial [\\Phi(\\cdot,y_k)](x_k)$, reads as\n\\begin{equation}%\n  \\label{eq:alt-gda-non-strongly}\n  (\\forall k \\ge 0)\n  \\left\\lfloor\n    \\begin{array}{l}\n      x_{k+1} = \\prox{\\eta_x f}{x_k - \\eta_x g_k} \\\\\n      y_{k+1} = \\prox{\\eta_y h}{y_k + \\eta_y \\nabla_y\\Phi(x_{k+1},y_k)}.\n    \\end{array}\\right.\n\\end{equation}","sha256_utf8":"9f4498f0ef6fda279be486959e2d8f935ad408632323baf405eddeea5bcd1e2b"},{"start":42971,"end":43271,"start_line":653,"end_line":656,"raw_tex":"  For algorithm~\\eqref{eq:alt-gda-non-strongly} with the step sizes\n  \\begin{equation*}\n  \\eta_x=\\min \\left\\{\\frac{\\epsilon^4}{L_{\\nabla\\Phi}\\rho^2 D_h^2{(L+L_f)}^2}, \\frac{\\epsilon^2}{\\rho L^2}\\right\\}, \\eta_y=\\frac{1}{L_{\\nabla\\Phi}} \\,\\, \\text{and}\\,\\, \\lambda = \\frac{1}{2\\rho}\n  \\end{equation*}\n","sha256_utf8":"87c824959ba7e79f853573f96bf9fbdc3454889b976f942f54b54a18676acf42"},{"start":56314,"end":56586,"start_line":868,"end_line":868,"raw_tex":"In order to estimate the summation of $\\Delta_k$ we will use a trick to sum over it in blocks, where the size $B$ of these blocks will depend on the total number of iterations $K$. Note that w.l.o.g.\\ we assume that the block size $B \\le K$ divides $K$ without remainder.\n","sha256_utf8":"aad86a00faf20dabaad216da41323e10038256d91291118bcce3c306426cddb7"},{"start":60844,"end":61237,"start_line":946,"end_line":955,"raw_tex":"For initial values $(x_0,y_0)\\in \\dom f \\times \\dom h$ the stochastic version of alternating GDA is given by\n\\begin{equation}%\n  \\label{eq:gda-non-strongly-stoch}\n  (\\forall k\\ge 0)\n  \\left\\lfloor\n    \\begin{array}{l}\n      x_{k+1} = \\prox{\\eta_x f}{x_k - \\eta_x g_k^\\xi} \\\\\n      y_{k+1} = \\prox{\\eta_y h}{y_k + \\eta_y \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k)},\n    \\end{array}\\right.\n\\end{equation}","sha256_utf8":"2725515712121322a7d623ca040e49e2d204f14d42b85c1c110fc6a0d5cd3c65"},{"start":61238,"end":61370,"start_line":956,"end_line":956,"raw_tex":"for $g_k^\\xi \\in \\partial[\\Phi(\\cdot,y_k;\\xi_k)](x_k)$ for $\\xi_k,\\zeta_k\\sim\\mathcal{D}$ independent from all previous iterates.\\\\\n","sha256_utf8":"bcb282e26ff2e91f0a07bdf83077ff9384fd4532cefe0522782fca9f2f43f496"},{"start":71268,"end":71492,"start_line":1112,"end_line":1112,"raw_tex":"In order to estimate the summation of $\\hat{\\Delta}_k$ we will use the same trick as in the deterministic setting and sum over it in blocks, where the size $B$ of these blocks will divide the total number of iterations $K$.\n","sha256_utf8":"a65210a6e1de56af591ac07cd1ed98aa1078547f9c30a4cf9451c63982da4b9d"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}%\n  \\label{thm:alternating-rate-stoch}\n  Let in addition to the assumptions of Theorem~\\ref{thm:alternating-rate} also Assumption~\\ref{ass:unbiased} and~\\ref{ass:bounded-variance} hold true. For algorithm~\\eqref{eq:gda-non-strongly-stoch} with step sizes\n  \\begin{equation*}\n  \\eta_x = \\min \\left\\{\\frac{\\epsilon^2}{\\rho(L^2+\\sigma^2)}, \\frac{\\epsilon^4}{\\rho^2 L(L+L_f+\\sigma)D_h^2 L_{\\nabla\\Phi}}, \\frac{\\epsilon^6}{\\rho^3L(L+L_f+\\sigma)\\sigma^2D_h^2 }\\right\\},\n  \\end{equation*}\n  $\\eta_y= \\min\\{\\frac{1}{2 L_{\\nabla\\Phi}}, \\frac{\\epsilon^2}{\\rho \\sigma^2}\\}$ and $\\lambda=\\frac{1}{2\\rho}$ the number of stochastic gradient evaluations $K$ required is\n  % iterates generated by~\\eqref{eq:gda-non-strongly-stoch} with $\\eta_y=\\mathcal{O}(\\epsilon^2)\\le \\nicefrac{1}{2L_{\\nabla\\Phi}}$, $\\eta_x=\\mathcal{O}(\\epsilon^6)<\\nicefrac{1}{2\\rho}$ and $\\lambda=\\nicefrac{1}{2\\rho}$ fulfill\n  \\begin{equation*}\n      \\mathcal{O}\\left(\\frac{\\Delta^*(L^2+L_f^2+\\sigma^2)\\rho}{\\epsilon^4}\\max\\Big\\{1, \\frac{\\rho L_{\\nabla\\Phi}LD_h^2}{\\epsilon^2}, \\frac{\\rho^2 D_h^2 \\sigma^2}{\\epsilon^4}\\Big\\}+ \\frac{\\Delta_0\\rho}{\\epsilon^2}\\right),\n  \\end{equation*}\n  where $\\Delta^* = \\psi(x_0) - \\inf_{x\\in\\R^d} \\psi(x)$ and $\\Delta_0=\\psi(x_0)-\\Psi(x_0,y_0)$, to visit an $\\epsilon$-stationary point in expectation such that $\\min_{1\\le k \\le K}\\E[\\Vert \\nabla \\psi_\\lambda(x_k) \\Vert] \\le \\epsilon$.\n\\end{theorem}","units":[{"start":61372,"end":62779,"start_line":959,"end_line":971,"raw_tex":"\\begin{theorem}%\n  \\label{thm:alternating-rate-stoch}\n  Let in addition to the assumptions of Theorem~\\ref{thm:alternating-rate} also Assumption~\\ref{ass:unbiased} and~\\ref{ass:bounded-variance} hold true. For algorithm~\\eqref{eq:gda-non-strongly-stoch} with step sizes\n  \\begin{equation*}\n  \\eta_x = \\min \\left\\{\\frac{\\epsilon^2}{\\rho(L^2+\\sigma^2)}, \\frac{\\epsilon^4}{\\rho^2 L(L+L_f+\\sigma)D_h^2 L_{\\nabla\\Phi}}, \\frac{\\epsilon^6}{\\rho^3L(L+L_f+\\sigma)\\sigma^2D_h^2 }\\right\\},\n  \\end{equation*}\n  $\\eta_y= \\min\\{\\frac{1}{2 L_{\\nabla\\Phi}}, \\frac{\\epsilon^2}{\\rho \\sigma^2}\\}$ and $\\lambda=\\frac{1}{2\\rho}$ the number of stochastic gradient evaluations $K$ required is\n  % iterates generated by~\\eqref{eq:gda-non-strongly-stoch} with $\\eta_y=\\mathcal{O}(\\epsilon^2)\\le \\nicefrac{1}{2L_{\\nabla\\Phi}}$, $\\eta_x=\\mathcal{O}(\\epsilon^6)<\\nicefrac{1}{2\\rho}$ and $\\lambda=\\nicefrac{1}{2\\rho}$ fulfill\n  \\begin{equation*}\n      \\mathcal{O}\\left(\\frac{\\Delta^*(L^2+L_f^2+\\sigma^2)\\rho}{\\epsilon^4}\\max\\Big\\{1, \\frac{\\rho L_{\\nabla\\Phi}LD_h^2}{\\epsilon^2}, \\frac{\\rho^2 D_h^2 \\sigma^2}{\\epsilon^4}\\Big\\}+ \\frac{\\Delta_0\\rho}{\\epsilon^2}\\right),\n  \\end{equation*}\n  where $\\Delta^* = \\psi(x_0) - \\inf_{x\\in\\R^d} \\psi(x)$ and $\\Delta_0=\\psi(x_0)-\\Psi(x_0,y_0)$, to visit an $\\epsilon$-stationary point in expectation such that $\\min_{1\\le k \\le K}\\E[\\Vert \\nabla \\psi_\\lambda(x_k) \\Vert] \\le \\epsilon$.\n\\end{theorem}","sha256_utf8":"6c0119cb8715502ae3a20f9caf14f56fc88ed8389eb703e54fba94cd9cb88265"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}[{see~\\cite[Theorem 3.52]{mordukhovich-variational-analysis}}]%\n  \\label{lem:lipschitz-bounded-subgradients}\n  For an $L_\\psi$-Lipschitz continuous function $\\psi:\\R^d \\to \\R$ every Fr\\'echet subgradient is bounded in norm by $L_\\psi$.\n\\end{lemma}\n\n\\begin{lemma}[{see~\\cite[Lemma 2.2]{stoch_weakly_model_based}}]%\n  \\label{lem:grad_smooth_is_prox}\n  Let $\\psi: \\R^d \\to \\R\\cup\\{+\\infty\\}$ be a proper, $\\rho$-weakly convex, and l.s.c.\\ function, and let $\\lambda \\in (0,\\rho^{-1})$. Then the Moreau envelope $\\psi_\\lambda$ is continuously differentiable\n  on $\\R^d$ with gradient\n  \\begin{equation*}\n    \\nabla \\psi_\\lambda(x) = \\frac{1}{\\lambda}\\left(x - \\prox{\\lambda \\psi}{x}\\right) \\quad \\mbox{for all $x \\in \\R^d$},\n  \\end{equation*}\n  and this gradient is Lipschitz continuous.\n\\end{lemma}\n\n\\begin{proposition}[Subgradient characterization of the max function]%\n  \\label{thm:extended-danskin}\n  Let Assumption~\\ref{ass:weakly-convex-gradient-lipschitz} and~\\ref{ass:regularizers} hold true.\n  Then, the function $\\phi$, see~\\eqref{eq:max-function}, fulfills\n  \\begin{equation*}\n    \\partial [\\Phi(\\cdot,y^*)](x) \\subseteq \\partial\\phi(x) \\quad \\forall y^*\\in Y(x), \\forall x\\in \\R^d.\n  \\end{equation*}\n  In particular, $\\phi$ is $\\rho$-weakly convex.\n\\end{proposition}\n\n\\begin{proof}\n  From the $\\rho$-weak convexity of $\\Phi(\\cdot,y)$, we have that $\\Phi(\\cdot,y) + \\frac{\\rho}{2} \\|\\cdot\\|^2$ is convex for all $y\\in\\R^n$. We define $\\tilde{\\Phi}(x,y) = \\Phi(x,y) + \\frac\\rho2 \\|x\\|^2$ and $\\tilde{\\Gamma}(x,y) = \\Gamma(x,y)+\\frac{\\rho}{2}\\Vert x \\Vert^2$ for $(x,y)\\in\\R^d\\times\\R^n$ as well as\n  \\begin{equation*}\n    \\tilde{\\phi}(x) = \\max_{y \\in \\R^n} \\tilde{\\Gamma}(x,y) = \\phi(x) + \\frac{\\rho}{2} \\|x\\|^2.\n  \\end{equation*}\n  Notice that $\\tilde{\\Gamma}(x,\\cdot)$ is concave for any $x \\in \\R^d$ and $\\tilde{\\Gamma}(\\cdot,y)$ is convex for any $y \\in \\R^n$.\n  Thus, the function $\\tilde{\\phi}$ is convex and $\\dom \\tilde{\\phi} = \\dom \\phi = \\R^d$. Therefore $\\phi$ is continuous, which implies that $\\partial \\phi(x) \\neq \\emptyset$ for any $x \\in \\R^d$.\n  Let $x \\in \\R^d$, $y \\in Y(x)$ and $v \\in \\R^d$. For any $\\alpha >0$ it holds\n  \\begin{equation*}\n    \\frac{\\tilde{\\phi}(x+\\alpha v) - \\tilde{\\phi}(x)}{\\alpha} \\geq \\frac{\\tilde{\\Gamma}(x+\\alpha v, y) - \\tilde{\\Gamma}(x,y)}{\\alpha} =  \\frac{\\tilde{\\Phi}(x+\\alpha v, y) - \\tilde{\\Phi}(x,y)}{\\alpha},\n  \\end{equation*}\n  thus\n  \\begin{equation}\n    \\begin{aligned}\n      \\tilde{\\phi}'(x;v) &= \\inf_{\\alpha >0} \\frac{\\tilde{\\phi}(x+\\alpha v) - \\tilde{\\phi}(x)}{\\alpha} \\geq \\inf_{\\alpha >0} \\frac{\\tilde{\\Phi}(x+\\alpha v, y) - \\tilde{\\Phi}(x,y)}{\\alpha} = [\\tilde{\\Phi}(\\cdot,y)]'(x;v),\n    \\end{aligned}\n  \\end{equation}\n  where $[\\Phi(\\cdot,y)]'(x;v)$ denotes the directional derivative of $\\Phi$ in the first component at $x$ in the direction $v$.\n  In conclusion,\n  \\begin{equation}\n    \\label{eq1}\n    \\tilde{\\phi}'(x;v) \\geq \\sup_{y \\in Y(x)}  [\\tilde{\\Phi}(\\cdot,y)]'(x;v) \\quad \\forall v \\in \\R^d\n  \\end{equation}\n  and for $y\\in Y(x)$ we therefore conclude $\\partial [\\tilde{\\Phi}(\\cdot,y)](x) \\subseteq \\partial\\tilde{\\phi}(x)$.\n  The first statement is obtained by subtracting $\\rho x$ on both sides of the inclusion.\n\\end{proof}\n\n\\begin{lemma}[Lipschitz continuity of the max function]%\n  \\label{lem:max-is-lipschitz}\n  The Lipschitz continuity of $\\Phi$ in its first component implies that $\\phi$ is Lipschitz with the same constant.\n\\end{lemma}\n\n\\begin{proof}\n  Let $x,x' \\in \\R^d$ and $y^*\\in Y(x)$. On the one hand\n  \\begin{equation*}\n    \\begin{aligned}\n      \\phi(x) - \\phi(x') &= \\Phi(x,y^*)-h(y^*) - \\phi(x')\\\\\n                        &\\le \\Phi(x,y^*)-h(y^*) - \\Phi(x',y^*)+h(y^*)\\le L \\Vert x-x' \\Vert.\n    \\end{aligned}\n  \\end{equation*}\n  The reverse direction $\\phi(x') - \\phi(x) \\le L\\Vert x-x' \\Vert$ follows analogously.\n\\end{proof}\n\n\\begin{lemma}%\n  \\label{lem:xhatk-as-prox}\n  For any $\\lambda\\in(0,\\rho^{-1})$ and all $k\\ge0$ the point $\\hat{x}_k := \\prox{\\lambda g}{x_k}$ can also be written for some $v_k\\in \\partial \\phi(\\hat{x}_k)$ as\n  \\begin{equation*}\n    \\hat{x}_k = \\prox{\\eta_x f}{\\eta_x\\lambda^{-1}x_k - \\eta_x v_k + (1-\\eta_x\\lambda^{-1})\\hat{x}_k}.\n  \\end{equation*}\n\\end{lemma}\n\n\\begin{proof}\n  Let $k\\ge0$ be arbitrary but fixed and recall that $g=f+\\phi$. By the definition of $\\hat{x}_k$ we have that\n  \\begin{equation*}\n      0 \\in \\partial \\psi(\\hat{x}_k) + \\frac1\\lambda(\\hat{x}_k-x_k) = \\partial(\\phi+f)(\\hat{x}_k) + \\frac1\\lambda(\\hat{x}_k-x_k).\n  \\end{equation*}\n  % TODO: reference\n  We can estimate through the continuity of $\\phi$ and subdifferential calculus\n  \\begin{equation*}\n    \\frac1\\lambda(x_k - \\hat{x}_k) \\in \\partial(\\phi+f)(\\hat{x}_k) \\subseteq \\partial \\phi(\\hat{x}_k) + \\partial f(\\hat{x}_k).\n  \\end{equation*}\n  Thus, there exists $v_k \\in \\partial \\phi(\\hat{x}_k)$ such that $\\frac1\\lambda (x_k - \\hat{x}_k) \\in v_k + \\partial f(\\hat{x}_k)$.\n  % \\begin{equation*}\n  %   \\frac1\\lambda (x_k - \\hat{x}_k) \\in v_k + \\partial f(\\hat{x}_k).\n  % \\end{equation*}\n  Also,\n  \\begin{equation*}\n    \\begin{aligned}\n      \\frac1\\lambda(x_k - \\hat{x}_k) \\in  \\partial f(\\hat{x}_k) +  v_k &\\Leftrightarrow \\frac{\\eta_x }{\\lambda}x_k - \\eta_x v_k + (1-\\frac{\\eta_x}{\\lambda})\\hat{x}_k \\in \\hat{x}_k + \\eta_x \\partial f(\\hat{x}_k) \\\\\n      &\\Leftrightarrow \\hat{x}_k = \\prox{\\eta_x f}{\\frac{\\eta_x}{\\lambda}x_k - \\eta_x v_k + \\big(1-\\frac{\\eta_x}{\\lambda}\\big)\\hat{x}_k}.\n    \\end{aligned}\n  \\end{equation*}\n\\end{proof}\n\n\\begin{lemma}%\n  \\label{lem:descent-on-moreau}\n  With $\\lambda = \\nicefrac{1}{2\\rho}$ and $\\eta_x \\ge 0$ we have for all $k\\ge0$ that\n  \\begin{equation*}\n    \\psi_\\lambda(x_{k+1}) \\le \\psi_\\lambda(x_k) + 2\\rho\\eta_x\\Delta_k - \\frac12\\eta_x \\Vert \\nabla \\psi_\\lambda(x_k) \\Vert^2  + 4\\rho\\eta_x^2L^2,\n  \\end{equation*}\n  where $\\Delta_k := \\psi(x_k) - \\Psi(x_k, y_k) \\ge 0$.\n\\end{lemma}\n\n\\begin{proof}\n  Let $k\\ge0$ be fixed. As before we denote $\\hat{x}_k = \\prox{\\lambda \\psi}{x_k}$. From the definition of the Moreau envelope we have that\n  \\begin{equation}\n    \\label{eq:moreau-is-min}\n    \\psi_\\lambda(x_{k+1}) = \\min_{x\\in\\R^d} \\Big\\{ \\psi(x) + \\frac{1}{2\\lambda} \\Vert x - x_{k+1} \\Vert^2\\Big\\} \\le \\psi(\\hat{x}_k) + \\frac{1}{2\\lambda} \\Vert \\hat{x}_k - x_{k+1} \\Vert^2.\n  \\end{equation}\n  Let now $v_k \\in \\partial\\phi(\\hat{x}_k)$ as in Lemma~\\ref{lem:xhatk-as-prox}. We successively deduce for $\\beta:=1-\\eta_x \\lambda^{-1}$\n  \\begin{align}\n    \\Vert \\hat{x}_k - x_{k+1} \\Vert^2 &= \\Vert \\prox{\\eta_x f}{\\eta_x\\lambda^{-1}x_k - \\eta_x v_k + \\beta\\hat{x}_k} - \\prox{\\eta_x f}{x_k - \\eta_x g_k}\\Vert^2 \\label{eq:defin-iter}\\\\\n     &\\le \\Vert \\beta(\\hat{x}_k - x_k) + \\eta_x (g_k - v_k)\\Vert^2 \\label{eq:1-Lipschitz}\\\\\n     &= {\\beta}^2\\Vert \\hat{x}_k - x_k \\Vert^2 + 2 \\eta_x\\beta\\langle g_k-v_k, \\hat{x}_k - x_k\\rangle\n     + \\eta_x^2\\Vert g_k - v_k \\Vert^2 \\notag\\\\\n     &\\le {\\beta}^2\\Vert \\hat{x}_k - x_k \\Vert^2 + 2 \\eta_x \\beta\\langle g_k-v_k, \\hat{x}_k - x_k\\rangle + 4\\eta_x^2L^2 \\label{eq:Lipschitz}\n  \\end{align}\n  where~\\eqref{eq:defin-iter} uses Lemma~\\ref{lem:xhatk-as-prox} and the definition of $x_{k+1}$, inequality~\\eqref{eq:1-Lipschitz} holds because of the nonexpansiveness of the proximal operator, and~\\eqref{eq:Lipschitz} follows from the Lipschitz continuity of $\\Phi$ and $\\phi$ (see Lemma~\\ref{lem:max-is-lipschitz}) and the fact that Lipschitz continuity implies bounded subgradients.\n  We are left with estimating the inner product in the above inequality and we do so by splitting it into two:\n  first of all, from the weak convexity of $\\Phi$ in $x$ we have that\n  \\begin{equation*}\n    \\begin{aligned}\n      \\langle g_k, \\hat{x}_k - x_k \\rangle &\\le \\Phi(\\hat{x}_k, y_k) - \\Phi(x_k, y_k) + \\frac{\\rho}{2} \\Vert \\hat{x}_k - x_k \\Vert^2 \\\\\n      % &= \\Phi(\\hat{x}_k, y_k)-h(y_k) - (\\Phi(x_k, y_k)-h(y_k)) + \\frac{\\rho}{2} \\Vert \\hat{x}_k - x_k \\Vert^2\\\\\n      &\\le \\phi(\\hat{x}_k) - \\Gamma(x_k,y_k) + \\frac{\\rho}{2} \\Vert \\hat{x}_k - x_k \\Vert^2.\n    \\end{aligned}\n  \\end{equation*}\n  Secondly, by the $\\rho$-weak convexity of $\\phi$\n  \\begin{equation*}\n    -\\langle v_k, \\hat{x}_k - x_k\\rangle \\le \\phi(x_k) - \\phi(\\hat{x}_k) + \\frac{\\rho}{2} \\Vert \\hat{x}_k - x_k \\Vert^2.\n  \\end{equation*}\n  Combining the last two inequalities we get that\n  \\begin{equation}\n    \\label{eq:estimate-middle}\n    \\langle g_k-v_k, \\hat{x}_k - x_k\\rangle \\le \\phi(x_k) - \\Gamma(x_k,y_k) + \\rho \\Vert \\hat{x}_k - x_k \\Vert^2.\n  \\end{equation}\n  Plugging~\\eqref{eq:estimate-middle} into~\\eqref{eq:Lipschitz} we deduce\n  \\begin{equation}\n    \\begin{aligned}\n    \\label{eq:ab3}\n    \\Vert \\hat{x}_k - x_{k+1} \\Vert^2 \\le \\underbrace{[{(1-\\eta_x\\lambda^{-1})}^2+2\\eta_x(1-\\eta_x \\lambda^{-1})\\rho]}_{= (*)} \\Vert \\hat{x}_k - x_k \\Vert^2 + 2\\eta_x \\Delta_k + 4\\eta_x^2L^2, \\\\\n    % &\\le \\Vert \\hat{x}_k - x_k \\Vert^2 + 2\\eta_x\\Delta_k -2\\eta_x\\rho \\Vert\\hat{x}_k - x_k \\Vert^2  + 2\\eta_x^2 L^2\n    \\end{aligned}\n  \\end{equation}\n  where we used the fact that $1-\\beta\\le1$ in the factor of $\\Delta_k$.\n  Now note that\n  \\begin{equation}\n    \\label{eq:terrible-algebra}\n    \\begin{aligned}\n      (*) &= 1 - 2\\eta_x \\lambda^{-1} + \\eta_x^2 \\lambda^{-2} + 2\\eta_x\\rho - 2\\eta_x^2 \\lambda^{-1}\\rho\\\\\n      &= 1 - 4\\eta_x\\rho + 4 \\eta_x^2\\rho^2 + 2\\eta_x\\rho - 4 \\eta_x^2\\rho^2 = 1-2\\eta_x\\rho.\n    \\end{aligned}\n  \\end{equation}\n  Combining~\\eqref{eq:moreau-is-min},~\\eqref{eq:ab3} and~\\eqref{eq:terrible-algebra} we deduce, using $\\lambda=\\nicefrac{1}{2\\rho}$,\n  % a lot of algebra happens here but it's correct\n  \\begin{equation*}\n    \\label{eq:descent-moreau}\n    \\begin{aligned}\n      \\psi_\\lambda(x_{k+1}) &\\le \\psi(\\hat{x}_k) + \\frac{1}{2\\lambda} \\left(\\Vert \\hat{x}_k - x_k \\Vert^2 + 2\\eta_x\\Delta_k -2\\eta_x\\rho \\Vert\\hat{x}_k - x_k \\Vert^2  + 4\\eta_x^2L^2\\right) \\\\\n      &= \\psi_\\lambda(x_k) + 2\\rho\\eta_x\\Delta_k - \\frac12\\eta_x \\Vert \\nabla \\psi_\\lambda(x_k) \\Vert^2  + 4\\rho\\eta_x^2L^2.\n    \\end{aligned}\n  \\end{equation*}\n  % where we used that $\\lambda=\\nicefrac{1}{2\\rho}$.\n\\end{proof}\n\n\\begin{lemma}%\n  \\label{lem:standard-gradient-ascent-estimation}\n  It holds for all $y\\in\\R^n$ and $k\\ge0$ that\n  \\begin{equation}\n    \\label{eq:gradient-ascent-estimation}\n    \\Psi(x_{k+1}, y) - \\Psi(x_{k+1}, y_{k+1}) \\le  \\frac{1}{2\\eta_y} \\Big(\\Vert y - y_k \\Vert^2 - \\Vert y - y_{k+1} \\Vert^2 \\Big).\n  \\end{equation}\n\\end{lemma}\n\n\\begin{proof}\n  This is a standard estimate on the improvement made by a single prox-gradient step for a convex (in this case concave) function, see for example~\\cite[Lemma 2.3]{fista}.\n  % By the definition of $y_{k+1}$ we have that\n  % \\begin{equation*}\n  %   y_{k+1} = \\argmin_{y\\in\\R^n} \\Big\\{ h(y) - \\langle\\nabla_y\\Phi(x_{k+1},y_k), y-y_k\\rangle + \\frac{1}{2\\eta_y}\\Vert y-y_k \\Vert^2 \\Big\\}.\n  % \\end{equation*}\n  % Let now $y\\in\\R^n$ be arbitrary but fixed. Since $y_{k+1}$ minimizes a $\\nicefrac{1}{\\eta_y}$-strongly convex function,\n  % \\begin{equation}\n  %   \\label{eq:decrease}\n  %   \\begin{aligned}\n  %     \\MoveEqLeft h(y_{k+1}) - \\langle \\nabla_y\\Phi(x_{k+1}, y_k), y_{k+1}-y_k \\rangle + \\frac{1}{2\\eta_y} \\Vert y_{k+1} -y_k \\Vert^2 + \\frac{1}{2\\eta_y} \\Vert y-y_{k+1} \\Vert^2  \\\\\n  %     &\\le h(y) - \\langle \\nabla_y\\Phi(x_{k+1}, y_k), y-y_k \\rangle + \\frac{1}{2\\eta_y} \\Vert y-y_k \\Vert^2.\n  %   \\end{aligned}\n  % \\end{equation}\n  % From the descent lemma (in ascent form) and the fact that $\\nicefrac{1}{\\eta_y} = L_{\\nabla\\Phi}$ we have that\n  % \\begin{equation}\n  %   \\label{eq:ascent-lemma}\n  %   \\Phi(x_{k+1}, y_k) + \\langle \\nabla_y\\Phi(x_{k+1}, y_k),  y_{k+1} - y_k\\rangle - \\frac{1}{2\\eta_y} \\Vert y_{k+1} - y_k \\Vert^2 \\le \\Phi(x_{k+1}, y_{k+1}).\n  % \\end{equation}\n  % From the concavity of $\\Phi$ in its second component we get that\n  % \\begin{equation}\n  %   \\label{eq:concavity}\n  %   \\Phi(x_{k+1}, y) \\le \\Phi(x_{k+1}, y_k) + \\langle \\nabla_y\\Phi(x_{k+1}, y_k), y-y_k \\rangle.\n  % \\end{equation}\n  % By plugging~\\eqref{eq:concavity} and~\\eqref{eq:ascent-lemma} into~\\eqref{eq:decrease} we deduce\n  % \\begin{equation*}\n  %   \\Phi(x_{k+1},y) -h(y) + \\frac{1}{2\\eta_y}\\Vert y-y_{k+1} \\Vert^2 \\le \\Phi(x_{k+1},y_{k+1})-h(y_{k+1}) + \\frac{1}{2\\eta_y}\\Vert y-y_k \\Vert^2.\n  % \\end{equation*}\n  % The statement of the lemma is obtained by rearranging the above inequality and adding $f(x_{k+1})$ on both sides.\n\\end{proof}\n\n\\begin{lemma}%\n  \\label{lem:estimate-Delta}\n  We have that for all $1 \\le m \\le k$,\n  \\begin{equation}\n    \\begin{aligned}\n      \\label{eq:estimate-Delta}\n      \\Delta_k \\le 2\\eta_x L(L+L_f)(k-m) + \\frac{1}{2\\eta_y} \\Big(\\Vert y_{k-1} - y^*_m \\Vert^2 - \\Vert y_k - y^*_m \\Vert^2 \\Big).\n    \\end{aligned}\n  \\end{equation}\n\\end{lemma}\n\n\\begin{proof}\n  % Let $1\\le m \\le k$ be fixed.\n  Plugging $y=y^*_m$ into~\\eqref{eq:gradient-ascent-estimation} we deduce that\n  \\begin{equation}\n    \\label{eq:standard_descent_with_ystar}\n    \\begin{aligned}\n       0 \\le \\Psi(x_k, y_k) - \\Psi(x_k, y^*_m) + \\frac{1}{2\\eta_y} \\Big(\\Vert y^*_m - y_{k-1} \\Vert^2 - \\Vert y^*_m - y_k \\Vert^2 \\Big).\n    \\end{aligned}\n  \\end{equation}\n  Starting from the definition of $\\Delta_k = \\Psi(x_k,y^*_k) - \\Psi(x_k,y_k)$, we add~\\eqref{eq:standard_descent_with_ystar} to obtain\n  \\begin{equation}\n    \\label{eq:diff-function-values}\n    \\begin{aligned}\n      \\Delta_k %&= \\Psi(x_k,y^*_k) - \\Psi(x_k,y_k) \\\\\n      &\\le \\Psi(x_k,y^*_k) - \\Psi(x_k,y_m^*) + \\frac{1}{2\\eta_y} \\Big(\\Vert y^*_m - y_{k-1} \\Vert^2 - \\Vert y^*_m - y_k \\Vert^2 \\Big).\n    \\end{aligned}\n  \\end{equation}\n  Due to the Lipschitz continuity of $\\Phi$, terms which only differ in their first argument will be easy to estimate. Therefore, we insert and subtract $\\Phi(x_m,y_{k}^*)$ to deduce\n  \\begin{equation}\n    \\label{eq:plus-minus-functionval}\n    \\begin{aligned}\n      \\MoveEqLeft \\Psi(x_k, y^*_k) - \\Psi(x_k, y^*_m)  \\\\ %&= \\Phi(x_k, y^*_k)-h(y^*_k) - \\Phi(x_k, y^*_m)+h(y^*_m) \\\\\n      &= \\Phi(x_k, y^*_k) - \\Phi(x_m, y^*_k) + \\Phi(x_m, y^*_k)-h(y^*_k) - \\Phi(x_k, y^*_m) + h(y^*_m) \\\\\n      &\\le \\Phi(x_k, y^*_k) - \\Phi(x_m, y^*_k) + \\Phi(x_m, y^*_m)-h(y^*_m) - \\Phi(x_k, y^*_m) + h(y^*_m)\\\\\n      &= \\Phi(x_k,y^*_k) - \\Phi(x_m,y^*_k) + \\Phi(x_m,y^*_m) - \\Phi(x_k, y^*_m).\n    \\end{aligned}\n  \\end{equation}\n  We estimate the above expression for $k>m$ by making use of the Lipschitz continuity of $\\Phi(\\cdot, y)$ and~\\eqref{eq:bound-prox-f} deducing\n  \\begin{equation}\n    \\label{eq:fun-val-same-y}\n    \\begin{aligned}\n      \\MoveEqLeft \\Phi(x_k, y^*_k) - \\Phi(x_m, y^*_k)  \\le L \\Vert x_k - x_m \\Vert \\le L \\sum_{l=m}^{k-1} \\Vert x_{l+1}-x_l \\Vert \\\\\n      &\\le L \\sum_{l=m}^{k-1} \\Big( \\Vert \\prox{\\eta_x f}{x_l -\\eta_x g_l} - \\prox{\\eta_x f}{x_l} \\Vert + \\Vert \\prox{\\eta_x f}{x_l} - x_l \\Vert \\Big)\\\\\n      & \\le \\eta_x L(L+L_f)(k-m).\n    \\end{aligned}\n  \\end{equation}\n  For $k=m$ the inequality follows trivially. Analogously, we deduce\n  \\begin{equation}\n    \\label{eq:fun-val-same-y-2}\n      \\Phi(x_m, y^*_m) - \\Phi(x_k, y^*_m) \\le \\eta_x L(L+L_f)(k-m).\n  \\end{equation}\n  Plugging~\\eqref{eq:plus-minus-functionval},~\\eqref{eq:fun-val-same-y} and~\\eqref{eq:fun-val-same-y-2} into~\\eqref{eq:diff-function-values} gives the statement of the lemma.\n\\end{proof}\n\n\\begin{lemma}%\n  \\label{lem:estimate-sum-of-delta}\n  It holds that for all $K\\ge1$\n  \\begin{equation}\n    \\label{eq:estimate-sum-of-delta}\n    \\frac{1}{K} \\sum_{k=0}^{K-1} \\Delta_k \\le \\eta_x L(L+L_f) B + \\frac{L_{\\nabla\\Phi}D_h^2}{2B} + \\frac{\\Delta_0}{K}.\n  \\end{equation}\n\\end{lemma}\n\n\\begin{proof}\n  By splitting the summation into blocks we get that\n  \\begin{equation}\n    \\label{eq:split--summation-into-blocks}\n    \\sum_{k=0}^{K-1} \\Delta_k  = \\sum_{j=0}^{K/B-1} \\ \\sum_{k=jB}^{(j+1)B-1} \\Delta_k.\n  \\end{equation}\n  By using~\\eqref{eq:estimate-Delta} from Lemma~\\ref{lem:estimate-Delta} with $j>0$ and $m=jB$ and the fact that $\\sum_{k=1}^{B-1} k \\le \\nicefrac{B^2}{2}$ we have\n  \\begin{equation}\n    \\label{eq:sum-over-any-block}\n    \\begin{aligned}\n      \\sum_{k=jB}^{(j+1)B-1} \\Delta_k &\\le \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y} \\Vert y_{jB-1}-y^*_{jB} \\Vert^2 \\\\\n      % &\\le \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y}D_h^2,\n    \\end{aligned}\n  \\end{equation}\n  where the last term can be bounded by $D_h^2$, which was defined in Assumption~\\ref{ass:regularizers} and denotes the diameter of $\\dom h$.\n  We do the same for the case $j=0$ but choose here $m=1$ and have separate out the first summand of $\\sum_{k=0}^{B-1} \\Delta_k$ as Lemma~\\ref{lem:estimate-Delta} does not hold for $k=0$ and therefore get an extra $\\Delta_0$ summand.\n  % with $m=1$ and the fact that $\\sum_{k=1}^{B-1} k \\le \\nicefrac{B^2}{2}$ we get that\n  % \\begin{equation}\n  %   \\label{eq:sum-first-block}\n  %   \\begin{aligned}\n  %     \\sum_{k=0}^{B-1} \\Delta_k &\\le \\Delta_0 + \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y} \\Vert y_{0}-y^*_{1} \\Vert^2 \\\\\n  %     &\\le \\Delta_0 + \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y}D_h^2,\n  %   \\end{aligned}\n  % \\end{equation}\n  where $D_h$ was defined in Assumption~\\ref{ass:regularizers} and denotes the diameter of $\\dom h$.\n  % Analogously, for $j>0$ and $m=jB$ we have that\n  % \\begin{equation}\n  %   \\label{eq:sum-over-any-block}\n  %   \\begin{aligned}\n  %     \\sum_{k=jB}^{(j+1)B-1} \\Delta_k &\\le \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y} \\Vert y_{jB-1}-y^*_{jB} \\Vert^2 \\\\\n  %     &\\le \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y}D_h^2.\n  %   \\end{aligned}\n  % \\end{equation}\n  Plugging~\\eqref{eq:sum-over-any-block} into~\\eqref{eq:split--summation-into-blocks} gives\n  \\begin{equation*}\n    % \\label{eq:sum-over-all}\n    \\frac{1}{K} \\sum_{k=0}^{K-1} \\Delta_k \\le \\eta_x L(L+L_f) B + \\frac{1}{2\\eta_y B}D_h^2 + \\frac{\\Delta_0}{K}.\n  \\end{equation*}\n  The desired statement is obtained by using the step size $\\eta_y=\\nicefrac{1}{L_{\\nabla\\Phi}}$.\n\\end{proof}\n\n\\begin{lemma}%\n  \\label{lem:stoch-descent-on-moreau}\n  With $\\lambda = \\nicefrac{1}{2\\rho}$ we have for all $k\\ge0$ that\n  \\begin{equation*}\n    \\Exp{\\psi_\\lambda(x_{k+1})} \\le \\Exp{\\psi_\\lambda(x_k)} + 2\\rho\\eta_x\\hat{\\Delta}_k - \\frac{\\eta_x}{2}\\Exp{\\Vert \\nabla \\psi_\\lambda(x_k) \\Vert^2} + 4\\rho\\eta_x^2(L^2+\\sigma^2)\n  \\end{equation*}\n  where $\\hat{\\Delta}_k := \\Exp{\\psi(x_k) - \\Psi(x_k, y_k)}$.\n\\end{lemma}\n\n\\begin{proof}\n  Let $k\\ge0$ be arbitrary but fixed. It follows easily from~\\eqref{eq:bounded-variance-subgradient} that\n  \\begin{equation}\n    \\label{eq:steiner}\n    \\Exp{\\Vert g_k^\\xi \\Vert^2} \\le \\Exp{\\Vert g_k \\Vert^2} + \\sigma^2 \\le L^2 + \\sigma^2,\n  \\end{equation}\n  where $\\Exp{g_k^\\xi}=g_k \\in \\partial_x\\Phi(x_k,y_k)$.\n  The definition of the Moreau envelope yields\n  \\begin{equation}\n    \\label{eq:defin-moreau}\n    \\Exp{g_\\lambda(x_{k+1})} \\le \\Exp{\\psi(\\hat{x}_k)} + \\frac{1}{2\\lambda}\\Exp{\\Vert \\hat{x}_k - x_{k+1} \\Vert^2}.\n  \\end{equation}\n  Similarly to Lemma~\\ref{lem:descent-on-moreau} we deduce that for $v_k\\in \\partial\\phi(\\hat{x}_k)$ (as given in Lemma~\\ref{lem:xhatk-as-prox}) and $\\beta=1-\\eta_x \\lambda^{-1}$\n  \\begin{equation*}\n    \\begin{aligned}\n      \\Vert\\hat{x}_k - x_{k+1} \\Vert^2 %&= \\Vert \\prox{\\eta_x f}{\\eta_x\\lambda^{-1}x_k - \\eta_x v_k + \\beta\\hat{x}_k} - \\prox{\\eta_x f}{x_k - \\eta_x g_k^\\xi}\\Vert^2 \\\\\n       %&\\le \\Vert \\beta(\\hat{x}_k - x_k) + \\eta_x (g_k^\\xi - v_k)\\Vert^2 \\\\\n       &= {\\beta}^2\\Vert \\hat{x}_k - x_k \\Vert^2 + 2 \\eta_x \\beta\\langle g_k^\\xi - v_k, \\hat{x}_k - x_k\\rangle\n       + \\eta_x^2\\Vert g_k^\\xi - v_k \\Vert^2.\n    \\end{aligned}\n  \\end{equation*}\n  By applying the conditional expectation $\\Expcond{\\cdot}{x_k,y_k}$, then the unconditional one and using~\\eqref{eq:steiner}, we get that\n  \\begin{equation*}\n    \\Exp{\\Vert \\hat{x}_k - x_{k+1} \\Vert^2} \\le \\Exp{\\Vert \\hat{x}_k - x_k \\Vert^2 + 2 \\eta_x(1-\\frac{\\eta_x}{\\lambda}) \\langle g_k-v_k, \\hat{x}_k - x_k\\rangle} + 4\\eta_x^2(L^2+\\sigma^2).\n  \\end{equation*}\n  where $g_k= \\Exp{g_k^\\xi}$.\n  Lastly, we combine the above inequality with~\\eqref{eq:defin-moreau} and the estimate for the inner product~\\eqref{eq:estimate-middle} as in Lemma~\\ref{lem:descent-on-moreau} to deduce the statement of the lemma.\n\\end{proof}\n\n\\begin{lemma}%\n  \\label{lem:stoch-standard-gradient-ascent-estimation}\n  With $\\eta_y\\le \\nicefrac{1}{2L_{\\nabla\\Phi}}$ we have for all $k\\ge0$ and all $y \\in \\R^n$\n  \\begin{equation}\n    \\label{eq:stoch-gradient-ascent-estimation}\n    \\begin{aligned}\n      \\MoveEqLeft \\Exp{\\Psi(x_{k+1}, y) - \\Psi(x_{k+1}, y_{k+1})} \\le  \\frac{1}{2\\eta_y} \\Big(\\Exp{\\Vert y^*_m - y_k \\Vert^2 - \\Vert y^*_m - y_{k+1} \\Vert^2} \\Big) + \\eta_y\\sigma^2.\n    \\end{aligned}\n  \\end{equation}\n\\end{lemma}\n\n\\begin{proof}\n  Let $k\\ge0$ and $y\\in\\R^n$ be arbitrary but fixed.\n  % By the definition of $y_{k+1}$ we have that\n  % \\begin{equation*} y_{k+1} = \\argmin_{y\\in\\R^n} \\Big\\{ h(y) - \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k), y-y_k\\rangle + \\frac{1}{2\\eta_y}\\Vert y-y_k \\Vert^2 \\Big\\}.\n  % \\end{equation*}\n  As in Lemma~\\ref{lem:standard-gradient-ascent-estimation}, we deduce\n  \\begin{equation*}\n    % \\label{eq:stoch-decrease}\n    \\begin{aligned}\n      h(y_{k+1}) - \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k), y_{k+1}-y_k \\rangle + \\frac{1}{2\\eta_y} \\Big(\\Vert y_{k+1} -y_k \\Vert^2 + \\Vert y-y_{k+1} \\Vert^2 \\Big)  \\\\\n      \\le h(y) - \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k), y-y_k \\rangle + \\frac{1}{2\\eta_y} \\Vert y-y_k \\Vert^2.\n    \\end{aligned}\n  \\end{equation*}\n  The term $\\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k), y_{k+1}-y_k \\rangle$ is problematic, because the right~hand side of the inner product is not measurable with respect to the sigma algebra generated by past iterates $\\mathcal{F}_k := \\sigma\\{x_{k+1},\\dots, x_1, y_k, \\dots, y_1\\}$, so we insert and subtract $\\langle \\nabla_y\\Phi(x_{k+1},y_k) , y_{k+1}-y_k\\rangle$\n  % to deduce\n  % \\begin{multline*}\n  %     h(y_{k+1}) + \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k),y-y_k\\rangle + \\frac{1}{2\\eta_y} \\Vert y_{k+1} -y_k \\Vert^2 + \\frac{1}{2\\eta_y} \\Vert y-y_{k+1} \\Vert^2 \\\\\n  %     \\le h(y) + \\langle \\nabla_y\\Phi(x_{k+1},y_k), y_{k+1}-y_k \\rangle  + \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k) - \\nabla_y\\Phi(x_{k+1},y_k), y_{k+1}-y_k \\rangle + \\frac{1}{2\\eta_y} \\Vert y-y_k \\Vert^2.\n  % \\end{multline*}\n  Now, using Young's inequality we estimate the resulting inner product\n  \\begin{multline*}\n    % \\label{eq:young2}\n    \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k) - \\nabla_y\\Phi(x_{k+1},y_k), y_{k+1}-y_k \\rangle \\\\\n    \\le \\eta_y\\Vert \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k) - \\nabla_{y}\\Phi(x_{k+1},y_k) \\Vert^2 + \\frac{1}{4\\eta_y} \\Vert y_{k+1} - y_k \\Vert^2.\n  \\end{multline*}\n  Combining the above two inequalities for $y=y^*_m$ with $1\\le m\\le k$ and taking the expectation together with the bounded variance assumption~\\eqref{eq:bounded-variance} gives\n  \\begin{equation}\n    \\label{eq:ab1}\n    \\begin{aligned}\n    \\MoveEqLeft \\Exp{\\Expcond{\\langle \\nabla_{y}\\Phi(x_{k+1},y_k;\\xi_k), y^*_m-y_k \\rangle}{\\mathcal{F}_k}}  + \\Exp{h(y_{k+1})+ \\frac{1}{2\\eta_y} \\Vert y^*_m-y_{k+1} \\Vert^2}\\\\\n    &\\le  \\Exp{\\langle \\nabla_{y}\\Phi(x_{k+1},y_k), y_{k+1}-y_k \\rangle - \\frac{1}{4\\eta_y} \\Vert y_{k+1} -y_k \\Vert^2}\n    \\\\\n    &\\quad+ \\eta_y \\sigma^2+ \\Exp{h(y^*_m) +\\frac{1}{2\\eta_y} \\Vert y^*_m-y_k \\Vert^2}.\n    \\end{aligned}\n  \\end{equation}\n  From the descent lemma (in ascent form) and the fact that $ \\eta_y \\le  \\nicefrac{1}{2L_{\\nabla\\Phi}}$ we have\n  \\begin{equation*}\n    % \\label{eq:stoch-ascent-lemma}\n    \\Phi(x_{k+1}, y_k) + \\langle y_{k+1} - y_k, \\nabla_y\\Phi(x_{k+1}, y_k)\\rangle - \\frac{1}{4\\eta_y} \\Vert y_{k+1} - y_k \\Vert^2 \\le \\Phi(x_{k+1}, y_{k+1}).\n  \\end{equation*}\n  We plug the above inequality into~\\eqref{eq:ab1}, make use of the concavity and add $f(x_{k+1})$ on both sides to deduce the statement of the lemma.\n\\end{proof}\n\n\\begin{lemma}%\n  \\label{lem:stoch-estimate-Delta}\n  For all $1 \\le m \\le k$, we have that\n  \\begin{equation}\n    \\label{eq:stoch-estimate-Delta}\n    \\begin{aligned}\n      \\MoveEqLeft \\hat{\\Delta}_k \\le 2\\eta_x L(L_f+L+\\sigma)(k-m) + \\frac{1}{2\\eta_y} \\Exp{\\Vert y_{k-1} - y^*_m \\Vert^2 - \\Vert y_k - y^*_m \\Vert^2} + \\eta_y\\sigma^2.\n    \\end{aligned}\n  \\end{equation}\n\\end{lemma}\n\n\\begin{proof}\n  Let the numbers $1 \\le m \\le k$ be fixed.\n  Starting from the definition of $\\hat{\\Delta}_k$, we add~\\eqref{eq:stoch-gradient-ascent-estimation} to obtain\n  % NOTE: Consider using $k\\ge1$ and get rid of the +1 everywhere\n  \\begin{equation}\n    \\label{eq:diff-function-values-stoch}\n    \\begin{aligned}\n      \\hat{\\Delta}_k &= \\Exp{\\psi(x_k) - \\Psi(x_k,y_k)} \\\\\n      &\\le \\Exp{\\psi(x_k) - \\Psi(x_k, y^*_m)} + \\frac{1}{2\\eta_y} \\Big(\\Exp{\\Vert y^*_m - y_{k-1} \\Vert^2 - \\Vert y^*_m - y_k \\Vert^2} \\Big)+ \\eta_y\\sigma^2.\n    \\end{aligned}\n  \\end{equation}\n  As in~\\eqref{eq:plus-minus-functionval} we deduce that\n  \\begin{equation*}\n    % \\label{eq:stoch-plus-minus}\n    \\Psi(x_k, y^*_k) - \\Psi(x_k, y^*_m) \\le \\Phi(x_k,y_k^*) -\\Phi(x_m,y_k^*) +\\Phi(x_m,y_m^*) -\\Phi(x_k,y_m^*).\n  \\end{equation*}\n  Together with the $L$-Lipschitz continuity of $\\Phi(\\cdot, y)$ and~\\eqref{eq:bound-prox-f} we estimate for $k\\ge m$ as in~\\eqref{eq:fun-val-same-y}, but using $\\Exp{\\Vert g_l^\\xi \\Vert}\\le L + \\sigma$,\n  \\begin{equation*}\n    \\begin{aligned}\n      \\Exp{\\Phi(x_k, y^*_k) - \\Phi(x_m, y^*_k)}\n      %&\\le L \\Exp{\\Vert x_k-x_m \\Vert} \\le L \\sum_{l=m}^{k-1}\\Exp{\\Vert x_{l+1} - x_l \\Vert} \\\\\n      %&\\le L \\sum_{l=m}^{k-1} \\Exp{\\Vert \\prox{\\eta_x f}{x_l -\\eta_x g_l^\\xi} - \\prox{\\eta_x f}{x_l} \\Vert+ \\Vert \\prox{\\eta_x f}{x_l} - x_l \\Vert} \\\\\n      &\\le \\eta_x L\\Big(L_f+\\sqrt{L^2+\\sigma^2}\\Big)(k-m).\n    \\end{aligned}\n  \\end{equation*}\n  % For $k=m$ the statement follows trivially.\n  % Analogously,\n  % \\begin{equation*}\n  %   \\Exp{\\Phi(x_m, y^*_m) - \\Phi(x_k, y^*_m)} \\le L \\Exp{\\Vert x_k - x_m \\Vert} \\le \\eta_x L\\Big(L_f+\\sqrt{L^2+\\sigma^2}\\Big)(k-m).\n  % \\end{equation*}\n  Plugging all of these into~\\eqref{eq:diff-function-values-stoch} gives the statement of the lemma.\n\\end{proof}\n\n\\begin{lemma}%\n  \\label{lem:stoch-estimate-sum-of-delta}\n  We have that for all $K\\ge1$\n  \\begin{equation}\n    \\label{eq:stoch-estimate-sum-of-delta}\n    \\frac{1}{K} \\sum_{k=0}^{K-1} \\hat{\\Delta}_k \\le \\eta_x L(L+L_f+\\sigma) B + \\frac{D_h^2}{2\\eta_y B} + \\eta_y\\sigma^2 + \\frac{\\Delta_0}{K}.\n  \\end{equation}\n\\end{lemma}\n\n\\begin{proof}\n  We proceed as in Lemma~\\ref{lem:estimate-sum-of-delta}. By using Lemma~\\ref{lem:stoch-estimate-Delta} we obtain\n  $j>0$ and $m=jB$ we have that\n  \\begin{equation}\n    \\label{eq:stoch--sum-over-any-block}\n    \\begin{aligned}\n      \\sum_{k=jB}^{(j+1)B-1} \\hat{\\Delta}_k %&\\le \\eta_x L(L+L_f+\\sigma)  B^2 + \\frac{1}{2\\eta_y} \\Exp{\\Vert y_{jB-1}-y^*_{jB} \\Vert^2} + B\\eta_y\\sigma^2  \\\\\n      &\\le \\eta_x L(L+L_f+\\sigma) B^2 + \\frac{1}{2\\eta_y}D_h^2 + B\\eta_y \\sigma^2.\n    \\end{aligned}\n  \\end{equation}\n  For $j=0$ we use $m=1$ and do not estimate $\\Delta_0$ but leave it there.\n  % with $m=1$ and the fact that $\\sum_{k=1}^{B-1} k \\le \\nicefrac{B^2}{2}$ we get that\n  % \\begin{equation}\n  %   \\label{eq:stoch--sum-first---block}\n  %   \\begin{aligned}\n  %     \\sum_{k=0}^{B-1} \\hat{\\Delta}_k &\\le \\Delta_0 + \\eta_x L(L+L_f+\\sigma)  B^2 + \\frac{1}{2\\eta_y} \\Exp{\\Vert y_{0}-y^*_{1} \\Vert^2} + B\\eta_y\\sigma^2  \\\\\n  %     &\\le \\Delta_0 + \\eta_x L(L+L_f+\\sigma) B^2 + \\frac{1}{2\\eta_y}D_h^2 + B\\eta_y \\sigma^2,\n  %   \\end{aligned}\n  % \\end{equation}\n  % where $D_h$ denotes the diameter of $\\dom h$, see Assumption~\\ref{ass:regularizers}.\n  % Analogously, for $j>0$ and $m=jB$ we have that\n  % \\begin{equation}\n  %   \\label{eq:stoch--sum-over-any-block}\n  %   \\begin{aligned}\n  %     \\sum_{k=jB}^{(j+1)B-1} \\hat{\\Delta}_k &\\le \\eta_x L(L+L_f+\\sigma)  B^2 + \\frac{1}{2\\eta_y} \\Exp{\\Vert y_{jB-1}-y^*_{jB} \\Vert^2} + B\\eta_y\\sigma^2  \\\\\n  %     &\\le \\eta_x L(L+L_f+\\sigma) B^2 + \\frac{1}{2\\eta_y}D_h^2 + B\\eta_y \\sigma^2,\n  %   \\end{aligned}\n  % \\end{equation}\n  Plugging~\\eqref{eq:stoch--sum-over-any-block} into~\\eqref{eq:split--summation-into-blocks} gives the statement of the lemma.\n\\end{proof}\n\n\\begin{proof}[Proof of Theorem~\\ref{thm:alternating-rate-stoch}]\n  We sum up the inequality of Lemma~\\ref{lem:stoch-descent-on-moreau} to deduce that\n  \\begin{equation*}\n    \\Exp{\\psi_\\lambda(x_{K})} \\le \\psi_\\lambda(x_0) + 2\\eta_x\\rho\\sum_{k=0}^{K-1}\\hat{\\Delta}_k - \\frac{\\eta_x}{2}\\sum_{k=0}^{K-1}\\Exp{\\Vert \\nabla \\psi_\\lambda(x_k) \\Vert^2} + 4K\\rho\\eta_x^2(L^2+\\sigma^2).\n  \\end{equation*}\n  Thus, by dividing by $K$ and $\\eta_x$ yields\n  \\begin{equation*}\n    \\frac{1}{K}\\sum_{k=0}^{K-1} \\Exp{\\Vert\\nabla \\psi_\\lambda(x_k)\\Vert^2}  \\le \\frac{2\\Delta^*}{\\eta_x K} + \\frac{4\\rho}{K}\\sum_{k=0}^{K-1}\\hat{\\Delta}_k + 8\\rho\\eta_x(L^2+\\sigma^2).\n  \\end{equation*}\n  Now we plug in~\\eqref{eq:stoch-estimate-sum-of-delta} to obtain\n  \\begin{equation*}\n    \\begin{aligned}\n      \\MoveEqLeft\\frac{1}{K}\\sum_{k=0}^{K-1} \\Exp{\\Vert\\nabla \\psi_\\lambda(x_k)\\Vert^2} \\\\\n      &\\le \\frac{2\\Delta^*}{\\eta_x K} + 4\\rho\\Big( \\eta_x L(L+L_f+\\sigma) B  + \\frac{D_h^2}{2\\eta_y B} + \\eta_y\\sigma^2 \\Big)  + 4\\rho\\frac{\\Delta_0}{K}+ 8\\rho\\eta_x (L^2+\\sigma^2).\n    \\end{aligned}\n  \\end{equation*}\n  With the block size $B=D_h \\sqrt{1/(\\eta_x\\eta_y L(L+L_f+\\sigma))}$ we have that\n  \\begin{equation*}\n    \\eta_x L(L+L_f+\\sigma) B  + \\frac{D_h^2}{2\\eta_y B} + \\eta_y\\sigma^2 = \\sqrt{\\frac{\\eta_x}{\\eta_y}}\\sqrt{L(L+L_f+\\sigma) }D_h + \\eta_y \\sigma^2\n  \\end{equation*}\n  % \\begin{equation*}\n  %   \\begin{aligned}\n  %     \\MoveEqLeft \\frac{1}{K}\\sum_{k=0}^{K-1} \\Exp{\\Vert \\nabla \\psi_\\lambda(x_k) \\Vert^2}\\\\\n  %     &\\le \\frac{2\\Delta^*}{\\eta_x K} + 4\\rho\\sqrt{\\frac{\\eta_x}{\\eta_y}}\\sqrt{L(L+L_f+\\sigma) }D_h + 4\\rho\\eta_y \\sigma^2 + 4\\rho\\frac{\\Delta_0}{K} + 8\\rho\\eta_x(L^2+\\sigma^2).\n  %   \\end{aligned}\n  % \\end{equation*}\n  Via the step size choice presented in the theorem we obtain the desired complexity.\n  % \\begin{equation}\n  % \\eta_x = \\min \\left\\{\\frac{\\epsilon^2}{\\rho(L^2+\\sigma^2)}, \\frac{\\epsilon^4}{\\rho^2 L(L+L_f+\\sigma)D_h^2 L_{\\nabla\\Phi}}, \\frac{\\epsilon^6}{\\rho^3L(L+L_f+\\sigma)\\sigma^2D_h^2 }\\right\\}\n  % \\end{equation}\n  % and $\\eta_y= \\min\\{\\frac{1}{2 L_{\\nabla\\Phi}}, \\frac{\\epsilon^2}{\\rho \\sigma^2}\\}$ we obtain\n  % \\begin{equation*}\n  %   \\begin{aligned}\n  %     \\mathcal{O}\\left(\\frac{\\Delta^*(L^2+L_f^2+\\sigma^2)\\rho}{\\epsilon^4}\\max\\Big\\{1, \\frac{\\rho L_{\\nabla\\Phi}LD_h^2}{\\epsilon^2}, \\frac{\\rho^2 D_h^2 \\sigma^2}{\\epsilon^4}\\Big\\}+ \\frac{\\Delta_0\\rho}{\\epsilon^2}\\right)\n  %   \\end{aligned}\n  % \\end{equation*}\n  % which finishes the proof.\n\\end{proof}","units":[{"start":27218,"end":27478,"start_line":359,"end_line":362,"raw_tex":"\\begin{lemma}[{see~\\cite[Theorem 3.52]{mordukhovich-variational-analysis}}]%\n  \\label{lem:lipschitz-bounded-subgradients}\n  For an $L_\\psi$-Lipschitz continuous function $\\psi:\\R^d \\to \\R$ every Fr\\'echet subgradient is bounded in norm by $L_\\psi$.\n\\end{lemma}","sha256_utf8":"209aa48a210ac4676123f3b549e1d8f8810d152ec39cb2121c7595cab92c7a8a"},{"start":27563,"end":28109,"start_line":366,"end_line":374,"raw_tex":"\\begin{lemma}[{see~\\cite[Lemma 2.2]{stoch_weakly_model_based}}]%\n  \\label{lem:grad_smooth_is_prox}\n  Let $\\psi: \\R^d \\to \\R\\cup\\{+\\infty\\}$ be a proper, $\\rho$-weakly convex, and l.s.c.\\ function, and let $\\lambda \\in (0,\\rho^{-1})$. Then the Moreau envelope $\\psi_\\lambda$ is continuously differentiable\n  on $\\R^d$ with gradient\n  \\begin{equation*}\n    \\nabla \\psi_\\lambda(x) = \\frac{1}{\\lambda}\\left(x - \\prox{\\lambda \\psi}{x}\\right) \\quad \\mbox{for all $x \\in \\R^d$},\n  \\end{equation*}\n  and this gradient is Lipschitz continuous.\n\\end{lemma}","sha256_utf8":"68a3ea4c840463d434eb8e78e5c756cfbc4a3926ff39e07e4e1f1813d6915823"},{"start":39240,"end":39717,"start_line":582,"end_line":590,"raw_tex":"\\begin{proposition}[Subgradient characterization of the max function]%\n  \\label{thm:extended-danskin}\n  Let Assumption~\\ref{ass:weakly-convex-gradient-lipschitz} and~\\ref{ass:regularizers} hold true.\n  Then, the function $\\phi$, see~\\eqref{eq:max-function}, fulfills\n  \\begin{equation*}\n    \\partial [\\Phi(\\cdot,y^*)](x) \\subseteq \\partial\\phi(x) \\quad \\forall y^*\\in Y(x), \\forall x\\in \\R^d.\n  \\end{equation*}\n  In particular, $\\phi$ is $\\rho$-weakly convex.\n\\end{proposition}","sha256_utf8":"81ffb8d587200258579469f8a02d57bedf3ed6ba8dc21e54e11cbe4cc8b7382d"},{"start":39718,"end":41650,"start_line":591,"end_line":616,"raw_tex":"\\begin{proof}\n  From the $\\rho$-weak convexity of $\\Phi(\\cdot,y)$, we have that $\\Phi(\\cdot,y) + \\frac{\\rho}{2} \\|\\cdot\\|^2$ is convex for all $y\\in\\R^n$. We define $\\tilde{\\Phi}(x,y) = \\Phi(x,y) + \\frac\\rho2 \\|x\\|^2$ and $\\tilde{\\Gamma}(x,y) = \\Gamma(x,y)+\\frac{\\rho}{2}\\Vert x \\Vert^2$ for $(x,y)\\in\\R^d\\times\\R^n$ as well as\n  \\begin{equation*}\n    \\tilde{\\phi}(x) = \\max_{y \\in \\R^n} \\tilde{\\Gamma}(x,y) = \\phi(x) + \\frac{\\rho}{2} \\|x\\|^2.\n  \\end{equation*}\n  Notice that $\\tilde{\\Gamma}(x,\\cdot)$ is concave for any $x \\in \\R^d$ and $\\tilde{\\Gamma}(\\cdot,y)$ is convex for any $y \\in \\R^n$.\n  Thus, the function $\\tilde{\\phi}$ is convex and $\\dom \\tilde{\\phi} = \\dom \\phi = \\R^d$. Therefore $\\phi$ is continuous, which implies that $\\partial \\phi(x) \\neq \\emptyset$ for any $x \\in \\R^d$.\n  Let $x \\in \\R^d$, $y \\in Y(x)$ and $v \\in \\R^d$. For any $\\alpha >0$ it holds\n  \\begin{equation*}\n    \\frac{\\tilde{\\phi}(x+\\alpha v) - \\tilde{\\phi}(x)}{\\alpha} \\geq \\frac{\\tilde{\\Gamma}(x+\\alpha v, y) - \\tilde{\\Gamma}(x,y)}{\\alpha} =  \\frac{\\tilde{\\Phi}(x+\\alpha v, y) - \\tilde{\\Phi}(x,y)}{\\alpha},\n  \\end{equation*}\n  thus\n  \\begin{equation}\n    \\begin{aligned}\n      \\tilde{\\phi}'(x;v) &= \\inf_{\\alpha >0} \\frac{\\tilde{\\phi}(x+\\alpha v) - \\tilde{\\phi}(x)}{\\alpha} \\geq \\inf_{\\alpha >0} \\frac{\\tilde{\\Phi}(x+\\alpha v, y) - \\tilde{\\Phi}(x,y)}{\\alpha} = [\\tilde{\\Phi}(\\cdot,y)]'(x;v),\n    \\end{aligned}\n  \\end{equation}\n  where $[\\Phi(\\cdot,y)]'(x;v)$ denotes the directional derivative of $\\Phi$ in the first component at $x$ in the direction $v$.\n  In conclusion,\n  \\begin{equation}\n    \\label{eq1}\n    \\tilde{\\phi}'(x;v) \\geq \\sup_{y \\in Y(x)}  [\\tilde{\\Phi}(\\cdot,y)]'(x;v) \\quad \\forall v \\in \\R^d\n  \\end{equation}\n  and for $y\\in Y(x)$ we therefore conclude $\\partial [\\tilde{\\Phi}(\\cdot,y)](x) \\subseteq \\partial\\tilde{\\phi}(x)$.\n  The first statement is obtained by subtracting $\\rho x$ on both sides of the inclusion.\n\\end{proof}","sha256_utf8":"499890d036e110b62080b55881945f929d1c69bb80119d48e62f3857e54f5c89"},{"start":41654,"end":41870,"start_line":620,"end_line":623,"raw_tex":"\\begin{lemma}[Lipschitz continuity of the max function]%\n  \\label{lem:max-is-lipschitz}\n  The Lipschitz continuity of $\\Phi$ in its first component implies that $\\phi$ is Lipschitz with the same constant.\n\\end{lemma}","sha256_utf8":"15b5c6dfe5920f32d591531ab4cbcff8775142ce7374fae53d3c27fee38d1cf8"},{"start":41871,"end":42270,"start_line":624,"end_line":633,"raw_tex":"\\begin{proof}\n  Let $x,x' \\in \\R^d$ and $y^*\\in Y(x)$. On the one hand\n  \\begin{equation*}\n    \\begin{aligned}\n      \\phi(x) - \\phi(x') &= \\Phi(x,y^*)-h(y^*) - \\phi(x')\\\\\n                        &\\le \\Phi(x,y^*)-h(y^*) - \\Phi(x',y^*)+h(y^*)\\le L \\Vert x-x' \\Vert.\n    \\end{aligned}\n  \\end{equation*}\n  The reverse direction $\\phi(x') - \\phi(x) \\le L\\Vert x-x' \\Vert$ follows analogously.\n\\end{proof}","sha256_utf8":"6a47b8443595fc564a9b5011c08c511085704a176e4b74056b584fb17c0f6360"},{"start":44534,"end":44894,"start_line":671,"end_line":677,"raw_tex":"\\begin{lemma}%\n  \\label{lem:xhatk-as-prox}\n  For any $\\lambda\\in(0,\\rho^{-1})$ and all $k\\ge0$ the point $\\hat{x}_k := \\prox{\\lambda g}{x_k}$ can also be written for some $v_k\\in \\partial \\phi(\\hat{x}_k)$ as\n  \\begin{equation*}\n    \\hat{x}_k = \\prox{\\eta_x f}{\\eta_x\\lambda^{-1}x_k - \\eta_x v_k + (1-\\eta_x\\lambda^{-1})\\hat{x}_k}.\n  \\end{equation*}\n\\end{lemma}","sha256_utf8":"8c317245875a37bc29aed88c101e758a012cb56a5baa4ebe0fba3cd42af5ee25"},{"start":44895,"end":46146,"start_line":678,"end_line":699,"raw_tex":"\\begin{proof}\n  Let $k\\ge0$ be arbitrary but fixed and recall that $g=f+\\phi$. By the definition of $\\hat{x}_k$ we have that\n  \\begin{equation*}\n      0 \\in \\partial \\psi(\\hat{x}_k) + \\frac1\\lambda(\\hat{x}_k-x_k) = \\partial(\\phi+f)(\\hat{x}_k) + \\frac1\\lambda(\\hat{x}_k-x_k).\n  \\end{equation*}\n  % TODO: reference\n  We can estimate through the continuity of $\\phi$ and subdifferential calculus\n  \\begin{equation*}\n    \\frac1\\lambda(x_k - \\hat{x}_k) \\in \\partial(\\phi+f)(\\hat{x}_k) \\subseteq \\partial \\phi(\\hat{x}_k) + \\partial f(\\hat{x}_k).\n  \\end{equation*}\n  Thus, there exists $v_k \\in \\partial \\phi(\\hat{x}_k)$ such that $\\frac1\\lambda (x_k - \\hat{x}_k) \\in v_k + \\partial f(\\hat{x}_k)$.\n  % \\begin{equation*}\n  %   \\frac1\\lambda (x_k - \\hat{x}_k) \\in v_k + \\partial f(\\hat{x}_k).\n  % \\end{equation*}\n  Also,\n  \\begin{equation*}\n    \\begin{aligned}\n      \\frac1\\lambda(x_k - \\hat{x}_k) \\in  \\partial f(\\hat{x}_k) +  v_k &\\Leftrightarrow \\frac{\\eta_x }{\\lambda}x_k - \\eta_x v_k + (1-\\frac{\\eta_x}{\\lambda})\\hat{x}_k \\in \\hat{x}_k + \\eta_x \\partial f(\\hat{x}_k) \\\\\n      &\\Leftrightarrow \\hat{x}_k = \\prox{\\eta_x f}{\\frac{\\eta_x}{\\lambda}x_k - \\eta_x v_k + \\big(1-\\frac{\\eta_x}{\\lambda}\\big)\\hat{x}_k}.\n    \\end{aligned}\n  \\end{equation*}\n\\end{proof}","sha256_utf8":"a04c7e7961fbafdaafa52af5358db9f4819e248d4fb6b849790e7024d89c07ff"},{"start":46263,"end":46648,"start_line":703,"end_line":710,"raw_tex":"\\begin{lemma}%\n  \\label{lem:descent-on-moreau}\n  With $\\lambda = \\nicefrac{1}{2\\rho}$ and $\\eta_x \\ge 0$ we have for all $k\\ge0$ that\n  \\begin{equation*}\n    \\psi_\\lambda(x_{k+1}) \\le \\psi_\\lambda(x_k) + 2\\rho\\eta_x\\Delta_k - \\frac12\\eta_x \\Vert \\nabla \\psi_\\lambda(x_k) \\Vert^2  + 4\\rho\\eta_x^2L^2,\n  \\end{equation*}\n  where $\\Delta_k := \\psi(x_k) - \\Psi(x_k, y_k) \\ge 0$.\n\\end{lemma}","sha256_utf8":"6f565ab9b81e50fa8ade12bb8fa7d424fafbfbd8bd5d9f05fb3dbcce7c3d6357"},{"start":46649,"end":50768,"start_line":711,"end_line":771,"raw_tex":"\\begin{proof}\n  Let $k\\ge0$ be fixed. As before we denote $\\hat{x}_k = \\prox{\\lambda \\psi}{x_k}$. From the definition of the Moreau envelope we have that\n  \\begin{equation}\n    \\label{eq:moreau-is-min}\n    \\psi_\\lambda(x_{k+1}) = \\min_{x\\in\\R^d} \\Big\\{ \\psi(x) + \\frac{1}{2\\lambda} \\Vert x - x_{k+1} \\Vert^2\\Big\\} \\le \\psi(\\hat{x}_k) + \\frac{1}{2\\lambda} \\Vert \\hat{x}_k - x_{k+1} \\Vert^2.\n  \\end{equation}\n  Let now $v_k \\in \\partial\\phi(\\hat{x}_k)$ as in Lemma~\\ref{lem:xhatk-as-prox}. We successively deduce for $\\beta:=1-\\eta_x \\lambda^{-1}$\n  \\begin{align}\n    \\Vert \\hat{x}_k - x_{k+1} \\Vert^2 &= \\Vert \\prox{\\eta_x f}{\\eta_x\\lambda^{-1}x_k - \\eta_x v_k + \\beta\\hat{x}_k} - \\prox{\\eta_x f}{x_k - \\eta_x g_k}\\Vert^2 \\label{eq:defin-iter}\\\\\n     &\\le \\Vert \\beta(\\hat{x}_k - x_k) + \\eta_x (g_k - v_k)\\Vert^2 \\label{eq:1-Lipschitz}\\\\\n     &= {\\beta}^2\\Vert \\hat{x}_k - x_k \\Vert^2 + 2 \\eta_x\\beta\\langle g_k-v_k, \\hat{x}_k - x_k\\rangle\n     + \\eta_x^2\\Vert g_k - v_k \\Vert^2 \\notag\\\\\n     &\\le {\\beta}^2\\Vert \\hat{x}_k - x_k \\Vert^2 + 2 \\eta_x \\beta\\langle g_k-v_k, \\hat{x}_k - x_k\\rangle + 4\\eta_x^2L^2 \\label{eq:Lipschitz}\n  \\end{align}\n  where~\\eqref{eq:defin-iter} uses Lemma~\\ref{lem:xhatk-as-prox} and the definition of $x_{k+1}$, inequality~\\eqref{eq:1-Lipschitz} holds because of the nonexpansiveness of the proximal operator, and~\\eqref{eq:Lipschitz} follows from the Lipschitz continuity of $\\Phi$ and $\\phi$ (see Lemma~\\ref{lem:max-is-lipschitz}) and the fact that Lipschitz continuity implies bounded subgradients.\n  We are left with estimating the inner product in the above inequality and we do so by splitting it into two:\n  first of all, from the weak convexity of $\\Phi$ in $x$ we have that\n  \\begin{equation*}\n    \\begin{aligned}\n      \\langle g_k, \\hat{x}_k - x_k \\rangle &\\le \\Phi(\\hat{x}_k, y_k) - \\Phi(x_k, y_k) + \\frac{\\rho}{2} \\Vert \\hat{x}_k - x_k \\Vert^2 \\\\\n      % &= \\Phi(\\hat{x}_k, y_k)-h(y_k) - (\\Phi(x_k, y_k)-h(y_k)) + \\frac{\\rho}{2} \\Vert \\hat{x}_k - x_k \\Vert^2\\\\\n      &\\le \\phi(\\hat{x}_k) - \\Gamma(x_k,y_k) + \\frac{\\rho}{2} \\Vert \\hat{x}_k - x_k \\Vert^2.\n    \\end{aligned}\n  \\end{equation*}\n  Secondly, by the $\\rho$-weak convexity of $\\phi$\n  \\begin{equation*}\n    -\\langle v_k, \\hat{x}_k - x_k\\rangle \\le \\phi(x_k) - \\phi(\\hat{x}_k) + \\frac{\\rho}{2} \\Vert \\hat{x}_k - x_k \\Vert^2.\n  \\end{equation*}\n  Combining the last two inequalities we get that\n  \\begin{equation}\n    \\label{eq:estimate-middle}\n    \\langle g_k-v_k, \\hat{x}_k - x_k\\rangle \\le \\phi(x_k) - \\Gamma(x_k,y_k) + \\rho \\Vert \\hat{x}_k - x_k \\Vert^2.\n  \\end{equation}\n  Plugging~\\eqref{eq:estimate-middle} into~\\eqref{eq:Lipschitz} we deduce\n  \\begin{equation}\n    \\begin{aligned}\n    \\label{eq:ab3}\n    \\Vert \\hat{x}_k - x_{k+1} \\Vert^2 \\le \\underbrace{[{(1-\\eta_x\\lambda^{-1})}^2+2\\eta_x(1-\\eta_x \\lambda^{-1})\\rho]}_{= (*)} \\Vert \\hat{x}_k - x_k \\Vert^2 + 2\\eta_x \\Delta_k + 4\\eta_x^2L^2, \\\\\n    % &\\le \\Vert \\hat{x}_k - x_k \\Vert^2 + 2\\eta_x\\Delta_k -2\\eta_x\\rho \\Vert\\hat{x}_k - x_k \\Vert^2  + 2\\eta_x^2 L^2\n    \\end{aligned}\n  \\end{equation}\n  where we used the fact that $1-\\beta\\le1$ in the factor of $\\Delta_k$.\n  Now note that\n  \\begin{equation}\n    \\label{eq:terrible-algebra}\n    \\begin{aligned}\n      (*) &= 1 - 2\\eta_x \\lambda^{-1} + \\eta_x^2 \\lambda^{-2} + 2\\eta_x\\rho - 2\\eta_x^2 \\lambda^{-1}\\rho\\\\\n      &= 1 - 4\\eta_x\\rho + 4 \\eta_x^2\\rho^2 + 2\\eta_x\\rho - 4 \\eta_x^2\\rho^2 = 1-2\\eta_x\\rho.\n    \\end{aligned}\n  \\end{equation}\n  Combining~\\eqref{eq:moreau-is-min},~\\eqref{eq:ab3} and~\\eqref{eq:terrible-algebra} we deduce, using $\\lambda=\\nicefrac{1}{2\\rho}$,\n  % a lot of algebra happens here but it's correct\n  \\begin{equation*}\n    \\label{eq:descent-moreau}\n    \\begin{aligned}\n      \\psi_\\lambda(x_{k+1}) &\\le \\psi(\\hat{x}_k) + \\frac{1}{2\\lambda} \\left(\\Vert \\hat{x}_k - x_k \\Vert^2 + 2\\eta_x\\Delta_k -2\\eta_x\\rho \\Vert\\hat{x}_k - x_k \\Vert^2  + 4\\eta_x^2L^2\\right) \\\\\n      &= \\psi_\\lambda(x_k) + 2\\rho\\eta_x\\Delta_k - \\frac12\\eta_x \\Vert \\nabla \\psi_\\lambda(x_k) \\Vert^2  + 4\\rho\\eta_x^2L^2.\n    \\end{aligned}\n  \\end{equation*}\n  % where we used that $\\lambda=\\nicefrac{1}{2\\rho}$.\n\\end{proof}","sha256_utf8":"f23397374bcfd70e2dcf743bb026106c0ad6be36f45006e12a5ec085c25163ac"},{"start":51075,"end":51407,"start_line":774,"end_line":781,"raw_tex":"\\begin{lemma}%\n  \\label{lem:standard-gradient-ascent-estimation}\n  It holds for all $y\\in\\R^n$ and $k\\ge0$ that\n  \\begin{equation}\n    \\label{eq:gradient-ascent-estimation}\n    \\Psi(x_{k+1}, y) - \\Psi(x_{k+1}, y_{k+1}) \\le  \\frac{1}{2\\eta_y} \\Big(\\Vert y - y_k \\Vert^2 - \\Vert y - y_{k+1} \\Vert^2 \\Big).\n  \\end{equation}\n\\end{lemma}","sha256_utf8":"159d0eb207a11fbfcf162eba06dd636936560b5e393d099a427743d3634e5999"},{"start":51408,"end":53354,"start_line":782,"end_line":811,"raw_tex":"\\begin{proof}\n  This is a standard estimate on the improvement made by a single prox-gradient step for a convex (in this case concave) function, see for example~\\cite[Lemma 2.3]{fista}.\n  % By the definition of $y_{k+1}$ we have that\n  % \\begin{equation*}\n  %   y_{k+1} = \\argmin_{y\\in\\R^n} \\Big\\{ h(y) - \\langle\\nabla_y\\Phi(x_{k+1},y_k), y-y_k\\rangle + \\frac{1}{2\\eta_y}\\Vert y-y_k \\Vert^2 \\Big\\}.\n  % \\end{equation*}\n  % Let now $y\\in\\R^n$ be arbitrary but fixed. Since $y_{k+1}$ minimizes a $\\nicefrac{1}{\\eta_y}$-strongly convex function,\n  % \\begin{equation}\n  %   \\label{eq:decrease}\n  %   \\begin{aligned}\n  %     \\MoveEqLeft h(y_{k+1}) - \\langle \\nabla_y\\Phi(x_{k+1}, y_k), y_{k+1}-y_k \\rangle + \\frac{1}{2\\eta_y} \\Vert y_{k+1} -y_k \\Vert^2 + \\frac{1}{2\\eta_y} \\Vert y-y_{k+1} \\Vert^2  \\\\\n  %     &\\le h(y) - \\langle \\nabla_y\\Phi(x_{k+1}, y_k), y-y_k \\rangle + \\frac{1}{2\\eta_y} \\Vert y-y_k \\Vert^2.\n  %   \\end{aligned}\n  % \\end{equation}\n  % From the descent lemma (in ascent form) and the fact that $\\nicefrac{1}{\\eta_y} = L_{\\nabla\\Phi}$ we have that\n  % \\begin{equation}\n  %   \\label{eq:ascent-lemma}\n  %   \\Phi(x_{k+1}, y_k) + \\langle \\nabla_y\\Phi(x_{k+1}, y_k),  y_{k+1} - y_k\\rangle - \\frac{1}{2\\eta_y} \\Vert y_{k+1} - y_k \\Vert^2 \\le \\Phi(x_{k+1}, y_{k+1}).\n  % \\end{equation}\n  % From the concavity of $\\Phi$ in its second component we get that\n  % \\begin{equation}\n  %   \\label{eq:concavity}\n  %   \\Phi(x_{k+1}, y) \\le \\Phi(x_{k+1}, y_k) + \\langle \\nabla_y\\Phi(x_{k+1}, y_k), y-y_k \\rangle.\n  % \\end{equation}\n  % By plugging~\\eqref{eq:concavity} and~\\eqref{eq:ascent-lemma} into~\\eqref{eq:decrease} we deduce\n  % \\begin{equation*}\n  %   \\Phi(x_{k+1},y) -h(y) + \\frac{1}{2\\eta_y}\\Vert y-y_{k+1} \\Vert^2 \\le \\Phi(x_{k+1},y_{k+1})-h(y_{k+1}) + \\frac{1}{2\\eta_y}\\Vert y-y_k \\Vert^2.\n  % \\end{equation*}\n  % The statement of the lemma is obtained by rearranging the above inequality and adding $f(x_{k+1})$ on both sides.\n\\end{proof}","sha256_utf8":"68db736d96dc5e89ea6282415721f764c4e156924b36638233c9d0fc50892878"},{"start":53497,"end":53829,"start_line":814,"end_line":823,"raw_tex":"\\begin{lemma}%\n  \\label{lem:estimate-Delta}\n  We have that for all $1 \\le m \\le k$,\n  \\begin{equation}\n    \\begin{aligned}\n      \\label{eq:estimate-Delta}\n      \\Delta_k \\le 2\\eta_x L(L+L_f)(k-m) + \\frac{1}{2\\eta_y} \\Big(\\Vert y_{k-1} - y^*_m \\Vert^2 - \\Vert y_k - y^*_m \\Vert^2 \\Big).\n    \\end{aligned}\n  \\end{equation}\n\\end{lemma}","sha256_utf8":"c7d83f1816404d0251d07fa87db157f84c0837f6c73ee7ff6e36aebc6a3c90fe"},{"start":53830,"end":56312,"start_line":824,"end_line":866,"raw_tex":"\\begin{proof}\n  % Let $1\\le m \\le k$ be fixed.\n  Plugging $y=y^*_m$ into~\\eqref{eq:gradient-ascent-estimation} we deduce that\n  \\begin{equation}\n    \\label{eq:standard_descent_with_ystar}\n    \\begin{aligned}\n       0 \\le \\Psi(x_k, y_k) - \\Psi(x_k, y^*_m) + \\frac{1}{2\\eta_y} \\Big(\\Vert y^*_m - y_{k-1} \\Vert^2 - \\Vert y^*_m - y_k \\Vert^2 \\Big).\n    \\end{aligned}\n  \\end{equation}\n  Starting from the definition of $\\Delta_k = \\Psi(x_k,y^*_k) - \\Psi(x_k,y_k)$, we add~\\eqref{eq:standard_descent_with_ystar} to obtain\n  \\begin{equation}\n    \\label{eq:diff-function-values}\n    \\begin{aligned}\n      \\Delta_k %&= \\Psi(x_k,y^*_k) - \\Psi(x_k,y_k) \\\\\n      &\\le \\Psi(x_k,y^*_k) - \\Psi(x_k,y_m^*) + \\frac{1}{2\\eta_y} \\Big(\\Vert y^*_m - y_{k-1} \\Vert^2 - \\Vert y^*_m - y_k \\Vert^2 \\Big).\n    \\end{aligned}\n  \\end{equation}\n  Due to the Lipschitz continuity of $\\Phi$, terms which only differ in their first argument will be easy to estimate. Therefore, we insert and subtract $\\Phi(x_m,y_{k}^*)$ to deduce\n  \\begin{equation}\n    \\label{eq:plus-minus-functionval}\n    \\begin{aligned}\n      \\MoveEqLeft \\Psi(x_k, y^*_k) - \\Psi(x_k, y^*_m)  \\\\ %&= \\Phi(x_k, y^*_k)-h(y^*_k) - \\Phi(x_k, y^*_m)+h(y^*_m) \\\\\n      &= \\Phi(x_k, y^*_k) - \\Phi(x_m, y^*_k) + \\Phi(x_m, y^*_k)-h(y^*_k) - \\Phi(x_k, y^*_m) + h(y^*_m) \\\\\n      &\\le \\Phi(x_k, y^*_k) - \\Phi(x_m, y^*_k) + \\Phi(x_m, y^*_m)-h(y^*_m) - \\Phi(x_k, y^*_m) + h(y^*_m)\\\\\n      &= \\Phi(x_k,y^*_k) - \\Phi(x_m,y^*_k) + \\Phi(x_m,y^*_m) - \\Phi(x_k, y^*_m).\n    \\end{aligned}\n  \\end{equation}\n  We estimate the above expression for $k>m$ by making use of the Lipschitz continuity of $\\Phi(\\cdot, y)$ and~\\eqref{eq:bound-prox-f} deducing\n  \\begin{equation}\n    \\label{eq:fun-val-same-y}\n    \\begin{aligned}\n      \\MoveEqLeft \\Phi(x_k, y^*_k) - \\Phi(x_m, y^*_k)  \\le L \\Vert x_k - x_m \\Vert \\le L \\sum_{l=m}^{k-1} \\Vert x_{l+1}-x_l \\Vert \\\\\n      &\\le L \\sum_{l=m}^{k-1} \\Big( \\Vert \\prox{\\eta_x f}{x_l -\\eta_x g_l} - \\prox{\\eta_x f}{x_l} \\Vert + \\Vert \\prox{\\eta_x f}{x_l} - x_l \\Vert \\Big)\\\\\n      & \\le \\eta_x L(L+L_f)(k-m).\n    \\end{aligned}\n  \\end{equation}\n  For $k=m$ the inequality follows trivially. Analogously, we deduce\n  \\begin{equation}\n    \\label{eq:fun-val-same-y-2}\n      \\Phi(x_m, y^*_m) - \\Phi(x_k, y^*_m) \\le \\eta_x L(L+L_f)(k-m).\n  \\end{equation}\n  Plugging~\\eqref{eq:plus-minus-functionval},~\\eqref{eq:fun-val-same-y} and~\\eqref{eq:fun-val-same-y-2} into~\\eqref{eq:diff-function-values} gives the statement of the lemma.\n\\end{proof}","sha256_utf8":"b7c28f64215e74cd374d83a1daa8f7360ed135c3ff6e5d413715c3406fd0e384"},{"start":56587,"end":56873,"start_line":870,"end_line":877,"raw_tex":"\\begin{lemma}%\n  \\label{lem:estimate-sum-of-delta}\n  It holds that for all $K\\ge1$\n  \\begin{equation}\n    \\label{eq:estimate-sum-of-delta}\n    \\frac{1}{K} \\sum_{k=0}^{K-1} \\Delta_k \\le \\eta_x L(L+L_f) B + \\frac{L_{\\nabla\\Phi}D_h^2}{2B} + \\frac{\\Delta_0}{K}.\n  \\end{equation}\n\\end{lemma}","sha256_utf8":"1283100176363cbabe622b34f8906131e37a562f9b1ed160531520e714620fa9"},{"start":56874,"end":59158,"start_line":878,"end_line":917,"raw_tex":"\\begin{proof}\n  By splitting the summation into blocks we get that\n  \\begin{equation}\n    \\label{eq:split--summation-into-blocks}\n    \\sum_{k=0}^{K-1} \\Delta_k  = \\sum_{j=0}^{K/B-1} \\ \\sum_{k=jB}^{(j+1)B-1} \\Delta_k.\n  \\end{equation}\n  By using~\\eqref{eq:estimate-Delta} from Lemma~\\ref{lem:estimate-Delta} with $j>0$ and $m=jB$ and the fact that $\\sum_{k=1}^{B-1} k \\le \\nicefrac{B^2}{2}$ we have\n  \\begin{equation}\n    \\label{eq:sum-over-any-block}\n    \\begin{aligned}\n      \\sum_{k=jB}^{(j+1)B-1} \\Delta_k &\\le \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y} \\Vert y_{jB-1}-y^*_{jB} \\Vert^2 \\\\\n      % &\\le \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y}D_h^2,\n    \\end{aligned}\n  \\end{equation}\n  where the last term can be bounded by $D_h^2$, which was defined in Assumption~\\ref{ass:regularizers} and denotes the diameter of $\\dom h$.\n  We do the same for the case $j=0$ but choose here $m=1$ and have separate out the first summand of $\\sum_{k=0}^{B-1} \\Delta_k$ as Lemma~\\ref{lem:estimate-Delta} does not hold for $k=0$ and therefore get an extra $\\Delta_0$ summand.\n  % with $m=1$ and the fact that $\\sum_{k=1}^{B-1} k \\le \\nicefrac{B^2}{2}$ we get that\n  % \\begin{equation}\n  %   \\label{eq:sum-first-block}\n  %   \\begin{aligned}\n  %     \\sum_{k=0}^{B-1} \\Delta_k &\\le \\Delta_0 + \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y} \\Vert y_{0}-y^*_{1} \\Vert^2 \\\\\n  %     &\\le \\Delta_0 + \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y}D_h^2,\n  %   \\end{aligned}\n  % \\end{equation}\n  where $D_h$ was defined in Assumption~\\ref{ass:regularizers} and denotes the diameter of $\\dom h$.\n  % Analogously, for $j>0$ and $m=jB$ we have that\n  % \\begin{equation}\n  %   \\label{eq:sum-over-any-block}\n  %   \\begin{aligned}\n  %     \\sum_{k=jB}^{(j+1)B-1} \\Delta_k &\\le \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y} \\Vert y_{jB-1}-y^*_{jB} \\Vert^2 \\\\\n  %     &\\le \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y}D_h^2.\n  %   \\end{aligned}\n  % \\end{equation}\n  Plugging~\\eqref{eq:sum-over-any-block} into~\\eqref{eq:split--summation-into-blocks} gives\n  \\begin{equation*}\n    % \\label{eq:sum-over-all}\n    \\frac{1}{K} \\sum_{k=0}^{K-1} \\Delta_k \\le \\eta_x L(L+L_f) B + \\frac{1}{2\\eta_y B}D_h^2 + \\frac{\\Delta_0}{K}.\n  \\end{equation*}\n  The desired statement is obtained by using the step size $\\eta_y=\\nicefrac{1}{L_{\\nabla\\Phi}}$.\n\\end{proof}","sha256_utf8":"aad8a35b02681a34c6f2667802da0e177567d9ebf20c6238d7e7e47c8ac1fad2"},{"start":62924,"end":63337,"start_line":975,"end_line":982,"raw_tex":"\\begin{lemma}%\n  \\label{lem:stoch-descent-on-moreau}\n  With $\\lambda = \\nicefrac{1}{2\\rho}$ we have for all $k\\ge0$ that\n  \\begin{equation*}\n    \\Exp{\\psi_\\lambda(x_{k+1})} \\le \\Exp{\\psi_\\lambda(x_k)} + 2\\rho\\eta_x\\hat{\\Delta}_k - \\frac{\\eta_x}{2}\\Exp{\\Vert \\nabla \\psi_\\lambda(x_k) \\Vert^2} + 4\\rho\\eta_x^2(L^2+\\sigma^2)\n  \\end{equation*}\n  where $\\hat{\\Delta}_k := \\Exp{\\psi(x_k) - \\Psi(x_k, y_k)}$.\n\\end{lemma}","sha256_utf8":"09766523c8f38e3bbfdc0a1b5d8183d0bf728baacd1035d9d07a7516aad89c9a"},{"start":63338,"end":65170,"start_line":983,"end_line":1010,"raw_tex":"\\begin{proof}\n  Let $k\\ge0$ be arbitrary but fixed. It follows easily from~\\eqref{eq:bounded-variance-subgradient} that\n  \\begin{equation}\n    \\label{eq:steiner}\n    \\Exp{\\Vert g_k^\\xi \\Vert^2} \\le \\Exp{\\Vert g_k \\Vert^2} + \\sigma^2 \\le L^2 + \\sigma^2,\n  \\end{equation}\n  where $\\Exp{g_k^\\xi}=g_k \\in \\partial_x\\Phi(x_k,y_k)$.\n  The definition of the Moreau envelope yields\n  \\begin{equation}\n    \\label{eq:defin-moreau}\n    \\Exp{g_\\lambda(x_{k+1})} \\le \\Exp{\\psi(\\hat{x}_k)} + \\frac{1}{2\\lambda}\\Exp{\\Vert \\hat{x}_k - x_{k+1} \\Vert^2}.\n  \\end{equation}\n  Similarly to Lemma~\\ref{lem:descent-on-moreau} we deduce that for $v_k\\in \\partial\\phi(\\hat{x}_k)$ (as given in Lemma~\\ref{lem:xhatk-as-prox}) and $\\beta=1-\\eta_x \\lambda^{-1}$\n  \\begin{equation*}\n    \\begin{aligned}\n      \\Vert\\hat{x}_k - x_{k+1} \\Vert^2 %&= \\Vert \\prox{\\eta_x f}{\\eta_x\\lambda^{-1}x_k - \\eta_x v_k + \\beta\\hat{x}_k} - \\prox{\\eta_x f}{x_k - \\eta_x g_k^\\xi}\\Vert^2 \\\\\n       %&\\le \\Vert \\beta(\\hat{x}_k - x_k) + \\eta_x (g_k^\\xi - v_k)\\Vert^2 \\\\\n       &= {\\beta}^2\\Vert \\hat{x}_k - x_k \\Vert^2 + 2 \\eta_x \\beta\\langle g_k^\\xi - v_k, \\hat{x}_k - x_k\\rangle\n       + \\eta_x^2\\Vert g_k^\\xi - v_k \\Vert^2.\n    \\end{aligned}\n  \\end{equation*}\n  By applying the conditional expectation $\\Expcond{\\cdot}{x_k,y_k}$, then the unconditional one and using~\\eqref{eq:steiner}, we get that\n  \\begin{equation*}\n    \\Exp{\\Vert \\hat{x}_k - x_{k+1} \\Vert^2} \\le \\Exp{\\Vert \\hat{x}_k - x_k \\Vert^2 + 2 \\eta_x(1-\\frac{\\eta_x}{\\lambda}) \\langle g_k-v_k, \\hat{x}_k - x_k\\rangle} + 4\\eta_x^2(L^2+\\sigma^2).\n  \\end{equation*}\n  where $g_k= \\Exp{g_k^\\xi}$.\n  Lastly, we combine the above inequality with~\\eqref{eq:defin-moreau} and the estimate for the inner product~\\eqref{eq:estimate-middle} as in Lemma~\\ref{lem:descent-on-moreau} to deduce the statement of the lemma.\n\\end{proof}","sha256_utf8":"3e3abb9fff0466cf62c050d4cebb082665533fa9830ff930cfee73e65150e286"},{"start":65424,"end":65904,"start_line":1014,"end_line":1023,"raw_tex":"\\begin{lemma}%\n  \\label{lem:stoch-standard-gradient-ascent-estimation}\n  With $\\eta_y\\le \\nicefrac{1}{2L_{\\nabla\\Phi}}$ we have for all $k\\ge0$ and all $y \\in \\R^n$\n  \\begin{equation}\n    \\label{eq:stoch-gradient-ascent-estimation}\n    \\begin{aligned}\n      \\MoveEqLeft \\Exp{\\Psi(x_{k+1}, y) - \\Psi(x_{k+1}, y_{k+1})} \\le  \\frac{1}{2\\eta_y} \\Big(\\Exp{\\Vert y^*_m - y_k \\Vert^2 - \\Vert y^*_m - y_{k+1} \\Vert^2} \\Big) + \\eta_y\\sigma^2.\n    \\end{aligned}\n  \\end{equation}\n\\end{lemma}","sha256_utf8":"87bf85169b684b0a1f3d7efcef77295bf932097c586c99c6408b87abc574bc28"},{"start":65905,"end":69020,"start_line":1024,"end_line":1065,"raw_tex":"\\begin{proof}\n  Let $k\\ge0$ and $y\\in\\R^n$ be arbitrary but fixed.\n  % By the definition of $y_{k+1}$ we have that\n  % \\begin{equation*} y_{k+1} = \\argmin_{y\\in\\R^n} \\Big\\{ h(y) - \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k), y-y_k\\rangle + \\frac{1}{2\\eta_y}\\Vert y-y_k \\Vert^2 \\Big\\}.\n  % \\end{equation*}\n  As in Lemma~\\ref{lem:standard-gradient-ascent-estimation}, we deduce\n  \\begin{equation*}\n    % \\label{eq:stoch-decrease}\n    \\begin{aligned}\n      h(y_{k+1}) - \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k), y_{k+1}-y_k \\rangle + \\frac{1}{2\\eta_y} \\Big(\\Vert y_{k+1} -y_k \\Vert^2 + \\Vert y-y_{k+1} \\Vert^2 \\Big)  \\\\\n      \\le h(y) - \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k), y-y_k \\rangle + \\frac{1}{2\\eta_y} \\Vert y-y_k \\Vert^2.\n    \\end{aligned}\n  \\end{equation*}\n  The term $\\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k), y_{k+1}-y_k \\rangle$ is problematic, because the right~hand side of the inner product is not measurable with respect to the sigma algebra generated by past iterates $\\mathcal{F}_k := \\sigma\\{x_{k+1},\\dots, x_1, y_k, \\dots, y_1\\}$, so we insert and subtract $\\langle \\nabla_y\\Phi(x_{k+1},y_k) , y_{k+1}-y_k\\rangle$\n  % to deduce\n  % \\begin{multline*}\n  %     h(y_{k+1}) + \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k),y-y_k\\rangle + \\frac{1}{2\\eta_y} \\Vert y_{k+1} -y_k \\Vert^2 + \\frac{1}{2\\eta_y} \\Vert y-y_{k+1} \\Vert^2 \\\\\n  %     \\le h(y) + \\langle \\nabla_y\\Phi(x_{k+1},y_k), y_{k+1}-y_k \\rangle  + \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k) - \\nabla_y\\Phi(x_{k+1},y_k), y_{k+1}-y_k \\rangle + \\frac{1}{2\\eta_y} \\Vert y-y_k \\Vert^2.\n  % \\end{multline*}\n  Now, using Young's inequality we estimate the resulting inner product\n  \\begin{multline*}\n    % \\label{eq:young2}\n    \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k) - \\nabla_y\\Phi(x_{k+1},y_k), y_{k+1}-y_k \\rangle \\\\\n    \\le \\eta_y\\Vert \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k) - \\nabla_{y}\\Phi(x_{k+1},y_k) \\Vert^2 + \\frac{1}{4\\eta_y} \\Vert y_{k+1} - y_k \\Vert^2.\n  \\end{multline*}\n  Combining the above two inequalities for $y=y^*_m$ with $1\\le m\\le k$ and taking the expectation together with the bounded variance assumption~\\eqref{eq:bounded-variance} gives\n  \\begin{equation}\n    \\label{eq:ab1}\n    \\begin{aligned}\n    \\MoveEqLeft \\Exp{\\Expcond{\\langle \\nabla_{y}\\Phi(x_{k+1},y_k;\\xi_k), y^*_m-y_k \\rangle}{\\mathcal{F}_k}}  + \\Exp{h(y_{k+1})+ \\frac{1}{2\\eta_y} \\Vert y^*_m-y_{k+1} \\Vert^2}\\\\\n    &\\le  \\Exp{\\langle \\nabla_{y}\\Phi(x_{k+1},y_k), y_{k+1}-y_k \\rangle - \\frac{1}{4\\eta_y} \\Vert y_{k+1} -y_k \\Vert^2}\n    \\\\\n    &\\quad+ \\eta_y \\sigma^2+ \\Exp{h(y^*_m) +\\frac{1}{2\\eta_y} \\Vert y^*_m-y_k \\Vert^2}.\n    \\end{aligned}\n  \\end{equation}\n  From the descent lemma (in ascent form) and the fact that $ \\eta_y \\le  \\nicefrac{1}{2L_{\\nabla\\Phi}}$ we have\n  \\begin{equation*}\n    % \\label{eq:stoch-ascent-lemma}\n    \\Phi(x_{k+1}, y_k) + \\langle y_{k+1} - y_k, \\nabla_y\\Phi(x_{k+1}, y_k)\\rangle - \\frac{1}{4\\eta_y} \\Vert y_{k+1} - y_k \\Vert^2 \\le \\Phi(x_{k+1}, y_{k+1}).\n  \\end{equation*}\n  We plug the above inequality into~\\eqref{eq:ab1}, make use of the concavity and add $f(x_{k+1})$ on both sides to deduce the statement of the lemma.\n\\end{proof}","sha256_utf8":"4888d0adb8d9b7162f6c4fef9524cbb3e8e61a5c05f7f9e62024ad40d8151103"},{"start":69087,"end":69466,"start_line":1069,"end_line":1078,"raw_tex":"\\begin{lemma}%\n  \\label{lem:stoch-estimate-Delta}\n  For all $1 \\le m \\le k$, we have that\n  \\begin{equation}\n    \\label{eq:stoch-estimate-Delta}\n    \\begin{aligned}\n      \\MoveEqLeft \\hat{\\Delta}_k \\le 2\\eta_x L(L_f+L+\\sigma)(k-m) + \\frac{1}{2\\eta_y} \\Exp{\\Vert y_{k-1} - y^*_m \\Vert^2 - \\Vert y_k - y^*_m \\Vert^2} + \\eta_y\\sigma^2.\n    \\end{aligned}\n  \\end{equation}\n\\end{lemma}","sha256_utf8":"58a8211a03221d7e6dd03ced7a8da90c804c1ec83569aa1df798bd83f1c2e42c"},{"start":69467,"end":71266,"start_line":1079,"end_line":1110,"raw_tex":"\\begin{proof}\n  Let the numbers $1 \\le m \\le k$ be fixed.\n  Starting from the definition of $\\hat{\\Delta}_k$, we add~\\eqref{eq:stoch-gradient-ascent-estimation} to obtain\n  % NOTE: Consider using $k\\ge1$ and get rid of the +1 everywhere\n  \\begin{equation}\n    \\label{eq:diff-function-values-stoch}\n    \\begin{aligned}\n      \\hat{\\Delta}_k &= \\Exp{\\psi(x_k) - \\Psi(x_k,y_k)} \\\\\n      &\\le \\Exp{\\psi(x_k) - \\Psi(x_k, y^*_m)} + \\frac{1}{2\\eta_y} \\Big(\\Exp{\\Vert y^*_m - y_{k-1} \\Vert^2 - \\Vert y^*_m - y_k \\Vert^2} \\Big)+ \\eta_y\\sigma^2.\n    \\end{aligned}\n  \\end{equation}\n  As in~\\eqref{eq:plus-minus-functionval} we deduce that\n  \\begin{equation*}\n    % \\label{eq:stoch-plus-minus}\n    \\Psi(x_k, y^*_k) - \\Psi(x_k, y^*_m) \\le \\Phi(x_k,y_k^*) -\\Phi(x_m,y_k^*) +\\Phi(x_m,y_m^*) -\\Phi(x_k,y_m^*).\n  \\end{equation*}\n  Together with the $L$-Lipschitz continuity of $\\Phi(\\cdot, y)$ and~\\eqref{eq:bound-prox-f} we estimate for $k\\ge m$ as in~\\eqref{eq:fun-val-same-y}, but using $\\Exp{\\Vert g_l^\\xi \\Vert}\\le L + \\sigma$,\n  \\begin{equation*}\n    \\begin{aligned}\n      \\Exp{\\Phi(x_k, y^*_k) - \\Phi(x_m, y^*_k)}\n      %&\\le L \\Exp{\\Vert x_k-x_m \\Vert} \\le L \\sum_{l=m}^{k-1}\\Exp{\\Vert x_{l+1} - x_l \\Vert} \\\\\n      %&\\le L \\sum_{l=m}^{k-1} \\Exp{\\Vert \\prox{\\eta_x f}{x_l -\\eta_x g_l^\\xi} - \\prox{\\eta_x f}{x_l} \\Vert+ \\Vert \\prox{\\eta_x f}{x_l} - x_l \\Vert} \\\\\n      &\\le \\eta_x L\\Big(L_f+\\sqrt{L^2+\\sigma^2}\\Big)(k-m).\n    \\end{aligned}\n  \\end{equation*}\n  % For $k=m$ the statement follows trivially.\n  % Analogously,\n  % \\begin{equation*}\n  %   \\Exp{\\Phi(x_m, y^*_m) - \\Phi(x_k, y^*_m)} \\le L \\Exp{\\Vert x_k - x_m \\Vert} \\le \\eta_x L\\Big(L_f+\\sqrt{L^2+\\sigma^2}\\Big)(k-m).\n  % \\end{equation*}\n  Plugging all of these into~\\eqref{eq:diff-function-values-stoch} gives the statement of the lemma.\n\\end{proof}","sha256_utf8":"721cd4fd078a6dc2193cd6431c9e659108437237089e0c164c95771c05244e04"},{"start":71493,"end":71813,"start_line":1114,"end_line":1121,"raw_tex":"\\begin{lemma}%\n  \\label{lem:stoch-estimate-sum-of-delta}\n  We have that for all $K\\ge1$\n  \\begin{equation}\n    \\label{eq:stoch-estimate-sum-of-delta}\n    \\frac{1}{K} \\sum_{k=0}^{K-1} \\hat{\\Delta}_k \\le \\eta_x L(L+L_f+\\sigma) B + \\frac{D_h^2}{2\\eta_y B} + \\eta_y\\sigma^2 + \\frac{\\Delta_0}{K}.\n  \\end{equation}\n\\end{lemma}","sha256_utf8":"ed8d1125594f4720a489dc649c8b7791eccdb03f166c81a4420a2e6117db5c3a"},{"start":71814,"end":73521,"start_line":1122,"end_line":1151,"raw_tex":"\\begin{proof}\n  We proceed as in Lemma~\\ref{lem:estimate-sum-of-delta}. By using Lemma~\\ref{lem:stoch-estimate-Delta} we obtain\n  $j>0$ and $m=jB$ we have that\n  \\begin{equation}\n    \\label{eq:stoch--sum-over-any-block}\n    \\begin{aligned}\n      \\sum_{k=jB}^{(j+1)B-1} \\hat{\\Delta}_k %&\\le \\eta_x L(L+L_f+\\sigma)  B^2 + \\frac{1}{2\\eta_y} \\Exp{\\Vert y_{jB-1}-y^*_{jB} \\Vert^2} + B\\eta_y\\sigma^2  \\\\\n      &\\le \\eta_x L(L+L_f+\\sigma) B^2 + \\frac{1}{2\\eta_y}D_h^2 + B\\eta_y \\sigma^2.\n    \\end{aligned}\n  \\end{equation}\n  For $j=0$ we use $m=1$ and do not estimate $\\Delta_0$ but leave it there.\n  % with $m=1$ and the fact that $\\sum_{k=1}^{B-1} k \\le \\nicefrac{B^2}{2}$ we get that\n  % \\begin{equation}\n  %   \\label{eq:stoch--sum-first---block}\n  %   \\begin{aligned}\n  %     \\sum_{k=0}^{B-1} \\hat{\\Delta}_k &\\le \\Delta_0 + \\eta_x L(L+L_f+\\sigma)  B^2 + \\frac{1}{2\\eta_y} \\Exp{\\Vert y_{0}-y^*_{1} \\Vert^2} + B\\eta_y\\sigma^2  \\\\\n  %     &\\le \\Delta_0 + \\eta_x L(L+L_f+\\sigma) B^2 + \\frac{1}{2\\eta_y}D_h^2 + B\\eta_y \\sigma^2,\n  %   \\end{aligned}\n  % \\end{equation}\n  % where $D_h$ denotes the diameter of $\\dom h$, see Assumption~\\ref{ass:regularizers}.\n  % Analogously, for $j>0$ and $m=jB$ we have that\n  % \\begin{equation}\n  %   \\label{eq:stoch--sum-over-any-block}\n  %   \\begin{aligned}\n  %     \\sum_{k=jB}^{(j+1)B-1} \\hat{\\Delta}_k &\\le \\eta_x L(L+L_f+\\sigma)  B^2 + \\frac{1}{2\\eta_y} \\Exp{\\Vert y_{jB-1}-y^*_{jB} \\Vert^2} + B\\eta_y\\sigma^2  \\\\\n  %     &\\le \\eta_x L(L+L_f+\\sigma) B^2 + \\frac{1}{2\\eta_y}D_h^2 + B\\eta_y \\sigma^2,\n  %   \\end{aligned}\n  % \\end{equation}\n  Plugging~\\eqref{eq:stoch--sum-over-any-block} into~\\eqref{eq:split--summation-into-blocks} gives the statement of the lemma.\n\\end{proof}","sha256_utf8":"639c5c95a163db9b95868295b411fdf1c0f8c848c736e3c91d61f1de0d236e9e"},{"start":73594,"end":76061,"start_line":1155,"end_line":1192,"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm:alternating-rate-stoch}]\n  We sum up the inequality of Lemma~\\ref{lem:stoch-descent-on-moreau} to deduce that\n  \\begin{equation*}\n    \\Exp{\\psi_\\lambda(x_{K})} \\le \\psi_\\lambda(x_0) + 2\\eta_x\\rho\\sum_{k=0}^{K-1}\\hat{\\Delta}_k - \\frac{\\eta_x}{2}\\sum_{k=0}^{K-1}\\Exp{\\Vert \\nabla \\psi_\\lambda(x_k) \\Vert^2} + 4K\\rho\\eta_x^2(L^2+\\sigma^2).\n  \\end{equation*}\n  Thus, by dividing by $K$ and $\\eta_x$ yields\n  \\begin{equation*}\n    \\frac{1}{K}\\sum_{k=0}^{K-1} \\Exp{\\Vert\\nabla \\psi_\\lambda(x_k)\\Vert^2}  \\le \\frac{2\\Delta^*}{\\eta_x K} + \\frac{4\\rho}{K}\\sum_{k=0}^{K-1}\\hat{\\Delta}_k + 8\\rho\\eta_x(L^2+\\sigma^2).\n  \\end{equation*}\n  Now we plug in~\\eqref{eq:stoch-estimate-sum-of-delta} to obtain\n  \\begin{equation*}\n    \\begin{aligned}\n      \\MoveEqLeft\\frac{1}{K}\\sum_{k=0}^{K-1} \\Exp{\\Vert\\nabla \\psi_\\lambda(x_k)\\Vert^2} \\\\\n      &\\le \\frac{2\\Delta^*}{\\eta_x K} + 4\\rho\\Big( \\eta_x L(L+L_f+\\sigma) B  + \\frac{D_h^2}{2\\eta_y B} + \\eta_y\\sigma^2 \\Big)  + 4\\rho\\frac{\\Delta_0}{K}+ 8\\rho\\eta_x (L^2+\\sigma^2).\n    \\end{aligned}\n  \\end{equation*}\n  With the block size $B=D_h \\sqrt{1/(\\eta_x\\eta_y L(L+L_f+\\sigma))}$ we have that\n  \\begin{equation*}\n    \\eta_x L(L+L_f+\\sigma) B  + \\frac{D_h^2}{2\\eta_y B} + \\eta_y\\sigma^2 = \\sqrt{\\frac{\\eta_x}{\\eta_y}}\\sqrt{L(L+L_f+\\sigma) }D_h + \\eta_y \\sigma^2\n  \\end{equation*}\n  % \\begin{equation*}\n  %   \\begin{aligned}\n  %     \\MoveEqLeft \\frac{1}{K}\\sum_{k=0}^{K-1} \\Exp{\\Vert \\nabla \\psi_\\lambda(x_k) \\Vert^2}\\\\\n  %     &\\le \\frac{2\\Delta^*}{\\eta_x K} + 4\\rho\\sqrt{\\frac{\\eta_x}{\\eta_y}}\\sqrt{L(L+L_f+\\sigma) }D_h + 4\\rho\\eta_y \\sigma^2 + 4\\rho\\frac{\\Delta_0}{K} + 8\\rho\\eta_x(L^2+\\sigma^2).\n  %   \\end{aligned}\n  % \\end{equation*}\n  Via the step size choice presented in the theorem we obtain the desired complexity.\n  % \\begin{equation}\n  % \\eta_x = \\min \\left\\{\\frac{\\epsilon^2}{\\rho(L^2+\\sigma^2)}, \\frac{\\epsilon^4}{\\rho^2 L(L+L_f+\\sigma)D_h^2 L_{\\nabla\\Phi}}, \\frac{\\epsilon^6}{\\rho^3L(L+L_f+\\sigma)\\sigma^2D_h^2 }\\right\\}\n  % \\end{equation}\n  % and $\\eta_y= \\min\\{\\frac{1}{2 L_{\\nabla\\Phi}}, \\frac{\\epsilon^2}{\\rho \\sigma^2}\\}$ we obtain\n  % \\begin{equation*}\n  %   \\begin{aligned}\n  %     \\mathcal{O}\\left(\\frac{\\Delta^*(L^2+L_f^2+\\sigma^2)\\rho}{\\epsilon^4}\\max\\Big\\{1, \\frac{\\rho L_{\\nabla\\Phi}LD_h^2}{\\epsilon^2}, \\frac{\\rho^2 D_h^2 \\sigma^2}{\\epsilon^4}\\Big\\}+ \\frac{\\Delta_0\\rho}{\\epsilon^2}\\right)\n  %   \\end{aligned}\n  % \\end{equation*}\n  % which finishes the proof.\n\\end{proof}","sha256_utf8":"758ce4c123070eac0c7ee3226534f9992df82a383832fe1aa1e5274fee78639a"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"复合极小极大问题","source_units":[{"start":4238,"end":4463,"start_line":134,"end_line":138,"raw_tex":"\\begin{equation}\n  \\label{eq:minimax}\n  \\min_{x\\in\\R^d} \\max_{y\\in\\R^n}\\, \\Big\\{\\Psi(x,y):=f(x) + \\Phi(x,y) - h(y)\\Big\\}\n  % \\min_{x\\in\\R^d} \\Big\\{f(x) + \\max_{y\\in\\R^n}\\, \\{\\psi(x,y):= \\Phi(x,y) - h(y)\\}\\Big\\}\n\\end{equation}","sha256_utf8":"f6c28f2b0313ada42cfabd0c8a79ada240ef5c4bfd47ddda35ed7a571bd1897e"}]},{"id":"S002","semantic_type":"notation","kind":"notation","title":"目标与最大值函数记号","source_units":[{"start":10046,"end":10191,"start_line":196,"end_line":199,"raw_tex":"\\begin{equation}\n  \\label{eq:max-function}\n  \\phi(x) := \\max_{y\\in\\R^n}\\, \\Phi(x,y) - h(y),\\quad \\text{where $\\phi: \\R^d \\to \\R$}.\n\\end{equation}","sha256_utf8":"7b60d8ef29af71c2ef92b29692b4a4fb05880a100911b46e69e144a10343ab6d"},{"start":23907,"end":24014,"start_line":318,"end_line":320,"raw_tex":"\\begin{equation*}\n  \\psi := \\phi+f, \\quad \\text{where $\\psi:\\R^d \\to \\R \\cup\\{+\\infty\\}$}.\n\\end{equation*}\n","sha256_utf8":"1b69bc0a7004ff85d88728848048d46e6919db3f37b046ca7ce8cba424fd76b9"},{"start":33399,"end":33694,"start_line":476,"end_line":480,"raw_tex":"    Coupling function & $\\Phi(x,y)$ \\\\\n    Objective function & $\\Psi(x,y):= f(x) + \\Phi(x,y) - h(y)$ \\\\\n    Regularized coupling function & $\\Gamma(x,y):=\\Phi(x,y) - h(y)$ \\\\\n    Max function & $\\phi(x):= \\max_{y} \\Phi(x,y) - h(y)$ \\\\\n    Regularized max function & $\\psi(x):= f(x)+\\phi(x)$ \\\\\n","sha256_utf8":"04034e17235fb76680495142f90a62181133b9cab2ff1706fe922dba391ac6bf"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"弱凸性与光滑性","source_units":[{"start":24779,"end":25264,"start_line":330,"end_line":335,"raw_tex":"For some $\\rho \\ge 0$, we say that\n\\begin{equation*}\n  \\label{eq:weakly_convex}\n  \\mbox{$\\psi:\\R^d \\to \\R\\cup\\{+\\infty\\}$ is \\emph{$\\rho$-weakly convex} if $\\psi + (\\rho/2) \\Vert \\cdot \\Vert^2$ is convex.}\n\\end{equation*}\nAn example of a weakly convex function is one which is differentiable and the gradient is uniformly Lipschitz continuous with constant $L$ (we call such a function $L$-smooth). In this case, the weak convexity parameter $\\rho$ is given by the Lipschitz constant.\n","sha256_utf8":"63a0c164321048453ba65922aeebb3f1e3ddc21b8e21984f6ccd780869610c62"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"包络与近端算子","source_units":[{"start":25590,"end":26399,"start_line":339,"end_line":349,"raw_tex":"For a proper, $\\rho$-weakly convex and l.s.c.\\ function $\\psi: \\R^d \\rightarrow \\R\\cup\\{+\\infty\\}$, the Moreau envelope of $\\psi$ with the parameter $\\lambda \\in (0,\\rho^{-1})$ is the function from $\\R^d$ to $\\R$ defined by\n\\begin{equation*}\n  \\label{eq:moreau}\n  \\psi_\\lambda(x) := \\inf_{z \\in \\R^d} \\Big\\{ \\psi(z) + \\frac{1}{2\\lambda}\\Vert z-x \\Vert^2\\Big\\}.\n\\end{equation*}\nThe proximal operator of the function $\\lambda \\psi$ is the $\\arg\\min$ of the right-hand side in this definition, that is,\n\\begin{equation}\n  \\label{eq:prox}\n  \\prox{\\lambda \\psi}{x} := \\argmin_{z \\in \\R^d}\\Big\\{ \\psi(z) + \\frac{1}{2\\lambda}\\Vert z-x \\Vert^2\\Big\\}.\n\\end{equation}\nNote that $\\prox{\\lambda \\psi}{x}$ is uniquely defined by~\\eqref{eq:prox} because the function being minimized is proper, l.s.c.\\ and strongly convex.\n","sha256_utf8":"002f24b6c8bb31e44c955c1d7fb2b325d4a7838d266b937f9b45c00dc5241616"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"弗雷歇次微分","source_units":[{"start":26399,"end":27102,"start_line":350,"end_line":356,"raw_tex":"% TODO: consider moving this paragraph past the definition of \\epsilon-Stationarity??!!\nFor a function $\\psi:\\R^d \\to \\R\\cup\\{+\\infty\\}$ and a point $\\bar{x}$ such that $\\psi(\\bar{x})$ is finite, the \\emph{Fr\\'echet subdifferential} of $\\psi$ at $\\bar{x}$, denoted by $\\partial \\psi(\\bar{x})$, is the set of all vectors $v \\in \\R^d$ such that\n\\begin{equation*}\n  \\label{eq:subgradient}\n  \\psi(x) \\ge \\psi(\\bar{x}) + \\langle v, x- \\bar{x}\\rangle + o(\\Vert x-\\bar{x} \\Vert) \\quad \\text{as $x \\to \\bar{x}$}.\n\\end{equation*}\nFor weakly convex function the Fr\\'echet subdifferential can simply be expressed in terms of the convex subdifferential of the (convex) function $\\psi+ (\\rho/2)\\Vert \\cdot \\Vert^2$.\n","sha256_utf8":"1823137840aa602a9a2443dde646dc6cf701a8f2cc7f7e19e49961b9f979ce81"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"包络近似驻点","source_units":[{"start":28832,"end":29579,"start_line":385,"end_line":392,"raw_tex":"The Moreau envelope allows us to naturally define a notion of \\emph{near stationarity} even for nonsmooth and $\\rho$-weakly convex functions.\n% We say that a point $\\bar{x}$ is stationary for a function if the Fr\\'echet subdifferential of the function contains $0$ at $\\bar{x}$. The concept of \\emph{nearly stationary} is not quite as straightforward. We will use the norm of the gradient of the Moreau envelope as a proxy.\nWe say that for an $\\epsilon>0$ and a $\\lambda\\in(0,\\rho^{-1})$\n\\begin{equation}\n  \\label{eq:epsilon-stationary}\n  \\mbox{a point $x$ is $\\epsilon$-\\emph{stationary} for $\\psi$ if $\\Vert \\nabla \\psi_\\lambda(x) \\Vert \\le \\epsilon$}.\n\\end{equation}\nCanonically, we call a point stationary if the above holds for $\\epsilon=0$.\n","sha256_utf8":"5cdbd888a12d75fb46ef174908d6d05e1704111904a59b75cc58612fe40b6c00"}]},{"id":"S007","semantic_type":"problem_setting","kind":"problem_setting","title":"随机预言机与独立采样","source_units":[{"start":30547,"end":31042,"start_line":411,"end_line":415,"raw_tex":"We discuss the stochastic version of problem~\\eqref{eq:minimax} where the coupling function $\\Phi$ is actually given as an expectation,\n\\begin{equation*}\n  \\Phi(x,y) = \\E_{\\xi\\sim\\mathcal{D}}\\left[ \\Phi(x,y;\\xi)\\right] \\quad \\forall (x,y)\\in\\R^d\\times\\R^n\n\\end{equation*}\nand we can only access independent samples of the gradient $\\nabla_x\\Phi(x,y;\\xi)$ (or subgradient) and $\\nabla_y\\Phi(x,y;\\zeta)$, where $\\xi$ and $\\zeta$ are drawn from the (in general unknown) distribution $\\mathcal{D}$.\n","sha256_utf8":"4e8f602ab196959b0cf150a12dbcd795893675fae918ae1cfaab644347fd3fef"}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","title":"随机估计无偏","source_units":[{"start":31144,"end":31576,"start_line":418,"end_line":428,"raw_tex":"\\begin{assumption}[unbiased]%\n  \\label{ass:unbiased}\n  The stochastic gradient estimator is unbiased, i.e.\n  \\begin{equation*}\n    \\Exp{\\nabla\\Phi(x,y;\\xi)} = \\nabla\\Phi(x,y) \\quad \\forall (x,y)\\in \\R^d\\times \\R^n,\n  \\end{equation*}\n  or in the case of subgradients\n  \\begin{equation*}\n    \\Exp{g^\\xi} \\in \\partial [\\Phi(\\cdot,y)](x), \\quad\\text{where $g^\\xi \\in \\partial [\\Phi(\\cdot,y;\\xi)](x)$}.\n  \\end{equation*}\n\\end{assumption}","sha256_utf8":"a9ef4b60495ca37352d20354abc70d0ad93a06b86b20a973d96a067b8eec8b7e"}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","title":"随机估计方差有界","source_units":[{"start":31578,"end":32339,"start_line":430,"end_line":443,"raw_tex":"\\begin{assumption}[bounded variance]%\n  \\label{ass:bounded-variance}\n  The variance of the estimator is uniformly bounded, i.e.\\ for all $(x,y)\\in\\R^d\\times\\R^n$ and a variance $\\sigma^2\\ge0$ we have\n  \\begin{equation}\n    \\label{eq:bounded-variance}\n    \\Exp{\\Vert \\nabla\\Phi(x,y;\\xi) - \\nabla\\Phi(x,y) \\Vert^2} \\le \\sigma^2.\n  \\end{equation}\n  In the setting of Section~\\ref{sec:non-strongly-concave} where $\\Phi$ is not necessarily smooth in the first component, we make the analogous assumption for subgradients, i.e.\\\n  \\begin{equation}\n    \\label{eq:bounded-variance-subgradient}\n    \\Exp{\\left\\Vert g^\\xi-\\Exp{g^\\xi}  \\right\\Vert^2} \\le \\sigma^2,\n  \\end{equation}\n  for a stochastic subgradient $g^\\xi\\in \\partial[\\Phi(\\cdot,y;\\xi)](x)$.\n\\end{assumption}","sha256_utf8":"b5b82745109a2e997cbb2bdbc296e8a249e116079b67fb1c1af3874bacf27555"}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","title":"弱凸与光滑凹耦合","source_units":[{"start":34765,"end":35472,"start_line":501,"end_line":516,"raw_tex":"\\begin{assumption}%\n  \\label{ass:weakly-convex-gradient-lipschitz}\n  The coupling function $\\Phi$ is\n  \\begin{enumerate}\n    \\item[(i)]\n      concave and $L_{\\nabla\\Phi}$-smooth in the second component uniformly in $x$,\n      \\begin{equation*}\n        \\Vert \\nabla_y\\Phi (x,y) - \\nabla_y\\Phi(x, y') \\Vert \\le L_{\\nabla\\Phi} \\Vert y - y' \\Vert \\quad \\forall x \\in \\R^d \\, \\forall y,y' \\in \\R^n.\n      \\end{equation*}\n      \\item[(ii)]\n        $\\rho$-weakly convex in the first component uniformly in the second one, i.e.\n        \\begin{equation*}\n          \\Phi(\\cdot,y) + \\frac{\\rho}{2}\\Vert \\cdot \\Vert^2 \\quad \\text{is convex for all $y\\in\\R^n$}.\n        \\end{equation*}\n  \\end{enumerate}\n\\end{assumption}","sha256_utf8":"2c17a7f1ae57724433b36ad2a999be6393226cc519827f6b9ab1e7df2fb850f5"}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","title":"目标与包络下界","source_units":[{"start":35925,"end":36087,"start_line":525,"end_line":528,"raw_tex":"\\begin{assumption}%\n  \\label{ass:max-function-lower-bounded}\n  The function $\\psi$ is lower bounded, i.e.\\ $\\inf_{x\\in\\R^d}\\, \\psi(x) > -\\infty$.\n\\end{assumption}","sha256_utf8":"a60aa33303a01f00f540652d4f39fce96da596b060f362eff34905afc3f45087"},{"start":36088,"end":36397,"start_line":529,"end_line":532,"raw_tex":"In Section~\\ref{sec:non-strongly-concave} we will actually need to bound the Moreau envelope $\\psi_\\lambda$, but these two conditions are in fact equivalent as for all $x\\in\\R^d$ and any $\\lambda\\in (0,\\rho^{-1})$\n\\begin{equation*}\n  \\psi(x) \\ge \\psi_\\lambda(x) \\ge \\inf_{u\\in\\R^d}\\, \\psi(u).\n\\end{equation*}\n","sha256_utf8":"dccac1ca050b07e3415925355030c1e6d60f2f74b40f66e6054858f28a4ae345"}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","title":"第一分量利普希茨性","source_units":[{"start":36947,"end":37260,"start_line":542,"end_line":548,"raw_tex":"\\begin{assumption}%\n  \\label{ass:lipschitz}\n  $\\Phi$ is $L$-Lipschitz in the first component uniformly over $\\dom h$ in the second one, i.e.\n  \\begin{equation*}\n    \\Vert \\Phi(x, y) - \\Phi(x', y) \\Vert \\le L \\Vert x - x' \\Vert \\quad \\forall x,x' \\in \\R^d\\; \\forall y \\in \\dom h.\n  \\end{equation*}\n\\end{assumption}","sha256_utf8":"5d9c82e2a9f9f7715b96cbdcd243098331bde8ebb48a2b750d60eb1017248223"}]},{"id":"S013","semantic_type":"assumption","kind":"assumption","title":"正则项与有界对偶域","source_units":[{"start":37333,"end":38028,"start_line":551,"end_line":565,"raw_tex":"\\begin{assumption}%\n  \\label{ass:regularizers}\n  The regularizers $f$ and $h$ are proper, l.s.c.\\ and convex\n  \\begin{enumerate}\n    \\item[(i)]\n      Additionally, $f$ is either $L_f$-Lipschitz continuous on its domain, which is assumed to be open, or the indicator of a nonempty, convex and closed set.\n      Either of those assumptions guarantees for any $\\gamma>0$ the bound\n      \\begin{equation}\n        \\label{eq:bound-prox-f}\n        \\Vert \\prox{\\gamma f}{x}-x \\Vert \\le \\gamma L_f \\quad \\forall x \\in \\dom f.\n      \\end{equation}\n    \\item[(ii)]\n    Furthermore, $h$ has a bounded domain $\\dom h$ such that the diameter of $\\dom h$ is bounded by $D_h$.\n  \\end{enumerate}\n\\end{assumption}","sha256_utf8":"9c1757d7904b44f0c05b40f131a740952b778b662f5d5d93d37ca765f5e9630e"}]},{"id":"S014","semantic_type":"notation","kind":"notation","title":"最大化解集与选择","source_units":[{"start":38926,"end":39143,"start_line":574,"end_line":578,"raw_tex":"This implies that for every $x\\in\\R^d$ the set\n\\begin{equation}\n  \\label{eq:solution-set-mapping}\n  Y(x) := \\Big\\{y^* \\in \\R^n : \\phi(x) = \\Phi(x,y^*) -h(y^*) = \\max_{y\\in\\R^n} \\{\\Phi(x,y)-h(y)\\} \\Big\\}\n\\end{equation}","sha256_utf8":"8c0dc675ffbd96b97dfa0769b4731f58aa6dadeb206881551d8b6531d3a7c483"},{"start":39144,"end":39238,"start_line":579,"end_line":579,"raw_tex":"is nonempty. For brevity we denote arbitrary elements of $Y(x_k)$ by $y_k^*$ for all $k\\ge0$.\n","sha256_utf8":"45175b5a284f1791892092c5ab98ae8da269649fbce59b06f9108a82dc3c016d"}]},{"id":"S015","semantic_type":"algorithm","kind":"algorithm","title":"确定性交替近端迭代","source_units":[{"start":42336,"end":42764,"start_line":639,"end_line":648,"raw_tex":"For initial values $(x_0,y_0)\\in \\dom f \\times \\dom h$ the deterministic version of alternating GDA, for $g_k \\in \\partial [\\Phi(\\cdot,y_k)](x_k)$, reads as\n\\begin{equation}%\n  \\label{eq:alt-gda-non-strongly}\n  (\\forall k \\ge 0)\n  \\left\\lfloor\n    \\begin{array}{l}\n      x_{k+1} = \\prox{\\eta_x f}{x_k - \\eta_x g_k} \\\\\n      y_{k+1} = \\prox{\\eta_y h}{y_k + \\eta_y \\nabla_y\\Phi(x_{k+1},y_k)}.\n    \\end{array}\\right.\n\\end{equation}","sha256_utf8":"9f4498f0ef6fda279be486959e2d8f935ad408632323baf405eddeea5bcd1e2b"}]},{"id":"S016","semantic_type":"algorithm","kind":"algorithm","title":"随机交替近端迭代","source_units":[{"start":60844,"end":61237,"start_line":946,"end_line":955,"raw_tex":"For initial values $(x_0,y_0)\\in \\dom f \\times \\dom h$ the stochastic version of alternating GDA is given by\n\\begin{equation}%\n  \\label{eq:gda-non-strongly-stoch}\n  (\\forall k\\ge 0)\n  \\left\\lfloor\n    \\begin{array}{l}\n      x_{k+1} = \\prox{\\eta_x f}{x_k - \\eta_x g_k^\\xi} \\\\\n      y_{k+1} = \\prox{\\eta_y h}{y_k + \\eta_y \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k)},\n    \\end{array}\\right.\n\\end{equation}","sha256_utf8":"2725515712121322a7d623ca040e49e2d204f14d42b85c1c110fc6a0d5cd3c65"},{"start":61238,"end":61370,"start_line":956,"end_line":956,"raw_tex":"for $g_k^\\xi \\in \\partial[\\Phi(\\cdot,y_k;\\xi_k)](x_k)$ for $\\xi_k,\\zeta_k\\sim\\mathcal{D}$ independent from all previous iterates.\\\\\n","sha256_utf8":"bcb282e26ff2e91f0a07bdf83077ff9384fd4532cefe0522782fca9f2f43f496"}]},{"id":"S017","semantic_type":"notation","kind":"notation","title":"分块求和约定","source_units":[{"start":56314,"end":56586,"start_line":868,"end_line":868,"raw_tex":"In order to estimate the summation of $\\Delta_k$ we will use a trick to sum over it in blocks, where the size $B$ of these blocks will depend on the total number of iterations $K$. Note that w.l.o.g.\\ we assume that the block size $B \\le K$ divides $K$ without remainder.\n","sha256_utf8":"aad86a00faf20dabaad216da41323e10038256d91291118bcce3c306426cddb7"},{"start":71268,"end":71492,"start_line":1112,"end_line":1112,"raw_tex":"In order to estimate the summation of $\\hat{\\Delta}_k$ we will use the same trick as in the deterministic setting and sum over it in blocks, where the size $B$ of these blocks will divide the total number of iterations $K$.\n","sha256_utf8":"a65210a6e1de56af591ac07cd1ed98aa1078547f9c30a4cf9451c63982da4b9d"}]},{"id":"S018","semantic_type":"lemma","kind":"lemma","title":"利普希茨次梯度有界","source_units":[{"start":27218,"end":27478,"start_line":359,"end_line":362,"raw_tex":"\\begin{lemma}[{see~\\cite[Theorem 3.52]{mordukhovich-variational-analysis}}]%\n  \\label{lem:lipschitz-bounded-subgradients}\n  For an $L_\\psi$-Lipschitz continuous function $\\psi:\\R^d \\to \\R$ every Fr\\'echet subgradient is bounded in norm by $L_\\psi$.\n\\end{lemma}","sha256_utf8":"209aa48a210ac4676123f3b549e1d8f8810d152ec39cb2121c7595cab92c7a8a"}]},{"id":"S019","semantic_type":"lemma","kind":"lemma","title":"包络梯度近端表示","source_units":[{"start":27563,"end":28109,"start_line":366,"end_line":374,"raw_tex":"\\begin{lemma}[{see~\\cite[Lemma 2.2]{stoch_weakly_model_based}}]%\n  \\label{lem:grad_smooth_is_prox}\n  Let $\\psi: \\R^d \\to \\R\\cup\\{+\\infty\\}$ be a proper, $\\rho$-weakly convex, and l.s.c.\\ function, and let $\\lambda \\in (0,\\rho^{-1})$. Then the Moreau envelope $\\psi_\\lambda$ is continuously differentiable\n  on $\\R^d$ with gradient\n  \\begin{equation*}\n    \\nabla \\psi_\\lambda(x) = \\frac{1}{\\lambda}\\left(x - \\prox{\\lambda \\psi}{x}\\right) \\quad \\mbox{for all $x \\in \\R^d$},\n  \\end{equation*}\n  and this gradient is Lipschitz continuous.\n\\end{lemma}","sha256_utf8":"68a3ea4c840463d434eb8e78e5c756cfbc4a3926ff39e07e4e1f1813d6915823"}]},{"id":"S020","semantic_type":"proposition","kind":"proposition","title":"最大值函数弱凸性","source_units":[{"start":39240,"end":39717,"start_line":582,"end_line":590,"raw_tex":"\\begin{proposition}[Subgradient characterization of the max function]%\n  \\label{thm:extended-danskin}\n  Let Assumption~\\ref{ass:weakly-convex-gradient-lipschitz} and~\\ref{ass:regularizers} hold true.\n  Then, the function $\\phi$, see~\\eqref{eq:max-function}, fulfills\n  \\begin{equation*}\n    \\partial [\\Phi(\\cdot,y^*)](x) \\subseteq \\partial\\phi(x) \\quad \\forall y^*\\in Y(x), \\forall x\\in \\R^d.\n  \\end{equation*}\n  In particular, $\\phi$ is $\\rho$-weakly convex.\n\\end{proposition}","sha256_utf8":"81ffb8d587200258579469f8a02d57bedf3ed6ba8dc21e54e11cbe4cc8b7382d"}]},{"id":"S021","semantic_type":"proof","kind":"proof","title":"最大值次微分证明","source_units":[{"start":39718,"end":41650,"start_line":591,"end_line":616,"raw_tex":"\\begin{proof}\n  From the $\\rho$-weak convexity of $\\Phi(\\cdot,y)$, we have that $\\Phi(\\cdot,y) + \\frac{\\rho}{2} \\|\\cdot\\|^2$ is convex for all $y\\in\\R^n$. We define $\\tilde{\\Phi}(x,y) = \\Phi(x,y) + \\frac\\rho2 \\|x\\|^2$ and $\\tilde{\\Gamma}(x,y) = \\Gamma(x,y)+\\frac{\\rho}{2}\\Vert x \\Vert^2$ for $(x,y)\\in\\R^d\\times\\R^n$ as well as\n  \\begin{equation*}\n    \\tilde{\\phi}(x) = \\max_{y \\in \\R^n} \\tilde{\\Gamma}(x,y) = \\phi(x) + \\frac{\\rho}{2} \\|x\\|^2.\n  \\end{equation*}\n  Notice that $\\tilde{\\Gamma}(x,\\cdot)$ is concave for any $x \\in \\R^d$ and $\\tilde{\\Gamma}(\\cdot,y)$ is convex for any $y \\in \\R^n$.\n  Thus, the function $\\tilde{\\phi}$ is convex and $\\dom \\tilde{\\phi} = \\dom \\phi = \\R^d$. Therefore $\\phi$ is continuous, which implies that $\\partial \\phi(x) \\neq \\emptyset$ for any $x \\in \\R^d$.\n  Let $x \\in \\R^d$, $y \\in Y(x)$ and $v \\in \\R^d$. For any $\\alpha >0$ it holds\n  \\begin{equation*}\n    \\frac{\\tilde{\\phi}(x+\\alpha v) - \\tilde{\\phi}(x)}{\\alpha} \\geq \\frac{\\tilde{\\Gamma}(x+\\alpha v, y) - \\tilde{\\Gamma}(x,y)}{\\alpha} =  \\frac{\\tilde{\\Phi}(x+\\alpha v, y) - \\tilde{\\Phi}(x,y)}{\\alpha},\n  \\end{equation*}\n  thus\n  \\begin{equation}\n    \\begin{aligned}\n      \\tilde{\\phi}'(x;v) &= \\inf_{\\alpha >0} \\frac{\\tilde{\\phi}(x+\\alpha v) - \\tilde{\\phi}(x)}{\\alpha} \\geq \\inf_{\\alpha >0} \\frac{\\tilde{\\Phi}(x+\\alpha v, y) - \\tilde{\\Phi}(x,y)}{\\alpha} = [\\tilde{\\Phi}(\\cdot,y)]'(x;v),\n    \\end{aligned}\n  \\end{equation}\n  where $[\\Phi(\\cdot,y)]'(x;v)$ denotes the directional derivative of $\\Phi$ in the first component at $x$ in the direction $v$.\n  In conclusion,\n  \\begin{equation}\n    \\label{eq1}\n    \\tilde{\\phi}'(x;v) \\geq \\sup_{y \\in Y(x)}  [\\tilde{\\Phi}(\\cdot,y)]'(x;v) \\quad \\forall v \\in \\R^d\n  \\end{equation}\n  and for $y\\in Y(x)$ we therefore conclude $\\partial [\\tilde{\\Phi}(\\cdot,y)](x) \\subseteq \\partial\\tilde{\\phi}(x)$.\n  The first statement is obtained by subtracting $\\rho x$ on both sides of the inclusion.\n\\end{proof}","sha256_utf8":"499890d036e110b62080b55881945f929d1c69bb80119d48e62f3857e54f5c89"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"二次凸化与连续性","source_units":[{"start":39732,"end":40511,"start_line":592,"end_line":597,"raw_tex":"  From the $\\rho$-weak convexity of $\\Phi(\\cdot,y)$, we have that $\\Phi(\\cdot,y) + \\frac{\\rho}{2} \\|\\cdot\\|^2$ is convex for all $y\\in\\R^n$. We define $\\tilde{\\Phi}(x,y) = \\Phi(x,y) + \\frac\\rho2 \\|x\\|^2$ and $\\tilde{\\Gamma}(x,y) = \\Gamma(x,y)+\\frac{\\rho}{2}\\Vert x \\Vert^2$ for $(x,y)\\in\\R^d\\times\\R^n$ as well as\n  \\begin{equation*}\n    \\tilde{\\phi}(x) = \\max_{y \\in \\R^n} \\tilde{\\Gamma}(x,y) = \\phi(x) + \\frac{\\rho}{2} \\|x\\|^2.\n  \\end{equation*}\n  Notice that $\\tilde{\\Gamma}(x,\\cdot)$ is concave for any $x \\in \\R^d$ and $\\tilde{\\Gamma}(\\cdot,y)$ is convex for any $y \\in \\R^n$.\n  Thus, the function $\\tilde{\\phi}$ is convex and $\\dom \\tilde{\\phi} = \\dom \\phi = \\R^d$. Therefore $\\phi$ is continuous, which implies that $\\partial \\phi(x) \\neq \\emptyset$ for any $x \\in \\R^d$.\n","sha256_utf8":"10b9c16d71a4f53e6c78717f5a1666b5775d7c5e17da6d9f3ea71905c16a09b9"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"方向差商比较","source_units":[{"start":40511,"end":41261,"start_line":598,"end_line":608,"raw_tex":"  Let $x \\in \\R^d$, $y \\in Y(x)$ and $v \\in \\R^d$. For any $\\alpha >0$ it holds\n  \\begin{equation*}\n    \\frac{\\tilde{\\phi}(x+\\alpha v) - \\tilde{\\phi}(x)}{\\alpha} \\geq \\frac{\\tilde{\\Gamma}(x+\\alpha v, y) - \\tilde{\\Gamma}(x,y)}{\\alpha} =  \\frac{\\tilde{\\Phi}(x+\\alpha v, y) - \\tilde{\\Phi}(x,y)}{\\alpha},\n  \\end{equation*}\n  thus\n  \\begin{equation}\n    \\begin{aligned}\n      \\tilde{\\phi}'(x;v) &= \\inf_{\\alpha >0} \\frac{\\tilde{\\phi}(x+\\alpha v) - \\tilde{\\phi}(x)}{\\alpha} \\geq \\inf_{\\alpha >0} \\frac{\\tilde{\\Phi}(x+\\alpha v, y) - \\tilde{\\Phi}(x,y)}{\\alpha} = [\\tilde{\\Phi}(\\cdot,y)]'(x;v),\n    \\end{aligned}\n  \\end{equation}\n  where $[\\Phi(\\cdot,y)]'(x;v)$ denotes the directional derivative of $\\Phi$ in the first component at $x$ in the direction $v$.\n","sha256_utf8":"2227cf0ec377d059d3845defaddcb240fc7a8b42a97b2edaf01f3d73598cbfdd"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"次微分包含与去凸化","source_units":[{"start":41261,"end":41639,"start_line":609,"end_line":615,"raw_tex":"  In conclusion,\n  \\begin{equation}\n    \\label{eq1}\n    \\tilde{\\phi}'(x;v) \\geq \\sup_{y \\in Y(x)}  [\\tilde{\\Phi}(\\cdot,y)]'(x;v) \\quad \\forall v \\in \\R^d\n  \\end{equation}\n  and for $y\\in Y(x)$ we therefore conclude $\\partial [\\tilde{\\Phi}(\\cdot,y)](x) \\subseteq \\partial\\tilde{\\phi}(x)$.\n  The first statement is obtained by subtracting $\\rho x$ on both sides of the inclusion.\n","sha256_utf8":"acbd50b47d78dc49a1958e39acaf78cd42221eee3c47b1bfe94ecf84e707d851"}]},{"id":"S025","semantic_type":"lemma","kind":"lemma","title":"最大值函数利普希茨性","source_units":[{"start":41654,"end":41870,"start_line":620,"end_line":623,"raw_tex":"\\begin{lemma}[Lipschitz continuity of the max function]%\n  \\label{lem:max-is-lipschitz}\n  The Lipschitz continuity of $\\Phi$ in its first component implies that $\\phi$ is Lipschitz with the same constant.\n\\end{lemma}","sha256_utf8":"15b5c6dfe5920f32d591531ab4cbcff8775142ce7374fae53d3c27fee38d1cf8"}]},{"id":"S026","semantic_type":"proof","kind":"proof","title":"最大值利普希茨证明","source_units":[{"start":41871,"end":42270,"start_line":624,"end_line":633,"raw_tex":"\\begin{proof}\n  Let $x,x' \\in \\R^d$ and $y^*\\in Y(x)$. On the one hand\n  \\begin{equation*}\n    \\begin{aligned}\n      \\phi(x) - \\phi(x') &= \\Phi(x,y^*)-h(y^*) - \\phi(x')\\\\\n                        &\\le \\Phi(x,y^*)-h(y^*) - \\Phi(x',y^*)+h(y^*)\\le L \\Vert x-x' \\Vert.\n    \\end{aligned}\n  \\end{equation*}\n  The reverse direction $\\phi(x') - \\phi(x) \\le L\\Vert x-x' \\Vert$ follows analogously.\n\\end{proof}","sha256_utf8":"6a47b8443595fc564a9b5011c08c511085704a176e4b74056b584fb17c0f6360"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"固定最优对偶点比较","source_units":[{"start":41885,"end":42171,"start_line":625,"end_line":631,"raw_tex":"  Let $x,x' \\in \\R^d$ and $y^*\\in Y(x)$. On the one hand\n  \\begin{equation*}\n    \\begin{aligned}\n      \\phi(x) - \\phi(x') &= \\Phi(x,y^*)-h(y^*) - \\phi(x')\\\\\n                        &\\le \\Phi(x,y^*)-h(y^*) - \\Phi(x',y^*)+h(y^*)\\le L \\Vert x-x' \\Vert.\n    \\end{aligned}\n  \\end{equation*}\n","sha256_utf8":"112437374193eec8d497c7986c0f1e7825d9ac4d56cd2255daa0aa347fd13cd1"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"交换两点得双边界","source_units":[{"start":42171,"end":42259,"start_line":632,"end_line":632,"raw_tex":"  The reverse direction $\\phi(x') - \\phi(x) \\le L\\Vert x-x' \\Vert$ follows analogously.\n","sha256_utf8":"90e34c36aa6cc6281bffb785eafae7c9b003e161c897a3a25d3d1889e3b8af86"}]},{"id":"S029","semantic_type":"lemma","kind":"lemma","title":"辅助近端点重表示","source_units":[{"start":44534,"end":44894,"start_line":671,"end_line":677,"raw_tex":"\\begin{lemma}%\n  \\label{lem:xhatk-as-prox}\n  For any $\\lambda\\in(0,\\rho^{-1})$ and all $k\\ge0$ the point $\\hat{x}_k := \\prox{\\lambda g}{x_k}$ can also be written for some $v_k\\in \\partial \\phi(\\hat{x}_k)$ as\n  \\begin{equation*}\n    \\hat{x}_k = \\prox{\\eta_x f}{\\eta_x\\lambda^{-1}x_k - \\eta_x v_k + (1-\\eta_x\\lambda^{-1})\\hat{x}_k}.\n  \\end{equation*}\n\\end{lemma}","sha256_utf8":"8c317245875a37bc29aed88c101e758a012cb56a5baa4ebe0fba3cd42af5ee25"}]},{"id":"S030","semantic_type":"proof","kind":"proof","title":"辅助近端表示证明","source_units":[{"start":44895,"end":46146,"start_line":678,"end_line":699,"raw_tex":"\\begin{proof}\n  Let $k\\ge0$ be arbitrary but fixed and recall that $g=f+\\phi$. By the definition of $\\hat{x}_k$ we have that\n  \\begin{equation*}\n      0 \\in \\partial \\psi(\\hat{x}_k) + \\frac1\\lambda(\\hat{x}_k-x_k) = \\partial(\\phi+f)(\\hat{x}_k) + \\frac1\\lambda(\\hat{x}_k-x_k).\n  \\end{equation*}\n  % TODO: reference\n  We can estimate through the continuity of $\\phi$ and subdifferential calculus\n  \\begin{equation*}\n    \\frac1\\lambda(x_k - \\hat{x}_k) \\in \\partial(\\phi+f)(\\hat{x}_k) \\subseteq \\partial \\phi(\\hat{x}_k) + \\partial f(\\hat{x}_k).\n  \\end{equation*}\n  Thus, there exists $v_k \\in \\partial \\phi(\\hat{x}_k)$ such that $\\frac1\\lambda (x_k - \\hat{x}_k) \\in v_k + \\partial f(\\hat{x}_k)$.\n  % \\begin{equation*}\n  %   \\frac1\\lambda (x_k - \\hat{x}_k) \\in v_k + \\partial f(\\hat{x}_k).\n  % \\end{equation*}\n  Also,\n  \\begin{equation*}\n    \\begin{aligned}\n      \\frac1\\lambda(x_k - \\hat{x}_k) \\in  \\partial f(\\hat{x}_k) +  v_k &\\Leftrightarrow \\frac{\\eta_x }{\\lambda}x_k - \\eta_x v_k + (1-\\frac{\\eta_x}{\\lambda})\\hat{x}_k \\in \\hat{x}_k + \\eta_x \\partial f(\\hat{x}_k) \\\\\n      &\\Leftrightarrow \\hat{x}_k = \\prox{\\eta_x f}{\\frac{\\eta_x}{\\lambda}x_k - \\eta_x v_k + \\big(1-\\frac{\\eta_x}{\\lambda}\\big)\\hat{x}_k}.\n    \\end{aligned}\n  \\end{equation*}\n\\end{proof}","sha256_utf8":"a04c7e7961fbafdaafa52af5358db9f4819e248d4fb6b849790e7024d89c07ff"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"辅助点最优性条件","source_units":[{"start":44909,"end":45188,"start_line":679,"end_line":682,"raw_tex":"  Let $k\\ge0$ be arbitrary but fixed and recall that $g=f+\\phi$. By the definition of $\\hat{x}_k$ we have that\n  \\begin{equation*}\n      0 \\in \\partial \\psi(\\hat{x}_k) + \\frac1\\lambda(\\hat{x}_k-x_k) = \\partial(\\phi+f)(\\hat{x}_k) + \\frac1\\lambda(\\hat{x}_k-x_k).\n  \\end{equation*}\n","sha256_utf8":"906a8afd3811c95ca5d2152f41eeae3ee19ed51e4c8f5e6f923ad5d31f997e2a"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"分离正则项次梯度","source_units":[{"start":45188,"end":45699,"start_line":683,"end_line":691,"raw_tex":"  % TODO: reference\n  We can estimate through the continuity of $\\phi$ and subdifferential calculus\n  \\begin{equation*}\n    \\frac1\\lambda(x_k - \\hat{x}_k) \\in \\partial(\\phi+f)(\\hat{x}_k) \\subseteq \\partial \\phi(\\hat{x}_k) + \\partial f(\\hat{x}_k).\n  \\end{equation*}\n  Thus, there exists $v_k \\in \\partial \\phi(\\hat{x}_k)$ such that $\\frac1\\lambda (x_k - \\hat{x}_k) \\in v_k + \\partial f(\\hat{x}_k)$.\n  % \\begin{equation*}\n  %   \\frac1\\lambda (x_k - \\hat{x}_k) \\in v_k + \\partial f(\\hat{x}_k).\n  % \\end{equation*}\n","sha256_utf8":"1ab21e9541203b9c87902a11c9d1335b722abae41e38d583ddc5c43b76f27cb5"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"转换为近端不动点","source_units":[{"start":45699,"end":46135,"start_line":692,"end_line":698,"raw_tex":"  Also,\n  \\begin{equation*}\n    \\begin{aligned}\n      \\frac1\\lambda(x_k - \\hat{x}_k) \\in  \\partial f(\\hat{x}_k) +  v_k &\\Leftrightarrow \\frac{\\eta_x }{\\lambda}x_k - \\eta_x v_k + (1-\\frac{\\eta_x}{\\lambda})\\hat{x}_k \\in \\hat{x}_k + \\eta_x \\partial f(\\hat{x}_k) \\\\\n      &\\Leftrightarrow \\hat{x}_k = \\prox{\\eta_x f}{\\frac{\\eta_x}{\\lambda}x_k - \\eta_x v_k + \\big(1-\\frac{\\eta_x}{\\lambda}\\big)\\hat{x}_k}.\n    \\end{aligned}\n  \\end{equation*}\n","sha256_utf8":"e39fce2169ed4b281d0dd5c4c0d54600cca8e528b0204e1696384cb415de2d62"}]},{"id":"S034","semantic_type":"lemma","kind":"lemma","title":"确定性包络下降","source_units":[{"start":46263,"end":46648,"start_line":703,"end_line":710,"raw_tex":"\\begin{lemma}%\n  \\label{lem:descent-on-moreau}\n  With $\\lambda = \\nicefrac{1}{2\\rho}$ and $\\eta_x \\ge 0$ we have for all $k\\ge0$ that\n  \\begin{equation*}\n    \\psi_\\lambda(x_{k+1}) \\le \\psi_\\lambda(x_k) + 2\\rho\\eta_x\\Delta_k - \\frac12\\eta_x \\Vert \\nabla \\psi_\\lambda(x_k) \\Vert^2  + 4\\rho\\eta_x^2L^2,\n  \\end{equation*}\n  where $\\Delta_k := \\psi(x_k) - \\Psi(x_k, y_k) \\ge 0$.\n\\end{lemma}","sha256_utf8":"6f565ab9b81e50fa8ade12bb8fa7d424fafbfbd8bd5d9f05fb3dbcce7c3d6357"}]},{"id":"S035","semantic_type":"proof","kind":"proof","title":"确定性包络下降证明","source_units":[{"start":46649,"end":50768,"start_line":711,"end_line":771,"raw_tex":"\\begin{proof}\n  Let $k\\ge0$ be fixed. As before we denote $\\hat{x}_k = \\prox{\\lambda \\psi}{x_k}$. From the definition of the Moreau envelope we have that\n  \\begin{equation}\n    \\label{eq:moreau-is-min}\n    \\psi_\\lambda(x_{k+1}) = \\min_{x\\in\\R^d} \\Big\\{ \\psi(x) + \\frac{1}{2\\lambda} \\Vert x - x_{k+1} \\Vert^2\\Big\\} \\le \\psi(\\hat{x}_k) + \\frac{1}{2\\lambda} \\Vert \\hat{x}_k - x_{k+1} \\Vert^2.\n  \\end{equation}\n  Let now $v_k \\in \\partial\\phi(\\hat{x}_k)$ as in Lemma~\\ref{lem:xhatk-as-prox}. We successively deduce for $\\beta:=1-\\eta_x \\lambda^{-1}$\n  \\begin{align}\n    \\Vert \\hat{x}_k - x_{k+1} \\Vert^2 &= \\Vert \\prox{\\eta_x f}{\\eta_x\\lambda^{-1}x_k - \\eta_x v_k + \\beta\\hat{x}_k} - \\prox{\\eta_x f}{x_k - \\eta_x g_k}\\Vert^2 \\label{eq:defin-iter}\\\\\n     &\\le \\Vert \\beta(\\hat{x}_k - x_k) + \\eta_x (g_k - v_k)\\Vert^2 \\label{eq:1-Lipschitz}\\\\\n     &= {\\beta}^2\\Vert \\hat{x}_k - x_k \\Vert^2 + 2 \\eta_x\\beta\\langle g_k-v_k, \\hat{x}_k - x_k\\rangle\n     + \\eta_x^2\\Vert g_k - v_k \\Vert^2 \\notag\\\\\n     &\\le {\\beta}^2\\Vert \\hat{x}_k - x_k \\Vert^2 + 2 \\eta_x \\beta\\langle g_k-v_k, \\hat{x}_k - x_k\\rangle + 4\\eta_x^2L^2 \\label{eq:Lipschitz}\n  \\end{align}\n  where~\\eqref{eq:defin-iter} uses Lemma~\\ref{lem:xhatk-as-prox} and the definition of $x_{k+1}$, inequality~\\eqref{eq:1-Lipschitz} holds because of the nonexpansiveness of the proximal operator, and~\\eqref{eq:Lipschitz} follows from the Lipschitz continuity of $\\Phi$ and $\\phi$ (see Lemma~\\ref{lem:max-is-lipschitz}) and the fact that Lipschitz continuity implies bounded subgradients.\n  We are left with estimating the inner product in the above inequality and we do so by splitting it into two:\n  first of all, from the weak convexity of $\\Phi$ in $x$ we have that\n  \\begin{equation*}\n    \\begin{aligned}\n      \\langle g_k, \\hat{x}_k - x_k \\rangle &\\le \\Phi(\\hat{x}_k, y_k) - \\Phi(x_k, y_k) + \\frac{\\rho}{2} \\Vert \\hat{x}_k - x_k \\Vert^2 \\\\\n      % &= \\Phi(\\hat{x}_k, y_k)-h(y_k) - (\\Phi(x_k, y_k)-h(y_k)) + \\frac{\\rho}{2} \\Vert \\hat{x}_k - x_k \\Vert^2\\\\\n      &\\le \\phi(\\hat{x}_k) - \\Gamma(x_k,y_k) + \\frac{\\rho}{2} \\Vert \\hat{x}_k - x_k \\Vert^2.\n    \\end{aligned}\n  \\end{equation*}\n  Secondly, by the $\\rho$-weak convexity of $\\phi$\n  \\begin{equation*}\n    -\\langle v_k, \\hat{x}_k - x_k\\rangle \\le \\phi(x_k) - \\phi(\\hat{x}_k) + \\frac{\\rho}{2} \\Vert \\hat{x}_k - x_k \\Vert^2.\n  \\end{equation*}\n  Combining the last two inequalities we get that\n  \\begin{equation}\n    \\label{eq:estimate-middle}\n    \\langle g_k-v_k, \\hat{x}_k - x_k\\rangle \\le \\phi(x_k) - \\Gamma(x_k,y_k) + \\rho \\Vert \\hat{x}_k - x_k \\Vert^2.\n  \\end{equation}\n  Plugging~\\eqref{eq:estimate-middle} into~\\eqref{eq:Lipschitz} we deduce\n  \\begin{equation}\n    \\begin{aligned}\n    \\label{eq:ab3}\n    \\Vert \\hat{x}_k - x_{k+1} \\Vert^2 \\le \\underbrace{[{(1-\\eta_x\\lambda^{-1})}^2+2\\eta_x(1-\\eta_x \\lambda^{-1})\\rho]}_{= (*)} \\Vert \\hat{x}_k - x_k \\Vert^2 + 2\\eta_x \\Delta_k + 4\\eta_x^2L^2, \\\\\n    % &\\le \\Vert \\hat{x}_k - x_k \\Vert^2 + 2\\eta_x\\Delta_k -2\\eta_x\\rho \\Vert\\hat{x}_k - x_k \\Vert^2  + 2\\eta_x^2 L^2\n    \\end{aligned}\n  \\end{equation}\n  where we used the fact that $1-\\beta\\le1$ in the factor of $\\Delta_k$.\n  Now note that\n  \\begin{equation}\n    \\label{eq:terrible-algebra}\n    \\begin{aligned}\n      (*) &= 1 - 2\\eta_x \\lambda^{-1} + \\eta_x^2 \\lambda^{-2} + 2\\eta_x\\rho - 2\\eta_x^2 \\lambda^{-1}\\rho\\\\\n      &= 1 - 4\\eta_x\\rho + 4 \\eta_x^2\\rho^2 + 2\\eta_x\\rho - 4 \\eta_x^2\\rho^2 = 1-2\\eta_x\\rho.\n    \\end{aligned}\n  \\end{equation}\n  Combining~\\eqref{eq:moreau-is-min},~\\eqref{eq:ab3} and~\\eqref{eq:terrible-algebra} we deduce, using $\\lambda=\\nicefrac{1}{2\\rho}$,\n  % a lot of algebra happens here but it's correct\n  \\begin{equation*}\n    \\label{eq:descent-moreau}\n    \\begin{aligned}\n      \\psi_\\lambda(x_{k+1}) &\\le \\psi(\\hat{x}_k) + \\frac{1}{2\\lambda} \\left(\\Vert \\hat{x}_k - x_k \\Vert^2 + 2\\eta_x\\Delta_k -2\\eta_x\\rho \\Vert\\hat{x}_k - x_k \\Vert^2  + 4\\eta_x^2L^2\\right) \\\\\n      &= \\psi_\\lambda(x_k) + 2\\rho\\eta_x\\Delta_k - \\frac12\\eta_x \\Vert \\nabla \\psi_\\lambda(x_k) \\Vert^2  + 4\\rho\\eta_x^2L^2.\n    \\end{aligned}\n  \\end{equation*}\n  % where we used that $\\lambda=\\nicefrac{1}{2\\rho}$.\n\\end{proof}","sha256_utf8":"f23397374bcfd70e2dcf743bb026106c0ad6be36f45006e12a5ec085c25163ac"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"代入旧辅助点估计包络","source_units":[{"start":46663,"end":47056,"start_line":712,"end_line":716,"raw_tex":"  Let $k\\ge0$ be fixed. As before we denote $\\hat{x}_k = \\prox{\\lambda \\psi}{x_k}$. From the definition of the Moreau envelope we have that\n  \\begin{equation}\n    \\label{eq:moreau-is-min}\n    \\psi_\\lambda(x_{k+1}) = \\min_{x\\in\\R^d} \\Big\\{ \\psi(x) + \\frac{1}{2\\lambda} \\Vert x - x_{k+1} \\Vert^2\\Big\\} \\le \\psi(\\hat{x}_k) + \\frac{1}{2\\lambda} \\Vert \\hat{x}_k - x_{k+1} \\Vert^2.\n  \\end{equation}\n","sha256_utf8":"a91bf037687c75800553e42161499bc04fb1afee178d8883970a0f9b8976d05f"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"非扩张性展开距离","source_units":[{"start":47056,"end":48179,"start_line":717,"end_line":725,"raw_tex":"  Let now $v_k \\in \\partial\\phi(\\hat{x}_k)$ as in Lemma~\\ref{lem:xhatk-as-prox}. We successively deduce for $\\beta:=1-\\eta_x \\lambda^{-1}$\n  \\begin{align}\n    \\Vert \\hat{x}_k - x_{k+1} \\Vert^2 &= \\Vert \\prox{\\eta_x f}{\\eta_x\\lambda^{-1}x_k - \\eta_x v_k + \\beta\\hat{x}_k} - \\prox{\\eta_x f}{x_k - \\eta_x g_k}\\Vert^2 \\label{eq:defin-iter}\\\\\n     &\\le \\Vert \\beta(\\hat{x}_k - x_k) + \\eta_x (g_k - v_k)\\Vert^2 \\label{eq:1-Lipschitz}\\\\\n     &= {\\beta}^2\\Vert \\hat{x}_k - x_k \\Vert^2 + 2 \\eta_x\\beta\\langle g_k-v_k, \\hat{x}_k - x_k\\rangle\n     + \\eta_x^2\\Vert g_k - v_k \\Vert^2 \\notag\\\\\n     &\\le {\\beta}^2\\Vert \\hat{x}_k - x_k \\Vert^2 + 2 \\eta_x \\beta\\langle g_k-v_k, \\hat{x}_k - x_k\\rangle + 4\\eta_x^2L^2 \\label{eq:Lipschitz}\n  \\end{align}\n  where~\\eqref{eq:defin-iter} uses Lemma~\\ref{lem:xhatk-as-prox} and the definition of $x_{k+1}$, inequality~\\eqref{eq:1-Lipschitz} holds because of the nonexpansiveness of the proximal operator, and~\\eqref{eq:Lipschitz} follows from the Lipschitz continuity of $\\Phi$ and $\\phi$ (see Lemma~\\ref{lem:max-is-lipschitz}) and the fact that Lipschitz continuity implies bounded subgradients.\n","sha256_utf8":"022b5365f249a6d487eda0301b1e1f069fbb76ffc7c99be1e956d5b2d92225e8"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"弱凸性控制内积","source_units":[{"start":48179,"end":49220,"start_line":726,"end_line":743,"raw_tex":"  We are left with estimating the inner product in the above inequality and we do so by splitting it into two:\n  first of all, from the weak convexity of $\\Phi$ in $x$ we have that\n  \\begin{equation*}\n    \\begin{aligned}\n      \\langle g_k, \\hat{x}_k - x_k \\rangle &\\le \\Phi(\\hat{x}_k, y_k) - \\Phi(x_k, y_k) + \\frac{\\rho}{2} \\Vert \\hat{x}_k - x_k \\Vert^2 \\\\\n      % &= \\Phi(\\hat{x}_k, y_k)-h(y_k) - (\\Phi(x_k, y_k)-h(y_k)) + \\frac{\\rho}{2} \\Vert \\hat{x}_k - x_k \\Vert^2\\\\\n      &\\le \\phi(\\hat{x}_k) - \\Gamma(x_k,y_k) + \\frac{\\rho}{2} \\Vert \\hat{x}_k - x_k \\Vert^2.\n    \\end{aligned}\n  \\end{equation*}\n  Secondly, by the $\\rho$-weak convexity of $\\phi$\n  \\begin{equation*}\n    -\\langle v_k, \\hat{x}_k - x_k\\rangle \\le \\phi(x_k) - \\phi(\\hat{x}_k) + \\frac{\\rho}{2} \\Vert \\hat{x}_k - x_k \\Vert^2.\n  \\end{equation*}\n  Combining the last two inequalities we get that\n  \\begin{equation}\n    \\label{eq:estimate-middle}\n    \\langle g_k-v_k, \\hat{x}_k - x_k\\rangle \\le \\phi(x_k) - \\Gamma(x_k,y_k) + \\rho \\Vert \\hat{x}_k - x_k \\Vert^2.\n  \\end{equation}\n","sha256_utf8":"3ccc12a711c80c654c2e81ac3747577884a491aaa0eb1e904ed99ed0782a87bd"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"合并距离界并化简系数","source_units":[{"start":49220,"end":50096,"start_line":744,"end_line":760,"raw_tex":"  Plugging~\\eqref{eq:estimate-middle} into~\\eqref{eq:Lipschitz} we deduce\n  \\begin{equation}\n    \\begin{aligned}\n    \\label{eq:ab3}\n    \\Vert \\hat{x}_k - x_{k+1} \\Vert^2 \\le \\underbrace{[{(1-\\eta_x\\lambda^{-1})}^2+2\\eta_x(1-\\eta_x \\lambda^{-1})\\rho]}_{= (*)} \\Vert \\hat{x}_k - x_k \\Vert^2 + 2\\eta_x \\Delta_k + 4\\eta_x^2L^2, \\\\\n    % &\\le \\Vert \\hat{x}_k - x_k \\Vert^2 + 2\\eta_x\\Delta_k -2\\eta_x\\rho \\Vert\\hat{x}_k - x_k \\Vert^2  + 2\\eta_x^2 L^2\n    \\end{aligned}\n  \\end{equation}\n  where we used the fact that $1-\\beta\\le1$ in the factor of $\\Delta_k$.\n  Now note that\n  \\begin{equation}\n    \\label{eq:terrible-algebra}\n    \\begin{aligned}\n      (*) &= 1 - 2\\eta_x \\lambda^{-1} + \\eta_x^2 \\lambda^{-2} + 2\\eta_x\\rho - 2\\eta_x^2 \\lambda^{-1}\\rho\\\\\n      &= 1 - 4\\eta_x\\rho + 4 \\eta_x^2\\rho^2 + 2\\eta_x\\rho - 4 \\eta_x^2\\rho^2 = 1-2\\eta_x\\rho.\n    \\end{aligned}\n  \\end{equation}\n","sha256_utf8":"ada6a450cce6814918162fb6d390bb69507807589e930dc7b8611e095e9e9fab"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"转为包络梯度下降界","source_units":[{"start":50096,"end":50757,"start_line":761,"end_line":770,"raw_tex":"  Combining~\\eqref{eq:moreau-is-min},~\\eqref{eq:ab3} and~\\eqref{eq:terrible-algebra} we deduce, using $\\lambda=\\nicefrac{1}{2\\rho}$,\n  % a lot of algebra happens here but it's correct\n  \\begin{equation*}\n    \\label{eq:descent-moreau}\n    \\begin{aligned}\n      \\psi_\\lambda(x_{k+1}) &\\le \\psi(\\hat{x}_k) + \\frac{1}{2\\lambda} \\left(\\Vert \\hat{x}_k - x_k \\Vert^2 + 2\\eta_x\\Delta_k -2\\eta_x\\rho \\Vert\\hat{x}_k - x_k \\Vert^2  + 4\\eta_x^2L^2\\right) \\\\\n      &= \\psi_\\lambda(x_k) + 2\\rho\\eta_x\\Delta_k - \\frac12\\eta_x \\Vert \\nabla \\psi_\\lambda(x_k) \\Vert^2  + 4\\rho\\eta_x^2L^2.\n    \\end{aligned}\n  \\end{equation*}\n  % where we used that $\\lambda=\\nicefrac{1}{2\\rho}$.\n","sha256_utf8":"c07f8065957eafdd5eca96afb114e9faea6ba32122a1bdd17a1cfa253abb1466"}]},{"id":"S041","semantic_type":"lemma","kind":"lemma","title":"确定性近端上升比较","source_units":[{"start":51075,"end":51407,"start_line":774,"end_line":781,"raw_tex":"\\begin{lemma}%\n  \\label{lem:standard-gradient-ascent-estimation}\n  It holds for all $y\\in\\R^n$ and $k\\ge0$ that\n  \\begin{equation}\n    \\label{eq:gradient-ascent-estimation}\n    \\Psi(x_{k+1}, y) - \\Psi(x_{k+1}, y_{k+1}) \\le  \\frac{1}{2\\eta_y} \\Big(\\Vert y - y_k \\Vert^2 - \\Vert y - y_{k+1} \\Vert^2 \\Big).\n  \\end{equation}\n\\end{lemma}","sha256_utf8":"159d0eb207a11fbfcf162eba06dd636936560b5e393d099a427743d3634e5999"}]},{"id":"S042","semantic_type":"proof","kind":"proof","title":"标准近端上升引证","source_units":[{"start":51408,"end":53354,"start_line":782,"end_line":811,"raw_tex":"\\begin{proof}\n  This is a standard estimate on the improvement made by a single prox-gradient step for a convex (in this case concave) function, see for example~\\cite[Lemma 2.3]{fista}.\n  % By the definition of $y_{k+1}$ we have that\n  % \\begin{equation*}\n  %   y_{k+1} = \\argmin_{y\\in\\R^n} \\Big\\{ h(y) - \\langle\\nabla_y\\Phi(x_{k+1},y_k), y-y_k\\rangle + \\frac{1}{2\\eta_y}\\Vert y-y_k \\Vert^2 \\Big\\}.\n  % \\end{equation*}\n  % Let now $y\\in\\R^n$ be arbitrary but fixed. Since $y_{k+1}$ minimizes a $\\nicefrac{1}{\\eta_y}$-strongly convex function,\n  % \\begin{equation}\n  %   \\label{eq:decrease}\n  %   \\begin{aligned}\n  %     \\MoveEqLeft h(y_{k+1}) - \\langle \\nabla_y\\Phi(x_{k+1}, y_k), y_{k+1}-y_k \\rangle + \\frac{1}{2\\eta_y} \\Vert y_{k+1} -y_k \\Vert^2 + \\frac{1}{2\\eta_y} \\Vert y-y_{k+1} \\Vert^2  \\\\\n  %     &\\le h(y) - \\langle \\nabla_y\\Phi(x_{k+1}, y_k), y-y_k \\rangle + \\frac{1}{2\\eta_y} \\Vert y-y_k \\Vert^2.\n  %   \\end{aligned}\n  % \\end{equation}\n  % From the descent lemma (in ascent form) and the fact that $\\nicefrac{1}{\\eta_y} = L_{\\nabla\\Phi}$ we have that\n  % \\begin{equation}\n  %   \\label{eq:ascent-lemma}\n  %   \\Phi(x_{k+1}, y_k) + \\langle \\nabla_y\\Phi(x_{k+1}, y_k),  y_{k+1} - y_k\\rangle - \\frac{1}{2\\eta_y} \\Vert y_{k+1} - y_k \\Vert^2 \\le \\Phi(x_{k+1}, y_{k+1}).\n  % \\end{equation}\n  % From the concavity of $\\Phi$ in its second component we get that\n  % \\begin{equation}\n  %   \\label{eq:concavity}\n  %   \\Phi(x_{k+1}, y) \\le \\Phi(x_{k+1}, y_k) + \\langle \\nabla_y\\Phi(x_{k+1}, y_k), y-y_k \\rangle.\n  % \\end{equation}\n  % By plugging~\\eqref{eq:concavity} and~\\eqref{eq:ascent-lemma} into~\\eqref{eq:decrease} we deduce\n  % \\begin{equation*}\n  %   \\Phi(x_{k+1},y) -h(y) + \\frac{1}{2\\eta_y}\\Vert y-y_{k+1} \\Vert^2 \\le \\Phi(x_{k+1},y_{k+1})-h(y_{k+1}) + \\frac{1}{2\\eta_y}\\Vert y-y_k \\Vert^2.\n  % \\end{equation*}\n  % The statement of the lemma is obtained by rearranging the above inequality and adding $f(x_{k+1})$ on both sides.\n\\end{proof}","sha256_utf8":"68db736d96dc5e89ea6282415721f764c4e156924b36638233c9d0fc50892878"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"调用标准单步改进估计","source_units":[{"start":51422,"end":53343,"start_line":783,"end_line":810,"raw_tex":"  This is a standard estimate on the improvement made by a single prox-gradient step for a convex (in this case concave) function, see for example~\\cite[Lemma 2.3]{fista}.\n  % By the definition of $y_{k+1}$ we have that\n  % \\begin{equation*}\n  %   y_{k+1} = \\argmin_{y\\in\\R^n} \\Big\\{ h(y) - \\langle\\nabla_y\\Phi(x_{k+1},y_k), y-y_k\\rangle + \\frac{1}{2\\eta_y}\\Vert y-y_k \\Vert^2 \\Big\\}.\n  % \\end{equation*}\n  % Let now $y\\in\\R^n$ be arbitrary but fixed. Since $y_{k+1}$ minimizes a $\\nicefrac{1}{\\eta_y}$-strongly convex function,\n  % \\begin{equation}\n  %   \\label{eq:decrease}\n  %   \\begin{aligned}\n  %     \\MoveEqLeft h(y_{k+1}) - \\langle \\nabla_y\\Phi(x_{k+1}, y_k), y_{k+1}-y_k \\rangle + \\frac{1}{2\\eta_y} \\Vert y_{k+1} -y_k \\Vert^2 + \\frac{1}{2\\eta_y} \\Vert y-y_{k+1} \\Vert^2  \\\\\n  %     &\\le h(y) - \\langle \\nabla_y\\Phi(x_{k+1}, y_k), y-y_k \\rangle + \\frac{1}{2\\eta_y} \\Vert y-y_k \\Vert^2.\n  %   \\end{aligned}\n  % \\end{equation}\n  % From the descent lemma (in ascent form) and the fact that $\\nicefrac{1}{\\eta_y} = L_{\\nabla\\Phi}$ we have that\n  % \\begin{equation}\n  %   \\label{eq:ascent-lemma}\n  %   \\Phi(x_{k+1}, y_k) + \\langle \\nabla_y\\Phi(x_{k+1}, y_k),  y_{k+1} - y_k\\rangle - \\frac{1}{2\\eta_y} \\Vert y_{k+1} - y_k \\Vert^2 \\le \\Phi(x_{k+1}, y_{k+1}).\n  % \\end{equation}\n  % From the concavity of $\\Phi$ in its second component we get that\n  % \\begin{equation}\n  %   \\label{eq:concavity}\n  %   \\Phi(x_{k+1}, y) \\le \\Phi(x_{k+1}, y_k) + \\langle \\nabla_y\\Phi(x_{k+1}, y_k), y-y_k \\rangle.\n  % \\end{equation}\n  % By plugging~\\eqref{eq:concavity} and~\\eqref{eq:ascent-lemma} into~\\eqref{eq:decrease} we deduce\n  % \\begin{equation*}\n  %   \\Phi(x_{k+1},y) -h(y) + \\frac{1}{2\\eta_y}\\Vert y-y_{k+1} \\Vert^2 \\le \\Phi(x_{k+1},y_{k+1})-h(y_{k+1}) + \\frac{1}{2\\eta_y}\\Vert y-y_k \\Vert^2.\n  % \\end{equation*}\n  % The statement of the lemma is obtained by rearranging the above inequality and adding $f(x_{k+1})$ on both sides.\n","sha256_utf8":"c49cc8a20e1eedea0d3fc9da44a3291fb70110bf9207f015e5e288c7036b87d5"}]},{"id":"S044","semantic_type":"lemma","kind":"lemma","title":"确定性对偶间隙界","source_units":[{"start":53497,"end":53829,"start_line":814,"end_line":823,"raw_tex":"\\begin{lemma}%\n  \\label{lem:estimate-Delta}\n  We have that for all $1 \\le m \\le k$,\n  \\begin{equation}\n    \\begin{aligned}\n      \\label{eq:estimate-Delta}\n      \\Delta_k \\le 2\\eta_x L(L+L_f)(k-m) + \\frac{1}{2\\eta_y} \\Big(\\Vert y_{k-1} - y^*_m \\Vert^2 - \\Vert y_k - y^*_m \\Vert^2 \\Big).\n    \\end{aligned}\n  \\end{equation}\n\\end{lemma}","sha256_utf8":"c7d83f1816404d0251d07fa87db157f84c0837f6c73ee7ff6e36aebc6a3c90fe"}]},{"id":"S045","semantic_type":"proof","kind":"proof","title":"确定性间隙估计证明","source_units":[{"start":53830,"end":56312,"start_line":824,"end_line":866,"raw_tex":"\\begin{proof}\n  % Let $1\\le m \\le k$ be fixed.\n  Plugging $y=y^*_m$ into~\\eqref{eq:gradient-ascent-estimation} we deduce that\n  \\begin{equation}\n    \\label{eq:standard_descent_with_ystar}\n    \\begin{aligned}\n       0 \\le \\Psi(x_k, y_k) - \\Psi(x_k, y^*_m) + \\frac{1}{2\\eta_y} \\Big(\\Vert y^*_m - y_{k-1} \\Vert^2 - \\Vert y^*_m - y_k \\Vert^2 \\Big).\n    \\end{aligned}\n  \\end{equation}\n  Starting from the definition of $\\Delta_k = \\Psi(x_k,y^*_k) - \\Psi(x_k,y_k)$, we add~\\eqref{eq:standard_descent_with_ystar} to obtain\n  \\begin{equation}\n    \\label{eq:diff-function-values}\n    \\begin{aligned}\n      \\Delta_k %&= \\Psi(x_k,y^*_k) - \\Psi(x_k,y_k) \\\\\n      &\\le \\Psi(x_k,y^*_k) - \\Psi(x_k,y_m^*) + \\frac{1}{2\\eta_y} \\Big(\\Vert y^*_m - y_{k-1} \\Vert^2 - \\Vert y^*_m - y_k \\Vert^2 \\Big).\n    \\end{aligned}\n  \\end{equation}\n  Due to the Lipschitz continuity of $\\Phi$, terms which only differ in their first argument will be easy to estimate. Therefore, we insert and subtract $\\Phi(x_m,y_{k}^*)$ to deduce\n  \\begin{equation}\n    \\label{eq:plus-minus-functionval}\n    \\begin{aligned}\n      \\MoveEqLeft \\Psi(x_k, y^*_k) - \\Psi(x_k, y^*_m)  \\\\ %&= \\Phi(x_k, y^*_k)-h(y^*_k) - \\Phi(x_k, y^*_m)+h(y^*_m) \\\\\n      &= \\Phi(x_k, y^*_k) - \\Phi(x_m, y^*_k) + \\Phi(x_m, y^*_k)-h(y^*_k) - \\Phi(x_k, y^*_m) + h(y^*_m) \\\\\n      &\\le \\Phi(x_k, y^*_k) - \\Phi(x_m, y^*_k) + \\Phi(x_m, y^*_m)-h(y^*_m) - \\Phi(x_k, y^*_m) + h(y^*_m)\\\\\n      &= \\Phi(x_k,y^*_k) - \\Phi(x_m,y^*_k) + \\Phi(x_m,y^*_m) - \\Phi(x_k, y^*_m).\n    \\end{aligned}\n  \\end{equation}\n  We estimate the above expression for $k>m$ by making use of the Lipschitz continuity of $\\Phi(\\cdot, y)$ and~\\eqref{eq:bound-prox-f} deducing\n  \\begin{equation}\n    \\label{eq:fun-val-same-y}\n    \\begin{aligned}\n      \\MoveEqLeft \\Phi(x_k, y^*_k) - \\Phi(x_m, y^*_k)  \\le L \\Vert x_k - x_m \\Vert \\le L \\sum_{l=m}^{k-1} \\Vert x_{l+1}-x_l \\Vert \\\\\n      &\\le L \\sum_{l=m}^{k-1} \\Big( \\Vert \\prox{\\eta_x f}{x_l -\\eta_x g_l} - \\prox{\\eta_x f}{x_l} \\Vert + \\Vert \\prox{\\eta_x f}{x_l} - x_l \\Vert \\Big)\\\\\n      & \\le \\eta_x L(L+L_f)(k-m).\n    \\end{aligned}\n  \\end{equation}\n  For $k=m$ the inequality follows trivially. Analogously, we deduce\n  \\begin{equation}\n    \\label{eq:fun-val-same-y-2}\n      \\Phi(x_m, y^*_m) - \\Phi(x_k, y^*_m) \\le \\eta_x L(L+L_f)(k-m).\n  \\end{equation}\n  Plugging~\\eqref{eq:plus-minus-functionval},~\\eqref{eq:fun-val-same-y} and~\\eqref{eq:fun-val-same-y-2} into~\\eqref{eq:diff-function-values} gives the statement of the lemma.\n\\end{proof}","sha256_utf8":"b7c28f64215e74cd374d83a1daa8f7360ed135c3ff6e5d413715c3406fd0e384"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"上升比较加入间隙","source_units":[{"start":53844,"end":54645,"start_line":825,"end_line":840,"raw_tex":"  % Let $1\\le m \\le k$ be fixed.\n  Plugging $y=y^*_m$ into~\\eqref{eq:gradient-ascent-estimation} we deduce that\n  \\begin{equation}\n    \\label{eq:standard_descent_with_ystar}\n    \\begin{aligned}\n       0 \\le \\Psi(x_k, y_k) - \\Psi(x_k, y^*_m) + \\frac{1}{2\\eta_y} \\Big(\\Vert y^*_m - y_{k-1} \\Vert^2 - \\Vert y^*_m - y_k \\Vert^2 \\Big).\n    \\end{aligned}\n  \\end{equation}\n  Starting from the definition of $\\Delta_k = \\Psi(x_k,y^*_k) - \\Psi(x_k,y_k)$, we add~\\eqref{eq:standard_descent_with_ystar} to obtain\n  \\begin{equation}\n    \\label{eq:diff-function-values}\n    \\begin{aligned}\n      \\Delta_k %&= \\Psi(x_k,y^*_k) - \\Psi(x_k,y_k) \\\\\n      &\\le \\Psi(x_k,y^*_k) - \\Psi(x_k,y_m^*) + \\frac{1}{2\\eta_y} \\Big(\\Vert y^*_m - y_{k-1} \\Vert^2 - \\Vert y^*_m - y_k \\Vert^2 \\Big).\n    \\end{aligned}\n  \\end{equation}\n","sha256_utf8":"d580206e99dd562be5e8183e31ad94dbeaf42204a67cf89304d21bcd006b37d7"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"插入参考原始点分解","source_units":[{"start":54645,"end":55353,"start_line":841,"end_line":850,"raw_tex":"  Due to the Lipschitz continuity of $\\Phi$, terms which only differ in their first argument will be easy to estimate. Therefore, we insert and subtract $\\Phi(x_m,y_{k}^*)$ to deduce\n  \\begin{equation}\n    \\label{eq:plus-minus-functionval}\n    \\begin{aligned}\n      \\MoveEqLeft \\Psi(x_k, y^*_k) - \\Psi(x_k, y^*_m)  \\\\ %&= \\Phi(x_k, y^*_k)-h(y^*_k) - \\Phi(x_k, y^*_m)+h(y^*_m) \\\\\n      &= \\Phi(x_k, y^*_k) - \\Phi(x_m, y^*_k) + \\Phi(x_m, y^*_k)-h(y^*_k) - \\Phi(x_k, y^*_m) + h(y^*_m) \\\\\n      &\\le \\Phi(x_k, y^*_k) - \\Phi(x_m, y^*_k) + \\Phi(x_m, y^*_m)-h(y^*_m) - \\Phi(x_k, y^*_m) + h(y^*_m)\\\\\n      &= \\Phi(x_k,y^*_k) - \\Phi(x_m,y^*_k) + \\Phi(x_m,y^*_m) - \\Phi(x_k, y^*_m).\n    \\end{aligned}\n  \\end{equation}\n","sha256_utf8":"692fdad9d469a7b02c165bcecf7d99a0438f40ba26b5189399175a7656a90207"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"近端位移控制函数差","source_units":[{"start":55353,"end":56126,"start_line":851,"end_line":864,"raw_tex":"  We estimate the above expression for $k>m$ by making use of the Lipschitz continuity of $\\Phi(\\cdot, y)$ and~\\eqref{eq:bound-prox-f} deducing\n  \\begin{equation}\n    \\label{eq:fun-val-same-y}\n    \\begin{aligned}\n      \\MoveEqLeft \\Phi(x_k, y^*_k) - \\Phi(x_m, y^*_k)  \\le L \\Vert x_k - x_m \\Vert \\le L \\sum_{l=m}^{k-1} \\Vert x_{l+1}-x_l \\Vert \\\\\n      &\\le L \\sum_{l=m}^{k-1} \\Big( \\Vert \\prox{\\eta_x f}{x_l -\\eta_x g_l} - \\prox{\\eta_x f}{x_l} \\Vert + \\Vert \\prox{\\eta_x f}{x_l} - x_l \\Vert \\Big)\\\\\n      & \\le \\eta_x L(L+L_f)(k-m).\n    \\end{aligned}\n  \\end{equation}\n  For $k=m$ the inequality follows trivially. Analogously, we deduce\n  \\begin{equation}\n    \\label{eq:fun-val-same-y-2}\n      \\Phi(x_m, y^*_m) - \\Phi(x_k, y^*_m) \\le \\eta_x L(L+L_f)(k-m).\n  \\end{equation}\n","sha256_utf8":"0e17c55a82115a09b0eea86ec1ee113d691bde5a0861c09af148437bd0bd3a9e"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"合并确定性间隙估计","source_units":[{"start":56126,"end":56301,"start_line":865,"end_line":865,"raw_tex":"  Plugging~\\eqref{eq:plus-minus-functionval},~\\eqref{eq:fun-val-same-y} and~\\eqref{eq:fun-val-same-y-2} into~\\eqref{eq:diff-function-values} gives the statement of the lemma.\n","sha256_utf8":"fec2f4d9bda8dd2c4c4daea470e860a259ab51b4a359235b2bbc5796d20130da"}]},{"id":"S050","semantic_type":"lemma","kind":"lemma","title":"确定性平均间隙界","source_units":[{"start":56587,"end":56873,"start_line":870,"end_line":877,"raw_tex":"\\begin{lemma}%\n  \\label{lem:estimate-sum-of-delta}\n  It holds that for all $K\\ge1$\n  \\begin{equation}\n    \\label{eq:estimate-sum-of-delta}\n    \\frac{1}{K} \\sum_{k=0}^{K-1} \\Delta_k \\le \\eta_x L(L+L_f) B + \\frac{L_{\\nabla\\Phi}D_h^2}{2B} + \\frac{\\Delta_0}{K}.\n  \\end{equation}\n\\end{lemma}","sha256_utf8":"1283100176363cbabe622b34f8906131e37a562f9b1ed160531520e714620fa9"}]},{"id":"S051","semantic_type":"proof","kind":"proof","title":"确定性分块求和证明","source_units":[{"start":56874,"end":59158,"start_line":878,"end_line":917,"raw_tex":"\\begin{proof}\n  By splitting the summation into blocks we get that\n  \\begin{equation}\n    \\label{eq:split--summation-into-blocks}\n    \\sum_{k=0}^{K-1} \\Delta_k  = \\sum_{j=0}^{K/B-1} \\ \\sum_{k=jB}^{(j+1)B-1} \\Delta_k.\n  \\end{equation}\n  By using~\\eqref{eq:estimate-Delta} from Lemma~\\ref{lem:estimate-Delta} with $j>0$ and $m=jB$ and the fact that $\\sum_{k=1}^{B-1} k \\le \\nicefrac{B^2}{2}$ we have\n  \\begin{equation}\n    \\label{eq:sum-over-any-block}\n    \\begin{aligned}\n      \\sum_{k=jB}^{(j+1)B-1} \\Delta_k &\\le \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y} \\Vert y_{jB-1}-y^*_{jB} \\Vert^2 \\\\\n      % &\\le \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y}D_h^2,\n    \\end{aligned}\n  \\end{equation}\n  where the last term can be bounded by $D_h^2$, which was defined in Assumption~\\ref{ass:regularizers} and denotes the diameter of $\\dom h$.\n  We do the same for the case $j=0$ but choose here $m=1$ and have separate out the first summand of $\\sum_{k=0}^{B-1} \\Delta_k$ as Lemma~\\ref{lem:estimate-Delta} does not hold for $k=0$ and therefore get an extra $\\Delta_0$ summand.\n  % with $m=1$ and the fact that $\\sum_{k=1}^{B-1} k \\le \\nicefrac{B^2}{2}$ we get that\n  % \\begin{equation}\n  %   \\label{eq:sum-first-block}\n  %   \\begin{aligned}\n  %     \\sum_{k=0}^{B-1} \\Delta_k &\\le \\Delta_0 + \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y} \\Vert y_{0}-y^*_{1} \\Vert^2 \\\\\n  %     &\\le \\Delta_0 + \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y}D_h^2,\n  %   \\end{aligned}\n  % \\end{equation}\n  where $D_h$ was defined in Assumption~\\ref{ass:regularizers} and denotes the diameter of $\\dom h$.\n  % Analogously, for $j>0$ and $m=jB$ we have that\n  % \\begin{equation}\n  %   \\label{eq:sum-over-any-block}\n  %   \\begin{aligned}\n  %     \\sum_{k=jB}^{(j+1)B-1} \\Delta_k &\\le \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y} \\Vert y_{jB-1}-y^*_{jB} \\Vert^2 \\\\\n  %     &\\le \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y}D_h^2.\n  %   \\end{aligned}\n  % \\end{equation}\n  Plugging~\\eqref{eq:sum-over-any-block} into~\\eqref{eq:split--summation-into-blocks} gives\n  \\begin{equation*}\n    % \\label{eq:sum-over-all}\n    \\frac{1}{K} \\sum_{k=0}^{K-1} \\Delta_k \\le \\eta_x L(L+L_f) B + \\frac{1}{2\\eta_y B}D_h^2 + \\frac{\\Delta_0}{K}.\n  \\end{equation*}\n  The desired statement is obtained by using the step size $\\eta_y=\\nicefrac{1}{L_{\\nabla\\Phi}}$.\n\\end{proof}","sha256_utf8":"aad8a35b02681a34c6f2667802da0e177567d9ebf20c6238d7e7e47c8ac1fad2"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"完整求和分解为块","source_units":[{"start":56888,"end":57108,"start_line":879,"end_line":883,"raw_tex":"  By splitting the summation into blocks we get that\n  \\begin{equation}\n    \\label{eq:split--summation-into-blocks}\n    \\sum_{k=0}^{K-1} \\Delta_k  = \\sum_{j=0}^{K/B-1} \\ \\sum_{k=jB}^{(j+1)B-1} \\Delta_k.\n  \\end{equation}\n","sha256_utf8":"cb2f08b74a52ab678253083cc352b3db265ce43316cf6a9272cbbce59cd66c75"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"普通块望远镜估计","source_units":[{"start":57108,"end":57699,"start_line":884,"end_line":892,"raw_tex":"  By using~\\eqref{eq:estimate-Delta} from Lemma~\\ref{lem:estimate-Delta} with $j>0$ and $m=jB$ and the fact that $\\sum_{k=1}^{B-1} k \\le \\nicefrac{B^2}{2}$ we have\n  \\begin{equation}\n    \\label{eq:sum-over-any-block}\n    \\begin{aligned}\n      \\sum_{k=jB}^{(j+1)B-1} \\Delta_k &\\le \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y} \\Vert y_{jB-1}-y^*_{jB} \\Vert^2 \\\\\n      % &\\le \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y}D_h^2,\n    \\end{aligned}\n  \\end{equation}\n  where the last term can be bounded by $D_h^2$, which was defined in Assumption~\\ref{ass:regularizers} and denotes the diameter of $\\dom h$.\n","sha256_utf8":"0765f4a880e41f540520ecbb2ac9488e5831bf22c154f3877075b13acc1b179f"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"首块保留初始间隙","source_units":[{"start":57699,"end":58776,"start_line":893,"end_line":910,"raw_tex":"  We do the same for the case $j=0$ but choose here $m=1$ and have separate out the first summand of $\\sum_{k=0}^{B-1} \\Delta_k$ as Lemma~\\ref{lem:estimate-Delta} does not hold for $k=0$ and therefore get an extra $\\Delta_0$ summand.\n  % with $m=1$ and the fact that $\\sum_{k=1}^{B-1} k \\le \\nicefrac{B^2}{2}$ we get that\n  % \\begin{equation}\n  %   \\label{eq:sum-first-block}\n  %   \\begin{aligned}\n  %     \\sum_{k=0}^{B-1} \\Delta_k &\\le \\Delta_0 + \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y} \\Vert y_{0}-y^*_{1} \\Vert^2 \\\\\n  %     &\\le \\Delta_0 + \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y}D_h^2,\n  %   \\end{aligned}\n  % \\end{equation}\n  where $D_h$ was defined in Assumption~\\ref{ass:regularizers} and denotes the diameter of $\\dom h$.\n  % Analogously, for $j>0$ and $m=jB$ we have that\n  % \\begin{equation}\n  %   \\label{eq:sum-over-any-block}\n  %   \\begin{aligned}\n  %     \\sum_{k=jB}^{(j+1)B-1} \\Delta_k &\\le \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y} \\Vert y_{jB-1}-y^*_{jB} \\Vert^2 \\\\\n  %     &\\le \\eta_x L(L+L_f) B^2 + \\frac{1}{2\\eta_y}D_h^2.\n  %   \\end{aligned}\n  % \\end{equation}\n","sha256_utf8":"bf037dfd9e219932e87e8836c42b023e1a1cf4f4dea13411d8a24f1434c43a70"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"合并各块与代入步长","source_units":[{"start":58776,"end":59147,"start_line":911,"end_line":916,"raw_tex":"  Plugging~\\eqref{eq:sum-over-any-block} into~\\eqref{eq:split--summation-into-blocks} gives\n  \\begin{equation*}\n    % \\label{eq:sum-over-all}\n    \\frac{1}{K} \\sum_{k=0}^{K-1} \\Delta_k \\le \\eta_x L(L+L_f) B + \\frac{1}{2\\eta_y B}D_h^2 + \\frac{\\Delta_0}{K}.\n  \\end{equation*}\n  The desired statement is obtained by using the step size $\\eta_y=\\nicefrac{1}{L_{\\nabla\\Phi}}$.\n","sha256_utf8":"1fe60baa7dee23a6b148a157e8046f94a3ca276df2ee17705af033cb5ceb218e"}]},{"id":"S056","semantic_type":"lemma","kind":"lemma","title":"随机包络期望下降","source_units":[{"start":62924,"end":63337,"start_line":975,"end_line":982,"raw_tex":"\\begin{lemma}%\n  \\label{lem:stoch-descent-on-moreau}\n  With $\\lambda = \\nicefrac{1}{2\\rho}$ we have for all $k\\ge0$ that\n  \\begin{equation*}\n    \\Exp{\\psi_\\lambda(x_{k+1})} \\le \\Exp{\\psi_\\lambda(x_k)} + 2\\rho\\eta_x\\hat{\\Delta}_k - \\frac{\\eta_x}{2}\\Exp{\\Vert \\nabla \\psi_\\lambda(x_k) \\Vert^2} + 4\\rho\\eta_x^2(L^2+\\sigma^2)\n  \\end{equation*}\n  where $\\hat{\\Delta}_k := \\Exp{\\psi(x_k) - \\Psi(x_k, y_k)}$.\n\\end{lemma}","sha256_utf8":"09766523c8f38e3bbfdc0a1b5d8183d0bf728baacd1035d9d07a7516aad89c9a"}]},{"id":"S057","semantic_type":"proof","kind":"proof","title":"随机包络下降证明","source_units":[{"start":63338,"end":65170,"start_line":983,"end_line":1010,"raw_tex":"\\begin{proof}\n  Let $k\\ge0$ be arbitrary but fixed. It follows easily from~\\eqref{eq:bounded-variance-subgradient} that\n  \\begin{equation}\n    \\label{eq:steiner}\n    \\Exp{\\Vert g_k^\\xi \\Vert^2} \\le \\Exp{\\Vert g_k \\Vert^2} + \\sigma^2 \\le L^2 + \\sigma^2,\n  \\end{equation}\n  where $\\Exp{g_k^\\xi}=g_k \\in \\partial_x\\Phi(x_k,y_k)$.\n  The definition of the Moreau envelope yields\n  \\begin{equation}\n    \\label{eq:defin-moreau}\n    \\Exp{g_\\lambda(x_{k+1})} \\le \\Exp{\\psi(\\hat{x}_k)} + \\frac{1}{2\\lambda}\\Exp{\\Vert \\hat{x}_k - x_{k+1} \\Vert^2}.\n  \\end{equation}\n  Similarly to Lemma~\\ref{lem:descent-on-moreau} we deduce that for $v_k\\in \\partial\\phi(\\hat{x}_k)$ (as given in Lemma~\\ref{lem:xhatk-as-prox}) and $\\beta=1-\\eta_x \\lambda^{-1}$\n  \\begin{equation*}\n    \\begin{aligned}\n      \\Vert\\hat{x}_k - x_{k+1} \\Vert^2 %&= \\Vert \\prox{\\eta_x f}{\\eta_x\\lambda^{-1}x_k - \\eta_x v_k + \\beta\\hat{x}_k} - \\prox{\\eta_x f}{x_k - \\eta_x g_k^\\xi}\\Vert^2 \\\\\n       %&\\le \\Vert \\beta(\\hat{x}_k - x_k) + \\eta_x (g_k^\\xi - v_k)\\Vert^2 \\\\\n       &= {\\beta}^2\\Vert \\hat{x}_k - x_k \\Vert^2 + 2 \\eta_x \\beta\\langle g_k^\\xi - v_k, \\hat{x}_k - x_k\\rangle\n       + \\eta_x^2\\Vert g_k^\\xi - v_k \\Vert^2.\n    \\end{aligned}\n  \\end{equation*}\n  By applying the conditional expectation $\\Expcond{\\cdot}{x_k,y_k}$, then the unconditional one and using~\\eqref{eq:steiner}, we get that\n  \\begin{equation*}\n    \\Exp{\\Vert \\hat{x}_k - x_{k+1} \\Vert^2} \\le \\Exp{\\Vert \\hat{x}_k - x_k \\Vert^2 + 2 \\eta_x(1-\\frac{\\eta_x}{\\lambda}) \\langle g_k-v_k, \\hat{x}_k - x_k\\rangle} + 4\\eta_x^2(L^2+\\sigma^2).\n  \\end{equation*}\n  where $g_k= \\Exp{g_k^\\xi}$.\n  Lastly, we combine the above inequality with~\\eqref{eq:defin-moreau} and the estimate for the inner product~\\eqref{eq:estimate-middle} as in Lemma~\\ref{lem:descent-on-moreau} to deduce the statement of the lemma.\n\\end{proof}","sha256_utf8":"3e3abb9fff0466cf62c050d4cebb082665533fa9830ff930cfee73e65150e286"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"随机次梯度二阶矩界","source_units":[{"start":63352,"end":63665,"start_line":984,"end_line":989,"raw_tex":"  Let $k\\ge0$ be arbitrary but fixed. It follows easily from~\\eqref{eq:bounded-variance-subgradient} that\n  \\begin{equation}\n    \\label{eq:steiner}\n    \\Exp{\\Vert g_k^\\xi \\Vert^2} \\le \\Exp{\\Vert g_k \\Vert^2} + \\sigma^2 \\le L^2 + \\sigma^2,\n  \\end{equation}\n  where $\\Exp{g_k^\\xi}=g_k \\in \\partial_x\\Phi(x_k,y_k)$.\n","sha256_utf8":"5b27989839fe66701af31e3e8bc27ffc4cf1455f5c97bcf596d82edd2bb846a2"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"期望包络的辅助点界","source_units":[{"start":63665,"end":63892,"start_line":990,"end_line":994,"raw_tex":"  The definition of the Moreau envelope yields\n  \\begin{equation}\n    \\label{eq:defin-moreau}\n    \\Exp{g_\\lambda(x_{k+1})} \\le \\Exp{\\psi(\\hat{x}_k)} + \\frac{1}{2\\lambda}\\Exp{\\Vert \\hat{x}_k - x_{k+1} \\Vert^2}.\n  \\end{equation}\n","sha256_utf8":"174adaf30151260747516b48ee15f79efceed76aa5e4b0e3ca5a989af9711a81"}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","title":"随机近端距离展开","source_units":[{"start":63892,"end":64549,"start_line":995,"end_line":1003,"raw_tex":"  Similarly to Lemma~\\ref{lem:descent-on-moreau} we deduce that for $v_k\\in \\partial\\phi(\\hat{x}_k)$ (as given in Lemma~\\ref{lem:xhatk-as-prox}) and $\\beta=1-\\eta_x \\lambda^{-1}$\n  \\begin{equation*}\n    \\begin{aligned}\n      \\Vert\\hat{x}_k - x_{k+1} \\Vert^2 %&= \\Vert \\prox{\\eta_x f}{\\eta_x\\lambda^{-1}x_k - \\eta_x v_k + \\beta\\hat{x}_k} - \\prox{\\eta_x f}{x_k - \\eta_x g_k^\\xi}\\Vert^2 \\\\\n       %&\\le \\Vert \\beta(\\hat{x}_k - x_k) + \\eta_x (g_k^\\xi - v_k)\\Vert^2 \\\\\n       &= {\\beta}^2\\Vert \\hat{x}_k - x_k \\Vert^2 + 2 \\eta_x \\beta\\langle g_k^\\xi - v_k, \\hat{x}_k - x_k\\rangle\n       + \\eta_x^2\\Vert g_k^\\xi - v_k \\Vert^2.\n    \\end{aligned}\n  \\end{equation*}\n","sha256_utf8":"b272709de8597c7d5e02b514442c4cd230f069721fe78f09116f6289a9bbd274"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"条件期望消除噪声","source_units":[{"start":64549,"end":64944,"start_line":1004,"end_line":1008,"raw_tex":"  By applying the conditional expectation $\\Expcond{\\cdot}{x_k,y_k}$, then the unconditional one and using~\\eqref{eq:steiner}, we get that\n  \\begin{equation*}\n    \\Exp{\\Vert \\hat{x}_k - x_{k+1} \\Vert^2} \\le \\Exp{\\Vert \\hat{x}_k - x_k \\Vert^2 + 2 \\eta_x(1-\\frac{\\eta_x}{\\lambda}) \\langle g_k-v_k, \\hat{x}_k - x_k\\rangle} + 4\\eta_x^2(L^2+\\sigma^2).\n  \\end{equation*}\n  where $g_k= \\Exp{g_k^\\xi}$.\n","sha256_utf8":"dc3814ce3564257f4e85d2e5579a69ee79f386335c94f5ea052480595a242b6c"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"沿用弱凸内积与下降推导","source_units":[{"start":64944,"end":65159,"start_line":1009,"end_line":1009,"raw_tex":"  Lastly, we combine the above inequality with~\\eqref{eq:defin-moreau} and the estimate for the inner product~\\eqref{eq:estimate-middle} as in Lemma~\\ref{lem:descent-on-moreau} to deduce the statement of the lemma.\n","sha256_utf8":"d457894c3898d42863b7426f2bc31b62bf396fdd95853b05445c316317cbec85"}]},{"id":"S063","semantic_type":"lemma","kind":"lemma","title":"随机近端上升比较","source_units":[{"start":65424,"end":65904,"start_line":1014,"end_line":1023,"raw_tex":"\\begin{lemma}%\n  \\label{lem:stoch-standard-gradient-ascent-estimation}\n  With $\\eta_y\\le \\nicefrac{1}{2L_{\\nabla\\Phi}}$ we have for all $k\\ge0$ and all $y \\in \\R^n$\n  \\begin{equation}\n    \\label{eq:stoch-gradient-ascent-estimation}\n    \\begin{aligned}\n      \\MoveEqLeft \\Exp{\\Psi(x_{k+1}, y) - \\Psi(x_{k+1}, y_{k+1})} \\le  \\frac{1}{2\\eta_y} \\Big(\\Exp{\\Vert y^*_m - y_k \\Vert^2 - \\Vert y^*_m - y_{k+1} \\Vert^2} \\Big) + \\eta_y\\sigma^2.\n    \\end{aligned}\n  \\end{equation}\n\\end{lemma}","sha256_utf8":"87bf85169b684b0a1f3d7efcef77295bf932097c586c99c6408b87abc574bc28"}]},{"id":"S064","semantic_type":"proof","kind":"proof","title":"随机上升比较证明","source_units":[{"start":65905,"end":69020,"start_line":1024,"end_line":1065,"raw_tex":"\\begin{proof}\n  Let $k\\ge0$ and $y\\in\\R^n$ be arbitrary but fixed.\n  % By the definition of $y_{k+1}$ we have that\n  % \\begin{equation*} y_{k+1} = \\argmin_{y\\in\\R^n} \\Big\\{ h(y) - \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k), y-y_k\\rangle + \\frac{1}{2\\eta_y}\\Vert y-y_k \\Vert^2 \\Big\\}.\n  % \\end{equation*}\n  As in Lemma~\\ref{lem:standard-gradient-ascent-estimation}, we deduce\n  \\begin{equation*}\n    % \\label{eq:stoch-decrease}\n    \\begin{aligned}\n      h(y_{k+1}) - \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k), y_{k+1}-y_k \\rangle + \\frac{1}{2\\eta_y} \\Big(\\Vert y_{k+1} -y_k \\Vert^2 + \\Vert y-y_{k+1} \\Vert^2 \\Big)  \\\\\n      \\le h(y) - \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k), y-y_k \\rangle + \\frac{1}{2\\eta_y} \\Vert y-y_k \\Vert^2.\n    \\end{aligned}\n  \\end{equation*}\n  The term $\\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k), y_{k+1}-y_k \\rangle$ is problematic, because the right~hand side of the inner product is not measurable with respect to the sigma algebra generated by past iterates $\\mathcal{F}_k := \\sigma\\{x_{k+1},\\dots, x_1, y_k, \\dots, y_1\\}$, so we insert and subtract $\\langle \\nabla_y\\Phi(x_{k+1},y_k) , y_{k+1}-y_k\\rangle$\n  % to deduce\n  % \\begin{multline*}\n  %     h(y_{k+1}) + \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k),y-y_k\\rangle + \\frac{1}{2\\eta_y} \\Vert y_{k+1} -y_k \\Vert^2 + \\frac{1}{2\\eta_y} \\Vert y-y_{k+1} \\Vert^2 \\\\\n  %     \\le h(y) + \\langle \\nabla_y\\Phi(x_{k+1},y_k), y_{k+1}-y_k \\rangle  + \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k) - \\nabla_y\\Phi(x_{k+1},y_k), y_{k+1}-y_k \\rangle + \\frac{1}{2\\eta_y} \\Vert y-y_k \\Vert^2.\n  % \\end{multline*}\n  Now, using Young's inequality we estimate the resulting inner product\n  \\begin{multline*}\n    % \\label{eq:young2}\n    \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k) - \\nabla_y\\Phi(x_{k+1},y_k), y_{k+1}-y_k \\rangle \\\\\n    \\le \\eta_y\\Vert \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k) - \\nabla_{y}\\Phi(x_{k+1},y_k) \\Vert^2 + \\frac{1}{4\\eta_y} \\Vert y_{k+1} - y_k \\Vert^2.\n  \\end{multline*}\n  Combining the above two inequalities for $y=y^*_m$ with $1\\le m\\le k$ and taking the expectation together with the bounded variance assumption~\\eqref{eq:bounded-variance} gives\n  \\begin{equation}\n    \\label{eq:ab1}\n    \\begin{aligned}\n    \\MoveEqLeft \\Exp{\\Expcond{\\langle \\nabla_{y}\\Phi(x_{k+1},y_k;\\xi_k), y^*_m-y_k \\rangle}{\\mathcal{F}_k}}  + \\Exp{h(y_{k+1})+ \\frac{1}{2\\eta_y} \\Vert y^*_m-y_{k+1} \\Vert^2}\\\\\n    &\\le  \\Exp{\\langle \\nabla_{y}\\Phi(x_{k+1},y_k), y_{k+1}-y_k \\rangle - \\frac{1}{4\\eta_y} \\Vert y_{k+1} -y_k \\Vert^2}\n    \\\\\n    &\\quad+ \\eta_y \\sigma^2+ \\Exp{h(y^*_m) +\\frac{1}{2\\eta_y} \\Vert y^*_m-y_k \\Vert^2}.\n    \\end{aligned}\n  \\end{equation}\n  From the descent lemma (in ascent form) and the fact that $ \\eta_y \\le  \\nicefrac{1}{2L_{\\nabla\\Phi}}$ we have\n  \\begin{equation*}\n    % \\label{eq:stoch-ascent-lemma}\n    \\Phi(x_{k+1}, y_k) + \\langle y_{k+1} - y_k, \\nabla_y\\Phi(x_{k+1}, y_k)\\rangle - \\frac{1}{4\\eta_y} \\Vert y_{k+1} - y_k \\Vert^2 \\le \\Phi(x_{k+1}, y_{k+1}).\n  \\end{equation*}\n  We plug the above inequality into~\\eqref{eq:ab1}, make use of the concavity and add $f(x_{k+1})$ on both sides to deduce the statement of the lemma.\n\\end{proof}","sha256_utf8":"4888d0adb8d9b7162f6c4fef9524cbb3e8e61a5c05f7f9e62024ad40d8151103"}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","title":"近端最优性与可测性分析","source_units":[{"start":65919,"end":67041,"start_line":1025,"end_line":1037,"raw_tex":"  Let $k\\ge0$ and $y\\in\\R^n$ be arbitrary but fixed.\n  % By the definition of $y_{k+1}$ we have that\n  % \\begin{equation*} y_{k+1} = \\argmin_{y\\in\\R^n} \\Big\\{ h(y) - \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k), y-y_k\\rangle + \\frac{1}{2\\eta_y}\\Vert y-y_k \\Vert^2 \\Big\\}.\n  % \\end{equation*}\n  As in Lemma~\\ref{lem:standard-gradient-ascent-estimation}, we deduce\n  \\begin{equation*}\n    % \\label{eq:stoch-decrease}\n    \\begin{aligned}\n      h(y_{k+1}) - \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k), y_{k+1}-y_k \\rangle + \\frac{1}{2\\eta_y} \\Big(\\Vert y_{k+1} -y_k \\Vert^2 + \\Vert y-y_{k+1} \\Vert^2 \\Big)  \\\\\n      \\le h(y) - \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k), y-y_k \\rangle + \\frac{1}{2\\eta_y} \\Vert y-y_k \\Vert^2.\n    \\end{aligned}\n  \\end{equation*}\n  The term $\\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k), y_{k+1}-y_k \\rangle$ is problematic, because the right~hand side of the inner product is not measurable with respect to the sigma algebra generated by past iterates $\\mathcal{F}_k := \\sigma\\{x_{k+1},\\dots, x_1, y_k, \\dots, y_1\\}$, so we insert and subtract $\\langle \\nabla_y\\Phi(x_{k+1},y_k) , y_{k+1}-y_k\\rangle$\n","sha256_utf8":"a21c5205451ee76dfcfd3cbfadf71baaa34eb6a9937bbcc98fec1fceff7ba3f4"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"杨氏不等式分离梯度误差","source_units":[{"start":67041,"end":67849,"start_line":1038,"end_line":1048,"raw_tex":"  % to deduce\n  % \\begin{multline*}\n  %     h(y_{k+1}) + \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k),y-y_k\\rangle + \\frac{1}{2\\eta_y} \\Vert y_{k+1} -y_k \\Vert^2 + \\frac{1}{2\\eta_y} \\Vert y-y_{k+1} \\Vert^2 \\\\\n  %     \\le h(y) + \\langle \\nabla_y\\Phi(x_{k+1},y_k), y_{k+1}-y_k \\rangle  + \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k) - \\nabla_y\\Phi(x_{k+1},y_k), y_{k+1}-y_k \\rangle + \\frac{1}{2\\eta_y} \\Vert y-y_k \\Vert^2.\n  % \\end{multline*}\n  Now, using Young's inequality we estimate the resulting inner product\n  \\begin{multline*}\n    % \\label{eq:young2}\n    \\langle \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k) - \\nabla_y\\Phi(x_{k+1},y_k), y_{k+1}-y_k \\rangle \\\\\n    \\le \\eta_y\\Vert \\nabla_y\\Phi(x_{k+1},y_k;\\zeta_k) - \\nabla_{y}\\Phi(x_{k+1},y_k) \\Vert^2 + \\frac{1}{4\\eta_y} \\Vert y_{k+1} - y_k \\Vert^2.\n  \\end{multline*}\n","sha256_utf8":"f42efbc6aa7f02592cfbe46649d7676aefddff7bd03c025bcbbbbee21d4bbf0e"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"条件期望与方差控制","source_units":[{"start":67849,"end":68513,"start_line":1049,"end_line":1058,"raw_tex":"  Combining the above two inequalities for $y=y^*_m$ with $1\\le m\\le k$ and taking the expectation together with the bounded variance assumption~\\eqref{eq:bounded-variance} gives\n  \\begin{equation}\n    \\label{eq:ab1}\n    \\begin{aligned}\n    \\MoveEqLeft \\Exp{\\Expcond{\\langle \\nabla_{y}\\Phi(x_{k+1},y_k;\\xi_k), y^*_m-y_k \\rangle}{\\mathcal{F}_k}}  + \\Exp{h(y_{k+1})+ \\frac{1}{2\\eta_y} \\Vert y^*_m-y_{k+1} \\Vert^2}\\\\\n    &\\le  \\Exp{\\langle \\nabla_{y}\\Phi(x_{k+1},y_k), y_{k+1}-y_k \\rangle - \\frac{1}{4\\eta_y} \\Vert y_{k+1} -y_k \\Vert^2}\n    \\\\\n    &\\quad+ \\eta_y \\sigma^2+ \\Exp{h(y^*_m) +\\frac{1}{2\\eta_y} \\Vert y^*_m-y_k \\Vert^2}.\n    \\end{aligned}\n  \\end{equation}\n","sha256_utf8":"e9b2ef0ae174ab08c58aebad7382b796ee6417e55b89d618996e2b825663cb41"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"光滑上升界与凹性合并","source_units":[{"start":68513,"end":69009,"start_line":1059,"end_line":1064,"raw_tex":"  From the descent lemma (in ascent form) and the fact that $ \\eta_y \\le  \\nicefrac{1}{2L_{\\nabla\\Phi}}$ we have\n  \\begin{equation*}\n    % \\label{eq:stoch-ascent-lemma}\n    \\Phi(x_{k+1}, y_k) + \\langle y_{k+1} - y_k, \\nabla_y\\Phi(x_{k+1}, y_k)\\rangle - \\frac{1}{4\\eta_y} \\Vert y_{k+1} - y_k \\Vert^2 \\le \\Phi(x_{k+1}, y_{k+1}).\n  \\end{equation*}\n  We plug the above inequality into~\\eqref{eq:ab1}, make use of the concavity and add $f(x_{k+1})$ on both sides to deduce the statement of the lemma.\n","sha256_utf8":"a09e3ec2fa36f372566bd79435622b603c57bfdaefc905b47e408f0e48087ce9"}]},{"id":"S069","semantic_type":"lemma","kind":"lemma","title":"随机对偶间隙界","source_units":[{"start":69087,"end":69466,"start_line":1069,"end_line":1078,"raw_tex":"\\begin{lemma}%\n  \\label{lem:stoch-estimate-Delta}\n  For all $1 \\le m \\le k$, we have that\n  \\begin{equation}\n    \\label{eq:stoch-estimate-Delta}\n    \\begin{aligned}\n      \\MoveEqLeft \\hat{\\Delta}_k \\le 2\\eta_x L(L_f+L+\\sigma)(k-m) + \\frac{1}{2\\eta_y} \\Exp{\\Vert y_{k-1} - y^*_m \\Vert^2 - \\Vert y_k - y^*_m \\Vert^2} + \\eta_y\\sigma^2.\n    \\end{aligned}\n  \\end{equation}\n\\end{lemma}","sha256_utf8":"58a8211a03221d7e6dd03ced7a8da90c804c1ec83569aa1df798bd83f1c2e42c"}]},{"id":"S070","semantic_type":"proof","kind":"proof","title":"随机间隙估计证明","source_units":[{"start":69467,"end":71266,"start_line":1079,"end_line":1110,"raw_tex":"\\begin{proof}\n  Let the numbers $1 \\le m \\le k$ be fixed.\n  Starting from the definition of $\\hat{\\Delta}_k$, we add~\\eqref{eq:stoch-gradient-ascent-estimation} to obtain\n  % NOTE: Consider using $k\\ge1$ and get rid of the +1 everywhere\n  \\begin{equation}\n    \\label{eq:diff-function-values-stoch}\n    \\begin{aligned}\n      \\hat{\\Delta}_k &= \\Exp{\\psi(x_k) - \\Psi(x_k,y_k)} \\\\\n      &\\le \\Exp{\\psi(x_k) - \\Psi(x_k, y^*_m)} + \\frac{1}{2\\eta_y} \\Big(\\Exp{\\Vert y^*_m - y_{k-1} \\Vert^2 - \\Vert y^*_m - y_k \\Vert^2} \\Big)+ \\eta_y\\sigma^2.\n    \\end{aligned}\n  \\end{equation}\n  As in~\\eqref{eq:plus-minus-functionval} we deduce that\n  \\begin{equation*}\n    % \\label{eq:stoch-plus-minus}\n    \\Psi(x_k, y^*_k) - \\Psi(x_k, y^*_m) \\le \\Phi(x_k,y_k^*) -\\Phi(x_m,y_k^*) +\\Phi(x_m,y_m^*) -\\Phi(x_k,y_m^*).\n  \\end{equation*}\n  Together with the $L$-Lipschitz continuity of $\\Phi(\\cdot, y)$ and~\\eqref{eq:bound-prox-f} we estimate for $k\\ge m$ as in~\\eqref{eq:fun-val-same-y}, but using $\\Exp{\\Vert g_l^\\xi \\Vert}\\le L + \\sigma$,\n  \\begin{equation*}\n    \\begin{aligned}\n      \\Exp{\\Phi(x_k, y^*_k) - \\Phi(x_m, y^*_k)}\n      %&\\le L \\Exp{\\Vert x_k-x_m \\Vert} \\le L \\sum_{l=m}^{k-1}\\Exp{\\Vert x_{l+1} - x_l \\Vert} \\\\\n      %&\\le L \\sum_{l=m}^{k-1} \\Exp{\\Vert \\prox{\\eta_x f}{x_l -\\eta_x g_l^\\xi} - \\prox{\\eta_x f}{x_l} \\Vert+ \\Vert \\prox{\\eta_x f}{x_l} - x_l \\Vert} \\\\\n      &\\le \\eta_x L\\Big(L_f+\\sqrt{L^2+\\sigma^2}\\Big)(k-m).\n    \\end{aligned}\n  \\end{equation*}\n  % For $k=m$ the statement follows trivially.\n  % Analogously,\n  % \\begin{equation*}\n  %   \\Exp{\\Phi(x_m, y^*_m) - \\Phi(x_k, y^*_m)} \\le L \\Exp{\\Vert x_k - x_m \\Vert} \\le \\eta_x L\\Big(L_f+\\sqrt{L^2+\\sigma^2}\\Big)(k-m).\n  % \\end{equation*}\n  Plugging all of these into~\\eqref{eq:diff-function-values-stoch} gives the statement of the lemma.\n\\end{proof}","sha256_utf8":"721cd4fd078a6dc2193cd6431c9e659108437237089e0c164c95771c05244e04"}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","title":"随机上升界加入期望间隙","source_units":[{"start":69481,"end":70037,"start_line":1080,"end_line":1089,"raw_tex":"  Let the numbers $1 \\le m \\le k$ be fixed.\n  Starting from the definition of $\\hat{\\Delta}_k$, we add~\\eqref{eq:stoch-gradient-ascent-estimation} to obtain\n  % NOTE: Consider using $k\\ge1$ and get rid of the +1 everywhere\n  \\begin{equation}\n    \\label{eq:diff-function-values-stoch}\n    \\begin{aligned}\n      \\hat{\\Delta}_k &= \\Exp{\\psi(x_k) - \\Psi(x_k,y_k)} \\\\\n      &\\le \\Exp{\\psi(x_k) - \\Psi(x_k, y^*_m)} + \\frac{1}{2\\eta_y} \\Big(\\Exp{\\Vert y^*_m - y_{k-1} \\Vert^2 - \\Vert y^*_m - y_k \\Vert^2} \\Big)+ \\eta_y\\sigma^2.\n    \\end{aligned}\n  \\end{equation}\n","sha256_utf8":"9a4ad4bf9488ad2135df88ac0919d41b8104550bdbf62d7e76dc67a047f6c7a5"}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","title":"沿用最优点函数差分解","source_units":[{"start":70037,"end":70278,"start_line":1090,"end_line":1094,"raw_tex":"  As in~\\eqref{eq:plus-minus-functionval} we deduce that\n  \\begin{equation*}\n    % \\label{eq:stoch-plus-minus}\n    \\Psi(x_k, y^*_k) - \\Psi(x_k, y^*_m) \\le \\Phi(x_k,y_k^*) -\\Phi(x_m,y_k^*) +\\Phi(x_m,y_m^*) -\\Phi(x_k,y_m^*).\n  \\end{equation*}\n","sha256_utf8":"e072f454eda9aa6461680352e1840b4bda2ef0a06fe9d1e38e63fdb987c00d89"}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","title":"随机位移与函数差期望界","source_units":[{"start":70278,"end":71154,"start_line":1095,"end_line":1108,"raw_tex":"  Together with the $L$-Lipschitz continuity of $\\Phi(\\cdot, y)$ and~\\eqref{eq:bound-prox-f} we estimate for $k\\ge m$ as in~\\eqref{eq:fun-val-same-y}, but using $\\Exp{\\Vert g_l^\\xi \\Vert}\\le L + \\sigma$,\n  \\begin{equation*}\n    \\begin{aligned}\n      \\Exp{\\Phi(x_k, y^*_k) - \\Phi(x_m, y^*_k)}\n      %&\\le L \\Exp{\\Vert x_k-x_m \\Vert} \\le L \\sum_{l=m}^{k-1}\\Exp{\\Vert x_{l+1} - x_l \\Vert} \\\\\n      %&\\le L \\sum_{l=m}^{k-1} \\Exp{\\Vert \\prox{\\eta_x f}{x_l -\\eta_x g_l^\\xi} - \\prox{\\eta_x f}{x_l} \\Vert+ \\Vert \\prox{\\eta_x f}{x_l} - x_l \\Vert} \\\\\n      &\\le \\eta_x L\\Big(L_f+\\sqrt{L^2+\\sigma^2}\\Big)(k-m).\n    \\end{aligned}\n  \\end{equation*}\n  % For $k=m$ the statement follows trivially.\n  % Analogously,\n  % \\begin{equation*}\n  %   \\Exp{\\Phi(x_m, y^*_m) - \\Phi(x_k, y^*_m)} \\le L \\Exp{\\Vert x_k - x_m \\Vert} \\le \\eta_x L\\Big(L_f+\\sqrt{L^2+\\sigma^2}\\Big)(k-m).\n  % \\end{equation*}\n","sha256_utf8":"7351c7ce87df9c556058466d3a6e6379c28dac55d0128dfa2baeaf02952cc290"}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","title":"合并随机间隙估计","source_units":[{"start":71154,"end":71255,"start_line":1109,"end_line":1109,"raw_tex":"  Plugging all of these into~\\eqref{eq:diff-function-values-stoch} gives the statement of the lemma.\n","sha256_utf8":"f8aabb56f8cc3484af22b8fdc2aa5cbefbfd3a5e1bcb3dca3f0897b971e3ba88"}]},{"id":"S075","semantic_type":"lemma","kind":"lemma","title":"随机平均间隙界","source_units":[{"start":71493,"end":71813,"start_line":1114,"end_line":1121,"raw_tex":"\\begin{lemma}%\n  \\label{lem:stoch-estimate-sum-of-delta}\n  We have that for all $K\\ge1$\n  \\begin{equation}\n    \\label{eq:stoch-estimate-sum-of-delta}\n    \\frac{1}{K} \\sum_{k=0}^{K-1} \\hat{\\Delta}_k \\le \\eta_x L(L+L_f+\\sigma) B + \\frac{D_h^2}{2\\eta_y B} + \\eta_y\\sigma^2 + \\frac{\\Delta_0}{K}.\n  \\end{equation}\n\\end{lemma}","sha256_utf8":"ed8d1125594f4720a489dc649c8b7791eccdb03f166c81a4420a2e6117db5c3a"}]},{"id":"S076","semantic_type":"proof","kind":"proof","title":"随机分块求和证明","source_units":[{"start":71814,"end":73521,"start_line":1122,"end_line":1151,"raw_tex":"\\begin{proof}\n  We proceed as in Lemma~\\ref{lem:estimate-sum-of-delta}. By using Lemma~\\ref{lem:stoch-estimate-Delta} we obtain\n  $j>0$ and $m=jB$ we have that\n  \\begin{equation}\n    \\label{eq:stoch--sum-over-any-block}\n    \\begin{aligned}\n      \\sum_{k=jB}^{(j+1)B-1} \\hat{\\Delta}_k %&\\le \\eta_x L(L+L_f+\\sigma)  B^2 + \\frac{1}{2\\eta_y} \\Exp{\\Vert y_{jB-1}-y^*_{jB} \\Vert^2} + B\\eta_y\\sigma^2  \\\\\n      &\\le \\eta_x L(L+L_f+\\sigma) B^2 + \\frac{1}{2\\eta_y}D_h^2 + B\\eta_y \\sigma^2.\n    \\end{aligned}\n  \\end{equation}\n  For $j=0$ we use $m=1$ and do not estimate $\\Delta_0$ but leave it there.\n  % with $m=1$ and the fact that $\\sum_{k=1}^{B-1} k \\le \\nicefrac{B^2}{2}$ we get that\n  % \\begin{equation}\n  %   \\label{eq:stoch--sum-first---block}\n  %   \\begin{aligned}\n  %     \\sum_{k=0}^{B-1} \\hat{\\Delta}_k &\\le \\Delta_0 + \\eta_x L(L+L_f+\\sigma)  B^2 + \\frac{1}{2\\eta_y} \\Exp{\\Vert y_{0}-y^*_{1} \\Vert^2} + B\\eta_y\\sigma^2  \\\\\n  %     &\\le \\Delta_0 + \\eta_x L(L+L_f+\\sigma) B^2 + \\frac{1}{2\\eta_y}D_h^2 + B\\eta_y \\sigma^2,\n  %   \\end{aligned}\n  % \\end{equation}\n  % where $D_h$ denotes the diameter of $\\dom h$, see Assumption~\\ref{ass:regularizers}.\n  % Analogously, for $j>0$ and $m=jB$ we have that\n  % \\begin{equation}\n  %   \\label{eq:stoch--sum-over-any-block}\n  %   \\begin{aligned}\n  %     \\sum_{k=jB}^{(j+1)B-1} \\hat{\\Delta}_k &\\le \\eta_x L(L+L_f+\\sigma)  B^2 + \\frac{1}{2\\eta_y} \\Exp{\\Vert y_{jB-1}-y^*_{jB} \\Vert^2} + B\\eta_y\\sigma^2  \\\\\n  %     &\\le \\eta_x L(L+L_f+\\sigma) B^2 + \\frac{1}{2\\eta_y}D_h^2 + B\\eta_y \\sigma^2,\n  %   \\end{aligned}\n  % \\end{equation}\n  Plugging~\\eqref{eq:stoch--sum-over-any-block} into~\\eqref{eq:split--summation-into-blocks} gives the statement of the lemma.\n\\end{proof}","sha256_utf8":"639c5c95a163db9b95868295b411fdf1c0f8c848c736e3c91d61f1de0d236e9e"}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","title":"普通随机块的间隙估计","source_units":[{"start":71828,"end":72330,"start_line":1123,"end_line":1131,"raw_tex":"  We proceed as in Lemma~\\ref{lem:estimate-sum-of-delta}. By using Lemma~\\ref{lem:stoch-estimate-Delta} we obtain\n  $j>0$ and $m=jB$ we have that\n  \\begin{equation}\n    \\label{eq:stoch--sum-over-any-block}\n    \\begin{aligned}\n      \\sum_{k=jB}^{(j+1)B-1} \\hat{\\Delta}_k %&\\le \\eta_x L(L+L_f+\\sigma)  B^2 + \\frac{1}{2\\eta_y} \\Exp{\\Vert y_{jB-1}-y^*_{jB} \\Vert^2} + B\\eta_y\\sigma^2  \\\\\n      &\\le \\eta_x L(L+L_f+\\sigma) B^2 + \\frac{1}{2\\eta_y}D_h^2 + B\\eta_y \\sigma^2.\n    \\end{aligned}\n  \\end{equation}\n","sha256_utf8":"293acda28fa920b313638a3dba98185a35d3b3cf96c857b495542e9732ac57e1"}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","title":"随机首块保留初始间隙","source_units":[{"start":72330,"end":73383,"start_line":1132,"end_line":1149,"raw_tex":"  For $j=0$ we use $m=1$ and do not estimate $\\Delta_0$ but leave it there.\n  % with $m=1$ and the fact that $\\sum_{k=1}^{B-1} k \\le \\nicefrac{B^2}{2}$ we get that\n  % \\begin{equation}\n  %   \\label{eq:stoch--sum-first---block}\n  %   \\begin{aligned}\n  %     \\sum_{k=0}^{B-1} \\hat{\\Delta}_k &\\le \\Delta_0 + \\eta_x L(L+L_f+\\sigma)  B^2 + \\frac{1}{2\\eta_y} \\Exp{\\Vert y_{0}-y^*_{1} \\Vert^2} + B\\eta_y\\sigma^2  \\\\\n  %     &\\le \\Delta_0 + \\eta_x L(L+L_f+\\sigma) B^2 + \\frac{1}{2\\eta_y}D_h^2 + B\\eta_y \\sigma^2,\n  %   \\end{aligned}\n  % \\end{equation}\n  % where $D_h$ denotes the diameter of $\\dom h$, see Assumption~\\ref{ass:regularizers}.\n  % Analogously, for $j>0$ and $m=jB$ we have that\n  % \\begin{equation}\n  %   \\label{eq:stoch--sum-over-any-block}\n  %   \\begin{aligned}\n  %     \\sum_{k=jB}^{(j+1)B-1} \\hat{\\Delta}_k &\\le \\eta_x L(L+L_f+\\sigma)  B^2 + \\frac{1}{2\\eta_y} \\Exp{\\Vert y_{jB-1}-y^*_{jB} \\Vert^2} + B\\eta_y\\sigma^2  \\\\\n  %     &\\le \\eta_x L(L+L_f+\\sigma) B^2 + \\frac{1}{2\\eta_y}D_h^2 + B\\eta_y \\sigma^2,\n  %   \\end{aligned}\n  % \\end{equation}\n","sha256_utf8":"50e78bbf44d93257fbf7a905a6163417ca2cc9311fb7f26d3e6e2947e22ea12c"}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","title":"按分块恒等式汇总","source_units":[{"start":73383,"end":73510,"start_line":1150,"end_line":1150,"raw_tex":"  Plugging~\\eqref{eq:stoch--sum-over-any-block} into~\\eqref{eq:split--summation-into-blocks} gives the statement of the lemma.\n","sha256_utf8":"1ef3c7fbb29e284f8d07fa0df280bad852a11b0f23696f3e2720ac86cd024531"}]},{"id":"S080","semantic_type":"theorem","kind":"theorem","title":"随机交替近端法驻点复杂度","source_units":[{"start":61372,"end":62779,"start_line":959,"end_line":971,"raw_tex":"\\begin{theorem}%\n  \\label{thm:alternating-rate-stoch}\n  Let in addition to the assumptions of Theorem~\\ref{thm:alternating-rate} also Assumption~\\ref{ass:unbiased} and~\\ref{ass:bounded-variance} hold true. For algorithm~\\eqref{eq:gda-non-strongly-stoch} with step sizes\n  \\begin{equation*}\n  \\eta_x = \\min \\left\\{\\frac{\\epsilon^2}{\\rho(L^2+\\sigma^2)}, \\frac{\\epsilon^4}{\\rho^2 L(L+L_f+\\sigma)D_h^2 L_{\\nabla\\Phi}}, \\frac{\\epsilon^6}{\\rho^3L(L+L_f+\\sigma)\\sigma^2D_h^2 }\\right\\},\n  \\end{equation*}\n  $\\eta_y= \\min\\{\\frac{1}{2 L_{\\nabla\\Phi}}, \\frac{\\epsilon^2}{\\rho \\sigma^2}\\}$ and $\\lambda=\\frac{1}{2\\rho}$ the number of stochastic gradient evaluations $K$ required is\n  % iterates generated by~\\eqref{eq:gda-non-strongly-stoch} with $\\eta_y=\\mathcal{O}(\\epsilon^2)\\le \\nicefrac{1}{2L_{\\nabla\\Phi}}$, $\\eta_x=\\mathcal{O}(\\epsilon^6)<\\nicefrac{1}{2\\rho}$ and $\\lambda=\\nicefrac{1}{2\\rho}$ fulfill\n  \\begin{equation*}\n      \\mathcal{O}\\left(\\frac{\\Delta^*(L^2+L_f^2+\\sigma^2)\\rho}{\\epsilon^4}\\max\\Big\\{1, \\frac{\\rho L_{\\nabla\\Phi}LD_h^2}{\\epsilon^2}, \\frac{\\rho^2 D_h^2 \\sigma^2}{\\epsilon^4}\\Big\\}+ \\frac{\\Delta_0\\rho}{\\epsilon^2}\\right),\n  \\end{equation*}\n  where $\\Delta^* = \\psi(x_0) - \\inf_{x\\in\\R^d} \\psi(x)$ and $\\Delta_0=\\psi(x_0)-\\Psi(x_0,y_0)$, to visit an $\\epsilon$-stationary point in expectation such that $\\min_{1\\le k \\le K}\\E[\\Vert \\nabla \\psi_\\lambda(x_k) \\Vert] \\le \\epsilon$.\n\\end{theorem}","sha256_utf8":"6c0119cb8715502ae3a20f9caf14f56fc88ed8389eb703e54fba94cd9cb88265"}]},{"id":"S081","semantic_type":"proof","kind":"proof","title":"随机驻点复杂度完整证明","source_units":[{"start":73594,"end":76061,"start_line":1155,"end_line":1192,"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm:alternating-rate-stoch}]\n  We sum up the inequality of Lemma~\\ref{lem:stoch-descent-on-moreau} to deduce that\n  \\begin{equation*}\n    \\Exp{\\psi_\\lambda(x_{K})} \\le \\psi_\\lambda(x_0) + 2\\eta_x\\rho\\sum_{k=0}^{K-1}\\hat{\\Delta}_k - \\frac{\\eta_x}{2}\\sum_{k=0}^{K-1}\\Exp{\\Vert \\nabla \\psi_\\lambda(x_k) \\Vert^2} + 4K\\rho\\eta_x^2(L^2+\\sigma^2).\n  \\end{equation*}\n  Thus, by dividing by $K$ and $\\eta_x$ yields\n  \\begin{equation*}\n    \\frac{1}{K}\\sum_{k=0}^{K-1} \\Exp{\\Vert\\nabla \\psi_\\lambda(x_k)\\Vert^2}  \\le \\frac{2\\Delta^*}{\\eta_x K} + \\frac{4\\rho}{K}\\sum_{k=0}^{K-1}\\hat{\\Delta}_k + 8\\rho\\eta_x(L^2+\\sigma^2).\n  \\end{equation*}\n  Now we plug in~\\eqref{eq:stoch-estimate-sum-of-delta} to obtain\n  \\begin{equation*}\n    \\begin{aligned}\n      \\MoveEqLeft\\frac{1}{K}\\sum_{k=0}^{K-1} \\Exp{\\Vert\\nabla \\psi_\\lambda(x_k)\\Vert^2} \\\\\n      &\\le \\frac{2\\Delta^*}{\\eta_x K} + 4\\rho\\Big( \\eta_x L(L+L_f+\\sigma) B  + \\frac{D_h^2}{2\\eta_y B} + \\eta_y\\sigma^2 \\Big)  + 4\\rho\\frac{\\Delta_0}{K}+ 8\\rho\\eta_x (L^2+\\sigma^2).\n    \\end{aligned}\n  \\end{equation*}\n  With the block size $B=D_h \\sqrt{1/(\\eta_x\\eta_y L(L+L_f+\\sigma))}$ we have that\n  \\begin{equation*}\n    \\eta_x L(L+L_f+\\sigma) B  + \\frac{D_h^2}{2\\eta_y B} + \\eta_y\\sigma^2 = \\sqrt{\\frac{\\eta_x}{\\eta_y}}\\sqrt{L(L+L_f+\\sigma) }D_h + \\eta_y \\sigma^2\n  \\end{equation*}\n  % \\begin{equation*}\n  %   \\begin{aligned}\n  %     \\MoveEqLeft \\frac{1}{K}\\sum_{k=0}^{K-1} \\Exp{\\Vert \\nabla \\psi_\\lambda(x_k) \\Vert^2}\\\\\n  %     &\\le \\frac{2\\Delta^*}{\\eta_x K} + 4\\rho\\sqrt{\\frac{\\eta_x}{\\eta_y}}\\sqrt{L(L+L_f+\\sigma) }D_h + 4\\rho\\eta_y \\sigma^2 + 4\\rho\\frac{\\Delta_0}{K} + 8\\rho\\eta_x(L^2+\\sigma^2).\n  %   \\end{aligned}\n  % \\end{equation*}\n  Via the step size choice presented in the theorem we obtain the desired complexity.\n  % \\begin{equation}\n  % \\eta_x = \\min \\left\\{\\frac{\\epsilon^2}{\\rho(L^2+\\sigma^2)}, \\frac{\\epsilon^4}{\\rho^2 L(L+L_f+\\sigma)D_h^2 L_{\\nabla\\Phi}}, \\frac{\\epsilon^6}{\\rho^3L(L+L_f+\\sigma)\\sigma^2D_h^2 }\\right\\}\n  % \\end{equation}\n  % and $\\eta_y= \\min\\{\\frac{1}{2 L_{\\nabla\\Phi}}, \\frac{\\epsilon^2}{\\rho \\sigma^2}\\}$ we obtain\n  % \\begin{equation*}\n  %   \\begin{aligned}\n  %     \\mathcal{O}\\left(\\frac{\\Delta^*(L^2+L_f^2+\\sigma^2)\\rho}{\\epsilon^4}\\max\\Big\\{1, \\frac{\\rho L_{\\nabla\\Phi}LD_h^2}{\\epsilon^2}, \\frac{\\rho^2 D_h^2 \\sigma^2}{\\epsilon^4}\\Big\\}+ \\frac{\\Delta_0\\rho}{\\epsilon^2}\\right)\n  %   \\end{aligned}\n  % \\end{equation*}\n  % which finishes the proof.\n\\end{proof}","sha256_utf8":"758ce4c123070eac0c7ee3226534f9992df82a383832fe1aa1e5274fee78639a"}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","title":"累加期望包络下降","source_units":[{"start":73659,"end":73989,"start_line":1156,"end_line":1159,"raw_tex":"  We sum up the inequality of Lemma~\\ref{lem:stoch-descent-on-moreau} to deduce that\n  \\begin{equation*}\n    \\Exp{\\psi_\\lambda(x_{K})} \\le \\psi_\\lambda(x_0) + 2\\eta_x\\rho\\sum_{k=0}^{K-1}\\hat{\\Delta}_k - \\frac{\\eta_x}{2}\\sum_{k=0}^{K-1}\\Exp{\\Vert \\nabla \\psi_\\lambda(x_k) \\Vert^2} + 4K\\rho\\eta_x^2(L^2+\\sigma^2).\n  \\end{equation*}\n","sha256_utf8":"cb1bf1993951463b9011b003fa6947e0675c8805d65468ae1beb4c5e03293001"}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","title":"归一化得平均梯度界","source_units":[{"start":73989,"end":74258,"start_line":1160,"end_line":1163,"raw_tex":"  Thus, by dividing by $K$ and $\\eta_x$ yields\n  \\begin{equation*}\n    \\frac{1}{K}\\sum_{k=0}^{K-1} \\Exp{\\Vert\\nabla \\psi_\\lambda(x_k)\\Vert^2}  \\le \\frac{2\\Delta^*}{\\eta_x K} + \\frac{4\\rho}{K}\\sum_{k=0}^{K-1}\\hat{\\Delta}_k + 8\\rho\\eta_x(L^2+\\sigma^2).\n  \\end{equation*}\n","sha256_utf8":"238f29c61e0aa790f703c2f3061e1169029d64a550b30208b7d6b0d27c2e2dab"}]},{"id":"S084","semantic_type":"proof_step","kind":"proof_step","title":"代入随机平均间隙界","source_units":[{"start":74258,"end":74673,"start_line":1164,"end_line":1170,"raw_tex":"  Now we plug in~\\eqref{eq:stoch-estimate-sum-of-delta} to obtain\n  \\begin{equation*}\n    \\begin{aligned}\n      \\MoveEqLeft\\frac{1}{K}\\sum_{k=0}^{K-1} \\Exp{\\Vert\\nabla \\psi_\\lambda(x_k)\\Vert^2} \\\\\n      &\\le \\frac{2\\Delta^*}{\\eta_x K} + 4\\rho\\Big( \\eta_x L(L+L_f+\\sigma) B  + \\frac{D_h^2}{2\\eta_y B} + \\eta_y\\sigma^2 \\Big)  + 4\\rho\\frac{\\Delta_0}{K}+ 8\\rho\\eta_x (L^2+\\sigma^2).\n    \\end{aligned}\n  \\end{equation*}\n","sha256_utf8":"a52f3cfc70c820babcef74f874b650b0566b917260f0ba7c74404ec64e83575f"}]},{"id":"S085","semantic_type":"proof_step","kind":"proof_step","title":"选择块长平衡误差","source_units":[{"start":74673,"end":75301,"start_line":1171,"end_line":1180,"raw_tex":"  With the block size $B=D_h \\sqrt{1/(\\eta_x\\eta_y L(L+L_f+\\sigma))}$ we have that\n  \\begin{equation*}\n    \\eta_x L(L+L_f+\\sigma) B  + \\frac{D_h^2}{2\\eta_y B} + \\eta_y\\sigma^2 = \\sqrt{\\frac{\\eta_x}{\\eta_y}}\\sqrt{L(L+L_f+\\sigma) }D_h + \\eta_y \\sigma^2\n  \\end{equation*}\n  % \\begin{equation*}\n  %   \\begin{aligned}\n  %     \\MoveEqLeft \\frac{1}{K}\\sum_{k=0}^{K-1} \\Exp{\\Vert \\nabla \\psi_\\lambda(x_k) \\Vert^2}\\\\\n  %     &\\le \\frac{2\\Delta^*}{\\eta_x K} + 4\\rho\\sqrt{\\frac{\\eta_x}{\\eta_y}}\\sqrt{L(L+L_f+\\sigma) }D_h + 4\\rho\\eta_y \\sigma^2 + 4\\rho\\frac{\\Delta_0}{K} + 8\\rho\\eta_x(L^2+\\sigma^2).\n  %   \\end{aligned}\n  % \\end{equation*}\n","sha256_utf8":"67524ceada06b07a9f88bfb636e4afd39f7971d2f0df2a416edabf673b7575ac"}]},{"id":"S086","semantic_type":"proof_step","kind":"proof_step","title":"代入步长得所述复杂度","source_units":[{"start":75301,"end":76050,"start_line":1181,"end_line":1191,"raw_tex":"  Via the step size choice presented in the theorem we obtain the desired complexity.\n  % \\begin{equation}\n  % \\eta_x = \\min \\left\\{\\frac{\\epsilon^2}{\\rho(L^2+\\sigma^2)}, \\frac{\\epsilon^4}{\\rho^2 L(L+L_f+\\sigma)D_h^2 L_{\\nabla\\Phi}}, \\frac{\\epsilon^6}{\\rho^3L(L+L_f+\\sigma)\\sigma^2D_h^2 }\\right\\}\n  % \\end{equation}\n  % and $\\eta_y= \\min\\{\\frac{1}{2 L_{\\nabla\\Phi}}, \\frac{\\epsilon^2}{\\rho \\sigma^2}\\}$ we obtain\n  % \\begin{equation*}\n  %   \\begin{aligned}\n  %     \\mathcal{O}\\left(\\frac{\\Delta^*(L^2+L_f^2+\\sigma^2)\\rho}{\\epsilon^4}\\max\\Big\\{1, \\frac{\\rho L_{\\nabla\\Phi}LD_h^2}{\\epsilon^2}, \\frac{\\rho^2 D_h^2 \\sigma^2}{\\epsilon^4}\\Big\\}+ \\frac{\\Delta_0\\rho}{\\epsilon^2}\\right)\n  %   \\end{aligned}\n  % \\end{equation*}\n  % which finishes the proof.\n","sha256_utf8":"9d8d1bb5ecfaad5f5f54854d888eaa2e574acce3bb54d9b038633f87975afb6c"}]},{"id":"S087","semantic_type":"assumption","kind":"assumption","title":"随机定理步长与包络参数","source_units":[{"start":61642,"end":62042,"start_line":962,"end_line":965,"raw_tex":"  \\begin{equation*}\n  \\eta_x = \\min \\left\\{\\frac{\\epsilon^2}{\\rho(L^2+\\sigma^2)}, \\frac{\\epsilon^4}{\\rho^2 L(L+L_f+\\sigma)D_h^2 L_{\\nabla\\Phi}}, \\frac{\\epsilon^6}{\\rho^3L(L+L_f+\\sigma)\\sigma^2D_h^2 }\\right\\},\n  \\end{equation*}\n  $\\eta_y= \\min\\{\\frac{1}{2 L_{\\nabla\\Phi}}, \\frac{\\epsilon^2}{\\rho \\sigma^2}\\}$ and $\\lambda=\\frac{1}{2\\rho}$ the number of stochastic gradient evaluations $K$ required is\n","sha256_utf8":"256b3e3018b7a1440379a25a29d5f97086144b42f2452c9c4aae657dbb3ff85f"}]},{"id":"S088","semantic_type":"assumption","kind":"assumption","title":"确定性辅助分析步长","source_units":[{"start":42971,"end":43271,"start_line":653,"end_line":656,"raw_tex":"  For algorithm~\\eqref{eq:alt-gda-non-strongly} with the step sizes\n  \\begin{equation*}\n  \\eta_x=\\min \\left\\{\\frac{\\epsilon^4}{L_{\\nabla\\Phi}\\rho^2 D_h^2{(L+L_f)}^2}, \\frac{\\epsilon^2}{\\rho L^2}\\right\\}, \\eta_y=\\frac{1}{L_{\\nabla\\Phi}} \\,\\, \\text{and}\\,\\, \\lambda = \\frac{1}{2\\rho}\n  \\end{equation*}\n","sha256_utf8":"87c824959ba7e79f853573f96bf9fbdc3454889b976f942f54b54a18676acf42"}]},{"id":"S089","semantic_type":"external_fact","kind":"external_fact","title":"利普希茨次梯度标准界","source_units":[],"declared_external":true,"declaration_text":["Every Fréchet subgradient of an L-Lipschitz real-valued function has norm at most L, as cited in Mordukhovich, Theorem 3.52."]},{"id":"S090","semantic_type":"external_fact","kind":"external_fact","title":"弱凸包络梯度公式","source_units":[],"declared_external":true,"declaration_text":["For proper lower semicontinuous rho-weakly convex psi and 0<lambda<1/rho, psi_lambda is continuously differentiable and its gradient equals (x-prox_{lambda psi}(x))/lambda, as cited in Davis–Drusvyatskiy, Lemma 2.2."]},{"id":"S091","semantic_type":"external_fact","kind":"external_fact","title":"近端与凸次微分演算","source_units":[],"declared_external":true,"declaration_text":["For a proper lower semicontinuous convex function f, z=prox_{eta f}(u) iff u-z belongs to eta partial f(z); this proximal map is nonexpansive. Strong convexity gives the corresponding minimizer comparison inequality. A continuous convex summand permits the subdifferential sum rule; these facts also apply to weakly convex functions after quadratic convexification."]},{"id":"S092","semantic_type":"external_fact","kind":"external_fact","title":"有限凸函数方向导数演算","source_units":[],"declared_external":true,"declaration_text":["Pointwise suprema of convex functions are convex; a finite convex function on Euclidean space is continuous and has nonempty subdifferentials. Its directional derivative is the infimum of positive difference quotients, and domination of directional derivatives implies the corresponding inclusion of convex subdifferentials."]},{"id":"S093","semantic_type":"external_fact","kind":"external_fact","title":"标准近端上升比较界","source_units":[],"declared_external":true,"declaration_text":["For concave L-smooth q, convex proper lower semicontinuous h, eta<=1/L, and y_plus=prox_{eta h}(y_old+eta grad q(y_old)), (q-h)(y)-(q-h)(y_plus) <= (||y-y_old||^2-||y-y_plus||^2)/(2 eta), as cited in Beck–Teboulle, Lemma 2.3."]},{"id":"S094","semantic_type":"external_fact","kind":"external_fact","title":"期望与基本不等式","source_units":[],"declared_external":true,"declaration_text":["For independent unbiased samples, conditional centering and the tower property apply. Variance decomposition bounds second moments; Cauchy–Schwarz, Jensen and Young inequalities control first moments and cross terms. Finite arithmetic sums and telescoping sums have their usual bounds, and a bound on an average of expected squared norms yields a bound on the minimum expected norm."]},{"id":"S095","semantic_type":"external_fact","kind":"external_fact","title":"有界有效域上的极值存在","source_units":[],"declared_external":true,"declaration_text":["In finite-dimensional Euclidean space, a proper lower semicontinuous function with bounded effective domain attains a finite minimum by compactness of its nonempty sublevel sets. For fixed x, this applies to h(.)-Phi(x,.), yielding the maximizing selections used in the source."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S080","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S007","target_id":"S016","kind":"supports"},{"source_id":"S010","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S095","target_id":"S014","kind":"supports"},{"source_id":"S089","target_id":"S018","kind":"supports"},{"source_id":"S090","target_id":"S019","kind":"supports"},{"source_id":"S004","target_id":"S019","kind":"supports"},{"source_id":"S005","target_id":"S018","kind":"supports"},{"source_id":"S002","target_id":"S022","kind":"supports"},{"source_id":"S003","target_id":"S022","kind":"supports"},{"source_id":"S010","target_id":"S022","kind":"supports"},{"source_id":"S013","target_id":"S022","kind":"supports"},{"source_id":"S092","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S014","target_id":"S023","kind":"supports"},{"source_id":"S092","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S005","target_id":"S024","kind":"supports"},{"source_id":"S092","target_id":"S024","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S023","target_id":"S021","kind":"supports"},{"source_id":"S024","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S002","target_id":"S027","kind":"supports"},{"source_id":"S012","target_id":"S027","kind":"supports"},{"source_id":"S014","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S002","target_id":"S031","kind":"supports"},{"source_id":"S004","target_id":"S031","kind":"supports"},{"source_id":"S091","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S020","target_id":"S032","kind":"supports"},{"source_id":"S013","target_id":"S032","kind":"supports"},{"source_id":"S091","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S091","target_id":"S033","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S030","kind":"supports"},{"source_id":"S033","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S004","target_id":"S036","kind":"supports"},{"source_id":"S029","target_id":"S037","kind":"supports"},{"source_id":"S015","target_id":"S037","kind":"supports"},{"source_id":"S025","target_id":"S037","kind":"supports"},{"source_id":"S012","target_id":"S037","kind":"supports"},{"source_id":"S018","target_id":"S037","kind":"supports"},{"source_id":"S091","target_id":"S037","kind":"supports"},{"source_id":"S002","target_id":"S038","kind":"supports"},{"source_id":"S010","target_id":"S038","kind":"supports"},{"source_id":"S020","target_id":"S038","kind":"supports"},{"source_id":"S005","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S036","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S019","target_id":"S040","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S035","kind":"supports"},{"source_id":"S039","target_id":"S035","kind":"supports"},{"source_id":"S040","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S015","target_id":"S043","kind":"supports"},{"source_id":"S010","target_id":"S043","kind":"supports"},{"source_id":"S013","target_id":"S043","kind":"supports"},{"source_id":"S088","target_id":"S043","kind":"supports"},{"source_id":"S093","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S046","kind":"supports"},{"source_id":"S014","target_id":"S046","kind":"supports"},{"source_id":"S002","target_id":"S046","kind":"supports"},{"source_id":"S034","target_id":"S046","kind":"supports"},{"source_id":"S014","target_id":"S047","kind":"supports"},{"source_id":"S002","target_id":"S047","kind":"supports"},{"source_id":"S012","target_id":"S048","kind":"supports"},{"source_id":"S013","target_id":"S048","kind":"supports"},{"source_id":"S015","target_id":"S048","kind":"supports"},{"source_id":"S018","target_id":"S048","kind":"supports"},{"source_id":"S091","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S049","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S048","target_id":"S045","kind":"supports"},{"source_id":"S049","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S017","target_id":"S052","kind":"supports"},{"source_id":"S044","target_id":"S053","kind":"supports"},{"source_id":"S013","target_id":"S053","kind":"supports"},{"source_id":"S094","target_id":"S053","kind":"supports"},{"source_id":"S044","target_id":"S054","kind":"supports"},{"source_id":"S013","target_id":"S054","kind":"supports"},{"source_id":"S052","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S088","target_id":"S055","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S051","kind":"supports"},{"source_id":"S055","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S008","target_id":"S058","kind":"supports"},{"source_id":"S009","target_id":"S058","kind":"supports"},{"source_id":"S012","target_id":"S058","kind":"supports"},{"source_id":"S018","target_id":"S058","kind":"supports"},{"source_id":"S094","target_id":"S058","kind":"supports"},{"source_id":"S004","target_id":"S059","kind":"supports"},{"source_id":"S029","target_id":"S060","kind":"supports"},{"source_id":"S016","target_id":"S060","kind":"supports"},{"source_id":"S091","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S058","target_id":"S061","kind":"supports"},{"source_id":"S008","target_id":"S061","kind":"supports"},{"source_id":"S007","target_id":"S061","kind":"supports"},{"source_id":"S025","target_id":"S061","kind":"supports"},{"source_id":"S018","target_id":"S061","kind":"supports"},{"source_id":"S094","target_id":"S061","kind":"supports"},{"source_id":"S059","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S038","target_id":"S062","kind":"supports"},{"source_id":"S040","target_id":"S062","kind":"supports"},{"source_id":"S035","target_id":"S062","kind":"supports"},{"source_id":"S019","target_id":"S062","kind":"supports"},{"source_id":"S087","target_id":"S062","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S059","target_id":"S057","kind":"supports"},{"source_id":"S060","target_id":"S057","kind":"supports"},{"source_id":"S061","target_id":"S057","kind":"supports"},{"source_id":"S062","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S016","target_id":"S065","kind":"supports"},{"source_id":"S013","target_id":"S065","kind":"supports"},{"source_id":"S042","target_id":"S065","kind":"supports"},{"source_id":"S091","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S094","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S007","target_id":"S067","kind":"supports"},{"source_id":"S008","target_id":"S067","kind":"supports"},{"source_id":"S009","target_id":"S067","kind":"supports"},{"source_id":"S014","target_id":"S067","kind":"supports"},{"source_id":"S094","target_id":"S067","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S010","target_id":"S068","kind":"supports"},{"source_id":"S087","target_id":"S068","kind":"supports"},{"source_id":"S002","target_id":"S068","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S064","kind":"supports"},{"source_id":"S067","target_id":"S064","kind":"supports"},{"source_id":"S068","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S063","kind":"supports"},{"source_id":"S056","target_id":"S071","kind":"supports"},{"source_id":"S063","target_id":"S071","kind":"supports"},{"source_id":"S014","target_id":"S071","kind":"supports"},{"source_id":"S047","target_id":"S072","kind":"supports"},{"source_id":"S014","target_id":"S072","kind":"supports"},{"source_id":"S048","target_id":"S073","kind":"supports"},{"source_id":"S058","target_id":"S073","kind":"supports"},{"source_id":"S012","target_id":"S073","kind":"supports"},{"source_id":"S013","target_id":"S073","kind":"supports"},{"source_id":"S016","target_id":"S073","kind":"supports"},{"source_id":"S091","target_id":"S073","kind":"supports"},{"source_id":"S094","target_id":"S073","kind":"supports"},{"source_id":"S071","target_id":"S074","kind":"supports"},{"source_id":"S072","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S071","target_id":"S070","kind":"supports"},{"source_id":"S072","target_id":"S070","kind":"supports"},{"source_id":"S073","target_id":"S070","kind":"supports"},{"source_id":"S074","target_id":"S070","kind":"supports"},{"source_id":"S070","target_id":"S069","kind":"supports"},{"source_id":"S051","target_id":"S077","kind":"supports"},{"source_id":"S069","target_id":"S077","kind":"supports"},{"source_id":"S013","target_id":"S077","kind":"supports"},{"source_id":"S017","target_id":"S077","kind":"supports"},{"source_id":"S094","target_id":"S077","kind":"supports"},{"source_id":"S069","target_id":"S078","kind":"supports"},{"source_id":"S054","target_id":"S078","kind":"supports"},{"source_id":"S052","target_id":"S079","kind":"supports"},{"source_id":"S077","target_id":"S079","kind":"supports"},{"source_id":"S078","target_id":"S079","kind":"supports"},{"source_id":"S077","target_id":"S076","kind":"supports"},{"source_id":"S078","target_id":"S076","kind":"supports"},{"source_id":"S079","target_id":"S076","kind":"supports"},{"source_id":"S076","target_id":"S075","kind":"supports"},{"source_id":"S056","target_id":"S082","kind":"supports"},{"source_id":"S082","target_id":"S083","kind":"supports"},{"source_id":"S011","target_id":"S083","kind":"supports"},{"source_id":"S083","target_id":"S084","kind":"supports"},{"source_id":"S075","target_id":"S084","kind":"supports"},{"source_id":"S084","target_id":"S085","kind":"supports"},{"source_id":"S017","target_id":"S085","kind":"supports"},{"source_id":"S085","target_id":"S086","kind":"supports"},{"source_id":"S087","target_id":"S086","kind":"supports"},{"source_id":"S006","target_id":"S086","kind":"supports"},{"source_id":"S094","target_id":"S086","kind":"supports"},{"source_id":"S082","target_id":"S081","kind":"supports"},{"source_id":"S083","target_id":"S081","kind":"supports"},{"source_id":"S084","target_id":"S081","kind":"supports"},{"source_id":"S085","target_id":"S081","kind":"supports"},{"source_id":"S086","target_id":"S081","kind":"supports"},{"source_id":"S081","target_id":"S080","kind":"supports"}]}}
{"record_id":"2007.15845v2","paper":{"title":"A Method with Convergence Rates for Optimization Problems with Variational Inequality Constraints","authors":["Harshal D. Kaushik","Farzad Yousefian"],"arxiv_id":"2007.15845","version":"v2","journal":"SIAM Journal on Optimization","doi":"10.1137/20m1357378","abstract_url":"https://arxiv.org/abs/2007.15845v2","pdf_url":"https://arxiv.org/pdf/2007.15845v2"},"context":{"raw_tex":"\\begin{equation}\\label{prob:main}\n\\tag{$\\text{P}^f_{\\text{VI}}$} \n\\begin{aligned}\n& {\\text{minimize}}\n& & f(x) \\\\\n& \\text{subject to}\n& & x \\in \\text{SOL}(X,F).\n\\end{aligned}\n\\end{equation}\n\nHere, $f:\\mathbb{R}^n \\to \\mathbb{R}$ is a convex function and $X \\subseteq \\mathbb{R}^n$ is given as a Cartesian product, i.e., $ X \\triangleq \\prod_{i = 1}^{d}  X_i $, where $X_i \\subseteq\\mathbb{R}^{n_i}$ is convex for all $i=1,\\dots, d$ and $\\sum_{i=1}^dn_i=n$. We consider $F: X \\to \\mathbb{R}^n$ to be a monotone mapping. The term $\\text{SOL}(X,F)$ denotes the solution set of the variational inequality $\\text{VI}(X,F)$ defined as follows:  A vector $x \\in X$ is said to be a solution to $\\text{VI}(X,F)$ if for any $y \\in X$, we have $F(x)^T(y-x) \\geq 0$. Variational inequalities, first introduced in late 1950s, are an immensely powerful mathematical tool that can serve as a unifying framework for capturing a wide range of applications arising in operations research, finance, and economics (cf. \\cite{FacchineiPang2003, Rockafellar98,Lan-VI-13,wang2015}). Importantly, as it will be discussed shortly, the block structure of the set $X$ allows for addressing Nash games as well as high-dimensional optimization problems. We note that the problem \\cref{prob:main} can represent a variety of the standard problems in optimization and VI regimes. For example, when $F(x) := 0_{n}$, the problem \\cref{prob:main} is equivalent to the canonical optimization problem $\\min_{x \\in X} f(x)$. Also, when $f(x):=0$, the problem \\cref{prob:main} is equivalent to solving $\\text{VI}(X,F)$. More detailed examples that can be reformulated as the problem \\cref{prob:main} are presented below.\n\n\n\\textbf{Notation and preliminary definitions:} Throughout, a vector $ x \\in \\mathbb{R}^n$ is assumed to be a column vector and $ x^T$ denotes the transpose of $x$. We use $x^{(i)}\\in \\mathbb{R}^{n_i}$ to denote the $i^{\\text{th}}$ block-coordinate of vector $x$ where $x =\\left(x^{(1)};\\ldots;x^{(d)}\\right)$ and $\\sum_{i=1}^dn_i=n$. The Euclidean norm of a vector $x$ is denoted by $\\|x\\|$, i.e.,  $\\| x\\|\\triangleq \\sqrt{ x^T  x}.$  For a mapping $F:\\mathbb{R}^n \\to \\mathbb{R}^n$, we denote the $i^{\\text{th}}$ block-coordinate of $F$ by $F_i:\\mathbb{R}^n \\to \\mathbb{R}^{n_i}$, i.e., $F(x) =\\left(F_1(x);\\ldots;F_d(x)\\right)$. A mapping $F:\\mathbb{R}^n \\to \\mathbb{R}^n$ is said to be monotone on a convex set $X \\subseteq \\mathbb{R}^n$ if for any $x,y \\in X$, we have $(F(x)-F(y))^T(x-y)\\geq 0$. The mapping $F$ is said to be $\\mu$--strongly monotone on a convex set $X \\subseteq \\mathbb{R}^n$ if $\\mu>0$ and for any $x,y \\in X$, we have $(F(x)-F(y))^T(x-y)\\geq \\mu\\|x-y\\|^2$. Also, $F$ is said to be Lipschitz with parameter $L>0$ on the set $X$ if for any $x,y \\in X$, we have $\\|F(x)-F(y)\\|\\leq L\\|x-y\\|$. A continuously differentiable function $f:\\mathbb{R}^n \\to \\mathbb{R}$ is called $\\mu$--strongly convex on a convex set $X$ if $f(x) \\geq f(y)+\\nabla f(y)^T(x-y)+\\frac{\\mu}{2}\\|x-y\\|^2$. Function $f$ is $\\mu$--strongly convex if and only if $\\nabla f$ is $\\mu$--strongly monotone on $X$. For a convex function $f$ with the domain $\\text{dom}(f)$, the subgradient of $f$ at $x \\in \\text{dom}(f)$ is denoted by $\\tilde \\nabla f(x)$ and it satisfies $f(x)+\\tilde \\nabla f(x)^T(y-x) \\leq f(y)$ for all $y \\in \\text{dom}(f)$. The subdifferential set of $f$ at $x$ is the set of all subgradients of $f$ at $x$ and is denoted by $\\partial f(x)$. The Euclidean projection of vector $s$ onto a convex set $ S$ is denoted by $ \\proj[S]{ s} $, where $\\proj[S]{ s}\\triangleq  \\argmin_{ y \\in  S}\\| s- y\\|$. %Given two vectors $x,y\\in \\mathbb{R}^n$, we use $0\\leq x \\perp y \\geq 0$ to compactly denote the complementarity constraints $x\\geq 0$, $y \\geq 0$, and $x^Ty=0$. \n\n\nWe use $\\mathbf{I}_n$ to denote the identity matrix of size $n\\times n$. The probability of an event $Z$ is denoted by $\\text{Prob}(Z)$ and the expectation of a random variable $z$ is denoted by $\\EXP{z}$. We use $\\mathbb{R}^n_+$ and $\\mathbb{R}^n_{++}$ to denote $\\{x \\in \\mathbb{R}^n \\mid x \\geq 0\\}$ and  $\\{x \\in \\mathbb{R}^n \\mid x > 0\\}$, respectively.\n\n\n\\begin{assumption}\\label{assum:problem} Consider the problem \\cref{prob:main} under the following conditions:\n\n\\noindent (a) The set $X_i$ is nonempty, closed, and convex for all $i=1,\\dots, d$.\n\n\\noindent (b) The function $f$ is convex and has bounded subgradients over the set $X$.\n\n\\noindent  (c) The mapping $F:\\mathbb{R}^n \\to \\mathbb{R}^n$ is continuous, monotone, and bounded over the set $X$. \n\n\\noindent (d) The optimal solution set of problem \\cref{prob:main} in nonempty.\n\\end{assumption}\n\nThroughout, we let $C_F>0$ denote the bound on the Euclidean norm of the mapping $F$, i.e., $\\|F(x)\\| \\leq C_F$ for all $x \\in X$. Also, we let $C_f>0$ denote the bound on the norm of the subgradients of $f$, i.e., $\\|\\tilde \\nabla f (x)\\| \\leq C_f$ for all $\\tilde \\nabla f(x) \\in \\partial f(x)$ and $x \\in X$.\n\\begin{algorithm} [h]\n\t\\caption{aRB-IRG}\n\t\\begin{algorithmic}[1]\n\t\t\\STATE\\textbf{Input:} A random initial point $x_0 \\in X$, $\\bar{x}_0 := x_0$, initial stepsize $\\gamma_{0} > 0$, initial regularization parameter $\\eta_{0} > 0$, a scalar $0\\leq r<1$, and $S_0 := \\gamma_0^r$.\n\t\t\\FOR{k = 0, 1, \\dots}\n\t\t\\STATE Generate a realization of random variable $i_k$ according to \\cref{assum:random sample}.\n\t\t\\STATE Evaluate $F_{i_k}(x_k)$ and $\\tilde \\nabla_{i_k} f(x_k)$ where $\\tilde \\nabla f(x_k) \\in \\partial f(x_k)$.\n\n\t\t\\STATE Update $x_k$ as follows:\n%\t\t\t\t${x_{k+1}^{(i)}} :=$\n\t\t\\begin{align}\\label{equ:update_rule_aRBIRG} \n\t\t{x_{k+1}^{(i)}} :=\n\t\t\\begin{cases}\n\t\t\\proj[X_{i_k}]{ x_{k}^{(i_k)}-\\gamma_{k}\\left(F_{i_k}\\left( x_{k}\\right) + \\eta_{k} \\tilde \\nabla_{{i_k}} f\\left( x_{k}\\right)\\right)}& \\text{ if } i = i_k, \\\\\n\t\tx_k^{(i)} & \\text{ if } i \\neq i_k. \n\t\t\\end{cases}\n\t\t\\end{align}\n\t\t\\STATE Obtain $\\gamma_{k+1}$ and $\\eta_{k+1}$ (cf. \\cref{thm:a-IRG_rate_results} and \\cref{thm:convergence_RB-IRG} for the update rules).\n\n\t\t\\STATE Update the averaged iterate $\\bar{x}_{k}$ as follows:\n\t\t\t\\begin{align}\\label{eqn:ave_step_of_alg}\n\t\t\tS_{k+1} &:= S_k + \\gamma_{k+1}^r,\\quad\t\\bar{x}_{k+1}:= \\frac{S_k\\bar{x}_{k}+\\gamma_{k+1}^rx_{k+1}}{S_{k+1}}.\n\t\t\t\\end{align}\n\t\t\\ENDFOR\n\t\\end{algorithmic} \\label{alg:aRB-IRG}\n\\end{algorithm}\n\n\\begin{assumption}[Block-coordinate selection rule] \\label{assum:random sample}\nAt each iteration $k\\geq 0$, the random variable $i_k$ is generated from an independent and identically distributed discrete probability distribution such that $\\mathrm{Prob}\\left(i_k=i\\right) = \\prob{i}$ where $ \\prob{i}>0$ for $i \\in \\{1,\\ldots,d\\}$ and $\\sum_{i=1}^{d}\\prob{i}=1$.\n\\end{assumption}\n\n\\begin{definition}[Distance function]\\label{def:err_D}\n\tFor any $x,y \\in \\mathbb{R}^n$, function $\\mathcal{D}(x,y)$ is defined as $\n\t\\mathcal{D}(x,y)\\triangleq \\sum_{i=1}^{d} \\prob{i}^{-1} \\left\\Vert\\left. x^{(i)}-y^{(i)} \\right\\Vert\\right.^2$, where $\\prob{i}$ is given by \\cref{assum:random sample}.\n\t% \t, where $\\prob{i}$ is the uniform probability of selection of any block  $i$. From \\cref{assum:random sample}, we have $\\prob{i} = \\frac{1}{d}.$ So, we write  $ \\mathcal{D}(x,y) =  d \\sum_{i=1}^{d}  \\left\\Vert\\left. x^{(i)}-y^{(i)} \\right\\Vert\\right.^2$.\n\\end{definition}\n\n\\begin{remark} \\label{rem:err_D}\nUnder \\cref{assum:random sample}, we can relate the distance function $\\mathcal{D}$ with the $\\ell_2$--norm as follows: $\\prob{{min}}\t{\\mathcal{D}(x,y)} \\leq \\|x-y\\|^2 \\leq\\prob{{max}} {\\mathcal{D}(x,y)}$ for all $x,y \\in \\mathbb{R}^n$, where  $\\prob{{min}}\\triangleq \\min_{1\\leq i \\leq d}\\{\\prob{i}\\}$ and $\\prob{{max}}\\triangleq \\max_{1\\leq i \\leq d}\\{\\prob{i}\\}$.\n\\end{remark}\n\n\\begin{definition}[The dual gap function \\cite{MarcotteZhu1998}]\\label{def:gap}\nLet a nonempty closed set $X\\subseteq \\mathbb{R}^n$ and a mapping $F:X\\rightarrow\\mathbb{R}^n$ be given. Then, for any $x \\in X$, the dual gap function $\\mathrm{GAP}:X\\rightarrow \\mathbb{R}\\cup \\{+\\infty\\} $ is defined as $ \\mathrm{GAP}(x) \\triangleq \\sup_{y\\in X} F(y)^T(x-y)$.\n\\end{definition}\n\n\\begin{definition}[Regularized mapping]\\label{def:reg_map}\nGiven a vector $x \\in X$, a subgradient $\\tilde \\nabla f(x) \\in \\partial f(x)$, and an integer $k\\geq 0$, the regularized mapping $G_k: X \\to \\mathbb{R}$ is defined as $G_k(x) \\triangleq F(x)+\\eta_k\\tilde \\nabla f(x)$. The $i^\\text{th}$ {block-coordinate} of $G_k$ is denoted by $G_{k,i}$.\n\\end{definition}\n\n\\begin{definition}[History of the method]\\label{def:Fk}\nThroughout, we let the history of the algorithm to be denoted by $\\mathcal{F}_k \\triangleq \\{x_0,i_0,i_1,\\ldots,i_{k-1}\\}$ for $k\\geq 1$, with $\\mathcal{F}_0 \\triangleq \\{x_0\\}$.\n\\end{definition}\n\n\\begin{definition}[Randomized block error terms]\\label{def:deltas}\nLet ${\\mathbf{U}}_i \\in \\mathbb{R}^{n\\times n_i}$ for $i=1,\\ldots,d$ be the collection of matrices such that $\\mathbf{I}_n = \\left[{\\mathbf{U}}_1,\\ldots,{\\mathbf{U}}_d\\right] \\in \\mathbb{R}^{n\\times n}$. Consider the following definitions for $k\\geq 0$: \n\\begin{align}\\label{eqn:deltas}\n\\Delta_k\\triangleq F(x_k) - \\prob{{i_k}}^{-1}{\\mathbf{U}}_{i_k}F_{i_k}(x_k), \\qquad \\delta_k\\triangleq \\tilde \\nabla f(x_k) - \\prob{{i_k}}^{-1}{\\mathbf{U}}_{i_k} \\tilde \\nabla_{i_k}f (x_k).\n\\end{align}\n\\end{definition}","units":[{"start":7955,"end":8144,"start_line":194,"end_line":202,"raw_tex":"\\begin{equation}\\label{prob:main}\n\\tag{$\\text{P}^f_{\\text{VI}}$} \n\\begin{aligned}\n& {\\text{minimize}}\n& & f(x) \\\\\n& \\text{subject to}\n& & x \\in \\text{SOL}(X,F).\n\\end{aligned}\n\\end{equation}","sha256_utf8":"4984453dd494d2bffbcb96fc28b54e4f3c6aef71ac81d947d65268bf5a539117"},{"start":8145,"end":9636,"start_line":203,"end_line":203,"raw_tex":"Here, $f:\\mathbb{R}^n \\to \\mathbb{R}$ is a convex function and $X \\subseteq \\mathbb{R}^n$ is given as a Cartesian product, i.e., $ X \\triangleq \\prod_{i = 1}^{d}  X_i $, where $X_i \\subseteq\\mathbb{R}^{n_i}$ is convex for all $i=1,\\dots, d$ and $\\sum_{i=1}^dn_i=n$. We consider $F: X \\to \\mathbb{R}^n$ to be a monotone mapping. The term $\\text{SOL}(X,F)$ denotes the solution set of the variational inequality $\\text{VI}(X,F)$ defined as follows:  A vector $x \\in X$ is said to be a solution to $\\text{VI}(X,F)$ if for any $y \\in X$, we have $F(x)^T(y-x) \\geq 0$. Variational inequalities, first introduced in late 1950s, are an immensely powerful mathematical tool that can serve as a unifying framework for capturing a wide range of applications arising in operations research, finance, and economics (cf. \\cite{FacchineiPang2003, Rockafellar98,Lan-VI-13,wang2015}). Importantly, as it will be discussed shortly, the block structure of the set $X$ allows for addressing Nash games as well as high-dimensional optimization problems. We note that the problem \\cref{prob:main} can represent a variety of the standard problems in optimization and VI regimes. For example, when $F(x) := 0_{n}$, the problem \\cref{prob:main} is equivalent to the canonical optimization problem $\\min_{x \\in X} f(x)$. Also, when $f(x):=0$, the problem \\cref{prob:main} is equivalent to solving $\\text{VI}(X,F)$. More detailed examples that can be reformulated as the problem \\cref{prob:main} are presented below.\n","sha256_utf8":"88e9a0f9f04970e2b1794cc7aca652b60287d3acd2e948334f22d11a81f0925a"},{"start":28075,"end":30148,"start_line":300,"end_line":300,"raw_tex":"\\textbf{Notation and preliminary definitions:} Throughout, a vector $ x \\in \\mathbb{R}^n$ is assumed to be a column vector and $ x^T$ denotes the transpose of $x$. We use $x^{(i)}\\in \\mathbb{R}^{n_i}$ to denote the $i^{\\text{th}}$ block-coordinate of vector $x$ where $x =\\left(x^{(1)};\\ldots;x^{(d)}\\right)$ and $\\sum_{i=1}^dn_i=n$. The Euclidean norm of a vector $x$ is denoted by $\\|x\\|$, i.e.,  $\\| x\\|\\triangleq \\sqrt{ x^T  x}.$  For a mapping $F:\\mathbb{R}^n \\to \\mathbb{R}^n$, we denote the $i^{\\text{th}}$ block-coordinate of $F$ by $F_i:\\mathbb{R}^n \\to \\mathbb{R}^{n_i}$, i.e., $F(x) =\\left(F_1(x);\\ldots;F_d(x)\\right)$. A mapping $F:\\mathbb{R}^n \\to \\mathbb{R}^n$ is said to be monotone on a convex set $X \\subseteq \\mathbb{R}^n$ if for any $x,y \\in X$, we have $(F(x)-F(y))^T(x-y)\\geq 0$. The mapping $F$ is said to be $\\mu$--strongly monotone on a convex set $X \\subseteq \\mathbb{R}^n$ if $\\mu>0$ and for any $x,y \\in X$, we have $(F(x)-F(y))^T(x-y)\\geq \\mu\\|x-y\\|^2$. Also, $F$ is said to be Lipschitz with parameter $L>0$ on the set $X$ if for any $x,y \\in X$, we have $\\|F(x)-F(y)\\|\\leq L\\|x-y\\|$. A continuously differentiable function $f:\\mathbb{R}^n \\to \\mathbb{R}$ is called $\\mu$--strongly convex on a convex set $X$ if $f(x) \\geq f(y)+\\nabla f(y)^T(x-y)+\\frac{\\mu}{2}\\|x-y\\|^2$. Function $f$ is $\\mu$--strongly convex if and only if $\\nabla f$ is $\\mu$--strongly monotone on $X$. For a convex function $f$ with the domain $\\text{dom}(f)$, the subgradient of $f$ at $x \\in \\text{dom}(f)$ is denoted by $\\tilde \\nabla f(x)$ and it satisfies $f(x)+\\tilde \\nabla f(x)^T(y-x) \\leq f(y)$ for all $y \\in \\text{dom}(f)$. The subdifferential set of $f$ at $x$ is the set of all subgradients of $f$ at $x$ and is denoted by $\\partial f(x)$. The Euclidean projection of vector $s$ onto a convex set $ S$ is denoted by $ \\proj[S]{ s} $, where $\\proj[S]{ s}\\triangleq  \\argmin_{ y \\in  S}\\| s- y\\|$. %Given two vectors $x,y\\in \\mathbb{R}^n$, we use $0\\leq x \\perp y \\geq 0$ to compactly denote the complementarity constraints $x\\geq 0$, $y \\geq 0$, and $x^Ty=0$. \n","sha256_utf8":"b40c26a0663d5306046de9ab9419797b5efcbc293a49ba6db2cee6866ab053e7"},{"start":30273,"end":30632,"start_line":302,"end_line":302,"raw_tex":"We use $\\mathbf{I}_n$ to denote the identity matrix of size $n\\times n$. The probability of an event $Z$ is denoted by $\\text{Prob}(Z)$ and the expectation of a random variable $z$ is denoted by $\\EXP{z}$. We use $\\mathbb{R}^n_+$ and $\\mathbb{R}^n_{++}$ to denote $\\{x \\in \\mathbb{R}^n \\mid x \\geq 0\\}$ and  $\\{x \\in \\mathbb{R}^n \\mid x > 0\\}$, respectively.\n","sha256_utf8":"f87e5eb1f2eeaceb67275c0be17d8b055e01cf7bc6f43e0393f1d0c78589c2de"},{"start":30891,"end":31390,"start_line":306,"end_line":315,"raw_tex":"\\begin{assumption}\\label{assum:problem} Consider the problem \\cref{prob:main} under the following conditions:\n\n\\noindent (a) The set $X_i$ is nonempty, closed, and convex for all $i=1,\\dots, d$.\n\n\\noindent (b) The function $f$ is convex and has bounded subgradients over the set $X$.\n\n\\noindent  (c) The mapping $F:\\mathbb{R}^n \\to \\mathbb{R}^n$ is continuous, monotone, and bounded over the set $X$. \n\n\\noindent (d) The optimal solution set of problem \\cref{prob:main} in nonempty.\n\\end{assumption}","sha256_utf8":"c31e827daf3b4f9a9a084b577e12bf0729b0ea4ab58610068bb8e790e66413a5"},{"start":32584,"end":34223,"start_line":320,"end_line":346,"raw_tex":"Throughout, we let $C_F>0$ denote the bound on the Euclidean norm of the mapping $F$, i.e., $\\|F(x)\\| \\leq C_F$ for all $x \\in X$. Also, we let $C_f>0$ denote the bound on the norm of the subgradients of $f$, i.e., $\\|\\tilde \\nabla f (x)\\| \\leq C_f$ for all $\\tilde \\nabla f(x) \\in \\partial f(x)$ and $x \\in X$.\n\\begin{algorithm} [h]\n\t\\caption{aRB-IRG}\n\t\\begin{algorithmic}[1]\n\t\t\\STATE\\textbf{Input:} A random initial point $x_0 \\in X$, $\\bar{x}_0 := x_0$, initial stepsize $\\gamma_{0} > 0$, initial regularization parameter $\\eta_{0} > 0$, a scalar $0\\leq r<1$, and $S_0 := \\gamma_0^r$.\n\t\t\\FOR{k = 0, 1, \\dots}\n\t\t\\STATE Generate a realization of random variable $i_k$ according to \\cref{assum:random sample}.\n\t\t\\STATE Evaluate $F_{i_k}(x_k)$ and $\\tilde \\nabla_{i_k} f(x_k)$ where $\\tilde \\nabla f(x_k) \\in \\partial f(x_k)$.\n\n\t\t\\STATE Update $x_k$ as follows:\n%\t\t\t\t${x_{k+1}^{(i)}} :=$\n\t\t\\begin{align}\\label{equ:update_rule_aRBIRG} \n\t\t{x_{k+1}^{(i)}} :=\n\t\t\\begin{cases}\n\t\t\\proj[X_{i_k}]{ x_{k}^{(i_k)}-\\gamma_{k}\\left(F_{i_k}\\left( x_{k}\\right) + \\eta_{k} \\tilde \\nabla_{{i_k}} f\\left( x_{k}\\right)\\right)}& \\text{ if } i = i_k, \\\\\n\t\tx_k^{(i)} & \\text{ if } i \\neq i_k. \n\t\t\\end{cases}\n\t\t\\end{align}\n\t\t\\STATE Obtain $\\gamma_{k+1}$ and $\\eta_{k+1}$ (cf. \\cref{thm:a-IRG_rate_results} and \\cref{thm:convergence_RB-IRG} for the update rules).\n\n\t\t\\STATE Update the averaged iterate $\\bar{x}_{k}$ as follows:\n\t\t\t\\begin{align}\\label{eqn:ave_step_of_alg}\n\t\t\tS_{k+1} &:= S_k + \\gamma_{k+1}^r,\\quad\t\\bar{x}_{k+1}:= \\frac{S_k\\bar{x}_{k}+\\gamma_{k+1}^rx_{k+1}}{S_{k+1}}.\n\t\t\t\\end{align}\n\t\t\\ENDFOR\n\t\\end{algorithmic} \\label{alg:aRB-IRG}\n\\end{algorithm}","sha256_utf8":"b328af5a9e9ac4ce1c97581b988b93f719f897cc1dcf77f5a823afa8395c22b6"},{"start":34378,"end":34758,"start_line":348,"end_line":350,"raw_tex":"\\begin{assumption}[Block-coordinate selection rule] \\label{assum:random sample}\nAt each iteration $k\\geq 0$, the random variable $i_k$ is generated from an independent and identically distributed discrete probability distribution such that $\\mathrm{Prob}\\left(i_k=i\\right) = \\prob{i}$ where $ \\prob{i}>0$ for $i \\in \\{1,\\ldots,d\\}$ and $\\sum_{i=1}^{d}\\prob{i}=1$.\n\\end{assumption}","sha256_utf8":"4386d688bbb5422e043abb6c90cf467deabfb06b9f40629ac594579f5e6e35ed"},{"start":36922,"end":37499,"start_line":359,"end_line":363,"raw_tex":"\\begin{definition}[Distance function]\\label{def:err_D}\n\tFor any $x,y \\in \\mathbb{R}^n$, function $\\mathcal{D}(x,y)$ is defined as $\n\t\\mathcal{D}(x,y)\\triangleq \\sum_{i=1}^{d} \\prob{i}^{-1} \\left\\Vert\\left. x^{(i)}-y^{(i)} \\right\\Vert\\right.^2$, where $\\prob{i}$ is given by \\cref{assum:random sample}.\n\t% \t, where $\\prob{i}$ is the uniform probability of selection of any block  $i$. From \\cref{assum:random sample}, we have $\\prob{i} = \\frac{1}{d}.$ So, we write  $ \\mathcal{D}(x,y) =  d \\sum_{i=1}^{d}  \\left\\Vert\\left. x^{(i)}-y^{(i)} \\right\\Vert\\right.^2$.\n\\end{definition}","sha256_utf8":"9a65a9504cad447a57ca915f5cc4fcb35105952e11d4c3c6e64247b81d6805f9"},{"start":37500,"end":37912,"start_line":364,"end_line":366,"raw_tex":"\\begin{remark} \\label{rem:err_D}\nUnder \\cref{assum:random sample}, we can relate the distance function $\\mathcal{D}$ with the $\\ell_2$--norm as follows: $\\prob{{min}}\t{\\mathcal{D}(x,y)} \\leq \\|x-y\\|^2 \\leq\\prob{{max}} {\\mathcal{D}(x,y)}$ for all $x,y \\in \\mathbb{R}^n$, where  $\\prob{{min}}\\triangleq \\min_{1\\leq i \\leq d}\\{\\prob{i}\\}$ and $\\prob{{max}}\\triangleq \\max_{1\\leq i \\leq d}\\{\\prob{i}\\}$.\n\\end{remark}","sha256_utf8":"4d723b04cecfa80f1af38dd2fbe097068edacefeac6e93c365100322fdb893c5"},{"start":38548,"end":38923,"start_line":368,"end_line":370,"raw_tex":"\\begin{definition}[The dual gap function \\cite{MarcotteZhu1998}]\\label{def:gap}\nLet a nonempty closed set $X\\subseteq \\mathbb{R}^n$ and a mapping $F:X\\rightarrow\\mathbb{R}^n$ be given. Then, for any $x \\in X$, the dual gap function $\\mathrm{GAP}:X\\rightarrow \\mathbb{R}\\cup \\{+\\infty\\} $ is defined as $ \\mathrm{GAP}(x) \\triangleq \\sup_{y\\in X} F(y)^T(x-y)$.\n\\end{definition}","sha256_utf8":"55d8752b02626e85d6559f2a09a39917b61cd51aefa39bfc05de40295790860a"},{"start":39322,"end":39687,"start_line":374,"end_line":376,"raw_tex":"\\begin{definition}[Regularized mapping]\\label{def:reg_map}\nGiven a vector $x \\in X$, a subgradient $\\tilde \\nabla f(x) \\in \\partial f(x)$, and an integer $k\\geq 0$, the regularized mapping $G_k: X \\to \\mathbb{R}$ is defined as $G_k(x) \\triangleq F(x)+\\eta_k\\tilde \\nabla f(x)$. The $i^\\text{th}$ {block-coordinate} of $G_k$ is denoted by $G_{k,i}$.\n\\end{definition}","sha256_utf8":"a41df99d86d08da828c0f549785ce32f67ba602770106f160c7a13da5972d96c"},{"start":39688,"end":39939,"start_line":377,"end_line":379,"raw_tex":"\\begin{definition}[History of the method]\\label{def:Fk}\nThroughout, we let the history of the algorithm to be denoted by $\\mathcal{F}_k \\triangleq \\{x_0,i_0,i_1,\\ldots,i_{k-1}\\}$ for $k\\geq 1$, with $\\mathcal{F}_0 \\triangleq \\{x_0\\}$.\n\\end{definition}","sha256_utf8":"928d35e874f6181a29fd66572a03cce573567bc999465d6fcd6aba1f55bd8511"},{"start":41602,"end":42174,"start_line":396,"end_line":401,"raw_tex":"\\begin{definition}[Randomized block error terms]\\label{def:deltas}\nLet ${\\mathbf{U}}_i \\in \\mathbb{R}^{n\\times n_i}$ for $i=1,\\ldots,d$ be the collection of matrices such that $\\mathbf{I}_n = \\left[{\\mathbf{U}}_1,\\ldots,{\\mathbf{U}}_d\\right] \\in \\mathbb{R}^{n\\times n}$. Consider the following definitions for $k\\geq 0$: \n\\begin{align}\\label{eqn:deltas}\n\\Delta_k\\triangleq F(x_k) - \\prob{{i_k}}^{-1}{\\mathbf{U}}_{i_k}F_{i_k}(x_k), \\qquad \\delta_k\\triangleq \\tilde \\nabla f(x_k) - \\prob{{i_k}}^{-1}{\\mathbf{U}}_{i_k} \\tilde \\nabla_{i_k}f (x_k).\n\\end{align}\n\\end{definition}","sha256_utf8":"d1b9335df6d49ae9d929fe3c0db90771edce03e8e0bb3dfa9349b1062d4c24eb"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}[Convergence rate statements for \\cref{alg:aRB-IRG}]\\label{thm:a-IRG_rate_results}\nConsider \\cref{alg:aRB-IRG}. Let \\cref{assum:problem} and \\cref{assum:random sample} hold and assume {that} the set $X$ is bounded such that $\\|x\\|\\leq M$ for all $x \\in X$ and some $M>0$. Suppose for {all} $k\\geq 0$, $\\gamma_k:=\\frac{\\gamma_0}{\\sqrt{k+1}}$ and $\\eta_k:=\\frac{\\eta_0}{(k+1)^b}$, where $\\gamma_0>0$, $\\eta_0>0$, and $0<b<0.5$. Then, for any $0\\leq r <1$, the following results hold:\\\\\n\\noindent (i) Let $x^*$ be an optimal solution to the problem \\cref{prob:main}. Then, for {all} $N\\geq 2^{\\frac{2}{1-r}}-1$:\n\\begin{align}\\label{eqn:f_rate} \n&\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{2-r}{\\prob{{min}}\\eta_0}\\left(\\frac{4M^2}{\\gamma_0}+\\frac{\\gamma_0\\left(C_F^2+\\eta_0^2C_f^2\\right)}{0.5-0.5r+b}\\right)\\frac{1}{(N+1)^{0.5-b}}.\n\\end{align}\n\\noindent (ii) Consider the dual gap function in \\cref{def:gap}. Then, for {all} $N\\geq 2^{\\frac{2}{1-r}}-1$:\n\\begin{align}\\label{eqn:gap_rate} \n\\EXP{\\mathrm{GAP}\\left(\\bar{x}_N\\right)} \\leq \\frac{2-r}{\\prob{{min}}}\\left(\\frac{4M^2}{\\gamma_0}+\\frac{\\gamma_0\\left(C_F^2+\\eta_0^2C_f^2\\right)}{0.5-0.5r}+\\frac{2\\prob{{min}C_fM}\\eta_0}{1-0.5r-b}\\right)\\frac{1}{(N+1)^b}.\n\\end{align}\n\\end{theorem}","units":[{"start":64132,"end":65384,"start_line":643,"end_line":653,"raw_tex":"\\begin{theorem}[Convergence rate statements for \\cref{alg:aRB-IRG}]\\label{thm:a-IRG_rate_results}\nConsider \\cref{alg:aRB-IRG}. Let \\cref{assum:problem} and \\cref{assum:random sample} hold and assume {that} the set $X$ is bounded such that $\\|x\\|\\leq M$ for all $x \\in X$ and some $M>0$. Suppose for {all} $k\\geq 0$, $\\gamma_k:=\\frac{\\gamma_0}{\\sqrt{k+1}}$ and $\\eta_k:=\\frac{\\eta_0}{(k+1)^b}$, where $\\gamma_0>0$, $\\eta_0>0$, and $0<b<0.5$. Then, for any $0\\leq r <1$, the following results hold:\\\\\n\\noindent (i) Let $x^*$ be an optimal solution to the problem \\cref{prob:main}. Then, for {all} $N\\geq 2^{\\frac{2}{1-r}}-1$:\n\\begin{align}\\label{eqn:f_rate} \n&\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{2-r}{\\prob{{min}}\\eta_0}\\left(\\frac{4M^2}{\\gamma_0}+\\frac{\\gamma_0\\left(C_F^2+\\eta_0^2C_f^2\\right)}{0.5-0.5r+b}\\right)\\frac{1}{(N+1)^{0.5-b}}.\n\\end{align}\n\\noindent (ii) Consider the dual gap function in \\cref{def:gap}. Then, for {all} $N\\geq 2^{\\frac{2}{1-r}}-1$:\n\\begin{align}\\label{eqn:gap_rate} \n\\EXP{\\mathrm{GAP}\\left(\\bar{x}_N\\right)} \\leq \\frac{2-r}{\\prob{{min}}}\\left(\\frac{4M^2}{\\gamma_0}+\\frac{\\gamma_0\\left(C_F^2+\\eta_0^2C_f^2\\right)}{0.5-0.5r}+\\frac{2\\prob{{min}C_fM}\\eta_0}{1-0.5r-b}\\right)\\frac{1}{(N+1)^b}.\n\\end{align}\n\\end{theorem}","sha256_utf8":"6cca0c5095d06b9bced93c6c3b464f96a3be4e79d93f7c108c3905aa44c67b46"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}[Weighted averaging]\\label{Lemma 2.10}\nLet $\\{\\bar x_k\\}$ be generated by \\cref{alg:aRB-IRG}. Let us define the weights $\\lambda_{k,N} \\triangleq \\frac{\\gamma_k^r}{\\sum_{j=0}^N \\gamma_j^r}$ for $k \\in \\{0,\\ldots, N\\}$ and $N\\geq 0$. Then, for any $N\\geq 0$, we have $\\bar{x}_{N} = \\sum_{k=0}^N \\lambda_{k,N} x_k$. Also, when $X$ is a convex set, we have $\\bar x_N \\in X$.\n\\end{lemma}\n\n\\begin{proof}\n\\begin{comment}\nWe use induction to show that $\\bar{x}_{N} = \\sum_{k=0}^N \\lambda_{k,N} x_k$ for any $N\\geq 0$. For $N=0$, the relation holds due to the initialization $\\bar{x}_0 :=x_0$ in \\cref{alg:aRB-IRG} and that $\\lambda_{0,0}=1$. Next, let the relation hold for some $N\\geq 0$. From \\cref{eqn:ave_step_of_alg} and noting that $S_{N}=\\sum_{k=0}^N\\gamma_k^r$ for any $N\\geq 0$, we can write the following: \n\\begin{align*}\n\t\\bar{x}_{N+1} = \\frac{S_N\\bar{x}_N + \\gamma_{N+1}^r x_{N+1}}{S_{N+1}} \n\t%= \\frac{\\sum_{k = 0}^{N}\\gamma_k^r x_k + \\gamma_{N+1}^r x_{N+1}}{S_{N+1}}\n\t = \\frac{\\sum_{k=0}^{N+1}\\gamma_k^r x_k}{\\sum_{k = 0}^{N+1}\\gamma_k^r} = \\sum_{k=0}^{N+1} \\lambda_{k,N+1} x_k,\n\t\\end{align*}\nimplying that the induction hypothesis holds for $N+1$. Thus, we conclude that the desired averaging formula holds for $N\\geq 0$. To complete the proof, note that since $\\sum_{k=0}^N \\lambda_{k,N}=1$, under the convexity of the set $X$, we have $\\bar x_N \\in X$.\n\\end{comment}\nSee \\cref{app:ave_x_second_formula}. \n\\end{proof}\n\n\\begin{lemma}[Properties of $\\Delta_k$ and $\\delta_k$]\\label{Lemma 2.12}\nConsider \\cref{def:deltas}. We have:\\\\\n\\noindent (a)  $\\EXP{\\Delta_k\\mid \\mathcal{F}_k}=\\EXP{\\delta_k\\mid \\mathcal{F}_k}=0$.\\\\\n\\noindent (b) $\\EXP{\\|\\Delta_k\\|^2\\mid \\mathcal{F}_k} \\leq  \\left(\\prob{{min}}^{-1}-1\\right)C_F^2$ and $\\EXP{\\|\\delta_k\\|^2\\mid \\mathcal{F}_k} \\leq  \\left(\\prob{{min}}^{-1}-1\\right)C_f^2$.\n\\end{lemma}\n\n\\begin{proof} \n\\begin{comment}\n\\noindent (a) From the definition of $\\Delta_k$ and ${\\mathbf{U}}_i $, we can write:\n\\begin{align*} \n\\EXP{\\Delta_k\\mid \\mathcal{F}_k} =F(x_k)-\\sum_{i=1}^d\\prob{{i}}\\prob{{i}}^{-1}{\\mathbf{U}}_{i}F_{i}(x_k)= F(x_k) -\\sum_{i=1}^d{\\mathbf{U}}_{i}F_{i}(x_k) =0.\n\\end{align*}\nThe relation $\\EXP{\\delta_k\\mid \\mathcal{F}_k}=0$ can be shown in a similar fashion.\n\n\\noindent (b) We can write:\n\\begin{align*}\n \\EXP{\\|\\Delta_k\\|^2\\mid \\mathcal{F}_k}& = \\sum_{i=1}^d\\prob{{i}}\\left\\|F(x_k)-\\prob{{i}}^{-1}{\\mathbf{U}}_{i}F_{i}(x_k)\\right\\|^2\\\\\n &=  \\sum_{i=1}^d\\prob{{i}} \\left(\\|F(x_k)\\|^2 +\\prob{{i}}^{-2} \\left\\|{\\mathbf{U}}_{i}F_{i}(x_k)\\right\\|^2 - 2\\prob{{i}}^{-1}F(x_k)^T{\\mathbf{U}}_{i}F_{i}(x_k)\\right)\\\\\n & = \\|F(x_k)\\|^2 +\\sum_{i=1}^d\\prob{{i}}^{-1} \\left\\|{\\mathbf{U}}_{i}F_{i}(x_k)\\right\\|^2  -2\\sum_{i=1}^d\\|F_i(x_k)\\|^2\\\\\n & \\leq \\left(\\prob{{min}}^{-1}-1\\right)\\|F(x_k)\\|^2 \\leq \\left(\\prob{{min}}^{-1}-1\\right)C_F^2.\n\\end{align*}\nThe relation $\\EXP{\\|\\delta_k\\|^2\\mid \\mathcal{F}_k} \\leq  \\left(\\prob{{min}}^{-1}-1\\right)C_f^2$ can be shown using a similar approach. \n\\end{comment}\nSee \\cref{app:deltas}\n\\end{proof}\n\n\\begin{lemma}[Bounds on the harmonic series]\\label{Lemma 2.13}\nLet $0\\leq \\alpha <1$ be a given scalar. Then, for any integer $N \\geq 2^{\\frac{1}{1-\\alpha}}-1$, we have \n%\\begin{align*}\n$\\frac{(N+1)^{1-\\alpha}}{2(1-\\alpha)}  \\leq \\sum_{k=0}^{N}\\frac{1}{(k+1)^\\alpha}\\leq \\frac{(N+1)^{1-\\alpha}}{1-\\alpha}$.\n%\\end{align*}\n\\end{lemma}\n\n\\begin{proof}\n\\begin{comment}Given $0\\leq \\alpha <1$, let us define the function $\\phi:\\mathbb{R}_{++} \\to \\mathbb{R}$ as $\\phi(x) \\triangleq x^{-\\alpha}$ for all $x>0$. Since $\\alpha >0$, the function $\\phi$ is nonincreasing. We can write:\n\\begin{align*}\n\\sum_{k=0}^{N}\\frac{1}{(k+1)^\\alpha}\n%=\\sum_{k=1}^{N+1}k^{-\\alpha}\n= 1 + \\sum_{k=2}^{N+1}\\frac{1}{k^\\alpha}\\leq 1+\\int_{1}^{N+1}\\frac{dx}{x^\\alpha} =1+\\frac{(N+1)^{1-\\alpha}-1}{1-\\alpha}\\leq \\frac{(N+1)^{1-\\alpha}}{1-\\alpha},\n\\end{align*}\nimplying the desired upper bound. To show that the lower bound holds, we can write:\n\\begin{align*}\n\\sum_{k=0}^{N}\\frac{1}{(k+1)^\\alpha}=\\sum_{k=1}^{N+1}\\frac{1}{k^\\alpha}\\geq \\int_{1}^{N+2}\\frac{dx}{x^\\alpha} \\geq \\int_{1}^{N+1}\\frac{dx}{x^\\alpha}\n%=\\frac{(N+1)^{1-\\alpha}-1}{1-\\alpha} \n\\geq \\frac{(N+1)^{1-\\alpha}-0.5(N+1)^{1-\\alpha}}{1-\\alpha},\n\\end{align*}\nwhere the last inequality is obtained using the assumption that $N \\geq 2^{\\frac{1}{1-\\alpha}}-1$. Therefore, the desired lower bound holds as well. This completes the proof.\n\\end{comment}\nSee \\cref{app:harmonic_bounds}\n\\end{proof}\n\n\\begin{lemma}\\label{lem:prebound_gap}\nConsider the sequence $\\{x_k\\}$ in \\cref{alg:aRB-IRG}. Suppose $\\{\\gamma_k\\}$ and $\\{\\eta_k\\}$ are strictly positive sequences. Let \\cref{assum:problem} and \\cref{assum:random sample} hold. Let the auxiliary sequence $\\{u_k\\} \\subset X$ be defined as \n\t\t%\\begin{align}\\label{equ:update_rule_aux_u} \n\t\t{$u_{k+1} \\triangleq \\mathcal{P}_X\\left(u_k-\\gamma_k\\left(\\Delta_k+\\eta_k\\delta_k\\right)\\right)$},\n\t\t%\\end{align}\nwhere $u_0 \\in X$ is an arbitrary vector. Then, for {all $y\\in X$ and all $k \\geq 0$, we have}:\n\\begin{align}\\label{eqn:prebound_gap}\n&\\gamma_k^rF(y)^T(x_k-y) +\\gamma_k^r\\eta_k\\tilde \\nabla f(x_k)^T(x_k-y)\\leq \\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k}, y\\right) +\\|u_k-y\\|^2\\right) \\nonumber\\\\\n&-\\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k+1}, y\\right) +\\|u_{k+1}-y\\|^2\\right){+ \\gamma_k^r(x_k-u_k)^T\\left(\\Delta_k+\\eta_k\\delta_k\\right)}\\nonumber\\\\\n& +\\gamma_k^{r+1}\\left(\\|\\Delta_k\\|^2+\\eta_k^2\\|\\delta_k\\|^2\\right)+ 0.5\\prob{{i_k}}^{-1}\\gamma_k^{1+r}\\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\\end{align}\n\\end{lemma}\n\n\\begin{proof} Let $k\\geq 1$ be fixed. From \\cref{def:err_D} and \\cref{equ:update_rule_aRBIRG}, for any $y \\in X$, we have:\n\t\\begin{align}\\label{eqn:D_split}\n\t\\mathcal{D} \\left(x_{k+1}, y\\right)  =   \\prob{{i_k}}^{-1} \\left\\Vert\\left.  x_{k+1}^{(i_k)}-  y^{(i_k)} \\right\\Vert\\right.^2+\\sum_{i=1, \\thinspace\t i \\neq i_k}^{d} \\prob{i}^{-1} \\left\\Vert\\left.  x_{k}^{(i)}-  y^{(i)} \\right\\Vert\\right.^2.\n\t\\end{align}Next, we find a bound on the term $\\left\\Vert\\left.  x_{k+1}^{(i_k)}- y^{(i_k)} \\right\\Vert\\right.^2$. From the block structure of $X$, we have $y^{(i_k)}\\in X_{i_k}$. Invoking the nonexpansiveness property of the projection mapping, the update rule \\cref{equ:update_rule_aRBIRG}, \\cref{def:reg_map}, and the preceding relation, we obtain:\n\t\t\\begin{align*} \n\t\\left\\Vert\\left. x_{k+1}^{(i_k)}-  y^{(i_k)}\\right\\Vert\\right.^2 \\leq\\left\\Vert\\left.  x_{k}^{(i_k)}-\\gamma_{k}G_{k,i_k} \\left( x_{k}\\right) -y^{(i_k)}\\right\\Vert\\right.^2.\n\t\\end{align*}\n\t\tCombining the preceding relation with \\cref{eqn:D_split}, we obtain:\n\t\\begin{align*}\n\t \t\\mathcal{D}\\left(x_{k+1}, y\\right)& \\leq \n\t \t \\sum_{i=1, \\thinspace\t i \\neq i_k}^{d} \\prob{i}^{-1} \\left\\Vert\\left.  x_{k}^{(i)}-  y^{(i)} \\right\\Vert\\right.^2 +  \\prob{{i_k}}^{-1}\\left\\Vert\\left.  x_{k}^{(i_k)}-  y^{(i_k)} \\right\\Vert\\right.^2 \\nonumber \\\\\n& - 2\\,\\prob{{i_k}}^{-1}\\gamma_k\\left( x^{(i_k)}_{k} - y^{(i_k)}\\right)^TG_{k,i_k}\\left(x_k\\right)+ \\prob{{i_k}}^{-1}\\gamma_k^2 \\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\t\\end{align*}\nInvoking \\cref{def:err_D} again, we obtain:\t\n\t\t\\begin{align}\\label{eqn:erc_D_elementary_ineq1}\n\t \t\\mathcal{D}\\left(x_{k+1}, y\\right) & \\leq  \\mathcal{D}\\left(x_{k}, y\\right) - 2\\,\\prob{{i_k}}^{-1}\\gamma_k\\left( x^{(i_k)}_{k} - y^{(i_k)}\\right)^TG_{k,i_k}\\left(x_k\\right)+ \\prob{{i_k}}^{-1}\\gamma_k^2 \\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\t\\end{align}\nFrom \\cref{def:deltas} and \\cref{def:reg_map}, we can write:\n\\begin{align*}\n&\\prob{{i_k}}^{-1}\\left( x^{(i_k)}_{k} - y^{(i_k)}\\right)^TG_{k,i_k}\\left(x_k\\right) = \\prob{{i_k}}^{-1}(x_k-y)^T\\left({\\mathbf{U}}_{i_k}G_{k,i_k}\\left(x_k\\right)\\right) \\\\ \n&=  \\prob{{i_k}}^{-1}(x_k-y)^T\\left({\\mathbf{U}}_{i_k}F_{i_k}(x_k)+\\eta_k{\\mathbf{U}}_{i_k}\\tilde \\nabla_{i_k}f\\left(x_k\\right)\\right) \\\\\n& {=(x_k-y)^T\\left(F(x_k)-\\Delta_k+\\eta_k\\tilde \\nabla f(x_k)-\\eta_k\\delta_k\\right) = (x_k-y)^T\\left(G_k(x_k)-\\Delta_k-\\eta_k\\delta_k\\right).}\n\\end{align*}\n\tCombining the preceding inequality and relation \\cref{eqn:erc_D_elementary_ineq1}, we obtain:\n\t\\begin{align}\\label{eqn:erc_D_elementary_ineq2}\n\t \t\\mathcal{D}\\left(x_{k+1}, y\\right) & \\leq  \\mathcal{D}\\left(x_{k}, y\\right) {- 2 \\gamma_k(x_k-y)^T\\left(G_k(x_k)-\\Delta_k-\\eta_k\\delta_k\\right)}+ \\prob{{i_k}}^{-1}\\gamma_k^2 \\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\t\\end{align}\n\tConsider the definition of the auxiliary sequence $\\{u_k\\}$ in \\cref{lem:prebound_gap}. Invoking the nonexpansiveness property of the projection again, we can obtain:\n\\begin{align*}\n\\|u_{k+1}-y\\|^2  &\\leq{\\|u_k-\\gamma_k\\left(\\Delta_k+\\eta_k\\delta_k\\right)-y\\|^2}\\\\\n&=\\|u_k-y\\|^2{-2\\gamma_k(u_k-y)^T\\left(\\Delta_k+\\eta_k\\delta_k\\right)} + \\gamma_k^2\\|\\Delta_k+\\eta_k\\delta_k\\|^2.\n\\end{align*}\nThus, we have: \n\\begin{align*}\n\\|u_{k+1}-y\\|^2\\leq \\|u_k-y\\|^2{- 2\\gamma_k(u_k-y)^T\\left(\\Delta_k+\\eta_k\\delta_k\\right)} +2\\gamma_k^2\\|\\Delta_k\\|^2+2\\gamma_k^2\\eta_k^2\\|\\delta_k\\|^2.\n\\end{align*}\nAdding the preceding inequality and the inequality \\cref{eqn:erc_D_elementary_ineq2} together, we obtain:\n\\begin{align}\\label{eqn:erc_D_elementary_ineq3}\n&2\\gamma_k(x_k-y)^T G_k(x_k) \\leq \\left(\\mathcal{D}\\left(x_{k}, y\\right) +\\|u_k-y\\|^2\\right) -\\left(\\mathcal{D}\\left(x_{k+1}, y\\right) +\\|u_{k+1}-y\\|^2\\right)\\nonumber\\\\\n& {+ 2\\gamma_k(x_k-u_k)^T\\left(\\Delta_k+\\eta_k\\delta_k\\right)}+2\\gamma_k^2\\left(\\|\\Delta_k\\|^2+\\eta_k^2\\|\\delta_k\\|^2\\right)+ \\prob{{i_k}}^{-1}\\gamma_k^2 \\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\\end{align}\nFrom the monotonicity property of the mapping $F$ and \\cref{def:reg_map}, we have:\n\\begin{align*}\n(x_k-y)^TG_k(x_k)\n%=(x_k-y)^T\\left(F(x_k)+\\eta_k\\tilde \\nabla f(x_k)\\right) \n\\geq (x_k-y)^TF(y)+\\eta_k\\tilde \\nabla f(x_k)^T(x_k-y).\n\\end{align*}\nThis provides a lower bound on the left-hand side of \\cref{eqn:erc_D_elementary_ineq3}. The inequality \\cref{eqn:prebound_gap} is obtained by substituting this bound in \\cref{eqn:erc_D_elementary_ineq3} and multiplying both sides by $\\frac{\\gamma_{k}^{r-1}}{2}$.\n\\end{proof}\n\n\\begin{proposition}[Error bounds for \\cref{alg:aRB-IRG}]\\label{prop:dual_gap_bound}\nLet the sequence $\\{\\bar x_k\\}$ be generated by \\cref{alg:aRB-IRG}{, where $0\\leq r<1$}. Suppose $\\{\\gamma_k\\}$ and $\\{\\eta_k\\}$ are strictly positive and nonincreasing sequences. Let \\cref{assum:problem} and \\cref{assum:random sample} hold {and} assume {that} the set $X$ is bounded, i.e., {$\\|x\\|\\leq M$} for all $x \\in X$ and some $M>0$. \\\\\n\\noindent (a) Let $x^*$ be an optimal solution to problem \\cref{prob:main}. Then, for {all} $N\\geq 1$:\n\\begin{align}\\label{eqn:f_bound} \n&\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{\\frac{4M^2\\gamma_{{N}}^{r-1}}{\\eta_{N}} +\\sum_{k=0}^{N}\\eta_k^{-1}\\gamma_k^{r+1}\\left(C_F^2+\\eta_k^2C_f^2\\right)}{\\prob{{min}}\\sum_{k=0}^{N}\\gamma_k^r}.\n\\end{align}\n\\noindent (b) Consider the dual gap function in \\cref{def:gap}. Then, for {all} $N\\geq 1$:\n\\begin{align}\\label{eqn:gap_bound} \n\\EXP{\\mathrm{GAP}\\left(\\bar{x}_N\\right)} \\leq\\frac{ {4 M^2 \\gamma_{{N}}^{r-1} }  +\\sum_{k=0}^{N}\\gamma_{{k}}^r\\left( 2\\prob{{min}}\\eta_kC_fM+\\gamma_k C_F^2 +  \\gamma_{k}\\eta_k^2C_f ^2 \\right)}{\\prob{{min}}\\sum_{k=0}^{N} \\gamma_{{k}}^r}.\n\\end{align}\n\\end{proposition}\n\n\\begin{proof}\nWe define the following terms for all $k\\geq 0$, that appear in \\cref{eqn:prebound_gap}:\n\\begin{align}\\label{eqn:three_terms_abr}\n& {\\Theta_{k,1} \\triangleq \\gamma_k^r(x_k-u_k)^T\\left(\\Delta_k+\\eta_k\\delta_k\\right)}, \\quad \\Theta_{k,2} \\triangleq \\gamma_k^{r+1}\\left(\\|\\Delta_k\\|^2+\\eta_k^2\\|\\delta_k\\|^2\\right),\\nonumber \\\\ \n&\\Theta_{k,3} \\triangleq 0.5\\prob{{i_k}}^{-1}\\gamma_k^{1+r}\\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\\end{align}\nNext, we estimate the expected values of these terms. {Consider the notation of} $\\mathcal{F}_k$ given by \\cref{def:Fk}. Note that $x_k$ is $\\mathcal{F}_k$--measurable. Also, from the definition of $u_k$ in \\cref{lem:prebound_gap}, $u_k$ is $\\mathcal{F}_k$--measurable. Note, however, that $\\Theta_{k,j}$ is $\\mathcal{F}_{k+1}$--measurable for all $j \\in \\{1,2,3\\}$. Taking these into account and using the total probability law, for any $k\\geq 0$ and $j \\in \\{1,2,3\\}$, we have $ \\EXP{\\Theta_{k,j}} = \\mathsf{E}_{\\mathcal{F}_k}\\left[\\mathsf{E}_{i_k}\\left[\\Theta_{k,j}\\mid \\mathcal{F}_k\\right]\\right]$. From this relation and \\cref{Lemma 2.12}, we have for any $k\\geq 0$:\n\\begin{align}\\label{eqn:estimate_deltas_1_2}\n&\\EXP{\\Theta_{k,1}} = 0, \\qquad \\EXP{\\Theta_{k,2}} = \\left(\\prob{{min}}^{-1}-1\\right)\\gamma_k^{r+1}\\left(C_F^2+\\eta_k^2C_f^2\\right).\n\\end{align}\nAlso, using \\cref{def:reg_map} and the triangle inequality, we can write:\n\\begin{align*}\n&{\\mathsf{E}_{i_k}\\left[\\Theta_{k,3}\\mid \\mathcal{F}_k\\right]} = \\sum_{i=1}^d\\prob{i}\\left(0.5\\prob{{i}}^{-1}\\gamma_k^{1+r}\\left\\Vert\\left.G_{k,i}\\left(x_k\\right)\\right\\Vert\\right.^2\\right)\\nonumber\\\\\n&\\leq \\gamma_k^{1+r}\\sum_{i=1}^d\\left(\\|F_i(x_k)\\|^2+\\eta_k^2\\|\\tilde \\nabla_i f(x_k)\\|^2\\right)  = \\gamma_k^{1+r}\\|F(x_k)\\|^2+\\eta_k^2\\|\\tilde \\nabla f(x_k)\\|^2.\n\\end{align*}\n{From the preceding inequality, we obtain:\n\\begin{align}\\label{eqn:estimate_deltas_3}\n\\EXP{\\Theta_{k,3}} & \\leq  \\gamma_k^{1+r}\\left(C_F^2+\\eta_k^2C_f^2\\right).\n\\end{align}\n}We are now ready to show the inequalities \\cref{eqn:f_bound} and \\cref{eqn:gap_bound} as follows:\\\\\n\\noindent (a) Consider \\cref{eqn:prebound_gap}. From the definition of subgradients of the convex function $f$, we have that $f(x_k)-f(y) \\leq \\tilde \\nabla f(x_k)^T(x_k-y)$. Thus, {from \\cref{eqn:three_terms_abr}} we obtain for any $y \\in X$:\n\\begin{align*}\n&\\gamma_k^rF(y)^T(x_k-y) +\\gamma_k^r\\eta_k\\left(f(x_k)-f(y)\\right)\\leq \\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k}, y\\right) +\\|u_k-y\\|^2\\right) \\nonumber\\\\\n&-\\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k+1}, y\\right) +\\|u_{k+1}-y\\|^2\\right)+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}.\n\\end{align*}\nLet us substitute $y:=x^*$, where $x^*$ denotes an optimal solution to \\cref{prob:main}. Note that $x^*$ must be a feasible solution to \\cref{prob:main}, i.e., $F(x^*)^T(x_k-x^*) \\geq 0$. Thus, we obtain:\n\\begin{align}\\label{eqn:gap_prop_ineq1_a}\n\t\t\t&\\gamma_k^r\\eta_k\\left(f(x_k)-f(x^*)\\right) \\leq  \\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k}, x^*\\right) +\\|u_k-x^*\\|^2\\right) \\nonumber\\\\\n&-\\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k+1}, x^*\\right) +\\|u_{k+1}-x^*\\|^2\\right)+\\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}.\n\\end{align}\nDividing both sides by $\\eta_k$ and adding and subtracting $\\frac{\\gamma_{{k-1}}^{r-1}}{2\\eta_{k-1}} \\left( \\mathcal{D}\\left(x_k,x^*\\right) + \\left\\Vert u_k-x^* \\right\\Vert^2 \\right)$ in the right-hand side of \\cref{eqn:gap_prop_ineq1_a}, we obtain for $k\\geq 1$:\n\\begin{align}\\label{eqn:gap_prop_ineq2_a}\n&\\gamma_k^r\\left(f(x_k)-f(x^*)\\right) \\leq  \\frac{\\gamma_{{k-1}}^{r-1}}{2\\eta_{k-1}} \\left(  \\mathcal{D}\\left(x_k,x^*\\right)  + \\left\\Vert u_k-x^* \\right\\Vert^2 \\right) \\nonumber \\\\\n&- \\frac{\\gamma_{{k}}^{r-1}}{2\\eta_k} \\left(  \\mathcal{D}\\left(x_{k+1},x^*\\right)  + \\left\\Vert u_{k+1}-x^* \\right\\Vert^2 \\right)\\nonumber \\\\ \n&+ \\frac{1}{2} \\left(\\frac{\\gamma_{{k}}^{r-1}}{\\eta_k} - \\frac{\\gamma_{{k-1}}^{r-1}}{\\eta_{k-1}}\\right)\\left( \\mathcal{D}\\left(x_k,x^*\\right) + \\left\\Vert u_k-x^* \\right\\Vert^2 \\right)+\\eta_k^{-1}\\left( \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right).\n\\end{align}\nSince $r-1 < 0$ and that $\\{\\gamma_k\\}$ and $\\{\\eta_k\\}$ are nonincreasing, we have $\\frac{\\gamma_{{k}}^{r-1}}{\\eta_k} - \\frac{\\gamma_{{k-1}}^{r-1}}{\\eta_{k-1}} \\geq 0$. Also, from the boundedness of the set $X$, since $x_k$, $x^*$, and {$u_k$} belong to $ X$, using \\cref{rem:err_D} and the triangle inequality, we have:\n\\begin{align}\\label{eqn:gap_prop_ineq3_a}\n\\mathcal{D}\\left(x_k,x^*\\right)  + \\left\\Vert u_k-x^* \\right\\Vert^2 \n\\leq \\prob{{min}}^{-1}\\left\\Vert x_k-x^* \\right\\Vert^2 + \\left\\Vert u_k-x^* \\right\\Vert^2\n\\leq 4M^2\\left(\\prob{{min}}^{-1}+1\\right)\\leq \\frac{8M^2}{\\prob{{min}}}.\n\\end{align}\nSumming over \\cref{eqn:gap_prop_ineq2_a} from $k=1$ to $N$ and using \\cref{eqn:gap_prop_ineq3_a}, we obtain:\n\\begin{align*}\n&\\sum_{k=1}^{N}\\gamma_k^r\\left(f(x_k)-f(x^*)\\right) \\leq \\frac{\\gamma_{0}^{r-1}}{2\\eta_0} \\left(  \\mathcal{D}\\left(x_1,x^*\\right)  + \\left\\Vert u_1-x^* \\right\\Vert^2 \\right) \n\\nonumber \\\\ \n&+4M^2\\prob{{min}}^{-1}\\left(\\frac{\\gamma_{{N}}^{r-1}}{\\eta_{N}} - \\frac{\\gamma_{{0}}^{r-1}}{\\eta_{0}}\\right)+\\sum_{k=1}^{N}\\eta_k^{-1}\\left(\\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right),\n\\end{align*}\nwhere we drop the nonpositive term. From relation \\cref{eqn:gap_prop_ineq1_a} when $k=0$, we have:\n\\begin{align*}\n\\gamma_0^r\\left(f(x_0)-f(x^*)\\right)& \\leq  \\frac{\\gamma_0^{r-1}}{2\\eta_0}\\left(\\mathcal{D}\\left(x_{0}, x^*\\right) +\\|u_0-x^*\\|^2\\right) \\nonumber\\\\\n&-\\frac{\\gamma_0^{r-1}}{2\\eta_0}\\left(\\mathcal{D}\\left(x_{1}, x^*\\right) +\\|u_{1}-x^*\\|^2\\right)+ \\eta_0^{-1}\\left(\\Theta_{0,1}+\\Theta_{0,2}+\\Theta_{0,3}\\right).\n\\end{align*}\nAdding the last two inequalities, {multiplying and dividing the left-hand side by $\\sum_{k=0}^{N}\\gamma_k^r$, and then, invoking} \\cref{Lemma 2.10} and convexity of $f$, we obtain:\n\\begin{align*}\n&\\left(\\sum_{k=0}^{N}\\gamma_k^r\\right)\\left(f\\left(\\bar x_N\\right)-f(x^*)\\right) \\leq \\frac{\\gamma_{0}^{r-1}}{2\\eta_0} \\left(  \\mathcal{D}\\left(x_0,x^*\\right)  + \\left\\Vert u_0-x^* \\right\\Vert^2 \\right) \n\\nonumber \\\\ \n&+4M^2\\prob{{min}}^{-1}\\left(\\frac{\\gamma_{{N}}^{r-1}}{\\eta_{N}} - \\frac{\\gamma_{{0}}^{r-1}}{\\eta_{0}}\\right)+\\sum_{k=0}^{N}\\eta_k^{-1}\\left( \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right).\n\\end{align*}\nTaking the expectation on both sides and invoking \\cref{eqn:gap_prop_ineq3_a}, we obtain:\n\\begin{align*}\n&\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{\\frac{4M^2\\prob{{min}}^{-1}\\gamma_{{N}}^{r-1}}{\\eta_{N}} +\\sum_{k=0}^{N}\\eta_k^{-1}\\EXP{\\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}}}{\\sum_{k=0}^{N}\\gamma_k^r}.\n\\end{align*}\nFrom the relations \\cref{eqn:estimate_deltas_1_2} and \\cref{eqn:estimate_deltas_3}, we obtain:\n\\begin{align*}\n&\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{\\frac{4M^2\\prob{{min}}^{-1}\\gamma_{{N}}^{r-1}}{\\eta_{N}}+\\sum_{k=0}^{N}\\eta_k^{-1}\\left(\\prob{{min}}^{-1}\\gamma_k^{r+1}\\left(C_F^2+\\eta_k^2C_f^2\\right)\\right)}{\\sum_{k=0}^{N}\\gamma_k^r},\n\\end{align*}\nwhich implies the inequality \\cref{eqn:f_bound}. \n\n\\noindent (b) From the Cauchy-Schwarz inequality, the definitions of $C_f$ and $M$, and the triangle inequality, we have $ \\tilde \\nabla f(x_k)^T(y-x_k) \\leq \\left\\| \\tilde \\nabla f(x_k)\\right\\|\\|x_k-y\\| \\leq 2C_f M.$\nAdding the preceding inequality with the relation \\cref{eqn:prebound_gap}, {from \\cref{eqn:three_terms_abr}} we obtain:\n\\begin{align}\\label{eqn:prop_gap_proof_ineq0}\n&\\gamma_k^rF(y)^T(x_k-y) \\leq \\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k}, y\\right) +\\|u_k-y\\|^2\\right) \\nonumber\\\\\n&-\\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k+1}, y\\right) +\\|u_{k+1}-y\\|^2\\right)+ 2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}.\n\\end{align}\nAdding and subtracting the term $\\frac{\\gamma_{{k-1}}^{r-1}}{2} \\left( \\mathcal{D}\\left(x_k,y\\right) + \\left\\Vert u_k-y \\right\\Vert^2 \\right)$, we obtain:\n\\begin{align*}\n\t\t\t&\\gamma_k^rF(y)^T(x_k-y) \\leq  \\frac{\\gamma_{{k-1}}^{r-1}}{2} \\left(  \\mathcal{D}\\left(x_k,y\\right)  + \\left\\Vert u_k-y \\right\\Vert^2 \\right) \\nonumber \\\\\n\t\t\t&- \\frac{\\gamma_{{k}}^{r-1}}{2} \\left(  \\mathcal{D}\\left(x_{k+1},y\\right)  + \\left\\Vert u_{k+1}-y \\right\\Vert^2 \\right)+ \\frac{1}{2} \\left(\\gamma_{{k}}^{r-1} - \\gamma_{{k-1}}^{r-1}\\right)\\left( \\mathcal{D}\\left(x_k,y\\right) + \\left\\Vert u_k-y \\right\\Vert^2 \\right)\\nonumber \\\\\n&+ 2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}.\n\\end{align*}\nSubstituting the bound given by \\cref{eqn:gap_prop_ineq3_a} in the preceding relation, we obtain:\n\\begin{align*}\n\t\t\t&\\gamma_k^rF(y)^T(x_k-y) \\leq  \\frac{\\gamma_{{k-1}}^{r-1}}{2} \\left(  \\mathcal{D}\\left(x_k,y\\right)  + \\left\\Vert u_k-y \\right\\Vert^2 \\right) \\nonumber \\\\\n\t\t\t&- \\frac{\\gamma_{{k}}^{r-1}}{2} \\left(  \\mathcal{D}\\left(x_{k+1},y\\right)  + \\left\\Vert u_{k+1}-y \\right\\Vert^2 \\right)+ 4M^2\\prob{{min}}^{-1} \\left(\\gamma_{{k}}^{r-1} - \\gamma_{{k-1}}^{r-1}\\right)\\nonumber \\\\\n&+ 2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}.\n\\end{align*}\nSumming both sides from $k = 1 $ to $N$, we obtain:\n\\begin{align}\\label{eqn:prop_gap_proof_ineq1}\n\\sum_{k=1}^{N}\\gamma_{{k}}^r F(y)^T \\left(x_k-y\\right)& \\leq  \\frac{\\gamma_{{0}}^{r-1}}{2} \\left( \\mathcal{D}\\left(x_1,y\\right) + \\left\\Vert u_1-y \\right\\Vert^2 \\right)+ 4 M^2\\prob{{min}}^{-1} \\left( \\gamma_{{N}}^{r-1} - \\gamma_{{0}}^{r-1}\\right) \\nonumber \\\\\n\t\t\t%&- \\frac{\\gamma_{{N}}^{r-1}}{2} \\left(  \\mathcal{D}\\left(x_{N},y\\right)  + \\left\\Vert u_{N}-y \\right\\Vert^2 \\right) \\nonumber \\\\\n\t\t\t& +\\sum_{k=1}^{N}\\left(  2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right).\n\\end{align}\nWriting the inequality \\cref{eqn:prop_gap_proof_ineq0} for $k =0$, we have:\n\t\\begin{align}\\label{eqn:prop_gap_proof_ineq2}\n\\gamma_0^rF(y)^T(x_0-y) &\\leq \\frac{\\gamma_0^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{0}, y\\right) +\\|u_0-y\\|^2\\right)-\\frac{\\gamma_0^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{1}, y\\right) +\\|u_{1}-y\\|^2\\right) \\nonumber\\\\\n&+ 2\\gamma_0^r\\eta_0C_f M+ \\Theta_{0,1}+\\Theta_{0,2}+\\Theta_{0,3}.\n\\end{align}\nAdding \\cref{eqn:prop_gap_proof_ineq1} and \\cref{eqn:prop_gap_proof_ineq2} together, we obtain:\n\\begin{align}\\label{eqn:prop_gap_proof_ineq3}\n\\sum_{k=0}^{N}\\gamma_{{k}}^r F(y)^T \\left(x_k-y\\right) &\\leq  \\frac{\\gamma_{{0}}^{r-1}}{2} \\left( \\mathcal{D}\\left(x_0,y\\right) + \\left\\Vert u_0 -y \\right\\Vert^2 \\right) + 4 M^2\\prob{{min}}^{-1} \\left( \\gamma_{{N}}^{r-1} - \\gamma_{{0}}^{r-1}\\right) \\nonumber \\\\& +\\sum_{k=0}^{N}\\left(  2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right).\n\\end{align}\nRecalling $\\bar{x}_{N} = \\sum_{k=0}^N \\lambda_{k,N} x_k$ in \\cref{Lemma 2.10}, applying the bound given by \\cref{eqn:gap_prop_ineq3_a}, and using the triangle inequality, we obtain:\n\t\\begin{align*}\n\\left(\\sum_{k=0}^{N} \\gamma_{{k}}^r \\right) F(y)^T \\left(\\bar{x}_N-y\\right) \\leq& {4 M^2\\prob{{min}}^{-1}  \\gamma_{{N}}^{r-1} }  +\\sum_{k=0}^{N}\\left(  2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right).\n\\end{align*}\nTaking the supremum with respect to $y$ over the set $X$ from the left-hand side, invoking \\cref{def:gap}, and then dividing both sides by $\\sum_{k=0}^{N} \\gamma_{{k}}^r $, we obtain:\n\t\\begin{align*}\n\\text{GAP}\\left(\\bar x_N\\right) \\leq \\frac{4 M^2\\prob{{min}}^{-1}  \\gamma_{{N}}^{r-1}   +\\sum_{k=0}^{N}\\left(  2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right)}{\\sum_{k=0}^{N} \\gamma_{{k}}^r }.\n\\end{align*}\nTaking the expectation on both sides, using the relations \\cref{eqn:estimate_deltas_1_2} and \\cref{eqn:estimate_deltas_3}, and rearranging the terms, we obtain the inequality \\cref{eqn:gap_bound}.\n\\end{proof}\n\n\\begin{proof} Let us define the following terms: \n\\begin{align*}%\\label{eqn:three_terms_abr2}\n& \\Lambda_{N,1} \\triangleq \\prob{{min}}\\sum_{k=0}^{N}\\gamma_k^r, \\quad  \\Lambda_{N,2}\\triangleq  \\frac{4M^2\\gamma_{{N}}^{r-1}}{\\eta_{N}} ,\\quad \\Lambda_{N,3} \\triangleq \\left(C_F^2+\\eta_0^2C_f^2\\right)\\sum_{k=0}^{N}\\eta_k^{-1}\\gamma_k^{r+1},\\nonumber\\\\ \n& \\Lambda_{N,4} \\triangleq 4M^2\\gamma_{{N}}^{r-1}, \\quad  \\Lambda_{N,5} \\triangleq \\left(C_F^2+\\eta_0^2C_f^2\\right)\\sum_{k=0}^{N}\\gamma_k^{r+1}, \\quad \\Lambda_{N,6} \\triangleq 2\\prob{{min}}C_fM \\sum_{k=0}^{N}\\eta_k\\gamma_k^r.\n\\end{align*}\nNote that from \\cref{eqn:f_bound} and \\cref{eqn:gap_bound}, we have:\n\\begin{align}\\label{eqn:bounds_in_Lambdas}\n\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{\\Lambda_{N,2}+\\Lambda_{N,3}}{\\Lambda_{N,1}}, \\quad \\EXP{\\text{GAP}\\left(\\bar{x}_N\\right)} \\leq \\frac{\\Lambda_{N,4}+\\Lambda_{N,5}+\\Lambda_{N,6}}{\\Lambda_{N,1}}.\n\\end{align}\nNext, we apply \\cref{Lemma 2.13} to estimate the terms $\\Lambda_{N,i}$. Substituting $\\gamma_k$ and $\\eta_k$ by their update rules, we obtain:\n\\begin{align*}%\\label{eqn:Lambda1}\n\\Lambda_{N,1} &=\\prob{{min}}\\sum_{k=0}^{N}\\frac{\\gamma_0^r}{(k+1)^{0.5r}}\\geq \\frac{\\prob{{min}}\\gamma_0^r(N+1)^{1-0.5r}}{2(1-0.5r)},\\\\  \n\\Lambda_{N,2} &= \\frac{4M^2(N+1)^{0.5(1-r)+b}}{\\eta_0\\gamma_0^{1-r}},\\quad \\Lambda_{N,4} = \\frac{4M^2(N+1)^{0.5(1-r)}}{\\gamma_0^{1-r}}, \\\\\n\\Lambda_{N,3} &= \\sum_{k=0}^{N}\\frac{\\left(C_F^2+\\eta_0^2C_f^2\\right)\\gamma_0^{1+r}}{\\eta_0(k+1)^{0.5(1+r)-b}} \\leq\\frac{\\gamma_0^{1+r}\\left(C_F^2+\\eta_0^2C_f^2\\right)(N+1)^{1-0.5(1+r)+b}}{\\eta_0(1-0.5(1+r)+b)},\\\\\n\\Lambda_{N,5} &= \\left(C_F^2+\\eta_0^2C_f^2\\right)\\sum_{k=0}^{N}\\frac{\\gamma_0^{r+1}}{(k+1)^{0.5(1+r)}}\\leq \\frac{\\left(C_F^2+\\eta_0^2C_f^2\\right)\\gamma_0^{r+1}(N+1)^{1-0.5(1+r)}}{1-0.5(1+r)},\\\\\n\\Lambda_{N,6} &=2\\prob{{min}}C_fM \\eta_0\\gamma_0^r\\sum_{k=0}^{N}\\frac{1}{(k+1)^{0.5r+b}}\\leq  \\frac{2\\prob{{min}}C_fM \\eta_0\\gamma_0^r (N+1)^{1-0.5r-b}}{1-0.5r-b}.\n\\end{align*}\nFor these inequalities to hold, we need to ensure that {the} conditions of \\cref{Lemma 2.13} are met. Accordingly, we must have $0\\leq 0.5r <1$, $0\\leq 0.5(1+r)-b <1$, $0\\leq 0.5r+b<1$, and $0 \\leq 0.5(1+r)<1$. These relations hold because $0\\leq r<1$ and $0<b<0.5$. Another set of conditions when applying \\cref{Lemma 2.13} includes $N\\geq \\max\\left\\{2^{1/(1-0.5r)},2^{1/(1-0.5(1+r)+b)},2^{1/(1-0.5r-b)},2^{1/(1-0.5(1+r))}\\right\\}-1$. {This relation is indeed} satisfied as a consequence of $N\\geq 2^{\\frac{2}{1-r}}-1$, {$0<b<0.5$}, and $0\\leq r<1$. We conclude that all the necessary conditions for applying \\cref{Lemma 2.13} and obtaining the aforementioned bounds for the terms $\\Lambda_{N,i}$ are satisfied. To show that the inequalities \\cref{eqn:f_rate} and \\cref{eqn:gap_rate} hold, it suffices to substitute the preceding {bounds on} the terms $\\Lambda_{N,i}$ into the two inequalities given by \\cref{eqn:bounds_in_Lambdas}. The details are as follows:\n\\begin{align*}\n&\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{\\Lambda_{N,2}+\\Lambda_{N,3}}{\\Lambda_{N,1}}=\\frac{2-r}{\\prob{{min}}\\gamma_0^r(N+1)^{1-0.5r}}\\left(\\frac{4M^2(N+1)^{0.5-0.5r+b}}{\\eta_0\\gamma_0^{1-r}}\\right.\\\\\n&\\left.+\\left(\\frac{\\gamma_0^{1+r}}{\\eta_0}\\right)\\frac{\\left(C_F^2+\\eta_0^2C_f^2\\right)(N+1)^{0.5-0.5r+b}}{0.5-0.5r+b}\\right). \n\\end{align*}\nThe inequality \\cref{eqn:f_rate} is obtained by rearranging the terms in the preceding relation.\n\\begin{align*}\n&\\EXP{\\text{GAP}\\left(\\bar{x}_N\\right)} \\leq \\frac{\\Lambda_{N,4}+\\Lambda_{N,5}+\\Lambda_{N,6}}{\\Lambda_{N,1}} \\leq\\frac{2-r}{\\prob{{min}}\\gamma_0^r(N+1)^{1-0.5r}}\\left(\\frac{4M^2(N+1)^{0.5-0.5r}}{\\gamma_0^{1-r}} \\right.\\\\\n&\\left.+\\frac{\\left(C_F^2+\\eta_0^2C_f^2\\right)\\gamma_0^{r+1}(N+1)^{0.5-0.5r}}{0.5-0.5r}+ \\frac{2\\prob{{min}}C_fM \\eta_0\\gamma_0^r (N+1)^{1-0.5r-b}}{1-0.5r-b}\\right).\n\\end{align*}\nThen, \\cref{eqn:gap_rate} can be obtained by rearranging the terms in the preceding inequality.\n\\end{proof}\n\n\\subsection{Proof of \\cref{Lemma 2.10}}\\label{app:ave_x_second_formula}\nWe use induction to show $\\bar{x}_{N} = \\sum_{k=0}^N \\lambda_{k,N} x_k$ for any $N\\geq 0$. For $N=0$, the relation holds due to the initialization $\\bar{x}_0 :=x_0$ in \\cref{alg:aRB-IRG} and that $\\lambda_{0,0}=1$. Next, let the relation hold for some $N\\geq 0$. From {the hypothesis, equation \\cref{eqn:ave_step_of_alg}, and} that $S_{N}=\\sum_{k=0}^N\\gamma_k^r$ for all $N\\geq 0$, we can write: \n\\begin{align*}\n\t\\bar{x}_{N+1} = \\frac{S_N\\bar{x}_N + \\gamma_{N+1}^r x_{N+1}}{S_{N+1}} \n\t%= \\frac{\\sum_{k = 0}^{N}\\gamma_k^r x_k + \\gamma_{N+1}^r x_{N+1}}{S_{N+1}}\n\t = \\frac{\\sum_{k=0}^{N+1}\\gamma_k^r x_k}{\\sum_{k = 0}^{N+1}\\gamma_k^r} = \\sum_{k=0}^{N+1} \\lambda_{k,N+1} x_k,\n\t\\end{align*}\nimplying that the induction hypothesis holds for $N+1$. Thus, we conclude that the desired averaging formula holds for {all} $N\\geq 0$. To complete the proof, note that since $\\sum_{k=0}^N \\lambda_{k,N}=1$, under the convexity of the set $X$, we have $\\bar x_N \\in X$.\n%\\section*{Acknowledgments}\n%We would like to acknowledge the assistance of volunteers in putting\n%together this example manuscript and supplement.\n\\subsection{Proof of \\cref{Lemma 2.12}}\\label{app:deltas} (a) From \\cref{def:deltas}, we can write:\n\\begin{align*} \n\\EXP{\\Delta_k\\mid \\mathcal{F}_k} =F(x_k)-\\sum_{i=1}^d\\prob{{i}}\\prob{{i}}^{-1}{\\mathbf{U}}_{i}F_{i}(x_k)= F(x_k) -\\sum_{i=1}^d{\\mathbf{U}}_{i}F_{i}(x_k) =0.\n\\end{align*}\nThe relation $\\EXP{\\delta_k\\mid \\mathcal{F}_k}=0$ can be shown in a similar fashion.\n\n\\noindent (b) We can write:\n\\begin{align*}\n &\\EXP{\\|\\Delta_k\\|^2\\mid \\mathcal{F}_k} = \\sum_{i=1}^d\\prob{{i}}\\left\\|F(x_k)-\\prob{{i}}^{-1}{\\mathbf{U}}_{i}F_{i}(x_k)\\right\\|^2\\\\\n &=  \\sum_{i=1}^d\\prob{{i}} \\left(\\|F(x_k)\\|^2 +\\prob{{i}}^{-2} \\left\\|{\\mathbf{U}}_{i}F_{i}(x_k)\\right\\|^2 - 2\\prob{{i}}^{-1}F(x_k)^T{\\mathbf{U}}_{i}F_{i}(x_k)\\right)\\\\\n & = \\|F(x_k)\\|^2 +\\sum_{i=1}^d\\prob{{i}}^{-1} \\left\\|{\\mathbf{U}}_{i}F_{i}(x_k)\\right\\|^2  -2\\sum_{i=1}^d\\|F_i(x_k)\\|^2 \n % \\leq \\left(\\prob{{min}}^{-1}-1\\right)\\|F(x_k)\\|^2 \n \\leq \\left(\\prob{{min}}^{-1}-1\\right)C_F^2.\n\\end{align*}\nThe relation $\\EXP{\\|\\delta_k\\|^2\\mid \\mathcal{F}_k} \\leq  \\left(\\prob{{min}}^{-1}-1\\right)C_f^2$ can be shown using a similar approach. \n\n\n\n\\subsection{Proof of \\cref{Lemma 2.13}}\\label{app:harmonic_bounds} Given $0\\leq \\alpha <1$, let us define the function $\\phi:\\mathbb{R}_{++} \\to \\mathbb{R}$ as $\\phi(x) \\triangleq x^{-\\alpha}$ for all $x>0$. Since $\\alpha >0$, the function $\\phi$ is nonincreasing. We can write:\n\\begin{align*}\n\\sum_{k=0}^{N}\\frac{1}{(k+1)^\\alpha}\n%=\\sum_{k=1}^{N+1}k^{-\\alpha}\n= 1 + \\sum_{k=2}^{N+1}\\frac{1}{k^\\alpha}\\leq 1+\\int_{1}^{N+1}\\frac{dx}{x^\\alpha} =1+\\frac{(N+1)^{1-\\alpha}-1}{1-\\alpha}\\leq \\frac{(N+1)^{1-\\alpha}}{1-\\alpha},\n\\end{align*}\nimplying the desired upper bound. To show that the lower bound holds, we can write:\n\\begin{align*}\n\\sum_{k=0}^{N}\\frac{1}{(k+1)^\\alpha}=\\sum_{k=1}^{N+1}\\frac{1}{k^\\alpha}\\geq \\int_{1}^{N+2}\\frac{dx}{x^\\alpha} \\geq \\int_{1}^{N+1}\\frac{dx}{x^\\alpha}\n%=\\frac{(N+1)^{1-\\alpha}-1}{1-\\alpha} \n\\geq \\frac{(N+1)^{1-\\alpha}-0.5(N+1)^{1-\\alpha}}{1-\\alpha},\n\\end{align*}\nwhere the last inequality is obtained using the assumption that $N \\geq 2^{\\frac{1}{1-\\alpha}}-1$. Therefore, the desired lower bound holds as well. This completes the proof.\n\n ","units":[{"start":40037,"end":40433,"start_line":380,"end_line":382,"raw_tex":"\\begin{lemma}[Weighted averaging]\\label{Lemma 2.10}\nLet $\\{\\bar x_k\\}$ be generated by \\cref{alg:aRB-IRG}. Let us define the weights $\\lambda_{k,N} \\triangleq \\frac{\\gamma_k^r}{\\sum_{j=0}^N \\gamma_j^r}$ for $k \\in \\{0,\\ldots, N\\}$ and $N\\geq 0$. Then, for any $N\\geq 0$, we have $\\bar{x}_{N} = \\sum_{k=0}^N \\lambda_{k,N} x_k$. Also, when $X$ is a convex set, we have $\\bar x_N \\in X$.\n\\end{lemma}","sha256_utf8":"77f7d2cc7e3379a54233611da003b059b9ac835369cdcfe450c57d66b8fec428"},{"start":40434,"end":41474,"start_line":383,"end_line":394,"raw_tex":"\\begin{proof}\n\\begin{comment}\nWe use induction to show that $\\bar{x}_{N} = \\sum_{k=0}^N \\lambda_{k,N} x_k$ for any $N\\geq 0$. For $N=0$, the relation holds due to the initialization $\\bar{x}_0 :=x_0$ in \\cref{alg:aRB-IRG} and that $\\lambda_{0,0}=1$. Next, let the relation hold for some $N\\geq 0$. From \\cref{eqn:ave_step_of_alg} and noting that $S_{N}=\\sum_{k=0}^N\\gamma_k^r$ for any $N\\geq 0$, we can write the following: \n\\begin{align*}\n\t\\bar{x}_{N+1} = \\frac{S_N\\bar{x}_N + \\gamma_{N+1}^r x_{N+1}}{S_{N+1}} \n\t%= \\frac{\\sum_{k = 0}^{N}\\gamma_k^r x_k + \\gamma_{N+1}^r x_{N+1}}{S_{N+1}}\n\t = \\frac{\\sum_{k=0}^{N+1}\\gamma_k^r x_k}{\\sum_{k = 0}^{N+1}\\gamma_k^r} = \\sum_{k=0}^{N+1} \\lambda_{k,N+1} x_k,\n\t\\end{align*}\nimplying that the induction hypothesis holds for $N+1$. Thus, we conclude that the desired averaging formula holds for $N\\geq 0$. To complete the proof, note that since $\\sum_{k=0}^N \\lambda_{k,N}=1$, under the convexity of the set $X$, we have $\\bar x_N \\in X$.\n\\end{comment}\nSee \\cref{app:ave_x_second_formula}. \n\\end{proof}","sha256_utf8":"47f6338e8ea0cf2245e34780b04b970a9e6bde861851eed03949087e9c166252"},{"start":42175,"end":42575,"start_line":402,"end_line":406,"raw_tex":"\\begin{lemma}[Properties of $\\Delta_k$ and $\\delta_k$]\\label{Lemma 2.12}\nConsider \\cref{def:deltas}. We have:\\\\\n\\noindent (a)  $\\EXP{\\Delta_k\\mid \\mathcal{F}_k}=\\EXP{\\delta_k\\mid \\mathcal{F}_k}=0$.\\\\\n\\noindent (b) $\\EXP{\\|\\Delta_k\\|^2\\mid \\mathcal{F}_k} \\leq  \\left(\\prob{{min}}^{-1}-1\\right)C_F^2$ and $\\EXP{\\|\\delta_k\\|^2\\mid \\mathcal{F}_k} \\leq  \\left(\\prob{{min}}^{-1}-1\\right)C_f^2$.\n\\end{lemma}","sha256_utf8":"dcf2dfff93d44c36863963330bbb75f78ff2a116c5817cda1e17fe4aab334ffc"},{"start":42576,"end":43728,"start_line":407,"end_line":425,"raw_tex":"\\begin{proof} \n\\begin{comment}\n\\noindent (a) From the definition of $\\Delta_k$ and ${\\mathbf{U}}_i $, we can write:\n\\begin{align*} \n\\EXP{\\Delta_k\\mid \\mathcal{F}_k} =F(x_k)-\\sum_{i=1}^d\\prob{{i}}\\prob{{i}}^{-1}{\\mathbf{U}}_{i}F_{i}(x_k)= F(x_k) -\\sum_{i=1}^d{\\mathbf{U}}_{i}F_{i}(x_k) =0.\n\\end{align*}\nThe relation $\\EXP{\\delta_k\\mid \\mathcal{F}_k}=0$ can be shown in a similar fashion.\n\n\\noindent (b) We can write:\n\\begin{align*}\n \\EXP{\\|\\Delta_k\\|^2\\mid \\mathcal{F}_k}& = \\sum_{i=1}^d\\prob{{i}}\\left\\|F(x_k)-\\prob{{i}}^{-1}{\\mathbf{U}}_{i}F_{i}(x_k)\\right\\|^2\\\\\n &=  \\sum_{i=1}^d\\prob{{i}} \\left(\\|F(x_k)\\|^2 +\\prob{{i}}^{-2} \\left\\|{\\mathbf{U}}_{i}F_{i}(x_k)\\right\\|^2 - 2\\prob{{i}}^{-1}F(x_k)^T{\\mathbf{U}}_{i}F_{i}(x_k)\\right)\\\\\n & = \\|F(x_k)\\|^2 +\\sum_{i=1}^d\\prob{{i}}^{-1} \\left\\|{\\mathbf{U}}_{i}F_{i}(x_k)\\right\\|^2  -2\\sum_{i=1}^d\\|F_i(x_k)\\|^2\\\\\n & \\leq \\left(\\prob{{min}}^{-1}-1\\right)\\|F(x_k)\\|^2 \\leq \\left(\\prob{{min}}^{-1}-1\\right)C_F^2.\n\\end{align*}\nThe relation $\\EXP{\\|\\delta_k\\|^2\\mid \\mathcal{F}_k} \\leq  \\left(\\prob{{min}}^{-1}-1\\right)C_f^2$ can be shown using a similar approach. \n\\end{comment}\nSee \\cref{app:deltas}\n\\end{proof}","sha256_utf8":"82e5a7610352ff624fd6f32dafcb356b8909f6857a1c8e58e84d0a803f0392c1"},{"start":43818,"end":44150,"start_line":426,"end_line":431,"raw_tex":"\\begin{lemma}[Bounds on the harmonic series]\\label{Lemma 2.13}\nLet $0\\leq \\alpha <1$ be a given scalar. Then, for any integer $N \\geq 2^{\\frac{1}{1-\\alpha}}-1$, we have \n%\\begin{align*}\n$\\frac{(N+1)^{1-\\alpha}}{2(1-\\alpha)}  \\leq \\sum_{k=0}^{N}\\frac{1}{(k+1)^\\alpha}\\leq \\frac{(N+1)^{1-\\alpha}}{1-\\alpha}$.\n%\\end{align*}\n\\end{lemma}","sha256_utf8":"e845f6615c474cdbb3df456e7b8ecee12fe06ac70cd650f396b22468021fbb4d"},{"start":44151,"end":45237,"start_line":432,"end_line":448,"raw_tex":"\\begin{proof}\n\\begin{comment}Given $0\\leq \\alpha <1$, let us define the function $\\phi:\\mathbb{R}_{++} \\to \\mathbb{R}$ as $\\phi(x) \\triangleq x^{-\\alpha}$ for all $x>0$. Since $\\alpha >0$, the function $\\phi$ is nonincreasing. We can write:\n\\begin{align*}\n\\sum_{k=0}^{N}\\frac{1}{(k+1)^\\alpha}\n%=\\sum_{k=1}^{N+1}k^{-\\alpha}\n= 1 + \\sum_{k=2}^{N+1}\\frac{1}{k^\\alpha}\\leq 1+\\int_{1}^{N+1}\\frac{dx}{x^\\alpha} =1+\\frac{(N+1)^{1-\\alpha}-1}{1-\\alpha}\\leq \\frac{(N+1)^{1-\\alpha}}{1-\\alpha},\n\\end{align*}\nimplying the desired upper bound. To show that the lower bound holds, we can write:\n\\begin{align*}\n\\sum_{k=0}^{N}\\frac{1}{(k+1)^\\alpha}=\\sum_{k=1}^{N+1}\\frac{1}{k^\\alpha}\\geq \\int_{1}^{N+2}\\frac{dx}{x^\\alpha} \\geq \\int_{1}^{N+1}\\frac{dx}{x^\\alpha}\n%=\\frac{(N+1)^{1-\\alpha}-1}{1-\\alpha} \n\\geq \\frac{(N+1)^{1-\\alpha}-0.5(N+1)^{1-\\alpha}}{1-\\alpha},\n\\end{align*}\nwhere the last inequality is obtained using the assumption that $N \\geq 2^{\\frac{1}{1-\\alpha}}-1$. Therefore, the desired lower bound holds as well. This completes the proof.\n\\end{comment}\nSee \\cref{app:harmonic_bounds}\n\\end{proof}","sha256_utf8":"173edceaa559ae0d4dcdb3c012d8fabf81ceb90903c4688352a659ab8a162b53"},{"start":45511,"end":46621,"start_line":454,"end_line":465,"raw_tex":"\\begin{lemma}\\label{lem:prebound_gap}\nConsider the sequence $\\{x_k\\}$ in \\cref{alg:aRB-IRG}. Suppose $\\{\\gamma_k\\}$ and $\\{\\eta_k\\}$ are strictly positive sequences. Let \\cref{assum:problem} and \\cref{assum:random sample} hold. Let the auxiliary sequence $\\{u_k\\} \\subset X$ be defined as \n\t\t%\\begin{align}\\label{equ:update_rule_aux_u} \n\t\t{$u_{k+1} \\triangleq \\mathcal{P}_X\\left(u_k-\\gamma_k\\left(\\Delta_k+\\eta_k\\delta_k\\right)\\right)$},\n\t\t%\\end{align}\nwhere $u_0 \\in X$ is an arbitrary vector. Then, for {all $y\\in X$ and all $k \\geq 0$, we have}:\n\\begin{align}\\label{eqn:prebound_gap}\n&\\gamma_k^rF(y)^T(x_k-y) +\\gamma_k^r\\eta_k\\tilde \\nabla f(x_k)^T(x_k-y)\\leq \\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k}, y\\right) +\\|u_k-y\\|^2\\right) \\nonumber\\\\\n&-\\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k+1}, y\\right) +\\|u_{k+1}-y\\|^2\\right){+ \\gamma_k^r(x_k-u_k)^T\\left(\\Delta_k+\\eta_k\\delta_k\\right)}\\nonumber\\\\\n& +\\gamma_k^{r+1}\\left(\\|\\Delta_k\\|^2+\\eta_k^2\\|\\delta_k\\|^2\\right)+ 0.5\\prob{{i_k}}^{-1}\\gamma_k^{1+r}\\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\\end{align}\n\\end{lemma}","sha256_utf8":"fc6594965df5cdb372e2aa5d6e8a17b2c23ed1d4c84831481e3a5d02d0f4be63"},{"start":46623,"end":51110,"start_line":467,"end_line":515,"raw_tex":"\\begin{proof} Let $k\\geq 1$ be fixed. From \\cref{def:err_D} and \\cref{equ:update_rule_aRBIRG}, for any $y \\in X$, we have:\n\t\\begin{align}\\label{eqn:D_split}\n\t\\mathcal{D} \\left(x_{k+1}, y\\right)  =   \\prob{{i_k}}^{-1} \\left\\Vert\\left.  x_{k+1}^{(i_k)}-  y^{(i_k)} \\right\\Vert\\right.^2+\\sum_{i=1, \\thinspace\t i \\neq i_k}^{d} \\prob{i}^{-1} \\left\\Vert\\left.  x_{k}^{(i)}-  y^{(i)} \\right\\Vert\\right.^2.\n\t\\end{align}Next, we find a bound on the term $\\left\\Vert\\left.  x_{k+1}^{(i_k)}- y^{(i_k)} \\right\\Vert\\right.^2$. From the block structure of $X$, we have $y^{(i_k)}\\in X_{i_k}$. Invoking the nonexpansiveness property of the projection mapping, the update rule \\cref{equ:update_rule_aRBIRG}, \\cref{def:reg_map}, and the preceding relation, we obtain:\n\t\t\\begin{align*} \n\t\\left\\Vert\\left. x_{k+1}^{(i_k)}-  y^{(i_k)}\\right\\Vert\\right.^2 \\leq\\left\\Vert\\left.  x_{k}^{(i_k)}-\\gamma_{k}G_{k,i_k} \\left( x_{k}\\right) -y^{(i_k)}\\right\\Vert\\right.^2.\n\t\\end{align*}\n\t\tCombining the preceding relation with \\cref{eqn:D_split}, we obtain:\n\t\\begin{align*}\n\t \t\\mathcal{D}\\left(x_{k+1}, y\\right)& \\leq \n\t \t \\sum_{i=1, \\thinspace\t i \\neq i_k}^{d} \\prob{i}^{-1} \\left\\Vert\\left.  x_{k}^{(i)}-  y^{(i)} \\right\\Vert\\right.^2 +  \\prob{{i_k}}^{-1}\\left\\Vert\\left.  x_{k}^{(i_k)}-  y^{(i_k)} \\right\\Vert\\right.^2 \\nonumber \\\\\n& - 2\\,\\prob{{i_k}}^{-1}\\gamma_k\\left( x^{(i_k)}_{k} - y^{(i_k)}\\right)^TG_{k,i_k}\\left(x_k\\right)+ \\prob{{i_k}}^{-1}\\gamma_k^2 \\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\t\\end{align*}\nInvoking \\cref{def:err_D} again, we obtain:\t\n\t\t\\begin{align}\\label{eqn:erc_D_elementary_ineq1}\n\t \t\\mathcal{D}\\left(x_{k+1}, y\\right) & \\leq  \\mathcal{D}\\left(x_{k}, y\\right) - 2\\,\\prob{{i_k}}^{-1}\\gamma_k\\left( x^{(i_k)}_{k} - y^{(i_k)}\\right)^TG_{k,i_k}\\left(x_k\\right)+ \\prob{{i_k}}^{-1}\\gamma_k^2 \\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\t\\end{align}\nFrom \\cref{def:deltas} and \\cref{def:reg_map}, we can write:\n\\begin{align*}\n&\\prob{{i_k}}^{-1}\\left( x^{(i_k)}_{k} - y^{(i_k)}\\right)^TG_{k,i_k}\\left(x_k\\right) = \\prob{{i_k}}^{-1}(x_k-y)^T\\left({\\mathbf{U}}_{i_k}G_{k,i_k}\\left(x_k\\right)\\right) \\\\ \n&=  \\prob{{i_k}}^{-1}(x_k-y)^T\\left({\\mathbf{U}}_{i_k}F_{i_k}(x_k)+\\eta_k{\\mathbf{U}}_{i_k}\\tilde \\nabla_{i_k}f\\left(x_k\\right)\\right) \\\\\n& {=(x_k-y)^T\\left(F(x_k)-\\Delta_k+\\eta_k\\tilde \\nabla f(x_k)-\\eta_k\\delta_k\\right) = (x_k-y)^T\\left(G_k(x_k)-\\Delta_k-\\eta_k\\delta_k\\right).}\n\\end{align*}\n\tCombining the preceding inequality and relation \\cref{eqn:erc_D_elementary_ineq1}, we obtain:\n\t\\begin{align}\\label{eqn:erc_D_elementary_ineq2}\n\t \t\\mathcal{D}\\left(x_{k+1}, y\\right) & \\leq  \\mathcal{D}\\left(x_{k}, y\\right) {- 2 \\gamma_k(x_k-y)^T\\left(G_k(x_k)-\\Delta_k-\\eta_k\\delta_k\\right)}+ \\prob{{i_k}}^{-1}\\gamma_k^2 \\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\t\\end{align}\n\tConsider the definition of the auxiliary sequence $\\{u_k\\}$ in \\cref{lem:prebound_gap}. Invoking the nonexpansiveness property of the projection again, we can obtain:\n\\begin{align*}\n\\|u_{k+1}-y\\|^2  &\\leq{\\|u_k-\\gamma_k\\left(\\Delta_k+\\eta_k\\delta_k\\right)-y\\|^2}\\\\\n&=\\|u_k-y\\|^2{-2\\gamma_k(u_k-y)^T\\left(\\Delta_k+\\eta_k\\delta_k\\right)} + \\gamma_k^2\\|\\Delta_k+\\eta_k\\delta_k\\|^2.\n\\end{align*}\nThus, we have: \n\\begin{align*}\n\\|u_{k+1}-y\\|^2\\leq \\|u_k-y\\|^2{- 2\\gamma_k(u_k-y)^T\\left(\\Delta_k+\\eta_k\\delta_k\\right)} +2\\gamma_k^2\\|\\Delta_k\\|^2+2\\gamma_k^2\\eta_k^2\\|\\delta_k\\|^2.\n\\end{align*}\nAdding the preceding inequality and the inequality \\cref{eqn:erc_D_elementary_ineq2} together, we obtain:\n\\begin{align}\\label{eqn:erc_D_elementary_ineq3}\n&2\\gamma_k(x_k-y)^T G_k(x_k) \\leq \\left(\\mathcal{D}\\left(x_{k}, y\\right) +\\|u_k-y\\|^2\\right) -\\left(\\mathcal{D}\\left(x_{k+1}, y\\right) +\\|u_{k+1}-y\\|^2\\right)\\nonumber\\\\\n& {+ 2\\gamma_k(x_k-u_k)^T\\left(\\Delta_k+\\eta_k\\delta_k\\right)}+2\\gamma_k^2\\left(\\|\\Delta_k\\|^2+\\eta_k^2\\|\\delta_k\\|^2\\right)+ \\prob{{i_k}}^{-1}\\gamma_k^2 \\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\\end{align}\nFrom the monotonicity property of the mapping $F$ and \\cref{def:reg_map}, we have:\n\\begin{align*}\n(x_k-y)^TG_k(x_k)\n%=(x_k-y)^T\\left(F(x_k)+\\eta_k\\tilde \\nabla f(x_k)\\right) \n\\geq (x_k-y)^TF(y)+\\eta_k\\tilde \\nabla f(x_k)^T(x_k-y).\n\\end{align*}\nThis provides a lower bound on the left-hand side of \\cref{eqn:erc_D_elementary_ineq3}. The inequality \\cref{eqn:prebound_gap} is obtained by substituting this bound in \\cref{eqn:erc_D_elementary_ineq3} and multiplying both sides by $\\frac{\\gamma_{k}^{r-1}}{2}$.\n\\end{proof}","sha256_utf8":"107f5a742d72f72f9cdb96a5f1b22ab11c47f59f1f1041fea0e88642d8986aa0"},{"start":51360,"end":52531,"start_line":518,"end_line":528,"raw_tex":"\\begin{proposition}[Error bounds for \\cref{alg:aRB-IRG}]\\label{prop:dual_gap_bound}\nLet the sequence $\\{\\bar x_k\\}$ be generated by \\cref{alg:aRB-IRG}{, where $0\\leq r<1$}. Suppose $\\{\\gamma_k\\}$ and $\\{\\eta_k\\}$ are strictly positive and nonincreasing sequences. Let \\cref{assum:problem} and \\cref{assum:random sample} hold {and} assume {that} the set $X$ is bounded, i.e., {$\\|x\\|\\leq M$} for all $x \\in X$ and some $M>0$. \\\\\n\\noindent (a) Let $x^*$ be an optimal solution to problem \\cref{prob:main}. Then, for {all} $N\\geq 1$:\n\\begin{align}\\label{eqn:f_bound} \n&\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{\\frac{4M^2\\gamma_{{N}}^{r-1}}{\\eta_{N}} +\\sum_{k=0}^{N}\\eta_k^{-1}\\gamma_k^{r+1}\\left(C_F^2+\\eta_k^2C_f^2\\right)}{\\prob{{min}}\\sum_{k=0}^{N}\\gamma_k^r}.\n\\end{align}\n\\noindent (b) Consider the dual gap function in \\cref{def:gap}. Then, for {all} $N\\geq 1$:\n\\begin{align}\\label{eqn:gap_bound} \n\\EXP{\\mathrm{GAP}\\left(\\bar{x}_N\\right)} \\leq\\frac{ {4 M^2 \\gamma_{{N}}^{r-1} }  +\\sum_{k=0}^{N}\\gamma_{{k}}^r\\left( 2\\prob{{min}}\\eta_kC_fM+\\gamma_k C_F^2 +  \\gamma_{k}\\eta_k^2C_f ^2 \\right)}{\\prob{{min}}\\sum_{k=0}^{N} \\gamma_{{k}}^r}.\n\\end{align}\n\\end{proposition}","sha256_utf8":"547ef868d469f1e4adad12508140b2d7fe9c230b4234254007680f7475cc3c7d"},{"start":52533,"end":64049,"start_line":530,"end_line":641,"raw_tex":"\\begin{proof}\nWe define the following terms for all $k\\geq 0$, that appear in \\cref{eqn:prebound_gap}:\n\\begin{align}\\label{eqn:three_terms_abr}\n& {\\Theta_{k,1} \\triangleq \\gamma_k^r(x_k-u_k)^T\\left(\\Delta_k+\\eta_k\\delta_k\\right)}, \\quad \\Theta_{k,2} \\triangleq \\gamma_k^{r+1}\\left(\\|\\Delta_k\\|^2+\\eta_k^2\\|\\delta_k\\|^2\\right),\\nonumber \\\\ \n&\\Theta_{k,3} \\triangleq 0.5\\prob{{i_k}}^{-1}\\gamma_k^{1+r}\\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\\end{align}\nNext, we estimate the expected values of these terms. {Consider the notation of} $\\mathcal{F}_k$ given by \\cref{def:Fk}. Note that $x_k$ is $\\mathcal{F}_k$--measurable. Also, from the definition of $u_k$ in \\cref{lem:prebound_gap}, $u_k$ is $\\mathcal{F}_k$--measurable. Note, however, that $\\Theta_{k,j}$ is $\\mathcal{F}_{k+1}$--measurable for all $j \\in \\{1,2,3\\}$. Taking these into account and using the total probability law, for any $k\\geq 0$ and $j \\in \\{1,2,3\\}$, we have $ \\EXP{\\Theta_{k,j}} = \\mathsf{E}_{\\mathcal{F}_k}\\left[\\mathsf{E}_{i_k}\\left[\\Theta_{k,j}\\mid \\mathcal{F}_k\\right]\\right]$. From this relation and \\cref{Lemma 2.12}, we have for any $k\\geq 0$:\n\\begin{align}\\label{eqn:estimate_deltas_1_2}\n&\\EXP{\\Theta_{k,1}} = 0, \\qquad \\EXP{\\Theta_{k,2}} = \\left(\\prob{{min}}^{-1}-1\\right)\\gamma_k^{r+1}\\left(C_F^2+\\eta_k^2C_f^2\\right).\n\\end{align}\nAlso, using \\cref{def:reg_map} and the triangle inequality, we can write:\n\\begin{align*}\n&{\\mathsf{E}_{i_k}\\left[\\Theta_{k,3}\\mid \\mathcal{F}_k\\right]} = \\sum_{i=1}^d\\prob{i}\\left(0.5\\prob{{i}}^{-1}\\gamma_k^{1+r}\\left\\Vert\\left.G_{k,i}\\left(x_k\\right)\\right\\Vert\\right.^2\\right)\\nonumber\\\\\n&\\leq \\gamma_k^{1+r}\\sum_{i=1}^d\\left(\\|F_i(x_k)\\|^2+\\eta_k^2\\|\\tilde \\nabla_i f(x_k)\\|^2\\right)  = \\gamma_k^{1+r}\\|F(x_k)\\|^2+\\eta_k^2\\|\\tilde \\nabla f(x_k)\\|^2.\n\\end{align*}\n{From the preceding inequality, we obtain:\n\\begin{align}\\label{eqn:estimate_deltas_3}\n\\EXP{\\Theta_{k,3}} & \\leq  \\gamma_k^{1+r}\\left(C_F^2+\\eta_k^2C_f^2\\right).\n\\end{align}\n}We are now ready to show the inequalities \\cref{eqn:f_bound} and \\cref{eqn:gap_bound} as follows:\\\\\n\\noindent (a) Consider \\cref{eqn:prebound_gap}. From the definition of subgradients of the convex function $f$, we have that $f(x_k)-f(y) \\leq \\tilde \\nabla f(x_k)^T(x_k-y)$. Thus, {from \\cref{eqn:three_terms_abr}} we obtain for any $y \\in X$:\n\\begin{align*}\n&\\gamma_k^rF(y)^T(x_k-y) +\\gamma_k^r\\eta_k\\left(f(x_k)-f(y)\\right)\\leq \\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k}, y\\right) +\\|u_k-y\\|^2\\right) \\nonumber\\\\\n&-\\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k+1}, y\\right) +\\|u_{k+1}-y\\|^2\\right)+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}.\n\\end{align*}\nLet us substitute $y:=x^*$, where $x^*$ denotes an optimal solution to \\cref{prob:main}. Note that $x^*$ must be a feasible solution to \\cref{prob:main}, i.e., $F(x^*)^T(x_k-x^*) \\geq 0$. Thus, we obtain:\n\\begin{align}\\label{eqn:gap_prop_ineq1_a}\n\t\t\t&\\gamma_k^r\\eta_k\\left(f(x_k)-f(x^*)\\right) \\leq  \\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k}, x^*\\right) +\\|u_k-x^*\\|^2\\right) \\nonumber\\\\\n&-\\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k+1}, x^*\\right) +\\|u_{k+1}-x^*\\|^2\\right)+\\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}.\n\\end{align}\nDividing both sides by $\\eta_k$ and adding and subtracting $\\frac{\\gamma_{{k-1}}^{r-1}}{2\\eta_{k-1}} \\left( \\mathcal{D}\\left(x_k,x^*\\right) + \\left\\Vert u_k-x^* \\right\\Vert^2 \\right)$ in the right-hand side of \\cref{eqn:gap_prop_ineq1_a}, we obtain for $k\\geq 1$:\n\\begin{align}\\label{eqn:gap_prop_ineq2_a}\n&\\gamma_k^r\\left(f(x_k)-f(x^*)\\right) \\leq  \\frac{\\gamma_{{k-1}}^{r-1}}{2\\eta_{k-1}} \\left(  \\mathcal{D}\\left(x_k,x^*\\right)  + \\left\\Vert u_k-x^* \\right\\Vert^2 \\right) \\nonumber \\\\\n&- \\frac{\\gamma_{{k}}^{r-1}}{2\\eta_k} \\left(  \\mathcal{D}\\left(x_{k+1},x^*\\right)  + \\left\\Vert u_{k+1}-x^* \\right\\Vert^2 \\right)\\nonumber \\\\ \n&+ \\frac{1}{2} \\left(\\frac{\\gamma_{{k}}^{r-1}}{\\eta_k} - \\frac{\\gamma_{{k-1}}^{r-1}}{\\eta_{k-1}}\\right)\\left( \\mathcal{D}\\left(x_k,x^*\\right) + \\left\\Vert u_k-x^* \\right\\Vert^2 \\right)+\\eta_k^{-1}\\left( \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right).\n\\end{align}\nSince $r-1 < 0$ and that $\\{\\gamma_k\\}$ and $\\{\\eta_k\\}$ are nonincreasing, we have $\\frac{\\gamma_{{k}}^{r-1}}{\\eta_k} - \\frac{\\gamma_{{k-1}}^{r-1}}{\\eta_{k-1}} \\geq 0$. Also, from the boundedness of the set $X$, since $x_k$, $x^*$, and {$u_k$} belong to $ X$, using \\cref{rem:err_D} and the triangle inequality, we have:\n\\begin{align}\\label{eqn:gap_prop_ineq3_a}\n\\mathcal{D}\\left(x_k,x^*\\right)  + \\left\\Vert u_k-x^* \\right\\Vert^2 \n\\leq \\prob{{min}}^{-1}\\left\\Vert x_k-x^* \\right\\Vert^2 + \\left\\Vert u_k-x^* \\right\\Vert^2\n\\leq 4M^2\\left(\\prob{{min}}^{-1}+1\\right)\\leq \\frac{8M^2}{\\prob{{min}}}.\n\\end{align}\nSumming over \\cref{eqn:gap_prop_ineq2_a} from $k=1$ to $N$ and using \\cref{eqn:gap_prop_ineq3_a}, we obtain:\n\\begin{align*}\n&\\sum_{k=1}^{N}\\gamma_k^r\\left(f(x_k)-f(x^*)\\right) \\leq \\frac{\\gamma_{0}^{r-1}}{2\\eta_0} \\left(  \\mathcal{D}\\left(x_1,x^*\\right)  + \\left\\Vert u_1-x^* \\right\\Vert^2 \\right) \n\\nonumber \\\\ \n&+4M^2\\prob{{min}}^{-1}\\left(\\frac{\\gamma_{{N}}^{r-1}}{\\eta_{N}} - \\frac{\\gamma_{{0}}^{r-1}}{\\eta_{0}}\\right)+\\sum_{k=1}^{N}\\eta_k^{-1}\\left(\\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right),\n\\end{align*}\nwhere we drop the nonpositive term. From relation \\cref{eqn:gap_prop_ineq1_a} when $k=0$, we have:\n\\begin{align*}\n\\gamma_0^r\\left(f(x_0)-f(x^*)\\right)& \\leq  \\frac{\\gamma_0^{r-1}}{2\\eta_0}\\left(\\mathcal{D}\\left(x_{0}, x^*\\right) +\\|u_0-x^*\\|^2\\right) \\nonumber\\\\\n&-\\frac{\\gamma_0^{r-1}}{2\\eta_0}\\left(\\mathcal{D}\\left(x_{1}, x^*\\right) +\\|u_{1}-x^*\\|^2\\right)+ \\eta_0^{-1}\\left(\\Theta_{0,1}+\\Theta_{0,2}+\\Theta_{0,3}\\right).\n\\end{align*}\nAdding the last two inequalities, {multiplying and dividing the left-hand side by $\\sum_{k=0}^{N}\\gamma_k^r$, and then, invoking} \\cref{Lemma 2.10} and convexity of $f$, we obtain:\n\\begin{align*}\n&\\left(\\sum_{k=0}^{N}\\gamma_k^r\\right)\\left(f\\left(\\bar x_N\\right)-f(x^*)\\right) \\leq \\frac{\\gamma_{0}^{r-1}}{2\\eta_0} \\left(  \\mathcal{D}\\left(x_0,x^*\\right)  + \\left\\Vert u_0-x^* \\right\\Vert^2 \\right) \n\\nonumber \\\\ \n&+4M^2\\prob{{min}}^{-1}\\left(\\frac{\\gamma_{{N}}^{r-1}}{\\eta_{N}} - \\frac{\\gamma_{{0}}^{r-1}}{\\eta_{0}}\\right)+\\sum_{k=0}^{N}\\eta_k^{-1}\\left( \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right).\n\\end{align*}\nTaking the expectation on both sides and invoking \\cref{eqn:gap_prop_ineq3_a}, we obtain:\n\\begin{align*}\n&\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{\\frac{4M^2\\prob{{min}}^{-1}\\gamma_{{N}}^{r-1}}{\\eta_{N}} +\\sum_{k=0}^{N}\\eta_k^{-1}\\EXP{\\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}}}{\\sum_{k=0}^{N}\\gamma_k^r}.\n\\end{align*}\nFrom the relations \\cref{eqn:estimate_deltas_1_2} and \\cref{eqn:estimate_deltas_3}, we obtain:\n\\begin{align*}\n&\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{\\frac{4M^2\\prob{{min}}^{-1}\\gamma_{{N}}^{r-1}}{\\eta_{N}}+\\sum_{k=0}^{N}\\eta_k^{-1}\\left(\\prob{{min}}^{-1}\\gamma_k^{r+1}\\left(C_F^2+\\eta_k^2C_f^2\\right)\\right)}{\\sum_{k=0}^{N}\\gamma_k^r},\n\\end{align*}\nwhich implies the inequality \\cref{eqn:f_bound}. \n\n\\noindent (b) From the Cauchy-Schwarz inequality, the definitions of $C_f$ and $M$, and the triangle inequality, we have $ \\tilde \\nabla f(x_k)^T(y-x_k) \\leq \\left\\| \\tilde \\nabla f(x_k)\\right\\|\\|x_k-y\\| \\leq 2C_f M.$\nAdding the preceding inequality with the relation \\cref{eqn:prebound_gap}, {from \\cref{eqn:three_terms_abr}} we obtain:\n\\begin{align}\\label{eqn:prop_gap_proof_ineq0}\n&\\gamma_k^rF(y)^T(x_k-y) \\leq \\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k}, y\\right) +\\|u_k-y\\|^2\\right) \\nonumber\\\\\n&-\\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k+1}, y\\right) +\\|u_{k+1}-y\\|^2\\right)+ 2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}.\n\\end{align}\nAdding and subtracting the term $\\frac{\\gamma_{{k-1}}^{r-1}}{2} \\left( \\mathcal{D}\\left(x_k,y\\right) + \\left\\Vert u_k-y \\right\\Vert^2 \\right)$, we obtain:\n\\begin{align*}\n\t\t\t&\\gamma_k^rF(y)^T(x_k-y) \\leq  \\frac{\\gamma_{{k-1}}^{r-1}}{2} \\left(  \\mathcal{D}\\left(x_k,y\\right)  + \\left\\Vert u_k-y \\right\\Vert^2 \\right) \\nonumber \\\\\n\t\t\t&- \\frac{\\gamma_{{k}}^{r-1}}{2} \\left(  \\mathcal{D}\\left(x_{k+1},y\\right)  + \\left\\Vert u_{k+1}-y \\right\\Vert^2 \\right)+ \\frac{1}{2} \\left(\\gamma_{{k}}^{r-1} - \\gamma_{{k-1}}^{r-1}\\right)\\left( \\mathcal{D}\\left(x_k,y\\right) + \\left\\Vert u_k-y \\right\\Vert^2 \\right)\\nonumber \\\\\n&+ 2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}.\n\\end{align*}\nSubstituting the bound given by \\cref{eqn:gap_prop_ineq3_a} in the preceding relation, we obtain:\n\\begin{align*}\n\t\t\t&\\gamma_k^rF(y)^T(x_k-y) \\leq  \\frac{\\gamma_{{k-1}}^{r-1}}{2} \\left(  \\mathcal{D}\\left(x_k,y\\right)  + \\left\\Vert u_k-y \\right\\Vert^2 \\right) \\nonumber \\\\\n\t\t\t&- \\frac{\\gamma_{{k}}^{r-1}}{2} \\left(  \\mathcal{D}\\left(x_{k+1},y\\right)  + \\left\\Vert u_{k+1}-y \\right\\Vert^2 \\right)+ 4M^2\\prob{{min}}^{-1} \\left(\\gamma_{{k}}^{r-1} - \\gamma_{{k-1}}^{r-1}\\right)\\nonumber \\\\\n&+ 2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}.\n\\end{align*}\nSumming both sides from $k = 1 $ to $N$, we obtain:\n\\begin{align}\\label{eqn:prop_gap_proof_ineq1}\n\\sum_{k=1}^{N}\\gamma_{{k}}^r F(y)^T \\left(x_k-y\\right)& \\leq  \\frac{\\gamma_{{0}}^{r-1}}{2} \\left( \\mathcal{D}\\left(x_1,y\\right) + \\left\\Vert u_1-y \\right\\Vert^2 \\right)+ 4 M^2\\prob{{min}}^{-1} \\left( \\gamma_{{N}}^{r-1} - \\gamma_{{0}}^{r-1}\\right) \\nonumber \\\\\n\t\t\t%&- \\frac{\\gamma_{{N}}^{r-1}}{2} \\left(  \\mathcal{D}\\left(x_{N},y\\right)  + \\left\\Vert u_{N}-y \\right\\Vert^2 \\right) \\nonumber \\\\\n\t\t\t& +\\sum_{k=1}^{N}\\left(  2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right).\n\\end{align}\nWriting the inequality \\cref{eqn:prop_gap_proof_ineq0} for $k =0$, we have:\n\t\\begin{align}\\label{eqn:prop_gap_proof_ineq2}\n\\gamma_0^rF(y)^T(x_0-y) &\\leq \\frac{\\gamma_0^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{0}, y\\right) +\\|u_0-y\\|^2\\right)-\\frac{\\gamma_0^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{1}, y\\right) +\\|u_{1}-y\\|^2\\right) \\nonumber\\\\\n&+ 2\\gamma_0^r\\eta_0C_f M+ \\Theta_{0,1}+\\Theta_{0,2}+\\Theta_{0,3}.\n\\end{align}\nAdding \\cref{eqn:prop_gap_proof_ineq1} and \\cref{eqn:prop_gap_proof_ineq2} together, we obtain:\n\\begin{align}\\label{eqn:prop_gap_proof_ineq3}\n\\sum_{k=0}^{N}\\gamma_{{k}}^r F(y)^T \\left(x_k-y\\right) &\\leq  \\frac{\\gamma_{{0}}^{r-1}}{2} \\left( \\mathcal{D}\\left(x_0,y\\right) + \\left\\Vert u_0 -y \\right\\Vert^2 \\right) + 4 M^2\\prob{{min}}^{-1} \\left( \\gamma_{{N}}^{r-1} - \\gamma_{{0}}^{r-1}\\right) \\nonumber \\\\& +\\sum_{k=0}^{N}\\left(  2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right).\n\\end{align}\nRecalling $\\bar{x}_{N} = \\sum_{k=0}^N \\lambda_{k,N} x_k$ in \\cref{Lemma 2.10}, applying the bound given by \\cref{eqn:gap_prop_ineq3_a}, and using the triangle inequality, we obtain:\n\t\\begin{align*}\n\\left(\\sum_{k=0}^{N} \\gamma_{{k}}^r \\right) F(y)^T \\left(\\bar{x}_N-y\\right) \\leq& {4 M^2\\prob{{min}}^{-1}  \\gamma_{{N}}^{r-1} }  +\\sum_{k=0}^{N}\\left(  2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right).\n\\end{align*}\nTaking the supremum with respect to $y$ over the set $X$ from the left-hand side, invoking \\cref{def:gap}, and then dividing both sides by $\\sum_{k=0}^{N} \\gamma_{{k}}^r $, we obtain:\n\t\\begin{align*}\n\\text{GAP}\\left(\\bar x_N\\right) \\leq \\frac{4 M^2\\prob{{min}}^{-1}  \\gamma_{{N}}^{r-1}   +\\sum_{k=0}^{N}\\left(  2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right)}{\\sum_{k=0}^{N} \\gamma_{{k}}^r }.\n\\end{align*}\nTaking the expectation on both sides, using the relations \\cref{eqn:estimate_deltas_1_2} and \\cref{eqn:estimate_deltas_3}, and rearranging the terms, we obtain the inequality \\cref{eqn:gap_bound}.\n\\end{proof}","sha256_utf8":"ebc9c4301e75fabeed8684375d096c873ce00354b290b7f025a83c12a85239aa"},{"start":65385,"end":69288,"start_line":654,"end_line":682,"raw_tex":"\\begin{proof} Let us define the following terms: \n\\begin{align*}%\\label{eqn:three_terms_abr2}\n& \\Lambda_{N,1} \\triangleq \\prob{{min}}\\sum_{k=0}^{N}\\gamma_k^r, \\quad  \\Lambda_{N,2}\\triangleq  \\frac{4M^2\\gamma_{{N}}^{r-1}}{\\eta_{N}} ,\\quad \\Lambda_{N,3} \\triangleq \\left(C_F^2+\\eta_0^2C_f^2\\right)\\sum_{k=0}^{N}\\eta_k^{-1}\\gamma_k^{r+1},\\nonumber\\\\ \n& \\Lambda_{N,4} \\triangleq 4M^2\\gamma_{{N}}^{r-1}, \\quad  \\Lambda_{N,5} \\triangleq \\left(C_F^2+\\eta_0^2C_f^2\\right)\\sum_{k=0}^{N}\\gamma_k^{r+1}, \\quad \\Lambda_{N,6} \\triangleq 2\\prob{{min}}C_fM \\sum_{k=0}^{N}\\eta_k\\gamma_k^r.\n\\end{align*}\nNote that from \\cref{eqn:f_bound} and \\cref{eqn:gap_bound}, we have:\n\\begin{align}\\label{eqn:bounds_in_Lambdas}\n\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{\\Lambda_{N,2}+\\Lambda_{N,3}}{\\Lambda_{N,1}}, \\quad \\EXP{\\text{GAP}\\left(\\bar{x}_N\\right)} \\leq \\frac{\\Lambda_{N,4}+\\Lambda_{N,5}+\\Lambda_{N,6}}{\\Lambda_{N,1}}.\n\\end{align}\nNext, we apply \\cref{Lemma 2.13} to estimate the terms $\\Lambda_{N,i}$. Substituting $\\gamma_k$ and $\\eta_k$ by their update rules, we obtain:\n\\begin{align*}%\\label{eqn:Lambda1}\n\\Lambda_{N,1} &=\\prob{{min}}\\sum_{k=0}^{N}\\frac{\\gamma_0^r}{(k+1)^{0.5r}}\\geq \\frac{\\prob{{min}}\\gamma_0^r(N+1)^{1-0.5r}}{2(1-0.5r)},\\\\  \n\\Lambda_{N,2} &= \\frac{4M^2(N+1)^{0.5(1-r)+b}}{\\eta_0\\gamma_0^{1-r}},\\quad \\Lambda_{N,4} = \\frac{4M^2(N+1)^{0.5(1-r)}}{\\gamma_0^{1-r}}, \\\\\n\\Lambda_{N,3} &= \\sum_{k=0}^{N}\\frac{\\left(C_F^2+\\eta_0^2C_f^2\\right)\\gamma_0^{1+r}}{\\eta_0(k+1)^{0.5(1+r)-b}} \\leq\\frac{\\gamma_0^{1+r}\\left(C_F^2+\\eta_0^2C_f^2\\right)(N+1)^{1-0.5(1+r)+b}}{\\eta_0(1-0.5(1+r)+b)},\\\\\n\\Lambda_{N,5} &= \\left(C_F^2+\\eta_0^2C_f^2\\right)\\sum_{k=0}^{N}\\frac{\\gamma_0^{r+1}}{(k+1)^{0.5(1+r)}}\\leq \\frac{\\left(C_F^2+\\eta_0^2C_f^2\\right)\\gamma_0^{r+1}(N+1)^{1-0.5(1+r)}}{1-0.5(1+r)},\\\\\n\\Lambda_{N,6} &=2\\prob{{min}}C_fM \\eta_0\\gamma_0^r\\sum_{k=0}^{N}\\frac{1}{(k+1)^{0.5r+b}}\\leq  \\frac{2\\prob{{min}}C_fM \\eta_0\\gamma_0^r (N+1)^{1-0.5r-b}}{1-0.5r-b}.\n\\end{align*}\nFor these inequalities to hold, we need to ensure that {the} conditions of \\cref{Lemma 2.13} are met. Accordingly, we must have $0\\leq 0.5r <1$, $0\\leq 0.5(1+r)-b <1$, $0\\leq 0.5r+b<1$, and $0 \\leq 0.5(1+r)<1$. These relations hold because $0\\leq r<1$ and $0<b<0.5$. Another set of conditions when applying \\cref{Lemma 2.13} includes $N\\geq \\max\\left\\{2^{1/(1-0.5r)},2^{1/(1-0.5(1+r)+b)},2^{1/(1-0.5r-b)},2^{1/(1-0.5(1+r))}\\right\\}-1$. {This relation is indeed} satisfied as a consequence of $N\\geq 2^{\\frac{2}{1-r}}-1$, {$0<b<0.5$}, and $0\\leq r<1$. We conclude that all the necessary conditions for applying \\cref{Lemma 2.13} and obtaining the aforementioned bounds for the terms $\\Lambda_{N,i}$ are satisfied. To show that the inequalities \\cref{eqn:f_rate} and \\cref{eqn:gap_rate} hold, it suffices to substitute the preceding {bounds on} the terms $\\Lambda_{N,i}$ into the two inequalities given by \\cref{eqn:bounds_in_Lambdas}. The details are as follows:\n\\begin{align*}\n&\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{\\Lambda_{N,2}+\\Lambda_{N,3}}{\\Lambda_{N,1}}=\\frac{2-r}{\\prob{{min}}\\gamma_0^r(N+1)^{1-0.5r}}\\left(\\frac{4M^2(N+1)^{0.5-0.5r+b}}{\\eta_0\\gamma_0^{1-r}}\\right.\\\\\n&\\left.+\\left(\\frac{\\gamma_0^{1+r}}{\\eta_0}\\right)\\frac{\\left(C_F^2+\\eta_0^2C_f^2\\right)(N+1)^{0.5-0.5r+b}}{0.5-0.5r+b}\\right). \n\\end{align*}\nThe inequality \\cref{eqn:f_rate} is obtained by rearranging the terms in the preceding relation.\n\\begin{align*}\n&\\EXP{\\text{GAP}\\left(\\bar{x}_N\\right)} \\leq \\frac{\\Lambda_{N,4}+\\Lambda_{N,5}+\\Lambda_{N,6}}{\\Lambda_{N,1}} \\leq\\frac{2-r}{\\prob{{min}}\\gamma_0^r(N+1)^{1-0.5r}}\\left(\\frac{4M^2(N+1)^{0.5-0.5r}}{\\gamma_0^{1-r}} \\right.\\\\\n&\\left.+\\frac{\\left(C_F^2+\\eta_0^2C_f^2\\right)\\gamma_0^{r+1}(N+1)^{0.5-0.5r}}{0.5-0.5r}+ \\frac{2\\prob{{min}}C_fM \\eta_0\\gamma_0^r (N+1)^{1-0.5r-b}}{1-0.5r-b}\\right).\n\\end{align*}\nThen, \\cref{eqn:gap_rate} can be obtained by rearranging the terms in the preceding inequality.\n\\end{proof}","sha256_utf8":"cdea17fe10aca883d97f9824ee3dbadc0e7dcaf1eebf470249fff6a69a752d6c"},{"start":106027,"end":109365,"start_line":975,"end_line":1018,"raw_tex":"\\subsection{Proof of \\cref{Lemma 2.10}}\\label{app:ave_x_second_formula}\nWe use induction to show $\\bar{x}_{N} = \\sum_{k=0}^N \\lambda_{k,N} x_k$ for any $N\\geq 0$. For $N=0$, the relation holds due to the initialization $\\bar{x}_0 :=x_0$ in \\cref{alg:aRB-IRG} and that $\\lambda_{0,0}=1$. Next, let the relation hold for some $N\\geq 0$. From {the hypothesis, equation \\cref{eqn:ave_step_of_alg}, and} that $S_{N}=\\sum_{k=0}^N\\gamma_k^r$ for all $N\\geq 0$, we can write: \n\\begin{align*}\n\t\\bar{x}_{N+1} = \\frac{S_N\\bar{x}_N + \\gamma_{N+1}^r x_{N+1}}{S_{N+1}} \n\t%= \\frac{\\sum_{k = 0}^{N}\\gamma_k^r x_k + \\gamma_{N+1}^r x_{N+1}}{S_{N+1}}\n\t = \\frac{\\sum_{k=0}^{N+1}\\gamma_k^r x_k}{\\sum_{k = 0}^{N+1}\\gamma_k^r} = \\sum_{k=0}^{N+1} \\lambda_{k,N+1} x_k,\n\t\\end{align*}\nimplying that the induction hypothesis holds for $N+1$. Thus, we conclude that the desired averaging formula holds for {all} $N\\geq 0$. To complete the proof, note that since $\\sum_{k=0}^N \\lambda_{k,N}=1$, under the convexity of the set $X$, we have $\\bar x_N \\in X$.\n%\\section*{Acknowledgments}\n%We would like to acknowledge the assistance of volunteers in putting\n%together this example manuscript and supplement.\n\\subsection{Proof of \\cref{Lemma 2.12}}\\label{app:deltas} (a) From \\cref{def:deltas}, we can write:\n\\begin{align*} \n\\EXP{\\Delta_k\\mid \\mathcal{F}_k} =F(x_k)-\\sum_{i=1}^d\\prob{{i}}\\prob{{i}}^{-1}{\\mathbf{U}}_{i}F_{i}(x_k)= F(x_k) -\\sum_{i=1}^d{\\mathbf{U}}_{i}F_{i}(x_k) =0.\n\\end{align*}\nThe relation $\\EXP{\\delta_k\\mid \\mathcal{F}_k}=0$ can be shown in a similar fashion.\n\n\\noindent (b) We can write:\n\\begin{align*}\n &\\EXP{\\|\\Delta_k\\|^2\\mid \\mathcal{F}_k} = \\sum_{i=1}^d\\prob{{i}}\\left\\|F(x_k)-\\prob{{i}}^{-1}{\\mathbf{U}}_{i}F_{i}(x_k)\\right\\|^2\\\\\n &=  \\sum_{i=1}^d\\prob{{i}} \\left(\\|F(x_k)\\|^2 +\\prob{{i}}^{-2} \\left\\|{\\mathbf{U}}_{i}F_{i}(x_k)\\right\\|^2 - 2\\prob{{i}}^{-1}F(x_k)^T{\\mathbf{U}}_{i}F_{i}(x_k)\\right)\\\\\n & = \\|F(x_k)\\|^2 +\\sum_{i=1}^d\\prob{{i}}^{-1} \\left\\|{\\mathbf{U}}_{i}F_{i}(x_k)\\right\\|^2  -2\\sum_{i=1}^d\\|F_i(x_k)\\|^2 \n % \\leq \\left(\\prob{{min}}^{-1}-1\\right)\\|F(x_k)\\|^2 \n \\leq \\left(\\prob{{min}}^{-1}-1\\right)C_F^2.\n\\end{align*}\nThe relation $\\EXP{\\|\\delta_k\\|^2\\mid \\mathcal{F}_k} \\leq  \\left(\\prob{{min}}^{-1}-1\\right)C_f^2$ can be shown using a similar approach. \n\n\n\n\\subsection{Proof of \\cref{Lemma 2.13}}\\label{app:harmonic_bounds} Given $0\\leq \\alpha <1$, let us define the function $\\phi:\\mathbb{R}_{++} \\to \\mathbb{R}$ as $\\phi(x) \\triangleq x^{-\\alpha}$ for all $x>0$. Since $\\alpha >0$, the function $\\phi$ is nonincreasing. We can write:\n\\begin{align*}\n\\sum_{k=0}^{N}\\frac{1}{(k+1)^\\alpha}\n%=\\sum_{k=1}^{N+1}k^{-\\alpha}\n= 1 + \\sum_{k=2}^{N+1}\\frac{1}{k^\\alpha}\\leq 1+\\int_{1}^{N+1}\\frac{dx}{x^\\alpha} =1+\\frac{(N+1)^{1-\\alpha}-1}{1-\\alpha}\\leq \\frac{(N+1)^{1-\\alpha}}{1-\\alpha},\n\\end{align*}\nimplying the desired upper bound. To show that the lower bound holds, we can write:\n\\begin{align*}\n\\sum_{k=0}^{N}\\frac{1}{(k+1)^\\alpha}=\\sum_{k=1}^{N+1}\\frac{1}{k^\\alpha}\\geq \\int_{1}^{N+2}\\frac{dx}{x^\\alpha} \\geq \\int_{1}^{N+1}\\frac{dx}{x^\\alpha}\n%=\\frac{(N+1)^{1-\\alpha}-1}{1-\\alpha} \n\\geq \\frac{(N+1)^{1-\\alpha}-0.5(N+1)^{1-\\alpha}}{1-\\alpha},\n\\end{align*}\nwhere the last inequality is obtained using the assumption that $N \\geq 2^{\\frac{1}{1-\\alpha}}-1$. Therefore, the desired lower bound holds as well. This completes the proof.\n\n ","sha256_utf8":"3a534ac1322bc94d8d924dde2e7bb2babeafac1f199d4cdeb32f13a3a3c7e4ba"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"变分不等式约束优化问题","source_units":[{"start":7955,"end":8144,"start_line":194,"end_line":202,"raw_tex":"\\begin{equation}\\label{prob:main}\n\\tag{$\\text{P}^f_{\\text{VI}}$} \n\\begin{aligned}\n& {\\text{minimize}}\n& & f(x) \\\\\n& \\text{subject to}\n& & x \\in \\text{SOL}(X,F).\n\\end{aligned}\n\\end{equation}","sha256_utf8":"4984453dd494d2bffbcb96fc28b54e4f3c6aef71ac81d947d65268bf5a539117"},{"start":8145,"end":9636,"start_line":203,"end_line":203,"raw_tex":"Here, $f:\\mathbb{R}^n \\to \\mathbb{R}$ is a convex function and $X \\subseteq \\mathbb{R}^n$ is given as a Cartesian product, i.e., $ X \\triangleq \\prod_{i = 1}^{d}  X_i $, where $X_i \\subseteq\\mathbb{R}^{n_i}$ is convex for all $i=1,\\dots, d$ and $\\sum_{i=1}^dn_i=n$. We consider $F: X \\to \\mathbb{R}^n$ to be a monotone mapping. The term $\\text{SOL}(X,F)$ denotes the solution set of the variational inequality $\\text{VI}(X,F)$ defined as follows:  A vector $x \\in X$ is said to be a solution to $\\text{VI}(X,F)$ if for any $y \\in X$, we have $F(x)^T(y-x) \\geq 0$. Variational inequalities, first introduced in late 1950s, are an immensely powerful mathematical tool that can serve as a unifying framework for capturing a wide range of applications arising in operations research, finance, and economics (cf. \\cite{FacchineiPang2003, Rockafellar98,Lan-VI-13,wang2015}). Importantly, as it will be discussed shortly, the block structure of the set $X$ allows for addressing Nash games as well as high-dimensional optimization problems. We note that the problem \\cref{prob:main} can represent a variety of the standard problems in optimization and VI regimes. For example, when $F(x) := 0_{n}$, the problem \\cref{prob:main} is equivalent to the canonical optimization problem $\\min_{x \\in X} f(x)$. Also, when $f(x):=0$, the problem \\cref{prob:main} is equivalent to solving $\\text{VI}(X,F)$. More detailed examples that can be reformulated as the problem \\cref{prob:main} are presented below.\n","sha256_utf8":"88e9a0f9f04970e2b1794cc7aca652b60287d3acd2e948334f22d11a81f0925a"}]},{"id":"S002","semantic_type":"notation","kind":"notation","title":"分块向量与凸分析记号","source_units":[{"start":28075,"end":30148,"start_line":300,"end_line":300,"raw_tex":"\\textbf{Notation and preliminary definitions:} Throughout, a vector $ x \\in \\mathbb{R}^n$ is assumed to be a column vector and $ x^T$ denotes the transpose of $x$. We use $x^{(i)}\\in \\mathbb{R}^{n_i}$ to denote the $i^{\\text{th}}$ block-coordinate of vector $x$ where $x =\\left(x^{(1)};\\ldots;x^{(d)}\\right)$ and $\\sum_{i=1}^dn_i=n$. The Euclidean norm of a vector $x$ is denoted by $\\|x\\|$, i.e.,  $\\| x\\|\\triangleq \\sqrt{ x^T  x}.$  For a mapping $F:\\mathbb{R}^n \\to \\mathbb{R}^n$, we denote the $i^{\\text{th}}$ block-coordinate of $F$ by $F_i:\\mathbb{R}^n \\to \\mathbb{R}^{n_i}$, i.e., $F(x) =\\left(F_1(x);\\ldots;F_d(x)\\right)$. A mapping $F:\\mathbb{R}^n \\to \\mathbb{R}^n$ is said to be monotone on a convex set $X \\subseteq \\mathbb{R}^n$ if for any $x,y \\in X$, we have $(F(x)-F(y))^T(x-y)\\geq 0$. The mapping $F$ is said to be $\\mu$--strongly monotone on a convex set $X \\subseteq \\mathbb{R}^n$ if $\\mu>0$ and for any $x,y \\in X$, we have $(F(x)-F(y))^T(x-y)\\geq \\mu\\|x-y\\|^2$. Also, $F$ is said to be Lipschitz with parameter $L>0$ on the set $X$ if for any $x,y \\in X$, we have $\\|F(x)-F(y)\\|\\leq L\\|x-y\\|$. A continuously differentiable function $f:\\mathbb{R}^n \\to \\mathbb{R}$ is called $\\mu$--strongly convex on a convex set $X$ if $f(x) \\geq f(y)+\\nabla f(y)^T(x-y)+\\frac{\\mu}{2}\\|x-y\\|^2$. Function $f$ is $\\mu$--strongly convex if and only if $\\nabla f$ is $\\mu$--strongly monotone on $X$. For a convex function $f$ with the domain $\\text{dom}(f)$, the subgradient of $f$ at $x \\in \\text{dom}(f)$ is denoted by $\\tilde \\nabla f(x)$ and it satisfies $f(x)+\\tilde \\nabla f(x)^T(y-x) \\leq f(y)$ for all $y \\in \\text{dom}(f)$. The subdifferential set of $f$ at $x$ is the set of all subgradients of $f$ at $x$ and is denoted by $\\partial f(x)$. The Euclidean projection of vector $s$ onto a convex set $ S$ is denoted by $ \\proj[S]{ s} $, where $\\proj[S]{ s}\\triangleq  \\argmin_{ y \\in  S}\\| s- y\\|$. %Given two vectors $x,y\\in \\mathbb{R}^n$, we use $0\\leq x \\perp y \\geq 0$ to compactly denote the complementarity constraints $x\\geq 0$, $y \\geq 0$, and $x^Ty=0$. \n","sha256_utf8":"b40c26a0663d5306046de9ab9419797b5efcbc293a49ba6db2cee6866ab053e7"},{"start":30273,"end":30632,"start_line":302,"end_line":302,"raw_tex":"We use $\\mathbf{I}_n$ to denote the identity matrix of size $n\\times n$. The probability of an event $Z$ is denoted by $\\text{Prob}(Z)$ and the expectation of a random variable $z$ is denoted by $\\EXP{z}$. We use $\\mathbb{R}^n_+$ and $\\mathbb{R}^n_{++}$ to denote $\\{x \\in \\mathbb{R}^n \\mid x \\geq 0\\}$ and  $\\{x \\in \\mathbb{R}^n \\mid x > 0\\}$, respectively.\n","sha256_utf8":"f87e5eb1f2eeaceb67275c0be17d8b055e01cf7bc6f43e0393f1d0c78589c2de"}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","title":"基本凸性与有界性假设","source_units":[{"start":30891,"end":31390,"start_line":306,"end_line":315,"raw_tex":"\\begin{assumption}\\label{assum:problem} Consider the problem \\cref{prob:main} under the following conditions:\n\n\\noindent (a) The set $X_i$ is nonempty, closed, and convex for all $i=1,\\dots, d$.\n\n\\noindent (b) The function $f$ is convex and has bounded subgradients over the set $X$.\n\n\\noindent  (c) The mapping $F:\\mathbb{R}^n \\to \\mathbb{R}^n$ is continuous, monotone, and bounded over the set $X$. \n\n\\noindent (d) The optimal solution set of problem \\cref{prob:main} in nonempty.\n\\end{assumption}","sha256_utf8":"c31e827daf3b4f9a9a084b577e12bf0729b0ea4ab58610068bb8e790e66413a5"}]},{"id":"S004","semantic_type":"notation","kind":"notation","title":"映射与次梯度界常数","source_units":[{"start":32584,"end":32896,"start_line":320,"end_line":320,"raw_tex":"Throughout, we let $C_F>0$ denote the bound on the Euclidean norm of the mapping $F$, i.e., $\\|F(x)\\| \\leq C_F$ for all $x \\in X$. Also, we let $C_f>0$ denote the bound on the norm of the subgradients of $f$, i.e., $\\|\\tilde \\nabla f (x)\\| \\leq C_f$ for all $\\tilde \\nabla f(x) \\in \\partial f(x)$ and $x \\in X$.\n","sha256_utf8":"ca645cd96475465537452679433601d82259fe2c606485f1a934e41f17c36389"}]},{"id":"S005","semantic_type":"algorithm","kind":"algorithm","title":"随机分块正则化平均算法","source_units":[{"start":32896,"end":34223,"start_line":321,"end_line":346,"raw_tex":"\\begin{algorithm} [h]\n\t\\caption{aRB-IRG}\n\t\\begin{algorithmic}[1]\n\t\t\\STATE\\textbf{Input:} A random initial point $x_0 \\in X$, $\\bar{x}_0 := x_0$, initial stepsize $\\gamma_{0} > 0$, initial regularization parameter $\\eta_{0} > 0$, a scalar $0\\leq r<1$, and $S_0 := \\gamma_0^r$.\n\t\t\\FOR{k = 0, 1, \\dots}\n\t\t\\STATE Generate a realization of random variable $i_k$ according to \\cref{assum:random sample}.\n\t\t\\STATE Evaluate $F_{i_k}(x_k)$ and $\\tilde \\nabla_{i_k} f(x_k)$ where $\\tilde \\nabla f(x_k) \\in \\partial f(x_k)$.\n\n\t\t\\STATE Update $x_k$ as follows:\n%\t\t\t\t${x_{k+1}^{(i)}} :=$\n\t\t\\begin{align}\\label{equ:update_rule_aRBIRG} \n\t\t{x_{k+1}^{(i)}} :=\n\t\t\\begin{cases}\n\t\t\\proj[X_{i_k}]{ x_{k}^{(i_k)}-\\gamma_{k}\\left(F_{i_k}\\left( x_{k}\\right) + \\eta_{k} \\tilde \\nabla_{{i_k}} f\\left( x_{k}\\right)\\right)}& \\text{ if } i = i_k, \\\\\n\t\tx_k^{(i)} & \\text{ if } i \\neq i_k. \n\t\t\\end{cases}\n\t\t\\end{align}\n\t\t\\STATE Obtain $\\gamma_{k+1}$ and $\\eta_{k+1}$ (cf. \\cref{thm:a-IRG_rate_results} and \\cref{thm:convergence_RB-IRG} for the update rules).\n\n\t\t\\STATE Update the averaged iterate $\\bar{x}_{k}$ as follows:\n\t\t\t\\begin{align}\\label{eqn:ave_step_of_alg}\n\t\t\tS_{k+1} &:= S_k + \\gamma_{k+1}^r,\\quad\t\\bar{x}_{k+1}:= \\frac{S_k\\bar{x}_{k}+\\gamma_{k+1}^rx_{k+1}}{S_{k+1}}.\n\t\t\t\\end{align}\n\t\t\\ENDFOR\n\t\\end{algorithmic} \\label{alg:aRB-IRG}\n\\end{algorithm}","sha256_utf8":"01269de1055981190928ccafdf697bb061595f6e645510dc31d5c929fa03a137"}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","title":"独立同分布分块抽样","source_units":[{"start":34378,"end":34758,"start_line":348,"end_line":350,"raw_tex":"\\begin{assumption}[Block-coordinate selection rule] \\label{assum:random sample}\nAt each iteration $k\\geq 0$, the random variable $i_k$ is generated from an independent and identically distributed discrete probability distribution such that $\\mathrm{Prob}\\left(i_k=i\\right) = \\prob{i}$ where $ \\prob{i}>0$ for $i \\in \\{1,\\ldots,d\\}$ and $\\sum_{i=1}^{d}\\prob{i}=1$.\n\\end{assumption}","sha256_utf8":"4386d688bbb5422e043abb6c90cf467deabfb06b9f40629ac594579f5e6e35ed"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"概率加权距离","source_units":[{"start":36922,"end":37499,"start_line":359,"end_line":363,"raw_tex":"\\begin{definition}[Distance function]\\label{def:err_D}\n\tFor any $x,y \\in \\mathbb{R}^n$, function $\\mathcal{D}(x,y)$ is defined as $\n\t\\mathcal{D}(x,y)\\triangleq \\sum_{i=1}^{d} \\prob{i}^{-1} \\left\\Vert\\left. x^{(i)}-y^{(i)} \\right\\Vert\\right.^2$, where $\\prob{i}$ is given by \\cref{assum:random sample}.\n\t% \t, where $\\prob{i}$ is the uniform probability of selection of any block  $i$. From \\cref{assum:random sample}, we have $\\prob{i} = \\frac{1}{d}.$ So, we write  $ \\mathcal{D}(x,y) =  d \\sum_{i=1}^{d}  \\left\\Vert\\left. x^{(i)}-y^{(i)} \\right\\Vert\\right.^2$.\n\\end{definition}","sha256_utf8":"9a65a9504cad447a57ca915f5cc4fcb35105952e11d4c3c6e64247b81d6805f9"}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","title":"距离与欧氏范数比较","source_units":[{"start":37500,"end":37912,"start_line":364,"end_line":366,"raw_tex":"\\begin{remark} \\label{rem:err_D}\nUnder \\cref{assum:random sample}, we can relate the distance function $\\mathcal{D}$ with the $\\ell_2$--norm as follows: $\\prob{{min}}\t{\\mathcal{D}(x,y)} \\leq \\|x-y\\|^2 \\leq\\prob{{max}} {\\mathcal{D}(x,y)}$ for all $x,y \\in \\mathbb{R}^n$, where  $\\prob{{min}}\\triangleq \\min_{1\\leq i \\leq d}\\{\\prob{i}\\}$ and $\\prob{{max}}\\triangleq \\max_{1\\leq i \\leq d}\\{\\prob{i}\\}$.\n\\end{remark}","sha256_utf8":"4d723b04cecfa80f1af38dd2fbe097068edacefeac6e93c365100322fdb893c5"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"对偶间隙函数","source_units":[{"start":38548,"end":38923,"start_line":368,"end_line":370,"raw_tex":"\\begin{definition}[The dual gap function \\cite{MarcotteZhu1998}]\\label{def:gap}\nLet a nonempty closed set $X\\subseteq \\mathbb{R}^n$ and a mapping $F:X\\rightarrow\\mathbb{R}^n$ be given. Then, for any $x \\in X$, the dual gap function $\\mathrm{GAP}:X\\rightarrow \\mathbb{R}\\cup \\{+\\infty\\} $ is defined as $ \\mathrm{GAP}(x) \\triangleq \\sup_{y\\in X} F(y)^T(x-y)$.\n\\end{definition}","sha256_utf8":"55d8752b02626e85d6559f2a09a39917b61cd51aefa39bfc05de40295790860a"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"正则化映射","source_units":[{"start":39322,"end":39687,"start_line":374,"end_line":376,"raw_tex":"\\begin{definition}[Regularized mapping]\\label{def:reg_map}\nGiven a vector $x \\in X$, a subgradient $\\tilde \\nabla f(x) \\in \\partial f(x)$, and an integer $k\\geq 0$, the regularized mapping $G_k: X \\to \\mathbb{R}$ is defined as $G_k(x) \\triangleq F(x)+\\eta_k\\tilde \\nabla f(x)$. The $i^\\text{th}$ {block-coordinate} of $G_k$ is denoted by $G_{k,i}$.\n\\end{definition}","sha256_utf8":"a41df99d86d08da828c0f549785ce32f67ba602770106f160c7a13da5972d96c"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"算法历史信息","source_units":[{"start":39688,"end":39939,"start_line":377,"end_line":379,"raw_tex":"\\begin{definition}[History of the method]\\label{def:Fk}\nThroughout, we let the history of the algorithm to be denoted by $\\mathcal{F}_k \\triangleq \\{x_0,i_0,i_1,\\ldots,i_{k-1}\\}$ for $k\\geq 1$, with $\\mathcal{F}_0 \\triangleq \\{x_0\\}$.\n\\end{definition}","sha256_utf8":"928d35e874f6181a29fd66572a03cce573567bc999465d6fcd6aba1f55bd8511"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"随机分块误差","source_units":[{"start":41602,"end":42174,"start_line":396,"end_line":401,"raw_tex":"\\begin{definition}[Randomized block error terms]\\label{def:deltas}\nLet ${\\mathbf{U}}_i \\in \\mathbb{R}^{n\\times n_i}$ for $i=1,\\ldots,d$ be the collection of matrices such that $\\mathbf{I}_n = \\left[{\\mathbf{U}}_1,\\ldots,{\\mathbf{U}}_d\\right] \\in \\mathbb{R}^{n\\times n}$. Consider the following definitions for $k\\geq 0$: \n\\begin{align}\\label{eqn:deltas}\n\\Delta_k\\triangleq F(x_k) - \\prob{{i_k}}^{-1}{\\mathbf{U}}_{i_k}F_{i_k}(x_k), \\qquad \\delta_k\\triangleq \\tilde \\nabla f(x_k) - \\prob{{i_k}}^{-1}{\\mathbf{U}}_{i_k} \\tilde \\nabla_{i_k}f (x_k).\n\\end{align}\n\\end{definition}","sha256_utf8":"d1b9335df6d49ae9d929fe3c0db90771edce03e8e0bb3dfa9349b1062d4c24eb"}]},{"id":"S013","semantic_type":"theorem","kind":"theorem","title":"随机分块法的双误差速率","source_units":[{"start":64132,"end":65384,"start_line":643,"end_line":653,"raw_tex":"\\begin{theorem}[Convergence rate statements for \\cref{alg:aRB-IRG}]\\label{thm:a-IRG_rate_results}\nConsider \\cref{alg:aRB-IRG}. Let \\cref{assum:problem} and \\cref{assum:random sample} hold and assume {that} the set $X$ is bounded such that $\\|x\\|\\leq M$ for all $x \\in X$ and some $M>0$. Suppose for {all} $k\\geq 0$, $\\gamma_k:=\\frac{\\gamma_0}{\\sqrt{k+1}}$ and $\\eta_k:=\\frac{\\eta_0}{(k+1)^b}$, where $\\gamma_0>0$, $\\eta_0>0$, and $0<b<0.5$. Then, for any $0\\leq r <1$, the following results hold:\\\\\n\\noindent (i) Let $x^*$ be an optimal solution to the problem \\cref{prob:main}. Then, for {all} $N\\geq 2^{\\frac{2}{1-r}}-1$:\n\\begin{align}\\label{eqn:f_rate} \n&\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{2-r}{\\prob{{min}}\\eta_0}\\left(\\frac{4M^2}{\\gamma_0}+\\frac{\\gamma_0\\left(C_F^2+\\eta_0^2C_f^2\\right)}{0.5-0.5r+b}\\right)\\frac{1}{(N+1)^{0.5-b}}.\n\\end{align}\n\\noindent (ii) Consider the dual gap function in \\cref{def:gap}. Then, for {all} $N\\geq 2^{\\frac{2}{1-r}}-1$:\n\\begin{align}\\label{eqn:gap_rate} \n\\EXP{\\mathrm{GAP}\\left(\\bar{x}_N\\right)} \\leq \\frac{2-r}{\\prob{{min}}}\\left(\\frac{4M^2}{\\gamma_0}+\\frac{\\gamma_0\\left(C_F^2+\\eta_0^2C_f^2\\right)}{0.5-0.5r}+\\frac{2\\prob{{min}C_fM}\\eta_0}{1-0.5r-b}\\right)\\frac{1}{(N+1)^b}.\n\\end{align}\n\\end{theorem}","sha256_utf8":"6cca0c5095d06b9bced93c6c3b464f96a3be4e79d93f7c108c3905aa44c67b46"}]},{"id":"S014","semantic_type":"assumption","kind":"assumption","title":"有界域与幂次参数规则","source_units":[{"start":64230,"end":64756,"start_line":644,"end_line":645,"raw_tex":"Consider \\cref{alg:aRB-IRG}. Let \\cref{assum:problem} and \\cref{assum:random sample} hold and assume {that} the set $X$ is bounded such that $\\|x\\|\\leq M$ for all $x \\in X$ and some $M>0$. Suppose for {all} $k\\geq 0$, $\\gamma_k:=\\frac{\\gamma_0}{\\sqrt{k+1}}$ and $\\eta_k:=\\frac{\\eta_0}{(k+1)^b}$, where $\\gamma_0>0$, $\\eta_0>0$, and $0<b<0.5$. Then, for any $0\\leq r <1$, the following results hold:\\\\\n\\noindent (i) Let $x^*$ be an optimal solution to the problem \\cref{prob:main}. Then, for {all} $N\\geq 2^{\\frac{2}{1-r}}-1$:\n","sha256_utf8":"b77ae464ab98bfbe2969b5b01c7a254d96b0df00545f8cc7cc1c263b4ab868ba"}]},{"id":"S015","semantic_type":"proof","kind":"proof","title":"双误差速率完整证明","source_units":[{"start":65385,"end":69288,"start_line":654,"end_line":682,"raw_tex":"\\begin{proof} Let us define the following terms: \n\\begin{align*}%\\label{eqn:three_terms_abr2}\n& \\Lambda_{N,1} \\triangleq \\prob{{min}}\\sum_{k=0}^{N}\\gamma_k^r, \\quad  \\Lambda_{N,2}\\triangleq  \\frac{4M^2\\gamma_{{N}}^{r-1}}{\\eta_{N}} ,\\quad \\Lambda_{N,3} \\triangleq \\left(C_F^2+\\eta_0^2C_f^2\\right)\\sum_{k=0}^{N}\\eta_k^{-1}\\gamma_k^{r+1},\\nonumber\\\\ \n& \\Lambda_{N,4} \\triangleq 4M^2\\gamma_{{N}}^{r-1}, \\quad  \\Lambda_{N,5} \\triangleq \\left(C_F^2+\\eta_0^2C_f^2\\right)\\sum_{k=0}^{N}\\gamma_k^{r+1}, \\quad \\Lambda_{N,6} \\triangleq 2\\prob{{min}}C_fM \\sum_{k=0}^{N}\\eta_k\\gamma_k^r.\n\\end{align*}\nNote that from \\cref{eqn:f_bound} and \\cref{eqn:gap_bound}, we have:\n\\begin{align}\\label{eqn:bounds_in_Lambdas}\n\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{\\Lambda_{N,2}+\\Lambda_{N,3}}{\\Lambda_{N,1}}, \\quad \\EXP{\\text{GAP}\\left(\\bar{x}_N\\right)} \\leq \\frac{\\Lambda_{N,4}+\\Lambda_{N,5}+\\Lambda_{N,6}}{\\Lambda_{N,1}}.\n\\end{align}\nNext, we apply \\cref{Lemma 2.13} to estimate the terms $\\Lambda_{N,i}$. Substituting $\\gamma_k$ and $\\eta_k$ by their update rules, we obtain:\n\\begin{align*}%\\label{eqn:Lambda1}\n\\Lambda_{N,1} &=\\prob{{min}}\\sum_{k=0}^{N}\\frac{\\gamma_0^r}{(k+1)^{0.5r}}\\geq \\frac{\\prob{{min}}\\gamma_0^r(N+1)^{1-0.5r}}{2(1-0.5r)},\\\\  \n\\Lambda_{N,2} &= \\frac{4M^2(N+1)^{0.5(1-r)+b}}{\\eta_0\\gamma_0^{1-r}},\\quad \\Lambda_{N,4} = \\frac{4M^2(N+1)^{0.5(1-r)}}{\\gamma_0^{1-r}}, \\\\\n\\Lambda_{N,3} &= \\sum_{k=0}^{N}\\frac{\\left(C_F^2+\\eta_0^2C_f^2\\right)\\gamma_0^{1+r}}{\\eta_0(k+1)^{0.5(1+r)-b}} \\leq\\frac{\\gamma_0^{1+r}\\left(C_F^2+\\eta_0^2C_f^2\\right)(N+1)^{1-0.5(1+r)+b}}{\\eta_0(1-0.5(1+r)+b)},\\\\\n\\Lambda_{N,5} &= \\left(C_F^2+\\eta_0^2C_f^2\\right)\\sum_{k=0}^{N}\\frac{\\gamma_0^{r+1}}{(k+1)^{0.5(1+r)}}\\leq \\frac{\\left(C_F^2+\\eta_0^2C_f^2\\right)\\gamma_0^{r+1}(N+1)^{1-0.5(1+r)}}{1-0.5(1+r)},\\\\\n\\Lambda_{N,6} &=2\\prob{{min}}C_fM \\eta_0\\gamma_0^r\\sum_{k=0}^{N}\\frac{1}{(k+1)^{0.5r+b}}\\leq  \\frac{2\\prob{{min}}C_fM \\eta_0\\gamma_0^r (N+1)^{1-0.5r-b}}{1-0.5r-b}.\n\\end{align*}\nFor these inequalities to hold, we need to ensure that {the} conditions of \\cref{Lemma 2.13} are met. Accordingly, we must have $0\\leq 0.5r <1$, $0\\leq 0.5(1+r)-b <1$, $0\\leq 0.5r+b<1$, and $0 \\leq 0.5(1+r)<1$. These relations hold because $0\\leq r<1$ and $0<b<0.5$. Another set of conditions when applying \\cref{Lemma 2.13} includes $N\\geq \\max\\left\\{2^{1/(1-0.5r)},2^{1/(1-0.5(1+r)+b)},2^{1/(1-0.5r-b)},2^{1/(1-0.5(1+r))}\\right\\}-1$. {This relation is indeed} satisfied as a consequence of $N\\geq 2^{\\frac{2}{1-r}}-1$, {$0<b<0.5$}, and $0\\leq r<1$. We conclude that all the necessary conditions for applying \\cref{Lemma 2.13} and obtaining the aforementioned bounds for the terms $\\Lambda_{N,i}$ are satisfied. To show that the inequalities \\cref{eqn:f_rate} and \\cref{eqn:gap_rate} hold, it suffices to substitute the preceding {bounds on} the terms $\\Lambda_{N,i}$ into the two inequalities given by \\cref{eqn:bounds_in_Lambdas}. The details are as follows:\n\\begin{align*}\n&\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{\\Lambda_{N,2}+\\Lambda_{N,3}}{\\Lambda_{N,1}}=\\frac{2-r}{\\prob{{min}}\\gamma_0^r(N+1)^{1-0.5r}}\\left(\\frac{4M^2(N+1)^{0.5-0.5r+b}}{\\eta_0\\gamma_0^{1-r}}\\right.\\\\\n&\\left.+\\left(\\frac{\\gamma_0^{1+r}}{\\eta_0}\\right)\\frac{\\left(C_F^2+\\eta_0^2C_f^2\\right)(N+1)^{0.5-0.5r+b}}{0.5-0.5r+b}\\right). \n\\end{align*}\nThe inequality \\cref{eqn:f_rate} is obtained by rearranging the terms in the preceding relation.\n\\begin{align*}\n&\\EXP{\\text{GAP}\\left(\\bar{x}_N\\right)} \\leq \\frac{\\Lambda_{N,4}+\\Lambda_{N,5}+\\Lambda_{N,6}}{\\Lambda_{N,1}} \\leq\\frac{2-r}{\\prob{{min}}\\gamma_0^r(N+1)^{1-0.5r}}\\left(\\frac{4M^2(N+1)^{0.5-0.5r}}{\\gamma_0^{1-r}} \\right.\\\\\n&\\left.+\\frac{\\left(C_F^2+\\eta_0^2C_f^2\\right)\\gamma_0^{r+1}(N+1)^{0.5-0.5r}}{0.5-0.5r}+ \\frac{2\\prob{{min}}C_fM \\eta_0\\gamma_0^r (N+1)^{1-0.5r-b}}{1-0.5r-b}\\right).\n\\end{align*}\nThen, \\cref{eqn:gap_rate} can be obtained by rearranging the terms in the preceding inequality.\n\\end{proof}","sha256_utf8":"cdea17fe10aca883d97f9824ee3dbadc0e7dcaf1eebf470249fff6a69a752d6c"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"定义六个速率估计量","source_units":[{"start":65385,"end":65972,"start_line":654,"end_line":658,"raw_tex":"\\begin{proof} Let us define the following terms: \n\\begin{align*}%\\label{eqn:three_terms_abr2}\n& \\Lambda_{N,1} \\triangleq \\prob{{min}}\\sum_{k=0}^{N}\\gamma_k^r, \\quad  \\Lambda_{N,2}\\triangleq  \\frac{4M^2\\gamma_{{N}}^{r-1}}{\\eta_{N}} ,\\quad \\Lambda_{N,3} \\triangleq \\left(C_F^2+\\eta_0^2C_f^2\\right)\\sum_{k=0}^{N}\\eta_k^{-1}\\gamma_k^{r+1},\\nonumber\\\\ \n& \\Lambda_{N,4} \\triangleq 4M^2\\gamma_{{N}}^{r-1}, \\quad  \\Lambda_{N,5} \\triangleq \\left(C_F^2+\\eta_0^2C_f^2\\right)\\sum_{k=0}^{N}\\gamma_k^{r+1}, \\quad \\Lambda_{N,6} \\triangleq 2\\prob{{min}}C_fM \\sum_{k=0}^{N}\\eta_k\\gamma_k^r.\n\\end{align*}\n","sha256_utf8":"5abe24b08fa8940db113a4d714b89a1e95060f877def7eb0d97dff0be0126ad4"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"将误差界写成估计量之比","source_units":[{"start":65972,"end":66303,"start_line":659,"end_line":662,"raw_tex":"Note that from \\cref{eqn:f_bound} and \\cref{eqn:gap_bound}, we have:\n\\begin{align}\\label{eqn:bounds_in_Lambdas}\n\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{\\Lambda_{N,2}+\\Lambda_{N,3}}{\\Lambda_{N,1}}, \\quad \\EXP{\\text{GAP}\\left(\\bar{x}_N\\right)} \\leq \\frac{\\Lambda_{N,4}+\\Lambda_{N,5}+\\Lambda_{N,6}}{\\Lambda_{N,1}}.\n\\end{align}\n","sha256_utf8":"3457bc528f7eb62962dfbbebaab9f700e33b51c2466d4271a79c0c3d83e30d48"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"估计六个幂次求和项","source_units":[{"start":66303,"end":67343,"start_line":663,"end_line":670,"raw_tex":"Next, we apply \\cref{Lemma 2.13} to estimate the terms $\\Lambda_{N,i}$. Substituting $\\gamma_k$ and $\\eta_k$ by their update rules, we obtain:\n\\begin{align*}%\\label{eqn:Lambda1}\n\\Lambda_{N,1} &=\\prob{{min}}\\sum_{k=0}^{N}\\frac{\\gamma_0^r}{(k+1)^{0.5r}}\\geq \\frac{\\prob{{min}}\\gamma_0^r(N+1)^{1-0.5r}}{2(1-0.5r)},\\\\  \n\\Lambda_{N,2} &= \\frac{4M^2(N+1)^{0.5(1-r)+b}}{\\eta_0\\gamma_0^{1-r}},\\quad \\Lambda_{N,4} = \\frac{4M^2(N+1)^{0.5(1-r)}}{\\gamma_0^{1-r}}, \\\\\n\\Lambda_{N,3} &= \\sum_{k=0}^{N}\\frac{\\left(C_F^2+\\eta_0^2C_f^2\\right)\\gamma_0^{1+r}}{\\eta_0(k+1)^{0.5(1+r)-b}} \\leq\\frac{\\gamma_0^{1+r}\\left(C_F^2+\\eta_0^2C_f^2\\right)(N+1)^{1-0.5(1+r)+b}}{\\eta_0(1-0.5(1+r)+b)},\\\\\n\\Lambda_{N,5} &= \\left(C_F^2+\\eta_0^2C_f^2\\right)\\sum_{k=0}^{N}\\frac{\\gamma_0^{r+1}}{(k+1)^{0.5(1+r)}}\\leq \\frac{\\left(C_F^2+\\eta_0^2C_f^2\\right)\\gamma_0^{r+1}(N+1)^{1-0.5(1+r)}}{1-0.5(1+r)},\\\\\n\\Lambda_{N,6} &=2\\prob{{min}}C_fM \\eta_0\\gamma_0^r\\sum_{k=0}^{N}\\frac{1}{(k+1)^{0.5r+b}}\\leq  \\frac{2\\prob{{min}}C_fM \\eta_0\\gamma_0^r (N+1)^{1-0.5r-b}}{1-0.5r-b}.\n\\end{align*}\n","sha256_utf8":"70a8f7a2c4943a63875634c74b0f250a44ab9dbb96edc8ba5d25a8eaf2455652"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"核验级数估计适用条件","source_units":[{"start":67343,"end":68305,"start_line":671,"end_line":671,"raw_tex":"For these inequalities to hold, we need to ensure that {the} conditions of \\cref{Lemma 2.13} are met. Accordingly, we must have $0\\leq 0.5r <1$, $0\\leq 0.5(1+r)-b <1$, $0\\leq 0.5r+b<1$, and $0 \\leq 0.5(1+r)<1$. These relations hold because $0\\leq r<1$ and $0<b<0.5$. Another set of conditions when applying \\cref{Lemma 2.13} includes $N\\geq \\max\\left\\{2^{1/(1-0.5r)},2^{1/(1-0.5(1+r)+b)},2^{1/(1-0.5r-b)},2^{1/(1-0.5(1+r))}\\right\\}-1$. {This relation is indeed} satisfied as a consequence of $N\\geq 2^{\\frac{2}{1-r}}-1$, {$0<b<0.5$}, and $0\\leq r<1$. We conclude that all the necessary conditions for applying \\cref{Lemma 2.13} and obtaining the aforementioned bounds for the terms $\\Lambda_{N,i}$ are satisfied. To show that the inequalities \\cref{eqn:f_rate} and \\cref{eqn:gap_rate} hold, it suffices to substitute the preceding {bounds on} the terms $\\Lambda_{N,i}$ into the two inequalities given by \\cref{eqn:bounds_in_Lambdas}. The details are as follows:\n","sha256_utf8":"354e293383fae7968ec786af4d4e726a4ee156cd7518021e6e26cf6d016c2929"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"代入并整理次优性速率","source_units":[{"start":68305,"end":68766,"start_line":672,"end_line":676,"raw_tex":"\\begin{align*}\n&\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{\\Lambda_{N,2}+\\Lambda_{N,3}}{\\Lambda_{N,1}}=\\frac{2-r}{\\prob{{min}}\\gamma_0^r(N+1)^{1-0.5r}}\\left(\\frac{4M^2(N+1)^{0.5-0.5r+b}}{\\eta_0\\gamma_0^{1-r}}\\right.\\\\\n&\\left.+\\left(\\frac{\\gamma_0^{1+r}}{\\eta_0}\\right)\\frac{\\left(C_F^2+\\eta_0^2C_f^2\\right)(N+1)^{0.5-0.5r+b}}{0.5-0.5r+b}\\right). \n\\end{align*}\nThe inequality \\cref{eqn:f_rate} is obtained by rearranging the terms in the preceding relation.\n","sha256_utf8":"5476918d5cff2580bdbcf9488f22f1e680682a2ffb5cf59be8eed69050946106"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"代入并整理间隙速率","source_units":[{"start":68766,"end":69288,"start_line":677,"end_line":682,"raw_tex":"\\begin{align*}\n&\\EXP{\\text{GAP}\\left(\\bar{x}_N\\right)} \\leq \\frac{\\Lambda_{N,4}+\\Lambda_{N,5}+\\Lambda_{N,6}}{\\Lambda_{N,1}} \\leq\\frac{2-r}{\\prob{{min}}\\gamma_0^r(N+1)^{1-0.5r}}\\left(\\frac{4M^2(N+1)^{0.5-0.5r}}{\\gamma_0^{1-r}} \\right.\\\\\n&\\left.+\\frac{\\left(C_F^2+\\eta_0^2C_f^2\\right)\\gamma_0^{r+1}(N+1)^{0.5-0.5r}}{0.5-0.5r}+ \\frac{2\\prob{{min}}C_fM \\eta_0\\gamma_0^r (N+1)^{1-0.5r-b}}{1-0.5r-b}\\right).\n\\end{align*}\nThen, \\cref{eqn:gap_rate} can be obtained by rearranging the terms in the preceding inequality.\n\\end{proof}","sha256_utf8":"025b45c511bffb5af5595e302517473458b13a0bf32ca5c8bbabbac5c1588f3b"}]},{"id":"S022","semantic_type":"proposition","kind":"proposition","title":"平均迭代的双误差上界","source_units":[{"start":51360,"end":52531,"start_line":518,"end_line":528,"raw_tex":"\\begin{proposition}[Error bounds for \\cref{alg:aRB-IRG}]\\label{prop:dual_gap_bound}\nLet the sequence $\\{\\bar x_k\\}$ be generated by \\cref{alg:aRB-IRG}{, where $0\\leq r<1$}. Suppose $\\{\\gamma_k\\}$ and $\\{\\eta_k\\}$ are strictly positive and nonincreasing sequences. Let \\cref{assum:problem} and \\cref{assum:random sample} hold {and} assume {that} the set $X$ is bounded, i.e., {$\\|x\\|\\leq M$} for all $x \\in X$ and some $M>0$. \\\\\n\\noindent (a) Let $x^*$ be an optimal solution to problem \\cref{prob:main}. Then, for {all} $N\\geq 1$:\n\\begin{align}\\label{eqn:f_bound} \n&\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{\\frac{4M^2\\gamma_{{N}}^{r-1}}{\\eta_{N}} +\\sum_{k=0}^{N}\\eta_k^{-1}\\gamma_k^{r+1}\\left(C_F^2+\\eta_k^2C_f^2\\right)}{\\prob{{min}}\\sum_{k=0}^{N}\\gamma_k^r}.\n\\end{align}\n\\noindent (b) Consider the dual gap function in \\cref{def:gap}. Then, for {all} $N\\geq 1$:\n\\begin{align}\\label{eqn:gap_bound} \n\\EXP{\\mathrm{GAP}\\left(\\bar{x}_N\\right)} \\leq\\frac{ {4 M^2 \\gamma_{{N}}^{r-1} }  +\\sum_{k=0}^{N}\\gamma_{{k}}^r\\left( 2\\prob{{min}}\\eta_kC_fM+\\gamma_k C_F^2 +  \\gamma_{k}\\eta_k^2C_f ^2 \\right)}{\\prob{{min}}\\sum_{k=0}^{N} \\gamma_{{k}}^r}.\n\\end{align}\n\\end{proposition}","sha256_utf8":"547ef868d469f1e4adad12508140b2d7fe9c230b4234254007680f7475cc3c7d"}]},{"id":"S023","semantic_type":"assumption","kind":"assumption","title":"误差界的参数与有界域假设","source_units":[{"start":51444,"end":51891,"start_line":519,"end_line":520,"raw_tex":"Let the sequence $\\{\\bar x_k\\}$ be generated by \\cref{alg:aRB-IRG}{, where $0\\leq r<1$}. Suppose $\\{\\gamma_k\\}$ and $\\{\\eta_k\\}$ are strictly positive and nonincreasing sequences. Let \\cref{assum:problem} and \\cref{assum:random sample} hold {and} assume {that} the set $X$ is bounded, i.e., {$\\|x\\|\\leq M$} for all $x \\in X$ and some $M>0$. \\\\\n\\noindent (a) Let $x^*$ be an optimal solution to problem \\cref{prob:main}. Then, for {all} $N\\geq 1$:\n","sha256_utf8":"0ee986dcfb77dc67cb79ea3845e6fbb284183d42bba90b0c7ee30f98c7ebedfd"}]},{"id":"S024","semantic_type":"proof","kind":"proof","title":"双误差上界完整证明","source_units":[{"start":52533,"end":64049,"start_line":530,"end_line":641,"raw_tex":"\\begin{proof}\nWe define the following terms for all $k\\geq 0$, that appear in \\cref{eqn:prebound_gap}:\n\\begin{align}\\label{eqn:three_terms_abr}\n& {\\Theta_{k,1} \\triangleq \\gamma_k^r(x_k-u_k)^T\\left(\\Delta_k+\\eta_k\\delta_k\\right)}, \\quad \\Theta_{k,2} \\triangleq \\gamma_k^{r+1}\\left(\\|\\Delta_k\\|^2+\\eta_k^2\\|\\delta_k\\|^2\\right),\\nonumber \\\\ \n&\\Theta_{k,3} \\triangleq 0.5\\prob{{i_k}}^{-1}\\gamma_k^{1+r}\\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\\end{align}\nNext, we estimate the expected values of these terms. {Consider the notation of} $\\mathcal{F}_k$ given by \\cref{def:Fk}. Note that $x_k$ is $\\mathcal{F}_k$--measurable. Also, from the definition of $u_k$ in \\cref{lem:prebound_gap}, $u_k$ is $\\mathcal{F}_k$--measurable. Note, however, that $\\Theta_{k,j}$ is $\\mathcal{F}_{k+1}$--measurable for all $j \\in \\{1,2,3\\}$. Taking these into account and using the total probability law, for any $k\\geq 0$ and $j \\in \\{1,2,3\\}$, we have $ \\EXP{\\Theta_{k,j}} = \\mathsf{E}_{\\mathcal{F}_k}\\left[\\mathsf{E}_{i_k}\\left[\\Theta_{k,j}\\mid \\mathcal{F}_k\\right]\\right]$. From this relation and \\cref{Lemma 2.12}, we have for any $k\\geq 0$:\n\\begin{align}\\label{eqn:estimate_deltas_1_2}\n&\\EXP{\\Theta_{k,1}} = 0, \\qquad \\EXP{\\Theta_{k,2}} = \\left(\\prob{{min}}^{-1}-1\\right)\\gamma_k^{r+1}\\left(C_F^2+\\eta_k^2C_f^2\\right).\n\\end{align}\nAlso, using \\cref{def:reg_map} and the triangle inequality, we can write:\n\\begin{align*}\n&{\\mathsf{E}_{i_k}\\left[\\Theta_{k,3}\\mid \\mathcal{F}_k\\right]} = \\sum_{i=1}^d\\prob{i}\\left(0.5\\prob{{i}}^{-1}\\gamma_k^{1+r}\\left\\Vert\\left.G_{k,i}\\left(x_k\\right)\\right\\Vert\\right.^2\\right)\\nonumber\\\\\n&\\leq \\gamma_k^{1+r}\\sum_{i=1}^d\\left(\\|F_i(x_k)\\|^2+\\eta_k^2\\|\\tilde \\nabla_i f(x_k)\\|^2\\right)  = \\gamma_k^{1+r}\\|F(x_k)\\|^2+\\eta_k^2\\|\\tilde \\nabla f(x_k)\\|^2.\n\\end{align*}\n{From the preceding inequality, we obtain:\n\\begin{align}\\label{eqn:estimate_deltas_3}\n\\EXP{\\Theta_{k,3}} & \\leq  \\gamma_k^{1+r}\\left(C_F^2+\\eta_k^2C_f^2\\right).\n\\end{align}\n}We are now ready to show the inequalities \\cref{eqn:f_bound} and \\cref{eqn:gap_bound} as follows:\\\\\n\\noindent (a) Consider \\cref{eqn:prebound_gap}. From the definition of subgradients of the convex function $f$, we have that $f(x_k)-f(y) \\leq \\tilde \\nabla f(x_k)^T(x_k-y)$. Thus, {from \\cref{eqn:three_terms_abr}} we obtain for any $y \\in X$:\n\\begin{align*}\n&\\gamma_k^rF(y)^T(x_k-y) +\\gamma_k^r\\eta_k\\left(f(x_k)-f(y)\\right)\\leq \\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k}, y\\right) +\\|u_k-y\\|^2\\right) \\nonumber\\\\\n&-\\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k+1}, y\\right) +\\|u_{k+1}-y\\|^2\\right)+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}.\n\\end{align*}\nLet us substitute $y:=x^*$, where $x^*$ denotes an optimal solution to \\cref{prob:main}. Note that $x^*$ must be a feasible solution to \\cref{prob:main}, i.e., $F(x^*)^T(x_k-x^*) \\geq 0$. Thus, we obtain:\n\\begin{align}\\label{eqn:gap_prop_ineq1_a}\n\t\t\t&\\gamma_k^r\\eta_k\\left(f(x_k)-f(x^*)\\right) \\leq  \\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k}, x^*\\right) +\\|u_k-x^*\\|^2\\right) \\nonumber\\\\\n&-\\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k+1}, x^*\\right) +\\|u_{k+1}-x^*\\|^2\\right)+\\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}.\n\\end{align}\nDividing both sides by $\\eta_k$ and adding and subtracting $\\frac{\\gamma_{{k-1}}^{r-1}}{2\\eta_{k-1}} \\left( \\mathcal{D}\\left(x_k,x^*\\right) + \\left\\Vert u_k-x^* \\right\\Vert^2 \\right)$ in the right-hand side of \\cref{eqn:gap_prop_ineq1_a}, we obtain for $k\\geq 1$:\n\\begin{align}\\label{eqn:gap_prop_ineq2_a}\n&\\gamma_k^r\\left(f(x_k)-f(x^*)\\right) \\leq  \\frac{\\gamma_{{k-1}}^{r-1}}{2\\eta_{k-1}} \\left(  \\mathcal{D}\\left(x_k,x^*\\right)  + \\left\\Vert u_k-x^* \\right\\Vert^2 \\right) \\nonumber \\\\\n&- \\frac{\\gamma_{{k}}^{r-1}}{2\\eta_k} \\left(  \\mathcal{D}\\left(x_{k+1},x^*\\right)  + \\left\\Vert u_{k+1}-x^* \\right\\Vert^2 \\right)\\nonumber \\\\ \n&+ \\frac{1}{2} \\left(\\frac{\\gamma_{{k}}^{r-1}}{\\eta_k} - \\frac{\\gamma_{{k-1}}^{r-1}}{\\eta_{k-1}}\\right)\\left( \\mathcal{D}\\left(x_k,x^*\\right) + \\left\\Vert u_k-x^* \\right\\Vert^2 \\right)+\\eta_k^{-1}\\left( \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right).\n\\end{align}\nSince $r-1 < 0$ and that $\\{\\gamma_k\\}$ and $\\{\\eta_k\\}$ are nonincreasing, we have $\\frac{\\gamma_{{k}}^{r-1}}{\\eta_k} - \\frac{\\gamma_{{k-1}}^{r-1}}{\\eta_{k-1}} \\geq 0$. Also, from the boundedness of the set $X$, since $x_k$, $x^*$, and {$u_k$} belong to $ X$, using \\cref{rem:err_D} and the triangle inequality, we have:\n\\begin{align}\\label{eqn:gap_prop_ineq3_a}\n\\mathcal{D}\\left(x_k,x^*\\right)  + \\left\\Vert u_k-x^* \\right\\Vert^2 \n\\leq \\prob{{min}}^{-1}\\left\\Vert x_k-x^* \\right\\Vert^2 + \\left\\Vert u_k-x^* \\right\\Vert^2\n\\leq 4M^2\\left(\\prob{{min}}^{-1}+1\\right)\\leq \\frac{8M^2}{\\prob{{min}}}.\n\\end{align}\nSumming over \\cref{eqn:gap_prop_ineq2_a} from $k=1$ to $N$ and using \\cref{eqn:gap_prop_ineq3_a}, we obtain:\n\\begin{align*}\n&\\sum_{k=1}^{N}\\gamma_k^r\\left(f(x_k)-f(x^*)\\right) \\leq \\frac{\\gamma_{0}^{r-1}}{2\\eta_0} \\left(  \\mathcal{D}\\left(x_1,x^*\\right)  + \\left\\Vert u_1-x^* \\right\\Vert^2 \\right) \n\\nonumber \\\\ \n&+4M^2\\prob{{min}}^{-1}\\left(\\frac{\\gamma_{{N}}^{r-1}}{\\eta_{N}} - \\frac{\\gamma_{{0}}^{r-1}}{\\eta_{0}}\\right)+\\sum_{k=1}^{N}\\eta_k^{-1}\\left(\\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right),\n\\end{align*}\nwhere we drop the nonpositive term. From relation \\cref{eqn:gap_prop_ineq1_a} when $k=0$, we have:\n\\begin{align*}\n\\gamma_0^r\\left(f(x_0)-f(x^*)\\right)& \\leq  \\frac{\\gamma_0^{r-1}}{2\\eta_0}\\left(\\mathcal{D}\\left(x_{0}, x^*\\right) +\\|u_0-x^*\\|^2\\right) \\nonumber\\\\\n&-\\frac{\\gamma_0^{r-1}}{2\\eta_0}\\left(\\mathcal{D}\\left(x_{1}, x^*\\right) +\\|u_{1}-x^*\\|^2\\right)+ \\eta_0^{-1}\\left(\\Theta_{0,1}+\\Theta_{0,2}+\\Theta_{0,3}\\right).\n\\end{align*}\nAdding the last two inequalities, {multiplying and dividing the left-hand side by $\\sum_{k=0}^{N}\\gamma_k^r$, and then, invoking} \\cref{Lemma 2.10} and convexity of $f$, we obtain:\n\\begin{align*}\n&\\left(\\sum_{k=0}^{N}\\gamma_k^r\\right)\\left(f\\left(\\bar x_N\\right)-f(x^*)\\right) \\leq \\frac{\\gamma_{0}^{r-1}}{2\\eta_0} \\left(  \\mathcal{D}\\left(x_0,x^*\\right)  + \\left\\Vert u_0-x^* \\right\\Vert^2 \\right) \n\\nonumber \\\\ \n&+4M^2\\prob{{min}}^{-1}\\left(\\frac{\\gamma_{{N}}^{r-1}}{\\eta_{N}} - \\frac{\\gamma_{{0}}^{r-1}}{\\eta_{0}}\\right)+\\sum_{k=0}^{N}\\eta_k^{-1}\\left( \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right).\n\\end{align*}\nTaking the expectation on both sides and invoking \\cref{eqn:gap_prop_ineq3_a}, we obtain:\n\\begin{align*}\n&\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{\\frac{4M^2\\prob{{min}}^{-1}\\gamma_{{N}}^{r-1}}{\\eta_{N}} +\\sum_{k=0}^{N}\\eta_k^{-1}\\EXP{\\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}}}{\\sum_{k=0}^{N}\\gamma_k^r}.\n\\end{align*}\nFrom the relations \\cref{eqn:estimate_deltas_1_2} and \\cref{eqn:estimate_deltas_3}, we obtain:\n\\begin{align*}\n&\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{\\frac{4M^2\\prob{{min}}^{-1}\\gamma_{{N}}^{r-1}}{\\eta_{N}}+\\sum_{k=0}^{N}\\eta_k^{-1}\\left(\\prob{{min}}^{-1}\\gamma_k^{r+1}\\left(C_F^2+\\eta_k^2C_f^2\\right)\\right)}{\\sum_{k=0}^{N}\\gamma_k^r},\n\\end{align*}\nwhich implies the inequality \\cref{eqn:f_bound}. \n\n\\noindent (b) From the Cauchy-Schwarz inequality, the definitions of $C_f$ and $M$, and the triangle inequality, we have $ \\tilde \\nabla f(x_k)^T(y-x_k) \\leq \\left\\| \\tilde \\nabla f(x_k)\\right\\|\\|x_k-y\\| \\leq 2C_f M.$\nAdding the preceding inequality with the relation \\cref{eqn:prebound_gap}, {from \\cref{eqn:three_terms_abr}} we obtain:\n\\begin{align}\\label{eqn:prop_gap_proof_ineq0}\n&\\gamma_k^rF(y)^T(x_k-y) \\leq \\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k}, y\\right) +\\|u_k-y\\|^2\\right) \\nonumber\\\\\n&-\\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k+1}, y\\right) +\\|u_{k+1}-y\\|^2\\right)+ 2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}.\n\\end{align}\nAdding and subtracting the term $\\frac{\\gamma_{{k-1}}^{r-1}}{2} \\left( \\mathcal{D}\\left(x_k,y\\right) + \\left\\Vert u_k-y \\right\\Vert^2 \\right)$, we obtain:\n\\begin{align*}\n\t\t\t&\\gamma_k^rF(y)^T(x_k-y) \\leq  \\frac{\\gamma_{{k-1}}^{r-1}}{2} \\left(  \\mathcal{D}\\left(x_k,y\\right)  + \\left\\Vert u_k-y \\right\\Vert^2 \\right) \\nonumber \\\\\n\t\t\t&- \\frac{\\gamma_{{k}}^{r-1}}{2} \\left(  \\mathcal{D}\\left(x_{k+1},y\\right)  + \\left\\Vert u_{k+1}-y \\right\\Vert^2 \\right)+ \\frac{1}{2} \\left(\\gamma_{{k}}^{r-1} - \\gamma_{{k-1}}^{r-1}\\right)\\left( \\mathcal{D}\\left(x_k,y\\right) + \\left\\Vert u_k-y \\right\\Vert^2 \\right)\\nonumber \\\\\n&+ 2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}.\n\\end{align*}\nSubstituting the bound given by \\cref{eqn:gap_prop_ineq3_a} in the preceding relation, we obtain:\n\\begin{align*}\n\t\t\t&\\gamma_k^rF(y)^T(x_k-y) \\leq  \\frac{\\gamma_{{k-1}}^{r-1}}{2} \\left(  \\mathcal{D}\\left(x_k,y\\right)  + \\left\\Vert u_k-y \\right\\Vert^2 \\right) \\nonumber \\\\\n\t\t\t&- \\frac{\\gamma_{{k}}^{r-1}}{2} \\left(  \\mathcal{D}\\left(x_{k+1},y\\right)  + \\left\\Vert u_{k+1}-y \\right\\Vert^2 \\right)+ 4M^2\\prob{{min}}^{-1} \\left(\\gamma_{{k}}^{r-1} - \\gamma_{{k-1}}^{r-1}\\right)\\nonumber \\\\\n&+ 2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}.\n\\end{align*}\nSumming both sides from $k = 1 $ to $N$, we obtain:\n\\begin{align}\\label{eqn:prop_gap_proof_ineq1}\n\\sum_{k=1}^{N}\\gamma_{{k}}^r F(y)^T \\left(x_k-y\\right)& \\leq  \\frac{\\gamma_{{0}}^{r-1}}{2} \\left( \\mathcal{D}\\left(x_1,y\\right) + \\left\\Vert u_1-y \\right\\Vert^2 \\right)+ 4 M^2\\prob{{min}}^{-1} \\left( \\gamma_{{N}}^{r-1} - \\gamma_{{0}}^{r-1}\\right) \\nonumber \\\\\n\t\t\t%&- \\frac{\\gamma_{{N}}^{r-1}}{2} \\left(  \\mathcal{D}\\left(x_{N},y\\right)  + \\left\\Vert u_{N}-y \\right\\Vert^2 \\right) \\nonumber \\\\\n\t\t\t& +\\sum_{k=1}^{N}\\left(  2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right).\n\\end{align}\nWriting the inequality \\cref{eqn:prop_gap_proof_ineq0} for $k =0$, we have:\n\t\\begin{align}\\label{eqn:prop_gap_proof_ineq2}\n\\gamma_0^rF(y)^T(x_0-y) &\\leq \\frac{\\gamma_0^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{0}, y\\right) +\\|u_0-y\\|^2\\right)-\\frac{\\gamma_0^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{1}, y\\right) +\\|u_{1}-y\\|^2\\right) \\nonumber\\\\\n&+ 2\\gamma_0^r\\eta_0C_f M+ \\Theta_{0,1}+\\Theta_{0,2}+\\Theta_{0,3}.\n\\end{align}\nAdding \\cref{eqn:prop_gap_proof_ineq1} and \\cref{eqn:prop_gap_proof_ineq2} together, we obtain:\n\\begin{align}\\label{eqn:prop_gap_proof_ineq3}\n\\sum_{k=0}^{N}\\gamma_{{k}}^r F(y)^T \\left(x_k-y\\right) &\\leq  \\frac{\\gamma_{{0}}^{r-1}}{2} \\left( \\mathcal{D}\\left(x_0,y\\right) + \\left\\Vert u_0 -y \\right\\Vert^2 \\right) + 4 M^2\\prob{{min}}^{-1} \\left( \\gamma_{{N}}^{r-1} - \\gamma_{{0}}^{r-1}\\right) \\nonumber \\\\& +\\sum_{k=0}^{N}\\left(  2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right).\n\\end{align}\nRecalling $\\bar{x}_{N} = \\sum_{k=0}^N \\lambda_{k,N} x_k$ in \\cref{Lemma 2.10}, applying the bound given by \\cref{eqn:gap_prop_ineq3_a}, and using the triangle inequality, we obtain:\n\t\\begin{align*}\n\\left(\\sum_{k=0}^{N} \\gamma_{{k}}^r \\right) F(y)^T \\left(\\bar{x}_N-y\\right) \\leq& {4 M^2\\prob{{min}}^{-1}  \\gamma_{{N}}^{r-1} }  +\\sum_{k=0}^{N}\\left(  2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right).\n\\end{align*}\nTaking the supremum with respect to $y$ over the set $X$ from the left-hand side, invoking \\cref{def:gap}, and then dividing both sides by $\\sum_{k=0}^{N} \\gamma_{{k}}^r $, we obtain:\n\t\\begin{align*}\n\\text{GAP}\\left(\\bar x_N\\right) \\leq \\frac{4 M^2\\prob{{min}}^{-1}  \\gamma_{{N}}^{r-1}   +\\sum_{k=0}^{N}\\left(  2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right)}{\\sum_{k=0}^{N} \\gamma_{{k}}^r }.\n\\end{align*}\nTaking the expectation on both sides, using the relations \\cref{eqn:estimate_deltas_1_2} and \\cref{eqn:estimate_deltas_3}, and rearranging the terms, we obtain the inequality \\cref{eqn:gap_bound}.\n\\end{proof}","sha256_utf8":"ebc9c4301e75fabeed8684375d096c873ce00354b290b7f025a83c12a85239aa"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"定义三个随机余项","source_units":[{"start":52533,"end":53007,"start_line":530,"end_line":535,"raw_tex":"\\begin{proof}\nWe define the following terms for all $k\\geq 0$, that appear in \\cref{eqn:prebound_gap}:\n\\begin{align}\\label{eqn:three_terms_abr}\n& {\\Theta_{k,1} \\triangleq \\gamma_k^r(x_k-u_k)^T\\left(\\Delta_k+\\eta_k\\delta_k\\right)}, \\quad \\Theta_{k,2} \\triangleq \\gamma_k^{r+1}\\left(\\|\\Delta_k\\|^2+\\eta_k^2\\|\\delta_k\\|^2\\right),\\nonumber \\\\ \n&\\Theta_{k,3} \\triangleq 0.5\\prob{{i_k}}^{-1}\\gamma_k^{1+r}\\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\\end{align}\n","sha256_utf8":"117da5388a6e3ec9b12b4ec510c0b6a12ecfa61032d751d2d14e7c96a4ac298b"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"估计前两个余项的期望","source_units":[{"start":53007,"end":53869,"start_line":536,"end_line":539,"raw_tex":"Next, we estimate the expected values of these terms. {Consider the notation of} $\\mathcal{F}_k$ given by \\cref{def:Fk}. Note that $x_k$ is $\\mathcal{F}_k$--measurable. Also, from the definition of $u_k$ in \\cref{lem:prebound_gap}, $u_k$ is $\\mathcal{F}_k$--measurable. Note, however, that $\\Theta_{k,j}$ is $\\mathcal{F}_{k+1}$--measurable for all $j \\in \\{1,2,3\\}$. Taking these into account and using the total probability law, for any $k\\geq 0$ and $j \\in \\{1,2,3\\}$, we have $ \\EXP{\\Theta_{k,j}} = \\mathsf{E}_{\\mathcal{F}_k}\\left[\\mathsf{E}_{i_k}\\left[\\Theta_{k,j}\\mid \\mathcal{F}_k\\right]\\right]$. From this relation and \\cref{Lemma 2.12}, we have for any $k\\geq 0$:\n\\begin{align}\\label{eqn:estimate_deltas_1_2}\n&\\EXP{\\Theta_{k,1}} = 0, \\qquad \\EXP{\\Theta_{k,2}} = \\left(\\prob{{min}}^{-1}-1\\right)\\gamma_k^{r+1}\\left(C_F^2+\\eta_k^2C_f^2\\right).\n\\end{align}\n","sha256_utf8":"409dc1bd3df39b5f4a7b78cd5b7210952aef79bee7e994a11806a63e70a8bf34"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"估计正则化映射余项","source_units":[{"start":53869,"end":54609,"start_line":540,"end_line":549,"raw_tex":"Also, using \\cref{def:reg_map} and the triangle inequality, we can write:\n\\begin{align*}\n&{\\mathsf{E}_{i_k}\\left[\\Theta_{k,3}\\mid \\mathcal{F}_k\\right]} = \\sum_{i=1}^d\\prob{i}\\left(0.5\\prob{{i}}^{-1}\\gamma_k^{1+r}\\left\\Vert\\left.G_{k,i}\\left(x_k\\right)\\right\\Vert\\right.^2\\right)\\nonumber\\\\\n&\\leq \\gamma_k^{1+r}\\sum_{i=1}^d\\left(\\|F_i(x_k)\\|^2+\\eta_k^2\\|\\tilde \\nabla_i f(x_k)\\|^2\\right)  = \\gamma_k^{1+r}\\|F(x_k)\\|^2+\\eta_k^2\\|\\tilde \\nabla f(x_k)\\|^2.\n\\end{align*}\n{From the preceding inequality, we obtain:\n\\begin{align}\\label{eqn:estimate_deltas_3}\n\\EXP{\\Theta_{k,3}} & \\leq  \\gamma_k^{1+r}\\left(C_F^2+\\eta_k^2C_f^2\\right).\n\\end{align}\n}We are now ready to show the inequalities \\cref{eqn:f_bound} and \\cref{eqn:gap_bound} as follows:\\\\\n","sha256_utf8":"e4f349c4e017eeeeeb1a8f3be3295e75c0ef7785165f2e394c66231bdff1925f"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"以次梯度不等式引入目标差","source_units":[{"start":54609,"end":55179,"start_line":550,"end_line":554,"raw_tex":"\\noindent (a) Consider \\cref{eqn:prebound_gap}. From the definition of subgradients of the convex function $f$, we have that $f(x_k)-f(y) \\leq \\tilde \\nabla f(x_k)^T(x_k-y)$. Thus, {from \\cref{eqn:three_terms_abr}} we obtain for any $y \\in X$:\n\\begin{align*}\n&\\gamma_k^rF(y)^T(x_k-y) +\\gamma_k^r\\eta_k\\left(f(x_k)-f(y)\\right)\\leq \\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k}, y\\right) +\\|u_k-y\\|^2\\right) \\nonumber\\\\\n&-\\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k+1}, y\\right) +\\|u_{k+1}-y\\|^2\\right)+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}.\n\\end{align*}\n","sha256_utf8":"ad4fe09301d894bc176cdabf8956e07907eb290ed7ad7012adfd50489a8d4525"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"利用最优解的变分可行性","source_units":[{"start":55179,"end":55725,"start_line":555,"end_line":559,"raw_tex":"Let us substitute $y:=x^*$, where $x^*$ denotes an optimal solution to \\cref{prob:main}. Note that $x^*$ must be a feasible solution to \\cref{prob:main}, i.e., $F(x^*)^T(x_k-x^*) \\geq 0$. Thus, we obtain:\n\\begin{align}\\label{eqn:gap_prop_ineq1_a}\n\t\t\t&\\gamma_k^r\\eta_k\\left(f(x_k)-f(x^*)\\right) \\leq  \\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k}, x^*\\right) +\\|u_k-x^*\\|^2\\right) \\nonumber\\\\\n&-\\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k+1}, x^*\\right) +\\|u_{k+1}-x^*\\|^2\\right)+\\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}.\n\\end{align}\n","sha256_utf8":"8a63797f2eda34554a11fc155aa2cbc5387c7d5bc3f7b1942bca84dde640adbe"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"构造目标差的望远镜形式","source_units":[{"start":55725,"end":56618,"start_line":560,"end_line":565,"raw_tex":"Dividing both sides by $\\eta_k$ and adding and subtracting $\\frac{\\gamma_{{k-1}}^{r-1}}{2\\eta_{k-1}} \\left( \\mathcal{D}\\left(x_k,x^*\\right) + \\left\\Vert u_k-x^* \\right\\Vert^2 \\right)$ in the right-hand side of \\cref{eqn:gap_prop_ineq1_a}, we obtain for $k\\geq 1$:\n\\begin{align}\\label{eqn:gap_prop_ineq2_a}\n&\\gamma_k^r\\left(f(x_k)-f(x^*)\\right) \\leq  \\frac{\\gamma_{{k-1}}^{r-1}}{2\\eta_{k-1}} \\left(  \\mathcal{D}\\left(x_k,x^*\\right)  + \\left\\Vert u_k-x^* \\right\\Vert^2 \\right) \\nonumber \\\\\n&- \\frac{\\gamma_{{k}}^{r-1}}{2\\eta_k} \\left(  \\mathcal{D}\\left(x_{k+1},x^*\\right)  + \\left\\Vert u_{k+1}-x^* \\right\\Vert^2 \\right)\\nonumber \\\\ \n&+ \\frac{1}{2} \\left(\\frac{\\gamma_{{k}}^{r-1}}{\\eta_k} - \\frac{\\gamma_{{k-1}}^{r-1}}{\\eta_{k-1}}\\right)\\left( \\mathcal{D}\\left(x_k,x^*\\right) + \\left\\Vert u_k-x^* \\right\\Vert^2 \\right)+\\eta_k^{-1}\\left( \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right).\n\\end{align}\n","sha256_utf8":"4bf5fcca13f880e943d1c199d32e7db5fd9ec7a592ff9783c49dfe724e55045a"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"以有界域控制距离项","source_units":[{"start":56618,"end":57226,"start_line":566,"end_line":571,"raw_tex":"Since $r-1 < 0$ and that $\\{\\gamma_k\\}$ and $\\{\\eta_k\\}$ are nonincreasing, we have $\\frac{\\gamma_{{k}}^{r-1}}{\\eta_k} - \\frac{\\gamma_{{k-1}}^{r-1}}{\\eta_{k-1}} \\geq 0$. Also, from the boundedness of the set $X$, since $x_k$, $x^*$, and {$u_k$} belong to $ X$, using \\cref{rem:err_D} and the triangle inequality, we have:\n\\begin{align}\\label{eqn:gap_prop_ineq3_a}\n\\mathcal{D}\\left(x_k,x^*\\right)  + \\left\\Vert u_k-x^* \\right\\Vert^2 \n\\leq \\prob{{min}}^{-1}\\left\\Vert x_k-x^* \\right\\Vert^2 + \\left\\Vert u_k-x^* \\right\\Vert^2\n\\leq 4M^2\\left(\\prob{{min}}^{-1}+1\\right)\\leq \\frac{8M^2}{\\prob{{min}}}.\n\\end{align}\n","sha256_utf8":"a75db1c00e974485f34d567c78ddb8cfd2db6e325546c1f2f9cced5e21f5585f"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"累加正指标目标差","source_units":[{"start":57226,"end":57740,"start_line":572,"end_line":577,"raw_tex":"Summing over \\cref{eqn:gap_prop_ineq2_a} from $k=1$ to $N$ and using \\cref{eqn:gap_prop_ineq3_a}, we obtain:\n\\begin{align*}\n&\\sum_{k=1}^{N}\\gamma_k^r\\left(f(x_k)-f(x^*)\\right) \\leq \\frac{\\gamma_{0}^{r-1}}{2\\eta_0} \\left(  \\mathcal{D}\\left(x_1,x^*\\right)  + \\left\\Vert u_1-x^* \\right\\Vert^2 \\right) \n\\nonumber \\\\ \n&+4M^2\\prob{{min}}^{-1}\\left(\\frac{\\gamma_{{N}}^{r-1}}{\\eta_{N}} - \\frac{\\gamma_{{0}}^{r-1}}{\\eta_{0}}\\right)+\\sum_{k=1}^{N}\\eta_k^{-1}\\left(\\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right),\n\\end{align*}\n","sha256_utf8":"025a09ad75d9925a1333d24b3ae2e26d560fa71f2e3dcf63a85cdd12240c6bc6"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"补入初始目标差","source_units":[{"start":57740,"end":58178,"start_line":578,"end_line":582,"raw_tex":"where we drop the nonpositive term. From relation \\cref{eqn:gap_prop_ineq1_a} when $k=0$, we have:\n\\begin{align*}\n\\gamma_0^r\\left(f(x_0)-f(x^*)\\right)& \\leq  \\frac{\\gamma_0^{r-1}}{2\\eta_0}\\left(\\mathcal{D}\\left(x_{0}, x^*\\right) +\\|u_0-x^*\\|^2\\right) \\nonumber\\\\\n&-\\frac{\\gamma_0^{r-1}}{2\\eta_0}\\left(\\mathcal{D}\\left(x_{1}, x^*\\right) +\\|u_{1}-x^*\\|^2\\right)+ \\eta_0^{-1}\\left(\\Theta_{0,1}+\\Theta_{0,2}+\\Theta_{0,3}\\right).\n\\end{align*}\n","sha256_utf8":"d64e961cd55e3345998baa63bd265e9070dde9c671811852cec68369839012f5"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"由凸性转为平均点目标差","source_units":[{"start":58178,"end":58794,"start_line":583,"end_line":588,"raw_tex":"Adding the last two inequalities, {multiplying and dividing the left-hand side by $\\sum_{k=0}^{N}\\gamma_k^r$, and then, invoking} \\cref{Lemma 2.10} and convexity of $f$, we obtain:\n\\begin{align*}\n&\\left(\\sum_{k=0}^{N}\\gamma_k^r\\right)\\left(f\\left(\\bar x_N\\right)-f(x^*)\\right) \\leq \\frac{\\gamma_{0}^{r-1}}{2\\eta_0} \\left(  \\mathcal{D}\\left(x_0,x^*\\right)  + \\left\\Vert u_0-x^* \\right\\Vert^2 \\right) \n\\nonumber \\\\ \n&+4M^2\\prob{{min}}^{-1}\\left(\\frac{\\gamma_{{N}}^{r-1}}{\\eta_{N}} - \\frac{\\gamma_{{0}}^{r-1}}{\\eta_{0}}\\right)+\\sum_{k=0}^{N}\\eta_k^{-1}\\left( \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right).\n\\end{align*}\n","sha256_utf8":"b155d53ae201ce0f1c24a2a585fa3a6bd03c151ad312168aa37fe8955c802921"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"取期望得到次优性上界","source_units":[{"start":58794,"end":59524,"start_line":589,"end_line":597,"raw_tex":"Taking the expectation on both sides and invoking \\cref{eqn:gap_prop_ineq3_a}, we obtain:\n\\begin{align*}\n&\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{\\frac{4M^2\\prob{{min}}^{-1}\\gamma_{{N}}^{r-1}}{\\eta_{N}} +\\sum_{k=0}^{N}\\eta_k^{-1}\\EXP{\\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}}}{\\sum_{k=0}^{N}\\gamma_k^r}.\n\\end{align*}\nFrom the relations \\cref{eqn:estimate_deltas_1_2} and \\cref{eqn:estimate_deltas_3}, we obtain:\n\\begin{align*}\n&\\EXP{f\\left(\\bar x_N\\right)}-f(x^*) \\leq \\frac{\\frac{4M^2\\prob{{min}}^{-1}\\gamma_{{N}}^{r-1}}{\\eta_{N}}+\\sum_{k=0}^{N}\\eta_k^{-1}\\left(\\prob{{min}}^{-1}\\gamma_k^{r+1}\\left(C_F^2+\\eta_k^2C_f^2\\right)\\right)}{\\sum_{k=0}^{N}\\gamma_k^r},\n\\end{align*}\nwhich implies the inequality \\cref{eqn:f_bound}. \n","sha256_utf8":"15dd4683317f3a625188080fe3adfffb0271f48984e54d94360e98f32ddeee4e"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"控制正则项并保留间隙项","source_units":[{"start":59525,"end":60202,"start_line":599,"end_line":604,"raw_tex":"\\noindent (b) From the Cauchy-Schwarz inequality, the definitions of $C_f$ and $M$, and the triangle inequality, we have $ \\tilde \\nabla f(x_k)^T(y-x_k) \\leq \\left\\| \\tilde \\nabla f(x_k)\\right\\|\\|x_k-y\\| \\leq 2C_f M.$\nAdding the preceding inequality with the relation \\cref{eqn:prebound_gap}, {from \\cref{eqn:three_terms_abr}} we obtain:\n\\begin{align}\\label{eqn:prop_gap_proof_ineq0}\n&\\gamma_k^rF(y)^T(x_k-y) \\leq \\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k}, y\\right) +\\|u_k-y\\|^2\\right) \\nonumber\\\\\n&-\\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k+1}, y\\right) +\\|u_{k+1}-y\\|^2\\right)+ 2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}.\n\\end{align}\n","sha256_utf8":"1025ef2a9b41f7f2458a85d8d49890209050ca4f5881f8a3ec4fc5da6f391eac"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"构造间隙项的望远镜形式","source_units":[{"start":60202,"end":60890,"start_line":605,"end_line":610,"raw_tex":"Adding and subtracting the term $\\frac{\\gamma_{{k-1}}^{r-1}}{2} \\left( \\mathcal{D}\\left(x_k,y\\right) + \\left\\Vert u_k-y \\right\\Vert^2 \\right)$, we obtain:\n\\begin{align*}\n\t\t\t&\\gamma_k^rF(y)^T(x_k-y) \\leq  \\frac{\\gamma_{{k-1}}^{r-1}}{2} \\left(  \\mathcal{D}\\left(x_k,y\\right)  + \\left\\Vert u_k-y \\right\\Vert^2 \\right) \\nonumber \\\\\n\t\t\t&- \\frac{\\gamma_{{k}}^{r-1}}{2} \\left(  \\mathcal{D}\\left(x_{k+1},y\\right)  + \\left\\Vert u_{k+1}-y \\right\\Vert^2 \\right)+ \\frac{1}{2} \\left(\\gamma_{{k}}^{r-1} - \\gamma_{{k-1}}^{r-1}\\right)\\left( \\mathcal{D}\\left(x_k,y\\right) + \\left\\Vert u_k-y \\right\\Vert^2 \\right)\\nonumber \\\\\n&+ 2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}.\n\\end{align*}\n","sha256_utf8":"ee74f76e65ad9807f6a0155342c4e2d39d33eab9872600d0dd4801b9671d958d"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"代入统一距离界","source_units":[{"start":60890,"end":61454,"start_line":611,"end_line":616,"raw_tex":"Substituting the bound given by \\cref{eqn:gap_prop_ineq3_a} in the preceding relation, we obtain:\n\\begin{align*}\n\t\t\t&\\gamma_k^rF(y)^T(x_k-y) \\leq  \\frac{\\gamma_{{k-1}}^{r-1}}{2} \\left(  \\mathcal{D}\\left(x_k,y\\right)  + \\left\\Vert u_k-y \\right\\Vert^2 \\right) \\nonumber \\\\\n\t\t\t&- \\frac{\\gamma_{{k}}^{r-1}}{2} \\left(  \\mathcal{D}\\left(x_{k+1},y\\right)  + \\left\\Vert u_{k+1}-y \\right\\Vert^2 \\right)+ 4M^2\\prob{{min}}^{-1} \\left(\\gamma_{{k}}^{r-1} - \\gamma_{{k-1}}^{r-1}\\right)\\nonumber \\\\\n&+ 2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}.\n\\end{align*}\n","sha256_utf8":"4e93b0c5fd2d2aea98f35d27a898fa46b9dfbd23e4b30311a55c523f5a127ea2"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"累加正指标间隙项","source_units":[{"start":61454,"end":62056,"start_line":617,"end_line":622,"raw_tex":"Summing both sides from $k = 1 $ to $N$, we obtain:\n\\begin{align}\\label{eqn:prop_gap_proof_ineq1}\n\\sum_{k=1}^{N}\\gamma_{{k}}^r F(y)^T \\left(x_k-y\\right)& \\leq  \\frac{\\gamma_{{0}}^{r-1}}{2} \\left( \\mathcal{D}\\left(x_1,y\\right) + \\left\\Vert u_1-y \\right\\Vert^2 \\right)+ 4 M^2\\prob{{min}}^{-1} \\left( \\gamma_{{N}}^{r-1} - \\gamma_{{0}}^{r-1}\\right) \\nonumber \\\\\n\t\t\t%&- \\frac{\\gamma_{{N}}^{r-1}}{2} \\left(  \\mathcal{D}\\left(x_{N},y\\right)  + \\left\\Vert u_{N}-y \\right\\Vert^2 \\right) \\nonumber \\\\\n\t\t\t& +\\sum_{k=1}^{N}\\left(  2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right).\n\\end{align}\n","sha256_utf8":"e508fe5595dc0ae01f0d1ecbb11eb69adb8b02b029f1c59c73208508cf5e8223"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"补入初始间隙项","source_units":[{"start":62056,"end":62468,"start_line":623,"end_line":627,"raw_tex":"Writing the inequality \\cref{eqn:prop_gap_proof_ineq0} for $k =0$, we have:\n\t\\begin{align}\\label{eqn:prop_gap_proof_ineq2}\n\\gamma_0^rF(y)^T(x_0-y) &\\leq \\frac{\\gamma_0^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{0}, y\\right) +\\|u_0-y\\|^2\\right)-\\frac{\\gamma_0^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{1}, y\\right) +\\|u_{1}-y\\|^2\\right) \\nonumber\\\\\n&+ 2\\gamma_0^r\\eta_0C_f M+ \\Theta_{0,1}+\\Theta_{0,2}+\\Theta_{0,3}.\n\\end{align}\n","sha256_utf8":"348fae06424d0ce7f927e59ad79e88b04bdabe9c5b40c0efb95a9277ff85aaa1"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"合并全部指标的间隙和","source_units":[{"start":62468,"end":62979,"start_line":628,"end_line":631,"raw_tex":"Adding \\cref{eqn:prop_gap_proof_ineq1} and \\cref{eqn:prop_gap_proof_ineq2} together, we obtain:\n\\begin{align}\\label{eqn:prop_gap_proof_ineq3}\n\\sum_{k=0}^{N}\\gamma_{{k}}^r F(y)^T \\left(x_k-y\\right) &\\leq  \\frac{\\gamma_{{0}}^{r-1}}{2} \\left( \\mathcal{D}\\left(x_0,y\\right) + \\left\\Vert u_0 -y \\right\\Vert^2 \\right) + 4 M^2\\prob{{min}}^{-1} \\left( \\gamma_{{N}}^{r-1} - \\gamma_{{0}}^{r-1}\\right) \\nonumber \\\\& +\\sum_{k=0}^{N}\\left(  2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right).\n\\end{align}\n","sha256_utf8":"a8600117d66969f67a16216836cc4c27d19406465f4a5b5546968af5588825d6"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"转为平均点的线性间隙界","source_units":[{"start":62979,"end":63413,"start_line":632,"end_line":635,"raw_tex":"Recalling $\\bar{x}_{N} = \\sum_{k=0}^N \\lambda_{k,N} x_k$ in \\cref{Lemma 2.10}, applying the bound given by \\cref{eqn:gap_prop_ineq3_a}, and using the triangle inequality, we obtain:\n\t\\begin{align*}\n\\left(\\sum_{k=0}^{N} \\gamma_{{k}}^r \\right) F(y)^T \\left(\\bar{x}_N-y\\right) \\leq& {4 M^2\\prob{{min}}^{-1}  \\gamma_{{N}}^{r-1} }  +\\sum_{k=0}^{N}\\left(  2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right).\n\\end{align*}\n","sha256_utf8":"0a2616620aa4ae4f4312d1c74e8820b03730ad035dc5c3d2a42d869edebc65c9"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"取上确界与期望完成间隙界","source_units":[{"start":63413,"end":64049,"start_line":636,"end_line":641,"raw_tex":"Taking the supremum with respect to $y$ over the set $X$ from the left-hand side, invoking \\cref{def:gap}, and then dividing both sides by $\\sum_{k=0}^{N} \\gamma_{{k}}^r $, we obtain:\n\t\\begin{align*}\n\\text{GAP}\\left(\\bar x_N\\right) \\leq \\frac{4 M^2\\prob{{min}}^{-1}  \\gamma_{{N}}^{r-1}   +\\sum_{k=0}^{N}\\left(  2\\gamma_k^r\\eta_kC_f M+ \\Theta_{k,1}+\\Theta_{k,2}+\\Theta_{k,3}\\right)}{\\sum_{k=0}^{N} \\gamma_{{k}}^r }.\n\\end{align*}\nTaking the expectation on both sides, using the relations \\cref{eqn:estimate_deltas_1_2} and \\cref{eqn:estimate_deltas_3}, and rearranging the terms, we obtain the inequality \\cref{eqn:gap_bound}.\n\\end{proof}","sha256_utf8":"20deb3184c814b1e0b06d7fae09efe58b65dad50ca4aac80f006e4011896c062"}]},{"id":"S044","semantic_type":"lemma","kind":"lemma","title":"单步目标与间隙联合不等式","source_units":[{"start":45511,"end":46621,"start_line":454,"end_line":465,"raw_tex":"\\begin{lemma}\\label{lem:prebound_gap}\nConsider the sequence $\\{x_k\\}$ in \\cref{alg:aRB-IRG}. Suppose $\\{\\gamma_k\\}$ and $\\{\\eta_k\\}$ are strictly positive sequences. Let \\cref{assum:problem} and \\cref{assum:random sample} hold. Let the auxiliary sequence $\\{u_k\\} \\subset X$ be defined as \n\t\t%\\begin{align}\\label{equ:update_rule_aux_u} \n\t\t{$u_{k+1} \\triangleq \\mathcal{P}_X\\left(u_k-\\gamma_k\\left(\\Delta_k+\\eta_k\\delta_k\\right)\\right)$},\n\t\t%\\end{align}\nwhere $u_0 \\in X$ is an arbitrary vector. Then, for {all $y\\in X$ and all $k \\geq 0$, we have}:\n\\begin{align}\\label{eqn:prebound_gap}\n&\\gamma_k^rF(y)^T(x_k-y) +\\gamma_k^r\\eta_k\\tilde \\nabla f(x_k)^T(x_k-y)\\leq \\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k}, y\\right) +\\|u_k-y\\|^2\\right) \\nonumber\\\\\n&-\\frac{\\gamma_k^{r-1}}{2}\\left(\\mathcal{D}\\left(x_{k+1}, y\\right) +\\|u_{k+1}-y\\|^2\\right){+ \\gamma_k^r(x_k-u_k)^T\\left(\\Delta_k+\\eta_k\\delta_k\\right)}\\nonumber\\\\\n& +\\gamma_k^{r+1}\\left(\\|\\Delta_k\\|^2+\\eta_k^2\\|\\delta_k\\|^2\\right)+ 0.5\\prob{{i_k}}^{-1}\\gamma_k^{1+r}\\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\\end{align}\n\\end{lemma}","sha256_utf8":"fc6594965df5cdb372e2aa5d6e8a17b2c23ed1d4c84831481e3a5d02d0f4be63"}]},{"id":"S045","semantic_type":"assumption","kind":"assumption","title":"正参数与辅助投影序列","source_units":[{"start":45549,"end":46060,"start_line":455,"end_line":459,"raw_tex":"Consider the sequence $\\{x_k\\}$ in \\cref{alg:aRB-IRG}. Suppose $\\{\\gamma_k\\}$ and $\\{\\eta_k\\}$ are strictly positive sequences. Let \\cref{assum:problem} and \\cref{assum:random sample} hold. Let the auxiliary sequence $\\{u_k\\} \\subset X$ be defined as \n\t\t%\\begin{align}\\label{equ:update_rule_aux_u} \n\t\t{$u_{k+1} \\triangleq \\mathcal{P}_X\\left(u_k-\\gamma_k\\left(\\Delta_k+\\eta_k\\delta_k\\right)\\right)$},\n\t\t%\\end{align}\nwhere $u_0 \\in X$ is an arbitrary vector. Then, for {all $y\\in X$ and all $k \\geq 0$, we have}:\n","sha256_utf8":"5bd5a77529d0c297abc7c8c578d27e4dcea6b6d2e1078319bd0945d5c418bdad"}]},{"id":"S046","semantic_type":"proof","kind":"proof","title":"单步联合不等式完整证明","source_units":[{"start":46623,"end":51110,"start_line":467,"end_line":515,"raw_tex":"\\begin{proof} Let $k\\geq 1$ be fixed. From \\cref{def:err_D} and \\cref{equ:update_rule_aRBIRG}, for any $y \\in X$, we have:\n\t\\begin{align}\\label{eqn:D_split}\n\t\\mathcal{D} \\left(x_{k+1}, y\\right)  =   \\prob{{i_k}}^{-1} \\left\\Vert\\left.  x_{k+1}^{(i_k)}-  y^{(i_k)} \\right\\Vert\\right.^2+\\sum_{i=1, \\thinspace\t i \\neq i_k}^{d} \\prob{i}^{-1} \\left\\Vert\\left.  x_{k}^{(i)}-  y^{(i)} \\right\\Vert\\right.^2.\n\t\\end{align}Next, we find a bound on the term $\\left\\Vert\\left.  x_{k+1}^{(i_k)}- y^{(i_k)} \\right\\Vert\\right.^2$. From the block structure of $X$, we have $y^{(i_k)}\\in X_{i_k}$. Invoking the nonexpansiveness property of the projection mapping, the update rule \\cref{equ:update_rule_aRBIRG}, \\cref{def:reg_map}, and the preceding relation, we obtain:\n\t\t\\begin{align*} \n\t\\left\\Vert\\left. x_{k+1}^{(i_k)}-  y^{(i_k)}\\right\\Vert\\right.^2 \\leq\\left\\Vert\\left.  x_{k}^{(i_k)}-\\gamma_{k}G_{k,i_k} \\left( x_{k}\\right) -y^{(i_k)}\\right\\Vert\\right.^2.\n\t\\end{align*}\n\t\tCombining the preceding relation with \\cref{eqn:D_split}, we obtain:\n\t\\begin{align*}\n\t \t\\mathcal{D}\\left(x_{k+1}, y\\right)& \\leq \n\t \t \\sum_{i=1, \\thinspace\t i \\neq i_k}^{d} \\prob{i}^{-1} \\left\\Vert\\left.  x_{k}^{(i)}-  y^{(i)} \\right\\Vert\\right.^2 +  \\prob{{i_k}}^{-1}\\left\\Vert\\left.  x_{k}^{(i_k)}-  y^{(i_k)} \\right\\Vert\\right.^2 \\nonumber \\\\\n& - 2\\,\\prob{{i_k}}^{-1}\\gamma_k\\left( x^{(i_k)}_{k} - y^{(i_k)}\\right)^TG_{k,i_k}\\left(x_k\\right)+ \\prob{{i_k}}^{-1}\\gamma_k^2 \\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\t\\end{align*}\nInvoking \\cref{def:err_D} again, we obtain:\t\n\t\t\\begin{align}\\label{eqn:erc_D_elementary_ineq1}\n\t \t\\mathcal{D}\\left(x_{k+1}, y\\right) & \\leq  \\mathcal{D}\\left(x_{k}, y\\right) - 2\\,\\prob{{i_k}}^{-1}\\gamma_k\\left( x^{(i_k)}_{k} - y^{(i_k)}\\right)^TG_{k,i_k}\\left(x_k\\right)+ \\prob{{i_k}}^{-1}\\gamma_k^2 \\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\t\\end{align}\nFrom \\cref{def:deltas} and \\cref{def:reg_map}, we can write:\n\\begin{align*}\n&\\prob{{i_k}}^{-1}\\left( x^{(i_k)}_{k} - y^{(i_k)}\\right)^TG_{k,i_k}\\left(x_k\\right) = \\prob{{i_k}}^{-1}(x_k-y)^T\\left({\\mathbf{U}}_{i_k}G_{k,i_k}\\left(x_k\\right)\\right) \\\\ \n&=  \\prob{{i_k}}^{-1}(x_k-y)^T\\left({\\mathbf{U}}_{i_k}F_{i_k}(x_k)+\\eta_k{\\mathbf{U}}_{i_k}\\tilde \\nabla_{i_k}f\\left(x_k\\right)\\right) \\\\\n& {=(x_k-y)^T\\left(F(x_k)-\\Delta_k+\\eta_k\\tilde \\nabla f(x_k)-\\eta_k\\delta_k\\right) = (x_k-y)^T\\left(G_k(x_k)-\\Delta_k-\\eta_k\\delta_k\\right).}\n\\end{align*}\n\tCombining the preceding inequality and relation \\cref{eqn:erc_D_elementary_ineq1}, we obtain:\n\t\\begin{align}\\label{eqn:erc_D_elementary_ineq2}\n\t \t\\mathcal{D}\\left(x_{k+1}, y\\right) & \\leq  \\mathcal{D}\\left(x_{k}, y\\right) {- 2 \\gamma_k(x_k-y)^T\\left(G_k(x_k)-\\Delta_k-\\eta_k\\delta_k\\right)}+ \\prob{{i_k}}^{-1}\\gamma_k^2 \\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\t\\end{align}\n\tConsider the definition of the auxiliary sequence $\\{u_k\\}$ in \\cref{lem:prebound_gap}. Invoking the nonexpansiveness property of the projection again, we can obtain:\n\\begin{align*}\n\\|u_{k+1}-y\\|^2  &\\leq{\\|u_k-\\gamma_k\\left(\\Delta_k+\\eta_k\\delta_k\\right)-y\\|^2}\\\\\n&=\\|u_k-y\\|^2{-2\\gamma_k(u_k-y)^T\\left(\\Delta_k+\\eta_k\\delta_k\\right)} + \\gamma_k^2\\|\\Delta_k+\\eta_k\\delta_k\\|^2.\n\\end{align*}\nThus, we have: \n\\begin{align*}\n\\|u_{k+1}-y\\|^2\\leq \\|u_k-y\\|^2{- 2\\gamma_k(u_k-y)^T\\left(\\Delta_k+\\eta_k\\delta_k\\right)} +2\\gamma_k^2\\|\\Delta_k\\|^2+2\\gamma_k^2\\eta_k^2\\|\\delta_k\\|^2.\n\\end{align*}\nAdding the preceding inequality and the inequality \\cref{eqn:erc_D_elementary_ineq2} together, we obtain:\n\\begin{align}\\label{eqn:erc_D_elementary_ineq3}\n&2\\gamma_k(x_k-y)^T G_k(x_k) \\leq \\left(\\mathcal{D}\\left(x_{k}, y\\right) +\\|u_k-y\\|^2\\right) -\\left(\\mathcal{D}\\left(x_{k+1}, y\\right) +\\|u_{k+1}-y\\|^2\\right)\\nonumber\\\\\n& {+ 2\\gamma_k(x_k-u_k)^T\\left(\\Delta_k+\\eta_k\\delta_k\\right)}+2\\gamma_k^2\\left(\\|\\Delta_k\\|^2+\\eta_k^2\\|\\delta_k\\|^2\\right)+ \\prob{{i_k}}^{-1}\\gamma_k^2 \\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\\end{align}\nFrom the monotonicity property of the mapping $F$ and \\cref{def:reg_map}, we have:\n\\begin{align*}\n(x_k-y)^TG_k(x_k)\n%=(x_k-y)^T\\left(F(x_k)+\\eta_k\\tilde \\nabla f(x_k)\\right) \n\\geq (x_k-y)^TF(y)+\\eta_k\\tilde \\nabla f(x_k)^T(x_k-y).\n\\end{align*}\nThis provides a lower bound on the left-hand side of \\cref{eqn:erc_D_elementary_ineq3}. The inequality \\cref{eqn:prebound_gap} is obtained by substituting this bound in \\cref{eqn:erc_D_elementary_ineq3} and multiplying both sides by $\\frac{\\gamma_{k}^{r-1}}{2}$.\n\\end{proof}","sha256_utf8":"107f5a742d72f72f9cdb96a5f1b22ab11c47f59f1f1041fea0e88642d8986aa0"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"分解距离并应用投影非扩张性","source_units":[{"start":46623,"end":47580,"start_line":467,"end_line":473,"raw_tex":"\\begin{proof} Let $k\\geq 1$ be fixed. From \\cref{def:err_D} and \\cref{equ:update_rule_aRBIRG}, for any $y \\in X$, we have:\n\t\\begin{align}\\label{eqn:D_split}\n\t\\mathcal{D} \\left(x_{k+1}, y\\right)  =   \\prob{{i_k}}^{-1} \\left\\Vert\\left.  x_{k+1}^{(i_k)}-  y^{(i_k)} \\right\\Vert\\right.^2+\\sum_{i=1, \\thinspace\t i \\neq i_k}^{d} \\prob{i}^{-1} \\left\\Vert\\left.  x_{k}^{(i)}-  y^{(i)} \\right\\Vert\\right.^2.\n\t\\end{align}Next, we find a bound on the term $\\left\\Vert\\left.  x_{k+1}^{(i_k)}- y^{(i_k)} \\right\\Vert\\right.^2$. From the block structure of $X$, we have $y^{(i_k)}\\in X_{i_k}$. Invoking the nonexpansiveness property of the projection mapping, the update rule \\cref{equ:update_rule_aRBIRG}, \\cref{def:reg_map}, and the preceding relation, we obtain:\n\t\t\\begin{align*} \n\t\\left\\Vert\\left. x_{k+1}^{(i_k)}-  y^{(i_k)}\\right\\Vert\\right.^2 \\leq\\left\\Vert\\left.  x_{k}^{(i_k)}-\\gamma_{k}G_{k,i_k} \\left( x_{k}\\right) -y^{(i_k)}\\right\\Vert\\right.^2.\n\t\\end{align*}\n","sha256_utf8":"71934b69323bbd1284aa050f33bcf2932aa455dbe44d3754e6e96705ec6c00da"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"展开分块更新的距离递推","source_units":[{"start":47580,"end":48509,"start_line":474,"end_line":483,"raw_tex":"\t\tCombining the preceding relation with \\cref{eqn:D_split}, we obtain:\n\t\\begin{align*}\n\t \t\\mathcal{D}\\left(x_{k+1}, y\\right)& \\leq \n\t \t \\sum_{i=1, \\thinspace\t i \\neq i_k}^{d} \\prob{i}^{-1} \\left\\Vert\\left.  x_{k}^{(i)}-  y^{(i)} \\right\\Vert\\right.^2 +  \\prob{{i_k}}^{-1}\\left\\Vert\\left.  x_{k}^{(i_k)}-  y^{(i_k)} \\right\\Vert\\right.^2 \\nonumber \\\\\n& - 2\\,\\prob{{i_k}}^{-1}\\gamma_k\\left( x^{(i_k)}_{k} - y^{(i_k)}\\right)^TG_{k,i_k}\\left(x_k\\right)+ \\prob{{i_k}}^{-1}\\gamma_k^2 \\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\t\\end{align*}\nInvoking \\cref{def:err_D} again, we obtain:\t\n\t\t\\begin{align}\\label{eqn:erc_D_elementary_ineq1}\n\t \t\\mathcal{D}\\left(x_{k+1}, y\\right) & \\leq  \\mathcal{D}\\left(x_{k}, y\\right) - 2\\,\\prob{{i_k}}^{-1}\\gamma_k\\left( x^{(i_k)}_{k} - y^{(i_k)}\\right)^TG_{k,i_k}\\left(x_k\\right)+ \\prob{{i_k}}^{-1}\\gamma_k^2 \\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\t\\end{align}\n","sha256_utf8":"d1e8143d3653b3b605b50ea217d8ca5c64c1c113ca2647f89ef89a888cb5bb14"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"以随机误差表示分块内积","source_units":[{"start":48509,"end":49450,"start_line":484,"end_line":493,"raw_tex":"From \\cref{def:deltas} and \\cref{def:reg_map}, we can write:\n\\begin{align*}\n&\\prob{{i_k}}^{-1}\\left( x^{(i_k)}_{k} - y^{(i_k)}\\right)^TG_{k,i_k}\\left(x_k\\right) = \\prob{{i_k}}^{-1}(x_k-y)^T\\left({\\mathbf{U}}_{i_k}G_{k,i_k}\\left(x_k\\right)\\right) \\\\ \n&=  \\prob{{i_k}}^{-1}(x_k-y)^T\\left({\\mathbf{U}}_{i_k}F_{i_k}(x_k)+\\eta_k{\\mathbf{U}}_{i_k}\\tilde \\nabla_{i_k}f\\left(x_k\\right)\\right) \\\\\n& {=(x_k-y)^T\\left(F(x_k)-\\Delta_k+\\eta_k\\tilde \\nabla f(x_k)-\\eta_k\\delta_k\\right) = (x_k-y)^T\\left(G_k(x_k)-\\Delta_k-\\eta_k\\delta_k\\right).}\n\\end{align*}\n\tCombining the preceding inequality and relation \\cref{eqn:erc_D_elementary_ineq1}, we obtain:\n\t\\begin{align}\\label{eqn:erc_D_elementary_ineq2}\n\t \t\\mathcal{D}\\left(x_{k+1}, y\\right) & \\leq  \\mathcal{D}\\left(x_{k}, y\\right) {- 2 \\gamma_k(x_k-y)^T\\left(G_k(x_k)-\\Delta_k-\\eta_k\\delta_k\\right)}+ \\prob{{i_k}}^{-1}\\gamma_k^2 \\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\t\\end{align}\n","sha256_utf8":"af1eee21535a6eac1e4ae128bfc6f6d9c007f1f23762d91e13ca97ed83fa5026"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"估计辅助投影序列距离","source_units":[{"start":49450,"end":50039,"start_line":494,"end_line":502,"raw_tex":"\tConsider the definition of the auxiliary sequence $\\{u_k\\}$ in \\cref{lem:prebound_gap}. Invoking the nonexpansiveness property of the projection again, we can obtain:\n\\begin{align*}\n\\|u_{k+1}-y\\|^2  &\\leq{\\|u_k-\\gamma_k\\left(\\Delta_k+\\eta_k\\delta_k\\right)-y\\|^2}\\\\\n&=\\|u_k-y\\|^2{-2\\gamma_k(u_k-y)^T\\left(\\Delta_k+\\eta_k\\delta_k\\right)} + \\gamma_k^2\\|\\Delta_k+\\eta_k\\delta_k\\|^2.\n\\end{align*}\nThus, we have: \n\\begin{align*}\n\\|u_{k+1}-y\\|^2\\leq \\|u_k-y\\|^2{- 2\\gamma_k(u_k-y)^T\\left(\\Delta_k+\\eta_k\\delta_k\\right)} +2\\gamma_k^2\\|\\Delta_k\\|^2+2\\gamma_k^2\\eta_k^2\\|\\delta_k\\|^2.\n\\end{align*}\n","sha256_utf8":"f581ef377388cce5ac94acf9a80b38cad3b34a17ad98e848451b37364a554755"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"合并两个距离递推","source_units":[{"start":50039,"end":50592,"start_line":503,"end_line":507,"raw_tex":"Adding the preceding inequality and the inequality \\cref{eqn:erc_D_elementary_ineq2} together, we obtain:\n\\begin{align}\\label{eqn:erc_D_elementary_ineq3}\n&2\\gamma_k(x_k-y)^T G_k(x_k) \\leq \\left(\\mathcal{D}\\left(x_{k}, y\\right) +\\|u_k-y\\|^2\\right) -\\left(\\mathcal{D}\\left(x_{k+1}, y\\right) +\\|u_{k+1}-y\\|^2\\right)\\nonumber\\\\\n& {+ 2\\gamma_k(x_k-u_k)^T\\left(\\Delta_k+\\eta_k\\delta_k\\right)}+2\\gamma_k^2\\left(\\|\\Delta_k\\|^2+\\eta_k^2\\|\\delta_k\\|^2\\right)+ \\prob{{i_k}}^{-1}\\gamma_k^2 \\left\\Vert\\left.G_{k,i_k}\\left(x_k\\right)\\right\\Vert\\right.^2.\n\\end{align}\n","sha256_utf8":"bc4b142276a38c4845f711ae3663dd105c2cd3e63a95bed29cefbd20851ca1fb"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"用单调性替换映射并缩放","source_units":[{"start":50592,"end":51110,"start_line":508,"end_line":515,"raw_tex":"From the monotonicity property of the mapping $F$ and \\cref{def:reg_map}, we have:\n\\begin{align*}\n(x_k-y)^TG_k(x_k)\n%=(x_k-y)^T\\left(F(x_k)+\\eta_k\\tilde \\nabla f(x_k)\\right) \n\\geq (x_k-y)^TF(y)+\\eta_k\\tilde \\nabla f(x_k)^T(x_k-y).\n\\end{align*}\nThis provides a lower bound on the left-hand side of \\cref{eqn:erc_D_elementary_ineq3}. The inequality \\cref{eqn:prebound_gap} is obtained by substituting this bound in \\cref{eqn:erc_D_elementary_ineq3} and multiplying both sides by $\\frac{\\gamma_{k}^{r-1}}{2}$.\n\\end{proof}","sha256_utf8":"bb377b27a968e8f5405b620004053757e3655e69eb2bc61b988ffd02f0e27eef"}]},{"id":"S053","semantic_type":"lemma","kind":"lemma","title":"平均迭代的加权表示","source_units":[{"start":40037,"end":40433,"start_line":380,"end_line":382,"raw_tex":"\\begin{lemma}[Weighted averaging]\\label{Lemma 2.10}\nLet $\\{\\bar x_k\\}$ be generated by \\cref{alg:aRB-IRG}. Let us define the weights $\\lambda_{k,N} \\triangleq \\frac{\\gamma_k^r}{\\sum_{j=0}^N \\gamma_j^r}$ for $k \\in \\{0,\\ldots, N\\}$ and $N\\geq 0$. Then, for any $N\\geq 0$, we have $\\bar{x}_{N} = \\sum_{k=0}^N \\lambda_{k,N} x_k$. Also, when $X$ is a convex set, we have $\\bar x_N \\in X$.\n\\end{lemma}","sha256_utf8":"77f7d2cc7e3379a54233611da003b059b9ac835369cdcfe450c57d66b8fec428"}]},{"id":"S054","semantic_type":"proof","kind":"proof","title":"加权表示的正文证明指引","source_units":[{"start":40434,"end":41474,"start_line":383,"end_line":394,"raw_tex":"\\begin{proof}\n\\begin{comment}\nWe use induction to show that $\\bar{x}_{N} = \\sum_{k=0}^N \\lambda_{k,N} x_k$ for any $N\\geq 0$. For $N=0$, the relation holds due to the initialization $\\bar{x}_0 :=x_0$ in \\cref{alg:aRB-IRG} and that $\\lambda_{0,0}=1$. Next, let the relation hold for some $N\\geq 0$. From \\cref{eqn:ave_step_of_alg} and noting that $S_{N}=\\sum_{k=0}^N\\gamma_k^r$ for any $N\\geq 0$, we can write the following: \n\\begin{align*}\n\t\\bar{x}_{N+1} = \\frac{S_N\\bar{x}_N + \\gamma_{N+1}^r x_{N+1}}{S_{N+1}} \n\t%= \\frac{\\sum_{k = 0}^{N}\\gamma_k^r x_k + \\gamma_{N+1}^r x_{N+1}}{S_{N+1}}\n\t = \\frac{\\sum_{k=0}^{N+1}\\gamma_k^r x_k}{\\sum_{k = 0}^{N+1}\\gamma_k^r} = \\sum_{k=0}^{N+1} \\lambda_{k,N+1} x_k,\n\t\\end{align*}\nimplying that the induction hypothesis holds for $N+1$. Thus, we conclude that the desired averaging formula holds for $N\\geq 0$. To complete the proof, note that since $\\sum_{k=0}^N \\lambda_{k,N}=1$, under the convexity of the set $X$, we have $\\bar x_N \\in X$.\n\\end{comment}\nSee \\cref{app:ave_x_second_formula}. \n\\end{proof}","sha256_utf8":"47f6338e8ea0cf2245e34780b04b970a9e6bde861851eed03949087e9c166252"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"将加权表示证明转至附录","source_units":[{"start":41425,"end":41474,"start_line":393,"end_line":394,"raw_tex":"See \\cref{app:ave_x_second_formula}. \n\\end{proof}","sha256_utf8":"986a55349956938b4db7083727da0c8a67855990bd0beffd7f637eda25cbd369"}]},{"id":"S056","semantic_type":"proof","kind":"proof","title":"附录加权表示完整证明","source_units":[{"start":106027,"end":107202,"start_line":975,"end_line":985,"raw_tex":"\\subsection{Proof of \\cref{Lemma 2.10}}\\label{app:ave_x_second_formula}\nWe use induction to show $\\bar{x}_{N} = \\sum_{k=0}^N \\lambda_{k,N} x_k$ for any $N\\geq 0$. For $N=0$, the relation holds due to the initialization $\\bar{x}_0 :=x_0$ in \\cref{alg:aRB-IRG} and that $\\lambda_{0,0}=1$. Next, let the relation hold for some $N\\geq 0$. From {the hypothesis, equation \\cref{eqn:ave_step_of_alg}, and} that $S_{N}=\\sum_{k=0}^N\\gamma_k^r$ for all $N\\geq 0$, we can write: \n\\begin{align*}\n\t\\bar{x}_{N+1} = \\frac{S_N\\bar{x}_N + \\gamma_{N+1}^r x_{N+1}}{S_{N+1}} \n\t%= \\frac{\\sum_{k = 0}^{N}\\gamma_k^r x_k + \\gamma_{N+1}^r x_{N+1}}{S_{N+1}}\n\t = \\frac{\\sum_{k=0}^{N+1}\\gamma_k^r x_k}{\\sum_{k = 0}^{N+1}\\gamma_k^r} = \\sum_{k=0}^{N+1} \\lambda_{k,N+1} x_k,\n\t\\end{align*}\nimplying that the induction hypothesis holds for $N+1$. Thus, we conclude that the desired averaging formula holds for {all} $N\\geq 0$. To complete the proof, note that since $\\sum_{k=0}^N \\lambda_{k,N}=1$, under the convexity of the set $X$, we have $\\bar x_N \\in X$.\n%\\section*{Acknowledgments}\n%We would like to acknowledge the assistance of volunteers in putting\n%together this example manuscript and supplement.\n","sha256_utf8":"9e49a04870587e2834dc68c27e63985bdda20405c702b982e70013bcaa8befd0"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"归纳基步与归纳假设","source_units":[{"start":106027,"end":106496,"start_line":975,"end_line":976,"raw_tex":"\\subsection{Proof of \\cref{Lemma 2.10}}\\label{app:ave_x_second_formula}\nWe use induction to show $\\bar{x}_{N} = \\sum_{k=0}^N \\lambda_{k,N} x_k$ for any $N\\geq 0$. For $N=0$, the relation holds due to the initialization $\\bar{x}_0 :=x_0$ in \\cref{alg:aRB-IRG} and that $\\lambda_{0,0}=1$. Next, let the relation hold for some $N\\geq 0$. From {the hypothesis, equation \\cref{eqn:ave_step_of_alg}, and} that $S_{N}=\\sum_{k=0}^N\\gamma_k^r$ for all $N\\geq 0$, we can write: \n","sha256_utf8":"d083587fe794f1c6680db7d14c8eb3db9560f07517ad956bdf6291a4c6206d4f"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"展开平均更新完成归纳步","source_units":[{"start":106496,"end":106785,"start_line":977,"end_line":981,"raw_tex":"\\begin{align*}\n\t\\bar{x}_{N+1} = \\frac{S_N\\bar{x}_N + \\gamma_{N+1}^r x_{N+1}}{S_{N+1}} \n\t%= \\frac{\\sum_{k = 0}^{N}\\gamma_k^r x_k + \\gamma_{N+1}^r x_{N+1}}{S_{N+1}}\n\t = \\frac{\\sum_{k=0}^{N+1}\\gamma_k^r x_k}{\\sum_{k = 0}^{N+1}\\gamma_k^r} = \\sum_{k=0}^{N+1} \\lambda_{k,N+1} x_k,\n\t\\end{align*}\n","sha256_utf8":"3a25b27bcdc82b79914a0d7dfd6e88cf3fce43e2492d6b83bf85dd714b587c47"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"归纳结论与凸组合可行性","source_units":[{"start":106785,"end":107054,"start_line":982,"end_line":982,"raw_tex":"implying that the induction hypothesis holds for $N+1$. Thus, we conclude that the desired averaging formula holds for {all} $N\\geq 0$. To complete the proof, note that since $\\sum_{k=0}^N \\lambda_{k,N}=1$, under the convexity of the set $X$, we have $\\bar x_N \\in X$.\n","sha256_utf8":"72c0e27c42f67db0cc6400bce87795d3e9ccad7d69a6567e67438522ddc2939d"}]},{"id":"S060","semantic_type":"lemma","kind":"lemma","title":"随机误差的条件矩性质","source_units":[{"start":42175,"end":42575,"start_line":402,"end_line":406,"raw_tex":"\\begin{lemma}[Properties of $\\Delta_k$ and $\\delta_k$]\\label{Lemma 2.12}\nConsider \\cref{def:deltas}. We have:\\\\\n\\noindent (a)  $\\EXP{\\Delta_k\\mid \\mathcal{F}_k}=\\EXP{\\delta_k\\mid \\mathcal{F}_k}=0$.\\\\\n\\noindent (b) $\\EXP{\\|\\Delta_k\\|^2\\mid \\mathcal{F}_k} \\leq  \\left(\\prob{{min}}^{-1}-1\\right)C_F^2$ and $\\EXP{\\|\\delta_k\\|^2\\mid \\mathcal{F}_k} \\leq  \\left(\\prob{{min}}^{-1}-1\\right)C_f^2$.\n\\end{lemma}","sha256_utf8":"dcf2dfff93d44c36863963330bbb75f78ff2a116c5817cda1e17fe4aab334ffc"}]},{"id":"S061","semantic_type":"proof","kind":"proof","title":"条件矩的正文证明指引","source_units":[{"start":42576,"end":43728,"start_line":407,"end_line":425,"raw_tex":"\\begin{proof} \n\\begin{comment}\n\\noindent (a) From the definition of $\\Delta_k$ and ${\\mathbf{U}}_i $, we can write:\n\\begin{align*} \n\\EXP{\\Delta_k\\mid \\mathcal{F}_k} =F(x_k)-\\sum_{i=1}^d\\prob{{i}}\\prob{{i}}^{-1}{\\mathbf{U}}_{i}F_{i}(x_k)= F(x_k) -\\sum_{i=1}^d{\\mathbf{U}}_{i}F_{i}(x_k) =0.\n\\end{align*}\nThe relation $\\EXP{\\delta_k\\mid \\mathcal{F}_k}=0$ can be shown in a similar fashion.\n\n\\noindent (b) We can write:\n\\begin{align*}\n \\EXP{\\|\\Delta_k\\|^2\\mid \\mathcal{F}_k}& = \\sum_{i=1}^d\\prob{{i}}\\left\\|F(x_k)-\\prob{{i}}^{-1}{\\mathbf{U}}_{i}F_{i}(x_k)\\right\\|^2\\\\\n &=  \\sum_{i=1}^d\\prob{{i}} \\left(\\|F(x_k)\\|^2 +\\prob{{i}}^{-2} \\left\\|{\\mathbf{U}}_{i}F_{i}(x_k)\\right\\|^2 - 2\\prob{{i}}^{-1}F(x_k)^T{\\mathbf{U}}_{i}F_{i}(x_k)\\right)\\\\\n & = \\|F(x_k)\\|^2 +\\sum_{i=1}^d\\prob{{i}}^{-1} \\left\\|{\\mathbf{U}}_{i}F_{i}(x_k)\\right\\|^2  -2\\sum_{i=1}^d\\|F_i(x_k)\\|^2\\\\\n & \\leq \\left(\\prob{{min}}^{-1}-1\\right)\\|F(x_k)\\|^2 \\leq \\left(\\prob{{min}}^{-1}-1\\right)C_F^2.\n\\end{align*}\nThe relation $\\EXP{\\|\\delta_k\\|^2\\mid \\mathcal{F}_k} \\leq  \\left(\\prob{{min}}^{-1}-1\\right)C_f^2$ can be shown using a similar approach. \n\\end{comment}\nSee \\cref{app:deltas}\n\\end{proof}","sha256_utf8":"82e5a7610352ff624fd6f32dafcb356b8909f6857a1c8e58e84d0a803f0392c1"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"将条件矩证明转至附录","source_units":[{"start":43695,"end":43728,"start_line":424,"end_line":425,"raw_tex":"See \\cref{app:deltas}\n\\end{proof}","sha256_utf8":"56d15b8dafb62c56f775bb301c08c02948f8b339a67aad88c979f6313a3822ab"}]},{"id":"S063","semantic_type":"proof","kind":"proof","title":"附录随机误差矩完整证明","source_units":[{"start":107202,"end":108295,"start_line":986,"end_line":1003,"raw_tex":"\\subsection{Proof of \\cref{Lemma 2.12}}\\label{app:deltas} (a) From \\cref{def:deltas}, we can write:\n\\begin{align*} \n\\EXP{\\Delta_k\\mid \\mathcal{F}_k} =F(x_k)-\\sum_{i=1}^d\\prob{{i}}\\prob{{i}}^{-1}{\\mathbf{U}}_{i}F_{i}(x_k)= F(x_k) -\\sum_{i=1}^d{\\mathbf{U}}_{i}F_{i}(x_k) =0.\n\\end{align*}\nThe relation $\\EXP{\\delta_k\\mid \\mathcal{F}_k}=0$ can be shown in a similar fashion.\n\n\\noindent (b) We can write:\n\\begin{align*}\n &\\EXP{\\|\\Delta_k\\|^2\\mid \\mathcal{F}_k} = \\sum_{i=1}^d\\prob{{i}}\\left\\|F(x_k)-\\prob{{i}}^{-1}{\\mathbf{U}}_{i}F_{i}(x_k)\\right\\|^2\\\\\n &=  \\sum_{i=1}^d\\prob{{i}} \\left(\\|F(x_k)\\|^2 +\\prob{{i}}^{-2} \\left\\|{\\mathbf{U}}_{i}F_{i}(x_k)\\right\\|^2 - 2\\prob{{i}}^{-1}F(x_k)^T{\\mathbf{U}}_{i}F_{i}(x_k)\\right)\\\\\n & = \\|F(x_k)\\|^2 +\\sum_{i=1}^d\\prob{{i}}^{-1} \\left\\|{\\mathbf{U}}_{i}F_{i}(x_k)\\right\\|^2  -2\\sum_{i=1}^d\\|F_i(x_k)\\|^2 \n % \\leq \\left(\\prob{{min}}^{-1}-1\\right)\\|F(x_k)\\|^2 \n \\leq \\left(\\prob{{min}}^{-1}-1\\right)C_F^2.\n\\end{align*}\nThe relation $\\EXP{\\|\\delta_k\\|^2\\mid \\mathcal{F}_k} \\leq  \\left(\\prob{{min}}^{-1}-1\\right)C_f^2$ can be shown using a similar approach. \n\n\n\n","sha256_utf8":"03dc128ee265be9a69f619222d7ab148c54e041d08ec27cf97d878a16b868361"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"计算随机误差的条件均值","source_units":[{"start":107202,"end":107573,"start_line":986,"end_line":990,"raw_tex":"\\subsection{Proof of \\cref{Lemma 2.12}}\\label{app:deltas} (a) From \\cref{def:deltas}, we can write:\n\\begin{align*} \n\\EXP{\\Delta_k\\mid \\mathcal{F}_k} =F(x_k)-\\sum_{i=1}^d\\prob{{i}}\\prob{{i}}^{-1}{\\mathbf{U}}_{i}F_{i}(x_k)= F(x_k) -\\sum_{i=1}^d{\\mathbf{U}}_{i}F_{i}(x_k) =0.\n\\end{align*}\nThe relation $\\EXP{\\delta_k\\mid \\mathcal{F}_k}=0$ can be shown in a similar fashion.\n","sha256_utf8":"31af8f52d713841ac80883b0ac69b2fd6ff5c5955b3baae0b4dc3c1b9f1b8a1b"}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","title":"计算随机误差的条件二阶矩界","source_units":[{"start":107574,"end":108292,"start_line":992,"end_line":1000,"raw_tex":"\\noindent (b) We can write:\n\\begin{align*}\n &\\EXP{\\|\\Delta_k\\|^2\\mid \\mathcal{F}_k} = \\sum_{i=1}^d\\prob{{i}}\\left\\|F(x_k)-\\prob{{i}}^{-1}{\\mathbf{U}}_{i}F_{i}(x_k)\\right\\|^2\\\\\n &=  \\sum_{i=1}^d\\prob{{i}} \\left(\\|F(x_k)\\|^2 +\\prob{{i}}^{-2} \\left\\|{\\mathbf{U}}_{i}F_{i}(x_k)\\right\\|^2 - 2\\prob{{i}}^{-1}F(x_k)^T{\\mathbf{U}}_{i}F_{i}(x_k)\\right)\\\\\n & = \\|F(x_k)\\|^2 +\\sum_{i=1}^d\\prob{{i}}^{-1} \\left\\|{\\mathbf{U}}_{i}F_{i}(x_k)\\right\\|^2  -2\\sum_{i=1}^d\\|F_i(x_k)\\|^2 \n % \\leq \\left(\\prob{{min}}^{-1}-1\\right)\\|F(x_k)\\|^2 \n \\leq \\left(\\prob{{min}}^{-1}-1\\right)C_F^2.\n\\end{align*}\nThe relation $\\EXP{\\|\\delta_k\\|^2\\mid \\mathcal{F}_k} \\leq  \\left(\\prob{{min}}^{-1}-1\\right)C_f^2$ can be shown using a similar approach. \n","sha256_utf8":"e7ec13eb62b9399d0b414d44ce19140511fcd4d15ad15773fe05f77b4ea563d6"}]},{"id":"S066","semantic_type":"lemma","kind":"lemma","title":"幂次级数的双边界","source_units":[{"start":43818,"end":44150,"start_line":426,"end_line":431,"raw_tex":"\\begin{lemma}[Bounds on the harmonic series]\\label{Lemma 2.13}\nLet $0\\leq \\alpha <1$ be a given scalar. Then, for any integer $N \\geq 2^{\\frac{1}{1-\\alpha}}-1$, we have \n%\\begin{align*}\n$\\frac{(N+1)^{1-\\alpha}}{2(1-\\alpha)}  \\leq \\sum_{k=0}^{N}\\frac{1}{(k+1)^\\alpha}\\leq \\frac{(N+1)^{1-\\alpha}}{1-\\alpha}$.\n%\\end{align*}\n\\end{lemma}","sha256_utf8":"e845f6615c474cdbb3df456e7b8ecee12fe06ac70cd650f396b22468021fbb4d"}]},{"id":"S067","semantic_type":"assumption","kind":"assumption","title":"级数指数与长度条件","source_units":[{"start":43881,"end":43988,"start_line":427,"end_line":427,"raw_tex":"Let $0\\leq \\alpha <1$ be a given scalar. Then, for any integer $N \\geq 2^{\\frac{1}{1-\\alpha}}-1$, we have \n","sha256_utf8":"bf23a9a4aabf51ad24f773f59fb77de86a690beab01ea0e138526eb374aea63f"}]},{"id":"S068","semantic_type":"proof","kind":"proof","title":"级数界的正文证明指引","source_units":[{"start":44151,"end":45237,"start_line":432,"end_line":448,"raw_tex":"\\begin{proof}\n\\begin{comment}Given $0\\leq \\alpha <1$, let us define the function $\\phi:\\mathbb{R}_{++} \\to \\mathbb{R}$ as $\\phi(x) \\triangleq x^{-\\alpha}$ for all $x>0$. Since $\\alpha >0$, the function $\\phi$ is nonincreasing. We can write:\n\\begin{align*}\n\\sum_{k=0}^{N}\\frac{1}{(k+1)^\\alpha}\n%=\\sum_{k=1}^{N+1}k^{-\\alpha}\n= 1 + \\sum_{k=2}^{N+1}\\frac{1}{k^\\alpha}\\leq 1+\\int_{1}^{N+1}\\frac{dx}{x^\\alpha} =1+\\frac{(N+1)^{1-\\alpha}-1}{1-\\alpha}\\leq \\frac{(N+1)^{1-\\alpha}}{1-\\alpha},\n\\end{align*}\nimplying the desired upper bound. To show that the lower bound holds, we can write:\n\\begin{align*}\n\\sum_{k=0}^{N}\\frac{1}{(k+1)^\\alpha}=\\sum_{k=1}^{N+1}\\frac{1}{k^\\alpha}\\geq \\int_{1}^{N+2}\\frac{dx}{x^\\alpha} \\geq \\int_{1}^{N+1}\\frac{dx}{x^\\alpha}\n%=\\frac{(N+1)^{1-\\alpha}-1}{1-\\alpha} \n\\geq \\frac{(N+1)^{1-\\alpha}-0.5(N+1)^{1-\\alpha}}{1-\\alpha},\n\\end{align*}\nwhere the last inequality is obtained using the assumption that $N \\geq 2^{\\frac{1}{1-\\alpha}}-1$. Therefore, the desired lower bound holds as well. This completes the proof.\n\\end{comment}\nSee \\cref{app:harmonic_bounds}\n\\end{proof}","sha256_utf8":"173edceaa559ae0d4dcdb3c012d8fabf81ceb90903c4688352a659ab8a162b53"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"将级数界证明转至附录","source_units":[{"start":45195,"end":45237,"start_line":447,"end_line":448,"raw_tex":"See \\cref{app:harmonic_bounds}\n\\end{proof}","sha256_utf8":"ec79afed6166959c4e7db542b14c0193d31787eac3f6630071fc5eac8edf5f87"}]},{"id":"S070","semantic_type":"proof","kind":"proof","title":"附录幂次级数界完整证明","source_units":[{"start":108295,"end":109363,"start_line":1004,"end_line":1016,"raw_tex":"\\subsection{Proof of \\cref{Lemma 2.13}}\\label{app:harmonic_bounds} Given $0\\leq \\alpha <1$, let us define the function $\\phi:\\mathbb{R}_{++} \\to \\mathbb{R}$ as $\\phi(x) \\triangleq x^{-\\alpha}$ for all $x>0$. Since $\\alpha >0$, the function $\\phi$ is nonincreasing. We can write:\n\\begin{align*}\n\\sum_{k=0}^{N}\\frac{1}{(k+1)^\\alpha}\n%=\\sum_{k=1}^{N+1}k^{-\\alpha}\n= 1 + \\sum_{k=2}^{N+1}\\frac{1}{k^\\alpha}\\leq 1+\\int_{1}^{N+1}\\frac{dx}{x^\\alpha} =1+\\frac{(N+1)^{1-\\alpha}-1}{1-\\alpha}\\leq \\frac{(N+1)^{1-\\alpha}}{1-\\alpha},\n\\end{align*}\nimplying the desired upper bound. To show that the lower bound holds, we can write:\n\\begin{align*}\n\\sum_{k=0}^{N}\\frac{1}{(k+1)^\\alpha}=\\sum_{k=1}^{N+1}\\frac{1}{k^\\alpha}\\geq \\int_{1}^{N+2}\\frac{dx}{x^\\alpha} \\geq \\int_{1}^{N+1}\\frac{dx}{x^\\alpha}\n%=\\frac{(N+1)^{1-\\alpha}-1}{1-\\alpha} \n\\geq \\frac{(N+1)^{1-\\alpha}-0.5(N+1)^{1-\\alpha}}{1-\\alpha},\n\\end{align*}\nwhere the last inequality is obtained using the assumption that $N \\geq 2^{\\frac{1}{1-\\alpha}}-1$. Therefore, the desired lower bound holds as well. This completes the proof.\n","sha256_utf8":"d8fd46e12811ce0295bcac9b67b4ac6ac6b477c756953185484532328a20c504"}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","title":"以积分比较得到级数上界","source_units":[{"start":108295,"end":108828,"start_line":1004,"end_line":1009,"raw_tex":"\\subsection{Proof of \\cref{Lemma 2.13}}\\label{app:harmonic_bounds} Given $0\\leq \\alpha <1$, let us define the function $\\phi:\\mathbb{R}_{++} \\to \\mathbb{R}$ as $\\phi(x) \\triangleq x^{-\\alpha}$ for all $x>0$. Since $\\alpha >0$, the function $\\phi$ is nonincreasing. We can write:\n\\begin{align*}\n\\sum_{k=0}^{N}\\frac{1}{(k+1)^\\alpha}\n%=\\sum_{k=1}^{N+1}k^{-\\alpha}\n= 1 + \\sum_{k=2}^{N+1}\\frac{1}{k^\\alpha}\\leq 1+\\int_{1}^{N+1}\\frac{dx}{x^\\alpha} =1+\\frac{(N+1)^{1-\\alpha}-1}{1-\\alpha}\\leq \\frac{(N+1)^{1-\\alpha}}{1-\\alpha},\n\\end{align*}\n","sha256_utf8":"0459a847384d26caebe7a777dd485ba910a0559197c624c011230ede91eeb3f9"}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","title":"以积分比较得到级数下界","source_units":[{"start":108828,"end":109363,"start_line":1010,"end_line":1016,"raw_tex":"implying the desired upper bound. To show that the lower bound holds, we can write:\n\\begin{align*}\n\\sum_{k=0}^{N}\\frac{1}{(k+1)^\\alpha}=\\sum_{k=1}^{N+1}\\frac{1}{k^\\alpha}\\geq \\int_{1}^{N+2}\\frac{dx}{x^\\alpha} \\geq \\int_{1}^{N+1}\\frac{dx}{x^\\alpha}\n%=\\frac{(N+1)^{1-\\alpha}-1}{1-\\alpha} \n\\geq \\frac{(N+1)^{1-\\alpha}-0.5(N+1)^{1-\\alpha}}{1-\\alpha},\n\\end{align*}\nwhere the last inequality is obtained using the assumption that $N \\geq 2^{\\frac{1}{1-\\alpha}}-1$. Therefore, the desired lower bound holds as well. This completes the proof.\n","sha256_utf8":"996db24372fb4f3a363223bee90d2cb4cb5a9f97279f5b68f97cccfa1921ddae"}]},{"id":"S073","semantic_type":"external_fact","kind":"external_fact","title":"欧氏投影非扩张性","source_units":[],"declared_external":true,"declaration_text":["非空闭凸集上的欧氏投影满足非扩张性，且对集合内的点保持不变。"]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","title":"凸组合与凸函数不等式","source_units":[],"declared_external":true,"declaration_text":["凸函数满足有限加权 Jensen 不等式；凸集包含其元素的有限凸组合。"]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","title":"条件期望运算法则","source_units":[],"declared_external":true,"declaration_text":["条件离散分布下的期望可按各取值概率求和；可测因子可提出条件期望，并可使用全期望公式。"]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","title":"欧氏范数基本不等式","source_units":[],"declared_external":true,"declaration_text":["欧氏范数满足三角不等式、Cauchy–Schwarz 不等式及 ||u+v||²≤2||u||²+2||v||²。"]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","title":"单调级数的积分比较","source_units":[],"declared_external":true,"declaration_text":["非负单调函数的级数可用相应积分作上下界估计；当 0≤α<1 时，∫x^(−α)dx=x^(1−α)/(1−α)。"]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S013","edges":[{"source_id":"S001","target_id":"S013","kind":"supports"},{"source_id":"S003","target_id":"S013","kind":"supports"},{"source_id":"S005","target_id":"S013","kind":"supports"},{"source_id":"S006","target_id":"S013","kind":"supports"},{"source_id":"S009","target_id":"S013","kind":"supports"},{"source_id":"S004","target_id":"S016","kind":"supports"},{"source_id":"S014","target_id":"S017","kind":"supports"},{"source_id":"S014","target_id":"S018","kind":"supports"},{"source_id":"S014","target_id":"S019","kind":"supports"},{"source_id":"S014","target_id":"S021","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S018","kind":"supports"},{"source_id":"S022","target_id":"S017","kind":"supports"},{"source_id":"S066","target_id":"S018","kind":"supports"},{"source_id":"S066","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S020","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S017","target_id":"S021","kind":"supports"},{"source_id":"S018","target_id":"S021","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S015","kind":"supports"},{"source_id":"S018","target_id":"S015","kind":"supports"},{"source_id":"S019","target_id":"S015","kind":"supports"},{"source_id":"S020","target_id":"S015","kind":"supports"},{"source_id":"S021","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S013","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S010","target_id":"S025","kind":"supports"},{"source_id":"S012","target_id":"S025","kind":"supports"},{"source_id":"S011","target_id":"S026","kind":"supports"},{"source_id":"S045","target_id":"S026","kind":"supports"},{"source_id":"S060","target_id":"S026","kind":"supports"},{"source_id":"S075","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S027","kind":"supports"},{"source_id":"S010","target_id":"S027","kind":"supports"},{"source_id":"S004","target_id":"S027","kind":"supports"},{"source_id":"S006","target_id":"S027","kind":"supports"},{"source_id":"S075","target_id":"S027","kind":"supports"},{"source_id":"S076","target_id":"S027","kind":"supports"},{"source_id":"S044","target_id":"S028","kind":"supports"},{"source_id":"S002","target_id":"S028","kind":"supports"},{"source_id":"S003","target_id":"S028","kind":"supports"},{"source_id":"S025","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S001","target_id":"S029","kind":"supports"},{"source_id":"S023","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S023","target_id":"S030","kind":"supports"},{"source_id":"S023","target_id":"S031","kind":"supports"},{"source_id":"S008","target_id":"S031","kind":"supports"},{"source_id":"S045","target_id":"S031","kind":"supports"},{"source_id":"S005","target_id":"S031","kind":"supports"},{"source_id":"S076","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S029","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S053","target_id":"S034","kind":"supports"},{"source_id":"S003","target_id":"S034","kind":"supports"},{"source_id":"S074","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S031","target_id":"S035","kind":"supports"},{"source_id":"S026","target_id":"S035","kind":"supports"},{"source_id":"S027","target_id":"S035","kind":"supports"},{"source_id":"S044","target_id":"S036","kind":"supports"},{"source_id":"S025","target_id":"S036","kind":"supports"},{"source_id":"S004","target_id":"S036","kind":"supports"},{"source_id":"S023","target_id":"S036","kind":"supports"},{"source_id":"S076","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S023","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S031","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S036","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S031","target_id":"S042","kind":"supports"},{"source_id":"S053","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S009","target_id":"S043","kind":"supports"},{"source_id":"S026","target_id":"S043","kind":"supports"},{"source_id":"S027","target_id":"S043","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S027","target_id":"S024","kind":"supports"},{"source_id":"S028","target_id":"S024","kind":"supports"},{"source_id":"S029","target_id":"S024","kind":"supports"},{"source_id":"S030","target_id":"S024","kind":"supports"},{"source_id":"S031","target_id":"S024","kind":"supports"},{"source_id":"S032","target_id":"S024","kind":"supports"},{"source_id":"S033","target_id":"S024","kind":"supports"},{"source_id":"S034","target_id":"S024","kind":"supports"},{"source_id":"S035","target_id":"S024","kind":"supports"},{"source_id":"S036","target_id":"S024","kind":"supports"},{"source_id":"S037","target_id":"S024","kind":"supports"},{"source_id":"S038","target_id":"S024","kind":"supports"},{"source_id":"S039","target_id":"S024","kind":"supports"},{"source_id":"S040","target_id":"S024","kind":"supports"},{"source_id":"S041","target_id":"S024","kind":"supports"},{"source_id":"S042","target_id":"S024","kind":"supports"},{"source_id":"S043","target_id":"S024","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S007","target_id":"S047","kind":"supports"},{"source_id":"S005","target_id":"S047","kind":"supports"},{"source_id":"S010","target_id":"S047","kind":"supports"},{"source_id":"S003","target_id":"S047","kind":"supports"},{"source_id":"S073","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S007","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S012","target_id":"S049","kind":"supports"},{"source_id":"S010","target_id":"S049","kind":"supports"},{"source_id":"S045","target_id":"S050","kind":"supports"},{"source_id":"S073","target_id":"S050","kind":"supports"},{"source_id":"S076","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S051","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S003","target_id":"S052","kind":"supports"},{"source_id":"S010","target_id":"S052","kind":"supports"},{"source_id":"S045","target_id":"S052","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S049","target_id":"S046","kind":"supports"},{"source_id":"S050","target_id":"S046","kind":"supports"},{"source_id":"S051","target_id":"S046","kind":"supports"},{"source_id":"S052","target_id":"S046","kind":"supports"},{"source_id":"S056","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S005","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S005","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S003","target_id":"S059","kind":"supports"},{"source_id":"S074","target_id":"S059","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S058","target_id":"S056","kind":"supports"},{"source_id":"S059","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S053","kind":"supports"},{"source_id":"S063","target_id":"S062","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S012","target_id":"S064","kind":"supports"},{"source_id":"S006","target_id":"S064","kind":"supports"},{"source_id":"S011","target_id":"S064","kind":"supports"},{"source_id":"S075","target_id":"S064","kind":"supports"},{"source_id":"S012","target_id":"S065","kind":"supports"},{"source_id":"S006","target_id":"S065","kind":"supports"},{"source_id":"S004","target_id":"S065","kind":"supports"},{"source_id":"S008","target_id":"S065","kind":"supports"},{"source_id":"S075","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S063","kind":"supports"},{"source_id":"S065","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S060","kind":"supports"},{"source_id":"S070","target_id":"S069","kind":"supports"},{"source_id":"S069","target_id":"S068","kind":"supports"},{"source_id":"S068","target_id":"S066","kind":"supports"},{"source_id":"S067","target_id":"S071","kind":"supports"},{"source_id":"S077","target_id":"S071","kind":"supports"},{"source_id":"S067","target_id":"S072","kind":"supports"},{"source_id":"S077","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S070","kind":"supports"},{"source_id":"S072","target_id":"S070","kind":"supports"},{"source_id":"S070","target_id":"S066","kind":"supports"}]}}
{"record_id":"2009.04014v2","paper":{"title":"Multi-Block Nonconvex Nonsmooth Proximal ADMM: Convergence and Rates under Kurdyka-Łojasiewicz Property","authors":["Maryam Yashtini"],"arxiv_id":"2009.04014","version":"v2","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-021-01919-7","abstract_url":"https://arxiv.org/abs/2009.04014v2","pdf_url":"https://arxiv.org/pdf/2009.04014v2"},"context":{"raw_tex":"\\begin{eqnarray}\\label{OP}\n\\begin{array}{ccc}\n&\\displaystyle{\\min_{x_1,x_2,\\dots, x_p,y}}&\n\\sum_{i=1}^p f_i(x_i)+h(y)\\\\[.1in]\n&\\textrm{s.t.}& \\sum_{i=1}^pA_ix_i+By+b=0, \n\\end{array}\n\\end{eqnarray}\n\nwhere $p\\ge 1$, $x_i\\in\\mathbb R^{n_i}$ are variables with their coefficient matrix \n$A_i\\in\\mathbb R^{m\\times n_i}$, $i=1,2,\\dots,p$, $y\\in\\mathbb R^q$ is the last variable \nwith its coefficient matrix $B\\in\\mathbb R^{m\\times q}$, and $b\\in\\mathbb R^m$. \nThe model remains as general without $y$\nand $By$; but we keep $y$ and $By$ to simplify the notation.\nThe functions $f_i:\\mathbb  R^{n_i}\\to\\bar{\\mathbb R}$ for $i=1,2,\\dots,p$ are proper \nand lower-semicontinuous,\nand $h:\\mathbb R^{q}\\to \\bar{\\mathbb R}$ is proper smooth function.\n All functions can be nonconvex.\n We denote ${\\bf x}=[x_1,\\dots,x_p]\\in\\mathbb R^n$ where $n=\\sum_{i=1}^p n_i$,\n${\\bf A}:=[A_1\\dots A_p]\\in\\mathbb R^{m\\times n}$, and ${\\bf Ax}=\\sum_{i=1}^p A_ix_i\\in\\mathbb R^m$.\nWith these notations, the problem (\\ref{OP}) can be written  \n\n\n\\begin{eqnarray}\\label{OPE}\n\\displaystyle{\\min_{{\\bf x},y}}\n\\sum_{i=1}^p f_i(x_i)+h(y)\n\\quad\\quad\\textrm{s.t.} \\;\\;\\;{\\bf Ax}+By+b=0.\n\\end{eqnarray}\n\n\\begin{eqnarray}\\label{augL}\n&\\mathcal L^{\\alpha}: \\mathbb R^n\\times\\mathbb R^q\\times \\mathbb R^m\\to\\mathbb R\\nonumber&\\\\[.1in]\n&\\mathcal L^{\\alpha} ({\\bf x},y,z)=\\sum_{i=1}^p f_i(x_i)+h(y)\n+\\langle z, {\\bf Ax}+By+b\\rangle \n+\\frac{\\alpha}{2}\\big\\|{\\bf Ax}+By+b\\big\\|^2,&\n\\end{eqnarray}\n\nwhere $\\alpha>0$ and the vector $z\\in\\mathbb R^m$ is the Lagrange multiplier associated with the constraint\n${\\bf Ax}+By+b=0$. In each iteration, the augmented Lagrangian is minimized over the primal variables \n$x_i, i=1,\\dots,p$ and $y$ separately and one after each other, followed by a dual update for $z$.\n\n\nHere $\\beta\\in (0,2)$ is the stepsize or damping parameter, $x_{<i}:=[x_1;\\dots;x_{i-1}]\\in\\mathbb R^{n_1+\\dots+n_{i-1}}$\nand $x_{>i}:=[x_{i+1};\\dots;x_{p}]\\in\\mathbb R^{n_{i+1}+\\dots+n_{p}}$\n(clearly, $x_{<0}$ and $x_{>p}$ are null variables, which may be used for notational ease).\n\n\n In this paper, we consider a variant/modification of ADMM that is called Proximal ADMM (PADMM).\nTo state the algorithm let $\\{Q_i\\}_{i=1,\\dots,p}\\subseteq \\mathbb R^{n_i\\times n_i}$\nand $P\\in \\mathbb R^{q\\times q}$ be symmetric positive semidefinite matrices and \n$\\|x\\|^2_M=x^TMx$, then the nonconvex possibly nonsmooth PADMM algorithm for solving \n(\\ref{OPE}) is given by:\n\n\n\\begin{eqnarray}\\label{padmm}\n\\left\\{\n\\begin{array}{lll}\nx_i^{k+1}&=&\\displaystyle{{\\arg\\min}_{x_i}} \\;\n\\mathcal L^{\\alpha}(x^{k+1}_{<i}, x_i, x^k_{>i},y^k,z^k)+\\frac{1}{2}\\|x_i-x_i^k\\|^2_{Q_i};\n\\quad i=1,\\dots, p \\\\[.1in]\ny^{k+1}&=&\\displaystyle{\\arg\\min}_{y}\\; \\mathcal L^{\\alpha}({\\bf x}^{k+1},y,z^k)+\\frac 12 \\|y-y^k\\|^2_P; \\\\[.1in]\nz^{k+1}&=&z^k+\\alpha\\beta ({\\bf Ax}^{k+1}+By^{k+1}+b).\n\\end{array}\n\\right.\n\\end{eqnarray}\n\nThroughout this paper, we denote $\\mathbb R$ as the real number set\nwhile $\\mathbb Z$ as the set of integers. \nThe set $\\bar{\\mathbb R}:=\\mathbb R\\cup\\{+\\infty\\}$ is the extended real numbers,\n$\\mathbb R_+$ is the positive real number set,\nand $\\mathbb Z_+$ is the set of positive integers.\nGiven the matrix $X$, ${\\rm Im}(X)$ denotes its image. \nWe denote by $I_n$ the $n\\times n$ identity matrix. The minimum and maximum eigenvalues of the matrix \n$X\\in\\mathbb R^{n\\times n}$ are denoted respectively by $\\lambda_{\\min}(X)$,\nand $\\lambda_{\\max}(X)$,\nand $\\lambda_{++}^X$ refers to the smallest strictly positive eigenvalue of $X$.\n%\nThe Euclidean scalar product of $\\mathbb R^n$ and its corresponding \nnorms are, respectively, denoted by $\\langle \\cdot, \\cdot\\rangle$\nand $\\|\\cdot\\|=\\sqrt{\\langle \\cdot,\\cdot\\rangle}$. For a vector $x\\in\\mathbb R^n$,\n\n\n If  $n_1,\\dots, n_p\\in\\mathbb Z_+$ and $p\\in\\mathbb Z_+$,\nthen for any $v:=(v_1,\\dots,v_p)\\in\\mathbb R^{n_1}\\times\\mathbb R^{n_2}\\times\\dots\\times\\mathbb R^{n_p}$ and $v':=(v'_1,\\dots,v'_p)\\in \\mathbb R^{n_1}\\times\\mathbb R^{n_2}\\times\\dots\\times\\mathbb R^{n_p}$ the Cartesian product and its norm are defined by\n\\[\n\\ll v,v'\\gg =\\sum_{i=1}^{p}\\langle v_i,v_i'\\rangle\\quad\\quad\n%\n\\frac{1}{\\sqrt{p}}  \\sum_{i=1}^{p} \\| v_i\\| \\le |||v||| = \\sqrt{\\sum_{i=1}^{p} \\| v_i\\|^2}\\le  \\sum_{i=1}^{p} \\| v_i\\|.\n\\]\n\n\n Let $\\Phi:\\mathbb R^d\\to \\bar{\\mathbb R}$ be a proper and lower semicontinuous function. \n The domain  of $\\Phi$, denoted ${\\rm dom}\\; \\Phi$, is defined by \n${\\rm dom}\\; \\Phi:=\\{x\\in\\mathbb R^d:\\; \\Phi(x)<+\\infty\\}.\n$\n%\n%The graph of the $\\Phi$, denoted  ${\\rm Graph}\\; \\Phi$, defined by\n%$\n% {\\rm Graph}\\; \\Phi:=\\{(x,y)\\in\\mathbb R^d \\times\\mathbb R: y=\\Phi(x)\\}.\n%$\nFor any $x\\in {\\rm dom}\\; \\Phi$, the {\\it Fr\\'echet (viscosity) subdifferential} of $\\Phi$ at $x$,\ndenoted $\\hat\\partial\\Phi(x)$, is defined by\n\\[\n\\hat\\partial\\Phi(x)=\n\\Big\\{\ns\\in\\mathbb R^d: \\;\\;\\lim_{y\\neq x}\\inf_{y\\to x} \\frac{\\Phi(y)-\\Phi(x)-\\langle s, y-x\\rangle}{\\|y-x\\|}\\ge 0\n\\Big\\}.\n\\]\nFor $x\\notin {\\rm dom}\\;\\Phi$, then $\\hat\\partial\\Phi (x)=\\emptyset$.\nThe {\\it limiting (Mordukhovich) subdifferential}, or simply the subdifferential for short,\n of $\\Phi$ at $x\\in {\\rm dom}\\;\\Phi$, denoted $\\partial\\Phi(x)$, is defined by\n %\n\\begin{eqnarray*}\n\\partial\\Phi(x):=\\{s\\in\\mathbb R^d: \\exists x^k\\to x,\\;\\Phi(x^k)\\to\\Phi(x)\\;\\;\n{\\rm and} \\;\\; s^k\\in\\hat\\Phi(x^k)\\to s\\;\\;{\\rm as}\\; k\\to+\\infty\\}.\n\\end{eqnarray*}\n\n\nLet $(x^k,s^k)\\in {\\rm Graph}\\; \\partial\\Phi:=\\{(x,s)\\in\\mathbb R^d\\times\\mathbb R^d: s\\in\\partial \\Phi(x)\\}.\n$\nLet $(x^k,s^k)\\to (x^*,s^*)$ as $k\\to\\infty$. Since $s^k\\in\\partial \\Phi(x^k)$, \nand $\\Phi(x^k)\\to \\Phi(x^*)$ as $k\\to\\infty$ we have  $(x^*,s^*)\\in {\\rm Graph}\\; \\partial\\Phi$.\nThe well-known Fermat's rule ``$x\\in\\mathbb R^d$ is a local minimizer of $\\Phi$, then $\\partial \\Phi(x)\\ni 0$''\nremains unchanged. If $x\\in\\mathbb R^d$ such that $\\partial \\Phi(x)\\ni 0$ the point $x$ is called \na critical point. We denote by ${\\rm crit}\\; \\Phi$ the set of {\\it critical points} of $\\Phi$,\nthat is \n$\n{\\rm crit}\\; \\Phi=\\{x\\in\\mathbb R^d:0\\in\\partial\\Phi(x)\\}.\n$\n\n\n Let $\\Omega$ be a subset of $\\mathbb R^d$  and $x$ is any point in $\\mathbb R^d$. The distance from $x$\nto $\\Omega$, denoted ${\\rm dist}(x,\\Omega)$, is defined by \n${\\rm dist}(x,\\Omega)=\\inf\\{\\|x-z\\|:\\;z\\in\\Omega\\}$, and the set of points in $\\Omega$ that \nachieve this infimum (the projection of $x$ onto $\\Omega$) is defined by ${\\rm Proj}_{\\Omega}(x)$.\nThe set ${\\rm Proj}_{\\Omega}(x)$ becomes a singleton if $\\Omega$ is a closed convex \nset. If $\\Omega=\\emptyset$, then ${\\rm dist}(x,\\Omega)=+\\infty$ for all $x\\in\\mathbb R^d$. \nFor any real-valued function $\\Phi$ on $\\mathbb R^d$ we have\n\\[\n{\\rm dist}(0,\\partial\\Phi(x))=\\inf\\{\\|s^*\\|: \\; s^*\\in\\partial \\Phi(x)\\}.\n\\]  \n%\nLet $F:\\mathbb R^n\\times\\mathbb R^m\\to \\bar{\\mathbb R}$ be a lower semicontinuous function. The subdifferentiation of $F$ at the point $(\\hat x,\\hat y)$ is defined by \n$\n\\partial F(\\hat x, \\hat y)=\\big(\\partial_x F(\\hat x,\\hat y), \\partial_y F(\\hat x,\\hat y)\\big),\n$\nwhere $\\partial_x F$ and $\\partial_y F$ are respectively the differential of the function \n$F(\\cdot,y)$ when $y\\in\\mathbb R^m$ is fixed, and $F(x,\\cdot)$ when $x$ is fixed.\n\n\nLet $\\Phi:\\mathbb R^d\\to \\bar{\\mathbb R}$  be a proper lower semicontinuous function. \nFor $-\\infty<\\eta_1<\\eta_2\\le +\\infty$, we define \n$[\\eta_1<\\Phi<\\eta_2] =\\{x\\in\\mathbb R^d: \\; \\eta_1<\\Phi(x)<\\eta_2\\}.$\n%\nLet $\\eta\\in (0,+\\infty]$. We denote by $\\Psi_{\\eta}$ the set of all continuous concave functions \n $\\psi:[0,\\eta]\\to [0,+\\infty)$ that $\\psi(0)=0$ and\n$\\psi$ is continuously differentiable on $(0,\\eta)$ with $\\psi'(s)>0$ over $(0,\\eta)$.\n\n\n\\begin{definition}\\label{KLprop}\nLet $\\Phi:\\mathbb R^d\\to \\bar{\\mathbb R}$  be a proper  lower semicontinuous function.\nWe say that $\\Phi$ has the Kurdyka-{\\L}ojasiewicz (K{\\L}) property at \n$x^*\\in {\\rm dom} \\;\\partial \\Phi$ if there exists a neighborhood $U$ of $x^*$, \n$\\eta\\in(0,\\infty)$ and a continuous concave function $\\psi\\in\\Psi_{\\eta}$ such that\n%\n\\begin{eqnarray}\\label{KLine}\n\\psi'\\big(\\Phi(x)-\\Phi(x^*)\\big) {\\rm dist}\\big(0,\\partial \\Phi(x)\\big)\\ge 1.\n\\end{eqnarray}\n If $\\Phi$ satisfies the  property at each point of $ {\\rm dom} \\;\\partial \\Phi$, then $\\Phi$ is a \n{K\\L} function. \n\\end{definition}\n\n\\begin{definition}\\label{KLset}\nLet $\\Phi:\\mathbb R^d\\to \\bar{\\mathbb R}$  be a proper  lower semicontinuous function that\n takes constant value on $\\Omega$ and satisfies the {K\\L} property at each point of $\\Omega$.\nWe say $\\Phi$ satisfies a K{\\L} property on $\\Omega$ if there exists\n$\\epsilon>0$, $\\eta>0$, and $\\psi\\in\\Psi_{\\eta}$ such that \nfor every $x^*\\in\\Omega$ and every element $x$ belongs to the intersection\n$ \\{x\\in\\mathbb R^d: {\\rm dist} (x,\\Omega)<\\epsilon\\} \\cap [\\Phi(x^*) <\\Phi(x) <   \\Phi(x^*)+\\eta],$\n (\\ref{KLine}) holds. \n \\end{definition}\n\n\\begin{itemize}\\label{assumptionA}\n\\item[{A1.}] $f_i, i=1,\\dots, p $ are lower-semicontinuous and coercive;\n\\item[{A2.}] $h$ is bounded from below, Lipschitz differentiable with constant $L_h$;\n\\item[{A3.}] ${\\rm Im}({\\bf A})\\subseteq {\\rm Im}(B)$ and $b\\in {\\rm Im}(B)$;\n%\\item[{A4.}] $\\exists\\eta>0$ such that $B\\tr B\\succeq \\eta I_q$ and $A_i\\tr A_i\\succeq \\eta I_{n_i}, i=1,\\dots,p$;\n\\item[{A4.}] For $i=1,\\dots, p$ one of the followings hold: \n\\begin{itemize}\n\\item[(i)] $\\exists q_i>0$ such that $Q_i\\succeq q_i I_{n_i}$; or \n\\item[(ii)] $\\exists \\epsilon_i > 0$ such that \n$f_i(v)-f_i(\\tilde v)- \\langle \\tilde s_i , v-\\tilde v \\rangle \\ge -\\frac{\\epsilon_i}{2} \\| v-\\tilde v\\|^2$ \nfor  $v,\\tilde v\\in\\mathbb R^{n_i}$ and $\\tilde s_i\\in\\partial f_i(\\tilde v)$. \nThis is equivalent to the weak convexity of the function $f_i$, $i=1,\\dots,p$.\n \\end{itemize}\n \\end{itemize}\n\n\n\\begin{eqnarray}\\label{quantities}\n\\left\\{\n\\begin{tabular}{ll}\n$\\rho(\\beta):=1-|1-\\beta| $,&  \\\\\n$c_1:=\\displaystyle{\\frac{1}{\\alpha\\rho(\\beta)\\lambda_{++}^{B\\tr B}}}$,&\n$c_2:= \\displaystyle{  \\frac{2\\beta}{\\alpha\\rho(\\beta)^2\\lambda_{++}^{B\\tr B}}}$, \\\\\n$c_3:=c_2\\|P\\|^2$, & $c_4:=   c_2 (\\|P\\|+L_h)^2$, \\\\\n$c_5:=\\beta^{-1}|1-\\beta| c_1$, \n&$\\displaystyle{c_6:=\\frac{|1-\\beta| }{2\\alpha\\beta^2\\lambda_{++}^{B\\tr B}}}$, \\\\\n$D=2P+\\alpha B\\tr B-L_h I_m$, &$\\bar D=D-2\\epsilon_0(c_3+c_4)I$, \\\\\n\\end{tabular}\n \\right.\n\\end{eqnarray}\n\nwhere $\\epsilon_0>1$, $\\lambda_{++}^{B\\tr B}$ is  the smallest \nstrictly-positive eigenvalue of $B\\tr B$, and $I_m$ denotes a $m\\times m$ identity matrix.\n\n\nWe define a modified version of the augmented Lagrangian function (\\ref{augL}).\nWe consider $\\bar{\\mathcal L}:\\mathbb R^n\\times \\mathbb R^q\\times \\mathbb R^m\\times \\mathbb R^q\\times \\mathbb R^m\\to (-\\infty,+\\infty] $ defined by \n%\n\\begin{eqnarray}\\label{Lbar}\n\\bar{\\mathcal L}(x,y,z,y',z')=\\mathcal L^{\\alpha}(x,y,z)+\n\\epsilon_0 c_5 \\|B\\tr(z-z')\\|^2+\n\\epsilon_0  c_3 \\|y-y'\\|^2,\n\\end{eqnarray}\n\nwhere $\\epsilon_0>1$, and $c_3$ and $c_5$ are given in (\\ref{quantities}). \nEvaluating $\\bar{\\mathcal L}$ at the point $({\\bf x}^{k}, y^{k},z^{k},y^{k-1},z^{k-1})$ where \n$\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 1}$ is the sequence generated by the PADMM algorithm (\\ref{padmm})\ngives\n\\begin{eqnarray}\\label{Lbark}\n\\bar{\\mathcal L}_k:=\n\\bar{\\mathcal L}({\\bf x}^{k}, y^{k},z^{k},y^{k-1},z^{k-1})=\\mathcal L^{\\alpha}({\\bf x}^k,y^k,z^k)+\n\\epsilon_0 c_5 \\|B\\tr\\Delta z^k\\|^2+\n\\epsilon_0 c_3 \\|\\Delta y^k\\|^2.\n\\end{eqnarray}","units":[{"start":2731,"end":2927,"start_line":75,"end_line":81,"raw_tex":"\\begin{eqnarray}\\label{OP}\n\\begin{array}{ccc}\n&\\displaystyle{\\min_{x_1,x_2,\\dots, x_p,y}}&\n\\sum_{i=1}^p f_i(x_i)+h(y)\\\\[.1in]\n&\\textrm{s.t.}& \\sum_{i=1}^pA_ix_i+By+b=0, \n\\end{array}\n\\end{eqnarray}","sha256_utf8":"216ba35e902c291c5a9addfe1a3e60973206b54aae15403a0167f7e38a722a00"},{"start":2932,"end":3746,"start_line":84,"end_line":95,"raw_tex":"where $p\\ge 1$, $x_i\\in\\mathbb R^{n_i}$ are variables with their coefficient matrix \n$A_i\\in\\mathbb R^{m\\times n_i}$, $i=1,2,\\dots,p$, $y\\in\\mathbb R^q$ is the last variable \nwith its coefficient matrix $B\\in\\mathbb R^{m\\times q}$, and $b\\in\\mathbb R^m$. \nThe model remains as general without $y$\nand $By$; but we keep $y$ and $By$ to simplify the notation.\nThe functions $f_i:\\mathbb  R^{n_i}\\to\\bar{\\mathbb R}$ for $i=1,2,\\dots,p$ are proper \nand lower-semicontinuous,\nand $h:\\mathbb R^{q}\\to \\bar{\\mathbb R}$ is proper smooth function.\n All functions can be nonconvex.\n We denote ${\\bf x}=[x_1,\\dots,x_p]\\in\\mathbb R^n$ where $n=\\sum_{i=1}^p n_i$,\n${\\bf A}:=[A_1\\dots A_p]\\in\\mathbb R^{m\\times n}$, and ${\\bf Ax}=\\sum_{i=1}^p A_ix_i\\in\\mathbb R^m$.\nWith these notations, the problem (\\ref{OP}) can be written  \n","sha256_utf8":"063574d6521b7e21fd614215b15fc2f01b7573649f4185801ce522aa22c0c9bf"},{"start":3748,"end":3896,"start_line":97,"end_line":101,"raw_tex":"\\begin{eqnarray}\\label{OPE}\n\\displaystyle{\\min_{{\\bf x},y}}\n\\sum_{i=1}^p f_i(x_i)+h(y)\n\\quad\\quad\\textrm{s.t.} \\;\\;\\;{\\bf Ax}+By+b=0.\n\\end{eqnarray}","sha256_utf8":"fb06fe9ea39ade70cb3c8bc92f40509524e85625fa7613d04cab6f72d6d27177"},{"start":4947,"end":5232,"start_line":114,"end_line":119,"raw_tex":"\\begin{eqnarray}\\label{augL}\n&\\mathcal L^{\\alpha}: \\mathbb R^n\\times\\mathbb R^q\\times \\mathbb R^m\\to\\mathbb R\\nonumber&\\\\[.1in]\n&\\mathcal L^{\\alpha} ({\\bf x},y,z)=\\sum_{i=1}^p f_i(x_i)+h(y)\n+\\langle z, {\\bf Ax}+By+b\\rangle \n+\\frac{\\alpha}{2}\\big\\|{\\bf Ax}+By+b\\big\\|^2,&\n\\end{eqnarray}","sha256_utf8":"d41af35eef0268fb73e61a4670a3cc804f7b362991d0621303c88498029bc785"},{"start":5235,"end":5545,"start_line":121,"end_line":123,"raw_tex":"where $\\alpha>0$ and the vector $z\\in\\mathbb R^m$ is the Lagrange multiplier associated with the constraint\n${\\bf Ax}+By+b=0$. In each iteration, the augmented Lagrangian is minimized over the primal variables \n$x_i, i=1,\\dots,p$ and $y$ separately and one after each other, followed by a dual update for $z$.\n","sha256_utf8":"d2a61527cc0644cf9a39b0e9855c200684b17eaf7bb7ce826b54572c87eec62a"},{"start":6011,"end":6295,"start_line":137,"end_line":139,"raw_tex":"Here $\\beta\\in (0,2)$ is the stepsize or damping parameter, $x_{<i}:=[x_1;\\dots;x_{i-1}]\\in\\mathbb R^{n_1+\\dots+n_{i-1}}$\nand $x_{>i}:=[x_{i+1};\\dots;x_{p}]\\in\\mathbb R^{n_{i+1}+\\dots+n_{p}}$\n(clearly, $x_{<0}$ and $x_{>p}$ are null variables, which may be used for notational ease).\n","sha256_utf8":"a636765fbc8ac1fe05b4d22ca985a711860aba50c69d293ed576c22cd6de7416"},{"start":11499,"end":11875,"start_line":222,"end_line":226,"raw_tex":" In this paper, we consider a variant/modification of ADMM that is called Proximal ADMM (PADMM).\nTo state the algorithm let $\\{Q_i\\}_{i=1,\\dots,p}\\subseteq \\mathbb R^{n_i\\times n_i}$\nand $P\\in \\mathbb R^{q\\times q}$ be symmetric positive semidefinite matrices and \n$\\|x\\|^2_M=x^TMx$, then the nonconvex possibly nonsmooth PADMM algorithm for solving \n(\\ref{OPE}) is given by:\n","sha256_utf8":"86c5b6dd7992df2564bcbc468498ecdd23084177ac274e2c94f1d5fcf7dd1b79"},{"start":11924,"end":12350,"start_line":229,"end_line":239,"raw_tex":"\\begin{eqnarray}\\label{padmm}\n\\left\\{\n\\begin{array}{lll}\nx_i^{k+1}&=&\\displaystyle{{\\arg\\min}_{x_i}} \\;\n\\mathcal L^{\\alpha}(x^{k+1}_{<i}, x_i, x^k_{>i},y^k,z^k)+\\frac{1}{2}\\|x_i-x_i^k\\|^2_{Q_i};\n\\quad i=1,\\dots, p \\\\[.1in]\ny^{k+1}&=&\\displaystyle{\\arg\\min}_{y}\\; \\mathcal L^{\\alpha}({\\bf x}^{k+1},y,z^k)+\\frac 12 \\|y-y^k\\|^2_P; \\\\[.1in]\nz^{k+1}&=&z^k+\\alpha\\beta ({\\bf Ax}^{k+1}+By^{k+1}+b).\n\\end{array}\n\\right.\n\\end{eqnarray}","sha256_utf8":"dd4c8ebc1416671f03e2e38d7177fe901661c311073efae9e8daca30827c0d5a"},{"start":15791,"end":16644,"start_line":287,"end_line":300,"raw_tex":"Throughout this paper, we denote $\\mathbb R$ as the real number set\nwhile $\\mathbb Z$ as the set of integers. \nThe set $\\bar{\\mathbb R}:=\\mathbb R\\cup\\{+\\infty\\}$ is the extended real numbers,\n$\\mathbb R_+$ is the positive real number set,\nand $\\mathbb Z_+$ is the set of positive integers.\nGiven the matrix $X$, ${\\rm Im}(X)$ denotes its image. \nWe denote by $I_n$ the $n\\times n$ identity matrix. The minimum and maximum eigenvalues of the matrix \n$X\\in\\mathbb R^{n\\times n}$ are denoted respectively by $\\lambda_{\\min}(X)$,\nand $\\lambda_{\\max}(X)$,\nand $\\lambda_{++}^X$ refers to the smallest strictly positive eigenvalue of $X$.\n%\nThe Euclidean scalar product of $\\mathbb R^n$ and its corresponding \nnorms are, respectively, denoted by $\\langle \\cdot, \\cdot\\rangle$\nand $\\|\\cdot\\|=\\sqrt{\\langle \\cdot,\\cdot\\rangle}$. For a vector $x\\in\\mathbb R^n$,\n","sha256_utf8":"888a2a2366b49bab6bccaa71f7d3a16331efce5a4c7bfb69ea5757a53ec46a1a"},{"start":17031,"end":17534,"start_line":314,"end_line":320,"raw_tex":" If  $n_1,\\dots, n_p\\in\\mathbb Z_+$ and $p\\in\\mathbb Z_+$,\nthen for any $v:=(v_1,\\dots,v_p)\\in\\mathbb R^{n_1}\\times\\mathbb R^{n_2}\\times\\dots\\times\\mathbb R^{n_p}$ and $v':=(v'_1,\\dots,v'_p)\\in \\mathbb R^{n_1}\\times\\mathbb R^{n_2}\\times\\dots\\times\\mathbb R^{n_p}$ the Cartesian product and its norm are defined by\n\\[\n\\ll v,v'\\gg =\\sum_{i=1}^{p}\\langle v_i,v_i'\\rangle\\quad\\quad\n%\n\\frac{1}{\\sqrt{p}}  \\sum_{i=1}^{p} \\| v_i\\| \\le |||v||| = \\sqrt{\\sum_{i=1}^{p} \\| v_i\\|^2}\\le  \\sum_{i=1}^{p} \\| v_i\\|.\n\\]\n","sha256_utf8":"d72a1037b64e8e140a7ab01e413c36645a6cec6c689fe304fd748ac8775b3d9c"},{"start":17788,"end":18877,"start_line":329,"end_line":353,"raw_tex":" Let $\\Phi:\\mathbb R^d\\to \\bar{\\mathbb R}$ be a proper and lower semicontinuous function. \n The domain  of $\\Phi$, denoted ${\\rm dom}\\; \\Phi$, is defined by \n${\\rm dom}\\; \\Phi:=\\{x\\in\\mathbb R^d:\\; \\Phi(x)<+\\infty\\}.\n$\n%\n%The graph of the $\\Phi$, denoted  ${\\rm Graph}\\; \\Phi$, defined by\n%$\n% {\\rm Graph}\\; \\Phi:=\\{(x,y)\\in\\mathbb R^d \\times\\mathbb R: y=\\Phi(x)\\}.\n%$\nFor any $x\\in {\\rm dom}\\; \\Phi$, the {\\it Fr\\'echet (viscosity) subdifferential} of $\\Phi$ at $x$,\ndenoted $\\hat\\partial\\Phi(x)$, is defined by\n\\[\n\\hat\\partial\\Phi(x)=\n\\Big\\{\ns\\in\\mathbb R^d: \\;\\;\\lim_{y\\neq x}\\inf_{y\\to x} \\frac{\\Phi(y)-\\Phi(x)-\\langle s, y-x\\rangle}{\\|y-x\\|}\\ge 0\n\\Big\\}.\n\\]\nFor $x\\notin {\\rm dom}\\;\\Phi$, then $\\hat\\partial\\Phi (x)=\\emptyset$.\nThe {\\it limiting (Mordukhovich) subdifferential}, or simply the subdifferential for short,\n of $\\Phi$ at $x\\in {\\rm dom}\\;\\Phi$, denoted $\\partial\\Phi(x)$, is defined by\n %\n\\begin{eqnarray*}\n\\partial\\Phi(x):=\\{s\\in\\mathbb R^d: \\exists x^k\\to x,\\;\\Phi(x^k)\\to\\Phi(x)\\;\\;\n{\\rm and} \\;\\; s^k\\in\\hat\\Phi(x^k)\\to s\\;\\;{\\rm as}\\; k\\to+\\infty\\}.\n\\end{eqnarray*}\n","sha256_utf8":"3f23a983a3d884692be6621a02ece2aefb5c602e9c75fc908a9500d297951dae"},{"start":19101,"end":19769,"start_line":361,"end_line":371,"raw_tex":"Let $(x^k,s^k)\\in {\\rm Graph}\\; \\partial\\Phi:=\\{(x,s)\\in\\mathbb R^d\\times\\mathbb R^d: s\\in\\partial \\Phi(x)\\}.\n$\nLet $(x^k,s^k)\\to (x^*,s^*)$ as $k\\to\\infty$. Since $s^k\\in\\partial \\Phi(x^k)$, \nand $\\Phi(x^k)\\to \\Phi(x^*)$ as $k\\to\\infty$ we have  $(x^*,s^*)\\in {\\rm Graph}\\; \\partial\\Phi$.\nThe well-known Fermat's rule ``$x\\in\\mathbb R^d$ is a local minimizer of $\\Phi$, then $\\partial \\Phi(x)\\ni 0$''\nremains unchanged. If $x\\in\\mathbb R^d$ such that $\\partial \\Phi(x)\\ni 0$ the point $x$ is called \na critical point. We denote by ${\\rm crit}\\; \\Phi$ the set of {\\it critical points} of $\\Phi$,\nthat is \n$\n{\\rm crit}\\; \\Phi=\\{x\\in\\mathbb R^d:0\\in\\partial\\Phi(x)\\}.\n$\n","sha256_utf8":"f509d5d522b34aeaae1b5fe6b6a196c3f9645222e2366f091815a295879e5a9c"},{"start":19955,"end":21074,"start_line":378,"end_line":394,"raw_tex":" Let $\\Omega$ be a subset of $\\mathbb R^d$  and $x$ is any point in $\\mathbb R^d$. The distance from $x$\nto $\\Omega$, denoted ${\\rm dist}(x,\\Omega)$, is defined by \n${\\rm dist}(x,\\Omega)=\\inf\\{\\|x-z\\|:\\;z\\in\\Omega\\}$, and the set of points in $\\Omega$ that \nachieve this infimum (the projection of $x$ onto $\\Omega$) is defined by ${\\rm Proj}_{\\Omega}(x)$.\nThe set ${\\rm Proj}_{\\Omega}(x)$ becomes a singleton if $\\Omega$ is a closed convex \nset. If $\\Omega=\\emptyset$, then ${\\rm dist}(x,\\Omega)=+\\infty$ for all $x\\in\\mathbb R^d$. \nFor any real-valued function $\\Phi$ on $\\mathbb R^d$ we have\n\\[\n{\\rm dist}(0,\\partial\\Phi(x))=\\inf\\{\\|s^*\\|: \\; s^*\\in\\partial \\Phi(x)\\}.\n\\]  \n%\nLet $F:\\mathbb R^n\\times\\mathbb R^m\\to \\bar{\\mathbb R}$ be a lower semicontinuous function. The subdifferentiation of $F$ at the point $(\\hat x,\\hat y)$ is defined by \n$\n\\partial F(\\hat x, \\hat y)=\\big(\\partial_x F(\\hat x,\\hat y), \\partial_y F(\\hat x,\\hat y)\\big),\n$\nwhere $\\partial_x F$ and $\\partial_y F$ are respectively the differential of the function \n$F(\\cdot,y)$ when $y\\in\\mathbb R^m$ is fixed, and $F(x,\\cdot)$ when $x$ is fixed.\n","sha256_utf8":"560ee069cfdd0166760de9735d0a17f6700113967cfec6b04aaaf001dec7d905"},{"start":22414,"end":22864,"start_line":435,"end_line":441,"raw_tex":"Let $\\Phi:\\mathbb R^d\\to \\bar{\\mathbb R}$  be a proper lower semicontinuous function. \nFor $-\\infty<\\eta_1<\\eta_2\\le +\\infty$, we define \n$[\\eta_1<\\Phi<\\eta_2] =\\{x\\in\\mathbb R^d: \\; \\eta_1<\\Phi(x)<\\eta_2\\}.$\n%\nLet $\\eta\\in (0,+\\infty]$. We denote by $\\Psi_{\\eta}$ the set of all continuous concave functions \n $\\psi:[0,\\eta]\\to [0,+\\infty)$ that $\\psi(0)=0$ and\n$\\psi$ is continuously differentiable on $(0,\\eta)$ with $\\psi'(s)>0$ over $(0,\\eta)$.\n","sha256_utf8":"4e77acbd56be9321f040a533e31cc7bbbc2be5fbd5a3f52b4010114e068b0ec7"},{"start":22867,"end":23482,"start_line":444,"end_line":455,"raw_tex":"\\begin{definition}\\label{KLprop}\nLet $\\Phi:\\mathbb R^d\\to \\bar{\\mathbb R}$  be a proper  lower semicontinuous function.\nWe say that $\\Phi$ has the Kurdyka-{\\L}ojasiewicz (K{\\L}) property at \n$x^*\\in {\\rm dom} \\;\\partial \\Phi$ if there exists a neighborhood $U$ of $x^*$, \n$\\eta\\in(0,\\infty)$ and a continuous concave function $\\psi\\in\\Psi_{\\eta}$ such that\n%\n\\begin{eqnarray}\\label{KLine}\n\\psi'\\big(\\Phi(x)-\\Phi(x^*)\\big) {\\rm dist}\\big(0,\\partial \\Phi(x)\\big)\\ge 1.\n\\end{eqnarray}\n If $\\Phi$ satisfies the  property at each point of $ {\\rm dom} \\;\\partial \\Phi$, then $\\Phi$ is a \n{K\\L} function. \n\\end{definition}","sha256_utf8":"e76027cffadd9c795ba47740456ab943173fa9d03fb7ad3473d9903300fe9287"},{"start":23484,"end":24046,"start_line":457,"end_line":465,"raw_tex":"\\begin{definition}\\label{KLset}\nLet $\\Phi:\\mathbb R^d\\to \\bar{\\mathbb R}$  be a proper  lower semicontinuous function that\n takes constant value on $\\Omega$ and satisfies the {K\\L} property at each point of $\\Omega$.\nWe say $\\Phi$ satisfies a K{\\L} property on $\\Omega$ if there exists\n$\\epsilon>0$, $\\eta>0$, and $\\psi\\in\\Psi_{\\eta}$ such that \nfor every $x^*\\in\\Omega$ and every element $x$ belongs to the intersection\n$ \\{x\\in\\mathbb R^d: {\\rm dist} (x,\\Omega)<\\epsilon\\} \\cap [\\Phi(x^*) <\\Phi(x) <   \\Phi(x^*)+\\eta],$\n (\\ref{KLine}) holds. \n \\end{definition}","sha256_utf8":"dc6a892339d4a25a2a1185b35a52fb6e78c8d53f9d9678db6b2ad4d8ebee4026"},{"start":26470,"end":27348,"start_line":511,"end_line":524,"raw_tex":"\\begin{itemize}\\label{assumptionA}\n\\item[{A1.}] $f_i, i=1,\\dots, p $ are lower-semicontinuous and coercive;\n\\item[{A2.}] $h$ is bounded from below, Lipschitz differentiable with constant $L_h$;\n\\item[{A3.}] ${\\rm Im}({\\bf A})\\subseteq {\\rm Im}(B)$ and $b\\in {\\rm Im}(B)$;\n%\\item[{A4.}] $\\exists\\eta>0$ such that $B\\tr B\\succeq \\eta I_q$ and $A_i\\tr A_i\\succeq \\eta I_{n_i}, i=1,\\dots,p$;\n\\item[{A4.}] For $i=1,\\dots, p$ one of the followings hold: \n\\begin{itemize}\n\\item[(i)] $\\exists q_i>0$ such that $Q_i\\succeq q_i I_{n_i}$; or \n\\item[(ii)] $\\exists \\epsilon_i > 0$ such that \n$f_i(v)-f_i(\\tilde v)- \\langle \\tilde s_i , v-\\tilde v \\rangle \\ge -\\frac{\\epsilon_i}{2} \\| v-\\tilde v\\|^2$ \nfor  $v,\\tilde v\\in\\mathbb R^{n_i}$ and $\\tilde s_i\\in\\partial f_i(\\tilde v)$. \nThis is equivalent to the weak convexity of the function $f_i$, $i=1,\\dots,p$.\n \\end{itemize}\n \\end{itemize}\n","sha256_utf8":"46569f889da1620518d29912d04c404c916746941374925d34dcbf3ca706ddf6"},{"start":27464,"end":27992,"start_line":527,"end_line":539,"raw_tex":"\\begin{eqnarray}\\label{quantities}\n\\left\\{\n\\begin{tabular}{ll}\n$\\rho(\\beta):=1-|1-\\beta| $,&  \\\\\n$c_1:=\\displaystyle{\\frac{1}{\\alpha\\rho(\\beta)\\lambda_{++}^{B\\tr B}}}$,&\n$c_2:= \\displaystyle{  \\frac{2\\beta}{\\alpha\\rho(\\beta)^2\\lambda_{++}^{B\\tr B}}}$, \\\\\n$c_3:=c_2\\|P\\|^2$, & $c_4:=   c_2 (\\|P\\|+L_h)^2$, \\\\\n$c_5:=\\beta^{-1}|1-\\beta| c_1$, \n&$\\displaystyle{c_6:=\\frac{|1-\\beta| }{2\\alpha\\beta^2\\lambda_{++}^{B\\tr B}}}$, \\\\\n$D=2P+\\alpha B\\tr B-L_h I_m$, &$\\bar D=D-2\\epsilon_0(c_3+c_4)I$, \\\\\n\\end{tabular}\n \\right.\n\\end{eqnarray}","sha256_utf8":"8a17376d5b908f5b98175471ec42a48eb2717d4372594e8db320db82b8141c48"},{"start":27993,"end":28148,"start_line":540,"end_line":541,"raw_tex":"where $\\epsilon_0>1$, $\\lambda_{++}^{B\\tr B}$ is  the smallest \nstrictly-positive eigenvalue of $B\\tr B$, and $I_m$ denotes a $m\\times m$ identity matrix.\n","sha256_utf8":"883074af689ddc2d320035aaefb507a8f11aab5330c229fa500470093f6b134c"},{"start":46977,"end":47370,"start_line":1065,"end_line":1072,"raw_tex":"We define a modified version of the augmented Lagrangian function (\\ref{augL}).\nWe consider $\\bar{\\mathcal L}:\\mathbb R^n\\times \\mathbb R^q\\times \\mathbb R^m\\times \\mathbb R^q\\times \\mathbb R^m\\to (-\\infty,+\\infty] $ defined by \n%\n\\begin{eqnarray}\\label{Lbar}\n\\bar{\\mathcal L}(x,y,z,y',z')=\\mathcal L^{\\alpha}(x,y,z)+\n\\epsilon_0 c_5 \\|B\\tr(z-z')\\|^2+\n\\epsilon_0  c_3 \\|y-y'\\|^2,\n\\end{eqnarray}","sha256_utf8":"51ac1e56493ad5b4cc5aed53db0fa534c54982070fd8bb3e8a9f6fe06d85a670"},{"start":47373,"end":47882,"start_line":1074,"end_line":1083,"raw_tex":"where $\\epsilon_0>1$, and $c_3$ and $c_5$ are given in (\\ref{quantities}). \nEvaluating $\\bar{\\mathcal L}$ at the point $({\\bf x}^{k}, y^{k},z^{k},y^{k-1},z^{k-1})$ where \n$\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 1}$ is the sequence generated by the PADMM algorithm (\\ref{padmm})\ngives\n\\begin{eqnarray}\\label{Lbark}\n\\bar{\\mathcal L}_k:=\n\\bar{\\mathcal L}({\\bf x}^{k}, y^{k},z^{k},y^{k-1},z^{k-1})=\\mathcal L^{\\alpha}({\\bf x}^k,y^k,z^k)+\n\\epsilon_0 c_5 \\|B\\tr\\Delta z^k\\|^2+\n\\epsilon_0 c_3 \\|\\Delta y^k\\|^2.\n\\end{eqnarray}","sha256_utf8":"17b7232908cde026635c6b4629bc977b02176f07e569355b2df6ade97983a746"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{conv}\nSuppose that  A1-A4 hold and $\\sigma$ defined in Lemma \\ref{thm1} is positive.\nLet the sequence \n$\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ generated by the PADMM algorithm (\\ref{padmm}) be bounded and \n$\\bar{\\mathcal L}({\\bf x},y,z,y',z')$ satisfies the {K\\L} property on $\\Omega$ (\\ref{omega});\nthat is, for every \n$v^*:=({\\bf x}^*, y^*, z^*,  y^*, z^*) \\in \\Omega$\nthere exists $\\epsilon>0$, $\\eta\\in [0,+\\infty)$, and a function\n$\\psi\\in\\Psi_{\\eta}$ such that for any  \n$v:=({\\bf x},y,z,y',z')\\in\\mathcal S$ where \n%\n\\begin{eqnarray}\\label{eq:intersec}\n\\mathcal S:=\\Big\\{v \\in \\mathbb R^n\\times\\mathbb R^m\\times\\mathbb R^p\\times\\mathbb R^m\\times\\mathbb R^p:\n\\;{\\rm dist}(v,\\Omega)<\\epsilon\\} \\cap [\\bar{\\mathcal L}(v^*) <\\bar{\\mathcal L}< \\bar{\\mathcal L}(v^*)+\\eta],\n\\end{eqnarray}\nit holds \n%\n\\begin{eqnarray}\\label{eq:thm1KL}\n\\psi'\\Big(\\bar{\\mathcal L}(v)-\\bar{\\mathcal L}(v^*)\\Big) \\cdot\n{\\rm dist}\\big(0,\\partial\\bar{\\mathcal L}(v)\\big)\\ge 1.\n\\end{eqnarray}\n%\n Then $\\{u^k\\}_{k\\ge 0}:=\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$\n satisfies \n\\[\n\\sum_{k=1}^{\\infty} |||\\Delta u^k|||\\le \\sum_{k=1}^{\\infty} \n\\Big\\{\\sum_{i=1}^p\\| \\Delta x_i^{k}\\|\n+ \\| \\Delta y^{k}\\|\n+ \\| \\Delta z^{k}\\|\\Big\\}<+\\infty,\n\\]\nand consequently it converges to a stationary point of (\\ref{OPE}).\n\\end{theorem}","units":[{"start":61778,"end":63079,"start_line":1463,"end_line":1494,"raw_tex":"\\begin{theorem}\\label{conv}\nSuppose that  A1-A4 hold and $\\sigma$ defined in Lemma \\ref{thm1} is positive.\nLet the sequence \n$\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ generated by the PADMM algorithm (\\ref{padmm}) be bounded and \n$\\bar{\\mathcal L}({\\bf x},y,z,y',z')$ satisfies the {K\\L} property on $\\Omega$ (\\ref{omega});\nthat is, for every \n$v^*:=({\\bf x}^*, y^*, z^*,  y^*, z^*) \\in \\Omega$\nthere exists $\\epsilon>0$, $\\eta\\in [0,+\\infty)$, and a function\n$\\psi\\in\\Psi_{\\eta}$ such that for any  \n$v:=({\\bf x},y,z,y',z')\\in\\mathcal S$ where \n%\n\\begin{eqnarray}\\label{eq:intersec}\n\\mathcal S:=\\Big\\{v \\in \\mathbb R^n\\times\\mathbb R^m\\times\\mathbb R^p\\times\\mathbb R^m\\times\\mathbb R^p:\n\\;{\\rm dist}(v,\\Omega)<\\epsilon\\} \\cap [\\bar{\\mathcal L}(v^*) <\\bar{\\mathcal L}< \\bar{\\mathcal L}(v^*)+\\eta],\n\\end{eqnarray}\nit holds \n%\n\\begin{eqnarray}\\label{eq:thm1KL}\n\\psi'\\Big(\\bar{\\mathcal L}(v)-\\bar{\\mathcal L}(v^*)\\Big) \\cdot\n{\\rm dist}\\big(0,\\partial\\bar{\\mathcal L}(v)\\big)\\ge 1.\n\\end{eqnarray}\n%\n Then $\\{u^k\\}_{k\\ge 0}:=\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$\n satisfies \n\\[\n\\sum_{k=1}^{\\infty} |||\\Delta u^k|||\\le \\sum_{k=1}^{\\infty} \n\\Big\\{\\sum_{i=1}^p\\| \\Delta x_i^{k}\\|\n+ \\| \\Delta y^{k}\\|\n+ \\| \\Delta z^{k}\\|\\Big\\}<+\\infty,\n\\]\nand consequently it converges to a stationary point of (\\ref{OPE}).\n\\end{theorem}","sha256_utf8":"4e62a748c730eeca3563d93bc2ba564d66ab38a7e75b050fb1827954dab543c7"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\nLet $\\Phi:\\mathbb R^d\\to\\mathbb R$ be Fr\\'echet differentiable such that \nits gradient is Lipschitz continuous with constant $L_{\\Phi}>0$. Then \nfor every $u,v\\in\\mathbb R^d$ and every $\\xi\\in[u,v]=\\{(1-t)u+tv:\\;t\\in[0,1]\\}$\nit holds \n\\begin{eqnarray}\\label{Lip}\n\\Phi(v)\\le \\Phi(u)+\\langle \\nabla \\Phi(\\xi),v-u\\rangle +\\frac{L_{\\Phi}}{2}\\|v-u\\|^2.\n\\end{eqnarray}\n\n\\end{lemma}\n\n\\begin{proof}\nLet $u, v\\in\\mathbb R^d$, and $\\xi=(1-t)u+tv$ for $t\\in [0,1]$. Then we have\n%\n\\begin{eqnarray*}\n&\\Phi(v) -\\Phi(u) = \\int_0^1 \\langle \\nabla \\Phi((1-r) u+rv), v-u \\rangle dr &\\\\\n&=  \\int_0^1  \\langle \\nabla \\Phi((1-r) u+rv)-\\nabla \\Phi(\\xi), v-u \\rangle dr\n+\\langle \\nabla \\Phi(\\xi) , v-u\\rangle. &\n\\end{eqnarray*}\n\nSince $\\Phi$ has Lipschitz continuous gradients with constant $L_{\\Phi}>0$, then we have\n\\begin{eqnarray*}\n&|\\Phi(v) -\\Phi(u) - \\langle \\nabla \\Phi(\\xi) , v-u\\rangle|\n=\\Big| \\int_0^1  \\langle \\nabla \\Phi((1-r) u+rv)-\\nabla \\Phi(\\xi), v-u \\rangle dr\\Big | \n&\\\\\n&\n\\le  \\int_0^1  \\Big\\|  \\nabla \\Phi((1-r) u+rv)-\\nabla \\Phi(\\xi)\\Big\\|\\cdot \\Big\\|v-u\\Big \\|dr \n\\le  L_{\\Phi} \\|v-u\\|^2 \\int_0^1 |r-t| dr\n& \\\\ &\n=  L_{\\Phi} \n\\|v-u\\|^2 \\Big( \\int_0^t (-r+t) dr + \\int_t^1 (r-t) dr \\Big)\n& \\\\&\n=  L_{\\Phi} \\Big(\\frac 12 -t(1-t)\\Big) \\|v-u\\|^2 \n\\le \\frac {L_{\\Phi}}{2}\\|v-u\\|^2.\\quad  \\square &\n\\end{eqnarray*}\n\n\n\\end{proof}\n\n\\begin{lemma}\\label{thm-d}\n{(Subgradient bound)} Let $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ be a sequence generated by the PADMM\n algorithm (\\ref{padmm}). Then there exists \n$d^{k}:=\\big(\\{d^{k}_{x_i}\\}_{i=1}^p,d^{k}_y,d^{k}_z\\big)\\in\\partial \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k})$ and  a constant $\\rho>0$\nsuch that \n%\n\\begin{eqnarray}\\label{nd}\n|||d^{k}||| &\\le&\n \\rho \\Big(\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|+ \\|\\Delta y^{k}\\|+ \\|\\Delta z^{k}\\|\\Big), \n\\end{eqnarray}\n%\nwhere %\n\\begin{eqnarray}\\label{rho}\n\\rho:=\\max\\Big\\{\n\\|{\\bf Q}\\|+\\|{\\bf A}\\|^2,\\;\\;\\;\n\\|P\\|+\n\\alpha \\|{\\bf A}\\|\\|B\\|\n, \\;\\;\\;\n\\|{\\bf A}\\|  +\\|B\\|+\\frac{1}{\\alpha\\beta}\\Big\\},\n\\end{eqnarray}\n%\n$\\|{\\bf Q}\\|:=\\sum_{i=1}^p\\|Q_i\\|$ and $\\|{\\bf A}\\|:=\\sum_{i=1}^p\\|A_i\\|$,\nand for any sequence $\\{u^k\\}_{k\\ge 0}$,\n$\\Delta u^{k+1}=u^{k+1}-u^k$.\n\\end{lemma}\n\n\\begin{proof}\nLet $k\\ge 1$ be fixed. By the optimality condition of the $x_i$ subproblem\nof PADMM algorithm for \n$i=1,\\dots, p$ we have\n%\n\\begin{eqnarray*}\n-A_i\\tr z^{k-1}\n-\\alpha A_i\\tr (A_{<i}x_{<i}^{k}+A_ix_i^{k}+A_{>i}x_{>i}^{k-1}+By^{k-1}+b)\n-Q_i\\Delta x_i^{k}\n\\in\\partial f_i(x_i^{k}).\n\\end{eqnarray*}\n%\nTaking partial differential of $\\mathcal L^{\\alpha}$ with respect to $x_i$\nand evaluating the result at the point $({\\bf x}^{k},y^{k},z^{k})$ yields\n\\[\n\\partial_{x_i} \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k})=\n\\partial f_i(x_i^{k})+A_i\\tr z^{k}+\n\\alpha A_i\\tr \\big({\\bf Ax}^{k}+By^{k}+ b\\big).\n\\]\n%\nTherefore \n\\begin{eqnarray}\\label{dxinL}\n\\begin{array}{lll}\nd_{x_i}^{k}:=\nA_i\\tr \\Delta z^{k}+\n\\alpha A_i\\tr A_{>i}\\Delta x_{>i}^{k}\n+\\alpha A_i\\tr B\\Delta y^{k}\n-Q_i\\Delta x_i^{k}\n\\in \\partial_{x_i} \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k}).\n\\end{array}\n \\end{eqnarray}\n%\nThe optimality criterion of $y$ subproblem gives\n\\[\n\\nabla h(y^{k})=-B\\tr z^{k-1}- \\alpha B\\tr ({\\bf A x}^{k}+By^{k}+b) \n- {P} \\Delta y^{k}.\n\\]\nBy taking partial differential of $\\mathcal L^{\\alpha}$ with respect to $y$\nand evaluating it at $({\\bf x}^{k},y^{k},z^{k}) $ we obtain\n\\[\n\\nabla_y \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k}) =\n\\nabla h(y^{k})+B\\tr z^{k}+\\alpha B\\tr \\big({\\bf Ax}^{k}+By^{k}+b\\big).\n\\]\nThe last two equations follow that\n\\begin{eqnarray}\\label{dyinL}\nd_y^{k}:=B\\tr \\Delta z^{k}- P \\Delta y^{k}\n= \\nabla_y \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k}).\n\\end{eqnarray}\n%\nLastly, by the $z$ subproblem we have ${\\bf Ax}^{k}+By^{k}+b=\\frac{1}{\\alpha\\beta}\n \\Delta z^k$ hence \n\\begin{eqnarray}\\label{dzinL}\nd_z^{k}:=\\frac{1}{\\alpha\\beta}\n \\Delta z^{k}= \\nabla_z \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k}).\n\\end{eqnarray}\nHence, by (\\ref{dxinL}), (\\ref{dyinL}), , and (\\ref{dzinL}), we \n have \n$\nd^{k}:=\\big(\\{d^{k}_{x_i}\\}_{i=1}^p,d^{k}_y,d^{k}_z\\big)\\in\\partial \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k}).\n$\n%\nBy the norm property we have $\\|d_z^{k}\\|=\\frac{1}{\\alpha\\beta} \\|\\Delta z^{k}\\|$ \nand by the  triangle inequality from (\\ref{dxinL}) and (\\ref{dyinL}) we obtain \n%\n\\begin{eqnarray*} \\label{ndx}\n&\\|d_{x_i}^{k}\\|\\le\n\\|A_i\\|\\|\\Delta z^{k}\\|+\\alpha \\|A_i\\|\\Big({\\sum_{j=i+1}^p}\\|A_j\\|\\|\\Delta x_j^{k}\\|\\Big)\n+\\alpha\\|A_i\\|\\|B\\|\\|\\Delta y^{k}\\|+\\|Q_i\\|\\|\\Delta x_i^{k}\\|, &\\\\\n%\n&\\|d_y^{k}\\|\\le\\|B\\| \\|\\Delta z^{k}\\| +\\|P\\| \\|\\Delta y^{k}\\|. &\n%\\|d_z^{k}\\|=\\frac{1}{\\alpha\\beta} \\|\\Delta z^{k}\\|.\n\\end{eqnarray*}\n%\nHence we get\n%\n\\begin{eqnarray*}\n&|||d^{k}||| \\le  \\sum_{i=1}^p\\|d_{x_i}^{k}\\|+ \\|d_y^{k}\\|+\\|d_z^{k}\\|\n\\le \\sum_{i=1}^p \\|Q_i\\|\\|\\Delta x_i^{k}\\|\n+ \\sum_{i=2}^p \\|A_i\\| \\|\\Delta x_i^{k}\\|\\sum_{j=1}^{i-1}\\|A_j\\| \n& \\\\&\n+\\Big(\\|P\\|+\\alpha\\|B\\|\\sum_{i=1}^p \\|A_i\\|  \\Big)      \\|\\Delta y^{k}\\|\n+\\Big( \\sum_{i=1}^p \\|A_i\\|  +\\|B\\|+\\frac{1}{\\alpha\\beta}\\Big)       \\|\\Delta z^{k}\\|\n&\\\\&\n\\le \n\\Big(\\|{\\bf Q}\\|+\\|{\\bf A}\\|^2\\Big) \\sum_{i=1}^p\\|\\Delta x_i^{k}\\|\n+\\Big(\\|P\\|+\\alpha \\|{\\bf A}\\| \\|B\\| \\Big)      \\|\\Delta y^{k}\\|\n+\\Big(\\|{\\bf A}\\|  +\\|B\\|+\\frac{1}{\\alpha\\beta}\\Big)  \\|\\Delta z^{k}\\|\n&\\\\&\n\\le \\rho \\Big( \\sum_{i=1}^p\\|\\Delta x_i^{k}\\|+   \\|\\Delta y^{k}\\|+  \\|\\Delta z^{k}\\|\\Big).\n\\quad\\square &\n\\end{eqnarray*}\n%\n\\end{proof}\n\n\\begin{lemma}\\label{thm-cluster}\n{(Limiting continuity)} \nSuppose that A1-A2  hold. \nIf $({\\bf x}^*, y^*, z^*)$ is the limit point of $\\{({\\bf x}^{k_j},y^{k_j},z^{k_j})\\}_{j\\ge 0}$, a subsequence of \n$\\{{\\bf x}^k, y^k, z^k\\}_{k\\ge 0}$ generated by the PADMM algorithm (\\ref{padmm}). Then \n$ \\mathcal L^{\\alpha} ({\\bf x}^*, y^*, z^*)=\\lim_{j\\to\\infty} \\mathcal L^{\\alpha} ({\\bf x}^{k_j},y^{k_j},z^{k_j}).\n$\n\\end{lemma}\n\n\\begin{proof}\nLet $\\{({\\bf x}^{k_j},y^{k_j},z^{k_j})\\}_{j\\ge 0}$ be a subsequence of the sequence \n$\\{({\\bf x}^{k},y^{k},z^{k})\\}_{j\\ge 0}$\ngenerated by the PADMM algorithm such that\n$\\lim_{j\\to\\infty} ({\\bf x}^{k_j},y^{k_j},z^{k_j})=({\\bf x}^*, y^*, z^*).$\nBy A1, for $i=1,\\dots,p$ the functions $f_i$ are lower semicontinuous hence\n\n\\begin{eqnarray}\\label{eq:flsc0}\nf_i(x_i^*)\\le \\lim\\inf_{j\\to\\infty} f(x_i^{k_j}).\n\\end{eqnarray}\n%\nBy the $x_i$-subproblem of the PADMM algorithm for any $v\\in\\mathbb R^{n_i}$ we have \n\\[\n\\mathcal L^{\\alpha} (x_{<i}^{k_j+1},x_i^{k_j+1}, x^{k_j}_{>i}, y^{k_j},z^{k_j})+\n\\frac{1}{2}\\|\\Delta x_i^{k_j+1}\\|_{Q_i}^2\n\\le \\mathcal L^{\\alpha} (x_{<i}^{k_j+1},v, x^{k_j}_{>i}, y^{k_j},z^{k_j})\n+\\frac{1}{2}\\|v- x_i^{k_j}\\|_{Q_i}^2.\n\\]\nWe choose $v=x_i^*$ and exploit the cosine rule\n$\\|a_1+c\\|^2-\\|a_2+c\\|^2=\\|a_2-a_1\\|^2+2\\langle a_2+c, a_2-a_1\\rangle$\nwith \n$a_1=A_ix_i^*$, $a_2=A_ix_i^{k_j+1}$ \nand $c=A_{<i}x_{<i}^{k_j+1}+A_{>i}x^{k_j}_{>i}+By^{k_j}+b$ to obtain\n %\n\\begin{eqnarray*}\n&f_i(x_i^{k_j+1})- f_i(x_i^*) +\n\\langle A_i\\tr z^{k_j},x^{k_j+1}_i-x_i^*\\rangle+\\|\\Delta x_i^{k_j+1}\\|^2_{Q_i} \n\\le \n\\frac{\\alpha}{2}\\|A_i (x_i^{k_j+1}-x_i^*)\\|^2&\n\\\\\n&+\\alpha \\Big\\langle A_i (x_i^*-x_i^{k_j+1}), A_{<i}x^{k_j+1}_{<i}+A_ix_i^{k_j+1}+A_{>i}x_{>i}^{k_j}+By^{k_j}+b\\Big\\rangle.\n&\n\\end{eqnarray*}\n%\nSince $\\{({\\bf x}^{k_j},y^{k_j},z^{k_j})\\}_{j\\ge 0}$  is a convergent sequence, \n it is bounded and the difference of consequtive terms approaches  zero. \nThus the latter inequality reduces to \n$\\displaystyle{\\lim\\sup_{j\\to \\infty}} f_i(x_i^{k_j+1})-f_i(x_i^*)\\le 0$ and \nin view of (\\ref{eq:flsc0}) we then have\n$\n\\lim_{j\\to \\infty} f_i(x_i^{k_j})=f_i(x_i^*),\n$\nwhich follows that\n$\n\\lim_{j\\to \\infty} \\sum_{i=1}^p f_i(x_i^{k_j})=\\sum_{i=1}^p f_i(x_i^*).\n$\n%\nBy A2, $h(y)$ is smooth hence $\\lim_{j\\to\\infty} h(y^{k_j})= h(y^*)$.\nClearly, ${\\bf Ax}^{k_j}+By^{k_j}+b\\to {\\bf Ax}^{*}+By^{*}+b$\nas $j\\to\\infty$.\nThus $\\displaystyle{\\lim_{j\\to\\infty}} \\mathcal L^{\\alpha} ({\\bf x}^{k_j},y^{k_j},z^{k_j})\n= \\mathcal L^{\\alpha} ({\\bf x}^*, y^*, z^*).$ $\\quad\\square$\n\\end{proof}\n\n\\begin{lemma}\\label{thm-cluster0}\n{(Limit point is critical point)} The set of limit points of the sequence\n$\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ generated by the PADMM algorithm (\\ref{padmm}),\ndenoted \n$\\omega\\big(\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}\\big)$, satisfies\n\\[\n\\omega\\big( \\{ ({\\bf x}^k,y^k,z^k) \\}_{k\\ge 0} \\big) \\subseteq {\\rm crit} \\;\\mathcal L^{\\alpha}\n:=\n\\left\\{\n({\\bf x}^*,y^*,z^*):\n\\begin{array}{l}\n 0\\in\\partial f_i(x_i^*)+A_i\\tr z^*,\\; i=1,\\dots, p\\\\\n\\nabla h(y^*)+ B\\tr z^*=0,\\\\\n{\\bf Ax}^*+By^*+b=0\n\\end{array}\n\\right\\},\n\\]\nwhere ${\\rm crit}\\;\\mathcal L^{\\alpha}$ denotes the set of critical points of $\\mathcal L^{\\alpha}$.\n%\n\\end{lemma}\n\n\\begin{proof}\nLet $(\\hat {\\bf x}, \\hat y, \\hat z)\\in \\omega\\big( \\{ ({\\bf x}^k,y^k,z^k) \\}_{k\\ge 0} \\big)$,\nhence there exists a subsequence $\\{({\\bf x}^{k_j},y^{k_j},z^{k_j})\\}_{j\\ge 0}$\nof $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ such that \n$\\lim_{j\\to\\infty} ({\\bf x}^{k_j},y^{k_j},z^{k_j})=(\\hat {\\bf x}, \\hat y, \\hat z).$\nFor the sequence $\\{d^k\\}_{k\\ge 0}$, we have $d^{k_j}\\in\\partial \\mathcal L^{\\alpha}(x^{k_j}, y^{k_j}, z^{k_j})$\nfor $j\\ge 1$ and $d^{k_j}\\to 0$ as $j\\to \\infty$ while $({\\bf x}^{k_j}, y^{k_j}, z^{k_j})\\to (\\hat {\\bf x}, \\hat y,\\hat z)$\nand $\\displaystyle{\\lim_{j\\to\\infty}} \\mathcal L^{\\alpha} ({\\bf x}^{k_j},y^{k_j},z^{k_j})\n= \\mathcal L^{\\alpha} (\\hat {\\bf x}, \\hat y, \\hat z).$\n By the closeness criterion of the limiting sub-differential \nwe then have $0\\in \\partial \\mathcal L^{\\alpha}(\\hat{\\bf x}, \\hat{y}, \\hat {z})$,\nand equivalently, $(\\hat{\\bf x}, \\hat{y}, \\hat {z})\\in {\\rm crit} (\\mathcal L^{\\alpha})$.\n $\\square$\n\\end{proof}\n\n\\begin{lemma}{(bound dual)}\\label{bounddual}\nSuppose that A2 and A3 hold.\n%\\[\n%\\rho(\\beta):=1-|1-\\beta|,\n%\\quad \n%c_1:=  \\frac{1}{\\alpha\\rho(\\beta)\\lambda_{++}^{B\\tr B}},\n%\\quad\n%c_2:=  \\frac{2\\beta}{\\alpha\\rho(\\beta)^2\\lambda_{++}^{B\\tr B}}.\n%\\]\n%where  $\\lambda_{++}^{B\\tr B}$ be the smallest \n%strictly-positive eigenvalue of $B\\tr B$.\n%\nThen for all $k\\in\\mathbb Z_+$ the following assertions are true.\n\\begin{itemize}\n%\n\\item [(i)] $B\\tr z^{k}=\\beta w^{k}+(1-\\beta) B\\tr z^{k-1},$ where \n$w^{k}=-P\\Delta y^{k}-\\nabla h(y^{k}),$\n%\n\\item [(ii)]\n$\\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2\n\\le c_4 \\|\\Delta y^{k+1}\\|^2\n+c_3 \\|\\Delta y^{k}\\|^2+c_5 \\|B\\tr \\Delta z^{k}\\|^2\n-c_5\\| B\\tr \\Delta z^{k+1}\\|^2$,\n%\n\\item [(iii)]\n$\\frac{1}{2\\alpha}\\|z^{k+1}\\|^2\\le c_1  \\|P\\|^2 \\|\\Delta y^{k+1}\\|^2\n+ c_1 \\|\\nabla h(y^{k+1})\\|^2\n+c_6\\|B\\tr \\Delta z^{k+1}\\|^2$,\n\\end{itemize}\nwhere $c_i$, $i=2,\\dots,6$ are defined in (\\ref{quantities}).\n\\end{lemma}\n\n\\begin{proof}\n{\\bf (i)} \nBy the the optimality condition of $y$-subproblem we have\n\\[\n\\alpha B\\tr ({\\bf Ax}^{k}+By^{k}+b)+B\\tr z^{k-1}-w^k=0\n\\]\nwhich together with the $z$-iterate ${\\bf Ax}^{k}+By^{k}+b=\\frac{1}{\\alpha\\beta}(z^k-z^{k-1})$ we obtain the result.\n\n\n{\\bf (ii)} By A3 and the $z$ iterate of the PADMM algorithm we have $\\Delta z^{k+1}\\in {\\rm Im}(B)$, thus \n $\\lambda_{++}^{B\\tr B}\\|\\Delta z^{k+1}\\|^2\\le \\|B\\tr \\Delta z^{k+1}\\|^2$.\nPart (i) follows \n$B\\tr \\Delta z^{k+1}=\\beta\\Delta w^{k+1}+(1-\\beta) B\\tr \\Delta z^{k}$ and \nequivalently \n%\n\\begin{eqnarray*}\nB\\tr \\Delta z^{k+1}=\\rho(\\beta)\n\\Big(\\frac{\\beta\\Delta w^{k+1}}{\\rho(\\beta)} \\Big)\n+|1-\\beta| \\Big({\\rm sign}(1-\\beta) B\\tr \\Delta z^k\\Big),\n\\end{eqnarray*}\nwhere ${\\rm sign} (\\lambda)=1$ if $\\lambda\\ge 0$ and ${\\rm sign} (\\lambda)=-1$ if $\\lambda<0$.\nBy $\\rho(\\beta)+|1-\\beta|=1$ and the convexity of $\\|\\cdot\\|^2$ \nwe have\n%\n\\begin{eqnarray}\\label{gg}\n\\|B\\tr \\Delta z^{k+1}\\|^2\n\\le  \\frac{\\beta^2}{\\rho(\\beta)}  \\|\\Delta w^{k+1}\\|^2\n+|1-\\beta| \\|B\\tr \\Delta z^{k}\\|^2.%-|1-\\beta| \\Big\\| B^*\\Delta z^{k+1}\\Big\\|^2.\n\\end{eqnarray}\nThis also follows that \n%\n\\begin{eqnarray*}\\label{gg}\n\\rho(\\beta)\\lambda_{++}^{B\\tr B} \\|\\Delta z^{k+1}\\|^2&\\le& \\rho(\\beta)\\|B\\tr \\Delta z^{k+1}\\|^2\\\\\n%\n&\\le&  \\frac{\\beta^2}{\\rho(\\beta)}  \\|\\Delta w^{k+1}\\|^2\n+|1-\\beta| \\|B\\tr \\Delta z^{k}\\|^2-|1-\\beta| \\| B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray*}\n%\nRearrange this to obtain\n%\n\\begin{eqnarray}\\label{gg}\n\\|\\Delta z^{k+1}\\|^2\\le\n \\frac{\\beta^2}{\\rho(\\beta)^2\\lambda_{++}^{B\\tr B} }  \\|\\Delta w^{k+1}\\|^2\n+\\frac{|1-\\beta|}{\\rho(\\beta)\\lambda_{++}^{B\\tr B} } \\|B\\tr \\Delta z^{k}\\|^2\n-\\frac{|1-\\beta|}{\\rho(\\beta)\\lambda_{++}^{B\\tr B} } \\| B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray}\n%\nBy the optimality condition of $y$ subproblem we have \n\\[\n\\Delta w^{k+1}=-P\\Delta y^{k+1}+P\\Delta y^k+\\nabla h(y^{k})-\\nabla h(y^{k+1}).\n\\]\nBy $\\|a+b\\|^2\\le 2\\|a\\|^2+2\\|b\\|^2$ and the fact that $\\nabla h$ has $L_h$ Lipschitz gradients\nwe obtain\n\\[\n\\|\\Delta w^{k+1}\\|^2\\le 2(\\|P\\|+L_h)^2\\|\\Delta y^{k+1}\\|^2+2\\|P\\|^2 \\|\\Delta y^k\\|^2.\n\\]\nExploit this in (\\ref{gg}) to get\n\\begin{eqnarray*}\\label{gg0}\n\\|\\Delta z^{k+1}\\|^2&\\le&\n \\frac{2\\beta^2 (\\|P\\|+L_h)^2}{\\rho(\\beta)^2\\lambda_{++}^{B\\tr B} }          \\|\\Delta y^{k+1}\\|^2\n + \\frac{2\\beta^2\\|P\\|^2}{\\rho(\\beta)^2\\lambda_{++}^{B\\tr B} }    \\|\\Delta y^k\\|^2\\\\\n &&+\\frac{|1-\\beta|}{\\rho(\\beta)\\lambda_{++}^{B\\tr B} } \\|B\\tr \\Delta z^{k}\\|^2\n-\\frac{|1-\\beta|}{\\rho(\\beta)\\lambda_{++}^{B\\tr B} } \\| B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray*}\nWe divide both sides by $\\alpha\\beta$ to get \n%\n\\begin{eqnarray*}\n\\frac{1}{\\alpha\\beta}\\Big\\|\\Delta z^{k+1}\\Big\\|^2\n\\le c_4 \\Big\\|\\Delta y^{k+1}\\Big\\|^2\n+c_3 \\|\\Delta y^{k}\\|^2+c_5 \\Big\\|B\\tr \\Delta z^{k}\\Big\\|^2\n-c_5 \\Big\\| B\\tr \\Delta z^{k+1}\\Big\\|^2.\n\\end{eqnarray*}\n\n{\\bf (iii).} By part (i) we have\n%\n$\\beta B\\tr z^{k+1}=\\beta w^{k+1}+(1-\\beta) B\\tr (z^k-z^{k+1})$ \nwhich can be written equivalently as follows\n%\n\\begin{eqnarray*}\n\\beta B\\tr z^{k+1}= \\rho(\\beta)\\cdot \\frac{\\beta w^{k+1}}{\\rho(\\beta)}\n+|1-\\beta|\\cdot {\\rm sign}(1-\\beta) B\\tr (z^k-z^{k+1}).\n\\end{eqnarray*}\nBy $\\rho(\\beta)+|1-\\beta|=1$ and the convexity of $\\|\\cdot\\|^2$\nwe get\n%\n\\begin{eqnarray*}\n\\beta^2\\lambda_{++}^{B\\tr B} \\|z^{k+1}\\|^2\n\\le   \\beta^2 \\|B\\tr z^{k+1}\\|^2\n\\le\n\\rho(\\beta)\\cdot \\Big\\| \\frac{\\beta w^{k+1}}{\\rho(\\beta)}\\Big\\|^2\n+|1-\\beta| \\|B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray*}\n%\nSimplify and rearrange to get\n%\n\\begin{eqnarray*}\n\\|z^{k+1}\\|^2\n\\le \n\\frac{1}{\\rho(\\beta)\\lambda_{++}^{B\\tr B}}      \\|w^{k+1}\\|^2\n+\\frac{|1-\\beta|}{\\beta^2 \\lambda_{++}^{B\\tr B}      } \\|B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray*}\n%\nExploit $\\|w^{k+1}\\|^2\\le 2 \\|P\\|^2 \\|\\Delta y^{k+1}\\|^2+2\\|\\nabla h(y^{k+1})\\|^2$\nto obtain \n%\n\\begin{eqnarray*}\n\\|z^{k+1}\\|^2\n\\le\n\\frac{2\\|P\\|^2}{\\rho(\\beta)\\lambda_{++}^{B\\tr B}}  \\|\\Delta y^{k+1}\\|^2\n+ \\frac{2}{\\rho(\\beta)\\lambda_{++}^{B\\tr B}} \\|\\nabla h(y^{k+1})\\|^2\n+\\frac{|1-\\beta|}{\\beta^2\\lambda_{++}^{B\\tr B}} \\|B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray*}\n%\nDivide both sides by $2\\alpha$ to obtain the result.\nThis concludes the proof.  $\\square$\n\\end{proof}\n\n\\begin{lemma}\\label{chgL}\n{(change of $\\mathcal L^{\\alpha}$ during each variable update)} \nSuppose that A1-A4 hold. The iterates in PADMM algorithm (\\ref{padmm}) satisfy\n%\n\\begin{itemize}\n\\item [(i)] $\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^k, x^{k}_{>i},y^k,z^k)-\n\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^{k+1}, x^{k}_{>i},y^k,z^k)\\ge \\frac{1}{2} \\|\\Delta x_i^{k+1}\\|_{T_i}^2$ where\n$T_i=Q_i$ or $T_i=\\alpha A_i\\tr A_i-\\epsilon_i I$ where $\\epsilon_i >0$ defined in A4 for $i=1,\\dots, p$.\n%\n\\item [(ii)]\n$\\mathcal L^{\\alpha}({\\bf x}^{k+1},y^k, z^k)-\\mathcal L^{\\alpha}({\\bf x}^{k+1},y^{k+1},z^{k})\n\\ge \\frac{1}{2} \\|\\Delta y^{k+1}\\|_D^2$, where $D$ is defined in (\\ref{quantities}).\n\\item [(iii)] $\\mathcal L^{\\alpha}({\\bf x}^{k+1},y^{k+1}, z^k)-\n\\mathcal L^{\\alpha}({\\bf x}^{k+1},y^{k+1},z^{k+1})=-\\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2.$\n%\n\\end{itemize}\n\\end{lemma}\n\n\\begin{proof}\n{\\bf (i)} Let $k\\in\\mathbb Z_+$ and $i\\in\\{1,\\dots,p\\}$ be fixed.\nBy the $x_i$ subproblem we have\n\\[\n\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^{k}, x^{k}_{>i},y^k,z^k)-\n\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^{k+1}, x^{k}_{>i},y^k,z^k)\n\\ge \\|\\Delta x_i^{k+1}\\|_{Q_i}^2.\n\\]\n%\nOn the other hand, by the optimality condition of $x_i$-sub-minimization problem we have \n$ \\bar{d_i}\\in\\partial f_i(x_i^{k+1})$\nwhere \n\\[\n\\bar{d_i}:=-A_i \\tr z^k-\n\\alpha A_i\\tr \\big(A_{<i} x_{<i}^{k+1}+A_ix_i^{k+1}+A_{>i}x_{>i}^k+By^k+b\n\\big).\n\\]\nBy this we then have\n\\begin{eqnarray*}\n&\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^k, x^{k}_{>i},y^k,z^k)-\n\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^{k+1}, x^{k}_{>i},y^k,z^k)& \\\\[.1in]\n&=f_i(x_i^k)-f_i(x_i^{k+1})\n+\\frac{\\alpha}{2}\\|A_i\\Delta x_i^{k+1}\\|^2-\\langle \\bar{d_i}, \\Delta x_i^{k+1}\\rangle&\n\\\\[.1in]\n&\\ge \\frac{\\alpha}{2}\\|A_i\\Delta x_i^{k+1}\\|^2\n- \\frac{\\epsilon_i}{2}\\|\\Delta x_i^{k+1}\\|^2\n=\\frac 12 \\|\\Delta x_i^{k+1}\\|_{\\alpha A_i\\tr A_i-\\epsilon_iI}^2.&\n\\end{eqnarray*}\n\n%%%%\n{\\bf (ii)}\nLet $k\\ge 0$ be fixed. From the optimality condition of $y$ minimization problem \nwe have\n\\begin{eqnarray*}\n\\nabla h(y^{k+1})+\\alpha B\\tr ({\\bf Ax}^{k+1}+By^{k+1}+b)\n+B\\tr z^k+P \\Delta y^{k+1}=0.\n\\end{eqnarray*}\nMultiply this equation by $\\Delta y^{k+1}$ and rearrange to obtain\n\\begin{eqnarray}\\label{eq:2}\n- \\alpha\\big\\langle B\\tr ({\\bf Ax}^{k+1}+ By^{k+1}+b),\\Delta y^{k+1}\\big\\rangle \n-\\langle B\\tr z^k, \\Delta y^{k+1}\\rangle\n=\\langle \\nabla h(y^{k+1}),\\Delta y^{k+1}\\rangle +\\|\\Delta y^{k+1}\\|_P^2.\n\\end{eqnarray}\nBy this and the fact that $h(y)$ is $L_h$ Lipschitz continuous we then get \n%\n\\begin{eqnarray*}\n&\\mathcal L^{\\alpha}(x^{k+1},y^k,z^k)-\\mathcal L^{\\alpha}(x^{k+1},y^{k+1},z^{k})&\n\\\\[.1in]\n&= h(y^k)-h(y^{k+1})\n+\\frac{\\alpha}{2}\\|{\\bf Ax}^{k+1}+By^k+b\\|^2\n-\\frac{\\alpha}{2}\\|{\\bf Ax}^{k+1}+By^{k+1}+b\\|^2\n-\\langle B\\tr z^k, \\Delta y^{k+1} \\rangle&\n\\\\[.1in]\n &=h(y^k)-h(y^{k+1})\n+\\frac{\\alpha}{2}\\|B\\Delta y^{k+1}\\|^2\n-\\alpha \\langle \\Delta y^{k+1}, B\\tr ({\\bf Ax}^{k+1}+By^{k+1}+b)\\rangle \n-\\langle B\\tr z^k, \\Delta y^{k+1} \\rangle&\n\\\\[.1in]\n&=h(y^k)-h(y^{k+1})\n+\\langle \\nabla h(y^{k+1}),\\Delta y^{k+1}\\rangle +\\|\\Delta y^{k+1}\\|_P^2\n+\\frac{\\alpha}{2}\\|B\\Delta y^{k+1}\\|^2\n&\\\\[.1in]\n&\\ge\\|\\Delta y^{k+1}\\|_P^2\n+\\frac{\\alpha}{2}\\|B\\Delta y^{k+1}\\|^2\n-\\frac{L_h}{2}\\|\\Delta y^{k+1}\\|^2 = \n\\frac{1}{2} \\|\\Delta y^{k+1}\\|_{D}^2.&\n\\end{eqnarray*}\n%\n%\n{\\bf (iii)} By the $z$ iterate of the algorithm we obtain\n\\begin{eqnarray*}\n&\\mathcal L^{\\alpha}({\\bf x}^{k+1},y^{k+1}, z^k)-\n\\mathcal L^{\\alpha}({\\bf x}^{k+1},y^{k+1},z^{k+1})=\n\\langle z^k-z^{k+1}, {\\bf Ax}^{k+1}+By^{k+1}+b\\rangle\n=-\\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2.&\n\\end{eqnarray*}\n%\nThat completes the proof. $\\square$\n\\end{proof}\n\n\\begin{lemma}\\label{lem:chgLiter}\n{(change of $\\mathcal L^{\\alpha}$ after each iteration)} \nWe assume that A1-A4 hold.\nThe iterates in PADMM algorithm (\\ref{padmm}) satisfy\n%\n\\begin{eqnarray}\\label{chgLiter}\n\\begin{array}{c}\n\\mathcal L^{\\alpha}\\big({\\bf x}^{k+1},y^{k+1},z^{k+1}\\big)\n+\\frac{1}{2}\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|_{T_i}^2\n+ \\frac{1}{2} \\|\\Delta y^{k+1}\\|_{D-2\\epsilon_0c_4 I_m}^2\\\\[.1in]\n+\\displaystyle{\\frac{\\epsilon_0-1}{\\alpha\\beta}}\\big\\|\\Delta z^{k+1}\\big\\|^2\n+\\epsilon_0c_5 \\big\\|B\\tr \\Delta z^{k+1}\\big\\|^2\n\\le  \\mathcal L^{\\alpha}\\big(x^{k},y^k,z^k\\big)+\n\\epsilon_0c_5 \\big\\|B\\tr \\Delta z^k\\big\\|^2\n+\\epsilon_0c_3 \\big\\|\\Delta y^k\\big\\|^2,\n\\end{array}\n\\end{eqnarray}\nwhere $\\epsilon_0>1$, and $c_3,c_4,c_5$ are defined in (\\ref{quantities}).\n%\n\\end{lemma}\n\n\\begin{proof}\nBy Lemma \\ref{chgL} we have \n\\begin{eqnarray*}\n&\\mathcal L^{\\alpha}(x^{k+1},y^{k+1},z^{k+1})=\\mathcal L^{\\alpha}(x^{k+1},y^{k+1},z^{k}) \n+ \\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2\n&\\\\[.1in]&\n\\le  \\mathcal L^{\\alpha}(x^{k+1},y^{k},z^{k}) + \\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2\n-\\frac{1}{2} \\|\\Delta y^{k+1}\\|_{D}^2 \n&\\\\[.1in]&\n\\le \\mathcal L^{\\alpha}(x^{k},y^{k},z^{k})\n+ \\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2\n-\\frac{1}{2} \\|\\Delta y^{k+1}\\|_{D}^2 -\\frac{1}{2}\\sum_{i=1}^p \\|\\Delta x_i^{k+1}\\|^2_{T_i}.&\n\\end{eqnarray*}\nBy Lemma \\ref{bounddual} for $\\epsilon_0>1$ we have \n\\[\n\\frac{\\epsilon_0}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2\n\\le\\epsilon_0c_4 \\|\\Delta y^{k+1}\\|^2\n+\\epsilon_0c_3 \\|\\Delta y^{k}\\|^2+\n\\epsilon_0c_5 \\|B\\tr \\Delta z^{k}\\|^2\n-\\epsilon_0c_5\\| B\\tr \\Delta z^{k+1}\\|^2.\n\\]\nAdd this to the latter inequality and rearrange to obtain (\\ref{chgLiter}). $\\square$\n\\end{proof}\n\n\\begin{lemma}\\label{thm1}\n%{\\bf (monotonicity/convergence of $\\bar{\\mathcal L}$)}\nSuppose that A1-A4 hold, and \n\\[\n\\bar\\tau =\\min\\{ \\max \\{ q_i,\\lambda_{++}^{\\alpha A_i\\tr A_i-\\epsilon_i I} \\} : i=1,\\dots, p\\}.\n\\]\n If $\\sigma=\\min\\{\\lambda_{\\min}(\\bar D), \\bar\\tau,\\frac{\\epsilon_0-1}{\\alpha\\beta}\\}>0$ where \n $\\bar D$ is defined in (\\ref{quantities}), then\nthe following assertions hold.\n%\n\\begin{itemize}\n\\item [(i)] The sequence $\\{\\bar {\\mathcal L_k}\\}_{k\\ge 1}$ is monotonically decreasing and \n%\n\\begin{eqnarray}\\label{Lbarmon}\n\\bar{\\mathcal L}_{k+1}+\\sigma\n\\big[\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|^2\n+ \\|\\Delta y^{k+1}\\|^2+\n\\|\\Delta z^{k+1}\\|^2\\big]\n\\le \\bar{\\mathcal L}_k.\n\\end{eqnarray}\n%\n%\n\\item [(ii)] If the sequence $\\{{\\bf x}^k, y^k, z^k\\}_{k\\ge 0}$ generated by the PADMM is bounded \nthen $\\bar{\\mathcal L}_k$ is bounded from below for all $k \\in \\mathbb N$  and converges as $k$ approaches infinity.\n%%\n%\\item [(iii)] If $B\\tr B$ is invertible, then $\\{{\\bf x}^k, y^k, z^k\\}_{k\\ge 0}$ is bounded.\n%\\item [(iv)]  If $\\{{\\bf x}^k, y^k, z^k\\}_{k\\ge 0}$ is bounded\n%then \n%$\\lim_{k\\to \\infty} \\|\\Delta x_i^{k}\\|=\\lim_{k\\to \\infty} \\|\\Delta y^{k}\\|=\\lim_{k\\to \\infty} \\|\\Delta z^{k}\\|=0.$\n\\end{itemize}\n\\end{lemma}\n\n\\begin{proof}\n{\\bf (i)} By (\\ref{chgLiter}) and (\\ref{Lbark}) we obtain \n\\[\n\\bar{\\mathcal L}_{k+1}\n+\\frac{1}{2}\\sum_{i=1}^p \\|\\Delta x_i^{k+1}\\|^2_{T_i}\n+\\frac 12\\|\\Delta y^{k+1}\\|^2_{D-2\\epsilon_0(c_3+c_4)}+\n\\displaystyle{\\frac{\\epsilon_0-1}{\\alpha\\beta}}\\big\\|\\Delta z^{k+1}\\big\\|^2\n\\le \\bar{\\mathcal L}_k.\n\\]\nWe have $T_i=Q_i\\succeq q_i I$ where $q_i>0$ or $T_i=\\alpha A_i\\tr A_i-\\epsilon_i I$.\nWe let $\\bar\\tau =\\min\\{ \\max \\{ q_i,\\lambda_{++}^{\\alpha A_i\\tr A_i-\\epsilon_i I} \\} : i=1,\\dots, p\\}$\nand $\\bar D=D-2\\epsilon_0(c_3+c_4)I$. For \n$\\sigma=\\min\\{\\lambda_{\\min}(\\bar D), \\bar\\tau,\\frac{\\epsilon_0-1}{\\alpha\\beta}\\}>0$\nwe arrive at the inequality (\\ref{Lbarmon}), showing monotonicity of $\\{\\bar {\\mathcal L_k}\\}_{k\\ge 1}$.\n%\n\n{\\bf (ii)}  If $\\{{\\bf x}^k, y^k, z^k\\}_{k\\ge 0}$ is a bounded  sequence,\nthen \n$\\|B\\tr\\Delta z^k\\|^2+\n\\|\\Delta y^k\\|^2$ is also bounded, and bounded from below by zero.\nHence we only need to show that \n$\\mathcal L^{\\alpha}({\\bf x}^k,y^k,z^k)$ is bounded. \nBy the boundedness of $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$,\nthe terms  $\\langle z^k, {\\bf Ax}^k+By^k+b\\rangle$ and $\\|{\\bf Ax}^k+By^k+b\\|^2$\nare bounded. By A1 the function $f_i, i=1,\\dots,p$ are coercive,\nsince $\\{{\\bf x}^k\\}_{k\\ge 0}$ is bounded, then $\\{f_i(x_i^k)\\}_{k\\ge 0}$ is bounded for $i=1,\\dots,p$. \nBy A2, the function $h$ is bounded from below.\nThus, $\\{\\mathcal L^{\\alpha}({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$\nis bounded from below.\n%\nSince $\\{\\bar{\\mathcal L}_k\\}_{k\\ge 1}$ is bounded from below and by part (i) it is \nmonotonically decreasing then we conclude that $\\{\\bar{\\mathcal L}_k\\}_{k\\ge 1}$ it converges.  $\\square$\n\n\\end{proof}\n\n\\begin{lemma}\\label{samelimit}\nSuppose that A1-A4 hold and   $({\\bf x}^*,y^*,z^*)\\in\\omega(\\{ ({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0})$,\nthen\n\\[\n\\bar{\\mathcal L}({\\bf x}^*,y^*,z^*, y^*,z^*)\n=\n\\mathcal L^{\\alpha} ({\\bf x}^*,y^*,z^*) \n=\n\\sum_{i=1}^p f_i(x_i^*)+h(y^*).\n\\]\n\n\\end{lemma}\n\n\\begin{proof}\nLet $({\\bf x}^*,y^*,z^*)\\in\\omega(\\{ ({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0})$\nand let the subsequence $\\{({\\bf x}^{k_j},y^{k_j},z^{k_j})\\}_{j\\ge 0}$ of the \nPADMM algorithm converges to \n$({\\bf x}^*,y^*,z^*)$.\nClearly $\\|\\Delta y^{k_j}\\|$ and $\\|B\\tr \\Delta z^{k_j}\\|$ both converge to zero as $j$ goes to infinity.\nThis, together with Lemma\n Lemma \\ref{thm-cluster} and \\ref{thm-cluster0} give\n%\n\\begin{eqnarray*}\n\\lim_{j\\to\\infty} \\bar {\\mathcal L}_{k_j} =\\lim_{j\\to\\infty} \\mathcal L^{\\alpha} (x^{k_j},y^{k_j},z^{k_j})\n&=&\\mathcal L^{\\alpha}(x^*,y^*,z^*)\\\\\n&=&\\sum_{i=1}^pf_i(x_i^*)+h(y^*)\n=\\bar{\\mathcal L}({\\bf x}^*,y^*,z^*, y^*,z^*). \\quad\\quad\\square\n\\end{eqnarray*}\n%\n\\end{proof}\n\n\\begin{lemma}\\label{lma-D}\n{(subgradient bound for $\\bar{\\mathcal L}_k$)}\nLet $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ be a sequence generated by the PADMM algorithm\nand assume that $\\big(\\{d_{x_i}^k\\}_{i=1}^p,d_y^k,d_z^k\\big)\\in\\partial \\mathcal L^{\\alpha}({\\bf x}^k,y^k,z^k)$. Then \n%\n\\[\ns^{k}:=\\Big( \\{s^{k}_{x_i}\\}_{i=1}^p, s^{k}_y, s^{k}_z, s^{k}_{y'},s^{k}_{z'} \\Big)\\in\n\\partial \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1}),\\quad k\\ge 1\n\\]\n%\nwhere\n%\n$s_{x_i}^{k}:=d^{k}_{x_i}, i=1,\\dots, p$,\n$s_y^{k}:=d^{k}_y+2\\epsilon_0c_3 \\Delta y^{k}$,\n$s_z^{k}:=d^{k}_z+2\\epsilon_0c_5BB\\tr \\Delta z^{k}$,\n$s_{y'}^{k}:=-2\\epsilon_0c_3 \\Delta y^{k}$,\n$s_{z'}^{k}:=-2\\epsilon_0c_5BB\\tr \\Delta z^{k}$,\nand $\\epsilon_0>1$, and $c_3$ and $c_5$ are defined in  (\\ref{quantities}).\nMoreover,\n\\begin{eqnarray}\\label{nD}\n|||s^{k}|||\\le \\tilde\\rho \\Big(\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|+\\|\\Delta y^{k}\\|+\\|\\Delta z^{k}\\|\\Big),\n\\end{eqnarray}\nwhere \n$\n\\tilde\\rho:=\\max\\Big\\{\n\\|{\\bf Q}\\|+\\|{\\bf A}\\|^2,\\;\n\\|P\\|+\\alpha \\|{\\bf A}\\|\\|B\\|+4\\epsilon_0c_3, \\;\n\\|{\\bf A}\\|  +\\|B\\|+4\\epsilon_0c_5\\|B\\|^2 + \\frac{1}{\\alpha\\beta}\\Big\\},\n$\nand $\\|{\\bf Q}$ and $\\|{\\bf A}\\|$ are defined in Lemma \\ref{thm-d}.\n\\end{lemma}\n\n\\begin{proof}\nLet $k\\ge 1$ be fixed and $(\\{d_{x_i}^{k}\\}_{i=1}^p, d_y^{k}, d_z^{k})\\in \\partial \\mathcal L^{\\alpha}(x^k,y^k,z^k)$. By taking partial derivatives of $\\bar{\\mathcal L}$ with respect to $x,y,z,y',z'$ and evaluating \nthem at $({\\bf x}^k,y^k,z^k,y^{k-1},z^{k-1})$ we obtain\n%\n\\begin{eqnarray*}\n%s_{x_i}^{k}&:=& \n\\partial_{x_i} \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1})\n&=&\\partial_{x_i} \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k})\n\\ni d_{x_i}^{k},\\; i=1,\\dots,p;\\\\\n%s_y^{k}&:=&\n\\nabla_y  \\bar{\\mathcal L} ({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1})\n&=&\\nabla_y \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k})\n+2\\epsilon_0c_3 \\Delta y^k=\nd_y^{k}+2\\epsilon_0c_3 \\Delta y^k;\\\\\n%s_z^{k}&:=&\n\\nabla_z  \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1})\n&=&\\nabla_z \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k})\n+2\\epsilon_0c_5 BB\\tr \\Delta z^{k}=\nd_z^{k}\n+2\\epsilon_0c_5 BB\\tr\\Delta z^{k};\\\\\n%s_{y'}^{k}&:=&\n\\nabla_{y'}  \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1})\n&=&-2\\epsilon_0c_3\\Delta y^k;\\\\\n%d_{z'}^{k}&:=&\n\\nabla_{z'}  \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1})\n&=&-2\\epsilon_0c_5BB\\tr \\Delta z^k.\n\\end{eqnarray*}\n%\nThese equations follow that \n\\begin{eqnarray*}\ns^k =\n\\left[\n\\begin{array}{c}\ns^k_{x_1}\\\\\n\\vdots\\\\\ns^k_{x_p}\\\\\ns^k_{y} \\\\ \ns^k_{z}\\\\\ns^k_{y'}\\\\\ns^k_{z'}\n\\end{array}\n\\right]\n=\n\\left[\n\\begin{array}{l}\nd^k_{x_1}\\\\\n\\vdots\\\\[.1in]\nd^k_{x_p}\\\\\nd_y^{k}+2\\epsilon_0c_3 \\Delta y^k \\\\ \nd_z^{k}+2\\epsilon_0c_5 BB\\tr\\Delta z^{k}\\\\\n-2\\epsilon_0c_3\\Delta y^k\\\\\n-2\\epsilon_0c_5BB\\tr \\Delta z^k\n\\end{array}\n\\right]\\in \\partial \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1}).\n\\end{eqnarray*} \n%\nBy the triangle inequality we have\n%\n\\begin{eqnarray*}\n&\\|s_y^{k}\\|\\le \\|d^{k}_y\\|+2\\epsilon_0c_3\\|\\Delta y^k\\|,\\quad\\quad\n\\|s_z^{k}\\|\\le\\|d^{k}_z\\|+2\\epsilon_0c_5\\|B\\|^2 \\|\\Delta z^k\\|,&\\\\\n&\\|s_{y'}^{k}\\|\\le 2\\epsilon_0c_3\\|\\Delta y^k\\|,\\quad\\quad\n\\|s_z^{k}\\|\\le 2\\epsilon_0c_5 \\|B\\|^2 \\|\\Delta z^k\\|.&\n\\end{eqnarray*}\n%\nThus\n\\begin{eqnarray*}\n|||s^{k}|||&\\le&  \\sum_{i=1}^p\\|s_{x_i}^{k}\\|+\\|s_y^{k}\\|+\\|s_z^{k}\\|+ \\|s_{y'}^{k}\\|+\\|s_z^{k}\\|\\\\\n&\\le& \\sum_{i=1}^p \\|d^{k}_{x_i}\\|+  \\|d^{k}_y\\|+\\|d^{k}_z\\|+\n4\\epsilon_0c_3\\|\\Delta y^{k}\\|+\n4\\epsilon_0c_5 \\|B\\|^2 \\|\\Delta z^k\\|.\n\\end{eqnarray*}\n%\nBy Lemma \\ref{thm-d}, we have \n%\n\\begin{eqnarray*}\n&|||s^{k}|||\\le\n\\Big(\\|{\\bf Q}\\|+\\|{\\bf A}\\|^2\\Big) \\sum_{i=1}^p\\|\\Delta x_i^{k}\\|\n+\\Big(\\|P\\|+\\alpha \\|{\\bf A}\\|\\|B\\| +4\\epsilon_0c_3 \\Big)      \\|\\Delta y^{k}\\|&\\\\\n&+\\Big(\\|{\\bf A}\\|  +\\|B\\|+4\\epsilon_0c_5\\|B\\|^2+\\frac{1}{\\alpha\\beta}\\Big)  \\|\\Delta z^{k}\\|\n\\le \\tilde\\rho \\Big( \\sum_{i=1}^p\\|\\Delta x_i^{k}\\|+   \\|\\Delta y^{k}\\|+  \\|\\Delta z^{k}\\|\\Big). \\quad \\square&\n\\end{eqnarray*}\n\n\\end{proof}\n\n\\begin{lemma} \\label{bddseq0}\nSuppose that A1-A4 hold and $\\sigma$ defined in Lemma \\ref{thm1} is positive.\nThen the sequence \n$\\{{\\bf x}^k, y^k, z^k\\}_{k\\ge 0}$ generated by the PADMM algorithm (\\ref{padmm}) is bounded,\nand $\n\\lim_{k\\to \\infty} \\|\\Delta x_1^{k}\\|=\\dots=\\lim_{k\\to \\infty} \\|\\Delta x_p^{k}\\|=\n\\lim_{k\\to \\infty} \\|\\Delta y^{k}\\|=\\lim_{k\\to \\infty} \\|\\Delta z^{k}\\|=0.\n$\n\\end{lemma}\n\n\\begin{proof}\nLet $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ be a sequence generated by the PADMM algorithm.\n%\nBy $\\sigma>0$, the sequence $\\{\\bar{\\mathcal L}_k\\}_{k\\ge 1}$ \ndefined in (\\ref{Lbark}) is monotonically decreasing, hence it is upper bounded by \n$\\bar{\\mathcal L}_1=\\bar{\\mathcal L}({\\bf x}^1,y^1,z^1,y^0,z^0)$.\nThis follows that \n%\n\\begin{eqnarray}\\label{ineq10}\n\\begin{array}{c}\n\\sum_{i=1}^pf(x_i^{k})+h(y^{k})\n+\\frac{\\alpha}{2}\\|{\\bf Ax}^{k}+By^{k}\n+\\alpha^{-1}z^{k}+b\\|^2-\\frac{1}{2\\alpha}\\|z^{k}\\|^2\n\\\\[.1in]\n+\\sigma \\sum_{i=1}^p \\|\\Delta x_i^{k}\\|^2\n +\\sigma\\|\\Delta z^{k}\\|^2\n+(\\sigma+\\epsilon_0c_3) \\|\\Delta y^{k}\\|^2\n+\\epsilon_0c_5\\|B\\tr \\Delta z^{k}\\|^2\n\\le \\bar{\\mathcal L}_1\n \\end{array}\n\\end{eqnarray}\n%\nfor all $k\\in\\mathbb Z_+$.\nBy Lemma \\ref{bounddual} (iii), (\\ref{ineq10}) leads to \n%\n\\begin{eqnarray}\\label{thmbdd-eq1}\n\\begin{array}{c}\n\\sum_{i=1}^pf(x_i^{k})+\\frac{\\alpha}{2}\\|{\\bf Ax}^{k}+By^{k}+\\alpha^{-1}z^{k}+b\\|^2\n+r_1\\|\\Delta y^{k}\\|^2\\\\[.1in]\n+ \\sigma\\sum_{i=1}^p\\|\\Delta x^{k}\\|^2\n+ r_2\\|B\\tr \\Delta z^{k}\\|^2\n+\\sigma\\|\\Delta z^{k}\\|^2\n\\le\\bar{\\mathcal L}_1\n%- \\inf_{x,y}\\Big\\{g(x)- c_1 \\Big\\|\\nabla g(x)\\Big\\|^2\\Big\\}\n-\\inf_y\\Big\\{ h(y)-c_1 \\|\\nabla h(y)\\|^2 \\Big\\},\n\\end{array}\n\\end{eqnarray}\n%\nwhere $r_1:= \\epsilon_0c_3+\\sigma-\\|P\\|^2c_1>0$\nand $r_2:=\\epsilon_0c_5-c_6>0$. \nBy A2,  since $h$ is $L_h$ Lipschitz differentiable then for any $k\\ge 1$ \nand $\\delta>0$ we have\n%\n$\nh\\big(y^k-\\delta \\nabla h(y^k)\\big)\\le h(y^k)- \n\\big(\\delta-\\frac{L_h\\delta^2}{2}\\big)\\|\\nabla h(y^k)\\|^2.\n$\n%\nAgain by A2, $h$ is bounded from below then\n%\n\\begin{eqnarray}\\label{hbd}\n-\\infty<\\inf\\{h(y)- \\Big(\\delta-\\frac{L_h\\delta^2}{2}\\Big)\\|\\nabla h(y)\\|^2: \\; y\\in\\mathbb R^q\\}.\n\\end{eqnarray}\n%\nWe choose $\\delta (>0)$ such that \n$\\delta-\\frac{L_h\\delta^2}{2}=c_1$. By (\\ref{hbd}), the right hand side of (\\ref{thmbdd-eq1}) is finite\nhence\n%\n\\begin{eqnarray}\\label{finite}\n\\sum_{i=1}^pf(x_i^{k})+\\|{\\bf Ax}^{k}+By^{k}+\\alpha^{-1}z^{k}+b\\|^2\n+\\|\\Delta y^{k}\\|^2 +\\|\\Delta z^k\\|^2<+\\infty.\n\\end{eqnarray}\n%\nBy A1, since $f_i, i=1,\\dots,p$ are coercive and $\\sum_{i=1}^pf(x_i^{k})<+\\infty$, then the sequence \n$\\{x_i^k\\}_{k\\ge 0}$  is bounded for $i=1,\\dots,p$, and so are \n$\\{{\\bf x}^k\\}_{k\\ge 0}$ and $\\{{\\bf Ax}^k\\}_{k\\ge 0}$.\n%\n%In addition, by (\\ref{finite}) the sequences \n%$\\{Ax^{k}+By^{k}+\\alpha^{-1}z^{k}+c\\}_{k\\ge k_0}$ and $\\{ \\Delta y^{k}\\}_{k\\ge k_0}$ are bounded, \n%then $\\{A\\Delta x^k\\}_{k\\ge k_0}$ and $\\{B\\Delta y^k\\}_{k\\ge k_0}$\n%are bounded which follows that $\\{\\Delta z^k\\}_{k\\ge 0}$ is bounded.\n\nBy the $z$ iterate of PADMM we have\n$\nBy^{k} = \\frac{1}{\\alpha\\beta} \\Delta z^{k} -{\\bf Ax}^{k}-b.\n$\nBy (\\ref{finite}), $\\{\\Delta z^k\\}_{k\\ge 0}$ is bounded.\nBy A3, $B\\tr B$ is invertible, thus $\\{y^k\\}_{k\\ge 0}$ is bounded.\nFinally, by (\\ref{finite}) $\\{{\\bf Ax}^{k}+By^{k}+\\frac{1}{\\alpha}z^{k}\\}_{k\\ge 0}$ is bounded\nand ${\\bf Ax}^k$ and $By^k$ are bounded, so \n$\\{z^k\\}_{k\\ge 0}$ is then bounded.\n%\nSumming up (\\ref{Lbarmon}) for $k=1,\\dots, K-1$, $K\\ge 2$ gives\n%\n\\[\n\\sum_{k=1}^{K}\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|^2\n+\\|\\Delta y^{k}\\|^2\n+\\|\\Delta z^{k}\\|^2\\Big\\}\\le \\frac{1}{\\sigma}(\\bar{\\mathcal L}_1 - \\bar{\\mathcal L}_K) .\n\\]\nSince $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ is bounded, by Lemma \\ref{thm1} (ii) the right hand side is finite for any $K$.\nWe let $K$ approach to infinity to obtain \n\\begin{eqnarray*}\n\\sum_{k=1}^{\\infty}\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|^2 +\\|\\Delta y^{k}\\|^2+\\|\\Delta z^{k}\\|^2\\Big\\}<\n\\frac{1}{\\sigma}(\\bar{\\mathcal L}_1 - \\inf_{k\\ge 1}\\bar{\\mathcal L}_k) <\n+\\infty.\n\\end{eqnarray*}\nThis follows \n$\n\\displaystyle{\\lim_{k\\to \\infty}} \\Big(\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|^2 +\\|\\Delta y^{k}\\|^2+\\|\\Delta z^{k}\\|^2\\Big)=0,\n$\nand consequently \n$\n\\displaystyle{\\lim_{k\\to \\infty}} \\|\\Delta x_1^{k}\\|\n=\\dots\n=\\displaystyle{\\lim_{k\\to \\infty}} \\|\\Delta x_p^{k}\\|\n=\n\\displaystyle{\\lim_{k\\to \\infty}}\\|\\Delta y^{k}\\|\n=\n\\displaystyle{\\lim_{k\\to \\infty}}\\|\\Delta z^{k}\\| =0.\\quad\\square\n$\n\n\n\\end{proof}\n\n\\begin{lemma}\\label{lma-cluster2}\nLet $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ be a sequence generated by the PADMM algorithm (\\ref{padmm}),\nand denote\n\\begin{eqnarray}\\label{omega}\n\\Omega:=\\omega \\Big(\\{({\\bf x}^k,y^k,z^k,y^{k-1},z^{k-1})\\}_{k\\ge 1}\\Big).\n\\end{eqnarray}\nThen the following statements are true\n\\begin{itemize}\n\\item [(i)] $\\Omega$ is nonempty, connected, and compact.\n\\item [(ii)] $\\Omega \\subseteq \\{({\\bf x}^*, y^*, z^*,  y^*, z^*)\n\\in\\mathbb R^n\\times\\mathbb R^q\\times\\mathbb R^m\\times\\mathbb R^q\\times\\mathbb R^m:\n\\;({\\bf x}^*, y^*, z^*)\\in {\\rm crit} (\\mathcal L^{\\alpha})\\big\\}$\n%\n\\item [(iii)] $\\lim_{k\\to\\infty} {\\rm dist} \\big[ ({\\bf x}^k, y^k, z^k, y^{k-1}, x^{k-1}), \\Omega \\big]=0$.\n\\item [(iv)] The sequences $\\{\\bar{\\mathcal L}_k\\}_{k\\ge 1}$,\n $\\{\\bar{\\mathcal L}^{\\alpha}({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$,\n and $\\{\\sum_{i=1}^pf_i(x_i^k)+h(y^k)\\}_{k\\ge 0}$ approaches the same limit on $\\Omega$. \n\\end{itemize}\n\\end{lemma}\n\n\\begin{proof}\nThese results follow from Lemmas \\ref{thm1}, \\ref{samelimit}, and \\ref{bddseq0}. Thus we omit the proof. $\\square$\n\\end{proof}\n\n\\begin{proof}\nBy Lemma \\ref{thm1}, the sequence $\\{\\bar{\\mathcal L}_k\\}_{k\\ge 1}$ is monotonically\n decreasing and converges. Let $\\bar{\\mathcal L}_*:=\\lim_{k\\to\\infty}\\bar{\\mathcal L}_k$, then \nthe error sequence \n$e_k:=\\bar{\\mathcal L}_k-\\bar{\\mathcal L}_*, k\\ge 1$\nis non-negative, monotonically decreasing, and converges to zero. \nWe prove under two cases.\n\n {\\it Case 1:} let assume that \nthere is a $k_1\\in\\mathbb Z_+$ such that $e_{k_1}=0$. \nSince $\\{e_k\\}_{k\\ge 1}$ is monotonically decreasing, \n$e_k=0$ for all $k\\ge k_1$.\nBy Lemma \\ref{thm1}(i) then we have\n\\[\n\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|^2+\\|\\Delta y^{k+1}\\|^2+\\|\\Delta z^{k+1}\\|^2 \\le \n \\frac{1}{\\sigma}(e_k-e_{k+1})=0,\\quad \\forall k\\ge k_1.\n\\]\nHence  as $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ is a bounded sequence\nthen \n\\[\n\\sum_{k=1}^{\\infty}\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|\n+\\|\\Delta y^{k}\\|\n+\\|\\Delta z^{k}\\|\\Big\\}\n=\\sum_{k=1}^{k_1} \\Big\\{\n\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|\n+\\|\\Delta y^{k}\\|\n+\\|\\Delta z^{k}\\|\\Big\\}\n\\] \nis bounded.\n\n{\\it Case 2:}\nwe let  $e_k>0$ for all $k\\ge 1$. \nBy Lemma \\ref{thm1}(i) we have\n%\n\\begin{eqnarray}\\label{s1}\n%|||\\Delta u^{k+1}|||^2=\n\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|^2+\\|\\Delta y^{k+1}\\|^2+\\|\\Delta z^{k+1}\\|^2 \\le \n \\frac{1}{\\sigma}( e_k-e_{k+1}),\\quad \\forall k\\ge 1.\n\\end{eqnarray}\n%\nBy Lemma \\ref{lma-cluster2}, $\\Omega$ is nonempty, compact, and connected,\nand $\\bar{\\mathcal L}$ takes on a constant value $\\bar{\\mathcal L}_*$ on $\\Omega$. \nSince $\\bar{\\mathcal L}_{k}\\searrow\\bar{\\mathcal L}_*$ as $k\\to\\infty$ and $\\bar{\\mathcal{L}}$ satisfies \nthe {K\\L} property then there exists  $ k_1\\ge 1$ such that \n$\\bar{\\mathcal L}_*<\\bar{\\mathcal L}_k<\\bar{\\mathcal L}_*+\\eta$\nfor all $k\\ge k_1$ and some $\\eta>0$.\n%\nMoreover, $\\lim_{k\\to\\infty} {\\rm dist} \\big[ ({\\bf x}^k,y^k,z^k,y^{k-1},x^{k-1}), \\Omega\\big]=0$ by\nLemma \\ref{lma-cluster2}, \nthus for a given $\\epsilon>0$ there exists $k_2\\ge 1$ such that for all $k\\ge k_2$ it holds\n\\[\n{\\rm dist} \\Big[ ({\\bf x}^k,y^k,z^k,y^{k-1},x^{k-1}), \\Omega\\Big]<\\epsilon. \n\\]\nLet  $\\tilde k:=\\max\\{ k_1,k_2,3 \\}$, \nthen $({\\bf x}^k,y^k,z^k,y^{k-1},x^{k-1})\\in\\mathcal S$\n and\n%\n$\n\\psi'(\\bar{\\mathcal L}_k-\\bar{\\mathcal L}_*)\\cdot {\\rm dist}\\Big(0, \n\\partial \\bar{\\mathcal L}_k \\Big)\n\\ge 1$ for all $k\\ge \\tilde k$, or equivalently\n%\n\\begin{eqnarray}\\label{KL}\n\\psi'(e_k)\n\\cdot {\\rm dist}\\Big(0, \n\\partial \\bar{\\mathcal L}_k \\Big)\n \\ge1.\n\\end{eqnarray}\n%\nMultiply both sides of this inequality by $ |||\\Delta u^{k+1}|||^2 $ to get\n%\n\\begin{eqnarray}\\label{eq:001}\n \\psi'(e_{k}) \\cdot {\\rm dist}\\Big(0, \n\\partial \\bar{\\mathcal L}_k \\Big) |||\\Delta u^{k+1}|||^2 \\ge |||\\Delta u^{k+1}|||^2,\n\\end{eqnarray}\nwhere \n$|||\\Delta u^k|||^2=\\sum_{i=1}^p\\|\\Delta x_i^k\\|^2+\\|\\Delta y^k\\|^2+\\|\\Delta z^k\\|^2.$\n%\nBy (\\ref{s1}), (\\ref{eq:001}) leads to \n\\begin{eqnarray}\\label{eq:002}\n \\frac {1}{\\sigma} \\psi'(e_{k}) (e_k-e_{k+1})\\cdot {\\rm dist}\\Big(0, \n\\partial \\bar{\\mathcal L}_k \\Big)  \\ge |||\\Delta u^{k+1}|||^2.\n\\end{eqnarray}\n%\nBy the concavity of $\\psi$, $\\psi(e_{k}) -\\psi(e_{k+1})\\ge  \\psi'(e_{k}) (e_{k} - e_{k+1})$, hence\n%\n\\begin{eqnarray}\\label{eq:003}\n|||\\Delta u^{k+1}|||^2 \\le \n \\frac{1}{\\sigma} \\big(\\psi(e_{k}) -\\psi(e_{k+1})\\big) \n \\cdot {\\rm dist}(0, \\partial \\bar{\\mathcal L}_k ).\n\\end{eqnarray}\nBy the arithmetic mean-geometric mean inequality, for any $\\gamma>0$ we  have\n%\n\\begin{eqnarray}\\label{artith-geo}\n\\sqrt{ \\frac{1}{\\sigma} \\big(\\psi(e_{k}) -\\psi(e_{k+1})\\big) \n \\cdot {\\rm dist}(0, \\partial \\bar{\\mathcal L}_k )}\n \\le \\frac{\\gamma}{2\\sigma}  \\big(\\psi(e_{k}) -\\psi(e_{k+1})\\big) \n+\\frac{1}{2\\gamma}{\\rm dist}(0, \\partial \\bar{\\mathcal L}_k ).\n \\end{eqnarray}\n%\nMoreover,\n\\begin{eqnarray}\\label{eq:004}\n\\frac{1}{\\sqrt{p+2}} \\Big(\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\\Big)\n\\le |||\\Delta u^{k+1}|||.\n\\end{eqnarray}\n%\nThus  (\\ref{eq:003}), (\\ref{artith-geo}), and (\\ref{eq:004}) give rise to\n%\n\\begin{eqnarray}\\label{main}\n\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n\\le \\frac{\\gamma\\sqrt{p+2}}{2\\sigma}  \n\\big(\\psi(e_{k}) -\\psi(e_{k+1})\\big) \n+\\frac{\\sqrt{p+2}}{2\\gamma}{\\rm dist}(0, \\partial \\bar{\\mathcal L}_k ).\n\\end{eqnarray}\n%\nNote that ${\\rm dist}(0, \\partial \\bar{\\mathcal L}_k)=|||s^k|||$, where $s^k\\in\\partial \\bar{\\mathcal L}_k$ \ndefined in Lemma \\ref{lma-D}, hence \n%\n\\begin{eqnarray}\\label{eq:import}\n\\begin{array}{c}\n\\displaystyle{\n\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n}\\\\[.1in]\n\\le \n\\displaystyle{\n\\frac{\\gamma\\sqrt{p+2}}{2\\sigma}  \n\\big(\\psi(e_{k}) -\\psi(e_{k+1})\\big) \n+\n\\frac{\\tilde \\rho\\sqrt{p+2}}{2\\gamma}\n\\Big(\n\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|\n+\\|\\Delta y^{k}\\|\n+\\|\\Delta z^{k}\\|\\Big).\n}\n\\end{array}\n\\end{eqnarray}\n%\nFor any $K\\ge k\\ge \\underline{k}$, the following identity holds\n\\[\n\\sum_{k=\\underline k}^K\\|\\Delta w^{k}\\| =\n\\sum_{k=\\underline k}^{K}\n\\|\\Delta w^{k+1}\\|\n+\\|\\Delta w^{\\underline k}\\|-\\|\\Delta w^{K+1}\\|.\n\\]\nChoose $\\gamma>0$ large enough such that $2\\gamma>\\tilde \\rho\\sqrt{p+2}$.\nLet $\\delta_0=1-\\frac{\\tilde \\rho\\sqrt{p+2}}{2\\gamma}>0$.\n%\nSumming up (\\ref{eq:import}) from $k=\\underline k\\ge \\tilde k $ to $K\\ge \\underline k$ leads to\n\\begin{eqnarray*}\n&\\sum_{k=\\underline k}^K\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\\Big\\}\n\\le\n\\delta_1\n\\big(\\psi(e_{\\underline k}) -\\psi(e_{K+1})\\big) \n&\\\\\n&\n+\\delta_2\n\\Big(\n\\sum_{i=1}^p\\|\\Delta x_i^{\\underline k}\\|\n+\\|\\Delta y^{\\underline k}\\|\n+\\|\\Delta z^{\\underline k}\\|\\Big)\n-\\delta_2\n\\Big(\n\\sum_{i=1}^p\\|\\Delta x_i^{K+1}\\|\n+\\|\\Delta y^{K+1}\\|\n+\\|\\Delta z^{K+1}\\|\\Big),&\n\\end{eqnarray*}\nwhere $\\delta_1:={\\gamma\\sqrt{p+2}}/{2\\sigma\\delta_0} $ \nand \n$\\delta_2= {\\tilde \\rho\\sqrt{p+2}}/{2\\gamma\\delta_0}$.\n%\nRecall that $e_k$ is monotonically decreasing and\n$\\psi'>0$, therefore\n%\n\\begin{eqnarray*}\n&\\sum_{k=\\underline k}^K\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n\\Big\\}\\le  \n\\delta_1\n\\psi(e_{\\underline k})\n+\\delta_2\\big(\n\\sum_{i=1}^p\\|\\Delta x_i^{\\underline k}\\|\n+\\|\\Delta y^{\\underline k}\\|\n+\\|\\Delta z^{\\underline k}\\|\\big).\n&\n\\end{eqnarray*}\n%\nThe right hand side of this inequality is bounded for any $K\\ge \\underline k$, we let $K\\to\\infty$ to obtain\n\\begin{eqnarray}\\label{eq:0005}\n\\sum_{k=\\underline k}^{\\infty}\n\\Big(\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n\\Big)\n\\le \n\\delta_1\n\\psi(e_{\\underline k})\n+\\delta_2\n\\Big(\n\\sum_{i=1}^p\\|\\Delta x_i^{\\underline k}\\|\n+\\|\\Delta y^{\\underline k}\\|\n+\\|\\Delta z^{\\underline k}\\|\\Big).\n\\end{eqnarray}\n%\nThen (\\ref{eq:0005}) together with the fact that $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ is  bounded gives\n%\n\\begin{eqnarray}\\label{eq:000000005}\n\\sum_{k=0}^{\\infty}\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n\\Big\\}\n<+\\infty.\n\\end{eqnarray}\n\n\nFor any $p,q,K\\in\\mathbb Z_+$ where $q\\ge p$ we have \n\\begin{eqnarray*}\n|||u^q-u^p||| =||| \\sum_{k=p}^{q-1} \\Delta u^{k+1}|||\n\\le\n \\sum_{k=p}^{q-1} |||\\Delta u^{k+1}|||\n \\le \n\\sum_{k=1}^{\\infty}|||\\Delta u^{k+1}|||.\n\\end{eqnarray*}\n%\nSince by (\\ref{eq:000000005}) the right hand side is finite \nthen \n\\[\n\\sum_{k=1}^{\\infty}|||\\Delta u^{k+1}|||\n\\le \\sum_{k=0}^{\\infty}\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n\\Big\\}\n<+\\infty.\n\\]\nThis implies that $\\{u^k\\}_{k\\ge 0}=\\{({\\bf x}, y^k,z^k)\\}_{k\\ge 0}$ is a Cauchy sequence. \nBy Lemma \\ref{thm-cluster} it converges to a stationary point.  $\\square$\n \\end{proof}","units":[{"start":21077,"end":21466,"start_line":397,"end_line":406,"raw_tex":"\\begin{lemma}\nLet $\\Phi:\\mathbb R^d\\to\\mathbb R$ be Fr\\'echet differentiable such that \nits gradient is Lipschitz continuous with constant $L_{\\Phi}>0$. Then \nfor every $u,v\\in\\mathbb R^d$ and every $\\xi\\in[u,v]=\\{(1-t)u+tv:\\;t\\in[0,1]\\}$\nit holds \n\\begin{eqnarray}\\label{Lip}\n\\Phi(v)\\le \\Phi(u)+\\langle \\nabla \\Phi(\\xi),v-u\\rangle +\\frac{L_{\\Phi}}{2}\\|v-u\\|^2.\n\\end{eqnarray}\n\n\\end{lemma}","sha256_utf8":"7240c80bfda2ec4073da8ddbb878ddcce69b7ce229e8f8c1cc4966a4ae8c56ae"},{"start":21467,"end":22412,"start_line":407,"end_line":433,"raw_tex":"\\begin{proof}\nLet $u, v\\in\\mathbb R^d$, and $\\xi=(1-t)u+tv$ for $t\\in [0,1]$. Then we have\n%\n\\begin{eqnarray*}\n&\\Phi(v) -\\Phi(u) = \\int_0^1 \\langle \\nabla \\Phi((1-r) u+rv), v-u \\rangle dr &\\\\\n&=  \\int_0^1  \\langle \\nabla \\Phi((1-r) u+rv)-\\nabla \\Phi(\\xi), v-u \\rangle dr\n+\\langle \\nabla \\Phi(\\xi) , v-u\\rangle. &\n\\end{eqnarray*}\n\nSince $\\Phi$ has Lipschitz continuous gradients with constant $L_{\\Phi}>0$, then we have\n\\begin{eqnarray*}\n&|\\Phi(v) -\\Phi(u) - \\langle \\nabla \\Phi(\\xi) , v-u\\rangle|\n=\\Big| \\int_0^1  \\langle \\nabla \\Phi((1-r) u+rv)-\\nabla \\Phi(\\xi), v-u \\rangle dr\\Big | \n&\\\\\n&\n\\le  \\int_0^1  \\Big\\|  \\nabla \\Phi((1-r) u+rv)-\\nabla \\Phi(\\xi)\\Big\\|\\cdot \\Big\\|v-u\\Big \\|dr \n\\le  L_{\\Phi} \\|v-u\\|^2 \\int_0^1 |r-t| dr\n& \\\\ &\n=  L_{\\Phi} \n\\|v-u\\|^2 \\Big( \\int_0^t (-r+t) dr + \\int_t^1 (r-t) dr \\Big)\n& \\\\&\n=  L_{\\Phi} \\Big(\\frac 12 -t(1-t)\\Big) \\|v-u\\|^2 \n\\le \\frac {L_{\\Phi}}{2}\\|v-u\\|^2.\\quad  \\square &\n\\end{eqnarray*}\n\n\n\\end{proof}","sha256_utf8":"0446b4ea17ae77d7fc2fafc1f0412c33f07dff96fea23625b73b663c079e99a5"},{"start":28184,"end":29003,"start_line":545,"end_line":569,"raw_tex":"\\begin{lemma}\\label{thm-d}\n{(Subgradient bound)} Let $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ be a sequence generated by the PADMM\n algorithm (\\ref{padmm}). Then there exists \n$d^{k}:=\\big(\\{d^{k}_{x_i}\\}_{i=1}^p,d^{k}_y,d^{k}_z\\big)\\in\\partial \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k})$ and  a constant $\\rho>0$\nsuch that \n%\n\\begin{eqnarray}\\label{nd}\n|||d^{k}||| &\\le&\n \\rho \\Big(\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|+ \\|\\Delta y^{k}\\|+ \\|\\Delta z^{k}\\|\\Big), \n\\end{eqnarray}\n%\nwhere %\n\\begin{eqnarray}\\label{rho}\n\\rho:=\\max\\Big\\{\n\\|{\\bf Q}\\|+\\|{\\bf A}\\|^2,\\;\\;\\;\n\\|P\\|+\n\\alpha \\|{\\bf A}\\|\\|B\\|\n, \\;\\;\\;\n\\|{\\bf A}\\|  +\\|B\\|+\\frac{1}{\\alpha\\beta}\\Big\\},\n\\end{eqnarray}\n%\n$\\|{\\bf Q}\\|:=\\sum_{i=1}^p\\|Q_i\\|$ and $\\|{\\bf A}\\|:=\\sum_{i=1}^p\\|A_i\\|$,\nand for any sequence $\\{u^k\\}_{k\\ge 0}$,\n$\\Delta u^{k+1}=u^{k+1}-u^k$.\n\\end{lemma}","sha256_utf8":"cf2238b0490aa018b5f4060893e774dfddd2e17748056f72997fb7bfa3bc3c6f"},{"start":29008,"end":32174,"start_line":572,"end_line":664,"raw_tex":"\\begin{proof}\nLet $k\\ge 1$ be fixed. By the optimality condition of the $x_i$ subproblem\nof PADMM algorithm for \n$i=1,\\dots, p$ we have\n%\n\\begin{eqnarray*}\n-A_i\\tr z^{k-1}\n-\\alpha A_i\\tr (A_{<i}x_{<i}^{k}+A_ix_i^{k}+A_{>i}x_{>i}^{k-1}+By^{k-1}+b)\n-Q_i\\Delta x_i^{k}\n\\in\\partial f_i(x_i^{k}).\n\\end{eqnarray*}\n%\nTaking partial differential of $\\mathcal L^{\\alpha}$ with respect to $x_i$\nand evaluating the result at the point $({\\bf x}^{k},y^{k},z^{k})$ yields\n\\[\n\\partial_{x_i} \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k})=\n\\partial f_i(x_i^{k})+A_i\\tr z^{k}+\n\\alpha A_i\\tr \\big({\\bf Ax}^{k}+By^{k}+ b\\big).\n\\]\n%\nTherefore \n\\begin{eqnarray}\\label{dxinL}\n\\begin{array}{lll}\nd_{x_i}^{k}:=\nA_i\\tr \\Delta z^{k}+\n\\alpha A_i\\tr A_{>i}\\Delta x_{>i}^{k}\n+\\alpha A_i\\tr B\\Delta y^{k}\n-Q_i\\Delta x_i^{k}\n\\in \\partial_{x_i} \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k}).\n\\end{array}\n \\end{eqnarray}\n%\nThe optimality criterion of $y$ subproblem gives\n\\[\n\\nabla h(y^{k})=-B\\tr z^{k-1}- \\alpha B\\tr ({\\bf A x}^{k}+By^{k}+b) \n- {P} \\Delta y^{k}.\n\\]\nBy taking partial differential of $\\mathcal L^{\\alpha}$ with respect to $y$\nand evaluating it at $({\\bf x}^{k},y^{k},z^{k}) $ we obtain\n\\[\n\\nabla_y \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k}) =\n\\nabla h(y^{k})+B\\tr z^{k}+\\alpha B\\tr \\big({\\bf Ax}^{k}+By^{k}+b\\big).\n\\]\nThe last two equations follow that\n\\begin{eqnarray}\\label{dyinL}\nd_y^{k}:=B\\tr \\Delta z^{k}- P \\Delta y^{k}\n= \\nabla_y \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k}).\n\\end{eqnarray}\n%\nLastly, by the $z$ subproblem we have ${\\bf Ax}^{k}+By^{k}+b=\\frac{1}{\\alpha\\beta}\n \\Delta z^k$ hence \n\\begin{eqnarray}\\label{dzinL}\nd_z^{k}:=\\frac{1}{\\alpha\\beta}\n \\Delta z^{k}= \\nabla_z \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k}).\n\\end{eqnarray}\nHence, by (\\ref{dxinL}), (\\ref{dyinL}), , and (\\ref{dzinL}), we \n have \n$\nd^{k}:=\\big(\\{d^{k}_{x_i}\\}_{i=1}^p,d^{k}_y,d^{k}_z\\big)\\in\\partial \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k}).\n$\n%\nBy the norm property we have $\\|d_z^{k}\\|=\\frac{1}{\\alpha\\beta} \\|\\Delta z^{k}\\|$ \nand by the  triangle inequality from (\\ref{dxinL}) and (\\ref{dyinL}) we obtain \n%\n\\begin{eqnarray*} \\label{ndx}\n&\\|d_{x_i}^{k}\\|\\le\n\\|A_i\\|\\|\\Delta z^{k}\\|+\\alpha \\|A_i\\|\\Big({\\sum_{j=i+1}^p}\\|A_j\\|\\|\\Delta x_j^{k}\\|\\Big)\n+\\alpha\\|A_i\\|\\|B\\|\\|\\Delta y^{k}\\|+\\|Q_i\\|\\|\\Delta x_i^{k}\\|, &\\\\\n%\n&\\|d_y^{k}\\|\\le\\|B\\| \\|\\Delta z^{k}\\| +\\|P\\| \\|\\Delta y^{k}\\|. &\n%\\|d_z^{k}\\|=\\frac{1}{\\alpha\\beta} \\|\\Delta z^{k}\\|.\n\\end{eqnarray*}\n%\nHence we get\n%\n\\begin{eqnarray*}\n&|||d^{k}||| \\le  \\sum_{i=1}^p\\|d_{x_i}^{k}\\|+ \\|d_y^{k}\\|+\\|d_z^{k}\\|\n\\le \\sum_{i=1}^p \\|Q_i\\|\\|\\Delta x_i^{k}\\|\n+ \\sum_{i=2}^p \\|A_i\\| \\|\\Delta x_i^{k}\\|\\sum_{j=1}^{i-1}\\|A_j\\| \n& \\\\&\n+\\Big(\\|P\\|+\\alpha\\|B\\|\\sum_{i=1}^p \\|A_i\\|  \\Big)      \\|\\Delta y^{k}\\|\n+\\Big( \\sum_{i=1}^p \\|A_i\\|  +\\|B\\|+\\frac{1}{\\alpha\\beta}\\Big)       \\|\\Delta z^{k}\\|\n&\\\\&\n\\le \n\\Big(\\|{\\bf Q}\\|+\\|{\\bf A}\\|^2\\Big) \\sum_{i=1}^p\\|\\Delta x_i^{k}\\|\n+\\Big(\\|P\\|+\\alpha \\|{\\bf A}\\| \\|B\\| \\Big)      \\|\\Delta y^{k}\\|\n+\\Big(\\|{\\bf A}\\|  +\\|B\\|+\\frac{1}{\\alpha\\beta}\\Big)  \\|\\Delta z^{k}\\|\n&\\\\&\n\\le \\rho \\Big( \\sum_{i=1}^p\\|\\Delta x_i^{k}\\|+   \\|\\Delta y^{k}\\|+  \\|\\Delta z^{k}\\|\\Big).\n\\quad\\square &\n\\end{eqnarray*}\n%\n\\end{proof}","sha256_utf8":"8de583d2e75ad98a019f9ed85e9c94f70d31f11e504d314a89718d78453d3549"},{"start":32230,"end":32647,"start_line":667,"end_line":674,"raw_tex":"\\begin{lemma}\\label{thm-cluster}\n{(Limiting continuity)} \nSuppose that A1-A2  hold. \nIf $({\\bf x}^*, y^*, z^*)$ is the limit point of $\\{({\\bf x}^{k_j},y^{k_j},z^{k_j})\\}_{j\\ge 0}$, a subsequence of \n$\\{{\\bf x}^k, y^k, z^k\\}_{k\\ge 0}$ generated by the PADMM algorithm (\\ref{padmm}). Then \n$ \\mathcal L^{\\alpha} ({\\bf x}^*, y^*, z^*)=\\lim_{j\\to\\infty} \\mathcal L^{\\alpha} ({\\bf x}^{k_j},y^{k_j},z^{k_j}).\n$\n\\end{lemma}","sha256_utf8":"fe580e61d696d6f56439b2eb6a45dffa831d461134f291ef6327b7eae7f87d91"},{"start":32648,"end":34746,"start_line":675,"end_line":727,"raw_tex":"\\begin{proof}\nLet $\\{({\\bf x}^{k_j},y^{k_j},z^{k_j})\\}_{j\\ge 0}$ be a subsequence of the sequence \n$\\{({\\bf x}^{k},y^{k},z^{k})\\}_{j\\ge 0}$\ngenerated by the PADMM algorithm such that\n$\\lim_{j\\to\\infty} ({\\bf x}^{k_j},y^{k_j},z^{k_j})=({\\bf x}^*, y^*, z^*).$\nBy A1, for $i=1,\\dots,p$ the functions $f_i$ are lower semicontinuous hence\n\n\\begin{eqnarray}\\label{eq:flsc0}\nf_i(x_i^*)\\le \\lim\\inf_{j\\to\\infty} f(x_i^{k_j}).\n\\end{eqnarray}\n%\nBy the $x_i$-subproblem of the PADMM algorithm for any $v\\in\\mathbb R^{n_i}$ we have \n\\[\n\\mathcal L^{\\alpha} (x_{<i}^{k_j+1},x_i^{k_j+1}, x^{k_j}_{>i}, y^{k_j},z^{k_j})+\n\\frac{1}{2}\\|\\Delta x_i^{k_j+1}\\|_{Q_i}^2\n\\le \\mathcal L^{\\alpha} (x_{<i}^{k_j+1},v, x^{k_j}_{>i}, y^{k_j},z^{k_j})\n+\\frac{1}{2}\\|v- x_i^{k_j}\\|_{Q_i}^2.\n\\]\nWe choose $v=x_i^*$ and exploit the cosine rule\n$\\|a_1+c\\|^2-\\|a_2+c\\|^2=\\|a_2-a_1\\|^2+2\\langle a_2+c, a_2-a_1\\rangle$\nwith \n$a_1=A_ix_i^*$, $a_2=A_ix_i^{k_j+1}$ \nand $c=A_{<i}x_{<i}^{k_j+1}+A_{>i}x^{k_j}_{>i}+By^{k_j}+b$ to obtain\n %\n\\begin{eqnarray*}\n&f_i(x_i^{k_j+1})- f_i(x_i^*) +\n\\langle A_i\\tr z^{k_j},x^{k_j+1}_i-x_i^*\\rangle+\\|\\Delta x_i^{k_j+1}\\|^2_{Q_i} \n\\le \n\\frac{\\alpha}{2}\\|A_i (x_i^{k_j+1}-x_i^*)\\|^2&\n\\\\\n&+\\alpha \\Big\\langle A_i (x_i^*-x_i^{k_j+1}), A_{<i}x^{k_j+1}_{<i}+A_ix_i^{k_j+1}+A_{>i}x_{>i}^{k_j}+By^{k_j}+b\\Big\\rangle.\n&\n\\end{eqnarray*}\n%\nSince $\\{({\\bf x}^{k_j},y^{k_j},z^{k_j})\\}_{j\\ge 0}$  is a convergent sequence, \n it is bounded and the difference of consequtive terms approaches  zero. \nThus the latter inequality reduces to \n$\\displaystyle{\\lim\\sup_{j\\to \\infty}} f_i(x_i^{k_j+1})-f_i(x_i^*)\\le 0$ and \nin view of (\\ref{eq:flsc0}) we then have\n$\n\\lim_{j\\to \\infty} f_i(x_i^{k_j})=f_i(x_i^*),\n$\nwhich follows that\n$\n\\lim_{j\\to \\infty} \\sum_{i=1}^p f_i(x_i^{k_j})=\\sum_{i=1}^p f_i(x_i^*).\n$\n%\nBy A2, $h(y)$ is smooth hence $\\lim_{j\\to\\infty} h(y^{k_j})= h(y^*)$.\nClearly, ${\\bf Ax}^{k_j}+By^{k_j}+b\\to {\\bf Ax}^{*}+By^{*}+b$\nas $j\\to\\infty$.\nThus $\\displaystyle{\\lim_{j\\to\\infty}} \\mathcal L^{\\alpha} ({\\bf x}^{k_j},y^{k_j},z^{k_j})\n= \\mathcal L^{\\alpha} ({\\bf x}^*, y^*, z^*).$ $\\quad\\square$\n\\end{proof}","sha256_utf8":"89d6dceb5102678cd45cc38524a91043f328f0b19808ec2ca296ee0e0344f895"},{"start":34797,"end":35449,"start_line":731,"end_line":750,"raw_tex":"\\begin{lemma}\\label{thm-cluster0}\n{(Limit point is critical point)} The set of limit points of the sequence\n$\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ generated by the PADMM algorithm (\\ref{padmm}),\ndenoted \n$\\omega\\big(\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}\\big)$, satisfies\n\\[\n\\omega\\big( \\{ ({\\bf x}^k,y^k,z^k) \\}_{k\\ge 0} \\big) \\subseteq {\\rm crit} \\;\\mathcal L^{\\alpha}\n:=\n\\left\\{\n({\\bf x}^*,y^*,z^*):\n\\begin{array}{l}\n 0\\in\\partial f_i(x_i^*)+A_i\\tr z^*,\\; i=1,\\dots, p\\\\\n\\nabla h(y^*)+ B\\tr z^*=0,\\\\\n{\\bf Ax}^*+By^*+b=0\n\\end{array}\n\\right\\},\n\\]\nwhere ${\\rm crit}\\;\\mathcal L^{\\alpha}$ denotes the set of critical points of $\\mathcal L^{\\alpha}$.\n%\n\\end{lemma}","sha256_utf8":"f11b535d6788061e7fcd4bab0a52e081b95c60e2c9747300ed8263e80b61f6f0"},{"start":35451,"end":36410,"start_line":752,"end_line":765,"raw_tex":"\\begin{proof}\nLet $(\\hat {\\bf x}, \\hat y, \\hat z)\\in \\omega\\big( \\{ ({\\bf x}^k,y^k,z^k) \\}_{k\\ge 0} \\big)$,\nhence there exists a subsequence $\\{({\\bf x}^{k_j},y^{k_j},z^{k_j})\\}_{j\\ge 0}$\nof $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ such that \n$\\lim_{j\\to\\infty} ({\\bf x}^{k_j},y^{k_j},z^{k_j})=(\\hat {\\bf x}, \\hat y, \\hat z).$\nFor the sequence $\\{d^k\\}_{k\\ge 0}$, we have $d^{k_j}\\in\\partial \\mathcal L^{\\alpha}(x^{k_j}, y^{k_j}, z^{k_j})$\nfor $j\\ge 1$ and $d^{k_j}\\to 0$ as $j\\to \\infty$ while $({\\bf x}^{k_j}, y^{k_j}, z^{k_j})\\to (\\hat {\\bf x}, \\hat y,\\hat z)$\nand $\\displaystyle{\\lim_{j\\to\\infty}} \\mathcal L^{\\alpha} ({\\bf x}^{k_j},y^{k_j},z^{k_j})\n= \\mathcal L^{\\alpha} (\\hat {\\bf x}, \\hat y, \\hat z).$\n By the closeness criterion of the limiting sub-differential \nwe then have $0\\in \\partial \\mathcal L^{\\alpha}(\\hat{\\bf x}, \\hat{y}, \\hat {z})$,\nand equivalently, $(\\hat{\\bf x}, \\hat{y}, \\hat {z})\\in {\\rm crit} (\\mathcal L^{\\alpha})$.\n $\\square$\n\\end{proof}","sha256_utf8":"5668ad3aa04cca0c545a2ad2cc54328f4c3235f570fe89e7e81980b87d127005"},{"start":36468,"end":37407,"start_line":768,"end_line":798,"raw_tex":"\\begin{lemma}{(bound dual)}\\label{bounddual}\nSuppose that A2 and A3 hold.\n%\\[\n%\\rho(\\beta):=1-|1-\\beta|,\n%\\quad \n%c_1:=  \\frac{1}{\\alpha\\rho(\\beta)\\lambda_{++}^{B\\tr B}},\n%\\quad\n%c_2:=  \\frac{2\\beta}{\\alpha\\rho(\\beta)^2\\lambda_{++}^{B\\tr B}}.\n%\\]\n%where  $\\lambda_{++}^{B\\tr B}$ be the smallest \n%strictly-positive eigenvalue of $B\\tr B$.\n%\nThen for all $k\\in\\mathbb Z_+$ the following assertions are true.\n\\begin{itemize}\n%\n\\item [(i)] $B\\tr z^{k}=\\beta w^{k}+(1-\\beta) B\\tr z^{k-1},$ where \n$w^{k}=-P\\Delta y^{k}-\\nabla h(y^{k}),$\n%\n\\item [(ii)]\n$\\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2\n\\le c_4 \\|\\Delta y^{k+1}\\|^2\n+c_3 \\|\\Delta y^{k}\\|^2+c_5 \\|B\\tr \\Delta z^{k}\\|^2\n-c_5\\| B\\tr \\Delta z^{k+1}\\|^2$,\n%\n\\item [(iii)]\n$\\frac{1}{2\\alpha}\\|z^{k+1}\\|^2\\le c_1  \\|P\\|^2 \\|\\Delta y^{k+1}\\|^2\n+ c_1 \\|\\nabla h(y^{k+1})\\|^2\n+c_6\\|B\\tr \\Delta z^{k+1}\\|^2$,\n\\end{itemize}\nwhere $c_i$, $i=2,\\dots,6$ are defined in (\\ref{quantities}).\n\\end{lemma}","sha256_utf8":"22038a1c0b81d7f63c120ee31d02452d4bd06c2c74376ffe9c31506cbf1b3d8a"},{"start":37409,"end":41498,"start_line":800,"end_line":915,"raw_tex":"\\begin{proof}\n{\\bf (i)} \nBy the the optimality condition of $y$-subproblem we have\n\\[\n\\alpha B\\tr ({\\bf Ax}^{k}+By^{k}+b)+B\\tr z^{k-1}-w^k=0\n\\]\nwhich together with the $z$-iterate ${\\bf Ax}^{k}+By^{k}+b=\\frac{1}{\\alpha\\beta}(z^k-z^{k-1})$ we obtain the result.\n\n\n{\\bf (ii)} By A3 and the $z$ iterate of the PADMM algorithm we have $\\Delta z^{k+1}\\in {\\rm Im}(B)$, thus \n $\\lambda_{++}^{B\\tr B}\\|\\Delta z^{k+1}\\|^2\\le \\|B\\tr \\Delta z^{k+1}\\|^2$.\nPart (i) follows \n$B\\tr \\Delta z^{k+1}=\\beta\\Delta w^{k+1}+(1-\\beta) B\\tr \\Delta z^{k}$ and \nequivalently \n%\n\\begin{eqnarray*}\nB\\tr \\Delta z^{k+1}=\\rho(\\beta)\n\\Big(\\frac{\\beta\\Delta w^{k+1}}{\\rho(\\beta)} \\Big)\n+|1-\\beta| \\Big({\\rm sign}(1-\\beta) B\\tr \\Delta z^k\\Big),\n\\end{eqnarray*}\nwhere ${\\rm sign} (\\lambda)=1$ if $\\lambda\\ge 0$ and ${\\rm sign} (\\lambda)=-1$ if $\\lambda<0$.\nBy $\\rho(\\beta)+|1-\\beta|=1$ and the convexity of $\\|\\cdot\\|^2$ \nwe have\n%\n\\begin{eqnarray}\\label{gg}\n\\|B\\tr \\Delta z^{k+1}\\|^2\n\\le  \\frac{\\beta^2}{\\rho(\\beta)}  \\|\\Delta w^{k+1}\\|^2\n+|1-\\beta| \\|B\\tr \\Delta z^{k}\\|^2.%-|1-\\beta| \\Big\\| B^*\\Delta z^{k+1}\\Big\\|^2.\n\\end{eqnarray}\nThis also follows that \n%\n\\begin{eqnarray*}\\label{gg}\n\\rho(\\beta)\\lambda_{++}^{B\\tr B} \\|\\Delta z^{k+1}\\|^2&\\le& \\rho(\\beta)\\|B\\tr \\Delta z^{k+1}\\|^2\\\\\n%\n&\\le&  \\frac{\\beta^2}{\\rho(\\beta)}  \\|\\Delta w^{k+1}\\|^2\n+|1-\\beta| \\|B\\tr \\Delta z^{k}\\|^2-|1-\\beta| \\| B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray*}\n%\nRearrange this to obtain\n%\n\\begin{eqnarray}\\label{gg}\n\\|\\Delta z^{k+1}\\|^2\\le\n \\frac{\\beta^2}{\\rho(\\beta)^2\\lambda_{++}^{B\\tr B} }  \\|\\Delta w^{k+1}\\|^2\n+\\frac{|1-\\beta|}{\\rho(\\beta)\\lambda_{++}^{B\\tr B} } \\|B\\tr \\Delta z^{k}\\|^2\n-\\frac{|1-\\beta|}{\\rho(\\beta)\\lambda_{++}^{B\\tr B} } \\| B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray}\n%\nBy the optimality condition of $y$ subproblem we have \n\\[\n\\Delta w^{k+1}=-P\\Delta y^{k+1}+P\\Delta y^k+\\nabla h(y^{k})-\\nabla h(y^{k+1}).\n\\]\nBy $\\|a+b\\|^2\\le 2\\|a\\|^2+2\\|b\\|^2$ and the fact that $\\nabla h$ has $L_h$ Lipschitz gradients\nwe obtain\n\\[\n\\|\\Delta w^{k+1}\\|^2\\le 2(\\|P\\|+L_h)^2\\|\\Delta y^{k+1}\\|^2+2\\|P\\|^2 \\|\\Delta y^k\\|^2.\n\\]\nExploit this in (\\ref{gg}) to get\n\\begin{eqnarray*}\\label{gg0}\n\\|\\Delta z^{k+1}\\|^2&\\le&\n \\frac{2\\beta^2 (\\|P\\|+L_h)^2}{\\rho(\\beta)^2\\lambda_{++}^{B\\tr B} }          \\|\\Delta y^{k+1}\\|^2\n + \\frac{2\\beta^2\\|P\\|^2}{\\rho(\\beta)^2\\lambda_{++}^{B\\tr B} }    \\|\\Delta y^k\\|^2\\\\\n &&+\\frac{|1-\\beta|}{\\rho(\\beta)\\lambda_{++}^{B\\tr B} } \\|B\\tr \\Delta z^{k}\\|^2\n-\\frac{|1-\\beta|}{\\rho(\\beta)\\lambda_{++}^{B\\tr B} } \\| B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray*}\nWe divide both sides by $\\alpha\\beta$ to get \n%\n\\begin{eqnarray*}\n\\frac{1}{\\alpha\\beta}\\Big\\|\\Delta z^{k+1}\\Big\\|^2\n\\le c_4 \\Big\\|\\Delta y^{k+1}\\Big\\|^2\n+c_3 \\|\\Delta y^{k}\\|^2+c_5 \\Big\\|B\\tr \\Delta z^{k}\\Big\\|^2\n-c_5 \\Big\\| B\\tr \\Delta z^{k+1}\\Big\\|^2.\n\\end{eqnarray*}\n\n{\\bf (iii).} By part (i) we have\n%\n$\\beta B\\tr z^{k+1}=\\beta w^{k+1}+(1-\\beta) B\\tr (z^k-z^{k+1})$ \nwhich can be written equivalently as follows\n%\n\\begin{eqnarray*}\n\\beta B\\tr z^{k+1}= \\rho(\\beta)\\cdot \\frac{\\beta w^{k+1}}{\\rho(\\beta)}\n+|1-\\beta|\\cdot {\\rm sign}(1-\\beta) B\\tr (z^k-z^{k+1}).\n\\end{eqnarray*}\nBy $\\rho(\\beta)+|1-\\beta|=1$ and the convexity of $\\|\\cdot\\|^2$\nwe get\n%\n\\begin{eqnarray*}\n\\beta^2\\lambda_{++}^{B\\tr B} \\|z^{k+1}\\|^2\n\\le   \\beta^2 \\|B\\tr z^{k+1}\\|^2\n\\le\n\\rho(\\beta)\\cdot \\Big\\| \\frac{\\beta w^{k+1}}{\\rho(\\beta)}\\Big\\|^2\n+|1-\\beta| \\|B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray*}\n%\nSimplify and rearrange to get\n%\n\\begin{eqnarray*}\n\\|z^{k+1}\\|^2\n\\le \n\\frac{1}{\\rho(\\beta)\\lambda_{++}^{B\\tr B}}      \\|w^{k+1}\\|^2\n+\\frac{|1-\\beta|}{\\beta^2 \\lambda_{++}^{B\\tr B}      } \\|B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray*}\n%\nExploit $\\|w^{k+1}\\|^2\\le 2 \\|P\\|^2 \\|\\Delta y^{k+1}\\|^2+2\\|\\nabla h(y^{k+1})\\|^2$\nto obtain \n%\n\\begin{eqnarray*}\n\\|z^{k+1}\\|^2\n\\le\n\\frac{2\\|P\\|^2}{\\rho(\\beta)\\lambda_{++}^{B\\tr B}}  \\|\\Delta y^{k+1}\\|^2\n+ \\frac{2}{\\rho(\\beta)\\lambda_{++}^{B\\tr B}} \\|\\nabla h(y^{k+1})\\|^2\n+\\frac{|1-\\beta|}{\\beta^2\\lambda_{++}^{B\\tr B}} \\|B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray*}\n%\nDivide both sides by $2\\alpha$ to obtain the result.\nThis concludes the proof.  $\\square$\n\\end{proof}","sha256_utf8":"143f56ab5161b89e9b90baf3c7d009a1d9048c25e90e0fc5943de10cee899318"},{"start":41653,"end":42517,"start_line":920,"end_line":936,"raw_tex":"\\begin{lemma}\\label{chgL}\n{(change of $\\mathcal L^{\\alpha}$ during each variable update)} \nSuppose that A1-A4 hold. The iterates in PADMM algorithm (\\ref{padmm}) satisfy\n%\n\\begin{itemize}\n\\item [(i)] $\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^k, x^{k}_{>i},y^k,z^k)-\n\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^{k+1}, x^{k}_{>i},y^k,z^k)\\ge \\frac{1}{2} \\|\\Delta x_i^{k+1}\\|_{T_i}^2$ where\n$T_i=Q_i$ or $T_i=\\alpha A_i\\tr A_i-\\epsilon_i I$ where $\\epsilon_i >0$ defined in A4 for $i=1,\\dots, p$.\n%\n\\item [(ii)]\n$\\mathcal L^{\\alpha}({\\bf x}^{k+1},y^k, z^k)-\\mathcal L^{\\alpha}({\\bf x}^{k+1},y^{k+1},z^{k})\n\\ge \\frac{1}{2} \\|\\Delta y^{k+1}\\|_D^2$, where $D$ is defined in (\\ref{quantities}).\n\\item [(iii)] $\\mathcal L^{\\alpha}({\\bf x}^{k+1},y^{k+1}, z^k)-\n\\mathcal L^{\\alpha}({\\bf x}^{k+1},y^{k+1},z^{k+1})=-\\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2.$\n%\n\\end{itemize}\n\\end{lemma}","sha256_utf8":"442a549970f364763fafac394fc26be1db1c8ff14c644c1fe418cc2956e17a16"},{"start":42519,"end":45245,"start_line":938,"end_line":1016,"raw_tex":"\\begin{proof}\n{\\bf (i)} Let $k\\in\\mathbb Z_+$ and $i\\in\\{1,\\dots,p\\}$ be fixed.\nBy the $x_i$ subproblem we have\n\\[\n\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^{k}, x^{k}_{>i},y^k,z^k)-\n\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^{k+1}, x^{k}_{>i},y^k,z^k)\n\\ge \\|\\Delta x_i^{k+1}\\|_{Q_i}^2.\n\\]\n%\nOn the other hand, by the optimality condition of $x_i$-sub-minimization problem we have \n$ \\bar{d_i}\\in\\partial f_i(x_i^{k+1})$\nwhere \n\\[\n\\bar{d_i}:=-A_i \\tr z^k-\n\\alpha A_i\\tr \\big(A_{<i} x_{<i}^{k+1}+A_ix_i^{k+1}+A_{>i}x_{>i}^k+By^k+b\n\\big).\n\\]\nBy this we then have\n\\begin{eqnarray*}\n&\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^k, x^{k}_{>i},y^k,z^k)-\n\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^{k+1}, x^{k}_{>i},y^k,z^k)& \\\\[.1in]\n&=f_i(x_i^k)-f_i(x_i^{k+1})\n+\\frac{\\alpha}{2}\\|A_i\\Delta x_i^{k+1}\\|^2-\\langle \\bar{d_i}, \\Delta x_i^{k+1}\\rangle&\n\\\\[.1in]\n&\\ge \\frac{\\alpha}{2}\\|A_i\\Delta x_i^{k+1}\\|^2\n- \\frac{\\epsilon_i}{2}\\|\\Delta x_i^{k+1}\\|^2\n=\\frac 12 \\|\\Delta x_i^{k+1}\\|_{\\alpha A_i\\tr A_i-\\epsilon_iI}^2.&\n\\end{eqnarray*}\n\n%%%%\n{\\bf (ii)}\nLet $k\\ge 0$ be fixed. From the optimality condition of $y$ minimization problem \nwe have\n\\begin{eqnarray*}\n\\nabla h(y^{k+1})+\\alpha B\\tr ({\\bf Ax}^{k+1}+By^{k+1}+b)\n+B\\tr z^k+P \\Delta y^{k+1}=0.\n\\end{eqnarray*}\nMultiply this equation by $\\Delta y^{k+1}$ and rearrange to obtain\n\\begin{eqnarray}\\label{eq:2}\n- \\alpha\\big\\langle B\\tr ({\\bf Ax}^{k+1}+ By^{k+1}+b),\\Delta y^{k+1}\\big\\rangle \n-\\langle B\\tr z^k, \\Delta y^{k+1}\\rangle\n=\\langle \\nabla h(y^{k+1}),\\Delta y^{k+1}\\rangle +\\|\\Delta y^{k+1}\\|_P^2.\n\\end{eqnarray}\nBy this and the fact that $h(y)$ is $L_h$ Lipschitz continuous we then get \n%\n\\begin{eqnarray*}\n&\\mathcal L^{\\alpha}(x^{k+1},y^k,z^k)-\\mathcal L^{\\alpha}(x^{k+1},y^{k+1},z^{k})&\n\\\\[.1in]\n&= h(y^k)-h(y^{k+1})\n+\\frac{\\alpha}{2}\\|{\\bf Ax}^{k+1}+By^k+b\\|^2\n-\\frac{\\alpha}{2}\\|{\\bf Ax}^{k+1}+By^{k+1}+b\\|^2\n-\\langle B\\tr z^k, \\Delta y^{k+1} \\rangle&\n\\\\[.1in]\n &=h(y^k)-h(y^{k+1})\n+\\frac{\\alpha}{2}\\|B\\Delta y^{k+1}\\|^2\n-\\alpha \\langle \\Delta y^{k+1}, B\\tr ({\\bf Ax}^{k+1}+By^{k+1}+b)\\rangle \n-\\langle B\\tr z^k, \\Delta y^{k+1} \\rangle&\n\\\\[.1in]\n&=h(y^k)-h(y^{k+1})\n+\\langle \\nabla h(y^{k+1}),\\Delta y^{k+1}\\rangle +\\|\\Delta y^{k+1}\\|_P^2\n+\\frac{\\alpha}{2}\\|B\\Delta y^{k+1}\\|^2\n&\\\\[.1in]\n&\\ge\\|\\Delta y^{k+1}\\|_P^2\n+\\frac{\\alpha}{2}\\|B\\Delta y^{k+1}\\|^2\n-\\frac{L_h}{2}\\|\\Delta y^{k+1}\\|^2 = \n\\frac{1}{2} \\|\\Delta y^{k+1}\\|_{D}^2.&\n\\end{eqnarray*}\n%\n%\n{\\bf (iii)} By the $z$ iterate of the algorithm we obtain\n\\begin{eqnarray*}\n&\\mathcal L^{\\alpha}({\\bf x}^{k+1},y^{k+1}, z^k)-\n\\mathcal L^{\\alpha}({\\bf x}^{k+1},y^{k+1},z^{k+1})=\n\\langle z^k-z^{k+1}, {\\bf Ax}^{k+1}+By^{k+1}+b\\rangle\n=-\\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2.&\n\\end{eqnarray*}\n%\nThat completes the proof. $\\square$\n\\end{proof}","sha256_utf8":"d5f49798c3bafb67d1b4ce2727309c7b7a2e205e570b410bf91f0021ec4362e6"},{"start":45292,"end":46070,"start_line":1020,"end_line":1039,"raw_tex":"\\begin{lemma}\\label{lem:chgLiter}\n{(change of $\\mathcal L^{\\alpha}$ after each iteration)} \nWe assume that A1-A4 hold.\nThe iterates in PADMM algorithm (\\ref{padmm}) satisfy\n%\n\\begin{eqnarray}\\label{chgLiter}\n\\begin{array}{c}\n\\mathcal L^{\\alpha}\\big({\\bf x}^{k+1},y^{k+1},z^{k+1}\\big)\n+\\frac{1}{2}\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|_{T_i}^2\n+ \\frac{1}{2} \\|\\Delta y^{k+1}\\|_{D-2\\epsilon_0c_4 I_m}^2\\\\[.1in]\n+\\displaystyle{\\frac{\\epsilon_0-1}{\\alpha\\beta}}\\big\\|\\Delta z^{k+1}\\big\\|^2\n+\\epsilon_0c_5 \\big\\|B\\tr \\Delta z^{k+1}\\big\\|^2\n\\le  \\mathcal L^{\\alpha}\\big(x^{k},y^k,z^k\\big)+\n\\epsilon_0c_5 \\big\\|B\\tr \\Delta z^k\\big\\|^2\n+\\epsilon_0c_3 \\big\\|\\Delta y^k\\big\\|^2,\n\\end{array}\n\\end{eqnarray}\nwhere $\\epsilon_0>1$, and $c_3,c_4,c_5$ are defined in (\\ref{quantities}).\n%\n\\end{lemma}","sha256_utf8":"3866b3a0bd5f9d648372782d02cb97b24e0e5e81a6412c96e07adfd89260d55c"},{"start":46071,"end":46974,"start_line":1040,"end_line":1062,"raw_tex":"\\begin{proof}\nBy Lemma \\ref{chgL} we have \n\\begin{eqnarray*}\n&\\mathcal L^{\\alpha}(x^{k+1},y^{k+1},z^{k+1})=\\mathcal L^{\\alpha}(x^{k+1},y^{k+1},z^{k}) \n+ \\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2\n&\\\\[.1in]&\n\\le  \\mathcal L^{\\alpha}(x^{k+1},y^{k},z^{k}) + \\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2\n-\\frac{1}{2} \\|\\Delta y^{k+1}\\|_{D}^2 \n&\\\\[.1in]&\n\\le \\mathcal L^{\\alpha}(x^{k},y^{k},z^{k})\n+ \\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2\n-\\frac{1}{2} \\|\\Delta y^{k+1}\\|_{D}^2 -\\frac{1}{2}\\sum_{i=1}^p \\|\\Delta x_i^{k+1}\\|^2_{T_i}.&\n\\end{eqnarray*}\nBy Lemma \\ref{bounddual} for $\\epsilon_0>1$ we have \n\\[\n\\frac{\\epsilon_0}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2\n\\le\\epsilon_0c_4 \\|\\Delta y^{k+1}\\|^2\n+\\epsilon_0c_3 \\|\\Delta y^{k}\\|^2+\n\\epsilon_0c_5 \\|B\\tr \\Delta z^{k}\\|^2\n-\\epsilon_0c_5\\| B\\tr \\Delta z^{k+1}\\|^2.\n\\]\nAdd this to the latter inequality and rearrange to obtain (\\ref{chgLiter}). $\\square$\n\\end{proof}","sha256_utf8":"5132e47a6c707194b86aa2d4e4e539b2a542a1199fde6d6bc56a4b2f7d9be152"},{"start":47931,"end":49154,"start_line":1088,"end_line":1118,"raw_tex":"\\begin{lemma}\\label{thm1}\n%{\\bf (monotonicity/convergence of $\\bar{\\mathcal L}$)}\nSuppose that A1-A4 hold, and \n\\[\n\\bar\\tau =\\min\\{ \\max \\{ q_i,\\lambda_{++}^{\\alpha A_i\\tr A_i-\\epsilon_i I} \\} : i=1,\\dots, p\\}.\n\\]\n If $\\sigma=\\min\\{\\lambda_{\\min}(\\bar D), \\bar\\tau,\\frac{\\epsilon_0-1}{\\alpha\\beta}\\}>0$ where \n $\\bar D$ is defined in (\\ref{quantities}), then\nthe following assertions hold.\n%\n\\begin{itemize}\n\\item [(i)] The sequence $\\{\\bar {\\mathcal L_k}\\}_{k\\ge 1}$ is monotonically decreasing and \n%\n\\begin{eqnarray}\\label{Lbarmon}\n\\bar{\\mathcal L}_{k+1}+\\sigma\n\\big[\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|^2\n+ \\|\\Delta y^{k+1}\\|^2+\n\\|\\Delta z^{k+1}\\|^2\\big]\n\\le \\bar{\\mathcal L}_k.\n\\end{eqnarray}\n%\n%\n\\item [(ii)] If the sequence $\\{{\\bf x}^k, y^k, z^k\\}_{k\\ge 0}$ generated by the PADMM is bounded \nthen $\\bar{\\mathcal L}_k$ is bounded from below for all $k \\in \\mathbb N$  and converges as $k$ approaches infinity.\n%%\n%\\item [(iii)] If $B\\tr B$ is invertible, then $\\{{\\bf x}^k, y^k, z^k\\}_{k\\ge 0}$ is bounded.\n%\\item [(iv)]  If $\\{{\\bf x}^k, y^k, z^k\\}_{k\\ge 0}$ is bounded\n%then \n%$\\lim_{k\\to \\infty} \\|\\Delta x_i^{k}\\|=\\lim_{k\\to \\infty} \\|\\Delta y^{k}\\|=\\lim_{k\\to \\infty} \\|\\Delta z^{k}\\|=0.$\n\\end{itemize}\n\\end{lemma}","sha256_utf8":"13bea07d676c4a91ff7e9da5c79db97687fe6e5869c402f73c64fdd3609f5bf1"},{"start":49157,"end":50796,"start_line":1120,"end_line":1153,"raw_tex":"\\begin{proof}\n{\\bf (i)} By (\\ref{chgLiter}) and (\\ref{Lbark}) we obtain \n\\[\n\\bar{\\mathcal L}_{k+1}\n+\\frac{1}{2}\\sum_{i=1}^p \\|\\Delta x_i^{k+1}\\|^2_{T_i}\n+\\frac 12\\|\\Delta y^{k+1}\\|^2_{D-2\\epsilon_0(c_3+c_4)}+\n\\displaystyle{\\frac{\\epsilon_0-1}{\\alpha\\beta}}\\big\\|\\Delta z^{k+1}\\big\\|^2\n\\le \\bar{\\mathcal L}_k.\n\\]\nWe have $T_i=Q_i\\succeq q_i I$ where $q_i>0$ or $T_i=\\alpha A_i\\tr A_i-\\epsilon_i I$.\nWe let $\\bar\\tau =\\min\\{ \\max \\{ q_i,\\lambda_{++}^{\\alpha A_i\\tr A_i-\\epsilon_i I} \\} : i=1,\\dots, p\\}$\nand $\\bar D=D-2\\epsilon_0(c_3+c_4)I$. For \n$\\sigma=\\min\\{\\lambda_{\\min}(\\bar D), \\bar\\tau,\\frac{\\epsilon_0-1}{\\alpha\\beta}\\}>0$\nwe arrive at the inequality (\\ref{Lbarmon}), showing monotonicity of $\\{\\bar {\\mathcal L_k}\\}_{k\\ge 1}$.\n%\n\n{\\bf (ii)}  If $\\{{\\bf x}^k, y^k, z^k\\}_{k\\ge 0}$ is a bounded  sequence,\nthen \n$\\|B\\tr\\Delta z^k\\|^2+\n\\|\\Delta y^k\\|^2$ is also bounded, and bounded from below by zero.\nHence we only need to show that \n$\\mathcal L^{\\alpha}({\\bf x}^k,y^k,z^k)$ is bounded. \nBy the boundedness of $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$,\nthe terms  $\\langle z^k, {\\bf Ax}^k+By^k+b\\rangle$ and $\\|{\\bf Ax}^k+By^k+b\\|^2$\nare bounded. By A1 the function $f_i, i=1,\\dots,p$ are coercive,\nsince $\\{{\\bf x}^k\\}_{k\\ge 0}$ is bounded, then $\\{f_i(x_i^k)\\}_{k\\ge 0}$ is bounded for $i=1,\\dots,p$. \nBy A2, the function $h$ is bounded from below.\nThus, $\\{\\mathcal L^{\\alpha}({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$\nis bounded from below.\n%\nSince $\\{\\bar{\\mathcal L}_k\\}_{k\\ge 1}$ is bounded from below and by part (i) it is \nmonotonically decreasing then we conclude that $\\{\\bar{\\mathcal L}_k\\}_{k\\ge 1}$ it converges.  $\\square$\n\n\\end{proof}","sha256_utf8":"b5db1b6cc455d4e4355ddffece3c324ff5b452d8e706313947d7f905b96881d1"},{"start":50897,"end":51170,"start_line":1157,"end_line":1168,"raw_tex":"\\begin{lemma}\\label{samelimit}\nSuppose that A1-A4 hold and   $({\\bf x}^*,y^*,z^*)\\in\\omega(\\{ ({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0})$,\nthen\n\\[\n\\bar{\\mathcal L}({\\bf x}^*,y^*,z^*, y^*,z^*)\n=\n\\mathcal L^{\\alpha} ({\\bf x}^*,y^*,z^*) \n=\n\\sum_{i=1}^p f_i(x_i^*)+h(y^*).\n\\]\n\n\\end{lemma}","sha256_utf8":"309450f1ed8b1d93e95fa1ad59dd2275cf8f73746209f0fea700b25f44bbbe16"},{"start":51171,"end":51864,"start_line":1169,"end_line":1185,"raw_tex":"\\begin{proof}\nLet $({\\bf x}^*,y^*,z^*)\\in\\omega(\\{ ({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0})$\nand let the subsequence $\\{({\\bf x}^{k_j},y^{k_j},z^{k_j})\\}_{j\\ge 0}$ of the \nPADMM algorithm converges to \n$({\\bf x}^*,y^*,z^*)$.\nClearly $\\|\\Delta y^{k_j}\\|$ and $\\|B\\tr \\Delta z^{k_j}\\|$ both converge to zero as $j$ goes to infinity.\nThis, together with Lemma\n Lemma \\ref{thm-cluster} and \\ref{thm-cluster0} give\n%\n\\begin{eqnarray*}\n\\lim_{j\\to\\infty} \\bar {\\mathcal L}_{k_j} =\\lim_{j\\to\\infty} \\mathcal L^{\\alpha} (x^{k_j},y^{k_j},z^{k_j})\n&=&\\mathcal L^{\\alpha}(x^*,y^*,z^*)\\\\\n&=&\\sum_{i=1}^pf_i(x_i^*)+h(y^*)\n=\\bar{\\mathcal L}({\\bf x}^*,y^*,z^*, y^*,z^*). \\quad\\quad\\square\n\\end{eqnarray*}\n%\n\\end{proof}","sha256_utf8":"78d55783d8c2e08a299b03076e283ce5d115d5acd80f5ca81e9901d858aefa9f"},{"start":51905,"end":53099,"start_line":1187,"end_line":1217,"raw_tex":"\\begin{lemma}\\label{lma-D}\n{(subgradient bound for $\\bar{\\mathcal L}_k$)}\nLet $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ be a sequence generated by the PADMM algorithm\nand assume that $\\big(\\{d_{x_i}^k\\}_{i=1}^p,d_y^k,d_z^k\\big)\\in\\partial \\mathcal L^{\\alpha}({\\bf x}^k,y^k,z^k)$. Then \n%\n\\[\ns^{k}:=\\Big( \\{s^{k}_{x_i}\\}_{i=1}^p, s^{k}_y, s^{k}_z, s^{k}_{y'},s^{k}_{z'} \\Big)\\in\n\\partial \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1}),\\quad k\\ge 1\n\\]\n%\nwhere\n%\n$s_{x_i}^{k}:=d^{k}_{x_i}, i=1,\\dots, p$,\n$s_y^{k}:=d^{k}_y+2\\epsilon_0c_3 \\Delta y^{k}$,\n$s_z^{k}:=d^{k}_z+2\\epsilon_0c_5BB\\tr \\Delta z^{k}$,\n$s_{y'}^{k}:=-2\\epsilon_0c_3 \\Delta y^{k}$,\n$s_{z'}^{k}:=-2\\epsilon_0c_5BB\\tr \\Delta z^{k}$,\nand $\\epsilon_0>1$, and $c_3$ and $c_5$ are defined in  (\\ref{quantities}).\nMoreover,\n\\begin{eqnarray}\\label{nD}\n|||s^{k}|||\\le \\tilde\\rho \\Big(\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|+\\|\\Delta y^{k}\\|+\\|\\Delta z^{k}\\|\\Big),\n\\end{eqnarray}\nwhere \n$\n\\tilde\\rho:=\\max\\Big\\{\n\\|{\\bf Q}\\|+\\|{\\bf A}\\|^2,\\;\n\\|P\\|+\\alpha \\|{\\bf A}\\|\\|B\\|+4\\epsilon_0c_3, \\;\n\\|{\\bf A}\\|  +\\|B\\|+4\\epsilon_0c_5\\|B\\|^2 + \\frac{1}{\\alpha\\beta}\\Big\\},\n$\nand $\\|{\\bf Q}$ and $\\|{\\bf A}\\|$ are defined in Lemma \\ref{thm-d}.\n\\end{lemma}","sha256_utf8":"10a474ad587b6506b308f045bb1a2060c8cd33035ef4ee453b5d67cfab1e622b"},{"start":53102,"end":55814,"start_line":1219,"end_line":1302,"raw_tex":"\\begin{proof}\nLet $k\\ge 1$ be fixed and $(\\{d_{x_i}^{k}\\}_{i=1}^p, d_y^{k}, d_z^{k})\\in \\partial \\mathcal L^{\\alpha}(x^k,y^k,z^k)$. By taking partial derivatives of $\\bar{\\mathcal L}$ with respect to $x,y,z,y',z'$ and evaluating \nthem at $({\\bf x}^k,y^k,z^k,y^{k-1},z^{k-1})$ we obtain\n%\n\\begin{eqnarray*}\n%s_{x_i}^{k}&:=& \n\\partial_{x_i} \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1})\n&=&\\partial_{x_i} \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k})\n\\ni d_{x_i}^{k},\\; i=1,\\dots,p;\\\\\n%s_y^{k}&:=&\n\\nabla_y  \\bar{\\mathcal L} ({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1})\n&=&\\nabla_y \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k})\n+2\\epsilon_0c_3 \\Delta y^k=\nd_y^{k}+2\\epsilon_0c_3 \\Delta y^k;\\\\\n%s_z^{k}&:=&\n\\nabla_z  \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1})\n&=&\\nabla_z \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k})\n+2\\epsilon_0c_5 BB\\tr \\Delta z^{k}=\nd_z^{k}\n+2\\epsilon_0c_5 BB\\tr\\Delta z^{k};\\\\\n%s_{y'}^{k}&:=&\n\\nabla_{y'}  \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1})\n&=&-2\\epsilon_0c_3\\Delta y^k;\\\\\n%d_{z'}^{k}&:=&\n\\nabla_{z'}  \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1})\n&=&-2\\epsilon_0c_5BB\\tr \\Delta z^k.\n\\end{eqnarray*}\n%\nThese equations follow that \n\\begin{eqnarray*}\ns^k =\n\\left[\n\\begin{array}{c}\ns^k_{x_1}\\\\\n\\vdots\\\\\ns^k_{x_p}\\\\\ns^k_{y} \\\\ \ns^k_{z}\\\\\ns^k_{y'}\\\\\ns^k_{z'}\n\\end{array}\n\\right]\n=\n\\left[\n\\begin{array}{l}\nd^k_{x_1}\\\\\n\\vdots\\\\[.1in]\nd^k_{x_p}\\\\\nd_y^{k}+2\\epsilon_0c_3 \\Delta y^k \\\\ \nd_z^{k}+2\\epsilon_0c_5 BB\\tr\\Delta z^{k}\\\\\n-2\\epsilon_0c_3\\Delta y^k\\\\\n-2\\epsilon_0c_5BB\\tr \\Delta z^k\n\\end{array}\n\\right]\\in \\partial \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1}).\n\\end{eqnarray*} \n%\nBy the triangle inequality we have\n%\n\\begin{eqnarray*}\n&\\|s_y^{k}\\|\\le \\|d^{k}_y\\|+2\\epsilon_0c_3\\|\\Delta y^k\\|,\\quad\\quad\n\\|s_z^{k}\\|\\le\\|d^{k}_z\\|+2\\epsilon_0c_5\\|B\\|^2 \\|\\Delta z^k\\|,&\\\\\n&\\|s_{y'}^{k}\\|\\le 2\\epsilon_0c_3\\|\\Delta y^k\\|,\\quad\\quad\n\\|s_z^{k}\\|\\le 2\\epsilon_0c_5 \\|B\\|^2 \\|\\Delta z^k\\|.&\n\\end{eqnarray*}\n%\nThus\n\\begin{eqnarray*}\n|||s^{k}|||&\\le&  \\sum_{i=1}^p\\|s_{x_i}^{k}\\|+\\|s_y^{k}\\|+\\|s_z^{k}\\|+ \\|s_{y'}^{k}\\|+\\|s_z^{k}\\|\\\\\n&\\le& \\sum_{i=1}^p \\|d^{k}_{x_i}\\|+  \\|d^{k}_y\\|+\\|d^{k}_z\\|+\n4\\epsilon_0c_3\\|\\Delta y^{k}\\|+\n4\\epsilon_0c_5 \\|B\\|^2 \\|\\Delta z^k\\|.\n\\end{eqnarray*}\n%\nBy Lemma \\ref{thm-d}, we have \n%\n\\begin{eqnarray*}\n&|||s^{k}|||\\le\n\\Big(\\|{\\bf Q}\\|+\\|{\\bf A}\\|^2\\Big) \\sum_{i=1}^p\\|\\Delta x_i^{k}\\|\n+\\Big(\\|P\\|+\\alpha \\|{\\bf A}\\|\\|B\\| +4\\epsilon_0c_3 \\Big)      \\|\\Delta y^{k}\\|&\\\\\n&+\\Big(\\|{\\bf A}\\|  +\\|B\\|+4\\epsilon_0c_5\\|B\\|^2+\\frac{1}{\\alpha\\beta}\\Big)  \\|\\Delta z^{k}\\|\n\\le \\tilde\\rho \\Big( \\sum_{i=1}^p\\|\\Delta x_i^{k}\\|+   \\|\\Delta y^{k}\\|+  \\|\\Delta z^{k}\\|\\Big). \\quad \\square&\n\\end{eqnarray*}\n\n\\end{proof}","sha256_utf8":"4e60370f42a9fabd20b4cfa7c7285e75adc7ecc49bfee69050efab4fe484294b"},{"start":56288,"end":56686,"start_line":1313,"end_line":1321,"raw_tex":"\\begin{lemma} \\label{bddseq0}\nSuppose that A1-A4 hold and $\\sigma$ defined in Lemma \\ref{thm1} is positive.\nThen the sequence \n$\\{{\\bf x}^k, y^k, z^k\\}_{k\\ge 0}$ generated by the PADMM algorithm (\\ref{padmm}) is bounded,\nand $\n\\lim_{k\\to \\infty} \\|\\Delta x_1^{k}\\|=\\dots=\\lim_{k\\to \\infty} \\|\\Delta x_p^{k}\\|=\n\\lim_{k\\to \\infty} \\|\\Delta y^{k}\\|=\\lim_{k\\to \\infty} \\|\\Delta z^{k}\\|=0.\n$\n\\end{lemma}","sha256_utf8":"0bb40970e52304d514c05b51dfd9e52592e639b7ae79ced4c08259b5b0393d77"},{"start":56688,"end":60650,"start_line":1323,"end_line":1437,"raw_tex":"\\begin{proof}\nLet $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ be a sequence generated by the PADMM algorithm.\n%\nBy $\\sigma>0$, the sequence $\\{\\bar{\\mathcal L}_k\\}_{k\\ge 1}$ \ndefined in (\\ref{Lbark}) is monotonically decreasing, hence it is upper bounded by \n$\\bar{\\mathcal L}_1=\\bar{\\mathcal L}({\\bf x}^1,y^1,z^1,y^0,z^0)$.\nThis follows that \n%\n\\begin{eqnarray}\\label{ineq10}\n\\begin{array}{c}\n\\sum_{i=1}^pf(x_i^{k})+h(y^{k})\n+\\frac{\\alpha}{2}\\|{\\bf Ax}^{k}+By^{k}\n+\\alpha^{-1}z^{k}+b\\|^2-\\frac{1}{2\\alpha}\\|z^{k}\\|^2\n\\\\[.1in]\n+\\sigma \\sum_{i=1}^p \\|\\Delta x_i^{k}\\|^2\n +\\sigma\\|\\Delta z^{k}\\|^2\n+(\\sigma+\\epsilon_0c_3) \\|\\Delta y^{k}\\|^2\n+\\epsilon_0c_5\\|B\\tr \\Delta z^{k}\\|^2\n\\le \\bar{\\mathcal L}_1\n \\end{array}\n\\end{eqnarray}\n%\nfor all $k\\in\\mathbb Z_+$.\nBy Lemma \\ref{bounddual} (iii), (\\ref{ineq10}) leads to \n%\n\\begin{eqnarray}\\label{thmbdd-eq1}\n\\begin{array}{c}\n\\sum_{i=1}^pf(x_i^{k})+\\frac{\\alpha}{2}\\|{\\bf Ax}^{k}+By^{k}+\\alpha^{-1}z^{k}+b\\|^2\n+r_1\\|\\Delta y^{k}\\|^2\\\\[.1in]\n+ \\sigma\\sum_{i=1}^p\\|\\Delta x^{k}\\|^2\n+ r_2\\|B\\tr \\Delta z^{k}\\|^2\n+\\sigma\\|\\Delta z^{k}\\|^2\n\\le\\bar{\\mathcal L}_1\n%- \\inf_{x,y}\\Big\\{g(x)- c_1 \\Big\\|\\nabla g(x)\\Big\\|^2\\Big\\}\n-\\inf_y\\Big\\{ h(y)-c_1 \\|\\nabla h(y)\\|^2 \\Big\\},\n\\end{array}\n\\end{eqnarray}\n%\nwhere $r_1:= \\epsilon_0c_3+\\sigma-\\|P\\|^2c_1>0$\nand $r_2:=\\epsilon_0c_5-c_6>0$. \nBy A2,  since $h$ is $L_h$ Lipschitz differentiable then for any $k\\ge 1$ \nand $\\delta>0$ we have\n%\n$\nh\\big(y^k-\\delta \\nabla h(y^k)\\big)\\le h(y^k)- \n\\big(\\delta-\\frac{L_h\\delta^2}{2}\\big)\\|\\nabla h(y^k)\\|^2.\n$\n%\nAgain by A2, $h$ is bounded from below then\n%\n\\begin{eqnarray}\\label{hbd}\n-\\infty<\\inf\\{h(y)- \\Big(\\delta-\\frac{L_h\\delta^2}{2}\\Big)\\|\\nabla h(y)\\|^2: \\; y\\in\\mathbb R^q\\}.\n\\end{eqnarray}\n%\nWe choose $\\delta (>0)$ such that \n$\\delta-\\frac{L_h\\delta^2}{2}=c_1$. By (\\ref{hbd}), the right hand side of (\\ref{thmbdd-eq1}) is finite\nhence\n%\n\\begin{eqnarray}\\label{finite}\n\\sum_{i=1}^pf(x_i^{k})+\\|{\\bf Ax}^{k}+By^{k}+\\alpha^{-1}z^{k}+b\\|^2\n+\\|\\Delta y^{k}\\|^2 +\\|\\Delta z^k\\|^2<+\\infty.\n\\end{eqnarray}\n%\nBy A1, since $f_i, i=1,\\dots,p$ are coercive and $\\sum_{i=1}^pf(x_i^{k})<+\\infty$, then the sequence \n$\\{x_i^k\\}_{k\\ge 0}$  is bounded for $i=1,\\dots,p$, and so are \n$\\{{\\bf x}^k\\}_{k\\ge 0}$ and $\\{{\\bf Ax}^k\\}_{k\\ge 0}$.\n%\n%In addition, by (\\ref{finite}) the sequences \n%$\\{Ax^{k}+By^{k}+\\alpha^{-1}z^{k}+c\\}_{k\\ge k_0}$ and $\\{ \\Delta y^{k}\\}_{k\\ge k_0}$ are bounded, \n%then $\\{A\\Delta x^k\\}_{k\\ge k_0}$ and $\\{B\\Delta y^k\\}_{k\\ge k_0}$\n%are bounded which follows that $\\{\\Delta z^k\\}_{k\\ge 0}$ is bounded.\n\nBy the $z$ iterate of PADMM we have\n$\nBy^{k} = \\frac{1}{\\alpha\\beta} \\Delta z^{k} -{\\bf Ax}^{k}-b.\n$\nBy (\\ref{finite}), $\\{\\Delta z^k\\}_{k\\ge 0}$ is bounded.\nBy A3, $B\\tr B$ is invertible, thus $\\{y^k\\}_{k\\ge 0}$ is bounded.\nFinally, by (\\ref{finite}) $\\{{\\bf Ax}^{k}+By^{k}+\\frac{1}{\\alpha}z^{k}\\}_{k\\ge 0}$ is bounded\nand ${\\bf Ax}^k$ and $By^k$ are bounded, so \n$\\{z^k\\}_{k\\ge 0}$ is then bounded.\n%\nSumming up (\\ref{Lbarmon}) for $k=1,\\dots, K-1$, $K\\ge 2$ gives\n%\n\\[\n\\sum_{k=1}^{K}\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|^2\n+\\|\\Delta y^{k}\\|^2\n+\\|\\Delta z^{k}\\|^2\\Big\\}\\le \\frac{1}{\\sigma}(\\bar{\\mathcal L}_1 - \\bar{\\mathcal L}_K) .\n\\]\nSince $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ is bounded, by Lemma \\ref{thm1} (ii) the right hand side is finite for any $K$.\nWe let $K$ approach to infinity to obtain \n\\begin{eqnarray*}\n\\sum_{k=1}^{\\infty}\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|^2 +\\|\\Delta y^{k}\\|^2+\\|\\Delta z^{k}\\|^2\\Big\\}<\n\\frac{1}{\\sigma}(\\bar{\\mathcal L}_1 - \\inf_{k\\ge 1}\\bar{\\mathcal L}_k) <\n+\\infty.\n\\end{eqnarray*}\nThis follows \n$\n\\displaystyle{\\lim_{k\\to \\infty}} \\Big(\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|^2 +\\|\\Delta y^{k}\\|^2+\\|\\Delta z^{k}\\|^2\\Big)=0,\n$\nand consequently \n$\n\\displaystyle{\\lim_{k\\to \\infty}} \\|\\Delta x_1^{k}\\|\n=\\dots\n=\\displaystyle{\\lim_{k\\to \\infty}} \\|\\Delta x_p^{k}\\|\n=\n\\displaystyle{\\lim_{k\\to \\infty}}\\|\\Delta y^{k}\\|\n=\n\\displaystyle{\\lim_{k\\to \\infty}}\\|\\Delta z^{k}\\| =0.\\quad\\square\n$\n\n\n\\end{proof}","sha256_utf8":"bac6175215f7924fe0bcaa0ce68f6bfd2e5cbc5bb6d8e1a3f46283d0da83dd27"},{"start":60690,"end":61635,"start_line":1440,"end_line":1458,"raw_tex":"\\begin{lemma}\\label{lma-cluster2}\nLet $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ be a sequence generated by the PADMM algorithm (\\ref{padmm}),\nand denote\n\\begin{eqnarray}\\label{omega}\n\\Omega:=\\omega \\Big(\\{({\\bf x}^k,y^k,z^k,y^{k-1},z^{k-1})\\}_{k\\ge 1}\\Big).\n\\end{eqnarray}\nThen the following statements are true\n\\begin{itemize}\n\\item [(i)] $\\Omega$ is nonempty, connected, and compact.\n\\item [(ii)] $\\Omega \\subseteq \\{({\\bf x}^*, y^*, z^*,  y^*, z^*)\n\\in\\mathbb R^n\\times\\mathbb R^q\\times\\mathbb R^m\\times\\mathbb R^q\\times\\mathbb R^m:\n\\;({\\bf x}^*, y^*, z^*)\\in {\\rm crit} (\\mathcal L^{\\alpha})\\big\\}$\n%\n\\item [(iii)] $\\lim_{k\\to\\infty} {\\rm dist} \\big[ ({\\bf x}^k, y^k, z^k, y^{k-1}, x^{k-1}), \\Omega \\big]=0$.\n\\item [(iv)] The sequences $\\{\\bar{\\mathcal L}_k\\}_{k\\ge 1}$,\n $\\{\\bar{\\mathcal L}^{\\alpha}({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$,\n and $\\{\\sum_{i=1}^pf_i(x_i^k)+h(y^k)\\}_{k\\ge 0}$ approaches the same limit on $\\Omega$. \n\\end{itemize}\n\\end{lemma}","sha256_utf8":"c6323881f43cf6fca7c823643bab033bd315f9acc7fe2201eb7751b871d7651f"},{"start":61636,"end":61776,"start_line":1459,"end_line":1461,"raw_tex":"\\begin{proof}\nThese results follow from Lemmas \\ref{thm1}, \\ref{samelimit}, and \\ref{bddseq0}. Thus we omit the proof. $\\square$\n\\end{proof}","sha256_utf8":"3c0364c46fddc96f3f9d6991df4ef0f182e576f0e8a4b8af0532cbe42f5dcf46"},{"start":63102,"end":70515,"start_line":1496,"end_line":1745,"raw_tex":"\\begin{proof}\nBy Lemma \\ref{thm1}, the sequence $\\{\\bar{\\mathcal L}_k\\}_{k\\ge 1}$ is monotonically\n decreasing and converges. Let $\\bar{\\mathcal L}_*:=\\lim_{k\\to\\infty}\\bar{\\mathcal L}_k$, then \nthe error sequence \n$e_k:=\\bar{\\mathcal L}_k-\\bar{\\mathcal L}_*, k\\ge 1$\nis non-negative, monotonically decreasing, and converges to zero. \nWe prove under two cases.\n\n {\\it Case 1:} let assume that \nthere is a $k_1\\in\\mathbb Z_+$ such that $e_{k_1}=0$. \nSince $\\{e_k\\}_{k\\ge 1}$ is monotonically decreasing, \n$e_k=0$ for all $k\\ge k_1$.\nBy Lemma \\ref{thm1}(i) then we have\n\\[\n\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|^2+\\|\\Delta y^{k+1}\\|^2+\\|\\Delta z^{k+1}\\|^2 \\le \n \\frac{1}{\\sigma}(e_k-e_{k+1})=0,\\quad \\forall k\\ge k_1.\n\\]\nHence  as $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ is a bounded sequence\nthen \n\\[\n\\sum_{k=1}^{\\infty}\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|\n+\\|\\Delta y^{k}\\|\n+\\|\\Delta z^{k}\\|\\Big\\}\n=\\sum_{k=1}^{k_1} \\Big\\{\n\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|\n+\\|\\Delta y^{k}\\|\n+\\|\\Delta z^{k}\\|\\Big\\}\n\\] \nis bounded.\n\n{\\it Case 2:}\nwe let  $e_k>0$ for all $k\\ge 1$. \nBy Lemma \\ref{thm1}(i) we have\n%\n\\begin{eqnarray}\\label{s1}\n%|||\\Delta u^{k+1}|||^2=\n\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|^2+\\|\\Delta y^{k+1}\\|^2+\\|\\Delta z^{k+1}\\|^2 \\le \n \\frac{1}{\\sigma}( e_k-e_{k+1}),\\quad \\forall k\\ge 1.\n\\end{eqnarray}\n%\nBy Lemma \\ref{lma-cluster2}, $\\Omega$ is nonempty, compact, and connected,\nand $\\bar{\\mathcal L}$ takes on a constant value $\\bar{\\mathcal L}_*$ on $\\Omega$. \nSince $\\bar{\\mathcal L}_{k}\\searrow\\bar{\\mathcal L}_*$ as $k\\to\\infty$ and $\\bar{\\mathcal{L}}$ satisfies \nthe {K\\L} property then there exists  $ k_1\\ge 1$ such that \n$\\bar{\\mathcal L}_*<\\bar{\\mathcal L}_k<\\bar{\\mathcal L}_*+\\eta$\nfor all $k\\ge k_1$ and some $\\eta>0$.\n%\nMoreover, $\\lim_{k\\to\\infty} {\\rm dist} \\big[ ({\\bf x}^k,y^k,z^k,y^{k-1},x^{k-1}), \\Omega\\big]=0$ by\nLemma \\ref{lma-cluster2}, \nthus for a given $\\epsilon>0$ there exists $k_2\\ge 1$ such that for all $k\\ge k_2$ it holds\n\\[\n{\\rm dist} \\Big[ ({\\bf x}^k,y^k,z^k,y^{k-1},x^{k-1}), \\Omega\\Big]<\\epsilon. \n\\]\nLet  $\\tilde k:=\\max\\{ k_1,k_2,3 \\}$, \nthen $({\\bf x}^k,y^k,z^k,y^{k-1},x^{k-1})\\in\\mathcal S$\n and\n%\n$\n\\psi'(\\bar{\\mathcal L}_k-\\bar{\\mathcal L}_*)\\cdot {\\rm dist}\\Big(0, \n\\partial \\bar{\\mathcal L}_k \\Big)\n\\ge 1$ for all $k\\ge \\tilde k$, or equivalently\n%\n\\begin{eqnarray}\\label{KL}\n\\psi'(e_k)\n\\cdot {\\rm dist}\\Big(0, \n\\partial \\bar{\\mathcal L}_k \\Big)\n \\ge1.\n\\end{eqnarray}\n%\nMultiply both sides of this inequality by $ |||\\Delta u^{k+1}|||^2 $ to get\n%\n\\begin{eqnarray}\\label{eq:001}\n \\psi'(e_{k}) \\cdot {\\rm dist}\\Big(0, \n\\partial \\bar{\\mathcal L}_k \\Big) |||\\Delta u^{k+1}|||^2 \\ge |||\\Delta u^{k+1}|||^2,\n\\end{eqnarray}\nwhere \n$|||\\Delta u^k|||^2=\\sum_{i=1}^p\\|\\Delta x_i^k\\|^2+\\|\\Delta y^k\\|^2+\\|\\Delta z^k\\|^2.$\n%\nBy (\\ref{s1}), (\\ref{eq:001}) leads to \n\\begin{eqnarray}\\label{eq:002}\n \\frac {1}{\\sigma} \\psi'(e_{k}) (e_k-e_{k+1})\\cdot {\\rm dist}\\Big(0, \n\\partial \\bar{\\mathcal L}_k \\Big)  \\ge |||\\Delta u^{k+1}|||^2.\n\\end{eqnarray}\n%\nBy the concavity of $\\psi$, $\\psi(e_{k}) -\\psi(e_{k+1})\\ge  \\psi'(e_{k}) (e_{k} - e_{k+1})$, hence\n%\n\\begin{eqnarray}\\label{eq:003}\n|||\\Delta u^{k+1}|||^2 \\le \n \\frac{1}{\\sigma} \\big(\\psi(e_{k}) -\\psi(e_{k+1})\\big) \n \\cdot {\\rm dist}(0, \\partial \\bar{\\mathcal L}_k ).\n\\end{eqnarray}\nBy the arithmetic mean-geometric mean inequality, for any $\\gamma>0$ we  have\n%\n\\begin{eqnarray}\\label{artith-geo}\n\\sqrt{ \\frac{1}{\\sigma} \\big(\\psi(e_{k}) -\\psi(e_{k+1})\\big) \n \\cdot {\\rm dist}(0, \\partial \\bar{\\mathcal L}_k )}\n \\le \\frac{\\gamma}{2\\sigma}  \\big(\\psi(e_{k}) -\\psi(e_{k+1})\\big) \n+\\frac{1}{2\\gamma}{\\rm dist}(0, \\partial \\bar{\\mathcal L}_k ).\n \\end{eqnarray}\n%\nMoreover,\n\\begin{eqnarray}\\label{eq:004}\n\\frac{1}{\\sqrt{p+2}} \\Big(\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\\Big)\n\\le |||\\Delta u^{k+1}|||.\n\\end{eqnarray}\n%\nThus  (\\ref{eq:003}), (\\ref{artith-geo}), and (\\ref{eq:004}) give rise to\n%\n\\begin{eqnarray}\\label{main}\n\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n\\le \\frac{\\gamma\\sqrt{p+2}}{2\\sigma}  \n\\big(\\psi(e_{k}) -\\psi(e_{k+1})\\big) \n+\\frac{\\sqrt{p+2}}{2\\gamma}{\\rm dist}(0, \\partial \\bar{\\mathcal L}_k ).\n\\end{eqnarray}\n%\nNote that ${\\rm dist}(0, \\partial \\bar{\\mathcal L}_k)=|||s^k|||$, where $s^k\\in\\partial \\bar{\\mathcal L}_k$ \ndefined in Lemma \\ref{lma-D}, hence \n%\n\\begin{eqnarray}\\label{eq:import}\n\\begin{array}{c}\n\\displaystyle{\n\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n}\\\\[.1in]\n\\le \n\\displaystyle{\n\\frac{\\gamma\\sqrt{p+2}}{2\\sigma}  \n\\big(\\psi(e_{k}) -\\psi(e_{k+1})\\big) \n+\n\\frac{\\tilde \\rho\\sqrt{p+2}}{2\\gamma}\n\\Big(\n\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|\n+\\|\\Delta y^{k}\\|\n+\\|\\Delta z^{k}\\|\\Big).\n}\n\\end{array}\n\\end{eqnarray}\n%\nFor any $K\\ge k\\ge \\underline{k}$, the following identity holds\n\\[\n\\sum_{k=\\underline k}^K\\|\\Delta w^{k}\\| =\n\\sum_{k=\\underline k}^{K}\n\\|\\Delta w^{k+1}\\|\n+\\|\\Delta w^{\\underline k}\\|-\\|\\Delta w^{K+1}\\|.\n\\]\nChoose $\\gamma>0$ large enough such that $2\\gamma>\\tilde \\rho\\sqrt{p+2}$.\nLet $\\delta_0=1-\\frac{\\tilde \\rho\\sqrt{p+2}}{2\\gamma}>0$.\n%\nSumming up (\\ref{eq:import}) from $k=\\underline k\\ge \\tilde k $ to $K\\ge \\underline k$ leads to\n\\begin{eqnarray*}\n&\\sum_{k=\\underline k}^K\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\\Big\\}\n\\le\n\\delta_1\n\\big(\\psi(e_{\\underline k}) -\\psi(e_{K+1})\\big) \n&\\\\\n&\n+\\delta_2\n\\Big(\n\\sum_{i=1}^p\\|\\Delta x_i^{\\underline k}\\|\n+\\|\\Delta y^{\\underline k}\\|\n+\\|\\Delta z^{\\underline k}\\|\\Big)\n-\\delta_2\n\\Big(\n\\sum_{i=1}^p\\|\\Delta x_i^{K+1}\\|\n+\\|\\Delta y^{K+1}\\|\n+\\|\\Delta z^{K+1}\\|\\Big),&\n\\end{eqnarray*}\nwhere $\\delta_1:={\\gamma\\sqrt{p+2}}/{2\\sigma\\delta_0} $ \nand \n$\\delta_2= {\\tilde \\rho\\sqrt{p+2}}/{2\\gamma\\delta_0}$.\n%\nRecall that $e_k$ is monotonically decreasing and\n$\\psi'>0$, therefore\n%\n\\begin{eqnarray*}\n&\\sum_{k=\\underline k}^K\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n\\Big\\}\\le  \n\\delta_1\n\\psi(e_{\\underline k})\n+\\delta_2\\big(\n\\sum_{i=1}^p\\|\\Delta x_i^{\\underline k}\\|\n+\\|\\Delta y^{\\underline k}\\|\n+\\|\\Delta z^{\\underline k}\\|\\big).\n&\n\\end{eqnarray*}\n%\nThe right hand side of this inequality is bounded for any $K\\ge \\underline k$, we let $K\\to\\infty$ to obtain\n\\begin{eqnarray}\\label{eq:0005}\n\\sum_{k=\\underline k}^{\\infty}\n\\Big(\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n\\Big)\n\\le \n\\delta_1\n\\psi(e_{\\underline k})\n+\\delta_2\n\\Big(\n\\sum_{i=1}^p\\|\\Delta x_i^{\\underline k}\\|\n+\\|\\Delta y^{\\underline k}\\|\n+\\|\\Delta z^{\\underline k}\\|\\Big).\n\\end{eqnarray}\n%\nThen (\\ref{eq:0005}) together with the fact that $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ is  bounded gives\n%\n\\begin{eqnarray}\\label{eq:000000005}\n\\sum_{k=0}^{\\infty}\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n\\Big\\}\n<+\\infty.\n\\end{eqnarray}\n\n\nFor any $p,q,K\\in\\mathbb Z_+$ where $q\\ge p$ we have \n\\begin{eqnarray*}\n|||u^q-u^p||| =||| \\sum_{k=p}^{q-1} \\Delta u^{k+1}|||\n\\le\n \\sum_{k=p}^{q-1} |||\\Delta u^{k+1}|||\n \\le \n\\sum_{k=1}^{\\infty}|||\\Delta u^{k+1}|||.\n\\end{eqnarray*}\n%\nSince by (\\ref{eq:000000005}) the right hand side is finite \nthen \n\\[\n\\sum_{k=1}^{\\infty}|||\\Delta u^{k+1}|||\n\\le \\sum_{k=0}^{\\infty}\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n\\Big\\}\n<+\\infty.\n\\]\nThis implies that $\\{u^k\\}_{k\\ge 0}=\\{({\\bf x}, y^k,z^k)\\}_{k\\ge 0}$ is a Cauchy sequence. \nBy Lemma \\ref{thm-cluster} it converges to a stationary point.  $\\square$\n \\end{proof}","sha256_utf8":"c28a6af64bb19720e70343a83fe4684cc2eaf07b3050a4c9850056e690039bf1"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"线性约束分块优化模型","source_units":[{"start":2731,"end":2927,"start_line":75,"end_line":81,"raw_tex":"\\begin{eqnarray}\\label{OP}\n\\begin{array}{ccc}\n&\\displaystyle{\\min_{x_1,x_2,\\dots, x_p,y}}&\n\\sum_{i=1}^p f_i(x_i)+h(y)\\\\[.1in]\n&\\textrm{s.t.}& \\sum_{i=1}^pA_ix_i+By+b=0, \n\\end{array}\n\\end{eqnarray}","sha256_utf8":"216ba35e902c291c5a9addfe1a3e60973206b54aae15403a0167f7e38a722a00"},{"start":2932,"end":3746,"start_line":84,"end_line":95,"raw_tex":"where $p\\ge 1$, $x_i\\in\\mathbb R^{n_i}$ are variables with their coefficient matrix \n$A_i\\in\\mathbb R^{m\\times n_i}$, $i=1,2,\\dots,p$, $y\\in\\mathbb R^q$ is the last variable \nwith its coefficient matrix $B\\in\\mathbb R^{m\\times q}$, and $b\\in\\mathbb R^m$. \nThe model remains as general without $y$\nand $By$; but we keep $y$ and $By$ to simplify the notation.\nThe functions $f_i:\\mathbb  R^{n_i}\\to\\bar{\\mathbb R}$ for $i=1,2,\\dots,p$ are proper \nand lower-semicontinuous,\nand $h:\\mathbb R^{q}\\to \\bar{\\mathbb R}$ is proper smooth function.\n All functions can be nonconvex.\n We denote ${\\bf x}=[x_1,\\dots,x_p]\\in\\mathbb R^n$ where $n=\\sum_{i=1}^p n_i$,\n${\\bf A}:=[A_1\\dots A_p]\\in\\mathbb R^{m\\times n}$, and ${\\bf Ax}=\\sum_{i=1}^p A_ix_i\\in\\mathbb R^m$.\nWith these notations, the problem (\\ref{OP}) can be written  \n","sha256_utf8":"063574d6521b7e21fd614215b15fc2f01b7573649f4185801ce522aa22c0c9bf"},{"start":3748,"end":3896,"start_line":97,"end_line":101,"raw_tex":"\\begin{eqnarray}\\label{OPE}\n\\displaystyle{\\min_{{\\bf x},y}}\n\\sum_{i=1}^p f_i(x_i)+h(y)\n\\quad\\quad\\textrm{s.t.} \\;\\;\\;{\\bf Ax}+By+b=0.\n\\end{eqnarray}","sha256_utf8":"fb06fe9ea39ade70cb3c8bc92f40509524e85625fa7613d04cab6f72d6d27177"}]},{"id":"S002","semantic_type":"definition","kind":"definition","title":"增广拉格朗日函数","source_units":[{"start":4947,"end":5232,"start_line":114,"end_line":119,"raw_tex":"\\begin{eqnarray}\\label{augL}\n&\\mathcal L^{\\alpha}: \\mathbb R^n\\times\\mathbb R^q\\times \\mathbb R^m\\to\\mathbb R\\nonumber&\\\\[.1in]\n&\\mathcal L^{\\alpha} ({\\bf x},y,z)=\\sum_{i=1}^p f_i(x_i)+h(y)\n+\\langle z, {\\bf Ax}+By+b\\rangle \n+\\frac{\\alpha}{2}\\big\\|{\\bf Ax}+By+b\\big\\|^2,&\n\\end{eqnarray}","sha256_utf8":"d41af35eef0268fb73e61a4670a3cc804f7b362991d0621303c88498029bc785"},{"start":5235,"end":5545,"start_line":121,"end_line":123,"raw_tex":"where $\\alpha>0$ and the vector $z\\in\\mathbb R^m$ is the Lagrange multiplier associated with the constraint\n${\\bf Ax}+By+b=0$. In each iteration, the augmented Lagrangian is minimized over the primal variables \n$x_i, i=1,\\dots,p$ and $y$ separately and one after each other, followed by a dual update for $z$.\n","sha256_utf8":"d2a61527cc0644cf9a39b0e9855c200684b17eaf7bb7ce826b54572c87eec62a"}]},{"id":"S003","semantic_type":"algorithm","kind":"algorithm","title":"近端交替方向乘子迭代","source_units":[{"start":6011,"end":6295,"start_line":137,"end_line":139,"raw_tex":"Here $\\beta\\in (0,2)$ is the stepsize or damping parameter, $x_{<i}:=[x_1;\\dots;x_{i-1}]\\in\\mathbb R^{n_1+\\dots+n_{i-1}}$\nand $x_{>i}:=[x_{i+1};\\dots;x_{p}]\\in\\mathbb R^{n_{i+1}+\\dots+n_{p}}$\n(clearly, $x_{<0}$ and $x_{>p}$ are null variables, which may be used for notational ease).\n","sha256_utf8":"a636765fbc8ac1fe05b4d22ca985a711860aba50c69d293ed576c22cd6de7416"},{"start":11499,"end":11875,"start_line":222,"end_line":226,"raw_tex":" In this paper, we consider a variant/modification of ADMM that is called Proximal ADMM (PADMM).\nTo state the algorithm let $\\{Q_i\\}_{i=1,\\dots,p}\\subseteq \\mathbb R^{n_i\\times n_i}$\nand $P\\in \\mathbb R^{q\\times q}$ be symmetric positive semidefinite matrices and \n$\\|x\\|^2_M=x^TMx$, then the nonconvex possibly nonsmooth PADMM algorithm for solving \n(\\ref{OPE}) is given by:\n","sha256_utf8":"86c5b6dd7992df2564bcbc468498ecdd23084177ac274e2c94f1d5fcf7dd1b79"},{"start":11924,"end":12350,"start_line":229,"end_line":239,"raw_tex":"\\begin{eqnarray}\\label{padmm}\n\\left\\{\n\\begin{array}{lll}\nx_i^{k+1}&=&\\displaystyle{{\\arg\\min}_{x_i}} \\;\n\\mathcal L^{\\alpha}(x^{k+1}_{<i}, x_i, x^k_{>i},y^k,z^k)+\\frac{1}{2}\\|x_i-x_i^k\\|^2_{Q_i};\n\\quad i=1,\\dots, p \\\\[.1in]\ny^{k+1}&=&\\displaystyle{\\arg\\min}_{y}\\; \\mathcal L^{\\alpha}({\\bf x}^{k+1},y,z^k)+\\frac 12 \\|y-y^k\\|^2_P; \\\\[.1in]\nz^{k+1}&=&z^k+\\alpha\\beta ({\\bf Ax}^{k+1}+By^{k+1}+b).\n\\end{array}\n\\right.\n\\end{eqnarray}","sha256_utf8":"dd4c8ebc1416671f03e2e38d7177fe901661c311073efae9e8daca30827c0d5a"}]},{"id":"S004","semantic_type":"notation","kind":"notation","title":"谱符号与乘积范数","source_units":[{"start":15791,"end":16644,"start_line":287,"end_line":300,"raw_tex":"Throughout this paper, we denote $\\mathbb R$ as the real number set\nwhile $\\mathbb Z$ as the set of integers. \nThe set $\\bar{\\mathbb R}:=\\mathbb R\\cup\\{+\\infty\\}$ is the extended real numbers,\n$\\mathbb R_+$ is the positive real number set,\nand $\\mathbb Z_+$ is the set of positive integers.\nGiven the matrix $X$, ${\\rm Im}(X)$ denotes its image. \nWe denote by $I_n$ the $n\\times n$ identity matrix. The minimum and maximum eigenvalues of the matrix \n$X\\in\\mathbb R^{n\\times n}$ are denoted respectively by $\\lambda_{\\min}(X)$,\nand $\\lambda_{\\max}(X)$,\nand $\\lambda_{++}^X$ refers to the smallest strictly positive eigenvalue of $X$.\n%\nThe Euclidean scalar product of $\\mathbb R^n$ and its corresponding \nnorms are, respectively, denoted by $\\langle \\cdot, \\cdot\\rangle$\nand $\\|\\cdot\\|=\\sqrt{\\langle \\cdot,\\cdot\\rangle}$. For a vector $x\\in\\mathbb R^n$,\n","sha256_utf8":"888a2a2366b49bab6bccaa71f7d3a16331efce5a4c7bfb69ea5757a53ec46a1a"},{"start":17031,"end":17534,"start_line":314,"end_line":320,"raw_tex":" If  $n_1,\\dots, n_p\\in\\mathbb Z_+$ and $p\\in\\mathbb Z_+$,\nthen for any $v:=(v_1,\\dots,v_p)\\in\\mathbb R^{n_1}\\times\\mathbb R^{n_2}\\times\\dots\\times\\mathbb R^{n_p}$ and $v':=(v'_1,\\dots,v'_p)\\in \\mathbb R^{n_1}\\times\\mathbb R^{n_2}\\times\\dots\\times\\mathbb R^{n_p}$ the Cartesian product and its norm are defined by\n\\[\n\\ll v,v'\\gg =\\sum_{i=1}^{p}\\langle v_i,v_i'\\rangle\\quad\\quad\n%\n\\frac{1}{\\sqrt{p}}  \\sum_{i=1}^{p} \\| v_i\\| \\le |||v||| = \\sqrt{\\sum_{i=1}^{p} \\| v_i\\|^2}\\le  \\sum_{i=1}^{p} \\| v_i\\|.\n\\]\n","sha256_utf8":"d72a1037b64e8e140a7ab01e413c36645a6cec6c689fe304fd748ac8775b3d9c"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"次微分与临界点及距离","source_units":[{"start":17788,"end":18877,"start_line":329,"end_line":353,"raw_tex":" Let $\\Phi:\\mathbb R^d\\to \\bar{\\mathbb R}$ be a proper and lower semicontinuous function. \n The domain  of $\\Phi$, denoted ${\\rm dom}\\; \\Phi$, is defined by \n${\\rm dom}\\; \\Phi:=\\{x\\in\\mathbb R^d:\\; \\Phi(x)<+\\infty\\}.\n$\n%\n%The graph of the $\\Phi$, denoted  ${\\rm Graph}\\; \\Phi$, defined by\n%$\n% {\\rm Graph}\\; \\Phi:=\\{(x,y)\\in\\mathbb R^d \\times\\mathbb R: y=\\Phi(x)\\}.\n%$\nFor any $x\\in {\\rm dom}\\; \\Phi$, the {\\it Fr\\'echet (viscosity) subdifferential} of $\\Phi$ at $x$,\ndenoted $\\hat\\partial\\Phi(x)$, is defined by\n\\[\n\\hat\\partial\\Phi(x)=\n\\Big\\{\ns\\in\\mathbb R^d: \\;\\;\\lim_{y\\neq x}\\inf_{y\\to x} \\frac{\\Phi(y)-\\Phi(x)-\\langle s, y-x\\rangle}{\\|y-x\\|}\\ge 0\n\\Big\\}.\n\\]\nFor $x\\notin {\\rm dom}\\;\\Phi$, then $\\hat\\partial\\Phi (x)=\\emptyset$.\nThe {\\it limiting (Mordukhovich) subdifferential}, or simply the subdifferential for short,\n of $\\Phi$ at $x\\in {\\rm dom}\\;\\Phi$, denoted $\\partial\\Phi(x)$, is defined by\n %\n\\begin{eqnarray*}\n\\partial\\Phi(x):=\\{s\\in\\mathbb R^d: \\exists x^k\\to x,\\;\\Phi(x^k)\\to\\Phi(x)\\;\\;\n{\\rm and} \\;\\; s^k\\in\\hat\\Phi(x^k)\\to s\\;\\;{\\rm as}\\; k\\to+\\infty\\}.\n\\end{eqnarray*}\n","sha256_utf8":"3f23a983a3d884692be6621a02ece2aefb5c602e9c75fc908a9500d297951dae"},{"start":19101,"end":19769,"start_line":361,"end_line":371,"raw_tex":"Let $(x^k,s^k)\\in {\\rm Graph}\\; \\partial\\Phi:=\\{(x,s)\\in\\mathbb R^d\\times\\mathbb R^d: s\\in\\partial \\Phi(x)\\}.\n$\nLet $(x^k,s^k)\\to (x^*,s^*)$ as $k\\to\\infty$. Since $s^k\\in\\partial \\Phi(x^k)$, \nand $\\Phi(x^k)\\to \\Phi(x^*)$ as $k\\to\\infty$ we have  $(x^*,s^*)\\in {\\rm Graph}\\; \\partial\\Phi$.\nThe well-known Fermat's rule ``$x\\in\\mathbb R^d$ is a local minimizer of $\\Phi$, then $\\partial \\Phi(x)\\ni 0$''\nremains unchanged. If $x\\in\\mathbb R^d$ such that $\\partial \\Phi(x)\\ni 0$ the point $x$ is called \na critical point. We denote by ${\\rm crit}\\; \\Phi$ the set of {\\it critical points} of $\\Phi$,\nthat is \n$\n{\\rm crit}\\; \\Phi=\\{x\\in\\mathbb R^d:0\\in\\partial\\Phi(x)\\}.\n$\n","sha256_utf8":"f509d5d522b34aeaae1b5fe6b6a196c3f9645222e2366f091815a295879e5a9c"},{"start":19955,"end":21074,"start_line":378,"end_line":394,"raw_tex":" Let $\\Omega$ be a subset of $\\mathbb R^d$  and $x$ is any point in $\\mathbb R^d$. The distance from $x$\nto $\\Omega$, denoted ${\\rm dist}(x,\\Omega)$, is defined by \n${\\rm dist}(x,\\Omega)=\\inf\\{\\|x-z\\|:\\;z\\in\\Omega\\}$, and the set of points in $\\Omega$ that \nachieve this infimum (the projection of $x$ onto $\\Omega$) is defined by ${\\rm Proj}_{\\Omega}(x)$.\nThe set ${\\rm Proj}_{\\Omega}(x)$ becomes a singleton if $\\Omega$ is a closed convex \nset. If $\\Omega=\\emptyset$, then ${\\rm dist}(x,\\Omega)=+\\infty$ for all $x\\in\\mathbb R^d$. \nFor any real-valued function $\\Phi$ on $\\mathbb R^d$ we have\n\\[\n{\\rm dist}(0,\\partial\\Phi(x))=\\inf\\{\\|s^*\\|: \\; s^*\\in\\partial \\Phi(x)\\}.\n\\]  \n%\nLet $F:\\mathbb R^n\\times\\mathbb R^m\\to \\bar{\\mathbb R}$ be a lower semicontinuous function. The subdifferentiation of $F$ at the point $(\\hat x,\\hat y)$ is defined by \n$\n\\partial F(\\hat x, \\hat y)=\\big(\\partial_x F(\\hat x,\\hat y), \\partial_y F(\\hat x,\\hat y)\\big),\n$\nwhere $\\partial_x F$ and $\\partial_y F$ are respectively the differential of the function \n$F(\\cdot,y)$ when $y\\in\\mathbb R^m$ is fixed, and $F(x,\\cdot)$ when $x$ is fixed.\n","sha256_utf8":"560ee069cfdd0166760de9735d0a17f6700113967cfec6b04aaaf001dec7d905"}]},{"id":"S006","semantic_type":"notation","kind":"notation","title":"函数值层集与凹脱奇异函数","source_units":[{"start":22414,"end":22864,"start_line":435,"end_line":441,"raw_tex":"Let $\\Phi:\\mathbb R^d\\to \\bar{\\mathbb R}$  be a proper lower semicontinuous function. \nFor $-\\infty<\\eta_1<\\eta_2\\le +\\infty$, we define \n$[\\eta_1<\\Phi<\\eta_2] =\\{x\\in\\mathbb R^d: \\; \\eta_1<\\Phi(x)<\\eta_2\\}.$\n%\nLet $\\eta\\in (0,+\\infty]$. We denote by $\\Psi_{\\eta}$ the set of all continuous concave functions \n $\\psi:[0,\\eta]\\to [0,+\\infty)$ that $\\psi(0)=0$ and\n$\\psi$ is continuously differentiable on $(0,\\eta)$ with $\\psi'(s)>0$ over $(0,\\eta)$.\n","sha256_utf8":"4e77acbd56be9321f040a533e31cc7bbbc2be5fbd5a3f52b4010114e068b0ec7"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"点上及集合上的KL性质","source_units":[{"start":22867,"end":23482,"start_line":444,"end_line":455,"raw_tex":"\\begin{definition}\\label{KLprop}\nLet $\\Phi:\\mathbb R^d\\to \\bar{\\mathbb R}$  be a proper  lower semicontinuous function.\nWe say that $\\Phi$ has the Kurdyka-{\\L}ojasiewicz (K{\\L}) property at \n$x^*\\in {\\rm dom} \\;\\partial \\Phi$ if there exists a neighborhood $U$ of $x^*$, \n$\\eta\\in(0,\\infty)$ and a continuous concave function $\\psi\\in\\Psi_{\\eta}$ such that\n%\n\\begin{eqnarray}\\label{KLine}\n\\psi'\\big(\\Phi(x)-\\Phi(x^*)\\big) {\\rm dist}\\big(0,\\partial \\Phi(x)\\big)\\ge 1.\n\\end{eqnarray}\n If $\\Phi$ satisfies the  property at each point of $ {\\rm dom} \\;\\partial \\Phi$, then $\\Phi$ is a \n{K\\L} function. \n\\end{definition}","sha256_utf8":"e76027cffadd9c795ba47740456ab943173fa9d03fb7ad3473d9903300fe9287"},{"start":23484,"end":24046,"start_line":457,"end_line":465,"raw_tex":"\\begin{definition}\\label{KLset}\nLet $\\Phi:\\mathbb R^d\\to \\bar{\\mathbb R}$  be a proper  lower semicontinuous function that\n takes constant value on $\\Omega$ and satisfies the {K\\L} property at each point of $\\Omega$.\nWe say $\\Phi$ satisfies a K{\\L} property on $\\Omega$ if there exists\n$\\epsilon>0$, $\\eta>0$, and $\\psi\\in\\Psi_{\\eta}$ such that \nfor every $x^*\\in\\Omega$ and every element $x$ belongs to the intersection\n$ \\{x\\in\\mathbb R^d: {\\rm dist} (x,\\Omega)<\\epsilon\\} \\cap [\\Phi(x^*) <\\Phi(x) <   \\Phi(x^*)+\\eta],$\n (\\ref{KLine}) holds. \n \\end{definition}","sha256_utf8":"dc6a892339d4a25a2a1185b35a52fb6e78c8d53f9d9678db6b2ad4d8ebee4026"}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","title":"目标函数与约束矩阵假设","source_units":[{"start":26470,"end":27348,"start_line":511,"end_line":524,"raw_tex":"\\begin{itemize}\\label{assumptionA}\n\\item[{A1.}] $f_i, i=1,\\dots, p $ are lower-semicontinuous and coercive;\n\\item[{A2.}] $h$ is bounded from below, Lipschitz differentiable with constant $L_h$;\n\\item[{A3.}] ${\\rm Im}({\\bf A})\\subseteq {\\rm Im}(B)$ and $b\\in {\\rm Im}(B)$;\n%\\item[{A4.}] $\\exists\\eta>0$ such that $B\\tr B\\succeq \\eta I_q$ and $A_i\\tr A_i\\succeq \\eta I_{n_i}, i=1,\\dots,p$;\n\\item[{A4.}] For $i=1,\\dots, p$ one of the followings hold: \n\\begin{itemize}\n\\item[(i)] $\\exists q_i>0$ such that $Q_i\\succeq q_i I_{n_i}$; or \n\\item[(ii)] $\\exists \\epsilon_i > 0$ such that \n$f_i(v)-f_i(\\tilde v)- \\langle \\tilde s_i , v-\\tilde v \\rangle \\ge -\\frac{\\epsilon_i}{2} \\| v-\\tilde v\\|^2$ \nfor  $v,\\tilde v\\in\\mathbb R^{n_i}$ and $\\tilde s_i\\in\\partial f_i(\\tilde v)$. \nThis is equivalent to the weak convexity of the function $f_i$, $i=1,\\dots,p$.\n \\end{itemize}\n \\end{itemize}\n","sha256_utf8":"46569f889da1620518d29912d04c404c916746941374925d34dcbf3ca706ddf6"}]},{"id":"S009","semantic_type":"notation","kind":"notation","title":"对偶估计常数与下降矩阵","source_units":[{"start":27464,"end":27992,"start_line":527,"end_line":539,"raw_tex":"\\begin{eqnarray}\\label{quantities}\n\\left\\{\n\\begin{tabular}{ll}\n$\\rho(\\beta):=1-|1-\\beta| $,&  \\\\\n$c_1:=\\displaystyle{\\frac{1}{\\alpha\\rho(\\beta)\\lambda_{++}^{B\\tr B}}}$,&\n$c_2:= \\displaystyle{  \\frac{2\\beta}{\\alpha\\rho(\\beta)^2\\lambda_{++}^{B\\tr B}}}$, \\\\\n$c_3:=c_2\\|P\\|^2$, & $c_4:=   c_2 (\\|P\\|+L_h)^2$, \\\\\n$c_5:=\\beta^{-1}|1-\\beta| c_1$, \n&$\\displaystyle{c_6:=\\frac{|1-\\beta| }{2\\alpha\\beta^2\\lambda_{++}^{B\\tr B}}}$, \\\\\n$D=2P+\\alpha B\\tr B-L_h I_m$, &$\\bar D=D-2\\epsilon_0(c_3+c_4)I$, \\\\\n\\end{tabular}\n \\right.\n\\end{eqnarray}","sha256_utf8":"8a17376d5b908f5b98175471ec42a48eb2717d4372594e8db320db82b8141c48"},{"start":27993,"end":28148,"start_line":540,"end_line":541,"raw_tex":"where $\\epsilon_0>1$, $\\lambda_{++}^{B\\tr B}$ is  the smallest \nstrictly-positive eigenvalue of $B\\tr B$, and $I_m$ denotes a $m\\times m$ identity matrix.\n","sha256_utf8":"883074af689ddc2d320035aaefb507a8f11aab5330c229fa500470093f6b134c"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"修正势函数及其迭代值","source_units":[{"start":46977,"end":47370,"start_line":1065,"end_line":1072,"raw_tex":"We define a modified version of the augmented Lagrangian function (\\ref{augL}).\nWe consider $\\bar{\\mathcal L}:\\mathbb R^n\\times \\mathbb R^q\\times \\mathbb R^m\\times \\mathbb R^q\\times \\mathbb R^m\\to (-\\infty,+\\infty] $ defined by \n%\n\\begin{eqnarray}\\label{Lbar}\n\\bar{\\mathcal L}(x,y,z,y',z')=\\mathcal L^{\\alpha}(x,y,z)+\n\\epsilon_0 c_5 \\|B\\tr(z-z')\\|^2+\n\\epsilon_0  c_3 \\|y-y'\\|^2,\n\\end{eqnarray}","sha256_utf8":"51ac1e56493ad5b4cc5aed53db0fa534c54982070fd8bb3e8a9f6fe06d85a670"},{"start":47373,"end":47882,"start_line":1074,"end_line":1083,"raw_tex":"where $\\epsilon_0>1$, and $c_3$ and $c_5$ are given in (\\ref{quantities}). \nEvaluating $\\bar{\\mathcal L}$ at the point $({\\bf x}^{k}, y^{k},z^{k},y^{k-1},z^{k-1})$ where \n$\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 1}$ is the sequence generated by the PADMM algorithm (\\ref{padmm})\ngives\n\\begin{eqnarray}\\label{Lbark}\n\\bar{\\mathcal L}_k:=\n\\bar{\\mathcal L}({\\bf x}^{k}, y^{k},z^{k},y^{k-1},z^{k-1})=\\mathcal L^{\\alpha}({\\bf x}^k,y^k,z^k)+\n\\epsilon_0 c_5 \\|B\\tr\\Delta z^k\\|^2+\n\\epsilon_0 c_3 \\|\\Delta y^k\\|^2.\n\\end{eqnarray}","sha256_utf8":"17b7232908cde026635c6b4629bc977b02176f07e569355b2df6ade97983a746"}]},{"id":"S011","semantic_type":"theorem","kind":"theorem","title":"迭代有限长度与驻点收敛","source_units":[{"start":61778,"end":63079,"start_line":1463,"end_line":1494,"raw_tex":"\\begin{theorem}\\label{conv}\nSuppose that  A1-A4 hold and $\\sigma$ defined in Lemma \\ref{thm1} is positive.\nLet the sequence \n$\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ generated by the PADMM algorithm (\\ref{padmm}) be bounded and \n$\\bar{\\mathcal L}({\\bf x},y,z,y',z')$ satisfies the {K\\L} property on $\\Omega$ (\\ref{omega});\nthat is, for every \n$v^*:=({\\bf x}^*, y^*, z^*,  y^*, z^*) \\in \\Omega$\nthere exists $\\epsilon>0$, $\\eta\\in [0,+\\infty)$, and a function\n$\\psi\\in\\Psi_{\\eta}$ such that for any  \n$v:=({\\bf x},y,z,y',z')\\in\\mathcal S$ where \n%\n\\begin{eqnarray}\\label{eq:intersec}\n\\mathcal S:=\\Big\\{v \\in \\mathbb R^n\\times\\mathbb R^m\\times\\mathbb R^p\\times\\mathbb R^m\\times\\mathbb R^p:\n\\;{\\rm dist}(v,\\Omega)<\\epsilon\\} \\cap [\\bar{\\mathcal L}(v^*) <\\bar{\\mathcal L}< \\bar{\\mathcal L}(v^*)+\\eta],\n\\end{eqnarray}\nit holds \n%\n\\begin{eqnarray}\\label{eq:thm1KL}\n\\psi'\\Big(\\bar{\\mathcal L}(v)-\\bar{\\mathcal L}(v^*)\\Big) \\cdot\n{\\rm dist}\\big(0,\\partial\\bar{\\mathcal L}(v)\\big)\\ge 1.\n\\end{eqnarray}\n%\n Then $\\{u^k\\}_{k\\ge 0}:=\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$\n satisfies \n\\[\n\\sum_{k=1}^{\\infty} |||\\Delta u^k|||\\le \\sum_{k=1}^{\\infty} \n\\Big\\{\\sum_{i=1}^p\\| \\Delta x_i^{k}\\|\n+ \\| \\Delta y^{k}\\|\n+ \\| \\Delta z^{k}\\|\\Big\\}<+\\infty,\n\\]\nand consequently it converges to a stationary point of (\\ref{OPE}).\n\\end{theorem}","sha256_utf8":"4e62a748c730eeca3563d93bc2ba564d66ab38a7e75b050fb1827954dab543c7"}]},{"id":"S012","semantic_type":"proof","kind":"proof","title":"有限长度收敛定理完整证明","source_units":[{"start":63102,"end":70515,"start_line":1496,"end_line":1745,"raw_tex":"\\begin{proof}\nBy Lemma \\ref{thm1}, the sequence $\\{\\bar{\\mathcal L}_k\\}_{k\\ge 1}$ is monotonically\n decreasing and converges. Let $\\bar{\\mathcal L}_*:=\\lim_{k\\to\\infty}\\bar{\\mathcal L}_k$, then \nthe error sequence \n$e_k:=\\bar{\\mathcal L}_k-\\bar{\\mathcal L}_*, k\\ge 1$\nis non-negative, monotonically decreasing, and converges to zero. \nWe prove under two cases.\n\n {\\it Case 1:} let assume that \nthere is a $k_1\\in\\mathbb Z_+$ such that $e_{k_1}=0$. \nSince $\\{e_k\\}_{k\\ge 1}$ is monotonically decreasing, \n$e_k=0$ for all $k\\ge k_1$.\nBy Lemma \\ref{thm1}(i) then we have\n\\[\n\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|^2+\\|\\Delta y^{k+1}\\|^2+\\|\\Delta z^{k+1}\\|^2 \\le \n \\frac{1}{\\sigma}(e_k-e_{k+1})=0,\\quad \\forall k\\ge k_1.\n\\]\nHence  as $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ is a bounded sequence\nthen \n\\[\n\\sum_{k=1}^{\\infty}\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|\n+\\|\\Delta y^{k}\\|\n+\\|\\Delta z^{k}\\|\\Big\\}\n=\\sum_{k=1}^{k_1} \\Big\\{\n\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|\n+\\|\\Delta y^{k}\\|\n+\\|\\Delta z^{k}\\|\\Big\\}\n\\] \nis bounded.\n\n{\\it Case 2:}\nwe let  $e_k>0$ for all $k\\ge 1$. \nBy Lemma \\ref{thm1}(i) we have\n%\n\\begin{eqnarray}\\label{s1}\n%|||\\Delta u^{k+1}|||^2=\n\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|^2+\\|\\Delta y^{k+1}\\|^2+\\|\\Delta z^{k+1}\\|^2 \\le \n \\frac{1}{\\sigma}( e_k-e_{k+1}),\\quad \\forall k\\ge 1.\n\\end{eqnarray}\n%\nBy Lemma \\ref{lma-cluster2}, $\\Omega$ is nonempty, compact, and connected,\nand $\\bar{\\mathcal L}$ takes on a constant value $\\bar{\\mathcal L}_*$ on $\\Omega$. \nSince $\\bar{\\mathcal L}_{k}\\searrow\\bar{\\mathcal L}_*$ as $k\\to\\infty$ and $\\bar{\\mathcal{L}}$ satisfies \nthe {K\\L} property then there exists  $ k_1\\ge 1$ such that \n$\\bar{\\mathcal L}_*<\\bar{\\mathcal L}_k<\\bar{\\mathcal L}_*+\\eta$\nfor all $k\\ge k_1$ and some $\\eta>0$.\n%\nMoreover, $\\lim_{k\\to\\infty} {\\rm dist} \\big[ ({\\bf x}^k,y^k,z^k,y^{k-1},x^{k-1}), \\Omega\\big]=0$ by\nLemma \\ref{lma-cluster2}, \nthus for a given $\\epsilon>0$ there exists $k_2\\ge 1$ such that for all $k\\ge k_2$ it holds\n\\[\n{\\rm dist} \\Big[ ({\\bf x}^k,y^k,z^k,y^{k-1},x^{k-1}), \\Omega\\Big]<\\epsilon. \n\\]\nLet  $\\tilde k:=\\max\\{ k_1,k_2,3 \\}$, \nthen $({\\bf x}^k,y^k,z^k,y^{k-1},x^{k-1})\\in\\mathcal S$\n and\n%\n$\n\\psi'(\\bar{\\mathcal L}_k-\\bar{\\mathcal L}_*)\\cdot {\\rm dist}\\Big(0, \n\\partial \\bar{\\mathcal L}_k \\Big)\n\\ge 1$ for all $k\\ge \\tilde k$, or equivalently\n%\n\\begin{eqnarray}\\label{KL}\n\\psi'(e_k)\n\\cdot {\\rm dist}\\Big(0, \n\\partial \\bar{\\mathcal L}_k \\Big)\n \\ge1.\n\\end{eqnarray}\n%\nMultiply both sides of this inequality by $ |||\\Delta u^{k+1}|||^2 $ to get\n%\n\\begin{eqnarray}\\label{eq:001}\n \\psi'(e_{k}) \\cdot {\\rm dist}\\Big(0, \n\\partial \\bar{\\mathcal L}_k \\Big) |||\\Delta u^{k+1}|||^2 \\ge |||\\Delta u^{k+1}|||^2,\n\\end{eqnarray}\nwhere \n$|||\\Delta u^k|||^2=\\sum_{i=1}^p\\|\\Delta x_i^k\\|^2+\\|\\Delta y^k\\|^2+\\|\\Delta z^k\\|^2.$\n%\nBy (\\ref{s1}), (\\ref{eq:001}) leads to \n\\begin{eqnarray}\\label{eq:002}\n \\frac {1}{\\sigma} \\psi'(e_{k}) (e_k-e_{k+1})\\cdot {\\rm dist}\\Big(0, \n\\partial \\bar{\\mathcal L}_k \\Big)  \\ge |||\\Delta u^{k+1}|||^2.\n\\end{eqnarray}\n%\nBy the concavity of $\\psi$, $\\psi(e_{k}) -\\psi(e_{k+1})\\ge  \\psi'(e_{k}) (e_{k} - e_{k+1})$, hence\n%\n\\begin{eqnarray}\\label{eq:003}\n|||\\Delta u^{k+1}|||^2 \\le \n \\frac{1}{\\sigma} \\big(\\psi(e_{k}) -\\psi(e_{k+1})\\big) \n \\cdot {\\rm dist}(0, \\partial \\bar{\\mathcal L}_k ).\n\\end{eqnarray}\nBy the arithmetic mean-geometric mean inequality, for any $\\gamma>0$ we  have\n%\n\\begin{eqnarray}\\label{artith-geo}\n\\sqrt{ \\frac{1}{\\sigma} \\big(\\psi(e_{k}) -\\psi(e_{k+1})\\big) \n \\cdot {\\rm dist}(0, \\partial \\bar{\\mathcal L}_k )}\n \\le \\frac{\\gamma}{2\\sigma}  \\big(\\psi(e_{k}) -\\psi(e_{k+1})\\big) \n+\\frac{1}{2\\gamma}{\\rm dist}(0, \\partial \\bar{\\mathcal L}_k ).\n \\end{eqnarray}\n%\nMoreover,\n\\begin{eqnarray}\\label{eq:004}\n\\frac{1}{\\sqrt{p+2}} \\Big(\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\\Big)\n\\le |||\\Delta u^{k+1}|||.\n\\end{eqnarray}\n%\nThus  (\\ref{eq:003}), (\\ref{artith-geo}), and (\\ref{eq:004}) give rise to\n%\n\\begin{eqnarray}\\label{main}\n\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n\\le \\frac{\\gamma\\sqrt{p+2}}{2\\sigma}  \n\\big(\\psi(e_{k}) -\\psi(e_{k+1})\\big) \n+\\frac{\\sqrt{p+2}}{2\\gamma}{\\rm dist}(0, \\partial \\bar{\\mathcal L}_k ).\n\\end{eqnarray}\n%\nNote that ${\\rm dist}(0, \\partial \\bar{\\mathcal L}_k)=|||s^k|||$, where $s^k\\in\\partial \\bar{\\mathcal L}_k$ \ndefined in Lemma \\ref{lma-D}, hence \n%\n\\begin{eqnarray}\\label{eq:import}\n\\begin{array}{c}\n\\displaystyle{\n\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n}\\\\[.1in]\n\\le \n\\displaystyle{\n\\frac{\\gamma\\sqrt{p+2}}{2\\sigma}  \n\\big(\\psi(e_{k}) -\\psi(e_{k+1})\\big) \n+\n\\frac{\\tilde \\rho\\sqrt{p+2}}{2\\gamma}\n\\Big(\n\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|\n+\\|\\Delta y^{k}\\|\n+\\|\\Delta z^{k}\\|\\Big).\n}\n\\end{array}\n\\end{eqnarray}\n%\nFor any $K\\ge k\\ge \\underline{k}$, the following identity holds\n\\[\n\\sum_{k=\\underline k}^K\\|\\Delta w^{k}\\| =\n\\sum_{k=\\underline k}^{K}\n\\|\\Delta w^{k+1}\\|\n+\\|\\Delta w^{\\underline k}\\|-\\|\\Delta w^{K+1}\\|.\n\\]\nChoose $\\gamma>0$ large enough such that $2\\gamma>\\tilde \\rho\\sqrt{p+2}$.\nLet $\\delta_0=1-\\frac{\\tilde \\rho\\sqrt{p+2}}{2\\gamma}>0$.\n%\nSumming up (\\ref{eq:import}) from $k=\\underline k\\ge \\tilde k $ to $K\\ge \\underline k$ leads to\n\\begin{eqnarray*}\n&\\sum_{k=\\underline k}^K\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\\Big\\}\n\\le\n\\delta_1\n\\big(\\psi(e_{\\underline k}) -\\psi(e_{K+1})\\big) \n&\\\\\n&\n+\\delta_2\n\\Big(\n\\sum_{i=1}^p\\|\\Delta x_i^{\\underline k}\\|\n+\\|\\Delta y^{\\underline k}\\|\n+\\|\\Delta z^{\\underline k}\\|\\Big)\n-\\delta_2\n\\Big(\n\\sum_{i=1}^p\\|\\Delta x_i^{K+1}\\|\n+\\|\\Delta y^{K+1}\\|\n+\\|\\Delta z^{K+1}\\|\\Big),&\n\\end{eqnarray*}\nwhere $\\delta_1:={\\gamma\\sqrt{p+2}}/{2\\sigma\\delta_0} $ \nand \n$\\delta_2= {\\tilde \\rho\\sqrt{p+2}}/{2\\gamma\\delta_0}$.\n%\nRecall that $e_k$ is monotonically decreasing and\n$\\psi'>0$, therefore\n%\n\\begin{eqnarray*}\n&\\sum_{k=\\underline k}^K\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n\\Big\\}\\le  \n\\delta_1\n\\psi(e_{\\underline k})\n+\\delta_2\\big(\n\\sum_{i=1}^p\\|\\Delta x_i^{\\underline k}\\|\n+\\|\\Delta y^{\\underline k}\\|\n+\\|\\Delta z^{\\underline k}\\|\\big).\n&\n\\end{eqnarray*}\n%\nThe right hand side of this inequality is bounded for any $K\\ge \\underline k$, we let $K\\to\\infty$ to obtain\n\\begin{eqnarray}\\label{eq:0005}\n\\sum_{k=\\underline k}^{\\infty}\n\\Big(\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n\\Big)\n\\le \n\\delta_1\n\\psi(e_{\\underline k})\n+\\delta_2\n\\Big(\n\\sum_{i=1}^p\\|\\Delta x_i^{\\underline k}\\|\n+\\|\\Delta y^{\\underline k}\\|\n+\\|\\Delta z^{\\underline k}\\|\\Big).\n\\end{eqnarray}\n%\nThen (\\ref{eq:0005}) together with the fact that $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ is  bounded gives\n%\n\\begin{eqnarray}\\label{eq:000000005}\n\\sum_{k=0}^{\\infty}\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n\\Big\\}\n<+\\infty.\n\\end{eqnarray}\n\n\nFor any $p,q,K\\in\\mathbb Z_+$ where $q\\ge p$ we have \n\\begin{eqnarray*}\n|||u^q-u^p||| =||| \\sum_{k=p}^{q-1} \\Delta u^{k+1}|||\n\\le\n \\sum_{k=p}^{q-1} |||\\Delta u^{k+1}|||\n \\le \n\\sum_{k=1}^{\\infty}|||\\Delta u^{k+1}|||.\n\\end{eqnarray*}\n%\nSince by (\\ref{eq:000000005}) the right hand side is finite \nthen \n\\[\n\\sum_{k=1}^{\\infty}|||\\Delta u^{k+1}|||\n\\le \\sum_{k=0}^{\\infty}\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n\\Big\\}\n<+\\infty.\n\\]\nThis implies that $\\{u^k\\}_{k\\ge 0}=\\{({\\bf x}, y^k,z^k)\\}_{k\\ge 0}$ is a Cauchy sequence. \nBy Lemma \\ref{thm-cluster} it converges to a stationary point.  $\\square$\n \\end{proof}","sha256_utf8":"c28a6af64bb19720e70343a83fe4684cc2eaf07b3050a4c9850056e690039bf1"}]},{"id":"S013","semantic_type":"proof_step","kind":"proof_step","title":"建立非负递减势函数误差","source_units":[{"start":63102,"end":63463,"start_line":1496,"end_line":1502,"raw_tex":"\\begin{proof}\nBy Lemma \\ref{thm1}, the sequence $\\{\\bar{\\mathcal L}_k\\}_{k\\ge 1}$ is monotonically\n decreasing and converges. Let $\\bar{\\mathcal L}_*:=\\lim_{k\\to\\infty}\\bar{\\mathcal L}_k$, then \nthe error sequence \n$e_k:=\\bar{\\mathcal L}_k-\\bar{\\mathcal L}_*, k\\ge 1$\nis non-negative, monotonically decreasing, and converges to zero. \nWe prove under two cases.\n","sha256_utf8":"272c48d4f96fddb83ba9bd31c2f035414fff8505fb1065dfd5ad59e050bbbfb3"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"误差归零时迭代有限停止","source_units":[{"start":63464,"end":64104,"start_line":1504,"end_line":1525,"raw_tex":" {\\it Case 1:} let assume that \nthere is a $k_1\\in\\mathbb Z_+$ such that $e_{k_1}=0$. \nSince $\\{e_k\\}_{k\\ge 1}$ is monotonically decreasing, \n$e_k=0$ for all $k\\ge k_1$.\nBy Lemma \\ref{thm1}(i) then we have\n\\[\n\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|^2+\\|\\Delta y^{k+1}\\|^2+\\|\\Delta z^{k+1}\\|^2 \\le \n \\frac{1}{\\sigma}(e_k-e_{k+1})=0,\\quad \\forall k\\ge k_1.\n\\]\nHence  as $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ is a bounded sequence\nthen \n\\[\n\\sum_{k=1}^{\\infty}\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|\n+\\|\\Delta y^{k}\\|\n+\\|\\Delta z^{k}\\|\\Big\\}\n=\\sum_{k=1}^{k_1} \\Big\\{\n\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|\n+\\|\\Delta y^{k}\\|\n+\\|\\Delta z^{k}\\|\\Big\\}\n\\] \nis bounded.\n","sha256_utf8":"b3519d0048e8cd9ddf4ba820f692eb76bcd5191f0cea38dae06d9dbd6686ceff"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"正误差情形的平方增量界","source_units":[{"start":64105,"end":64390,"start_line":1527,"end_line":1535,"raw_tex":"{\\it Case 2:}\nwe let  $e_k>0$ for all $k\\ge 1$. \nBy Lemma \\ref{thm1}(i) we have\n%\n\\begin{eqnarray}\\label{s1}\n%|||\\Delta u^{k+1}|||^2=\n\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|^2+\\|\\Delta y^{k+1}\\|^2+\\|\\Delta z^{k+1}\\|^2 \\le \n \\frac{1}{\\sigma}( e_k-e_{k+1}),\\quad \\forall k\\ge 1.\n\\end{eqnarray}\n","sha256_utf8":"1a23f3c5f4c4350e94d3cb1352717a8f78e4de0cd3238487e24bd7edb82c2c4d"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"迭代最终进入KL邻域层集","source_units":[{"start":64392,"end":65125,"start_line":1537,"end_line":1549,"raw_tex":"By Lemma \\ref{lma-cluster2}, $\\Omega$ is nonempty, compact, and connected,\nand $\\bar{\\mathcal L}$ takes on a constant value $\\bar{\\mathcal L}_*$ on $\\Omega$. \nSince $\\bar{\\mathcal L}_{k}\\searrow\\bar{\\mathcal L}_*$ as $k\\to\\infty$ and $\\bar{\\mathcal{L}}$ satisfies \nthe {K\\L} property then there exists  $ k_1\\ge 1$ such that \n$\\bar{\\mathcal L}_*<\\bar{\\mathcal L}_k<\\bar{\\mathcal L}_*+\\eta$\nfor all $k\\ge k_1$ and some $\\eta>0$.\n%\nMoreover, $\\lim_{k\\to\\infty} {\\rm dist} \\big[ ({\\bf x}^k,y^k,z^k,y^{k-1},x^{k-1}), \\Omega\\big]=0$ by\nLemma \\ref{lma-cluster2}, \nthus for a given $\\epsilon>0$ there exists $k_2\\ge 1$ such that for all $k\\ge k_2$ it holds\n\\[\n{\\rm dist} \\Big[ ({\\bf x}^k,y^k,z^k,y^{k-1},x^{k-1}), \\Omega\\Big]<\\epsilon. \n\\]\n","sha256_utf8":"325e22404bf877ec3b02e4d5846aac807ffadf919a71bd74f8a0cb3b9abf9371"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"沿迭代应用一致KL不等式","source_units":[{"start":65125,"end":65501,"start_line":1550,"end_line":1564,"raw_tex":"Let  $\\tilde k:=\\max\\{ k_1,k_2,3 \\}$, \nthen $({\\bf x}^k,y^k,z^k,y^{k-1},x^{k-1})\\in\\mathcal S$\n and\n%\n$\n\\psi'(\\bar{\\mathcal L}_k-\\bar{\\mathcal L}_*)\\cdot {\\rm dist}\\Big(0, \n\\partial \\bar{\\mathcal L}_k \\Big)\n\\ge 1$ for all $k\\ge \\tilde k$, or equivalently\n%\n\\begin{eqnarray}\\label{KL}\n\\psi'(e_k)\n\\cdot {\\rm dist}\\Big(0, \n\\partial \\bar{\\mathcal L}_k \\Big)\n \\ge1.\n\\end{eqnarray}\n","sha256_utf8":"d55c36e40f4aabb093ea65cef9ab676b298e65488b1599da9d6bca29576bc1de"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"结合KL界与充分下降","source_units":[{"start":65503,"end":66066,"start_line":1566,"end_line":1579,"raw_tex":"Multiply both sides of this inequality by $ |||\\Delta u^{k+1}|||^2 $ to get\n%\n\\begin{eqnarray}\\label{eq:001}\n \\psi'(e_{k}) \\cdot {\\rm dist}\\Big(0, \n\\partial \\bar{\\mathcal L}_k \\Big) |||\\Delta u^{k+1}|||^2 \\ge |||\\Delta u^{k+1}|||^2,\n\\end{eqnarray}\nwhere \n$|||\\Delta u^k|||^2=\\sum_{i=1}^p\\|\\Delta x_i^k\\|^2+\\|\\Delta y^k\\|^2+\\|\\Delta z^k\\|^2.$\n%\nBy (\\ref{s1}), (\\ref{eq:001}) leads to \n\\begin{eqnarray}\\label{eq:002}\n \\frac {1}{\\sigma} \\psi'(e_{k}) (e_k-e_{k+1})\\cdot {\\rm dist}\\Big(0, \n\\partial \\bar{\\mathcal L}_k \\Big)  \\ge |||\\Delta u^{k+1}|||^2.\n\\end{eqnarray}\n","sha256_utf8":"c352e9a721bb0ad080de9e09915f22fa42abd0fc909b633e97f119b11f61145c"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"凹性转换为脱奇异函数差","source_units":[{"start":66068,"end":66351,"start_line":1581,"end_line":1587,"raw_tex":"By the concavity of $\\psi$, $\\psi(e_{k}) -\\psi(e_{k+1})\\ge  \\psi'(e_{k}) (e_{k} - e_{k+1})$, hence\n%\n\\begin{eqnarray}\\label{eq:003}\n|||\\Delta u^{k+1}|||^2 \\le \n \\frac{1}{\\sigma} \\big(\\psi(e_{k}) -\\psi(e_{k+1})\\big) \n \\cdot {\\rm dist}(0, \\partial \\bar{\\mathcal L}_k ).\n\\end{eqnarray}\n","sha256_utf8":"3ca1f838bc6d57644707c9647a4fedfa847303e70b2e6d2f35ebf571cd36e836"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"均值不等式分离乘积项","source_units":[{"start":66351,"end":66726,"start_line":1588,"end_line":1595,"raw_tex":"By the arithmetic mean-geometric mean inequality, for any $\\gamma>0$ we  have\n%\n\\begin{eqnarray}\\label{artith-geo}\n\\sqrt{ \\frac{1}{\\sigma} \\big(\\psi(e_{k}) -\\psi(e_{k+1})\\big) \n \\cdot {\\rm dist}(0, \\partial \\bar{\\mathcal L}_k )}\n \\le \\frac{\\gamma}{2\\sigma}  \\big(\\psi(e_{k}) -\\psi(e_{k+1})\\big) \n+\\frac{1}{2\\gamma}{\\rm dist}(0, \\partial \\bar{\\mathcal L}_k ).\n \\end{eqnarray}\n","sha256_utf8":"c0242768716f3f29d6214d10f05e79e2119e0efbbbbb76308e92e5fb47985c33"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"乘积范数控制分块增量和","source_units":[{"start":66728,"end":66914,"start_line":1597,"end_line":1603,"raw_tex":"Moreover,\n\\begin{eqnarray}\\label{eq:004}\n\\frac{1}{\\sqrt{p+2}} \\Big(\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\\Big)\n\\le |||\\Delta u^{k+1}|||.\n\\end{eqnarray}\n","sha256_utf8":"95e051c0aa25280f23b6c26e5e89317b2243f818953a0e99138a63a569e1c6ba"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"得到一步长度估计","source_units":[{"start":66916,"end":67258,"start_line":1605,"end_line":1614,"raw_tex":"Thus  (\\ref{eq:003}), (\\ref{artith-geo}), and (\\ref{eq:004}) give rise to\n%\n\\begin{eqnarray}\\label{main}\n\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n\\le \\frac{\\gamma\\sqrt{p+2}}{2\\sigma}  \n\\big(\\psi(e_{k}) -\\psi(e_{k+1})\\big) \n+\\frac{\\sqrt{p+2}}{2\\gamma}{\\rm dist}(0, \\partial \\bar{\\mathcal L}_k ).\n\\end{eqnarray}\n","sha256_utf8":"1ba018f88524c52fa2e6fcc6bfe216fefdaec93a7df657b439202018d0f87192"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"代入次梯度界形成长度递推","source_units":[{"start":67260,"end":67798,"start_line":1616,"end_line":1638,"raw_tex":"Note that ${\\rm dist}(0, \\partial \\bar{\\mathcal L}_k)=|||s^k|||$, where $s^k\\in\\partial \\bar{\\mathcal L}_k$ \ndefined in Lemma \\ref{lma-D}, hence \n%\n\\begin{eqnarray}\\label{eq:import}\n\\begin{array}{c}\n\\displaystyle{\n\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n}\\\\[.1in]\n\\le \n\\displaystyle{\n\\frac{\\gamma\\sqrt{p+2}}{2\\sigma}  \n\\big(\\psi(e_{k}) -\\psi(e_{k+1})\\big) \n+\n\\frac{\\tilde \\rho\\sqrt{p+2}}{2\\gamma}\n\\Big(\n\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|\n+\\|\\Delta y^{k}\\|\n+\\|\\Delta z^{k}\\|\\Big).\n}\n\\end{array}\n\\end{eqnarray}\n","sha256_utf8":"7bd87a28a14f7432bce264528eb7ff956bd3dbb431a4cde9bd1f021b5aa16bd8"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"求和移位并选择吸收参数","source_units":[{"start":67800,"end":68138,"start_line":1640,"end_line":1648,"raw_tex":"For any $K\\ge k\\ge \\underline{k}$, the following identity holds\n\\[\n\\sum_{k=\\underline k}^K\\|\\Delta w^{k}\\| =\n\\sum_{k=\\underline k}^{K}\n\\|\\Delta w^{k+1}\\|\n+\\|\\Delta w^{\\underline k}\\|-\\|\\Delta w^{K+1}\\|.\n\\]\nChoose $\\gamma>0$ large enough such that $2\\gamma>\\tilde \\rho\\sqrt{p+2}$.\nLet $\\delta_0=1-\\frac{\\tilde \\rho\\sqrt{p+2}}{2\\gamma}>0$.\n","sha256_utf8":"5801c47020650b6842bcfe229a9919c397935c3d783c5c9f0ea767c1d5e65bc2"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"望远镜求和控制有限尾段","source_units":[{"start":68140,"end":68782,"start_line":1650,"end_line":1674,"raw_tex":"Summing up (\\ref{eq:import}) from $k=\\underline k\\ge \\tilde k $ to $K\\ge \\underline k$ leads to\n\\begin{eqnarray*}\n&\\sum_{k=\\underline k}^K\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\\Big\\}\n\\le\n\\delta_1\n\\big(\\psi(e_{\\underline k}) -\\psi(e_{K+1})\\big) \n&\\\\\n&\n+\\delta_2\n\\Big(\n\\sum_{i=1}^p\\|\\Delta x_i^{\\underline k}\\|\n+\\|\\Delta y^{\\underline k}\\|\n+\\|\\Delta z^{\\underline k}\\|\\Big)\n-\\delta_2\n\\Big(\n\\sum_{i=1}^p\\|\\Delta x_i^{K+1}\\|\n+\\|\\Delta y^{K+1}\\|\n+\\|\\Delta z^{K+1}\\|\\Big),&\n\\end{eqnarray*}\nwhere $\\delta_1:={\\gamma\\sqrt{p+2}}/{2\\sigma\\delta_0} $ \nand \n$\\delta_2= {\\tilde \\rho\\sqrt{p+2}}/{2\\gamma\\delta_0}$.\n","sha256_utf8":"0e76d00e46e4ac0913409a464c4473d0bf6661ad7d6bad90441ecdb6aacac690"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"单调性给出统一尾段上界","source_units":[{"start":68784,"end":69162,"start_line":1676,"end_line":1692,"raw_tex":"Recall that $e_k$ is monotonically decreasing and\n$\\psi'>0$, therefore\n%\n\\begin{eqnarray*}\n&\\sum_{k=\\underline k}^K\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n\\Big\\}\\le  \n\\delta_1\n\\psi(e_{\\underline k})\n+\\delta_2\\big(\n\\sum_{i=1}^p\\|\\Delta x_i^{\\underline k}\\|\n+\\|\\Delta y^{\\underline k}\\|\n+\\|\\Delta z^{\\underline k}\\|\\big).\n&\n\\end{eqnarray*}\n","sha256_utf8":"f3a6979cf5222a9a717f0f2ce60db73ad5a7ae5122156815dc733a52b6e498b7"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"取极限获得无穷尾和界","source_units":[{"start":69164,"end":69594,"start_line":1694,"end_line":1709,"raw_tex":"The right hand side of this inequality is bounded for any $K\\ge \\underline k$, we let $K\\to\\infty$ to obtain\n\\begin{eqnarray}\\label{eq:0005}\n\\sum_{k=\\underline k}^{\\infty}\n\\Big(\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n\\Big)\n\\le \n\\delta_1\n\\psi(e_{\\underline k})\n+\\delta_2\n\\Big(\n\\sum_{i=1}^p\\|\\Delta x_i^{\\underline k}\\|\n+\\|\\Delta y^{\\underline k}\\|\n+\\|\\Delta z^{\\underline k}\\|\\Big).\n\\end{eqnarray}\n","sha256_utf8":"1c74e205795659bab838071112628b6e59eea96ed3f1ca7eee36d72a4ae75a1a"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"补入有限前缀得到有限长度","source_units":[{"start":69596,"end":69868,"start_line":1711,"end_line":1720,"raw_tex":"Then (\\ref{eq:0005}) together with the fact that $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ is  bounded gives\n%\n\\begin{eqnarray}\\label{eq:000000005}\n\\sum_{k=0}^{\\infty}\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n\\Big\\}\n<+\\infty.\n\\end{eqnarray}\n","sha256_utf8":"802a97ed02e2f8265627091b4e1b40e5bfb8732142834c1561f9651ad0b42a08"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"有限长度推出柯西与驻点极限","source_units":[{"start":69870,"end":70515,"start_line":1723,"end_line":1745,"raw_tex":"For any $p,q,K\\in\\mathbb Z_+$ where $q\\ge p$ we have \n\\begin{eqnarray*}\n|||u^q-u^p||| =||| \\sum_{k=p}^{q-1} \\Delta u^{k+1}|||\n\\le\n \\sum_{k=p}^{q-1} |||\\Delta u^{k+1}|||\n \\le \n\\sum_{k=1}^{\\infty}|||\\Delta u^{k+1}|||.\n\\end{eqnarray*}\n%\nSince by (\\ref{eq:000000005}) the right hand side is finite \nthen \n\\[\n\\sum_{k=1}^{\\infty}|||\\Delta u^{k+1}|||\n\\le \\sum_{k=0}^{\\infty}\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|\n+\\|\\Delta y^{k+1}\\|\n+\\|\\Delta z^{k+1}\\|\n\\Big\\}\n<+\\infty.\n\\]\nThis implies that $\\{u^k\\}_{k\\ge 0}=\\{({\\bf x}, y^k,z^k)\\}_{k\\ge 0}$ is a Cauchy sequence. \nBy Lemma \\ref{thm-cluster} it converges to a stationary point.  $\\square$\n \\end{proof}","sha256_utf8":"7fd6ba8e85d271905ef25015560576a9c3b4507bd2fc1afd2f1f9fd323df5cdb"}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","title":"光滑函数线段梯度估计","source_units":[{"start":21077,"end":21466,"start_line":397,"end_line":406,"raw_tex":"\\begin{lemma}\nLet $\\Phi:\\mathbb R^d\\to\\mathbb R$ be Fr\\'echet differentiable such that \nits gradient is Lipschitz continuous with constant $L_{\\Phi}>0$. Then \nfor every $u,v\\in\\mathbb R^d$ and every $\\xi\\in[u,v]=\\{(1-t)u+tv:\\;t\\in[0,1]\\}$\nit holds \n\\begin{eqnarray}\\label{Lip}\n\\Phi(v)\\le \\Phi(u)+\\langle \\nabla \\Phi(\\xi),v-u\\rangle +\\frac{L_{\\Phi}}{2}\\|v-u\\|^2.\n\\end{eqnarray}\n\n\\end{lemma}","sha256_utf8":"7240c80bfda2ec4073da8ddbb878ddcce69b7ce229e8f8c1cc4966a4ae8c56ae"}]},{"id":"S031","semantic_type":"proof","kind":"proof","title":"线段梯度估计完整证明","source_units":[{"start":21467,"end":22412,"start_line":407,"end_line":433,"raw_tex":"\\begin{proof}\nLet $u, v\\in\\mathbb R^d$, and $\\xi=(1-t)u+tv$ for $t\\in [0,1]$. Then we have\n%\n\\begin{eqnarray*}\n&\\Phi(v) -\\Phi(u) = \\int_0^1 \\langle \\nabla \\Phi((1-r) u+rv), v-u \\rangle dr &\\\\\n&=  \\int_0^1  \\langle \\nabla \\Phi((1-r) u+rv)-\\nabla \\Phi(\\xi), v-u \\rangle dr\n+\\langle \\nabla \\Phi(\\xi) , v-u\\rangle. &\n\\end{eqnarray*}\n\nSince $\\Phi$ has Lipschitz continuous gradients with constant $L_{\\Phi}>0$, then we have\n\\begin{eqnarray*}\n&|\\Phi(v) -\\Phi(u) - \\langle \\nabla \\Phi(\\xi) , v-u\\rangle|\n=\\Big| \\int_0^1  \\langle \\nabla \\Phi((1-r) u+rv)-\\nabla \\Phi(\\xi), v-u \\rangle dr\\Big | \n&\\\\\n&\n\\le  \\int_0^1  \\Big\\|  \\nabla \\Phi((1-r) u+rv)-\\nabla \\Phi(\\xi)\\Big\\|\\cdot \\Big\\|v-u\\Big \\|dr \n\\le  L_{\\Phi} \\|v-u\\|^2 \\int_0^1 |r-t| dr\n& \\\\ &\n=  L_{\\Phi} \n\\|v-u\\|^2 \\Big( \\int_0^t (-r+t) dr + \\int_t^1 (r-t) dr \\Big)\n& \\\\&\n=  L_{\\Phi} \\Big(\\frac 12 -t(1-t)\\Big) \\|v-u\\|^2 \n\\le \\frac {L_{\\Phi}}{2}\\|v-u\\|^2.\\quad  \\square &\n\\end{eqnarray*}\n\n\n\\end{proof}","sha256_utf8":"0446b4ea17ae77d7fc2fafc1f0412c33f07dff96fea23625b73b663c079e99a5"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"积分表示函数差并减去梯度项","source_units":[{"start":21467,"end":21796,"start_line":407,"end_line":414,"raw_tex":"\\begin{proof}\nLet $u, v\\in\\mathbb R^d$, and $\\xi=(1-t)u+tv$ for $t\\in [0,1]$. Then we have\n%\n\\begin{eqnarray*}\n&\\Phi(v) -\\Phi(u) = \\int_0^1 \\langle \\nabla \\Phi((1-r) u+rv), v-u \\rangle dr &\\\\\n&=  \\int_0^1  \\langle \\nabla \\Phi((1-r) u+rv)-\\nabla \\Phi(\\xi), v-u \\rangle dr\n+\\langle \\nabla \\Phi(\\xi) , v-u\\rangle. &\n\\end{eqnarray*}\n","sha256_utf8":"beb4c41432f10c239fbb45ac8b4356567c73a4b071a2fa9b64d26070189598c5"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"Lipschitz梯度控制积分余项","source_units":[{"start":21797,"end":22412,"start_line":416,"end_line":433,"raw_tex":"Since $\\Phi$ has Lipschitz continuous gradients with constant $L_{\\Phi}>0$, then we have\n\\begin{eqnarray*}\n&|\\Phi(v) -\\Phi(u) - \\langle \\nabla \\Phi(\\xi) , v-u\\rangle|\n=\\Big| \\int_0^1  \\langle \\nabla \\Phi((1-r) u+rv)-\\nabla \\Phi(\\xi), v-u \\rangle dr\\Big | \n&\\\\\n&\n\\le  \\int_0^1  \\Big\\|  \\nabla \\Phi((1-r) u+rv)-\\nabla \\Phi(\\xi)\\Big\\|\\cdot \\Big\\|v-u\\Big \\|dr \n\\le  L_{\\Phi} \\|v-u\\|^2 \\int_0^1 |r-t| dr\n& \\\\ &\n=  L_{\\Phi} \n\\|v-u\\|^2 \\Big( \\int_0^t (-r+t) dr + \\int_t^1 (r-t) dr \\Big)\n& \\\\&\n=  L_{\\Phi} \\Big(\\frac 12 -t(1-t)\\Big) \\|v-u\\|^2 \n\\le \\frac {L_{\\Phi}}{2}\\|v-u\\|^2.\\quad  \\square &\n\\end{eqnarray*}\n\n\n\\end{proof}","sha256_utf8":"88b8f9434806ec0033f1a455b523a0496ced9b3daf0c8d5c219a6648278b8d73"}]},{"id":"S034","semantic_type":"lemma","kind":"lemma","title":"增广拉格朗日次梯度界","source_units":[{"start":28184,"end":29003,"start_line":545,"end_line":569,"raw_tex":"\\begin{lemma}\\label{thm-d}\n{(Subgradient bound)} Let $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ be a sequence generated by the PADMM\n algorithm (\\ref{padmm}). Then there exists \n$d^{k}:=\\big(\\{d^{k}_{x_i}\\}_{i=1}^p,d^{k}_y,d^{k}_z\\big)\\in\\partial \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k})$ and  a constant $\\rho>0$\nsuch that \n%\n\\begin{eqnarray}\\label{nd}\n|||d^{k}||| &\\le&\n \\rho \\Big(\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|+ \\|\\Delta y^{k}\\|+ \\|\\Delta z^{k}\\|\\Big), \n\\end{eqnarray}\n%\nwhere %\n\\begin{eqnarray}\\label{rho}\n\\rho:=\\max\\Big\\{\n\\|{\\bf Q}\\|+\\|{\\bf A}\\|^2,\\;\\;\\;\n\\|P\\|+\n\\alpha \\|{\\bf A}\\|\\|B\\|\n, \\;\\;\\;\n\\|{\\bf A}\\|  +\\|B\\|+\\frac{1}{\\alpha\\beta}\\Big\\},\n\\end{eqnarray}\n%\n$\\|{\\bf Q}\\|:=\\sum_{i=1}^p\\|Q_i\\|$ and $\\|{\\bf A}\\|:=\\sum_{i=1}^p\\|A_i\\|$,\nand for any sequence $\\{u^k\\}_{k\\ge 0}$,\n$\\Delta u^{k+1}=u^{k+1}-u^k$.\n\\end{lemma}","sha256_utf8":"cf2238b0490aa018b5f4060893e774dfddd2e17748056f72997fb7bfa3bc3c6f"}]},{"id":"S035","semantic_type":"proof","kind":"proof","title":"增广次梯度界完整证明","source_units":[{"start":29008,"end":32174,"start_line":572,"end_line":664,"raw_tex":"\\begin{proof}\nLet $k\\ge 1$ be fixed. By the optimality condition of the $x_i$ subproblem\nof PADMM algorithm for \n$i=1,\\dots, p$ we have\n%\n\\begin{eqnarray*}\n-A_i\\tr z^{k-1}\n-\\alpha A_i\\tr (A_{<i}x_{<i}^{k}+A_ix_i^{k}+A_{>i}x_{>i}^{k-1}+By^{k-1}+b)\n-Q_i\\Delta x_i^{k}\n\\in\\partial f_i(x_i^{k}).\n\\end{eqnarray*}\n%\nTaking partial differential of $\\mathcal L^{\\alpha}$ with respect to $x_i$\nand evaluating the result at the point $({\\bf x}^{k},y^{k},z^{k})$ yields\n\\[\n\\partial_{x_i} \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k})=\n\\partial f_i(x_i^{k})+A_i\\tr z^{k}+\n\\alpha A_i\\tr \\big({\\bf Ax}^{k}+By^{k}+ b\\big).\n\\]\n%\nTherefore \n\\begin{eqnarray}\\label{dxinL}\n\\begin{array}{lll}\nd_{x_i}^{k}:=\nA_i\\tr \\Delta z^{k}+\n\\alpha A_i\\tr A_{>i}\\Delta x_{>i}^{k}\n+\\alpha A_i\\tr B\\Delta y^{k}\n-Q_i\\Delta x_i^{k}\n\\in \\partial_{x_i} \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k}).\n\\end{array}\n \\end{eqnarray}\n%\nThe optimality criterion of $y$ subproblem gives\n\\[\n\\nabla h(y^{k})=-B\\tr z^{k-1}- \\alpha B\\tr ({\\bf A x}^{k}+By^{k}+b) \n- {P} \\Delta y^{k}.\n\\]\nBy taking partial differential of $\\mathcal L^{\\alpha}$ with respect to $y$\nand evaluating it at $({\\bf x}^{k},y^{k},z^{k}) $ we obtain\n\\[\n\\nabla_y \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k}) =\n\\nabla h(y^{k})+B\\tr z^{k}+\\alpha B\\tr \\big({\\bf Ax}^{k}+By^{k}+b\\big).\n\\]\nThe last two equations follow that\n\\begin{eqnarray}\\label{dyinL}\nd_y^{k}:=B\\tr \\Delta z^{k}- P \\Delta y^{k}\n= \\nabla_y \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k}).\n\\end{eqnarray}\n%\nLastly, by the $z$ subproblem we have ${\\bf Ax}^{k}+By^{k}+b=\\frac{1}{\\alpha\\beta}\n \\Delta z^k$ hence \n\\begin{eqnarray}\\label{dzinL}\nd_z^{k}:=\\frac{1}{\\alpha\\beta}\n \\Delta z^{k}= \\nabla_z \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k}).\n\\end{eqnarray}\nHence, by (\\ref{dxinL}), (\\ref{dyinL}), , and (\\ref{dzinL}), we \n have \n$\nd^{k}:=\\big(\\{d^{k}_{x_i}\\}_{i=1}^p,d^{k}_y,d^{k}_z\\big)\\in\\partial \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k}).\n$\n%\nBy the norm property we have $\\|d_z^{k}\\|=\\frac{1}{\\alpha\\beta} \\|\\Delta z^{k}\\|$ \nand by the  triangle inequality from (\\ref{dxinL}) and (\\ref{dyinL}) we obtain \n%\n\\begin{eqnarray*} \\label{ndx}\n&\\|d_{x_i}^{k}\\|\\le\n\\|A_i\\|\\|\\Delta z^{k}\\|+\\alpha \\|A_i\\|\\Big({\\sum_{j=i+1}^p}\\|A_j\\|\\|\\Delta x_j^{k}\\|\\Big)\n+\\alpha\\|A_i\\|\\|B\\|\\|\\Delta y^{k}\\|+\\|Q_i\\|\\|\\Delta x_i^{k}\\|, &\\\\\n%\n&\\|d_y^{k}\\|\\le\\|B\\| \\|\\Delta z^{k}\\| +\\|P\\| \\|\\Delta y^{k}\\|. &\n%\\|d_z^{k}\\|=\\frac{1}{\\alpha\\beta} \\|\\Delta z^{k}\\|.\n\\end{eqnarray*}\n%\nHence we get\n%\n\\begin{eqnarray*}\n&|||d^{k}||| \\le  \\sum_{i=1}^p\\|d_{x_i}^{k}\\|+ \\|d_y^{k}\\|+\\|d_z^{k}\\|\n\\le \\sum_{i=1}^p \\|Q_i\\|\\|\\Delta x_i^{k}\\|\n+ \\sum_{i=2}^p \\|A_i\\| \\|\\Delta x_i^{k}\\|\\sum_{j=1}^{i-1}\\|A_j\\| \n& \\\\&\n+\\Big(\\|P\\|+\\alpha\\|B\\|\\sum_{i=1}^p \\|A_i\\|  \\Big)      \\|\\Delta y^{k}\\|\n+\\Big( \\sum_{i=1}^p \\|A_i\\|  +\\|B\\|+\\frac{1}{\\alpha\\beta}\\Big)       \\|\\Delta z^{k}\\|\n&\\\\&\n\\le \n\\Big(\\|{\\bf Q}\\|+\\|{\\bf A}\\|^2\\Big) \\sum_{i=1}^p\\|\\Delta x_i^{k}\\|\n+\\Big(\\|P\\|+\\alpha \\|{\\bf A}\\| \\|B\\| \\Big)      \\|\\Delta y^{k}\\|\n+\\Big(\\|{\\bf A}\\|  +\\|B\\|+\\frac{1}{\\alpha\\beta}\\Big)  \\|\\Delta z^{k}\\|\n&\\\\&\n\\le \\rho \\Big( \\sum_{i=1}^p\\|\\Delta x_i^{k}\\|+   \\|\\Delta y^{k}\\|+  \\|\\Delta z^{k}\\|\\Big).\n\\quad\\square &\n\\end{eqnarray*}\n%\n\\end{proof}","sha256_utf8":"8de583d2e75ad98a019f9ed85e9c94f70d31f11e504d314a89718d78453d3549"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"分块最优性构造原变量次梯度","source_units":[{"start":29008,"end":29894,"start_line":572,"end_line":602,"raw_tex":"\\begin{proof}\nLet $k\\ge 1$ be fixed. By the optimality condition of the $x_i$ subproblem\nof PADMM algorithm for \n$i=1,\\dots, p$ we have\n%\n\\begin{eqnarray*}\n-A_i\\tr z^{k-1}\n-\\alpha A_i\\tr (A_{<i}x_{<i}^{k}+A_ix_i^{k}+A_{>i}x_{>i}^{k-1}+By^{k-1}+b)\n-Q_i\\Delta x_i^{k}\n\\in\\partial f_i(x_i^{k}).\n\\end{eqnarray*}\n%\nTaking partial differential of $\\mathcal L^{\\alpha}$ with respect to $x_i$\nand evaluating the result at the point $({\\bf x}^{k},y^{k},z^{k})$ yields\n\\[\n\\partial_{x_i} \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k})=\n\\partial f_i(x_i^{k})+A_i\\tr z^{k}+\n\\alpha A_i\\tr \\big({\\bf Ax}^{k}+By^{k}+ b\\big).\n\\]\n%\nTherefore \n\\begin{eqnarray}\\label{dxinL}\n\\begin{array}{lll}\nd_{x_i}^{k}:=\nA_i\\tr \\Delta z^{k}+\n\\alpha A_i\\tr A_{>i}\\Delta x_{>i}^{k}\n+\\alpha A_i\\tr B\\Delta y^{k}\n-Q_i\\Delta x_i^{k}\n\\in \\partial_{x_i} \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k}).\n\\end{array}\n \\end{eqnarray}\n","sha256_utf8":"b974f8beb21d2b6bf0ee079bb35ecb832a80e089ff365a8e391e24ca021f5b64"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"末块最优性构造光滑梯度","source_units":[{"start":29896,"end":30490,"start_line":604,"end_line":619,"raw_tex":"The optimality criterion of $y$ subproblem gives\n\\[\n\\nabla h(y^{k})=-B\\tr z^{k-1}- \\alpha B\\tr ({\\bf A x}^{k}+By^{k}+b) \n- {P} \\Delta y^{k}.\n\\]\nBy taking partial differential of $\\mathcal L^{\\alpha}$ with respect to $y$\nand evaluating it at $({\\bf x}^{k},y^{k},z^{k}) $ we obtain\n\\[\n\\nabla_y \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k}) =\n\\nabla h(y^{k})+B\\tr z^{k}+\\alpha B\\tr \\big({\\bf Ax}^{k}+By^{k}+b\\big).\n\\]\nThe last two equations follow that\n\\begin{eqnarray}\\label{dyinL}\nd_y^{k}:=B\\tr \\Delta z^{k}- P \\Delta y^{k}\n= \\nabla_y \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k}).\n\\end{eqnarray}\n","sha256_utf8":"e38e842362dd1a0044d5c821101f8357327ca5fc9c5d5d844eaeb3f6785e1933"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"对偶更新构造约束残差梯度","source_units":[{"start":30492,"end":30931,"start_line":621,"end_line":631,"raw_tex":"Lastly, by the $z$ subproblem we have ${\\bf Ax}^{k}+By^{k}+b=\\frac{1}{\\alpha\\beta}\n \\Delta z^k$ hence \n\\begin{eqnarray}\\label{dzinL}\nd_z^{k}:=\\frac{1}{\\alpha\\beta}\n \\Delta z^{k}= \\nabla_z \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k}).\n\\end{eqnarray}\nHence, by (\\ref{dxinL}), (\\ref{dyinL}), , and (\\ref{dzinL}), we \n have \n$\nd^{k}:=\\big(\\{d^{k}_{x_i}\\}_{i=1}^p,d^{k}_y,d^{k}_z\\big)\\in\\partial \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k}).\n$\n","sha256_utf8":"0fcbcb4df114e931a737245138e06bfff906a3336fba8893f9bd8126aef36c17"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"分别估计次梯度分量范数","source_units":[{"start":30933,"end":31441,"start_line":633,"end_line":643,"raw_tex":"By the norm property we have $\\|d_z^{k}\\|=\\frac{1}{\\alpha\\beta} \\|\\Delta z^{k}\\|$ \nand by the  triangle inequality from (\\ref{dxinL}) and (\\ref{dyinL}) we obtain \n%\n\\begin{eqnarray*} \\label{ndx}\n&\\|d_{x_i}^{k}\\|\\le\n\\|A_i\\|\\|\\Delta z^{k}\\|+\\alpha \\|A_i\\|\\Big({\\sum_{j=i+1}^p}\\|A_j\\|\\|\\Delta x_j^{k}\\|\\Big)\n+\\alpha\\|A_i\\|\\|B\\|\\|\\Delta y^{k}\\|+\\|Q_i\\|\\|\\Delta x_i^{k}\\|, &\\\\\n%\n&\\|d_y^{k}\\|\\le\\|B\\| \\|\\Delta z^{k}\\| +\\|P\\| \\|\\Delta y^{k}\\|. &\n%\\|d_z^{k}\\|=\\frac{1}{\\alpha\\beta} \\|\\Delta z^{k}\\|.\n\\end{eqnarray*}\n","sha256_utf8":"96e2e36e4426c007b218a61fc1a5add52c8d1c31e5ce954a1a2aa84fa7a5a703"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"汇总分量得到统一常数界","source_units":[{"start":31443,"end":32174,"start_line":645,"end_line":664,"raw_tex":"Hence we get\n%\n\\begin{eqnarray*}\n&|||d^{k}||| \\le  \\sum_{i=1}^p\\|d_{x_i}^{k}\\|+ \\|d_y^{k}\\|+\\|d_z^{k}\\|\n\\le \\sum_{i=1}^p \\|Q_i\\|\\|\\Delta x_i^{k}\\|\n+ \\sum_{i=2}^p \\|A_i\\| \\|\\Delta x_i^{k}\\|\\sum_{j=1}^{i-1}\\|A_j\\| \n& \\\\&\n+\\Big(\\|P\\|+\\alpha\\|B\\|\\sum_{i=1}^p \\|A_i\\|  \\Big)      \\|\\Delta y^{k}\\|\n+\\Big( \\sum_{i=1}^p \\|A_i\\|  +\\|B\\|+\\frac{1}{\\alpha\\beta}\\Big)       \\|\\Delta z^{k}\\|\n&\\\\&\n\\le \n\\Big(\\|{\\bf Q}\\|+\\|{\\bf A}\\|^2\\Big) \\sum_{i=1}^p\\|\\Delta x_i^{k}\\|\n+\\Big(\\|P\\|+\\alpha \\|{\\bf A}\\| \\|B\\| \\Big)      \\|\\Delta y^{k}\\|\n+\\Big(\\|{\\bf A}\\|  +\\|B\\|+\\frac{1}{\\alpha\\beta}\\Big)  \\|\\Delta z^{k}\\|\n&\\\\&\n\\le \\rho \\Big( \\sum_{i=1}^p\\|\\Delta x_i^{k}\\|+   \\|\\Delta y^{k}\\|+  \\|\\Delta z^{k}\\|\\Big).\n\\quad\\square &\n\\end{eqnarray*}\n%\n\\end{proof}","sha256_utf8":"eaa9c24c93dc9bfc33647cff2ca307d1bc876ac0ef7ddd1b68d190e0fd70251f"}]},{"id":"S041","semantic_type":"lemma","kind":"lemma","title":"子列极限上的函数值连续性","source_units":[{"start":32230,"end":32647,"start_line":667,"end_line":674,"raw_tex":"\\begin{lemma}\\label{thm-cluster}\n{(Limiting continuity)} \nSuppose that A1-A2  hold. \nIf $({\\bf x}^*, y^*, z^*)$ is the limit point of $\\{({\\bf x}^{k_j},y^{k_j},z^{k_j})\\}_{j\\ge 0}$, a subsequence of \n$\\{{\\bf x}^k, y^k, z^k\\}_{k\\ge 0}$ generated by the PADMM algorithm (\\ref{padmm}). Then \n$ \\mathcal L^{\\alpha} ({\\bf x}^*, y^*, z^*)=\\lim_{j\\to\\infty} \\mathcal L^{\\alpha} ({\\bf x}^{k_j},y^{k_j},z^{k_j}).\n$\n\\end{lemma}","sha256_utf8":"fe580e61d696d6f56439b2eb6a45dffa831d461134f291ef6327b7eae7f87d91"}]},{"id":"S042","semantic_type":"proof","kind":"proof","title":"极限连续性完整证明","source_units":[{"start":32648,"end":34746,"start_line":675,"end_line":727,"raw_tex":"\\begin{proof}\nLet $\\{({\\bf x}^{k_j},y^{k_j},z^{k_j})\\}_{j\\ge 0}$ be a subsequence of the sequence \n$\\{({\\bf x}^{k},y^{k},z^{k})\\}_{j\\ge 0}$\ngenerated by the PADMM algorithm such that\n$\\lim_{j\\to\\infty} ({\\bf x}^{k_j},y^{k_j},z^{k_j})=({\\bf x}^*, y^*, z^*).$\nBy A1, for $i=1,\\dots,p$ the functions $f_i$ are lower semicontinuous hence\n\n\\begin{eqnarray}\\label{eq:flsc0}\nf_i(x_i^*)\\le \\lim\\inf_{j\\to\\infty} f(x_i^{k_j}).\n\\end{eqnarray}\n%\nBy the $x_i$-subproblem of the PADMM algorithm for any $v\\in\\mathbb R^{n_i}$ we have \n\\[\n\\mathcal L^{\\alpha} (x_{<i}^{k_j+1},x_i^{k_j+1}, x^{k_j}_{>i}, y^{k_j},z^{k_j})+\n\\frac{1}{2}\\|\\Delta x_i^{k_j+1}\\|_{Q_i}^2\n\\le \\mathcal L^{\\alpha} (x_{<i}^{k_j+1},v, x^{k_j}_{>i}, y^{k_j},z^{k_j})\n+\\frac{1}{2}\\|v- x_i^{k_j}\\|_{Q_i}^2.\n\\]\nWe choose $v=x_i^*$ and exploit the cosine rule\n$\\|a_1+c\\|^2-\\|a_2+c\\|^2=\\|a_2-a_1\\|^2+2\\langle a_2+c, a_2-a_1\\rangle$\nwith \n$a_1=A_ix_i^*$, $a_2=A_ix_i^{k_j+1}$ \nand $c=A_{<i}x_{<i}^{k_j+1}+A_{>i}x^{k_j}_{>i}+By^{k_j}+b$ to obtain\n %\n\\begin{eqnarray*}\n&f_i(x_i^{k_j+1})- f_i(x_i^*) +\n\\langle A_i\\tr z^{k_j},x^{k_j+1}_i-x_i^*\\rangle+\\|\\Delta x_i^{k_j+1}\\|^2_{Q_i} \n\\le \n\\frac{\\alpha}{2}\\|A_i (x_i^{k_j+1}-x_i^*)\\|^2&\n\\\\\n&+\\alpha \\Big\\langle A_i (x_i^*-x_i^{k_j+1}), A_{<i}x^{k_j+1}_{<i}+A_ix_i^{k_j+1}+A_{>i}x_{>i}^{k_j}+By^{k_j}+b\\Big\\rangle.\n&\n\\end{eqnarray*}\n%\nSince $\\{({\\bf x}^{k_j},y^{k_j},z^{k_j})\\}_{j\\ge 0}$  is a convergent sequence, \n it is bounded and the difference of consequtive terms approaches  zero. \nThus the latter inequality reduces to \n$\\displaystyle{\\lim\\sup_{j\\to \\infty}} f_i(x_i^{k_j+1})-f_i(x_i^*)\\le 0$ and \nin view of (\\ref{eq:flsc0}) we then have\n$\n\\lim_{j\\to \\infty} f_i(x_i^{k_j})=f_i(x_i^*),\n$\nwhich follows that\n$\n\\lim_{j\\to \\infty} \\sum_{i=1}^p f_i(x_i^{k_j})=\\sum_{i=1}^p f_i(x_i^*).\n$\n%\nBy A2, $h(y)$ is smooth hence $\\lim_{j\\to\\infty} h(y^{k_j})= h(y^*)$.\nClearly, ${\\bf Ax}^{k_j}+By^{k_j}+b\\to {\\bf Ax}^{*}+By^{*}+b$\nas $j\\to\\infty$.\nThus $\\displaystyle{\\lim_{j\\to\\infty}} \\mathcal L^{\\alpha} ({\\bf x}^{k_j},y^{k_j},z^{k_j})\n= \\mathcal L^{\\alpha} ({\\bf x}^*, y^*, z^*).$ $\\quad\\square$\n\\end{proof}","sha256_utf8":"89d6dceb5102678cd45cc38524a91043f328f0b19808ec2ca296ee0e0344f895"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"选取收敛子列并用下半连续性","source_units":[{"start":32648,"end":33081,"start_line":675,"end_line":684,"raw_tex":"\\begin{proof}\nLet $\\{({\\bf x}^{k_j},y^{k_j},z^{k_j})\\}_{j\\ge 0}$ be a subsequence of the sequence \n$\\{({\\bf x}^{k},y^{k},z^{k})\\}_{j\\ge 0}$\ngenerated by the PADMM algorithm such that\n$\\lim_{j\\to\\infty} ({\\bf x}^{k_j},y^{k_j},z^{k_j})=({\\bf x}^*, y^*, z^*).$\nBy A1, for $i=1,\\dots,p$ the functions $f_i$ are lower semicontinuous hence\n\n\\begin{eqnarray}\\label{eq:flsc0}\nf_i(x_i^*)\\le \\lim\\inf_{j\\to\\infty} f(x_i^{k_j}).\n\\end{eqnarray}\n","sha256_utf8":"08ea579bb67c1a8ff51d442e4c44494f11c526a74a09bcb6ac871fe4f1842be4"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"子问题比较给出函数值上界","source_units":[{"start":33083,"end":33972,"start_line":686,"end_line":707,"raw_tex":"By the $x_i$-subproblem of the PADMM algorithm for any $v\\in\\mathbb R^{n_i}$ we have \n\\[\n\\mathcal L^{\\alpha} (x_{<i}^{k_j+1},x_i^{k_j+1}, x^{k_j}_{>i}, y^{k_j},z^{k_j})+\n\\frac{1}{2}\\|\\Delta x_i^{k_j+1}\\|_{Q_i}^2\n\\le \\mathcal L^{\\alpha} (x_{<i}^{k_j+1},v, x^{k_j}_{>i}, y^{k_j},z^{k_j})\n+\\frac{1}{2}\\|v- x_i^{k_j}\\|_{Q_i}^2.\n\\]\nWe choose $v=x_i^*$ and exploit the cosine rule\n$\\|a_1+c\\|^2-\\|a_2+c\\|^2=\\|a_2-a_1\\|^2+2\\langle a_2+c, a_2-a_1\\rangle$\nwith \n$a_1=A_ix_i^*$, $a_2=A_ix_i^{k_j+1}$ \nand $c=A_{<i}x_{<i}^{k_j+1}+A_{>i}x^{k_j}_{>i}+By^{k_j}+b$ to obtain\n %\n\\begin{eqnarray*}\n&f_i(x_i^{k_j+1})- f_i(x_i^*) +\n\\langle A_i\\tr z^{k_j},x^{k_j+1}_i-x_i^*\\rangle+\\|\\Delta x_i^{k_j+1}\\|^2_{Q_i} \n\\le \n\\frac{\\alpha}{2}\\|A_i (x_i^{k_j+1}-x_i^*)\\|^2&\n\\\\\n&+\\alpha \\Big\\langle A_i (x_i^*-x_i^{k_j+1}), A_{<i}x^{k_j+1}_{<i}+A_ix_i^{k_j+1}+A_{>i}x_{>i}^{k_j}+By^{k_j}+b\\Big\\rangle.\n&\n\\end{eqnarray*}\n","sha256_utf8":"1afd69b8a09705e2febb163cc2eb4136e59e04b417f2eedb48c27c2e11ec1273"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"合并上下极限及连续项","source_units":[{"start":33974,"end":34746,"start_line":709,"end_line":727,"raw_tex":"Since $\\{({\\bf x}^{k_j},y^{k_j},z^{k_j})\\}_{j\\ge 0}$  is a convergent sequence, \n it is bounded and the difference of consequtive terms approaches  zero. \nThus the latter inequality reduces to \n$\\displaystyle{\\lim\\sup_{j\\to \\infty}} f_i(x_i^{k_j+1})-f_i(x_i^*)\\le 0$ and \nin view of (\\ref{eq:flsc0}) we then have\n$\n\\lim_{j\\to \\infty} f_i(x_i^{k_j})=f_i(x_i^*),\n$\nwhich follows that\n$\n\\lim_{j\\to \\infty} \\sum_{i=1}^p f_i(x_i^{k_j})=\\sum_{i=1}^p f_i(x_i^*).\n$\n%\nBy A2, $h(y)$ is smooth hence $\\lim_{j\\to\\infty} h(y^{k_j})= h(y^*)$.\nClearly, ${\\bf Ax}^{k_j}+By^{k_j}+b\\to {\\bf Ax}^{*}+By^{*}+b$\nas $j\\to\\infty$.\nThus $\\displaystyle{\\lim_{j\\to\\infty}} \\mathcal L^{\\alpha} ({\\bf x}^{k_j},y^{k_j},z^{k_j})\n= \\mathcal L^{\\alpha} ({\\bf x}^*, y^*, z^*).$ $\\quad\\square$\n\\end{proof}","sha256_utf8":"cb4da7178e0e9cf2f8fc5dde8a91d64f6e923260e2772ebb9fa4d792af38156f"}]},{"id":"S046","semantic_type":"lemma","kind":"lemma","title":"聚点满足驻点条件","source_units":[{"start":34797,"end":35449,"start_line":731,"end_line":750,"raw_tex":"\\begin{lemma}\\label{thm-cluster0}\n{(Limit point is critical point)} The set of limit points of the sequence\n$\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ generated by the PADMM algorithm (\\ref{padmm}),\ndenoted \n$\\omega\\big(\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}\\big)$, satisfies\n\\[\n\\omega\\big( \\{ ({\\bf x}^k,y^k,z^k) \\}_{k\\ge 0} \\big) \\subseteq {\\rm crit} \\;\\mathcal L^{\\alpha}\n:=\n\\left\\{\n({\\bf x}^*,y^*,z^*):\n\\begin{array}{l}\n 0\\in\\partial f_i(x_i^*)+A_i\\tr z^*,\\; i=1,\\dots, p\\\\\n\\nabla h(y^*)+ B\\tr z^*=0,\\\\\n{\\bf Ax}^*+By^*+b=0\n\\end{array}\n\\right\\},\n\\]\nwhere ${\\rm crit}\\;\\mathcal L^{\\alpha}$ denotes the set of critical points of $\\mathcal L^{\\alpha}$.\n%\n\\end{lemma}","sha256_utf8":"f11b535d6788061e7fcd4bab0a52e081b95c60e2c9747300ed8263e80b61f6f0"}]},{"id":"S047","semantic_type":"proof","kind":"proof","title":"聚点临界性完整证明","source_units":[{"start":35451,"end":36410,"start_line":752,"end_line":765,"raw_tex":"\\begin{proof}\nLet $(\\hat {\\bf x}, \\hat y, \\hat z)\\in \\omega\\big( \\{ ({\\bf x}^k,y^k,z^k) \\}_{k\\ge 0} \\big)$,\nhence there exists a subsequence $\\{({\\bf x}^{k_j},y^{k_j},z^{k_j})\\}_{j\\ge 0}$\nof $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ such that \n$\\lim_{j\\to\\infty} ({\\bf x}^{k_j},y^{k_j},z^{k_j})=(\\hat {\\bf x}, \\hat y, \\hat z).$\nFor the sequence $\\{d^k\\}_{k\\ge 0}$, we have $d^{k_j}\\in\\partial \\mathcal L^{\\alpha}(x^{k_j}, y^{k_j}, z^{k_j})$\nfor $j\\ge 1$ and $d^{k_j}\\to 0$ as $j\\to \\infty$ while $({\\bf x}^{k_j}, y^{k_j}, z^{k_j})\\to (\\hat {\\bf x}, \\hat y,\\hat z)$\nand $\\displaystyle{\\lim_{j\\to\\infty}} \\mathcal L^{\\alpha} ({\\bf x}^{k_j},y^{k_j},z^{k_j})\n= \\mathcal L^{\\alpha} (\\hat {\\bf x}, \\hat y, \\hat z).$\n By the closeness criterion of the limiting sub-differential \nwe then have $0\\in \\partial \\mathcal L^{\\alpha}(\\hat{\\bf x}, \\hat{y}, \\hat {z})$,\nand equivalently, $(\\hat{\\bf x}, \\hat{y}, \\hat {z})\\in {\\rm crit} (\\mathcal L^{\\alpha})$.\n $\\square$\n\\end{proof}","sha256_utf8":"5668ad3aa04cca0c545a2ad2cc54328f4c3235f570fe89e7e81980b87d127005"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"以收敛子列表示任意聚点","source_units":[{"start":35451,"end":35772,"start_line":752,"end_line":756,"raw_tex":"\\begin{proof}\nLet $(\\hat {\\bf x}, \\hat y, \\hat z)\\in \\omega\\big( \\{ ({\\bf x}^k,y^k,z^k) \\}_{k\\ge 0} \\big)$,\nhence there exists a subsequence $\\{({\\bf x}^{k_j},y^{k_j},z^{k_j})\\}_{j\\ge 0}$\nof $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ such that \n$\\lim_{j\\to\\infty} ({\\bf x}^{k_j},y^{k_j},z^{k_j})=(\\hat {\\bf x}, \\hat y, \\hat z).$\n","sha256_utf8":"d50ef85e361b04a9fcb250a396d476765a362ddef5defaddce36c4f6d5ad5617"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"次梯度闭性推出极限临界性","source_units":[{"start":35772,"end":36410,"start_line":757,"end_line":765,"raw_tex":"For the sequence $\\{d^k\\}_{k\\ge 0}$, we have $d^{k_j}\\in\\partial \\mathcal L^{\\alpha}(x^{k_j}, y^{k_j}, z^{k_j})$\nfor $j\\ge 1$ and $d^{k_j}\\to 0$ as $j\\to \\infty$ while $({\\bf x}^{k_j}, y^{k_j}, z^{k_j})\\to (\\hat {\\bf x}, \\hat y,\\hat z)$\nand $\\displaystyle{\\lim_{j\\to\\infty}} \\mathcal L^{\\alpha} ({\\bf x}^{k_j},y^{k_j},z^{k_j})\n= \\mathcal L^{\\alpha} (\\hat {\\bf x}, \\hat y, \\hat z).$\n By the closeness criterion of the limiting sub-differential \nwe then have $0\\in \\partial \\mathcal L^{\\alpha}(\\hat{\\bf x}, \\hat{y}, \\hat {z})$,\nand equivalently, $(\\hat{\\bf x}, \\hat{y}, \\hat {z})\\in {\\rm crit} (\\mathcal L^{\\alpha})$.\n $\\square$\n\\end{proof}","sha256_utf8":"675b18ed74c1e2a4e1d783251ac8f0eded0a1d3bd205e020a8da0a959bb5334c"}]},{"id":"S050","semantic_type":"lemma","kind":"lemma","title":"对偶递推及对偶范数估计","source_units":[{"start":36468,"end":37407,"start_line":768,"end_line":798,"raw_tex":"\\begin{lemma}{(bound dual)}\\label{bounddual}\nSuppose that A2 and A3 hold.\n%\\[\n%\\rho(\\beta):=1-|1-\\beta|,\n%\\quad \n%c_1:=  \\frac{1}{\\alpha\\rho(\\beta)\\lambda_{++}^{B\\tr B}},\n%\\quad\n%c_2:=  \\frac{2\\beta}{\\alpha\\rho(\\beta)^2\\lambda_{++}^{B\\tr B}}.\n%\\]\n%where  $\\lambda_{++}^{B\\tr B}$ be the smallest \n%strictly-positive eigenvalue of $B\\tr B$.\n%\nThen for all $k\\in\\mathbb Z_+$ the following assertions are true.\n\\begin{itemize}\n%\n\\item [(i)] $B\\tr z^{k}=\\beta w^{k}+(1-\\beta) B\\tr z^{k-1},$ where \n$w^{k}=-P\\Delta y^{k}-\\nabla h(y^{k}),$\n%\n\\item [(ii)]\n$\\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2\n\\le c_4 \\|\\Delta y^{k+1}\\|^2\n+c_3 \\|\\Delta y^{k}\\|^2+c_5 \\|B\\tr \\Delta z^{k}\\|^2\n-c_5\\| B\\tr \\Delta z^{k+1}\\|^2$,\n%\n\\item [(iii)]\n$\\frac{1}{2\\alpha}\\|z^{k+1}\\|^2\\le c_1  \\|P\\|^2 \\|\\Delta y^{k+1}\\|^2\n+ c_1 \\|\\nabla h(y^{k+1})\\|^2\n+c_6\\|B\\tr \\Delta z^{k+1}\\|^2$,\n\\end{itemize}\nwhere $c_i$, $i=2,\\dots,6$ are defined in (\\ref{quantities}).\n\\end{lemma}","sha256_utf8":"22038a1c0b81d7f63c120ee31d02452d4bd06c2c74376ffe9c31506cbf1b3d8a"}]},{"id":"S051","semantic_type":"proof","kind":"proof","title":"对偶估计完整证明","source_units":[{"start":37409,"end":41498,"start_line":800,"end_line":915,"raw_tex":"\\begin{proof}\n{\\bf (i)} \nBy the the optimality condition of $y$-subproblem we have\n\\[\n\\alpha B\\tr ({\\bf Ax}^{k}+By^{k}+b)+B\\tr z^{k-1}-w^k=0\n\\]\nwhich together with the $z$-iterate ${\\bf Ax}^{k}+By^{k}+b=\\frac{1}{\\alpha\\beta}(z^k-z^{k-1})$ we obtain the result.\n\n\n{\\bf (ii)} By A3 and the $z$ iterate of the PADMM algorithm we have $\\Delta z^{k+1}\\in {\\rm Im}(B)$, thus \n $\\lambda_{++}^{B\\tr B}\\|\\Delta z^{k+1}\\|^2\\le \\|B\\tr \\Delta z^{k+1}\\|^2$.\nPart (i) follows \n$B\\tr \\Delta z^{k+1}=\\beta\\Delta w^{k+1}+(1-\\beta) B\\tr \\Delta z^{k}$ and \nequivalently \n%\n\\begin{eqnarray*}\nB\\tr \\Delta z^{k+1}=\\rho(\\beta)\n\\Big(\\frac{\\beta\\Delta w^{k+1}}{\\rho(\\beta)} \\Big)\n+|1-\\beta| \\Big({\\rm sign}(1-\\beta) B\\tr \\Delta z^k\\Big),\n\\end{eqnarray*}\nwhere ${\\rm sign} (\\lambda)=1$ if $\\lambda\\ge 0$ and ${\\rm sign} (\\lambda)=-1$ if $\\lambda<0$.\nBy $\\rho(\\beta)+|1-\\beta|=1$ and the convexity of $\\|\\cdot\\|^2$ \nwe have\n%\n\\begin{eqnarray}\\label{gg}\n\\|B\\tr \\Delta z^{k+1}\\|^2\n\\le  \\frac{\\beta^2}{\\rho(\\beta)}  \\|\\Delta w^{k+1}\\|^2\n+|1-\\beta| \\|B\\tr \\Delta z^{k}\\|^2.%-|1-\\beta| \\Big\\| B^*\\Delta z^{k+1}\\Big\\|^2.\n\\end{eqnarray}\nThis also follows that \n%\n\\begin{eqnarray*}\\label{gg}\n\\rho(\\beta)\\lambda_{++}^{B\\tr B} \\|\\Delta z^{k+1}\\|^2&\\le& \\rho(\\beta)\\|B\\tr \\Delta z^{k+1}\\|^2\\\\\n%\n&\\le&  \\frac{\\beta^2}{\\rho(\\beta)}  \\|\\Delta w^{k+1}\\|^2\n+|1-\\beta| \\|B\\tr \\Delta z^{k}\\|^2-|1-\\beta| \\| B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray*}\n%\nRearrange this to obtain\n%\n\\begin{eqnarray}\\label{gg}\n\\|\\Delta z^{k+1}\\|^2\\le\n \\frac{\\beta^2}{\\rho(\\beta)^2\\lambda_{++}^{B\\tr B} }  \\|\\Delta w^{k+1}\\|^2\n+\\frac{|1-\\beta|}{\\rho(\\beta)\\lambda_{++}^{B\\tr B} } \\|B\\tr \\Delta z^{k}\\|^2\n-\\frac{|1-\\beta|}{\\rho(\\beta)\\lambda_{++}^{B\\tr B} } \\| B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray}\n%\nBy the optimality condition of $y$ subproblem we have \n\\[\n\\Delta w^{k+1}=-P\\Delta y^{k+1}+P\\Delta y^k+\\nabla h(y^{k})-\\nabla h(y^{k+1}).\n\\]\nBy $\\|a+b\\|^2\\le 2\\|a\\|^2+2\\|b\\|^2$ and the fact that $\\nabla h$ has $L_h$ Lipschitz gradients\nwe obtain\n\\[\n\\|\\Delta w^{k+1}\\|^2\\le 2(\\|P\\|+L_h)^2\\|\\Delta y^{k+1}\\|^2+2\\|P\\|^2 \\|\\Delta y^k\\|^2.\n\\]\nExploit this in (\\ref{gg}) to get\n\\begin{eqnarray*}\\label{gg0}\n\\|\\Delta z^{k+1}\\|^2&\\le&\n \\frac{2\\beta^2 (\\|P\\|+L_h)^2}{\\rho(\\beta)^2\\lambda_{++}^{B\\tr B} }          \\|\\Delta y^{k+1}\\|^2\n + \\frac{2\\beta^2\\|P\\|^2}{\\rho(\\beta)^2\\lambda_{++}^{B\\tr B} }    \\|\\Delta y^k\\|^2\\\\\n &&+\\frac{|1-\\beta|}{\\rho(\\beta)\\lambda_{++}^{B\\tr B} } \\|B\\tr \\Delta z^{k}\\|^2\n-\\frac{|1-\\beta|}{\\rho(\\beta)\\lambda_{++}^{B\\tr B} } \\| B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray*}\nWe divide both sides by $\\alpha\\beta$ to get \n%\n\\begin{eqnarray*}\n\\frac{1}{\\alpha\\beta}\\Big\\|\\Delta z^{k+1}\\Big\\|^2\n\\le c_4 \\Big\\|\\Delta y^{k+1}\\Big\\|^2\n+c_3 \\|\\Delta y^{k}\\|^2+c_5 \\Big\\|B\\tr \\Delta z^{k}\\Big\\|^2\n-c_5 \\Big\\| B\\tr \\Delta z^{k+1}\\Big\\|^2.\n\\end{eqnarray*}\n\n{\\bf (iii).} By part (i) we have\n%\n$\\beta B\\tr z^{k+1}=\\beta w^{k+1}+(1-\\beta) B\\tr (z^k-z^{k+1})$ \nwhich can be written equivalently as follows\n%\n\\begin{eqnarray*}\n\\beta B\\tr z^{k+1}= \\rho(\\beta)\\cdot \\frac{\\beta w^{k+1}}{\\rho(\\beta)}\n+|1-\\beta|\\cdot {\\rm sign}(1-\\beta) B\\tr (z^k-z^{k+1}).\n\\end{eqnarray*}\nBy $\\rho(\\beta)+|1-\\beta|=1$ and the convexity of $\\|\\cdot\\|^2$\nwe get\n%\n\\begin{eqnarray*}\n\\beta^2\\lambda_{++}^{B\\tr B} \\|z^{k+1}\\|^2\n\\le   \\beta^2 \\|B\\tr z^{k+1}\\|^2\n\\le\n\\rho(\\beta)\\cdot \\Big\\| \\frac{\\beta w^{k+1}}{\\rho(\\beta)}\\Big\\|^2\n+|1-\\beta| \\|B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray*}\n%\nSimplify and rearrange to get\n%\n\\begin{eqnarray*}\n\\|z^{k+1}\\|^2\n\\le \n\\frac{1}{\\rho(\\beta)\\lambda_{++}^{B\\tr B}}      \\|w^{k+1}\\|^2\n+\\frac{|1-\\beta|}{\\beta^2 \\lambda_{++}^{B\\tr B}      } \\|B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray*}\n%\nExploit $\\|w^{k+1}\\|^2\\le 2 \\|P\\|^2 \\|\\Delta y^{k+1}\\|^2+2\\|\\nabla h(y^{k+1})\\|^2$\nto obtain \n%\n\\begin{eqnarray*}\n\\|z^{k+1}\\|^2\n\\le\n\\frac{2\\|P\\|^2}{\\rho(\\beta)\\lambda_{++}^{B\\tr B}}  \\|\\Delta y^{k+1}\\|^2\n+ \\frac{2}{\\rho(\\beta)\\lambda_{++}^{B\\tr B}} \\|\\nabla h(y^{k+1})\\|^2\n+\\frac{|1-\\beta|}{\\beta^2\\lambda_{++}^{B\\tr B}} \\|B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray*}\n%\nDivide both sides by $2\\alpha$ to obtain the result.\nThis concludes the proof.  $\\square$\n\\end{proof}","sha256_utf8":"143f56ab5161b89e9b90baf3c7d009a1d9048c25e90e0fc5943de10cee899318"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"末块最优性导出对偶递推","source_units":[{"start":37409,"end":37670,"start_line":800,"end_line":806,"raw_tex":"\\begin{proof}\n{\\bf (i)} \nBy the the optimality condition of $y$-subproblem we have\n\\[\n\\alpha B\\tr ({\\bf Ax}^{k}+By^{k}+b)+B\\tr z^{k-1}-w^k=0\n\\]\nwhich together with the $z$-iterate ${\\bf Ax}^{k}+By^{k}+b=\\frac{1}{\\alpha\\beta}(z^k-z^{k-1})$ we obtain the result.\n","sha256_utf8":"f3a9f9df4880794359ec6d6d2ead30fe7491859a8ae8253d656d0a5e1cdb9a9f"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"像空间谱界与凸组合估计","source_units":[{"start":37672,"end":38512,"start_line":809,"end_line":828,"raw_tex":"{\\bf (ii)} By A3 and the $z$ iterate of the PADMM algorithm we have $\\Delta z^{k+1}\\in {\\rm Im}(B)$, thus \n $\\lambda_{++}^{B\\tr B}\\|\\Delta z^{k+1}\\|^2\\le \\|B\\tr \\Delta z^{k+1}\\|^2$.\nPart (i) follows \n$B\\tr \\Delta z^{k+1}=\\beta\\Delta w^{k+1}+(1-\\beta) B\\tr \\Delta z^{k}$ and \nequivalently \n%\n\\begin{eqnarray*}\nB\\tr \\Delta z^{k+1}=\\rho(\\beta)\n\\Big(\\frac{\\beta\\Delta w^{k+1}}{\\rho(\\beta)} \\Big)\n+|1-\\beta| \\Big({\\rm sign}(1-\\beta) B\\tr \\Delta z^k\\Big),\n\\end{eqnarray*}\nwhere ${\\rm sign} (\\lambda)=1$ if $\\lambda\\ge 0$ and ${\\rm sign} (\\lambda)=-1$ if $\\lambda<0$.\nBy $\\rho(\\beta)+|1-\\beta|=1$ and the convexity of $\\|\\cdot\\|^2$ \nwe have\n%\n\\begin{eqnarray}\\label{gg}\n\\|B\\tr \\Delta z^{k+1}\\|^2\n\\le  \\frac{\\beta^2}{\\rho(\\beta)}  \\|\\Delta w^{k+1}\\|^2\n+|1-\\beta| \\|B\\tr \\Delta z^{k}\\|^2.%-|1-\\beta| \\Big\\| B^*\\Delta z^{k+1}\\Big\\|^2.\n\\end{eqnarray}\n","sha256_utf8":"aee619f75bbac186e1c515fef0cd7bc68db3826185ad5eaed6273ef8ad2bfda7"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"整理对偶增量平方递推","source_units":[{"start":38512,"end":39140,"start_line":829,"end_line":845,"raw_tex":"This also follows that \n%\n\\begin{eqnarray*}\\label{gg}\n\\rho(\\beta)\\lambda_{++}^{B\\tr B} \\|\\Delta z^{k+1}\\|^2&\\le& \\rho(\\beta)\\|B\\tr \\Delta z^{k+1}\\|^2\\\\\n%\n&\\le&  \\frac{\\beta^2}{\\rho(\\beta)}  \\|\\Delta w^{k+1}\\|^2\n+|1-\\beta| \\|B\\tr \\Delta z^{k}\\|^2-|1-\\beta| \\| B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray*}\n%\nRearrange this to obtain\n%\n\\begin{eqnarray}\\label{gg}\n\\|\\Delta z^{k+1}\\|^2\\le\n \\frac{\\beta^2}{\\rho(\\beta)^2\\lambda_{++}^{B\\tr B} }  \\|\\Delta w^{k+1}\\|^2\n+\\frac{|1-\\beta|}{\\rho(\\beta)\\lambda_{++}^{B\\tr B} } \\|B\\tr \\Delta z^{k}\\|^2\n-\\frac{|1-\\beta|}{\\rho(\\beta)\\lambda_{++}^{B\\tr B} } \\| B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray}\n","sha256_utf8":"982acb4909b4bb910722b1a850459eeb32169d61c7819b3a095f0a75e21315dc"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"梯度光滑性控制辅助增量","source_units":[{"start":39142,"end":40198,"start_line":847,"end_line":871,"raw_tex":"By the optimality condition of $y$ subproblem we have \n\\[\n\\Delta w^{k+1}=-P\\Delta y^{k+1}+P\\Delta y^k+\\nabla h(y^{k})-\\nabla h(y^{k+1}).\n\\]\nBy $\\|a+b\\|^2\\le 2\\|a\\|^2+2\\|b\\|^2$ and the fact that $\\nabla h$ has $L_h$ Lipschitz gradients\nwe obtain\n\\[\n\\|\\Delta w^{k+1}\\|^2\\le 2(\\|P\\|+L_h)^2\\|\\Delta y^{k+1}\\|^2+2\\|P\\|^2 \\|\\Delta y^k\\|^2.\n\\]\nExploit this in (\\ref{gg}) to get\n\\begin{eqnarray*}\\label{gg0}\n\\|\\Delta z^{k+1}\\|^2&\\le&\n \\frac{2\\beta^2 (\\|P\\|+L_h)^2}{\\rho(\\beta)^2\\lambda_{++}^{B\\tr B} }          \\|\\Delta y^{k+1}\\|^2\n + \\frac{2\\beta^2\\|P\\|^2}{\\rho(\\beta)^2\\lambda_{++}^{B\\tr B} }    \\|\\Delta y^k\\|^2\\\\\n &&+\\frac{|1-\\beta|}{\\rho(\\beta)\\lambda_{++}^{B\\tr B} } \\|B\\tr \\Delta z^{k}\\|^2\n-\\frac{|1-\\beta|}{\\rho(\\beta)\\lambda_{++}^{B\\tr B} } \\| B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray*}\nWe divide both sides by $\\alpha\\beta$ to get \n%\n\\begin{eqnarray*}\n\\frac{1}{\\alpha\\beta}\\Big\\|\\Delta z^{k+1}\\Big\\|^2\n\\le c_4 \\Big\\|\\Delta y^{k+1}\\Big\\|^2\n+c_3 \\|\\Delta y^{k}\\|^2+c_5 \\Big\\|B\\tr \\Delta z^{k}\\Big\\|^2\n-c_5 \\Big\\| B\\tr \\Delta z^{k+1}\\Big\\|^2.\n\\end{eqnarray*}\n","sha256_utf8":"cedcfeb20435f5d0af76da1d26a4be12696dea36006e0d99161c0bd40cd30fad"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"重写对偶递推估计对偶范数","source_units":[{"start":40199,"end":41029,"start_line":873,"end_line":900,"raw_tex":"{\\bf (iii).} By part (i) we have\n%\n$\\beta B\\tr z^{k+1}=\\beta w^{k+1}+(1-\\beta) B\\tr (z^k-z^{k+1})$ \nwhich can be written equivalently as follows\n%\n\\begin{eqnarray*}\n\\beta B\\tr z^{k+1}= \\rho(\\beta)\\cdot \\frac{\\beta w^{k+1}}{\\rho(\\beta)}\n+|1-\\beta|\\cdot {\\rm sign}(1-\\beta) B\\tr (z^k-z^{k+1}).\n\\end{eqnarray*}\nBy $\\rho(\\beta)+|1-\\beta|=1$ and the convexity of $\\|\\cdot\\|^2$\nwe get\n%\n\\begin{eqnarray*}\n\\beta^2\\lambda_{++}^{B\\tr B} \\|z^{k+1}\\|^2\n\\le   \\beta^2 \\|B\\tr z^{k+1}\\|^2\n\\le\n\\rho(\\beta)\\cdot \\Big\\| \\frac{\\beta w^{k+1}}{\\rho(\\beta)}\\Big\\|^2\n+|1-\\beta| \\|B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray*}\n%\nSimplify and rearrange to get\n%\n\\begin{eqnarray*}\n\\|z^{k+1}\\|^2\n\\le \n\\frac{1}{\\rho(\\beta)\\lambda_{++}^{B\\tr B}}      \\|w^{k+1}\\|^2\n+\\frac{|1-\\beta|}{\\beta^2 \\lambda_{++}^{B\\tr B}      } \\|B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray*}\n","sha256_utf8":"4aceb1b3337ddff216cd69d2e1af00ecd19b29d2818a98a1f02ffdc864e59cc3"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"代入辅助向量界完成范数估计","source_units":[{"start":41031,"end":41498,"start_line":902,"end_line":915,"raw_tex":"Exploit $\\|w^{k+1}\\|^2\\le 2 \\|P\\|^2 \\|\\Delta y^{k+1}\\|^2+2\\|\\nabla h(y^{k+1})\\|^2$\nto obtain \n%\n\\begin{eqnarray*}\n\\|z^{k+1}\\|^2\n\\le\n\\frac{2\\|P\\|^2}{\\rho(\\beta)\\lambda_{++}^{B\\tr B}}  \\|\\Delta y^{k+1}\\|^2\n+ \\frac{2}{\\rho(\\beta)\\lambda_{++}^{B\\tr B}} \\|\\nabla h(y^{k+1})\\|^2\n+\\frac{|1-\\beta|}{\\beta^2\\lambda_{++}^{B\\tr B}} \\|B\\tr \\Delta z^{k+1}\\|^2.\n\\end{eqnarray*}\n%\nDivide both sides by $2\\alpha$ to obtain the result.\nThis concludes the proof.  $\\square$\n\\end{proof}","sha256_utf8":"7be7ecd10998e4e34cdbfb6d9a3641d8fdc3bb23ae484e4176e10a20db0b5a4b"}]},{"id":"S058","semantic_type":"lemma","kind":"lemma","title":"各变量更新的函数值变化","source_units":[{"start":41653,"end":42517,"start_line":920,"end_line":936,"raw_tex":"\\begin{lemma}\\label{chgL}\n{(change of $\\mathcal L^{\\alpha}$ during each variable update)} \nSuppose that A1-A4 hold. The iterates in PADMM algorithm (\\ref{padmm}) satisfy\n%\n\\begin{itemize}\n\\item [(i)] $\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^k, x^{k}_{>i},y^k,z^k)-\n\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^{k+1}, x^{k}_{>i},y^k,z^k)\\ge \\frac{1}{2} \\|\\Delta x_i^{k+1}\\|_{T_i}^2$ where\n$T_i=Q_i$ or $T_i=\\alpha A_i\\tr A_i-\\epsilon_i I$ where $\\epsilon_i >0$ defined in A4 for $i=1,\\dots, p$.\n%\n\\item [(ii)]\n$\\mathcal L^{\\alpha}({\\bf x}^{k+1},y^k, z^k)-\\mathcal L^{\\alpha}({\\bf x}^{k+1},y^{k+1},z^{k})\n\\ge \\frac{1}{2} \\|\\Delta y^{k+1}\\|_D^2$, where $D$ is defined in (\\ref{quantities}).\n\\item [(iii)] $\\mathcal L^{\\alpha}({\\bf x}^{k+1},y^{k+1}, z^k)-\n\\mathcal L^{\\alpha}({\\bf x}^{k+1},y^{k+1},z^{k+1})=-\\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2.$\n%\n\\end{itemize}\n\\end{lemma}","sha256_utf8":"442a549970f364763fafac394fc26be1db1c8ff14c644c1fe418cc2956e17a16"}]},{"id":"S059","semantic_type":"proof","kind":"proof","title":"逐变量变化界完整证明","source_units":[{"start":42519,"end":45245,"start_line":938,"end_line":1016,"raw_tex":"\\begin{proof}\n{\\bf (i)} Let $k\\in\\mathbb Z_+$ and $i\\in\\{1,\\dots,p\\}$ be fixed.\nBy the $x_i$ subproblem we have\n\\[\n\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^{k}, x^{k}_{>i},y^k,z^k)-\n\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^{k+1}, x^{k}_{>i},y^k,z^k)\n\\ge \\|\\Delta x_i^{k+1}\\|_{Q_i}^2.\n\\]\n%\nOn the other hand, by the optimality condition of $x_i$-sub-minimization problem we have \n$ \\bar{d_i}\\in\\partial f_i(x_i^{k+1})$\nwhere \n\\[\n\\bar{d_i}:=-A_i \\tr z^k-\n\\alpha A_i\\tr \\big(A_{<i} x_{<i}^{k+1}+A_ix_i^{k+1}+A_{>i}x_{>i}^k+By^k+b\n\\big).\n\\]\nBy this we then have\n\\begin{eqnarray*}\n&\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^k, x^{k}_{>i},y^k,z^k)-\n\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^{k+1}, x^{k}_{>i},y^k,z^k)& \\\\[.1in]\n&=f_i(x_i^k)-f_i(x_i^{k+1})\n+\\frac{\\alpha}{2}\\|A_i\\Delta x_i^{k+1}\\|^2-\\langle \\bar{d_i}, \\Delta x_i^{k+1}\\rangle&\n\\\\[.1in]\n&\\ge \\frac{\\alpha}{2}\\|A_i\\Delta x_i^{k+1}\\|^2\n- \\frac{\\epsilon_i}{2}\\|\\Delta x_i^{k+1}\\|^2\n=\\frac 12 \\|\\Delta x_i^{k+1}\\|_{\\alpha A_i\\tr A_i-\\epsilon_iI}^2.&\n\\end{eqnarray*}\n\n%%%%\n{\\bf (ii)}\nLet $k\\ge 0$ be fixed. From the optimality condition of $y$ minimization problem \nwe have\n\\begin{eqnarray*}\n\\nabla h(y^{k+1})+\\alpha B\\tr ({\\bf Ax}^{k+1}+By^{k+1}+b)\n+B\\tr z^k+P \\Delta y^{k+1}=0.\n\\end{eqnarray*}\nMultiply this equation by $\\Delta y^{k+1}$ and rearrange to obtain\n\\begin{eqnarray}\\label{eq:2}\n- \\alpha\\big\\langle B\\tr ({\\bf Ax}^{k+1}+ By^{k+1}+b),\\Delta y^{k+1}\\big\\rangle \n-\\langle B\\tr z^k, \\Delta y^{k+1}\\rangle\n=\\langle \\nabla h(y^{k+1}),\\Delta y^{k+1}\\rangle +\\|\\Delta y^{k+1}\\|_P^2.\n\\end{eqnarray}\nBy this and the fact that $h(y)$ is $L_h$ Lipschitz continuous we then get \n%\n\\begin{eqnarray*}\n&\\mathcal L^{\\alpha}(x^{k+1},y^k,z^k)-\\mathcal L^{\\alpha}(x^{k+1},y^{k+1},z^{k})&\n\\\\[.1in]\n&= h(y^k)-h(y^{k+1})\n+\\frac{\\alpha}{2}\\|{\\bf Ax}^{k+1}+By^k+b\\|^2\n-\\frac{\\alpha}{2}\\|{\\bf Ax}^{k+1}+By^{k+1}+b\\|^2\n-\\langle B\\tr z^k, \\Delta y^{k+1} \\rangle&\n\\\\[.1in]\n &=h(y^k)-h(y^{k+1})\n+\\frac{\\alpha}{2}\\|B\\Delta y^{k+1}\\|^2\n-\\alpha \\langle \\Delta y^{k+1}, B\\tr ({\\bf Ax}^{k+1}+By^{k+1}+b)\\rangle \n-\\langle B\\tr z^k, \\Delta y^{k+1} \\rangle&\n\\\\[.1in]\n&=h(y^k)-h(y^{k+1})\n+\\langle \\nabla h(y^{k+1}),\\Delta y^{k+1}\\rangle +\\|\\Delta y^{k+1}\\|_P^2\n+\\frac{\\alpha}{2}\\|B\\Delta y^{k+1}\\|^2\n&\\\\[.1in]\n&\\ge\\|\\Delta y^{k+1}\\|_P^2\n+\\frac{\\alpha}{2}\\|B\\Delta y^{k+1}\\|^2\n-\\frac{L_h}{2}\\|\\Delta y^{k+1}\\|^2 = \n\\frac{1}{2} \\|\\Delta y^{k+1}\\|_{D}^2.&\n\\end{eqnarray*}\n%\n%\n{\\bf (iii)} By the $z$ iterate of the algorithm we obtain\n\\begin{eqnarray*}\n&\\mathcal L^{\\alpha}({\\bf x}^{k+1},y^{k+1}, z^k)-\n\\mathcal L^{\\alpha}({\\bf x}^{k+1},y^{k+1},z^{k+1})=\n\\langle z^k-z^{k+1}, {\\bf Ax}^{k+1}+By^{k+1}+b\\rangle\n=-\\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2.&\n\\end{eqnarray*}\n%\nThat completes the proof. $\\square$\n\\end{proof}","sha256_utf8":"d5f49798c3bafb67d1b4ce2727309c7b7a2e205e570b410bf91f0021ec4362e6"}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","title":"近端子问题比较给出分块下降","source_units":[{"start":42519,"end":42798,"start_line":938,"end_line":945,"raw_tex":"\\begin{proof}\n{\\bf (i)} Let $k\\in\\mathbb Z_+$ and $i\\in\\{1,\\dots,p\\}$ be fixed.\nBy the $x_i$ subproblem we have\n\\[\n\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^{k}, x^{k}_{>i},y^k,z^k)-\n\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^{k+1}, x^{k}_{>i},y^k,z^k)\n\\ge \\|\\Delta x_i^{k+1}\\|_{Q_i}^2.\n\\]\n","sha256_utf8":"503dadf1d0d67cff06f5615664bf067a7aa5dcde7a2d01ca6b4c9b20b6f9f5ea"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"弱凸性给出另一分块下降界","source_units":[{"start":42800,"end":43522,"start_line":947,"end_line":965,"raw_tex":"On the other hand, by the optimality condition of $x_i$-sub-minimization problem we have \n$ \\bar{d_i}\\in\\partial f_i(x_i^{k+1})$\nwhere \n\\[\n\\bar{d_i}:=-A_i \\tr z^k-\n\\alpha A_i\\tr \\big(A_{<i} x_{<i}^{k+1}+A_ix_i^{k+1}+A_{>i}x_{>i}^k+By^k+b\n\\big).\n\\]\nBy this we then have\n\\begin{eqnarray*}\n&\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^k, x^{k}_{>i},y^k,z^k)-\n\\mathcal L^{\\alpha}(x_{<i}^{k+1},x_i^{k+1}, x^{k}_{>i},y^k,z^k)& \\\\[.1in]\n&=f_i(x_i^k)-f_i(x_i^{k+1})\n+\\frac{\\alpha}{2}\\|A_i\\Delta x_i^{k+1}\\|^2-\\langle \\bar{d_i}, \\Delta x_i^{k+1}\\rangle&\n\\\\[.1in]\n&\\ge \\frac{\\alpha}{2}\\|A_i\\Delta x_i^{k+1}\\|^2\n- \\frac{\\epsilon_i}{2}\\|\\Delta x_i^{k+1}\\|^2\n=\\frac 12 \\|\\Delta x_i^{k+1}\\|_{\\alpha A_i\\tr A_i-\\epsilon_iI}^2.&\n\\end{eqnarray*}\n","sha256_utf8":"338ed6aeac83a0c15ae89a349201c1ff3f1aa3c0932d1c3ead4221d916fa1e02"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"末块最优性与光滑性导出下降","source_units":[{"start":43528,"end":44898,"start_line":968,"end_line":1004,"raw_tex":"{\\bf (ii)}\nLet $k\\ge 0$ be fixed. From the optimality condition of $y$ minimization problem \nwe have\n\\begin{eqnarray*}\n\\nabla h(y^{k+1})+\\alpha B\\tr ({\\bf Ax}^{k+1}+By^{k+1}+b)\n+B\\tr z^k+P \\Delta y^{k+1}=0.\n\\end{eqnarray*}\nMultiply this equation by $\\Delta y^{k+1}$ and rearrange to obtain\n\\begin{eqnarray}\\label{eq:2}\n- \\alpha\\big\\langle B\\tr ({\\bf Ax}^{k+1}+ By^{k+1}+b),\\Delta y^{k+1}\\big\\rangle \n-\\langle B\\tr z^k, \\Delta y^{k+1}\\rangle\n=\\langle \\nabla h(y^{k+1}),\\Delta y^{k+1}\\rangle +\\|\\Delta y^{k+1}\\|_P^2.\n\\end{eqnarray}\nBy this and the fact that $h(y)$ is $L_h$ Lipschitz continuous we then get \n%\n\\begin{eqnarray*}\n&\\mathcal L^{\\alpha}(x^{k+1},y^k,z^k)-\\mathcal L^{\\alpha}(x^{k+1},y^{k+1},z^{k})&\n\\\\[.1in]\n&= h(y^k)-h(y^{k+1})\n+\\frac{\\alpha}{2}\\|{\\bf Ax}^{k+1}+By^k+b\\|^2\n-\\frac{\\alpha}{2}\\|{\\bf Ax}^{k+1}+By^{k+1}+b\\|^2\n-\\langle B\\tr z^k, \\Delta y^{k+1} \\rangle&\n\\\\[.1in]\n &=h(y^k)-h(y^{k+1})\n+\\frac{\\alpha}{2}\\|B\\Delta y^{k+1}\\|^2\n-\\alpha \\langle \\Delta y^{k+1}, B\\tr ({\\bf Ax}^{k+1}+By^{k+1}+b)\\rangle \n-\\langle B\\tr z^k, \\Delta y^{k+1} \\rangle&\n\\\\[.1in]\n&=h(y^k)-h(y^{k+1})\n+\\langle \\nabla h(y^{k+1}),\\Delta y^{k+1}\\rangle +\\|\\Delta y^{k+1}\\|_P^2\n+\\frac{\\alpha}{2}\\|B\\Delta y^{k+1}\\|^2\n&\\\\[.1in]\n&\\ge\\|\\Delta y^{k+1}\\|_P^2\n+\\frac{\\alpha}{2}\\|B\\Delta y^{k+1}\\|^2\n-\\frac{L_h}{2}\\|\\Delta y^{k+1}\\|^2 = \n\\frac{1}{2} \\|\\Delta y^{k+1}\\|_{D}^2.&\n\\end{eqnarray*}\n","sha256_utf8":"71e73ad866ae7ccad59fab0cde82e63d6135beba3a96b4c058d76f1d8373cf4b"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"计算对偶更新造成的函数变化","source_units":[{"start":44902,"end":45245,"start_line":1007,"end_line":1016,"raw_tex":"{\\bf (iii)} By the $z$ iterate of the algorithm we obtain\n\\begin{eqnarray*}\n&\\mathcal L^{\\alpha}({\\bf x}^{k+1},y^{k+1}, z^k)-\n\\mathcal L^{\\alpha}({\\bf x}^{k+1},y^{k+1},z^{k+1})=\n\\langle z^k-z^{k+1}, {\\bf Ax}^{k+1}+By^{k+1}+b\\rangle\n=-\\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2.&\n\\end{eqnarray*}\n%\nThat completes the proof. $\\square$\n\\end{proof}","sha256_utf8":"0ba373c706317734f4a28b891030caf109ce4c7d2a2dd9367aea1406d48e38b9"}]},{"id":"S064","semantic_type":"lemma","kind":"lemma","title":"完整迭代的修正下降界","source_units":[{"start":45292,"end":46070,"start_line":1020,"end_line":1039,"raw_tex":"\\begin{lemma}\\label{lem:chgLiter}\n{(change of $\\mathcal L^{\\alpha}$ after each iteration)} \nWe assume that A1-A4 hold.\nThe iterates in PADMM algorithm (\\ref{padmm}) satisfy\n%\n\\begin{eqnarray}\\label{chgLiter}\n\\begin{array}{c}\n\\mathcal L^{\\alpha}\\big({\\bf x}^{k+1},y^{k+1},z^{k+1}\\big)\n+\\frac{1}{2}\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|_{T_i}^2\n+ \\frac{1}{2} \\|\\Delta y^{k+1}\\|_{D-2\\epsilon_0c_4 I_m}^2\\\\[.1in]\n+\\displaystyle{\\frac{\\epsilon_0-1}{\\alpha\\beta}}\\big\\|\\Delta z^{k+1}\\big\\|^2\n+\\epsilon_0c_5 \\big\\|B\\tr \\Delta z^{k+1}\\big\\|^2\n\\le  \\mathcal L^{\\alpha}\\big(x^{k},y^k,z^k\\big)+\n\\epsilon_0c_5 \\big\\|B\\tr \\Delta z^k\\big\\|^2\n+\\epsilon_0c_3 \\big\\|\\Delta y^k\\big\\|^2,\n\\end{array}\n\\end{eqnarray}\nwhere $\\epsilon_0>1$, and $c_3,c_4,c_5$ are defined in (\\ref{quantities}).\n%\n\\end{lemma}","sha256_utf8":"3866b3a0bd5f9d648372782d02cb97b24e0e5e81a6412c96e07adfd89260d55c"}]},{"id":"S065","semantic_type":"proof","kind":"proof","title":"完整迭代下降界证明","source_units":[{"start":46071,"end":46974,"start_line":1040,"end_line":1062,"raw_tex":"\\begin{proof}\nBy Lemma \\ref{chgL} we have \n\\begin{eqnarray*}\n&\\mathcal L^{\\alpha}(x^{k+1},y^{k+1},z^{k+1})=\\mathcal L^{\\alpha}(x^{k+1},y^{k+1},z^{k}) \n+ \\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2\n&\\\\[.1in]&\n\\le  \\mathcal L^{\\alpha}(x^{k+1},y^{k},z^{k}) + \\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2\n-\\frac{1}{2} \\|\\Delta y^{k+1}\\|_{D}^2 \n&\\\\[.1in]&\n\\le \\mathcal L^{\\alpha}(x^{k},y^{k},z^{k})\n+ \\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2\n-\\frac{1}{2} \\|\\Delta y^{k+1}\\|_{D}^2 -\\frac{1}{2}\\sum_{i=1}^p \\|\\Delta x_i^{k+1}\\|^2_{T_i}.&\n\\end{eqnarray*}\nBy Lemma \\ref{bounddual} for $\\epsilon_0>1$ we have \n\\[\n\\frac{\\epsilon_0}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2\n\\le\\epsilon_0c_4 \\|\\Delta y^{k+1}\\|^2\n+\\epsilon_0c_3 \\|\\Delta y^{k}\\|^2+\n\\epsilon_0c_5 \\|B\\tr \\Delta z^{k}\\|^2\n-\\epsilon_0c_5\\| B\\tr \\Delta z^{k+1}\\|^2.\n\\]\nAdd this to the latter inequality and rearrange to obtain (\\ref{chgLiter}). $\\square$\n\\end{proof}","sha256_utf8":"5132e47a6c707194b86aa2d4e4e539b2a542a1199fde6d6bc56a4b2f7d9be152"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"累加逐变量变化不等式","source_units":[{"start":46071,"end":46614,"start_line":1040,"end_line":1052,"raw_tex":"\\begin{proof}\nBy Lemma \\ref{chgL} we have \n\\begin{eqnarray*}\n&\\mathcal L^{\\alpha}(x^{k+1},y^{k+1},z^{k+1})=\\mathcal L^{\\alpha}(x^{k+1},y^{k+1},z^{k}) \n+ \\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2\n&\\\\[.1in]&\n\\le  \\mathcal L^{\\alpha}(x^{k+1},y^{k},z^{k}) + \\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2\n-\\frac{1}{2} \\|\\Delta y^{k+1}\\|_{D}^2 \n&\\\\[.1in]&\n\\le \\mathcal L^{\\alpha}(x^{k},y^{k},z^{k})\n+ \\frac{1}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2\n-\\frac{1}{2} \\|\\Delta y^{k+1}\\|_{D}^2 -\\frac{1}{2}\\sum_{i=1}^p \\|\\Delta x_i^{k+1}\\|^2_{T_i}.&\n\\end{eqnarray*}\n","sha256_utf8":"f9b415d33c965e5fc3ca94d0b8e186e07a966cbbb967083654cdfcc2ebba8bbc"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"用对偶界吸收对偶增量","source_units":[{"start":46614,"end":46974,"start_line":1053,"end_line":1062,"raw_tex":"By Lemma \\ref{bounddual} for $\\epsilon_0>1$ we have \n\\[\n\\frac{\\epsilon_0}{\\alpha\\beta}\\|\\Delta z^{k+1}\\|^2\n\\le\\epsilon_0c_4 \\|\\Delta y^{k+1}\\|^2\n+\\epsilon_0c_3 \\|\\Delta y^{k}\\|^2+\n\\epsilon_0c_5 \\|B\\tr \\Delta z^{k}\\|^2\n-\\epsilon_0c_5\\| B\\tr \\Delta z^{k+1}\\|^2.\n\\]\nAdd this to the latter inequality and rearrange to obtain (\\ref{chgLiter}). $\\square$\n\\end{proof}","sha256_utf8":"a368411838c1d55eb6e1621b09abad1b2d2d22dc13ca0dd25eb37e7ca59dda4c"}]},{"id":"S068","semantic_type":"lemma","kind":"lemma","title":"修正势函数充分下降与收敛","source_units":[{"start":47931,"end":49154,"start_line":1088,"end_line":1118,"raw_tex":"\\begin{lemma}\\label{thm1}\n%{\\bf (monotonicity/convergence of $\\bar{\\mathcal L}$)}\nSuppose that A1-A4 hold, and \n\\[\n\\bar\\tau =\\min\\{ \\max \\{ q_i,\\lambda_{++}^{\\alpha A_i\\tr A_i-\\epsilon_i I} \\} : i=1,\\dots, p\\}.\n\\]\n If $\\sigma=\\min\\{\\lambda_{\\min}(\\bar D), \\bar\\tau,\\frac{\\epsilon_0-1}{\\alpha\\beta}\\}>0$ where \n $\\bar D$ is defined in (\\ref{quantities}), then\nthe following assertions hold.\n%\n\\begin{itemize}\n\\item [(i)] The sequence $\\{\\bar {\\mathcal L_k}\\}_{k\\ge 1}$ is monotonically decreasing and \n%\n\\begin{eqnarray}\\label{Lbarmon}\n\\bar{\\mathcal L}_{k+1}+\\sigma\n\\big[\\sum_{i=1}^p\\|\\Delta x_i^{k+1}\\|^2\n+ \\|\\Delta y^{k+1}\\|^2+\n\\|\\Delta z^{k+1}\\|^2\\big]\n\\le \\bar{\\mathcal L}_k.\n\\end{eqnarray}\n%\n%\n\\item [(ii)] If the sequence $\\{{\\bf x}^k, y^k, z^k\\}_{k\\ge 0}$ generated by the PADMM is bounded \nthen $\\bar{\\mathcal L}_k$ is bounded from below for all $k \\in \\mathbb N$  and converges as $k$ approaches infinity.\n%%\n%\\item [(iii)] If $B\\tr B$ is invertible, then $\\{{\\bf x}^k, y^k, z^k\\}_{k\\ge 0}$ is bounded.\n%\\item [(iv)]  If $\\{{\\bf x}^k, y^k, z^k\\}_{k\\ge 0}$ is bounded\n%then \n%$\\lim_{k\\to \\infty} \\|\\Delta x_i^{k}\\|=\\lim_{k\\to \\infty} \\|\\Delta y^{k}\\|=\\lim_{k\\to \\infty} \\|\\Delta z^{k}\\|=0.$\n\\end{itemize}\n\\end{lemma}","sha256_utf8":"13bea07d676c4a91ff7e9da5c79db97687fe6e5869c402f73c64fdd3609f5bf1"}]},{"id":"S069","semantic_type":"proof","kind":"proof","title":"势函数下降收敛完整证明","source_units":[{"start":49157,"end":50796,"start_line":1120,"end_line":1153,"raw_tex":"\\begin{proof}\n{\\bf (i)} By (\\ref{chgLiter}) and (\\ref{Lbark}) we obtain \n\\[\n\\bar{\\mathcal L}_{k+1}\n+\\frac{1}{2}\\sum_{i=1}^p \\|\\Delta x_i^{k+1}\\|^2_{T_i}\n+\\frac 12\\|\\Delta y^{k+1}\\|^2_{D-2\\epsilon_0(c_3+c_4)}+\n\\displaystyle{\\frac{\\epsilon_0-1}{\\alpha\\beta}}\\big\\|\\Delta z^{k+1}\\big\\|^2\n\\le \\bar{\\mathcal L}_k.\n\\]\nWe have $T_i=Q_i\\succeq q_i I$ where $q_i>0$ or $T_i=\\alpha A_i\\tr A_i-\\epsilon_i I$.\nWe let $\\bar\\tau =\\min\\{ \\max \\{ q_i,\\lambda_{++}^{\\alpha A_i\\tr A_i-\\epsilon_i I} \\} : i=1,\\dots, p\\}$\nand $\\bar D=D-2\\epsilon_0(c_3+c_4)I$. For \n$\\sigma=\\min\\{\\lambda_{\\min}(\\bar D), \\bar\\tau,\\frac{\\epsilon_0-1}{\\alpha\\beta}\\}>0$\nwe arrive at the inequality (\\ref{Lbarmon}), showing monotonicity of $\\{\\bar {\\mathcal L_k}\\}_{k\\ge 1}$.\n%\n\n{\\bf (ii)}  If $\\{{\\bf x}^k, y^k, z^k\\}_{k\\ge 0}$ is a bounded  sequence,\nthen \n$\\|B\\tr\\Delta z^k\\|^2+\n\\|\\Delta y^k\\|^2$ is also bounded, and bounded from below by zero.\nHence we only need to show that \n$\\mathcal L^{\\alpha}({\\bf x}^k,y^k,z^k)$ is bounded. \nBy the boundedness of $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$,\nthe terms  $\\langle z^k, {\\bf Ax}^k+By^k+b\\rangle$ and $\\|{\\bf Ax}^k+By^k+b\\|^2$\nare bounded. By A1 the function $f_i, i=1,\\dots,p$ are coercive,\nsince $\\{{\\bf x}^k\\}_{k\\ge 0}$ is bounded, then $\\{f_i(x_i^k)\\}_{k\\ge 0}$ is bounded for $i=1,\\dots,p$. \nBy A2, the function $h$ is bounded from below.\nThus, $\\{\\mathcal L^{\\alpha}({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$\nis bounded from below.\n%\nSince $\\{\\bar{\\mathcal L}_k\\}_{k\\ge 1}$ is bounded from below and by part (i) it is \nmonotonically decreasing then we conclude that $\\{\\bar{\\mathcal L}_k\\}_{k\\ge 1}$ it converges.  $\\square$\n\n\\end{proof}","sha256_utf8":"b5db1b6cc455d4e4355ddffece3c324ff5b452d8e706313947d7f905b96881d1"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"正谱参数给出充分下降","source_units":[{"start":49157,"end":49892,"start_line":1120,"end_line":1133,"raw_tex":"\\begin{proof}\n{\\bf (i)} By (\\ref{chgLiter}) and (\\ref{Lbark}) we obtain \n\\[\n\\bar{\\mathcal L}_{k+1}\n+\\frac{1}{2}\\sum_{i=1}^p \\|\\Delta x_i^{k+1}\\|^2_{T_i}\n+\\frac 12\\|\\Delta y^{k+1}\\|^2_{D-2\\epsilon_0(c_3+c_4)}+\n\\displaystyle{\\frac{\\epsilon_0-1}{\\alpha\\beta}}\\big\\|\\Delta z^{k+1}\\big\\|^2\n\\le \\bar{\\mathcal L}_k.\n\\]\nWe have $T_i=Q_i\\succeq q_i I$ where $q_i>0$ or $T_i=\\alpha A_i\\tr A_i-\\epsilon_i I$.\nWe let $\\bar\\tau =\\min\\{ \\max \\{ q_i,\\lambda_{++}^{\\alpha A_i\\tr A_i-\\epsilon_i I} \\} : i=1,\\dots, p\\}$\nand $\\bar D=D-2\\epsilon_0(c_3+c_4)I$. For \n$\\sigma=\\min\\{\\lambda_{\\min}(\\bar D), \\bar\\tau,\\frac{\\epsilon_0-1}{\\alpha\\beta}\\}>0$\nwe arrive at the inequality (\\ref{Lbarmon}), showing monotonicity of $\\{\\bar {\\mathcal L_k}\\}_{k\\ge 1}$.\n","sha256_utf8":"9a6dd5be1194f1e972e6cc7fa1d9d3413c7e37ba5058c27ebc096cefccad06b9"}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","title":"有界迭代下势函数有下界并收敛","source_units":[{"start":49895,"end":50796,"start_line":1136,"end_line":1153,"raw_tex":"{\\bf (ii)}  If $\\{{\\bf x}^k, y^k, z^k\\}_{k\\ge 0}$ is a bounded  sequence,\nthen \n$\\|B\\tr\\Delta z^k\\|^2+\n\\|\\Delta y^k\\|^2$ is also bounded, and bounded from below by zero.\nHence we only need to show that \n$\\mathcal L^{\\alpha}({\\bf x}^k,y^k,z^k)$ is bounded. \nBy the boundedness of $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$,\nthe terms  $\\langle z^k, {\\bf Ax}^k+By^k+b\\rangle$ and $\\|{\\bf Ax}^k+By^k+b\\|^2$\nare bounded. By A1 the function $f_i, i=1,\\dots,p$ are coercive,\nsince $\\{{\\bf x}^k\\}_{k\\ge 0}$ is bounded, then $\\{f_i(x_i^k)\\}_{k\\ge 0}$ is bounded for $i=1,\\dots,p$. \nBy A2, the function $h$ is bounded from below.\nThus, $\\{\\mathcal L^{\\alpha}({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$\nis bounded from below.\n%\nSince $\\{\\bar{\\mathcal L}_k\\}_{k\\ge 1}$ is bounded from below and by part (i) it is \nmonotonically decreasing then we conclude that $\\{\\bar{\\mathcal L}_k\\}_{k\\ge 1}$ it converges.  $\\square$\n\n\\end{proof}","sha256_utf8":"60ea754ff2a428727691b7ab121994fa6adaa9c73bd248275a3a677bbd026d0f"}]},{"id":"S072","semantic_type":"lemma","kind":"lemma","title":"聚点上的势函数与目标值相同","source_units":[{"start":50897,"end":51170,"start_line":1157,"end_line":1168,"raw_tex":"\\begin{lemma}\\label{samelimit}\nSuppose that A1-A4 hold and   $({\\bf x}^*,y^*,z^*)\\in\\omega(\\{ ({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0})$,\nthen\n\\[\n\\bar{\\mathcal L}({\\bf x}^*,y^*,z^*, y^*,z^*)\n=\n\\mathcal L^{\\alpha} ({\\bf x}^*,y^*,z^*) \n=\n\\sum_{i=1}^p f_i(x_i^*)+h(y^*).\n\\]\n\n\\end{lemma}","sha256_utf8":"309450f1ed8b1d93e95fa1ad59dd2275cf8f73746209f0fea700b25f44bbbe16"}]},{"id":"S073","semantic_type":"proof","kind":"proof","title":"聚点函数值一致性完整证明","source_units":[{"start":51171,"end":51864,"start_line":1169,"end_line":1185,"raw_tex":"\\begin{proof}\nLet $({\\bf x}^*,y^*,z^*)\\in\\omega(\\{ ({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0})$\nand let the subsequence $\\{({\\bf x}^{k_j},y^{k_j},z^{k_j})\\}_{j\\ge 0}$ of the \nPADMM algorithm converges to \n$({\\bf x}^*,y^*,z^*)$.\nClearly $\\|\\Delta y^{k_j}\\|$ and $\\|B\\tr \\Delta z^{k_j}\\|$ both converge to zero as $j$ goes to infinity.\nThis, together with Lemma\n Lemma \\ref{thm-cluster} and \\ref{thm-cluster0} give\n%\n\\begin{eqnarray*}\n\\lim_{j\\to\\infty} \\bar {\\mathcal L}_{k_j} =\\lim_{j\\to\\infty} \\mathcal L^{\\alpha} (x^{k_j},y^{k_j},z^{k_j})\n&=&\\mathcal L^{\\alpha}(x^*,y^*,z^*)\\\\\n&=&\\sum_{i=1}^pf_i(x_i^*)+h(y^*)\n=\\bar{\\mathcal L}({\\bf x}^*,y^*,z^*, y^*,z^*). \\quad\\quad\\square\n\\end{eqnarray*}\n%\n\\end{proof}","sha256_utf8":"78d55783d8c2e08a299b03076e283ce5d115d5acd80f5ca81e9901d858aefa9f"}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","title":"利用增量消失及聚点可行性取极限","source_units":[{"start":51171,"end":51864,"start_line":1169,"end_line":1185,"raw_tex":"\\begin{proof}\nLet $({\\bf x}^*,y^*,z^*)\\in\\omega(\\{ ({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0})$\nand let the subsequence $\\{({\\bf x}^{k_j},y^{k_j},z^{k_j})\\}_{j\\ge 0}$ of the \nPADMM algorithm converges to \n$({\\bf x}^*,y^*,z^*)$.\nClearly $\\|\\Delta y^{k_j}\\|$ and $\\|B\\tr \\Delta z^{k_j}\\|$ both converge to zero as $j$ goes to infinity.\nThis, together with Lemma\n Lemma \\ref{thm-cluster} and \\ref{thm-cluster0} give\n%\n\\begin{eqnarray*}\n\\lim_{j\\to\\infty} \\bar {\\mathcal L}_{k_j} =\\lim_{j\\to\\infty} \\mathcal L^{\\alpha} (x^{k_j},y^{k_j},z^{k_j})\n&=&\\mathcal L^{\\alpha}(x^*,y^*,z^*)\\\\\n&=&\\sum_{i=1}^pf_i(x_i^*)+h(y^*)\n=\\bar{\\mathcal L}({\\bf x}^*,y^*,z^*, y^*,z^*). \\quad\\quad\\square\n\\end{eqnarray*}\n%\n\\end{proof}","sha256_utf8":"78d55783d8c2e08a299b03076e283ce5d115d5acd80f5ca81e9901d858aefa9f"}]},{"id":"S075","semantic_type":"lemma","kind":"lemma","title":"修正势函数的次梯度界","source_units":[{"start":51905,"end":53099,"start_line":1187,"end_line":1217,"raw_tex":"\\begin{lemma}\\label{lma-D}\n{(subgradient bound for $\\bar{\\mathcal L}_k$)}\nLet $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ be a sequence generated by the PADMM algorithm\nand assume that $\\big(\\{d_{x_i}^k\\}_{i=1}^p,d_y^k,d_z^k\\big)\\in\\partial \\mathcal L^{\\alpha}({\\bf x}^k,y^k,z^k)$. Then \n%\n\\[\ns^{k}:=\\Big( \\{s^{k}_{x_i}\\}_{i=1}^p, s^{k}_y, s^{k}_z, s^{k}_{y'},s^{k}_{z'} \\Big)\\in\n\\partial \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1}),\\quad k\\ge 1\n\\]\n%\nwhere\n%\n$s_{x_i}^{k}:=d^{k}_{x_i}, i=1,\\dots, p$,\n$s_y^{k}:=d^{k}_y+2\\epsilon_0c_3 \\Delta y^{k}$,\n$s_z^{k}:=d^{k}_z+2\\epsilon_0c_5BB\\tr \\Delta z^{k}$,\n$s_{y'}^{k}:=-2\\epsilon_0c_3 \\Delta y^{k}$,\n$s_{z'}^{k}:=-2\\epsilon_0c_5BB\\tr \\Delta z^{k}$,\nand $\\epsilon_0>1$, and $c_3$ and $c_5$ are defined in  (\\ref{quantities}).\nMoreover,\n\\begin{eqnarray}\\label{nD}\n|||s^{k}|||\\le \\tilde\\rho \\Big(\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|+\\|\\Delta y^{k}\\|+\\|\\Delta z^{k}\\|\\Big),\n\\end{eqnarray}\nwhere \n$\n\\tilde\\rho:=\\max\\Big\\{\n\\|{\\bf Q}\\|+\\|{\\bf A}\\|^2,\\;\n\\|P\\|+\\alpha \\|{\\bf A}\\|\\|B\\|+4\\epsilon_0c_3, \\;\n\\|{\\bf A}\\|  +\\|B\\|+4\\epsilon_0c_5\\|B\\|^2 + \\frac{1}{\\alpha\\beta}\\Big\\},\n$\nand $\\|{\\bf Q}$ and $\\|{\\bf A}\\|$ are defined in Lemma \\ref{thm-d}.\n\\end{lemma}","sha256_utf8":"10a474ad587b6506b308f045bb1a2060c8cd33035ef4ee453b5d67cfab1e622b"}]},{"id":"S076","semantic_type":"proof","kind":"proof","title":"修正次梯度界完整证明","source_units":[{"start":53102,"end":55814,"start_line":1219,"end_line":1302,"raw_tex":"\\begin{proof}\nLet $k\\ge 1$ be fixed and $(\\{d_{x_i}^{k}\\}_{i=1}^p, d_y^{k}, d_z^{k})\\in \\partial \\mathcal L^{\\alpha}(x^k,y^k,z^k)$. By taking partial derivatives of $\\bar{\\mathcal L}$ with respect to $x,y,z,y',z'$ and evaluating \nthem at $({\\bf x}^k,y^k,z^k,y^{k-1},z^{k-1})$ we obtain\n%\n\\begin{eqnarray*}\n%s_{x_i}^{k}&:=& \n\\partial_{x_i} \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1})\n&=&\\partial_{x_i} \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k})\n\\ni d_{x_i}^{k},\\; i=1,\\dots,p;\\\\\n%s_y^{k}&:=&\n\\nabla_y  \\bar{\\mathcal L} ({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1})\n&=&\\nabla_y \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k})\n+2\\epsilon_0c_3 \\Delta y^k=\nd_y^{k}+2\\epsilon_0c_3 \\Delta y^k;\\\\\n%s_z^{k}&:=&\n\\nabla_z  \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1})\n&=&\\nabla_z \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k})\n+2\\epsilon_0c_5 BB\\tr \\Delta z^{k}=\nd_z^{k}\n+2\\epsilon_0c_5 BB\\tr\\Delta z^{k};\\\\\n%s_{y'}^{k}&:=&\n\\nabla_{y'}  \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1})\n&=&-2\\epsilon_0c_3\\Delta y^k;\\\\\n%d_{z'}^{k}&:=&\n\\nabla_{z'}  \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1})\n&=&-2\\epsilon_0c_5BB\\tr \\Delta z^k.\n\\end{eqnarray*}\n%\nThese equations follow that \n\\begin{eqnarray*}\ns^k =\n\\left[\n\\begin{array}{c}\ns^k_{x_1}\\\\\n\\vdots\\\\\ns^k_{x_p}\\\\\ns^k_{y} \\\\ \ns^k_{z}\\\\\ns^k_{y'}\\\\\ns^k_{z'}\n\\end{array}\n\\right]\n=\n\\left[\n\\begin{array}{l}\nd^k_{x_1}\\\\\n\\vdots\\\\[.1in]\nd^k_{x_p}\\\\\nd_y^{k}+2\\epsilon_0c_3 \\Delta y^k \\\\ \nd_z^{k}+2\\epsilon_0c_5 BB\\tr\\Delta z^{k}\\\\\n-2\\epsilon_0c_3\\Delta y^k\\\\\n-2\\epsilon_0c_5BB\\tr \\Delta z^k\n\\end{array}\n\\right]\\in \\partial \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1}).\n\\end{eqnarray*} \n%\nBy the triangle inequality we have\n%\n\\begin{eqnarray*}\n&\\|s_y^{k}\\|\\le \\|d^{k}_y\\|+2\\epsilon_0c_3\\|\\Delta y^k\\|,\\quad\\quad\n\\|s_z^{k}\\|\\le\\|d^{k}_z\\|+2\\epsilon_0c_5\\|B\\|^2 \\|\\Delta z^k\\|,&\\\\\n&\\|s_{y'}^{k}\\|\\le 2\\epsilon_0c_3\\|\\Delta y^k\\|,\\quad\\quad\n\\|s_z^{k}\\|\\le 2\\epsilon_0c_5 \\|B\\|^2 \\|\\Delta z^k\\|.&\n\\end{eqnarray*}\n%\nThus\n\\begin{eqnarray*}\n|||s^{k}|||&\\le&  \\sum_{i=1}^p\\|s_{x_i}^{k}\\|+\\|s_y^{k}\\|+\\|s_z^{k}\\|+ \\|s_{y'}^{k}\\|+\\|s_z^{k}\\|\\\\\n&\\le& \\sum_{i=1}^p \\|d^{k}_{x_i}\\|+  \\|d^{k}_y\\|+\\|d^{k}_z\\|+\n4\\epsilon_0c_3\\|\\Delta y^{k}\\|+\n4\\epsilon_0c_5 \\|B\\|^2 \\|\\Delta z^k\\|.\n\\end{eqnarray*}\n%\nBy Lemma \\ref{thm-d}, we have \n%\n\\begin{eqnarray*}\n&|||s^{k}|||\\le\n\\Big(\\|{\\bf Q}\\|+\\|{\\bf A}\\|^2\\Big) \\sum_{i=1}^p\\|\\Delta x_i^{k}\\|\n+\\Big(\\|P\\|+\\alpha \\|{\\bf A}\\|\\|B\\| +4\\epsilon_0c_3 \\Big)      \\|\\Delta y^{k}\\|&\\\\\n&+\\Big(\\|{\\bf A}\\|  +\\|B\\|+4\\epsilon_0c_5\\|B\\|^2+\\frac{1}{\\alpha\\beta}\\Big)  \\|\\Delta z^{k}\\|\n\\le \\tilde\\rho \\Big( \\sum_{i=1}^p\\|\\Delta x_i^{k}\\|+   \\|\\Delta y^{k}\\|+  \\|\\Delta z^{k}\\|\\Big). \\quad \\square&\n\\end{eqnarray*}\n\n\\end{proof}","sha256_utf8":"4e60370f42a9fabd20b4cfa7c7285e75adc7ecc49bfee69050efab4fe484294b"}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","title":"计算修正势函数各偏导数","source_units":[{"start":53102,"end":54277,"start_line":1219,"end_line":1245,"raw_tex":"\\begin{proof}\nLet $k\\ge 1$ be fixed and $(\\{d_{x_i}^{k}\\}_{i=1}^p, d_y^{k}, d_z^{k})\\in \\partial \\mathcal L^{\\alpha}(x^k,y^k,z^k)$. By taking partial derivatives of $\\bar{\\mathcal L}$ with respect to $x,y,z,y',z'$ and evaluating \nthem at $({\\bf x}^k,y^k,z^k,y^{k-1},z^{k-1})$ we obtain\n%\n\\begin{eqnarray*}\n%s_{x_i}^{k}&:=& \n\\partial_{x_i} \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1})\n&=&\\partial_{x_i} \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k})\n\\ni d_{x_i}^{k},\\; i=1,\\dots,p;\\\\\n%s_y^{k}&:=&\n\\nabla_y  \\bar{\\mathcal L} ({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1})\n&=&\\nabla_y \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k})\n+2\\epsilon_0c_3 \\Delta y^k=\nd_y^{k}+2\\epsilon_0c_3 \\Delta y^k;\\\\\n%s_z^{k}&:=&\n\\nabla_z  \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1})\n&=&\\nabla_z \\mathcal L^{\\alpha}({\\bf x}^{k},y^{k},z^{k})\n+2\\epsilon_0c_5 BB\\tr \\Delta z^{k}=\nd_z^{k}\n+2\\epsilon_0c_5 BB\\tr\\Delta z^{k};\\\\\n%s_{y'}^{k}&:=&\n\\nabla_{y'}  \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1})\n&=&-2\\epsilon_0c_3\\Delta y^k;\\\\\n%d_{z'}^{k}&:=&\n\\nabla_{z'}  \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1})\n&=&-2\\epsilon_0c_5BB\\tr \\Delta z^k.\n\\end{eqnarray*}\n","sha256_utf8":"b1584772a371f7aba1d3a132d6eddf03d535e1ddec1d01ccf9b365592f3eb4ce"}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","title":"组装修正势函数次梯度向量","source_units":[{"start":54279,"end":54765,"start_line":1247,"end_line":1273,"raw_tex":"These equations follow that \n\\begin{eqnarray*}\ns^k =\n\\left[\n\\begin{array}{c}\ns^k_{x_1}\\\\\n\\vdots\\\\\ns^k_{x_p}\\\\\ns^k_{y} \\\\ \ns^k_{z}\\\\\ns^k_{y'}\\\\\ns^k_{z'}\n\\end{array}\n\\right]\n=\n\\left[\n\\begin{array}{l}\nd^k_{x_1}\\\\\n\\vdots\\\\[.1in]\nd^k_{x_p}\\\\\nd_y^{k}+2\\epsilon_0c_3 \\Delta y^k \\\\ \nd_z^{k}+2\\epsilon_0c_5 BB\\tr\\Delta z^{k}\\\\\n-2\\epsilon_0c_3\\Delta y^k\\\\\n-2\\epsilon_0c_5BB\\tr \\Delta z^k\n\\end{array}\n\\right]\\in \\partial \\bar{\\mathcal L}({\\bf x}^{k},y^{k},z^{k},y^{k-1},z^{k-1}).\n\\end{eqnarray*} \n","sha256_utf8":"0c77ad1dec4fd8f4cf8a12fdd035ae247634d6634cda5ae1c539db76ac292b46"}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","title":"估计修正次梯度分量及总范数","source_units":[{"start":54767,"end":55361,"start_line":1275,"end_line":1290,"raw_tex":"By the triangle inequality we have\n%\n\\begin{eqnarray*}\n&\\|s_y^{k}\\|\\le \\|d^{k}_y\\|+2\\epsilon_0c_3\\|\\Delta y^k\\|,\\quad\\quad\n\\|s_z^{k}\\|\\le\\|d^{k}_z\\|+2\\epsilon_0c_5\\|B\\|^2 \\|\\Delta z^k\\|,&\\\\\n&\\|s_{y'}^{k}\\|\\le 2\\epsilon_0c_3\\|\\Delta y^k\\|,\\quad\\quad\n\\|s_z^{k}\\|\\le 2\\epsilon_0c_5 \\|B\\|^2 \\|\\Delta z^k\\|.&\n\\end{eqnarray*}\n%\nThus\n\\begin{eqnarray*}\n|||s^{k}|||&\\le&  \\sum_{i=1}^p\\|s_{x_i}^{k}\\|+\\|s_y^{k}\\|+\\|s_z^{k}\\|+ \\|s_{y'}^{k}\\|+\\|s_z^{k}\\|\\\\\n&\\le& \\sum_{i=1}^p \\|d^{k}_{x_i}\\|+  \\|d^{k}_y\\|+\\|d^{k}_z\\|+\n4\\epsilon_0c_3\\|\\Delta y^{k}\\|+\n4\\epsilon_0c_5 \\|B\\|^2 \\|\\Delta z^k\\|.\n\\end{eqnarray*}\n","sha256_utf8":"f523a010f2f49c3993c8d5d875573421b6e47e408e161d517c13b3cfb522e87d"}]},{"id":"S080","semantic_type":"proof_step","kind":"proof_step","title":"代入原次梯度界确定统一常数","source_units":[{"start":55363,"end":55814,"start_line":1292,"end_line":1302,"raw_tex":"By Lemma \\ref{thm-d}, we have \n%\n\\begin{eqnarray*}\n&|||s^{k}|||\\le\n\\Big(\\|{\\bf Q}\\|+\\|{\\bf A}\\|^2\\Big) \\sum_{i=1}^p\\|\\Delta x_i^{k}\\|\n+\\Big(\\|P\\|+\\alpha \\|{\\bf A}\\|\\|B\\| +4\\epsilon_0c_3 \\Big)      \\|\\Delta y^{k}\\|&\\\\\n&+\\Big(\\|{\\bf A}\\|  +\\|B\\|+4\\epsilon_0c_5\\|B\\|^2+\\frac{1}{\\alpha\\beta}\\Big)  \\|\\Delta z^{k}\\|\n\\le \\tilde\\rho \\Big( \\sum_{i=1}^p\\|\\Delta x_i^{k}\\|+   \\|\\Delta y^{k}\\|+  \\|\\Delta z^{k}\\|\\Big). \\quad \\square&\n\\end{eqnarray*}\n\n\\end{proof}","sha256_utf8":"1502cf7015a7fffb5d21f43821979ba7804de5baaa0be09390d957bd31f0058f"}]},{"id":"S081","semantic_type":"lemma","kind":"lemma","title":"迭代有界与相邻增量消失","source_units":[{"start":56288,"end":56686,"start_line":1313,"end_line":1321,"raw_tex":"\\begin{lemma} \\label{bddseq0}\nSuppose that A1-A4 hold and $\\sigma$ defined in Lemma \\ref{thm1} is positive.\nThen the sequence \n$\\{{\\bf x}^k, y^k, z^k\\}_{k\\ge 0}$ generated by the PADMM algorithm (\\ref{padmm}) is bounded,\nand $\n\\lim_{k\\to \\infty} \\|\\Delta x_1^{k}\\|=\\dots=\\lim_{k\\to \\infty} \\|\\Delta x_p^{k}\\|=\n\\lim_{k\\to \\infty} \\|\\Delta y^{k}\\|=\\lim_{k\\to \\infty} \\|\\Delta z^{k}\\|=0.\n$\n\\end{lemma}","sha256_utf8":"0bb40970e52304d514c05b51dfd9e52592e639b7ae79ced4c08259b5b0393d77"}]},{"id":"S082","semantic_type":"proof","kind":"proof","title":"有界性及增量消失完整证明","source_units":[{"start":56688,"end":60650,"start_line":1323,"end_line":1437,"raw_tex":"\\begin{proof}\nLet $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ be a sequence generated by the PADMM algorithm.\n%\nBy $\\sigma>0$, the sequence $\\{\\bar{\\mathcal L}_k\\}_{k\\ge 1}$ \ndefined in (\\ref{Lbark}) is monotonically decreasing, hence it is upper bounded by \n$\\bar{\\mathcal L}_1=\\bar{\\mathcal L}({\\bf x}^1,y^1,z^1,y^0,z^0)$.\nThis follows that \n%\n\\begin{eqnarray}\\label{ineq10}\n\\begin{array}{c}\n\\sum_{i=1}^pf(x_i^{k})+h(y^{k})\n+\\frac{\\alpha}{2}\\|{\\bf Ax}^{k}+By^{k}\n+\\alpha^{-1}z^{k}+b\\|^2-\\frac{1}{2\\alpha}\\|z^{k}\\|^2\n\\\\[.1in]\n+\\sigma \\sum_{i=1}^p \\|\\Delta x_i^{k}\\|^2\n +\\sigma\\|\\Delta z^{k}\\|^2\n+(\\sigma+\\epsilon_0c_3) \\|\\Delta y^{k}\\|^2\n+\\epsilon_0c_5\\|B\\tr \\Delta z^{k}\\|^2\n\\le \\bar{\\mathcal L}_1\n \\end{array}\n\\end{eqnarray}\n%\nfor all $k\\in\\mathbb Z_+$.\nBy Lemma \\ref{bounddual} (iii), (\\ref{ineq10}) leads to \n%\n\\begin{eqnarray}\\label{thmbdd-eq1}\n\\begin{array}{c}\n\\sum_{i=1}^pf(x_i^{k})+\\frac{\\alpha}{2}\\|{\\bf Ax}^{k}+By^{k}+\\alpha^{-1}z^{k}+b\\|^2\n+r_1\\|\\Delta y^{k}\\|^2\\\\[.1in]\n+ \\sigma\\sum_{i=1}^p\\|\\Delta x^{k}\\|^2\n+ r_2\\|B\\tr \\Delta z^{k}\\|^2\n+\\sigma\\|\\Delta z^{k}\\|^2\n\\le\\bar{\\mathcal L}_1\n%- \\inf_{x,y}\\Big\\{g(x)- c_1 \\Big\\|\\nabla g(x)\\Big\\|^2\\Big\\}\n-\\inf_y\\Big\\{ h(y)-c_1 \\|\\nabla h(y)\\|^2 \\Big\\},\n\\end{array}\n\\end{eqnarray}\n%\nwhere $r_1:= \\epsilon_0c_3+\\sigma-\\|P\\|^2c_1>0$\nand $r_2:=\\epsilon_0c_5-c_6>0$. \nBy A2,  since $h$ is $L_h$ Lipschitz differentiable then for any $k\\ge 1$ \nand $\\delta>0$ we have\n%\n$\nh\\big(y^k-\\delta \\nabla h(y^k)\\big)\\le h(y^k)- \n\\big(\\delta-\\frac{L_h\\delta^2}{2}\\big)\\|\\nabla h(y^k)\\|^2.\n$\n%\nAgain by A2, $h$ is bounded from below then\n%\n\\begin{eqnarray}\\label{hbd}\n-\\infty<\\inf\\{h(y)- \\Big(\\delta-\\frac{L_h\\delta^2}{2}\\Big)\\|\\nabla h(y)\\|^2: \\; y\\in\\mathbb R^q\\}.\n\\end{eqnarray}\n%\nWe choose $\\delta (>0)$ such that \n$\\delta-\\frac{L_h\\delta^2}{2}=c_1$. By (\\ref{hbd}), the right hand side of (\\ref{thmbdd-eq1}) is finite\nhence\n%\n\\begin{eqnarray}\\label{finite}\n\\sum_{i=1}^pf(x_i^{k})+\\|{\\bf Ax}^{k}+By^{k}+\\alpha^{-1}z^{k}+b\\|^2\n+\\|\\Delta y^{k}\\|^2 +\\|\\Delta z^k\\|^2<+\\infty.\n\\end{eqnarray}\n%\nBy A1, since $f_i, i=1,\\dots,p$ are coercive and $\\sum_{i=1}^pf(x_i^{k})<+\\infty$, then the sequence \n$\\{x_i^k\\}_{k\\ge 0}$  is bounded for $i=1,\\dots,p$, and so are \n$\\{{\\bf x}^k\\}_{k\\ge 0}$ and $\\{{\\bf Ax}^k\\}_{k\\ge 0}$.\n%\n%In addition, by (\\ref{finite}) the sequences \n%$\\{Ax^{k}+By^{k}+\\alpha^{-1}z^{k}+c\\}_{k\\ge k_0}$ and $\\{ \\Delta y^{k}\\}_{k\\ge k_0}$ are bounded, \n%then $\\{A\\Delta x^k\\}_{k\\ge k_0}$ and $\\{B\\Delta y^k\\}_{k\\ge k_0}$\n%are bounded which follows that $\\{\\Delta z^k\\}_{k\\ge 0}$ is bounded.\n\nBy the $z$ iterate of PADMM we have\n$\nBy^{k} = \\frac{1}{\\alpha\\beta} \\Delta z^{k} -{\\bf Ax}^{k}-b.\n$\nBy (\\ref{finite}), $\\{\\Delta z^k\\}_{k\\ge 0}$ is bounded.\nBy A3, $B\\tr B$ is invertible, thus $\\{y^k\\}_{k\\ge 0}$ is bounded.\nFinally, by (\\ref{finite}) $\\{{\\bf Ax}^{k}+By^{k}+\\frac{1}{\\alpha}z^{k}\\}_{k\\ge 0}$ is bounded\nand ${\\bf Ax}^k$ and $By^k$ are bounded, so \n$\\{z^k\\}_{k\\ge 0}$ is then bounded.\n%\nSumming up (\\ref{Lbarmon}) for $k=1,\\dots, K-1$, $K\\ge 2$ gives\n%\n\\[\n\\sum_{k=1}^{K}\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|^2\n+\\|\\Delta y^{k}\\|^2\n+\\|\\Delta z^{k}\\|^2\\Big\\}\\le \\frac{1}{\\sigma}(\\bar{\\mathcal L}_1 - \\bar{\\mathcal L}_K) .\n\\]\nSince $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ is bounded, by Lemma \\ref{thm1} (ii) the right hand side is finite for any $K$.\nWe let $K$ approach to infinity to obtain \n\\begin{eqnarray*}\n\\sum_{k=1}^{\\infty}\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|^2 +\\|\\Delta y^{k}\\|^2+\\|\\Delta z^{k}\\|^2\\Big\\}<\n\\frac{1}{\\sigma}(\\bar{\\mathcal L}_1 - \\inf_{k\\ge 1}\\bar{\\mathcal L}_k) <\n+\\infty.\n\\end{eqnarray*}\nThis follows \n$\n\\displaystyle{\\lim_{k\\to \\infty}} \\Big(\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|^2 +\\|\\Delta y^{k}\\|^2+\\|\\Delta z^{k}\\|^2\\Big)=0,\n$\nand consequently \n$\n\\displaystyle{\\lim_{k\\to \\infty}} \\|\\Delta x_1^{k}\\|\n=\\dots\n=\\displaystyle{\\lim_{k\\to \\infty}} \\|\\Delta x_p^{k}\\|\n=\n\\displaystyle{\\lim_{k\\to \\infty}}\\|\\Delta y^{k}\\|\n=\n\\displaystyle{\\lim_{k\\to \\infty}}\\|\\Delta z^{k}\\| =0.\\quad\\square\n$\n\n\n\\end{proof}","sha256_utf8":"bac6175215f7924fe0bcaa0ce68f6bfd2e5cbc5bb6d8e1a3f46283d0da83dd27"}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","title":"势函数下降与平方配方给出上界","source_units":[{"start":56688,"end":57436,"start_line":1323,"end_line":1345,"raw_tex":"\\begin{proof}\nLet $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ be a sequence generated by the PADMM algorithm.\n%\nBy $\\sigma>0$, the sequence $\\{\\bar{\\mathcal L}_k\\}_{k\\ge 1}$ \ndefined in (\\ref{Lbark}) is monotonically decreasing, hence it is upper bounded by \n$\\bar{\\mathcal L}_1=\\bar{\\mathcal L}({\\bf x}^1,y^1,z^1,y^0,z^0)$.\nThis follows that \n%\n\\begin{eqnarray}\\label{ineq10}\n\\begin{array}{c}\n\\sum_{i=1}^pf(x_i^{k})+h(y^{k})\n+\\frac{\\alpha}{2}\\|{\\bf Ax}^{k}+By^{k}\n+\\alpha^{-1}z^{k}+b\\|^2-\\frac{1}{2\\alpha}\\|z^{k}\\|^2\n\\\\[.1in]\n+\\sigma \\sum_{i=1}^p \\|\\Delta x_i^{k}\\|^2\n +\\sigma\\|\\Delta z^{k}\\|^2\n+(\\sigma+\\epsilon_0c_3) \\|\\Delta y^{k}\\|^2\n+\\epsilon_0c_5\\|B\\tr \\Delta z^{k}\\|^2\n\\le \\bar{\\mathcal L}_1\n \\end{array}\n\\end{eqnarray}\n%\nfor all $k\\in\\mathbb Z_+$.\n","sha256_utf8":"c42354ae514018a964e3e88b2d8a1194492fdb419aa75a9c4f7a37c39fd02c21"}]},{"id":"S084","semantic_type":"proof_step","kind":"proof_step","title":"对偶范数界导出目标与增量估计","source_units":[{"start":57436,"end":57998,"start_line":1346,"end_line":1362,"raw_tex":"By Lemma \\ref{bounddual} (iii), (\\ref{ineq10}) leads to \n%\n\\begin{eqnarray}\\label{thmbdd-eq1}\n\\begin{array}{c}\n\\sum_{i=1}^pf(x_i^{k})+\\frac{\\alpha}{2}\\|{\\bf Ax}^{k}+By^{k}+\\alpha^{-1}z^{k}+b\\|^2\n+r_1\\|\\Delta y^{k}\\|^2\\\\[.1in]\n+ \\sigma\\sum_{i=1}^p\\|\\Delta x^{k}\\|^2\n+ r_2\\|B\\tr \\Delta z^{k}\\|^2\n+\\sigma\\|\\Delta z^{k}\\|^2\n\\le\\bar{\\mathcal L}_1\n%- \\inf_{x,y}\\Big\\{g(x)- c_1 \\Big\\|\\nabla g(x)\\Big\\|^2\\Big\\}\n-\\inf_y\\Big\\{ h(y)-c_1 \\|\\nabla h(y)\\|^2 \\Big\\},\n\\end{array}\n\\end{eqnarray}\n%\nwhere $r_1:= \\epsilon_0c_3+\\sigma-\\|P\\|^2c_1>0$\nand $r_2:=\\epsilon_0c_5-c_6>0$. \n","sha256_utf8":"3383111b18681036854ff1e0e5924171dbe44354619cadd94f013bed03e717cd"}]},{"id":"S085","semantic_type":"proof_step","kind":"proof_step","title":"梯度下降估计控制右端下确界","source_units":[{"start":57998,"end":58709,"start_line":1363,"end_line":1384,"raw_tex":"By A2,  since $h$ is $L_h$ Lipschitz differentiable then for any $k\\ge 1$ \nand $\\delta>0$ we have\n%\n$\nh\\big(y^k-\\delta \\nabla h(y^k)\\big)\\le h(y^k)- \n\\big(\\delta-\\frac{L_h\\delta^2}{2}\\big)\\|\\nabla h(y^k)\\|^2.\n$\n%\nAgain by A2, $h$ is bounded from below then\n%\n\\begin{eqnarray}\\label{hbd}\n-\\infty<\\inf\\{h(y)- \\Big(\\delta-\\frac{L_h\\delta^2}{2}\\Big)\\|\\nabla h(y)\\|^2: \\; y\\in\\mathbb R^q\\}.\n\\end{eqnarray}\n%\nWe choose $\\delta (>0)$ such that \n$\\delta-\\frac{L_h\\delta^2}{2}=c_1$. By (\\ref{hbd}), the right hand side of (\\ref{thmbdd-eq1}) is finite\nhence\n%\n\\begin{eqnarray}\\label{finite}\n\\sum_{i=1}^pf(x_i^{k})+\\|{\\bf Ax}^{k}+By^{k}+\\alpha^{-1}z^{k}+b\\|^2\n+\\|\\Delta y^{k}\\|^2 +\\|\\Delta z^k\\|^2<+\\infty.\n\\end{eqnarray}\n","sha256_utf8":"7d0a9f7f79cd35dea9fbae2e35bd5c65dd9f013e9fc212ed7d79e71938c3f629"}]},{"id":"S086","semantic_type":"proof_step","kind":"proof_step","title":"由强制性及约束更新推导迭代有界","source_units":[{"start":58711,"end":59622,"start_line":1386,"end_line":1403,"raw_tex":"By A1, since $f_i, i=1,\\dots,p$ are coercive and $\\sum_{i=1}^pf(x_i^{k})<+\\infty$, then the sequence \n$\\{x_i^k\\}_{k\\ge 0}$  is bounded for $i=1,\\dots,p$, and so are \n$\\{{\\bf x}^k\\}_{k\\ge 0}$ and $\\{{\\bf Ax}^k\\}_{k\\ge 0}$.\n%\n%In addition, by (\\ref{finite}) the sequences \n%$\\{Ax^{k}+By^{k}+\\alpha^{-1}z^{k}+c\\}_{k\\ge k_0}$ and $\\{ \\Delta y^{k}\\}_{k\\ge k_0}$ are bounded, \n%then $\\{A\\Delta x^k\\}_{k\\ge k_0}$ and $\\{B\\Delta y^k\\}_{k\\ge k_0}$\n%are bounded which follows that $\\{\\Delta z^k\\}_{k\\ge 0}$ is bounded.\n\nBy the $z$ iterate of PADMM we have\n$\nBy^{k} = \\frac{1}{\\alpha\\beta} \\Delta z^{k} -{\\bf Ax}^{k}-b.\n$\nBy (\\ref{finite}), $\\{\\Delta z^k\\}_{k\\ge 0}$ is bounded.\nBy A3, $B\\tr B$ is invertible, thus $\\{y^k\\}_{k\\ge 0}$ is bounded.\nFinally, by (\\ref{finite}) $\\{{\\bf Ax}^{k}+By^{k}+\\frac{1}{\\alpha}z^{k}\\}_{k\\ge 0}$ is bounded\nand ${\\bf Ax}^k$ and $By^k$ are bounded, so \n$\\{z^k\\}_{k\\ge 0}$ is then bounded.\n","sha256_utf8":"9e14b8112dda7edeb1dbd0d1f4913041e214e209f47a2f0ab37df9cf4cfd71f5"}]},{"id":"S087","semantic_type":"proof_step","kind":"proof_step","title":"求和充分下降得到平方增量可和","source_units":[{"start":59624,"end":60244,"start_line":1405,"end_line":1420,"raw_tex":"Summing up (\\ref{Lbarmon}) for $k=1,\\dots, K-1$, $K\\ge 2$ gives\n%\n\\[\n\\sum_{k=1}^{K}\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|^2\n+\\|\\Delta y^{k}\\|^2\n+\\|\\Delta z^{k}\\|^2\\Big\\}\\le \\frac{1}{\\sigma}(\\bar{\\mathcal L}_1 - \\bar{\\mathcal L}_K) .\n\\]\nSince $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ is bounded, by Lemma \\ref{thm1} (ii) the right hand side is finite for any $K$.\nWe let $K$ approach to infinity to obtain \n\\begin{eqnarray*}\n\\sum_{k=1}^{\\infty}\n\\Big\\{\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|^2 +\\|\\Delta y^{k}\\|^2+\\|\\Delta z^{k}\\|^2\\Big\\}<\n\\frac{1}{\\sigma}(\\bar{\\mathcal L}_1 - \\inf_{k\\ge 1}\\bar{\\mathcal L}_k) <\n+\\infty.\n\\end{eqnarray*}\n","sha256_utf8":"b1eebb4c1b11bc7e5b05eec863a06442fa411c55fbbd0622fd48b6e86e5c798e"}]},{"id":"S088","semantic_type":"proof_step","kind":"proof_step","title":"平方可和推出各相邻增量消失","source_units":[{"start":60244,"end":60650,"start_line":1421,"end_line":1437,"raw_tex":"This follows \n$\n\\displaystyle{\\lim_{k\\to \\infty}} \\Big(\\sum_{i=1}^p\\|\\Delta x_i^{k}\\|^2 +\\|\\Delta y^{k}\\|^2+\\|\\Delta z^{k}\\|^2\\Big)=0,\n$\nand consequently \n$\n\\displaystyle{\\lim_{k\\to \\infty}} \\|\\Delta x_1^{k}\\|\n=\\dots\n=\\displaystyle{\\lim_{k\\to \\infty}} \\|\\Delta x_p^{k}\\|\n=\n\\displaystyle{\\lim_{k\\to \\infty}}\\|\\Delta y^{k}\\|\n=\n\\displaystyle{\\lim_{k\\to \\infty}}\\|\\Delta z^{k}\\| =0.\\quad\\square\n$\n\n\n\\end{proof}","sha256_utf8":"ef567cc17fd6f9c9463142b087d064e23f87ae6f38401f0c176121ba9f946094"}]},{"id":"S089","semantic_type":"lemma","kind":"lemma","title":"扩展迭代聚点集及共同极限","source_units":[{"start":60690,"end":61635,"start_line":1440,"end_line":1458,"raw_tex":"\\begin{lemma}\\label{lma-cluster2}\nLet $\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ be a sequence generated by the PADMM algorithm (\\ref{padmm}),\nand denote\n\\begin{eqnarray}\\label{omega}\n\\Omega:=\\omega \\Big(\\{({\\bf x}^k,y^k,z^k,y^{k-1},z^{k-1})\\}_{k\\ge 1}\\Big).\n\\end{eqnarray}\nThen the following statements are true\n\\begin{itemize}\n\\item [(i)] $\\Omega$ is nonempty, connected, and compact.\n\\item [(ii)] $\\Omega \\subseteq \\{({\\bf x}^*, y^*, z^*,  y^*, z^*)\n\\in\\mathbb R^n\\times\\mathbb R^q\\times\\mathbb R^m\\times\\mathbb R^q\\times\\mathbb R^m:\n\\;({\\bf x}^*, y^*, z^*)\\in {\\rm crit} (\\mathcal L^{\\alpha})\\big\\}$\n%\n\\item [(iii)] $\\lim_{k\\to\\infty} {\\rm dist} \\big[ ({\\bf x}^k, y^k, z^k, y^{k-1}, x^{k-1}), \\Omega \\big]=0$.\n\\item [(iv)] The sequences $\\{\\bar{\\mathcal L}_k\\}_{k\\ge 1}$,\n $\\{\\bar{\\mathcal L}^{\\alpha}({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$,\n and $\\{\\sum_{i=1}^pf_i(x_i^k)+h(y^k)\\}_{k\\ge 0}$ approaches the same limit on $\\Omega$. \n\\end{itemize}\n\\end{lemma}","sha256_utf8":"c6323881f43cf6fca7c823643bab033bd315f9acc7fe2201eb7751b871d7651f"}]},{"id":"S090","semantic_type":"proof","kind":"proof","title":"聚点集性质的作者归约证明","source_units":[{"start":61636,"end":61776,"start_line":1459,"end_line":1461,"raw_tex":"\\begin{proof}\nThese results follow from Lemmas \\ref{thm1}, \\ref{samelimit}, and \\ref{bddseq0}. Thus we omit the proof. $\\square$\n\\end{proof}","sha256_utf8":"3c0364c46fddc96f3f9d6991df4ef0f182e576f0e8a4b8af0532cbe42f5dcf46"}]},{"id":"S091","semantic_type":"proof_step","kind":"proof_step","title":"归约至下降同值与有界性引理","source_units":[{"start":61636,"end":61776,"start_line":1459,"end_line":1461,"raw_tex":"\\begin{proof}\nThese results follow from Lemmas \\ref{thm1}, \\ref{samelimit}, and \\ref{bddseq0}. Thus we omit the proof. $\\square$\n\\end{proof}","sha256_utf8":"3c0364c46fddc96f3f9d6991df4ef0f182e576f0e8a4b8af0532cbe42f5dcf46"}]},{"id":"S092","semantic_type":"external_fact","kind":"external_fact","title":"极限次微分的标准演算规则","source_units":[],"declared_external":true,"declaration_text":["Fermat's rule holds for proper lower-semicontinuous functions. Adding a continuously differentiable function adds its gradient to the limiting subdifferential. If x_j→x, f(x_j)→f(x), s_j∈∂f(x_j), and s_j→s, then s∈∂f(x)."]},{"id":"S093","semantic_type":"external_fact","kind":"external_fact","title":"有界渐近正则序列的聚点集性质","source_units":[],"declared_external":true,"declaration_text":["For a bounded sequence v_k in a finite-dimensional Euclidean space with ‖v_{k+1}−v_k‖→0, its cluster set is nonempty, compact and connected, and dist(v_k,ω(v_k))→0. Coordinates differing by quantities tending to zero agree at every cluster point."]},{"id":"S094","semantic_type":"external_fact","kind":"external_fact","title":"有限长度序列的完备性结论","source_units":[],"declared_external":true,"declaration_text":["If ∑_k ‖u_{k+1}−u_k‖<∞ in a finite-dimensional normed space, then ‖u_q−u_p‖ is bounded by the corresponding summable tail; hence the sequence is Cauchy and converges."]},{"id":"S095","semantic_type":"external_fact","kind":"external_fact","title":"范数凹性及谱估计","source_units":[],"declared_external":true,"declaration_text":["Use triangle and Cauchy–Schwarz inequalities, convexity of the squared norm, and √(ab)≤γa/2+b/(2γ) for a,b≥0 and γ>0. For differentiable concave ψ and a≥b, ψ(a)−ψ(b)≥ψ′(a)(a−b). For v∈Im(B), ‖Bᵀv‖²≥λ₊₊(BᵀB)‖v‖²."]},{"id":"S096","semantic_type":"assumption","kind":"assumption","title":"主定理的正下降与有界KL假设","source_units":[{"start":61806,"end":62765,"start_line":1464,"end_line":1483,"raw_tex":"Suppose that  A1-A4 hold and $\\sigma$ defined in Lemma \\ref{thm1} is positive.\nLet the sequence \n$\\{({\\bf x}^k,y^k,z^k)\\}_{k\\ge 0}$ generated by the PADMM algorithm (\\ref{padmm}) be bounded and \n$\\bar{\\mathcal L}({\\bf x},y,z,y',z')$ satisfies the {K\\L} property on $\\Omega$ (\\ref{omega});\nthat is, for every \n$v^*:=({\\bf x}^*, y^*, z^*,  y^*, z^*) \\in \\Omega$\nthere exists $\\epsilon>0$, $\\eta\\in [0,+\\infty)$, and a function\n$\\psi\\in\\Psi_{\\eta}$ such that for any  \n$v:=({\\bf x},y,z,y',z')\\in\\mathcal S$ where \n%\n\\begin{eqnarray}\\label{eq:intersec}\n\\mathcal S:=\\Big\\{v \\in \\mathbb R^n\\times\\mathbb R^m\\times\\mathbb R^p\\times\\mathbb R^m\\times\\mathbb R^p:\n\\;{\\rm dist}(v,\\Omega)<\\epsilon\\} \\cap [\\bar{\\mathcal L}(v^*) <\\bar{\\mathcal L}< \\bar{\\mathcal L}(v^*)+\\eta],\n\\end{eqnarray}\nit holds \n%\n\\begin{eqnarray}\\label{eq:thm1KL}\n\\psi'\\Big(\\bar{\\mathcal L}(v)-\\bar{\\mathcal L}(v^*)\\Big) \\cdot\n{\\rm dist}\\big(0,\\partial\\bar{\\mathcal L}(v)\\big)\\ge 1.\n\\end{eqnarray}\n","sha256_utf8":"85bcf1139d539c020a5baf54f63d39baa99e86fe7fc3e2df65ebabc34c7f5b11"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S011","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S001","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S002","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S007","target_id":"S096","kind":"supports"},{"source_id":"S008","target_id":"S096","kind":"supports"},{"source_id":"S009","target_id":"S096","kind":"supports"},{"source_id":"S010","target_id":"S096","kind":"supports"},{"source_id":"S096","target_id":"S013","kind":"supports"},{"source_id":"S068","target_id":"S013","kind":"supports"},{"source_id":"S010","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S068","target_id":"S014","kind":"supports"},{"source_id":"S096","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S015","kind":"supports"},{"source_id":"S068","target_id":"S015","kind":"supports"},{"source_id":"S089","target_id":"S016","kind":"supports"},{"source_id":"S013","target_id":"S016","kind":"supports"},{"source_id":"S096","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S096","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S015","target_id":"S018","kind":"supports"},{"source_id":"S004","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S006","target_id":"S019","kind":"supports"},{"source_id":"S095","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S095","target_id":"S020","kind":"supports"},{"source_id":"S004","target_id":"S021","kind":"supports"},{"source_id":"S019","target_id":"S022","kind":"supports"},{"source_id":"S020","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S075","target_id":"S023","kind":"supports"},{"source_id":"S005","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S013","target_id":"S026","kind":"supports"},{"source_id":"S006","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S096","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S014","target_id":"S029","kind":"supports"},{"source_id":"S004","target_id":"S029","kind":"supports"},{"source_id":"S041","target_id":"S029","kind":"supports"},{"source_id":"S046","target_id":"S029","kind":"supports"},{"source_id":"S089","target_id":"S029","kind":"supports"},{"source_id":"S094","target_id":"S029","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S014","target_id":"S012","kind":"supports"},{"source_id":"S015","target_id":"S012","kind":"supports"},{"source_id":"S016","target_id":"S012","kind":"supports"},{"source_id":"S017","target_id":"S012","kind":"supports"},{"source_id":"S018","target_id":"S012","kind":"supports"},{"source_id":"S019","target_id":"S012","kind":"supports"},{"source_id":"S020","target_id":"S012","kind":"supports"},{"source_id":"S021","target_id":"S012","kind":"supports"},{"source_id":"S022","target_id":"S012","kind":"supports"},{"source_id":"S023","target_id":"S012","kind":"supports"},{"source_id":"S024","target_id":"S012","kind":"supports"},{"source_id":"S025","target_id":"S012","kind":"supports"},{"source_id":"S026","target_id":"S012","kind":"supports"},{"source_id":"S027","target_id":"S012","kind":"supports"},{"source_id":"S028","target_id":"S012","kind":"supports"},{"source_id":"S029","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S011","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S095","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S003","target_id":"S036","kind":"supports"},{"source_id":"S002","target_id":"S036","kind":"supports"},{"source_id":"S005","target_id":"S036","kind":"supports"},{"source_id":"S092","target_id":"S036","kind":"supports"},{"source_id":"S003","target_id":"S037","kind":"supports"},{"source_id":"S002","target_id":"S037","kind":"supports"},{"source_id":"S003","target_id":"S038","kind":"supports"},{"source_id":"S002","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S095","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S004","target_id":"S040","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S035","kind":"supports"},{"source_id":"S039","target_id":"S035","kind":"supports"},{"source_id":"S040","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S008","target_id":"S043","kind":"supports"},{"source_id":"S003","target_id":"S044","kind":"supports"},{"source_id":"S002","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S081","target_id":"S045","kind":"supports"},{"source_id":"S008","target_id":"S045","kind":"supports"},{"source_id":"S002","target_id":"S045","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S044","target_id":"S042","kind":"supports"},{"source_id":"S045","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S034","target_id":"S049","kind":"supports"},{"source_id":"S041","target_id":"S049","kind":"supports"},{"source_id":"S081","target_id":"S049","kind":"supports"},{"source_id":"S005","target_id":"S049","kind":"supports"},{"source_id":"S092","target_id":"S049","kind":"supports"},{"source_id":"S002","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S049","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S003","target_id":"S052","kind":"supports"},{"source_id":"S008","target_id":"S053","kind":"supports"},{"source_id":"S003","target_id":"S053","kind":"supports"},{"source_id":"S009","target_id":"S053","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S095","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S052","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S008","target_id":"S055","kind":"supports"},{"source_id":"S009","target_id":"S055","kind":"supports"},{"source_id":"S095","target_id":"S055","kind":"supports"},{"source_id":"S052","target_id":"S056","kind":"supports"},{"source_id":"S009","target_id":"S056","kind":"supports"},{"source_id":"S095","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S052","target_id":"S057","kind":"supports"},{"source_id":"S009","target_id":"S057","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S051","kind":"supports"},{"source_id":"S055","target_id":"S051","kind":"supports"},{"source_id":"S056","target_id":"S051","kind":"supports"},{"source_id":"S057","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S003","target_id":"S060","kind":"supports"},{"source_id":"S002","target_id":"S060","kind":"supports"},{"source_id":"S003","target_id":"S061","kind":"supports"},{"source_id":"S002","target_id":"S061","kind":"supports"},{"source_id":"S008","target_id":"S061","kind":"supports"},{"source_id":"S092","target_id":"S061","kind":"supports"},{"source_id":"S003","target_id":"S062","kind":"supports"},{"source_id":"S002","target_id":"S062","kind":"supports"},{"source_id":"S008","target_id":"S062","kind":"supports"},{"source_id":"S009","target_id":"S062","kind":"supports"},{"source_id":"S030","target_id":"S062","kind":"supports"},{"source_id":"S003","target_id":"S063","kind":"supports"},{"source_id":"S002","target_id":"S063","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S061","target_id":"S059","kind":"supports"},{"source_id":"S062","target_id":"S059","kind":"supports"},{"source_id":"S063","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S050","target_id":"S067","kind":"supports"},{"source_id":"S009","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"},{"source_id":"S067","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S070","kind":"supports"},{"source_id":"S010","target_id":"S070","kind":"supports"},{"source_id":"S009","target_id":"S070","kind":"supports"},{"source_id":"S008","target_id":"S070","kind":"supports"},{"source_id":"S096","target_id":"S070","kind":"supports"},{"source_id":"S008","target_id":"S071","kind":"supports"},{"source_id":"S002","target_id":"S071","kind":"supports"},{"source_id":"S010","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S069","kind":"supports"},{"source_id":"S071","target_id":"S069","kind":"supports"},{"source_id":"S069","target_id":"S068","kind":"supports"},{"source_id":"S041","target_id":"S074","kind":"supports"},{"source_id":"S046","target_id":"S074","kind":"supports"},{"source_id":"S081","target_id":"S074","kind":"supports"},{"source_id":"S010","target_id":"S074","kind":"supports"},{"source_id":"S002","target_id":"S074","kind":"supports"},{"source_id":"S074","target_id":"S073","kind":"supports"},{"source_id":"S073","target_id":"S072","kind":"supports"},{"source_id":"S010","target_id":"S077","kind":"supports"},{"source_id":"S009","target_id":"S077","kind":"supports"},{"source_id":"S005","target_id":"S077","kind":"supports"},{"source_id":"S092","target_id":"S077","kind":"supports"},{"source_id":"S077","target_id":"S078","kind":"supports"},{"source_id":"S078","target_id":"S079","kind":"supports"},{"source_id":"S095","target_id":"S079","kind":"supports"},{"source_id":"S079","target_id":"S080","kind":"supports"},{"source_id":"S034","target_id":"S080","kind":"supports"},{"source_id":"S004","target_id":"S080","kind":"supports"},{"source_id":"S077","target_id":"S076","kind":"supports"},{"source_id":"S078","target_id":"S076","kind":"supports"},{"source_id":"S079","target_id":"S076","kind":"supports"},{"source_id":"S080","target_id":"S076","kind":"supports"},{"source_id":"S076","target_id":"S075","kind":"supports"},{"source_id":"S068","target_id":"S083","kind":"supports"},{"source_id":"S010","target_id":"S083","kind":"supports"},{"source_id":"S002","target_id":"S083","kind":"supports"},{"source_id":"S096","target_id":"S083","kind":"supports"},{"source_id":"S083","target_id":"S084","kind":"supports"},{"source_id":"S050","target_id":"S084","kind":"supports"},{"source_id":"S009","target_id":"S084","kind":"supports"},{"source_id":"S008","target_id":"S085","kind":"supports"},{"source_id":"S030","target_id":"S085","kind":"supports"},{"source_id":"S084","target_id":"S085","kind":"supports"},{"source_id":"S085","target_id":"S086","kind":"supports"},{"source_id":"S008","target_id":"S086","kind":"supports"},{"source_id":"S003","target_id":"S086","kind":"supports"},{"source_id":"S068","target_id":"S087","kind":"supports"},{"source_id":"S086","target_id":"S087","kind":"supports"},{"source_id":"S087","target_id":"S088","kind":"supports"},{"source_id":"S083","target_id":"S082","kind":"supports"},{"source_id":"S084","target_id":"S082","kind":"supports"},{"source_id":"S085","target_id":"S082","kind":"supports"},{"source_id":"S086","target_id":"S082","kind":"supports"},{"source_id":"S087","target_id":"S082","kind":"supports"},{"source_id":"S088","target_id":"S082","kind":"supports"},{"source_id":"S082","target_id":"S081","kind":"supports"},{"source_id":"S068","target_id":"S091","kind":"supports"},{"source_id":"S072","target_id":"S091","kind":"supports"},{"source_id":"S081","target_id":"S091","kind":"supports"},{"source_id":"S093","target_id":"S091","kind":"supports"},{"source_id":"S091","target_id":"S090","kind":"supports"},{"source_id":"S090","target_id":"S089","kind":"supports"}]}}
{"record_id":"2010.06759v4","paper":{"title":"On Distributionally Robust Multistage Convex Optimization: New Algorithms and Complexity Analysis","authors":["Shixuan Zhang","Xu Andy Sun"],"arxiv_id":"2010.06759","version":"v4","journal":"SIAM Journal on Optimization","doi":"10.1137/25m1761690","abstract_url":"https://arxiv.org/abs/2010.06759v4","pdf_url":"https://arxiv.org/pdf/2010.06759v4"},"context":{"raw_tex":"Let \\(\\calT\\coloneqq\\{1,2,\\dots,T\\}\\) denote the set of stage indices and \\(\\calT'\\coloneqq\\calT\\setminus\\{1\\}\\). \nFor each \\(t\\in\\calT\\), the decision variable in stage \\(t\\) is denoted as \\(x_t\\) and constrained in a compact convex set \\(\\calX_t\\subset\\bbR^{d_t}\\) with dimension \\(d_t\\in\\bbZ_{\\ge0}\\).\nThe uncertainty in stage \\(t\\) is modeled as a random vector \\(\\xi_t\\) with some possibly unknown probability distribution and its convex support set denoted as \\(\\Xi_t\\).\nFor simplicity, we use \\(x_0\\) and \\(\\xi_1\\) to denote deterministic parameters, known as the initial conditions.\nThe cost incurred by the decisions and the uncertainty outcomes in stage \\(t\\) is modeled by a nonnegative lower semicontinuous function \\(f_t(x_{t-1},x_t;\\xi_t)\\), that is convex in \\((x_{t-1},x_t)\\) for each \\(\\xi_t\\in\\Xi_t\\) and allowed to take \\(+\\infty\\) for infeasibility.\n\n\nLet \\(\\calP_t\\) denote a set of Borel probability measures supported on the uncertainty set \\(\\Xi_t\\), called the ambiguity set, for \\(t\\in\\calT'\\).\nOur DR-MCO is defined as\n\\begin{align}\\label{eq:DR-MSCP-Def}\n    \\min_{x_1\\in\\calX_1}\\quad f_1(x_0,x_1;\\xi_1)\\ +&\\sup_{p_2\\in\\calP_2}\\bbE_{\\xi_2\\sim p_2} \\min_{x_2\\in\\calX_2}\\Bigg[f_2(x_1,x_2;\\xi_2) + \\\\\n    &+\\sup_{p_3\\in\\calP_3}\\bbE_{\\xi_3\\sim p_3} \\min_{x_3\\in\\calX_3}\\bigg[f_3(x_2,x_3;\\xi_3)+\\cdots\\notag\\\\\n    &+\\sup_{p_T\\in\\calP_{T}}\\bbE_{\\xi_T\\sim p_T} \\min_{x_T\\in\\calX_T}f_T(x_{T-1},x_T;\\xi_T)\\bigg]\\Bigg].\\notag\n\\end{align}\n\nHere, each expectation \\(\\bbE_{\\xi_t\\sim p_t}\\) is taken with respect to some given probability measure \\(p_t\\in\\calP_t\\).\nOur problem~\\eqref{eq:DR-MSCP-Def} relies on the SI assumption in the sense that the ambiguity sets and the expectations are independent across stages.\nConsequently, we can define the (worst-case expected) cost-to-go functions recursively from \\(t=T\\) to \\(t=2\\) as:\n\\begin{equation}\\label{eq:DR-MSCP-Recursion}\n    \\calQ_{t-1}(x_{t-1})\\coloneqq\\sup_{p_t\\in\\calP_t}\\bbE_{\\xi_t\\sim p_t}\\min_{x_t\\in\\calX_t}f_t(x_{t-1},x_{t};\\xi_{t})+\\calQ_{t}(x_{t}),\n\\end{equation}\n\nwith \\(\\calQ_T(x_T)\\equiv0\\) for any \\(x_T\\in\\calX_T\\).\nTo simplify the notation, we also define \n\\begin{equation}\\label{eq:DR-MSCP-ValueFunction}\n    Q_t(x_{t-1};\\xi_t)\\coloneqq\\min_{x_t\\in\\calX_t}f_t(x_{t-1},x_t;\\xi_t)+\\calQ_t(x_t)\n\\end{equation}\n\nas the value function,\nsuch that \\(\\calQ_{t-1}(x_{t-1})=\\sup_{p_t\\in\\calP_t}\\bbE_{\\xi_t\\sim p_t} Q_t(x_{t-1};\\xi_t)\\) for all \\(t\\in\\calT'\\).\nOur algorithmic goal in this paper is to find a near-optimal first stage solution\n\\begin{equation}\n    x_1^*\\in\\argmin_{x_1\\in\\calX_1} f_1(x_0,x_1;\\xi_1)+\\calQ_1(x_1),\n\\end{equation}\n\nas a multistage solution may already take exponentially many operations in \\(T\\) to be written down, e.g., \\(2^{T-1}\\) if we have an MSCO with \\(\\abs{\\Xi_t}=2\\) for \\(t=2,\\dots,T\\).\nWe say that \\(x_1^*\\) is an \\(\\epsilon\\)-optimal first stage solution if \n\\begin{equation}\\label{eq:DR-MSCP-NearOptimalSolution}\n    f_1(x_0,x_1^*;\\xi_1)+Q_1(x_1^*)\\le \\varepsilon+\\Big(\\min_{x_1\\in\\calX_1}f_1(x_0,x_1;\\xi_1)+\\calQ_1(x_1)\\Big).\n\\end{equation}\n\nTo begin with, let us fix algorithmic parameters \\(M_t>0\\) for stages \\(t\\in\\calT'\\) and define alternative value function recursively\n\\begin{equation}\\label{eq:RegularizedValueFunction}\n    Q_t^\\Reg(x_{t-1};\\xi_t)\\coloneqq\\max_{\\nVert{\\lambda_t}_*\\le M_t}\\inf_{\\substack{x_t\\in\\calX_t,\\\\z_t\\in\\bbR^{d_{t-1}}}}f_t(x_{t-1},x_t;\\xi_t)+\\calQ_t^\\Reg(x_t)+\\bangle{\\lambda_t}{x_{t-1}-z_t},\n\\end{equation}\n\nwith \\(\\calQ_T^\\Reg(x_T)\\equiv0\\) on \\(\\calX_T\\) and\n\\begin{equation}\\label{eq:RegularizedCostToGo}\n    \\calQ_{t-1}^\\Reg(x_{t-1})\\coloneqq\\sup_{p_t\\in\\calP_t}\\bbE_{\\xi_t\\sim p_t} Q_t^\\Reg(x_{t-1};\\xi_t)\n\\end{equation}\n\nfor \\(t=T,T-1,\\dots,2\\).\n\n\n\\begin{definition}[Initial stage subproblem oracle]\n    \\label{def:InitialStageOracle}\n    Let \\(\\ulcQ_1,\\olcQ_1:\\calX_1\\to\\bbR\\cup\\{+\\infty\\}\\) denote two lsc, convex functions, with \\(\\ulcQ_1(x_1)\\le\\calQ_1^\\Reg(x_1)\\le\\olcQ_1(x_1)\\) for any \\(x_1\\in\\calX_1\\).\n    Consider the following subproblem for the first stage \\(t=1\\),\n    \\begin{equation}\\label{eq:InitialStageOracle}\n        \\min_{x_1\\in\\calX_1}f_1(x_0,x_1;\\xi_1)+\\ulcQ_1(x_1).\\tag{I}\n    \\end{equation}\n    The initial stage subproblem oracle provides an optimal solution \\(x_1\\) to~\\eqref{eq:InitialStageOracle} and calculates the approximation gap \\(\\gamma_1\\coloneqq\\olcQ_1(x_1)-\\ulcQ_1(x_1)\\).\n    We thus define the subproblem oracle formally as a map \\(\\scrO_1:(\\ulcQ_1,\\olcQ_1)\\mapsto(x_1;\\gamma_1)\\).\n\\end{definition}\n\n\\begin{definition}[Noninitial stage subproblem oracle]\n    \\label{def:NoninitialStageOracle}\n    For any \\(t\\in\\calT'\\), let \\(\\ulcQ_t,\\olcQ_t:\\calX_t\\to\\bbR\\cup\\{+\\infty\\}\\) denote two lsc, convex functions, with \\(\\ulcQ_t(x_t)\\le\\calQ_t^\\Reg(x_t)\\le\\olcQ_t(x_t)\\) for all \\(x_t\\in\\calX_t\\).\n    Then given a feasible state \\(x_{t-1}\\in\\calX_{t-1}\\), the stage-\\(t\\) subproblem oracle provides a feasible state \\(x_t\\in\\calX_t\\), an \\(M_t\\)-Lipschitz continuous linear cut \\(\\calV_{t-1}(\\,\\cdot\\,)\\), and an over-estimate value \\(v_{t-1}\\) such that\n    \\begin{compactitem}\n        \\item they are valid, i.e., \\(\\calV_{t-1}(\\cdot)\\le\\calQ_{t-1}^\\Reg(\\cdot)\\) on \\(\\calX_{t-1}\\) and \\(v_{t-1}\\ge\\calQ_{t-1}^\\Reg(x_{t-1})\\);\n        \\item the gap is controlled, i.e., \\(v_{t-1}-\\calV_{t-1}(x_{t-1})\\le\\gamma_t\\coloneqq\\olcQ_t(x_t)-\\ulcQ_t(x_t)\\).\n    \\end{compactitem}\n    We thus define the subproblem oracle formally as a map \\(\\scrO_t:(x_{t-1},\\ulcQ_t,\\olcQ_t)\\mapsto(\\calV_{t-1},v_{t-1},x_t;\\gamma_t)\\).\n\\end{definition}\n\nTo ease the notation, we use \\(\\conv\\{h_1,h_2\\}\\) to denote the function corresponding to the closed convex hull of the epigraphs of functions \\(h_1\\) and \\(h_2\\).\nMore precisely, we define\n\\[\n    \\conv\\{h_1,h_2\\}(x)\\coloneqq \\sup_{\\lambda}\\inf_{z}\\left\\{\\min\\{h_1(z),h_2(z)\\}+\\bangle{\\lambda}{x-z}\\right\\}.\n\\]\n\n\\begin{algorithm}[ht]\n    \\caption{Consecutive Dual Dynamic Programming Algorithm}\n    \\label{alg:ConsecutiveDualDP}\n    \\begin{algorithmic}[1]\n        \\Require{subproblem oracles \\(\\scrO_t\\) for \\(t\\in\\calT\\), target optimality gap \\(\\epsilon>0\\)} \n        \\Ensure{an \\(\\epsilon\\)-optimal first stage solution \\(x_1^*\\)}\n        \\State{initialize: \\(\\ulcQ_t^0\\leftarrow 0,\\olcQ_t^0\\leftarrow +\\infty,t\\in\\calT\\backslash\\{T\\}\\); \\(\\ulcQ_T^j,\\olcQ_T^j\\leftarrow0,j\\in\\bbN\\); \\(i\\leftarrow 1\\)}\n        \\State{evaluate \\((x_1^1;\\gamma_1^1)\\leftarrow\\scrO_1(\\ulcQ_1^0,\\olcQ_1^0)\\)}\n        \\State{set \\(\\LB\\leftarrow f_1(x_0,x_1^1;\\xi_1),\\ \\UB\\leftarrow +\\infty\\)}\n        \\While{\\(\\UB-\\LB >\\epsilon\\)}\n        \\For{\\(t=2,\\dots,T\\)}\n        \\State{evaluate \\((\\calV_{t-1}^{i},v_{t-1}^{i},x_t^i;\\gamma_t^i)=\\scrO_t(x_{t-1}^i,\\ulcQ_t^{i-1},\\olcQ_t^{i-1})\\)}\n        \\Comment{forward step}\n        \\EndFor\n        \\For{\\(t=T,\\dots,2\\)}\n        \\State{update \\(\\ulcQ_{t-1}^i(x)\\leftarrow \\max\\{\\ulcQ_{t-1}^{i-1}(x),\\calV_{t-1}^i(x)\\}\\)}\n        \\Comment{backward step}\n        \\State{update \\(\\olcQ_{t-1}^i(x)\\leftarrow \\conv\\{\\olcQ_{t-1}^{i-1}(x),v_{t-1}^i+M_{t}\\nVert{x-x_{t-1}^i}\\}\\)}\n        \\EndFor\n        \\State{evaluate \\((x_1^{i+1};\\gamma_1^{i+1})\\leftarrow\\scrO_1(\\ulcQ_1^i,\\olcQ_1^i)\\)}\n        \\Comment{initial stage step}\n        \\State{update \\(\\LB\\leftarrow f_1(x_0,x_1^{i+1};\\xi_1)+\\ulcQ_1^i(x_1^{i+1})\\)}\n        \\State{update \\(\\UB'\\leftarrow f_1(x_0,x_1^{i+1};\\xi_1)+\\olcQ_1^i(x_1^{i+1})\\)}\n        \\If{\\(\\UB'<\\UB\\)}\n        \\State{set \\(x_1^*\\leftarrow x_1^{i+1}\\),  \\(\\UB\\leftarrow\\UB'\\)}\n        \\EndIf\n        \\State{update \\(i\\leftarrow i+1\\)}\n        \\EndWhile\n    \\end{algorithmic}\n\\end{algorithm}\n\n\\begin{algorithm}[ht]\n    \\caption{Nonconsecutive Dual Dynamic Programming (NDDP) Algorithm}\n    \\label{alg:NonconsecutiveDualDP}\n    \\begin{algorithmic}[1]\n        \\Require{subproblem oracles \\(\\scrO_t\\) for \\(t\\in\\calT\\), opt.\\ and approx.\\ gaps \\(\\epsilon=\\delta_1>\\cdots>\\delta_T=0\\)} \n        \\Ensure{an \\(\\epsilon\\)-optimal first stage solution \\(x_1^*\\)}\n        \\State{initialize: \\(\\ulcQ_t^0\\leftarrow 0,\\olcQ_t^0\\leftarrow +\\infty,t\\in\\calT\\backslash\\{T\\}\\); \\(\\ulcQ_T^j,\\olcQ_T^j\\leftarrow0,j\\in\\bbN\\); \\(i_t\\leftarrow 0,t\\in\\calT\\)}\n        \\State{set \\(\\LB\\leftarrow 0,\\ \\UB\\leftarrow +\\infty\\), \\(t\\leftarrow 1\\)}\n        \\Loop\n        \\State{update \\(i_t\\leftarrow i_t+1\\)}\n        \\If{\\(t=1\\)}\n        \\State{evaluate \\((x_1^{i_1};\\gamma_1^{i_1})\\leftarrow\\scrO_1(\\ulcQ_1^{i_1},\\olcQ_1^{i_1})\\)}\n        \\Comment{initial stage step}\n        \\State{update \\(\\LB\\leftarrow f_1(x_0,x_1^{i_1};\\xi_1)+\\ulcQ_1^{i_1}(x_1^{i_1})\\)}\n        \\State{update \\(\\UB'\\leftarrow f_1(x_0,x_1^{i_1};\\xi_1)+\\olcQ_1^{i_1}(x_1^{i_1})\\)}\n        \\If{\\(\\UB'<\\UB\\)}\n        \\State{set \\(x_1^*\\leftarrow x_1^{i_1}\\),  \\(\\UB\\leftarrow\\UB'\\)}\n        \\EndIf\n        \\If{\\(\\UB-\\LB\\le\\epsilon\\)}\n        \\State{\\bfseries{break the loop}}\n        \\EndIf\n        \\State{maintain \\(\\ulcQ_2^{i_2+1}(x)\\leftarrow\\ulcQ_2^{i_2}(x),\\;\\olcQ_2^{i_2+1}(x)\\leftarrow\\olcQ_2^{i_2}(x)\\)}\n        \\State{set \\(t\\leftarrow t+1\\)}\n        \\Else\n        \\State{evaluate \\((\\calV_{t-1}^{i_t},v_{t-1}^{i_t},x_t^{i_t};\\gamma_t^{i_t})\\leftarrow\\scrO_t(x_{t-1}^{i_{t-1}},\\ulcQ_t^{i_t},\\olcQ_t^{i_t})\\)}\n        \\Comment{noninitial stage step}\n        \\If{\\(t<T\\) \\textbf{and} \\(\\gamma_t^{i_t}>\\delta_t\\)}\n        \\State{maintain \\(\\ulcQ_{t+1}^{i_{t+1}+1}(x)\\leftarrow\\ulcQ_{t+1}^{i_{t+1}}(x),\\;\\olcQ_{t+1}^{i_{t+1}+1}(x)\\leftarrow\\olcQ_{t+1}^{i_{t+1}}(x)\\)}\n        \\State{set \\(t\\leftarrow t+1\\)}\n        \\Else\n        \\State{update \\(\\ulcQ_{t-1}^{i_{t-1}+1}(x)\\leftarrow \\max\\{\\ulcQ_{t-1}^{i_{t-1}}(x),\\calV_{t-1}^{i_t}(x)\\}\\)}\n        \\State{update \\(\\olcQ_{t-1}^{i_{t-1}+1}(x)\\leftarrow \\conv\\{\\olcQ_{t-1}^{i_{t-1}}(x),v_{t-1}^{i_t}+M_{t-1}\\Vert{x-x_{t-1}^{i_{t-1}}}\\Vert\\}\\)}\n        \\State{set \\(t\\leftarrow t-1\\)}\n        \\EndIf\n        \\EndIf\n        \\EndLoop\n    \\end{algorithmic}\n\\end{algorithm}\n\nTo begin with, let \\(\\calJ_t,t>1\\), denote the set of pairs of indices \\((i_{t-1},i_t)\\) such that the noninitial stage subproblem oracle is evaluated at the \\(i_t\\)-th time at the state \\(x_{t-1}^{i_{t-1}}\\), i.e., \\((\\calV_{t-1}^{i_t},v_{t-1}^{i_t},x_t^{i_t};\\gamma_t^{i_t})=\\scrO_t(x_{t-1}^{i_{t-1}},\\ulcQ_t^{i_t},\\olcQ_t^{i_t})\\).\nFor the CDDP algorithm, all stages share the same iteration index \\(i_t=i\\), so \\(\\calJ_t=\\{(i,i):i\\in\\bbN\\}\\) for all \\(t>1\\).\nWe define the following sets of indices for each \\(t\\in\\calT\\setminus\\{T\\}\\):\n\\begin{equation}\n    \\calI_t(\\delta)\\coloneqq\\left\\{i_t\\in\\bbN:\\gamma_t^{i_t}>\\delta_t\\text{ and } \\gamma_{t+1}^{i_{t+1}}\\le\\delta_{t+1},\\ (i_t,i_{t+1})\\in\\calJ_{t+1}\\right\\}.\n\\end{equation}\n\nHere, for NDDP algorithm, \\(\\delta\\) is the given approximation gap vector, while for CDDP algorithm, \\(\\delta=(\\delta_t)_{t=1}^T\\) can be any vector satisfying \\(\\epsilon=\\delta_1>\\delta_2>\\cdots>\\delta_T=0\\) for the purpose of analysis, since it is not required for the CDDP algorithm.\nWe adopt the convention that the gap for the last stage \\(\\gamma_T^{i_T}\\equiv0\\) such that \\(i_{T-1}\\in\\calI_{T-1}(\\delta)\\) if and only if \\(\\gamma_{T-1}^{i_{T-1}}>\\delta_{T-1}\\) and \\((i_{T-1},i_T)\\in\\calJ_T\\).\n","units":[{"start":14353,"end":15223,"start_line":247,"end_line":251,"raw_tex":"Let \\(\\calT\\coloneqq\\{1,2,\\dots,T\\}\\) denote the set of stage indices and \\(\\calT'\\coloneqq\\calT\\setminus\\{1\\}\\). \nFor each \\(t\\in\\calT\\), the decision variable in stage \\(t\\) is denoted as \\(x_t\\) and constrained in a compact convex set \\(\\calX_t\\subset\\bbR^{d_t}\\) with dimension \\(d_t\\in\\bbZ_{\\ge0}\\).\nThe uncertainty in stage \\(t\\) is modeled as a random vector \\(\\xi_t\\) with some possibly unknown probability distribution and its convex support set denoted as \\(\\Xi_t\\).\nFor simplicity, we use \\(x_0\\) and \\(\\xi_1\\) to denote deterministic parameters, known as the initial conditions.\nThe cost incurred by the decisions and the uncertainty outcomes in stage \\(t\\) is modeled by a nonnegative lower semicontinuous function \\(f_t(x_{t-1},x_t;\\xi_t)\\), that is convex in \\((x_{t-1},x_t)\\) for each \\(\\xi_t\\in\\Xi_t\\) and allowed to take \\(+\\infty\\) for infeasibility.\n","sha256_utf8":"68b96328380576cdfb282706a6fa8a075328156e7f98241040ee5a5fd9c422c0"},{"start":16312,"end":16894,"start_line":265,"end_line":271,"raw_tex":"Let \\(\\calP_t\\) denote a set of Borel probability measures supported on the uncertainty set \\(\\Xi_t\\), called the ambiguity set, for \\(t\\in\\calT'\\).\nOur DR-MCO is defined as\n\\begin{align}\\label{eq:DR-MSCP-Def}\n    \\min_{x_1\\in\\calX_1}\\quad f_1(x_0,x_1;\\xi_1)\\ +&\\sup_{p_2\\in\\calP_2}\\bbE_{\\xi_2\\sim p_2} \\min_{x_2\\in\\calX_2}\\Bigg[f_2(x_1,x_2;\\xi_2) + \\\\\n    &+\\sup_{p_3\\in\\calP_3}\\bbE_{\\xi_3\\sim p_3} \\min_{x_3\\in\\calX_3}\\bigg[f_3(x_2,x_3;\\xi_3)+\\cdots\\notag\\\\\n    &+\\sup_{p_T\\in\\calP_{T}}\\bbE_{\\xi_T\\sim p_T} \\min_{x_T\\in\\calX_T}f_T(x_{T-1},x_T;\\xi_T)\\bigg]\\Bigg].\\notag\n\\end{align}","sha256_utf8":"9551f0cc58c9a3217f3a64736c906a2e07e5527f0e6ce4e9bbb568b923cfe1b4"},{"start":16895,"end":17482,"start_line":272,"end_line":277,"raw_tex":"Here, each expectation \\(\\bbE_{\\xi_t\\sim p_t}\\) is taken with respect to some given probability measure \\(p_t\\in\\calP_t\\).\nOur problem~\\eqref{eq:DR-MSCP-Def} relies on the SI assumption in the sense that the ambiguity sets and the expectations are independent across stages.\nConsequently, we can define the (worst-case expected) cost-to-go functions recursively from \\(t=T\\) to \\(t=2\\) as:\n\\begin{equation}\\label{eq:DR-MSCP-Recursion}\n    \\calQ_{t-1}(x_{t-1})\\coloneqq\\sup_{p_t\\in\\calP_t}\\bbE_{\\xi_t\\sim p_t}\\min_{x_t\\in\\calX_t}f_t(x_{t-1},x_{t};\\xi_{t})+\\calQ_{t}(x_{t}),\n\\end{equation}","sha256_utf8":"21722e6c895381fcb94e56f213bc104e3e90b517fea02eafb6e4bc62c6178af7"},{"start":17483,"end":17731,"start_line":278,"end_line":282,"raw_tex":"with \\(\\calQ_T(x_T)\\equiv0\\) for any \\(x_T\\in\\calX_T\\).\nTo simplify the notation, we also define \n\\begin{equation}\\label{eq:DR-MSCP-ValueFunction}\n    Q_t(x_{t-1};\\xi_t)\\coloneqq\\min_{x_t\\in\\calX_t}f_t(x_{t-1},x_t;\\xi_t)+\\calQ_t(x_t)\n\\end{equation}","sha256_utf8":"c8975e980003ee0ecdcbf981579cc33d3a6881e76809d39647c47edabceeeeb8"},{"start":17732,"end":18056,"start_line":283,"end_line":288,"raw_tex":"as the value function,\nsuch that \\(\\calQ_{t-1}(x_{t-1})=\\sup_{p_t\\in\\calP_t}\\bbE_{\\xi_t\\sim p_t} Q_t(x_{t-1};\\xi_t)\\) for all \\(t\\in\\calT'\\).\nOur algorithmic goal in this paper is to find a near-optimal first stage solution\n\\begin{equation}\n    x_1^*\\in\\argmin_{x_1\\in\\calX_1} f_1(x_0,x_1;\\xi_1)+\\calQ_1(x_1),\n\\end{equation}","sha256_utf8":"4debc3d933db65ef2a4eef85acae381bbe60edf61b2ff0bba81b38a744a4beb8"},{"start":18057,"end":18496,"start_line":289,"end_line":293,"raw_tex":"as a multistage solution may already take exponentially many operations in \\(T\\) to be written down, e.g., \\(2^{T-1}\\) if we have an MSCO with \\(\\abs{\\Xi_t}=2\\) for \\(t=2,\\dots,T\\).\nWe say that \\(x_1^*\\) is an \\(\\epsilon\\)-optimal first stage solution if \n\\begin{equation}\\label{eq:DR-MSCP-NearOptimalSolution}\n    f_1(x_0,x_1^*;\\xi_1)+Q_1(x_1^*)\\le \\varepsilon+\\Big(\\min_{x_1\\in\\calX_1}f_1(x_0,x_1;\\xi_1)+\\calQ_1(x_1)\\Big).\n\\end{equation}","sha256_utf8":"e1158e8abe2a6d708921f22e7b579ee413b98786a038b08f8e8efa9bd6e061b0"},{"start":32334,"end":32732,"start_line":472,"end_line":475,"raw_tex":"To begin with, let us fix algorithmic parameters \\(M_t>0\\) for stages \\(t\\in\\calT'\\) and define alternative value function recursively\n\\begin{equation}\\label{eq:RegularizedValueFunction}\n    Q_t^\\Reg(x_{t-1};\\xi_t)\\coloneqq\\max_{\\nVert{\\lambda_t}_*\\le M_t}\\inf_{\\substack{x_t\\in\\calX_t,\\\\z_t\\in\\bbR^{d_{t-1}}}}f_t(x_{t-1},x_t;\\xi_t)+\\calQ_t^\\Reg(x_t)+\\bangle{\\lambda_t}{x_{t-1}-z_t},\n\\end{equation}","sha256_utf8":"ed00ccf4aff72439585597ca579ae47d8ee495cca14c9ced76f648d8c4266b24"},{"start":32733,"end":32950,"start_line":476,"end_line":479,"raw_tex":"with \\(\\calQ_T^\\Reg(x_T)\\equiv0\\) on \\(\\calX_T\\) and\n\\begin{equation}\\label{eq:RegularizedCostToGo}\n    \\calQ_{t-1}^\\Reg(x_{t-1})\\coloneqq\\sup_{p_t\\in\\calP_t}\\bbE_{\\xi_t\\sim p_t} Q_t^\\Reg(x_{t-1};\\xi_t)\n\\end{equation}","sha256_utf8":"ce2d48cd3b6f74b52e97476f2b493f61e06effe1b1cc3818101099614e840524"},{"start":32951,"end":32976,"start_line":480,"end_line":480,"raw_tex":"for \\(t=T,T-1,\\dots,2\\).\n","sha256_utf8":"f6655ec6887a18cae0cfa9a5207e472ddef92bd27b1f992002338d8726495bd3"},{"start":41403,"end":42192,"start_line":571,"end_line":580,"raw_tex":"\\begin{definition}[Initial stage subproblem oracle]\n    \\label{def:InitialStageOracle}\n    Let \\(\\ulcQ_1,\\olcQ_1:\\calX_1\\to\\bbR\\cup\\{+\\infty\\}\\) denote two lsc, convex functions, with \\(\\ulcQ_1(x_1)\\le\\calQ_1^\\Reg(x_1)\\le\\olcQ_1(x_1)\\) for any \\(x_1\\in\\calX_1\\).\n    Consider the following subproblem for the first stage \\(t=1\\),\n    \\begin{equation}\\label{eq:InitialStageOracle}\n        \\min_{x_1\\in\\calX_1}f_1(x_0,x_1;\\xi_1)+\\ulcQ_1(x_1).\\tag{I}\n    \\end{equation}\n    The initial stage subproblem oracle provides an optimal solution \\(x_1\\) to~\\eqref{eq:InitialStageOracle} and calculates the approximation gap \\(\\gamma_1\\coloneqq\\olcQ_1(x_1)-\\ulcQ_1(x_1)\\).\n    We thus define the subproblem oracle formally as a map \\(\\scrO_1:(\\ulcQ_1,\\olcQ_1)\\mapsto(x_1;\\gamma_1)\\).\n\\end{definition}","sha256_utf8":"3cb8a92a37ef0b2aa4f47dfe60af48672394b0843b804777a4288ca63e692948"},{"start":42194,"end":43217,"start_line":582,"end_line":591,"raw_tex":"\\begin{definition}[Noninitial stage subproblem oracle]\n    \\label{def:NoninitialStageOracle}\n    For any \\(t\\in\\calT'\\), let \\(\\ulcQ_t,\\olcQ_t:\\calX_t\\to\\bbR\\cup\\{+\\infty\\}\\) denote two lsc, convex functions, with \\(\\ulcQ_t(x_t)\\le\\calQ_t^\\Reg(x_t)\\le\\olcQ_t(x_t)\\) for all \\(x_t\\in\\calX_t\\).\n    Then given a feasible state \\(x_{t-1}\\in\\calX_{t-1}\\), the stage-\\(t\\) subproblem oracle provides a feasible state \\(x_t\\in\\calX_t\\), an \\(M_t\\)-Lipschitz continuous linear cut \\(\\calV_{t-1}(\\,\\cdot\\,)\\), and an over-estimate value \\(v_{t-1}\\) such that\n    \\begin{compactitem}\n        \\item they are valid, i.e., \\(\\calV_{t-1}(\\cdot)\\le\\calQ_{t-1}^\\Reg(\\cdot)\\) on \\(\\calX_{t-1}\\) and \\(v_{t-1}\\ge\\calQ_{t-1}^\\Reg(x_{t-1})\\);\n        \\item the gap is controlled, i.e., \\(v_{t-1}-\\calV_{t-1}(x_{t-1})\\le\\gamma_t\\coloneqq\\olcQ_t(x_t)-\\ulcQ_t(x_t)\\).\n    \\end{compactitem}\n    We thus define the subproblem oracle formally as a map \\(\\scrO_t:(x_{t-1},\\ulcQ_t,\\olcQ_t)\\mapsto(\\calV_{t-1},v_{t-1},x_t;\\gamma_t)\\).\n\\end{definition}","sha256_utf8":"e7f91629b893bf59df05aeceded2ee42c9046b46a8475eecb01e0a920dcd71a5"},{"start":50226,"end":50536,"start_line":661,"end_line":665,"raw_tex":"To ease the notation, we use \\(\\conv\\{h_1,h_2\\}\\) to denote the function corresponding to the closed convex hull of the epigraphs of functions \\(h_1\\) and \\(h_2\\).\nMore precisely, we define\n\\[\n    \\conv\\{h_1,h_2\\}(x)\\coloneqq \\sup_{\\lambda}\\inf_{z}\\left\\{\\min\\{h_1(z),h_2(z)\\}+\\bangle{\\lambda}{x-z}\\right\\}.\n\\]","sha256_utf8":"a701f68ba201e2755574227d22deeb6c589e8c15d24bf4f8cf86e239458202b9"},{"start":50990,"end":52706,"start_line":677,"end_line":706,"raw_tex":"\\begin{algorithm}[ht]\n    \\caption{Consecutive Dual Dynamic Programming Algorithm}\n    \\label{alg:ConsecutiveDualDP}\n    \\begin{algorithmic}[1]\n        \\Require{subproblem oracles \\(\\scrO_t\\) for \\(t\\in\\calT\\), target optimality gap \\(\\epsilon>0\\)} \n        \\Ensure{an \\(\\epsilon\\)-optimal first stage solution \\(x_1^*\\)}\n        \\State{initialize: \\(\\ulcQ_t^0\\leftarrow 0,\\olcQ_t^0\\leftarrow +\\infty,t\\in\\calT\\backslash\\{T\\}\\); \\(\\ulcQ_T^j,\\olcQ_T^j\\leftarrow0,j\\in\\bbN\\); \\(i\\leftarrow 1\\)}\n        \\State{evaluate \\((x_1^1;\\gamma_1^1)\\leftarrow\\scrO_1(\\ulcQ_1^0,\\olcQ_1^0)\\)}\n        \\State{set \\(\\LB\\leftarrow f_1(x_0,x_1^1;\\xi_1),\\ \\UB\\leftarrow +\\infty\\)}\n        \\While{\\(\\UB-\\LB >\\epsilon\\)}\n        \\For{\\(t=2,\\dots,T\\)}\n        \\State{evaluate \\((\\calV_{t-1}^{i},v_{t-1}^{i},x_t^i;\\gamma_t^i)=\\scrO_t(x_{t-1}^i,\\ulcQ_t^{i-1},\\olcQ_t^{i-1})\\)}\n        \\Comment{forward step}\n        \\EndFor\n        \\For{\\(t=T,\\dots,2\\)}\n        \\State{update \\(\\ulcQ_{t-1}^i(x)\\leftarrow \\max\\{\\ulcQ_{t-1}^{i-1}(x),\\calV_{t-1}^i(x)\\}\\)}\n        \\Comment{backward step}\n        \\State{update \\(\\olcQ_{t-1}^i(x)\\leftarrow \\conv\\{\\olcQ_{t-1}^{i-1}(x),v_{t-1}^i+M_{t}\\nVert{x-x_{t-1}^i}\\}\\)}\n        \\EndFor\n        \\State{evaluate \\((x_1^{i+1};\\gamma_1^{i+1})\\leftarrow\\scrO_1(\\ulcQ_1^i,\\olcQ_1^i)\\)}\n        \\Comment{initial stage step}\n        \\State{update \\(\\LB\\leftarrow f_1(x_0,x_1^{i+1};\\xi_1)+\\ulcQ_1^i(x_1^{i+1})\\)}\n        \\State{update \\(\\UB'\\leftarrow f_1(x_0,x_1^{i+1};\\xi_1)+\\olcQ_1^i(x_1^{i+1})\\)}\n        \\If{\\(\\UB'<\\UB\\)}\n        \\State{set \\(x_1^*\\leftarrow x_1^{i+1}\\),  \\(\\UB\\leftarrow\\UB'\\)}\n        \\EndIf\n        \\State{update \\(i\\leftarrow i+1\\)}\n        \\EndWhile\n    \\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"ec76286ea3d78b3761ff6544f7ba7801f31b5b282caec4e382278317bc75c5a1"},{"start":55303,"end":57572,"start_line":729,"end_line":766,"raw_tex":"\\begin{algorithm}[ht]\n    \\caption{Nonconsecutive Dual Dynamic Programming (NDDP) Algorithm}\n    \\label{alg:NonconsecutiveDualDP}\n    \\begin{algorithmic}[1]\n        \\Require{subproblem oracles \\(\\scrO_t\\) for \\(t\\in\\calT\\), opt.\\ and approx.\\ gaps \\(\\epsilon=\\delta_1>\\cdots>\\delta_T=0\\)} \n        \\Ensure{an \\(\\epsilon\\)-optimal first stage solution \\(x_1^*\\)}\n        \\State{initialize: \\(\\ulcQ_t^0\\leftarrow 0,\\olcQ_t^0\\leftarrow +\\infty,t\\in\\calT\\backslash\\{T\\}\\); \\(\\ulcQ_T^j,\\olcQ_T^j\\leftarrow0,j\\in\\bbN\\); \\(i_t\\leftarrow 0,t\\in\\calT\\)}\n        \\State{set \\(\\LB\\leftarrow 0,\\ \\UB\\leftarrow +\\infty\\), \\(t\\leftarrow 1\\)}\n        \\Loop\n        \\State{update \\(i_t\\leftarrow i_t+1\\)}\n        \\If{\\(t=1\\)}\n        \\State{evaluate \\((x_1^{i_1};\\gamma_1^{i_1})\\leftarrow\\scrO_1(\\ulcQ_1^{i_1},\\olcQ_1^{i_1})\\)}\n        \\Comment{initial stage step}\n        \\State{update \\(\\LB\\leftarrow f_1(x_0,x_1^{i_1};\\xi_1)+\\ulcQ_1^{i_1}(x_1^{i_1})\\)}\n        \\State{update \\(\\UB'\\leftarrow f_1(x_0,x_1^{i_1};\\xi_1)+\\olcQ_1^{i_1}(x_1^{i_1})\\)}\n        \\If{\\(\\UB'<\\UB\\)}\n        \\State{set \\(x_1^*\\leftarrow x_1^{i_1}\\),  \\(\\UB\\leftarrow\\UB'\\)}\n        \\EndIf\n        \\If{\\(\\UB-\\LB\\le\\epsilon\\)}\n        \\State{\\bfseries{break the loop}}\n        \\EndIf\n        \\State{maintain \\(\\ulcQ_2^{i_2+1}(x)\\leftarrow\\ulcQ_2^{i_2}(x),\\;\\olcQ_2^{i_2+1}(x)\\leftarrow\\olcQ_2^{i_2}(x)\\)}\n        \\State{set \\(t\\leftarrow t+1\\)}\n        \\Else\n        \\State{evaluate \\((\\calV_{t-1}^{i_t},v_{t-1}^{i_t},x_t^{i_t};\\gamma_t^{i_t})\\leftarrow\\scrO_t(x_{t-1}^{i_{t-1}},\\ulcQ_t^{i_t},\\olcQ_t^{i_t})\\)}\n        \\Comment{noninitial stage step}\n        \\If{\\(t<T\\) \\textbf{and} \\(\\gamma_t^{i_t}>\\delta_t\\)}\n        \\State{maintain \\(\\ulcQ_{t+1}^{i_{t+1}+1}(x)\\leftarrow\\ulcQ_{t+1}^{i_{t+1}}(x),\\;\\olcQ_{t+1}^{i_{t+1}+1}(x)\\leftarrow\\olcQ_{t+1}^{i_{t+1}}(x)\\)}\n        \\State{set \\(t\\leftarrow t+1\\)}\n        \\Else\n        \\State{update \\(\\ulcQ_{t-1}^{i_{t-1}+1}(x)\\leftarrow \\max\\{\\ulcQ_{t-1}^{i_{t-1}}(x),\\calV_{t-1}^{i_t}(x)\\}\\)}\n        \\State{update \\(\\olcQ_{t-1}^{i_{t-1}+1}(x)\\leftarrow \\conv\\{\\olcQ_{t-1}^{i_{t-1}}(x),v_{t-1}^{i_t}+M_{t-1}\\Vert{x-x_{t-1}^{i_{t-1}}}\\Vert\\}\\)}\n        \\State{set \\(t\\leftarrow t-1\\)}\n        \\EndIf\n        \\EndIf\n        \\EndLoop\n    \\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"16b0fca40a420db9c52d61403690ba852826da3e909613c05b6e36c8c6ba3642"},{"start":59053,"end":59784,"start_line":783,"end_line":788,"raw_tex":"To begin with, let \\(\\calJ_t,t>1\\), denote the set of pairs of indices \\((i_{t-1},i_t)\\) such that the noninitial stage subproblem oracle is evaluated at the \\(i_t\\)-th time at the state \\(x_{t-1}^{i_{t-1}}\\), i.e., \\((\\calV_{t-1}^{i_t},v_{t-1}^{i_t},x_t^{i_t};\\gamma_t^{i_t})=\\scrO_t(x_{t-1}^{i_{t-1}},\\ulcQ_t^{i_t},\\olcQ_t^{i_t})\\).\nFor the CDDP algorithm, all stages share the same iteration index \\(i_t=i\\), so \\(\\calJ_t=\\{(i,i):i\\in\\bbN\\}\\) for all \\(t>1\\).\nWe define the following sets of indices for each \\(t\\in\\calT\\setminus\\{T\\}\\):\n\\begin{equation}\n    \\calI_t(\\delta)\\coloneqq\\left\\{i_t\\in\\bbN:\\gamma_t^{i_t}>\\delta_t\\text{ and } \\gamma_{t+1}^{i_{t+1}}\\le\\delta_{t+1},\\ (i_t,i_{t+1})\\in\\calJ_{t+1}\\right\\}.\n\\end{equation}","sha256_utf8":"ea2ed4643714878cd81c11c0c54b4cb1ba36cfdfda763b4175835dfcac2a5b51"},{"start":59785,"end":60287,"start_line":789,"end_line":790,"raw_tex":"Here, for NDDP algorithm, \\(\\delta\\) is the given approximation gap vector, while for CDDP algorithm, \\(\\delta=(\\delta_t)_{t=1}^T\\) can be any vector satisfying \\(\\epsilon=\\delta_1>\\delta_2>\\cdots>\\delta_T=0\\) for the purpose of analysis, since it is not required for the CDDP algorithm.\nWe adopt the convention that the gap for the last stage \\(\\gamma_T^{i_T}\\equiv0\\) such that \\(i_{T-1}\\in\\calI_{T-1}(\\delta)\\) if and only if \\(\\gamma_{T-1}^{i_{T-1}}>\\delta_{T-1}\\) and \\((i_{T-1},i_T)\\in\\calJ_T\\).\n","sha256_utf8":"2a7cb2d7e26008014eaffbd58a6cdf5e6a70119d5799230d238b4bffe93f4eca"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{thm:NDDPComplexityBound}\n    Suppose the state spaces \\(\\calX_t\\subset\\bbR^{d_t}\\) are contained in balls, each with diameter \\(D_t>0\\).\n    Then, for the NDDP algorithm (Algorithm~\\ref{alg:NonconsecutiveDualDP}) with the predetermined approximation gap vector \\((\\delta_t)_{t=1}^{T}\\) satisfying \\(\\epsilon=\\delta_1>\\delta_2>\\cdots>\\delta_T=0\\), \n    the total number of subproblem oracle evaluations \\(\\NumEval_\\NDDP\\) before termination is bounded by\n    \\begin{equation*}\n        \\NumEval_\\NDDP\\le 1+2\\cdot\\sum_{t=1}^{T-1}\\left(1+\\frac{2M_{t+1}D_t}{\\delta_t-\\delta_{t+1}}\\right)^{d_t}.\n    \\end{equation*}\n\\end{theorem}","units":[{"start":67023,"end":67668,"start_line":867,"end_line":874,"raw_tex":"\\begin{theorem}\\label{thm:NDDPComplexityBound}\n    Suppose the state spaces \\(\\calX_t\\subset\\bbR^{d_t}\\) are contained in balls, each with diameter \\(D_t>0\\).\n    Then, for the NDDP algorithm (Algorithm~\\ref{alg:NonconsecutiveDualDP}) with the predetermined approximation gap vector \\((\\delta_t)_{t=1}^{T}\\) satisfying \\(\\epsilon=\\delta_1>\\delta_2>\\cdots>\\delta_T=0\\), \n    the total number of subproblem oracle evaluations \\(\\NumEval_\\NDDP\\) before termination is bounded by\n    \\begin{equation*}\n        \\NumEval_\\NDDP\\le 1+2\\cdot\\sum_{t=1}^{T-1}\\left(1+\\frac{2M_{t+1}D_t}{\\delta_t-\\delta_{t+1}}\\right)^{d_t}.\n    \\end{equation*}\n\\end{theorem}","sha256_utf8":"fdb162343a6420b649090c6d381b782e4a46fb510ee4fa5ded71b6680387101a"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\\label{lemma:FiniteIndexSet}\n    For stage any \\(t<T\\), suppose the state space \\(\\calX_t\\subset\\bbR^{d_t}\\) is contained in a ball with diameter \\(D_t>0\\).\n    Then,\n    \\begin{equation}\\label{eq:FiniteIndexSet}\n        \\abs{\\calI_t(\\delta)}\\le\\left(1+\\frac{2M_{t+1} D_t}{\\delta_t-\\delta_{t+1}}\\right)^{d_t}.\n    \\end{equation}\n\\end{lemma}\n\n\\begin{proof}\n    We claim that for any \\(j,k\\in\\calI_t\\), \\(j\\neq k\\), it holds that \\(\\nVert{x_t^j-x_t^k}>(\\delta_t-\\delta_{t+1})/(2M_{t+1})\\).\n    Assume for contradiction that \\(\\nVert{x_t^j-x_t^k}\\le(\\delta_t-\\delta_{t+1})/(2M_{t+1})\\) for some \\(j<k\\), \\(j,k\\in\\calI_t(\\delta)\\).\n    By definition of \\(\\calI_t(\\delta)\\), the \\((t+1)\\)-th subproblem oracle is evaluated at the state \\(x_t^j\\), and in both the CDDP and the NDDP algorithms, the approximations \\(\\ulcQ_t^j\\) and \\(\\olcQ_t^j\\) are updated since \\(\\gamma_{t+1}^{i_{t+1}}\\le\\delta_{t+1}\\) for some \\(i_{t+1}\\in\\bbN\\) with \\((j,i_{t+1})\\in\\calJ_{t+1}\\).\n    Then by Definition~\\ref{def:NoninitialStageOracle} of the noninitial stage subproblem oracle, we have \\(\\olcQ_t^{j+1}(x_t^j)-\\ulcQ_t^{j+1}(x_t^j)\\le \\delta_{t+1}\\).\n    Note that for any point \\(x\\in\\calX_t\\) with \\(\\nVert{x-x_t^j}\\le(\\delta_t-\\delta_{t+1})/(2M_{t+1})\\), we have \\(\\olcQ_t^{j+1}(x)-\\ulcQ_t^{j+1}(x)\\le \\delta_{t}\\) because of the \\(M_{t+1}\\)-Lipschitz continuity of the approximations.\n    Since \\(\\olcQ_t^{k}(x)\\le\\olcQ_t^{j+1}(x)\\) and \\(\\ulcQ_t^{k}(x)\\ge\\ulcQ_t^{j+1}(x)\\), by setting \\(x=x_t^k\\), we see a contradiction with the assumption that \\(k\\in\\calI_t(\\delta)\\), which proves the claim.\n\n    To ease the notation, let \\(r_t\\coloneqq(\\delta_t-\\delta_{t+1})/(2M_{t+1})\\) denote the radius of the \\(d_t\\)-dimensional balls \\(\\calB^{d_t}(x_t^j;r_t)\\) centered at \\(x_t^j\\) for \\(j\\in\\calI_t(\\delta)\\), and let \\(\\calB_t\\supseteq\\calX_t\\) denote a ball with diameter \\(D_t\\).\n    From the above claim, we know that \\(x_t^k\\notin\\calB^{d_t}(x_t^j;r_t)\\) for any \\(j,k\\in\\calI_t(\\delta)\\) with \\(j<k\\).\n    In other words, the smaller balls \\(\\calB^{d_t}(x_t^j;r_t/2)\\) are disjoint.\n    Meanwhile, note that each of these smaller balls satisfies \\(\\calB^{d_t}(x_t^j;r_t/2)\\subset\\calB_t+\\calB^{d_t}(0;r_t/2)\\) (the Minkowski sum in the Euclidean space \\(\\bbR^{d_t}\\)).\n    Therefore, the volumes satisfy the relation\n    \\begin{equation*}\n     \\Vol\\left(\\textstyle\\bigcup_{j\\in\\calI_t(\\delta)}\\calB^{d_t}(x_t^j;r_t/2)\\right)\n        =\\abs{\\calI_t(\\delta)}\\cdot\\Vol\\calB^{d_t}(0;r_t/2)\n        \\le\\Vol\\left(\\calB_t+\\calB^{d_t}(0;r_t/2)\\right),\n    \\end{equation*}\n    which implies that\n    \\begin{equation*}\n        \\abs{\\calI_t(\\delta)}\\le\\frac{\\Vol(\\calB_t+\\calB^{d_t}(0;r_t/2))}{\\Vol\\calB^{d_t}(0;r_t/2)}\n        =\\left(\\frac{D_t/2+r_t/2}{r_t/2}\\right)^{d_t}\n        =\\left(1+\\frac{2M_{t+1}D_t}{\\delta_t-\\delta_{t+1}}\\right)^{d_t}.\n    \\end{equation*}\n\\end{proof}\n\n\\begin{proof}\n    For the NDDP algorithm, each time when it decides to go back to the precedent stage \\(t\\leftarrow t-1\\), we must have \\(\\gamma_t^{i_t}\\le\\delta_t\\) while \\(\\gamma_{t-1}^{i_{t-1}}>\\delta_{t-1}\\) for some \\((i_{t-1},i_t)\\in\\calJ_t\\).\n    In this case, we have by definition that \\(i_{t-1}\\in\\calI_{t-1}(\\delta)\\).\n    By Lemma~\\ref{lemma:FiniteIndexSet}, such ``going back'' steps can only happen at most \\(\\sum_{t=1}^{T-1}\\abs{\\calI_t(\\delta)}\\) times.\n    The total number of evaluations is then bounded by two times the number of ``going back'' steps due to the corresponding ``going forward'' steps, plus an additional evaluation of the initial stage oracle for checking the termination criterion.\n\\end{proof}","units":[{"start":60990,"end":61343,"start_line":794,"end_line":800,"raw_tex":"\\begin{lemma}\\label{lemma:FiniteIndexSet}\n    For stage any \\(t<T\\), suppose the state space \\(\\calX_t\\subset\\bbR^{d_t}\\) is contained in a ball with diameter \\(D_t>0\\).\n    Then,\n    \\begin{equation}\\label{eq:FiniteIndexSet}\n        \\abs{\\calI_t(\\delta)}\\le\\left(1+\\frac{2M_{t+1} D_t}{\\delta_t-\\delta_{t+1}}\\right)^{d_t}.\n    \\end{equation}\n\\end{lemma}","sha256_utf8":"4292406ac4093927c70c2644cdda15c4308b887b3799f21c91c6a5f2bfb5ff93"},{"start":61344,"end":63857,"start_line":801,"end_line":825,"raw_tex":"\\begin{proof}\n    We claim that for any \\(j,k\\in\\calI_t\\), \\(j\\neq k\\), it holds that \\(\\nVert{x_t^j-x_t^k}>(\\delta_t-\\delta_{t+1})/(2M_{t+1})\\).\n    Assume for contradiction that \\(\\nVert{x_t^j-x_t^k}\\le(\\delta_t-\\delta_{t+1})/(2M_{t+1})\\) for some \\(j<k\\), \\(j,k\\in\\calI_t(\\delta)\\).\n    By definition of \\(\\calI_t(\\delta)\\), the \\((t+1)\\)-th subproblem oracle is evaluated at the state \\(x_t^j\\), and in both the CDDP and the NDDP algorithms, the approximations \\(\\ulcQ_t^j\\) and \\(\\olcQ_t^j\\) are updated since \\(\\gamma_{t+1}^{i_{t+1}}\\le\\delta_{t+1}\\) for some \\(i_{t+1}\\in\\bbN\\) with \\((j,i_{t+1})\\in\\calJ_{t+1}\\).\n    Then by Definition~\\ref{def:NoninitialStageOracle} of the noninitial stage subproblem oracle, we have \\(\\olcQ_t^{j+1}(x_t^j)-\\ulcQ_t^{j+1}(x_t^j)\\le \\delta_{t+1}\\).\n    Note that for any point \\(x\\in\\calX_t\\) with \\(\\nVert{x-x_t^j}\\le(\\delta_t-\\delta_{t+1})/(2M_{t+1})\\), we have \\(\\olcQ_t^{j+1}(x)-\\ulcQ_t^{j+1}(x)\\le \\delta_{t}\\) because of the \\(M_{t+1}\\)-Lipschitz continuity of the approximations.\n    Since \\(\\olcQ_t^{k}(x)\\le\\olcQ_t^{j+1}(x)\\) and \\(\\ulcQ_t^{k}(x)\\ge\\ulcQ_t^{j+1}(x)\\), by setting \\(x=x_t^k\\), we see a contradiction with the assumption that \\(k\\in\\calI_t(\\delta)\\), which proves the claim.\n\n    To ease the notation, let \\(r_t\\coloneqq(\\delta_t-\\delta_{t+1})/(2M_{t+1})\\) denote the radius of the \\(d_t\\)-dimensional balls \\(\\calB^{d_t}(x_t^j;r_t)\\) centered at \\(x_t^j\\) for \\(j\\in\\calI_t(\\delta)\\), and let \\(\\calB_t\\supseteq\\calX_t\\) denote a ball with diameter \\(D_t\\).\n    From the above claim, we know that \\(x_t^k\\notin\\calB^{d_t}(x_t^j;r_t)\\) for any \\(j,k\\in\\calI_t(\\delta)\\) with \\(j<k\\).\n    In other words, the smaller balls \\(\\calB^{d_t}(x_t^j;r_t/2)\\) are disjoint.\n    Meanwhile, note that each of these smaller balls satisfies \\(\\calB^{d_t}(x_t^j;r_t/2)\\subset\\calB_t+\\calB^{d_t}(0;r_t/2)\\) (the Minkowski sum in the Euclidean space \\(\\bbR^{d_t}\\)).\n    Therefore, the volumes satisfy the relation\n    \\begin{equation*}\n     \\Vol\\left(\\textstyle\\bigcup_{j\\in\\calI_t(\\delta)}\\calB^{d_t}(x_t^j;r_t/2)\\right)\n        =\\abs{\\calI_t(\\delta)}\\cdot\\Vol\\calB^{d_t}(0;r_t/2)\n        \\le\\Vol\\left(\\calB_t+\\calB^{d_t}(0;r_t/2)\\right),\n    \\end{equation*}\n    which implies that\n    \\begin{equation*}\n        \\abs{\\calI_t(\\delta)}\\le\\frac{\\Vol(\\calB_t+\\calB^{d_t}(0;r_t/2))}{\\Vol\\calB^{d_t}(0;r_t/2)}\n        =\\left(\\frac{D_t/2+r_t/2}{r_t/2}\\right)^{d_t}\n        =\\left(1+\\frac{2M_{t+1}D_t}{\\delta_t-\\delta_{t+1}}\\right)^{d_t}.\n    \\end{equation*}\n\\end{proof}","sha256_utf8":"06cd4c3ba4fab57918ceca49f528508a3eaaa0f4a66021cd04d13b54c476f977"},{"start":67669,"end":68398,"start_line":875,"end_line":880,"raw_tex":"\\begin{proof}\n    For the NDDP algorithm, each time when it decides to go back to the precedent stage \\(t\\leftarrow t-1\\), we must have \\(\\gamma_t^{i_t}\\le\\delta_t\\) while \\(\\gamma_{t-1}^{i_{t-1}}>\\delta_{t-1}\\) for some \\((i_{t-1},i_t)\\in\\calJ_t\\).\n    In this case, we have by definition that \\(i_{t-1}\\in\\calI_{t-1}(\\delta)\\).\n    By Lemma~\\ref{lemma:FiniteIndexSet}, such ``going back'' steps can only happen at most \\(\\sum_{t=1}^{T-1}\\abs{\\calI_t(\\delta)}\\) times.\n    The total number of evaluations is then bounded by two times the number of ``going back'' steps due to the corresponding ``going forward'' steps, plus an additional evaluation of the initial stage oracle for checking the termination criterion.\n\\end{proof}","sha256_utf8":"d5079d15cf88278667177062f3b06bc1a1ee6af9be0f774aac9c8973bd1ce250"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"阶段、状态与成本条件","source_units":[{"start":14353,"end":15223,"start_line":247,"end_line":251,"raw_tex":"Let \\(\\calT\\coloneqq\\{1,2,\\dots,T\\}\\) denote the set of stage indices and \\(\\calT'\\coloneqq\\calT\\setminus\\{1\\}\\). \nFor each \\(t\\in\\calT\\), the decision variable in stage \\(t\\) is denoted as \\(x_t\\) and constrained in a compact convex set \\(\\calX_t\\subset\\bbR^{d_t}\\) with dimension \\(d_t\\in\\bbZ_{\\ge0}\\).\nThe uncertainty in stage \\(t\\) is modeled as a random vector \\(\\xi_t\\) with some possibly unknown probability distribution and its convex support set denoted as \\(\\Xi_t\\).\nFor simplicity, we use \\(x_0\\) and \\(\\xi_1\\) to denote deterministic parameters, known as the initial conditions.\nThe cost incurred by the decisions and the uncertainty outcomes in stage \\(t\\) is modeled by a nonnegative lower semicontinuous function \\(f_t(x_{t-1},x_t;\\xi_t)\\), that is convex in \\((x_{t-1},x_t)\\) for each \\(\\xi_t\\in\\Xi_t\\) and allowed to take \\(+\\infty\\) for infeasibility.\n","sha256_utf8":"68b96328380576cdfb282706a6fa8a075328156e7f98241040ee5a5fd9c422c0"}]},{"id":"S002","semantic_type":"definition","kind":"definition","title":"分布模糊集","source_units":[{"start":16312,"end":16486,"start_line":265,"end_line":266,"raw_tex":"Let \\(\\calP_t\\) denote a set of Borel probability measures supported on the uncertainty set \\(\\Xi_t\\), called the ambiguity set, for \\(t\\in\\calT'\\).\nOur DR-MCO is defined as\n","sha256_utf8":"a34c874041f85deb00b36a2b85124128d4776207582eea4548d4543cf25d2351"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"阶段独立的鲁棒模型","source_units":[{"start":16486,"end":16894,"start_line":267,"end_line":271,"raw_tex":"\\begin{align}\\label{eq:DR-MSCP-Def}\n    \\min_{x_1\\in\\calX_1}\\quad f_1(x_0,x_1;\\xi_1)\\ +&\\sup_{p_2\\in\\calP_2}\\bbE_{\\xi_2\\sim p_2} \\min_{x_2\\in\\calX_2}\\Bigg[f_2(x_1,x_2;\\xi_2) + \\\\\n    &+\\sup_{p_3\\in\\calP_3}\\bbE_{\\xi_3\\sim p_3} \\min_{x_3\\in\\calX_3}\\bigg[f_3(x_2,x_3;\\xi_3)+\\cdots\\notag\\\\\n    &+\\sup_{p_T\\in\\calP_{T}}\\bbE_{\\xi_T\\sim p_T} \\min_{x_T\\in\\calX_T}f_T(x_{T-1},x_T;\\xi_T)\\bigg]\\Bigg].\\notag\n\\end{align}","sha256_utf8":"e88652d76e004d654997491022a0ef3583595116b67bf71dd777e8233a64dcb9"},{"start":16895,"end":17170,"start_line":272,"end_line":273,"raw_tex":"Here, each expectation \\(\\bbE_{\\xi_t\\sim p_t}\\) is taken with respect to some given probability measure \\(p_t\\in\\calP_t\\).\nOur problem~\\eqref{eq:DR-MSCP-Def} relies on the SI assumption in the sense that the ambiguity sets and the expectations are independent across stages.\n","sha256_utf8":"b66d7b582ce446e7e41231722197368e867bdd8e9fc76da8cfc5e8506fb0e1c6"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"值函数与成本递推","source_units":[{"start":17170,"end":17482,"start_line":274,"end_line":277,"raw_tex":"Consequently, we can define the (worst-case expected) cost-to-go functions recursively from \\(t=T\\) to \\(t=2\\) as:\n\\begin{equation}\\label{eq:DR-MSCP-Recursion}\n    \\calQ_{t-1}(x_{t-1})\\coloneqq\\sup_{p_t\\in\\calP_t}\\bbE_{\\xi_t\\sim p_t}\\min_{x_t\\in\\calX_t}f_t(x_{t-1},x_{t};\\xi_{t})+\\calQ_{t}(x_{t}),\n\\end{equation}","sha256_utf8":"b750e593ba7e723436afc585b582cc5a4678e54fd85ded3630355a1e0576cabc"},{"start":17483,"end":17731,"start_line":278,"end_line":282,"raw_tex":"with \\(\\calQ_T(x_T)\\equiv0\\) for any \\(x_T\\in\\calX_T\\).\nTo simplify the notation, we also define \n\\begin{equation}\\label{eq:DR-MSCP-ValueFunction}\n    Q_t(x_{t-1};\\xi_t)\\coloneqq\\min_{x_t\\in\\calX_t}f_t(x_{t-1},x_t;\\xi_t)+\\calQ_t(x_t)\n\\end{equation}","sha256_utf8":"c8975e980003ee0ecdcbf981579cc33d3a6881e76809d39647c47edabceeeeb8"},{"start":17732,"end":17874,"start_line":283,"end_line":284,"raw_tex":"as the value function,\nsuch that \\(\\calQ_{t-1}(x_{t-1})=\\sup_{p_t\\in\\calP_t}\\bbE_{\\xi_t\\sim p_t} Q_t(x_{t-1};\\xi_t)\\) for all \\(t\\in\\calT'\\).\n","sha256_utf8":"97ea847a37f4ac0ce1b540e8303ed2e8af2abde2511d29b764d3adbc80adc880"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"首阶段近似最优解","source_units":[{"start":17874,"end":18056,"start_line":285,"end_line":288,"raw_tex":"Our algorithmic goal in this paper is to find a near-optimal first stage solution\n\\begin{equation}\n    x_1^*\\in\\argmin_{x_1\\in\\calX_1} f_1(x_0,x_1;\\xi_1)+\\calQ_1(x_1),\n\\end{equation}","sha256_utf8":"0be05017f991bbe4e9a8d471babc8c42f4db56d333b204e27b3bd5b97261a188"},{"start":18057,"end":18496,"start_line":289,"end_line":293,"raw_tex":"as a multistage solution may already take exponentially many operations in \\(T\\) to be written down, e.g., \\(2^{T-1}\\) if we have an MSCO with \\(\\abs{\\Xi_t}=2\\) for \\(t=2,\\dots,T\\).\nWe say that \\(x_1^*\\) is an \\(\\epsilon\\)-optimal first stage solution if \n\\begin{equation}\\label{eq:DR-MSCP-NearOptimalSolution}\n    f_1(x_0,x_1^*;\\xi_1)+Q_1(x_1^*)\\le \\varepsilon+\\Big(\\min_{x_1\\in\\calX_1}f_1(x_0,x_1;\\xi_1)+\\calQ_1(x_1)\\Big).\n\\end{equation}","sha256_utf8":"e1158e8abe2a6d708921f22e7b579ee413b98786a038b08f8e8efa9bd6e061b0"}]},{"id":"S006","semantic_type":"notation","kind":"notation","title":"正的对偶界参数","source_units":[{"start":32334,"end":32469,"start_line":472,"end_line":472,"raw_tex":"To begin with, let us fix algorithmic parameters \\(M_t>0\\) for stages \\(t\\in\\calT'\\) and define alternative value function recursively\n","sha256_utf8":"e97977e9bd8c22ee0ba98e300e02859842351d94f08afa6ad655bc38c38c0ed5"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"正则化值函数递推","source_units":[{"start":32469,"end":32732,"start_line":473,"end_line":475,"raw_tex":"\\begin{equation}\\label{eq:RegularizedValueFunction}\n    Q_t^\\Reg(x_{t-1};\\xi_t)\\coloneqq\\max_{\\nVert{\\lambda_t}_*\\le M_t}\\inf_{\\substack{x_t\\in\\calX_t,\\\\z_t\\in\\bbR^{d_{t-1}}}}f_t(x_{t-1},x_t;\\xi_t)+\\calQ_t^\\Reg(x_t)+\\bangle{\\lambda_t}{x_{t-1}-z_t},\n\\end{equation}","sha256_utf8":"7953c6e4b6188ba5ab93a2d5b023bd97b3ddd049ac330c1e3a3f84b483e2e6e6"},{"start":32733,"end":32950,"start_line":476,"end_line":479,"raw_tex":"with \\(\\calQ_T^\\Reg(x_T)\\equiv0\\) on \\(\\calX_T\\) and\n\\begin{equation}\\label{eq:RegularizedCostToGo}\n    \\calQ_{t-1}^\\Reg(x_{t-1})\\coloneqq\\sup_{p_t\\in\\calP_t}\\bbE_{\\xi_t\\sim p_t} Q_t^\\Reg(x_{t-1};\\xi_t)\n\\end{equation}","sha256_utf8":"ce2d48cd3b6f74b52e97476f2b493f61e06effe1b1cc3818101099614e840524"},{"start":32951,"end":32976,"start_line":480,"end_line":480,"raw_tex":"for \\(t=T,T-1,\\dots,2\\).\n","sha256_utf8":"f6655ec6887a18cae0cfa9a5207e472ddef92bd27b1f992002338d8726495bd3"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"初始阶段子问题预言机","source_units":[{"start":41403,"end":42192,"start_line":571,"end_line":580,"raw_tex":"\\begin{definition}[Initial stage subproblem oracle]\n    \\label{def:InitialStageOracle}\n    Let \\(\\ulcQ_1,\\olcQ_1:\\calX_1\\to\\bbR\\cup\\{+\\infty\\}\\) denote two lsc, convex functions, with \\(\\ulcQ_1(x_1)\\le\\calQ_1^\\Reg(x_1)\\le\\olcQ_1(x_1)\\) for any \\(x_1\\in\\calX_1\\).\n    Consider the following subproblem for the first stage \\(t=1\\),\n    \\begin{equation}\\label{eq:InitialStageOracle}\n        \\min_{x_1\\in\\calX_1}f_1(x_0,x_1;\\xi_1)+\\ulcQ_1(x_1).\\tag{I}\n    \\end{equation}\n    The initial stage subproblem oracle provides an optimal solution \\(x_1\\) to~\\eqref{eq:InitialStageOracle} and calculates the approximation gap \\(\\gamma_1\\coloneqq\\olcQ_1(x_1)-\\ulcQ_1(x_1)\\).\n    We thus define the subproblem oracle formally as a map \\(\\scrO_1:(\\ulcQ_1,\\olcQ_1)\\mapsto(x_1;\\gamma_1)\\).\n\\end{definition}","sha256_utf8":"3cb8a92a37ef0b2aa4f47dfe60af48672394b0843b804777a4288ca63e692948"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"非初始阶段预言机","source_units":[{"start":42194,"end":43217,"start_line":582,"end_line":591,"raw_tex":"\\begin{definition}[Noninitial stage subproblem oracle]\n    \\label{def:NoninitialStageOracle}\n    For any \\(t\\in\\calT'\\), let \\(\\ulcQ_t,\\olcQ_t:\\calX_t\\to\\bbR\\cup\\{+\\infty\\}\\) denote two lsc, convex functions, with \\(\\ulcQ_t(x_t)\\le\\calQ_t^\\Reg(x_t)\\le\\olcQ_t(x_t)\\) for all \\(x_t\\in\\calX_t\\).\n    Then given a feasible state \\(x_{t-1}\\in\\calX_{t-1}\\), the stage-\\(t\\) subproblem oracle provides a feasible state \\(x_t\\in\\calX_t\\), an \\(M_t\\)-Lipschitz continuous linear cut \\(\\calV_{t-1}(\\,\\cdot\\,)\\), and an over-estimate value \\(v_{t-1}\\) such that\n    \\begin{compactitem}\n        \\item they are valid, i.e., \\(\\calV_{t-1}(\\cdot)\\le\\calQ_{t-1}^\\Reg(\\cdot)\\) on \\(\\calX_{t-1}\\) and \\(v_{t-1}\\ge\\calQ_{t-1}^\\Reg(x_{t-1})\\);\n        \\item the gap is controlled, i.e., \\(v_{t-1}-\\calV_{t-1}(x_{t-1})\\le\\gamma_t\\coloneqq\\olcQ_t(x_t)-\\ulcQ_t(x_t)\\).\n    \\end{compactitem}\n    We thus define the subproblem oracle formally as a map \\(\\scrO_t:(x_{t-1},\\ulcQ_t,\\olcQ_t)\\mapsto(\\calV_{t-1},v_{t-1},x_t;\\gamma_t)\\).\n\\end{definition}","sha256_utf8":"e7f91629b893bf59df05aeceded2ee42c9046b46a8475eecb01e0a920dcd71a5"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"上图闭凸包运算","source_units":[{"start":50226,"end":50536,"start_line":661,"end_line":665,"raw_tex":"To ease the notation, we use \\(\\conv\\{h_1,h_2\\}\\) to denote the function corresponding to the closed convex hull of the epigraphs of functions \\(h_1\\) and \\(h_2\\).\nMore precisely, we define\n\\[\n    \\conv\\{h_1,h_2\\}(x)\\coloneqq \\sup_{\\lambda}\\inf_{z}\\left\\{\\min\\{h_1(z),h_2(z)\\}+\\bangle{\\lambda}{x-z}\\right\\}.\n\\]","sha256_utf8":"a701f68ba201e2755574227d22deeb6c589e8c15d24bf4f8cf86e239458202b9"}]},{"id":"S011","semantic_type":"algorithm","kind":"algorithm","title":"连续动态规划算法","source_units":[{"start":50990,"end":52706,"start_line":677,"end_line":706,"raw_tex":"\\begin{algorithm}[ht]\n    \\caption{Consecutive Dual Dynamic Programming Algorithm}\n    \\label{alg:ConsecutiveDualDP}\n    \\begin{algorithmic}[1]\n        \\Require{subproblem oracles \\(\\scrO_t\\) for \\(t\\in\\calT\\), target optimality gap \\(\\epsilon>0\\)} \n        \\Ensure{an \\(\\epsilon\\)-optimal first stage solution \\(x_1^*\\)}\n        \\State{initialize: \\(\\ulcQ_t^0\\leftarrow 0,\\olcQ_t^0\\leftarrow +\\infty,t\\in\\calT\\backslash\\{T\\}\\); \\(\\ulcQ_T^j,\\olcQ_T^j\\leftarrow0,j\\in\\bbN\\); \\(i\\leftarrow 1\\)}\n        \\State{evaluate \\((x_1^1;\\gamma_1^1)\\leftarrow\\scrO_1(\\ulcQ_1^0,\\olcQ_1^0)\\)}\n        \\State{set \\(\\LB\\leftarrow f_1(x_0,x_1^1;\\xi_1),\\ \\UB\\leftarrow +\\infty\\)}\n        \\While{\\(\\UB-\\LB >\\epsilon\\)}\n        \\For{\\(t=2,\\dots,T\\)}\n        \\State{evaluate \\((\\calV_{t-1}^{i},v_{t-1}^{i},x_t^i;\\gamma_t^i)=\\scrO_t(x_{t-1}^i,\\ulcQ_t^{i-1},\\olcQ_t^{i-1})\\)}\n        \\Comment{forward step}\n        \\EndFor\n        \\For{\\(t=T,\\dots,2\\)}\n        \\State{update \\(\\ulcQ_{t-1}^i(x)\\leftarrow \\max\\{\\ulcQ_{t-1}^{i-1}(x),\\calV_{t-1}^i(x)\\}\\)}\n        \\Comment{backward step}\n        \\State{update \\(\\olcQ_{t-1}^i(x)\\leftarrow \\conv\\{\\olcQ_{t-1}^{i-1}(x),v_{t-1}^i+M_{t}\\nVert{x-x_{t-1}^i}\\}\\)}\n        \\EndFor\n        \\State{evaluate \\((x_1^{i+1};\\gamma_1^{i+1})\\leftarrow\\scrO_1(\\ulcQ_1^i,\\olcQ_1^i)\\)}\n        \\Comment{initial stage step}\n        \\State{update \\(\\LB\\leftarrow f_1(x_0,x_1^{i+1};\\xi_1)+\\ulcQ_1^i(x_1^{i+1})\\)}\n        \\State{update \\(\\UB'\\leftarrow f_1(x_0,x_1^{i+1};\\xi_1)+\\olcQ_1^i(x_1^{i+1})\\)}\n        \\If{\\(\\UB'<\\UB\\)}\n        \\State{set \\(x_1^*\\leftarrow x_1^{i+1}\\),  \\(\\UB\\leftarrow\\UB'\\)}\n        \\EndIf\n        \\State{update \\(i\\leftarrow i+1\\)}\n        \\EndWhile\n    \\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"ec76286ea3d78b3761ff6544f7ba7801f31b5b282caec4e382278317bc75c5a1"}]},{"id":"S012","semantic_type":"algorithm","kind":"algorithm","title":"非连续动态规划算法","source_units":[{"start":55303,"end":57572,"start_line":729,"end_line":766,"raw_tex":"\\begin{algorithm}[ht]\n    \\caption{Nonconsecutive Dual Dynamic Programming (NDDP) Algorithm}\n    \\label{alg:NonconsecutiveDualDP}\n    \\begin{algorithmic}[1]\n        \\Require{subproblem oracles \\(\\scrO_t\\) for \\(t\\in\\calT\\), opt.\\ and approx.\\ gaps \\(\\epsilon=\\delta_1>\\cdots>\\delta_T=0\\)} \n        \\Ensure{an \\(\\epsilon\\)-optimal first stage solution \\(x_1^*\\)}\n        \\State{initialize: \\(\\ulcQ_t^0\\leftarrow 0,\\olcQ_t^0\\leftarrow +\\infty,t\\in\\calT\\backslash\\{T\\}\\); \\(\\ulcQ_T^j,\\olcQ_T^j\\leftarrow0,j\\in\\bbN\\); \\(i_t\\leftarrow 0,t\\in\\calT\\)}\n        \\State{set \\(\\LB\\leftarrow 0,\\ \\UB\\leftarrow +\\infty\\), \\(t\\leftarrow 1\\)}\n        \\Loop\n        \\State{update \\(i_t\\leftarrow i_t+1\\)}\n        \\If{\\(t=1\\)}\n        \\State{evaluate \\((x_1^{i_1};\\gamma_1^{i_1})\\leftarrow\\scrO_1(\\ulcQ_1^{i_1},\\olcQ_1^{i_1})\\)}\n        \\Comment{initial stage step}\n        \\State{update \\(\\LB\\leftarrow f_1(x_0,x_1^{i_1};\\xi_1)+\\ulcQ_1^{i_1}(x_1^{i_1})\\)}\n        \\State{update \\(\\UB'\\leftarrow f_1(x_0,x_1^{i_1};\\xi_1)+\\olcQ_1^{i_1}(x_1^{i_1})\\)}\n        \\If{\\(\\UB'<\\UB\\)}\n        \\State{set \\(x_1^*\\leftarrow x_1^{i_1}\\),  \\(\\UB\\leftarrow\\UB'\\)}\n        \\EndIf\n        \\If{\\(\\UB-\\LB\\le\\epsilon\\)}\n        \\State{\\bfseries{break the loop}}\n        \\EndIf\n        \\State{maintain \\(\\ulcQ_2^{i_2+1}(x)\\leftarrow\\ulcQ_2^{i_2}(x),\\;\\olcQ_2^{i_2+1}(x)\\leftarrow\\olcQ_2^{i_2}(x)\\)}\n        \\State{set \\(t\\leftarrow t+1\\)}\n        \\Else\n        \\State{evaluate \\((\\calV_{t-1}^{i_t},v_{t-1}^{i_t},x_t^{i_t};\\gamma_t^{i_t})\\leftarrow\\scrO_t(x_{t-1}^{i_{t-1}},\\ulcQ_t^{i_t},\\olcQ_t^{i_t})\\)}\n        \\Comment{noninitial stage step}\n        \\If{\\(t<T\\) \\textbf{and} \\(\\gamma_t^{i_t}>\\delta_t\\)}\n        \\State{maintain \\(\\ulcQ_{t+1}^{i_{t+1}+1}(x)\\leftarrow\\ulcQ_{t+1}^{i_{t+1}}(x),\\;\\olcQ_{t+1}^{i_{t+1}+1}(x)\\leftarrow\\olcQ_{t+1}^{i_{t+1}}(x)\\)}\n        \\State{set \\(t\\leftarrow t+1\\)}\n        \\Else\n        \\State{update \\(\\ulcQ_{t-1}^{i_{t-1}+1}(x)\\leftarrow \\max\\{\\ulcQ_{t-1}^{i_{t-1}}(x),\\calV_{t-1}^{i_t}(x)\\}\\)}\n        \\State{update \\(\\olcQ_{t-1}^{i_{t-1}+1}(x)\\leftarrow \\conv\\{\\olcQ_{t-1}^{i_{t-1}}(x),v_{t-1}^{i_t}+M_{t-1}\\Vert{x-x_{t-1}^{i_{t-1}}}\\Vert\\}\\)}\n        \\State{set \\(t\\leftarrow t-1\\)}\n        \\EndIf\n        \\EndIf\n        \\EndLoop\n    \\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"16b0fca40a420db9c52d61403690ba852826da3e909613c05b6e36c8c6ba3642"}]},{"id":"S013","semantic_type":"notation","kind":"notation","title":"阶段预言机调用索引","source_units":[{"start":59053,"end":59594,"start_line":783,"end_line":785,"raw_tex":"To begin with, let \\(\\calJ_t,t>1\\), denote the set of pairs of indices \\((i_{t-1},i_t)\\) such that the noninitial stage subproblem oracle is evaluated at the \\(i_t\\)-th time at the state \\(x_{t-1}^{i_{t-1}}\\), i.e., \\((\\calV_{t-1}^{i_t},v_{t-1}^{i_t},x_t^{i_t};\\gamma_t^{i_t})=\\scrO_t(x_{t-1}^{i_{t-1}},\\ulcQ_t^{i_t},\\olcQ_t^{i_t})\\).\nFor the CDDP algorithm, all stages share the same iteration index \\(i_t=i\\), so \\(\\calJ_t=\\{(i,i):i\\in\\bbN\\}\\) for all \\(t>1\\).\nWe define the following sets of indices for each \\(t\\in\\calT\\setminus\\{T\\}\\):\n","sha256_utf8":"d8d01519ea1abee7c7ce86f874cfc02f5c0fbcf1384d260392e21dfb19286f78"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"跨阶段间隙索引集","source_units":[{"start":59594,"end":59784,"start_line":786,"end_line":788,"raw_tex":"\\begin{equation}\n    \\calI_t(\\delta)\\coloneqq\\left\\{i_t\\in\\bbN:\\gamma_t^{i_t}>\\delta_t\\text{ and } \\gamma_{t+1}^{i_{t+1}}\\le\\delta_{t+1},\\ (i_t,i_{t+1})\\in\\calJ_{t+1}\\right\\}.\n\\end{equation}","sha256_utf8":"7dfa9bae8705417d2a209036a7c19d95a9ef5bf5164ee7f308f171b962909d66"},{"start":59785,"end":60287,"start_line":789,"end_line":790,"raw_tex":"Here, for NDDP algorithm, \\(\\delta\\) is the given approximation gap vector, while for CDDP algorithm, \\(\\delta=(\\delta_t)_{t=1}^T\\) can be any vector satisfying \\(\\epsilon=\\delta_1>\\delta_2>\\cdots>\\delta_T=0\\) for the purpose of analysis, since it is not required for the CDDP algorithm.\nWe adopt the convention that the gap for the last stage \\(\\gamma_T^{i_T}\\equiv0\\) such that \\(i_{T-1}\\in\\calI_{T-1}(\\delta)\\) if and only if \\(\\gamma_{T-1}^{i_{T-1}}>\\delta_{T-1}\\) and \\((i_{T-1},i_T)\\in\\calJ_T\\).\n","sha256_utf8":"2a7cb2d7e26008014eaffbd58a6cdf5e6a70119d5799230d238b4bffe93f4eca"}]},{"id":"S015","semantic_type":"lemma","kind":"lemma","title":"间隙索引集的容量界","source_units":[{"start":60990,"end":61343,"start_line":794,"end_line":800,"raw_tex":"\\begin{lemma}\\label{lemma:FiniteIndexSet}\n    For stage any \\(t<T\\), suppose the state space \\(\\calX_t\\subset\\bbR^{d_t}\\) is contained in a ball with diameter \\(D_t>0\\).\n    Then,\n    \\begin{equation}\\label{eq:FiniteIndexSet}\n        \\abs{\\calI_t(\\delta)}\\le\\left(1+\\frac{2M_{t+1} D_t}{\\delta_t-\\delta_{t+1}}\\right)^{d_t}.\n    \\end{equation}\n\\end{lemma}","sha256_utf8":"4292406ac4093927c70c2644cdda15c4308b887b3799f21c91c6a5f2bfb5ff93"}]},{"id":"S016","semantic_type":"assumption","kind":"assumption","title":"辅助引理的状态球界","source_units":[{"start":61032,"end":61160,"start_line":795,"end_line":795,"raw_tex":"    For stage any \\(t<T\\), suppose the state space \\(\\calX_t\\subset\\bbR^{d_t}\\) is contained in a ball with diameter \\(D_t>0\\).\n","sha256_utf8":"87fa93410959776f552ff67799df6f305d71a737329e119d3beb5d7d02e52887"}]},{"id":"S017","semantic_type":"proof","kind":"proof","title":"索引集容量界完整证明","source_units":[{"start":61344,"end":63857,"start_line":801,"end_line":825,"raw_tex":"\\begin{proof}\n    We claim that for any \\(j,k\\in\\calI_t\\), \\(j\\neq k\\), it holds that \\(\\nVert{x_t^j-x_t^k}>(\\delta_t-\\delta_{t+1})/(2M_{t+1})\\).\n    Assume for contradiction that \\(\\nVert{x_t^j-x_t^k}\\le(\\delta_t-\\delta_{t+1})/(2M_{t+1})\\) for some \\(j<k\\), \\(j,k\\in\\calI_t(\\delta)\\).\n    By definition of \\(\\calI_t(\\delta)\\), the \\((t+1)\\)-th subproblem oracle is evaluated at the state \\(x_t^j\\), and in both the CDDP and the NDDP algorithms, the approximations \\(\\ulcQ_t^j\\) and \\(\\olcQ_t^j\\) are updated since \\(\\gamma_{t+1}^{i_{t+1}}\\le\\delta_{t+1}\\) for some \\(i_{t+1}\\in\\bbN\\) with \\((j,i_{t+1})\\in\\calJ_{t+1}\\).\n    Then by Definition~\\ref{def:NoninitialStageOracle} of the noninitial stage subproblem oracle, we have \\(\\olcQ_t^{j+1}(x_t^j)-\\ulcQ_t^{j+1}(x_t^j)\\le \\delta_{t+1}\\).\n    Note that for any point \\(x\\in\\calX_t\\) with \\(\\nVert{x-x_t^j}\\le(\\delta_t-\\delta_{t+1})/(2M_{t+1})\\), we have \\(\\olcQ_t^{j+1}(x)-\\ulcQ_t^{j+1}(x)\\le \\delta_{t}\\) because of the \\(M_{t+1}\\)-Lipschitz continuity of the approximations.\n    Since \\(\\olcQ_t^{k}(x)\\le\\olcQ_t^{j+1}(x)\\) and \\(\\ulcQ_t^{k}(x)\\ge\\ulcQ_t^{j+1}(x)\\), by setting \\(x=x_t^k\\), we see a contradiction with the assumption that \\(k\\in\\calI_t(\\delta)\\), which proves the claim.\n\n    To ease the notation, let \\(r_t\\coloneqq(\\delta_t-\\delta_{t+1})/(2M_{t+1})\\) denote the radius of the \\(d_t\\)-dimensional balls \\(\\calB^{d_t}(x_t^j;r_t)\\) centered at \\(x_t^j\\) for \\(j\\in\\calI_t(\\delta)\\), and let \\(\\calB_t\\supseteq\\calX_t\\) denote a ball with diameter \\(D_t\\).\n    From the above claim, we know that \\(x_t^k\\notin\\calB^{d_t}(x_t^j;r_t)\\) for any \\(j,k\\in\\calI_t(\\delta)\\) with \\(j<k\\).\n    In other words, the smaller balls \\(\\calB^{d_t}(x_t^j;r_t/2)\\) are disjoint.\n    Meanwhile, note that each of these smaller balls satisfies \\(\\calB^{d_t}(x_t^j;r_t/2)\\subset\\calB_t+\\calB^{d_t}(0;r_t/2)\\) (the Minkowski sum in the Euclidean space \\(\\bbR^{d_t}\\)).\n    Therefore, the volumes satisfy the relation\n    \\begin{equation*}\n     \\Vol\\left(\\textstyle\\bigcup_{j\\in\\calI_t(\\delta)}\\calB^{d_t}(x_t^j;r_t/2)\\right)\n        =\\abs{\\calI_t(\\delta)}\\cdot\\Vol\\calB^{d_t}(0;r_t/2)\n        \\le\\Vol\\left(\\calB_t+\\calB^{d_t}(0;r_t/2)\\right),\n    \\end{equation*}\n    which implies that\n    \\begin{equation*}\n        \\abs{\\calI_t(\\delta)}\\le\\frac{\\Vol(\\calB_t+\\calB^{d_t}(0;r_t/2))}{\\Vol\\calB^{d_t}(0;r_t/2)}\n        =\\left(\\frac{D_t/2+r_t/2}{r_t/2}\\right)^{d_t}\n        =\\left(1+\\frac{2M_{t+1}D_t}{\\delta_t-\\delta_{t+1}}\\right)^{d_t}.\n    \\end{equation*}\n\\end{proof}","sha256_utf8":"06cd4c3ba4fab57918ceca49f528508a3eaaa0f4a66021cd04d13b54c476f977"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"提出状态分离并设反例","source_units":[{"start":61358,"end":61630,"start_line":802,"end_line":803,"raw_tex":"    We claim that for any \\(j,k\\in\\calI_t\\), \\(j\\neq k\\), it holds that \\(\\nVert{x_t^j-x_t^k}>(\\delta_t-\\delta_{t+1})/(2M_{t+1})\\).\n    Assume for contradiction that \\(\\nVert{x_t^j-x_t^k}\\le(\\delta_t-\\delta_{t+1})/(2M_{t+1})\\) for some \\(j<k\\), \\(j,k\\in\\calI_t(\\delta)\\).\n","sha256_utf8":"820c5cbf4f7a9c296131243fceab1939a70e6daf94d3654b75ede98e2c13c0a6"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"预言机更新缩小访问点间隙","source_units":[{"start":61630,"end":62134,"start_line":804,"end_line":805,"raw_tex":"    By definition of \\(\\calI_t(\\delta)\\), the \\((t+1)\\)-th subproblem oracle is evaluated at the state \\(x_t^j\\), and in both the CDDP and the NDDP algorithms, the approximations \\(\\ulcQ_t^j\\) and \\(\\olcQ_t^j\\) are updated since \\(\\gamma_{t+1}^{i_{t+1}}\\le\\delta_{t+1}\\) for some \\(i_{t+1}\\in\\bbN\\) with \\((j,i_{t+1})\\in\\calJ_{t+1}\\).\n    Then by Definition~\\ref{def:NoninitialStageOracle} of the noninitial stage subproblem oracle, we have \\(\\olcQ_t^{j+1}(x_t^j)-\\ulcQ_t^{j+1}(x_t^j)\\le \\delta_{t+1}\\).\n","sha256_utf8":"39eeb4be5dd39b188f4405fa56717bc7cb0a716004d2c9e544b4b6f6e3ca00ec"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"连续性与单调性排除近邻","source_units":[{"start":62134,"end":62584,"start_line":806,"end_line":807,"raw_tex":"    Note that for any point \\(x\\in\\calX_t\\) with \\(\\nVert{x-x_t^j}\\le(\\delta_t-\\delta_{t+1})/(2M_{t+1})\\), we have \\(\\olcQ_t^{j+1}(x)-\\ulcQ_t^{j+1}(x)\\le \\delta_{t}\\) because of the \\(M_{t+1}\\)-Lipschitz continuity of the approximations.\n    Since \\(\\olcQ_t^{k}(x)\\le\\olcQ_t^{j+1}(x)\\) and \\(\\ulcQ_t^{k}(x)\\ge\\ulcQ_t^{j+1}(x)\\), by setting \\(x=x_t^k\\), we see a contradiction with the assumption that \\(k\\in\\calI_t(\\delta)\\), which proves the claim.\n","sha256_utf8":"3b921b0b21ddfbd99ddc8174dbea9e9c6158a95d81df5ffa6410275adeb61527"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"构造互不相交的小球","source_units":[{"start":62585,"end":63260,"start_line":809,"end_line":812,"raw_tex":"    To ease the notation, let \\(r_t\\coloneqq(\\delta_t-\\delta_{t+1})/(2M_{t+1})\\) denote the radius of the \\(d_t\\)-dimensional balls \\(\\calB^{d_t}(x_t^j;r_t)\\) centered at \\(x_t^j\\) for \\(j\\in\\calI_t(\\delta)\\), and let \\(\\calB_t\\supseteq\\calX_t\\) denote a ball with diameter \\(D_t\\).\n    From the above claim, we know that \\(x_t^k\\notin\\calB^{d_t}(x_t^j;r_t)\\) for any \\(j,k\\in\\calI_t(\\delta)\\) with \\(j<k\\).\n    In other words, the smaller balls \\(\\calB^{d_t}(x_t^j;r_t/2)\\) are disjoint.\n    Meanwhile, note that each of these smaller balls satisfies \\(\\calB^{d_t}(x_t^j;r_t/2)\\subset\\calB_t+\\calB^{d_t}(0;r_t/2)\\) (the Minkowski sum in the Euclidean space \\(\\bbR^{d_t}\\)).\n","sha256_utf8":"e27a74d9f1b26b487d7dfbabdcf7a94172ce947898041dda486b89ca8a9d1288"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"比较小球并集与包围球体积","source_units":[{"start":63260,"end":63554,"start_line":813,"end_line":818,"raw_tex":"    Therefore, the volumes satisfy the relation\n    \\begin{equation*}\n     \\Vol\\left(\\textstyle\\bigcup_{j\\in\\calI_t(\\delta)}\\calB^{d_t}(x_t^j;r_t/2)\\right)\n        =\\abs{\\calI_t(\\delta)}\\cdot\\Vol\\calB^{d_t}(0;r_t/2)\n        \\le\\Vol\\left(\\calB_t+\\calB^{d_t}(0;r_t/2)\\right),\n    \\end{equation*}\n","sha256_utf8":"ac9d406703b837b21d21fef119f82218f11a6cbbb4524d2fe8eee7de08bd3bf0"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"计算体积比得到容量界","source_units":[{"start":63554,"end":63846,"start_line":819,"end_line":824,"raw_tex":"    which implies that\n    \\begin{equation*}\n        \\abs{\\calI_t(\\delta)}\\le\\frac{\\Vol(\\calB_t+\\calB^{d_t}(0;r_t/2))}{\\Vol\\calB^{d_t}(0;r_t/2)}\n        =\\left(\\frac{D_t/2+r_t/2}{r_t/2}\\right)^{d_t}\n        =\\left(1+\\frac{2M_{t+1}D_t}{\\delta_t-\\delta_{t+1}}\\right)^{d_t}.\n    \\end{equation*}\n","sha256_utf8":"807865a6479a312e65a0c35c30da4782b9dee79fff81f6ea3913b7bdcc105caf"}]},{"id":"S024","semantic_type":"theorem","kind":"theorem","title":"非连续算法的预言机复杂度","source_units":[{"start":67023,"end":67668,"start_line":867,"end_line":874,"raw_tex":"\\begin{theorem}\\label{thm:NDDPComplexityBound}\n    Suppose the state spaces \\(\\calX_t\\subset\\bbR^{d_t}\\) are contained in balls, each with diameter \\(D_t>0\\).\n    Then, for the NDDP algorithm (Algorithm~\\ref{alg:NonconsecutiveDualDP}) with the predetermined approximation gap vector \\((\\delta_t)_{t=1}^{T}\\) satisfying \\(\\epsilon=\\delta_1>\\delta_2>\\cdots>\\delta_T=0\\), \n    the total number of subproblem oracle evaluations \\(\\NumEval_\\NDDP\\) before termination is bounded by\n    \\begin{equation*}\n        \\NumEval_\\NDDP\\le 1+2\\cdot\\sum_{t=1}^{T-1}\\left(1+\\frac{2M_{t+1}D_t}{\\delta_t-\\delta_{t+1}}\\right)^{d_t}.\n    \\end{equation*}\n\\end{theorem}","sha256_utf8":"fdb162343a6420b649090c6d381b782e4a46fb510ee4fa5ded71b6680387101a"}]},{"id":"S025","semantic_type":"assumption","kind":"assumption","title":"主定理的状态界与间隙条件","source_units":[{"start":67070,"end":67393,"start_line":868,"end_line":869,"raw_tex":"    Suppose the state spaces \\(\\calX_t\\subset\\bbR^{d_t}\\) are contained in balls, each with diameter \\(D_t>0\\).\n    Then, for the NDDP algorithm (Algorithm~\\ref{alg:NonconsecutiveDualDP}) with the predetermined approximation gap vector \\((\\delta_t)_{t=1}^{T}\\) satisfying \\(\\epsilon=\\delta_1>\\delta_2>\\cdots>\\delta_T=0\\), \n","sha256_utf8":"edeafa5ab2b401e9317b2bcb2d0179baabc678f831fe6dcbd76ea8fdec98b687"}]},{"id":"S026","semantic_type":"proof","kind":"proof","title":"非连续算法复杂度完整证明","source_units":[{"start":67669,"end":68398,"start_line":875,"end_line":880,"raw_tex":"\\begin{proof}\n    For the NDDP algorithm, each time when it decides to go back to the precedent stage \\(t\\leftarrow t-1\\), we must have \\(\\gamma_t^{i_t}\\le\\delta_t\\) while \\(\\gamma_{t-1}^{i_{t-1}}>\\delta_{t-1}\\) for some \\((i_{t-1},i_t)\\in\\calJ_t\\).\n    In this case, we have by definition that \\(i_{t-1}\\in\\calI_{t-1}(\\delta)\\).\n    By Lemma~\\ref{lemma:FiniteIndexSet}, such ``going back'' steps can only happen at most \\(\\sum_{t=1}^{T-1}\\abs{\\calI_t(\\delta)}\\) times.\n    The total number of evaluations is then bounded by two times the number of ``going back'' steps due to the corresponding ``going forward'' steps, plus an additional evaluation of the initial stage oracle for checking the termination criterion.\n\\end{proof}","sha256_utf8":"d5079d15cf88278667177062f3b06bc1a1ee6af9be0f774aac9c8973bd1ce250"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"回退步骤对应间隙索引","source_units":[{"start":67683,"end":67999,"start_line":876,"end_line":877,"raw_tex":"    For the NDDP algorithm, each time when it decides to go back to the precedent stage \\(t\\leftarrow t-1\\), we must have \\(\\gamma_t^{i_t}\\le\\delta_t\\) while \\(\\gamma_{t-1}^{i_{t-1}}>\\delta_{t-1}\\) for some \\((i_{t-1},i_t)\\in\\calJ_t\\).\n    In this case, we have by definition that \\(i_{t-1}\\in\\calI_{t-1}(\\delta)\\).\n","sha256_utf8":"76833c0462ca03cef0db388e3167c114029b721d8ffade48c255149683816660"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"容量引理限制回退次数","source_units":[{"start":67999,"end":68139,"start_line":878,"end_line":878,"raw_tex":"    By Lemma~\\ref{lemma:FiniteIndexSet}, such ``going back'' steps can only happen at most \\(\\sum_{t=1}^{T-1}\\abs{\\calI_t(\\delta)}\\) times.\n","sha256_utf8":"bcf121861d1228b2a993bde16a7b44b9556bf6f7463edef5eb8356cc71f6016f"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"配对前进回退并计入终止调用","source_units":[{"start":68139,"end":68387,"start_line":879,"end_line":879,"raw_tex":"    The total number of evaluations is then bounded by two times the number of ``going back'' steps due to the corresponding ``going forward'' steps, plus an additional evaluation of the initial stage oracle for checking the termination criterion.\n","sha256_utf8":"a81e32453919390ff727288819d47a4c8087e8253cb21346cfc1d5100ade70e5"}]},{"id":"S030","semantic_type":"external_fact","kind":"external_fact","title":"近似更新保持共同连续性界","source_units":[],"declared_external":true,"declaration_text":["The pointwise maximum of affine functions whose slopes have dual norm at most M is M-Lipschitz wherever finite. Taking the closed convex envelope of M-Lipschitz cone functions preserves their common Lipschitz bound wherever finite."]},{"id":"S031","semantic_type":"external_fact","kind":"external_fact","title":"欧氏球的体积与闵可夫斯基和","source_units":[],"declared_external":true,"declaration_text":["Euclidean volume is additive on disjoint measurable sets and monotone under inclusion. A d-dimensional ball of radius r has volume c_d r^d, and the Minkowski sum of two Euclidean balls is a ball with the sum of their radii."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S024","edges":[{"source_id":"S001","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S009","kind":"supports"},{"source_id":"S005","target_id":"S011","kind":"supports"},{"source_id":"S005","target_id":"S012","kind":"supports"},{"source_id":"S008","target_id":"S011","kind":"supports"},{"source_id":"S008","target_id":"S012","kind":"supports"},{"source_id":"S009","target_id":"S011","kind":"supports"},{"source_id":"S009","target_id":"S012","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S018","kind":"supports"},{"source_id":"S006","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S014","target_id":"S019","kind":"supports"},{"source_id":"S009","target_id":"S019","kind":"supports"},{"source_id":"S011","target_id":"S019","kind":"supports"},{"source_id":"S012","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S009","target_id":"S020","kind":"supports"},{"source_id":"S011","target_id":"S020","kind":"supports"},{"source_id":"S012","target_id":"S020","kind":"supports"},{"source_id":"S014","target_id":"S020","kind":"supports"},{"source_id":"S030","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S016","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S031","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S031","target_id":"S023","kind":"supports"},{"source_id":"S016","target_id":"S023","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S017","kind":"supports"},{"source_id":"S020","target_id":"S017","kind":"supports"},{"source_id":"S021","target_id":"S017","kind":"supports"},{"source_id":"S022","target_id":"S017","kind":"supports"},{"source_id":"S023","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S015","kind":"supports"},{"source_id":"S012","target_id":"S027","kind":"supports"},{"source_id":"S014","target_id":"S027","kind":"supports"},{"source_id":"S025","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S015","target_id":"S028","kind":"supports"},{"source_id":"S025","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S012","target_id":"S029","kind":"supports"},{"source_id":"S008","target_id":"S029","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S029","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"}]}}
{"record_id":"2010.11379v2","paper":{"title":"Local Convergence Analysis of Augmented Lagrangian Methods for Piecewise Linear-Quadratic Composite Optimization Problems","authors":["Nguyen T. V. Hang","M. Ebrahim Sarabi"],"arxiv_id":"2010.11379","version":"v2","journal":"SIAM Journal on Optimization","doi":"10.1137/20m1375188","abstract_url":"https://arxiv.org/abs/2010.11379v2","pdf_url":"https://arxiv.org/pdf/2010.11379v2"},"context":{"raw_tex":"Given twice continuously differentiable functions   $\\ph:\\R^n\\to \\R$ and  $\\Phi:\\R^n\\to \\R^m$,  a convex polyhedral set  $\\Th$ in $\\R^n$, and a convex piecewise linear-quadratic (CPLQ) function $g:\\R^m\\to \\oR$,\n we aim to present a local convergence analysis of the ALM for  the composite optimization problem \n\\begin{equation}\\label{comp}\n\\mbox{minimize}\\;\\;\\ph(x)+g\\big(\\Phi(x)\\big)\\quad \\mbox{subject to}\\;\\; x\\in \\Th.\n\\end{equation} \n\n\nThe augmented Lagrangian $\\L:\\R^n\\times \\R^m\\times (0,\\infty)\\to \\R$ associated with the composite problem \\eqref{comp} is defined  by \n\\begin{equation}\\label{aug}\n\\L(x,\\lm,\\rho):=\\ph(x)+e_{ {1}/{\\rho}} g \\big(\\Phi(x)+\\rho^{-1}\\lm\\big)-\\sm\\rho^{-1}\\|\\lm\\|^2\n\\end{equation}\nwith $e_{1/\\rho} g$ signifying  the Moreau envelope of   $g$, given  by \n\\begin{equation*}\\label{Menv}\ne_{1/\\rho} g(y):=\\inf_{z\\in \\R^m}\\big\\{g(z)+\\frac{\\rho}{2}\\|y-z\\|^2\\big\\}, \\quad y\\in \\R^m.\n\\end{equation*}\nIt is well-known that the augmented Lagrangian $\\L$ is continuously differentiable with respect to both $x$ and $\\lm$ and is nondecreasing with the respect to $\\rho$, the {\\em penalty} parameter of the augmented Lagrangian. \nAt  each iteration for a given pair $(\\lm,\\rho)\\in \\R^m\\times (0,\\infty)$,  our proposed  ALM for  the  composite problem \\eqref{comp}\nrequires solving the {\\em partially} augmented problem \n\\begin{equation}\\label{augp}\n\\mini \\L(x, \\lm, \\rho)\\quad \\mbox{subject to}\\;\\; x\\in \\Th.\n\\end{equation} \n\n\nIn this section, we first briefly review basic constructions of variational analysis and generalized differentiation employed in the paper. In what follows,    we \n denote by $\\B$ the closed unit ball in the space in question and by $\\B_r(x):=x+r\\B$ the closed ball centered at $x$ with radius $r>0$. \n In the  product space $\\R^n\\times \\R^m$, we use the norm $\\|(w,u)\\|=\\sqrt{\\|w\\|^2+\\|u\\|^2}$ for any $(w,u)\\in \\R^n\\times \\R^m$.\n Given a nonempty set $C\\subset\\R^n$, the symbols $\\inte C$,  $\\cone C$, and $\\span C$ signify its interior, conic hull, and the   linear space generated by  $C$, respectively. \n For any set $C$ in $\\R^n$, its indicator function is defined by $\\dd_C(x)=0$ for $x\\in C$ and $\\dd_C(x)=\\infty$ otherwise. We denote\n by $P_C$ the projection mapping onto $C$ and  by $\\dist(x,C)$  the distance between $x\\in \\R^n$ and a set $C$.\n For a vector $w\\in \\R^n$, the subspace $\\{tw |\\, t\\in \\R\\}$ is denoted by $[w]$. \nWe write $x(t)=o(t)$ with $x(t)\\in \\R^n$ and $t>0$ to mean that ${\\|x(t)\\|}/{t}$ goes to $0$ as $t\\dn 0$.\nFinally, we denote by $\\R_+$ (respectively,  $\\R_-$) the set of non-negative (respectively, non-positive) real numbers.\nFor an $n\\times m$ matrix $A$, we denote by $\\ker A$ and $\\rge A$ its null and range spaces, respectively.\n\n\nA sequence    $\\{C^k\\b$ of subsets in $\\R^n$ is called convergent to a set $C\\subset\\R^n$ as $k\\to \\infty$ if $C$ is closed and\n\\begin{equation*}\n\\lim_{k\\to \\infty}{\\rm dist}(w,C^k)={\\rm dist}(w,C)\\;\\mbox{ for all }\\;w\\in\\R^n.\n\\end{equation*}\nGiven a nonempty set $C\\subset\\R^n$ with $\\ox\\in C$, the  tangent cone $T_ C(\\ox)$ to $C$ at $\\ox$ is defined by\n\\begin{equation*}\\label{2.5}\nT_C(\\ox)=\\big\\{w\\in\\R^n|\\;\\exists\\,t_k{\\dn}0,\\;\\;w_k\\to w\\;\\;\\mbox{ as }\\;k\\to\\infty\\;\\;\\mbox{with}\\;\\;\\ox+t_kw_k\\in C\\big\\}.\n\\end{equation*}\nWhen $C$ is convex, the  {normal cone} to $C$ at $\\ox\\in C$ in the sense of convex analysis  is given by\n\\begin{equation*}\nN_C(\\ox)=\\big\\{v\\in\\R^n\\;\\big|\\; \\la v,x-\\ox\\ra\\le 0\\quad \\mbox{for all}\\;\\; x\\in C\\big\\}.\n\\end{equation*}\nFor a convex  function $f:\\R^n \\to \\oR:=(-\\infty,\\infty]$  with $\\ox\\in \\dom f$, its   subdifferential at $\\ox$   is defined  by \n\\begin{equation*}\\label{sub}\n\\partial f(\\ox)=\\big\\{v\\in\\R^n\\;\\big|\\;  \\la v,x-\\ox\\ra\\le f(x)-f(\\ox)\\;\\;\\mbox{for all}\\;x\\in \\R^n\\big\\}.\n\\end{equation*}\n\n\nA sequence $\\{f^k\\b$ of functions $f^k:\\R^n\\to \\oR$ is said to {\\em epi-converge} to a function $f:\\R^n\\to [-\\infty,\\infty]$ if we have $\\epi f^k\\to \\epi f$ as $k\\to \\infty$.\nWe denote by $f^k\\overset {\\ss \\mbox{e}}{\\to} f$ the  epi-convergence of the sequence $\\{f^k\\b$ to $f$.\nGiven  a function  $f:\\R^n \\to \\oR$ and  a point $\\ox$ with $f(\\ox)$ finite, the subderivative function $\\d f(\\ox)\\colon\\R^n\\to[-\\infty,\\infty]$ is defined by\n\\begin{equation}\\label{fsud}\n{\\mathrm d}f(\\ox)(w)=\\liminf_{\\substack{\n   t\\dn 0 \\\\\n  w'\\to w\n  }} {\\frac{f(\\ox+tw')-f(\\ox)}{t}}.\n\\end{equation}\n The critical cone of $f$ at $\\ox$ for $\\ov$ with $(\\ox,\\ov)\\in \\gph \\sub f$ is defined by \n$$\n{K_f}(\\ox,\\ov)=\\big\\{w\\in \\R^n\\big|\\;\\la\\bar v,w\\ra=\\d f(\\ox)(w)\\big\\}.\n$$\nWhen $f=\\dd_C$, namely the indicator function of $C$, the critical cone of $\\dd_C$ at $\\ox$ for $\\ov$ is denoted by $K_C(\\ox,\\ov)$. In this case, the above definition of the critical cone of a function \nboils down to  the well-known concept of the critical cone of a set (see \\cite[page~109]{dr}), namely $K_C(\\ox,\\ov)=T_C(\\ox)\\cap [\\ov]^\\perp$.\nDefine the parametric  family of \nsecond-order difference quotients for $f$ at $\\ox$ for $\\ov\\in \\R^n$ by \n\\begin{equation*}\\label{lk01}\n\\Delta_t^2 f(\\bar x , \\ov)(w)=\\dfrac{f(\\ox+tw)-f(\\ox)-t\\langle \\ov,\\,w\\rangle}{\\frac{1}{2}t^2}\\quad\\quad\\mbox{with}\\;\\;w\\in \\R^n, \\;\\;t>0.\n\\end{equation*}\nIf $f(\\ox)$ is finite, then the {second subderivative} of $f$ at $\\ox$ for $\\ov$   is given by \n\\begin{equation*}\\label{ssd}\n\\d^2 f(\\bar x , \\ov)(w)= \\liminf_{\\substack{\n   t\\dn 0 \\\\\n  w'\\to w\n  }} \\Delta_t^2 f(\\ox , \\ov)(w'),\\;\\; w\\in \\R^n.\n\\end{equation*}\nThe second subderivative of $f$ at $\\ox$ (without mention of $\\ov$) is defined by \n\\begin{equation}\\label{ssud}\n\\d^2 f(\\ox)(w)=\\liminf_{\\substack{t\\dn 0\\\\\nw'\\to w}} \\frac{f(\\ox+tw')-f(\\ox)-t\\d f(\\ox)(w')}{\\sm t^2},\\quad w\\in \\R^n.\n\\end{equation}\nFollowing \\cite[Definition~13.6]{rw}, a function $f:\\R^n \\to \\oR$ is said to be {twice epi-differentiable} at $\\bar x$ for $\\ov\\in\\R^n$, with $f(\\ox) $ finite, \nif the sets $\\epi \\Delta_t^2 f(\\bar x , \\ov)$ converge to $\\epi \\d^2 f(\\bar x,\\ov)$ as $t\\downarrow 0$. \nIn what follows, we refer to the second subderivative of $f$ at $\\bar x$ for $\\ov$ by the {\\em second-order epi-derivative} of $f$ when it is twice epi-differentiable at   $\\bar x$ for $\\ov$.\n\n\nRecall  that a function $g:\\R^m\\to \\oR$ is called  piecewise linear-quadratic if $\\dom g = \\cup_{i=1}^{s} C_i$ with $s\\in \\N$ and $C_i $ being polyhedral convex  sets for $i = 1, \\ldots, s$, and if $g$ has a representation of the form\n\\begin{equation} \\label{PWLQ}\ng(z) = \\sm \\langle A_i z ,z \\rangle + \\langle a_i ,z \\rangle + \\alpha_i  \\quad \\mbox{for all} \\quad  z \\in C_i,\n\\end{equation}\nwhere $A_i$ is an $m \\times m$ symmetric matrix, $a_i\\in \\R^m$, and $\\alpha_i\\in \\R$ for all $i = 1, \\cdots, s$. Take $\\oz\\in \\dom g$ and define the {\\em active} indices  of the domain of $g$ at $\\oz$ by\n\\begin{equation}\\label{act}\nI(\\oz)=\\big \\{i\\in \\{1,\\ldots, m\\} |\\, \\oz\\in C_i\\big \\}.\n\\end{equation}\nIf in addition $g$ is convex, \nit follows   from \\cite[page~487]{rw} that \nif  $\\oz\\in \\dom g$, the subdifferential of $g$ at  $\\oz$ can be calculated by \n\\begin{equation}\\label{sub}\n\\sub g(\\oz)=\\bigcap_{i\\in I(\\oz)} \\big \\{v\\in \\R^m |\\, v-A_i \\oz- a_i\\in N_{ C_i}(\\oz)\\big \\}.\n\\end{equation}\n\n\nThe   Karush-Kuhn-Tucker (KKT)   system associated with the composite problem \\eqref{comp} is given by \n\\begin{equation}\\label{vs} \n0\\in \\nabla_x L(x,\\lm)+N_\\Th(x),\\;\\;\\lm\\in \\sub g(\\Phi(x)),\n\\end{equation}\nwhere $L(x,\\lm)= \\ph(x)+\\la \\Phi(x),\\lm\\ra$ for any   $(x,\\lm)\\in \\R^n\\times \\R^m$. \nGiven a point $\\ox\\in\\R^n$, we define the set of {  Lagrange multipliers} of the KKT system \\eqref{vs} associated with $\\ox$ by\n\\begin{equation}\\label{laset}\n\\Lambda(\\ox):=\\big\\{\\lm\\in\\R^m\\;\\big|\\; 0\\in \\nabla_x L(\\ox,\\lm)+N_\\Th(\\ox),\\;\\lm\\in \\sub g(\\Phi(\\ox))\\big\\}.\n\\end{equation}\nIf  $(\\ox,\\olm)$ is a solution to the KKT  system \\eqref{vs},   we clearly have   $\\olm\\in\\Lambda(\\ox)$.\n Given  a solution $(\\ox,\\olm)$  to \\eqref{vs}, the second-order sufficient condition (SOSC)\nfor the composite problem \\eqref{comp} at $(\\ox,\\olm)$ is formulated by \n\\begin{equation}\\label{sosc}\n\\langle\\nabla_{xx}^2L(\\bar x,\\olm)w,w\\rangle+\\d^2g (\\Phi(\\bar x),\\olm) (\\nabla \\Phi(\\ox)w )>0\\quad \\mbox{for all }\\; w\\in \\D\\setminus\\{0\\},\n\\end{equation}\nwhere the convex cone $\\D$ is defined by \n \\begin{equation}\\label{coned}\n \\D:=K_\\Th\\big(\\ox,-\\nabla_x L(\\ox,\\olm)\\big)\\cap \\big\\{w\\in \\R^n\\big|\\; \\nabla \\Phi(\\ox)w\\in K_g \\big(\\Phi(\\ox),\\olm \\big)\\big\\}.\n \\end{equation}\n\n\na result, obtained recently in \\cite{s20}, showing that it yields an error bound estimate for the KKT system \\eqref{vs}, which is central to our   local convergence analysis  in this paper. To this end,  recall that, given a constant $r>0$,    the proximal mapping of  a convex function $g:\\R^m\\to \\oR$ is defined by \n\\begin{equation*}\\label{pr}\n{\\rm{prox}}_{rg}(x):=\\underset{z\\in \\R^n}{\\argmin} \\big\\{g(z)+\\rsm\\|x-z\\|^2\\big\\},\\quad x\\in\\R^m.\n\\end{equation*}\nIn what follows, when $r=1$, the proximal mapping of $g$ will be denoted by ${\\rm{prox}_g}$. For such a convex function, \nit is known that for any $x\\in \\R^m$ we have \n\\begin{equation}\\label{prm}\n{\\rm{prox}}_{rg}(x)=\\big(I+r\\sub g\\big)^{-1}(x) \\quad \\mbox{and}\\quad \\nabla e_rg(x)=\\big(rI+(\\sub g)^{-1}\\big)^{-1}(x),\n\\end{equation}\nwhere $I$ stands for the $m\\times m$ identity matrix; see \\cite[Proposition~12.19]{rw} and \\cite[Theorem~2.26]{rw} for more details. \n\n\nA function $f:\\R^n\\to \\oR$ is called fully amenable at $\\ox\\in \\dom f$ if there is a neighborhood ${\\cal O}$ of $\\ox$ on which \n$f$ can be represented in the form $f=\\psi\\circ h$, where $\\psi:\\R^m\\to \\oR$ is a CPLQ function and   $h:\\R^n\\to \\R^m$ is a ${\\cal C}^2$ function, and the \nbasic constraint qualification \n\\begin{equation}\\label{bcq}\nN_{\\ss \\dom \\psi}\\big(h(\\ox)\\big)\\cap \\ker \\nabla h(\\ox)^*=\\{0\\}\n\\end{equation}\nis satisfied. \n\n\nRecall that a function $f:\\R^n\\to \\oR$ is called semidifferentiable at $\\ox$ if it is finite at $\\ox$ and the ``$\\liminf$\" in \\eqref{fsud} is a ``$\\lim$.\"\nFurthermore, $f$ is said to be twice semidifferentiable at $\\ox$ if it is semidifferentiable at $\\ox$ and  the ``$\\liminf$\" in \\eqref{ssud} is a ``$\\lim$.\"\n\n\nRecall that for  the current triple $(\\xk, \\lk,\\rok)\\in \\R^n\\times \\R^m\\times (0,\\infty)$, the next primal iterate $\\xkk$ in the ALM is computed as an optimal solution to \n  the subproblem\n\\begin{equation}\\label{conv1}\n\\mini \\L(x, \\lk, \\rok)\\quad \\mbox{subject to}\\quad x\\in \\Th.\n\\end{equation} \nThe next dual iterate $\\lkk$ is  then calculated as the Lagrange multiplier associated with the optimal solution $\\xkk$ to the subproblem \\eqref{conv1}, namely  \n\\begin{equation}\\label{lkk}\n \\lkk := \\nabla\\menvk\\big(\\Phi(\\xkk)+ \\rho_k^{-1}{\\lk}\\big).\n\\end{equation}\n Since an optimal solution  $\\xkk$ to the subproblem \\eqref{conv1} always satisfies the {\\em stationary}  condition\n\\begin{equation}\\label{conv2}\n0\\in \\nabla_x\\L(\\xkk, \\lk, \\rok) +N_\\Th(\\xkk),\n\\end{equation} \nand since it is often difficult to find an exact solution to \\eqref{conv1} in practice, it is more convenient to look for a solution $\\xkk$ satisfying the {\\em approximate} stationary condition\n\\begin{equation}\\label{xkk}\n\\dist\\big(-\\nabla_x\\L(\\xkk, \\lk, \\rok),N_\\Th(\\xkk)\\big)\\leq \\ek,\n\\end{equation} \nwhere $\\ek\\geq 0$ is a   tolerance parameter.\n\n\n\\begin{Algorithm}[augmented Lagrangian method]\\label{Alg1}\nChoose $(x^0,\\lambda^0)\\in\\Th\\times \\R^m$ and $\\bar\\rho>0$. Pick a sequence of tolerance parameters $\\{\\ek\\c$ with $\\ek>0$ for all $k$ and $\\ek\\to   0$ as $k\\to\\infty$ and a sequence of $\\{\\rok\\c$ such that  $\\rok\\ge \\bar\\rho$ for all $k$ and set $k:=0$. Then\n\\begin{itemize}[noitemsep,topsep=0pt]\n\\item [{\\rm (1)}] if $(\\xk,\\lk)$ satisfies a suitable termination criterion, stop;\n\\item[{\\rm (2)}]  otherwise, choose  $\\xkk$ such that  \\eqref{xkk} holds and then update the Lagrange multiplier  $\\lkk$ by \\eqref{lkk};\n\\item[{\\rm (3)}] set $k\\leftarrow k+1$ and go to Step~1.\n\\end{itemize}\n\\end{Algorithm}\n\nThus, in the rest of this paper, we assume further that the  polyhedral convex set  $\\Th$ has a representation of the form \n\\begin{equation}\\label{Th}\n\\Th:=\\big\\{x\\in\\R^n\\,|\\, Bx = b\\big \\},\n\\end{equation} \nwhere $B$ is a $s\\times n$ matrix and $b$ is a vector in $\\R^s$. Furthermore, our convergence analysis of the ALM  requires to consider the \\textit{residual function} $R: \\R^n\\times \\R^m \\to \\R$ of the KKT system \\eqref{vs}, given by\n\\begin{equation}\\label{res}\nR(x, \\lm) : =  \\dist\\big(-\\nabla_x L(x, \\lm), N_\\Th(x)\\big)+\\big\\|\\Phi(x)-{\\rm{prox}}_g\\big(\\Phi(x)+\\lm\\big)\\big\\|, \\quad (x, \\lm) \\in \\R^n\\times \\R^m.\n\\end{equation}\n\n\n convergence of generalized equations with non-isolated solutions. To this end, let  $\\sigma:\\R^n\\times \\R^m\\to \\R_+$ be a function that satisfies  the condition\n \\begin{equation}\\label{error}\n \\sigma(x,\\lm)=o\\big(R(x,\\lm)\\big)\\quad \\mbox{as}\\;\\;(x,\\lm)\\to (\\ox,\\olm),\n \\end{equation}\nwhere $R$ is the residual function \\eqref{res} and $(\\ox,\\olm)$ is a solution to the KKT system \\eqref{vs}.\n","units":[{"start":7530,"end":7968,"start_line":215,"end_line":219,"raw_tex":"Given twice continuously differentiable functions   $\\ph:\\R^n\\to \\R$ and  $\\Phi:\\R^n\\to \\R^m$,  a convex polyhedral set  $\\Th$ in $\\R^n$, and a convex piecewise linear-quadratic (CPLQ) function $g:\\R^m\\to \\oR$,\n we aim to present a local convergence analysis of the ALM for  the composite optimization problem \n\\begin{equation}\\label{comp}\n\\mbox{minimize}\\;\\;\\ph(x)+g\\big(\\Phi(x)\\big)\\quad \\mbox{subject to}\\;\\; x\\in \\Th.\n\\end{equation} \n","sha256_utf8":"ec3bf2accbeb796338f368c588ce9b2d319cd76aba05dfd4f18d48493fcf67e2"},{"start":8916,"end":9921,"start_line":227,"end_line":240,"raw_tex":"The augmented Lagrangian $\\L:\\R^n\\times \\R^m\\times (0,\\infty)\\to \\R$ associated with the composite problem \\eqref{comp} is defined  by \n\\begin{equation}\\label{aug}\n\\L(x,\\lm,\\rho):=\\ph(x)+e_{ {1}/{\\rho}} g \\big(\\Phi(x)+\\rho^{-1}\\lm\\big)-\\sm\\rho^{-1}\\|\\lm\\|^2\n\\end{equation}\nwith $e_{1/\\rho} g$ signifying  the Moreau envelope of   $g$, given  by \n\\begin{equation*}\\label{Menv}\ne_{1/\\rho} g(y):=\\inf_{z\\in \\R^m}\\big\\{g(z)+\\frac{\\rho}{2}\\|y-z\\|^2\\big\\}, \\quad y\\in \\R^m.\n\\end{equation*}\nIt is well-known that the augmented Lagrangian $\\L$ is continuously differentiable with respect to both $x$ and $\\lm$ and is nondecreasing with the respect to $\\rho$, the {\\em penalty} parameter of the augmented Lagrangian. \nAt  each iteration for a given pair $(\\lm,\\rho)\\in \\R^m\\times (0,\\infty)$,  our proposed  ALM for  the  composite problem \\eqref{comp}\nrequires solving the {\\em partially} augmented problem \n\\begin{equation}\\label{augp}\n\\mini \\L(x, \\lm, \\rho)\\quad \\mbox{subject to}\\;\\; x\\in \\Th.\n\\end{equation} \n","sha256_utf8":"e1d931ab1360ca165290e6dff66d97899c4ff365650f37c5cc6c7801e13591ec"},{"start":14622,"end":15893,"start_line":277,"end_line":286,"raw_tex":"In this section, we first briefly review basic constructions of variational analysis and generalized differentiation employed in the paper. In what follows,    we \n denote by $\\B$ the closed unit ball in the space in question and by $\\B_r(x):=x+r\\B$ the closed ball centered at $x$ with radius $r>0$. \n In the  product space $\\R^n\\times \\R^m$, we use the norm $\\|(w,u)\\|=\\sqrt{\\|w\\|^2+\\|u\\|^2}$ for any $(w,u)\\in \\R^n\\times \\R^m$.\n Given a nonempty set $C\\subset\\R^n$, the symbols $\\inte C$,  $\\cone C$, and $\\span C$ signify its interior, conic hull, and the   linear space generated by  $C$, respectively. \n For any set $C$ in $\\R^n$, its indicator function is defined by $\\dd_C(x)=0$ for $x\\in C$ and $\\dd_C(x)=\\infty$ otherwise. We denote\n by $P_C$ the projection mapping onto $C$ and  by $\\dist(x,C)$  the distance between $x\\in \\R^n$ and a set $C$.\n For a vector $w\\in \\R^n$, the subspace $\\{tw |\\, t\\in \\R\\}$ is denoted by $[w]$. \nWe write $x(t)=o(t)$ with $x(t)\\in \\R^n$ and $t>0$ to mean that ${\\|x(t)\\|}/{t}$ goes to $0$ as $t\\dn 0$.\nFinally, we denote by $\\R_+$ (respectively,  $\\R_-$) the set of non-negative (respectively, non-positive) real numbers.\nFor an $n\\times m$ matrix $A$, we denote by $\\ker A$ and $\\rge A$ its null and range spaces, respectively.\n","sha256_utf8":"05e44e4d2ae58771a47aeddcd12b42ef23e5d5ae25c4d3332398d743fd83e068"},{"start":15894,"end":16933,"start_line":288,"end_line":303,"raw_tex":"A sequence    $\\{C^k\\b$ of subsets in $\\R^n$ is called convergent to a set $C\\subset\\R^n$ as $k\\to \\infty$ if $C$ is closed and\n\\begin{equation*}\n\\lim_{k\\to \\infty}{\\rm dist}(w,C^k)={\\rm dist}(w,C)\\;\\mbox{ for all }\\;w\\in\\R^n.\n\\end{equation*}\nGiven a nonempty set $C\\subset\\R^n$ with $\\ox\\in C$, the  tangent cone $T_ C(\\ox)$ to $C$ at $\\ox$ is defined by\n\\begin{equation*}\\label{2.5}\nT_C(\\ox)=\\big\\{w\\in\\R^n|\\;\\exists\\,t_k{\\dn}0,\\;\\;w_k\\to w\\;\\;\\mbox{ as }\\;k\\to\\infty\\;\\;\\mbox{with}\\;\\;\\ox+t_kw_k\\in C\\big\\}.\n\\end{equation*}\nWhen $C$ is convex, the  {normal cone} to $C$ at $\\ox\\in C$ in the sense of convex analysis  is given by\n\\begin{equation*}\nN_C(\\ox)=\\big\\{v\\in\\R^n\\;\\big|\\; \\la v,x-\\ox\\ra\\le 0\\quad \\mbox{for all}\\;\\; x\\in C\\big\\}.\n\\end{equation*}\nFor a convex  function $f:\\R^n \\to \\oR:=(-\\infty,\\infty]$  with $\\ox\\in \\dom f$, its   subdifferential at $\\ox$   is defined  by \n\\begin{equation*}\\label{sub}\n\\partial f(\\ox)=\\big\\{v\\in\\R^n\\;\\big|\\;  \\la v,x-\\ox\\ra\\le f(x)-f(\\ox)\\;\\;\\mbox{for all}\\;x\\in \\R^n\\big\\}.\n\\end{equation*}\n","sha256_utf8":"7ac4752a82697888da7bd7fad1a2f1dc44333a141550b14b10028dc407bd9dea"},{"start":16934,"end":19286,"start_line":305,"end_line":339,"raw_tex":"A sequence $\\{f^k\\b$ of functions $f^k:\\R^n\\to \\oR$ is said to {\\em epi-converge} to a function $f:\\R^n\\to [-\\infty,\\infty]$ if we have $\\epi f^k\\to \\epi f$ as $k\\to \\infty$.\nWe denote by $f^k\\overset {\\ss \\mbox{e}}{\\to} f$ the  epi-convergence of the sequence $\\{f^k\\b$ to $f$.\nGiven  a function  $f:\\R^n \\to \\oR$ and  a point $\\ox$ with $f(\\ox)$ finite, the subderivative function $\\d f(\\ox)\\colon\\R^n\\to[-\\infty,\\infty]$ is defined by\n\\begin{equation}\\label{fsud}\n{\\mathrm d}f(\\ox)(w)=\\liminf_{\\substack{\n   t\\dn 0 \\\\\n  w'\\to w\n  }} {\\frac{f(\\ox+tw')-f(\\ox)}{t}}.\n\\end{equation}\n The critical cone of $f$ at $\\ox$ for $\\ov$ with $(\\ox,\\ov)\\in \\gph \\sub f$ is defined by \n$$\n{K_f}(\\ox,\\ov)=\\big\\{w\\in \\R^n\\big|\\;\\la\\bar v,w\\ra=\\d f(\\ox)(w)\\big\\}.\n$$\nWhen $f=\\dd_C$, namely the indicator function of $C$, the critical cone of $\\dd_C$ at $\\ox$ for $\\ov$ is denoted by $K_C(\\ox,\\ov)$. In this case, the above definition of the critical cone of a function \nboils down to  the well-known concept of the critical cone of a set (see \\cite[page~109]{dr}), namely $K_C(\\ox,\\ov)=T_C(\\ox)\\cap [\\ov]^\\perp$.\nDefine the parametric  family of \nsecond-order difference quotients for $f$ at $\\ox$ for $\\ov\\in \\R^n$ by \n\\begin{equation*}\\label{lk01}\n\\Delta_t^2 f(\\bar x , \\ov)(w)=\\dfrac{f(\\ox+tw)-f(\\ox)-t\\langle \\ov,\\,w\\rangle}{\\frac{1}{2}t^2}\\quad\\quad\\mbox{with}\\;\\;w\\in \\R^n, \\;\\;t>0.\n\\end{equation*}\nIf $f(\\ox)$ is finite, then the {second subderivative} of $f$ at $\\ox$ for $\\ov$   is given by \n\\begin{equation*}\\label{ssd}\n\\d^2 f(\\bar x , \\ov)(w)= \\liminf_{\\substack{\n   t\\dn 0 \\\\\n  w'\\to w\n  }} \\Delta_t^2 f(\\ox , \\ov)(w'),\\;\\; w\\in \\R^n.\n\\end{equation*}\nThe second subderivative of $f$ at $\\ox$ (without mention of $\\ov$) is defined by \n\\begin{equation}\\label{ssud}\n\\d^2 f(\\ox)(w)=\\liminf_{\\substack{t\\dn 0\\\\\nw'\\to w}} \\frac{f(\\ox+tw')-f(\\ox)-t\\d f(\\ox)(w')}{\\sm t^2},\\quad w\\in \\R^n.\n\\end{equation}\nFollowing \\cite[Definition~13.6]{rw}, a function $f:\\R^n \\to \\oR$ is said to be {twice epi-differentiable} at $\\bar x$ for $\\ov\\in\\R^n$, with $f(\\ox) $ finite, \nif the sets $\\epi \\Delta_t^2 f(\\bar x , \\ov)$ converge to $\\epi \\d^2 f(\\bar x,\\ov)$ as $t\\downarrow 0$. \nIn what follows, we refer to the second subderivative of $f$ at $\\bar x$ for $\\ov$ by the {\\em second-order epi-derivative} of $f$ when it is twice epi-differentiable at   $\\bar x$ for $\\ov$.\n","sha256_utf8":"a0d8aa2b15b72b1ad1095ceabd37106583aebf6bf6ad0bcba4fc70f336c7a456"},{"start":19288,"end":20278,"start_line":341,"end_line":354,"raw_tex":"Recall  that a function $g:\\R^m\\to \\oR$ is called  piecewise linear-quadratic if $\\dom g = \\cup_{i=1}^{s} C_i$ with $s\\in \\N$ and $C_i $ being polyhedral convex  sets for $i = 1, \\ldots, s$, and if $g$ has a representation of the form\n\\begin{equation} \\label{PWLQ}\ng(z) = \\sm \\langle A_i z ,z \\rangle + \\langle a_i ,z \\rangle + \\alpha_i  \\quad \\mbox{for all} \\quad  z \\in C_i,\n\\end{equation}\nwhere $A_i$ is an $m \\times m$ symmetric matrix, $a_i\\in \\R^m$, and $\\alpha_i\\in \\R$ for all $i = 1, \\cdots, s$. Take $\\oz\\in \\dom g$ and define the {\\em active} indices  of the domain of $g$ at $\\oz$ by\n\\begin{equation}\\label{act}\nI(\\oz)=\\big \\{i\\in \\{1,\\ldots, m\\} |\\, \\oz\\in C_i\\big \\}.\n\\end{equation}\nIf in addition $g$ is convex, \nit follows   from \\cite[page~487]{rw} that \nif  $\\oz\\in \\dom g$, the subdifferential of $g$ at  $\\oz$ can be calculated by \n\\begin{equation}\\label{sub}\n\\sub g(\\oz)=\\bigcap_{i\\in I(\\oz)} \\big \\{v\\in \\R^m |\\, v-A_i \\oz- a_i\\in N_{ C_i}(\\oz)\\big \\}.\n\\end{equation}\n","sha256_utf8":"0941f0498ff46f11d09c57cc82e42e27ce665b9c29a86a117e3829b5ef69e535"},{"start":21537,"end":22787,"start_line":383,"end_line":401,"raw_tex":"The   Karush-Kuhn-Tucker (KKT)   system associated with the composite problem \\eqref{comp} is given by \n\\begin{equation}\\label{vs} \n0\\in \\nabla_x L(x,\\lm)+N_\\Th(x),\\;\\;\\lm\\in \\sub g(\\Phi(x)),\n\\end{equation}\nwhere $L(x,\\lm)= \\ph(x)+\\la \\Phi(x),\\lm\\ra$ for any   $(x,\\lm)\\in \\R^n\\times \\R^m$. \nGiven a point $\\ox\\in\\R^n$, we define the set of {  Lagrange multipliers} of the KKT system \\eqref{vs} associated with $\\ox$ by\n\\begin{equation}\\label{laset}\n\\Lambda(\\ox):=\\big\\{\\lm\\in\\R^m\\;\\big|\\; 0\\in \\nabla_x L(\\ox,\\lm)+N_\\Th(\\ox),\\;\\lm\\in \\sub g(\\Phi(\\ox))\\big\\}.\n\\end{equation}\nIf  $(\\ox,\\olm)$ is a solution to the KKT  system \\eqref{vs},   we clearly have   $\\olm\\in\\Lambda(\\ox)$.\n Given  a solution $(\\ox,\\olm)$  to \\eqref{vs}, the second-order sufficient condition (SOSC)\nfor the composite problem \\eqref{comp} at $(\\ox,\\olm)$ is formulated by \n\\begin{equation}\\label{sosc}\n\\langle\\nabla_{xx}^2L(\\bar x,\\olm)w,w\\rangle+\\d^2g (\\Phi(\\bar x),\\olm) (\\nabla \\Phi(\\ox)w )>0\\quad \\mbox{for all }\\; w\\in \\D\\setminus\\{0\\},\n\\end{equation}\nwhere the convex cone $\\D$ is defined by \n \\begin{equation}\\label{coned}\n \\D:=K_\\Th\\big(\\ox,-\\nabla_x L(\\ox,\\olm)\\big)\\cap \\big\\{w\\in \\R^n\\big|\\; \\nabla \\Phi(\\ox)w\\in K_g \\big(\\Phi(\\ox),\\olm \\big)\\big\\}.\n \\end{equation}\n","sha256_utf8":"69e82eb1b6dd68d312f418a87010dd2708ea57f20daff1d1d55203714898e30c"},{"start":23093,"end":24019,"start_line":405,"end_line":414,"raw_tex":"a result, obtained recently in \\cite{s20}, showing that it yields an error bound estimate for the KKT system \\eqref{vs}, which is central to our   local convergence analysis  in this paper. To this end,  recall that, given a constant $r>0$,    the proximal mapping of  a convex function $g:\\R^m\\to \\oR$ is defined by \n\\begin{equation*}\\label{pr}\n{\\rm{prox}}_{rg}(x):=\\underset{z\\in \\R^n}{\\argmin} \\big\\{g(z)+\\rsm\\|x-z\\|^2\\big\\},\\quad x\\in\\R^m.\n\\end{equation*}\nIn what follows, when $r=1$, the proximal mapping of $g$ will be denoted by ${\\rm{prox}_g}$. For such a convex function, \nit is known that for any $x\\in \\R^m$ we have \n\\begin{equation}\\label{prm}\n{\\rm{prox}}_{rg}(x)=\\big(I+r\\sub g\\big)^{-1}(x) \\quad \\mbox{and}\\quad \\nabla e_rg(x)=\\big(rI+(\\sub g)^{-1}\\big)^{-1}(x),\n\\end{equation}\nwhere $I$ stands for the $m\\times m$ identity matrix; see \\cite[Proposition~12.19]{rw} and \\cite[Theorem~2.26]{rw} for more details. \n","sha256_utf8":"a3b18c6f796e8e8b27fe85d15d365d68dc812c56b3f7a71cf939d24f1661c64c"},{"start":25969,"end":26408,"start_line":439,"end_line":445,"raw_tex":"A function $f:\\R^n\\to \\oR$ is called fully amenable at $\\ox\\in \\dom f$ if there is a neighborhood ${\\cal O}$ of $\\ox$ on which \n$f$ can be represented in the form $f=\\psi\\circ h$, where $\\psi:\\R^m\\to \\oR$ is a CPLQ function and   $h:\\R^n\\to \\R^m$ is a ${\\cal C}^2$ function, and the \nbasic constraint qualification \n\\begin{equation}\\label{bcq}\nN_{\\ss \\dom \\psi}\\big(h(\\ox)\\big)\\cap \\ker \\nabla h(\\ox)^*=\\{0\\}\n\\end{equation}\nis satisfied. \n","sha256_utf8":"80e655996c2b3fb754867b1fa908629fff395e47e5d56b7dde95bf4f43f3bc92"},{"start":32774,"end":33085,"start_line":536,"end_line":537,"raw_tex":"Recall that a function $f:\\R^n\\to \\oR$ is called semidifferentiable at $\\ox$ if it is finite at $\\ox$ and the ``$\\liminf$\" in \\eqref{fsud} is a ``$\\lim$.\"\nFurthermore, $f$ is said to be twice semidifferentiable at $\\ox$ if it is semidifferentiable at $\\ox$ and  the ``$\\liminf$\" in \\eqref{ssud} is a ``$\\lim$.\"\n","sha256_utf8":"1773ba9ca83097660e14f087928954be82359f9d9956255600fa1adb7bf9a2c0"},{"start":57280,"end":58400,"start_line":829,"end_line":846,"raw_tex":"Recall that for  the current triple $(\\xk, \\lk,\\rok)\\in \\R^n\\times \\R^m\\times (0,\\infty)$, the next primal iterate $\\xkk$ in the ALM is computed as an optimal solution to \n  the subproblem\n\\begin{equation}\\label{conv1}\n\\mini \\L(x, \\lk, \\rok)\\quad \\mbox{subject to}\\quad x\\in \\Th.\n\\end{equation} \nThe next dual iterate $\\lkk$ is  then calculated as the Lagrange multiplier associated with the optimal solution $\\xkk$ to the subproblem \\eqref{conv1}, namely  \n\\begin{equation}\\label{lkk}\n \\lkk := \\nabla\\menvk\\big(\\Phi(\\xkk)+ \\rho_k^{-1}{\\lk}\\big).\n\\end{equation}\n Since an optimal solution  $\\xkk$ to the subproblem \\eqref{conv1} always satisfies the {\\em stationary}  condition\n\\begin{equation}\\label{conv2}\n0\\in \\nabla_x\\L(\\xkk, \\lk, \\rok) +N_\\Th(\\xkk),\n\\end{equation} \nand since it is often difficult to find an exact solution to \\eqref{conv1} in practice, it is more convenient to look for a solution $\\xkk$ satisfying the {\\em approximate} stationary condition\n\\begin{equation}\\label{xkk}\n\\dist\\big(-\\nabla_x\\L(\\xkk, \\lk, \\rok),N_\\Th(\\xkk)\\big)\\leq \\ek,\n\\end{equation} \nwhere $\\ek\\geq 0$ is a   tolerance parameter.\n","sha256_utf8":"53a5e8fabfcf52f8a23f0dfb5a5c3e837653e94ce25fd9deef7d17848a598a55"},{"start":58433,"end":59096,"start_line":848,"end_line":855,"raw_tex":"\\begin{Algorithm}[augmented Lagrangian method]\\label{Alg1}\nChoose $(x^0,\\lambda^0)\\in\\Th\\times \\R^m$ and $\\bar\\rho>0$. Pick a sequence of tolerance parameters $\\{\\ek\\c$ with $\\ek>0$ for all $k$ and $\\ek\\to   0$ as $k\\to\\infty$ and a sequence of $\\{\\rok\\c$ such that  $\\rok\\ge \\bar\\rho$ for all $k$ and set $k:=0$. Then\n\\begin{itemize}[noitemsep,topsep=0pt]\n\\item [{\\rm (1)}] if $(\\xk,\\lk)$ satisfies a suitable termination criterion, stop;\n\\item[{\\rm (2)}]  otherwise, choose  $\\xkk$ such that  \\eqref{xkk} holds and then update the Lagrange multiplier  $\\lkk$ by \\eqref{lkk};\n\\item[{\\rm (3)}] set $k\\leftarrow k+1$ and go to Step~1.\n\\end{itemize}\n\\end{Algorithm}","sha256_utf8":"5f89db04668bdd41a3c9893ba6b400a27904f6b609011e603d5ef7f17bbe4446"},{"start":65793,"end":66429,"start_line":953,"end_line":960,"raw_tex":"Thus, in the rest of this paper, we assume further that the  polyhedral convex set  $\\Th$ has a representation of the form \n\\begin{equation}\\label{Th}\n\\Th:=\\big\\{x\\in\\R^n\\,|\\, Bx = b\\big \\},\n\\end{equation} \nwhere $B$ is a $s\\times n$ matrix and $b$ is a vector in $\\R^s$. Furthermore, our convergence analysis of the ALM  requires to consider the \\textit{residual function} $R: \\R^n\\times \\R^m \\to \\R$ of the KKT system \\eqref{vs}, given by\n\\begin{equation}\\label{res}\nR(x, \\lm) : =  \\dist\\big(-\\nabla_x L(x, \\lm), N_\\Th(x)\\big)+\\big\\|\\Phi(x)-{\\rm{prox}}_g\\big(\\Phi(x)+\\lm\\big)\\big\\|, \\quad (x, \\lm) \\in \\R^n\\times \\R^m.\n\\end{equation}\n","sha256_utf8":"c05f54e30c75b6847ad6396287a02521d1d7af0820aa0b469ef26b3922de4467"},{"start":85121,"end":85514,"start_line":1192,"end_line":1196,"raw_tex":" convergence of generalized equations with non-isolated solutions. To this end, let  $\\sigma:\\R^n\\times \\R^m\\to \\R_+$ be a function that satisfies  the condition\n \\begin{equation}\\label{error}\n \\sigma(x,\\lm)=o\\big(R(x,\\lm)\\big)\\quad \\mbox{as}\\;\\;(x,\\lm)\\to (\\ox,\\olm),\n \\end{equation}\nwhere $R$ is the residual function \\eqref{res} and $(\\ox,\\olm)$ is a solution to the KKT system \\eqref{vs}.\n","sha256_utf8":"9181051fe87d434bf9d08b18011cfa15462e7704952599873004d77cc02a0587"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{Theorem}[primal-dual linear convergence of ALM]\\label{locpd} \n Let $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and  let the SOSC \\eqref{sosc} hold at $(\\ox,\\olm)$.\n Then there exist positive constants $\\bar\\gg$, $\\bar\\tau$, and $\\bar\\varrho$ such that for any starting point $(x^0,\\lm^0)\\in \\B_{\\bar\\gg}(\\ox,\\olm)\\cap\\big(\\Th\\times \\R^m\\big)$\n there is a primal-dual sequence   $\\{(\\xk,\\lk)\\c$,  generated by Algorithm~{\\rm\\ref{Alg1}} with $\\rok \\ge \\bar\\varrho$ and   $\\ek=\\sigma(\\xk,\\lk)$ for all $k$, where $\\sigma$ is defined by \\eqref{error}, satisfying the  estimate\n \\begin{equation}\\label{cone}\n \\|x^{k+1}-x^k\\|+\\|\\lm^{k+1}-\\lk\\|\\le \\bar\\tau R(x^k,\\lk),\n \\end{equation}\n where $R$ is the residual function defined by  \\eqref{res}.\n  Moreover, every such a sequence is convergent to $(\\ox,\\hat\\lambda)$ for some $\\hat\\lambda\\in\\Lm(\\ox)$, and its rate of convergence is linear.\n  Furthermore, if $\\rok\\to \\infty$,  the rate of convergence  of $\\{(\\xk,\\lk)\\c$ is superlinear. \n\\end{Theorem}","units":[{"start":85556,"end":86570,"start_line":1198,"end_line":1208,"raw_tex":"\\begin{Theorem}[primal-dual linear convergence of ALM]\\label{locpd} \n Let $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and  let the SOSC \\eqref{sosc} hold at $(\\ox,\\olm)$.\n Then there exist positive constants $\\bar\\gg$, $\\bar\\tau$, and $\\bar\\varrho$ such that for any starting point $(x^0,\\lm^0)\\in \\B_{\\bar\\gg}(\\ox,\\olm)\\cap\\big(\\Th\\times \\R^m\\big)$\n there is a primal-dual sequence   $\\{(\\xk,\\lk)\\c$,  generated by Algorithm~{\\rm\\ref{Alg1}} with $\\rok \\ge \\bar\\varrho$ and   $\\ek=\\sigma(\\xk,\\lk)$ for all $k$, where $\\sigma$ is defined by \\eqref{error}, satisfying the  estimate\n \\begin{equation}\\label{cone}\n \\|x^{k+1}-x^k\\|+\\|\\lm^{k+1}-\\lk\\|\\le \\bar\\tau R(x^k,\\lk),\n \\end{equation}\n where $R$ is the residual function defined by  \\eqref{res}.\n  Moreover, every such a sequence is convergent to $(\\ox,\\hat\\lambda)$ for some $\\hat\\lambda\\in\\Lm(\\ox)$, and its rate of convergence is linear.\n  Furthermore, if $\\rok\\to \\infty$,  the rate of convergence  of $\\{(\\xk,\\lk)\\c$ is superlinear. \n\\end{Theorem}","sha256_utf8":"f00825ffe5fb50a1d09e85d1ca01cf7c8fe27f7e4f56fd499759d22d4d352f92"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{Proposition}[second-order variational properties of CPLQ] \\label{sop} Assume that $g:\\R^m\\to \\oR$ is a CPLQ function with representation \\eqref{PWLQ}\nand that $\\oz\\in \\dom g$ and $\\ov\\in \\sub g(\\oz)$. Set $\\ov_i:=\\ov -A_i \\oz -a_i$ for all $i\\in I(\\oz)$. \nThen the following conditions hold:\n \\begin{itemize}[noitemsep,topsep=0pt]\n \\item [{\\rm (a)}]   the critical cone of $g$ at $\\oz$ for $\\ov$ can be calculated by $K_g(\\oz,\\ov)=N_{\\sub g(\\oz)}(\\ov)$. Moreover, we have \n\\begin{equation}\\label{cc2}\n  \\d^2 g(\\oz , \\ov ) (w)=\\d^2 g(\\oz)(w)+\\dd_{K_g(\\oz,\\ov)}(w), \\quad w\\in \\R^m.\n\\end{equation}\n \\item [{\\rm (b)}] the function $g$ is twice epi-differentiable at $\\ox$ for $\\ov$ and its second-order epi-derivative  at this point can be calculated by \n\\begin{equation}\\label{pwfor}\n \\d^2 g(\\oz , \\ov ) (w)  \n=\\begin{cases}\n\\la A_i w, w\\ra&\\mbox{if}\\;\\; w\\in {K}_{C_i}(\\oz,\\ov_i),\\\\\n\\infty&\\mbox{otherwise}.\n\\end{cases}\n\\end{equation}\nMoreover, we have \n\\begin{equation}\\label{domcr}\nK_g(\\oz,\\ov)=\\bigcup_{i\\in I(\\oz)} {K}_{C_i}(\\oz,\\ov_i).\n\\end{equation}\n  \\end{itemize}\n\\end{Proposition}\n\n\\begin{Theorem}[error bound via second-order optimality conditions]\\label{sooc} \n Assume that $(\\ox,\\olm)$ is a solution to the KKT system \\eqref{vs}. If the SOSC \\eqref{sosc} holds at $(\\ox,\\olm)$, then\n there are positive constants $\\gg_1$ and $\\kappa_1$ for which  the error bound estimate\n\\begin{equation}\\label{subr3}\n\\|x-\\ox\\|+\\dist\\big(\\lm,\\Lambda(\\ox)\\big)\\le\\kappa_1\\Big(\\dist\\big(-\\nabla_x L(x,\\lm),N_\\Th(x)\\big)+\\| \\Phi(x)-{\\rm{prox}}_g\\big(\\lm+ \\Phi(x)\\big)\\|\\Big)\n\\end{equation}\nholds for any $(x,\\lm)\\in\\B_{\\gg_1}(\\ox,\\olm)$.\n \\end{Theorem}\n\n\\begin{Proposition}[properties of the augmented Lagrangian]\\label{paug}\nAssume that  $(x, \\lm, \\rho)\\in \\R^n\\times \\R^m\\times (0, \\infty)$ and $\\ox\\in \\R^n$. Then the following conditions hold for the  augmented Lagrangian   \\eqref{aug}:\n\\begin{enumerate}[noitemsep,topsep=0pt]\n\\item the function $x \\mapsto \\L(x, \\lm, \\rho)$ is $\\mathcal{C}^1$ and fully amenable at $\\ox$;\n\\item the function $\\lm \\mapsto \\L(x, \\lm, \\rho)$ is concave;\n\\item the function $\\rho \\mapsto \\L(x, \\lm, \\rho)$ is nondecreasing.\n\\end{enumerate}\n\\end{Proposition}\n\n\\begin{proof} Let $(x, \\lm, \\rho)\\in \\R^n\\times \\R^m\\times (0, \\infty)$ and $\\ox\\in \\R^n$. To justify (a), it follows from \\cite[Proposition~12.30]{rw} that the envelope function $\\menv$ is CPLQ. \nMoreover, the envelope function $\\menv$  is continuously differentiable on $\\R^m$ and thus the function $x\\mapsto e_{ {1}/{\\rho}} g \\big(\\Phi(x)+\\rho^{-1}\\lm\\big)$\nis $\\mathcal{C}^1$ and fully amenable at $\\ox$. In this case, the basic qualification condition \\eqref{bcq} holds automatically at any $\\ox\\in \\R^n$ since $\\dom \\menv=\\R^m$. Clearly the mapping $x\\mapsto \\ph(x)$\nis fully amenable at $\\ox$ since $\\ph$ is ${\\cal C}^2$. Combining these and remembering that the sum of two ${\\cal C}^1$ and fully amenable functions is fully amenable (cf. \\cite[Exercise~10.22]{rw})\nindicate that the augmented Lagrangian \\eqref{aug} is fully amenable at $\\ox$. Parts (b) and (c) were established in \\cite[Exercise~11.56]{rw}.\n\\end{proof}\n\n\\begin{Proposition}[propagation of stationary conditions]\\label{fopag}\nLet $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs}. Then for any $\\rho>0$,  the following conditions are satisfied:\n\\begin{enumerate}[noitemsep,topsep=0pt]\n\\item we always have  $\\L(\\ox,\\olm,\\rho)=\\ph(\\ox)+g\\big(\\Phi(\\ox)\\big)$;\n\\item $\\ox$ is a stationary point of the augmented  problem \n\\begin{equation}\\label{ap}\n\\mini\\; \\L(x,\\olm,\\rho)\\quad \\mbox{subject to}\\;\\; x\\in \\Th.\n\\end{equation}\n\\end{enumerate}\n\\end{Proposition}\n\n\\begin{proof} Given $\\rho>0$, it follows from the first equality in \\eqref{prm} and  $\\olm\\in \\sub g\\big(\\Phi(\\ox)\\big)$ that \n $$\n \\prox\\big(\\Phi(\\ox)+\\rho^{-1}\\olm\\big)=\\Phi(\\ox),\n $$ which clearly proves (a).\nWe proceed now with the proof of  (b). Since $g$ is convex, we conclude from the second equality in \\eqref{prm} that the condition $\\olm\\in \\sub g\\big(\\Phi(\\ox)\\big)$ is equivalent to $\\nabla(e_{ {1}/{\\rho}} g)\\big(\\Phi(\\ox)+\\rho^{-1}\\olm\\big)=\\olm$\nfor any $\\rho>0$.  Thus we get \n\\begin{equation}\\label{kkt2}\n\\nabla_x \\L(\\ox,\\olm,\\rho)=\\nabla \\ph(\\ox)+ \\nabla \\Phi(\\ox)^*\\nabla(e_{ {1}/{\\rho}} g)(\\Phi(\\ox)+\\rho^{-1}\\olm)=\\nabla_x L(\\ox,\\olm).\n\\end{equation}\nCombining these and \\eqref{vs} justifies  (b).\n\\end{proof}\n\n\\begin{Proposition}[quadratic growth condition for augmented Lagrangians]\\label{2ndgr}\nLet $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and set $\\ov:= \\nabla_x L(\\ox,\\olm)$. Then for any $\\rho>0$,  the following conditions are equivalent: \n\\begin{enumerate}[noitemsep,topsep=0pt]\n\\item    there exists a constant $\\bar\\rho>0$ such that for any $\\rho\\ge\\bar\\rho$ the second-order condition \n\\begin{equation}\\label{eq26}\n\\d^2_x\\L\\big((\\ox,\\olm,\\rho),\\ov\\big)(w)>0\\;\\mbox{ for all }\\;w\\in K_\\Th (\\ox,-\\ov ) \\setminus\\{0\\}\n\\end{equation}\nholds;\n \\item  there exist positive constants $\\bar\\rho$, $\\gamma$, and $\\ell$ such that for any $\\rho\\ge\\bar \\rho$, the quadratic growth condition \n \\begin{equation}\\label{eq16}\n\\L(x,\\olm,\\rho)\\ge \\ph(\\ox)+g\\big(\\Phi(\\ox)\\big)+\\ell\\| x-\\bar x\\|^2\\quad \\mbox{ for all }\\;x\\in\\B_\\gg (\\ox)\\cap \\Th\n\\end{equation}\nholds. \n\\end{enumerate}\n\\end{Proposition}\n\n\\begin{proof} Pick $\\rho>0$ and    $x\\in \\R^n$, and \n define the function $  f_\\rho(x) := \\L(x,\\olm, \\rho)+\\delta_\\Th(x)$.  Remember that $(\\ox,\\olm)$ is a solution to the KKT system  \\eqref{vs}.\nSince $\\partial   f_\\rho(\\ox) = \\nabla_x \\L(\\ox,\\olm,\\rho)+N_\\Th(\\ox)$, it follows from \\eqref{kkt2} that $0\\in \\sub f_\\rho(\\ox)$. \nEmploying \\cite[Proposition~13.19]{rw} and Proposition~\\ref{paug}(a), we conclude  that  \n\\begin{eqnarray}\n\\d^2  f_\\rho (\\ox, 0)(w) &=&\\d^2_x\\L\\big((\\ox,\\olm,\\rho),\\ov\\big)(w)+\\d^2\\delta_\\Th(\\ox, -\\ov)(w)\\nonumber \\\\\n&=&\\d^2_x\\L\\big((\\ox,\\olm,\\rho),\\ov\\big)(w)+\\delta_{K_\\Th(\\ox, -\\ov)}(w),\\label{sumr}\n\\end{eqnarray} \nwhere the formula for the second subderivative of $\\dd_\\Th$ was taken from \\cite[Example~3.4]{mmstams}.\n\nAfter this preparation,  assume that   (a)  holds. So we find   $\\bar \\rho>0$ for which \\eqref{eq26} is satisfied. \nThis, combined with \\eqref{sumr}, tells us that $\\d^2  f_{\\bar\\rho}(\\ox, 0)(w)>0$  for all $w\\in \\R^n\\setminus\\{0\\}$.\nEmploying now   \\cite[Theorem~13.24(c)]{rw}, we find positive constants  $\\gg$ and $\\ell$ for which the quadratic \ngrowth condition \n$$\n\\L(x,\\olm,\\bar \\rho)\\ge \\L(\\ox,\\olm,\\bar \\rho) +\\ell\\| x-\\bar x\\|^2 =\\ph(\\ox)+g\\big(\\Phi(\\ox)\\big)+\\ell\\| x-\\bar x\\|^2\n$$\nholds for all $x\\in\\B_\\gg (\\ox)\\cap \\Th$, where the last equality comes from Proposition~\\ref{fopag}(a). By Proposition~\\ref{paug}(c), the mapping $\\rho\\mapsto \\L(x,\\olm,  \\rho)$ is nondecereasing. Combining this with the above quadratic growth condition justifies \\eqref{eq16} for any $\\rho\\ge \\bar\\rho$.\n\nTurning now to the opposite implication, assume that (b) holds and then pick $\\rho\\in [\\bar\\rho, \\infty)$. By  Proposition~\\ref{fopag}(a), we have $f_\\rho(\\ox)= \\ph(\\ox)+g\\big(\\Phi(\\ox)\\big)$, which yields \n$$\nf_\\rho(x) \\ge f_\\rho(\\ox)+\\ell\\| x-\\bar x\\|^2\\quad \\mbox{ for all }\\;x\\in\\B_\\gg (\\ox).\n$$\nThis implies via the definition of the second subderivative that $\\d^2  f_{\\rho}(\\ox, 0)(w)>0$  for all $w\\in \\R^n\\setminus\\{0\\}$. Appealing now to \\eqref{sumr} verifies \\eqref{eq26} and hence ends the proof.\n\\end{proof}\n\n\\begin{Theorem}[second subderivatives of augmented Lagrangians]\\label{ssc} \nAssume that  $(x, \\lm, \\rho)\\in \\R^n\\times \\R^m\\times (0, \\infty)$ and $\\ox\\in \\R^n$. \nThen  the function $x\\mapsto\\L(x,\\lm,\\rho)$, defined via the augmented Lagrangian \\eqref{aug}, is twice epi-differentiable at $\\ox$ for $\\ov:=\\nabla_x \\L(\\ox,\\lm,\\rho)$ and its second-order epi-derivative \ncan be calculated for any $w\\in \\R^n$ by\n\\begin{equation}\\label{eq40}\n\\d^2_x\\L\\big((\\ox,\\lm,\\rho),\\ov\\big)(w)=\\big\\langle \\nabla^2_{xx}L(\\ox,\\mu)w,w\\big\\rangle+ e_{1/{2\\rho}}\\big(\\d^2 g\\big(\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu),\\mu\\big)\\big) \\big(\\nabla \\Phi(\\ox)w \\big), \n\\end{equation}\nwhere $\\mu=\\nabla_x \\big(e_{1/{\\rho}}g\\big)\\big(\\Phi(\\ox)+\\rho^{-1}\\lm\\big)$.\n\\end{Theorem}\n\n\\begin{proof}  Given $\\rho>0$,  it follows from  the second equality in \\eqref{prm} that $\\mu\\in \\sub g\\big(\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu)\\big)$. \nAccording to Proposition~\\ref{sop}(b), the CPLQ function $g$ is twice epi-differentiable at $\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu)$ for $\\mu$. Appealing now \nto  \\cite[Proposition~4.1]{hms20} tells us that    the   envelope function $e_{1/\\rho} g$ is twice epi-differentiable at $\\Phi(\\ox)+\\rho^{-1}\\lm$ for $\\mu$ and that \n\\begin{equation}\\label{more}\n\\d^2 (e_{1/\\rho} g )\\big(\\Phi(\\ox)+\\rho^{-1}\\lm,\\mu\\big)=e_{1/2\\rho}\\big(\\d^2 g(\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu),\\mu)\\big).\n\\end{equation}\nBy \\eqref{pwfor}, the second-order epi-derivative of $g$ is CPLQ. This, combined with \\eqref{more}, indicates that $ e_{1/2\\rho}\\big(\\d^2 g(\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu),\\mu)\\big)$\nis finite on $\\R^m$. Employing now \\cite[Theorem~4.3]{r20}, we conclude that  $e_{1/\\rho}g$ is twice semidifferentiable at $\\Phi(\\ox)+\\rho^{-1}\\lm$ and its second semiderivative \ncoincides with its second-order epi-derivative, namely \n\\begin{equation}\\label{ssed}\n\\d^2\\big(e_{1/\\rho}g\\big)\\big(\\Phi(\\ox)+\\rho^{-1}\\lm\\big)=\\d^2 (e_{1/\\rho} g )\\big(\\Phi(\\ox)+\\rho^{-1}\\lm,\\mu\\big).\n\\end{equation}\nThis, together with the chain and sum rules for twice semidifferentiability, established in  \\cite[Propositions~8.2(i) and 8.1]{mmstams}, \nshows that the mapping $x\\mapsto\\L(x,\\lm,\\rho)$ is twice semidifferentiable at $\\ox$. Thus, by   \\cite[Propositions~8.2(iii)]{mmstams}, the latter mapping is twice epi-differentiable at $\\ox$ for $\\ov$ and \n\\begin{equation}\\label{lss}\n\\d_x^2{\\L}(\\ox,\\lm,\\rho)(w)=\\d^2_x\\L\\big((\\ox,\\lm,\\rho),\\ov\\big)(w), \\quad w\\in \\R^n.\n\\end{equation}\nBy  \\cite[Propositions~8.2(i) and 8.1]{mmstams},  the second  semiderivative of $x\\mapsto\\L(x,\\lm,\\rho)$ at $\\ox$ can be calculated for any $w\\in \\R^n$ by\n\\begin{eqnarray*}\n\\d_x^2{\\L}(\\ox,\\lm,\\rho)(w)&=&\\d^2\\ph(\\ox)(w)+\\big\\la\\mu,\\nabla^2\\Phi(\\ox)(w,w)\\big\\ra+\\d^2\\big(e_{1/\\rho} g\\big)\\big(\\Phi(\\ox)+\\rho^{-1}\\lm\\big)\\big(\\nabla \\Phi(\\ox)w\\big)\\\\\n&=&\\la\\nabla^2\\ph(\\ox)w,w\\ra+\\big\\la\\mu,\\nabla^2\\Phi(\\ox)(w,w)\\big\\ra+\\d^2\\big(e_{1/\\rho} g\\big)\\big(\\Phi(\\ox)+\\rho^{-1}\\lm,\\mu\\big)\\big(\\nabla \\Phi(\\ox)w\\big)\\\\\n&=&\\big\\la\\nabla_{xx}^2L(\\ox,\\mu)w,w\\big\\ra+e_{1/2\\rho}\\big(\\d^2 g(\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu),\\mu)\\big)\\big(\\nabla \\Phi(\\ox)w\\big),\n\\end{eqnarray*}\nwhere the second equality comes from \\eqref{ssed} and the last one results from \\eqref{more}. This together with \\eqref{lss}\njustifies \\eqref{eq40} and hence completes the proof.\n\\end{proof}\n\n\\begin{Corollary}[second-order epi-derivatives] Let $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and set $\\ov:= \\nabla_x L(\\ox,\\olm)$. Then for any $\\rho>0$,\nthe function $x\\mapsto\\L(x,\\lm,\\rho)$, defined via the augmented Lagrangian \\eqref{aug}, is twice epi-differentiable at $\\ox$ for $\\ov $ and its second-order epi-derivative \ncan be calculated for any $w\\in \\R^n$ by\n\\begin{equation}\\label{eq402}\n\\d^2_x\\L\\big((\\ox,\\olm,\\rho),\\ov\\big)(w)=\\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w,w\\big\\rangle+ e_{1/{2\\rho}}\\big(\\d^2 g\\big(\\Phi(\\ox),\\olm\\big)\\big) \\big(\\nabla \\Phi(\\ox)w \\big).\n\\end{equation}\n\\end{Corollary}\n\n\\begin{proof} Because  $(\\ox,\\olm)$ is a solution to the KKT system \\eqref{vs}, we get $\\olm\\in \\sub g\\big(\\Phi(\\ox)\\big)$.\nSince $g$ is CPLQ, we conclude from the second equality in \\eqref{prm} that the latter inclusion is equivalent to $\\nabla(e_{ {1}/{\\rho}} g)\\big(\\Phi(\\ox)+\\rho^{-1}\\olm\\big)=\\olm$\nfor any $\\rho>0$. This means that $\\mu=\\olm$, where $\\mu$ was taken from Theorem~\\ref{ssc}. Moreover, it follows from \n\\eqref{kkt2} that $\\nabla_x \\L(\\ox,\\olm,\\rho)=\\nabla_x L(\\ox,\\olm)=\\ov$.\nCombining these and the second-order epi-derivative formula \\eqref{eq40}, \nwe arrive at \\eqref{eq402}.\n\\end{proof}\n\n\\begin{Theorem}[characterizations of quadratic growth condition]\\label{growth}\nLet $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and set $\\ov:= \\nabla_x L(\\ox,\\olm)$. Then the following conditions are equivalent: \n \\begin{enumerate}[noitemsep,topsep=0pt]\n \\item  the SOSC \\eqref{sosc} holds at $(\\ox,\\olm)$;\n \\item   there exists a constant $\\bar\\rho>0$ such that for any $\\rho\\ge\\bar\\rho$ the second-order condition \\eqref{eq26} is satisfied;\n \\item there exist positive constants $\\bar\\rho$, $\\gg$, and $\\ell$ such that for any $\\rho\\ge\\bar \\rho$, the quadratic growth condition \\eqref{eq16} \nholds.\n\\end{enumerate}\n\\end{Theorem}\n\n\\begin{proof} The equivalence between (b) and (c) was already established in Proposition~\\ref{2ndgr}.  Suppose that (b) is satisfied. Since we always have \n$$\ne_{1/{2\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\olm)\\big)(\\nabla \\Phi(\\ox)w)\\le \\d^2 g(\\Phi(\\ox),\\olm)(\\nabla \\Phi(\\ox)w)\\quad \\mbox{for all}\\;\\; \\rho>0, \n$$\nthe validity of the second-order condition \\eqref{eq402} yields the SOSC \\eqref{sosc} and so proves (a). \n\nAssume now that (a) holds. Since the second subderivative is lower semicontinuous and positive homogeneous of degree $2$ (cf. \\cite[Proposition~13.5]{rw}), the SOSC \\eqref{sosc} amounts to the existence of \na positive constant $\\ell$ so that we have \n\\begin{equation}\\label{som}\n\\langle\\nabla_{xx}^2L(\\bar x,\\olm)w,w\\rangle+\\d^2g (\\Phi(\\bar x),\\olm) (\\nabla \\Phi(\\ox)w )\\ge \\ell \\quad \\mbox{for all }\\; w\\in \\D\\cap \\S,\n\\end{equation}\nwhere $\\S$ stands for the unit sphere in $\\R^n$. Let $w\\in \\S$ and denote the left-hand side of \\eqref{som} by $h(w)$.\n  Given $\\rho>0$ and $w\\in \\R^n$, define the function $h_\\rho:\\R^n\\to \\R$ by \n$$\nh_\\rho(w)=\\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w,w\\big\\rangle+ e_{1/{2\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\olm)\\big)(\\nabla \\Phi(\\ox)w).\n$$\nWe understand from \\cite[Theorem~1.25]{rw} that \n$$\nh_\\rho(w) \\nearrow h(w)\\;\\; as \\;\\; \\rho \\nearrow \\infty\n$$\n for all $w\\in\\R^n$. Employing now \\cite[Proposition~7.4]{rw} tells us that  $h_\\rho\\overset {\\ss \\mbox{e}}{\\to} h$ as $\\rho \\nearrow \\infty$.\n This, combined with \\cite[Theorem~7.46]{rw}, implies that $h_\\rho+\\dd_{\\S\\cap K_\\Th(\\ox,-\\ov)}\\overset {\\ss \\mbox{e}}{\\to} h+ \\dd_{\\S\\cap K_\\Th(\\ox,-\\ov)}$ as  $\\rho \\nearrow \\infty$.\n Appealing now to \\cite[Theorem~7.33]{rw} indicates that \n\\begin{equation}\\label{infco}\n \\inf \\big\\{  h_\\rho(w)|\\; w\\in \\S\\cap K_\\Th(\\ox,-\\ov)\\big\\} \\to \\inf\\big\\{ h(w)|\\; w\\in \\S\\cap K_\\Th(\\ox,-\\ov)\\big\\}\\;\\;\\mbox{as}\\;\\;\\rho \\nearrow \\infty.\n\\end{equation}\nIt follows from \\eqref{pwfor} and \\eqref{domcr} that  for any $w\\in \\R^n$ with $\\nabla \\Phi(\\ox)w \\notin K_g(\\Phi(\\ox),\\olm)$, we have  $\\d^2g (\\Phi(\\bar x),\\olm) (\\nabla \\Phi(\\ox)w )=\\infty$.\nThis,  \\eqref{som}, and the definition of the convex cone $\\D$ from \\eqref{coned}  bring us to \n$$\n\\inf\\big\\{ h(w)|\\; w\\in \\S\\cap K_\\Th(\\ox,-\\ov)\\big\\}\\ge \\ell.\n$$\nCombining this and \\eqref{infco}, we can find a positive constant $ \\bar \\rho$ for which we have \n\\begin{equation}\\label{rb1}\nh_\\rho(w)>\\frac{\\ell}{2}\\quad \\mbox{for all}\\;\\;\\rho\\ge  \\bar \\rho\\;\\;\\mbox{and all}\\;\\;w\\in    \\S\\cap K_\\Th(\\ox,-\\ov).\n\\end{equation}\nGiven $\\rho\\ge \\bar\\rho$, it follows from \\eqref{eq402} and the definition of $h_\\rho$  that \n$$\n\\d^2_x\\L\\big((\\ox,\\olm,\\rho),\\ov\\big)(w) >0\\quad \\mbox{for all} \\;\\; w\\in \\S\\cap K_\\Th(\\ox,-\\ov).\n$$\nRemember that the second-order epi-derivative  is positive homogeneous of degree $2$. So we arrive at the second-order condition \\eqref{eq26} for any $\\rho\\ge \\bar\\rho $. \nThis proves (b) and hence completes the proof.\n\\end{proof}\n\n\\begin{Lemma}[second-order epi-derivatives of augmented Lagrangians] \\label{usog}\nLet $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and set $\\ov:= \\nabla_x L(\\ox,\\olm)$. \nThen the following conditions are equivalent: \n \\begin{enumerate}[noitemsep,topsep=0pt]\n \\item the  SOSC \\eqref{sosc} holds at $(\\ox,\\olm)$;\n \\item there exist positive constants $\\bar\\rho$, $\\ell_1$, $\\ve_1$ such that for all $\\rho\\ge\\bar\\rho$ and\nall  $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ we have\n\\begin{equation}\\label{eq44}\n\\d^2_x\\L \\big((\\ox,\\lm,\\rho), \\nabla_x L(\\ox,\\lm)\\big)(w)\\ge\\ell_1\\|w\\|^2\\;\\mbox{ for all }\\;w\\in K_\\Th\\big(\\ox, - \\nabla_x L(\\ox,\\olm)\\big).\n\\end{equation}\n\\end{enumerate}\n\\end{Lemma}\n\n\\begin{proof} The implication (b)$\\implies$(a)   results directly from Theorem~\\ref{growth}. To prove the opposite implication, assume that (a) holds. \nIt follows from Theorem~\\ref{growth} that there exists a constant $\\bar\\rho>0$ such that for all $\\rho\\ge\\bar\\rho$ the second-order condition \\eqref{eq26} is satisfied. \nRecall from  \\cite[Proposition~13.5]{rw} that  the second-order epi-derivative  is lower semicontinuous and positive homogenous of degree $2$. \nThus the second-order condition  \\eqref{eq26} is equivalent  to the existence of a constant $\\ell_1>0$ such that the estimate \n\\begin{equation}\\label{eq41}\n\\d^2_x\\L \\big((\\ox,\\olm,\\bar\\rho),\\ov \\big)(w)=\\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w, w\\big\\rangle+ e_{ {1}/{2\\bar\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\olm)\\big)(\\nabla\\Phi(\\ox)w)\\ge 2\\ell_1\n\\end{equation}\nholds for all $w\\in \\S\\cap K_\\Th(\\ox, -\\ov)$, where the equality in \\eqref{eq41} comes from  \\eqref{eq40}. We proceed now by proving the following claim:\n\n{\\bf Claim A.} {\\em There exists an $\\varepsilon_1>0$ such that for any $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ the estimate\n\\begin{equation}\\label{clq2}\n\\d^2_x\\L \\big((\\ox,\\lm,\\bar\\rho), \\nabla_x L(\\ox,\\lm) \\big)(w)=\\big\\langle \\nabla^2_{xx}L(\\ox,\\lm)w,w\\big\\rangle+ e_{ {1}/{2\\bar\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\lm)\\big)(\\nabla\\Phi(\\ox)w)\\ge \\ell_1\n\\end{equation} \n holds for all $w\\in \\S\\cap K_\\Th\\big(\\ox, -\\nabla_x L(\\ox,\\lm)\\big)$.}\n \n  To furnish this claim, choose $\\varepsilon_1\\in (0, {\\ell_1}/{\\|\\nabla^2\\Phi(\\ox)\\|})$ if $\\|\\nabla^2\\Phi(\\ox)\\|\\neq 0$ and $\\varepsilon_1 > 0$ otherwise\n  and let $w\\in \\S$.\n Thus for all     $\\lm\\in \\B_{\\ve_1}(\\olm)$, we obtain   \n\\begin{eqnarray}\\label{clq3}\n\\big\\langle \\nabla^2_{xx}L(\\ox,\\lm)w, w\\big\\rangle &=& \\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w, w\\big\\rangle+\\big\\langle\\lm -\\olm, \\nabla^2  \\Phi (\\ox)(w, w)\\big\\rangle\\nonumber\\\\\n&\\geq&\\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w, w\\big\\rangle - \\|\\nabla^2\\Phi(\\ox)\\|\\cdot\\|\\lm - \\olm\\|\\nonumber\\\\\n&\\geq&\\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w, w\\big\\rangle - \\ell_1.\n\\end{eqnarray}\nMoreover, it follows from \\eqref{cc2} that \n\\begin{eqnarray}\\label{clq4}\ne_{ {1}/{2\\bar\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\lm)\\big)(\\nabla\\Phi(\\ox)w)&=&\\inf_{v\\in \\R^m}\\big\\{\\d^2g\\big(\\Phi(\\ox)\\big)(v)+\\delta_{K_g(\\Phi(\\ox), \\lm)}(v)+\\bar\\rho\\|v-\\nabla\\Phi(\\ox)w\\|^2\\big\\}\\nonumber\\\\ \n&=& \\inf_{v\\in K_g(\\Phi(\\ox), \\lm)}\\big\\{\\d^2g \\big(\\Phi(\\ox) \\big)(v)+\\bar\\rho\\|v-\\nabla\\Phi(\\ox)w\\|^2\\big\\}. \n\\end{eqnarray}\nNote that $\\d^2g(\\Phi(\\ox))$ does not depend on $\\lambda$. Shrinking $\\ve_1$ if necessary and recalling that $\\sub g\\big(\\Phi(\\ox)\\big)$ is a polyhedral convex set, we conclude from   Proposition~\\ref{sop}(a) that  \nthe relationships \n\\begin{equation*}\nK_g(\\Phi(\\ox), \\lm) = N_{\\partial g(\\Phi(\\ox))}(\\lm) \\subset N_{\\partial g(\\Phi(\\ox))}(\\olm) = K_g(\\Phi(\\ox), \\olm) \n\\end{equation*} \nhold for all $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$.  These, together with  \\eqref{clq4}, tell us  that for any $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ we have \n\\begin{equation}\\label{clq5}\ne_{ {1}/{2\\bar\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\lm)\\big)(\\nabla\\Phi(\\ox)w) \\geq e_{ {1}/{2\\bar\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\olm)\\big)(\\nabla\\Phi(\\ox)w).\n\\end{equation} \n Combining \\eqref{eq41}, \\eqref{clq3}, and \\eqref{clq5} verifies   \\eqref{clq2} for all $w\\in \\S\\cap K_\\Th(\\ox, -\\ov)$ and   all $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$. \n To justify \\eqref{clq2} for all $w\\in \\S\\cap  K_\\Th \\big(\\ox, -\\nabla_x L(\\ox, \\lm) \\big)$, observe that by the  polyhedrality of $\\Th$,   we can assume without loss of generality that \n $$\n K_\\Th \\big(\\ox, -\\nabla_x L(\\ox, \\lm) \\big)=N_{N_\\Th(\\ox)}\\big(-\\nabla_x L(\\ox, \\lm)\\big) \\subset N_{N_\\Th(\\ox)}\\big(-\\nabla_x L(\\ox, \\olm)\\big) = K_\\Th(\\ox, -\\ov)\n $$\n  for all $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$. This clearly justifies Claim A. \n  \nUsing the positive homogeneity of the second-order epi-derivative yields \\eqref{eq44} for $\\rho = \\bar\\rho$ and for all $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$. Recall that the function $$\\rho\\mapsto e_{ {1}/{2\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\lm)\\big)(\\nabla\\Phi(\\ox)w)$$ is nondecreasing. So is the function $\\rho \\mapsto\\d^2\\L( (\\ox, \\lm, \\rho), \\nabla_x L(\\ox, \\lm))(w)$. This  \nyields  \\eqref{eq44}  for all $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ and all $\\rho \\geq \\bar\\rho$, and hence completes the   proof.\n\\end{proof}\n\n\\begin{Theorem}[uniform quadratic growth condition for augmented Lagrangians]\\label{ugrowth}\nLet $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs}. Then the following conditions are equivalent:\n \\begin{enumerate}[noitemsep,topsep=0pt]\n \\item    the  SOSC \\eqref{sosc} holds at $(\\ox,\\olm)$;\n \\item there are positive constants $\\bar\\rho, \\gamma, \\varepsilon,\\ell$ such that for all $\\lm\\in\\Lambda(\\ox)\\cap\\B_\\varepsilon(\\olm)$ and all $\\rho\\ge\\bar\\rho$  the uniform quadratic growth condition\n\\begin{equation}\\label{eq14}\n\\L(x,\\lm,\\rho)\\ge \\ph(\\ox)+g(\\Phi(\\ox))+\\ell\\| x-\\ox\\|^2\\;\\mbox{ for all }\\;x\\in\\B_\\gamma(\\ox)\\cap \\Th\n\\end{equation}\nis satisfied.\n\\end{enumerate}\n\\end{Theorem}\n\n\\begin{proof} The implication (b)$\\implies$(a)   results directly from Theorem~\\ref{growth}. To prove the opposite implication, assume that (a) holds. \nIt follows from Lemma~\\ref{usog} that there exist the positive constants $\\ell_1$, $\\varepsilon_1$, and $\\bar\\rho$  for which \\eqref{eq44} is satisfied  for all \n $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ and all $\\rho\\ge\\bar\\rho$. Using this and  Proposition~\\ref{fopag}(a), \n we deduce from \\eqref{eq44} that for any $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ there exists $\\gamma_{\\lm}>0$ for which we have \n\\begin{equation}\\label{eq22}\n\\L(x,\\lm,\\bar\\rho)\\ge \\ph(\\ox)+g(\\Phi(\\ox))+\\dfrac{\\ell_1}{2}\\Vert x-\\ox\\Vert^2\\;\\textrm{ for all }\\;x\\in\\B_{\\gamma_\\lm}(\\ox)\\cap \\Th,\n\\end{equation}\nwhere the constant $ {\\ell_1}$ can be chosen the same for all the multipliers $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$. The\nlatter results directly from the definition of the second subderivative. The radii of the balls centered at $\\ox$ in \\eqref{eq22}, however,  depend on $\\lm$. We argue below that  a common radius can be chosen for all the multipliers $\\lm\\in\\Lambda(\\ox)$ that are sufficiently close to $\\olm$. To this end, assume that \n$$\n\\tilde\\B_{\\varepsilon_1}(\\olm)=\\big\\{\\lm\\in \\R^m|\\; \\|\\lm-\\olm\\|_1\\le \\ve_1\\big\\}\n$$\nwhere $\\|\\lm-\\olm\\|_1$ stands for $L_1$-norm of the vector $\\lm-\\olm$, namely the sum of the absolute values of the components of $\\lm-\\olm$. Clearly, we have $\\tilde\\B_{\\varepsilon_1}(\\olm)\\subset \\B_{\\varepsilon_1}(\\olm)$.  Define the function $\\psi\\colon\\R^m\\to\\oR$ by\n\\begin{equation}\\label{eq23}\n\\psi(\\lm):=\\sup_{x\\in\\big(\\B_{\\gamma_{\\olm}}(\\ox)\\cap \\Th\\big)\\setminus \\{\\ox\\}}\\frac{\\ph(\\ox)+g(\\Phi(\\ox))-\\L(x,\\lambda,\\bar\\rho)}{\\Vert x-\\ox\\Vert^2}\n+\\delta_{\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm)} (\\lambda),\\quad\\lm\\in\\R^m.\n\\end{equation}\nBy Proposition~\\ref{paug}(b), the function $\\lambda\\mapsto\\L(x,\\lm,\\bar\\rho)$ is concave. This, combined with the convexity of the set $\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm)$, \n guarantees  that $\\psi$ in \\eqref{eq23} is a convex function. Moreover, $\\psi$ is lower semicontinuous since  the first term on the right-hand side of \\eqref{eq23} is the pointwise supremum of a collection of continuous functions.\n Now we claim  that for any $\\lm\\in\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm)$ the value $\\psi(\\lm)$ is finite. To justify this claim, let $\\lm\\in\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm)$. If  $\\gamma_\\lm\\ge\\gamma_{\\olm}$,  we conclude from    \\eqref{eq22} that \n\\begin{equation*}\n\\psi(\\lm)\\le\\sup_{x\\in \\big(\\B_{\\gamma_{\\lm}}(\\ox)\\cap \\Th\\big)\\setminus \\{\\ox\\}}\\frac{\\ph(\\ox)+g(\\Phi(\\ox))-\\L(x,\\lm,\\bar\\rho)}{\\Vert x-\\ox\\Vert^2}\\le-\\dfrac{\\ell_1}{2},\n\\end{equation*}\nwhich, in particular,   implies that $\\psi(\\olm)\\le- {\\ell_1}/{2}$. On the other hand, if $\\gamma_\\lm<\\gamma_{\\olm}$, we get  \n\\begin{eqnarray*}\n &&\\psi(\\lm) \\le\\\\\n&&  \\max\\left\\{\\sup_{x\\in \\big(\\B_{\\gamma_{\\lm}}(\\ox)\\cap \\Th\\big)\\setminus \\{\\ox\\}}\\frac{\\ph(\\ox)+g(\\Phi(\\ox))-\\L(x,\\lm,\\bar\\rho)}{\\Vert x-\\ox\\Vert^2},\\max_{\\substack{\\gamma_\\lm\\le\\Vert x-\\ox\\Vert\\le\\gamma_{\\olm}\\\\ x\\in \\Th}}\\dfrac{\\ph(\\ox)+g(\\Phi(\\ox))-\\L(x,\\lm,\\bar\\rho)}{\\Vert x-\\ox\\Vert^2}\\right\\}.\n\\end{eqnarray*}\nBy \\eqref{eq22}, the first term inside the maximum is bounded above by  $- {\\ell_1}/{2}$.  Also, the second term is finite since it is the maximum of a continuous function over a compact set. \nCombining these tells us that $\\psi(\\lm)$ is finite for all $\\lm\\in\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm)$, which  results in \n$$\n\\dom\\psi=\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm).\n$$\nSince both sets  $\\Lambda(\\ox)$ and $\\tilde\\B_{\\varepsilon_1}(\\olm)$ are polyhedral, $\\dom\\psi$ is   a compact polyhedral convex set.\nAppealing now to   \\cite[Theorem~2.35]{rw} ensures that  $\\psi$ is continuous relative to its domain.  \nThus, we can find  $\\varepsilon >0$ such that $\\B_\\varepsilon(\\olm)\\subset \\tilde\\B_{\\varepsilon_1}(\\olm)$   and that\n\\begin{equation*}\n\\psi(\\lm)\\leq \\psi(\\olm)+\\frac{\\ell_1}{4}\\leq -\\frac{\\ell_1}{4}  \n\\end{equation*} \nfor all $\\lm \\in \\dom\\psi\\cap \\B_\\varepsilon(\\olm)=\\Lambda(\\ox)\\cap \\B_\\varepsilon(\\olm)$. This, combined with the definition of $\\psi$ in \\eqref{eq23}, tells us that for \nall $x\\in\\B_{\\gamma_{\\olm}}(\\ox)\\cap \\Th$ and all $\\lm\\in\\Lambda(\\ox)\\cap\\B_\\varepsilon(\\olm)$ we have \n\\begin{equation}\\label{eq25}\n\\L(x,\\lm,\\bar\\rho)\\ge \\ph(\\ox)+g(\\Phi(\\ox))+\\frac{\\ell_1}{4}\\Vert x-\\bar x\\Vert^2.\n\\end{equation}\nRemember that by  Proposition~\\ref{paug}(c) we always have \n\\begin{equation*}\n\\L(x,\\lm,\\rho)\\ge\\L(x,\\lm,\\bar\\rho)\\;\\mbox{ for all}\\;\\;\\rho\\ge\\bar\\rho.\n\\end{equation*}\nCombining this with \\eqref{eq25} and setting $\\ell:= {\\ell_1}/{4}$ verify the uniform quadratic growth condition \\eqref{eq14} and hence complete the proof.\n\\end{proof}\n\n\\begin{Proposition}[solvability of subproblems]\\label{solv} \nLet $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and let  the  SOSC \\eqref{sosc} hold at $(\\ox,\\olm)$.\nThen there exist positive constants      $\\tau$ and $\\hat\\gamma\\le \\gg$  with $\\gg$ taken from Theorem~{\\rm \\ref{ugrowth}}  such that for any $\\rho\\ge \\bar\\rho $ with $\\bar\\rho$ taken from Theorem~{\\rm\\ref{ugrowth}},\n the optimal solution mapping $S_\\rho:\\R^m\\to \\R^n$, defined by \n\\begin{equation}\\label{eq177}\nS_\\rho(\\lambda):=\\argmin\\big\\{\\L(x,\\lambda,\\rho)\\;\\big|\\; x\\in\\Th\\cap\\B_{\\hat\\gamma}(\\ox)\\big\\}, \\;\\;\\lm\\in \\R^m,\n\\end{equation}\nenjoys the uniform  isolated calmness property \n\\begin{equation}\\label{pt9}\n   S_\\rho(\\lambda) \\subset\\{\\ox\\} +\\tau \\Vert\\lambda-\\olm\\Vert\\B\n\\end{equation}\n and satisfies the condition $\\emp \\neq S_\\rho(\\lambda)\\subset  \\inte  \\B_{\\hat\\gamma}(\\ox)$ for all $\\lm\\in \\B_{\\hat\\gamma/2\\tau}(\\olm)$.  \n \\end{Proposition}\n\n\\begin{proof} Since the SOSC \\eqref{sosc} is satisfied at $(\\ox,\\olm)$, it follows from Theorem~\\ref{ugrowth}\nthat there exist positive constants $\\bar\\rho$, $\\gg$, and $\\ell$ such that for any $\\rho\\ge\\bar \\rho$, the uniform quadratic growth condition \\eqref{eq14} holds for $\\lm=\\olm$. Moreover, \nby the twice differentiability of  $\\Phi$  at $\\ox$, we find some  constants $\\hat\\gamma\\in(0,\\gamma]$ and $\\kappa_\\Phi>0$ for which we have \n\\begin{equation}\\label{eq49}\n\\Vert\\Phi(x)-\\Phi(\\ox)\\Vert\\le\\kappa_\\Phi\\Vert x-\\ox\\Vert\\;\\textrm{ for all }\\;x\\in\\B_{\\hat\\gamma}(\\ox).\n\\end{equation}\nClearly,  the uniform quadratic growth condition \\eqref{eq14} for $\\lm=\\olm$ indicates  that  $S_\\rho(\\olm)\\cap \\B_{\\gamma}(\\ox)=\\{\\ox\\}$ for all $\\rho\\ge\\bar\\rho$. \nDefine   the positive constant\n\\begin{equation}\\label{eq21}\n\\tau:=\\frac{\\kappa_\\Phi}{\\ell}+\\sqrt{\\frac{\\kappa_\\Phi^2}{\\ell^2}+\\frac{1}{\\ell\\bar\\rho}},\n\\end{equation}\nand fix   $\\lambda\\in \\B_{\\hat\\gamma/2\\tau} (\\olm)$ and $\\rho\\geq\\bar\\rho$.\nObserve from the classical Weierstrass theorem that $S_\\rho(\\lm)\\neq\\emp$.   Assume that  $u\\in S_\\rho(\\lambda)$.\nBy  Proposition~\\ref{paug}(b),   the function $\\lambda\\mapsto\\L(u,\\lambda,\\rho)$ is concave. \nBy \\cite[Theorem~2.26(b)]{rw}, we have \n\\begin{equation}\\label{ueq19}\n\\nabla_\\lm\\L(u, \\lm, \\rho)=\\rho^{-1}\\big(\\nabla\\menv\\big(\\Phi(u)+\\rho^{-1}{\\lm}\\big) - \\lm\\big)\n= \\Phi(u)-\\prox \\big(\\Phi(u)+ \\rho^{-1}\\lm\\big),\n\\end{equation} \n which in turn yields  the relationships\n\\begin{eqnarray*} \n\\L(u,\\lambda,\\rho)&\\ge&\\L(u,\\olm,\\rho)-\\langle\\nabla_\\lambda\\L(u,\\lambda,\\rho),\\olm-\\lambda\\rangle\\nonumber\\\\\n&=&\\L(u,\\olm,\\rho)-\\big\\langle \\Phi(u)-\\prox\\big(\\Phi(u)+ \\rho^{-1}{\\lm}\\big),\\olm-\\lambda\\big\\rangle\\nonumber\\\\\n&\\ge& \\ph(\\ox)+g(\\Phi(\\ox))+\\ell\\Vert u-\\ox\\Vert^2-\\big\\langle \\Phi(u)-\\prox\\big(\\Phi(u)+ \\rho^{-1}{\\lm} \\big),\\olm-\\lambda\\big\\rangle\n\\end{eqnarray*}\nwith the last inequality resulting from  \\eqref{eq16}. We conclude from  $u\\in S_\\rho(\\lambda)$ that \n\\begin{equation*}\n\\L(u,\\lambda,\\rho)\\le\\L(\\ox,\\lambda,\\rho)=\\ph(\\ox)+\\menv\\big(\\Phi(\\ox)+\\rho^{-1}{\\lm}\\big) - \\sm \\rho^{-1}{\\|\\lm\\|^2}\\leq \\ph(\\ox)+g \\big(\\Phi(\\ox) \\big).\n\\end{equation*}\nCombining these brings us to \n\\begin{equation}\\label{eq20}\n\\Vert u-\\ox\\Vert^2\\le\\frac{1}{\\ell}\\big\\langle \\Phi(u)-\\prox\\big(\\Phi(u)+\\rho^{-1}\\lm\\big),\\olm-\\lambda\\big\\rangle.\n\\end{equation}\nBy the first equality in \\eqref{prm}, the inclusion  $\\olm \\in \\partial g\\big(\\Phi(\\ox)\\big)$ amounts to $\\Phi(\\ox) = \\prox\\big(\\Phi(\\ox)+ \\rho^{-1}\\olm\\big)$. This leads us to \n\\begin{eqnarray}\\label{clq7}\n&&\\big\\Vert\\Phi(u)-\\prox\\big(\\Phi(u)+\\rho^{-1}\\lm\\big)\\big\\Vert \\nonumber\\\\\n&=&\\big\\Vert\\Phi(u)-\\Phi(\\ox)+\\prox\\big(\\Phi(\\ox)+\\rho^{-1}\\olm\\big)-\\prox\\big(\\Phi(u)+\\rho^{-1}\\lm\\big)\\big\\Vert\\nonumber\\\\\n&\\le&2\\Vert\\Phi(u)-\\Phi(\\ox)\\Vert+\\rho^{-1}{\\Vert\\olm-\\lambda\\Vert}\\nonumber\\\\\n&\\le&2\\kappa_\\Phi\\Vert u-\\ox\\Vert+\\rho^{-1}{\\Vert\\olm-\\lambda\\Vert},\n\\end{eqnarray}\nwhere the first inequality comes from  the fact that proximal mapping of a convex function is always nonexpansive (cf.  \\cite[Proposition~12.19]{rw}), \nand where the last inequality results from \\eqref{eq49}. This, combined with  \\eqref{eq20},  brings us   to \n\\begin{equation*}\n\\Vert u-\\ox\\Vert^2\\le\\frac{1}{\\ell}\\Big(2\\kappa_\\Phi\\Vert u-\\ox\\Vert+\\rho^{-1}{\\Vert\\lambda-\\olm\\Vert}\\Big)\\Vert\\lambda-\\olm\\Vert.\n\\end{equation*}\nSince the latter inequality  can be expressed  equivalently as \n\\begin{equation*}\n\\ell\\Vert u-\\ox\\Vert^2-2\\kappa_\\Phi\\Vert \\lambda-\\olm\\Vert\\cdot\\Vert u-\\ox\\Vert-\\rho^{-1}{\\Vert \\lambda-\\olm\\Vert^2 }\\le 0,\n\\end{equation*}\nwe arrive at  the estimate\n\\begin{equation*}\n\\Vert u-\\ox\\Vert\\le\\left(\\dfrac{\\kappa_\\Phi}{\\ell}+\\sqrt{\\frac{\\kappa_\\Phi^2}{\\ell^2}+\\frac{1}{\\ell\\rho}}\\right)\\Vert\\lambda-\\olm\\Vert\\le\\tau\\Vert\\lambda-\\olm\\Vert \\le \\tau \\hat\\gamma/2\\tau < \\hat\\gamma.\n\\end{equation*}\nThis clearly justifies   \\eqref{pt9}   and  the inclusion $S_\\rho(\\lambda)\\subset  \\inte  \\B_{\\hat\\gamma}(\\ox)$ for all  $\\lambda\\in \\B_{\\hat\\gamma/2\\tau} (\\olm)$ and $\\rho\\geq\\bar\\rho$,  and hence completes the proof.\n\\end{proof}\n\n\\begin{Remark}{\\rm Assume that  the SOSC \\eqref{sosc} holds at $(\\ox,\\olm)$. It is not hard to see that a similar argument as those for \\eqref{eq20}\nshows that for any $\\lm\\in \\R^m$, any $u\\in S_\\rho(\\lm)$,  any $\\mu \\in \\Lm(\\ox)\\cap \\B_\\ve(\\olm)$, and any $\\rho\\ge \\bar\\rho$ we have \n\\begin{equation*} \n\\Vert u-\\ox\\Vert^2\\le\\frac{1}{\\ell}\\big\\langle \\Phi(u)-\\prox\\big(\\Phi(u)+\\rho^{-1}\\lm\\big),\\mu-\\lambda\\big\\rangle,\n\\end{equation*}\nwhere constants $\\bar\\rho$, $\\ve$, and $\\ell$ come from Theorem~\\ref{ugrowth}.\nCombining this and the second equality in \\eqref{ueq19} brings us to \n\\begin{equation}\\label{ueq20}\n\\Vert u-\\ox\\Vert^2\\le\\frac{1}{\\ell\\rho}\\big\\langle \\nabla e_{1/\\rho}\\big(\\Phi(u)+\\rho^{-1}\\lm\\big)-\\lm,\\mu-\\lambda\\big\\rangle,\n\\end{equation}\nwhich will be utilized in the proof of Theorem~\\ref{est} later in this section.\n}\n\\end{Remark}\n\n\n\\begin{Proposition}[estimate for residual functions] \\label{erct}  Assume that $(\\ox,\\olm)$ is a solution to the KKT system \\eqref{vs} with $\\Th$ taken from \\eqref{Th}. Then \nthere exist positive constants $\\gamma_2$ and $\\kappa_2$ such that\n\\begin{equation}\\label{eb1}\nR(x, \\lambda) \\leq \\kappa_2\\big(\\|x-\\ox\\|+\\dist(\\lambda, \\Lambda(\\ox))\\big)\\quad\\textrm{for all }\\;\\; (x, \\lambda)\\in \\B_{\\gamma_2}(\\ox, \\olm).\n\\end{equation} \n\\end{Proposition}\n\n\\begin{proof} By \\eqref{Th},  we have $N_\\Th(x) = N_\\Th(\\ox)$ for all $x\\in \\Th$. Let $\\mu=P_{\\Lm(\\ox)}(\\lm)$\nand observe that  $(\\ox,\\mu)$ is a solution to the KKT system \\eqref{vs}. So, we know from \\eqref{prm} that    $\\mu \\in \\partial g \\big(\\Phi(\\ox)\\big)$ yields   $\\Phi(\\ox) = {\\rm prox}_g\\big(\\Phi(\\ox)+ \\mu\\big)$.\nUsing these tells us that \n\\begin{eqnarray*}\nR(x, \\lambda)&=& \\dist\\big(-\\nabla_x L(x, \\lm), N_\\Th(x)\\big)+\\big\\|\\Phi(x)-{\\rm{prox}}_g\\big(\\Phi(x)+\\lm\\big)\\big\\|\\\\\n&\\le & \\|\\nabla_x L(x, \\lm)-\\nabla_x L(\\ox, \\olm)  \\|+  \\dist\\big(-\\nabla_x L(\\ox, \\olm), N_\\Th(\\ox)\\big)\\\\\n&&+  \\|\\Phi(x)-\\Phi(\\ox)\\|+ \\big\\|{\\rm{prox}}_g\\big(\\Phi(x)+\\lm\\big)-{\\rm{prox}}_g\\big(\\Phi(\\ox)+\\mu\\big)\\big\\|\\\\\n&\\le &\\|\\nabla_x L(x, \\lm)-\\nabla_x L(\\ox, \\olm)  \\| +  2\\|\\Phi(x)-\\Phi(\\ox)\\|+ \\dist(\\lambda, \\Lambda(\\ox)).\n\\end{eqnarray*}\nThis, together with \\eqref{eq49},  ensures the existence of positive \nconstants  $\\gamma_2$ and $\\kappa_2$ for which the estimate \\eqref{eb1} is satisfied. \n\\end{proof}\n\n\\begin{Theorem}[error bound for consecutive iterates of ALM]\\label{est} \nLet $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and  let the SOSC \\eqref{sosc} hold at $(\\ox,\\olm)$. Then there exist positive constants   $\\gamma_3$, and $\\kappa_3$ such that for any $\\rho\\geq \\bar\\rho$ with \n$\\bar\\rho$ taken from Theorem~{\\rm\\ref{ugrowth}}, any $(x,\\lambda)\\in\\B_{\\gamma_3}(\\ox,\\olm)$ with $x\\in \\Th$ and  $R(x,\\lambda)>0$, and any   optimal solution $u$ to the regularized problem \n\\begin{equation}\\label{regp}\n\\mini \\L(w, \\lm, \\rho)\\quad \\mbox{subject to}\\quad w\\in \\Th\\cap \\B_{\\hat\\gg}(\\ox)\n\\end{equation} \nwith $\\hat\\gg$ taken from Proposition~{\\rm\\ref{solv}},   the error bound estimate\n\\begin{equation}\\label{est0}\n\\Vert u-x\\Vert+\\big\\Vert\\nabla\\menv\\big(\\Phi(u)+ \\rho^{-1}{\\lm}\\big)-\\lm\\big\\Vert\\le\\kappa_3\\,R(x,\\lambda)\n\\end{equation}\nholds, where the residual function  $R$ is  taken from \\eqref{res}.\n\\end{Theorem}\n\n\\begin{proof} It follows from the SOSC \\eqref{sosc} and    Theorem~\\ref{ugrowth} that there are positive  constants $\\bar\\rho$, $\\gamma$, $\\varepsilon$, and $\\ell$ for which \nthe uniform quadratic  growth condition \\eqref{eq14} holds  for all $\\rho\\ge\\bar\\rho$ and all $\\lambda\\in\\Lambda(\\ox)\\cap\\B_\\varepsilon(\\olm)$. \nAccording to Proposition~\\ref{solv}, the solution mapping to \\eqref{regp}, denoted by $S_\\rho(\\lm)$ therein, satisfies the uniform isolated calmness \\eqref{pt9} and the condition $S_\\rho(\\lm)\\subset  \\inte  \\B_{\\hat\\gamma}(\\ox)$ for all $\\lm\\in \\B_{\\hat\\gamma/2\\tau}(\\olm)$ and all $\\rho\\ge \\bar\\rho$, where both positive constants $\\tau$ and $\\hat\\gg$ are taken from this proposition. This, in particular, tells us that for every $\\lm\\in \\B_{\\hat\\gamma/2\\tau}(\\olm)$ and every $\\rho\\ge \\bar\\rho$\nany optimal solution $u$ to \\eqref{regp} satisfies the first-order optimality condition \n\\begin{equation}\\label{optc}\n0\\in \\nabla_x\\L(u, \\lm, \\rho)+ N_\\Th(u).\n\\end{equation}\nAfter this presentation, assume by contradiction that the error bound estimate \\eqref{est0} fails. The latter implies that  there exists a sequence $\\{(\\xk, \\lk, \\rok)\\b\\subset \\Th \\times \\R^m\\times [\\bar\\rho, \\infty)$ \nwith $\\xk\\to\\ox, \\lk\\to\\olm$ as $k\\to\\infty$ such that\n\\begin{equation}\\label{est1}\n\\Vert u^k-\\xk\\Vert+\\| d^k-\\lk\\|>kR_k\\quad \\mbox{with}\\;\\;d^k:=\\nabla\\menvk\\big(\\Phi(u^k)+\\rho_k^{-1}\\lk\\big),\n\\end{equation} \nwhere $u^k$ is an optimal solution to \\eqref{regp} for $(\\lambda, \\rho):=(\\lk, \\rok)$, and where $R_k:=R(\\xk, \\lk)$ for each $k \\in \\mathbb{N}$. Denote by $\\beta_k$ the left-hand side of \\eqref{est1}, and thus get $R_k=o(\\beta_k)$.\n By the definition of $R_k$, the latter yields\n\\begin{equation}\\label{est2}\n-\\nabla_xL(\\xk, \\lk)+o(\\xik)\\in N_\\Th(\\xk)\\quad\\textrm{and}\\quad \\Phi(\\xk)+o(\\xik)= {\\rm{prox}}_g\\big(\\Phi (\\xk)+\\lk\\big).\n\\end{equation}\nPassing to a subsequence if necessary, we can find $(\\xi,\\eta)\\in \\R^n\\times \\R^m$ such that \n\\begin{equation}\\label{est4}\n\\dfrac{u^k-\\xk}{\\xik}\\to \\xi\\quad \\mbox{and}\\quad  \\dfrac{ d^k-\\lk}{\\xik} \\to  \\eta \\quad \\mbox{with}\\;\\; (\\xi,\\eta)\\neq 0.\n\\end{equation}\nWe know from Theorem~\\ref{sooc} that the error bound \\eqref{subr3} holds for all $(x,\\lm)$ sufficiently close to  $(\\ox, \\olm)$. \nSet $\\mu^k:=P_{\\Lambda(\\ox)}(\\lk)$ and remember  that $(\\xk, \\lk)\\to(\\ox, \\olm)$.  By  \\eqref{subr3}, we can assume without loss of generality that $\\xk-\\ox=O(R_k)$ and $\\lk-\\mu^k=O(R_k)$ for all $k\\in \\N$, which in turn results in\n\\begin{equation}\\label{est3}\n\\xk-\\ox=o(\\xik)\\quad\\textrm{and}\\quad\\lk-\\mu^k=o(\\xik)\\quad\\textrm{as } k\\to\\infty.\n\\end{equation}\nThe latter along with $\\lk\\to \\olm$ tells us that $\\mu^k\\to \\olm$. So, we get   $\\mu^k\\in\\Lambda(\\ox)\\cap\\B_\\varepsilon(\\olm)$ for all $k $ sufficiently large. \nThis, combined with  $\\rho_k\\ge\\bar\\rho$ and \\eqref{ueq20}, implies that \n\\begin{eqnarray}\\label{est5}\n\\Vert u^k-\\ox\\Vert^2&\\le&\\dfrac{1}{\\rok\\ell} \\langle d^k-\\lk,\\mu^k-\\lk \\rangle\\le\\dfrac{1}{\\rok\\ell}\\| d^k-\\lk\\|\\cdot\\Vert\\mu^k-\\lk\\Vert,\n\\end{eqnarray}\nwhich together with \\eqref{est4} and  \\eqref{est3} results in \n\\begin{equation*}\n\\dfrac{\\Vert u^k-\\ox\\Vert^2}{\\xik^2}\\le\\dfrac{1}{\\rok\\ell}\\dfrac{\\| d^k-\\lk\\|}{\\xik}\\cdot\\dfrac{\\Vert\\mu^k-\\lk\\Vert}{\\xik}\\to 0\\quad\\textrm{ as }\\;k\\to\\infty.\n\\end{equation*}\nThus, we get $u^k-\\ox=o(\\beta_k)$. Combining this with \\eqref{est3} clearly shows that \n\\begin{equation*}\n\\xi=\\lim_{k\\to\\infty}\\dfrac{u^k-\\xk}{\\xik}=\\lim_{k\\to\\infty}\\dfrac{u^k-\\ox}{\\xik}-\\lim_{k\\to\\infty}\\dfrac{\\xk-\\ox}{\\xik}=0-0=0.\n\\end{equation*}\nIf either the sequence $\\{\\rok\\b$ or  $\\{{\\rok}/{\\xik}\\b$ is bounded, it follows from $u^k-\\ox=o(\\beta_k)$ that  $\\dfrac{\\rok}{\\xik}\\Vert u^k-\\ox\\Vert\\to0$ as $k\\to\\infty$. \nThis, \\eqref{ueq19}, and a similar argument as those for \\eqref{clq7} bring us to \n\\begin{eqnarray*}\n \\dfrac{ \\|d^k-\\lk\\|}{\\xik}\n&=&\\frac{\\rok}{\\xik} \\big\\Vert \\Phi(u^k)-\\proxk\\big(\\Phi(u^k)+\\rho_k^{-1}\\lk\\big)\\big\\Vert\\\\\n&\\leq& 2\\kappa_\\Phi\\frac{\\rok}{\\xik}\\|u^k-\\ox\\|+\\dfrac{\\Vert \\mu^k-\\lk\\Vert }{\\xik}\\to0\\quad\\textrm{as } k\\to\\infty,  \n\\end{eqnarray*} \nwhich by \\eqref{est4} yields $\\eta =0$. This is  a contradiction with \\eqref{est4} since we showed that $(\\xi,\\eta)=0$.\n\nAssume now that both sequences $\\{\\rok\\b$ and  $\\{{\\rok}/{\\xik}\\b$  are unbounded.  We can assume by passing  to a subsequence if necessary that \n\\begin{equation}\\label{unb}\n\\rok\\to \\infty\\quad \\mbox{and}\\quad \\frac{\\rok}{\\beta_k}\\to \\infty\\;\\;\\mbox{as}\\;\\; k\\to \\infty.\n\\end{equation}\nSince $u^k$ is an optimal solution to \\eqref{regp} associated with $(\\lk,\\rok)$, we deduce from \\eqref{optc} that \n\\begin{equation*}\n0\\in \\nabla_x\\L(u^k, \\lk, \\rok)+ N_\\Th(u^k).\n\\end{equation*} \nBy \\eqref{Th}, we get  $N_{\\Th}(u^k)=N_\\Th(\\xk) = \\rge B^*$. These along with  \\eqref{est2} tell us that \n\\begin{equation*}\n\\nabla_x\\L(u^k, \\lk, \\rok)-\\nabla_xL(\\xk, \\lk)+o(\\xik)\\in N_\\Th(\\xk)-N_\\Th(u^k)= \\rge B^*.\n\\end{equation*}\nUsing the definitions of the augmented Lagrangian $\\L$ and  $L$, we get \n\\begin{eqnarray*}\n\\rge B^*&\\ni& \\nabla_x\\L(u^k, \\lk, \\rok)-\\nabla_xL(\\xk, \\lk)+o(\\xik)\\\\\n&=&\\nabla \\varphi(u^k)-\\nabla \\varphi(\\xk)+\\nabla\\Phi(u^k)^* \\nabla\\menvk\\big(\\Phi(u^k)+\\rho_k^{-1}\\lk\\big)-\\nabla\\Phi(\\xk)^*\\lk+o(\\xik)\\\\\n&=&\\big(\\nabla\\Phi(u^k)-\\nabla\\Phi(\\xk)\\big) ^*\\lk+\\nabla\\Phi(u^k)^*\\big (d^k-\\lk\\big )+o(\\xik)\\\\\n&=&\\nabla\\Phi(u^k)^* \\big(d^k-\\lk\\big)+o(\\xik),\n\\end{eqnarray*}\nwhere the second and last equalities result from  the Lipschitz continuity of  $\\nabla \\varphi$ and $\\nabla \\Phi$ around $\\ox$ and from  the fact that $u^k-\\xk=o(\\xik)$. \nDividing both sides  by $\\xik$ and then letting  $k\\to\\infty$ confirm via \\eqref{est4} that \n\\begin{equation*}\n\\nabla\\Phi(\\ox)^*\\eta \\in\\rge B^*.\n\\end{equation*} \nWe are going to show that $\\eta=0$, which contradicts \\eqref{est4} and thus completes the proof. To furnish this, set $E:=\\big\\{z \\in \\R^n\\, |\\, \\nabla \\Phi(\\ox)^*z\\in \\rge B^*\\big\\}$\nand observe from \\cite[Corollary~11.25(d)]{rw} that $E^\\perp=\\big\\{\\nabla\\Phi(\\ox)y|\\;y\\in \\ker B\\big\\}$. So, we conclude  from  $\\ox,u^k\\in \\Th$ and  \\eqref{Th} that $\\nabla\\Phi(\\ox)(u^k-\\ox)\\in E^\\perp$.\nMoreover, we obtain  from \\eqref{est4}-\\eqref{est5} that \n\\begin{equation}\\label{est6}\n\\dfrac{\\rok}{\\xik}\\Vert u^k-\\ox\\Vert^2\\leq\\dfrac{1}{\\ell} \\dfrac{\\| d^k-\\lk\\|}{\\xik} \\cdot\\Vert \\mu^k-\\lk\\Vert  \\to 0 \\quad\\textrm{ as }k\\to\\infty.\n\\end{equation} \nSince $P_E$ is a linear mapping due to $E$ being a subspace, we arrive at \n\\begin{eqnarray}\\label{est7}\nP_E\\Big(\\dfrac{\\rok}{\\xik}\\big (\\Phi(u^k)-\\Phi(\\ox)\\big )\\Big)&=&P_E\\Big (\\dfrac{\\rok}{\\xik}\\nabla\\Phi(\\ox) (u^k-\\ox )+O\\big (\\dfrac{\\rok}{\\xik}\\Vert u^k-\\ox\\Vert^2 \\big)\\Big  )\\nonumber\\\\\n&=&P_E \\Big(O\\big(\\dfrac{\\rok}{\\xik}\\Vert u^k-\\ox\\Vert^2 \\big )\\Big)\\to 0 \\quad \\textrm{as } k\\to\\infty,\n\\end{eqnarray} \nwhere the last equality results from $\\nabla\\Phi(\\ox)(u^k-\\ox)\\in E^\\perp$.\nFix $k\\in \\N$ and set  \n$\nz^k:= \\Phi(u^k)+ \\rho_k^{-1}\\big(\\lk -d^k\\big).\n$\nIt follows from $d^k=\\nabla\\menvk\\big(\\Phi(u^k)+ \\rho_k^{-1}{\\lk}\\big)$ and the second equation in \\eqref{prm} that \n\\begin{equation}\\label{clq8}\nd^k \\in \\partial g(z^k).\n\\end{equation} \nUsing  \\eqref{est4} and \\eqref{unb} allows us to arrive at \n\\begin{equation*}\nz^k =\\Phi(u^k) - \\frac{d^k-\\lk}{\\xik}\\cdot\\frac{\\xik}{\\rok} \\to \\Phi(\\ox)=:\\bar z\\quad\\textrm{as }\\; k\\to\\infty.\n\\end{equation*} \nBy \\eqref{clq8}, we get $z^k\\in \\dom g=\\cup_{i=1}^s C_i$ for all $k$, where the polyhedral convex sets $C_i$ are taken from \\eqref{PWLQ}. \nBy passing to a subsequence if necessary, we can assume without loss of generality that  for some $\\oi\\in I(\\oz)$, we have $z^k \\in C_\\oi$ for all $k$, where the index set  $  I(\\bar z)$  is defined by \\eqref{act}.\nSince $C_\\oi$ is  polyhedral, we find   $l_\\oi\\in \\N$, $c_{\\oi, j}\\in \\R^m$, and $\\al_{\\oi, j}\\in \\R$,  $j=1,\\ldots,l_\\oi$,  such that \n\\begin{equation*}\nC_\\oi= \\big\\{z\\in \\R^m\\, |\\, \\langle c_{\\oi, j}, z\\rangle \\leq \\al_{\\oi, j}, \\quad j= 1, \\ldots, l_\\oi\\big\\}.\n\\end{equation*} \nDenote by $J(\\bar z)$ and $J(z^k)$ the sets of the indexes of active inequalities in $C_\\oi$ at $\\bar z$ and $z^k$, respectively. Since $z^k\\to \\oz$, we obtain  $J(z^k)\\subset J(\\bar z)$ for all $k$ sufficiently large. \nPassing to a subsequence, if necessary, we can assume that there exists a subset $\\bar J\\subset J(\\bar z)$ such that $J(z^k)=\\bar J$ for all $k$. Thus, we have \n\\begin{equation*}\nz^k -\\bar z \\in \\big\\{v\\in \\R^m\\, \\big|\\, \\langle c_{\\oi, j}, v\\rangle= 0\\;\\textrm{ for } \\;j \\in \\bar J, \\; \\langle c_{\\oi, j}, v\\rangle\\leq 0\\; \\textrm{ for }\\; j\\in J(\\bar z)\\setminus \\bar J \\big\\}=: \\Omega.\n\\end{equation*} \nIt follows from  the definitions of $z^k$ and $\\bar z$ that \n\\begin{equation*}\n\\Phi(u^k)-\\Phi(\\ox)-\\frac{ d^k-\\lk}{\\rok}=z^k-\\oz \\in \\Omega. \n\\end{equation*}\nMultiplying by $ {\\rok}/{\\xik}$ and using the linearity of  $P_E$  allow us to get \n\\begin{equation*}\nP_E\\Big(\\dfrac{\\rok}{\\xik}\\big (\\Phi(u^k)-\\Phi(\\ox)\\big )\\Big)- P_E \\big(\\frac{ d^k-\\lk}{\\xik}\\big) \\in P_E(\\Omega),\n\\end{equation*} \nwhich, together with \\eqref{est4}, \\eqref{est7}, and the closedness of $P_E(\\Omega)$, yields $-P_E(\\eta) \\in P_E(\\Omega)$. Since $\\eta\\in E$, the latter confirms that \n\\begin{equation}\\label{est8}\n-\\eta=-P_E(\\eta) = P_E(\\zeta)\\quad\\textrm{for some }\\;\\; \\zeta\\in \\Omega.\n\\end{equation}\nRemembering $\\mu^k\\in \\Lm(\\ox)$, we obtain $\\mu^k \\in \\partial g(\\bar z)$. By the representation of $\\partial g(\\bar z)$ from \\eqref{sub} and the fact that  $\\oi\\in I(\\bar z)$, we conclude that \n\\begin{equation*}\\label{clq9}\n\\mu^k - A_\\oi\\bar z -a_\\oi \\in N_{C_\\oi}(\\bar z) = \\cone\\big\\{c_{\\oi, j}\\, | \\;j\\in J(\\bar z)\\big\\}.\n\\end{equation*} \nSince we also have $\\oi\\in I(z^k)$, it follows again from \\eqref{clq8} and  \\eqref{sub} that \n\\begin{equation*}\\label{clq10}\nd^k -A_\\oi z^k-a_\\oi \\in N_{C_\\oi}(z^k) = \\cone\\big\\{c_{\\oi, j}\\,|\\; j\\in \\bar J\\big\\}.\n\\end{equation*} \nCombining these results in \n\\begin{eqnarray*}\n \\mu^k-d^k + A_\\oi(z^k-\\bar z)&\\in&   \\cone\\big\\{c_{\\oi, j}\\, | \\;j\\in J(\\bar z)\\big\\}-\\cone\\big\\{c_{\\oi, j}\\,|\\; j\\in \\bar J\\big\\}\\\\\n&=&\\span\\big\\{c_{\\oi, j}\\,|\\;  j\\in \\bar J\\big\\}+\\cone\\big\\{c_{\\oi, j}\\,|\\; j\\in J(\\bar z)\\setminus\\bar J\\big\\} =  \\Omega^*.\n\\end{eqnarray*} \nSince $\\lk -\\mu^k=o(\\xik)$ and $u^k-\\ox=o(\\beta_k)$, we deduce from \\eqref{est4} and \\eqref{unb} that \n\\begin{eqnarray*}\n\\frac{ \\mu^k-d^k}{\\beta_k}+A_\\oi\\big(\\frac{z^k-\\oz}{\\beta_k}\\big)&=& \\frac{ \\mu^k-\\lk}{\\beta_k}+\\frac{ \\lk-d^k}{\\beta_k} +A_\\oi\\big(\\frac{\\Phi(u^k)-\\Phi(\\ox)}{\\beta_k}\\big)\\\\\n&&-\\frac{1}{\\rok}A_\\oi\\big(\\frac{d^k-\\lk}{\\beta_k}\\big)\\to -\\eta \\;\\;\\;\\;\\mbox{as}\\;\\;k\\to \\infty,\n\\end{eqnarray*}\nwhich in turn demonstrates that \n$-\\eta \\in \\Omega^*$. This, combined with \\eqref{est8} and $\\eta\\in E$, shows that \n\\begin{equation*}\n\\|\\eta\\|^2= \\langle \\eta, \\eta \\rangle = \\langle \\eta, P_E(\\eta) \\rangle= -\\langle \\eta, P_E(\\zeta) \\rangle= -\\big\\langle \\eta, P_E(\\zeta)+P_{E^{\\perp}}(\\zeta) \\big\\rangle = -    \\langle \\eta, \\zeta\\rangle \\le 0,\n\\end{equation*} \nimplying that  $\\eta =0$. This clearly contradicts \\eqref{est4} and thus completes the proof.\n\\end{proof}\n\n\\begin{proof}\n Define $R_k:=R(\\xk,\\lk)$   and observe that if $R_k=0$ for some $k$, then the pair $(\\xk,\\lk)$ satisfies the KKT system \\eqref{vs}, and thus Algorithm \\ref{Alg1} should stop. From now on assume that $R_k>0$ for all $k\\in\\N$. \n  Pick the positive constants $\\kappa_1$ and $\\gg_1$ from Theorem~\\ref{sooc}, the positive constants $\\gamma_2$ and $\\kappa_2$ from Proposition~\\ref{erct}, the positive constants \n  $\\gamma_3$ and $\\kappa_3$ from Theorem~\\ref{est}, and the positive constants $\\hat\\gg$ and $ \\tau$ from Proposition~\\ref{solv}. By \\eqref{error}, we can find $\\gamma_4>0$ such that\n\\begin{equation}\\label{ek}\n\\sigma(x,\\lm)\\le\\dfrac{1}{4\\kappa_1\\kappa_2}R(x,\\lm)\\quad\\textrm{ whenever }\\;(x,\\lm)\\in\\B_{\\gg_4}(\\ox,\\olm).\n\\end{equation}\nDefine further the  positive  constants  $\\bar\\tau:=\\kappa_3$ and \n\\begin{equation}\\label{const}\n\\bar\\gamma:=\\dfrac{\\gamma'}{1+2\\sqrt{2}\\bar\\tau\\kappa_2} \\quad\\mbox{with}\\quad\n\\gamma':=\\min\\Big\\{\\frac{\\hat\\gg}{2\\tau},\\gamma_3,\\gamma_4,\\dfrac{\\gamma_1}{1+\\sqrt{2}\\bar\\tau\\kappa_2},\\dfrac{\\gamma_2}{1+\\sqrt{2}\n\\bar\\tau\\kappa_2}\\Big\\}.\n\\end{equation}\nMoreover, pick $q\\in (0,1) $ and set \n\\begin{equation}\\label{rhof}\n\\bar\\varrho:=\\max\\big\\{\\bar\\rho, 4\\kappa_1\\kappa_2\\bar\\tau, \\frac{2\\sqrt{2} \\kappa_1\\kappa_2^2\\bar\\tau^2}{q}\\big\\},\n\\end{equation}\nwhere  $\\bar\\rho$ is taken from Theorem~\\ref{ugrowth}. \nArguing by induction, we aim at showing that for any starting point $(x^0,\\lambda^0)\\in\\big(\\Th\\times\\R^m\\big)\\cap\\B_{\\bar\\gamma}(\\ox,\\olm)$ there exists a sequence $\\{(\\xk,\\lk)\\c\\subset \\Th \\times\\R^m$, generated by Algorithm~\\ref{Alg1}\nwith $\\rok \\ge \\bar\\varrho$ and   $\\ek=\\sigma(\\xk,\\lk)$ for all $k$,\n such that for all $k=0,1,\\ldots$ the relationships\n\\begin{equation}\\label{c1}\n(\\xk,\\lk)\\in\\B_{\\gamma'}(\\ox,\\olm),\n\\end{equation}\n\\begin{equation}\\label{c2}\n\\dist\\big(-\\nabla_x L(\\xkk,\\lkk), N_\\Th(\\xkk)\\big)\\le\\ek\\quad\\textrm{ and }\\quad\\Vert\\xkk-\\xk\\Vert+\\Vert\\lkk-\\lk\\Vert\\le\\bar\\tau R_k\n\\end{equation}\nhold. To achieve this goal, set $k=0$, pick $(x^0,\\lambda^0)\\in\\big(\\Th\\times\\R^m\\big)\\cap\\B_{\\bar\\gamma}(\\ox,\\olm)$, and observe from  $\\bar\\gamma\\le\\gamma'$ that \\eqref{c1} holds in this case. We also have $\\lambda^0\\in\\B_{\\hat\\gg/2\\tau}(\\olm)$ and $\\rho_0\\ge \\bar\\varrho$ and then find by Proposition~\\ref{solv} a primal iterate $x^1\\in \\inte \\B_{\\hat\\gamma}(\\ox)$ satisfying $-\\nabla_x\\L(x^1,\\lambda^0,\\rho_0)\\in N_\\Th(x^1)$. Define further \nthe multiplier $\\lm^1:=\\nabla e_{1^{}/\\rho_0}g\\big(\\Phi(x^1)+ \\rho_0^{-1}\\lambda^0\\big)$ and observe that\n\\begin{equation*}\n-\\nabla_x L(x^1,\\lm^1)=-\\nabla_x\\L(x^1,\\lambda^0,\\rho_0)\\in N_\\Th(x^1).\n\\end{equation*}\nMoreover, employing  Theorem~\\ref{est} confirms that \n\\begin{equation*}\n\\Vert x^1-x^0\\Vert+\\Vert\\lm^1-\\lambda^0\\Vert\\le\\bar\\tau R_0.\n\\end{equation*}\nThus $(x^1,\\lambda^1)$ is well defined and satisfies the claimed condition \\eqref{c2} for $k=0$. Assume by induction that the iterates $(\\xk,\\lk)$, $k=0,1,\\ldots,s+1$, are well defined  and that   both conditions \\eqref{c1} and \\eqref{c2} hold for $k=0,1,\\ldots,s$. We verify now the existence of $(x^{s+2},\\lambda^{s+2})$ satisfying \\eqref{c1} and \\eqref{c2} for $k=s+1$. To this end, we first show that $(x^{s+1},\\lambda^{s+1})\\in\\B_{\\gamma'}(\\ox,\\olm)$. \nTo proceed, fix an integer   $k$ with $0\\le k\\le s$.\nSince $(\\xk,\\lk)\\in\\B_{\\gamma'}(\\ox,\\olm)$, it follows from \\eqref{eb1} and \\eqref{c1} that\n\\begin{eqnarray}\\label{c3}\nR_k&\\le&\\kappa_2\\big(\\Vert\\xk-\\ox\\Vert+\\dist(\\lk,\\Lambda(\\ox))\\big)\\\\\n&\\le&\\kappa_2\\big(\\Vert\\xk-\\ox\\Vert+\\Vert\\lk-\\olm\\Vert\\big)\\le\\sqrt{2}\\kappa_2\\Vert(\\xk,\\lk)-(\\ox,\\olm)\\Vert\\le\\sqrt{2}\\kappa_2\\gamma'.\\nonumber\n\\end{eqnarray}\nCombining this with \\eqref{c1} and the second relation in \\eqref{c2}, we conclude that\n\\begin{eqnarray}\\label{c4}\n\\Vert(\\xkk,\\lkk)-(\\ox,\\olm)\\Vert&\\le&\\Vert\\xkk-\\xk\\Vert+\\Vert\\lkk-\\lk\\Vert+\\Vert(\\xk,\\lk)-(\\ox,\\olm)\\Vert\\nonumber\\\\\n&\\le&\\bar\\tau R_k+\\gamma'\\le(\\sqrt{2}\\bar\\tau\\kappa_2+1)\\gamma'\\le\\gamma_1,\n\\end{eqnarray}\nwhich implies that $(\\xkk,\\lkk)\\in\\B_{\\gamma_1}(\\ox,\\olm)$. Using \\eqref{subr3}, we obtain\n\\begin{eqnarray}\\label{c5}\n&&\\Vert\\xkk-\\ox\\Vert+\\dist\\big(\\lkk,\\Lambda(\\ox)\\big)\\nonumber\\\\\n&\\le&\\kappa_1 R_{k+1} =\\kappa_1\\Big(\\dist\\big(-\\nabla_x L(\\xkk,\\lkk), N_\\Th(\\xkk)\\big) +\\big\\Vert\\Phi(\\xkk)-{\\rm{prox}}_g\\big(\\Phi(\\xkk)+\\lkk\\big)\\big\\Vert\\Big)\\nonumber\\\\\n&\\le&\\kappa_1\\ek+\\kappa_1\\big\\Vert\\Phi(\\xkk)-{\\rm{prox}}_g\\big(\\Phi(\\xkk)+\\lkk\\big)\\big\\Vert.\n\\end{eqnarray}\nTo proceed further,  define\n\\begin{equation}\\label{skk}\n\\skk:=\\proxk\\big(\\Phi(\\xkk)+ \\rho_k^{-1}{\\lk}\\big ).\n\\end{equation}\nIt follows from  \\eqref{lkk}  and \\eqref{ueq19} that\n\\begin{equation}\\label{eq3}\n\\Phi(\\xkk)-\\skk= \\rho_k^{-1}(\\lkk-\\lk).\n\\end{equation}\nSince $\\lkk=\\nabla\\menvk\\big(\\Phi(\\xkk)+ \\rho_k^{-1}{\\lk}\\big)$, we deduce from the second equality in \\eqref{prm} that \n\\begin{equation*}\\label{clq17}\n\\lkk   \\in \\sub g\\big(\\Phi(\\xkk)+ \\rho_k^{-1}(\\lk-\\lkk)\\big)=\\partial g(\\skk).\n\\end{equation*} \nAppealing now to the first equality in \\eqref{prm} yields $\\skk={\\rm{prox}}_g(\\skk+\\lkk)$. Since the mapping $y\\mapsto y-{\\rm{prox}}_g(y+\\lkk)$ is nonexpansive (cf. \\cite[Proposition~12.27]{bc}), we arrive at the relationships\n\\begin{eqnarray}\\label{eq2}\n&&\\big\\Vert\\Phi(\\xkk)-{\\rm{prox}}_g\\big(\\Phi(\\xkk)+\\lkk\\big)\\big\\Vert\\nonumber\\\\\n&=&\\big\\Vert\\Phi(\\xkk)-{\\rm{prox}}_g\\big(\\Phi(\\xkk)+\\lkk\\big)\\big\\Vert-\\big\\Vert\\skk-{\\rm{prox}}_g(\\skk+\\lkk)\\big\\Vert\\nonumber\\\\\n&\\le&\\big\\Vert\\Phi(\\xkk)-{\\rm{prox}}_g\\big(\\Phi(\\xkk)+\\lkk\\big)-\\big(\\skk-{\\rm{prox}}_g(\\skk+\\lkk)\\big)\\big\\Vert\\nonumber\\\\\n&\\le&\\Vert\\Phi(\\xkk)-\\skk\\Vert.\n\\end{eqnarray}\nCombining \\eqref{c5}, \\eqref{eq3}, and \\eqref{eq2} brings us to\n\\begin{equation*}\n\\Vert\\xkk-\\ox\\Vert+\\dist\\big(\\lkk,\\Lambda(\\ox)\\big)\\le\\kappa_1\\ek+\\rho_k^{-1}{\\kappa_1}\\Vert\\lkk-\\lk\\Vert.\n\\end{equation*}\nThis,  together with \\eqref{ek}, \\eqref{rhof}, the second inequality in \\eqref{c2}, and \\eqref{c3} results in \n\\begin{eqnarray}\\label{c10}\n\\Vert \\xkk-\\ox\\Vert+\\dist\\big(\\lkk,\\Lambda(\\ox)\\big)&\\le&\\kappa_1\\ek+\\dfrac{\\bar\\tau\\kappa_1}{\\rok}R_k\\le\\dfrac{1}{4\\kappa_2}R_k +\\dfrac{1}{4\\kappa_2}R_k\\\\\n&\\le&\\dfrac{1}{2}\\Big(\\Vert\\xk-\\ox\\Vert+\\dist\\big(\\lk,\\Lambda(\\ox)\\big)\\Big)\\nonumber,\n\\end{eqnarray}\nwhich in turn implies the primal-dual estimates\n\\begin{equation}\\label{c6}\n\\Vert\\xkk-\\ox\\Vert+\\dist\\big(\\lkk,\\Lambda(\\ox)\\big)\\le\\dfrac{1}{2^{k+1}}\\Big(\\Vert x^0-\\ox\\Vert+\\dist\\big(\\lambda^0,\\Lambda(\\ox)\\big)\\Big).\n\\end{equation}\nThis estimate, \\eqref{c3},  and the second inequality in \\eqref{c2} tell us that \n\\begin{eqnarray}\\label{c7}\n\\Vert(x^{s+1},\\lambda^{s+1})-(x^0,\\lambda^0)\\Vert&\\le&\\sum\\limits_{k=0}^s\\Vert(\\xkk,\\lkk)-(\\xk,\\lk)\\Vert \\le\\bar\\tau\\sum\\limits_{k=0}^s R_k\\nonumber\\\\\n&\\le&\\bar\\tau\\kappa_2\\sum\\limits_{k=0}^s\\Vert\\xk-\\ox\\Vert+\\dist\\big(\\lk,\\Lambda(\\ox)\\big)\\nonumber\\\\\n&\\le&\\bar\\tau\\kappa_2\\sum\\limits_{k=0}^s\\dfrac{1}{2^k}\\Big(\\Vert x^0-\\ox\\Vert+\\dist\\big(\\lambda^0,\\Lambda(\\ox)\\big)\\Big)\\\\\n&\\le&2\\bar\\tau\\kappa_2\\big(\\Vert x^0-\\ox\\Vert+\\dist(\\lambda^0,\\Lambda(\\ox))\\big)\\nonumber\\\\\n&\\le&2\\bar\\tau\\kappa_2\\big(\\Vert x^0-\\ox\\Vert+\\Vert\\lambda^0-\\olm\\Vert\\big).\\nonumber\n\\end{eqnarray}\nThus, we arrive at the estimates\n\\begin{eqnarray*}\n\\Vert(x^{s+1},\\lambda^{s+1})-(\\ox,\\olm)\\Vert&\\le&\\Vert(x^{s+1},\\lambda^{s+1})-(x^0,\\lambda^0)\\Vert+\\Vert(x^0,\\lambda^0)-(\\ox,\\olm)\\Vert\\\\\n&\\le&2\\bar\\tau\\kappa_2\\big(\\Vert x^0-\\ox\\Vert+\\Vert\\lambda^0-\\olm\\Vert\\big)+\\Vert(x^0,\\lambda^0)-(\\ox,\\olm)\\Vert\\\\\n&\\le&(2\\sqrt{2}\\bar\\tau\\kappa_2+1)\\Vert(x^0,\\lambda^0)-(\\ox,\\olm)\\Vert\\le(2\\sqrt{2}\\bar\\tau\\kappa_2+1)\\bar\\gamma=\\gamma',\n\\end{eqnarray*}\nwhere the last inequality follows from $(x^0,\\lambda^0)\\in\\B_{\\bar\\gamma}(\\ox,\\olm)$, while the last equality comes from the definition of $\\bar\\gamma$ in \\eqref{const}. \nThis verifies that $(x^{s+1},\\lambda^{s+1})\\in\\B_{\\gamma'}(\\ox,\\olm)$. By  \\eqref{const}, we get  $\\lambda^{s+1}\\in\\B_{\\hat\\gg/2\\tau}(\\olm)$, and hence Proposition~\\ref{solv} ensures the existence of of an optimal solution $x^{s+2}$ \nto \\eqref{regp} associated with $(\\lm^{s+1},\\rho_{s+1})$ such that $x^{s+2}\\in \\inte\\B_{\\hat\\gamma}(\\ox)$. The latter yields \n\\begin{equation*}\n-\\nabla_x\\L(x^{s+2},\\lambda^{s+1},\\rho_{s+1})\\in N_\\Th(x^{s+2}).\n\\end{equation*} \nSet now $\\lm^{s+2}:=\\nabla e_{1^{}/\\rho_{s+1}}\\big(\\Phi(x^{s+2})+\\rho_{s+1}^{-1}\\lambda^{s+1}\\big)$ and observe that\n\\begin{equation*}\n -\\nabla_x L(x^{s+2},\\lm^{s+2}) =  -\\nabla_x\\L(x^{s+2},\\lambda^{s+1},\\rho_{s+1}) \\in N_\\Th(x^{s+2}),\n\\end{equation*}\nwhich by Proposition~\\ref{est} brings us to the estimate\n\\begin{equation*}\n\\Vert x^{s+2}-x^{s+1}\\Vert+\\Vert\\lm^{s+2}-\\lambda^{s+1}\\Vert\\le\\bar\\tau R_{s+1}.\n\\end{equation*}\nThis tells us that the iterate $(x^{s+2},\\lambda^{s+2})$ satisfies   \\eqref{c2} for $k=s+1$, which completes  the proof of  induction. Thus there exists a sequence  $\\{(\\xk,\\lk)\\c\\subset\\Th\\times\\R^m$, generated by Algorithm~\\ref{Alg1} \nwith $\\rok \\ge \\bar\\varrho$ and   $\\ek=o\\big(R(\\xk,\\lk)\\big)$ for all $k$, for which  \\eqref{c1} and \\eqref{c2} fulfill.\n\n \nNext, we are going to justify  the convergence of this primal-dual   sequence. Using the same arguments as in the proofs of \\eqref{c6} and \\eqref{c7} leads us to\n\\begin{equation}\\label{c8}\n\\Vert(x^{k+l},\\lambda^{k+l})-(\\xk,\\lk)\\Vert\\le 2\\bar\\tau\\kappa_2\\big(\\Vert\\xk-\\ox\\Vert+\\dist(\\lk,\\Lambda(\\ox))\\big)\\quad\\mbox{for all}\\;\\;k,l\\in\\N.\n\\end{equation}\nIt follows from \\eqref{c6} that the right-hand side of \\eqref{c8} goes to zero as $k\\to\\infty$, which tells us that the sequence $\\{(\\xk,\\lk)\\c$ is Cauchy. By \\eqref{c6},   this sequence converges  to $(\\ox,\\hat\\lambda)$ for some $\\hat\\lambda\\in\\Lambda(\\ox)$.\nNow, we show  that  the rate of this convergence is linear. Indeed, letting $l\\to\\infty$ in \\eqref{c8} tells us that for all $k\\in\\N$ we have\n\\begin{equation*}\\label{c9}\n\\Vert(\\xk,\\lk)-(\\ox,\\hat\\lambda)\\Vert\\le 2\\bar\\tau\\kappa_2\\big(\\Vert\\xk-\\ox\\Vert+\\dist(\\lk,\\Lambda(\\ox))\\big),\n\\end{equation*}\nwhich together with the first inequality in \\eqref{c10} verifies that\n\\begin{eqnarray*}\n\\Vert(\\xkk,\\lkk)-(\\ox,\\hat\\lambda)\\Vert&\\le&2\\bar\\tau\\kappa_2\\big(\\Vert\\xkk-\\ox\\Vert+\\dist(\\lkk,\\Lambda(\\ox))\\big)\\nonumber\\\\\n&\\le&2\\bar\\tau\\kappa_1\\kappa_2\\big(R_k^{-1}{\\ek}+\\rho_k^{-1}\\bar\\tau\\big)R_k\\nonumber\\\\\n&\\le&2\\bar\\tau\\kappa_1\\kappa_2^2\\big(R_k^{-1}{\\ek}+\\rho_k^{-1}\\bar\\tau\\big)\\big(\\Vert\\xk-\\ox\\Vert+\\dist(\\lk,\\Lambda(\\ox))\\big)\\nonumber\\\\\n&\\le&2\\bar\\tau\\kappa_1\\kappa_2^2\\big(R_k^{-1}{\\ek}+\\rho_k^{-1}\\bar\\tau\\big)\\big(\\Vert\\xk-\\ox\\Vert+\\Vert\\lk-\\hat\\lambda\\Vert\\big)\\nonumber\\\\\n&\\le& 2\\sqrt{2}\\bar\\tau\\kappa_1\\kappa_2^2\\big(R_k^{-1}{\\ek}+\\rho_k^{-1}\\bar\\tau\\big)\\Vert(\\xk,\\lk)-(\\ox,\\hat\\lambda)\\Vert.\n\\end{eqnarray*}\nCombining this with $\\rok\\ge \\bar\\varrho$  with $ \\bar\\varrho$ taken from \\eqref{rhof} and using the choice of the tolerance $\\ek=o(R_k)$ result in \n\\begin{equation*}\n\\limsup_{k\\to\\infty}\\dfrac{\\Vert(\\xkk,\\lkk)-(\\ox,\\hat\\lambda)\\Vert}{\\Vert(\\xk,\\lk)-(\\ox,\\hat\\lambda)\\Vert}\\le\\limsup_{k\\to\\infty}2\\sqrt{2}\\bar\\tau\n\\kappa_1\\kappa_2^2\\big(R_k^{-1}{\\ek}+\\rho_k^{-1}\\bar\\tau\\big)\\le q.\n\\end{equation*}\nIt follows from  $q\\in (0,1)$ that the rate of   convergence of $\\{(\\xk,\\lk)\\c$ is linear. Furthermore, it is clear from the above inequality that if $\\rok\\to \\infty$, we \narrive at the superlinear convergence of the latter primal-dual sequence, which completes the proof.\n\\end{proof}","units":[{"start":20439,"end":21534,"start_line":359,"end_line":380,"raw_tex":"\\begin{Proposition}[second-order variational properties of CPLQ] \\label{sop} Assume that $g:\\R^m\\to \\oR$ is a CPLQ function with representation \\eqref{PWLQ}\nand that $\\oz\\in \\dom g$ and $\\ov\\in \\sub g(\\oz)$. Set $\\ov_i:=\\ov -A_i \\oz -a_i$ for all $i\\in I(\\oz)$. \nThen the following conditions hold:\n \\begin{itemize}[noitemsep,topsep=0pt]\n \\item [{\\rm (a)}]   the critical cone of $g$ at $\\oz$ for $\\ov$ can be calculated by $K_g(\\oz,\\ov)=N_{\\sub g(\\oz)}(\\ov)$. Moreover, we have \n\\begin{equation}\\label{cc2}\n  \\d^2 g(\\oz , \\ov ) (w)=\\d^2 g(\\oz)(w)+\\dd_{K_g(\\oz,\\ov)}(w), \\quad w\\in \\R^m.\n\\end{equation}\n \\item [{\\rm (b)}] the function $g$ is twice epi-differentiable at $\\ox$ for $\\ov$ and its second-order epi-derivative  at this point can be calculated by \n\\begin{equation}\\label{pwfor}\n \\d^2 g(\\oz , \\ov ) (w)  \n=\\begin{cases}\n\\la A_i w, w\\ra&\\mbox{if}\\;\\; w\\in {K}_{C_i}(\\oz,\\ov_i),\\\\\n\\infty&\\mbox{otherwise}.\n\\end{cases}\n\\end{equation}\nMoreover, we have \n\\begin{equation}\\label{domcr}\nK_g(\\oz,\\ov)=\\bigcup_{i\\in I(\\oz)} {K}_{C_i}(\\oz,\\ov_i).\n\\end{equation}\n  \\end{itemize}\n\\end{Proposition}","sha256_utf8":"cd541f1fc2d4d0ab68abe0cc5a1944986ac765a5d2d8f7961c3c83a33df52a68"},{"start":24206,"end":24760,"start_line":416,"end_line":423,"raw_tex":"\\begin{Theorem}[error bound via second-order optimality conditions]\\label{sooc} \n Assume that $(\\ox,\\olm)$ is a solution to the KKT system \\eqref{vs}. If the SOSC \\eqref{sosc} holds at $(\\ox,\\olm)$, then\n there are positive constants $\\gg_1$ and $\\kappa_1$ for which  the error bound estimate\n\\begin{equation}\\label{subr3}\n\\|x-\\ox\\|+\\dist\\big(\\lm,\\Lambda(\\ox)\\big)\\le\\kappa_1\\Big(\\dist\\big(-\\nabla_x L(x,\\lm),N_\\Th(x)\\big)+\\| \\Phi(x)-{\\rm{prox}}_g\\big(\\lm+ \\Phi(x)\\big)\\|\\Big)\n\\end{equation}\nholds for any $(x,\\lm)\\in\\B_{\\gg_1}(\\ox,\\olm)$.\n \\end{Theorem}","sha256_utf8":"434fac25ee63d2ded34f673c60922f4b390cdfe4b513c4c575470ee9c5c4cfcc"},{"start":26450,"end":26988,"start_line":448,"end_line":455,"raw_tex":"\\begin{Proposition}[properties of the augmented Lagrangian]\\label{paug}\nAssume that  $(x, \\lm, \\rho)\\in \\R^n\\times \\R^m\\times (0, \\infty)$ and $\\ox\\in \\R^n$. Then the following conditions hold for the  augmented Lagrangian   \\eqref{aug}:\n\\begin{enumerate}[noitemsep,topsep=0pt]\n\\item the function $x \\mapsto \\L(x, \\lm, \\rho)$ is $\\mathcal{C}^1$ and fully amenable at $\\ox$;\n\\item the function $\\lm \\mapsto \\L(x, \\lm, \\rho)$ is concave;\n\\item the function $\\rho \\mapsto \\L(x, \\lm, \\rho)$ is nondecreasing.\n\\end{enumerate}\n\\end{Proposition}","sha256_utf8":"e1523de839f3c46d7c1970d9665062a2cc32753fa32a8564771245a64afd18f2"},{"start":26991,"end":27919,"start_line":457,"end_line":462,"raw_tex":"\\begin{proof} Let $(x, \\lm, \\rho)\\in \\R^n\\times \\R^m\\times (0, \\infty)$ and $\\ox\\in \\R^n$. To justify (a), it follows from \\cite[Proposition~12.30]{rw} that the envelope function $\\menv$ is CPLQ. \nMoreover, the envelope function $\\menv$  is continuously differentiable on $\\R^m$ and thus the function $x\\mapsto e_{ {1}/{\\rho}} g \\big(\\Phi(x)+\\rho^{-1}\\lm\\big)$\nis $\\mathcal{C}^1$ and fully amenable at $\\ox$. In this case, the basic qualification condition \\eqref{bcq} holds automatically at any $\\ox\\in \\R^n$ since $\\dom \\menv=\\R^m$. Clearly the mapping $x\\mapsto \\ph(x)$\nis fully amenable at $\\ox$ since $\\ph$ is ${\\cal C}^2$. Combining these and remembering that the sum of two ${\\cal C}^1$ and fully amenable functions is fully amenable (cf. \\cite[Exercise~10.22]{rw})\nindicate that the augmented Lagrangian \\eqref{aug} is fully amenable at $\\ox$. Parts (b) and (c) were established in \\cite[Exercise~11.56]{rw}.\n\\end{proof}","sha256_utf8":"0f35b8f474b5644d3d043fd1d793de0ca3f81406524eb7485b67209547c414dc"},{"start":28057,"end":28563,"start_line":467,"end_line":476,"raw_tex":"\\begin{Proposition}[propagation of stationary conditions]\\label{fopag}\nLet $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs}. Then for any $\\rho>0$,  the following conditions are satisfied:\n\\begin{enumerate}[noitemsep,topsep=0pt]\n\\item we always have  $\\L(\\ox,\\olm,\\rho)=\\ph(\\ox)+g\\big(\\Phi(\\ox)\\big)$;\n\\item $\\ox$ is a stationary point of the augmented  problem \n\\begin{equation}\\label{ap}\n\\mini\\; \\L(x,\\olm,\\rho)\\quad \\mbox{subject to}\\;\\; x\\in \\Th.\n\\end{equation}\n\\end{enumerate}\n\\end{Proposition}","sha256_utf8":"17069c83e069b5a7bb15c07051c6fd6c6a5952731d3b254f904c63639a6f51ee"},{"start":28564,"end":29295,"start_line":477,"end_line":487,"raw_tex":"\\begin{proof} Given $\\rho>0$, it follows from the first equality in \\eqref{prm} and  $\\olm\\in \\sub g\\big(\\Phi(\\ox)\\big)$ that \n $$\n \\prox\\big(\\Phi(\\ox)+\\rho^{-1}\\olm\\big)=\\Phi(\\ox),\n $$ which clearly proves (a).\nWe proceed now with the proof of  (b). Since $g$ is convex, we conclude from the second equality in \\eqref{prm} that the condition $\\olm\\in \\sub g\\big(\\Phi(\\ox)\\big)$ is equivalent to $\\nabla(e_{ {1}/{\\rho}} g)\\big(\\Phi(\\ox)+\\rho^{-1}\\olm\\big)=\\olm$\nfor any $\\rho>0$.  Thus we get \n\\begin{equation}\\label{kkt2}\n\\nabla_x \\L(\\ox,\\olm,\\rho)=\\nabla \\ph(\\ox)+ \\nabla \\Phi(\\ox)^*\\nabla(e_{ {1}/{\\rho}} g)(\\Phi(\\ox)+\\rho^{-1}\\olm)=\\nabla_x L(\\ox,\\olm).\n\\end{equation}\nCombining these and \\eqref{vs} justifies  (b).\n\\end{proof}","sha256_utf8":"4197cd112458f282a7380fcb9448379374994f4a55472a42550b795652842ddd"},{"start":29679,"end":30576,"start_line":493,"end_line":507,"raw_tex":"\\begin{Proposition}[quadratic growth condition for augmented Lagrangians]\\label{2ndgr}\nLet $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and set $\\ov:= \\nabla_x L(\\ox,\\olm)$. Then for any $\\rho>0$,  the following conditions are equivalent: \n\\begin{enumerate}[noitemsep,topsep=0pt]\n\\item    there exists a constant $\\bar\\rho>0$ such that for any $\\rho\\ge\\bar\\rho$ the second-order condition \n\\begin{equation}\\label{eq26}\n\\d^2_x\\L\\big((\\ox,\\olm,\\rho),\\ov\\big)(w)>0\\;\\mbox{ for all }\\;w\\in K_\\Th (\\ox,-\\ov ) \\setminus\\{0\\}\n\\end{equation}\nholds;\n \\item  there exist positive constants $\\bar\\rho$, $\\gamma$, and $\\ell$ such that for any $\\rho\\ge\\bar \\rho$, the quadratic growth condition \n \\begin{equation}\\label{eq16}\n\\L(x,\\olm,\\rho)\\ge \\ph(\\ox)+g\\big(\\Phi(\\ox)\\big)+\\ell\\| x-\\bar x\\|^2\\quad \\mbox{ for all }\\;x\\in\\B_\\gg (\\ox)\\cap \\Th\n\\end{equation}\nholds. \n\\end{enumerate}\n\\end{Proposition}","sha256_utf8":"4008cc810b81a47bd139f210daec161bf084d60348fbdfdbc703706eab435604"},{"start":30579,"end":32650,"start_line":509,"end_line":533,"raw_tex":"\\begin{proof} Pick $\\rho>0$ and    $x\\in \\R^n$, and \n define the function $  f_\\rho(x) := \\L(x,\\olm, \\rho)+\\delta_\\Th(x)$.  Remember that $(\\ox,\\olm)$ is a solution to the KKT system  \\eqref{vs}.\nSince $\\partial   f_\\rho(\\ox) = \\nabla_x \\L(\\ox,\\olm,\\rho)+N_\\Th(\\ox)$, it follows from \\eqref{kkt2} that $0\\in \\sub f_\\rho(\\ox)$. \nEmploying \\cite[Proposition~13.19]{rw} and Proposition~\\ref{paug}(a), we conclude  that  \n\\begin{eqnarray}\n\\d^2  f_\\rho (\\ox, 0)(w) &=&\\d^2_x\\L\\big((\\ox,\\olm,\\rho),\\ov\\big)(w)+\\d^2\\delta_\\Th(\\ox, -\\ov)(w)\\nonumber \\\\\n&=&\\d^2_x\\L\\big((\\ox,\\olm,\\rho),\\ov\\big)(w)+\\delta_{K_\\Th(\\ox, -\\ov)}(w),\\label{sumr}\n\\end{eqnarray} \nwhere the formula for the second subderivative of $\\dd_\\Th$ was taken from \\cite[Example~3.4]{mmstams}.\n\nAfter this preparation,  assume that   (a)  holds. So we find   $\\bar \\rho>0$ for which \\eqref{eq26} is satisfied. \nThis, combined with \\eqref{sumr}, tells us that $\\d^2  f_{\\bar\\rho}(\\ox, 0)(w)>0$  for all $w\\in \\R^n\\setminus\\{0\\}$.\nEmploying now   \\cite[Theorem~13.24(c)]{rw}, we find positive constants  $\\gg$ and $\\ell$ for which the quadratic \ngrowth condition \n$$\n\\L(x,\\olm,\\bar \\rho)\\ge \\L(\\ox,\\olm,\\bar \\rho) +\\ell\\| x-\\bar x\\|^2 =\\ph(\\ox)+g\\big(\\Phi(\\ox)\\big)+\\ell\\| x-\\bar x\\|^2\n$$\nholds for all $x\\in\\B_\\gg (\\ox)\\cap \\Th$, where the last equality comes from Proposition~\\ref{fopag}(a). By Proposition~\\ref{paug}(c), the mapping $\\rho\\mapsto \\L(x,\\olm,  \\rho)$ is nondecereasing. Combining this with the above quadratic growth condition justifies \\eqref{eq16} for any $\\rho\\ge \\bar\\rho$.\n\nTurning now to the opposite implication, assume that (b) holds and then pick $\\rho\\in [\\bar\\rho, \\infty)$. By  Proposition~\\ref{fopag}(a), we have $f_\\rho(\\ox)= \\ph(\\ox)+g\\big(\\Phi(\\ox)\\big)$, which yields \n$$\nf_\\rho(x) \\ge f_\\rho(\\ox)+\\ell\\| x-\\bar x\\|^2\\quad \\mbox{ for all }\\;x\\in\\B_\\gg (\\ox).\n$$\nThis implies via the definition of the second subderivative that $\\d^2  f_{\\rho}(\\ox, 0)(w)>0$  for all $w\\in \\R^n\\setminus\\{0\\}$. Appealing now to \\eqref{sumr} verifies \\eqref{eq26} and hence ends the proof.\n\\end{proof}","sha256_utf8":"4fed42d2728b346b0dc4425c8e11e7065a8f2fa5269392c0217b2005b3bd400c"},{"start":33721,"end":34460,"start_line":544,"end_line":552,"raw_tex":"\\begin{Theorem}[second subderivatives of augmented Lagrangians]\\label{ssc} \nAssume that  $(x, \\lm, \\rho)\\in \\R^n\\times \\R^m\\times (0, \\infty)$ and $\\ox\\in \\R^n$. \nThen  the function $x\\mapsto\\L(x,\\lm,\\rho)$, defined via the augmented Lagrangian \\eqref{aug}, is twice epi-differentiable at $\\ox$ for $\\ov:=\\nabla_x \\L(\\ox,\\lm,\\rho)$ and its second-order epi-derivative \ncan be calculated for any $w\\in \\R^n$ by\n\\begin{equation}\\label{eq40}\n\\d^2_x\\L\\big((\\ox,\\lm,\\rho),\\ov\\big)(w)=\\big\\langle \\nabla^2_{xx}L(\\ox,\\mu)w,w\\big\\rangle+ e_{1/{2\\rho}}\\big(\\d^2 g\\big(\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu),\\mu\\big)\\big) \\big(\\nabla \\Phi(\\ox)w \\big), \n\\end{equation}\nwhere $\\mu=\\nabla_x \\big(e_{1/{\\rho}}g\\big)\\big(\\Phi(\\ox)+\\rho^{-1}\\lm\\big)$.\n\\end{Theorem}","sha256_utf8":"6bc46b104c9de17e743c5d2e238f5e5c5958b6caea88c8016efbb486891b8a79"},{"start":34461,"end":36985,"start_line":553,"end_line":578,"raw_tex":"\\begin{proof}  Given $\\rho>0$,  it follows from  the second equality in \\eqref{prm} that $\\mu\\in \\sub g\\big(\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu)\\big)$. \nAccording to Proposition~\\ref{sop}(b), the CPLQ function $g$ is twice epi-differentiable at $\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu)$ for $\\mu$. Appealing now \nto  \\cite[Proposition~4.1]{hms20} tells us that    the   envelope function $e_{1/\\rho} g$ is twice epi-differentiable at $\\Phi(\\ox)+\\rho^{-1}\\lm$ for $\\mu$ and that \n\\begin{equation}\\label{more}\n\\d^2 (e_{1/\\rho} g )\\big(\\Phi(\\ox)+\\rho^{-1}\\lm,\\mu\\big)=e_{1/2\\rho}\\big(\\d^2 g(\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu),\\mu)\\big).\n\\end{equation}\nBy \\eqref{pwfor}, the second-order epi-derivative of $g$ is CPLQ. This, combined with \\eqref{more}, indicates that $ e_{1/2\\rho}\\big(\\d^2 g(\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu),\\mu)\\big)$\nis finite on $\\R^m$. Employing now \\cite[Theorem~4.3]{r20}, we conclude that  $e_{1/\\rho}g$ is twice semidifferentiable at $\\Phi(\\ox)+\\rho^{-1}\\lm$ and its second semiderivative \ncoincides with its second-order epi-derivative, namely \n\\begin{equation}\\label{ssed}\n\\d^2\\big(e_{1/\\rho}g\\big)\\big(\\Phi(\\ox)+\\rho^{-1}\\lm\\big)=\\d^2 (e_{1/\\rho} g )\\big(\\Phi(\\ox)+\\rho^{-1}\\lm,\\mu\\big).\n\\end{equation}\nThis, together with the chain and sum rules for twice semidifferentiability, established in  \\cite[Propositions~8.2(i) and 8.1]{mmstams}, \nshows that the mapping $x\\mapsto\\L(x,\\lm,\\rho)$ is twice semidifferentiable at $\\ox$. Thus, by   \\cite[Propositions~8.2(iii)]{mmstams}, the latter mapping is twice epi-differentiable at $\\ox$ for $\\ov$ and \n\\begin{equation}\\label{lss}\n\\d_x^2{\\L}(\\ox,\\lm,\\rho)(w)=\\d^2_x\\L\\big((\\ox,\\lm,\\rho),\\ov\\big)(w), \\quad w\\in \\R^n.\n\\end{equation}\nBy  \\cite[Propositions~8.2(i) and 8.1]{mmstams},  the second  semiderivative of $x\\mapsto\\L(x,\\lm,\\rho)$ at $\\ox$ can be calculated for any $w\\in \\R^n$ by\n\\begin{eqnarray*}\n\\d_x^2{\\L}(\\ox,\\lm,\\rho)(w)&=&\\d^2\\ph(\\ox)(w)+\\big\\la\\mu,\\nabla^2\\Phi(\\ox)(w,w)\\big\\ra+\\d^2\\big(e_{1/\\rho} g\\big)\\big(\\Phi(\\ox)+\\rho^{-1}\\lm\\big)\\big(\\nabla \\Phi(\\ox)w\\big)\\\\\n&=&\\la\\nabla^2\\ph(\\ox)w,w\\ra+\\big\\la\\mu,\\nabla^2\\Phi(\\ox)(w,w)\\big\\ra+\\d^2\\big(e_{1/\\rho} g\\big)\\big(\\Phi(\\ox)+\\rho^{-1}\\lm,\\mu\\big)\\big(\\nabla \\Phi(\\ox)w\\big)\\\\\n&=&\\big\\la\\nabla_{xx}^2L(\\ox,\\mu)w,w\\big\\ra+e_{1/2\\rho}\\big(\\d^2 g(\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu),\\mu)\\big)\\big(\\nabla \\Phi(\\ox)w\\big),\n\\end{eqnarray*}\nwhere the second equality comes from \\eqref{ssed} and the last one results from \\eqref{more}. This together with \\eqref{lss}\njustifies \\eqref{eq40} and hence completes the proof.\n\\end{proof}","sha256_utf8":"ba1e1a1aa8bfab836fede0462fca1954d1c803456d95d3d890e4177545055516"},{"start":37207,"end":37828,"start_line":582,"end_line":588,"raw_tex":"\\begin{Corollary}[second-order epi-derivatives] Let $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and set $\\ov:= \\nabla_x L(\\ox,\\olm)$. Then for any $\\rho>0$,\nthe function $x\\mapsto\\L(x,\\lm,\\rho)$, defined via the augmented Lagrangian \\eqref{aug}, is twice epi-differentiable at $\\ox$ for $\\ov $ and its second-order epi-derivative \ncan be calculated for any $w\\in \\R^n$ by\n\\begin{equation}\\label{eq402}\n\\d^2_x\\L\\big((\\ox,\\olm,\\rho),\\ov\\big)(w)=\\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w,w\\big\\rangle+ e_{1/{2\\rho}}\\big(\\d^2 g\\big(\\Phi(\\ox),\\olm\\big)\\big) \\big(\\nabla \\Phi(\\ox)w \\big).\n\\end{equation}\n\\end{Corollary}","sha256_utf8":"5490fbb7ea56ef512ffdfa8ad43d02160b6454c87019d3938bd46418d1b90139"},{"start":37829,"end":38439,"start_line":589,"end_line":595,"raw_tex":"\\begin{proof} Because  $(\\ox,\\olm)$ is a solution to the KKT system \\eqref{vs}, we get $\\olm\\in \\sub g\\big(\\Phi(\\ox)\\big)$.\nSince $g$ is CPLQ, we conclude from the second equality in \\eqref{prm} that the latter inclusion is equivalent to $\\nabla(e_{ {1}/{\\rho}} g)\\big(\\Phi(\\ox)+\\rho^{-1}\\olm\\big)=\\olm$\nfor any $\\rho>0$. This means that $\\mu=\\olm$, where $\\mu$ was taken from Theorem~\\ref{ssc}. Moreover, it follows from \n\\eqref{kkt2} that $\\nabla_x \\L(\\ox,\\olm,\\rho)=\\nabla_x L(\\ox,\\olm)=\\ov$.\nCombining these and the second-order epi-derivative formula \\eqref{eq40}, \nwe arrive at \\eqref{eq402}.\n\\end{proof}","sha256_utf8":"7457ecd416ad05f4c48e96b444fb431ae3cba3d8e39c65d86737b154dc75d796"},{"start":39169,"end":39810,"start_line":611,"end_line":619,"raw_tex":"\\begin{Theorem}[characterizations of quadratic growth condition]\\label{growth}\nLet $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and set $\\ov:= \\nabla_x L(\\ox,\\olm)$. Then the following conditions are equivalent: \n \\begin{enumerate}[noitemsep,topsep=0pt]\n \\item  the SOSC \\eqref{sosc} holds at $(\\ox,\\olm)$;\n \\item   there exists a constant $\\bar\\rho>0$ such that for any $\\rho\\ge\\bar\\rho$ the second-order condition \\eqref{eq26} is satisfied;\n \\item there exist positive constants $\\bar\\rho$, $\\gg$, and $\\ell$ such that for any $\\rho\\ge\\bar \\rho$, the quadratic growth condition \\eqref{eq16} \nholds.\n\\end{enumerate}\n\\end{Theorem}","sha256_utf8":"93d7697ab5a7c69b2bc405d4ec6f00f602ac5a2de6711ff371be7027eb5c1c6d"},{"start":39811,"end":42733,"start_line":620,"end_line":661,"raw_tex":"\\begin{proof} The equivalence between (b) and (c) was already established in Proposition~\\ref{2ndgr}.  Suppose that (b) is satisfied. Since we always have \n$$\ne_{1/{2\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\olm)\\big)(\\nabla \\Phi(\\ox)w)\\le \\d^2 g(\\Phi(\\ox),\\olm)(\\nabla \\Phi(\\ox)w)\\quad \\mbox{for all}\\;\\; \\rho>0, \n$$\nthe validity of the second-order condition \\eqref{eq402} yields the SOSC \\eqref{sosc} and so proves (a). \n\nAssume now that (a) holds. Since the second subderivative is lower semicontinuous and positive homogeneous of degree $2$ (cf. \\cite[Proposition~13.5]{rw}), the SOSC \\eqref{sosc} amounts to the existence of \na positive constant $\\ell$ so that we have \n\\begin{equation}\\label{som}\n\\langle\\nabla_{xx}^2L(\\bar x,\\olm)w,w\\rangle+\\d^2g (\\Phi(\\bar x),\\olm) (\\nabla \\Phi(\\ox)w )\\ge \\ell \\quad \\mbox{for all }\\; w\\in \\D\\cap \\S,\n\\end{equation}\nwhere $\\S$ stands for the unit sphere in $\\R^n$. Let $w\\in \\S$ and denote the left-hand side of \\eqref{som} by $h(w)$.\n  Given $\\rho>0$ and $w\\in \\R^n$, define the function $h_\\rho:\\R^n\\to \\R$ by \n$$\nh_\\rho(w)=\\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w,w\\big\\rangle+ e_{1/{2\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\olm)\\big)(\\nabla \\Phi(\\ox)w).\n$$\nWe understand from \\cite[Theorem~1.25]{rw} that \n$$\nh_\\rho(w) \\nearrow h(w)\\;\\; as \\;\\; \\rho \\nearrow \\infty\n$$\n for all $w\\in\\R^n$. Employing now \\cite[Proposition~7.4]{rw} tells us that  $h_\\rho\\overset {\\ss \\mbox{e}}{\\to} h$ as $\\rho \\nearrow \\infty$.\n This, combined with \\cite[Theorem~7.46]{rw}, implies that $h_\\rho+\\dd_{\\S\\cap K_\\Th(\\ox,-\\ov)}\\overset {\\ss \\mbox{e}}{\\to} h+ \\dd_{\\S\\cap K_\\Th(\\ox,-\\ov)}$ as  $\\rho \\nearrow \\infty$.\n Appealing now to \\cite[Theorem~7.33]{rw} indicates that \n\\begin{equation}\\label{infco}\n \\inf \\big\\{  h_\\rho(w)|\\; w\\in \\S\\cap K_\\Th(\\ox,-\\ov)\\big\\} \\to \\inf\\big\\{ h(w)|\\; w\\in \\S\\cap K_\\Th(\\ox,-\\ov)\\big\\}\\;\\;\\mbox{as}\\;\\;\\rho \\nearrow \\infty.\n\\end{equation}\nIt follows from \\eqref{pwfor} and \\eqref{domcr} that  for any $w\\in \\R^n$ with $\\nabla \\Phi(\\ox)w \\notin K_g(\\Phi(\\ox),\\olm)$, we have  $\\d^2g (\\Phi(\\bar x),\\olm) (\\nabla \\Phi(\\ox)w )=\\infty$.\nThis,  \\eqref{som}, and the definition of the convex cone $\\D$ from \\eqref{coned}  bring us to \n$$\n\\inf\\big\\{ h(w)|\\; w\\in \\S\\cap K_\\Th(\\ox,-\\ov)\\big\\}\\ge \\ell.\n$$\nCombining this and \\eqref{infco}, we can find a positive constant $ \\bar \\rho$ for which we have \n\\begin{equation}\\label{rb1}\nh_\\rho(w)>\\frac{\\ell}{2}\\quad \\mbox{for all}\\;\\;\\rho\\ge  \\bar \\rho\\;\\;\\mbox{and all}\\;\\;w\\in    \\S\\cap K_\\Th(\\ox,-\\ov).\n\\end{equation}\nGiven $\\rho\\ge \\bar\\rho$, it follows from \\eqref{eq402} and the definition of $h_\\rho$  that \n$$\n\\d^2_x\\L\\big((\\ox,\\olm,\\rho),\\ov\\big)(w) >0\\quad \\mbox{for all} \\;\\; w\\in \\S\\cap K_\\Th(\\ox,-\\ov).\n$$\nRemember that the second-order epi-derivative  is positive homogeneous of degree $2$. So we arrive at the second-order condition \\eqref{eq26} for any $\\rho\\ge \\bar\\rho $. \nThis proves (b) and hence completes the proof.\n\\end{proof}","sha256_utf8":"70af987513278af7fa8141f0f37c3d43d9d149b62640b35f59de45e1bd333adf"},{"start":44440,"end":45145,"start_line":682,"end_line":693,"raw_tex":"\\begin{Lemma}[second-order epi-derivatives of augmented Lagrangians] \\label{usog}\nLet $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and set $\\ov:= \\nabla_x L(\\ox,\\olm)$. \nThen the following conditions are equivalent: \n \\begin{enumerate}[noitemsep,topsep=0pt]\n \\item the  SOSC \\eqref{sosc} holds at $(\\ox,\\olm)$;\n \\item there exist positive constants $\\bar\\rho$, $\\ell_1$, $\\ve_1$ such that for all $\\rho\\ge\\bar\\rho$ and\nall  $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ we have\n\\begin{equation}\\label{eq44}\n\\d^2_x\\L \\big((\\ox,\\lm,\\rho), \\nabla_x L(\\ox,\\lm)\\big)(w)\\ge\\ell_1\\|w\\|^2\\;\\mbox{ for all }\\;w\\in K_\\Th\\big(\\ox, - \\nabla_x L(\\ox,\\olm)\\big).\n\\end{equation}\n\\end{enumerate}\n\\end{Lemma}","sha256_utf8":"55fea279c03c42ece5cef5f74592da9d820940e12af6adaf8e4117629b9a39ff"},{"start":45146,"end":49544,"start_line":694,"end_line":740,"raw_tex":"\\begin{proof} The implication (b)$\\implies$(a)   results directly from Theorem~\\ref{growth}. To prove the opposite implication, assume that (a) holds. \nIt follows from Theorem~\\ref{growth} that there exists a constant $\\bar\\rho>0$ such that for all $\\rho\\ge\\bar\\rho$ the second-order condition \\eqref{eq26} is satisfied. \nRecall from  \\cite[Proposition~13.5]{rw} that  the second-order epi-derivative  is lower semicontinuous and positive homogenous of degree $2$. \nThus the second-order condition  \\eqref{eq26} is equivalent  to the existence of a constant $\\ell_1>0$ such that the estimate \n\\begin{equation}\\label{eq41}\n\\d^2_x\\L \\big((\\ox,\\olm,\\bar\\rho),\\ov \\big)(w)=\\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w, w\\big\\rangle+ e_{ {1}/{2\\bar\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\olm)\\big)(\\nabla\\Phi(\\ox)w)\\ge 2\\ell_1\n\\end{equation}\nholds for all $w\\in \\S\\cap K_\\Th(\\ox, -\\ov)$, where the equality in \\eqref{eq41} comes from  \\eqref{eq40}. We proceed now by proving the following claim:\n\n{\\bf Claim A.} {\\em There exists an $\\varepsilon_1>0$ such that for any $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ the estimate\n\\begin{equation}\\label{clq2}\n\\d^2_x\\L \\big((\\ox,\\lm,\\bar\\rho), \\nabla_x L(\\ox,\\lm) \\big)(w)=\\big\\langle \\nabla^2_{xx}L(\\ox,\\lm)w,w\\big\\rangle+ e_{ {1}/{2\\bar\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\lm)\\big)(\\nabla\\Phi(\\ox)w)\\ge \\ell_1\n\\end{equation} \n holds for all $w\\in \\S\\cap K_\\Th\\big(\\ox, -\\nabla_x L(\\ox,\\lm)\\big)$.}\n \n  To furnish this claim, choose $\\varepsilon_1\\in (0, {\\ell_1}/{\\|\\nabla^2\\Phi(\\ox)\\|})$ if $\\|\\nabla^2\\Phi(\\ox)\\|\\neq 0$ and $\\varepsilon_1 > 0$ otherwise\n  and let $w\\in \\S$.\n Thus for all     $\\lm\\in \\B_{\\ve_1}(\\olm)$, we obtain   \n\\begin{eqnarray}\\label{clq3}\n\\big\\langle \\nabla^2_{xx}L(\\ox,\\lm)w, w\\big\\rangle &=& \\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w, w\\big\\rangle+\\big\\langle\\lm -\\olm, \\nabla^2  \\Phi (\\ox)(w, w)\\big\\rangle\\nonumber\\\\\n&\\geq&\\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w, w\\big\\rangle - \\|\\nabla^2\\Phi(\\ox)\\|\\cdot\\|\\lm - \\olm\\|\\nonumber\\\\\n&\\geq&\\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w, w\\big\\rangle - \\ell_1.\n\\end{eqnarray}\nMoreover, it follows from \\eqref{cc2} that \n\\begin{eqnarray}\\label{clq4}\ne_{ {1}/{2\\bar\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\lm)\\big)(\\nabla\\Phi(\\ox)w)&=&\\inf_{v\\in \\R^m}\\big\\{\\d^2g\\big(\\Phi(\\ox)\\big)(v)+\\delta_{K_g(\\Phi(\\ox), \\lm)}(v)+\\bar\\rho\\|v-\\nabla\\Phi(\\ox)w\\|^2\\big\\}\\nonumber\\\\ \n&=& \\inf_{v\\in K_g(\\Phi(\\ox), \\lm)}\\big\\{\\d^2g \\big(\\Phi(\\ox) \\big)(v)+\\bar\\rho\\|v-\\nabla\\Phi(\\ox)w\\|^2\\big\\}. \n\\end{eqnarray}\nNote that $\\d^2g(\\Phi(\\ox))$ does not depend on $\\lambda$. Shrinking $\\ve_1$ if necessary and recalling that $\\sub g\\big(\\Phi(\\ox)\\big)$ is a polyhedral convex set, we conclude from   Proposition~\\ref{sop}(a) that  \nthe relationships \n\\begin{equation*}\nK_g(\\Phi(\\ox), \\lm) = N_{\\partial g(\\Phi(\\ox))}(\\lm) \\subset N_{\\partial g(\\Phi(\\ox))}(\\olm) = K_g(\\Phi(\\ox), \\olm) \n\\end{equation*} \nhold for all $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$.  These, together with  \\eqref{clq4}, tell us  that for any $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ we have \n\\begin{equation}\\label{clq5}\ne_{ {1}/{2\\bar\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\lm)\\big)(\\nabla\\Phi(\\ox)w) \\geq e_{ {1}/{2\\bar\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\olm)\\big)(\\nabla\\Phi(\\ox)w).\n\\end{equation} \n Combining \\eqref{eq41}, \\eqref{clq3}, and \\eqref{clq5} verifies   \\eqref{clq2} for all $w\\in \\S\\cap K_\\Th(\\ox, -\\ov)$ and   all $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$. \n To justify \\eqref{clq2} for all $w\\in \\S\\cap  K_\\Th \\big(\\ox, -\\nabla_x L(\\ox, \\lm) \\big)$, observe that by the  polyhedrality of $\\Th$,   we can assume without loss of generality that \n $$\n K_\\Th \\big(\\ox, -\\nabla_x L(\\ox, \\lm) \\big)=N_{N_\\Th(\\ox)}\\big(-\\nabla_x L(\\ox, \\lm)\\big) \\subset N_{N_\\Th(\\ox)}\\big(-\\nabla_x L(\\ox, \\olm)\\big) = K_\\Th(\\ox, -\\ov)\n $$\n  for all $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$. This clearly justifies Claim A. \n  \nUsing the positive homogeneity of the second-order epi-derivative yields \\eqref{eq44} for $\\rho = \\bar\\rho$ and for all $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$. Recall that the function $$\\rho\\mapsto e_{ {1}/{2\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\lm)\\big)(\\nabla\\Phi(\\ox)w)$$ is nondecreasing. So is the function $\\rho \\mapsto\\d^2\\L( (\\ox, \\lm, \\rho), \\nabla_x L(\\ox, \\lm))(w)$. This  \nyields  \\eqref{eq44}  for all $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ and all $\\rho \\geq \\bar\\rho$, and hence completes the   proof.\n\\end{proof}","sha256_utf8":"5138874805de656bbd7ac89695e35e41136e7834f4b37d20cd8307668f694357"},{"start":49685,"end":50374,"start_line":744,"end_line":754,"raw_tex":"\\begin{Theorem}[uniform quadratic growth condition for augmented Lagrangians]\\label{ugrowth}\nLet $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs}. Then the following conditions are equivalent:\n \\begin{enumerate}[noitemsep,topsep=0pt]\n \\item    the  SOSC \\eqref{sosc} holds at $(\\ox,\\olm)$;\n \\item there are positive constants $\\bar\\rho, \\gamma, \\varepsilon,\\ell$ such that for all $\\lm\\in\\Lambda(\\ox)\\cap\\B_\\varepsilon(\\olm)$ and all $\\rho\\ge\\bar\\rho$  the uniform quadratic growth condition\n\\begin{equation}\\label{eq14}\n\\L(x,\\lm,\\rho)\\ge \\ph(\\ox)+g(\\Phi(\\ox))+\\ell\\| x-\\ox\\|^2\\;\\mbox{ for all }\\;x\\in\\B_\\gamma(\\ox)\\cap \\Th\n\\end{equation}\nis satisfied.\n\\end{enumerate}\n\\end{Theorem}","sha256_utf8":"83c468b4ca006e3d37ddba8824534654eecbff0277261073c840ca3342747595"},{"start":50377,"end":55194,"start_line":756,"end_line":805,"raw_tex":"\\begin{proof} The implication (b)$\\implies$(a)   results directly from Theorem~\\ref{growth}. To prove the opposite implication, assume that (a) holds. \nIt follows from Lemma~\\ref{usog} that there exist the positive constants $\\ell_1$, $\\varepsilon_1$, and $\\bar\\rho$  for which \\eqref{eq44} is satisfied  for all \n $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ and all $\\rho\\ge\\bar\\rho$. Using this and  Proposition~\\ref{fopag}(a), \n we deduce from \\eqref{eq44} that for any $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ there exists $\\gamma_{\\lm}>0$ for which we have \n\\begin{equation}\\label{eq22}\n\\L(x,\\lm,\\bar\\rho)\\ge \\ph(\\ox)+g(\\Phi(\\ox))+\\dfrac{\\ell_1}{2}\\Vert x-\\ox\\Vert^2\\;\\textrm{ for all }\\;x\\in\\B_{\\gamma_\\lm}(\\ox)\\cap \\Th,\n\\end{equation}\nwhere the constant $ {\\ell_1}$ can be chosen the same for all the multipliers $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$. The\nlatter results directly from the definition of the second subderivative. The radii of the balls centered at $\\ox$ in \\eqref{eq22}, however,  depend on $\\lm$. We argue below that  a common radius can be chosen for all the multipliers $\\lm\\in\\Lambda(\\ox)$ that are sufficiently close to $\\olm$. To this end, assume that \n$$\n\\tilde\\B_{\\varepsilon_1}(\\olm)=\\big\\{\\lm\\in \\R^m|\\; \\|\\lm-\\olm\\|_1\\le \\ve_1\\big\\}\n$$\nwhere $\\|\\lm-\\olm\\|_1$ stands for $L_1$-norm of the vector $\\lm-\\olm$, namely the sum of the absolute values of the components of $\\lm-\\olm$. Clearly, we have $\\tilde\\B_{\\varepsilon_1}(\\olm)\\subset \\B_{\\varepsilon_1}(\\olm)$.  Define the function $\\psi\\colon\\R^m\\to\\oR$ by\n\\begin{equation}\\label{eq23}\n\\psi(\\lm):=\\sup_{x\\in\\big(\\B_{\\gamma_{\\olm}}(\\ox)\\cap \\Th\\big)\\setminus \\{\\ox\\}}\\frac{\\ph(\\ox)+g(\\Phi(\\ox))-\\L(x,\\lambda,\\bar\\rho)}{\\Vert x-\\ox\\Vert^2}\n+\\delta_{\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm)} (\\lambda),\\quad\\lm\\in\\R^m.\n\\end{equation}\nBy Proposition~\\ref{paug}(b), the function $\\lambda\\mapsto\\L(x,\\lm,\\bar\\rho)$ is concave. This, combined with the convexity of the set $\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm)$, \n guarantees  that $\\psi$ in \\eqref{eq23} is a convex function. Moreover, $\\psi$ is lower semicontinuous since  the first term on the right-hand side of \\eqref{eq23} is the pointwise supremum of a collection of continuous functions.\n Now we claim  that for any $\\lm\\in\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm)$ the value $\\psi(\\lm)$ is finite. To justify this claim, let $\\lm\\in\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm)$. If  $\\gamma_\\lm\\ge\\gamma_{\\olm}$,  we conclude from    \\eqref{eq22} that \n\\begin{equation*}\n\\psi(\\lm)\\le\\sup_{x\\in \\big(\\B_{\\gamma_{\\lm}}(\\ox)\\cap \\Th\\big)\\setminus \\{\\ox\\}}\\frac{\\ph(\\ox)+g(\\Phi(\\ox))-\\L(x,\\lm,\\bar\\rho)}{\\Vert x-\\ox\\Vert^2}\\le-\\dfrac{\\ell_1}{2},\n\\end{equation*}\nwhich, in particular,   implies that $\\psi(\\olm)\\le- {\\ell_1}/{2}$. On the other hand, if $\\gamma_\\lm<\\gamma_{\\olm}$, we get  \n\\begin{eqnarray*}\n &&\\psi(\\lm) \\le\\\\\n&&  \\max\\left\\{\\sup_{x\\in \\big(\\B_{\\gamma_{\\lm}}(\\ox)\\cap \\Th\\big)\\setminus \\{\\ox\\}}\\frac{\\ph(\\ox)+g(\\Phi(\\ox))-\\L(x,\\lm,\\bar\\rho)}{\\Vert x-\\ox\\Vert^2},\\max_{\\substack{\\gamma_\\lm\\le\\Vert x-\\ox\\Vert\\le\\gamma_{\\olm}\\\\ x\\in \\Th}}\\dfrac{\\ph(\\ox)+g(\\Phi(\\ox))-\\L(x,\\lm,\\bar\\rho)}{\\Vert x-\\ox\\Vert^2}\\right\\}.\n\\end{eqnarray*}\nBy \\eqref{eq22}, the first term inside the maximum is bounded above by  $- {\\ell_1}/{2}$.  Also, the second term is finite since it is the maximum of a continuous function over a compact set. \nCombining these tells us that $\\psi(\\lm)$ is finite for all $\\lm\\in\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm)$, which  results in \n$$\n\\dom\\psi=\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm).\n$$\nSince both sets  $\\Lambda(\\ox)$ and $\\tilde\\B_{\\varepsilon_1}(\\olm)$ are polyhedral, $\\dom\\psi$ is   a compact polyhedral convex set.\nAppealing now to   \\cite[Theorem~2.35]{rw} ensures that  $\\psi$ is continuous relative to its domain.  \nThus, we can find  $\\varepsilon >0$ such that $\\B_\\varepsilon(\\olm)\\subset \\tilde\\B_{\\varepsilon_1}(\\olm)$   and that\n\\begin{equation*}\n\\psi(\\lm)\\leq \\psi(\\olm)+\\frac{\\ell_1}{4}\\leq -\\frac{\\ell_1}{4}  \n\\end{equation*} \nfor all $\\lm \\in \\dom\\psi\\cap \\B_\\varepsilon(\\olm)=\\Lambda(\\ox)\\cap \\B_\\varepsilon(\\olm)$. This, combined with the definition of $\\psi$ in \\eqref{eq23}, tells us that for \nall $x\\in\\B_{\\gamma_{\\olm}}(\\ox)\\cap \\Th$ and all $\\lm\\in\\Lambda(\\ox)\\cap\\B_\\varepsilon(\\olm)$ we have \n\\begin{equation}\\label{eq25}\n\\L(x,\\lm,\\bar\\rho)\\ge \\ph(\\ox)+g(\\Phi(\\ox))+\\frac{\\ell_1}{4}\\Vert x-\\bar x\\Vert^2.\n\\end{equation}\nRemember that by  Proposition~\\ref{paug}(c) we always have \n\\begin{equation*}\n\\L(x,\\lm,\\rho)\\ge\\L(x,\\lm,\\bar\\rho)\\;\\mbox{ for all}\\;\\;\\rho\\ge\\bar\\rho.\n\\end{equation*}\nCombining this with \\eqref{eq25} and setting $\\ell:= {\\ell_1}/{4}$ verify the uniform quadratic growth condition \\eqref{eq14} and hence complete the proof.\n\\end{proof}","sha256_utf8":"a36125c20f38934e4565ddfbbd4004f6bfaa9d3ee51f7b428778734a3007dcb8"},{"start":59607,"end":60538,"start_line":866,"end_line":878,"raw_tex":"\\begin{Proposition}[solvability of subproblems]\\label{solv} \nLet $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and let  the  SOSC \\eqref{sosc} hold at $(\\ox,\\olm)$.\nThen there exist positive constants      $\\tau$ and $\\hat\\gamma\\le \\gg$  with $\\gg$ taken from Theorem~{\\rm \\ref{ugrowth}}  such that for any $\\rho\\ge \\bar\\rho $ with $\\bar\\rho$ taken from Theorem~{\\rm\\ref{ugrowth}},\n the optimal solution mapping $S_\\rho:\\R^m\\to \\R^n$, defined by \n\\begin{equation}\\label{eq177}\nS_\\rho(\\lambda):=\\argmin\\big\\{\\L(x,\\lambda,\\rho)\\;\\big|\\; x\\in\\Th\\cap\\B_{\\hat\\gamma}(\\ox)\\big\\}, \\;\\;\\lm\\in \\R^m,\n\\end{equation}\nenjoys the uniform  isolated calmness property \n\\begin{equation}\\label{pt9}\n   S_\\rho(\\lambda) \\subset\\{\\ox\\} +\\tau \\Vert\\lambda-\\olm\\Vert\\B\n\\end{equation}\n and satisfies the condition $\\emp \\neq S_\\rho(\\lambda)\\subset  \\inte  \\B_{\\hat\\gamma}(\\ox)$ for all $\\lm\\in \\B_{\\hat\\gamma/2\\tau}(\\olm)$.  \n \\end{Proposition}","sha256_utf8":"7ee844e9a8198dfcb67e8a2a699fb9c704f8587478b8eb88971f570ab5166aed"},{"start":60540,"end":64606,"start_line":880,"end_line":934,"raw_tex":"\\begin{proof} Since the SOSC \\eqref{sosc} is satisfied at $(\\ox,\\olm)$, it follows from Theorem~\\ref{ugrowth}\nthat there exist positive constants $\\bar\\rho$, $\\gg$, and $\\ell$ such that for any $\\rho\\ge\\bar \\rho$, the uniform quadratic growth condition \\eqref{eq14} holds for $\\lm=\\olm$. Moreover, \nby the twice differentiability of  $\\Phi$  at $\\ox$, we find some  constants $\\hat\\gamma\\in(0,\\gamma]$ and $\\kappa_\\Phi>0$ for which we have \n\\begin{equation}\\label{eq49}\n\\Vert\\Phi(x)-\\Phi(\\ox)\\Vert\\le\\kappa_\\Phi\\Vert x-\\ox\\Vert\\;\\textrm{ for all }\\;x\\in\\B_{\\hat\\gamma}(\\ox).\n\\end{equation}\nClearly,  the uniform quadratic growth condition \\eqref{eq14} for $\\lm=\\olm$ indicates  that  $S_\\rho(\\olm)\\cap \\B_{\\gamma}(\\ox)=\\{\\ox\\}$ for all $\\rho\\ge\\bar\\rho$. \nDefine   the positive constant\n\\begin{equation}\\label{eq21}\n\\tau:=\\frac{\\kappa_\\Phi}{\\ell}+\\sqrt{\\frac{\\kappa_\\Phi^2}{\\ell^2}+\\frac{1}{\\ell\\bar\\rho}},\n\\end{equation}\nand fix   $\\lambda\\in \\B_{\\hat\\gamma/2\\tau} (\\olm)$ and $\\rho\\geq\\bar\\rho$.\nObserve from the classical Weierstrass theorem that $S_\\rho(\\lm)\\neq\\emp$.   Assume that  $u\\in S_\\rho(\\lambda)$.\nBy  Proposition~\\ref{paug}(b),   the function $\\lambda\\mapsto\\L(u,\\lambda,\\rho)$ is concave. \nBy \\cite[Theorem~2.26(b)]{rw}, we have \n\\begin{equation}\\label{ueq19}\n\\nabla_\\lm\\L(u, \\lm, \\rho)=\\rho^{-1}\\big(\\nabla\\menv\\big(\\Phi(u)+\\rho^{-1}{\\lm}\\big) - \\lm\\big)\n= \\Phi(u)-\\prox \\big(\\Phi(u)+ \\rho^{-1}\\lm\\big),\n\\end{equation} \n which in turn yields  the relationships\n\\begin{eqnarray*} \n\\L(u,\\lambda,\\rho)&\\ge&\\L(u,\\olm,\\rho)-\\langle\\nabla_\\lambda\\L(u,\\lambda,\\rho),\\olm-\\lambda\\rangle\\nonumber\\\\\n&=&\\L(u,\\olm,\\rho)-\\big\\langle \\Phi(u)-\\prox\\big(\\Phi(u)+ \\rho^{-1}{\\lm}\\big),\\olm-\\lambda\\big\\rangle\\nonumber\\\\\n&\\ge& \\ph(\\ox)+g(\\Phi(\\ox))+\\ell\\Vert u-\\ox\\Vert^2-\\big\\langle \\Phi(u)-\\prox\\big(\\Phi(u)+ \\rho^{-1}{\\lm} \\big),\\olm-\\lambda\\big\\rangle\n\\end{eqnarray*}\nwith the last inequality resulting from  \\eqref{eq16}. We conclude from  $u\\in S_\\rho(\\lambda)$ that \n\\begin{equation*}\n\\L(u,\\lambda,\\rho)\\le\\L(\\ox,\\lambda,\\rho)=\\ph(\\ox)+\\menv\\big(\\Phi(\\ox)+\\rho^{-1}{\\lm}\\big) - \\sm \\rho^{-1}{\\|\\lm\\|^2}\\leq \\ph(\\ox)+g \\big(\\Phi(\\ox) \\big).\n\\end{equation*}\nCombining these brings us to \n\\begin{equation}\\label{eq20}\n\\Vert u-\\ox\\Vert^2\\le\\frac{1}{\\ell}\\big\\langle \\Phi(u)-\\prox\\big(\\Phi(u)+\\rho^{-1}\\lm\\big),\\olm-\\lambda\\big\\rangle.\n\\end{equation}\nBy the first equality in \\eqref{prm}, the inclusion  $\\olm \\in \\partial g\\big(\\Phi(\\ox)\\big)$ amounts to $\\Phi(\\ox) = \\prox\\big(\\Phi(\\ox)+ \\rho^{-1}\\olm\\big)$. This leads us to \n\\begin{eqnarray}\\label{clq7}\n&&\\big\\Vert\\Phi(u)-\\prox\\big(\\Phi(u)+\\rho^{-1}\\lm\\big)\\big\\Vert \\nonumber\\\\\n&=&\\big\\Vert\\Phi(u)-\\Phi(\\ox)+\\prox\\big(\\Phi(\\ox)+\\rho^{-1}\\olm\\big)-\\prox\\big(\\Phi(u)+\\rho^{-1}\\lm\\big)\\big\\Vert\\nonumber\\\\\n&\\le&2\\Vert\\Phi(u)-\\Phi(\\ox)\\Vert+\\rho^{-1}{\\Vert\\olm-\\lambda\\Vert}\\nonumber\\\\\n&\\le&2\\kappa_\\Phi\\Vert u-\\ox\\Vert+\\rho^{-1}{\\Vert\\olm-\\lambda\\Vert},\n\\end{eqnarray}\nwhere the first inequality comes from  the fact that proximal mapping of a convex function is always nonexpansive (cf.  \\cite[Proposition~12.19]{rw}), \nand where the last inequality results from \\eqref{eq49}. This, combined with  \\eqref{eq20},  brings us   to \n\\begin{equation*}\n\\Vert u-\\ox\\Vert^2\\le\\frac{1}{\\ell}\\Big(2\\kappa_\\Phi\\Vert u-\\ox\\Vert+\\rho^{-1}{\\Vert\\lambda-\\olm\\Vert}\\Big)\\Vert\\lambda-\\olm\\Vert.\n\\end{equation*}\nSince the latter inequality  can be expressed  equivalently as \n\\begin{equation*}\n\\ell\\Vert u-\\ox\\Vert^2-2\\kappa_\\Phi\\Vert \\lambda-\\olm\\Vert\\cdot\\Vert u-\\ox\\Vert-\\rho^{-1}{\\Vert \\lambda-\\olm\\Vert^2 }\\le 0,\n\\end{equation*}\nwe arrive at  the estimate\n\\begin{equation*}\n\\Vert u-\\ox\\Vert\\le\\left(\\dfrac{\\kappa_\\Phi}{\\ell}+\\sqrt{\\frac{\\kappa_\\Phi^2}{\\ell^2}+\\frac{1}{\\ell\\rho}}\\right)\\Vert\\lambda-\\olm\\Vert\\le\\tau\\Vert\\lambda-\\olm\\Vert \\le \\tau \\hat\\gamma/2\\tau < \\hat\\gamma.\n\\end{equation*}\nThis clearly justifies   \\eqref{pt9}   and  the inclusion $S_\\rho(\\lambda)\\subset  \\inte  \\B_{\\hat\\gamma}(\\ox)$ for all  $\\lambda\\in \\B_{\\hat\\gamma/2\\tau} (\\olm)$ and $\\rho\\geq\\bar\\rho$,  and hence completes the proof.\n\\end{proof}","sha256_utf8":"fc629cefe138ce9ffcf929a13326e06425788aeeda21494f3f66bed123e7d757"},{"start":64609,"end":65460,"start_line":936,"end_line":948,"raw_tex":"\\begin{Remark}{\\rm Assume that  the SOSC \\eqref{sosc} holds at $(\\ox,\\olm)$. It is not hard to see that a similar argument as those for \\eqref{eq20}\nshows that for any $\\lm\\in \\R^m$, any $u\\in S_\\rho(\\lm)$,  any $\\mu \\in \\Lm(\\ox)\\cap \\B_\\ve(\\olm)$, and any $\\rho\\ge \\bar\\rho$ we have \n\\begin{equation*} \n\\Vert u-\\ox\\Vert^2\\le\\frac{1}{\\ell}\\big\\langle \\Phi(u)-\\prox\\big(\\Phi(u)+\\rho^{-1}\\lm\\big),\\mu-\\lambda\\big\\rangle,\n\\end{equation*}\nwhere constants $\\bar\\rho$, $\\ve$, and $\\ell$ come from Theorem~\\ref{ugrowth}.\nCombining this and the second equality in \\eqref{ueq19} brings us to \n\\begin{equation}\\label{ueq20}\n\\Vert u-\\ox\\Vert^2\\le\\frac{1}{\\ell\\rho}\\big\\langle \\nabla e_{1/\\rho}\\big(\\Phi(u)+\\rho^{-1}\\lm\\big)-\\lm,\\mu-\\lambda\\big\\rangle,\n\\end{equation}\nwhich will be utilized in the proof of Theorem~\\ref{est} later in this section.\n}\n\\end{Remark}\n","sha256_utf8":"59f40b05f3eae17aea1301094e9ce8e2f2fbeaf2b89a20548b3ebf24dcc65135"},{"start":66893,"end":67340,"start_line":967,"end_line":972,"raw_tex":"\\begin{Proposition}[estimate for residual functions] \\label{erct}  Assume that $(\\ox,\\olm)$ is a solution to the KKT system \\eqref{vs} with $\\Th$ taken from \\eqref{Th}. Then \nthere exist positive constants $\\gamma_2$ and $\\kappa_2$ such that\n\\begin{equation}\\label{eb1}\nR(x, \\lambda) \\leq \\kappa_2\\big(\\|x-\\ox\\|+\\dist(\\lambda, \\Lambda(\\ox))\\big)\\quad\\textrm{for all }\\;\\; (x, \\lambda)\\in \\B_{\\gamma_2}(\\ox, \\olm).\n\\end{equation} \n\\end{Proposition}","sha256_utf8":"dfcd47912be3b0503359236e28d38df76af54ed6b70b2d30da56906460755769"},{"start":67342,"end":68345,"start_line":974,"end_line":985,"raw_tex":"\\begin{proof} By \\eqref{Th},  we have $N_\\Th(x) = N_\\Th(\\ox)$ for all $x\\in \\Th$. Let $\\mu=P_{\\Lm(\\ox)}(\\lm)$\nand observe that  $(\\ox,\\mu)$ is a solution to the KKT system \\eqref{vs}. So, we know from \\eqref{prm} that    $\\mu \\in \\partial g \\big(\\Phi(\\ox)\\big)$ yields   $\\Phi(\\ox) = {\\rm prox}_g\\big(\\Phi(\\ox)+ \\mu\\big)$.\nUsing these tells us that \n\\begin{eqnarray*}\nR(x, \\lambda)&=& \\dist\\big(-\\nabla_x L(x, \\lm), N_\\Th(x)\\big)+\\big\\|\\Phi(x)-{\\rm{prox}}_g\\big(\\Phi(x)+\\lm\\big)\\big\\|\\\\\n&\\le & \\|\\nabla_x L(x, \\lm)-\\nabla_x L(\\ox, \\olm)  \\|+  \\dist\\big(-\\nabla_x L(\\ox, \\olm), N_\\Th(\\ox)\\big)\\\\\n&&+  \\|\\Phi(x)-\\Phi(\\ox)\\|+ \\big\\|{\\rm{prox}}_g\\big(\\Phi(x)+\\lm\\big)-{\\rm{prox}}_g\\big(\\Phi(\\ox)+\\mu\\big)\\big\\|\\\\\n&\\le &\\|\\nabla_x L(x, \\lm)-\\nabla_x L(\\ox, \\olm)  \\| +  2\\|\\Phi(x)-\\Phi(\\ox)\\|+ \\dist(\\lambda, \\Lambda(\\ox)).\n\\end{eqnarray*}\nThis, together with \\eqref{eq49},  ensures the existence of positive \nconstants  $\\gamma_2$ and $\\kappa_2$ for which the estimate \\eqref{eb1} is satisfied. \n\\end{proof}","sha256_utf8":"8b084850e6d3159bc030fbad7a9ad4a76167e88a0119ca6d358946efdd19cf0e"},{"start":68519,"end":69447,"start_line":990,"end_line":1001,"raw_tex":"\\begin{Theorem}[error bound for consecutive iterates of ALM]\\label{est} \nLet $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and  let the SOSC \\eqref{sosc} hold at $(\\ox,\\olm)$. Then there exist positive constants   $\\gamma_3$, and $\\kappa_3$ such that for any $\\rho\\geq \\bar\\rho$ with \n$\\bar\\rho$ taken from Theorem~{\\rm\\ref{ugrowth}}, any $(x,\\lambda)\\in\\B_{\\gamma_3}(\\ox,\\olm)$ with $x\\in \\Th$ and  $R(x,\\lambda)>0$, and any   optimal solution $u$ to the regularized problem \n\\begin{equation}\\label{regp}\n\\mini \\L(w, \\lm, \\rho)\\quad \\mbox{subject to}\\quad w\\in \\Th\\cap \\B_{\\hat\\gg}(\\ox)\n\\end{equation} \nwith $\\hat\\gg$ taken from Proposition~{\\rm\\ref{solv}},   the error bound estimate\n\\begin{equation}\\label{est0}\n\\Vert u-x\\Vert+\\big\\Vert\\nabla\\menv\\big(\\Phi(u)+ \\rho^{-1}{\\lm}\\big)-\\lm\\big\\Vert\\le\\kappa_3\\,R(x,\\lambda)\n\\end{equation}\nholds, where the residual function  $R$ is  taken from \\eqref{res}.\n\\end{Theorem}","sha256_utf8":"d080f766bd7bd341a940424a1fdc1303fdbe779a1716c60d6054ac7d83f9c3b9"},{"start":69450,"end":80474,"start_line":1004,"end_line":1147,"raw_tex":"\\begin{proof} It follows from the SOSC \\eqref{sosc} and    Theorem~\\ref{ugrowth} that there are positive  constants $\\bar\\rho$, $\\gamma$, $\\varepsilon$, and $\\ell$ for which \nthe uniform quadratic  growth condition \\eqref{eq14} holds  for all $\\rho\\ge\\bar\\rho$ and all $\\lambda\\in\\Lambda(\\ox)\\cap\\B_\\varepsilon(\\olm)$. \nAccording to Proposition~\\ref{solv}, the solution mapping to \\eqref{regp}, denoted by $S_\\rho(\\lm)$ therein, satisfies the uniform isolated calmness \\eqref{pt9} and the condition $S_\\rho(\\lm)\\subset  \\inte  \\B_{\\hat\\gamma}(\\ox)$ for all $\\lm\\in \\B_{\\hat\\gamma/2\\tau}(\\olm)$ and all $\\rho\\ge \\bar\\rho$, where both positive constants $\\tau$ and $\\hat\\gg$ are taken from this proposition. This, in particular, tells us that for every $\\lm\\in \\B_{\\hat\\gamma/2\\tau}(\\olm)$ and every $\\rho\\ge \\bar\\rho$\nany optimal solution $u$ to \\eqref{regp} satisfies the first-order optimality condition \n\\begin{equation}\\label{optc}\n0\\in \\nabla_x\\L(u, \\lm, \\rho)+ N_\\Th(u).\n\\end{equation}\nAfter this presentation, assume by contradiction that the error bound estimate \\eqref{est0} fails. The latter implies that  there exists a sequence $\\{(\\xk, \\lk, \\rok)\\b\\subset \\Th \\times \\R^m\\times [\\bar\\rho, \\infty)$ \nwith $\\xk\\to\\ox, \\lk\\to\\olm$ as $k\\to\\infty$ such that\n\\begin{equation}\\label{est1}\n\\Vert u^k-\\xk\\Vert+\\| d^k-\\lk\\|>kR_k\\quad \\mbox{with}\\;\\;d^k:=\\nabla\\menvk\\big(\\Phi(u^k)+\\rho_k^{-1}\\lk\\big),\n\\end{equation} \nwhere $u^k$ is an optimal solution to \\eqref{regp} for $(\\lambda, \\rho):=(\\lk, \\rok)$, and where $R_k:=R(\\xk, \\lk)$ for each $k \\in \\mathbb{N}$. Denote by $\\beta_k$ the left-hand side of \\eqref{est1}, and thus get $R_k=o(\\beta_k)$.\n By the definition of $R_k$, the latter yields\n\\begin{equation}\\label{est2}\n-\\nabla_xL(\\xk, \\lk)+o(\\xik)\\in N_\\Th(\\xk)\\quad\\textrm{and}\\quad \\Phi(\\xk)+o(\\xik)= {\\rm{prox}}_g\\big(\\Phi (\\xk)+\\lk\\big).\n\\end{equation}\nPassing to a subsequence if necessary, we can find $(\\xi,\\eta)\\in \\R^n\\times \\R^m$ such that \n\\begin{equation}\\label{est4}\n\\dfrac{u^k-\\xk}{\\xik}\\to \\xi\\quad \\mbox{and}\\quad  \\dfrac{ d^k-\\lk}{\\xik} \\to  \\eta \\quad \\mbox{with}\\;\\; (\\xi,\\eta)\\neq 0.\n\\end{equation}\nWe know from Theorem~\\ref{sooc} that the error bound \\eqref{subr3} holds for all $(x,\\lm)$ sufficiently close to  $(\\ox, \\olm)$. \nSet $\\mu^k:=P_{\\Lambda(\\ox)}(\\lk)$ and remember  that $(\\xk, \\lk)\\to(\\ox, \\olm)$.  By  \\eqref{subr3}, we can assume without loss of generality that $\\xk-\\ox=O(R_k)$ and $\\lk-\\mu^k=O(R_k)$ for all $k\\in \\N$, which in turn results in\n\\begin{equation}\\label{est3}\n\\xk-\\ox=o(\\xik)\\quad\\textrm{and}\\quad\\lk-\\mu^k=o(\\xik)\\quad\\textrm{as } k\\to\\infty.\n\\end{equation}\nThe latter along with $\\lk\\to \\olm$ tells us that $\\mu^k\\to \\olm$. So, we get   $\\mu^k\\in\\Lambda(\\ox)\\cap\\B_\\varepsilon(\\olm)$ for all $k $ sufficiently large. \nThis, combined with  $\\rho_k\\ge\\bar\\rho$ and \\eqref{ueq20}, implies that \n\\begin{eqnarray}\\label{est5}\n\\Vert u^k-\\ox\\Vert^2&\\le&\\dfrac{1}{\\rok\\ell} \\langle d^k-\\lk,\\mu^k-\\lk \\rangle\\le\\dfrac{1}{\\rok\\ell}\\| d^k-\\lk\\|\\cdot\\Vert\\mu^k-\\lk\\Vert,\n\\end{eqnarray}\nwhich together with \\eqref{est4} and  \\eqref{est3} results in \n\\begin{equation*}\n\\dfrac{\\Vert u^k-\\ox\\Vert^2}{\\xik^2}\\le\\dfrac{1}{\\rok\\ell}\\dfrac{\\| d^k-\\lk\\|}{\\xik}\\cdot\\dfrac{\\Vert\\mu^k-\\lk\\Vert}{\\xik}\\to 0\\quad\\textrm{ as }\\;k\\to\\infty.\n\\end{equation*}\nThus, we get $u^k-\\ox=o(\\beta_k)$. Combining this with \\eqref{est3} clearly shows that \n\\begin{equation*}\n\\xi=\\lim_{k\\to\\infty}\\dfrac{u^k-\\xk}{\\xik}=\\lim_{k\\to\\infty}\\dfrac{u^k-\\ox}{\\xik}-\\lim_{k\\to\\infty}\\dfrac{\\xk-\\ox}{\\xik}=0-0=0.\n\\end{equation*}\nIf either the sequence $\\{\\rok\\b$ or  $\\{{\\rok}/{\\xik}\\b$ is bounded, it follows from $u^k-\\ox=o(\\beta_k)$ that  $\\dfrac{\\rok}{\\xik}\\Vert u^k-\\ox\\Vert\\to0$ as $k\\to\\infty$. \nThis, \\eqref{ueq19}, and a similar argument as those for \\eqref{clq7} bring us to \n\\begin{eqnarray*}\n \\dfrac{ \\|d^k-\\lk\\|}{\\xik}\n&=&\\frac{\\rok}{\\xik} \\big\\Vert \\Phi(u^k)-\\proxk\\big(\\Phi(u^k)+\\rho_k^{-1}\\lk\\big)\\big\\Vert\\\\\n&\\leq& 2\\kappa_\\Phi\\frac{\\rok}{\\xik}\\|u^k-\\ox\\|+\\dfrac{\\Vert \\mu^k-\\lk\\Vert }{\\xik}\\to0\\quad\\textrm{as } k\\to\\infty,  \n\\end{eqnarray*} \nwhich by \\eqref{est4} yields $\\eta =0$. This is  a contradiction with \\eqref{est4} since we showed that $(\\xi,\\eta)=0$.\n\nAssume now that both sequences $\\{\\rok\\b$ and  $\\{{\\rok}/{\\xik}\\b$  are unbounded.  We can assume by passing  to a subsequence if necessary that \n\\begin{equation}\\label{unb}\n\\rok\\to \\infty\\quad \\mbox{and}\\quad \\frac{\\rok}{\\beta_k}\\to \\infty\\;\\;\\mbox{as}\\;\\; k\\to \\infty.\n\\end{equation}\nSince $u^k$ is an optimal solution to \\eqref{regp} associated with $(\\lk,\\rok)$, we deduce from \\eqref{optc} that \n\\begin{equation*}\n0\\in \\nabla_x\\L(u^k, \\lk, \\rok)+ N_\\Th(u^k).\n\\end{equation*} \nBy \\eqref{Th}, we get  $N_{\\Th}(u^k)=N_\\Th(\\xk) = \\rge B^*$. These along with  \\eqref{est2} tell us that \n\\begin{equation*}\n\\nabla_x\\L(u^k, \\lk, \\rok)-\\nabla_xL(\\xk, \\lk)+o(\\xik)\\in N_\\Th(\\xk)-N_\\Th(u^k)= \\rge B^*.\n\\end{equation*}\nUsing the definitions of the augmented Lagrangian $\\L$ and  $L$, we get \n\\begin{eqnarray*}\n\\rge B^*&\\ni& \\nabla_x\\L(u^k, \\lk, \\rok)-\\nabla_xL(\\xk, \\lk)+o(\\xik)\\\\\n&=&\\nabla \\varphi(u^k)-\\nabla \\varphi(\\xk)+\\nabla\\Phi(u^k)^* \\nabla\\menvk\\big(\\Phi(u^k)+\\rho_k^{-1}\\lk\\big)-\\nabla\\Phi(\\xk)^*\\lk+o(\\xik)\\\\\n&=&\\big(\\nabla\\Phi(u^k)-\\nabla\\Phi(\\xk)\\big) ^*\\lk+\\nabla\\Phi(u^k)^*\\big (d^k-\\lk\\big )+o(\\xik)\\\\\n&=&\\nabla\\Phi(u^k)^* \\big(d^k-\\lk\\big)+o(\\xik),\n\\end{eqnarray*}\nwhere the second and last equalities result from  the Lipschitz continuity of  $\\nabla \\varphi$ and $\\nabla \\Phi$ around $\\ox$ and from  the fact that $u^k-\\xk=o(\\xik)$. \nDividing both sides  by $\\xik$ and then letting  $k\\to\\infty$ confirm via \\eqref{est4} that \n\\begin{equation*}\n\\nabla\\Phi(\\ox)^*\\eta \\in\\rge B^*.\n\\end{equation*} \nWe are going to show that $\\eta=0$, which contradicts \\eqref{est4} and thus completes the proof. To furnish this, set $E:=\\big\\{z \\in \\R^n\\, |\\, \\nabla \\Phi(\\ox)^*z\\in \\rge B^*\\big\\}$\nand observe from \\cite[Corollary~11.25(d)]{rw} that $E^\\perp=\\big\\{\\nabla\\Phi(\\ox)y|\\;y\\in \\ker B\\big\\}$. So, we conclude  from  $\\ox,u^k\\in \\Th$ and  \\eqref{Th} that $\\nabla\\Phi(\\ox)(u^k-\\ox)\\in E^\\perp$.\nMoreover, we obtain  from \\eqref{est4}-\\eqref{est5} that \n\\begin{equation}\\label{est6}\n\\dfrac{\\rok}{\\xik}\\Vert u^k-\\ox\\Vert^2\\leq\\dfrac{1}{\\ell} \\dfrac{\\| d^k-\\lk\\|}{\\xik} \\cdot\\Vert \\mu^k-\\lk\\Vert  \\to 0 \\quad\\textrm{ as }k\\to\\infty.\n\\end{equation} \nSince $P_E$ is a linear mapping due to $E$ being a subspace, we arrive at \n\\begin{eqnarray}\\label{est7}\nP_E\\Big(\\dfrac{\\rok}{\\xik}\\big (\\Phi(u^k)-\\Phi(\\ox)\\big )\\Big)&=&P_E\\Big (\\dfrac{\\rok}{\\xik}\\nabla\\Phi(\\ox) (u^k-\\ox )+O\\big (\\dfrac{\\rok}{\\xik}\\Vert u^k-\\ox\\Vert^2 \\big)\\Big  )\\nonumber\\\\\n&=&P_E \\Big(O\\big(\\dfrac{\\rok}{\\xik}\\Vert u^k-\\ox\\Vert^2 \\big )\\Big)\\to 0 \\quad \\textrm{as } k\\to\\infty,\n\\end{eqnarray} \nwhere the last equality results from $\\nabla\\Phi(\\ox)(u^k-\\ox)\\in E^\\perp$.\nFix $k\\in \\N$ and set  \n$\nz^k:= \\Phi(u^k)+ \\rho_k^{-1}\\big(\\lk -d^k\\big).\n$\nIt follows from $d^k=\\nabla\\menvk\\big(\\Phi(u^k)+ \\rho_k^{-1}{\\lk}\\big)$ and the second equation in \\eqref{prm} that \n\\begin{equation}\\label{clq8}\nd^k \\in \\partial g(z^k).\n\\end{equation} \nUsing  \\eqref{est4} and \\eqref{unb} allows us to arrive at \n\\begin{equation*}\nz^k =\\Phi(u^k) - \\frac{d^k-\\lk}{\\xik}\\cdot\\frac{\\xik}{\\rok} \\to \\Phi(\\ox)=:\\bar z\\quad\\textrm{as }\\; k\\to\\infty.\n\\end{equation*} \nBy \\eqref{clq8}, we get $z^k\\in \\dom g=\\cup_{i=1}^s C_i$ for all $k$, where the polyhedral convex sets $C_i$ are taken from \\eqref{PWLQ}. \nBy passing to a subsequence if necessary, we can assume without loss of generality that  for some $\\oi\\in I(\\oz)$, we have $z^k \\in C_\\oi$ for all $k$, where the index set  $  I(\\bar z)$  is defined by \\eqref{act}.\nSince $C_\\oi$ is  polyhedral, we find   $l_\\oi\\in \\N$, $c_{\\oi, j}\\in \\R^m$, and $\\al_{\\oi, j}\\in \\R$,  $j=1,\\ldots,l_\\oi$,  such that \n\\begin{equation*}\nC_\\oi= \\big\\{z\\in \\R^m\\, |\\, \\langle c_{\\oi, j}, z\\rangle \\leq \\al_{\\oi, j}, \\quad j= 1, \\ldots, l_\\oi\\big\\}.\n\\end{equation*} \nDenote by $J(\\bar z)$ and $J(z^k)$ the sets of the indexes of active inequalities in $C_\\oi$ at $\\bar z$ and $z^k$, respectively. Since $z^k\\to \\oz$, we obtain  $J(z^k)\\subset J(\\bar z)$ for all $k$ sufficiently large. \nPassing to a subsequence, if necessary, we can assume that there exists a subset $\\bar J\\subset J(\\bar z)$ such that $J(z^k)=\\bar J$ for all $k$. Thus, we have \n\\begin{equation*}\nz^k -\\bar z \\in \\big\\{v\\in \\R^m\\, \\big|\\, \\langle c_{\\oi, j}, v\\rangle= 0\\;\\textrm{ for } \\;j \\in \\bar J, \\; \\langle c_{\\oi, j}, v\\rangle\\leq 0\\; \\textrm{ for }\\; j\\in J(\\bar z)\\setminus \\bar J \\big\\}=: \\Omega.\n\\end{equation*} \nIt follows from  the definitions of $z^k$ and $\\bar z$ that \n\\begin{equation*}\n\\Phi(u^k)-\\Phi(\\ox)-\\frac{ d^k-\\lk}{\\rok}=z^k-\\oz \\in \\Omega. \n\\end{equation*}\nMultiplying by $ {\\rok}/{\\xik}$ and using the linearity of  $P_E$  allow us to get \n\\begin{equation*}\nP_E\\Big(\\dfrac{\\rok}{\\xik}\\big (\\Phi(u^k)-\\Phi(\\ox)\\big )\\Big)- P_E \\big(\\frac{ d^k-\\lk}{\\xik}\\big) \\in P_E(\\Omega),\n\\end{equation*} \nwhich, together with \\eqref{est4}, \\eqref{est7}, and the closedness of $P_E(\\Omega)$, yields $-P_E(\\eta) \\in P_E(\\Omega)$. Since $\\eta\\in E$, the latter confirms that \n\\begin{equation}\\label{est8}\n-\\eta=-P_E(\\eta) = P_E(\\zeta)\\quad\\textrm{for some }\\;\\; \\zeta\\in \\Omega.\n\\end{equation}\nRemembering $\\mu^k\\in \\Lm(\\ox)$, we obtain $\\mu^k \\in \\partial g(\\bar z)$. By the representation of $\\partial g(\\bar z)$ from \\eqref{sub} and the fact that  $\\oi\\in I(\\bar z)$, we conclude that \n\\begin{equation*}\\label{clq9}\n\\mu^k - A_\\oi\\bar z -a_\\oi \\in N_{C_\\oi}(\\bar z) = \\cone\\big\\{c_{\\oi, j}\\, | \\;j\\in J(\\bar z)\\big\\}.\n\\end{equation*} \nSince we also have $\\oi\\in I(z^k)$, it follows again from \\eqref{clq8} and  \\eqref{sub} that \n\\begin{equation*}\\label{clq10}\nd^k -A_\\oi z^k-a_\\oi \\in N_{C_\\oi}(z^k) = \\cone\\big\\{c_{\\oi, j}\\,|\\; j\\in \\bar J\\big\\}.\n\\end{equation*} \nCombining these results in \n\\begin{eqnarray*}\n \\mu^k-d^k + A_\\oi(z^k-\\bar z)&\\in&   \\cone\\big\\{c_{\\oi, j}\\, | \\;j\\in J(\\bar z)\\big\\}-\\cone\\big\\{c_{\\oi, j}\\,|\\; j\\in \\bar J\\big\\}\\\\\n&=&\\span\\big\\{c_{\\oi, j}\\,|\\;  j\\in \\bar J\\big\\}+\\cone\\big\\{c_{\\oi, j}\\,|\\; j\\in J(\\bar z)\\setminus\\bar J\\big\\} =  \\Omega^*.\n\\end{eqnarray*} \nSince $\\lk -\\mu^k=o(\\xik)$ and $u^k-\\ox=o(\\beta_k)$, we deduce from \\eqref{est4} and \\eqref{unb} that \n\\begin{eqnarray*}\n\\frac{ \\mu^k-d^k}{\\beta_k}+A_\\oi\\big(\\frac{z^k-\\oz}{\\beta_k}\\big)&=& \\frac{ \\mu^k-\\lk}{\\beta_k}+\\frac{ \\lk-d^k}{\\beta_k} +A_\\oi\\big(\\frac{\\Phi(u^k)-\\Phi(\\ox)}{\\beta_k}\\big)\\\\\n&&-\\frac{1}{\\rok}A_\\oi\\big(\\frac{d^k-\\lk}{\\beta_k}\\big)\\to -\\eta \\;\\;\\;\\;\\mbox{as}\\;\\;k\\to \\infty,\n\\end{eqnarray*}\nwhich in turn demonstrates that \n$-\\eta \\in \\Omega^*$. This, combined with \\eqref{est8} and $\\eta\\in E$, shows that \n\\begin{equation*}\n\\|\\eta\\|^2= \\langle \\eta, \\eta \\rangle = \\langle \\eta, P_E(\\eta) \\rangle= -\\langle \\eta, P_E(\\zeta) \\rangle= -\\big\\langle \\eta, P_E(\\zeta)+P_{E^{\\perp}}(\\zeta) \\big\\rangle = -    \\langle \\eta, \\zeta\\rangle \\le 0,\n\\end{equation*} \nimplying that  $\\eta =0$. This clearly contradicts \\eqref{est4} and thus completes the proof.\n\\end{proof}","sha256_utf8":"7604770e3663f1dd6f1c001fdb63748bda5cda1fbeabd2f6be2d534f1458b5ff"},{"start":86571,"end":97816,"start_line":1209,"end_line":1351,"raw_tex":"\\begin{proof}\n Define $R_k:=R(\\xk,\\lk)$   and observe that if $R_k=0$ for some $k$, then the pair $(\\xk,\\lk)$ satisfies the KKT system \\eqref{vs}, and thus Algorithm \\ref{Alg1} should stop. From now on assume that $R_k>0$ for all $k\\in\\N$. \n  Pick the positive constants $\\kappa_1$ and $\\gg_1$ from Theorem~\\ref{sooc}, the positive constants $\\gamma_2$ and $\\kappa_2$ from Proposition~\\ref{erct}, the positive constants \n  $\\gamma_3$ and $\\kappa_3$ from Theorem~\\ref{est}, and the positive constants $\\hat\\gg$ and $ \\tau$ from Proposition~\\ref{solv}. By \\eqref{error}, we can find $\\gamma_4>0$ such that\n\\begin{equation}\\label{ek}\n\\sigma(x,\\lm)\\le\\dfrac{1}{4\\kappa_1\\kappa_2}R(x,\\lm)\\quad\\textrm{ whenever }\\;(x,\\lm)\\in\\B_{\\gg_4}(\\ox,\\olm).\n\\end{equation}\nDefine further the  positive  constants  $\\bar\\tau:=\\kappa_3$ and \n\\begin{equation}\\label{const}\n\\bar\\gamma:=\\dfrac{\\gamma'}{1+2\\sqrt{2}\\bar\\tau\\kappa_2} \\quad\\mbox{with}\\quad\n\\gamma':=\\min\\Big\\{\\frac{\\hat\\gg}{2\\tau},\\gamma_3,\\gamma_4,\\dfrac{\\gamma_1}{1+\\sqrt{2}\\bar\\tau\\kappa_2},\\dfrac{\\gamma_2}{1+\\sqrt{2}\n\\bar\\tau\\kappa_2}\\Big\\}.\n\\end{equation}\nMoreover, pick $q\\in (0,1) $ and set \n\\begin{equation}\\label{rhof}\n\\bar\\varrho:=\\max\\big\\{\\bar\\rho, 4\\kappa_1\\kappa_2\\bar\\tau, \\frac{2\\sqrt{2} \\kappa_1\\kappa_2^2\\bar\\tau^2}{q}\\big\\},\n\\end{equation}\nwhere  $\\bar\\rho$ is taken from Theorem~\\ref{ugrowth}. \nArguing by induction, we aim at showing that for any starting point $(x^0,\\lambda^0)\\in\\big(\\Th\\times\\R^m\\big)\\cap\\B_{\\bar\\gamma}(\\ox,\\olm)$ there exists a sequence $\\{(\\xk,\\lk)\\c\\subset \\Th \\times\\R^m$, generated by Algorithm~\\ref{Alg1}\nwith $\\rok \\ge \\bar\\varrho$ and   $\\ek=\\sigma(\\xk,\\lk)$ for all $k$,\n such that for all $k=0,1,\\ldots$ the relationships\n\\begin{equation}\\label{c1}\n(\\xk,\\lk)\\in\\B_{\\gamma'}(\\ox,\\olm),\n\\end{equation}\n\\begin{equation}\\label{c2}\n\\dist\\big(-\\nabla_x L(\\xkk,\\lkk), N_\\Th(\\xkk)\\big)\\le\\ek\\quad\\textrm{ and }\\quad\\Vert\\xkk-\\xk\\Vert+\\Vert\\lkk-\\lk\\Vert\\le\\bar\\tau R_k\n\\end{equation}\nhold. To achieve this goal, set $k=0$, pick $(x^0,\\lambda^0)\\in\\big(\\Th\\times\\R^m\\big)\\cap\\B_{\\bar\\gamma}(\\ox,\\olm)$, and observe from  $\\bar\\gamma\\le\\gamma'$ that \\eqref{c1} holds in this case. We also have $\\lambda^0\\in\\B_{\\hat\\gg/2\\tau}(\\olm)$ and $\\rho_0\\ge \\bar\\varrho$ and then find by Proposition~\\ref{solv} a primal iterate $x^1\\in \\inte \\B_{\\hat\\gamma}(\\ox)$ satisfying $-\\nabla_x\\L(x^1,\\lambda^0,\\rho_0)\\in N_\\Th(x^1)$. Define further \nthe multiplier $\\lm^1:=\\nabla e_{1^{}/\\rho_0}g\\big(\\Phi(x^1)+ \\rho_0^{-1}\\lambda^0\\big)$ and observe that\n\\begin{equation*}\n-\\nabla_x L(x^1,\\lm^1)=-\\nabla_x\\L(x^1,\\lambda^0,\\rho_0)\\in N_\\Th(x^1).\n\\end{equation*}\nMoreover, employing  Theorem~\\ref{est} confirms that \n\\begin{equation*}\n\\Vert x^1-x^0\\Vert+\\Vert\\lm^1-\\lambda^0\\Vert\\le\\bar\\tau R_0.\n\\end{equation*}\nThus $(x^1,\\lambda^1)$ is well defined and satisfies the claimed condition \\eqref{c2} for $k=0$. Assume by induction that the iterates $(\\xk,\\lk)$, $k=0,1,\\ldots,s+1$, are well defined  and that   both conditions \\eqref{c1} and \\eqref{c2} hold for $k=0,1,\\ldots,s$. We verify now the existence of $(x^{s+2},\\lambda^{s+2})$ satisfying \\eqref{c1} and \\eqref{c2} for $k=s+1$. To this end, we first show that $(x^{s+1},\\lambda^{s+1})\\in\\B_{\\gamma'}(\\ox,\\olm)$. \nTo proceed, fix an integer   $k$ with $0\\le k\\le s$.\nSince $(\\xk,\\lk)\\in\\B_{\\gamma'}(\\ox,\\olm)$, it follows from \\eqref{eb1} and \\eqref{c1} that\n\\begin{eqnarray}\\label{c3}\nR_k&\\le&\\kappa_2\\big(\\Vert\\xk-\\ox\\Vert+\\dist(\\lk,\\Lambda(\\ox))\\big)\\\\\n&\\le&\\kappa_2\\big(\\Vert\\xk-\\ox\\Vert+\\Vert\\lk-\\olm\\Vert\\big)\\le\\sqrt{2}\\kappa_2\\Vert(\\xk,\\lk)-(\\ox,\\olm)\\Vert\\le\\sqrt{2}\\kappa_2\\gamma'.\\nonumber\n\\end{eqnarray}\nCombining this with \\eqref{c1} and the second relation in \\eqref{c2}, we conclude that\n\\begin{eqnarray}\\label{c4}\n\\Vert(\\xkk,\\lkk)-(\\ox,\\olm)\\Vert&\\le&\\Vert\\xkk-\\xk\\Vert+\\Vert\\lkk-\\lk\\Vert+\\Vert(\\xk,\\lk)-(\\ox,\\olm)\\Vert\\nonumber\\\\\n&\\le&\\bar\\tau R_k+\\gamma'\\le(\\sqrt{2}\\bar\\tau\\kappa_2+1)\\gamma'\\le\\gamma_1,\n\\end{eqnarray}\nwhich implies that $(\\xkk,\\lkk)\\in\\B_{\\gamma_1}(\\ox,\\olm)$. Using \\eqref{subr3}, we obtain\n\\begin{eqnarray}\\label{c5}\n&&\\Vert\\xkk-\\ox\\Vert+\\dist\\big(\\lkk,\\Lambda(\\ox)\\big)\\nonumber\\\\\n&\\le&\\kappa_1 R_{k+1} =\\kappa_1\\Big(\\dist\\big(-\\nabla_x L(\\xkk,\\lkk), N_\\Th(\\xkk)\\big) +\\big\\Vert\\Phi(\\xkk)-{\\rm{prox}}_g\\big(\\Phi(\\xkk)+\\lkk\\big)\\big\\Vert\\Big)\\nonumber\\\\\n&\\le&\\kappa_1\\ek+\\kappa_1\\big\\Vert\\Phi(\\xkk)-{\\rm{prox}}_g\\big(\\Phi(\\xkk)+\\lkk\\big)\\big\\Vert.\n\\end{eqnarray}\nTo proceed further,  define\n\\begin{equation}\\label{skk}\n\\skk:=\\proxk\\big(\\Phi(\\xkk)+ \\rho_k^{-1}{\\lk}\\big ).\n\\end{equation}\nIt follows from  \\eqref{lkk}  and \\eqref{ueq19} that\n\\begin{equation}\\label{eq3}\n\\Phi(\\xkk)-\\skk= \\rho_k^{-1}(\\lkk-\\lk).\n\\end{equation}\nSince $\\lkk=\\nabla\\menvk\\big(\\Phi(\\xkk)+ \\rho_k^{-1}{\\lk}\\big)$, we deduce from the second equality in \\eqref{prm} that \n\\begin{equation*}\\label{clq17}\n\\lkk   \\in \\sub g\\big(\\Phi(\\xkk)+ \\rho_k^{-1}(\\lk-\\lkk)\\big)=\\partial g(\\skk).\n\\end{equation*} \nAppealing now to the first equality in \\eqref{prm} yields $\\skk={\\rm{prox}}_g(\\skk+\\lkk)$. Since the mapping $y\\mapsto y-{\\rm{prox}}_g(y+\\lkk)$ is nonexpansive (cf. \\cite[Proposition~12.27]{bc}), we arrive at the relationships\n\\begin{eqnarray}\\label{eq2}\n&&\\big\\Vert\\Phi(\\xkk)-{\\rm{prox}}_g\\big(\\Phi(\\xkk)+\\lkk\\big)\\big\\Vert\\nonumber\\\\\n&=&\\big\\Vert\\Phi(\\xkk)-{\\rm{prox}}_g\\big(\\Phi(\\xkk)+\\lkk\\big)\\big\\Vert-\\big\\Vert\\skk-{\\rm{prox}}_g(\\skk+\\lkk)\\big\\Vert\\nonumber\\\\\n&\\le&\\big\\Vert\\Phi(\\xkk)-{\\rm{prox}}_g\\big(\\Phi(\\xkk)+\\lkk\\big)-\\big(\\skk-{\\rm{prox}}_g(\\skk+\\lkk)\\big)\\big\\Vert\\nonumber\\\\\n&\\le&\\Vert\\Phi(\\xkk)-\\skk\\Vert.\n\\end{eqnarray}\nCombining \\eqref{c5}, \\eqref{eq3}, and \\eqref{eq2} brings us to\n\\begin{equation*}\n\\Vert\\xkk-\\ox\\Vert+\\dist\\big(\\lkk,\\Lambda(\\ox)\\big)\\le\\kappa_1\\ek+\\rho_k^{-1}{\\kappa_1}\\Vert\\lkk-\\lk\\Vert.\n\\end{equation*}\nThis,  together with \\eqref{ek}, \\eqref{rhof}, the second inequality in \\eqref{c2}, and \\eqref{c3} results in \n\\begin{eqnarray}\\label{c10}\n\\Vert \\xkk-\\ox\\Vert+\\dist\\big(\\lkk,\\Lambda(\\ox)\\big)&\\le&\\kappa_1\\ek+\\dfrac{\\bar\\tau\\kappa_1}{\\rok}R_k\\le\\dfrac{1}{4\\kappa_2}R_k +\\dfrac{1}{4\\kappa_2}R_k\\\\\n&\\le&\\dfrac{1}{2}\\Big(\\Vert\\xk-\\ox\\Vert+\\dist\\big(\\lk,\\Lambda(\\ox)\\big)\\Big)\\nonumber,\n\\end{eqnarray}\nwhich in turn implies the primal-dual estimates\n\\begin{equation}\\label{c6}\n\\Vert\\xkk-\\ox\\Vert+\\dist\\big(\\lkk,\\Lambda(\\ox)\\big)\\le\\dfrac{1}{2^{k+1}}\\Big(\\Vert x^0-\\ox\\Vert+\\dist\\big(\\lambda^0,\\Lambda(\\ox)\\big)\\Big).\n\\end{equation}\nThis estimate, \\eqref{c3},  and the second inequality in \\eqref{c2} tell us that \n\\begin{eqnarray}\\label{c7}\n\\Vert(x^{s+1},\\lambda^{s+1})-(x^0,\\lambda^0)\\Vert&\\le&\\sum\\limits_{k=0}^s\\Vert(\\xkk,\\lkk)-(\\xk,\\lk)\\Vert \\le\\bar\\tau\\sum\\limits_{k=0}^s R_k\\nonumber\\\\\n&\\le&\\bar\\tau\\kappa_2\\sum\\limits_{k=0}^s\\Vert\\xk-\\ox\\Vert+\\dist\\big(\\lk,\\Lambda(\\ox)\\big)\\nonumber\\\\\n&\\le&\\bar\\tau\\kappa_2\\sum\\limits_{k=0}^s\\dfrac{1}{2^k}\\Big(\\Vert x^0-\\ox\\Vert+\\dist\\big(\\lambda^0,\\Lambda(\\ox)\\big)\\Big)\\\\\n&\\le&2\\bar\\tau\\kappa_2\\big(\\Vert x^0-\\ox\\Vert+\\dist(\\lambda^0,\\Lambda(\\ox))\\big)\\nonumber\\\\\n&\\le&2\\bar\\tau\\kappa_2\\big(\\Vert x^0-\\ox\\Vert+\\Vert\\lambda^0-\\olm\\Vert\\big).\\nonumber\n\\end{eqnarray}\nThus, we arrive at the estimates\n\\begin{eqnarray*}\n\\Vert(x^{s+1},\\lambda^{s+1})-(\\ox,\\olm)\\Vert&\\le&\\Vert(x^{s+1},\\lambda^{s+1})-(x^0,\\lambda^0)\\Vert+\\Vert(x^0,\\lambda^0)-(\\ox,\\olm)\\Vert\\\\\n&\\le&2\\bar\\tau\\kappa_2\\big(\\Vert x^0-\\ox\\Vert+\\Vert\\lambda^0-\\olm\\Vert\\big)+\\Vert(x^0,\\lambda^0)-(\\ox,\\olm)\\Vert\\\\\n&\\le&(2\\sqrt{2}\\bar\\tau\\kappa_2+1)\\Vert(x^0,\\lambda^0)-(\\ox,\\olm)\\Vert\\le(2\\sqrt{2}\\bar\\tau\\kappa_2+1)\\bar\\gamma=\\gamma',\n\\end{eqnarray*}\nwhere the last inequality follows from $(x^0,\\lambda^0)\\in\\B_{\\bar\\gamma}(\\ox,\\olm)$, while the last equality comes from the definition of $\\bar\\gamma$ in \\eqref{const}. \nThis verifies that $(x^{s+1},\\lambda^{s+1})\\in\\B_{\\gamma'}(\\ox,\\olm)$. By  \\eqref{const}, we get  $\\lambda^{s+1}\\in\\B_{\\hat\\gg/2\\tau}(\\olm)$, and hence Proposition~\\ref{solv} ensures the existence of of an optimal solution $x^{s+2}$ \nto \\eqref{regp} associated with $(\\lm^{s+1},\\rho_{s+1})$ such that $x^{s+2}\\in \\inte\\B_{\\hat\\gamma}(\\ox)$. The latter yields \n\\begin{equation*}\n-\\nabla_x\\L(x^{s+2},\\lambda^{s+1},\\rho_{s+1})\\in N_\\Th(x^{s+2}).\n\\end{equation*} \nSet now $\\lm^{s+2}:=\\nabla e_{1^{}/\\rho_{s+1}}\\big(\\Phi(x^{s+2})+\\rho_{s+1}^{-1}\\lambda^{s+1}\\big)$ and observe that\n\\begin{equation*}\n -\\nabla_x L(x^{s+2},\\lm^{s+2}) =  -\\nabla_x\\L(x^{s+2},\\lambda^{s+1},\\rho_{s+1}) \\in N_\\Th(x^{s+2}),\n\\end{equation*}\nwhich by Proposition~\\ref{est} brings us to the estimate\n\\begin{equation*}\n\\Vert x^{s+2}-x^{s+1}\\Vert+\\Vert\\lm^{s+2}-\\lambda^{s+1}\\Vert\\le\\bar\\tau R_{s+1}.\n\\end{equation*}\nThis tells us that the iterate $(x^{s+2},\\lambda^{s+2})$ satisfies   \\eqref{c2} for $k=s+1$, which completes  the proof of  induction. Thus there exists a sequence  $\\{(\\xk,\\lk)\\c\\subset\\Th\\times\\R^m$, generated by Algorithm~\\ref{Alg1} \nwith $\\rok \\ge \\bar\\varrho$ and   $\\ek=o\\big(R(\\xk,\\lk)\\big)$ for all $k$, for which  \\eqref{c1} and \\eqref{c2} fulfill.\n\n \nNext, we are going to justify  the convergence of this primal-dual   sequence. Using the same arguments as in the proofs of \\eqref{c6} and \\eqref{c7} leads us to\n\\begin{equation}\\label{c8}\n\\Vert(x^{k+l},\\lambda^{k+l})-(\\xk,\\lk)\\Vert\\le 2\\bar\\tau\\kappa_2\\big(\\Vert\\xk-\\ox\\Vert+\\dist(\\lk,\\Lambda(\\ox))\\big)\\quad\\mbox{for all}\\;\\;k,l\\in\\N.\n\\end{equation}\nIt follows from \\eqref{c6} that the right-hand side of \\eqref{c8} goes to zero as $k\\to\\infty$, which tells us that the sequence $\\{(\\xk,\\lk)\\c$ is Cauchy. By \\eqref{c6},   this sequence converges  to $(\\ox,\\hat\\lambda)$ for some $\\hat\\lambda\\in\\Lambda(\\ox)$.\nNow, we show  that  the rate of this convergence is linear. Indeed, letting $l\\to\\infty$ in \\eqref{c8} tells us that for all $k\\in\\N$ we have\n\\begin{equation*}\\label{c9}\n\\Vert(\\xk,\\lk)-(\\ox,\\hat\\lambda)\\Vert\\le 2\\bar\\tau\\kappa_2\\big(\\Vert\\xk-\\ox\\Vert+\\dist(\\lk,\\Lambda(\\ox))\\big),\n\\end{equation*}\nwhich together with the first inequality in \\eqref{c10} verifies that\n\\begin{eqnarray*}\n\\Vert(\\xkk,\\lkk)-(\\ox,\\hat\\lambda)\\Vert&\\le&2\\bar\\tau\\kappa_2\\big(\\Vert\\xkk-\\ox\\Vert+\\dist(\\lkk,\\Lambda(\\ox))\\big)\\nonumber\\\\\n&\\le&2\\bar\\tau\\kappa_1\\kappa_2\\big(R_k^{-1}{\\ek}+\\rho_k^{-1}\\bar\\tau\\big)R_k\\nonumber\\\\\n&\\le&2\\bar\\tau\\kappa_1\\kappa_2^2\\big(R_k^{-1}{\\ek}+\\rho_k^{-1}\\bar\\tau\\big)\\big(\\Vert\\xk-\\ox\\Vert+\\dist(\\lk,\\Lambda(\\ox))\\big)\\nonumber\\\\\n&\\le&2\\bar\\tau\\kappa_1\\kappa_2^2\\big(R_k^{-1}{\\ek}+\\rho_k^{-1}\\bar\\tau\\big)\\big(\\Vert\\xk-\\ox\\Vert+\\Vert\\lk-\\hat\\lambda\\Vert\\big)\\nonumber\\\\\n&\\le& 2\\sqrt{2}\\bar\\tau\\kappa_1\\kappa_2^2\\big(R_k^{-1}{\\ek}+\\rho_k^{-1}\\bar\\tau\\big)\\Vert(\\xk,\\lk)-(\\ox,\\hat\\lambda)\\Vert.\n\\end{eqnarray*}\nCombining this with $\\rok\\ge \\bar\\varrho$  with $ \\bar\\varrho$ taken from \\eqref{rhof} and using the choice of the tolerance $\\ek=o(R_k)$ result in \n\\begin{equation*}\n\\limsup_{k\\to\\infty}\\dfrac{\\Vert(\\xkk,\\lkk)-(\\ox,\\hat\\lambda)\\Vert}{\\Vert(\\xk,\\lk)-(\\ox,\\hat\\lambda)\\Vert}\\le\\limsup_{k\\to\\infty}2\\sqrt{2}\\bar\\tau\n\\kappa_1\\kappa_2^2\\big(R_k^{-1}{\\ek}+\\rho_k^{-1}\\bar\\tau\\big)\\le q.\n\\end{equation*}\nIt follows from  $q\\in (0,1)$ that the rate of   convergence of $\\{(\\xk,\\lk)\\c$ is linear. Furthermore, it is clear from the above inequality that if $\\rok\\to \\infty$, we \narrive at the superlinear convergence of the latter primal-dual sequence, which completes the proof.\n\\end{proof}","sha256_utf8":"f9acbf089f1af9f52c52af8d8ab8a5fbdabd05f0adf4cb349a0c822f8f35dee8"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"增广拉格朗日法原始对偶收敛","source_units":[{"start":85556,"end":86570,"start_line":1198,"end_line":1208,"raw_tex":"\\begin{Theorem}[primal-dual linear convergence of ALM]\\label{locpd} \n Let $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and  let the SOSC \\eqref{sosc} hold at $(\\ox,\\olm)$.\n Then there exist positive constants $\\bar\\gg$, $\\bar\\tau$, and $\\bar\\varrho$ such that for any starting point $(x^0,\\lm^0)\\in \\B_{\\bar\\gg}(\\ox,\\olm)\\cap\\big(\\Th\\times \\R^m\\big)$\n there is a primal-dual sequence   $\\{(\\xk,\\lk)\\c$,  generated by Algorithm~{\\rm\\ref{Alg1}} with $\\rok \\ge \\bar\\varrho$ and   $\\ek=\\sigma(\\xk,\\lk)$ for all $k$, where $\\sigma$ is defined by \\eqref{error}, satisfying the  estimate\n \\begin{equation}\\label{cone}\n \\|x^{k+1}-x^k\\|+\\|\\lm^{k+1}-\\lk\\|\\le \\bar\\tau R(x^k,\\lk),\n \\end{equation}\n where $R$ is the residual function defined by  \\eqref{res}.\n  Moreover, every such a sequence is convergent to $(\\ox,\\hat\\lambda)$ for some $\\hat\\lambda\\in\\Lm(\\ox)$, and its rate of convergence is linear.\n  Furthermore, if $\\rok\\to \\infty$,  the rate of convergence  of $\\{(\\xk,\\lk)\\c$ is superlinear. \n\\end{Theorem}","sha256_utf8":"f00825ffe5fb50a1d09e85d1ca01cf7c8fe27f7e4f56fd499759d22d4d352f92"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"主收敛定理完整证明","source_units":[{"start":86571,"end":97816,"start_line":1209,"end_line":1351,"raw_tex":"\\begin{proof}\n Define $R_k:=R(\\xk,\\lk)$   and observe that if $R_k=0$ for some $k$, then the pair $(\\xk,\\lk)$ satisfies the KKT system \\eqref{vs}, and thus Algorithm \\ref{Alg1} should stop. From now on assume that $R_k>0$ for all $k\\in\\N$. \n  Pick the positive constants $\\kappa_1$ and $\\gg_1$ from Theorem~\\ref{sooc}, the positive constants $\\gamma_2$ and $\\kappa_2$ from Proposition~\\ref{erct}, the positive constants \n  $\\gamma_3$ and $\\kappa_3$ from Theorem~\\ref{est}, and the positive constants $\\hat\\gg$ and $ \\tau$ from Proposition~\\ref{solv}. By \\eqref{error}, we can find $\\gamma_4>0$ such that\n\\begin{equation}\\label{ek}\n\\sigma(x,\\lm)\\le\\dfrac{1}{4\\kappa_1\\kappa_2}R(x,\\lm)\\quad\\textrm{ whenever }\\;(x,\\lm)\\in\\B_{\\gg_4}(\\ox,\\olm).\n\\end{equation}\nDefine further the  positive  constants  $\\bar\\tau:=\\kappa_3$ and \n\\begin{equation}\\label{const}\n\\bar\\gamma:=\\dfrac{\\gamma'}{1+2\\sqrt{2}\\bar\\tau\\kappa_2} \\quad\\mbox{with}\\quad\n\\gamma':=\\min\\Big\\{\\frac{\\hat\\gg}{2\\tau},\\gamma_3,\\gamma_4,\\dfrac{\\gamma_1}{1+\\sqrt{2}\\bar\\tau\\kappa_2},\\dfrac{\\gamma_2}{1+\\sqrt{2}\n\\bar\\tau\\kappa_2}\\Big\\}.\n\\end{equation}\nMoreover, pick $q\\in (0,1) $ and set \n\\begin{equation}\\label{rhof}\n\\bar\\varrho:=\\max\\big\\{\\bar\\rho, 4\\kappa_1\\kappa_2\\bar\\tau, \\frac{2\\sqrt{2} \\kappa_1\\kappa_2^2\\bar\\tau^2}{q}\\big\\},\n\\end{equation}\nwhere  $\\bar\\rho$ is taken from Theorem~\\ref{ugrowth}. \nArguing by induction, we aim at showing that for any starting point $(x^0,\\lambda^0)\\in\\big(\\Th\\times\\R^m\\big)\\cap\\B_{\\bar\\gamma}(\\ox,\\olm)$ there exists a sequence $\\{(\\xk,\\lk)\\c\\subset \\Th \\times\\R^m$, generated by Algorithm~\\ref{Alg1}\nwith $\\rok \\ge \\bar\\varrho$ and   $\\ek=\\sigma(\\xk,\\lk)$ for all $k$,\n such that for all $k=0,1,\\ldots$ the relationships\n\\begin{equation}\\label{c1}\n(\\xk,\\lk)\\in\\B_{\\gamma'}(\\ox,\\olm),\n\\end{equation}\n\\begin{equation}\\label{c2}\n\\dist\\big(-\\nabla_x L(\\xkk,\\lkk), N_\\Th(\\xkk)\\big)\\le\\ek\\quad\\textrm{ and }\\quad\\Vert\\xkk-\\xk\\Vert+\\Vert\\lkk-\\lk\\Vert\\le\\bar\\tau R_k\n\\end{equation}\nhold. To achieve this goal, set $k=0$, pick $(x^0,\\lambda^0)\\in\\big(\\Th\\times\\R^m\\big)\\cap\\B_{\\bar\\gamma}(\\ox,\\olm)$, and observe from  $\\bar\\gamma\\le\\gamma'$ that \\eqref{c1} holds in this case. We also have $\\lambda^0\\in\\B_{\\hat\\gg/2\\tau}(\\olm)$ and $\\rho_0\\ge \\bar\\varrho$ and then find by Proposition~\\ref{solv} a primal iterate $x^1\\in \\inte \\B_{\\hat\\gamma}(\\ox)$ satisfying $-\\nabla_x\\L(x^1,\\lambda^0,\\rho_0)\\in N_\\Th(x^1)$. Define further \nthe multiplier $\\lm^1:=\\nabla e_{1^{}/\\rho_0}g\\big(\\Phi(x^1)+ \\rho_0^{-1}\\lambda^0\\big)$ and observe that\n\\begin{equation*}\n-\\nabla_x L(x^1,\\lm^1)=-\\nabla_x\\L(x^1,\\lambda^0,\\rho_0)\\in N_\\Th(x^1).\n\\end{equation*}\nMoreover, employing  Theorem~\\ref{est} confirms that \n\\begin{equation*}\n\\Vert x^1-x^0\\Vert+\\Vert\\lm^1-\\lambda^0\\Vert\\le\\bar\\tau R_0.\n\\end{equation*}\nThus $(x^1,\\lambda^1)$ is well defined and satisfies the claimed condition \\eqref{c2} for $k=0$. Assume by induction that the iterates $(\\xk,\\lk)$, $k=0,1,\\ldots,s+1$, are well defined  and that   both conditions \\eqref{c1} and \\eqref{c2} hold for $k=0,1,\\ldots,s$. We verify now the existence of $(x^{s+2},\\lambda^{s+2})$ satisfying \\eqref{c1} and \\eqref{c2} for $k=s+1$. To this end, we first show that $(x^{s+1},\\lambda^{s+1})\\in\\B_{\\gamma'}(\\ox,\\olm)$. \nTo proceed, fix an integer   $k$ with $0\\le k\\le s$.\nSince $(\\xk,\\lk)\\in\\B_{\\gamma'}(\\ox,\\olm)$, it follows from \\eqref{eb1} and \\eqref{c1} that\n\\begin{eqnarray}\\label{c3}\nR_k&\\le&\\kappa_2\\big(\\Vert\\xk-\\ox\\Vert+\\dist(\\lk,\\Lambda(\\ox))\\big)\\\\\n&\\le&\\kappa_2\\big(\\Vert\\xk-\\ox\\Vert+\\Vert\\lk-\\olm\\Vert\\big)\\le\\sqrt{2}\\kappa_2\\Vert(\\xk,\\lk)-(\\ox,\\olm)\\Vert\\le\\sqrt{2}\\kappa_2\\gamma'.\\nonumber\n\\end{eqnarray}\nCombining this with \\eqref{c1} and the second relation in \\eqref{c2}, we conclude that\n\\begin{eqnarray}\\label{c4}\n\\Vert(\\xkk,\\lkk)-(\\ox,\\olm)\\Vert&\\le&\\Vert\\xkk-\\xk\\Vert+\\Vert\\lkk-\\lk\\Vert+\\Vert(\\xk,\\lk)-(\\ox,\\olm)\\Vert\\nonumber\\\\\n&\\le&\\bar\\tau R_k+\\gamma'\\le(\\sqrt{2}\\bar\\tau\\kappa_2+1)\\gamma'\\le\\gamma_1,\n\\end{eqnarray}\nwhich implies that $(\\xkk,\\lkk)\\in\\B_{\\gamma_1}(\\ox,\\olm)$. Using \\eqref{subr3}, we obtain\n\\begin{eqnarray}\\label{c5}\n&&\\Vert\\xkk-\\ox\\Vert+\\dist\\big(\\lkk,\\Lambda(\\ox)\\big)\\nonumber\\\\\n&\\le&\\kappa_1 R_{k+1} =\\kappa_1\\Big(\\dist\\big(-\\nabla_x L(\\xkk,\\lkk), N_\\Th(\\xkk)\\big) +\\big\\Vert\\Phi(\\xkk)-{\\rm{prox}}_g\\big(\\Phi(\\xkk)+\\lkk\\big)\\big\\Vert\\Big)\\nonumber\\\\\n&\\le&\\kappa_1\\ek+\\kappa_1\\big\\Vert\\Phi(\\xkk)-{\\rm{prox}}_g\\big(\\Phi(\\xkk)+\\lkk\\big)\\big\\Vert.\n\\end{eqnarray}\nTo proceed further,  define\n\\begin{equation}\\label{skk}\n\\skk:=\\proxk\\big(\\Phi(\\xkk)+ \\rho_k^{-1}{\\lk}\\big ).\n\\end{equation}\nIt follows from  \\eqref{lkk}  and \\eqref{ueq19} that\n\\begin{equation}\\label{eq3}\n\\Phi(\\xkk)-\\skk= \\rho_k^{-1}(\\lkk-\\lk).\n\\end{equation}\nSince $\\lkk=\\nabla\\menvk\\big(\\Phi(\\xkk)+ \\rho_k^{-1}{\\lk}\\big)$, we deduce from the second equality in \\eqref{prm} that \n\\begin{equation*}\\label{clq17}\n\\lkk   \\in \\sub g\\big(\\Phi(\\xkk)+ \\rho_k^{-1}(\\lk-\\lkk)\\big)=\\partial g(\\skk).\n\\end{equation*} \nAppealing now to the first equality in \\eqref{prm} yields $\\skk={\\rm{prox}}_g(\\skk+\\lkk)$. Since the mapping $y\\mapsto y-{\\rm{prox}}_g(y+\\lkk)$ is nonexpansive (cf. \\cite[Proposition~12.27]{bc}), we arrive at the relationships\n\\begin{eqnarray}\\label{eq2}\n&&\\big\\Vert\\Phi(\\xkk)-{\\rm{prox}}_g\\big(\\Phi(\\xkk)+\\lkk\\big)\\big\\Vert\\nonumber\\\\\n&=&\\big\\Vert\\Phi(\\xkk)-{\\rm{prox}}_g\\big(\\Phi(\\xkk)+\\lkk\\big)\\big\\Vert-\\big\\Vert\\skk-{\\rm{prox}}_g(\\skk+\\lkk)\\big\\Vert\\nonumber\\\\\n&\\le&\\big\\Vert\\Phi(\\xkk)-{\\rm{prox}}_g\\big(\\Phi(\\xkk)+\\lkk\\big)-\\big(\\skk-{\\rm{prox}}_g(\\skk+\\lkk)\\big)\\big\\Vert\\nonumber\\\\\n&\\le&\\Vert\\Phi(\\xkk)-\\skk\\Vert.\n\\end{eqnarray}\nCombining \\eqref{c5}, \\eqref{eq3}, and \\eqref{eq2} brings us to\n\\begin{equation*}\n\\Vert\\xkk-\\ox\\Vert+\\dist\\big(\\lkk,\\Lambda(\\ox)\\big)\\le\\kappa_1\\ek+\\rho_k^{-1}{\\kappa_1}\\Vert\\lkk-\\lk\\Vert.\n\\end{equation*}\nThis,  together with \\eqref{ek}, \\eqref{rhof}, the second inequality in \\eqref{c2}, and \\eqref{c3} results in \n\\begin{eqnarray}\\label{c10}\n\\Vert \\xkk-\\ox\\Vert+\\dist\\big(\\lkk,\\Lambda(\\ox)\\big)&\\le&\\kappa_1\\ek+\\dfrac{\\bar\\tau\\kappa_1}{\\rok}R_k\\le\\dfrac{1}{4\\kappa_2}R_k +\\dfrac{1}{4\\kappa_2}R_k\\\\\n&\\le&\\dfrac{1}{2}\\Big(\\Vert\\xk-\\ox\\Vert+\\dist\\big(\\lk,\\Lambda(\\ox)\\big)\\Big)\\nonumber,\n\\end{eqnarray}\nwhich in turn implies the primal-dual estimates\n\\begin{equation}\\label{c6}\n\\Vert\\xkk-\\ox\\Vert+\\dist\\big(\\lkk,\\Lambda(\\ox)\\big)\\le\\dfrac{1}{2^{k+1}}\\Big(\\Vert x^0-\\ox\\Vert+\\dist\\big(\\lambda^0,\\Lambda(\\ox)\\big)\\Big).\n\\end{equation}\nThis estimate, \\eqref{c3},  and the second inequality in \\eqref{c2} tell us that \n\\begin{eqnarray}\\label{c7}\n\\Vert(x^{s+1},\\lambda^{s+1})-(x^0,\\lambda^0)\\Vert&\\le&\\sum\\limits_{k=0}^s\\Vert(\\xkk,\\lkk)-(\\xk,\\lk)\\Vert \\le\\bar\\tau\\sum\\limits_{k=0}^s R_k\\nonumber\\\\\n&\\le&\\bar\\tau\\kappa_2\\sum\\limits_{k=0}^s\\Vert\\xk-\\ox\\Vert+\\dist\\big(\\lk,\\Lambda(\\ox)\\big)\\nonumber\\\\\n&\\le&\\bar\\tau\\kappa_2\\sum\\limits_{k=0}^s\\dfrac{1}{2^k}\\Big(\\Vert x^0-\\ox\\Vert+\\dist\\big(\\lambda^0,\\Lambda(\\ox)\\big)\\Big)\\\\\n&\\le&2\\bar\\tau\\kappa_2\\big(\\Vert x^0-\\ox\\Vert+\\dist(\\lambda^0,\\Lambda(\\ox))\\big)\\nonumber\\\\\n&\\le&2\\bar\\tau\\kappa_2\\big(\\Vert x^0-\\ox\\Vert+\\Vert\\lambda^0-\\olm\\Vert\\big).\\nonumber\n\\end{eqnarray}\nThus, we arrive at the estimates\n\\begin{eqnarray*}\n\\Vert(x^{s+1},\\lambda^{s+1})-(\\ox,\\olm)\\Vert&\\le&\\Vert(x^{s+1},\\lambda^{s+1})-(x^0,\\lambda^0)\\Vert+\\Vert(x^0,\\lambda^0)-(\\ox,\\olm)\\Vert\\\\\n&\\le&2\\bar\\tau\\kappa_2\\big(\\Vert x^0-\\ox\\Vert+\\Vert\\lambda^0-\\olm\\Vert\\big)+\\Vert(x^0,\\lambda^0)-(\\ox,\\olm)\\Vert\\\\\n&\\le&(2\\sqrt{2}\\bar\\tau\\kappa_2+1)\\Vert(x^0,\\lambda^0)-(\\ox,\\olm)\\Vert\\le(2\\sqrt{2}\\bar\\tau\\kappa_2+1)\\bar\\gamma=\\gamma',\n\\end{eqnarray*}\nwhere the last inequality follows from $(x^0,\\lambda^0)\\in\\B_{\\bar\\gamma}(\\ox,\\olm)$, while the last equality comes from the definition of $\\bar\\gamma$ in \\eqref{const}. \nThis verifies that $(x^{s+1},\\lambda^{s+1})\\in\\B_{\\gamma'}(\\ox,\\olm)$. By  \\eqref{const}, we get  $\\lambda^{s+1}\\in\\B_{\\hat\\gg/2\\tau}(\\olm)$, and hence Proposition~\\ref{solv} ensures the existence of of an optimal solution $x^{s+2}$ \nto \\eqref{regp} associated with $(\\lm^{s+1},\\rho_{s+1})$ such that $x^{s+2}\\in \\inte\\B_{\\hat\\gamma}(\\ox)$. The latter yields \n\\begin{equation*}\n-\\nabla_x\\L(x^{s+2},\\lambda^{s+1},\\rho_{s+1})\\in N_\\Th(x^{s+2}).\n\\end{equation*} \nSet now $\\lm^{s+2}:=\\nabla e_{1^{}/\\rho_{s+1}}\\big(\\Phi(x^{s+2})+\\rho_{s+1}^{-1}\\lambda^{s+1}\\big)$ and observe that\n\\begin{equation*}\n -\\nabla_x L(x^{s+2},\\lm^{s+2}) =  -\\nabla_x\\L(x^{s+2},\\lambda^{s+1},\\rho_{s+1}) \\in N_\\Th(x^{s+2}),\n\\end{equation*}\nwhich by Proposition~\\ref{est} brings us to the estimate\n\\begin{equation*}\n\\Vert x^{s+2}-x^{s+1}\\Vert+\\Vert\\lm^{s+2}-\\lambda^{s+1}\\Vert\\le\\bar\\tau R_{s+1}.\n\\end{equation*}\nThis tells us that the iterate $(x^{s+2},\\lambda^{s+2})$ satisfies   \\eqref{c2} for $k=s+1$, which completes  the proof of  induction. Thus there exists a sequence  $\\{(\\xk,\\lk)\\c\\subset\\Th\\times\\R^m$, generated by Algorithm~\\ref{Alg1} \nwith $\\rok \\ge \\bar\\varrho$ and   $\\ek=o\\big(R(\\xk,\\lk)\\big)$ for all $k$, for which  \\eqref{c1} and \\eqref{c2} fulfill.\n\n \nNext, we are going to justify  the convergence of this primal-dual   sequence. Using the same arguments as in the proofs of \\eqref{c6} and \\eqref{c7} leads us to\n\\begin{equation}\\label{c8}\n\\Vert(x^{k+l},\\lambda^{k+l})-(\\xk,\\lk)\\Vert\\le 2\\bar\\tau\\kappa_2\\big(\\Vert\\xk-\\ox\\Vert+\\dist(\\lk,\\Lambda(\\ox))\\big)\\quad\\mbox{for all}\\;\\;k,l\\in\\N.\n\\end{equation}\nIt follows from \\eqref{c6} that the right-hand side of \\eqref{c8} goes to zero as $k\\to\\infty$, which tells us that the sequence $\\{(\\xk,\\lk)\\c$ is Cauchy. By \\eqref{c6},   this sequence converges  to $(\\ox,\\hat\\lambda)$ for some $\\hat\\lambda\\in\\Lambda(\\ox)$.\nNow, we show  that  the rate of this convergence is linear. Indeed, letting $l\\to\\infty$ in \\eqref{c8} tells us that for all $k\\in\\N$ we have\n\\begin{equation*}\\label{c9}\n\\Vert(\\xk,\\lk)-(\\ox,\\hat\\lambda)\\Vert\\le 2\\bar\\tau\\kappa_2\\big(\\Vert\\xk-\\ox\\Vert+\\dist(\\lk,\\Lambda(\\ox))\\big),\n\\end{equation*}\nwhich together with the first inequality in \\eqref{c10} verifies that\n\\begin{eqnarray*}\n\\Vert(\\xkk,\\lkk)-(\\ox,\\hat\\lambda)\\Vert&\\le&2\\bar\\tau\\kappa_2\\big(\\Vert\\xkk-\\ox\\Vert+\\dist(\\lkk,\\Lambda(\\ox))\\big)\\nonumber\\\\\n&\\le&2\\bar\\tau\\kappa_1\\kappa_2\\big(R_k^{-1}{\\ek}+\\rho_k^{-1}\\bar\\tau\\big)R_k\\nonumber\\\\\n&\\le&2\\bar\\tau\\kappa_1\\kappa_2^2\\big(R_k^{-1}{\\ek}+\\rho_k^{-1}\\bar\\tau\\big)\\big(\\Vert\\xk-\\ox\\Vert+\\dist(\\lk,\\Lambda(\\ox))\\big)\\nonumber\\\\\n&\\le&2\\bar\\tau\\kappa_1\\kappa_2^2\\big(R_k^{-1}{\\ek}+\\rho_k^{-1}\\bar\\tau\\big)\\big(\\Vert\\xk-\\ox\\Vert+\\Vert\\lk-\\hat\\lambda\\Vert\\big)\\nonumber\\\\\n&\\le& 2\\sqrt{2}\\bar\\tau\\kappa_1\\kappa_2^2\\big(R_k^{-1}{\\ek}+\\rho_k^{-1}\\bar\\tau\\big)\\Vert(\\xk,\\lk)-(\\ox,\\hat\\lambda)\\Vert.\n\\end{eqnarray*}\nCombining this with $\\rok\\ge \\bar\\varrho$  with $ \\bar\\varrho$ taken from \\eqref{rhof} and using the choice of the tolerance $\\ek=o(R_k)$ result in \n\\begin{equation*}\n\\limsup_{k\\to\\infty}\\dfrac{\\Vert(\\xkk,\\lkk)-(\\ox,\\hat\\lambda)\\Vert}{\\Vert(\\xk,\\lk)-(\\ox,\\hat\\lambda)\\Vert}\\le\\limsup_{k\\to\\infty}2\\sqrt{2}\\bar\\tau\n\\kappa_1\\kappa_2^2\\big(R_k^{-1}{\\ek}+\\rho_k^{-1}\\bar\\tau\\big)\\le q.\n\\end{equation*}\nIt follows from  $q\\in (0,1)$ that the rate of   convergence of $\\{(\\xk,\\lk)\\c$ is linear. Furthermore, it is clear from the above inequality that if $\\rok\\to \\infty$, we \narrive at the superlinear convergence of the latter primal-dual sequence, which completes the proof.\n\\end{proof}","sha256_utf8":"f9acbf089f1af9f52c52af8d8ab8a5fbdabd05f0adf4cb349a0c822f8f35dee8"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"复合优化模型与光滑性","source_units":[{"start":7530,"end":7968,"start_line":215,"end_line":219,"raw_tex":"Given twice continuously differentiable functions   $\\ph:\\R^n\\to \\R$ and  $\\Phi:\\R^n\\to \\R^m$,  a convex polyhedral set  $\\Th$ in $\\R^n$, and a convex piecewise linear-quadratic (CPLQ) function $g:\\R^m\\to \\oR$,\n we aim to present a local convergence analysis of the ALM for  the composite optimization problem \n\\begin{equation}\\label{comp}\n\\mbox{minimize}\\;\\;\\ph(x)+g\\big(\\Phi(x)\\big)\\quad \\mbox{subject to}\\;\\; x\\in \\Th.\n\\end{equation} \n","sha256_utf8":"ec3bf2accbeb796338f368c588ce9b2d319cd76aba05dfd4f18d48493fcf67e2"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"包络与增广子问题","source_units":[{"start":8916,"end":9921,"start_line":227,"end_line":240,"raw_tex":"The augmented Lagrangian $\\L:\\R^n\\times \\R^m\\times (0,\\infty)\\to \\R$ associated with the composite problem \\eqref{comp} is defined  by \n\\begin{equation}\\label{aug}\n\\L(x,\\lm,\\rho):=\\ph(x)+e_{ {1}/{\\rho}} g \\big(\\Phi(x)+\\rho^{-1}\\lm\\big)-\\sm\\rho^{-1}\\|\\lm\\|^2\n\\end{equation}\nwith $e_{1/\\rho} g$ signifying  the Moreau envelope of   $g$, given  by \n\\begin{equation*}\\label{Menv}\ne_{1/\\rho} g(y):=\\inf_{z\\in \\R^m}\\big\\{g(z)+\\frac{\\rho}{2}\\|y-z\\|^2\\big\\}, \\quad y\\in \\R^m.\n\\end{equation*}\nIt is well-known that the augmented Lagrangian $\\L$ is continuously differentiable with respect to both $x$ and $\\lm$ and is nondecreasing with the respect to $\\rho$, the {\\em penalty} parameter of the augmented Lagrangian. \nAt  each iteration for a given pair $(\\lm,\\rho)\\in \\R^m\\times (0,\\infty)$,  our proposed  ALM for  the  composite problem \\eqref{comp}\nrequires solving the {\\em partially} augmented problem \n\\begin{equation}\\label{augp}\n\\mini \\L(x, \\lm, \\rho)\\quad \\mbox{subject to}\\;\\; x\\in \\Th.\n\\end{equation} \n","sha256_utf8":"e1d931ab1360ca165290e6dff66d97899c4ff365650f37c5cc6c7801e13591ec"}]},{"id":"S005","semantic_type":"notation","kind":"notation","title":"球范数投影及渐近记号","source_units":[{"start":14622,"end":15893,"start_line":277,"end_line":286,"raw_tex":"In this section, we first briefly review basic constructions of variational analysis and generalized differentiation employed in the paper. In what follows,    we \n denote by $\\B$ the closed unit ball in the space in question and by $\\B_r(x):=x+r\\B$ the closed ball centered at $x$ with radius $r>0$. \n In the  product space $\\R^n\\times \\R^m$, we use the norm $\\|(w,u)\\|=\\sqrt{\\|w\\|^2+\\|u\\|^2}$ for any $(w,u)\\in \\R^n\\times \\R^m$.\n Given a nonempty set $C\\subset\\R^n$, the symbols $\\inte C$,  $\\cone C$, and $\\span C$ signify its interior, conic hull, and the   linear space generated by  $C$, respectively. \n For any set $C$ in $\\R^n$, its indicator function is defined by $\\dd_C(x)=0$ for $x\\in C$ and $\\dd_C(x)=\\infty$ otherwise. We denote\n by $P_C$ the projection mapping onto $C$ and  by $\\dist(x,C)$  the distance between $x\\in \\R^n$ and a set $C$.\n For a vector $w\\in \\R^n$, the subspace $\\{tw |\\, t\\in \\R\\}$ is denoted by $[w]$. \nWe write $x(t)=o(t)$ with $x(t)\\in \\R^n$ and $t>0$ to mean that ${\\|x(t)\\|}/{t}$ goes to $0$ as $t\\dn 0$.\nFinally, we denote by $\\R_+$ (respectively,  $\\R_-$) the set of non-negative (respectively, non-positive) real numbers.\nFor an $n\\times m$ matrix $A$, we denote by $\\ker A$ and $\\rge A$ its null and range spaces, respectively.\n","sha256_utf8":"05e44e4d2ae58771a47aeddcd12b42ef23e5d5ae25c4d3332398d743fd83e068"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"集合收敛与一阶锥构造","source_units":[{"start":15894,"end":16933,"start_line":288,"end_line":303,"raw_tex":"A sequence    $\\{C^k\\b$ of subsets in $\\R^n$ is called convergent to a set $C\\subset\\R^n$ as $k\\to \\infty$ if $C$ is closed and\n\\begin{equation*}\n\\lim_{k\\to \\infty}{\\rm dist}(w,C^k)={\\rm dist}(w,C)\\;\\mbox{ for all }\\;w\\in\\R^n.\n\\end{equation*}\nGiven a nonempty set $C\\subset\\R^n$ with $\\ox\\in C$, the  tangent cone $T_ C(\\ox)$ to $C$ at $\\ox$ is defined by\n\\begin{equation*}\\label{2.5}\nT_C(\\ox)=\\big\\{w\\in\\R^n|\\;\\exists\\,t_k{\\dn}0,\\;\\;w_k\\to w\\;\\;\\mbox{ as }\\;k\\to\\infty\\;\\;\\mbox{with}\\;\\;\\ox+t_kw_k\\in C\\big\\}.\n\\end{equation*}\nWhen $C$ is convex, the  {normal cone} to $C$ at $\\ox\\in C$ in the sense of convex analysis  is given by\n\\begin{equation*}\nN_C(\\ox)=\\big\\{v\\in\\R^n\\;\\big|\\; \\la v,x-\\ox\\ra\\le 0\\quad \\mbox{for all}\\;\\; x\\in C\\big\\}.\n\\end{equation*}\nFor a convex  function $f:\\R^n \\to \\oR:=(-\\infty,\\infty]$  with $\\ox\\in \\dom f$, its   subdifferential at $\\ox$   is defined  by \n\\begin{equation*}\\label{sub}\n\\partial f(\\ox)=\\big\\{v\\in\\R^n\\;\\big|\\;  \\la v,x-\\ox\\ra\\le f(x)-f(\\ox)\\;\\;\\mbox{for all}\\;x\\in \\R^n\\big\\}.\n\\end{equation*}\n","sha256_utf8":"7ac4752a82697888da7bd7fad1a2f1dc44333a141550b14b10028dc407bd9dea"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"临界锥与二阶上图导数","source_units":[{"start":16934,"end":19286,"start_line":305,"end_line":339,"raw_tex":"A sequence $\\{f^k\\b$ of functions $f^k:\\R^n\\to \\oR$ is said to {\\em epi-converge} to a function $f:\\R^n\\to [-\\infty,\\infty]$ if we have $\\epi f^k\\to \\epi f$ as $k\\to \\infty$.\nWe denote by $f^k\\overset {\\ss \\mbox{e}}{\\to} f$ the  epi-convergence of the sequence $\\{f^k\\b$ to $f$.\nGiven  a function  $f:\\R^n \\to \\oR$ and  a point $\\ox$ with $f(\\ox)$ finite, the subderivative function $\\d f(\\ox)\\colon\\R^n\\to[-\\infty,\\infty]$ is defined by\n\\begin{equation}\\label{fsud}\n{\\mathrm d}f(\\ox)(w)=\\liminf_{\\substack{\n   t\\dn 0 \\\\\n  w'\\to w\n  }} {\\frac{f(\\ox+tw')-f(\\ox)}{t}}.\n\\end{equation}\n The critical cone of $f$ at $\\ox$ for $\\ov$ with $(\\ox,\\ov)\\in \\gph \\sub f$ is defined by \n$$\n{K_f}(\\ox,\\ov)=\\big\\{w\\in \\R^n\\big|\\;\\la\\bar v,w\\ra=\\d f(\\ox)(w)\\big\\}.\n$$\nWhen $f=\\dd_C$, namely the indicator function of $C$, the critical cone of $\\dd_C$ at $\\ox$ for $\\ov$ is denoted by $K_C(\\ox,\\ov)$. In this case, the above definition of the critical cone of a function \nboils down to  the well-known concept of the critical cone of a set (see \\cite[page~109]{dr}), namely $K_C(\\ox,\\ov)=T_C(\\ox)\\cap [\\ov]^\\perp$.\nDefine the parametric  family of \nsecond-order difference quotients for $f$ at $\\ox$ for $\\ov\\in \\R^n$ by \n\\begin{equation*}\\label{lk01}\n\\Delta_t^2 f(\\bar x , \\ov)(w)=\\dfrac{f(\\ox+tw)-f(\\ox)-t\\langle \\ov,\\,w\\rangle}{\\frac{1}{2}t^2}\\quad\\quad\\mbox{with}\\;\\;w\\in \\R^n, \\;\\;t>0.\n\\end{equation*}\nIf $f(\\ox)$ is finite, then the {second subderivative} of $f$ at $\\ox$ for $\\ov$   is given by \n\\begin{equation*}\\label{ssd}\n\\d^2 f(\\bar x , \\ov)(w)= \\liminf_{\\substack{\n   t\\dn 0 \\\\\n  w'\\to w\n  }} \\Delta_t^2 f(\\ox , \\ov)(w'),\\;\\; w\\in \\R^n.\n\\end{equation*}\nThe second subderivative of $f$ at $\\ox$ (without mention of $\\ov$) is defined by \n\\begin{equation}\\label{ssud}\n\\d^2 f(\\ox)(w)=\\liminf_{\\substack{t\\dn 0\\\\\nw'\\to w}} \\frac{f(\\ox+tw')-f(\\ox)-t\\d f(\\ox)(w')}{\\sm t^2},\\quad w\\in \\R^n.\n\\end{equation}\nFollowing \\cite[Definition~13.6]{rw}, a function $f:\\R^n \\to \\oR$ is said to be {twice epi-differentiable} at $\\bar x$ for $\\ov\\in\\R^n$, with $f(\\ox) $ finite, \nif the sets $\\epi \\Delta_t^2 f(\\bar x , \\ov)$ converge to $\\epi \\d^2 f(\\bar x,\\ov)$ as $t\\downarrow 0$. \nIn what follows, we refer to the second subderivative of $f$ at $\\bar x$ for $\\ov$ by the {\\em second-order epi-derivative} of $f$ when it is twice epi-differentiable at   $\\bar x$ for $\\ov$.\n","sha256_utf8":"a0d8aa2b15b72b1ad1095ceabd37106583aebf6bf6ad0bcba4fc70f336c7a456"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"分片二次表示及次微分","source_units":[{"start":19288,"end":20278,"start_line":341,"end_line":354,"raw_tex":"Recall  that a function $g:\\R^m\\to \\oR$ is called  piecewise linear-quadratic if $\\dom g = \\cup_{i=1}^{s} C_i$ with $s\\in \\N$ and $C_i $ being polyhedral convex  sets for $i = 1, \\ldots, s$, and if $g$ has a representation of the form\n\\begin{equation} \\label{PWLQ}\ng(z) = \\sm \\langle A_i z ,z \\rangle + \\langle a_i ,z \\rangle + \\alpha_i  \\quad \\mbox{for all} \\quad  z \\in C_i,\n\\end{equation}\nwhere $A_i$ is an $m \\times m$ symmetric matrix, $a_i\\in \\R^m$, and $\\alpha_i\\in \\R$ for all $i = 1, \\cdots, s$. Take $\\oz\\in \\dom g$ and define the {\\em active} indices  of the domain of $g$ at $\\oz$ by\n\\begin{equation}\\label{act}\nI(\\oz)=\\big \\{i\\in \\{1,\\ldots, m\\} |\\, \\oz\\in C_i\\big \\}.\n\\end{equation}\nIf in addition $g$ is convex, \nit follows   from \\cite[page~487]{rw} that \nif  $\\oz\\in \\dom g$, the subdifferential of $g$ at  $\\oz$ can be calculated by \n\\begin{equation}\\label{sub}\n\\sub g(\\oz)=\\bigcap_{i\\in I(\\oz)} \\big \\{v\\in \\R^m |\\, v-A_i \\oz- a_i\\in N_{ C_i}(\\oz)\\big \\}.\n\\end{equation}\n","sha256_utf8":"0941f0498ff46f11d09c57cc82e42e27ce665b9c29a86a117e3829b5ef69e535"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"KKT系统与乘子集合","source_units":[{"start":21537,"end":22217,"start_line":383,"end_line":392,"raw_tex":"The   Karush-Kuhn-Tucker (KKT)   system associated with the composite problem \\eqref{comp} is given by \n\\begin{equation}\\label{vs} \n0\\in \\nabla_x L(x,\\lm)+N_\\Th(x),\\;\\;\\lm\\in \\sub g(\\Phi(x)),\n\\end{equation}\nwhere $L(x,\\lm)= \\ph(x)+\\la \\Phi(x),\\lm\\ra$ for any   $(x,\\lm)\\in \\R^n\\times \\R^m$. \nGiven a point $\\ox\\in\\R^n$, we define the set of {  Lagrange multipliers} of the KKT system \\eqref{vs} associated with $\\ox$ by\n\\begin{equation}\\label{laset}\n\\Lambda(\\ox):=\\big\\{\\lm\\in\\R^m\\;\\big|\\; 0\\in \\nabla_x L(\\ox,\\lm)+N_\\Th(\\ox),\\;\\lm\\in \\sub g(\\Phi(\\ox))\\big\\}.\n\\end{equation}\nIf  $(\\ox,\\olm)$ is a solution to the KKT  system \\eqref{vs},   we clearly have   $\\olm\\in\\Lambda(\\ox)$.\n","sha256_utf8":"0f8f3196f35e163b54bb3d3d9702fe5396cbc7cbeff626e5880945b1d56119b8"}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","title":"二阶充分条件及方向锥","source_units":[{"start":22217,"end":22787,"start_line":393,"end_line":401,"raw_tex":" Given  a solution $(\\ox,\\olm)$  to \\eqref{vs}, the second-order sufficient condition (SOSC)\nfor the composite problem \\eqref{comp} at $(\\ox,\\olm)$ is formulated by \n\\begin{equation}\\label{sosc}\n\\langle\\nabla_{xx}^2L(\\bar x,\\olm)w,w\\rangle+\\d^2g (\\Phi(\\bar x),\\olm) (\\nabla \\Phi(\\ox)w )>0\\quad \\mbox{for all }\\; w\\in \\D\\setminus\\{0\\},\n\\end{equation}\nwhere the convex cone $\\D$ is defined by \n \\begin{equation}\\label{coned}\n \\D:=K_\\Th\\big(\\ox,-\\nabla_x L(\\ox,\\olm)\\big)\\cap \\big\\{w\\in \\R^n\\big|\\; \\nabla \\Phi(\\ox)w\\in K_g \\big(\\Phi(\\ox),\\olm \\big)\\big\\}.\n \\end{equation}\n","sha256_utf8":"8df635aa47fdaa756ae80cdb936d269edf0acd3dab331707cfe7065644fb5f23"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"近端映射与预解恒等式","source_units":[{"start":23093,"end":24019,"start_line":405,"end_line":414,"raw_tex":"a result, obtained recently in \\cite{s20}, showing that it yields an error bound estimate for the KKT system \\eqref{vs}, which is central to our   local convergence analysis  in this paper. To this end,  recall that, given a constant $r>0$,    the proximal mapping of  a convex function $g:\\R^m\\to \\oR$ is defined by \n\\begin{equation*}\\label{pr}\n{\\rm{prox}}_{rg}(x):=\\underset{z\\in \\R^n}{\\argmin} \\big\\{g(z)+\\rsm\\|x-z\\|^2\\big\\},\\quad x\\in\\R^m.\n\\end{equation*}\nIn what follows, when $r=1$, the proximal mapping of $g$ will be denoted by ${\\rm{prox}_g}$. For such a convex function, \nit is known that for any $x\\in \\R^m$ we have \n\\begin{equation}\\label{prm}\n{\\rm{prox}}_{rg}(x)=\\big(I+r\\sub g\\big)^{-1}(x) \\quad \\mbox{and}\\quad \\nabla e_rg(x)=\\big(rI+(\\sub g)^{-1}\\big)^{-1}(x),\n\\end{equation}\nwhere $I$ stands for the $m\\times m$ identity matrix; see \\cite[Proposition~12.19]{rw} and \\cite[Theorem~2.26]{rw} for more details. \n","sha256_utf8":"a3b18c6f796e8e8b27fe85d15d365d68dc812c56b3f7a71cf939d24f1661c64c"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"完全可处理性","source_units":[{"start":25969,"end":26408,"start_line":439,"end_line":445,"raw_tex":"A function $f:\\R^n\\to \\oR$ is called fully amenable at $\\ox\\in \\dom f$ if there is a neighborhood ${\\cal O}$ of $\\ox$ on which \n$f$ can be represented in the form $f=\\psi\\circ h$, where $\\psi:\\R^m\\to \\oR$ is a CPLQ function and   $h:\\R^n\\to \\R^m$ is a ${\\cal C}^2$ function, and the \nbasic constraint qualification \n\\begin{equation}\\label{bcq}\nN_{\\ss \\dom \\psi}\\big(h(\\ox)\\big)\\cap \\ker \\nabla h(\\ox)^*=\\{0\\}\n\\end{equation}\nis satisfied. \n","sha256_utf8":"80e655996c2b3fb754867b1fa908629fff395e47e5d56b7dde95bf4f43f3bc92"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"一次与二次半可微性","source_units":[{"start":32774,"end":33085,"start_line":536,"end_line":537,"raw_tex":"Recall that a function $f:\\R^n\\to \\oR$ is called semidifferentiable at $\\ox$ if it is finite at $\\ox$ and the ``$\\liminf$\" in \\eqref{fsud} is a ``$\\lim$.\"\nFurthermore, $f$ is said to be twice semidifferentiable at $\\ox$ if it is semidifferentiable at $\\ox$ and  the ``$\\liminf$\" in \\eqref{ssud} is a ``$\\lim$.\"\n","sha256_utf8":"1773ba9ca83097660e14f087928954be82359f9d9956255600fa1adb7bf9a2c0"}]},{"id":"S014","semantic_type":"algorithm","kind":"algorithm","title":"原始更新与近似驻点准则","source_units":[{"start":57280,"end":58400,"start_line":829,"end_line":846,"raw_tex":"Recall that for  the current triple $(\\xk, \\lk,\\rok)\\in \\R^n\\times \\R^m\\times (0,\\infty)$, the next primal iterate $\\xkk$ in the ALM is computed as an optimal solution to \n  the subproblem\n\\begin{equation}\\label{conv1}\n\\mini \\L(x, \\lk, \\rok)\\quad \\mbox{subject to}\\quad x\\in \\Th.\n\\end{equation} \nThe next dual iterate $\\lkk$ is  then calculated as the Lagrange multiplier associated with the optimal solution $\\xkk$ to the subproblem \\eqref{conv1}, namely  \n\\begin{equation}\\label{lkk}\n \\lkk := \\nabla\\menvk\\big(\\Phi(\\xkk)+ \\rho_k^{-1}{\\lk}\\big).\n\\end{equation}\n Since an optimal solution  $\\xkk$ to the subproblem \\eqref{conv1} always satisfies the {\\em stationary}  condition\n\\begin{equation}\\label{conv2}\n0\\in \\nabla_x\\L(\\xkk, \\lk, \\rok) +N_\\Th(\\xkk),\n\\end{equation} \nand since it is often difficult to find an exact solution to \\eqref{conv1} in practice, it is more convenient to look for a solution $\\xkk$ satisfying the {\\em approximate} stationary condition\n\\begin{equation}\\label{xkk}\n\\dist\\big(-\\nabla_x\\L(\\xkk, \\lk, \\rok),N_\\Th(\\xkk)\\big)\\leq \\ek,\n\\end{equation} \nwhere $\\ek\\geq 0$ is a   tolerance parameter.\n","sha256_utf8":"53a5e8fabfcf52f8a23f0dfb5a5c3e837653e94ce25fd9deef7d17848a598a55"}]},{"id":"S015","semantic_type":"algorithm","kind":"algorithm","title":"增广拉格朗日迭代算法","source_units":[{"start":58433,"end":59096,"start_line":848,"end_line":855,"raw_tex":"\\begin{Algorithm}[augmented Lagrangian method]\\label{Alg1}\nChoose $(x^0,\\lambda^0)\\in\\Th\\times \\R^m$ and $\\bar\\rho>0$. Pick a sequence of tolerance parameters $\\{\\ek\\c$ with $\\ek>0$ for all $k$ and $\\ek\\to   0$ as $k\\to\\infty$ and a sequence of $\\{\\rok\\c$ such that  $\\rok\\ge \\bar\\rho$ for all $k$ and set $k:=0$. Then\n\\begin{itemize}[noitemsep,topsep=0pt]\n\\item [{\\rm (1)}] if $(\\xk,\\lk)$ satisfies a suitable termination criterion, stop;\n\\item[{\\rm (2)}]  otherwise, choose  $\\xkk$ such that  \\eqref{xkk} holds and then update the Lagrange multiplier  $\\lkk$ by \\eqref{lkk};\n\\item[{\\rm (3)}] set $k\\leftarrow k+1$ and go to Step~1.\n\\end{itemize}\n\\end{Algorithm}","sha256_utf8":"5f89db04668bdd41a3c9893ba6b400a27904f6b609011e603d5ef7f17bbe4446"}]},{"id":"S016","semantic_type":"assumption","kind":"assumption","title":"约束集的仿射表示","source_units":[{"start":65793,"end":66234,"start_line":953,"end_line":957,"raw_tex":"Thus, in the rest of this paper, we assume further that the  polyhedral convex set  $\\Th$ has a representation of the form \n\\begin{equation}\\label{Th}\n\\Th:=\\big\\{x\\in\\R^n\\,|\\, Bx = b\\big \\},\n\\end{equation} \nwhere $B$ is a $s\\times n$ matrix and $b$ is a vector in $\\R^s$. Furthermore, our convergence analysis of the ALM  requires to consider the \\textit{residual function} $R: \\R^n\\times \\R^m \\to \\R$ of the KKT system \\eqref{vs}, given by\n","sha256_utf8":"71ca0c9e198b3d28208910ab236cadffa317d983a5401bf588cbf3c6812760c3"}]},{"id":"S017","semantic_type":"definition","kind":"definition","title":"KKT残差函数","source_units":[{"start":66000,"end":66429,"start_line":957,"end_line":960,"raw_tex":"where $B$ is a $s\\times n$ matrix and $b$ is a vector in $\\R^s$. Furthermore, our convergence analysis of the ALM  requires to consider the \\textit{residual function} $R: \\R^n\\times \\R^m \\to \\R$ of the KKT system \\eqref{vs}, given by\n\\begin{equation}\\label{res}\nR(x, \\lm) : =  \\dist\\big(-\\nabla_x L(x, \\lm), N_\\Th(x)\\big)+\\big\\|\\Phi(x)-{\\rm{prox}}_g\\big(\\Phi(x)+\\lm\\big)\\big\\|, \\quad (x, \\lm) \\in \\R^n\\times \\R^m.\n\\end{equation}\n","sha256_utf8":"b286a32defb0ba1811a3259af23dd116f8023f290cfc7250523136c4730d9299"}]},{"id":"S018","semantic_type":"assumption","kind":"assumption","title":"相对残差消失的容差","source_units":[{"start":85121,"end":85514,"start_line":1192,"end_line":1196,"raw_tex":" convergence of generalized equations with non-isolated solutions. To this end, let  $\\sigma:\\R^n\\times \\R^m\\to \\R_+$ be a function that satisfies  the condition\n \\begin{equation}\\label{error}\n \\sigma(x,\\lm)=o\\big(R(x,\\lm)\\big)\\quad \\mbox{as}\\;\\;(x,\\lm)\\to (\\ox,\\olm),\n \\end{equation}\nwhere $R$ is the residual function \\eqref{res} and $(\\ox,\\olm)$ is a solution to the KKT system \\eqref{vs}.\n","sha256_utf8":"9181051fe87d434bf9d08b18011cfa15462e7704952599873004d77cc02a0587"}]},{"id":"S019","semantic_type":"proposition","kind":"proposition","title":"分片二次函数二阶性质","source_units":[{"start":20439,"end":21534,"start_line":359,"end_line":380,"raw_tex":"\\begin{Proposition}[second-order variational properties of CPLQ] \\label{sop} Assume that $g:\\R^m\\to \\oR$ is a CPLQ function with representation \\eqref{PWLQ}\nand that $\\oz\\in \\dom g$ and $\\ov\\in \\sub g(\\oz)$. Set $\\ov_i:=\\ov -A_i \\oz -a_i$ for all $i\\in I(\\oz)$. \nThen the following conditions hold:\n \\begin{itemize}[noitemsep,topsep=0pt]\n \\item [{\\rm (a)}]   the critical cone of $g$ at $\\oz$ for $\\ov$ can be calculated by $K_g(\\oz,\\ov)=N_{\\sub g(\\oz)}(\\ov)$. Moreover, we have \n\\begin{equation}\\label{cc2}\n  \\d^2 g(\\oz , \\ov ) (w)=\\d^2 g(\\oz)(w)+\\dd_{K_g(\\oz,\\ov)}(w), \\quad w\\in \\R^m.\n\\end{equation}\n \\item [{\\rm (b)}] the function $g$ is twice epi-differentiable at $\\ox$ for $\\ov$ and its second-order epi-derivative  at this point can be calculated by \n\\begin{equation}\\label{pwfor}\n \\d^2 g(\\oz , \\ov ) (w)  \n=\\begin{cases}\n\\la A_i w, w\\ra&\\mbox{if}\\;\\; w\\in {K}_{C_i}(\\oz,\\ov_i),\\\\\n\\infty&\\mbox{otherwise}.\n\\end{cases}\n\\end{equation}\nMoreover, we have \n\\begin{equation}\\label{domcr}\nK_g(\\oz,\\ov)=\\bigcup_{i\\in I(\\oz)} {K}_{C_i}(\\oz,\\ov_i).\n\\end{equation}\n  \\end{itemize}\n\\end{Proposition}","sha256_utf8":"cd541f1fc2d4d0ab68abe0cc5a1944986ac765a5d2d8f7961c3c83a33df52a68"}]},{"id":"S020","semantic_type":"theorem","kind":"theorem","title":"二阶充分条件导出误差界","source_units":[{"start":24206,"end":24760,"start_line":416,"end_line":423,"raw_tex":"\\begin{Theorem}[error bound via second-order optimality conditions]\\label{sooc} \n Assume that $(\\ox,\\olm)$ is a solution to the KKT system \\eqref{vs}. If the SOSC \\eqref{sosc} holds at $(\\ox,\\olm)$, then\n there are positive constants $\\gg_1$ and $\\kappa_1$ for which  the error bound estimate\n\\begin{equation}\\label{subr3}\n\\|x-\\ox\\|+\\dist\\big(\\lm,\\Lambda(\\ox)\\big)\\le\\kappa_1\\Big(\\dist\\big(-\\nabla_x L(x,\\lm),N_\\Th(x)\\big)+\\| \\Phi(x)-{\\rm{prox}}_g\\big(\\lm+ \\Phi(x)\\big)\\|\\Big)\n\\end{equation}\nholds for any $(x,\\lm)\\in\\B_{\\gg_1}(\\ox,\\olm)$.\n \\end{Theorem}","sha256_utf8":"434fac25ee63d2ded34f673c60922f4b390cdfe4b513c4c575470ee9c5c4cfcc"}]},{"id":"S021","semantic_type":"proposition","kind":"proposition","title":"增广函数的基本性质","source_units":[{"start":26450,"end":26988,"start_line":448,"end_line":455,"raw_tex":"\\begin{Proposition}[properties of the augmented Lagrangian]\\label{paug}\nAssume that  $(x, \\lm, \\rho)\\in \\R^n\\times \\R^m\\times (0, \\infty)$ and $\\ox\\in \\R^n$. Then the following conditions hold for the  augmented Lagrangian   \\eqref{aug}:\n\\begin{enumerate}[noitemsep,topsep=0pt]\n\\item the function $x \\mapsto \\L(x, \\lm, \\rho)$ is $\\mathcal{C}^1$ and fully amenable at $\\ox$;\n\\item the function $\\lm \\mapsto \\L(x, \\lm, \\rho)$ is concave;\n\\item the function $\\rho \\mapsto \\L(x, \\lm, \\rho)$ is nondecreasing.\n\\end{enumerate}\n\\end{Proposition}","sha256_utf8":"e1523de839f3c46d7c1970d9665062a2cc32753fa32a8564771245a64afd18f2"}]},{"id":"S022","semantic_type":"proof","kind":"proof","title":"基本性质完整证明","source_units":[{"start":26991,"end":27919,"start_line":457,"end_line":462,"raw_tex":"\\begin{proof} Let $(x, \\lm, \\rho)\\in \\R^n\\times \\R^m\\times (0, \\infty)$ and $\\ox\\in \\R^n$. To justify (a), it follows from \\cite[Proposition~12.30]{rw} that the envelope function $\\menv$ is CPLQ. \nMoreover, the envelope function $\\menv$  is continuously differentiable on $\\R^m$ and thus the function $x\\mapsto e_{ {1}/{\\rho}} g \\big(\\Phi(x)+\\rho^{-1}\\lm\\big)$\nis $\\mathcal{C}^1$ and fully amenable at $\\ox$. In this case, the basic qualification condition \\eqref{bcq} holds automatically at any $\\ox\\in \\R^n$ since $\\dom \\menv=\\R^m$. Clearly the mapping $x\\mapsto \\ph(x)$\nis fully amenable at $\\ox$ since $\\ph$ is ${\\cal C}^2$. Combining these and remembering that the sum of two ${\\cal C}^1$ and fully amenable functions is fully amenable (cf. \\cite[Exercise~10.22]{rw})\nindicate that the augmented Lagrangian \\eqref{aug} is fully amenable at $\\ox$. Parts (b) and (c) were established in \\cite[Exercise~11.56]{rw}.\n\\end{proof}","sha256_utf8":"0f35b8f474b5644d3d043fd1d793de0ca3f81406524eb7485b67209547c414dc"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"包络光滑性及资格条件","source_units":[{"start":26991,"end":27564,"start_line":457,"end_line":459,"raw_tex":"\\begin{proof} Let $(x, \\lm, \\rho)\\in \\R^n\\times \\R^m\\times (0, \\infty)$ and $\\ox\\in \\R^n$. To justify (a), it follows from \\cite[Proposition~12.30]{rw} that the envelope function $\\menv$ is CPLQ. \nMoreover, the envelope function $\\menv$  is continuously differentiable on $\\R^m$ and thus the function $x\\mapsto e_{ {1}/{\\rho}} g \\big(\\Phi(x)+\\rho^{-1}\\lm\\big)$\nis $\\mathcal{C}^1$ and fully amenable at $\\ox$. In this case, the basic qualification condition \\eqref{bcq} holds automatically at any $\\ox\\in \\R^n$ since $\\dom \\menv=\\R^m$. Clearly the mapping $x\\mapsto \\ph(x)$\n","sha256_utf8":"72b561774fdb35660de3d11365dd17072e8c5ac29df4c705b5bc4004923a0b94"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"合并可处理性与参数性质","source_units":[{"start":27564,"end":27919,"start_line":460,"end_line":462,"raw_tex":"is fully amenable at $\\ox$ since $\\ph$ is ${\\cal C}^2$. Combining these and remembering that the sum of two ${\\cal C}^1$ and fully amenable functions is fully amenable (cf. \\cite[Exercise~10.22]{rw})\nindicate that the augmented Lagrangian \\eqref{aug} is fully amenable at $\\ox$. Parts (b) and (c) were established in \\cite[Exercise~11.56]{rw}.\n\\end{proof}","sha256_utf8":"6229acd16bd5c0206869744c2b5cba391a44216f8215c44a2a9e0ee4095bfb2f"}]},{"id":"S025","semantic_type":"proposition","kind":"proposition","title":"KKT驻点向子问题传播","source_units":[{"start":28057,"end":28563,"start_line":467,"end_line":476,"raw_tex":"\\begin{Proposition}[propagation of stationary conditions]\\label{fopag}\nLet $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs}. Then for any $\\rho>0$,  the following conditions are satisfied:\n\\begin{enumerate}[noitemsep,topsep=0pt]\n\\item we always have  $\\L(\\ox,\\olm,\\rho)=\\ph(\\ox)+g\\big(\\Phi(\\ox)\\big)$;\n\\item $\\ox$ is a stationary point of the augmented  problem \n\\begin{equation}\\label{ap}\n\\mini\\; \\L(x,\\olm,\\rho)\\quad \\mbox{subject to}\\;\\; x\\in \\Th.\n\\end{equation}\n\\end{enumerate}\n\\end{Proposition}","sha256_utf8":"17069c83e069b5a7bb15c07051c6fd6c6a5952731d3b254f904c63639a6f51ee"}]},{"id":"S026","semantic_type":"proof","kind":"proof","title":"驻点传播完整证明","source_units":[{"start":28564,"end":29295,"start_line":477,"end_line":487,"raw_tex":"\\begin{proof} Given $\\rho>0$, it follows from the first equality in \\eqref{prm} and  $\\olm\\in \\sub g\\big(\\Phi(\\ox)\\big)$ that \n $$\n \\prox\\big(\\Phi(\\ox)+\\rho^{-1}\\olm\\big)=\\Phi(\\ox),\n $$ which clearly proves (a).\nWe proceed now with the proof of  (b). Since $g$ is convex, we conclude from the second equality in \\eqref{prm} that the condition $\\olm\\in \\sub g\\big(\\Phi(\\ox)\\big)$ is equivalent to $\\nabla(e_{ {1}/{\\rho}} g)\\big(\\Phi(\\ox)+\\rho^{-1}\\olm\\big)=\\olm$\nfor any $\\rho>0$.  Thus we get \n\\begin{equation}\\label{kkt2}\n\\nabla_x \\L(\\ox,\\olm,\\rho)=\\nabla \\ph(\\ox)+ \\nabla \\Phi(\\ox)^*\\nabla(e_{ {1}/{\\rho}} g)(\\Phi(\\ox)+\\rho^{-1}\\olm)=\\nabla_x L(\\ox,\\olm).\n\\end{equation}\nCombining these and \\eqref{vs} justifies  (b).\n\\end{proof}","sha256_utf8":"4197cd112458f282a7380fcb9448379374994f4a55472a42550b795652842ddd"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"近端固定点给出函数值","source_units":[{"start":28564,"end":28776,"start_line":477,"end_line":480,"raw_tex":"\\begin{proof} Given $\\rho>0$, it follows from the first equality in \\eqref{prm} and  $\\olm\\in \\sub g\\big(\\Phi(\\ox)\\big)$ that \n $$\n \\prox\\big(\\Phi(\\ox)+\\rho^{-1}\\olm\\big)=\\Phi(\\ox),\n $$ which clearly proves (a).\n","sha256_utf8":"9b41215b7e44f7f24e7dcad5c2255bb56314648eae4832e2c1aba17dfe8c68d4"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"梯度恒等式传递驻点性","source_units":[{"start":28776,"end":29295,"start_line":481,"end_line":487,"raw_tex":"We proceed now with the proof of  (b). Since $g$ is convex, we conclude from the second equality in \\eqref{prm} that the condition $\\olm\\in \\sub g\\big(\\Phi(\\ox)\\big)$ is equivalent to $\\nabla(e_{ {1}/{\\rho}} g)\\big(\\Phi(\\ox)+\\rho^{-1}\\olm\\big)=\\olm$\nfor any $\\rho>0$.  Thus we get \n\\begin{equation}\\label{kkt2}\n\\nabla_x \\L(\\ox,\\olm,\\rho)=\\nabla \\ph(\\ox)+ \\nabla \\Phi(\\ox)^*\\nabla(e_{ {1}/{\\rho}} g)(\\Phi(\\ox)+\\rho^{-1}\\olm)=\\nabla_x L(\\ox,\\olm).\n\\end{equation}\nCombining these and \\eqref{vs} justifies  (b).\n\\end{proof}","sha256_utf8":"f17dced2ec147579d935113cf65e360c4b4ae9e2977a538902620ce852ea7270"}]},{"id":"S029","semantic_type":"proposition","kind":"proposition","title":"二阶正性与二次增长等价","source_units":[{"start":29679,"end":30576,"start_line":493,"end_line":507,"raw_tex":"\\begin{Proposition}[quadratic growth condition for augmented Lagrangians]\\label{2ndgr}\nLet $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and set $\\ov:= \\nabla_x L(\\ox,\\olm)$. Then for any $\\rho>0$,  the following conditions are equivalent: \n\\begin{enumerate}[noitemsep,topsep=0pt]\n\\item    there exists a constant $\\bar\\rho>0$ such that for any $\\rho\\ge\\bar\\rho$ the second-order condition \n\\begin{equation}\\label{eq26}\n\\d^2_x\\L\\big((\\ox,\\olm,\\rho),\\ov\\big)(w)>0\\;\\mbox{ for all }\\;w\\in K_\\Th (\\ox,-\\ov ) \\setminus\\{0\\}\n\\end{equation}\nholds;\n \\item  there exist positive constants $\\bar\\rho$, $\\gamma$, and $\\ell$ such that for any $\\rho\\ge\\bar \\rho$, the quadratic growth condition \n \\begin{equation}\\label{eq16}\n\\L(x,\\olm,\\rho)\\ge \\ph(\\ox)+g\\big(\\Phi(\\ox)\\big)+\\ell\\| x-\\bar x\\|^2\\quad \\mbox{ for all }\\;x\\in\\B_\\gg (\\ox)\\cap \\Th\n\\end{equation}\nholds. \n\\end{enumerate}\n\\end{Proposition}","sha256_utf8":"4008cc810b81a47bd139f210daec161bf084d60348fbdfdbc703706eab435604"}]},{"id":"S030","semantic_type":"proof","kind":"proof","title":"增长判据完整证明","source_units":[{"start":30579,"end":32650,"start_line":509,"end_line":533,"raw_tex":"\\begin{proof} Pick $\\rho>0$ and    $x\\in \\R^n$, and \n define the function $  f_\\rho(x) := \\L(x,\\olm, \\rho)+\\delta_\\Th(x)$.  Remember that $(\\ox,\\olm)$ is a solution to the KKT system  \\eqref{vs}.\nSince $\\partial   f_\\rho(\\ox) = \\nabla_x \\L(\\ox,\\olm,\\rho)+N_\\Th(\\ox)$, it follows from \\eqref{kkt2} that $0\\in \\sub f_\\rho(\\ox)$. \nEmploying \\cite[Proposition~13.19]{rw} and Proposition~\\ref{paug}(a), we conclude  that  \n\\begin{eqnarray}\n\\d^2  f_\\rho (\\ox, 0)(w) &=&\\d^2_x\\L\\big((\\ox,\\olm,\\rho),\\ov\\big)(w)+\\d^2\\delta_\\Th(\\ox, -\\ov)(w)\\nonumber \\\\\n&=&\\d^2_x\\L\\big((\\ox,\\olm,\\rho),\\ov\\big)(w)+\\delta_{K_\\Th(\\ox, -\\ov)}(w),\\label{sumr}\n\\end{eqnarray} \nwhere the formula for the second subderivative of $\\dd_\\Th$ was taken from \\cite[Example~3.4]{mmstams}.\n\nAfter this preparation,  assume that   (a)  holds. So we find   $\\bar \\rho>0$ for which \\eqref{eq26} is satisfied. \nThis, combined with \\eqref{sumr}, tells us that $\\d^2  f_{\\bar\\rho}(\\ox, 0)(w)>0$  for all $w\\in \\R^n\\setminus\\{0\\}$.\nEmploying now   \\cite[Theorem~13.24(c)]{rw}, we find positive constants  $\\gg$ and $\\ell$ for which the quadratic \ngrowth condition \n$$\n\\L(x,\\olm,\\bar \\rho)\\ge \\L(\\ox,\\olm,\\bar \\rho) +\\ell\\| x-\\bar x\\|^2 =\\ph(\\ox)+g\\big(\\Phi(\\ox)\\big)+\\ell\\| x-\\bar x\\|^2\n$$\nholds for all $x\\in\\B_\\gg (\\ox)\\cap \\Th$, where the last equality comes from Proposition~\\ref{fopag}(a). By Proposition~\\ref{paug}(c), the mapping $\\rho\\mapsto \\L(x,\\olm,  \\rho)$ is nondecereasing. Combining this with the above quadratic growth condition justifies \\eqref{eq16} for any $\\rho\\ge \\bar\\rho$.\n\nTurning now to the opposite implication, assume that (b) holds and then pick $\\rho\\in [\\bar\\rho, \\infty)$. By  Proposition~\\ref{fopag}(a), we have $f_\\rho(\\ox)= \\ph(\\ox)+g\\big(\\Phi(\\ox)\\big)$, which yields \n$$\nf_\\rho(x) \\ge f_\\rho(\\ox)+\\ell\\| x-\\bar x\\|^2\\quad \\mbox{ for all }\\;x\\in\\B_\\gg (\\ox).\n$$\nThis implies via the definition of the second subderivative that $\\d^2  f_{\\rho}(\\ox, 0)(w)>0$  for all $w\\in \\R^n\\setminus\\{0\\}$. Appealing now to \\eqref{sumr} verifies \\eqref{eq26} and hence ends the proof.\n\\end{proof}","sha256_utf8":"4fed42d2728b346b0dc4425c8e11e7065a8f2fa5269392c0217b2005b3bd400c"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"含指标目标的二阶分解","source_units":[{"start":30579,"end":31330,"start_line":509,"end_line":517,"raw_tex":"\\begin{proof} Pick $\\rho>0$ and    $x\\in \\R^n$, and \n define the function $  f_\\rho(x) := \\L(x,\\olm, \\rho)+\\delta_\\Th(x)$.  Remember that $(\\ox,\\olm)$ is a solution to the KKT system  \\eqref{vs}.\nSince $\\partial   f_\\rho(\\ox) = \\nabla_x \\L(\\ox,\\olm,\\rho)+N_\\Th(\\ox)$, it follows from \\eqref{kkt2} that $0\\in \\sub f_\\rho(\\ox)$. \nEmploying \\cite[Proposition~13.19]{rw} and Proposition~\\ref{paug}(a), we conclude  that  \n\\begin{eqnarray}\n\\d^2  f_\\rho (\\ox, 0)(w) &=&\\d^2_x\\L\\big((\\ox,\\olm,\\rho),\\ov\\big)(w)+\\d^2\\delta_\\Th(\\ox, -\\ov)(w)\\nonumber \\\\\n&=&\\d^2_x\\L\\big((\\ox,\\olm,\\rho),\\ov\\big)(w)+\\delta_{K_\\Th(\\ox, -\\ov)}(w),\\label{sumr}\n\\end{eqnarray} \nwhere the formula for the second subderivative of $\\dd_\\Th$ was taken from \\cite[Example~3.4]{mmstams}.\n","sha256_utf8":"ef9d6fdddac12ab79e254f4d1cb0b9994ebc949d130c420d53c6615a409e9342"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"二阶正性推出统一增长","source_units":[{"start":31331,"end":32129,"start_line":519,"end_line":526,"raw_tex":"After this preparation,  assume that   (a)  holds. So we find   $\\bar \\rho>0$ for which \\eqref{eq26} is satisfied. \nThis, combined with \\eqref{sumr}, tells us that $\\d^2  f_{\\bar\\rho}(\\ox, 0)(w)>0$  for all $w\\in \\R^n\\setminus\\{0\\}$.\nEmploying now   \\cite[Theorem~13.24(c)]{rw}, we find positive constants  $\\gg$ and $\\ell$ for which the quadratic \ngrowth condition \n$$\n\\L(x,\\olm,\\bar \\rho)\\ge \\L(\\ox,\\olm,\\bar \\rho) +\\ell\\| x-\\bar x\\|^2 =\\ph(\\ox)+g\\big(\\Phi(\\ox)\\big)+\\ell\\| x-\\bar x\\|^2\n$$\nholds for all $x\\in\\B_\\gg (\\ox)\\cap \\Th$, where the last equality comes from Proposition~\\ref{fopag}(a). By Proposition~\\ref{paug}(c), the mapping $\\rho\\mapsto \\L(x,\\olm,  \\rho)$ is nondecereasing. Combining this with the above quadratic growth condition justifies \\eqref{eq16} for any $\\rho\\ge \\bar\\rho$.\n","sha256_utf8":"a2fa8685ce39e018018382f8546b5bbacf485b4ddc36b8dbd21568e87b4772f5"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"二次增长反推二阶正性","source_units":[{"start":32130,"end":32650,"start_line":528,"end_line":533,"raw_tex":"Turning now to the opposite implication, assume that (b) holds and then pick $\\rho\\in [\\bar\\rho, \\infty)$. By  Proposition~\\ref{fopag}(a), we have $f_\\rho(\\ox)= \\ph(\\ox)+g\\big(\\Phi(\\ox)\\big)$, which yields \n$$\nf_\\rho(x) \\ge f_\\rho(\\ox)+\\ell\\| x-\\bar x\\|^2\\quad \\mbox{ for all }\\;x\\in\\B_\\gg (\\ox).\n$$\nThis implies via the definition of the second subderivative that $\\d^2  f_{\\rho}(\\ox, 0)(w)>0$  for all $w\\in \\R^n\\setminus\\{0\\}$. Appealing now to \\eqref{sumr} verifies \\eqref{eq26} and hence ends the proof.\n\\end{proof}","sha256_utf8":"206af8084ac967015a4baf202f0f29b904d7f7ec9c8c82c71b8bdd54982fefa4"}]},{"id":"S034","semantic_type":"theorem","kind":"theorem","title":"增广函数二阶导数公式","source_units":[{"start":33721,"end":34460,"start_line":544,"end_line":552,"raw_tex":"\\begin{Theorem}[second subderivatives of augmented Lagrangians]\\label{ssc} \nAssume that  $(x, \\lm, \\rho)\\in \\R^n\\times \\R^m\\times (0, \\infty)$ and $\\ox\\in \\R^n$. \nThen  the function $x\\mapsto\\L(x,\\lm,\\rho)$, defined via the augmented Lagrangian \\eqref{aug}, is twice epi-differentiable at $\\ox$ for $\\ov:=\\nabla_x \\L(\\ox,\\lm,\\rho)$ and its second-order epi-derivative \ncan be calculated for any $w\\in \\R^n$ by\n\\begin{equation}\\label{eq40}\n\\d^2_x\\L\\big((\\ox,\\lm,\\rho),\\ov\\big)(w)=\\big\\langle \\nabla^2_{xx}L(\\ox,\\mu)w,w\\big\\rangle+ e_{1/{2\\rho}}\\big(\\d^2 g\\big(\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu),\\mu\\big)\\big) \\big(\\nabla \\Phi(\\ox)w \\big), \n\\end{equation}\nwhere $\\mu=\\nabla_x \\big(e_{1/{\\rho}}g\\big)\\big(\\Phi(\\ox)+\\rho^{-1}\\lm\\big)$.\n\\end{Theorem}","sha256_utf8":"6bc46b104c9de17e743c5d2e238f5e5c5958b6caea88c8016efbb486891b8a79"}]},{"id":"S035","semantic_type":"proof","kind":"proof","title":"二阶公式完整证明","source_units":[{"start":34461,"end":36985,"start_line":553,"end_line":578,"raw_tex":"\\begin{proof}  Given $\\rho>0$,  it follows from  the second equality in \\eqref{prm} that $\\mu\\in \\sub g\\big(\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu)\\big)$. \nAccording to Proposition~\\ref{sop}(b), the CPLQ function $g$ is twice epi-differentiable at $\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu)$ for $\\mu$. Appealing now \nto  \\cite[Proposition~4.1]{hms20} tells us that    the   envelope function $e_{1/\\rho} g$ is twice epi-differentiable at $\\Phi(\\ox)+\\rho^{-1}\\lm$ for $\\mu$ and that \n\\begin{equation}\\label{more}\n\\d^2 (e_{1/\\rho} g )\\big(\\Phi(\\ox)+\\rho^{-1}\\lm,\\mu\\big)=e_{1/2\\rho}\\big(\\d^2 g(\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu),\\mu)\\big).\n\\end{equation}\nBy \\eqref{pwfor}, the second-order epi-derivative of $g$ is CPLQ. This, combined with \\eqref{more}, indicates that $ e_{1/2\\rho}\\big(\\d^2 g(\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu),\\mu)\\big)$\nis finite on $\\R^m$. Employing now \\cite[Theorem~4.3]{r20}, we conclude that  $e_{1/\\rho}g$ is twice semidifferentiable at $\\Phi(\\ox)+\\rho^{-1}\\lm$ and its second semiderivative \ncoincides with its second-order epi-derivative, namely \n\\begin{equation}\\label{ssed}\n\\d^2\\big(e_{1/\\rho}g\\big)\\big(\\Phi(\\ox)+\\rho^{-1}\\lm\\big)=\\d^2 (e_{1/\\rho} g )\\big(\\Phi(\\ox)+\\rho^{-1}\\lm,\\mu\\big).\n\\end{equation}\nThis, together with the chain and sum rules for twice semidifferentiability, established in  \\cite[Propositions~8.2(i) and 8.1]{mmstams}, \nshows that the mapping $x\\mapsto\\L(x,\\lm,\\rho)$ is twice semidifferentiable at $\\ox$. Thus, by   \\cite[Propositions~8.2(iii)]{mmstams}, the latter mapping is twice epi-differentiable at $\\ox$ for $\\ov$ and \n\\begin{equation}\\label{lss}\n\\d_x^2{\\L}(\\ox,\\lm,\\rho)(w)=\\d^2_x\\L\\big((\\ox,\\lm,\\rho),\\ov\\big)(w), \\quad w\\in \\R^n.\n\\end{equation}\nBy  \\cite[Propositions~8.2(i) and 8.1]{mmstams},  the second  semiderivative of $x\\mapsto\\L(x,\\lm,\\rho)$ at $\\ox$ can be calculated for any $w\\in \\R^n$ by\n\\begin{eqnarray*}\n\\d_x^2{\\L}(\\ox,\\lm,\\rho)(w)&=&\\d^2\\ph(\\ox)(w)+\\big\\la\\mu,\\nabla^2\\Phi(\\ox)(w,w)\\big\\ra+\\d^2\\big(e_{1/\\rho} g\\big)\\big(\\Phi(\\ox)+\\rho^{-1}\\lm\\big)\\big(\\nabla \\Phi(\\ox)w\\big)\\\\\n&=&\\la\\nabla^2\\ph(\\ox)w,w\\ra+\\big\\la\\mu,\\nabla^2\\Phi(\\ox)(w,w)\\big\\ra+\\d^2\\big(e_{1/\\rho} g\\big)\\big(\\Phi(\\ox)+\\rho^{-1}\\lm,\\mu\\big)\\big(\\nabla \\Phi(\\ox)w\\big)\\\\\n&=&\\big\\la\\nabla_{xx}^2L(\\ox,\\mu)w,w\\big\\ra+e_{1/2\\rho}\\big(\\d^2 g(\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu),\\mu)\\big)\\big(\\nabla \\Phi(\\ox)w\\big),\n\\end{eqnarray*}\nwhere the second equality comes from \\eqref{ssed} and the last one results from \\eqref{more}. This together with \\eqref{lss}\njustifies \\eqref{eq40} and hence completes the proof.\n\\end{proof}","sha256_utf8":"ba1e1a1aa8bfab836fede0462fca1954d1c803456d95d3d890e4177545055516"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"包络二阶上图导数","source_units":[{"start":34461,"end":35085,"start_line":553,"end_line":558,"raw_tex":"\\begin{proof}  Given $\\rho>0$,  it follows from  the second equality in \\eqref{prm} that $\\mu\\in \\sub g\\big(\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu)\\big)$. \nAccording to Proposition~\\ref{sop}(b), the CPLQ function $g$ is twice epi-differentiable at $\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu)$ for $\\mu$. Appealing now \nto  \\cite[Proposition~4.1]{hms20} tells us that    the   envelope function $e_{1/\\rho} g$ is twice epi-differentiable at $\\Phi(\\ox)+\\rho^{-1}\\lm$ for $\\mu$ and that \n\\begin{equation}\\label{more}\n\\d^2 (e_{1/\\rho} g )\\big(\\Phi(\\ox)+\\rho^{-1}\\lm,\\mu\\big)=e_{1/2\\rho}\\big(\\d^2 g(\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu),\\mu)\\big).\n\\end{equation}\n","sha256_utf8":"9958a5f2df450b4a0e575bf151d7b29113f0e795b4a8d7a4482c5d488cab3c7b"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"有限二阶导数导出半可微","source_units":[{"start":35085,"end":35660,"start_line":559,"end_line":564,"raw_tex":"By \\eqref{pwfor}, the second-order epi-derivative of $g$ is CPLQ. This, combined with \\eqref{more}, indicates that $ e_{1/2\\rho}\\big(\\d^2 g(\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu),\\mu)\\big)$\nis finite on $\\R^m$. Employing now \\cite[Theorem~4.3]{r20}, we conclude that  $e_{1/\\rho}g$ is twice semidifferentiable at $\\Phi(\\ox)+\\rho^{-1}\\lm$ and its second semiderivative \ncoincides with its second-order epi-derivative, namely \n\\begin{equation}\\label{ssed}\n\\d^2\\big(e_{1/\\rho}g\\big)\\big(\\Phi(\\ox)+\\rho^{-1}\\lm\\big)=\\d^2 (e_{1/\\rho} g )\\big(\\Phi(\\ox)+\\rho^{-1}\\lm,\\mu\\big).\n\\end{equation}\n","sha256_utf8":"16a0fdc094fc0e3fc19c38ad7204b032b18d8e33714ce9469fddb431dae25d50"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"复合与求和保持二次可微","source_units":[{"start":35660,"end":36135,"start_line":565,"end_line":569,"raw_tex":"This, together with the chain and sum rules for twice semidifferentiability, established in  \\cite[Propositions~8.2(i) and 8.1]{mmstams}, \nshows that the mapping $x\\mapsto\\L(x,\\lm,\\rho)$ is twice semidifferentiable at $\\ox$. Thus, by   \\cite[Propositions~8.2(iii)]{mmstams}, the latter mapping is twice epi-differentiable at $\\ox$ for $\\ov$ and \n\\begin{equation}\\label{lss}\n\\d_x^2{\\L}(\\ox,\\lm,\\rho)(w)=\\d^2_x\\L\\big((\\ox,\\lm,\\rho),\\ov\\big)(w), \\quad w\\in \\R^n.\n\\end{equation}\n","sha256_utf8":"39b0c00a5f5d7249d7c9e4fc358a69eefa070640fcf2db01d06a3166d5c71cee"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"展开并识别二阶公式","source_units":[{"start":36135,"end":36985,"start_line":570,"end_line":578,"raw_tex":"By  \\cite[Propositions~8.2(i) and 8.1]{mmstams},  the second  semiderivative of $x\\mapsto\\L(x,\\lm,\\rho)$ at $\\ox$ can be calculated for any $w\\in \\R^n$ by\n\\begin{eqnarray*}\n\\d_x^2{\\L}(\\ox,\\lm,\\rho)(w)&=&\\d^2\\ph(\\ox)(w)+\\big\\la\\mu,\\nabla^2\\Phi(\\ox)(w,w)\\big\\ra+\\d^2\\big(e_{1/\\rho} g\\big)\\big(\\Phi(\\ox)+\\rho^{-1}\\lm\\big)\\big(\\nabla \\Phi(\\ox)w\\big)\\\\\n&=&\\la\\nabla^2\\ph(\\ox)w,w\\ra+\\big\\la\\mu,\\nabla^2\\Phi(\\ox)(w,w)\\big\\ra+\\d^2\\big(e_{1/\\rho} g\\big)\\big(\\Phi(\\ox)+\\rho^{-1}\\lm,\\mu\\big)\\big(\\nabla \\Phi(\\ox)w\\big)\\\\\n&=&\\big\\la\\nabla_{xx}^2L(\\ox,\\mu)w,w\\big\\ra+e_{1/2\\rho}\\big(\\d^2 g(\\Phi(\\ox)+\\rho^{-1}(\\lm-\\mu),\\mu)\\big)\\big(\\nabla \\Phi(\\ox)w\\big),\n\\end{eqnarray*}\nwhere the second equality comes from \\eqref{ssed} and the last one results from \\eqref{more}. This together with \\eqref{lss}\njustifies \\eqref{eq40} and hence completes the proof.\n\\end{proof}","sha256_utf8":"23562b46f07fef09c0cf20f16fdcbb8cbc41cfbba0135cb34b3156bd1f001054"}]},{"id":"S040","semantic_type":"corollary","kind":"corollary","title":"KKT点处二阶公式","source_units":[{"start":37207,"end":37828,"start_line":582,"end_line":588,"raw_tex":"\\begin{Corollary}[second-order epi-derivatives] Let $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and set $\\ov:= \\nabla_x L(\\ox,\\olm)$. Then for any $\\rho>0$,\nthe function $x\\mapsto\\L(x,\\lm,\\rho)$, defined via the augmented Lagrangian \\eqref{aug}, is twice epi-differentiable at $\\ox$ for $\\ov $ and its second-order epi-derivative \ncan be calculated for any $w\\in \\R^n$ by\n\\begin{equation}\\label{eq402}\n\\d^2_x\\L\\big((\\ox,\\olm,\\rho),\\ov\\big)(w)=\\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w,w\\big\\rangle+ e_{1/{2\\rho}}\\big(\\d^2 g\\big(\\Phi(\\ox),\\olm\\big)\\big) \\big(\\nabla \\Phi(\\ox)w \\big).\n\\end{equation}\n\\end{Corollary}","sha256_utf8":"5490fbb7ea56ef512ffdfa8ad43d02160b6454c87019d3938bd46418d1b90139"}]},{"id":"S041","semantic_type":"proof","kind":"proof","title":"KKT二阶公式完整证明","source_units":[{"start":37829,"end":38439,"start_line":589,"end_line":595,"raw_tex":"\\begin{proof} Because  $(\\ox,\\olm)$ is a solution to the KKT system \\eqref{vs}, we get $\\olm\\in \\sub g\\big(\\Phi(\\ox)\\big)$.\nSince $g$ is CPLQ, we conclude from the second equality in \\eqref{prm} that the latter inclusion is equivalent to $\\nabla(e_{ {1}/{\\rho}} g)\\big(\\Phi(\\ox)+\\rho^{-1}\\olm\\big)=\\olm$\nfor any $\\rho>0$. This means that $\\mu=\\olm$, where $\\mu$ was taken from Theorem~\\ref{ssc}. Moreover, it follows from \n\\eqref{kkt2} that $\\nabla_x \\L(\\ox,\\olm,\\rho)=\\nabla_x L(\\ox,\\olm)=\\ov$.\nCombining these and the second-order epi-derivative formula \\eqref{eq40}, \nwe arrive at \\eqref{eq402}.\n\\end{proof}","sha256_utf8":"7457ecd416ad05f4c48e96b444fb431ae3cba3d8e39c65d86737b154dc75d796"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"识别包络乘子并代入公式","source_units":[{"start":37829,"end":38439,"start_line":589,"end_line":595,"raw_tex":"\\begin{proof} Because  $(\\ox,\\olm)$ is a solution to the KKT system \\eqref{vs}, we get $\\olm\\in \\sub g\\big(\\Phi(\\ox)\\big)$.\nSince $g$ is CPLQ, we conclude from the second equality in \\eqref{prm} that the latter inclusion is equivalent to $\\nabla(e_{ {1}/{\\rho}} g)\\big(\\Phi(\\ox)+\\rho^{-1}\\olm\\big)=\\olm$\nfor any $\\rho>0$. This means that $\\mu=\\olm$, where $\\mu$ was taken from Theorem~\\ref{ssc}. Moreover, it follows from \n\\eqref{kkt2} that $\\nabla_x \\L(\\ox,\\olm,\\rho)=\\nabla_x L(\\ox,\\olm)=\\ov$.\nCombining these and the second-order epi-derivative formula \\eqref{eq40}, \nwe arrive at \\eqref{eq402}.\n\\end{proof}","sha256_utf8":"7457ecd416ad05f4c48e96b444fb431ae3cba3d8e39c65d86737b154dc75d796"}]},{"id":"S043","semantic_type":"theorem","kind":"theorem","title":"充分条件与增长等价","source_units":[{"start":39169,"end":39810,"start_line":611,"end_line":619,"raw_tex":"\\begin{Theorem}[characterizations of quadratic growth condition]\\label{growth}\nLet $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and set $\\ov:= \\nabla_x L(\\ox,\\olm)$. Then the following conditions are equivalent: \n \\begin{enumerate}[noitemsep,topsep=0pt]\n \\item  the SOSC \\eqref{sosc} holds at $(\\ox,\\olm)$;\n \\item   there exists a constant $\\bar\\rho>0$ such that for any $\\rho\\ge\\bar\\rho$ the second-order condition \\eqref{eq26} is satisfied;\n \\item there exist positive constants $\\bar\\rho$, $\\gg$, and $\\ell$ such that for any $\\rho\\ge\\bar \\rho$, the quadratic growth condition \\eqref{eq16} \nholds.\n\\end{enumerate}\n\\end{Theorem}","sha256_utf8":"93d7697ab5a7c69b2bc405d4ec6f00f602ac5a2de6711ff371be7027eb5c1c6d"}]},{"id":"S044","semantic_type":"proof","kind":"proof","title":"增长等价定理完整证明","source_units":[{"start":39811,"end":42733,"start_line":620,"end_line":661,"raw_tex":"\\begin{proof} The equivalence between (b) and (c) was already established in Proposition~\\ref{2ndgr}.  Suppose that (b) is satisfied. Since we always have \n$$\ne_{1/{2\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\olm)\\big)(\\nabla \\Phi(\\ox)w)\\le \\d^2 g(\\Phi(\\ox),\\olm)(\\nabla \\Phi(\\ox)w)\\quad \\mbox{for all}\\;\\; \\rho>0, \n$$\nthe validity of the second-order condition \\eqref{eq402} yields the SOSC \\eqref{sosc} and so proves (a). \n\nAssume now that (a) holds. Since the second subderivative is lower semicontinuous and positive homogeneous of degree $2$ (cf. \\cite[Proposition~13.5]{rw}), the SOSC \\eqref{sosc} amounts to the existence of \na positive constant $\\ell$ so that we have \n\\begin{equation}\\label{som}\n\\langle\\nabla_{xx}^2L(\\bar x,\\olm)w,w\\rangle+\\d^2g (\\Phi(\\bar x),\\olm) (\\nabla \\Phi(\\ox)w )\\ge \\ell \\quad \\mbox{for all }\\; w\\in \\D\\cap \\S,\n\\end{equation}\nwhere $\\S$ stands for the unit sphere in $\\R^n$. Let $w\\in \\S$ and denote the left-hand side of \\eqref{som} by $h(w)$.\n  Given $\\rho>0$ and $w\\in \\R^n$, define the function $h_\\rho:\\R^n\\to \\R$ by \n$$\nh_\\rho(w)=\\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w,w\\big\\rangle+ e_{1/{2\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\olm)\\big)(\\nabla \\Phi(\\ox)w).\n$$\nWe understand from \\cite[Theorem~1.25]{rw} that \n$$\nh_\\rho(w) \\nearrow h(w)\\;\\; as \\;\\; \\rho \\nearrow \\infty\n$$\n for all $w\\in\\R^n$. Employing now \\cite[Proposition~7.4]{rw} tells us that  $h_\\rho\\overset {\\ss \\mbox{e}}{\\to} h$ as $\\rho \\nearrow \\infty$.\n This, combined with \\cite[Theorem~7.46]{rw}, implies that $h_\\rho+\\dd_{\\S\\cap K_\\Th(\\ox,-\\ov)}\\overset {\\ss \\mbox{e}}{\\to} h+ \\dd_{\\S\\cap K_\\Th(\\ox,-\\ov)}$ as  $\\rho \\nearrow \\infty$.\n Appealing now to \\cite[Theorem~7.33]{rw} indicates that \n\\begin{equation}\\label{infco}\n \\inf \\big\\{  h_\\rho(w)|\\; w\\in \\S\\cap K_\\Th(\\ox,-\\ov)\\big\\} \\to \\inf\\big\\{ h(w)|\\; w\\in \\S\\cap K_\\Th(\\ox,-\\ov)\\big\\}\\;\\;\\mbox{as}\\;\\;\\rho \\nearrow \\infty.\n\\end{equation}\nIt follows from \\eqref{pwfor} and \\eqref{domcr} that  for any $w\\in \\R^n$ with $\\nabla \\Phi(\\ox)w \\notin K_g(\\Phi(\\ox),\\olm)$, we have  $\\d^2g (\\Phi(\\bar x),\\olm) (\\nabla \\Phi(\\ox)w )=\\infty$.\nThis,  \\eqref{som}, and the definition of the convex cone $\\D$ from \\eqref{coned}  bring us to \n$$\n\\inf\\big\\{ h(w)|\\; w\\in \\S\\cap K_\\Th(\\ox,-\\ov)\\big\\}\\ge \\ell.\n$$\nCombining this and \\eqref{infco}, we can find a positive constant $ \\bar \\rho$ for which we have \n\\begin{equation}\\label{rb1}\nh_\\rho(w)>\\frac{\\ell}{2}\\quad \\mbox{for all}\\;\\;\\rho\\ge  \\bar \\rho\\;\\;\\mbox{and all}\\;\\;w\\in    \\S\\cap K_\\Th(\\ox,-\\ov).\n\\end{equation}\nGiven $\\rho\\ge \\bar\\rho$, it follows from \\eqref{eq402} and the definition of $h_\\rho$  that \n$$\n\\d^2_x\\L\\big((\\ox,\\olm,\\rho),\\ov\\big)(w) >0\\quad \\mbox{for all} \\;\\; w\\in \\S\\cap K_\\Th(\\ox,-\\ov).\n$$\nRemember that the second-order epi-derivative  is positive homogeneous of degree $2$. So we arrive at the second-order condition \\eqref{eq26} for any $\\rho\\ge \\bar\\rho $. \nThis proves (b) and hence completes the proof.\n\\end{proof}","sha256_utf8":"70af987513278af7fa8141f0f37c3d43d9d149b62640b35f59de45e1bd333adf"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"增长判据及必要方向","source_units":[{"start":39811,"end":40222,"start_line":620,"end_line":624,"raw_tex":"\\begin{proof} The equivalence between (b) and (c) was already established in Proposition~\\ref{2ndgr}.  Suppose that (b) is satisfied. Since we always have \n$$\ne_{1/{2\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\olm)\\big)(\\nabla \\Phi(\\ox)w)\\le \\d^2 g(\\Phi(\\ox),\\olm)(\\nabla \\Phi(\\ox)w)\\quad \\mbox{for all}\\;\\; \\rho>0, \n$$\nthe validity of the second-order condition \\eqref{eq402} yields the SOSC \\eqref{sosc} and so proves (a). \n","sha256_utf8":"7da69664a0c1b1671be9846b259eacd559eccb1631d53c688b03a22aef42bc37"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"球面上取得正下界","source_units":[{"start":40223,"end":40776,"start_line":626,"end_line":631,"raw_tex":"Assume now that (a) holds. Since the second subderivative is lower semicontinuous and positive homogeneous of degree $2$ (cf. \\cite[Proposition~13.5]{rw}), the SOSC \\eqref{sosc} amounts to the existence of \na positive constant $\\ell$ so that we have \n\\begin{equation}\\label{som}\n\\langle\\nabla_{xx}^2L(\\bar x,\\olm)w,w\\rangle+\\d^2g (\\Phi(\\bar x),\\olm) (\\nabla \\Phi(\\ox)w )\\ge \\ell \\quad \\mbox{for all }\\; w\\in \\D\\cap \\S,\n\\end{equation}\nwhere $\\S$ stands for the unit sphere in $\\R^n$. Let $w\\in \\S$ and denote the left-hand side of \\eqref{som} by $h(w)$.\n","sha256_utf8":"e07defd71f893361856da7005d70eba1490f08a2dab9cd8292880b834a6dc699"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"包络逼近与最小值收敛","source_units":[{"start":40776,"end":41687,"start_line":632,"end_line":645,"raw_tex":"  Given $\\rho>0$ and $w\\in \\R^n$, define the function $h_\\rho:\\R^n\\to \\R$ by \n$$\nh_\\rho(w)=\\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w,w\\big\\rangle+ e_{1/{2\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\olm)\\big)(\\nabla \\Phi(\\ox)w).\n$$\nWe understand from \\cite[Theorem~1.25]{rw} that \n$$\nh_\\rho(w) \\nearrow h(w)\\;\\; as \\;\\; \\rho \\nearrow \\infty\n$$\n for all $w\\in\\R^n$. Employing now \\cite[Proposition~7.4]{rw} tells us that  $h_\\rho\\overset {\\ss \\mbox{e}}{\\to} h$ as $\\rho \\nearrow \\infty$.\n This, combined with \\cite[Theorem~7.46]{rw}, implies that $h_\\rho+\\dd_{\\S\\cap K_\\Th(\\ox,-\\ov)}\\overset {\\ss \\mbox{e}}{\\to} h+ \\dd_{\\S\\cap K_\\Th(\\ox,-\\ov)}$ as  $\\rho \\nearrow \\infty$.\n Appealing now to \\cite[Theorem~7.33]{rw} indicates that \n\\begin{equation}\\label{infco}\n \\inf \\big\\{  h_\\rho(w)|\\; w\\in \\S\\cap K_\\Th(\\ox,-\\ov)\\big\\} \\to \\inf\\big\\{ h(w)|\\; w\\in \\S\\cap K_\\Th(\\ox,-\\ov)\\big\\}\\;\\;\\mbox{as}\\;\\;\\rho \\nearrow \\infty.\n\\end{equation}\n","sha256_utf8":"3f3039229459a7947e39c0d07e7dd4209fa54e0e1abfd86ebd4afbd8b89b7b11"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"极限正性传给大罚参数","source_units":[{"start":41687,"end":42305,"start_line":646,"end_line":654,"raw_tex":"It follows from \\eqref{pwfor} and \\eqref{domcr} that  for any $w\\in \\R^n$ with $\\nabla \\Phi(\\ox)w \\notin K_g(\\Phi(\\ox),\\olm)$, we have  $\\d^2g (\\Phi(\\bar x),\\olm) (\\nabla \\Phi(\\ox)w )=\\infty$.\nThis,  \\eqref{som}, and the definition of the convex cone $\\D$ from \\eqref{coned}  bring us to \n$$\n\\inf\\big\\{ h(w)|\\; w\\in \\S\\cap K_\\Th(\\ox,-\\ov)\\big\\}\\ge \\ell.\n$$\nCombining this and \\eqref{infco}, we can find a positive constant $ \\bar \\rho$ for which we have \n\\begin{equation}\\label{rb1}\nh_\\rho(w)>\\frac{\\ell}{2}\\quad \\mbox{for all}\\;\\;\\rho\\ge  \\bar \\rho\\;\\;\\mbox{and all}\\;\\;w\\in    \\S\\cap K_\\Th(\\ox,-\\ov).\n\\end{equation}\n","sha256_utf8":"a0757975a2df7b53c43ae7e87538f58299949da32f51c4e7bfbdf98a95be7c1b"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"齐次性恢复所有方向正性","source_units":[{"start":42305,"end":42733,"start_line":655,"end_line":661,"raw_tex":"Given $\\rho\\ge \\bar\\rho$, it follows from \\eqref{eq402} and the definition of $h_\\rho$  that \n$$\n\\d^2_x\\L\\big((\\ox,\\olm,\\rho),\\ov\\big)(w) >0\\quad \\mbox{for all} \\;\\; w\\in \\S\\cap K_\\Th(\\ox,-\\ov).\n$$\nRemember that the second-order epi-derivative  is positive homogeneous of degree $2$. So we arrive at the second-order condition \\eqref{eq26} for any $\\rho\\ge \\bar\\rho $. \nThis proves (b) and hence completes the proof.\n\\end{proof}","sha256_utf8":"8ed98aefbbcb4719aa1bf7a763812f17c701c550baa1b8e1e44a13af26d15c15"}]},{"id":"S050","semantic_type":"lemma","kind":"lemma","title":"邻近乘子的统一二阶下界","source_units":[{"start":44440,"end":45145,"start_line":682,"end_line":693,"raw_tex":"\\begin{Lemma}[second-order epi-derivatives of augmented Lagrangians] \\label{usog}\nLet $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and set $\\ov:= \\nabla_x L(\\ox,\\olm)$. \nThen the following conditions are equivalent: \n \\begin{enumerate}[noitemsep,topsep=0pt]\n \\item the  SOSC \\eqref{sosc} holds at $(\\ox,\\olm)$;\n \\item there exist positive constants $\\bar\\rho$, $\\ell_1$, $\\ve_1$ such that for all $\\rho\\ge\\bar\\rho$ and\nall  $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ we have\n\\begin{equation}\\label{eq44}\n\\d^2_x\\L \\big((\\ox,\\lm,\\rho), \\nabla_x L(\\ox,\\lm)\\big)(w)\\ge\\ell_1\\|w\\|^2\\;\\mbox{ for all }\\;w\\in K_\\Th\\big(\\ox, - \\nabla_x L(\\ox,\\olm)\\big).\n\\end{equation}\n\\end{enumerate}\n\\end{Lemma}","sha256_utf8":"55fea279c03c42ece5cef5f74592da9d820940e12af6adaf8e4117629b9a39ff"}]},{"id":"S051","semantic_type":"proof","kind":"proof","title":"统一二阶下界完整证明","source_units":[{"start":45146,"end":49544,"start_line":694,"end_line":740,"raw_tex":"\\begin{proof} The implication (b)$\\implies$(a)   results directly from Theorem~\\ref{growth}. To prove the opposite implication, assume that (a) holds. \nIt follows from Theorem~\\ref{growth} that there exists a constant $\\bar\\rho>0$ such that for all $\\rho\\ge\\bar\\rho$ the second-order condition \\eqref{eq26} is satisfied. \nRecall from  \\cite[Proposition~13.5]{rw} that  the second-order epi-derivative  is lower semicontinuous and positive homogenous of degree $2$. \nThus the second-order condition  \\eqref{eq26} is equivalent  to the existence of a constant $\\ell_1>0$ such that the estimate \n\\begin{equation}\\label{eq41}\n\\d^2_x\\L \\big((\\ox,\\olm,\\bar\\rho),\\ov \\big)(w)=\\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w, w\\big\\rangle+ e_{ {1}/{2\\bar\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\olm)\\big)(\\nabla\\Phi(\\ox)w)\\ge 2\\ell_1\n\\end{equation}\nholds for all $w\\in \\S\\cap K_\\Th(\\ox, -\\ov)$, where the equality in \\eqref{eq41} comes from  \\eqref{eq40}. We proceed now by proving the following claim:\n\n{\\bf Claim A.} {\\em There exists an $\\varepsilon_1>0$ such that for any $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ the estimate\n\\begin{equation}\\label{clq2}\n\\d^2_x\\L \\big((\\ox,\\lm,\\bar\\rho), \\nabla_x L(\\ox,\\lm) \\big)(w)=\\big\\langle \\nabla^2_{xx}L(\\ox,\\lm)w,w\\big\\rangle+ e_{ {1}/{2\\bar\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\lm)\\big)(\\nabla\\Phi(\\ox)w)\\ge \\ell_1\n\\end{equation} \n holds for all $w\\in \\S\\cap K_\\Th\\big(\\ox, -\\nabla_x L(\\ox,\\lm)\\big)$.}\n \n  To furnish this claim, choose $\\varepsilon_1\\in (0, {\\ell_1}/{\\|\\nabla^2\\Phi(\\ox)\\|})$ if $\\|\\nabla^2\\Phi(\\ox)\\|\\neq 0$ and $\\varepsilon_1 > 0$ otherwise\n  and let $w\\in \\S$.\n Thus for all     $\\lm\\in \\B_{\\ve_1}(\\olm)$, we obtain   \n\\begin{eqnarray}\\label{clq3}\n\\big\\langle \\nabla^2_{xx}L(\\ox,\\lm)w, w\\big\\rangle &=& \\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w, w\\big\\rangle+\\big\\langle\\lm -\\olm, \\nabla^2  \\Phi (\\ox)(w, w)\\big\\rangle\\nonumber\\\\\n&\\geq&\\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w, w\\big\\rangle - \\|\\nabla^2\\Phi(\\ox)\\|\\cdot\\|\\lm - \\olm\\|\\nonumber\\\\\n&\\geq&\\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w, w\\big\\rangle - \\ell_1.\n\\end{eqnarray}\nMoreover, it follows from \\eqref{cc2} that \n\\begin{eqnarray}\\label{clq4}\ne_{ {1}/{2\\bar\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\lm)\\big)(\\nabla\\Phi(\\ox)w)&=&\\inf_{v\\in \\R^m}\\big\\{\\d^2g\\big(\\Phi(\\ox)\\big)(v)+\\delta_{K_g(\\Phi(\\ox), \\lm)}(v)+\\bar\\rho\\|v-\\nabla\\Phi(\\ox)w\\|^2\\big\\}\\nonumber\\\\ \n&=& \\inf_{v\\in K_g(\\Phi(\\ox), \\lm)}\\big\\{\\d^2g \\big(\\Phi(\\ox) \\big)(v)+\\bar\\rho\\|v-\\nabla\\Phi(\\ox)w\\|^2\\big\\}. \n\\end{eqnarray}\nNote that $\\d^2g(\\Phi(\\ox))$ does not depend on $\\lambda$. Shrinking $\\ve_1$ if necessary and recalling that $\\sub g\\big(\\Phi(\\ox)\\big)$ is a polyhedral convex set, we conclude from   Proposition~\\ref{sop}(a) that  \nthe relationships \n\\begin{equation*}\nK_g(\\Phi(\\ox), \\lm) = N_{\\partial g(\\Phi(\\ox))}(\\lm) \\subset N_{\\partial g(\\Phi(\\ox))}(\\olm) = K_g(\\Phi(\\ox), \\olm) \n\\end{equation*} \nhold for all $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$.  These, together with  \\eqref{clq4}, tell us  that for any $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ we have \n\\begin{equation}\\label{clq5}\ne_{ {1}/{2\\bar\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\lm)\\big)(\\nabla\\Phi(\\ox)w) \\geq e_{ {1}/{2\\bar\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\olm)\\big)(\\nabla\\Phi(\\ox)w).\n\\end{equation} \n Combining \\eqref{eq41}, \\eqref{clq3}, and \\eqref{clq5} verifies   \\eqref{clq2} for all $w\\in \\S\\cap K_\\Th(\\ox, -\\ov)$ and   all $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$. \n To justify \\eqref{clq2} for all $w\\in \\S\\cap  K_\\Th \\big(\\ox, -\\nabla_x L(\\ox, \\lm) \\big)$, observe that by the  polyhedrality of $\\Th$,   we can assume without loss of generality that \n $$\n K_\\Th \\big(\\ox, -\\nabla_x L(\\ox, \\lm) \\big)=N_{N_\\Th(\\ox)}\\big(-\\nabla_x L(\\ox, \\lm)\\big) \\subset N_{N_\\Th(\\ox)}\\big(-\\nabla_x L(\\ox, \\olm)\\big) = K_\\Th(\\ox, -\\ov)\n $$\n  for all $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$. This clearly justifies Claim A. \n  \nUsing the positive homogeneity of the second-order epi-derivative yields \\eqref{eq44} for $\\rho = \\bar\\rho$ and for all $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$. Recall that the function $$\\rho\\mapsto e_{ {1}/{2\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\lm)\\big)(\\nabla\\Phi(\\ox)w)$$ is nondecreasing. So is the function $\\rho \\mapsto\\d^2\\L( (\\ox, \\lm, \\rho), \\nabla_x L(\\ox, \\lm))(w)$. This  \nyields  \\eqref{eq44}  for all $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ and all $\\rho \\geq \\bar\\rho$, and hence completes the   proof.\n\\end{proof}","sha256_utf8":"5138874805de656bbd7ac89695e35e41136e7834f4b37d20cd8307668f694357"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"固定乘子的严格二阶余量","source_units":[{"start":45146,"end":46119,"start_line":694,"end_line":701,"raw_tex":"\\begin{proof} The implication (b)$\\implies$(a)   results directly from Theorem~\\ref{growth}. To prove the opposite implication, assume that (a) holds. \nIt follows from Theorem~\\ref{growth} that there exists a constant $\\bar\\rho>0$ such that for all $\\rho\\ge\\bar\\rho$ the second-order condition \\eqref{eq26} is satisfied. \nRecall from  \\cite[Proposition~13.5]{rw} that  the second-order epi-derivative  is lower semicontinuous and positive homogenous of degree $2$. \nThus the second-order condition  \\eqref{eq26} is equivalent  to the existence of a constant $\\ell_1>0$ such that the estimate \n\\begin{equation}\\label{eq41}\n\\d^2_x\\L \\big((\\ox,\\olm,\\bar\\rho),\\ov \\big)(w)=\\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w, w\\big\\rangle+ e_{ {1}/{2\\bar\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\olm)\\big)(\\nabla\\Phi(\\ox)w)\\ge 2\\ell_1\n\\end{equation}\nholds for all $w\\in \\S\\cap K_\\Th(\\ox, -\\ov)$, where the equality in \\eqref{eq41} comes from  \\eqref{eq40}. We proceed now by proving the following claim:\n","sha256_utf8":"26020ef2d708aa449d4e31e173843b80707dcb8387350ed51747d2e491ce8248"}]},{"id":"S053","semantic_type":"derived_fact","kind":"derived_fact","title":"邻近乘子下界之声明","source_units":[{"start":46120,"end":46565,"start_line":703,"end_line":707,"raw_tex":"{\\bf Claim A.} {\\em There exists an $\\varepsilon_1>0$ such that for any $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ the estimate\n\\begin{equation}\\label{clq2}\n\\d^2_x\\L \\big((\\ox,\\lm,\\bar\\rho), \\nabla_x L(\\ox,\\lm) \\big)(w)=\\big\\langle \\nabla^2_{xx}L(\\ox,\\lm)w,w\\big\\rangle+ e_{ {1}/{2\\bar\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\lm)\\big)(\\nabla\\Phi(\\ox)w)\\ge \\ell_1\n\\end{equation} \n holds for all $w\\in \\S\\cap K_\\Th\\big(\\ox, -\\nabla_x L(\\ox,\\lm)\\big)$.}\n","sha256_utf8":"7c10b1abe249f884b13c440feda43bcef88c0f351e84aa1f9521c8063ad3e64e"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"控制拉格朗日海森扰动","source_units":[{"start":46567,"end":47204,"start_line":709,"end_line":716,"raw_tex":"  To furnish this claim, choose $\\varepsilon_1\\in (0, {\\ell_1}/{\\|\\nabla^2\\Phi(\\ox)\\|})$ if $\\|\\nabla^2\\Phi(\\ox)\\|\\neq 0$ and $\\varepsilon_1 > 0$ otherwise\n  and let $w\\in \\S$.\n Thus for all     $\\lm\\in \\B_{\\ve_1}(\\olm)$, we obtain   \n\\begin{eqnarray}\\label{clq3}\n\\big\\langle \\nabla^2_{xx}L(\\ox,\\lm)w, w\\big\\rangle &=& \\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w, w\\big\\rangle+\\big\\langle\\lm -\\olm, \\nabla^2  \\Phi (\\ox)(w, w)\\big\\rangle\\nonumber\\\\\n&\\geq&\\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w, w\\big\\rangle - \\|\\nabla^2\\Phi(\\ox)\\|\\cdot\\|\\lm - \\olm\\|\\nonumber\\\\\n&\\geq&\\big\\langle \\nabla^2_{xx}L(\\ox,\\olm)w, w\\big\\rangle - \\ell_1.\n\\end{eqnarray}\n","sha256_utf8":"49a496229bc3b8e1153527bf55efdf4f255353dc07ff36bfc30bdb3dd2cd22b3"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"将包络写为临界锥极小化","source_units":[{"start":47204,"end":47609,"start_line":717,"end_line":721,"raw_tex":"Moreover, it follows from \\eqref{cc2} that \n\\begin{eqnarray}\\label{clq4}\ne_{ {1}/{2\\bar\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\lm)\\big)(\\nabla\\Phi(\\ox)w)&=&\\inf_{v\\in \\R^m}\\big\\{\\d^2g\\big(\\Phi(\\ox)\\big)(v)+\\delta_{K_g(\\Phi(\\ox), \\lm)}(v)+\\bar\\rho\\|v-\\nabla\\Phi(\\ox)w\\|^2\\big\\}\\nonumber\\\\ \n&=& \\inf_{v\\in K_g(\\Phi(\\ox), \\lm)}\\big\\{\\d^2g \\big(\\Phi(\\ox) \\big)(v)+\\bar\\rho\\|v-\\nabla\\Phi(\\ox)w\\|^2\\big\\}. \n\\end{eqnarray}\n","sha256_utf8":"46e51d1b00bbccbdbf786b9ac06ba8821c9525d881ab4dc46e91d9a05e979006"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"临界锥包含给出包络下界","source_units":[{"start":47609,"end":48368,"start_line":722,"end_line":730,"raw_tex":"Note that $\\d^2g(\\Phi(\\ox))$ does not depend on $\\lambda$. Shrinking $\\ve_1$ if necessary and recalling that $\\sub g\\big(\\Phi(\\ox)\\big)$ is a polyhedral convex set, we conclude from   Proposition~\\ref{sop}(a) that  \nthe relationships \n\\begin{equation*}\nK_g(\\Phi(\\ox), \\lm) = N_{\\partial g(\\Phi(\\ox))}(\\lm) \\subset N_{\\partial g(\\Phi(\\ox))}(\\olm) = K_g(\\Phi(\\ox), \\olm) \n\\end{equation*} \nhold for all $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$.  These, together with  \\eqref{clq4}, tell us  that for any $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ we have \n\\begin{equation}\\label{clq5}\ne_{ {1}/{2\\bar\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\lm)\\big)(\\nabla\\Phi(\\ox)w) \\geq e_{ {1}/{2\\bar\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\olm)\\big)(\\nabla\\Phi(\\ox)w).\n\\end{equation} \n","sha256_utf8":"fe82c5fd4eec4428aa00b6d20a510ab96c45df2c423acbb866697d6c6dc09998"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"合并下界并比较约束临界锥","source_units":[{"start":48368,"end":49001,"start_line":731,"end_line":736,"raw_tex":" Combining \\eqref{eq41}, \\eqref{clq3}, and \\eqref{clq5} verifies   \\eqref{clq2} for all $w\\in \\S\\cap K_\\Th(\\ox, -\\ov)$ and   all $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$. \n To justify \\eqref{clq2} for all $w\\in \\S\\cap  K_\\Th \\big(\\ox, -\\nabla_x L(\\ox, \\lm) \\big)$, observe that by the  polyhedrality of $\\Th$,   we can assume without loss of generality that \n $$\n K_\\Th \\big(\\ox, -\\nabla_x L(\\ox, \\lm) \\big)=N_{N_\\Th(\\ox)}\\big(-\\nabla_x L(\\ox, \\lm)\\big) \\subset N_{N_\\Th(\\ox)}\\big(-\\nabla_x L(\\ox, \\olm)\\big) = K_\\Th(\\ox, -\\ov)\n $$\n  for all $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$. This clearly justifies Claim A. \n","sha256_utf8":"7453ce17357e1768a93a7e0863707d4ae2fe132452d00c02e547bc8b03804cc6"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"齐次性与罚参数单调推广","source_units":[{"start":49004,"end":49544,"start_line":738,"end_line":740,"raw_tex":"Using the positive homogeneity of the second-order epi-derivative yields \\eqref{eq44} for $\\rho = \\bar\\rho$ and for all $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$. Recall that the function $$\\rho\\mapsto e_{ {1}/{2\\rho}}\\big(\\d^2 g(\\Phi(\\ox),\\lm)\\big)(\\nabla\\Phi(\\ox)w)$$ is nondecreasing. So is the function $\\rho \\mapsto\\d^2\\L( (\\ox, \\lm, \\rho), \\nabla_x L(\\ox, \\lm))(w)$. This  \nyields  \\eqref{eq44}  for all $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ and all $\\rho \\geq \\bar\\rho$, and hence completes the   proof.\n\\end{proof}","sha256_utf8":"97fd6d8a2525f98869263bed078d2169a1079b7b19c229c12dfe0e0f97e15f04"}]},{"id":"S059","semantic_type":"theorem","kind":"theorem","title":"乘子邻域内统一二次增长","source_units":[{"start":49685,"end":50374,"start_line":744,"end_line":754,"raw_tex":"\\begin{Theorem}[uniform quadratic growth condition for augmented Lagrangians]\\label{ugrowth}\nLet $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs}. Then the following conditions are equivalent:\n \\begin{enumerate}[noitemsep,topsep=0pt]\n \\item    the  SOSC \\eqref{sosc} holds at $(\\ox,\\olm)$;\n \\item there are positive constants $\\bar\\rho, \\gamma, \\varepsilon,\\ell$ such that for all $\\lm\\in\\Lambda(\\ox)\\cap\\B_\\varepsilon(\\olm)$ and all $\\rho\\ge\\bar\\rho$  the uniform quadratic growth condition\n\\begin{equation}\\label{eq14}\n\\L(x,\\lm,\\rho)\\ge \\ph(\\ox)+g(\\Phi(\\ox))+\\ell\\| x-\\ox\\|^2\\;\\mbox{ for all }\\;x\\in\\B_\\gamma(\\ox)\\cap \\Th\n\\end{equation}\nis satisfied.\n\\end{enumerate}\n\\end{Theorem}","sha256_utf8":"83c468b4ca006e3d37ddba8824534654eecbff0277261073c840ca3342747595"}]},{"id":"S060","semantic_type":"proof","kind":"proof","title":"统一增长完整证明","source_units":[{"start":50377,"end":55194,"start_line":756,"end_line":805,"raw_tex":"\\begin{proof} The implication (b)$\\implies$(a)   results directly from Theorem~\\ref{growth}. To prove the opposite implication, assume that (a) holds. \nIt follows from Lemma~\\ref{usog} that there exist the positive constants $\\ell_1$, $\\varepsilon_1$, and $\\bar\\rho$  for which \\eqref{eq44} is satisfied  for all \n $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ and all $\\rho\\ge\\bar\\rho$. Using this and  Proposition~\\ref{fopag}(a), \n we deduce from \\eqref{eq44} that for any $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ there exists $\\gamma_{\\lm}>0$ for which we have \n\\begin{equation}\\label{eq22}\n\\L(x,\\lm,\\bar\\rho)\\ge \\ph(\\ox)+g(\\Phi(\\ox))+\\dfrac{\\ell_1}{2}\\Vert x-\\ox\\Vert^2\\;\\textrm{ for all }\\;x\\in\\B_{\\gamma_\\lm}(\\ox)\\cap \\Th,\n\\end{equation}\nwhere the constant $ {\\ell_1}$ can be chosen the same for all the multipliers $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$. The\nlatter results directly from the definition of the second subderivative. The radii of the balls centered at $\\ox$ in \\eqref{eq22}, however,  depend on $\\lm$. We argue below that  a common radius can be chosen for all the multipliers $\\lm\\in\\Lambda(\\ox)$ that are sufficiently close to $\\olm$. To this end, assume that \n$$\n\\tilde\\B_{\\varepsilon_1}(\\olm)=\\big\\{\\lm\\in \\R^m|\\; \\|\\lm-\\olm\\|_1\\le \\ve_1\\big\\}\n$$\nwhere $\\|\\lm-\\olm\\|_1$ stands for $L_1$-norm of the vector $\\lm-\\olm$, namely the sum of the absolute values of the components of $\\lm-\\olm$. Clearly, we have $\\tilde\\B_{\\varepsilon_1}(\\olm)\\subset \\B_{\\varepsilon_1}(\\olm)$.  Define the function $\\psi\\colon\\R^m\\to\\oR$ by\n\\begin{equation}\\label{eq23}\n\\psi(\\lm):=\\sup_{x\\in\\big(\\B_{\\gamma_{\\olm}}(\\ox)\\cap \\Th\\big)\\setminus \\{\\ox\\}}\\frac{\\ph(\\ox)+g(\\Phi(\\ox))-\\L(x,\\lambda,\\bar\\rho)}{\\Vert x-\\ox\\Vert^2}\n+\\delta_{\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm)} (\\lambda),\\quad\\lm\\in\\R^m.\n\\end{equation}\nBy Proposition~\\ref{paug}(b), the function $\\lambda\\mapsto\\L(x,\\lm,\\bar\\rho)$ is concave. This, combined with the convexity of the set $\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm)$, \n guarantees  that $\\psi$ in \\eqref{eq23} is a convex function. Moreover, $\\psi$ is lower semicontinuous since  the first term on the right-hand side of \\eqref{eq23} is the pointwise supremum of a collection of continuous functions.\n Now we claim  that for any $\\lm\\in\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm)$ the value $\\psi(\\lm)$ is finite. To justify this claim, let $\\lm\\in\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm)$. If  $\\gamma_\\lm\\ge\\gamma_{\\olm}$,  we conclude from    \\eqref{eq22} that \n\\begin{equation*}\n\\psi(\\lm)\\le\\sup_{x\\in \\big(\\B_{\\gamma_{\\lm}}(\\ox)\\cap \\Th\\big)\\setminus \\{\\ox\\}}\\frac{\\ph(\\ox)+g(\\Phi(\\ox))-\\L(x,\\lm,\\bar\\rho)}{\\Vert x-\\ox\\Vert^2}\\le-\\dfrac{\\ell_1}{2},\n\\end{equation*}\nwhich, in particular,   implies that $\\psi(\\olm)\\le- {\\ell_1}/{2}$. On the other hand, if $\\gamma_\\lm<\\gamma_{\\olm}$, we get  \n\\begin{eqnarray*}\n &&\\psi(\\lm) \\le\\\\\n&&  \\max\\left\\{\\sup_{x\\in \\big(\\B_{\\gamma_{\\lm}}(\\ox)\\cap \\Th\\big)\\setminus \\{\\ox\\}}\\frac{\\ph(\\ox)+g(\\Phi(\\ox))-\\L(x,\\lm,\\bar\\rho)}{\\Vert x-\\ox\\Vert^2},\\max_{\\substack{\\gamma_\\lm\\le\\Vert x-\\ox\\Vert\\le\\gamma_{\\olm}\\\\ x\\in \\Th}}\\dfrac{\\ph(\\ox)+g(\\Phi(\\ox))-\\L(x,\\lm,\\bar\\rho)}{\\Vert x-\\ox\\Vert^2}\\right\\}.\n\\end{eqnarray*}\nBy \\eqref{eq22}, the first term inside the maximum is bounded above by  $- {\\ell_1}/{2}$.  Also, the second term is finite since it is the maximum of a continuous function over a compact set. \nCombining these tells us that $\\psi(\\lm)$ is finite for all $\\lm\\in\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm)$, which  results in \n$$\n\\dom\\psi=\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm).\n$$\nSince both sets  $\\Lambda(\\ox)$ and $\\tilde\\B_{\\varepsilon_1}(\\olm)$ are polyhedral, $\\dom\\psi$ is   a compact polyhedral convex set.\nAppealing now to   \\cite[Theorem~2.35]{rw} ensures that  $\\psi$ is continuous relative to its domain.  \nThus, we can find  $\\varepsilon >0$ such that $\\B_\\varepsilon(\\olm)\\subset \\tilde\\B_{\\varepsilon_1}(\\olm)$   and that\n\\begin{equation*}\n\\psi(\\lm)\\leq \\psi(\\olm)+\\frac{\\ell_1}{4}\\leq -\\frac{\\ell_1}{4}  \n\\end{equation*} \nfor all $\\lm \\in \\dom\\psi\\cap \\B_\\varepsilon(\\olm)=\\Lambda(\\ox)\\cap \\B_\\varepsilon(\\olm)$. This, combined with the definition of $\\psi$ in \\eqref{eq23}, tells us that for \nall $x\\in\\B_{\\gamma_{\\olm}}(\\ox)\\cap \\Th$ and all $\\lm\\in\\Lambda(\\ox)\\cap\\B_\\varepsilon(\\olm)$ we have \n\\begin{equation}\\label{eq25}\n\\L(x,\\lm,\\bar\\rho)\\ge \\ph(\\ox)+g(\\Phi(\\ox))+\\frac{\\ell_1}{4}\\Vert x-\\bar x\\Vert^2.\n\\end{equation}\nRemember that by  Proposition~\\ref{paug}(c) we always have \n\\begin{equation*}\n\\L(x,\\lm,\\rho)\\ge\\L(x,\\lm,\\bar\\rho)\\;\\mbox{ for all}\\;\\;\\rho\\ge\\bar\\rho.\n\\end{equation*}\nCombining this with \\eqref{eq25} and setting $\\ell:= {\\ell_1}/{4}$ verify the uniform quadratic growth condition \\eqref{eq14} and hence complete the proof.\n\\end{proof}","sha256_utf8":"a36125c20f38934e4565ddfbbd4004f6bfaa9d3ee51f7b428778734a3007dcb8"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"各乘子共享增长系数","source_units":[{"start":50377,"end":51583,"start_line":756,"end_line":764,"raw_tex":"\\begin{proof} The implication (b)$\\implies$(a)   results directly from Theorem~\\ref{growth}. To prove the opposite implication, assume that (a) holds. \nIt follows from Lemma~\\ref{usog} that there exist the positive constants $\\ell_1$, $\\varepsilon_1$, and $\\bar\\rho$  for which \\eqref{eq44} is satisfied  for all \n $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ and all $\\rho\\ge\\bar\\rho$. Using this and  Proposition~\\ref{fopag}(a), \n we deduce from \\eqref{eq44} that for any $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$ there exists $\\gamma_{\\lm}>0$ for which we have \n\\begin{equation}\\label{eq22}\n\\L(x,\\lm,\\bar\\rho)\\ge \\ph(\\ox)+g(\\Phi(\\ox))+\\dfrac{\\ell_1}{2}\\Vert x-\\ox\\Vert^2\\;\\textrm{ for all }\\;x\\in\\B_{\\gamma_\\lm}(\\ox)\\cap \\Th,\n\\end{equation}\nwhere the constant $ {\\ell_1}$ can be chosen the same for all the multipliers $\\lm\\in\\Lambda(\\ox)\\cap\\B_{\\varepsilon_1}(\\olm)$. The\nlatter results directly from the definition of the second subderivative. The radii of the balls centered at $\\ox$ in \\eqref{eq22}, however,  depend on $\\lm$. We argue below that  a common radius can be chosen for all the multipliers $\\lm\\in\\Lambda(\\ox)$ that are sufficiently close to $\\olm$. To this end, assume that \n","sha256_utf8":"5ba8d85f46adb4f589fc0664ab3761c3e0882d0acb3699b993c0e6ebd61cd941"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"构造凸下半连续上确界","source_units":[{"start":51583,"end":52643,"start_line":765,"end_line":774,"raw_tex":"$$\n\\tilde\\B_{\\varepsilon_1}(\\olm)=\\big\\{\\lm\\in \\R^m|\\; \\|\\lm-\\olm\\|_1\\le \\ve_1\\big\\}\n$$\nwhere $\\|\\lm-\\olm\\|_1$ stands for $L_1$-norm of the vector $\\lm-\\olm$, namely the sum of the absolute values of the components of $\\lm-\\olm$. Clearly, we have $\\tilde\\B_{\\varepsilon_1}(\\olm)\\subset \\B_{\\varepsilon_1}(\\olm)$.  Define the function $\\psi\\colon\\R^m\\to\\oR$ by\n\\begin{equation}\\label{eq23}\n\\psi(\\lm):=\\sup_{x\\in\\big(\\B_{\\gamma_{\\olm}}(\\ox)\\cap \\Th\\big)\\setminus \\{\\ox\\}}\\frac{\\ph(\\ox)+g(\\Phi(\\ox))-\\L(x,\\lambda,\\bar\\rho)}{\\Vert x-\\ox\\Vert^2}\n+\\delta_{\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm)} (\\lambda),\\quad\\lm\\in\\R^m.\n\\end{equation}\nBy Proposition~\\ref{paug}(b), the function $\\lambda\\mapsto\\L(x,\\lm,\\bar\\rho)$ is concave. This, combined with the convexity of the set $\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm)$, \n guarantees  that $\\psi$ in \\eqref{eq23} is a convex function. Moreover, $\\psi$ is lower semicontinuous since  the first term on the right-hand side of \\eqref{eq23} is the pointwise supremum of a collection of continuous functions.\n","sha256_utf8":"53f9626c05894dab0fcff4607cb31d33995baa31e1520b735debbeaea7ebfb21"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"内球与紧环带保证有限性","source_units":[{"start":52643,"end":54000,"start_line":775,"end_line":788,"raw_tex":" Now we claim  that for any $\\lm\\in\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm)$ the value $\\psi(\\lm)$ is finite. To justify this claim, let $\\lm\\in\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm)$. If  $\\gamma_\\lm\\ge\\gamma_{\\olm}$,  we conclude from    \\eqref{eq22} that \n\\begin{equation*}\n\\psi(\\lm)\\le\\sup_{x\\in \\big(\\B_{\\gamma_{\\lm}}(\\ox)\\cap \\Th\\big)\\setminus \\{\\ox\\}}\\frac{\\ph(\\ox)+g(\\Phi(\\ox))-\\L(x,\\lm,\\bar\\rho)}{\\Vert x-\\ox\\Vert^2}\\le-\\dfrac{\\ell_1}{2},\n\\end{equation*}\nwhich, in particular,   implies that $\\psi(\\olm)\\le- {\\ell_1}/{2}$. On the other hand, if $\\gamma_\\lm<\\gamma_{\\olm}$, we get  \n\\begin{eqnarray*}\n &&\\psi(\\lm) \\le\\\\\n&&  \\max\\left\\{\\sup_{x\\in \\big(\\B_{\\gamma_{\\lm}}(\\ox)\\cap \\Th\\big)\\setminus \\{\\ox\\}}\\frac{\\ph(\\ox)+g(\\Phi(\\ox))-\\L(x,\\lm,\\bar\\rho)}{\\Vert x-\\ox\\Vert^2},\\max_{\\substack{\\gamma_\\lm\\le\\Vert x-\\ox\\Vert\\le\\gamma_{\\olm}\\\\ x\\in \\Th}}\\dfrac{\\ph(\\ox)+g(\\Phi(\\ox))-\\L(x,\\lm,\\bar\\rho)}{\\Vert x-\\ox\\Vert^2}\\right\\}.\n\\end{eqnarray*}\nBy \\eqref{eq22}, the first term inside the maximum is bounded above by  $- {\\ell_1}/{2}$.  Also, the second term is finite since it is the maximum of a continuous function over a compact set. \nCombining these tells us that $\\psi(\\lm)$ is finite for all $\\lm\\in\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm)$, which  results in \n$$\n\\dom\\psi=\\Lambda(\\ox)\\cap \\tilde\\B_{\\varepsilon_1}(\\olm).\n$$\n","sha256_utf8":"8454040243d0779966d4b3453acf1d81fca98144cf0eff74343331de3e55985b"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"相对连续性给出共同半径","source_units":[{"start":54000,"end":54860,"start_line":789,"end_line":799,"raw_tex":"Since both sets  $\\Lambda(\\ox)$ and $\\tilde\\B_{\\varepsilon_1}(\\olm)$ are polyhedral, $\\dom\\psi$ is   a compact polyhedral convex set.\nAppealing now to   \\cite[Theorem~2.35]{rw} ensures that  $\\psi$ is continuous relative to its domain.  \nThus, we can find  $\\varepsilon >0$ such that $\\B_\\varepsilon(\\olm)\\subset \\tilde\\B_{\\varepsilon_1}(\\olm)$   and that\n\\begin{equation*}\n\\psi(\\lm)\\leq \\psi(\\olm)+\\frac{\\ell_1}{4}\\leq -\\frac{\\ell_1}{4}  \n\\end{equation*} \nfor all $\\lm \\in \\dom\\psi\\cap \\B_\\varepsilon(\\olm)=\\Lambda(\\ox)\\cap \\B_\\varepsilon(\\olm)$. This, combined with the definition of $\\psi$ in \\eqref{eq23}, tells us that for \nall $x\\in\\B_{\\gamma_{\\olm}}(\\ox)\\cap \\Th$ and all $\\lm\\in\\Lambda(\\ox)\\cap\\B_\\varepsilon(\\olm)$ we have \n\\begin{equation}\\label{eq25}\n\\L(x,\\lm,\\bar\\rho)\\ge \\ph(\\ox)+g(\\Phi(\\ox))+\\frac{\\ell_1}{4}\\Vert x-\\bar x\\Vert^2.\n\\end{equation}\n","sha256_utf8":"0c1a30f743278c86cf8282613dd2666f6a672cdcdf15a29a7aaf5482f6f962b8"}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","title":"单调性推广至全部大罚参数","source_units":[{"start":54860,"end":55194,"start_line":800,"end_line":805,"raw_tex":"Remember that by  Proposition~\\ref{paug}(c) we always have \n\\begin{equation*}\n\\L(x,\\lm,\\rho)\\ge\\L(x,\\lm,\\bar\\rho)\\;\\mbox{ for all}\\;\\;\\rho\\ge\\bar\\rho.\n\\end{equation*}\nCombining this with \\eqref{eq25} and setting $\\ell:= {\\ell_1}/{4}$ verify the uniform quadratic growth condition \\eqref{eq14} and hence complete the proof.\n\\end{proof}","sha256_utf8":"1002e4618016e6204d1cfc0ceca289afc96b75cd5aa5d4acd120542b18b3d8f6"}]},{"id":"S066","semantic_type":"proposition","kind":"proposition","title":"子问题可解与统一孤立平静","source_units":[{"start":59607,"end":60538,"start_line":866,"end_line":878,"raw_tex":"\\begin{Proposition}[solvability of subproblems]\\label{solv} \nLet $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and let  the  SOSC \\eqref{sosc} hold at $(\\ox,\\olm)$.\nThen there exist positive constants      $\\tau$ and $\\hat\\gamma\\le \\gg$  with $\\gg$ taken from Theorem~{\\rm \\ref{ugrowth}}  such that for any $\\rho\\ge \\bar\\rho $ with $\\bar\\rho$ taken from Theorem~{\\rm\\ref{ugrowth}},\n the optimal solution mapping $S_\\rho:\\R^m\\to \\R^n$, defined by \n\\begin{equation}\\label{eq177}\nS_\\rho(\\lambda):=\\argmin\\big\\{\\L(x,\\lambda,\\rho)\\;\\big|\\; x\\in\\Th\\cap\\B_{\\hat\\gamma}(\\ox)\\big\\}, \\;\\;\\lm\\in \\R^m,\n\\end{equation}\nenjoys the uniform  isolated calmness property \n\\begin{equation}\\label{pt9}\n   S_\\rho(\\lambda) \\subset\\{\\ox\\} +\\tau \\Vert\\lambda-\\olm\\Vert\\B\n\\end{equation}\n and satisfies the condition $\\emp \\neq S_\\rho(\\lambda)\\subset  \\inte  \\B_{\\hat\\gamma}(\\ox)$ for all $\\lm\\in \\B_{\\hat\\gamma/2\\tau}(\\olm)$.  \n \\end{Proposition}","sha256_utf8":"7ee844e9a8198dfcb67e8a2a699fb9c704f8587478b8eb88971f570ab5166aed"}]},{"id":"S067","semantic_type":"proof","kind":"proof","title":"子问题可解性完整证明","source_units":[{"start":60540,"end":64606,"start_line":880,"end_line":934,"raw_tex":"\\begin{proof} Since the SOSC \\eqref{sosc} is satisfied at $(\\ox,\\olm)$, it follows from Theorem~\\ref{ugrowth}\nthat there exist positive constants $\\bar\\rho$, $\\gg$, and $\\ell$ such that for any $\\rho\\ge\\bar \\rho$, the uniform quadratic growth condition \\eqref{eq14} holds for $\\lm=\\olm$. Moreover, \nby the twice differentiability of  $\\Phi$  at $\\ox$, we find some  constants $\\hat\\gamma\\in(0,\\gamma]$ and $\\kappa_\\Phi>0$ for which we have \n\\begin{equation}\\label{eq49}\n\\Vert\\Phi(x)-\\Phi(\\ox)\\Vert\\le\\kappa_\\Phi\\Vert x-\\ox\\Vert\\;\\textrm{ for all }\\;x\\in\\B_{\\hat\\gamma}(\\ox).\n\\end{equation}\nClearly,  the uniform quadratic growth condition \\eqref{eq14} for $\\lm=\\olm$ indicates  that  $S_\\rho(\\olm)\\cap \\B_{\\gamma}(\\ox)=\\{\\ox\\}$ for all $\\rho\\ge\\bar\\rho$. \nDefine   the positive constant\n\\begin{equation}\\label{eq21}\n\\tau:=\\frac{\\kappa_\\Phi}{\\ell}+\\sqrt{\\frac{\\kappa_\\Phi^2}{\\ell^2}+\\frac{1}{\\ell\\bar\\rho}},\n\\end{equation}\nand fix   $\\lambda\\in \\B_{\\hat\\gamma/2\\tau} (\\olm)$ and $\\rho\\geq\\bar\\rho$.\nObserve from the classical Weierstrass theorem that $S_\\rho(\\lm)\\neq\\emp$.   Assume that  $u\\in S_\\rho(\\lambda)$.\nBy  Proposition~\\ref{paug}(b),   the function $\\lambda\\mapsto\\L(u,\\lambda,\\rho)$ is concave. \nBy \\cite[Theorem~2.26(b)]{rw}, we have \n\\begin{equation}\\label{ueq19}\n\\nabla_\\lm\\L(u, \\lm, \\rho)=\\rho^{-1}\\big(\\nabla\\menv\\big(\\Phi(u)+\\rho^{-1}{\\lm}\\big) - \\lm\\big)\n= \\Phi(u)-\\prox \\big(\\Phi(u)+ \\rho^{-1}\\lm\\big),\n\\end{equation} \n which in turn yields  the relationships\n\\begin{eqnarray*} \n\\L(u,\\lambda,\\rho)&\\ge&\\L(u,\\olm,\\rho)-\\langle\\nabla_\\lambda\\L(u,\\lambda,\\rho),\\olm-\\lambda\\rangle\\nonumber\\\\\n&=&\\L(u,\\olm,\\rho)-\\big\\langle \\Phi(u)-\\prox\\big(\\Phi(u)+ \\rho^{-1}{\\lm}\\big),\\olm-\\lambda\\big\\rangle\\nonumber\\\\\n&\\ge& \\ph(\\ox)+g(\\Phi(\\ox))+\\ell\\Vert u-\\ox\\Vert^2-\\big\\langle \\Phi(u)-\\prox\\big(\\Phi(u)+ \\rho^{-1}{\\lm} \\big),\\olm-\\lambda\\big\\rangle\n\\end{eqnarray*}\nwith the last inequality resulting from  \\eqref{eq16}. We conclude from  $u\\in S_\\rho(\\lambda)$ that \n\\begin{equation*}\n\\L(u,\\lambda,\\rho)\\le\\L(\\ox,\\lambda,\\rho)=\\ph(\\ox)+\\menv\\big(\\Phi(\\ox)+\\rho^{-1}{\\lm}\\big) - \\sm \\rho^{-1}{\\|\\lm\\|^2}\\leq \\ph(\\ox)+g \\big(\\Phi(\\ox) \\big).\n\\end{equation*}\nCombining these brings us to \n\\begin{equation}\\label{eq20}\n\\Vert u-\\ox\\Vert^2\\le\\frac{1}{\\ell}\\big\\langle \\Phi(u)-\\prox\\big(\\Phi(u)+\\rho^{-1}\\lm\\big),\\olm-\\lambda\\big\\rangle.\n\\end{equation}\nBy the first equality in \\eqref{prm}, the inclusion  $\\olm \\in \\partial g\\big(\\Phi(\\ox)\\big)$ amounts to $\\Phi(\\ox) = \\prox\\big(\\Phi(\\ox)+ \\rho^{-1}\\olm\\big)$. This leads us to \n\\begin{eqnarray}\\label{clq7}\n&&\\big\\Vert\\Phi(u)-\\prox\\big(\\Phi(u)+\\rho^{-1}\\lm\\big)\\big\\Vert \\nonumber\\\\\n&=&\\big\\Vert\\Phi(u)-\\Phi(\\ox)+\\prox\\big(\\Phi(\\ox)+\\rho^{-1}\\olm\\big)-\\prox\\big(\\Phi(u)+\\rho^{-1}\\lm\\big)\\big\\Vert\\nonumber\\\\\n&\\le&2\\Vert\\Phi(u)-\\Phi(\\ox)\\Vert+\\rho^{-1}{\\Vert\\olm-\\lambda\\Vert}\\nonumber\\\\\n&\\le&2\\kappa_\\Phi\\Vert u-\\ox\\Vert+\\rho^{-1}{\\Vert\\olm-\\lambda\\Vert},\n\\end{eqnarray}\nwhere the first inequality comes from  the fact that proximal mapping of a convex function is always nonexpansive (cf.  \\cite[Proposition~12.19]{rw}), \nand where the last inequality results from \\eqref{eq49}. This, combined with  \\eqref{eq20},  brings us   to \n\\begin{equation*}\n\\Vert u-\\ox\\Vert^2\\le\\frac{1}{\\ell}\\Big(2\\kappa_\\Phi\\Vert u-\\ox\\Vert+\\rho^{-1}{\\Vert\\lambda-\\olm\\Vert}\\Big)\\Vert\\lambda-\\olm\\Vert.\n\\end{equation*}\nSince the latter inequality  can be expressed  equivalently as \n\\begin{equation*}\n\\ell\\Vert u-\\ox\\Vert^2-2\\kappa_\\Phi\\Vert \\lambda-\\olm\\Vert\\cdot\\Vert u-\\ox\\Vert-\\rho^{-1}{\\Vert \\lambda-\\olm\\Vert^2 }\\le 0,\n\\end{equation*}\nwe arrive at  the estimate\n\\begin{equation*}\n\\Vert u-\\ox\\Vert\\le\\left(\\dfrac{\\kappa_\\Phi}{\\ell}+\\sqrt{\\frac{\\kappa_\\Phi^2}{\\ell^2}+\\frac{1}{\\ell\\rho}}\\right)\\Vert\\lambda-\\olm\\Vert\\le\\tau\\Vert\\lambda-\\olm\\Vert \\le \\tau \\hat\\gamma/2\\tau < \\hat\\gamma.\n\\end{equation*}\nThis clearly justifies   \\eqref{pt9}   and  the inclusion $S_\\rho(\\lambda)\\subset  \\inte  \\B_{\\hat\\gamma}(\\ox)$ for all  $\\lambda\\in \\B_{\\hat\\gamma/2\\tau} (\\olm)$ and $\\rho\\geq\\bar\\rho$,  and hence completes the proof.\n\\end{proof}","sha256_utf8":"fc629cefe138ce9ffcf929a13326e06425788aeeda21494f3f66bed123e7d757"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"增长条件与局部光滑界","source_units":[{"start":60540,"end":61296,"start_line":880,"end_line":886,"raw_tex":"\\begin{proof} Since the SOSC \\eqref{sosc} is satisfied at $(\\ox,\\olm)$, it follows from Theorem~\\ref{ugrowth}\nthat there exist positive constants $\\bar\\rho$, $\\gg$, and $\\ell$ such that for any $\\rho\\ge\\bar \\rho$, the uniform quadratic growth condition \\eqref{eq14} holds for $\\lm=\\olm$. Moreover, \nby the twice differentiability of  $\\Phi$  at $\\ox$, we find some  constants $\\hat\\gamma\\in(0,\\gamma]$ and $\\kappa_\\Phi>0$ for which we have \n\\begin{equation}\\label{eq49}\n\\Vert\\Phi(x)-\\Phi(\\ox)\\Vert\\le\\kappa_\\Phi\\Vert x-\\ox\\Vert\\;\\textrm{ for all }\\;x\\in\\B_{\\hat\\gamma}(\\ox).\n\\end{equation}\nClearly,  the uniform quadratic growth condition \\eqref{eq14} for $\\lm=\\olm$ indicates  that  $S_\\rho(\\olm)\\cap \\B_{\\gamma}(\\ox)=\\{\\ox\\}$ for all $\\rho\\ge\\bar\\rho$. \n","sha256_utf8":"614846a44c05ddb57eb364ff12c7bd6f7addf28eff904ca838848652d94a34cc"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"选取平静常数并取得最优解","source_units":[{"start":61296,"end":61977,"start_line":887,"end_line":898,"raw_tex":"Define   the positive constant\n\\begin{equation}\\label{eq21}\n\\tau:=\\frac{\\kappa_\\Phi}{\\ell}+\\sqrt{\\frac{\\kappa_\\Phi^2}{\\ell^2}+\\frac{1}{\\ell\\bar\\rho}},\n\\end{equation}\nand fix   $\\lambda\\in \\B_{\\hat\\gamma/2\\tau} (\\olm)$ and $\\rho\\geq\\bar\\rho$.\nObserve from the classical Weierstrass theorem that $S_\\rho(\\lm)\\neq\\emp$.   Assume that  $u\\in S_\\rho(\\lambda)$.\nBy  Proposition~\\ref{paug}(b),   the function $\\lambda\\mapsto\\L(u,\\lambda,\\rho)$ is concave. \nBy \\cite[Theorem~2.26(b)]{rw}, we have \n\\begin{equation}\\label{ueq19}\n\\nabla_\\lm\\L(u, \\lm, \\rho)=\\rho^{-1}\\big(\\nabla\\menv\\big(\\Phi(u)+\\rho^{-1}{\\lm}\\big) - \\lm\\big)\n= \\Phi(u)-\\prox \\big(\\Phi(u)+ \\rho^{-1}\\lm\\big),\n\\end{equation} \n","sha256_utf8":"dae55ffbf71559bd4d1ed2c94c6b764d0efe7e968400cf2b1cbbcdf1a75e4495"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"乘子凹性控制原始距离","source_units":[{"start":61977,"end":62892,"start_line":899,"end_line":912,"raw_tex":" which in turn yields  the relationships\n\\begin{eqnarray*} \n\\L(u,\\lambda,\\rho)&\\ge&\\L(u,\\olm,\\rho)-\\langle\\nabla_\\lambda\\L(u,\\lambda,\\rho),\\olm-\\lambda\\rangle\\nonumber\\\\\n&=&\\L(u,\\olm,\\rho)-\\big\\langle \\Phi(u)-\\prox\\big(\\Phi(u)+ \\rho^{-1}{\\lm}\\big),\\olm-\\lambda\\big\\rangle\\nonumber\\\\\n&\\ge& \\ph(\\ox)+g(\\Phi(\\ox))+\\ell\\Vert u-\\ox\\Vert^2-\\big\\langle \\Phi(u)-\\prox\\big(\\Phi(u)+ \\rho^{-1}{\\lm} \\big),\\olm-\\lambda\\big\\rangle\n\\end{eqnarray*}\nwith the last inequality resulting from  \\eqref{eq16}. We conclude from  $u\\in S_\\rho(\\lambda)$ that \n\\begin{equation*}\n\\L(u,\\lambda,\\rho)\\le\\L(\\ox,\\lambda,\\rho)=\\ph(\\ox)+\\menv\\big(\\Phi(\\ox)+\\rho^{-1}{\\lm}\\big) - \\sm \\rho^{-1}{\\|\\lm\\|^2}\\leq \\ph(\\ox)+g \\big(\\Phi(\\ox) \\big).\n\\end{equation*}\nCombining these brings us to \n\\begin{equation}\\label{eq20}\n\\Vert u-\\ox\\Vert^2\\le\\frac{1}{\\ell}\\big\\langle \\Phi(u)-\\prox\\big(\\Phi(u)+\\rho^{-1}\\lm\\big),\\olm-\\lambda\\big\\rangle.\n\\end{equation}\n","sha256_utf8":"777e13a0470ef528bbd913669b545818357a6ffa9ec4fde3d508f139637bad2b"}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","title":"近端非扩张控制乘子扰动","source_units":[{"start":62892,"end":63889,"start_line":913,"end_line":924,"raw_tex":"By the first equality in \\eqref{prm}, the inclusion  $\\olm \\in \\partial g\\big(\\Phi(\\ox)\\big)$ amounts to $\\Phi(\\ox) = \\prox\\big(\\Phi(\\ox)+ \\rho^{-1}\\olm\\big)$. This leads us to \n\\begin{eqnarray}\\label{clq7}\n&&\\big\\Vert\\Phi(u)-\\prox\\big(\\Phi(u)+\\rho^{-1}\\lm\\big)\\big\\Vert \\nonumber\\\\\n&=&\\big\\Vert\\Phi(u)-\\Phi(\\ox)+\\prox\\big(\\Phi(\\ox)+\\rho^{-1}\\olm\\big)-\\prox\\big(\\Phi(u)+\\rho^{-1}\\lm\\big)\\big\\Vert\\nonumber\\\\\n&\\le&2\\Vert\\Phi(u)-\\Phi(\\ox)\\Vert+\\rho^{-1}{\\Vert\\olm-\\lambda\\Vert}\\nonumber\\\\\n&\\le&2\\kappa_\\Phi\\Vert u-\\ox\\Vert+\\rho^{-1}{\\Vert\\olm-\\lambda\\Vert},\n\\end{eqnarray}\nwhere the first inequality comes from  the fact that proximal mapping of a convex function is always nonexpansive (cf.  \\cite[Proposition~12.19]{rw}), \nand where the last inequality results from \\eqref{eq49}. This, combined with  \\eqref{eq20},  brings us   to \n\\begin{equation*}\n\\Vert u-\\ox\\Vert^2\\le\\frac{1}{\\ell}\\Big(2\\kappa_\\Phi\\Vert u-\\ox\\Vert+\\rho^{-1}{\\Vert\\lambda-\\olm\\Vert}\\Big)\\Vert\\lambda-\\olm\\Vert.\n\\end{equation*}\n","sha256_utf8":"759d50ddd0569b28033105989a702615eac0c862789fae5fdd7282762a4af12c"}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","title":"解二次不等式取得内点性","source_units":[{"start":63889,"end":64606,"start_line":925,"end_line":934,"raw_tex":"Since the latter inequality  can be expressed  equivalently as \n\\begin{equation*}\n\\ell\\Vert u-\\ox\\Vert^2-2\\kappa_\\Phi\\Vert \\lambda-\\olm\\Vert\\cdot\\Vert u-\\ox\\Vert-\\rho^{-1}{\\Vert \\lambda-\\olm\\Vert^2 }\\le 0,\n\\end{equation*}\nwe arrive at  the estimate\n\\begin{equation*}\n\\Vert u-\\ox\\Vert\\le\\left(\\dfrac{\\kappa_\\Phi}{\\ell}+\\sqrt{\\frac{\\kappa_\\Phi^2}{\\ell^2}+\\frac{1}{\\ell\\rho}}\\right)\\Vert\\lambda-\\olm\\Vert\\le\\tau\\Vert\\lambda-\\olm\\Vert \\le \\tau \\hat\\gamma/2\\tau < \\hat\\gamma.\n\\end{equation*}\nThis clearly justifies   \\eqref{pt9}   and  the inclusion $S_\\rho(\\lambda)\\subset  \\inte  \\B_{\\hat\\gamma}(\\ox)$ for all  $\\lambda\\in \\B_{\\hat\\gamma/2\\tau} (\\olm)$ and $\\rho\\geq\\bar\\rho$,  and hence completes the proof.\n\\end{proof}","sha256_utf8":"330a7dbaf455be301a2522e3f49899d819a630ec90b004f7899b7360a9de52df"}]},{"id":"S073","semantic_type":"proof","kind":"proof","title":"任意邻近真乘子的估计推导","source_units":[{"start":64609,"end":65460,"start_line":936,"end_line":948,"raw_tex":"\\begin{Remark}{\\rm Assume that  the SOSC \\eqref{sosc} holds at $(\\ox,\\olm)$. It is not hard to see that a similar argument as those for \\eqref{eq20}\nshows that for any $\\lm\\in \\R^m$, any $u\\in S_\\rho(\\lm)$,  any $\\mu \\in \\Lm(\\ox)\\cap \\B_\\ve(\\olm)$, and any $\\rho\\ge \\bar\\rho$ we have \n\\begin{equation*} \n\\Vert u-\\ox\\Vert^2\\le\\frac{1}{\\ell}\\big\\langle \\Phi(u)-\\prox\\big(\\Phi(u)+\\rho^{-1}\\lm\\big),\\mu-\\lambda\\big\\rangle,\n\\end{equation*}\nwhere constants $\\bar\\rho$, $\\ve$, and $\\ell$ come from Theorem~\\ref{ugrowth}.\nCombining this and the second equality in \\eqref{ueq19} brings us to \n\\begin{equation}\\label{ueq20}\n\\Vert u-\\ox\\Vert^2\\le\\frac{1}{\\ell\\rho}\\big\\langle \\nabla e_{1/\\rho}\\big(\\Phi(u)+\\rho^{-1}\\lm\\big)-\\lm,\\mu-\\lambda\\big\\rangle,\n\\end{equation}\nwhich will be utilized in the proof of Theorem~\\ref{est} later in this section.\n}\n\\end{Remark}\n","sha256_utf8":"59f40b05f3eae17aea1301094e9ce8e2f2fbeaf2b89a20548b3ebf24dcc65135"}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","title":"统一增长替换基准乘子","source_units":[{"start":64609,"end":65123,"start_line":936,"end_line":941,"raw_tex":"\\begin{Remark}{\\rm Assume that  the SOSC \\eqref{sosc} holds at $(\\ox,\\olm)$. It is not hard to see that a similar argument as those for \\eqref{eq20}\nshows that for any $\\lm\\in \\R^m$, any $u\\in S_\\rho(\\lm)$,  any $\\mu \\in \\Lm(\\ox)\\cap \\B_\\ve(\\olm)$, and any $\\rho\\ge \\bar\\rho$ we have \n\\begin{equation*} \n\\Vert u-\\ox\\Vert^2\\le\\frac{1}{\\ell}\\big\\langle \\Phi(u)-\\prox\\big(\\Phi(u)+\\rho^{-1}\\lm\\big),\\mu-\\lambda\\big\\rangle,\n\\end{equation*}\nwhere constants $\\bar\\rho$, $\\ve$, and $\\ell$ come from Theorem~\\ref{ugrowth}.\n","sha256_utf8":"b72210f72e4491610948bffd114401b8a6e9aa661511bd0a93322ffa13e5e418"}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","title":"转写为包络梯度内积估计","source_units":[{"start":65123,"end":65460,"start_line":942,"end_line":948,"raw_tex":"Combining this and the second equality in \\eqref{ueq19} brings us to \n\\begin{equation}\\label{ueq20}\n\\Vert u-\\ox\\Vert^2\\le\\frac{1}{\\ell\\rho}\\big\\langle \\nabla e_{1/\\rho}\\big(\\Phi(u)+\\rho^{-1}\\lm\\big)-\\lm,\\mu-\\lambda\\big\\rangle,\n\\end{equation}\nwhich will be utilized in the proof of Theorem~\\ref{est} later in this section.\n}\n\\end{Remark}\n","sha256_utf8":"960fbafff9b3558f6192e165b6f539c0d043701c5a5f1840b9839fd587f4bda5"}]},{"id":"S076","semantic_type":"proposition","kind":"proposition","title":"残差的局部上界","source_units":[{"start":66893,"end":67340,"start_line":967,"end_line":972,"raw_tex":"\\begin{Proposition}[estimate for residual functions] \\label{erct}  Assume that $(\\ox,\\olm)$ is a solution to the KKT system \\eqref{vs} with $\\Th$ taken from \\eqref{Th}. Then \nthere exist positive constants $\\gamma_2$ and $\\kappa_2$ such that\n\\begin{equation}\\label{eb1}\nR(x, \\lambda) \\leq \\kappa_2\\big(\\|x-\\ox\\|+\\dist(\\lambda, \\Lambda(\\ox))\\big)\\quad\\textrm{for all }\\;\\; (x, \\lambda)\\in \\B_{\\gamma_2}(\\ox, \\olm).\n\\end{equation} \n\\end{Proposition}","sha256_utf8":"dfcd47912be3b0503359236e28d38df76af54ed6b70b2d30da56906460755769"}]},{"id":"S077","semantic_type":"proof","kind":"proof","title":"残差上界完整证明","source_units":[{"start":67342,"end":68345,"start_line":974,"end_line":985,"raw_tex":"\\begin{proof} By \\eqref{Th},  we have $N_\\Th(x) = N_\\Th(\\ox)$ for all $x\\in \\Th$. Let $\\mu=P_{\\Lm(\\ox)}(\\lm)$\nand observe that  $(\\ox,\\mu)$ is a solution to the KKT system \\eqref{vs}. So, we know from \\eqref{prm} that    $\\mu \\in \\partial g \\big(\\Phi(\\ox)\\big)$ yields   $\\Phi(\\ox) = {\\rm prox}_g\\big(\\Phi(\\ox)+ \\mu\\big)$.\nUsing these tells us that \n\\begin{eqnarray*}\nR(x, \\lambda)&=& \\dist\\big(-\\nabla_x L(x, \\lm), N_\\Th(x)\\big)+\\big\\|\\Phi(x)-{\\rm{prox}}_g\\big(\\Phi(x)+\\lm\\big)\\big\\|\\\\\n&\\le & \\|\\nabla_x L(x, \\lm)-\\nabla_x L(\\ox, \\olm)  \\|+  \\dist\\big(-\\nabla_x L(\\ox, \\olm), N_\\Th(\\ox)\\big)\\\\\n&&+  \\|\\Phi(x)-\\Phi(\\ox)\\|+ \\big\\|{\\rm{prox}}_g\\big(\\Phi(x)+\\lm\\big)-{\\rm{prox}}_g\\big(\\Phi(\\ox)+\\mu\\big)\\big\\|\\\\\n&\\le &\\|\\nabla_x L(x, \\lm)-\\nabla_x L(\\ox, \\olm)  \\| +  2\\|\\Phi(x)-\\Phi(\\ox)\\|+ \\dist(\\lambda, \\Lambda(\\ox)).\n\\end{eqnarray*}\nThis, together with \\eqref{eq49},  ensures the existence of positive \nconstants  $\\gamma_2$ and $\\kappa_2$ for which the estimate \\eqref{eb1} is satisfied. \n\\end{proof}","sha256_utf8":"8b084850e6d3159bc030fbad7a9ad4a76167e88a0119ca6d358946efdd19cf0e"}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","title":"恒定法空间与乘子投影","source_units":[{"start":67342,"end":67692,"start_line":974,"end_line":976,"raw_tex":"\\begin{proof} By \\eqref{Th},  we have $N_\\Th(x) = N_\\Th(\\ox)$ for all $x\\in \\Th$. Let $\\mu=P_{\\Lm(\\ox)}(\\lm)$\nand observe that  $(\\ox,\\mu)$ is a solution to the KKT system \\eqref{vs}. So, we know from \\eqref{prm} that    $\\mu \\in \\partial g \\big(\\Phi(\\ox)\\big)$ yields   $\\Phi(\\ox) = {\\rm prox}_g\\big(\\Phi(\\ox)+ \\mu\\big)$.\nUsing these tells us that \n","sha256_utf8":"7b9d459601763f9b431edab26975c058b141910d44e6ba0988699c37a4315837"}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","title":"非扩张性及光滑界控制残差","source_units":[{"start":67692,"end":68345,"start_line":977,"end_line":985,"raw_tex":"\\begin{eqnarray*}\nR(x, \\lambda)&=& \\dist\\big(-\\nabla_x L(x, \\lm), N_\\Th(x)\\big)+\\big\\|\\Phi(x)-{\\rm{prox}}_g\\big(\\Phi(x)+\\lm\\big)\\big\\|\\\\\n&\\le & \\|\\nabla_x L(x, \\lm)-\\nabla_x L(\\ox, \\olm)  \\|+  \\dist\\big(-\\nabla_x L(\\ox, \\olm), N_\\Th(\\ox)\\big)\\\\\n&&+  \\|\\Phi(x)-\\Phi(\\ox)\\|+ \\big\\|{\\rm{prox}}_g\\big(\\Phi(x)+\\lm\\big)-{\\rm{prox}}_g\\big(\\Phi(\\ox)+\\mu\\big)\\big\\|\\\\\n&\\le &\\|\\nabla_x L(x, \\lm)-\\nabla_x L(\\ox, \\olm)  \\| +  2\\|\\Phi(x)-\\Phi(\\ox)\\|+ \\dist(\\lambda, \\Lambda(\\ox)).\n\\end{eqnarray*}\nThis, together with \\eqref{eq49},  ensures the existence of positive \nconstants  $\\gamma_2$ and $\\kappa_2$ for which the estimate \\eqref{eb1} is satisfied. \n\\end{proof}","sha256_utf8":"116d603f70846834cacab664ffa05d4cc72cb105419121d16727ab6c3626dfc5"}]},{"id":"S080","semantic_type":"theorem","kind":"theorem","title":"连续迭代的残差误差界","source_units":[{"start":68519,"end":69447,"start_line":990,"end_line":1001,"raw_tex":"\\begin{Theorem}[error bound for consecutive iterates of ALM]\\label{est} \nLet $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and  let the SOSC \\eqref{sosc} hold at $(\\ox,\\olm)$. Then there exist positive constants   $\\gamma_3$, and $\\kappa_3$ such that for any $\\rho\\geq \\bar\\rho$ with \n$\\bar\\rho$ taken from Theorem~{\\rm\\ref{ugrowth}}, any $(x,\\lambda)\\in\\B_{\\gamma_3}(\\ox,\\olm)$ with $x\\in \\Th$ and  $R(x,\\lambda)>0$, and any   optimal solution $u$ to the regularized problem \n\\begin{equation}\\label{regp}\n\\mini \\L(w, \\lm, \\rho)\\quad \\mbox{subject to}\\quad w\\in \\Th\\cap \\B_{\\hat\\gg}(\\ox)\n\\end{equation} \nwith $\\hat\\gg$ taken from Proposition~{\\rm\\ref{solv}},   the error bound estimate\n\\begin{equation}\\label{est0}\n\\Vert u-x\\Vert+\\big\\Vert\\nabla\\menv\\big(\\Phi(u)+ \\rho^{-1}{\\lm}\\big)-\\lm\\big\\Vert\\le\\kappa_3\\,R(x,\\lambda)\n\\end{equation}\nholds, where the residual function  $R$ is  taken from \\eqref{res}.\n\\end{Theorem}","sha256_utf8":"d080f766bd7bd341a940424a1fdc1303fdbe779a1716c60d6054ac7d83f9c3b9"}]},{"id":"S081","semantic_type":"proof","kind":"proof","title":"连续迭代误差界完整证明","source_units":[{"start":69450,"end":80474,"start_line":1004,"end_line":1147,"raw_tex":"\\begin{proof} It follows from the SOSC \\eqref{sosc} and    Theorem~\\ref{ugrowth} that there are positive  constants $\\bar\\rho$, $\\gamma$, $\\varepsilon$, and $\\ell$ for which \nthe uniform quadratic  growth condition \\eqref{eq14} holds  for all $\\rho\\ge\\bar\\rho$ and all $\\lambda\\in\\Lambda(\\ox)\\cap\\B_\\varepsilon(\\olm)$. \nAccording to Proposition~\\ref{solv}, the solution mapping to \\eqref{regp}, denoted by $S_\\rho(\\lm)$ therein, satisfies the uniform isolated calmness \\eqref{pt9} and the condition $S_\\rho(\\lm)\\subset  \\inte  \\B_{\\hat\\gamma}(\\ox)$ for all $\\lm\\in \\B_{\\hat\\gamma/2\\tau}(\\olm)$ and all $\\rho\\ge \\bar\\rho$, where both positive constants $\\tau$ and $\\hat\\gg$ are taken from this proposition. This, in particular, tells us that for every $\\lm\\in \\B_{\\hat\\gamma/2\\tau}(\\olm)$ and every $\\rho\\ge \\bar\\rho$\nany optimal solution $u$ to \\eqref{regp} satisfies the first-order optimality condition \n\\begin{equation}\\label{optc}\n0\\in \\nabla_x\\L(u, \\lm, \\rho)+ N_\\Th(u).\n\\end{equation}\nAfter this presentation, assume by contradiction that the error bound estimate \\eqref{est0} fails. The latter implies that  there exists a sequence $\\{(\\xk, \\lk, \\rok)\\b\\subset \\Th \\times \\R^m\\times [\\bar\\rho, \\infty)$ \nwith $\\xk\\to\\ox, \\lk\\to\\olm$ as $k\\to\\infty$ such that\n\\begin{equation}\\label{est1}\n\\Vert u^k-\\xk\\Vert+\\| d^k-\\lk\\|>kR_k\\quad \\mbox{with}\\;\\;d^k:=\\nabla\\menvk\\big(\\Phi(u^k)+\\rho_k^{-1}\\lk\\big),\n\\end{equation} \nwhere $u^k$ is an optimal solution to \\eqref{regp} for $(\\lambda, \\rho):=(\\lk, \\rok)$, and where $R_k:=R(\\xk, \\lk)$ for each $k \\in \\mathbb{N}$. Denote by $\\beta_k$ the left-hand side of \\eqref{est1}, and thus get $R_k=o(\\beta_k)$.\n By the definition of $R_k$, the latter yields\n\\begin{equation}\\label{est2}\n-\\nabla_xL(\\xk, \\lk)+o(\\xik)\\in N_\\Th(\\xk)\\quad\\textrm{and}\\quad \\Phi(\\xk)+o(\\xik)= {\\rm{prox}}_g\\big(\\Phi (\\xk)+\\lk\\big).\n\\end{equation}\nPassing to a subsequence if necessary, we can find $(\\xi,\\eta)\\in \\R^n\\times \\R^m$ such that \n\\begin{equation}\\label{est4}\n\\dfrac{u^k-\\xk}{\\xik}\\to \\xi\\quad \\mbox{and}\\quad  \\dfrac{ d^k-\\lk}{\\xik} \\to  \\eta \\quad \\mbox{with}\\;\\; (\\xi,\\eta)\\neq 0.\n\\end{equation}\nWe know from Theorem~\\ref{sooc} that the error bound \\eqref{subr3} holds for all $(x,\\lm)$ sufficiently close to  $(\\ox, \\olm)$. \nSet $\\mu^k:=P_{\\Lambda(\\ox)}(\\lk)$ and remember  that $(\\xk, \\lk)\\to(\\ox, \\olm)$.  By  \\eqref{subr3}, we can assume without loss of generality that $\\xk-\\ox=O(R_k)$ and $\\lk-\\mu^k=O(R_k)$ for all $k\\in \\N$, which in turn results in\n\\begin{equation}\\label{est3}\n\\xk-\\ox=o(\\xik)\\quad\\textrm{and}\\quad\\lk-\\mu^k=o(\\xik)\\quad\\textrm{as } k\\to\\infty.\n\\end{equation}\nThe latter along with $\\lk\\to \\olm$ tells us that $\\mu^k\\to \\olm$. So, we get   $\\mu^k\\in\\Lambda(\\ox)\\cap\\B_\\varepsilon(\\olm)$ for all $k $ sufficiently large. \nThis, combined with  $\\rho_k\\ge\\bar\\rho$ and \\eqref{ueq20}, implies that \n\\begin{eqnarray}\\label{est5}\n\\Vert u^k-\\ox\\Vert^2&\\le&\\dfrac{1}{\\rok\\ell} \\langle d^k-\\lk,\\mu^k-\\lk \\rangle\\le\\dfrac{1}{\\rok\\ell}\\| d^k-\\lk\\|\\cdot\\Vert\\mu^k-\\lk\\Vert,\n\\end{eqnarray}\nwhich together with \\eqref{est4} and  \\eqref{est3} results in \n\\begin{equation*}\n\\dfrac{\\Vert u^k-\\ox\\Vert^2}{\\xik^2}\\le\\dfrac{1}{\\rok\\ell}\\dfrac{\\| d^k-\\lk\\|}{\\xik}\\cdot\\dfrac{\\Vert\\mu^k-\\lk\\Vert}{\\xik}\\to 0\\quad\\textrm{ as }\\;k\\to\\infty.\n\\end{equation*}\nThus, we get $u^k-\\ox=o(\\beta_k)$. Combining this with \\eqref{est3} clearly shows that \n\\begin{equation*}\n\\xi=\\lim_{k\\to\\infty}\\dfrac{u^k-\\xk}{\\xik}=\\lim_{k\\to\\infty}\\dfrac{u^k-\\ox}{\\xik}-\\lim_{k\\to\\infty}\\dfrac{\\xk-\\ox}{\\xik}=0-0=0.\n\\end{equation*}\nIf either the sequence $\\{\\rok\\b$ or  $\\{{\\rok}/{\\xik}\\b$ is bounded, it follows from $u^k-\\ox=o(\\beta_k)$ that  $\\dfrac{\\rok}{\\xik}\\Vert u^k-\\ox\\Vert\\to0$ as $k\\to\\infty$. \nThis, \\eqref{ueq19}, and a similar argument as those for \\eqref{clq7} bring us to \n\\begin{eqnarray*}\n \\dfrac{ \\|d^k-\\lk\\|}{\\xik}\n&=&\\frac{\\rok}{\\xik} \\big\\Vert \\Phi(u^k)-\\proxk\\big(\\Phi(u^k)+\\rho_k^{-1}\\lk\\big)\\big\\Vert\\\\\n&\\leq& 2\\kappa_\\Phi\\frac{\\rok}{\\xik}\\|u^k-\\ox\\|+\\dfrac{\\Vert \\mu^k-\\lk\\Vert }{\\xik}\\to0\\quad\\textrm{as } k\\to\\infty,  \n\\end{eqnarray*} \nwhich by \\eqref{est4} yields $\\eta =0$. This is  a contradiction with \\eqref{est4} since we showed that $(\\xi,\\eta)=0$.\n\nAssume now that both sequences $\\{\\rok\\b$ and  $\\{{\\rok}/{\\xik}\\b$  are unbounded.  We can assume by passing  to a subsequence if necessary that \n\\begin{equation}\\label{unb}\n\\rok\\to \\infty\\quad \\mbox{and}\\quad \\frac{\\rok}{\\beta_k}\\to \\infty\\;\\;\\mbox{as}\\;\\; k\\to \\infty.\n\\end{equation}\nSince $u^k$ is an optimal solution to \\eqref{regp} associated with $(\\lk,\\rok)$, we deduce from \\eqref{optc} that \n\\begin{equation*}\n0\\in \\nabla_x\\L(u^k, \\lk, \\rok)+ N_\\Th(u^k).\n\\end{equation*} \nBy \\eqref{Th}, we get  $N_{\\Th}(u^k)=N_\\Th(\\xk) = \\rge B^*$. These along with  \\eqref{est2} tell us that \n\\begin{equation*}\n\\nabla_x\\L(u^k, \\lk, \\rok)-\\nabla_xL(\\xk, \\lk)+o(\\xik)\\in N_\\Th(\\xk)-N_\\Th(u^k)= \\rge B^*.\n\\end{equation*}\nUsing the definitions of the augmented Lagrangian $\\L$ and  $L$, we get \n\\begin{eqnarray*}\n\\rge B^*&\\ni& \\nabla_x\\L(u^k, \\lk, \\rok)-\\nabla_xL(\\xk, \\lk)+o(\\xik)\\\\\n&=&\\nabla \\varphi(u^k)-\\nabla \\varphi(\\xk)+\\nabla\\Phi(u^k)^* \\nabla\\menvk\\big(\\Phi(u^k)+\\rho_k^{-1}\\lk\\big)-\\nabla\\Phi(\\xk)^*\\lk+o(\\xik)\\\\\n&=&\\big(\\nabla\\Phi(u^k)-\\nabla\\Phi(\\xk)\\big) ^*\\lk+\\nabla\\Phi(u^k)^*\\big (d^k-\\lk\\big )+o(\\xik)\\\\\n&=&\\nabla\\Phi(u^k)^* \\big(d^k-\\lk\\big)+o(\\xik),\n\\end{eqnarray*}\nwhere the second and last equalities result from  the Lipschitz continuity of  $\\nabla \\varphi$ and $\\nabla \\Phi$ around $\\ox$ and from  the fact that $u^k-\\xk=o(\\xik)$. \nDividing both sides  by $\\xik$ and then letting  $k\\to\\infty$ confirm via \\eqref{est4} that \n\\begin{equation*}\n\\nabla\\Phi(\\ox)^*\\eta \\in\\rge B^*.\n\\end{equation*} \nWe are going to show that $\\eta=0$, which contradicts \\eqref{est4} and thus completes the proof. To furnish this, set $E:=\\big\\{z \\in \\R^n\\, |\\, \\nabla \\Phi(\\ox)^*z\\in \\rge B^*\\big\\}$\nand observe from \\cite[Corollary~11.25(d)]{rw} that $E^\\perp=\\big\\{\\nabla\\Phi(\\ox)y|\\;y\\in \\ker B\\big\\}$. So, we conclude  from  $\\ox,u^k\\in \\Th$ and  \\eqref{Th} that $\\nabla\\Phi(\\ox)(u^k-\\ox)\\in E^\\perp$.\nMoreover, we obtain  from \\eqref{est4}-\\eqref{est5} that \n\\begin{equation}\\label{est6}\n\\dfrac{\\rok}{\\xik}\\Vert u^k-\\ox\\Vert^2\\leq\\dfrac{1}{\\ell} \\dfrac{\\| d^k-\\lk\\|}{\\xik} \\cdot\\Vert \\mu^k-\\lk\\Vert  \\to 0 \\quad\\textrm{ as }k\\to\\infty.\n\\end{equation} \nSince $P_E$ is a linear mapping due to $E$ being a subspace, we arrive at \n\\begin{eqnarray}\\label{est7}\nP_E\\Big(\\dfrac{\\rok}{\\xik}\\big (\\Phi(u^k)-\\Phi(\\ox)\\big )\\Big)&=&P_E\\Big (\\dfrac{\\rok}{\\xik}\\nabla\\Phi(\\ox) (u^k-\\ox )+O\\big (\\dfrac{\\rok}{\\xik}\\Vert u^k-\\ox\\Vert^2 \\big)\\Big  )\\nonumber\\\\\n&=&P_E \\Big(O\\big(\\dfrac{\\rok}{\\xik}\\Vert u^k-\\ox\\Vert^2 \\big )\\Big)\\to 0 \\quad \\textrm{as } k\\to\\infty,\n\\end{eqnarray} \nwhere the last equality results from $\\nabla\\Phi(\\ox)(u^k-\\ox)\\in E^\\perp$.\nFix $k\\in \\N$ and set  \n$\nz^k:= \\Phi(u^k)+ \\rho_k^{-1}\\big(\\lk -d^k\\big).\n$\nIt follows from $d^k=\\nabla\\menvk\\big(\\Phi(u^k)+ \\rho_k^{-1}{\\lk}\\big)$ and the second equation in \\eqref{prm} that \n\\begin{equation}\\label{clq8}\nd^k \\in \\partial g(z^k).\n\\end{equation} \nUsing  \\eqref{est4} and \\eqref{unb} allows us to arrive at \n\\begin{equation*}\nz^k =\\Phi(u^k) - \\frac{d^k-\\lk}{\\xik}\\cdot\\frac{\\xik}{\\rok} \\to \\Phi(\\ox)=:\\bar z\\quad\\textrm{as }\\; k\\to\\infty.\n\\end{equation*} \nBy \\eqref{clq8}, we get $z^k\\in \\dom g=\\cup_{i=1}^s C_i$ for all $k$, where the polyhedral convex sets $C_i$ are taken from \\eqref{PWLQ}. \nBy passing to a subsequence if necessary, we can assume without loss of generality that  for some $\\oi\\in I(\\oz)$, we have $z^k \\in C_\\oi$ for all $k$, where the index set  $  I(\\bar z)$  is defined by \\eqref{act}.\nSince $C_\\oi$ is  polyhedral, we find   $l_\\oi\\in \\N$, $c_{\\oi, j}\\in \\R^m$, and $\\al_{\\oi, j}\\in \\R$,  $j=1,\\ldots,l_\\oi$,  such that \n\\begin{equation*}\nC_\\oi= \\big\\{z\\in \\R^m\\, |\\, \\langle c_{\\oi, j}, z\\rangle \\leq \\al_{\\oi, j}, \\quad j= 1, \\ldots, l_\\oi\\big\\}.\n\\end{equation*} \nDenote by $J(\\bar z)$ and $J(z^k)$ the sets of the indexes of active inequalities in $C_\\oi$ at $\\bar z$ and $z^k$, respectively. Since $z^k\\to \\oz$, we obtain  $J(z^k)\\subset J(\\bar z)$ for all $k$ sufficiently large. \nPassing to a subsequence, if necessary, we can assume that there exists a subset $\\bar J\\subset J(\\bar z)$ such that $J(z^k)=\\bar J$ for all $k$. Thus, we have \n\\begin{equation*}\nz^k -\\bar z \\in \\big\\{v\\in \\R^m\\, \\big|\\, \\langle c_{\\oi, j}, v\\rangle= 0\\;\\textrm{ for } \\;j \\in \\bar J, \\; \\langle c_{\\oi, j}, v\\rangle\\leq 0\\; \\textrm{ for }\\; j\\in J(\\bar z)\\setminus \\bar J \\big\\}=: \\Omega.\n\\end{equation*} \nIt follows from  the definitions of $z^k$ and $\\bar z$ that \n\\begin{equation*}\n\\Phi(u^k)-\\Phi(\\ox)-\\frac{ d^k-\\lk}{\\rok}=z^k-\\oz \\in \\Omega. \n\\end{equation*}\nMultiplying by $ {\\rok}/{\\xik}$ and using the linearity of  $P_E$  allow us to get \n\\begin{equation*}\nP_E\\Big(\\dfrac{\\rok}{\\xik}\\big (\\Phi(u^k)-\\Phi(\\ox)\\big )\\Big)- P_E \\big(\\frac{ d^k-\\lk}{\\xik}\\big) \\in P_E(\\Omega),\n\\end{equation*} \nwhich, together with \\eqref{est4}, \\eqref{est7}, and the closedness of $P_E(\\Omega)$, yields $-P_E(\\eta) \\in P_E(\\Omega)$. Since $\\eta\\in E$, the latter confirms that \n\\begin{equation}\\label{est8}\n-\\eta=-P_E(\\eta) = P_E(\\zeta)\\quad\\textrm{for some }\\;\\; \\zeta\\in \\Omega.\n\\end{equation}\nRemembering $\\mu^k\\in \\Lm(\\ox)$, we obtain $\\mu^k \\in \\partial g(\\bar z)$. By the representation of $\\partial g(\\bar z)$ from \\eqref{sub} and the fact that  $\\oi\\in I(\\bar z)$, we conclude that \n\\begin{equation*}\\label{clq9}\n\\mu^k - A_\\oi\\bar z -a_\\oi \\in N_{C_\\oi}(\\bar z) = \\cone\\big\\{c_{\\oi, j}\\, | \\;j\\in J(\\bar z)\\big\\}.\n\\end{equation*} \nSince we also have $\\oi\\in I(z^k)$, it follows again from \\eqref{clq8} and  \\eqref{sub} that \n\\begin{equation*}\\label{clq10}\nd^k -A_\\oi z^k-a_\\oi \\in N_{C_\\oi}(z^k) = \\cone\\big\\{c_{\\oi, j}\\,|\\; j\\in \\bar J\\big\\}.\n\\end{equation*} \nCombining these results in \n\\begin{eqnarray*}\n \\mu^k-d^k + A_\\oi(z^k-\\bar z)&\\in&   \\cone\\big\\{c_{\\oi, j}\\, | \\;j\\in J(\\bar z)\\big\\}-\\cone\\big\\{c_{\\oi, j}\\,|\\; j\\in \\bar J\\big\\}\\\\\n&=&\\span\\big\\{c_{\\oi, j}\\,|\\;  j\\in \\bar J\\big\\}+\\cone\\big\\{c_{\\oi, j}\\,|\\; j\\in J(\\bar z)\\setminus\\bar J\\big\\} =  \\Omega^*.\n\\end{eqnarray*} \nSince $\\lk -\\mu^k=o(\\xik)$ and $u^k-\\ox=o(\\beta_k)$, we deduce from \\eqref{est4} and \\eqref{unb} that \n\\begin{eqnarray*}\n\\frac{ \\mu^k-d^k}{\\beta_k}+A_\\oi\\big(\\frac{z^k-\\oz}{\\beta_k}\\big)&=& \\frac{ \\mu^k-\\lk}{\\beta_k}+\\frac{ \\lk-d^k}{\\beta_k} +A_\\oi\\big(\\frac{\\Phi(u^k)-\\Phi(\\ox)}{\\beta_k}\\big)\\\\\n&&-\\frac{1}{\\rok}A_\\oi\\big(\\frac{d^k-\\lk}{\\beta_k}\\big)\\to -\\eta \\;\\;\\;\\;\\mbox{as}\\;\\;k\\to \\infty,\n\\end{eqnarray*}\nwhich in turn demonstrates that \n$-\\eta \\in \\Omega^*$. This, combined with \\eqref{est8} and $\\eta\\in E$, shows that \n\\begin{equation*}\n\\|\\eta\\|^2= \\langle \\eta, \\eta \\rangle = \\langle \\eta, P_E(\\eta) \\rangle= -\\langle \\eta, P_E(\\zeta) \\rangle= -\\big\\langle \\eta, P_E(\\zeta)+P_{E^{\\perp}}(\\zeta) \\big\\rangle = -    \\langle \\eta, \\zeta\\rangle \\le 0,\n\\end{equation*} \nimplying that  $\\eta =0$. This clearly contradicts \\eqref{est4} and thus completes the proof.\n\\end{proof}","sha256_utf8":"7604770e3663f1dd6f1c001fdb63748bda5cda1fbeabd2f6be2d534f1458b5ff"}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","title":"统一增长与最优性条件","source_units":[{"start":69450,"end":70441,"start_line":1004,"end_line":1010,"raw_tex":"\\begin{proof} It follows from the SOSC \\eqref{sosc} and    Theorem~\\ref{ugrowth} that there are positive  constants $\\bar\\rho$, $\\gamma$, $\\varepsilon$, and $\\ell$ for which \nthe uniform quadratic  growth condition \\eqref{eq14} holds  for all $\\rho\\ge\\bar\\rho$ and all $\\lambda\\in\\Lambda(\\ox)\\cap\\B_\\varepsilon(\\olm)$. \nAccording to Proposition~\\ref{solv}, the solution mapping to \\eqref{regp}, denoted by $S_\\rho(\\lm)$ therein, satisfies the uniform isolated calmness \\eqref{pt9} and the condition $S_\\rho(\\lm)\\subset  \\inte  \\B_{\\hat\\gamma}(\\ox)$ for all $\\lm\\in \\B_{\\hat\\gamma/2\\tau}(\\olm)$ and all $\\rho\\ge \\bar\\rho$, where both positive constants $\\tau$ and $\\hat\\gg$ are taken from this proposition. This, in particular, tells us that for every $\\lm\\in \\B_{\\hat\\gamma/2\\tau}(\\olm)$ and every $\\rho\\ge \\bar\\rho$\nany optimal solution $u$ to \\eqref{regp} satisfies the first-order optimality condition \n\\begin{equation}\\label{optc}\n0\\in \\nabla_x\\L(u, \\lm, \\rho)+ N_\\Th(u).\n\\end{equation}\n","sha256_utf8":"7cb52c82c3e4aa591918ac9288a06ae9a58d712fbbb4608b0fb893190e790c67"}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","title":"反设失效并归一化反例","source_units":[{"start":70441,"end":71579,"start_line":1011,"end_line":1024,"raw_tex":"After this presentation, assume by contradiction that the error bound estimate \\eqref{est0} fails. The latter implies that  there exists a sequence $\\{(\\xk, \\lk, \\rok)\\b\\subset \\Th \\times \\R^m\\times [\\bar\\rho, \\infty)$ \nwith $\\xk\\to\\ox, \\lk\\to\\olm$ as $k\\to\\infty$ such that\n\\begin{equation}\\label{est1}\n\\Vert u^k-\\xk\\Vert+\\| d^k-\\lk\\|>kR_k\\quad \\mbox{with}\\;\\;d^k:=\\nabla\\menvk\\big(\\Phi(u^k)+\\rho_k^{-1}\\lk\\big),\n\\end{equation} \nwhere $u^k$ is an optimal solution to \\eqref{regp} for $(\\lambda, \\rho):=(\\lk, \\rok)$, and where $R_k:=R(\\xk, \\lk)$ for each $k \\in \\mathbb{N}$. Denote by $\\beta_k$ the left-hand side of \\eqref{est1}, and thus get $R_k=o(\\beta_k)$.\n By the definition of $R_k$, the latter yields\n\\begin{equation}\\label{est2}\n-\\nabla_xL(\\xk, \\lk)+o(\\xik)\\in N_\\Th(\\xk)\\quad\\textrm{and}\\quad \\Phi(\\xk)+o(\\xik)= {\\rm{prox}}_g\\big(\\Phi (\\xk)+\\lk\\big).\n\\end{equation}\nPassing to a subsequence if necessary, we can find $(\\xi,\\eta)\\in \\R^n\\times \\R^m$ such that \n\\begin{equation}\\label{est4}\n\\dfrac{u^k-\\xk}{\\xik}\\to \\xi\\quad \\mbox{and}\\quad  \\dfrac{ d^k-\\lk}{\\xik} \\to  \\eta \\quad \\mbox{with}\\;\\; (\\xi,\\eta)\\neq 0.\n\\end{equation}\n","sha256_utf8":"d363acfd6fb2154ffbcd10140d50fd5720c31aebe84a8857b787ff77b4a02994"}]},{"id":"S084","semantic_type":"proof_step","kind":"proof_step","title":"KKT误差界压低基点偏差","source_units":[{"start":71579,"end":72230,"start_line":1025,"end_line":1030,"raw_tex":"We know from Theorem~\\ref{sooc} that the error bound \\eqref{subr3} holds for all $(x,\\lm)$ sufficiently close to  $(\\ox, \\olm)$. \nSet $\\mu^k:=P_{\\Lambda(\\ox)}(\\lk)$ and remember  that $(\\xk, \\lk)\\to(\\ox, \\olm)$.  By  \\eqref{subr3}, we can assume without loss of generality that $\\xk-\\ox=O(R_k)$ and $\\lk-\\mu^k=O(R_k)$ for all $k\\in \\N$, which in turn results in\n\\begin{equation}\\label{est3}\n\\xk-\\ox=o(\\xik)\\quad\\textrm{and}\\quad\\lk-\\mu^k=o(\\xik)\\quad\\textrm{as } k\\to\\infty.\n\\end{equation}\nThe latter along with $\\lk\\to \\olm$ tells us that $\\mu^k\\to \\olm$. So, we get   $\\mu^k\\in\\Lambda(\\ox)\\cap\\B_\\varepsilon(\\olm)$ for all $k $ sufficiently large. \n","sha256_utf8":"167ce213e823c434391aea3580a0a82b3dc8a44cf543c491958b9495aae63073"}]},{"id":"S085","semantic_type":"proof_step","kind":"proof_step","title":"内积估计消去原始极限方向","source_units":[{"start":72230,"end":72992,"start_line":1031,"end_line":1042,"raw_tex":"This, combined with  $\\rho_k\\ge\\bar\\rho$ and \\eqref{ueq20}, implies that \n\\begin{eqnarray}\\label{est5}\n\\Vert u^k-\\ox\\Vert^2&\\le&\\dfrac{1}{\\rok\\ell} \\langle d^k-\\lk,\\mu^k-\\lk \\rangle\\le\\dfrac{1}{\\rok\\ell}\\| d^k-\\lk\\|\\cdot\\Vert\\mu^k-\\lk\\Vert,\n\\end{eqnarray}\nwhich together with \\eqref{est4} and  \\eqref{est3} results in \n\\begin{equation*}\n\\dfrac{\\Vert u^k-\\ox\\Vert^2}{\\xik^2}\\le\\dfrac{1}{\\rok\\ell}\\dfrac{\\| d^k-\\lk\\|}{\\xik}\\cdot\\dfrac{\\Vert\\mu^k-\\lk\\Vert}{\\xik}\\to 0\\quad\\textrm{ as }\\;k\\to\\infty.\n\\end{equation*}\nThus, we get $u^k-\\ox=o(\\beta_k)$. Combining this with \\eqref{est3} clearly shows that \n\\begin{equation*}\n\\xi=\\lim_{k\\to\\infty}\\dfrac{u^k-\\xk}{\\xik}=\\lim_{k\\to\\infty}\\dfrac{u^k-\\ox}{\\xik}-\\lim_{k\\to\\infty}\\dfrac{\\xk-\\ox}{\\xik}=0-0=0.\n\\end{equation*}\n","sha256_utf8":"20916e5ae0235ee071928457605d22864d4a7c062ead88c526f8fd613984a73f"}]},{"id":"S086","semantic_type":"proof_step","kind":"proof_step","title":"有界参数情形消去对偶方向","source_units":[{"start":72992,"end":73644,"start_line":1043,"end_line":1050,"raw_tex":"If either the sequence $\\{\\rok\\b$ or  $\\{{\\rok}/{\\xik}\\b$ is bounded, it follows from $u^k-\\ox=o(\\beta_k)$ that  $\\dfrac{\\rok}{\\xik}\\Vert u^k-\\ox\\Vert\\to0$ as $k\\to\\infty$. \nThis, \\eqref{ueq19}, and a similar argument as those for \\eqref{clq7} bring us to \n\\begin{eqnarray*}\n \\dfrac{ \\|d^k-\\lk\\|}{\\xik}\n&=&\\frac{\\rok}{\\xik} \\big\\Vert \\Phi(u^k)-\\proxk\\big(\\Phi(u^k)+\\rho_k^{-1}\\lk\\big)\\big\\Vert\\\\\n&\\leq& 2\\kappa_\\Phi\\frac{\\rok}{\\xik}\\|u^k-\\ox\\|+\\dfrac{\\Vert \\mu^k-\\lk\\Vert }{\\xik}\\to0\\quad\\textrm{as } k\\to\\infty,  \n\\end{eqnarray*} \nwhich by \\eqref{est4} yields $\\eta =0$. This is  a contradiction with \\eqref{est4} since we showed that $(\\xi,\\eta)=0$.\n","sha256_utf8":"45aaab9091059a5887a84b6225ff57d3b83c65f296153f45133d14480b6e7c85"}]},{"id":"S087","semantic_type":"proof_step","kind":"proof_step","title":"无界参数情形的法空间极限","source_units":[{"start":73645,"end":75154,"start_line":1052,"end_line":1075,"raw_tex":"Assume now that both sequences $\\{\\rok\\b$ and  $\\{{\\rok}/{\\xik}\\b$  are unbounded.  We can assume by passing  to a subsequence if necessary that \n\\begin{equation}\\label{unb}\n\\rok\\to \\infty\\quad \\mbox{and}\\quad \\frac{\\rok}{\\beta_k}\\to \\infty\\;\\;\\mbox{as}\\;\\; k\\to \\infty.\n\\end{equation}\nSince $u^k$ is an optimal solution to \\eqref{regp} associated with $(\\lk,\\rok)$, we deduce from \\eqref{optc} that \n\\begin{equation*}\n0\\in \\nabla_x\\L(u^k, \\lk, \\rok)+ N_\\Th(u^k).\n\\end{equation*} \nBy \\eqref{Th}, we get  $N_{\\Th}(u^k)=N_\\Th(\\xk) = \\rge B^*$. These along with  \\eqref{est2} tell us that \n\\begin{equation*}\n\\nabla_x\\L(u^k, \\lk, \\rok)-\\nabla_xL(\\xk, \\lk)+o(\\xik)\\in N_\\Th(\\xk)-N_\\Th(u^k)= \\rge B^*.\n\\end{equation*}\nUsing the definitions of the augmented Lagrangian $\\L$ and  $L$, we get \n\\begin{eqnarray*}\n\\rge B^*&\\ni& \\nabla_x\\L(u^k, \\lk, \\rok)-\\nabla_xL(\\xk, \\lk)+o(\\xik)\\\\\n&=&\\nabla \\varphi(u^k)-\\nabla \\varphi(\\xk)+\\nabla\\Phi(u^k)^* \\nabla\\menvk\\big(\\Phi(u^k)+\\rho_k^{-1}\\lk\\big)-\\nabla\\Phi(\\xk)^*\\lk+o(\\xik)\\\\\n&=&\\big(\\nabla\\Phi(u^k)-\\nabla\\Phi(\\xk)\\big) ^*\\lk+\\nabla\\Phi(u^k)^*\\big (d^k-\\lk\\big )+o(\\xik)\\\\\n&=&\\nabla\\Phi(u^k)^* \\big(d^k-\\lk\\big)+o(\\xik),\n\\end{eqnarray*}\nwhere the second and last equalities result from  the Lipschitz continuity of  $\\nabla \\varphi$ and $\\nabla \\Phi$ around $\\ox$ and from  the fact that $u^k-\\xk=o(\\xik)$. \nDividing both sides  by $\\xik$ and then letting  $k\\to\\infty$ confirm via \\eqref{est4} that \n\\begin{equation*}\n\\nabla\\Phi(\\ox)^*\\eta \\in\\rge B^*.\n\\end{equation*} \n","sha256_utf8":"51824e97b2752007649410860581f7c00982f417b9d50d1ee627590ac3d61b23"}]},{"id":"S088","semantic_type":"proof_step","kind":"proof_step","title":"正交子空间投影消去线性项","source_units":[{"start":75154,"end":76285,"start_line":1076,"end_line":1087,"raw_tex":"We are going to show that $\\eta=0$, which contradicts \\eqref{est4} and thus completes the proof. To furnish this, set $E:=\\big\\{z \\in \\R^n\\, |\\, \\nabla \\Phi(\\ox)^*z\\in \\rge B^*\\big\\}$\nand observe from \\cite[Corollary~11.25(d)]{rw} that $E^\\perp=\\big\\{\\nabla\\Phi(\\ox)y|\\;y\\in \\ker B\\big\\}$. So, we conclude  from  $\\ox,u^k\\in \\Th$ and  \\eqref{Th} that $\\nabla\\Phi(\\ox)(u^k-\\ox)\\in E^\\perp$.\nMoreover, we obtain  from \\eqref{est4}-\\eqref{est5} that \n\\begin{equation}\\label{est6}\n\\dfrac{\\rok}{\\xik}\\Vert u^k-\\ox\\Vert^2\\leq\\dfrac{1}{\\ell} \\dfrac{\\| d^k-\\lk\\|}{\\xik} \\cdot\\Vert \\mu^k-\\lk\\Vert  \\to 0 \\quad\\textrm{ as }k\\to\\infty.\n\\end{equation} \nSince $P_E$ is a linear mapping due to $E$ being a subspace, we arrive at \n\\begin{eqnarray}\\label{est7}\nP_E\\Big(\\dfrac{\\rok}{\\xik}\\big (\\Phi(u^k)-\\Phi(\\ox)\\big )\\Big)&=&P_E\\Big (\\dfrac{\\rok}{\\xik}\\nabla\\Phi(\\ox) (u^k-\\ox )+O\\big (\\dfrac{\\rok}{\\xik}\\Vert u^k-\\ox\\Vert^2 \\big)\\Big  )\\nonumber\\\\\n&=&P_E \\Big(O\\big(\\dfrac{\\rok}{\\xik}\\Vert u^k-\\ox\\Vert^2 \\big )\\Big)\\to 0 \\quad \\textrm{as } k\\to\\infty,\n\\end{eqnarray} \nwhere the last equality results from $\\nabla\\Phi(\\ox)(u^k-\\ox)\\in E^\\perp$.\n","sha256_utf8":"64b6bed94a4e6057717526d21049232fc8d07681d455891b825f6ad8d5c03ad1"}]},{"id":"S089","semantic_type":"proof_step","kind":"proof_step","title":"近端点收敛并固定二次片","source_units":[{"start":76285,"end":77110,"start_line":1088,"end_line":1101,"raw_tex":"Fix $k\\in \\N$ and set  \n$\nz^k:= \\Phi(u^k)+ \\rho_k^{-1}\\big(\\lk -d^k\\big).\n$\nIt follows from $d^k=\\nabla\\menvk\\big(\\Phi(u^k)+ \\rho_k^{-1}{\\lk}\\big)$ and the second equation in \\eqref{prm} that \n\\begin{equation}\\label{clq8}\nd^k \\in \\partial g(z^k).\n\\end{equation} \nUsing  \\eqref{est4} and \\eqref{unb} allows us to arrive at \n\\begin{equation*}\nz^k =\\Phi(u^k) - \\frac{d^k-\\lk}{\\xik}\\cdot\\frac{\\xik}{\\rok} \\to \\Phi(\\ox)=:\\bar z\\quad\\textrm{as }\\; k\\to\\infty.\n\\end{equation*} \nBy \\eqref{clq8}, we get $z^k\\in \\dom g=\\cup_{i=1}^s C_i$ for all $k$, where the polyhedral convex sets $C_i$ are taken from \\eqref{PWLQ}. \nBy passing to a subsequence if necessary, we can assume without loss of generality that  for some $\\oi\\in I(\\oz)$, we have $z^k \\in C_\\oi$ for all $k$, where the index set  $  I(\\bar z)$  is defined by \\eqref{act}.\n","sha256_utf8":"da7c8f08f00734841d0f47c03fb26f4c3bcfd818c5d91fdb8fd82c7aa808f227"}]},{"id":"S090","semantic_type":"proof_step","kind":"proof_step","title":"固定活跃面构造多面体锥","source_units":[{"start":77110,"end":78176,"start_line":1102,"end_line":1114,"raw_tex":"Since $C_\\oi$ is  polyhedral, we find   $l_\\oi\\in \\N$, $c_{\\oi, j}\\in \\R^m$, and $\\al_{\\oi, j}\\in \\R$,  $j=1,\\ldots,l_\\oi$,  such that \n\\begin{equation*}\nC_\\oi= \\big\\{z\\in \\R^m\\, |\\, \\langle c_{\\oi, j}, z\\rangle \\leq \\al_{\\oi, j}, \\quad j= 1, \\ldots, l_\\oi\\big\\}.\n\\end{equation*} \nDenote by $J(\\bar z)$ and $J(z^k)$ the sets of the indexes of active inequalities in $C_\\oi$ at $\\bar z$ and $z^k$, respectively. Since $z^k\\to \\oz$, we obtain  $J(z^k)\\subset J(\\bar z)$ for all $k$ sufficiently large. \nPassing to a subsequence, if necessary, we can assume that there exists a subset $\\bar J\\subset J(\\bar z)$ such that $J(z^k)=\\bar J$ for all $k$. Thus, we have \n\\begin{equation*}\nz^k -\\bar z \\in \\big\\{v\\in \\R^m\\, \\big|\\, \\langle c_{\\oi, j}, v\\rangle= 0\\;\\textrm{ for } \\;j \\in \\bar J, \\; \\langle c_{\\oi, j}, v\\rangle\\leq 0\\; \\textrm{ for }\\; j\\in J(\\bar z)\\setminus \\bar J \\big\\}=: \\Omega.\n\\end{equation*} \nIt follows from  the definitions of $z^k$ and $\\bar z$ that \n\\begin{equation*}\n\\Phi(u^k)-\\Phi(\\ox)-\\frac{ d^k-\\lk}{\\rok}=z^k-\\oz \\in \\Omega. \n\\end{equation*}\n","sha256_utf8":"8c53adaa2761e3a5f30b9e771e55758936740aa51de102d9b5c33b8bf5bf1d76"}]},{"id":"S091","semantic_type":"proof_step","kind":"proof_step","title":"投影极限给出锥中表示","source_units":[{"start":78176,"end":78698,"start_line":1115,"end_line":1122,"raw_tex":"Multiplying by $ {\\rok}/{\\xik}$ and using the linearity of  $P_E$  allow us to get \n\\begin{equation*}\nP_E\\Big(\\dfrac{\\rok}{\\xik}\\big (\\Phi(u^k)-\\Phi(\\ox)\\big )\\Big)- P_E \\big(\\frac{ d^k-\\lk}{\\xik}\\big) \\in P_E(\\Omega),\n\\end{equation*} \nwhich, together with \\eqref{est4}, \\eqref{est7}, and the closedness of $P_E(\\Omega)$, yields $-P_E(\\eta) \\in P_E(\\Omega)$. Since $\\eta\\in E$, the latter confirms that \n\\begin{equation}\\label{est8}\n-\\eta=-P_E(\\eta) = P_E(\\zeta)\\quad\\textrm{for some }\\;\\; \\zeta\\in \\Omega.\n\\end{equation}\n","sha256_utf8":"22f47d3e14b8b4c8d0e475170f4bc04eb0e3db94856bc01c71666a4b7ea98f78"}]},{"id":"S092","semantic_type":"proof_step","kind":"proof_step","title":"分片次微分给出极锥归属","source_units":[{"start":78698,"end":79593,"start_line":1123,"end_line":1135,"raw_tex":"Remembering $\\mu^k\\in \\Lm(\\ox)$, we obtain $\\mu^k \\in \\partial g(\\bar z)$. By the representation of $\\partial g(\\bar z)$ from \\eqref{sub} and the fact that  $\\oi\\in I(\\bar z)$, we conclude that \n\\begin{equation*}\\label{clq9}\n\\mu^k - A_\\oi\\bar z -a_\\oi \\in N_{C_\\oi}(\\bar z) = \\cone\\big\\{c_{\\oi, j}\\, | \\;j\\in J(\\bar z)\\big\\}.\n\\end{equation*} \nSince we also have $\\oi\\in I(z^k)$, it follows again from \\eqref{clq8} and  \\eqref{sub} that \n\\begin{equation*}\\label{clq10}\nd^k -A_\\oi z^k-a_\\oi \\in N_{C_\\oi}(z^k) = \\cone\\big\\{c_{\\oi, j}\\,|\\; j\\in \\bar J\\big\\}.\n\\end{equation*} \nCombining these results in \n\\begin{eqnarray*}\n \\mu^k-d^k + A_\\oi(z^k-\\bar z)&\\in&   \\cone\\big\\{c_{\\oi, j}\\, | \\;j\\in J(\\bar z)\\big\\}-\\cone\\big\\{c_{\\oi, j}\\,|\\; j\\in \\bar J\\big\\}\\\\\n&=&\\span\\big\\{c_{\\oi, j}\\,|\\;  j\\in \\bar J\\big\\}+\\cone\\big\\{c_{\\oi, j}\\,|\\; j\\in J(\\bar z)\\setminus\\bar J\\big\\} =  \\Omega^*.\n\\end{eqnarray*} \n","sha256_utf8":"e667c733ce20735bccf62f721dcd045330055c2af7ce7b08d83dd0a46617ec14"}]},{"id":"S093","semantic_type":"proof_step","kind":"proof_step","title":"极锥配对迫使对偶方向为零","source_units":[{"start":79593,"end":80474,"start_line":1136,"end_line":1147,"raw_tex":"Since $\\lk -\\mu^k=o(\\xik)$ and $u^k-\\ox=o(\\beta_k)$, we deduce from \\eqref{est4} and \\eqref{unb} that \n\\begin{eqnarray*}\n\\frac{ \\mu^k-d^k}{\\beta_k}+A_\\oi\\big(\\frac{z^k-\\oz}{\\beta_k}\\big)&=& \\frac{ \\mu^k-\\lk}{\\beta_k}+\\frac{ \\lk-d^k}{\\beta_k} +A_\\oi\\big(\\frac{\\Phi(u^k)-\\Phi(\\ox)}{\\beta_k}\\big)\\\\\n&&-\\frac{1}{\\rok}A_\\oi\\big(\\frac{d^k-\\lk}{\\beta_k}\\big)\\to -\\eta \\;\\;\\;\\;\\mbox{as}\\;\\;k\\to \\infty,\n\\end{eqnarray*}\nwhich in turn demonstrates that \n$-\\eta \\in \\Omega^*$. This, combined with \\eqref{est8} and $\\eta\\in E$, shows that \n\\begin{equation*}\n\\|\\eta\\|^2= \\langle \\eta, \\eta \\rangle = \\langle \\eta, P_E(\\eta) \\rangle= -\\langle \\eta, P_E(\\zeta) \\rangle= -\\big\\langle \\eta, P_E(\\zeta)+P_{E^{\\perp}}(\\zeta) \\big\\rangle = -    \\langle \\eta, \\zeta\\rangle \\le 0,\n\\end{equation*} \nimplying that  $\\eta =0$. This clearly contradicts \\eqref{est4} and thus completes the proof.\n\\end{proof}","sha256_utf8":"03c70d95dfef6af95f6412f65df2f1f96b4adf70dc894e42c4257ae097cfe787"}]},{"id":"S094","semantic_type":"proof_step","kind":"proof_step","title":"排除终止情形并控制容差","source_units":[{"start":86571,"end":87327,"start_line":1209,"end_line":1215,"raw_tex":"\\begin{proof}\n Define $R_k:=R(\\xk,\\lk)$   and observe that if $R_k=0$ for some $k$, then the pair $(\\xk,\\lk)$ satisfies the KKT system \\eqref{vs}, and thus Algorithm \\ref{Alg1} should stop. From now on assume that $R_k>0$ for all $k\\in\\N$. \n  Pick the positive constants $\\kappa_1$ and $\\gg_1$ from Theorem~\\ref{sooc}, the positive constants $\\gamma_2$ and $\\kappa_2$ from Proposition~\\ref{erct}, the positive constants \n  $\\gamma_3$ and $\\kappa_3$ from Theorem~\\ref{est}, and the positive constants $\\hat\\gg$ and $ \\tau$ from Proposition~\\ref{solv}. By \\eqref{error}, we can find $\\gamma_4>0$ such that\n\\begin{equation}\\label{ek}\n\\sigma(x,\\lm)\\le\\dfrac{1}{4\\kappa_1\\kappa_2}R(x,\\lm)\\quad\\textrm{ whenever }\\;(x,\\lm)\\in\\B_{\\gg_4}(\\ox,\\olm).\n\\end{equation}\n","sha256_utf8":"31895fde3da789ebcbb235a8a92e62de37318f44988a0694b2651adbb6b2d14a"}]},{"id":"S095","semantic_type":"proof_step","kind":"proof_step","title":"选取邻域与罚参数阈值","source_units":[{"start":87327,"end":87929,"start_line":1216,"end_line":1226,"raw_tex":"Define further the  positive  constants  $\\bar\\tau:=\\kappa_3$ and \n\\begin{equation}\\label{const}\n\\bar\\gamma:=\\dfrac{\\gamma'}{1+2\\sqrt{2}\\bar\\tau\\kappa_2} \\quad\\mbox{with}\\quad\n\\gamma':=\\min\\Big\\{\\frac{\\hat\\gg}{2\\tau},\\gamma_3,\\gamma_4,\\dfrac{\\gamma_1}{1+\\sqrt{2}\\bar\\tau\\kappa_2},\\dfrac{\\gamma_2}{1+\\sqrt{2}\n\\bar\\tau\\kappa_2}\\Big\\}.\n\\end{equation}\nMoreover, pick $q\\in (0,1) $ and set \n\\begin{equation}\\label{rhof}\n\\bar\\varrho:=\\max\\big\\{\\bar\\rho, 4\\kappa_1\\kappa_2\\bar\\tau, \\frac{2\\sqrt{2} \\kappa_1\\kappa_2^2\\bar\\tau^2}{q}\\big\\},\n\\end{equation}\nwhere  $\\bar\\rho$ is taken from Theorem~\\ref{ugrowth}. \n","sha256_utf8":"550939cc5d78202f53213585d9d379b76adb10984f0e0c7845055c2bd9a777b4"}]},{"id":"S096","semantic_type":"proof_step","kind":"proof_step","title":"提出归纳不变量","source_units":[{"start":87929,"end":88541,"start_line":1227,"end_line":1235,"raw_tex":"Arguing by induction, we aim at showing that for any starting point $(x^0,\\lambda^0)\\in\\big(\\Th\\times\\R^m\\big)\\cap\\B_{\\bar\\gamma}(\\ox,\\olm)$ there exists a sequence $\\{(\\xk,\\lk)\\c\\subset \\Th \\times\\R^m$, generated by Algorithm~\\ref{Alg1}\nwith $\\rok \\ge \\bar\\varrho$ and   $\\ek=\\sigma(\\xk,\\lk)$ for all $k$,\n such that for all $k=0,1,\\ldots$ the relationships\n\\begin{equation}\\label{c1}\n(\\xk,\\lk)\\in\\B_{\\gamma'}(\\ox,\\olm),\n\\end{equation}\n\\begin{equation}\\label{c2}\n\\dist\\big(-\\nabla_x L(\\xkk,\\lkk), N_\\Th(\\xkk)\\big)\\le\\ek\\quad\\textrm{ and }\\quad\\Vert\\xkk-\\xk\\Vert+\\Vert\\lkk-\\lk\\Vert\\le\\bar\\tau R_k\n\\end{equation}\n","sha256_utf8":"440f306a4824075734db470e322ef948b78ed0b1c18f2ef4b3e61e224a77f313"}]},{"id":"S097","semantic_type":"proof_step","kind":"proof_step","title":"构造首步并建立归纳假设","source_units":[{"start":88541,"end":89806,"start_line":1236,"end_line":1245,"raw_tex":"hold. To achieve this goal, set $k=0$, pick $(x^0,\\lambda^0)\\in\\big(\\Th\\times\\R^m\\big)\\cap\\B_{\\bar\\gamma}(\\ox,\\olm)$, and observe from  $\\bar\\gamma\\le\\gamma'$ that \\eqref{c1} holds in this case. We also have $\\lambda^0\\in\\B_{\\hat\\gg/2\\tau}(\\olm)$ and $\\rho_0\\ge \\bar\\varrho$ and then find by Proposition~\\ref{solv} a primal iterate $x^1\\in \\inte \\B_{\\hat\\gamma}(\\ox)$ satisfying $-\\nabla_x\\L(x^1,\\lambda^0,\\rho_0)\\in N_\\Th(x^1)$. Define further \nthe multiplier $\\lm^1:=\\nabla e_{1^{}/\\rho_0}g\\big(\\Phi(x^1)+ \\rho_0^{-1}\\lambda^0\\big)$ and observe that\n\\begin{equation*}\n-\\nabla_x L(x^1,\\lm^1)=-\\nabla_x\\L(x^1,\\lambda^0,\\rho_0)\\in N_\\Th(x^1).\n\\end{equation*}\nMoreover, employing  Theorem~\\ref{est} confirms that \n\\begin{equation*}\n\\Vert x^1-x^0\\Vert+\\Vert\\lm^1-\\lambda^0\\Vert\\le\\bar\\tau R_0.\n\\end{equation*}\nThus $(x^1,\\lambda^1)$ is well defined and satisfies the claimed condition \\eqref{c2} for $k=0$. Assume by induction that the iterates $(\\xk,\\lk)$, $k=0,1,\\ldots,s+1$, are well defined  and that   both conditions \\eqref{c1} and \\eqref{c2} hold for $k=0,1,\\ldots,s$. We verify now the existence of $(x^{s+2},\\lambda^{s+2})$ satisfying \\eqref{c1} and \\eqref{c2} for $k=s+1$. To this end, we first show that $(x^{s+1},\\lambda^{s+1})\\in\\B_{\\gamma'}(\\ox,\\olm)$. \n","sha256_utf8":"31d418b28df569baa77191d027643f9a9ffd25e09d372d1570f801cde5a7af0c"}]},{"id":"S098","semantic_type":"proof_step","kind":"proof_step","title":"残差上界保持误差界邻域","source_units":[{"start":89806,"end":90530,"start_line":1246,"end_line":1256,"raw_tex":"To proceed, fix an integer   $k$ with $0\\le k\\le s$.\nSince $(\\xk,\\lk)\\in\\B_{\\gamma'}(\\ox,\\olm)$, it follows from \\eqref{eb1} and \\eqref{c1} that\n\\begin{eqnarray}\\label{c3}\nR_k&\\le&\\kappa_2\\big(\\Vert\\xk-\\ox\\Vert+\\dist(\\lk,\\Lambda(\\ox))\\big)\\\\\n&\\le&\\kappa_2\\big(\\Vert\\xk-\\ox\\Vert+\\Vert\\lk-\\olm\\Vert\\big)\\le\\sqrt{2}\\kappa_2\\Vert(\\xk,\\lk)-(\\ox,\\olm)\\Vert\\le\\sqrt{2}\\kappa_2\\gamma'.\\nonumber\n\\end{eqnarray}\nCombining this with \\eqref{c1} and the second relation in \\eqref{c2}, we conclude that\n\\begin{eqnarray}\\label{c4}\n\\Vert(\\xkk,\\lkk)-(\\ox,\\olm)\\Vert&\\le&\\Vert\\xkk-\\xk\\Vert+\\Vert\\lkk-\\lk\\Vert+\\Vert(\\xk,\\lk)-(\\ox,\\olm)\\Vert\\nonumber\\\\\n&\\le&\\bar\\tau R_k+\\gamma'\\le(\\sqrt{2}\\bar\\tau\\kappa_2+1)\\gamma'\\le\\gamma_1,\n\\end{eqnarray}\n","sha256_utf8":"1de4e4ca0a0bc690968303192d9523042c3ff266f6d3f15afae11c1ce2c35ca9"}]},{"id":"S099","semantic_type":"proof_step","kind":"proof_step","title":"KKT误差界控制下步距离","source_units":[{"start":90530,"end":90994,"start_line":1257,"end_line":1262,"raw_tex":"which implies that $(\\xkk,\\lkk)\\in\\B_{\\gamma_1}(\\ox,\\olm)$. Using \\eqref{subr3}, we obtain\n\\begin{eqnarray}\\label{c5}\n&&\\Vert\\xkk-\\ox\\Vert+\\dist\\big(\\lkk,\\Lambda(\\ox)\\big)\\nonumber\\\\\n&\\le&\\kappa_1 R_{k+1} =\\kappa_1\\Big(\\dist\\big(-\\nabla_x L(\\xkk,\\lkk), N_\\Th(\\xkk)\\big) +\\big\\Vert\\Phi(\\xkk)-{\\rm{prox}}_g\\big(\\Phi(\\xkk)+\\lkk\\big)\\big\\Vert\\Big)\\nonumber\\\\\n&\\le&\\kappa_1\\ek+\\kappa_1\\big\\Vert\\Phi(\\xkk)-{\\rm{prox}}_g\\big(\\Phi(\\xkk)+\\lkk\\big)\\big\\Vert.\n\\end{eqnarray}\n","sha256_utf8":"8ddecf0c4df7275fe30b4ce1905dd8169a6771da17db2ff95bf52e5e386bbb1a"}]},{"id":"S100","semantic_type":"proof_step","kind":"proof_step","title":"近端固定点控制可行性残差","source_units":[{"start":90994,"end":92139,"start_line":1263,"end_line":1281,"raw_tex":"To proceed further,  define\n\\begin{equation}\\label{skk}\n\\skk:=\\proxk\\big(\\Phi(\\xkk)+ \\rho_k^{-1}{\\lk}\\big ).\n\\end{equation}\nIt follows from  \\eqref{lkk}  and \\eqref{ueq19} that\n\\begin{equation}\\label{eq3}\n\\Phi(\\xkk)-\\skk= \\rho_k^{-1}(\\lkk-\\lk).\n\\end{equation}\nSince $\\lkk=\\nabla\\menvk\\big(\\Phi(\\xkk)+ \\rho_k^{-1}{\\lk}\\big)$, we deduce from the second equality in \\eqref{prm} that \n\\begin{equation*}\\label{clq17}\n\\lkk   \\in \\sub g\\big(\\Phi(\\xkk)+ \\rho_k^{-1}(\\lk-\\lkk)\\big)=\\partial g(\\skk).\n\\end{equation*} \nAppealing now to the first equality in \\eqref{prm} yields $\\skk={\\rm{prox}}_g(\\skk+\\lkk)$. Since the mapping $y\\mapsto y-{\\rm{prox}}_g(y+\\lkk)$ is nonexpansive (cf. \\cite[Proposition~12.27]{bc}), we arrive at the relationships\n\\begin{eqnarray}\\label{eq2}\n&&\\big\\Vert\\Phi(\\xkk)-{\\rm{prox}}_g\\big(\\Phi(\\xkk)+\\lkk\\big)\\big\\Vert\\nonumber\\\\\n&=&\\big\\Vert\\Phi(\\xkk)-{\\rm{prox}}_g\\big(\\Phi(\\xkk)+\\lkk\\big)\\big\\Vert-\\big\\Vert\\skk-{\\rm{prox}}_g(\\skk+\\lkk)\\big\\Vert\\nonumber\\\\\n&\\le&\\big\\Vert\\Phi(\\xkk)-{\\rm{prox}}_g\\big(\\Phi(\\xkk)+\\lkk\\big)-\\big(\\skk-{\\rm{prox}}_g(\\skk+\\lkk)\\big)\\big\\Vert\\nonumber\\\\\n&\\le&\\Vert\\Phi(\\xkk)-\\skk\\Vert.\n\\end{eqnarray}\n","sha256_utf8":"aa0e818061e816e8f9e558ec1d37cfaae0809eda771398b344d92be605625699"}]},{"id":"S101","semantic_type":"proof_step","kind":"proof_step","title":"组合容差与罚参数取得收缩","source_units":[{"start":92139,"end":92741,"start_line":1282,"end_line":1290,"raw_tex":"Combining \\eqref{c5}, \\eqref{eq3}, and \\eqref{eq2} brings us to\n\\begin{equation*}\n\\Vert\\xkk-\\ox\\Vert+\\dist\\big(\\lkk,\\Lambda(\\ox)\\big)\\le\\kappa_1\\ek+\\rho_k^{-1}{\\kappa_1}\\Vert\\lkk-\\lk\\Vert.\n\\end{equation*}\nThis,  together with \\eqref{ek}, \\eqref{rhof}, the second inequality in \\eqref{c2}, and \\eqref{c3} results in \n\\begin{eqnarray}\\label{c10}\n\\Vert \\xkk-\\ox\\Vert+\\dist\\big(\\lkk,\\Lambda(\\ox)\\big)&\\le&\\kappa_1\\ek+\\dfrac{\\bar\\tau\\kappa_1}{\\rok}R_k\\le\\dfrac{1}{4\\kappa_2}R_k +\\dfrac{1}{4\\kappa_2}R_k\\\\\n&\\le&\\dfrac{1}{2}\\Big(\\Vert\\xk-\\ox\\Vert+\\dist\\big(\\lk,\\Lambda(\\ox)\\big)\\Big)\\nonumber,\n\\end{eqnarray}\n","sha256_utf8":"d17cb505e9fc8792be8eb938d5b769e3af447485e6416c4e30e28451f04e692f"}]},{"id":"S102","semantic_type":"proof_step","kind":"proof_step","title":"几何衰减保证迭代留在邻域","source_units":[{"start":92741,"end":94261,"start_line":1291,"end_line":1309,"raw_tex":"which in turn implies the primal-dual estimates\n\\begin{equation}\\label{c6}\n\\Vert\\xkk-\\ox\\Vert+\\dist\\big(\\lkk,\\Lambda(\\ox)\\big)\\le\\dfrac{1}{2^{k+1}}\\Big(\\Vert x^0-\\ox\\Vert+\\dist\\big(\\lambda^0,\\Lambda(\\ox)\\big)\\Big).\n\\end{equation}\nThis estimate, \\eqref{c3},  and the second inequality in \\eqref{c2} tell us that \n\\begin{eqnarray}\\label{c7}\n\\Vert(x^{s+1},\\lambda^{s+1})-(x^0,\\lambda^0)\\Vert&\\le&\\sum\\limits_{k=0}^s\\Vert(\\xkk,\\lkk)-(\\xk,\\lk)\\Vert \\le\\bar\\tau\\sum\\limits_{k=0}^s R_k\\nonumber\\\\\n&\\le&\\bar\\tau\\kappa_2\\sum\\limits_{k=0}^s\\Vert\\xk-\\ox\\Vert+\\dist\\big(\\lk,\\Lambda(\\ox)\\big)\\nonumber\\\\\n&\\le&\\bar\\tau\\kappa_2\\sum\\limits_{k=0}^s\\dfrac{1}{2^k}\\Big(\\Vert x^0-\\ox\\Vert+\\dist\\big(\\lambda^0,\\Lambda(\\ox)\\big)\\Big)\\\\\n&\\le&2\\bar\\tau\\kappa_2\\big(\\Vert x^0-\\ox\\Vert+\\dist(\\lambda^0,\\Lambda(\\ox))\\big)\\nonumber\\\\\n&\\le&2\\bar\\tau\\kappa_2\\big(\\Vert x^0-\\ox\\Vert+\\Vert\\lambda^0-\\olm\\Vert\\big).\\nonumber\n\\end{eqnarray}\nThus, we arrive at the estimates\n\\begin{eqnarray*}\n\\Vert(x^{s+1},\\lambda^{s+1})-(\\ox,\\olm)\\Vert&\\le&\\Vert(x^{s+1},\\lambda^{s+1})-(x^0,\\lambda^0)\\Vert+\\Vert(x^0,\\lambda^0)-(\\ox,\\olm)\\Vert\\\\\n&\\le&2\\bar\\tau\\kappa_2\\big(\\Vert x^0-\\ox\\Vert+\\Vert\\lambda^0-\\olm\\Vert\\big)+\\Vert(x^0,\\lambda^0)-(\\ox,\\olm)\\Vert\\\\\n&\\le&(2\\sqrt{2}\\bar\\tau\\kappa_2+1)\\Vert(x^0,\\lambda^0)-(\\ox,\\olm)\\Vert\\le(2\\sqrt{2}\\bar\\tau\\kappa_2+1)\\bar\\gamma=\\gamma',\n\\end{eqnarray*}\nwhere the last inequality follows from $(x^0,\\lambda^0)\\in\\B_{\\bar\\gamma}(\\ox,\\olm)$, while the last equality comes from the definition of $\\bar\\gamma$ in \\eqref{const}. \n","sha256_utf8":"04b56060073d676c71585106621d6c96ee589169ca98757f0ea4bb9e92c78273"}]},{"id":"S103","semantic_type":"proof_step","kind":"proof_step","title":"构造后继迭代完成归纳","source_units":[{"start":94261,"end":95503,"start_line":1310,"end_line":1324,"raw_tex":"This verifies that $(x^{s+1},\\lambda^{s+1})\\in\\B_{\\gamma'}(\\ox,\\olm)$. By  \\eqref{const}, we get  $\\lambda^{s+1}\\in\\B_{\\hat\\gg/2\\tau}(\\olm)$, and hence Proposition~\\ref{solv} ensures the existence of of an optimal solution $x^{s+2}$ \nto \\eqref{regp} associated with $(\\lm^{s+1},\\rho_{s+1})$ such that $x^{s+2}\\in \\inte\\B_{\\hat\\gamma}(\\ox)$. The latter yields \n\\begin{equation*}\n-\\nabla_x\\L(x^{s+2},\\lambda^{s+1},\\rho_{s+1})\\in N_\\Th(x^{s+2}).\n\\end{equation*} \nSet now $\\lm^{s+2}:=\\nabla e_{1^{}/\\rho_{s+1}}\\big(\\Phi(x^{s+2})+\\rho_{s+1}^{-1}\\lambda^{s+1}\\big)$ and observe that\n\\begin{equation*}\n -\\nabla_x L(x^{s+2},\\lm^{s+2}) =  -\\nabla_x\\L(x^{s+2},\\lambda^{s+1},\\rho_{s+1}) \\in N_\\Th(x^{s+2}),\n\\end{equation*}\nwhich by Proposition~\\ref{est} brings us to the estimate\n\\begin{equation*}\n\\Vert x^{s+2}-x^{s+1}\\Vert+\\Vert\\lm^{s+2}-\\lambda^{s+1}\\Vert\\le\\bar\\tau R_{s+1}.\n\\end{equation*}\nThis tells us that the iterate $(x^{s+2},\\lambda^{s+2})$ satisfies   \\eqref{c2} for $k=s+1$, which completes  the proof of  induction. Thus there exists a sequence  $\\{(\\xk,\\lk)\\c\\subset\\Th\\times\\R^m$, generated by Algorithm~\\ref{Alg1} \nwith $\\rok \\ge \\bar\\varrho$ and   $\\ek=o\\big(R(\\xk,\\lk)\\big)$ for all $k$, for which  \\eqref{c1} and \\eqref{c2} fulfill.\n","sha256_utf8":"9a121a139dbf3688c555dc9af8ab2c2388f181680c205db5c6835eb680704b72"}]},{"id":"S104","semantic_type":"proof_step","kind":"proof_step","title":"尾和界证明原始对偶收敛","source_units":[{"start":95506,"end":96118,"start_line":1327,"end_line":1331,"raw_tex":"Next, we are going to justify  the convergence of this primal-dual   sequence. Using the same arguments as in the proofs of \\eqref{c6} and \\eqref{c7} leads us to\n\\begin{equation}\\label{c8}\n\\Vert(x^{k+l},\\lambda^{k+l})-(\\xk,\\lk)\\Vert\\le 2\\bar\\tau\\kappa_2\\big(\\Vert\\xk-\\ox\\Vert+\\dist(\\lk,\\Lambda(\\ox))\\big)\\quad\\mbox{for all}\\;\\;k,l\\in\\N.\n\\end{equation}\nIt follows from \\eqref{c6} that the right-hand side of \\eqref{c8} goes to zero as $k\\to\\infty$, which tells us that the sequence $\\{(\\xk,\\lk)\\c$ is Cauchy. By \\eqref{c6},   this sequence converges  to $(\\ox,\\hat\\lambda)$ for some $\\hat\\lambda\\in\\Lambda(\\ox)$.\n","sha256_utf8":"a7e71810e56ae7f585d63ca7e7feebe2631f2cedee64cb391cd6b110c20039e6"}]},{"id":"S105","semantic_type":"proof_step","kind":"proof_step","title":"极限距离的相邻步比率界","source_units":[{"start":96118,"end":97134,"start_line":1332,"end_line":1343,"raw_tex":"Now, we show  that  the rate of this convergence is linear. Indeed, letting $l\\to\\infty$ in \\eqref{c8} tells us that for all $k\\in\\N$ we have\n\\begin{equation*}\\label{c9}\n\\Vert(\\xk,\\lk)-(\\ox,\\hat\\lambda)\\Vert\\le 2\\bar\\tau\\kappa_2\\big(\\Vert\\xk-\\ox\\Vert+\\dist(\\lk,\\Lambda(\\ox))\\big),\n\\end{equation*}\nwhich together with the first inequality in \\eqref{c10} verifies that\n\\begin{eqnarray*}\n\\Vert(\\xkk,\\lkk)-(\\ox,\\hat\\lambda)\\Vert&\\le&2\\bar\\tau\\kappa_2\\big(\\Vert\\xkk-\\ox\\Vert+\\dist(\\lkk,\\Lambda(\\ox))\\big)\\nonumber\\\\\n&\\le&2\\bar\\tau\\kappa_1\\kappa_2\\big(R_k^{-1}{\\ek}+\\rho_k^{-1}\\bar\\tau\\big)R_k\\nonumber\\\\\n&\\le&2\\bar\\tau\\kappa_1\\kappa_2^2\\big(R_k^{-1}{\\ek}+\\rho_k^{-1}\\bar\\tau\\big)\\big(\\Vert\\xk-\\ox\\Vert+\\dist(\\lk,\\Lambda(\\ox))\\big)\\nonumber\\\\\n&\\le&2\\bar\\tau\\kappa_1\\kappa_2^2\\big(R_k^{-1}{\\ek}+\\rho_k^{-1}\\bar\\tau\\big)\\big(\\Vert\\xk-\\ox\\Vert+\\Vert\\lk-\\hat\\lambda\\Vert\\big)\\nonumber\\\\\n&\\le& 2\\sqrt{2}\\bar\\tau\\kappa_1\\kappa_2^2\\big(R_k^{-1}{\\ek}+\\rho_k^{-1}\\bar\\tau\\big)\\Vert(\\xk,\\lk)-(\\ox,\\hat\\lambda)\\Vert.\n\\end{eqnarray*}\n","sha256_utf8":"00ec4fff68e30978f6db3097dc3f0699185f5bda5224734b658da1a9ba56ca5a"}]},{"id":"S106","semantic_type":"proof_step","kind":"proof_step","title":"线性及超线性速率结论","source_units":[{"start":97134,"end":97816,"start_line":1344,"end_line":1351,"raw_tex":"Combining this with $\\rok\\ge \\bar\\varrho$  with $ \\bar\\varrho$ taken from \\eqref{rhof} and using the choice of the tolerance $\\ek=o(R_k)$ result in \n\\begin{equation*}\n\\limsup_{k\\to\\infty}\\dfrac{\\Vert(\\xkk,\\lkk)-(\\ox,\\hat\\lambda)\\Vert}{\\Vert(\\xk,\\lk)-(\\ox,\\hat\\lambda)\\Vert}\\le\\limsup_{k\\to\\infty}2\\sqrt{2}\\bar\\tau\n\\kappa_1\\kappa_2^2\\big(R_k^{-1}{\\ek}+\\rho_k^{-1}\\bar\\tau\\big)\\le q.\n\\end{equation*}\nIt follows from  $q\\in (0,1)$ that the rate of   convergence of $\\{(\\xk,\\lk)\\c$ is linear. Furthermore, it is clear from the above inequality that if $\\rok\\to \\infty$, we \narrive at the superlinear convergence of the latter primal-dual sequence, which completes the proof.\n\\end{proof}","sha256_utf8":"89771fc5cf6c259c1dcabf81c71799248cc1b206801e596c903d076254f08b85"}]},{"id":"S107","semantic_type":"external_fact","kind":"external_fact","title":"近端与包络标准性质","source_units":[],"declared_external":true,"declaration_text":["凸函数的近端与包络满足源文 prm；近端映射及其补映射非扩张。CPLQ 函数的 Moreau 包络有限、连续可微且为 CPLQ；相应增广函数关于乘子凹、关于罚参数单调不减。引用 rw，Propositions 12.19、12.30，Theorem 2.26，Exercise 11.56；bc，Proposition 12.27。"]},{"id":"S108","semantic_type":"external_fact","kind":"external_fact","title":"二阶求和及增长标准判据","source_units":[],"declared_external":true,"declaration_text":["完全可处理性具有源文使用的和规则；相应二阶次导数可按和规则计算，多面体指标函数的二阶次导数为临界锥指标函数；驻点处所有非零方向上的二阶严格正性与局部二次增长满足源文使用的标准判据。引用 rw，Exercise 10.22、Proposition 13.19、Theorem 13.24(c)；mmstams，Example 3.4。"]},{"id":"S109","semantic_type":"external_fact","kind":"external_fact","title":"已知分片二次变分公式","source_units":[],"declared_external":true,"declaration_text":["CPLQ 函数满足源文 B0019 的临界锥、二次上图可微性及二阶导数公式，且其次微分满足源文 352–354 行公式。源文将这些作为 rw，Theorem 13.14、Proposition 13.9 及第 487 页的已知结果回顾。"]},{"id":"S110","semantic_type":"external_fact","kind":"external_fact","title":"已知充分条件下KKT误差界","source_units":[],"declared_external":true,"declaration_text":["在源文模型及 SOSC 下，源文 B0029 的局部 KKT 误差界 subr3 成立。源文明确将该定理归于 s20，Theorem 3.6 和 Propositions 3.8、3.9 的组合。"]},{"id":"S111","semantic_type":"external_fact","kind":"external_fact","title":"包络二阶上图导数规则","source_units":[],"declared_external":true,"declaration_text":["在源文满足的凸性及二次上图可微条件下，Moreau 包络二次上图可微，其二阶上图导数满足源文 more 的包络公式。引用 hms20，Proposition 4.1。"]},{"id":"S112","semantic_type":"external_fact","kind":"external_fact","title":"二次半可微计算规则","source_units":[],"declared_external":true,"declaration_text":["源文所用凸函数的有限二阶上图导数给出二次半可微性；二次半导数满足所用求和与复合规则，并与相应二阶上图导数一致。引用 r20，Theorem 4.3；mmstams，Propositions 8.1、8.2(i)、8.2(iii)。"]},{"id":"S113","semantic_type":"external_fact","kind":"external_fact","title":"齐次性及上图收敛规则","source_units":[],"declared_external":true,"declaration_text":["二阶次导数下半连续且二次正齐次；Moreau 包络随正则化消失单调趋于闭真凸函数；源文所用单调上图收敛、加指标函数的上图收敛及紧约束集上的最小值收敛规则成立。引用 rw，Proposition 13.5、Theorem 1.25、Proposition 7.4、Theorems 7.46、7.33。"]},{"id":"S114","semantic_type":"external_fact","kind":"external_fact","title":"多面体锥与活跃面几何","source_units":[],"declared_external":true,"declaration_text":["多面体集合满足源文使用的切锥、法锥及临界锥公式，邻近点法锥有局部包含关系，活跃面只有有限种，且多面体锥的线性像闭。凸集临界锥等于切锥与给定法向量正交空间之交，见 dr，第 109 页。"]},{"id":"S115","semantic_type":"external_fact","kind":"external_fact","title":"多面体域上的相对连续性","source_units":[],"declared_external":true,"declaration_text":["源文使用的下半连续凸函数在其多面体定义域上相对连续的结论，引用 rw，Theorem 2.35。"]},{"id":"S116","semantic_type":"external_fact","kind":"external_fact","title":"正交补与子空间投影","source_units":[],"declared_external":true,"declaration_text":["对于源文定义的 E，有 E^perp={∇Φ(x̄)y:By=0}；子空间的正交投影线性并满足正交分解。正交补公式引用 rw，Corollary 11.25(d)。"]},{"id":"S117","semantic_type":"external_fact","kind":"external_fact","title":"有限维分析基本事实","source_units":[],"declared_external":true,"declaration_text":["有限维 Weierstrass 定理、紧性子列原理、二阶 Taylor 展开、光滑映射局部 Lipschitz 性、几何级数求和以及 Cauchy 序列完备性。"]},{"id":"S118","semantic_type":"assumption","kind":"assumption","title":"主定理的KKT与充分性假设","source_units":[{"start":85625,"end":85739,"start_line":1199,"end_line":1199,"raw_tex":" Let $(\\ox,\\olm)$ be a solution to the KKT system \\eqref{vs} and  let the SOSC \\eqref{sosc} hold at $(\\ox,\\olm)$.\n","sha256_utf8":"78b3d7dcfd20b95f2eda9f5cee7b52c2cc8ecbc43042a2ed62f5a74ad9e8dc15"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S009","kind":"supports"},{"source_id":"S003","target_id":"S023","kind":"supports"},{"source_id":"S003","target_id":"S039","kind":"supports"},{"source_id":"S003","target_id":"S068","kind":"supports"},{"source_id":"S003","target_id":"S087","kind":"supports"},{"source_id":"S004","target_id":"S023","kind":"supports"},{"source_id":"S004","target_id":"S027","kind":"supports"},{"source_id":"S004","target_id":"S028","kind":"supports"},{"source_id":"S004","target_id":"S039","kind":"supports"},{"source_id":"S004","target_id":"S047","kind":"supports"},{"source_id":"S004","target_id":"S055","kind":"supports"},{"source_id":"S004","target_id":"S070","kind":"supports"},{"source_id":"S004","target_id":"S087","kind":"supports"},{"source_id":"S005","target_id":"S046","kind":"supports"},{"source_id":"S005","target_id":"S083","kind":"supports"},{"source_id":"S005","target_id":"S095","kind":"supports"},{"source_id":"S005","target_id":"S098","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S019","kind":"supports"},{"source_id":"S006","target_id":"S031","kind":"supports"},{"source_id":"S006","target_id":"S092","kind":"supports"},{"source_id":"S007","target_id":"S010","kind":"supports"},{"source_id":"S007","target_id":"S019","kind":"supports"},{"source_id":"S007","target_id":"S031","kind":"supports"},{"source_id":"S007","target_id":"S033","kind":"supports"},{"source_id":"S007","target_id":"S046","kind":"supports"},{"source_id":"S007","target_id":"S047","kind":"supports"},{"source_id":"S007","target_id":"S061","kind":"supports"},{"source_id":"S008","target_id":"S019","kind":"supports"},{"source_id":"S008","target_id":"S089","kind":"supports"},{"source_id":"S008","target_id":"S092","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S020","kind":"supports"},{"source_id":"S009","target_id":"S027","kind":"supports"},{"source_id":"S009","target_id":"S028","kind":"supports"},{"source_id":"S009","target_id":"S042","kind":"supports"},{"source_id":"S009","target_id":"S054","kind":"supports"},{"source_id":"S009","target_id":"S078","kind":"supports"},{"source_id":"S009","target_id":"S084","kind":"supports"},{"source_id":"S009","target_id":"S094","kind":"supports"},{"source_id":"S010","target_id":"S020","kind":"supports"},{"source_id":"S010","target_id":"S045","kind":"supports"},{"source_id":"S010","target_id":"S046","kind":"supports"},{"source_id":"S011","target_id":"S027","kind":"supports"},{"source_id":"S011","target_id":"S028","kind":"supports"},{"source_id":"S011","target_id":"S036","kind":"supports"},{"source_id":"S011","target_id":"S042","kind":"supports"},{"source_id":"S011","target_id":"S069","kind":"supports"},{"source_id":"S011","target_id":"S071","kind":"supports"},{"source_id":"S011","target_id":"S078","kind":"supports"},{"source_id":"S011","target_id":"S089","kind":"supports"},{"source_id":"S011","target_id":"S100","kind":"supports"},{"source_id":"S012","target_id":"S023","kind":"supports"},{"source_id":"S012","target_id":"S024","kind":"supports"},{"source_id":"S013","target_id":"S037","kind":"supports"},{"source_id":"S013","target_id":"S038","kind":"supports"},{"source_id":"S013","target_id":"S039","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S097","kind":"supports"},{"source_id":"S014","target_id":"S100","kind":"supports"},{"source_id":"S014","target_id":"S103","kind":"supports"},{"source_id":"S015","target_id":"S094","kind":"supports"},{"source_id":"S015","target_id":"S096","kind":"supports"},{"source_id":"S015","target_id":"S097","kind":"supports"},{"source_id":"S015","target_id":"S103","kind":"supports"},{"source_id":"S016","target_id":"S078","kind":"supports"},{"source_id":"S016","target_id":"S087","kind":"supports"},{"source_id":"S016","target_id":"S088","kind":"supports"},{"source_id":"S017","target_id":"S079","kind":"supports"},{"source_id":"S017","target_id":"S083","kind":"supports"},{"source_id":"S017","target_id":"S094","kind":"supports"},{"source_id":"S017","target_id":"S099","kind":"supports"},{"source_id":"S018","target_id":"S094","kind":"supports"},{"source_id":"S018","target_id":"S101","kind":"supports"},{"source_id":"S018","target_id":"S106","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S030","kind":"supports"},{"source_id":"S033","target_id":"S030","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S033","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S021","target_id":"S031","kind":"supports"},{"source_id":"S021","target_id":"S032","kind":"supports"},{"source_id":"S025","target_id":"S031","kind":"supports"},{"source_id":"S025","target_id":"S032","kind":"supports"},{"source_id":"S025","target_id":"S033","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S035","kind":"supports"},{"source_id":"S039","target_id":"S035","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S019","target_id":"S036","kind":"supports"},{"source_id":"S019","target_id":"S037","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S025","target_id":"S042","kind":"supports"},{"source_id":"S034","target_id":"S042","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S044","kind":"supports"},{"source_id":"S049","target_id":"S044","kind":"supports"},{"source_id":"S029","target_id":"S045","kind":"supports"},{"source_id":"S040","target_id":"S045","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S019","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S040","target_id":"S049","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S051","kind":"supports"},{"source_id":"S055","target_id":"S051","kind":"supports"},{"source_id":"S056","target_id":"S051","kind":"supports"},{"source_id":"S057","target_id":"S051","kind":"supports"},{"source_id":"S058","target_id":"S051","kind":"supports"},{"source_id":"S043","target_id":"S052","kind":"supports"},{"source_id":"S034","target_id":"S052","kind":"supports"},{"source_id":"S040","target_id":"S052","kind":"supports"},{"source_id":"S040","target_id":"S053","kind":"supports"},{"source_id":"S019","target_id":"S055","kind":"supports"},{"source_id":"S019","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S052","target_id":"S057","kind":"supports"},{"source_id":"S054","target_id":"S057","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S060","kind":"supports"},{"source_id":"S063","target_id":"S060","kind":"supports"},{"source_id":"S064","target_id":"S060","kind":"supports"},{"source_id":"S065","target_id":"S060","kind":"supports"},{"source_id":"S043","target_id":"S061","kind":"supports"},{"source_id":"S050","target_id":"S061","kind":"supports"},{"source_id":"S025","target_id":"S061","kind":"supports"},{"source_id":"S021","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S063","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S062","target_id":"S064","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S021","target_id":"S065","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S068","target_id":"S067","kind":"supports"},{"source_id":"S069","target_id":"S067","kind":"supports"},{"source_id":"S070","target_id":"S067","kind":"supports"},{"source_id":"S071","target_id":"S067","kind":"supports"},{"source_id":"S072","target_id":"S067","kind":"supports"},{"source_id":"S059","target_id":"S068","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S021","target_id":"S069","kind":"supports"},{"source_id":"S021","target_id":"S070","kind":"supports"},{"source_id":"S069","target_id":"S070","kind":"supports"},{"source_id":"S068","target_id":"S070","kind":"supports"},{"source_id":"S068","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S069","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S067","target_id":"S066","kind":"supports"},{"source_id":"S074","target_id":"S073","kind":"supports"},{"source_id":"S075","target_id":"S073","kind":"supports"},{"source_id":"S059","target_id":"S074","kind":"supports"},{"source_id":"S070","target_id":"S074","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S069","target_id":"S075","kind":"supports"},{"source_id":"S078","target_id":"S077","kind":"supports"},{"source_id":"S079","target_id":"S077","kind":"supports"},{"source_id":"S078","target_id":"S079","kind":"supports"},{"source_id":"S068","target_id":"S079","kind":"supports"},{"source_id":"S077","target_id":"S076","kind":"supports"},{"source_id":"S082","target_id":"S081","kind":"supports"},{"source_id":"S083","target_id":"S081","kind":"supports"},{"source_id":"S084","target_id":"S081","kind":"supports"},{"source_id":"S085","target_id":"S081","kind":"supports"},{"source_id":"S086","target_id":"S081","kind":"supports"},{"source_id":"S087","target_id":"S081","kind":"supports"},{"source_id":"S088","target_id":"S081","kind":"supports"},{"source_id":"S089","target_id":"S081","kind":"supports"},{"source_id":"S090","target_id":"S081","kind":"supports"},{"source_id":"S091","target_id":"S081","kind":"supports"},{"source_id":"S092","target_id":"S081","kind":"supports"},{"source_id":"S093","target_id":"S081","kind":"supports"},{"source_id":"S059","target_id":"S082","kind":"supports"},{"source_id":"S066","target_id":"S082","kind":"supports"},{"source_id":"S082","target_id":"S083","kind":"supports"},{"source_id":"S083","target_id":"S084","kind":"supports"},{"source_id":"S020","target_id":"S084","kind":"supports"},{"source_id":"S083","target_id":"S085","kind":"supports"},{"source_id":"S084","target_id":"S085","kind":"supports"},{"source_id":"S073","target_id":"S085","kind":"supports"},{"source_id":"S085","target_id":"S086","kind":"supports"},{"source_id":"S084","target_id":"S086","kind":"supports"},{"source_id":"S069","target_id":"S086","kind":"supports"},{"source_id":"S071","target_id":"S086","kind":"supports"},{"source_id":"S083","target_id":"S087","kind":"supports"},{"source_id":"S082","target_id":"S087","kind":"supports"},{"source_id":"S085","target_id":"S087","kind":"supports"},{"source_id":"S083","target_id":"S088","kind":"supports"},{"source_id":"S085","target_id":"S088","kind":"supports"},{"source_id":"S087","target_id":"S088","kind":"supports"},{"source_id":"S083","target_id":"S089","kind":"supports"},{"source_id":"S085","target_id":"S089","kind":"supports"},{"source_id":"S087","target_id":"S089","kind":"supports"},{"source_id":"S089","target_id":"S090","kind":"supports"},{"source_id":"S083","target_id":"S091","kind":"supports"},{"source_id":"S087","target_id":"S091","kind":"supports"},{"source_id":"S088","target_id":"S091","kind":"supports"},{"source_id":"S090","target_id":"S091","kind":"supports"},{"source_id":"S084","target_id":"S092","kind":"supports"},{"source_id":"S089","target_id":"S092","kind":"supports"},{"source_id":"S090","target_id":"S092","kind":"supports"},{"source_id":"S083","target_id":"S093","kind":"supports"},{"source_id":"S084","target_id":"S093","kind":"supports"},{"source_id":"S085","target_id":"S093","kind":"supports"},{"source_id":"S087","target_id":"S093","kind":"supports"},{"source_id":"S089","target_id":"S093","kind":"supports"},{"source_id":"S091","target_id":"S093","kind":"supports"},{"source_id":"S092","target_id":"S093","kind":"supports"},{"source_id":"S081","target_id":"S080","kind":"supports"},{"source_id":"S094","target_id":"S002","kind":"supports"},{"source_id":"S095","target_id":"S002","kind":"supports"},{"source_id":"S096","target_id":"S002","kind":"supports"},{"source_id":"S097","target_id":"S002","kind":"supports"},{"source_id":"S098","target_id":"S002","kind":"supports"},{"source_id":"S099","target_id":"S002","kind":"supports"},{"source_id":"S100","target_id":"S002","kind":"supports"},{"source_id":"S101","target_id":"S002","kind":"supports"},{"source_id":"S102","target_id":"S002","kind":"supports"},{"source_id":"S103","target_id":"S002","kind":"supports"},{"source_id":"S104","target_id":"S002","kind":"supports"},{"source_id":"S105","target_id":"S002","kind":"supports"},{"source_id":"S106","target_id":"S002","kind":"supports"},{"source_id":"S118","target_id":"S094","kind":"supports"},{"source_id":"S118","target_id":"S095","kind":"supports"},{"source_id":"S020","target_id":"S094","kind":"supports"},{"source_id":"S076","target_id":"S094","kind":"supports"},{"source_id":"S080","target_id":"S094","kind":"supports"},{"source_id":"S066","target_id":"S094","kind":"supports"},{"source_id":"S094","target_id":"S095","kind":"supports"},{"source_id":"S059","target_id":"S095","kind":"supports"},{"source_id":"S095","target_id":"S096","kind":"supports"},{"source_id":"S095","target_id":"S097","kind":"supports"},{"source_id":"S096","target_id":"S097","kind":"supports"},{"source_id":"S066","target_id":"S097","kind":"supports"},{"source_id":"S080","target_id":"S097","kind":"supports"},{"source_id":"S096","target_id":"S098","kind":"supports"},{"source_id":"S097","target_id":"S098","kind":"supports"},{"source_id":"S076","target_id":"S098","kind":"supports"},{"source_id":"S095","target_id":"S098","kind":"supports"},{"source_id":"S098","target_id":"S099","kind":"supports"},{"source_id":"S096","target_id":"S099","kind":"supports"},{"source_id":"S020","target_id":"S099","kind":"supports"},{"source_id":"S069","target_id":"S100","kind":"supports"},{"source_id":"S099","target_id":"S101","kind":"supports"},{"source_id":"S100","target_id":"S101","kind":"supports"},{"source_id":"S094","target_id":"S101","kind":"supports"},{"source_id":"S095","target_id":"S101","kind":"supports"},{"source_id":"S096","target_id":"S101","kind":"supports"},{"source_id":"S098","target_id":"S101","kind":"supports"},{"source_id":"S101","target_id":"S102","kind":"supports"},{"source_id":"S098","target_id":"S102","kind":"supports"},{"source_id":"S096","target_id":"S102","kind":"supports"},{"source_id":"S095","target_id":"S102","kind":"supports"},{"source_id":"S102","target_id":"S103","kind":"supports"},{"source_id":"S066","target_id":"S103","kind":"supports"},{"source_id":"S080","target_id":"S103","kind":"supports"},{"source_id":"S101","target_id":"S104","kind":"supports"},{"source_id":"S102","target_id":"S104","kind":"supports"},{"source_id":"S103","target_id":"S104","kind":"supports"},{"source_id":"S104","target_id":"S105","kind":"supports"},{"source_id":"S101","target_id":"S105","kind":"supports"},{"source_id":"S076","target_id":"S105","kind":"supports"},{"source_id":"S105","target_id":"S106","kind":"supports"},{"source_id":"S095","target_id":"S106","kind":"supports"},{"source_id":"S107","target_id":"S011","kind":"supports"},{"source_id":"S107","target_id":"S023","kind":"supports"},{"source_id":"S107","target_id":"S024","kind":"supports"},{"source_id":"S107","target_id":"S037","kind":"supports"},{"source_id":"S107","target_id":"S045","kind":"supports"},{"source_id":"S107","target_id":"S058","kind":"supports"},{"source_id":"S107","target_id":"S069","kind":"supports"},{"source_id":"S107","target_id":"S071","kind":"supports"},{"source_id":"S107","target_id":"S079","kind":"supports"},{"source_id":"S107","target_id":"S100","kind":"supports"},{"source_id":"S108","target_id":"S024","kind":"supports"},{"source_id":"S108","target_id":"S031","kind":"supports"},{"source_id":"S108","target_id":"S032","kind":"supports"},{"source_id":"S109","target_id":"S008","kind":"supports"},{"source_id":"S109","target_id":"S019","kind":"supports"},{"source_id":"S110","target_id":"S020","kind":"supports"},{"source_id":"S111","target_id":"S036","kind":"supports"},{"source_id":"S112","target_id":"S037","kind":"supports"},{"source_id":"S112","target_id":"S038","kind":"supports"},{"source_id":"S112","target_id":"S039","kind":"supports"},{"source_id":"S113","target_id":"S046","kind":"supports"},{"source_id":"S113","target_id":"S047","kind":"supports"},{"source_id":"S113","target_id":"S049","kind":"supports"},{"source_id":"S113","target_id":"S052","kind":"supports"},{"source_id":"S113","target_id":"S058","kind":"supports"},{"source_id":"S114","target_id":"S007","kind":"supports"},{"source_id":"S114","target_id":"S056","kind":"supports"},{"source_id":"S114","target_id":"S057","kind":"supports"},{"source_id":"S114","target_id":"S064","kind":"supports"},{"source_id":"S114","target_id":"S078","kind":"supports"},{"source_id":"S114","target_id":"S087","kind":"supports"},{"source_id":"S114","target_id":"S089","kind":"supports"},{"source_id":"S114","target_id":"S090","kind":"supports"},{"source_id":"S114","target_id":"S091","kind":"supports"},{"source_id":"S114","target_id":"S092","kind":"supports"},{"source_id":"S114","target_id":"S093","kind":"supports"},{"source_id":"S115","target_id":"S064","kind":"supports"},{"source_id":"S116","target_id":"S088","kind":"supports"},{"source_id":"S116","target_id":"S091","kind":"supports"},{"source_id":"S116","target_id":"S093","kind":"supports"},{"source_id":"S117","target_id":"S046","kind":"supports"},{"source_id":"S117","target_id":"S063","kind":"supports"},{"source_id":"S117","target_id":"S068","kind":"supports"},{"source_id":"S117","target_id":"S069","kind":"supports"},{"source_id":"S117","target_id":"S079","kind":"supports"},{"source_id":"S117","target_id":"S083","kind":"supports"},{"source_id":"S117","target_id":"S087","kind":"supports"},{"source_id":"S117","target_id":"S088","kind":"supports"},{"source_id":"S117","target_id":"S102","kind":"supports"},{"source_id":"S117","target_id":"S104","kind":"supports"}]}}
{"record_id":"2010.14314v4","paper":{"title":"Faster Lagrangian-Based Methods in Convex Optimization","authors":["Shoham Sabach","Marc Teboulle"],"arxiv_id":"2010.14314","version":"v4","journal":"SIAM Journal on Optimization","doi":"10.1137/20m1375358","abstract_url":"https://arxiv.org/abs/2010.14314v4","pdf_url":"https://arxiv.org/pdf/2010.14314v4"},"context":{"raw_tex":"\tIn this paper, we consider the linearly constrained convex optimization model,\n\n\n\\begin{equation*}\n\t\t\\mbox{(P)} \\quad\\quad \\min  \\left\\{ \\Psi\\left(x\\right) \\, : \\, \\AAA x = b, \\; x \\in \\real^n\\right\\},\n\t\\end{equation*}\n\n\twhere $\\Psi:\\real^{n} \\rightarrow \\erl$ is a proper, lower semi-continuous and convex function, $\\AAA: \\real^n\\to \\real^m$ is a linear mapping, and $b \\in \\real^{m}$. Throughout this work, the feasible set of problem (P) is denoted by  $\\FFF = \\left\\{ x \\in \\rr^{n} : \\, \\AAA x = b \\right\\}$.\n\\medskip\n\t\n\t Despite its apparent simplicity, with an additional structural/data information on the objective function or/and on the linear mapping (see examples below), the optimization model (P) essentially captures a very wide range of convex optimization problems arising in many modern applications. Since we are interested in analyzing this model also in the strongly convex case, we instead assume that $\\Psi\\left(\\cdot\\right)$ is $\\sigma$-strongly convex with $\\sigma \\geq 0$ (\\ie when $\\sigma = 0$ we recover the convex setting). At this juncture, it is important to mention that in many applied problems the function $\\Psi$ is defined by the sum of two or more functions, see \\eg \\cite{CP2016-B}. In such cases,  it is enough to assume that {\\em one} of the functions is $\\sigma$-strongly convex, see Section \\ref{SSec:Block} for more details.\n\n\n\tWe conclude this section with some basic properties of Lagrangians in convex optimization. For the equality constrained problem given in model (P), the {\\em Lagrangian} $\\Lag: \\rr^{n} \\times \\rr^{m} \\to \\erl$ is defined by\n\n\n\\begin{equation} \\label{D:Lag}\n\t\t\\Lag\\left(x , y\\right) \\equiv \\Psi\\left(x\\right) + \\act{y , \\AAA x - b},\n\t\\end{equation}\n\n\twhere $y \\in \\real^{m}$ is the {\\em Lagrange multiplier} associated with the linear constraint $\\AAA x = b$. In this study, we will also use the \\textit{augmented Lagrangian} associated to problem (P), which is defined by \t\n\n\n\\begin{equation} \\label{D:AugLA}\n\t\t\\Lag_{\\rho}\\left(x ,  y\\right) := \\Lag\\left(x , y\\right) + \\frac{\\rho}{2}\\norm{\\AAA x - b}^{2} = \\Psi\\left(x\\right) + \\act{y , \\AAA x - b} + \\frac{\\rho}{2}\\norm{\\AAA x - b}^{2},\n\t\\end{equation}\n\n\twhere $\\rho > 0$ is a {\\em penalty parameter}. Clearly, when $\\rho = 0$ we recover the Lagrangian itself, \\ie $\\Lag \\equiv \\Lag_{0}$.\n\n\n\\begin{assumption} \\label{A:ConvLag}\n\t\tThe Lagrangian $\\Lag$ has a saddle point, that is, there exists a pair $\\left(x^{\\ast} , y^{\\ast}\\right)$ such that\n\t\t\\begin{equation} \\label{A:ConvLag:1}\n\t\t\t\\Lag\\left(x^{\\ast} , y\\right) \\leq \\Lag\\left(x^{\\ast} , y^{\\ast}\\right) \\leq \\Lag\\left(x , y^{\\ast}\\right), \\quad \\forall \\,\\, x \\in \\real^{n}, \\quad \\forall \\,\\, y\\in \\real^{m}.\n\t\t\\end{equation}\n\t\\end{assumption}\n\n\tWe end this section with some notations that will be used throughout the paper. We denote by $\\bbS_{+}^{n}$ and $\\bbS_{++}^{n}$ the sets of all $n \\times n$ {\\em positive semi-definite matrices} and {\\em positive definite matrices}, respectively. Given $P \\in \\bbS_{+}^{n}$, $\\norm{u}_{P} := \\sqrt{\\act{u , Pu}}$ stands for the semi-norm of $u \\in \\real^{n}$. For any $u , v , w \\in \\real^{n}$, we define the following quantity \t\t\n\n\n\\begin{equation} \\label{D:Delta}\n\t\t\\Delta_{P}\\left(u , v , w\\right) := \\frac{1}{2}\\left(\\norm{u - v}_{P}^{2} - \\norm{u - w}_{P}^{2}\\right).\n\t\\end{equation}\n\n\tIn the case that $P = I$, where $I_{n}$ denotes the $n \\times n$ {\\em identity matrix}, we simply denote $\\Delta := \\Delta_{P}$.\n\n\n\\begin{definition}[Nice primal algorithmic map] \\label{D:AlgoMapNice}\n\t\tGiven the parameters $\\rho , t > 0$, we let $\\rho_{t} = \\rho$ and $\\tau_{t} = 1$ (when $\\sigma = 0$) or $\\rho_{t} = \\rho t$ and $\\tau_{t} = t$ (when $\\sigma > 0$). A primal algorithmic map $\\SSS_{t} : \\real^{n} \\times \\real^{m} \\rightarrow \\real^{n}$, which is applied on the augmented Lagrangian $\\Lag_{\\rho_{t}}\\left(z , \\lambda\\right)$, that generates $z^{+}$ via $z^{+} \\in \\SSS_{t}\\left(z , \\lambda\\right)$, is called {\\em \\nice}\\hspace{-0.05in} if there exist a parameter $\\delta \\in \\left(0 , 1\\right]$ and matrices $P , Q \\in \\bbS_{+}^{n}$, such that for any $\\xi \\in \\FFF$ we have\n\t\t\\begin{equation} \\label{D:AlgoMapNice:1}\n\t\t\t\\Lag_{\\rho_{t}}\\left(z^{+} , \\lambda\\right) - \\Lag_{\\rho_{t}}\\left(\\xi , \\lambda\\right) \\leq \\tau_{t}\\Delta_{P}\\left(\\xi , z , z^{+}\\right) - \\frac{\\tau_{t}}{2}\\norm{z^{+} - z}_{Q}^{2} - \\frac{\\sigma}{2}\\norm{\\xi - z^{+}}^{2} - \\frac{\\delta\\rho_{t}}{2}\\norm{\\AAA z^{+} - b}^{2}.\n\t\t\\end{equation}\t\t\t\t\n\t\\end{definition}\n\n    \\begin{center}\n    \t\t\\fbox{\\parbox{16.5cm}{{\\bf FLAG -- Faster LAGrangian based method}\n\t\t\t\\begin{enumerate}\n\t\t\t\t\\item[1.] {\\bf Input:} The problem data $[\\Psi, \\AAA, b, \\sigma]$, and a \\nice primal algorithmic map $\\SSS_{t}\\left(\\cdot\\right)$.\n\t\t\t\t\\item[2.] {\\bf Initialization:} Set $\\mu \\in \\left(0 , \\delta\\right]$ and $\\rho > 0$. Start with any $\\left(x^{0} , z^{0} , y^{0}\\right) \\in \\rr^{n} \\times \\rr^{n} \\times \\rr^{m}$, $t_{0} = 1$.\n            \t\t\\item[3.] {\\bf Iterations:} Sequences $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ and $\\seq{t}{k}$ are generated as follows: for each $k \\geq 0$\n                \t\t\\begin{enumerate}\n                    \t\t\\item[3.1.] If $\\sigma = 0$, let $\\rho_{k} = \\rho$, or, if $\\sigma > 0$, let $\\rho_{k} = \\rho t_{k}$. Compute\n                    \t\t\t\\begin{equation} \\label{GL:Lambda}\n                    \t\t\t\t\\lambda^{k} = y^{k} + \\rho_{k}\\left(t_{k} - 1\\right)\\left(\\AAA x^{k} -b\\right).\n                    \t\t\t\\end{equation}    \t\t\n                    \t\t\\item[3.2.] Update the sequence $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ by\n                       \t\t\\begin{align}\n                           \t\tz^{k + 1} & \\in \\SSS_{t_{k}}\\left(z^{k} , \\lambda^{k}\\right), \\label{FGL:PriStep} \\\\\n                           \t\ty^{k + 1} & = y^{k} + \\mu\\rho_{k}\\left(\\AAA z^{k + 1} - b\\right), \\label{FGL:MultiStep} \\\\\n                           \t\tx^{k + 1} & = \\left(1 - t_{k}^{-1}\\right)x^{k} + t_{k}^{-1}z^{k + 1}. \\label{FGL:AccStep}\n                       \t\t\\end{align}\n \t\t\t\t\t\t\\item[3.3.] Update the sequence $\\seq{t}{k}$ by solving the equation $t_{k + 1}^{p} - t_{k}^{p} = t_{k + 1}^{p - 1}$, \\ie\n                       \t\t\\begin{equation} \\label{FGL:Tk}\n                        \t\t\tt_{k + 1} = \\left\\{\\begin{matrix}\n                        \t\t\tt_{k}  + 1, & \\hspace{-0.6in} p = 1 \\,\\, \\text{(convex case)}, \\vspace{0.1in} \\\\\n                        \t\t\t(1 + \\sqrt{1 + 4t_{k}^{2}})/2, & p =2 \\,\\, \\text{(strongly convex case)}.\n                        \t\t\t\\end{matrix} \\right.\n\t\t\t\t\t\t\t\\end{equation}\n                    \t\t\\end{enumerate}\n\t\t\t\\end{enumerate}\\vspace{-0.2in}}}\n\t\\end{center}\n\n\n\tIn FLAG, once the \\nice algorithmic map $\\SSS_{t}\\left(\\cdot\\right)$ has been chosen, \\ie a parameter $\\delta \\in \\left(0 , 1\\right]$ and matrices $P , Q \\in \\bbS_{+}^{n}$, have been determined, Definition \\ref{D:AlgoMapNice} translates in terms of the iterative sequence $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ produced by FLAG as follows: for each $k \\geq 0$ and $\\xi \\in \\FFF$,\n\n\n\\begin{align}\n\t\t\\Lag_{\\rho_{k}}\\left(z^{k + 1} , \\lambda^{k}\\right) - \\Lag_{\\rho_{k}}\\left(\\xi , \\lambda^{k}\\right) & \\leq \\tau_{k}\\Delta_{P}\\left(\\xi , z^{k} , z^{k + 1}\\right) - \\frac{\\tau_{k}}{2}\\norm{z^{k + 1} - z^{k}}_{Q}^{2}  - \\frac{\\sigma}{2}\\norm{\\xi- z^{k + 1}}^{2} \\nonumber \\\\\n\t\t& - \\frac{\\delta\\rho_{k}}{2}\\norm{\\AAA z^{k + 1} - b}^{2}, \\label{AlgoMapNiceProperty}\n\t\\end{align}\n\n\twhere $\\tau_{k} = 1$ and $\\rho_{k} = \\rho$ (when $\\sigma = 0$) or $\\tau_{k} = t_{k}$ and $\\rho_{k} = \\rho t_{k}$ (when $\\sigma > 0$).\n\n\n\tBefore proceeding, we recall the following fundamental {\\em Pythagoras three-points identity} that is frequently used below. Given any matrix $P \\in \\bbS_{+}^{n}$, we have\n\n\n\\begin{equation} \\label{Pyth}\n\t\t2\\act{u - w , P\\left(w - v\\right)} =  \\norm{u - v}_{P}^{2}- \\norm{u - w}_{P}^{2} - \\norm{v - w}_{P}^{2}, \\quad \\forall \\,\\, u , v , w \\in \\real^{n}.\n\t\\end{equation}\n\n\tWe also recall the main parameter sequences introduced in FLAG and which, for convenience, will be used often and systematically as follows:\n   \\begin{equation*}\n    \t\t\\rho_{k} = \\rho t_{k}^{p - 1} ,\\; (\\rho>0)\\; \\quad \\text{and} \\quad t_{k}^{p} - t_{k - 1}^{p} = t_{k}^{p - 1},\\; (t_{-1} \\equiv 0 \\;\\text{and}\\; t_0 = 1),\n\t\\end{equation*}\t\n\twhere $p = 1$ (if $\\sigma = 0$) or $p = 2$ (if $\\sigma > 0$), and $\\tau_{k} = 1$ if $\\sigma = 0$, or $\\tau_{k} = t_{k}$ if $\\sigma > 0$. We will also often use the following expression. For any $k\\geq 0$ and any $\\xi \\in {\\cal F}$ we define,\n\n\n\\begin{equation}\\label{laggap}\n\t\ts_{k} := \\Lag_{\\rho t_{k - 1}^{p}}\\left(x^{k} , \\eta\\right) - \\Lag_{\\rho t_{k - 1}^{p}}\\left(\\xi , \\eta\\right) \\equiv \\Lag_{\\rho t_{k - 1}^{p}}\\left(x^{k} , \\eta\\right) - \\Psi(\\xi).\n\t \\end{equation}\n\n\tThroughout the rest of this section, recalling the fact that the set of optimal dual solutions is compact (see Section \\ref{Sec:Form}), we take $c > 0$ to be a constant for which $c \\geq 2\\norm{y^{\\ast}}$, where $y^{\\ast}$ is an optimal solution of the dual problem.\n","units":[{"start":12937,"end":13017,"start_line":131,"end_line":131,"raw_tex":"\tIn this paper, we consider the linearly constrained convex optimization model,\n","sha256_utf8":"5e2a842966d59be0a498190c55db3ee0865377352885fd3fa69ac7855e044e6b"},{"start":13018,"end":13155,"start_line":132,"end_line":134,"raw_tex":"\\begin{equation*}\n\t\t\\mbox{(P)} \\quad\\quad \\min  \\left\\{ \\Psi\\left(x\\right) \\, : \\, \\AAA x = b, \\; x \\in \\real^n\\right\\},\n\t\\end{equation*}","sha256_utf8":"45bf5b1e239fd483804e2ae205a2d1e01871c3f691887865bdfbdc4105118cd2"},{"start":13156,"end":14304,"start_line":135,"end_line":138,"raw_tex":"\twhere $\\Psi:\\real^{n} \\rightarrow \\erl$ is a proper, lower semi-continuous and convex function, $\\AAA: \\real^n\\to \\real^m$ is a linear mapping, and $b \\in \\real^{m}$. Throughout this work, the feasible set of problem (P) is denoted by  $\\FFF = \\left\\{ x \\in \\rr^{n} : \\, \\AAA x = b \\right\\}$.\n\\medskip\n\t\n\t Despite its apparent simplicity, with an additional structural/data information on the objective function or/and on the linear mapping (see examples below), the optimization model (P) essentially captures a very wide range of convex optimization problems arising in many modern applications. Since we are interested in analyzing this model also in the strongly convex case, we instead assume that $\\Psi\\left(\\cdot\\right)$ is $\\sigma$-strongly convex with $\\sigma \\geq 0$ (\\ie when $\\sigma = 0$ we recover the convex setting). At this juncture, it is important to mention that in many applied problems the function $\\Psi$ is defined by the sum of two or more functions, see \\eg \\cite{CP2016-B}. In such cases,  it is enough to assume that {\\em one} of the functions is $\\sigma$-strongly convex, see Section \\ref{SSec:Block} for more details.\n","sha256_utf8":"90fd9a420174f44e7b47c97ca9c81de6d880d572a2e5f175fadf1a044b1d738f"},{"start":17236,"end":17460,"start_line":167,"end_line":167,"raw_tex":"\tWe conclude this section with some basic properties of Lagrangians in convex optimization. For the equality constrained problem given in model (P), the {\\em Lagrangian} $\\Lag: \\rr^{n} \\times \\rr^{m} \\to \\erl$ is defined by\n","sha256_utf8":"e108869db10bdcf10bc968611700fbb068ff30e5f8807c989c92725cdb20388f"},{"start":17461,"end":17582,"start_line":168,"end_line":170,"raw_tex":"\\begin{equation} \\label{D:Lag}\n\t\t\\Lag\\left(x , y\\right) \\equiv \\Psi\\left(x\\right) + \\act{y , \\AAA x - b},\n\t\\end{equation}","sha256_utf8":"5f9a7cbfe614826c9fb7fe804dd63922e69472e5d7fe1a270fbe8ca158fe7e69"},{"start":17583,"end":17808,"start_line":171,"end_line":171,"raw_tex":"\twhere $y \\in \\real^{m}$ is the {\\em Lagrange multiplier} associated with the linear constraint $\\AAA x = b$. In this study, we will also use the \\textit{augmented Lagrangian} associated to problem (P), which is defined by \t\n","sha256_utf8":"6a756fc6c6073e70f6a3c9c70b1003c38d1365da742d7d62d26ff46704d401c8"},{"start":17809,"end":18037,"start_line":172,"end_line":174,"raw_tex":"\\begin{equation} \\label{D:AugLA}\n\t\t\\Lag_{\\rho}\\left(x ,  y\\right) := \\Lag\\left(x , y\\right) + \\frac{\\rho}{2}\\norm{\\AAA x - b}^{2} = \\Psi\\left(x\\right) + \\act{y , \\AAA x - b} + \\frac{\\rho}{2}\\norm{\\AAA x - b}^{2},\n\t\\end{equation}","sha256_utf8":"03f9ee54c061248f25b124c49afaea46ecfe48e42eaeeeee2996671c6e5e3b11"},{"start":18038,"end":18173,"start_line":175,"end_line":175,"raw_tex":"\twhere $\\rho > 0$ is a {\\em penalty parameter}. Clearly, when $\\rho = 0$ we recover the Lagrangian itself, \\ie $\\Lag \\equiv \\Lag_{0}$.\n","sha256_utf8":"e2690b65128788da1a5fb1c6777d00715ecfea6e5d7c05262d919173ce606660"},{"start":18249,"end":18659,"start_line":179,"end_line":184,"raw_tex":"\\begin{assumption} \\label{A:ConvLag}\n\t\tThe Lagrangian $\\Lag$ has a saddle point, that is, there exists a pair $\\left(x^{\\ast} , y^{\\ast}\\right)$ such that\n\t\t\\begin{equation} \\label{A:ConvLag:1}\n\t\t\t\\Lag\\left(x^{\\ast} , y\\right) \\leq \\Lag\\left(x^{\\ast} , y^{\\ast}\\right) \\leq \\Lag\\left(x , y^{\\ast}\\right), \\quad \\forall \\,\\, x \\in \\real^{n}, \\quad \\forall \\,\\, y\\in \\real^{m}.\n\t\t\\end{equation}\n\t\\end{assumption}","sha256_utf8":"030f92eac44b146f5b3ac99a347421ef4035e0473c4705ad19453f58c8ba2beb"},{"start":20003,"end":20435,"start_line":197,"end_line":197,"raw_tex":"\tWe end this section with some notations that will be used throughout the paper. We denote by $\\bbS_{+}^{n}$ and $\\bbS_{++}^{n}$ the sets of all $n \\times n$ {\\em positive semi-definite matrices} and {\\em positive definite matrices}, respectively. Given $P \\in \\bbS_{+}^{n}$, $\\norm{u}_{P} := \\sqrt{\\act{u , Pu}}$ stands for the semi-norm of $u \\in \\real^{n}$. For any $u , v , w \\in \\real^{n}$, we define the following quantity \t\t\n","sha256_utf8":"b00d99d78f5a31c97dbbd66f639624cdbfdea09bc35a90c24901a792b43a143f"},{"start":20436,"end":20591,"start_line":198,"end_line":200,"raw_tex":"\\begin{equation} \\label{D:Delta}\n\t\t\\Delta_{P}\\left(u , v , w\\right) := \\frac{1}{2}\\left(\\norm{u - v}_{P}^{2} - \\norm{u - w}_{P}^{2}\\right).\n\t\\end{equation}","sha256_utf8":"99602ce2dbfa4b55f4a7982bd4847a1bf43e4089fa59028300a53fba9de512cf"},{"start":20592,"end":20722,"start_line":201,"end_line":201,"raw_tex":"\tIn the case that $P = I$, where $I_{n}$ denotes the $n \\times n$ {\\em identity matrix}, we simply denote $\\Delta := \\Delta_{P}$.\n","sha256_utf8":"0a52df2ed32855d7d2ee6ae0e4310ed6dbcbf9cc8961feccc21d1229522a2257"},{"start":22814,"end":23839,"start_line":213,"end_line":218,"raw_tex":"\\begin{definition}[Nice primal algorithmic map] \\label{D:AlgoMapNice}\n\t\tGiven the parameters $\\rho , t > 0$, we let $\\rho_{t} = \\rho$ and $\\tau_{t} = 1$ (when $\\sigma = 0$) or $\\rho_{t} = \\rho t$ and $\\tau_{t} = t$ (when $\\sigma > 0$). A primal algorithmic map $\\SSS_{t} : \\real^{n} \\times \\real^{m} \\rightarrow \\real^{n}$, which is applied on the augmented Lagrangian $\\Lag_{\\rho_{t}}\\left(z , \\lambda\\right)$, that generates $z^{+}$ via $z^{+} \\in \\SSS_{t}\\left(z , \\lambda\\right)$, is called {\\em \\nice}\\hspace{-0.05in} if there exist a parameter $\\delta \\in \\left(0 , 1\\right]$ and matrices $P , Q \\in \\bbS_{+}^{n}$, such that for any $\\xi \\in \\FFF$ we have\n\t\t\\begin{equation} \\label{D:AlgoMapNice:1}\n\t\t\t\\Lag_{\\rho_{t}}\\left(z^{+} , \\lambda\\right) - \\Lag_{\\rho_{t}}\\left(\\xi , \\lambda\\right) \\leq \\tau_{t}\\Delta_{P}\\left(\\xi , z , z^{+}\\right) - \\frac{\\tau_{t}}{2}\\norm{z^{+} - z}_{Q}^{2} - \\frac{\\sigma}{2}\\norm{\\xi - z^{+}}^{2} - \\frac{\\delta\\rho_{t}}{2}\\norm{\\AAA z^{+} - b}^{2}.\n\t\t\\end{equation}\t\t\t\t\n\t\\end{definition}","sha256_utf8":"e81c0823bbbff82ee03e68595b3a07ff450c1d5c175e67ef171b32483108a7a1"},{"start":24250,"end":26418,"start_line":222,"end_line":248,"raw_tex":"    \\begin{center}\n    \t\t\\fbox{\\parbox{16.5cm}{{\\bf FLAG -- Faster LAGrangian based method}\n\t\t\t\\begin{enumerate}\n\t\t\t\t\\item[1.] {\\bf Input:} The problem data $[\\Psi, \\AAA, b, \\sigma]$, and a \\nice primal algorithmic map $\\SSS_{t}\\left(\\cdot\\right)$.\n\t\t\t\t\\item[2.] {\\bf Initialization:} Set $\\mu \\in \\left(0 , \\delta\\right]$ and $\\rho > 0$. Start with any $\\left(x^{0} , z^{0} , y^{0}\\right) \\in \\rr^{n} \\times \\rr^{n} \\times \\rr^{m}$, $t_{0} = 1$.\n            \t\t\\item[3.] {\\bf Iterations:} Sequences $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ and $\\seq{t}{k}$ are generated as follows: for each $k \\geq 0$\n                \t\t\\begin{enumerate}\n                    \t\t\\item[3.1.] If $\\sigma = 0$, let $\\rho_{k} = \\rho$, or, if $\\sigma > 0$, let $\\rho_{k} = \\rho t_{k}$. Compute\n                    \t\t\t\\begin{equation} \\label{GL:Lambda}\n                    \t\t\t\t\\lambda^{k} = y^{k} + \\rho_{k}\\left(t_{k} - 1\\right)\\left(\\AAA x^{k} -b\\right).\n                    \t\t\t\\end{equation}    \t\t\n                    \t\t\\item[3.2.] Update the sequence $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ by\n                       \t\t\\begin{align}\n                           \t\tz^{k + 1} & \\in \\SSS_{t_{k}}\\left(z^{k} , \\lambda^{k}\\right), \\label{FGL:PriStep} \\\\\n                           \t\ty^{k + 1} & = y^{k} + \\mu\\rho_{k}\\left(\\AAA z^{k + 1} - b\\right), \\label{FGL:MultiStep} \\\\\n                           \t\tx^{k + 1} & = \\left(1 - t_{k}^{-1}\\right)x^{k} + t_{k}^{-1}z^{k + 1}. \\label{FGL:AccStep}\n                       \t\t\\end{align}\n \t\t\t\t\t\t\\item[3.3.] Update the sequence $\\seq{t}{k}$ by solving the equation $t_{k + 1}^{p} - t_{k}^{p} = t_{k + 1}^{p - 1}$, \\ie\n                       \t\t\\begin{equation} \\label{FGL:Tk}\n                        \t\t\tt_{k + 1} = \\left\\{\\begin{matrix}\n                        \t\t\tt_{k}  + 1, & \\hspace{-0.6in} p = 1 \\,\\, \\text{(convex case)}, \\vspace{0.1in} \\\\\n                        \t\t\t(1 + \\sqrt{1 + 4t_{k}^{2}})/2, & p =2 \\,\\, \\text{(strongly convex case)}.\n                        \t\t\t\\end{matrix} \\right.\n\t\t\t\t\t\t\t\\end{equation}\n                    \t\t\\end{enumerate}\n\t\t\t\\end{enumerate}\\vspace{-0.2in}}}\n\t\\end{center}\n","sha256_utf8":"3f8de3e95c28a9f84d4a16114603906e5e9769b116b4bc0b6a462cbb6923351c"},{"start":27735,"end":28143,"start_line":258,"end_line":258,"raw_tex":"\tIn FLAG, once the \\nice algorithmic map $\\SSS_{t}\\left(\\cdot\\right)$ has been chosen, \\ie a parameter $\\delta \\in \\left(0 , 1\\right]$ and matrices $P , Q \\in \\bbS_{+}^{n}$, have been determined, Definition \\ref{D:AlgoMapNice} translates in terms of the iterative sequence $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ produced by FLAG as follows: for each $k \\geq 0$ and $\\xi \\in \\FFF$,\n","sha256_utf8":"b5980546cadd271552eae5f6b155e60790e8b9626acb159865ffb7c32984dfd1"},{"start":28144,"end":28534,"start_line":259,"end_line":262,"raw_tex":"\\begin{align}\n\t\t\\Lag_{\\rho_{k}}\\left(z^{k + 1} , \\lambda^{k}\\right) - \\Lag_{\\rho_{k}}\\left(\\xi , \\lambda^{k}\\right) & \\leq \\tau_{k}\\Delta_{P}\\left(\\xi , z^{k} , z^{k + 1}\\right) - \\frac{\\tau_{k}}{2}\\norm{z^{k + 1} - z^{k}}_{Q}^{2}  - \\frac{\\sigma}{2}\\norm{\\xi- z^{k + 1}}^{2} \\nonumber \\\\\n\t\t& - \\frac{\\delta\\rho_{k}}{2}\\norm{\\AAA z^{k + 1} - b}^{2}, \\label{AlgoMapNiceProperty}\n\t\\end{align}","sha256_utf8":"096f7d9baf38022613aafe7e0bb20835a7d611d57e00ce6f3fadfa828d86f906"},{"start":28537,"end":28672,"start_line":263,"end_line":263,"raw_tex":"\twhere $\\tau_{k} = 1$ and $\\rho_{k} = \\rho$ (when $\\sigma = 0$) or $\\tau_{k} = t_{k}$ and $\\rho_{k} = \\rho t_{k}$ (when $\\sigma > 0$).\n","sha256_utf8":"6772e6485d972856f37bc57e021b469b9e52999a966fb7f878e232f3bc64d156"},{"start":29882,"end":30055,"start_line":272,"end_line":272,"raw_tex":"\tBefore proceeding, we recall the following fundamental {\\em Pythagoras three-points identity} that is frequently used below. Given any matrix $P \\in \\bbS_{+}^{n}$, we have\n","sha256_utf8":"0dffd7a28355788f7c685152d93ce0dd5c8badcd678b565ccc62b8da307fc18c"},{"start":30056,"end":30252,"start_line":273,"end_line":275,"raw_tex":"\\begin{equation} \\label{Pyth}\n\t\t2\\act{u - w , P\\left(w - v\\right)} =  \\norm{u - v}_{P}^{2}- \\norm{u - w}_{P}^{2} - \\norm{v - w}_{P}^{2}, \\quad \\forall \\,\\, u , v , w \\in \\real^{n}.\n\t\\end{equation}","sha256_utf8":"dae84b2e696ff10649f2e5cd6c5ddae3953ba97f12346087babaae7560dcead1"},{"start":30254,"end":30839,"start_line":276,"end_line":280,"raw_tex":"\tWe also recall the main parameter sequences introduced in FLAG and which, for convenience, will be used often and systematically as follows:\n   \\begin{equation*}\n    \t\t\\rho_{k} = \\rho t_{k}^{p - 1} ,\\; (\\rho>0)\\; \\quad \\text{and} \\quad t_{k}^{p} - t_{k - 1}^{p} = t_{k}^{p - 1},\\; (t_{-1} \\equiv 0 \\;\\text{and}\\; t_0 = 1),\n\t\\end{equation*}\t\n\twhere $p = 1$ (if $\\sigma = 0$) or $p = 2$ (if $\\sigma > 0$), and $\\tau_{k} = 1$ if $\\sigma = 0$, or $\\tau_{k} = t_{k}$ if $\\sigma > 0$. We will also often use the following expression. For any $k\\geq 0$ and any $\\xi \\in {\\cal F}$ we define,\n","sha256_utf8":"9f4cd3dd1731d509ac4f6e5acd3a9025a7dda196638b0bd84d2e23080faa5543"},{"start":30840,"end":31071,"start_line":281,"end_line":283,"raw_tex":"\\begin{equation}\\label{laggap}\n\t\ts_{k} := \\Lag_{\\rho t_{k - 1}^{p}}\\left(x^{k} , \\eta\\right) - \\Lag_{\\rho t_{k - 1}^{p}}\\left(\\xi , \\eta\\right) \\equiv \\Lag_{\\rho t_{k - 1}^{p}}\\left(x^{k} , \\eta\\right) - \\Psi(\\xi).\n\t \\end{equation}","sha256_utf8":"cc92e0d403e33726be695c0828a3475dc54bfc75c2769562e1d71f564e01583b"},{"start":40533,"end":40801,"start_line":382,"end_line":382,"raw_tex":"\tThroughout the rest of this section, recalling the fact that the set of optimal dual solutions is compact (see Section \\ref{Sec:Form}), we take $c > 0$ to be a constant for which $c \\geq 2\\norm{y^{\\ast}}$, where $y^{\\ast}$ is an optimal solution of the dual problem.\n","sha256_utf8":"c10d888eaebc0b99192dea74544af5b32d55fe65b0d0f68e2729193d553a97b1"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}[A fast non-ergodic function values and feasibility violation rates] \\label{T:FGLrate}\n\t\tLet $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ be a sequence generated by FLAG. Suppose that $\\sigma > 0$ and $0 \\preceq P \\preceq \\left(\\sigma/2\\right)I_{n}$. Then, for any optimal solution $x^{\\ast}$ of problem (P), we have\n   \t\t\\begin{align}\n\t\t\t\\Psi\\left(x^{N}\\right) - \\Psi\\left(x^{\\ast}\\right) & \\leq \\frac{B_{\\rho , c}\\left(x^{\\ast}\\right)}{2N^{2}}, \\label{T:FGLrate:1} \\\\\n       \t\t\\norm{\\AAA x^{N} - b} & \\leq \\frac{B_{\\rho , c}\\left(x^{\\ast}\\right)}{cN^{2}}, \\label{T:FGLrate:2}\n\t\t\\end{align}\n\t\twhere $B_{\\rho , c}\\left(x^{\\ast}\\right) := 4\\left(\\norm{x^{\\ast} - z^{0}}_{P}^{2} + \\frac{1}{\\mu\\rho}\\left(\\norm{y^{0}} + c\\right)^{2}\\right)$.\n\t\\end{theorem}","units":[{"start":40924,"end":41713,"start_line":386,"end_line":393,"raw_tex":"\\begin{theorem}[A fast non-ergodic function values and feasibility violation rates] \\label{T:FGLrate}\n\t\tLet $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ be a sequence generated by FLAG. Suppose that $\\sigma > 0$ and $0 \\preceq P \\preceq \\left(\\sigma/2\\right)I_{n}$. Then, for any optimal solution $x^{\\ast}$ of problem (P), we have\n   \t\t\\begin{align}\n\t\t\t\\Psi\\left(x^{N}\\right) - \\Psi\\left(x^{\\ast}\\right) & \\leq \\frac{B_{\\rho , c}\\left(x^{\\ast}\\right)}{2N^{2}}, \\label{T:FGLrate:1} \\\\\n       \t\t\\norm{\\AAA x^{N} - b} & \\leq \\frac{B_{\\rho , c}\\left(x^{\\ast}\\right)}{cN^{2}}, \\label{T:FGLrate:2}\n\t\t\\end{align}\n\t\twhere $B_{\\rho , c}\\left(x^{\\ast}\\right) := 4\\left(\\norm{x^{\\ast} - z^{0}}_{P}^{2} + \\frac{1}{\\mu\\rho}\\left(\\norm{y^{0}} + c\\right)^{2}\\right)$.\n\t\\end{theorem}","sha256_utf8":"ed41a025f1c33bde99b8940d7f76f3fad50663432f4b1b9308c85846d48d029d"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma} \\label{L:FGLIter}\n\t\tLet $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ be a sequence generated by FLAG. Then, for any $\\xi \\in \\FFF$, $\\eta \\in \\real^{m}$ and $k \\geq 0$, we have\n\t\t\\begin{align}\n\t\t\t\\Lag_{\\rho_{k}}\\left(z^{k + 1} , \\eta\\right) - \\Lag_{\\rho_{k}}\\left(\\xi , \\eta\\right) & \\leq \\tau_{k}\\Delta_{P}\\left(\\xi , z^{k} , z^{k + 1}\\right) - \\frac{\\sigma}{2}\\norm{\\xi - z^{k + 1}}^{2} + \\frac{1}{\\mu\\rho_{k}}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right) \\nonumber \\\\\n\t\t\t& - \\rho t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b}. \\label{L:FGLIter:0}\n\t\t\\end{align}\t\t\n\t\\end{lemma}\n\n\\begin{proof}\n\t\tSince $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ is generated by FLAG, we obtain from \\eqref{AlgoMapNiceProperty} after omitting the non-negative term $\\left(\\tau_{k}/2\\right)\\norm{z^{k + 1} - z^{k}}_{Q}^{2}$, that\n\t\t\\begin{equation} \\label{L:FGLIter:1}\n\t\t\t\\Lag_{\\rho_{k}}\\left(z^{k + 1} , \\lambda^{k}\\right) - \\Lag_{\\rho_{k}}\\left(\\xi , \\lambda^{k}\\right) \\leq \\tau_{k}\\Delta_{P}\\left(\\xi , z^{k} , z^{k + 1}\\right) - \\frac{\\sigma}{2}\\norm{\\xi - z^{k + 1}}^{2} - \\frac{\\delta\\rho_{k}}{2}\\norm{\\AAA z^{k + 1} - b}^{2}.\n\t\t\\end{equation}\n\t\tFrom the multiplier update \\eqref{FGL:MultiStep} and the three-points identity \\eqref{Pyth}, we obtain, for all $\\eta \\in \\real^{m}$,\n\t\t\\begin{align}\n\t\t\t\\act{\\eta - y^{k} , \\AAA z^{k + 1} - b}  = \\frac{1}{\\mu\\rho_{k}}\\act{\\eta - y^{k} , y^{k + 1} - y^{k}} & = \\frac{1}{\\mu\\rho_{k}}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right) + \\frac{1}{2\\mu\\rho_{k}}\\norm{y^{k + 1} - y^{k}}^{2} \\nonumber \\\\\n\t\t\t& = \\frac{1}{\\mu\\rho_{k}}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right) + \\frac{\\mu\\rho_{k}}{2}\\norm{\\AAA z^{k + 1} - b}^{2}. \\label{L:FGLIter:2}\n\t\t\\end{align}\n\t\tUsing the update rule of the sequence $\\seq{t}{k}$ (\\cf \\eqref{FGL:Tk}) we have that $\\rho_{k}\\left(t_{k} - 1\\right) =\\rho t_{k - 1}^{p} $ and thus\n\t\t\\begin{equation*}\n\t\t\t\\lambda^{k} = y^{k} + \\rho_{k}\\left(t_{k} - 1\\right)\\left(\\AAA x^{k} -b\\right) = y^{k} + \\rho t_{k - 1}^{p}\\left(\\AAA x^{k} -b\\right).\n\t\t\\end{equation*}\n\t\tUsing this together with \\eqref{L:FGLIter:2} yields, for all $\\eta \\in \\real^{m}$, that\n\t\t\\begin{align}\t\t\n\t\t\t\\act{\\eta - \\lambda^{k} , \\AAA z^{k + 1} - b} & = \\act{\\eta - y^{k} , \\AAA z^{k + 1} - b} - \\rho t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b} \\nonumber \\\\\n\t\t\t& = \\frac{1}{\\mu\\rho_{k}}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right) + \\frac{\\mu\\rho_{k}}{2}\\norm{\\AAA z^{k + 1} - b}^{2} - \\rho t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b}. \\label{L:FGLIter:3}\n\t\t\\end{align}\n\t\tAdding \\eqref{L:FGLIter:1} to \\eqref{L:FGLIter:3} yields (recall that $\\mu \\leq \\delta$) the desired result.\n\t\\end{proof}\n\n\\begin{lemma} \\label{L:FGLMain}\n\t\tLet $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ be a sequence generated by FLAG. Then, for any $\\xi \\in \\FFF$, $\\eta \\in \\real^{m}$ and $k \\geq 0$, we have\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{p}s_{k + 1} - t_{k - 1}^{p}s_{k} \\leq \\frac{\\tau_{k}\\rho_{k}}{\\rho}\\Delta_{P}\\left(\\xi , z^{k} , z^{k + 1}\\right) - \\frac{\\rho_{k}\\sigma}{2\\rho}\\norm{\\xi - z^{k + 1}}^{2} + \\frac{1}{\\mu\\rho}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right),\n\t\t\\end{equation*}\n\t\twhere $s_{k} = \\Lag_{\\rho t_{k - 1}^{p}}\\left(x^{k} , \\eta\\right) - \\Lag_{\\rho t_{k - 1}^{p}}\\left(\\xi , \\eta\\right)$ (\\cf \\ref{laggap}).\n\t\\end{lemma}\n\n\\begin{proof}\n\t\tSince $x^{k + 1} = \\left(1 - t_{k}^{-1}\\right)x^{k} + t_{k}^{-1}z^{k + 1}$ (\\cf \\eqref{FGL:AccStep}) we obtain from the convexity of $\\Psi\\left(\\cdot\\right)$ that\n\t\t\\begin{equation*}\n\t\t\t\\Psi\\left(x^{k + 1}\\right) \\leq \\left(1 - t_{k}^{-1}\\right)\\Psi\\left(x^{k}\\right) + t_{k}^{-1}\\Psi\\left(z^{k + 1}\\right).\n\t\t\\end{equation*}\n\t\tTherefore, multiplying both sides by $t_{k}^{p}$ (recalling that $t_{k}^{p} - t_{k}^{p - 1} = t_{k - 1}^{p}$), we obtain\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{p}\\left(\\Psi\\left(x^{k + 1}\\right) - \\Psi\\left(\\xi\\right)\\right) - t_{k - 1}^{p}\\left(\\Psi\\left(x^{k}\\right) - \\Psi\\left(\\xi\\right)\\right) \\leq t_{k}^{p - 1}\\left(\\Psi\\left(z^{k + 1}\\right) - \\Psi\\left(\\xi\\right)\\right).\n\t\t\\end{equation*}\n\t\tIn addition, using again \\eqref{FGL:AccStep}, yields that\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{p}\\act{\\eta , \\AAA x^{k + 1} - b} - t_{k - 1}^{p}\\act{\\eta , \\AAA x^{k} - b} = t_{k}^{p - 1}\\act{\\eta , \\AAA z^{k + 1} - b}.\n\t\t\\end{equation*}\t\t\t\n\t\tCombining these two facts yields (using the definition of the Lagrangian $\\Lag$)\n\t\t\\begin{equation} \\label{L:FGLMain:1}\n\t\t\tt_{k}^{p}\\left(\\Lag\\left(x^{k + 1} , \\eta\\right) - \\Lag\\left(\\xi , \\eta\\right)\\right) - t_{k - 1}^{p}\\left(\\Lag\\left(x^{k} , \\eta\\right) - \\Lag\\left(\\xi , \\eta\\right)\\right) \\leq t_{k}^{p - 1}\\left(\\Lag\\left(z^{k + 1} , \\eta\\right) - \\Lag\\left(\\xi , \\eta\\right)\\right).\n\t\t\\end{equation}\t\t\t\n\t\tNow, we will again use the relation $x^{k + 1} = \\left(1 - t_{k}^{-1}\\right)x^{k} + t_{k}^{-1}z^{k + 1}$ to obtain\n\t\t\\begin{equation*}\n\t\t\t\\norm{\\AAA x^{k + 1} - b}^{2}  = \\left(1 - t_{k}^{-1}\\right)^{2}\\norm{\\AAA x^{k} - b}^{2} + t_{k}^{-2}\\norm{\\AAA z^{k + 1} - b}^{2} + 2\\left(1 - t_{k}^{-1}\\right)t_{k}^{-1}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b}.\n\t\t\\end{equation*}\t\t\n\t\tMultiplying both sides of the above equality by $\\rho t_{k}^{2p}/2$ (recalling $\\rho_{k} = \\rho t_{k}^{p - 1}$ and $t_{k}^{p} - t_{k}^{p - 1} = t_{k - 1}^{p}$) yields\n\t\t\\begin{align}\n\t\t\t\\frac{\\rho t_{k}^{2p}}{2}\\norm{\\AAA x^{k + 1} - b}^{2} - \\frac{\\rho t_{k - 1}^{2p}}{2}\\norm{\\AAA x^{k} - b}^{2} = \\frac{\\rho_{k}t_{k}^{p - 1}}{2}\\norm{\\AAA z^{k + 1} - b}^{2} + \\rho_{k}t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b}. \\label{L:FGLMain:2}\n\t\t\\end{align}\n\t\tTherefore, adding \\eqref{L:FGLMain:1} to \\eqref{L:FGLMain:2} yields (recall that $\\norm{\\AAA\\xi - b}^{2} = 0$) with $s_{k} = \\Lag_{\\rho t_{k - 1}^{p}}\\left(x^{k} , \\eta\\right) - \\Lag_{\\rho t_{k - 1}^{p}}\\left(\\xi , \\eta\\right)$,\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{p}s_{k + 1} - t_{k - 1}^{p}s_{k} \\leq t_{k}^{p - 1}\\left(\\Lag_{\\rho_{k}}\\left(z^{k + 1} , \\eta\\right) - \\Lag_{\\rho_{k}}\\left(\\xi , \\eta\\right)\\right) + \\rho_{k}t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b}.\n\t\t\\end{equation*}\t\t\t\n\t\tFrom Lemma \\ref{L:FGLIter}, after we multiplied both sides of \\eqref{L:FGLIter:0} by $t_{k}^{p - 1}$ (recall that $\\rho_{k} = \\rho t_{k}^{p - 1}$), we obtain that\n\t\t\\begin{align*}\n\t\t\tt_{k}^{p - 1}\\left(\\Lag_{\\rho_{k}}\\left(z^{k + 1} , \\eta\\right) - \\Lag_{\\rho_{k}}\\left(\\xi , \\eta\\right)\\right) + \\rho_{k}t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b} & \\leq \\frac{\\tau_{k}\\rho_{k}}{\\rho}\\Delta_{P}\\left(\\xi , z^{k} , z^{k + 1}\\right) - \\frac{\\rho_{k}\\sigma}{2\\rho}\\norm{\\xi - z^{k + 1}}^{2} \\\\\n\t\t\t& + \\frac{1}{\\mu\\rho}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right).\n\t\t\\end{align*}\t\t\n\t\tBy combining the last two inequalities, we thus get\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{p}s_{k + 1} - t_{k - 1}^{p}s_{k} \\leq \\frac{\\tau_{k}\\rho_{k}}{\\rho}\\Delta_{P}\\left(\\xi , z^{k} , z^{k + 1}\\right) - \\frac{\\rho_{k}\\sigma}{2\\rho}\\norm{\\xi - z^{k + 1}}^{2} + \\frac{1}{\\mu\\rho}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right),\n\t\t\\end{equation*}\t\t\t\t\n\t\twhich proves the desired result.\n\t\\end{proof}\n\n\\begin{lemma} \\label{L:Tk}\n\t\tLet $\\seq{t}{k}$ be the sequence of positive numbers generated from $t_{0} = 1$ via the equation $t_{k + 1}^{p} - t_{k}^{p} = t_{k + 1}^{p - 1}$, with $p = 1$ or $p = 2$. Then, for any $k \\geq 0$, we have\n\t\\begin{itemize}\n\t\t\\item[$\\rm{(i)}$] For $p = 1$, we have $t_{k} = k + 1$.\n\t\t\\item[$\\rm{(ii)}$] For $p = 2$, we have $t_{k} \\geq \\left(k + 1\\right)/2$. Moreover, one has $t_{k + 1}^{2} \\leq t_{k}^{2} + 2t_{k}$.\n\t\\end{itemize}\t\t\t\t\n\t\\end{lemma}\n\n\\begin{proof}\n\t\tThe first item is immediate as well as the first part of the second item (see, for instance, \\cite[Lemma 4.3]{BT2009}). Moreover, note that by the given recursion $t_{k + 1}^{2} - t_{k}^{2} = t_{k + 1}$, one also has $t_{k + 1} = t_{k} + t_{k + 1}/\\left(t_{k} + t_{k + 1}\\right)$, and hence it follows that $t_{k + 1}^{2} = t_{k}^{2} + t_{k + 1} \\leq t_{k}^{2} + 2t_{k}$, since $t_{k} \\geq 1$.\n\t\\end{proof}\n\n\\begin{proof}\n\t\tFrom Lemma \\ref{L:FGLMain}, after we substitute $\\xi = x^{\\ast}$ and $p = 2$ (recall that $\\rho_{k} = \\rho t_{k}$ and $\\tau_{k} = t_{k}$), we obtain\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{2}s_{k + 1} - t_{k - 1}^{2}s_{k} \\leq t_{k}^{2}\\Delta_{P}\\left(x^{\\ast} , z^{k} , z^{k + 1}\\right) - \\frac{t_{k}\\sigma}{2}\\norm{x^{\\ast} - z^{k + 1}}^{2} + \\frac{1}{\\mu\\rho}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right).\n\t\t\\end{equation*}\n\t\tUsing the definition of $\\Delta_{P}$ and $\\Delta$ (see \\eqref{D:Delta}) and the fact that $P \\preceq \\left(\\sigma/2\\right)I_{n}$ we have that\n\t\t\\begin{align*}\n\t\t\tt_{k}^{2}s_{k + 1} - t_{k - 1}^{2}s_{k} & \\leq \\frac{t_{k}^{2}}{2}\\left(\\norm{x^{\\ast} - z^{k}}_{P}^{2} - \\norm{x^{\\ast} - z^{k + 1}}_{P}^{2}\\right) - t_{k}\\norm{x^{\\ast} - z^{k + 1}}_{P}^{2} + \\frac{1}{2\\mu\\rho}\\left(\\norm{\\eta - y^{k}}^{2} -  \\norm{\\eta - y^{k + 1}}^{2}\\right) \\nonumber \\\\\n\t\t\t& \\leq \\frac{1}{2}\\left(t_{k}^{2}\\norm{x^{\\ast} - z^{k}}_{P}^{2} - t_{k + 1}^{2}\\norm{x^{\\ast} - z^{k + 1}}_{P}^{2}\\right) + \\frac{1}{2\\mu\\rho}\\left(\\norm{\\eta - y^{k}}^{2} - \\norm{\\eta - y^{k + 1}}^{2}\\right),\n\t\t\\end{align*}\n\t\twhere the last inequality follows from Lemma \\ref{L:Tk}(ii). Summing this inequality for all $k = 0 , 1 , \\ldots , N - 1$ (recall that $t_{-1} = 0$ and $t_{0} = 1$), it follows (where we use  the definition of $s_{N}$, see \\eqref{laggap}, for the first inequality below)\n\t\t\\begin{equation*}\n\tt_{N - 1}^{2}\\left(\\Psi\\left(x^{N}\\right) - \\Psi\\left(x^{\\ast}\\right) + \\act{\\eta , \\AAA x^{N} - b}\\right) \\leq\tt_{N - 1}^{2}s_{N} - t_{-1}^{2}s_{0} \\leq \\frac{1}{2}\\norm{x^{\\ast} - z^{0}}_{P}^{2} + \\frac{1}{2\\mu\\rho}\\norm{\\eta - y^{0}}^{2}.\n\t\t\\end{equation*}\n\t\tTherefore, by taking the maximum of both sides over $\\norm{\\eta} \\leq c$, proves that (recall that $t_{N - 1}^{2} \\geq N^{2}/4$ using Lemma \\ref{L:Tk}(ii))\n\t\t\\begin{equation*}\n\t\t\t\\Psi\\left(x^{N}\\right) - \\Psi\\left(x^{\\ast}\\right) + c\\norm{\\AAA x^{N} - b} \\leq \\frac{B_{\\rho , c}\\left(x^{\\ast}\\right)}{2N^{2}} := \\beta,\n\t\t\\end{equation*}\n\t\tfrom which the estimate \\eqref{T:FGLrate:1} immediately follows. Moreover, since $\\left(x^{\\ast} , y^{\\ast}\\right)$ is a saddle point of problem (P), with $c \\geq 2\\norm{y^{\\ast}}$, using the above inequality it follows that\n\t\t\\begin{equation*}\n\t\t\tc\\norm{\\AAA x^{N} - b} \\leq \\Psi\\left(x^{\\ast}\\right) - \\Psi\\left(x^{N}\\right) + \\beta \\leq \\act{y^{\\ast} , \\AAA x^{N} - b} + \\beta \\leq \\frac{c}{2}\\norm{\\AAA x^{N} - b} + \\beta,\n\t\t\\end{equation*}\n\t\tand the desired estimate \\eqref{T:FGLrate:2} is proved.\n\t\\end{proof}","units":[{"start":31144,"end":31764,"start_line":287,"end_line":293,"raw_tex":"\\begin{lemma} \\label{L:FGLIter}\n\t\tLet $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ be a sequence generated by FLAG. Then, for any $\\xi \\in \\FFF$, $\\eta \\in \\real^{m}$ and $k \\geq 0$, we have\n\t\t\\begin{align}\n\t\t\t\\Lag_{\\rho_{k}}\\left(z^{k + 1} , \\eta\\right) - \\Lag_{\\rho_{k}}\\left(\\xi , \\eta\\right) & \\leq \\tau_{k}\\Delta_{P}\\left(\\xi , z^{k} , z^{k + 1}\\right) - \\frac{\\sigma}{2}\\norm{\\xi - z^{k + 1}}^{2} + \\frac{1}{\\mu\\rho_{k}}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right) \\nonumber \\\\\n\t\t\t& - \\rho t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b}. \\label{L:FGLIter:0}\n\t\t\\end{align}\t\t\n\t\\end{lemma}","sha256_utf8":"1004d2ac2e8e8af827effbe1b2529c80728f952ca3f3e239dabc5695daf344e8"},{"start":31766,"end":33841,"start_line":294,"end_line":314,"raw_tex":"\\begin{proof}\n\t\tSince $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ is generated by FLAG, we obtain from \\eqref{AlgoMapNiceProperty} after omitting the non-negative term $\\left(\\tau_{k}/2\\right)\\norm{z^{k + 1} - z^{k}}_{Q}^{2}$, that\n\t\t\\begin{equation} \\label{L:FGLIter:1}\n\t\t\t\\Lag_{\\rho_{k}}\\left(z^{k + 1} , \\lambda^{k}\\right) - \\Lag_{\\rho_{k}}\\left(\\xi , \\lambda^{k}\\right) \\leq \\tau_{k}\\Delta_{P}\\left(\\xi , z^{k} , z^{k + 1}\\right) - \\frac{\\sigma}{2}\\norm{\\xi - z^{k + 1}}^{2} - \\frac{\\delta\\rho_{k}}{2}\\norm{\\AAA z^{k + 1} - b}^{2}.\n\t\t\\end{equation}\n\t\tFrom the multiplier update \\eqref{FGL:MultiStep} and the three-points identity \\eqref{Pyth}, we obtain, for all $\\eta \\in \\real^{m}$,\n\t\t\\begin{align}\n\t\t\t\\act{\\eta - y^{k} , \\AAA z^{k + 1} - b}  = \\frac{1}{\\mu\\rho_{k}}\\act{\\eta - y^{k} , y^{k + 1} - y^{k}} & = \\frac{1}{\\mu\\rho_{k}}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right) + \\frac{1}{2\\mu\\rho_{k}}\\norm{y^{k + 1} - y^{k}}^{2} \\nonumber \\\\\n\t\t\t& = \\frac{1}{\\mu\\rho_{k}}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right) + \\frac{\\mu\\rho_{k}}{2}\\norm{\\AAA z^{k + 1} - b}^{2}. \\label{L:FGLIter:2}\n\t\t\\end{align}\n\t\tUsing the update rule of the sequence $\\seq{t}{k}$ (\\cf \\eqref{FGL:Tk}) we have that $\\rho_{k}\\left(t_{k} - 1\\right) =\\rho t_{k - 1}^{p} $ and thus\n\t\t\\begin{equation*}\n\t\t\t\\lambda^{k} = y^{k} + \\rho_{k}\\left(t_{k} - 1\\right)\\left(\\AAA x^{k} -b\\right) = y^{k} + \\rho t_{k - 1}^{p}\\left(\\AAA x^{k} -b\\right).\n\t\t\\end{equation*}\n\t\tUsing this together with \\eqref{L:FGLIter:2} yields, for all $\\eta \\in \\real^{m}$, that\n\t\t\\begin{align}\t\t\n\t\t\t\\act{\\eta - \\lambda^{k} , \\AAA z^{k + 1} - b} & = \\act{\\eta - y^{k} , \\AAA z^{k + 1} - b} - \\rho t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b} \\nonumber \\\\\n\t\t\t& = \\frac{1}{\\mu\\rho_{k}}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right) + \\frac{\\mu\\rho_{k}}{2}\\norm{\\AAA z^{k + 1} - b}^{2} - \\rho t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b}. \\label{L:FGLIter:3}\n\t\t\\end{align}\n\t\tAdding \\eqref{L:FGLIter:1} to \\eqref{L:FGLIter:3} yields (recall that $\\mu \\leq \\delta$) the desired result.\n\t\\end{proof}","sha256_utf8":"ea8edced767fef02131980ad9d0eff69c85688169ca98365858fe24fa408d5fc"},{"start":34255,"end":34902,"start_line":316,"end_line":322,"raw_tex":"\\begin{lemma} \\label{L:FGLMain}\n\t\tLet $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ be a sequence generated by FLAG. Then, for any $\\xi \\in \\FFF$, $\\eta \\in \\real^{m}$ and $k \\geq 0$, we have\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{p}s_{k + 1} - t_{k - 1}^{p}s_{k} \\leq \\frac{\\tau_{k}\\rho_{k}}{\\rho}\\Delta_{P}\\left(\\xi , z^{k} , z^{k + 1}\\right) - \\frac{\\rho_{k}\\sigma}{2\\rho}\\norm{\\xi - z^{k + 1}}^{2} + \\frac{1}{\\mu\\rho}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right),\n\t\t\\end{equation*}\n\t\twhere $s_{k} = \\Lag_{\\rho t_{k - 1}^{p}}\\left(x^{k} , \\eta\\right) - \\Lag_{\\rho t_{k - 1}^{p}}\\left(\\xi , \\eta\\right)$ (\\cf \\ref{laggap}).\n\t\\end{lemma}","sha256_utf8":"02b00e6d2a09321d2ae9da247c7364201f5b0792e683d98e1844641a7d1550e6"},{"start":34905,"end":38605,"start_line":323,"end_line":362,"raw_tex":"\\begin{proof}\n\t\tSince $x^{k + 1} = \\left(1 - t_{k}^{-1}\\right)x^{k} + t_{k}^{-1}z^{k + 1}$ (\\cf \\eqref{FGL:AccStep}) we obtain from the convexity of $\\Psi\\left(\\cdot\\right)$ that\n\t\t\\begin{equation*}\n\t\t\t\\Psi\\left(x^{k + 1}\\right) \\leq \\left(1 - t_{k}^{-1}\\right)\\Psi\\left(x^{k}\\right) + t_{k}^{-1}\\Psi\\left(z^{k + 1}\\right).\n\t\t\\end{equation*}\n\t\tTherefore, multiplying both sides by $t_{k}^{p}$ (recalling that $t_{k}^{p} - t_{k}^{p - 1} = t_{k - 1}^{p}$), we obtain\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{p}\\left(\\Psi\\left(x^{k + 1}\\right) - \\Psi\\left(\\xi\\right)\\right) - t_{k - 1}^{p}\\left(\\Psi\\left(x^{k}\\right) - \\Psi\\left(\\xi\\right)\\right) \\leq t_{k}^{p - 1}\\left(\\Psi\\left(z^{k + 1}\\right) - \\Psi\\left(\\xi\\right)\\right).\n\t\t\\end{equation*}\n\t\tIn addition, using again \\eqref{FGL:AccStep}, yields that\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{p}\\act{\\eta , \\AAA x^{k + 1} - b} - t_{k - 1}^{p}\\act{\\eta , \\AAA x^{k} - b} = t_{k}^{p - 1}\\act{\\eta , \\AAA z^{k + 1} - b}.\n\t\t\\end{equation*}\t\t\t\n\t\tCombining these two facts yields (using the definition of the Lagrangian $\\Lag$)\n\t\t\\begin{equation} \\label{L:FGLMain:1}\n\t\t\tt_{k}^{p}\\left(\\Lag\\left(x^{k + 1} , \\eta\\right) - \\Lag\\left(\\xi , \\eta\\right)\\right) - t_{k - 1}^{p}\\left(\\Lag\\left(x^{k} , \\eta\\right) - \\Lag\\left(\\xi , \\eta\\right)\\right) \\leq t_{k}^{p - 1}\\left(\\Lag\\left(z^{k + 1} , \\eta\\right) - \\Lag\\left(\\xi , \\eta\\right)\\right).\n\t\t\\end{equation}\t\t\t\n\t\tNow, we will again use the relation $x^{k + 1} = \\left(1 - t_{k}^{-1}\\right)x^{k} + t_{k}^{-1}z^{k + 1}$ to obtain\n\t\t\\begin{equation*}\n\t\t\t\\norm{\\AAA x^{k + 1} - b}^{2}  = \\left(1 - t_{k}^{-1}\\right)^{2}\\norm{\\AAA x^{k} - b}^{2} + t_{k}^{-2}\\norm{\\AAA z^{k + 1} - b}^{2} + 2\\left(1 - t_{k}^{-1}\\right)t_{k}^{-1}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b}.\n\t\t\\end{equation*}\t\t\n\t\tMultiplying both sides of the above equality by $\\rho t_{k}^{2p}/2$ (recalling $\\rho_{k} = \\rho t_{k}^{p - 1}$ and $t_{k}^{p} - t_{k}^{p - 1} = t_{k - 1}^{p}$) yields\n\t\t\\begin{align}\n\t\t\t\\frac{\\rho t_{k}^{2p}}{2}\\norm{\\AAA x^{k + 1} - b}^{2} - \\frac{\\rho t_{k - 1}^{2p}}{2}\\norm{\\AAA x^{k} - b}^{2} = \\frac{\\rho_{k}t_{k}^{p - 1}}{2}\\norm{\\AAA z^{k + 1} - b}^{2} + \\rho_{k}t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b}. \\label{L:FGLMain:2}\n\t\t\\end{align}\n\t\tTherefore, adding \\eqref{L:FGLMain:1} to \\eqref{L:FGLMain:2} yields (recall that $\\norm{\\AAA\\xi - b}^{2} = 0$) with $s_{k} = \\Lag_{\\rho t_{k - 1}^{p}}\\left(x^{k} , \\eta\\right) - \\Lag_{\\rho t_{k - 1}^{p}}\\left(\\xi , \\eta\\right)$,\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{p}s_{k + 1} - t_{k - 1}^{p}s_{k} \\leq t_{k}^{p - 1}\\left(\\Lag_{\\rho_{k}}\\left(z^{k + 1} , \\eta\\right) - \\Lag_{\\rho_{k}}\\left(\\xi , \\eta\\right)\\right) + \\rho_{k}t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b}.\n\t\t\\end{equation*}\t\t\t\n\t\tFrom Lemma \\ref{L:FGLIter}, after we multiplied both sides of \\eqref{L:FGLIter:0} by $t_{k}^{p - 1}$ (recall that $\\rho_{k} = \\rho t_{k}^{p - 1}$), we obtain that\n\t\t\\begin{align*}\n\t\t\tt_{k}^{p - 1}\\left(\\Lag_{\\rho_{k}}\\left(z^{k + 1} , \\eta\\right) - \\Lag_{\\rho_{k}}\\left(\\xi , \\eta\\right)\\right) + \\rho_{k}t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b} & \\leq \\frac{\\tau_{k}\\rho_{k}}{\\rho}\\Delta_{P}\\left(\\xi , z^{k} , z^{k + 1}\\right) - \\frac{\\rho_{k}\\sigma}{2\\rho}\\norm{\\xi - z^{k + 1}}^{2} \\\\\n\t\t\t& + \\frac{1}{\\mu\\rho}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right).\n\t\t\\end{align*}\t\t\n\t\tBy combining the last two inequalities, we thus get\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{p}s_{k + 1} - t_{k - 1}^{p}s_{k} \\leq \\frac{\\tau_{k}\\rho_{k}}{\\rho}\\Delta_{P}\\left(\\xi , z^{k} , z^{k + 1}\\right) - \\frac{\\rho_{k}\\sigma}{2\\rho}\\norm{\\xi - z^{k + 1}}^{2} + \\frac{1}{\\mu\\rho}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right),\n\t\t\\end{equation*}\t\t\t\t\n\t\twhich proves the desired result.\n\t\\end{proof}","sha256_utf8":"b967cbd4fd0b27cffd5fa62d44654cb1deb49c37706c4e16982abe97caa92947"},{"start":38783,"end":39259,"start_line":364,"end_line":370,"raw_tex":"\\begin{lemma} \\label{L:Tk}\n\t\tLet $\\seq{t}{k}$ be the sequence of positive numbers generated from $t_{0} = 1$ via the equation $t_{k + 1}^{p} - t_{k}^{p} = t_{k + 1}^{p - 1}$, with $p = 1$ or $p = 2$. Then, for any $k \\geq 0$, we have\n\t\\begin{itemize}\n\t\t\\item[$\\rm{(i)}$] For $p = 1$, we have $t_{k} = k + 1$.\n\t\t\\item[$\\rm{(ii)}$] For $p = 2$, we have $t_{k} \\geq \\left(k + 1\\right)/2$. Moreover, one has $t_{k + 1}^{2} \\leq t_{k}^{2} + 2t_{k}$.\n\t\\end{itemize}\t\t\t\t\n\t\\end{lemma}","sha256_utf8":"222a94addff0949b903956bc0ddf482e0b14cb68b0d1d6904a6618f7e17754e7"},{"start":39261,"end":39683,"start_line":371,"end_line":373,"raw_tex":"\\begin{proof}\n\t\tThe first item is immediate as well as the first part of the second item (see, for instance, \\cite[Lemma 4.3]{BT2009}). Moreover, note that by the given recursion $t_{k + 1}^{2} - t_{k}^{2} = t_{k + 1}$, one also has $t_{k + 1} = t_{k} + t_{k + 1}/\\left(t_{k} + t_{k + 1}\\right)$, and hence it follows that $t_{k + 1}^{2} = t_{k}^{2} + t_{k + 1} \\leq t_{k}^{2} + 2t_{k}$, since $t_{k} \\geq 1$.\n\t\\end{proof}","sha256_utf8":"df4f8ac1179b03fa414a856a78499b1b0e393e2fd6a59dbda71ac1f6cb6d2ffb"},{"start":41715,"end":44242,"start_line":394,"end_line":417,"raw_tex":"\\begin{proof}\n\t\tFrom Lemma \\ref{L:FGLMain}, after we substitute $\\xi = x^{\\ast}$ and $p = 2$ (recall that $\\rho_{k} = \\rho t_{k}$ and $\\tau_{k} = t_{k}$), we obtain\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{2}s_{k + 1} - t_{k - 1}^{2}s_{k} \\leq t_{k}^{2}\\Delta_{P}\\left(x^{\\ast} , z^{k} , z^{k + 1}\\right) - \\frac{t_{k}\\sigma}{2}\\norm{x^{\\ast} - z^{k + 1}}^{2} + \\frac{1}{\\mu\\rho}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right).\n\t\t\\end{equation*}\n\t\tUsing the definition of $\\Delta_{P}$ and $\\Delta$ (see \\eqref{D:Delta}) and the fact that $P \\preceq \\left(\\sigma/2\\right)I_{n}$ we have that\n\t\t\\begin{align*}\n\t\t\tt_{k}^{2}s_{k + 1} - t_{k - 1}^{2}s_{k} & \\leq \\frac{t_{k}^{2}}{2}\\left(\\norm{x^{\\ast} - z^{k}}_{P}^{2} - \\norm{x^{\\ast} - z^{k + 1}}_{P}^{2}\\right) - t_{k}\\norm{x^{\\ast} - z^{k + 1}}_{P}^{2} + \\frac{1}{2\\mu\\rho}\\left(\\norm{\\eta - y^{k}}^{2} -  \\norm{\\eta - y^{k + 1}}^{2}\\right) \\nonumber \\\\\n\t\t\t& \\leq \\frac{1}{2}\\left(t_{k}^{2}\\norm{x^{\\ast} - z^{k}}_{P}^{2} - t_{k + 1}^{2}\\norm{x^{\\ast} - z^{k + 1}}_{P}^{2}\\right) + \\frac{1}{2\\mu\\rho}\\left(\\norm{\\eta - y^{k}}^{2} - \\norm{\\eta - y^{k + 1}}^{2}\\right),\n\t\t\\end{align*}\n\t\twhere the last inequality follows from Lemma \\ref{L:Tk}(ii). Summing this inequality for all $k = 0 , 1 , \\ldots , N - 1$ (recall that $t_{-1} = 0$ and $t_{0} = 1$), it follows (where we use  the definition of $s_{N}$, see \\eqref{laggap}, for the first inequality below)\n\t\t\\begin{equation*}\n\tt_{N - 1}^{2}\\left(\\Psi\\left(x^{N}\\right) - \\Psi\\left(x^{\\ast}\\right) + \\act{\\eta , \\AAA x^{N} - b}\\right) \\leq\tt_{N - 1}^{2}s_{N} - t_{-1}^{2}s_{0} \\leq \\frac{1}{2}\\norm{x^{\\ast} - z^{0}}_{P}^{2} + \\frac{1}{2\\mu\\rho}\\norm{\\eta - y^{0}}^{2}.\n\t\t\\end{equation*}\n\t\tTherefore, by taking the maximum of both sides over $\\norm{\\eta} \\leq c$, proves that (recall that $t_{N - 1}^{2} \\geq N^{2}/4$ using Lemma \\ref{L:Tk}(ii))\n\t\t\\begin{equation*}\n\t\t\t\\Psi\\left(x^{N}\\right) - \\Psi\\left(x^{\\ast}\\right) + c\\norm{\\AAA x^{N} - b} \\leq \\frac{B_{\\rho , c}\\left(x^{\\ast}\\right)}{2N^{2}} := \\beta,\n\t\t\\end{equation*}\n\t\tfrom which the estimate \\eqref{T:FGLrate:1} immediately follows. Moreover, since $\\left(x^{\\ast} , y^{\\ast}\\right)$ is a saddle point of problem (P), with $c \\geq 2\\norm{y^{\\ast}}$, using the above inequality it follows that\n\t\t\\begin{equation*}\n\t\t\tc\\norm{\\AAA x^{N} - b} \\leq \\Psi\\left(x^{\\ast}\\right) - \\Psi\\left(x^{N}\\right) + \\beta \\leq \\act{y^{\\ast} , \\AAA x^{N} - b} + \\beta \\leq \\frac{c}{2}\\norm{\\AAA x^{N} - b} + \\beta,\n\t\t\\end{equation*}\n\t\tand the desired estimate \\eqref{T:FGLrate:2} is proved.\n\t\\end{proof}","sha256_utf8":"c7240249be930a99ed7b7762ebeac54828b34da2834be7b2a4708f507e3d893c"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"线性约束强凸模型","source_units":[{"start":12937,"end":13017,"start_line":131,"end_line":131,"raw_tex":"\tIn this paper, we consider the linearly constrained convex optimization model,\n","sha256_utf8":"5e2a842966d59be0a498190c55db3ee0865377352885fd3fa69ac7855e044e6b"},{"start":13018,"end":13155,"start_line":132,"end_line":134,"raw_tex":"\\begin{equation*}\n\t\t\\mbox{(P)} \\quad\\quad \\min  \\left\\{ \\Psi\\left(x\\right) \\, : \\, \\AAA x = b, \\; x \\in \\real^n\\right\\},\n\t\\end{equation*}","sha256_utf8":"45bf5b1e239fd483804e2ae205a2d1e01871c3f691887865bdfbdc4105118cd2"},{"start":13156,"end":14304,"start_line":135,"end_line":138,"raw_tex":"\twhere $\\Psi:\\real^{n} \\rightarrow \\erl$ is a proper, lower semi-continuous and convex function, $\\AAA: \\real^n\\to \\real^m$ is a linear mapping, and $b \\in \\real^{m}$. Throughout this work, the feasible set of problem (P) is denoted by  $\\FFF = \\left\\{ x \\in \\rr^{n} : \\, \\AAA x = b \\right\\}$.\n\\medskip\n\t\n\t Despite its apparent simplicity, with an additional structural/data information on the objective function or/and on the linear mapping (see examples below), the optimization model (P) essentially captures a very wide range of convex optimization problems arising in many modern applications. Since we are interested in analyzing this model also in the strongly convex case, we instead assume that $\\Psi\\left(\\cdot\\right)$ is $\\sigma$-strongly convex with $\\sigma \\geq 0$ (\\ie when $\\sigma = 0$ we recover the convex setting). At this juncture, it is important to mention that in many applied problems the function $\\Psi$ is defined by the sum of two or more functions, see \\eg \\cite{CP2016-B}. In such cases,  it is enough to assume that {\\em one} of the functions is $\\sigma$-strongly convex, see Section \\ref{SSec:Block} for more details.\n","sha256_utf8":"90fd9a420174f44e7b47c97ca9c81de6d880d572a2e5f175fadf1a044b1d738f"}]},{"id":"S002","semantic_type":"definition","kind":"definition","title":"拉格朗日函数与罚项","source_units":[{"start":17236,"end":17460,"start_line":167,"end_line":167,"raw_tex":"\tWe conclude this section with some basic properties of Lagrangians in convex optimization. For the equality constrained problem given in model (P), the {\\em Lagrangian} $\\Lag: \\rr^{n} \\times \\rr^{m} \\to \\erl$ is defined by\n","sha256_utf8":"e108869db10bdcf10bc968611700fbb068ff30e5f8807c989c92725cdb20388f"},{"start":17461,"end":17582,"start_line":168,"end_line":170,"raw_tex":"\\begin{equation} \\label{D:Lag}\n\t\t\\Lag\\left(x , y\\right) \\equiv \\Psi\\left(x\\right) + \\act{y , \\AAA x - b},\n\t\\end{equation}","sha256_utf8":"5f9a7cbfe614826c9fb7fe804dd63922e69472e5d7fe1a270fbe8ca158fe7e69"},{"start":17583,"end":17808,"start_line":171,"end_line":171,"raw_tex":"\twhere $y \\in \\real^{m}$ is the {\\em Lagrange multiplier} associated with the linear constraint $\\AAA x = b$. In this study, we will also use the \\textit{augmented Lagrangian} associated to problem (P), which is defined by \t\n","sha256_utf8":"6a756fc6c6073e70f6a3c9c70b1003c38d1365da742d7d62d26ff46704d401c8"},{"start":17809,"end":18037,"start_line":172,"end_line":174,"raw_tex":"\\begin{equation} \\label{D:AugLA}\n\t\t\\Lag_{\\rho}\\left(x ,  y\\right) := \\Lag\\left(x , y\\right) + \\frac{\\rho}{2}\\norm{\\AAA x - b}^{2} = \\Psi\\left(x\\right) + \\act{y , \\AAA x - b} + \\frac{\\rho}{2}\\norm{\\AAA x - b}^{2},\n\t\\end{equation}","sha256_utf8":"03f9ee54c061248f25b124c49afaea46ecfe48e42eaeeeee2996671c6e5e3b11"},{"start":18038,"end":18173,"start_line":175,"end_line":175,"raw_tex":"\twhere $\\rho > 0$ is a {\\em penalty parameter}. Clearly, when $\\rho = 0$ we recover the Lagrangian itself, \\ie $\\Lag \\equiv \\Lag_{0}$.\n","sha256_utf8":"e2690b65128788da1a5fb1c6777d00715ecfea6e5d7c05262d919173ce606660"}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","title":"鞍点存在假设","source_units":[{"start":18249,"end":18659,"start_line":179,"end_line":184,"raw_tex":"\\begin{assumption} \\label{A:ConvLag}\n\t\tThe Lagrangian $\\Lag$ has a saddle point, that is, there exists a pair $\\left(x^{\\ast} , y^{\\ast}\\right)$ such that\n\t\t\\begin{equation} \\label{A:ConvLag:1}\n\t\t\t\\Lag\\left(x^{\\ast} , y\\right) \\leq \\Lag\\left(x^{\\ast} , y^{\\ast}\\right) \\leq \\Lag\\left(x , y^{\\ast}\\right), \\quad \\forall \\,\\, x \\in \\real^{n}, \\quad \\forall \\,\\, y\\in \\real^{m}.\n\t\t\\end{equation}\n\t\\end{assumption}","sha256_utf8":"030f92eac44b146f5b3ac99a347421ef4035e0473c4705ad19453f58c8ba2beb"}]},{"id":"S004","semantic_type":"notation","kind":"notation","title":"矩阵半范数与差分","source_units":[{"start":20003,"end":20435,"start_line":197,"end_line":197,"raw_tex":"\tWe end this section with some notations that will be used throughout the paper. We denote by $\\bbS_{+}^{n}$ and $\\bbS_{++}^{n}$ the sets of all $n \\times n$ {\\em positive semi-definite matrices} and {\\em positive definite matrices}, respectively. Given $P \\in \\bbS_{+}^{n}$, $\\norm{u}_{P} := \\sqrt{\\act{u , Pu}}$ stands for the semi-norm of $u \\in \\real^{n}$. For any $u , v , w \\in \\real^{n}$, we define the following quantity \t\t\n","sha256_utf8":"b00d99d78f5a31c97dbbd66f639624cdbfdea09bc35a90c24901a792b43a143f"},{"start":20436,"end":20591,"start_line":198,"end_line":200,"raw_tex":"\\begin{equation} \\label{D:Delta}\n\t\t\\Delta_{P}\\left(u , v , w\\right) := \\frac{1}{2}\\left(\\norm{u - v}_{P}^{2} - \\norm{u - w}_{P}^{2}\\right).\n\t\\end{equation}","sha256_utf8":"99602ce2dbfa4b55f4a7982bd4847a1bf43e4089fa59028300a53fba9de512cf"},{"start":20592,"end":20722,"start_line":201,"end_line":201,"raw_tex":"\tIn the case that $P = I$, where $I_{n}$ denotes the $n \\times n$ {\\em identity matrix}, we simply denote $\\Delta := \\Delta_{P}$.\n","sha256_utf8":"0a52df2ed32855d7d2ee6ae0e4310ed6dbcbf9cc8961feccc21d1229522a2257"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"良好原始算法映射","source_units":[{"start":22814,"end":23839,"start_line":213,"end_line":218,"raw_tex":"\\begin{definition}[Nice primal algorithmic map] \\label{D:AlgoMapNice}\n\t\tGiven the parameters $\\rho , t > 0$, we let $\\rho_{t} = \\rho$ and $\\tau_{t} = 1$ (when $\\sigma = 0$) or $\\rho_{t} = \\rho t$ and $\\tau_{t} = t$ (when $\\sigma > 0$). A primal algorithmic map $\\SSS_{t} : \\real^{n} \\times \\real^{m} \\rightarrow \\real^{n}$, which is applied on the augmented Lagrangian $\\Lag_{\\rho_{t}}\\left(z , \\lambda\\right)$, that generates $z^{+}$ via $z^{+} \\in \\SSS_{t}\\left(z , \\lambda\\right)$, is called {\\em \\nice}\\hspace{-0.05in} if there exist a parameter $\\delta \\in \\left(0 , 1\\right]$ and matrices $P , Q \\in \\bbS_{+}^{n}$, such that for any $\\xi \\in \\FFF$ we have\n\t\t\\begin{equation} \\label{D:AlgoMapNice:1}\n\t\t\t\\Lag_{\\rho_{t}}\\left(z^{+} , \\lambda\\right) - \\Lag_{\\rho_{t}}\\left(\\xi , \\lambda\\right) \\leq \\tau_{t}\\Delta_{P}\\left(\\xi , z , z^{+}\\right) - \\frac{\\tau_{t}}{2}\\norm{z^{+} - z}_{Q}^{2} - \\frac{\\sigma}{2}\\norm{\\xi - z^{+}}^{2} - \\frac{\\delta\\rho_{t}}{2}\\norm{\\AAA z^{+} - b}^{2}.\n\t\t\\end{equation}\t\t\t\t\n\t\\end{definition}","sha256_utf8":"e81c0823bbbff82ee03e68595b3a07ff450c1d5c175e67ef171b32483108a7a1"}]},{"id":"S006","semantic_type":"algorithm","kind":"algorithm","title":"FLAG完整迭代规则","source_units":[{"start":24250,"end":26418,"start_line":222,"end_line":248,"raw_tex":"    \\begin{center}\n    \t\t\\fbox{\\parbox{16.5cm}{{\\bf FLAG -- Faster LAGrangian based method}\n\t\t\t\\begin{enumerate}\n\t\t\t\t\\item[1.] {\\bf Input:} The problem data $[\\Psi, \\AAA, b, \\sigma]$, and a \\nice primal algorithmic map $\\SSS_{t}\\left(\\cdot\\right)$.\n\t\t\t\t\\item[2.] {\\bf Initialization:} Set $\\mu \\in \\left(0 , \\delta\\right]$ and $\\rho > 0$. Start with any $\\left(x^{0} , z^{0} , y^{0}\\right) \\in \\rr^{n} \\times \\rr^{n} \\times \\rr^{m}$, $t_{0} = 1$.\n            \t\t\\item[3.] {\\bf Iterations:} Sequences $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ and $\\seq{t}{k}$ are generated as follows: for each $k \\geq 0$\n                \t\t\\begin{enumerate}\n                    \t\t\\item[3.1.] If $\\sigma = 0$, let $\\rho_{k} = \\rho$, or, if $\\sigma > 0$, let $\\rho_{k} = \\rho t_{k}$. Compute\n                    \t\t\t\\begin{equation} \\label{GL:Lambda}\n                    \t\t\t\t\\lambda^{k} = y^{k} + \\rho_{k}\\left(t_{k} - 1\\right)\\left(\\AAA x^{k} -b\\right).\n                    \t\t\t\\end{equation}    \t\t\n                    \t\t\\item[3.2.] Update the sequence $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ by\n                       \t\t\\begin{align}\n                           \t\tz^{k + 1} & \\in \\SSS_{t_{k}}\\left(z^{k} , \\lambda^{k}\\right), \\label{FGL:PriStep} \\\\\n                           \t\ty^{k + 1} & = y^{k} + \\mu\\rho_{k}\\left(\\AAA z^{k + 1} - b\\right), \\label{FGL:MultiStep} \\\\\n                           \t\tx^{k + 1} & = \\left(1 - t_{k}^{-1}\\right)x^{k} + t_{k}^{-1}z^{k + 1}. \\label{FGL:AccStep}\n                       \t\t\\end{align}\n \t\t\t\t\t\t\\item[3.3.] Update the sequence $\\seq{t}{k}$ by solving the equation $t_{k + 1}^{p} - t_{k}^{p} = t_{k + 1}^{p - 1}$, \\ie\n                       \t\t\\begin{equation} \\label{FGL:Tk}\n                        \t\t\tt_{k + 1} = \\left\\{\\begin{matrix}\n                        \t\t\tt_{k}  + 1, & \\hspace{-0.6in} p = 1 \\,\\, \\text{(convex case)}, \\vspace{0.1in} \\\\\n                        \t\t\t(1 + \\sqrt{1 + 4t_{k}^{2}})/2, & p =2 \\,\\, \\text{(strongly convex case)}.\n                        \t\t\t\\end{matrix} \\right.\n\t\t\t\t\t\t\t\\end{equation}\n                    \t\t\\end{enumerate}\n\t\t\t\\end{enumerate}\\vspace{-0.2in}}}\n\t\\end{center}\n","sha256_utf8":"3f8de3e95c28a9f84d4a16114603906e5e9769b116b4bc0b6a462cbb6923351c"}]},{"id":"S007","semantic_type":"derived_fact","kind":"derived_fact","title":"迭代映射基本不等式","source_units":[{"start":27735,"end":28143,"start_line":258,"end_line":258,"raw_tex":"\tIn FLAG, once the \\nice algorithmic map $\\SSS_{t}\\left(\\cdot\\right)$ has been chosen, \\ie a parameter $\\delta \\in \\left(0 , 1\\right]$ and matrices $P , Q \\in \\bbS_{+}^{n}$, have been determined, Definition \\ref{D:AlgoMapNice} translates in terms of the iterative sequence $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ produced by FLAG as follows: for each $k \\geq 0$ and $\\xi \\in \\FFF$,\n","sha256_utf8":"b5980546cadd271552eae5f6b155e60790e8b9626acb159865ffb7c32984dfd1"},{"start":28144,"end":28534,"start_line":259,"end_line":262,"raw_tex":"\\begin{align}\n\t\t\\Lag_{\\rho_{k}}\\left(z^{k + 1} , \\lambda^{k}\\right) - \\Lag_{\\rho_{k}}\\left(\\xi , \\lambda^{k}\\right) & \\leq \\tau_{k}\\Delta_{P}\\left(\\xi , z^{k} , z^{k + 1}\\right) - \\frac{\\tau_{k}}{2}\\norm{z^{k + 1} - z^{k}}_{Q}^{2}  - \\frac{\\sigma}{2}\\norm{\\xi- z^{k + 1}}^{2} \\nonumber \\\\\n\t\t& - \\frac{\\delta\\rho_{k}}{2}\\norm{\\AAA z^{k + 1} - b}^{2}, \\label{AlgoMapNiceProperty}\n\t\\end{align}","sha256_utf8":"096f7d9baf38022613aafe7e0bb20835a7d611d57e00ce6f3fadfa828d86f906"},{"start":28537,"end":28672,"start_line":263,"end_line":263,"raw_tex":"\twhere $\\tau_{k} = 1$ and $\\rho_{k} = \\rho$ (when $\\sigma = 0$) or $\\tau_{k} = t_{k}$ and $\\rho_{k} = \\rho t_{k}$ (when $\\sigma > 0$).\n","sha256_utf8":"6772e6485d972856f37bc57e021b469b9e52999a966fb7f878e232f3bc64d156"}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","title":"三点恒等式","source_units":[{"start":29882,"end":30055,"start_line":272,"end_line":272,"raw_tex":"\tBefore proceeding, we recall the following fundamental {\\em Pythagoras three-points identity} that is frequently used below. Given any matrix $P \\in \\bbS_{+}^{n}$, we have\n","sha256_utf8":"0dffd7a28355788f7c685152d93ce0dd5c8badcd678b565ccc62b8da307fc18c"},{"start":30056,"end":30252,"start_line":273,"end_line":275,"raw_tex":"\\begin{equation} \\label{Pyth}\n\t\t2\\act{u - w , P\\left(w - v\\right)} =  \\norm{u - v}_{P}^{2}- \\norm{u - w}_{P}^{2} - \\norm{v - w}_{P}^{2}, \\quad \\forall \\,\\, u , v , w \\in \\real^{n}.\n\t\\end{equation}","sha256_utf8":"dae84b2e696ff10649f2e5cd6c5ddae3953ba97f12346087babaae7560dcead1"}]},{"id":"S009","semantic_type":"notation","kind":"notation","title":"参数递推与增广间隙","source_units":[{"start":30254,"end":30839,"start_line":276,"end_line":280,"raw_tex":"\tWe also recall the main parameter sequences introduced in FLAG and which, for convenience, will be used often and systematically as follows:\n   \\begin{equation*}\n    \t\t\\rho_{k} = \\rho t_{k}^{p - 1} ,\\; (\\rho>0)\\; \\quad \\text{and} \\quad t_{k}^{p} - t_{k - 1}^{p} = t_{k}^{p - 1},\\; (t_{-1} \\equiv 0 \\;\\text{and}\\; t_0 = 1),\n\t\\end{equation*}\t\n\twhere $p = 1$ (if $\\sigma = 0$) or $p = 2$ (if $\\sigma > 0$), and $\\tau_{k} = 1$ if $\\sigma = 0$, or $\\tau_{k} = t_{k}$ if $\\sigma > 0$. We will also often use the following expression. For any $k\\geq 0$ and any $\\xi \\in {\\cal F}$ we define,\n","sha256_utf8":"9f4cd3dd1731d509ac4f6e5acd3a9025a7dda196638b0bd84d2e23080faa5543"},{"start":30840,"end":31071,"start_line":281,"end_line":283,"raw_tex":"\\begin{equation}\\label{laggap}\n\t\ts_{k} := \\Lag_{\\rho t_{k - 1}^{p}}\\left(x^{k} , \\eta\\right) - \\Lag_{\\rho t_{k - 1}^{p}}\\left(\\xi , \\eta\\right) \\equiv \\Lag_{\\rho t_{k - 1}^{p}}\\left(x^{k} , \\eta\\right) - \\Psi(\\xi).\n\t \\end{equation}","sha256_utf8":"cc92e0d403e33726be695c0828a3475dc54bfc75c2769562e1d71f564e01583b"}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","title":"对偶范数控制常数","source_units":[{"start":40533,"end":40801,"start_line":382,"end_line":382,"raw_tex":"\tThroughout the rest of this section, recalling the fact that the set of optimal dual solutions is compact (see Section \\ref{Sec:Form}), we take $c > 0$ to be a constant for which $c \\geq 2\\norm{y^{\\ast}}$, where $y^{\\ast}$ is an optimal solution of the dual problem.\n","sha256_utf8":"c10d888eaebc0b99192dea74544af5b32d55fe65b0d0f68e2729193d553a97b1"}]},{"id":"S011","semantic_type":"theorem","kind":"theorem","title":"强凸非遍历二阶收敛率","source_units":[{"start":40924,"end":41713,"start_line":386,"end_line":393,"raw_tex":"\\begin{theorem}[A fast non-ergodic function values and feasibility violation rates] \\label{T:FGLrate}\n\t\tLet $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ be a sequence generated by FLAG. Suppose that $\\sigma > 0$ and $0 \\preceq P \\preceq \\left(\\sigma/2\\right)I_{n}$. Then, for any optimal solution $x^{\\ast}$ of problem (P), we have\n   \t\t\\begin{align}\n\t\t\t\\Psi\\left(x^{N}\\right) - \\Psi\\left(x^{\\ast}\\right) & \\leq \\frac{B_{\\rho , c}\\left(x^{\\ast}\\right)}{2N^{2}}, \\label{T:FGLrate:1} \\\\\n       \t\t\\norm{\\AAA x^{N} - b} & \\leq \\frac{B_{\\rho , c}\\left(x^{\\ast}\\right)}{cN^{2}}, \\label{T:FGLrate:2}\n\t\t\\end{align}\n\t\twhere $B_{\\rho , c}\\left(x^{\\ast}\\right) := 4\\left(\\norm{x^{\\ast} - z^{0}}_{P}^{2} + \\frac{1}{\\mu\\rho}\\left(\\norm{y^{0}} + c\\right)^{2}\\right)$.\n\t\\end{theorem}","sha256_utf8":"ed41a025f1c33bde99b8940d7f76f3fad50663432f4b1b9308c85846d48d029d"}]},{"id":"S012","semantic_type":"proof","kind":"proof","title":"非遍历收敛率完整证明","source_units":[{"start":41715,"end":44242,"start_line":394,"end_line":417,"raw_tex":"\\begin{proof}\n\t\tFrom Lemma \\ref{L:FGLMain}, after we substitute $\\xi = x^{\\ast}$ and $p = 2$ (recall that $\\rho_{k} = \\rho t_{k}$ and $\\tau_{k} = t_{k}$), we obtain\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{2}s_{k + 1} - t_{k - 1}^{2}s_{k} \\leq t_{k}^{2}\\Delta_{P}\\left(x^{\\ast} , z^{k} , z^{k + 1}\\right) - \\frac{t_{k}\\sigma}{2}\\norm{x^{\\ast} - z^{k + 1}}^{2} + \\frac{1}{\\mu\\rho}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right).\n\t\t\\end{equation*}\n\t\tUsing the definition of $\\Delta_{P}$ and $\\Delta$ (see \\eqref{D:Delta}) and the fact that $P \\preceq \\left(\\sigma/2\\right)I_{n}$ we have that\n\t\t\\begin{align*}\n\t\t\tt_{k}^{2}s_{k + 1} - t_{k - 1}^{2}s_{k} & \\leq \\frac{t_{k}^{2}}{2}\\left(\\norm{x^{\\ast} - z^{k}}_{P}^{2} - \\norm{x^{\\ast} - z^{k + 1}}_{P}^{2}\\right) - t_{k}\\norm{x^{\\ast} - z^{k + 1}}_{P}^{2} + \\frac{1}{2\\mu\\rho}\\left(\\norm{\\eta - y^{k}}^{2} -  \\norm{\\eta - y^{k + 1}}^{2}\\right) \\nonumber \\\\\n\t\t\t& \\leq \\frac{1}{2}\\left(t_{k}^{2}\\norm{x^{\\ast} - z^{k}}_{P}^{2} - t_{k + 1}^{2}\\norm{x^{\\ast} - z^{k + 1}}_{P}^{2}\\right) + \\frac{1}{2\\mu\\rho}\\left(\\norm{\\eta - y^{k}}^{2} - \\norm{\\eta - y^{k + 1}}^{2}\\right),\n\t\t\\end{align*}\n\t\twhere the last inequality follows from Lemma \\ref{L:Tk}(ii). Summing this inequality for all $k = 0 , 1 , \\ldots , N - 1$ (recall that $t_{-1} = 0$ and $t_{0} = 1$), it follows (where we use  the definition of $s_{N}$, see \\eqref{laggap}, for the first inequality below)\n\t\t\\begin{equation*}\n\tt_{N - 1}^{2}\\left(\\Psi\\left(x^{N}\\right) - \\Psi\\left(x^{\\ast}\\right) + \\act{\\eta , \\AAA x^{N} - b}\\right) \\leq\tt_{N - 1}^{2}s_{N} - t_{-1}^{2}s_{0} \\leq \\frac{1}{2}\\norm{x^{\\ast} - z^{0}}_{P}^{2} + \\frac{1}{2\\mu\\rho}\\norm{\\eta - y^{0}}^{2}.\n\t\t\\end{equation*}\n\t\tTherefore, by taking the maximum of both sides over $\\norm{\\eta} \\leq c$, proves that (recall that $t_{N - 1}^{2} \\geq N^{2}/4$ using Lemma \\ref{L:Tk}(ii))\n\t\t\\begin{equation*}\n\t\t\t\\Psi\\left(x^{N}\\right) - \\Psi\\left(x^{\\ast}\\right) + c\\norm{\\AAA x^{N} - b} \\leq \\frac{B_{\\rho , c}\\left(x^{\\ast}\\right)}{2N^{2}} := \\beta,\n\t\t\\end{equation*}\n\t\tfrom which the estimate \\eqref{T:FGLrate:1} immediately follows. Moreover, since $\\left(x^{\\ast} , y^{\\ast}\\right)$ is a saddle point of problem (P), with $c \\geq 2\\norm{y^{\\ast}}$, using the above inequality it follows that\n\t\t\\begin{equation*}\n\t\t\tc\\norm{\\AAA x^{N} - b} \\leq \\Psi\\left(x^{\\ast}\\right) - \\Psi\\left(x^{N}\\right) + \\beta \\leq \\act{y^{\\ast} , \\AAA x^{N} - b} + \\beta \\leq \\frac{c}{2}\\norm{\\AAA x^{N} - b} + \\beta,\n\t\t\\end{equation*}\n\t\tand the desired estimate \\eqref{T:FGLrate:2} is proved.\n\t\\end{proof}","sha256_utf8":"c7240249be930a99ed7b7762ebeac54828b34da2834be7b2a4708f507e3d893c"}]},{"id":"S013","semantic_type":"assumption","kind":"assumption","title":"强凸性与度量上界","source_units":[{"start":41026,"end":41277,"start_line":387,"end_line":387,"raw_tex":"\t\tLet $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ be a sequence generated by FLAG. Suppose that $\\sigma > 0$ and $0 \\preceq P \\preceq \\left(\\sigma/2\\right)I_{n}$. Then, for any optimal solution $x^{\\ast}$ of problem (P), we have\n","sha256_utf8":"07ffbdb2df282e1049fdca4835e6c20d95272b7f5fe6695da77cffc0735c073b"}]},{"id":"S014","semantic_type":"lemma","kind":"lemma","title":"原始对偶单步估计","source_units":[{"start":31144,"end":31764,"start_line":287,"end_line":293,"raw_tex":"\\begin{lemma} \\label{L:FGLIter}\n\t\tLet $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ be a sequence generated by FLAG. Then, for any $\\xi \\in \\FFF$, $\\eta \\in \\real^{m}$ and $k \\geq 0$, we have\n\t\t\\begin{align}\n\t\t\t\\Lag_{\\rho_{k}}\\left(z^{k + 1} , \\eta\\right) - \\Lag_{\\rho_{k}}\\left(\\xi , \\eta\\right) & \\leq \\tau_{k}\\Delta_{P}\\left(\\xi , z^{k} , z^{k + 1}\\right) - \\frac{\\sigma}{2}\\norm{\\xi - z^{k + 1}}^{2} + \\frac{1}{\\mu\\rho_{k}}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right) \\nonumber \\\\\n\t\t\t& - \\rho t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b}. \\label{L:FGLIter:0}\n\t\t\\end{align}\t\t\n\t\\end{lemma}","sha256_utf8":"1004d2ac2e8e8af827effbe1b2529c80728f952ca3f3e239dabc5695daf344e8"}]},{"id":"S015","semantic_type":"proof","kind":"proof","title":"单步估计完整证明","source_units":[{"start":31766,"end":33841,"start_line":294,"end_line":314,"raw_tex":"\\begin{proof}\n\t\tSince $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ is generated by FLAG, we obtain from \\eqref{AlgoMapNiceProperty} after omitting the non-negative term $\\left(\\tau_{k}/2\\right)\\norm{z^{k + 1} - z^{k}}_{Q}^{2}$, that\n\t\t\\begin{equation} \\label{L:FGLIter:1}\n\t\t\t\\Lag_{\\rho_{k}}\\left(z^{k + 1} , \\lambda^{k}\\right) - \\Lag_{\\rho_{k}}\\left(\\xi , \\lambda^{k}\\right) \\leq \\tau_{k}\\Delta_{P}\\left(\\xi , z^{k} , z^{k + 1}\\right) - \\frac{\\sigma}{2}\\norm{\\xi - z^{k + 1}}^{2} - \\frac{\\delta\\rho_{k}}{2}\\norm{\\AAA z^{k + 1} - b}^{2}.\n\t\t\\end{equation}\n\t\tFrom the multiplier update \\eqref{FGL:MultiStep} and the three-points identity \\eqref{Pyth}, we obtain, for all $\\eta \\in \\real^{m}$,\n\t\t\\begin{align}\n\t\t\t\\act{\\eta - y^{k} , \\AAA z^{k + 1} - b}  = \\frac{1}{\\mu\\rho_{k}}\\act{\\eta - y^{k} , y^{k + 1} - y^{k}} & = \\frac{1}{\\mu\\rho_{k}}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right) + \\frac{1}{2\\mu\\rho_{k}}\\norm{y^{k + 1} - y^{k}}^{2} \\nonumber \\\\\n\t\t\t& = \\frac{1}{\\mu\\rho_{k}}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right) + \\frac{\\mu\\rho_{k}}{2}\\norm{\\AAA z^{k + 1} - b}^{2}. \\label{L:FGLIter:2}\n\t\t\\end{align}\n\t\tUsing the update rule of the sequence $\\seq{t}{k}$ (\\cf \\eqref{FGL:Tk}) we have that $\\rho_{k}\\left(t_{k} - 1\\right) =\\rho t_{k - 1}^{p} $ and thus\n\t\t\\begin{equation*}\n\t\t\t\\lambda^{k} = y^{k} + \\rho_{k}\\left(t_{k} - 1\\right)\\left(\\AAA x^{k} -b\\right) = y^{k} + \\rho t_{k - 1}^{p}\\left(\\AAA x^{k} -b\\right).\n\t\t\\end{equation*}\n\t\tUsing this together with \\eqref{L:FGLIter:2} yields, for all $\\eta \\in \\real^{m}$, that\n\t\t\\begin{align}\t\t\n\t\t\t\\act{\\eta - \\lambda^{k} , \\AAA z^{k + 1} - b} & = \\act{\\eta - y^{k} , \\AAA z^{k + 1} - b} - \\rho t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b} \\nonumber \\\\\n\t\t\t& = \\frac{1}{\\mu\\rho_{k}}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right) + \\frac{\\mu\\rho_{k}}{2}\\norm{\\AAA z^{k + 1} - b}^{2} - \\rho t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b}. \\label{L:FGLIter:3}\n\t\t\\end{align}\n\t\tAdding \\eqref{L:FGLIter:1} to \\eqref{L:FGLIter:3} yields (recall that $\\mu \\leq \\delta$) the desired result.\n\t\\end{proof}","sha256_utf8":"ea8edced767fef02131980ad9d0eff69c85688169ca98365858fe24fa408d5fc"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"舍去映射非负项","source_units":[{"start":31780,"end":32341,"start_line":295,"end_line":298,"raw_tex":"\t\tSince $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ is generated by FLAG, we obtain from \\eqref{AlgoMapNiceProperty} after omitting the non-negative term $\\left(\\tau_{k}/2\\right)\\norm{z^{k + 1} - z^{k}}_{Q}^{2}$, that\n\t\t\\begin{equation} \\label{L:FGLIter:1}\n\t\t\t\\Lag_{\\rho_{k}}\\left(z^{k + 1} , \\lambda^{k}\\right) - \\Lag_{\\rho_{k}}\\left(\\xi , \\lambda^{k}\\right) \\leq \\tau_{k}\\Delta_{P}\\left(\\xi , z^{k} , z^{k + 1}\\right) - \\frac{\\sigma}{2}\\norm{\\xi - z^{k + 1}}^{2} - \\frac{\\delta\\rho_{k}}{2}\\norm{\\AAA z^{k + 1} - b}^{2}.\n\t\t\\end{equation}\n","sha256_utf8":"0a60de41260cc7725bd706bfd30a7bb7162196d3e6a20d8c53491f5ad09f9568"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"乘子更新转化为差分","source_units":[{"start":32341,"end":32894,"start_line":299,"end_line":303,"raw_tex":"\t\tFrom the multiplier update \\eqref{FGL:MultiStep} and the three-points identity \\eqref{Pyth}, we obtain, for all $\\eta \\in \\real^{m}$,\n\t\t\\begin{align}\n\t\t\t\\act{\\eta - y^{k} , \\AAA z^{k + 1} - b}  = \\frac{1}{\\mu\\rho_{k}}\\act{\\eta - y^{k} , y^{k + 1} - y^{k}} & = \\frac{1}{\\mu\\rho_{k}}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right) + \\frac{1}{2\\mu\\rho_{k}}\\norm{y^{k + 1} - y^{k}}^{2} \\nonumber \\\\\n\t\t\t& = \\frac{1}{\\mu\\rho_{k}}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right) + \\frac{\\mu\\rho_{k}}{2}\\norm{\\AAA z^{k + 1} - b}^{2}. \\label{L:FGLIter:2}\n\t\t\\end{align}\n","sha256_utf8":"fd0a639eced6c8689068031dfe0cca93fdbf99afbd569cce8cdc75f8d1151662"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"改写辅助乘子","source_units":[{"start":32894,"end":33220,"start_line":304,"end_line":307,"raw_tex":"\t\tUsing the update rule of the sequence $\\seq{t}{k}$ (\\cf \\eqref{FGL:Tk}) we have that $\\rho_{k}\\left(t_{k} - 1\\right) =\\rho t_{k - 1}^{p} $ and thus\n\t\t\\begin{equation*}\n\t\t\t\\lambda^{k} = y^{k} + \\rho_{k}\\left(t_{k} - 1\\right)\\left(\\AAA x^{k} -b\\right) = y^{k} + \\rho t_{k - 1}^{p}\\left(\\AAA x^{k} -b\\right).\n\t\t\\end{equation*}\n","sha256_utf8":"73789f51607057f172427849f0a000d06ea622bbeebd0b95a3fb61eb2eb8820e"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"计算乘子偏移内积","source_units":[{"start":33220,"end":33718,"start_line":308,"end_line":312,"raw_tex":"\t\tUsing this together with \\eqref{L:FGLIter:2} yields, for all $\\eta \\in \\real^{m}$, that\n\t\t\\begin{align}\t\t\n\t\t\t\\act{\\eta - \\lambda^{k} , \\AAA z^{k + 1} - b} & = \\act{\\eta - y^{k} , \\AAA z^{k + 1} - b} - \\rho t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b} \\nonumber \\\\\n\t\t\t& = \\frac{1}{\\mu\\rho_{k}}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right) + \\frac{\\mu\\rho_{k}}{2}\\norm{\\AAA z^{k + 1} - b}^{2} - \\rho t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b}. \\label{L:FGLIter:3}\n\t\t\\end{align}\n","sha256_utf8":"abb1e3e2edb5d0cb7b472b323dd1547d15ec3e139e01c75b35446a9e18b43657"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"合并并消去残差项","source_units":[{"start":33718,"end":33829,"start_line":313,"end_line":313,"raw_tex":"\t\tAdding \\eqref{L:FGLIter:1} to \\eqref{L:FGLIter:3} yields (recall that $\\mu \\leq \\delta$) the desired result.\n","sha256_utf8":"1a86722250f043913c3afc4cb1ec518fa236eec38300fe652b037bc881f827f8"}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","title":"加权增广间隙递推","source_units":[{"start":34255,"end":34902,"start_line":316,"end_line":322,"raw_tex":"\\begin{lemma} \\label{L:FGLMain}\n\t\tLet $\\left\\{ \\left(x^{k} , z^{k} , y^{k}\\right) \\right\\}_{k \\in \\nn}$ be a sequence generated by FLAG. Then, for any $\\xi \\in \\FFF$, $\\eta \\in \\real^{m}$ and $k \\geq 0$, we have\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{p}s_{k + 1} - t_{k - 1}^{p}s_{k} \\leq \\frac{\\tau_{k}\\rho_{k}}{\\rho}\\Delta_{P}\\left(\\xi , z^{k} , z^{k + 1}\\right) - \\frac{\\rho_{k}\\sigma}{2\\rho}\\norm{\\xi - z^{k + 1}}^{2} + \\frac{1}{\\mu\\rho}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right),\n\t\t\\end{equation*}\n\t\twhere $s_{k} = \\Lag_{\\rho t_{k - 1}^{p}}\\left(x^{k} , \\eta\\right) - \\Lag_{\\rho t_{k - 1}^{p}}\\left(\\xi , \\eta\\right)$ (\\cf \\ref{laggap}).\n\t\\end{lemma}","sha256_utf8":"02b00e6d2a09321d2ae9da247c7364201f5b0792e683d98e1844641a7d1550e6"}]},{"id":"S022","semantic_type":"proof","kind":"proof","title":"间隙递推完整证明","source_units":[{"start":34905,"end":38605,"start_line":323,"end_line":362,"raw_tex":"\\begin{proof}\n\t\tSince $x^{k + 1} = \\left(1 - t_{k}^{-1}\\right)x^{k} + t_{k}^{-1}z^{k + 1}$ (\\cf \\eqref{FGL:AccStep}) we obtain from the convexity of $\\Psi\\left(\\cdot\\right)$ that\n\t\t\\begin{equation*}\n\t\t\t\\Psi\\left(x^{k + 1}\\right) \\leq \\left(1 - t_{k}^{-1}\\right)\\Psi\\left(x^{k}\\right) + t_{k}^{-1}\\Psi\\left(z^{k + 1}\\right).\n\t\t\\end{equation*}\n\t\tTherefore, multiplying both sides by $t_{k}^{p}$ (recalling that $t_{k}^{p} - t_{k}^{p - 1} = t_{k - 1}^{p}$), we obtain\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{p}\\left(\\Psi\\left(x^{k + 1}\\right) - \\Psi\\left(\\xi\\right)\\right) - t_{k - 1}^{p}\\left(\\Psi\\left(x^{k}\\right) - \\Psi\\left(\\xi\\right)\\right) \\leq t_{k}^{p - 1}\\left(\\Psi\\left(z^{k + 1}\\right) - \\Psi\\left(\\xi\\right)\\right).\n\t\t\\end{equation*}\n\t\tIn addition, using again \\eqref{FGL:AccStep}, yields that\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{p}\\act{\\eta , \\AAA x^{k + 1} - b} - t_{k - 1}^{p}\\act{\\eta , \\AAA x^{k} - b} = t_{k}^{p - 1}\\act{\\eta , \\AAA z^{k + 1} - b}.\n\t\t\\end{equation*}\t\t\t\n\t\tCombining these two facts yields (using the definition of the Lagrangian $\\Lag$)\n\t\t\\begin{equation} \\label{L:FGLMain:1}\n\t\t\tt_{k}^{p}\\left(\\Lag\\left(x^{k + 1} , \\eta\\right) - \\Lag\\left(\\xi , \\eta\\right)\\right) - t_{k - 1}^{p}\\left(\\Lag\\left(x^{k} , \\eta\\right) - \\Lag\\left(\\xi , \\eta\\right)\\right) \\leq t_{k}^{p - 1}\\left(\\Lag\\left(z^{k + 1} , \\eta\\right) - \\Lag\\left(\\xi , \\eta\\right)\\right).\n\t\t\\end{equation}\t\t\t\n\t\tNow, we will again use the relation $x^{k + 1} = \\left(1 - t_{k}^{-1}\\right)x^{k} + t_{k}^{-1}z^{k + 1}$ to obtain\n\t\t\\begin{equation*}\n\t\t\t\\norm{\\AAA x^{k + 1} - b}^{2}  = \\left(1 - t_{k}^{-1}\\right)^{2}\\norm{\\AAA x^{k} - b}^{2} + t_{k}^{-2}\\norm{\\AAA z^{k + 1} - b}^{2} + 2\\left(1 - t_{k}^{-1}\\right)t_{k}^{-1}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b}.\n\t\t\\end{equation*}\t\t\n\t\tMultiplying both sides of the above equality by $\\rho t_{k}^{2p}/2$ (recalling $\\rho_{k} = \\rho t_{k}^{p - 1}$ and $t_{k}^{p} - t_{k}^{p - 1} = t_{k - 1}^{p}$) yields\n\t\t\\begin{align}\n\t\t\t\\frac{\\rho t_{k}^{2p}}{2}\\norm{\\AAA x^{k + 1} - b}^{2} - \\frac{\\rho t_{k - 1}^{2p}}{2}\\norm{\\AAA x^{k} - b}^{2} = \\frac{\\rho_{k}t_{k}^{p - 1}}{2}\\norm{\\AAA z^{k + 1} - b}^{2} + \\rho_{k}t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b}. \\label{L:FGLMain:2}\n\t\t\\end{align}\n\t\tTherefore, adding \\eqref{L:FGLMain:1} to \\eqref{L:FGLMain:2} yields (recall that $\\norm{\\AAA\\xi - b}^{2} = 0$) with $s_{k} = \\Lag_{\\rho t_{k - 1}^{p}}\\left(x^{k} , \\eta\\right) - \\Lag_{\\rho t_{k - 1}^{p}}\\left(\\xi , \\eta\\right)$,\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{p}s_{k + 1} - t_{k - 1}^{p}s_{k} \\leq t_{k}^{p - 1}\\left(\\Lag_{\\rho_{k}}\\left(z^{k + 1} , \\eta\\right) - \\Lag_{\\rho_{k}}\\left(\\xi , \\eta\\right)\\right) + \\rho_{k}t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b}.\n\t\t\\end{equation*}\t\t\t\n\t\tFrom Lemma \\ref{L:FGLIter}, after we multiplied both sides of \\eqref{L:FGLIter:0} by $t_{k}^{p - 1}$ (recall that $\\rho_{k} = \\rho t_{k}^{p - 1}$), we obtain that\n\t\t\\begin{align*}\n\t\t\tt_{k}^{p - 1}\\left(\\Lag_{\\rho_{k}}\\left(z^{k + 1} , \\eta\\right) - \\Lag_{\\rho_{k}}\\left(\\xi , \\eta\\right)\\right) + \\rho_{k}t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b} & \\leq \\frac{\\tau_{k}\\rho_{k}}{\\rho}\\Delta_{P}\\left(\\xi , z^{k} , z^{k + 1}\\right) - \\frac{\\rho_{k}\\sigma}{2\\rho}\\norm{\\xi - z^{k + 1}}^{2} \\\\\n\t\t\t& + \\frac{1}{\\mu\\rho}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right).\n\t\t\\end{align*}\t\t\n\t\tBy combining the last two inequalities, we thus get\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{p}s_{k + 1} - t_{k - 1}^{p}s_{k} \\leq \\frac{\\tau_{k}\\rho_{k}}{\\rho}\\Delta_{P}\\left(\\xi , z^{k} , z^{k + 1}\\right) - \\frac{\\rho_{k}\\sigma}{2\\rho}\\norm{\\xi - z^{k + 1}}^{2} + \\frac{1}{\\mu\\rho}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right),\n\t\t\\end{equation*}\t\t\t\t\n\t\twhich proves the desired result.\n\t\\end{proof}","sha256_utf8":"b967cbd4fd0b27cffd5fa62d44654cb1deb49c37706c4e16982abe97caa92947"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"凸组合控制目标函数","source_units":[{"start":34919,"end":35247,"start_line":324,"end_line":327,"raw_tex":"\t\tSince $x^{k + 1} = \\left(1 - t_{k}^{-1}\\right)x^{k} + t_{k}^{-1}z^{k + 1}$ (\\cf \\eqref{FGL:AccStep}) we obtain from the convexity of $\\Psi\\left(\\cdot\\right)$ that\n\t\t\\begin{equation*}\n\t\t\t\\Psi\\left(x^{k + 1}\\right) \\leq \\left(1 - t_{k}^{-1}\\right)\\Psi\\left(x^{k}\\right) + t_{k}^{-1}\\Psi\\left(z^{k + 1}\\right).\n\t\t\\end{equation*}\n","sha256_utf8":"e08575853f88260884f1a996dd80b9c3521c3d46ba12328d615618f07136dcc2"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"目标间隙加权递推","source_units":[{"start":35247,"end":35639,"start_line":328,"end_line":331,"raw_tex":"\t\tTherefore, multiplying both sides by $t_{k}^{p}$ (recalling that $t_{k}^{p} - t_{k}^{p - 1} = t_{k - 1}^{p}$), we obtain\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{p}\\left(\\Psi\\left(x^{k + 1}\\right) - \\Psi\\left(\\xi\\right)\\right) - t_{k - 1}^{p}\\left(\\Psi\\left(x^{k}\\right) - \\Psi\\left(\\xi\\right)\\right) \\leq t_{k}^{p - 1}\\left(\\Psi\\left(z^{k + 1}\\right) - \\Psi\\left(\\xi\\right)\\right).\n\t\t\\end{equation*}\n","sha256_utf8":"d420023d7e3266409a8833cda21188c716f95f375efe633a841496c69cfd27c4"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"约束线性项递推","source_units":[{"start":35639,"end":35875,"start_line":332,"end_line":335,"raw_tex":"\t\tIn addition, using again \\eqref{FGL:AccStep}, yields that\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{p}\\act{\\eta , \\AAA x^{k + 1} - b} - t_{k - 1}^{p}\\act{\\eta , \\AAA x^{k} - b} = t_{k}^{p - 1}\\act{\\eta , \\AAA z^{k + 1} - b}.\n\t\t\\end{equation*}\t\t\t\n","sha256_utf8":"0e15bc8b3667dfb7a9937510c3ed87945f38bc102b19d0bd0b0bd12c4518c871"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"合成拉格朗日间隙","source_units":[{"start":35875,"end":36290,"start_line":336,"end_line":339,"raw_tex":"\t\tCombining these two facts yields (using the definition of the Lagrangian $\\Lag$)\n\t\t\\begin{equation} \\label{L:FGLMain:1}\n\t\t\tt_{k}^{p}\\left(\\Lag\\left(x^{k + 1} , \\eta\\right) - \\Lag\\left(\\xi , \\eta\\right)\\right) - t_{k - 1}^{p}\\left(\\Lag\\left(x^{k} , \\eta\\right) - \\Lag\\left(\\xi , \\eta\\right)\\right) \\leq t_{k}^{p - 1}\\left(\\Lag\\left(z^{k + 1} , \\eta\\right) - \\Lag\\left(\\xi , \\eta\\right)\\right).\n\t\t\\end{equation}\t\t\t\n","sha256_utf8":"157e080b634514f5fd433d38ad0a9e4a89bb1fd21bf8716860be529520949687"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"展开约束残差平方","source_units":[{"start":36290,"end":36665,"start_line":340,"end_line":343,"raw_tex":"\t\tNow, we will again use the relation $x^{k + 1} = \\left(1 - t_{k}^{-1}\\right)x^{k} + t_{k}^{-1}z^{k + 1}$ to obtain\n\t\t\\begin{equation*}\n\t\t\t\\norm{\\AAA x^{k + 1} - b}^{2}  = \\left(1 - t_{k}^{-1}\\right)^{2}\\norm{\\AAA x^{k} - b}^{2} + t_{k}^{-2}\\norm{\\AAA z^{k + 1} - b}^{2} + 2\\left(1 - t_{k}^{-1}\\right)t_{k}^{-1}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b}.\n\t\t\\end{equation*}\t\t\n","sha256_utf8":"9c5fc8f958c6711cfdfafa185ede4931becfac46a8f1b6624b911fc3169967f9"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"罚项加权差分恒等式","source_units":[{"start":36665,"end":37128,"start_line":344,"end_line":347,"raw_tex":"\t\tMultiplying both sides of the above equality by $\\rho t_{k}^{2p}/2$ (recalling $\\rho_{k} = \\rho t_{k}^{p - 1}$ and $t_{k}^{p} - t_{k}^{p - 1} = t_{k - 1}^{p}$) yields\n\t\t\\begin{align}\n\t\t\t\\frac{\\rho t_{k}^{2p}}{2}\\norm{\\AAA x^{k + 1} - b}^{2} - \\frac{\\rho t_{k - 1}^{2p}}{2}\\norm{\\AAA x^{k} - b}^{2} = \\frac{\\rho_{k}t_{k}^{p - 1}}{2}\\norm{\\AAA z^{k + 1} - b}^{2} + \\rho_{k}t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b}. \\label{L:FGLMain:2}\n\t\t\\end{align}\n","sha256_utf8":"98777165148c5feb61f805c8efd26afb493f33bb478dba64cb0c1e00084bc7ee"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"组合增广间隙差分","source_units":[{"start":37128,"end":37626,"start_line":348,"end_line":351,"raw_tex":"\t\tTherefore, adding \\eqref{L:FGLMain:1} to \\eqref{L:FGLMain:2} yields (recall that $\\norm{\\AAA\\xi - b}^{2} = 0$) with $s_{k} = \\Lag_{\\rho t_{k - 1}^{p}}\\left(x^{k} , \\eta\\right) - \\Lag_{\\rho t_{k - 1}^{p}}\\left(\\xi , \\eta\\right)$,\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{p}s_{k + 1} - t_{k - 1}^{p}s_{k} \\leq t_{k}^{p - 1}\\left(\\Lag_{\\rho_{k}}\\left(z^{k + 1} , \\eta\\right) - \\Lag_{\\rho_{k}}\\left(\\xi , \\eta\\right)\\right) + \\rho_{k}t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b}.\n\t\t\\end{equation*}\t\t\t\n","sha256_utf8":"cf525bdf3aecc38b8d26cce5557f77a1486e0820350970dfe8268c32b20d0628"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"代入单步估计","source_units":[{"start":37626,"end":38217,"start_line":352,"end_line":356,"raw_tex":"\t\tFrom Lemma \\ref{L:FGLIter}, after we multiplied both sides of \\eqref{L:FGLIter:0} by $t_{k}^{p - 1}$ (recall that $\\rho_{k} = \\rho t_{k}^{p - 1}$), we obtain that\n\t\t\\begin{align*}\n\t\t\tt_{k}^{p - 1}\\left(\\Lag_{\\rho_{k}}\\left(z^{k + 1} , \\eta\\right) - \\Lag_{\\rho_{k}}\\left(\\xi , \\eta\\right)\\right) + \\rho_{k}t_{k - 1}^{p}\\act{\\AAA x^{k} - b , \\AAA z^{k + 1} - b} & \\leq \\frac{\\tau_{k}\\rho_{k}}{\\rho}\\Delta_{P}\\left(\\xi , z^{k} , z^{k + 1}\\right) - \\frac{\\rho_{k}\\sigma}{2\\rho}\\norm{\\xi - z^{k + 1}}^{2} \\\\\n\t\t\t& + \\frac{1}{\\mu\\rho}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right).\n\t\t\\end{align*}\t\t\n","sha256_utf8":"3e6bf90bf629457ab85a4ef77ca134ad51607be09303ef36f3cb10c9da520f22"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"得到间隙递推结论","source_units":[{"start":38217,"end":38593,"start_line":357,"end_line":361,"raw_tex":"\t\tBy combining the last two inequalities, we thus get\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{p}s_{k + 1} - t_{k - 1}^{p}s_{k} \\leq \\frac{\\tau_{k}\\rho_{k}}{\\rho}\\Delta_{P}\\left(\\xi , z^{k} , z^{k + 1}\\right) - \\frac{\\rho_{k}\\sigma}{2\\rho}\\norm{\\xi - z^{k + 1}}^{2} + \\frac{1}{\\mu\\rho}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right),\n\t\t\\end{equation*}\t\t\t\t\n\t\twhich proves the desired result.\n","sha256_utf8":"328809ba031f7703e3bbc073be5d470dc8acb853ebb3ee979a7e3eb1aa8cee18"}]},{"id":"S032","semantic_type":"lemma","kind":"lemma","title":"加速参数增长界","source_units":[{"start":38783,"end":39259,"start_line":364,"end_line":370,"raw_tex":"\\begin{lemma} \\label{L:Tk}\n\t\tLet $\\seq{t}{k}$ be the sequence of positive numbers generated from $t_{0} = 1$ via the equation $t_{k + 1}^{p} - t_{k}^{p} = t_{k + 1}^{p - 1}$, with $p = 1$ or $p = 2$. Then, for any $k \\geq 0$, we have\n\t\\begin{itemize}\n\t\t\\item[$\\rm{(i)}$] For $p = 1$, we have $t_{k} = k + 1$.\n\t\t\\item[$\\rm{(ii)}$] For $p = 2$, we have $t_{k} \\geq \\left(k + 1\\right)/2$. Moreover, one has $t_{k + 1}^{2} \\leq t_{k}^{2} + 2t_{k}$.\n\t\\end{itemize}\t\t\t\t\n\t\\end{lemma}","sha256_utf8":"222a94addff0949b903956bc0ddf482e0b14cb68b0d1d6904a6618f7e17754e7"}]},{"id":"S033","semantic_type":"proof","kind":"proof","title":"参数增长界完整证明","source_units":[{"start":39261,"end":39683,"start_line":371,"end_line":373,"raw_tex":"\\begin{proof}\n\t\tThe first item is immediate as well as the first part of the second item (see, for instance, \\cite[Lemma 4.3]{BT2009}). Moreover, note that by the given recursion $t_{k + 1}^{2} - t_{k}^{2} = t_{k + 1}$, one also has $t_{k + 1} = t_{k} + t_{k + 1}/\\left(t_{k} + t_{k + 1}\\right)$, and hence it follows that $t_{k + 1}^{2} = t_{k}^{2} + t_{k + 1} \\leq t_{k}^{2} + 2t_{k}$, since $t_{k} \\geq 1$.\n\t\\end{proof}","sha256_utf8":"df4f8ac1179b03fa414a856a78499b1b0e393e2fd6a59dbda71ac1f6cb6d2ffb"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"递推导出参数上下界","source_units":[{"start":39275,"end":39671,"start_line":372,"end_line":372,"raw_tex":"\t\tThe first item is immediate as well as the first part of the second item (see, for instance, \\cite[Lemma 4.3]{BT2009}). Moreover, note that by the given recursion $t_{k + 1}^{2} - t_{k}^{2} = t_{k + 1}$, one also has $t_{k + 1} = t_{k} + t_{k + 1}/\\left(t_{k} + t_{k + 1}\\right)$, and hence it follows that $t_{k + 1}^{2} = t_{k}^{2} + t_{k + 1} \\leq t_{k}^{2} + 2t_{k}$, since $t_{k} \\geq 1$.\n","sha256_utf8":"eb70a7d5449e2d26cd9ed6566bbb377a8f0812d85d84c7b30075f1e4f057349c"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"强凸情形代入间隙递推","source_units":[{"start":41729,"end":42146,"start_line":395,"end_line":398,"raw_tex":"\t\tFrom Lemma \\ref{L:FGLMain}, after we substitute $\\xi = x^{\\ast}$ and $p = 2$ (recall that $\\rho_{k} = \\rho t_{k}$ and $\\tau_{k} = t_{k}$), we obtain\n\t\t\\begin{equation*}\n\t\t\tt_{k}^{2}s_{k + 1} - t_{k - 1}^{2}s_{k} \\leq t_{k}^{2}\\Delta_{P}\\left(x^{\\ast} , z^{k} , z^{k + 1}\\right) - \\frac{t_{k}\\sigma}{2}\\norm{x^{\\ast} - z^{k + 1}}^{2} + \\frac{1}{\\mu\\rho}\\Delta\\left(\\eta , y^{k} , y^{k + 1}\\right).\n\t\t\\end{equation*}\n","sha256_utf8":"715024329e020ba2b02d80f009898c4fd893b5e1ea8e7f237fbd25b7df04ed9b"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"构造可望远镜求和势能","source_units":[{"start":42146,"end":42832,"start_line":399,"end_line":403,"raw_tex":"\t\tUsing the definition of $\\Delta_{P}$ and $\\Delta$ (see \\eqref{D:Delta}) and the fact that $P \\preceq \\left(\\sigma/2\\right)I_{n}$ we have that\n\t\t\\begin{align*}\n\t\t\tt_{k}^{2}s_{k + 1} - t_{k - 1}^{2}s_{k} & \\leq \\frac{t_{k}^{2}}{2}\\left(\\norm{x^{\\ast} - z^{k}}_{P}^{2} - \\norm{x^{\\ast} - z^{k + 1}}_{P}^{2}\\right) - t_{k}\\norm{x^{\\ast} - z^{k + 1}}_{P}^{2} + \\frac{1}{2\\mu\\rho}\\left(\\norm{\\eta - y^{k}}^{2} -  \\norm{\\eta - y^{k + 1}}^{2}\\right) \\nonumber \\\\\n\t\t\t& \\leq \\frac{1}{2}\\left(t_{k}^{2}\\norm{x^{\\ast} - z^{k}}_{P}^{2} - t_{k + 1}^{2}\\norm{x^{\\ast} - z^{k + 1}}_{P}^{2}\\right) + \\frac{1}{2\\mu\\rho}\\left(\\norm{\\eta - y^{k}}^{2} - \\norm{\\eta - y^{k + 1}}^{2}\\right),\n\t\t\\end{align*}\n","sha256_utf8":"d3772153483e73c882e9c8fbb521878c114ae41d74843665a927f6ffbdff90d3"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"求和得到终点间隙界","source_units":[{"start":42832,"end":43386,"start_line":404,"end_line":407,"raw_tex":"\t\twhere the last inequality follows from Lemma \\ref{L:Tk}(ii). Summing this inequality for all $k = 0 , 1 , \\ldots , N - 1$ (recall that $t_{-1} = 0$ and $t_{0} = 1$), it follows (where we use  the definition of $s_{N}$, see \\eqref{laggap}, for the first inequality below)\n\t\t\\begin{equation*}\n\tt_{N - 1}^{2}\\left(\\Psi\\left(x^{N}\\right) - \\Psi\\left(x^{\\ast}\\right) + \\act{\\eta , \\AAA x^{N} - b}\\right) \\leq\tt_{N - 1}^{2}s_{N} - t_{-1}^{2}s_{0} \\leq \\frac{1}{2}\\norm{x^{\\ast} - z^{0}}_{P}^{2} + \\frac{1}{2\\mu\\rho}\\norm{\\eta - y^{0}}^{2}.\n\t\t\\end{equation*}\n","sha256_utf8":"2ece63f9cf9ef444151a364275cb12d5950f39112a60546a863aaccf5d9bd5dc"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"对偶球最大化获得联合界","source_units":[{"start":43386,"end":43725,"start_line":408,"end_line":411,"raw_tex":"\t\tTherefore, by taking the maximum of both sides over $\\norm{\\eta} \\leq c$, proves that (recall that $t_{N - 1}^{2} \\geq N^{2}/4$ using Lemma \\ref{L:Tk}(ii))\n\t\t\\begin{equation*}\n\t\t\t\\Psi\\left(x^{N}\\right) - \\Psi\\left(x^{\\ast}\\right) + c\\norm{\\AAA x^{N} - b} \\leq \\frac{B_{\\rho , c}\\left(x^{\\ast}\\right)}{2N^{2}} := \\beta,\n\t\t\\end{equation*}\n","sha256_utf8":"ed5e1cad0512bc28d7b289cf1a01fe915268583436e192986de6c5e0877cf333"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"分离目标误差与可行性界","source_units":[{"start":43725,"end":44230,"start_line":412,"end_line":416,"raw_tex":"\t\tfrom which the estimate \\eqref{T:FGLrate:1} immediately follows. Moreover, since $\\left(x^{\\ast} , y^{\\ast}\\right)$ is a saddle point of problem (P), with $c \\geq 2\\norm{y^{\\ast}}$, using the above inequality it follows that\n\t\t\\begin{equation*}\n\t\t\tc\\norm{\\AAA x^{N} - b} \\leq \\Psi\\left(x^{\\ast}\\right) - \\Psi\\left(x^{N}\\right) + \\beta \\leq \\act{y^{\\ast} , \\AAA x^{N} - b} + \\beta \\leq \\frac{c}{2}\\norm{\\AAA x^{N} - b} + \\beta,\n\t\t\\end{equation*}\n\t\tand the desired estimate \\eqref{T:FGLrate:2} is proved.\n","sha256_utf8":"59b95f265dccd36f246e3cb9b88d0b6375440da638506f5b2c346dc62a7054a5"}]},{"id":"S040","semantic_type":"external_fact","kind":"external_fact","title":"标准加速序列下界","source_units":[],"declared_external":true,"declaration_text":["BT2009, Lemma 4.3 supplies the standard acceleration-sequence lower bound: if t_0=1 and t_{k+1}=(1+sqrt(1+4t_k^2))/2, then t_k >= (k+1)/2."]},{"id":"S041","semantic_type":"external_fact","kind":"external_fact","title":"欧氏范数基本性质","source_units":[],"declared_external":true,"declaration_text":["For c>0, sup_{||eta||<=c}<eta,r>=c||r||. The triangle inequality gives ||eta-y||<=||eta||+||y||, and Cauchy–Schwarz gives <u,v><=||u||||v||."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S011","edges":[{"source_id":"S001","target_id":"S005","kind":"supports"},{"source_id":"S002","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S007","target_id":"S016","kind":"supports"},{"source_id":"S004","target_id":"S016","kind":"supports"},{"source_id":"S006","target_id":"S017","kind":"supports"},{"source_id":"S008","target_id":"S017","kind":"supports"},{"source_id":"S004","target_id":"S017","kind":"supports"},{"source_id":"S006","target_id":"S018","kind":"supports"},{"source_id":"S009","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S016","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S006","target_id":"S020","kind":"supports"},{"source_id":"S002","target_id":"S020","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S015","kind":"supports"},{"source_id":"S018","target_id":"S015","kind":"supports"},{"source_id":"S019","target_id":"S015","kind":"supports"},{"source_id":"S020","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S001","target_id":"S023","kind":"supports"},{"source_id":"S006","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S009","target_id":"S024","kind":"supports"},{"source_id":"S006","target_id":"S025","kind":"supports"},{"source_id":"S009","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S002","target_id":"S026","kind":"supports"},{"source_id":"S006","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S009","target_id":"S028","kind":"supports"},{"source_id":"S026","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S001","target_id":"S029","kind":"supports"},{"source_id":"S002","target_id":"S029","kind":"supports"},{"source_id":"S009","target_id":"S029","kind":"supports"},{"source_id":"S014","target_id":"S030","kind":"supports"},{"source_id":"S009","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S022","kind":"supports"},{"source_id":"S026","target_id":"S022","kind":"supports"},{"source_id":"S027","target_id":"S022","kind":"supports"},{"source_id":"S028","target_id":"S022","kind":"supports"},{"source_id":"S029","target_id":"S022","kind":"supports"},{"source_id":"S030","target_id":"S022","kind":"supports"},{"source_id":"S031","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S006","target_id":"S034","kind":"supports"},{"source_id":"S040","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S021","target_id":"S035","kind":"supports"},{"source_id":"S013","target_id":"S035","kind":"supports"},{"source_id":"S003","target_id":"S035","kind":"supports"},{"source_id":"S009","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S004","target_id":"S036","kind":"supports"},{"source_id":"S013","target_id":"S036","kind":"supports"},{"source_id":"S032","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S009","target_id":"S037","kind":"supports"},{"source_id":"S002","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S032","target_id":"S038","kind":"supports"},{"source_id":"S010","target_id":"S038","kind":"supports"},{"source_id":"S041","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S003","target_id":"S039","kind":"supports"},{"source_id":"S002","target_id":"S039","kind":"supports"},{"source_id":"S010","target_id":"S039","kind":"supports"},{"source_id":"S041","target_id":"S039","kind":"supports"},{"source_id":"S035","target_id":"S012","kind":"supports"},{"source_id":"S036","target_id":"S012","kind":"supports"},{"source_id":"S037","target_id":"S012","kind":"supports"},{"source_id":"S038","target_id":"S012","kind":"supports"},{"source_id":"S039","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S011","kind":"supports"}]}}
{"record_id":"2011.08166v2","paper":{"title":"A Globally Convergent Proximal Newton-Type Method in Nonsmooth Convex Optimization","authors":["Boris S. Mordukhovich","Xiaoming Yuan","Shangzhi Zeng","Jin Zhang"],"arxiv_id":"2011.08166","version":"v2","journal":"Mathematical Programming","doi":"10.1007/s10107-022-01797-5","abstract_url":"https://arxiv.org/abs/2011.08166v2","pdf_url":"https://arxiv.org/pdf/2011.08166v2"},"context":{"raw_tex":"In this paper we consider a class of optimization problems of the\nfollowing type: \\begin{equation}\\label{prob_ori_1}\n\\min_{x\\in\\mathbb{R}^n}~F(x):=f(x)+g(x), \\end{equation} where both\nfunctions $f,g\\colon\\R^n\\to\\oR:=(-\\infty,\\infty]$ are proper,\nconvex, and lower semicontinuous (l.s.c.), while being structurally\ndifferent from each other. Namely, $f$ is assumed to be twice\ncontinuously differentiable with the Lipschitz continuous gradient\n$\\nabla f$ on its domain. On the other hand, $g$ is merely\ncontinuous on its domain; see Assumption~\\ref{assumptions_f} for the\nprecise formulations. It has been well recognized that model\n\\eqref{prob_ori_1}, known as a {\\em composite convex optimization\nproblem}, frequently appears in a variety of applications including,\ne.g., machine learning, signal processing, and statistics, where $f$\nis a {\\em loss function} and $g$ is a {\\em regularizer}; we keep\nthis terminology here. Note that problem \\eqref{prob_ori_1} contains\nin fact implicit constraints written as $x\\in\\O:=\\dom g$.\n\n\n\\begin{equation}\\label{lin} \\min_{x\\in\\mathbb{R}^n}~l_k(x) +\n\\frac{1}{2t}\\|x-x_k\\|^2 \\;\\; \\text{with} \\;\\; l_k(x):=f(x^k)+\\nabla\nf(x^k)^T(x-x^k)+g(x), \\end{equation}\n\n\\begin{equation}\\label{subproblem}\n\\min_{x\\in\\mathbb{R}^n}~q_k(x):=f(x^k)+\\nabla\nf(x^k)^T(x-x^k)+\\frac{1}{2}(x-x^k)^TH_k(x-x^k)+g(x), \\end{equation}\n\n\\begin{assumption}\\label{assumptions_f} Impose the following properties of the loss function and the regularizer in \\eqref{prob_ori_1}:\n\\begin{itemize}\n\\item[{\\bf(i)}] Both functions $f,g:{\\R^n}\\rightarrow (-\\infty,\\infty]$ are proper, l.s.c., and convex.\n\n\\item[{\\bf(ii)}] The \\revise{effective} domain of the loss function $\\dom f:=\\{x~|~f(x)<\\infty\\}$ is open, and $f(x)$ is twice continuously differentiable on a closed set $\\Omega\\supset \\dom f$.\n\n\\item[{\\bf(iii)}] The regularizer $g(x)$ is continuous on its domain and $\\varnothing \\ne\\dom g \\subset\\dom f$.\n\n\\item[{\\bf(iv)}] The gradient $\\nabla f(x)$ is Lipschitz continuous on a closed set $\\Omega$ from {\\rm(ii)} with Lipschitz constant $L_1>0$.\n\n\\item[{\\bf(v)}] Problem \\eqref{prob_ori_1} has a nonempty solution set ${\\cal{X}^*}:=\\arg\\min_{x\\in\\R^n}F(x)$ with the optimal value $F^*$.\n\\end{itemize}\n\\end{assumption}\n\n\\begin{equation}\\label{prox-map}\n\\hbox{Prox}_{g}(u):={\\rm argmin}\\Big\\{g(x)+\\frac{1}{2}\\|x-u\\|^2\\;\\Big|\\;x\\in\\R^n\\Big\\},\\quad u\\in\\R^n,\n\\end{equation}\n\n\\begin{equation}\\label{prox-gra}\n\\mathcal{G}(x):=x-\\mathrm{Prox}_{g}\\big(x-\\nabla f(x)\\big),\\quad x\\in\\R^n,\n\\end{equation}\n\nGiven a current iterate $x^k$ for each $k=0,1,\\ldots$, we select a {\\em positive semidefinite matrix} $B_k$ as an arbitrary approximation of the Hessian $\\nabla^2 f(x^k)$ satisfying the {\\em standing\nboundedness assumption}:\n\\begin{equation}\\label{assum_B_bounded}\n\\mbox{ there exists }\\;M\\ge 0\\;\\mbox{ such that }\\;\\|B_k\\|\\le M\\;\\mbox{ whenever }\\;k=0,1,\\ldots.\n\\end{equation}\nIf the gradient mapping $\\nabla f$ is uniformly Lipschitz continuous along the sequence of iterates with constant $L_1$, then \\eqref{assum_B_bounded} holds for $B_k=\\nabla^2 f(x^k)$ with $M=L_1$.\nIn the general case of $B_k$, pick any constants $c>0$ and $\\rho\\in(0,1]$ and, using the prox-regular mapping \\eqref{prox-gra}, consider the positive number $\\alpha_k:=c\\|\\mathcal{G}(x^k)\\|^{\\rho}$\nand define the {\\em quasi-Newton approximation} of the Hessian of $f$ at $x^k$ by\n\\begin{equation}\\label{alg-app}\nH_k:=B_k+\\alpha_kI\\;\\mbox{ for all }\\;k=0,1,\\ldots,\n\\end{equation}\nwhich is a positive definite matrix. Then similarly to \\cite{byrd2016inexact}, but with the different approximation \\eqref{alg-app}, denote\n\\begin{equation}\\label{opt-mes}\nr_k(x):=x-\\mathrm{Prox}_{g}\\left(x-\\nabla f(x^k)-H_k(x-x^k)\\right)\n\\end{equation}\nand select $\\hat{x}^k$ as an {\\em approximate minimizer} of the quadratic {\\em subproblem} for \\eqref{prob_ori_1} given by\n\\begin{equation}\\label{subproblem1}\n\\min_{x\\in\\mathbb{R}^n}~q_k(x):=f(x^k)+\\nabla f(x^k)^T(x-x^k)+\\frac{1}{2}(x-x^k)H_k(x-x^k)+g(x)\n\\end{equation}\nwith the {\\em residual} number $\\|r_k(\\hat x^k)\\|$ measuring the approximate optimality of $\\hat x^k$ in \\eqref{subproblem1}. Observing that $\\|r_k(\\hat{x}^k)\\|=0$ if and only if $\\hat{x}^k$ is an\n{\\em exact} solution to subproblem \\eqref{subproblem1}, we use the nonnegative number {\\color{black} $\\|r_k(\\hat{x}^k)\\|$}\nwith $r_k(x)$ taken from \\eqref{opt-mes} as the {\\em optimality measure} of $\\hat x^k$ in subproblem \\eqref{subproblem1}. Adapting the scheme of \\cite{byrd2016inexact} in our new setting, let us impose\nthe following two estimates as {\\em inexact conditions} for choosing $\\hat{x}^k$ as an approximate solution to subproblem \\eqref{subproblem1}:\n{\\color{black}\\begin{equation}\\label{inexact_condition_1}\n\\|r_k(\\hat{x}^k)\\|\\le\\eta_k\\|\\mathcal{G}(x^k)\\|\\;\\mbox{ and }\\;q_k(\\hat{x}^k)\\le q_k(x^k)\n\\end{equation}}%\nwith the parameter $\\eta_k:=\\nu\\min\\{1,\\|\\mathcal{G}(x^k)\\|^{\\varrho}\\}$ defined via \\eqref{opt-mes} and some numbers $\\nu\\in[0,1)$ and $\\varrho>0$. \\revise{Since any exact solution to subproblem \\eqref{subproblem1} always fulfills the two inexact conditions in \\eqref{inexact_condition_1}, a point $\\hat x^k$ satisfying \\eqref{inexact_condition_1} always exists. For many application problems, the mapping $\\mathrm{Prox}_{g}$ is easy to calculate, thus inexact conditions \\eqref{inexact_condition_1} can be readily verified.\n\n\n\\begin{algorithm}\n\\caption{Proximal Newton-type method}\\label{Newton}\n\\begin{algorithmic}[1]\n\\State Choose $x^0\\in\\mathbb{R}^n$, $0<\\theta,\\sigma,\\gamma<1$, $C>F(x^0)$, $\\revise{\\bar{\\alpha}}, c>0$, and $\\rho\\in(0,1]$.\n\\For{$k=0,1,\\ldots$}{\n\\begin{itemize}\n\\item[1.] Update the approximation of the Hessian matrix $B_k$.\n\\item[2.] Form the quadratic model \\eqref{subproblem} with $H_k:=B_k+\\alpha_kI$ and \\revise{$\\alpha_k:=\\min \\left\\{ \\bar{\\alpha}, c\\|\\mathcal{G}(x^k)\\|^{\\rho}\\right\\} $  }.\n\\item[3.] Obtain an inexact solution $\\hat{x}^k$ of \\eqref{subproblem} satisfying the conditions in \\eqref{inexact_condition_1}.\n\\item[4.] If $k=0$, let $\\vartheta_1:=\\mathcal{G}(x^0)$ and go to Step~5. For $k\\ge 1$, if $\\|\\mathcal{G}(\\hat{x}^k)\\|\\le\\sigma\\vartheta_k$ and $F(\\hat{x}^k)\\le C$, let $t_k:=1$, $\\vartheta_{k+1}:=\n\\|\\mathcal{G}(\\hat{x}^k)\\|$,\nand go to Step~6. Otherwise, let $\\vartheta_{k+1}:=\\vartheta_k$ and go to Step~5.\n\\item[5.] Perform a backtracking line search along the direction $d^k:=\\hat{x}^k-x^k$ by setting $t_k:=\\gamma^{m_k}$, where $m_k$ is the smallest nonnegative integer $m$ such that\n\\begin{equation}\\label{line_search}\nF(x^k+\\gamma^md^k)\\le F(x^k)-\\theta\\alpha_k\\gamma^m\\|d^k\\|^2.\n\\end{equation}\n\\item[6.] Set $x^{k+1}:=x^k+t_kd^k$.\n\\end{itemize}}\n\\EndFor\n\\end{algorithmic}\n\\end{algorithm}\n\n\\begin{equation}\\label{K}\nK:=\\big\\{0,1,\\ldots\\big\\}\\;\\mbox{ and }\\;K_0:=\\{0\\}\\cup\\big\\{k+1\\in K ~\\big|~\\text{Step~$5$ is not applied at iteration $k$}\\big\\}.\n\\end{equation}","units":[{"start":6005,"end":7033,"start_line":150,"end_line":165,"raw_tex":"In this paper we consider a class of optimization problems of the\nfollowing type: \\begin{equation}\\label{prob_ori_1}\n\\min_{x\\in\\mathbb{R}^n}~F(x):=f(x)+g(x), \\end{equation} where both\nfunctions $f,g\\colon\\R^n\\to\\oR:=(-\\infty,\\infty]$ are proper,\nconvex, and lower semicontinuous (l.s.c.), while being structurally\ndifferent from each other. Namely, $f$ is assumed to be twice\ncontinuously differentiable with the Lipschitz continuous gradient\n$\\nabla f$ on its domain. On the other hand, $g$ is merely\ncontinuous on its domain; see Assumption~\\ref{assumptions_f} for the\nprecise formulations. It has been well recognized that model\n\\eqref{prob_ori_1}, known as a {\\em composite convex optimization\nproblem}, frequently appears in a variety of applications including,\ne.g., machine learning, signal processing, and statistics, where $f$\nis a {\\em loss function} and $g$ is a {\\em regularizer}; we keep\nthis terminology here. Note that problem \\eqref{prob_ori_1} contains\nin fact implicit constraints written as $x\\in\\O:=\\dom g$.\n","sha256_utf8":"2bfecaa3e2439243be0f0c31c6319efe50f0e42b73e7b2690ac4603076d1a282"},{"start":7457,"end":7622,"start_line":174,"end_line":176,"raw_tex":"\\begin{equation}\\label{lin} \\min_{x\\in\\mathbb{R}^n}~l_k(x) +\n\\frac{1}{2t}\\|x-x_k\\|^2 \\;\\; \\text{with} \\;\\; l_k(x):=f(x^k)+\\nabla\nf(x^k)^T(x-x^k)+g(x), \\end{equation}","sha256_utf8":"4a27be743f633e30244ca2ee528a031abb9b42a3200ec02a3353d671ca83c64a"},{"start":8521,"end":8669,"start_line":191,"end_line":193,"raw_tex":"\\begin{equation}\\label{subproblem}\n\\min_{x\\in\\mathbb{R}^n}~q_k(x):=f(x^k)+\\nabla\nf(x^k)^T(x-x^k)+\\frac{1}{2}(x-x^k)^TH_k(x-x^k)+g(x), \\end{equation}","sha256_utf8":"aa57a0728448c3bfbdc6413a8a6cfe380c3bd53162b2cf12b65040a23f405df7"},{"start":14383,"end":15261,"start_line":267,"end_line":279,"raw_tex":"\\begin{assumption}\\label{assumptions_f} Impose the following properties of the loss function and the regularizer in \\eqref{prob_ori_1}:\n\\begin{itemize}\n\\item[{\\bf(i)}] Both functions $f,g:{\\R^n}\\rightarrow (-\\infty,\\infty]$ are proper, l.s.c., and convex.\n\n\\item[{\\bf(ii)}] The \\revise{effective} domain of the loss function $\\dom f:=\\{x~|~f(x)<\\infty\\}$ is open, and $f(x)$ is twice continuously differentiable on a closed set $\\Omega\\supset \\dom f$.\n\n\\item[{\\bf(iii)}] The regularizer $g(x)$ is continuous on its domain and $\\varnothing \\ne\\dom g \\subset\\dom f$.\n\n\\item[{\\bf(iv)}] The gradient $\\nabla f(x)$ is Lipschitz continuous on a closed set $\\Omega$ from {\\rm(ii)} with Lipschitz constant $L_1>0$.\n\n\\item[{\\bf(v)}] Problem \\eqref{prob_ori_1} has a nonempty solution set ${\\cal{X}^*}:=\\arg\\min_{x\\in\\R^n}F(x)$ with the optimal value $F^*$.\n\\end{itemize}\n\\end{assumption}","sha256_utf8":"ec355afa1c82b2c19d9c77d58e51d6b3f2f3fa863fb1f531032291f595d65028"},{"start":23094,"end":23244,"start_line":359,"end_line":361,"raw_tex":"\\begin{equation}\\label{prox-map}\n\\hbox{Prox}_{g}(u):={\\rm argmin}\\Big\\{g(x)+\\frac{1}{2}\\|x-u\\|^2\\;\\Big|\\;x\\in\\R^n\\Big\\},\\quad u\\in\\R^n,\n\\end{equation}","sha256_utf8":"b38b9b586b705472f2e5898b35d76ec78a8810186d30520f4361f618f7eff3ae"},{"start":24621,"end":24743,"start_line":373,"end_line":375,"raw_tex":"\\begin{equation}\\label{prox-gra}\n\\mathcal{G}(x):=x-\\mathrm{Prox}_{g}\\big(x-\\nabla f(x)\\big),\\quad x\\in\\R^n,\n\\end{equation}","sha256_utf8":"6a7e540055408a592f3b6cfb6ddfaebed37360d6b3810441d8d596b70e5a3c7e"},{"start":34933,"end":37766,"start_line":544,"end_line":570,"raw_tex":"Given a current iterate $x^k$ for each $k=0,1,\\ldots$, we select a {\\em positive semidefinite matrix} $B_k$ as an arbitrary approximation of the Hessian $\\nabla^2 f(x^k)$ satisfying the {\\em standing\nboundedness assumption}:\n\\begin{equation}\\label{assum_B_bounded}\n\\mbox{ there exists }\\;M\\ge 0\\;\\mbox{ such that }\\;\\|B_k\\|\\le M\\;\\mbox{ whenever }\\;k=0,1,\\ldots.\n\\end{equation}\nIf the gradient mapping $\\nabla f$ is uniformly Lipschitz continuous along the sequence of iterates with constant $L_1$, then \\eqref{assum_B_bounded} holds for $B_k=\\nabla^2 f(x^k)$ with $M=L_1$.\nIn the general case of $B_k$, pick any constants $c>0$ and $\\rho\\in(0,1]$ and, using the prox-regular mapping \\eqref{prox-gra}, consider the positive number $\\alpha_k:=c\\|\\mathcal{G}(x^k)\\|^{\\rho}$\nand define the {\\em quasi-Newton approximation} of the Hessian of $f$ at $x^k$ by\n\\begin{equation}\\label{alg-app}\nH_k:=B_k+\\alpha_kI\\;\\mbox{ for all }\\;k=0,1,\\ldots,\n\\end{equation}\nwhich is a positive definite matrix. Then similarly to \\cite{byrd2016inexact}, but with the different approximation \\eqref{alg-app}, denote\n\\begin{equation}\\label{opt-mes}\nr_k(x):=x-\\mathrm{Prox}_{g}\\left(x-\\nabla f(x^k)-H_k(x-x^k)\\right)\n\\end{equation}\nand select $\\hat{x}^k$ as an {\\em approximate minimizer} of the quadratic {\\em subproblem} for \\eqref{prob_ori_1} given by\n\\begin{equation}\\label{subproblem1}\n\\min_{x\\in\\mathbb{R}^n}~q_k(x):=f(x^k)+\\nabla f(x^k)^T(x-x^k)+\\frac{1}{2}(x-x^k)H_k(x-x^k)+g(x)\n\\end{equation}\nwith the {\\em residual} number $\\|r_k(\\hat x^k)\\|$ measuring the approximate optimality of $\\hat x^k$ in \\eqref{subproblem1}. Observing that $\\|r_k(\\hat{x}^k)\\|=0$ if and only if $\\hat{x}^k$ is an\n{\\em exact} solution to subproblem \\eqref{subproblem1}, we use the nonnegative number {\\color{black} $\\|r_k(\\hat{x}^k)\\|$}\nwith $r_k(x)$ taken from \\eqref{opt-mes} as the {\\em optimality measure} of $\\hat x^k$ in subproblem \\eqref{subproblem1}. Adapting the scheme of \\cite{byrd2016inexact} in our new setting, let us impose\nthe following two estimates as {\\em inexact conditions} for choosing $\\hat{x}^k$ as an approximate solution to subproblem \\eqref{subproblem1}:\n{\\color{black}\\begin{equation}\\label{inexact_condition_1}\n\\|r_k(\\hat{x}^k)\\|\\le\\eta_k\\|\\mathcal{G}(x^k)\\|\\;\\mbox{ and }\\;q_k(\\hat{x}^k)\\le q_k(x^k)\n\\end{equation}}%\nwith the parameter $\\eta_k:=\\nu\\min\\{1,\\|\\mathcal{G}(x^k)\\|^{\\varrho}\\}$ defined via \\eqref{opt-mes} and some numbers $\\nu\\in[0,1)$ and $\\varrho>0$. \\revise{Since any exact solution to subproblem \\eqref{subproblem1} always fulfills the two inexact conditions in \\eqref{inexact_condition_1}, a point $\\hat x^k$ satisfying \\eqref{inexact_condition_1} always exists. For many application problems, the mapping $\\mathrm{Prox}_{g}$ is easy to calculate, thus inexact conditions \\eqref{inexact_condition_1} can be readily verified.\n","sha256_utf8":"b705ac270e86660a00d29c9023db0aebfaa6a0be17db4f6e74287ca80e4b84d2"},{"start":38388,"end":39706,"start_line":574,"end_line":594,"raw_tex":"\\begin{algorithm}\n\\caption{Proximal Newton-type method}\\label{Newton}\n\\begin{algorithmic}[1]\n\\State Choose $x^0\\in\\mathbb{R}^n$, $0<\\theta,\\sigma,\\gamma<1$, $C>F(x^0)$, $\\revise{\\bar{\\alpha}}, c>0$, and $\\rho\\in(0,1]$.\n\\For{$k=0,1,\\ldots$}{\n\\begin{itemize}\n\\item[1.] Update the approximation of the Hessian matrix $B_k$.\n\\item[2.] Form the quadratic model \\eqref{subproblem} with $H_k:=B_k+\\alpha_kI$ and \\revise{$\\alpha_k:=\\min \\left\\{ \\bar{\\alpha}, c\\|\\mathcal{G}(x^k)\\|^{\\rho}\\right\\} $  }.\n\\item[3.] Obtain an inexact solution $\\hat{x}^k$ of \\eqref{subproblem} satisfying the conditions in \\eqref{inexact_condition_1}.\n\\item[4.] If $k=0$, let $\\vartheta_1:=\\mathcal{G}(x^0)$ and go to Step~5. For $k\\ge 1$, if $\\|\\mathcal{G}(\\hat{x}^k)\\|\\le\\sigma\\vartheta_k$ and $F(\\hat{x}^k)\\le C$, let $t_k:=1$, $\\vartheta_{k+1}:=\n\\|\\mathcal{G}(\\hat{x}^k)\\|$,\nand go to Step~6. Otherwise, let $\\vartheta_{k+1}:=\\vartheta_k$ and go to Step~5.\n\\item[5.] Perform a backtracking line search along the direction $d^k:=\\hat{x}^k-x^k$ by setting $t_k:=\\gamma^{m_k}$, where $m_k$ is the smallest nonnegative integer $m$ such that\n\\begin{equation}\\label{line_search}\nF(x^k+\\gamma^md^k)\\le F(x^k)-\\theta\\alpha_k\\gamma^m\\|d^k\\|^2.\n\\end{equation}\n\\item[6.] Set $x^{k+1}:=x^k+t_kd^k$.\n\\end{itemize}}\n\\EndFor\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"01e8b9f676483377385b71da4b97e8b00dad5222da42386e38109e901214d970"},{"start":46469,"end":46641,"start_line":699,"end_line":701,"raw_tex":"\\begin{equation}\\label{K}\nK:=\\big\\{0,1,\\ldots\\big\\}\\;\\mbox{ and }\\;K_0:=\\{0\\}\\cup\\big\\{k+1\\in K ~\\big|~\\text{Step~$5$ is not applied at iteration $k$}\\big\\}.\n\\end{equation}","sha256_utf8":"bae38c21497112117ff40593a0de12d3a03a560b219195c059b90fd599a697d2"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{algorithm_convergence}\nLet $\\{x^k\\}$ be the sequence of iterates generated by Algorithm~{\\rm\\ref{Newton}} with an arbitrarily chosen starting point $x^0\\in\\R^n$ under the standing assumptions made. Then we have the residual\ncondition\n\\begin{equation}\\label{glob}\n\\liminf_{k\\rightarrow\\infty}\\|\\mathcal{G}(x^k)\\|=0\n\\end{equation}\nalong the prox-gradient mapping \\eqref{prox-gra}. Furthermore, the boundedness of $\\{x^k\\}$ yields the convergence to the optimal value $\\lim\\limits_{k\\rightarrow\\infty}F(x^{k})=F^*$ and ensures that any\nlimiting point of $\\{x^k\\}$ is a global minimizer in \\eqref{prob_ori_1}.\n\\end{theorem}","units":[{"start":46643,"end":47284,"start_line":703,"end_line":711,"raw_tex":"\\begin{theorem}\\label{algorithm_convergence}\nLet $\\{x^k\\}$ be the sequence of iterates generated by Algorithm~{\\rm\\ref{Newton}} with an arbitrarily chosen starting point $x^0\\in\\R^n$ under the standing assumptions made. Then we have the residual\ncondition\n\\begin{equation}\\label{glob}\n\\liminf_{k\\rightarrow\\infty}\\|\\mathcal{G}(x^k)\\|=0\n\\end{equation}\nalong the prox-gradient mapping \\eqref{prox-gra}. Furthermore, the boundedness of $\\{x^k\\}$ yields the convergence to the optimal value $\\lim\\limits_{k\\rightarrow\\infty}F(x^{k})=F^*$ and ensures that any\nlimiting point of $\\{x^k\\}$ is a global minimizer in \\eqref{prob_ori_1}.\n\\end{theorem}","sha256_utf8":"70b182ea8e688767f26ca96575f83cbe3ea7e0d9cb4954670beade94aaf1e40a"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\\label{lem1} Given an approximate solution $\\hat{x}^k$ to \\eqref{subproblem1}, there exists a vector $e_k\\in\\R^n$ such that\n\\begin{equation}\\label{ek}\ne_k\\in\\nabla f(x^k)+H_k(\\hat{x}^k-x^k)+\\partial g(\\hat{x}^k-e_k)\\;\\mbox{ and }\\;\\|e_k\\|\\le\\nu\\min\\big\\{\\|\\mathcal{G}(x^k)\\|,\\|\\mathcal{G}(x^k)\\|^{1+\\varrho}\\big\\}.\n\\end{equation}\n\\end{lemma}\n\n\\begin{proof}\nLet $e_k:=r_k(\\hat{x}^k)=\\hat{x}^k-\\mathrm{Prox}_{g}(\\hat{x}^k-\\nabla f(x^k)-H_k(\\hat{x}^k-x^k))$. Then we have\n\\[\ne_k\\in\\nabla f(x^k)+H_k(\\hat{x}^k-x^k)+\\partial g(\\hat{x}^k-e_k),\n\\]\nwhich follow from \\eqref{prox-map}.\nUsing finally the inexact conditions \\eqref{inexact_condition_1} for $\\hat x^k$, we verify the claim of the lemma.\n\\end{proof}\n\n\\begin{lemma}\\label{decrease} Let $t_k$ be chosen by the backtracking line search in Step~{\\rm 5} of Algorithm~{\\rm\\ref{Newton}} at iteration $k$. Then we have the step size estimate\n\\begin{equation}\\label{tk}\nt_k \\ge \\frac{\\gamma(1-\\theta)\\alpha_k}{L_1}\n\\end{equation}\nwith the cost function decrease satisfying\n\\begin{equation}\\label{decrease_eq}\nF(x^{k+1})-F(x^k)\\le-\\frac{\\gamma\\theta(1-\\theta)}{2L_1}\\left(\\frac{(1-\\nu)\\alpha_k}{1+M+\\alpha_k}\\right)^2\\|\\mathcal{G}(x^k)\\|^2.\n\\end{equation}\n\\end{lemma}\n\n\\begin{proof} Since $\\hat{x}^k$ is an inexact solution to \\eqref{subproblem} obeying the conditions in \\eqref{inexact_condition_1}, it follows that\n\\[\n0\\ge q_k(\\hat{x}^k)-q_k(x^k)=l_k(\\hat{x}^k)-l_k(x^k)+\\frac{1}{2}(\\hat{x}^k-x^k)^TH_k(\\hat{x}^k-x^k),\n\\]\nwhere $l_k$ is the linear part of $q_k$ defined in \\eqref{lin}. This yields\n\\begin{equation}\\label{decrease_eq1}\nl_k(x^k)-l_k(\\hat{x}^k)\\ge\\frac{1}{2}(\\hat{x}^k-x^k)^TH_k(\\hat{x}^k-x^k)\\ge\\frac{1}{2}\\alpha_k\\|\\hat{x}^k-x^k\\|^2.\n\\end{equation}\nBy $\\mathcal{G}(x^k)=x^k-\\mathrm{Prox}_{g}(x^k-\\nabla f(x^k))$ we deduce from the stationary and subdifferential sum rules that\n\\[\n\\mathcal{G}(x^k)\\in\\nabla f(x^k)+\\partial g\\big(x^k-\\mathcal{G}(x^k)\\big).\n\\]\nFurthermore, Lemma~\\ref{lem1} gives us the condition $e_k\\in\\nabla f(x^k)+H_k(\\hat{x}^k-x^k)+\\partial g(\\hat{x}^k-e_k)$ for $\\hat x^k$ with $e_k$ satisfying the estimate\n$\\|e_k\\|\\le\\nu\\|\\mathcal{G}(x^k)\\|$. The monotonicity of the subgradient mapping $\\partial g$ ensures that\n\\[\n\\big\\langle \\mathcal{G}(x^k)+H_k(\\hat{x}^k-x^k)-e_k,x^k-\\mathcal{G}(x^k)-\\hat{x}^k+e_k\\big\\rangle\\ge 0,\n\\]\nwhich therefore leads us to the inequality\n\\begin{equation*}\n\\begin{aligned}\n\\| \\mathcal{G}(x^k) - e_k\\|^2 & \\le \\|\\mathcal{G}(x^k)\\|^2 -2\\big\\langle e_k,\\mathcal{G}(x^k)\\big\\rangle +\\|e_k\\|^2+(\\hat{x}^k-x^k)^TH_k(\\hat{x}^k-x^k) \\\\\n&\\le\\big\\langle \\mathcal{G}(x^k) - e_k,x^k-\\hat{x}^k+H_k(x^k-\\hat{x}^k)\\big\\rangle \\\\\n&\\le\\big\\| \\mathcal{G}(x^k) - e_k\\big\\| \\cdot \\big\\| x^k-\\hat{x}^k+H_k(x^k-\\hat{x}^k)\\big\\|.\n\\end{aligned}\n\\end{equation*}\nUsing again the condition $\\|e_k\\|\\le\\nu\\|\\mathcal{G}(x^k)\\|$ together with $\\|B_k\\|\\le M$ from \\eqref{assum_B_bounded} results in\n\\begin{equation*}\n\\|\\mathcal{G}(x^k)\\| \\le \\| \\mathcal{G}(x^k) - e_k\\| + \\|e_k\\| \\le(1+M+\\alpha_k)\\|\\hat{x}^k-x^k\\|+ \\nu\\|\\mathcal{G}(x^k)\\|.\n\\end{equation*}\nRemembering the choice of $\\nu\\in[0, 1)$, we estimate the prox-gradient mapping \\eqref{prox-gra} at the iterate $x^k$ by\n\\begin{equation}\\label{decrease_eq2}\n\\|\\mathcal{G}(x^k)\\|\\le\\frac{1+M+\\alpha_k}{1-\\nu}\\|\\hat{x}^k-x^k\\|.\n\\end{equation}\n\nNext let us show that the backtracking line search along the direction $d^k=\\hat{x}^k- x^k$ in Step~5 is well-defined and the proposed step size ensures a sufficient decrease in the cost function $F$.\nIt follows from the Lipschitz continuity of $\\nabla f$ that\n\\[\nf(x^k+\\tau d^k)\\le f(x^k)+\\tau\\nabla f(x^k)^Td^k+\\frac{L_1}{2}\\tau^2\\|d^k\\|^2\\;\\mbox{ for any }\\;\\tau\\ge 0,\n\\]\nand thus we deduce from the definition of $l_k$ in \\eqref{lin} that\n\\[\nF(x^k)-F(x^k+\\tau d^k)\\ge l_k(x^k)-l_k(x^k+\\tau d^k)-\\frac{L_1}{2}\\tau^2\\|d^k\\|^2.\n\\]\nThis implies by the convexity of $g$ that\n\\[\nl_k(x^k)-l_k(x^k+\\tau d^k)\\ge\\tau\\big(l_k(x^k)-l_k(x^k+d^k)\\big).\n\\]\nCombining the latter with \\eqref{decrease_eq1} and using the choice of $\\theta\\in(0,1)$ yield the relationships\n\\begin{equation}\\label{decrease_eq3}\n\\begin{aligned}\n&F(x^k)-F(x^k+\\tau d^k)-\\frac{\\theta\\alpha_k\\tau}{2}\\|d^k\\|^2\\\\\n\\ge~&l_k(x^k)-l_k(x^k+\\tau d^k)-\\frac{L_1}{2}\\tau^2\\|d^k\\|^2-\\frac{\\theta\\alpha_k\\tau}{2}\\|d^k\\|^2\\\\\n\\ge~&\\tau\\big(l_k(x^k)-l_k(x^k+d^k)\\big)-\\frac{L_1}{2}\\tau^2\\|d^k\\|^2-\\frac{\\theta\\alpha_k\\tau}{2}\\|d^k\\|^2\\\\\n\\ge~&(1-\\theta)\\tau\\frac{\\alpha_k}{2}\\|d^k\\|^2-\\frac{L_1}{2}\\tau^2\\|d^k\\|^2\\\\\n=~&\\frac{\\tau}{2}\\|d^k\\|^2\\big((1-\\theta)\\alpha_k-L_1\\tau\\big).\n\\end{aligned}\n\\end{equation}\nThis tells us that the backtracking line search criterion \\eqref{line_search} fulfills when $0<\\tau\\le\\frac{(1-\\theta)\\alpha_k}{L_1}$, and thus the step size $t_k$ satisfies the claimed condition\n\\eqref{tk}. Substituting now $\\tau:=t_k\\ge\\frac{\\gamma(1-\\theta)\\alpha_k}{L_1}$ into \\eqref{decrease_eq3} and employing the estimate of $\\|\\mathcal{G}(x^k)\\|^2$ from \\eqref{decrease_eq2}, we arrive at the\ninequalities\n\\[\n\\begin{aligned}\nF(x^k)-F(x^k+t_k d^k)&\\ge\\frac{\\theta\\alpha_kt_k}{2}\\|d^k\\|^2\\\\\n&\\ge\\frac{\\gamma\\theta(1-\\theta)\\alpha_k^2}{2L_1}\\left(\\frac{1-\\nu}{1+M+\\alpha_k}\\right)^2\\|\\mathcal{G}(x^k)\\|^2,\n\\end{aligned}\n\\]\nwhich verify the decreasing condition \\eqref{decrease_eq} and thus completes the proof of the lemma.\n\\end{proof}\n\n\\begin{proof}\nFirst we consider the case where the set $K_0$ is infinite. We can reorganize $K_0$ in such a way that $0=k_0<k_1<k_2<\\ldots$. It follows from Step~4 of Algorithm~\\ref{Newton} that the estimate\n\\[\n\\|\\mathcal{G}(x^{k_{\\ell+1}})\\|\\le\\sigma\\|\\mathcal{G}(x^{k_{\\ell}})\\|\\;\\mbox{ whenever }\\;\\ell=0,1,\\ldots\n\\]\nholds with the chosen number $\\sigma\\in(0,1)$ in the algorithm, and therefore we get\n\\revise{\\begin{equation*}\n0\\le \\liminf_{k\\rightarrow\\infty}\\|\\mathcal{G}(x^k)\\| \\le \\limsup\\limits_{\\ell\\rightarrow\\infty}\\|\\mathcal{G}(x^{k_{\\ell}})\\|\\le\\lim\\limits_{\\ell\\rightarrow\\infty}\\sigma^{\\ell}\\|\\mathcal{G}(x^{k_{0}})\\|=0,\n\\end{equation*}}%\nwhich clearly yields \\eqref{glob}. The continuity of $\\mathcal{G}(\\cdot)$ ensures that $\\|\\mathcal{G}(\\bar{x})\\|=0$ for a limiting point $\\bar{x}$ of the sequence $\\{x^k\\}_{k\\in K_0}$, and thus $\\bar{x}\n\\in\\mathcal{X}^*$.\nConsider now any limiting point $\\ox$ of the entire sequence of iterates $\\{x^k\\}_{k\\in K}$. If there exists $\\bar{k}$ such that $k\\in K_0$ for all $k\\ge\\bar{k}$, it is easy to see that $\\ox$ is a global\nminimizer of \\eqref{prob_ori_1}. Otherwise, for any $k\\notin K_0$, denote by $k_{\\ell}\\in K_0$ the largest number satisfying $k_{\\ell}<k$, and hence we get the following estimate from Step~5:\n\\[\n F^*\\le F(x^{k})\\le F(x^{k-1})\\le\\ldots\\le F(x^{k_\\ell}).\n\\]\nWhen the sequence $\\{x^k\\}_{k\\in K_0}$ is bounded, since any limiting point of $\\{x^k\\}_{k\\in K_0}$ is a global minimizer of \\eqref{prob_ori_1} as already shown, it follows that\n$\\lim\\limits_{\\ell\\rightarrow\\infty}F(x^{k_\\ell})=F^*$. This readily verifies by the constructions above that $\\lim\\limits_{k\\rightarrow\\infty}F(x^{k})=F^*$, and thus any limiting point\nof $\\{x^k\\}_{k\\in K}$ provides a global minimum to \\eqref{prob_ori_1}.\n\nNext we consider the case where $K_0$ is finite and denote $\\bar{k}:=\\max_{k\\in K_0}k$. It follows from Lemma~\\ref{decrease} that for any $k>\\bar{k}$ we get\n\\[\nF(x^{k+1})-F(x^k)\\le-\\frac{\\gamma\\theta(1-\\theta)}{2L_1}\\left(\\frac{(1-\\nu)\\alpha_k}{1+M+\\alpha_k}\\right)^2\\|\\mathcal{G}(x^k)\\|^2,\n\\]\nwhich therefore tells us that\n\\[\n\\sum_{k=\\bar{k}}^{\\infty}\\frac{\\gamma\\theta(1-\\theta)}{2L_1}\\left(\\frac{(1-\\nu)\\alpha_k}{1+M+\\alpha_k}\\right)^2\\|\\mathcal{G}(x^k)\\|^2\\le F(x^{\\bar{k}})-F^*\\le 0.\n\\]\nThe latter implies in turn that\n\\[\n\\lim_{k\\rightarrow \\infty}\\frac{\\gamma\\theta(1-\\theta)}{2L_1}\\left(\\frac{(1-\\nu)\\alpha_k}{1+M+\\alpha_k}\\right)^2\\|\\mathcal{G}(x^k)\\|^2=0.\n\\]\nRemembering the choice of \\revise{$\\alpha_k=\\min \\left\\{ \\bar{\\alpha}, c\\|\\mathcal{G}(x^k)\\|^{\\rho}\\right\\}$} with $\\bar{\\alpha}, c, \\rho>0$ ensures that\n\\[\n\\lim_{k\\rightarrow\\infty}\\|\\mathcal{G}(x^k)\\|=0,\n\\]\nhence \\eqref{glob} holds. This readily verifies by the continuity of $\\mathcal{G}(\\cdot)$ that any limiting point of $\\{x^k\\}_{k\\in K}$ provides a global minimum to \\eqref{prob_ori_1}.\n\\end{proof}","units":[{"start":40908,"end":41262,"start_line":604,"end_line":608,"raw_tex":"\\begin{lemma}\\label{lem1} Given an approximate solution $\\hat{x}^k$ to \\eqref{subproblem1}, there exists a vector $e_k\\in\\R^n$ such that\n\\begin{equation}\\label{ek}\ne_k\\in\\nabla f(x^k)+H_k(\\hat{x}^k-x^k)+\\partial g(\\hat{x}^k-e_k)\\;\\mbox{ and }\\;\\|e_k\\|\\le\\nu\\min\\big\\{\\|\\mathcal{G}(x^k)\\|,\\|\\mathcal{G}(x^k)\\|^{1+\\varrho}\\big\\}.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"27a9a97b4ba2d3830b6dd37d571a17da55a34ace66d634ba245884fa80a8fc8d"},{"start":41271,"end":41631,"start_line":609,"end_line":616,"raw_tex":"\\begin{proof}\nLet $e_k:=r_k(\\hat{x}^k)=\\hat{x}^k-\\mathrm{Prox}_{g}(\\hat{x}^k-\\nabla f(x^k)-H_k(\\hat{x}^k-x^k))$. Then we have\n\\[\ne_k\\in\\nabla f(x^k)+H_k(\\hat{x}^k-x^k)+\\partial g(\\hat{x}^k-e_k),\n\\]\nwhich follow from \\eqref{prox-map}.\nUsing finally the inexact conditions \\eqref{inexact_condition_1} for $\\hat x^k$, we verify the claim of the lemma.\n\\end{proof}","sha256_utf8":"f164b5d42b6c82f458f96bfaf58ae65e323772af7a6c4dedc45fe04160fbbe07"},{"start":41805,"end":42311,"start_line":620,"end_line":628,"raw_tex":"\\begin{lemma}\\label{decrease} Let $t_k$ be chosen by the backtracking line search in Step~{\\rm 5} of Algorithm~{\\rm\\ref{Newton}} at iteration $k$. Then we have the step size estimate\n\\begin{equation}\\label{tk}\nt_k \\ge \\frac{\\gamma(1-\\theta)\\alpha_k}{L_1}\n\\end{equation}\nwith the cost function decrease satisfying\n\\begin{equation}\\label{decrease_eq}\nF(x^{k+1})-F(x^k)\\le-\\frac{\\gamma\\theta(1-\\theta)}{2L_1}\\left(\\frac{(1-\\nu)\\alpha_k}{1+M+\\alpha_k}\\right)^2\\|\\mathcal{G}(x^k)\\|^2.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"4ece636a2d1f49766f213a450278110f258c813446d18934d6a36a706aa8af23"},{"start":42312,"end":46375,"start_line":629,"end_line":696,"raw_tex":"\\begin{proof} Since $\\hat{x}^k$ is an inexact solution to \\eqref{subproblem} obeying the conditions in \\eqref{inexact_condition_1}, it follows that\n\\[\n0\\ge q_k(\\hat{x}^k)-q_k(x^k)=l_k(\\hat{x}^k)-l_k(x^k)+\\frac{1}{2}(\\hat{x}^k-x^k)^TH_k(\\hat{x}^k-x^k),\n\\]\nwhere $l_k$ is the linear part of $q_k$ defined in \\eqref{lin}. This yields\n\\begin{equation}\\label{decrease_eq1}\nl_k(x^k)-l_k(\\hat{x}^k)\\ge\\frac{1}{2}(\\hat{x}^k-x^k)^TH_k(\\hat{x}^k-x^k)\\ge\\frac{1}{2}\\alpha_k\\|\\hat{x}^k-x^k\\|^2.\n\\end{equation}\nBy $\\mathcal{G}(x^k)=x^k-\\mathrm{Prox}_{g}(x^k-\\nabla f(x^k))$ we deduce from the stationary and subdifferential sum rules that\n\\[\n\\mathcal{G}(x^k)\\in\\nabla f(x^k)+\\partial g\\big(x^k-\\mathcal{G}(x^k)\\big).\n\\]\nFurthermore, Lemma~\\ref{lem1} gives us the condition $e_k\\in\\nabla f(x^k)+H_k(\\hat{x}^k-x^k)+\\partial g(\\hat{x}^k-e_k)$ for $\\hat x^k$ with $e_k$ satisfying the estimate\n$\\|e_k\\|\\le\\nu\\|\\mathcal{G}(x^k)\\|$. The monotonicity of the subgradient mapping $\\partial g$ ensures that\n\\[\n\\big\\langle \\mathcal{G}(x^k)+H_k(\\hat{x}^k-x^k)-e_k,x^k-\\mathcal{G}(x^k)-\\hat{x}^k+e_k\\big\\rangle\\ge 0,\n\\]\nwhich therefore leads us to the inequality\n\\begin{equation*}\n\\begin{aligned}\n\\| \\mathcal{G}(x^k) - e_k\\|^2 & \\le \\|\\mathcal{G}(x^k)\\|^2 -2\\big\\langle e_k,\\mathcal{G}(x^k)\\big\\rangle +\\|e_k\\|^2+(\\hat{x}^k-x^k)^TH_k(\\hat{x}^k-x^k) \\\\\n&\\le\\big\\langle \\mathcal{G}(x^k) - e_k,x^k-\\hat{x}^k+H_k(x^k-\\hat{x}^k)\\big\\rangle \\\\\n&\\le\\big\\| \\mathcal{G}(x^k) - e_k\\big\\| \\cdot \\big\\| x^k-\\hat{x}^k+H_k(x^k-\\hat{x}^k)\\big\\|.\n\\end{aligned}\n\\end{equation*}\nUsing again the condition $\\|e_k\\|\\le\\nu\\|\\mathcal{G}(x^k)\\|$ together with $\\|B_k\\|\\le M$ from \\eqref{assum_B_bounded} results in\n\\begin{equation*}\n\\|\\mathcal{G}(x^k)\\| \\le \\| \\mathcal{G}(x^k) - e_k\\| + \\|e_k\\| \\le(1+M+\\alpha_k)\\|\\hat{x}^k-x^k\\|+ \\nu\\|\\mathcal{G}(x^k)\\|.\n\\end{equation*}\nRemembering the choice of $\\nu\\in[0, 1)$, we estimate the prox-gradient mapping \\eqref{prox-gra} at the iterate $x^k$ by\n\\begin{equation}\\label{decrease_eq2}\n\\|\\mathcal{G}(x^k)\\|\\le\\frac{1+M+\\alpha_k}{1-\\nu}\\|\\hat{x}^k-x^k\\|.\n\\end{equation}\n\nNext let us show that the backtracking line search along the direction $d^k=\\hat{x}^k- x^k$ in Step~5 is well-defined and the proposed step size ensures a sufficient decrease in the cost function $F$.\nIt follows from the Lipschitz continuity of $\\nabla f$ that\n\\[\nf(x^k+\\tau d^k)\\le f(x^k)+\\tau\\nabla f(x^k)^Td^k+\\frac{L_1}{2}\\tau^2\\|d^k\\|^2\\;\\mbox{ for any }\\;\\tau\\ge 0,\n\\]\nand thus we deduce from the definition of $l_k$ in \\eqref{lin} that\n\\[\nF(x^k)-F(x^k+\\tau d^k)\\ge l_k(x^k)-l_k(x^k+\\tau d^k)-\\frac{L_1}{2}\\tau^2\\|d^k\\|^2.\n\\]\nThis implies by the convexity of $g$ that\n\\[\nl_k(x^k)-l_k(x^k+\\tau d^k)\\ge\\tau\\big(l_k(x^k)-l_k(x^k+d^k)\\big).\n\\]\nCombining the latter with \\eqref{decrease_eq1} and using the choice of $\\theta\\in(0,1)$ yield the relationships\n\\begin{equation}\\label{decrease_eq3}\n\\begin{aligned}\n&F(x^k)-F(x^k+\\tau d^k)-\\frac{\\theta\\alpha_k\\tau}{2}\\|d^k\\|^2\\\\\n\\ge~&l_k(x^k)-l_k(x^k+\\tau d^k)-\\frac{L_1}{2}\\tau^2\\|d^k\\|^2-\\frac{\\theta\\alpha_k\\tau}{2}\\|d^k\\|^2\\\\\n\\ge~&\\tau\\big(l_k(x^k)-l_k(x^k+d^k)\\big)-\\frac{L_1}{2}\\tau^2\\|d^k\\|^2-\\frac{\\theta\\alpha_k\\tau}{2}\\|d^k\\|^2\\\\\n\\ge~&(1-\\theta)\\tau\\frac{\\alpha_k}{2}\\|d^k\\|^2-\\frac{L_1}{2}\\tau^2\\|d^k\\|^2\\\\\n=~&\\frac{\\tau}{2}\\|d^k\\|^2\\big((1-\\theta)\\alpha_k-L_1\\tau\\big).\n\\end{aligned}\n\\end{equation}\nThis tells us that the backtracking line search criterion \\eqref{line_search} fulfills when $0<\\tau\\le\\frac{(1-\\theta)\\alpha_k}{L_1}$, and thus the step size $t_k$ satisfies the claimed condition\n\\eqref{tk}. Substituting now $\\tau:=t_k\\ge\\frac{\\gamma(1-\\theta)\\alpha_k}{L_1}$ into \\eqref{decrease_eq3} and employing the estimate of $\\|\\mathcal{G}(x^k)\\|^2$ from \\eqref{decrease_eq2}, we arrive at the\ninequalities\n\\[\n\\begin{aligned}\nF(x^k)-F(x^k+t_k d^k)&\\ge\\frac{\\theta\\alpha_kt_k}{2}\\|d^k\\|^2\\\\\n&\\ge\\frac{\\gamma\\theta(1-\\theta)\\alpha_k^2}{2L_1}\\left(\\frac{1-\\nu}{1+M+\\alpha_k}\\right)^2\\|\\mathcal{G}(x^k)\\|^2,\n\\end{aligned}\n\\]\nwhich verify the decreasing condition \\eqref{decrease_eq} and thus completes the proof of the lemma.\n\\end{proof}","sha256_utf8":"3089c13e1469c1a6ad1d935d142fb4700013335ec1a1e6bbd6b4728f4b80fe5a"},{"start":47285,"end":50131,"start_line":712,"end_line":749,"raw_tex":"\\begin{proof}\nFirst we consider the case where the set $K_0$ is infinite. We can reorganize $K_0$ in such a way that $0=k_0<k_1<k_2<\\ldots$. It follows from Step~4 of Algorithm~\\ref{Newton} that the estimate\n\\[\n\\|\\mathcal{G}(x^{k_{\\ell+1}})\\|\\le\\sigma\\|\\mathcal{G}(x^{k_{\\ell}})\\|\\;\\mbox{ whenever }\\;\\ell=0,1,\\ldots\n\\]\nholds with the chosen number $\\sigma\\in(0,1)$ in the algorithm, and therefore we get\n\\revise{\\begin{equation*}\n0\\le \\liminf_{k\\rightarrow\\infty}\\|\\mathcal{G}(x^k)\\| \\le \\limsup\\limits_{\\ell\\rightarrow\\infty}\\|\\mathcal{G}(x^{k_{\\ell}})\\|\\le\\lim\\limits_{\\ell\\rightarrow\\infty}\\sigma^{\\ell}\\|\\mathcal{G}(x^{k_{0}})\\|=0,\n\\end{equation*}}%\nwhich clearly yields \\eqref{glob}. The continuity of $\\mathcal{G}(\\cdot)$ ensures that $\\|\\mathcal{G}(\\bar{x})\\|=0$ for a limiting point $\\bar{x}$ of the sequence $\\{x^k\\}_{k\\in K_0}$, and thus $\\bar{x}\n\\in\\mathcal{X}^*$.\nConsider now any limiting point $\\ox$ of the entire sequence of iterates $\\{x^k\\}_{k\\in K}$. If there exists $\\bar{k}$ such that $k\\in K_0$ for all $k\\ge\\bar{k}$, it is easy to see that $\\ox$ is a global\nminimizer of \\eqref{prob_ori_1}. Otherwise, for any $k\\notin K_0$, denote by $k_{\\ell}\\in K_0$ the largest number satisfying $k_{\\ell}<k$, and hence we get the following estimate from Step~5:\n\\[\n F^*\\le F(x^{k})\\le F(x^{k-1})\\le\\ldots\\le F(x^{k_\\ell}).\n\\]\nWhen the sequence $\\{x^k\\}_{k\\in K_0}$ is bounded, since any limiting point of $\\{x^k\\}_{k\\in K_0}$ is a global minimizer of \\eqref{prob_ori_1} as already shown, it follows that\n$\\lim\\limits_{\\ell\\rightarrow\\infty}F(x^{k_\\ell})=F^*$. This readily verifies by the constructions above that $\\lim\\limits_{k\\rightarrow\\infty}F(x^{k})=F^*$, and thus any limiting point\nof $\\{x^k\\}_{k\\in K}$ provides a global minimum to \\eqref{prob_ori_1}.\n\nNext we consider the case where $K_0$ is finite and denote $\\bar{k}:=\\max_{k\\in K_0}k$. It follows from Lemma~\\ref{decrease} that for any $k>\\bar{k}$ we get\n\\[\nF(x^{k+1})-F(x^k)\\le-\\frac{\\gamma\\theta(1-\\theta)}{2L_1}\\left(\\frac{(1-\\nu)\\alpha_k}{1+M+\\alpha_k}\\right)^2\\|\\mathcal{G}(x^k)\\|^2,\n\\]\nwhich therefore tells us that\n\\[\n\\sum_{k=\\bar{k}}^{\\infty}\\frac{\\gamma\\theta(1-\\theta)}{2L_1}\\left(\\frac{(1-\\nu)\\alpha_k}{1+M+\\alpha_k}\\right)^2\\|\\mathcal{G}(x^k)\\|^2\\le F(x^{\\bar{k}})-F^*\\le 0.\n\\]\nThe latter implies in turn that\n\\[\n\\lim_{k\\rightarrow \\infty}\\frac{\\gamma\\theta(1-\\theta)}{2L_1}\\left(\\frac{(1-\\nu)\\alpha_k}{1+M+\\alpha_k}\\right)^2\\|\\mathcal{G}(x^k)\\|^2=0.\n\\]\nRemembering the choice of \\revise{$\\alpha_k=\\min \\left\\{ \\bar{\\alpha}, c\\|\\mathcal{G}(x^k)\\|^{\\rho}\\right\\}$} with $\\bar{\\alpha}, c, \\rho>0$ ensures that\n\\[\n\\lim_{k\\rightarrow\\infty}\\|\\mathcal{G}(x^k)\\|=0,\n\\]\nhence \\eqref{glob} holds. This readily verifies by the continuity of $\\mathcal{G}(\\cdot)$ that any limiting point of $\\{x^k\\}_{k\\in K}$ provides a global minimum to \\eqref{prob_ori_1}.\n\\end{proof}","sha256_utf8":"e4ee1d72859b8ff67d568694d638d15d3bb4a45fbc94c4deeca3fef087c866d0"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"复合凸优化问题","source_units":[{"start":6005,"end":7033,"start_line":150,"end_line":165,"raw_tex":"In this paper we consider a class of optimization problems of the\nfollowing type: \\begin{equation}\\label{prob_ori_1}\n\\min_{x\\in\\mathbb{R}^n}~F(x):=f(x)+g(x), \\end{equation} where both\nfunctions $f,g\\colon\\R^n\\to\\oR:=(-\\infty,\\infty]$ are proper,\nconvex, and lower semicontinuous (l.s.c.), while being structurally\ndifferent from each other. Namely, $f$ is assumed to be twice\ncontinuously differentiable with the Lipschitz continuous gradient\n$\\nabla f$ on its domain. On the other hand, $g$ is merely\ncontinuous on its domain; see Assumption~\\ref{assumptions_f} for the\nprecise formulations. It has been well recognized that model\n\\eqref{prob_ori_1}, known as a {\\em composite convex optimization\nproblem}, frequently appears in a variety of applications including,\ne.g., machine learning, signal processing, and statistics, where $f$\nis a {\\em loss function} and $g$ is a {\\em regularizer}; we keep\nthis terminology here. Note that problem \\eqref{prob_ori_1} contains\nin fact implicit constraints written as $x\\in\\O:=\\dom g$.\n","sha256_utf8":"2bfecaa3e2439243be0f0c31c6319efe50f0e42b73e7b2690ac4603076d1a282"}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","title":"函数与解集基本假设","source_units":[{"start":14383,"end":15261,"start_line":267,"end_line":279,"raw_tex":"\\begin{assumption}\\label{assumptions_f} Impose the following properties of the loss function and the regularizer in \\eqref{prob_ori_1}:\n\\begin{itemize}\n\\item[{\\bf(i)}] Both functions $f,g:{\\R^n}\\rightarrow (-\\infty,\\infty]$ are proper, l.s.c., and convex.\n\n\\item[{\\bf(ii)}] The \\revise{effective} domain of the loss function $\\dom f:=\\{x~|~f(x)<\\infty\\}$ is open, and $f(x)$ is twice continuously differentiable on a closed set $\\Omega\\supset \\dom f$.\n\n\\item[{\\bf(iii)}] The regularizer $g(x)$ is continuous on its domain and $\\varnothing \\ne\\dom g \\subset\\dom f$.\n\n\\item[{\\bf(iv)}] The gradient $\\nabla f(x)$ is Lipschitz continuous on a closed set $\\Omega$ from {\\rm(ii)} with Lipschitz constant $L_1>0$.\n\n\\item[{\\bf(v)}] Problem \\eqref{prob_ori_1} has a nonempty solution set ${\\cal{X}^*}:=\\arg\\min_{x\\in\\R^n}F(x)$ with the optimal value $F^*$.\n\\end{itemize}\n\\end{assumption}","sha256_utf8":"ec355afa1c82b2c19d9c77d58e51d6b3f2f3fa863fb1f531032291f595d65028"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"线性复合模型","source_units":[{"start":7457,"end":7622,"start_line":174,"end_line":176,"raw_tex":"\\begin{equation}\\label{lin} \\min_{x\\in\\mathbb{R}^n}~l_k(x) +\n\\frac{1}{2t}\\|x-x_k\\|^2 \\;\\; \\text{with} \\;\\; l_k(x):=f(x^k)+\\nabla\nf(x^k)^T(x-x^k)+g(x), \\end{equation}","sha256_utf8":"4a27be743f633e30244ca2ee528a031abb9b42a3200ec02a3353d671ca83c64a"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"二次子问题模型","source_units":[{"start":8521,"end":8669,"start_line":191,"end_line":193,"raw_tex":"\\begin{equation}\\label{subproblem}\n\\min_{x\\in\\mathbb{R}^n}~q_k(x):=f(x^k)+\\nabla\nf(x^k)^T(x-x^k)+\\frac{1}{2}(x-x^k)^TH_k(x-x^k)+g(x), \\end{equation}","sha256_utf8":"aa57a0728448c3bfbdc6413a8a6cfe380c3bd53162b2cf12b65040a23f405df7"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"近端映射","source_units":[{"start":23094,"end":23244,"start_line":359,"end_line":361,"raw_tex":"\\begin{equation}\\label{prox-map}\n\\hbox{Prox}_{g}(u):={\\rm argmin}\\Big\\{g(x)+\\frac{1}{2}\\|x-u\\|^2\\;\\Big|\\;x\\in\\R^n\\Big\\},\\quad u\\in\\R^n,\n\\end{equation}","sha256_utf8":"b38b9b586b705472f2e5898b35d76ec78a8810186d30520f4361f618f7eff3ae"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"近端梯度残差映射","source_units":[{"start":24621,"end":24743,"start_line":373,"end_line":375,"raw_tex":"\\begin{equation}\\label{prox-gra}\n\\mathcal{G}(x):=x-\\mathrm{Prox}_{g}\\big(x-\\nabla f(x)\\big),\\quad x\\in\\R^n,\n\\end{equation}","sha256_utf8":"6a7e540055408a592f3b6cfb6ddfaebed37360d6b3810441d8d596b70e5a3c7e"}]},{"id":"S007","semantic_type":"assumption","kind":"assumption","title":"半正定矩阵及有界性","source_units":[{"start":34933,"end":35507,"start_line":544,"end_line":549,"raw_tex":"Given a current iterate $x^k$ for each $k=0,1,\\ldots$, we select a {\\em positive semidefinite matrix} $B_k$ as an arbitrary approximation of the Hessian $\\nabla^2 f(x^k)$ satisfying the {\\em standing\nboundedness assumption}:\n\\begin{equation}\\label{assum_B_bounded}\n\\mbox{ there exists }\\;M\\ge 0\\;\\mbox{ such that }\\;\\|B_k\\|\\le M\\;\\mbox{ whenever }\\;k=0,1,\\ldots.\n\\end{equation}\nIf the gradient mapping $\\nabla f$ is uniformly Lipschitz continuous along the sequence of iterates with constant $L_1$, then \\eqref{assum_B_bounded} holds for $B_k=\\nabla^2 f(x^k)$ with $M=L_1$.\n","sha256_utf8":"60018652f3c88b336118b470b86c02791bda01f3a636c45fca4828addc7724b4"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"正则化矩阵与子问题残差","source_units":[{"start":35507,"end":36932,"start_line":550,"end_line":565,"raw_tex":"In the general case of $B_k$, pick any constants $c>0$ and $\\rho\\in(0,1]$ and, using the prox-regular mapping \\eqref{prox-gra}, consider the positive number $\\alpha_k:=c\\|\\mathcal{G}(x^k)\\|^{\\rho}$\nand define the {\\em quasi-Newton approximation} of the Hessian of $f$ at $x^k$ by\n\\begin{equation}\\label{alg-app}\nH_k:=B_k+\\alpha_kI\\;\\mbox{ for all }\\;k=0,1,\\ldots,\n\\end{equation}\nwhich is a positive definite matrix. Then similarly to \\cite{byrd2016inexact}, but with the different approximation \\eqref{alg-app}, denote\n\\begin{equation}\\label{opt-mes}\nr_k(x):=x-\\mathrm{Prox}_{g}\\left(x-\\nabla f(x^k)-H_k(x-x^k)\\right)\n\\end{equation}\nand select $\\hat{x}^k$ as an {\\em approximate minimizer} of the quadratic {\\em subproblem} for \\eqref{prob_ori_1} given by\n\\begin{equation}\\label{subproblem1}\n\\min_{x\\in\\mathbb{R}^n}~q_k(x):=f(x^k)+\\nabla f(x^k)^T(x-x^k)+\\frac{1}{2}(x-x^k)H_k(x-x^k)+g(x)\n\\end{equation}\nwith the {\\em residual} number $\\|r_k(\\hat x^k)\\|$ measuring the approximate optimality of $\\hat x^k$ in \\eqref{subproblem1}. Observing that $\\|r_k(\\hat{x}^k)\\|=0$ if and only if $\\hat{x}^k$ is an\n{\\em exact} solution to subproblem \\eqref{subproblem1}, we use the nonnegative number {\\color{black} $\\|r_k(\\hat{x}^k)\\|$}\nwith $r_k(x)$ taken from \\eqref{opt-mes} as the {\\em optimality measure} of $\\hat x^k$ in subproblem \\eqref{subproblem1}. Adapting the scheme of \\cite{byrd2016inexact} in our new setting, let us impose\n","sha256_utf8":"92bd9cc5360558b7c7a6bc85fb453d06f9d3f9355988584d5a469502f2557c00"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"非精确解准则与参数","source_units":[{"start":36932,"end":37766,"start_line":566,"end_line":570,"raw_tex":"the following two estimates as {\\em inexact conditions} for choosing $\\hat{x}^k$ as an approximate solution to subproblem \\eqref{subproblem1}:\n{\\color{black}\\begin{equation}\\label{inexact_condition_1}\n\\|r_k(\\hat{x}^k)\\|\\le\\eta_k\\|\\mathcal{G}(x^k)\\|\\;\\mbox{ and }\\;q_k(\\hat{x}^k)\\le q_k(x^k)\n\\end{equation}}%\nwith the parameter $\\eta_k:=\\nu\\min\\{1,\\|\\mathcal{G}(x^k)\\|^{\\varrho}\\}$ defined via \\eqref{opt-mes} and some numbers $\\nu\\in[0,1)$ and $\\varrho>0$. \\revise{Since any exact solution to subproblem \\eqref{subproblem1} always fulfills the two inexact conditions in \\eqref{inexact_condition_1}, a point $\\hat x^k$ satisfying \\eqref{inexact_condition_1} always exists. For many application problems, the mapping $\\mathrm{Prox}_{g}$ is easy to calculate, thus inexact conditions \\eqref{inexact_condition_1} can be readily verified.\n","sha256_utf8":"8c0787f2e7e6663977723e38b70e4649fa12118568864dbff967da09f4b47632"}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","title":"近端牛顿迭代与回溯","source_units":[{"start":38388,"end":39706,"start_line":574,"end_line":594,"raw_tex":"\\begin{algorithm}\n\\caption{Proximal Newton-type method}\\label{Newton}\n\\begin{algorithmic}[1]\n\\State Choose $x^0\\in\\mathbb{R}^n$, $0<\\theta,\\sigma,\\gamma<1$, $C>F(x^0)$, $\\revise{\\bar{\\alpha}}, c>0$, and $\\rho\\in(0,1]$.\n\\For{$k=0,1,\\ldots$}{\n\\begin{itemize}\n\\item[1.] Update the approximation of the Hessian matrix $B_k$.\n\\item[2.] Form the quadratic model \\eqref{subproblem} with $H_k:=B_k+\\alpha_kI$ and \\revise{$\\alpha_k:=\\min \\left\\{ \\bar{\\alpha}, c\\|\\mathcal{G}(x^k)\\|^{\\rho}\\right\\} $  }.\n\\item[3.] Obtain an inexact solution $\\hat{x}^k$ of \\eqref{subproblem} satisfying the conditions in \\eqref{inexact_condition_1}.\n\\item[4.] If $k=0$, let $\\vartheta_1:=\\mathcal{G}(x^0)$ and go to Step~5. For $k\\ge 1$, if $\\|\\mathcal{G}(\\hat{x}^k)\\|\\le\\sigma\\vartheta_k$ and $F(\\hat{x}^k)\\le C$, let $t_k:=1$, $\\vartheta_{k+1}:=\n\\|\\mathcal{G}(\\hat{x}^k)\\|$,\nand go to Step~6. Otherwise, let $\\vartheta_{k+1}:=\\vartheta_k$ and go to Step~5.\n\\item[5.] Perform a backtracking line search along the direction $d^k:=\\hat{x}^k-x^k$ by setting $t_k:=\\gamma^{m_k}$, where $m_k$ is the smallest nonnegative integer $m$ such that\n\\begin{equation}\\label{line_search}\nF(x^k+\\gamma^md^k)\\le F(x^k)-\\theta\\alpha_k\\gamma^m\\|d^k\\|^2.\n\\end{equation}\n\\item[6.] Set $x^{k+1}:=x^k+t_kd^k$.\n\\end{itemize}}\n\\EndFor\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"01e8b9f676483377385b71da4b97e8b00dad5222da42386e38109e901214d970"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"迭代及直接接受指标集","source_units":[{"start":46469,"end":46641,"start_line":699,"end_line":701,"raw_tex":"\\begin{equation}\\label{K}\nK:=\\big\\{0,1,\\ldots\\big\\}\\;\\mbox{ and }\\;K_0:=\\{0\\}\\cup\\big\\{k+1\\in K ~\\big|~\\text{Step~$5$ is not applied at iteration $k$}\\big\\}.\n\\end{equation}","sha256_utf8":"bae38c21497112117ff40593a0de12d3a03a560b219195c059b90fd599a697d2"}]},{"id":"S012","semantic_type":"theorem","kind":"theorem","title":"近端牛顿法全局收敛","source_units":[{"start":46643,"end":47284,"start_line":703,"end_line":711,"raw_tex":"\\begin{theorem}\\label{algorithm_convergence}\nLet $\\{x^k\\}$ be the sequence of iterates generated by Algorithm~{\\rm\\ref{Newton}} with an arbitrarily chosen starting point $x^0\\in\\R^n$ under the standing assumptions made. Then we have the residual\ncondition\n\\begin{equation}\\label{glob}\n\\liminf_{k\\rightarrow\\infty}\\|\\mathcal{G}(x^k)\\|=0\n\\end{equation}\nalong the prox-gradient mapping \\eqref{prox-gra}. Furthermore, the boundedness of $\\{x^k\\}$ yields the convergence to the optimal value $\\lim\\limits_{k\\rightarrow\\infty}F(x^{k})=F^*$ and ensures that any\nlimiting point of $\\{x^k\\}$ is a global minimizer in \\eqref{prob_ori_1}.\n\\end{theorem}","sha256_utf8":"70b182ea8e688767f26ca96575f83cbe3ea7e0d9cb4954670beade94aaf1e40a"}]},{"id":"S013","semantic_type":"proof","kind":"proof","title":"全局收敛完整证明","source_units":[{"start":47285,"end":50131,"start_line":712,"end_line":749,"raw_tex":"\\begin{proof}\nFirst we consider the case where the set $K_0$ is infinite. We can reorganize $K_0$ in such a way that $0=k_0<k_1<k_2<\\ldots$. It follows from Step~4 of Algorithm~\\ref{Newton} that the estimate\n\\[\n\\|\\mathcal{G}(x^{k_{\\ell+1}})\\|\\le\\sigma\\|\\mathcal{G}(x^{k_{\\ell}})\\|\\;\\mbox{ whenever }\\;\\ell=0,1,\\ldots\n\\]\nholds with the chosen number $\\sigma\\in(0,1)$ in the algorithm, and therefore we get\n\\revise{\\begin{equation*}\n0\\le \\liminf_{k\\rightarrow\\infty}\\|\\mathcal{G}(x^k)\\| \\le \\limsup\\limits_{\\ell\\rightarrow\\infty}\\|\\mathcal{G}(x^{k_{\\ell}})\\|\\le\\lim\\limits_{\\ell\\rightarrow\\infty}\\sigma^{\\ell}\\|\\mathcal{G}(x^{k_{0}})\\|=0,\n\\end{equation*}}%\nwhich clearly yields \\eqref{glob}. The continuity of $\\mathcal{G}(\\cdot)$ ensures that $\\|\\mathcal{G}(\\bar{x})\\|=0$ for a limiting point $\\bar{x}$ of the sequence $\\{x^k\\}_{k\\in K_0}$, and thus $\\bar{x}\n\\in\\mathcal{X}^*$.\nConsider now any limiting point $\\ox$ of the entire sequence of iterates $\\{x^k\\}_{k\\in K}$. If there exists $\\bar{k}$ such that $k\\in K_0$ for all $k\\ge\\bar{k}$, it is easy to see that $\\ox$ is a global\nminimizer of \\eqref{prob_ori_1}. Otherwise, for any $k\\notin K_0$, denote by $k_{\\ell}\\in K_0$ the largest number satisfying $k_{\\ell}<k$, and hence we get the following estimate from Step~5:\n\\[\n F^*\\le F(x^{k})\\le F(x^{k-1})\\le\\ldots\\le F(x^{k_\\ell}).\n\\]\nWhen the sequence $\\{x^k\\}_{k\\in K_0}$ is bounded, since any limiting point of $\\{x^k\\}_{k\\in K_0}$ is a global minimizer of \\eqref{prob_ori_1} as already shown, it follows that\n$\\lim\\limits_{\\ell\\rightarrow\\infty}F(x^{k_\\ell})=F^*$. This readily verifies by the constructions above that $\\lim\\limits_{k\\rightarrow\\infty}F(x^{k})=F^*$, and thus any limiting point\nof $\\{x^k\\}_{k\\in K}$ provides a global minimum to \\eqref{prob_ori_1}.\n\nNext we consider the case where $K_0$ is finite and denote $\\bar{k}:=\\max_{k\\in K_0}k$. It follows from Lemma~\\ref{decrease} that for any $k>\\bar{k}$ we get\n\\[\nF(x^{k+1})-F(x^k)\\le-\\frac{\\gamma\\theta(1-\\theta)}{2L_1}\\left(\\frac{(1-\\nu)\\alpha_k}{1+M+\\alpha_k}\\right)^2\\|\\mathcal{G}(x^k)\\|^2,\n\\]\nwhich therefore tells us that\n\\[\n\\sum_{k=\\bar{k}}^{\\infty}\\frac{\\gamma\\theta(1-\\theta)}{2L_1}\\left(\\frac{(1-\\nu)\\alpha_k}{1+M+\\alpha_k}\\right)^2\\|\\mathcal{G}(x^k)\\|^2\\le F(x^{\\bar{k}})-F^*\\le 0.\n\\]\nThe latter implies in turn that\n\\[\n\\lim_{k\\rightarrow \\infty}\\frac{\\gamma\\theta(1-\\theta)}{2L_1}\\left(\\frac{(1-\\nu)\\alpha_k}{1+M+\\alpha_k}\\right)^2\\|\\mathcal{G}(x^k)\\|^2=0.\n\\]\nRemembering the choice of \\revise{$\\alpha_k=\\min \\left\\{ \\bar{\\alpha}, c\\|\\mathcal{G}(x^k)\\|^{\\rho}\\right\\}$} with $\\bar{\\alpha}, c, \\rho>0$ ensures that\n\\[\n\\lim_{k\\rightarrow\\infty}\\|\\mathcal{G}(x^k)\\|=0,\n\\]\nhence \\eqref{glob} holds. This readily verifies by the continuity of $\\mathcal{G}(\\cdot)$ that any limiting point of $\\{x^k\\}_{k\\in K}$ provides a global minimum to \\eqref{prob_ori_1}.\n\\end{proof}","sha256_utf8":"e4ee1d72859b8ff67d568694d638d15d3bb4a45fbc94c4deeca3fef087c866d0"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"无限直接接受时残差趋零","source_units":[{"start":47285,"end":48162,"start_line":712,"end_line":722,"raw_tex":"\\begin{proof}\nFirst we consider the case where the set $K_0$ is infinite. We can reorganize $K_0$ in such a way that $0=k_0<k_1<k_2<\\ldots$. It follows from Step~4 of Algorithm~\\ref{Newton} that the estimate\n\\[\n\\|\\mathcal{G}(x^{k_{\\ell+1}})\\|\\le\\sigma\\|\\mathcal{G}(x^{k_{\\ell}})\\|\\;\\mbox{ whenever }\\;\\ell=0,1,\\ldots\n\\]\nholds with the chosen number $\\sigma\\in(0,1)$ in the algorithm, and therefore we get\n\\revise{\\begin{equation*}\n0\\le \\liminf_{k\\rightarrow\\infty}\\|\\mathcal{G}(x^k)\\| \\le \\limsup\\limits_{\\ell\\rightarrow\\infty}\\|\\mathcal{G}(x^{k_{\\ell}})\\|\\le\\lim\\limits_{\\ell\\rightarrow\\infty}\\sigma^{\\ell}\\|\\mathcal{G}(x^{k_{0}})\\|=0,\n\\end{equation*}}%\nwhich clearly yields \\eqref{glob}. The continuity of $\\mathcal{G}(\\cdot)$ ensures that $\\|\\mathcal{G}(\\bar{x})\\|=0$ for a limiting point $\\bar{x}$ of the sequence $\\{x^k\\}_{k\\in K_0}$, and thus $\\bar{x}\n\\in\\mathcal{X}^*$.\n","sha256_utf8":"b7958d3ec4a5e9f7acd52cd2ea69aad1fd0e2758f37d96ca3b2a4fef6dc76b10"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"接受指标之间目标值夹逼","source_units":[{"start":48162,"end":48622,"start_line":723,"end_line":727,"raw_tex":"Consider now any limiting point $\\ox$ of the entire sequence of iterates $\\{x^k\\}_{k\\in K}$. If there exists $\\bar{k}$ such that $k\\in K_0$ for all $k\\ge\\bar{k}$, it is easy to see that $\\ox$ is a global\nminimizer of \\eqref{prob_ori_1}. Otherwise, for any $k\\notin K_0$, denote by $k_{\\ell}\\in K_0$ the largest number satisfying $k_{\\ell}<k$, and hence we get the following estimate from Step~5:\n\\[\n F^*\\le F(x^{k})\\le F(x^{k-1})\\le\\ldots\\le F(x^{k_\\ell}).\n\\]\n","sha256_utf8":"98208bfc00a8d51ae1f945f5d062a471b28db9e7ec7190ec67a2867dcec95a60"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"有界时目标值及聚点最优","source_units":[{"start":48622,"end":49057,"start_line":728,"end_line":730,"raw_tex":"When the sequence $\\{x^k\\}_{k\\in K_0}$ is bounded, since any limiting point of $\\{x^k\\}_{k\\in K_0}$ is a global minimizer of \\eqref{prob_ori_1} as already shown, it follows that\n$\\lim\\limits_{\\ell\\rightarrow\\infty}F(x^{k_\\ell})=F^*$. This readily verifies by the constructions above that $\\lim\\limits_{k\\rightarrow\\infty}F(x^{k})=F^*$, and thus any limiting point\nof $\\{x^k\\}_{k\\in K}$ provides a global minimum to \\eqref{prob_ori_1}.\n","sha256_utf8":"17db554ff69aa6411bd4f8eec5239443b9dad56804262f5074ca1f6e44196d78"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"有限直接接受时下降求和","source_units":[{"start":49058,"end":49550,"start_line":732,"end_line":739,"raw_tex":"Next we consider the case where $K_0$ is finite and denote $\\bar{k}:=\\max_{k\\in K_0}k$. It follows from Lemma~\\ref{decrease} that for any $k>\\bar{k}$ we get\n\\[\nF(x^{k+1})-F(x^k)\\le-\\frac{\\gamma\\theta(1-\\theta)}{2L_1}\\left(\\frac{(1-\\nu)\\alpha_k}{1+M+\\alpha_k}\\right)^2\\|\\mathcal{G}(x^k)\\|^2,\n\\]\nwhich therefore tells us that\n\\[\n\\sum_{k=\\bar{k}}^{\\infty}\\frac{\\gamma\\theta(1-\\theta)}{2L_1}\\left(\\frac{(1-\\nu)\\alpha_k}{1+M+\\alpha_k}\\right)^2\\|\\mathcal{G}(x^k)\\|^2\\le F(x^{\\bar{k}})-F^*\\le 0.\n\\]\n","sha256_utf8":"14f0357fd3019c2eae669dece69053729909eb2b891005e8d28130911ec33496"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"正则化关系推出残差极限","source_units":[{"start":49550,"end":50131,"start_line":740,"end_line":749,"raw_tex":"The latter implies in turn that\n\\[\n\\lim_{k\\rightarrow \\infty}\\frac{\\gamma\\theta(1-\\theta)}{2L_1}\\left(\\frac{(1-\\nu)\\alpha_k}{1+M+\\alpha_k}\\right)^2\\|\\mathcal{G}(x^k)\\|^2=0.\n\\]\nRemembering the choice of \\revise{$\\alpha_k=\\min \\left\\{ \\bar{\\alpha}, c\\|\\mathcal{G}(x^k)\\|^{\\rho}\\right\\}$} with $\\bar{\\alpha}, c, \\rho>0$ ensures that\n\\[\n\\lim_{k\\rightarrow\\infty}\\|\\mathcal{G}(x^k)\\|=0,\n\\]\nhence \\eqref{glob} holds. This readily verifies by the continuity of $\\mathcal{G}(\\cdot)$ that any limiting point of $\\{x^k\\}_{k\\in K}$ provides a global minimum to \\eqref{prob_ori_1}.\n\\end{proof}","sha256_utf8":"1f1c84fc9a1a7b715c59909778026f4bb360c7717088511d2d2a15b570e92546"}]},{"id":"S019","semantic_type":"lemma","kind":"lemma","title":"非精确解的次梯度误差","source_units":[{"start":40908,"end":41262,"start_line":604,"end_line":608,"raw_tex":"\\begin{lemma}\\label{lem1} Given an approximate solution $\\hat{x}^k$ to \\eqref{subproblem1}, there exists a vector $e_k\\in\\R^n$ such that\n\\begin{equation}\\label{ek}\ne_k\\in\\nabla f(x^k)+H_k(\\hat{x}^k-x^k)+\\partial g(\\hat{x}^k-e_k)\\;\\mbox{ and }\\;\\|e_k\\|\\le\\nu\\min\\big\\{\\|\\mathcal{G}(x^k)\\|,\\|\\mathcal{G}(x^k)\\|^{1+\\varrho}\\big\\}.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"27a9a97b4ba2d3830b6dd37d571a17da55a34ace66d634ba245884fa80a8fc8d"}]},{"id":"S020","semantic_type":"proof","kind":"proof","title":"次梯度误差完整证明","source_units":[{"start":41271,"end":41631,"start_line":609,"end_line":616,"raw_tex":"\\begin{proof}\nLet $e_k:=r_k(\\hat{x}^k)=\\hat{x}^k-\\mathrm{Prox}_{g}(\\hat{x}^k-\\nabla f(x^k)-H_k(\\hat{x}^k-x^k))$. Then we have\n\\[\ne_k\\in\\nabla f(x^k)+H_k(\\hat{x}^k-x^k)+\\partial g(\\hat{x}^k-e_k),\n\\]\nwhich follow from \\eqref{prox-map}.\nUsing finally the inexact conditions \\eqref{inexact_condition_1} for $\\hat x^k$, we verify the claim of the lemma.\n\\end{proof}","sha256_utf8":"f164b5d42b6c82f458f96bfaf58ae65e323772af7a6c4dedc45fe04160fbbe07"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"残差构造次梯度包含式","source_units":[{"start":41271,"end":41505,"start_line":609,"end_line":614,"raw_tex":"\\begin{proof}\nLet $e_k:=r_k(\\hat{x}^k)=\\hat{x}^k-\\mathrm{Prox}_{g}(\\hat{x}^k-\\nabla f(x^k)-H_k(\\hat{x}^k-x^k))$. Then we have\n\\[\ne_k\\in\\nabla f(x^k)+H_k(\\hat{x}^k-x^k)+\\partial g(\\hat{x}^k-e_k),\n\\]\nwhich follow from \\eqref{prox-map}.\n","sha256_utf8":"152b278119618a706cdc689b9ea037b8b487f8259457057fe28b274629c2e3b2"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"非精确准则给出误差界","source_units":[{"start":41505,"end":41631,"start_line":615,"end_line":616,"raw_tex":"Using finally the inexact conditions \\eqref{inexact_condition_1} for $\\hat x^k$, we verify the claim of the lemma.\n\\end{proof}","sha256_utf8":"94d63b4b0da9f43e8cddde6d5d99d11a0ac286b017302e647348068d18177a96"}]},{"id":"S023","semantic_type":"lemma","kind":"lemma","title":"回溯步长与充分下降","source_units":[{"start":41805,"end":42311,"start_line":620,"end_line":628,"raw_tex":"\\begin{lemma}\\label{decrease} Let $t_k$ be chosen by the backtracking line search in Step~{\\rm 5} of Algorithm~{\\rm\\ref{Newton}} at iteration $k$. Then we have the step size estimate\n\\begin{equation}\\label{tk}\nt_k \\ge \\frac{\\gamma(1-\\theta)\\alpha_k}{L_1}\n\\end{equation}\nwith the cost function decrease satisfying\n\\begin{equation}\\label{decrease_eq}\nF(x^{k+1})-F(x^k)\\le-\\frac{\\gamma\\theta(1-\\theta)}{2L_1}\\left(\\frac{(1-\\nu)\\alpha_k}{1+M+\\alpha_k}\\right)^2\\|\\mathcal{G}(x^k)\\|^2.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"4ece636a2d1f49766f213a450278110f258c813446d18934d6a36a706aa8af23"}]},{"id":"S024","semantic_type":"proof","kind":"proof","title":"步长及下降完整证明","source_units":[{"start":42312,"end":46375,"start_line":629,"end_line":696,"raw_tex":"\\begin{proof} Since $\\hat{x}^k$ is an inexact solution to \\eqref{subproblem} obeying the conditions in \\eqref{inexact_condition_1}, it follows that\n\\[\n0\\ge q_k(\\hat{x}^k)-q_k(x^k)=l_k(\\hat{x}^k)-l_k(x^k)+\\frac{1}{2}(\\hat{x}^k-x^k)^TH_k(\\hat{x}^k-x^k),\n\\]\nwhere $l_k$ is the linear part of $q_k$ defined in \\eqref{lin}. This yields\n\\begin{equation}\\label{decrease_eq1}\nl_k(x^k)-l_k(\\hat{x}^k)\\ge\\frac{1}{2}(\\hat{x}^k-x^k)^TH_k(\\hat{x}^k-x^k)\\ge\\frac{1}{2}\\alpha_k\\|\\hat{x}^k-x^k\\|^2.\n\\end{equation}\nBy $\\mathcal{G}(x^k)=x^k-\\mathrm{Prox}_{g}(x^k-\\nabla f(x^k))$ we deduce from the stationary and subdifferential sum rules that\n\\[\n\\mathcal{G}(x^k)\\in\\nabla f(x^k)+\\partial g\\big(x^k-\\mathcal{G}(x^k)\\big).\n\\]\nFurthermore, Lemma~\\ref{lem1} gives us the condition $e_k\\in\\nabla f(x^k)+H_k(\\hat{x}^k-x^k)+\\partial g(\\hat{x}^k-e_k)$ for $\\hat x^k$ with $e_k$ satisfying the estimate\n$\\|e_k\\|\\le\\nu\\|\\mathcal{G}(x^k)\\|$. The monotonicity of the subgradient mapping $\\partial g$ ensures that\n\\[\n\\big\\langle \\mathcal{G}(x^k)+H_k(\\hat{x}^k-x^k)-e_k,x^k-\\mathcal{G}(x^k)-\\hat{x}^k+e_k\\big\\rangle\\ge 0,\n\\]\nwhich therefore leads us to the inequality\n\\begin{equation*}\n\\begin{aligned}\n\\| \\mathcal{G}(x^k) - e_k\\|^2 & \\le \\|\\mathcal{G}(x^k)\\|^2 -2\\big\\langle e_k,\\mathcal{G}(x^k)\\big\\rangle +\\|e_k\\|^2+(\\hat{x}^k-x^k)^TH_k(\\hat{x}^k-x^k) \\\\\n&\\le\\big\\langle \\mathcal{G}(x^k) - e_k,x^k-\\hat{x}^k+H_k(x^k-\\hat{x}^k)\\big\\rangle \\\\\n&\\le\\big\\| \\mathcal{G}(x^k) - e_k\\big\\| \\cdot \\big\\| x^k-\\hat{x}^k+H_k(x^k-\\hat{x}^k)\\big\\|.\n\\end{aligned}\n\\end{equation*}\nUsing again the condition $\\|e_k\\|\\le\\nu\\|\\mathcal{G}(x^k)\\|$ together with $\\|B_k\\|\\le M$ from \\eqref{assum_B_bounded} results in\n\\begin{equation*}\n\\|\\mathcal{G}(x^k)\\| \\le \\| \\mathcal{G}(x^k) - e_k\\| + \\|e_k\\| \\le(1+M+\\alpha_k)\\|\\hat{x}^k-x^k\\|+ \\nu\\|\\mathcal{G}(x^k)\\|.\n\\end{equation*}\nRemembering the choice of $\\nu\\in[0, 1)$, we estimate the prox-gradient mapping \\eqref{prox-gra} at the iterate $x^k$ by\n\\begin{equation}\\label{decrease_eq2}\n\\|\\mathcal{G}(x^k)\\|\\le\\frac{1+M+\\alpha_k}{1-\\nu}\\|\\hat{x}^k-x^k\\|.\n\\end{equation}\n\nNext let us show that the backtracking line search along the direction $d^k=\\hat{x}^k- x^k$ in Step~5 is well-defined and the proposed step size ensures a sufficient decrease in the cost function $F$.\nIt follows from the Lipschitz continuity of $\\nabla f$ that\n\\[\nf(x^k+\\tau d^k)\\le f(x^k)+\\tau\\nabla f(x^k)^Td^k+\\frac{L_1}{2}\\tau^2\\|d^k\\|^2\\;\\mbox{ for any }\\;\\tau\\ge 0,\n\\]\nand thus we deduce from the definition of $l_k$ in \\eqref{lin} that\n\\[\nF(x^k)-F(x^k+\\tau d^k)\\ge l_k(x^k)-l_k(x^k+\\tau d^k)-\\frac{L_1}{2}\\tau^2\\|d^k\\|^2.\n\\]\nThis implies by the convexity of $g$ that\n\\[\nl_k(x^k)-l_k(x^k+\\tau d^k)\\ge\\tau\\big(l_k(x^k)-l_k(x^k+d^k)\\big).\n\\]\nCombining the latter with \\eqref{decrease_eq1} and using the choice of $\\theta\\in(0,1)$ yield the relationships\n\\begin{equation}\\label{decrease_eq3}\n\\begin{aligned}\n&F(x^k)-F(x^k+\\tau d^k)-\\frac{\\theta\\alpha_k\\tau}{2}\\|d^k\\|^2\\\\\n\\ge~&l_k(x^k)-l_k(x^k+\\tau d^k)-\\frac{L_1}{2}\\tau^2\\|d^k\\|^2-\\frac{\\theta\\alpha_k\\tau}{2}\\|d^k\\|^2\\\\\n\\ge~&\\tau\\big(l_k(x^k)-l_k(x^k+d^k)\\big)-\\frac{L_1}{2}\\tau^2\\|d^k\\|^2-\\frac{\\theta\\alpha_k\\tau}{2}\\|d^k\\|^2\\\\\n\\ge~&(1-\\theta)\\tau\\frac{\\alpha_k}{2}\\|d^k\\|^2-\\frac{L_1}{2}\\tau^2\\|d^k\\|^2\\\\\n=~&\\frac{\\tau}{2}\\|d^k\\|^2\\big((1-\\theta)\\alpha_k-L_1\\tau\\big).\n\\end{aligned}\n\\end{equation}\nThis tells us that the backtracking line search criterion \\eqref{line_search} fulfills when $0<\\tau\\le\\frac{(1-\\theta)\\alpha_k}{L_1}$, and thus the step size $t_k$ satisfies the claimed condition\n\\eqref{tk}. Substituting now $\\tau:=t_k\\ge\\frac{\\gamma(1-\\theta)\\alpha_k}{L_1}$ into \\eqref{decrease_eq3} and employing the estimate of $\\|\\mathcal{G}(x^k)\\|^2$ from \\eqref{decrease_eq2}, we arrive at the\ninequalities\n\\[\n\\begin{aligned}\nF(x^k)-F(x^k+t_k d^k)&\\ge\\frac{\\theta\\alpha_kt_k}{2}\\|d^k\\|^2\\\\\n&\\ge\\frac{\\gamma\\theta(1-\\theta)\\alpha_k^2}{2L_1}\\left(\\frac{1-\\nu}{1+M+\\alpha_k}\\right)^2\\|\\mathcal{G}(x^k)\\|^2,\n\\end{aligned}\n\\]\nwhich verify the decreasing condition \\eqref{decrease_eq} and thus completes the proof of the lemma.\n\\end{proof}","sha256_utf8":"3089c13e1469c1a6ad1d935d142fb4700013335ec1a1e6bbd6b4728f4b80fe5a"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"模型下降控制方向平方","source_units":[{"start":42312,"end":42810,"start_line":629,"end_line":636,"raw_tex":"\\begin{proof} Since $\\hat{x}^k$ is an inexact solution to \\eqref{subproblem} obeying the conditions in \\eqref{inexact_condition_1}, it follows that\n\\[\n0\\ge q_k(\\hat{x}^k)-q_k(x^k)=l_k(\\hat{x}^k)-l_k(x^k)+\\frac{1}{2}(\\hat{x}^k-x^k)^TH_k(\\hat{x}^k-x^k),\n\\]\nwhere $l_k$ is the linear part of $q_k$ defined in \\eqref{lin}. This yields\n\\begin{equation}\\label{decrease_eq1}\nl_k(x^k)-l_k(\\hat{x}^k)\\ge\\frac{1}{2}(\\hat{x}^k-x^k)^TH_k(\\hat{x}^k-x^k)\\ge\\frac{1}{2}\\alpha_k\\|\\hat{x}^k-x^k\\|^2.\n\\end{equation}\n","sha256_utf8":"23402a0dc1d101686b61997ebe408e0d8883942aa513ea93f55906ffbc2d97a3"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"近端梯度的次梯度包含式","source_units":[{"start":42810,"end":43019,"start_line":637,"end_line":640,"raw_tex":"By $\\mathcal{G}(x^k)=x^k-\\mathrm{Prox}_{g}(x^k-\\nabla f(x^k))$ we deduce from the stationary and subdifferential sum rules that\n\\[\n\\mathcal{G}(x^k)\\in\\nabla f(x^k)+\\partial g\\big(x^k-\\mathcal{G}(x^k)\\big).\n\\]\n","sha256_utf8":"ec51bc19ef2553142b97faf5b5ab36384a3a4e2d2d2751743fcd1a6b0db50323"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"单调性比较两种残差","source_units":[{"start":43019,"end":43847,"start_line":641,"end_line":653,"raw_tex":"Furthermore, Lemma~\\ref{lem1} gives us the condition $e_k\\in\\nabla f(x^k)+H_k(\\hat{x}^k-x^k)+\\partial g(\\hat{x}^k-e_k)$ for $\\hat x^k$ with $e_k$ satisfying the estimate\n$\\|e_k\\|\\le\\nu\\|\\mathcal{G}(x^k)\\|$. The monotonicity of the subgradient mapping $\\partial g$ ensures that\n\\[\n\\big\\langle \\mathcal{G}(x^k)+H_k(\\hat{x}^k-x^k)-e_k,x^k-\\mathcal{G}(x^k)-\\hat{x}^k+e_k\\big\\rangle\\ge 0,\n\\]\nwhich therefore leads us to the inequality\n\\begin{equation*}\n\\begin{aligned}\n\\| \\mathcal{G}(x^k) - e_k\\|^2 & \\le \\|\\mathcal{G}(x^k)\\|^2 -2\\big\\langle e_k,\\mathcal{G}(x^k)\\big\\rangle +\\|e_k\\|^2+(\\hat{x}^k-x^k)^TH_k(\\hat{x}^k-x^k) \\\\\n&\\le\\big\\langle \\mathcal{G}(x^k) - e_k,x^k-\\hat{x}^k+H_k(x^k-\\hat{x}^k)\\big\\rangle \\\\\n&\\le\\big\\| \\mathcal{G}(x^k) - e_k\\big\\| \\cdot \\big\\| x^k-\\hat{x}^k+H_k(x^k-\\hat{x}^k)\\big\\|.\n\\end{aligned}\n\\end{equation*}\n","sha256_utf8":"6809d5560dda6de3575f18cff61ccc7a17e2904d91ee1e96c9f2a01ec29eef15"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"残差范数受方向范数控制","source_units":[{"start":43847,"end":44377,"start_line":654,"end_line":661,"raw_tex":"Using again the condition $\\|e_k\\|\\le\\nu\\|\\mathcal{G}(x^k)\\|$ together with $\\|B_k\\|\\le M$ from \\eqref{assum_B_bounded} results in\n\\begin{equation*}\n\\|\\mathcal{G}(x^k)\\| \\le \\| \\mathcal{G}(x^k) - e_k\\| + \\|e_k\\| \\le(1+M+\\alpha_k)\\|\\hat{x}^k-x^k\\|+ \\nu\\|\\mathcal{G}(x^k)\\|.\n\\end{equation*}\nRemembering the choice of $\\nu\\in[0, 1)$, we estimate the prox-gradient mapping \\eqref{prox-gra} at the iterate $x^k$ by\n\\begin{equation}\\label{decrease_eq2}\n\\|\\mathcal{G}(x^k)\\|\\le\\frac{1+M+\\alpha_k}{1-\\nu}\\|\\hat{x}^k-x^k\\|.\n\\end{equation}\n","sha256_utf8":"943666a991fcc290697060b3e33fa50b63bb8225f6cf97b35c0043fb9188916b"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"光滑性与凸性估计目标下降","source_units":[{"start":44378,"end":45024,"start_line":663,"end_line":675,"raw_tex":"Next let us show that the backtracking line search along the direction $d^k=\\hat{x}^k- x^k$ in Step~5 is well-defined and the proposed step size ensures a sufficient decrease in the cost function $F$.\nIt follows from the Lipschitz continuity of $\\nabla f$ that\n\\[\nf(x^k+\\tau d^k)\\le f(x^k)+\\tau\\nabla f(x^k)^Td^k+\\frac{L_1}{2}\\tau^2\\|d^k\\|^2\\;\\mbox{ for any }\\;\\tau\\ge 0,\n\\]\nand thus we deduce from the definition of $l_k$ in \\eqref{lin} that\n\\[\nF(x^k)-F(x^k+\\tau d^k)\\ge l_k(x^k)-l_k(x^k+\\tau d^k)-\\frac{L_1}{2}\\tau^2\\|d^k\\|^2.\n\\]\nThis implies by the convexity of $g$ that\n\\[\nl_k(x^k)-l_k(x^k+\\tau d^k)\\ge\\tau\\big(l_k(x^k)-l_k(x^k+d^k)\\big).\n\\]\n","sha256_utf8":"bdafe654575f9dfbc68ea3e15dea9a954abf81563d3013b45e4a2734de432630"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"合并模型界得回溯充分条件","source_units":[{"start":45024,"end":45635,"start_line":676,"end_line":685,"raw_tex":"Combining the latter with \\eqref{decrease_eq1} and using the choice of $\\theta\\in(0,1)$ yield the relationships\n\\begin{equation}\\label{decrease_eq3}\n\\begin{aligned}\n&F(x^k)-F(x^k+\\tau d^k)-\\frac{\\theta\\alpha_k\\tau}{2}\\|d^k\\|^2\\\\\n\\ge~&l_k(x^k)-l_k(x^k+\\tau d^k)-\\frac{L_1}{2}\\tau^2\\|d^k\\|^2-\\frac{\\theta\\alpha_k\\tau}{2}\\|d^k\\|^2\\\\\n\\ge~&\\tau\\big(l_k(x^k)-l_k(x^k+d^k)\\big)-\\frac{L_1}{2}\\tau^2\\|d^k\\|^2-\\frac{\\theta\\alpha_k\\tau}{2}\\|d^k\\|^2\\\\\n\\ge~&(1-\\theta)\\tau\\frac{\\alpha_k}{2}\\|d^k\\|^2-\\frac{L_1}{2}\\tau^2\\|d^k\\|^2\\\\\n=~&\\frac{\\tau}{2}\\|d^k\\|^2\\big((1-\\theta)\\alpha_k-L_1\\tau\\big).\n\\end{aligned}\n\\end{equation}\n","sha256_utf8":"79c48c125cfce9cd8ad90a91a1dbdfc435bf513d9f5924ec9fd7722f73705edd"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"步长下界及残差下降估计","source_units":[{"start":45635,"end":46375,"start_line":686,"end_line":696,"raw_tex":"This tells us that the backtracking line search criterion \\eqref{line_search} fulfills when $0<\\tau\\le\\frac{(1-\\theta)\\alpha_k}{L_1}$, and thus the step size $t_k$ satisfies the claimed condition\n\\eqref{tk}. Substituting now $\\tau:=t_k\\ge\\frac{\\gamma(1-\\theta)\\alpha_k}{L_1}$ into \\eqref{decrease_eq3} and employing the estimate of $\\|\\mathcal{G}(x^k)\\|^2$ from \\eqref{decrease_eq2}, we arrive at the\ninequalities\n\\[\n\\begin{aligned}\nF(x^k)-F(x^k+t_k d^k)&\\ge\\frac{\\theta\\alpha_kt_k}{2}\\|d^k\\|^2\\\\\n&\\ge\\frac{\\gamma\\theta(1-\\theta)\\alpha_k^2}{2L_1}\\left(\\frac{1-\\nu}{1+M+\\alpha_k}\\right)^2\\|\\mathcal{G}(x^k)\\|^2,\n\\end{aligned}\n\\]\nwhich verify the decreasing condition \\eqref{decrease_eq} and thus completes the proof of the lemma.\n\\end{proof}","sha256_utf8":"386f08cf7db5019685085c5401fc17a0ea16493081568748513663175f07b4b0"}]},{"id":"S032","semantic_type":"external_fact","kind":"external_fact","title":"近端最优性与非扩张性","source_units":[],"declared_external":true,"declaration_text":["For proper lower semicontinuous convex g, Prox_g is single-valued and nonexpansive, and z=Prox_g(u) iff u-z∈∂g(z). With continuous ∇f, G(x)=x-Prox_g(x-∇f(x)) is continuous."]},{"id":"S033","semantic_type":"external_fact","kind":"external_fact","title":"凸次梯度单调性与最优性","source_units":[],"declared_external":true,"declaration_text":["The convex subdifferential ∂g is monotone. Under the stated smoothness and domain assumptions, ∂(f+g)=∇f+∂g, and x globally minimizes f+g iff 0∈∇f(x)+∂g(x), equivalently G(x)=0."]},{"id":"S034","semantic_type":"external_fact","kind":"external_fact","title":"下降引理与凸插值界","source_units":[],"declared_external":true,"declaration_text":["An L-Lipschitz gradient gives f(x+s)≤f(x)+⟨∇f(x),s⟩+(L/2)||s||² on admissible segments. A convex function at an interpolated point is bounded above by the corresponding interpolation of endpoint values."]},{"id":"S035","semantic_type":"external_fact","kind":"external_fact","title":"有限维序列与极限基本事实","source_units":[],"declared_external":true,"declaration_text":["Bounded finite-dimensional sequences have convergent subsequences. Continuity preserves subsequential limits. Geometric sequences with ratio in (0,1) vanish, nonnegative summable terms vanish, and scalar monotonicity and squeeze arguments give the associated limits."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S012","edges":[{"source_id":"S001","target_id":"S008","kind":"supports"},{"source_id":"S002","target_id":"S014","kind":"supports"},{"source_id":"S002","target_id":"S016","kind":"supports"},{"source_id":"S002","target_id":"S017","kind":"supports"},{"source_id":"S002","target_id":"S018","kind":"supports"},{"source_id":"S002","target_id":"S029","kind":"supports"},{"source_id":"S003","target_id":"S025","kind":"supports"},{"source_id":"S003","target_id":"S029","kind":"supports"},{"source_id":"S004","target_id":"S010","kind":"supports"},{"source_id":"S004","target_id":"S025","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S008","kind":"supports"},{"source_id":"S005","target_id":"S021","kind":"supports"},{"source_id":"S005","target_id":"S026","kind":"supports"},{"source_id":"S006","target_id":"S008","kind":"supports"},{"source_id":"S006","target_id":"S009","kind":"supports"},{"source_id":"S006","target_id":"S014","kind":"supports"},{"source_id":"S006","target_id":"S018","kind":"supports"},{"source_id":"S006","target_id":"S026","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S025","kind":"supports"},{"source_id":"S007","target_id":"S028","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S021","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S022","kind":"supports"},{"source_id":"S009","target_id":"S025","kind":"supports"},{"source_id":"S009","target_id":"S028","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S014","kind":"supports"},{"source_id":"S010","target_id":"S015","kind":"supports"},{"source_id":"S010","target_id":"S018","kind":"supports"},{"source_id":"S010","target_id":"S031","kind":"supports"},{"source_id":"S011","target_id":"S014","kind":"supports"},{"source_id":"S011","target_id":"S015","kind":"supports"},{"source_id":"S011","target_id":"S017","kind":"supports"},{"source_id":"S014","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S015","target_id":"S013","kind":"supports"},{"source_id":"S016","target_id":"S013","kind":"supports"},{"source_id":"S017","target_id":"S013","kind":"supports"},{"source_id":"S018","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S027","kind":"supports"},{"source_id":"S023","target_id":"S017","kind":"supports"},{"source_id":"S025","target_id":"S030","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S028","target_id":"S031","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S027","target_id":"S024","kind":"supports"},{"source_id":"S028","target_id":"S024","kind":"supports"},{"source_id":"S029","target_id":"S024","kind":"supports"},{"source_id":"S030","target_id":"S024","kind":"supports"},{"source_id":"S031","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S032","target_id":"S021","kind":"supports"},{"source_id":"S032","target_id":"S026","kind":"supports"},{"source_id":"S032","target_id":"S014","kind":"supports"},{"source_id":"S032","target_id":"S018","kind":"supports"},{"source_id":"S033","target_id":"S026","kind":"supports"},{"source_id":"S033","target_id":"S027","kind":"supports"},{"source_id":"S033","target_id":"S014","kind":"supports"},{"source_id":"S033","target_id":"S018","kind":"supports"},{"source_id":"S034","target_id":"S029","kind":"supports"},{"source_id":"S035","target_id":"S014","kind":"supports"},{"source_id":"S035","target_id":"S016","kind":"supports"},{"source_id":"S035","target_id":"S018","kind":"supports"}]}}
{"record_id":"2012.12936v4","paper":{"title":"Newton acceleration on manifolds identified by proximal-gradient methods","authors":["Gilles Bareilles","Franck Iutzeler","Jérôme Malick"],"arxiv_id":"2012.12936","version":"v4","journal":"Mathematical Programming","doi":"10.1007/s10107-022-01873-w","abstract_url":"https://arxiv.org/abs/2012.12936v4","pdf_url":"https://arxiv.org/pdf/2012.12936v4"},"context":{"raw_tex":"\\begin{equation}\\tag{$\\mathcal P$} \\label{pb:compositepb}\n    \\min_{x\\in\\mathbb R^n} F(x) \\triangleq f(x)+g(x)\n\\end{equation}\n\n\\revise{In the rest of the paper, $\\M$ denotes a submanifold of $\\bbR^{n}$ or $\\bbR^{m \\times n}$.}\n\n\nA subset $\\M$ of $\\RR^n$ is said to be a \\emph{$p$-dimensional $\\C^2$-submanifold} of $\\RR^n$ around $\\bar{x} \\in \\M$ if there exists %\n$\\C^2$ function $\\varphi : \\RR^p \\to \\RR^n$ such that $\\varphi$ \\revise{maps} a neighborhood of $0 \\in \\RR^p$ \\revise{to} a neighborhood of $\\bar{x} \\in\\M$, \\revise{that admits a smooth (local) inverse,} and \\revise{which derivative} at $\\varphi^{-1}(\\bar{x})=0$ is injective.\nA $p$-dimensional $\\C^2$-submanifold of $\\RR^n$ can alternatively be defined via a local equation, that is, a $\\C^2$ function $\\Phi : \\RR^n \\to \\RR^{n-p}$ with a surjective derivative at $\\bar x \\in \\M$ that satisfies for all $x$ close enough to $\\bar x$: $x \\in\\M \\Leftrightarrow \\Phi(x) = 0$.\n\n\nA basic tool to investigate approximations on manifolds is notion of the \\emph{smooth curves}. A smooth curve on $\\M$ is a $\\C^2$ application $\\gamma : I\\subset \\RR \\to \\M\\subset \\RR^n$, where $I$ is an open interval containing $0$. At each point $x\\in\\M$, the \\emph{tangent space}, noted $\\tangentM$, can be defined as the velocities of all smooth curves passing by $x$ at $0$:\n\\begin{equation*}\n    \\tangentM \\triangleq \\left\\{ \\smoothcurve'(0) ~|~ \\smoothcurve : I \\to \\M \\text{ is a smooth curve around $0$ and } \\smoothcurve(0)=x \\right\\}.\n\\end{equation*}\nThe tangent space is a $p$-dimensional %\nspace containing \\emph{tangent vectors}. Each tangent space $\\tangentM$ is equipped with a scalar product $\\langle\\cdot, \\cdot\\rangle_x : \\tangentM\\times\\tangentM\\to\\RR$, and the associated norm $\\|\\cdot\\|_x$. In many cases, the tangent metric varies smoothly with $x$, making the manifold \\emph{Riemannian}. In this paper, we use the ambient space scalar product to define the scalar product on tangent spaces; we will thus drop the subscript in the tangent scalar product and norm notations when there is no confusion possible. Related to the tangent space, we will also consider the \\emph{normal space} $\\normalM$ at $x\\in\\M$, defined as the orthogonal space to $\\tangentM $ in $\\RR^n$, and the \\emph{tangent bundle manifold} defined by:\n\\[\n\\tangentBundle \\triangleq \\bigcup_{x\\in\\M}(x,\\tangentM).\n\\]\nNote also that both tangent and normal spaces at $x\\in\\M$  admit explicit expressions from derivatives of local parametrization $\\varphi$ or local equations $\\Phi$ defining $\\M$:\n\\begin{equation*}\n    \\tangentM = \\text{Im}~ \\D_\\varphi (0) = \\text{Ker}~ \\D_\\Phi (x) \\qquad \\normalM = \\text{Ker}~ \\D_\\varphi (0)^* = \\text{Im}~ \\D_\\Phi (x)^*\n\\end{equation*}\n\nA \\emph{metric} on $\\M$ can be defined as the minimal length over all curves joining two points $x, y\\in\\M$, ie. $\\distM(x, y) = \\inf_{\\smoothcurve \\in C_{x,y}}  \\int_0^1 \\|\\smoothcurve'(t)\\|_{\\smoothcurve(t)} \\mathrm{d}t$, where $C_{x,y}$ is the set of $[0, 1]\\to\\M$ smooth curves $\\smoothcurve$ such that $\\smoothcurve(0)=x$, $\\smoothcurve(1)=y$. The minimizing curves generalize the notion of straight line between two points to manifolds. The constant speed parametrization of any minimizing curve is called a \\emph{geodesic}.\n\n\nLet $\\funman:\\M\\to\\RR$, the \\emph{Riemannian differential} of $\\funman$ at\\;$x$ is the linear operator $\\D \\funman(x):\\tangentM\\to\\RR$ defined by $\\D \\funman(x)[\\eta]\\triangleq\\left. \\frac{\\mathrm{d}}{\\mathrm{d}t}\\funman\\circ \\smoothcurve(t) \\right|_{t=0}$, where $\\smoothcurve$ is a smooth curve such that $\\smoothcurve(0)=x$ and $\\smoothcurve'(0)=\\eta$. In turn, the \\emph{Riemannian gradient} $\\grad \\funman(x)$ is the unique vector of $\\tangentM$ such that, for any tangent vector\\;$\\eta$, $\\D \\funman(x)[\\eta] = \\langle \\grad \\funman(x), \\eta \\rangle$.\nIf $\\grad \\funman(x)$ exists, a first order Taylor development can be formulated. Let $x\\in\\M$, $\\eta\\in\\tangentM$ and $\\smoothcurve$ denote a smooth curve passing by $x$, with velocity $\\eta$ at $0$; then, for $t$ near $0$,\n\\begin{equation*}\\label{eq:manifoldFirstOrderDevSmoothCurve}\n    \\funman\\circ \\smoothcurve(t) = \\funman(x) + t \\langle \\grad \\funman(x), \\eta \\rangle + o(t).\n\\end{equation*}\n\nNotions of derivation for vector fields and of acceleration for curves are used to define second-order objects. Let a curve $\\smoothcurve:I\\to\\M$ and a smooth vector field $Z$ on $\\smoothcurve$, ie.\\;a smooth map such that $Z(t)\\in\\tangentM[c(t)]$ for $t\\in I$. The \\emph{covariant derivative} of $Z$ on the curve $\\smoothcurve$, denoted $\\frac{\\D}{\\mathrm{d}t}Z:I\\to\\tangentBundle$, is defined by $\\frac{\\D}{\\mathrm{d}t}Z(t) \\triangleq \\proj_{\\smoothcurve(t)} Z'(t)$, where $Z'(t)$ denotes the derivative in the ambient space $\\bbR^n$ and $\\proj_{x}$ corresponds to the orthogonal projector from $\\RR^n$ to $\\tangentM$. The \\emph{acceleration} of a curve $\\smoothcurve$ is defined as the covariant derivative of its velocity: $\\smoothcurve''(t) \\triangleq \\frac{\\D}{\\mathrm{d}t}\\smoothcurve'(0)$. \n\nThe \\emph{Riemannian Hessian}  of $\\funman$ at $x$ along $\\eta$ is the linear operator $\\Hess \\funman(x):\\tangentM\\to\\tangentM$ defined by the relation $\\Hess \\funman(x)[\\eta] \\triangleq \\left. \\frac{\\D}{\\mathrm{d}t} \\grad \\funman(\\smoothcurve(t)) \\right|_{t=0}$, where $\\smoothcurve$ is a smooth curve such that $\\smoothcurve(0)=x$, $\\smoothcurve'(0)=\\eta$. \nEquivalently, we have\n$\\langle\\Hess \\funman(x)[\\eta], \\eta\\rangle = \\left. \\frac{\\mathrm{d}^2}{\\mathrm{d}t^2}\\funman\\circ \\geocurve(t) \\right|_{t=0}$, where $\\geocurve$ is a geodesic such that $\\geocurve(0)=x$, $\\gamma'(0) = \\eta$.\nA second order Taylor development can now be formulated. Let $x\\in\\M$, $\\eta\\in\\tangentM$, and $\\smoothcurve$ be a smooth curve such that $\\smoothcurve(0)=x$, $\\smoothcurve'(0)=\\eta$. Then, for $t$ near $0$,\n\\begin{equation*}%\n    \\funman\\!\\circ\\!\\smoothcurve(t) = \\funman(x) \\!+ t \\langle \\grad \\funman(x), \\eta \\rangle + \\frac{t^2}{2}\\!(\\langle \\Hess \\funman(x)[\\eta], \\eta \\rangle\\!+\\!\\langle \\grad \\funman(x), c''(0)\\rangle)+ o(t^2).\n\\end{equation*}\n\n\n\\noindent\\textbf{Algorithms on manifolds: retractions and convergence rates.}\nIterative Riemannian methods require a way to produce curves on $\\M$ given a point $x$ and a tangent vector $\\eta$. A geodesic curve passing at $(x, \\eta)$, while attractive as the generalization of the straight line, has a prohibitive computational cost. We thus \\emph{retractions}, i.e. approximations of it, defined on a manifold $\\M$ as a smooth map $\\R : \\tangentBundle \\to \\M$ such that\n\\[%\n    \\text{$\\R_x (0) = x\\quad$ and $\\quad \\D\\R_x (0) : \\tangentM \\to\\tangentM$ is the identity map: $D\\R_x (0 )[v] = v$,}\n\\]%\nwhere, for each $x \\in\\M$, $\\R_x : \\tangentM\\to\\M$ is defined as the restriction of $\\R$ at $x$, so that $\\R_x(v) = \\R(x, v)$. A \\emph{second-order retraction} is a retraction $\\R$ such that, for all $(x, \\eta)\\in\\tangentBundle$, the curve $c(t)=\\R_x(t\\eta)$ has zero acceleration at 0: $c''(0)=0$.\nThus $t\\mapsto\\R_x(t\\eta)$ is a practical curve passing by $(x, \\eta)$ at $0$, and provides a similar development as above: %\nfor $t$ near 0,\n\\begin{equation}\\label{eq:manifoldSecondOrderDevSecRetraction}\n    \\funman\\circ\\R_x(t \\eta) = \\funman(x) + t \\langle \\grad \\funman(x), \\eta\\rangle + \\frac{t^2}{2}\\langle \\Hess \\funman(x)[\\eta], \\eta\\rangle + o(t^2 \\|\\eta\\|^2).\n\\end{equation}\n\n\nFinally, the convergence rates on manifolds are defined as follows. A sequence of points $(x_k)$ \\emph{converges (Q-)linearly} to some point $\\bar{x}\\in\\M$ if there exist an integer $K>0$ and a constant $q\\in(0,1)$ such that, for all $k\\ge K$, there holds\n\\begin{equation*}\n    \\distM(x_{k+1}, \\bar{x})\\le q~ \\distM(x_{k}, \\bar{x}).\n\\end{equation*}\nThe sequence \\emph{converges with order at least $p$} if there exists an integer $K>0$ and a constant $q\\in(0,1)$ such that, for all $k\\ge K$, there holds\n\\begin{equation*}\n    \\distM(x_{k+1}, \\bar{x})\\le q~\\distM(x_{k}, \\bar{x})^p.\n\\end{equation*}\nThe convergence is \\emph{superlinear} when $p>1$ and \\emph{quadratic} when $p=2$.\n\n\nConsider a point $\\bar{x}$ with $\\funns(\\bar{x})$ finite. The set of \\emph{regular subgradients} %\n\\begin{equation*}\n    \\widehat\\partial \\funns(\\bar{x}) \\triangleq \\left\\{ v : \\funns(x) \\ge \\funns(\\bar{x}) + \\langle v, x-\\bar{x} \\rangle + o(\\|x-\\bar{x}\\|) \\text{ for all } x\\in\\bbR^n \\right\\}\n\\end{equation*}\nis closed and convex, but the subdifferential mapping $\\widehat\\partial \\funns(\\cdot)$ may not be outer semi-continuous \\cite[Th. 8.6, Prop. 8.7]{rockafellar2009variational}. To overcome this problem, the set of \\emph{(general or limiting) subgradients} is defined as\n\\begin{equation*}\n    \\partial \\funns(\\bar{x}) \\triangleq \\left\\{ \\lim_r v_r : v_r\\in\\widehat\\partial \\funns(x_r), x_r\\to\\bar{x}, \\funns(x_r)\\to \\funns(\\bar{x}) \\right\\}.\n\\end{equation*}\nThe limiting subdifferential is by design outer semi-continuous:\n\\[\n\\limsup_{x\\to\\bar{x}} \\partial \\funns (x) = \\{u : \\exists x_r\\to\\bar{x}, \\exists u_r \\to u \\text{ with } u_r\\in  \\partial \\funns (x_r) \\} ~\\subset~ \\partial \\funns (\\bar{x}),\n\\]\nwhich is an attractive property to study the properties of sequences of points whose subgradients converge.\nWe say that a function is \\emph{(Clarke) regular} at $\\bar{x}$ if the regular and limiting subdifferentials at $\\bar{x}$ coincide \\cite[Def. 7.25, Cor. 8.11]{rockafellar2009variational}. This is notably the case for convex functions where the two above definitions coincide with the convex subdifferential \\cite[Prop. 8.12]{rockafellar2009variational}.\n\n\nThe subdifferential allows to derive optimality conditions: for a local minimizer $\\bar x$ of $F$, we have $0\\in \\partial F(\\bar x)$. For the objective function of \\eqref{pb:compositepb}, this writes\n\\[\n0\\in \\nabla f (\\bar x) + \\partial g(\\bar x) \\qquad\\text{or equivalently}\\qquad -\\nabla f (\\bar x) \\in \\partial g(\\bar x).\n\\]\nA point satisfying these conditions is called a critical point. The analysis of the algorithms of this paper will provide convergence guarantees towards critical points. %\n\n\nA central tool to tackle non-differentiable functions is the \\emph{proximity operator}.\nFor\\;$\\gamma\\!>\\!0$ and a function $\\funns$; it is defined as the set-valued mapping\n\\begin{equation*}\n    \\prox_{\\gamma \\funns}(y) \\triangleq \\argmin_{u\\in\\bbR^n} \\left\\{ \\funns(u) + \\frac{1}{2\\gamma}\\|u-y\\|^2\\right\\}.\n\\end{equation*}\n\n\nA function $\\funns$ is \\emph{prox-regular} at a point $\\bar{x}$ for a subgradient $\\bar{v} \\in \\partial \\funns(\\bar{x})$ if $\\funns$ is finite, locally lower semi-continuous at $\\bar{x}$, and there exists $r>0$ and $\\varepsilon>0$ such~that\n$%\n    \\funns(x') \\ge \\funns(x) + \\langle v, x'-x\\rangle - \\frac{r}{2}\\|x'-x\\|^2\n$ %\nwhenever $v \\in \\partial \\funns(x)$, $\\|x-\\bar{x}\\|<\\varepsilon$, $\\|x'-\\bar{x}\\|<\\varepsilon$, $\\|v-\\bar{v}\\|<\\varepsilon$ and $\\funns(x)<\\funns(\\bar{x})+\\varepsilon$.\nWhen this holds for all $\\bar{v} \\in\\partial \\funns(\\bar{x})$, we say that $\\funns$ is prox-regular at $\\bar{x}$ \\cite[Def. 13.27]{rockafellar2009variational}.\n\n\n\\begin{assumption}\\label{asm:fun}\n    The functions $f$ and $g$ are proper and \n    \\begin{itemize}\n        \\item[i)] $f$ is $\\C^2(\\bbR^n)$ with an $L$-Lipschitz continuous gradient;\n        \\item[ii)] $g$ is lower semi-continuous;\n        \\item[iii)] $\\prox_{\\gamma g}$ is non-empty on $\\mathbb R^n$ for any $\\gamma>0$;\n        \\item[iv)] $F(x) = f(x) + g(x)$ is bounded below.\n    \\end{itemize}\n\\end{assumption}\n\n\\begin{algorithm}\n    \\caption{General structure exploiting algorithm\\label{alg:generic}}\n    \\begin{algorithmic}[1]\n        \\Require \\revise{Pick $x_{0}$ arbitrary, $\\gamma < 1 / L$.}\n        \\Repeat\n        \\State Compute $x_{k}\\in\\prox_{\\gamma g}(y_{k-1} - \\gamma \\nabla f(y_{k-1}))$ and get $\\M_{k}\\ni x_{k}$ %\n        \\State Update $y_{k} = \\manupdate_{\\M_{k}} (x_{k})$ on the current manifold %\n        \\Until{stopping criterion}\n    \\end{algorithmic}\n\\end{algorithm}\n\n\\begin{definition}[partial smoothness]\n    A function $g$ is ($\\C^2$-)\\emph{partly smooth} at a point $\\bar{x}$ relative to a set $\\M$ containing $\\bar{x}$ if $\\M$ is a $\\C^2$ manifold around $\\bar{x}$ and:\n    \\begin{itemize}\n        \\item (smoothness) the restriction of $g$ to $\\M$ is a $\\C^2$ function near $\\bar{x}$;\n        \\item (regularity) $g$ is (Clarke) regular at all points $x\\in\\M$ near $\\bar{x}$, with $\\partial g(x)\\neq\\emptyset$;\n        \\item (sharpness) the affine span of $\\partial g(\\bar{x})$ is a translate of $\\normalM[\\bar{x}]$;\n        \\item (sub-continuity) the set-valued mapping $\\partial g$ restricted to $\\M$ is continuous at $\\bar{x}$.\n    \\end{itemize}\n\\end{definition}\n\n\\begin{definition}\\label{def:structqual}\n   A point $\\bar{x}$ of a $C^2$ submanifold $\\M$ is a $r$-\\emph{structured critical} point for $(f,g)$~if we have:\n    \\begin{itemize}\n        \\item[i)] proximal gradient stability: $\\bar{x} = \\prox_{g/r}(\\bar{x}-1/r \\nabla f( \\bar{x} ) )$ ;\n        \\item[ii)] qualification condition: $0 \\in \\ri (\\nabla f + \\partial g)(\\bar{x})$;\n        \\item[iii)] prox-regularity: $g$ is $r$-prox-regular at $\\bar{x}$;\n        \\item[iv)] partial smoothness: $g$ is partly-smooth at $\\bar{x}$ with respect to $\\M$.\n    \\end{itemize}\n\\end{definition}\n\nStanding at point $x\\in\\M$ with a proposed direction $\\eta\\in\\tangentM$, a stepsize $\\alpha>0$ is \\emph{acceptable} \\revise{if} it satisfies the following \\emph{Armijo} condition %\n\\begin{equation}\\label{eq:Armijo}\n    \\funtot(\\R_x(\\alpha \\eta)) \\le \\funtot(x) + m_1 \\alpha \\langle \\grad \\funtot(x), \\eta\\rangle, \\qquad\\text{for $0 < m_1 < 1/2$.}\n\\end{equation}\n\nThe line search employs a second-order retraction $\\R_x$, \\revisebis{e.g.\\;the exponential map, a projection retraction \\cite{Absil2012ProjectionlikeRO}, or any other second-order retraction \\cite{boumal2022intromanifolds}.\\footnote{\\revisebis{Indeed, in many applications of Riemannian optimization, computing geodesics and the exponential map can be costly and then retractions provide an efficient alternative. For this reason, we consider here second-order retractions \\cite{absil2009optimization,boumal2022intromanifolds}.}}} %\n\n\nIn the following, we will consider a \\emph{backtracking} \\revise{line search} for finding an acceptable stepsize $\\alpha$: the unit stepsize is first tried, and then the search space is reduced geometrically. %\nIn practice, we use exactly \\cite[Alg. A6.3.1]{dennis1996numerical}, which features polynomial interpolation of $\\funman$ in the search space.\n\n\n\\begin{algorithm}\n    \\caption{$\\manupdate$-Newton\\label{alg:ManUpNewton}}\n    \\begin{algorithmic}[1]\n        \\Require Manifold $\\M$, point $x\\in\\M$\n        \\State Find $d$ in $\\tangentM$ that solves\n        \\begin{equation*}\\tag{Newton equation}\\label{eq:Newtoneq}\n            \\grad F(x) + \\Hess F(x)[d] = 0\n        \\end{equation*}\n        \\State Find $\\alpha$ satisfying the Armijo condition \\eqref{eq:Armijo} with direction $d$\n        \\State \\Return $y = R_x(\\alpha d)$\n    \\end{algorithmic}\n\\end{algorithm}","units":[{"start":11512,"end":11637,"start_line":391,"end_line":393,"raw_tex":"\\begin{equation}\\tag{$\\mathcal P$} \\label{pb:compositepb}\n    \\min_{x\\in\\mathbb R^n} F(x) \\triangleq f(x)+g(x)\n\\end{equation}","sha256_utf8":"eae5d284ff3c6a0eaa73eb6a9f325c9fcc60ddbcee29f25005ad2340d34e08ae"},{"start":18060,"end":18160,"start_line":460,"end_line":460,"raw_tex":"\\revise{In the rest of the paper, $\\M$ denotes a submanifold of $\\bbR^{n}$ or $\\bbR^{m \\times n}$.}\n","sha256_utf8":"ce58f5b3ea502169d2177c909e98e0764707f216a0445821bc1cf47bdc094e86"},{"start":18202,"end":21205,"start_line":464,"end_line":483,"raw_tex":"A subset $\\M$ of $\\RR^n$ is said to be a \\emph{$p$-dimensional $\\C^2$-submanifold} of $\\RR^n$ around $\\bar{x} \\in \\M$ if there exists %\n$\\C^2$ function $\\varphi : \\RR^p \\to \\RR^n$ such that $\\varphi$ \\revise{maps} a neighborhood of $0 \\in \\RR^p$ \\revise{to} a neighborhood of $\\bar{x} \\in\\M$, \\revise{that admits a smooth (local) inverse,} and \\revise{which derivative} at $\\varphi^{-1}(\\bar{x})=0$ is injective.\nA $p$-dimensional $\\C^2$-submanifold of $\\RR^n$ can alternatively be defined via a local equation, that is, a $\\C^2$ function $\\Phi : \\RR^n \\to \\RR^{n-p}$ with a surjective derivative at $\\bar x \\in \\M$ that satisfies for all $x$ close enough to $\\bar x$: $x \\in\\M \\Leftrightarrow \\Phi(x) = 0$.\n\n\nA basic tool to investigate approximations on manifolds is notion of the \\emph{smooth curves}. A smooth curve on $\\M$ is a $\\C^2$ application $\\gamma : I\\subset \\RR \\to \\M\\subset \\RR^n$, where $I$ is an open interval containing $0$. At each point $x\\in\\M$, the \\emph{tangent space}, noted $\\tangentM$, can be defined as the velocities of all smooth curves passing by $x$ at $0$:\n\\begin{equation*}\n    \\tangentM \\triangleq \\left\\{ \\smoothcurve'(0) ~|~ \\smoothcurve : I \\to \\M \\text{ is a smooth curve around $0$ and } \\smoothcurve(0)=x \\right\\}.\n\\end{equation*}\nThe tangent space is a $p$-dimensional %\nspace containing \\emph{tangent vectors}. Each tangent space $\\tangentM$ is equipped with a scalar product $\\langle\\cdot, \\cdot\\rangle_x : \\tangentM\\times\\tangentM\\to\\RR$, and the associated norm $\\|\\cdot\\|_x$. In many cases, the tangent metric varies smoothly with $x$, making the manifold \\emph{Riemannian}. In this paper, we use the ambient space scalar product to define the scalar product on tangent spaces; we will thus drop the subscript in the tangent scalar product and norm notations when there is no confusion possible. Related to the tangent space, we will also consider the \\emph{normal space} $\\normalM$ at $x\\in\\M$, defined as the orthogonal space to $\\tangentM $ in $\\RR^n$, and the \\emph{tangent bundle manifold} defined by:\n\\[\n\\tangentBundle \\triangleq \\bigcup_{x\\in\\M}(x,\\tangentM).\n\\]\nNote also that both tangent and normal spaces at $x\\in\\M$  admit explicit expressions from derivatives of local parametrization $\\varphi$ or local equations $\\Phi$ defining $\\M$:\n\\begin{equation*}\n    \\tangentM = \\text{Im}~ \\D_\\varphi (0) = \\text{Ker}~ \\D_\\Phi (x) \\qquad \\normalM = \\text{Ker}~ \\D_\\varphi (0)^* = \\text{Im}~ \\D_\\Phi (x)^*\n\\end{equation*}\n\nA \\emph{metric} on $\\M$ can be defined as the minimal length over all curves joining two points $x, y\\in\\M$, ie. $\\distM(x, y) = \\inf_{\\smoothcurve \\in C_{x,y}}  \\int_0^1 \\|\\smoothcurve'(t)\\|_{\\smoothcurve(t)} \\mathrm{d}t$, where $C_{x,y}$ is the set of $[0, 1]\\to\\M$ smooth curves $\\smoothcurve$ such that $\\smoothcurve(0)=x$, $\\smoothcurve(1)=y$. The minimizing curves generalize the notion of straight line between two points to manifolds. The constant speed parametrization of any minimizing curve is called a \\emph{geodesic}.\n","sha256_utf8":"5c0c38c60d43885b47a7782ec85acfbe38fe6199bf7a05df73cb7400da9799fc"},{"start":21267,"end":24071,"start_line":487,"end_line":501,"raw_tex":"Let $\\funman:\\M\\to\\RR$, the \\emph{Riemannian differential} of $\\funman$ at\\;$x$ is the linear operator $\\D \\funman(x):\\tangentM\\to\\RR$ defined by $\\D \\funman(x)[\\eta]\\triangleq\\left. \\frac{\\mathrm{d}}{\\mathrm{d}t}\\funman\\circ \\smoothcurve(t) \\right|_{t=0}$, where $\\smoothcurve$ is a smooth curve such that $\\smoothcurve(0)=x$ and $\\smoothcurve'(0)=\\eta$. In turn, the \\emph{Riemannian gradient} $\\grad \\funman(x)$ is the unique vector of $\\tangentM$ such that, for any tangent vector\\;$\\eta$, $\\D \\funman(x)[\\eta] = \\langle \\grad \\funman(x), \\eta \\rangle$.\nIf $\\grad \\funman(x)$ exists, a first order Taylor development can be formulated. Let $x\\in\\M$, $\\eta\\in\\tangentM$ and $\\smoothcurve$ denote a smooth curve passing by $x$, with velocity $\\eta$ at $0$; then, for $t$ near $0$,\n\\begin{equation*}\\label{eq:manifoldFirstOrderDevSmoothCurve}\n    \\funman\\circ \\smoothcurve(t) = \\funman(x) + t \\langle \\grad \\funman(x), \\eta \\rangle + o(t).\n\\end{equation*}\n\nNotions of derivation for vector fields and of acceleration for curves are used to define second-order objects. Let a curve $\\smoothcurve:I\\to\\M$ and a smooth vector field $Z$ on $\\smoothcurve$, ie.\\;a smooth map such that $Z(t)\\in\\tangentM[c(t)]$ for $t\\in I$. The \\emph{covariant derivative} of $Z$ on the curve $\\smoothcurve$, denoted $\\frac{\\D}{\\mathrm{d}t}Z:I\\to\\tangentBundle$, is defined by $\\frac{\\D}{\\mathrm{d}t}Z(t) \\triangleq \\proj_{\\smoothcurve(t)} Z'(t)$, where $Z'(t)$ denotes the derivative in the ambient space $\\bbR^n$ and $\\proj_{x}$ corresponds to the orthogonal projector from $\\RR^n$ to $\\tangentM$. The \\emph{acceleration} of a curve $\\smoothcurve$ is defined as the covariant derivative of its velocity: $\\smoothcurve''(t) \\triangleq \\frac{\\D}{\\mathrm{d}t}\\smoothcurve'(0)$. \n\nThe \\emph{Riemannian Hessian}  of $\\funman$ at $x$ along $\\eta$ is the linear operator $\\Hess \\funman(x):\\tangentM\\to\\tangentM$ defined by the relation $\\Hess \\funman(x)[\\eta] \\triangleq \\left. \\frac{\\D}{\\mathrm{d}t} \\grad \\funman(\\smoothcurve(t)) \\right|_{t=0}$, where $\\smoothcurve$ is a smooth curve such that $\\smoothcurve(0)=x$, $\\smoothcurve'(0)=\\eta$. \nEquivalently, we have\n$\\langle\\Hess \\funman(x)[\\eta], \\eta\\rangle = \\left. \\frac{\\mathrm{d}^2}{\\mathrm{d}t^2}\\funman\\circ \\geocurve(t) \\right|_{t=0}$, where $\\geocurve$ is a geodesic such that $\\geocurve(0)=x$, $\\gamma'(0) = \\eta$.\nA second order Taylor development can now be formulated. Let $x\\in\\M$, $\\eta\\in\\tangentM$, and $\\smoothcurve$ be a smooth curve such that $\\smoothcurve(0)=x$, $\\smoothcurve'(0)=\\eta$. Then, for $t$ near $0$,\n\\begin{equation*}%\n    \\funman\\!\\circ\\!\\smoothcurve(t) = \\funman(x) \\!+ t \\langle \\grad \\funman(x), \\eta \\rangle + \\frac{t^2}{2}\\!(\\langle \\Hess \\funman(x)[\\eta], \\eta \\rangle\\!+\\!\\langle \\grad \\funman(x), c''(0)\\rangle)+ o(t^2).\n\\end{equation*}\n","sha256_utf8":"4c798af3142bd9c159465a3f6e4740264bc3925b9f93f584abb9987dce86a650"},{"start":24650,"end":26615,"start_line":513,"end_line":534,"raw_tex":"\\noindent\\textbf{Algorithms on manifolds: retractions and convergence rates.}\nIterative Riemannian methods require a way to produce curves on $\\M$ given a point $x$ and a tangent vector $\\eta$. A geodesic curve passing at $(x, \\eta)$, while attractive as the generalization of the straight line, has a prohibitive computational cost. We thus \\emph{retractions}, i.e. approximations of it, defined on a manifold $\\M$ as a smooth map $\\R : \\tangentBundle \\to \\M$ such that\n\\[%\n    \\text{$\\R_x (0) = x\\quad$ and $\\quad \\D\\R_x (0) : \\tangentM \\to\\tangentM$ is the identity map: $D\\R_x (0 )[v] = v$,}\n\\]%\nwhere, for each $x \\in\\M$, $\\R_x : \\tangentM\\to\\M$ is defined as the restriction of $\\R$ at $x$, so that $\\R_x(v) = \\R(x, v)$. A \\emph{second-order retraction} is a retraction $\\R$ such that, for all $(x, \\eta)\\in\\tangentBundle$, the curve $c(t)=\\R_x(t\\eta)$ has zero acceleration at 0: $c''(0)=0$.\nThus $t\\mapsto\\R_x(t\\eta)$ is a practical curve passing by $(x, \\eta)$ at $0$, and provides a similar development as above: %\nfor $t$ near 0,\n\\begin{equation}\\label{eq:manifoldSecondOrderDevSecRetraction}\n    \\funman\\circ\\R_x(t \\eta) = \\funman(x) + t \\langle \\grad \\funman(x), \\eta\\rangle + \\frac{t^2}{2}\\langle \\Hess \\funman(x)[\\eta], \\eta\\rangle + o(t^2 \\|\\eta\\|^2).\n\\end{equation}\n\n\nFinally, the convergence rates on manifolds are defined as follows. A sequence of points $(x_k)$ \\emph{converges (Q-)linearly} to some point $\\bar{x}\\in\\M$ if there exist an integer $K>0$ and a constant $q\\in(0,1)$ such that, for all $k\\ge K$, there holds\n\\begin{equation*}\n    \\distM(x_{k+1}, \\bar{x})\\le q~ \\distM(x_{k}, \\bar{x}).\n\\end{equation*}\nThe sequence \\emph{converges with order at least $p$} if there exists an integer $K>0$ and a constant $q\\in(0,1)$ such that, for all $k\\ge K$, there holds\n\\begin{equation*}\n    \\distM(x_{k+1}, \\bar{x})\\le q~\\distM(x_{k}, \\bar{x})^p.\n\\end{equation*}\nThe convergence is \\emph{superlinear} when $p>1$ and \\emph{quadratic} when $p=2$.\n","sha256_utf8":"50dbdc3520f12282bccceb954f34302281d099cce4b5d84b6f7fa73e494326a3"},{"start":30057,"end":31529,"start_line":586,"end_line":599,"raw_tex":"Consider a point $\\bar{x}$ with $\\funns(\\bar{x})$ finite. The set of \\emph{regular subgradients} %\n\\begin{equation*}\n    \\widehat\\partial \\funns(\\bar{x}) \\triangleq \\left\\{ v : \\funns(x) \\ge \\funns(\\bar{x}) + \\langle v, x-\\bar{x} \\rangle + o(\\|x-\\bar{x}\\|) \\text{ for all } x\\in\\bbR^n \\right\\}\n\\end{equation*}\nis closed and convex, but the subdifferential mapping $\\widehat\\partial \\funns(\\cdot)$ may not be outer semi-continuous \\cite[Th. 8.6, Prop. 8.7]{rockafellar2009variational}. To overcome this problem, the set of \\emph{(general or limiting) subgradients} is defined as\n\\begin{equation*}\n    \\partial \\funns(\\bar{x}) \\triangleq \\left\\{ \\lim_r v_r : v_r\\in\\widehat\\partial \\funns(x_r), x_r\\to\\bar{x}, \\funns(x_r)\\to \\funns(\\bar{x}) \\right\\}.\n\\end{equation*}\nThe limiting subdifferential is by design outer semi-continuous:\n\\[\n\\limsup_{x\\to\\bar{x}} \\partial \\funns (x) = \\{u : \\exists x_r\\to\\bar{x}, \\exists u_r \\to u \\text{ with } u_r\\in  \\partial \\funns (x_r) \\} ~\\subset~ \\partial \\funns (\\bar{x}),\n\\]\nwhich is an attractive property to study the properties of sequences of points whose subgradients converge.\nWe say that a function is \\emph{(Clarke) regular} at $\\bar{x}$ if the regular and limiting subdifferentials at $\\bar{x}$ coincide \\cite[Def. 7.25, Cor. 8.11]{rockafellar2009variational}. This is notably the case for convex functions where the two above definitions coincide with the convex subdifferential \\cite[Prop. 8.12]{rockafellar2009variational}.\n","sha256_utf8":"08ba300b62da1f464007eda5f41cea7d1053f889f042dbecc3a704a7cfdff6c8"},{"start":31600,"end":32100,"start_line":603,"end_line":607,"raw_tex":"The subdifferential allows to derive optimality conditions: for a local minimizer $\\bar x$ of $F$, we have $0\\in \\partial F(\\bar x)$. For the objective function of \\eqref{pb:compositepb}, this writes\n\\[\n0\\in \\nabla f (\\bar x) + \\partial g(\\bar x) \\qquad\\text{or equivalently}\\qquad -\\nabla f (\\bar x) \\in \\partial g(\\bar x).\n\\]\nA point satisfying these conditions is called a critical point. The analysis of the algorithms of this paper will provide convergence guarantees towards critical points. %\n","sha256_utf8":"3c882cb445555b7edcc83cc961181510da6aaa9ffab65adce7ef5e29930bb02c"},{"start":32148,"end":32472,"start_line":611,"end_line":615,"raw_tex":"A central tool to tackle non-differentiable functions is the \\emph{proximity operator}.\nFor\\;$\\gamma\\!>\\!0$ and a function $\\funns$; it is defined as the set-valued mapping\n\\begin{equation*}\n    \\prox_{\\gamma \\funns}(y) \\triangleq \\argmin_{u\\in\\bbR^n} \\left\\{ \\funns(u) + \\frac{1}{2\\gamma}\\|u-y\\|^2\\right\\}.\n\\end{equation*}\n","sha256_utf8":"1d24b438bbe3405f60426b77fa288b6364c98f643f123298641b5ee30a90a220"},{"start":33259,"end":33914,"start_line":625,"end_line":630,"raw_tex":"A function $\\funns$ is \\emph{prox-regular} at a point $\\bar{x}$ for a subgradient $\\bar{v} \\in \\partial \\funns(\\bar{x})$ if $\\funns$ is finite, locally lower semi-continuous at $\\bar{x}$, and there exists $r>0$ and $\\varepsilon>0$ such~that\n$%\n    \\funns(x') \\ge \\funns(x) + \\langle v, x'-x\\rangle - \\frac{r}{2}\\|x'-x\\|^2\n$ %\nwhenever $v \\in \\partial \\funns(x)$, $\\|x-\\bar{x}\\|<\\varepsilon$, $\\|x'-\\bar{x}\\|<\\varepsilon$, $\\|v-\\bar{v}\\|<\\varepsilon$ and $\\funns(x)<\\funns(\\bar{x})+\\varepsilon$.\nWhen this holds for all $\\bar{v} \\in\\partial \\funns(\\bar{x})$, we say that $\\funns$ is prox-regular at $\\bar{x}$ \\cite[Def. 13.27]{rockafellar2009variational}.\n","sha256_utf8":"d5dfffff8cb59effe6bc65bd06d44465f80fa85ce9c4df07ab9c93371b44b598"},{"start":38431,"end":38844,"start_line":708,"end_line":716,"raw_tex":"\\begin{assumption}\\label{asm:fun}\n    The functions $f$ and $g$ are proper and \n    \\begin{itemize}\n        \\item[i)] $f$ is $\\C^2(\\bbR^n)$ with an $L$-Lipschitz continuous gradient;\n        \\item[ii)] $g$ is lower semi-continuous;\n        \\item[iii)] $\\prox_{\\gamma g}$ is non-empty on $\\mathbb R^n$ for any $\\gamma>0$;\n        \\item[iv)] $F(x) = f(x) + g(x)$ is bounded below.\n    \\end{itemize}\n\\end{assumption}","sha256_utf8":"b094b90e4a82afeb4132a0aaa7efc6f5415fea29dfb226b10878c278a8e45bb3"},{"start":39849,"end":40322,"start_line":729,"end_line":738,"raw_tex":"\\begin{algorithm}\n    \\caption{General structure exploiting algorithm\\label{alg:generic}}\n    \\begin{algorithmic}[1]\n        \\Require \\revise{Pick $x_{0}$ arbitrary, $\\gamma < 1 / L$.}\n        \\Repeat\n        \\State Compute $x_{k}\\in\\prox_{\\gamma g}(y_{k-1} - \\gamma \\nabla f(y_{k-1}))$ and get $\\M_{k}\\ni x_{k}$ %\n        \\State Update $y_{k} = \\manupdate_{\\M_{k}} (x_{k})$ on the current manifold %\n        \\Until{stopping criterion}\n    \\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"e5634b8ce6bbba4498cbecced71fd8cbad0f1e548d2ab16a28f06cb6945b900c"},{"start":40542,"end":41243,"start_line":755,"end_line":763,"raw_tex":"\\begin{definition}[partial smoothness]\n    A function $g$ is ($\\C^2$-)\\emph{partly smooth} at a point $\\bar{x}$ relative to a set $\\M$ containing $\\bar{x}$ if $\\M$ is a $\\C^2$ manifold around $\\bar{x}$ and:\n    \\begin{itemize}\n        \\item (smoothness) the restriction of $g$ to $\\M$ is a $\\C^2$ function near $\\bar{x}$;\n        \\item (regularity) $g$ is (Clarke) regular at all points $x\\in\\M$ near $\\bar{x}$, with $\\partial g(x)\\neq\\emptyset$;\n        \\item (sharpness) the affine span of $\\partial g(\\bar{x})$ is a translate of $\\normalM[\\bar{x}]$;\n        \\item (sub-continuity) the set-valued mapping $\\partial g$ restricted to $\\M$ is continuous at $\\bar{x}$.\n    \\end{itemize}\n\\end{definition}","sha256_utf8":"ef53f97690627049d4e8996cde379c4f6ed1cd9ef89e14dbf4520fc8b83c11e6"},{"start":48752,"end":49329,"start_line":836,"end_line":844,"raw_tex":"\\begin{definition}\\label{def:structqual}\n   A point $\\bar{x}$ of a $C^2$ submanifold $\\M$ is a $r$-\\emph{structured critical} point for $(f,g)$~if we have:\n    \\begin{itemize}\n        \\item[i)] proximal gradient stability: $\\bar{x} = \\prox_{g/r}(\\bar{x}-1/r \\nabla f( \\bar{x} ) )$ ;\n        \\item[ii)] qualification condition: $0 \\in \\ri (\\nabla f + \\partial g)(\\bar{x})$;\n        \\item[iii)] prox-regularity: $g$ is $r$-prox-regular at $\\bar{x}$;\n        \\item[iv)] partial smoothness: $g$ is partly-smooth at $\\bar{x}$ with respect to $\\M$.\n    \\end{itemize}\n\\end{definition}","sha256_utf8":"ba0853430fb160cda17c0535b85d0b60f08963c788a6a621c6b2adac931bbbf9"},{"start":58063,"end":58424,"start_line":987,"end_line":990,"raw_tex":"Standing at point $x\\in\\M$ with a proposed direction $\\eta\\in\\tangentM$, a stepsize $\\alpha>0$ is \\emph{acceptable} \\revise{if} it satisfies the following \\emph{Armijo} condition %\n\\begin{equation}\\label{eq:Armijo}\n    \\funtot(\\R_x(\\alpha \\eta)) \\le \\funtot(x) + m_1 \\alpha \\langle \\grad \\funtot(x), \\eta\\rangle, \\qquad\\text{for $0 < m_1 < 1/2$.}\n\\end{equation}","sha256_utf8":"ef915adf2f06e30edcc90f0ec29ac5a42f371d769483822f5f84d50169eda699"},{"start":58425,"end":58958,"start_line":991,"end_line":991,"raw_tex":"The line search employs a second-order retraction $\\R_x$, \\revisebis{e.g.\\;the exponential map, a projection retraction \\cite{Absil2012ProjectionlikeRO}, or any other second-order retraction \\cite{boumal2022intromanifolds}.\\footnote{\\revisebis{Indeed, in many applications of Riemannian optimization, computing geodesics and the exponential map can be costly and then retractions provide an efficient alternative. For this reason, we consider here second-order retractions \\cite{absil2009optimization,boumal2022intromanifolds}.}}} %\n","sha256_utf8":"a07d85dc76e8382bb4cebd0f18f2a08f5ca08513d8b4504a3307f229f3bf76c8"},{"start":61257,"end":61611,"start_line":1015,"end_line":1016,"raw_tex":"In the following, we will consider a \\emph{backtracking} \\revise{line search} for finding an acceptable stepsize $\\alpha$: the unit stepsize is first tried, and then the search space is reduced geometrically. %\nIn practice, we use exactly \\cite[Alg. A6.3.1]{dennis1996numerical}, which features polynomial interpolation of $\\funman$ in the search space.\n","sha256_utf8":"38fcf8c0a8abf063993d487c1ec3731cb332fa03aedbe0f794e971b2d055f75b"},{"start":62137,"end":62648,"start_line":1037,"end_line":1048,"raw_tex":"\\begin{algorithm}\n    \\caption{$\\manupdate$-Newton\\label{alg:ManUpNewton}}\n    \\begin{algorithmic}[1]\n        \\Require Manifold $\\M$, point $x\\in\\M$\n        \\State Find $d$ in $\\tangentM$ that solves\n        \\begin{equation*}\\tag{Newton equation}\\label{eq:Newtoneq}\n            \\grad F(x) + \\Hess F(x)[d] = 0\n        \\end{equation*}\n        \\State Find $\\alpha$ satisfying the Armijo condition \\eqref{eq:Armijo} with direction $d$\n        \\State \\Return $y = R_x(\\alpha d)$\n    \\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"e036c4af6e54f74427477eaaf8bb0062abde8b38daf799980efe4830e989ad7f"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{th:Newton}\n    Let \\cref{asm:fun} hold and take $\\gamma\\in(0, 1/L)$. Consider the sequence of iterates $(x_k)$ generated by \\cref{alg:generic} equipped with the Riemannian Newton manifold update (\\cref{alg:ManUpNewton}). If $~ \\Hess \\funtot(x_k)$ is positive definite at each step, then all limit points of $(x_k)$ are critical points of $\\funtot$ and share the same functional value.\n\n    Furthermore, assume that the sequence $(y_k)$ admits %\n    a limit point $x^\\star$ such\\;that\n    \\begin{itemize}\n        \\item[i)] $x^\\star\\in\\M$ is a $r$-structured critical point for $(f,g)$  with $r<1/\\gamma$;\n        \\item[ii)] $\\Hess_{\\M} \\funtot(x^\\star)\\succ 0$ and  $\\Hess_{\\M} \\funtot$ is locally Lipschitz around $x^\\star$.\n    \\end{itemize}\n    Then, after some finite time,\n    \\begin{itemize}\n        \\item[a)] the sequence $(x_k)$ lies on $\\M$;\n        \\item[b)] $x_k$ converges to $x^\\star$ quadratically{: for large $k$, there exists $c>0$ such that}\n            \\begin{equation*}\n                \\distM(x_{k+1}, x^\\star) \\le {c ~} \\distM(x_k, x^\\star)^2.\n            \\end{equation*}\n    \\end{itemize}\n\\end{theorem}","units":[{"start":62651,"end":63795,"start_line":1051,"end_line":1068,"raw_tex":"\\begin{theorem}\\label{th:Newton}\n    Let \\cref{asm:fun} hold and take $\\gamma\\in(0, 1/L)$. Consider the sequence of iterates $(x_k)$ generated by \\cref{alg:generic} equipped with the Riemannian Newton manifold update (\\cref{alg:ManUpNewton}). If $~ \\Hess \\funtot(x_k)$ is positive definite at each step, then all limit points of $(x_k)$ are critical points of $\\funtot$ and share the same functional value.\n\n    Furthermore, assume that the sequence $(y_k)$ admits %\n    a limit point $x^\\star$ such\\;that\n    \\begin{itemize}\n        \\item[i)] $x^\\star\\in\\M$ is a $r$-structured critical point for $(f,g)$  with $r<1/\\gamma$;\n        \\item[ii)] $\\Hess_{\\M} \\funtot(x^\\star)\\succ 0$ and  $\\Hess_{\\M} \\funtot$ is locally Lipschitz around $x^\\star$.\n    \\end{itemize}\n    Then, after some finite time,\n    \\begin{itemize}\n        \\item[a)] the sequence $(x_k)$ lies on $\\M$;\n        \\item[b)] $x_k$ converges to $x^\\star$ quadratically{: for large $k$, there exists $c>0$ such that}\n            \\begin{equation*}\n                \\distM(x_{k+1}, x^\\star) \\le {c ~} \\distM(x_k, x^\\star)^2.\n            \\end{equation*}\n    \\end{itemize}\n\\end{theorem}","sha256_utf8":"dbf3100410ae9ede761bfb287eb17ec954b4e55724250c7722aa0e1b16297146"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{theorem}[Proximal gradient points smoothly locate manifolds] \\label{th:PGpointlocatemanifolds}\n    Let $f$ be a $\\C^2$ function on $\\RR^n$ and $g$ a lower semi-continuous function on $\\RR^n$. Suppose that $g$ is both $r$-prox-regular at $\\bar{x}$ and partly-smooth relative to $\\M$ at $\\bar{x}$.\n\n    Take $\\gamma$, $\\bar \\gamma$ such that $0<\\gamma<\\bar \\gamma \\leq 1/r$ and $\\bar{x}=\\prox_{\\bar\\gamma g}(\\bar{y}-\\bar\\gamma \\nabla f(\\bar{y}))$. If\n    \\begin{itemize}\n        \\item[i)] $\\frac{1}{\\gamma}(\\bar{y}-\\bar{x}) - \\nabla f(\\bar{y})  \\in \\ri \\partial g(\\bar{x})$ ~\\revise{(the relative interior of the subdifferential at $\\bar{x}$)};\n        \\item[ii)] either a) $\\gamma$ is sufficiently close to $\\bar{\\gamma}$, or b) $\\bar{y}$ is sufficiently close to $\\bar{x}$;\n    \\end{itemize}\n    then, the proximal gradient $y \\mapsto \\prox_{\\gamma g}(y - \\gamma\\nabla f(y))$ is $\\C^{1}$ and $\\M$-valued near $\\bar{y}$.\n\\end{theorem}\n\n\\begin{proof}\n    Adopting the same reasoning as in \\cite[Sec. 5]{lewis2002active} and \\cite[Sec. 4.1]{daniilidis2006geometrical}, we consider the function\n    \\begin{equation*}\n        \\begin{array}{rl}\n            \\rho : \\bbR^n\\times\\bbR^n & \\to \\bbR                                                 \\\\\n            (x, y)                    & \\mapsto g(x)+\\frac{1}{2\\gamma}\\|x-y+\\gamma\\nabla f(y)\\|^2,\n        \\end{array}\n    \\end{equation*}\n    and denote by $\\rho_y = \\rho(\\cdot,y)$. Computing the proximal gradient $\\prox_{\\gamma g}(y-\\gamma\\nabla f(y))$ can then be seen as minimizing the parametrized function $\\rho_y$.\n\n    \\noindent\\underline{Step 1.} As a first step, we study the minimizers of $\\rho_y$ restricted to $\\M$, for $y$ near\\;$\\bar{y}$. %\n    We consider the parametric manifold optimization problem, for $y$ near $\\bar{y}$:\n    \\begin{equation}\\tag{$ P_\\M(y)$}\n        \\min_{x\\in\\M} \\rho_y(x). \\label{pb:pb_mancstr}\n    \\end{equation}\n    Since $g$ is $\\C^2$-partly-smooth relative to $\\M$ and $f$ is $\\C^2(\\bbR^n)$, $\\rho_y$ is twice continuously differentiable on $\\M$. \\revise{Moreover, the $r$-prox-regularity gives easily (see \\cref{lemma:prox_strong_min}) that $\\rho_{\\bar{y}}$ is lower-bounded by $(\\frac{1}{\\gamma}-r)\\|\\cdot - \\bar{x}\\|^2/2$ on a neighborhood of $\\bar x$ in $\\bbR^n$ and, a fortiori, in $\\M$. From usual rationale (see e.g.\n    \\cite[Chap 4.2, 6.1]{boumal2022intromanifolds}), this implies}\n    \\begin{equation*}\n        \\grad \\rho_{\\bar{y}}(\\bar{x}) = 0 \\qquad \\Hess \\rho_{\\bar{y}}(\\bar{x}) \\succeq \\Big(\\frac{1}{\\gamma}-r\\Big) I \\succ 0,\n    \\end{equation*}\n    \\revise{which are the conditions to apply the implicit functions theorem, as follows.}\n\n    We consider the equation $\\Phi(x, y) = 0$, for $x, y$ near $\\bar{x}, \\bar{y}$, where $\\Phi:\\M\\times\\bbR^n\\to\\tangentBundle$ is defined as $\\Phi(x, y) = \\grad \\rho_y(x)$. This function is continuously differentiable on a neighborhood of $(\\bar{x}, \\bar{y})$, and its differential relative to $\\bar x$ at that point, $\\Hess \\rho_{\\bar{y}}(\\bar{x})$, is invertible. The implicit function theorem thus grants the existence of neighborhoods $\\N_{\\bar{x}}$, $\\N_{\\bar{y}}$ of $\\bar{x}$, $\\bar{y}$ in $\\M$, $\\bbR^n$, and a continuously differentiable function $\\hat{x}:\\N_{\\bar{y}} \\to \\N_{\\bar{x}}$ such that, for any $y$ in $\\N_{\\bar{y}}$, $\\Phi(\\hat{x}(y), y) = \\grad \\rho_y(\\hat{x}(y)) = 0$. Actually, $\\hat{x}(y)$ is a strong minimizer of $\\rho_y$ on $\\M$ for $y$ close enough to $\\bar{y}$. Indeed, the mapping $\\hat{x}$ is continuous on $\\N_{\\bar{y}}$, so that $y \\mapsto \\Hess \\rho_y(\\hat{x}(y))$ is also continuous there and the property $\\Hess \\rho_{\\bar{y}}(\\hat{x}(\\bar{y}))\\succ 0$ extends locally around $\\bar{y}$.\n\n    \\noindent\\underline{Step 2.}  As a second step, we turn to show that the minimizer $\\hat{x}(y)$ of $\\rho_y$ on $\\M$ is actually a strong critical point of $\\rho_y$ in $\\bbR^n$ \\cite[Def. 5.3]{lewis2002active}, and thus the proximal gradient of point $y$. More precisely, we claim that, for $y$ near $\\bar{y}$ and $x=\\hat{x}(y)$, there holds $0\\in\\ri \\partial \\rho_y(x)$, that is\n    \\begin{equation*}\n        \\frac{1}{\\gamma}(y-x) - \\nabla f(y)  \\in \\ri \\partial g(x).\n    \\end{equation*}\n\n    This property holds at $(\\bar{x}, \\bar{y})$ by assumption. By contradiction, assume there exist sequences of points $(y_r)$ with limit $\\bar{y}$, $(x_r) = (\\hat x(y_r))$ with limit $\\bar{x} = \\hat x(\\bar{y})$ and $(h_r)$ of unit norm $\\|h_r\\|=1$ such that for all $r$, $h_r$ separates 0 from $\\partial \\rho_{y_r}(x_r)$:\n    \\begin{equation*}\n        \\inf_{h\\in\\partial \\rho_{y_r}(x_r)} \\langle h_r, h\\rangle \\ge 0.\n    \\end{equation*}\n    Since $(h_r)$ is bounded, a converging subsequence can be extracted from it, let $\\bar{h}$ denote its limit. At the cost of renaming iterates, we assume that $\\lim_{r\\to\\infty} h_r = \\bar{h}$. The above property still holds at the limit $r\\to\\infty$. Indeed, let $\\bar{u}\\in\\partial \\rho_{\\bar{y}}(\\bar{x})$. Since $g$ is partly smooth, the mapping $(x, y)\\in \\N_{\\bar{x}}\\times\\N_{\\bar{y}} \\mapsto\\partial \\rho_y(x) = \\partial g(x) + \\frac{1}{\\gamma}(x-y)$ is continuous. Therefore, there exists a sequence $(u_r)$ such that $u_r\\in\\partial\\rho_{y_r}(x_r)$ and $\\lim_{r\\to\\infty} u_r=\\bar{u}$. We have for all $r$: $\\langle u_r, h_r\\rangle \\ge 0$, which yields at the limit $\\langle\\bar{u}, \\bar{h}\\rangle\\ge 0$. Thus $\\bar{h}$ separates $0$ from $\\partial \\rho_{\\bar{y}}(\\bar{x})$, which contradicts our assumption.\n\n    \\noindent\\underline{Conclusion.} We thus have a continuously differentiable function $\\hat{x}$ defined on a neighborhood of $\\bar{y}$ such that i) $\\hat{x}(\\bar{y}) = \\bar{x}$, ii) $\\hat{x}(y)$ is a strong minimizer of $\\rho_y$ on $\\M$, iii) $0\\in\\ri\\partial \\rho_y(\\hat{x}(y))$.\n\n    This last point tells us that $(y-\\hat x(y))/\\gamma -\\nabla f(y) \\in \\partial g(\\hat x(y))$. The characterization of proximity by the optimality condition (\\cref{th:proxex}) gives\nthat $\\hat{x}(y)=\\prox_{\\gamma g}(y-\\gamma \\nabla f(y))$ for $y$ close enough to $\\bar{y}$.\n\\end{proof}\n\n\\begin{theorem}%\n    \\label{th:identiflocalcv_sperlin}\n    Let \\cref{asm:fun} hold and take $\\gamma\\in(0, 1/L)$\\revise{, where $L$ is the Lipschitz constant for $\\nabla f$}. Assume that \\cref{alg:generic} generates a sequence $(y_k)$ which admits at least one limit point $\\bar{x}$ such that:\n    \\begin{itemize}\n        \\item[i)] $\\bar{x}\\in\\M$ is a $r$-structured critical point for $(f,g)$ with $r<1/\\gamma$;\n        \\item[ii)] $\\manupdate_{\\M}$ has superlinear convergence rate of order $1+\\theta\\in(1,2)$ %\n        near $\\bar{x}$ in $\\M$.\n    \\end{itemize}\n    Then, after some finite time:\n    \\begin{itemize}\n        \\item[a)] the full sequence $(x_k)$ lies on $\\M$;\n        \\item[b)] $x_k$ converges to $\\bar{x}$ superlinearly with the same order as $\\manupdate$:\n              \\begin{equation}\\label{eq:super}\n                  \\distM(x_{k+1},\\bar{x}) \\le c ~ \\distM(x_k, \\bar{x})^{1+\\theta} \\qquad \\text{ for some $c>0$.}\n              \\end{equation}\n    \\end{itemize}\n\\end{theorem}\n\n\\begin{proof}\n    Let us note $ \\PGop(y) = \\prox_{\\gamma g}(y - \\gamma \\nabla f(y)) $ for $y\\in\\mathbb{R}^n$.\n    The part i) of the assumptions enables us to show the existence of some neighborhood of $\\bar{x}$ on which the proximal gradient operation is $\\M$-valued and Lipschitz continuous. More precisely, \n    \\cref{th:PGpointlocatemanifolds} \n    implies that there exists $\\delta_1>0$ and $C>0$ such that, \n    \\[%\n      \\PGop(y)\\in\\M \\label{proof:inM} \\quad \\text{ and } \\quad \\|\\PGop(y)- \\PGop(\\bar{x})\\| \\le C \\|y-\\bar{x}\\| \\qquad \\text{for all $y$ in $\\B(\\bar{x}, \\delta_1)$}.\n    \\]%\n    Now, if $y$ belongs to $\\M$, we get that there exists $\\varepsilon_1>0$ such that for any $y$ in $\\B_{\\M}(\\bar{x}, \\varepsilon_1)$, $\\PGop(y)\\in\\M$; but in addition, the Euclidean Lipschitz continuity can be translated into a Riemannian one (see \\cref{lemma:Euclidean_to_Riemannian}) since   for some $\\delta>0$,\n    \\begin{multline}\\label{eq:EuctoRie}\n        (1-\\delta)\\distM(\\PGop(y), \\bar{x}) = (1-\\delta)\\distM(\\PGop(y), \\PGop(\\bar{x}))  \\le \\|\\PGop(y)-\\PGop(\\bar{x})\\| \\\\\n        \\le C \\|y-\\bar{x}\\| \\le C (1+\\delta)\\distM(y, \\bar{x})\n    \\end{multline}\n    Hence, there is $q_1>0$ such that for any $y$ in $\\B_{\\M}(\\bar{x}, \\varepsilon_1)$\n    \\begin{equation}\\label{proof:PGopEstimate}\n        \\distM(\\PGop(y), \\bar{x}) =  \\distM(\\PGop(y), \\PGop(\\bar{x}))   \\le  q_1 ~ \\distM(y, \\bar{x}).\n    \\end{equation}\n\n    Then, the part ii) of the assumptions gives us the existence of $\\varepsilon_2$, $q_2>0$ and $\\theta\\in(0,1)$ such that, for any $x$ in $\\B_{\\M}(\\bar{x}, \\varepsilon_2)$,\n    \\begin{equation}\\label{proof:manupdateEstimate}\n        \\distM(\\manupdate_{\\M}(x), \\bar{x}) \\le q_2 ~ \\distM(x, \\bar{x})^{1+\\theta}.\n    \\end{equation}\n    Let us now take any $x\\in\\B_{\\M}(\\bar{x}, \\varepsilon)$ where $\\varepsilon=\\min(\\varepsilon_1,\\varepsilon_2,(\\varepsilon_1 / q_2)^{\\frac{1}{1+\\theta}},(q_2 q_1)^{-\\frac{1}{\\theta}})$: \\\\\n    \\noindent (\\emph{i}) Since $x\\in\\B_{\\M}(\\bar{x}, \\varepsilon_2)$, the manifold update \\eqref{proof:manupdateEstimate} yields\n    \\begin{equation*}\n        \\distM(\\manupdate_{\\M}(x), \\bar{x}) \\le q_2 ~ \\distM(x, \\bar{x})^{1+\\theta} \\le q_2 ~ \\varepsilon^{1+\\theta} \\le \\varepsilon_1.\n    \\end{equation*}\n    \\noindent (\\emph{ii}) As $\\manupdate_{\\M}(x)$ lies in $\\B_{\\M}(\\bar{x}, \\varepsilon_1)$, the proximal gradient update \\eqref{proof:PGopEstimate} applied to $y = \\manupdate_{\\M}(x)$ gives\n    \\begin{multline}\\label{eq:superlinear}\n        \\distM(\\PGop(\\manupdate_{\\M}(x)), \\bar{x}) \\le    q_1 \\distM(\\manupdate_{\\M}(x), \\bar{x}) \\\\ \\le q_1 q_2 ~ \\distM(x, \\bar{x})^{1+\\theta} \\le q_1 q_2 ~ \\varepsilon^\\theta \\distM(x, \\bar{x}).\n    \\end{multline}\n    Since $q_2 q_1 \\varepsilon^\\theta \\le 1$  by construction, this gives %\n    \\begin{equation}\\label{eq:manstable}\n        \\distM(\\PGop(\\manupdate_{\\M}(x)), \\bar{x}) \\le \\distM(x, \\bar{x}) \\qquad \\text{for any $x\\in\\B_{\\M}(\\bar{x}, \\varepsilon)$}.\n    \\end{equation}\n    We have thus proved the existence of a neighborhood $\\B_\\M(\\bar{x}, \\varepsilon)$ of $\\bar{x}$ in $\\M$ which is stable for an iteration of \\cref{alg:generic} and over which one iteration has a superlinear improvement of order $1+\\theta$ (by \\eqref{eq:superlinear}).\n\n\n    Finally, since $\\bar{x}$ is a limit point of $(y_k)$, there exists $K<\\infty$ such that $y_K\\in\\B(\\bar{x},(1-\\delta)\\varepsilon/C)$. Besides, \\eqref{eq:EuctoRie} tells us that $ \\distM(\\PGop(y_K), \\bar{x})\\le \\varepsilon$ and thus $x_k$ and $y_k$ belong to $\\B_{\\M}(\\bar{x}, \\varepsilon)$ for all $k>K$ by \\eqref{eq:manstable}. %\n    We conclude that $x_{k+1}=\\PGop(y_k)\\in\\M$ for all $k\\geq K$, and, using \\eqref{eq:superlinear}, that we have \\eqref{eq:super} with $c= q_1 q_2$, for all $k>K$. \n\\end{proof}\n\n\\begin{lemma}\\label{lemma:LS_exists_valid_stepsizes}\n    Let \\cref{asm:fun} hold and consider a manifold $\\M$ equipped with a retraction $\\R$ and a pair $(x, \\eta)\\in\\tangentBundle$. If $\\funtot$ is differentiable on $\\M$ at $x$, $\\langle \\grad \\funtot (x), \\eta\\rangle <0$, and $m_1<1$, then there exists $\\hat\\alpha>0$ such that any step size $\\alpha\\in(0, \\hat\\alpha)$ is acceptable by the Armijo rule \\eqref{eq:Armijo}.\n\\end{lemma}\n\n\\begin{proof}\n    We adapt a part of the proof of  \\cite[Th. 6.3.2]{dennis1996numerical} for the Armijo rule and the Riemannian setting. Since $m_1<1/2$, for any $\\alpha$ sufficiently small there holds\n    \\begin{equation*}\n        \\funtot\\circ \\R_x(\\alpha \\eta) \\le \\funtot\\circ\\R_x(0) + m_1 \\D\\left(\\funtot\\circ\\R_x \\right)(0)[\\alpha\\eta] = \\funtot(x) + m_1 \\alpha \\langle \\grad \\funtot(x), \\eta\\rangle.\n    \\end{equation*}\n    Since $\\funtot$ is bounded below, there exists a smallest $\\hat{\\alpha}$ such that $\\funtot(\\R_x(\\hat\\alpha \\eta)) = \\funtot(x) + m_1 \\hat\\alpha \\langle \\grad \\funtot(x), \\eta\\rangle$. Thus all stepsizes in $(0, \\hat\\alpha)$ are acceptable by\\;\\eqref{eq:Armijo}.\n\\end{proof}\n\nIn addition, a \\revise{line search} performed near a minimizer with a Newton direction\nshould accept the unit stepsize, so that a full step may be taken. This is the case when the Riemannian Hessian around this minimizer is positive definite as stated by the next lemma, \\revise{which is a direct corollary of \\cref{th:unitstepsuperlinearcv}.}\n\\begin{lemma}\\label{lemma:LS_unit_stepsize}\n    Let \\cref{asm:fun} hold and consider a manifold $\\M$ equipped with a retraction $\\R$, a point $x^\\star\\in\\M$ and a pair $(x, \\eta)\\in\\tangentBundle$. Assume that $\\funtot$ is twice differentiable on $\\M$ near a strong local minimizer  $x^\\star$  on $\\M$, that is $\\Hess \\funman(x^\\star)$ is positive definite.\n    If the direction $\\eta$ brings a superlinear improvement towards $x^\\star$, that is $\\distM(\\R_x(\\eta), x^\\star) = o(\\distM(x, x^\\star))$ as $x\\to x^\\star$, and $0<m_1<1/2$, then $\\eta$ is acceptable by the Armijo rule\\;\\eqref{eq:Armijo} with unit stepsize $\\alpha=1$.\n\\end{lemma}\n\n\\begin{proof}\n    As the Riemannian Hessian is assumed to be positive definite, Newton's direction is a descent direction:\n    \\begin{equation*}\n        \\langle \\grad \\funtot(x_k), d_k \\rangle = -\\langle \\grad \\funtot(x_k), \\Hess \\funtot(x_k)^{-1} \\grad \\funtot(x_k) \\rangle < 0.\n    \\end{equation*}\n    The Riemannian Newton manifold step is therefore well-defined, and the \\revise{line search} terminates by \\cref{lemma:LS_exists_valid_stepsizes}, so that the manifold update is well-defined and provides descent \\revise{($F(y_{k})\\le F(x_{k})$)}. \n\n\n    \\revise{Now, since the proximal gradient update provides a descent (see \\cite[Lem. 10.4]{beck2017first}),\n    \\begin{equation}\\label{eq:proofglobalalg_descent}\n        F(x_{k+1}) \\le F(y_k) - \\frac{1-\\gamma L}{2\\gamma} \\| x_{k+1}-y_k \\|^2 \\le F(x_k) - \\frac{1-\\gamma L}{2\\gamma} \\| x_{k+1}-y_k \\|^2.\n    \\end{equation}\n    The sequence $(F(x_k))$ is thus non-increasing and lower-bounded, therefore it converges.\n    Besides, any accumulation point of $(x_k)$ is a critical point of $F$. Indeed, summing equation \\eqref{eq:proofglobalalg_descent} for $k=1, \\ldots, n$ yields:\n    \\begin{equation*}\n        \\frac{1-\\gamma L}{2\\gamma} \\sum_{k=1}^n \\| x_{k+1}-y_k \\|^2 \\le F(x_1) - F(x_{n+1}) \\le F(x_1)-\\inf F < +\\infty.\n    \\end{equation*}\n    Since $ \\dist (0, \\partial F(x_{k+1})) \\le \\frac{L\\gamma + 1}{\\gamma}\\|x_{k+1}-y_k\\|$ (see e.g. the proof of\\;\\cite[Prop.\\;13]{bolte2015error}), we have that \n    $\\dist (0, \\partial F(x_{k+1}))$\n    converges to $0$. The outer-semi continuity of the limiting subdifferential then yields criticality of accumulation points.}\n\n\n\n\n\n    Now we apply the local convergence of Riemannian Newton \\cite[Th. 6.3.2]{absil2009optimization}:\n    assumption ii) ensures that\n    the Riemannian Newton direction $d$ computed in step\\;1 of\\;\\cref{alg:ManUpNewton} provides a quadratic improvement %\n    on a neighborhood of $x^\\star$ on\\;$\\M$. Moreover, the \\revise{line search} returns the unit-stepsize after some finite time: $\\alpha=1$ is tried first, and is acceptable for directions providing superlinear improvement by \\cref{lemma:LS_unit_stepsize}. Thus the whole Riemannian Newton update provides quadratic improvement after some finite time. Using this and assumption i), \\cref{th:identiflocalcv_sperlin} applies and yields the results.\n\\end{proof}\n\n\\begin{lemma}\n    \\label{th:proxex}\n    Consider a function $\\funns:\\bbR^n\\to\\bbR$, a pair of points $\\bar{x}, \\bar{y}$ and a \\revise{step length} $\\bar{\\gamma}>0$ such that $\\bar{x}=\\prox_{\\bar\\gamma \\funns}(\\bar{y})$ and $\\funns$ is $r$ prox-regular at $\\bar{x}$ for subgradient $\\bar{v} \\triangleq (\\bar{y}-\\bar{x}) / \\bar{\\gamma}$.\n\n    Then, for any $\\gamma\\in(0, \\min(1/r, \\bar\\gamma))$, there exists a neighborhood $\\N_{\\bar{y}}$ of $\\bar{y}$ over which $\\prox_{\\gamma \\funns}$ is single-valued and $(1-\\gamma r)^{-1}$-Lipschitz continuous. Furthermore, there holds\n    \\begin{equation*}\n        x = \\prox_{\\gamma \\funns}(y) \\Leftrightarrow (y-x)/\\gamma \\in\\partial \\funns(x)\n    \\end{equation*}\n    for $y\\in\\N_{\\bar{y}}$ and $x$ near $\\bar{x}$ in the sense $\\|x-\\bar{x}\\|<\\varepsilon$, $|\\funns(x)-\\funns(\\bar{x})|<\\varepsilon$ and $\\|(y-x)/\\gamma - \\bar{v}\\|<\\varepsilon$. %\n\\end{lemma}\n\n\\begin{proof}\n    One can easily check that prox-regularity of $\\funns$ at $\\bar{x}$ for subgradient $\\bar{v}$ is equivalent to prox-regularity of function $\\tilde{\\funns}$ around $0$ for subgradient $0$, with $\\tilde{\\funns} = \\funns(\\cdot + \\bar{x}) - \\langle \\bar{v}, \\cdot\\rangle - \\funns(\\bar{x})$ and a change of variable $\\tilde{x} = x-\\bar{x}$. Similarly, $\\bar{x}=\\prox_{\\bar\\gamma \\funns}(\\bar{y})$ is characterized by its global optimality~condition\n    \\begin{equation*}\n        \\funns(x) + \\frac{1}{2\\bar\\gamma}\\|x-\\bar{y}\\| > \\funns(\\bar{x}) + \\frac{1}{2\\bar\\gamma}\\|\\bar{x}-\\bar{y}\\|^2 \\qquad \\text{ for all } x\\neq\\bar{x},\n    \\end{equation*}\n    which we may write as\n    \\begin{equation*}\n        \\funns(x) > \\funns(\\bar{x}) + \\langle \\bar{v}, x-\\bar{x} \\rangle - \\frac{1}{2\\bar\\gamma}\\|x - \\bar{x}\\|^2 \\qquad \\text{ for all } x\\neq\\bar{x}.\n    \\end{equation*}\n    Under that same change of variables, since $\\tilde{\\funns}(0) = 0$, this optimality condition rewrites as\n    \\begin{equation*}\n        \\tilde{\\funns}(\\tilde{x}) > - \\frac{1}{2\\bar\\gamma}\\|\\tilde{x}\\|^2 \\qquad \\text{ for all } \\tilde{x}\\neq 0.\n    \\end{equation*}\n    We may thus apply Theorem 4.4 from \\cite{poliquin1996prox} to get the claimed result on $\\tilde{\\funns}$, which transfers back to $\\funns$ as our change of function and variable is bijective. We thus obtain that for $\\gamma\\in(0, \\min(1/r, \\bar\\gamma))$, on a neighborhood $\\N_{\\bar{y}}$ of $\\bar{y}$, $\\prox_{\\gamma \\funns}$ is single-valued, $(1-\\gamma r)^{-1}$-Lipschitz continuous and $\\prox_{\\gamma \\funns}(y)=[I+\\gamma T]^{-1}(y)$, where $T$ denotes the $\\funns$-attentive $\\varepsilon$-localization of $\\partial \\funns$ at $\\bar{x}$. Taking $y$ near $\\bar{y}$ and $x$ near $\\bar{x}$ such that $\\|x-\\bar{x}\\|<\\varepsilon$, $|\\funns(x)-\\funns(\\bar{x})|<\\varepsilon$ and $\\|(y-x)/\\gamma - \\bar{v}\\|<\\varepsilon$ allows to identify the localization of $\\partial \\funns(x)$ with $\\partial \\funns(x)$, so that\n    \\begin{equation*}\n        \\frac{y-x}{\\gamma}\\in \\partial \\funns(x)\\Leftrightarrow \\frac{y-x}{\\gamma}\\in T(x)\\Leftrightarrow (I+\\gamma T)(x)=y\\Leftrightarrow x=\\prox_{\\gamma \\funns}(y).\n    \\end{equation*}\n    Note that the proof of \\cite[Th. 4.4]{poliquin1996prox} includes a minor error relative to the Lipschitz constant computation, we report here a corrected value.\n\\end{proof}\n\n\\begin{lemma}\\label{lemma:prox_strong_min}\n    Let $f$ and $g$ denote two functions and $\\bar{x}$, $\\bar{y}$ two points such that $f$ is differentiable at $\\bar{y}$ and $g$ is $r$-prox-regular at $\\bar{x}$ for subgradient $\\frac{1}{\\gamma}(\\bar{y}-\\bar{x}) - \\nabla f(\\bar{y}) \\in \\partial g(\\bar{x})$ with $\\gamma \\in (0, 1/r)$.\n    Then, the function $\\rho_{\\bar{y}} : x\\mapsto g(x) + \\frac{1}{2\\gamma}\\|\\bar{y}-\\gamma\\nabla f(\\bar{y}) - x\\|^2$ satisfies %\n    \\begin{equation*}\n        \\rho_{\\bar{y}}(x) \\ge \\rho_{\\bar{y}}(\\bar{x}) + \\frac{1}{2} \\left( \\frac{1}{\\gamma} - r \\right) \\| x-\\bar{x} \\|^2, \\qquad \\text{for all $x$ near $\\bar{x}$.}\n    \\end{equation*}\n\\end{lemma}\n\n\\begin{proof}\n    Prox-regularity of $g$ at $\\bar{x}$ with subgradient $\\frac{1}{\\gamma} (\\bar{y}-\\gamma \\nabla f(\\bar{y}) - \\bar{x}) \\in \\partial g(\\bar{x})$ writes\n    \\begin{equation*}\n        g(x) \\ge g(\\bar{x}) + \\frac{1}{\\gamma}\\langle \\bar{y}-\\gamma \\nabla f(\\bar{y}) - \\bar{x} , x-\\bar{x} \\rangle - \\frac{r}{2} \\| x-\\bar{x} \\|^2.\n    \\end{equation*}\n    The identity $ 2\\langle b-a, c-a\\rangle = \\|b-a\\|^2 + \\|c-a\\|^2 -   \\|b-c\\|^2 $ applied to the previous scalar product yields:\n    \\begin{equation*}\n        g(x) \\ge g(\\bar{x}) + \\frac{1}{2\\gamma} \\| \\bar{y}-\\gamma \\nabla f(\\bar{y}) - \\bar{x} \\|^2 + \\frac{1}{2\\gamma} \\| x-\\bar{x} \\|^2 - \\frac{1}{2\\gamma} \\|\\bar{y}-\\gamma \\nabla f(\\bar{y}) - x \\|^2 - \\frac{r}{2} \\| x-\\bar{x} \\|^2,\n    \\end{equation*}\n    which rewrites\n    \\begin{equation*}\n        \\underbrace{ g(x) + \\frac{1}{2\\gamma} \\|\\bar{y}-\\gamma \\nabla f(\\bar{y}) - x \\|^2 }_{=\\rho_{\\bar{y}}(x) } \\ge  \\underbrace{  g(\\bar{x}) + \\frac{1}{2\\gamma} \\| \\bar{y}-\\gamma \\nabla f(\\bar{y}) - \\bar{x} \\|^2   }_{=\\rho_{\\bar{y}}(\\bar{x}) } + \\frac{1}{2} \\left( \\frac{1}{\\gamma} - r \\right) \\| x-\\bar{x} \\|^2,\n    \\end{equation*}\n    \\vspace*{-1ex}\nwhich is the claimed inequality.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:InvRetractionApproxLog}\n    Consider a point $\\bar{x}$ of a Riemannian manifold $\\M$, equipped with a retraction $\\R$ such that $\\R_{\\bar{x}}$ is $\\C^2$. For any $\\varepsilon>0$, there exists a neighborhood $\\U$ of $\\bar{x}$ in $\\M$ such that \n    \\begin{equation*}\n        (1-\\varepsilon)\\distM(x, \\bar{x}) \\le \\|\\R^{-1}_{\\bar{x}}(x)\\| \\le (1+\\varepsilon)\\distM(x, \\bar{x}) \\qquad \\text{ for all $x\\in\\U$.}\n    \\end{equation*}\n    where $\\R^{-1}_{\\bar{x}}:\\M\\to\\tangent{\\bar{x}}{\\M}$ is the smooth inverse of $\\R_{\\bar{x}}$ defined locally around $\\bar{x}$.\n\\end{lemma}\n\n\\begin{proof}\n    The retraction at $\\bar{x}$ can be inverted locally around $0$. Indeed, as $\\D\\R_{\\bar{x}}(0_{\\tangent{\\bar{x}}{\\M}})=I$ is invertible and $\\R_{\\bar{x}}$ is $\\C^2$, the implicit function theorem provides the existence of a $\\C^2$ inverse function $\\R^{-1}_{\\bar{x}}:\\M\\to\\tangent{\\bar{x}}{\\M}$ defined locally around $\\bar{x}$. Furthermore, one shows by differentiating the relation $\\R_{\\bar{x}}\\circ\\R_{\\bar{x}}^{-1}$ that the differential of $\\R^{-1}_{\\bar{x}}$ at $\\bar{x}$ is the identity.\n\n    We consider the function $f:\\M\\to\\RR$ defined by $f(x)=\\|\\log_{\\bar{x}}(x)\\|-\\|\\R_{\\bar{x}}^{-1}(x)\\|$. Clearly $f(\\bar{x})=0$, and $\\D f(\\bar{x})=0$ as the differentials of both $\\R_{\\bar{x}}^{-1}$ and logarithm at $\\bar{x}$ are the identity. In local coordinates $\\hat{x}=\\log_{\\bar{x}}{x}$ around $\\bar{x}$, $f$ is represented by the function $\\hat{f}=f\\circ\\exp_{\\bar{x}}: \\tangent{\\bar{x}}{\\M}\\to\\RR$. As $\\hat{f}(\\hat{\\bar{x}})=0$, $\\D \\hat{f}(\\hat{\\bar{x}})=0$ and $\\hat{f}$ is $\\C^2$, there exists some $C>0$ such that\n    \\begin{equation*}\n        -C \\|\\hat{x} - \\hat{\\bar{x}}\\|^2 \\le \\hat{f}(\\hat{x}) \\le C \\|\\hat{x} - \\hat{\\bar{x}}\\|^2 \\qquad\\text{in a neighborhood $\\hat{\\U}$ of $\\hat{\\bar{x}}$,}.\n    \\end{equation*}\n    For any $\\varepsilon>0$, by taking a small enough neighborhood $\\hat{\\U}'\\subset\\hat{\\U}$, there holds\n    \\begin{equation*}\n        -\\varepsilon \\|\\hat{x} - \\hat{\\bar{x}}\\| \\le \\hat{f}(\\hat{x}) \\le \\varepsilon \\|\\hat{x} - \\hat{\\bar{x}}\\|.\n    \\end{equation*}\n    Thus for all $x$ in $\\U = \\R_{\\bar{x}}(\\hat{\\U}')$,\n    \\begin{equation*}\n        -\\varepsilon \\|\\log_{\\bar{x}}(x)\\| \\le \\|\\log_{\\bar{x}}(x)\\|-\\|\\R_{\\bar{x}}^{-1}(x)\\| \\le \\varepsilon \\|\\log_{\\bar{x}}(x)\\|,\n    \\end{equation*}\n    as $\\hat{x}=\\log_{\\bar{x}}(x)$, $\\hat{\\bar{x}}=0$. We conclude with %\n    $\\distM(x, \\bar{x})=\\|\\hat{x} - \\hat{\\bar{x}}\\| = \\|\\log_{\\bar{x}}(x)\\|$.\n\\end{proof}\n\n\\begin{proposition}\\label{prop:tangretraction}\n    Consider a $p$-dimensional $\\C^k$-submanifold $\\M$ of $\\bbR^n$ around a point $\\bar{x}\\in\\M$. The mapping $\\R : \\tangentBundle\\to\\M$, defined for $(x, \\eta)\\in\\tangentBundle$ near $(\\bar{x}, 0)$ by $\\proj_x(\\R(x, \\eta)) = \\eta$ defines a second-order retraction near $(\\bar{x}, 0)$. The point-wise retraction, defined as $\\R_x = \\R(x, \\cdot)$, is locally invertible with inverse $\\R_{x}^{-1}=\\proj_{x}$.\n\\end{proposition}\n\n\\begin{proof}\n    Let $\\Psi : \\bbR^n\\to\\bbR^{n-p}$ denote a $\\C^k$ function defining $\\M$ around $\\bar{x}$: for all $x$ close enough to $\\bar{x}$, there holds $x\\in\\M \\Leftrightarrow \\Psi(x)=0$, and $\\D\\Psi(x)$ is surjective. Consider the equation $\\Phi(x, \\eta_t, \\eta_n) = 0$ around $(\\bar{x}, 0, 0)$, with\n    \\begin{equation*}\n        \\begin{array}{rl}\n            \\Phi : \\left\\{ x, \\eta_t, \\eta_n : x\\in\\M, \\eta_t\\in\\tangentM, \\eta_n\\in\\normalM \\right\\} & \\to \\bbR                       \\\\\n            x, \\eta_t, \\eta_n                                                                         & \\mapsto \\Psi(x+\\eta_t+\\eta_n).\n        \\end{array}\n    \\end{equation*}\n    The partial differential $\\D_{\\eta_n}\\Phi(\\bar{x}, 0, 0)$ is, for $\\xi_n\\in\\normalM[\\bar{x}]$,\n    \\begin{equation*}\n        \\D_{\\eta_n}\\Phi(\\bar{x}, 0, 0)[\\xi_n] = \\D\\Psi(\\bar{x})[\\xi_n].\n    \\end{equation*}\n    Since $\\bar{x}\\in\\M$, $\\D_{\\eta_n}\\Phi(\\bar{x}, 0, 0)$ is surjective from $\\normalM[\\bar{x}]$ to $\\bbR^{n-p}$ so its a bijection. The implicit function theorem provides the existence of neighborhoods $\\N_{\\bar{x}}^1\\subset\\M$, $\\N_{0}^2\\subset\\cup_{x\\in\\M} \\tangentM$, $\\N_{0}^3\\subset\\cup_{x\\in\\M}\\normalM$ and a unique $\\C^k$ function $\\eta_n : \\N_{\\bar{x}}^1 \\times \\N_0^2 \\to \\N_0^3$ such that, for all $x\\in\\N_{\\bar{x}}^1$, $\\eta_t\\in\\N_0^2$ and $\\eta_n\\in\\N_0^3$, $\\eta_n(\\bar{x}, 0) = 0$ and\n    \\begin{equation*}\n        \\Phi(x, \\eta_t, \\eta_n(x, \\eta_t)) = 0 \\Leftrightarrow x+\\eta_t+\\eta_n(x, \\eta_t) \\in \\M.\n    \\end{equation*}\n    It also provides an expression for the partial derivative of $\\eta_n$ at $(x, 0)$ along $\\eta_t$: for $\\xi_t\\in\\tangentM$,\n    \\begin{equation*}\n        \\D_{\\eta_t} \\eta_n(x, 0)[\\xi_t] = - \\left[ \\D_{\\eta_n}\\Phi(x, 0, 0) \\right]^{-1} \\D_{\\eta_t} \\Phi(x, 0, 0)[\\xi_t].\n    \\end{equation*}\n    As noted before, $\\D_{\\eta_n}\\Phi(x, 0, 0)$ is bijective since $x\\in\\M$. Besides, $\\D_{\\eta_t} \\Phi(x, 0, 0) = \\D\\Phi(x)[\\xi_t]=0$ since $\\tangentM$ identifies as the kernel of $\\D\\Phi(x)$. Thus $\\D_{\\eta_t} \\eta_n(x, 0) = 0$.\n\n    Now, define a map $\\R:\\N_{\\bar{x}}^1\\times\\N_0^2 \\to \\M$ by $\\R(x, \\eta_t) = x + \\eta_t + \\eta_n(x, \\eta_t)$. This map has degree of smoothness $\\C^k$ since $\\eta_n$ is $\\C^k$, satisfies $\\R(x, 0) = x$ since $\\eta_n(x, 0)=0$ and satisfies $\\D_{\\eta_t}\\eta_n(x, 0) = I + \\D_{\\eta_t}(x, 0) = I$. Thus $\\R$ defines a retraction on a neighborhood of $(\\bar{x}, 0)$.\n\n    We turn to show the second-order property of $\\R$. Consider the smooth curve $\\smoothcurve$ defined as $c(t) = \\R(x, t\\eta)$ for some $x\\in\\N_{\\bar{x}}^1$, $\\eta_t\\in\\tangentM\\cap\\N_0^2$. It's first derivative writes\n    \\begin{equation*}\n        c'(t) = \\eta + \\D_{\\eta_t}\\eta_n(x, t\\eta)[\\eta] = \\eta.\n    \\end{equation*}\n    The acceleration of the curve $c$ is obtained by computing the derivative of $c'(\\cdot)$ in the ambient space and then projecting onto $\\tangentM$. Thus $c''(t)=0$ and in particular, $c''(0)=0$ which makes $\\R$ a second-order retraction.\n\\end{proof}\n\n\\begin{lemma}\\label{lemma:Euclidean_to_Riemannian}\n    Consider a point $\\bar{x}$ of a Riemannian manifold $\\M$. For any $\\varepsilon>0$, there exists a neighborhood $\\U$ of $\\bar{x}$ in $\\M$ such that, for all $x\\in\\U$,\n    \\begin{equation*}\n        (1-\\varepsilon)\\distM(x, \\bar{x}) \\le \\|x-\\bar{x}\\| \\le (1+\\varepsilon)\\distM(x, \\bar{x}),\n    \\end{equation*}\n    where $\\|x-\\bar{x}\\|$ is the Euclidean distance in the ambient space.\n\\end{lemma}\n\n\\begin{proof}\n    Let $\\bar{x}$, $x$ denote two close points on $\\M$. Consider the tangential retraction introduced in \\cref{prop:tangretraction}. As a retraction, it satisfies:\n    \\begin{equation*}\n        \\R_{\\bar{x}}(\\eta) = \\R_{\\bar{x}}(0) + \\D\\R_{\\bar{x}}(0)[\\eta] + \\mathcal O(\\|\\eta\\|^2) = \\bar{x} + \\mathcal O(\\|\\eta\\|^2).\n    \\end{equation*}\n    Taking $x=\\R_{\\bar{x}}(\\eta)$ allows to write $x = \\bar{x} + \\mathcal O(\\|\\R_{\\bar{x}}^{-1}(x)\\|^2)$, so that for any small $\\varepsilon_1>0$, there exists a small enough neighborhood $\\U_1\\subset\\U$ of $\\bar{x}$ in $\\M$ such that\n    \\begin{equation*}\n        (1-\\varepsilon_1) \\|\\R_{\\bar{x}}^{-1}(x)\\| \\le \\|x-\\bar{x}\\| \\le (1+\\varepsilon_1)\\|\\R_{\\bar{x}}^{-1}(x)\\|.\n    \\end{equation*}\n    By \\cref{lem:InvRetractionApproxLog}, for $\\varepsilon_2>0$ small enough, there exists a neighborhood $\\U_2\\subset\\U$ of $\\bar{x}$ such that,\n    \\begin{equation*}\n        (1-\\varepsilon_2)\\distM(x, \\bar{x}) \\le \\|\\R^{-1}_{\\bar{x}}(x)\\| \\le (1+\\varepsilon_2)\\distM(x, \\bar{x}).\n    \\end{equation*}\n    With $\\varepsilon_1$, $\\varepsilon_2$ such that $1-\\varepsilon=(1-\\varepsilon_1)(1-\\varepsilon_2)$, we combine the two estimates to conclude.%\n\\end{proof}\n\n\\begin{theorem}[Soundness of the Riemannian \\revise{line search}]\\label{th:unitstepsuperlinearcv}\n    Consider a manifold $\\M$ equipped with a retraction $\\R$ and a twice differentiable function $\\funman:\\M\\to\\RR$ that admits a strong local minimizer $x^\\star$, that is, a point such that $\\Hess \\funman(x^\\star)$ is positive definite. If $x$ is close to $x^\\star$, $\\eta$ brings a superlinear improvement towards $x^\\star$, that is $\\distM(\\R_x(\\eta), x^\\star) = o(\\distM(x, x^\\star))$ as $x\\to x^\\star$, and $0<m_1<1/2$, then $\\eta$ is acceptable by the Armijo rule\\;\\eqref{eq:Armijo} with unit stepsize $\\alpha=1$.\n\\end{theorem}\n\n\\begin{proof}\n\n    Let $x, \\eta\\in\\tangentBundle$ denote a pair such that $x$ is close to $x^\\star$ and $\\distM(\\R_x(\\eta), x^\\star) = o(\\distM(x, x^\\star))$. For convenience, let $x_+ = \\R_x(\\eta)$ denote the next point.\n\n    Following \\cite{absil2009optimization} (see e.g. the proof of Th. 6.3.2), we work in local coordinates around $x^\\star$, representing any point $x\\in\\M$ by $\\widehat{x} = \\log_{x^\\star}(x)$ and any tangent vector $\\eta\\in\\tangentM$ by $\\widehat{\\eta}_x = \\D \\log_{x^\\star}(x)[\\eta]$. The function $\\funman$ is represented by $\\widehat{\\funman} = \\funman\\circ\\exp_{x^\\star}:\\tangentM[x^\\star]\\to\\bbR$. Defining the coordinates via the logarithm grants the useful property that the Riemannian distance of any two points $x, y\\in\\M$ matches the euclidean distance between their representatives: $\\distM(x, y) = \\|\\widehat{x}-\\widehat{y}\\|$. Besides, there holds\n    \\begin{equation}\\label{eq:gradHess_to_localcoords}\n        \\D \\funman(x)[\\eta]=\\D \\widehat{\\funman}(\\widehat{x})[\\widehat{\\eta}] \\quad \\text{ and } \\quad \\Hess \\funman(x)[\\eta, \\eta]=\\D \\widehat{\\funman}(\\widehat{x})[\\widehat{\\eta}, \\widehat{\\eta}].\n    \\end{equation}\n    Indeed, $\\D \\funman(x)[\\eta] = (\\funman\\circ\\geocurve)'(0)$ and $\\Hess \\funman(x)[\\eta, \\eta] = (\\funman\\circ\\geocurve)''(0)$, where $\\geocurve$ denotes the geodesic curve defined by $\\widehat{\\geocurve}(t) = \\widehat{x} + t\\widehat{\\eta}$. Using $\\funman\\circ\\geocurve=\\widehat{\\funman}\\circ\\widehat{\\geocurve}$, one obtains the result.\n\n    \\underline{Step 1.} We derive an approximation of $\\D \\funman(x)[\\eta] = \\langle \\grad \\funman(x), \\eta\\rangle$ in terms of $\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x}-\\widehat{x^\\star}]^2$. To do so, we go through the intermediate quantity $\\D \\widehat{\\funman}(\\widehat{x}) [\\widehat{x_+}-\\widehat{x}]$, and handle precisely the $o(\\cdot)$ terms. By smoothness of $\\widehat{\\funman}$ and since $\\D \\widehat{\\funman}(\\widehat{x^\\star})=0$, Taylor's formula for $\\D \\widehat{\\funman}$ writes\n    \\begin{align*}\n        \\D \\widehat{\\funman}(\\widehat{x}) [\\widehat{x_+}-\\widehat{x}] &= \\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x_+}-\\widehat{x}, \\widehat{x}-\\widehat{x^\\star}] + o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2)\\\\\n        &= -\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x}-\\widehat{x^\\star}]^2 + \\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x_+}-\\widehat{x^\\star}, \\widehat{x}-\\widehat{x^\\star}] + o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2)\\\\\n        &= -\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x}-\\widehat{x^\\star}]^2 + o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2),\n    \\end{align*}\n    where, in the last step, we used that $\\|\\widehat{x_+}-\\widehat{x^\\star}\\| = o(\\|\\widehat{x}-\\widehat{x^\\star}\\|)$ to get that $\\|\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x_+}-\\widehat{x^\\star}, \\widehat{x}-\\widehat{x^\\star}]\\| = \\|\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})\\| \\|\\widehat{x_+}-\\widehat{x^\\star}\\|\\|\\widehat{x}-\\widehat{x^\\star}\\| = o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2)$.\n    We now turn to show that $\\D \\widehat{\\funman}(\\widehat{x}) [\\widehat{x_+}-\\widehat{x}]$ behaves as $\\D \\funman(x)[\\eta]$ up to $o(\\|\\widehat{x_+}-\\widehat{x}\\|^2)$. Since $\\D \\funman(x)[\\eta]=\\D\\widehat{\\funman}(\\widehat{x})[\\widehat{\\eta}]$ by \\eqref{eq:gradHess_to_localcoords}, there holds:\n    \\begin{equation*}\n        \\|\\D \\funman(x)[\\eta] - \\D\\widehat{\\funman}(\\widehat{x})[\\widehat{x_+} - \\widehat{x}]\\| = \\|\\D\\widehat{\\funman}(\\widehat{x})[\\widehat{\\eta} - (\\widehat{x_+} - \\widehat{x})]\\| \\le \\|\\D\\widehat{\\funman}(\\widehat{x}) \\| \\|\\widehat{\\eta} - (\\widehat{x_+} - \\widehat{x})\\|.\n    \\end{equation*}\n\n    As $\\funman$ is twice differentiable and $\\exp$ is $\\C^\\infty$, $\\widehat{\\funman}$ is twice differentiable as well. In particular its derivative is locally Lipschitz continuous, so that for $\\widehat{x}$ near $\\widehat{x^\\star}$, we obtain a first estimate:\n    \\begin{equation*}\n        \\|\\D\\widehat{\\funman}(\\widehat{x})\\| = \\|\\D\\widehat{\\funman}(\\widehat{x}) - \\D\\widehat{\\funman}(\\widehat{x^\\star})\\| = \\mathcal O (\\| \\widehat{x}-\\widehat{x^\\star} \\|).\n    \\end{equation*}\n\n    Besides, the following estimate holds $\\|\\widehat{\\eta} - (\\widehat{x_+} - \\widehat{x})\\| = o (\\| \\widehat{x}-\\widehat{x^\\star} \\|)$. Indeed, as the function $\\log_{x^\\star}\\circ\\R_x : \\tangentM\\to\\tangentM[x^\\star]$ is differentiable, there holds for $\\eta\\in\\tangentM$ small,\n    \\begin{equation*}\n        \\log_{x^\\star}(\\R_x(\\eta)) = \\log_{x^\\star}(\\R_x(0)) + \\D\\log_{x^\\star}(\\R_x(0))[\\D\\R_x(0)[\\eta]] + o(\\|\\eta\\|),\n    \\end{equation*}\n    which simplifies to $\\widehat{x_+} = \\widehat{x} + \\widehat{\\eta} + o(\\|\\eta\\|)$. \\Cref{lem:InvRetractionApproxLog} allows to write $\\|\\eta\\| = \\|\\R_x^{-1}(x_+)\\| = \\mathcal O(\\distM(x, x_+))$. Using the triangular inequality and the assumption that $\\distM(x_+, x^\\star) = o(\\distM(x, x^*))$ we get\n    \\begin{equation*}\n        \\distM(x, x_+) \\le \\distM(x, x^\\star) + \\distM(x^\\star, x_+) = \\mathcal O(\\distM(x, x^*)) = \\mathcal O(\\|\\widehat{x}-\\widehat{x^\\star}\\|),\n    \\end{equation*}\n    so that the second estimate holds.\n\n    Combining the two above estimates allows to conclude that\n    \\begin{equation*}\n        \\|\\D \\funman(x)[\\eta] - \\D\\widehat{\\funman}(\\widehat{x})[\\widehat{x_+} - \\widehat{x}]\\| = o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2),\n    \\end{equation*}\n    so that overall,\n    \\begin{equation*}\n        \\D \\funman(x)[\\eta] = \\D \\widehat{\\funman}(\\widehat{x}) [\\widehat{x_+}-\\widehat{x}] + o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2) = -\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x}-\\widehat{x^\\star}]^2 + o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2).\n    \\end{equation*}\n\n\n\n    Using that $\\|\\widehat{x}-\\widehat{x^\\star}\\| = \\distM(x, x^\\star)$ and $\\D^2 \\widehat{\\funman}(\\widehat{x^\\star}) = \\Hess \\funman(x^\\star)$ \\eqref{eq:gradHess_to_localcoords}, we obtain\n    \\begin{equation}\\label{eq:proofunitstepsize_gradhess}\n        \\D \\funman(x)[\\eta] = -\\Hess \\funman(x^\\star)[\\widehat{x}-\\widehat{x^\\star}]^2 + o(\\distM(x, x^\\star)^2).\n    \\end{equation}\n\n    \\underline{Step 2.} The function $\\funman$ admits a second-order development around $x^\\star$: applying \\cref{eq:manifoldSecondOrderDevSecRetraction} with the exponential map $\\exp_{x^\\star}$ as a second-order retraction yields\n    \\begin{equation}\\label{eq:proofunitstepsize_fdev}\n        \\funman(x) = \\funman(x^\\star) + \\frac{1}{2} \\Hess \\funman(x^\\star)[\\widehat{x}-\\widehat{x^\\star}]^2 + o(\\distM(x, x^\\star)^2),\n    \\end{equation}\n    where we used that $\\distM(x, x^\\star) = \\|\\log_{x^\\star}(x)-\\log_{x^\\star}(x^\\star)\\|$. Denote $0 < l \\le L$ the lower and upper eigenvalues of $\\Hess \\funman(x^\\star)$. The combination $\\eqref{eq:proofunitstepsize_fdev} + m_1 \\eqref{eq:proofunitstepsize_gradhess}$ writes\n    \\begin{align*}\n        \\funman(x) + m_1 \\D \\funman(x)[\\eta] &= \\funman(x^\\star) + (\\frac{1}{2}-m_1) \\Hess \\funman(x^\\star)[\\widehat{x}-\\widehat{x^\\star}]^2 + o(\\distM(x, x^\\star)^2) \\\\\n        &\\ge \\funman(x^\\star) + (\\frac{1}{2}-m_1) l \\distM(x, x^\\star)^2 + o(\\distM(x, x^\\star)^2),\n    \\end{align*}\n\n    Let $\\varepsilon >0$ such that $\\frac{1}{2}L \\varepsilon^2  < (\\frac{1}{2}-m_1) l$. As $\\distM(x_+, x^\\star)=o(\\distM(x, x^\\star))$, for $x$ close enough to $x^\\star$ there holds $\\distM(x_+, x^\\star) \\le \\varepsilon \\distM(x, x^\\star)$. Combining this with the second-order development of $f$ at $x_+$, there holds:\n    \\begin{align*}\n        \\funman(x_+) &= \\funman(x^\\star) + \\frac{1}{2} \\Hess \\funman(x^\\star)[\\widehat{x_+}-\\widehat{x^\\star}]^2 + o(\\distM(x_+, x^\\star)^2) \\\\\n        &\\le \\funman(x^\\star) + \\frac{1}{2}L \\distM(x_+, x^\\star)^2 + o(\\distM(x_+, x^\\star)^2)\\\\\n        &\\le \\funman(x^\\star) + \\frac{1}{2}L \\varepsilon^2 \\distM(x, x^\\star)^2 + o(\\distM(x, x^\\star)^2).\n    \\end{align*}\n\n    Subtracting the two estimates yields\n    \\begin{equation*}\n        \\funman(x_+) - (\\funman(x) + m_1 \\D \\funman(x)[\\eta]) \\le \\left(\\frac{1}{2}L \\varepsilon^2 - (\\frac{1}{2}-m_1) l \\right) \\distM(x, x^\\star)^2 + o(\\distM(x, x^\\star)^2),\n    \\end{equation*}\n    which ensures that the Armijo condition is satisfied.\n\n\n\n\n\n\\end{proof}","units":[{"start":41893,"end":42833,"start_line":770,"end_line":779,"raw_tex":"\\begin{theorem}[Proximal gradient points smoothly locate manifolds] \\label{th:PGpointlocatemanifolds}\n    Let $f$ be a $\\C^2$ function on $\\RR^n$ and $g$ a lower semi-continuous function on $\\RR^n$. Suppose that $g$ is both $r$-prox-regular at $\\bar{x}$ and partly-smooth relative to $\\M$ at $\\bar{x}$.\n\n    Take $\\gamma$, $\\bar \\gamma$ such that $0<\\gamma<\\bar \\gamma \\leq 1/r$ and $\\bar{x}=\\prox_{\\bar\\gamma g}(\\bar{y}-\\bar\\gamma \\nabla f(\\bar{y}))$. If\n    \\begin{itemize}\n        \\item[i)] $\\frac{1}{\\gamma}(\\bar{y}-\\bar{x}) - \\nabla f(\\bar{y})  \\in \\ri \\partial g(\\bar{x})$ ~\\revise{(the relative interior of the subdifferential at $\\bar{x}$)};\n        \\item[ii)] either a) $\\gamma$ is sufficiently close to $\\bar{\\gamma}$, or b) $\\bar{y}$ is sufficiently close to $\\bar{x}$;\n    \\end{itemize}\n    then, the proximal gradient $y \\mapsto \\prox_{\\gamma g}(y - \\gamma\\nabla f(y))$ is $\\C^{1}$ and $\\M$-valued near $\\bar{y}$.\n\\end{theorem}","sha256_utf8":"61ed7d5643fd2d937656148e05f2510cd79c54fcfbb8c2502e8ad849c9e6e396"},{"start":42834,"end":47887,"start_line":780,"end_line":819,"raw_tex":"\\begin{proof}\n    Adopting the same reasoning as in \\cite[Sec. 5]{lewis2002active} and \\cite[Sec. 4.1]{daniilidis2006geometrical}, we consider the function\n    \\begin{equation*}\n        \\begin{array}{rl}\n            \\rho : \\bbR^n\\times\\bbR^n & \\to \\bbR                                                 \\\\\n            (x, y)                    & \\mapsto g(x)+\\frac{1}{2\\gamma}\\|x-y+\\gamma\\nabla f(y)\\|^2,\n        \\end{array}\n    \\end{equation*}\n    and denote by $\\rho_y = \\rho(\\cdot,y)$. Computing the proximal gradient $\\prox_{\\gamma g}(y-\\gamma\\nabla f(y))$ can then be seen as minimizing the parametrized function $\\rho_y$.\n\n    \\noindent\\underline{Step 1.} As a first step, we study the minimizers of $\\rho_y$ restricted to $\\M$, for $y$ near\\;$\\bar{y}$. %\n    We consider the parametric manifold optimization problem, for $y$ near $\\bar{y}$:\n    \\begin{equation}\\tag{$ P_\\M(y)$}\n        \\min_{x\\in\\M} \\rho_y(x). \\label{pb:pb_mancstr}\n    \\end{equation}\n    Since $g$ is $\\C^2$-partly-smooth relative to $\\M$ and $f$ is $\\C^2(\\bbR^n)$, $\\rho_y$ is twice continuously differentiable on $\\M$. \\revise{Moreover, the $r$-prox-regularity gives easily (see \\cref{lemma:prox_strong_min}) that $\\rho_{\\bar{y}}$ is lower-bounded by $(\\frac{1}{\\gamma}-r)\\|\\cdot - \\bar{x}\\|^2/2$ on a neighborhood of $\\bar x$ in $\\bbR^n$ and, a fortiori, in $\\M$. From usual rationale (see e.g.\n    \\cite[Chap 4.2, 6.1]{boumal2022intromanifolds}), this implies}\n    \\begin{equation*}\n        \\grad \\rho_{\\bar{y}}(\\bar{x}) = 0 \\qquad \\Hess \\rho_{\\bar{y}}(\\bar{x}) \\succeq \\Big(\\frac{1}{\\gamma}-r\\Big) I \\succ 0,\n    \\end{equation*}\n    \\revise{which are the conditions to apply the implicit functions theorem, as follows.}\n\n    We consider the equation $\\Phi(x, y) = 0$, for $x, y$ near $\\bar{x}, \\bar{y}$, where $\\Phi:\\M\\times\\bbR^n\\to\\tangentBundle$ is defined as $\\Phi(x, y) = \\grad \\rho_y(x)$. This function is continuously differentiable on a neighborhood of $(\\bar{x}, \\bar{y})$, and its differential relative to $\\bar x$ at that point, $\\Hess \\rho_{\\bar{y}}(\\bar{x})$, is invertible. The implicit function theorem thus grants the existence of neighborhoods $\\N_{\\bar{x}}$, $\\N_{\\bar{y}}$ of $\\bar{x}$, $\\bar{y}$ in $\\M$, $\\bbR^n$, and a continuously differentiable function $\\hat{x}:\\N_{\\bar{y}} \\to \\N_{\\bar{x}}$ such that, for any $y$ in $\\N_{\\bar{y}}$, $\\Phi(\\hat{x}(y), y) = \\grad \\rho_y(\\hat{x}(y)) = 0$. Actually, $\\hat{x}(y)$ is a strong minimizer of $\\rho_y$ on $\\M$ for $y$ close enough to $\\bar{y}$. Indeed, the mapping $\\hat{x}$ is continuous on $\\N_{\\bar{y}}$, so that $y \\mapsto \\Hess \\rho_y(\\hat{x}(y))$ is also continuous there and the property $\\Hess \\rho_{\\bar{y}}(\\hat{x}(\\bar{y}))\\succ 0$ extends locally around $\\bar{y}$.\n\n    \\noindent\\underline{Step 2.}  As a second step, we turn to show that the minimizer $\\hat{x}(y)$ of $\\rho_y$ on $\\M$ is actually a strong critical point of $\\rho_y$ in $\\bbR^n$ \\cite[Def. 5.3]{lewis2002active}, and thus the proximal gradient of point $y$. More precisely, we claim that, for $y$ near $\\bar{y}$ and $x=\\hat{x}(y)$, there holds $0\\in\\ri \\partial \\rho_y(x)$, that is\n    \\begin{equation*}\n        \\frac{1}{\\gamma}(y-x) - \\nabla f(y)  \\in \\ri \\partial g(x).\n    \\end{equation*}\n\n    This property holds at $(\\bar{x}, \\bar{y})$ by assumption. By contradiction, assume there exist sequences of points $(y_r)$ with limit $\\bar{y}$, $(x_r) = (\\hat x(y_r))$ with limit $\\bar{x} = \\hat x(\\bar{y})$ and $(h_r)$ of unit norm $\\|h_r\\|=1$ such that for all $r$, $h_r$ separates 0 from $\\partial \\rho_{y_r}(x_r)$:\n    \\begin{equation*}\n        \\inf_{h\\in\\partial \\rho_{y_r}(x_r)} \\langle h_r, h\\rangle \\ge 0.\n    \\end{equation*}\n    Since $(h_r)$ is bounded, a converging subsequence can be extracted from it, let $\\bar{h}$ denote its limit. At the cost of renaming iterates, we assume that $\\lim_{r\\to\\infty} h_r = \\bar{h}$. The above property still holds at the limit $r\\to\\infty$. Indeed, let $\\bar{u}\\in\\partial \\rho_{\\bar{y}}(\\bar{x})$. Since $g$ is partly smooth, the mapping $(x, y)\\in \\N_{\\bar{x}}\\times\\N_{\\bar{y}} \\mapsto\\partial \\rho_y(x) = \\partial g(x) + \\frac{1}{\\gamma}(x-y)$ is continuous. Therefore, there exists a sequence $(u_r)$ such that $u_r\\in\\partial\\rho_{y_r}(x_r)$ and $\\lim_{r\\to\\infty} u_r=\\bar{u}$. We have for all $r$: $\\langle u_r, h_r\\rangle \\ge 0$, which yields at the limit $\\langle\\bar{u}, \\bar{h}\\rangle\\ge 0$. Thus $\\bar{h}$ separates $0$ from $\\partial \\rho_{\\bar{y}}(\\bar{x})$, which contradicts our assumption.\n\n    \\noindent\\underline{Conclusion.} We thus have a continuously differentiable function $\\hat{x}$ defined on a neighborhood of $\\bar{y}$ such that i) $\\hat{x}(\\bar{y}) = \\bar{x}$, ii) $\\hat{x}(y)$ is a strong minimizer of $\\rho_y$ on $\\M$, iii) $0\\in\\ri\\partial \\rho_y(\\hat{x}(y))$.\n\n    This last point tells us that $(y-\\hat x(y))/\\gamma -\\nabla f(y) \\in \\partial g(\\hat x(y))$. The characterization of proximity by the optimality condition (\\cref{th:proxex}) gives\nthat $\\hat{x}(y)=\\prox_{\\gamma g}(y-\\gamma \\nabla f(y))$ for $y$ close enough to $\\bar{y}$.\n\\end{proof}","sha256_utf8":"7eee440ebef07eabf94a432927500ce43f5877271b3e979d504271441d27f0b1"},{"start":51892,"end":52884,"start_line":904,"end_line":920,"raw_tex":"\\begin{theorem}%\n    \\label{th:identiflocalcv_sperlin}\n    Let \\cref{asm:fun} hold and take $\\gamma\\in(0, 1/L)$\\revise{, where $L$ is the Lipschitz constant for $\\nabla f$}. Assume that \\cref{alg:generic} generates a sequence $(y_k)$ which admits at least one limit point $\\bar{x}$ such that:\n    \\begin{itemize}\n        \\item[i)] $\\bar{x}\\in\\M$ is a $r$-structured critical point for $(f,g)$ with $r<1/\\gamma$;\n        \\item[ii)] $\\manupdate_{\\M}$ has superlinear convergence rate of order $1+\\theta\\in(1,2)$ %\n        near $\\bar{x}$ in $\\M$.\n    \\end{itemize}\n    Then, after some finite time:\n    \\begin{itemize}\n        \\item[a)] the full sequence $(x_k)$ lies on $\\M$;\n        \\item[b)] $x_k$ converges to $\\bar{x}$ superlinearly with the same order as $\\manupdate$:\n              \\begin{equation}\\label{eq:super}\n                  \\distM(x_{k+1},\\bar{x}) \\le c ~ \\distM(x_k, \\bar{x})^{1+\\theta} \\qquad \\text{ for some $c>0$.}\n              \\end{equation}\n    \\end{itemize}\n\\end{theorem}","sha256_utf8":"185dec797db7e909bae9e82dc922d24c957ed90d202f35c89d8a9dfda89dba79"},{"start":52888,"end":56637,"start_line":924,"end_line":964,"raw_tex":"\\begin{proof}\n    Let us note $ \\PGop(y) = \\prox_{\\gamma g}(y - \\gamma \\nabla f(y)) $ for $y\\in\\mathbb{R}^n$.\n    The part i) of the assumptions enables us to show the existence of some neighborhood of $\\bar{x}$ on which the proximal gradient operation is $\\M$-valued and Lipschitz continuous. More precisely, \n    \\cref{th:PGpointlocatemanifolds} \n    implies that there exists $\\delta_1>0$ and $C>0$ such that, \n    \\[%\n      \\PGop(y)\\in\\M \\label{proof:inM} \\quad \\text{ and } \\quad \\|\\PGop(y)- \\PGop(\\bar{x})\\| \\le C \\|y-\\bar{x}\\| \\qquad \\text{for all $y$ in $\\B(\\bar{x}, \\delta_1)$}.\n    \\]%\n    Now, if $y$ belongs to $\\M$, we get that there exists $\\varepsilon_1>0$ such that for any $y$ in $\\B_{\\M}(\\bar{x}, \\varepsilon_1)$, $\\PGop(y)\\in\\M$; but in addition, the Euclidean Lipschitz continuity can be translated into a Riemannian one (see \\cref{lemma:Euclidean_to_Riemannian}) since   for some $\\delta>0$,\n    \\begin{multline}\\label{eq:EuctoRie}\n        (1-\\delta)\\distM(\\PGop(y), \\bar{x}) = (1-\\delta)\\distM(\\PGop(y), \\PGop(\\bar{x}))  \\le \\|\\PGop(y)-\\PGop(\\bar{x})\\| \\\\\n        \\le C \\|y-\\bar{x}\\| \\le C (1+\\delta)\\distM(y, \\bar{x})\n    \\end{multline}\n    Hence, there is $q_1>0$ such that for any $y$ in $\\B_{\\M}(\\bar{x}, \\varepsilon_1)$\n    \\begin{equation}\\label{proof:PGopEstimate}\n        \\distM(\\PGop(y), \\bar{x}) =  \\distM(\\PGop(y), \\PGop(\\bar{x}))   \\le  q_1 ~ \\distM(y, \\bar{x}).\n    \\end{equation}\n\n    Then, the part ii) of the assumptions gives us the existence of $\\varepsilon_2$, $q_2>0$ and $\\theta\\in(0,1)$ such that, for any $x$ in $\\B_{\\M}(\\bar{x}, \\varepsilon_2)$,\n    \\begin{equation}\\label{proof:manupdateEstimate}\n        \\distM(\\manupdate_{\\M}(x), \\bar{x}) \\le q_2 ~ \\distM(x, \\bar{x})^{1+\\theta}.\n    \\end{equation}\n    Let us now take any $x\\in\\B_{\\M}(\\bar{x}, \\varepsilon)$ where $\\varepsilon=\\min(\\varepsilon_1,\\varepsilon_2,(\\varepsilon_1 / q_2)^{\\frac{1}{1+\\theta}},(q_2 q_1)^{-\\frac{1}{\\theta}})$: \\\\\n    \\noindent (\\emph{i}) Since $x\\in\\B_{\\M}(\\bar{x}, \\varepsilon_2)$, the manifold update \\eqref{proof:manupdateEstimate} yields\n    \\begin{equation*}\n        \\distM(\\manupdate_{\\M}(x), \\bar{x}) \\le q_2 ~ \\distM(x, \\bar{x})^{1+\\theta} \\le q_2 ~ \\varepsilon^{1+\\theta} \\le \\varepsilon_1.\n    \\end{equation*}\n    \\noindent (\\emph{ii}) As $\\manupdate_{\\M}(x)$ lies in $\\B_{\\M}(\\bar{x}, \\varepsilon_1)$, the proximal gradient update \\eqref{proof:PGopEstimate} applied to $y = \\manupdate_{\\M}(x)$ gives\n    \\begin{multline}\\label{eq:superlinear}\n        \\distM(\\PGop(\\manupdate_{\\M}(x)), \\bar{x}) \\le    q_1 \\distM(\\manupdate_{\\M}(x), \\bar{x}) \\\\ \\le q_1 q_2 ~ \\distM(x, \\bar{x})^{1+\\theta} \\le q_1 q_2 ~ \\varepsilon^\\theta \\distM(x, \\bar{x}).\n    \\end{multline}\n    Since $q_2 q_1 \\varepsilon^\\theta \\le 1$  by construction, this gives %\n    \\begin{equation}\\label{eq:manstable}\n        \\distM(\\PGop(\\manupdate_{\\M}(x)), \\bar{x}) \\le \\distM(x, \\bar{x}) \\qquad \\text{for any $x\\in\\B_{\\M}(\\bar{x}, \\varepsilon)$}.\n    \\end{equation}\n    We have thus proved the existence of a neighborhood $\\B_\\M(\\bar{x}, \\varepsilon)$ of $\\bar{x}$ in $\\M$ which is stable for an iteration of \\cref{alg:generic} and over which one iteration has a superlinear improvement of order $1+\\theta$ (by \\eqref{eq:superlinear}).\n\n\n    Finally, since $\\bar{x}$ is a limit point of $(y_k)$, there exists $K<\\infty$ such that $y_K\\in\\B(\\bar{x},(1-\\delta)\\varepsilon/C)$. Besides, \\eqref{eq:EuctoRie} tells us that $ \\distM(\\PGop(y_K), \\bar{x})\\le \\varepsilon$ and thus $x_k$ and $y_k$ belong to $\\B_{\\M}(\\bar{x}, \\varepsilon)$ for all $k>K$ by \\eqref{eq:manstable}. %\n    We conclude that $x_{k+1}=\\PGop(y_k)\\in\\M$ for all $k\\geq K$, and, using \\eqref{eq:superlinear}, that we have \\eqref{eq:super} with $c= q_1 q_2$, for all $k>K$. \n\\end{proof}","sha256_utf8":"cc51dcdfba46de7deb8816f183f1473b6c7fe8af32cdb6cd1de04104c298d46d"},{"start":59124,"end":59559,"start_line":994,"end_line":996,"raw_tex":"\\begin{lemma}\\label{lemma:LS_exists_valid_stepsizes}\n    Let \\cref{asm:fun} hold and consider a manifold $\\M$ equipped with a retraction $\\R$ and a pair $(x, \\eta)\\in\\tangentBundle$. If $\\funtot$ is differentiable on $\\M$ at $x$, $\\langle \\grad \\funtot (x), \\eta\\rangle <0$, and $m_1<1$, then there exists $\\hat\\alpha>0$ such that any step size $\\alpha\\in(0, \\hat\\alpha)$ is acceptable by the Armijo rule \\eqref{eq:Armijo}.\n\\end{lemma}","sha256_utf8":"79565d6da0ca9cc53921cf70fd4b0d50439e66bf818d423f756b76e7829f8e84"},{"start":59561,"end":60265,"start_line":998,"end_line":1004,"raw_tex":"\\begin{proof}\n    We adapt a part of the proof of  \\cite[Th. 6.3.2]{dennis1996numerical} for the Armijo rule and the Riemannian setting. Since $m_1<1/2$, for any $\\alpha$ sufficiently small there holds\n    \\begin{equation*}\n        \\funtot\\circ \\R_x(\\alpha \\eta) \\le \\funtot\\circ\\R_x(0) + m_1 \\D\\left(\\funtot\\circ\\R_x \\right)(0)[\\alpha\\eta] = \\funtot(x) + m_1 \\alpha \\langle \\grad \\funtot(x), \\eta\\rangle.\n    \\end{equation*}\n    Since $\\funtot$ is bounded below, there exists a smallest $\\hat{\\alpha}$ such that $\\funtot(\\R_x(\\hat\\alpha \\eta)) = \\funtot(x) + m_1 \\hat\\alpha \\langle \\grad \\funtot(x), \\eta\\rangle$. Thus all stepsizes in $(0, \\hat\\alpha)$ are acceptable by\\;\\eqref{eq:Armijo}.\n\\end{proof}","sha256_utf8":"ae9779db8ea8c09b4f2150d350d1e9e5f1c18d2156eee32dbe51664a9c1519cf"},{"start":60268,"end":61254,"start_line":1007,"end_line":1012,"raw_tex":"In addition, a \\revise{line search} performed near a minimizer with a Newton direction\nshould accept the unit stepsize, so that a full step may be taken. This is the case when the Riemannian Hessian around this minimizer is positive definite as stated by the next lemma, \\revise{which is a direct corollary of \\cref{th:unitstepsuperlinearcv}.}\n\\begin{lemma}\\label{lemma:LS_unit_stepsize}\n    Let \\cref{asm:fun} hold and consider a manifold $\\M$ equipped with a retraction $\\R$, a point $x^\\star\\in\\M$ and a pair $(x, \\eta)\\in\\tangentBundle$. Assume that $\\funtot$ is twice differentiable on $\\M$ near a strong local minimizer  $x^\\star$  on $\\M$, that is $\\Hess \\funman(x^\\star)$ is positive definite.\n    If the direction $\\eta$ brings a superlinear improvement towards $x^\\star$, that is $\\distM(\\R_x(\\eta), x^\\star) = o(\\distM(x, x^\\star))$ as $x\\to x^\\star$, and $0<m_1<1/2$, then $\\eta$ is acceptable by the Armijo rule\\;\\eqref{eq:Armijo} with unit stepsize $\\alpha=1$.\n\\end{lemma}","sha256_utf8":"c2221fa4f299a69a9b6bdf414f16bb5e8c3f4d976238720ab42bbb1dbe0dd349"},{"start":63797,"end":66141,"start_line":1070,"end_line":1099,"raw_tex":"\\begin{proof}\n    As the Riemannian Hessian is assumed to be positive definite, Newton's direction is a descent direction:\n    \\begin{equation*}\n        \\langle \\grad \\funtot(x_k), d_k \\rangle = -\\langle \\grad \\funtot(x_k), \\Hess \\funtot(x_k)^{-1} \\grad \\funtot(x_k) \\rangle < 0.\n    \\end{equation*}\n    The Riemannian Newton manifold step is therefore well-defined, and the \\revise{line search} terminates by \\cref{lemma:LS_exists_valid_stepsizes}, so that the manifold update is well-defined and provides descent \\revise{($F(y_{k})\\le F(x_{k})$)}. \n\n\n    \\revise{Now, since the proximal gradient update provides a descent (see \\cite[Lem. 10.4]{beck2017first}),\n    \\begin{equation}\\label{eq:proofglobalalg_descent}\n        F(x_{k+1}) \\le F(y_k) - \\frac{1-\\gamma L}{2\\gamma} \\| x_{k+1}-y_k \\|^2 \\le F(x_k) - \\frac{1-\\gamma L}{2\\gamma} \\| x_{k+1}-y_k \\|^2.\n    \\end{equation}\n    The sequence $(F(x_k))$ is thus non-increasing and lower-bounded, therefore it converges.\n    Besides, any accumulation point of $(x_k)$ is a critical point of $F$. Indeed, summing equation \\eqref{eq:proofglobalalg_descent} for $k=1, \\ldots, n$ yields:\n    \\begin{equation*}\n        \\frac{1-\\gamma L}{2\\gamma} \\sum_{k=1}^n \\| x_{k+1}-y_k \\|^2 \\le F(x_1) - F(x_{n+1}) \\le F(x_1)-\\inf F < +\\infty.\n    \\end{equation*}\n    Since $ \\dist (0, \\partial F(x_{k+1})) \\le \\frac{L\\gamma + 1}{\\gamma}\\|x_{k+1}-y_k\\|$ (see e.g. the proof of\\;\\cite[Prop.\\;13]{bolte2015error}), we have that \n    $\\dist (0, \\partial F(x_{k+1}))$\n    converges to $0$. The outer-semi continuity of the limiting subdifferential then yields criticality of accumulation points.}\n\n\n\n\n\n    Now we apply the local convergence of Riemannian Newton \\cite[Th. 6.3.2]{absil2009optimization}:\n    assumption ii) ensures that\n    the Riemannian Newton direction $d$ computed in step\\;1 of\\;\\cref{alg:ManUpNewton} provides a quadratic improvement %\n    on a neighborhood of $x^\\star$ on\\;$\\M$. Moreover, the \\revise{line search} returns the unit-stepsize after some finite time: $\\alpha=1$ is tried first, and is acceptable for directions providing superlinear improvement by \\cref{lemma:LS_unit_stepsize}. Thus the whole Riemannian Newton update provides quadratic improvement after some finite time. Using this and assumption i), \\cref{th:identiflocalcv_sperlin} applies and yields the results.\n\\end{proof}","sha256_utf8":"848ac24f841a1f40a87201227b7af239bd4661ea99fcec15f4e58b6690112fad"},{"start":82908,"end":83804,"start_line":1400,"end_line":1409,"raw_tex":"\\begin{lemma}\n    \\label{th:proxex}\n    Consider a function $\\funns:\\bbR^n\\to\\bbR$, a pair of points $\\bar{x}, \\bar{y}$ and a \\revise{step length} $\\bar{\\gamma}>0$ such that $\\bar{x}=\\prox_{\\bar\\gamma \\funns}(\\bar{y})$ and $\\funns$ is $r$ prox-regular at $\\bar{x}$ for subgradient $\\bar{v} \\triangleq (\\bar{y}-\\bar{x}) / \\bar{\\gamma}$.\n\n    Then, for any $\\gamma\\in(0, \\min(1/r, \\bar\\gamma))$, there exists a neighborhood $\\N_{\\bar{y}}$ of $\\bar{y}$ over which $\\prox_{\\gamma \\funns}$ is single-valued and $(1-\\gamma r)^{-1}$-Lipschitz continuous. Furthermore, there holds\n    \\begin{equation*}\n        x = \\prox_{\\gamma \\funns}(y) \\Leftrightarrow (y-x)/\\gamma \\in\\partial \\funns(x)\n    \\end{equation*}\n    for $y\\in\\N_{\\bar{y}}$ and $x$ near $\\bar{x}$ in the sense $\\|x-\\bar{x}\\|<\\varepsilon$, $|\\funns(x)-\\funns(\\bar{x})|<\\varepsilon$ and $\\|(y-x)/\\gamma - \\bar{v}\\|<\\varepsilon$. %\n\\end{lemma}","sha256_utf8":"6cdf1e0a5f14689b6324c3e8e81578c80bad292b007b7bfbc5881462f149d2a5"},{"start":83806,"end":86153,"start_line":1411,"end_line":1429,"raw_tex":"\\begin{proof}\n    One can easily check that prox-regularity of $\\funns$ at $\\bar{x}$ for subgradient $\\bar{v}$ is equivalent to prox-regularity of function $\\tilde{\\funns}$ around $0$ for subgradient $0$, with $\\tilde{\\funns} = \\funns(\\cdot + \\bar{x}) - \\langle \\bar{v}, \\cdot\\rangle - \\funns(\\bar{x})$ and a change of variable $\\tilde{x} = x-\\bar{x}$. Similarly, $\\bar{x}=\\prox_{\\bar\\gamma \\funns}(\\bar{y})$ is characterized by its global optimality~condition\n    \\begin{equation*}\n        \\funns(x) + \\frac{1}{2\\bar\\gamma}\\|x-\\bar{y}\\| > \\funns(\\bar{x}) + \\frac{1}{2\\bar\\gamma}\\|\\bar{x}-\\bar{y}\\|^2 \\qquad \\text{ for all } x\\neq\\bar{x},\n    \\end{equation*}\n    which we may write as\n    \\begin{equation*}\n        \\funns(x) > \\funns(\\bar{x}) + \\langle \\bar{v}, x-\\bar{x} \\rangle - \\frac{1}{2\\bar\\gamma}\\|x - \\bar{x}\\|^2 \\qquad \\text{ for all } x\\neq\\bar{x}.\n    \\end{equation*}\n    Under that same change of variables, since $\\tilde{\\funns}(0) = 0$, this optimality condition rewrites as\n    \\begin{equation*}\n        \\tilde{\\funns}(\\tilde{x}) > - \\frac{1}{2\\bar\\gamma}\\|\\tilde{x}\\|^2 \\qquad \\text{ for all } \\tilde{x}\\neq 0.\n    \\end{equation*}\n    We may thus apply Theorem 4.4 from \\cite{poliquin1996prox} to get the claimed result on $\\tilde{\\funns}$, which transfers back to $\\funns$ as our change of function and variable is bijective. We thus obtain that for $\\gamma\\in(0, \\min(1/r, \\bar\\gamma))$, on a neighborhood $\\N_{\\bar{y}}$ of $\\bar{y}$, $\\prox_{\\gamma \\funns}$ is single-valued, $(1-\\gamma r)^{-1}$-Lipschitz continuous and $\\prox_{\\gamma \\funns}(y)=[I+\\gamma T]^{-1}(y)$, where $T$ denotes the $\\funns$-attentive $\\varepsilon$-localization of $\\partial \\funns$ at $\\bar{x}$. Taking $y$ near $\\bar{y}$ and $x$ near $\\bar{x}$ such that $\\|x-\\bar{x}\\|<\\varepsilon$, $|\\funns(x)-\\funns(\\bar{x})|<\\varepsilon$ and $\\|(y-x)/\\gamma - \\bar{v}\\|<\\varepsilon$ allows to identify the localization of $\\partial \\funns(x)$ with $\\partial \\funns(x)$, so that\n    \\begin{equation*}\n        \\frac{y-x}{\\gamma}\\in \\partial \\funns(x)\\Leftrightarrow \\frac{y-x}{\\gamma}\\in T(x)\\Leftrightarrow (I+\\gamma T)(x)=y\\Leftrightarrow x=\\prox_{\\gamma \\funns}(y).\n    \\end{equation*}\n    Note that the proof of \\cite[Th. 4.4]{poliquin1996prox} includes a minor error relative to the Lipschitz constant computation, we report here a corrected value.\n\\end{proof}","sha256_utf8":"268fac5359ba1169849c6a34a27267e5f81ae0369f77d30904c7fa7289f99ce7"},{"start":86364,"end":87041,"start_line":1445,"end_line":1451,"raw_tex":"\\begin{lemma}\\label{lemma:prox_strong_min}\n    Let $f$ and $g$ denote two functions and $\\bar{x}$, $\\bar{y}$ two points such that $f$ is differentiable at $\\bar{y}$ and $g$ is $r$-prox-regular at $\\bar{x}$ for subgradient $\\frac{1}{\\gamma}(\\bar{y}-\\bar{x}) - \\nabla f(\\bar{y}) \\in \\partial g(\\bar{x})$ with $\\gamma \\in (0, 1/r)$.\n    Then, the function $\\rho_{\\bar{y}} : x\\mapsto g(x) + \\frac{1}{2\\gamma}\\|\\bar{y}-\\gamma\\nabla f(\\bar{y}) - x\\|^2$ satisfies %\n    \\begin{equation*}\n        \\rho_{\\bar{y}}(x) \\ge \\rho_{\\bar{y}}(\\bar{x}) + \\frac{1}{2} \\left( \\frac{1}{\\gamma} - r \\right) \\| x-\\bar{x} \\|^2, \\qquad \\text{for all $x$ near $\\bar{x}$.}\n    \\end{equation*}\n\\end{lemma}","sha256_utf8":"ab2ff1fdb658830a5372da3b199c88b4d75891cef28818d4220469589bc43c35"},{"start":87043,"end":88248,"start_line":1453,"end_line":1468,"raw_tex":"\\begin{proof}\n    Prox-regularity of $g$ at $\\bar{x}$ with subgradient $\\frac{1}{\\gamma} (\\bar{y}-\\gamma \\nabla f(\\bar{y}) - \\bar{x}) \\in \\partial g(\\bar{x})$ writes\n    \\begin{equation*}\n        g(x) \\ge g(\\bar{x}) + \\frac{1}{\\gamma}\\langle \\bar{y}-\\gamma \\nabla f(\\bar{y}) - \\bar{x} , x-\\bar{x} \\rangle - \\frac{r}{2} \\| x-\\bar{x} \\|^2.\n    \\end{equation*}\n    The identity $ 2\\langle b-a, c-a\\rangle = \\|b-a\\|^2 + \\|c-a\\|^2 -   \\|b-c\\|^2 $ applied to the previous scalar product yields:\n    \\begin{equation*}\n        g(x) \\ge g(\\bar{x}) + \\frac{1}{2\\gamma} \\| \\bar{y}-\\gamma \\nabla f(\\bar{y}) - \\bar{x} \\|^2 + \\frac{1}{2\\gamma} \\| x-\\bar{x} \\|^2 - \\frac{1}{2\\gamma} \\|\\bar{y}-\\gamma \\nabla f(\\bar{y}) - x \\|^2 - \\frac{r}{2} \\| x-\\bar{x} \\|^2,\n    \\end{equation*}\n    which rewrites\n    \\begin{equation*}\n        \\underbrace{ g(x) + \\frac{1}{2\\gamma} \\|\\bar{y}-\\gamma \\nabla f(\\bar{y}) - x \\|^2 }_{=\\rho_{\\bar{y}}(x) } \\ge  \\underbrace{  g(\\bar{x}) + \\frac{1}{2\\gamma} \\| \\bar{y}-\\gamma \\nabla f(\\bar{y}) - \\bar{x} \\|^2   }_{=\\rho_{\\bar{y}}(\\bar{x}) } + \\frac{1}{2} \\left( \\frac{1}{\\gamma} - r \\right) \\| x-\\bar{x} \\|^2,\n    \\end{equation*}\n    \\vspace*{-1ex}\nwhich is the claimed inequality.\n\\end{proof}","sha256_utf8":"9c59053cc5c0bd3d197b537fcb212dfacffb6d1d2682ff3e2d46f4b01d54f9fb"},{"start":88667,"end":89261,"start_line":1497,"end_line":1503,"raw_tex":"\\begin{lemma}\\label{lem:InvRetractionApproxLog}\n    Consider a point $\\bar{x}$ of a Riemannian manifold $\\M$, equipped with a retraction $\\R$ such that $\\R_{\\bar{x}}$ is $\\C^2$. For any $\\varepsilon>0$, there exists a neighborhood $\\U$ of $\\bar{x}$ in $\\M$ such that \n    \\begin{equation*}\n        (1-\\varepsilon)\\distM(x, \\bar{x}) \\le \\|\\R^{-1}_{\\bar{x}}(x)\\| \\le (1+\\varepsilon)\\distM(x, \\bar{x}) \\qquad \\text{ for all $x\\in\\U$.}\n    \\end{equation*}\n    where $\\R^{-1}_{\\bar{x}}:\\M\\to\\tangent{\\bar{x}}{\\M}$ is the smooth inverse of $\\R_{\\bar{x}}$ defined locally around $\\bar{x}$.\n\\end{lemma}","sha256_utf8":"c8c0dee5ab2e5f2312c4de750a5d08e8ea6b7185e017732a68954d661ef6f422"},{"start":89262,"end":91168,"start_line":1504,"end_line":1521,"raw_tex":"\\begin{proof}\n    The retraction at $\\bar{x}$ can be inverted locally around $0$. Indeed, as $\\D\\R_{\\bar{x}}(0_{\\tangent{\\bar{x}}{\\M}})=I$ is invertible and $\\R_{\\bar{x}}$ is $\\C^2$, the implicit function theorem provides the existence of a $\\C^2$ inverse function $\\R^{-1}_{\\bar{x}}:\\M\\to\\tangent{\\bar{x}}{\\M}$ defined locally around $\\bar{x}$. Furthermore, one shows by differentiating the relation $\\R_{\\bar{x}}\\circ\\R_{\\bar{x}}^{-1}$ that the differential of $\\R^{-1}_{\\bar{x}}$ at $\\bar{x}$ is the identity.\n\n    We consider the function $f:\\M\\to\\RR$ defined by $f(x)=\\|\\log_{\\bar{x}}(x)\\|-\\|\\R_{\\bar{x}}^{-1}(x)\\|$. Clearly $f(\\bar{x})=0$, and $\\D f(\\bar{x})=0$ as the differentials of both $\\R_{\\bar{x}}^{-1}$ and logarithm at $\\bar{x}$ are the identity. In local coordinates $\\hat{x}=\\log_{\\bar{x}}{x}$ around $\\bar{x}$, $f$ is represented by the function $\\hat{f}=f\\circ\\exp_{\\bar{x}}: \\tangent{\\bar{x}}{\\M}\\to\\RR$. As $\\hat{f}(\\hat{\\bar{x}})=0$, $\\D \\hat{f}(\\hat{\\bar{x}})=0$ and $\\hat{f}$ is $\\C^2$, there exists some $C>0$ such that\n    \\begin{equation*}\n        -C \\|\\hat{x} - \\hat{\\bar{x}}\\|^2 \\le \\hat{f}(\\hat{x}) \\le C \\|\\hat{x} - \\hat{\\bar{x}}\\|^2 \\qquad\\text{in a neighborhood $\\hat{\\U}$ of $\\hat{\\bar{x}}$,}.\n    \\end{equation*}\n    For any $\\varepsilon>0$, by taking a small enough neighborhood $\\hat{\\U}'\\subset\\hat{\\U}$, there holds\n    \\begin{equation*}\n        -\\varepsilon \\|\\hat{x} - \\hat{\\bar{x}}\\| \\le \\hat{f}(\\hat{x}) \\le \\varepsilon \\|\\hat{x} - \\hat{\\bar{x}}\\|.\n    \\end{equation*}\n    Thus for all $x$ in $\\U = \\R_{\\bar{x}}(\\hat{\\U}')$,\n    \\begin{equation*}\n        -\\varepsilon \\|\\log_{\\bar{x}}(x)\\| \\le \\|\\log_{\\bar{x}}(x)\\|-\\|\\R_{\\bar{x}}^{-1}(x)\\| \\le \\varepsilon \\|\\log_{\\bar{x}}(x)\\|,\n    \\end{equation*}\n    as $\\hat{x}=\\log_{\\bar{x}}(x)$, $\\hat{\\bar{x}}=0$. We conclude with %\n    $\\distM(x, \\bar{x})=\\|\\hat{x} - \\hat{\\bar{x}}\\| = \\|\\log_{\\bar{x}}(x)\\|$.\n\\end{proof}","sha256_utf8":"10c6acd26a14554271456d01555ecab3bf4aac551c804200158edc4d6d1f9a97"},{"start":91349,"end":91821,"start_line":1525,"end_line":1527,"raw_tex":"\\begin{proposition}\\label{prop:tangretraction}\n    Consider a $p$-dimensional $\\C^k$-submanifold $\\M$ of $\\bbR^n$ around a point $\\bar{x}\\in\\M$. The mapping $\\R : \\tangentBundle\\to\\M$, defined for $(x, \\eta)\\in\\tangentBundle$ near $(\\bar{x}, 0)$ by $\\proj_x(\\R(x, \\eta)) = \\eta$ defines a second-order retraction near $(\\bar{x}, 0)$. The point-wise retraction, defined as $\\R_x = \\R(x, \\cdot)$, is locally invertible with inverse $\\R_{x}^{-1}=\\proj_{x}$.\n\\end{proposition}","sha256_utf8":"3f115bf1cd06623e1aff53e5de4b555f8ebaf93f54370b8788fad7775a69a210"},{"start":91823,"end":94821,"start_line":1529,"end_line":1558,"raw_tex":"\\begin{proof}\n    Let $\\Psi : \\bbR^n\\to\\bbR^{n-p}$ denote a $\\C^k$ function defining $\\M$ around $\\bar{x}$: for all $x$ close enough to $\\bar{x}$, there holds $x\\in\\M \\Leftrightarrow \\Psi(x)=0$, and $\\D\\Psi(x)$ is surjective. Consider the equation $\\Phi(x, \\eta_t, \\eta_n) = 0$ around $(\\bar{x}, 0, 0)$, with\n    \\begin{equation*}\n        \\begin{array}{rl}\n            \\Phi : \\left\\{ x, \\eta_t, \\eta_n : x\\in\\M, \\eta_t\\in\\tangentM, \\eta_n\\in\\normalM \\right\\} & \\to \\bbR                       \\\\\n            x, \\eta_t, \\eta_n                                                                         & \\mapsto \\Psi(x+\\eta_t+\\eta_n).\n        \\end{array}\n    \\end{equation*}\n    The partial differential $\\D_{\\eta_n}\\Phi(\\bar{x}, 0, 0)$ is, for $\\xi_n\\in\\normalM[\\bar{x}]$,\n    \\begin{equation*}\n        \\D_{\\eta_n}\\Phi(\\bar{x}, 0, 0)[\\xi_n] = \\D\\Psi(\\bar{x})[\\xi_n].\n    \\end{equation*}\n    Since $\\bar{x}\\in\\M$, $\\D_{\\eta_n}\\Phi(\\bar{x}, 0, 0)$ is surjective from $\\normalM[\\bar{x}]$ to $\\bbR^{n-p}$ so its a bijection. The implicit function theorem provides the existence of neighborhoods $\\N_{\\bar{x}}^1\\subset\\M$, $\\N_{0}^2\\subset\\cup_{x\\in\\M} \\tangentM$, $\\N_{0}^3\\subset\\cup_{x\\in\\M}\\normalM$ and a unique $\\C^k$ function $\\eta_n : \\N_{\\bar{x}}^1 \\times \\N_0^2 \\to \\N_0^3$ such that, for all $x\\in\\N_{\\bar{x}}^1$, $\\eta_t\\in\\N_0^2$ and $\\eta_n\\in\\N_0^3$, $\\eta_n(\\bar{x}, 0) = 0$ and\n    \\begin{equation*}\n        \\Phi(x, \\eta_t, \\eta_n(x, \\eta_t)) = 0 \\Leftrightarrow x+\\eta_t+\\eta_n(x, \\eta_t) \\in \\M.\n    \\end{equation*}\n    It also provides an expression for the partial derivative of $\\eta_n$ at $(x, 0)$ along $\\eta_t$: for $\\xi_t\\in\\tangentM$,\n    \\begin{equation*}\n        \\D_{\\eta_t} \\eta_n(x, 0)[\\xi_t] = - \\left[ \\D_{\\eta_n}\\Phi(x, 0, 0) \\right]^{-1} \\D_{\\eta_t} \\Phi(x, 0, 0)[\\xi_t].\n    \\end{equation*}\n    As noted before, $\\D_{\\eta_n}\\Phi(x, 0, 0)$ is bijective since $x\\in\\M$. Besides, $\\D_{\\eta_t} \\Phi(x, 0, 0) = \\D\\Phi(x)[\\xi_t]=0$ since $\\tangentM$ identifies as the kernel of $\\D\\Phi(x)$. Thus $\\D_{\\eta_t} \\eta_n(x, 0) = 0$.\n\n    Now, define a map $\\R:\\N_{\\bar{x}}^1\\times\\N_0^2 \\to \\M$ by $\\R(x, \\eta_t) = x + \\eta_t + \\eta_n(x, \\eta_t)$. This map has degree of smoothness $\\C^k$ since $\\eta_n$ is $\\C^k$, satisfies $\\R(x, 0) = x$ since $\\eta_n(x, 0)=0$ and satisfies $\\D_{\\eta_t}\\eta_n(x, 0) = I + \\D_{\\eta_t}(x, 0) = I$. Thus $\\R$ defines a retraction on a neighborhood of $(\\bar{x}, 0)$.\n\n    We turn to show the second-order property of $\\R$. Consider the smooth curve $\\smoothcurve$ defined as $c(t) = \\R(x, t\\eta)$ for some $x\\in\\N_{\\bar{x}}^1$, $\\eta_t\\in\\tangentM\\cap\\N_0^2$. It's first derivative writes\n    \\begin{equation*}\n        c'(t) = \\eta + \\D_{\\eta_t}\\eta_n(x, t\\eta)[\\eta] = \\eta.\n    \\end{equation*}\n    The acceleration of the curve $c$ is obtained by computing the derivative of $c'(\\cdot)$ in the ambient space and then projecting onto $\\tangentM$. Thus $c''(t)=0$ and in particular, $c''(0)=0$ which makes $\\R$ a second-order retraction.\n\\end{proof}","sha256_utf8":"caaef7b94d9397e950071b85762f52f44e34a2f4714e9ec618da63bab28ace67"},{"start":94823,"end":95270,"start_line":1560,"end_line":1566,"raw_tex":"\\begin{lemma}\\label{lemma:Euclidean_to_Riemannian}\n    Consider a point $\\bar{x}$ of a Riemannian manifold $\\M$. For any $\\varepsilon>0$, there exists a neighborhood $\\U$ of $\\bar{x}$ in $\\M$ such that, for all $x\\in\\U$,\n    \\begin{equation*}\n        (1-\\varepsilon)\\distM(x, \\bar{x}) \\le \\|x-\\bar{x}\\| \\le (1+\\varepsilon)\\distM(x, \\bar{x}),\n    \\end{equation*}\n    where $\\|x-\\bar{x}\\|$ is the Euclidean distance in the ambient space.\n\\end{lemma}","sha256_utf8":"68af2ec1d70c4e8712b7652ab5afdff71a405dc5d266a13b29fcfdf525cfe0f5"},{"start":95271,"end":96476,"start_line":1567,"end_line":1581,"raw_tex":"\\begin{proof}\n    Let $\\bar{x}$, $x$ denote two close points on $\\M$. Consider the tangential retraction introduced in \\cref{prop:tangretraction}. As a retraction, it satisfies:\n    \\begin{equation*}\n        \\R_{\\bar{x}}(\\eta) = \\R_{\\bar{x}}(0) + \\D\\R_{\\bar{x}}(0)[\\eta] + \\mathcal O(\\|\\eta\\|^2) = \\bar{x} + \\mathcal O(\\|\\eta\\|^2).\n    \\end{equation*}\n    Taking $x=\\R_{\\bar{x}}(\\eta)$ allows to write $x = \\bar{x} + \\mathcal O(\\|\\R_{\\bar{x}}^{-1}(x)\\|^2)$, so that for any small $\\varepsilon_1>0$, there exists a small enough neighborhood $\\U_1\\subset\\U$ of $\\bar{x}$ in $\\M$ such that\n    \\begin{equation*}\n        (1-\\varepsilon_1) \\|\\R_{\\bar{x}}^{-1}(x)\\| \\le \\|x-\\bar{x}\\| \\le (1+\\varepsilon_1)\\|\\R_{\\bar{x}}^{-1}(x)\\|.\n    \\end{equation*}\n    By \\cref{lem:InvRetractionApproxLog}, for $\\varepsilon_2>0$ small enough, there exists a neighborhood $\\U_2\\subset\\U$ of $\\bar{x}$ such that,\n    \\begin{equation*}\n        (1-\\varepsilon_2)\\distM(x, \\bar{x}) \\le \\|\\R^{-1}_{\\bar{x}}(x)\\| \\le (1+\\varepsilon_2)\\distM(x, \\bar{x}).\n    \\end{equation*}\n    With $\\varepsilon_1$, $\\varepsilon_2$ such that $1-\\varepsilon=(1-\\varepsilon_1)(1-\\varepsilon_2)$, we combine the two estimates to conclude.%\n\\end{proof}","sha256_utf8":"b0388e057828718c41d78f8f67644744bbe4683d9611b7559e970584f6cd222f"},{"start":96929,"end":97560,"start_line":1591,"end_line":1593,"raw_tex":"\\begin{theorem}[Soundness of the Riemannian \\revise{line search}]\\label{th:unitstepsuperlinearcv}\n    Consider a manifold $\\M$ equipped with a retraction $\\R$ and a twice differentiable function $\\funman:\\M\\to\\RR$ that admits a strong local minimizer $x^\\star$, that is, a point such that $\\Hess \\funman(x^\\star)$ is positive definite. If $x$ is close to $x^\\star$, $\\eta$ brings a superlinear improvement towards $x^\\star$, that is $\\distM(\\R_x(\\eta), x^\\star) = o(\\distM(x, x^\\star))$ as $x\\to x^\\star$, and $0<m_1<1/2$, then $\\eta$ is acceptable by the Armijo rule\\;\\eqref{eq:Armijo} with unit stepsize $\\alpha=1$.\n\\end{theorem}","sha256_utf8":"33149545c985ef5da6376801b552249e1b277828db4c33f7e7bf5cfd7cce2b1f"},{"start":97562,"end":105682,"start_line":1595,"end_line":1675,"raw_tex":"\\begin{proof}\n\n    Let $x, \\eta\\in\\tangentBundle$ denote a pair such that $x$ is close to $x^\\star$ and $\\distM(\\R_x(\\eta), x^\\star) = o(\\distM(x, x^\\star))$. For convenience, let $x_+ = \\R_x(\\eta)$ denote the next point.\n\n    Following \\cite{absil2009optimization} (see e.g. the proof of Th. 6.3.2), we work in local coordinates around $x^\\star$, representing any point $x\\in\\M$ by $\\widehat{x} = \\log_{x^\\star}(x)$ and any tangent vector $\\eta\\in\\tangentM$ by $\\widehat{\\eta}_x = \\D \\log_{x^\\star}(x)[\\eta]$. The function $\\funman$ is represented by $\\widehat{\\funman} = \\funman\\circ\\exp_{x^\\star}:\\tangentM[x^\\star]\\to\\bbR$. Defining the coordinates via the logarithm grants the useful property that the Riemannian distance of any two points $x, y\\in\\M$ matches the euclidean distance between their representatives: $\\distM(x, y) = \\|\\widehat{x}-\\widehat{y}\\|$. Besides, there holds\n    \\begin{equation}\\label{eq:gradHess_to_localcoords}\n        \\D \\funman(x)[\\eta]=\\D \\widehat{\\funman}(\\widehat{x})[\\widehat{\\eta}] \\quad \\text{ and } \\quad \\Hess \\funman(x)[\\eta, \\eta]=\\D \\widehat{\\funman}(\\widehat{x})[\\widehat{\\eta}, \\widehat{\\eta}].\n    \\end{equation}\n    Indeed, $\\D \\funman(x)[\\eta] = (\\funman\\circ\\geocurve)'(0)$ and $\\Hess \\funman(x)[\\eta, \\eta] = (\\funman\\circ\\geocurve)''(0)$, where $\\geocurve$ denotes the geodesic curve defined by $\\widehat{\\geocurve}(t) = \\widehat{x} + t\\widehat{\\eta}$. Using $\\funman\\circ\\geocurve=\\widehat{\\funman}\\circ\\widehat{\\geocurve}$, one obtains the result.\n\n    \\underline{Step 1.} We derive an approximation of $\\D \\funman(x)[\\eta] = \\langle \\grad \\funman(x), \\eta\\rangle$ in terms of $\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x}-\\widehat{x^\\star}]^2$. To do so, we go through the intermediate quantity $\\D \\widehat{\\funman}(\\widehat{x}) [\\widehat{x_+}-\\widehat{x}]$, and handle precisely the $o(\\cdot)$ terms. By smoothness of $\\widehat{\\funman}$ and since $\\D \\widehat{\\funman}(\\widehat{x^\\star})=0$, Taylor's formula for $\\D \\widehat{\\funman}$ writes\n    \\begin{align*}\n        \\D \\widehat{\\funman}(\\widehat{x}) [\\widehat{x_+}-\\widehat{x}] &= \\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x_+}-\\widehat{x}, \\widehat{x}-\\widehat{x^\\star}] + o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2)\\\\\n        &= -\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x}-\\widehat{x^\\star}]^2 + \\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x_+}-\\widehat{x^\\star}, \\widehat{x}-\\widehat{x^\\star}] + o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2)\\\\\n        &= -\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x}-\\widehat{x^\\star}]^2 + o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2),\n    \\end{align*}\n    where, in the last step, we used that $\\|\\widehat{x_+}-\\widehat{x^\\star}\\| = o(\\|\\widehat{x}-\\widehat{x^\\star}\\|)$ to get that $\\|\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x_+}-\\widehat{x^\\star}, \\widehat{x}-\\widehat{x^\\star}]\\| = \\|\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})\\| \\|\\widehat{x_+}-\\widehat{x^\\star}\\|\\|\\widehat{x}-\\widehat{x^\\star}\\| = o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2)$.\n    We now turn to show that $\\D \\widehat{\\funman}(\\widehat{x}) [\\widehat{x_+}-\\widehat{x}]$ behaves as $\\D \\funman(x)[\\eta]$ up to $o(\\|\\widehat{x_+}-\\widehat{x}\\|^2)$. Since $\\D \\funman(x)[\\eta]=\\D\\widehat{\\funman}(\\widehat{x})[\\widehat{\\eta}]$ by \\eqref{eq:gradHess_to_localcoords}, there holds:\n    \\begin{equation*}\n        \\|\\D \\funman(x)[\\eta] - \\D\\widehat{\\funman}(\\widehat{x})[\\widehat{x_+} - \\widehat{x}]\\| = \\|\\D\\widehat{\\funman}(\\widehat{x})[\\widehat{\\eta} - (\\widehat{x_+} - \\widehat{x})]\\| \\le \\|\\D\\widehat{\\funman}(\\widehat{x}) \\| \\|\\widehat{\\eta} - (\\widehat{x_+} - \\widehat{x})\\|.\n    \\end{equation*}\n\n    As $\\funman$ is twice differentiable and $\\exp$ is $\\C^\\infty$, $\\widehat{\\funman}$ is twice differentiable as well. In particular its derivative is locally Lipschitz continuous, so that for $\\widehat{x}$ near $\\widehat{x^\\star}$, we obtain a first estimate:\n    \\begin{equation*}\n        \\|\\D\\widehat{\\funman}(\\widehat{x})\\| = \\|\\D\\widehat{\\funman}(\\widehat{x}) - \\D\\widehat{\\funman}(\\widehat{x^\\star})\\| = \\mathcal O (\\| \\widehat{x}-\\widehat{x^\\star} \\|).\n    \\end{equation*}\n\n    Besides, the following estimate holds $\\|\\widehat{\\eta} - (\\widehat{x_+} - \\widehat{x})\\| = o (\\| \\widehat{x}-\\widehat{x^\\star} \\|)$. Indeed, as the function $\\log_{x^\\star}\\circ\\R_x : \\tangentM\\to\\tangentM[x^\\star]$ is differentiable, there holds for $\\eta\\in\\tangentM$ small,\n    \\begin{equation*}\n        \\log_{x^\\star}(\\R_x(\\eta)) = \\log_{x^\\star}(\\R_x(0)) + \\D\\log_{x^\\star}(\\R_x(0))[\\D\\R_x(0)[\\eta]] + o(\\|\\eta\\|),\n    \\end{equation*}\n    which simplifies to $\\widehat{x_+} = \\widehat{x} + \\widehat{\\eta} + o(\\|\\eta\\|)$. \\Cref{lem:InvRetractionApproxLog} allows to write $\\|\\eta\\| = \\|\\R_x^{-1}(x_+)\\| = \\mathcal O(\\distM(x, x_+))$. Using the triangular inequality and the assumption that $\\distM(x_+, x^\\star) = o(\\distM(x, x^*))$ we get\n    \\begin{equation*}\n        \\distM(x, x_+) \\le \\distM(x, x^\\star) + \\distM(x^\\star, x_+) = \\mathcal O(\\distM(x, x^*)) = \\mathcal O(\\|\\widehat{x}-\\widehat{x^\\star}\\|),\n    \\end{equation*}\n    so that the second estimate holds.\n\n    Combining the two above estimates allows to conclude that\n    \\begin{equation*}\n        \\|\\D \\funman(x)[\\eta] - \\D\\widehat{\\funman}(\\widehat{x})[\\widehat{x_+} - \\widehat{x}]\\| = o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2),\n    \\end{equation*}\n    so that overall,\n    \\begin{equation*}\n        \\D \\funman(x)[\\eta] = \\D \\widehat{\\funman}(\\widehat{x}) [\\widehat{x_+}-\\widehat{x}] + o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2) = -\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x}-\\widehat{x^\\star}]^2 + o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2).\n    \\end{equation*}\n\n\n\n    Using that $\\|\\widehat{x}-\\widehat{x^\\star}\\| = \\distM(x, x^\\star)$ and $\\D^2 \\widehat{\\funman}(\\widehat{x^\\star}) = \\Hess \\funman(x^\\star)$ \\eqref{eq:gradHess_to_localcoords}, we obtain\n    \\begin{equation}\\label{eq:proofunitstepsize_gradhess}\n        \\D \\funman(x)[\\eta] = -\\Hess \\funman(x^\\star)[\\widehat{x}-\\widehat{x^\\star}]^2 + o(\\distM(x, x^\\star)^2).\n    \\end{equation}\n\n    \\underline{Step 2.} The function $\\funman$ admits a second-order development around $x^\\star$: applying \\cref{eq:manifoldSecondOrderDevSecRetraction} with the exponential map $\\exp_{x^\\star}$ as a second-order retraction yields\n    \\begin{equation}\\label{eq:proofunitstepsize_fdev}\n        \\funman(x) = \\funman(x^\\star) + \\frac{1}{2} \\Hess \\funman(x^\\star)[\\widehat{x}-\\widehat{x^\\star}]^2 + o(\\distM(x, x^\\star)^2),\n    \\end{equation}\n    where we used that $\\distM(x, x^\\star) = \\|\\log_{x^\\star}(x)-\\log_{x^\\star}(x^\\star)\\|$. Denote $0 < l \\le L$ the lower and upper eigenvalues of $\\Hess \\funman(x^\\star)$. The combination $\\eqref{eq:proofunitstepsize_fdev} + m_1 \\eqref{eq:proofunitstepsize_gradhess}$ writes\n    \\begin{align*}\n        \\funman(x) + m_1 \\D \\funman(x)[\\eta] &= \\funman(x^\\star) + (\\frac{1}{2}-m_1) \\Hess \\funman(x^\\star)[\\widehat{x}-\\widehat{x^\\star}]^2 + o(\\distM(x, x^\\star)^2) \\\\\n        &\\ge \\funman(x^\\star) + (\\frac{1}{2}-m_1) l \\distM(x, x^\\star)^2 + o(\\distM(x, x^\\star)^2),\n    \\end{align*}\n\n    Let $\\varepsilon >0$ such that $\\frac{1}{2}L \\varepsilon^2  < (\\frac{1}{2}-m_1) l$. As $\\distM(x_+, x^\\star)=o(\\distM(x, x^\\star))$, for $x$ close enough to $x^\\star$ there holds $\\distM(x_+, x^\\star) \\le \\varepsilon \\distM(x, x^\\star)$. Combining this with the second-order development of $f$ at $x_+$, there holds:\n    \\begin{align*}\n        \\funman(x_+) &= \\funman(x^\\star) + \\frac{1}{2} \\Hess \\funman(x^\\star)[\\widehat{x_+}-\\widehat{x^\\star}]^2 + o(\\distM(x_+, x^\\star)^2) \\\\\n        &\\le \\funman(x^\\star) + \\frac{1}{2}L \\distM(x_+, x^\\star)^2 + o(\\distM(x_+, x^\\star)^2)\\\\\n        &\\le \\funman(x^\\star) + \\frac{1}{2}L \\varepsilon^2 \\distM(x, x^\\star)^2 + o(\\distM(x, x^\\star)^2).\n    \\end{align*}\n\n    Subtracting the two estimates yields\n    \\begin{equation*}\n        \\funman(x_+) - (\\funman(x) + m_1 \\D \\funman(x)[\\eta]) \\le \\left(\\frac{1}{2}L \\varepsilon^2 - (\\frac{1}{2}-m_1) l \\right) \\distM(x, x^\\star)^2 + o(\\distM(x, x^\\star)^2),\n    \\end{equation*}\n    which ensures that the Armijo condition is satisfied.\n\n\n\n\n\n\\end{proof}","sha256_utf8":"03fbdd0077dcc1116c80fd235f1bfe340748428e57ba066b1faf4b7a505f8d7f"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"交替近端与牛顿法的二次收敛","source_units":[{"start":62651,"end":63795,"start_line":1051,"end_line":1068,"raw_tex":"\\begin{theorem}\\label{th:Newton}\n    Let \\cref{asm:fun} hold and take $\\gamma\\in(0, 1/L)$. Consider the sequence of iterates $(x_k)$ generated by \\cref{alg:generic} equipped with the Riemannian Newton manifold update (\\cref{alg:ManUpNewton}). If $~ \\Hess \\funtot(x_k)$ is positive definite at each step, then all limit points of $(x_k)$ are critical points of $\\funtot$ and share the same functional value.\n\n    Furthermore, assume that the sequence $(y_k)$ admits %\n    a limit point $x^\\star$ such\\;that\n    \\begin{itemize}\n        \\item[i)] $x^\\star\\in\\M$ is a $r$-structured critical point for $(f,g)$  with $r<1/\\gamma$;\n        \\item[ii)] $\\Hess_{\\M} \\funtot(x^\\star)\\succ 0$ and  $\\Hess_{\\M} \\funtot$ is locally Lipschitz around $x^\\star$.\n    \\end{itemize}\n    Then, after some finite time,\n    \\begin{itemize}\n        \\item[a)] the sequence $(x_k)$ lies on $\\M$;\n        \\item[b)] $x_k$ converges to $x^\\star$ quadratically{: for large $k$, there exists $c>0$ such that}\n            \\begin{equation*}\n                \\distM(x_{k+1}, x^\\star) \\le {c ~} \\distM(x_k, x^\\star)^2.\n            \\end{equation*}\n    \\end{itemize}\n\\end{theorem}","sha256_utf8":"dbf3100410ae9ede761bfb287eb17ec954b4e55724250c7722aa0e1b16297146"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"二次收敛定理完整证明","source_units":[{"start":63797,"end":66141,"start_line":1070,"end_line":1099,"raw_tex":"\\begin{proof}\n    As the Riemannian Hessian is assumed to be positive definite, Newton's direction is a descent direction:\n    \\begin{equation*}\n        \\langle \\grad \\funtot(x_k), d_k \\rangle = -\\langle \\grad \\funtot(x_k), \\Hess \\funtot(x_k)^{-1} \\grad \\funtot(x_k) \\rangle < 0.\n    \\end{equation*}\n    The Riemannian Newton manifold step is therefore well-defined, and the \\revise{line search} terminates by \\cref{lemma:LS_exists_valid_stepsizes}, so that the manifold update is well-defined and provides descent \\revise{($F(y_{k})\\le F(x_{k})$)}. \n\n\n    \\revise{Now, since the proximal gradient update provides a descent (see \\cite[Lem. 10.4]{beck2017first}),\n    \\begin{equation}\\label{eq:proofglobalalg_descent}\n        F(x_{k+1}) \\le F(y_k) - \\frac{1-\\gamma L}{2\\gamma} \\| x_{k+1}-y_k \\|^2 \\le F(x_k) - \\frac{1-\\gamma L}{2\\gamma} \\| x_{k+1}-y_k \\|^2.\n    \\end{equation}\n    The sequence $(F(x_k))$ is thus non-increasing and lower-bounded, therefore it converges.\n    Besides, any accumulation point of $(x_k)$ is a critical point of $F$. Indeed, summing equation \\eqref{eq:proofglobalalg_descent} for $k=1, \\ldots, n$ yields:\n    \\begin{equation*}\n        \\frac{1-\\gamma L}{2\\gamma} \\sum_{k=1}^n \\| x_{k+1}-y_k \\|^2 \\le F(x_1) - F(x_{n+1}) \\le F(x_1)-\\inf F < +\\infty.\n    \\end{equation*}\n    Since $ \\dist (0, \\partial F(x_{k+1})) \\le \\frac{L\\gamma + 1}{\\gamma}\\|x_{k+1}-y_k\\|$ (see e.g. the proof of\\;\\cite[Prop.\\;13]{bolte2015error}), we have that \n    $\\dist (0, \\partial F(x_{k+1}))$\n    converges to $0$. The outer-semi continuity of the limiting subdifferential then yields criticality of accumulation points.}\n\n\n\n\n\n    Now we apply the local convergence of Riemannian Newton \\cite[Th. 6.3.2]{absil2009optimization}:\n    assumption ii) ensures that\n    the Riemannian Newton direction $d$ computed in step\\;1 of\\;\\cref{alg:ManUpNewton} provides a quadratic improvement %\n    on a neighborhood of $x^\\star$ on\\;$\\M$. Moreover, the \\revise{line search} returns the unit-stepsize after some finite time: $\\alpha=1$ is tried first, and is acceptable for directions providing superlinear improvement by \\cref{lemma:LS_unit_stepsize}. Thus the whole Riemannian Newton update provides quadratic improvement after some finite time. Using this and assumption i), \\cref{th:identiflocalcv_sperlin} applies and yields the results.\n\\end{proof}","sha256_utf8":"848ac24f841a1f40a87201227b7af239bd4661ea99fcec15f4e58b6690112fad"}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","title":"结构化聚点与局部海森条件","source_units":[{"start":63059,"end":63416,"start_line":1054,"end_line":1059,"raw_tex":"    Furthermore, assume that the sequence $(y_k)$ admits %\n    a limit point $x^\\star$ such\\;that\n    \\begin{itemize}\n        \\item[i)] $x^\\star\\in\\M$ is a $r$-structured critical point for $(f,g)$  with $r<1/\\gamma$;\n        \\item[ii)] $\\Hess_{\\M} \\funtot(x^\\star)\\succ 0$ and  $\\Hess_{\\M} \\funtot$ is locally Lipschitz around $x^\\star$.\n    \\end{itemize}\n","sha256_utf8":"cdfd949d81cb0666fa3be3c81c3cef026b6098729a8f845fa75ca894b1092e1e"}]},{"id":"S004","semantic_type":"problem_setting","kind":"problem_setting","title":"复合非光滑优化问题","source_units":[{"start":11512,"end":11637,"start_line":391,"end_line":393,"raw_tex":"\\begin{equation}\\tag{$\\mathcal P$} \\label{pb:compositepb}\n    \\min_{x\\in\\mathbb R^n} F(x) \\triangleq f(x)+g(x)\n\\end{equation}","sha256_utf8":"eae5d284ff3c6a0eaa73eb6a9f325c9fcc60ddbcee29f25005ad2340d34e08ae"}]},{"id":"S005","semantic_type":"assumption","kind":"assumption","title":"目标函数基本假设","source_units":[{"start":38431,"end":38844,"start_line":708,"end_line":716,"raw_tex":"\\begin{assumption}\\label{asm:fun}\n    The functions $f$ and $g$ are proper and \n    \\begin{itemize}\n        \\item[i)] $f$ is $\\C^2(\\bbR^n)$ with an $L$-Lipschitz continuous gradient;\n        \\item[ii)] $g$ is lower semi-continuous;\n        \\item[iii)] $\\prox_{\\gamma g}$ is non-empty on $\\mathbb R^n$ for any $\\gamma>0$;\n        \\item[iv)] $F(x) = f(x) + g(x)$ is bounded below.\n    \\end{itemize}\n\\end{assumption}","sha256_utf8":"b094b90e4a82afeb4132a0aaa7efc6f5415fea29dfb226b10878c278a8e45bb3"}]},{"id":"S006","semantic_type":"algorithm","kind":"algorithm","title":"近端梯度与流形更新交替","source_units":[{"start":39849,"end":40322,"start_line":729,"end_line":738,"raw_tex":"\\begin{algorithm}\n    \\caption{General structure exploiting algorithm\\label{alg:generic}}\n    \\begin{algorithmic}[1]\n        \\Require \\revise{Pick $x_{0}$ arbitrary, $\\gamma < 1 / L$.}\n        \\Repeat\n        \\State Compute $x_{k}\\in\\prox_{\\gamma g}(y_{k-1} - \\gamma \\nabla f(y_{k-1}))$ and get $\\M_{k}\\ni x_{k}$ %\n        \\State Update $y_{k} = \\manupdate_{\\M_{k}} (x_{k})$ on the current manifold %\n        \\Until{stopping criterion}\n    \\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"e5634b8ce6bbba4498cbecced71fd8cbad0f1e548d2ab16a28f06cb6945b900c"}]},{"id":"S007","semantic_type":"algorithm","kind":"algorithm","title":"黎曼牛顿流形更新","source_units":[{"start":62137,"end":62648,"start_line":1037,"end_line":1048,"raw_tex":"\\begin{algorithm}\n    \\caption{$\\manupdate$-Newton\\label{alg:ManUpNewton}}\n    \\begin{algorithmic}[1]\n        \\Require Manifold $\\M$, point $x\\in\\M$\n        \\State Find $d$ in $\\tangentM$ that solves\n        \\begin{equation*}\\tag{Newton equation}\\label{eq:Newtoneq}\n            \\grad F(x) + \\Hess F(x)[d] = 0\n        \\end{equation*}\n        \\State Find $\\alpha$ satisfying the Armijo condition \\eqref{eq:Armijo} with direction $d$\n        \\State \\Return $y = R_x(\\alpha d)$\n    \\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"e036c4af6e54f74427477eaaf8bb0062abde8b38daf799980efe4830e989ad7f"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"阿米霍接受条件","source_units":[{"start":58063,"end":58424,"start_line":987,"end_line":990,"raw_tex":"Standing at point $x\\in\\M$ with a proposed direction $\\eta\\in\\tangentM$, a stepsize $\\alpha>0$ is \\emph{acceptable} \\revise{if} it satisfies the following \\emph{Armijo} condition %\n\\begin{equation}\\label{eq:Armijo}\n    \\funtot(\\R_x(\\alpha \\eta)) \\le \\funtot(x) + m_1 \\alpha \\langle \\grad \\funtot(x), \\eta\\rangle, \\qquad\\text{for $0 < m_1 < 1/2$.}\n\\end{equation}","sha256_utf8":"ef915adf2f06e30edcc90f0ec29ac5a42f371d769483822f5f84d50169eda699"},{"start":58425,"end":58958,"start_line":991,"end_line":991,"raw_tex":"The line search employs a second-order retraction $\\R_x$, \\revisebis{e.g.\\;the exponential map, a projection retraction \\cite{Absil2012ProjectionlikeRO}, or any other second-order retraction \\cite{boumal2022intromanifolds}.\\footnote{\\revisebis{Indeed, in many applications of Riemannian optimization, computing geodesics and the exponential map can be costly and then retractions provide an efficient alternative. For this reason, we consider here second-order retractions \\cite{absil2009optimization,boumal2022intromanifolds}.}}} %\n","sha256_utf8":"a07d85dc76e8382bb4cebd0f18f2a08f5ca08513d8b4504a3307f229f3bf76c8"}]},{"id":"S009","semantic_type":"algorithm","kind":"algorithm","title":"从单位步长开始回溯","source_units":[{"start":61257,"end":61611,"start_line":1015,"end_line":1016,"raw_tex":"In the following, we will consider a \\emph{backtracking} \\revise{line search} for finding an acceptable stepsize $\\alpha$: the unit stepsize is first tried, and then the search space is reduced geometrically. %\nIn practice, we use exactly \\cite[Alg. A6.3.1]{dennis1996numerical}, which features polynomial interpolation of $\\funman$ in the search space.\n","sha256_utf8":"38fcf8c0a8abf063993d487c1ec3731cb332fa03aedbe0f794e971b2d055f75b"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"子流形切空间与距离","source_units":[{"start":18060,"end":18160,"start_line":460,"end_line":460,"raw_tex":"\\revise{In the rest of the paper, $\\M$ denotes a submanifold of $\\bbR^{n}$ or $\\bbR^{m \\times n}$.}\n","sha256_utf8":"ce58f5b3ea502169d2177c909e98e0764707f216a0445821bc1cf47bdc094e86"},{"start":18202,"end":21205,"start_line":464,"end_line":483,"raw_tex":"A subset $\\M$ of $\\RR^n$ is said to be a \\emph{$p$-dimensional $\\C^2$-submanifold} of $\\RR^n$ around $\\bar{x} \\in \\M$ if there exists %\n$\\C^2$ function $\\varphi : \\RR^p \\to \\RR^n$ such that $\\varphi$ \\revise{maps} a neighborhood of $0 \\in \\RR^p$ \\revise{to} a neighborhood of $\\bar{x} \\in\\M$, \\revise{that admits a smooth (local) inverse,} and \\revise{which derivative} at $\\varphi^{-1}(\\bar{x})=0$ is injective.\nA $p$-dimensional $\\C^2$-submanifold of $\\RR^n$ can alternatively be defined via a local equation, that is, a $\\C^2$ function $\\Phi : \\RR^n \\to \\RR^{n-p}$ with a surjective derivative at $\\bar x \\in \\M$ that satisfies for all $x$ close enough to $\\bar x$: $x \\in\\M \\Leftrightarrow \\Phi(x) = 0$.\n\n\nA basic tool to investigate approximations on manifolds is notion of the \\emph{smooth curves}. A smooth curve on $\\M$ is a $\\C^2$ application $\\gamma : I\\subset \\RR \\to \\M\\subset \\RR^n$, where $I$ is an open interval containing $0$. At each point $x\\in\\M$, the \\emph{tangent space}, noted $\\tangentM$, can be defined as the velocities of all smooth curves passing by $x$ at $0$:\n\\begin{equation*}\n    \\tangentM \\triangleq \\left\\{ \\smoothcurve'(0) ~|~ \\smoothcurve : I \\to \\M \\text{ is a smooth curve around $0$ and } \\smoothcurve(0)=x \\right\\}.\n\\end{equation*}\nThe tangent space is a $p$-dimensional %\nspace containing \\emph{tangent vectors}. Each tangent space $\\tangentM$ is equipped with a scalar product $\\langle\\cdot, \\cdot\\rangle_x : \\tangentM\\times\\tangentM\\to\\RR$, and the associated norm $\\|\\cdot\\|_x$. In many cases, the tangent metric varies smoothly with $x$, making the manifold \\emph{Riemannian}. In this paper, we use the ambient space scalar product to define the scalar product on tangent spaces; we will thus drop the subscript in the tangent scalar product and norm notations when there is no confusion possible. Related to the tangent space, we will also consider the \\emph{normal space} $\\normalM$ at $x\\in\\M$, defined as the orthogonal space to $\\tangentM $ in $\\RR^n$, and the \\emph{tangent bundle manifold} defined by:\n\\[\n\\tangentBundle \\triangleq \\bigcup_{x\\in\\M}(x,\\tangentM).\n\\]\nNote also that both tangent and normal spaces at $x\\in\\M$  admit explicit expressions from derivatives of local parametrization $\\varphi$ or local equations $\\Phi$ defining $\\M$:\n\\begin{equation*}\n    \\tangentM = \\text{Im}~ \\D_\\varphi (0) = \\text{Ker}~ \\D_\\Phi (x) \\qquad \\normalM = \\text{Ker}~ \\D_\\varphi (0)^* = \\text{Im}~ \\D_\\Phi (x)^*\n\\end{equation*}\n\nA \\emph{metric} on $\\M$ can be defined as the minimal length over all curves joining two points $x, y\\in\\M$, ie. $\\distM(x, y) = \\inf_{\\smoothcurve \\in C_{x,y}}  \\int_0^1 \\|\\smoothcurve'(t)\\|_{\\smoothcurve(t)} \\mathrm{d}t$, where $C_{x,y}$ is the set of $[0, 1]\\to\\M$ smooth curves $\\smoothcurve$ such that $\\smoothcurve(0)=x$, $\\smoothcurve(1)=y$. The minimizing curves generalize the notion of straight line between two points to manifolds. The constant speed parametrization of any minimizing curve is called a \\emph{geodesic}.\n","sha256_utf8":"5c0c38c60d43885b47a7782ec85acfbe38fe6199bf7a05df73cb7400da9799fc"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"黎曼梯度与海森算子","source_units":[{"start":21267,"end":24071,"start_line":487,"end_line":501,"raw_tex":"Let $\\funman:\\M\\to\\RR$, the \\emph{Riemannian differential} of $\\funman$ at\\;$x$ is the linear operator $\\D \\funman(x):\\tangentM\\to\\RR$ defined by $\\D \\funman(x)[\\eta]\\triangleq\\left. \\frac{\\mathrm{d}}{\\mathrm{d}t}\\funman\\circ \\smoothcurve(t) \\right|_{t=0}$, where $\\smoothcurve$ is a smooth curve such that $\\smoothcurve(0)=x$ and $\\smoothcurve'(0)=\\eta$. In turn, the \\emph{Riemannian gradient} $\\grad \\funman(x)$ is the unique vector of $\\tangentM$ such that, for any tangent vector\\;$\\eta$, $\\D \\funman(x)[\\eta] = \\langle \\grad \\funman(x), \\eta \\rangle$.\nIf $\\grad \\funman(x)$ exists, a first order Taylor development can be formulated. Let $x\\in\\M$, $\\eta\\in\\tangentM$ and $\\smoothcurve$ denote a smooth curve passing by $x$, with velocity $\\eta$ at $0$; then, for $t$ near $0$,\n\\begin{equation*}\\label{eq:manifoldFirstOrderDevSmoothCurve}\n    \\funman\\circ \\smoothcurve(t) = \\funman(x) + t \\langle \\grad \\funman(x), \\eta \\rangle + o(t).\n\\end{equation*}\n\nNotions of derivation for vector fields and of acceleration for curves are used to define second-order objects. Let a curve $\\smoothcurve:I\\to\\M$ and a smooth vector field $Z$ on $\\smoothcurve$, ie.\\;a smooth map such that $Z(t)\\in\\tangentM[c(t)]$ for $t\\in I$. The \\emph{covariant derivative} of $Z$ on the curve $\\smoothcurve$, denoted $\\frac{\\D}{\\mathrm{d}t}Z:I\\to\\tangentBundle$, is defined by $\\frac{\\D}{\\mathrm{d}t}Z(t) \\triangleq \\proj_{\\smoothcurve(t)} Z'(t)$, where $Z'(t)$ denotes the derivative in the ambient space $\\bbR^n$ and $\\proj_{x}$ corresponds to the orthogonal projector from $\\RR^n$ to $\\tangentM$. The \\emph{acceleration} of a curve $\\smoothcurve$ is defined as the covariant derivative of its velocity: $\\smoothcurve''(t) \\triangleq \\frac{\\D}{\\mathrm{d}t}\\smoothcurve'(0)$. \n\nThe \\emph{Riemannian Hessian}  of $\\funman$ at $x$ along $\\eta$ is the linear operator $\\Hess \\funman(x):\\tangentM\\to\\tangentM$ defined by the relation $\\Hess \\funman(x)[\\eta] \\triangleq \\left. \\frac{\\D}{\\mathrm{d}t} \\grad \\funman(\\smoothcurve(t)) \\right|_{t=0}$, where $\\smoothcurve$ is a smooth curve such that $\\smoothcurve(0)=x$, $\\smoothcurve'(0)=\\eta$. \nEquivalently, we have\n$\\langle\\Hess \\funman(x)[\\eta], \\eta\\rangle = \\left. \\frac{\\mathrm{d}^2}{\\mathrm{d}t^2}\\funman\\circ \\geocurve(t) \\right|_{t=0}$, where $\\geocurve$ is a geodesic such that $\\geocurve(0)=x$, $\\gamma'(0) = \\eta$.\nA second order Taylor development can now be formulated. Let $x\\in\\M$, $\\eta\\in\\tangentM$, and $\\smoothcurve$ be a smooth curve such that $\\smoothcurve(0)=x$, $\\smoothcurve'(0)=\\eta$. Then, for $t$ near $0$,\n\\begin{equation*}%\n    \\funman\\!\\circ\\!\\smoothcurve(t) = \\funman(x) \\!+ t \\langle \\grad \\funman(x), \\eta \\rangle + \\frac{t^2}{2}\\!(\\langle \\Hess \\funman(x)[\\eta], \\eta \\rangle\\!+\\!\\langle \\grad \\funman(x), c''(0)\\rangle)+ o(t^2).\n\\end{equation*}\n","sha256_utf8":"4c798af3142bd9c159465a3f6e4740264bc3925b9f93f584abb9987dce86a650"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"回缩与二阶展开","source_units":[{"start":24650,"end":25933,"start_line":513,"end_line":523,"raw_tex":"\\noindent\\textbf{Algorithms on manifolds: retractions and convergence rates.}\nIterative Riemannian methods require a way to produce curves on $\\M$ given a point $x$ and a tangent vector $\\eta$. A geodesic curve passing at $(x, \\eta)$, while attractive as the generalization of the straight line, has a prohibitive computational cost. We thus \\emph{retractions}, i.e. approximations of it, defined on a manifold $\\M$ as a smooth map $\\R : \\tangentBundle \\to \\M$ such that\n\\[%\n    \\text{$\\R_x (0) = x\\quad$ and $\\quad \\D\\R_x (0) : \\tangentM \\to\\tangentM$ is the identity map: $D\\R_x (0 )[v] = v$,}\n\\]%\nwhere, for each $x \\in\\M$, $\\R_x : \\tangentM\\to\\M$ is defined as the restriction of $\\R$ at $x$, so that $\\R_x(v) = \\R(x, v)$. A \\emph{second-order retraction} is a retraction $\\R$ such that, for all $(x, \\eta)\\in\\tangentBundle$, the curve $c(t)=\\R_x(t\\eta)$ has zero acceleration at 0: $c''(0)=0$.\nThus $t\\mapsto\\R_x(t\\eta)$ is a practical curve passing by $(x, \\eta)$ at $0$, and provides a similar development as above: %\nfor $t$ near 0,\n\\begin{equation}\\label{eq:manifoldSecondOrderDevSecRetraction}\n    \\funman\\circ\\R_x(t \\eta) = \\funman(x) + t \\langle \\grad \\funman(x), \\eta\\rangle + \\frac{t^2}{2}\\langle \\Hess \\funman(x)[\\eta], \\eta\\rangle + o(t^2 \\|\\eta\\|^2).\n\\end{equation}\n","sha256_utf8":"02e776574ca9cc55006c23f43413297a3430ac2f497b1bd30157dd2ab0fcb413"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"流形上的收敛阶","source_units":[{"start":25935,"end":26615,"start_line":526,"end_line":534,"raw_tex":"Finally, the convergence rates on manifolds are defined as follows. A sequence of points $(x_k)$ \\emph{converges (Q-)linearly} to some point $\\bar{x}\\in\\M$ if there exist an integer $K>0$ and a constant $q\\in(0,1)$ such that, for all $k\\ge K$, there holds\n\\begin{equation*}\n    \\distM(x_{k+1}, \\bar{x})\\le q~ \\distM(x_{k}, \\bar{x}).\n\\end{equation*}\nThe sequence \\emph{converges with order at least $p$} if there exists an integer $K>0$ and a constant $q\\in(0,1)$ such that, for all $k\\ge K$, there holds\n\\begin{equation*}\n    \\distM(x_{k+1}, \\bar{x})\\le q~\\distM(x_{k}, \\bar{x})^p.\n\\end{equation*}\nThe convergence is \\emph{superlinear} when $p>1$ and \\emph{quadratic} when $p=2$.\n","sha256_utf8":"cfc653e66916546560442512e41eb556445930d1df329d5f6d501971abdaadeb"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"次微分与正则性","source_units":[{"start":30057,"end":31529,"start_line":586,"end_line":599,"raw_tex":"Consider a point $\\bar{x}$ with $\\funns(\\bar{x})$ finite. The set of \\emph{regular subgradients} %\n\\begin{equation*}\n    \\widehat\\partial \\funns(\\bar{x}) \\triangleq \\left\\{ v : \\funns(x) \\ge \\funns(\\bar{x}) + \\langle v, x-\\bar{x} \\rangle + o(\\|x-\\bar{x}\\|) \\text{ for all } x\\in\\bbR^n \\right\\}\n\\end{equation*}\nis closed and convex, but the subdifferential mapping $\\widehat\\partial \\funns(\\cdot)$ may not be outer semi-continuous \\cite[Th. 8.6, Prop. 8.7]{rockafellar2009variational}. To overcome this problem, the set of \\emph{(general or limiting) subgradients} is defined as\n\\begin{equation*}\n    \\partial \\funns(\\bar{x}) \\triangleq \\left\\{ \\lim_r v_r : v_r\\in\\widehat\\partial \\funns(x_r), x_r\\to\\bar{x}, \\funns(x_r)\\to \\funns(\\bar{x}) \\right\\}.\n\\end{equation*}\nThe limiting subdifferential is by design outer semi-continuous:\n\\[\n\\limsup_{x\\to\\bar{x}} \\partial \\funns (x) = \\{u : \\exists x_r\\to\\bar{x}, \\exists u_r \\to u \\text{ with } u_r\\in  \\partial \\funns (x_r) \\} ~\\subset~ \\partial \\funns (\\bar{x}),\n\\]\nwhich is an attractive property to study the properties of sequences of points whose subgradients converge.\nWe say that a function is \\emph{(Clarke) regular} at $\\bar{x}$ if the regular and limiting subdifferentials at $\\bar{x}$ coincide \\cite[Def. 7.25, Cor. 8.11]{rockafellar2009variational}. This is notably the case for convex functions where the two above definitions coincide with the convex subdifferential \\cite[Prop. 8.12]{rockafellar2009variational}.\n","sha256_utf8":"08ba300b62da1f464007eda5f41cea7d1053f889f042dbecc3a704a7cfdff6c8"}]},{"id":"S015","semantic_type":"definition","kind":"definition","title":"复合目标的临界点","source_units":[{"start":31600,"end":32100,"start_line":603,"end_line":607,"raw_tex":"The subdifferential allows to derive optimality conditions: for a local minimizer $\\bar x$ of $F$, we have $0\\in \\partial F(\\bar x)$. For the objective function of \\eqref{pb:compositepb}, this writes\n\\[\n0\\in \\nabla f (\\bar x) + \\partial g(\\bar x) \\qquad\\text{or equivalently}\\qquad -\\nabla f (\\bar x) \\in \\partial g(\\bar x).\n\\]\nA point satisfying these conditions is called a critical point. The analysis of the algorithms of this paper will provide convergence guarantees towards critical points. %\n","sha256_utf8":"3c882cb445555b7edcc83cc961181510da6aaa9ffab65adce7ef5e29930bb02c"}]},{"id":"S016","semantic_type":"definition","kind":"definition","title":"近端映射","source_units":[{"start":32148,"end":32472,"start_line":611,"end_line":615,"raw_tex":"A central tool to tackle non-differentiable functions is the \\emph{proximity operator}.\nFor\\;$\\gamma\\!>\\!0$ and a function $\\funns$; it is defined as the set-valued mapping\n\\begin{equation*}\n    \\prox_{\\gamma \\funns}(y) \\triangleq \\argmin_{u\\in\\bbR^n} \\left\\{ \\funns(u) + \\frac{1}{2\\gamma}\\|u-y\\|^2\\right\\}.\n\\end{equation*}\n","sha256_utf8":"1d24b438bbe3405f60426b77fa288b6364c98f643f123298641b5ee30a90a220"}]},{"id":"S017","semantic_type":"definition","kind":"definition","title":"近端正则性","source_units":[{"start":33259,"end":33914,"start_line":625,"end_line":630,"raw_tex":"A function $\\funns$ is \\emph{prox-regular} at a point $\\bar{x}$ for a subgradient $\\bar{v} \\in \\partial \\funns(\\bar{x})$ if $\\funns$ is finite, locally lower semi-continuous at $\\bar{x}$, and there exists $r>0$ and $\\varepsilon>0$ such~that\n$%\n    \\funns(x') \\ge \\funns(x) + \\langle v, x'-x\\rangle - \\frac{r}{2}\\|x'-x\\|^2\n$ %\nwhenever $v \\in \\partial \\funns(x)$, $\\|x-\\bar{x}\\|<\\varepsilon$, $\\|x'-\\bar{x}\\|<\\varepsilon$, $\\|v-\\bar{v}\\|<\\varepsilon$ and $\\funns(x)<\\funns(\\bar{x})+\\varepsilon$.\nWhen this holds for all $\\bar{v} \\in\\partial \\funns(\\bar{x})$, we say that $\\funns$ is prox-regular at $\\bar{x}$ \\cite[Def. 13.27]{rockafellar2009variational}.\n","sha256_utf8":"d5dfffff8cb59effe6bc65bd06d44465f80fa85ce9c4df07ab9c93371b44b598"}]},{"id":"S018","semantic_type":"definition","kind":"definition","title":"部分光滑性","source_units":[{"start":40542,"end":41243,"start_line":755,"end_line":763,"raw_tex":"\\begin{definition}[partial smoothness]\n    A function $g$ is ($\\C^2$-)\\emph{partly smooth} at a point $\\bar{x}$ relative to a set $\\M$ containing $\\bar{x}$ if $\\M$ is a $\\C^2$ manifold around $\\bar{x}$ and:\n    \\begin{itemize}\n        \\item (smoothness) the restriction of $g$ to $\\M$ is a $\\C^2$ function near $\\bar{x}$;\n        \\item (regularity) $g$ is (Clarke) regular at all points $x\\in\\M$ near $\\bar{x}$, with $\\partial g(x)\\neq\\emptyset$;\n        \\item (sharpness) the affine span of $\\partial g(\\bar{x})$ is a translate of $\\normalM[\\bar{x}]$;\n        \\item (sub-continuity) the set-valued mapping $\\partial g$ restricted to $\\M$ is continuous at $\\bar{x}$.\n    \\end{itemize}\n\\end{definition}","sha256_utf8":"ef53f97690627049d4e8996cde379c4f6ed1cd9ef89e14dbf4520fc8b83c11e6"}]},{"id":"S019","semantic_type":"definition","kind":"definition","title":"结构化临界点","source_units":[{"start":48752,"end":49329,"start_line":836,"end_line":844,"raw_tex":"\\begin{definition}\\label{def:structqual}\n   A point $\\bar{x}$ of a $C^2$ submanifold $\\M$ is a $r$-\\emph{structured critical} point for $(f,g)$~if we have:\n    \\begin{itemize}\n        \\item[i)] proximal gradient stability: $\\bar{x} = \\prox_{g/r}(\\bar{x}-1/r \\nabla f( \\bar{x} ) )$ ;\n        \\item[ii)] qualification condition: $0 \\in \\ri (\\nabla f + \\partial g)(\\bar{x})$;\n        \\item[iii)] prox-regularity: $g$ is $r$-prox-regular at $\\bar{x}$;\n        \\item[iv)] partial smoothness: $g$ is partly-smooth at $\\bar{x}$ with respect to $\\M$.\n    \\end{itemize}\n\\end{definition}","sha256_utf8":"ba0853430fb160cda17c0535b85d0b60f08963c788a6a621c6b2adac931bbbf9"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"牛顿方向下降且线搜索终止","source_units":[{"start":63811,"end":64348,"start_line":1071,"end_line":1075,"raw_tex":"    As the Riemannian Hessian is assumed to be positive definite, Newton's direction is a descent direction:\n    \\begin{equation*}\n        \\langle \\grad \\funtot(x_k), d_k \\rangle = -\\langle \\grad \\funtot(x_k), \\Hess \\funtot(x_k)^{-1} \\grad \\funtot(x_k) \\rangle < 0.\n    \\end{equation*}\n    The Riemannian Newton manifold step is therefore well-defined, and the \\revise{line search} terminates by \\cref{lemma:LS_exists_valid_stepsizes}, so that the manifold update is well-defined and provides descent \\revise{($F(y_{k})\\le F(x_{k})$)}. \n","sha256_utf8":"e916ff19d7bd78ca85c8ff697fc1734636598a43b2fee2b9ee98cc789312e0c3"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"充分下降与目标值收敛","source_units":[{"start":64350,"end":64767,"start_line":1078,"end_line":1082,"raw_tex":"    \\revise{Now, since the proximal gradient update provides a descent (see \\cite[Lem. 10.4]{beck2017first}),\n    \\begin{equation}\\label{eq:proofglobalalg_descent}\n        F(x_{k+1}) \\le F(y_k) - \\frac{1-\\gamma L}{2\\gamma} \\| x_{k+1}-y_k \\|^2 \\le F(x_k) - \\frac{1-\\gamma L}{2\\gamma} \\| x_{k+1}-y_k \\|^2.\n    \\end{equation}\n    The sequence $(F(x_k))$ is thus non-increasing and lower-bounded, therefore it converges.\n","sha256_utf8":"7c04db91fd978709f428dcc259856624ee04c13766020bc8f6232e4ad9e61189"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"残差消失与聚点临界性","source_units":[{"start":64767,"end":65422,"start_line":1083,"end_line":1089,"raw_tex":"    Besides, any accumulation point of $(x_k)$ is a critical point of $F$. Indeed, summing equation \\eqref{eq:proofglobalalg_descent} for $k=1, \\ldots, n$ yields:\n    \\begin{equation*}\n        \\frac{1-\\gamma L}{2\\gamma} \\sum_{k=1}^n \\| x_{k+1}-y_k \\|^2 \\le F(x_1) - F(x_{n+1}) \\le F(x_1)-\\inf F < +\\infty.\n    \\end{equation*}\n    Since $ \\dist (0, \\partial F(x_{k+1})) \\le \\frac{L\\gamma + 1}{\\gamma}\\|x_{k+1}-y_k\\|$ (see e.g. the proof of\\;\\cite[Prop.\\;13]{bolte2015error}), we have that \n    $\\dist (0, \\partial F(x_{k+1}))$\n    converges to $0$. The outer-semi continuity of the limiting subdifferential then yields criticality of accumulation points.}\n","sha256_utf8":"0ae6825e1c1abe44a46334dae6a7421469c8f1b72aabe2d93d5f51f24932b9fb"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"单位牛顿步传递二次收敛","source_units":[{"start":65427,"end":66130,"start_line":1095,"end_line":1098,"raw_tex":"    Now we apply the local convergence of Riemannian Newton \\cite[Th. 6.3.2]{absil2009optimization}:\n    assumption ii) ensures that\n    the Riemannian Newton direction $d$ computed in step\\;1 of\\;\\cref{alg:ManUpNewton} provides a quadratic improvement %\n    on a neighborhood of $x^\\star$ on\\;$\\M$. Moreover, the \\revise{line search} returns the unit-stepsize after some finite time: $\\alpha=1$ is tried first, and is acceptable for directions providing superlinear improvement by \\cref{lemma:LS_unit_stepsize}. Thus the whole Riemannian Newton update provides quadratic improvement after some finite time. Using this and assumption i), \\cref{th:identiflocalcv_sperlin} applies and yields the results.\n","sha256_utf8":"cae8c87780592a1422ab0be918933c5a8be1c820ff170e08cb0a37d4b746ed7c"}]},{"id":"S024","semantic_type":"lemma","kind":"lemma","title":"可接受小步长存在","source_units":[{"start":59124,"end":59559,"start_line":994,"end_line":996,"raw_tex":"\\begin{lemma}\\label{lemma:LS_exists_valid_stepsizes}\n    Let \\cref{asm:fun} hold and consider a manifold $\\M$ equipped with a retraction $\\R$ and a pair $(x, \\eta)\\in\\tangentBundle$. If $\\funtot$ is differentiable on $\\M$ at $x$, $\\langle \\grad \\funtot (x), \\eta\\rangle <0$, and $m_1<1$, then there exists $\\hat\\alpha>0$ such that any step size $\\alpha\\in(0, \\hat\\alpha)$ is acceptable by the Armijo rule \\eqref{eq:Armijo}.\n\\end{lemma}","sha256_utf8":"79565d6da0ca9cc53921cf70fd4b0d50439e66bf818d423f756b76e7829f8e84"}]},{"id":"S025","semantic_type":"proof","kind":"proof","title":"小步长存在性完整证明","source_units":[{"start":59561,"end":60265,"start_line":998,"end_line":1004,"raw_tex":"\\begin{proof}\n    We adapt a part of the proof of  \\cite[Th. 6.3.2]{dennis1996numerical} for the Armijo rule and the Riemannian setting. Since $m_1<1/2$, for any $\\alpha$ sufficiently small there holds\n    \\begin{equation*}\n        \\funtot\\circ \\R_x(\\alpha \\eta) \\le \\funtot\\circ\\R_x(0) + m_1 \\D\\left(\\funtot\\circ\\R_x \\right)(0)[\\alpha\\eta] = \\funtot(x) + m_1 \\alpha \\langle \\grad \\funtot(x), \\eta\\rangle.\n    \\end{equation*}\n    Since $\\funtot$ is bounded below, there exists a smallest $\\hat{\\alpha}$ such that $\\funtot(\\R_x(\\hat\\alpha \\eta)) = \\funtot(x) + m_1 \\hat\\alpha \\langle \\grad \\funtot(x), \\eta\\rangle$. Thus all stepsizes in $(0, \\hat\\alpha)$ are acceptable by\\;\\eqref{eq:Armijo}.\n\\end{proof}","sha256_utf8":"ae9779db8ea8c09b4f2150d350d1e9e5f1c18d2156eee32dbe51664a9c1519cf"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"一阶展开给出小步长下降","source_units":[{"start":59575,"end":59987,"start_line":999,"end_line":1002,"raw_tex":"    We adapt a part of the proof of  \\cite[Th. 6.3.2]{dennis1996numerical} for the Armijo rule and the Riemannian setting. Since $m_1<1/2$, for any $\\alpha$ sufficiently small there holds\n    \\begin{equation*}\n        \\funtot\\circ \\R_x(\\alpha \\eta) \\le \\funtot\\circ\\R_x(0) + m_1 \\D\\left(\\funtot\\circ\\R_x \\right)(0)[\\alpha\\eta] = \\funtot(x) + m_1 \\alpha \\langle \\grad \\funtot(x), \\eta\\rangle.\n    \\end{equation*}\n","sha256_utf8":"dbe4a5aeece21a04efca46763484cc867b7b94f6f9d8ee43aed9196280b5dd7e"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"目标下界确定可接受区间","source_units":[{"start":59987,"end":60254,"start_line":1003,"end_line":1003,"raw_tex":"    Since $\\funtot$ is bounded below, there exists a smallest $\\hat{\\alpha}$ such that $\\funtot(\\R_x(\\hat\\alpha \\eta)) = \\funtot(x) + m_1 \\hat\\alpha \\langle \\grad \\funtot(x), \\eta\\rangle$. Thus all stepsizes in $(0, \\hat\\alpha)$ are acceptable by\\;\\eqref{eq:Armijo}.\n","sha256_utf8":"f97b983d0962d334286c026ce20e4f29fbd7888f8e2aaa1049405a8478f15e59"}]},{"id":"S028","semantic_type":"lemma","kind":"lemma","title":"超线性方向接受单位步长","source_units":[{"start":60612,"end":61254,"start_line":1009,"end_line":1012,"raw_tex":"\\begin{lemma}\\label{lemma:LS_unit_stepsize}\n    Let \\cref{asm:fun} hold and consider a manifold $\\M$ equipped with a retraction $\\R$, a point $x^\\star\\in\\M$ and a pair $(x, \\eta)\\in\\tangentBundle$. Assume that $\\funtot$ is twice differentiable on $\\M$ near a strong local minimizer  $x^\\star$  on $\\M$, that is $\\Hess \\funman(x^\\star)$ is positive definite.\n    If the direction $\\eta$ brings a superlinear improvement towards $x^\\star$, that is $\\distM(\\R_x(\\eta), x^\\star) = o(\\distM(x, x^\\star))$ as $x\\to x^\\star$, and $0<m_1<1/2$, then $\\eta$ is acceptable by the Armijo rule\\;\\eqref{eq:Armijo} with unit stepsize $\\alpha=1$.\n\\end{lemma}","sha256_utf8":"04c61a7daf548b5b42c3261c6d8213accb5d56054cb0bccf82326f8711eac6ef"}]},{"id":"S029","semantic_type":"proof","kind":"proof","title":"单位步长引理的推论说明","source_units":[{"start":60268,"end":60612,"start_line":1007,"end_line":1008,"raw_tex":"In addition, a \\revise{line search} performed near a minimizer with a Newton direction\nshould accept the unit stepsize, so that a full step may be taken. This is the case when the Riemannian Hessian around this minimizer is positive definite as stated by the next lemma, \\revise{which is a direct corollary of \\cref{th:unitstepsuperlinearcv}.}\n","sha256_utf8":"b11b791cfc82a8fe7cc7a1244af98a95ebdd45052b072905996d4d96e1148675"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"由线搜索定理直接推出","source_units":[{"start":60268,"end":60612,"start_line":1007,"end_line":1008,"raw_tex":"In addition, a \\revise{line search} performed near a minimizer with a Newton direction\nshould accept the unit stepsize, so that a full step may be taken. This is the case when the Riemannian Hessian around this minimizer is positive definite as stated by the next lemma, \\revise{which is a direct corollary of \\cref{th:unitstepsuperlinearcv}.}\n","sha256_utf8":"b11b791cfc82a8fe7cc7a1244af98a95ebdd45052b072905996d4d96e1148675"}]},{"id":"S031","semantic_type":"theorem","kind":"theorem","title":"黎曼线搜索的单位步长保证","source_units":[{"start":96929,"end":97560,"start_line":1591,"end_line":1593,"raw_tex":"\\begin{theorem}[Soundness of the Riemannian \\revise{line search}]\\label{th:unitstepsuperlinearcv}\n    Consider a manifold $\\M$ equipped with a retraction $\\R$ and a twice differentiable function $\\funman:\\M\\to\\RR$ that admits a strong local minimizer $x^\\star$, that is, a point such that $\\Hess \\funman(x^\\star)$ is positive definite. If $x$ is close to $x^\\star$, $\\eta$ brings a superlinear improvement towards $x^\\star$, that is $\\distM(\\R_x(\\eta), x^\\star) = o(\\distM(x, x^\\star))$ as $x\\to x^\\star$, and $0<m_1<1/2$, then $\\eta$ is acceptable by the Armijo rule\\;\\eqref{eq:Armijo} with unit stepsize $\\alpha=1$.\n\\end{theorem}","sha256_utf8":"33149545c985ef5da6376801b552249e1b277828db4c33f7e7bf5cfd7cce2b1f"}]},{"id":"S032","semantic_type":"proof","kind":"proof","title":"线搜索保证完整证明","source_units":[{"start":97562,"end":105682,"start_line":1595,"end_line":1675,"raw_tex":"\\begin{proof}\n\n    Let $x, \\eta\\in\\tangentBundle$ denote a pair such that $x$ is close to $x^\\star$ and $\\distM(\\R_x(\\eta), x^\\star) = o(\\distM(x, x^\\star))$. For convenience, let $x_+ = \\R_x(\\eta)$ denote the next point.\n\n    Following \\cite{absil2009optimization} (see e.g. the proof of Th. 6.3.2), we work in local coordinates around $x^\\star$, representing any point $x\\in\\M$ by $\\widehat{x} = \\log_{x^\\star}(x)$ and any tangent vector $\\eta\\in\\tangentM$ by $\\widehat{\\eta}_x = \\D \\log_{x^\\star}(x)[\\eta]$. The function $\\funman$ is represented by $\\widehat{\\funman} = \\funman\\circ\\exp_{x^\\star}:\\tangentM[x^\\star]\\to\\bbR$. Defining the coordinates via the logarithm grants the useful property that the Riemannian distance of any two points $x, y\\in\\M$ matches the euclidean distance between their representatives: $\\distM(x, y) = \\|\\widehat{x}-\\widehat{y}\\|$. Besides, there holds\n    \\begin{equation}\\label{eq:gradHess_to_localcoords}\n        \\D \\funman(x)[\\eta]=\\D \\widehat{\\funman}(\\widehat{x})[\\widehat{\\eta}] \\quad \\text{ and } \\quad \\Hess \\funman(x)[\\eta, \\eta]=\\D \\widehat{\\funman}(\\widehat{x})[\\widehat{\\eta}, \\widehat{\\eta}].\n    \\end{equation}\n    Indeed, $\\D \\funman(x)[\\eta] = (\\funman\\circ\\geocurve)'(0)$ and $\\Hess \\funman(x)[\\eta, \\eta] = (\\funman\\circ\\geocurve)''(0)$, where $\\geocurve$ denotes the geodesic curve defined by $\\widehat{\\geocurve}(t) = \\widehat{x} + t\\widehat{\\eta}$. Using $\\funman\\circ\\geocurve=\\widehat{\\funman}\\circ\\widehat{\\geocurve}$, one obtains the result.\n\n    \\underline{Step 1.} We derive an approximation of $\\D \\funman(x)[\\eta] = \\langle \\grad \\funman(x), \\eta\\rangle$ in terms of $\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x}-\\widehat{x^\\star}]^2$. To do so, we go through the intermediate quantity $\\D \\widehat{\\funman}(\\widehat{x}) [\\widehat{x_+}-\\widehat{x}]$, and handle precisely the $o(\\cdot)$ terms. By smoothness of $\\widehat{\\funman}$ and since $\\D \\widehat{\\funman}(\\widehat{x^\\star})=0$, Taylor's formula for $\\D \\widehat{\\funman}$ writes\n    \\begin{align*}\n        \\D \\widehat{\\funman}(\\widehat{x}) [\\widehat{x_+}-\\widehat{x}] &= \\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x_+}-\\widehat{x}, \\widehat{x}-\\widehat{x^\\star}] + o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2)\\\\\n        &= -\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x}-\\widehat{x^\\star}]^2 + \\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x_+}-\\widehat{x^\\star}, \\widehat{x}-\\widehat{x^\\star}] + o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2)\\\\\n        &= -\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x}-\\widehat{x^\\star}]^2 + o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2),\n    \\end{align*}\n    where, in the last step, we used that $\\|\\widehat{x_+}-\\widehat{x^\\star}\\| = o(\\|\\widehat{x}-\\widehat{x^\\star}\\|)$ to get that $\\|\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x_+}-\\widehat{x^\\star}, \\widehat{x}-\\widehat{x^\\star}]\\| = \\|\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})\\| \\|\\widehat{x_+}-\\widehat{x^\\star}\\|\\|\\widehat{x}-\\widehat{x^\\star}\\| = o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2)$.\n    We now turn to show that $\\D \\widehat{\\funman}(\\widehat{x}) [\\widehat{x_+}-\\widehat{x}]$ behaves as $\\D \\funman(x)[\\eta]$ up to $o(\\|\\widehat{x_+}-\\widehat{x}\\|^2)$. Since $\\D \\funman(x)[\\eta]=\\D\\widehat{\\funman}(\\widehat{x})[\\widehat{\\eta}]$ by \\eqref{eq:gradHess_to_localcoords}, there holds:\n    \\begin{equation*}\n        \\|\\D \\funman(x)[\\eta] - \\D\\widehat{\\funman}(\\widehat{x})[\\widehat{x_+} - \\widehat{x}]\\| = \\|\\D\\widehat{\\funman}(\\widehat{x})[\\widehat{\\eta} - (\\widehat{x_+} - \\widehat{x})]\\| \\le \\|\\D\\widehat{\\funman}(\\widehat{x}) \\| \\|\\widehat{\\eta} - (\\widehat{x_+} - \\widehat{x})\\|.\n    \\end{equation*}\n\n    As $\\funman$ is twice differentiable and $\\exp$ is $\\C^\\infty$, $\\widehat{\\funman}$ is twice differentiable as well. In particular its derivative is locally Lipschitz continuous, so that for $\\widehat{x}$ near $\\widehat{x^\\star}$, we obtain a first estimate:\n    \\begin{equation*}\n        \\|\\D\\widehat{\\funman}(\\widehat{x})\\| = \\|\\D\\widehat{\\funman}(\\widehat{x}) - \\D\\widehat{\\funman}(\\widehat{x^\\star})\\| = \\mathcal O (\\| \\widehat{x}-\\widehat{x^\\star} \\|).\n    \\end{equation*}\n\n    Besides, the following estimate holds $\\|\\widehat{\\eta} - (\\widehat{x_+} - \\widehat{x})\\| = o (\\| \\widehat{x}-\\widehat{x^\\star} \\|)$. Indeed, as the function $\\log_{x^\\star}\\circ\\R_x : \\tangentM\\to\\tangentM[x^\\star]$ is differentiable, there holds for $\\eta\\in\\tangentM$ small,\n    \\begin{equation*}\n        \\log_{x^\\star}(\\R_x(\\eta)) = \\log_{x^\\star}(\\R_x(0)) + \\D\\log_{x^\\star}(\\R_x(0))[\\D\\R_x(0)[\\eta]] + o(\\|\\eta\\|),\n    \\end{equation*}\n    which simplifies to $\\widehat{x_+} = \\widehat{x} + \\widehat{\\eta} + o(\\|\\eta\\|)$. \\Cref{lem:InvRetractionApproxLog} allows to write $\\|\\eta\\| = \\|\\R_x^{-1}(x_+)\\| = \\mathcal O(\\distM(x, x_+))$. Using the triangular inequality and the assumption that $\\distM(x_+, x^\\star) = o(\\distM(x, x^*))$ we get\n    \\begin{equation*}\n        \\distM(x, x_+) \\le \\distM(x, x^\\star) + \\distM(x^\\star, x_+) = \\mathcal O(\\distM(x, x^*)) = \\mathcal O(\\|\\widehat{x}-\\widehat{x^\\star}\\|),\n    \\end{equation*}\n    so that the second estimate holds.\n\n    Combining the two above estimates allows to conclude that\n    \\begin{equation*}\n        \\|\\D \\funman(x)[\\eta] - \\D\\widehat{\\funman}(\\widehat{x})[\\widehat{x_+} - \\widehat{x}]\\| = o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2),\n    \\end{equation*}\n    so that overall,\n    \\begin{equation*}\n        \\D \\funman(x)[\\eta] = \\D \\widehat{\\funman}(\\widehat{x}) [\\widehat{x_+}-\\widehat{x}] + o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2) = -\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x}-\\widehat{x^\\star}]^2 + o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2).\n    \\end{equation*}\n\n\n\n    Using that $\\|\\widehat{x}-\\widehat{x^\\star}\\| = \\distM(x, x^\\star)$ and $\\D^2 \\widehat{\\funman}(\\widehat{x^\\star}) = \\Hess \\funman(x^\\star)$ \\eqref{eq:gradHess_to_localcoords}, we obtain\n    \\begin{equation}\\label{eq:proofunitstepsize_gradhess}\n        \\D \\funman(x)[\\eta] = -\\Hess \\funman(x^\\star)[\\widehat{x}-\\widehat{x^\\star}]^2 + o(\\distM(x, x^\\star)^2).\n    \\end{equation}\n\n    \\underline{Step 2.} The function $\\funman$ admits a second-order development around $x^\\star$: applying \\cref{eq:manifoldSecondOrderDevSecRetraction} with the exponential map $\\exp_{x^\\star}$ as a second-order retraction yields\n    \\begin{equation}\\label{eq:proofunitstepsize_fdev}\n        \\funman(x) = \\funman(x^\\star) + \\frac{1}{2} \\Hess \\funman(x^\\star)[\\widehat{x}-\\widehat{x^\\star}]^2 + o(\\distM(x, x^\\star)^2),\n    \\end{equation}\n    where we used that $\\distM(x, x^\\star) = \\|\\log_{x^\\star}(x)-\\log_{x^\\star}(x^\\star)\\|$. Denote $0 < l \\le L$ the lower and upper eigenvalues of $\\Hess \\funman(x^\\star)$. The combination $\\eqref{eq:proofunitstepsize_fdev} + m_1 \\eqref{eq:proofunitstepsize_gradhess}$ writes\n    \\begin{align*}\n        \\funman(x) + m_1 \\D \\funman(x)[\\eta] &= \\funman(x^\\star) + (\\frac{1}{2}-m_1) \\Hess \\funman(x^\\star)[\\widehat{x}-\\widehat{x^\\star}]^2 + o(\\distM(x, x^\\star)^2) \\\\\n        &\\ge \\funman(x^\\star) + (\\frac{1}{2}-m_1) l \\distM(x, x^\\star)^2 + o(\\distM(x, x^\\star)^2),\n    \\end{align*}\n\n    Let $\\varepsilon >0$ such that $\\frac{1}{2}L \\varepsilon^2  < (\\frac{1}{2}-m_1) l$. As $\\distM(x_+, x^\\star)=o(\\distM(x, x^\\star))$, for $x$ close enough to $x^\\star$ there holds $\\distM(x_+, x^\\star) \\le \\varepsilon \\distM(x, x^\\star)$. Combining this with the second-order development of $f$ at $x_+$, there holds:\n    \\begin{align*}\n        \\funman(x_+) &= \\funman(x^\\star) + \\frac{1}{2} \\Hess \\funman(x^\\star)[\\widehat{x_+}-\\widehat{x^\\star}]^2 + o(\\distM(x_+, x^\\star)^2) \\\\\n        &\\le \\funman(x^\\star) + \\frac{1}{2}L \\distM(x_+, x^\\star)^2 + o(\\distM(x_+, x^\\star)^2)\\\\\n        &\\le \\funman(x^\\star) + \\frac{1}{2}L \\varepsilon^2 \\distM(x, x^\\star)^2 + o(\\distM(x, x^\\star)^2).\n    \\end{align*}\n\n    Subtracting the two estimates yields\n    \\begin{equation*}\n        \\funman(x_+) - (\\funman(x) + m_1 \\D \\funman(x)[\\eta]) \\le \\left(\\frac{1}{2}L \\varepsilon^2 - (\\frac{1}{2}-m_1) l \\right) \\distM(x, x^\\star)^2 + o(\\distM(x, x^\\star)^2),\n    \\end{equation*}\n    which ensures that the Armijo condition is satisfied.\n\n\n\n\n\n\\end{proof}","sha256_utf8":"03fbdd0077dcc1116c80fd235f1bfe340748428e57ba066b1faf4b7a505f8d7f"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"引入更新点与局部坐标","source_units":[{"start":97577,"end":99063,"start_line":1597,"end_line":1603,"raw_tex":"    Let $x, \\eta\\in\\tangentBundle$ denote a pair such that $x$ is close to $x^\\star$ and $\\distM(\\R_x(\\eta), x^\\star) = o(\\distM(x, x^\\star))$. For convenience, let $x_+ = \\R_x(\\eta)$ denote the next point.\n\n    Following \\cite{absil2009optimization} (see e.g. the proof of Th. 6.3.2), we work in local coordinates around $x^\\star$, representing any point $x\\in\\M$ by $\\widehat{x} = \\log_{x^\\star}(x)$ and any tangent vector $\\eta\\in\\tangentM$ by $\\widehat{\\eta}_x = \\D \\log_{x^\\star}(x)[\\eta]$. The function $\\funman$ is represented by $\\widehat{\\funman} = \\funman\\circ\\exp_{x^\\star}:\\tangentM[x^\\star]\\to\\bbR$. Defining the coordinates via the logarithm grants the useful property that the Riemannian distance of any two points $x, y\\in\\M$ matches the euclidean distance between their representatives: $\\distM(x, y) = \\|\\widehat{x}-\\widehat{y}\\|$. Besides, there holds\n    \\begin{equation}\\label{eq:gradHess_to_localcoords}\n        \\D \\funman(x)[\\eta]=\\D \\widehat{\\funman}(\\widehat{x})[\\widehat{\\eta}] \\quad \\text{ and } \\quad \\Hess \\funman(x)[\\eta, \\eta]=\\D \\widehat{\\funman}(\\widehat{x})[\\widehat{\\eta}, \\widehat{\\eta}].\n    \\end{equation}\n    Indeed, $\\D \\funman(x)[\\eta] = (\\funman\\circ\\geocurve)'(0)$ and $\\Hess \\funman(x)[\\eta, \\eta] = (\\funman\\circ\\geocurve)''(0)$, where $\\geocurve$ denotes the geodesic curve defined by $\\widehat{\\geocurve}(t) = \\widehat{x} + t\\widehat{\\eta}$. Using $\\funman\\circ\\geocurve=\\widehat{\\funman}\\circ\\widehat{\\geocurve}$, one obtains the result.\n","sha256_utf8":"5d61d64d2a7fff4672553d23e13d8370785c8d21792b9076ebf882b292da1bee"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"位移方向导数的二次主项","source_units":[{"start":99064,"end":100592,"start_line":1605,"end_line":1611,"raw_tex":"    \\underline{Step 1.} We derive an approximation of $\\D \\funman(x)[\\eta] = \\langle \\grad \\funman(x), \\eta\\rangle$ in terms of $\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x}-\\widehat{x^\\star}]^2$. To do so, we go through the intermediate quantity $\\D \\widehat{\\funman}(\\widehat{x}) [\\widehat{x_+}-\\widehat{x}]$, and handle precisely the $o(\\cdot)$ terms. By smoothness of $\\widehat{\\funman}$ and since $\\D \\widehat{\\funman}(\\widehat{x^\\star})=0$, Taylor's formula for $\\D \\widehat{\\funman}$ writes\n    \\begin{align*}\n        \\D \\widehat{\\funman}(\\widehat{x}) [\\widehat{x_+}-\\widehat{x}] &= \\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x_+}-\\widehat{x}, \\widehat{x}-\\widehat{x^\\star}] + o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2)\\\\\n        &= -\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x}-\\widehat{x^\\star}]^2 + \\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x_+}-\\widehat{x^\\star}, \\widehat{x}-\\widehat{x^\\star}] + o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2)\\\\\n        &= -\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x}-\\widehat{x^\\star}]^2 + o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2),\n    \\end{align*}\n    where, in the last step, we used that $\\|\\widehat{x_+}-\\widehat{x^\\star}\\| = o(\\|\\widehat{x}-\\widehat{x^\\star}\\|)$ to get that $\\|\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x_+}-\\widehat{x^\\star}, \\widehat{x}-\\widehat{x^\\star}]\\| = \\|\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})\\| \\|\\widehat{x_+}-\\widehat{x^\\star}\\|\\|\\widehat{x}-\\widehat{x^\\star}\\| = o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2)$.\n","sha256_utf8":"e8cce5ca64b15fe879979f49fdee0b9989295146f4696810527da5222e64f392"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"真实方向与坐标位移比较","source_units":[{"start":100592,"end":101210,"start_line":1612,"end_line":1615,"raw_tex":"    We now turn to show that $\\D \\widehat{\\funman}(\\widehat{x}) [\\widehat{x_+}-\\widehat{x}]$ behaves as $\\D \\funman(x)[\\eta]$ up to $o(\\|\\widehat{x_+}-\\widehat{x}\\|^2)$. Since $\\D \\funman(x)[\\eta]=\\D\\widehat{\\funman}(\\widehat{x})[\\widehat{\\eta}]$ by \\eqref{eq:gradHess_to_localcoords}, there holds:\n    \\begin{equation*}\n        \\|\\D \\funman(x)[\\eta] - \\D\\widehat{\\funman}(\\widehat{x})[\\widehat{x_+} - \\widehat{x}]\\| = \\|\\D\\widehat{\\funman}(\\widehat{x})[\\widehat{\\eta} - (\\widehat{x_+} - \\widehat{x})]\\| \\le \\|\\D\\widehat{\\funman}(\\widehat{x}) \\| \\|\\widehat{\\eta} - (\\widehat{x_+} - \\widehat{x})\\|.\n    \\end{equation*}\n","sha256_utf8":"b3bfd6d77f1a15c8eb53e35cd5cbf6ae0fca38d49edc9831b54b15274065a8aa"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"局部导数范数估计","source_units":[{"start":101211,"end":101693,"start_line":1617,"end_line":1620,"raw_tex":"    As $\\funman$ is twice differentiable and $\\exp$ is $\\C^\\infty$, $\\widehat{\\funman}$ is twice differentiable as well. In particular its derivative is locally Lipschitz continuous, so that for $\\widehat{x}$ near $\\widehat{x^\\star}$, we obtain a first estimate:\n    \\begin{equation*}\n        \\|\\D\\widehat{\\funman}(\\widehat{x})\\| = \\|\\D\\widehat{\\funman}(\\widehat{x}) - \\D\\widehat{\\funman}(\\widehat{x^\\star})\\| = \\mathcal O (\\| \\widehat{x}-\\widehat{x^\\star} \\|).\n    \\end{equation*}\n","sha256_utf8":"13fd7e0d54236c7a94af788d1d37033df10eec4b386a487ded41cf6935fb4dc2"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"回缩误差与步向量估计","source_units":[{"start":101694,"end":102671,"start_line":1622,"end_line":1630,"raw_tex":"    Besides, the following estimate holds $\\|\\widehat{\\eta} - (\\widehat{x_+} - \\widehat{x})\\| = o (\\| \\widehat{x}-\\widehat{x^\\star} \\|)$. Indeed, as the function $\\log_{x^\\star}\\circ\\R_x : \\tangentM\\to\\tangentM[x^\\star]$ is differentiable, there holds for $\\eta\\in\\tangentM$ small,\n    \\begin{equation*}\n        \\log_{x^\\star}(\\R_x(\\eta)) = \\log_{x^\\star}(\\R_x(0)) + \\D\\log_{x^\\star}(\\R_x(0))[\\D\\R_x(0)[\\eta]] + o(\\|\\eta\\|),\n    \\end{equation*}\n    which simplifies to $\\widehat{x_+} = \\widehat{x} + \\widehat{\\eta} + o(\\|\\eta\\|)$. \\Cref{lem:InvRetractionApproxLog} allows to write $\\|\\eta\\| = \\|\\R_x^{-1}(x_+)\\| = \\mathcal O(\\distM(x, x_+))$. Using the triangular inequality and the assumption that $\\distM(x_+, x^\\star) = o(\\distM(x, x^*))$ we get\n    \\begin{equation*}\n        \\distM(x, x_+) \\le \\distM(x, x^\\star) + \\distM(x^\\star, x_+) = \\mathcal O(\\distM(x, x^*)) = \\mathcal O(\\|\\widehat{x}-\\widehat{x^\\star}\\|),\n    \\end{equation*}\n    so that the second estimate holds.\n","sha256_utf8":"775b43a5c3453fda752fe1e12fd21975ce6e283b514f183ed7ceb620559ec6f6"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"得到方向导数的海森展开","source_units":[{"start":102672,"end":103615,"start_line":1632,"end_line":1646,"raw_tex":"    Combining the two above estimates allows to conclude that\n    \\begin{equation*}\n        \\|\\D \\funman(x)[\\eta] - \\D\\widehat{\\funman}(\\widehat{x})[\\widehat{x_+} - \\widehat{x}]\\| = o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2),\n    \\end{equation*}\n    so that overall,\n    \\begin{equation*}\n        \\D \\funman(x)[\\eta] = \\D \\widehat{\\funman}(\\widehat{x}) [\\widehat{x_+}-\\widehat{x}] + o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2) = -\\D^2 \\widehat{\\funman}(\\widehat{x^\\star})[\\widehat{x}-\\widehat{x^\\star}]^2 + o(\\|\\widehat{x}-\\widehat{x^\\star}\\|^2).\n    \\end{equation*}\n\n\n\n    Using that $\\|\\widehat{x}-\\widehat{x^\\star}\\| = \\distM(x, x^\\star)$ and $\\D^2 \\widehat{\\funman}(\\widehat{x^\\star}) = \\Hess \\funman(x^\\star)$ \\eqref{eq:gradHess_to_localcoords}, we obtain\n    \\begin{equation}\\label{eq:proofunitstepsize_gradhess}\n        \\D \\funman(x)[\\eta] = -\\Hess \\funman(x^\\star)[\\widehat{x}-\\widehat{x^\\star}]^2 + o(\\distM(x, x^\\star)^2).\n    \\end{equation}\n","sha256_utf8":"bc8a7241293bf08fc0060eba07a505f6dd3470abd5671f6f0e012958b0b67fa1"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"阿米霍右端的正二次余量","source_units":[{"start":103616,"end":104640,"start_line":1648,"end_line":1656,"raw_tex":"    \\underline{Step 2.} The function $\\funman$ admits a second-order development around $x^\\star$: applying \\cref{eq:manifoldSecondOrderDevSecRetraction} with the exponential map $\\exp_{x^\\star}$ as a second-order retraction yields\n    \\begin{equation}\\label{eq:proofunitstepsize_fdev}\n        \\funman(x) = \\funman(x^\\star) + \\frac{1}{2} \\Hess \\funman(x^\\star)[\\widehat{x}-\\widehat{x^\\star}]^2 + o(\\distM(x, x^\\star)^2),\n    \\end{equation}\n    where we used that $\\distM(x, x^\\star) = \\|\\log_{x^\\star}(x)-\\log_{x^\\star}(x^\\star)\\|$. Denote $0 < l \\le L$ the lower and upper eigenvalues of $\\Hess \\funman(x^\\star)$. The combination $\\eqref{eq:proofunitstepsize_fdev} + m_1 \\eqref{eq:proofunitstepsize_gradhess}$ writes\n    \\begin{align*}\n        \\funman(x) + m_1 \\D \\funman(x)[\\eta] &= \\funman(x^\\star) + (\\frac{1}{2}-m_1) \\Hess \\funman(x^\\star)[\\widehat{x}-\\widehat{x^\\star}]^2 + o(\\distM(x, x^\\star)^2) \\\\\n        &\\ge \\funman(x^\\star) + (\\frac{1}{2}-m_1) l \\distM(x, x^\\star)^2 + o(\\distM(x, x^\\star)^2),\n    \\end{align*}\n","sha256_utf8":"e6447032fee33e7009848be256e7a0efb2089ca09dd421d882db009bb60b44ff"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"超线性更新点的函数值上界","source_units":[{"start":104641,"end":105347,"start_line":1658,"end_line":1663,"raw_tex":"    Let $\\varepsilon >0$ such that $\\frac{1}{2}L \\varepsilon^2  < (\\frac{1}{2}-m_1) l$. As $\\distM(x_+, x^\\star)=o(\\distM(x, x^\\star))$, for $x$ close enough to $x^\\star$ there holds $\\distM(x_+, x^\\star) \\le \\varepsilon \\distM(x, x^\\star)$. Combining this with the second-order development of $f$ at $x_+$, there holds:\n    \\begin{align*}\n        \\funman(x_+) &= \\funman(x^\\star) + \\frac{1}{2} \\Hess \\funman(x^\\star)[\\widehat{x_+}-\\widehat{x^\\star}]^2 + o(\\distM(x_+, x^\\star)^2) \\\\\n        &\\le \\funman(x^\\star) + \\frac{1}{2}L \\distM(x_+, x^\\star)^2 + o(\\distM(x_+, x^\\star)^2)\\\\\n        &\\le \\funman(x^\\star) + \\frac{1}{2}L \\varepsilon^2 \\distM(x, x^\\star)^2 + o(\\distM(x, x^\\star)^2).\n    \\end{align*}\n","sha256_utf8":"5fcd919db0133e6efeb89fae859357deb174a9ee4f89bf24677039cc3a791e52"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"比较余量得到单位步长接受","source_units":[{"start":105348,"end":105666,"start_line":1665,"end_line":1669,"raw_tex":"    Subtracting the two estimates yields\n    \\begin{equation*}\n        \\funman(x_+) - (\\funman(x) + m_1 \\D \\funman(x)[\\eta]) \\le \\left(\\frac{1}{2}L \\varepsilon^2 - (\\frac{1}{2}-m_1) l \\right) \\distM(x, x^\\star)^2 + o(\\distM(x, x^\\star)^2),\n    \\end{equation*}\n    which ensures that the Armijo condition is satisfied.\n","sha256_utf8":"f8689aafc28a9fa80f6d006c48c22b098995b253853183f291a40ccd6e1e4f57"}]},{"id":"S042","semantic_type":"theorem","kind":"theorem","title":"交替算法继承超线性收敛阶","source_units":[{"start":51892,"end":52884,"start_line":904,"end_line":920,"raw_tex":"\\begin{theorem}%\n    \\label{th:identiflocalcv_sperlin}\n    Let \\cref{asm:fun} hold and take $\\gamma\\in(0, 1/L)$\\revise{, where $L$ is the Lipschitz constant for $\\nabla f$}. Assume that \\cref{alg:generic} generates a sequence $(y_k)$ which admits at least one limit point $\\bar{x}$ such that:\n    \\begin{itemize}\n        \\item[i)] $\\bar{x}\\in\\M$ is a $r$-structured critical point for $(f,g)$ with $r<1/\\gamma$;\n        \\item[ii)] $\\manupdate_{\\M}$ has superlinear convergence rate of order $1+\\theta\\in(1,2)$ %\n        near $\\bar{x}$ in $\\M$.\n    \\end{itemize}\n    Then, after some finite time:\n    \\begin{itemize}\n        \\item[a)] the full sequence $(x_k)$ lies on $\\M$;\n        \\item[b)] $x_k$ converges to $\\bar{x}$ superlinearly with the same order as $\\manupdate$:\n              \\begin{equation}\\label{eq:super}\n                  \\distM(x_{k+1},\\bar{x}) \\le c ~ \\distM(x_k, \\bar{x})^{1+\\theta} \\qquad \\text{ for some $c>0$.}\n              \\end{equation}\n    \\end{itemize}\n\\end{theorem}","sha256_utf8":"185dec797db7e909bae9e82dc922d24c957ed90d202f35c89d8a9dfda89dba79"}]},{"id":"S043","semantic_type":"proof","kind":"proof","title":"收敛阶传递完整证明","source_units":[{"start":52888,"end":56637,"start_line":924,"end_line":964,"raw_tex":"\\begin{proof}\n    Let us note $ \\PGop(y) = \\prox_{\\gamma g}(y - \\gamma \\nabla f(y)) $ for $y\\in\\mathbb{R}^n$.\n    The part i) of the assumptions enables us to show the existence of some neighborhood of $\\bar{x}$ on which the proximal gradient operation is $\\M$-valued and Lipschitz continuous. More precisely, \n    \\cref{th:PGpointlocatemanifolds} \n    implies that there exists $\\delta_1>0$ and $C>0$ such that, \n    \\[%\n      \\PGop(y)\\in\\M \\label{proof:inM} \\quad \\text{ and } \\quad \\|\\PGop(y)- \\PGop(\\bar{x})\\| \\le C \\|y-\\bar{x}\\| \\qquad \\text{for all $y$ in $\\B(\\bar{x}, \\delta_1)$}.\n    \\]%\n    Now, if $y$ belongs to $\\M$, we get that there exists $\\varepsilon_1>0$ such that for any $y$ in $\\B_{\\M}(\\bar{x}, \\varepsilon_1)$, $\\PGop(y)\\in\\M$; but in addition, the Euclidean Lipschitz continuity can be translated into a Riemannian one (see \\cref{lemma:Euclidean_to_Riemannian}) since   for some $\\delta>0$,\n    \\begin{multline}\\label{eq:EuctoRie}\n        (1-\\delta)\\distM(\\PGop(y), \\bar{x}) = (1-\\delta)\\distM(\\PGop(y), \\PGop(\\bar{x}))  \\le \\|\\PGop(y)-\\PGop(\\bar{x})\\| \\\\\n        \\le C \\|y-\\bar{x}\\| \\le C (1+\\delta)\\distM(y, \\bar{x})\n    \\end{multline}\n    Hence, there is $q_1>0$ such that for any $y$ in $\\B_{\\M}(\\bar{x}, \\varepsilon_1)$\n    \\begin{equation}\\label{proof:PGopEstimate}\n        \\distM(\\PGop(y), \\bar{x}) =  \\distM(\\PGop(y), \\PGop(\\bar{x}))   \\le  q_1 ~ \\distM(y, \\bar{x}).\n    \\end{equation}\n\n    Then, the part ii) of the assumptions gives us the existence of $\\varepsilon_2$, $q_2>0$ and $\\theta\\in(0,1)$ such that, for any $x$ in $\\B_{\\M}(\\bar{x}, \\varepsilon_2)$,\n    \\begin{equation}\\label{proof:manupdateEstimate}\n        \\distM(\\manupdate_{\\M}(x), \\bar{x}) \\le q_2 ~ \\distM(x, \\bar{x})^{1+\\theta}.\n    \\end{equation}\n    Let us now take any $x\\in\\B_{\\M}(\\bar{x}, \\varepsilon)$ where $\\varepsilon=\\min(\\varepsilon_1,\\varepsilon_2,(\\varepsilon_1 / q_2)^{\\frac{1}{1+\\theta}},(q_2 q_1)^{-\\frac{1}{\\theta}})$: \\\\\n    \\noindent (\\emph{i}) Since $x\\in\\B_{\\M}(\\bar{x}, \\varepsilon_2)$, the manifold update \\eqref{proof:manupdateEstimate} yields\n    \\begin{equation*}\n        \\distM(\\manupdate_{\\M}(x), \\bar{x}) \\le q_2 ~ \\distM(x, \\bar{x})^{1+\\theta} \\le q_2 ~ \\varepsilon^{1+\\theta} \\le \\varepsilon_1.\n    \\end{equation*}\n    \\noindent (\\emph{ii}) As $\\manupdate_{\\M}(x)$ lies in $\\B_{\\M}(\\bar{x}, \\varepsilon_1)$, the proximal gradient update \\eqref{proof:PGopEstimate} applied to $y = \\manupdate_{\\M}(x)$ gives\n    \\begin{multline}\\label{eq:superlinear}\n        \\distM(\\PGop(\\manupdate_{\\M}(x)), \\bar{x}) \\le    q_1 \\distM(\\manupdate_{\\M}(x), \\bar{x}) \\\\ \\le q_1 q_2 ~ \\distM(x, \\bar{x})^{1+\\theta} \\le q_1 q_2 ~ \\varepsilon^\\theta \\distM(x, \\bar{x}).\n    \\end{multline}\n    Since $q_2 q_1 \\varepsilon^\\theta \\le 1$  by construction, this gives %\n    \\begin{equation}\\label{eq:manstable}\n        \\distM(\\PGop(\\manupdate_{\\M}(x)), \\bar{x}) \\le \\distM(x, \\bar{x}) \\qquad \\text{for any $x\\in\\B_{\\M}(\\bar{x}, \\varepsilon)$}.\n    \\end{equation}\n    We have thus proved the existence of a neighborhood $\\B_\\M(\\bar{x}, \\varepsilon)$ of $\\bar{x}$ in $\\M$ which is stable for an iteration of \\cref{alg:generic} and over which one iteration has a superlinear improvement of order $1+\\theta$ (by \\eqref{eq:superlinear}).\n\n\n    Finally, since $\\bar{x}$ is a limit point of $(y_k)$, there exists $K<\\infty$ such that $y_K\\in\\B(\\bar{x},(1-\\delta)\\varepsilon/C)$. Besides, \\eqref{eq:EuctoRie} tells us that $ \\distM(\\PGop(y_K), \\bar{x})\\le \\varepsilon$ and thus $x_k$ and $y_k$ belong to $\\B_{\\M}(\\bar{x}, \\varepsilon)$ for all $k>K$ by \\eqref{eq:manstable}. %\n    We conclude that $x_{k+1}=\\PGop(y_k)\\in\\M$ for all $k\\geq K$, and, using \\eqref{eq:superlinear}, that we have \\eqref{eq:super} with $c= q_1 q_2$, for all $k>K$. \n\\end{proof}","sha256_utf8":"cc51dcdfba46de7deb8816f183f1473b6c7fe8af32cdb6cd1de04104c298d46d"}]},{"id":"S044","semantic_type":"assumption","kind":"assumption","title":"聚点结构与流形更新收敛阶","source_units":[{"start":51947,"end":52454,"start_line":906,"end_line":911,"raw_tex":"    Let \\cref{asm:fun} hold and take $\\gamma\\in(0, 1/L)$\\revise{, where $L$ is the Lipschitz constant for $\\nabla f$}. Assume that \\cref{alg:generic} generates a sequence $(y_k)$ which admits at least one limit point $\\bar{x}$ such that:\n    \\begin{itemize}\n        \\item[i)] $\\bar{x}\\in\\M$ is a $r$-structured critical point for $(f,g)$ with $r<1/\\gamma$;\n        \\item[ii)] $\\manupdate_{\\M}$ has superlinear convergence rate of order $1+\\theta\\in(1,2)$ %\n        near $\\bar{x}$ in $\\M$.\n    \\end{itemize}\n","sha256_utf8":"4eff563184b84ea61ff5e8f597f800e9b2d940f6bfd752778b7413183c92a0be"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"近端梯度局部定位与利普希茨性","source_units":[{"start":52902,"end":53484,"start_line":925,"end_line":931,"raw_tex":"    Let us note $ \\PGop(y) = \\prox_{\\gamma g}(y - \\gamma \\nabla f(y)) $ for $y\\in\\mathbb{R}^n$.\n    The part i) of the assumptions enables us to show the existence of some neighborhood of $\\bar{x}$ on which the proximal gradient operation is $\\M$-valued and Lipschitz continuous. More precisely, \n    \\cref{th:PGpointlocatemanifolds} \n    implies that there exists $\\delta_1>0$ and $C>0$ such that, \n    \\[%\n      \\PGop(y)\\in\\M \\label{proof:inM} \\quad \\text{ and } \\quad \\|\\PGop(y)- \\PGop(\\bar{x})\\| \\le C \\|y-\\bar{x}\\| \\qquad \\text{for all $y$ in $\\B(\\bar{x}, \\delta_1)$}.\n    \\]%\n","sha256_utf8":"c999099310ead28b5370b7c82be833c7a0e52c7071b480be2e4c645d0113dc05"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"转换为黎曼距离估计","source_units":[{"start":53484,"end":54304,"start_line":932,"end_line":940,"raw_tex":"    Now, if $y$ belongs to $\\M$, we get that there exists $\\varepsilon_1>0$ such that for any $y$ in $\\B_{\\M}(\\bar{x}, \\varepsilon_1)$, $\\PGop(y)\\in\\M$; but in addition, the Euclidean Lipschitz continuity can be translated into a Riemannian one (see \\cref{lemma:Euclidean_to_Riemannian}) since   for some $\\delta>0$,\n    \\begin{multline}\\label{eq:EuctoRie}\n        (1-\\delta)\\distM(\\PGop(y), \\bar{x}) = (1-\\delta)\\distM(\\PGop(y), \\PGop(\\bar{x}))  \\le \\|\\PGop(y)-\\PGop(\\bar{x})\\| \\\\\n        \\le C \\|y-\\bar{x}\\| \\le C (1+\\delta)\\distM(y, \\bar{x})\n    \\end{multline}\n    Hence, there is $q_1>0$ such that for any $y$ in $\\B_{\\M}(\\bar{x}, \\varepsilon_1)$\n    \\begin{equation}\\label{proof:PGopEstimate}\n        \\distM(\\PGop(y), \\bar{x}) =  \\distM(\\PGop(y), \\PGop(\\bar{x}))   \\le  q_1 ~ \\distM(y, \\bar{x}).\n    \\end{equation}\n","sha256_utf8":"30818d6d6cbf139e7c9be18ae055aaf3d3181112350663855f61916f11038dd7"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"流形更新的超线性估计","source_units":[{"start":54305,"end":54636,"start_line":942,"end_line":945,"raw_tex":"    Then, the part ii) of the assumptions gives us the existence of $\\varepsilon_2$, $q_2>0$ and $\\theta\\in(0,1)$ such that, for any $x$ in $\\B_{\\M}(\\bar{x}, \\varepsilon_2)$,\n    \\begin{equation}\\label{proof:manupdateEstimate}\n        \\distM(\\manupdate_{\\M}(x), \\bar{x}) \\le q_2 ~ \\distM(x, \\bar{x})^{1+\\theta}.\n    \\end{equation}\n","sha256_utf8":"eb49a7e3ee763f1c15eaa32f92061c9d1049db8870bec4a197fd656ce387bd6e"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"选择邻域控制流形更新","source_units":[{"start":54636,"end":55134,"start_line":946,"end_line":950,"raw_tex":"    Let us now take any $x\\in\\B_{\\M}(\\bar{x}, \\varepsilon)$ where $\\varepsilon=\\min(\\varepsilon_1,\\varepsilon_2,(\\varepsilon_1 / q_2)^{\\frac{1}{1+\\theta}},(q_2 q_1)^{-\\frac{1}{\\theta}})$: \\\\\n    \\noindent (\\emph{i}) Since $x\\in\\B_{\\M}(\\bar{x}, \\varepsilon_2)$, the manifold update \\eqref{proof:manupdateEstimate} yields\n    \\begin{equation*}\n        \\distM(\\manupdate_{\\M}(x), \\bar{x}) \\le q_2 ~ \\distM(x, \\bar{x})^{1+\\theta} \\le q_2 ~ \\varepsilon^{1+\\theta} \\le \\varepsilon_1.\n    \\end{equation*}\n","sha256_utf8":"ef41e10a8185797955d71b6fe89a9a7b6e9ab67e8350c511813faf6616b29c4e"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"复合迭代的超线性与稳定性","source_units":[{"start":55134,"end":56124,"start_line":951,"end_line":959,"raw_tex":"    \\noindent (\\emph{ii}) As $\\manupdate_{\\M}(x)$ lies in $\\B_{\\M}(\\bar{x}, \\varepsilon_1)$, the proximal gradient update \\eqref{proof:PGopEstimate} applied to $y = \\manupdate_{\\M}(x)$ gives\n    \\begin{multline}\\label{eq:superlinear}\n        \\distM(\\PGop(\\manupdate_{\\M}(x)), \\bar{x}) \\le    q_1 \\distM(\\manupdate_{\\M}(x), \\bar{x}) \\\\ \\le q_1 q_2 ~ \\distM(x, \\bar{x})^{1+\\theta} \\le q_1 q_2 ~ \\varepsilon^\\theta \\distM(x, \\bar{x}).\n    \\end{multline}\n    Since $q_2 q_1 \\varepsilon^\\theta \\le 1$  by construction, this gives %\n    \\begin{equation}\\label{eq:manstable}\n        \\distM(\\PGop(\\manupdate_{\\M}(x)), \\bar{x}) \\le \\distM(x, \\bar{x}) \\qquad \\text{for any $x\\in\\B_{\\M}(\\bar{x}, \\varepsilon)$}.\n    \\end{equation}\n    We have thus proved the existence of a neighborhood $\\B_\\M(\\bar{x}, \\varepsilon)$ of $\\bar{x}$ in $\\M$ which is stable for an iteration of \\cref{alg:generic} and over which one iteration has a superlinear improvement of order $1+\\theta$ (by \\eqref{eq:superlinear}).\n","sha256_utf8":"c8e64a245dce071defef414d46782aa979db4e9338df74a8c48ac5a378142685"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"聚点触发有限识别与全序列收敛","source_units":[{"start":56126,"end":56626,"start_line":962,"end_line":963,"raw_tex":"    Finally, since $\\bar{x}$ is a limit point of $(y_k)$, there exists $K<\\infty$ such that $y_K\\in\\B(\\bar{x},(1-\\delta)\\varepsilon/C)$. Besides, \\eqref{eq:EuctoRie} tells us that $ \\distM(\\PGop(y_K), \\bar{x})\\le \\varepsilon$ and thus $x_k$ and $y_k$ belong to $\\B_{\\M}(\\bar{x}, \\varepsilon)$ for all $k>K$ by \\eqref{eq:manstable}. %\n    We conclude that $x_{k+1}=\\PGop(y_k)\\in\\M$ for all $k\\geq K$, and, using \\eqref{eq:superlinear}, that we have \\eqref{eq:super} with $c= q_1 q_2$, for all $k>K$. \n","sha256_utf8":"7a694b128874b8389094dc8bc72fb1dfc5654a9b8e1c08536fa010b884cff7cc"}]},{"id":"S051","semantic_type":"theorem","kind":"theorem","title":"近端梯度光滑定位流形","source_units":[{"start":41893,"end":42833,"start_line":770,"end_line":779,"raw_tex":"\\begin{theorem}[Proximal gradient points smoothly locate manifolds] \\label{th:PGpointlocatemanifolds}\n    Let $f$ be a $\\C^2$ function on $\\RR^n$ and $g$ a lower semi-continuous function on $\\RR^n$. Suppose that $g$ is both $r$-prox-regular at $\\bar{x}$ and partly-smooth relative to $\\M$ at $\\bar{x}$.\n\n    Take $\\gamma$, $\\bar \\gamma$ such that $0<\\gamma<\\bar \\gamma \\leq 1/r$ and $\\bar{x}=\\prox_{\\bar\\gamma g}(\\bar{y}-\\bar\\gamma \\nabla f(\\bar{y}))$. If\n    \\begin{itemize}\n        \\item[i)] $\\frac{1}{\\gamma}(\\bar{y}-\\bar{x}) - \\nabla f(\\bar{y})  \\in \\ri \\partial g(\\bar{x})$ ~\\revise{(the relative interior of the subdifferential at $\\bar{x}$)};\n        \\item[ii)] either a) $\\gamma$ is sufficiently close to $\\bar{\\gamma}$, or b) $\\bar{y}$ is sufficiently close to $\\bar{x}$;\n    \\end{itemize}\n    then, the proximal gradient $y \\mapsto \\prox_{\\gamma g}(y - \\gamma\\nabla f(y))$ is $\\C^{1}$ and $\\M$-valued near $\\bar{y}$.\n\\end{theorem}","sha256_utf8":"61ed7d5643fd2d937656148e05f2510cd79c54fcfbb8c2502e8ad849c9e6e396"}]},{"id":"S052","semantic_type":"proof","kind":"proof","title":"流形定位完整证明","source_units":[{"start":42834,"end":47887,"start_line":780,"end_line":819,"raw_tex":"\\begin{proof}\n    Adopting the same reasoning as in \\cite[Sec. 5]{lewis2002active} and \\cite[Sec. 4.1]{daniilidis2006geometrical}, we consider the function\n    \\begin{equation*}\n        \\begin{array}{rl}\n            \\rho : \\bbR^n\\times\\bbR^n & \\to \\bbR                                                 \\\\\n            (x, y)                    & \\mapsto g(x)+\\frac{1}{2\\gamma}\\|x-y+\\gamma\\nabla f(y)\\|^2,\n        \\end{array}\n    \\end{equation*}\n    and denote by $\\rho_y = \\rho(\\cdot,y)$. Computing the proximal gradient $\\prox_{\\gamma g}(y-\\gamma\\nabla f(y))$ can then be seen as minimizing the parametrized function $\\rho_y$.\n\n    \\noindent\\underline{Step 1.} As a first step, we study the minimizers of $\\rho_y$ restricted to $\\M$, for $y$ near\\;$\\bar{y}$. %\n    We consider the parametric manifold optimization problem, for $y$ near $\\bar{y}$:\n    \\begin{equation}\\tag{$ P_\\M(y)$}\n        \\min_{x\\in\\M} \\rho_y(x). \\label{pb:pb_mancstr}\n    \\end{equation}\n    Since $g$ is $\\C^2$-partly-smooth relative to $\\M$ and $f$ is $\\C^2(\\bbR^n)$, $\\rho_y$ is twice continuously differentiable on $\\M$. \\revise{Moreover, the $r$-prox-regularity gives easily (see \\cref{lemma:prox_strong_min}) that $\\rho_{\\bar{y}}$ is lower-bounded by $(\\frac{1}{\\gamma}-r)\\|\\cdot - \\bar{x}\\|^2/2$ on a neighborhood of $\\bar x$ in $\\bbR^n$ and, a fortiori, in $\\M$. From usual rationale (see e.g.\n    \\cite[Chap 4.2, 6.1]{boumal2022intromanifolds}), this implies}\n    \\begin{equation*}\n        \\grad \\rho_{\\bar{y}}(\\bar{x}) = 0 \\qquad \\Hess \\rho_{\\bar{y}}(\\bar{x}) \\succeq \\Big(\\frac{1}{\\gamma}-r\\Big) I \\succ 0,\n    \\end{equation*}\n    \\revise{which are the conditions to apply the implicit functions theorem, as follows.}\n\n    We consider the equation $\\Phi(x, y) = 0$, for $x, y$ near $\\bar{x}, \\bar{y}$, where $\\Phi:\\M\\times\\bbR^n\\to\\tangentBundle$ is defined as $\\Phi(x, y) = \\grad \\rho_y(x)$. This function is continuously differentiable on a neighborhood of $(\\bar{x}, \\bar{y})$, and its differential relative to $\\bar x$ at that point, $\\Hess \\rho_{\\bar{y}}(\\bar{x})$, is invertible. The implicit function theorem thus grants the existence of neighborhoods $\\N_{\\bar{x}}$, $\\N_{\\bar{y}}$ of $\\bar{x}$, $\\bar{y}$ in $\\M$, $\\bbR^n$, and a continuously differentiable function $\\hat{x}:\\N_{\\bar{y}} \\to \\N_{\\bar{x}}$ such that, for any $y$ in $\\N_{\\bar{y}}$, $\\Phi(\\hat{x}(y), y) = \\grad \\rho_y(\\hat{x}(y)) = 0$. Actually, $\\hat{x}(y)$ is a strong minimizer of $\\rho_y$ on $\\M$ for $y$ close enough to $\\bar{y}$. Indeed, the mapping $\\hat{x}$ is continuous on $\\N_{\\bar{y}}$, so that $y \\mapsto \\Hess \\rho_y(\\hat{x}(y))$ is also continuous there and the property $\\Hess \\rho_{\\bar{y}}(\\hat{x}(\\bar{y}))\\succ 0$ extends locally around $\\bar{y}$.\n\n    \\noindent\\underline{Step 2.}  As a second step, we turn to show that the minimizer $\\hat{x}(y)$ of $\\rho_y$ on $\\M$ is actually a strong critical point of $\\rho_y$ in $\\bbR^n$ \\cite[Def. 5.3]{lewis2002active}, and thus the proximal gradient of point $y$. More precisely, we claim that, for $y$ near $\\bar{y}$ and $x=\\hat{x}(y)$, there holds $0\\in\\ri \\partial \\rho_y(x)$, that is\n    \\begin{equation*}\n        \\frac{1}{\\gamma}(y-x) - \\nabla f(y)  \\in \\ri \\partial g(x).\n    \\end{equation*}\n\n    This property holds at $(\\bar{x}, \\bar{y})$ by assumption. By contradiction, assume there exist sequences of points $(y_r)$ with limit $\\bar{y}$, $(x_r) = (\\hat x(y_r))$ with limit $\\bar{x} = \\hat x(\\bar{y})$ and $(h_r)$ of unit norm $\\|h_r\\|=1$ such that for all $r$, $h_r$ separates 0 from $\\partial \\rho_{y_r}(x_r)$:\n    \\begin{equation*}\n        \\inf_{h\\in\\partial \\rho_{y_r}(x_r)} \\langle h_r, h\\rangle \\ge 0.\n    \\end{equation*}\n    Since $(h_r)$ is bounded, a converging subsequence can be extracted from it, let $\\bar{h}$ denote its limit. At the cost of renaming iterates, we assume that $\\lim_{r\\to\\infty} h_r = \\bar{h}$. The above property still holds at the limit $r\\to\\infty$. Indeed, let $\\bar{u}\\in\\partial \\rho_{\\bar{y}}(\\bar{x})$. Since $g$ is partly smooth, the mapping $(x, y)\\in \\N_{\\bar{x}}\\times\\N_{\\bar{y}} \\mapsto\\partial \\rho_y(x) = \\partial g(x) + \\frac{1}{\\gamma}(x-y)$ is continuous. Therefore, there exists a sequence $(u_r)$ such that $u_r\\in\\partial\\rho_{y_r}(x_r)$ and $\\lim_{r\\to\\infty} u_r=\\bar{u}$. We have for all $r$: $\\langle u_r, h_r\\rangle \\ge 0$, which yields at the limit $\\langle\\bar{u}, \\bar{h}\\rangle\\ge 0$. Thus $\\bar{h}$ separates $0$ from $\\partial \\rho_{\\bar{y}}(\\bar{x})$, which contradicts our assumption.\n\n    \\noindent\\underline{Conclusion.} We thus have a continuously differentiable function $\\hat{x}$ defined on a neighborhood of $\\bar{y}$ such that i) $\\hat{x}(\\bar{y}) = \\bar{x}$, ii) $\\hat{x}(y)$ is a strong minimizer of $\\rho_y$ on $\\M$, iii) $0\\in\\ri\\partial \\rho_y(\\hat{x}(y))$.\n\n    This last point tells us that $(y-\\hat x(y))/\\gamma -\\nabla f(y) \\in \\partial g(\\hat x(y))$. The characterization of proximity by the optimality condition (\\cref{th:proxex}) gives\nthat $\\hat{x}(y)=\\prox_{\\gamma g}(y-\\gamma \\nabla f(y))$ for $y$ close enough to $\\bar{y}$.\n\\end{proof}","sha256_utf8":"7eee440ebef07eabf94a432927500ce43f5877271b3e979d504271441d27f0b1"}]},{"id":"S053","semantic_type":"assumption","kind":"assumption","title":"定位定理的正则与资格条件","source_units":[{"start":41995,"end":42692,"start_line":771,"end_line":777,"raw_tex":"    Let $f$ be a $\\C^2$ function on $\\RR^n$ and $g$ a lower semi-continuous function on $\\RR^n$. Suppose that $g$ is both $r$-prox-regular at $\\bar{x}$ and partly-smooth relative to $\\M$ at $\\bar{x}$.\n\n    Take $\\gamma$, $\\bar \\gamma$ such that $0<\\gamma<\\bar \\gamma \\leq 1/r$ and $\\bar{x}=\\prox_{\\bar\\gamma g}(\\bar{y}-\\bar\\gamma \\nabla f(\\bar{y}))$. If\n    \\begin{itemize}\n        \\item[i)] $\\frac{1}{\\gamma}(\\bar{y}-\\bar{x}) - \\nabla f(\\bar{y})  \\in \\ri \\partial g(\\bar{x})$ ~\\revise{(the relative interior of the subdifferential at $\\bar{x}$)};\n        \\item[ii)] either a) $\\gamma$ is sufficiently close to $\\bar{\\gamma}$, or b) $\\bar{y}$ is sufficiently close to $\\bar{x}$;\n    \\end{itemize}\n","sha256_utf8":"6091aae4c74b054373b514965152a6b01cecd26ba1e0a1a00fcd7f57de415932"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"构造参数化近端目标","source_units":[{"start":42848,"end":43460,"start_line":781,"end_line":788,"raw_tex":"    Adopting the same reasoning as in \\cite[Sec. 5]{lewis2002active} and \\cite[Sec. 4.1]{daniilidis2006geometrical}, we consider the function\n    \\begin{equation*}\n        \\begin{array}{rl}\n            \\rho : \\bbR^n\\times\\bbR^n & \\to \\bbR                                                 \\\\\n            (x, y)                    & \\mapsto g(x)+\\frac{1}{2\\gamma}\\|x-y+\\gamma\\nabla f(y)\\|^2,\n        \\end{array}\n    \\end{equation*}\n    and denote by $\\rho_y = \\rho(\\cdot,y)$. Computing the proximal gradient $\\prox_{\\gamma g}(y-\\gamma\\nabla f(y))$ can then be seen as minimizing the parametrized function $\\rho_y$.\n","sha256_utf8":"74cd88fbb853d0986bc19c3bb9faafd01cb384cbea21782627fadba36374e095"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"限制问题的正定海森","source_units":[{"start":43461,"end":44532,"start_line":790,"end_line":800,"raw_tex":"    \\noindent\\underline{Step 1.} As a first step, we study the minimizers of $\\rho_y$ restricted to $\\M$, for $y$ near\\;$\\bar{y}$. %\n    We consider the parametric manifold optimization problem, for $y$ near $\\bar{y}$:\n    \\begin{equation}\\tag{$ P_\\M(y)$}\n        \\min_{x\\in\\M} \\rho_y(x). \\label{pb:pb_mancstr}\n    \\end{equation}\n    Since $g$ is $\\C^2$-partly-smooth relative to $\\M$ and $f$ is $\\C^2(\\bbR^n)$, $\\rho_y$ is twice continuously differentiable on $\\M$. \\revise{Moreover, the $r$-prox-regularity gives easily (see \\cref{lemma:prox_strong_min}) that $\\rho_{\\bar{y}}$ is lower-bounded by $(\\frac{1}{\\gamma}-r)\\|\\cdot - \\bar{x}\\|^2/2$ on a neighborhood of $\\bar x$ in $\\bbR^n$ and, a fortiori, in $\\M$. From usual rationale (see e.g.\n    \\cite[Chap 4.2, 6.1]{boumal2022intromanifolds}), this implies}\n    \\begin{equation*}\n        \\grad \\rho_{\\bar{y}}(\\bar{x}) = 0 \\qquad \\Hess \\rho_{\\bar{y}}(\\bar{x}) \\succeq \\Big(\\frac{1}{\\gamma}-r\\Big) I \\succ 0,\n    \\end{equation*}\n    \\revise{which are the conditions to apply the implicit functions theorem, as follows.}\n","sha256_utf8":"e49b88e78e94e1249079966ca182e3fd24252d7e89ec6be75d724c64ded4dff2"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"隐函数定理构造光滑极小点","source_units":[{"start":44533,"end":45558,"start_line":802,"end_line":802,"raw_tex":"    We consider the equation $\\Phi(x, y) = 0$, for $x, y$ near $\\bar{x}, \\bar{y}$, where $\\Phi:\\M\\times\\bbR^n\\to\\tangentBundle$ is defined as $\\Phi(x, y) = \\grad \\rho_y(x)$. This function is continuously differentiable on a neighborhood of $(\\bar{x}, \\bar{y})$, and its differential relative to $\\bar x$ at that point, $\\Hess \\rho_{\\bar{y}}(\\bar{x})$, is invertible. The implicit function theorem thus grants the existence of neighborhoods $\\N_{\\bar{x}}$, $\\N_{\\bar{y}}$ of $\\bar{x}$, $\\bar{y}$ in $\\M$, $\\bbR^n$, and a continuously differentiable function $\\hat{x}:\\N_{\\bar{y}} \\to \\N_{\\bar{x}}$ such that, for any $y$ in $\\N_{\\bar{y}}$, $\\Phi(\\hat{x}(y), y) = \\grad \\rho_y(\\hat{x}(y)) = 0$. Actually, $\\hat{x}(y)$ is a strong minimizer of $\\rho_y$ on $\\M$ for $y$ close enough to $\\bar{y}$. Indeed, the mapping $\\hat{x}$ is continuous on $\\N_{\\bar{y}}$, so that $y \\mapsto \\Hess \\rho_y(\\hat{x}(y))$ is also continuous there and the property $\\Hess \\rho_{\\bar{y}}(\\hat{x}(\\bar{y}))\\succ 0$ extends locally around $\\bar{y}$.\n","sha256_utf8":"72f576476992c7a6ed7b6aa29afbf06593962ebecffe45ec592a75666577e5e3"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"提出环境空间强临界性","source_units":[{"start":45559,"end":46052,"start_line":804,"end_line":807,"raw_tex":"    \\noindent\\underline{Step 2.}  As a second step, we turn to show that the minimizer $\\hat{x}(y)$ of $\\rho_y$ on $\\M$ is actually a strong critical point of $\\rho_y$ in $\\bbR^n$ \\cite[Def. 5.3]{lewis2002active}, and thus the proximal gradient of point $y$. More precisely, we claim that, for $y$ near $\\bar{y}$ and $x=\\hat{x}(y)$, there holds $0\\in\\ri \\partial \\rho_y(x)$, that is\n    \\begin{equation*}\n        \\frac{1}{\\gamma}(y-x) - \\nabla f(y)  \\in \\ri \\partial g(x).\n    \\end{equation*}\n","sha256_utf8":"2fc639c4dba2c11f53e51df00b4dd1fec1386474436e540005d99a06dc035d1a"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"分离反证与次微分连续性","source_units":[{"start":46053,"end":47314,"start_line":809,"end_line":813,"raw_tex":"    This property holds at $(\\bar{x}, \\bar{y})$ by assumption. By contradiction, assume there exist sequences of points $(y_r)$ with limit $\\bar{y}$, $(x_r) = (\\hat x(y_r))$ with limit $\\bar{x} = \\hat x(\\bar{y})$ and $(h_r)$ of unit norm $\\|h_r\\|=1$ such that for all $r$, $h_r$ separates 0 from $\\partial \\rho_{y_r}(x_r)$:\n    \\begin{equation*}\n        \\inf_{h\\in\\partial \\rho_{y_r}(x_r)} \\langle h_r, h\\rangle \\ge 0.\n    \\end{equation*}\n    Since $(h_r)$ is bounded, a converging subsequence can be extracted from it, let $\\bar{h}$ denote its limit. At the cost of renaming iterates, we assume that $\\lim_{r\\to\\infty} h_r = \\bar{h}$. The above property still holds at the limit $r\\to\\infty$. Indeed, let $\\bar{u}\\in\\partial \\rho_{\\bar{y}}(\\bar{x})$. Since $g$ is partly smooth, the mapping $(x, y)\\in \\N_{\\bar{x}}\\times\\N_{\\bar{y}} \\mapsto\\partial \\rho_y(x) = \\partial g(x) + \\frac{1}{\\gamma}(x-y)$ is continuous. Therefore, there exists a sequence $(u_r)$ such that $u_r\\in\\partial\\rho_{y_r}(x_r)$ and $\\lim_{r\\to\\infty} u_r=\\bar{u}$. We have for all $r$: $\\langle u_r, h_r\\rangle \\ge 0$, which yields at the limit $\\langle\\bar{u}, \\bar{h}\\rangle\\ge 0$. Thus $\\bar{h}$ separates $0$ from $\\partial \\rho_{\\bar{y}}(\\bar{x})$, which contradicts our assumption.\n","sha256_utf8":"4da58f411b44c05c0eb126ddc0a7836fce1424a7998c7c6f9ff7a856e7012c26"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"最优性刻画识别近端梯度","source_units":[{"start":47315,"end":47876,"start_line":815,"end_line":818,"raw_tex":"    \\noindent\\underline{Conclusion.} We thus have a continuously differentiable function $\\hat{x}$ defined on a neighborhood of $\\bar{y}$ such that i) $\\hat{x}(\\bar{y}) = \\bar{x}$, ii) $\\hat{x}(y)$ is a strong minimizer of $\\rho_y$ on $\\M$, iii) $0\\in\\ri\\partial \\rho_y(\\hat{x}(y))$.\n\n    This last point tells us that $(y-\\hat x(y))/\\gamma -\\nabla f(y) \\in \\partial g(\\hat x(y))$. The characterization of proximity by the optimality condition (\\cref{th:proxex}) gives\nthat $\\hat{x}(y)=\\prox_{\\gamma g}(y-\\gamma \\nabla f(y))$ for $y$ close enough to $\\bar{y}$.\n","sha256_utf8":"550a985c18a47593260750ed7826558bde20ffe54456bc300fd2adf18790d16e"}]},{"id":"S060","semantic_type":"lemma","kind":"lemma","title":"近端目标的局部二次增长","source_units":[{"start":86364,"end":87041,"start_line":1445,"end_line":1451,"raw_tex":"\\begin{lemma}\\label{lemma:prox_strong_min}\n    Let $f$ and $g$ denote two functions and $\\bar{x}$, $\\bar{y}$ two points such that $f$ is differentiable at $\\bar{y}$ and $g$ is $r$-prox-regular at $\\bar{x}$ for subgradient $\\frac{1}{\\gamma}(\\bar{y}-\\bar{x}) - \\nabla f(\\bar{y}) \\in \\partial g(\\bar{x})$ with $\\gamma \\in (0, 1/r)$.\n    Then, the function $\\rho_{\\bar{y}} : x\\mapsto g(x) + \\frac{1}{2\\gamma}\\|\\bar{y}-\\gamma\\nabla f(\\bar{y}) - x\\|^2$ satisfies %\n    \\begin{equation*}\n        \\rho_{\\bar{y}}(x) \\ge \\rho_{\\bar{y}}(\\bar{x}) + \\frac{1}{2} \\left( \\frac{1}{\\gamma} - r \\right) \\| x-\\bar{x} \\|^2, \\qquad \\text{for all $x$ near $\\bar{x}$.}\n    \\end{equation*}\n\\end{lemma}","sha256_utf8":"ab2ff1fdb658830a5372da3b199c88b4d75891cef28818d4220469589bc43c35"}]},{"id":"S061","semantic_type":"proof","kind":"proof","title":"二次增长完整证明","source_units":[{"start":87043,"end":88248,"start_line":1453,"end_line":1468,"raw_tex":"\\begin{proof}\n    Prox-regularity of $g$ at $\\bar{x}$ with subgradient $\\frac{1}{\\gamma} (\\bar{y}-\\gamma \\nabla f(\\bar{y}) - \\bar{x}) \\in \\partial g(\\bar{x})$ writes\n    \\begin{equation*}\n        g(x) \\ge g(\\bar{x}) + \\frac{1}{\\gamma}\\langle \\bar{y}-\\gamma \\nabla f(\\bar{y}) - \\bar{x} , x-\\bar{x} \\rangle - \\frac{r}{2} \\| x-\\bar{x} \\|^2.\n    \\end{equation*}\n    The identity $ 2\\langle b-a, c-a\\rangle = \\|b-a\\|^2 + \\|c-a\\|^2 -   \\|b-c\\|^2 $ applied to the previous scalar product yields:\n    \\begin{equation*}\n        g(x) \\ge g(\\bar{x}) + \\frac{1}{2\\gamma} \\| \\bar{y}-\\gamma \\nabla f(\\bar{y}) - \\bar{x} \\|^2 + \\frac{1}{2\\gamma} \\| x-\\bar{x} \\|^2 - \\frac{1}{2\\gamma} \\|\\bar{y}-\\gamma \\nabla f(\\bar{y}) - x \\|^2 - \\frac{r}{2} \\| x-\\bar{x} \\|^2,\n    \\end{equation*}\n    which rewrites\n    \\begin{equation*}\n        \\underbrace{ g(x) + \\frac{1}{2\\gamma} \\|\\bar{y}-\\gamma \\nabla f(\\bar{y}) - x \\|^2 }_{=\\rho_{\\bar{y}}(x) } \\ge  \\underbrace{  g(\\bar{x}) + \\frac{1}{2\\gamma} \\| \\bar{y}-\\gamma \\nabla f(\\bar{y}) - \\bar{x} \\|^2   }_{=\\rho_{\\bar{y}}(\\bar{x}) } + \\frac{1}{2} \\left( \\frac{1}{\\gamma} - r \\right) \\| x-\\bar{x} \\|^2,\n    \\end{equation*}\n    \\vspace*{-1ex}\nwhich is the claimed inequality.\n\\end{proof}","sha256_utf8":"9c59053cc5c0bd3d197b537fcb212dfacffb6d1d2682ff3e2d46f4b01d54f9fb"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"写出近端正则下界","source_units":[{"start":87057,"end":87401,"start_line":1454,"end_line":1457,"raw_tex":"    Prox-regularity of $g$ at $\\bar{x}$ with subgradient $\\frac{1}{\\gamma} (\\bar{y}-\\gamma \\nabla f(\\bar{y}) - \\bar{x}) \\in \\partial g(\\bar{x})$ writes\n    \\begin{equation*}\n        g(x) \\ge g(\\bar{x}) + \\frac{1}{\\gamma}\\langle \\bar{y}-\\gamma \\nabla f(\\bar{y}) - \\bar{x} , x-\\bar{x} \\rangle - \\frac{r}{2} \\| x-\\bar{x} \\|^2.\n    \\end{equation*}\n","sha256_utf8":"420481862a851f72d817dec5c94e20f971afb71ab894e95c2e093b81dd8f270d"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"内积恒等式展开下界","source_units":[{"start":87401,"end":87808,"start_line":1458,"end_line":1461,"raw_tex":"    The identity $ 2\\langle b-a, c-a\\rangle = \\|b-a\\|^2 + \\|c-a\\|^2 -   \\|b-c\\|^2 $ applied to the previous scalar product yields:\n    \\begin{equation*}\n        g(x) \\ge g(\\bar{x}) + \\frac{1}{2\\gamma} \\| \\bar{y}-\\gamma \\nabla f(\\bar{y}) - \\bar{x} \\|^2 + \\frac{1}{2\\gamma} \\| x-\\bar{x} \\|^2 - \\frac{1}{2\\gamma} \\|\\bar{y}-\\gamma \\nabla f(\\bar{y}) - x \\|^2 - \\frac{r}{2} \\| x-\\bar{x} \\|^2,\n    \\end{equation*}\n","sha256_utf8":"ae21faec40375f314b96e3b6c6555241d5290fdbe93f966b596acee8bc03d54f"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"整理得到二次增长不等式","source_units":[{"start":87808,"end":88237,"start_line":1462,"end_line":1467,"raw_tex":"    which rewrites\n    \\begin{equation*}\n        \\underbrace{ g(x) + \\frac{1}{2\\gamma} \\|\\bar{y}-\\gamma \\nabla f(\\bar{y}) - x \\|^2 }_{=\\rho_{\\bar{y}}(x) } \\ge  \\underbrace{  g(\\bar{x}) + \\frac{1}{2\\gamma} \\| \\bar{y}-\\gamma \\nabla f(\\bar{y}) - \\bar{x} \\|^2   }_{=\\rho_{\\bar{y}}(\\bar{x}) } + \\frac{1}{2} \\left( \\frac{1}{\\gamma} - r \\right) \\| x-\\bar{x} \\|^2,\n    \\end{equation*}\n    \\vspace*{-1ex}\nwhich is the claimed inequality.\n","sha256_utf8":"0b992f943e4c83a543b234c59fffad8580be69f6e734cae5805efed703a62f39"}]},{"id":"S065","semantic_type":"lemma","kind":"lemma","title":"近端映射局部单值与最优性","source_units":[{"start":82908,"end":83804,"start_line":1400,"end_line":1409,"raw_tex":"\\begin{lemma}\n    \\label{th:proxex}\n    Consider a function $\\funns:\\bbR^n\\to\\bbR$, a pair of points $\\bar{x}, \\bar{y}$ and a \\revise{step length} $\\bar{\\gamma}>0$ such that $\\bar{x}=\\prox_{\\bar\\gamma \\funns}(\\bar{y})$ and $\\funns$ is $r$ prox-regular at $\\bar{x}$ for subgradient $\\bar{v} \\triangleq (\\bar{y}-\\bar{x}) / \\bar{\\gamma}$.\n\n    Then, for any $\\gamma\\in(0, \\min(1/r, \\bar\\gamma))$, there exists a neighborhood $\\N_{\\bar{y}}$ of $\\bar{y}$ over which $\\prox_{\\gamma \\funns}$ is single-valued and $(1-\\gamma r)^{-1}$-Lipschitz continuous. Furthermore, there holds\n    \\begin{equation*}\n        x = \\prox_{\\gamma \\funns}(y) \\Leftrightarrow (y-x)/\\gamma \\in\\partial \\funns(x)\n    \\end{equation*}\n    for $y\\in\\N_{\\bar{y}}$ and $x$ near $\\bar{x}$ in the sense $\\|x-\\bar{x}\\|<\\varepsilon$, $|\\funns(x)-\\funns(\\bar{x})|<\\varepsilon$ and $\\|(y-x)/\\gamma - \\bar{v}\\|<\\varepsilon$. %\n\\end{lemma}","sha256_utf8":"6cdf1e0a5f14689b6324c3e8e81578c80bad292b007b7bfbc5881462f149d2a5"}]},{"id":"S066","semantic_type":"proof","kind":"proof","title":"近端映射性质完整证明","source_units":[{"start":83806,"end":86153,"start_line":1411,"end_line":1429,"raw_tex":"\\begin{proof}\n    One can easily check that prox-regularity of $\\funns$ at $\\bar{x}$ for subgradient $\\bar{v}$ is equivalent to prox-regularity of function $\\tilde{\\funns}$ around $0$ for subgradient $0$, with $\\tilde{\\funns} = \\funns(\\cdot + \\bar{x}) - \\langle \\bar{v}, \\cdot\\rangle - \\funns(\\bar{x})$ and a change of variable $\\tilde{x} = x-\\bar{x}$. Similarly, $\\bar{x}=\\prox_{\\bar\\gamma \\funns}(\\bar{y})$ is characterized by its global optimality~condition\n    \\begin{equation*}\n        \\funns(x) + \\frac{1}{2\\bar\\gamma}\\|x-\\bar{y}\\| > \\funns(\\bar{x}) + \\frac{1}{2\\bar\\gamma}\\|\\bar{x}-\\bar{y}\\|^2 \\qquad \\text{ for all } x\\neq\\bar{x},\n    \\end{equation*}\n    which we may write as\n    \\begin{equation*}\n        \\funns(x) > \\funns(\\bar{x}) + \\langle \\bar{v}, x-\\bar{x} \\rangle - \\frac{1}{2\\bar\\gamma}\\|x - \\bar{x}\\|^2 \\qquad \\text{ for all } x\\neq\\bar{x}.\n    \\end{equation*}\n    Under that same change of variables, since $\\tilde{\\funns}(0) = 0$, this optimality condition rewrites as\n    \\begin{equation*}\n        \\tilde{\\funns}(\\tilde{x}) > - \\frac{1}{2\\bar\\gamma}\\|\\tilde{x}\\|^2 \\qquad \\text{ for all } \\tilde{x}\\neq 0.\n    \\end{equation*}\n    We may thus apply Theorem 4.4 from \\cite{poliquin1996prox} to get the claimed result on $\\tilde{\\funns}$, which transfers back to $\\funns$ as our change of function and variable is bijective. We thus obtain that for $\\gamma\\in(0, \\min(1/r, \\bar\\gamma))$, on a neighborhood $\\N_{\\bar{y}}$ of $\\bar{y}$, $\\prox_{\\gamma \\funns}$ is single-valued, $(1-\\gamma r)^{-1}$-Lipschitz continuous and $\\prox_{\\gamma \\funns}(y)=[I+\\gamma T]^{-1}(y)$, where $T$ denotes the $\\funns$-attentive $\\varepsilon$-localization of $\\partial \\funns$ at $\\bar{x}$. Taking $y$ near $\\bar{y}$ and $x$ near $\\bar{x}$ such that $\\|x-\\bar{x}\\|<\\varepsilon$, $|\\funns(x)-\\funns(\\bar{x})|<\\varepsilon$ and $\\|(y-x)/\\gamma - \\bar{v}\\|<\\varepsilon$ allows to identify the localization of $\\partial \\funns(x)$ with $\\partial \\funns(x)$, so that\n    \\begin{equation*}\n        \\frac{y-x}{\\gamma}\\in \\partial \\funns(x)\\Leftrightarrow \\frac{y-x}{\\gamma}\\in T(x)\\Leftrightarrow (I+\\gamma T)(x)=y\\Leftrightarrow x=\\prox_{\\gamma \\funns}(y).\n    \\end{equation*}\n    Note that the proof of \\cite[Th. 4.4]{poliquin1996prox} includes a minor error relative to the Lipschitz constant computation, we report here a corrected value.\n\\end{proof}","sha256_utf8":"268fac5359ba1169849c6a34a27267e5f81ae0369f77d30904c7fa7289f99ce7"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"平移归一化与全局最优性","source_units":[{"start":83820,"end":84465,"start_line":1412,"end_line":1415,"raw_tex":"    One can easily check that prox-regularity of $\\funns$ at $\\bar{x}$ for subgradient $\\bar{v}$ is equivalent to prox-regularity of function $\\tilde{\\funns}$ around $0$ for subgradient $0$, with $\\tilde{\\funns} = \\funns(\\cdot + \\bar{x}) - \\langle \\bar{v}, \\cdot\\rangle - \\funns(\\bar{x})$ and a change of variable $\\tilde{x} = x-\\bar{x}$. Similarly, $\\bar{x}=\\prox_{\\bar\\gamma \\funns}(\\bar{y})$ is characterized by its global optimality~condition\n    \\begin{equation*}\n        \\funns(x) + \\frac{1}{2\\bar\\gamma}\\|x-\\bar{y}\\| > \\funns(\\bar{x}) + \\frac{1}{2\\bar\\gamma}\\|\\bar{x}-\\bar{y}\\|^2 \\qquad \\text{ for all } x\\neq\\bar{x},\n    \\end{equation*}\n","sha256_utf8":"7b22b92b04c830aee013a146ed2145765c4dde02b3ccf6f34b68ae30a9dcd003"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"转换为原点处二次支撑条件","source_units":[{"start":84465,"end":84953,"start_line":1416,"end_line":1423,"raw_tex":"    which we may write as\n    \\begin{equation*}\n        \\funns(x) > \\funns(\\bar{x}) + \\langle \\bar{v}, x-\\bar{x} \\rangle - \\frac{1}{2\\bar\\gamma}\\|x - \\bar{x}\\|^2 \\qquad \\text{ for all } x\\neq\\bar{x}.\n    \\end{equation*}\n    Under that same change of variables, since $\\tilde{\\funns}(0) = 0$, this optimality condition rewrites as\n    \\begin{equation*}\n        \\tilde{\\funns}(\\tilde{x}) > - \\frac{1}{2\\bar\\gamma}\\|\\tilde{x}\\|^2 \\qquad \\text{ for all } \\tilde{x}\\neq 0.\n    \\end{equation*}\n","sha256_utf8":"8d21a095a85644912e06e79fb7c962557e8b122ea4e35c37591ddf07aa8028ec"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"应用局部预解式定理","source_units":[{"start":84953,"end":85977,"start_line":1424,"end_line":1427,"raw_tex":"    We may thus apply Theorem 4.4 from \\cite{poliquin1996prox} to get the claimed result on $\\tilde{\\funns}$, which transfers back to $\\funns$ as our change of function and variable is bijective. We thus obtain that for $\\gamma\\in(0, \\min(1/r, \\bar\\gamma))$, on a neighborhood $\\N_{\\bar{y}}$ of $\\bar{y}$, $\\prox_{\\gamma \\funns}$ is single-valued, $(1-\\gamma r)^{-1}$-Lipschitz continuous and $\\prox_{\\gamma \\funns}(y)=[I+\\gamma T]^{-1}(y)$, where $T$ denotes the $\\funns$-attentive $\\varepsilon$-localization of $\\partial \\funns$ at $\\bar{x}$. Taking $y$ near $\\bar{y}$ and $x$ near $\\bar{x}$ such that $\\|x-\\bar{x}\\|<\\varepsilon$, $|\\funns(x)-\\funns(\\bar{x})|<\\varepsilon$ and $\\|(y-x)/\\gamma - \\bar{v}\\|<\\varepsilon$ allows to identify the localization of $\\partial \\funns(x)$ with $\\partial \\funns(x)$, so that\n    \\begin{equation*}\n        \\frac{y-x}{\\gamma}\\in \\partial \\funns(x)\\Leftrightarrow \\frac{y-x}{\\gamma}\\in T(x)\\Leftrightarrow (I+\\gamma T)(x)=y\\Leftrightarrow x=\\prox_{\\gamma \\funns}(y).\n    \\end{equation*}\n","sha256_utf8":"0c4508949160f9186673122e5613b491039d3a0aeb8f1ff27892608542d3cc32"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"说明利普希茨常数修正","source_units":[{"start":85977,"end":86142,"start_line":1428,"end_line":1428,"raw_tex":"    Note that the proof of \\cite[Th. 4.4]{poliquin1996prox} includes a minor error relative to the Lipschitz constant computation, we report here a corrected value.\n","sha256_utf8":"c0a6b9b75a1e523a12701a1ec4d0376970d099079f98ddae56c65c1117e702c2"}]},{"id":"S071","semantic_type":"lemma","kind":"lemma","title":"欧氏距离与黎曼距离局部等价","source_units":[{"start":94823,"end":95270,"start_line":1560,"end_line":1566,"raw_tex":"\\begin{lemma}\\label{lemma:Euclidean_to_Riemannian}\n    Consider a point $\\bar{x}$ of a Riemannian manifold $\\M$. For any $\\varepsilon>0$, there exists a neighborhood $\\U$ of $\\bar{x}$ in $\\M$ such that, for all $x\\in\\U$,\n    \\begin{equation*}\n        (1-\\varepsilon)\\distM(x, \\bar{x}) \\le \\|x-\\bar{x}\\| \\le (1+\\varepsilon)\\distM(x, \\bar{x}),\n    \\end{equation*}\n    where $\\|x-\\bar{x}\\|$ is the Euclidean distance in the ambient space.\n\\end{lemma}","sha256_utf8":"68af2ec1d70c4e8712b7652ab5afdff71a405dc5d266a13b29fcfdf525cfe0f5"}]},{"id":"S072","semantic_type":"proof","kind":"proof","title":"距离等价完整证明","source_units":[{"start":95271,"end":96476,"start_line":1567,"end_line":1581,"raw_tex":"\\begin{proof}\n    Let $\\bar{x}$, $x$ denote two close points on $\\M$. Consider the tangential retraction introduced in \\cref{prop:tangretraction}. As a retraction, it satisfies:\n    \\begin{equation*}\n        \\R_{\\bar{x}}(\\eta) = \\R_{\\bar{x}}(0) + \\D\\R_{\\bar{x}}(0)[\\eta] + \\mathcal O(\\|\\eta\\|^2) = \\bar{x} + \\mathcal O(\\|\\eta\\|^2).\n    \\end{equation*}\n    Taking $x=\\R_{\\bar{x}}(\\eta)$ allows to write $x = \\bar{x} + \\mathcal O(\\|\\R_{\\bar{x}}^{-1}(x)\\|^2)$, so that for any small $\\varepsilon_1>0$, there exists a small enough neighborhood $\\U_1\\subset\\U$ of $\\bar{x}$ in $\\M$ such that\n    \\begin{equation*}\n        (1-\\varepsilon_1) \\|\\R_{\\bar{x}}^{-1}(x)\\| \\le \\|x-\\bar{x}\\| \\le (1+\\varepsilon_1)\\|\\R_{\\bar{x}}^{-1}(x)\\|.\n    \\end{equation*}\n    By \\cref{lem:InvRetractionApproxLog}, for $\\varepsilon_2>0$ small enough, there exists a neighborhood $\\U_2\\subset\\U$ of $\\bar{x}$ such that,\n    \\begin{equation*}\n        (1-\\varepsilon_2)\\distM(x, \\bar{x}) \\le \\|\\R^{-1}_{\\bar{x}}(x)\\| \\le (1+\\varepsilon_2)\\distM(x, \\bar{x}).\n    \\end{equation*}\n    With $\\varepsilon_1$, $\\varepsilon_2$ such that $1-\\varepsilon=(1-\\varepsilon_1)(1-\\varepsilon_2)$, we combine the two estimates to conclude.%\n\\end{proof}","sha256_utf8":"b0388e057828718c41d78f8f67644744bbe4683d9611b7559e970584f6cd222f"}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","title":"采用切向回缩并作展开","source_units":[{"start":95285,"end":95623,"start_line":1568,"end_line":1571,"raw_tex":"    Let $\\bar{x}$, $x$ denote two close points on $\\M$. Consider the tangential retraction introduced in \\cref{prop:tangretraction}. As a retraction, it satisfies:\n    \\begin{equation*}\n        \\R_{\\bar{x}}(\\eta) = \\R_{\\bar{x}}(0) + \\D\\R_{\\bar{x}}(0)[\\eta] + \\mathcal O(\\|\\eta\\|^2) = \\bar{x} + \\mathcal O(\\|\\eta\\|^2).\n    \\end{equation*}\n","sha256_utf8":"399cbcd8e33ae4a9395daf398fb67ed2690cfa43a110146db34d2f74fde7dab4"}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","title":"欧氏位移与逆回缩范数比较","source_units":[{"start":95623,"end":96016,"start_line":1572,"end_line":1575,"raw_tex":"    Taking $x=\\R_{\\bar{x}}(\\eta)$ allows to write $x = \\bar{x} + \\mathcal O(\\|\\R_{\\bar{x}}^{-1}(x)\\|^2)$, so that for any small $\\varepsilon_1>0$, there exists a small enough neighborhood $\\U_1\\subset\\U$ of $\\bar{x}$ in $\\M$ such that\n    \\begin{equation*}\n        (1-\\varepsilon_1) \\|\\R_{\\bar{x}}^{-1}(x)\\| \\le \\|x-\\bar{x}\\| \\le (1+\\varepsilon_1)\\|\\R_{\\bar{x}}^{-1}(x)\\|.\n    \\end{equation*}\n","sha256_utf8":"4faa6cc90bcabc71b9f0125d9a1c6f29516426f0a663c111adbb98a911b5a3e6"}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","title":"组合逆回缩距离估计","source_units":[{"start":96016,"end":96465,"start_line":1576,"end_line":1580,"raw_tex":"    By \\cref{lem:InvRetractionApproxLog}, for $\\varepsilon_2>0$ small enough, there exists a neighborhood $\\U_2\\subset\\U$ of $\\bar{x}$ such that,\n    \\begin{equation*}\n        (1-\\varepsilon_2)\\distM(x, \\bar{x}) \\le \\|\\R^{-1}_{\\bar{x}}(x)\\| \\le (1+\\varepsilon_2)\\distM(x, \\bar{x}).\n    \\end{equation*}\n    With $\\varepsilon_1$, $\\varepsilon_2$ such that $1-\\varepsilon=(1-\\varepsilon_1)(1-\\varepsilon_2)$, we combine the two estimates to conclude.%\n","sha256_utf8":"afaf591186d4f88f4745ebec7d79d0ab8bb52dbca9f4b956c1ff1cc398f1f8bc"}]},{"id":"S076","semantic_type":"proposition","kind":"proposition","title":"切向投影定义二阶回缩","source_units":[{"start":91349,"end":91821,"start_line":1525,"end_line":1527,"raw_tex":"\\begin{proposition}\\label{prop:tangretraction}\n    Consider a $p$-dimensional $\\C^k$-submanifold $\\M$ of $\\bbR^n$ around a point $\\bar{x}\\in\\M$. The mapping $\\R : \\tangentBundle\\to\\M$, defined for $(x, \\eta)\\in\\tangentBundle$ near $(\\bar{x}, 0)$ by $\\proj_x(\\R(x, \\eta)) = \\eta$ defines a second-order retraction near $(\\bar{x}, 0)$. The point-wise retraction, defined as $\\R_x = \\R(x, \\cdot)$, is locally invertible with inverse $\\R_{x}^{-1}=\\proj_{x}$.\n\\end{proposition}","sha256_utf8":"3f115bf1cd06623e1aff53e5de4b555f8ebaf93f54370b8788fad7775a69a210"}]},{"id":"S077","semantic_type":"proof","kind":"proof","title":"切向回缩完整证明","source_units":[{"start":91823,"end":94821,"start_line":1529,"end_line":1558,"raw_tex":"\\begin{proof}\n    Let $\\Psi : \\bbR^n\\to\\bbR^{n-p}$ denote a $\\C^k$ function defining $\\M$ around $\\bar{x}$: for all $x$ close enough to $\\bar{x}$, there holds $x\\in\\M \\Leftrightarrow \\Psi(x)=0$, and $\\D\\Psi(x)$ is surjective. Consider the equation $\\Phi(x, \\eta_t, \\eta_n) = 0$ around $(\\bar{x}, 0, 0)$, with\n    \\begin{equation*}\n        \\begin{array}{rl}\n            \\Phi : \\left\\{ x, \\eta_t, \\eta_n : x\\in\\M, \\eta_t\\in\\tangentM, \\eta_n\\in\\normalM \\right\\} & \\to \\bbR                       \\\\\n            x, \\eta_t, \\eta_n                                                                         & \\mapsto \\Psi(x+\\eta_t+\\eta_n).\n        \\end{array}\n    \\end{equation*}\n    The partial differential $\\D_{\\eta_n}\\Phi(\\bar{x}, 0, 0)$ is, for $\\xi_n\\in\\normalM[\\bar{x}]$,\n    \\begin{equation*}\n        \\D_{\\eta_n}\\Phi(\\bar{x}, 0, 0)[\\xi_n] = \\D\\Psi(\\bar{x})[\\xi_n].\n    \\end{equation*}\n    Since $\\bar{x}\\in\\M$, $\\D_{\\eta_n}\\Phi(\\bar{x}, 0, 0)$ is surjective from $\\normalM[\\bar{x}]$ to $\\bbR^{n-p}$ so its a bijection. The implicit function theorem provides the existence of neighborhoods $\\N_{\\bar{x}}^1\\subset\\M$, $\\N_{0}^2\\subset\\cup_{x\\in\\M} \\tangentM$, $\\N_{0}^3\\subset\\cup_{x\\in\\M}\\normalM$ and a unique $\\C^k$ function $\\eta_n : \\N_{\\bar{x}}^1 \\times \\N_0^2 \\to \\N_0^3$ such that, for all $x\\in\\N_{\\bar{x}}^1$, $\\eta_t\\in\\N_0^2$ and $\\eta_n\\in\\N_0^3$, $\\eta_n(\\bar{x}, 0) = 0$ and\n    \\begin{equation*}\n        \\Phi(x, \\eta_t, \\eta_n(x, \\eta_t)) = 0 \\Leftrightarrow x+\\eta_t+\\eta_n(x, \\eta_t) \\in \\M.\n    \\end{equation*}\n    It also provides an expression for the partial derivative of $\\eta_n$ at $(x, 0)$ along $\\eta_t$: for $\\xi_t\\in\\tangentM$,\n    \\begin{equation*}\n        \\D_{\\eta_t} \\eta_n(x, 0)[\\xi_t] = - \\left[ \\D_{\\eta_n}\\Phi(x, 0, 0) \\right]^{-1} \\D_{\\eta_t} \\Phi(x, 0, 0)[\\xi_t].\n    \\end{equation*}\n    As noted before, $\\D_{\\eta_n}\\Phi(x, 0, 0)$ is bijective since $x\\in\\M$. Besides, $\\D_{\\eta_t} \\Phi(x, 0, 0) = \\D\\Phi(x)[\\xi_t]=0$ since $\\tangentM$ identifies as the kernel of $\\D\\Phi(x)$. Thus $\\D_{\\eta_t} \\eta_n(x, 0) = 0$.\n\n    Now, define a map $\\R:\\N_{\\bar{x}}^1\\times\\N_0^2 \\to \\M$ by $\\R(x, \\eta_t) = x + \\eta_t + \\eta_n(x, \\eta_t)$. This map has degree of smoothness $\\C^k$ since $\\eta_n$ is $\\C^k$, satisfies $\\R(x, 0) = x$ since $\\eta_n(x, 0)=0$ and satisfies $\\D_{\\eta_t}\\eta_n(x, 0) = I + \\D_{\\eta_t}(x, 0) = I$. Thus $\\R$ defines a retraction on a neighborhood of $(\\bar{x}, 0)$.\n\n    We turn to show the second-order property of $\\R$. Consider the smooth curve $\\smoothcurve$ defined as $c(t) = \\R(x, t\\eta)$ for some $x\\in\\N_{\\bar{x}}^1$, $\\eta_t\\in\\tangentM\\cap\\N_0^2$. It's first derivative writes\n    \\begin{equation*}\n        c'(t) = \\eta + \\D_{\\eta_t}\\eta_n(x, t\\eta)[\\eta] = \\eta.\n    \\end{equation*}\n    The acceleration of the curve $c$ is obtained by computing the derivative of $c'(\\cdot)$ in the ambient space and then projecting onto $\\tangentM$. Thus $c''(t)=0$ and in particular, $c''(0)=0$ which makes $\\R$ a second-order retraction.\n\\end{proof}","sha256_utf8":"caaef7b94d9397e950071b85762f52f44e34a2f4714e9ec618da63bab28ace67"}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","title":"建立流形方程与法向修正","source_units":[{"start":91837,"end":92493,"start_line":1530,"end_line":1536,"raw_tex":"    Let $\\Psi : \\bbR^n\\to\\bbR^{n-p}$ denote a $\\C^k$ function defining $\\M$ around $\\bar{x}$: for all $x$ close enough to $\\bar{x}$, there holds $x\\in\\M \\Leftrightarrow \\Psi(x)=0$, and $\\D\\Psi(x)$ is surjective. Consider the equation $\\Phi(x, \\eta_t, \\eta_n) = 0$ around $(\\bar{x}, 0, 0)$, with\n    \\begin{equation*}\n        \\begin{array}{rl}\n            \\Phi : \\left\\{ x, \\eta_t, \\eta_n : x\\in\\M, \\eta_t\\in\\tangentM, \\eta_n\\in\\normalM \\right\\} & \\to \\bbR                       \\\\\n            x, \\eta_t, \\eta_n                                                                         & \\mapsto \\Psi(x+\\eta_t+\\eta_n).\n        \\end{array}\n    \\end{equation*}\n","sha256_utf8":"ccf439cd6b6932a422a35a79017a48918f8a46d4f992d096aee1ac60e5c00e89"}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","title":"隐函数定理求法向分量","source_units":[{"start":92493,"end":93349,"start_line":1537,"end_line":1544,"raw_tex":"    The partial differential $\\D_{\\eta_n}\\Phi(\\bar{x}, 0, 0)$ is, for $\\xi_n\\in\\normalM[\\bar{x}]$,\n    \\begin{equation*}\n        \\D_{\\eta_n}\\Phi(\\bar{x}, 0, 0)[\\xi_n] = \\D\\Psi(\\bar{x})[\\xi_n].\n    \\end{equation*}\n    Since $\\bar{x}\\in\\M$, $\\D_{\\eta_n}\\Phi(\\bar{x}, 0, 0)$ is surjective from $\\normalM[\\bar{x}]$ to $\\bbR^{n-p}$ so its a bijection. The implicit function theorem provides the existence of neighborhoods $\\N_{\\bar{x}}^1\\subset\\M$, $\\N_{0}^2\\subset\\cup_{x\\in\\M} \\tangentM$, $\\N_{0}^3\\subset\\cup_{x\\in\\M}\\normalM$ and a unique $\\C^k$ function $\\eta_n : \\N_{\\bar{x}}^1 \\times \\N_0^2 \\to \\N_0^3$ such that, for all $x\\in\\N_{\\bar{x}}^1$, $\\eta_t\\in\\N_0^2$ and $\\eta_n\\in\\N_0^3$, $\\eta_n(\\bar{x}, 0) = 0$ and\n    \\begin{equation*}\n        \\Phi(x, \\eta_t, \\eta_n(x, \\eta_t)) = 0 \\Leftrightarrow x+\\eta_t+\\eta_n(x, \\eta_t) \\in \\M.\n    \\end{equation*}\n","sha256_utf8":"5fc02d36c1a4dcc391f257a8ece873303925c2d60eac187fb71c4a4fb8e61729"}]},{"id":"S080","semantic_type":"proof_step","kind":"proof_step","title":"法向修正的一阶导数为零","source_units":[{"start":93349,"end":93872,"start_line":1545,"end_line":1549,"raw_tex":"    It also provides an expression for the partial derivative of $\\eta_n$ at $(x, 0)$ along $\\eta_t$: for $\\xi_t\\in\\tangentM$,\n    \\begin{equation*}\n        \\D_{\\eta_t} \\eta_n(x, 0)[\\xi_t] = - \\left[ \\D_{\\eta_n}\\Phi(x, 0, 0) \\right]^{-1} \\D_{\\eta_t} \\Phi(x, 0, 0)[\\xi_t].\n    \\end{equation*}\n    As noted before, $\\D_{\\eta_n}\\Phi(x, 0, 0)$ is bijective since $x\\in\\M$. Besides, $\\D_{\\eta_t} \\Phi(x, 0, 0) = \\D\\Phi(x)[\\xi_t]=0$ since $\\tangentM$ identifies as the kernel of $\\D\\Phi(x)$. Thus $\\D_{\\eta_t} \\eta_n(x, 0) = 0$.\n","sha256_utf8":"a4565f6fbfc9e1ee7fb2c2478244668a72668d1bfe155b143c6ac887f0f53579"}]},{"id":"S081","semantic_type":"proof_step","kind":"proof_step","title":"构造并验证回缩","source_units":[{"start":93873,"end":94239,"start_line":1551,"end_line":1551,"raw_tex":"    Now, define a map $\\R:\\N_{\\bar{x}}^1\\times\\N_0^2 \\to \\M$ by $\\R(x, \\eta_t) = x + \\eta_t + \\eta_n(x, \\eta_t)$. This map has degree of smoothness $\\C^k$ since $\\eta_n$ is $\\C^k$, satisfies $\\R(x, 0) = x$ since $\\eta_n(x, 0)=0$ and satisfies $\\D_{\\eta_t}\\eta_n(x, 0) = I + \\D_{\\eta_t}(x, 0) = I$. Thus $\\R$ defines a retraction on a neighborhood of $(\\bar{x}, 0)$.\n","sha256_utf8":"17a304a1cba0a82aea5e935872ed8a8002b5666f08cd8cbcc6cc94ef2296eeab"}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","title":"曲线加速度验证二阶性质","source_units":[{"start":94240,"end":94810,"start_line":1553,"end_line":1557,"raw_tex":"    We turn to show the second-order property of $\\R$. Consider the smooth curve $\\smoothcurve$ defined as $c(t) = \\R(x, t\\eta)$ for some $x\\in\\N_{\\bar{x}}^1$, $\\eta_t\\in\\tangentM\\cap\\N_0^2$. It's first derivative writes\n    \\begin{equation*}\n        c'(t) = \\eta + \\D_{\\eta_t}\\eta_n(x, t\\eta)[\\eta] = \\eta.\n    \\end{equation*}\n    The acceleration of the curve $c$ is obtained by computing the derivative of $c'(\\cdot)$ in the ambient space and then projecting onto $\\tangentM$. Thus $c''(t)=0$ and in particular, $c''(0)=0$ which makes $\\R$ a second-order retraction.\n","sha256_utf8":"3063127e98b315a7481354a567cb280f13e46c1cdc54408066c6c7bd8e719460"}]},{"id":"S083","semantic_type":"lemma","kind":"lemma","title":"逆回缩范数近似黎曼距离","source_units":[{"start":88667,"end":89261,"start_line":1497,"end_line":1503,"raw_tex":"\\begin{lemma}\\label{lem:InvRetractionApproxLog}\n    Consider a point $\\bar{x}$ of a Riemannian manifold $\\M$, equipped with a retraction $\\R$ such that $\\R_{\\bar{x}}$ is $\\C^2$. For any $\\varepsilon>0$, there exists a neighborhood $\\U$ of $\\bar{x}$ in $\\M$ such that \n    \\begin{equation*}\n        (1-\\varepsilon)\\distM(x, \\bar{x}) \\le \\|\\R^{-1}_{\\bar{x}}(x)\\| \\le (1+\\varepsilon)\\distM(x, \\bar{x}) \\qquad \\text{ for all $x\\in\\U$.}\n    \\end{equation*}\n    where $\\R^{-1}_{\\bar{x}}:\\M\\to\\tangent{\\bar{x}}{\\M}$ is the smooth inverse of $\\R_{\\bar{x}}$ defined locally around $\\bar{x}$.\n\\end{lemma}","sha256_utf8":"c8c0dee5ab2e5f2312c4de750a5d08e8ea6b7185e017732a68954d661ef6f422"}]},{"id":"S084","semantic_type":"proof","kind":"proof","title":"逆回缩近似完整证明","source_units":[{"start":89262,"end":91168,"start_line":1504,"end_line":1521,"raw_tex":"\\begin{proof}\n    The retraction at $\\bar{x}$ can be inverted locally around $0$. Indeed, as $\\D\\R_{\\bar{x}}(0_{\\tangent{\\bar{x}}{\\M}})=I$ is invertible and $\\R_{\\bar{x}}$ is $\\C^2$, the implicit function theorem provides the existence of a $\\C^2$ inverse function $\\R^{-1}_{\\bar{x}}:\\M\\to\\tangent{\\bar{x}}{\\M}$ defined locally around $\\bar{x}$. Furthermore, one shows by differentiating the relation $\\R_{\\bar{x}}\\circ\\R_{\\bar{x}}^{-1}$ that the differential of $\\R^{-1}_{\\bar{x}}$ at $\\bar{x}$ is the identity.\n\n    We consider the function $f:\\M\\to\\RR$ defined by $f(x)=\\|\\log_{\\bar{x}}(x)\\|-\\|\\R_{\\bar{x}}^{-1}(x)\\|$. Clearly $f(\\bar{x})=0$, and $\\D f(\\bar{x})=0$ as the differentials of both $\\R_{\\bar{x}}^{-1}$ and logarithm at $\\bar{x}$ are the identity. In local coordinates $\\hat{x}=\\log_{\\bar{x}}{x}$ around $\\bar{x}$, $f$ is represented by the function $\\hat{f}=f\\circ\\exp_{\\bar{x}}: \\tangent{\\bar{x}}{\\M}\\to\\RR$. As $\\hat{f}(\\hat{\\bar{x}})=0$, $\\D \\hat{f}(\\hat{\\bar{x}})=0$ and $\\hat{f}$ is $\\C^2$, there exists some $C>0$ such that\n    \\begin{equation*}\n        -C \\|\\hat{x} - \\hat{\\bar{x}}\\|^2 \\le \\hat{f}(\\hat{x}) \\le C \\|\\hat{x} - \\hat{\\bar{x}}\\|^2 \\qquad\\text{in a neighborhood $\\hat{\\U}$ of $\\hat{\\bar{x}}$,}.\n    \\end{equation*}\n    For any $\\varepsilon>0$, by taking a small enough neighborhood $\\hat{\\U}'\\subset\\hat{\\U}$, there holds\n    \\begin{equation*}\n        -\\varepsilon \\|\\hat{x} - \\hat{\\bar{x}}\\| \\le \\hat{f}(\\hat{x}) \\le \\varepsilon \\|\\hat{x} - \\hat{\\bar{x}}\\|.\n    \\end{equation*}\n    Thus for all $x$ in $\\U = \\R_{\\bar{x}}(\\hat{\\U}')$,\n    \\begin{equation*}\n        -\\varepsilon \\|\\log_{\\bar{x}}(x)\\| \\le \\|\\log_{\\bar{x}}(x)\\|-\\|\\R_{\\bar{x}}^{-1}(x)\\| \\le \\varepsilon \\|\\log_{\\bar{x}}(x)\\|,\n    \\end{equation*}\n    as $\\hat{x}=\\log_{\\bar{x}}(x)$, $\\hat{\\bar{x}}=0$. We conclude with %\n    $\\distM(x, \\bar{x})=\\|\\hat{x} - \\hat{\\bar{x}}\\| = \\|\\log_{\\bar{x}}(x)\\|$.\n\\end{proof}","sha256_utf8":"10c6acd26a14554271456d01555ecab3bf4aac551c804200158edc4d6d1f9a97"}]},{"id":"S085","semantic_type":"proof_step","kind":"proof_step","title":"局部逆回缩及其微分","source_units":[{"start":89276,"end":89775,"start_line":1505,"end_line":1505,"raw_tex":"    The retraction at $\\bar{x}$ can be inverted locally around $0$. Indeed, as $\\D\\R_{\\bar{x}}(0_{\\tangent{\\bar{x}}{\\M}})=I$ is invertible and $\\R_{\\bar{x}}$ is $\\C^2$, the implicit function theorem provides the existence of a $\\C^2$ inverse function $\\R^{-1}_{\\bar{x}}:\\M\\to\\tangent{\\bar{x}}{\\M}$ defined locally around $\\bar{x}$. Furthermore, one shows by differentiating the relation $\\R_{\\bar{x}}\\circ\\R_{\\bar{x}}^{-1}$ that the differential of $\\R^{-1}_{\\bar{x}}$ at $\\bar{x}$ is the identity.\n","sha256_utf8":"58239ad205a00477f8eb1114bbb4131990f36856291dc00e589393ac651e5822"}]},{"id":"S086","semantic_type":"proof_step","kind":"proof_step","title":"比较对数与逆回缩范数","source_units":[{"start":89776,"end":90510,"start_line":1507,"end_line":1510,"raw_tex":"    We consider the function $f:\\M\\to\\RR$ defined by $f(x)=\\|\\log_{\\bar{x}}(x)\\|-\\|\\R_{\\bar{x}}^{-1}(x)\\|$. Clearly $f(\\bar{x})=0$, and $\\D f(\\bar{x})=0$ as the differentials of both $\\R_{\\bar{x}}^{-1}$ and logarithm at $\\bar{x}$ are the identity. In local coordinates $\\hat{x}=\\log_{\\bar{x}}{x}$ around $\\bar{x}$, $f$ is represented by the function $\\hat{f}=f\\circ\\exp_{\\bar{x}}: \\tangent{\\bar{x}}{\\M}\\to\\RR$. As $\\hat{f}(\\hat{\\bar{x}})=0$, $\\D \\hat{f}(\\hat{\\bar{x}})=0$ and $\\hat{f}$ is $\\C^2$, there exists some $C>0$ such that\n    \\begin{equation*}\n        -C \\|\\hat{x} - \\hat{\\bar{x}}\\|^2 \\le \\hat{f}(\\hat{x}) \\le C \\|\\hat{x} - \\hat{\\bar{x}}\\|^2 \\qquad\\text{in a neighborhood $\\hat{\\U}$ of $\\hat{\\bar{x}}$,}.\n    \\end{equation*}\n","sha256_utf8":"6f20eabbd1694f3165cf95a6e3a1e8423b10763aee1bd54fef961d6451e27a49"}]},{"id":"S087","semantic_type":"proof_step","kind":"proof_step","title":"缩小邻域得到相对误差","source_units":[{"start":90510,"end":91005,"start_line":1511,"end_line":1518,"raw_tex":"    For any $\\varepsilon>0$, by taking a small enough neighborhood $\\hat{\\U}'\\subset\\hat{\\U}$, there holds\n    \\begin{equation*}\n        -\\varepsilon \\|\\hat{x} - \\hat{\\bar{x}}\\| \\le \\hat{f}(\\hat{x}) \\le \\varepsilon \\|\\hat{x} - \\hat{\\bar{x}}\\|.\n    \\end{equation*}\n    Thus for all $x$ in $\\U = \\R_{\\bar{x}}(\\hat{\\U}')$,\n    \\begin{equation*}\n        -\\varepsilon \\|\\log_{\\bar{x}}(x)\\| \\le \\|\\log_{\\bar{x}}(x)\\|-\\|\\R_{\\bar{x}}^{-1}(x)\\| \\le \\varepsilon \\|\\log_{\\bar{x}}(x)\\|,\n    \\end{equation*}\n","sha256_utf8":"7285358c494936c54576b64a90d4f8bc4e09c6c2e8dc7d0872310c0b54980c81"}]},{"id":"S088","semantic_type":"proof_step","kind":"proof_step","title":"用径向距离恒等式结束","source_units":[{"start":91005,"end":91157,"start_line":1519,"end_line":1520,"raw_tex":"    as $\\hat{x}=\\log_{\\bar{x}}(x)$, $\\hat{\\bar{x}}=0$. We conclude with %\n    $\\distM(x, \\bar{x})=\\|\\hat{x} - \\hat{\\bar{x}}\\| = \\|\\log_{\\bar{x}}(x)\\|$.\n","sha256_utf8":"7325951434f2c924af52f84b2a90f7e59ee1391031e2139e195ed2b2d063aca9"}]},{"id":"S089","semantic_type":"external_fact","kind":"external_fact","title":"有限维微积分与分离工具","source_units":[],"declared_external":true,"declaration_text":["使用隐函数定理、逆函数定理、链式法则、Taylor 展开、有限维有界序列的子列紧性及凸集分离定理。"]},{"id":"S090","semantic_type":"external_fact","kind":"external_fact","title":"近端正则映射的局部预解式","source_units":[],"declared_external":true,"declaration_text":["Poliquin–Rockafellar (1996), Theorem 4.4：在作者归一化后的近端正则与二次支撑条件下，局部近端映射单值、Lipschitz，并等于适当次微分局部化的预解式。本文关于常数的作者修正另行保留。"]},{"id":"S091","semantic_type":"external_fact","kind":"external_fact","title":"黎曼牛顿的局部二次改进","source_units":[],"declared_external":true,"declaration_text":["Absil–Mahony–Sepulchre (2009), Theorem 6.3.2：正定且局部 Lipschitz 的黎曼 Hessian 在非退化极小点附近保证完整 Newton 步的局部二次收敛。"]},{"id":"S092","semantic_type":"external_fact","kind":"external_fact","title":"近端梯度充分下降","source_units":[],"declared_external":true,"declaration_text":["Beck (2017), Lemma 10.4：若 ∇f 为 L-Lipschitz，x⁺ 为步长 γ 的近端梯度点，则 F(x⁺) ≤ F(y) − (1−γL)‖x⁺−y‖²/(2γ)。"]},{"id":"S093","semantic_type":"external_fact","kind":"external_fact","title":"近端梯度次微分残差估计","source_units":[],"declared_external":true,"declaration_text":["由近端子问题最优性和梯度 Lipschitz 性，dist(0, ∂F(x⁺)) ≤ (L+1/γ)‖x⁺−y‖；作者引用 Bolte 等 (2015), Proposition 13 的证明，并使用极限次微分相应的闭性来推断聚点临界性。"]},{"id":"S094","semantic_type":"external_fact","kind":"external_fact","title":"指数映射与对数映射","source_units":[],"declared_external":true,"declaration_text":["在基点的充分小正规邻域内，指数映射和对数映射光滑互逆，基点处微分为恒等映射，且 dist_M(x, x̄)=‖log_x̄(x)‖；指数映射为二阶回缩。"]},{"id":"S095","semantic_type":"external_fact","kind":"external_fact","title":"二次增长推出海森下界","source_units":[],"declared_external":true,"declaration_text":["对光滑流形上的 C² 函数，局部二次增长下界蕴含基点处黎曼梯度为零，且黎曼 Hessian 具有相应正定下界；作者引用 Boumal (2022), Chapters 4.2 and 6.1。"]},{"id":"S096","semantic_type":"external_fact","kind":"external_fact","title":"部分光滑分析与次微分求和","source_units":[],"declared_external":true,"declaration_text":["Lewis (2002), Section 5 and Definition 5.3，以及 Daniilidis 等 (2006), Section 4.1 提供部分光滑敏感性分析框架和强临界点术语；加入光滑函数时，次微分按其梯度平移。"]},{"id":"S097","semantic_type":"external_fact","kind":"external_fact","title":"阿米霍回溯的标准性质","source_units":[],"declared_external":true,"declaration_text":["Dennis–Schnabel (1996), Theorem 6.3.2 and Algorithm A6.3.1：下降方向的一阶展开保证充分小步长满足 Armijo 条件；回溯先尝试单位步长，再几何缩小。"]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S020","target_id":"S002","kind":"supports"},{"source_id":"S021","target_id":"S002","kind":"supports"},{"source_id":"S022","target_id":"S002","kind":"supports"},{"source_id":"S023","target_id":"S002","kind":"supports"},{"source_id":"S003","target_id":"S023","kind":"supports"},{"source_id":"S004","target_id":"S021","kind":"supports"},{"source_id":"S005","target_id":"S021","kind":"supports"},{"source_id":"S005","target_id":"S027","kind":"supports"},{"source_id":"S006","target_id":"S021","kind":"supports"},{"source_id":"S006","target_id":"S049","kind":"supports"},{"source_id":"S006","target_id":"S050","kind":"supports"},{"source_id":"S007","target_id":"S020","kind":"supports"},{"source_id":"S007","target_id":"S023","kind":"supports"},{"source_id":"S008","target_id":"S027","kind":"supports"},{"source_id":"S008","target_id":"S041","kind":"supports"},{"source_id":"S009","target_id":"S023","kind":"supports"},{"source_id":"S010","target_id":"S078","kind":"supports"},{"source_id":"S010","target_id":"S080","kind":"supports"},{"source_id":"S010","target_id":"S033","kind":"supports"},{"source_id":"S010","target_id":"S046","kind":"supports"},{"source_id":"S011","target_id":"S020","kind":"supports"},{"source_id":"S011","target_id":"S026","kind":"supports"},{"source_id":"S011","target_id":"S033","kind":"supports"},{"source_id":"S011","target_id":"S055","kind":"supports"},{"source_id":"S012","target_id":"S026","kind":"supports"},{"source_id":"S012","target_id":"S037","kind":"supports"},{"source_id":"S012","target_id":"S039","kind":"supports"},{"source_id":"S012","target_id":"S081","kind":"supports"},{"source_id":"S012","target_id":"S082","kind":"supports"},{"source_id":"S012","target_id":"S085","kind":"supports"},{"source_id":"S013","target_id":"S047","kind":"supports"},{"source_id":"S013","target_id":"S023","kind":"supports"},{"source_id":"S014","target_id":"S022","kind":"supports"},{"source_id":"S014","target_id":"S058","kind":"supports"},{"source_id":"S014","target_id":"S069","kind":"supports"},{"source_id":"S015","target_id":"S022","kind":"supports"},{"source_id":"S016","target_id":"S054","kind":"supports"},{"source_id":"S016","target_id":"S067","kind":"supports"},{"source_id":"S017","target_id":"S062","kind":"supports"},{"source_id":"S017","target_id":"S067","kind":"supports"},{"source_id":"S018","target_id":"S055","kind":"supports"},{"source_id":"S018","target_id":"S056","kind":"supports"},{"source_id":"S018","target_id":"S058","kind":"supports"},{"source_id":"S019","target_id":"S045","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S020","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S027","target_id":"S025","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S028","target_id":"S023","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S032","kind":"supports"},{"source_id":"S035","target_id":"S032","kind":"supports"},{"source_id":"S036","target_id":"S032","kind":"supports"},{"source_id":"S037","target_id":"S032","kind":"supports"},{"source_id":"S038","target_id":"S032","kind":"supports"},{"source_id":"S039","target_id":"S032","kind":"supports"},{"source_id":"S040","target_id":"S032","kind":"supports"},{"source_id":"S041","target_id":"S032","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S033","target_id":"S036","kind":"supports"},{"source_id":"S033","target_id":"S037","kind":"supports"},{"source_id":"S034","target_id":"S038","kind":"supports"},{"source_id":"S035","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S042","target_id":"S023","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S047","kind":"supports"},{"source_id":"S044","target_id":"S050","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"},{"source_id":"S046","target_id":"S043","kind":"supports"},{"source_id":"S047","target_id":"S043","kind":"supports"},{"source_id":"S048","target_id":"S043","kind":"supports"},{"source_id":"S049","target_id":"S043","kind":"supports"},{"source_id":"S050","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S049","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S045","target_id":"S050","kind":"supports"},{"source_id":"S046","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S051","target_id":"S045","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S057","kind":"supports"},{"source_id":"S053","target_id":"S058","kind":"supports"},{"source_id":"S053","target_id":"S059","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S055","target_id":"S052","kind":"supports"},{"source_id":"S056","target_id":"S052","kind":"supports"},{"source_id":"S057","target_id":"S052","kind":"supports"},{"source_id":"S058","target_id":"S052","kind":"supports"},{"source_id":"S059","target_id":"S052","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S060","target_id":"S055","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S063","target_id":"S061","kind":"supports"},{"source_id":"S064","target_id":"S061","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S065","target_id":"S059","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"},{"source_id":"S067","target_id":"S066","kind":"supports"},{"source_id":"S068","target_id":"S066","kind":"supports"},{"source_id":"S069","target_id":"S066","kind":"supports"},{"source_id":"S070","target_id":"S066","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S071","target_id":"S046","kind":"supports"},{"source_id":"S072","target_id":"S071","kind":"supports"},{"source_id":"S073","target_id":"S072","kind":"supports"},{"source_id":"S074","target_id":"S072","kind":"supports"},{"source_id":"S075","target_id":"S072","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S076","target_id":"S073","kind":"supports"},{"source_id":"S077","target_id":"S076","kind":"supports"},{"source_id":"S078","target_id":"S077","kind":"supports"},{"source_id":"S079","target_id":"S077","kind":"supports"},{"source_id":"S080","target_id":"S077","kind":"supports"},{"source_id":"S081","target_id":"S077","kind":"supports"},{"source_id":"S082","target_id":"S077","kind":"supports"},{"source_id":"S078","target_id":"S079","kind":"supports"},{"source_id":"S079","target_id":"S080","kind":"supports"},{"source_id":"S079","target_id":"S081","kind":"supports"},{"source_id":"S080","target_id":"S081","kind":"supports"},{"source_id":"S080","target_id":"S082","kind":"supports"},{"source_id":"S081","target_id":"S082","kind":"supports"},{"source_id":"S083","target_id":"S075","kind":"supports"},{"source_id":"S083","target_id":"S037","kind":"supports"},{"source_id":"S084","target_id":"S083","kind":"supports"},{"source_id":"S085","target_id":"S084","kind":"supports"},{"source_id":"S086","target_id":"S084","kind":"supports"},{"source_id":"S087","target_id":"S084","kind":"supports"},{"source_id":"S088","target_id":"S084","kind":"supports"},{"source_id":"S085","target_id":"S086","kind":"supports"},{"source_id":"S086","target_id":"S087","kind":"supports"},{"source_id":"S087","target_id":"S088","kind":"supports"},{"source_id":"S089","target_id":"S026","kind":"supports"},{"source_id":"S089","target_id":"S034","kind":"supports"},{"source_id":"S089","target_id":"S035","kind":"supports"},{"source_id":"S089","target_id":"S036","kind":"supports"},{"source_id":"S089","target_id":"S037","kind":"supports"},{"source_id":"S089","target_id":"S056","kind":"supports"},{"source_id":"S089","target_id":"S058","kind":"supports"},{"source_id":"S089","target_id":"S079","kind":"supports"},{"source_id":"S089","target_id":"S080","kind":"supports"},{"source_id":"S089","target_id":"S085","kind":"supports"},{"source_id":"S089","target_id":"S086","kind":"supports"},{"source_id":"S089","target_id":"S073","kind":"supports"},{"source_id":"S090","target_id":"S069","kind":"supports"},{"source_id":"S090","target_id":"S070","kind":"supports"},{"source_id":"S091","target_id":"S023","kind":"supports"},{"source_id":"S092","target_id":"S021","kind":"supports"},{"source_id":"S093","target_id":"S022","kind":"supports"},{"source_id":"S094","target_id":"S033","kind":"supports"},{"source_id":"S094","target_id":"S039","kind":"supports"},{"source_id":"S094","target_id":"S086","kind":"supports"},{"source_id":"S094","target_id":"S088","kind":"supports"},{"source_id":"S095","target_id":"S055","kind":"supports"},{"source_id":"S096","target_id":"S057","kind":"supports"},{"source_id":"S096","target_id":"S058","kind":"supports"},{"source_id":"S097","target_id":"S026","kind":"supports"},{"source_id":"S097","target_id":"S023","kind":"supports"}]}}
{"record_id":"2101.01323v2","paper":{"title":"On the global convergence of randomized coordinate gradient descent for non-convex optimization","authors":["Ziang Chen","Yingzhou Li","Jianfeng Lu"],"arxiv_id":"2101.01323","version":"v2","journal":"SIAM Journal on Optimization","doi":"10.1137/21m1460375","abstract_url":"https://arxiv.org/abs/2101.01323v2","pdf_url":"https://arxiv.org/pdf/2101.01323v2"},"context":{"raw_tex":"\tIn this paper, we analyze the global convergence of coordinate gradient descent algorithm for smooth but non-convex optimization problem\n\n\n\\begin{equation}\\label{obj-fun}\n\t\t\\min_{x\\in \\bR^d} f(x).\n\t\\end{equation}\n\n\\begin{algorithm}[H]\n\t\t\\caption{Randomized coordinate gradient descent}\n\t\t\\label{alg:random CGD}\n\t\t\\begin{algorithmic}\n\t\t\t\\State Initialization: $x_0\\in\\bR^d$, $t=0$.\n\t\t\t\\While {not convergent}\n\t\t\t\\State Draw a coordinate $i_t$ uniformly random from $\\{1,2,\\dots,d\\}$.\n\t\t\t\\State Draw a stepsize $\\alpha_t$ uniformly random in $[\\alpha_{\\min},\\alpha_{\\max}]$.\n\t\t\t\\State $x_{t+1}\\leftarrow x_t- \\alpha_t e_{i_t} \\partial_{i_t} f(x_t)$.\n\t\t\t\\State $t\\leftarrow t+1$.\n\t\t\t\\EndWhile\n\t\t\\end{algorithmic}\n\t\\end{algorithm}\n\n\tLet $(\\Omega,\\mathcal{F},\\bP)$ be a probability space and let $\\mathbb{T}$ be a semigroup with $\\mathcal{B}(\\mathbb{T})$ being its Borel $\\sigma$-algebra. $\\mathbb{T}$ serves as the notion of time. In the setting of Algorithm~\\ref{alg:random CGD}, we have $\\mathbb{T}=\\bN$, corresponding to the one-sided discrete time setting. Other possible examples of $\\mathbb{T}$ include $\\mathbb{T}=\\bZ$, $\\mathbb{T}=\\bR_{\\geq 0}$, and $\\mathbb{T}=\\bR$, with the assumption that $0\\in\\mathbb{T}$.\n\n\n\\begin{definition}[Metric dynamical system] A metric dynamical system on a probability space $(\\Omega,\\mathcal{F},\\bP)$ is a family of maps $\\{\\theta(t):\\Omega\\rightarrow\\Omega\\}_{t\\in\\mathbb{T}}$ satisfying that\n\t\t\\begin{itemize}\n\t\t\t\\item[(i)] The mapping $\\mathbb{T}\\times\\Omega\\rightarrow\\Omega,\\ (t,\\omega)\\mapsto\\theta(t)\\omega$ is measurable;\n\t\t\t\n\t\t\t\\item[(ii)] It holds that $\\theta(0)=\\mathrm{Id}_{\\Omega}$ and $\\theta(t+s)=\\theta(t)\\circ\\theta(s),\\ \\forall\\ s,t\\in\\mathbb{T}$;\n\t\t\t\n\t\t\t\\item[(iii)] $\\theta(t)$ is $\\bP$-preserving for any $t\\in\\mathbb{T}$, where we say a map $\\theta:\\Omega\\rightarrow\\Omega$ is $\\bP$-preserving if \n\t\t\t\\begin{equation*}\n\t\t\t\t\\bP(\\theta^{-1}B)=\\bP(B),\\quad \\forall\\ B\\in\\mathcal{F}.\n\t\t\t\\end{equation*}\n\t\t\\end{itemize}\n\t\\end{definition}\n\n\\begin{definition}[Random dynamical system]\n\t\tLet $(X,\\mathcal{F}_X)$ be a measurable space and let $\\{\\theta(t):\\Omega\\rightarrow\\Omega\\}_{t\\in\\mathbb{T}}$ be a metric dynamical system on $(\\Omega,\\mathcal{F},\\bP)$. Then a random dynamical system on $(X,\\mathcal{F}_X)$ \n\t\tover $\\{\\theta(t)\\}_{t\\in\\mathbb{T}}$ is a measurable map \n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\varphi:\\mathbb{T}\\times\\Omega\\times X&\\rightarrow\\quad X,\\\\\n\t\t\t\t(t,\\omega,x)\\ \\  & \\mapsto \\varphi(t,\\omega,x),\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tsatisfying the following cocycle property: for any $\\omega\\in\\Omega$, $x\\in X$, and $s,t\\in \\mathbb{T}$, it holds that \n\t\t\\begin{equation*}\n\t\t\t\\varphi(0,\\omega,x)=x,\n\t\t\\end{equation*}\n\t\tand that\n\t\t\\begin{equation}\\label{cocycle}\n\t\t\t\\varphi(t+s,\\omega,x)=\\varphi(t,\\theta(s)\\omega,\\varphi(s,\\omega,x)).\n\t\t\\end{equation}\n\t\\end{definition}\n\n\tThe cocycle property \\eqref{cocycle} is a key property of random dynamical system: After time $s$, if we restart the system at $x_s$, the future dynamic corresponds to the sample $\\theta(s)\\omega$. Note that $\\varphi(t,\\omega,\\cdot)$ is a map on $X$, with some ambiguity of notation, we also use $\\varphi(t, \\omega)$ to denote this map on $X$ and write $\\varphi(t,\\omega)x=\\varphi(t,\\omega,x)$. Then the cocycle property \\eqref{cocycle} can be written as \n\t\\begin{equation*}\n\t\t\\varphi(t+s,\\omega)=\\varphi(t,\\theta(s)\\omega)\\circ\\varphi(s,\\omega).\n\t\\end{equation*}\n\t\n\tIn this work, we will focus on the one-sided discrete time $\\mathbb{T}=\\bN$ and $\\theta(t)=\\theta^t$, where $\\theta$ is $\\bP$-preserving and $\\theta^t$ is the $t$-fold composition of $\\theta$. \n\tSuppose that $X=\\bR^d$ and $A:\\Omega\\rightarrow\\text{GL}(d,\\bR)$ is measurable. Consider a linear random dynamical system defined as (we use $\\Phi$ for the linear system, while reserving $\\varphi$ for nonlinear dynamics considered later)\n\t\\begin{equation*}\n\t\t\\Phi(t,\\omega)=A(\\theta^{t-1}\\omega)\\cdots A(\\theta\\omega)A(\\omega),\n\t\\end{equation*}\n\twhere the right-hand side is the product of a sequences of random matrices. In this setting, the behavior of the linear system $x_t=\\Phi(t,\\omega)x_0$ is well understood by the celebrated multiplicative ergodic theorem, also known as the Oseledets theorem, which we recall in Theorem~\\ref{thm: MET}. Such type of results was first established by V.I.{} Oseledets~\\cite{oseledets} and was further developed in many works such as \\cite{raghunathan, Ruelle-1979, Walters-93}.\n\n\n\\begin{theorem}[Multiplicative ergodic theorem, {\\cite[Theorem 3.4.1]{Arnold_RDS}}]\n\t\t\\label{thm: MET}Suppose that \n\t\t\\begin{equation*}\n\t\t\t\\left(\\log\\norm{A(\\cdot)}\\right)_+,\\ \\left(\\log\\norm{A(\\cdot)^{-1}}\\right)_+\\in L^1(\\Omega,\\mathcal{F},\\bP),\n\t\t\\end{equation*}\n\t\twhere we have used the short-hand $a_+:=\\max\\{a,0\\}$. Then there exists an $\\theta$-invariant $\\widetilde{\\Omega}\\in \\mathcal{F}$ with $\\bP(\\widetilde{\\Omega})=1$, such that the followings hold for any $\\omega\\in\\widetilde{\\Omega}$:\n\t\t\\begin{itemize}\n\t\t\t\\item[(i)] It holds that the limit\n\t\t\t\\begin{equation}\\label{conv: Lamb}\n\t\t\t\t\\Lambda(\\omega)=\\lim_{t\\rightarrow\\infty}\\left(\\Phi(t,\\omega)^{\\top}\\Phi(t,\\omega)\\right)^{1/2t},\n\t\t\t\\end{equation}\n\t\t\texists and is a positive definite matrix. Here $\\Phi(t, \\omega)^{\\top}$ denotes the transposition of the matrix (as $\\Phi(t, \\omega)$ is a linear map on $X$).\n\t\t\t\\item[(ii)] Suppose  $\\Lambda(\\omega)$ has $p(\\omega)$ distinct eigenvalues, which are ordered as $e^{\\lambda_1(\\omega)}>e^{\\lambda_2(\\omega)}>\\cdots> e^{\\lambda_{p(\\omega)}(\\omega)}$. Denote $V_i(\\omega)$ the corresponding eigenspace, with dimension $d_i(\\omega)$, for  $i=1,2,\\dots,p(\\omega)$. Then the functions $p(\\cdot)$, $\\lambda_i(\\cdot)$, and $d_i(\\cdot)$, $i=1,2,\\dots,p(\\cdot)$, are all measurable and  $\\theta$-invariant on $\\widetilde{\\Omega}$.\n\t\t\t\\item[(iii)] Set $W_i(\\omega)=\\bigoplus_{j\\geq i}V_j(\\omega),\\ i=1,2,\\dots,p(\\omega)$,  \n\t\t\tand $W_{p(\\omega)+1}(\\omega)=\\{0\\}$. Then it holds that\n\t\t\t\\begin{equation}\\label{conv: Lya-exp}\n\t\t\t\t\\lim_{t\\rightarrow\\infty}\\frac{1}{t}\\log\\norm{\\Phi(t,\\omega)x}=\\lambda_i(\\omega),\\quad\\forall\\ x\\in W_i(\\omega)\\backslash W_{i+1}(\\omega),\n\t\t\t\\end{equation}  \n\t\t\tfor $i=1,2,\\dots,p(\\omega)$. The maps $V(\\cdot)$ and $W(\\cdot)$ from $\\widetilde{\\Omega}$ to the Grassmannian manifold are measurable. \n\t\t\t\\item[(iv)] It holds that\n\t\t\t\\begin{equation*}\n\t\t\t\tW_i(\\theta\\omega)=A(\\omega)W_i(\\omega).\n\t\t\t\\end{equation*}\n\t\t\t\\item[(v)] When $(\\Omega,\\mathcal{F},\\bP,\\theta)$ is ergodic, i.e., every $B\\in\\mathcal{F}$ with $\\theta^{-1}B=B$ satisfies $\\bP(B)=0$ or $\\bP(B)=1$, the functions  \n\t\t\t$p(\\cdot)$, $\\lambda_i(\\cdot)$, and $d_i(\\cdot)$, $i=1,2,\\dots,p(\\cdot)$, are constant on $\\widetilde{\\Omega}$.\n\t\t\\end{itemize}\n\t\\end{theorem}\n\n\t\t\\item{\\emph{Probability space}.} For each $t \\in \\bN$, denote  $(\\Omega_t,\\Sigma_t,\\bP_t)$ the usual probability space for the distribution $\\mathcal{U}_{\\{1,2,\\dots,n\\}}\\times \\mathcal{U}_{[\\alpha_{\\min},\\alpha_{\\max}]}$, where $\\mathcal{U}_{\\{1,2,\\dots,n\\}}$ and $\\mathcal{U}_{[\\alpha_{\\min},\\alpha_{\\max}]}$ are the uniform distributions on the set $\\{1,2,\\dots,n\\}$ and interval $[\\alpha_{\\min},\\alpha_{\\max}]$ respectively. \n\t\tLet $(\\Omega,\\mathcal{F},\\bP)$ be the product probability space of all $(\\Omega_t,\\Sigma_t,\\bP_t),\\ t\\in\\bN$. Denote $\\pi_t$ as the projection from $(\\Omega,\\mathcal{F},\\bP)$ onto $(\\Omega_t,\\Sigma_t,\\bP_t), \\ t \\in \\bN$. Thus, a sample $\\omega\\in\\Omega$ can be represented as a sequence  $((i_0,\\alpha_0),(i_1,\\alpha_1),\\dots)$, where $(i_t,\\alpha_t)=\\pi_t(\\omega),\\ t\\in\\bN$. Let $\\{\\mathcal{F}_t\\}_{t\\in \\bN}$ be the filtration defined by $$\\mathcal{F}_t=\\sigma\\left\\{(B_0\\times\\cdots\\times B_t)\\times \\left(\\prod_{j> t}\\Omega_j\\right):B_i\\in \\Sigma_i,\\ i=0,1,\\dots,t\\right\\}.$$\n\n\n\t\t\\item{\\emph{Metric dynamical system}.}\n\t\tThe metric dynamical system on $\\Omega$ is constructed by the (left) shifting operator $\\tau:\\Omega\\rightarrow\\Omega$ defined as \n\t\t\\begin{equation*}\n\t\t\t\\tau(\\omega)=\\tau(\\pi_0(\\omega),\\pi_1(\\omega),\\cdots) := (\\pi_1(\\omega),\\pi_2(\\omega),\\cdots),\n\t\t\\end{equation*}\n\t\twhich is clearly measurable and $\\bP$-preserving. The metric dynamical system is then given by $\\theta(t)=\\tau^t$ for $t\\in\\bN$. \n\n\n\t\t\\item{\\emph{Random dynamical system}.}\n\t\tFor any $\\omega\\in\\Omega$ and $t\\in\\bN$, we define $\\phi(\\omega)$ to be a (nonlinear) map on $\\bR^d$ as \n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\phi(\\omega):\\bR^d&\\rightarrow\\quad\\quad\\quad \\bR^d\\\\\n\t\t\t\tx\\ &\\mapsto x-\\alpha e_i e_i^{\\top} \\nabla f(x),\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\twhere $(i,\\alpha)=\\pi_0(\\omega)$ is the first pair/element in the sequence $\\omega$, and we define the map $\\varphi(t,\\omega)$ via \n\t\t\\begin{equation*}\n\t\t\t\\varphi(t,\\omega)=\\phi(\\tau^{t-1}\\omega)\\circ\\cdots \\circ \\phi(\\tau\\omega)\\circ \\phi(\\omega), \\qquad \\text{for } t \\geq 1,\n\t\t\\end{equation*}\n\t\twhile $\\varphi(0, \\omega)$ is the identity operator. It is clear that $\\varphi(t,\\omega)$ satisfies the cocycle property \\eqref{cocycle} and hence defines a random dynamical system on $X=\\bR^d$ over $\\{\\tau^t\\}_{t\\in\\bN}$.\n\t\tThe iterate of Algorithm~\\ref{alg:random CGD} follows the random dynamical system as\n\t\t\\begin{equation*}\n\t\t\tx_t=\\phi(\\tau^{t-1}\\omega)x_{t-1}=\\cdots=\\phi(\\tau^{t-1}\\omega)\\circ\\cdots \\circ \\phi(\\tau\\omega)\\circ \\phi(\\omega)x_0=\\varphi(t,\\omega)x_0.\n\t\t\\end{equation*}\n\t\tIt can be seen that $\\{x_t\\}_{t\\in\\bN}$ is $\\{\\mathcal{F}_t\\}$-predictable, i.e., $x_t$ is $\\mathcal{F}_{t-1}$-measurable for any $t\\in\\bNp$, since $x_t$ is determined by samples $(i_0,\\alpha_0),(i_1,\\alpha_1),\\dots,(i_{t-1},\\alpha_{t-1})$.\n\n\n\tIn our analysis, we will use linearization of the dynamical system $\\varphi(t,\\omega)$ at a critical point $x^*$ of $f$. Without loss of generality, we assume $x^*=0$; otherwise we consider the system with state being $x-x^*$. The resulting linear system, which depends on $H=\\nabla^2 f(x^*)=(H_{ij})_{1\\leq i,j\\leq d}$, is given by (here and in the sequel, we use the superscript $H$ to indicate dependence on the matrix)\n\t\\begin{equation}\\label{linear: Phi}\n\t\t\\Phi^H(t,\\omega)=A^H(\\tau^{t-1}\\omega)\\cdots A^H(\\tau\\omega)A^H(\\omega),\n\t\\end{equation}\n\twhere \n\t\\begin{equation}\\label{linear: A}\n\t\tA^H(\\omega)=I-\\alpha e_i e_i^{\\top} H,\\quad (i,\\alpha)=\\pi_0(\\omega). \n\t\\end{equation}\n\tNote that $A^H(\\cdot)$ is bounded in $\\Omega$. \n\tWe know that $\\left(\\log\\norm{A^H(\\cdot)}\\right)_+$ is integrable. When $\\alpha< 1/|H_{ii}|$, \n\tthe matrix $A^H(\\omega)=I-\\alpha e_i e_i^{\\top} H$ is invertable, and the inverse is given explicitly by applying the Sherman-Morrison formula:\n\t\\begin{equation}\\label{inverse-AH}\n\t\t\\begin{split}\n\t\t\tA^H(\\omega)^{-1}&=\\left(I-\\alpha e_i e_i^{\\top} H\\right)^{-1} = I + \\frac{\\alpha e_i e_i^{\\top} H}{1 - \\alpha H_{ii}}. \n\t\t\\end{split}\n\t\\end{equation}\n\tIn particular, we have \n\t\\begin{equation}\n\t\t\\norm{A^H(\\omega)^{-1}} \\leq 1 + \\frac{\\alpha \\norm{H}}{1 - \\alpha |H_{ii}|}. \n\t\\end{equation}\n\tThus, if we take the maximal stepsize $\\alpha_{\\max}$ such that $\\alpha_{\\max} < 1/\\max_{1\\leq i\\leq d}|H_{ii}|$,  $\\norm{A^H(\\cdot)^{-1}}$ is bounded in $\\Omega$, and as a result $\\left(\\log\\norm{A^H(\\cdot)^{-1}}\\right)_+$ is also integrable. \n\tTherefore, the assumptions of Theorem~\\ref{thm: MET} hold. The shifting operator $\\tau$ is ergodic on $(\\Omega,\\mathcal{F},\\bP)$ by Kolmogorov's $0$--$1$ law. Then Theorem~\\ref{thm: MET} applies for $\\theta=\\tau$ with $p^H(\\cdot)$, $\\lambda_i^H(\\cdot)$, and $d_i^H(\\cdot)$ all being a.e. constant. For any $\\omega\\in\\widetilde{\\Omega}$ that is the set in Theorem~\\ref{thm: MET} satisfying $\\bP(\\widetilde{\\Omega})=1$, \n\twe denote\n\t\\begin{equation}\\label{W_pm}\n\t\tW_+^H(\\omega)=\\bigoplus_{\\lambda_i>0}V_i^H(\\omega),\\ \\text{and}\\ W_-^H(\\omega)=\\bigoplus_{\\lambda_i\\leq 0}V_i^H(\\omega).\n\t\\end{equation}\n\tThen the following invariant property holds \n\t\\begin{equation*}\n\t\tW_-^H(\\tau\\omega)=A^H(\\omega)W_-^H(\\omega).\n\t\\end{equation*}\n\tNote that $W_-^H(\\omega)$ works as a center-stable subspace. That is, for any $x\\in W_-^H(\\omega)$ and any $\\epsilon>0$, it holds that $\\norm{\\Phi(t,\\omega)x}\\leq e^{t\\epsilon}$, for sufficiently large $t$, and for $x\\notin W_-^H(\\omega)$, $\\norm{\\Phi(t,\\omega)x}$ grows exponentially as $t\\rightarrow\\infty$ with rate greater than $\\min_{\\lambda_i>0}\\lambda_i -\\epsilon$ for any $\\epsilon>0$.\n\n\n\\begin{assumption}\\label{Assump: Hess_bdd} $f \\in C^2(\\bR^d)$ and the Hessian $\\nabla^2 f$ is uniformly bounded, i.e., there exists $M>0$ such that $\\norm{\\nabla^2 f(x)}\\leq M$ for all $x\\in\\bR^d$.\n\t\\end{assumption}\n\n\tAn optimization algorithm is expected to converge, under some reasonable assumptions, to a critical point of $f$ where the gradient vanishes. Our aim is to further characterize the possible limits of the algorithm iterates. For this purpose, we distinguish $\\calA$, the set of all strict saddle points (including local maxima with non-degenerate Hessian) of $f$:\n\t\\begin{equation*}\n\t\t\\calA := \\{x\\in\\bR^d:\\nabla f(x)=0,\\\n\t\t\\lambda_{\\min}(\\nabla^2 f(x))<0\\},\n\t\\end{equation*}\n\twhere we use the subscript $s$ to emphasize that it is strict. \n\n\n\\begin{assumption}\\label{Assump: non-degenerate}\n\t\tFor every $x^*\\in \\calA$, $\\nabla^2 f(x^*)$ is non-degenerate, i.e., $x^*$ is a non-degenerate critical point of $f$ in the sense that any eigenvalue of $\\nabla^2 f(x^*)$ is nonzero.\n\t\\end{assumption}\n\n\\begin{assumption}\\label{Assump: non-zero-proj}\n\t\tFor every $x^*\\in \\calA$, it holds that $\\mathcal{P}_+^H(\\omega) e_i\\neq 0$, for every $i\\in\\{1,2,\\dots,d\\}$ and almost every $\\omega\\in\\Omega$, where $\\mathcal{P}_+^H(\\omega)$ is the orthogonal projection onto $W_+^H(\\omega)$ with $H=\\nabla^2 f(x^*)$.\n\t\\end{assumption}\n\n\tGiven an initial guess $x_0$, the behavior of the algorithm, in particular, the limit of $x_t$, depends on the particular sample $\\omega \\in \\Omega$.\n\tFor any $x^*\\in\\calA$, we denote the set of all $\\omega$ such that the algorithm starting at $x_0$ would converge to $x^*$:\n\t\\begin{equation*}\n\t\t\\calC(x^*,x_0) :=\\left\\{\\omega\\in\\Omega:\\lim_{t\\rightarrow\\infty}x_t=\\lim_{t\\rightarrow\\infty}\\varphi(t,\\omega)x_0=x^*\\right\\}.\n\t\\end{equation*}\n\tWe further define the set $\\calC(\\calA,x_0)$ as the union of all $\\calC(x^*,x_0)$ over $x^*\\in\\calA$,\n\t\\begin{equation*}\n\t\t\\calC(\\calA,x_0):=\\bigcup_{x^*\\in\\calA}\\calC(x^*,x_0).\n\t\\end{equation*}\n\n\n\tWe will first study the linear dynamical system, for which we assume the objective function is given by \n\n\n\\begin{equation}\\label{obj: quadratic}\n\t\tf^H(x)=\\frac{1}{2}x^{\\top} H x,\n\t\\end{equation}\n\n\twhere $H$ is a symmetric matrix in $\\bR^{d\\times d}$ with at least one negative eigenvalue. In this case, the coordinate descent algorithm is given by \n\t\\begin{equation*}\n\t\tx_{t+1}=\\left(I-\\alpha_t e_{i_t} e_{i_t}^{\\top} H\\right)x_t,\n\t\\end{equation*}\n\twhich corresponds to the linear dynamical system  $\\Phi^H(t,\\omega)$ with single step map $A^H(\\omega)$,  defined in \\eqref{linear: Phi} and \\eqref{linear: A} respectively. \n\n\n\tIn this subsection, we study the behavior of the nonlinear dynamical system near a strict saddle point of $f$, which without loss of generality can be assumed to be $x^*=0$. As mentioned above, in a small neighborhood of $x^*$, while it is not possible to control the difference between nonlinear and linear systems for infinite time,  the nonlinear system can be approximated by the linear system during a finite time horizon. \n\n\n\tWe first set the small constant $\\epsilon$ in Theorem~\\ref{thm: local-growth} which controls the failure probability of the amplification bound. Let $\\lambda_1>\\lambda_2>\\cdots>\\lambda_p$ be the Lyapunov exponents of the linearized system at $x^*=0$. We set\n\t\\begin{equation}\\label{lambplus-gamma}\n\t\t\\lambda_+=\\min_{\\lambda_i>0}\\lambda_i\\quad\\text{and}\\quad 0<\\gamma<\\frac{1}{2}\\min\\left\\{\\min_{1\\leq i< p}|\\lambda_i-\\lambda_{i+1}|,\\lambda_+\\right\\}. \n\t\\end{equation}\n\tNote that $\\gamma<\\lambda_+$.\n\tLet $\\epsilon_*\\in(0,1/6)$ be sufficiently small such that\n\t\\begin{equation}\\label{def: eps}\n\t\t(1-6\\epsilon)(\\lambda_+-\\gamma)+6\\epsilon \\cdot \\log(1-M\\alpha_{\\max})>0,\\quad \\forall\\ \\epsilon\\in(0,\\epsilon_*). \n\t\\end{equation}\n\tThe reason for such choice will become clear later (see \\eqref{large S2}). For the rest of the section, we will consider a fixed $\\epsilon\\in(0,\\epsilon_*)$. \n\n\n\tWe now carry out the amplification part of the finite block analysis \n\tfor the linearized dynamics at $x^* = 0$. To simplify expressions in the following, for $t_1<t_2$, we introduce the short-hand notation\n\t\\begin{equation*}\n\t\t(i,\\alpha)_{t_1:t_2-1}=\\big((i_{t_1},\\alpha_{t_1}),\\dots,(i_{t_2-1},\\alpha_{t_2-1})\\big)\\in\\Omega_{t_1}\\times\\cdots\\times\\Omega_{t_2-1},\n\t\\end{equation*}\n\tand the finite time transition matrix (i.e., composition of linear maps)\n\t\\begin{equation*}\n\t\t\\Phi^H\\bigl((i,\\alpha)_{t_1:t_2-1}\\bigr)=(I-\\alpha_{t_2-1}e_{i_{t_2-1}}e_{i_{t_2-1}}^{\\top} H)\\cdots (I-\\alpha_{t_1} e_{i_{t_1}}e_{i_{t_1}}^{\\top} H).\n\t\\end{equation*}\n\tRecall that $(\\Omega_t,\\Sigma_t,\\bP_t)$ is the probability space for $\\mathcal{U}_{\\{1,2,\\dots,d\\}}\\times \\mathcal{U}_{[\\alpha_{\\min},\\alpha_{\\max}]}$ for $t\\in\\bN$. We also denote $\\mathcal{P}_+^H\\bigl((i,\\alpha)_{t_1:t_2-1}\\bigr)$ as the projection operator onto the subspace spanned by the right singular vectors of $\\Phi^H\\bigl((i,\\alpha)_{t_1:t_2-1}\\bigr)$ corresponding to $d_+$ largest singular values, where $d_+=\\sum_{\\lambda_i>0}d_i$ and $d_i$ is the dimension of the $i$-th eigenspace as in Theorem~\\ref{thm: MET} (ii) for the linearized system at $x^*$.\n\n\n\tWe will then be able to handle $\\norm{\\mathcal{P}_+^H((i,\\alpha)_{0:S-1})e_j}$ using Lemma~\\ref{lem: Omega1} and the closeness between $\\left(\\Phi^H(S,\\omega)^{\\top} \\Phi^H(S,\\omega)\\right)^{1/2S}$ with $\\Lambda(\\omega)$ as the former converges to the latter as $S \\to \\infty$ by Theorem~\\ref{thm: MET}. \n\tMore precisely, denote the singular values of $X\\in\\bR^{d\\times d}$ by $s_1(X)\\geq s_2(X)\\geq \\cdots\\geq s_d(X)$. Then for $S\\in\\bNp$ sufficiently large,\n\twe have \n\t\\begin{equation}\\label{close: singular value}\n\t\t\\left|\\frac{1}{S}\\log s_j\\left(\\Phi^H(S,\\omega)\\right)- \\lambda_{\\mu(j)}\\right|=\\left|\\frac{1}{S}\\log s_j\\left(\\Phi^H\\bigl((i,\\alpha)_{0:S-1}\\bigr)\\right)- \\lambda_{\\mu(j)}\\right|\\leq \\gamma,\n\t\\end{equation}\n\tfor every $j\\in\\{1,2,\\dots,d\\}$, where $\\lambda_1>\\lambda_2>\\dots>\\lambda_p$ are the Lyapunov exponents from Theorem~\\ref{thm: MET}, $\\gamma$ is given by \\eqref{lambplus-gamma}, and the map $\\mu:\\{1,2,\\dots,d\\}\\rightarrow\\{1,2,\\dots,p\\}$ satisfies that $\\mu(j)=i$ if and only if $d_1+\\cdots +d_{i-1}<j\\leq d_1+\\cdots +d_i$, so $\\mu$ corresponds the index for the singular values with that of the Lyapunov exponents. \n\tMoreover,\n\tthe convergence also implies that \n\t\\begin{equation*}\n\t\t\\norm{\\mathcal{P}_+^H(S,\\omega)-\\mathcal{P}_+^H(\\omega)}\\leq\\frac{\\delta}{2},\n\t\\end{equation*}\n\tfor sufficiently large $S$, which then leads to\n\t\\begin{equation}\\label{close: singular space}\n\t\t\\norm{\\mathcal{P}^H_+\\left(S,\\omega\\right)e_j}=\\norm{\\mathcal{P}^H_+\\left((i,\\alpha)_{0:S-1}\\right)e_j}\\geq \\frac{\\delta}{2},\n\t\\end{equation}\n\tfor every $j\\in\\{1,2,\\dots,d\\}$,\n\twhere $\\mathcal{P}^H_+(S,\\omega)$ is the projection operator onto the subspace spanned by the right singular vectors of $\\Phi^H(S,\\omega)$ corresponding to $d_+$ largest singular values. Let \n\t\\begin{equation}\n\t\t\\Omega^S=\\bigl\\{(i,\\alpha)_{0:S-1}\\in \\Omega_0\\times\\cdots\\times\\Omega_{S-1}:\\eqref{close: singular value}\\text{ and }\\eqref{close: singular space}\\text{ hold}\\bigr\\}.\n\t\\end{equation}\n\tThe following lemma states that $\\Omega^S$ has high probability for sufficiently large $S$, where with slight abuse of notation, we write $\\bP(\\Omega^S)=\\bP\\left(\\Omega^S\\times\\left(\\bigtimes_{t\\geq S}\\Omega_t\\right)\\right)$.\n\n\n\tNote that $\\alpha_{\\varrho_t-1}\\sim\\mathcal{U}_{[\\alpha_{\\min},\\alpha_{\\max}]}$ is independent of $\\mathcal{F}_{\\varrho_t-2}$, $i_{\\varrho_t-1}$, and $(i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}$. \n\tThe next lemma shows that with high probability, the choice of $\\alpha_{\\varrho_t-1}$ will lead to a nontrivial orthogonal projection of $x_{\\varrho_t}$ onto the unstable subspace of $\\Phi^H(S,\\tau^{\\varrho_t}\\omega)=\\Phi^H\\bigl((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\bigr)$.\n","units":[{"start":3457,"end":3595,"start_line":86,"end_line":86,"raw_tex":"\tIn this paper, we analyze the global convergence of coordinate gradient descent algorithm for smooth but non-convex optimization problem\n","sha256_utf8":"f3c8569970a20a027fe2c6fe9ca68d360d864251c658ded2b6f4c0b4262e2b26"},{"start":3596,"end":3669,"start_line":87,"end_line":89,"raw_tex":"\\begin{equation}\\label{obj-fun}\n\t\t\\min_{x\\in \\bR^d} f(x).\n\t\\end{equation}","sha256_utf8":"07b9351ce3419cc85c7c52f16a8afa4777bd871178b01a0302ab5da59f78cd9e"},{"start":3821,"end":4333,"start_line":91,"end_line":103,"raw_tex":"\\begin{algorithm}[H]\n\t\t\\caption{Randomized coordinate gradient descent}\n\t\t\\label{alg:random CGD}\n\t\t\\begin{algorithmic}\n\t\t\t\\State Initialization: $x_0\\in\\bR^d$, $t=0$.\n\t\t\t\\While {not convergent}\n\t\t\t\\State Draw a coordinate $i_t$ uniformly random from $\\{1,2,\\dots,d\\}$.\n\t\t\t\\State Draw a stepsize $\\alpha_t$ uniformly random in $[\\alpha_{\\min},\\alpha_{\\max}]$.\n\t\t\t\\State $x_{t+1}\\leftarrow x_t- \\alpha_t e_{i_t} \\partial_{i_t} f(x_t)$.\n\t\t\t\\State $t\\leftarrow t+1$.\n\t\t\t\\EndWhile\n\t\t\\end{algorithmic}\n\t\\end{algorithm}","sha256_utf8":"04f89a486c1d27a4cf0a4509b99428618df495eccb555dee67a3256556f2beac"},{"start":13735,"end":14222,"start_line":144,"end_line":144,"raw_tex":"\tLet $(\\Omega,\\mathcal{F},\\bP)$ be a probability space and let $\\mathbb{T}$ be a semigroup with $\\mathcal{B}(\\mathbb{T})$ being its Borel $\\sigma$-algebra. $\\mathbb{T}$ serves as the notion of time. In the setting of Algorithm~\\ref{alg:random CGD}, we have $\\mathbb{T}=\\bN$, corresponding to the one-sided discrete time setting. Other possible examples of $\\mathbb{T}$ include $\\mathbb{T}=\\bZ$, $\\mathbb{T}=\\bR_{\\geq 0}$, and $\\mathbb{T}=\\bR$, with the assumption that $0\\in\\mathbb{T}$.\n","sha256_utf8":"500be49d4b708b2ac1c8da04db8176443e3b0e0a788019b970e230a5d701e351"},{"start":14842,"end":15616,"start_line":147,"end_line":158,"raw_tex":"\\begin{definition}[Metric dynamical system] A metric dynamical system on a probability space $(\\Omega,\\mathcal{F},\\bP)$ is a family of maps $\\{\\theta(t):\\Omega\\rightarrow\\Omega\\}_{t\\in\\mathbb{T}}$ satisfying that\n\t\t\\begin{itemize}\n\t\t\t\\item[(i)] The mapping $\\mathbb{T}\\times\\Omega\\rightarrow\\Omega,\\ (t,\\omega)\\mapsto\\theta(t)\\omega$ is measurable;\n\t\t\t\n\t\t\t\\item[(ii)] It holds that $\\theta(0)=\\mathrm{Id}_{\\Omega}$ and $\\theta(t+s)=\\theta(t)\\circ\\theta(s),\\ \\forall\\ s,t\\in\\mathbb{T}$;\n\t\t\t\n\t\t\t\\item[(iii)] $\\theta(t)$ is $\\bP$-preserving for any $t\\in\\mathbb{T}$, where we say a map $\\theta:\\Omega\\rightarrow\\Omega$ is $\\bP$-preserving if \n\t\t\t\\begin{equation*}\n\t\t\t\t\\bP(\\theta^{-1}B)=\\bP(B),\\quad \\forall\\ B\\in\\mathcal{F}.\n\t\t\t\\end{equation*}\n\t\t\\end{itemize}\n\t\\end{definition}","sha256_utf8":"aeced4d24abe3c9853b4c0b9db8dd9038f69b0a463c8e35f82928c83b03b5a29"},{"start":15684,"end":16542,"start_line":162,"end_line":179,"raw_tex":"\\begin{definition}[Random dynamical system]\n\t\tLet $(X,\\mathcal{F}_X)$ be a measurable space and let $\\{\\theta(t):\\Omega\\rightarrow\\Omega\\}_{t\\in\\mathbb{T}}$ be a metric dynamical system on $(\\Omega,\\mathcal{F},\\bP)$. Then a random dynamical system on $(X,\\mathcal{F}_X)$ \n\t\tover $\\{\\theta(t)\\}_{t\\in\\mathbb{T}}$ is a measurable map \n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\varphi:\\mathbb{T}\\times\\Omega\\times X&\\rightarrow\\quad X,\\\\\n\t\t\t\t(t,\\omega,x)\\ \\  & \\mapsto \\varphi(t,\\omega,x),\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tsatisfying the following cocycle property: for any $\\omega\\in\\Omega$, $x\\in X$, and $s,t\\in \\mathbb{T}$, it holds that \n\t\t\\begin{equation*}\n\t\t\t\\varphi(0,\\omega,x)=x,\n\t\t\\end{equation*}\n\t\tand that\n\t\t\\begin{equation}\\label{cocycle}\n\t\t\t\\varphi(t+s,\\omega,x)=\\varphi(t,\\theta(s)\\omega,\\varphi(s,\\omega,x)).\n\t\t\\end{equation}\n\t\\end{definition}","sha256_utf8":"03bc5585f50b44a45b3516f1fd639a5b1364c9a6f9ff4626e3a68697c8c114f2"},{"start":16545,"end":18127,"start_line":181,"end_line":191,"raw_tex":"\tThe cocycle property \\eqref{cocycle} is a key property of random dynamical system: After time $s$, if we restart the system at $x_s$, the future dynamic corresponds to the sample $\\theta(s)\\omega$. Note that $\\varphi(t,\\omega,\\cdot)$ is a map on $X$, with some ambiguity of notation, we also use $\\varphi(t, \\omega)$ to denote this map on $X$ and write $\\varphi(t,\\omega)x=\\varphi(t,\\omega,x)$. Then the cocycle property \\eqref{cocycle} can be written as \n\t\\begin{equation*}\n\t\t\\varphi(t+s,\\omega)=\\varphi(t,\\theta(s)\\omega)\\circ\\varphi(s,\\omega).\n\t\\end{equation*}\n\t\n\tIn this work, we will focus on the one-sided discrete time $\\mathbb{T}=\\bN$ and $\\theta(t)=\\theta^t$, where $\\theta$ is $\\bP$-preserving and $\\theta^t$ is the $t$-fold composition of $\\theta$. \n\tSuppose that $X=\\bR^d$ and $A:\\Omega\\rightarrow\\text{GL}(d,\\bR)$ is measurable. Consider a linear random dynamical system defined as (we use $\\Phi$ for the linear system, while reserving $\\varphi$ for nonlinear dynamics considered later)\n\t\\begin{equation*}\n\t\t\\Phi(t,\\omega)=A(\\theta^{t-1}\\omega)\\cdots A(\\theta\\omega)A(\\omega),\n\t\\end{equation*}\n\twhere the right-hand side is the product of a sequences of random matrices. In this setting, the behavior of the linear system $x_t=\\Phi(t,\\omega)x_0$ is well understood by the celebrated multiplicative ergodic theorem, also known as the Oseledets theorem, which we recall in Theorem~\\ref{thm: MET}. Such type of results was first established by V.I.{} Oseledets~\\cite{oseledets} and was further developed in many works such as \\cite{raghunathan, Ruelle-1979, Walters-93}.\n","sha256_utf8":"500b57180f8f1f6df46722d295466cb5590fdb80a139ceb80e1ea818e4aabf19"},{"start":18130,"end":20388,"start_line":193,"end_line":219,"raw_tex":"\\begin{theorem}[Multiplicative ergodic theorem, {\\cite[Theorem 3.4.1]{Arnold_RDS}}]\n\t\t\\label{thm: MET}Suppose that \n\t\t\\begin{equation*}\n\t\t\t\\left(\\log\\norm{A(\\cdot)}\\right)_+,\\ \\left(\\log\\norm{A(\\cdot)^{-1}}\\right)_+\\in L^1(\\Omega,\\mathcal{F},\\bP),\n\t\t\\end{equation*}\n\t\twhere we have used the short-hand $a_+:=\\max\\{a,0\\}$. Then there exists an $\\theta$-invariant $\\widetilde{\\Omega}\\in \\mathcal{F}$ with $\\bP(\\widetilde{\\Omega})=1$, such that the followings hold for any $\\omega\\in\\widetilde{\\Omega}$:\n\t\t\\begin{itemize}\n\t\t\t\\item[(i)] It holds that the limit\n\t\t\t\\begin{equation}\\label{conv: Lamb}\n\t\t\t\t\\Lambda(\\omega)=\\lim_{t\\rightarrow\\infty}\\left(\\Phi(t,\\omega)^{\\top}\\Phi(t,\\omega)\\right)^{1/2t},\n\t\t\t\\end{equation}\n\t\t\texists and is a positive definite matrix. Here $\\Phi(t, \\omega)^{\\top}$ denotes the transposition of the matrix (as $\\Phi(t, \\omega)$ is a linear map on $X$).\n\t\t\t\\item[(ii)] Suppose  $\\Lambda(\\omega)$ has $p(\\omega)$ distinct eigenvalues, which are ordered as $e^{\\lambda_1(\\omega)}>e^{\\lambda_2(\\omega)}>\\cdots> e^{\\lambda_{p(\\omega)}(\\omega)}$. Denote $V_i(\\omega)$ the corresponding eigenspace, with dimension $d_i(\\omega)$, for  $i=1,2,\\dots,p(\\omega)$. Then the functions $p(\\cdot)$, $\\lambda_i(\\cdot)$, and $d_i(\\cdot)$, $i=1,2,\\dots,p(\\cdot)$, are all measurable and  $\\theta$-invariant on $\\widetilde{\\Omega}$.\n\t\t\t\\item[(iii)] Set $W_i(\\omega)=\\bigoplus_{j\\geq i}V_j(\\omega),\\ i=1,2,\\dots,p(\\omega)$,  \n\t\t\tand $W_{p(\\omega)+1}(\\omega)=\\{0\\}$. Then it holds that\n\t\t\t\\begin{equation}\\label{conv: Lya-exp}\n\t\t\t\t\\lim_{t\\rightarrow\\infty}\\frac{1}{t}\\log\\norm{\\Phi(t,\\omega)x}=\\lambda_i(\\omega),\\quad\\forall\\ x\\in W_i(\\omega)\\backslash W_{i+1}(\\omega),\n\t\t\t\\end{equation}  \n\t\t\tfor $i=1,2,\\dots,p(\\omega)$. The maps $V(\\cdot)$ and $W(\\cdot)$ from $\\widetilde{\\Omega}$ to the Grassmannian manifold are measurable. \n\t\t\t\\item[(iv)] It holds that\n\t\t\t\\begin{equation*}\n\t\t\t\tW_i(\\theta\\omega)=A(\\omega)W_i(\\omega).\n\t\t\t\\end{equation*}\n\t\t\t\\item[(v)] When $(\\Omega,\\mathcal{F},\\bP,\\theta)$ is ergodic, i.e., every $B\\in\\mathcal{F}$ with $\\theta^{-1}B=B$ satisfies $\\bP(B)=0$ or $\\bP(B)=1$, the functions  \n\t\t\t$p(\\cdot)$, $\\lambda_i(\\cdot)$, and $d_i(\\cdot)$, $i=1,2,\\dots,p(\\cdot)$, are constant on $\\widetilde{\\Omega}$.\n\t\t\\end{itemize}\n\t\\end{theorem}","sha256_utf8":"01c47d7709ebaa95163798e274b82b72da8754d4f15f278b3212e99635a9c492"},{"start":22767,"end":23783,"start_line":239,"end_line":240,"raw_tex":"\t\t\\item{\\emph{Probability space}.} For each $t \\in \\bN$, denote  $(\\Omega_t,\\Sigma_t,\\bP_t)$ the usual probability space for the distribution $\\mathcal{U}_{\\{1,2,\\dots,n\\}}\\times \\mathcal{U}_{[\\alpha_{\\min},\\alpha_{\\max}]}$, where $\\mathcal{U}_{\\{1,2,\\dots,n\\}}$ and $\\mathcal{U}_{[\\alpha_{\\min},\\alpha_{\\max}]}$ are the uniform distributions on the set $\\{1,2,\\dots,n\\}$ and interval $[\\alpha_{\\min},\\alpha_{\\max}]$ respectively. \n\t\tLet $(\\Omega,\\mathcal{F},\\bP)$ be the product probability space of all $(\\Omega_t,\\Sigma_t,\\bP_t),\\ t\\in\\bN$. Denote $\\pi_t$ as the projection from $(\\Omega,\\mathcal{F},\\bP)$ onto $(\\Omega_t,\\Sigma_t,\\bP_t), \\ t \\in \\bN$. Thus, a sample $\\omega\\in\\Omega$ can be represented as a sequence  $((i_0,\\alpha_0),(i_1,\\alpha_1),\\dots)$, where $(i_t,\\alpha_t)=\\pi_t(\\omega),\\ t\\in\\bN$. Let $\\{\\mathcal{F}_t\\}_{t\\in \\bN}$ be the filtration defined by $$\\mathcal{F}_t=\\sigma\\left\\{(B_0\\times\\cdots\\times B_t)\\times \\left(\\prod_{j> t}\\Omega_j\\right):B_i\\in \\Sigma_i,\\ i=0,1,\\dots,t\\right\\}.$$\n","sha256_utf8":"55c6a8feaf27783989e42df40cf710532b1f33c9d0a6dda31eeef0564afef879"},{"start":23786,"end":24227,"start_line":242,"end_line":247,"raw_tex":"\t\t\\item{\\emph{Metric dynamical system}.}\n\t\tThe metric dynamical system on $\\Omega$ is constructed by the (left) shifting operator $\\tau:\\Omega\\rightarrow\\Omega$ defined as \n\t\t\\begin{equation*}\n\t\t\t\\tau(\\omega)=\\tau(\\pi_0(\\omega),\\pi_1(\\omega),\\cdots) := (\\pi_1(\\omega),\\pi_2(\\omega),\\cdots),\n\t\t\\end{equation*}\n\t\twhich is clearly measurable and $\\bP$-preserving. The metric dynamical system is then given by $\\theta(t)=\\tau^t$ for $t\\in\\bN$. \n","sha256_utf8":"932e2558f24d5d577131b609f0216d20ed2ce06b6e2a93993974b3111f573536"},{"start":24230,"end":25594,"start_line":249,"end_line":266,"raw_tex":"\t\t\\item{\\emph{Random dynamical system}.}\n\t\tFor any $\\omega\\in\\Omega$ and $t\\in\\bN$, we define $\\phi(\\omega)$ to be a (nonlinear) map on $\\bR^d$ as \n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\phi(\\omega):\\bR^d&\\rightarrow\\quad\\quad\\quad \\bR^d\\\\\n\t\t\t\tx\\ &\\mapsto x-\\alpha e_i e_i^{\\top} \\nabla f(x),\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\twhere $(i,\\alpha)=\\pi_0(\\omega)$ is the first pair/element in the sequence $\\omega$, and we define the map $\\varphi(t,\\omega)$ via \n\t\t\\begin{equation*}\n\t\t\t\\varphi(t,\\omega)=\\phi(\\tau^{t-1}\\omega)\\circ\\cdots \\circ \\phi(\\tau\\omega)\\circ \\phi(\\omega), \\qquad \\text{for } t \\geq 1,\n\t\t\\end{equation*}\n\t\twhile $\\varphi(0, \\omega)$ is the identity operator. It is clear that $\\varphi(t,\\omega)$ satisfies the cocycle property \\eqref{cocycle} and hence defines a random dynamical system on $X=\\bR^d$ over $\\{\\tau^t\\}_{t\\in\\bN}$.\n\t\tThe iterate of Algorithm~\\ref{alg:random CGD} follows the random dynamical system as\n\t\t\\begin{equation*}\n\t\t\tx_t=\\phi(\\tau^{t-1}\\omega)x_{t-1}=\\cdots=\\phi(\\tau^{t-1}\\omega)\\circ\\cdots \\circ \\phi(\\tau\\omega)\\circ \\phi(\\omega)x_0=\\varphi(t,\\omega)x_0.\n\t\t\\end{equation*}\n\t\tIt can be seen that $\\{x_t\\}_{t\\in\\bN}$ is $\\{\\mathcal{F}_t\\}$-predictable, i.e., $x_t$ is $\\mathcal{F}_{t-1}$-measurable for any $t\\in\\bNp$, since $x_t$ is determined by samples $(i_0,\\alpha_0),(i_1,\\alpha_1),\\dots,(i_{t-1},\\alpha_{t-1})$.\n","sha256_utf8":"8fba8e88a94bd4d2371b22f80d938f28cf34c4e2ee41b6c5f4146c24bc42ec6a"},{"start":25626,"end":28314,"start_line":271,"end_line":301,"raw_tex":"\tIn our analysis, we will use linearization of the dynamical system $\\varphi(t,\\omega)$ at a critical point $x^*$ of $f$. Without loss of generality, we assume $x^*=0$; otherwise we consider the system with state being $x-x^*$. The resulting linear system, which depends on $H=\\nabla^2 f(x^*)=(H_{ij})_{1\\leq i,j\\leq d}$, is given by (here and in the sequel, we use the superscript $H$ to indicate dependence on the matrix)\n\t\\begin{equation}\\label{linear: Phi}\n\t\t\\Phi^H(t,\\omega)=A^H(\\tau^{t-1}\\omega)\\cdots A^H(\\tau\\omega)A^H(\\omega),\n\t\\end{equation}\n\twhere \n\t\\begin{equation}\\label{linear: A}\n\t\tA^H(\\omega)=I-\\alpha e_i e_i^{\\top} H,\\quad (i,\\alpha)=\\pi_0(\\omega). \n\t\\end{equation}\n\tNote that $A^H(\\cdot)$ is bounded in $\\Omega$. \n\tWe know that $\\left(\\log\\norm{A^H(\\cdot)}\\right)_+$ is integrable. When $\\alpha< 1/|H_{ii}|$, \n\tthe matrix $A^H(\\omega)=I-\\alpha e_i e_i^{\\top} H$ is invertable, and the inverse is given explicitly by applying the Sherman-Morrison formula:\n\t\\begin{equation}\\label{inverse-AH}\n\t\t\\begin{split}\n\t\t\tA^H(\\omega)^{-1}&=\\left(I-\\alpha e_i e_i^{\\top} H\\right)^{-1} = I + \\frac{\\alpha e_i e_i^{\\top} H}{1 - \\alpha H_{ii}}. \n\t\t\\end{split}\n\t\\end{equation}\n\tIn particular, we have \n\t\\begin{equation}\n\t\t\\norm{A^H(\\omega)^{-1}} \\leq 1 + \\frac{\\alpha \\norm{H}}{1 - \\alpha |H_{ii}|}. \n\t\\end{equation}\n\tThus, if we take the maximal stepsize $\\alpha_{\\max}$ such that $\\alpha_{\\max} < 1/\\max_{1\\leq i\\leq d}|H_{ii}|$,  $\\norm{A^H(\\cdot)^{-1}}$ is bounded in $\\Omega$, and as a result $\\left(\\log\\norm{A^H(\\cdot)^{-1}}\\right)_+$ is also integrable. \n\tTherefore, the assumptions of Theorem~\\ref{thm: MET} hold. The shifting operator $\\tau$ is ergodic on $(\\Omega,\\mathcal{F},\\bP)$ by Kolmogorov's $0$--$1$ law. Then Theorem~\\ref{thm: MET} applies for $\\theta=\\tau$ with $p^H(\\cdot)$, $\\lambda_i^H(\\cdot)$, and $d_i^H(\\cdot)$ all being a.e. constant. For any $\\omega\\in\\widetilde{\\Omega}$ that is the set in Theorem~\\ref{thm: MET} satisfying $\\bP(\\widetilde{\\Omega})=1$, \n\twe denote\n\t\\begin{equation}\\label{W_pm}\n\t\tW_+^H(\\omega)=\\bigoplus_{\\lambda_i>0}V_i^H(\\omega),\\ \\text{and}\\ W_-^H(\\omega)=\\bigoplus_{\\lambda_i\\leq 0}V_i^H(\\omega).\n\t\\end{equation}\n\tThen the following invariant property holds \n\t\\begin{equation*}\n\t\tW_-^H(\\tau\\omega)=A^H(\\omega)W_-^H(\\omega).\n\t\\end{equation*}\n\tNote that $W_-^H(\\omega)$ works as a center-stable subspace. That is, for any $x\\in W_-^H(\\omega)$ and any $\\epsilon>0$, it holds that $\\norm{\\Phi(t,\\omega)x}\\leq e^{t\\epsilon}$, for sufficiently large $t$, and for $x\\notin W_-^H(\\omega)$, $\\norm{\\Phi(t,\\omega)x}$ grows exponentially as $t\\rightarrow\\infty$ with rate greater than $\\min_{\\lambda_i>0}\\lambda_i -\\epsilon$ for any $\\epsilon>0$.\n","sha256_utf8":"2581da52440ee8527c309e01fc9c3d531bc634026ef8c33ddad13f375c54fa42"},{"start":28494,"end":28709,"start_line":308,"end_line":309,"raw_tex":"\\begin{assumption}\\label{Assump: Hess_bdd} $f \\in C^2(\\bR^d)$ and the Hessian $\\nabla^2 f$ is uniformly bounded, i.e., there exists $M>0$ such that $\\norm{\\nabla^2 f(x)}\\leq M$ for all $x\\in\\bR^d$.\n\t\\end{assumption}","sha256_utf8":"1e4caacea1942e28a9caadca1a73f886dd6a63937142be5dcca9121116d3d824"},{"start":28712,"end":29253,"start_line":311,"end_line":316,"raw_tex":"\tAn optimization algorithm is expected to converge, under some reasonable assumptions, to a critical point of $f$ where the gradient vanishes. Our aim is to further characterize the possible limits of the algorithm iterates. For this purpose, we distinguish $\\calA$, the set of all strict saddle points (including local maxima with non-degenerate Hessian) of $f$:\n\t\\begin{equation*}\n\t\t\\calA := \\{x\\in\\bR^d:\\nabla f(x)=0,\\\n\t\t\\lambda_{\\min}(\\nabla^2 f(x))<0\\},\n\t\\end{equation*}\n\twhere we use the subscript $s$ to emphasize that it is strict. \n","sha256_utf8":"e963ad05f10574dbb524f82b225b19d58bfcc62fe5046e3b2ccd4f37a5a21c5b"},{"start":30349,"end":30600,"start_line":327,"end_line":329,"raw_tex":"\\begin{assumption}\\label{Assump: non-degenerate}\n\t\tFor every $x^*\\in \\calA$, $\\nabla^2 f(x^*)$ is non-degenerate, i.e., $x^*$ is a non-degenerate critical point of $f$ in the sense that any eigenvalue of $\\nabla^2 f(x^*)$ is nonzero.\n\t\\end{assumption}","sha256_utf8":"935e17776ddfaebe2a87b22af6125fba1c2ffb97913f35abbfeaffec8716ce83"},{"start":31038,"end":31358,"start_line":331,"end_line":333,"raw_tex":"\\begin{assumption}\\label{Assump: non-zero-proj}\n\t\tFor every $x^*\\in \\calA$, it holds that $\\mathcal{P}_+^H(\\omega) e_i\\neq 0$, for every $i\\in\\{1,2,\\dots,d\\}$ and almost every $\\omega\\in\\Omega$, where $\\mathcal{P}_+^H(\\omega)$ is the orthogonal projection onto $W_+^H(\\omega)$ with $H=\\nabla^2 f(x^*)$.\n\t\\end{assumption}","sha256_utf8":"fe479be9d5b12ec318996b1de61fa8135d720e4c2af914471071406468c27f51"},{"start":32330,"end":32968,"start_line":341,"end_line":349,"raw_tex":"\tGiven an initial guess $x_0$, the behavior of the algorithm, in particular, the limit of $x_t$, depends on the particular sample $\\omega \\in \\Omega$.\n\tFor any $x^*\\in\\calA$, we denote the set of all $\\omega$ such that the algorithm starting at $x_0$ would converge to $x^*$:\n\t\\begin{equation*}\n\t\t\\calC(x^*,x_0) :=\\left\\{\\omega\\in\\Omega:\\lim_{t\\rightarrow\\infty}x_t=\\lim_{t\\rightarrow\\infty}\\varphi(t,\\omega)x_0=x^*\\right\\}.\n\t\\end{equation*}\n\tWe further define the set $\\calC(\\calA,x_0)$ as the union of all $\\calC(x^*,x_0)$ over $x^*\\in\\calA$,\n\t\\begin{equation*}\n\t\t\\calC(\\calA,x_0):=\\bigcup_{x^*\\in\\calA}\\calC(x^*,x_0).\n\t\\end{equation*}\n","sha256_utf8":"5345a4bff28f3052e5e1e8de7506928fc3cc10c35aad92d014c5a800d7ca9e0b"},{"start":38129,"end":38235,"start_line":394,"end_line":394,"raw_tex":"\tWe will first study the linear dynamical system, for which we assume the objective function is given by \n","sha256_utf8":"3f96632409c50293478ebaa88f0a203beda6e2d2b40fe8aef244e4fda4283fd0"},{"start":38236,"end":38324,"start_line":395,"end_line":397,"raw_tex":"\\begin{equation}\\label{obj: quadratic}\n\t\tf^H(x)=\\frac{1}{2}x^{\\top} H x,\n\t\\end{equation}","sha256_utf8":"8244bc2c4ab1b9fad06cc2f6a6f53912c77455b4f5f91cc1a2330a82fc7f0edf"},{"start":38325,"end":38752,"start_line":398,"end_line":402,"raw_tex":"\twhere $H$ is a symmetric matrix in $\\bR^{d\\times d}$ with at least one negative eigenvalue. In this case, the coordinate descent algorithm is given by \n\t\\begin{equation*}\n\t\tx_{t+1}=\\left(I-\\alpha_t e_{i_t} e_{i_t}^{\\top} H\\right)x_t,\n\t\\end{equation*}\n\twhich corresponds to the linear dynamical system  $\\Phi^H(t,\\omega)$ with single step map $A^H(\\omega)$,  defined in \\eqref{linear: Phi} and \\eqref{linear: A} respectively. \n","sha256_utf8":"ad36ffb708d68ea24d1228610c6608b1b29137b4bebf16a4ad609fe6975afecb"},{"start":47758,"end":48188,"start_line":535,"end_line":535,"raw_tex":"\tIn this subsection, we study the behavior of the nonlinear dynamical system near a strict saddle point of $f$, which without loss of generality can be assumed to be $x^*=0$. As mentioned above, in a small neighborhood of $x^*$, while it is not possible to control the difference between nonlinear and linear systems for infinite time,  the nonlinear system can be approximated by the linear system during a finite time horizon. \n","sha256_utf8":"3b7933522103958cd601901c15d3e39a7a39f40ff9443a36b04ca6a24c20bf34"},{"start":51129,"end":52017,"start_line":557,"end_line":566,"raw_tex":"\tWe first set the small constant $\\epsilon$ in Theorem~\\ref{thm: local-growth} which controls the failure probability of the amplification bound. Let $\\lambda_1>\\lambda_2>\\cdots>\\lambda_p$ be the Lyapunov exponents of the linearized system at $x^*=0$. We set\n\t\\begin{equation}\\label{lambplus-gamma}\n\t\t\\lambda_+=\\min_{\\lambda_i>0}\\lambda_i\\quad\\text{and}\\quad 0<\\gamma<\\frac{1}{2}\\min\\left\\{\\min_{1\\leq i< p}|\\lambda_i-\\lambda_{i+1}|,\\lambda_+\\right\\}. \n\t\\end{equation}\n\tNote that $\\gamma<\\lambda_+$.\n\tLet $\\epsilon_*\\in(0,1/6)$ be sufficiently small such that\n\t\\begin{equation}\\label{def: eps}\n\t\t(1-6\\epsilon)(\\lambda_+-\\gamma)+6\\epsilon \\cdot \\log(1-M\\alpha_{\\max})>0,\\quad \\forall\\ \\epsilon\\in(0,\\epsilon_*). \n\t\\end{equation}\n\tThe reason for such choice will become clear later (see \\eqref{large S2}). For the rest of the section, we will consider a fixed $\\epsilon\\in(0,\\epsilon_*)$. \n","sha256_utf8":"8eba350a0a946124c0c923e2ee624368863b7568ffdf0dae69409eaefc7fcd78"},{"start":53666,"end":54879,"start_line":595,"end_line":604,"raw_tex":"\tWe now carry out the amplification part of the finite block analysis \n\tfor the linearized dynamics at $x^* = 0$. To simplify expressions in the following, for $t_1<t_2$, we introduce the short-hand notation\n\t\\begin{equation*}\n\t\t(i,\\alpha)_{t_1:t_2-1}=\\big((i_{t_1},\\alpha_{t_1}),\\dots,(i_{t_2-1},\\alpha_{t_2-1})\\big)\\in\\Omega_{t_1}\\times\\cdots\\times\\Omega_{t_2-1},\n\t\\end{equation*}\n\tand the finite time transition matrix (i.e., composition of linear maps)\n\t\\begin{equation*}\n\t\t\\Phi^H\\bigl((i,\\alpha)_{t_1:t_2-1}\\bigr)=(I-\\alpha_{t_2-1}e_{i_{t_2-1}}e_{i_{t_2-1}}^{\\top} H)\\cdots (I-\\alpha_{t_1} e_{i_{t_1}}e_{i_{t_1}}^{\\top} H).\n\t\\end{equation*}\n\tRecall that $(\\Omega_t,\\Sigma_t,\\bP_t)$ is the probability space for $\\mathcal{U}_{\\{1,2,\\dots,d\\}}\\times \\mathcal{U}_{[\\alpha_{\\min},\\alpha_{\\max}]}$ for $t\\in\\bN$. We also denote $\\mathcal{P}_+^H\\bigl((i,\\alpha)_{t_1:t_2-1}\\bigr)$ as the projection operator onto the subspace spanned by the right singular vectors of $\\Phi^H\\bigl((i,\\alpha)_{t_1:t_2-1}\\bigr)$ corresponding to $d_+$ largest singular values, where $d_+=\\sum_{\\lambda_i>0}d_i$ and $d_i$ is the dimension of the $i$-th eigenspace as in Theorem~\\ref{thm: MET} (ii) for the linearized system at $x^*$.\n","sha256_utf8":"b5a6419f5916712b87a16246a376763c4c7322b2423f1c93e2e3c82b7abaa26b"},{"start":57271,"end":59478,"start_line":645,"end_line":666,"raw_tex":"\tWe will then be able to handle $\\norm{\\mathcal{P}_+^H((i,\\alpha)_{0:S-1})e_j}$ using Lemma~\\ref{lem: Omega1} and the closeness between $\\left(\\Phi^H(S,\\omega)^{\\top} \\Phi^H(S,\\omega)\\right)^{1/2S}$ with $\\Lambda(\\omega)$ as the former converges to the latter as $S \\to \\infty$ by Theorem~\\ref{thm: MET}. \n\tMore precisely, denote the singular values of $X\\in\\bR^{d\\times d}$ by $s_1(X)\\geq s_2(X)\\geq \\cdots\\geq s_d(X)$. Then for $S\\in\\bNp$ sufficiently large,\n\twe have \n\t\\begin{equation}\\label{close: singular value}\n\t\t\\left|\\frac{1}{S}\\log s_j\\left(\\Phi^H(S,\\omega)\\right)- \\lambda_{\\mu(j)}\\right|=\\left|\\frac{1}{S}\\log s_j\\left(\\Phi^H\\bigl((i,\\alpha)_{0:S-1}\\bigr)\\right)- \\lambda_{\\mu(j)}\\right|\\leq \\gamma,\n\t\\end{equation}\n\tfor every $j\\in\\{1,2,\\dots,d\\}$, where $\\lambda_1>\\lambda_2>\\dots>\\lambda_p$ are the Lyapunov exponents from Theorem~\\ref{thm: MET}, $\\gamma$ is given by \\eqref{lambplus-gamma}, and the map $\\mu:\\{1,2,\\dots,d\\}\\rightarrow\\{1,2,\\dots,p\\}$ satisfies that $\\mu(j)=i$ if and only if $d_1+\\cdots +d_{i-1}<j\\leq d_1+\\cdots +d_i$, so $\\mu$ corresponds the index for the singular values with that of the Lyapunov exponents. \n\tMoreover,\n\tthe convergence also implies that \n\t\\begin{equation*}\n\t\t\\norm{\\mathcal{P}_+^H(S,\\omega)-\\mathcal{P}_+^H(\\omega)}\\leq\\frac{\\delta}{2},\n\t\\end{equation*}\n\tfor sufficiently large $S$, which then leads to\n\t\\begin{equation}\\label{close: singular space}\n\t\t\\norm{\\mathcal{P}^H_+\\left(S,\\omega\\right)e_j}=\\norm{\\mathcal{P}^H_+\\left((i,\\alpha)_{0:S-1}\\right)e_j}\\geq \\frac{\\delta}{2},\n\t\\end{equation}\n\tfor every $j\\in\\{1,2,\\dots,d\\}$,\n\twhere $\\mathcal{P}^H_+(S,\\omega)$ is the projection operator onto the subspace spanned by the right singular vectors of $\\Phi^H(S,\\omega)$ corresponding to $d_+$ largest singular values. Let \n\t\\begin{equation}\n\t\t\\Omega^S=\\bigl\\{(i,\\alpha)_{0:S-1}\\in \\Omega_0\\times\\cdots\\times\\Omega_{S-1}:\\eqref{close: singular value}\\text{ and }\\eqref{close: singular space}\\text{ hold}\\bigr\\}.\n\t\\end{equation}\n\tThe following lemma states that $\\Omega^S$ has high probability for sufficiently large $S$, where with slight abuse of notation, we write $\\bP(\\Omega^S)=\\bP\\left(\\Omega^S\\times\\left(\\bigtimes_{t\\geq S}\\Omega_t\\right)\\right)$.\n","sha256_utf8":"f4e1192d049dae11014692daf5c1ff553c912478e080d4068247bce53c8477f6"},{"start":61773,"end":62240,"start_line":707,"end_line":708,"raw_tex":"\tNote that $\\alpha_{\\varrho_t-1}\\sim\\mathcal{U}_{[\\alpha_{\\min},\\alpha_{\\max}]}$ is independent of $\\mathcal{F}_{\\varrho_t-2}$, $i_{\\varrho_t-1}$, and $(i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}$. \n\tThe next lemma shows that with high probability, the choice of $\\alpha_{\\varrho_t-1}$ will lead to a nontrivial orthogonal projection of $x_{\\varrho_t}$ onto the unstable subspace of $\\Phi^H(S,\\tau^{\\varrho_t}\\omega)=\\Phi^H\\bigl((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\bigr)$.\n","sha256_utf8":"5fea48a65bdbf19a124c6efd8388daee059d4838da32f567db5460921d90d89f"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{maintheorem}\\label{thm: main}\n\t\tSuppose that Assumptions~\\ref{Assump: Hess_bdd}, \\ref{Assump: non-degenerate}, and \\ref{Assump: non-zero-proj} hold and that $0 < \\alpha_{\\min} < \\alpha_{\\max} < 1/M$, then  \n\t\tfor any $x_0\\in \\bR^d\\backslash\\calA$, it holds that\n\t\t\\begin{equation*}\n\t\t\t\\bP(\\calC(\\calA,x_0))=0.\n\t\t\\end{equation*}\n\t\\end{maintheorem}","units":[{"start":33342,"end":33695,"start_line":351,"end_line":357,"raw_tex":"\\begin{maintheorem}\\label{thm: main}\n\t\tSuppose that Assumptions~\\ref{Assump: Hess_bdd}, \\ref{Assump: non-degenerate}, and \\ref{Assump: non-zero-proj} hold and that $0 < \\alpha_{\\min} < \\alpha_{\\max} < 1/M$, then  \n\t\tfor any $x_0\\in \\bR^d\\backslash\\calA$, it holds that\n\t\t\\begin{equation*}\n\t\t\t\\bP(\\calC(\\calA,x_0))=0.\n\t\t\\end{equation*}\n\t\\end{maintheorem}","sha256_utf8":"17b6ad0cfc9c50927d80708e6b895748eb00370ff1e96ad1acde29638a16dee1"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{proposition}\\label{prop: posi-lamb1}\n\t\t%For any $x^*\\in\\calA$, if \n\t\tLet $H$ have a negative eigenvalue and\n\t\t$0<\\alpha_{\\min}<\\alpha_{\\max}<1/\\max_{1\\leq i\\leq d}|H_{ii}|$, then the largest Lyapunov exponent of $\\Phi^H(t,\\omega)$ is positive. \n\t\\end{proposition}\n\n\\begin{proposition}\\label{prop: linear_decay}\n\t\tLet $m\\geq d$ be fixed. For the objective function \\eqref{obj: quadratic} with $\\lambda_{\\min}(H)<0$, suppose \n\t\tthat $0<\\alpha_{\\min}<\\alpha_{\\max}<1/\\max_{1\\leq i\\leq d}|H_{ii}|$, there exists $c\\in(0,1)$ depending on $m$, $H$, $\\alpha_{\\min}$, and $\\alpha_{\\max}$, \n\t\tsuch that \n\t\t\\begin{equation*}\n\t\t\tf^H(x_{t+m})-f^H(x_t)\\leq c\\, f^H(x_t),\n\t\t\\end{equation*}\n\t\tholds as long as $f^H(x_t)<0$ and $\\{1,2,\\dots,d\\}=\\{i_t,i_{t+1},\\dots,i_{t+m-1}\\}$ (in the sense of sets). \n\t\\end{proposition}\n\n\\begin{proof} \n\t\tWithout loss of generality, we assume that $t=0$. \n\t\tDue to the choice of $\\alpha_{\\max}$, we have the following simple non-increasing property for any $t'\\in\\bN$:\n\t\t\\begin{equation}\\label{decay: quad}\n\t\t\t\\begin{split}\n\t\t\t\tf^H(x_{t'+1})&=\\frac{1}{2}x_{t'+1}^{\\top} H x_{t'+1}\\\\\n\t\t\t\t&=\\frac{1}{2}x_{t'}^{\\top} \\left(I-\\alpha_{t'} e_{i_{t'}} e_{i_{t'}}^{\\top} H\\right)^{\\top} H \\left(I-\\alpha_{t'} e_{i_{t'}} e_{i_{t'}}^{\\top} H\\right) x_{t'}\\\\\n\t\t\t\t&=f^H(x_{t'})-\\alpha_{t'} \\left(e_{i_{t'}}^{\\top} H x_{t'}\\right)^2+\\frac{1}{2}\\alpha_t^2 e_{i_{t'}}^{\\top} H e_{i_{t'}} \\left(e_{i_{t'}}^{\\top} H x_{t'}\\right)^2\\\\\n\t\t\t\t&\\leq f^H(x_{t'})-\\frac{\\alpha_{t'}}{2} \\left(e_{i_{t'}}^{\\top} H x_{t'}\\right)^2.\n\t\t\t\\end{split}\n\t\t\\end{equation}\n\t\t\n\t\tWrite $x_0=y^*+y_0$ with $y^*\\in\\ker(H)$ and $y_0\\in\\ran(H)$. Let \n\t\t\\begin{equation}\n\t\t\t\\label{def: y}\n\t\t\ty_{t'+1}=y_{t'}-\\alpha_{t'} e_{i_{t'}} e_{i_{t'}}^{\\top} H y_{t'}, \\qquad t'=0,1,\\dots,m-1.\n\t\t\\end{equation} \n\t\tThen $x_{t'}=y^*+y_{t'}$ holds for any $t'=0,1,\\dots,m$. Using \\eqref{decay: quad}, to give an upper bound for $f^H(x_{t+m})-f^H(x_t)$, we would like a nontrivial lower bound for $\\alpha_{t'}\\bigl(e_{i_{t'}}^{\\top} H x_{t'}\\bigr)^2=\\alpha_{t'}\\bigl(e_{i_{t'}}^{\\top} H y_{t'}\\bigr)^2$ for some $t'\\in\\{t,t+1,\\dots,t+m-1\\}$, which is guaranteed by Lemma~\\ref{lem: linear_decay}, whose proof will be postponed. \n\t\t\n\t\t\\begin{lemma} \\label{lem: linear_decay}\n\t\t\tSuppose that $\\{1,2,\\dots,d\\}=\\{i_0,i_1,\\dots,i_{m-1}\\}$. For any\n\t\t\t\\begin{equation}\\label{def:delta}\n\t\t\t\t0<\\delta\\leq\\min\\left\\{\\frac{1}{2m},\\frac{\\alpha_{\\min}\\sigma_{\\min}(H)}{2\\sqrt{m}(m\\alpha_{\\max}\\sigma_{\\max}(H)+1)}\\right\\},\n\t\t\t\\end{equation}\n\t\t\twhere $\\sigma_{\\min}(H)$ and $\\sigma_{\\max}(H)$ are the smallest and largest positive singular values of $H$ respectively, \n\t\t\tif $0\\neq y_0\\in \\ran(H)$, then there exists $T\\in \\{0,1,\\dots,m-1\\}$ such that $\\alpha_{T}\\left\\vert e_{i_{T}}^{\\top} H y_{T}\\right\\vert\\geq \\delta\\norm{y_{T}}$, where the sequence $y_t$ is given as in \\eqref{def: y}. \n\t\t\\end{lemma}\n\t\t\n\t\tAssuming the lemma, there exists $T\\in\\{0,1,\\dots,m-1\\}$ that \n\t\t\\begin{equation*}\n\t\t\t\\alpha_T\\left\\vert e_{i_{T}}^{\\top} H y_{T}\\right\\vert\\geq \\delta\\norm{y_{T}},\n\t\t\\end{equation*}\n\t\twith a fixed $\\delta>0$ satisfying \\eqref{def:delta}. We can further constrain that $\\frac{\\delta^2}{\\alpha_{\\min}\\sigma_{\\max}(H)}<1$. Thus, we have\n\t\t\\begin{equation*}\\label{decay esti}\n\t\t\t\\begin{split}\n\t\t\t\tf^H(x_m)&\\leq f^H (x_{T+1})\\leq f^H(x_T)- \\frac{\\alpha_T}{2}\\left(e_{i_{T}}^{\\top} H x_{T}\\right)^2\\leq f^H (x_{T}) -\\frac{\\delta^2}{2\\alpha_T}\\norm{y_{T}}^2,\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\twhich combined with $\\sigma_{\\max}(H)\\norm{y_{T}}^2\\geq -y_{T}^{\\top} H y_{T}=-x_{T}^{\\top} H x_{T}=-2f^H(x_T)$,\n\t\tyields that \n\t\t\\begin{equation*}\n\t\t\tf^H (x_m) \\leq\\left( 1 +\\frac{\\delta^2}{\\alpha_T \\sigma_{\\max}(H)}\\right)f^H (x_{T}) \\leq\\left( 1 +\\frac{\\delta^2}{\\alpha_T \\sigma_{\\max}(H)}\\right)f^H (x_0). \n\t\t\\end{equation*}\n\t\tSet $c=\\frac{\\delta^2}{\\alpha_{\\max} \\sigma_{\\max}(H)}$ and we get that $f^H(x_m)-f^H(x_0)\\leq c f^H(x_0)$.\n\t\\end{proof}\n\n\\begin{proof}[Proof of Lemma~\\ref{lem: linear_decay}] Suppose on the contrary that $\\alpha_t\\left\\vert e_{i_t}^{\\top} H y_t\\right\\vert< \\delta\\norm{y_t}$ for any $t\\in \\{0,1,\\dots,m-1\\}$. It holds that\n\t\t\\begin{equation*}\n\t\t\t\\norm{y_1-y_0}=\\alpha_0\\left\\vert e_{i_0}^{\\top} H y_0\\right\\vert<\\delta\\norm{y_0}<2\\delta\\norm{y_0}.\n\t\t\\end{equation*}\n\t\tWe claim that \n\t\t\\begin{equation} \\label{eq: disty}\n\t\t\t\\norm{y_t-y_0}<2 t\\delta\\norm{y_0},\n\t\t\\end{equation}\n\t\tfor any $t=1,2,\\dots,m-1$. By induction, assume that  $\\norm{y_t-y_0}<2t\\delta\\norm{y_0}$ holds for some $t\\in\\{1,2,\\dots,m-2\\}$, then \n\t\t\\begin{equation*}\n\t\t\t\\norm{y_{t+1}-y_t}=\\alpha_t\\left\\vert e_{i_t}^{\\top} H y_t\\right\\vert< \\delta\\norm{y_t}<\\delta(2t\\delta+1)\\norm{y_0}<2\\delta\\norm{y_0},\n\t\t\\end{equation*}\n\t\twhere the last inequality uses $2 t \\delta < 2m \\delta \\leq 1 $. It follows $\\norm{y_{t+1}-y_0}\\leq \\norm{y_t-y_0}+\\norm{y_{t+1}-y_t}<2(t+1)\\delta\\norm{y_0}$.\n\t\t\n\t\tUsing \\eqref{eq: disty} and $\\max_{1\\leq i\\leq d} \\norm{H e_i}\\leq \\sigma_{\\max}(H)$, we have \n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\alpha_t\\bigl\\lvert e_{i_t}^{\\top} H y_0\\bigr\\rvert&\\leq \\alpha_t\\bigl\\lvert e_{i_t}^{\\top} H (y_t-y_0)\\bigr\\rvert+\\alpha_t\\bigl\\lvert e_{i_t}^{\\top} H y_t\\bigr\\rvert\\\\\n\t\t\t\t&<\\alpha_{\\max}\\sigma_{\\max}(H)\\cdot 2 t\\delta\\norm{y_0}+2\\delta\\norm{y_0}\\\\\n\t\t\t\t&<2\\delta(m\\alpha_{\\max}\\sigma_{\\max}(H)+1)\\norm{y_0},\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tfor $t=0,1,\\dots,m-1$. Since $\\text{span}\\{e_{i_k}:k=0,1,\\dots,m-1\\}=\\bR^d$, noticing that $y_0\\in\\ran{H}$, we have\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\alpha_{\\min}\\sigma_{\\min}(H) \\norm{y_0}&\\leq \\alpha_{\\min}\\norm{H y_0}\\leq\\left(\\sum_{t=0}^{m-1} \\left(\\alpha_t\\bigl\\lvert e_{i_t}^{\\top} H y_0\\bigr\\rvert\\right)^2\\right)^{1/2}\\\\\n\t\t\t\t& <2\\delta\\sqrt{m}(m\\alpha_{\\max}\\sigma_{\\max}(H)+1)\\norm{y_0},\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\twhich contradicts with the choice of $\\delta$ in \\eqref{def:delta}.\n\t\\end{proof}\n\n\\begin{proof}[Proof of Proposition~\\ref{prop: posi-lamb1}] It suffices to show that for almost every $\\omega\\in\\Omega$ there exist some $x_0\\in \\bR^d$, $\\epsilon>0$, and $T>0$, such that $x_t=\\Phi(t,\\omega)x_0$ satisfies $\\norm{x_t}\\geq e^{\\epsilon t}$ for any $t>T$. Let $x_0$ be an eigenvector corresponding to a negative eigenvalue of $H$. Then it holds that $f^H(x_0)< 0$. Consider a fixed $m\\geq d$. For any $k\\in\\bN$, set $I_k=1$ if $\\{1,2,\\dots,d\\}=\\{i_{km},i_{km+1},\\dots,i_{km+m-1}\\}$ \n\t\tand $I_k=0$ otherwise. We can see that the random variables, $I_0,I_1,I_2,\\dots$, are independent and identically distributed with $\\bE I_0=\\bP(I_0=1)\\in(0,1)$. By Proposition~\\ref{prop: linear_decay}, we obtain that\n\t\t\\begin{equation*}\n\t\t\tf^H(x_{(k+1)m})\\leq\\begin{cases}\n\t\t\t\t(1+c)f^H(x_{km}),&\\text{if }I_k=1,\\\\\n\t\t\t\tf^H(x_{km}),&\\text{if }I_k=0,\n\t\t\t\\end{cases}\n\t\t\\end{equation*}\t\n\t\twhere $c$ is the constant from Proposition~\\ref{prop: linear_decay}.\n\t\tTherefore, %\\jl{missing $1/2$}\n\t\t\\begin{equation*}\n\t\t\t\\frac{\\lambda_{\\min}(H)}{2}\\norm{x_{km}}^2\\leq f^H(x_{km})\\leq(1+c)^{\\sum_{j=0}^{k-1} I_j}f^H(x_0),\n\t\t\\end{equation*}\n\t\twhich implies that\n\t\t\\begin{equation}\\label{norm:xkm}\n\t\t\t\\norm{x_{km}}\\geq \\left(\\frac{2f^H(x_0)}{\\lambda_{\\min}(H)}\\right)^{1/2}\\cdot (1+c)^{\\frac{1}{2}\\sum_{j=0}^{k-1} I_j}.\n\t\t\\end{equation}\n\t\tNote that $\\bE |I_0|=\\bE I_0<\\infty$. The strong law of large number suggests that for almost every $\\omega\\in\\Omega$, there exists some $K$, such that for all $k \\geq K$\n\t\t\\begin{equation}\\label{SLLN}\n\t\t\t\\sum_{j=0}^{k-1} I_j\\geq \\frac{\\bE I_0}{2}k.\n\t\t\\end{equation}\n\t\tCombining \\eqref{norm:xkm} and \\eqref{SLLN}, we arrive at\n\t\t\\begin{equation*}\n\t\t\t\\norm{x_{km}}\\geq \\left(\\frac{2f^H(x_0)}{\\lambda_{\\min}(H)}\\right)^{1/2}\\cdot (1+c)^{\\frac{\\bE I_0}{4m}\\cdot km}.\n\t\t\\end{equation*}\n\t\tNoticing that $(1+c)^{\\frac{\\bE I_0}{4m}}$ is greater than $1$, $\\norm{x_{km}}$ grows exponentially in $km$ and we complete the proof.\n\t\\end{proof}\n\n\\begin{theorem}\\label{thm: local-growth}\n\t\tSuppose that Assumptions~\\ref{Assump: Hess_bdd}, \\ref{Assump: non-degenerate}, and \\ref{Assump: non-zero-proj} hold and that $0<\\alpha_{\\min}<\\alpha_{\\max}<1/M$. There exists $\\epsilon_*\\in(0,1/6)$ such that for any $\\epsilon\\in(0,\\epsilon_*)$, we have $T_* = T_*(\\epsilon) \\in\\bNp$ that for any $T \\in \\bNp$ with $T\\geq T_*$ and any $t\\in\\bN$, conditioned on $\\mathcal{F}_{t-1}$, with probability at least $1-4\\epsilon$, it holds for all $x_t \\in V$ that\n\t\t\\begin{equation}\\label{local-growth}\n\t\t\t\\norm{x_{t+T}}\\geq \\exp\\biggl(\\frac{6\\epsilon}{1 - 6 \\epsilon} \\bigl\\lvert \\log(1-M\\alpha_{\\max}) \\bigr\\rvert \\, T\\biggr) \\norm{x_t},\n\t\t\\end{equation}\n\t\twhere $V$ is a neighborhood of $x^* = 0$, depending on $\\epsilon$, $T$, and $f$ near $x^*$. \n\t\\end{theorem}\n\n\\begin{lemma}\\label{lem: L}\n\t\tLet $0<\\mu\\leq\\frac{1}{\\sqrt{d}}$ be a fixed constant. \n\t\tThere exists some constant $L>0$, such that for any $t\\in\\bN$, there exists a measurable $\\varrho_t:\\Omega\\rightarrow \\bNp$ such that $t < \\varrho_t \\leq t + L$ and\n\t\t\\begin{equation}\\label{eq: rho_t}\n\t\t\t\\bP\\Bigl(\\bigl\\lvert e_{i_{\\varrho_t-1}}^{\\top} \\nabla f(x_{\\varrho_t-1})\\bigr\\rvert \\geq \\mu\\norm{\\nabla f(x_{\\varrho_t-1})}\\;\\Big\\vert\\; \\mathcal{F}_{t-1}\\Bigr)\\geq 1-\\epsilon.\n\t\t\\end{equation}\n\t\\end{lemma}\n\n\\begin{proof}\n\t\tFor any $t\\in\\bN$, use $\\ell_0$ to denote the smallest non-negative integer $\\ell$, such that\n\t\t\\begin{equation*}\n\t\t\t\\ell_0 = \\arg\\min_{\\ell} \\Bigl\\{ \\ell \\in \\mathbb{N}_+ \\,:\\,\t\\bigl\\lvert e_{i_{t+\\ell-1}}^{\\top} \\nabla f(x_{t+\\ell-1})\\bigr\\rvert\\geq \\mu\\norm{\\nabla f(x_{t+\\ell-1})} \\Bigr\\}.\n\t\t\\end{equation*}\n\t\tIt is clear that $\\bP(\\ell_0 >\\ell\\mid\\mathcal{F}_{t-1})\\leq (1-1/d)^\\ell$, since for each step the coordinate is randomly chosen. Hence, there exists some $L>0$, such that \n\t\t\\begin{equation*}\n\t\t\t\\bP(\\ell_0 \\leq L\\mid\\mathcal{F}_{t-1})\\geq 1-\\epsilon.\n\t\t\\end{equation*}\n\t\tWe finish the proof by setting $\\varrho_t = t + \\min\\{ \\ell_0, L \\}$\n\t\twhich has the desired property. \n\t\\end{proof}\n\n\\begin{lemma}\\label{lem: Omega1}\n\t\tUnder Assumption~\\ref{Assump: non-zero-proj}, \n\t\tthere exist $\\delta>0$ and measurable $\\Omega_1^\\epsilon\\subset\\widetilde{\\Omega}$, where $\\widetilde{\\Omega}$ is from Theorem~\\ref{thm: MET}, \n\t\tsuch that $\\bP(\\Omega_1^\\epsilon)\\geq 1-\\epsilon$ \n\t\tand \n\t\t\\begin{equation*}\n\t\t\t\\norm{\\mathcal{P}^H_+(\\omega) e_i}\\geq\\delta,\\quad \\forall\\ \\omega\\in\\Omega_1^\\epsilon,\\ i\\in\\{1,2,\\dots,d\\}.\n\t\t\\end{equation*}\n\t\\end{lemma}\n\n\\begin{proof}\n\t\tAssumption~\\ref{Assump: non-zero-proj} implies that \n\t\t\\begin{equation*}\n\t\t\t\\bP\\bigl(\\{\\omega\\in\\widetilde{\\Omega}:\\norm{\\mathcal{P}^H_+(\\omega) e_i}>0,\\quad \\forall\\ i\\in\\{1,2,\\dots,d\\}\\}\\bigr) = 1\n\t\t\\end{equation*}\n\t\tNotice that \n\t\t\\begin{multline*}\n\t\t\t\\bigl\\{\\omega\\in\\widetilde{\\Omega}:\\norm{\\mathcal{P}^H_+(\\omega) e_i}>0,\\quad \\forall\\ i\\in\\{1,2,\\dots,d\\}\\bigr\\}\n\t\t\t= \\\\ \n\t\t\t=\\bigcup_{n\\in\\bNp}\\Bigl\\{\\omega\\in\\widetilde{\\Omega}:\\norm{\\mathcal{P}^H_+(\\omega) e_i}\\geq \\frac{1}{n},\\quad \\forall\\ i\\in\\{1,2,\\dots,d\\}\\Bigr\\}.\n\t\t\\end{multline*}\n\t\tThe Lemma follows from continuity of measure. \n\t\\end{proof}\n\n\\begin{lemma}\\label{lem: T-Omega2}\n\t\tUnder the same assumptions of Lemma~\\ref{lem: Omega1}, there exists some $S_*>0$, such that for every $S \\in \\bNp, S \\geq S_*$, it holds $\\bP(\\Omega^S)\\geq 1-2\\epsilon$.\n\t\\end{lemma}\n\n\\begin{proof}\n\t\tFor $a.e.\\ \\omega\\in\\Omega$, it follows from Theorem~\\ref{thm: MET}, in particular \\eqref{conv: Lamb}, and standard matrix perturbation analysis (see e.g., \\cite[Theorem VI.2.1, Theorem VII.3.1]{bhatia}) that \n\t\t\\begin{equation}\\label{conv_lya_component}\n\t\t\t\\frac{1}{S}s_j(\\Phi^H(S,\\omega))\\rightarrow\\lambda_{\\mu(j)},\\quad S\\rightarrow\\infty.\n\t\t\\end{equation}\n\t\tfor any $j\\in\\{1,2,\\dots,d\\}$, and that\n\t\t\\begin{equation}\\label{conv_subspace}\n\t\t\t\\mathcal{P}^H_+(S,\\omega)\\rightarrow\\mathcal{P}^H_+(\\omega),\\quad S\\rightarrow\\infty.\n\t\t\\end{equation}\n\t\t\n\t\tBy Egorov's theorem, there exists $\\Omega_2^\\epsilon\\subset\\Omega_1^\\epsilon$ with $\\bP(\\Omega_2^\\epsilon) \\geq 1 - 2 \\epsilon$, such that the convergences in \\eqref{conv_lya_component} and \\eqref{conv_subspace} are both uniform on $\\Omega_2^\\epsilon$. Here $\\Omega_1^\\epsilon$ is as in Lemma~\\ref{lem: Omega1}. Therefore, for some $S_*$ sufficiently large, we have\n\t\t\\begin{equation*}\\label{uniconv: singval}\n\t\t\t\\left|\\frac{1}{S}\\log s_j\\left(\\Phi^H(S,\\omega)\\right)- \\lambda_{\\mu(j)}\\right|\\leq \\gamma,\\quad\\forall\\ j\\in\\{1,2,\\dots,d\\},\\ S\\geq S_*,\\ \\omega\\in\\Omega_2^\\epsilon,\n\t\t\\end{equation*}\n\t\tand\n\t\t\\begin{equation}\\label{uniconv: singspace}\n\t\t\t\\norm{\\mathcal{P}^H_+(S,\\omega)-\\mathcal{P}^H_+(\\omega)}\\leq \\frac{\\delta}{2},\\quad\\forall\\ S\\geq S_*,\\ \\omega\\in\\Omega_2^\\epsilon.\n\t\t\\end{equation}\n\t\tCombining Lemma~\\ref{lem: Omega1} and \\eqref{uniconv: singspace}, we obtain that\n\t\t\\begin{equation*}\n\t\t\t\\norm{\\mathcal{P}^H_+(S,\\omega)e_i}\\geq \\frac{\\delta}{2},\\quad \\forall\\ i\\in\\{1,2,\\dots,d\\},\\ \\forall\\ S\\geq S_*,\\ \\omega\\in\\Omega_2^\\epsilon.\n\t\t\\end{equation*}\n\t\t\n\t\tFor any $S\\geq S_*$, \n\t\tby the definition of $\\Omega^S$, it holds that \n\t\t\\begin{equation*}\n\t\t\t\\Omega_2^\\epsilon\\subset\\Omega^S\\times\\Bigl(\\bigtimes_{t\\geq S}\\Omega_t\\Bigr),\n\t\t\\end{equation*}\n\t\twhich implies the desired estimate \n\t\t\\begin{equation*}\n\t\t\t\\bP(\\Omega^S)=\\bP\\biggl(\\Omega^S\\times\\Bigl(\\bigtimes_{t\\geq S}\\Omega_t\\Bigr)\\biggr)\\geq\\bP(\\Omega_2^\\epsilon)\\geq 1-2\\epsilon. %\\qedhere\n\t\t\\end{equation*}\n\t\\end{proof}\n\n\\begin{lemma}\\label{lem: nontrial-proj}\n\t\tFor any $S\\in\\bNp$, $x_{\\varrho_t-1}$, $i_{\\varrho_t-1}$, \n\t\tand $(i,\\alpha)_{\\varrho_t:\\varrho_t+S-1} \\in\\Omega^S$, there exists $I\\subset[\\alpha_{\\min},\\alpha_{\\max}]$ with $m(I)\\geq (1-\\epsilon)(\\alpha_{\\max}-\\alpha_{\\min})$ where $m(\\cdot)$ is the Lebesgue measure,  such that for any $\\alpha_{\\varrho_t-1}\\in I$, it holds that \n\t\t\\begin{equation}\\label{eq: nontrial-proj}\n\t\t\t\\norm{\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\right)x_{\\varrho_t}}\\geq \\frac{\\epsilon\\delta (\\alpha_{\\max}-\\alpha_{\\min})}{4}\\bigl\\lvert e_{i_{\\varrho_t-1}}^{\\top} \\nabla f(x_{\\varrho_t-1})\\bigr\\rvert.\n\t\t\\end{equation}\n\t\\end{lemma}\n\n\\begin{proof}\n\t\tWe assume that $\\bigl\\lvert e_{i_{\\varrho_t-1}}^{\\top} \\nabla f(x_{\\varrho_t-1})\\bigr\\rvert\\neq 0$; otherwise the result is trivial. \n\t\t\n\t\tFor simplicity of notation, we write \n\t\t\\begin{equation*}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\right)x_{\\varrho_t} & = \n\t\t\t\t\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\right)x_{\\varrho_t-1} \\\\\n\t\t\t\t& \\qquad \n\t\t\t\t- \\alpha_{\\varrho_t - 1} e_{i_{\\varrho_t-1}}^{\\top}\\nabla f(x_{\\varrho_t-1})\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\right)e_{i_{\\varrho_t-1}} \\\\\n\t\t\t\t& =: y_2-\\alpha_{\\varrho_t-1} y_1,\n\t\t\t\\end{aligned}\n\t\t\\end{equation*}\n\t\twhere the last line defines $y_1$ and $y_2$. Using the short-hand notation \n\t\t\\begin{equation*}\n\t\t\tr=\\frac{\\epsilon\\delta (\\alpha_{\\max}-\\alpha_{\\min})}{4}\\bigl\\lvert e_{i_{\\varrho_t-1}}^{\\top} \\nabla f(x_{\\varrho_t-1})\\bigr\\rvert, \n\t\t\\end{equation*}\n\t\twe observe then \\eqref{eq: nontrial-proj} holds if and only if $\\alpha_{\\varrho_t-1} y_1$ is not located in a ball with radius $r$ centered at $y_2$.\n\t\t\n\t\tIt follows from the definition of $\\Omega^S$ and \\eqref{close: singular space} that $ \\norm{\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\right)e_{i_{\\varrho_t-1}}} \\geq \\frac{\\delta}{2}$, \n\t\twhich then leads to\n\t\t\\begin{equation*}\n\t\t\t\\norm{y_1}\\geq \\frac{\\delta}{2}\\bigl\\lvert e_{i_{\\varrho_t-1}}^{\\top} \\nabla f(x_{\\varrho_t-1})\\bigr\\rvert=:\\frac{2r}{\\epsilon(\\alpha_{\\max}-\\alpha_{\\min})}.\n\t\t\\end{equation*}\n\t\tThus, the set of $\\alpha_{\\varrho_t-1}$ such that $\\alpha_{\\varrho_t -1} y_1 \\in \\mathcal{B}_r(y_2)$ consists of an interval $J$ in $\\bR$ with $\\|\\sup(J)\\cdot y_1 - \\inf(J)\\cdot y_1\\|\\leq 2r$ as the diameter of $\\mathcal{B}_r(y_2)$ is $2r$, which implies $m(J)\\leq 2r/\\norm{y_1}\\leq \\epsilon(\\alpha_{\\max}-\\alpha_{\\min})$. The lemma is proved then by setting $I=[\\alpha_{\\max}-\\alpha_{\\min}]\\backslash J$.\n\t\\end{proof}\n\n\\begin{proof}[Proof of Theorem~\\ref{thm: local-growth}]\n\t\tWithout loss of generality, we will assume $t=0$ in the proof to simplify notation.\n\t\tSince $H=\\nabla^2 f(x^*)$ is non-degenerate, we can take a neighborhood \n\t\t$U$ of $x^*=0$ and some fixed $\\sigma > 0$ such that\n\t\t\\begin{equation}\\label{eq: sigma}\n\t\t\t\\norm{\\nabla f(x)}\\geq \\sigma\\norm{x},\\quad \\forall\\ x\\in U.    \n\t\t\\end{equation}\n\t\tAssumption~\\ref{Assump: Hess_bdd} implies that\n\t\t\\begin{equation*}\n\t\t\t\\norm{\\nabla f(x)}=\\norm{\\nabla f(x)-\\nabla f(x^*)}\\leq M\\norm{x-x^*}=M\\norm{x},\\quad\\forall\\ x\\in\\bR^d.\n\t\t\\end{equation*}\n\t\tUsing the above inequality and $\\alpha_{\\max}<1/M$, it holds for every $\\omega\\in\\Omega$ and $t'\\in\\bN$ that \n\t\t\\begin{equation}\\label{r-minus}\n\t\t\t\\begin{split}\n\t\t\t\t\\norm{x_{t'+1}}&=\\norm{x_{t'}-\\alpha_{t'} e_{i_{t'}}e_{i_{t'}}^{\\top}\\nabla f(x_{t'})} \\\\\n\t\t\t\t& \\geq \\norm{x_{t'}}-\\alpha_{t'}\\norm{e_{i_{t'}}e_{i_{t'}}^{\\top}}\\cdot \\norm{\\nabla f(x_{t'})}\\\\\n\t\t\t\t&\\geq (1-M\\alpha_{\\max})\\norm{x_{t'}},\n\t\t\t\\end{split}\n\t\t\\end{equation}\n\t\tand similarly that\n\t\t\\begin{equation*}\n\t\t\t\\norm{x_{t'+1}}\\leq(1+M\\alpha_{\\max})\\norm{x_{t'}}.\n\t\t\\end{equation*}\n\t\tWe thus define\n\t\t\\begin{equation*}\n\t\t\tr_-:=1-M\\alpha_{\\max}\\quad\\text{and}\\quad r_+:=1+M\\alpha_{\\max}, \n\t\t\\end{equation*}\n\t\tso that\n\t\t\\begin{equation}\\label{eq:compareiterate}\n\t\t\tr_- \\norm{x_{t'}} \\leq \\norm{x_{t'+1}} \\leq r_+ \\norm{x_{t'}}.\n\t\t\\end{equation}\n\t\t\n\t\tWe now choose the time duration $S$ large enough in the finite block analysis to guarantee significant amplification. More specifically, we choose $S$ so large that $S \\geq S_*$ ($S_*$ defined in Lemma~\\ref{lem: T-Omega2}) and that the following two inequalities hold: \n\t\t\\begin{equation}\\label{large S1}\n\t\t\t\\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta \\mu\\sigma (r_-)^{L-1}(\\alpha_{\\max}-\\alpha_{\\min})}{8}\\geq\n\t\t\t(r_+)^{L},\n\t\t\\end{equation}\n\t\tand\n\t\t\\begin{equation}\\label{large S2}\n\t\t\t(1-6\\epsilon)\\left(S(\\lambda_+-\\gamma)+\\log \\frac{\\epsilon\\delta \\mu\\sigma (r_-)^{2(L-1)}(\\alpha_{\\max}-\\alpha_{\\min})}{8}\\right)+6\\epsilon(L+S)\\log r_->0,\n\t\t\\end{equation}\n\t\twhere $L$ is the upper bound defined in Lemma~\\ref{lem: L}, $\\mu\\leq 1/\\sqrt{d}$ is a fixed constant as in Lemma~\\ref{lem: L}, $\\delta$ is from Lemma~\\ref{lem: Omega1}, and $\\sigma$ is set in \\eqref{eq: sigma}. \n\t\tThanks to \\eqref{def: eps} for our choice of $\\epsilon$ and that $\\gamma < \\lambda_+$ from \\eqref{lambplus-gamma}, \n\t\t\\eqref{large S1} and \\eqref{large S2} are satisfied for sufficiently large $S$. \n\t\t\n\t\tNext, we show that, for any $S$ sufficiently large as above, there exists a convex neighborhood $U_1\\subset U$ of $x^*=0$, such that for any $t'\\in\\bN$, any $x_{t'}\\in U_1$, and any $(i,\\alpha)_{t':t'+S-1}$, it holds that\n\t\t\\begin{equation}\\label{block-error}\n\t\t\t\\norm{x_{t'+S}} \\geq \\norm{\\Phi^H\\bigl((i,\\alpha)_{t':t'+S-1}\\bigr)x_{t'}}-\\norm{x_{t'}}.\n\t\t\\end{equation}\n\t\tWe first define a convex neighborhood $U_0 \\subset U$ of $x^*=0$ such that\n\t\t\\begin{multline*}\n\t\t\t\\norm{\\left(x-\\alpha e_i e_i^{\\top}\\nabla f(x)\\right)-\\left(I-\\alpha e_i e_i^{\\top}H\\right)x}=\\norm{\\alpha e_i e_i^{\\top}\\left(\\nabla f(x)-H x\\right)}\\\\\n\t\t\t=\\norm{\\alpha e_i e_i^{\\top}\\int_0^1 (\\nabla^2 f(\\eta x)-Hx) \\mathrm{d} \\eta}\\leq \\frac{1}{S(r_+)^{S-1}} \\norm{x},\n\t\t\\end{multline*}\n\t\tfor any $x \\in U_0$, any $i\\in\\{1,2,\\dots,d\\}$, and any $\\alpha\\in[\\alpha_{\\min},\\alpha_{\\max}]$.\n\t\tApplying the inequality $S$ times for $x_{t'}\\in U_1 = (r_+)^{-(S-1)}U_0$, we have,\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t&\\norm{x_{t'+S}-\\Phi^H\\bigl((i,\\alpha)_{t':t'+S-1}\\bigr)x_{t'}}\\\\\n\t\t\t\t\\leq & \\norm{x_{t'+S}-\\left(I-\\alpha_{t'+S-1} e_{i_{t'+S-1}}e_{i_{t'+S-1}}^{\\top}H\\right)x_{t'+S-1}}\\\\\n\t\t\t\t&+\\norm{I-\\alpha_{t'+S-1} e_{i_{t'+S-1}}e_{i_{t'+S-1}}^{\\top}H}\\cdot \\norm{x_{t'+S-1}-\\Phi^H\\bigl((i,\\alpha)_{t':t'+S-2}\\bigr)x_{t'}}\\\\\n\t\t\t\t\\leq & \\frac{1}{S(r_+)^{S-1}} \\norm{x_{t'+S-1}}+r_+\\norm{x_{t'+S-1}-\\Phi^H\\bigl((i,\\alpha)_{t':t'+S-2}\\bigr)x_{t'}}\\\\\n\t\t\t\t\\leq& \\frac{1}{S(r_+)^{S-1}} \\left(\\norm{x_{t'+S-1}}+r_+\\norm{x_{t'+S-2}}+\\cdots+(r_+)^{S-1}\\norm{x_{t'}}\\right)\\\\\n\t\t\t\t\\leq& \\norm{x_{t'}},\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tand hence, inequality \\eqref{block-error}.\n\t\t\n\t\tSetting $V=(r_+)^{-(L-1)} U_1$, we then have $x_{\\varrho_0-1}\\in U$ for any $x_0\\in V$, which implies that $\\norm{\\nabla f(x_{\\varrho_0-1})}\\geq\\sigma\\norm{x_{\\varrho_0-1}}$ as $\\varrho_0\\leq L$. \n\t\tAccording to Lemma~\\ref{lem: L}--\\ref{lem: nontrial-proj}, for any given $x_0\\in V$, with probability at least $1-4\\epsilon$, we have $(i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\in\\Omega^S$, and  the followings hold:\n\t\t\\begin{align}\n\t\t\t\\label{ineq1} & \\bigl\\lvert e_{i_{\\varrho_0-1}}^{\\top} \\nabla f(x_{\\varrho_0-1})\\bigr\\rvert\\geq \\mu\\norm{\\nabla f(x_{\\varrho_0-1})}, \\\\\n\t\t\t\\label{ineq2} & \\norm{\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)x_{\\varrho_0}}\\geq \\frac{\\epsilon\\delta (\\alpha_{\\max}-\\alpha_{\\min})}{4}\\bigl\\lvert e_{i_{\\varrho_0-1}}^{\\top} \\nabla f(x_{\\varrho_0-1})\\bigr\\rvert,\n\t\t\\end{align}\n\t\twhere the probability is the marginal probability on $(i,\\alpha)_{0:L+S-1}\\in\\Omega_0\\times\\cdots\\times \\Omega_{L+S-1}$.  \n\t\t\n\t\t\\smallskip \n\t\t\n\t\tRecall $\\lambda_+$ and $\\gamma$ in \\eqref{lambplus-gamma}, and $d_+=\\sum_{\\lambda_i>0}d_i$. \n\t\tIt follows from \\eqref{close: singular value} of the construction of the set $\\Omega^S$ that\n\t\t\\begin{equation*}\n\t\t\t\\frac{1}{S}\\log s_j\\left(\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)\\right)\\geq \\lambda_{\\mu(j)}-\\gamma\\geq \\lambda_+-\\gamma,\\quad\\forall\\ j\\leq d_+.\n\t\t\\end{equation*}\n\t\tThis is to say that the $d_+$ largest singular values of $\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)$\n\t\tare all greater than or equal to $\\exp(S(\\lambda_+-\\gamma))$. \n\t\tTherefore, it holds that \n\t\t\\begin{equation}\\label{linear-growth}\n\t\t\t\\begin{split}\n\t\t\t\t\\norm{\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)x_{\\varrho_0}}&\\geq \\norm{\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)x_{\\varrho_0}}\\\\\n\t\t\t\t&  \\leftstackrel{\\eqref{ineq2}}{\\geq} \\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta (\\alpha_{\\max}-\\alpha_{\\min})}{4}\\bigl\\lvert e_{i_{\\varrho_0-1}}^{\\top} \\nabla f(x_{\\varrho_0-1})\\bigr\\rvert\\\\\n\t\t\t\t&  \\leftstackrel{\\eqref{ineq1}}{\\geq}\n\t\t\t\t\\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta\\mu (\\alpha_{\\max}-\\alpha_{\\min})}{4}\\norm{\\nabla f(x_{\\varrho_0-1})},\n\t\t\t\\end{split}\n\t\t\\end{equation}\n\t\twhere the first inequality follows from the fact that $\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)x_{\\varrho_0}$ and $\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)\\left(I-\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)\\right)x_{\\varrho_0}$ are orthogonal. Combining \\eqref{block-error} and \\eqref{linear-growth}, we obtain that\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\norm{x_{\\varrho_0+S}} & \\geq \\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta \\mu(\\alpha_{\\max}-\\alpha_{\\min})}{4}\\norm{\\nabla f(x_{\\varrho_0-1})}-\\norm{x_{\\varrho_0}}\\\\\n\t\t\t\t& \\leftstackrel{\\eqref{eq:compareiterate}}{\\geq} \\left(\\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta \\mu\\sigma (r_-)^{L-1}(\\alpha_{\\max}-\\alpha_{\\min})}{4}-(r_+)^{L}\\right)\\cdot\\norm{x_0}\\\\\n\t\t\t\t& \\leftstackrel{\\eqref{large S1}}{\\geq} \\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta \\mu\\sigma (r_-)^{L-1}(\\alpha_{\\max}-\\alpha_{\\min})}{8}\\cdot\\norm{x_0}.\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tTherefore, it holds that\n\t\t\\begin{equation*}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\norm{x_{L+S}} & \\; \\leftstackrel{\\eqref{eq:compareiterate}}{\\geq} (r_-)^{L-1} \\norm{x_{\\varrho_0+S}} \\\\\n\t\t\t\t& \\; \\geq \\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta \\mu\\sigma (r_-)^{2(L-1)}(\\alpha_{\\max}-\\alpha_{\\min})}{8}\\cdot\\norm{x_0}.\n\t\t\t\\end{aligned}\n\t\t\\end{equation*}\n\t\tWe finally arrive at \\eqref{local-growth} by setting $T = L + S$ and combining the above with \\eqref{large S2}.\n\t\\end{proof}\n\n\\begin{theorem}\\label{thm: single-pt}\n\t\tSuppose that Assumption~\\ref{Assump: Hess_bdd}, \\ref{Assump: non-degenerate}, and \\ref{Assump: non-zero-proj} hold and that $0<\\alpha_{\\min}<\\alpha_{\\max}<1/M$. Then for every $x^*\\in\\calA$ \n\t\tand every $x_0\\in\\bR^d\\backslash\\{x^*\\}$, it holds that\n\t\t\\begin{equation*}\n\t\t\t\\bP(\\calC(x^*,x_0))=0.\n\t\t\\end{equation*}\n\t\\end{theorem}\n\n\\begin{proof}\n\t\tWithout loss of generality, we assume that $x^*=0$. Conditioned on $\\mathcal{F}_{t-1}$ with $x_t\\in V$, where $V$ can be assumed to be bounded, Theorem~\\ref{thm: local-growth} states that with probability at least $1-4\\epsilon$, \n\t\t\\begin{equation*}\n\t\t\t\\norm{x_{t+T}} \\geq A \\norm{x_t},\n\t\t\\end{equation*}\n\t\twhere to simplify notation we denote \n\t\t\\begin{equation*}\n\t\t\tA := \\exp\\biggl(\\frac{6\\epsilon}{1 - 6 \\epsilon} \\bigl\\lvert \\log(1-M\\alpha_{\\max}) \\bigr\\rvert \\, T\\biggr)\n\t\t\\end{equation*}\n\t\tthe amplification factor appearing on the right-hand side of \\eqref{local-growth}. \n\t\tNotice also that, due to \\eqref{eq:compareiterate}, we always have\n\t\t\\begin{equation*}\n\t\t\t\\norm{x_{t+T}}\\geq (r_-)^{T}\\norm{x_t}.\n\t\t\\end{equation*}\n\t\tIt suffices to show that for any $x_0\\in V\\backslash\\{x^*\\}$, with probability $1$ there exists some $t\\in\\bNp$ such that $x_t\\notin V$.\n\t\t\n\t\tLet us consider the iterates every $T$ steps: Denote $y_t=x_{T t}$ and $\\mathcal{G}_t=\\mathcal{F}_{T t-1}$ for $t\\in\\bN$. Denote stopping time\n\t\t\\begin{equation*}\n\t\t\t\\rho=\\inf\\{t\\in\\bN:y_t\\notin V\\},\n\t\t\\end{equation*}\n\t\tit suffices to show that $\\bP(\\rho<\\infty)=1$. We define a sequence of random variables $I_t$ as follows:\n\t\t\\begin{equation*}\n\t\t\tI_t(\\omega)=\\begin{cases}1,&\\text{if }\\norm{y_{t+1}}\\geq A\\norm{y_t},\\\\\n\t\t\t\t0,&\\text{otherwise}.\\end{cases}\n\t\t\\end{equation*}\n\t\tBy the discussion in the beginning of the proof, we have  \n\t\t\\begin{equation*}\n\t\t\t\\bP(I_t = 1\\ |\\ \\mathcal{G}_t, t<\\rho)\\geq 1-4\\epsilon,\n\t\t\\end{equation*}\n\t\tand moreover, setting $S_t(\\omega)=\\sum_{0\\leq s<t} I_t(\\omega)$, we have for $t<\\rho(\\omega)$,\n\t\t\\begin{equation*}\n\t\t\t\\frac{\\norm{y_t}}{\\norm{y_0}} \\geq A^{S_t(\\omega)} \\cdot (r_-)^{T(t-S_t(\\omega))}.\n\t\t\\end{equation*}\n\t\tDenote $R:=\\sup_{x\\in V}\\norm{x}<\\infty$. Since  $(1-5\\epsilon)\\log A+5\\epsilon T\\log r_->0$, there exists $t_*\\in\\bN$, such that\n\t\t\\begin{equation*}\n\t\t\t\\left(A^{1-5\\epsilon}\\cdot (r_-)^{5\\epsilon T}\\right)^t>\\frac{R}{\\norm{y_0}},\\quad\\forall\\ t\\geq t_*.\n\t\t\\end{equation*}\n\t\tTherefore, for any $t\\geq t_*$, it holds that \n\t\t\\begin{equation*}\n\t\t\t\\bP(\\rho> t) = \\bP\\bigl(\\rho> t,S_t\\leq (1-5\\epsilon)t\\bigr)\\leq \\sum_{i\\leq (1-5\\epsilon)t} \\binom{t}{i} (1-4\\epsilon)^i (4\\epsilon)^{t-i}.\n\t\t\\end{equation*}\n\t\tAs we will show in the next lemma that the right-hand side of above goes to $0$ as $t \\to \\infty$, and thus $\\lim_{t\\rightarrow\\infty}\\bP(\\rho> t)=0$, which implies that $\\bP(\\rho<\\infty)=1$.\n\t\\end{proof}\n\n\\begin{lemma}\\label{lem: binomial}\n\t\tFor any $\\epsilon\\in(0,1/4)$, it holds that\n\t\t\\begin{equation*}\n\t\t\t\\lim_{t\\rightarrow\\infty}\\sum_{i\\leq(1-5\\epsilon)t} \\binom{t}{i} (1-4\\epsilon)^i (4\\epsilon)^{t-i}=0.\n\t\t\\end{equation*}\n\t\\end{lemma}\n\n\\begin{proof}\n\t\tLet $X_0,X_1,X_2,\\dots$ be a sequence of i.i.d. random variables with $X_i$ being a Bernoulli random variable with expectation $1-4\\epsilon$. Denote the average $\\bar{X}_t=\\frac{1}{t} \\sum_{0\\leq s<t} X_s$. The weak law of large numbers yields that\n\t\t\\begin{equation*}\n\t\t\t\\sum_{i\\leq(1-5\\epsilon)t} \\binom{t}{i} (1-4\\epsilon)^i (4\\epsilon)^{t-i}=\\bP\\left(\\bar{X}_t\\leq 1-5\\epsilon\\right)\\leq \\bP\\left(|\\bar{X}_t-\\bE X_0|\\geq \\epsilon\\right)\\rightarrow 0,\n\t\t\\end{equation*}\n\t\tas $t\\rightarrow\\infty$.\n\t\\end{proof}\n\n\\begin{proof}[Proof of Theorem~\\ref{thm: main}]\n\t\tAssumption~\\ref{Assump: non-degenerate} guarantees that, in a small neighborhood of $x^*$, the gradient $\\nabla f(x)=\\nabla^2 f(x^*)(x-x^*)+o(\\norm{x-x^*})$ is non-vanishing as long as $x\\neq x^*$, which implies that $x^*$ is an isolated stationary point. Therefore, $\\calA$ is countable. Then Theorem~\\ref{thm: main} follows directly from Theorem~\\ref{thm: single-pt} and the countability of $\\calA$.\n\t\\end{proof}","units":[{"start":29900,"end":30170,"start_line":320,"end_line":324,"raw_tex":"\\begin{proposition}\\label{prop: posi-lamb1}\n\t\t%For any $x^*\\in\\calA$, if \n\t\tLet $H$ have a negative eigenvalue and\n\t\t$0<\\alpha_{\\min}<\\alpha_{\\max}<1/\\max_{1\\leq i\\leq d}|H_{ii}|$, then the largest Lyapunov exponent of $\\Phi^H(t,\\omega)$ is positive. \n\t\\end{proposition}","sha256_utf8":"9a726910fbfbb8cf39c3f2882bba16c805613be7b655bcf1f1c0824d53f9abf2"},{"start":39709,"end":40249,"start_line":406,"end_line":414,"raw_tex":"\\begin{proposition}\\label{prop: linear_decay}\n\t\tLet $m\\geq d$ be fixed. For the objective function \\eqref{obj: quadratic} with $\\lambda_{\\min}(H)<0$, suppose \n\t\tthat $0<\\alpha_{\\min}<\\alpha_{\\max}<1/\\max_{1\\leq i\\leq d}|H_{ii}|$, there exists $c\\in(0,1)$ depending on $m$, $H$, $\\alpha_{\\min}$, and $\\alpha_{\\max}$, \n\t\tsuch that \n\t\t\\begin{equation*}\n\t\t\tf^H(x_{t+m})-f^H(x_t)\\leq c\\, f^H(x_t),\n\t\t\\end{equation*}\n\t\tholds as long as $f^H(x_t)<0$ and $\\{1,2,\\dots,d\\}=\\{i_t,i_{t+1},\\dots,i_{t+m-1}\\}$ (in the sense of sets). \n\t\\end{proposition}","sha256_utf8":"6ef21af3194063a11aa3669b29941aa369d7d53fe6ac5130157cc6b0ada6fbf0"},{"start":40473,"end":43563,"start_line":421,"end_line":465,"raw_tex":"\\begin{proof} \n\t\tWithout loss of generality, we assume that $t=0$. \n\t\tDue to the choice of $\\alpha_{\\max}$, we have the following simple non-increasing property for any $t'\\in\\bN$:\n\t\t\\begin{equation}\\label{decay: quad}\n\t\t\t\\begin{split}\n\t\t\t\tf^H(x_{t'+1})&=\\frac{1}{2}x_{t'+1}^{\\top} H x_{t'+1}\\\\\n\t\t\t\t&=\\frac{1}{2}x_{t'}^{\\top} \\left(I-\\alpha_{t'} e_{i_{t'}} e_{i_{t'}}^{\\top} H\\right)^{\\top} H \\left(I-\\alpha_{t'} e_{i_{t'}} e_{i_{t'}}^{\\top} H\\right) x_{t'}\\\\\n\t\t\t\t&=f^H(x_{t'})-\\alpha_{t'} \\left(e_{i_{t'}}^{\\top} H x_{t'}\\right)^2+\\frac{1}{2}\\alpha_t^2 e_{i_{t'}}^{\\top} H e_{i_{t'}} \\left(e_{i_{t'}}^{\\top} H x_{t'}\\right)^2\\\\\n\t\t\t\t&\\leq f^H(x_{t'})-\\frac{\\alpha_{t'}}{2} \\left(e_{i_{t'}}^{\\top} H x_{t'}\\right)^2.\n\t\t\t\\end{split}\n\t\t\\end{equation}\n\t\t\n\t\tWrite $x_0=y^*+y_0$ with $y^*\\in\\ker(H)$ and $y_0\\in\\ran(H)$. Let \n\t\t\\begin{equation}\n\t\t\t\\label{def: y}\n\t\t\ty_{t'+1}=y_{t'}-\\alpha_{t'} e_{i_{t'}} e_{i_{t'}}^{\\top} H y_{t'}, \\qquad t'=0,1,\\dots,m-1.\n\t\t\\end{equation} \n\t\tThen $x_{t'}=y^*+y_{t'}$ holds for any $t'=0,1,\\dots,m$. Using \\eqref{decay: quad}, to give an upper bound for $f^H(x_{t+m})-f^H(x_t)$, we would like a nontrivial lower bound for $\\alpha_{t'}\\bigl(e_{i_{t'}}^{\\top} H x_{t'}\\bigr)^2=\\alpha_{t'}\\bigl(e_{i_{t'}}^{\\top} H y_{t'}\\bigr)^2$ for some $t'\\in\\{t,t+1,\\dots,t+m-1\\}$, which is guaranteed by Lemma~\\ref{lem: linear_decay}, whose proof will be postponed. \n\t\t\n\t\t\\begin{lemma} \\label{lem: linear_decay}\n\t\t\tSuppose that $\\{1,2,\\dots,d\\}=\\{i_0,i_1,\\dots,i_{m-1}\\}$. For any\n\t\t\t\\begin{equation}\\label{def:delta}\n\t\t\t\t0<\\delta\\leq\\min\\left\\{\\frac{1}{2m},\\frac{\\alpha_{\\min}\\sigma_{\\min}(H)}{2\\sqrt{m}(m\\alpha_{\\max}\\sigma_{\\max}(H)+1)}\\right\\},\n\t\t\t\\end{equation}\n\t\t\twhere $\\sigma_{\\min}(H)$ and $\\sigma_{\\max}(H)$ are the smallest and largest positive singular values of $H$ respectively, \n\t\t\tif $0\\neq y_0\\in \\ran(H)$, then there exists $T\\in \\{0,1,\\dots,m-1\\}$ such that $\\alpha_{T}\\left\\vert e_{i_{T}}^{\\top} H y_{T}\\right\\vert\\geq \\delta\\norm{y_{T}}$, where the sequence $y_t$ is given as in \\eqref{def: y}. \n\t\t\\end{lemma}\n\t\t\n\t\tAssuming the lemma, there exists $T\\in\\{0,1,\\dots,m-1\\}$ that \n\t\t\\begin{equation*}\n\t\t\t\\alpha_T\\left\\vert e_{i_{T}}^{\\top} H y_{T}\\right\\vert\\geq \\delta\\norm{y_{T}},\n\t\t\\end{equation*}\n\t\twith a fixed $\\delta>0$ satisfying \\eqref{def:delta}. We can further constrain that $\\frac{\\delta^2}{\\alpha_{\\min}\\sigma_{\\max}(H)}<1$. Thus, we have\n\t\t\\begin{equation*}\\label{decay esti}\n\t\t\t\\begin{split}\n\t\t\t\tf^H(x_m)&\\leq f^H (x_{T+1})\\leq f^H(x_T)- \\frac{\\alpha_T}{2}\\left(e_{i_{T}}^{\\top} H x_{T}\\right)^2\\leq f^H (x_{T}) -\\frac{\\delta^2}{2\\alpha_T}\\norm{y_{T}}^2,\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\twhich combined with $\\sigma_{\\max}(H)\\norm{y_{T}}^2\\geq -y_{T}^{\\top} H y_{T}=-x_{T}^{\\top} H x_{T}=-2f^H(x_T)$,\n\t\tyields that \n\t\t\\begin{equation*}\n\t\t\tf^H (x_m) \\leq\\left( 1 +\\frac{\\delta^2}{\\alpha_T \\sigma_{\\max}(H)}\\right)f^H (x_{T}) \\leq\\left( 1 +\\frac{\\delta^2}{\\alpha_T \\sigma_{\\max}(H)}\\right)f^H (x_0). \n\t\t\\end{equation*}\n\t\tSet $c=\\frac{\\delta^2}{\\alpha_{\\max} \\sigma_{\\max}(H)}$ and we get that $f^H(x_m)-f^H(x_0)\\leq c f^H(x_0)$.\n\t\\end{proof}","sha256_utf8":"ce8bc2145cd3e5f05c1009b995275b1bd4ec844259a542b3520371247002474f"},{"start":43646,"end":45582,"start_line":470,"end_line":500,"raw_tex":"\\begin{proof}[Proof of Lemma~\\ref{lem: linear_decay}] Suppose on the contrary that $\\alpha_t\\left\\vert e_{i_t}^{\\top} H y_t\\right\\vert< \\delta\\norm{y_t}$ for any $t\\in \\{0,1,\\dots,m-1\\}$. It holds that\n\t\t\\begin{equation*}\n\t\t\t\\norm{y_1-y_0}=\\alpha_0\\left\\vert e_{i_0}^{\\top} H y_0\\right\\vert<\\delta\\norm{y_0}<2\\delta\\norm{y_0}.\n\t\t\\end{equation*}\n\t\tWe claim that \n\t\t\\begin{equation} \\label{eq: disty}\n\t\t\t\\norm{y_t-y_0}<2 t\\delta\\norm{y_0},\n\t\t\\end{equation}\n\t\tfor any $t=1,2,\\dots,m-1$. By induction, assume that  $\\norm{y_t-y_0}<2t\\delta\\norm{y_0}$ holds for some $t\\in\\{1,2,\\dots,m-2\\}$, then \n\t\t\\begin{equation*}\n\t\t\t\\norm{y_{t+1}-y_t}=\\alpha_t\\left\\vert e_{i_t}^{\\top} H y_t\\right\\vert< \\delta\\norm{y_t}<\\delta(2t\\delta+1)\\norm{y_0}<2\\delta\\norm{y_0},\n\t\t\\end{equation*}\n\t\twhere the last inequality uses $2 t \\delta < 2m \\delta \\leq 1 $. It follows $\\norm{y_{t+1}-y_0}\\leq \\norm{y_t-y_0}+\\norm{y_{t+1}-y_t}<2(t+1)\\delta\\norm{y_0}$.\n\t\t\n\t\tUsing \\eqref{eq: disty} and $\\max_{1\\leq i\\leq d} \\norm{H e_i}\\leq \\sigma_{\\max}(H)$, we have \n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\alpha_t\\bigl\\lvert e_{i_t}^{\\top} H y_0\\bigr\\rvert&\\leq \\alpha_t\\bigl\\lvert e_{i_t}^{\\top} H (y_t-y_0)\\bigr\\rvert+\\alpha_t\\bigl\\lvert e_{i_t}^{\\top} H y_t\\bigr\\rvert\\\\\n\t\t\t\t&<\\alpha_{\\max}\\sigma_{\\max}(H)\\cdot 2 t\\delta\\norm{y_0}+2\\delta\\norm{y_0}\\\\\n\t\t\t\t&<2\\delta(m\\alpha_{\\max}\\sigma_{\\max}(H)+1)\\norm{y_0},\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tfor $t=0,1,\\dots,m-1$. Since $\\text{span}\\{e_{i_k}:k=0,1,\\dots,m-1\\}=\\bR^d$, noticing that $y_0\\in\\ran{H}$, we have\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\alpha_{\\min}\\sigma_{\\min}(H) \\norm{y_0}&\\leq \\alpha_{\\min}\\norm{H y_0}\\leq\\left(\\sum_{t=0}^{m-1} \\left(\\alpha_t\\bigl\\lvert e_{i_t}^{\\top} H y_0\\bigr\\rvert\\right)^2\\right)^{1/2}\\\\\n\t\t\t\t& <2\\delta\\sqrt{m}(m\\alpha_{\\max}\\sigma_{\\max}(H)+1)\\norm{y_0},\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\twhich contradicts with the choice of $\\delta$ in \\eqref{def:delta}.\n\t\\end{proof}","sha256_utf8":"f425fc3bebacb66a85873f471543e908f9e14a65caa00b824bf4e39204efdf07"},{"start":45742,"end":47694,"start_line":505,"end_line":531,"raw_tex":"\\begin{proof}[Proof of Proposition~\\ref{prop: posi-lamb1}] It suffices to show that for almost every $\\omega\\in\\Omega$ there exist some $x_0\\in \\bR^d$, $\\epsilon>0$, and $T>0$, such that $x_t=\\Phi(t,\\omega)x_0$ satisfies $\\norm{x_t}\\geq e^{\\epsilon t}$ for any $t>T$. Let $x_0$ be an eigenvector corresponding to a negative eigenvalue of $H$. Then it holds that $f^H(x_0)< 0$. Consider a fixed $m\\geq d$. For any $k\\in\\bN$, set $I_k=1$ if $\\{1,2,\\dots,d\\}=\\{i_{km},i_{km+1},\\dots,i_{km+m-1}\\}$ \n\t\tand $I_k=0$ otherwise. We can see that the random variables, $I_0,I_1,I_2,\\dots$, are independent and identically distributed with $\\bE I_0=\\bP(I_0=1)\\in(0,1)$. By Proposition~\\ref{prop: linear_decay}, we obtain that\n\t\t\\begin{equation*}\n\t\t\tf^H(x_{(k+1)m})\\leq\\begin{cases}\n\t\t\t\t(1+c)f^H(x_{km}),&\\text{if }I_k=1,\\\\\n\t\t\t\tf^H(x_{km}),&\\text{if }I_k=0,\n\t\t\t\\end{cases}\n\t\t\\end{equation*}\t\n\t\twhere $c$ is the constant from Proposition~\\ref{prop: linear_decay}.\n\t\tTherefore, %\\jl{missing $1/2$}\n\t\t\\begin{equation*}\n\t\t\t\\frac{\\lambda_{\\min}(H)}{2}\\norm{x_{km}}^2\\leq f^H(x_{km})\\leq(1+c)^{\\sum_{j=0}^{k-1} I_j}f^H(x_0),\n\t\t\\end{equation*}\n\t\twhich implies that\n\t\t\\begin{equation}\\label{norm:xkm}\n\t\t\t\\norm{x_{km}}\\geq \\left(\\frac{2f^H(x_0)}{\\lambda_{\\min}(H)}\\right)^{1/2}\\cdot (1+c)^{\\frac{1}{2}\\sum_{j=0}^{k-1} I_j}.\n\t\t\\end{equation}\n\t\tNote that $\\bE |I_0|=\\bE I_0<\\infty$. The strong law of large number suggests that for almost every $\\omega\\in\\Omega$, there exists some $K$, such that for all $k \\geq K$\n\t\t\\begin{equation}\\label{SLLN}\n\t\t\t\\sum_{j=0}^{k-1} I_j\\geq \\frac{\\bE I_0}{2}k.\n\t\t\\end{equation}\n\t\tCombining \\eqref{norm:xkm} and \\eqref{SLLN}, we arrive at\n\t\t\\begin{equation*}\n\t\t\t\\norm{x_{km}}\\geq \\left(\\frac{2f^H(x_0)}{\\lambda_{\\min}(H)}\\right)^{1/2}\\cdot (1+c)^{\\frac{\\bE I_0}{4m}\\cdot km}.\n\t\t\\end{equation*}\n\t\tNoticing that $(1+c)^{\\frac{\\bE I_0}{4m}}$ is greater than $1$, $\\norm{x_{km}}$ grows exponentially in $km$ and we complete the proof.\n\t\\end{proof}","sha256_utf8":"83bd23b4b13160a77b83f169b859b7165544c5568720980ffdcf9d0e1e45a138"},{"start":48416,"end":49217,"start_line":538,"end_line":544,"raw_tex":"\\begin{theorem}\\label{thm: local-growth}\n\t\tSuppose that Assumptions~\\ref{Assump: Hess_bdd}, \\ref{Assump: non-degenerate}, and \\ref{Assump: non-zero-proj} hold and that $0<\\alpha_{\\min}<\\alpha_{\\max}<1/M$. There exists $\\epsilon_*\\in(0,1/6)$ such that for any $\\epsilon\\in(0,\\epsilon_*)$, we have $T_* = T_*(\\epsilon) \\in\\bNp$ that for any $T \\in \\bNp$ with $T\\geq T_*$ and any $t\\in\\bN$, conditioned on $\\mathcal{F}_{t-1}$, with probability at least $1-4\\epsilon$, it holds for all $x_t \\in V$ that\n\t\t\\begin{equation}\\label{local-growth}\n\t\t\t\\norm{x_{t+T}}\\geq \\exp\\biggl(\\frac{6\\epsilon}{1 - 6 \\epsilon} \\bigl\\lvert \\log(1-M\\alpha_{\\max}) \\bigr\\rvert \\, T\\biggr) \\norm{x_t},\n\t\t\\end{equation}\n\t\twhere $V$ is a neighborhood of $x^* = 0$, depending on $\\epsilon$, $T$, and $f$ near $x^*$. \n\t\\end{theorem}","sha256_utf8":"86ab076bc46e3a0d18eb73f6055208d2772f8d8dadaf5eb1553f26d21cf9860c"},{"start":52428,"end":52928,"start_line":572,"end_line":578,"raw_tex":"\\begin{lemma}\\label{lem: L}\n\t\tLet $0<\\mu\\leq\\frac{1}{\\sqrt{d}}$ be a fixed constant. \n\t\tThere exists some constant $L>0$, such that for any $t\\in\\bN$, there exists a measurable $\\varrho_t:\\Omega\\rightarrow \\bNp$ such that $t < \\varrho_t \\leq t + L$ and\n\t\t\\begin{equation}\\label{eq: rho_t}\n\t\t\t\\bP\\Bigl(\\bigl\\lvert e_{i_{\\varrho_t-1}}^{\\top} \\nabla f(x_{\\varrho_t-1})\\bigr\\rvert \\geq \\mu\\norm{\\nabla f(x_{\\varrho_t-1})}\\;\\Big\\vert\\; \\mathcal{F}_{t-1}\\Bigr)\\geq 1-\\epsilon.\n\t\t\\end{equation}\n\t\\end{lemma}","sha256_utf8":"bade027f138f6e63725c2f2cb2556c4be3fea3e07966287a0b9c4a9385690e0a"},{"start":52932,"end":53651,"start_line":580,"end_line":591,"raw_tex":"\\begin{proof}\n\t\tFor any $t\\in\\bN$, use $\\ell_0$ to denote the smallest non-negative integer $\\ell$, such that\n\t\t\\begin{equation*}\n\t\t\t\\ell_0 = \\arg\\min_{\\ell} \\Bigl\\{ \\ell \\in \\mathbb{N}_+ \\,:\\,\t\\bigl\\lvert e_{i_{t+\\ell-1}}^{\\top} \\nabla f(x_{t+\\ell-1})\\bigr\\rvert\\geq \\mu\\norm{\\nabla f(x_{t+\\ell-1})} \\Bigr\\}.\n\t\t\\end{equation*}\n\t\tIt is clear that $\\bP(\\ell_0 >\\ell\\mid\\mathcal{F}_{t-1})\\leq (1-1/d)^\\ell$, since for each step the coordinate is randomly chosen. Hence, there exists some $L>0$, such that \n\t\t\\begin{equation*}\n\t\t\t\\bP(\\ell_0 \\leq L\\mid\\mathcal{F}_{t-1})\\geq 1-\\epsilon.\n\t\t\\end{equation*}\n\t\tWe finish the proof by setting $\\varrho_t = t + \\min\\{ \\ell_0, L \\}$\n\t\twhich has the desired property. \n\t\\end{proof}","sha256_utf8":"2cb23b47832c04700f6ee8de72c2f2bd06baee51d26c6d17970755d076e8b1d7"},{"start":56185,"end":56636,"start_line":619,"end_line":627,"raw_tex":"\\begin{lemma}\\label{lem: Omega1}\n\t\tUnder Assumption~\\ref{Assump: non-zero-proj}, \n\t\tthere exist $\\delta>0$ and measurable $\\Omega_1^\\epsilon\\subset\\widetilde{\\Omega}$, where $\\widetilde{\\Omega}$ is from Theorem~\\ref{thm: MET}, \n\t\tsuch that $\\bP(\\Omega_1^\\epsilon)\\geq 1-\\epsilon$ \n\t\tand \n\t\t\\begin{equation*}\n\t\t\t\\norm{\\mathcal{P}^H_+(\\omega) e_i}\\geq\\delta,\\quad \\forall\\ \\omega\\in\\Omega_1^\\epsilon,\\ i\\in\\{1,2,\\dots,d\\}.\n\t\t\\end{equation*}\n\t\\end{lemma}","sha256_utf8":"35aa7d7fb1905abf3ebbe6d17619b7962120e94ff0ab50ec22cb33b322b0b003"},{"start":56640,"end":57264,"start_line":629,"end_line":641,"raw_tex":"\\begin{proof}\n\t\tAssumption~\\ref{Assump: non-zero-proj} implies that \n\t\t\\begin{equation*}\n\t\t\t\\bP\\bigl(\\{\\omega\\in\\widetilde{\\Omega}:\\norm{\\mathcal{P}^H_+(\\omega) e_i}>0,\\quad \\forall\\ i\\in\\{1,2,\\dots,d\\}\\}\\bigr) = 1\n\t\t\\end{equation*}\n\t\tNotice that \n\t\t\\begin{multline*}\n\t\t\t\\bigl\\{\\omega\\in\\widetilde{\\Omega}:\\norm{\\mathcal{P}^H_+(\\omega) e_i}>0,\\quad \\forall\\ i\\in\\{1,2,\\dots,d\\}\\bigr\\}\n\t\t\t= \\\\ \n\t\t\t=\\bigcup_{n\\in\\bNp}\\Bigl\\{\\omega\\in\\widetilde{\\Omega}:\\norm{\\mathcal{P}^H_+(\\omega) e_i}\\geq \\frac{1}{n},\\quad \\forall\\ i\\in\\{1,2,\\dots,d\\}\\Bigr\\}.\n\t\t\\end{multline*}\n\t\tThe Lemma follows from continuity of measure. \n\t\\end{proof}","sha256_utf8":"32c89ccc6733c938a2f3c6d2c8c2724c64d5bfbb66ff0a8e70b7157459555955"},{"start":59483,"end":59702,"start_line":669,"end_line":671,"raw_tex":"\\begin{lemma}\\label{lem: T-Omega2}\n\t\tUnder the same assumptions of Lemma~\\ref{lem: Omega1}, there exists some $S_*>0$, such that for every $S \\in \\bNp, S \\geq S_*$, it holds $\\bP(\\Omega^S)\\geq 1-2\\epsilon$.\n\t\\end{lemma}","sha256_utf8":"106e7ae66b74216aae30dee1c1714d245d072c0c6c5622b677757e5fe85b4dd2"},{"start":59706,"end":61770,"start_line":673,"end_line":705,"raw_tex":"\\begin{proof}\n\t\tFor $a.e.\\ \\omega\\in\\Omega$, it follows from Theorem~\\ref{thm: MET}, in particular \\eqref{conv: Lamb}, and standard matrix perturbation analysis (see e.g., \\cite[Theorem VI.2.1, Theorem VII.3.1]{bhatia}) that \n\t\t\\begin{equation}\\label{conv_lya_component}\n\t\t\t\\frac{1}{S}s_j(\\Phi^H(S,\\omega))\\rightarrow\\lambda_{\\mu(j)},\\quad S\\rightarrow\\infty.\n\t\t\\end{equation}\n\t\tfor any $j\\in\\{1,2,\\dots,d\\}$, and that\n\t\t\\begin{equation}\\label{conv_subspace}\n\t\t\t\\mathcal{P}^H_+(S,\\omega)\\rightarrow\\mathcal{P}^H_+(\\omega),\\quad S\\rightarrow\\infty.\n\t\t\\end{equation}\n\t\t\n\t\tBy Egorov's theorem, there exists $\\Omega_2^\\epsilon\\subset\\Omega_1^\\epsilon$ with $\\bP(\\Omega_2^\\epsilon) \\geq 1 - 2 \\epsilon$, such that the convergences in \\eqref{conv_lya_component} and \\eqref{conv_subspace} are both uniform on $\\Omega_2^\\epsilon$. Here $\\Omega_1^\\epsilon$ is as in Lemma~\\ref{lem: Omega1}. Therefore, for some $S_*$ sufficiently large, we have\n\t\t\\begin{equation*}\\label{uniconv: singval}\n\t\t\t\\left|\\frac{1}{S}\\log s_j\\left(\\Phi^H(S,\\omega)\\right)- \\lambda_{\\mu(j)}\\right|\\leq \\gamma,\\quad\\forall\\ j\\in\\{1,2,\\dots,d\\},\\ S\\geq S_*,\\ \\omega\\in\\Omega_2^\\epsilon,\n\t\t\\end{equation*}\n\t\tand\n\t\t\\begin{equation}\\label{uniconv: singspace}\n\t\t\t\\norm{\\mathcal{P}^H_+(S,\\omega)-\\mathcal{P}^H_+(\\omega)}\\leq \\frac{\\delta}{2},\\quad\\forall\\ S\\geq S_*,\\ \\omega\\in\\Omega_2^\\epsilon.\n\t\t\\end{equation}\n\t\tCombining Lemma~\\ref{lem: Omega1} and \\eqref{uniconv: singspace}, we obtain that\n\t\t\\begin{equation*}\n\t\t\t\\norm{\\mathcal{P}^H_+(S,\\omega)e_i}\\geq \\frac{\\delta}{2},\\quad \\forall\\ i\\in\\{1,2,\\dots,d\\},\\ \\forall\\ S\\geq S_*,\\ \\omega\\in\\Omega_2^\\epsilon.\n\t\t\\end{equation*}\n\t\t\n\t\tFor any $S\\geq S_*$, \n\t\tby the definition of $\\Omega^S$, it holds that \n\t\t\\begin{equation*}\n\t\t\t\\Omega_2^\\epsilon\\subset\\Omega^S\\times\\Bigl(\\bigtimes_{t\\geq S}\\Omega_t\\Bigr),\n\t\t\\end{equation*}\n\t\twhich implies the desired estimate \n\t\t\\begin{equation*}\n\t\t\t\\bP(\\Omega^S)=\\bP\\biggl(\\Omega^S\\times\\Bigl(\\bigtimes_{t\\geq S}\\Omega_t\\Bigr)\\biggr)\\geq\\bP(\\Omega_2^\\epsilon)\\geq 1-2\\epsilon. %\\qedhere\n\t\t\\end{equation*}\n\t\\end{proof}","sha256_utf8":"5904ce0f28ecd6f5e1910f9d5df4e4c8fef80b76b89590c42bcc456608df700c"},{"start":62243,"end":62914,"start_line":710,"end_line":716,"raw_tex":"\\begin{lemma}\\label{lem: nontrial-proj}\n\t\tFor any $S\\in\\bNp$, $x_{\\varrho_t-1}$, $i_{\\varrho_t-1}$, \n\t\tand $(i,\\alpha)_{\\varrho_t:\\varrho_t+S-1} \\in\\Omega^S$, there exists $I\\subset[\\alpha_{\\min},\\alpha_{\\max}]$ with $m(I)\\geq (1-\\epsilon)(\\alpha_{\\max}-\\alpha_{\\min})$ where $m(\\cdot)$ is the Lebesgue measure,  such that for any $\\alpha_{\\varrho_t-1}\\in I$, it holds that \n\t\t\\begin{equation}\\label{eq: nontrial-proj}\n\t\t\t\\norm{\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\right)x_{\\varrho_t}}\\geq \\frac{\\epsilon\\delta (\\alpha_{\\max}-\\alpha_{\\min})}{4}\\bigl\\lvert e_{i_{\\varrho_t-1}}^{\\top} \\nabla f(x_{\\varrho_t-1})\\bigr\\rvert.\n\t\t\\end{equation}\n\t\\end{lemma}","sha256_utf8":"0f195d5ec1f8ec81609f8a4aa87568e936e1c02fd1c3199107374fedf6965438"},{"start":62918,"end":64832,"start_line":718,"end_line":743,"raw_tex":"\\begin{proof}\n\t\tWe assume that $\\bigl\\lvert e_{i_{\\varrho_t-1}}^{\\top} \\nabla f(x_{\\varrho_t-1})\\bigr\\rvert\\neq 0$; otherwise the result is trivial. \n\t\t\n\t\tFor simplicity of notation, we write \n\t\t\\begin{equation*}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\right)x_{\\varrho_t} & = \n\t\t\t\t\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\right)x_{\\varrho_t-1} \\\\\n\t\t\t\t& \\qquad \n\t\t\t\t- \\alpha_{\\varrho_t - 1} e_{i_{\\varrho_t-1}}^{\\top}\\nabla f(x_{\\varrho_t-1})\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\right)e_{i_{\\varrho_t-1}} \\\\\n\t\t\t\t& =: y_2-\\alpha_{\\varrho_t-1} y_1,\n\t\t\t\\end{aligned}\n\t\t\\end{equation*}\n\t\twhere the last line defines $y_1$ and $y_2$. Using the short-hand notation \n\t\t\\begin{equation*}\n\t\t\tr=\\frac{\\epsilon\\delta (\\alpha_{\\max}-\\alpha_{\\min})}{4}\\bigl\\lvert e_{i_{\\varrho_t-1}}^{\\top} \\nabla f(x_{\\varrho_t-1})\\bigr\\rvert, \n\t\t\\end{equation*}\n\t\twe observe then \\eqref{eq: nontrial-proj} holds if and only if $\\alpha_{\\varrho_t-1} y_1$ is not located in a ball with radius $r$ centered at $y_2$.\n\t\t\n\t\tIt follows from the definition of $\\Omega^S$ and \\eqref{close: singular space} that $ \\norm{\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\right)e_{i_{\\varrho_t-1}}} \\geq \\frac{\\delta}{2}$, \n\t\twhich then leads to\n\t\t\\begin{equation*}\n\t\t\t\\norm{y_1}\\geq \\frac{\\delta}{2}\\bigl\\lvert e_{i_{\\varrho_t-1}}^{\\top} \\nabla f(x_{\\varrho_t-1})\\bigr\\rvert=:\\frac{2r}{\\epsilon(\\alpha_{\\max}-\\alpha_{\\min})}.\n\t\t\\end{equation*}\n\t\tThus, the set of $\\alpha_{\\varrho_t-1}$ such that $\\alpha_{\\varrho_t -1} y_1 \\in \\mathcal{B}_r(y_2)$ consists of an interval $J$ in $\\bR$ with $\\|\\sup(J)\\cdot y_1 - \\inf(J)\\cdot y_1\\|\\leq 2r$ as the diameter of $\\mathcal{B}_r(y_2)$ is $2r$, which implies $m(J)\\leq 2r/\\norm{y_1}\\leq \\epsilon(\\alpha_{\\max}-\\alpha_{\\min})$. The lemma is proved then by setting $I=[\\alpha_{\\max}-\\alpha_{\\min}]\\backslash J$.\n\t\\end{proof}","sha256_utf8":"1ef03a784d2b33aebff839547f076f6cad1e158d3105fd0e5b25b06e5de508d1"},{"start":65093,"end":73009,"start_line":749,"end_line":859,"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm: local-growth}]\n\t\tWithout loss of generality, we will assume $t=0$ in the proof to simplify notation.\n\t\tSince $H=\\nabla^2 f(x^*)$ is non-degenerate, we can take a neighborhood \n\t\t$U$ of $x^*=0$ and some fixed $\\sigma > 0$ such that\n\t\t\\begin{equation}\\label{eq: sigma}\n\t\t\t\\norm{\\nabla f(x)}\\geq \\sigma\\norm{x},\\quad \\forall\\ x\\in U.    \n\t\t\\end{equation}\n\t\tAssumption~\\ref{Assump: Hess_bdd} implies that\n\t\t\\begin{equation*}\n\t\t\t\\norm{\\nabla f(x)}=\\norm{\\nabla f(x)-\\nabla f(x^*)}\\leq M\\norm{x-x^*}=M\\norm{x},\\quad\\forall\\ x\\in\\bR^d.\n\t\t\\end{equation*}\n\t\tUsing the above inequality and $\\alpha_{\\max}<1/M$, it holds for every $\\omega\\in\\Omega$ and $t'\\in\\bN$ that \n\t\t\\begin{equation}\\label{r-minus}\n\t\t\t\\begin{split}\n\t\t\t\t\\norm{x_{t'+1}}&=\\norm{x_{t'}-\\alpha_{t'} e_{i_{t'}}e_{i_{t'}}^{\\top}\\nabla f(x_{t'})} \\\\\n\t\t\t\t& \\geq \\norm{x_{t'}}-\\alpha_{t'}\\norm{e_{i_{t'}}e_{i_{t'}}^{\\top}}\\cdot \\norm{\\nabla f(x_{t'})}\\\\\n\t\t\t\t&\\geq (1-M\\alpha_{\\max})\\norm{x_{t'}},\n\t\t\t\\end{split}\n\t\t\\end{equation}\n\t\tand similarly that\n\t\t\\begin{equation*}\n\t\t\t\\norm{x_{t'+1}}\\leq(1+M\\alpha_{\\max})\\norm{x_{t'}}.\n\t\t\\end{equation*}\n\t\tWe thus define\n\t\t\\begin{equation*}\n\t\t\tr_-:=1-M\\alpha_{\\max}\\quad\\text{and}\\quad r_+:=1+M\\alpha_{\\max}, \n\t\t\\end{equation*}\n\t\tso that\n\t\t\\begin{equation}\\label{eq:compareiterate}\n\t\t\tr_- \\norm{x_{t'}} \\leq \\norm{x_{t'+1}} \\leq r_+ \\norm{x_{t'}}.\n\t\t\\end{equation}\n\t\t\n\t\tWe now choose the time duration $S$ large enough in the finite block analysis to guarantee significant amplification. More specifically, we choose $S$ so large that $S \\geq S_*$ ($S_*$ defined in Lemma~\\ref{lem: T-Omega2}) and that the following two inequalities hold: \n\t\t\\begin{equation}\\label{large S1}\n\t\t\t\\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta \\mu\\sigma (r_-)^{L-1}(\\alpha_{\\max}-\\alpha_{\\min})}{8}\\geq\n\t\t\t(r_+)^{L},\n\t\t\\end{equation}\n\t\tand\n\t\t\\begin{equation}\\label{large S2}\n\t\t\t(1-6\\epsilon)\\left(S(\\lambda_+-\\gamma)+\\log \\frac{\\epsilon\\delta \\mu\\sigma (r_-)^{2(L-1)}(\\alpha_{\\max}-\\alpha_{\\min})}{8}\\right)+6\\epsilon(L+S)\\log r_->0,\n\t\t\\end{equation}\n\t\twhere $L$ is the upper bound defined in Lemma~\\ref{lem: L}, $\\mu\\leq 1/\\sqrt{d}$ is a fixed constant as in Lemma~\\ref{lem: L}, $\\delta$ is from Lemma~\\ref{lem: Omega1}, and $\\sigma$ is set in \\eqref{eq: sigma}. \n\t\tThanks to \\eqref{def: eps} for our choice of $\\epsilon$ and that $\\gamma < \\lambda_+$ from \\eqref{lambplus-gamma}, \n\t\t\\eqref{large S1} and \\eqref{large S2} are satisfied for sufficiently large $S$. \n\t\t\n\t\tNext, we show that, for any $S$ sufficiently large as above, there exists a convex neighborhood $U_1\\subset U$ of $x^*=0$, such that for any $t'\\in\\bN$, any $x_{t'}\\in U_1$, and any $(i,\\alpha)_{t':t'+S-1}$, it holds that\n\t\t\\begin{equation}\\label{block-error}\n\t\t\t\\norm{x_{t'+S}} \\geq \\norm{\\Phi^H\\bigl((i,\\alpha)_{t':t'+S-1}\\bigr)x_{t'}}-\\norm{x_{t'}}.\n\t\t\\end{equation}\n\t\tWe first define a convex neighborhood $U_0 \\subset U$ of $x^*=0$ such that\n\t\t\\begin{multline*}\n\t\t\t\\norm{\\left(x-\\alpha e_i e_i^{\\top}\\nabla f(x)\\right)-\\left(I-\\alpha e_i e_i^{\\top}H\\right)x}=\\norm{\\alpha e_i e_i^{\\top}\\left(\\nabla f(x)-H x\\right)}\\\\\n\t\t\t=\\norm{\\alpha e_i e_i^{\\top}\\int_0^1 (\\nabla^2 f(\\eta x)-Hx) \\mathrm{d} \\eta}\\leq \\frac{1}{S(r_+)^{S-1}} \\norm{x},\n\t\t\\end{multline*}\n\t\tfor any $x \\in U_0$, any $i\\in\\{1,2,\\dots,d\\}$, and any $\\alpha\\in[\\alpha_{\\min},\\alpha_{\\max}]$.\n\t\tApplying the inequality $S$ times for $x_{t'}\\in U_1 = (r_+)^{-(S-1)}U_0$, we have,\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t&\\norm{x_{t'+S}-\\Phi^H\\bigl((i,\\alpha)_{t':t'+S-1}\\bigr)x_{t'}}\\\\\n\t\t\t\t\\leq & \\norm{x_{t'+S}-\\left(I-\\alpha_{t'+S-1} e_{i_{t'+S-1}}e_{i_{t'+S-1}}^{\\top}H\\right)x_{t'+S-1}}\\\\\n\t\t\t\t&+\\norm{I-\\alpha_{t'+S-1} e_{i_{t'+S-1}}e_{i_{t'+S-1}}^{\\top}H}\\cdot \\norm{x_{t'+S-1}-\\Phi^H\\bigl((i,\\alpha)_{t':t'+S-2}\\bigr)x_{t'}}\\\\\n\t\t\t\t\\leq & \\frac{1}{S(r_+)^{S-1}} \\norm{x_{t'+S-1}}+r_+\\norm{x_{t'+S-1}-\\Phi^H\\bigl((i,\\alpha)_{t':t'+S-2}\\bigr)x_{t'}}\\\\\n\t\t\t\t\\leq& \\frac{1}{S(r_+)^{S-1}} \\left(\\norm{x_{t'+S-1}}+r_+\\norm{x_{t'+S-2}}+\\cdots+(r_+)^{S-1}\\norm{x_{t'}}\\right)\\\\\n\t\t\t\t\\leq& \\norm{x_{t'}},\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tand hence, inequality \\eqref{block-error}.\n\t\t\n\t\tSetting $V=(r_+)^{-(L-1)} U_1$, we then have $x_{\\varrho_0-1}\\in U$ for any $x_0\\in V$, which implies that $\\norm{\\nabla f(x_{\\varrho_0-1})}\\geq\\sigma\\norm{x_{\\varrho_0-1}}$ as $\\varrho_0\\leq L$. \n\t\tAccording to Lemma~\\ref{lem: L}--\\ref{lem: nontrial-proj}, for any given $x_0\\in V$, with probability at least $1-4\\epsilon$, we have $(i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\in\\Omega^S$, and  the followings hold:\n\t\t\\begin{align}\n\t\t\t\\label{ineq1} & \\bigl\\lvert e_{i_{\\varrho_0-1}}^{\\top} \\nabla f(x_{\\varrho_0-1})\\bigr\\rvert\\geq \\mu\\norm{\\nabla f(x_{\\varrho_0-1})}, \\\\\n\t\t\t\\label{ineq2} & \\norm{\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)x_{\\varrho_0}}\\geq \\frac{\\epsilon\\delta (\\alpha_{\\max}-\\alpha_{\\min})}{4}\\bigl\\lvert e_{i_{\\varrho_0-1}}^{\\top} \\nabla f(x_{\\varrho_0-1})\\bigr\\rvert,\n\t\t\\end{align}\n\t\twhere the probability is the marginal probability on $(i,\\alpha)_{0:L+S-1}\\in\\Omega_0\\times\\cdots\\times \\Omega_{L+S-1}$.  \n\t\t\n\t\t\\smallskip \n\t\t\n\t\tRecall $\\lambda_+$ and $\\gamma$ in \\eqref{lambplus-gamma}, and $d_+=\\sum_{\\lambda_i>0}d_i$. \n\t\tIt follows from \\eqref{close: singular value} of the construction of the set $\\Omega^S$ that\n\t\t\\begin{equation*}\n\t\t\t\\frac{1}{S}\\log s_j\\left(\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)\\right)\\geq \\lambda_{\\mu(j)}-\\gamma\\geq \\lambda_+-\\gamma,\\quad\\forall\\ j\\leq d_+.\n\t\t\\end{equation*}\n\t\tThis is to say that the $d_+$ largest singular values of $\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)$\n\t\tare all greater than or equal to $\\exp(S(\\lambda_+-\\gamma))$. \n\t\tTherefore, it holds that \n\t\t\\begin{equation}\\label{linear-growth}\n\t\t\t\\begin{split}\n\t\t\t\t\\norm{\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)x_{\\varrho_0}}&\\geq \\norm{\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)x_{\\varrho_0}}\\\\\n\t\t\t\t&  \\leftstackrel{\\eqref{ineq2}}{\\geq} \\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta (\\alpha_{\\max}-\\alpha_{\\min})}{4}\\bigl\\lvert e_{i_{\\varrho_0-1}}^{\\top} \\nabla f(x_{\\varrho_0-1})\\bigr\\rvert\\\\\n\t\t\t\t&  \\leftstackrel{\\eqref{ineq1}}{\\geq}\n\t\t\t\t\\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta\\mu (\\alpha_{\\max}-\\alpha_{\\min})}{4}\\norm{\\nabla f(x_{\\varrho_0-1})},\n\t\t\t\\end{split}\n\t\t\\end{equation}\n\t\twhere the first inequality follows from the fact that $\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)x_{\\varrho_0}$ and $\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)\\left(I-\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)\\right)x_{\\varrho_0}$ are orthogonal. Combining \\eqref{block-error} and \\eqref{linear-growth}, we obtain that\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\norm{x_{\\varrho_0+S}} & \\geq \\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta \\mu(\\alpha_{\\max}-\\alpha_{\\min})}{4}\\norm{\\nabla f(x_{\\varrho_0-1})}-\\norm{x_{\\varrho_0}}\\\\\n\t\t\t\t& \\leftstackrel{\\eqref{eq:compareiterate}}{\\geq} \\left(\\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta \\mu\\sigma (r_-)^{L-1}(\\alpha_{\\max}-\\alpha_{\\min})}{4}-(r_+)^{L}\\right)\\cdot\\norm{x_0}\\\\\n\t\t\t\t& \\leftstackrel{\\eqref{large S1}}{\\geq} \\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta \\mu\\sigma (r_-)^{L-1}(\\alpha_{\\max}-\\alpha_{\\min})}{8}\\cdot\\norm{x_0}.\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tTherefore, it holds that\n\t\t\\begin{equation*}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\norm{x_{L+S}} & \\; \\leftstackrel{\\eqref{eq:compareiterate}}{\\geq} (r_-)^{L-1} \\norm{x_{\\varrho_0+S}} \\\\\n\t\t\t\t& \\; \\geq \\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta \\mu\\sigma (r_-)^{2(L-1)}(\\alpha_{\\max}-\\alpha_{\\min})}{8}\\cdot\\norm{x_0}.\n\t\t\t\\end{aligned}\n\t\t\\end{equation*}\n\t\tWe finally arrive at \\eqref{local-growth} by setting $T = L + S$ and combining the above with \\eqref{large S2}.\n\t\\end{proof}","sha256_utf8":"6753f7c1fac9e35a1bfd924389dae20a014acc79ad35aa2987e10adefb6c52e6"},{"start":73392,"end":73759,"start_line":868,"end_line":874,"raw_tex":"\\begin{theorem}\\label{thm: single-pt}\n\t\tSuppose that Assumption~\\ref{Assump: Hess_bdd}, \\ref{Assump: non-degenerate}, and \\ref{Assump: non-zero-proj} hold and that $0<\\alpha_{\\min}<\\alpha_{\\max}<1/M$. Then for every $x^*\\in\\calA$ \n\t\tand every $x_0\\in\\bR^d\\backslash\\{x^*\\}$, it holds that\n\t\t\\begin{equation*}\n\t\t\t\\bP(\\calC(x^*,x_0))=0.\n\t\t\\end{equation*}\n\t\\end{theorem}","sha256_utf8":"f118e581a9045f85feb0b97b824beaeda10e7fe85a647180ca0d3925fc3fce4b"},{"start":73763,"end":76220,"start_line":876,"end_line":918,"raw_tex":"\\begin{proof}\n\t\tWithout loss of generality, we assume that $x^*=0$. Conditioned on $\\mathcal{F}_{t-1}$ with $x_t\\in V$, where $V$ can be assumed to be bounded, Theorem~\\ref{thm: local-growth} states that with probability at least $1-4\\epsilon$, \n\t\t\\begin{equation*}\n\t\t\t\\norm{x_{t+T}} \\geq A \\norm{x_t},\n\t\t\\end{equation*}\n\t\twhere to simplify notation we denote \n\t\t\\begin{equation*}\n\t\t\tA := \\exp\\biggl(\\frac{6\\epsilon}{1 - 6 \\epsilon} \\bigl\\lvert \\log(1-M\\alpha_{\\max}) \\bigr\\rvert \\, T\\biggr)\n\t\t\\end{equation*}\n\t\tthe amplification factor appearing on the right-hand side of \\eqref{local-growth}. \n\t\tNotice also that, due to \\eqref{eq:compareiterate}, we always have\n\t\t\\begin{equation*}\n\t\t\t\\norm{x_{t+T}}\\geq (r_-)^{T}\\norm{x_t}.\n\t\t\\end{equation*}\n\t\tIt suffices to show that for any $x_0\\in V\\backslash\\{x^*\\}$, with probability $1$ there exists some $t\\in\\bNp$ such that $x_t\\notin V$.\n\t\t\n\t\tLet us consider the iterates every $T$ steps: Denote $y_t=x_{T t}$ and $\\mathcal{G}_t=\\mathcal{F}_{T t-1}$ for $t\\in\\bN$. Denote stopping time\n\t\t\\begin{equation*}\n\t\t\t\\rho=\\inf\\{t\\in\\bN:y_t\\notin V\\},\n\t\t\\end{equation*}\n\t\tit suffices to show that $\\bP(\\rho<\\infty)=1$. We define a sequence of random variables $I_t$ as follows:\n\t\t\\begin{equation*}\n\t\t\tI_t(\\omega)=\\begin{cases}1,&\\text{if }\\norm{y_{t+1}}\\geq A\\norm{y_t},\\\\\n\t\t\t\t0,&\\text{otherwise}.\\end{cases}\n\t\t\\end{equation*}\n\t\tBy the discussion in the beginning of the proof, we have  \n\t\t\\begin{equation*}\n\t\t\t\\bP(I_t = 1\\ |\\ \\mathcal{G}_t, t<\\rho)\\geq 1-4\\epsilon,\n\t\t\\end{equation*}\n\t\tand moreover, setting $S_t(\\omega)=\\sum_{0\\leq s<t} I_t(\\omega)$, we have for $t<\\rho(\\omega)$,\n\t\t\\begin{equation*}\n\t\t\t\\frac{\\norm{y_t}}{\\norm{y_0}} \\geq A^{S_t(\\omega)} \\cdot (r_-)^{T(t-S_t(\\omega))}.\n\t\t\\end{equation*}\n\t\tDenote $R:=\\sup_{x\\in V}\\norm{x}<\\infty$. Since  $(1-5\\epsilon)\\log A+5\\epsilon T\\log r_->0$, there exists $t_*\\in\\bN$, such that\n\t\t\\begin{equation*}\n\t\t\t\\left(A^{1-5\\epsilon}\\cdot (r_-)^{5\\epsilon T}\\right)^t>\\frac{R}{\\norm{y_0}},\\quad\\forall\\ t\\geq t_*.\n\t\t\\end{equation*}\n\t\tTherefore, for any $t\\geq t_*$, it holds that \n\t\t\\begin{equation*}\n\t\t\t\\bP(\\rho> t) = \\bP\\bigl(\\rho> t,S_t\\leq (1-5\\epsilon)t\\bigr)\\leq \\sum_{i\\leq (1-5\\epsilon)t} \\binom{t}{i} (1-4\\epsilon)^i (4\\epsilon)^{t-i}.\n\t\t\\end{equation*}\n\t\tAs we will show in the next lemma that the right-hand side of above goes to $0$ as $t \\to \\infty$, and thus $\\lim_{t\\rightarrow\\infty}\\bP(\\rho> t)=0$, which implies that $\\bP(\\rho<\\infty)=1$.\n\t\\end{proof}","sha256_utf8":"996daa1c545ea7f85fe8802596f5f5aa4335bfcfd3092f3d4a06428019d10476"},{"start":76224,"end":76460,"start_line":920,"end_line":925,"raw_tex":"\\begin{lemma}\\label{lem: binomial}\n\t\tFor any $\\epsilon\\in(0,1/4)$, it holds that\n\t\t\\begin{equation*}\n\t\t\t\\lim_{t\\rightarrow\\infty}\\sum_{i\\leq(1-5\\epsilon)t} \\binom{t}{i} (1-4\\epsilon)^i (4\\epsilon)^{t-i}=0.\n\t\t\\end{equation*}\n\t\\end{lemma}","sha256_utf8":"d3f4834bd54bc64bae4371f4cbb09367aa42c7af3b3b414ae0549ece1ecb1255"},{"start":76464,"end":76992,"start_line":927,"end_line":933,"raw_tex":"\\begin{proof}\n\t\tLet $X_0,X_1,X_2,\\dots$ be a sequence of i.i.d. random variables with $X_i$ being a Bernoulli random variable with expectation $1-4\\epsilon$. Denote the average $\\bar{X}_t=\\frac{1}{t} \\sum_{0\\leq s<t} X_s$. The weak law of large numbers yields that\n\t\t\\begin{equation*}\n\t\t\t\\sum_{i\\leq(1-5\\epsilon)t} \\binom{t}{i} (1-4\\epsilon)^i (4\\epsilon)^{t-i}=\\bP\\left(\\bar{X}_t\\leq 1-5\\epsilon\\right)\\leq \\bP\\left(|\\bar{X}_t-\\bE X_0|\\geq \\epsilon\\right)\\rightarrow 0,\n\t\t\\end{equation*}\n\t\tas $t\\rightarrow\\infty$.\n\t\\end{proof}","sha256_utf8":"61ff2bcf462d8143a12711d37ff2e49201d9aee9c5a8cdd2bbc9b23fcc891e9b"},{"start":77075,"end":77539,"start_line":938,"end_line":940,"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm: main}]\n\t\tAssumption~\\ref{Assump: non-degenerate} guarantees that, in a small neighborhood of $x^*$, the gradient $\\nabla f(x)=\\nabla^2 f(x^*)(x-x^*)+o(\\norm{x-x^*})$ is non-vanishing as long as $x\\neq x^*$, which implies that $x^*$ is an isolated stationary point. Therefore, $\\calA$ is countable. Then Theorem~\\ref{thm: main} follows directly from Theorem~\\ref{thm: single-pt} and the countability of $\\calA$.\n\t\\end{proof}","sha256_utf8":"7b532dd451d2a0f1a2216767a37e11770c6a7a1d41cc7ac4a08560cc2e2d8dd9"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"非凸优化问题","source_units":[{"start":3457,"end":3595,"start_line":86,"end_line":86,"raw_tex":"\tIn this paper, we analyze the global convergence of coordinate gradient descent algorithm for smooth but non-convex optimization problem\n","sha256_utf8":"f3c8569970a20a027fe2c6fe9ca68d360d864251c658ded2b6f4c0b4262e2b26"},{"start":3596,"end":3669,"start_line":87,"end_line":89,"raw_tex":"\\begin{equation}\\label{obj-fun}\n\t\t\\min_{x\\in \\bR^d} f(x).\n\t\\end{equation}","sha256_utf8":"07b9351ce3419cc85c7c52f16a8afa4777bd871178b01a0302ab5da59f78cd9e"}]},{"id":"S002","semantic_type":"algorithm","kind":"algorithm","title":"随机坐标与随机步长","source_units":[{"start":3821,"end":4333,"start_line":91,"end_line":103,"raw_tex":"\\begin{algorithm}[H]\n\t\t\\caption{Randomized coordinate gradient descent}\n\t\t\\label{alg:random CGD}\n\t\t\\begin{algorithmic}\n\t\t\t\\State Initialization: $x_0\\in\\bR^d$, $t=0$.\n\t\t\t\\While {not convergent}\n\t\t\t\\State Draw a coordinate $i_t$ uniformly random from $\\{1,2,\\dots,d\\}$.\n\t\t\t\\State Draw a stepsize $\\alpha_t$ uniformly random in $[\\alpha_{\\min},\\alpha_{\\max}]$.\n\t\t\t\\State $x_{t+1}\\leftarrow x_t- \\alpha_t e_{i_t} \\partial_{i_t} f(x_t)$.\n\t\t\t\\State $t\\leftarrow t+1$.\n\t\t\t\\EndWhile\n\t\t\\end{algorithmic}\n\t\\end{algorithm}","sha256_utf8":"04f89a486c1d27a4cf0a4509b99428618df495eccb555dee67a3256556f2beac"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"度量动力系统","source_units":[{"start":13735,"end":14222,"start_line":144,"end_line":144,"raw_tex":"\tLet $(\\Omega,\\mathcal{F},\\bP)$ be a probability space and let $\\mathbb{T}$ be a semigroup with $\\mathcal{B}(\\mathbb{T})$ being its Borel $\\sigma$-algebra. $\\mathbb{T}$ serves as the notion of time. In the setting of Algorithm~\\ref{alg:random CGD}, we have $\\mathbb{T}=\\bN$, corresponding to the one-sided discrete time setting. Other possible examples of $\\mathbb{T}$ include $\\mathbb{T}=\\bZ$, $\\mathbb{T}=\\bR_{\\geq 0}$, and $\\mathbb{T}=\\bR$, with the assumption that $0\\in\\mathbb{T}$.\n","sha256_utf8":"500be49d4b708b2ac1c8da04db8176443e3b0e0a788019b970e230a5d701e351"},{"start":14842,"end":15616,"start_line":147,"end_line":158,"raw_tex":"\\begin{definition}[Metric dynamical system] A metric dynamical system on a probability space $(\\Omega,\\mathcal{F},\\bP)$ is a family of maps $\\{\\theta(t):\\Omega\\rightarrow\\Omega\\}_{t\\in\\mathbb{T}}$ satisfying that\n\t\t\\begin{itemize}\n\t\t\t\\item[(i)] The mapping $\\mathbb{T}\\times\\Omega\\rightarrow\\Omega,\\ (t,\\omega)\\mapsto\\theta(t)\\omega$ is measurable;\n\t\t\t\n\t\t\t\\item[(ii)] It holds that $\\theta(0)=\\mathrm{Id}_{\\Omega}$ and $\\theta(t+s)=\\theta(t)\\circ\\theta(s),\\ \\forall\\ s,t\\in\\mathbb{T}$;\n\t\t\t\n\t\t\t\\item[(iii)] $\\theta(t)$ is $\\bP$-preserving for any $t\\in\\mathbb{T}$, where we say a map $\\theta:\\Omega\\rightarrow\\Omega$ is $\\bP$-preserving if \n\t\t\t\\begin{equation*}\n\t\t\t\t\\bP(\\theta^{-1}B)=\\bP(B),\\quad \\forall\\ B\\in\\mathcal{F}.\n\t\t\t\\end{equation*}\n\t\t\\end{itemize}\n\t\\end{definition}","sha256_utf8":"aeced4d24abe3c9853b4c0b9db8dd9038f69b0a463c8e35f82928c83b03b5a29"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"随机动力系统","source_units":[{"start":15684,"end":16542,"start_line":162,"end_line":179,"raw_tex":"\\begin{definition}[Random dynamical system]\n\t\tLet $(X,\\mathcal{F}_X)$ be a measurable space and let $\\{\\theta(t):\\Omega\\rightarrow\\Omega\\}_{t\\in\\mathbb{T}}$ be a metric dynamical system on $(\\Omega,\\mathcal{F},\\bP)$. Then a random dynamical system on $(X,\\mathcal{F}_X)$ \n\t\tover $\\{\\theta(t)\\}_{t\\in\\mathbb{T}}$ is a measurable map \n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\varphi:\\mathbb{T}\\times\\Omega\\times X&\\rightarrow\\quad X,\\\\\n\t\t\t\t(t,\\omega,x)\\ \\  & \\mapsto \\varphi(t,\\omega,x),\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tsatisfying the following cocycle property: for any $\\omega\\in\\Omega$, $x\\in X$, and $s,t\\in \\mathbb{T}$, it holds that \n\t\t\\begin{equation*}\n\t\t\t\\varphi(0,\\omega,x)=x,\n\t\t\\end{equation*}\n\t\tand that\n\t\t\\begin{equation}\\label{cocycle}\n\t\t\t\\varphi(t+s,\\omega,x)=\\varphi(t,\\theta(s)\\omega,\\varphi(s,\\omega,x)).\n\t\t\\end{equation}\n\t\\end{definition}","sha256_utf8":"03bc5585f50b44a45b3516f1fd639a5b1364c9a6f9ff4626e3a68697c8c114f2"}]},{"id":"S005","semantic_type":"notation","kind":"notation","title":"余循环与线性矩阵积","source_units":[{"start":16545,"end":18127,"start_line":181,"end_line":191,"raw_tex":"\tThe cocycle property \\eqref{cocycle} is a key property of random dynamical system: After time $s$, if we restart the system at $x_s$, the future dynamic corresponds to the sample $\\theta(s)\\omega$. Note that $\\varphi(t,\\omega,\\cdot)$ is a map on $X$, with some ambiguity of notation, we also use $\\varphi(t, \\omega)$ to denote this map on $X$ and write $\\varphi(t,\\omega)x=\\varphi(t,\\omega,x)$. Then the cocycle property \\eqref{cocycle} can be written as \n\t\\begin{equation*}\n\t\t\\varphi(t+s,\\omega)=\\varphi(t,\\theta(s)\\omega)\\circ\\varphi(s,\\omega).\n\t\\end{equation*}\n\t\n\tIn this work, we will focus on the one-sided discrete time $\\mathbb{T}=\\bN$ and $\\theta(t)=\\theta^t$, where $\\theta$ is $\\bP$-preserving and $\\theta^t$ is the $t$-fold composition of $\\theta$. \n\tSuppose that $X=\\bR^d$ and $A:\\Omega\\rightarrow\\text{GL}(d,\\bR)$ is measurable. Consider a linear random dynamical system defined as (we use $\\Phi$ for the linear system, while reserving $\\varphi$ for nonlinear dynamics considered later)\n\t\\begin{equation*}\n\t\t\\Phi(t,\\omega)=A(\\theta^{t-1}\\omega)\\cdots A(\\theta\\omega)A(\\omega),\n\t\\end{equation*}\n\twhere the right-hand side is the product of a sequences of random matrices. In this setting, the behavior of the linear system $x_t=\\Phi(t,\\omega)x_0$ is well understood by the celebrated multiplicative ergodic theorem, also known as the Oseledets theorem, which we recall in Theorem~\\ref{thm: MET}. Such type of results was first established by V.I.{} Oseledets~\\cite{oseledets} and was further developed in many works such as \\cite{raghunathan, Ruelle-1979, Walters-93}.\n","sha256_utf8":"500b57180f8f1f6df46722d295466cb5590fdb80a139ceb80e1ea818e4aabf19"}]},{"id":"S006","semantic_type":"theorem","kind":"theorem","title":"乘法遍历定理","source_units":[{"start":18130,"end":20388,"start_line":193,"end_line":219,"raw_tex":"\\begin{theorem}[Multiplicative ergodic theorem, {\\cite[Theorem 3.4.1]{Arnold_RDS}}]\n\t\t\\label{thm: MET}Suppose that \n\t\t\\begin{equation*}\n\t\t\t\\left(\\log\\norm{A(\\cdot)}\\right)_+,\\ \\left(\\log\\norm{A(\\cdot)^{-1}}\\right)_+\\in L^1(\\Omega,\\mathcal{F},\\bP),\n\t\t\\end{equation*}\n\t\twhere we have used the short-hand $a_+:=\\max\\{a,0\\}$. Then there exists an $\\theta$-invariant $\\widetilde{\\Omega}\\in \\mathcal{F}$ with $\\bP(\\widetilde{\\Omega})=1$, such that the followings hold for any $\\omega\\in\\widetilde{\\Omega}$:\n\t\t\\begin{itemize}\n\t\t\t\\item[(i)] It holds that the limit\n\t\t\t\\begin{equation}\\label{conv: Lamb}\n\t\t\t\t\\Lambda(\\omega)=\\lim_{t\\rightarrow\\infty}\\left(\\Phi(t,\\omega)^{\\top}\\Phi(t,\\omega)\\right)^{1/2t},\n\t\t\t\\end{equation}\n\t\t\texists and is a positive definite matrix. Here $\\Phi(t, \\omega)^{\\top}$ denotes the transposition of the matrix (as $\\Phi(t, \\omega)$ is a linear map on $X$).\n\t\t\t\\item[(ii)] Suppose  $\\Lambda(\\omega)$ has $p(\\omega)$ distinct eigenvalues, which are ordered as $e^{\\lambda_1(\\omega)}>e^{\\lambda_2(\\omega)}>\\cdots> e^{\\lambda_{p(\\omega)}(\\omega)}$. Denote $V_i(\\omega)$ the corresponding eigenspace, with dimension $d_i(\\omega)$, for  $i=1,2,\\dots,p(\\omega)$. Then the functions $p(\\cdot)$, $\\lambda_i(\\cdot)$, and $d_i(\\cdot)$, $i=1,2,\\dots,p(\\cdot)$, are all measurable and  $\\theta$-invariant on $\\widetilde{\\Omega}$.\n\t\t\t\\item[(iii)] Set $W_i(\\omega)=\\bigoplus_{j\\geq i}V_j(\\omega),\\ i=1,2,\\dots,p(\\omega)$,  \n\t\t\tand $W_{p(\\omega)+1}(\\omega)=\\{0\\}$. Then it holds that\n\t\t\t\\begin{equation}\\label{conv: Lya-exp}\n\t\t\t\t\\lim_{t\\rightarrow\\infty}\\frac{1}{t}\\log\\norm{\\Phi(t,\\omega)x}=\\lambda_i(\\omega),\\quad\\forall\\ x\\in W_i(\\omega)\\backslash W_{i+1}(\\omega),\n\t\t\t\\end{equation}  \n\t\t\tfor $i=1,2,\\dots,p(\\omega)$. The maps $V(\\cdot)$ and $W(\\cdot)$ from $\\widetilde{\\Omega}$ to the Grassmannian manifold are measurable. \n\t\t\t\\item[(iv)] It holds that\n\t\t\t\\begin{equation*}\n\t\t\t\tW_i(\\theta\\omega)=A(\\omega)W_i(\\omega).\n\t\t\t\\end{equation*}\n\t\t\t\\item[(v)] When $(\\Omega,\\mathcal{F},\\bP,\\theta)$ is ergodic, i.e., every $B\\in\\mathcal{F}$ with $\\theta^{-1}B=B$ satisfies $\\bP(B)=0$ or $\\bP(B)=1$, the functions  \n\t\t\t$p(\\cdot)$, $\\lambda_i(\\cdot)$, and $d_i(\\cdot)$, $i=1,2,\\dots,p(\\cdot)$, are constant on $\\widetilde{\\Omega}$.\n\t\t\\end{itemize}\n\t\\end{theorem}","sha256_utf8":"01c47d7709ebaa95163798e274b82b72da8754d4f15f278b3212e99635a9c492"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"乘积概率空间与滤过","source_units":[{"start":22767,"end":23783,"start_line":239,"end_line":240,"raw_tex":"\t\t\\item{\\emph{Probability space}.} For each $t \\in \\bN$, denote  $(\\Omega_t,\\Sigma_t,\\bP_t)$ the usual probability space for the distribution $\\mathcal{U}_{\\{1,2,\\dots,n\\}}\\times \\mathcal{U}_{[\\alpha_{\\min},\\alpha_{\\max}]}$, where $\\mathcal{U}_{\\{1,2,\\dots,n\\}}$ and $\\mathcal{U}_{[\\alpha_{\\min},\\alpha_{\\max}]}$ are the uniform distributions on the set $\\{1,2,\\dots,n\\}$ and interval $[\\alpha_{\\min},\\alpha_{\\max}]$ respectively. \n\t\tLet $(\\Omega,\\mathcal{F},\\bP)$ be the product probability space of all $(\\Omega_t,\\Sigma_t,\\bP_t),\\ t\\in\\bN$. Denote $\\pi_t$ as the projection from $(\\Omega,\\mathcal{F},\\bP)$ onto $(\\Omega_t,\\Sigma_t,\\bP_t), \\ t \\in \\bN$. Thus, a sample $\\omega\\in\\Omega$ can be represented as a sequence  $((i_0,\\alpha_0),(i_1,\\alpha_1),\\dots)$, where $(i_t,\\alpha_t)=\\pi_t(\\omega),\\ t\\in\\bN$. Let $\\{\\mathcal{F}_t\\}_{t\\in \\bN}$ be the filtration defined by $$\\mathcal{F}_t=\\sigma\\left\\{(B_0\\times\\cdots\\times B_t)\\times \\left(\\prod_{j> t}\\Omega_j\\right):B_i\\in \\Sigma_i,\\ i=0,1,\\dots,t\\right\\}.$$\n","sha256_utf8":"55c6a8feaf27783989e42df40cf710532b1f33c9d0a6dda31eeef0564afef879"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"左移算子","source_units":[{"start":23786,"end":24227,"start_line":242,"end_line":247,"raw_tex":"\t\t\\item{\\emph{Metric dynamical system}.}\n\t\tThe metric dynamical system on $\\Omega$ is constructed by the (left) shifting operator $\\tau:\\Omega\\rightarrow\\Omega$ defined as \n\t\t\\begin{equation*}\n\t\t\t\\tau(\\omega)=\\tau(\\pi_0(\\omega),\\pi_1(\\omega),\\cdots) := (\\pi_1(\\omega),\\pi_2(\\omega),\\cdots),\n\t\t\\end{equation*}\n\t\twhich is clearly measurable and $\\bP$-preserving. The metric dynamical system is then given by $\\theta(t)=\\tau^t$ for $t\\in\\bN$. \n","sha256_utf8":"932e2558f24d5d577131b609f0216d20ed2ce06b6e2a93993974b3111f573536"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"算法的非线性余循环","source_units":[{"start":24230,"end":25594,"start_line":249,"end_line":266,"raw_tex":"\t\t\\item{\\emph{Random dynamical system}.}\n\t\tFor any $\\omega\\in\\Omega$ and $t\\in\\bN$, we define $\\phi(\\omega)$ to be a (nonlinear) map on $\\bR^d$ as \n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\phi(\\omega):\\bR^d&\\rightarrow\\quad\\quad\\quad \\bR^d\\\\\n\t\t\t\tx\\ &\\mapsto x-\\alpha e_i e_i^{\\top} \\nabla f(x),\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\twhere $(i,\\alpha)=\\pi_0(\\omega)$ is the first pair/element in the sequence $\\omega$, and we define the map $\\varphi(t,\\omega)$ via \n\t\t\\begin{equation*}\n\t\t\t\\varphi(t,\\omega)=\\phi(\\tau^{t-1}\\omega)\\circ\\cdots \\circ \\phi(\\tau\\omega)\\circ \\phi(\\omega), \\qquad \\text{for } t \\geq 1,\n\t\t\\end{equation*}\n\t\twhile $\\varphi(0, \\omega)$ is the identity operator. It is clear that $\\varphi(t,\\omega)$ satisfies the cocycle property \\eqref{cocycle} and hence defines a random dynamical system on $X=\\bR^d$ over $\\{\\tau^t\\}_{t\\in\\bN}$.\n\t\tThe iterate of Algorithm~\\ref{alg:random CGD} follows the random dynamical system as\n\t\t\\begin{equation*}\n\t\t\tx_t=\\phi(\\tau^{t-1}\\omega)x_{t-1}=\\cdots=\\phi(\\tau^{t-1}\\omega)\\circ\\cdots \\circ \\phi(\\tau\\omega)\\circ \\phi(\\omega)x_0=\\varphi(t,\\omega)x_0.\n\t\t\\end{equation*}\n\t\tIt can be seen that $\\{x_t\\}_{t\\in\\bN}$ is $\\{\\mathcal{F}_t\\}$-predictable, i.e., $x_t$ is $\\mathcal{F}_{t-1}$-measurable for any $t\\in\\bNp$, since $x_t$ is determined by samples $(i_0,\\alpha_0),(i_1,\\alpha_1),\\dots,(i_{t-1},\\alpha_{t-1})$.\n","sha256_utf8":"8fba8e88a94bd4d2371b22f80d938f28cf34c4e2ee41b6c5f4146c24bc42ec6a"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"临界点处的线性化","source_units":[{"start":25626,"end":26310,"start_line":271,"end_line":278,"raw_tex":"\tIn our analysis, we will use linearization of the dynamical system $\\varphi(t,\\omega)$ at a critical point $x^*$ of $f$. Without loss of generality, we assume $x^*=0$; otherwise we consider the system with state being $x-x^*$. The resulting linear system, which depends on $H=\\nabla^2 f(x^*)=(H_{ij})_{1\\leq i,j\\leq d}$, is given by (here and in the sequel, we use the superscript $H$ to indicate dependence on the matrix)\n\t\\begin{equation}\\label{linear: Phi}\n\t\t\\Phi^H(t,\\omega)=A^H(\\tau^{t-1}\\omega)\\cdots A^H(\\tau\\omega)A^H(\\omega),\n\t\\end{equation}\n\twhere \n\t\\begin{equation}\\label{linear: A}\n\t\tA^H(\\omega)=I-\\alpha e_i e_i^{\\top} H,\\quad (i,\\alpha)=\\pi_0(\\omega). \n\t\\end{equation}\n","sha256_utf8":"b9a8b62693bf82656738850036e99a8525b5d1d35ec7a7d1d5680fbbb40fe449"}]},{"id":"S011","semantic_type":"derived_fact","kind":"derived_fact","title":"遍历定理的适用条件","source_units":[{"start":26310,"end":27611,"start_line":279,"end_line":292,"raw_tex":"\tNote that $A^H(\\cdot)$ is bounded in $\\Omega$. \n\tWe know that $\\left(\\log\\norm{A^H(\\cdot)}\\right)_+$ is integrable. When $\\alpha< 1/|H_{ii}|$, \n\tthe matrix $A^H(\\omega)=I-\\alpha e_i e_i^{\\top} H$ is invertable, and the inverse is given explicitly by applying the Sherman-Morrison formula:\n\t\\begin{equation}\\label{inverse-AH}\n\t\t\\begin{split}\n\t\t\tA^H(\\omega)^{-1}&=\\left(I-\\alpha e_i e_i^{\\top} H\\right)^{-1} = I + \\frac{\\alpha e_i e_i^{\\top} H}{1 - \\alpha H_{ii}}. \n\t\t\\end{split}\n\t\\end{equation}\n\tIn particular, we have \n\t\\begin{equation}\n\t\t\\norm{A^H(\\omega)^{-1}} \\leq 1 + \\frac{\\alpha \\norm{H}}{1 - \\alpha |H_{ii}|}. \n\t\\end{equation}\n\tThus, if we take the maximal stepsize $\\alpha_{\\max}$ such that $\\alpha_{\\max} < 1/\\max_{1\\leq i\\leq d}|H_{ii}|$,  $\\norm{A^H(\\cdot)^{-1}}$ is bounded in $\\Omega$, and as a result $\\left(\\log\\norm{A^H(\\cdot)^{-1}}\\right)_+$ is also integrable. \n\tTherefore, the assumptions of Theorem~\\ref{thm: MET} hold. The shifting operator $\\tau$ is ergodic on $(\\Omega,\\mathcal{F},\\bP)$ by Kolmogorov's $0$--$1$ law. Then Theorem~\\ref{thm: MET} applies for $\\theta=\\tau$ with $p^H(\\cdot)$, $\\lambda_i^H(\\cdot)$, and $d_i^H(\\cdot)$ all being a.e. constant. For any $\\omega\\in\\widetilde{\\Omega}$ that is the set in Theorem~\\ref{thm: MET} satisfying $\\bP(\\widetilde{\\Omega})=1$, \n","sha256_utf8":"205e24871f04c24bee76d79ddc673d5af88ce5d7fe7b4529be1d8bf6d74f047c"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"不稳定与中心稳定子空间","source_units":[{"start":27611,"end":28314,"start_line":293,"end_line":301,"raw_tex":"\twe denote\n\t\\begin{equation}\\label{W_pm}\n\t\tW_+^H(\\omega)=\\bigoplus_{\\lambda_i>0}V_i^H(\\omega),\\ \\text{and}\\ W_-^H(\\omega)=\\bigoplus_{\\lambda_i\\leq 0}V_i^H(\\omega).\n\t\\end{equation}\n\tThen the following invariant property holds \n\t\\begin{equation*}\n\t\tW_-^H(\\tau\\omega)=A^H(\\omega)W_-^H(\\omega).\n\t\\end{equation*}\n\tNote that $W_-^H(\\omega)$ works as a center-stable subspace. That is, for any $x\\in W_-^H(\\omega)$ and any $\\epsilon>0$, it holds that $\\norm{\\Phi(t,\\omega)x}\\leq e^{t\\epsilon}$, for sufficiently large $t$, and for $x\\notin W_-^H(\\omega)$, $\\norm{\\Phi(t,\\omega)x}$ grows exponentially as $t\\rightarrow\\infty$ with rate greater than $\\min_{\\lambda_i>0}\\lambda_i -\\epsilon$ for any $\\epsilon>0$.\n","sha256_utf8":"38c3cc0f9abbe911469ab9d98bb636f181717eaf3e98e95a9705de2fe8a89abf"}]},{"id":"S013","semantic_type":"assumption","kind":"assumption","title":"二阶光滑与海森有界","source_units":[{"start":28494,"end":28709,"start_line":308,"end_line":309,"raw_tex":"\\begin{assumption}\\label{Assump: Hess_bdd} $f \\in C^2(\\bR^d)$ and the Hessian $\\nabla^2 f$ is uniformly bounded, i.e., there exists $M>0$ such that $\\norm{\\nabla^2 f(x)}\\leq M$ for all $x\\in\\bR^d$.\n\t\\end{assumption}","sha256_utf8":"1e4caacea1942e28a9caadca1a73f886dd6a63937142be5dcca9121116d3d824"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"严格鞍点集合","source_units":[{"start":28712,"end":29253,"start_line":311,"end_line":316,"raw_tex":"\tAn optimization algorithm is expected to converge, under some reasonable assumptions, to a critical point of $f$ where the gradient vanishes. Our aim is to further characterize the possible limits of the algorithm iterates. For this purpose, we distinguish $\\calA$, the set of all strict saddle points (including local maxima with non-degenerate Hessian) of $f$:\n\t\\begin{equation*}\n\t\t\\calA := \\{x\\in\\bR^d:\\nabla f(x)=0,\\\n\t\t\\lambda_{\\min}(\\nabla^2 f(x))<0\\},\n\t\\end{equation*}\n\twhere we use the subscript $s$ to emphasize that it is strict. \n","sha256_utf8":"e963ad05f10574dbb524f82b225b19d58bfcc62fe5046e3b2ccd4f37a5a21c5b"}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","title":"严格鞍点非退化","source_units":[{"start":30349,"end":30600,"start_line":327,"end_line":329,"raw_tex":"\\begin{assumption}\\label{Assump: non-degenerate}\n\t\tFor every $x^*\\in \\calA$, $\\nabla^2 f(x^*)$ is non-degenerate, i.e., $x^*$ is a non-degenerate critical point of $f$ in the sense that any eigenvalue of $\\nabla^2 f(x^*)$ is nonzero.\n\t\\end{assumption}","sha256_utf8":"935e17776ddfaebe2a87b22af6125fba1c2ffb97913f35abbfeaffec8716ce83"}]},{"id":"S016","semantic_type":"assumption","kind":"assumption","title":"坐标的不稳定投影非零","source_units":[{"start":31038,"end":31358,"start_line":331,"end_line":333,"raw_tex":"\\begin{assumption}\\label{Assump: non-zero-proj}\n\t\tFor every $x^*\\in \\calA$, it holds that $\\mathcal{P}_+^H(\\omega) e_i\\neq 0$, for every $i\\in\\{1,2,\\dots,d\\}$ and almost every $\\omega\\in\\Omega$, where $\\mathcal{P}_+^H(\\omega)$ is the orthogonal projection onto $W_+^H(\\omega)$ with $H=\\nabla^2 f(x^*)$.\n\t\\end{assumption}","sha256_utf8":"fe479be9d5b12ec318996b1de61fa8135d720e4c2af914471071406468c27f51"}]},{"id":"S017","semantic_type":"definition","kind":"definition","title":"收敛到鞍点的事件","source_units":[{"start":32330,"end":32968,"start_line":341,"end_line":349,"raw_tex":"\tGiven an initial guess $x_0$, the behavior of the algorithm, in particular, the limit of $x_t$, depends on the particular sample $\\omega \\in \\Omega$.\n\tFor any $x^*\\in\\calA$, we denote the set of all $\\omega$ such that the algorithm starting at $x_0$ would converge to $x^*$:\n\t\\begin{equation*}\n\t\t\\calC(x^*,x_0) :=\\left\\{\\omega\\in\\Omega:\\lim_{t\\rightarrow\\infty}x_t=\\lim_{t\\rightarrow\\infty}\\varphi(t,\\omega)x_0=x^*\\right\\}.\n\t\\end{equation*}\n\tWe further define the set $\\calC(\\calA,x_0)$ as the union of all $\\calC(x^*,x_0)$ over $x^*\\in\\calA$,\n\t\\begin{equation*}\n\t\t\\calC(\\calA,x_0):=\\bigcup_{x^*\\in\\calA}\\calC(x^*,x_0).\n\t\\end{equation*}\n","sha256_utf8":"5345a4bff28f3052e5e1e8de7506928fc3cc10c35aad92d014c5a800d7ca9e0b"}]},{"id":"S018","semantic_type":"problem_setting","kind":"problem_setting","title":"不定二次型动力系统","source_units":[{"start":38129,"end":38235,"start_line":394,"end_line":394,"raw_tex":"\tWe will first study the linear dynamical system, for which we assume the objective function is given by \n","sha256_utf8":"3f96632409c50293478ebaa88f0a203beda6e2d2b40fe8aef244e4fda4283fd0"},{"start":38236,"end":38324,"start_line":395,"end_line":397,"raw_tex":"\\begin{equation}\\label{obj: quadratic}\n\t\tf^H(x)=\\frac{1}{2}x^{\\top} H x,\n\t\\end{equation}","sha256_utf8":"8244bc2c4ab1b9fad06cc2f6a6f53912c77455b4f5f91cc1a2330a82fc7f0edf"},{"start":38325,"end":38752,"start_line":398,"end_line":402,"raw_tex":"\twhere $H$ is a symmetric matrix in $\\bR^{d\\times d}$ with at least one negative eigenvalue. In this case, the coordinate descent algorithm is given by \n\t\\begin{equation*}\n\t\tx_{t+1}=\\left(I-\\alpha_t e_{i_t} e_{i_t}^{\\top} H\\right)x_t,\n\t\\end{equation*}\n\twhich corresponds to the linear dynamical system  $\\Phi^H(t,\\omega)$ with single step map $A^H(\\omega)$,  defined in \\eqref{linear: Phi} and \\eqref{linear: A} respectively. \n","sha256_utf8":"ad36ffb708d68ea24d1228610c6608b1b29137b4bebf16a4ad609fe6975afecb"}]},{"id":"S019","semantic_type":"problem_setting","kind":"problem_setting","title":"鞍点平移到原点","source_units":[{"start":47758,"end":48188,"start_line":535,"end_line":535,"raw_tex":"\tIn this subsection, we study the behavior of the nonlinear dynamical system near a strict saddle point of $f$, which without loss of generality can be assumed to be $x^*=0$. As mentioned above, in a small neighborhood of $x^*$, while it is not possible to control the difference between nonlinear and linear systems for infinite time,  the nonlinear system can be approximated by the linear system during a finite time horizon. \n","sha256_utf8":"3b7933522103958cd601901c15d3e39a7a39f40ff9443a36b04ca6a24c20bf34"}]},{"id":"S020","semantic_type":"notation","kind":"notation","title":"正指数间隙与失败概率","source_units":[{"start":51129,"end":52017,"start_line":557,"end_line":566,"raw_tex":"\tWe first set the small constant $\\epsilon$ in Theorem~\\ref{thm: local-growth} which controls the failure probability of the amplification bound. Let $\\lambda_1>\\lambda_2>\\cdots>\\lambda_p$ be the Lyapunov exponents of the linearized system at $x^*=0$. We set\n\t\\begin{equation}\\label{lambplus-gamma}\n\t\t\\lambda_+=\\min_{\\lambda_i>0}\\lambda_i\\quad\\text{and}\\quad 0<\\gamma<\\frac{1}{2}\\min\\left\\{\\min_{1\\leq i< p}|\\lambda_i-\\lambda_{i+1}|,\\lambda_+\\right\\}. \n\t\\end{equation}\n\tNote that $\\gamma<\\lambda_+$.\n\tLet $\\epsilon_*\\in(0,1/6)$ be sufficiently small such that\n\t\\begin{equation}\\label{def: eps}\n\t\t(1-6\\epsilon)(\\lambda_+-\\gamma)+6\\epsilon \\cdot \\log(1-M\\alpha_{\\max})>0,\\quad \\forall\\ \\epsilon\\in(0,\\epsilon_*). \n\t\\end{equation}\n\tThe reason for such choice will become clear later (see \\eqref{large S2}). For the rest of the section, we will consider a fixed $\\epsilon\\in(0,\\epsilon_*)$. \n","sha256_utf8":"8eba350a0a946124c0c923e2ee624368863b7568ffdf0dae69409eaefc7fcd78"}]},{"id":"S021","semantic_type":"notation","kind":"notation","title":"有限块矩阵与奇异投影","source_units":[{"start":53666,"end":54879,"start_line":595,"end_line":604,"raw_tex":"\tWe now carry out the amplification part of the finite block analysis \n\tfor the linearized dynamics at $x^* = 0$. To simplify expressions in the following, for $t_1<t_2$, we introduce the short-hand notation\n\t\\begin{equation*}\n\t\t(i,\\alpha)_{t_1:t_2-1}=\\big((i_{t_1},\\alpha_{t_1}),\\dots,(i_{t_2-1},\\alpha_{t_2-1})\\big)\\in\\Omega_{t_1}\\times\\cdots\\times\\Omega_{t_2-1},\n\t\\end{equation*}\n\tand the finite time transition matrix (i.e., composition of linear maps)\n\t\\begin{equation*}\n\t\t\\Phi^H\\bigl((i,\\alpha)_{t_1:t_2-1}\\bigr)=(I-\\alpha_{t_2-1}e_{i_{t_2-1}}e_{i_{t_2-1}}^{\\top} H)\\cdots (I-\\alpha_{t_1} e_{i_{t_1}}e_{i_{t_1}}^{\\top} H).\n\t\\end{equation*}\n\tRecall that $(\\Omega_t,\\Sigma_t,\\bP_t)$ is the probability space for $\\mathcal{U}_{\\{1,2,\\dots,d\\}}\\times \\mathcal{U}_{[\\alpha_{\\min},\\alpha_{\\max}]}$ for $t\\in\\bN$. We also denote $\\mathcal{P}_+^H\\bigl((i,\\alpha)_{t_1:t_2-1}\\bigr)$ as the projection operator onto the subspace spanned by the right singular vectors of $\\Phi^H\\bigl((i,\\alpha)_{t_1:t_2-1}\\bigr)$ corresponding to $d_+$ largest singular values, where $d_+=\\sum_{\\lambda_i>0}d_i$ and $d_i$ is the dimension of the $i$-th eigenspace as in Theorem~\\ref{thm: MET} (ii) for the linearized system at $x^*$.\n","sha256_utf8":"b5a6419f5916712b87a16246a376763c4c7322b2423f1c93e2e3c82b7abaa26b"}]},{"id":"S022","semantic_type":"definition","kind":"definition","title":"良好有限块事件","source_units":[{"start":57271,"end":59478,"start_line":645,"end_line":666,"raw_tex":"\tWe will then be able to handle $\\norm{\\mathcal{P}_+^H((i,\\alpha)_{0:S-1})e_j}$ using Lemma~\\ref{lem: Omega1} and the closeness between $\\left(\\Phi^H(S,\\omega)^{\\top} \\Phi^H(S,\\omega)\\right)^{1/2S}$ with $\\Lambda(\\omega)$ as the former converges to the latter as $S \\to \\infty$ by Theorem~\\ref{thm: MET}. \n\tMore precisely, denote the singular values of $X\\in\\bR^{d\\times d}$ by $s_1(X)\\geq s_2(X)\\geq \\cdots\\geq s_d(X)$. Then for $S\\in\\bNp$ sufficiently large,\n\twe have \n\t\\begin{equation}\\label{close: singular value}\n\t\t\\left|\\frac{1}{S}\\log s_j\\left(\\Phi^H(S,\\omega)\\right)- \\lambda_{\\mu(j)}\\right|=\\left|\\frac{1}{S}\\log s_j\\left(\\Phi^H\\bigl((i,\\alpha)_{0:S-1}\\bigr)\\right)- \\lambda_{\\mu(j)}\\right|\\leq \\gamma,\n\t\\end{equation}\n\tfor every $j\\in\\{1,2,\\dots,d\\}$, where $\\lambda_1>\\lambda_2>\\dots>\\lambda_p$ are the Lyapunov exponents from Theorem~\\ref{thm: MET}, $\\gamma$ is given by \\eqref{lambplus-gamma}, and the map $\\mu:\\{1,2,\\dots,d\\}\\rightarrow\\{1,2,\\dots,p\\}$ satisfies that $\\mu(j)=i$ if and only if $d_1+\\cdots +d_{i-1}<j\\leq d_1+\\cdots +d_i$, so $\\mu$ corresponds the index for the singular values with that of the Lyapunov exponents. \n\tMoreover,\n\tthe convergence also implies that \n\t\\begin{equation*}\n\t\t\\norm{\\mathcal{P}_+^H(S,\\omega)-\\mathcal{P}_+^H(\\omega)}\\leq\\frac{\\delta}{2},\n\t\\end{equation*}\n\tfor sufficiently large $S$, which then leads to\n\t\\begin{equation}\\label{close: singular space}\n\t\t\\norm{\\mathcal{P}^H_+\\left(S,\\omega\\right)e_j}=\\norm{\\mathcal{P}^H_+\\left((i,\\alpha)_{0:S-1}\\right)e_j}\\geq \\frac{\\delta}{2},\n\t\\end{equation}\n\tfor every $j\\in\\{1,2,\\dots,d\\}$,\n\twhere $\\mathcal{P}^H_+(S,\\omega)$ is the projection operator onto the subspace spanned by the right singular vectors of $\\Phi^H(S,\\omega)$ corresponding to $d_+$ largest singular values. Let \n\t\\begin{equation}\n\t\t\\Omega^S=\\bigl\\{(i,\\alpha)_{0:S-1}\\in \\Omega_0\\times\\cdots\\times\\Omega_{S-1}:\\eqref{close: singular value}\\text{ and }\\eqref{close: singular space}\\text{ hold}\\bigr\\}.\n\t\\end{equation}\n\tThe following lemma states that $\\Omega^S$ has high probability for sufficiently large $S$, where with slight abuse of notation, we write $\\bP(\\Omega^S)=\\bP\\left(\\Omega^S\\times\\left(\\bigtimes_{t\\geq S}\\Omega_t\\right)\\right)$.\n","sha256_utf8":"f4e1192d049dae11014692daf5c1ff553c912478e080d4068247bce53c8477f6"}]},{"id":"S023","semantic_type":"derived_fact","kind":"derived_fact","title":"选定步长的独立性","source_units":[{"start":61773,"end":62240,"start_line":707,"end_line":708,"raw_tex":"\tNote that $\\alpha_{\\varrho_t-1}\\sim\\mathcal{U}_{[\\alpha_{\\min},\\alpha_{\\max}]}$ is independent of $\\mathcal{F}_{\\varrho_t-2}$, $i_{\\varrho_t-1}$, and $(i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}$. \n\tThe next lemma shows that with high probability, the choice of $\\alpha_{\\varrho_t-1}$ will lead to a nontrivial orthogonal projection of $x_{\\varrho_t}$ onto the unstable subspace of $\\Phi^H(S,\\tau^{\\varrho_t}\\omega)=\\Phi^H\\bigl((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\bigr)$.\n","sha256_utf8":"5fea48a65bdbf19a124c6efd8388daee059d4838da32f567db5460921d90d89f"}]},{"id":"S024","semantic_type":"theorem","kind":"theorem","title":"几乎必然不收敛到严格鞍点","source_units":[{"start":33342,"end":33695,"start_line":351,"end_line":357,"raw_tex":"\\begin{maintheorem}\\label{thm: main}\n\t\tSuppose that Assumptions~\\ref{Assump: Hess_bdd}, \\ref{Assump: non-degenerate}, and \\ref{Assump: non-zero-proj} hold and that $0 < \\alpha_{\\min} < \\alpha_{\\max} < 1/M$, then  \n\t\tfor any $x_0\\in \\bR^d\\backslash\\calA$, it holds that\n\t\t\\begin{equation*}\n\t\t\t\\bP(\\calC(\\calA,x_0))=0.\n\t\t\\end{equation*}\n\t\\end{maintheorem}","sha256_utf8":"17b6ad0cfc9c50927d80708e6b895748eb00370ff1e96ad1acde29638a16dee1"}]},{"id":"S025","semantic_type":"proof","kind":"proof","title":"主定理完整证明","source_units":[{"start":77075,"end":77539,"start_line":938,"end_line":940,"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm: main}]\n\t\tAssumption~\\ref{Assump: non-degenerate} guarantees that, in a small neighborhood of $x^*$, the gradient $\\nabla f(x)=\\nabla^2 f(x^*)(x-x^*)+o(\\norm{x-x^*})$ is non-vanishing as long as $x\\neq x^*$, which implies that $x^*$ is an isolated stationary point. Therefore, $\\calA$ is countable. Then Theorem~\\ref{thm: main} follows directly from Theorem~\\ref{thm: single-pt} and the countability of $\\calA$.\n\t\\end{proof}","sha256_utf8":"7b532dd451d2a0f1a2216767a37e11770c6a7a1d41cc7ac4a08560cc2e2d8dd9"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"鞍点可数与零测事件并集","source_units":[{"start":77123,"end":77527,"start_line":939,"end_line":939,"raw_tex":"\t\tAssumption~\\ref{Assump: non-degenerate} guarantees that, in a small neighborhood of $x^*$, the gradient $\\nabla f(x)=\\nabla^2 f(x^*)(x-x^*)+o(\\norm{x-x^*})$ is non-vanishing as long as $x\\neq x^*$, which implies that $x^*$ is an isolated stationary point. Therefore, $\\calA$ is countable. Then Theorem~\\ref{thm: main} follows directly from Theorem~\\ref{thm: single-pt} and the countability of $\\calA$.\n","sha256_utf8":"2d0af1ece261d2e5cae63ac30425877cca79a8c0f32a061eebc718e70bcbcfa4"}]},{"id":"S027","semantic_type":"proposition","kind":"proposition","title":"最大李雅普诺夫指数为正","source_units":[{"start":29900,"end":30170,"start_line":320,"end_line":324,"raw_tex":"\\begin{proposition}\\label{prop: posi-lamb1}\n\t\t%For any $x^*\\in\\calA$, if \n\t\tLet $H$ have a negative eigenvalue and\n\t\t$0<\\alpha_{\\min}<\\alpha_{\\max}<1/\\max_{1\\leq i\\leq d}|H_{ii}|$, then the largest Lyapunov exponent of $\\Phi^H(t,\\omega)$ is positive. \n\t\\end{proposition}","sha256_utf8":"9a726910fbfbb8cf39c3f2882bba16c805613be7b655bcf1f1c0824d53f9abf2"}]},{"id":"S028","semantic_type":"proof","kind":"proof","title":"正指数命题完整证明","source_units":[{"start":45742,"end":47694,"start_line":505,"end_line":531,"raw_tex":"\\begin{proof}[Proof of Proposition~\\ref{prop: posi-lamb1}] It suffices to show that for almost every $\\omega\\in\\Omega$ there exist some $x_0\\in \\bR^d$, $\\epsilon>0$, and $T>0$, such that $x_t=\\Phi(t,\\omega)x_0$ satisfies $\\norm{x_t}\\geq e^{\\epsilon t}$ for any $t>T$. Let $x_0$ be an eigenvector corresponding to a negative eigenvalue of $H$. Then it holds that $f^H(x_0)< 0$. Consider a fixed $m\\geq d$. For any $k\\in\\bN$, set $I_k=1$ if $\\{1,2,\\dots,d\\}=\\{i_{km},i_{km+1},\\dots,i_{km+m-1}\\}$ \n\t\tand $I_k=0$ otherwise. We can see that the random variables, $I_0,I_1,I_2,\\dots$, are independent and identically distributed with $\\bE I_0=\\bP(I_0=1)\\in(0,1)$. By Proposition~\\ref{prop: linear_decay}, we obtain that\n\t\t\\begin{equation*}\n\t\t\tf^H(x_{(k+1)m})\\leq\\begin{cases}\n\t\t\t\t(1+c)f^H(x_{km}),&\\text{if }I_k=1,\\\\\n\t\t\t\tf^H(x_{km}),&\\text{if }I_k=0,\n\t\t\t\\end{cases}\n\t\t\\end{equation*}\t\n\t\twhere $c$ is the constant from Proposition~\\ref{prop: linear_decay}.\n\t\tTherefore, %\\jl{missing $1/2$}\n\t\t\\begin{equation*}\n\t\t\t\\frac{\\lambda_{\\min}(H)}{2}\\norm{x_{km}}^2\\leq f^H(x_{km})\\leq(1+c)^{\\sum_{j=0}^{k-1} I_j}f^H(x_0),\n\t\t\\end{equation*}\n\t\twhich implies that\n\t\t\\begin{equation}\\label{norm:xkm}\n\t\t\t\\norm{x_{km}}\\geq \\left(\\frac{2f^H(x_0)}{\\lambda_{\\min}(H)}\\right)^{1/2}\\cdot (1+c)^{\\frac{1}{2}\\sum_{j=0}^{k-1} I_j}.\n\t\t\\end{equation}\n\t\tNote that $\\bE |I_0|=\\bE I_0<\\infty$. The strong law of large number suggests that for almost every $\\omega\\in\\Omega$, there exists some $K$, such that for all $k \\geq K$\n\t\t\\begin{equation}\\label{SLLN}\n\t\t\t\\sum_{j=0}^{k-1} I_j\\geq \\frac{\\bE I_0}{2}k.\n\t\t\\end{equation}\n\t\tCombining \\eqref{norm:xkm} and \\eqref{SLLN}, we arrive at\n\t\t\\begin{equation*}\n\t\t\t\\norm{x_{km}}\\geq \\left(\\frac{2f^H(x_0)}{\\lambda_{\\min}(H)}\\right)^{1/2}\\cdot (1+c)^{\\frac{\\bE I_0}{4m}\\cdot km}.\n\t\t\\end{equation*}\n\t\tNoticing that $(1+c)^{\\frac{\\bE I_0}{4m}}$ is greater than $1$, $\\norm{x_{km}}$ grows exponentially in $km$ and we complete the proof.\n\t\\end{proof}","sha256_utf8":"83bd23b4b13160a77b83f169b859b7165544c5568720980ffdcf9d0e1e45a138"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"覆盖坐标的独立块与能量衰减","source_units":[{"start":45742,"end":46692,"start_line":505,"end_line":513,"raw_tex":"\\begin{proof}[Proof of Proposition~\\ref{prop: posi-lamb1}] It suffices to show that for almost every $\\omega\\in\\Omega$ there exist some $x_0\\in \\bR^d$, $\\epsilon>0$, and $T>0$, such that $x_t=\\Phi(t,\\omega)x_0$ satisfies $\\norm{x_t}\\geq e^{\\epsilon t}$ for any $t>T$. Let $x_0$ be an eigenvector corresponding to a negative eigenvalue of $H$. Then it holds that $f^H(x_0)< 0$. Consider a fixed $m\\geq d$. For any $k\\in\\bN$, set $I_k=1$ if $\\{1,2,\\dots,d\\}=\\{i_{km},i_{km+1},\\dots,i_{km+m-1}\\}$ \n\t\tand $I_k=0$ otherwise. We can see that the random variables, $I_0,I_1,I_2,\\dots$, are independent and identically distributed with $\\bE I_0=\\bP(I_0=1)\\in(0,1)$. By Proposition~\\ref{prop: linear_decay}, we obtain that\n\t\t\\begin{equation*}\n\t\t\tf^H(x_{(k+1)m})\\leq\\begin{cases}\n\t\t\t\t(1+c)f^H(x_{km}),&\\text{if }I_k=1,\\\\\n\t\t\t\tf^H(x_{km}),&\\text{if }I_k=0,\n\t\t\t\\end{cases}\n\t\t\\end{equation*}\t\n\t\twhere $c$ is the constant from Proposition~\\ref{prop: linear_decay}.\n","sha256_utf8":"e3b9f60584abaa01f600d0a3487344afa373f305b87957fbaf0b8046af696adb"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"能量衰减推出范数下界","source_units":[{"start":46692,"end":47061,"start_line":514,"end_line":521,"raw_tex":"\t\tTherefore, %\\jl{missing $1/2$}\n\t\t\\begin{equation*}\n\t\t\t\\frac{\\lambda_{\\min}(H)}{2}\\norm{x_{km}}^2\\leq f^H(x_{km})\\leq(1+c)^{\\sum_{j=0}^{k-1} I_j}f^H(x_0),\n\t\t\\end{equation*}\n\t\twhich implies that\n\t\t\\begin{equation}\\label{norm:xkm}\n\t\t\t\\norm{x_{km}}\\geq \\left(\\frac{2f^H(x_0)}{\\lambda_{\\min}(H)}\\right)^{1/2}\\cdot (1+c)^{\\frac{1}{2}\\sum_{j=0}^{k-1} I_j}.\n\t\t\\end{equation}\n","sha256_utf8":"a050e5aa4ba764bb6b1fe0cb74af524e6b3a4df4a5eefa359a0f99cafb530ca6"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"大数定律推出指数增长","source_units":[{"start":47061,"end":47682,"start_line":522,"end_line":530,"raw_tex":"\t\tNote that $\\bE |I_0|=\\bE I_0<\\infty$. The strong law of large number suggests that for almost every $\\omega\\in\\Omega$, there exists some $K$, such that for all $k \\geq K$\n\t\t\\begin{equation}\\label{SLLN}\n\t\t\t\\sum_{j=0}^{k-1} I_j\\geq \\frac{\\bE I_0}{2}k.\n\t\t\\end{equation}\n\t\tCombining \\eqref{norm:xkm} and \\eqref{SLLN}, we arrive at\n\t\t\\begin{equation*}\n\t\t\t\\norm{x_{km}}\\geq \\left(\\frac{2f^H(x_0)}{\\lambda_{\\min}(H)}\\right)^{1/2}\\cdot (1+c)^{\\frac{\\bE I_0}{4m}\\cdot km}.\n\t\t\\end{equation*}\n\t\tNoticing that $(1+c)^{\\frac{\\bE I_0}{4m}}$ is greater than $1$, $\\norm{x_{km}}$ grows exponentially in $km$ and we complete the proof.\n","sha256_utf8":"76bd996ec3fdb15a874be785f8c33e0f18bd2b838350cfa875d9606d525dd8ed"}]},{"id":"S032","semantic_type":"proposition","kind":"proposition","title":"完整坐标块的能量下降","source_units":[{"start":39709,"end":40249,"start_line":406,"end_line":414,"raw_tex":"\\begin{proposition}\\label{prop: linear_decay}\n\t\tLet $m\\geq d$ be fixed. For the objective function \\eqref{obj: quadratic} with $\\lambda_{\\min}(H)<0$, suppose \n\t\tthat $0<\\alpha_{\\min}<\\alpha_{\\max}<1/\\max_{1\\leq i\\leq d}|H_{ii}|$, there exists $c\\in(0,1)$ depending on $m$, $H$, $\\alpha_{\\min}$, and $\\alpha_{\\max}$, \n\t\tsuch that \n\t\t\\begin{equation*}\n\t\t\tf^H(x_{t+m})-f^H(x_t)\\leq c\\, f^H(x_t),\n\t\t\\end{equation*}\n\t\tholds as long as $f^H(x_t)<0$ and $\\{1,2,\\dots,d\\}=\\{i_t,i_{t+1},\\dots,i_{t+m-1}\\}$ (in the sense of sets). \n\t\\end{proposition}","sha256_utf8":"6ef21af3194063a11aa3669b29941aa369d7d53fe6ac5130157cc6b0ada6fbf0"}]},{"id":"S033","semantic_type":"proof","kind":"proof","title":"块能量下降完整证明","source_units":[{"start":40473,"end":41855,"start_line":421,"end_line":438,"raw_tex":"\\begin{proof} \n\t\tWithout loss of generality, we assume that $t=0$. \n\t\tDue to the choice of $\\alpha_{\\max}$, we have the following simple non-increasing property for any $t'\\in\\bN$:\n\t\t\\begin{equation}\\label{decay: quad}\n\t\t\t\\begin{split}\n\t\t\t\tf^H(x_{t'+1})&=\\frac{1}{2}x_{t'+1}^{\\top} H x_{t'+1}\\\\\n\t\t\t\t&=\\frac{1}{2}x_{t'}^{\\top} \\left(I-\\alpha_{t'} e_{i_{t'}} e_{i_{t'}}^{\\top} H\\right)^{\\top} H \\left(I-\\alpha_{t'} e_{i_{t'}} e_{i_{t'}}^{\\top} H\\right) x_{t'}\\\\\n\t\t\t\t&=f^H(x_{t'})-\\alpha_{t'} \\left(e_{i_{t'}}^{\\top} H x_{t'}\\right)^2+\\frac{1}{2}\\alpha_t^2 e_{i_{t'}}^{\\top} H e_{i_{t'}} \\left(e_{i_{t'}}^{\\top} H x_{t'}\\right)^2\\\\\n\t\t\t\t&\\leq f^H(x_{t'})-\\frac{\\alpha_{t'}}{2} \\left(e_{i_{t'}}^{\\top} H x_{t'}\\right)^2.\n\t\t\t\\end{split}\n\t\t\\end{equation}\n\t\t\n\t\tWrite $x_0=y^*+y_0$ with $y^*\\in\\ker(H)$ and $y_0\\in\\ran(H)$. Let \n\t\t\\begin{equation}\n\t\t\t\\label{def: y}\n\t\t\ty_{t'+1}=y_{t'}-\\alpha_{t'} e_{i_{t'}} e_{i_{t'}}^{\\top} H y_{t'}, \\qquad t'=0,1,\\dots,m-1.\n\t\t\\end{equation} \n\t\tThen $x_{t'}=y^*+y_{t'}$ holds for any $t'=0,1,\\dots,m$. Using \\eqref{decay: quad}, to give an upper bound for $f^H(x_{t+m})-f^H(x_t)$, we would like a nontrivial lower bound for $\\alpha_{t'}\\bigl(e_{i_{t'}}^{\\top} H x_{t'}\\bigr)^2=\\alpha_{t'}\\bigl(e_{i_{t'}}^{\\top} H y_{t'}\\bigr)^2$ for some $t'\\in\\{t,t+1,\\dots,t+m-1\\}$, which is guaranteed by Lemma~\\ref{lem: linear_decay}, whose proof will be postponed. \n","sha256_utf8":"730b89b763e5102f079f5a6dd2da5923b4986e21392ef91c4240793b99c7484b"},{"start":42522,"end":43563,"start_line":449,"end_line":465,"raw_tex":"\t\tAssuming the lemma, there exists $T\\in\\{0,1,\\dots,m-1\\}$ that \n\t\t\\begin{equation*}\n\t\t\t\\alpha_T\\left\\vert e_{i_{T}}^{\\top} H y_{T}\\right\\vert\\geq \\delta\\norm{y_{T}},\n\t\t\\end{equation*}\n\t\twith a fixed $\\delta>0$ satisfying \\eqref{def:delta}. We can further constrain that $\\frac{\\delta^2}{\\alpha_{\\min}\\sigma_{\\max}(H)}<1$. Thus, we have\n\t\t\\begin{equation*}\\label{decay esti}\n\t\t\t\\begin{split}\n\t\t\t\tf^H(x_m)&\\leq f^H (x_{T+1})\\leq f^H(x_T)- \\frac{\\alpha_T}{2}\\left(e_{i_{T}}^{\\top} H x_{T}\\right)^2\\leq f^H (x_{T}) -\\frac{\\delta^2}{2\\alpha_T}\\norm{y_{T}}^2,\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\twhich combined with $\\sigma_{\\max}(H)\\norm{y_{T}}^2\\geq -y_{T}^{\\top} H y_{T}=-x_{T}^{\\top} H x_{T}=-2f^H(x_T)$,\n\t\tyields that \n\t\t\\begin{equation*}\n\t\t\tf^H (x_m) \\leq\\left( 1 +\\frac{\\delta^2}{\\alpha_T \\sigma_{\\max}(H)}\\right)f^H (x_{T}) \\leq\\left( 1 +\\frac{\\delta^2}{\\alpha_T \\sigma_{\\max}(H)}\\right)f^H (x_0). \n\t\t\\end{equation*}\n\t\tSet $c=\\frac{\\delta^2}{\\alpha_{\\max} \\sigma_{\\max}(H)}$ and we get that $f^H(x_m)-f^H(x_0)\\leq c f^H(x_0)$.\n\t\\end{proof}","sha256_utf8":"0b3c4917d32b0207e29e02f51c399609d2a421564c18786b8c5db14703181acb"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"二次能量单步下降","source_units":[{"start":40488,"end":41221,"start_line":422,"end_line":431,"raw_tex":"\t\tWithout loss of generality, we assume that $t=0$. \n\t\tDue to the choice of $\\alpha_{\\max}$, we have the following simple non-increasing property for any $t'\\in\\bN$:\n\t\t\\begin{equation}\\label{decay: quad}\n\t\t\t\\begin{split}\n\t\t\t\tf^H(x_{t'+1})&=\\frac{1}{2}x_{t'+1}^{\\top} H x_{t'+1}\\\\\n\t\t\t\t&=\\frac{1}{2}x_{t'}^{\\top} \\left(I-\\alpha_{t'} e_{i_{t'}} e_{i_{t'}}^{\\top} H\\right)^{\\top} H \\left(I-\\alpha_{t'} e_{i_{t'}} e_{i_{t'}}^{\\top} H\\right) x_{t'}\\\\\n\t\t\t\t&=f^H(x_{t'})-\\alpha_{t'} \\left(e_{i_{t'}}^{\\top} H x_{t'}\\right)^2+\\frac{1}{2}\\alpha_t^2 e_{i_{t'}}^{\\top} H e_{i_{t'}} \\left(e_{i_{t'}}^{\\top} H x_{t'}\\right)^2\\\\\n\t\t\t\t&\\leq f^H(x_{t'})-\\frac{\\alpha_{t'}}{2} \\left(e_{i_{t'}}^{\\top} H x_{t'}\\right)^2.\n\t\t\t\\end{split}\n\t\t\\end{equation}\n","sha256_utf8":"0362fd47e8fc8488b15dccef363d8450f0ed83b21881c071d4c4b8d908642af8"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"核空间分解与辅助迭代","source_units":[{"start":41224,"end":41855,"start_line":433,"end_line":438,"raw_tex":"\t\tWrite $x_0=y^*+y_0$ with $y^*\\in\\ker(H)$ and $y_0\\in\\ran(H)$. Let \n\t\t\\begin{equation}\n\t\t\t\\label{def: y}\n\t\t\ty_{t'+1}=y_{t'}-\\alpha_{t'} e_{i_{t'}} e_{i_{t'}}^{\\top} H y_{t'}, \\qquad t'=0,1,\\dots,m-1.\n\t\t\\end{equation} \n\t\tThen $x_{t'}=y^*+y_{t'}$ holds for any $t'=0,1,\\dots,m$. Using \\eqref{decay: quad}, to give an upper bound for $f^H(x_{t+m})-f^H(x_t)$, we would like a nontrivial lower bound for $\\alpha_{t'}\\bigl(e_{i_{t'}}^{\\top} H x_{t'}\\bigr)^2=\\alpha_{t'}\\bigl(e_{i_{t'}}^{\\top} H y_{t'}\\bigr)^2$ for some $t'\\in\\{t,t+1,\\dots,t+m-1\\}$, which is guaranteed by Lemma~\\ref{lem: linear_decay}, whose proof will be postponed. \n","sha256_utf8":"26900761f573b0af322def71e23950e72cd3ef4f60191782dbd50b96ce1f3b97"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"有效坐标产生定量下降","source_units":[{"start":42522,"end":43110,"start_line":449,"end_line":458,"raw_tex":"\t\tAssuming the lemma, there exists $T\\in\\{0,1,\\dots,m-1\\}$ that \n\t\t\\begin{equation*}\n\t\t\t\\alpha_T\\left\\vert e_{i_{T}}^{\\top} H y_{T}\\right\\vert\\geq \\delta\\norm{y_{T}},\n\t\t\\end{equation*}\n\t\twith a fixed $\\delta>0$ satisfying \\eqref{def:delta}. We can further constrain that $\\frac{\\delta^2}{\\alpha_{\\min}\\sigma_{\\max}(H)}<1$. Thus, we have\n\t\t\\begin{equation*}\\label{decay esti}\n\t\t\t\\begin{split}\n\t\t\t\tf^H(x_m)&\\leq f^H (x_{T+1})\\leq f^H(x_T)- \\frac{\\alpha_T}{2}\\left(e_{i_{T}}^{\\top} H x_{T}\\right)^2\\leq f^H (x_{T}) -\\frac{\\delta^2}{2\\alpha_T}\\norm{y_{T}}^2,\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n","sha256_utf8":"9329dee5a706812b4f84de8ead13db3f6fab1deaa4055db588607fb02f83dcfe"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"奇异值界确定下降常数","source_units":[{"start":43110,"end":43551,"start_line":459,"end_line":464,"raw_tex":"\t\twhich combined with $\\sigma_{\\max}(H)\\norm{y_{T}}^2\\geq -y_{T}^{\\top} H y_{T}=-x_{T}^{\\top} H x_{T}=-2f^H(x_T)$,\n\t\tyields that \n\t\t\\begin{equation*}\n\t\t\tf^H (x_m) \\leq\\left( 1 +\\frac{\\delta^2}{\\alpha_T \\sigma_{\\max}(H)}\\right)f^H (x_{T}) \\leq\\left( 1 +\\frac{\\delta^2}{\\alpha_T \\sigma_{\\max}(H)}\\right)f^H (x_0). \n\t\t\\end{equation*}\n\t\tSet $c=\\frac{\\delta^2}{\\alpha_{\\max} \\sigma_{\\max}(H)}$ and we get that $f^H(x_m)-f^H(x_0)\\leq c f^H(x_0)$.\n","sha256_utf8":"e1ffda298ce53286d6ca7260d551a84907472cd35755588b32b8ba8c54822544"}]},{"id":"S038","semantic_type":"lemma","kind":"lemma","title":"块内存在有效坐标更新","source_units":[{"start":41860,"end":42518,"start_line":440,"end_line":447,"raw_tex":"\\begin{lemma} \\label{lem: linear_decay}\n\t\t\tSuppose that $\\{1,2,\\dots,d\\}=\\{i_0,i_1,\\dots,i_{m-1}\\}$. For any\n\t\t\t\\begin{equation}\\label{def:delta}\n\t\t\t\t0<\\delta\\leq\\min\\left\\{\\frac{1}{2m},\\frac{\\alpha_{\\min}\\sigma_{\\min}(H)}{2\\sqrt{m}(m\\alpha_{\\max}\\sigma_{\\max}(H)+1)}\\right\\},\n\t\t\t\\end{equation}\n\t\t\twhere $\\sigma_{\\min}(H)$ and $\\sigma_{\\max}(H)$ are the smallest and largest positive singular values of $H$ respectively, \n\t\t\tif $0\\neq y_0\\in \\ran(H)$, then there exists $T\\in \\{0,1,\\dots,m-1\\}$ such that $\\alpha_{T}\\left\\vert e_{i_{T}}^{\\top} H y_{T}\\right\\vert\\geq \\delta\\norm{y_{T}}$, where the sequence $y_t$ is given as in \\eqref{def: y}. \n\t\t\\end{lemma}","sha256_utf8":"8ddd55f207ec736a429ab07faaebbc6c37a1290669a568c346d86d2fe200527b"}]},{"id":"S039","semantic_type":"proof","kind":"proof","title":"有效更新引理完整证明","source_units":[{"start":43646,"end":45582,"start_line":470,"end_line":500,"raw_tex":"\\begin{proof}[Proof of Lemma~\\ref{lem: linear_decay}] Suppose on the contrary that $\\alpha_t\\left\\vert e_{i_t}^{\\top} H y_t\\right\\vert< \\delta\\norm{y_t}$ for any $t\\in \\{0,1,\\dots,m-1\\}$. It holds that\n\t\t\\begin{equation*}\n\t\t\t\\norm{y_1-y_0}=\\alpha_0\\left\\vert e_{i_0}^{\\top} H y_0\\right\\vert<\\delta\\norm{y_0}<2\\delta\\norm{y_0}.\n\t\t\\end{equation*}\n\t\tWe claim that \n\t\t\\begin{equation} \\label{eq: disty}\n\t\t\t\\norm{y_t-y_0}<2 t\\delta\\norm{y_0},\n\t\t\\end{equation}\n\t\tfor any $t=1,2,\\dots,m-1$. By induction, assume that  $\\norm{y_t-y_0}<2t\\delta\\norm{y_0}$ holds for some $t\\in\\{1,2,\\dots,m-2\\}$, then \n\t\t\\begin{equation*}\n\t\t\t\\norm{y_{t+1}-y_t}=\\alpha_t\\left\\vert e_{i_t}^{\\top} H y_t\\right\\vert< \\delta\\norm{y_t}<\\delta(2t\\delta+1)\\norm{y_0}<2\\delta\\norm{y_0},\n\t\t\\end{equation*}\n\t\twhere the last inequality uses $2 t \\delta < 2m \\delta \\leq 1 $. It follows $\\norm{y_{t+1}-y_0}\\leq \\norm{y_t-y_0}+\\norm{y_{t+1}-y_t}<2(t+1)\\delta\\norm{y_0}$.\n\t\t\n\t\tUsing \\eqref{eq: disty} and $\\max_{1\\leq i\\leq d} \\norm{H e_i}\\leq \\sigma_{\\max}(H)$, we have \n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\alpha_t\\bigl\\lvert e_{i_t}^{\\top} H y_0\\bigr\\rvert&\\leq \\alpha_t\\bigl\\lvert e_{i_t}^{\\top} H (y_t-y_0)\\bigr\\rvert+\\alpha_t\\bigl\\lvert e_{i_t}^{\\top} H y_t\\bigr\\rvert\\\\\n\t\t\t\t&<\\alpha_{\\max}\\sigma_{\\max}(H)\\cdot 2 t\\delta\\norm{y_0}+2\\delta\\norm{y_0}\\\\\n\t\t\t\t&<2\\delta(m\\alpha_{\\max}\\sigma_{\\max}(H)+1)\\norm{y_0},\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tfor $t=0,1,\\dots,m-1$. Since $\\text{span}\\{e_{i_k}:k=0,1,\\dots,m-1\\}=\\bR^d$, noticing that $y_0\\in\\ran{H}$, we have\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\alpha_{\\min}\\sigma_{\\min}(H) \\norm{y_0}&\\leq \\alpha_{\\min}\\norm{H y_0}\\leq\\left(\\sum_{t=0}^{m-1} \\left(\\alpha_t\\bigl\\lvert e_{i_t}^{\\top} H y_0\\bigr\\rvert\\right)^2\\right)^{1/2}\\\\\n\t\t\t\t& <2\\delta\\sqrt{m}(m\\alpha_{\\max}\\sigma_{\\max}(H)+1)\\norm{y_0},\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\twhich contradicts with the choice of $\\delta$ in \\eqref{def:delta}.\n\t\\end{proof}","sha256_utf8":"f425fc3bebacb66a85873f471543e908f9e14a65caa00b824bf4e39204efdf07"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"反设更新小并归纳控制位移","source_units":[{"start":43646,"end":44577,"start_line":470,"end_line":482,"raw_tex":"\\begin{proof}[Proof of Lemma~\\ref{lem: linear_decay}] Suppose on the contrary that $\\alpha_t\\left\\vert e_{i_t}^{\\top} H y_t\\right\\vert< \\delta\\norm{y_t}$ for any $t\\in \\{0,1,\\dots,m-1\\}$. It holds that\n\t\t\\begin{equation*}\n\t\t\t\\norm{y_1-y_0}=\\alpha_0\\left\\vert e_{i_0}^{\\top} H y_0\\right\\vert<\\delta\\norm{y_0}<2\\delta\\norm{y_0}.\n\t\t\\end{equation*}\n\t\tWe claim that \n\t\t\\begin{equation} \\label{eq: disty}\n\t\t\t\\norm{y_t-y_0}<2 t\\delta\\norm{y_0},\n\t\t\\end{equation}\n\t\tfor any $t=1,2,\\dots,m-1$. By induction, assume that  $\\norm{y_t-y_0}<2t\\delta\\norm{y_0}$ holds for some $t\\in\\{1,2,\\dots,m-2\\}$, then \n\t\t\\begin{equation*}\n\t\t\t\\norm{y_{t+1}-y_t}=\\alpha_t\\left\\vert e_{i_t}^{\\top} H y_t\\right\\vert< \\delta\\norm{y_t}<\\delta(2t\\delta+1)\\norm{y_0}<2\\delta\\norm{y_0},\n\t\t\\end{equation*}\n\t\twhere the last inequality uses $2 t \\delta < 2m \\delta \\leq 1 $. It follows $\\norm{y_{t+1}-y_0}\\leq \\norm{y_t-y_0}+\\norm{y_{t+1}-y_t}<2(t+1)\\delta\\norm{y_0}$.\n","sha256_utf8":"5c4fa578d365021417a8c22ae0291b4ebcd2f54cd37848fff4bdd94144550f27"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"初始梯度各坐标上界","source_units":[{"start":44580,"end":45178,"start_line":484,"end_line":492,"raw_tex":"\t\tUsing \\eqref{eq: disty} and $\\max_{1\\leq i\\leq d} \\norm{H e_i}\\leq \\sigma_{\\max}(H)$, we have \n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\alpha_t\\bigl\\lvert e_{i_t}^{\\top} H y_0\\bigr\\rvert&\\leq \\alpha_t\\bigl\\lvert e_{i_t}^{\\top} H (y_t-y_0)\\bigr\\rvert+\\alpha_t\\bigl\\lvert e_{i_t}^{\\top} H y_t\\bigr\\rvert\\\\\n\t\t\t\t&<\\alpha_{\\max}\\sigma_{\\max}(H)\\cdot 2 t\\delta\\norm{y_0}+2\\delta\\norm{y_0}\\\\\n\t\t\t\t&<2\\delta(m\\alpha_{\\max}\\sigma_{\\max}(H)+1)\\norm{y_0},\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tfor $t=0,1,\\dots,m-1$. Since $\\text{span}\\{e_{i_k}:k=0,1,\\dots,m-1\\}=\\bR^d$, noticing that $y_0\\in\\ran{H}$, we have\n","sha256_utf8":"4879f2d720a2b8ce887de2431f01b658597a40eefb7024dc66f3134e28d87fe8"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"坐标覆盖与最小奇异值矛盾","source_units":[{"start":45178,"end":45570,"start_line":493,"end_line":499,"raw_tex":"\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\alpha_{\\min}\\sigma_{\\min}(H) \\norm{y_0}&\\leq \\alpha_{\\min}\\norm{H y_0}\\leq\\left(\\sum_{t=0}^{m-1} \\left(\\alpha_t\\bigl\\lvert e_{i_t}^{\\top} H y_0\\bigr\\rvert\\right)^2\\right)^{1/2}\\\\\n\t\t\t\t& <2\\delta\\sqrt{m}(m\\alpha_{\\max}\\sigma_{\\max}(H)+1)\\norm{y_0},\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\twhich contradicts with the choice of $\\delta$ in \\eqref{def:delta}.\n","sha256_utf8":"1d8bbe697be76505825cc64bc45366e88fbde7c047cb5931e9516274fbcc411e"}]},{"id":"S043","semantic_type":"theorem","kind":"theorem","title":"有限时间高概率放大","source_units":[{"start":48416,"end":49217,"start_line":538,"end_line":544,"raw_tex":"\\begin{theorem}\\label{thm: local-growth}\n\t\tSuppose that Assumptions~\\ref{Assump: Hess_bdd}, \\ref{Assump: non-degenerate}, and \\ref{Assump: non-zero-proj} hold and that $0<\\alpha_{\\min}<\\alpha_{\\max}<1/M$. There exists $\\epsilon_*\\in(0,1/6)$ such that for any $\\epsilon\\in(0,\\epsilon_*)$, we have $T_* = T_*(\\epsilon) \\in\\bNp$ that for any $T \\in \\bNp$ with $T\\geq T_*$ and any $t\\in\\bN$, conditioned on $\\mathcal{F}_{t-1}$, with probability at least $1-4\\epsilon$, it holds for all $x_t \\in V$ that\n\t\t\\begin{equation}\\label{local-growth}\n\t\t\t\\norm{x_{t+T}}\\geq \\exp\\biggl(\\frac{6\\epsilon}{1 - 6 \\epsilon} \\bigl\\lvert \\log(1-M\\alpha_{\\max}) \\bigr\\rvert \\, T\\biggr) \\norm{x_t},\n\t\t\\end{equation}\n\t\twhere $V$ is a neighborhood of $x^* = 0$, depending on $\\epsilon$, $T$, and $f$ near $x^*$. \n\t\\end{theorem}","sha256_utf8":"86ab076bc46e3a0d18eb73f6055208d2772f8d8dadaf5eb1553f26d21cf9860c"}]},{"id":"S044","semantic_type":"proof","kind":"proof","title":"局部放大定理完整证明","source_units":[{"start":65093,"end":73009,"start_line":749,"end_line":859,"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm: local-growth}]\n\t\tWithout loss of generality, we will assume $t=0$ in the proof to simplify notation.\n\t\tSince $H=\\nabla^2 f(x^*)$ is non-degenerate, we can take a neighborhood \n\t\t$U$ of $x^*=0$ and some fixed $\\sigma > 0$ such that\n\t\t\\begin{equation}\\label{eq: sigma}\n\t\t\t\\norm{\\nabla f(x)}\\geq \\sigma\\norm{x},\\quad \\forall\\ x\\in U.    \n\t\t\\end{equation}\n\t\tAssumption~\\ref{Assump: Hess_bdd} implies that\n\t\t\\begin{equation*}\n\t\t\t\\norm{\\nabla f(x)}=\\norm{\\nabla f(x)-\\nabla f(x^*)}\\leq M\\norm{x-x^*}=M\\norm{x},\\quad\\forall\\ x\\in\\bR^d.\n\t\t\\end{equation*}\n\t\tUsing the above inequality and $\\alpha_{\\max}<1/M$, it holds for every $\\omega\\in\\Omega$ and $t'\\in\\bN$ that \n\t\t\\begin{equation}\\label{r-minus}\n\t\t\t\\begin{split}\n\t\t\t\t\\norm{x_{t'+1}}&=\\norm{x_{t'}-\\alpha_{t'} e_{i_{t'}}e_{i_{t'}}^{\\top}\\nabla f(x_{t'})} \\\\\n\t\t\t\t& \\geq \\norm{x_{t'}}-\\alpha_{t'}\\norm{e_{i_{t'}}e_{i_{t'}}^{\\top}}\\cdot \\norm{\\nabla f(x_{t'})}\\\\\n\t\t\t\t&\\geq (1-M\\alpha_{\\max})\\norm{x_{t'}},\n\t\t\t\\end{split}\n\t\t\\end{equation}\n\t\tand similarly that\n\t\t\\begin{equation*}\n\t\t\t\\norm{x_{t'+1}}\\leq(1+M\\alpha_{\\max})\\norm{x_{t'}}.\n\t\t\\end{equation*}\n\t\tWe thus define\n\t\t\\begin{equation*}\n\t\t\tr_-:=1-M\\alpha_{\\max}\\quad\\text{and}\\quad r_+:=1+M\\alpha_{\\max}, \n\t\t\\end{equation*}\n\t\tso that\n\t\t\\begin{equation}\\label{eq:compareiterate}\n\t\t\tr_- \\norm{x_{t'}} \\leq \\norm{x_{t'+1}} \\leq r_+ \\norm{x_{t'}}.\n\t\t\\end{equation}\n\t\t\n\t\tWe now choose the time duration $S$ large enough in the finite block analysis to guarantee significant amplification. More specifically, we choose $S$ so large that $S \\geq S_*$ ($S_*$ defined in Lemma~\\ref{lem: T-Omega2}) and that the following two inequalities hold: \n\t\t\\begin{equation}\\label{large S1}\n\t\t\t\\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta \\mu\\sigma (r_-)^{L-1}(\\alpha_{\\max}-\\alpha_{\\min})}{8}\\geq\n\t\t\t(r_+)^{L},\n\t\t\\end{equation}\n\t\tand\n\t\t\\begin{equation}\\label{large S2}\n\t\t\t(1-6\\epsilon)\\left(S(\\lambda_+-\\gamma)+\\log \\frac{\\epsilon\\delta \\mu\\sigma (r_-)^{2(L-1)}(\\alpha_{\\max}-\\alpha_{\\min})}{8}\\right)+6\\epsilon(L+S)\\log r_->0,\n\t\t\\end{equation}\n\t\twhere $L$ is the upper bound defined in Lemma~\\ref{lem: L}, $\\mu\\leq 1/\\sqrt{d}$ is a fixed constant as in Lemma~\\ref{lem: L}, $\\delta$ is from Lemma~\\ref{lem: Omega1}, and $\\sigma$ is set in \\eqref{eq: sigma}. \n\t\tThanks to \\eqref{def: eps} for our choice of $\\epsilon$ and that $\\gamma < \\lambda_+$ from \\eqref{lambplus-gamma}, \n\t\t\\eqref{large S1} and \\eqref{large S2} are satisfied for sufficiently large $S$. \n\t\t\n\t\tNext, we show that, for any $S$ sufficiently large as above, there exists a convex neighborhood $U_1\\subset U$ of $x^*=0$, such that for any $t'\\in\\bN$, any $x_{t'}\\in U_1$, and any $(i,\\alpha)_{t':t'+S-1}$, it holds that\n\t\t\\begin{equation}\\label{block-error}\n\t\t\t\\norm{x_{t'+S}} \\geq \\norm{\\Phi^H\\bigl((i,\\alpha)_{t':t'+S-1}\\bigr)x_{t'}}-\\norm{x_{t'}}.\n\t\t\\end{equation}\n\t\tWe first define a convex neighborhood $U_0 \\subset U$ of $x^*=0$ such that\n\t\t\\begin{multline*}\n\t\t\t\\norm{\\left(x-\\alpha e_i e_i^{\\top}\\nabla f(x)\\right)-\\left(I-\\alpha e_i e_i^{\\top}H\\right)x}=\\norm{\\alpha e_i e_i^{\\top}\\left(\\nabla f(x)-H x\\right)}\\\\\n\t\t\t=\\norm{\\alpha e_i e_i^{\\top}\\int_0^1 (\\nabla^2 f(\\eta x)-Hx) \\mathrm{d} \\eta}\\leq \\frac{1}{S(r_+)^{S-1}} \\norm{x},\n\t\t\\end{multline*}\n\t\tfor any $x \\in U_0$, any $i\\in\\{1,2,\\dots,d\\}$, and any $\\alpha\\in[\\alpha_{\\min},\\alpha_{\\max}]$.\n\t\tApplying the inequality $S$ times for $x_{t'}\\in U_1 = (r_+)^{-(S-1)}U_0$, we have,\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t&\\norm{x_{t'+S}-\\Phi^H\\bigl((i,\\alpha)_{t':t'+S-1}\\bigr)x_{t'}}\\\\\n\t\t\t\t\\leq & \\norm{x_{t'+S}-\\left(I-\\alpha_{t'+S-1} e_{i_{t'+S-1}}e_{i_{t'+S-1}}^{\\top}H\\right)x_{t'+S-1}}\\\\\n\t\t\t\t&+\\norm{I-\\alpha_{t'+S-1} e_{i_{t'+S-1}}e_{i_{t'+S-1}}^{\\top}H}\\cdot \\norm{x_{t'+S-1}-\\Phi^H\\bigl((i,\\alpha)_{t':t'+S-2}\\bigr)x_{t'}}\\\\\n\t\t\t\t\\leq & \\frac{1}{S(r_+)^{S-1}} \\norm{x_{t'+S-1}}+r_+\\norm{x_{t'+S-1}-\\Phi^H\\bigl((i,\\alpha)_{t':t'+S-2}\\bigr)x_{t'}}\\\\\n\t\t\t\t\\leq& \\frac{1}{S(r_+)^{S-1}} \\left(\\norm{x_{t'+S-1}}+r_+\\norm{x_{t'+S-2}}+\\cdots+(r_+)^{S-1}\\norm{x_{t'}}\\right)\\\\\n\t\t\t\t\\leq& \\norm{x_{t'}},\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tand hence, inequality \\eqref{block-error}.\n\t\t\n\t\tSetting $V=(r_+)^{-(L-1)} U_1$, we then have $x_{\\varrho_0-1}\\in U$ for any $x_0\\in V$, which implies that $\\norm{\\nabla f(x_{\\varrho_0-1})}\\geq\\sigma\\norm{x_{\\varrho_0-1}}$ as $\\varrho_0\\leq L$. \n\t\tAccording to Lemma~\\ref{lem: L}--\\ref{lem: nontrial-proj}, for any given $x_0\\in V$, with probability at least $1-4\\epsilon$, we have $(i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\in\\Omega^S$, and  the followings hold:\n\t\t\\begin{align}\n\t\t\t\\label{ineq1} & \\bigl\\lvert e_{i_{\\varrho_0-1}}^{\\top} \\nabla f(x_{\\varrho_0-1})\\bigr\\rvert\\geq \\mu\\norm{\\nabla f(x_{\\varrho_0-1})}, \\\\\n\t\t\t\\label{ineq2} & \\norm{\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)x_{\\varrho_0}}\\geq \\frac{\\epsilon\\delta (\\alpha_{\\max}-\\alpha_{\\min})}{4}\\bigl\\lvert e_{i_{\\varrho_0-1}}^{\\top} \\nabla f(x_{\\varrho_0-1})\\bigr\\rvert,\n\t\t\\end{align}\n\t\twhere the probability is the marginal probability on $(i,\\alpha)_{0:L+S-1}\\in\\Omega_0\\times\\cdots\\times \\Omega_{L+S-1}$.  \n\t\t\n\t\t\\smallskip \n\t\t\n\t\tRecall $\\lambda_+$ and $\\gamma$ in \\eqref{lambplus-gamma}, and $d_+=\\sum_{\\lambda_i>0}d_i$. \n\t\tIt follows from \\eqref{close: singular value} of the construction of the set $\\Omega^S$ that\n\t\t\\begin{equation*}\n\t\t\t\\frac{1}{S}\\log s_j\\left(\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)\\right)\\geq \\lambda_{\\mu(j)}-\\gamma\\geq \\lambda_+-\\gamma,\\quad\\forall\\ j\\leq d_+.\n\t\t\\end{equation*}\n\t\tThis is to say that the $d_+$ largest singular values of $\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)$\n\t\tare all greater than or equal to $\\exp(S(\\lambda_+-\\gamma))$. \n\t\tTherefore, it holds that \n\t\t\\begin{equation}\\label{linear-growth}\n\t\t\t\\begin{split}\n\t\t\t\t\\norm{\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)x_{\\varrho_0}}&\\geq \\norm{\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)x_{\\varrho_0}}\\\\\n\t\t\t\t&  \\leftstackrel{\\eqref{ineq2}}{\\geq} \\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta (\\alpha_{\\max}-\\alpha_{\\min})}{4}\\bigl\\lvert e_{i_{\\varrho_0-1}}^{\\top} \\nabla f(x_{\\varrho_0-1})\\bigr\\rvert\\\\\n\t\t\t\t&  \\leftstackrel{\\eqref{ineq1}}{\\geq}\n\t\t\t\t\\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta\\mu (\\alpha_{\\max}-\\alpha_{\\min})}{4}\\norm{\\nabla f(x_{\\varrho_0-1})},\n\t\t\t\\end{split}\n\t\t\\end{equation}\n\t\twhere the first inequality follows from the fact that $\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)x_{\\varrho_0}$ and $\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)\\left(I-\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)\\right)x_{\\varrho_0}$ are orthogonal. Combining \\eqref{block-error} and \\eqref{linear-growth}, we obtain that\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\norm{x_{\\varrho_0+S}} & \\geq \\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta \\mu(\\alpha_{\\max}-\\alpha_{\\min})}{4}\\norm{\\nabla f(x_{\\varrho_0-1})}-\\norm{x_{\\varrho_0}}\\\\\n\t\t\t\t& \\leftstackrel{\\eqref{eq:compareiterate}}{\\geq} \\left(\\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta \\mu\\sigma (r_-)^{L-1}(\\alpha_{\\max}-\\alpha_{\\min})}{4}-(r_+)^{L}\\right)\\cdot\\norm{x_0}\\\\\n\t\t\t\t& \\leftstackrel{\\eqref{large S1}}{\\geq} \\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta \\mu\\sigma (r_-)^{L-1}(\\alpha_{\\max}-\\alpha_{\\min})}{8}\\cdot\\norm{x_0}.\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tTherefore, it holds that\n\t\t\\begin{equation*}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\norm{x_{L+S}} & \\; \\leftstackrel{\\eqref{eq:compareiterate}}{\\geq} (r_-)^{L-1} \\norm{x_{\\varrho_0+S}} \\\\\n\t\t\t\t& \\; \\geq \\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta \\mu\\sigma (r_-)^{2(L-1)}(\\alpha_{\\max}-\\alpha_{\\min})}{8}\\cdot\\norm{x_0}.\n\t\t\t\\end{aligned}\n\t\t\\end{equation*}\n\t\tWe finally arrive at \\eqref{local-growth} by setting $T = L + S$ and combining the above with \\eqref{large S2}.\n\t\\end{proof}","sha256_utf8":"6753f7c1fac9e35a1bfd924389dae20a014acc79ad35aa2987e10adefb6c52e6"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"非退化性给出梯度下界","source_units":[{"start":65149,"end":65486,"start_line":750,"end_line":755,"raw_tex":"\t\tWithout loss of generality, we will assume $t=0$ in the proof to simplify notation.\n\t\tSince $H=\\nabla^2 f(x^*)$ is non-degenerate, we can take a neighborhood \n\t\t$U$ of $x^*=0$ and some fixed $\\sigma > 0$ such that\n\t\t\\begin{equation}\\label{eq: sigma}\n\t\t\t\\norm{\\nabla f(x)}\\geq \\sigma\\norm{x},\\quad \\forall\\ x\\in U.    \n\t\t\\end{equation}\n","sha256_utf8":"d23e6d4e71e69fbef8804b338f9363bbfb3c32be4f186944858fcfdf3594cf82"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"逐步范数的双侧比较","source_units":[{"start":65486,"end":66490,"start_line":756,"end_line":779,"raw_tex":"\t\tAssumption~\\ref{Assump: Hess_bdd} implies that\n\t\t\\begin{equation*}\n\t\t\t\\norm{\\nabla f(x)}=\\norm{\\nabla f(x)-\\nabla f(x^*)}\\leq M\\norm{x-x^*}=M\\norm{x},\\quad\\forall\\ x\\in\\bR^d.\n\t\t\\end{equation*}\n\t\tUsing the above inequality and $\\alpha_{\\max}<1/M$, it holds for every $\\omega\\in\\Omega$ and $t'\\in\\bN$ that \n\t\t\\begin{equation}\\label{r-minus}\n\t\t\t\\begin{split}\n\t\t\t\t\\norm{x_{t'+1}}&=\\norm{x_{t'}-\\alpha_{t'} e_{i_{t'}}e_{i_{t'}}^{\\top}\\nabla f(x_{t'})} \\\\\n\t\t\t\t& \\geq \\norm{x_{t'}}-\\alpha_{t'}\\norm{e_{i_{t'}}e_{i_{t'}}^{\\top}}\\cdot \\norm{\\nabla f(x_{t'})}\\\\\n\t\t\t\t&\\geq (1-M\\alpha_{\\max})\\norm{x_{t'}},\n\t\t\t\\end{split}\n\t\t\\end{equation}\n\t\tand similarly that\n\t\t\\begin{equation*}\n\t\t\t\\norm{x_{t'+1}}\\leq(1+M\\alpha_{\\max})\\norm{x_{t'}}.\n\t\t\\end{equation*}\n\t\tWe thus define\n\t\t\\begin{equation*}\n\t\t\tr_-:=1-M\\alpha_{\\max}\\quad\\text{and}\\quad r_+:=1+M\\alpha_{\\max}, \n\t\t\\end{equation*}\n\t\tso that\n\t\t\\begin{equation}\\label{eq:compareiterate}\n\t\t\tr_- \\norm{x_{t'}} \\leq \\norm{x_{t'+1}} \\leq r_+ \\norm{x_{t'}}.\n\t\t\\end{equation}\n","sha256_utf8":"3f90029b40da955ee94633278ceed2b07b5cbe6723516774924368120ef7b7ef"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"选取足够长的放大时间","source_units":[{"start":66493,"end":67577,"start_line":781,"end_line":792,"raw_tex":"\t\tWe now choose the time duration $S$ large enough in the finite block analysis to guarantee significant amplification. More specifically, we choose $S$ so large that $S \\geq S_*$ ($S_*$ defined in Lemma~\\ref{lem: T-Omega2}) and that the following two inequalities hold: \n\t\t\\begin{equation}\\label{large S1}\n\t\t\t\\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta \\mu\\sigma (r_-)^{L-1}(\\alpha_{\\max}-\\alpha_{\\min})}{8}\\geq\n\t\t\t(r_+)^{L},\n\t\t\\end{equation}\n\t\tand\n\t\t\\begin{equation}\\label{large S2}\n\t\t\t(1-6\\epsilon)\\left(S(\\lambda_+-\\gamma)+\\log \\frac{\\epsilon\\delta \\mu\\sigma (r_-)^{2(L-1)}(\\alpha_{\\max}-\\alpha_{\\min})}{8}\\right)+6\\epsilon(L+S)\\log r_->0,\n\t\t\\end{equation}\n\t\twhere $L$ is the upper bound defined in Lemma~\\ref{lem: L}, $\\mu\\leq 1/\\sqrt{d}$ is a fixed constant as in Lemma~\\ref{lem: L}, $\\delta$ is from Lemma~\\ref{lem: Omega1}, and $\\sigma$ is set in \\eqref{eq: sigma}. \n\t\tThanks to \\eqref{def: eps} for our choice of $\\epsilon$ and that $\\gamma < \\lambda_+$ from \\eqref{lambplus-gamma}, \n\t\t\\eqref{large S1} and \\eqref{large S2} are satisfied for sufficiently large $S$. \n","sha256_utf8":"871628ef3b4024728eb0be1ef16ce269f8a176f984c706fce8e8987e4df12541"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"小邻域中的线性化单步误差","source_units":[{"start":67580,"end":68441,"start_line":794,"end_line":803,"raw_tex":"\t\tNext, we show that, for any $S$ sufficiently large as above, there exists a convex neighborhood $U_1\\subset U$ of $x^*=0$, such that for any $t'\\in\\bN$, any $x_{t'}\\in U_1$, and any $(i,\\alpha)_{t':t'+S-1}$, it holds that\n\t\t\\begin{equation}\\label{block-error}\n\t\t\t\\norm{x_{t'+S}} \\geq \\norm{\\Phi^H\\bigl((i,\\alpha)_{t':t'+S-1}\\bigr)x_{t'}}-\\norm{x_{t'}}.\n\t\t\\end{equation}\n\t\tWe first define a convex neighborhood $U_0 \\subset U$ of $x^*=0$ such that\n\t\t\\begin{multline*}\n\t\t\t\\norm{\\left(x-\\alpha e_i e_i^{\\top}\\nabla f(x)\\right)-\\left(I-\\alpha e_i e_i^{\\top}H\\right)x}=\\norm{\\alpha e_i e_i^{\\top}\\left(\\nabla f(x)-H x\\right)}\\\\\n\t\t\t=\\norm{\\alpha e_i e_i^{\\top}\\int_0^1 (\\nabla^2 f(\\eta x)-Hx) \\mathrm{d} \\eta}\\leq \\frac{1}{S(r_+)^{S-1}} \\norm{x},\n\t\t\\end{multline*}\n\t\tfor any $x \\in U_0$, any $i\\in\\{1,2,\\dots,d\\}$, and any $\\alpha\\in[\\alpha_{\\min},\\alpha_{\\max}]$.\n","sha256_utf8":"cce01256e16b6d8b62df00757c0642a60c8ca96309df2926c1eba9b847566e3c"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"累积有限块线性化误差","source_units":[{"start":68441,"end":69225,"start_line":804,"end_line":815,"raw_tex":"\t\tApplying the inequality $S$ times for $x_{t'}\\in U_1 = (r_+)^{-(S-1)}U_0$, we have,\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t&\\norm{x_{t'+S}-\\Phi^H\\bigl((i,\\alpha)_{t':t'+S-1}\\bigr)x_{t'}}\\\\\n\t\t\t\t\\leq & \\norm{x_{t'+S}-\\left(I-\\alpha_{t'+S-1} e_{i_{t'+S-1}}e_{i_{t'+S-1}}^{\\top}H\\right)x_{t'+S-1}}\\\\\n\t\t\t\t&+\\norm{I-\\alpha_{t'+S-1} e_{i_{t'+S-1}}e_{i_{t'+S-1}}^{\\top}H}\\cdot \\norm{x_{t'+S-1}-\\Phi^H\\bigl((i,\\alpha)_{t':t'+S-2}\\bigr)x_{t'}}\\\\\n\t\t\t\t\\leq & \\frac{1}{S(r_+)^{S-1}} \\norm{x_{t'+S-1}}+r_+\\norm{x_{t'+S-1}-\\Phi^H\\bigl((i,\\alpha)_{t':t'+S-2}\\bigr)x_{t'}}\\\\\n\t\t\t\t\\leq& \\frac{1}{S(r_+)^{S-1}} \\left(\\norm{x_{t'+S-1}}+r_+\\norm{x_{t'+S-2}}+\\cdots+(r_+)^{S-1}\\norm{x_{t'}}\\right)\\\\\n\t\t\t\t\\leq& \\norm{x_{t'}},\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tand hence, inequality \\eqref{block-error}.\n","sha256_utf8":"b42f10526d342262ebeb69023bbcbd375c916b9cb3da1421bc95270608ab7704"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"合并高概率梯度与投影事件","source_units":[{"start":69228,"end":70193,"start_line":817,"end_line":826,"raw_tex":"\t\tSetting $V=(r_+)^{-(L-1)} U_1$, we then have $x_{\\varrho_0-1}\\in U$ for any $x_0\\in V$, which implies that $\\norm{\\nabla f(x_{\\varrho_0-1})}\\geq\\sigma\\norm{x_{\\varrho_0-1}}$ as $\\varrho_0\\leq L$. \n\t\tAccording to Lemma~\\ref{lem: L}--\\ref{lem: nontrial-proj}, for any given $x_0\\in V$, with probability at least $1-4\\epsilon$, we have $(i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\in\\Omega^S$, and  the followings hold:\n\t\t\\begin{align}\n\t\t\t\\label{ineq1} & \\bigl\\lvert e_{i_{\\varrho_0-1}}^{\\top} \\nabla f(x_{\\varrho_0-1})\\bigr\\rvert\\geq \\mu\\norm{\\nabla f(x_{\\varrho_0-1})}, \\\\\n\t\t\t\\label{ineq2} & \\norm{\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)x_{\\varrho_0}}\\geq \\frac{\\epsilon\\delta (\\alpha_{\\max}-\\alpha_{\\min})}{4}\\bigl\\lvert e_{i_{\\varrho_0-1}}^{\\top} \\nabla f(x_{\\varrho_0-1})\\bigr\\rvert,\n\t\t\\end{align}\n\t\twhere the probability is the marginal probability on $(i,\\alpha)_{0:L+S-1}\\in\\Omega_0\\times\\cdots\\times \\Omega_{L+S-1}$.  \n\t\t\n\t\t\\smallskip \n\t\t\n","sha256_utf8":"ae0657e953e0a33af584d1f692d98b43ec831d39458e5868c046096ddab68093"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"奇异值放大传递到非线性迭代","source_units":[{"start":70193,"end":72997,"start_line":827,"end_line":858,"raw_tex":"\t\tRecall $\\lambda_+$ and $\\gamma$ in \\eqref{lambplus-gamma}, and $d_+=\\sum_{\\lambda_i>0}d_i$. \n\t\tIt follows from \\eqref{close: singular value} of the construction of the set $\\Omega^S$ that\n\t\t\\begin{equation*}\n\t\t\t\\frac{1}{S}\\log s_j\\left(\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)\\right)\\geq \\lambda_{\\mu(j)}-\\gamma\\geq \\lambda_+-\\gamma,\\quad\\forall\\ j\\leq d_+.\n\t\t\\end{equation*}\n\t\tThis is to say that the $d_+$ largest singular values of $\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)$\n\t\tare all greater than or equal to $\\exp(S(\\lambda_+-\\gamma))$. \n\t\tTherefore, it holds that \n\t\t\\begin{equation}\\label{linear-growth}\n\t\t\t\\begin{split}\n\t\t\t\t\\norm{\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)x_{\\varrho_0}}&\\geq \\norm{\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)x_{\\varrho_0}}\\\\\n\t\t\t\t&  \\leftstackrel{\\eqref{ineq2}}{\\geq} \\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta (\\alpha_{\\max}-\\alpha_{\\min})}{4}\\bigl\\lvert e_{i_{\\varrho_0-1}}^{\\top} \\nabla f(x_{\\varrho_0-1})\\bigr\\rvert\\\\\n\t\t\t\t&  \\leftstackrel{\\eqref{ineq1}}{\\geq}\n\t\t\t\t\\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta\\mu (\\alpha_{\\max}-\\alpha_{\\min})}{4}\\norm{\\nabla f(x_{\\varrho_0-1})},\n\t\t\t\\end{split}\n\t\t\\end{equation}\n\t\twhere the first inequality follows from the fact that $\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)x_{\\varrho_0}$ and $\\Phi^H\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)\\left(I-\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_0:\\varrho_0+S-1}\\right)\\right)x_{\\varrho_0}$ are orthogonal. Combining \\eqref{block-error} and \\eqref{linear-growth}, we obtain that\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\norm{x_{\\varrho_0+S}} & \\geq \\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta \\mu(\\alpha_{\\max}-\\alpha_{\\min})}{4}\\norm{\\nabla f(x_{\\varrho_0-1})}-\\norm{x_{\\varrho_0}}\\\\\n\t\t\t\t& \\leftstackrel{\\eqref{eq:compareiterate}}{\\geq} \\left(\\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta \\mu\\sigma (r_-)^{L-1}(\\alpha_{\\max}-\\alpha_{\\min})}{4}-(r_+)^{L}\\right)\\cdot\\norm{x_0}\\\\\n\t\t\t\t& \\leftstackrel{\\eqref{large S1}}{\\geq} \\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta \\mu\\sigma (r_-)^{L-1}(\\alpha_{\\max}-\\alpha_{\\min})}{8}\\cdot\\norm{x_0}.\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tTherefore, it holds that\n\t\t\\begin{equation*}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\norm{x_{L+S}} & \\; \\leftstackrel{\\eqref{eq:compareiterate}}{\\geq} (r_-)^{L-1} \\norm{x_{\\varrho_0+S}} \\\\\n\t\t\t\t& \\; \\geq \\exp(S(\\lambda_+-\\gamma))\\cdot \\frac{\\epsilon\\delta \\mu\\sigma (r_-)^{2(L-1)}(\\alpha_{\\max}-\\alpha_{\\min})}{8}\\cdot\\norm{x_0}.\n\t\t\t\\end{aligned}\n\t\t\\end{equation*}\n\t\tWe finally arrive at \\eqref{local-growth} by setting $T = L + S$ and combining the above with \\eqref{large S2}.\n","sha256_utf8":"6baeea8f176dd6253f2bd33ae522ffc8067379c0e700ccb7d718f101b1986329"}]},{"id":"S052","semantic_type":"lemma","kind":"lemma","title":"有限等待获得显著梯度坐标","source_units":[{"start":52428,"end":52928,"start_line":572,"end_line":578,"raw_tex":"\\begin{lemma}\\label{lem: L}\n\t\tLet $0<\\mu\\leq\\frac{1}{\\sqrt{d}}$ be a fixed constant. \n\t\tThere exists some constant $L>0$, such that for any $t\\in\\bN$, there exists a measurable $\\varrho_t:\\Omega\\rightarrow \\bNp$ such that $t < \\varrho_t \\leq t + L$ and\n\t\t\\begin{equation}\\label{eq: rho_t}\n\t\t\t\\bP\\Bigl(\\bigl\\lvert e_{i_{\\varrho_t-1}}^{\\top} \\nabla f(x_{\\varrho_t-1})\\bigr\\rvert \\geq \\mu\\norm{\\nabla f(x_{\\varrho_t-1})}\\;\\Big\\vert\\; \\mathcal{F}_{t-1}\\Bigr)\\geq 1-\\epsilon.\n\t\t\\end{equation}\n\t\\end{lemma}","sha256_utf8":"bade027f138f6e63725c2f2cb2556c4be3fea3e07966287a0b9c4a9385690e0a"}]},{"id":"S053","semantic_type":"proof","kind":"proof","title":"有限等待引理完整证明","source_units":[{"start":52932,"end":53651,"start_line":580,"end_line":591,"raw_tex":"\\begin{proof}\n\t\tFor any $t\\in\\bN$, use $\\ell_0$ to denote the smallest non-negative integer $\\ell$, such that\n\t\t\\begin{equation*}\n\t\t\t\\ell_0 = \\arg\\min_{\\ell} \\Bigl\\{ \\ell \\in \\mathbb{N}_+ \\,:\\,\t\\bigl\\lvert e_{i_{t+\\ell-1}}^{\\top} \\nabla f(x_{t+\\ell-1})\\bigr\\rvert\\geq \\mu\\norm{\\nabla f(x_{t+\\ell-1})} \\Bigr\\}.\n\t\t\\end{equation*}\n\t\tIt is clear that $\\bP(\\ell_0 >\\ell\\mid\\mathcal{F}_{t-1})\\leq (1-1/d)^\\ell$, since for each step the coordinate is randomly chosen. Hence, there exists some $L>0$, such that \n\t\t\\begin{equation*}\n\t\t\t\\bP(\\ell_0 \\leq L\\mid\\mathcal{F}_{t-1})\\geq 1-\\epsilon.\n\t\t\\end{equation*}\n\t\tWe finish the proof by setting $\\varrho_t = t + \\min\\{ \\ell_0, L \\}$\n\t\twhich has the desired property. \n\t\\end{proof}","sha256_utf8":"2cb23b47832c04700f6ee8de72c2f2bd06baee51d26c6d17970755d076e8b1d7"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"首次有效坐标的等待时间","source_units":[{"start":52946,"end":53260,"start_line":581,"end_line":584,"raw_tex":"\t\tFor any $t\\in\\bN$, use $\\ell_0$ to denote the smallest non-negative integer $\\ell$, such that\n\t\t\\begin{equation*}\n\t\t\t\\ell_0 = \\arg\\min_{\\ell} \\Bigl\\{ \\ell \\in \\mathbb{N}_+ \\,:\\,\t\\bigl\\lvert e_{i_{t+\\ell-1}}^{\\top} \\nabla f(x_{t+\\ell-1})\\bigr\\rvert\\geq \\mu\\norm{\\nabla f(x_{t+\\ell-1})} \\Bigr\\}.\n\t\t\\end{equation*}\n","sha256_utf8":"5ef291e9f244a5caf322f5763f769b9248342266fb25aa18a672cfbd3679fa59"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"几何尾界与有界截断","source_units":[{"start":53260,"end":53639,"start_line":585,"end_line":590,"raw_tex":"\t\tIt is clear that $\\bP(\\ell_0 >\\ell\\mid\\mathcal{F}_{t-1})\\leq (1-1/d)^\\ell$, since for each step the coordinate is randomly chosen. Hence, there exists some $L>0$, such that \n\t\t\\begin{equation*}\n\t\t\t\\bP(\\ell_0 \\leq L\\mid\\mathcal{F}_{t-1})\\geq 1-\\epsilon.\n\t\t\\end{equation*}\n\t\tWe finish the proof by setting $\\varrho_t = t + \\min\\{ \\ell_0, L \\}$\n\t\twhich has the desired property. \n","sha256_utf8":"b8f8b8c938136554db7ce256ddf7983d8c37796802f620749ccf959e58330754"}]},{"id":"S056","semantic_type":"lemma","kind":"lemma","title":"坐标不稳定投影的一致下界","source_units":[{"start":56185,"end":56636,"start_line":619,"end_line":627,"raw_tex":"\\begin{lemma}\\label{lem: Omega1}\n\t\tUnder Assumption~\\ref{Assump: non-zero-proj}, \n\t\tthere exist $\\delta>0$ and measurable $\\Omega_1^\\epsilon\\subset\\widetilde{\\Omega}$, where $\\widetilde{\\Omega}$ is from Theorem~\\ref{thm: MET}, \n\t\tsuch that $\\bP(\\Omega_1^\\epsilon)\\geq 1-\\epsilon$ \n\t\tand \n\t\t\\begin{equation*}\n\t\t\t\\norm{\\mathcal{P}^H_+(\\omega) e_i}\\geq\\delta,\\quad \\forall\\ \\omega\\in\\Omega_1^\\epsilon,\\ i\\in\\{1,2,\\dots,d\\}.\n\t\t\\end{equation*}\n\t\\end{lemma}","sha256_utf8":"35aa7d7fb1905abf3ebbe6d17619b7962120e94ff0ab50ec22cb33b322b0b003"}]},{"id":"S057","semantic_type":"proof","kind":"proof","title":"投影下界引理完整证明","source_units":[{"start":56640,"end":57264,"start_line":629,"end_line":641,"raw_tex":"\\begin{proof}\n\t\tAssumption~\\ref{Assump: non-zero-proj} implies that \n\t\t\\begin{equation*}\n\t\t\t\\bP\\bigl(\\{\\omega\\in\\widetilde{\\Omega}:\\norm{\\mathcal{P}^H_+(\\omega) e_i}>0,\\quad \\forall\\ i\\in\\{1,2,\\dots,d\\}\\}\\bigr) = 1\n\t\t\\end{equation*}\n\t\tNotice that \n\t\t\\begin{multline*}\n\t\t\t\\bigl\\{\\omega\\in\\widetilde{\\Omega}:\\norm{\\mathcal{P}^H_+(\\omega) e_i}>0,\\quad \\forall\\ i\\in\\{1,2,\\dots,d\\}\\bigr\\}\n\t\t\t= \\\\ \n\t\t\t=\\bigcup_{n\\in\\bNp}\\Bigl\\{\\omega\\in\\widetilde{\\Omega}:\\norm{\\mathcal{P}^H_+(\\omega) e_i}\\geq \\frac{1}{n},\\quad \\forall\\ i\\in\\{1,2,\\dots,d\\}\\Bigr\\}.\n\t\t\\end{multline*}\n\t\tThe Lemma follows from continuity of measure. \n\t\\end{proof}","sha256_utf8":"32c89ccc6733c938a2f3c6d2c8c2724c64d5bfbb66ff0a8e70b7157459555955"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"非零投影事件具有全概率","source_units":[{"start":56654,"end":56873,"start_line":630,"end_line":633,"raw_tex":"\t\tAssumption~\\ref{Assump: non-zero-proj} implies that \n\t\t\\begin{equation*}\n\t\t\t\\bP\\bigl(\\{\\omega\\in\\widetilde{\\Omega}:\\norm{\\mathcal{P}^H_+(\\omega) e_i}>0,\\quad \\forall\\ i\\in\\{1,2,\\dots,d\\}\\}\\bigr) = 1\n\t\t\\end{equation*}\n","sha256_utf8":"6e2d27d2e137282e725ed0d4619e2d58cfc463f4863fa65f8b04be564057d6ec"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"测度连续性给出统一阈值","source_units":[{"start":56873,"end":57252,"start_line":634,"end_line":640,"raw_tex":"\t\tNotice that \n\t\t\\begin{multline*}\n\t\t\t\\bigl\\{\\omega\\in\\widetilde{\\Omega}:\\norm{\\mathcal{P}^H_+(\\omega) e_i}>0,\\quad \\forall\\ i\\in\\{1,2,\\dots,d\\}\\bigr\\}\n\t\t\t= \\\\ \n\t\t\t=\\bigcup_{n\\in\\bNp}\\Bigl\\{\\omega\\in\\widetilde{\\Omega}:\\norm{\\mathcal{P}^H_+(\\omega) e_i}\\geq \\frac{1}{n},\\quad \\forall\\ i\\in\\{1,2,\\dots,d\\}\\Bigr\\}.\n\t\t\\end{multline*}\n\t\tThe Lemma follows from continuity of measure. \n","sha256_utf8":"9d254ff9befe7317ad3865dc53626afb199b15e0a02eecc50f48fca309d25d6c"}]},{"id":"S060","semantic_type":"lemma","kind":"lemma","title":"良好有限块具有高概率","source_units":[{"start":59483,"end":59702,"start_line":669,"end_line":671,"raw_tex":"\\begin{lemma}\\label{lem: T-Omega2}\n\t\tUnder the same assumptions of Lemma~\\ref{lem: Omega1}, there exists some $S_*>0$, such that for every $S \\in \\bNp, S \\geq S_*$, it holds $\\bP(\\Omega^S)\\geq 1-2\\epsilon$.\n\t\\end{lemma}","sha256_utf8":"106e7ae66b74216aae30dee1c1714d245d072c0c6c5622b677757e5fe85b4dd2"}]},{"id":"S061","semantic_type":"proof","kind":"proof","title":"良好块概率引理完整证明","source_units":[{"start":59706,"end":61770,"start_line":673,"end_line":705,"raw_tex":"\\begin{proof}\n\t\tFor $a.e.\\ \\omega\\in\\Omega$, it follows from Theorem~\\ref{thm: MET}, in particular \\eqref{conv: Lamb}, and standard matrix perturbation analysis (see e.g., \\cite[Theorem VI.2.1, Theorem VII.3.1]{bhatia}) that \n\t\t\\begin{equation}\\label{conv_lya_component}\n\t\t\t\\frac{1}{S}s_j(\\Phi^H(S,\\omega))\\rightarrow\\lambda_{\\mu(j)},\\quad S\\rightarrow\\infty.\n\t\t\\end{equation}\n\t\tfor any $j\\in\\{1,2,\\dots,d\\}$, and that\n\t\t\\begin{equation}\\label{conv_subspace}\n\t\t\t\\mathcal{P}^H_+(S,\\omega)\\rightarrow\\mathcal{P}^H_+(\\omega),\\quad S\\rightarrow\\infty.\n\t\t\\end{equation}\n\t\t\n\t\tBy Egorov's theorem, there exists $\\Omega_2^\\epsilon\\subset\\Omega_1^\\epsilon$ with $\\bP(\\Omega_2^\\epsilon) \\geq 1 - 2 \\epsilon$, such that the convergences in \\eqref{conv_lya_component} and \\eqref{conv_subspace} are both uniform on $\\Omega_2^\\epsilon$. Here $\\Omega_1^\\epsilon$ is as in Lemma~\\ref{lem: Omega1}. Therefore, for some $S_*$ sufficiently large, we have\n\t\t\\begin{equation*}\\label{uniconv: singval}\n\t\t\t\\left|\\frac{1}{S}\\log s_j\\left(\\Phi^H(S,\\omega)\\right)- \\lambda_{\\mu(j)}\\right|\\leq \\gamma,\\quad\\forall\\ j\\in\\{1,2,\\dots,d\\},\\ S\\geq S_*,\\ \\omega\\in\\Omega_2^\\epsilon,\n\t\t\\end{equation*}\n\t\tand\n\t\t\\begin{equation}\\label{uniconv: singspace}\n\t\t\t\\norm{\\mathcal{P}^H_+(S,\\omega)-\\mathcal{P}^H_+(\\omega)}\\leq \\frac{\\delta}{2},\\quad\\forall\\ S\\geq S_*,\\ \\omega\\in\\Omega_2^\\epsilon.\n\t\t\\end{equation}\n\t\tCombining Lemma~\\ref{lem: Omega1} and \\eqref{uniconv: singspace}, we obtain that\n\t\t\\begin{equation*}\n\t\t\t\\norm{\\mathcal{P}^H_+(S,\\omega)e_i}\\geq \\frac{\\delta}{2},\\quad \\forall\\ i\\in\\{1,2,\\dots,d\\},\\ \\forall\\ S\\geq S_*,\\ \\omega\\in\\Omega_2^\\epsilon.\n\t\t\\end{equation*}\n\t\t\n\t\tFor any $S\\geq S_*$, \n\t\tby the definition of $\\Omega^S$, it holds that \n\t\t\\begin{equation*}\n\t\t\t\\Omega_2^\\epsilon\\subset\\Omega^S\\times\\Bigl(\\bigtimes_{t\\geq S}\\Omega_t\\Bigr),\n\t\t\\end{equation*}\n\t\twhich implies the desired estimate \n\t\t\\begin{equation*}\n\t\t\t\\bP(\\Omega^S)=\\bP\\biggl(\\Omega^S\\times\\Bigl(\\bigtimes_{t\\geq S}\\Omega_t\\Bigr)\\biggr)\\geq\\bP(\\Omega_2^\\epsilon)\\geq 1-2\\epsilon. %\\qedhere\n\t\t\\end{equation*}\n\t\\end{proof}","sha256_utf8":"5904ce0f28ecd6f5e1910f9d5df4e4c8fef80b76b89590c42bcc456608df700c"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"遍历极限与谱投影收敛","source_units":[{"start":59720,"end":60271,"start_line":674,"end_line":681,"raw_tex":"\t\tFor $a.e.\\ \\omega\\in\\Omega$, it follows from Theorem~\\ref{thm: MET}, in particular \\eqref{conv: Lamb}, and standard matrix perturbation analysis (see e.g., \\cite[Theorem VI.2.1, Theorem VII.3.1]{bhatia}) that \n\t\t\\begin{equation}\\label{conv_lya_component}\n\t\t\t\\frac{1}{S}s_j(\\Phi^H(S,\\omega))\\rightarrow\\lambda_{\\mu(j)},\\quad S\\rightarrow\\infty.\n\t\t\\end{equation}\n\t\tfor any $j\\in\\{1,2,\\dots,d\\}$, and that\n\t\t\\begin{equation}\\label{conv_subspace}\n\t\t\t\\mathcal{P}^H_+(S,\\omega)\\rightarrow\\mathcal{P}^H_+(\\omega),\\quad S\\rightarrow\\infty.\n\t\t\\end{equation}\n","sha256_utf8":"e9a77dae98799f6ee971b08184dfa18acfb38f4d1a3c64fcb49a2a52ef22c30b"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"埃戈罗夫定理给出一致控制","source_units":[{"start":60274,"end":61344,"start_line":683,"end_line":694,"raw_tex":"\t\tBy Egorov's theorem, there exists $\\Omega_2^\\epsilon\\subset\\Omega_1^\\epsilon$ with $\\bP(\\Omega_2^\\epsilon) \\geq 1 - 2 \\epsilon$, such that the convergences in \\eqref{conv_lya_component} and \\eqref{conv_subspace} are both uniform on $\\Omega_2^\\epsilon$. Here $\\Omega_1^\\epsilon$ is as in Lemma~\\ref{lem: Omega1}. Therefore, for some $S_*$ sufficiently large, we have\n\t\t\\begin{equation*}\\label{uniconv: singval}\n\t\t\t\\left|\\frac{1}{S}\\log s_j\\left(\\Phi^H(S,\\omega)\\right)- \\lambda_{\\mu(j)}\\right|\\leq \\gamma,\\quad\\forall\\ j\\in\\{1,2,\\dots,d\\},\\ S\\geq S_*,\\ \\omega\\in\\Omega_2^\\epsilon,\n\t\t\\end{equation*}\n\t\tand\n\t\t\\begin{equation}\\label{uniconv: singspace}\n\t\t\t\\norm{\\mathcal{P}^H_+(S,\\omega)-\\mathcal{P}^H_+(\\omega)}\\leq \\frac{\\delta}{2},\\quad\\forall\\ S\\geq S_*,\\ \\omega\\in\\Omega_2^\\epsilon.\n\t\t\\end{equation}\n\t\tCombining Lemma~\\ref{lem: Omega1} and \\eqref{uniconv: singspace}, we obtain that\n\t\t\\begin{equation*}\n\t\t\t\\norm{\\mathcal{P}^H_+(S,\\omega)e_i}\\geq \\frac{\\delta}{2},\\quad \\forall\\ i\\in\\{1,2,\\dots,d\\},\\ \\forall\\ S\\geq S_*,\\ \\omega\\in\\Omega_2^\\epsilon.\n\t\t\\end{equation*}\n","sha256_utf8":"b7c8d6f57591e67aa355989612b6971452b33ca912e4c2e119fda32ac49f067c"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"集合包含推出良好块概率界","source_units":[{"start":61347,"end":61758,"start_line":696,"end_line":704,"raw_tex":"\t\tFor any $S\\geq S_*$, \n\t\tby the definition of $\\Omega^S$, it holds that \n\t\t\\begin{equation*}\n\t\t\t\\Omega_2^\\epsilon\\subset\\Omega^S\\times\\Bigl(\\bigtimes_{t\\geq S}\\Omega_t\\Bigr),\n\t\t\\end{equation*}\n\t\twhich implies the desired estimate \n\t\t\\begin{equation*}\n\t\t\t\\bP(\\Omega^S)=\\bP\\biggl(\\Omega^S\\times\\Bigl(\\bigtimes_{t\\geq S}\\Omega_t\\Bigr)\\biggr)\\geq\\bP(\\Omega_2^\\epsilon)\\geq 1-2\\epsilon. %\\qedhere\n\t\t\\end{equation*}\n","sha256_utf8":"49cad74779c0e068b4f3d3f671132c772dd6cba6e9c1d5aad4408f84de43bcdb"}]},{"id":"S065","semantic_type":"lemma","kind":"lemma","title":"随机步长产生显著不稳定分量","source_units":[{"start":62243,"end":62914,"start_line":710,"end_line":716,"raw_tex":"\\begin{lemma}\\label{lem: nontrial-proj}\n\t\tFor any $S\\in\\bNp$, $x_{\\varrho_t-1}$, $i_{\\varrho_t-1}$, \n\t\tand $(i,\\alpha)_{\\varrho_t:\\varrho_t+S-1} \\in\\Omega^S$, there exists $I\\subset[\\alpha_{\\min},\\alpha_{\\max}]$ with $m(I)\\geq (1-\\epsilon)(\\alpha_{\\max}-\\alpha_{\\min})$ where $m(\\cdot)$ is the Lebesgue measure,  such that for any $\\alpha_{\\varrho_t-1}\\in I$, it holds that \n\t\t\\begin{equation}\\label{eq: nontrial-proj}\n\t\t\t\\norm{\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\right)x_{\\varrho_t}}\\geq \\frac{\\epsilon\\delta (\\alpha_{\\max}-\\alpha_{\\min})}{4}\\bigl\\lvert e_{i_{\\varrho_t-1}}^{\\top} \\nabla f(x_{\\varrho_t-1})\\bigr\\rvert.\n\t\t\\end{equation}\n\t\\end{lemma}","sha256_utf8":"0f195d5ec1f8ec81609f8a4aa87568e936e1c02fd1c3199107374fedf6965438"}]},{"id":"S066","semantic_type":"proof","kind":"proof","title":"随机步长投影引理完整证明","source_units":[{"start":62918,"end":64832,"start_line":718,"end_line":743,"raw_tex":"\\begin{proof}\n\t\tWe assume that $\\bigl\\lvert e_{i_{\\varrho_t-1}}^{\\top} \\nabla f(x_{\\varrho_t-1})\\bigr\\rvert\\neq 0$; otherwise the result is trivial. \n\t\t\n\t\tFor simplicity of notation, we write \n\t\t\\begin{equation*}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\right)x_{\\varrho_t} & = \n\t\t\t\t\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\right)x_{\\varrho_t-1} \\\\\n\t\t\t\t& \\qquad \n\t\t\t\t- \\alpha_{\\varrho_t - 1} e_{i_{\\varrho_t-1}}^{\\top}\\nabla f(x_{\\varrho_t-1})\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\right)e_{i_{\\varrho_t-1}} \\\\\n\t\t\t\t& =: y_2-\\alpha_{\\varrho_t-1} y_1,\n\t\t\t\\end{aligned}\n\t\t\\end{equation*}\n\t\twhere the last line defines $y_1$ and $y_2$. Using the short-hand notation \n\t\t\\begin{equation*}\n\t\t\tr=\\frac{\\epsilon\\delta (\\alpha_{\\max}-\\alpha_{\\min})}{4}\\bigl\\lvert e_{i_{\\varrho_t-1}}^{\\top} \\nabla f(x_{\\varrho_t-1})\\bigr\\rvert, \n\t\t\\end{equation*}\n\t\twe observe then \\eqref{eq: nontrial-proj} holds if and only if $\\alpha_{\\varrho_t-1} y_1$ is not located in a ball with radius $r$ centered at $y_2$.\n\t\t\n\t\tIt follows from the definition of $\\Omega^S$ and \\eqref{close: singular space} that $ \\norm{\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\right)e_{i_{\\varrho_t-1}}} \\geq \\frac{\\delta}{2}$, \n\t\twhich then leads to\n\t\t\\begin{equation*}\n\t\t\t\\norm{y_1}\\geq \\frac{\\delta}{2}\\bigl\\lvert e_{i_{\\varrho_t-1}}^{\\top} \\nabla f(x_{\\varrho_t-1})\\bigr\\rvert=:\\frac{2r}{\\epsilon(\\alpha_{\\max}-\\alpha_{\\min})}.\n\t\t\\end{equation*}\n\t\tThus, the set of $\\alpha_{\\varrho_t-1}$ such that $\\alpha_{\\varrho_t -1} y_1 \\in \\mathcal{B}_r(y_2)$ consists of an interval $J$ in $\\bR$ with $\\|\\sup(J)\\cdot y_1 - \\inf(J)\\cdot y_1\\|\\leq 2r$ as the diameter of $\\mathcal{B}_r(y_2)$ is $2r$, which implies $m(J)\\leq 2r/\\norm{y_1}\\leq \\epsilon(\\alpha_{\\max}-\\alpha_{\\min})$. The lemma is proved then by setting $I=[\\alpha_{\\max}-\\alpha_{\\min}]\\backslash J$.\n\t\\end{proof}","sha256_utf8":"1ef03a784d2b33aebff839547f076f6cad1e158d3105fd0e5b25b06e5de508d1"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"投影仿射表达与避球条件","source_units":[{"start":62932,"end":63984,"start_line":719,"end_line":735,"raw_tex":"\t\tWe assume that $\\bigl\\lvert e_{i_{\\varrho_t-1}}^{\\top} \\nabla f(x_{\\varrho_t-1})\\bigr\\rvert\\neq 0$; otherwise the result is trivial. \n\t\t\n\t\tFor simplicity of notation, we write \n\t\t\\begin{equation*}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\right)x_{\\varrho_t} & = \n\t\t\t\t\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\right)x_{\\varrho_t-1} \\\\\n\t\t\t\t& \\qquad \n\t\t\t\t- \\alpha_{\\varrho_t - 1} e_{i_{\\varrho_t-1}}^{\\top}\\nabla f(x_{\\varrho_t-1})\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\right)e_{i_{\\varrho_t-1}} \\\\\n\t\t\t\t& =: y_2-\\alpha_{\\varrho_t-1} y_1,\n\t\t\t\\end{aligned}\n\t\t\\end{equation*}\n\t\twhere the last line defines $y_1$ and $y_2$. Using the short-hand notation \n\t\t\\begin{equation*}\n\t\t\tr=\\frac{\\epsilon\\delta (\\alpha_{\\max}-\\alpha_{\\min})}{4}\\bigl\\lvert e_{i_{\\varrho_t-1}}^{\\top} \\nabla f(x_{\\varrho_t-1})\\bigr\\rvert, \n\t\t\\end{equation*}\n\t\twe observe then \\eqref{eq: nontrial-proj} holds if and only if $\\alpha_{\\varrho_t-1} y_1$ is not located in a ball with radius $r$ centered at $y_2$.\n","sha256_utf8":"96d0da8dbdd5b98a8a10a469a26f886bb1ee2710e3d18ffc0dfd75f2eca95371"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"坏步长区间的长度界","source_units":[{"start":63987,"end":64820,"start_line":737,"end_line":742,"raw_tex":"\t\tIt follows from the definition of $\\Omega^S$ and \\eqref{close: singular space} that $ \\norm{\\mathcal{P}^H_+\\left((i,\\alpha)_{\\varrho_t:\\varrho_t+S-1}\\right)e_{i_{\\varrho_t-1}}} \\geq \\frac{\\delta}{2}$, \n\t\twhich then leads to\n\t\t\\begin{equation*}\n\t\t\t\\norm{y_1}\\geq \\frac{\\delta}{2}\\bigl\\lvert e_{i_{\\varrho_t-1}}^{\\top} \\nabla f(x_{\\varrho_t-1})\\bigr\\rvert=:\\frac{2r}{\\epsilon(\\alpha_{\\max}-\\alpha_{\\min})}.\n\t\t\\end{equation*}\n\t\tThus, the set of $\\alpha_{\\varrho_t-1}$ such that $\\alpha_{\\varrho_t -1} y_1 \\in \\mathcal{B}_r(y_2)$ consists of an interval $J$ in $\\bR$ with $\\|\\sup(J)\\cdot y_1 - \\inf(J)\\cdot y_1\\|\\leq 2r$ as the diameter of $\\mathcal{B}_r(y_2)$ is $2r$, which implies $m(J)\\leq 2r/\\norm{y_1}\\leq \\epsilon(\\alpha_{\\max}-\\alpha_{\\min})$. The lemma is proved then by setting $I=[\\alpha_{\\max}-\\alpha_{\\min}]\\backslash J$.\n","sha256_utf8":"d5e203a95fc624ef41648c31b24eb34d57379449d73427d1fe7a0cc3bbddab91"}]},{"id":"S069","semantic_type":"theorem","kind":"theorem","title":"单个严格鞍点的收敛概率为零","source_units":[{"start":73392,"end":73759,"start_line":868,"end_line":874,"raw_tex":"\\begin{theorem}\\label{thm: single-pt}\n\t\tSuppose that Assumption~\\ref{Assump: Hess_bdd}, \\ref{Assump: non-degenerate}, and \\ref{Assump: non-zero-proj} hold and that $0<\\alpha_{\\min}<\\alpha_{\\max}<1/M$. Then for every $x^*\\in\\calA$ \n\t\tand every $x_0\\in\\bR^d\\backslash\\{x^*\\}$, it holds that\n\t\t\\begin{equation*}\n\t\t\t\\bP(\\calC(x^*,x_0))=0.\n\t\t\\end{equation*}\n\t\\end{theorem}","sha256_utf8":"f118e581a9045f85feb0b97b824beaeda10e7fe85a647180ca0d3925fc3fce4b"}]},{"id":"S070","semantic_type":"proof","kind":"proof","title":"单鞍点定理完整证明","source_units":[{"start":73763,"end":76220,"start_line":876,"end_line":918,"raw_tex":"\\begin{proof}\n\t\tWithout loss of generality, we assume that $x^*=0$. Conditioned on $\\mathcal{F}_{t-1}$ with $x_t\\in V$, where $V$ can be assumed to be bounded, Theorem~\\ref{thm: local-growth} states that with probability at least $1-4\\epsilon$, \n\t\t\\begin{equation*}\n\t\t\t\\norm{x_{t+T}} \\geq A \\norm{x_t},\n\t\t\\end{equation*}\n\t\twhere to simplify notation we denote \n\t\t\\begin{equation*}\n\t\t\tA := \\exp\\biggl(\\frac{6\\epsilon}{1 - 6 \\epsilon} \\bigl\\lvert \\log(1-M\\alpha_{\\max}) \\bigr\\rvert \\, T\\biggr)\n\t\t\\end{equation*}\n\t\tthe amplification factor appearing on the right-hand side of \\eqref{local-growth}. \n\t\tNotice also that, due to \\eqref{eq:compareiterate}, we always have\n\t\t\\begin{equation*}\n\t\t\t\\norm{x_{t+T}}\\geq (r_-)^{T}\\norm{x_t}.\n\t\t\\end{equation*}\n\t\tIt suffices to show that for any $x_0\\in V\\backslash\\{x^*\\}$, with probability $1$ there exists some $t\\in\\bNp$ such that $x_t\\notin V$.\n\t\t\n\t\tLet us consider the iterates every $T$ steps: Denote $y_t=x_{T t}$ and $\\mathcal{G}_t=\\mathcal{F}_{T t-1}$ for $t\\in\\bN$. Denote stopping time\n\t\t\\begin{equation*}\n\t\t\t\\rho=\\inf\\{t\\in\\bN:y_t\\notin V\\},\n\t\t\\end{equation*}\n\t\tit suffices to show that $\\bP(\\rho<\\infty)=1$. We define a sequence of random variables $I_t$ as follows:\n\t\t\\begin{equation*}\n\t\t\tI_t(\\omega)=\\begin{cases}1,&\\text{if }\\norm{y_{t+1}}\\geq A\\norm{y_t},\\\\\n\t\t\t\t0,&\\text{otherwise}.\\end{cases}\n\t\t\\end{equation*}\n\t\tBy the discussion in the beginning of the proof, we have  \n\t\t\\begin{equation*}\n\t\t\t\\bP(I_t = 1\\ |\\ \\mathcal{G}_t, t<\\rho)\\geq 1-4\\epsilon,\n\t\t\\end{equation*}\n\t\tand moreover, setting $S_t(\\omega)=\\sum_{0\\leq s<t} I_t(\\omega)$, we have for $t<\\rho(\\omega)$,\n\t\t\\begin{equation*}\n\t\t\t\\frac{\\norm{y_t}}{\\norm{y_0}} \\geq A^{S_t(\\omega)} \\cdot (r_-)^{T(t-S_t(\\omega))}.\n\t\t\\end{equation*}\n\t\tDenote $R:=\\sup_{x\\in V}\\norm{x}<\\infty$. Since  $(1-5\\epsilon)\\log A+5\\epsilon T\\log r_->0$, there exists $t_*\\in\\bN$, such that\n\t\t\\begin{equation*}\n\t\t\t\\left(A^{1-5\\epsilon}\\cdot (r_-)^{5\\epsilon T}\\right)^t>\\frac{R}{\\norm{y_0}},\\quad\\forall\\ t\\geq t_*.\n\t\t\\end{equation*}\n\t\tTherefore, for any $t\\geq t_*$, it holds that \n\t\t\\begin{equation*}\n\t\t\t\\bP(\\rho> t) = \\bP\\bigl(\\rho> t,S_t\\leq (1-5\\epsilon)t\\bigr)\\leq \\sum_{i\\leq (1-5\\epsilon)t} \\binom{t}{i} (1-4\\epsilon)^i (4\\epsilon)^{t-i}.\n\t\t\\end{equation*}\n\t\tAs we will show in the next lemma that the right-hand side of above goes to $0$ as $t \\to \\infty$, and thus $\\lim_{t\\rightarrow\\infty}\\bP(\\rho> t)=0$, which implies that $\\bP(\\rho<\\infty)=1$.\n\t\\end{proof}","sha256_utf8":"996daa1c545ea7f85fe8802596f5f5aa4335bfcfd3092f3d4a06428019d10476"}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","title":"块放大概率与最坏收缩界","source_units":[{"start":73777,"end":74509,"start_line":877,"end_line":889,"raw_tex":"\t\tWithout loss of generality, we assume that $x^*=0$. Conditioned on $\\mathcal{F}_{t-1}$ with $x_t\\in V$, where $V$ can be assumed to be bounded, Theorem~\\ref{thm: local-growth} states that with probability at least $1-4\\epsilon$, \n\t\t\\begin{equation*}\n\t\t\t\\norm{x_{t+T}} \\geq A \\norm{x_t},\n\t\t\\end{equation*}\n\t\twhere to simplify notation we denote \n\t\t\\begin{equation*}\n\t\t\tA := \\exp\\biggl(\\frac{6\\epsilon}{1 - 6 \\epsilon} \\bigl\\lvert \\log(1-M\\alpha_{\\max}) \\bigr\\rvert \\, T\\biggr)\n\t\t\\end{equation*}\n\t\tthe amplification factor appearing on the right-hand side of \\eqref{local-growth}. \n\t\tNotice also that, due to \\eqref{eq:compareiterate}, we always have\n\t\t\\begin{equation*}\n\t\t\t\\norm{x_{t+T}}\\geq (r_-)^{T}\\norm{x_t}.\n\t\t\\end{equation*}\n","sha256_utf8":"0432dee347dffafaac16216715027d74f3120391f6861e373e69a27b73865868"}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","title":"出邻域停止时间与成功指标","source_units":[{"start":74509,"end":75128,"start_line":890,"end_line":900,"raw_tex":"\t\tIt suffices to show that for any $x_0\\in V\\backslash\\{x^*\\}$, with probability $1$ there exists some $t\\in\\bNp$ such that $x_t\\notin V$.\n\t\t\n\t\tLet us consider the iterates every $T$ steps: Denote $y_t=x_{T t}$ and $\\mathcal{G}_t=\\mathcal{F}_{T t-1}$ for $t\\in\\bN$. Denote stopping time\n\t\t\\begin{equation*}\n\t\t\t\\rho=\\inf\\{t\\in\\bN:y_t\\notin V\\},\n\t\t\\end{equation*}\n\t\tit suffices to show that $\\bP(\\rho<\\infty)=1$. We define a sequence of random variables $I_t$ as follows:\n\t\t\\begin{equation*}\n\t\t\tI_t(\\omega)=\\begin{cases}1,&\\text{if }\\norm{y_{t+1}}\\geq A\\norm{y_t},\\\\\n\t\t\t\t0,&\\text{otherwise}.\\end{cases}\n\t\t\\end{equation*}\n","sha256_utf8":"78cb8623a96fe2cfeb3ac8e3a126cf42094ce6b8008564e24c522fa47db527fc"}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","title":"成功块计数控制迭代范数","source_units":[{"start":75128,"end":75508,"start_line":901,"end_line":908,"raw_tex":"\t\tBy the discussion in the beginning of the proof, we have  \n\t\t\\begin{equation*}\n\t\t\t\\bP(I_t = 1\\ |\\ \\mathcal{G}_t, t<\\rho)\\geq 1-4\\epsilon,\n\t\t\\end{equation*}\n\t\tand moreover, setting $S_t(\\omega)=\\sum_{0\\leq s<t} I_t(\\omega)$, we have for $t<\\rho(\\omega)$,\n\t\t\\begin{equation*}\n\t\t\t\\frac{\\norm{y_t}}{\\norm{y_0}} \\geq A^{S_t(\\omega)} \\cdot (r_-)^{T(t-S_t(\\omega))}.\n\t\t\\end{equation*}\n","sha256_utf8":"3bec288737c9a11294fa76752c7aa4099e2dbd37cc37622cb9ddd9ab9267c29e"}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","title":"二项尾界推出几乎必然离开","source_units":[{"start":75508,"end":76208,"start_line":909,"end_line":917,"raw_tex":"\t\tDenote $R:=\\sup_{x\\in V}\\norm{x}<\\infty$. Since  $(1-5\\epsilon)\\log A+5\\epsilon T\\log r_->0$, there exists $t_*\\in\\bN$, such that\n\t\t\\begin{equation*}\n\t\t\t\\left(A^{1-5\\epsilon}\\cdot (r_-)^{5\\epsilon T}\\right)^t>\\frac{R}{\\norm{y_0}},\\quad\\forall\\ t\\geq t_*.\n\t\t\\end{equation*}\n\t\tTherefore, for any $t\\geq t_*$, it holds that \n\t\t\\begin{equation*}\n\t\t\t\\bP(\\rho> t) = \\bP\\bigl(\\rho> t,S_t\\leq (1-5\\epsilon)t\\bigr)\\leq \\sum_{i\\leq (1-5\\epsilon)t} \\binom{t}{i} (1-4\\epsilon)^i (4\\epsilon)^{t-i}.\n\t\t\\end{equation*}\n\t\tAs we will show in the next lemma that the right-hand side of above goes to $0$ as $t \\to \\infty$, and thus $\\lim_{t\\rightarrow\\infty}\\bP(\\rho> t)=0$, which implies that $\\bP(\\rho<\\infty)=1$.\n","sha256_utf8":"51e795bc3bb47fe7835303d7a17f0f7d60439e09b032d6ee4f0af656135fa3cf"}]},{"id":"S075","semantic_type":"lemma","kind":"lemma","title":"二项下尾概率趋零","source_units":[{"start":76224,"end":76460,"start_line":920,"end_line":925,"raw_tex":"\\begin{lemma}\\label{lem: binomial}\n\t\tFor any $\\epsilon\\in(0,1/4)$, it holds that\n\t\t\\begin{equation*}\n\t\t\t\\lim_{t\\rightarrow\\infty}\\sum_{i\\leq(1-5\\epsilon)t} \\binom{t}{i} (1-4\\epsilon)^i (4\\epsilon)^{t-i}=0.\n\t\t\\end{equation*}\n\t\\end{lemma}","sha256_utf8":"d3f4834bd54bc64bae4371f4cbb09367aa42c7af3b3b414ae0549ece1ecb1255"}]},{"id":"S076","semantic_type":"proof","kind":"proof","title":"二项尾引理完整证明","source_units":[{"start":76464,"end":76992,"start_line":927,"end_line":933,"raw_tex":"\\begin{proof}\n\t\tLet $X_0,X_1,X_2,\\dots$ be a sequence of i.i.d. random variables with $X_i$ being a Bernoulli random variable with expectation $1-4\\epsilon$. Denote the average $\\bar{X}_t=\\frac{1}{t} \\sum_{0\\leq s<t} X_s$. The weak law of large numbers yields that\n\t\t\\begin{equation*}\n\t\t\t\\sum_{i\\leq(1-5\\epsilon)t} \\binom{t}{i} (1-4\\epsilon)^i (4\\epsilon)^{t-i}=\\bP\\left(\\bar{X}_t\\leq 1-5\\epsilon\\right)\\leq \\bP\\left(|\\bar{X}_t-\\bE X_0|\\geq \\epsilon\\right)\\rightarrow 0,\n\t\t\\end{equation*}\n\t\tas $t\\rightarrow\\infty$.\n\t\\end{proof}","sha256_utf8":"61ff2bcf462d8143a12711d37ff2e49201d9aee9c5a8cdd2bbc9b23fcc891e9b"}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","title":"伯努利均值与弱大数定律","source_units":[{"start":76478,"end":76980,"start_line":928,"end_line":932,"raw_tex":"\t\tLet $X_0,X_1,X_2,\\dots$ be a sequence of i.i.d. random variables with $X_i$ being a Bernoulli random variable with expectation $1-4\\epsilon$. Denote the average $\\bar{X}_t=\\frac{1}{t} \\sum_{0\\leq s<t} X_s$. The weak law of large numbers yields that\n\t\t\\begin{equation*}\n\t\t\t\\sum_{i\\leq(1-5\\epsilon)t} \\binom{t}{i} (1-4\\epsilon)^i (4\\epsilon)^{t-i}=\\bP\\left(\\bar{X}_t\\leq 1-5\\epsilon\\right)\\leq \\bP\\left(|\\bar{X}_t-\\bE X_0|\\geq \\epsilon\\right)\\rightarrow 0,\n\t\t\\end{equation*}\n\t\tas $t\\rightarrow\\infty$.\n","sha256_utf8":"78cfd54d663d78c5696742a684e9f0a19f45fa55adb376c36fc6d6b3057543fd"}]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","title":"乘法遍历定理边界事实","source_units":[],"declared_external":true,"declaration_text":["For a measurable invertible matrix cocycle with integrable positive parts of log norms of its generator and inverse, the multiplicative ergodic theorem gives the positive-definite limiting root matrix, its measurable Lyapunov spectrum and filtration, vector growth rates, forward invariance of the filtration, and almost surely constant spectral data when the base is ergodic, as stated in B0016."]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","title":"对称矩阵谱扰动","source_units":[],"declared_external":true,"declaration_text":["If real symmetric matrices converge in operator norm, their ordered eigenvalues converge; spectral projections onto a cluster separated by a spectral gap converge in operator norm. Applied to the limiting root matrices, this controls logarithmic singular-value growth rates and the positive-exponent spectral subspace."]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","title":"标准概率极限定理","source_units":[],"declared_external":true,"declaration_text":["Integrable independent identically distributed variables obey the strong and weak laws of large numbers. Almost sure convergence on a finite measure space is uniform outside a set of arbitrarily small measure by Egorov's theorem. Probability is continuous under increasing unions. Independent tail events have probability zero or one; hence the independent product shift is ergodic."]},{"id":"S081","semantic_type":"external_fact","kind":"external_fact","title":"光滑函数的局部估计","source_units":[],"declared_external":true,"declaration_text":["For a C² function, the gradient admits its first-order Taylor expansion and an integral Hessian representation. A uniformly bounded Hessian makes the gradient Lipschitz. At a critical point with invertible Hessian, the gradient norm is locally bounded below by a positive constant times the distance to that point."]},{"id":"S082","semantic_type":"external_fact","kind":"external_fact","title":"离散集合与零测可数并","source_units":[],"declared_external":true,"declaration_text":["A subset of Euclidean space whose every point is isolated is countable, by second countability. A countable union of measurable events of probability zero has probability zero."]},{"id":"S083","semantic_type":"external_fact","kind":"external_fact","title":"条件伯努利计数比较","source_units":[],"declared_external":true,"declaration_text":["For adapted zero-one trials with conditional success probability at least p, the probability of at most k successes in t trials is bounded above by the corresponding Binomial(t,p) lower tail. If the bound is imposed only before a stopping time, the probability of surviving through t with at most k successes obeys the same upper bound."]},{"id":"S084","semantic_type":"external_fact","kind":"external_fact","title":"矩阵范数与直线测度估计","source_units":[],"declared_external":true,"declaration_text":["The Sherman-Morrison formula gives the inverse of an invertible rank-one perturbation of the identity. Singular values bound matrix action on the corresponding singular subspaces, and images of orthogonal right singular components are orthogonal. For nonzero v, the scalar set {a: ||av-w||<r} is an interval of length at most 2r/||v||."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S024","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S078","target_id":"S006","kind":"supports"},{"source_id":"S002","target_id":"S007","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S002","target_id":"S009","kind":"supports"},{"source_id":"S004","target_id":"S009","kind":"supports"},{"source_id":"S007","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S006","target_id":"S011","kind":"supports"},{"source_id":"S007","target_id":"S011","kind":"supports"},{"source_id":"S008","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S080","target_id":"S011","kind":"supports"},{"source_id":"S084","target_id":"S011","kind":"supports"},{"source_id":"S006","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S016","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S016","kind":"supports"},{"source_id":"S009","target_id":"S017","kind":"supports"},{"source_id":"S014","target_id":"S017","kind":"supports"},{"source_id":"S010","target_id":"S018","kind":"supports"},{"source_id":"S014","target_id":"S019","kind":"supports"},{"source_id":"S012","target_id":"S020","kind":"supports"},{"source_id":"S013","target_id":"S020","kind":"supports"},{"source_id":"S027","target_id":"S020","kind":"supports"},{"source_id":"S006","target_id":"S021","kind":"supports"},{"source_id":"S007","target_id":"S021","kind":"supports"},{"source_id":"S010","target_id":"S021","kind":"supports"},{"source_id":"S012","target_id":"S021","kind":"supports"},{"source_id":"S006","target_id":"S022","kind":"supports"},{"source_id":"S020","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S056","target_id":"S022","kind":"supports"},{"source_id":"S007","target_id":"S023","kind":"supports"},{"source_id":"S054","target_id":"S023","kind":"supports"},{"source_id":"S055","target_id":"S023","kind":"supports"},{"source_id":"S013","target_id":"S024","kind":"supports"},{"source_id":"S015","target_id":"S024","kind":"supports"},{"source_id":"S016","target_id":"S024","kind":"supports"},{"source_id":"S017","target_id":"S024","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S015","target_id":"S026","kind":"supports"},{"source_id":"S017","target_id":"S026","kind":"supports"},{"source_id":"S069","target_id":"S026","kind":"supports"},{"source_id":"S081","target_id":"S026","kind":"supports"},{"source_id":"S082","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S028","kind":"supports"},{"source_id":"S031","target_id":"S028","kind":"supports"},{"source_id":"S007","target_id":"S029","kind":"supports"},{"source_id":"S018","target_id":"S029","kind":"supports"},{"source_id":"S032","target_id":"S029","kind":"supports"},{"source_id":"S034","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S018","target_id":"S030","kind":"supports"},{"source_id":"S084","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S006","target_id":"S031","kind":"supports"},{"source_id":"S011","target_id":"S031","kind":"supports"},{"source_id":"S080","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S035","target_id":"S033","kind":"supports"},{"source_id":"S036","target_id":"S033","kind":"supports"},{"source_id":"S037","target_id":"S033","kind":"supports"},{"source_id":"S018","target_id":"S034","kind":"supports"},{"source_id":"S018","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S034","target_id":"S037","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S084","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S041","target_id":"S039","kind":"supports"},{"source_id":"S042","target_id":"S039","kind":"supports"},{"source_id":"S035","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S084","target_id":"S042","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S044","kind":"supports"},{"source_id":"S049","target_id":"S044","kind":"supports"},{"source_id":"S050","target_id":"S044","kind":"supports"},{"source_id":"S051","target_id":"S044","kind":"supports"},{"source_id":"S010","target_id":"S045","kind":"supports"},{"source_id":"S013","target_id":"S045","kind":"supports"},{"source_id":"S015","target_id":"S045","kind":"supports"},{"source_id":"S019","target_id":"S045","kind":"supports"},{"source_id":"S081","target_id":"S045","kind":"supports"},{"source_id":"S009","target_id":"S046","kind":"supports"},{"source_id":"S013","target_id":"S046","kind":"supports"},{"source_id":"S019","target_id":"S046","kind":"supports"},{"source_id":"S081","target_id":"S046","kind":"supports"},{"source_id":"S020","target_id":"S047","kind":"supports"},{"source_id":"S045","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S052","target_id":"S047","kind":"supports"},{"source_id":"S056","target_id":"S047","kind":"supports"},{"source_id":"S060","target_id":"S047","kind":"supports"},{"source_id":"S009","target_id":"S048","kind":"supports"},{"source_id":"S010","target_id":"S048","kind":"supports"},{"source_id":"S013","target_id":"S048","kind":"supports"},{"source_id":"S045","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S081","target_id":"S048","kind":"supports"},{"source_id":"S021","target_id":"S049","kind":"supports"},{"source_id":"S046","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S023","target_id":"S050","kind":"supports"},{"source_id":"S045","target_id":"S050","kind":"supports"},{"source_id":"S046","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S060","target_id":"S050","kind":"supports"},{"source_id":"S065","target_id":"S050","kind":"supports"},{"source_id":"S020","target_id":"S051","kind":"supports"},{"source_id":"S021","target_id":"S051","kind":"supports"},{"source_id":"S022","target_id":"S051","kind":"supports"},{"source_id":"S045","target_id":"S051","kind":"supports"},{"source_id":"S046","target_id":"S051","kind":"supports"},{"source_id":"S047","target_id":"S051","kind":"supports"},{"source_id":"S049","target_id":"S051","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S084","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S055","target_id":"S053","kind":"supports"},{"source_id":"S007","target_id":"S054","kind":"supports"},{"source_id":"S009","target_id":"S054","kind":"supports"},{"source_id":"S007","target_id":"S055","kind":"supports"},{"source_id":"S020","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S059","target_id":"S057","kind":"supports"},{"source_id":"S012","target_id":"S058","kind":"supports"},{"source_id":"S016","target_id":"S058","kind":"supports"},{"source_id":"S020","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S080","target_id":"S059","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S063","target_id":"S061","kind":"supports"},{"source_id":"S064","target_id":"S061","kind":"supports"},{"source_id":"S006","target_id":"S062","kind":"supports"},{"source_id":"S011","target_id":"S062","kind":"supports"},{"source_id":"S012","target_id":"S062","kind":"supports"},{"source_id":"S021","target_id":"S062","kind":"supports"},{"source_id":"S079","target_id":"S062","kind":"supports"},{"source_id":"S020","target_id":"S063","kind":"supports"},{"source_id":"S056","target_id":"S063","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S080","target_id":"S063","kind":"supports"},{"source_id":"S022","target_id":"S064","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"},{"source_id":"S067","target_id":"S066","kind":"supports"},{"source_id":"S068","target_id":"S066","kind":"supports"},{"source_id":"S009","target_id":"S067","kind":"supports"},{"source_id":"S021","target_id":"S067","kind":"supports"},{"source_id":"S052","target_id":"S067","kind":"supports"},{"source_id":"S022","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S084","target_id":"S068","kind":"supports"},{"source_id":"S070","target_id":"S069","kind":"supports"},{"source_id":"S071","target_id":"S070","kind":"supports"},{"source_id":"S072","target_id":"S070","kind":"supports"},{"source_id":"S073","target_id":"S070","kind":"supports"},{"source_id":"S074","target_id":"S070","kind":"supports"},{"source_id":"S043","target_id":"S071","kind":"supports"},{"source_id":"S046","target_id":"S071","kind":"supports"},{"source_id":"S007","target_id":"S072","kind":"supports"},{"source_id":"S017","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S071","target_id":"S074","kind":"supports"},{"source_id":"S072","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S075","target_id":"S074","kind":"supports"},{"source_id":"S080","target_id":"S074","kind":"supports"},{"source_id":"S083","target_id":"S074","kind":"supports"},{"source_id":"S076","target_id":"S075","kind":"supports"},{"source_id":"S077","target_id":"S076","kind":"supports"},{"source_id":"S080","target_id":"S077","kind":"supports"}]}}
{"record_id":"2101.06504v2","paper":{"title":"Convex Generalized Nash Equilibrium Problems and Polynomial Optimization","authors":["Jiawang Nie","Xindong Tang"],"arxiv_id":"2101.06504","version":"v2","journal":"Mathematical Programming","doi":"10.1007/s10107-021-01739-7","abstract_url":"https://arxiv.org/abs/2101.06504v2","pdf_url":"https://arxiv.org/pdf/2101.06504v2"},"context":{"raw_tex":"The Generalized Nash Equilibrium Problem (GNEP) is a kind of game to find strategies\r\nfor a group of players such that each player's objective function is optimized,\r\nfor given other players' strategies.\r\nSuppose there are  $N$  players and the $i$th player's strategy is a vector\r\n$x_{i}\\in\\mathbb{R}^{n_{i}}$ (the $n_i$-dimensional real Euclidean space).\r\nWe write that\r\n\\[x_i:=(x_{i,1},\\ldots,x_{i,n_i}),\\quad x:=(x_1,\\ldots,x_N).\\]\r\nThe total dimension of all strategies is $n := n_1+ \\ldots + n_N.$\r\nThe main task of the GNEP is to find a tuple\r\n$u = (u_1, \\ldots, u_N)$ of strategies\r\nsuch that each $u_i$ is a minimizer of the $i$th player's optimization\r\n\\be\r\n\\label{eq:GNEP}\r\n%\\tag{$\\mathcal{SP}$-i}\r\n\\mbox{F}_i(u_{-i}): \\,\r\n\\left\\{ \\begin{array}{cl}\r\n\\min\\limits_{\\xpi\\in \\Rni}  &  \\fpi(u_1, \\ldots, u_{i-1}, x_i, u_{i+1},\\ldots, u_N) \\\\\r\n\\st & g_{i,j}(u_1, \\ldots, u_{i-1}, x_i, u_{i+1},\\ldots, u_N)  = 0 \\,\r\n        (j\\in \\mathcal{E}_i),\\\\\r\n\t& g_{i,j}(u_1, \\ldots, u_{i-1}, x_i, u_{i+1},\\ldots, u_N)  \\geq 0 \\,\r\n         (j\\in \\mathcal{I}_i),\r\n\\end{array} \\right.\r\n\\ee\n\nwhere $u_{-i} := (u_1, \\ldots, u_{i-1}, u_{i+1},\\ldots, u_N)$,\r\nthe $f_i$ and $g_{i,j}$ are continuously differentiable functions in $x_i$,\r\nand the $\\mc{E}_i$, $\\mc{I}_i$ are disjoint finite (possibly empty) labeling sets.\r\nThe point $u$ satisfying the above is called a Generalized Nash Equilibrium (GNE).\r\nFor notational convenience, when the $i$th player's strategy is considered,\r\nwe use $x_{-i}$ to denote the subvector of all players' strategies\r\nexcept the $i$th one, i.e.,\r\n\\[\r\nx_{-i} \\, := \\, (x_1, \\ldots, x_{i-1}, x_{i+1}, \\ldots, x_N),\r\n\\]\r\nand write $x=(x_{i},x_{-i})$ accordingly.\r\n\r\nThis paper focuses on the Generalized Nash Equilibrium Problem of Polynomials (GNEPP),\r\ni.e., all the functions $f_i$ and $g_{i,j}$ are polynomials in $x$.\r\nFor each $i=1,\\ldots,N$, let $X_i$ be the point-to-set map such that\r\n\\be \\label{eq:feaset}\r\nX_i(\\xmi) \\, := \\,\r\n\\left\\{x_i \\in \\Rni \\left|\\begin{array}{l}\r\ng_{i,j}(\\xpi,\\xmi) = 0 ,\\, j\\in \\mathcal{E}_i,\\\\\r\ng_{i,j}(\\xpi,\\xmi) \\geq 0 ,\\, j\\in \\mathcal{I}_i\r\n\\end{array}\\right.\\right\\}.\r\n\\ee\n\nThe $X_i(\\xmi)$ is the feasible strategy set of $\\mbox{F}_i(\\xmi)$.\r\nThe domain of $X_i$ is\r\n\\[\\dom(X_i):=\\{x_{-i}\\in\\re^{n-n_i}:X_i(\\xmi)\\ne\\emptyset\\}.\\]\r\nThe tuple $x$ is said to be a feasible point of the GNEP if\r\n$\\xpi\\in X_i(\\xmi)$ for all $i$. Denote the set\r\n\\be\r\nX :=\r\n\\left\\{  x\\in\\re^n\r\n\\left| \\begin{array}{l}\r\n g_{i,j}(\\xpi,\\xmi) = 0, \\, j\\in \\mathcal{E}_i,\\, i=1,\\ldots, N,\\\\\r\n g_{i,j}(\\xpi,\\xmi) \\geq 0, \\, j\\in \\mathcal{I}_i,\\, i=1,\\ldots, N\r\n\\end{array} \\right.\r\n\\right \\} .\r\n\\ee\n\nThen $x$ is a feasible point for the GNEP if and only if $x\\in X.$\r\n\n\n\\begin{defi} \\label{df:cgnep} \\rm\r\nThe GNEP given by (\\ref{eq:GNEP}) is called {\\it convex}\r\n\\footnote{\r\nIn some literature, this is also called player-convex,\r\nto distinguish from jointly-convex GNEPs; see \\cite{dreves2012nonsmooth}.}\r\nif for all\r\n$i=1,\\ldots, N$ and for all given $\\xmi\\in\\dom(X_i)$,\r\nthe objective $f_i(\\xpi,\\xmi)$ is convex in $x_i$ on $X_i(\\xmi)$,\r\nall $g_{i,j}(\\xpi,\\xmi) \\, (j\\in\\mc{E}_i)$\r\nare affine linear in $\\xpi$,\r\nand all $g_{i,j}(\\xpi,\\xmi) \\, (j\\in\\mc{I}_i)$ are concave in $\\xpi$.\r\n\\end{defi}\n\nThe symbol $\\mathbb N$ (resp., $\\mathbb R$, $\\mathbb C$) stands for the set of\r\nnonnegative integers (resp., real numbers, complex numbers).\r\nFor a positive integer $k$, denote the set $[k] := \\{1, \\ldots, k\\}$.\r\nFor a real number $t$, $\\lceil t \\rceil$ (resp., $\\lfloor t \\rfloor$)\r\ndenotes the smallest integer not smaller than $t$\r\n(resp., the biggest integer not bigger than $t$).\r\nWe use $e_i$ to denote the vector such that the $i$th entry is\r\n$1$ and all others are zeros.\r\nBy writing $A\\succeq0$ (resp., $A\\succ0$), we mean that the matrix $A$\r\nis symmetric positive semidefinite (resp., positive definite).\r\nFor the $i$th player's strategy vector $x_i \\in \\re^{n_i}$,\r\nthe $x_{i,j}$ denotes the $j$th entry of $x_i$, for $j = 1, \\ldots, n_i$.\r\nWhen we write $(y,x_{-i})$, it means that the $i$th player's strategy is $y\\in\\Rni$,\r\nwhile the vector of all other players' strategy is fixed to be $\\xmi$.\r\nLet $\\nfR[x]$ denote the ring of polynomials\r\nwith real coefficients in $x$,\r\nand $\\nfR[x]_d$ denote its subset of polynomials whose degrees are not greater than $d$.\r\nFor the $i$th player's strategy vector $x_i$,\r\nthe notation $\\re[x_i]$ and $\\re[x_i]_d$\r\nare defined in the same way.\r\nFor $i$th player's objective $f_i(x)$,\r\nthe notation $\\nabla_{x_i}f_i$, $\\nabla^2_{x_i}f_i$\r\nrespectively denote its gradient and Hessian with respect to $x_i$.\r\n\n\nIn the following, we use the letter $z$ to represent either\r\n$x$, $x_i$ or $(x,\\omega)$ for some new variables $\\omega$,\r\nfor convenience of discussion.\r\nSuppose $z := (z_1,\\ldots,z_l)$.\r\nFor a polynomial $p(z)\\in\\re[z]$,\r\nthe $p=0$ means $p(z)$ is identically zero on $\\re^l$.\r\nWe say the polynomial $p$ is nonzero if $p\\ne0$.\r\nLet $\\af := (\\af_1, \\ldots, \\af_l) \\in \\N^{l}$, and we denote\r\n\\[\r\nz^\\alpha := z_1^{\\alpha_1} \\cdots z_l^{\\alpha_l}, \\quad\r\n|\\alpha|:=\\alpha_1+\\ldots+\\alpha_l.\r\n\\]\r\nFor an integer $d >0$, denote the monomial power set\r\n\\[\r\n{\\mathbb{N}}_d^l \\, := \\,\r\n\\{\\alpha\\in {\\mathbb{N}}^l: \\, \\ |\\alpha| \\le d \\}.\r\n\\]\r\nWe use $[z]_d$ to denote the vector of all monomials in $z$\r\nwhose degree is at most $d$, ordered in the graded alphabetical ordering.\r\n\n\nThroughout the paper, a property is said to hold {\\it generically}\r\nif it holds for all points in the space of input data\r\nexcept a set of Lebesgue measure zero.\r\n\n\nWe study optimality conditions for Generalized Nash Equilibrium Problems.\r\nConsider the $i$th player's optimization.\r\nFor convenience, suppose $\\mc{E}_i\\cup\\mc{I}_i = [m_i]$\r\nand $g_i:=(g_{i,1},\\ldots,g_{i,m_i})$. For a given $\\xmi$, \r\nunder some suitable constraint qualifications\r\n(e.g., the linear independence constraint qualification (LICQ),\r\nMangasarian-Fromovite constraint qualification (MFCQ),\r\nor the Slater's Condition; see \\cite{Brks} for them),\r\nif $x_i$ is a minimizer of $\\mbox{F}_i(\\xmi)$,\r\nthen there exists a Lagrange multiplier vector\r\n$\\lambda_i:=(\\lambda_{i,1},\\ldots,\\lambda_{i,m_i})$ such that\r\n\\be\r\n\\label{eq:KKTwithLM}\r\n\\left\\{\r\n\\begin{array}{l}\r\n  \\nabla_{x_i} f_i(x)-\\sum_{j=1}^{m_i}\\lambda_{i,j}\\nabla_{x_i} g_{i,j}(x)=0,\\\\\r\n  \\lambda_i\\perp g_i(x),\\,g_{i,j}(x)=0\\,(j\\in\\mathcal{E}_i),\\\\\r\n  \\lambda_{i,j}\\ge0\\,(j\\in\\mathcal{I}_i),\\,g_{i,j}(x)\\ge0\\,(j\\in\\mathcal{I}_i).\r\n\\end{array}\r\n\\right.\r\n\\ee\n\nThis is called the first order Karush-Kuhn-Tucker system for $\\mbox{F}_i(\\xmi)$.\r\nSuch $(x_i,\\lambda_i)$ is called a critical pair of $\\mbox{F}_i(\\xmi)$.\r\n\n\n\\be\r\n\\label{eq:KKTwithLMalp}\r\n\\left\\{\r\n\\begin{array}{l}\r\n\\nabla_{x_i} f_i(x)-\\sum_{j=1}^{m_i}\\lambda_{i,j}\\nabla_{x_i} g_{i,j}(x)=0\\,(i\\in[N]),\\\\\r\n\\lambda_i\\perp g_i(x)\\,(i\\in[N]),\\,g_{i,j}(x)=0\\,(i\\in[N],j\\in\\mathcal{E}_i),\\\\\r\n\\lambda_{i,j}\\ge0\\,(i\\in[N],j\\in\\mathcal{I}_i),\\,g_{i,j}(x)\\ge0\\,(i\\in[N],j\\in\\mathcal{I}_i).\\\\\r\n\\end{array}\r\n\\right.\r\n\\ee\n\nA point $x$ satisfying (\\ref{eq:KKTwithLMalp}) is called a KKT point for the GNEP.\r\nFor convex GNEPs, each KKT point is a GNE \\cite[Theorem~4.6]{Facchinei2010}.\r\n\n\n\\begin{defi} \\label{LME:rat} \\rm\r\nFor the $i$th player's optimization $\\mbox{F}_i(\\xmi)$,\r\nif there exist polynomials $\\hat{\\lambda}_{i,1},\\ldots,\\hat{\\lambda}_{i,m_i}$ \r\nand a nonzero polynomial $q_i$ such that\r\n$q_i(x)\\ge 0$ for all $x\\in X$, and $\\hat{\\lambda}_{i,j}(x)=q_i(x)\\lambda_{i,j}$ \r\nholds for all critical pairs $(x_i,\\lambda_i)$,\r\nthen we call the tuple\r\n\\[\r\n\\hat{\\lambda}_i/q_i:=(\\hat{\\lambda}_{i,1}(x)/q_i(x),\r\n\\ldots,\\hat{\\lambda}_{i,m_i}(x)/q_i(x))\r\n\\]\r\na rational expression for Lagrange multipliers.\r\n\\end{defi}\n\n\\be\r\n\\label{eq:KKTfeasoptrtn}\r\n\\left\\{\r\n\\begin{array}{rll}\r\n\\min\\limits_{x\\in X} & [x]_1^T\\Theta[x]_1 \\\\\r\n\\st &q_i(x)\\nabla_{x_i} f_i(x)-\\sum\\nolimits_{j=1}^{m_i}\\hat{\\lambda}_{i,j}(x)\\nabla_{x_i} g_{i,j}(x)=0\\,(i\\in[N]),\\\\\r\n\t&\\hat{\\lambda}_{i,j}(x)\\perp g_{i,j}(x)\\,(j\\in\\mathcal{E}_i\\cup\\mc{I}_i,i\\in[N]),\\\\\r\n\t% &g_{i,j}(x)=0\\,(j\\in\\mathcal{E}_i,i\\in[N]),\\\\\r\n\t% &g_{i,j}(x)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]),\\\\\r\n\t&\\hat{\\lambda}_{i,j}(x)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]).\r\n\\end{array}\r\n\\right.\r\n\\ee\n\nIn the above,\r\n$\\Theta$ is a generically chosen positive definite matrix.\r\n\n\n\\begin{defi} \\label{def:para:LME} \\rm\r\nFor the $i$th player's optimization $\\mbox{F}_i(\\xmi)$,\r\na parametric expression for the Lagrange multipliers is a tuple of polynomials\r\n\\[ \\hat{\\lambda}_{i}(x, \\omega_i) := (\\hat{\\lambda}_{i,1}(x, \\omega_i),\\ldots,\\hat{\\lambda}_{i,m_i}(x, \\omega_i) ), \\]\r\nin $x$ and in a parameter $\\omega_i:=(\\omega_{i,1},\\ldots,\\omega_{i,s_i})$ with $s_i\\le m_i$,\r\nsuch that $(x_i,\\lambda_i)$ is a critical pair if and only if\r\nthere is a value of $\\omega_i$ such that (\\ref{eq:KKTwithLM}) is satisfied for\r\n$\\lambda_{i,j}=\\hat{\\lambda}_{i,j}(x,\\omega_i)$ with $j\\in[m_i]$.\r\n\\end{defi}\n\nSuppose all players have parametric expressions for their Lagrange multipliers\r\nas in Definition~\\ref{def:para:LME}. \r\nLet $s := s_1+\\ldots+s_N$, and denote\r\n\\[\r\n\\allx \\,:=\\, (x,\\omega_1,\\ldots,\\omega_N). \r\n\\]\r\n\n\n\\be\r\n\\label{eq:KKTfeasoptprt}\r\n\\left\\{\r\n\\begin{array}{rll}\r\n\\min\\limits_{\\allx\\in X\\times\\re^s} & \\lvt\\allx\\rvt_1^T\\Theta\\lvt\\allx\\rvt_1 \\\\\r\n\\st &\\nabla_{x_i} f_i(x)-\\sum\\nolimits_{j=1}^{m_i}\\hat{\\lambda}_{i,j}(\\allx)\\nabla_{x_i} g_{i,j}(x)=0 \\, (i\\in[N]),\\\\\r\n\t&\\hat{\\lambda}_{i,j}(\\allx)\\perp g_{i,j}(x)\\,(j\\in\\mathcal{E}_i\\cup\\mc{I}_i,i\\in[N]),\\\\\r\n\t&\\hat{\\lambda}_{i,j}(\\allx)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]).\r\n\\end{array}\r\n\\right.\r\n\\ee\n\nIn the above, the $\\Theta$ is a generically chosen positive definite matrix.\r\n\n\nIn this section, we give an algorithm for solving convex GNEPs.\r\nWe assume each $\\lambda_i$ has either a rational or parametric expression,\r\nas in Definition~\\ref{LME:rat} or \\ref{def:para:LME}.\r\nIf $\\lambda_i$ has a polynomial or parametric expression,\r\nwe let $q_i(x) :=1$.\r\nIf $\\lambda_i$ has a polynomial or rational expression,\r\nthen we let $s_i=0$. Recall the notation\r\n\\[\r\n\\allx \\,:= \\, (x,\\omega_1,\\ldots,\\omega_N).\r\n\\]\r\nChoose a generic positive definite matrix $\\Theta$.\r\nThen solve the following polynomial optimization\r\n\\be\r\n\\label{eq:KKTfeasopt}\r\n\\left\\{\r\n\\begin{array}{rll}\r\n\\min\\limits_{\\allx} & \\lvt\\allx\\rvt_1^T\\Theta\\lvt\\allx\\rvt_1 \\\\\r\n\\st &q_i(x)\\nabla_{x_i} f_i(x)-\\sum\\nolimits_{j=1}^{m_i}\\hat{\\lambda}_{i,j}(\\allx)\\nabla_{x_i} g_{i,j}(x)=0 \\, (i\\in[N]),\\\\\r\n\t&\\hat{\\lambda}_{i,j}(\\allx)\\perp g_{i,j}(x)\\,(j\\in\\mathcal{E}_i\\cup\\mc{I}_i,i\\in[N]),\\\\\r\n\t&g_{i,j}(x)=0\\,(j\\in\\mathcal{E}_i,i\\in[N]),\\\\\r\n\t&g_{i,j}(x)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]),\\\\\r\n\t&\\hat{\\lambda}_{i,j}(\\allx)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]).\r\n\\end{array}\r\n\\right.\r\n\\ee\n\n\\begin{equation}\r\n\\left\\{\r\n\\begin{array}{llll}\r\n\\delta_i := & \\min\\limits_{\\xpi}& f_i(x_i,u_{-i})-f_i(u_{i},u_{-i})&\\\\\r\n &\\st  &  g_{i,j}(x_{i},u_{-i})=0 \\, (j\\in\\mc{E}_i), \\\r\n g_{i,j}(x_{i},u_{-i})\\ge0 \\, (j\\in\\mc{I}_i).\r\n\\end{array}\r\n\\right.\r\n\\label{eq:checkopt}\r\n\\end{equation}\n\n\\begin{alg}\r\n\\label{ag:KKTSDP} \\rm\r\nFor the convex GNEP given by (\\ref{eq:GNEP}),\r\ndo the following:\r\n\r\n\\begin{description}\r\n\r\n\\item [Step~0]\r\nChoose a generic positive definite matrix $\\Theta$ of length $n+s+1$.\r\n\r\n\\item [Step~1]\r\nSolve the polynomial optimization \\reff{eq:KKTfeasopt}.\r\nIf it is infeasible, then there are no KKT points and stop;\r\notherwise, solve it for a minimizer $(u,w)$.\r\n\r\n\\item [Step~2]\r\nIf all $q_i(u) >0$, then $u$ is a GNE. Otherwise,\r\nfor those $i$ with $q_i(u) \\le 0$, solve the optimization \\reff{eq:checkopt}\r\nfor the minimum value $\\delta_i$.\r\nIf $\\delta_i \\ge 0$ for all such $i$, then $u$ is a GNE;\r\notherwise, it is not.\r\n\r\n\\end{description}\r\n\\end{alg}","units":[{"start":6665,"end":7745,"start_line":238,"end_line":260,"raw_tex":"The Generalized Nash Equilibrium Problem (GNEP) is a kind of game to find strategies\r\nfor a group of players such that each player's objective function is optimized,\r\nfor given other players' strategies.\r\nSuppose there are  $N$  players and the $i$th player's strategy is a vector\r\n$x_{i}\\in\\mathbb{R}^{n_{i}}$ (the $n_i$-dimensional real Euclidean space).\r\nWe write that\r\n\\[x_i:=(x_{i,1},\\ldots,x_{i,n_i}),\\quad x:=(x_1,\\ldots,x_N).\\]\r\nThe total dimension of all strategies is $n := n_1+ \\ldots + n_N.$\r\nThe main task of the GNEP is to find a tuple\r\n$u = (u_1, \\ldots, u_N)$ of strategies\r\nsuch that each $u_i$ is a minimizer of the $i$th player's optimization\r\n\\be\r\n\\label{eq:GNEP}\r\n%\\tag{$\\mathcal{SP}$-i}\r\n\\mbox{F}_i(u_{-i}): \\,\r\n\\left\\{ \\begin{array}{cl}\r\n\\min\\limits_{\\xpi\\in \\Rni}  &  \\fpi(u_1, \\ldots, u_{i-1}, x_i, u_{i+1},\\ldots, u_N) \\\\\r\n\\st & g_{i,j}(u_1, \\ldots, u_{i-1}, x_i, u_{i+1},\\ldots, u_N)  = 0 \\,\r\n        (j\\in \\mathcal{E}_i),\\\\\r\n\t& g_{i,j}(u_1, \\ldots, u_{i-1}, x_i, u_{i+1},\\ldots, u_N)  \\geq 0 \\,\r\n         (j\\in \\mathcal{I}_i),\r\n\\end{array} \\right.\r\n\\ee","sha256_utf8":"ca82c1000570c98196f24e9a534a6be0b63b04f0d8cd608fcfb43e4edb24b60d"},{"start":7747,"end":8792,"start_line":261,"end_line":282,"raw_tex":"where $u_{-i} := (u_1, \\ldots, u_{i-1}, u_{i+1},\\ldots, u_N)$,\r\nthe $f_i$ and $g_{i,j}$ are continuously differentiable functions in $x_i$,\r\nand the $\\mc{E}_i$, $\\mc{I}_i$ are disjoint finite (possibly empty) labeling sets.\r\nThe point $u$ satisfying the above is called a Generalized Nash Equilibrium (GNE).\r\nFor notational convenience, when the $i$th player's strategy is considered,\r\nwe use $x_{-i}$ to denote the subvector of all players' strategies\r\nexcept the $i$th one, i.e.,\r\n\\[\r\nx_{-i} \\, := \\, (x_1, \\ldots, x_{i-1}, x_{i+1}, \\ldots, x_N),\r\n\\]\r\nand write $x=(x_{i},x_{-i})$ accordingly.\r\n\r\nThis paper focuses on the Generalized Nash Equilibrium Problem of Polynomials (GNEPP),\r\ni.e., all the functions $f_i$ and $g_{i,j}$ are polynomials in $x$.\r\nFor each $i=1,\\ldots,N$, let $X_i$ be the point-to-set map such that\r\n\\be \\label{eq:feaset}\r\nX_i(\\xmi) \\, := \\,\r\n\\left\\{x_i \\in \\Rni \\left|\\begin{array}{l}\r\ng_{i,j}(\\xpi,\\xmi) = 0 ,\\, j\\in \\mathcal{E}_i,\\\\\r\ng_{i,j}(\\xpi,\\xmi) \\geq 0 ,\\, j\\in \\mathcal{I}_i\r\n\\end{array}\\right.\\right\\}.\r\n\\ee","sha256_utf8":"ff2a6caf8bcbfff8e26ea47409905e377ab3392d23ac470aa8a5f208bc2d15e9"},{"start":8794,"end":9290,"start_line":283,"end_line":296,"raw_tex":"The $X_i(\\xmi)$ is the feasible strategy set of $\\mbox{F}_i(\\xmi)$.\r\nThe domain of $X_i$ is\r\n\\[\\dom(X_i):=\\{x_{-i}\\in\\re^{n-n_i}:X_i(\\xmi)\\ne\\emptyset\\}.\\]\r\nThe tuple $x$ is said to be a feasible point of the GNEP if\r\n$\\xpi\\in X_i(\\xmi)$ for all $i$. Denote the set\r\n\\be\r\nX :=\r\n\\left\\{  x\\in\\re^n\r\n\\left| \\begin{array}{l}\r\n g_{i,j}(\\xpi,\\xmi) = 0, \\, j\\in \\mathcal{E}_i,\\, i=1,\\ldots, N,\\\\\r\n g_{i,j}(\\xpi,\\xmi) \\geq 0, \\, j\\in \\mathcal{I}_i,\\, i=1,\\ldots, N\r\n\\end{array} \\right.\r\n\\right \\} .\r\n\\ee","sha256_utf8":"a320df36c9c2366d51419902af8fd33b6a0c97959e596b1f677cb8d753b29e92"},{"start":9292,"end":9360,"start_line":297,"end_line":297,"raw_tex":"Then $x$ is a feasible point for the GNEP if and only if $x\\in X.$\r\n","sha256_utf8":"2602a6d0d7159e0b14de8e8c8701891d3205ca8e6a59f4bb7d636cf87335280d"},{"start":9364,"end":9890,"start_line":300,"end_line":311,"raw_tex":"\\begin{defi} \\label{df:cgnep} \\rm\r\nThe GNEP given by (\\ref{eq:GNEP}) is called {\\it convex}\r\n\\footnote{\r\nIn some literature, this is also called player-convex,\r\nto distinguish from jointly-convex GNEPs; see \\cite{dreves2012nonsmooth}.}\r\nif for all\r\n$i=1,\\ldots, N$ and for all given $\\xmi\\in\\dom(X_i)$,\r\nthe objective $f_i(\\xpi,\\xmi)$ is convex in $x_i$ on $X_i(\\xmi)$,\r\nall $g_{i,j}(\\xpi,\\xmi) \\, (j\\in\\mc{E}_i)$\r\nare affine linear in $\\xpi$,\r\nand all $g_{i,j}(\\xpi,\\xmi) \\, (j\\in\\mc{I}_i)$ are concave in $\\xpi$.\r\n\\end{defi}","sha256_utf8":"b6e9e7285b96728b2bc55460f8c948fb1c8e691ec1f4358c059ab38f715c613a"},{"start":14508,"end":15868,"start_line":412,"end_line":434,"raw_tex":"The symbol $\\mathbb N$ (resp., $\\mathbb R$, $\\mathbb C$) stands for the set of\r\nnonnegative integers (resp., real numbers, complex numbers).\r\nFor a positive integer $k$, denote the set $[k] := \\{1, \\ldots, k\\}$.\r\nFor a real number $t$, $\\lceil t \\rceil$ (resp., $\\lfloor t \\rfloor$)\r\ndenotes the smallest integer not smaller than $t$\r\n(resp., the biggest integer not bigger than $t$).\r\nWe use $e_i$ to denote the vector such that the $i$th entry is\r\n$1$ and all others are zeros.\r\nBy writing $A\\succeq0$ (resp., $A\\succ0$), we mean that the matrix $A$\r\nis symmetric positive semidefinite (resp., positive definite).\r\nFor the $i$th player's strategy vector $x_i \\in \\re^{n_i}$,\r\nthe $x_{i,j}$ denotes the $j$th entry of $x_i$, for $j = 1, \\ldots, n_i$.\r\nWhen we write $(y,x_{-i})$, it means that the $i$th player's strategy is $y\\in\\Rni$,\r\nwhile the vector of all other players' strategy is fixed to be $\\xmi$.\r\nLet $\\nfR[x]$ denote the ring of polynomials\r\nwith real coefficients in $x$,\r\nand $\\nfR[x]_d$ denote its subset of polynomials whose degrees are not greater than $d$.\r\nFor the $i$th player's strategy vector $x_i$,\r\nthe notation $\\re[x_i]$ and $\\re[x_i]_d$\r\nare defined in the same way.\r\nFor $i$th player's objective $f_i(x)$,\r\nthe notation $\\nabla_{x_i}f_i$, $\\nabla^2_{x_i}f_i$\r\nrespectively denote its gradient and Hessian with respect to $x_i$.\r\n","sha256_utf8":"12ce49683b8730d7fbfb0ed5ea9c5559c583e484a94ae2e9fe02b817309c7590"},{"start":15870,"end":16642,"start_line":436,"end_line":454,"raw_tex":"In the following, we use the letter $z$ to represent either\r\n$x$, $x_i$ or $(x,\\omega)$ for some new variables $\\omega$,\r\nfor convenience of discussion.\r\nSuppose $z := (z_1,\\ldots,z_l)$.\r\nFor a polynomial $p(z)\\in\\re[z]$,\r\nthe $p=0$ means $p(z)$ is identically zero on $\\re^l$.\r\nWe say the polynomial $p$ is nonzero if $p\\ne0$.\r\nLet $\\af := (\\af_1, \\ldots, \\af_l) \\in \\N^{l}$, and we denote\r\n\\[\r\nz^\\alpha := z_1^{\\alpha_1} \\cdots z_l^{\\alpha_l}, \\quad\r\n|\\alpha|:=\\alpha_1+\\ldots+\\alpha_l.\r\n\\]\r\nFor an integer $d >0$, denote the monomial power set\r\n\\[\r\n{\\mathbb{N}}_d^l \\, := \\,\r\n\\{\\alpha\\in {\\mathbb{N}}^l: \\, \\ |\\alpha| \\le d \\}.\r\n\\]\r\nWe use $[z]_d$ to denote the vector of all monomials in $z$\r\nwhose degree is at most $d$, ordered in the graded alphabetical ordering.\r\n","sha256_utf8":"5e2feb69d200e66f24fc56328e7ec4e4b9f42f83309aec258665589643205f7d"},{"start":16772,"end":16935,"start_line":460,"end_line":462,"raw_tex":"Throughout the paper, a property is said to hold {\\it generically}\r\nif it holds for all points in the space of input data\r\nexcept a set of Lebesgue measure zero.\r\n","sha256_utf8":"f89456d59f24cd7681bea4953a38fe9074348dcd2dd4e5461684c3c13c32034d"},{"start":21555,"end":22478,"start_line":583,"end_line":603,"raw_tex":"We study optimality conditions for Generalized Nash Equilibrium Problems.\r\nConsider the $i$th player's optimization.\r\nFor convenience, suppose $\\mc{E}_i\\cup\\mc{I}_i = [m_i]$\r\nand $g_i:=(g_{i,1},\\ldots,g_{i,m_i})$. For a given $\\xmi$, \r\nunder some suitable constraint qualifications\r\n(e.g., the linear independence constraint qualification (LICQ),\r\nMangasarian-Fromovite constraint qualification (MFCQ),\r\nor the Slater's Condition; see \\cite{Brks} for them),\r\nif $x_i$ is a minimizer of $\\mbox{F}_i(\\xmi)$,\r\nthen there exists a Lagrange multiplier vector\r\n$\\lambda_i:=(\\lambda_{i,1},\\ldots,\\lambda_{i,m_i})$ such that\r\n\\be\r\n\\label{eq:KKTwithLM}\r\n\\left\\{\r\n\\begin{array}{l}\r\n  \\nabla_{x_i} f_i(x)-\\sum_{j=1}^{m_i}\\lambda_{i,j}\\nabla_{x_i} g_{i,j}(x)=0,\\\\\r\n  \\lambda_i\\perp g_i(x),\\,g_{i,j}(x)=0\\,(j\\in\\mathcal{E}_i),\\\\\r\n  \\lambda_{i,j}\\ge0\\,(j\\in\\mathcal{I}_i),\\,g_{i,j}(x)\\ge0\\,(j\\in\\mathcal{I}_i).\r\n\\end{array}\r\n\\right.\r\n\\ee","sha256_utf8":"04e8ba95d060b662a5132a1f0fce335889dcc14cbc9bc1af2c31814540e74b2e"},{"start":22480,"end":22635,"start_line":604,"end_line":605,"raw_tex":"This is called the first order Karush-Kuhn-Tucker system for $\\mbox{F}_i(\\xmi)$.\r\nSuch $(x_i,\\lambda_i)$ is called a critical pair of $\\mbox{F}_i(\\xmi)$.\r\n","sha256_utf8":"8ceb8a383beee247a90aec2fc62036cebe9029134560f3e7a6cd9a1eb8c2b2b0"},{"start":22835,"end":23185,"start_line":609,"end_line":618,"raw_tex":"\\be\r\n\\label{eq:KKTwithLMalp}\r\n\\left\\{\r\n\\begin{array}{l}\r\n\\nabla_{x_i} f_i(x)-\\sum_{j=1}^{m_i}\\lambda_{i,j}\\nabla_{x_i} g_{i,j}(x)=0\\,(i\\in[N]),\\\\\r\n\\lambda_i\\perp g_i(x)\\,(i\\in[N]),\\,g_{i,j}(x)=0\\,(i\\in[N],j\\in\\mathcal{E}_i),\\\\\r\n\\lambda_{i,j}\\ge0\\,(i\\in[N],j\\in\\mathcal{I}_i),\\,g_{i,j}(x)\\ge0\\,(i\\in[N],j\\in\\mathcal{I}_i).\\\\\r\n\\end{array}\r\n\\right.\r\n\\ee","sha256_utf8":"26d824d1e03e9efcbcc863427d8fb59cfd313881588a939df7b59dfe1512a91c"},{"start":23187,"end":23349,"start_line":619,"end_line":620,"raw_tex":"A point $x$ satisfying (\\ref{eq:KKTwithLMalp}) is called a KKT point for the GNEP.\r\nFor convex GNEPs, each KKT point is a GNE \\cite[Theorem~4.6]{Facchinei2010}.\r\n","sha256_utf8":"b3fadb18fac39dedeefa14ff3ee46a0ff15b30fd0b862955941b7b5234d913fc"},{"start":26364,"end":26895,"start_line":700,"end_line":712,"raw_tex":"\\begin{defi} \\label{LME:rat} \\rm\r\nFor the $i$th player's optimization $\\mbox{F}_i(\\xmi)$,\r\nif there exist polynomials $\\hat{\\lambda}_{i,1},\\ldots,\\hat{\\lambda}_{i,m_i}$ \r\nand a nonzero polynomial $q_i$ such that\r\n$q_i(x)\\ge 0$ for all $x\\in X$, and $\\hat{\\lambda}_{i,j}(x)=q_i(x)\\lambda_{i,j}$ \r\nholds for all critical pairs $(x_i,\\lambda_i)$,\r\nthen we call the tuple\r\n\\[\r\n\\hat{\\lambda}_i/q_i:=(\\hat{\\lambda}_{i,1}(x)/q_i(x),\r\n\\ldots,\\hat{\\lambda}_{i,m_i}(x)/q_i(x))\r\n\\]\r\na rational expression for Lagrange multipliers.\r\n\\end{defi}","sha256_utf8":"0cb4fbc76548a756bf059e036f6a3cd687add9d2fb9530d9b6a74a5f38a1e985"},{"start":29672,"end":30170,"start_line":797,"end_line":809,"raw_tex":"\\be\r\n\\label{eq:KKTfeasoptrtn}\r\n\\left\\{\r\n\\begin{array}{rll}\r\n\\min\\limits_{x\\in X} & [x]_1^T\\Theta[x]_1 \\\\\r\n\\st &q_i(x)\\nabla_{x_i} f_i(x)-\\sum\\nolimits_{j=1}^{m_i}\\hat{\\lambda}_{i,j}(x)\\nabla_{x_i} g_{i,j}(x)=0\\,(i\\in[N]),\\\\\r\n\t&\\hat{\\lambda}_{i,j}(x)\\perp g_{i,j}(x)\\,(j\\in\\mathcal{E}_i\\cup\\mc{I}_i,i\\in[N]),\\\\\r\n\t% &g_{i,j}(x)=0\\,(j\\in\\mathcal{E}_i,i\\in[N]),\\\\\r\n\t% &g_{i,j}(x)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]),\\\\\r\n\t&\\hat{\\lambda}_{i,j}(x)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]).\r\n\\end{array}\r\n\\right.\r\n\\ee","sha256_utf8":"c432e46a6ce67ecea46394e82b27b5192f1bda950a50c01e7fbb23c4b89f7870"},{"start":30172,"end":30247,"start_line":810,"end_line":811,"raw_tex":"In the above,\r\n$\\Theta$ is a generically chosen positive definite matrix.\r\n","sha256_utf8":"2d790c9fe152cea026d4875649df2ce0548df829347389c7816982ffb88a547f"},{"start":40221,"end":40832,"start_line":1061,"end_line":1069,"raw_tex":"\\begin{defi} \\label{def:para:LME} \\rm\r\nFor the $i$th player's optimization $\\mbox{F}_i(\\xmi)$,\r\na parametric expression for the Lagrange multipliers is a tuple of polynomials\r\n\\[ \\hat{\\lambda}_{i}(x, \\omega_i) := (\\hat{\\lambda}_{i,1}(x, \\omega_i),\\ldots,\\hat{\\lambda}_{i,m_i}(x, \\omega_i) ), \\]\r\nin $x$ and in a parameter $\\omega_i:=(\\omega_{i,1},\\ldots,\\omega_{i,s_i})$ with $s_i\\le m_i$,\r\nsuch that $(x_i,\\lambda_i)$ is a critical pair if and only if\r\nthere is a value of $\\omega_i$ such that (\\ref{eq:KKTwithLM}) is satisfied for\r\n$\\lambda_{i,j}=\\hat{\\lambda}_{i,j}(x,\\omega_i)$ with $j\\in[m_i]$.\r\n\\end{defi}","sha256_utf8":"bf7edd2ef661d5c37f3ac8a4c919f4d85985c8748d14b8f5450eef9a6e7ef6c8"},{"start":46557,"end":46768,"start_line":1238,"end_line":1243,"raw_tex":"Suppose all players have parametric expressions for their Lagrange multipliers\r\nas in Definition~\\ref{def:para:LME}. \r\nLet $s := s_1+\\ldots+s_N$, and denote\r\n\\[\r\n\\allx \\,:=\\, (x,\\omega_1,\\ldots,\\omega_N). \r\n\\]\r\n","sha256_utf8":"2987f401beb304a312ea56c3acb04fab938760683d68ec28bd21972cead47908"},{"start":47400,"end":47839,"start_line":1259,"end_line":1269,"raw_tex":"\\be\r\n\\label{eq:KKTfeasoptprt}\r\n\\left\\{\r\n\\begin{array}{rll}\r\n\\min\\limits_{\\allx\\in X\\times\\re^s} & \\lvt\\allx\\rvt_1^T\\Theta\\lvt\\allx\\rvt_1 \\\\\r\n\\st &\\nabla_{x_i} f_i(x)-\\sum\\nolimits_{j=1}^{m_i}\\hat{\\lambda}_{i,j}(\\allx)\\nabla_{x_i} g_{i,j}(x)=0 \\, (i\\in[N]),\\\\\r\n\t&\\hat{\\lambda}_{i,j}(\\allx)\\perp g_{i,j}(x)\\,(j\\in\\mathcal{E}_i\\cup\\mc{I}_i,i\\in[N]),\\\\\r\n\t&\\hat{\\lambda}_{i,j}(\\allx)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]).\r\n\\end{array}\r\n\\right.\r\n\\ee","sha256_utf8":"bbfe3ee5d8b30ec19efa1b499777d0008198206364150926eb8b2af7615b422f"},{"start":47841,"end":47919,"start_line":1270,"end_line":1270,"raw_tex":"In the above, the $\\Theta$ is a generically chosen positive definite matrix.\r\n","sha256_utf8":"4d782b2985f1099db1cb61da0e55ae9c7abbf4dde7f2f47e5c3f64882efa8ec0"},{"start":48622,"end":49678,"start_line":1299,"end_line":1323,"raw_tex":"In this section, we give an algorithm for solving convex GNEPs.\r\nWe assume each $\\lambda_i$ has either a rational or parametric expression,\r\nas in Definition~\\ref{LME:rat} or \\ref{def:para:LME}.\r\nIf $\\lambda_i$ has a polynomial or parametric expression,\r\nwe let $q_i(x) :=1$.\r\nIf $\\lambda_i$ has a polynomial or rational expression,\r\nthen we let $s_i=0$. Recall the notation\r\n\\[\r\n\\allx \\,:= \\, (x,\\omega_1,\\ldots,\\omega_N).\r\n\\]\r\nChoose a generic positive definite matrix $\\Theta$.\r\nThen solve the following polynomial optimization\r\n\\be\r\n\\label{eq:KKTfeasopt}\r\n\\left\\{\r\n\\begin{array}{rll}\r\n\\min\\limits_{\\allx} & \\lvt\\allx\\rvt_1^T\\Theta\\lvt\\allx\\rvt_1 \\\\\r\n\\st &q_i(x)\\nabla_{x_i} f_i(x)-\\sum\\nolimits_{j=1}^{m_i}\\hat{\\lambda}_{i,j}(\\allx)\\nabla_{x_i} g_{i,j}(x)=0 \\, (i\\in[N]),\\\\\r\n\t&\\hat{\\lambda}_{i,j}(\\allx)\\perp g_{i,j}(x)\\,(j\\in\\mathcal{E}_i\\cup\\mc{I}_i,i\\in[N]),\\\\\r\n\t&g_{i,j}(x)=0\\,(j\\in\\mathcal{E}_i,i\\in[N]),\\\\\r\n\t&g_{i,j}(x)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]),\\\\\r\n\t&\\hat{\\lambda}_{i,j}(\\allx)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]).\r\n\\end{array}\r\n\\right.\r\n\\ee","sha256_utf8":"8f64e081bf2fe98afa33e8076f862cd3139b986c3109e48e594fb6ba44938e0d"},{"start":50124,"end":50404,"start_line":1333,"end_line":1342,"raw_tex":"\\begin{equation}\r\n\\left\\{\r\n\\begin{array}{llll}\r\n\\delta_i := & \\min\\limits_{\\xpi}& f_i(x_i,u_{-i})-f_i(u_{i},u_{-i})&\\\\\r\n &\\st  &  g_{i,j}(x_{i},u_{-i})=0 \\, (j\\in\\mc{E}_i), \\\r\n g_{i,j}(x_{i},u_{-i})\\ge0 \\, (j\\in\\mc{I}_i).\r\n\\end{array}\r\n\\right.\r\n\\label{eq:checkopt}\r\n\\end{equation}","sha256_utf8":"dabada5f689b17922ec50edd55c9a2dd46efd7e83423fd6ddf446b148841aa5f"},{"start":50655,"end":51344,"start_line":1350,"end_line":1373,"raw_tex":"\\begin{alg}\r\n\\label{ag:KKTSDP} \\rm\r\nFor the convex GNEP given by (\\ref{eq:GNEP}),\r\ndo the following:\r\n\r\n\\begin{description}\r\n\r\n\\item [Step~0]\r\nChoose a generic positive definite matrix $\\Theta$ of length $n+s+1$.\r\n\r\n\\item [Step~1]\r\nSolve the polynomial optimization \\reff{eq:KKTfeasopt}.\r\nIf it is infeasible, then there are no KKT points and stop;\r\notherwise, solve it for a minimizer $(u,w)$.\r\n\r\n\\item [Step~2]\r\nIf all $q_i(u) >0$, then $u$ is a GNE. Otherwise,\r\nfor those $i$ with $q_i(u) \\le 0$, solve the optimization \\reff{eq:checkopt}\r\nfor the minimum value $\\delta_i$.\r\nIf $\\delta_i \\ge 0$ for all such $i$, then $u$ is a GNE;\r\notherwise, it is not.\r\n\r\n\\end{description}\r\n\\end{alg}","sha256_utf8":"87d436b302f9cd3e486a3090cae081d83aeb6d31fe74e557db17b6f996eedbc2"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{thm}\r\n\\label{tm:KKTSDPconv}\r\nFor the convex GNEPP given by (\\ref{eq:GNEP}),\r\nsuppose each Lagrange multiplier vector $\\lambda_i$ has a rational expression\r\nas in Definition~\\ref{LME:rat}\r\nor a parametric expression as in Definition~\\ref{def:para:LME}.\r\n\r\n\\begin{enumerate}[(i)]\r\n\r\n\r\n\\item If $(u,w)$ is a feasible point of (\\ref{eq:KKTfeasopt})\r\nsuch that $q_i(u)>0$ for all $i$, then $u$ is a GNE.\r\n\r\n\r\n\\item Assume every GNE is a KKT point.\r\nIf (\\ref{eq:KKTfeasopt}) is infeasible, then the GNEP has no GNEs.\r\nIf $\\Theta$ is positive definite and every\r\n$q_i(x)>0$ for all feasible $x$ of (\\ref{eq:KKTfeasopt}),\r\nthen Algorithm~\\ref{ag:KKTSDP} will find a GNE if it exists.\r\n\\end{enumerate}\r\n\\end{thm}","units":[{"start":52274,"end":52984,"start_line":1394,"end_line":1414,"raw_tex":"\\begin{thm}\r\n\\label{tm:KKTSDPconv}\r\nFor the convex GNEPP given by (\\ref{eq:GNEP}),\r\nsuppose each Lagrange multiplier vector $\\lambda_i$ has a rational expression\r\nas in Definition~\\ref{LME:rat}\r\nor a parametric expression as in Definition~\\ref{def:para:LME}.\r\n\r\n\\begin{enumerate}[(i)]\r\n\r\n\r\n\\item If $(u,w)$ is a feasible point of (\\ref{eq:KKTfeasopt})\r\nsuch that $q_i(u)>0$ for all $i$, then $u$ is a GNE.\r\n\r\n\r\n\\item Assume every GNE is a KKT point.\r\nIf (\\ref{eq:KKTfeasopt}) is infeasible, then the GNEP has no GNEs.\r\nIf $\\Theta$ is positive definite and every\r\n$q_i(x)>0$ for all feasible $x$ of (\\ref{eq:KKTfeasopt}),\r\nthen Algorithm~\\ref{ag:KKTSDP} will find a GNE if it exists.\r\n\\end{enumerate}\r\n\\end{thm}","sha256_utf8":"75c4c80b82db94282c7462e43824b8fa3f33cbced350dbcba7eee81699b34ba6"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"Suppose for each $i$, there exists a rational expression\r\n$\\hat{\\lambda}_i/q_i$ for the $i$th player's Lagrange multiplier vector.\r\nSince $q_{i}(x)\\lambda_{i,j}=\\hat{\\lambda}_i(x)$\r\nand $q_i(x)\\ge0$ for all $x\\in X$,\r\nthe following holds for all KKT points\r\n\\be\r\n\\label{eq:KKTrational}\r\n\\left\\{\r\n\\begin{array}{l}\r\nq_i(x)\\nabla_{x_i} f_i(x)-\\sum\\nolimits_{j=1}^{m_i}\r\n \\hat{\\lambda}_{i,j}(x)\\nabla_{x_i} g_{i,j}(x)=0 \\, (i\\in[N]),\\\\\r\n\\hat{\\lambda}_i(x)\\perp g_i(x),g_{i,j}(x)=0 \\, (j\\in\\mathcal{E}_i,i\\in[N]),\\\\\r\ng_{i,j}(x)\\ge0,\\hat{\\lambda}_{i,j}(x)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]).\r\n\\end{array}\r\n\\right.\r\n\\ee\r\nUnder some constraint qualifications,\r\nif $x$ is a GNE, then it satisfies (\\ref{eq:KKTrational}).\r\nFor convex GNEPs, if $x$ satisfies (\\ref{eq:KKTrational}) and $q_i(x)>0$,\r\nthen $x$ must be a GNE, since it satisfies\r\n(\\ref{eq:KKTwithLMalp}) with $\\lambda_{i,j}$ given by\r\n$\\lambda_{i,j}=\\hat{\\lambda}_{i,j}(x)/{q_i(x)}.$\r\n\n\n\\begin{prop}\r\n\\label{pr:rtn}\r\nFor the GNEPP given by (\\ref{eq:GNEP}), suppose for each $i\\in[N]$,\r\nthe Lagrange multiplier vector $\\lambda_i$\r\nhas the rational expression as in Definition~\\ref{LME:rat}.\r\n\\begin{enumerate}[(i)]\r\n\\item If (\\ref{eq:KKTfeasoptrtn}) is infeasible,\r\nthen the GNEP has no KKT points. Therefore,\r\nif every GNE is a KKT point,\r\nthen the infeasibility of (\\ref{eq:KKTfeasoptrtn})\r\nimplies the nonexistence of GNEs.\r\n\r\n\\item Assume the GNEP is convex.\r\nIf $u$ is a feasible point of (\\ref{eq:KKTfeasoptrtn})\r\nand $q_i(u)>0$ for all $i\\in[N]$, then $u$ must be a GNE.\r\n\\end{enumerate}\r\n\\end{prop}\n\nThe optimality conditions (\\ref{eq:KKTwithLMalp})\r\ncan be equivalently expressed as\r\n\\be\r\n\\label{eq:KKTpartial}\r\n\\left\\{\r\n\\begin{array}{l}\r\n\\nabla_{x_i} f_i(x)-\\sum_{j=1}^{m_i}\\hat{\\lambda}_{i,j}(\\allx)\\nabla_{x_i} g_{i,j}(x)=0 \\, (i\\in[N]),\\\\\r\n\\hat{\\lambda}_i(\\allx)\\perp g_i(x),g_{i,j}(x)=0 \\, (j\\in\\mathcal{E}_i, i\\in[N]),\\\\\r\ng_{i,j}(x)\\ge0,\\hat{\\lambda}_{i,j}(\\allx)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]).\r\n\\end{array}\r\n\\right.\r\n\\ee\r\nFor convex GNEPs, a point $x$ is a GNE if and only if there exists\r\n$\\omega:= (\\omega_1,\\ldots,\\omega_N)$ such that $\\allx$ satisfies (\\ref{eq:KKTpartial}).\r\n\n\n\\begin{prop}\r\n\\label{pr:prt}\r\nFor the GNEPP given by (\\ref{eq:GNEP}),\r\nsuppose each player's optimization has a\r\nparametric expression for their Lagrange multipliers\r\nas in Definition~\\ref{def:para:LME}.\r\n\\begin{enumerate}[(i)]\r\n\r\n\\item If (\\ref{eq:KKTfeasoptprt}) is infeasible,\r\nthen the GNEP has no KKT points. If every GNE is a KKT point,\r\nthen the infeasibility of (\\ref{eq:KKTfeasoptprt})\r\nimplies nonexistence of GNEs.\r\n\r\n\\item Assume the GNEP is convex.\r\nIf $(u,w)$ is a feasible point of (\\ref{eq:KKTfeasoptprt}),\r\nthen $u$ is a GNE.\r\n\\end{enumerate}\r\n\\end{prop}\n\n\\begin{proof}\r\n(i) This is directly implied by Propositions~\\ref{pr:rtn} and \\ref{pr:prt}.\r\n\r\n(ii) If (\\ref{eq:KKTfeasopt}) is infeasible,\r\nthen there is no GNE, because every GNE is assumed to be a KKT point\r\nand it must be feasible for (\\ref{eq:KKTfeasopt}).\r\nNext, assume (\\ref{eq:KKTfeasopt}) is feasible.\r\nSince $\\Theta$ is positive definite,\r\nthe optimization (\\ref{eq:KKTfeasopt}) has a minimizer,\r\nsay, $(u,w)$. By the given assumption,  we have $q_i(u)>0$ for all $i$.\r\nSo $u$ is a GNE, by (i).\r\n\\end{proof}","units":[{"start":28675,"end":29610,"start_line":774,"end_line":795,"raw_tex":"Suppose for each $i$, there exists a rational expression\r\n$\\hat{\\lambda}_i/q_i$ for the $i$th player's Lagrange multiplier vector.\r\nSince $q_{i}(x)\\lambda_{i,j}=\\hat{\\lambda}_i(x)$\r\nand $q_i(x)\\ge0$ for all $x\\in X$,\r\nthe following holds for all KKT points\r\n\\be\r\n\\label{eq:KKTrational}\r\n\\left\\{\r\n\\begin{array}{l}\r\nq_i(x)\\nabla_{x_i} f_i(x)-\\sum\\nolimits_{j=1}^{m_i}\r\n \\hat{\\lambda}_{i,j}(x)\\nabla_{x_i} g_{i,j}(x)=0 \\, (i\\in[N]),\\\\\r\n\\hat{\\lambda}_i(x)\\perp g_i(x),g_{i,j}(x)=0 \\, (j\\in\\mathcal{E}_i,i\\in[N]),\\\\\r\ng_{i,j}(x)\\ge0,\\hat{\\lambda}_{i,j}(x)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]).\r\n\\end{array}\r\n\\right.\r\n\\ee\r\nUnder some constraint qualifications,\r\nif $x$ is a GNE, then it satisfies (\\ref{eq:KKTrational}).\r\nFor convex GNEPs, if $x$ satisfies (\\ref{eq:KKTrational}) and $q_i(x)>0$,\r\nthen $x$ must be a GNE, since it satisfies\r\n(\\ref{eq:KKTwithLMalp}) with $\\lambda_{i,j}$ given by\r\n$\\lambda_{i,j}=\\hat{\\lambda}_{i,j}(x)/{q_i(x)}.$\r\n","sha256_utf8":"f67305ad83b9f2ddf9da2449748f0abb1d0b6c7eaadac1b876dd029e9e610013"},{"start":30294,"end":30912,"start_line":813,"end_line":829,"raw_tex":"\\begin{prop}\r\n\\label{pr:rtn}\r\nFor the GNEPP given by (\\ref{eq:GNEP}), suppose for each $i\\in[N]$,\r\nthe Lagrange multiplier vector $\\lambda_i$\r\nhas the rational expression as in Definition~\\ref{LME:rat}.\r\n\\begin{enumerate}[(i)]\r\n\\item If (\\ref{eq:KKTfeasoptrtn}) is infeasible,\r\nthen the GNEP has no KKT points. Therefore,\r\nif every GNE is a KKT point,\r\nthen the infeasibility of (\\ref{eq:KKTfeasoptrtn})\r\nimplies the nonexistence of GNEs.\r\n\r\n\\item Assume the GNEP is convex.\r\nIf $u$ is a feasible point of (\\ref{eq:KKTfeasoptrtn})\r\nand $q_i(u)>0$ for all $i\\in[N]$, then $u$ must be a GNE.\r\n\\end{enumerate}\r\n\\end{prop}","sha256_utf8":"b3144e7c8cbb6fe04cd9980d3e079d9eb28166c8fae490b5b9ebd75cf0607d79"},{"start":46768,"end":47359,"start_line":1244,"end_line":1257,"raw_tex":"The optimality conditions (\\ref{eq:KKTwithLMalp})\r\ncan be equivalently expressed as\r\n\\be\r\n\\label{eq:KKTpartial}\r\n\\left\\{\r\n\\begin{array}{l}\r\n\\nabla_{x_i} f_i(x)-\\sum_{j=1}^{m_i}\\hat{\\lambda}_{i,j}(\\allx)\\nabla_{x_i} g_{i,j}(x)=0 \\, (i\\in[N]),\\\\\r\n\\hat{\\lambda}_i(\\allx)\\perp g_i(x),g_{i,j}(x)=0 \\, (j\\in\\mathcal{E}_i, i\\in[N]),\\\\\r\ng_{i,j}(x)\\ge0,\\hat{\\lambda}_{i,j}(\\allx)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]).\r\n\\end{array}\r\n\\right.\r\n\\ee\r\nFor convex GNEPs, a point $x$ is a GNE if and only if there exists\r\n$\\omega:= (\\omega_1,\\ldots,\\omega_N)$ such that $\\allx$ satisfies (\\ref{eq:KKTpartial}).\r\n","sha256_utf8":"e73d82c78b80c9742a583068130e033915acc5ac5a173d7c7d12bb69af82a69c"},{"start":47966,"end":48537,"start_line":1272,"end_line":1289,"raw_tex":"\\begin{prop}\r\n\\label{pr:prt}\r\nFor the GNEPP given by (\\ref{eq:GNEP}),\r\nsuppose each player's optimization has a\r\nparametric expression for their Lagrange multipliers\r\nas in Definition~\\ref{def:para:LME}.\r\n\\begin{enumerate}[(i)]\r\n\r\n\\item If (\\ref{eq:KKTfeasoptprt}) is infeasible,\r\nthen the GNEP has no KKT points. If every GNE is a KKT point,\r\nthen the infeasibility of (\\ref{eq:KKTfeasoptprt})\r\nimplies nonexistence of GNEs.\r\n\r\n\\item Assume the GNEP is convex.\r\nIf $(u,w)$ is a feasible point of (\\ref{eq:KKTfeasoptprt}),\r\nthen $u$ is a GNE.\r\n\\end{enumerate}\r\n\\end{prop}","sha256_utf8":"4c444a57e65cb9b26cdcc9f87fc45b24a3cb30ec9b7dbdb95746c7c59bedc6f4"},{"start":52986,"end":53502,"start_line":1415,"end_line":1426,"raw_tex":"\\begin{proof}\r\n(i) This is directly implied by Propositions~\\ref{pr:rtn} and \\ref{pr:prt}.\r\n\r\n(ii) If (\\ref{eq:KKTfeasopt}) is infeasible,\r\nthen there is no GNE, because every GNE is assumed to be a KKT point\r\nand it must be feasible for (\\ref{eq:KKTfeasopt}).\r\nNext, assume (\\ref{eq:KKTfeasopt}) is feasible.\r\nSince $\\Theta$ is positive definite,\r\nthe optimization (\\ref{eq:KKTfeasopt}) has a minimizer,\r\nsay, $(u,w)$. By the given assumption,  we have $q_i(u)>0$ for all $i$.\r\nSo $u$ is a GNE, by (i).\r\n\\end{proof}","sha256_utf8":"10dcb6822ce4f62bceeefa59fa0ede919ef6db4bb76fa3b18e19c2e1665fe1b4"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"多项式广义纳什问题","source_units":[{"start":6665,"end":7745,"start_line":238,"end_line":260,"raw_tex":"The Generalized Nash Equilibrium Problem (GNEP) is a kind of game to find strategies\r\nfor a group of players such that each player's objective function is optimized,\r\nfor given other players' strategies.\r\nSuppose there are  $N$  players and the $i$th player's strategy is a vector\r\n$x_{i}\\in\\mathbb{R}^{n_{i}}$ (the $n_i$-dimensional real Euclidean space).\r\nWe write that\r\n\\[x_i:=(x_{i,1},\\ldots,x_{i,n_i}),\\quad x:=(x_1,\\ldots,x_N).\\]\r\nThe total dimension of all strategies is $n := n_1+ \\ldots + n_N.$\r\nThe main task of the GNEP is to find a tuple\r\n$u = (u_1, \\ldots, u_N)$ of strategies\r\nsuch that each $u_i$ is a minimizer of the $i$th player's optimization\r\n\\be\r\n\\label{eq:GNEP}\r\n%\\tag{$\\mathcal{SP}$-i}\r\n\\mbox{F}_i(u_{-i}): \\,\r\n\\left\\{ \\begin{array}{cl}\r\n\\min\\limits_{\\xpi\\in \\Rni}  &  \\fpi(u_1, \\ldots, u_{i-1}, x_i, u_{i+1},\\ldots, u_N) \\\\\r\n\\st & g_{i,j}(u_1, \\ldots, u_{i-1}, x_i, u_{i+1},\\ldots, u_N)  = 0 \\,\r\n        (j\\in \\mathcal{E}_i),\\\\\r\n\t& g_{i,j}(u_1, \\ldots, u_{i-1}, x_i, u_{i+1},\\ldots, u_N)  \\geq 0 \\,\r\n         (j\\in \\mathcal{I}_i),\r\n\\end{array} \\right.\r\n\\ee","sha256_utf8":"ca82c1000570c98196f24e9a534a6be0b63b04f0d8cd608fcfb43e4edb24b60d"},{"start":7747,"end":8573,"start_line":261,"end_line":275,"raw_tex":"where $u_{-i} := (u_1, \\ldots, u_{i-1}, u_{i+1},\\ldots, u_N)$,\r\nthe $f_i$ and $g_{i,j}$ are continuously differentiable functions in $x_i$,\r\nand the $\\mc{E}_i$, $\\mc{I}_i$ are disjoint finite (possibly empty) labeling sets.\r\nThe point $u$ satisfying the above is called a Generalized Nash Equilibrium (GNE).\r\nFor notational convenience, when the $i$th player's strategy is considered,\r\nwe use $x_{-i}$ to denote the subvector of all players' strategies\r\nexcept the $i$th one, i.e.,\r\n\\[\r\nx_{-i} \\, := \\, (x_1, \\ldots, x_{i-1}, x_{i+1}, \\ldots, x_N),\r\n\\]\r\nand write $x=(x_{i},x_{-i})$ accordingly.\r\n\r\nThis paper focuses on the Generalized Nash Equilibrium Problem of Polynomials (GNEPP),\r\ni.e., all the functions $f_i$ and $g_{i,j}$ are polynomials in $x$.\r\nFor each $i=1,\\ldots,N$, let $X_i$ be the point-to-set map such that\r\n","sha256_utf8":"4e241a0b5f7112f44813035e2aeac4588a021ee77509dde5c57fa0d54718490e"}]},{"id":"S002","semantic_type":"definition","kind":"definition","title":"策略集与联合可行集","source_units":[{"start":8573,"end":8792,"start_line":276,"end_line":282,"raw_tex":"\\be \\label{eq:feaset}\r\nX_i(\\xmi) \\, := \\,\r\n\\left\\{x_i \\in \\Rni \\left|\\begin{array}{l}\r\ng_{i,j}(\\xpi,\\xmi) = 0 ,\\, j\\in \\mathcal{E}_i,\\\\\r\ng_{i,j}(\\xpi,\\xmi) \\geq 0 ,\\, j\\in \\mathcal{I}_i\r\n\\end{array}\\right.\\right\\}.\r\n\\ee","sha256_utf8":"6eb377580155abfce6c3f5018505a79fc678d6c84621fd3f5827e1a0fb9a4f4b"},{"start":8794,"end":9290,"start_line":283,"end_line":296,"raw_tex":"The $X_i(\\xmi)$ is the feasible strategy set of $\\mbox{F}_i(\\xmi)$.\r\nThe domain of $X_i$ is\r\n\\[\\dom(X_i):=\\{x_{-i}\\in\\re^{n-n_i}:X_i(\\xmi)\\ne\\emptyset\\}.\\]\r\nThe tuple $x$ is said to be a feasible point of the GNEP if\r\n$\\xpi\\in X_i(\\xmi)$ for all $i$. Denote the set\r\n\\be\r\nX :=\r\n\\left\\{  x\\in\\re^n\r\n\\left| \\begin{array}{l}\r\n g_{i,j}(\\xpi,\\xmi) = 0, \\, j\\in \\mathcal{E}_i,\\, i=1,\\ldots, N,\\\\\r\n g_{i,j}(\\xpi,\\xmi) \\geq 0, \\, j\\in \\mathcal{I}_i,\\, i=1,\\ldots, N\r\n\\end{array} \\right.\r\n\\right \\} .\r\n\\ee","sha256_utf8":"a320df36c9c2366d51419902af8fd33b6a0c97959e596b1f677cb8d753b29e92"},{"start":9292,"end":9360,"start_line":297,"end_line":297,"raw_tex":"Then $x$ is a feasible point for the GNEP if and only if $x\\in X.$\r\n","sha256_utf8":"2602a6d0d7159e0b14de8e8c8701891d3205ca8e6a59f4bb7d636cf87335280d"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"玩家凸性","source_units":[{"start":9364,"end":9890,"start_line":300,"end_line":311,"raw_tex":"\\begin{defi} \\label{df:cgnep} \\rm\r\nThe GNEP given by (\\ref{eq:GNEP}) is called {\\it convex}\r\n\\footnote{\r\nIn some literature, this is also called player-convex,\r\nto distinguish from jointly-convex GNEPs; see \\cite{dreves2012nonsmooth}.}\r\nif for all\r\n$i=1,\\ldots, N$ and for all given $\\xmi\\in\\dom(X_i)$,\r\nthe objective $f_i(\\xpi,\\xmi)$ is convex in $x_i$ on $X_i(\\xmi)$,\r\nall $g_{i,j}(\\xpi,\\xmi) \\, (j\\in\\mc{E}_i)$\r\nare affine linear in $\\xpi$,\r\nand all $g_{i,j}(\\xpi,\\xmi) \\, (j\\in\\mc{I}_i)$ are concave in $\\xpi$.\r\n\\end{defi}","sha256_utf8":"b6e9e7285b96728b2bc55460f8c948fb1c8e691ec1f4358c059ab38f715c613a"}]},{"id":"S004","semantic_type":"notation","kind":"notation","title":"多项式与单项式记号","source_units":[{"start":14508,"end":15868,"start_line":412,"end_line":434,"raw_tex":"The symbol $\\mathbb N$ (resp., $\\mathbb R$, $\\mathbb C$) stands for the set of\r\nnonnegative integers (resp., real numbers, complex numbers).\r\nFor a positive integer $k$, denote the set $[k] := \\{1, \\ldots, k\\}$.\r\nFor a real number $t$, $\\lceil t \\rceil$ (resp., $\\lfloor t \\rfloor$)\r\ndenotes the smallest integer not smaller than $t$\r\n(resp., the biggest integer not bigger than $t$).\r\nWe use $e_i$ to denote the vector such that the $i$th entry is\r\n$1$ and all others are zeros.\r\nBy writing $A\\succeq0$ (resp., $A\\succ0$), we mean that the matrix $A$\r\nis symmetric positive semidefinite (resp., positive definite).\r\nFor the $i$th player's strategy vector $x_i \\in \\re^{n_i}$,\r\nthe $x_{i,j}$ denotes the $j$th entry of $x_i$, for $j = 1, \\ldots, n_i$.\r\nWhen we write $(y,x_{-i})$, it means that the $i$th player's strategy is $y\\in\\Rni$,\r\nwhile the vector of all other players' strategy is fixed to be $\\xmi$.\r\nLet $\\nfR[x]$ denote the ring of polynomials\r\nwith real coefficients in $x$,\r\nand $\\nfR[x]_d$ denote its subset of polynomials whose degrees are not greater than $d$.\r\nFor the $i$th player's strategy vector $x_i$,\r\nthe notation $\\re[x_i]$ and $\\re[x_i]_d$\r\nare defined in the same way.\r\nFor $i$th player's objective $f_i(x)$,\r\nthe notation $\\nabla_{x_i}f_i$, $\\nabla^2_{x_i}f_i$\r\nrespectively denote its gradient and Hessian with respect to $x_i$.\r\n","sha256_utf8":"12ce49683b8730d7fbfb0ed5ea9c5559c583e484a94ae2e9fe02b817309c7590"},{"start":15870,"end":16642,"start_line":436,"end_line":454,"raw_tex":"In the following, we use the letter $z$ to represent either\r\n$x$, $x_i$ or $(x,\\omega)$ for some new variables $\\omega$,\r\nfor convenience of discussion.\r\nSuppose $z := (z_1,\\ldots,z_l)$.\r\nFor a polynomial $p(z)\\in\\re[z]$,\r\nthe $p=0$ means $p(z)$ is identically zero on $\\re^l$.\r\nWe say the polynomial $p$ is nonzero if $p\\ne0$.\r\nLet $\\af := (\\af_1, \\ldots, \\af_l) \\in \\N^{l}$, and we denote\r\n\\[\r\nz^\\alpha := z_1^{\\alpha_1} \\cdots z_l^{\\alpha_l}, \\quad\r\n|\\alpha|:=\\alpha_1+\\ldots+\\alpha_l.\r\n\\]\r\nFor an integer $d >0$, denote the monomial power set\r\n\\[\r\n{\\mathbb{N}}_d^l \\, := \\,\r\n\\{\\alpha\\in {\\mathbb{N}}^l: \\, \\ |\\alpha| \\le d \\}.\r\n\\]\r\nWe use $[z]_d$ to denote the vector of all monomials in $z$\r\nwhose degree is at most $d$, ordered in the graded alphabetical ordering.\r\n","sha256_utf8":"5e2feb69d200e66f24fc56328e7ec4e4b9f42f83309aec258665589643205f7d"},{"start":16772,"end":16935,"start_line":460,"end_line":462,"raw_tex":"Throughout the paper, a property is said to hold {\\it generically}\r\nif it holds for all points in the space of input data\r\nexcept a set of Lebesgue measure zero.\r\n","sha256_utf8":"f89456d59f24cd7681bea4953a38fe9074348dcd2dd4e5461684c3c13c32034d"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"临界对与KKT条件","source_units":[{"start":21555,"end":22478,"start_line":583,"end_line":603,"raw_tex":"We study optimality conditions for Generalized Nash Equilibrium Problems.\r\nConsider the $i$th player's optimization.\r\nFor convenience, suppose $\\mc{E}_i\\cup\\mc{I}_i = [m_i]$\r\nand $g_i:=(g_{i,1},\\ldots,g_{i,m_i})$. For a given $\\xmi$, \r\nunder some suitable constraint qualifications\r\n(e.g., the linear independence constraint qualification (LICQ),\r\nMangasarian-Fromovite constraint qualification (MFCQ),\r\nor the Slater's Condition; see \\cite{Brks} for them),\r\nif $x_i$ is a minimizer of $\\mbox{F}_i(\\xmi)$,\r\nthen there exists a Lagrange multiplier vector\r\n$\\lambda_i:=(\\lambda_{i,1},\\ldots,\\lambda_{i,m_i})$ such that\r\n\\be\r\n\\label{eq:KKTwithLM}\r\n\\left\\{\r\n\\begin{array}{l}\r\n  \\nabla_{x_i} f_i(x)-\\sum_{j=1}^{m_i}\\lambda_{i,j}\\nabla_{x_i} g_{i,j}(x)=0,\\\\\r\n  \\lambda_i\\perp g_i(x),\\,g_{i,j}(x)=0\\,(j\\in\\mathcal{E}_i),\\\\\r\n  \\lambda_{i,j}\\ge0\\,(j\\in\\mathcal{I}_i),\\,g_{i,j}(x)\\ge0\\,(j\\in\\mathcal{I}_i).\r\n\\end{array}\r\n\\right.\r\n\\ee","sha256_utf8":"04e8ba95d060b662a5132a1f0fce335889dcc14cbc9bc1af2c31814540e74b2e"},{"start":22480,"end":22635,"start_line":604,"end_line":605,"raw_tex":"This is called the first order Karush-Kuhn-Tucker system for $\\mbox{F}_i(\\xmi)$.\r\nSuch $(x_i,\\lambda_i)$ is called a critical pair of $\\mbox{F}_i(\\xmi)$.\r\n","sha256_utf8":"8ceb8a383beee247a90aec2fc62036cebe9029134560f3e7a6cd9a1eb8c2b2b0"},{"start":22835,"end":23185,"start_line":609,"end_line":618,"raw_tex":"\\be\r\n\\label{eq:KKTwithLMalp}\r\n\\left\\{\r\n\\begin{array}{l}\r\n\\nabla_{x_i} f_i(x)-\\sum_{j=1}^{m_i}\\lambda_{i,j}\\nabla_{x_i} g_{i,j}(x)=0\\,(i\\in[N]),\\\\\r\n\\lambda_i\\perp g_i(x)\\,(i\\in[N]),\\,g_{i,j}(x)=0\\,(i\\in[N],j\\in\\mathcal{E}_i),\\\\\r\n\\lambda_{i,j}\\ge0\\,(i\\in[N],j\\in\\mathcal{I}_i),\\,g_{i,j}(x)\\ge0\\,(i\\in[N],j\\in\\mathcal{I}_i).\\\\\r\n\\end{array}\r\n\\right.\r\n\\ee","sha256_utf8":"26d824d1e03e9efcbcc863427d8fb59cfd313881588a939df7b59dfe1512a91c"},{"start":23187,"end":23349,"start_line":619,"end_line":620,"raw_tex":"A point $x$ satisfying (\\ref{eq:KKTwithLMalp}) is called a KKT point for the GNEP.\r\nFor convex GNEPs, each KKT point is a GNE \\cite[Theorem~4.6]{Facchinei2010}.\r\n","sha256_utf8":"b3fadb18fac39dedeefa14ff3ee46a0ff15b30fd0b862955941b7b5234d913fc"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"有理乘子表达","source_units":[{"start":26364,"end":26895,"start_line":700,"end_line":712,"raw_tex":"\\begin{defi} \\label{LME:rat} \\rm\r\nFor the $i$th player's optimization $\\mbox{F}_i(\\xmi)$,\r\nif there exist polynomials $\\hat{\\lambda}_{i,1},\\ldots,\\hat{\\lambda}_{i,m_i}$ \r\nand a nonzero polynomial $q_i$ such that\r\n$q_i(x)\\ge 0$ for all $x\\in X$, and $\\hat{\\lambda}_{i,j}(x)=q_i(x)\\lambda_{i,j}$ \r\nholds for all critical pairs $(x_i,\\lambda_i)$,\r\nthen we call the tuple\r\n\\[\r\n\\hat{\\lambda}_i/q_i:=(\\hat{\\lambda}_{i,1}(x)/q_i(x),\r\n\\ldots,\\hat{\\lambda}_{i,m_i}(x)/q_i(x))\r\n\\]\r\na rational expression for Lagrange multipliers.\r\n\\end{defi}","sha256_utf8":"0cb4fbc76548a756bf059e036f6a3cd687add9d2fb9530d9b6a74a5f38a1e985"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"参数乘子表达","source_units":[{"start":40221,"end":40832,"start_line":1061,"end_line":1069,"raw_tex":"\\begin{defi} \\label{def:para:LME} \\rm\r\nFor the $i$th player's optimization $\\mbox{F}_i(\\xmi)$,\r\na parametric expression for the Lagrange multipliers is a tuple of polynomials\r\n\\[ \\hat{\\lambda}_{i}(x, \\omega_i) := (\\hat{\\lambda}_{i,1}(x, \\omega_i),\\ldots,\\hat{\\lambda}_{i,m_i}(x, \\omega_i) ), \\]\r\nin $x$ and in a parameter $\\omega_i:=(\\omega_{i,1},\\ldots,\\omega_{i,s_i})$ with $s_i\\le m_i$,\r\nsuch that $(x_i,\\lambda_i)$ is a critical pair if and only if\r\nthere is a value of $\\omega_i$ such that (\\ref{eq:KKTwithLM}) is satisfied for\r\n$\\lambda_{i,j}=\\hat{\\lambda}_{i,j}(x,\\omega_i)$ with $j\\in[m_i]$.\r\n\\end{defi}","sha256_utf8":"bf7edd2ef661d5c37f3ac8a4c919f4d85985c8748d14b8f5450eef9a6e7ef6c8"}]},{"id":"S008","semantic_type":"problem_setting","kind":"problem_setting","title":"有理表达优化模型","source_units":[{"start":29672,"end":30170,"start_line":797,"end_line":809,"raw_tex":"\\be\r\n\\label{eq:KKTfeasoptrtn}\r\n\\left\\{\r\n\\begin{array}{rll}\r\n\\min\\limits_{x\\in X} & [x]_1^T\\Theta[x]_1 \\\\\r\n\\st &q_i(x)\\nabla_{x_i} f_i(x)-\\sum\\nolimits_{j=1}^{m_i}\\hat{\\lambda}_{i,j}(x)\\nabla_{x_i} g_{i,j}(x)=0\\,(i\\in[N]),\\\\\r\n\t&\\hat{\\lambda}_{i,j}(x)\\perp g_{i,j}(x)\\,(j\\in\\mathcal{E}_i\\cup\\mc{I}_i,i\\in[N]),\\\\\r\n\t% &g_{i,j}(x)=0\\,(j\\in\\mathcal{E}_i,i\\in[N]),\\\\\r\n\t% &g_{i,j}(x)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]),\\\\\r\n\t&\\hat{\\lambda}_{i,j}(x)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]).\r\n\\end{array}\r\n\\right.\r\n\\ee","sha256_utf8":"c432e46a6ce67ecea46394e82b27b5192f1bda950a50c01e7fbb23c4b89f7870"},{"start":30172,"end":30247,"start_line":810,"end_line":811,"raw_tex":"In the above,\r\n$\\Theta$ is a generically chosen positive definite matrix.\r\n","sha256_utf8":"2d790c9fe152cea026d4875649df2ce0548df829347389c7816982ffb88a547f"}]},{"id":"S009","semantic_type":"problem_setting","kind":"problem_setting","title":"参数表达优化模型","source_units":[{"start":47400,"end":47839,"start_line":1259,"end_line":1269,"raw_tex":"\\be\r\n\\label{eq:KKTfeasoptprt}\r\n\\left\\{\r\n\\begin{array}{rll}\r\n\\min\\limits_{\\allx\\in X\\times\\re^s} & \\lvt\\allx\\rvt_1^T\\Theta\\lvt\\allx\\rvt_1 \\\\\r\n\\st &\\nabla_{x_i} f_i(x)-\\sum\\nolimits_{j=1}^{m_i}\\hat{\\lambda}_{i,j}(\\allx)\\nabla_{x_i} g_{i,j}(x)=0 \\, (i\\in[N]),\\\\\r\n\t&\\hat{\\lambda}_{i,j}(\\allx)\\perp g_{i,j}(x)\\,(j\\in\\mathcal{E}_i\\cup\\mc{I}_i,i\\in[N]),\\\\\r\n\t&\\hat{\\lambda}_{i,j}(\\allx)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]).\r\n\\end{array}\r\n\\right.\r\n\\ee","sha256_utf8":"bbfe3ee5d8b30ec19efa1b499777d0008198206364150926eb8b2af7615b422f"},{"start":47841,"end":47919,"start_line":1270,"end_line":1270,"raw_tex":"In the above, the $\\Theta$ is a generically chosen positive definite matrix.\r\n","sha256_utf8":"4d782b2985f1099db1cb61da0e55ae9c7abbf4dde7f2f47e5c3f64882efa8ec0"}]},{"id":"S010","semantic_type":"notation","kind":"notation","title":"扩展变量与混合约定","source_units":[{"start":46557,"end":46768,"start_line":1238,"end_line":1243,"raw_tex":"Suppose all players have parametric expressions for their Lagrange multipliers\r\nas in Definition~\\ref{def:para:LME}. \r\nLet $s := s_1+\\ldots+s_N$, and denote\r\n\\[\r\n\\allx \\,:=\\, (x,\\omega_1,\\ldots,\\omega_N). \r\n\\]\r\n","sha256_utf8":"2987f401beb304a312ea56c3acb04fab938760683d68ec28bd21972cead47908"},{"start":48622,"end":49154,"start_line":1299,"end_line":1310,"raw_tex":"In this section, we give an algorithm for solving convex GNEPs.\r\nWe assume each $\\lambda_i$ has either a rational or parametric expression,\r\nas in Definition~\\ref{LME:rat} or \\ref{def:para:LME}.\r\nIf $\\lambda_i$ has a polynomial or parametric expression,\r\nwe let $q_i(x) :=1$.\r\nIf $\\lambda_i$ has a polynomial or rational expression,\r\nthen we let $s_i=0$. Recall the notation\r\n\\[\r\n\\allx \\,:= \\, (x,\\omega_1,\\ldots,\\omega_N).\r\n\\]\r\nChoose a generic positive definite matrix $\\Theta$.\r\nThen solve the following polynomial optimization\r\n","sha256_utf8":"e55ef0468f81ed0592d32f3a46a9f7cdc27a6b9e6ddb8e46b548b8bf34f2cf1d"}]},{"id":"S011","semantic_type":"problem_setting","kind":"problem_setting","title":"统一多项式优化模型","source_units":[{"start":49154,"end":49678,"start_line":1311,"end_line":1323,"raw_tex":"\\be\r\n\\label{eq:KKTfeasopt}\r\n\\left\\{\r\n\\begin{array}{rll}\r\n\\min\\limits_{\\allx} & \\lvt\\allx\\rvt_1^T\\Theta\\lvt\\allx\\rvt_1 \\\\\r\n\\st &q_i(x)\\nabla_{x_i} f_i(x)-\\sum\\nolimits_{j=1}^{m_i}\\hat{\\lambda}_{i,j}(\\allx)\\nabla_{x_i} g_{i,j}(x)=0 \\, (i\\in[N]),\\\\\r\n\t&\\hat{\\lambda}_{i,j}(\\allx)\\perp g_{i,j}(x)\\,(j\\in\\mathcal{E}_i\\cup\\mc{I}_i,i\\in[N]),\\\\\r\n\t&g_{i,j}(x)=0\\,(j\\in\\mathcal{E}_i,i\\in[N]),\\\\\r\n\t&g_{i,j}(x)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]),\\\\\r\n\t&\\hat{\\lambda}_{i,j}(\\allx)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]).\r\n\\end{array}\r\n\\right.\r\n\\ee","sha256_utf8":"0e795842d8d7ac6664f40fc258d539d219e749819d79854179ab2d8ff1e9d3a3"}]},{"id":"S012","semantic_type":"problem_setting","kind":"problem_setting","title":"单玩家均衡检验","source_units":[{"start":50124,"end":50404,"start_line":1333,"end_line":1342,"raw_tex":"\\begin{equation}\r\n\\left\\{\r\n\\begin{array}{llll}\r\n\\delta_i := & \\min\\limits_{\\xpi}& f_i(x_i,u_{-i})-f_i(u_{i},u_{-i})&\\\\\r\n &\\st  &  g_{i,j}(x_{i},u_{-i})=0 \\, (j\\in\\mc{E}_i), \\\r\n g_{i,j}(x_{i},u_{-i})\\ge0 \\, (j\\in\\mc{I}_i).\r\n\\end{array}\r\n\\right.\r\n\\label{eq:checkopt}\r\n\\end{equation}","sha256_utf8":"dabada5f689b17922ec50edd55c9a2dd46efd7e83423fd6ddf446b148841aa5f"}]},{"id":"S013","semantic_type":"algorithm","kind":"algorithm","title":"均衡求解算法","source_units":[{"start":50655,"end":51344,"start_line":1350,"end_line":1373,"raw_tex":"\\begin{alg}\r\n\\label{ag:KKTSDP} \\rm\r\nFor the convex GNEP given by (\\ref{eq:GNEP}),\r\ndo the following:\r\n\r\n\\begin{description}\r\n\r\n\\item [Step~0]\r\nChoose a generic positive definite matrix $\\Theta$ of length $n+s+1$.\r\n\r\n\\item [Step~1]\r\nSolve the polynomial optimization \\reff{eq:KKTfeasopt}.\r\nIf it is infeasible, then there are no KKT points and stop;\r\notherwise, solve it for a minimizer $(u,w)$.\r\n\r\n\\item [Step~2]\r\nIf all $q_i(u) >0$, then $u$ is a GNE. Otherwise,\r\nfor those $i$ with $q_i(u) \\le 0$, solve the optimization \\reff{eq:checkopt}\r\nfor the minimum value $\\delta_i$.\r\nIf $\\delta_i \\ge 0$ for all such $i$, then $u$ is a GNE;\r\notherwise, it is not.\r\n\r\n\\end{description}\r\n\\end{alg}","sha256_utf8":"87d436b302f9cd3e486a3090cae081d83aeb6d31fe74e557db17b6f996eedbc2"}]},{"id":"S014","semantic_type":"proposition","kind":"proposition","title":"有理表达均衡判据","source_units":[{"start":30294,"end":30912,"start_line":813,"end_line":829,"raw_tex":"\\begin{prop}\r\n\\label{pr:rtn}\r\nFor the GNEPP given by (\\ref{eq:GNEP}), suppose for each $i\\in[N]$,\r\nthe Lagrange multiplier vector $\\lambda_i$\r\nhas the rational expression as in Definition~\\ref{LME:rat}.\r\n\\begin{enumerate}[(i)]\r\n\\item If (\\ref{eq:KKTfeasoptrtn}) is infeasible,\r\nthen the GNEP has no KKT points. Therefore,\r\nif every GNE is a KKT point,\r\nthen the infeasibility of (\\ref{eq:KKTfeasoptrtn})\r\nimplies the nonexistence of GNEs.\r\n\r\n\\item Assume the GNEP is convex.\r\nIf $u$ is a feasible point of (\\ref{eq:KKTfeasoptrtn})\r\nand $q_i(u)>0$ for all $i\\in[N]$, then $u$ must be a GNE.\r\n\\end{enumerate}\r\n\\end{prop}","sha256_utf8":"b3144e7c8cbb6fe04cd9980d3e079d9eb28166c8fae490b5b9ebd75cf0607d79"}]},{"id":"S015","semantic_type":"proof","kind":"proof","title":"有理表达判据论证","source_units":[{"start":28675,"end":29610,"start_line":774,"end_line":795,"raw_tex":"Suppose for each $i$, there exists a rational expression\r\n$\\hat{\\lambda}_i/q_i$ for the $i$th player's Lagrange multiplier vector.\r\nSince $q_{i}(x)\\lambda_{i,j}=\\hat{\\lambda}_i(x)$\r\nand $q_i(x)\\ge0$ for all $x\\in X$,\r\nthe following holds for all KKT points\r\n\\be\r\n\\label{eq:KKTrational}\r\n\\left\\{\r\n\\begin{array}{l}\r\nq_i(x)\\nabla_{x_i} f_i(x)-\\sum\\nolimits_{j=1}^{m_i}\r\n \\hat{\\lambda}_{i,j}(x)\\nabla_{x_i} g_{i,j}(x)=0 \\, (i\\in[N]),\\\\\r\n\\hat{\\lambda}_i(x)\\perp g_i(x),g_{i,j}(x)=0 \\, (j\\in\\mathcal{E}_i,i\\in[N]),\\\\\r\ng_{i,j}(x)\\ge0,\\hat{\\lambda}_{i,j}(x)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]).\r\n\\end{array}\r\n\\right.\r\n\\ee\r\nUnder some constraint qualifications,\r\nif $x$ is a GNE, then it satisfies (\\ref{eq:KKTrational}).\r\nFor convex GNEPs, if $x$ satisfies (\\ref{eq:KKTrational}) and $q_i(x)>0$,\r\nthen $x$ must be a GNE, since it satisfies\r\n(\\ref{eq:KKTwithLMalp}) with $\\lambda_{i,j}$ given by\r\n$\\lambda_{i,j}=\\hat{\\lambda}_{i,j}(x)/{q_i(x)}.$\r\n","sha256_utf8":"f67305ad83b9f2ddf9da2449748f0abb1d0b6c7eaadac1b876dd029e9e610013"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"缩放KKT条件","source_units":[{"start":28675,"end":29287,"start_line":774,"end_line":789,"raw_tex":"Suppose for each $i$, there exists a rational expression\r\n$\\hat{\\lambda}_i/q_i$ for the $i$th player's Lagrange multiplier vector.\r\nSince $q_{i}(x)\\lambda_{i,j}=\\hat{\\lambda}_i(x)$\r\nand $q_i(x)\\ge0$ for all $x\\in X$,\r\nthe following holds for all KKT points\r\n\\be\r\n\\label{eq:KKTrational}\r\n\\left\\{\r\n\\begin{array}{l}\r\nq_i(x)\\nabla_{x_i} f_i(x)-\\sum\\nolimits_{j=1}^{m_i}\r\n \\hat{\\lambda}_{i,j}(x)\\nabla_{x_i} g_{i,j}(x)=0 \\, (i\\in[N]),\\\\\r\n\\hat{\\lambda}_i(x)\\perp g_i(x),g_{i,j}(x)=0 \\, (j\\in\\mathcal{E}_i,i\\in[N]),\\\\\r\ng_{i,j}(x)\\ge0,\\hat{\\lambda}_{i,j}(x)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]).\r\n\\end{array}\r\n\\right.\r\n\\ee\r\n","sha256_utf8":"165fe669c1dd168c322289f8732393f6768bd7fb0915a060202363f42bfce87e"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"正分母恢复均衡","source_units":[{"start":29287,"end":29610,"start_line":790,"end_line":795,"raw_tex":"Under some constraint qualifications,\r\nif $x$ is a GNE, then it satisfies (\\ref{eq:KKTrational}).\r\nFor convex GNEPs, if $x$ satisfies (\\ref{eq:KKTrational}) and $q_i(x)>0$,\r\nthen $x$ must be a GNE, since it satisfies\r\n(\\ref{eq:KKTwithLMalp}) with $\\lambda_{i,j}$ given by\r\n$\\lambda_{i,j}=\\hat{\\lambda}_{i,j}(x)/{q_i(x)}.$\r\n","sha256_utf8":"109cdb6448837c713b79553207a6e971d1198fa7c1dd308d0b02a8013f75ba6a"}]},{"id":"S018","semantic_type":"proposition","kind":"proposition","title":"参数表达均衡判据","source_units":[{"start":47966,"end":48537,"start_line":1272,"end_line":1289,"raw_tex":"\\begin{prop}\r\n\\label{pr:prt}\r\nFor the GNEPP given by (\\ref{eq:GNEP}),\r\nsuppose each player's optimization has a\r\nparametric expression for their Lagrange multipliers\r\nas in Definition~\\ref{def:para:LME}.\r\n\\begin{enumerate}[(i)]\r\n\r\n\\item If (\\ref{eq:KKTfeasoptprt}) is infeasible,\r\nthen the GNEP has no KKT points. If every GNE is a KKT point,\r\nthen the infeasibility of (\\ref{eq:KKTfeasoptprt})\r\nimplies nonexistence of GNEs.\r\n\r\n\\item Assume the GNEP is convex.\r\nIf $(u,w)$ is a feasible point of (\\ref{eq:KKTfeasoptprt}),\r\nthen $u$ is a GNE.\r\n\\end{enumerate}\r\n\\end{prop}","sha256_utf8":"4c444a57e65cb9b26cdcc9f87fc45b24a3cb30ec9b7dbdb95746c7c59bedc6f4"}]},{"id":"S019","semantic_type":"proof","kind":"proof","title":"参数表达判据论证","source_units":[{"start":46768,"end":47359,"start_line":1244,"end_line":1257,"raw_tex":"The optimality conditions (\\ref{eq:KKTwithLMalp})\r\ncan be equivalently expressed as\r\n\\be\r\n\\label{eq:KKTpartial}\r\n\\left\\{\r\n\\begin{array}{l}\r\n\\nabla_{x_i} f_i(x)-\\sum_{j=1}^{m_i}\\hat{\\lambda}_{i,j}(\\allx)\\nabla_{x_i} g_{i,j}(x)=0 \\, (i\\in[N]),\\\\\r\n\\hat{\\lambda}_i(\\allx)\\perp g_i(x),g_{i,j}(x)=0 \\, (j\\in\\mathcal{E}_i, i\\in[N]),\\\\\r\ng_{i,j}(x)\\ge0,\\hat{\\lambda}_{i,j}(\\allx)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]).\r\n\\end{array}\r\n\\right.\r\n\\ee\r\nFor convex GNEPs, a point $x$ is a GNE if and only if there exists\r\n$\\omega:= (\\omega_1,\\ldots,\\omega_N)$ such that $\\allx$ satisfies (\\ref{eq:KKTpartial}).\r\n","sha256_utf8":"e73d82c78b80c9742a583068130e033915acc5ac5a173d7c7d12bb69af82a69c"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"参数化KKT等价改写","source_units":[{"start":46768,"end":47201,"start_line":1244,"end_line":1255,"raw_tex":"The optimality conditions (\\ref{eq:KKTwithLMalp})\r\ncan be equivalently expressed as\r\n\\be\r\n\\label{eq:KKTpartial}\r\n\\left\\{\r\n\\begin{array}{l}\r\n\\nabla_{x_i} f_i(x)-\\sum_{j=1}^{m_i}\\hat{\\lambda}_{i,j}(\\allx)\\nabla_{x_i} g_{i,j}(x)=0 \\, (i\\in[N]),\\\\\r\n\\hat{\\lambda}_i(\\allx)\\perp g_i(x),g_{i,j}(x)=0 \\, (j\\in\\mathcal{E}_i, i\\in[N]),\\\\\r\ng_{i,j}(x)\\ge0,\\hat{\\lambda}_{i,j}(\\allx)\\ge0\\,(j\\in\\mathcal{I}_i,i\\in[N]).\r\n\\end{array}\r\n\\right.\r\n\\ee\r\n","sha256_utf8":"d3f0f53dfcb9dcff7759def8dab4a71a457517d4654b83f7e13ca62d8373d7ff"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"参数可行性联系均衡","source_units":[{"start":47201,"end":47359,"start_line":1256,"end_line":1257,"raw_tex":"For convex GNEPs, a point $x$ is a GNE if and only if there exists\r\n$\\omega:= (\\omega_1,\\ldots,\\omega_N)$ such that $\\allx$ satisfies (\\ref{eq:KKTpartial}).\r\n","sha256_utf8":"2c3fc3cae8cdcdb868af035db1a8ad7007385e25346141de8f2ab969452739b9"}]},{"id":"S022","semantic_type":"theorem","kind":"theorem","title":"均衡算法求解保证","source_units":[{"start":52274,"end":52984,"start_line":1394,"end_line":1414,"raw_tex":"\\begin{thm}\r\n\\label{tm:KKTSDPconv}\r\nFor the convex GNEPP given by (\\ref{eq:GNEP}),\r\nsuppose each Lagrange multiplier vector $\\lambda_i$ has a rational expression\r\nas in Definition~\\ref{LME:rat}\r\nor a parametric expression as in Definition~\\ref{def:para:LME}.\r\n\r\n\\begin{enumerate}[(i)]\r\n\r\n\r\n\\item If $(u,w)$ is a feasible point of (\\ref{eq:KKTfeasopt})\r\nsuch that $q_i(u)>0$ for all $i$, then $u$ is a GNE.\r\n\r\n\r\n\\item Assume every GNE is a KKT point.\r\nIf (\\ref{eq:KKTfeasopt}) is infeasible, then the GNEP has no GNEs.\r\nIf $\\Theta$ is positive definite and every\r\n$q_i(x)>0$ for all feasible $x$ of (\\ref{eq:KKTfeasopt}),\r\nthen Algorithm~\\ref{ag:KKTSDP} will find a GNE if it exists.\r\n\\end{enumerate}\r\n\\end{thm}","sha256_utf8":"75c4c80b82db94282c7462e43824b8fa3f33cbced350dbcba7eee81699b34ba6"}]},{"id":"S023","semantic_type":"proof","kind":"proof","title":"均衡求解保证证明","source_units":[{"start":52986,"end":53502,"start_line":1415,"end_line":1426,"raw_tex":"\\begin{proof}\r\n(i) This is directly implied by Propositions~\\ref{pr:rtn} and \\ref{pr:prt}.\r\n\r\n(ii) If (\\ref{eq:KKTfeasopt}) is infeasible,\r\nthen there is no GNE, because every GNE is assumed to be a KKT point\r\nand it must be feasible for (\\ref{eq:KKTfeasopt}).\r\nNext, assume (\\ref{eq:KKTfeasopt}) is feasible.\r\nSince $\\Theta$ is positive definite,\r\nthe optimization (\\ref{eq:KKTfeasopt}) has a minimizer,\r\nsay, $(u,w)$. By the given assumption,  we have $q_i(u)>0$ for all $i$.\r\nSo $u$ is a GNE, by (i).\r\n\\end{proof}","sha256_utf8":"10dcb6822ce4f62bceeefa59fa0ede919ef6db4bb76fa3b18e19c2e1665fe1b4"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"应用两类表达判据","source_units":[{"start":53001,"end":53078,"start_line":1416,"end_line":1416,"raw_tex":"(i) This is directly implied by Propositions~\\ref{pr:rtn} and \\ref{pr:prt}.\r\n","sha256_utf8":"f70f458c496e4d803fb1fa568ff0f00b03968288a48632918fd10bfe23a92161"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"不可行排除均衡","source_units":[{"start":53080,"end":53248,"start_line":1418,"end_line":1420,"raw_tex":"(ii) If (\\ref{eq:KKTfeasopt}) is infeasible,\r\nthen there is no GNE, because every GNE is assumed to be a KKT point\r\nand it must be feasible for (\\ref{eq:KKTfeasopt}).\r\n","sha256_utf8":"f1b6746f61cfe16a1513395293355c1cd59484f6f2e5f3120b17d5fc0dc920e6"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"正定目标取得极小值","source_units":[{"start":53248,"end":53465,"start_line":1421,"end_line":1424,"raw_tex":"Next, assume (\\ref{eq:KKTfeasopt}) is feasible.\r\nSince $\\Theta$ is positive definite,\r\nthe optimization (\\ref{eq:KKTfeasopt}) has a minimizer,\r\nsay, $(u,w)$. By the given assumption,  we have $q_i(u)>0$ for all $i$.\r\n","sha256_utf8":"23f0f6aa80ee6d422837b638f74ab1dd09d903bf7e440898b72bd15f00272e49"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"极小点给出均衡","source_units":[{"start":53465,"end":53491,"start_line":1425,"end_line":1425,"raw_tex":"So $u$ is a GNE, by (i).\r\n","sha256_utf8":"0b119dfb215c1de502e3f347a4cbf5e162d05cd0bba3edca5156f172676769cb"}]},{"id":"S028","semantic_type":"external_fact","kind":"external_fact","title":"凸优化KKT最优性","source_units":[],"declared_external":true,"declaration_text":["Under suitable constraint qualifications, a minimizer satisfies the KKT conditions. For a differentiable convex objective with affine equality constraints and concave inequality functions, every feasible KKT point is a global minimizer. Applied player by player, a KKT point of a convex GNEP is a GNE."]},{"id":"S029","semantic_type":"external_fact","kind":"external_fact","title":"强制目标的最小值存在","source_units":[],"declared_external":true,"declaration_text":["A continuous coercive function attains its minimum on every nonempty closed subset of finite-dimensional Euclidean space. Positive definiteness makes the quadratic objective coercive, and polynomial equalities and non-strict inequalities give a closed feasible set."]},{"id":"S030","semantic_type":"assumption","kind":"assumption","title":"凸性与乘子表达假设","source_units":[{"start":52310,"end":52534,"start_line":1396,"end_line":1399,"raw_tex":"For the convex GNEPP given by (\\ref{eq:GNEP}),\r\nsuppose each Lagrange multiplier vector $\\lambda_i$ has a rational expression\r\nas in Definition~\\ref{LME:rat}\r\nor a parametric expression as in Definition~\\ref{def:para:LME}.\r\n","sha256_utf8":"49db96b32935968f5c1a0354d6d2fae76c0cdbd1f0f1b4bd181cb3fc8e70d02e"}]},{"id":"S031","semantic_type":"assumption","kind":"assumption","title":"均衡满足KKT假设","source_units":[{"start":52685,"end":52725,"start_line":1408,"end_line":1408,"raw_tex":"\\item Assume every GNE is a KKT point.\r\n","sha256_utf8":"67341f2933423cc789b3f711356928b991b830333bd7573022b41f8db17337e7"}]},{"id":"S032","semantic_type":"assumption","kind":"assumption","title":"目标正定与分母正性","source_units":[{"start":52793,"end":52896,"start_line":1410,"end_line":1411,"raw_tex":"If $\\Theta$ is positive definite and every\r\n$q_i(x)>0$ for all feasible $x$ of (\\ref{eq:KKTfeasopt}),\r\n","sha256_utf8":"cb8dc9c279b56b05b8d349338e99b9a2551566cbb4e593007170fc344dba74bf"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S022","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S001","target_id":"S003","kind":"supports"},{"source_id":"S001","target_id":"S005","kind":"supports"},{"source_id":"S002","target_id":"S006","kind":"supports"},{"source_id":"S002","target_id":"S008","kind":"supports"},{"source_id":"S002","target_id":"S009","kind":"supports"},{"source_id":"S002","target_id":"S011","kind":"supports"},{"source_id":"S002","target_id":"S012","kind":"supports"},{"source_id":"S003","target_id":"S017","kind":"supports"},{"source_id":"S003","target_id":"S021","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S008","kind":"supports"},{"source_id":"S004","target_id":"S009","kind":"supports"},{"source_id":"S004","target_id":"S010","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S016","kind":"supports"},{"source_id":"S005","target_id":"S020","kind":"supports"},{"source_id":"S006","target_id":"S016","kind":"supports"},{"source_id":"S007","target_id":"S020","kind":"supports"},{"source_id":"S008","target_id":"S015","kind":"supports"},{"source_id":"S009","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S009","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S013","kind":"supports"},{"source_id":"S010","target_id":"S024","kind":"supports"},{"source_id":"S010","target_id":"S025","kind":"supports"},{"source_id":"S011","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S011","target_id":"S024","kind":"supports"},{"source_id":"S011","target_id":"S025","kind":"supports"},{"source_id":"S011","target_id":"S026","kind":"supports"},{"source_id":"S013","target_id":"S027","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S014","target_id":"S024","kind":"supports"},{"source_id":"S018","target_id":"S024","kind":"supports"},{"source_id":"S014","target_id":"S025","kind":"supports"},{"source_id":"S018","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S023","kind":"supports"},{"source_id":"S026","target_id":"S023","kind":"supports"},{"source_id":"S027","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S028","target_id":"S017","kind":"supports"},{"source_id":"S028","target_id":"S021","kind":"supports"},{"source_id":"S029","target_id":"S026","kind":"supports"},{"source_id":"S030","target_id":"S024","kind":"supports"},{"source_id":"S030","target_id":"S025","kind":"supports"},{"source_id":"S031","target_id":"S025","kind":"supports"},{"source_id":"S032","target_id":"S026","kind":"supports"}]}}
{"record_id":"2102.00135v6","paper":{"title":"Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classes","authors":["Guanghui Lan"],"arxiv_id":"2102.00135","version":"v6","journal":"Mathematical Programming","doi":"10.1007/s10107-022-01816-5","abstract_url":"https://arxiv.org/abs/2102.00135v6","pdf_url":"https://arxiv.org/pdf/2102.00135v6"},"context":{"raw_tex":"Consider the finite Markov decision process\n$M = (\\cS, \\cA, \\cP, c, \\gamma)$, where $\\cS$\nis a finite state space, $\\cA$ is a finite action space,\n$P: \\cS \\times \\cS \\times \\cA \\to \\bbr$ is transition \nmodel, $c: \\cS \\times \\cA \\to \\bbr$ is the cost function,\nand $\\gamma \\in (0,1)$ is the discount factor.\n%Observe that here we use the cost function \n%$c(s_t, a_t)$, rather than the usual reward function $r(s_t, a_t)$,\n%in order to facilitate the use of certain terminology (e.g., mirror descent)\n%from the optimization literature.\nA policy $\\pi: \\cA \\times \\cS \\to \\bbr$ determines\nthe probability of selecting a particular action at a given state.\n\n\nFor a given policy $\\pi$,\nwe measure its performance by the action-value\nfunction ($Q$-function) $Q^\\pi: \\cS \\times \\cA \\to \\bbr$\ndefined as\n\\begin{align}\nQ^\\pi(s,a) &:=\n\\bbe\\left[\\tsum_{t=0}^\\infty \\gamma^t [c(s_t, a_t) + h^\\pi(s_t)] \\right. \\nn \\\\\n&\\quad \\quad\\quad\\quad \\left. \\mid s_0 = s, a_0 = a, a_t \\sim \\pi(\\cdot | s_t), s_{t+1} \\sim \\cP(\\cdot | s_t, a_t)\\right]. \\label{eq:def_Q_function}\n\\end{align}\n\nHere $h^\\pi$ is a closed convex function w.r.t. the policy $\\pi$, i.e., there exist some\n$ \\mu \\ge 0$ s.t.\n\\beq \\label{eq:convex_obj}\nh^\\pi(s) - [h^{\\pi'}(s) + \\langle  (h')^{\\pi'}(s, \\cdot), \\pi(\\cdot|s ) -  \\pi'(\\cdot|s )\\rangle ] \\ge \\mu D_{\\pi'}^\\pi(s),\n\\eeq\n\nwhere $\\langle \\cdot, \\cdot \\rangle$ denotes\nthe inner product over the action space $\\cA$, $(h')^{\\pi'}(s, \\cdot)$ \ndenotes a subgradient of $h(s)$ at $\\pi'$, and $D_{\\pi'}^\\pi(s)$ is the \nBregman's distance or Kullback–Leibler (KL) divergence  between $\\pi$ and $\\pi'$ (see Subsection~\\ref{sec:notations_PMD} for\nmore discussion).\n\n\nThroughout this paper, we say that $h^\\pi$ is a strongly convex regularizer if $\\mu > 0$.\nOtherwise, we call $h^\\pi$ a general convex regularizer. Clearly\nthe latter class of problems covers\nthe regular case with $h^\\pi = 0$.\n\n\nWe define the state-value function\n$V^\\pi: \\cS \\to \\bbr$ associated with $\\pi$\nas\n\\begin{align} \nV^\\pi(s) &:=\n\\bbe\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [c(s_t, a_t) + h^\\pi(s_t)] \\right.\\nn \\\\\n & \\quad \\quad \\quad \\quad \\left. \\mid\ns_0 = s, a_t \\sim \\pi(\\cdot | s_t),\ns_{t+1} \\sim \\cP(\\cdot | s_t, a_t)\n\\right].\\label{eq:def_V_function}\n\\end{align}\n\nIt can be easily seen \nfrom the definitions of $Q^\\pi$ and $V^\\pi$ that\n\\begin{align}\nV^\\pi(s) &= \\tsum_{a \\in \\cA} \\pi( a | s) Q^\\pi(s, a)  = \\langle Q^\\pi(s, \\cdot),  \\pi( \\cdot | s) \\rangle, \\label{eq:QV1}  \\\\\nQ^\\pi(s, a) &= c(s, a) + h^\\pi(s) + \\gamma \\tsum_{s' \\in \\cS} \\cP(s'| s, a) V^\\pi(s'). \\label{eq:QV2}\n\\end{align}\n\nThe main objective in RL is to find an optimal policy $\\pi^*: \\cS \\times \\cA \\to \\bbr$ s.t.\n\\beq \\label{eq:opt_objective}\nV^{\\pi^*}(s) \\le V^\\pi(s), \\forall \\pi(\\cdot | s) \\in \\Delta_{|\\cA|}, \\forall s \\in \\cS.\n\\eeq\n\nfor any $s \\in \\cS$. Here $\\Delta_{|\\cA|}$ denotes the \nsimplex constraint given by\n\\beq \\label{eq:def_simplex}\n \\Delta_{|\\cA|} := \\{ p \\in \\bbr^{|\\cA|} | \\tsum_{i =1}^{|\\cA|} p_i = 1, p_i \\ge 0\\}, \\forall s \\in \\cS.\n\\eeq\n\nBy examining Bellman's optimality condition\nfor dynamic programming (\\cite{BellmanDreyfus1959} and Chapter 6 of \\cite{PutermanBook1994}), we can show the existence\nof a policy $\\pi^*$ which satisfies \\eqnok{eq:opt_objective} simultaneously \nfor all $s \\in \\cS$. Hence, we can formulate \\eqnok{eq:opt_objective}\nas an optimization problem with a single objective by taking the weighted sum of\n$V^\\pi$ over $s$ (with weights $\\rho_s >0$ and $\\tsum_{s \\in S} \\rho_s = 1$):\n\\beq \\label{eq:opt_objective1}\n\\begin{array}{ll}\n\\min_\\pi &  \\bbe_{s \\sim \\rho} [V^{\\pi}(s)] \\\\\n\\mbox{s.t.} & \\pi(\\cdot | s) \\in \\Delta_{|\\cA|}, \\forall s \\in \\cS.\n\\end{array}\n\\eeq\n\nWhile the weights $\\rho$ can be arbitrarily chosen, a reasonable selection\nof $\\rho$ would be the stationary state distribution\ninduced by the optimal policy $\\pi^*$, denoted by $\\nu^* \\equiv \\nu(\\pi^*)$. As such,\nproblem~\\eqnok{eq:opt_objective1} reduces to \n\\beq \\label{eq:MDP_OPT}\n\\begin{array}{ll}\n\\min_\\pi & \\left\\{ f(\\pi) := \\bbe_{s \\sim \\nu^*} [V^{\\pi}(s)] \\right\\}\\\\\n\\mbox{s.t.} & \\pi(\\cdot | s) \\in \\Delta_{|\\cA|}, \\forall s \\in \\cS.\n\\end{array}\n\\eeq\n\nFor any two points $\\pi(\\cdot|s), \\pi'(\\cdot|s) \\in \\Delta_{|\\cA|}$,\nwe measure their Kullback–Leibler (KL)  divergence by\n\\[\n\\KL(\\pi(\\cdot|s) \\parallel \\pi'(\\cdot|s)) = \\tsum_{a \\in \\cA} \\pi(a|s) \\log \\tfrac{\\pi(a|s)}{\\pi'(a|s)}.\n\\]\n\nObserve that the KL divergence can be viewed as\nis a special instance of the \nBregman's distance (or prox-function)\nwidely used in the optimization literature. \nLet the distance generating function $\\omega(\\pi(\\cdot|s)) := \\tsum_{a \\in \\cA} \\pi(a|s) \\log \\pi(a|s)$~\\footnote{It is worth noting that\nwe do not enforce $\\pi(a|s) > 0$ when defining $\\omega(\\pi(\\cdot|s))$ as all the search points generated by our algorithms\nwill satisfy this assumption.}. The Bregman's distance\nassociated with $\\omega$ is given by\n\\begin{align}\nD_{\\pi'}^\\pi(s) &:= \\omega(\\pi(\\cdot|s)) - [\\omega(\\pi'(\\cdot|s)) + \\langle \\nabla \\omega(\\pi'(\\cdot|s)), \\pi(\\cdot|s) - \\pi'(\\cdot|s)\\rangle ] \\nn\\\\\n&= \\tsum_{a \\in \\cA} \\left[ \\pi(a|s) \\log \\pi(a|s) - \\pi'(a|s) \\log \\pi'(a|s) \\right. \\nn\\\\\n& \\quad \\quad \\quad \\quad \\quad \\left. - (1 + \\log \\pi'(a|s)) (\\pi(a|s)- \\pi'(a|s))\\right] \\nn\\\\\n&= \\tsum_{a \\in \\cA} \\pi(a|s) \\log \\tfrac{\\pi(a|s)}{\\pi'(a|s)},\n\\end{align}\n\nwhere the last equation follows from the fact that $\\tsum_{a \\in \\cA}(\\pi(a|s)- \\pi'(a|s)) = 0$.\nTherefore, we will use the KL divergence $\\KL(\\pi(\\cdot|s) \\parallel \\pi'(\\cdot|s))$\nand Bregman's distance $D_{\\pi'}^\\pi(s)$ interchangeably throughout this paper. \n\n\nFor simplicity, we assume for now that $h^\\pi$ is differentiable and will relax this assumption later. \nFor a given policy $\\pi$, we define the discounted state visitation distribution by\n\\beq \\label{eq:def_visitation}\nd_{s_0}^\\pi(s) := (1 - \\gamma) \\tsum_{t=0}^\\infty \\gamma^t \\Pr^\\pi(s_t = s | s_0),\n\\eeq\n\nwhere $\\Pr^\\pi(s_t = s | s_0)$ denotes the state visitation probability of $s_t =s$ after we follow\nthe policy $\\pi$ starting at state $s_0$. Let $\\cP^\\pi$ denote the transition probability\nmatrix associated with policy $\\pi$, i.e., $\\cP^\\pi(i,j) = \\tsum_{a \\in \\cA} \\pi(a | i) \\cP(j| i, a)$,\nand  $e_i$ be the $i$-th unit vector.\nThen $\\Pr^\\pi(s_t = s | s_0) = e_{s_0}^T(\\cP^\\pi)^t e_s$ and\n\\beq \\label{eq:def_visitation1}\nd_{s_0}^\\pi(s) = (1 - \\gamma) \\tsum_{t=0}^\\infty \\gamma^t e_{s_0}^T(\\cP^\\pi)^t e_s.\n\\eeq\n\n\\beq\\label{eq:VI_MDP}\n\\bbe_{s \\sim \\nu^*} \\left[ \\langle Q^{\\pi^*}(s,\\cdot), \\pi(\\cdot |s) - \\pi^*(\\cdot |s)\\rangle + h^\\pi(s) - h^{\\pi^*}(s) \\right]  \\ge 0.\n\\eeq\n\n For the sake of simplicity, we will assume throughout the paper that\n\n\n\\beq \\label{eq:initial_p0}\n \\pi_0(a | s) = 1/|\\cA|, \\ \\forall a \\in \\cA, \\forall s \\in \\cS.\n \\eeq\n\nIn this case, we have \n\\beq \\label{eq:boundnessofdomain}\nD_{\\pi_0}^\\pi(s) = \\tsum_{a \\in \\cA} \\pi(a | s) \\log \\pi(a| s) + \\log |\\cA| \\le \\log |\\cA|, \\ \\forall \\pi(\\cdot|s) \\in \\Delta_{|\\cA|}.\n\\eeq\n\n\\begin{algorithm}[H]\n\\caption{The policy mirror descent (PMD) method}\n\\begin{algorithmic}\n\\State {\\bf Input:} initial points $\\pi_0$ and stepsizes $\\eta_k\\ge 0$.\n\\For {$k =0,1,\\ldots,$}\n%\\State  \n\\beq \\label{eq:PMD_step}\n\\pi_{k+1}(\\cdot|s) = \\argmin_{p(\\cdot|s) \\in \\Delta_{|\\cA|}} \\left\\{ \\eta_k[ \\langle Q^{\\pi_{k}}(s, \\cdot), p(\\cdot|s) \\rangle + h^{p}(s)] + D_{\\pi_k}^p(s)\\right\\},\n\\forall s \\in \\cS.\n\\eeq\n\\EndFor\n\\end{algorithmic} \\label{basic_pmd}\n\\end{algorithm}\n\n\\beq \\label{eq:def_distance_measure}\n\\cD (\\pi_k, \\pi^*) := \\bbe_{s\\sim \\nu^*} [D_{\\pi_{k}}^{\\pi^*}(s)].\n\\eeq\n\nIn this subsection, we assume that for a given policy $\\pi_k$,\nthere exists a stochastic estimator $\\cQ^{\\pi_k,\\xi_k}$ s.t.\n\\begin{align}\n \\bbe_{\\xi_k} [\\cQ^{\\pi_k,\\xi_k}] &= \\bar \\cQ^{\\pi_k}, \\label{eq:expectation_ass} \\\\\n\\bbe_{\\xi_k}[\\|\\cQ^{\\pi_k,\\xi_k} - Q^{\\pi_k} \\|_\\infty^2] &\\le \\sigma_k^2, \\label{eq:variance_ass}\\\\\n\\|\\bar \\cQ^{\\pi_k} - Q^{\\pi_k}\\|_\\infty &\\le \\varsigma_k, \\label{eq:bias_ass}\n\\end{align}\n\nfor some $\\sigma_k\\ge$ and $\\varsigma_k \\ge 0$,\nwhere $\\xi_k$ denotes the random vector used to generate the stochastic\nestimator $\\cQ^{\\pi_k,\\xi_k}$.  Clearly, if $\\sigma_k = 0$, then\nwe have exact information about $Q^{\\pi_k}$. \n\n\nThe stochastic policy mirror descent (SPMD)\nis obtained by replacing $Q^{\\pi_k}$ in \\eqnok{eq:PMD_step} \nwith its stochastic estimator $\\cQ^{\\pi_k,\\xi_k}$, i.e.,\n\\beq \\label{eq:SPMD_step}\n\\pi_{k+1}(\\cdot|s) = \\argmin_{p(\\cdot|s) \\in \\Delta_{|\\cA|}} \\left\\{ \\Phi_k(p):=\\eta_k [\\langle \\cQ^{\\pi_{k},\\xi_k}(s, \\cdot), p(\\cdot|s) \\rangle + h^p(s)]+ D_{\\pi_k}^p(s)\\right\\}.\n\\eeq\n\nIn the sequel, we denote $\\xi_{\\lceil k \\rceil}$ the sequence of random vectors $\\xi_0, \\ldots, \\xi_k$ and define\n\n\n\\beq \\label{eq:def_SPMD_delta}\n\\delta_k := \\cQ^{\\pi_k,\\xi_k} - Q^{\\pi_k}.\n\\eeq\n\nWe are now ready to establish the convergence rate of the SPMD method.  \nWe start with the case when $\\mu > 0$ and state a constant stepsize rule which requires both\n$\\varsigma_k$ and $\\sigma_k$, $k \\ge 0$, to be small enough to guarantee the convergence of \nthe SPMD method. \n","units":[{"start":4725,"end":5377,"start_line":150,"end_line":161,"raw_tex":"Consider the finite Markov decision process\n$M = (\\cS, \\cA, \\cP, c, \\gamma)$, where $\\cS$\nis a finite state space, $\\cA$ is a finite action space,\n$P: \\cS \\times \\cS \\times \\cA \\to \\bbr$ is transition \nmodel, $c: \\cS \\times \\cA \\to \\bbr$ is the cost function,\nand $\\gamma \\in (0,1)$ is the discount factor.\n%Observe that here we use the cost function \n%$c(s_t, a_t)$, rather than the usual reward function $r(s_t, a_t)$,\n%in order to facilitate the use of certain terminology (e.g., mirror descent)\n%from the optimization literature.\nA policy $\\pi: \\cA \\times \\cS \\to \\bbr$ determines\nthe probability of selecting a particular action at a given state.\n","sha256_utf8":"816aa9ab76dd52f2ea5c87c381bda019b68c206e8adf153e37c85e1b62f786d7"},{"start":5378,"end":5788,"start_line":163,"end_line":171,"raw_tex":"For a given policy $\\pi$,\nwe measure its performance by the action-value\nfunction ($Q$-function) $Q^\\pi: \\cS \\times \\cA \\to \\bbr$\ndefined as\n\\begin{align}\nQ^\\pi(s,a) &:=\n\\bbe\\left[\\tsum_{t=0}^\\infty \\gamma^t [c(s_t, a_t) + h^\\pi(s_t)] \\right. \\nn \\\\\n&\\quad \\quad\\quad\\quad \\left. \\mid s_0 = s, a_0 = a, a_t \\sim \\pi(\\cdot | s_t), s_{t+1} \\sim \\cP(\\cdot | s_t, a_t)\\right]. \\label{eq:def_Q_function}\n\\end{align}","sha256_utf8":"d92c254a530ed4654d1eab9dc59c2909ce1881ce6c82fd4a8aaa2cfaa5b11bdd"},{"start":5789,"end":6051,"start_line":172,"end_line":176,"raw_tex":"Here $h^\\pi$ is a closed convex function w.r.t. the policy $\\pi$, i.e., there exist some\n$ \\mu \\ge 0$ s.t.\n\\beq \\label{eq:convex_obj}\nh^\\pi(s) - [h^{\\pi'}(s) + \\langle  (h')^{\\pi'}(s, \\cdot), \\pi(\\cdot|s ) -  \\pi'(\\cdot|s )\\rangle ] \\ge \\mu D_{\\pi'}^\\pi(s),\n\\eeq","sha256_utf8":"bf95dd804f3e9d08c9120278c62d2747bfe8e85e438b90c504b85f0180c21a3c"},{"start":6052,"end":6385,"start_line":177,"end_line":181,"raw_tex":"where $\\langle \\cdot, \\cdot \\rangle$ denotes\nthe inner product over the action space $\\cA$, $(h')^{\\pi'}(s, \\cdot)$ \ndenotes a subgradient of $h(s)$ at $\\pi'$, and $D_{\\pi'}^\\pi(s)$ is the \nBregman's distance or Kullback–Leibler (KL) divergence  between $\\pi$ and $\\pi'$ (see Subsection~\\ref{sec:notations_PMD} for\nmore discussion).\n","sha256_utf8":"9bb43de8248d1630766ec823f8e12cd34d443af75a4299b07facd908bb00dce6"},{"start":7135,"end":7361,"start_line":193,"end_line":196,"raw_tex":"Throughout this paper, we say that $h^\\pi$ is a strongly convex regularizer if $\\mu > 0$.\nOtherwise, we call $h^\\pi$ a general convex regularizer. Clearly\nthe latter class of problems covers\nthe regular case with $h^\\pi = 0$.\n","sha256_utf8":"c6dc768e473fb61b00f1673b12efd895e7919ff8ad636c06e2f44b5691f5f1ea"},{"start":7362,"end":7710,"start_line":198,"end_line":210,"raw_tex":"We define the state-value function\n$V^\\pi: \\cS \\to \\bbr$ associated with $\\pi$\nas\n\\begin{align} \nV^\\pi(s) &:=\n\\bbe\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [c(s_t, a_t) + h^\\pi(s_t)] \\right.\\nn \\\\\n & \\quad \\quad \\quad \\quad \\left. \\mid\ns_0 = s, a_t \\sim \\pi(\\cdot | s_t),\ns_{t+1} \\sim \\cP(\\cdot | s_t, a_t)\n\\right].\\label{eq:def_V_function}\n\\end{align}","sha256_utf8":"316b4e1cc0268f9ef48ab3b8b1961337d1d36c832bad74bbbe8a664fcad7ee10"},{"start":7711,"end":8037,"start_line":211,"end_line":216,"raw_tex":"It can be easily seen \nfrom the definitions of $Q^\\pi$ and $V^\\pi$ that\n\\begin{align}\nV^\\pi(s) &= \\tsum_{a \\in \\cA} \\pi( a | s) Q^\\pi(s, a)  = \\langle Q^\\pi(s, \\cdot),  \\pi( \\cdot | s) \\rangle, \\label{eq:QV1}  \\\\\nQ^\\pi(s, a) &= c(s, a) + h^\\pi(s) + \\gamma \\tsum_{s' \\in \\cS} \\cP(s'| s, a) V^\\pi(s'). \\label{eq:QV2}\n\\end{align}","sha256_utf8":"b938b67dc6ab0a44fab1d550f120658912db3ff02f93e85d48421420e1206287"},{"start":8038,"end":8253,"start_line":217,"end_line":220,"raw_tex":"The main objective in RL is to find an optimal policy $\\pi^*: \\cS \\times \\cA \\to \\bbr$ s.t.\n\\beq \\label{eq:opt_objective}\nV^{\\pi^*}(s) \\le V^\\pi(s), \\forall \\pi(\\cdot | s) \\in \\Delta_{|\\cA|}, \\forall s \\in \\cS.\n\\eeq","sha256_utf8":"30d5770e26894e9ec91caa2ab56a10e1c5e397da3edb83711c20452c36c35875"},{"start":8255,"end":8476,"start_line":221,"end_line":225,"raw_tex":"for any $s \\in \\cS$. Here $\\Delta_{|\\cA|}$ denotes the \nsimplex constraint given by\n\\beq \\label{eq:def_simplex}\n \\Delta_{|\\cA|} := \\{ p \\in \\bbr^{|\\cA|} | \\tsum_{i =1}^{|\\cA|} p_i = 1, p_i \\ge 0\\}, \\forall s \\in \\cS.\n\\eeq","sha256_utf8":"2a7f6e8d5c3d4ae4c080ec4355e1940a89f044408ae571ae9fbc1583fe6c9d0f"},{"start":8478,"end":9128,"start_line":227,"end_line":238,"raw_tex":"By examining Bellman's optimality condition\nfor dynamic programming (\\cite{BellmanDreyfus1959} and Chapter 6 of \\cite{PutermanBook1994}), we can show the existence\nof a policy $\\pi^*$ which satisfies \\eqnok{eq:opt_objective} simultaneously \nfor all $s \\in \\cS$. Hence, we can formulate \\eqnok{eq:opt_objective}\nas an optimization problem with a single objective by taking the weighted sum of\n$V^\\pi$ over $s$ (with weights $\\rho_s >0$ and $\\tsum_{s \\in S} \\rho_s = 1$):\n\\beq \\label{eq:opt_objective1}\n\\begin{array}{ll}\n\\min_\\pi &  \\bbe_{s \\sim \\rho} [V^{\\pi}(s)] \\\\\n\\mbox{s.t.} & \\pi(\\cdot | s) \\in \\Delta_{|\\cA|}, \\forall s \\in \\cS.\n\\end{array}\n\\eeq","sha256_utf8":"bce9937c0896c2144f15675ec5d854e721792b7c7a951c5e9b2a04123ef53396"},{"start":9129,"end":9588,"start_line":239,"end_line":248,"raw_tex":"While the weights $\\rho$ can be arbitrarily chosen, a reasonable selection\nof $\\rho$ would be the stationary state distribution\ninduced by the optimal policy $\\pi^*$, denoted by $\\nu^* \\equiv \\nu(\\pi^*)$. As such,\nproblem~\\eqnok{eq:opt_objective1} reduces to \n\\beq \\label{eq:MDP_OPT}\n\\begin{array}{ll}\n\\min_\\pi & \\left\\{ f(\\pi) := \\bbe_{s \\sim \\nu^*} [V^{\\pi}(s)] \\right\\}\\\\\n\\mbox{s.t.} & \\pi(\\cdot | s) \\in \\Delta_{|\\cA|}, \\forall s \\in \\cS.\n\\end{array}\n\\eeq","sha256_utf8":"7071125284363431f9ececeef1ff3383ddba13ce15449375b02dfcbd1c8bae6c"},{"start":17315,"end":17548,"start_line":366,"end_line":370,"raw_tex":"For any two points $\\pi(\\cdot|s), \\pi'(\\cdot|s) \\in \\Delta_{|\\cA|}$,\nwe measure their Kullback–Leibler (KL)  divergence by\n\\[\n\\KL(\\pi(\\cdot|s) \\parallel \\pi'(\\cdot|s)) = \\tsum_{a \\in \\cA} \\pi(a|s) \\log \\tfrac{\\pi(a|s)}{\\pi'(a|s)}.\n\\]","sha256_utf8":"681ec8113666fadc87d5290c575a4d7077c85fa5066a12550b3c5c001a09b149"},{"start":17549,"end":18491,"start_line":371,"end_line":384,"raw_tex":"Observe that the KL divergence can be viewed as\nis a special instance of the \nBregman's distance (or prox-function)\nwidely used in the optimization literature. \nLet the distance generating function $\\omega(\\pi(\\cdot|s)) := \\tsum_{a \\in \\cA} \\pi(a|s) \\log \\pi(a|s)$~\\footnote{It is worth noting that\nwe do not enforce $\\pi(a|s) > 0$ when defining $\\omega(\\pi(\\cdot|s))$ as all the search points generated by our algorithms\nwill satisfy this assumption.}. The Bregman's distance\nassociated with $\\omega$ is given by\n\\begin{align}\nD_{\\pi'}^\\pi(s) &:= \\omega(\\pi(\\cdot|s)) - [\\omega(\\pi'(\\cdot|s)) + \\langle \\nabla \\omega(\\pi'(\\cdot|s)), \\pi(\\cdot|s) - \\pi'(\\cdot|s)\\rangle ] \\nn\\\\\n&= \\tsum_{a \\in \\cA} \\left[ \\pi(a|s) \\log \\pi(a|s) - \\pi'(a|s) \\log \\pi'(a|s) \\right. \\nn\\\\\n& \\quad \\quad \\quad \\quad \\quad \\left. - (1 + \\log \\pi'(a|s)) (\\pi(a|s)- \\pi'(a|s))\\right] \\nn\\\\\n&= \\tsum_{a \\in \\cA} \\pi(a|s) \\log \\tfrac{\\pi(a|s)}{\\pi'(a|s)},\n\\end{align}","sha256_utf8":"38c252f9b6b903f2571d87676fab367466b74909134889a67711ba43089ebb52"},{"start":18492,"end":18755,"start_line":385,"end_line":387,"raw_tex":"where the last equation follows from the fact that $\\tsum_{a \\in \\cA}(\\pi(a|s)- \\pi'(a|s)) = 0$.\nTherefore, we will use the KL divergence $\\KL(\\pi(\\cdot|s) \\parallel \\pi'(\\cdot|s))$\nand Bregman's distance $D_{\\pi'}^\\pi(s)$ interchangeably throughout this paper. \n","sha256_utf8":"d54eac2545890b4358f9cbc5f33de64489ad5e611e0a40986c12961500f642c5"},{"start":19776,"end":20082,"start_line":403,"end_line":407,"raw_tex":"For simplicity, we assume for now that $h^\\pi$ is differentiable and will relax this assumption later. \nFor a given policy $\\pi$, we define the discounted state visitation distribution by\n\\beq \\label{eq:def_visitation}\nd_{s_0}^\\pi(s) := (1 - \\gamma) \\tsum_{t=0}^\\infty \\gamma^t \\Pr^\\pi(s_t = s | s_0),\n\\eeq","sha256_utf8":"d25fae81d85549fbad3fc69109a00d8d6253809f22a1f4f8fb0ffa329aa7235c"},{"start":20083,"end":20595,"start_line":408,"end_line":415,"raw_tex":"where $\\Pr^\\pi(s_t = s | s_0)$ denotes the state visitation probability of $s_t =s$ after we follow\nthe policy $\\pi$ starting at state $s_0$. Let $\\cP^\\pi$ denote the transition probability\nmatrix associated with policy $\\pi$, i.e., $\\cP^\\pi(i,j) = \\tsum_{a \\in \\cA} \\pi(a | i) \\cP(j| i, a)$,\nand  $e_i$ be the $i$-th unit vector.\nThen $\\Pr^\\pi(s_t = s | s_0) = e_{s_0}^T(\\cP^\\pi)^t e_s$ and\n\\beq \\label{eq:def_visitation1}\nd_{s_0}^\\pi(s) = (1 - \\gamma) \\tsum_{t=0}^\\infty \\gamma^t e_{s_0}^T(\\cP^\\pi)^t e_s.\n\\eeq","sha256_utf8":"e4d4e48e9b36dfa52fcbb679ce070e50036bf6f7d490f2b0684b073033f99482"},{"start":24420,"end":24582,"start_line":499,"end_line":501,"raw_tex":"\\beq\\label{eq:VI_MDP}\n\\bbe_{s \\sim \\nu^*} \\left[ \\langle Q^{\\pi^*}(s,\\cdot), \\pi(\\cdot |s) - \\pi^*(\\cdot |s)\\rangle + h^\\pi(s) - h^{\\pi^*}(s) \\right]  \\ge 0.\n\\eeq","sha256_utf8":"0dc37a5947bd5c1526db2ac47e5cff34094014c9c30fecc442d5bae951df6c91"},{"start":32383,"end":32453,"start_line":684,"end_line":684,"raw_tex":" For the sake of simplicity, we will assume throughout the paper that\n","sha256_utf8":"af09c8d9e3e72fcb5e7d8b815cb8e9219cd672ef5864934558c8d59ef1ed30ad"},{"start":32454,"end":32551,"start_line":685,"end_line":687,"raw_tex":"\\beq \\label{eq:initial_p0}\n \\pi_0(a | s) = 1/|\\cA|, \\ \\forall a \\in \\cA, \\forall s \\in \\cS.\n \\eeq","sha256_utf8":"6b8f9b0ad89861657d22329c87c536fac874845661247b4992465a9ea6aea7bf"},{"start":32552,"end":32748,"start_line":688,"end_line":691,"raw_tex":"In this case, we have \n\\beq \\label{eq:boundnessofdomain}\nD_{\\pi_0}^\\pi(s) = \\tsum_{a \\in \\cA} \\pi(a | s) \\log \\pi(a| s) + \\log |\\cA| \\le \\log |\\cA|, \\ \\forall \\pi(\\cdot|s) \\in \\Delta_{|\\cA|}.\n\\eeq","sha256_utf8":"aa0bd8f2b306e444d323cb6d751a6d3f2ba5a97e64f7a4a0c444fdf5a55af269"},{"start":33046,"end":33515,"start_line":697,"end_line":709,"raw_tex":"\\begin{algorithm}[H]\n\\caption{The policy mirror descent (PMD) method}\n\\begin{algorithmic}\n\\State {\\bf Input:} initial points $\\pi_0$ and stepsizes $\\eta_k\\ge 0$.\n\\For {$k =0,1,\\ldots,$}\n%\\State  \n\\beq \\label{eq:PMD_step}\n\\pi_{k+1}(\\cdot|s) = \\argmin_{p(\\cdot|s) \\in \\Delta_{|\\cA|}} \\left\\{ \\eta_k[ \\langle Q^{\\pi_{k}}(s, \\cdot), p(\\cdot|s) \\rangle + h^{p}(s)] + D_{\\pi_k}^p(s)\\right\\},\n\\forall s \\in \\cS.\n\\eeq\n\\EndFor\n\\end{algorithmic} \\label{basic_pmd}\n\\end{algorithm}","sha256_utf8":"d36d2cfec2de4a4c7ac5283f5f9530761bbd95fc65a6678617c267fa6d4ecc5e"},{"start":39626,"end":39734,"start_line":833,"end_line":835,"raw_tex":"\\beq \\label{eq:def_distance_measure}\n\\cD (\\pi_k, \\pi^*) := \\bbe_{s\\sim \\nu^*} [D_{\\pi_{k}}^{\\pi^*}(s)].\n\\eeq","sha256_utf8":"7c3b7940426e7aeedb81529778c989ea176c34ee45e56670036cd9c1110e1220"},{"start":65211,"end":65624,"start_line":1348,"end_line":1354,"raw_tex":"In this subsection, we assume that for a given policy $\\pi_k$,\nthere exists a stochastic estimator $\\cQ^{\\pi_k,\\xi_k}$ s.t.\n\\begin{align}\n \\bbe_{\\xi_k} [\\cQ^{\\pi_k,\\xi_k}] &= \\bar \\cQ^{\\pi_k}, \\label{eq:expectation_ass} \\\\\n\\bbe_{\\xi_k}[\\|\\cQ^{\\pi_k,\\xi_k} - Q^{\\pi_k} \\|_\\infty^2] &\\le \\sigma_k^2, \\label{eq:variance_ass}\\\\\n\\|\\bar \\cQ^{\\pi_k} - Q^{\\pi_k}\\|_\\infty &\\le \\varsigma_k, \\label{eq:bias_ass}\n\\end{align}","sha256_utf8":"9e276b2f32b2551ac93ee53a89c7aa536bc162dfbbcd81a78a74b118b0146e5a"},{"start":65625,"end":65856,"start_line":1355,"end_line":1358,"raw_tex":"for some $\\sigma_k\\ge$ and $\\varsigma_k \\ge 0$,\nwhere $\\xi_k$ denotes the random vector used to generate the stochastic\nestimator $\\cQ^{\\pi_k,\\xi_k}$.  Clearly, if $\\sigma_k = 0$, then\nwe have exact information about $Q^{\\pi_k}$. \n","sha256_utf8":"d0f7a295b66f49564e6f29ba44c908c8b3f9c4aa9d4fd30a74ed713fbb411371"},{"start":66503,"end":66876,"start_line":1369,"end_line":1374,"raw_tex":"The stochastic policy mirror descent (SPMD)\nis obtained by replacing $Q^{\\pi_k}$ in \\eqnok{eq:PMD_step} \nwith its stochastic estimator $\\cQ^{\\pi_k,\\xi_k}$, i.e.,\n\\beq \\label{eq:SPMD_step}\n\\pi_{k+1}(\\cdot|s) = \\argmin_{p(\\cdot|s) \\in \\Delta_{|\\cA|}} \\left\\{ \\Phi_k(p):=\\eta_k [\\langle \\cQ^{\\pi_{k},\\xi_k}(s, \\cdot), p(\\cdot|s) \\rangle + h^p(s)]+ D_{\\pi_k}^p(s)\\right\\}.\n\\eeq","sha256_utf8":"77463837fa9cc1fe019792382601cb92b1ea393a817b8bf5f348f065365a8e0e"},{"start":66877,"end":66991,"start_line":1375,"end_line":1375,"raw_tex":"In the sequel, we denote $\\xi_{\\lceil k \\rceil}$ the sequence of random vectors $\\xi_0, \\ldots, \\xi_k$ and define\n","sha256_utf8":"33af0e0a9a5e5751ea2e9d608d122d34cae60c6535f89a0f302e7aafcd1027a4"},{"start":66992,"end":67070,"start_line":1376,"end_line":1378,"raw_tex":"\\beq \\label{eq:def_SPMD_delta}\n\\delta_k := \\cQ^{\\pi_k,\\xi_k} - Q^{\\pi_k}.\n\\eeq","sha256_utf8":"269d22f6629fc32aa13ee469529d31b77b2fc09b753a183c7bd735946c8bca19"},{"start":73787,"end":74064,"start_line":1512,"end_line":1515,"raw_tex":"We are now ready to establish the convergence rate of the SPMD method.  \nWe start with the case when $\\mu > 0$ and state a constant stepsize rule which requires both\n$\\varsigma_k$ and $\\sigma_k$, $k \\ge 0$, to be small enough to guarantee the convergence of \nthe SPMD method. \n","sha256_utf8":"4f77b7c13fc89ae07bffe82a650e32d66d4b6ea0f0f63e5d8cbb0ad5b7d7da40"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem} \\label{theorem:SPMD_reg}\nSuppose that \n$\\eta_k = \\eta = \\tfrac{1-\\gamma}{\\gamma \\mu}$\nin the SPMD method. \nIf $\\varsigma_k =2^{-(\\lfloor k/l \\rfloor+2)}$ and $\\sigma_k^2 = 2^{-(\\lfloor k/l \\rfloor+2)}$ \nfor any $k \\ge 0$ with $l :=\\left \\lceil  \\log_\\gamma (1/4) \\right \\rceil\n$,\nthen \n\\begin{align}\n& \\bbe_{\\xi_{\\lceil k-1 \\rceil}} [f(\\pi_{k}) - f(\\pi^*) \n+ \\tfrac{\\mu}{1-\\gamma} \\cD(\\pi_{k},\\pi^*)]\\nn \\\\\n&\\le 2^{-\\lfloor k/l \\rfloor}\\left [f(\\pi_0) - f(\\pi^*) +\\tfrac{1}{1-\\gamma}( \\mu  \\log |\\cA|\n+ \\tfrac{5} {2} + \\tfrac{ 5}{8 \\gamma \\mu})\\right]. \\label{eq:SPMD_reg_MDP_results_uniform_constant1}\n\\end{align}\n\\end{theorem}","units":[{"start":74065,"end":74709,"start_line":1517,"end_line":1531,"raw_tex":"\\begin{theorem} \\label{theorem:SPMD_reg}\nSuppose that \n$\\eta_k = \\eta = \\tfrac{1-\\gamma}{\\gamma \\mu}$\nin the SPMD method. \nIf $\\varsigma_k =2^{-(\\lfloor k/l \\rfloor+2)}$ and $\\sigma_k^2 = 2^{-(\\lfloor k/l \\rfloor+2)}$ \nfor any $k \\ge 0$ with $l :=\\left \\lceil  \\log_\\gamma (1/4) \\right \\rceil\n$,\nthen \n\\begin{align}\n& \\bbe_{\\xi_{\\lceil k-1 \\rceil}} [f(\\pi_{k}) - f(\\pi^*) \n+ \\tfrac{\\mu}{1-\\gamma} \\cD(\\pi_{k},\\pi^*)]\\nn \\\\\n&\\le 2^{-\\lfloor k/l \\rfloor}\\left [f(\\pi_0) - f(\\pi^*) +\\tfrac{1}{1-\\gamma}( \\mu  \\log |\\cA|\n+ \\tfrac{5} {2} + \\tfrac{ 5}{8 \\gamma \\mu})\\right]. \\label{eq:SPMD_reg_MDP_results_uniform_constant1}\n\\end{align}\n\\end{theorem}","sha256_utf8":"d69d958218b745992d9289db9d7849ec217e1a75603ec29f37e34f267c5c8703"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma} \\label{lemma:gradient}\nFor any $(s_0, s, a) \\in \\cS \\times \\cS \\times \\cA$, we have\n\\[\n\\tfrac{\\partial V^\\pi(s_0)}{\\partial \\pi(a|s)}\n= \\tfrac{1}{1-\\gamma} d_{s_0}^\\pi(s)\n\\left[ \nQ^\\pi(s,a) +  \\nabla h^\\pi(s, a) \n\\right],\n\\]\nwhere $\\nabla h^\\pi(s,\\cdot)$ denotes the gradient of $h^\\pi(s)$ w.r.t. $\\pi$.\n\\end{lemma}\n\n\\begin{proof}\nIt follows from \\eqnok{eq:QV1} that\n\\begin{align*}\n\\tfrac{\\partial V^\\pi(s_0)}{\\partial \\pi(a|s)}\n&= \\tfrac{\\partial}{\\partial \\pi(a'|s)} \\tsum_{a'\\in \\cA} \\pi(a'|s_0) Q^\\pi(s_0, a')\\\\\n&=  \\tsum_{a' \\in \\cA} \n\\left[\n\\tfrac{\\partial \\pi(a'|s_0)}{\\partial \\pi(a|s)} Q^\\pi(s_0, a')\n+ \\pi(a'|s_0) \n\\tfrac{\\partial Q^\\pi(s_0, a')}{\\partial \\pi(a|s)}\n\\right].\n\\end{align*}\nAlso the relation in \\eqnok{eq:QV2}\nimplies that\n\\begin{align*}\n\\tfrac{\\partial Q^\\pi(s_0, a')}{\\partial \\pi(a|s)}\n= \\tfrac{  \\partial h^\\pi(s_0)}{\\partial \\pi (a|s)} +\n\\gamma \\tsum_{s' \\in \\cS} \n\\cP(s'| s_0, a') \\tfrac{\\partial V^\\pi(s')}{\\partial \\pi(a|s)}.\n\\end{align*}\nCombining the above two relations, we obtain\n\\begin{align*}\n\\tfrac{\\partial V^\\pi(s_0)}{\\partial \\pi(a|s)}\n&= \\tsum_{a' \\in \\cA} \\left[ \\tfrac{\\partial \\pi(a'|s_0)}{\\partial \\pi(a|s)} Q^\\pi(s_0, a')\n+ \\pi(a'|s_0)  \\tfrac{  \\partial h^\\pi(s_0)}{\\partial \\pi (a|s)}\\right]  \\\\\n& \\quad + \\gamma \\tsum_{a' \\in \\cA} \\pi(a'|s_0) \\tsum_{s' \\in \\cS} \n\\cP(s'| s_0, a') \\tfrac{\\partial V^\\pi(s')}{\\partial \\pi(a|s)} \\\\\n&=\\tsum_{x\\in \\cS} \\tsum_{t=0}^\\infty \\gamma^t \\Pr^\\pi(s_t = x | s_0) \\\\\n&\\quad \\quad  \\tsum_{a' \\in \\cA} \\left[ \\tfrac{\\partial \\pi(a'|x)}{\\partial \\pi(a|s)} Q^\\pi(x, a')\n+ \\pi(a'|x)  \\tfrac{   \\partial h^\\pi(x)}{\\partial \\pi (a|s)}\\right]\\\\\n&=\\tfrac{1}{1-\\gamma} \\tsum_{x\\in \\cS} d_{s_0}^\\pi(x) \\left\\{\n\\tsum_{a' \\in \\cA} \\left[ \\tfrac{\\partial \\pi(a'|x)}{\\partial \\pi(a|s)} Q^\\pi(x, a')\\right]\n+ \\tfrac{   \\partial h^\\pi(x)}{\\partial \\pi (a|s)}\\right\\}\\\\\n&= \\tfrac{1}{1-\\gamma} d_{s_0}^\\pi(s) \\left[ Q^\\pi(s, a) + \\tfrac{   \\partial h^\\pi(s)}{\\partial \\pi (a|s)}\\right],\n\\end{align*}\nwhere the second equality follows by\nexpanding $\\tfrac{\\partial V^\\pi(s')}{\\partial \\pi(a|s)}$ recursively,\nand the third equality follows from the definition of $d_{s_0}^\\pi(s)$\nin \\eqnok{eq:def_visitation}, and the last identity follows from $\\tfrac{\\partial \\pi(a'|x)}{\\partial \\pi(a|s)} =0$\nfor $x \\neq s$ or $a' \\neq a$, and $\\tfrac{\\partial h^\\pi(x)}{\\partial \\pi (a|s)} =0$\nfor $x \\neq s$.\n\\end{proof}\n\nIn view of Lemma~\\ref{lemma:gradient}, the gradient of the objective\n $f(\\pi)$ in \\eqnok{eq:MDP_OPT} at the optimal policy $\\pi^*$  is given by\n\\begin{align}\n\\tfrac{\\partial f(\\pi^*) }{\\partial \\pi(a|s)}\n&=  \\bbe_{s_0 \\sim \\nu^*}\\left[ \\tfrac{\\partial V^{\\pi^*}(s_0)}{\\partial \\pi(a|s)}\\right]\n=  \\tfrac{1}{1-\\gamma}  \\bbe_{s_0 \\sim \\nu^*}\\left[ d_{s_0}^{\\pi^*}(s)\n[Q^{\\pi^*}(s,a) + \\nabla h^{\\pi^*}(s, a)] \\right] \\nn \\\\\n&= \\tsum_{t=0}^\\infty \\gamma^t (\\nu^*)^T (\\cP^{\\pi^*})^t e_s \\, [Q^{\\pi^*}(s,a) + \\nabla h^{\\pi^*}(s, a)] \\nn \\\\\n&= \\tfrac{1}{1-\\gamma} (\\nu^*)^T e_s \\, [Q^{\\pi^*}(s,a) + \\nabla h^{\\pi^*}(s, a)] \\nn \\\\\n&= \\tfrac{1}{1-\\gamma} \\nu^*(s) \\, [Q^{\\pi^*}(s,a) + \\nabla h^{\\pi^*}(s, a)], \\label{eq:stationary_relation}\n\\end{align}\nwhere the third identity follows from \\eqnok{eq:def_visitation1} and\nthe last one follows from \nthe fact that $(\\nu^*)^T (\\cP^{\\pi^*})^t = (\\nu^*)^T$ for any $t \\ge 0$ \n since $\\nu^*$\nis the steady state distribution of $\\pi^*$. \n\n\n\\begin{lemma} \\label{lemma_per_diff}\nFor any two feasible policies $\\pi$ and $\\pi'$, we have\n\\[\nV^{\\pi'}(s) - V^{\\pi}(s) =\n\\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi'}}\n\\left[\n\\langle A^{\\pi}(s', \\cdot), \\pi'(\\cdot | s')\\rangle\n+ h^{\\pi'}(s') - h^{\\pi}(s')\n\\right],\n\\]\nwhere\n\\beq \\label{eq:def_advantage}\nA^{\\pi}(s', a) := Q^{\\pi}(s', a)-V^{\\pi}(s').\n\\eeq\n\\end{lemma}\n\n\\begin{proof}\nFor simplicity, let us denote $\\xi^{\\pi'}(s_0)$ the random process $(s_t, a_t, s_{t+1})$, $t \\ge 0$,\ngenerated by following the policy $\\pi'$ starting with the initial state $s_0$.\nIt then follows from the definition of $V^{\\pi'}$ that\n\\begin{align*}\n&V^{\\pi'}(s) - V^{\\pi}(s) \\\\\n&= \\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [c(s_t, a_t) + h^{\\pi'}(s_t)] \\right]- V^{\\pi}(s)\\\\\n&=\\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [ c(s_t, a_t) + h^{\\pi'}(s_t) + V^{\\pi}(s_t)-V^{\\pi}(s_t) ] \\right]- V^{\\pi}(s)\\\\\n &\\stackrel{(a)}{=} \\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [ c(s_t, a_t) + h^{\\pi'}(s_t) + \\gamma V^{\\pi}(s_{t+1})-V^{\\pi}(s_t) ] \\right] \\\\\n&\\quad \\quad \\quad \\quad \\quad+ \\bbe_{\\xi^{\\pi'}(s)} [V^{\\pi}(s_0)] - V^{\\pi}(s)\\\\\n&\\stackrel{(b)}{=} \\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [ c(s_t, a_t) + h^{\\pi'}(s_t) + \\gamma V^{\\pi}(s_{t+1})-V^{\\pi}(s_t) ]\\right] \\\\\n &= \\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [ c(s_t, a_t) + h^{\\pi}(s_t) + \\gamma V^{\\pi}(s_{t+1})-V^{\\pi}(s_t) \\right. \\\\\n&\\quad \\quad \\quad \\quad \\quad \\quad  \\quad \\quad  \\left. + h^{\\pi'}(s_t) -h^{\\pi}(s_t) ] \\right]\\\\\n &\\stackrel{(c)}{=}\\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t \\left[  Q^{\\pi}(s_t, a_t)-V^{\\pi}(s_t)\n +h^{\\pi'}(s_t) -h^{\\pi}(s_t) \\right] \\right],\n\\end{align*}\nwhere (a) follows by taking the term $V^{\\pi}(s_0)$ outside the summation,\n(b) follows from the fact that $\\bbe_{\\xi^{\\pi'}(s)} [V^{\\pi}(s_0)] = V^{\\pi}(s)$ \nsince the random process starts with $s_0 = s$, and (c) follows from \\eqnok{eq:QV2}.\nThe previous conclusion, together with \\eqnok{eq:def_advantage} and the definition $d_s^{\\pi'}$ in \\eqnok{eq:def_visitation}, then\nimply that\n\\begin{align*}\n&V^{\\pi'}(s) - V^{\\pi}(s) \\\\\n&= \\tfrac{1}{1-\\gamma}\\tsum_{s' \\in \\cS} \\tsum_{a' \\in \\cA}\nd_s^{\\pi'}(s') \\pi'(a' | s') \\left[ A^{\\pi}(s', a') +  h^{\\pi'}(s') - h^{\\pi}(s') \\right]\\\\\n&= \\tfrac{1}{1-\\gamma} \\tsum_{s' \\in \\cS} d_s^{\\pi'}(s')\n\\left[\n\\langle A^{\\pi}(s', \\cdot), \\pi'(\\cdot | s')\\rangle\n+ h^{\\pi'}(s') - h^{\\pi}(s')\n\\right], \n\\end{align*}\nwhich immediately implies the result.\n\\end{proof}\n\n\\begin{lemma} \\label{prop_strong_monotone}\nThe VI problem in \\eqnok{eq:VI_MDP} satisfies\n\\begin{align}\n&\\bbe_{s\\sim \\nu^*}\n\\left[\n \\langle Q^{\\pi}(s, \\cdot),  \\pi(\\cdot|s) - \\pi^*(\\cdot | s)\\rangle + h^\\pi(s) - h^{\\pi^*}(s)\n\\right] \\nn \\\\\n& \\quad \\quad =\n \\bbe_{s\\sim \\nu^*}[(1- \\gamma)(V^{\\pi}(s) - V^{\\pi^*}(s) ) ].\n\\end{align}\n\\end{lemma}\n\n\\begin{proof}\nIt follows from Lemma~\\ref{lemma_per_diff} (with $\\pi' = \\pi^*$) that\n\\begin{align*}\n(1- \\gamma) [V^{\\pi^*}(s) - V^{\\pi}(s)] \n=\\bbe_{s' \\sim d_s^{\\pi^*}}\n\\left[\n\\langle A^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s')\\rangle\n+ h^{\\pi^*}(s') - h^{\\pi}(s')\n\\right].\n\\end{align*}\nLet $e$ denote the vector of all $1$'s. Then, we have \n\\begin{align}\n\\langle A^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s')\\rangle\n&= \\langle Q^{\\pi}(s', \\cdot) - V^\\pi(s') e, \\pi^*(\\cdot | s')\\rangle \\nn\\\\\n&= \\langle Q^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s')\\rangle - V^\\pi(s') \\nn \\\\\n&=  \\langle Q^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s')\\rangle - \\langle Q^{\\pi}(s', \\cdot), \\pi(\\cdot|s') \\nn \\\\\n&= \\langle Q^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s') - \\pi(\\cdot|s')\\rangle, \\label{eq:inner_product_equivalence}\n\\end{align}\nwhere the first identity follows from the definition of $A^{\\pi}(s', \\cdot)$ in \\eqnok{eq:def_advantage},\nthe second equality follows from the fact that $\\langle e,  \\pi^*(\\cdot | s')\\rangle = 1$,\nand the third equality follows from the definition of $V^\\pi$ in \\eqnok{eq:def_V_function}.\nCombining the above two relations and taking expectation w.r.t. $\\nu^*$, we obtain\n\\begin{align*}\n&(1- \\gamma) \\bbe_{s\\sim \\nu^*}[V^{\\pi^*}(s) - V^{\\pi}(s)] \\\\\n&= \\bbe_{s\\sim \\nu^*, s' \\sim d_s^{\\pi^*}}\n\\left[\n \\langle Q^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s') - \\pi(\\cdot|s')\\rangle\n +   h^{\\pi^*}(s') - h^{\\pi}(s')\n\\right]\\\\\n&= \\bbe_{s\\sim \\nu^*}\n\\left[\n \\langle Q^{\\pi}(s, \\cdot), \\pi^*(\\cdot | s) - \\pi(\\cdot|s)\\rangle\n +   h^{\\pi^*}(s) - h^{\\pi}(s)\n\\right],\n\\end{align*}\nwhere the second identity follows similarly to \\eqnok{eq:stationary_relation}\nsince $\\nu^*$ is the steady state distribution induced by $\\pi^*$.\nThe result then follows by rearranging the terms.\n\\end{proof}\n\n\\begin{lemma} \\label{lemma:prox_optimality}\nFor any $p(\\cdot|s) \\in \\Delta_{|\\cA|}$,\nwe have\n\\begin{align*}\n&\\eta_k[ \\langle Q^{\\pi_k}(s, \\cdot), \\pi_{k+1}(\\cdot|s) - p(\\cdot|s) \\rangle + h^{\\pi_{k+1}}(s) - h^{p}(s)]\n+ D_{\\pi_k}^{\\pi_{k+1}}(s)\\\\\n& \\le D_{\\pi_k}^p(s) -  (1 + \\eta_k \\mu) D_{\\pi_{k+1}}^p(s).\n\\end{align*}\n\\end{lemma}\n\n\\begin{proof}\nBy the optimality condition of \\eqnok{eq:PMD_step},\n\\[\n\\langle \\eta_k [Q^{\\pi_{k}}(s, \\cdot) + (h')^{\\pi_{k+1}}(s, \\cdot)] + \\nabla D_{\\pi_k}^{\\pi_{k+1}}(s, \\cdot),  p(\\cdot|s) - \\pi_{k+1}(\\cdot|s)  \\rangle \\ge 0,\n\\ \\ \\forall p(\\cdot|s) \\in \\Delta_{|\\cA|},\n\\]\nwhere $(h')^{\\pi_{k+1}}$ denotes the subgradient of $h$ at $\\pi_{k+1}$ and\n$\\nabla D_{\\pi_k}^{\\pi_{k+1}}(s, \\cdot)$ denotes the gradient of $D_{\\pi_k}^{\\pi_{k+1}}(s)$ at $\\pi_{k+1}$.\nUsing the definition of Bregman's  distance, it is easy\nto verify that\n\\beq \\label{prox_identity}\nD_{\\pi_k}^p(s) = D_{\\pi_k}^{\\pi_{k+1}}(s) + \\langle \\nabla D_{\\pi_k}^{\\pi_{k+1}}(s, \\cdot), p(\\cdot|s) -\\pi_{k+1}(\\cdot|s) \\rangle + D_{\\pi_{k+1}}^p(s).\n\\eeq\nThe result then immediately follows by combining the above two relations together with \\eqnok{eq:convex_obj}.\n\\end{proof}\n\n\\begin{lemma} \\label{prop:PMD_generic}\nFor any $s \\in \\cS$, we have\n\\begin{align}\nV^{\\pi_{k+1}}(s) &\\le V^{\\pi_k}(s), \\label{eq:pmd_decrease1}\\\\\n\\langle Q^{\\pi_k}(s, \\cdot),\\, \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle + h^{\\pi_{k+1}}(s) - h^{\\pi_{k}}(s)  \n&\\ge V^{\\pi_{k+1}}(s) - V^{\\pi_k}(s). \\label{eq:pmd_decrease2}\n\\end{align}\n\\end{lemma}\n\n\\begin{proof}\nIt follows from Lemma~\\ref{lemma_per_diff} (with $\\pi' = \\pi_{k+1}$, $\\pi = \\pi_k$ and $\\tau = \\tau_k$) that\n\\begin{align}\n&V^{\\pi_{k+1}}(s) - V^{\\pi_k}(s) \\nn\\\\\n&=\n\\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle A^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s')\\rangle\n+ h^{\\pi_{k+1}}(s') - h^{\\pi_k}(s')\n\\right]. \\label{eq:perf_diff_pmd}\n\\end{align}\nSimilarly to \\eqnok{eq:inner_product_equivalence}, we can show that\n\\begin{align*}\n\\langle A^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s')\\rangle\n&= \\langle Q^{\\pi_k}(s', \\cdot) - V^{\\pi_k}(s') e, \\pi_{k+1}(\\cdot | s')\\rangle\\\\\n&= \\langle Q^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s')\\rangle - V^{\\pi_k}_{\\tau_k}(s')\\\\\n&= \\langle Q^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot| s')\\rangle.\n\\end{align*}\nCombining the above two identities, we then obtain\n\\begin{align}\nV^{\\pi_{k+1}}(s) - V^{\\pi_k}(s) &=\n\\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle Q^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle \\right.\\nn \\\\\n& \\quad \\quad \\quad \\quad \\quad \\quad \\quad \\quad\n\\left. + h^{\\pi_{k+1}}(s') - h^{\\pi_k}(s')\\right]. \\label{eq:improvement_in_one_step_temp}\n\\end{align}\nNow we conclude from Lemma~\\ref{lemma:prox_optimality} applied to \\eqnok{eq:PMD_step}\nwith $p(\\cdot|s') = \\pi_k(\\cdot|s')$ that\n\\begin{align}\n& \\langle Q^{\\pi_{k}}(s', \\cdot), \n\\pi_{k+1}(\\cdot|s') - \\pi_k(\\cdot|s') \\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_k}(s') \\nn \\\\\n&\\le  -\\tfrac{1}{\\eta_k} [(1+\\eta_k \\mu) D_{\\pi_{k+1}}^{\\pi_k}(s') + D_{\\pi_k}^{\\pi_{k+1}}(s')].\\label{eq:negative_term_pmd}\n\\end{align}\nThe previous two conclusions\nthen clearly imply the result in \\eqnok{eq:pmd_decrease1}.\nIt also follows from \\eqnok{eq:negative_term_pmd} that\n\\begin{align}\n&\\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle Q^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle \n + h^{\\pi_{k+1}}(s') - h^{\\pi_k}(s') \\right]\\nn \\\\\n& \\le d_s^{\\pi_{k+1}}(s) \\left[\n\\langle Q^{\\pi_k}(s, \\cdot), \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle \n + h^{\\pi_{k+1}}(s) - h^{\\pi_k}(s) \\right]\\nn \\\\\n &\\le (1-\\gamma) \\left[\n\\langle Q^{\\pi_k}(s, \\cdot), \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle \n + h^{\\pi_{k+1}}(s) - h^{\\pi_k}(s) \\right], \\label{eq:negative_term_pmd_prob_switch}\n\\end{align}\nwhere the last inequality follows from the fact that $d_s^{\\pi_{k+1}}(s) \\ge (1-\\gamma)$ due to the definition of $d_s^{\\pi_{k+1}}$\nin \\eqnok{eq:def_visitation}. The result in \\eqnok{eq:pmd_decrease1} \nthen follows immediately from \\eqnok{eq:improvement_in_one_step_temp} and the above inequality.\n\\end{proof}\n\n\\begin{lemma} \\label{lemma:induction_tech_sequence}\nAssume that the nonnegative sequences $\\{X_k\\}_{k \\ge 0}, \\{Y_k\\}_{k \\ge 0}$ and $\\{Z_k\\}_{k \\ge 0}$ satisfy\n\\beq \\label{eq:induction_tech_sequence}\nX_{k+1} \\le \\gamma X_{k} + (Y_k - Y_{k+1}) + Z_k. % \\ \\ \\mbox{and} \\ \\ Y_k \\ge Y_{k+1}.\n\\eeq\nLet us denote $l = \\left \\lceil  \\log_\\gamma \\tfrac{1}{4} \\right \\rceil$.\nIf $Y_k = Y \\cdot 2^{-(\\lfloor k/l \\rfloor+1)}$ and $Z_k = Z \\cdot 2^{-(\\lfloor k/l \\rfloor+2)}$ for some $Y\\ge 0$ and $Z \\ge 0$, \nthen \n\\beq  \\label{eq:induction_tech_sequence_result}\nX_k \\le 2^{-\\lfloor k/l \\rfloor} (X_0 + Y + \\tfrac{5Z}{4(1-\\gamma)}).\n\\eeq\n\\end{lemma}\n\n\\begin{proof}\nLet us group the indices $\\{0, \\ldots, k\\}$  into $\\bar p \\equiv \\lfloor k/l \\rfloor+1$ epochs\nwith each of the first $\\bar p - 1$ epochs consisting of $l$ iterations. Let $p=0, \\ldots, \\bar p$ be the epoch indices.\nWe first show that for any $p = 0, \\ldots, \\bar p - 1$,\n\\beq \\label{eq:induction_tech_result}\nX_{pl} \\le 2^{-p} (X_0 + Y + \\tfrac{Z}{1-\\gamma} ).\n\\eeq\nThis relation holds obviously for $p = 0$. Let us assume that \\eqnok{eq:induction_tech_result} holds at the beginning\nof epoch $p$ ad examine the progress made in epoch $p$.\nNote that for any indices  $k = p l, \\ldots, (p+1)l-1$ in epoch $p$,\nwe have $Y_{k} = Y \\cdot 2^{-(p+1)}$ and  $Z = Z \\cdot 2^{-(p+2)}$.\nBy applying \\eqnok{eq:induction_tech_sequence} recursively, we have\n\\begin{align*}\nX_{(p+1) l} &\\le \\gamma^l X_{pl} + Y_{pl}- Y_{(p+1)l} + Z_{pl}  \\tsum_{i=0}^{l-1} \\gamma^{i} \\\\\n&= \\gamma^l X_{pl} + Y_{(p+1)l} + Z_{pl} \\tfrac{1 - \\gamma^l}{1-\\gamma}\\\\\n&\\le \\gamma^l X_{pl} + Y \\cdot 2^{-(p+2)} +  \\tfrac{Z \\cdot 2^{-(p+2)}}{1-\\gamma}\\\\\n&\\le \\tfrac{1}{4}X_{pl} + Y \\cdot 2^{-(p+2)} +  \\tfrac{Z \\cdot 2^{-(p+2)}}{1-\\gamma}\\\\\n&\\le \\tfrac{1}{4} 2^{-p} (X_0 +  Y + \\tfrac{Z}{1-\\gamma} ) + Y \\cdot 2^{-(p+2)} +  \\tfrac{Z \\cdot 2^{-(p+2)}}{1-\\gamma}\\\\\n&\\le 2^{-(p+1)} (X_0 + Y + \\tfrac{Z}{1-\\gamma} ),\n\\end{align*}\nwhere the second inequality follows from the definition of $Z_{pl}$ and $\\gamma^l \\ge 0$,\nthe third one follows from $\\gamma^l \\le 1/4$, the fourth one follows by induction hypothesis, and\nthe last one follows by regrouping the terms.\nSince $k = (\\bar p -1) l + k\\pmod l$, we have\n\\begin{align*}\nX_k &\\le \\gamma^{{k\\pmod l}} X_{(\\bar p -1) l} + Z_{(\\bar p -1)l} \\tsum_{i=0}^{k\\pmod l - 1} \\gamma^i\\\\\n&\\le 2^{-(\\bar p -1)} (X_0 + Y + \\tfrac{Z}{1-\\gamma} ) + \\tfrac{Z \\cdot 2^{-(\\bar p +1)}}{1-\\gamma}\\\\\n&= 2^{-(\\bar p -1)} (X_0 + Y + \\tfrac{5 Z}{4(1-\\gamma)} ),\n\\end{align*}\nwhich implies the result.\n\\end{proof}\n\nBy using the assumptions in \\eqnok{eq:expectation_ass} and \\eqnok{eq:bias_ass} and\nthe decomposition \n\\begin{align*}\n \\langle \\cQ^{\\pi_{k},\\xi_k}(s, \\cdot), \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle\n &= \\langle Q^{\\pi_k}(s, \\cdot),  \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle \\\\\n & \\quad + \\langle \\bar \\cQ^{\\pi_k}(s, \\cdot) - Q^{\\pi_k}(s, \\cdot),  \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle \\\\\n & \\quad + \\langle  \\cQ^{\\pi_k,\\xi_k}(s, \\cdot) - \\bar \\cQ^{\\pi_k}(s, \\cdot),  \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle,\n\\end{align*}\nwe can see that %Taking expectation w.r.t. $\\xi_k$ conditional on $\\xi_{\\lceil \\xi_{k-1} \\rceil}$, \n\\begin{align}\n&\\bbe_{\\xi_k}[ \\langle \\cQ^{\\pi_{k},\\xi_k}(s, \\cdot), \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle \\mid \\xi_{\\lceil \\xi_{k-1} \\rceil}] \\nn \\\\\n&\\ge  \\langle Q^{\\pi_k}(s, \\cdot),  \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle - 2 \\varsigma_k. \\label{eq:bnd_bias_SPMD}\n\\end{align}\n\n\n\\begin{lemma} \\label{prop:SPMD_generic}\nFor any $s \\in \\cS$, we have\n\\begin{align}\nV^{\\pi_{k+1}}(s) - V^{\\pi_k}(s) &\\le\n\\langle \\cQ^{\\pi_k,\\xi_k}(s, \\cdot) , \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle + h^{\\pi_{k+1}}(s) - h^{\\pi_{k}}(s) \\nn\\\\\n& \\quad   +\n\\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi_{k+1}}(s)+ \\tfrac{\\eta_k \\|\\delta_k\\|_\\infty^2}{2(1-\\gamma)}.\n\\label{eq:spmd_decrease2}\n\\end{align}\n\\end{lemma}\n\n\\begin{proof}\n%We prove \\eqnok{eq:spmd_decrease2} first since its proof is slightly more complicated.\nObserve that \\eqnok{eq:improvement_in_one_step_temp}\nstill holds, and hence that\n\\begin{align}\n&V^{\\pi_{k+1}}(s) - V^{\\pi_k}(s)\\nn \\\\\n&=\n\\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle Q^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s') \\right] \\nn \\\\\n&=\n\\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s')\\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s') \\right.\\nn \\\\\n& \\quad \n\\left.  - \\langle \\delta_k, \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle \\right] \\nn\\\\\n&\\le \\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s') \\right.\\nn \\\\\n& \\quad\n\\left. +\n \\tfrac{1 }{2\\eta_k}\\|\\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\|_1^2 + \\tfrac{\\eta_k \\|\\delta_k\\|_\\infty^2}{2}  \\right] \\nn\\\\\n &\\le \\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s')\\right.\\nn \\\\\n& \\quad\n\\left. + \\tfrac{1}{\\eta_k} D_{\\pi_{k}}^{\\pi_{k+1}}(s') + \\tfrac{\\eta_k \\|\\delta_k\\|_\\infty^2}{2} \\right], \\label{eq:improvement_in_one_step_temp_stoch}\n\\end{align}\nwhere the first inequality follows from Young's inequality and\nthe second one follows from the strong convexity of $D_{\\pi_{k+1}}^{\\pi_{k}}$ w.r.t. to $\\|\\cdot\\|_1$.\nMoreover, we conclude from Lemma~\\ref{lemma:prox_optimality} \napplied to \\eqnok{eq:SPMD_step}\nwith $Q^{\\pi_{k}}$ replaced by $\\cQ^{\\pi_{k},\\xi_k}$ and $p(\\cdot|s') = \\pi_k(\\cdot|s')$ that\n\\begin{align*}\n& \\langle \\cQ^{\\pi_{k},\\xi_k}(s', \\cdot), \n\\pi_{k+1}(\\cdot|s') - \\pi_k(\\cdot|s') \\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s') + \\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi_{k+1}}(s')\\\\\n&\\le  -\\tfrac{1}{\\eta_k} [(1+ \\eta_k \\mu) D_{\\pi_{k+1}}^{\\pi_k}(s') ] \\le 0,\n\\end{align*}\nwhich implies that\n\\begin{align*}\n&\\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle \n+ h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s') +  \\tfrac{1}{\\eta_k} D_{\\pi_{k+1}}^{\\pi_{k}}(s') \\right]\\nn \\\\\n& \\le d_s^{\\pi_{k+1}}(s) \\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s, \\cdot), \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle + h^{\\pi_{k+1}}(s) - h^{\\pi_{k}}(s) \n + \\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi_{k+1}}(s) \\right]\\nn \\\\\n &\\le (1-\\gamma) \\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s, \\cdot), \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle + h^{\\pi_{k+1}}(s) - h^{\\pi_{k}}(s) \n + \\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi_{k+1}}(s) \\right], \n \\end{align*}\nwhere the last inequality follows from the fact that $d_s^{\\pi_{k+1}}(s) \\ge 1-\\gamma$ due to the definition of $d_s^{\\pi_{k+1}}$\nin \\eqnok{eq:def_visitation}. The result in \\eqnok{eq:spmd_decrease2} \nthen follows immediately from \\eqnok{eq:improvement_in_one_step_temp_stoch} and the above inequality.\n\\end{proof}\n\n\\begin{lemma} \\label{prop:SPMD_regularized_MDPs}\nFor any $k \\ge 0$, we have\n\\begin{align*}\n&  \\bbe_{\\xi_{\\lceil k \\rceil}} [f(\\pi_{k+1}) - f(\\pi^*) + (\\tfrac{1}{\\eta_k}+  \\mu) \\cD(\\pi_{k+1}, \\pi^*)] \\\\\n&\\le   \\bbe_{\\xi_{\\lceil k-1 \\rceil}} [\\gamma (f(\\pi_k) - f(\\pi^*))+ \\tfrac{1}{\\eta_k} \\cD(\\pi_k,\\pi^*) ]\n+ 2 \\varsigma_k + \\tfrac{ \\eta_k \\sigma_k^2}{2(1-\\gamma)}.\n\\end{align*}\n\\end{lemma}\n\n\\begin{proof}\nBy applying Lemma~\\ref{lemma:prox_optimality} to \\eqnok{eq:PMD_step}\n(with $Q^{\\pi_{k}}$ replaced by $\\cQ^{\\pi_{k},\\xi_k}$ and $p = \\pi^*$), we have\n\\begin{align*}\n&\\eta_k[ \\langle \\cQ^{\\pi_{k},\\xi_k}(s, \\cdot), \n\\pi_{k+1}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle \n+ h^{\\pi_{k+1}}(s) -  h^{\\pi^*}(s) ]\n+ D_{\\pi_k}^{\\pi_{k+1}}(s) \\\\\n&\\le D_{\\pi_k}^{\\pi^*}(s) - (1+\\eta_k \\mu) D_{\\pi_{k+1}}^{\\pi^*}(s),\n\\end{align*}\nwhich, in view of \\eqnok{eq:spmd_decrease2}, then implies that\n\\begin{align*}\n&\\langle \\cQ^{\\pi_{k},\\xi_k}(s, \\cdot) , \n\\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle + h^{\\pi_{k}}(s) -  h^{\\pi^*}(s) +  V^{\\pi_{k+1}}(s) - V^{\\pi_k}(s) \n \\\\\n& \n\\le \\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi^*}(s) - (\\tfrac{1}{\\eta_k} + \\mu) D_{\\pi_{k+1}}^{\\pi^*}(s) + \\tfrac{\\eta_k \\|\\delta_k\\|_\\infty^2}{2(1-\\gamma)}.\n\\end{align*}\nTaking expectation w.r.t. $\\xi_{\\lceil k \\rceil}$ and $\\nu^*$ on both sides of the above inequality,\nand using Lemma~\\ref{prop_strong_monotone} and the relation in \\eqnok{eq:bnd_bias_SPMD}, \nwe arrive at\n\\begin{align*}\n&\\bbe_{s\\sim \\nu^*, \\xi_{\\lceil k \\rceil}}\\left[(1- \\gamma)(V^{\\pi_k}(s) - V^{\\pi_\\tau^*}(s) ) \n+  V^{\\pi_{k+1}}(s) - V^{\\pi_k}(s) \\right] \\\\\n&\\le \\bbe_{s\\sim \\nu^*,\\xi_{\\lceil k \\rceil}}[\\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi^*}(s) - (\\tfrac{1}{\\eta_k}+ \\mu) D_{\\pi_{k+1}}^{\\pi^*}(s)] \n+ 2  \\varsigma_k + \\tfrac{ \\eta_k \\sigma_k^2}{2(1-\\gamma)}.\n\\end{align*}\nNoting $V^{\\pi_{k+1}}(s) -  V^{\\pi_k}(s)= \nV^{\\pi_{k+1}}(s) - V^{\\pi^*}(s) - [V^{\\pi_k}(s) - V^{\\pi^*}(s) ] $,\nrearranging the terms in the above inequality, and using the definition of $f$ in \\eqnok{eq:MDP_OPT}, we arrive at the result.\n\\end{proof}\n\n\\begin{proof}\nBy Lemma~\\ref{prop:SPMD_regularized_MDPs} and the selection of $\\eta$, we have\n\\begin{align*}\n& \\bbe_{\\xi_{\\lceil k \\rceil}} [f(\\pi_{k+1}) - f(\\pi^*) + \\tfrac{\\mu}{1-\\gamma}  \\cD(\\pi_{k+1},\\pi^*)] \\\\\n&\\le  \\gamma [ \\bbe_{\\xi_{\\lceil k-1 \\rceil}} [f(\\pi_k) - f(\\pi^*) + \\tfrac{\\mu}{1-\\gamma} \\cD(\\pi_k, \\pi^*) ]\n+ 2 \\varsigma_k + \\tfrac{ \\sigma_k^2}{2 \\gamma \\mu},\n\\end{align*}\nwhich, in view of Lemma~\\ref{lemma:induction_tech_sequence}\nwith $X_k =  \\bbe_{\\xi_{\\lceil k-1 \\rceil}} [f(\\pi_k) - f(\\pi^*) + \\tfrac{\\mu}{1-\\gamma} \\cD(\\pi_k, \\pi^*)$\nand $Z_k = 2 \\varsigma_k + \\tfrac{ \\sigma_k^2}{2 \\gamma \\mu}$,\nthen implies that\n\\begin{align*}\n&\\bbe_{\\xi_{\\lceil k-1 \\rceil}} [f(\\pi_k) - f(\\pi^*) + \\tfrac{\\mu}{1-\\gamma} \\cD(\\pi_k, \\pi^*)\\\\\n&\\le \\gamma^{\\lfloor k/l \\rfloor}\\left [f(\\pi_0) - f(\\pi^*) + \\tfrac{\\mu  \\cD(\\pi_0, \\pi^*)}{1-\\gamma}\n+ \\tfrac{5} {4}( \\tfrac{2}{1-\\gamma}+ \\tfrac{ 1}{2 \\gamma(1 -\\gamma) \\mu})\\right]\\\\\n&\\le \\gamma^{\\lfloor k/l \\rfloor}\\left [f(\\pi_0) - f(\\pi^*) +\\tfrac{1}{1-\\gamma}( \\mu  \\log |\\cA|\n+ \\tfrac{5} {2} + \\tfrac{ 5}{8 \\gamma \\mu})\\right].\n\\end{align*}\n\\end{proof}","units":[{"start":20604,"end":20933,"start_line":419,"end_line":429,"raw_tex":"\\begin{lemma} \\label{lemma:gradient}\nFor any $(s_0, s, a) \\in \\cS \\times \\cS \\times \\cA$, we have\n\\[\n\\tfrac{\\partial V^\\pi(s_0)}{\\partial \\pi(a|s)}\n= \\tfrac{1}{1-\\gamma} d_{s_0}^\\pi(s)\n\\left[ \nQ^\\pi(s,a) +  \\nabla h^\\pi(s, a) \n\\right],\n\\]\nwhere $\\nabla h^\\pi(s,\\cdot)$ denotes the gradient of $h^\\pi(s)$ w.r.t. $\\pi$.\n\\end{lemma}","sha256_utf8":"9e3bafb268b471a66e733c840a3f6e08fadcf9b4dd1eb4ffb4cdd5f02289717e"},{"start":20935,"end":22993,"start_line":431,"end_line":472,"raw_tex":"\\begin{proof}\nIt follows from \\eqnok{eq:QV1} that\n\\begin{align*}\n\\tfrac{\\partial V^\\pi(s_0)}{\\partial \\pi(a|s)}\n&= \\tfrac{\\partial}{\\partial \\pi(a'|s)} \\tsum_{a'\\in \\cA} \\pi(a'|s_0) Q^\\pi(s_0, a')\\\\\n&=  \\tsum_{a' \\in \\cA} \n\\left[\n\\tfrac{\\partial \\pi(a'|s_0)}{\\partial \\pi(a|s)} Q^\\pi(s_0, a')\n+ \\pi(a'|s_0) \n\\tfrac{\\partial Q^\\pi(s_0, a')}{\\partial \\pi(a|s)}\n\\right].\n\\end{align*}\nAlso the relation in \\eqnok{eq:QV2}\nimplies that\n\\begin{align*}\n\\tfrac{\\partial Q^\\pi(s_0, a')}{\\partial \\pi(a|s)}\n= \\tfrac{  \\partial h^\\pi(s_0)}{\\partial \\pi (a|s)} +\n\\gamma \\tsum_{s' \\in \\cS} \n\\cP(s'| s_0, a') \\tfrac{\\partial V^\\pi(s')}{\\partial \\pi(a|s)}.\n\\end{align*}\nCombining the above two relations, we obtain\n\\begin{align*}\n\\tfrac{\\partial V^\\pi(s_0)}{\\partial \\pi(a|s)}\n&= \\tsum_{a' \\in \\cA} \\left[ \\tfrac{\\partial \\pi(a'|s_0)}{\\partial \\pi(a|s)} Q^\\pi(s_0, a')\n+ \\pi(a'|s_0)  \\tfrac{  \\partial h^\\pi(s_0)}{\\partial \\pi (a|s)}\\right]  \\\\\n& \\quad + \\gamma \\tsum_{a' \\in \\cA} \\pi(a'|s_0) \\tsum_{s' \\in \\cS} \n\\cP(s'| s_0, a') \\tfrac{\\partial V^\\pi(s')}{\\partial \\pi(a|s)} \\\\\n&=\\tsum_{x\\in \\cS} \\tsum_{t=0}^\\infty \\gamma^t \\Pr^\\pi(s_t = x | s_0) \\\\\n&\\quad \\quad  \\tsum_{a' \\in \\cA} \\left[ \\tfrac{\\partial \\pi(a'|x)}{\\partial \\pi(a|s)} Q^\\pi(x, a')\n+ \\pi(a'|x)  \\tfrac{   \\partial h^\\pi(x)}{\\partial \\pi (a|s)}\\right]\\\\\n&=\\tfrac{1}{1-\\gamma} \\tsum_{x\\in \\cS} d_{s_0}^\\pi(x) \\left\\{\n\\tsum_{a' \\in \\cA} \\left[ \\tfrac{\\partial \\pi(a'|x)}{\\partial \\pi(a|s)} Q^\\pi(x, a')\\right]\n+ \\tfrac{   \\partial h^\\pi(x)}{\\partial \\pi (a|s)}\\right\\}\\\\\n&= \\tfrac{1}{1-\\gamma} d_{s_0}^\\pi(s) \\left[ Q^\\pi(s, a) + \\tfrac{   \\partial h^\\pi(s)}{\\partial \\pi (a|s)}\\right],\n\\end{align*}\nwhere the second equality follows by\nexpanding $\\tfrac{\\partial V^\\pi(s')}{\\partial \\pi(a|s)}$ recursively,\nand the third equality follows from the definition of $d_{s_0}^\\pi(s)$\nin \\eqnok{eq:def_visitation}, and the last identity follows from $\\tfrac{\\partial \\pi(a'|x)}{\\partial \\pi(a|s)} =0$\nfor $x \\neq s$ or $a' \\neq a$, and $\\tfrac{\\partial h^\\pi(x)}{\\partial \\pi (a|s)} =0$\nfor $x \\neq s$.\n\\end{proof}","sha256_utf8":"6eef8de53b13c87187827471487f830d5a349ad509ef73a132e59570e773527a"},{"start":23002,"end":23977,"start_line":476,"end_line":491,"raw_tex":"In view of Lemma~\\ref{lemma:gradient}, the gradient of the objective\n $f(\\pi)$ in \\eqnok{eq:MDP_OPT} at the optimal policy $\\pi^*$  is given by\n\\begin{align}\n\\tfrac{\\partial f(\\pi^*) }{\\partial \\pi(a|s)}\n&=  \\bbe_{s_0 \\sim \\nu^*}\\left[ \\tfrac{\\partial V^{\\pi^*}(s_0)}{\\partial \\pi(a|s)}\\right]\n=  \\tfrac{1}{1-\\gamma}  \\bbe_{s_0 \\sim \\nu^*}\\left[ d_{s_0}^{\\pi^*}(s)\n[Q^{\\pi^*}(s,a) + \\nabla h^{\\pi^*}(s, a)] \\right] \\nn \\\\\n&= \\tsum_{t=0}^\\infty \\gamma^t (\\nu^*)^T (\\cP^{\\pi^*})^t e_s \\, [Q^{\\pi^*}(s,a) + \\nabla h^{\\pi^*}(s, a)] \\nn \\\\\n&= \\tfrac{1}{1-\\gamma} (\\nu^*)^T e_s \\, [Q^{\\pi^*}(s,a) + \\nabla h^{\\pi^*}(s, a)] \\nn \\\\\n&= \\tfrac{1}{1-\\gamma} \\nu^*(s) \\, [Q^{\\pi^*}(s,a) + \\nabla h^{\\pi^*}(s, a)], \\label{eq:stationary_relation}\n\\end{align}\nwhere the third identity follows from \\eqnok{eq:def_visitation1} and\nthe last one follows from \nthe fact that $(\\nu^*)^T (\\cP^{\\pi^*})^t = (\\nu^*)^T$ for any $t \\ge 0$ \n since $\\nu^*$\nis the steady state distribution of $\\pi^*$. \n","sha256_utf8":"06ad5cf425b1a3f8ec3e21decfdbbf936c2db45adeb8355c925a471fa83bf445"},{"start":24828,"end":25195,"start_line":506,"end_line":520,"raw_tex":"\\begin{lemma} \\label{lemma_per_diff}\nFor any two feasible policies $\\pi$ and $\\pi'$, we have\n\\[\nV^{\\pi'}(s) - V^{\\pi}(s) =\n\\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi'}}\n\\left[\n\\langle A^{\\pi}(s', \\cdot), \\pi'(\\cdot | s')\\rangle\n+ h^{\\pi'}(s') - h^{\\pi}(s')\n\\right],\n\\]\nwhere\n\\beq \\label{eq:def_advantage}\nA^{\\pi}(s', a) := Q^{\\pi}(s', a)-V^{\\pi}(s').\n\\eeq\n\\end{lemma}","sha256_utf8":"03161ef7717ca928dc03128c49e268bc884f06ea39b102c12e65a4390d07ca49"},{"start":25197,"end":27362,"start_line":522,"end_line":566,"raw_tex":"\\begin{proof}\nFor simplicity, let us denote $\\xi^{\\pi'}(s_0)$ the random process $(s_t, a_t, s_{t+1})$, $t \\ge 0$,\ngenerated by following the policy $\\pi'$ starting with the initial state $s_0$.\nIt then follows from the definition of $V^{\\pi'}$ that\n\\begin{align*}\n&V^{\\pi'}(s) - V^{\\pi}(s) \\\\\n&= \\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [c(s_t, a_t) + h^{\\pi'}(s_t)] \\right]- V^{\\pi}(s)\\\\\n&=\\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [ c(s_t, a_t) + h^{\\pi'}(s_t) + V^{\\pi}(s_t)-V^{\\pi}(s_t) ] \\right]- V^{\\pi}(s)\\\\\n &\\stackrel{(a)}{=} \\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [ c(s_t, a_t) + h^{\\pi'}(s_t) + \\gamma V^{\\pi}(s_{t+1})-V^{\\pi}(s_t) ] \\right] \\\\\n&\\quad \\quad \\quad \\quad \\quad+ \\bbe_{\\xi^{\\pi'}(s)} [V^{\\pi}(s_0)] - V^{\\pi}(s)\\\\\n&\\stackrel{(b)}{=} \\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [ c(s_t, a_t) + h^{\\pi'}(s_t) + \\gamma V^{\\pi}(s_{t+1})-V^{\\pi}(s_t) ]\\right] \\\\\n &= \\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [ c(s_t, a_t) + h^{\\pi}(s_t) + \\gamma V^{\\pi}(s_{t+1})-V^{\\pi}(s_t) \\right. \\\\\n&\\quad \\quad \\quad \\quad \\quad \\quad  \\quad \\quad  \\left. + h^{\\pi'}(s_t) -h^{\\pi}(s_t) ] \\right]\\\\\n &\\stackrel{(c)}{=}\\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t \\left[  Q^{\\pi}(s_t, a_t)-V^{\\pi}(s_t)\n +h^{\\pi'}(s_t) -h^{\\pi}(s_t) \\right] \\right],\n\\end{align*}\nwhere (a) follows by taking the term $V^{\\pi}(s_0)$ outside the summation,\n(b) follows from the fact that $\\bbe_{\\xi^{\\pi'}(s)} [V^{\\pi}(s_0)] = V^{\\pi}(s)$ \nsince the random process starts with $s_0 = s$, and (c) follows from \\eqnok{eq:QV2}.\nThe previous conclusion, together with \\eqnok{eq:def_advantage} and the definition $d_s^{\\pi'}$ in \\eqnok{eq:def_visitation}, then\nimply that\n\\begin{align*}\n&V^{\\pi'}(s) - V^{\\pi}(s) \\\\\n&= \\tfrac{1}{1-\\gamma}\\tsum_{s' \\in \\cS} \\tsum_{a' \\in \\cA}\nd_s^{\\pi'}(s') \\pi'(a' | s') \\left[ A^{\\pi}(s', a') +  h^{\\pi'}(s') - h^{\\pi}(s') \\right]\\\\\n&= \\tfrac{1}{1-\\gamma} \\tsum_{s' \\in \\cS} d_s^{\\pi'}(s')\n\\left[\n\\langle A^{\\pi}(s', \\cdot), \\pi'(\\cdot | s')\\rangle\n+ h^{\\pi'}(s') - h^{\\pi}(s')\n\\right], \n\\end{align*}\nwhich immediately implies the result.\n\\end{proof}","sha256_utf8":"7708c88222a983b84704f91f1558d16b5b44f25cde95461387d5cee222579357"},{"start":27481,"end":27822,"start_line":573,"end_line":583,"raw_tex":"\\begin{lemma} \\label{prop_strong_monotone}\nThe VI problem in \\eqnok{eq:VI_MDP} satisfies\n\\begin{align}\n&\\bbe_{s\\sim \\nu^*}\n\\left[\n \\langle Q^{\\pi}(s, \\cdot),  \\pi(\\cdot|s) - \\pi^*(\\cdot | s)\\rangle + h^\\pi(s) - h^{\\pi^*}(s)\n\\right] \\nn \\\\\n& \\quad \\quad =\n \\bbe_{s\\sim \\nu^*}[(1- \\gamma)(V^{\\pi}(s) - V^{\\pi^*}(s) ) ].\n\\end{align}\n\\end{lemma}","sha256_utf8":"074a0aecf318571f7b902ffc2146b3c8f3d695500c1bf64b21e47c038c51567e"},{"start":27824,"end":29574,"start_line":585,"end_line":623,"raw_tex":"\\begin{proof}\nIt follows from Lemma~\\ref{lemma_per_diff} (with $\\pi' = \\pi^*$) that\n\\begin{align*}\n(1- \\gamma) [V^{\\pi^*}(s) - V^{\\pi}(s)] \n=\\bbe_{s' \\sim d_s^{\\pi^*}}\n\\left[\n\\langle A^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s')\\rangle\n+ h^{\\pi^*}(s') - h^{\\pi}(s')\n\\right].\n\\end{align*}\nLet $e$ denote the vector of all $1$'s. Then, we have \n\\begin{align}\n\\langle A^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s')\\rangle\n&= \\langle Q^{\\pi}(s', \\cdot) - V^\\pi(s') e, \\pi^*(\\cdot | s')\\rangle \\nn\\\\\n&= \\langle Q^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s')\\rangle - V^\\pi(s') \\nn \\\\\n&=  \\langle Q^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s')\\rangle - \\langle Q^{\\pi}(s', \\cdot), \\pi(\\cdot|s') \\nn \\\\\n&= \\langle Q^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s') - \\pi(\\cdot|s')\\rangle, \\label{eq:inner_product_equivalence}\n\\end{align}\nwhere the first identity follows from the definition of $A^{\\pi}(s', \\cdot)$ in \\eqnok{eq:def_advantage},\nthe second equality follows from the fact that $\\langle e,  \\pi^*(\\cdot | s')\\rangle = 1$,\nand the third equality follows from the definition of $V^\\pi$ in \\eqnok{eq:def_V_function}.\nCombining the above two relations and taking expectation w.r.t. $\\nu^*$, we obtain\n\\begin{align*}\n&(1- \\gamma) \\bbe_{s\\sim \\nu^*}[V^{\\pi^*}(s) - V^{\\pi}(s)] \\\\\n&= \\bbe_{s\\sim \\nu^*, s' \\sim d_s^{\\pi^*}}\n\\left[\n \\langle Q^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s') - \\pi(\\cdot|s')\\rangle\n +   h^{\\pi^*}(s') - h^{\\pi}(s')\n\\right]\\\\\n&= \\bbe_{s\\sim \\nu^*}\n\\left[\n \\langle Q^{\\pi}(s, \\cdot), \\pi^*(\\cdot | s) - \\pi(\\cdot|s)\\rangle\n +   h^{\\pi^*}(s) - h^{\\pi}(s)\n\\right],\n\\end{align*}\nwhere the second identity follows similarly to \\eqnok{eq:stationary_relation}\nsince $\\nu^*$ is the steady state distribution induced by $\\pi^*$.\nThe result then follows by rearranging the terms.\n\\end{proof}","sha256_utf8":"d33b67fb16aeb9492ef4738ce9aca30baa48146b05cdc8d328e13c2338b1eb7e"},{"start":34951,"end":35282,"start_line":729,"end_line":737,"raw_tex":"\\begin{lemma} \\label{lemma:prox_optimality}\nFor any $p(\\cdot|s) \\in \\Delta_{|\\cA|}$,\nwe have\n\\begin{align*}\n&\\eta_k[ \\langle Q^{\\pi_k}(s, \\cdot), \\pi_{k+1}(\\cdot|s) - p(\\cdot|s) \\rangle + h^{\\pi_{k+1}}(s) - h^{p}(s)]\n+ D_{\\pi_k}^{\\pi_{k+1}}(s)\\\\\n& \\le D_{\\pi_k}^p(s) -  (1 + \\eta_k \\mu) D_{\\pi_{k+1}}^p(s).\n\\end{align*}\n\\end{lemma}","sha256_utf8":"67dbdd1a63a6d827059580b0d3b622c8707f8e5877152cb6c025398fec679422"},{"start":35284,"end":36118,"start_line":739,"end_line":753,"raw_tex":"\\begin{proof}\nBy the optimality condition of \\eqnok{eq:PMD_step},\n\\[\n\\langle \\eta_k [Q^{\\pi_{k}}(s, \\cdot) + (h')^{\\pi_{k+1}}(s, \\cdot)] + \\nabla D_{\\pi_k}^{\\pi_{k+1}}(s, \\cdot),  p(\\cdot|s) - \\pi_{k+1}(\\cdot|s)  \\rangle \\ge 0,\n\\ \\ \\forall p(\\cdot|s) \\in \\Delta_{|\\cA|},\n\\]\nwhere $(h')^{\\pi_{k+1}}$ denotes the subgradient of $h$ at $\\pi_{k+1}$ and\n$\\nabla D_{\\pi_k}^{\\pi_{k+1}}(s, \\cdot)$ denotes the gradient of $D_{\\pi_k}^{\\pi_{k+1}}(s)$ at $\\pi_{k+1}$.\nUsing the definition of Bregman's  distance, it is easy\nto verify that\n\\beq \\label{prox_identity}\nD_{\\pi_k}^p(s) = D_{\\pi_k}^{\\pi_{k+1}}(s) + \\langle \\nabla D_{\\pi_k}^{\\pi_{k+1}}(s, \\cdot), p(\\cdot|s) -\\pi_{k+1}(\\cdot|s) \\rangle + D_{\\pi_{k+1}}^p(s).\n\\eeq\nThe result then immediately follows by combining the above two relations together with \\eqnok{eq:convex_obj}.\n\\end{proof}","sha256_utf8":"84742826d21e375b77bf8c08c0ebdeca899769e9e7c74def8c5066049c8c5148"},{"start":36120,"end":36466,"start_line":755,"end_line":762,"raw_tex":"\\begin{lemma} \\label{prop:PMD_generic}\nFor any $s \\in \\cS$, we have\n\\begin{align}\nV^{\\pi_{k+1}}(s) &\\le V^{\\pi_k}(s), \\label{eq:pmd_decrease1}\\\\\n\\langle Q^{\\pi_k}(s, \\cdot),\\, \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle + h^{\\pi_{k+1}}(s) - h^{\\pi_{k}}(s)  \n&\\ge V^{\\pi_{k+1}}(s) - V^{\\pi_k}(s). \\label{eq:pmd_decrease2}\n\\end{align}\n\\end{lemma}","sha256_utf8":"0a229271a1d90bee5d1d487abd1200cbd52025817e5f81959cfc5a009d2e70a2"},{"start":36468,"end":39049,"start_line":764,"end_line":816,"raw_tex":"\\begin{proof}\nIt follows from Lemma~\\ref{lemma_per_diff} (with $\\pi' = \\pi_{k+1}$, $\\pi = \\pi_k$ and $\\tau = \\tau_k$) that\n\\begin{align}\n&V^{\\pi_{k+1}}(s) - V^{\\pi_k}(s) \\nn\\\\\n&=\n\\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle A^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s')\\rangle\n+ h^{\\pi_{k+1}}(s') - h^{\\pi_k}(s')\n\\right]. \\label{eq:perf_diff_pmd}\n\\end{align}\nSimilarly to \\eqnok{eq:inner_product_equivalence}, we can show that\n\\begin{align*}\n\\langle A^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s')\\rangle\n&= \\langle Q^{\\pi_k}(s', \\cdot) - V^{\\pi_k}(s') e, \\pi_{k+1}(\\cdot | s')\\rangle\\\\\n&= \\langle Q^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s')\\rangle - V^{\\pi_k}_{\\tau_k}(s')\\\\\n&= \\langle Q^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot| s')\\rangle.\n\\end{align*}\nCombining the above two identities, we then obtain\n\\begin{align}\nV^{\\pi_{k+1}}(s) - V^{\\pi_k}(s) &=\n\\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle Q^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle \\right.\\nn \\\\\n& \\quad \\quad \\quad \\quad \\quad \\quad \\quad \\quad\n\\left. + h^{\\pi_{k+1}}(s') - h^{\\pi_k}(s')\\right]. \\label{eq:improvement_in_one_step_temp}\n\\end{align}\nNow we conclude from Lemma~\\ref{lemma:prox_optimality} applied to \\eqnok{eq:PMD_step}\nwith $p(\\cdot|s') = \\pi_k(\\cdot|s')$ that\n\\begin{align}\n& \\langle Q^{\\pi_{k}}(s', \\cdot), \n\\pi_{k+1}(\\cdot|s') - \\pi_k(\\cdot|s') \\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_k}(s') \\nn \\\\\n&\\le  -\\tfrac{1}{\\eta_k} [(1+\\eta_k \\mu) D_{\\pi_{k+1}}^{\\pi_k}(s') + D_{\\pi_k}^{\\pi_{k+1}}(s')].\\label{eq:negative_term_pmd}\n\\end{align}\nThe previous two conclusions\nthen clearly imply the result in \\eqnok{eq:pmd_decrease1}.\nIt also follows from \\eqnok{eq:negative_term_pmd} that\n\\begin{align}\n&\\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle Q^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle \n + h^{\\pi_{k+1}}(s') - h^{\\pi_k}(s') \\right]\\nn \\\\\n& \\le d_s^{\\pi_{k+1}}(s) \\left[\n\\langle Q^{\\pi_k}(s, \\cdot), \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle \n + h^{\\pi_{k+1}}(s) - h^{\\pi_k}(s) \\right]\\nn \\\\\n &\\le (1-\\gamma) \\left[\n\\langle Q^{\\pi_k}(s, \\cdot), \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle \n + h^{\\pi_{k+1}}(s) - h^{\\pi_k}(s) \\right], \\label{eq:negative_term_pmd_prob_switch}\n\\end{align}\nwhere the last inequality follows from the fact that $d_s^{\\pi_{k+1}}(s) \\ge (1-\\gamma)$ due to the definition of $d_s^{\\pi_{k+1}}$\nin \\eqnok{eq:def_visitation}. The result in \\eqnok{eq:pmd_decrease1} \nthen follows immediately from \\eqnok{eq:improvement_in_one_step_temp} and the above inequality.\n\\end{proof}","sha256_utf8":"fb4f2bd36662a9a2da2821de761009f9f63e5e96052792e3fdee6ef64ed32fd2"},{"start":59015,"end":59654,"start_line":1235,"end_line":1246,"raw_tex":"\\begin{lemma} \\label{lemma:induction_tech_sequence}\nAssume that the nonnegative sequences $\\{X_k\\}_{k \\ge 0}, \\{Y_k\\}_{k \\ge 0}$ and $\\{Z_k\\}_{k \\ge 0}$ satisfy\n\\beq \\label{eq:induction_tech_sequence}\nX_{k+1} \\le \\gamma X_{k} + (Y_k - Y_{k+1}) + Z_k. % \\ \\ \\mbox{and} \\ \\ Y_k \\ge Y_{k+1}.\n\\eeq\nLet us denote $l = \\left \\lceil  \\log_\\gamma \\tfrac{1}{4} \\right \\rceil$.\nIf $Y_k = Y \\cdot 2^{-(\\lfloor k/l \\rfloor+1)}$ and $Z_k = Z \\cdot 2^{-(\\lfloor k/l \\rfloor+2)}$ for some $Y\\ge 0$ and $Z \\ge 0$, \nthen \n\\beq  \\label{eq:induction_tech_sequence_result}\nX_k \\le 2^{-\\lfloor k/l \\rfloor} (X_0 + Y + \\tfrac{5Z}{4(1-\\gamma)}).\n\\eeq\n\\end{lemma}","sha256_utf8":"83c6be1b6fecf2852a74635b0dc1372c92f40b199ef8fd66f50542aca7f13906"},{"start":59656,"end":61568,"start_line":1248,"end_line":1278,"raw_tex":"\\begin{proof}\nLet us group the indices $\\{0, \\ldots, k\\}$  into $\\bar p \\equiv \\lfloor k/l \\rfloor+1$ epochs\nwith each of the first $\\bar p - 1$ epochs consisting of $l$ iterations. Let $p=0, \\ldots, \\bar p$ be the epoch indices.\nWe first show that for any $p = 0, \\ldots, \\bar p - 1$,\n\\beq \\label{eq:induction_tech_result}\nX_{pl} \\le 2^{-p} (X_0 + Y + \\tfrac{Z}{1-\\gamma} ).\n\\eeq\nThis relation holds obviously for $p = 0$. Let us assume that \\eqnok{eq:induction_tech_result} holds at the beginning\nof epoch $p$ ad examine the progress made in epoch $p$.\nNote that for any indices  $k = p l, \\ldots, (p+1)l-1$ in epoch $p$,\nwe have $Y_{k} = Y \\cdot 2^{-(p+1)}$ and  $Z = Z \\cdot 2^{-(p+2)}$.\nBy applying \\eqnok{eq:induction_tech_sequence} recursively, we have\n\\begin{align*}\nX_{(p+1) l} &\\le \\gamma^l X_{pl} + Y_{pl}- Y_{(p+1)l} + Z_{pl}  \\tsum_{i=0}^{l-1} \\gamma^{i} \\\\\n&= \\gamma^l X_{pl} + Y_{(p+1)l} + Z_{pl} \\tfrac{1 - \\gamma^l}{1-\\gamma}\\\\\n&\\le \\gamma^l X_{pl} + Y \\cdot 2^{-(p+2)} +  \\tfrac{Z \\cdot 2^{-(p+2)}}{1-\\gamma}\\\\\n&\\le \\tfrac{1}{4}X_{pl} + Y \\cdot 2^{-(p+2)} +  \\tfrac{Z \\cdot 2^{-(p+2)}}{1-\\gamma}\\\\\n&\\le \\tfrac{1}{4} 2^{-p} (X_0 +  Y + \\tfrac{Z}{1-\\gamma} ) + Y \\cdot 2^{-(p+2)} +  \\tfrac{Z \\cdot 2^{-(p+2)}}{1-\\gamma}\\\\\n&\\le 2^{-(p+1)} (X_0 + Y + \\tfrac{Z}{1-\\gamma} ),\n\\end{align*}\nwhere the second inequality follows from the definition of $Z_{pl}$ and $\\gamma^l \\ge 0$,\nthe third one follows from $\\gamma^l \\le 1/4$, the fourth one follows by induction hypothesis, and\nthe last one follows by regrouping the terms.\nSince $k = (\\bar p -1) l + k\\pmod l$, we have\n\\begin{align*}\nX_k &\\le \\gamma^{{k\\pmod l}} X_{(\\bar p -1) l} + Z_{(\\bar p -1)l} \\tsum_{i=0}^{k\\pmod l - 1} \\gamma^i\\\\\n&\\le 2^{-(\\bar p -1)} (X_0 + Y + \\tfrac{Z}{1-\\gamma} ) + \\tfrac{Z \\cdot 2^{-(\\bar p +1)}}{1-\\gamma}\\\\\n&= 2^{-(\\bar p -1)} (X_0 + Y + \\tfrac{5 Z}{4(1-\\gamma)} ),\n\\end{align*}\nwhich implies the result.\n\\end{proof}","sha256_utf8":"5221ae0c6bc21e512ce7e142d5dc668fdd05ac0b7cc4276eda78f9c427f6400f"},{"start":67071,"end":67984,"start_line":1379,"end_line":1391,"raw_tex":"By using the assumptions in \\eqnok{eq:expectation_ass} and \\eqnok{eq:bias_ass} and\nthe decomposition \n\\begin{align*}\n \\langle \\cQ^{\\pi_{k},\\xi_k}(s, \\cdot), \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle\n &= \\langle Q^{\\pi_k}(s, \\cdot),  \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle \\\\\n & \\quad + \\langle \\bar \\cQ^{\\pi_k}(s, \\cdot) - Q^{\\pi_k}(s, \\cdot),  \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle \\\\\n & \\quad + \\langle  \\cQ^{\\pi_k,\\xi_k}(s, \\cdot) - \\bar \\cQ^{\\pi_k}(s, \\cdot),  \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle,\n\\end{align*}\nwe can see that %Taking expectation w.r.t. $\\xi_k$ conditional on $\\xi_{\\lceil \\xi_{k-1} \\rceil}$, \n\\begin{align}\n&\\bbe_{\\xi_k}[ \\langle \\cQ^{\\pi_{k},\\xi_k}(s, \\cdot), \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle \\mid \\xi_{\\lceil \\xi_{k-1} \\rceil}] \\nn \\\\\n&\\ge  \\langle Q^{\\pi_k}(s, \\cdot),  \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle - 2 \\varsigma_k. \\label{eq:bnd_bias_SPMD}\n\\end{align}\n","sha256_utf8":"13591f91356a35508831835833bb237258cfd17a9c0989cd315aa049e895bfac"},{"start":68195,"end":68597,"start_line":1400,"end_line":1409,"raw_tex":"\\begin{lemma} \\label{prop:SPMD_generic}\nFor any $s \\in \\cS$, we have\n\\begin{align}\nV^{\\pi_{k+1}}(s) - V^{\\pi_k}(s) &\\le\n\\langle \\cQ^{\\pi_k,\\xi_k}(s, \\cdot) , \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle + h^{\\pi_{k+1}}(s) - h^{\\pi_{k}}(s) \\nn\\\\\n& \\quad   +\n\\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi_{k+1}}(s)+ \\tfrac{\\eta_k \\|\\delta_k\\|_\\infty^2}{2(1-\\gamma)}.\n\\label{eq:spmd_decrease2}\n\\end{align}\n\\end{lemma}","sha256_utf8":"66d4392b41e6226f1c1e8d3ef6f2d4d74b4ace6f0188e90bc7456a804fb77813"},{"start":68599,"end":71676,"start_line":1411,"end_line":1465,"raw_tex":"\\begin{proof}\n%We prove \\eqnok{eq:spmd_decrease2} first since its proof is slightly more complicated.\nObserve that \\eqnok{eq:improvement_in_one_step_temp}\nstill holds, and hence that\n\\begin{align}\n&V^{\\pi_{k+1}}(s) - V^{\\pi_k}(s)\\nn \\\\\n&=\n\\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle Q^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s') \\right] \\nn \\\\\n&=\n\\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s')\\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s') \\right.\\nn \\\\\n& \\quad \n\\left.  - \\langle \\delta_k, \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle \\right] \\nn\\\\\n&\\le \\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s') \\right.\\nn \\\\\n& \\quad\n\\left. +\n \\tfrac{1 }{2\\eta_k}\\|\\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\|_1^2 + \\tfrac{\\eta_k \\|\\delta_k\\|_\\infty^2}{2}  \\right] \\nn\\\\\n &\\le \\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s')\\right.\\nn \\\\\n& \\quad\n\\left. + \\tfrac{1}{\\eta_k} D_{\\pi_{k}}^{\\pi_{k+1}}(s') + \\tfrac{\\eta_k \\|\\delta_k\\|_\\infty^2}{2} \\right], \\label{eq:improvement_in_one_step_temp_stoch}\n\\end{align}\nwhere the first inequality follows from Young's inequality and\nthe second one follows from the strong convexity of $D_{\\pi_{k+1}}^{\\pi_{k}}$ w.r.t. to $\\|\\cdot\\|_1$.\nMoreover, we conclude from Lemma~\\ref{lemma:prox_optimality} \napplied to \\eqnok{eq:SPMD_step}\nwith $Q^{\\pi_{k}}$ replaced by $\\cQ^{\\pi_{k},\\xi_k}$ and $p(\\cdot|s') = \\pi_k(\\cdot|s')$ that\n\\begin{align*}\n& \\langle \\cQ^{\\pi_{k},\\xi_k}(s', \\cdot), \n\\pi_{k+1}(\\cdot|s') - \\pi_k(\\cdot|s') \\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s') + \\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi_{k+1}}(s')\\\\\n&\\le  -\\tfrac{1}{\\eta_k} [(1+ \\eta_k \\mu) D_{\\pi_{k+1}}^{\\pi_k}(s') ] \\le 0,\n\\end{align*}\nwhich implies that\n\\begin{align*}\n&\\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle \n+ h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s') +  \\tfrac{1}{\\eta_k} D_{\\pi_{k+1}}^{\\pi_{k}}(s') \\right]\\nn \\\\\n& \\le d_s^{\\pi_{k+1}}(s) \\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s, \\cdot), \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle + h^{\\pi_{k+1}}(s) - h^{\\pi_{k}}(s) \n + \\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi_{k+1}}(s) \\right]\\nn \\\\\n &\\le (1-\\gamma) \\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s, \\cdot), \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle + h^{\\pi_{k+1}}(s) - h^{\\pi_{k}}(s) \n + \\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi_{k+1}}(s) \\right], \n \\end{align*}\nwhere the last inequality follows from the fact that $d_s^{\\pi_{k+1}}(s) \\ge 1-\\gamma$ due to the definition of $d_s^{\\pi_{k+1}}$\nin \\eqnok{eq:def_visitation}. The result in \\eqnok{eq:spmd_decrease2} \nthen follows immediately from \\eqnok{eq:improvement_in_one_step_temp_stoch} and the above inequality.\n\\end{proof}","sha256_utf8":"34ab843e16126d6bb7ec28951b01d241142714e84d52e342b96d5d5e50092059"},{"start":71749,"end":72140,"start_line":1471,"end_line":1478,"raw_tex":"\\begin{lemma} \\label{prop:SPMD_regularized_MDPs}\nFor any $k \\ge 0$, we have\n\\begin{align*}\n&  \\bbe_{\\xi_{\\lceil k \\rceil}} [f(\\pi_{k+1}) - f(\\pi^*) + (\\tfrac{1}{\\eta_k}+  \\mu) \\cD(\\pi_{k+1}, \\pi^*)] \\\\\n&\\le   \\bbe_{\\xi_{\\lceil k-1 \\rceil}} [\\gamma (f(\\pi_k) - f(\\pi^*))+ \\tfrac{1}{\\eta_k} \\cD(\\pi_k,\\pi^*) ]\n+ 2 \\varsigma_k + \\tfrac{ \\eta_k \\sigma_k^2}{2(1-\\gamma)}.\n\\end{align*}\n\\end{lemma}","sha256_utf8":"f36c96be2decc58bbd7a2c1476d338e6ccde20d5847bd1f00ed34461e839c253"},{"start":72142,"end":73785,"start_line":1480,"end_line":1510,"raw_tex":"\\begin{proof}\nBy applying Lemma~\\ref{lemma:prox_optimality} to \\eqnok{eq:PMD_step}\n(with $Q^{\\pi_{k}}$ replaced by $\\cQ^{\\pi_{k},\\xi_k}$ and $p = \\pi^*$), we have\n\\begin{align*}\n&\\eta_k[ \\langle \\cQ^{\\pi_{k},\\xi_k}(s, \\cdot), \n\\pi_{k+1}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle \n+ h^{\\pi_{k+1}}(s) -  h^{\\pi^*}(s) ]\n+ D_{\\pi_k}^{\\pi_{k+1}}(s) \\\\\n&\\le D_{\\pi_k}^{\\pi^*}(s) - (1+\\eta_k \\mu) D_{\\pi_{k+1}}^{\\pi^*}(s),\n\\end{align*}\nwhich, in view of \\eqnok{eq:spmd_decrease2}, then implies that\n\\begin{align*}\n&\\langle \\cQ^{\\pi_{k},\\xi_k}(s, \\cdot) , \n\\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle + h^{\\pi_{k}}(s) -  h^{\\pi^*}(s) +  V^{\\pi_{k+1}}(s) - V^{\\pi_k}(s) \n \\\\\n& \n\\le \\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi^*}(s) - (\\tfrac{1}{\\eta_k} + \\mu) D_{\\pi_{k+1}}^{\\pi^*}(s) + \\tfrac{\\eta_k \\|\\delta_k\\|_\\infty^2}{2(1-\\gamma)}.\n\\end{align*}\nTaking expectation w.r.t. $\\xi_{\\lceil k \\rceil}$ and $\\nu^*$ on both sides of the above inequality,\nand using Lemma~\\ref{prop_strong_monotone} and the relation in \\eqnok{eq:bnd_bias_SPMD}, \nwe arrive at\n\\begin{align*}\n&\\bbe_{s\\sim \\nu^*, \\xi_{\\lceil k \\rceil}}\\left[(1- \\gamma)(V^{\\pi_k}(s) - V^{\\pi_\\tau^*}(s) ) \n+  V^{\\pi_{k+1}}(s) - V^{\\pi_k}(s) \\right] \\\\\n&\\le \\bbe_{s\\sim \\nu^*,\\xi_{\\lceil k \\rceil}}[\\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi^*}(s) - (\\tfrac{1}{\\eta_k}+ \\mu) D_{\\pi_{k+1}}^{\\pi^*}(s)] \n+ 2  \\varsigma_k + \\tfrac{ \\eta_k \\sigma_k^2}{2(1-\\gamma)}.\n\\end{align*}\nNoting $V^{\\pi_{k+1}}(s) -  V^{\\pi_k}(s)= \nV^{\\pi_{k+1}}(s) - V^{\\pi^*}(s) - [V^{\\pi_k}(s) - V^{\\pi^*}(s) ] $,\nrearranging the terms in the above inequality, and using the definition of $f$ in \\eqnok{eq:MDP_OPT}, we arrive at the result.\n\\end{proof}","sha256_utf8":"8c2bd37fe155e5fce049a7e206b931234f8ba88b8e82af40dd6d6bde10d6460a"},{"start":74711,"end":75824,"start_line":1533,"end_line":1551,"raw_tex":"\\begin{proof}\nBy Lemma~\\ref{prop:SPMD_regularized_MDPs} and the selection of $\\eta$, we have\n\\begin{align*}\n& \\bbe_{\\xi_{\\lceil k \\rceil}} [f(\\pi_{k+1}) - f(\\pi^*) + \\tfrac{\\mu}{1-\\gamma}  \\cD(\\pi_{k+1},\\pi^*)] \\\\\n&\\le  \\gamma [ \\bbe_{\\xi_{\\lceil k-1 \\rceil}} [f(\\pi_k) - f(\\pi^*) + \\tfrac{\\mu}{1-\\gamma} \\cD(\\pi_k, \\pi^*) ]\n+ 2 \\varsigma_k + \\tfrac{ \\sigma_k^2}{2 \\gamma \\mu},\n\\end{align*}\nwhich, in view of Lemma~\\ref{lemma:induction_tech_sequence}\nwith $X_k =  \\bbe_{\\xi_{\\lceil k-1 \\rceil}} [f(\\pi_k) - f(\\pi^*) + \\tfrac{\\mu}{1-\\gamma} \\cD(\\pi_k, \\pi^*)$\nand $Z_k = 2 \\varsigma_k + \\tfrac{ \\sigma_k^2}{2 \\gamma \\mu}$,\nthen implies that\n\\begin{align*}\n&\\bbe_{\\xi_{\\lceil k-1 \\rceil}} [f(\\pi_k) - f(\\pi^*) + \\tfrac{\\mu}{1-\\gamma} \\cD(\\pi_k, \\pi^*)\\\\\n&\\le \\gamma^{\\lfloor k/l \\rfloor}\\left [f(\\pi_0) - f(\\pi^*) + \\tfrac{\\mu  \\cD(\\pi_0, \\pi^*)}{1-\\gamma}\n+ \\tfrac{5} {4}( \\tfrac{2}{1-\\gamma}+ \\tfrac{ 1}{2 \\gamma(1 -\\gamma) \\mu})\\right]\\\\\n&\\le \\gamma^{\\lfloor k/l \\rfloor}\\left [f(\\pi_0) - f(\\pi^*) +\\tfrac{1}{1-\\gamma}( \\mu  \\log |\\cA|\n+ \\tfrac{5} {2} + \\tfrac{ 5}{8 \\gamma \\mu})\\right].\n\\end{align*}\n\\end{proof}","sha256_utf8":"7ba594e9c5afc38b9da47782d12f32cf477417d1dd35ced75238295e5eb65225"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"随机策略镜像下降线性收敛","source_units":[{"start":74065,"end":74709,"start_line":1517,"end_line":1531,"raw_tex":"\\begin{theorem} \\label{theorem:SPMD_reg}\nSuppose that \n$\\eta_k = \\eta = \\tfrac{1-\\gamma}{\\gamma \\mu}$\nin the SPMD method. \nIf $\\varsigma_k =2^{-(\\lfloor k/l \\rfloor+2)}$ and $\\sigma_k^2 = 2^{-(\\lfloor k/l \\rfloor+2)}$ \nfor any $k \\ge 0$ with $l :=\\left \\lceil  \\log_\\gamma (1/4) \\right \\rceil\n$,\nthen \n\\begin{align}\n& \\bbe_{\\xi_{\\lceil k-1 \\rceil}} [f(\\pi_{k}) - f(\\pi^*) \n+ \\tfrac{\\mu}{1-\\gamma} \\cD(\\pi_{k},\\pi^*)]\\nn \\\\\n&\\le 2^{-\\lfloor k/l \\rfloor}\\left [f(\\pi_0) - f(\\pi^*) +\\tfrac{1}{1-\\gamma}( \\mu  \\log |\\cA|\n+ \\tfrac{5} {2} + \\tfrac{ 5}{8 \\gamma \\mu})\\right]. \\label{eq:SPMD_reg_MDP_results_uniform_constant1}\n\\end{align}\n\\end{theorem}","sha256_utf8":"d69d958218b745992d9289db9d7849ec217e1a75603ec29f37e34f267c5c8703"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"随机线性收敛完整证明","source_units":[{"start":74711,"end":75824,"start_line":1533,"end_line":1551,"raw_tex":"\\begin{proof}\nBy Lemma~\\ref{prop:SPMD_regularized_MDPs} and the selection of $\\eta$, we have\n\\begin{align*}\n& \\bbe_{\\xi_{\\lceil k \\rceil}} [f(\\pi_{k+1}) - f(\\pi^*) + \\tfrac{\\mu}{1-\\gamma}  \\cD(\\pi_{k+1},\\pi^*)] \\\\\n&\\le  \\gamma [ \\bbe_{\\xi_{\\lceil k-1 \\rceil}} [f(\\pi_k) - f(\\pi^*) + \\tfrac{\\mu}{1-\\gamma} \\cD(\\pi_k, \\pi^*) ]\n+ 2 \\varsigma_k + \\tfrac{ \\sigma_k^2}{2 \\gamma \\mu},\n\\end{align*}\nwhich, in view of Lemma~\\ref{lemma:induction_tech_sequence}\nwith $X_k =  \\bbe_{\\xi_{\\lceil k-1 \\rceil}} [f(\\pi_k) - f(\\pi^*) + \\tfrac{\\mu}{1-\\gamma} \\cD(\\pi_k, \\pi^*)$\nand $Z_k = 2 \\varsigma_k + \\tfrac{ \\sigma_k^2}{2 \\gamma \\mu}$,\nthen implies that\n\\begin{align*}\n&\\bbe_{\\xi_{\\lceil k-1 \\rceil}} [f(\\pi_k) - f(\\pi^*) + \\tfrac{\\mu}{1-\\gamma} \\cD(\\pi_k, \\pi^*)\\\\\n&\\le \\gamma^{\\lfloor k/l \\rfloor}\\left [f(\\pi_0) - f(\\pi^*) + \\tfrac{\\mu  \\cD(\\pi_0, \\pi^*)}{1-\\gamma}\n+ \\tfrac{5} {4}( \\tfrac{2}{1-\\gamma}+ \\tfrac{ 1}{2 \\gamma(1 -\\gamma) \\mu})\\right]\\\\\n&\\le \\gamma^{\\lfloor k/l \\rfloor}\\left [f(\\pi_0) - f(\\pi^*) +\\tfrac{1}{1-\\gamma}( \\mu  \\log |\\cA|\n+ \\tfrac{5} {2} + \\tfrac{ 5}{8 \\gamma \\mu})\\right].\n\\end{align*}\n\\end{proof}","sha256_utf8":"7ba594e9c5afc38b9da47782d12f32cf477417d1dd35ced75238295e5eb65225"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"固定步长得到误差收缩递推","source_units":[{"start":74711,"end":75102,"start_line":1533,"end_line":1539,"raw_tex":"\\begin{proof}\nBy Lemma~\\ref{prop:SPMD_regularized_MDPs} and the selection of $\\eta$, we have\n\\begin{align*}\n& \\bbe_{\\xi_{\\lceil k \\rceil}} [f(\\pi_{k+1}) - f(\\pi^*) + \\tfrac{\\mu}{1-\\gamma}  \\cD(\\pi_{k+1},\\pi^*)] \\\\\n&\\le  \\gamma [ \\bbe_{\\xi_{\\lceil k-1 \\rceil}} [f(\\pi_k) - f(\\pi^*) + \\tfrac{\\mu}{1-\\gamma} \\cD(\\pi_k, \\pi^*) ]\n+ 2 \\varsigma_k + \\tfrac{ \\sigma_k^2}{2 \\gamma \\mu},\n\\end{align*}\n","sha256_utf8":"ea3fcd078447bceed29de52e4a47f5b66a33ea1db6adcc1f6229ef78cf4b272e"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"分期递推与初始距离界","source_units":[{"start":75102,"end":75824,"start_line":1540,"end_line":1551,"raw_tex":"which, in view of Lemma~\\ref{lemma:induction_tech_sequence}\nwith $X_k =  \\bbe_{\\xi_{\\lceil k-1 \\rceil}} [f(\\pi_k) - f(\\pi^*) + \\tfrac{\\mu}{1-\\gamma} \\cD(\\pi_k, \\pi^*)$\nand $Z_k = 2 \\varsigma_k + \\tfrac{ \\sigma_k^2}{2 \\gamma \\mu}$,\nthen implies that\n\\begin{align*}\n&\\bbe_{\\xi_{\\lceil k-1 \\rceil}} [f(\\pi_k) - f(\\pi^*) + \\tfrac{\\mu}{1-\\gamma} \\cD(\\pi_k, \\pi^*)\\\\\n&\\le \\gamma^{\\lfloor k/l \\rfloor}\\left [f(\\pi_0) - f(\\pi^*) + \\tfrac{\\mu  \\cD(\\pi_0, \\pi^*)}{1-\\gamma}\n+ \\tfrac{5} {4}( \\tfrac{2}{1-\\gamma}+ \\tfrac{ 1}{2 \\gamma(1 -\\gamma) \\mu})\\right]\\\\\n&\\le \\gamma^{\\lfloor k/l \\rfloor}\\left [f(\\pi_0) - f(\\pi^*) +\\tfrac{1}{1-\\gamma}( \\mu  \\log |\\cA|\n+ \\tfrac{5} {2} + \\tfrac{ 5}{8 \\gamma \\mu})\\right].\n\\end{align*}\n\\end{proof}","sha256_utf8":"203b628894c09986402064baabefc5e9cc2fb0543aa1e46a0c3e61b3e47b0ef8"}]},{"id":"S005","semantic_type":"problem_setting","kind":"problem_setting","title":"有限折扣决策过程与策略","source_units":[{"start":4725,"end":5377,"start_line":150,"end_line":161,"raw_tex":"Consider the finite Markov decision process\n$M = (\\cS, \\cA, \\cP, c, \\gamma)$, where $\\cS$\nis a finite state space, $\\cA$ is a finite action space,\n$P: \\cS \\times \\cS \\times \\cA \\to \\bbr$ is transition \nmodel, $c: \\cS \\times \\cA \\to \\bbr$ is the cost function,\nand $\\gamma \\in (0,1)$ is the discount factor.\n%Observe that here we use the cost function \n%$c(s_t, a_t)$, rather than the usual reward function $r(s_t, a_t)$,\n%in order to facilitate the use of certain terminology (e.g., mirror descent)\n%from the optimization literature.\nA policy $\\pi: \\cA \\times \\cS \\to \\bbr$ determines\nthe probability of selecting a particular action at a given state.\n","sha256_utf8":"816aa9ab76dd52f2ea5c87c381bda019b68c206e8adf153e37c85e1b62f786d7"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"正则化动作价值函数","source_units":[{"start":5378,"end":5788,"start_line":163,"end_line":171,"raw_tex":"For a given policy $\\pi$,\nwe measure its performance by the action-value\nfunction ($Q$-function) $Q^\\pi: \\cS \\times \\cA \\to \\bbr$\ndefined as\n\\begin{align}\nQ^\\pi(s,a) &:=\n\\bbe\\left[\\tsum_{t=0}^\\infty \\gamma^t [c(s_t, a_t) + h^\\pi(s_t)] \\right. \\nn \\\\\n&\\quad \\quad\\quad\\quad \\left. \\mid s_0 = s, a_0 = a, a_t \\sim \\pi(\\cdot | s_t), s_{t+1} \\sim \\cP(\\cdot | s_t, a_t)\\right]. \\label{eq:def_Q_function}\n\\end{align}","sha256_utf8":"d92c254a530ed4654d1eab9dc59c2909ce1881ce6c82fd4a8aaa2cfaa5b11bdd"}]},{"id":"S007","semantic_type":"assumption","kind":"assumption","title":"正则项相对强凸条件","source_units":[{"start":5789,"end":6051,"start_line":172,"end_line":176,"raw_tex":"Here $h^\\pi$ is a closed convex function w.r.t. the policy $\\pi$, i.e., there exist some\n$ \\mu \\ge 0$ s.t.\n\\beq \\label{eq:convex_obj}\nh^\\pi(s) - [h^{\\pi'}(s) + \\langle  (h')^{\\pi'}(s, \\cdot), \\pi(\\cdot|s ) -  \\pi'(\\cdot|s )\\rangle ] \\ge \\mu D_{\\pi'}^\\pi(s),\n\\eeq","sha256_utf8":"bf95dd804f3e9d08c9120278c62d2747bfe8e85e438b90c504b85f0180c21a3c"},{"start":6052,"end":6385,"start_line":177,"end_line":181,"raw_tex":"where $\\langle \\cdot, \\cdot \\rangle$ denotes\nthe inner product over the action space $\\cA$, $(h')^{\\pi'}(s, \\cdot)$ \ndenotes a subgradient of $h(s)$ at $\\pi'$, and $D_{\\pi'}^\\pi(s)$ is the \nBregman's distance or Kullback–Leibler (KL) divergence  between $\\pi$ and $\\pi'$ (see Subsection~\\ref{sec:notations_PMD} for\nmore discussion).\n","sha256_utf8":"9bb43de8248d1630766ec823f8e12cd34d443af75a4299b07facd908bb00dce6"},{"start":7135,"end":7361,"start_line":193,"end_line":196,"raw_tex":"Throughout this paper, we say that $h^\\pi$ is a strongly convex regularizer if $\\mu > 0$.\nOtherwise, we call $h^\\pi$ a general convex regularizer. Clearly\nthe latter class of problems covers\nthe regular case with $h^\\pi = 0$.\n","sha256_utf8":"c6dc768e473fb61b00f1673b12efd895e7919ff8ad636c06e2f44b5691f5f1ea"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"状态价值函数","source_units":[{"start":7362,"end":7710,"start_line":198,"end_line":210,"raw_tex":"We define the state-value function\n$V^\\pi: \\cS \\to \\bbr$ associated with $\\pi$\nas\n\\begin{align} \nV^\\pi(s) &:=\n\\bbe\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [c(s_t, a_t) + h^\\pi(s_t)] \\right.\\nn \\\\\n & \\quad \\quad \\quad \\quad \\left. \\mid\ns_0 = s, a_t \\sim \\pi(\\cdot | s_t),\ns_{t+1} \\sim \\cP(\\cdot | s_t, a_t)\n\\right].\\label{eq:def_V_function}\n\\end{align}","sha256_utf8":"316b4e1cc0268f9ef48ab3b8b1961337d1d36c832bad74bbbe8a664fcad7ee10"}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","title":"状态价值与动作价值关系","source_units":[{"start":7711,"end":8037,"start_line":211,"end_line":216,"raw_tex":"It can be easily seen \nfrom the definitions of $Q^\\pi$ and $V^\\pi$ that\n\\begin{align}\nV^\\pi(s) &= \\tsum_{a \\in \\cA} \\pi( a | s) Q^\\pi(s, a)  = \\langle Q^\\pi(s, \\cdot),  \\pi( \\cdot | s) \\rangle, \\label{eq:QV1}  \\\\\nQ^\\pi(s, a) &= c(s, a) + h^\\pi(s) + \\gamma \\tsum_{s' \\in \\cS} \\cP(s'| s, a) V^\\pi(s'). \\label{eq:QV2}\n\\end{align}","sha256_utf8":"b938b67dc6ab0a44fab1d550f120658912db3ff02f93e85d48421420e1206287"}]},{"id":"S010","semantic_type":"problem_setting","kind":"problem_setting","title":"最优策略与平稳加权目标","source_units":[{"start":8038,"end":8253,"start_line":217,"end_line":220,"raw_tex":"The main objective in RL is to find an optimal policy $\\pi^*: \\cS \\times \\cA \\to \\bbr$ s.t.\n\\beq \\label{eq:opt_objective}\nV^{\\pi^*}(s) \\le V^\\pi(s), \\forall \\pi(\\cdot | s) \\in \\Delta_{|\\cA|}, \\forall s \\in \\cS.\n\\eeq","sha256_utf8":"30d5770e26894e9ec91caa2ab56a10e1c5e397da3edb83711c20452c36c35875"},{"start":8255,"end":8476,"start_line":221,"end_line":225,"raw_tex":"for any $s \\in \\cS$. Here $\\Delta_{|\\cA|}$ denotes the \nsimplex constraint given by\n\\beq \\label{eq:def_simplex}\n \\Delta_{|\\cA|} := \\{ p \\in \\bbr^{|\\cA|} | \\tsum_{i =1}^{|\\cA|} p_i = 1, p_i \\ge 0\\}, \\forall s \\in \\cS.\n\\eeq","sha256_utf8":"2a7f6e8d5c3d4ae4c080ec4355e1940a89f044408ae571ae9fbc1583fe6c9d0f"},{"start":8478,"end":9128,"start_line":227,"end_line":238,"raw_tex":"By examining Bellman's optimality condition\nfor dynamic programming (\\cite{BellmanDreyfus1959} and Chapter 6 of \\cite{PutermanBook1994}), we can show the existence\nof a policy $\\pi^*$ which satisfies \\eqnok{eq:opt_objective} simultaneously \nfor all $s \\in \\cS$. Hence, we can formulate \\eqnok{eq:opt_objective}\nas an optimization problem with a single objective by taking the weighted sum of\n$V^\\pi$ over $s$ (with weights $\\rho_s >0$ and $\\tsum_{s \\in S} \\rho_s = 1$):\n\\beq \\label{eq:opt_objective1}\n\\begin{array}{ll}\n\\min_\\pi &  \\bbe_{s \\sim \\rho} [V^{\\pi}(s)] \\\\\n\\mbox{s.t.} & \\pi(\\cdot | s) \\in \\Delta_{|\\cA|}, \\forall s \\in \\cS.\n\\end{array}\n\\eeq","sha256_utf8":"bce9937c0896c2144f15675ec5d854e721792b7c7a951c5e9b2a04123ef53396"},{"start":9129,"end":9588,"start_line":239,"end_line":248,"raw_tex":"While the weights $\\rho$ can be arbitrarily chosen, a reasonable selection\nof $\\rho$ would be the stationary state distribution\ninduced by the optimal policy $\\pi^*$, denoted by $\\nu^* \\equiv \\nu(\\pi^*)$. As such,\nproblem~\\eqnok{eq:opt_objective1} reduces to \n\\beq \\label{eq:MDP_OPT}\n\\begin{array}{ll}\n\\min_\\pi & \\left\\{ f(\\pi) := \\bbe_{s \\sim \\nu^*} [V^{\\pi}(s)] \\right\\}\\\\\n\\mbox{s.t.} & \\pi(\\cdot | s) \\in \\Delta_{|\\cA|}, \\forall s \\in \\cS.\n\\end{array}\n\\eeq","sha256_utf8":"7071125284363431f9ececeef1ff3383ddba13ce15449375b02dfcbd1c8bae6c"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"熵距离与KL散度","source_units":[{"start":17315,"end":17548,"start_line":366,"end_line":370,"raw_tex":"For any two points $\\pi(\\cdot|s), \\pi'(\\cdot|s) \\in \\Delta_{|\\cA|}$,\nwe measure their Kullback–Leibler (KL)  divergence by\n\\[\n\\KL(\\pi(\\cdot|s) \\parallel \\pi'(\\cdot|s)) = \\tsum_{a \\in \\cA} \\pi(a|s) \\log \\tfrac{\\pi(a|s)}{\\pi'(a|s)}.\n\\]","sha256_utf8":"681ec8113666fadc87d5290c575a4d7077c85fa5066a12550b3c5c001a09b149"},{"start":17549,"end":18491,"start_line":371,"end_line":384,"raw_tex":"Observe that the KL divergence can be viewed as\nis a special instance of the \nBregman's distance (or prox-function)\nwidely used in the optimization literature. \nLet the distance generating function $\\omega(\\pi(\\cdot|s)) := \\tsum_{a \\in \\cA} \\pi(a|s) \\log \\pi(a|s)$~\\footnote{It is worth noting that\nwe do not enforce $\\pi(a|s) > 0$ when defining $\\omega(\\pi(\\cdot|s))$ as all the search points generated by our algorithms\nwill satisfy this assumption.}. The Bregman's distance\nassociated with $\\omega$ is given by\n\\begin{align}\nD_{\\pi'}^\\pi(s) &:= \\omega(\\pi(\\cdot|s)) - [\\omega(\\pi'(\\cdot|s)) + \\langle \\nabla \\omega(\\pi'(\\cdot|s)), \\pi(\\cdot|s) - \\pi'(\\cdot|s)\\rangle ] \\nn\\\\\n&= \\tsum_{a \\in \\cA} \\left[ \\pi(a|s) \\log \\pi(a|s) - \\pi'(a|s) \\log \\pi'(a|s) \\right. \\nn\\\\\n& \\quad \\quad \\quad \\quad \\quad \\left. - (1 + \\log \\pi'(a|s)) (\\pi(a|s)- \\pi'(a|s))\\right] \\nn\\\\\n&= \\tsum_{a \\in \\cA} \\pi(a|s) \\log \\tfrac{\\pi(a|s)}{\\pi'(a|s)},\n\\end{align}","sha256_utf8":"38c252f9b6b903f2571d87676fab367466b74909134889a67711ba43089ebb52"},{"start":18492,"end":18755,"start_line":385,"end_line":387,"raw_tex":"where the last equation follows from the fact that $\\tsum_{a \\in \\cA}(\\pi(a|s)- \\pi'(a|s)) = 0$.\nTherefore, we will use the KL divergence $\\KL(\\pi(\\cdot|s) \\parallel \\pi'(\\cdot|s))$\nand Bregman's distance $D_{\\pi'}^\\pi(s)$ interchangeably throughout this paper. \n","sha256_utf8":"d54eac2545890b4358f9cbc5f33de64489ad5e611e0a40986c12961500f642c5"}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","title":"梯度计算中的临时可微假设","source_units":[{"start":19776,"end":19880,"start_line":403,"end_line":403,"raw_tex":"For simplicity, we assume for now that $h^\\pi$ is differentiable and will relax this assumption later. \n","sha256_utf8":"c99d7f8ae6c24d9d87c3d8338dd4f13392d381ac563e4e31390a2c4d067777d8"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"折扣访问分布与转移矩阵","source_units":[{"start":19880,"end":20082,"start_line":404,"end_line":407,"raw_tex":"For a given policy $\\pi$, we define the discounted state visitation distribution by\n\\beq \\label{eq:def_visitation}\nd_{s_0}^\\pi(s) := (1 - \\gamma) \\tsum_{t=0}^\\infty \\gamma^t \\Pr^\\pi(s_t = s | s_0),\n\\eeq","sha256_utf8":"8c0dd5273b02b1f04bff0870d38a42bcf7ba9ed8fd28e256a32776b850632ced"},{"start":20083,"end":20595,"start_line":408,"end_line":415,"raw_tex":"where $\\Pr^\\pi(s_t = s | s_0)$ denotes the state visitation probability of $s_t =s$ after we follow\nthe policy $\\pi$ starting at state $s_0$. Let $\\cP^\\pi$ denote the transition probability\nmatrix associated with policy $\\pi$, i.e., $\\cP^\\pi(i,j) = \\tsum_{a \\in \\cA} \\pi(a | i) \\cP(j| i, a)$,\nand  $e_i$ be the $i$-th unit vector.\nThen $\\Pr^\\pi(s_t = s | s_0) = e_{s_0}^T(\\cP^\\pi)^t e_s$ and\n\\beq \\label{eq:def_visitation1}\nd_{s_0}^\\pi(s) = (1 - \\gamma) \\tsum_{t=0}^\\infty \\gamma^t e_{s_0}^T(\\cP^\\pi)^t e_s.\n\\eeq","sha256_utf8":"e4d4e48e9b36dfa52fcbb679ce070e50036bf6f7d490f2b0684b073033f99482"}]},{"id":"S014","semantic_type":"problem_setting","kind":"problem_setting","title":"非光滑正则化变分不等式","source_units":[{"start":24420,"end":24582,"start_line":499,"end_line":501,"raw_tex":"\\beq\\label{eq:VI_MDP}\n\\bbe_{s \\sim \\nu^*} \\left[ \\langle Q^{\\pi^*}(s,\\cdot), \\pi(\\cdot |s) - \\pi^*(\\cdot |s)\\rangle + h^\\pi(s) - h^{\\pi^*}(s) \\right]  \\ge 0.\n\\eeq","sha256_utf8":"0dc37a5947bd5c1526db2ac47e5cff34094014c9c30fecc442d5bae951df6c91"}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","title":"均匀初始化与距离上界","source_units":[{"start":32383,"end":32453,"start_line":684,"end_line":684,"raw_tex":" For the sake of simplicity, we will assume throughout the paper that\n","sha256_utf8":"af09c8d9e3e72fcb5e7d8b815cb8e9219cd672ef5864934558c8d59ef1ed30ad"},{"start":32454,"end":32551,"start_line":685,"end_line":687,"raw_tex":"\\beq \\label{eq:initial_p0}\n \\pi_0(a | s) = 1/|\\cA|, \\ \\forall a \\in \\cA, \\forall s \\in \\cS.\n \\eeq","sha256_utf8":"6b8f9b0ad89861657d22329c87c536fac874845661247b4992465a9ea6aea7bf"},{"start":32552,"end":32748,"start_line":688,"end_line":691,"raw_tex":"In this case, we have \n\\beq \\label{eq:boundnessofdomain}\nD_{\\pi_0}^\\pi(s) = \\tsum_{a \\in \\cA} \\pi(a | s) \\log \\pi(a| s) + \\log |\\cA| \\le \\log |\\cA|, \\ \\forall \\pi(\\cdot|s) \\in \\Delta_{|\\cA|}.\n\\eeq","sha256_utf8":"aa0bd8f2b306e444d323cb6d751a6d3f2ba5a97e64f7a4a0c444fdf5a55af269"}]},{"id":"S016","semantic_type":"algorithm","kind":"algorithm","title":"确定性策略镜像下降更新","source_units":[{"start":33046,"end":33515,"start_line":697,"end_line":709,"raw_tex":"\\begin{algorithm}[H]\n\\caption{The policy mirror descent (PMD) method}\n\\begin{algorithmic}\n\\State {\\bf Input:} initial points $\\pi_0$ and stepsizes $\\eta_k\\ge 0$.\n\\For {$k =0,1,\\ldots,$}\n%\\State  \n\\beq \\label{eq:PMD_step}\n\\pi_{k+1}(\\cdot|s) = \\argmin_{p(\\cdot|s) \\in \\Delta_{|\\cA|}} \\left\\{ \\eta_k[ \\langle Q^{\\pi_{k}}(s, \\cdot), p(\\cdot|s) \\rangle + h^{p}(s)] + D_{\\pi_k}^p(s)\\right\\},\n\\forall s \\in \\cS.\n\\eeq\n\\EndFor\n\\end{algorithmic} \\label{basic_pmd}\n\\end{algorithm}","sha256_utf8":"d36d2cfec2de4a4c7ac5283f5f9530761bbd95fc65a6678617c267fa6d4ecc5e"}]},{"id":"S017","semantic_type":"definition","kind":"definition","title":"平稳加权策略距离","source_units":[{"start":39626,"end":39734,"start_line":833,"end_line":835,"raw_tex":"\\beq \\label{eq:def_distance_measure}\n\\cD (\\pi_k, \\pi^*) := \\bbe_{s\\sim \\nu^*} [D_{\\pi_{k}}^{\\pi^*}(s)].\n\\eeq","sha256_utf8":"7c3b7940426e7aeedb81529778c989ea176c34ee45e56670036cd9c1110e1220"}]},{"id":"S018","semantic_type":"assumption","kind":"assumption","title":"随机估计偏差与均方误差","source_units":[{"start":65211,"end":65624,"start_line":1348,"end_line":1354,"raw_tex":"In this subsection, we assume that for a given policy $\\pi_k$,\nthere exists a stochastic estimator $\\cQ^{\\pi_k,\\xi_k}$ s.t.\n\\begin{align}\n \\bbe_{\\xi_k} [\\cQ^{\\pi_k,\\xi_k}] &= \\bar \\cQ^{\\pi_k}, \\label{eq:expectation_ass} \\\\\n\\bbe_{\\xi_k}[\\|\\cQ^{\\pi_k,\\xi_k} - Q^{\\pi_k} \\|_\\infty^2] &\\le \\sigma_k^2, \\label{eq:variance_ass}\\\\\n\\|\\bar \\cQ^{\\pi_k} - Q^{\\pi_k}\\|_\\infty &\\le \\varsigma_k, \\label{eq:bias_ass}\n\\end{align}","sha256_utf8":"9e276b2f32b2551ac93ee53a89c7aa536bc162dfbbcd81a78a74b118b0146e5a"},{"start":65625,"end":65856,"start_line":1355,"end_line":1358,"raw_tex":"for some $\\sigma_k\\ge$ and $\\varsigma_k \\ge 0$,\nwhere $\\xi_k$ denotes the random vector used to generate the stochastic\nestimator $\\cQ^{\\pi_k,\\xi_k}$.  Clearly, if $\\sigma_k = 0$, then\nwe have exact information about $Q^{\\pi_k}$. \n","sha256_utf8":"d0f7a295b66f49564e6f29ba44c908c8b3f9c4aa9d4fd30a74ed713fbb411371"}]},{"id":"S019","semantic_type":"algorithm","kind":"algorithm","title":"随机镜像更新与噪声记号","source_units":[{"start":66503,"end":66876,"start_line":1369,"end_line":1374,"raw_tex":"The stochastic policy mirror descent (SPMD)\nis obtained by replacing $Q^{\\pi_k}$ in \\eqnok{eq:PMD_step} \nwith its stochastic estimator $\\cQ^{\\pi_k,\\xi_k}$, i.e.,\n\\beq \\label{eq:SPMD_step}\n\\pi_{k+1}(\\cdot|s) = \\argmin_{p(\\cdot|s) \\in \\Delta_{|\\cA|}} \\left\\{ \\Phi_k(p):=\\eta_k [\\langle \\cQ^{\\pi_{k},\\xi_k}(s, \\cdot), p(\\cdot|s) \\rangle + h^p(s)]+ D_{\\pi_k}^p(s)\\right\\}.\n\\eeq","sha256_utf8":"77463837fa9cc1fe019792382601cb92b1ea393a817b8bf5f348f065365a8e0e"},{"start":66877,"end":66991,"start_line":1375,"end_line":1375,"raw_tex":"In the sequel, we denote $\\xi_{\\lceil k \\rceil}$ the sequence of random vectors $\\xi_0, \\ldots, \\xi_k$ and define\n","sha256_utf8":"33af0e0a9a5e5751ea2e9d608d122d34cae60c6535f89a0f302e7aafcd1027a4"},{"start":66992,"end":67070,"start_line":1376,"end_line":1378,"raw_tex":"\\beq \\label{eq:def_SPMD_delta}\n\\delta_k := \\cQ^{\\pi_k,\\xi_k} - Q^{\\pi_k}.\n\\eeq","sha256_utf8":"269d22f6629fc32aa13ee469529d31b77b2fc09b753a183c7bd735946c8bca19"}]},{"id":"S020","semantic_type":"assumption","kind":"assumption","title":"强凸随机收敛适用范围","source_units":[{"start":73787,"end":74064,"start_line":1512,"end_line":1515,"raw_tex":"We are now ready to establish the convergence rate of the SPMD method.  \nWe start with the case when $\\mu > 0$ and state a constant stepsize rule which requires both\n$\\varsigma_k$ and $\\sigma_k$, $k \\ge 0$, to be small enough to guarantee the convergence of \nthe SPMD method. \n","sha256_utf8":"4f77b7c13fc89ae07bffe82a650e32d66d4b6ea0f0f63e5d8cbb0ad5b7d7da40"}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","title":"价值函数策略梯度公式","source_units":[{"start":20604,"end":20933,"start_line":419,"end_line":429,"raw_tex":"\\begin{lemma} \\label{lemma:gradient}\nFor any $(s_0, s, a) \\in \\cS \\times \\cS \\times \\cA$, we have\n\\[\n\\tfrac{\\partial V^\\pi(s_0)}{\\partial \\pi(a|s)}\n= \\tfrac{1}{1-\\gamma} d_{s_0}^\\pi(s)\n\\left[ \nQ^\\pi(s,a) +  \\nabla h^\\pi(s, a) \n\\right],\n\\]\nwhere $\\nabla h^\\pi(s,\\cdot)$ denotes the gradient of $h^\\pi(s)$ w.r.t. $\\pi$.\n\\end{lemma}","sha256_utf8":"9e3bafb268b471a66e733c840a3f6e08fadcf9b4dd1eb4ffb4cdd5f02289717e"}]},{"id":"S022","semantic_type":"proof","kind":"proof","title":"策略梯度公式完整证明","source_units":[{"start":20935,"end":22993,"start_line":431,"end_line":472,"raw_tex":"\\begin{proof}\nIt follows from \\eqnok{eq:QV1} that\n\\begin{align*}\n\\tfrac{\\partial V^\\pi(s_0)}{\\partial \\pi(a|s)}\n&= \\tfrac{\\partial}{\\partial \\pi(a'|s)} \\tsum_{a'\\in \\cA} \\pi(a'|s_0) Q^\\pi(s_0, a')\\\\\n&=  \\tsum_{a' \\in \\cA} \n\\left[\n\\tfrac{\\partial \\pi(a'|s_0)}{\\partial \\pi(a|s)} Q^\\pi(s_0, a')\n+ \\pi(a'|s_0) \n\\tfrac{\\partial Q^\\pi(s_0, a')}{\\partial \\pi(a|s)}\n\\right].\n\\end{align*}\nAlso the relation in \\eqnok{eq:QV2}\nimplies that\n\\begin{align*}\n\\tfrac{\\partial Q^\\pi(s_0, a')}{\\partial \\pi(a|s)}\n= \\tfrac{  \\partial h^\\pi(s_0)}{\\partial \\pi (a|s)} +\n\\gamma \\tsum_{s' \\in \\cS} \n\\cP(s'| s_0, a') \\tfrac{\\partial V^\\pi(s')}{\\partial \\pi(a|s)}.\n\\end{align*}\nCombining the above two relations, we obtain\n\\begin{align*}\n\\tfrac{\\partial V^\\pi(s_0)}{\\partial \\pi(a|s)}\n&= \\tsum_{a' \\in \\cA} \\left[ \\tfrac{\\partial \\pi(a'|s_0)}{\\partial \\pi(a|s)} Q^\\pi(s_0, a')\n+ \\pi(a'|s_0)  \\tfrac{  \\partial h^\\pi(s_0)}{\\partial \\pi (a|s)}\\right]  \\\\\n& \\quad + \\gamma \\tsum_{a' \\in \\cA} \\pi(a'|s_0) \\tsum_{s' \\in \\cS} \n\\cP(s'| s_0, a') \\tfrac{\\partial V^\\pi(s')}{\\partial \\pi(a|s)} \\\\\n&=\\tsum_{x\\in \\cS} \\tsum_{t=0}^\\infty \\gamma^t \\Pr^\\pi(s_t = x | s_0) \\\\\n&\\quad \\quad  \\tsum_{a' \\in \\cA} \\left[ \\tfrac{\\partial \\pi(a'|x)}{\\partial \\pi(a|s)} Q^\\pi(x, a')\n+ \\pi(a'|x)  \\tfrac{   \\partial h^\\pi(x)}{\\partial \\pi (a|s)}\\right]\\\\\n&=\\tfrac{1}{1-\\gamma} \\tsum_{x\\in \\cS} d_{s_0}^\\pi(x) \\left\\{\n\\tsum_{a' \\in \\cA} \\left[ \\tfrac{\\partial \\pi(a'|x)}{\\partial \\pi(a|s)} Q^\\pi(x, a')\\right]\n+ \\tfrac{   \\partial h^\\pi(x)}{\\partial \\pi (a|s)}\\right\\}\\\\\n&= \\tfrac{1}{1-\\gamma} d_{s_0}^\\pi(s) \\left[ Q^\\pi(s, a) + \\tfrac{   \\partial h^\\pi(s)}{\\partial \\pi (a|s)}\\right],\n\\end{align*}\nwhere the second equality follows by\nexpanding $\\tfrac{\\partial V^\\pi(s')}{\\partial \\pi(a|s)}$ recursively,\nand the third equality follows from the definition of $d_{s_0}^\\pi(s)$\nin \\eqnok{eq:def_visitation}, and the last identity follows from $\\tfrac{\\partial \\pi(a'|x)}{\\partial \\pi(a|s)} =0$\nfor $x \\neq s$ or $a' \\neq a$, and $\\tfrac{\\partial h^\\pi(x)}{\\partial \\pi (a|s)} =0$\nfor $x \\neq s$.\n\\end{proof}","sha256_utf8":"6eef8de53b13c87187827471487f830d5a349ad509ef73a132e59570e773527a"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"状态价值求导分解","source_units":[{"start":20935,"end":21316,"start_line":431,"end_line":442,"raw_tex":"\\begin{proof}\nIt follows from \\eqnok{eq:QV1} that\n\\begin{align*}\n\\tfrac{\\partial V^\\pi(s_0)}{\\partial \\pi(a|s)}\n&= \\tfrac{\\partial}{\\partial \\pi(a'|s)} \\tsum_{a'\\in \\cA} \\pi(a'|s_0) Q^\\pi(s_0, a')\\\\\n&=  \\tsum_{a' \\in \\cA} \n\\left[\n\\tfrac{\\partial \\pi(a'|s_0)}{\\partial \\pi(a|s)} Q^\\pi(s_0, a')\n+ \\pi(a'|s_0) \n\\tfrac{\\partial Q^\\pi(s_0, a')}{\\partial \\pi(a|s)}\n\\right].\n\\end{align*}\n","sha256_utf8":"fb5d8c87bbe96ab8535f327f3e4aa0eccd70cfe939220454c07dcfd5e843629a"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"动作价值导数递推","source_units":[{"start":21316,"end":21589,"start_line":443,"end_line":450,"raw_tex":"Also the relation in \\eqnok{eq:QV2}\nimplies that\n\\begin{align*}\n\\tfrac{\\partial Q^\\pi(s_0, a')}{\\partial \\pi(a|s)}\n= \\tfrac{  \\partial h^\\pi(s_0)}{\\partial \\pi (a|s)} +\n\\gamma \\tsum_{s' \\in \\cS} \n\\cP(s'| s_0, a') \\tfrac{\\partial V^\\pi(s')}{\\partial \\pi(a|s)}.\n\\end{align*}\n","sha256_utf8":"17a95e340958c8e5e74419ea455eb4d70c8931955b9c1b7dcb0f1ad1bf1dfdfd"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"递归展开为访问分布","source_units":[{"start":21589,"end":22993,"start_line":451,"end_line":472,"raw_tex":"Combining the above two relations, we obtain\n\\begin{align*}\n\\tfrac{\\partial V^\\pi(s_0)}{\\partial \\pi(a|s)}\n&= \\tsum_{a' \\in \\cA} \\left[ \\tfrac{\\partial \\pi(a'|s_0)}{\\partial \\pi(a|s)} Q^\\pi(s_0, a')\n+ \\pi(a'|s_0)  \\tfrac{  \\partial h^\\pi(s_0)}{\\partial \\pi (a|s)}\\right]  \\\\\n& \\quad + \\gamma \\tsum_{a' \\in \\cA} \\pi(a'|s_0) \\tsum_{s' \\in \\cS} \n\\cP(s'| s_0, a') \\tfrac{\\partial V^\\pi(s')}{\\partial \\pi(a|s)} \\\\\n&=\\tsum_{x\\in \\cS} \\tsum_{t=0}^\\infty \\gamma^t \\Pr^\\pi(s_t = x | s_0) \\\\\n&\\quad \\quad  \\tsum_{a' \\in \\cA} \\left[ \\tfrac{\\partial \\pi(a'|x)}{\\partial \\pi(a|s)} Q^\\pi(x, a')\n+ \\pi(a'|x)  \\tfrac{   \\partial h^\\pi(x)}{\\partial \\pi (a|s)}\\right]\\\\\n&=\\tfrac{1}{1-\\gamma} \\tsum_{x\\in \\cS} d_{s_0}^\\pi(x) \\left\\{\n\\tsum_{a' \\in \\cA} \\left[ \\tfrac{\\partial \\pi(a'|x)}{\\partial \\pi(a|s)} Q^\\pi(x, a')\\right]\n+ \\tfrac{   \\partial h^\\pi(x)}{\\partial \\pi (a|s)}\\right\\}\\\\\n&= \\tfrac{1}{1-\\gamma} d_{s_0}^\\pi(s) \\left[ Q^\\pi(s, a) + \\tfrac{   \\partial h^\\pi(s)}{\\partial \\pi (a|s)}\\right],\n\\end{align*}\nwhere the second equality follows by\nexpanding $\\tfrac{\\partial V^\\pi(s')}{\\partial \\pi(a|s)}$ recursively,\nand the third equality follows from the definition of $d_{s_0}^\\pi(s)$\nin \\eqnok{eq:def_visitation}, and the last identity follows from $\\tfrac{\\partial \\pi(a'|x)}{\\partial \\pi(a|s)} =0$\nfor $x \\neq s$ or $a' \\neq a$, and $\\tfrac{\\partial h^\\pi(x)}{\\partial \\pi (a|s)} =0$\nfor $x \\neq s$.\n\\end{proof}","sha256_utf8":"192647367df259a3fe5ec4c4b9825e33a63cf58be9d4032c5a1c4c8a4d76a4a5"}]},{"id":"S026","semantic_type":"proof","kind":"proof","title":"平稳分布梯度关系推导","source_units":[{"start":23002,"end":23977,"start_line":476,"end_line":491,"raw_tex":"In view of Lemma~\\ref{lemma:gradient}, the gradient of the objective\n $f(\\pi)$ in \\eqnok{eq:MDP_OPT} at the optimal policy $\\pi^*$  is given by\n\\begin{align}\n\\tfrac{\\partial f(\\pi^*) }{\\partial \\pi(a|s)}\n&=  \\bbe_{s_0 \\sim \\nu^*}\\left[ \\tfrac{\\partial V^{\\pi^*}(s_0)}{\\partial \\pi(a|s)}\\right]\n=  \\tfrac{1}{1-\\gamma}  \\bbe_{s_0 \\sim \\nu^*}\\left[ d_{s_0}^{\\pi^*}(s)\n[Q^{\\pi^*}(s,a) + \\nabla h^{\\pi^*}(s, a)] \\right] \\nn \\\\\n&= \\tsum_{t=0}^\\infty \\gamma^t (\\nu^*)^T (\\cP^{\\pi^*})^t e_s \\, [Q^{\\pi^*}(s,a) + \\nabla h^{\\pi^*}(s, a)] \\nn \\\\\n&= \\tfrac{1}{1-\\gamma} (\\nu^*)^T e_s \\, [Q^{\\pi^*}(s,a) + \\nabla h^{\\pi^*}(s, a)] \\nn \\\\\n&= \\tfrac{1}{1-\\gamma} \\nu^*(s) \\, [Q^{\\pi^*}(s,a) + \\nabla h^{\\pi^*}(s, a)], \\label{eq:stationary_relation}\n\\end{align}\nwhere the third identity follows from \\eqnok{eq:def_visitation1} and\nthe last one follows from \nthe fact that $(\\nu^*)^T (\\cP^{\\pi^*})^t = (\\nu^*)^T$ for any $t \\ge 0$ \n since $\\nu^*$\nis the steady state distribution of $\\pi^*$. \n","sha256_utf8":"06ad5cf425b1a3f8ec3e21decfdbbf936c2db45adeb8355c925a471fa83bf445"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"目标梯度按平稳分布平均","source_units":[{"start":23002,"end":23746,"start_line":476,"end_line":486,"raw_tex":"In view of Lemma~\\ref{lemma:gradient}, the gradient of the objective\n $f(\\pi)$ in \\eqnok{eq:MDP_OPT} at the optimal policy $\\pi^*$  is given by\n\\begin{align}\n\\tfrac{\\partial f(\\pi^*) }{\\partial \\pi(a|s)}\n&=  \\bbe_{s_0 \\sim \\nu^*}\\left[ \\tfrac{\\partial V^{\\pi^*}(s_0)}{\\partial \\pi(a|s)}\\right]\n=  \\tfrac{1}{1-\\gamma}  \\bbe_{s_0 \\sim \\nu^*}\\left[ d_{s_0}^{\\pi^*}(s)\n[Q^{\\pi^*}(s,a) + \\nabla h^{\\pi^*}(s, a)] \\right] \\nn \\\\\n&= \\tsum_{t=0}^\\infty \\gamma^t (\\nu^*)^T (\\cP^{\\pi^*})^t e_s \\, [Q^{\\pi^*}(s,a) + \\nabla h^{\\pi^*}(s, a)] \\nn \\\\\n&= \\tfrac{1}{1-\\gamma} (\\nu^*)^T e_s \\, [Q^{\\pi^*}(s,a) + \\nabla h^{\\pi^*}(s, a)] \\nn \\\\\n&= \\tfrac{1}{1-\\gamma} \\nu^*(s) \\, [Q^{\\pi^*}(s,a) + \\nabla h^{\\pi^*}(s, a)], \\label{eq:stationary_relation}\n\\end{align}","sha256_utf8":"08ad2e25596088b2083f9604a36ef155a27a54576cd21b858795f275e9f4efb7"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"利用平稳性消去转移幂","source_units":[{"start":23747,"end":23977,"start_line":487,"end_line":491,"raw_tex":"where the third identity follows from \\eqnok{eq:def_visitation1} and\nthe last one follows from \nthe fact that $(\\nu^*)^T (\\cP^{\\pi^*})^t = (\\nu^*)^T$ for any $t \\ge 0$ \n since $\\nu^*$\nis the steady state distribution of $\\pi^*$. \n","sha256_utf8":"f7cc2703221c2ef287795c2ae5d8f29c84ac9c5ad5471ba36f31c6d968390301"}]},{"id":"S029","semantic_type":"lemma","kind":"lemma","title":"正则化性能差分恒等式","source_units":[{"start":24828,"end":25195,"start_line":506,"end_line":520,"raw_tex":"\\begin{lemma} \\label{lemma_per_diff}\nFor any two feasible policies $\\pi$ and $\\pi'$, we have\n\\[\nV^{\\pi'}(s) - V^{\\pi}(s) =\n\\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi'}}\n\\left[\n\\langle A^{\\pi}(s', \\cdot), \\pi'(\\cdot | s')\\rangle\n+ h^{\\pi'}(s') - h^{\\pi}(s')\n\\right],\n\\]\nwhere\n\\beq \\label{eq:def_advantage}\nA^{\\pi}(s', a) := Q^{\\pi}(s', a)-V^{\\pi}(s').\n\\eeq\n\\end{lemma}","sha256_utf8":"03161ef7717ca928dc03128c49e268bc884f06ea39b102c12e65a4390d07ca49"}]},{"id":"S030","semantic_type":"proof","kind":"proof","title":"性能差分完整证明","source_units":[{"start":25197,"end":27362,"start_line":522,"end_line":566,"raw_tex":"\\begin{proof}\nFor simplicity, let us denote $\\xi^{\\pi'}(s_0)$ the random process $(s_t, a_t, s_{t+1})$, $t \\ge 0$,\ngenerated by following the policy $\\pi'$ starting with the initial state $s_0$.\nIt then follows from the definition of $V^{\\pi'}$ that\n\\begin{align*}\n&V^{\\pi'}(s) - V^{\\pi}(s) \\\\\n&= \\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [c(s_t, a_t) + h^{\\pi'}(s_t)] \\right]- V^{\\pi}(s)\\\\\n&=\\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [ c(s_t, a_t) + h^{\\pi'}(s_t) + V^{\\pi}(s_t)-V^{\\pi}(s_t) ] \\right]- V^{\\pi}(s)\\\\\n &\\stackrel{(a)}{=} \\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [ c(s_t, a_t) + h^{\\pi'}(s_t) + \\gamma V^{\\pi}(s_{t+1})-V^{\\pi}(s_t) ] \\right] \\\\\n&\\quad \\quad \\quad \\quad \\quad+ \\bbe_{\\xi^{\\pi'}(s)} [V^{\\pi}(s_0)] - V^{\\pi}(s)\\\\\n&\\stackrel{(b)}{=} \\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [ c(s_t, a_t) + h^{\\pi'}(s_t) + \\gamma V^{\\pi}(s_{t+1})-V^{\\pi}(s_t) ]\\right] \\\\\n &= \\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [ c(s_t, a_t) + h^{\\pi}(s_t) + \\gamma V^{\\pi}(s_{t+1})-V^{\\pi}(s_t) \\right. \\\\\n&\\quad \\quad \\quad \\quad \\quad \\quad  \\quad \\quad  \\left. + h^{\\pi'}(s_t) -h^{\\pi}(s_t) ] \\right]\\\\\n &\\stackrel{(c)}{=}\\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t \\left[  Q^{\\pi}(s_t, a_t)-V^{\\pi}(s_t)\n +h^{\\pi'}(s_t) -h^{\\pi}(s_t) \\right] \\right],\n\\end{align*}\nwhere (a) follows by taking the term $V^{\\pi}(s_0)$ outside the summation,\n(b) follows from the fact that $\\bbe_{\\xi^{\\pi'}(s)} [V^{\\pi}(s_0)] = V^{\\pi}(s)$ \nsince the random process starts with $s_0 = s$, and (c) follows from \\eqnok{eq:QV2}.\nThe previous conclusion, together with \\eqnok{eq:def_advantage} and the definition $d_s^{\\pi'}$ in \\eqnok{eq:def_visitation}, then\nimply that\n\\begin{align*}\n&V^{\\pi'}(s) - V^{\\pi}(s) \\\\\n&= \\tfrac{1}{1-\\gamma}\\tsum_{s' \\in \\cS} \\tsum_{a' \\in \\cA}\nd_s^{\\pi'}(s') \\pi'(a' | s') \\left[ A^{\\pi}(s', a') +  h^{\\pi'}(s') - h^{\\pi}(s') \\right]\\\\\n&= \\tfrac{1}{1-\\gamma} \\tsum_{s' \\in \\cS} d_s^{\\pi'}(s')\n\\left[\n\\langle A^{\\pi}(s', \\cdot), \\pi'(\\cdot | s')\\rangle\n+ h^{\\pi'}(s') - h^{\\pi}(s')\n\\right], \n\\end{align*}\nwhich immediately implies the result.\n\\end{proof}","sha256_utf8":"7708c88222a983b84704f91f1558d16b5b44f25cde95461387d5cee222579357"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"沿轨迹望远镜展开价值差","source_units":[{"start":25197,"end":26807,"start_line":522,"end_line":552,"raw_tex":"\\begin{proof}\nFor simplicity, let us denote $\\xi^{\\pi'}(s_0)$ the random process $(s_t, a_t, s_{t+1})$, $t \\ge 0$,\ngenerated by following the policy $\\pi'$ starting with the initial state $s_0$.\nIt then follows from the definition of $V^{\\pi'}$ that\n\\begin{align*}\n&V^{\\pi'}(s) - V^{\\pi}(s) \\\\\n&= \\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [c(s_t, a_t) + h^{\\pi'}(s_t)] \\right]- V^{\\pi}(s)\\\\\n&=\\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [ c(s_t, a_t) + h^{\\pi'}(s_t) + V^{\\pi}(s_t)-V^{\\pi}(s_t) ] \\right]- V^{\\pi}(s)\\\\\n &\\stackrel{(a)}{=} \\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [ c(s_t, a_t) + h^{\\pi'}(s_t) + \\gamma V^{\\pi}(s_{t+1})-V^{\\pi}(s_t) ] \\right] \\\\\n&\\quad \\quad \\quad \\quad \\quad+ \\bbe_{\\xi^{\\pi'}(s)} [V^{\\pi}(s_0)] - V^{\\pi}(s)\\\\\n&\\stackrel{(b)}{=} \\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [ c(s_t, a_t) + h^{\\pi'}(s_t) + \\gamma V^{\\pi}(s_{t+1})-V^{\\pi}(s_t) ]\\right] \\\\\n &= \\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t [ c(s_t, a_t) + h^{\\pi}(s_t) + \\gamma V^{\\pi}(s_{t+1})-V^{\\pi}(s_t) \\right. \\\\\n&\\quad \\quad \\quad \\quad \\quad \\quad  \\quad \\quad  \\left. + h^{\\pi'}(s_t) -h^{\\pi}(s_t) ] \\right]\\\\\n &\\stackrel{(c)}{=}\\bbe_{\\xi^{\\pi'}(s)}\n \\left[\n \\tsum_{t=0}^\\infty \\gamma^t \\left[  Q^{\\pi}(s_t, a_t)-V^{\\pi}(s_t)\n +h^{\\pi'}(s_t) -h^{\\pi}(s_t) \\right] \\right],\n\\end{align*}\nwhere (a) follows by taking the term $V^{\\pi}(s_0)$ outside the summation,\n(b) follows from the fact that $\\bbe_{\\xi^{\\pi'}(s)} [V^{\\pi}(s_0)] = V^{\\pi}(s)$ \nsince the random process starts with $s_0 = s$, and (c) follows from \\eqnok{eq:QV2}.\n","sha256_utf8":"e8354f0779965e768190bb2d7ba90513bc6a53c8fbb61c93acea74a698243c0a"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"改写为折扣访问期望","source_units":[{"start":26807,"end":27362,"start_line":553,"end_line":566,"raw_tex":"The previous conclusion, together with \\eqnok{eq:def_advantage} and the definition $d_s^{\\pi'}$ in \\eqnok{eq:def_visitation}, then\nimply that\n\\begin{align*}\n&V^{\\pi'}(s) - V^{\\pi}(s) \\\\\n&= \\tfrac{1}{1-\\gamma}\\tsum_{s' \\in \\cS} \\tsum_{a' \\in \\cA}\nd_s^{\\pi'}(s') \\pi'(a' | s') \\left[ A^{\\pi}(s', a') +  h^{\\pi'}(s') - h^{\\pi}(s') \\right]\\\\\n&= \\tfrac{1}{1-\\gamma} \\tsum_{s' \\in \\cS} d_s^{\\pi'}(s')\n\\left[\n\\langle A^{\\pi}(s', \\cdot), \\pi'(\\cdot | s')\\rangle\n+ h^{\\pi'}(s') - h^{\\pi}(s')\n\\right], \n\\end{align*}\nwhich immediately implies the result.\n\\end{proof}","sha256_utf8":"9ad24edab9f120672a58edefcabd7bdf3cbef29545316c4b8013b356c239b6f6"}]},{"id":"S033","semantic_type":"lemma","kind":"lemma","title":"广义单调性等式","source_units":[{"start":27481,"end":27822,"start_line":573,"end_line":583,"raw_tex":"\\begin{lemma} \\label{prop_strong_monotone}\nThe VI problem in \\eqnok{eq:VI_MDP} satisfies\n\\begin{align}\n&\\bbe_{s\\sim \\nu^*}\n\\left[\n \\langle Q^{\\pi}(s, \\cdot),  \\pi(\\cdot|s) - \\pi^*(\\cdot | s)\\rangle + h^\\pi(s) - h^{\\pi^*}(s)\n\\right] \\nn \\\\\n& \\quad \\quad =\n \\bbe_{s\\sim \\nu^*}[(1- \\gamma)(V^{\\pi}(s) - V^{\\pi^*}(s) ) ].\n\\end{align}\n\\end{lemma}","sha256_utf8":"074a0aecf318571f7b902ffc2146b3c8f3d695500c1bf64b21e47c038c51567e"}]},{"id":"S034","semantic_type":"proof","kind":"proof","title":"广义单调性完整证明","source_units":[{"start":27824,"end":29574,"start_line":585,"end_line":623,"raw_tex":"\\begin{proof}\nIt follows from Lemma~\\ref{lemma_per_diff} (with $\\pi' = \\pi^*$) that\n\\begin{align*}\n(1- \\gamma) [V^{\\pi^*}(s) - V^{\\pi}(s)] \n=\\bbe_{s' \\sim d_s^{\\pi^*}}\n\\left[\n\\langle A^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s')\\rangle\n+ h^{\\pi^*}(s') - h^{\\pi}(s')\n\\right].\n\\end{align*}\nLet $e$ denote the vector of all $1$'s. Then, we have \n\\begin{align}\n\\langle A^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s')\\rangle\n&= \\langle Q^{\\pi}(s', \\cdot) - V^\\pi(s') e, \\pi^*(\\cdot | s')\\rangle \\nn\\\\\n&= \\langle Q^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s')\\rangle - V^\\pi(s') \\nn \\\\\n&=  \\langle Q^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s')\\rangle - \\langle Q^{\\pi}(s', \\cdot), \\pi(\\cdot|s') \\nn \\\\\n&= \\langle Q^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s') - \\pi(\\cdot|s')\\rangle, \\label{eq:inner_product_equivalence}\n\\end{align}\nwhere the first identity follows from the definition of $A^{\\pi}(s', \\cdot)$ in \\eqnok{eq:def_advantage},\nthe second equality follows from the fact that $\\langle e,  \\pi^*(\\cdot | s')\\rangle = 1$,\nand the third equality follows from the definition of $V^\\pi$ in \\eqnok{eq:def_V_function}.\nCombining the above two relations and taking expectation w.r.t. $\\nu^*$, we obtain\n\\begin{align*}\n&(1- \\gamma) \\bbe_{s\\sim \\nu^*}[V^{\\pi^*}(s) - V^{\\pi}(s)] \\\\\n&= \\bbe_{s\\sim \\nu^*, s' \\sim d_s^{\\pi^*}}\n\\left[\n \\langle Q^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s') - \\pi(\\cdot|s')\\rangle\n +   h^{\\pi^*}(s') - h^{\\pi}(s')\n\\right]\\\\\n&= \\bbe_{s\\sim \\nu^*}\n\\left[\n \\langle Q^{\\pi}(s, \\cdot), \\pi^*(\\cdot | s) - \\pi(\\cdot|s)\\rangle\n +   h^{\\pi^*}(s) - h^{\\pi}(s)\n\\right],\n\\end{align*}\nwhere the second identity follows similarly to \\eqnok{eq:stationary_relation}\nsince $\\nu^*$ is the steady state distribution induced by $\\pi^*$.\nThe result then follows by rearranging the terms.\n\\end{proof}","sha256_utf8":"d33b67fb16aeb9492ef4738ce9aca30baa48146b05cdc8d328e13c2338b1eb7e"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"对最优策略应用性能差分","source_units":[{"start":27824,"end":28104,"start_line":585,"end_line":594,"raw_tex":"\\begin{proof}\nIt follows from Lemma~\\ref{lemma_per_diff} (with $\\pi' = \\pi^*$) that\n\\begin{align*}\n(1- \\gamma) [V^{\\pi^*}(s) - V^{\\pi}(s)] \n=\\bbe_{s' \\sim d_s^{\\pi^*}}\n\\left[\n\\langle A^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s')\\rangle\n+ h^{\\pi^*}(s') - h^{\\pi}(s')\n\\right].\n\\end{align*}\n","sha256_utf8":"d1045d2bda82ebc80a9dff00da6d5ab83a8d7af0dfdb539050b91c6f05121830"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"优势内积改写为策略差","source_units":[{"start":28104,"end":28896,"start_line":595,"end_line":605,"raw_tex":"Let $e$ denote the vector of all $1$'s. Then, we have \n\\begin{align}\n\\langle A^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s')\\rangle\n&= \\langle Q^{\\pi}(s', \\cdot) - V^\\pi(s') e, \\pi^*(\\cdot | s')\\rangle \\nn\\\\\n&= \\langle Q^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s')\\rangle - V^\\pi(s') \\nn \\\\\n&=  \\langle Q^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s')\\rangle - \\langle Q^{\\pi}(s', \\cdot), \\pi(\\cdot|s') \\nn \\\\\n&= \\langle Q^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s') - \\pi(\\cdot|s')\\rangle, \\label{eq:inner_product_equivalence}\n\\end{align}\nwhere the first identity follows from the definition of $A^{\\pi}(s', \\cdot)$ in \\eqnok{eq:def_advantage},\nthe second equality follows from the fact that $\\langle e,  \\pi^*(\\cdot | s')\\rangle = 1$,\nand the third equality follows from the definition of $V^\\pi$ in \\eqnok{eq:def_V_function}.\n","sha256_utf8":"2cec7460161640233ca1c4ea829a01e2c2ae6005673007c431f6b02d47eb67f8"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"平稳平均得到目标差","source_units":[{"start":28896,"end":29574,"start_line":606,"end_line":623,"raw_tex":"Combining the above two relations and taking expectation w.r.t. $\\nu^*$, we obtain\n\\begin{align*}\n&(1- \\gamma) \\bbe_{s\\sim \\nu^*}[V^{\\pi^*}(s) - V^{\\pi}(s)] \\\\\n&= \\bbe_{s\\sim \\nu^*, s' \\sim d_s^{\\pi^*}}\n\\left[\n \\langle Q^{\\pi}(s', \\cdot), \\pi^*(\\cdot | s') - \\pi(\\cdot|s')\\rangle\n +   h^{\\pi^*}(s') - h^{\\pi}(s')\n\\right]\\\\\n&= \\bbe_{s\\sim \\nu^*}\n\\left[\n \\langle Q^{\\pi}(s, \\cdot), \\pi^*(\\cdot | s) - \\pi(\\cdot|s)\\rangle\n +   h^{\\pi^*}(s) - h^{\\pi}(s)\n\\right],\n\\end{align*}\nwhere the second identity follows similarly to \\eqnok{eq:stationary_relation}\nsince $\\nu^*$ is the steady state distribution induced by $\\pi^*$.\nThe result then follows by rearranging the terms.\n\\end{proof}","sha256_utf8":"5cedc0c93d2c7b4023b3bb8b338fae7e77ccd2acc18fdd91f4933849cbd34007"}]},{"id":"S038","semantic_type":"lemma","kind":"lemma","title":"镜像更新三点不等式","source_units":[{"start":34951,"end":35282,"start_line":729,"end_line":737,"raw_tex":"\\begin{lemma} \\label{lemma:prox_optimality}\nFor any $p(\\cdot|s) \\in \\Delta_{|\\cA|}$,\nwe have\n\\begin{align*}\n&\\eta_k[ \\langle Q^{\\pi_k}(s, \\cdot), \\pi_{k+1}(\\cdot|s) - p(\\cdot|s) \\rangle + h^{\\pi_{k+1}}(s) - h^{p}(s)]\n+ D_{\\pi_k}^{\\pi_{k+1}}(s)\\\\\n& \\le D_{\\pi_k}^p(s) -  (1 + \\eta_k \\mu) D_{\\pi_{k+1}}^p(s).\n\\end{align*}\n\\end{lemma}","sha256_utf8":"67dbdd1a63a6d827059580b0d3b622c8707f8e5877152cb6c025398fec679422"}]},{"id":"S039","semantic_type":"proof","kind":"proof","title":"三点不等式完整证明","source_units":[{"start":35284,"end":36118,"start_line":739,"end_line":753,"raw_tex":"\\begin{proof}\nBy the optimality condition of \\eqnok{eq:PMD_step},\n\\[\n\\langle \\eta_k [Q^{\\pi_{k}}(s, \\cdot) + (h')^{\\pi_{k+1}}(s, \\cdot)] + \\nabla D_{\\pi_k}^{\\pi_{k+1}}(s, \\cdot),  p(\\cdot|s) - \\pi_{k+1}(\\cdot|s)  \\rangle \\ge 0,\n\\ \\ \\forall p(\\cdot|s) \\in \\Delta_{|\\cA|},\n\\]\nwhere $(h')^{\\pi_{k+1}}$ denotes the subgradient of $h$ at $\\pi_{k+1}$ and\n$\\nabla D_{\\pi_k}^{\\pi_{k+1}}(s, \\cdot)$ denotes the gradient of $D_{\\pi_k}^{\\pi_{k+1}}(s)$ at $\\pi_{k+1}$.\nUsing the definition of Bregman's  distance, it is easy\nto verify that\n\\beq \\label{prox_identity}\nD_{\\pi_k}^p(s) = D_{\\pi_k}^{\\pi_{k+1}}(s) + \\langle \\nabla D_{\\pi_k}^{\\pi_{k+1}}(s, \\cdot), p(\\cdot|s) -\\pi_{k+1}(\\cdot|s) \\rangle + D_{\\pi_{k+1}}^p(s).\n\\eeq\nThe result then immediately follows by combining the above two relations together with \\eqnok{eq:convex_obj}.\n\\end{proof}","sha256_utf8":"84742826d21e375b77bf8c08c0ebdeca899769e9e7c74def8c5066049c8c5148"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"写出近端子问题最优性","source_units":[{"start":35284,"end":35741,"start_line":739,"end_line":746,"raw_tex":"\\begin{proof}\nBy the optimality condition of \\eqnok{eq:PMD_step},\n\\[\n\\langle \\eta_k [Q^{\\pi_{k}}(s, \\cdot) + (h')^{\\pi_{k+1}}(s, \\cdot)] + \\nabla D_{\\pi_k}^{\\pi_{k+1}}(s, \\cdot),  p(\\cdot|s) - \\pi_{k+1}(\\cdot|s)  \\rangle \\ge 0,\n\\ \\ \\forall p(\\cdot|s) \\in \\Delta_{|\\cA|},\n\\]\nwhere $(h')^{\\pi_{k+1}}$ denotes the subgradient of $h$ at $\\pi_{k+1}$ and\n$\\nabla D_{\\pi_k}^{\\pi_{k+1}}(s, \\cdot)$ denotes the gradient of $D_{\\pi_k}^{\\pi_{k+1}}(s)$ at $\\pi_{k+1}$.\n","sha256_utf8":"692c6a660d65b5c09e6f3c272d94cb260a1065ca90dc84f5cb7e5e31a4c65d9f"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"展开Bregman三点恒等式","source_units":[{"start":35741,"end":35997,"start_line":747,"end_line":751,"raw_tex":"Using the definition of Bregman's  distance, it is easy\nto verify that\n\\beq \\label{prox_identity}\nD_{\\pi_k}^p(s) = D_{\\pi_k}^{\\pi_{k+1}}(s) + \\langle \\nabla D_{\\pi_k}^{\\pi_{k+1}}(s, \\cdot), p(\\cdot|s) -\\pi_{k+1}(\\cdot|s) \\rangle + D_{\\pi_{k+1}}^p(s).\n\\eeq\n","sha256_utf8":"4e2ca3874bb86f061e080544961b8972c32eca10f0a85c98b9b489eeb7c77bc3"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"结合强凸条件完成估计","source_units":[{"start":35997,"end":36118,"start_line":752,"end_line":753,"raw_tex":"The result then immediately follows by combining the above two relations together with \\eqnok{eq:convex_obj}.\n\\end{proof}","sha256_utf8":"948518df51b14b3ae7ce4fdbbd468e5db548a6ad7221155dce4a29daef5c4c57"}]},{"id":"S043","semantic_type":"lemma","kind":"lemma","title":"确定性单步改进性质","source_units":[{"start":36120,"end":36466,"start_line":755,"end_line":762,"raw_tex":"\\begin{lemma} \\label{prop:PMD_generic}\nFor any $s \\in \\cS$, we have\n\\begin{align}\nV^{\\pi_{k+1}}(s) &\\le V^{\\pi_k}(s), \\label{eq:pmd_decrease1}\\\\\n\\langle Q^{\\pi_k}(s, \\cdot),\\, \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle + h^{\\pi_{k+1}}(s) - h^{\\pi_{k}}(s)  \n&\\ge V^{\\pi_{k+1}}(s) - V^{\\pi_k}(s). \\label{eq:pmd_decrease2}\n\\end{align}\n\\end{lemma}","sha256_utf8":"0a229271a1d90bee5d1d487abd1200cbd52025817e5f81959cfc5a009d2e70a2"}]},{"id":"S044","semantic_type":"proof","kind":"proof","title":"确定性改进及差分恒等式证明","source_units":[{"start":36468,"end":39049,"start_line":764,"end_line":816,"raw_tex":"\\begin{proof}\nIt follows from Lemma~\\ref{lemma_per_diff} (with $\\pi' = \\pi_{k+1}$, $\\pi = \\pi_k$ and $\\tau = \\tau_k$) that\n\\begin{align}\n&V^{\\pi_{k+1}}(s) - V^{\\pi_k}(s) \\nn\\\\\n&=\n\\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle A^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s')\\rangle\n+ h^{\\pi_{k+1}}(s') - h^{\\pi_k}(s')\n\\right]. \\label{eq:perf_diff_pmd}\n\\end{align}\nSimilarly to \\eqnok{eq:inner_product_equivalence}, we can show that\n\\begin{align*}\n\\langle A^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s')\\rangle\n&= \\langle Q^{\\pi_k}(s', \\cdot) - V^{\\pi_k}(s') e, \\pi_{k+1}(\\cdot | s')\\rangle\\\\\n&= \\langle Q^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s')\\rangle - V^{\\pi_k}_{\\tau_k}(s')\\\\\n&= \\langle Q^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot| s')\\rangle.\n\\end{align*}\nCombining the above two identities, we then obtain\n\\begin{align}\nV^{\\pi_{k+1}}(s) - V^{\\pi_k}(s) &=\n\\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle Q^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle \\right.\\nn \\\\\n& \\quad \\quad \\quad \\quad \\quad \\quad \\quad \\quad\n\\left. + h^{\\pi_{k+1}}(s') - h^{\\pi_k}(s')\\right]. \\label{eq:improvement_in_one_step_temp}\n\\end{align}\nNow we conclude from Lemma~\\ref{lemma:prox_optimality} applied to \\eqnok{eq:PMD_step}\nwith $p(\\cdot|s') = \\pi_k(\\cdot|s')$ that\n\\begin{align}\n& \\langle Q^{\\pi_{k}}(s', \\cdot), \n\\pi_{k+1}(\\cdot|s') - \\pi_k(\\cdot|s') \\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_k}(s') \\nn \\\\\n&\\le  -\\tfrac{1}{\\eta_k} [(1+\\eta_k \\mu) D_{\\pi_{k+1}}^{\\pi_k}(s') + D_{\\pi_k}^{\\pi_{k+1}}(s')].\\label{eq:negative_term_pmd}\n\\end{align}\nThe previous two conclusions\nthen clearly imply the result in \\eqnok{eq:pmd_decrease1}.\nIt also follows from \\eqnok{eq:negative_term_pmd} that\n\\begin{align}\n&\\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle Q^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle \n + h^{\\pi_{k+1}}(s') - h^{\\pi_k}(s') \\right]\\nn \\\\\n& \\le d_s^{\\pi_{k+1}}(s) \\left[\n\\langle Q^{\\pi_k}(s, \\cdot), \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle \n + h^{\\pi_{k+1}}(s) - h^{\\pi_k}(s) \\right]\\nn \\\\\n &\\le (1-\\gamma) \\left[\n\\langle Q^{\\pi_k}(s, \\cdot), \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle \n + h^{\\pi_{k+1}}(s) - h^{\\pi_k}(s) \\right], \\label{eq:negative_term_pmd_prob_switch}\n\\end{align}\nwhere the last inequality follows from the fact that $d_s^{\\pi_{k+1}}(s) \\ge (1-\\gamma)$ due to the definition of $d_s^{\\pi_{k+1}}$\nin \\eqnok{eq:def_visitation}. The result in \\eqnok{eq:pmd_decrease1} \nthen follows immediately from \\eqnok{eq:improvement_in_one_step_temp} and the above inequality.\n\\end{proof}","sha256_utf8":"fb4f2bd36662a9a2da2821de761009f9f63e5e96052792e3fdee6ef64ed32fd2"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"相邻策略性能差分","source_units":[{"start":36468,"end":36846,"start_line":764,"end_line":774,"raw_tex":"\\begin{proof}\nIt follows from Lemma~\\ref{lemma_per_diff} (with $\\pi' = \\pi_{k+1}$, $\\pi = \\pi_k$ and $\\tau = \\tau_k$) that\n\\begin{align}\n&V^{\\pi_{k+1}}(s) - V^{\\pi_k}(s) \\nn\\\\\n&=\n\\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle A^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s')\\rangle\n+ h^{\\pi_{k+1}}(s') - h^{\\pi_k}(s')\n\\right]. \\label{eq:perf_diff_pmd}\n\\end{align}\n","sha256_utf8":"d71e4f4b43221abcf8ee6deea9a47b0be034417629e04b624013f761663f6d6a"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"优势函数消去价值常数","source_units":[{"start":36846,"end":37254,"start_line":775,"end_line":781,"raw_tex":"Similarly to \\eqnok{eq:inner_product_equivalence}, we can show that\n\\begin{align*}\n\\langle A^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s')\\rangle\n&= \\langle Q^{\\pi_k}(s', \\cdot) - V^{\\pi_k}(s') e, \\pi_{k+1}(\\cdot | s')\\rangle\\\\\n&= \\langle Q^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s')\\rangle - V^{\\pi_k}_{\\tau_k}(s')\\\\\n&= \\langle Q^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot| s')\\rangle.\n\\end{align*}\n","sha256_utf8":"6d0cde227e1bfe30885f105e4e2ce0801f4a8738ba2849bb054708e46904ba53"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"得到单步价值差恒等式","source_units":[{"start":37254,"end":37657,"start_line":782,"end_line":790,"raw_tex":"Combining the above two identities, we then obtain\n\\begin{align}\nV^{\\pi_{k+1}}(s) - V^{\\pi_k}(s) &=\n\\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle Q^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle \\right.\\nn \\\\\n& \\quad \\quad \\quad \\quad \\quad \\quad \\quad \\quad\n\\left. + h^{\\pi_{k+1}}(s') - h^{\\pi_k}(s')\\right]. \\label{eq:improvement_in_one_step_temp}\n\\end{align}\n","sha256_utf8":"c26b3c65b5e5e4441bd4f32ccfd6b36f9d69d0a220e1736c0ff168eed4fc17ae"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"三点不等式保证非正改进","source_units":[{"start":37657,"end":38148,"start_line":791,"end_line":799,"raw_tex":"Now we conclude from Lemma~\\ref{lemma:prox_optimality} applied to \\eqnok{eq:PMD_step}\nwith $p(\\cdot|s') = \\pi_k(\\cdot|s')$ that\n\\begin{align}\n& \\langle Q^{\\pi_{k}}(s', \\cdot), \n\\pi_{k+1}(\\cdot|s') - \\pi_k(\\cdot|s') \\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_k}(s') \\nn \\\\\n&\\le  -\\tfrac{1}{\\eta_k} [(1+\\eta_k \\mu) D_{\\pi_{k+1}}^{\\pi_k}(s') + D_{\\pi_k}^{\\pi_{k+1}}(s')].\\label{eq:negative_term_pmd}\n\\end{align}\nThe previous two conclusions\nthen clearly imply the result in \\eqnok{eq:pmd_decrease1}.\n","sha256_utf8":"2ea34c52d843e62fb2b28641fbef5c1b069400f00e06cf1e284733dbfb0ecaf5"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"访问概率下界局部化期望","source_units":[{"start":38148,"end":39049,"start_line":800,"end_line":816,"raw_tex":"It also follows from \\eqnok{eq:negative_term_pmd} that\n\\begin{align}\n&\\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle Q^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle \n + h^{\\pi_{k+1}}(s') - h^{\\pi_k}(s') \\right]\\nn \\\\\n& \\le d_s^{\\pi_{k+1}}(s) \\left[\n\\langle Q^{\\pi_k}(s, \\cdot), \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle \n + h^{\\pi_{k+1}}(s) - h^{\\pi_k}(s) \\right]\\nn \\\\\n &\\le (1-\\gamma) \\left[\n\\langle Q^{\\pi_k}(s, \\cdot), \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle \n + h^{\\pi_{k+1}}(s) - h^{\\pi_k}(s) \\right], \\label{eq:negative_term_pmd_prob_switch}\n\\end{align}\nwhere the last inequality follows from the fact that $d_s^{\\pi_{k+1}}(s) \\ge (1-\\gamma)$ due to the definition of $d_s^{\\pi_{k+1}}$\nin \\eqnok{eq:def_visitation}. The result in \\eqnok{eq:pmd_decrease1} \nthen follows immediately from \\eqnok{eq:improvement_in_one_step_temp} and the above inequality.\n\\end{proof}","sha256_utf8":"f2c276316e90cf1e3a23ca7cc6dbaa54e1bec5b0392a42f356d2ebc04d265c1a"}]},{"id":"S050","semantic_type":"lemma","kind":"lemma","title":"分期衰减递推引理","source_units":[{"start":59015,"end":59654,"start_line":1235,"end_line":1246,"raw_tex":"\\begin{lemma} \\label{lemma:induction_tech_sequence}\nAssume that the nonnegative sequences $\\{X_k\\}_{k \\ge 0}, \\{Y_k\\}_{k \\ge 0}$ and $\\{Z_k\\}_{k \\ge 0}$ satisfy\n\\beq \\label{eq:induction_tech_sequence}\nX_{k+1} \\le \\gamma X_{k} + (Y_k - Y_{k+1}) + Z_k. % \\ \\ \\mbox{and} \\ \\ Y_k \\ge Y_{k+1}.\n\\eeq\nLet us denote $l = \\left \\lceil  \\log_\\gamma \\tfrac{1}{4} \\right \\rceil$.\nIf $Y_k = Y \\cdot 2^{-(\\lfloor k/l \\rfloor+1)}$ and $Z_k = Z \\cdot 2^{-(\\lfloor k/l \\rfloor+2)}$ for some $Y\\ge 0$ and $Z \\ge 0$, \nthen \n\\beq  \\label{eq:induction_tech_sequence_result}\nX_k \\le 2^{-\\lfloor k/l \\rfloor} (X_0 + Y + \\tfrac{5Z}{4(1-\\gamma)}).\n\\eeq\n\\end{lemma}","sha256_utf8":"83c6be1b6fecf2852a74635b0dc1372c92f40b199ef8fd66f50542aca7f13906"}]},{"id":"S051","semantic_type":"proof","kind":"proof","title":"分期递推完整证明","source_units":[{"start":59656,"end":61568,"start_line":1248,"end_line":1278,"raw_tex":"\\begin{proof}\nLet us group the indices $\\{0, \\ldots, k\\}$  into $\\bar p \\equiv \\lfloor k/l \\rfloor+1$ epochs\nwith each of the first $\\bar p - 1$ epochs consisting of $l$ iterations. Let $p=0, \\ldots, \\bar p$ be the epoch indices.\nWe first show that for any $p = 0, \\ldots, \\bar p - 1$,\n\\beq \\label{eq:induction_tech_result}\nX_{pl} \\le 2^{-p} (X_0 + Y + \\tfrac{Z}{1-\\gamma} ).\n\\eeq\nThis relation holds obviously for $p = 0$. Let us assume that \\eqnok{eq:induction_tech_result} holds at the beginning\nof epoch $p$ ad examine the progress made in epoch $p$.\nNote that for any indices  $k = p l, \\ldots, (p+1)l-1$ in epoch $p$,\nwe have $Y_{k} = Y \\cdot 2^{-(p+1)}$ and  $Z = Z \\cdot 2^{-(p+2)}$.\nBy applying \\eqnok{eq:induction_tech_sequence} recursively, we have\n\\begin{align*}\nX_{(p+1) l} &\\le \\gamma^l X_{pl} + Y_{pl}- Y_{(p+1)l} + Z_{pl}  \\tsum_{i=0}^{l-1} \\gamma^{i} \\\\\n&= \\gamma^l X_{pl} + Y_{(p+1)l} + Z_{pl} \\tfrac{1 - \\gamma^l}{1-\\gamma}\\\\\n&\\le \\gamma^l X_{pl} + Y \\cdot 2^{-(p+2)} +  \\tfrac{Z \\cdot 2^{-(p+2)}}{1-\\gamma}\\\\\n&\\le \\tfrac{1}{4}X_{pl} + Y \\cdot 2^{-(p+2)} +  \\tfrac{Z \\cdot 2^{-(p+2)}}{1-\\gamma}\\\\\n&\\le \\tfrac{1}{4} 2^{-p} (X_0 +  Y + \\tfrac{Z}{1-\\gamma} ) + Y \\cdot 2^{-(p+2)} +  \\tfrac{Z \\cdot 2^{-(p+2)}}{1-\\gamma}\\\\\n&\\le 2^{-(p+1)} (X_0 + Y + \\tfrac{Z}{1-\\gamma} ),\n\\end{align*}\nwhere the second inequality follows from the definition of $Z_{pl}$ and $\\gamma^l \\ge 0$,\nthe third one follows from $\\gamma^l \\le 1/4$, the fourth one follows by induction hypothesis, and\nthe last one follows by regrouping the terms.\nSince $k = (\\bar p -1) l + k\\pmod l$, we have\n\\begin{align*}\nX_k &\\le \\gamma^{{k\\pmod l}} X_{(\\bar p -1) l} + Z_{(\\bar p -1)l} \\tsum_{i=0}^{k\\pmod l - 1} \\gamma^i\\\\\n&\\le 2^{-(\\bar p -1)} (X_0 + Y + \\tfrac{Z}{1-\\gamma} ) + \\tfrac{Z \\cdot 2^{-(\\bar p +1)}}{1-\\gamma}\\\\\n&= 2^{-(\\bar p -1)} (X_0 + Y + \\tfrac{5 Z}{4(1-\\gamma)} ),\n\\end{align*}\nwhich implies the result.\n\\end{proof}","sha256_utf8":"5221ae0c6bc21e512ce7e142d5dc668fdd05ac0b7cc4276eda78f9c427f6400f"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"划分时期并提出端点界","source_units":[{"start":59656,"end":60037,"start_line":1248,"end_line":1254,"raw_tex":"\\begin{proof}\nLet us group the indices $\\{0, \\ldots, k\\}$  into $\\bar p \\equiv \\lfloor k/l \\rfloor+1$ epochs\nwith each of the first $\\bar p - 1$ epochs consisting of $l$ iterations. Let $p=0, \\ldots, \\bar p$ be the epoch indices.\nWe first show that for any $p = 0, \\ldots, \\bar p - 1$,\n\\beq \\label{eq:induction_tech_result}\nX_{pl} \\le 2^{-p} (X_0 + Y + \\tfrac{Z}{1-\\gamma} ).\n\\eeq\n","sha256_utf8":"898a750881ce793d63a8785485a5c5085c1892081a518b21126f93680e4b633e"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"归纳控制完整时期误差","source_units":[{"start":60037,"end":61192,"start_line":1255,"end_line":1270,"raw_tex":"This relation holds obviously for $p = 0$. Let us assume that \\eqnok{eq:induction_tech_result} holds at the beginning\nof epoch $p$ ad examine the progress made in epoch $p$.\nNote that for any indices  $k = p l, \\ldots, (p+1)l-1$ in epoch $p$,\nwe have $Y_{k} = Y \\cdot 2^{-(p+1)}$ and  $Z = Z \\cdot 2^{-(p+2)}$.\nBy applying \\eqnok{eq:induction_tech_sequence} recursively, we have\n\\begin{align*}\nX_{(p+1) l} &\\le \\gamma^l X_{pl} + Y_{pl}- Y_{(p+1)l} + Z_{pl}  \\tsum_{i=0}^{l-1} \\gamma^{i} \\\\\n&= \\gamma^l X_{pl} + Y_{(p+1)l} + Z_{pl} \\tfrac{1 - \\gamma^l}{1-\\gamma}\\\\\n&\\le \\gamma^l X_{pl} + Y \\cdot 2^{-(p+2)} +  \\tfrac{Z \\cdot 2^{-(p+2)}}{1-\\gamma}\\\\\n&\\le \\tfrac{1}{4}X_{pl} + Y \\cdot 2^{-(p+2)} +  \\tfrac{Z \\cdot 2^{-(p+2)}}{1-\\gamma}\\\\\n&\\le \\tfrac{1}{4} 2^{-p} (X_0 +  Y + \\tfrac{Z}{1-\\gamma} ) + Y \\cdot 2^{-(p+2)} +  \\tfrac{Z \\cdot 2^{-(p+2)}}{1-\\gamma}\\\\\n&\\le 2^{-(p+1)} (X_0 + Y + \\tfrac{Z}{1-\\gamma} ),\n\\end{align*}\nwhere the second inequality follows from the definition of $Z_{pl}$ and $\\gamma^l \\ge 0$,\nthe third one follows from $\\gamma^l \\le 1/4$, the fourth one follows by induction hypothesis, and\nthe last one follows by regrouping the terms.\n","sha256_utf8":"a3e1c88506cd28245ec01b56cfdffa724d203e6e2f6a79f5de9ffd7e64c9d260"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"控制末尾未完成时期","source_units":[{"start":61192,"end":61568,"start_line":1271,"end_line":1278,"raw_tex":"Since $k = (\\bar p -1) l + k\\pmod l$, we have\n\\begin{align*}\nX_k &\\le \\gamma^{{k\\pmod l}} X_{(\\bar p -1) l} + Z_{(\\bar p -1)l} \\tsum_{i=0}^{k\\pmod l - 1} \\gamma^i\\\\\n&\\le 2^{-(\\bar p -1)} (X_0 + Y + \\tfrac{Z}{1-\\gamma} ) + \\tfrac{Z \\cdot 2^{-(\\bar p +1)}}{1-\\gamma}\\\\\n&= 2^{-(\\bar p -1)} (X_0 + Y + \\tfrac{5 Z}{4(1-\\gamma)} ),\n\\end{align*}\nwhich implies the result.\n\\end{proof}","sha256_utf8":"19ba15eab16275268b511760b679d19ded3706da1f336688c4ae40ea0353843c"}]},{"id":"S055","semantic_type":"proof","kind":"proof","title":"随机方向偏差界推导","source_units":[{"start":67071,"end":67984,"start_line":1379,"end_line":1391,"raw_tex":"By using the assumptions in \\eqnok{eq:expectation_ass} and \\eqnok{eq:bias_ass} and\nthe decomposition \n\\begin{align*}\n \\langle \\cQ^{\\pi_{k},\\xi_k}(s, \\cdot), \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle\n &= \\langle Q^{\\pi_k}(s, \\cdot),  \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle \\\\\n & \\quad + \\langle \\bar \\cQ^{\\pi_k}(s, \\cdot) - Q^{\\pi_k}(s, \\cdot),  \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle \\\\\n & \\quad + \\langle  \\cQ^{\\pi_k,\\xi_k}(s, \\cdot) - \\bar \\cQ^{\\pi_k}(s, \\cdot),  \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle,\n\\end{align*}\nwe can see that %Taking expectation w.r.t. $\\xi_k$ conditional on $\\xi_{\\lceil \\xi_{k-1} \\rceil}$, \n\\begin{align}\n&\\bbe_{\\xi_k}[ \\langle \\cQ^{\\pi_{k},\\xi_k}(s, \\cdot), \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle \\mid \\xi_{\\lceil \\xi_{k-1} \\rceil}] \\nn \\\\\n&\\ge  \\langle Q^{\\pi_k}(s, \\cdot),  \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle - 2 \\varsigma_k. \\label{eq:bnd_bias_SPMD}\n\\end{align}\n","sha256_utf8":"13591f91356a35508831835833bb237258cfd17a9c0989cd315aa049e895bfac"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"分离偏差与条件零均值噪声","source_units":[{"start":67071,"end":67984,"start_line":1379,"end_line":1391,"raw_tex":"By using the assumptions in \\eqnok{eq:expectation_ass} and \\eqnok{eq:bias_ass} and\nthe decomposition \n\\begin{align*}\n \\langle \\cQ^{\\pi_{k},\\xi_k}(s, \\cdot), \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle\n &= \\langle Q^{\\pi_k}(s, \\cdot),  \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle \\\\\n & \\quad + \\langle \\bar \\cQ^{\\pi_k}(s, \\cdot) - Q^{\\pi_k}(s, \\cdot),  \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle \\\\\n & \\quad + \\langle  \\cQ^{\\pi_k,\\xi_k}(s, \\cdot) - \\bar \\cQ^{\\pi_k}(s, \\cdot),  \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle,\n\\end{align*}\nwe can see that %Taking expectation w.r.t. $\\xi_k$ conditional on $\\xi_{\\lceil \\xi_{k-1} \\rceil}$, \n\\begin{align}\n&\\bbe_{\\xi_k}[ \\langle \\cQ^{\\pi_{k},\\xi_k}(s, \\cdot), \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle \\mid \\xi_{\\lceil \\xi_{k-1} \\rceil}] \\nn \\\\\n&\\ge  \\langle Q^{\\pi_k}(s, \\cdot),  \\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle - 2 \\varsigma_k. \\label{eq:bnd_bias_SPMD}\n\\end{align}\n","sha256_utf8":"13591f91356a35508831835833bb237258cfd17a9c0989cd315aa049e895bfac"}]},{"id":"S057","semantic_type":"lemma","kind":"lemma","title":"随机更新单步价值估计","source_units":[{"start":68195,"end":68597,"start_line":1400,"end_line":1409,"raw_tex":"\\begin{lemma} \\label{prop:SPMD_generic}\nFor any $s \\in \\cS$, we have\n\\begin{align}\nV^{\\pi_{k+1}}(s) - V^{\\pi_k}(s) &\\le\n\\langle \\cQ^{\\pi_k,\\xi_k}(s, \\cdot) , \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle + h^{\\pi_{k+1}}(s) - h^{\\pi_{k}}(s) \\nn\\\\\n& \\quad   +\n\\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi_{k+1}}(s)+ \\tfrac{\\eta_k \\|\\delta_k\\|_\\infty^2}{2(1-\\gamma)}.\n\\label{eq:spmd_decrease2}\n\\end{align}\n\\end{lemma}","sha256_utf8":"66d4392b41e6226f1c1e8d3ef6f2d4d74b4ace6f0188e90bc7456a804fb77813"}]},{"id":"S058","semantic_type":"proof","kind":"proof","title":"随机单步估计完整证明","source_units":[{"start":68599,"end":71676,"start_line":1411,"end_line":1465,"raw_tex":"\\begin{proof}\n%We prove \\eqnok{eq:spmd_decrease2} first since its proof is slightly more complicated.\nObserve that \\eqnok{eq:improvement_in_one_step_temp}\nstill holds, and hence that\n\\begin{align}\n&V^{\\pi_{k+1}}(s) - V^{\\pi_k}(s)\\nn \\\\\n&=\n\\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle Q^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s') \\right] \\nn \\\\\n&=\n\\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s')\\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s') \\right.\\nn \\\\\n& \\quad \n\\left.  - \\langle \\delta_k, \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle \\right] \\nn\\\\\n&\\le \\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s') \\right.\\nn \\\\\n& \\quad\n\\left. +\n \\tfrac{1 }{2\\eta_k}\\|\\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\|_1^2 + \\tfrac{\\eta_k \\|\\delta_k\\|_\\infty^2}{2}  \\right] \\nn\\\\\n &\\le \\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s')\\right.\\nn \\\\\n& \\quad\n\\left. + \\tfrac{1}{\\eta_k} D_{\\pi_{k}}^{\\pi_{k+1}}(s') + \\tfrac{\\eta_k \\|\\delta_k\\|_\\infty^2}{2} \\right], \\label{eq:improvement_in_one_step_temp_stoch}\n\\end{align}\nwhere the first inequality follows from Young's inequality and\nthe second one follows from the strong convexity of $D_{\\pi_{k+1}}^{\\pi_{k}}$ w.r.t. to $\\|\\cdot\\|_1$.\nMoreover, we conclude from Lemma~\\ref{lemma:prox_optimality} \napplied to \\eqnok{eq:SPMD_step}\nwith $Q^{\\pi_{k}}$ replaced by $\\cQ^{\\pi_{k},\\xi_k}$ and $p(\\cdot|s') = \\pi_k(\\cdot|s')$ that\n\\begin{align*}\n& \\langle \\cQ^{\\pi_{k},\\xi_k}(s', \\cdot), \n\\pi_{k+1}(\\cdot|s') - \\pi_k(\\cdot|s') \\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s') + \\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi_{k+1}}(s')\\\\\n&\\le  -\\tfrac{1}{\\eta_k} [(1+ \\eta_k \\mu) D_{\\pi_{k+1}}^{\\pi_k}(s') ] \\le 0,\n\\end{align*}\nwhich implies that\n\\begin{align*}\n&\\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle \n+ h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s') +  \\tfrac{1}{\\eta_k} D_{\\pi_{k+1}}^{\\pi_{k}}(s') \\right]\\nn \\\\\n& \\le d_s^{\\pi_{k+1}}(s) \\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s, \\cdot), \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle + h^{\\pi_{k+1}}(s) - h^{\\pi_{k}}(s) \n + \\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi_{k+1}}(s) \\right]\\nn \\\\\n &\\le (1-\\gamma) \\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s, \\cdot), \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle + h^{\\pi_{k+1}}(s) - h^{\\pi_{k}}(s) \n + \\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi_{k+1}}(s) \\right], \n \\end{align*}\nwhere the last inequality follows from the fact that $d_s^{\\pi_{k+1}}(s) \\ge 1-\\gamma$ due to the definition of $d_s^{\\pi_{k+1}}$\nin \\eqnok{eq:def_visitation}. The result in \\eqnok{eq:spmd_decrease2} \nthen follows immediately from \\eqnok{eq:improvement_in_one_step_temp_stoch} and the above inequality.\n\\end{proof}","sha256_utf8":"34ab843e16126d6bb7ec28951b01d241142714e84d52e342b96d5d5e50092059"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"性能差分与噪声二次控制","source_units":[{"start":68599,"end":70207,"start_line":1411,"end_line":1440,"raw_tex":"\\begin{proof}\n%We prove \\eqnok{eq:spmd_decrease2} first since its proof is slightly more complicated.\nObserve that \\eqnok{eq:improvement_in_one_step_temp}\nstill holds, and hence that\n\\begin{align}\n&V^{\\pi_{k+1}}(s) - V^{\\pi_k}(s)\\nn \\\\\n&=\n\\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle Q^{\\pi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s') \\right] \\nn \\\\\n&=\n\\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s')\\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s') \\right.\\nn \\\\\n& \\quad \n\\left.  - \\langle \\delta_k, \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle \\right] \\nn\\\\\n&\\le \\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s') \\right.\\nn \\\\\n& \\quad\n\\left. +\n \\tfrac{1 }{2\\eta_k}\\|\\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\|_1^2 + \\tfrac{\\eta_k \\|\\delta_k\\|_\\infty^2}{2}  \\right] \\nn\\\\\n &\\le \\tfrac{1}{1-\\gamma} \\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s')\\right.\\nn \\\\\n& \\quad\n\\left. + \\tfrac{1}{\\eta_k} D_{\\pi_{k}}^{\\pi_{k+1}}(s') + \\tfrac{\\eta_k \\|\\delta_k\\|_\\infty^2}{2} \\right], \\label{eq:improvement_in_one_step_temp_stoch}\n\\end{align}\nwhere the first inequality follows from Young's inequality and\nthe second one follows from the strong convexity of $D_{\\pi_{k+1}}^{\\pi_{k}}$ w.r.t. to $\\|\\cdot\\|_1$.\n","sha256_utf8":"4825f09943545db31240eadb3561ac8732a94ea05f66bb4f844eebfd774435a5"}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","title":"随机近端最优性保证非正项","source_units":[{"start":70207,"end":70675,"start_line":1441,"end_line":1448,"raw_tex":"Moreover, we conclude from Lemma~\\ref{lemma:prox_optimality} \napplied to \\eqnok{eq:SPMD_step}\nwith $Q^{\\pi_{k}}$ replaced by $\\cQ^{\\pi_{k},\\xi_k}$ and $p(\\cdot|s') = \\pi_k(\\cdot|s')$ that\n\\begin{align*}\n& \\langle \\cQ^{\\pi_{k},\\xi_k}(s', \\cdot), \n\\pi_{k+1}(\\cdot|s') - \\pi_k(\\cdot|s') \\rangle + h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s') + \\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi_{k+1}}(s')\\\\\n&\\le  -\\tfrac{1}{\\eta_k} [(1+ \\eta_k \\mu) D_{\\pi_{k+1}}^{\\pi_k}(s') ] \\le 0,\n\\end{align*}\n","sha256_utf8":"7bb5df45e37be9faf3bb4edd88a869beb477d6c86573cce4c7d5517304f1e001"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"局部化访问期望完成单步界","source_units":[{"start":70675,"end":71676,"start_line":1449,"end_line":1465,"raw_tex":"which implies that\n\\begin{align*}\n&\\bbe_{s' \\sim d_s^{\\pi_{k+1}}}\n\\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s', \\cdot), \\pi_{k+1}(\\cdot | s') - \\pi_k(\\cdot|s') \\rangle \n+ h^{\\pi_{k+1}}(s') - h^{\\pi_{k}}(s') +  \\tfrac{1}{\\eta_k} D_{\\pi_{k+1}}^{\\pi_{k}}(s') \\right]\\nn \\\\\n& \\le d_s^{\\pi_{k+1}}(s) \\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s, \\cdot), \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle + h^{\\pi_{k+1}}(s) - h^{\\pi_{k}}(s) \n + \\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi_{k+1}}(s) \\right]\\nn \\\\\n &\\le (1-\\gamma) \\left[\n\\langle \\cQ^{\\pi_k,\\xi_k}(s, \\cdot), \\pi_{k+1}(\\cdot | s) - \\pi_k(\\cdot|s) \\rangle + h^{\\pi_{k+1}}(s) - h^{\\pi_{k}}(s) \n + \\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi_{k+1}}(s) \\right], \n \\end{align*}\nwhere the last inequality follows from the fact that $d_s^{\\pi_{k+1}}(s) \\ge 1-\\gamma$ due to the definition of $d_s^{\\pi_{k+1}}$\nin \\eqnok{eq:def_visitation}. The result in \\eqnok{eq:spmd_decrease2} \nthen follows immediately from \\eqnok{eq:improvement_in_one_step_temp_stoch} and the above inequality.\n\\end{proof}","sha256_utf8":"ff62c4da0a6879aeaabe26104af400c46829fb046a0ef764aec07ba325e460c8"}]},{"id":"S062","semantic_type":"lemma","kind":"lemma","title":"随机目标与距离联合递推","source_units":[{"start":71749,"end":72140,"start_line":1471,"end_line":1478,"raw_tex":"\\begin{lemma} \\label{prop:SPMD_regularized_MDPs}\nFor any $k \\ge 0$, we have\n\\begin{align*}\n&  \\bbe_{\\xi_{\\lceil k \\rceil}} [f(\\pi_{k+1}) - f(\\pi^*) + (\\tfrac{1}{\\eta_k}+  \\mu) \\cD(\\pi_{k+1}, \\pi^*)] \\\\\n&\\le   \\bbe_{\\xi_{\\lceil k-1 \\rceil}} [\\gamma (f(\\pi_k) - f(\\pi^*))+ \\tfrac{1}{\\eta_k} \\cD(\\pi_k,\\pi^*) ]\n+ 2 \\varsigma_k + \\tfrac{ \\eta_k \\sigma_k^2}{2(1-\\gamma)}.\n\\end{align*}\n\\end{lemma}","sha256_utf8":"f36c96be2decc58bbd7a2c1476d338e6ccde20d5847bd1f00ed34461e839c253"}]},{"id":"S063","semantic_type":"proof","kind":"proof","title":"联合误差递推完整证明","source_units":[{"start":72142,"end":73785,"start_line":1480,"end_line":1510,"raw_tex":"\\begin{proof}\nBy applying Lemma~\\ref{lemma:prox_optimality} to \\eqnok{eq:PMD_step}\n(with $Q^{\\pi_{k}}$ replaced by $\\cQ^{\\pi_{k},\\xi_k}$ and $p = \\pi^*$), we have\n\\begin{align*}\n&\\eta_k[ \\langle \\cQ^{\\pi_{k},\\xi_k}(s, \\cdot), \n\\pi_{k+1}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle \n+ h^{\\pi_{k+1}}(s) -  h^{\\pi^*}(s) ]\n+ D_{\\pi_k}^{\\pi_{k+1}}(s) \\\\\n&\\le D_{\\pi_k}^{\\pi^*}(s) - (1+\\eta_k \\mu) D_{\\pi_{k+1}}^{\\pi^*}(s),\n\\end{align*}\nwhich, in view of \\eqnok{eq:spmd_decrease2}, then implies that\n\\begin{align*}\n&\\langle \\cQ^{\\pi_{k},\\xi_k}(s, \\cdot) , \n\\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle + h^{\\pi_{k}}(s) -  h^{\\pi^*}(s) +  V^{\\pi_{k+1}}(s) - V^{\\pi_k}(s) \n \\\\\n& \n\\le \\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi^*}(s) - (\\tfrac{1}{\\eta_k} + \\mu) D_{\\pi_{k+1}}^{\\pi^*}(s) + \\tfrac{\\eta_k \\|\\delta_k\\|_\\infty^2}{2(1-\\gamma)}.\n\\end{align*}\nTaking expectation w.r.t. $\\xi_{\\lceil k \\rceil}$ and $\\nu^*$ on both sides of the above inequality,\nand using Lemma~\\ref{prop_strong_monotone} and the relation in \\eqnok{eq:bnd_bias_SPMD}, \nwe arrive at\n\\begin{align*}\n&\\bbe_{s\\sim \\nu^*, \\xi_{\\lceil k \\rceil}}\\left[(1- \\gamma)(V^{\\pi_k}(s) - V^{\\pi_\\tau^*}(s) ) \n+  V^{\\pi_{k+1}}(s) - V^{\\pi_k}(s) \\right] \\\\\n&\\le \\bbe_{s\\sim \\nu^*,\\xi_{\\lceil k \\rceil}}[\\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi^*}(s) - (\\tfrac{1}{\\eta_k}+ \\mu) D_{\\pi_{k+1}}^{\\pi^*}(s)] \n+ 2  \\varsigma_k + \\tfrac{ \\eta_k \\sigma_k^2}{2(1-\\gamma)}.\n\\end{align*}\nNoting $V^{\\pi_{k+1}}(s) -  V^{\\pi_k}(s)= \nV^{\\pi_{k+1}}(s) - V^{\\pi^*}(s) - [V^{\\pi_k}(s) - V^{\\pi^*}(s) ] $,\nrearranging the terms in the above inequality, and using the definition of $f$ in \\eqnok{eq:MDP_OPT}, we arrive at the result.\n\\end{proof}","sha256_utf8":"8c2bd37fe155e5fce049a7e206b931234f8ba88b8e82af40dd6d6bde10d6460a"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"以最优策略测试近端不等式","source_units":[{"start":72142,"end":72563,"start_line":1480,"end_line":1489,"raw_tex":"\\begin{proof}\nBy applying Lemma~\\ref{lemma:prox_optimality} to \\eqnok{eq:PMD_step}\n(with $Q^{\\pi_{k}}$ replaced by $\\cQ^{\\pi_{k},\\xi_k}$ and $p = \\pi^*$), we have\n\\begin{align*}\n&\\eta_k[ \\langle \\cQ^{\\pi_{k},\\xi_k}(s, \\cdot), \n\\pi_{k+1}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle \n+ h^{\\pi_{k+1}}(s) -  h^{\\pi^*}(s) ]\n+ D_{\\pi_k}^{\\pi_{k+1}}(s) \\\\\n&\\le D_{\\pi_k}^{\\pi^*}(s) - (1+\\eta_k \\mu) D_{\\pi_{k+1}}^{\\pi^*}(s),\n\\end{align*}\n","sha256_utf8":"a54e64c9dde779739ea2bda39452c5080941b51a7aef879963f8f7c444dd4cac"}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","title":"代入单步价值改进界","source_units":[{"start":72563,"end":72963,"start_line":1490,"end_line":1497,"raw_tex":"which, in view of \\eqnok{eq:spmd_decrease2}, then implies that\n\\begin{align*}\n&\\langle \\cQ^{\\pi_{k},\\xi_k}(s, \\cdot) , \n\\pi_{k}(\\cdot|s) - \\pi^*(\\cdot|s) \\rangle + h^{\\pi_{k}}(s) -  h^{\\pi^*}(s) +  V^{\\pi_{k+1}}(s) - V^{\\pi_k}(s) \n \\\\\n& \n\\le \\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi^*}(s) - (\\tfrac{1}{\\eta_k} + \\mu) D_{\\pi_{k+1}}^{\\pi^*}(s) + \\tfrac{\\eta_k \\|\\delta_k\\|_\\infty^2}{2(1-\\gamma)}.\n\\end{align*}\n","sha256_utf8":"1d2fb6c22abf9a73d14209e8524758fc2ba39920baa2f102805d7a7566fc0e9a"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"平均并使用单调性与偏差界","source_units":[{"start":72963,"end":73536,"start_line":1498,"end_line":1506,"raw_tex":"Taking expectation w.r.t. $\\xi_{\\lceil k \\rceil}$ and $\\nu^*$ on both sides of the above inequality,\nand using Lemma~\\ref{prop_strong_monotone} and the relation in \\eqnok{eq:bnd_bias_SPMD}, \nwe arrive at\n\\begin{align*}\n&\\bbe_{s\\sim \\nu^*, \\xi_{\\lceil k \\rceil}}\\left[(1- \\gamma)(V^{\\pi_k}(s) - V^{\\pi_\\tau^*}(s) ) \n+  V^{\\pi_{k+1}}(s) - V^{\\pi_k}(s) \\right] \\\\\n&\\le \\bbe_{s\\sim \\nu^*,\\xi_{\\lceil k \\rceil}}[\\tfrac{1}{\\eta_k} D_{\\pi_k}^{\\pi^*}(s) - (\\tfrac{1}{\\eta_k}+ \\mu) D_{\\pi_{k+1}}^{\\pi^*}(s)] \n+ 2  \\varsigma_k + \\tfrac{ \\eta_k \\sigma_k^2}{2(1-\\gamma)}.\n\\end{align*}\n","sha256_utf8":"b13f3aa9ba2472738d24c4ef25fe098379e56c64316c24404f3c79924471d809"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"重排为目标距离递推","source_units":[{"start":73536,"end":73785,"start_line":1507,"end_line":1510,"raw_tex":"Noting $V^{\\pi_{k+1}}(s) -  V^{\\pi_k}(s)= \nV^{\\pi_{k+1}}(s) - V^{\\pi^*}(s) - [V^{\\pi_k}(s) - V^{\\pi^*}(s) ] $,\nrearranging the terms in the above inequality, and using the definition of $f$ in \\eqnok{eq:MDP_OPT}, we arrive at the result.\n\\end{proof}","sha256_utf8":"871cf5fc7edb4fb7e4a97ff9d792bc0372928c65c566e96251cbd8c8955c6040"}]},{"id":"S068","semantic_type":"external_fact","kind":"external_fact","title":"Bellman最优策略存在性","source_units":[],"declared_external":true,"declaration_text":["Finite discounted Markov decision processes with statewise convex policy regularization admit a policy that is optimal simultaneously at every state, by Bellman optimality."]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","title":"KL二次下界与Young不等式","source_units":[],"declared_external":true,"declaration_text":["For probability vectors p and q, KL(p||q) >= ||p-q||_1^2/2. For eta > 0, |<u,v>| <= ||u||_infinity ||v||_1 <= eta ||u||_infinity^2/2 + ||v||_1^2/(2 eta)."]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","title":"凸约束最小化一阶条件","source_units":[],"declared_external":true,"declaration_text":["A minimizer x of a convex function g over a convex set satisfies <z,y-x> >= 0 for an appropriate subgradient z in the constrained first-order optimality condition."]},{"id":"S071","semantic_type":"assumption","kind":"assumption","title":"主定理步长与分期误差条件","source_units":[{"start":74106,"end":74361,"start_line":1518,"end_line":1523,"raw_tex":"Suppose that \n$\\eta_k = \\eta = \\tfrac{1-\\gamma}{\\gamma \\mu}$\nin the SPMD method. \nIf $\\varsigma_k =2^{-(\\lfloor k/l \\rfloor+2)}$ and $\\sigma_k^2 = 2^{-(\\lfloor k/l \\rfloor+2)}$ \nfor any $k \\ge 0$ with $l :=\\left \\lceil  \\log_\\gamma (1/4) \\right \\rceil\n$,\n","sha256_utf8":"dc0fb5bd3dd7650d40c49094b4539c82ae9ed01f3ce65c0c1abf2ab4c320c113"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S062","target_id":"S003","kind":"supports"},{"source_id":"S017","target_id":"S003","kind":"supports"},{"source_id":"S020","target_id":"S003","kind":"supports"},{"source_id":"S071","target_id":"S003","kind":"supports"},{"source_id":"S071","target_id":"S004","kind":"supports"},{"source_id":"S050","target_id":"S004","kind":"supports"},{"source_id":"S015","target_id":"S004","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S008","kind":"supports"},{"source_id":"S005","target_id":"S010","kind":"supports"},{"source_id":"S006","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S011","target_id":"S007","kind":"supports"},{"source_id":"S011","target_id":"S015","kind":"supports"},{"source_id":"S011","target_id":"S017","kind":"supports"},{"source_id":"S010","target_id":"S017","kind":"supports"},{"source_id":"S068","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S023","kind":"supports"},{"source_id":"S009","target_id":"S024","kind":"supports"},{"source_id":"S012","target_id":"S023","kind":"supports"},{"source_id":"S012","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S013","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S027","kind":"supports"},{"source_id":"S010","target_id":"S027","kind":"supports"},{"source_id":"S013","target_id":"S027","kind":"supports"},{"source_id":"S010","target_id":"S028","kind":"supports"},{"source_id":"S013","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S008","target_id":"S031","kind":"supports"},{"source_id":"S009","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S013","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S035","kind":"supports"},{"source_id":"S009","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S026","target_id":"S037","kind":"supports"},{"source_id":"S010","target_id":"S037","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S034","kind":"supports"},{"source_id":"S037","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S014","target_id":"S033","kind":"supports"},{"source_id":"S016","target_id":"S040","kind":"supports"},{"source_id":"S070","target_id":"S040","kind":"supports"},{"source_id":"S011","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S007","target_id":"S042","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S041","target_id":"S039","kind":"supports"},{"source_id":"S042","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S029","target_id":"S045","kind":"supports"},{"source_id":"S036","target_id":"S046","kind":"supports"},{"source_id":"S009","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S038","target_id":"S048","kind":"supports"},{"source_id":"S016","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S013","target_id":"S049","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S044","kind":"supports"},{"source_id":"S049","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S044","target_id":"S059","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S018","target_id":"S056","kind":"supports"},{"source_id":"S019","target_id":"S056","kind":"supports"},{"source_id":"S010","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S055","kind":"supports"},{"source_id":"S019","target_id":"S059","kind":"supports"},{"source_id":"S069","target_id":"S059","kind":"supports"},{"source_id":"S038","target_id":"S060","kind":"supports"},{"source_id":"S019","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S059","target_id":"S061","kind":"supports"},{"source_id":"S013","target_id":"S061","kind":"supports"},{"source_id":"S059","target_id":"S058","kind":"supports"},{"source_id":"S060","target_id":"S058","kind":"supports"},{"source_id":"S061","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S038","target_id":"S064","kind":"supports"},{"source_id":"S019","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S057","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S033","target_id":"S066","kind":"supports"},{"source_id":"S055","target_id":"S066","kind":"supports"},{"source_id":"S018","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S010","target_id":"S067","kind":"supports"},{"source_id":"S017","target_id":"S067","kind":"supports"},{"source_id":"S064","target_id":"S063","kind":"supports"},{"source_id":"S065","target_id":"S063","kind":"supports"},{"source_id":"S066","target_id":"S063","kind":"supports"},{"source_id":"S067","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S062","kind":"supports"}]}}
{"record_id":"2103.02855v4","paper":{"title":"A Semismooth Newton based Augmented Lagrangian Method for Nonsmooth Optimization on Matrix Manifolds","authors":["Yuhao Zhou","Chenglong Bao","Chao Ding","Jun Zhu"],"arxiv_id":"2103.02855","version":"v4","journal":"Mathematical Programming","doi":"10.1007/s10107-022-01898-1","abstract_url":"https://arxiv.org/abs/2103.02855v4","pdf_url":"https://arxiv.org/pdf/2103.02855v4"},"context":{"raw_tex":"\t\t\t$[u]_i$ & The $i$-th component of $u\\in\\R^d$ \\\\ \\hline\n\t\t\t$[M]$ & The set $\\{1,2,\\cdots,M\\}$, where $M$ is a positive integer \\\\ \\hline\n\t\t\t$\\cM$ & A complete $n$-dimensional smooth Riemannian manifold \\\\ \n\t\t\t\\hline\n\t\t\t$T_p\\cM$ & The tangent space at $p\\in\\cM$ \\\\ \\hline\n\t\t\t$T\\cM$ & The tangent bundle of $\\cM$ \\\\ \\hline\n\t\t\t$\\cD(\\cM)$ & The set of smooth functions on $\\cM$ with compact support \\\\ \\hline \n\t\t\t$\\dd\\varphi|_p$ & The differential of the smooth map $\\varphi$ at $p\\in\\cM$  \\\\ \\hline\n\t\t\t$\\grad\\varphi$ & The gradient of the function $\\varphi$ on manifolds\\\\ \\hline\n\t\t\t$\\partial \\varphi$ & The Clarke subgradient of the function $\\varphi$ on manifolds \\\\ \\hline\n\n\n\t\t\t$\\inner{\\eta}{\\xi}_p$,$\\inner{\\eta}{\\xi}$ & The Riemannian inner product of $\\eta, \\xi \\in T_p\\cM$  \\\\ \\hline\n\t\t\t$\\norm{\\eta}_p$, $\\norm{\\eta}$ & The norm of $\\eta \\in T_p\\cM$  \\\\ \\hline\n\n\nA Hausdorff topological space $\\cM$ is said to be an \\emph{$n$-dimensional manifold} if it has a countable basis and for each $p \\in \\cM$ there exist a neighborhood $U$ of $p$, an open subset $\\hat U \\subset \\R^n$ and a map $\\varphi: U \\to \\hat U$ such that $\\varphi$ is a homeomorphism. The pair $(U, \\varphi)$ is called a \\emph{chart}.\n\n\n\\begin{definition}  %\n    A \\emph{tangent vector} $\\xi_p:\\cD(\\cM)\\to\\R$ to a manifold $\\cM$ at a point $p$ is a linear operator such that for every $f \\in \\cD(\\cM)$, %\n        $\\xi_p f = \\dot\\gamma(0) f:= \\left.\\frac{\\dd(f(\\gamma(t)))}{\\dd t}\\right|_{t=0}$,\n    where $\\gamma: (-1, 1) \\to \\cM$ is a smooth curve on $\\cM$ with $\\gamma(0)=p$.\n\\end{definition}\n\nThe \\emph{tangent space} $T_p\\cM$ is the space containing all tangent vectors at $p$, which is an $n$-dimensional $\\R$-linear space. \nA \\emph{Riemannian metric} $\\langle\\cdot,\\cdot\\rangle_p$ gives an inner product on $T_p\\cM$, which smoothly depends on~$p$.\\footnote{The subscript $p$ in $\\inner{\\cdot}{\\cdot}_p$ is usually omitted for simplicity.}\nMoreover, a Riemannian metric gives a metric on $\\cM$ and the gradient of functions defined on $\\cM$.\nBelow we assume that $\\cM$ is equipped with a Riemannian metric.\n\\begin{definition} %\n\tGiven $p,q\\in\\cM$, the \\emph{distance between $p$ and $q$} is defined as \n\t\\begin{equation*}\n\td(p,q) = \\inf \\left\\{ \\ell(\\gamma)  :  \\ell(\\gamma) := \\int_0^1 \\sqrt{\\inner{\\dot \\gamma(t)}{\\dot\\gamma(t)}} \\dd t \\right\\},\n\t\\end{equation*}\n\twhere $\\inf$ is taken over all piecewise smooth curves $\\gamma:[0,1]\\to\\cM$  with $\\gamma(0)=p$ and $\\gamma(1)=q$.\n\\end{definition}\n\n\\begin{definition}\n    Let $f: \\cM \\to \\cN$ be a smooth map between smooth manifolds $\\cM, \\cN$, the \\emph{differential of $f$ at $p\\in\\cM$}, denoted by $\\dd f|_p$, is a map from $T_p\\cM$ to $T_{f(p)}\\cN$ such that $(\\dd f|_p\\eta) g:= \\eta(g \\circ f)$ for all $g \\in \\cD(\\cN)$ and $\\eta \\in T_p\\cM$.\n\\end{definition}\n\n\\begin{definition}%\n\t\\label{def:riemannian-gradient}\n\tLet $f:\\cM\\to\\R$ be a smooth function \n\tand $p\\in\\cM$, the \\emph{gradient} of $f$ at $p$ is defined as the unique tangent vector $\\grad f(p)\\in T_p\\cM$ that satisfies \n\t\t $$\\xi_p f = \\inner{\\xi_p}{\\grad f(p)},\\ \\forall\\xi_p \\in T_p\\cM.$$\n\\end{definition}\n\nSuppose $f: \\cM \\to \\R$ is a locally Lipschitz function\\footnote{We say a function $f$ on a manifold is locally Lipschitz if $f \\circ \\varphi^{-1}$ is locally Lipschitz in $U$ for every chart $(U, \\varphi)$.} on a manifold $\\cM$ and $(U, \\varphi)$ is a chart containing $p \\in \\cM$. \nFrom ~\\cite{azagra2005nonsmooth,dirr2007nonsmooth,hosseini2011generalized}, the \\emph{Clarke generalized directional derivative}  of $f$ at $p$, denoted by $f^\\circ(p; v)$, is defined by\n\\[ f^\\circ(p; v) := \\limsup_{y \\to p, t \\downarrow 0} \\frac{\\hat f(\\varphi(y) + t \\dd\\varphi|_pv) - \\hat f(\\varphi(y))}{t}, \\]\nwhere $\\hat f := f \\circ \\varphi^{-1}$ and $\\dd\\varphi|_p$ is the differential of $\\varphi$ at $p$. The \\emph{Clarke subgradient} is \n$$\\partial f(p) := \\{  \\xi \\in T_p\\cM : \\inner{\\xi}{v} \\leq f^\\circ (p; v),\\ \\forall v \\in T_p \\cM \\}.$$\n\n\n\\begin{equation}\n\t\\label{eqn:original-problem}\n\t\\begin{aligned}\n\t\t\\min_{x}\\  &\\{f(x) + \\psi(h_1(x)) \\}, \\quad \n\t\t\\mathrm{s.t.}\\  x \\in \\cM,\\ h_2(x) \\leq 0.\n\t\\end{aligned}\n\\end{equation}\n\n\\begin{assumption}\n\t$\\cM$ is a complete smooth Riemannian manifold.\n\\end{assumption}\n\n\\begin{assumption}\n\t$f : \\cM \\to \\R$, $h_1: \\cM \\to \\R^m$, $h_2: \\cM \\to \\R^{q}$ are continuously differentiable, and~$\\psi: \\R^m \\to \\R$ is a convex function.\n\t$f(x) + \\psi(y)$ is bounded below for $(x, y) \\in \\cM \\times \\R^m$.\n\\end{assumption}\n\n\\begin{equation}\n\t\\label{eqn:reformulated-problem}\n\t\\begin{aligned}\n\t\t\\min_{x, y, z}\\  &\\{f(x) + \\psi(y) \\}, \\quad\n\t\t\\mathrm{s.t.}\\  x \\in \\cM, \\\n\t\t y = h_1(x), \\\n\t\t z = h_2(x),\\ z \\leq 0.\n\t\\end{aligned}\n\\end{equation}\n\nThe augmented Lagrangian function $L_\\sigma: \\cM \\times \\R^m \\times \\R_-^q \\times \\R^m \\times \\R^q \\to \\R$ of \\eqref{eqn:reformulated-problem} is given by\n\\begin{equation}\n\t\\begin{aligned} \n\\peq\tL_\\sigma(x, y, z, \\lambda, \\gamma)  \n= f(x) + \\psi(y) \n+ \\frac{\\sigma}{2} \\norm{h_1(x) - y + \\frac{\\lambda}{\\sigma}}^2_2 \n+ \\frac{\\sigma}{2}\\norm{h_2(x) - z + \\frac{\\gamma}{\\sigma}}_2^2 \n- \\frac{\\norm{\\lambda}_2^2 + \\norm{\\gamma}_2^2}{2\\sigma}.\n\t\\end{aligned}\n\t\\label{eqn:L_sigma}\n\\end{equation}\n\nwhere $\\psi^\\sigma$, $\\delta_{\\R^q_-}^\\sigma$ are the \\emph{Moreau-Yosida regularization} of $\\psi$, $\\delta_{\\R^q_-}$, respectively. \nMore specifically, it holds that\n\\begin{align}\n    \\psi^\\sigma(x) &:= \\min_{y \\in \\R^m}  \\left \\{ \\psi(y)  \n    + \\frac{\\sigma}{2} \\norm{x - y}^2_2 \\right \\}, \\label{prob:y}\\\\\n    \\delta_{\\R_-^q}^\\sigma(x) &:= \\min_{z \\in \\R^q} \\left \\{ \\delta_{\\R_-^q}(z) + \\frac{\\sigma}{2}\\norm{x - z}_2^2 \\right \\}, \\label{prob:z}\n\\end{align}\n\nwhere $\\delta_{\\R_-^q}$ is the indicator function of $\\R_-^q$. \n\n\n\\begin{alg}[An augmented Lagrangian method for solving \\eqref{eqn:reformulated-problem}]\n\t\\label{alg:alm-outer}\n\tChoose initial values $x_0 \\in \\cM$, $\\gamma_0 \\in \\R^{q}_+$, $\\lambda_0 \\in \\R^m$, $\\sigma_0 > 0$, $\\alpha, \\tau \\in (0, 1)$, $\\rho > 1$ \n\tand a sequence $\\{ \\varepsilon_k \\} \\subseteq \\R_+$ converging to $0$.\n\tLet $y_0 = \\prox_{\\psi/\\sigma_0} ( h_1(x_0) + \\lambda_0 / \\sigma_0)$, $z_0 = \\Pi_{\\R_-^q} ( h_2(x_0) + \\gamma_0 / \\sigma_0)$, where $\\Pi_{\\R_-^q}$ is the projection onto $\\R_-^q$. %\n\tChoose a feasible point $x_\\feas$ and a constant $\\Phi$ such that\n\t\\begin{equation}\n\t\t\\label{eqn:initial-maximal}\n\t\t\\Phi \\geq \\max\\{  f(x_\\feas) + \\psi(h_1(x_\\feas)),\\ L_{\\sigma_0}(x_0, y_0, z_0, \\lambda_0, \\gamma_0)  \\}.\n\t\\end{equation}\n\tOur algorithm repeats the following steps for $k = 1, 2, \\dots$\n\n    \\begin{enumerate}[label=(\\roman*)]\n\t\\item\n\tFind $x_k\\in\\cM$ such that\n\t\\begin{equation}\n\t\t\\label{eqn:subproblem-stopping-rule}\n\t\t\\norm{\\grad L_k (x_k)} < \\varepsilon_k, \\quad L_k(x_k) \\leq \\Phi,\n\t\\end{equation}\n\twhere \n\t\\begin{equation}\n\tL_k(x) := f(x) \n\t+ \\psi^{\\sigma_k} \\left ( h_1(x) + \\frac{\\lambda_k}{\\sigma_k}  \\right )\n\t+ \\delta_{\\R_-^q}^{\\sigma_k} \\left ( h_2(x) + \\frac{\\gamma_k}{\\sigma_k}  \\right ).\n\t\\label{eqn:Lk}\n\t\\end{equation}\n\t\\item Update $y$ and $z$ using\n    \\begin{align}\n        \\label{eqn:subproblem-y}\n        y_k &= \\prox_{\\psi/\\sigma_k}\\left ( h_1(x_k) + \\frac{\\lambda_k}{\\sigma_k}  \\right ),\\\\\n        \\label{eqn:subproblem-z}\n        z_k &= \\Pi_{\\R_-^q} \\left ( h_2(x_k) +  \\frac{\\gamma_k}{ \\sigma_k} \\right).\n    \\end{align}\n        \n\t\\item %\n\tUpdate the multipliers:\n    \\[ \\begin{aligned}\n        \\lambda_{k+1} &= \\lambda_k + \\sigma_k (h_1(x_k) - y_k), \\quad \n        \\gamma_{k+1} = \\gamma_k +\\sigma_k (h_2(x_k) - z_k).\n    \\end{aligned} \\]\n\n\t\\item\n\tLet \n        $     \\delta_k = \\max\\left\\{ \\norm{h_1(x_k) - y_k}_2,\\ \\norm{h_2(x_k)-z_k}_2\\right\\}.\n        $ %\n\n\t\t If $\\delta_k \\leq \\tau \\delta_{k-1}$, then $\\sigma_{k+1} = \\sigma_k$.\n\t\t Otherwise, set\n         \\begin{equation}\n            \\label{eqn:alm-update-penalty}\n\t\t\t\\sigma_{k+1} = \\max \\left \\{ \\rho\\sigma_k,\\ \\norm{\\lambda_{k+1}}_2^{1+\\alpha},\\ \\norm{\\gamma_{k+1}}_2^{1+\\alpha} \\right \\}. \n\t\t \\end{equation}\n\t\\end{enumerate}\n\\end{alg}\n\nConsider the problem \\eqref{eqn:original-problem}, we define the \\emph{active set} of a feasible point $x$ to be\n$\\cA(x) := \\{ i \\in [q] : [h_2(x)]_i = 0 \\}$.\n\n\n\\begin{definition}[LICQ]\n\t\\label{def:LICQ}\n\tWe say that a feasible point $x \\in \\cM$ of \\eqref{eqn:original-problem} satisfies the \\emph{linear independence constraint qualification} (LICQ) if\n    $\\{ \\grad\\, [h_2(x)]_i : i \\in \\cA(x) \\}$ are linearly independent in $T_x\\cM$.\n\\end{definition}\n\n\\begin{definition}[CPLD]\n\t\\label{def:CPLD}\n\tLet $x \\in \\cM$ be a feasible point of \\eqref{eqn:original-problem} and define\n    $S(x) := \\{ \\grad\\, [h_2(x)]_i : i \\in \\cA(x) \\}$.\n\tWe say that $x$ satisfies the \\emph{constant positive linear dependence constraint qualification} (CPLD) \n\tif for each subset $S_0(x) \\subseteq S(x)$ whose elements are linearly dependent with non-negative coefficients, $S_0$ remains linearly dependent in a neighborhood of $x$. \n\\end{definition}\n\nThe Lagrangian of the equivalent problem \\eqref{eqn:reformulated-problem} is\n\\[ \\tilde \\cL(x, y, z, \\lambda, \\gamma) = f(x) + \\psi(y) + \\lambda^\\trans(h_1(x) - y) + \\gamma^\\trans(h_2(x) - z). \\]\n\n\n    \\begin{subequations}\n        \\label{eqn:reformulated-kkt}\n        \\begin{align}\n\t\t\t y = h_1(x),\\ z = h_2(x), \\label{eqn:reformulated-kkt-feas} \\\\\n\t\t\t z \\in \\R^q_-,\\ \n            \\gamma \\in \\R^q_+,\\ \\gamma^\\trans z = 0, \\label{eqn:reformulated-kkt-z} \\\\\n            0 \\in \\partial_y \\tilde \\cL(x, y, z, \\lambda, \\gamma), \\label{eqn:reformulated-kkt-y} \\\\\n            \\mathrm{grad}_x\\, \\tilde \\cL(x, y, z, \\lambda, \\gamma) = 0. \\label{eqn:reformulated-kkt-x}\n        \\end{align}\n\t\\end{subequations}\n","units":[{"start":17527,"end":18202,"start_line":337,"end_line":346,"raw_tex":"\t\t\t$[u]_i$ & The $i$-th component of $u\\in\\R^d$ \\\\ \\hline\n\t\t\t$[M]$ & The set $\\{1,2,\\cdots,M\\}$, where $M$ is a positive integer \\\\ \\hline\n\t\t\t$\\cM$ & A complete $n$-dimensional smooth Riemannian manifold \\\\ \n\t\t\t\\hline\n\t\t\t$T_p\\cM$ & The tangent space at $p\\in\\cM$ \\\\ \\hline\n\t\t\t$T\\cM$ & The tangent bundle of $\\cM$ \\\\ \\hline\n\t\t\t$\\cD(\\cM)$ & The set of smooth functions on $\\cM$ with compact support \\\\ \\hline \n\t\t\t$\\dd\\varphi|_p$ & The differential of the smooth map $\\varphi$ at $p\\in\\cM$  \\\\ \\hline\n\t\t\t$\\grad\\varphi$ & The gradient of the function $\\varphi$ on manifolds\\\\ \\hline\n\t\t\t$\\partial \\varphi$ & The Clarke subgradient of the function $\\varphi$ on manifolds \\\\ \\hline\n","sha256_utf8":"3d76497c8e172feb0fb65fc0827d0bb4fbbc6950cbda165dfa8757b93bdae46d"},{"start":19724,"end":19914,"start_line":365,"end_line":366,"raw_tex":"\t\t\t$\\inner{\\eta}{\\xi}_p$,$\\inner{\\eta}{\\xi}$ & The Riemannian inner product of $\\eta, \\xi \\in T_p\\cM$  \\\\ \\hline\n\t\t\t$\\norm{\\eta}_p$, $\\norm{\\eta}$ & The norm of $\\eta \\in T_p\\cM$  \\\\ \\hline\n","sha256_utf8":"ebe43e8a3fe3c756c60a3fae4f5276bb0b1b60a5c8992c9400e292e8135cc2e1"},{"start":20288,"end":20626,"start_line":376,"end_line":376,"raw_tex":"A Hausdorff topological space $\\cM$ is said to be an \\emph{$n$-dimensional manifold} if it has a countable basis and for each $p \\in \\cM$ there exist a neighborhood $U$ of $p$, an open subset $\\hat U \\subset \\R^n$ and a map $\\varphi: U \\to \\hat U$ such that $\\varphi$ is a homeomorphism. The pair $(U, \\varphi)$ is called a \\emph{chart}.\n","sha256_utf8":"5fc2673a7ce724d22ab45d5c99fe30db6e51e678662523d8a7789ed3da2a33d1"},{"start":21237,"end":21594,"start_line":383,"end_line":387,"raw_tex":"\\begin{definition}  %\n    A \\emph{tangent vector} $\\xi_p:\\cD(\\cM)\\to\\R$ to a manifold $\\cM$ at a point $p$ is a linear operator such that for every $f \\in \\cD(\\cM)$, %\n        $\\xi_p f = \\dot\\gamma(0) f:= \\left.\\frac{\\dd(f(\\gamma(t)))}{\\dd t}\\right|_{t=0}$,\n    where $\\gamma: (-1, 1) \\to \\cM$ is a smooth curve on $\\cM$ with $\\gamma(0)=p$.\n\\end{definition}","sha256_utf8":"b5dc4c9bbf4c1f0902681b4e640b848af0ae27384562078316f3dc82b2dce074"},{"start":21595,"end":22501,"start_line":388,"end_line":398,"raw_tex":"The \\emph{tangent space} $T_p\\cM$ is the space containing all tangent vectors at $p$, which is an $n$-dimensional $\\R$-linear space. \nA \\emph{Riemannian metric} $\\langle\\cdot,\\cdot\\rangle_p$ gives an inner product on $T_p\\cM$, which smoothly depends on~$p$.\\footnote{The subscript $p$ in $\\inner{\\cdot}{\\cdot}_p$ is usually omitted for simplicity.}\nMoreover, a Riemannian metric gives a metric on $\\cM$ and the gradient of functions defined on $\\cM$.\nBelow we assume that $\\cM$ is equipped with a Riemannian metric.\n\\begin{definition} %\n\tGiven $p,q\\in\\cM$, the \\emph{distance between $p$ and $q$} is defined as \n\t\\begin{equation*}\n\td(p,q) = \\inf \\left\\{ \\ell(\\gamma)  :  \\ell(\\gamma) := \\int_0^1 \\sqrt{\\inner{\\dot \\gamma(t)}{\\dot\\gamma(t)}} \\dd t \\right\\},\n\t\\end{equation*}\n\twhere $\\inf$ is taken over all piecewise smooth curves $\\gamma:[0,1]\\to\\cM$  with $\\gamma(0)=p$ and $\\gamma(1)=q$.\n\\end{definition}","sha256_utf8":"f75a5792d94356c081b517687e300d7ff11574cec9cf4f868d9f51f6d4842174"},{"start":22502,"end":22818,"start_line":399,"end_line":401,"raw_tex":"\\begin{definition}\n    Let $f: \\cM \\to \\cN$ be a smooth map between smooth manifolds $\\cM, \\cN$, the \\emph{differential of $f$ at $p\\in\\cM$}, denoted by $\\dd f|_p$, is a map from $T_p\\cM$ to $T_{f(p)}\\cN$ such that $(\\dd f|_p\\eta) g:= \\eta(g \\circ f)$ for all $g \\in \\cD(\\cN)$ and $\\eta \\in T_p\\cM$.\n\\end{definition}","sha256_utf8":"f20487e0e8c206e8161d8b778b99d895cb59b7b3910e9f193b3181c2c26c8482"},{"start":22819,"end":23127,"start_line":402,"end_line":407,"raw_tex":"\\begin{definition}%\n\t\\label{def:riemannian-gradient}\n\tLet $f:\\cM\\to\\R$ be a smooth function \n\tand $p\\in\\cM$, the \\emph{gradient} of $f$ at $p$ is defined as the unique tangent vector $\\grad f(p)\\in T_p\\cM$ that satisfies \n\t\t $$\\xi_p f = \\inner{\\xi_p}{\\grad f(p)},\\ \\forall\\xi_p \\in T_p\\cM.$$\n\\end{definition}","sha256_utf8":"d73b10650fb48c4e48f98742adcaa9705700084153314e7d4b2af5dd197cfde8"},{"start":27795,"end":28632,"start_line":460,"end_line":464,"raw_tex":"Suppose $f: \\cM \\to \\R$ is a locally Lipschitz function\\footnote{We say a function $f$ on a manifold is locally Lipschitz if $f \\circ \\varphi^{-1}$ is locally Lipschitz in $U$ for every chart $(U, \\varphi)$.} on a manifold $\\cM$ and $(U, \\varphi)$ is a chart containing $p \\in \\cM$. \nFrom ~\\cite{azagra2005nonsmooth,dirr2007nonsmooth,hosseini2011generalized}, the \\emph{Clarke generalized directional derivative}  of $f$ at $p$, denoted by $f^\\circ(p; v)$, is defined by\n\\[ f^\\circ(p; v) := \\limsup_{y \\to p, t \\downarrow 0} \\frac{\\hat f(\\varphi(y) + t \\dd\\varphi|_pv) - \\hat f(\\varphi(y))}{t}, \\]\nwhere $\\hat f := f \\circ \\varphi^{-1}$ and $\\dd\\varphi|_p$ is the differential of $\\varphi$ at $p$. The \\emph{Clarke subgradient} is \n$$\\partial f(p) := \\{  \\xi \\in T_p\\cM : \\inner{\\xi}{v} \\leq f^\\circ (p; v),\\ \\forall v \\in T_p \\cM \\}.$$\n","sha256_utf8":"f9e0a4052463460b4da0d8441194d6688e7d5437e4df126fed721bc79a1880cf"},{"start":43584,"end":43769,"start_line":619,"end_line":625,"raw_tex":"\\begin{equation}\n\t\\label{eqn:original-problem}\n\t\\begin{aligned}\n\t\t\\min_{x}\\  &\\{f(x) + \\psi(h_1(x)) \\}, \\quad \n\t\t\\mathrm{s.t.}\\  x \\in \\cM,\\ h_2(x) \\leq 0.\n\t\\end{aligned}\n\\end{equation}","sha256_utf8":"d88a12c95918665659ea6d34aa0eda2d334f7a33285197d107769e28af02eea1"},{"start":43836,"end":43920,"start_line":628,"end_line":630,"raw_tex":"\\begin{assumption}\n\t$\\cM$ is a complete smooth Riemannian manifold.\n\\end{assumption}","sha256_utf8":"3e175af8e06c0209d862425465114a832647f5fa907a1e811ebd8168856c3c67"},{"start":43921,"end":44166,"start_line":631,"end_line":634,"raw_tex":"\\begin{assumption}\n\t$f : \\cM \\to \\R$, $h_1: \\cM \\to \\R^m$, $h_2: \\cM \\to \\R^{q}$ are continuously differentiable, and~$\\psi: \\R^m \\to \\R$ is a convex function.\n\t$f(x) + \\psi(y)$ is bounded below for $(x, y) \\in \\cM \\times \\R^m$.\n\\end{assumption}","sha256_utf8":"baaef443a04957783ade16d362c9283c9772451d782bcb8d828cba138061c164"},{"start":44252,"end":44470,"start_line":637,"end_line":645,"raw_tex":"\\begin{equation}\n\t\\label{eqn:reformulated-problem}\n\t\\begin{aligned}\n\t\t\\min_{x, y, z}\\  &\\{f(x) + \\psi(y) \\}, \\quad\n\t\t\\mathrm{s.t.}\\  x \\in \\cM, \\\n\t\t y = h_1(x), \\\n\t\t z = h_2(x),\\ z \\leq 0.\n\t\\end{aligned}\n\\end{equation}","sha256_utf8":"5c392586beb3bff6341a731a935518bbac18d590b7bd1323659eab07d7adff24"},{"start":44471,"end":44961,"start_line":646,"end_line":656,"raw_tex":"The augmented Lagrangian function $L_\\sigma: \\cM \\times \\R^m \\times \\R_-^q \\times \\R^m \\times \\R^q \\to \\R$ of \\eqref{eqn:reformulated-problem} is given by\n\\begin{equation}\n\t\\begin{aligned} \n\\peq\tL_\\sigma(x, y, z, \\lambda, \\gamma)  \n= f(x) + \\psi(y) \n+ \\frac{\\sigma}{2} \\norm{h_1(x) - y + \\frac{\\lambda}{\\sigma}}^2_2 \n+ \\frac{\\sigma}{2}\\norm{h_2(x) - z + \\frac{\\gamma}{\\sigma}}_2^2 \n- \\frac{\\norm{\\lambda}_2^2 + \\norm{\\gamma}_2^2}{2\\sigma}.\n\t\\end{aligned}\n\t\\label{eqn:L_sigma}\n\\end{equation}","sha256_utf8":"76b20126ece98a36e501c851506d19c82377694ca3b5fc29aec4effbb325e68b"},{"start":45296,"end":45759,"start_line":665,"end_line":671,"raw_tex":"where $\\psi^\\sigma$, $\\delta_{\\R^q_-}^\\sigma$ are the \\emph{Moreau-Yosida regularization} of $\\psi$, $\\delta_{\\R^q_-}$, respectively. \nMore specifically, it holds that\n\\begin{align}\n    \\psi^\\sigma(x) &:= \\min_{y \\in \\R^m}  \\left \\{ \\psi(y)  \n    + \\frac{\\sigma}{2} \\norm{x - y}^2_2 \\right \\}, \\label{prob:y}\\\\\n    \\delta_{\\R_-^q}^\\sigma(x) &:= \\min_{z \\in \\R^q} \\left \\{ \\delta_{\\R_-^q}(z) + \\frac{\\sigma}{2}\\norm{x - z}_2^2 \\right \\}, \\label{prob:z}\n\\end{align}","sha256_utf8":"fff39cf7d3cf2ec0ac826bb0f445b180061b0bc3a9ed275e772a6ec37561fa77"},{"start":45760,"end":45824,"start_line":672,"end_line":672,"raw_tex":"where $\\delta_{\\R_-^q}$ is the indicator function of $\\R_-^q$. \n","sha256_utf8":"83384c2de53fd4c1a061294cff82295410aa6ee140ad6fe83ae60eae479e8238"},{"start":47100,"end":49341,"start_line":690,"end_line":743,"raw_tex":"\\begin{alg}[An augmented Lagrangian method for solving \\eqref{eqn:reformulated-problem}]\n\t\\label{alg:alm-outer}\n\tChoose initial values $x_0 \\in \\cM$, $\\gamma_0 \\in \\R^{q}_+$, $\\lambda_0 \\in \\R^m$, $\\sigma_0 > 0$, $\\alpha, \\tau \\in (0, 1)$, $\\rho > 1$ \n\tand a sequence $\\{ \\varepsilon_k \\} \\subseteq \\R_+$ converging to $0$.\n\tLet $y_0 = \\prox_{\\psi/\\sigma_0} ( h_1(x_0) + \\lambda_0 / \\sigma_0)$, $z_0 = \\Pi_{\\R_-^q} ( h_2(x_0) + \\gamma_0 / \\sigma_0)$, where $\\Pi_{\\R_-^q}$ is the projection onto $\\R_-^q$. %\n\tChoose a feasible point $x_\\feas$ and a constant $\\Phi$ such that\n\t\\begin{equation}\n\t\t\\label{eqn:initial-maximal}\n\t\t\\Phi \\geq \\max\\{  f(x_\\feas) + \\psi(h_1(x_\\feas)),\\ L_{\\sigma_0}(x_0, y_0, z_0, \\lambda_0, \\gamma_0)  \\}.\n\t\\end{equation}\n\tOur algorithm repeats the following steps for $k = 1, 2, \\dots$\n\n    \\begin{enumerate}[label=(\\roman*)]\n\t\\item\n\tFind $x_k\\in\\cM$ such that\n\t\\begin{equation}\n\t\t\\label{eqn:subproblem-stopping-rule}\n\t\t\\norm{\\grad L_k (x_k)} < \\varepsilon_k, \\quad L_k(x_k) \\leq \\Phi,\n\t\\end{equation}\n\twhere \n\t\\begin{equation}\n\tL_k(x) := f(x) \n\t+ \\psi^{\\sigma_k} \\left ( h_1(x) + \\frac{\\lambda_k}{\\sigma_k}  \\right )\n\t+ \\delta_{\\R_-^q}^{\\sigma_k} \\left ( h_2(x) + \\frac{\\gamma_k}{\\sigma_k}  \\right ).\n\t\\label{eqn:Lk}\n\t\\end{equation}\n\t\\item Update $y$ and $z$ using\n    \\begin{align}\n        \\label{eqn:subproblem-y}\n        y_k &= \\prox_{\\psi/\\sigma_k}\\left ( h_1(x_k) + \\frac{\\lambda_k}{\\sigma_k}  \\right ),\\\\\n        \\label{eqn:subproblem-z}\n        z_k &= \\Pi_{\\R_-^q} \\left ( h_2(x_k) +  \\frac{\\gamma_k}{ \\sigma_k} \\right).\n    \\end{align}\n        \n\t\\item %\n\tUpdate the multipliers:\n    \\[ \\begin{aligned}\n        \\lambda_{k+1} &= \\lambda_k + \\sigma_k (h_1(x_k) - y_k), \\quad \n        \\gamma_{k+1} = \\gamma_k +\\sigma_k (h_2(x_k) - z_k).\n    \\end{aligned} \\]\n\n\t\\item\n\tLet \n        $     \\delta_k = \\max\\left\\{ \\norm{h_1(x_k) - y_k}_2,\\ \\norm{h_2(x_k)-z_k}_2\\right\\}.\n        $ %\n\n\t\t If $\\delta_k \\leq \\tau \\delta_{k-1}$, then $\\sigma_{k+1} = \\sigma_k$.\n\t\t Otherwise, set\n         \\begin{equation}\n            \\label{eqn:alm-update-penalty}\n\t\t\t\\sigma_{k+1} = \\max \\left \\{ \\rho\\sigma_k,\\ \\norm{\\lambda_{k+1}}_2^{1+\\alpha},\\ \\norm{\\gamma_{k+1}}_2^{1+\\alpha} \\right \\}. \n\t\t \\end{equation}\n\t\\end{enumerate}\n\\end{alg}","sha256_utf8":"d7c250017c183a613acdb4f41459140390f132fdb4f914ed241a65d5790b1e91"},{"start":53265,"end":53424,"start_line":799,"end_line":800,"raw_tex":"Consider the problem \\eqref{eqn:original-problem}, we define the \\emph{active set} of a feasible point $x$ to be\n$\\cA(x) := \\{ i \\in [q] : [h_2(x)]_i = 0 \\}$.\n","sha256_utf8":"934fe422ece68974578c8d0ffa1b687c797204768494c522cb5e87c1cf6d787d"},{"start":53508,"end":53801,"start_line":802,"end_line":806,"raw_tex":"\\begin{definition}[LICQ]\n\t\\label{def:LICQ}\n\tWe say that a feasible point $x \\in \\cM$ of \\eqref{eqn:original-problem} satisfies the \\emph{linear independence constraint qualification} (LICQ) if\n    $\\{ \\grad\\, [h_2(x)]_i : i \\in \\cA(x) \\}$ are linearly independent in $T_x\\cM$.\n\\end{definition}","sha256_utf8":"6a7c1889cc817d416a7f930e5c98d582b011b8ff199abf0965583891604f2343"},{"start":53989,"end":54464,"start_line":810,"end_line":816,"raw_tex":"\\begin{definition}[CPLD]\n\t\\label{def:CPLD}\n\tLet $x \\in \\cM$ be a feasible point of \\eqref{eqn:original-problem} and define\n    $S(x) := \\{ \\grad\\, [h_2(x)]_i : i \\in \\cA(x) \\}$.\n\tWe say that $x$ satisfies the \\emph{constant positive linear dependence constraint qualification} (CPLD) \n\tif for each subset $S_0(x) \\subseteq S(x)$ whose elements are linearly dependent with non-negative coefficients, $S_0$ remains linearly dependent in a neighborhood of $x$. \n\\end{definition}","sha256_utf8":"0aa9efe63cbe41ca14aa2a4c2efe8e9b06ed6b0152b5989554c38a677b66df72"},{"start":55400,"end":55595,"start_line":834,"end_line":835,"raw_tex":"The Lagrangian of the equivalent problem \\eqref{eqn:reformulated-problem} is\n\\[ \\tilde \\cL(x, y, z, \\lambda, \\gamma) = f(x) + \\psi(y) + \\lambda^\\trans(h_1(x) - y) + \\gamma^\\trans(h_2(x) - z). \\]\n","sha256_utf8":"490813a9907096a5c9e1f860ed87544a38912152cd368e24eb56529e1c115568"},{"start":55709,"end":56210,"start_line":837,"end_line":846,"raw_tex":"    \\begin{subequations}\n        \\label{eqn:reformulated-kkt}\n        \\begin{align}\n\t\t\t y = h_1(x),\\ z = h_2(x), \\label{eqn:reformulated-kkt-feas} \\\\\n\t\t\t z \\in \\R^q_-,\\ \n            \\gamma \\in \\R^q_+,\\ \\gamma^\\trans z = 0, \\label{eqn:reformulated-kkt-z} \\\\\n            0 \\in \\partial_y \\tilde \\cL(x, y, z, \\lambda, \\gamma), \\label{eqn:reformulated-kkt-y} \\\\\n            \\mathrm{grad}_x\\, \\tilde \\cL(x, y, z, \\lambda, \\gamma) = 0. \\label{eqn:reformulated-kkt-x}\n        \\end{align}\n\t\\end{subequations}\n","sha256_utf8":"28bfed5fc2c03e1a79187de7bc4d158216af78983a014cf87d9822b15cf5ac8e"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\n    \\label{thm:kkt}\n    Suppose there exist $K \\subseteq \\N$ and $(x_*, y_*, z_*) \\in \\cM \\times \\R^m \\times \\R^q_-$ such that\n    \\[ \\lim_{K \\ni k \\to \\infty} \\left ( d(x_k, x_*) + \\norm{y_k - y_*}_2 + \\norm{z_k - z_*}_2 \\right ) = 0. \\]\n    If the CPLD condition holds at $x_*$, then there exist $K_0 \\subseteq K$ and $\\lambda_* \\in \\R^m, \\gamma_* \\in \\R^q_+$ such that $\\lim_{K_0 \\ni k \\to \\infty} \\lambda_{k+1} = \\lambda_*$, and the KKT conditions \\eqref{eqn:reformulated-kkt} hold at $(x_*, y_*, z_*, \\lambda_*, \\gamma_*)$. Moreover, when the LICQ holds at $x_*$, we can choose $\\gamma_*$ such that $\\lim_{K_0 \\ni k \\to \\infty} \\gamma_{k + 1} = \\gamma_*$.\n\\end{theorem}","units":[{"start":59440,"end":60130,"start_line":891,"end_line":896,"raw_tex":"\\begin{theorem}\n    \\label{thm:kkt}\n    Suppose there exist $K \\subseteq \\N$ and $(x_*, y_*, z_*) \\in \\cM \\times \\R^m \\times \\R^q_-$ such that\n    \\[ \\lim_{K \\ni k \\to \\infty} \\left ( d(x_k, x_*) + \\norm{y_k - y_*}_2 + \\norm{z_k - z_*}_2 \\right ) = 0. \\]\n    If the CPLD condition holds at $x_*$, then there exist $K_0 \\subseteq K$ and $\\lambda_* \\in \\R^m, \\gamma_* \\in \\R^q_+$ such that $\\lim_{K_0 \\ni k \\to \\infty} \\lambda_{k+1} = \\lambda_*$, and the KKT conditions \\eqref{eqn:reformulated-kkt} hold at $(x_*, y_*, z_*, \\lambda_*, \\gamma_*)$. Moreover, when the LICQ holds at $x_*$, we can choose $\\gamma_*$ such that $\\lim_{K_0 \\ni k \\to \\infty} \\gamma_{k + 1} = \\gamma_*$.\n\\end{theorem}","sha256_utf8":"504bd65f889ab9de001d31fdf204e17afd8d21ec541f1e5a7e50db6f38cb765c"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{proposition}[Theorem 4.1.4 in \\cite{hiriart1993convex}]\n\t\\label{prop:moreau-yosida}\n\tLet $\\sigma > 0$ and $f: \\R^n \\to \\R \\cup \\{ +\\infty \\}$ be a closed proper convex function, then the Moreau-Yosida regularization $f^\\sigma$ is continuously differentiable, and its gradient is \n\t\\begin{equation}\n\t\t\\label{eqn:moreau-yosida-gradient}\n\t\t\\nabla f^\\sigma(x) = \\frac{1}{\\sigma}(x - \\prox_{f/\\sigma}(x)),\n\t\\end{equation}\n\twhere $\\prox_{f / \\sigma}$ is the \\emph{proximal map} of the function $f / \\sigma$:\n\t \\begin{equation}\n\t\t\\prox_{f / \\sigma} (x) := \\argmin_{y} \\left \\{ f(y) + \\frac{\\sigma}{2}\\norm{x-y}_2^2 \\right \\}.\n\t\\end{equation}\n\\end{proposition}\n\n\\begin{theorem}\n\t\\label{thm:feasibility}\n\tLet $\\{ (x_k, y_k, z_k) \\}$ be the sequence generated by Algorithm~\\ref{alg:alm-outer}.\n\tThen, we have\n\t\\[ \\lim_{k\\to \\infty} \\left( \\norm{h_1(x_k) - y_k}_2 + \\norm{h_2(x_k) - z_k}_2 \\right) = 0. \\]\n\tConsequently, if $x_* \\in \\cM$ is an accumulation point of $\\{ x_k \\}$, then $(x_*, h_1(x_*), h_2(x_*))$ is a feasible accumulation point of $\\left\\{ (x_k, y_k, z_k) \\right\\}$. %\n\tMoreover, $\\{ x_k \\}$ always contains an accumulation point if $\\cM$ is compact.\n\\end{theorem}\n\n\\begin{proof}\n    First, consider the case where $\\{ \\sigma_k \\}$ is bounded. There exists $k_1 \\in \\N$ such that $\\delta_{k+1} \\leq \\tau \\delta_k$ for any $k \\geq k_1$.\n    Then, $\\lim_{k \\to \\infty} \\delta_k = 0$ since $\\tau < 1$. \n    By the definition of $\\delta_k$, we know $\\lim_{k \\to \\infty} \\norm{h_1(x_k) - y_k}_2 = 0$ and $\\lim_{k\\to \\infty} \\norm{h_2(x_k) - z_k}_2 = 0$. \n\n\tIn the case where $\\{ \\sigma_k \\}$ is unbounded. By the update rule, $\\sigma_k$ is updated for infinitely many times. Then, we can find $k_1 < k_2 < \\cdots$ such that\n\t\\[ \\begin{aligned}\n\t\t\t\\sigma_k = \\sigma_{k_i} &= \\max\\big\\{ \\rho \\sigma_{k_{i-1}}, \\norm{\\lambda_{k_i}}_2^{1 + \\alpha}, \\norm{\\gamma_{k_i}}_2^{1 + \\alpha} \\big\\},\\ \\forall \\, k_i \\leq k < k_{i+1}.\n\t\\end{aligned} \\]\n\n\n\tFrom \\eqref{eqn:subproblem-stopping-rule} and the definition of $L_k(x)$, we know that $L_{\\sigma_{k}}(x_{k}, y_{k}, z_{k}, \\lambda_{k}, \\gamma_{k}) \\leq \\Phi$, where $L_{\\sigma_k}$ is defined in \\eqref{eqn:L_sigma}. Therefore, we have\n\t\\begin{equation}\n\t\t\\norm{h_1(x_{k}) - y_{k} + \\frac{\\lambda_{k}}{\\sigma_{k}}}^2_2 \n\t\t+ \\norm{h_2(x_{k}) - z_{k} + \\frac{\\gamma_{k}}{\\sigma_{k}}}_2^2 \n\t\t\\leq 2\\frac{ \\Phi - f(x_{k}) - \\psi(y_{k}) }{\\sigma_{k}} \n\t\t+ \\frac{\\norm{\\lambda_{k}}_2^2 + \\norm{\\gamma_{k}}_2^2}{\\sigma_{k}^2}.\n\t\t\\label{eqn:feasibility-bounded}\n\t\\end{equation}\n\n\tSince $f + \\psi$ is bounded below and $\\sigma_{k} \\to \\infty$, then the first term of \\eqref{eqn:feasibility-bounded} converges to $0$. Next, we show that the second term also converges to $0$.\n\tNotice that $\\norm{\\lambda_{k_i}}_2^{1+\\alpha} \\leq \\sigma_{k_i}$ and $\\sigma_{k_i} \\to \\infty$, \n\twe have $\\norm{\\lambda_{k_i}}_2 / \\sigma_{k_i} \\leq \\sigma_{k_i}^{-\\frac{\\alpha}{1 + \\alpha}} \\to 0$ as $i \\to \\infty$. A similar result also holds for $\\norm{\\gamma_{k_i}}_2 / \\sigma_{k_i}$. \n\tThen by \\eqref{eqn:feasibility-bounded} and the definition of $\\delta_{k_i}$, we have $\\lim_{i \\to \\infty} \\delta_{k_i} = 0$.\n\tBy the update rule of $\\lambda_k$, for $k_i < k < k_{i+1}$, it holds that \n\t\\[\\begin{aligned}\n\t\t\t\\frac{\\norm{\\lambda_{k}}_2}{\\sigma_{k}} \n\t\t\t&\\leq \\frac{\\norm{\\lambda_{k - 1}}_2}{\\sigma_{k_i}} + \\norm{h_1(x_{k - 1}) - y_{k - 1}}_2 \n\t\t\t\\leq \\frac{\\norm{\\lambda_{k - 1}}_2}{\\sigma_{k_i}} + \\delta_{k-1}.\n\t\\end{aligned} \\]\n\n\tAlso note that $\\delta_k \\leq \\tau\\delta_{k-1}$ for all $k_i \\leq k < k_{i+1} - 1$. Then, for $k_i < k < k_{i+1}$ the following inequality holds by induction\n\t\\[\\begin{aligned}\n\t\t\t\\frac{\\norm{\\lambda_{k}}_2}{\\sigma_{k}} \n\t\t\t&\\leq \\frac{\\norm{\\lambda_{k_i}}_2}{\\sigma_{k_i}} + \\sum_{j=k_i}^{k-1} \\delta_j\n\t\t\t\\leq \\frac{\\norm{\\lambda_{k_i}}_2}{\\sigma_{k_i}} + \\delta_{k_i} \\sum_{j=0}^{k - k_i -1} \\tau^{j}\n\t\t\t\\leq \\frac{\\norm{\\lambda_{k_i}}_2}{\\sigma_{k_i}} + \\frac{\\delta_{k_i}}{1 - \\tau}.\n\t\\end{aligned} \\]\n\n\tThus, we have $\\lim_{k \\to \\infty} \\norm{\\lambda_k}_2 / \\sigma_k = 0$.\n\tSimilarly, $\\lim_{k \\to \\infty} \\norm{\\gamma_k}_2 / \\sigma_k = 0$. Therefore, from \\eqref{eqn:feasibility-bounded}, we conclude that $\\lim_{k \\to \\infty} \\delta_k = 0$. \\hfill $\\Box$\n\\end{proof}\n\n\\begin{proof}\n    First, from Theorem~\\ref{thm:feasibility}, we obtain the feasibility condition \\eqref{eqn:reformulated-kkt-feas}.\n    From \\eqref{eqn:subproblem-y}, \\eqref{eqn:subproblem-z} and the property of Moreau-Yosida regularizations in Proposition~\\ref{prop:moreau-yosida}, we know that\n    \\[ \\begin{aligned} \n        y_k %\n        = h_1(x_k) +  \\frac{\\lambda_k}{\\sigma_k} - \\frac{1}{\\sigma_k} \\nabla \\psi^{\\sigma_k}\\left (h_1(x_k) +  \\frac{\\lambda_k}{\\sigma_k}\\right), \\text{ and }\n        z_k %\n        = h_2(x_k) +  \\frac{\\gamma_k}{\\sigma_k} - \\frac{1}{\\sigma_k} \\nabla \\delta_{\\R_-^q}^{\\sigma_k}\\left (h_2(x_k) +  \\frac{\\gamma_k }{ \\sigma_k}\\right). \n\t\\end{aligned} \\]\n    From the definition of $\\lambda_{k+1}$ and $\\gamma_{k+1}$, combining the above two equations, we have\n\t\\begin{equation}\n\t\t\\label{eqn:grad_Lk}\n\t\t\\begin{aligned}\n\t\t\t& \\grad L_k(x_k)\n\t\t\t= \\grad f(x_k) + \\sum_{i=1}^m [\\lambda_{k+1}]_i \\grad\\,[h_1(x_k)]_i \n\t\t\t+ \\sum_{i=1}^q [\\gamma_{k+1}]_i \\grad\\, [h_2(x_k)]_i,  \n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere the chain rule for gradients on manifolds is used\\footnote{\n\tLet $g: \\cM \\to \\R^m$ and $f: \\R^m \\to \\R$ be two smooth maps,\n\twe can use Definition~\\ref{def:riemannian-gradient} to compute the gradient of $f \\circ g$.\n\tFor $p \\in \\cM$, $\\xi_p \\in T_p\\cM$, choosing any curve $\\gamma: (-1, 1) \\to \\cM$ such that $\\gamma(0) = p$ and $\\dot \\gamma(0) = \\xi_p$, we have\n\t$\\inner{\\xi_p}{\\grad\\,(f\\circ g)(p)} = \\xi_p (f \\circ g) \n\t= \\left.\\frac{\\dd (f \\circ g \\circ \\gamma)(t)}{\\dd t}\\right|_{t=0} \n\t= \\sum_{i=1}^m\\alpha_i\\left.\\frac{\\dd (g_i \\circ \\gamma)(t)}{\\dd t}\\right|_{t=0} \n\t= \\sum_{i=1}^m \\alpha_i \\cdot (\\xi_p g_i) \n\t= \\sum_{i=1}^m \\alpha_i \\inner{\\xi_p}{\\grad\\,[g(p)]_i}$, where $\\alpha := \\nabla f(q)$, $q := g(\\gamma(t))$ and $g_i$ is the $i$-th component of $g$.\n\tThus, $\\grad\\, (f \\circ g)(p) = \\sum_{i=1}^m \\alpha_i \\grad\\, [g(p)]_i$.\n\t}.\n\n    Since $y_k = \\prox_{\\psi/\\sigma_k} (h_1(x_k) +  \\lambda_k/\\sigma_k)$, then\n\t\\begin{equation}\n\t\t\\label{eqn:grad_yk}\n\t\t0 \\in \\partial \\psi(y_k) - (\\sigma_k(h_1(x_k) - y_k) +  \\lambda_k) = \\partial \\psi(y_k) - \\lambda_{k+1}. \n\t\\end{equation}\n\tNote that $\\lim_{K \\ni k \\to \\infty} y_k = y_*$, then $\\left\\{ y_k \\right\\}_{k\\in K}$ is bounded.\n\tBy the locally boundedness of the subdifferential~\\cite[Corollary 24.5.1]{rockafellar1970convex}, $\\bigcup_{k \\in K} \\partial \\psi(y_k)$ is also bounded.\n\tTherefore, $\\{ \\lambda_{k+1} \\}_{k \\in K}$ is bounded, so we can choose $K_1 \\subseteq K$, $\\lambda_* \\in \\R^m$ such that $\\lim_{ K_1 \\ni k \\to \\infty} \\lambda_{k+1} = \\lambda_*$, which implies $\\lambda_*\\in\\partial\\psi(y_*)$.\n\n\tOn the other hand, since \n\t\\[ z_k = \\Pi_{\\R_-^q} \\left ( h_2(x_k) +  \\frac{\\gamma_k}{\\sigma_k} \\right ) = \\argmin_{z \\leq 0} \\norm{h_2(x_k) +  \\frac{\\gamma_k}{\\sigma_k} - z}^2_2, \\]\nthen from the optimality condition of the above problem, it holds that\n\t\\begin{equation}\n\t\t\\label{eqn:grad_zk}\n\t\t0 = [\\sigma_k( h_2(x_k) - z_k) +  \\gamma_k]^\\trans z_k = \\gamma_{k+1}^\\trans z_k \\quad \\mathrm{and} \\quad \\gamma_{k+1} \\geq 0.\n\t\\end{equation}\n\n\n\n\tLet $\\cA_{k} = \\{ i \\in [q] : [z_k]_i = 0 \\}$, from \\eqref{eqn:grad_zk}, we know $[\\gamma_{k+1}]_i = 0$ for $i \\notin \\cA_k$. \n\tDefine $\\cA_* := \\{ i \\in [q] : [z_*]_i = 0 \\}$.\n\tSince $z_k \\to z_*$, we also find that for sufficiently large $k$ and $i \\notin \\cA_*$, $[\\gamma_{k+1}]_i = 0$.\n\tThen, when $k$ is large enough \\eqref{eqn:grad_Lk} can be written as \n\t\\begin{equation}\n\t\t\\label{eqn:alm-Astar-equ}\n\t\t\t\\grad L_k(x_k)\n\t\t\t= \\grad f(x_k) + \\sum_{i=1}^m [\\lambda_{k+1}]_i \\grad\\,[h_1(x_k)]_i \n\t\t\t+ \\sum_{i \\in \\cA_*} [\\gamma_{k+1}]_i \\grad\\, [h_2(x_k)]_i.  \n    \\end{equation}\n\tUsing the Carath\\'eodory's theorem of cones~\\cite{bertsekas1997nonlinear}, there exist $J_k \\subseteq \\cA_*$ and $[\\hat \\gamma_k]_j \\geq 0$, where $j \\in J_k$, such that $\\left\\{ \\grad\\, [h_2(x_k)]_j : j \\in J_k \\right\\}$ are linearly independent and\n\t\\begin{equation}\n\t\t\\label{eqn:alm-J_k-equ}\n\t\t\t\\grad L_k(x_k)\n\t\t\t= \\grad f(x_k) + \\sum_{i=1}^m [\\lambda_{k+1}]_i \\grad\\,[h_1(x_k)]_i \n\t\t\t+ \\sum_{j \\in J_k} [\\hat\\gamma_{k}]_j \\grad\\, [h_2(x_k)]_j,\n\t\\end{equation}\n\tSince $J_k \\subseteq [q]$ is a finite set, we can choose $J_*$ and $K_2 \\subseteq K_1$ such that $J_k = J_*$ for all $k \\in K_2$ and $|K_2| = \\infty$.\n\tDefine $M_k = \\max\\left\\{ [\\hat\\gamma_{k}]_i : i \\in J_* \\right\\}$ for $k \\in K_2$.\n\tWhen $\\{ M_k \\}_{k \\in K_2}$ is bounded, then we can find $K_3 \\subseteq K_2$ and $\\gamma_* \\in \\R^p_+$ such that $[\\gamma_*]_i = 0$ for $i \\notin J_*$ and $\\lim_{ K_3 \\ni k \\to \\infty} [\\hat \\gamma_k]_i = [\\gamma_*]_i$ for $i \\in J_*$.\n\tUsing the fact that $\\norm{\\grad L_k(x_k)} \\to 0$, \\eqref{eqn:grad_Lk} implies \\eqref{eqn:reformulated-kkt-x}.\n\tSince $J_* \\subseteq \\cA_*$ and $[\\gamma_*]_i = 0$ for $i \\notin J_*$, then \\eqref{eqn:reformulated-kkt-z} follows.\n\n\tWhen $\\{ M_k \\}_{k\\in K_2}$ is unbounded, we can find $K_4 \\subseteq K_2$ and $\\hat \\gamma \\in \\R_+^q$ such that $\\lim_{ K_4 \\ni k \\to \\infty} [\\hat \\gamma_k]_i / M_k = [\\hat \\gamma]_i$ for $i \\in J_*$ and $[\\hat \\gamma]_i = 0$ otherwise. By the definition of $M_k$, we have $\\hat\\gamma\\neq 0$ and $\\|\\hat\\gamma\\|_\\infty = 1$.\n\tDividing \\eqref{eqn:alm-J_k-equ} by $M_k$, using the boundedness of $\\grad L_k(x_k)$, $\\grad f(x_k)$ and $\\grad\\, [h_1(x_k)]_i$, letting $k \\in K_4 \\to \\infty$ and noticing that $\\lambda_{k + 1} \\to \\lambda_*$, $x_k \\to x_*$, we can get\n\t\\[  \\sum_{j\\in J_*} [\\hat \\gamma]_j \\grad\\, [h_2(x_*)]_j = 0.  \\]\n\tDue to $\\normz{\\hat \\gamma}_\\infty = 1$ and $\\hat \\gamma \\geq 0$, we know $\\left\\{ \\grad\\, [h_2(x_*)]_j : j \\in J_* \\right\\}$ are linearly dependent with non-negative coefficients. However, they are linearly independent near $x_*$, which contradicts to the CPLD assumption. Thus, $M_k$ is always bounded.\n\tMoreover, when the LICQ holds at $x_*$ but $\\{ \\gamma_{k + 1} \\}$ is unbounded, we can divide \\eqref{eqn:alm-Astar-equ} by $\\norm{\\gamma_{k + 1}}_2$ and yield a contradiction to the LICQ condition. Therefore, $\\{ \\gamma_{k+1} \\}$ is bounded and contains a convergent subsequence.\n\\hfill $\\Box$\n\\end{proof}","units":[{"start":45933,"end":46592,"start_line":674,"end_line":685,"raw_tex":"\\begin{proposition}[Theorem 4.1.4 in \\cite{hiriart1993convex}]\n\t\\label{prop:moreau-yosida}\n\tLet $\\sigma > 0$ and $f: \\R^n \\to \\R \\cup \\{ +\\infty \\}$ be a closed proper convex function, then the Moreau-Yosida regularization $f^\\sigma$ is continuously differentiable, and its gradient is \n\t\\begin{equation}\n\t\t\\label{eqn:moreau-yosida-gradient}\n\t\t\\nabla f^\\sigma(x) = \\frac{1}{\\sigma}(x - \\prox_{f/\\sigma}(x)),\n\t\\end{equation}\n\twhere $\\prox_{f / \\sigma}$ is the \\emph{proximal map} of the function $f / \\sigma$:\n\t \\begin{equation}\n\t\t\\prox_{f / \\sigma} (x) := \\argmin_{y} \\left \\{ f(y) + \\frac{\\sigma}{2}\\norm{x-y}_2^2 \\right \\}.\n\t\\end{equation}\n\\end{proposition}","sha256_utf8":"c46b9c9f038e94ad196aed423708b759f835ae2676a092020d2998cce0cafd83"},{"start":49514,"end":50030,"start_line":747,"end_line":754,"raw_tex":"\\begin{theorem}\n\t\\label{thm:feasibility}\n\tLet $\\{ (x_k, y_k, z_k) \\}$ be the sequence generated by Algorithm~\\ref{alg:alm-outer}.\n\tThen, we have\n\t\\[ \\lim_{k\\to \\infty} \\left( \\norm{h_1(x_k) - y_k}_2 + \\norm{h_2(x_k) - z_k}_2 \\right) = 0. \\]\n\tConsequently, if $x_* \\in \\cM$ is an accumulation point of $\\{ x_k \\}$, then $(x_*, h_1(x_*), h_2(x_*))$ is a feasible accumulation point of $\\left\\{ (x_k, y_k, z_k) \\right\\}$. %\n\tMoreover, $\\{ x_k \\}$ always contains an accumulation point if $\\cM$ is compact.\n\\end{theorem}","sha256_utf8":"848b332c3c18158456709a5436a3ce028d60984f58587b0c814734aeafd9b234"},{"start":50031,"end":53087,"start_line":755,"end_line":796,"raw_tex":"\\begin{proof}\n    First, consider the case where $\\{ \\sigma_k \\}$ is bounded. There exists $k_1 \\in \\N$ such that $\\delta_{k+1} \\leq \\tau \\delta_k$ for any $k \\geq k_1$.\n    Then, $\\lim_{k \\to \\infty} \\delta_k = 0$ since $\\tau < 1$. \n    By the definition of $\\delta_k$, we know $\\lim_{k \\to \\infty} \\norm{h_1(x_k) - y_k}_2 = 0$ and $\\lim_{k\\to \\infty} \\norm{h_2(x_k) - z_k}_2 = 0$. \n\n\tIn the case where $\\{ \\sigma_k \\}$ is unbounded. By the update rule, $\\sigma_k$ is updated for infinitely many times. Then, we can find $k_1 < k_2 < \\cdots$ such that\n\t\\[ \\begin{aligned}\n\t\t\t\\sigma_k = \\sigma_{k_i} &= \\max\\big\\{ \\rho \\sigma_{k_{i-1}}, \\norm{\\lambda_{k_i}}_2^{1 + \\alpha}, \\norm{\\gamma_{k_i}}_2^{1 + \\alpha} \\big\\},\\ \\forall \\, k_i \\leq k < k_{i+1}.\n\t\\end{aligned} \\]\n\n\n\tFrom \\eqref{eqn:subproblem-stopping-rule} and the definition of $L_k(x)$, we know that $L_{\\sigma_{k}}(x_{k}, y_{k}, z_{k}, \\lambda_{k}, \\gamma_{k}) \\leq \\Phi$, where $L_{\\sigma_k}$ is defined in \\eqref{eqn:L_sigma}. Therefore, we have\n\t\\begin{equation}\n\t\t\\norm{h_1(x_{k}) - y_{k} + \\frac{\\lambda_{k}}{\\sigma_{k}}}^2_2 \n\t\t+ \\norm{h_2(x_{k}) - z_{k} + \\frac{\\gamma_{k}}{\\sigma_{k}}}_2^2 \n\t\t\\leq 2\\frac{ \\Phi - f(x_{k}) - \\psi(y_{k}) }{\\sigma_{k}} \n\t\t+ \\frac{\\norm{\\lambda_{k}}_2^2 + \\norm{\\gamma_{k}}_2^2}{\\sigma_{k}^2}.\n\t\t\\label{eqn:feasibility-bounded}\n\t\\end{equation}\n\n\tSince $f + \\psi$ is bounded below and $\\sigma_{k} \\to \\infty$, then the first term of \\eqref{eqn:feasibility-bounded} converges to $0$. Next, we show that the second term also converges to $0$.\n\tNotice that $\\norm{\\lambda_{k_i}}_2^{1+\\alpha} \\leq \\sigma_{k_i}$ and $\\sigma_{k_i} \\to \\infty$, \n\twe have $\\norm{\\lambda_{k_i}}_2 / \\sigma_{k_i} \\leq \\sigma_{k_i}^{-\\frac{\\alpha}{1 + \\alpha}} \\to 0$ as $i \\to \\infty$. A similar result also holds for $\\norm{\\gamma_{k_i}}_2 / \\sigma_{k_i}$. \n\tThen by \\eqref{eqn:feasibility-bounded} and the definition of $\\delta_{k_i}$, we have $\\lim_{i \\to \\infty} \\delta_{k_i} = 0$.\n\tBy the update rule of $\\lambda_k$, for $k_i < k < k_{i+1}$, it holds that \n\t\\[\\begin{aligned}\n\t\t\t\\frac{\\norm{\\lambda_{k}}_2}{\\sigma_{k}} \n\t\t\t&\\leq \\frac{\\norm{\\lambda_{k - 1}}_2}{\\sigma_{k_i}} + \\norm{h_1(x_{k - 1}) - y_{k - 1}}_2 \n\t\t\t\\leq \\frac{\\norm{\\lambda_{k - 1}}_2}{\\sigma_{k_i}} + \\delta_{k-1}.\n\t\\end{aligned} \\]\n\n\tAlso note that $\\delta_k \\leq \\tau\\delta_{k-1}$ for all $k_i \\leq k < k_{i+1} - 1$. Then, for $k_i < k < k_{i+1}$ the following inequality holds by induction\n\t\\[\\begin{aligned}\n\t\t\t\\frac{\\norm{\\lambda_{k}}_2}{\\sigma_{k}} \n\t\t\t&\\leq \\frac{\\norm{\\lambda_{k_i}}_2}{\\sigma_{k_i}} + \\sum_{j=k_i}^{k-1} \\delta_j\n\t\t\t\\leq \\frac{\\norm{\\lambda_{k_i}}_2}{\\sigma_{k_i}} + \\delta_{k_i} \\sum_{j=0}^{k - k_i -1} \\tau^{j}\n\t\t\t\\leq \\frac{\\norm{\\lambda_{k_i}}_2}{\\sigma_{k_i}} + \\frac{\\delta_{k_i}}{1 - \\tau}.\n\t\\end{aligned} \\]\n\n\tThus, we have $\\lim_{k \\to \\infty} \\norm{\\lambda_k}_2 / \\sigma_k = 0$.\n\tSimilarly, $\\lim_{k \\to \\infty} \\norm{\\gamma_k}_2 / \\sigma_k = 0$. Therefore, from \\eqref{eqn:feasibility-bounded}, we conclude that $\\lim_{k \\to \\infty} \\delta_k = 0$. \\hfill $\\Box$\n\\end{proof}","sha256_utf8":"1ab9a93dd014e5926d818c90a7aa1a0960e30f7ab27ba4d50a3a8512dfabb610"},{"start":60131,"end":66182,"start_line":897,"end_line":975,"raw_tex":"\\begin{proof}\n    First, from Theorem~\\ref{thm:feasibility}, we obtain the feasibility condition \\eqref{eqn:reformulated-kkt-feas}.\n    From \\eqref{eqn:subproblem-y}, \\eqref{eqn:subproblem-z} and the property of Moreau-Yosida regularizations in Proposition~\\ref{prop:moreau-yosida}, we know that\n    \\[ \\begin{aligned} \n        y_k %\n        = h_1(x_k) +  \\frac{\\lambda_k}{\\sigma_k} - \\frac{1}{\\sigma_k} \\nabla \\psi^{\\sigma_k}\\left (h_1(x_k) +  \\frac{\\lambda_k}{\\sigma_k}\\right), \\text{ and }\n        z_k %\n        = h_2(x_k) +  \\frac{\\gamma_k}{\\sigma_k} - \\frac{1}{\\sigma_k} \\nabla \\delta_{\\R_-^q}^{\\sigma_k}\\left (h_2(x_k) +  \\frac{\\gamma_k }{ \\sigma_k}\\right). \n\t\\end{aligned} \\]\n    From the definition of $\\lambda_{k+1}$ and $\\gamma_{k+1}$, combining the above two equations, we have\n\t\\begin{equation}\n\t\t\\label{eqn:grad_Lk}\n\t\t\\begin{aligned}\n\t\t\t& \\grad L_k(x_k)\n\t\t\t= \\grad f(x_k) + \\sum_{i=1}^m [\\lambda_{k+1}]_i \\grad\\,[h_1(x_k)]_i \n\t\t\t+ \\sum_{i=1}^q [\\gamma_{k+1}]_i \\grad\\, [h_2(x_k)]_i,  \n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere the chain rule for gradients on manifolds is used\\footnote{\n\tLet $g: \\cM \\to \\R^m$ and $f: \\R^m \\to \\R$ be two smooth maps,\n\twe can use Definition~\\ref{def:riemannian-gradient} to compute the gradient of $f \\circ g$.\n\tFor $p \\in \\cM$, $\\xi_p \\in T_p\\cM$, choosing any curve $\\gamma: (-1, 1) \\to \\cM$ such that $\\gamma(0) = p$ and $\\dot \\gamma(0) = \\xi_p$, we have\n\t$\\inner{\\xi_p}{\\grad\\,(f\\circ g)(p)} = \\xi_p (f \\circ g) \n\t= \\left.\\frac{\\dd (f \\circ g \\circ \\gamma)(t)}{\\dd t}\\right|_{t=0} \n\t= \\sum_{i=1}^m\\alpha_i\\left.\\frac{\\dd (g_i \\circ \\gamma)(t)}{\\dd t}\\right|_{t=0} \n\t= \\sum_{i=1}^m \\alpha_i \\cdot (\\xi_p g_i) \n\t= \\sum_{i=1}^m \\alpha_i \\inner{\\xi_p}{\\grad\\,[g(p)]_i}$, where $\\alpha := \\nabla f(q)$, $q := g(\\gamma(t))$ and $g_i$ is the $i$-th component of $g$.\n\tThus, $\\grad\\, (f \\circ g)(p) = \\sum_{i=1}^m \\alpha_i \\grad\\, [g(p)]_i$.\n\t}.\n\n    Since $y_k = \\prox_{\\psi/\\sigma_k} (h_1(x_k) +  \\lambda_k/\\sigma_k)$, then\n\t\\begin{equation}\n\t\t\\label{eqn:grad_yk}\n\t\t0 \\in \\partial \\psi(y_k) - (\\sigma_k(h_1(x_k) - y_k) +  \\lambda_k) = \\partial \\psi(y_k) - \\lambda_{k+1}. \n\t\\end{equation}\n\tNote that $\\lim_{K \\ni k \\to \\infty} y_k = y_*$, then $\\left\\{ y_k \\right\\}_{k\\in K}$ is bounded.\n\tBy the locally boundedness of the subdifferential~\\cite[Corollary 24.5.1]{rockafellar1970convex}, $\\bigcup_{k \\in K} \\partial \\psi(y_k)$ is also bounded.\n\tTherefore, $\\{ \\lambda_{k+1} \\}_{k \\in K}$ is bounded, so we can choose $K_1 \\subseteq K$, $\\lambda_* \\in \\R^m$ such that $\\lim_{ K_1 \\ni k \\to \\infty} \\lambda_{k+1} = \\lambda_*$, which implies $\\lambda_*\\in\\partial\\psi(y_*)$.\n\n\tOn the other hand, since \n\t\\[ z_k = \\Pi_{\\R_-^q} \\left ( h_2(x_k) +  \\frac{\\gamma_k}{\\sigma_k} \\right ) = \\argmin_{z \\leq 0} \\norm{h_2(x_k) +  \\frac{\\gamma_k}{\\sigma_k} - z}^2_2, \\]\nthen from the optimality condition of the above problem, it holds that\n\t\\begin{equation}\n\t\t\\label{eqn:grad_zk}\n\t\t0 = [\\sigma_k( h_2(x_k) - z_k) +  \\gamma_k]^\\trans z_k = \\gamma_{k+1}^\\trans z_k \\quad \\mathrm{and} \\quad \\gamma_{k+1} \\geq 0.\n\t\\end{equation}\n\n\n\n\tLet $\\cA_{k} = \\{ i \\in [q] : [z_k]_i = 0 \\}$, from \\eqref{eqn:grad_zk}, we know $[\\gamma_{k+1}]_i = 0$ for $i \\notin \\cA_k$. \n\tDefine $\\cA_* := \\{ i \\in [q] : [z_*]_i = 0 \\}$.\n\tSince $z_k \\to z_*$, we also find that for sufficiently large $k$ and $i \\notin \\cA_*$, $[\\gamma_{k+1}]_i = 0$.\n\tThen, when $k$ is large enough \\eqref{eqn:grad_Lk} can be written as \n\t\\begin{equation}\n\t\t\\label{eqn:alm-Astar-equ}\n\t\t\t\\grad L_k(x_k)\n\t\t\t= \\grad f(x_k) + \\sum_{i=1}^m [\\lambda_{k+1}]_i \\grad\\,[h_1(x_k)]_i \n\t\t\t+ \\sum_{i \\in \\cA_*} [\\gamma_{k+1}]_i \\grad\\, [h_2(x_k)]_i.  \n    \\end{equation}\n\tUsing the Carath\\'eodory's theorem of cones~\\cite{bertsekas1997nonlinear}, there exist $J_k \\subseteq \\cA_*$ and $[\\hat \\gamma_k]_j \\geq 0$, where $j \\in J_k$, such that $\\left\\{ \\grad\\, [h_2(x_k)]_j : j \\in J_k \\right\\}$ are linearly independent and\n\t\\begin{equation}\n\t\t\\label{eqn:alm-J_k-equ}\n\t\t\t\\grad L_k(x_k)\n\t\t\t= \\grad f(x_k) + \\sum_{i=1}^m [\\lambda_{k+1}]_i \\grad\\,[h_1(x_k)]_i \n\t\t\t+ \\sum_{j \\in J_k} [\\hat\\gamma_{k}]_j \\grad\\, [h_2(x_k)]_j,\n\t\\end{equation}\n\tSince $J_k \\subseteq [q]$ is a finite set, we can choose $J_*$ and $K_2 \\subseteq K_1$ such that $J_k = J_*$ for all $k \\in K_2$ and $|K_2| = \\infty$.\n\tDefine $M_k = \\max\\left\\{ [\\hat\\gamma_{k}]_i : i \\in J_* \\right\\}$ for $k \\in K_2$.\n\tWhen $\\{ M_k \\}_{k \\in K_2}$ is bounded, then we can find $K_3 \\subseteq K_2$ and $\\gamma_* \\in \\R^p_+$ such that $[\\gamma_*]_i = 0$ for $i \\notin J_*$ and $\\lim_{ K_3 \\ni k \\to \\infty} [\\hat \\gamma_k]_i = [\\gamma_*]_i$ for $i \\in J_*$.\n\tUsing the fact that $\\norm{\\grad L_k(x_k)} \\to 0$, \\eqref{eqn:grad_Lk} implies \\eqref{eqn:reformulated-kkt-x}.\n\tSince $J_* \\subseteq \\cA_*$ and $[\\gamma_*]_i = 0$ for $i \\notin J_*$, then \\eqref{eqn:reformulated-kkt-z} follows.\n\n\tWhen $\\{ M_k \\}_{k\\in K_2}$ is unbounded, we can find $K_4 \\subseteq K_2$ and $\\hat \\gamma \\in \\R_+^q$ such that $\\lim_{ K_4 \\ni k \\to \\infty} [\\hat \\gamma_k]_i / M_k = [\\hat \\gamma]_i$ for $i \\in J_*$ and $[\\hat \\gamma]_i = 0$ otherwise. By the definition of $M_k$, we have $\\hat\\gamma\\neq 0$ and $\\|\\hat\\gamma\\|_\\infty = 1$.\n\tDividing \\eqref{eqn:alm-J_k-equ} by $M_k$, using the boundedness of $\\grad L_k(x_k)$, $\\grad f(x_k)$ and $\\grad\\, [h_1(x_k)]_i$, letting $k \\in K_4 \\to \\infty$ and noticing that $\\lambda_{k + 1} \\to \\lambda_*$, $x_k \\to x_*$, we can get\n\t\\[  \\sum_{j\\in J_*} [\\hat \\gamma]_j \\grad\\, [h_2(x_*)]_j = 0.  \\]\n\tDue to $\\normz{\\hat \\gamma}_\\infty = 1$ and $\\hat \\gamma \\geq 0$, we know $\\left\\{ \\grad\\, [h_2(x_*)]_j : j \\in J_* \\right\\}$ are linearly dependent with non-negative coefficients. However, they are linearly independent near $x_*$, which contradicts to the CPLD assumption. Thus, $M_k$ is always bounded.\n\tMoreover, when the LICQ holds at $x_*$ but $\\{ \\gamma_{k + 1} \\}$ is unbounded, we can divide \\eqref{eqn:alm-Astar-equ} by $\\norm{\\gamma_{k + 1}}_2$ and yield a contradiction to the LICQ condition. Therefore, $\\{ \\gamma_{k+1} \\}$ is bounded and contains a convergent subsequence.\n\\hfill $\\Box$\n\\end{proof}","sha256_utf8":"202a58af2145f3f067b25f9552a62633713d707fe2ab91eb82a9973faffc56a1"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"增广拉格朗日法的KKT收敛","source_units":[{"start":59440,"end":60130,"start_line":891,"end_line":896,"raw_tex":"\\begin{theorem}\n    \\label{thm:kkt}\n    Suppose there exist $K \\subseteq \\N$ and $(x_*, y_*, z_*) \\in \\cM \\times \\R^m \\times \\R^q_-$ such that\n    \\[ \\lim_{K \\ni k \\to \\infty} \\left ( d(x_k, x_*) + \\norm{y_k - y_*}_2 + \\norm{z_k - z_*}_2 \\right ) = 0. \\]\n    If the CPLD condition holds at $x_*$, then there exist $K_0 \\subseteq K$ and $\\lambda_* \\in \\R^m, \\gamma_* \\in \\R^q_+$ such that $\\lim_{K_0 \\ni k \\to \\infty} \\lambda_{k+1} = \\lambda_*$, and the KKT conditions \\eqref{eqn:reformulated-kkt} hold at $(x_*, y_*, z_*, \\lambda_*, \\gamma_*)$. Moreover, when the LICQ holds at $x_*$, we can choose $\\gamma_*$ such that $\\lim_{K_0 \\ni k \\to \\infty} \\gamma_{k + 1} = \\gamma_*$.\n\\end{theorem}","sha256_utf8":"504bd65f889ab9de001d31fdf204e17afd8d21ec541f1e5a7e50db6f38cb765c"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"KKT收敛完整证明","source_units":[{"start":60131,"end":66182,"start_line":897,"end_line":975,"raw_tex":"\\begin{proof}\n    First, from Theorem~\\ref{thm:feasibility}, we obtain the feasibility condition \\eqref{eqn:reformulated-kkt-feas}.\n    From \\eqref{eqn:subproblem-y}, \\eqref{eqn:subproblem-z} and the property of Moreau-Yosida regularizations in Proposition~\\ref{prop:moreau-yosida}, we know that\n    \\[ \\begin{aligned} \n        y_k %\n        = h_1(x_k) +  \\frac{\\lambda_k}{\\sigma_k} - \\frac{1}{\\sigma_k} \\nabla \\psi^{\\sigma_k}\\left (h_1(x_k) +  \\frac{\\lambda_k}{\\sigma_k}\\right), \\text{ and }\n        z_k %\n        = h_2(x_k) +  \\frac{\\gamma_k}{\\sigma_k} - \\frac{1}{\\sigma_k} \\nabla \\delta_{\\R_-^q}^{\\sigma_k}\\left (h_2(x_k) +  \\frac{\\gamma_k }{ \\sigma_k}\\right). \n\t\\end{aligned} \\]\n    From the definition of $\\lambda_{k+1}$ and $\\gamma_{k+1}$, combining the above two equations, we have\n\t\\begin{equation}\n\t\t\\label{eqn:grad_Lk}\n\t\t\\begin{aligned}\n\t\t\t& \\grad L_k(x_k)\n\t\t\t= \\grad f(x_k) + \\sum_{i=1}^m [\\lambda_{k+1}]_i \\grad\\,[h_1(x_k)]_i \n\t\t\t+ \\sum_{i=1}^q [\\gamma_{k+1}]_i \\grad\\, [h_2(x_k)]_i,  \n\t\t\\end{aligned}\n\t\\end{equation}\n\twhere the chain rule for gradients on manifolds is used\\footnote{\n\tLet $g: \\cM \\to \\R^m$ and $f: \\R^m \\to \\R$ be two smooth maps,\n\twe can use Definition~\\ref{def:riemannian-gradient} to compute the gradient of $f \\circ g$.\n\tFor $p \\in \\cM$, $\\xi_p \\in T_p\\cM$, choosing any curve $\\gamma: (-1, 1) \\to \\cM$ such that $\\gamma(0) = p$ and $\\dot \\gamma(0) = \\xi_p$, we have\n\t$\\inner{\\xi_p}{\\grad\\,(f\\circ g)(p)} = \\xi_p (f \\circ g) \n\t= \\left.\\frac{\\dd (f \\circ g \\circ \\gamma)(t)}{\\dd t}\\right|_{t=0} \n\t= \\sum_{i=1}^m\\alpha_i\\left.\\frac{\\dd (g_i \\circ \\gamma)(t)}{\\dd t}\\right|_{t=0} \n\t= \\sum_{i=1}^m \\alpha_i \\cdot (\\xi_p g_i) \n\t= \\sum_{i=1}^m \\alpha_i \\inner{\\xi_p}{\\grad\\,[g(p)]_i}$, where $\\alpha := \\nabla f(q)$, $q := g(\\gamma(t))$ and $g_i$ is the $i$-th component of $g$.\n\tThus, $\\grad\\, (f \\circ g)(p) = \\sum_{i=1}^m \\alpha_i \\grad\\, [g(p)]_i$.\n\t}.\n\n    Since $y_k = \\prox_{\\psi/\\sigma_k} (h_1(x_k) +  \\lambda_k/\\sigma_k)$, then\n\t\\begin{equation}\n\t\t\\label{eqn:grad_yk}\n\t\t0 \\in \\partial \\psi(y_k) - (\\sigma_k(h_1(x_k) - y_k) +  \\lambda_k) = \\partial \\psi(y_k) - \\lambda_{k+1}. \n\t\\end{equation}\n\tNote that $\\lim_{K \\ni k \\to \\infty} y_k = y_*$, then $\\left\\{ y_k \\right\\}_{k\\in K}$ is bounded.\n\tBy the locally boundedness of the subdifferential~\\cite[Corollary 24.5.1]{rockafellar1970convex}, $\\bigcup_{k \\in K} \\partial \\psi(y_k)$ is also bounded.\n\tTherefore, $\\{ \\lambda_{k+1} \\}_{k \\in K}$ is bounded, so we can choose $K_1 \\subseteq K$, $\\lambda_* \\in \\R^m$ such that $\\lim_{ K_1 \\ni k \\to \\infty} \\lambda_{k+1} = \\lambda_*$, which implies $\\lambda_*\\in\\partial\\psi(y_*)$.\n\n\tOn the other hand, since \n\t\\[ z_k = \\Pi_{\\R_-^q} \\left ( h_2(x_k) +  \\frac{\\gamma_k}{\\sigma_k} \\right ) = \\argmin_{z \\leq 0} \\norm{h_2(x_k) +  \\frac{\\gamma_k}{\\sigma_k} - z}^2_2, \\]\nthen from the optimality condition of the above problem, it holds that\n\t\\begin{equation}\n\t\t\\label{eqn:grad_zk}\n\t\t0 = [\\sigma_k( h_2(x_k) - z_k) +  \\gamma_k]^\\trans z_k = \\gamma_{k+1}^\\trans z_k \\quad \\mathrm{and} \\quad \\gamma_{k+1} \\geq 0.\n\t\\end{equation}\n\n\n\n\tLet $\\cA_{k} = \\{ i \\in [q] : [z_k]_i = 0 \\}$, from \\eqref{eqn:grad_zk}, we know $[\\gamma_{k+1}]_i = 0$ for $i \\notin \\cA_k$. \n\tDefine $\\cA_* := \\{ i \\in [q] : [z_*]_i = 0 \\}$.\n\tSince $z_k \\to z_*$, we also find that for sufficiently large $k$ and $i \\notin \\cA_*$, $[\\gamma_{k+1}]_i = 0$.\n\tThen, when $k$ is large enough \\eqref{eqn:grad_Lk} can be written as \n\t\\begin{equation}\n\t\t\\label{eqn:alm-Astar-equ}\n\t\t\t\\grad L_k(x_k)\n\t\t\t= \\grad f(x_k) + \\sum_{i=1}^m [\\lambda_{k+1}]_i \\grad\\,[h_1(x_k)]_i \n\t\t\t+ \\sum_{i \\in \\cA_*} [\\gamma_{k+1}]_i \\grad\\, [h_2(x_k)]_i.  \n    \\end{equation}\n\tUsing the Carath\\'eodory's theorem of cones~\\cite{bertsekas1997nonlinear}, there exist $J_k \\subseteq \\cA_*$ and $[\\hat \\gamma_k]_j \\geq 0$, where $j \\in J_k$, such that $\\left\\{ \\grad\\, [h_2(x_k)]_j : j \\in J_k \\right\\}$ are linearly independent and\n\t\\begin{equation}\n\t\t\\label{eqn:alm-J_k-equ}\n\t\t\t\\grad L_k(x_k)\n\t\t\t= \\grad f(x_k) + \\sum_{i=1}^m [\\lambda_{k+1}]_i \\grad\\,[h_1(x_k)]_i \n\t\t\t+ \\sum_{j \\in J_k} [\\hat\\gamma_{k}]_j \\grad\\, [h_2(x_k)]_j,\n\t\\end{equation}\n\tSince $J_k \\subseteq [q]$ is a finite set, we can choose $J_*$ and $K_2 \\subseteq K_1$ such that $J_k = J_*$ for all $k \\in K_2$ and $|K_2| = \\infty$.\n\tDefine $M_k = \\max\\left\\{ [\\hat\\gamma_{k}]_i : i \\in J_* \\right\\}$ for $k \\in K_2$.\n\tWhen $\\{ M_k \\}_{k \\in K_2}$ is bounded, then we can find $K_3 \\subseteq K_2$ and $\\gamma_* \\in \\R^p_+$ such that $[\\gamma_*]_i = 0$ for $i \\notin J_*$ and $\\lim_{ K_3 \\ni k \\to \\infty} [\\hat \\gamma_k]_i = [\\gamma_*]_i$ for $i \\in J_*$.\n\tUsing the fact that $\\norm{\\grad L_k(x_k)} \\to 0$, \\eqref{eqn:grad_Lk} implies \\eqref{eqn:reformulated-kkt-x}.\n\tSince $J_* \\subseteq \\cA_*$ and $[\\gamma_*]_i = 0$ for $i \\notin J_*$, then \\eqref{eqn:reformulated-kkt-z} follows.\n\n\tWhen $\\{ M_k \\}_{k\\in K_2}$ is unbounded, we can find $K_4 \\subseteq K_2$ and $\\hat \\gamma \\in \\R_+^q$ such that $\\lim_{ K_4 \\ni k \\to \\infty} [\\hat \\gamma_k]_i / M_k = [\\hat \\gamma]_i$ for $i \\in J_*$ and $[\\hat \\gamma]_i = 0$ otherwise. By the definition of $M_k$, we have $\\hat\\gamma\\neq 0$ and $\\|\\hat\\gamma\\|_\\infty = 1$.\n\tDividing \\eqref{eqn:alm-J_k-equ} by $M_k$, using the boundedness of $\\grad L_k(x_k)$, $\\grad f(x_k)$ and $\\grad\\, [h_1(x_k)]_i$, letting $k \\in K_4 \\to \\infty$ and noticing that $\\lambda_{k + 1} \\to \\lambda_*$, $x_k \\to x_*$, we can get\n\t\\[  \\sum_{j\\in J_*} [\\hat \\gamma]_j \\grad\\, [h_2(x_*)]_j = 0.  \\]\n\tDue to $\\normz{\\hat \\gamma}_\\infty = 1$ and $\\hat \\gamma \\geq 0$, we know $\\left\\{ \\grad\\, [h_2(x_*)]_j : j \\in J_* \\right\\}$ are linearly dependent with non-negative coefficients. However, they are linearly independent near $x_*$, which contradicts to the CPLD assumption. Thus, $M_k$ is always bounded.\n\tMoreover, when the LICQ holds at $x_*$ but $\\{ \\gamma_{k + 1} \\}$ is unbounded, we can divide \\eqref{eqn:alm-Astar-equ} by $\\norm{\\gamma_{k + 1}}_2$ and yield a contradiction to the LICQ condition. Therefore, $\\{ \\gamma_{k+1} \\}$ is bounded and contains a convergent subsequence.\n\\hfill $\\Box$\n\\end{proof}","sha256_utf8":"202a58af2145f3f067b25f9552a62633713d707fe2ab91eb82a9973faffc56a1"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"原问题与分裂重构","source_units":[{"start":43584,"end":43769,"start_line":619,"end_line":625,"raw_tex":"\\begin{equation}\n\t\\label{eqn:original-problem}\n\t\\begin{aligned}\n\t\t\\min_{x}\\  &\\{f(x) + \\psi(h_1(x)) \\}, \\quad \n\t\t\\mathrm{s.t.}\\  x \\in \\cM,\\ h_2(x) \\leq 0.\n\t\\end{aligned}\n\\end{equation}","sha256_utf8":"d88a12c95918665659ea6d34aa0eda2d334f7a33285197d107769e28af02eea1"},{"start":44252,"end":44470,"start_line":637,"end_line":645,"raw_tex":"\\begin{equation}\n\t\\label{eqn:reformulated-problem}\n\t\\begin{aligned}\n\t\t\\min_{x, y, z}\\  &\\{f(x) + \\psi(y) \\}, \\quad\n\t\t\\mathrm{s.t.}\\  x \\in \\cM, \\\n\t\t y = h_1(x), \\\n\t\t z = h_2(x),\\ z \\leq 0.\n\t\\end{aligned}\n\\end{equation}","sha256_utf8":"5c392586beb3bff6341a731a935518bbac18d590b7bd1323659eab07d7adff24"}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","title":"完备性与函数基本假设","source_units":[{"start":43836,"end":43920,"start_line":628,"end_line":630,"raw_tex":"\\begin{assumption}\n\t$\\cM$ is a complete smooth Riemannian manifold.\n\\end{assumption}","sha256_utf8":"3e175af8e06c0209d862425465114a832647f5fa907a1e811ebd8168856c3c67"},{"start":43921,"end":44166,"start_line":631,"end_line":634,"raw_tex":"\\begin{assumption}\n\t$f : \\cM \\to \\R$, $h_1: \\cM \\to \\R^m$, $h_2: \\cM \\to \\R^{q}$ are continuously differentiable, and~$\\psi: \\R^m \\to \\R$ is a convex function.\n\t$f(x) + \\psi(y)$ is bounded below for $(x, y) \\in \\cM \\times \\R^m$.\n\\end{assumption}","sha256_utf8":"baaef443a04957783ade16d362c9283c9772451d782bcb8d828cba138061c164"}]},{"id":"S005","semantic_type":"notation","kind":"notation","title":"流形与指标基本记号","source_units":[{"start":17527,"end":18202,"start_line":337,"end_line":346,"raw_tex":"\t\t\t$[u]_i$ & The $i$-th component of $u\\in\\R^d$ \\\\ \\hline\n\t\t\t$[M]$ & The set $\\{1,2,\\cdots,M\\}$, where $M$ is a positive integer \\\\ \\hline\n\t\t\t$\\cM$ & A complete $n$-dimensional smooth Riemannian manifold \\\\ \n\t\t\t\\hline\n\t\t\t$T_p\\cM$ & The tangent space at $p\\in\\cM$ \\\\ \\hline\n\t\t\t$T\\cM$ & The tangent bundle of $\\cM$ \\\\ \\hline\n\t\t\t$\\cD(\\cM)$ & The set of smooth functions on $\\cM$ with compact support \\\\ \\hline \n\t\t\t$\\dd\\varphi|_p$ & The differential of the smooth map $\\varphi$ at $p\\in\\cM$  \\\\ \\hline\n\t\t\t$\\grad\\varphi$ & The gradient of the function $\\varphi$ on manifolds\\\\ \\hline\n\t\t\t$\\partial \\varphi$ & The Clarke subgradient of the function $\\varphi$ on manifolds \\\\ \\hline\n","sha256_utf8":"3d76497c8e172feb0fb65fc0827d0bb4fbbc6950cbda165dfa8757b93bdae46d"},{"start":19724,"end":19914,"start_line":365,"end_line":366,"raw_tex":"\t\t\t$\\inner{\\eta}{\\xi}_p$,$\\inner{\\eta}{\\xi}$ & The Riemannian inner product of $\\eta, \\xi \\in T_p\\cM$  \\\\ \\hline\n\t\t\t$\\norm{\\eta}_p$, $\\norm{\\eta}$ & The norm of $\\eta \\in T_p\\cM$  \\\\ \\hline\n","sha256_utf8":"ebe43e8a3fe3c756c60a3fae4f5276bb0b1b60a5c8992c9400e292e8135cc2e1"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"坐标图与切空间","source_units":[{"start":20288,"end":20626,"start_line":376,"end_line":376,"raw_tex":"A Hausdorff topological space $\\cM$ is said to be an \\emph{$n$-dimensional manifold} if it has a countable basis and for each $p \\in \\cM$ there exist a neighborhood $U$ of $p$, an open subset $\\hat U \\subset \\R^n$ and a map $\\varphi: U \\to \\hat U$ such that $\\varphi$ is a homeomorphism. The pair $(U, \\varphi)$ is called a \\emph{chart}.\n","sha256_utf8":"5fc2673a7ce724d22ab45d5c99fe30db6e51e678662523d8a7789ed3da2a33d1"},{"start":21237,"end":21594,"start_line":383,"end_line":387,"raw_tex":"\\begin{definition}  %\n    A \\emph{tangent vector} $\\xi_p:\\cD(\\cM)\\to\\R$ to a manifold $\\cM$ at a point $p$ is a linear operator such that for every $f \\in \\cD(\\cM)$, %\n        $\\xi_p f = \\dot\\gamma(0) f:= \\left.\\frac{\\dd(f(\\gamma(t)))}{\\dd t}\\right|_{t=0}$,\n    where $\\gamma: (-1, 1) \\to \\cM$ is a smooth curve on $\\cM$ with $\\gamma(0)=p$.\n\\end{definition}","sha256_utf8":"b5dc4c9bbf4c1f0902681b4e640b848af0ae27384562078316f3dc82b2dce074"},{"start":21595,"end":22111,"start_line":388,"end_line":391,"raw_tex":"The \\emph{tangent space} $T_p\\cM$ is the space containing all tangent vectors at $p$, which is an $n$-dimensional $\\R$-linear space. \nA \\emph{Riemannian metric} $\\langle\\cdot,\\cdot\\rangle_p$ gives an inner product on $T_p\\cM$, which smoothly depends on~$p$.\\footnote{The subscript $p$ in $\\inner{\\cdot}{\\cdot}_p$ is usually omitted for simplicity.}\nMoreover, a Riemannian metric gives a metric on $\\cM$ and the gradient of functions defined on $\\cM$.\nBelow we assume that $\\cM$ is equipped with a Riemannian metric.\n","sha256_utf8":"5fe2674d77192f9df830479a059e905737f4949af981b046702a56c42edd8a9a"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"距离微分与黎曼梯度","source_units":[{"start":22111,"end":22501,"start_line":392,"end_line":398,"raw_tex":"\\begin{definition} %\n\tGiven $p,q\\in\\cM$, the \\emph{distance between $p$ and $q$} is defined as \n\t\\begin{equation*}\n\td(p,q) = \\inf \\left\\{ \\ell(\\gamma)  :  \\ell(\\gamma) := \\int_0^1 \\sqrt{\\inner{\\dot \\gamma(t)}{\\dot\\gamma(t)}} \\dd t \\right\\},\n\t\\end{equation*}\n\twhere $\\inf$ is taken over all piecewise smooth curves $\\gamma:[0,1]\\to\\cM$  with $\\gamma(0)=p$ and $\\gamma(1)=q$.\n\\end{definition}","sha256_utf8":"93176fccbb242a8f0fef0dc3cfedf6e71ba6fd38991d3dab9d43d9fcb293aa32"},{"start":22502,"end":22818,"start_line":399,"end_line":401,"raw_tex":"\\begin{definition}\n    Let $f: \\cM \\to \\cN$ be a smooth map between smooth manifolds $\\cM, \\cN$, the \\emph{differential of $f$ at $p\\in\\cM$}, denoted by $\\dd f|_p$, is a map from $T_p\\cM$ to $T_{f(p)}\\cN$ such that $(\\dd f|_p\\eta) g:= \\eta(g \\circ f)$ for all $g \\in \\cD(\\cN)$ and $\\eta \\in T_p\\cM$.\n\\end{definition}","sha256_utf8":"f20487e0e8c206e8161d8b778b99d895cb59b7b3910e9f193b3181c2c26c8482"},{"start":22819,"end":23127,"start_line":402,"end_line":407,"raw_tex":"\\begin{definition}%\n\t\\label{def:riemannian-gradient}\n\tLet $f:\\cM\\to\\R$ be a smooth function \n\tand $p\\in\\cM$, the \\emph{gradient} of $f$ at $p$ is defined as the unique tangent vector $\\grad f(p)\\in T_p\\cM$ that satisfies \n\t\t $$\\xi_p f = \\inner{\\xi_p}{\\grad f(p)},\\ \\forall\\xi_p \\in T_p\\cM.$$\n\\end{definition}","sha256_utf8":"d73b10650fb48c4e48f98742adcaa9705700084153314e7d4b2af5dd197cfde8"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"Clarke次梯度","source_units":[{"start":27795,"end":28632,"start_line":460,"end_line":464,"raw_tex":"Suppose $f: \\cM \\to \\R$ is a locally Lipschitz function\\footnote{We say a function $f$ on a manifold is locally Lipschitz if $f \\circ \\varphi^{-1}$ is locally Lipschitz in $U$ for every chart $(U, \\varphi)$.} on a manifold $\\cM$ and $(U, \\varphi)$ is a chart containing $p \\in \\cM$. \nFrom ~\\cite{azagra2005nonsmooth,dirr2007nonsmooth,hosseini2011generalized}, the \\emph{Clarke generalized directional derivative}  of $f$ at $p$, denoted by $f^\\circ(p; v)$, is defined by\n\\[ f^\\circ(p; v) := \\limsup_{y \\to p, t \\downarrow 0} \\frac{\\hat f(\\varphi(y) + t \\dd\\varphi|_pv) - \\hat f(\\varphi(y))}{t}, \\]\nwhere $\\hat f := f \\circ \\varphi^{-1}$ and $\\dd\\varphi|_p$ is the differential of $\\varphi$ at $p$. The \\emph{Clarke subgradient} is \n$$\\partial f(p) := \\{  \\xi \\in T_p\\cM : \\inner{\\xi}{v} \\leq f^\\circ (p; v),\\ \\forall v \\in T_p \\cM \\}.$$\n","sha256_utf8":"f9e0a4052463460b4da0d8441194d6688e7d5437e4df126fed721bc79a1880cf"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"增广拉格朗日函数","source_units":[{"start":44471,"end":44961,"start_line":646,"end_line":656,"raw_tex":"The augmented Lagrangian function $L_\\sigma: \\cM \\times \\R^m \\times \\R_-^q \\times \\R^m \\times \\R^q \\to \\R$ of \\eqref{eqn:reformulated-problem} is given by\n\\begin{equation}\n\t\\begin{aligned} \n\\peq\tL_\\sigma(x, y, z, \\lambda, \\gamma)  \n= f(x) + \\psi(y) \n+ \\frac{\\sigma}{2} \\norm{h_1(x) - y + \\frac{\\lambda}{\\sigma}}^2_2 \n+ \\frac{\\sigma}{2}\\norm{h_2(x) - z + \\frac{\\gamma}{\\sigma}}_2^2 \n- \\frac{\\norm{\\lambda}_2^2 + \\norm{\\gamma}_2^2}{2\\sigma}.\n\t\\end{aligned}\n\t\\label{eqn:L_sigma}\n\\end{equation}","sha256_utf8":"76b20126ece98a36e501c851506d19c82377694ca3b5fc29aec4effbb325e68b"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"Moreau包络与指示函数","source_units":[{"start":45296,"end":45759,"start_line":665,"end_line":671,"raw_tex":"where $\\psi^\\sigma$, $\\delta_{\\R^q_-}^\\sigma$ are the \\emph{Moreau-Yosida regularization} of $\\psi$, $\\delta_{\\R^q_-}$, respectively. \nMore specifically, it holds that\n\\begin{align}\n    \\psi^\\sigma(x) &:= \\min_{y \\in \\R^m}  \\left \\{ \\psi(y)  \n    + \\frac{\\sigma}{2} \\norm{x - y}^2_2 \\right \\}, \\label{prob:y}\\\\\n    \\delta_{\\R_-^q}^\\sigma(x) &:= \\min_{z \\in \\R^q} \\left \\{ \\delta_{\\R_-^q}(z) + \\frac{\\sigma}{2}\\norm{x - z}_2^2 \\right \\}, \\label{prob:z}\n\\end{align}","sha256_utf8":"fff39cf7d3cf2ec0ac826bb0f445b180061b0bc3a9ed275e772a6ec37561fa77"},{"start":45760,"end":45824,"start_line":672,"end_line":672,"raw_tex":"where $\\delta_{\\R_-^q}$ is the indicator function of $\\R_-^q$. \n","sha256_utf8":"83384c2de53fd4c1a061294cff82295410aa6ee140ad6fe83ae60eae479e8238"}]},{"id":"S011","semantic_type":"proposition","kind":"proposition","title":"包络梯度与近端映射","source_units":[{"start":45933,"end":46592,"start_line":674,"end_line":685,"raw_tex":"\\begin{proposition}[Theorem 4.1.4 in \\cite{hiriart1993convex}]\n\t\\label{prop:moreau-yosida}\n\tLet $\\sigma > 0$ and $f: \\R^n \\to \\R \\cup \\{ +\\infty \\}$ be a closed proper convex function, then the Moreau-Yosida regularization $f^\\sigma$ is continuously differentiable, and its gradient is \n\t\\begin{equation}\n\t\t\\label{eqn:moreau-yosida-gradient}\n\t\t\\nabla f^\\sigma(x) = \\frac{1}{\\sigma}(x - \\prox_{f/\\sigma}(x)),\n\t\\end{equation}\n\twhere $\\prox_{f / \\sigma}$ is the \\emph{proximal map} of the function $f / \\sigma$:\n\t \\begin{equation}\n\t\t\\prox_{f / \\sigma} (x) := \\argmin_{y} \\left \\{ f(y) + \\frac{\\sigma}{2}\\norm{x-y}_2^2 \\right \\}.\n\t\\end{equation}\n\\end{proposition}","sha256_utf8":"c46b9c9f038e94ad196aed423708b759f835ae2676a092020d2998cce0cafd83"}]},{"id":"S012","semantic_type":"algorithm","kind":"algorithm","title":"增广拉格朗日外层算法","source_units":[{"start":47100,"end":49341,"start_line":690,"end_line":743,"raw_tex":"\\begin{alg}[An augmented Lagrangian method for solving \\eqref{eqn:reformulated-problem}]\n\t\\label{alg:alm-outer}\n\tChoose initial values $x_0 \\in \\cM$, $\\gamma_0 \\in \\R^{q}_+$, $\\lambda_0 \\in \\R^m$, $\\sigma_0 > 0$, $\\alpha, \\tau \\in (0, 1)$, $\\rho > 1$ \n\tand a sequence $\\{ \\varepsilon_k \\} \\subseteq \\R_+$ converging to $0$.\n\tLet $y_0 = \\prox_{\\psi/\\sigma_0} ( h_1(x_0) + \\lambda_0 / \\sigma_0)$, $z_0 = \\Pi_{\\R_-^q} ( h_2(x_0) + \\gamma_0 / \\sigma_0)$, where $\\Pi_{\\R_-^q}$ is the projection onto $\\R_-^q$. %\n\tChoose a feasible point $x_\\feas$ and a constant $\\Phi$ such that\n\t\\begin{equation}\n\t\t\\label{eqn:initial-maximal}\n\t\t\\Phi \\geq \\max\\{  f(x_\\feas) + \\psi(h_1(x_\\feas)),\\ L_{\\sigma_0}(x_0, y_0, z_0, \\lambda_0, \\gamma_0)  \\}.\n\t\\end{equation}\n\tOur algorithm repeats the following steps for $k = 1, 2, \\dots$\n\n    \\begin{enumerate}[label=(\\roman*)]\n\t\\item\n\tFind $x_k\\in\\cM$ such that\n\t\\begin{equation}\n\t\t\\label{eqn:subproblem-stopping-rule}\n\t\t\\norm{\\grad L_k (x_k)} < \\varepsilon_k, \\quad L_k(x_k) \\leq \\Phi,\n\t\\end{equation}\n\twhere \n\t\\begin{equation}\n\tL_k(x) := f(x) \n\t+ \\psi^{\\sigma_k} \\left ( h_1(x) + \\frac{\\lambda_k}{\\sigma_k}  \\right )\n\t+ \\delta_{\\R_-^q}^{\\sigma_k} \\left ( h_2(x) + \\frac{\\gamma_k}{\\sigma_k}  \\right ).\n\t\\label{eqn:Lk}\n\t\\end{equation}\n\t\\item Update $y$ and $z$ using\n    \\begin{align}\n        \\label{eqn:subproblem-y}\n        y_k &= \\prox_{\\psi/\\sigma_k}\\left ( h_1(x_k) + \\frac{\\lambda_k}{\\sigma_k}  \\right ),\\\\\n        \\label{eqn:subproblem-z}\n        z_k &= \\Pi_{\\R_-^q} \\left ( h_2(x_k) +  \\frac{\\gamma_k}{ \\sigma_k} \\right).\n    \\end{align}\n        \n\t\\item %\n\tUpdate the multipliers:\n    \\[ \\begin{aligned}\n        \\lambda_{k+1} &= \\lambda_k + \\sigma_k (h_1(x_k) - y_k), \\quad \n        \\gamma_{k+1} = \\gamma_k +\\sigma_k (h_2(x_k) - z_k).\n    \\end{aligned} \\]\n\n\t\\item\n\tLet \n        $     \\delta_k = \\max\\left\\{ \\norm{h_1(x_k) - y_k}_2,\\ \\norm{h_2(x_k)-z_k}_2\\right\\}.\n        $ %\n\n\t\t If $\\delta_k \\leq \\tau \\delta_{k-1}$, then $\\sigma_{k+1} = \\sigma_k$.\n\t\t Otherwise, set\n         \\begin{equation}\n            \\label{eqn:alm-update-penalty}\n\t\t\t\\sigma_{k+1} = \\max \\left \\{ \\rho\\sigma_k,\\ \\norm{\\lambda_{k+1}}_2^{1+\\alpha},\\ \\norm{\\gamma_{k+1}}_2^{1+\\alpha} \\right \\}. \n\t\t \\end{equation}\n\t\\end{enumerate}\n\\end{alg}","sha256_utf8":"d7c250017c183a613acdb4f41459140390f132fdb4f914ed241a65d5790b1e91"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"活跃集与约束资格","source_units":[{"start":53265,"end":53424,"start_line":799,"end_line":800,"raw_tex":"Consider the problem \\eqref{eqn:original-problem}, we define the \\emph{active set} of a feasible point $x$ to be\n$\\cA(x) := \\{ i \\in [q] : [h_2(x)]_i = 0 \\}$.\n","sha256_utf8":"934fe422ece68974578c8d0ffa1b687c797204768494c522cb5e87c1cf6d787d"},{"start":53508,"end":53801,"start_line":802,"end_line":806,"raw_tex":"\\begin{definition}[LICQ]\n\t\\label{def:LICQ}\n\tWe say that a feasible point $x \\in \\cM$ of \\eqref{eqn:original-problem} satisfies the \\emph{linear independence constraint qualification} (LICQ) if\n    $\\{ \\grad\\, [h_2(x)]_i : i \\in \\cA(x) \\}$ are linearly independent in $T_x\\cM$.\n\\end{definition}","sha256_utf8":"6a7c1889cc817d416a7f930e5c98d582b011b8ff199abf0965583891604f2343"},{"start":53989,"end":54464,"start_line":810,"end_line":816,"raw_tex":"\\begin{definition}[CPLD]\n\t\\label{def:CPLD}\n\tLet $x \\in \\cM$ be a feasible point of \\eqref{eqn:original-problem} and define\n    $S(x) := \\{ \\grad\\, [h_2(x)]_i : i \\in \\cA(x) \\}$.\n\tWe say that $x$ satisfies the \\emph{constant positive linear dependence constraint qualification} (CPLD) \n\tif for each subset $S_0(x) \\subseteq S(x)$ whose elements are linearly dependent with non-negative coefficients, $S_0$ remains linearly dependent in a neighborhood of $x$. \n\\end{definition}","sha256_utf8":"0aa9efe63cbe41ca14aa2a4c2efe8e9b06ed6b0152b5989554c38a677b66df72"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"重构问题的KKT系统","source_units":[{"start":55400,"end":55595,"start_line":834,"end_line":835,"raw_tex":"The Lagrangian of the equivalent problem \\eqref{eqn:reformulated-problem} is\n\\[ \\tilde \\cL(x, y, z, \\lambda, \\gamma) = f(x) + \\psi(y) + \\lambda^\\trans(h_1(x) - y) + \\gamma^\\trans(h_2(x) - z). \\]\n","sha256_utf8":"490813a9907096a5c9e1f860ed87544a38912152cd368e24eb56529e1c115568"},{"start":55709,"end":56210,"start_line":837,"end_line":846,"raw_tex":"    \\begin{subequations}\n        \\label{eqn:reformulated-kkt}\n        \\begin{align}\n\t\t\t y = h_1(x),\\ z = h_2(x), \\label{eqn:reformulated-kkt-feas} \\\\\n\t\t\t z \\in \\R^q_-,\\ \n            \\gamma \\in \\R^q_+,\\ \\gamma^\\trans z = 0, \\label{eqn:reformulated-kkt-z} \\\\\n            0 \\in \\partial_y \\tilde \\cL(x, y, z, \\lambda, \\gamma), \\label{eqn:reformulated-kkt-y} \\\\\n            \\mathrm{grad}_x\\, \\tilde \\cL(x, y, z, \\lambda, \\gamma) = 0. \\label{eqn:reformulated-kkt-x}\n        \\end{align}\n\t\\end{subequations}\n","sha256_utf8":"28bfed5fc2c03e1a79187de7bc4d158216af78983a014cf87d9822b15cf5ac8e"}]},{"id":"S015","semantic_type":"theorem","kind":"theorem","title":"约束残差趋零","source_units":[{"start":49514,"end":50030,"start_line":747,"end_line":754,"raw_tex":"\\begin{theorem}\n\t\\label{thm:feasibility}\n\tLet $\\{ (x_k, y_k, z_k) \\}$ be the sequence generated by Algorithm~\\ref{alg:alm-outer}.\n\tThen, we have\n\t\\[ \\lim_{k\\to \\infty} \\left( \\norm{h_1(x_k) - y_k}_2 + \\norm{h_2(x_k) - z_k}_2 \\right) = 0. \\]\n\tConsequently, if $x_* \\in \\cM$ is an accumulation point of $\\{ x_k \\}$, then $(x_*, h_1(x_*), h_2(x_*))$ is a feasible accumulation point of $\\left\\{ (x_k, y_k, z_k) \\right\\}$. %\n\tMoreover, $\\{ x_k \\}$ always contains an accumulation point if $\\cM$ is compact.\n\\end{theorem}","sha256_utf8":"848b332c3c18158456709a5436a3ce028d60984f58587b0c814734aeafd9b234"}]},{"id":"S016","semantic_type":"proof","kind":"proof","title":"可行性完整证明","source_units":[{"start":50031,"end":53087,"start_line":755,"end_line":796,"raw_tex":"\\begin{proof}\n    First, consider the case where $\\{ \\sigma_k \\}$ is bounded. There exists $k_1 \\in \\N$ such that $\\delta_{k+1} \\leq \\tau \\delta_k$ for any $k \\geq k_1$.\n    Then, $\\lim_{k \\to \\infty} \\delta_k = 0$ since $\\tau < 1$. \n    By the definition of $\\delta_k$, we know $\\lim_{k \\to \\infty} \\norm{h_1(x_k) - y_k}_2 = 0$ and $\\lim_{k\\to \\infty} \\norm{h_2(x_k) - z_k}_2 = 0$. \n\n\tIn the case where $\\{ \\sigma_k \\}$ is unbounded. By the update rule, $\\sigma_k$ is updated for infinitely many times. Then, we can find $k_1 < k_2 < \\cdots$ such that\n\t\\[ \\begin{aligned}\n\t\t\t\\sigma_k = \\sigma_{k_i} &= \\max\\big\\{ \\rho \\sigma_{k_{i-1}}, \\norm{\\lambda_{k_i}}_2^{1 + \\alpha}, \\norm{\\gamma_{k_i}}_2^{1 + \\alpha} \\big\\},\\ \\forall \\, k_i \\leq k < k_{i+1}.\n\t\\end{aligned} \\]\n\n\n\tFrom \\eqref{eqn:subproblem-stopping-rule} and the definition of $L_k(x)$, we know that $L_{\\sigma_{k}}(x_{k}, y_{k}, z_{k}, \\lambda_{k}, \\gamma_{k}) \\leq \\Phi$, where $L_{\\sigma_k}$ is defined in \\eqref{eqn:L_sigma}. Therefore, we have\n\t\\begin{equation}\n\t\t\\norm{h_1(x_{k}) - y_{k} + \\frac{\\lambda_{k}}{\\sigma_{k}}}^2_2 \n\t\t+ \\norm{h_2(x_{k}) - z_{k} + \\frac{\\gamma_{k}}{\\sigma_{k}}}_2^2 \n\t\t\\leq 2\\frac{ \\Phi - f(x_{k}) - \\psi(y_{k}) }{\\sigma_{k}} \n\t\t+ \\frac{\\norm{\\lambda_{k}}_2^2 + \\norm{\\gamma_{k}}_2^2}{\\sigma_{k}^2}.\n\t\t\\label{eqn:feasibility-bounded}\n\t\\end{equation}\n\n\tSince $f + \\psi$ is bounded below and $\\sigma_{k} \\to \\infty$, then the first term of \\eqref{eqn:feasibility-bounded} converges to $0$. Next, we show that the second term also converges to $0$.\n\tNotice that $\\norm{\\lambda_{k_i}}_2^{1+\\alpha} \\leq \\sigma_{k_i}$ and $\\sigma_{k_i} \\to \\infty$, \n\twe have $\\norm{\\lambda_{k_i}}_2 / \\sigma_{k_i} \\leq \\sigma_{k_i}^{-\\frac{\\alpha}{1 + \\alpha}} \\to 0$ as $i \\to \\infty$. A similar result also holds for $\\norm{\\gamma_{k_i}}_2 / \\sigma_{k_i}$. \n\tThen by \\eqref{eqn:feasibility-bounded} and the definition of $\\delta_{k_i}$, we have $\\lim_{i \\to \\infty} \\delta_{k_i} = 0$.\n\tBy the update rule of $\\lambda_k$, for $k_i < k < k_{i+1}$, it holds that \n\t\\[\\begin{aligned}\n\t\t\t\\frac{\\norm{\\lambda_{k}}_2}{\\sigma_{k}} \n\t\t\t&\\leq \\frac{\\norm{\\lambda_{k - 1}}_2}{\\sigma_{k_i}} + \\norm{h_1(x_{k - 1}) - y_{k - 1}}_2 \n\t\t\t\\leq \\frac{\\norm{\\lambda_{k - 1}}_2}{\\sigma_{k_i}} + \\delta_{k-1}.\n\t\\end{aligned} \\]\n\n\tAlso note that $\\delta_k \\leq \\tau\\delta_{k-1}$ for all $k_i \\leq k < k_{i+1} - 1$. Then, for $k_i < k < k_{i+1}$ the following inequality holds by induction\n\t\\[\\begin{aligned}\n\t\t\t\\frac{\\norm{\\lambda_{k}}_2}{\\sigma_{k}} \n\t\t\t&\\leq \\frac{\\norm{\\lambda_{k_i}}_2}{\\sigma_{k_i}} + \\sum_{j=k_i}^{k-1} \\delta_j\n\t\t\t\\leq \\frac{\\norm{\\lambda_{k_i}}_2}{\\sigma_{k_i}} + \\delta_{k_i} \\sum_{j=0}^{k - k_i -1} \\tau^{j}\n\t\t\t\\leq \\frac{\\norm{\\lambda_{k_i}}_2}{\\sigma_{k_i}} + \\frac{\\delta_{k_i}}{1 - \\tau}.\n\t\\end{aligned} \\]\n\n\tThus, we have $\\lim_{k \\to \\infty} \\norm{\\lambda_k}_2 / \\sigma_k = 0$.\n\tSimilarly, $\\lim_{k \\to \\infty} \\norm{\\gamma_k}_2 / \\sigma_k = 0$. Therefore, from \\eqref{eqn:feasibility-bounded}, we conclude that $\\lim_{k \\to \\infty} \\delta_k = 0$. \\hfill $\\Box$\n\\end{proof}","sha256_utf8":"1ab9a93dd014e5926d818c90a7aa1a0960e30f7ab27ba4d50a3a8512dfabb610"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"有界罚参数下残差收缩","source_units":[{"start":50031,"end":50415,"start_line":755,"end_line":758,"raw_tex":"\\begin{proof}\n    First, consider the case where $\\{ \\sigma_k \\}$ is bounded. There exists $k_1 \\in \\N$ such that $\\delta_{k+1} \\leq \\tau \\delta_k$ for any $k \\geq k_1$.\n    Then, $\\lim_{k \\to \\infty} \\delta_k = 0$ since $\\tau < 1$. \n    By the definition of $\\delta_k$, we know $\\lim_{k \\to \\infty} \\norm{h_1(x_k) - y_k}_2 = 0$ and $\\lim_{k\\to \\infty} \\norm{h_2(x_k) - z_k}_2 = 0$. \n","sha256_utf8":"e69ebdcbd822c2716e8f98609dcd0bd5e75cce9d7edc04a6566edbdf71a56ff7"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"划分罚参数更新区间","source_units":[{"start":50416,"end":50800,"start_line":760,"end_line":763,"raw_tex":"\tIn the case where $\\{ \\sigma_k \\}$ is unbounded. By the update rule, $\\sigma_k$ is updated for infinitely many times. Then, we can find $k_1 < k_2 < \\cdots$ such that\n\t\\[ \\begin{aligned}\n\t\t\t\\sigma_k = \\sigma_{k_i} &= \\max\\big\\{ \\rho \\sigma_{k_{i-1}}, \\norm{\\lambda_{k_i}}_2^{1 + \\alpha}, \\norm{\\gamma_{k_i}}_2^{1 + \\alpha} \\big\\},\\ \\forall \\, k_i \\leq k < k_{i+1}.\n\t\\end{aligned} \\]\n","sha256_utf8":"7d017455db83d9686113c7458af6c7aadc34b4b6e81d0ac09e8f75a66eec7bdc"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"函数上界控制移位残差","source_units":[{"start":50802,"end":51373,"start_line":766,"end_line":773,"raw_tex":"\tFrom \\eqref{eqn:subproblem-stopping-rule} and the definition of $L_k(x)$, we know that $L_{\\sigma_{k}}(x_{k}, y_{k}, z_{k}, \\lambda_{k}, \\gamma_{k}) \\leq \\Phi$, where $L_{\\sigma_k}$ is defined in \\eqref{eqn:L_sigma}. Therefore, we have\n\t\\begin{equation}\n\t\t\\norm{h_1(x_{k}) - y_{k} + \\frac{\\lambda_{k}}{\\sigma_{k}}}^2_2 \n\t\t+ \\norm{h_2(x_{k}) - z_{k} + \\frac{\\gamma_{k}}{\\sigma_{k}}}_2^2 \n\t\t\\leq 2\\frac{ \\Phi - f(x_{k}) - \\psi(y_{k}) }{\\sigma_{k}} \n\t\t+ \\frac{\\norm{\\lambda_{k}}_2^2 + \\norm{\\gamma_{k}}_2^2}{\\sigma_{k}^2}.\n\t\t\\label{eqn:feasibility-bounded}\n\t\\end{equation}\n","sha256_utf8":"150b5a15b79315c18d53bbbeb8df7fcb1cb4f3e8a33f036fd251709b18df731f"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"更新时刻的残差趋零","source_units":[{"start":51374,"end":51989,"start_line":775,"end_line":778,"raw_tex":"\tSince $f + \\psi$ is bounded below and $\\sigma_{k} \\to \\infty$, then the first term of \\eqref{eqn:feasibility-bounded} converges to $0$. Next, we show that the second term also converges to $0$.\n\tNotice that $\\norm{\\lambda_{k_i}}_2^{1+\\alpha} \\leq \\sigma_{k_i}$ and $\\sigma_{k_i} \\to \\infty$, \n\twe have $\\norm{\\lambda_{k_i}}_2 / \\sigma_{k_i} \\leq \\sigma_{k_i}^{-\\frac{\\alpha}{1 + \\alpha}} \\to 0$ as $i \\to \\infty$. A similar result also holds for $\\norm{\\gamma_{k_i}}_2 / \\sigma_{k_i}$. \n\tThen by \\eqref{eqn:feasibility-bounded} and the definition of $\\delta_{k_i}$, we have $\\lim_{i \\to \\infty} \\delta_{k_i} = 0$.\n","sha256_utf8":"a0e7ded68b7181365c9277c0df3ac146f6880a1e81d71036b240493fc91520f0"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"区间内乘子比值递推","source_units":[{"start":51989,"end":52310,"start_line":779,"end_line":784,"raw_tex":"\tBy the update rule of $\\lambda_k$, for $k_i < k < k_{i+1}$, it holds that \n\t\\[\\begin{aligned}\n\t\t\t\\frac{\\norm{\\lambda_{k}}_2}{\\sigma_{k}} \n\t\t\t&\\leq \\frac{\\norm{\\lambda_{k - 1}}_2}{\\sigma_{k_i}} + \\norm{h_1(x_{k - 1}) - y_{k - 1}}_2 \n\t\t\t\\leq \\frac{\\norm{\\lambda_{k - 1}}_2}{\\sigma_{k_i}} + \\delta_{k-1}.\n\t\\end{aligned} \\]\n","sha256_utf8":"07ff87bfcc233760852d731fd02869ce1627d21375cfe1d21338cc12c1778834"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"几何求和控制乘子比值","source_units":[{"start":52311,"end":52819,"start_line":786,"end_line":792,"raw_tex":"\tAlso note that $\\delta_k \\leq \\tau\\delta_{k-1}$ for all $k_i \\leq k < k_{i+1} - 1$. Then, for $k_i < k < k_{i+1}$ the following inequality holds by induction\n\t\\[\\begin{aligned}\n\t\t\t\\frac{\\norm{\\lambda_{k}}_2}{\\sigma_{k}} \n\t\t\t&\\leq \\frac{\\norm{\\lambda_{k_i}}_2}{\\sigma_{k_i}} + \\sum_{j=k_i}^{k-1} \\delta_j\n\t\t\t\\leq \\frac{\\norm{\\lambda_{k_i}}_2}{\\sigma_{k_i}} + \\delta_{k_i} \\sum_{j=0}^{k - k_i -1} \\tau^{j}\n\t\t\t\\leq \\frac{\\norm{\\lambda_{k_i}}_2}{\\sigma_{k_i}} + \\frac{\\delta_{k_i}}{1 - \\tau}.\n\t\\end{aligned} \\]\n","sha256_utf8":"6e6a3bfd2711ada4c242240e730185bb30a33edfad0273cc1a00ced1de643293"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"全序列可行残差趋零","source_units":[{"start":52820,"end":53087,"start_line":794,"end_line":796,"raw_tex":"\tThus, we have $\\lim_{k \\to \\infty} \\norm{\\lambda_k}_2 / \\sigma_k = 0$.\n\tSimilarly, $\\lim_{k \\to \\infty} \\norm{\\gamma_k}_2 / \\sigma_k = 0$. Therefore, from \\eqref{eqn:feasibility-bounded}, we conclude that $\\lim_{k \\to \\infty} \\delta_k = 0$. \\hfill $\\Box$\n\\end{proof}","sha256_utf8":"5e41154769936da90aecf827b156345ac665d6e6adb62fc894aba6924b7d2847"}]},{"id":"S024","semantic_type":"external_fact","kind":"external_fact","title":"标准Moreau包络定理","source_units":[],"declared_external":true,"declaration_text":["Theorem 4.1.4 of hiriart1993convex supplies continuous differentiability of the Moreau–Yosida envelope of a proper closed convex function and the identity expressing its gradient as the scaled difference between the argument and its proximal point. The source statement B0057 and the author's subsequent use are retained without alteration."]},{"id":"S025","semantic_type":"external_fact","kind":"external_fact","title":"凸次微分有界性与闭图性","source_units":[],"declared_external":true,"declaration_text":["For a finite convex ψ:R^m→R, its subdifferential is locally bounded and has closed graph: if y_j→y and λ_j→λ with λ_j∈∂ψ(y_j), then λ∈∂ψ(y). Local boundedness is cited to Corollary 24.5.1 of rockafellar1970convex."]},{"id":"S026","semantic_type":"external_fact","kind":"external_fact","title":"近端与正交投影最优条件","source_units":[],"declared_external":true,"declaration_text":["For convex ψ and σ>0, y minimizes ψ(y)+(σ/2)||a-y||² if and only if 0∈∂ψ(y)+σ(y-a). If z=Π_{R_-^q}(a), then a-z≥0 and (a-z)^Tz=0."]},{"id":"S027","semantic_type":"external_fact","kind":"external_fact","title":"锥的Carathéodory约化","source_units":[],"declared_external":true,"declaration_text":["Any vector represented as a nonnegative combination of finitely many vectors admits a representation with nonnegative coefficients using a linearly independent subset of those vectors; this conic Carathéodory theorem is cited to bertsekas1997nonlinear."]},{"id":"S028","semantic_type":"external_fact","kind":"external_fact","title":"紧致性与连续极限","source_units":[],"declared_external":true,"declaration_text":["Bounded sequences in finite-dimensional Euclidean spaces have convergent subsequences, and sequences in compact manifolds have accumulation points. Continuous maps preserve limits, and limits in closed constraint sets remain feasible."]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"调用可行性定理","source_units":[{"start":60131,"end":60263,"start_line":897,"end_line":898,"raw_tex":"\\begin{proof}\n    First, from Theorem~\\ref{thm:feasibility}, we obtain the feasibility condition \\eqref{eqn:reformulated-kkt-feas}.\n","sha256_utf8":"f4d26c96997fa76202713c86038ad604684f726706ed90023224846fae1c3326"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"近端变量的包络梯度表达","source_units":[{"start":60263,"end":60814,"start_line":899,"end_line":905,"raw_tex":"    From \\eqref{eqn:subproblem-y}, \\eqref{eqn:subproblem-z} and the property of Moreau-Yosida regularizations in Proposition~\\ref{prop:moreau-yosida}, we know that\n    \\[ \\begin{aligned} \n        y_k %\n        = h_1(x_k) +  \\frac{\\lambda_k}{\\sigma_k} - \\frac{1}{\\sigma_k} \\nabla \\psi^{\\sigma_k}\\left (h_1(x_k) +  \\frac{\\lambda_k}{\\sigma_k}\\right), \\text{ and }\n        z_k %\n        = h_2(x_k) +  \\frac{\\gamma_k}{\\sigma_k} - \\frac{1}{\\sigma_k} \\nabla \\delta_{\\R_-^q}^{\\sigma_k}\\left (h_2(x_k) +  \\frac{\\gamma_k }{ \\sigma_k}\\right). \n\t\\end{aligned} \\]\n","sha256_utf8":"7e322a61c185fa27b3ee6bf28fc3a47110683facf13222133cfe5798f030ddc1"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"子问题梯度的乘子表达","source_units":[{"start":60814,"end":61161,"start_line":906,"end_line":914,"raw_tex":"    From the definition of $\\lambda_{k+1}$ and $\\gamma_{k+1}$, combining the above two equations, we have\n\t\\begin{equation}\n\t\t\\label{eqn:grad_Lk}\n\t\t\\begin{aligned}\n\t\t\t& \\grad L_k(x_k)\n\t\t\t= \\grad f(x_k) + \\sum_{i=1}^m [\\lambda_{k+1}]_i \\grad\\,[h_1(x_k)]_i \n\t\t\t+ \\sum_{i=1}^q [\\gamma_{k+1}]_i \\grad\\, [h_2(x_k)]_i,  \n\t\t\\end{aligned}\n\t\\end{equation}\n","sha256_utf8":"d6cec3d8eb3e91ece3d8125afe434e2413f0ea2e7c18b6a504ea20477c622a39"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"脚注证明黎曼梯度链式法则","source_units":[{"start":61161,"end":62016,"start_line":915,"end_line":925,"raw_tex":"\twhere the chain rule for gradients on manifolds is used\\footnote{\n\tLet $g: \\cM \\to \\R^m$ and $f: \\R^m \\to \\R$ be two smooth maps,\n\twe can use Definition~\\ref{def:riemannian-gradient} to compute the gradient of $f \\circ g$.\n\tFor $p \\in \\cM$, $\\xi_p \\in T_p\\cM$, choosing any curve $\\gamma: (-1, 1) \\to \\cM$ such that $\\gamma(0) = p$ and $\\dot \\gamma(0) = \\xi_p$, we have\n\t$\\inner{\\xi_p}{\\grad\\,(f\\circ g)(p)} = \\xi_p (f \\circ g) \n\t= \\left.\\frac{\\dd (f \\circ g \\circ \\gamma)(t)}{\\dd t}\\right|_{t=0} \n\t= \\sum_{i=1}^m\\alpha_i\\left.\\frac{\\dd (g_i \\circ \\gamma)(t)}{\\dd t}\\right|_{t=0} \n\t= \\sum_{i=1}^m \\alpha_i \\cdot (\\xi_p g_i) \n\t= \\sum_{i=1}^m \\alpha_i \\inner{\\xi_p}{\\grad\\,[g(p)]_i}$, where $\\alpha := \\nabla f(q)$, $q := g(\\gamma(t))$ and $g_i$ is the $i$-th component of $g$.\n\tThus, $\\grad\\, (f \\circ g)(p) = \\sum_{i=1}^m \\alpha_i \\grad\\, [g(p)]_i$.\n\t}.\n","sha256_utf8":"15b83aa6ccaf3b58c7c388513be68bdd3202bc168985fb4c1b8fd8ac312e2d84"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"等式乘子有界与次梯度极限","source_units":[{"start":62017,"end":62742,"start_line":927,"end_line":934,"raw_tex":"    Since $y_k = \\prox_{\\psi/\\sigma_k} (h_1(x_k) +  \\lambda_k/\\sigma_k)$, then\n\t\\begin{equation}\n\t\t\\label{eqn:grad_yk}\n\t\t0 \\in \\partial \\psi(y_k) - (\\sigma_k(h_1(x_k) - y_k) +  \\lambda_k) = \\partial \\psi(y_k) - \\lambda_{k+1}. \n\t\\end{equation}\n\tNote that $\\lim_{K \\ni k \\to \\infty} y_k = y_*$, then $\\left\\{ y_k \\right\\}_{k\\in K}$ is bounded.\n\tBy the locally boundedness of the subdifferential~\\cite[Corollary 24.5.1]{rockafellar1970convex}, $\\bigcup_{k \\in K} \\partial \\psi(y_k)$ is also bounded.\n\tTherefore, $\\{ \\lambda_{k+1} \\}_{k \\in K}$ is bounded, so we can choose $K_1 \\subseteq K$, $\\lambda_* \\in \\R^m$ such that $\\lim_{ K_1 \\ni k \\to \\infty} \\lambda_{k+1} = \\lambda_*$, which implies $\\lambda_*\\in\\partial\\psi(y_*)$.\n","sha256_utf8":"ecd6664a371b1a765abb7e595870f3e9b91d06741dec51a7babcd9d557346129"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"投影产生非负互补乘子","source_units":[{"start":62743,"end":63182,"start_line":936,"end_line":942,"raw_tex":"\tOn the other hand, since \n\t\\[ z_k = \\Pi_{\\R_-^q} \\left ( h_2(x_k) +  \\frac{\\gamma_k}{\\sigma_k} \\right ) = \\argmin_{z \\leq 0} \\norm{h_2(x_k) +  \\frac{\\gamma_k}{\\sigma_k} - z}^2_2, \\]\nthen from the optimality condition of the above problem, it holds that\n\t\\begin{equation}\n\t\t\\label{eqn:grad_zk}\n\t\t0 = [\\sigma_k( h_2(x_k) - z_k) +  \\gamma_k]^\\trans z_k = \\gamma_{k+1}^\\trans z_k \\quad \\mathrm{and} \\quad \\gamma_{k+1} \\geq 0.\n\t\\end{equation}\n","sha256_utf8":"0eb8f2f63f1f3fd5ebcdf1dc1fb6f2ddfedaa33936613f890a46f7054566208a"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"排除极限非活跃约束","source_units":[{"start":63185,"end":63767,"start_line":946,"end_line":955,"raw_tex":"\tLet $\\cA_{k} = \\{ i \\in [q] : [z_k]_i = 0 \\}$, from \\eqref{eqn:grad_zk}, we know $[\\gamma_{k+1}]_i = 0$ for $i \\notin \\cA_k$. \n\tDefine $\\cA_* := \\{ i \\in [q] : [z_*]_i = 0 \\}$.\n\tSince $z_k \\to z_*$, we also find that for sufficiently large $k$ and $i \\notin \\cA_*$, $[\\gamma_{k+1}]_i = 0$.\n\tThen, when $k$ is large enough \\eqref{eqn:grad_Lk} can be written as \n\t\\begin{equation}\n\t\t\\label{eqn:alm-Astar-equ}\n\t\t\t\\grad L_k(x_k)\n\t\t\t= \\grad f(x_k) + \\sum_{i=1}^m [\\lambda_{k+1}]_i \\grad\\,[h_1(x_k)]_i \n\t\t\t+ \\sum_{i \\in \\cA_*} [\\gamma_{k+1}]_i \\grad\\, [h_2(x_k)]_i.  \n    \\end{equation}\n","sha256_utf8":"b4345275a89630db603645f63b10658f9080e94a699c9f5d72f303c8afb229ec"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"约化为固定独立梯度子集","source_units":[{"start":63767,"end":64469,"start_line":956,"end_line":964,"raw_tex":"\tUsing the Carath\\'eodory's theorem of cones~\\cite{bertsekas1997nonlinear}, there exist $J_k \\subseteq \\cA_*$ and $[\\hat \\gamma_k]_j \\geq 0$, where $j \\in J_k$, such that $\\left\\{ \\grad\\, [h_2(x_k)]_j : j \\in J_k \\right\\}$ are linearly independent and\n\t\\begin{equation}\n\t\t\\label{eqn:alm-J_k-equ}\n\t\t\t\\grad L_k(x_k)\n\t\t\t= \\grad f(x_k) + \\sum_{i=1}^m [\\lambda_{k+1}]_i \\grad\\,[h_1(x_k)]_i \n\t\t\t+ \\sum_{j \\in J_k} [\\hat\\gamma_{k}]_j \\grad\\, [h_2(x_k)]_j,\n\t\\end{equation}\n\tSince $J_k \\subseteq [q]$ is a finite set, we can choose $J_*$ and $K_2 \\subseteq K_1$ such that $J_k = J_*$ for all $k \\in K_2$ and $|K_2| = \\infty$.\n\tDefine $M_k = \\max\\left\\{ [\\hat\\gamma_{k}]_i : i \\in J_* \\right\\}$ for $k \\in K_2$.\n","sha256_utf8":"57abe12f3c1168bf498088065cda51d0bdbbc9d0b3c9bde028cf812a9503e5d4"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"有界约化乘子给出KKT极限","source_units":[{"start":64469,"end":64936,"start_line":965,"end_line":967,"raw_tex":"\tWhen $\\{ M_k \\}_{k \\in K_2}$ is bounded, then we can find $K_3 \\subseteq K_2$ and $\\gamma_* \\in \\R^p_+$ such that $[\\gamma_*]_i = 0$ for $i \\notin J_*$ and $\\lim_{ K_3 \\ni k \\to \\infty} [\\hat \\gamma_k]_i = [\\gamma_*]_i$ for $i \\in J_*$.\n\tUsing the fact that $\\norm{\\grad L_k(x_k)} \\to 0$, \\eqref{eqn:grad_Lk} implies \\eqref{eqn:reformulated-kkt-x}.\n\tSince $J_* \\subseteq \\cA_*$ and $[\\gamma_*]_i = 0$ for $i \\notin J_*$, then \\eqref{eqn:reformulated-kkt-z} follows.\n","sha256_utf8":"11603131965d43dfe943714aa4df87db7e0917832465097155b071b8cfbd6575"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"CPLD排除约化乘子无界","source_units":[{"start":64937,"end":65876,"start_line":969,"end_line":972,"raw_tex":"\tWhen $\\{ M_k \\}_{k\\in K_2}$ is unbounded, we can find $K_4 \\subseteq K_2$ and $\\hat \\gamma \\in \\R_+^q$ such that $\\lim_{ K_4 \\ni k \\to \\infty} [\\hat \\gamma_k]_i / M_k = [\\hat \\gamma]_i$ for $i \\in J_*$ and $[\\hat \\gamma]_i = 0$ otherwise. By the definition of $M_k$, we have $\\hat\\gamma\\neq 0$ and $\\|\\hat\\gamma\\|_\\infty = 1$.\n\tDividing \\eqref{eqn:alm-J_k-equ} by $M_k$, using the boundedness of $\\grad L_k(x_k)$, $\\grad f(x_k)$ and $\\grad\\, [h_1(x_k)]_i$, letting $k \\in K_4 \\to \\infty$ and noticing that $\\lambda_{k + 1} \\to \\lambda_*$, $x_k \\to x_*$, we can get\n\t\\[  \\sum_{j\\in J_*} [\\hat \\gamma]_j \\grad\\, [h_2(x_*)]_j = 0.  \\]\n\tDue to $\\normz{\\hat \\gamma}_\\infty = 1$ and $\\hat \\gamma \\geq 0$, we know $\\left\\{ \\grad\\, [h_2(x_*)]_j : j \\in J_* \\right\\}$ are linearly dependent with non-negative coefficients. However, they are linearly independent near $x_*$, which contradicts to the CPLD assumption. Thus, $M_k$ is always bounded.\n","sha256_utf8":"bb9b84ab8dfe380a1f0c6f7bba139039c40c937f94ad78b494944859a9ed3aa7"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"LICQ保证原始乘子子列收敛","source_units":[{"start":65876,"end":66182,"start_line":973,"end_line":975,"raw_tex":"\tMoreover, when the LICQ holds at $x_*$ but $\\{ \\gamma_{k + 1} \\}$ is unbounded, we can divide \\eqref{eqn:alm-Astar-equ} by $\\norm{\\gamma_{k + 1}}_2$ and yield a contradiction to the LICQ condition. Therefore, $\\{ \\gamma_{k+1} \\}$ is bounded and contains a convergent subsequence.\n\\hfill $\\Box$\n\\end{proof}","sha256_utf8":"9dca49e6750be4f34d1273612406a562e82bf325fae38809a9a6a6209f9545ef"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S009","kind":"supports"},{"source_id":"S003","target_id":"S012","kind":"supports"},{"source_id":"S003","target_id":"S014","kind":"supports"},{"source_id":"S004","target_id":"S020","kind":"supports"},{"source_id":"S004","target_id":"S033","kind":"supports"},{"source_id":"S004","target_id":"S035","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S013","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S032","kind":"supports"},{"source_id":"S008","target_id":"S033","kind":"supports"},{"source_id":"S009","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S019","kind":"supports"},{"source_id":"S011","target_id":"S030","kind":"supports"},{"source_id":"S012","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S018","kind":"supports"},{"source_id":"S012","target_id":"S019","kind":"supports"},{"source_id":"S012","target_id":"S021","kind":"supports"},{"source_id":"S012","target_id":"S022","kind":"supports"},{"source_id":"S012","target_id":"S030","kind":"supports"},{"source_id":"S012","target_id":"S031","kind":"supports"},{"source_id":"S012","target_id":"S033","kind":"supports"},{"source_id":"S012","target_id":"S034","kind":"supports"},{"source_id":"S012","target_id":"S037","kind":"supports"},{"source_id":"S013","target_id":"S035","kind":"supports"},{"source_id":"S013","target_id":"S038","kind":"supports"},{"source_id":"S013","target_id":"S039","kind":"supports"},{"source_id":"S014","target_id":"S029","kind":"supports"},{"source_id":"S014","target_id":"S037","kind":"supports"},{"source_id":"S015","target_id":"S029","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S016","kind":"supports"},{"source_id":"S019","target_id":"S016","kind":"supports"},{"source_id":"S020","target_id":"S016","kind":"supports"},{"source_id":"S021","target_id":"S016","kind":"supports"},{"source_id":"S022","target_id":"S016","kind":"supports"},{"source_id":"S023","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S018","target_id":"S022","kind":"supports"},{"source_id":"S020","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S019","target_id":"S023","kind":"supports"},{"source_id":"S020","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S024","target_id":"S011","kind":"supports"},{"source_id":"S025","target_id":"S033","kind":"supports"},{"source_id":"S026","target_id":"S033","kind":"supports"},{"source_id":"S026","target_id":"S034","kind":"supports"},{"source_id":"S027","target_id":"S036","kind":"supports"},{"source_id":"S028","target_id":"S015","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S033","kind":"supports"},{"source_id":"S028","target_id":"S035","kind":"supports"},{"source_id":"S028","target_id":"S037","kind":"supports"},{"source_id":"S028","target_id":"S038","kind":"supports"},{"source_id":"S028","target_id":"S039","kind":"supports"},{"source_id":"S029","target_id":"S002","kind":"supports"},{"source_id":"S030","target_id":"S002","kind":"supports"},{"source_id":"S031","target_id":"S002","kind":"supports"},{"source_id":"S032","target_id":"S002","kind":"supports"},{"source_id":"S033","target_id":"S002","kind":"supports"},{"source_id":"S034","target_id":"S002","kind":"supports"},{"source_id":"S035","target_id":"S002","kind":"supports"},{"source_id":"S036","target_id":"S002","kind":"supports"},{"source_id":"S037","target_id":"S002","kind":"supports"},{"source_id":"S038","target_id":"S002","kind":"supports"},{"source_id":"S039","target_id":"S002","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S029","target_id":"S035","kind":"supports"},{"source_id":"S031","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S033","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S033","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S029","target_id":"S037","kind":"supports"},{"source_id":"S033","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S033","target_id":"S039","kind":"supports"},{"source_id":"S034","target_id":"S039","kind":"supports"},{"source_id":"S035","target_id":"S039","kind":"supports"}]}}
{"record_id":"2103.11026v1","paper":{"title":"Universal Conditional Gradient Sliding for Convex Optimization","authors":["Yuyuan Ouyang","Trevor Squires"],"arxiv_id":"2103.11026","version":"v1","journal":"SIAM Journal on Optimization","doi":"10.1137/21m1406234","abstract_url":"https://arxiv.org/abs/2103.11026v1","pdf_url":"https://arxiv.org/pdf/2103.11026v1"},"context":{"raw_tex":"In this paper, we study first-order projection-free methods for computing $\\varepsilon$-approximation solutions to convex optimization problems of form\n\\begin{equation}\\label{problem:CP}\n    f^* = \\underset{x\\in X}{\\min}\\: f(x).\n\\end{equation}\n\nHere $X\\subset \\R^n$ is a high-dimensional compact convex set, $f$ is a convex function, and our goal is to compute an $\\varepsilon$-solution $y\\in X$ such that $f(y)-f^*\\le\\varepsilon$. We make the following assumptions concerning the set $X$ and the objective function $f$. Under the Euclidean norm $\\|\\cdot\\|$, we assume that $X$ is compact with diameter\n\\begin{align}\n\t\\label{eq:D}\n\tD_X := & \\underset{x,y \\in X}{\\max}\\: \\norm{x-y} < \\infty\n\\end{align}\n\nand that there exists H\\\"older exponent $\\nu \\in (0,1]$ and constant $M_\\nu>0$ such that\n\\begin{equation}\n\t\\label{def:holderCont}\n\tf(y) \\leq f(x) + \\ip{\\grad f(x),y-x} + \\frac{M_\\nu}{1+\\nu}\\norm{x-y}^{1+\\nu},\\ \\forall x,y\\in X.\n\\end{equation}\n\n\\begin{algorithm}[t]\n\t\\caption{\\label{alg:UCGS} Universal conditional gradient sliding (UCGS) method}\n\t\\footnotesize\n\t\\begin{algorithmic}\n\t\t\\State\n\t\tStart: Choose tolerance $\\varepsilon > 0$ and initial iteration $x_0 \\in X$.  Set $y_0 = x_0$. \n\t\t\\For {$k=1,2,\\ldots,$}\n\t\t\\State Decide $L_k >0$ such that\n\t\t\\begin{align}\n\t\t\tf(y_k) \\leq f(z_k) + \\ip{\\grad f(z_k),y_k-z_k} + \\frac{L_k}{2}\\norm{y_k-z_k}^2 + \\frac{\\varepsilon}{2}\\gamma_k\\label{def:lk}\n\t\t\\end{align}\n\t\t\\State where \n\t\t\\begin{align}\n\t\t\t\\label{def:gammak} \\gamma_k &:= \\begin{cases} 1, &k = 1\\\\ \\text{positive solution to } \\Gamma_{k-1}(1-\\gamma_k)=\\frac{L_k\\gamma_k^2}{k}, &k > 1 \\end{cases}\\\\\n\t\t\t\\label{def:zk} z_k &:= (1-\\gamma_k)y_{k-1} + \\gamma_k x_{k-1}\\\\\n\t\t\t\\label{def:xk} x_k &:= \\text{ACGM}(\\grad f(z_k), x_{k-1},\\beta_k,\\eta_k)\\\\\n\t\t\t\\label{def:yk} y_k &:= (1-\\gamma_k)y_{k-1} + \\gamma_k x_k\\\\\n\t\t\t\\label{def:Gammakk}\\Gamma_k &:= \\frac{L_k\\gamma_k^2}{k}.\n\t\t\\end{align}\n\t\t\n\t\t\\State \n\t\tCompute an approximate solution $s_k$ to the problem \n\t\t\\begin{align}\n\t\t    \\label{def:lkproblem}\n\t\t    \\min_{x\\in X}\\ell_k(x) := \\Gamma_k \\sum_{i=1}^k \\frac{\\gamma_i}{\\Gamma_i}\\left( f(z_i) + \\ip{\\grad f(z_i),x-z_i}\\right)\n\t\t\\end{align}\n\t\tsuch that $\\ell_k(s_k) - \\min_{x\\in X}\\ell_k(x)\\le \\varepsilon_k$. Terminate and output $y_k$ as an approximate solution if\n\t\t\\begin{align}\\label{eq:stopCrit}\n\t\t\tf(y_k) - \\ell_k(s_k) + \\varepsilon_k \\leq \\varepsilon.\n\t\t\\end{align}\n\t\t\\EndFor\n\t\t\\\\\n\t\t\n\t\t\n\t\t\\Procedure{$u^+ =$ ACGM}{$g,u,\\beta, \\eta$} \n\t\t\\State Goal: Compute $u^+$ such that $\\max_{x\\in X}\\ip{\\grad\\phi(u^+), u^+-x}\\le \\eta$, where\n\t\t\\begin{align}\n\t\t    \\label{def:phi}\n\t\t    \\phi(x):=\\ip{g, x} + \\frac{\\beta}{2}\\|x - u\\|^2.\n\t\t\\end{align}\n\t\t\\State Start: Set $u^0 = u$. \n\t\t\\For {$t=1,2,\\ldots,$}\n\t\t\\State Compute a $\\delta^t$-approximate solution $v^t$ to the problem $\\min_{x\\in X}\\ip{\\nabla\\phi(u^{t-1}),x}$ such that\n\t\t\\begin{equation}\\label{eq:cndgsubproblem}\n\t\t\t\\ip{g + \\beta(u^{t-1}-u),v^t} - \\min_{x\\in X}\\ip{g + \\beta(u^{t-1}-u), x}  \\leq \\delta^t.\n\t\t\\end{equation}\n\t\t\n\t\t\\State Terminate with $u^+:=u^{t-1}$ if\n\t\t\\begin{align}\n\t\t    \\label{eq:actual_termi}\n\t\t    \\ip{g+\\beta(u^{t-1}-u),u^{t-1}-v^t} + \\delta^t \\leq \\eta.\n\t\t\\end{align}\n\t\t\\State Otherwise, compute\n% \t\t\\begin{align}\n            $\n\t\t\tu^{t}=(1-\\alpha^t)u^{t-1}+\\alpha^tv^t\n% \t\t\t, \\text{ where }\n% \t\t\t\\alpha^t := \\min\\: \\left\\{1,\\frac{\\ip{g-\\beta(u-u^{t-1}),u^{t-1}-v^t}}{\\beta\\norm{v^t-u^{t-1}}^2}\\right\\}\n\t\t\t.\n\t\t\t$\n% \t\t\\end{align}\n\t\t% \t\t\t\\begin{enumerate}\n\t\t% \t\t\t    \\item Set $u_1 = u$ and $t=1$.\n\t\t% \t\t\t    \\item Let $v_t$ be the optimal solution for the subproblem of\n\t\t\n\t\t% \t\t\t    \\item If $V_{g,u,\\beta}(u_t) \\leq \\eta$, set $u^+ = u_t$ and terminate.\n\t\t% \t\t\t    \\item Set $u_{t+1}=(1-\\alpha_t)u_t+\\alpha_tv_t$ where $\\alpha_t = \\min\\: \\{1,\\ip{\\beta(u-u_t)-g,v_t-u_t}/(\\beta\\norm{v_t-u_t}^2)\\}$.\n\t\t% \t\t\t    \\item Set $t\\leftarrow t+1$ and go to step 2. \n\t\t% \t\t\t\\end{enumerate}\n\t\t\\EndFor\n\t\t\\EndProcedure\n\t\\end{algorithmic}\n\\end{algorithm}\n\n\\begin{align}\n\t\\label{def:alpha}\n\t\\alpha^t := \\min\\: \\left\\{1,\\frac{\\ip{g-\\beta(u-u^{t-1}),u^{t-1}-v^t}}{\\beta\\norm{v^t-u^{t-1}}^2}\\right\\}.\n\\end{align}\n\nIt is easy to observe that the above $\\alpha^t$ is the optimal solution to the exact linesearch problem $\\min_{\\alpha\\in[0,1]}\\phi((1-\\alpha)u^{t-1} + \\alpha v^t)$.\n\n\n\\begin{equation}\\label{eq:GammaRelation}\n    \\Gamma_k = \\begin{cases} 1, &k = 1\\\\ \\Gamma_{k-1}(1-\\gamma_k), &k > 1\\end{cases}.\n\\end{equation}\n\nNote that for $\\nu = 1$, the term $\\left(\\frac{1-\\nu}{1+\\nu}\\right)^\\frac{1-\\nu}{1+\\nu}$ can be handled using a continuity argument $\\lim_{\\nu\\to 1}\\left(\\frac{1-\\nu}{1+\\nu}\\right)^\\frac{1-\\nu}{1+\\nu} = 1$. We state an immediate corollary of the above lemma below. \n","units":[{"start":4785,"end":5028,"start_line":119,"end_line":122,"raw_tex":"In this paper, we study first-order projection-free methods for computing $\\varepsilon$-approximation solutions to convex optimization problems of form\n\\begin{equation}\\label{problem:CP}\n    f^* = \\underset{x\\in X}{\\min}\\: f(x).\n\\end{equation}","sha256_utf8":"230f4a13004569898d7b8bfa6534359f5be26efa77dcd0e5986acf0eb9092e95"},{"start":5029,"end":5485,"start_line":123,"end_line":127,"raw_tex":"Here $X\\subset \\R^n$ is a high-dimensional compact convex set, $f$ is a convex function, and our goal is to compute an $\\varepsilon$-solution $y\\in X$ such that $f(y)-f^*\\le\\varepsilon$. We make the following assumptions concerning the set $X$ and the objective function $f$. Under the Euclidean norm $\\|\\cdot\\|$, we assume that $X$ is compact with diameter\n\\begin{align}\n\t\\label{eq:D}\n\tD_X := & \\underset{x,y \\in X}{\\max}\\: \\norm{x-y} < \\infty\n\\end{align}","sha256_utf8":"5521fb8ad9bbfaa8220f58b7e326920d5ae0b746af7becf1d4f21ec130f22a50"},{"start":5486,"end":5728,"start_line":128,"end_line":132,"raw_tex":"and that there exists H\\\"older exponent $\\nu \\in (0,1]$ and constant $M_\\nu>0$ such that\n\\begin{equation}\n\t\\label{def:holderCont}\n\tf(y) \\leq f(x) + \\ip{\\grad f(x),y-x} + \\frac{M_\\nu}{1+\\nu}\\norm{x-y}^{1+\\nu},\\ \\forall x,y\\in X.\n\\end{equation}","sha256_utf8":"c92fddb8f844645c2874950e82e32fdc75c0112cd55cd73f0f72f997e9116402"},{"start":58543,"end":61513,"start_line":581,"end_line":653,"raw_tex":"\\begin{algorithm}[t]\n\t\\caption{\\label{alg:UCGS} Universal conditional gradient sliding (UCGS) method}\n\t\\footnotesize\n\t\\begin{algorithmic}\n\t\t\\State\n\t\tStart: Choose tolerance $\\varepsilon > 0$ and initial iteration $x_0 \\in X$.  Set $y_0 = x_0$. \n\t\t\\For {$k=1,2,\\ldots,$}\n\t\t\\State Decide $L_k >0$ such that\n\t\t\\begin{align}\n\t\t\tf(y_k) \\leq f(z_k) + \\ip{\\grad f(z_k),y_k-z_k} + \\frac{L_k}{2}\\norm{y_k-z_k}^2 + \\frac{\\varepsilon}{2}\\gamma_k\\label{def:lk}\n\t\t\\end{align}\n\t\t\\State where \n\t\t\\begin{align}\n\t\t\t\\label{def:gammak} \\gamma_k &:= \\begin{cases} 1, &k = 1\\\\ \\text{positive solution to } \\Gamma_{k-1}(1-\\gamma_k)=\\frac{L_k\\gamma_k^2}{k}, &k > 1 \\end{cases}\\\\\n\t\t\t\\label{def:zk} z_k &:= (1-\\gamma_k)y_{k-1} + \\gamma_k x_{k-1}\\\\\n\t\t\t\\label{def:xk} x_k &:= \\text{ACGM}(\\grad f(z_k), x_{k-1},\\beta_k,\\eta_k)\\\\\n\t\t\t\\label{def:yk} y_k &:= (1-\\gamma_k)y_{k-1} + \\gamma_k x_k\\\\\n\t\t\t\\label{def:Gammakk}\\Gamma_k &:= \\frac{L_k\\gamma_k^2}{k}.\n\t\t\\end{align}\n\t\t\n\t\t\\State \n\t\tCompute an approximate solution $s_k$ to the problem \n\t\t\\begin{align}\n\t\t    \\label{def:lkproblem}\n\t\t    \\min_{x\\in X}\\ell_k(x) := \\Gamma_k \\sum_{i=1}^k \\frac{\\gamma_i}{\\Gamma_i}\\left( f(z_i) + \\ip{\\grad f(z_i),x-z_i}\\right)\n\t\t\\end{align}\n\t\tsuch that $\\ell_k(s_k) - \\min_{x\\in X}\\ell_k(x)\\le \\varepsilon_k$. Terminate and output $y_k$ as an approximate solution if\n\t\t\\begin{align}\\label{eq:stopCrit}\n\t\t\tf(y_k) - \\ell_k(s_k) + \\varepsilon_k \\leq \\varepsilon.\n\t\t\\end{align}\n\t\t\\EndFor\n\t\t\\\\\n\t\t\n\t\t\n\t\t\\Procedure{$u^+ =$ ACGM}{$g,u,\\beta, \\eta$} \n\t\t\\State Goal: Compute $u^+$ such that $\\max_{x\\in X}\\ip{\\grad\\phi(u^+), u^+-x}\\le \\eta$, where\n\t\t\\begin{align}\n\t\t    \\label{def:phi}\n\t\t    \\phi(x):=\\ip{g, x} + \\frac{\\beta}{2}\\|x - u\\|^2.\n\t\t\\end{align}\n\t\t\\State Start: Set $u^0 = u$. \n\t\t\\For {$t=1,2,\\ldots,$}\n\t\t\\State Compute a $\\delta^t$-approximate solution $v^t$ to the problem $\\min_{x\\in X}\\ip{\\nabla\\phi(u^{t-1}),x}$ such that\n\t\t\\begin{equation}\\label{eq:cndgsubproblem}\n\t\t\t\\ip{g + \\beta(u^{t-1}-u),v^t} - \\min_{x\\in X}\\ip{g + \\beta(u^{t-1}-u), x}  \\leq \\delta^t.\n\t\t\\end{equation}\n\t\t\n\t\t\\State Terminate with $u^+:=u^{t-1}$ if\n\t\t\\begin{align}\n\t\t    \\label{eq:actual_termi}\n\t\t    \\ip{g+\\beta(u^{t-1}-u),u^{t-1}-v^t} + \\delta^t \\leq \\eta.\n\t\t\\end{align}\n\t\t\\State Otherwise, compute\n% \t\t\\begin{align}\n            $\n\t\t\tu^{t}=(1-\\alpha^t)u^{t-1}+\\alpha^tv^t\n% \t\t\t, \\text{ where }\n% \t\t\t\\alpha^t := \\min\\: \\left\\{1,\\frac{\\ip{g-\\beta(u-u^{t-1}),u^{t-1}-v^t}}{\\beta\\norm{v^t-u^{t-1}}^2}\\right\\}\n\t\t\t.\n\t\t\t$\n% \t\t\\end{align}\n\t\t% \t\t\t\\begin{enumerate}\n\t\t% \t\t\t    \\item Set $u_1 = u$ and $t=1$.\n\t\t% \t\t\t    \\item Let $v_t$ be the optimal solution for the subproblem of\n\t\t\n\t\t% \t\t\t    \\item If $V_{g,u,\\beta}(u_t) \\leq \\eta$, set $u^+ = u_t$ and terminate.\n\t\t% \t\t\t    \\item Set $u_{t+1}=(1-\\alpha_t)u_t+\\alpha_tv_t$ where $\\alpha_t = \\min\\: \\{1,\\ip{\\beta(u-u_t)-g,v_t-u_t}/(\\beta\\norm{v_t-u_t}^2)\\}$.\n\t\t% \t\t\t    \\item Set $t\\leftarrow t+1$ and go to step 2. \n\t\t% \t\t\t\\end{enumerate}\n\t\t\\EndFor\n\t\t\\EndProcedure\n\t\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"06238334a17241bea76e825c1603b54c14e225e778cceb8e3dd5478d12a6fd20"},{"start":61943,"end":62095,"start_line":658,"end_line":661,"raw_tex":"\\begin{align}\n\t\\label{def:alpha}\n\t\\alpha^t := \\min\\: \\left\\{1,\\frac{\\ip{g-\\beta(u-u^{t-1}),u^{t-1}-v^t}}{\\beta\\norm{v^t-u^{t-1}}^2}\\right\\}.\n\\end{align}","sha256_utf8":"5663aacb29781d9f61e7b67dec37791e68743236898a666a324ce1887d04bbbe"},{"start":62096,"end":62261,"start_line":662,"end_line":662,"raw_tex":"It is easy to observe that the above $\\alpha^t$ is the optimal solution to the exact linesearch problem $\\min_{\\alpha\\in[0,1]}\\phi((1-\\alpha)u^{t-1} + \\alpha v^t)$.\n","sha256_utf8":"262570bd3e6f559d55d726c4057a362f7af261ca88920818829b9847866572e9"},{"start":63113,"end":63254,"start_line":669,"end_line":671,"raw_tex":"\\begin{equation}\\label{eq:GammaRelation}\n    \\Gamma_k = \\begin{cases} 1, &k = 1\\\\ \\Gamma_{k-1}(1-\\gamma_k), &k > 1\\end{cases}.\n\\end{equation}","sha256_utf8":"9ff5920435913df8722482a931d473362eba31abaeace1febf18960299e43782"},{"start":80455,"end":80721,"start_line":900,"end_line":900,"raw_tex":"Note that for $\\nu = 1$, the term $\\left(\\frac{1-\\nu}{1+\\nu}\\right)^\\frac{1-\\nu}{1+\\nu}$ can be handled using a continuity argument $\\lim_{\\nu\\to 1}\\left(\\frac{1-\\nu}{1+\\nu}\\right)^\\frac{1-\\nu}{1+\\nu} = 1$. We state an immediate corollary of the above lemma below. \n","sha256_utf8":"4a77d5759a0af24477717ee60e64ee8a721fdbc9da6afddb6dd3c6c08f336a56"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{thm:flkDiff}\n    Suppose that we apply UCGS described in Algorithm \\ref{alg:UCGS} with parameters\n    \\begin{equation}\\label{eq:param}\n        \\beta_k = L_k\\gamma_k,\\ \\eta_k = \\frac{L_k\\gamma_kD_X^2}{k},\\text{ and }\\varepsilon_k = \\frac{\\sigma L_k\\gamma_kD_X^2}{2},\n    \\end{equation}\n    and $\\alpha^t$ in \\eqref{def:alpha} and $\\delta^t = \\sigma\\beta D_X^2/t$ in the ACGM procedure, where $\\sigma\\ge 0$ is a parameter related to the accuracy of approximately solving linear objective optimization subproblems.\n    Then \n    % we have\n    % \\begin{align}\n    %     f(y_k) - \\ell_k(x) \\leq \\frac{\\varepsilon}{2} + \\frac{3}{2}L_k\\gamma_k^2 D_X^2,\\ \\forall x\\in X.\n    % \\end{align}\n    % Consequently, \n    Algorithm \\ref{alg:UCGS} terminates with an $\\varepsilon$-solution after at most $N_{\\text{grad}}$ gradient evaluations and $N_{\\text{lin}}$ linear objective optimizations, where\n    \\begin{align}\\label{eq:evaluations}\n    \\begin{aligned}\n        N_{\\text{grad}} :=&\\left\\lceil16\\left(\\frac{(3+\\sigma)^{\\frac{1+\\nu}{2}}M_\\nu D_X^{1+\\nu}}{\\varepsilon}\\right)^\\frac{2}{1+3\\nu}\\right\\rceil\n        \\text{ and } \n        \\\\\n        N_{\\text{lin}} :=& \\left\\lceil\\left(\\frac{7}{2}\\sigma + 3\\right)N_{\\text{grad}}^2 + \\left(\\frac{7}{2}\\sigma + 6\\right)N_{\\text{grad}}\\right\\rceil.\n    \\end{aligned} \n    \\end{align}\n    % As a consequence, the total number of gradient evaluations and linear objective optimizations performed by UCGS to find an $\\varepsilon$ solution of \\eqref{problem:CP} can be bounded by $\\cO((M_{\\nu}D_X^{1+\\nu}/\\varepsilon)^{\\frac{2}{1+3\\nu}})$ and $\\cO((M_{\\nu}D_X^{1+\\nu}/\\varepsilon)^{\\frac{4}{1+3\\nu}})$ respectively. \n\\end{theorem}","units":[{"start":93369,"end":95049,"start_line":1051,"end_line":1073,"raw_tex":"\\begin{theorem}\\label{thm:flkDiff}\n    Suppose that we apply UCGS described in Algorithm \\ref{alg:UCGS} with parameters\n    \\begin{equation}\\label{eq:param}\n        \\beta_k = L_k\\gamma_k,\\ \\eta_k = \\frac{L_k\\gamma_kD_X^2}{k},\\text{ and }\\varepsilon_k = \\frac{\\sigma L_k\\gamma_kD_X^2}{2},\n    \\end{equation}\n    and $\\alpha^t$ in \\eqref{def:alpha} and $\\delta^t = \\sigma\\beta D_X^2/t$ in the ACGM procedure, where $\\sigma\\ge 0$ is a parameter related to the accuracy of approximately solving linear objective optimization subproblems.\n    Then \n    % we have\n    % \\begin{align}\n    %     f(y_k) - \\ell_k(x) \\leq \\frac{\\varepsilon}{2} + \\frac{3}{2}L_k\\gamma_k^2 D_X^2,\\ \\forall x\\in X.\n    % \\end{align}\n    % Consequently, \n    Algorithm \\ref{alg:UCGS} terminates with an $\\varepsilon$-solution after at most $N_{\\text{grad}}$ gradient evaluations and $N_{\\text{lin}}$ linear objective optimizations, where\n    \\begin{align}\\label{eq:evaluations}\n    \\begin{aligned}\n        N_{\\text{grad}} :=&\\left\\lceil16\\left(\\frac{(3+\\sigma)^{\\frac{1+\\nu}{2}}M_\\nu D_X^{1+\\nu}}{\\varepsilon}\\right)^\\frac{2}{1+3\\nu}\\right\\rceil\n        \\text{ and } \n        \\\\\n        N_{\\text{lin}} :=& \\left\\lceil\\left(\\frac{7}{2}\\sigma + 3\\right)N_{\\text{grad}}^2 + \\left(\\frac{7}{2}\\sigma + 6\\right)N_{\\text{grad}}\\right\\rceil.\n    \\end{aligned} \n    \\end{align}\n    % As a consequence, the total number of gradient evaluations and linear objective optimizations performed by UCGS to find an $\\varepsilon$ solution of \\eqref{problem:CP} can be bounded by $\\cO((M_{\\nu}D_X^{1+\\nu}/\\varepsilon)^{\\frac{2}{1+3\\nu}})$ and $\\cO((M_{\\nu}D_X^{1+\\nu}/\\varepsilon)^{\\frac{4}{1+3\\nu}})$ respectively. \n\\end{theorem}","sha256_utf8":"140498d20838d27c867698735f73f8d90156e3a4acae9a244489c4f1d49568be"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\n\t\\label{lem:tele}\n\tSuppose that $\\{a_k\\},\\{b_k\\}\\subset\\R$ and $\\{\\gamma_k\\}\\subset [0,1]$ are sequences that satisfy $\\gamma_1=1$ and\n\t\\begin{align}\n\t\t\\label{eq:tele_assum}\n\t\ta_k \\le (1-\\gamma_k)a_{k-1} + \\gamma_k b_k,\\ \\forall k\\ge 1.\n\t\\end{align}\n\tThen we have\n\t\\begin{align}\n\t\t\\label{eq:tele_res}\n\t\ta_k\\le \\Gamma_k\\sum_{i=1}^{k}\\frac{\\gamma_i}{\\Gamma_i}b_i,\\ \\forall k\\ge 1,\\text{ where }\\Gamma_k:=\\begin{cases}\n\t\t\t1 & \\text{ when }k=1\n\t\t\t\\\\\n\t\t\t\\Gamma_{k-1}(1-\\gamma_k) & \\text{ when }k>1.\n\t\t\\end{cases}\n\t\\end{align}\n\\end{lemma}\n\n\\begin{proof}\n\tDividing both sides of \\eqref{eq:tele_assum} by $\\Gamma_k$ we obtain a series of inequalities with telescoping terms concerning sequence $\\{a_k/\\Gamma_k\\}$. Summing up we obtain \\eqref{eq:tele_res}.\n\\end{proof}\n\nWe proposed to search for such $L_k$ through a backtracking linesearch strategy. In particular, we initialize with any $L_0 \\in \\R$ and choose $L_1 = 2^i L_0$ where $i$ is the smallest integer such that \\eqref{def:lk} is satisfied. At the start of the $k$-th outer iteration where $k>1$, we set $L_k = L_{k-1}/2$ and assess the validity of $L_k$. If it does not satisfy $\\eqref{def:lk}$, we keep backtracking and replacing $L_k$ to $2L_k$ until \\eqref{def:lk} is satisfies. Through this backtracking linesearch strategy, we ensure that our choice of $L_k$ is adaptive and that performance is independent of the choice of $L_0$. Previous literature \\cite{nazari2020backtracking} on backtracking linesearch strategy of CGS require monotonicity of $L_k$ and may suffer from a poorly chosen $L_0$. Second, our termination criterion is based on \\eqref{eq:stopCrit}. We can observe immediately that if the parameter $\\varepsilon_k\\equiv 0$, i.e., $s_k$ is the exact solution to problem \\eqref{def:lkproblem}, then when \\eqref{eq:stopCrit} is satisfied, $y_k$ will be $\\varepsilon$-approximation solution to problem \\eqref{problem:CP}.  To see this, note from \\eqref{eq:GammaRelation} that\n    \\begin{equation}\\label{eq:gammaSum}\n        \\Gamma_k \\sum_{i=1}^k \\frac{\\gamma_i}{\\Gamma_i} = 1\n    \\end{equation}\n    and consequently\n    % \\begin{equation}\\label{eq:lkBound}\n    $$\n        f(y_k) - f^* \\le f(y_k) - \\min_{x\\in X}\\ell_k(x) =  f(y_k) - \\ell_k(s_k).\n    $$\n\n\n\\begin{lemma}\\label{lem:innerIterUCGS}\n    Suppose that $\\{\\lambda^t\\}\\in [0,1]$ is any predetermined sequence satisfying $\\lambda^1= 1$. In the ACGM procedure, if $\\alpha^t$ is chosen such that \n    \\begin{align}\n        \\label{eq:alpha_req}\n        \\phi(u^t)\\le \\phi((1-\\lambda^t)u^{t-1} + \\lambda^tv^t),\\ \\forall t\\ge 1,\n    \\end{align}\n    then we have\n    \\begin{align}\n        \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j}\\: \\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}),u^{j-1} - x}\n        % \\\\\n%        \\leq & \\delta^1 + \\frac{\\beta D_X^2}{2} + \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j}\\left(\\delta^j + \\frac{\\beta D_X^2\\lambda^j}{2} +  \\Lambda^{j-1}\\sum_{i=1}^{j-1}\\frac{\\lambda^i}{\\Lambda^i}\\left(\\delta^i + \\frac{\\beta D_X^2\\lambda^i}{2}\\right)\\right)\n        \\leq &\n        \\left[\\delta^1 + \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j}\\left(\\delta^j + \\Lambda^{j-1}\\sum_{i=1}^{j-1}\\frac{\\lambda^i}{\\Lambda^i}\\delta^i\\right)\\right] \n        \\\\\n        & +\n        \\frac{\\beta D_X^2}{2}\\left[1+\\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j} \\left( \\lambda^j + \\Lambda^{j-1}\\sum_{i=1}^{j-1}\\frac{(\\lambda^i)^2}{\\Lambda^i}\\right)\\right]\n    \\end{align}\n    for all $t \\geq 2$, where\n    \\begin{align}\n        \\label{def:Lambda}\n        \\Lambda^t:=\\begin{cases}\n            1 & \\text{ when }t=1\n            \\\\\n            \\Lambda^{t-1}(1-\\lambda^t) & \\text{ when } t>1.\n        \\end{cases}\n    \\end{align}\n\\end{lemma}\n\n\\begin{proof}\n    Observing that the function $\\phi(x)$ in \\eqref{def:phi} is a strongly convex function with Lipschitz continuous (with constant $\\beta$) gradient, using the assumption \\eqref{eq:alpha_req}, and noting the definition of approximate solution $v^t$ in \\eqref{eq:cndgsubproblem}, we have\n    \\begin{align}\n    \t\\label{eq:phiConvex}\n    \t\\begin{aligned}\n        &\\phi(u^t) - (1-\\lambda^t)\\phi(u^{t-1}) - \\lambda^t(\\phi(u^{t-1}) + \\ip{\\grad \\phi(u^{t-1}),x-u^{t-1}})\n        \\\\\n        \\leq& \\phi((1-\\lambda^t)u^{t-1} + \\lambda^tv^t) - \\phi(u^{t-1}) - \\lambda^t\\ip{\\grad\\phi(u^{t-1}),x-u^{t-1}}\n        \\\\\n        \\le & \\lambda^t\\ip{\\grad \\phi(u^{t-1}), v^t - u^{t-1}} + \\frac{\\beta(\\lambda^t)^2}{2}\\norm{v^t-u^{t-1}}^2 - \\lambda^t\\ip{\\grad \\phi(u^{t-1}),x-u^{t-1}}\n        \\\\\n        =& \\lambda^t\\ip{\\grad\\phi(u^{t-1}),v^t-x} + \\frac{\\beta(\\lambda^t)^2}{2}\\norm{v^t-u^{t-1}}^2\\\\\n        \\leq& \\lambda^t\\delta^t +\\frac{\\beta D_X^2(\\lambda^t)^2}{2},\\ \\forall x\\in X, t\\ge 1.\n\t    \\end{aligned}\n    \\end{align}\n\tDefining $x^*:=\\operatorname{argmin}_{x\\in X}\\phi(x)$, from the above relation we have for any $t\\ge 2$ that\n\t\\begin{align}\n\t\t\\label{eq:tmp1}\n\t\t\\begin{aligned}\n\t\t& \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j} \\underset{x\\in X}{\\max}\\:\\ip{\\grad\\phi(u^{j-1}), u^{j-1}-x}\n\t\t\\\\\n\t\t\\leq& \\sum_{j=2}^t \\frac{1}{\\Lambda^j}[\\phi(u^{j-1}) - \\phi(x^*)] - \\frac{1}{\\Lambda^j}[\\phi(u^j) - \\phi(x^*)] + \\frac{\\lambda^j}{\\Lambda^j}\\delta^j + \\frac{\\beta D_X^2 (\\lambda^j)^2}{2\\Lambda^j}\n\t\t\\\\\n\t\t= & [\\phi(u^1) - \\phi(x^*)] - \\frac{1}{\\Lambda^t}[\\phi(u^t) - \\phi(x^*)] \n\t\t\\\\\n\t\t& + \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j}[\\phi(u^{j-1}) - \\phi(x^*)] + \\frac{\\lambda^j}{\\Lambda^j}\\delta^j + \\frac{\\beta D_X^2(\\lambda^j)^2}{2\\Lambda^j}\n\t\t\\\\\n\t\t\\le & [\\phi(u^1) - \\phi(x^*)] + \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j}[\\phi(u^{j-1}) - \\phi(x^*)] + \\frac{\\lambda^j}{\\Lambda^j}\\delta^j + \\frac{\\beta D_X^2(\\lambda^j)^2}{2\\Lambda^j}.\n\t\t\\end{aligned}\n\t\\end{align}\n\tHere in the equality we use the following observations from the definition of $\\Lambda^t$ in \\eqref{def:Lambda}: $1/\\Lambda^1 = 1$ and $1/\\Lambda^j = 1/\\Lambda^{j-1} + \\lambda^j/\\Lambda^j$ for all $j\\ge 2$, \n%\t\\begin{align}\n%\t\t\\frac{1}{\\Lambda^j} = \\frac{1}{\\Lambda^{j-1}} + \\frac{\\lambda^j}{\\Lambda^{j}},\\ \\forall j\\ge 2,\n%\t\\end{align}\n\n    To finish the proof it suffices to bound $\\phi(u^{j-1})-\\phi(x^*)$ for any $j\\ge 2$. Observing that $\\phi(x)$ in \\eqref{def:phi} is strongly convex and quadratic with\n    \\begin{equation}\n        \\frac{\\beta}{2}\\norm{x-u^{t-1}}^2 = \\phi(x) - (\\phi(u^{t-1})+\\ip{\\grad \\phi(u^{t-1}), x-u^{t-1}}),\\ \\forall x\\in X, t\\ge 1,\n    \\end{equation}\n    we have from \\eqref{eq:phiConvex} (with $x = x^*$) that\n    \\begin{equation}\\label{eq:telescopingBound}\n        [\\phi(u^t)-\\phi(x^*)] - (1-\\lambda^t)[\\phi(u^{t-1})-\\phi(x^*)] \\leq \\lambda^t\\delta^t + \\frac{\\beta D_X^2(\\lambda^t)^2}{2} - \\frac{\\beta\\lambda^t}{2}\\norm{x^*-u^{t-1}}^2.\n    \\end{equation}\n    Applying Lemma \\ref{lem:tele} to the above recurrence relation and ignoring negative terms at the right hand side, we have\n    \\begin{align}\\label{eq:simplePhiBound}\n        \\phi(u^t) - \\phi(x^*) \\leq \\Lambda^t\\sum_{i=1}^t \\frac{\\lambda^i}{\\Lambda^i}\\delta^i + \\frac{\\beta D_X^2(\\lambda^i)^2}{2\\Lambda^i},\\ \\forall t\\ge 1.\n    \\end{align}\n\tWe conclude the lemma immediately by applying the above bound to \\eqref{eq:tmp1} and rearranging terms.\n%\t\\begin{align}\n%\t\t\t& \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j} \\underset{x\\in X}{\\max}\\:\\ip{\\grad\\phi(u^{j-1}), u^{j-1}-x}\n%\t\t\t\\\\\n%\t\t\t\\le & \\delta^1 + \\frac{\\beta D_X^2}{2} + \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j}\\Lambda^{j-1}\\left(\\sum_{i=1}^{j-1} \\frac{\\lambda^i}{\\Lambda^i}\\delta^i + \\frac{\\beta D_X^2(\\lambda^i)^2}{2\\Lambda^i}\\right) + \\frac{\\lambda^j}{\\Lambda^j}\\delta^j + \\frac{\\beta D_X^2(\\lambda^j)^2}{2\\Lambda^j}.\n%\t\\end{align}\n\\end{proof}\n\n\\begin{proposition}\\label{prop:innerIterUCGS}\n\tIn the ACGM procedure, at termination we have\n\t\\begin{align}\n\t    \\label{eq:innerIterUCGS_OC}\n\t    \\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^+), u^+-x} \\le \\eta.\n\t\\end{align}\n\tMoreover, if $\\delta^t = \\sigma\\beta D_X^2/t$ for certain $\\sigma\\ge 0$ and $\\alpha^t$ is chosen such that \n\t\\begin{align}\n\t\t\\label{eq:alpha_req_2t1}\n\t\t\\phi(u^t) \\le \\phi\\left(\\frac{t-1}{t+1}u^{t-1} + \\frac{2}{t+1}v^t\\right),\n\t\\end{align}\n\tthen we have for any $t\\ge 1$ that\n    \\begin{align}\n        \\label{eq:termi_rate}\n        \\begin{aligned}\n        \\underset{j=1, \\dots, t+1}{\\min}\\:\\ip{\\grad \\phi(u^{j-1}), u^{j-1}-v^{j}}\\le & \\underset{j=1, \\dots, t+1}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x}\n        \\\\\n        \\le & \\frac{6(\\sigma+1)\\beta D_X^2}{t}.\n        \\end{aligned}\n    \\end{align}\n    Specially, it takes at most\n    \\begin{align}\n        \\label{def:T}\n        T := 1 + \\left\\lceil \\frac{(7\\sigma+6)\\beta D_X^2}{\\eta}\\right\\rceil \n    \\end{align}\n    iterations for the ACGM procedure to terminate.\n\\end{proposition}\n\n\\begin{proof}\n    From the definition of the approximate solution $v^t$ in \\eqref{eq:cndgsubproblem}, if the termination criterion in \\eqref{eq:actual_termi} of the ACGM procedure is satisfied, then the output $u^+=u^{t-1}$ satisfies\n    \\begin{align}\n        \\max_{x\\in X}\\ip{\\grad \\phi(u^{t-1}), u^{t-1}-x} &= \\max_{x\\in X}\\ip{\\grad \\phi(u^{t-1}), u^{t-1}-v^t} + \\ip{\\grad \\phi(u^{t-1}), v^t - x}\n        \\\\\n        &\\le  (\\eta - \\delta^t) + \\delta^t = \\eta.\n    \\end{align}\n    Therefore \\eqref{eq:innerIterUCGS_OC} holds. To conclude the proposition it suffices to estimate the rate of convergence of $\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{t-1}), u^{t-1}-x}$. To analyze the rate, let us choose $\\lambda^t = 2/(t+1)$ and apply Lemma \\ref{lem:innerIterUCGS}. Then $\\Lambda^t = 2/(t(t+1))$ and\n    \\begin{align}\n        & \\sum_{j=2}^t j\\cdot\\:\\underset{x\\in X}{\\max}\\ip{\\phi(u^{j-1}),u^{j-1}-x}\n        \\\\\n%        \\leq& \\left[\\delta^1 + \\sum_{j=2}^t \\left(j\\delta^j + \\frac{2}{j-1}\\sum_{i=1}^{j-1} i \\delta^i\\right)\\right] + \\frac{\\beta D_X^2}{2}\\left[1+\\sum_{j=2}^t\\left( \\frac{2j}{j+1} + \\frac{2}{j-1}\\sum_{i=1}^{j-1} \\frac{2i}{i+1}\\right)\\right]\n        \\leq& \\sigma\\beta D_X^2\\left[1 + \\sum_{j=2}^t \\left(1 + \\frac{2}{j-1}\\sum_{i=1}^{j-1} 1\\right)\\right] + \\frac{\\beta D_X^2}{2}\\left[1+\\sum_{j=2}^t\\left( \\frac{2j}{j+1} + \\frac{2}{j-1}\\sum_{i=1}^{j-1} \\frac{2i}{i+1}\\right)\\right]\n        \\\\\n        < &  \\sigma\\beta D_X^2(3t-2)+ \\frac{\\beta D_X^2}{2}(6t-5),\\ \\forall t\\ge 2.\n    \\end{align}\n    Noting that $\\sum_{j=2}^t j = (t+2)(t-1)/2$, we have\n%    , after rearranging, we have \n%    \\begin{align}\n%        \\underset{j =2, \\dots, t}{\\min}\\:\\underset{x\\in X}{\\max}&\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x}\\\\\n%        \\leq& \\frac{2}{(t-1)(t+2)} \\left(\\sum_{j=2}^t j\\delta^j + \\frac{2}{j-1}\\sum_{i=1}^{j-1}i \\delta^i\\right) + \\frac{\\beta D_X^2(6t-5)}{(t-1)(t+2)}.\n%    \\end{align}\n%    A change of indexing allows us to conclude that\n\t\\begin{align}\n\t\t\\underset{j=2, \\dots, t}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x} \\le & \\frac{2}{(t+2)(t-1)}\\sum_{j=2}^t j\\cdot\\:\\underset{x\\in X}{\\max}\\ip{\\phi(u^{j-1}),u^{j-1}-x}\n\t\t\\\\\n%\t\t\\le & \\frac{2\\sigma\\beta D_X^2(3t-2)}{(t+2)(t-1)}+ \\frac{\\beta D_X^2(6t-5)}{(t+2)(t-1)}\n%\t\t\\\\\n%\t\t< & \\frac{6\\sigma\\beta D_X^2}{t-1}+ \\frac{6\\beta D_X^2}{t-1}\n%\t\t\\\\\n\t\t< & \\frac{6(\\sigma+1)\\beta D_X^2}{t-1}\n\t\t,\\ \\forall t\\ge 2.\n\t\\end{align}\n\tUsing the above result and observing that \n\t\\begin{align}\n\t\t\\underset{j=1, \\dots, t+1}{\\min}\\:\\ip{\\grad \\phi(u^{j-1}), u^{j-1}-v^{j}} \\le & \n\t\t\\underset{j=1, \\dots, t+1}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x}\n\t\t\\\\\n\t\t\\le & \n\t\t\\underset{j=2, \\dots, t+1}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x},\\ \\forall t\\ge 1\n\t\\end{align}\n\twe conclude \\eqref{eq:termi_rate}. Moreover, from \\eqref{eq:termi_rate} and noting the choice of $\\delta^t$, the termination criterion \\eqref{eq:actual_termi} holds whenever\n    \\begin{align}\n        \\frac{6(\\sigma+1)\\beta D_X^2}{t-1} + \\frac{\\sigma \\beta D_X^2}{t} \\le \\eta.\n    \\end{align}\n    Noting the definition of $T$ \\eqref{def:T}, the above condition clearly holds for all $t\\ge T$.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:LkLB}\nFor any $\\delta >0$ and any $L$ such that\n\\begin{align}\n    L \\geq \\left( \\frac{1-\\nu}{1+\\nu}\\cdot\\frac{1}{\\tau}\\right)^{\\frac{1-\\nu}{1+\\nu}} M_\\nu ^ \\frac{2}{1+\\nu},\n\\end{align} \nwhere $\\nu$ and $M_\\nu$ are the H\\\"older continuity exponent and constant in \\eqref{def:holderCont}, we have\n\\begin{equation}\\label{eq:LkUB}\n    f(y)\\leq f(x) + \\ip{\\grad f(x), y-x} + \\frac{L}{2}\\norm{y-x}^2 + \\frac{\\tau}{2}, \\ \\forall x,y \\in X.\n\\end{equation}\n\\end{lemma}\n\n\\begin{proof}\n    See Lemma 1 of \\cite{nesterov2015universal}.\n\\end{proof}\n\n\\begin{corollary}\\label{coro:LkLB}\n    Any $L_k>0$ chosen by Algorithm \\ref{alg:UCGS} according to \\eqref{def:lk} must also satisfy\n    \\begin{align}\n        L_k \\leq 2\\left(\\frac{1-\\nu}{1+\\nu}\\cdot\\frac{1}{\\varepsilon \\gamma_k}\\right)^{\\frac{1-\\nu}{1+\\nu}} M_\\nu^{\\frac{2}{1+\\nu}}\n    \\end{align}\n\\end{corollary}\n\n\\begin{proof}\n    Suppose that $L_k$ does not satisfy \\eqref{eq:LkUB}.  Then applying Proposition \\ref{prop:flkDiff}  with $\\tau = \\varepsilon \\gamma_k$ implies that $L_k/2$ satisfies \\eqref{def:lk}, contradicting the fact that $L_k$ was chosen at step $k$ following the proposed backtracking linesearch implementation (see the remark on practical implementation after the description of Algorithm \\ref{alg:UCGS}). \n\\end{proof}\n\n\\begin{lemma}\\label{lem:lkgammak}\n    Let $L_k>0$ be chosen by \\eqref{def:lk} of Algorithm \\ref{alg:UCGS} at step $k$, then\n    \\begin{align}\n        L_k\\gamma_k^2 \\leq \\frac{C_\\nu M_\\nu^{\\frac{2}{1+\\nu}}}{k^{\\frac{1+3\\nu}{1+\\nu}}\\varepsilon^{\\frac{1-\\nu}{1+\\nu}}} \n    \\end{align}\n    where \n    \\begin{align}\\label{eq:c}\n        C_\\nu := \\left( \\frac{1+2\\nu}{1+3\\nu}\\right)^{\\frac{1+3\\nu}{1+\\nu}}\\left(\\frac{1-\\nu}{1+\\nu}\\right)^\\frac{1-\\nu}{1+\\nu}2^{\\frac{4+10\\nu}{1+\\nu}} \n    \\end{align}\n    is a constant depending only on $\\nu$. \n\\end{lemma}\n\n\\begin{proof}\n    The case when $k=1$ is immediate from Corollary \\ref{coro:LkLB}. Therefore, throughout the proof we will assume that $k\\ge 2$.\n    Since we set $\\Gamma_k = L_k\\gamma_k^2/k$ in Algorithm \\ref{alg:UCGS}, we can prove this proposition by bounding $\\Gamma_k$. \n    Set $s:= (1+\\nu)/(1+3\\nu)$.  Since $\\nu \\in [0,1]$, we have $s\\in [{1}/{2},1]$.\n    % \\begin{align}\n    %     \\frac{1}{2} = \\frac{1}{3}\\left(1+\\frac{2}{1+3}\\right) \\leq \\frac{1}{3}\\left(1+\\frac{2}{1+3\\nu}\\right) = \\frac{1}{3}\\left(\\frac{3+3\\nu}{1+3\\nu}\\right) = s < 1.\n    % \\end{align}\n    We will study the quantity ${1}/{\\Gamma_k^s} - {1}/{\\Gamma_{k-1}^s}$, which can be rewritten as\n    \\begin{align}\n        \\frac{1}{\\Gamma_k^s} - \\frac{1}{\\Gamma_{k-1}^s} &= \\frac{\\left( \\frac{1}{\\Gamma_k^s} - \\frac{1}{\\Gamma_{k-1}^s}\\right)\\left( \\frac{1}{\\Gamma_k^{1-s}} + \\frac{1}{\\Gamma_{k-1}^{1-s}}\\right)}{ \\frac{1}{\\Gamma_k^{1-s}} + \\frac{1}{\\Gamma_{k-1}^{1-s}}}\\\\\n        &= \\frac{\\frac{1}{\\Gamma_k}-\\frac{1}{\\Gamma_{k-1}} - \\frac{1}{\\Gamma_k}\\left(\\frac{\\Gamma_k}{\\Gamma_{k-1}}\\right)^s + \\frac{1}{\\Gamma_{k-1}}\\left(\\frac{\\Gamma_{k-1}}{\\Gamma_k}\\right)^s}{\\frac{1}{\\Gamma_k^{1-s}} + \\frac{1}{\\Gamma_{k-1}^{1-s}}}.\n    \\end{align}\n    Here, noting from the relation of $\\Gamma_k$ and $\\Gamma_{k-1}$ in \\eqref{eq:GammaRelation} that $\\Gamma_k \\leq \\Gamma_{k-1}$ and recalling that $s \\in [1/2,1]$, we can make two observations. First, we have $\\Gamma_k^{2s-1} \\leq \\Gamma_{k-1}^{2s-1}$, and hence\n    \\begin{align}\n        % \\Gamma_k^{2s-1} &\\leq \\Gamma_{k-1}^{2s-1}\\\\\n        % \\frac{\\Gamma_k^s}{\\Gamma_{k-1}^{s-1}} &\\leq \\frac{\\Gamma_{k-1}^s}{\\Gamma_k^{s-1}}\\\\\n        % \\frac{1}{\\Gamma_{k}}\\left(\\frac{\\Gamma_k}{\\Gamma_{k-1}}\\right)^s &\\leq \\frac{1}{\\Gamma_{k-1}}\\left(\\frac{\\Gamma_{k-1}}{\\Gamma_k}\\right)^s\n    % \\end{align}\n    % and consequently, \n% \n    % \\begin{align}\n        - \\frac{1}{\\Gamma_k}\\left(\\frac{\\Gamma_k}{\\Gamma_{k-1}}\\right)^s + \\frac{1}{\\Gamma_{k-1}}\\left(\\frac{\\Gamma_{k-1}}{\\Gamma_k}\\right)^s \\geq 0.\n    \\end{align} \n    Second, we have $\\Gamma_{k-1}^{1-s}\\le \\Gamma_k^{1-s}$, and hence\n    \\begin{align}\n        \\frac{1}{\\Gamma_k^{1-s}} + \\frac{1}{\\Gamma_{k-1}^{1-s}}\\leq \\frac{2}{\\Gamma_{k-1}^{1-s}}.\n    \\end{align}\n    Combining the above two observations and recalling that $s=(1+\\nu)/(1+3\\nu)$ and the relations concerning $\\Gamma_k$ and $\\Gamma_{k-1}$ in  \\eqref{eq:GammaRelation}, we have that\n    \\begin{align}\n        \\label{eq:tmp2}\n        \\frac{1}{\\Gamma_k^s} - \\frac{1}{\\Gamma_{k-1}^s} \\geq \\frac{\\frac{1}{\\Gamma_k}-\\frac{1}{\\Gamma_{k-1}}}{\\frac{2}{\\Gamma_k^{1-s}}}\\nonumber\n        = \\frac{\\frac{\\gamma_k}{\\Gamma_k}}{\\frac{2}{\\Gamma_k^{1-s}}}\\label{eq:gammaRelation}\n        = \\frac{\\gamma_k}{2}\\Gamma_k^{-\\frac{1+\\nu}{1+3\\nu}}\\nonumber.\n    \\end{align}\n    We can further bound the last expression in the above relation.  Indeed, recalling that $\\Gamma_k = L_k\\gamma_k^2/k$ and applying Corollary \\ref{coro:LkLB}, we have the inequality\n    \\begin{align}\n        \\frac{\\gamma_k^2}{k\\Gamma_k} = \\frac{1}{L_k} \\geq \\frac{1}{2M_\\nu^{\\frac{2}{1+\\nu}}}\\left(\\frac{1+\\nu}{1-\\nu}\\cdot\\varepsilon \\gamma_k\\right)^{\\frac{1-\\nu}{1+\\nu}}.\n    \\end{align}\n    In the above recall that we can use a continuity argument for the $\\nu=1$ case since $(1-\\nu)^{-(1-\\nu)}\\to 1$ as $\\nu\\to 1$. Rearranging terms in the above relation, we have\n    \\begin{align}\n        \\gamma_k\\Gamma_k^{-\\frac{1+\\nu}{1+3\\nu}} \\geq \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu}\\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}k^\\frac{1+\\nu}{1+3\\nu}}{2^\\frac{1+\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}}.\n    \\end{align}\n    Applying the above bound to \\eqref{eq:tmp2}, it follows that\n    \\begin{align}\n        \\frac{1}{\\Gamma_k^s} - \\frac{1}{\\Gamma_{k-1}^s} \\geq \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu} \\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}k^\\frac{1+\\nu}{1+3\\nu}}{2^\\frac{2+4\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}}.\n    \\end{align}\n    Summing the above from $i = 2$ to $k$ and using the fact that\n    \\begin{align}\n        \\sum_{i=2}^k i^{\\frac{1+\\nu}{1+3\\nu}} \\ge \\int_1^k u^{\\frac{1+\\nu}{1+3\\nu}}du = \\frac{1+3\\nu}{2+4\\nu}\\cdot\\left(k^{\\frac{2+4\\nu}{1+3\\nu}}-1\\right) \\ge \\frac{1+3\\nu}{4+8\\nu}k^{\\frac{2+4\\nu}{1+3\\nu}},\\ \\forall k\\ge 2\n    \\end{align}\n    and the definition of $C_\\nu$ in \\eqref{eq:c}, we obtain\n    \\begin{align}\n        \\frac{1}{\\Gamma_k^s} \\geq \\frac{1}{\\Gamma_k^s} - \\frac{1}{\\Gamma_{1}^s} \n        &\n        \\geq \n        % \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu}\\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}}{2^\\frac{2+4\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}}\\frac{1+3\\nu}{4+8\\nu}k^{\\frac{2+4\\nu}{1+3\\nu}}\n        % \\\\\n        % = & \n        \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu}\\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}}{2^\\frac{4+10\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}}\\frac{1+3\\nu}{1+2\\nu}k^{\\frac{2+4\\nu}{1+3\\nu}}\n    \\end{align}\n    Recalling that $s=(1+\\nu)/(1+3\\nu)$ and $\\Gamma_k = L_k\\gamma_k^2/k$, we conclude the proposition immediately from the above result.\n    % \\begin{align}\n    %     \\frac{L_k\\gamma_k^2}{k} \\le \\left(\\frac{1-\\nu}{1+\\nu}\\right)^\\frac{1-\\nu}{1+\\nu}\\frac{2^\\frac{4+10\\nu}{1+\\nu}M_\\nu^\\frac{2}{1+\\nu}}{\\varepsilon^\\frac{1-\\nu}{1+\\nu}}\\left(\\frac{1+2\\nu}{1+3\\nu}\\right)^{\\frac{1+3\\nu}{1+\\nu}}k^{-\\frac{2+4\\nu}{1+\\nu}}\n    % \\end{align}\n    % \n    % Noting that for $k \\geq 2$, $k^{\\frac{2+4\\nu}{1+3\\nu}}-1 \\geq \\frac{1}{2}k^{\\frac{2+4\\nu}{1+3\\nu}}$, we continue with \n    % \\begin{align}\n    %     \\frac{1}{\\Gamma_k^s} &\\geq \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu}\\left(\\frac{1+3\\nu}{2+4\\nu}\\right)\\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}(k^{\\frac{2+4\\nu}{1+3\\nu}}-1)}{2^\\frac{2+4\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}} \\\\\n    %     &\\geq \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu}\\left(\\frac{1+3\\nu}{2+4\\nu}\\right)\\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}k^{\\frac{2+4\\nu}{1+3\\nu}}}{2^\\frac{3+7\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}}\\\\\n    %     &= \\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}k^{\\frac{2+4\\nu}{1+3\\nu}}}{M_\\nu^\\frac{2}{1+3\\nu} C_\\nu^\\frac{1+\\nu}{1+3\\nu}}.\n    % \\end{align}\n    % Solving for $\\Gamma_k$ above, recalling our definition of $s$, and noting the relationship in \\eqref{def:Gammak}, we obtain the desired result. \n\\end{proof}\n\n\\begin{proposition}\\label{prop:flkDiff}\n    Suppose that the parameters in Algorithm \\ref{alg:UCGS} satisfy $\\beta_k \\geq L_k \\gamma_k$ for all $k$. Then for any $x\\in X$, \n    \\begin{align}\n        f(y_k) - \\ell_k(x) \\leq \\frac{\\varepsilon}{2} + \\Gamma_k \\sum_{i=1}^k \\frac{\\gamma_i\\beta_i}{2\\Gamma_i}\\left(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2\\right)^2 + \\Gamma_k \\sum_{i=1}^k \\frac{\\gamma_i\\eta_i}{\\Gamma_i}.\n    \\end{align} \n\\end{proposition}\n\n\\begin{proof}\n    Fix any $x\\in X$. From the definitions of $\\ell_k(x)$ and $y_k$ in \\eqref{def:lk} and \\eqref{def:yk} respectively, we have\n    \\begin{align}\n        \\frac{1}{\\Gamma_k}\\ell_k(x) =& \\sum_{i=1}^k \\frac{1}{\\Gamma_i}\\left( \\gamma_i f(z_i) + \\gamma_i \\ip{\\grad f(z_i),x-x_i} + \\ip{\\grad f(z_i),\\gamma_i(x_i-z_i)}\\right)\\\\\n        =& \\sum_{i=1}^k \\frac{1}{\\Gamma_i}\\left( f(z_i) + \\ip{\\grad f(z_i), y_i-z_i}\\right)+ \\frac{\\gamma_i}{\\Gamma_i} \\ip{\\grad f(z_i),x-x_i}\n        \\\\\n        &\\qquad- \\frac{1-\\gamma_i}{\\Gamma_i}(f(z_i) + \\ip{\\grad f(z_i), y_{i-1}-z_i})\n    \\end{align}\n    We will now bound three terms in the above relation.  First, by convexity of $f$, \n    \\begin{align}\n        -(f(z_i) + \\ip{\\grad f(z_i),y_{i-1}-z_i}) \\ge -f(y_{i-1}).\n    \\end{align}\n    Second, by our choice of $L_k$ in \\eqref{def:lk} and the definitions of $y_k$ and $z_k$ in \\eqref{def:yk} and \\eqref{def:zk} respectively, we have\n    \\begin{align}\n        f(z_i) + \\ip{\\grad f(z_i),y_i-z_i} \\ge & f(y_i) - \\frac{L_i}{2}\\norm{y_i-z_i}^2 - \\frac{\\varepsilon}{2}\\gamma_i \n        \\\\\n        = & f(y_i) - \\frac{L_i\\gamma_i^2}{2}\\norm{x_i-x_{i-1}}^2 - \\frac{\\varepsilon}{2}\\gamma_i.\n    \\end{align}\n    Lastly, using the result \\eqref{eq:innerIterUCGS_OC} in Lemma \\ref{lem:innerIterUCGS} and noting the definition of $\\phi(x)$ in \\eqref{def:phi}, we obtain the following result during the termination of the ACGM procedure in computing $x_i$:\n    \\begin{align}\n         \\ip{\\grad f(z_i), x-x_i} \\geq&\\: \\beta_i\\ip{x_i-x_{i-1},x_i-x} - \\eta_i\\\\\n        =& -\\frac{\\beta_i}{2}\\left(\\norm{x-x_{i-1}}^2 - \\norm{x_i-x_{i-1}}^2 - \\norm{x-x_i}^2\\right) - \\eta_i\\\\\n        \\geq& -\\frac{\\beta_i}{2}\\left(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2\\right) -\\eta_i + \\frac{L_i^2}{2}\\norm{x_i-x_{i-1}}^2.\n    \\end{align}\n    In the last inequality above we use our assumption that $\\beta_k\\geq L_k\\gamma_k$ for all $k$. Based on the above three observations and rearranging terms we obtain that\n    \\begin{align}\n        \\frac{1}{\\Gamma_k}\\ell_k(x) \\geq& \\sum_{i=1}^k \\frac{1}{\\Gamma_i}\\left( f(y_i) - (1-\\gamma_i)f(y_{i-1}) - \\frac{\\gamma_i\\beta_i}{2}(\\norm{x-x_{i-1}}^2-\\norm{x-x_i}^2)\\right)\n        \\\\\n        &\\qquad -\\frac{1}{\\Gamma_i}\\left(\\frac{\\varepsilon}{2}\\gamma_i + \\gamma_i\\eta_i\\right)\n        \\\\\n        = & \n        \\frac{f(y_k)}{\\Gamma_k} -\n        \\sum_{i=1}^k \\frac{\\gamma_i\\beta_i}{2\\Gamma_i}(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2) -\\sum_{i=1}^k \\frac{\\gamma_i\\eta_i}{\\Gamma_i} - \\frac{\\varepsilon}{2\\Gamma_k}.\n    \\end{align}\n    Here in the last equality we use the relations \\eqref{eq:GammaRelation} and \\eqref{eq:gammaSum} and the fact that $\\gamma_1=1$ in its definition \\eqref{def:gammak}. We conclude the result by multiplying by $\\Gamma_k$ and rearranging  terms.\n\\end{proof}\n\n\\begin{proof}\n    From the definition of $s_k$ in Algorithm \\ref{alg:UCGS}, we have that if the termination criterion of UCGS in \\eqref{eq:stopCrit} holds, then $y_k$ is an $\\varepsilon$-solution to problem \\eqref{problem:CP}. Let us evaluate the number of gradient evaluations, or equivalently, the number of outer iterations of UCGS in order to compute an $\\varepsilon$-solution $y_k$.\n    % By the choice of parameters in \\eqref{eq:param}, we have $\\beta_k \\geq L_k\\gamma_k$.  Thus, applying Proposition\n    Applying Proposition \\ref{prop:flkDiff} with our choice of parameters we have\n    \\begin{align}\n        \\label{eq:flkStep1}\n        \\begin{aligned}\n        % & \n        f(y_k) - \\ell_k(x) \n        % \\\\\n        % \\leq& \\:\\frac{\\varepsilon}{2} + \\Gamma_k \\sum_{i=1}^k \\frac{L_i\\gamma_i^2}{2L_i\\gamma_i^2} i\\left(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2\\right) + \\Gamma_k \\sum_{i=1}^k \\frac{L_i\\gamma_i^2}{L_i\\gamma_i^2}D_X^2\n        % \\\\\n        =&\\: \\frac{\\varepsilon}{2} + \\frac{\\Gamma_k}{2}\\sum_{i=1}^k i(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2) + \\Gamma_kkD_X^2,\\ \\forall x\\in X.\n        \\end{aligned}\n    \\end{align}\n    The second term above can be further simplified by noting from the compactness of $X$ and the definition of the diameter $D_X$ in \\eqref{eq:D}. Indeed, we have\n    \\begin{align}\n    \\label{eq:telescope}\n    \\begin{aligned}\n        & \\sum_{i=1}^k i\\left(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2\\right)\n        \\\\\n        =& \\norm{x-x_0}^2 + \\sum_{i=2}^k (i-(i-1))\\norm{x-x_{i-1}}^2 - k\\norm{x-x_k}^2 \n        \\\\\n        \\leq&\\: D_X^2 + \\sum_{i=2}^k D_X^2 =\\: kD_X^2,\\ \\forall x\\in X.\n    \\end{aligned}\n    \\end{align}\n    Thus, we may continue by applying \\eqref{eq:telescope} to \\eqref{eq:flkStep1} with $x = s_k$ to conclude that\n    \\begin{align}\n        f(y_k) - \\ell_k(s_k) \\leq \\frac{\\varepsilon}{2} + \\frac{\\Gamma_k}{2}k D_X^2  + \\Gamma_kkD_X^2 = \\frac{\\varepsilon}{2} + \\frac{3L_k\\gamma_k^2}{2} D_X^2.\n    \\end{align}\n    Here we make use of the description of $\\Gamma_k$ in \\eqref{def:Gammakk} for the last equality. In view of the above result and the value of parameter $\\varepsilon_k$ in \\eqref{eq:param}, $y_k$ satisfies the termination criterion \\eqref{eq:stopCrit} of UCGS and hence becomes an $\\varepsilon$-solution whenever $k$ satisfies the relation \n    % \\begin{align}\n    %     \\frac{\\varepsilon}{2} + \\frac{3L_k\\gamma_k^2}{2} D_X^2 + \\frac{\\sigma L_k\\gamma_kD_X^2}{2} \\le \\varepsilon,\\text{ i.e., }(3+\\sigma)L_k\\gamma_k^2 D_X^2 \\leq \\varepsilon\n    % \\end{align}\n    $(3+\\sigma)L_k\\gamma_k^2 D_X^2 \\leq \\varepsilon$. Applying Lemma \\ref{lem:lkgammak}, it follows that such relation holds whenever\n    \\begin{align}\n        \\frac{(3+\\sigma)C_\\nu D_X^2  M_\\nu^{\\frac{2}{1+\\nu}}}{k^{\\frac{1+3\\nu}{1+\\nu}}\\varepsilon^{\\frac{1-\\nu}{1+\\nu}}} \\leq \\varepsilon,\\text{ i.e., }k\\ge C_\\nu^\\frac{1+\\nu}{1+3\\nu}\\left(\\frac{(3+\\sigma)^{\\frac{1+\\nu}{2}}M_\\nu D_X^{1+\\nu}}{\\varepsilon}\\right)^\\frac{2}{1+3\\nu}.\n    \\end{align}\n    Noting that $C_\\nu$ defined in \\eqref{eq:c} is a constant that depends only on $\\nu\\in[0,1]$ and observing that $C_\\nu^\\frac{1+\\nu}{1+3\\nu} \\leq 16$ for all $\\nu \\in [0,1]$, we conclude that whenever $k \\geq N_{\\text{grad}}$, $y_k$ is an $\\varepsilon$-solution. Therefore, UCGS requires at most $N_{\\text{grad}}$ gradient evaluations of $\\grad f$ to compute an $\\varepsilon$-solution.\n    % \\begin{align}\n    %     k \\geq 24\\left(\\frac{D_X^{1+\\nu}M_\\nu^\\frac{2}{1+\\nu}}{\\varepsilon}\\right)^\\frac{2}{1+3\\nu} = N_{grad}\n    % \\end{align}\n    \n    It suffices to compute the number of linear objective optimizations that UCGS requires for computing an $\\varepsilon$-solution. This is equivalent to estimating the total number of inner iterations that UCGS requires.\n    % Note first that whenever the termination criterion \\eqref{eq:actual_termi} is satisfied, then we have\n    % \\begin{align}\n    %     \\max_{x\\in X}\\ip{\\grad \\phi(u^{t-1}), u^{t-1}-x} &= \\max_{x\\in X}\\ip{\\grad \\phi(u^{t-1}), u^{t-1}-v^t} + \\ip{\\grad \\phi(u^{t-1}), v^t - x}\n    %     \\\\\n    %     &\\le  (\\eta - \\delta^t) + \\delta^t = \\eta.\n    % \\end{align}\n    % Here the inequality is from the termination criterion \\eqref{eq:actual_termi} and the fact that $v^t$ is an approximate solution that satisfies \\eqref{eq:cndgsubproblem}. \n    Let us estimate the maximal number of inner iterations required before the termination criterion \\eqref{eq:actual_termi} is satisfied.\n    Recall from the remark after \\eqref{def:alpha} that $\\alpha^t$ is the best linesearch parameter and hence satisfies assumption \\eqref{eq:alpha_req_2t1} in Proposition \\ref{prop:innerIterUCGS}. Applying Proposition \\ref{prop:innerIterUCGS} and noting the definition of approximate solution $v^t$ in \\eqref{eq:cndgsubproblem}, we have that\n    % \\begin{align}\n    %     \\underset{j=1, \\dots, t}{\\min}\\:\\ip{\\grad \\phi(u^{j-1}), u^{j-1}-v^{j}} \n    %     \\le & \\underset{j=1, \\dots, t}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x}\n    %     \\\\\n    %     = & \\underset{j=0, \\dots, t-1}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j}), u^{j}-x}\n    %     \\\\\n    %     \\le & \\frac{6(\\sigma+1)\\beta D_X^2}{t-1}\n    %     ,\\ \\forall t\\ge 2.\n    % \\end{align}\n    % From the above and noting the choice of $\\delta^t$, the termination criterion \\eqref{eq:actual_termi} holds whenever\n    % \\begin{align}\n    %     \\frac{6(\\sigma+1)\\beta D_X^2}{t-1} + \\frac{\\sigma \\beta D_X^2}{t} \\le \\eta.\n    % \\end{align}\n    % The above condition holds whenever\n    % \\begin{align}\n    %     t \\ge 1 + \\frac{(7\\sigma + 6)\\beta D_X^2}{\\eta}.\n    % \\end{align}\n    the maximal number of linear objective optimizations performed at the $k$-th call to the ACGM procedure is at most \n    \\begin{equation}\n        T_k := 1 + \\left\\lceil \\frac{(7\\sigma+6)\\beta_k D_X^2}{\\eta_k}\\right\\rceil = 1 + \\lceil (7\\sigma+6)k \\rceil.\n    \\end{equation}\n    Adding one linear objective optimization problem in \\eqref{def:lkproblem} in each other iteration concerning the termination criterion of UCGS, we conclude that the total number of linear objective optimizations for UCGS to compute an $\\varepsilon$-solution is bounded above by\n    \\begin{align}\n        \\sum_{k=1}^{N_{\\text{grad}}} (T_k + 1) \\le  \\sum_{k=1}^{N_{\\text{grad}}} 3 + (7\\sigma+6)k = \\left(\\frac{7}{2}\\sigma + 3\\right)N_{\\text{grad}}^2 + \\left(\\frac{7}{2}\\sigma + 6\\right)N_{\\text{grad}}.\n    \\end{align} \n\\end{proof}","units":[{"start":37019,"end":37565,"start_line":366,"end_line":382,"raw_tex":"\\begin{lemma}\n\t\\label{lem:tele}\n\tSuppose that $\\{a_k\\},\\{b_k\\}\\subset\\R$ and $\\{\\gamma_k\\}\\subset [0,1]$ are sequences that satisfy $\\gamma_1=1$ and\n\t\\begin{align}\n\t\t\\label{eq:tele_assum}\n\t\ta_k \\le (1-\\gamma_k)a_{k-1} + \\gamma_k b_k,\\ \\forall k\\ge 1.\n\t\\end{align}\n\tThen we have\n\t\\begin{align}\n\t\t\\label{eq:tele_res}\n\t\ta_k\\le \\Gamma_k\\sum_{i=1}^{k}\\frac{\\gamma_i}{\\Gamma_i}b_i,\\ \\forall k\\ge 1,\\text{ where }\\Gamma_k:=\\begin{cases}\n\t\t\t1 & \\text{ when }k=1\n\t\t\t\\\\\n\t\t\t\\Gamma_{k-1}(1-\\gamma_k) & \\text{ when }k>1.\n\t\t\\end{cases}\n\t\\end{align}\n\\end{lemma}","sha256_utf8":"d290f0c73dd3ff2c60e13e8d86bcae8ad94d9cb9c4c0338fd4b3d41d365b12d1"},{"start":37566,"end":37791,"start_line":383,"end_line":385,"raw_tex":"\\begin{proof}\n\tDividing both sides of \\eqref{eq:tele_assum} by $\\Gamma_k$ we obtain a series of inequalities with telescoping terms concerning sequence $\\{a_k/\\Gamma_k\\}$. Summing up we obtain \\eqref{eq:tele_res}.\n\\end{proof}","sha256_utf8":"a995989e283d3d147ef119e543670b9c6d67df4e5fff852b4efc1b4a66964867"},{"start":65053,"end":66513,"start_line":679,"end_line":687,"raw_tex":"We proposed to search for such $L_k$ through a backtracking linesearch strategy. In particular, we initialize with any $L_0 \\in \\R$ and choose $L_1 = 2^i L_0$ where $i$ is the smallest integer such that \\eqref{def:lk} is satisfied. At the start of the $k$-th outer iteration where $k>1$, we set $L_k = L_{k-1}/2$ and assess the validity of $L_k$. If it does not satisfy $\\eqref{def:lk}$, we keep backtracking and replacing $L_k$ to $2L_k$ until \\eqref{def:lk} is satisfies. Through this backtracking linesearch strategy, we ensure that our choice of $L_k$ is adaptive and that performance is independent of the choice of $L_0$. Previous literature \\cite{nazari2020backtracking} on backtracking linesearch strategy of CGS require monotonicity of $L_k$ and may suffer from a poorly chosen $L_0$. Second, our termination criterion is based on \\eqref{eq:stopCrit}. We can observe immediately that if the parameter $\\varepsilon_k\\equiv 0$, i.e., $s_k$ is the exact solution to problem \\eqref{def:lkproblem}, then when \\eqref{eq:stopCrit} is satisfied, $y_k$ will be $\\varepsilon$-approximation solution to problem \\eqref{problem:CP}.  To see this, note from \\eqref{eq:GammaRelation} that\n    \\begin{equation}\\label{eq:gammaSum}\n        \\Gamma_k \\sum_{i=1}^k \\frac{\\gamma_i}{\\Gamma_i} = 1\n    \\end{equation}\n    and consequently\n    % \\begin{equation}\\label{eq:lkBound}\n    $$\n        f(y_k) - f^* \\le f(y_k) - \\min_{x\\in X}\\ell_k(x) =  f(y_k) - \\ell_k(s_k).\n    $$\n","sha256_utf8":"d10cfb3592cb78573b68bcb6add6e04a10a1d54d99206f1b3362ca0df6d46297"},{"start":68146,"end":69559,"start_line":714,"end_line":740,"raw_tex":"\\begin{lemma}\\label{lem:innerIterUCGS}\n    Suppose that $\\{\\lambda^t\\}\\in [0,1]$ is any predetermined sequence satisfying $\\lambda^1= 1$. In the ACGM procedure, if $\\alpha^t$ is chosen such that \n    \\begin{align}\n        \\label{eq:alpha_req}\n        \\phi(u^t)\\le \\phi((1-\\lambda^t)u^{t-1} + \\lambda^tv^t),\\ \\forall t\\ge 1,\n    \\end{align}\n    then we have\n    \\begin{align}\n        \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j}\\: \\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}),u^{j-1} - x}\n        % \\\\\n%        \\leq & \\delta^1 + \\frac{\\beta D_X^2}{2} + \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j}\\left(\\delta^j + \\frac{\\beta D_X^2\\lambda^j}{2} +  \\Lambda^{j-1}\\sum_{i=1}^{j-1}\\frac{\\lambda^i}{\\Lambda^i}\\left(\\delta^i + \\frac{\\beta D_X^2\\lambda^i}{2}\\right)\\right)\n        \\leq &\n        \\left[\\delta^1 + \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j}\\left(\\delta^j + \\Lambda^{j-1}\\sum_{i=1}^{j-1}\\frac{\\lambda^i}{\\Lambda^i}\\delta^i\\right)\\right] \n        \\\\\n        & +\n        \\frac{\\beta D_X^2}{2}\\left[1+\\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j} \\left( \\lambda^j + \\Lambda^{j-1}\\sum_{i=1}^{j-1}\\frac{(\\lambda^i)^2}{\\Lambda^i}\\right)\\right]\n    \\end{align}\n    for all $t \\geq 2$, where\n    \\begin{align}\n        \\label{def:Lambda}\n        \\Lambda^t:=\\begin{cases}\n            1 & \\text{ when }t=1\n            \\\\\n            \\Lambda^{t-1}(1-\\lambda^t) & \\text{ when } t>1.\n        \\end{cases}\n    \\end{align}\n\\end{lemma}","sha256_utf8":"7abc49e2e7b3dbb1c4335b6cdd0e44b3e69a6ab007607a56c5007ccf54a994d8"},{"start":69561,"end":73402,"start_line":742,"end_line":795,"raw_tex":"\\begin{proof}\n    Observing that the function $\\phi(x)$ in \\eqref{def:phi} is a strongly convex function with Lipschitz continuous (with constant $\\beta$) gradient, using the assumption \\eqref{eq:alpha_req}, and noting the definition of approximate solution $v^t$ in \\eqref{eq:cndgsubproblem}, we have\n    \\begin{align}\n    \t\\label{eq:phiConvex}\n    \t\\begin{aligned}\n        &\\phi(u^t) - (1-\\lambda^t)\\phi(u^{t-1}) - \\lambda^t(\\phi(u^{t-1}) + \\ip{\\grad \\phi(u^{t-1}),x-u^{t-1}})\n        \\\\\n        \\leq& \\phi((1-\\lambda^t)u^{t-1} + \\lambda^tv^t) - \\phi(u^{t-1}) - \\lambda^t\\ip{\\grad\\phi(u^{t-1}),x-u^{t-1}}\n        \\\\\n        \\le & \\lambda^t\\ip{\\grad \\phi(u^{t-1}), v^t - u^{t-1}} + \\frac{\\beta(\\lambda^t)^2}{2}\\norm{v^t-u^{t-1}}^2 - \\lambda^t\\ip{\\grad \\phi(u^{t-1}),x-u^{t-1}}\n        \\\\\n        =& \\lambda^t\\ip{\\grad\\phi(u^{t-1}),v^t-x} + \\frac{\\beta(\\lambda^t)^2}{2}\\norm{v^t-u^{t-1}}^2\\\\\n        \\leq& \\lambda^t\\delta^t +\\frac{\\beta D_X^2(\\lambda^t)^2}{2},\\ \\forall x\\in X, t\\ge 1.\n\t    \\end{aligned}\n    \\end{align}\n\tDefining $x^*:=\\operatorname{argmin}_{x\\in X}\\phi(x)$, from the above relation we have for any $t\\ge 2$ that\n\t\\begin{align}\n\t\t\\label{eq:tmp1}\n\t\t\\begin{aligned}\n\t\t& \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j} \\underset{x\\in X}{\\max}\\:\\ip{\\grad\\phi(u^{j-1}), u^{j-1}-x}\n\t\t\\\\\n\t\t\\leq& \\sum_{j=2}^t \\frac{1}{\\Lambda^j}[\\phi(u^{j-1}) - \\phi(x^*)] - \\frac{1}{\\Lambda^j}[\\phi(u^j) - \\phi(x^*)] + \\frac{\\lambda^j}{\\Lambda^j}\\delta^j + \\frac{\\beta D_X^2 (\\lambda^j)^2}{2\\Lambda^j}\n\t\t\\\\\n\t\t= & [\\phi(u^1) - \\phi(x^*)] - \\frac{1}{\\Lambda^t}[\\phi(u^t) - \\phi(x^*)] \n\t\t\\\\\n\t\t& + \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j}[\\phi(u^{j-1}) - \\phi(x^*)] + \\frac{\\lambda^j}{\\Lambda^j}\\delta^j + \\frac{\\beta D_X^2(\\lambda^j)^2}{2\\Lambda^j}\n\t\t\\\\\n\t\t\\le & [\\phi(u^1) - \\phi(x^*)] + \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j}[\\phi(u^{j-1}) - \\phi(x^*)] + \\frac{\\lambda^j}{\\Lambda^j}\\delta^j + \\frac{\\beta D_X^2(\\lambda^j)^2}{2\\Lambda^j}.\n\t\t\\end{aligned}\n\t\\end{align}\n\tHere in the equality we use the following observations from the definition of $\\Lambda^t$ in \\eqref{def:Lambda}: $1/\\Lambda^1 = 1$ and $1/\\Lambda^j = 1/\\Lambda^{j-1} + \\lambda^j/\\Lambda^j$ for all $j\\ge 2$, \n%\t\\begin{align}\n%\t\t\\frac{1}{\\Lambda^j} = \\frac{1}{\\Lambda^{j-1}} + \\frac{\\lambda^j}{\\Lambda^{j}},\\ \\forall j\\ge 2,\n%\t\\end{align}\n\n    To finish the proof it suffices to bound $\\phi(u^{j-1})-\\phi(x^*)$ for any $j\\ge 2$. Observing that $\\phi(x)$ in \\eqref{def:phi} is strongly convex and quadratic with\n    \\begin{equation}\n        \\frac{\\beta}{2}\\norm{x-u^{t-1}}^2 = \\phi(x) - (\\phi(u^{t-1})+\\ip{\\grad \\phi(u^{t-1}), x-u^{t-1}}),\\ \\forall x\\in X, t\\ge 1,\n    \\end{equation}\n    we have from \\eqref{eq:phiConvex} (with $x = x^*$) that\n    \\begin{equation}\\label{eq:telescopingBound}\n        [\\phi(u^t)-\\phi(x^*)] - (1-\\lambda^t)[\\phi(u^{t-1})-\\phi(x^*)] \\leq \\lambda^t\\delta^t + \\frac{\\beta D_X^2(\\lambda^t)^2}{2} - \\frac{\\beta\\lambda^t}{2}\\norm{x^*-u^{t-1}}^2.\n    \\end{equation}\n    Applying Lemma \\ref{lem:tele} to the above recurrence relation and ignoring negative terms at the right hand side, we have\n    \\begin{align}\\label{eq:simplePhiBound}\n        \\phi(u^t) - \\phi(x^*) \\leq \\Lambda^t\\sum_{i=1}^t \\frac{\\lambda^i}{\\Lambda^i}\\delta^i + \\frac{\\beta D_X^2(\\lambda^i)^2}{2\\Lambda^i},\\ \\forall t\\ge 1.\n    \\end{align}\n\tWe conclude the lemma immediately by applying the above bound to \\eqref{eq:tmp1} and rearranging terms.\n%\t\\begin{align}\n%\t\t\t& \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j} \\underset{x\\in X}{\\max}\\:\\ip{\\grad\\phi(u^{j-1}), u^{j-1}-x}\n%\t\t\t\\\\\n%\t\t\t\\le & \\delta^1 + \\frac{\\beta D_X^2}{2} + \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j}\\Lambda^{j-1}\\left(\\sum_{i=1}^{j-1} \\frac{\\lambda^i}{\\Lambda^i}\\delta^i + \\frac{\\beta D_X^2(\\lambda^i)^2}{2\\Lambda^i}\\right) + \\frac{\\lambda^j}{\\Lambda^j}\\delta^j + \\frac{\\beta D_X^2(\\lambda^j)^2}{2\\Lambda^j}.\n%\t\\end{align}\n\\end{proof}","sha256_utf8":"79566cc6c60fb0a9ed1cdbcf98b3762c61d59e4f7480f9e3b368ed1d03b4718b"},{"start":73599,"end":74685,"start_line":799,"end_line":825,"raw_tex":"\\begin{proposition}\\label{prop:innerIterUCGS}\n\tIn the ACGM procedure, at termination we have\n\t\\begin{align}\n\t    \\label{eq:innerIterUCGS_OC}\n\t    \\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^+), u^+-x} \\le \\eta.\n\t\\end{align}\n\tMoreover, if $\\delta^t = \\sigma\\beta D_X^2/t$ for certain $\\sigma\\ge 0$ and $\\alpha^t$ is chosen such that \n\t\\begin{align}\n\t\t\\label{eq:alpha_req_2t1}\n\t\t\\phi(u^t) \\le \\phi\\left(\\frac{t-1}{t+1}u^{t-1} + \\frac{2}{t+1}v^t\\right),\n\t\\end{align}\n\tthen we have for any $t\\ge 1$ that\n    \\begin{align}\n        \\label{eq:termi_rate}\n        \\begin{aligned}\n        \\underset{j=1, \\dots, t+1}{\\min}\\:\\ip{\\grad \\phi(u^{j-1}), u^{j-1}-v^{j}}\\le & \\underset{j=1, \\dots, t+1}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x}\n        \\\\\n        \\le & \\frac{6(\\sigma+1)\\beta D_X^2}{t}.\n        \\end{aligned}\n    \\end{align}\n    Specially, it takes at most\n    \\begin{align}\n        \\label{def:T}\n        T := 1 + \\left\\lceil \\frac{(7\\sigma+6)\\beta D_X^2}{\\eta}\\right\\rceil \n    \\end{align}\n    iterations for the ACGM procedure to terminate.\n\\end{proposition}","sha256_utf8":"d985ae7f9ee055db54afb672269e0305ae5846deb6b05ca6f17bef3179010a07"},{"start":74687,"end":77877,"start_line":827,"end_line":873,"raw_tex":"\\begin{proof}\n    From the definition of the approximate solution $v^t$ in \\eqref{eq:cndgsubproblem}, if the termination criterion in \\eqref{eq:actual_termi} of the ACGM procedure is satisfied, then the output $u^+=u^{t-1}$ satisfies\n    \\begin{align}\n        \\max_{x\\in X}\\ip{\\grad \\phi(u^{t-1}), u^{t-1}-x} &= \\max_{x\\in X}\\ip{\\grad \\phi(u^{t-1}), u^{t-1}-v^t} + \\ip{\\grad \\phi(u^{t-1}), v^t - x}\n        \\\\\n        &\\le  (\\eta - \\delta^t) + \\delta^t = \\eta.\n    \\end{align}\n    Therefore \\eqref{eq:innerIterUCGS_OC} holds. To conclude the proposition it suffices to estimate the rate of convergence of $\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{t-1}), u^{t-1}-x}$. To analyze the rate, let us choose $\\lambda^t = 2/(t+1)$ and apply Lemma \\ref{lem:innerIterUCGS}. Then $\\Lambda^t = 2/(t(t+1))$ and\n    \\begin{align}\n        & \\sum_{j=2}^t j\\cdot\\:\\underset{x\\in X}{\\max}\\ip{\\phi(u^{j-1}),u^{j-1}-x}\n        \\\\\n%        \\leq& \\left[\\delta^1 + \\sum_{j=2}^t \\left(j\\delta^j + \\frac{2}{j-1}\\sum_{i=1}^{j-1} i \\delta^i\\right)\\right] + \\frac{\\beta D_X^2}{2}\\left[1+\\sum_{j=2}^t\\left( \\frac{2j}{j+1} + \\frac{2}{j-1}\\sum_{i=1}^{j-1} \\frac{2i}{i+1}\\right)\\right]\n        \\leq& \\sigma\\beta D_X^2\\left[1 + \\sum_{j=2}^t \\left(1 + \\frac{2}{j-1}\\sum_{i=1}^{j-1} 1\\right)\\right] + \\frac{\\beta D_X^2}{2}\\left[1+\\sum_{j=2}^t\\left( \\frac{2j}{j+1} + \\frac{2}{j-1}\\sum_{i=1}^{j-1} \\frac{2i}{i+1}\\right)\\right]\n        \\\\\n        < &  \\sigma\\beta D_X^2(3t-2)+ \\frac{\\beta D_X^2}{2}(6t-5),\\ \\forall t\\ge 2.\n    \\end{align}\n    Noting that $\\sum_{j=2}^t j = (t+2)(t-1)/2$, we have\n%    , after rearranging, we have \n%    \\begin{align}\n%        \\underset{j =2, \\dots, t}{\\min}\\:\\underset{x\\in X}{\\max}&\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x}\\\\\n%        \\leq& \\frac{2}{(t-1)(t+2)} \\left(\\sum_{j=2}^t j\\delta^j + \\frac{2}{j-1}\\sum_{i=1}^{j-1}i \\delta^i\\right) + \\frac{\\beta D_X^2(6t-5)}{(t-1)(t+2)}.\n%    \\end{align}\n%    A change of indexing allows us to conclude that\n\t\\begin{align}\n\t\t\\underset{j=2, \\dots, t}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x} \\le & \\frac{2}{(t+2)(t-1)}\\sum_{j=2}^t j\\cdot\\:\\underset{x\\in X}{\\max}\\ip{\\phi(u^{j-1}),u^{j-1}-x}\n\t\t\\\\\n%\t\t\\le & \\frac{2\\sigma\\beta D_X^2(3t-2)}{(t+2)(t-1)}+ \\frac{\\beta D_X^2(6t-5)}{(t+2)(t-1)}\n%\t\t\\\\\n%\t\t< & \\frac{6\\sigma\\beta D_X^2}{t-1}+ \\frac{6\\beta D_X^2}{t-1}\n%\t\t\\\\\n\t\t< & \\frac{6(\\sigma+1)\\beta D_X^2}{t-1}\n\t\t,\\ \\forall t\\ge 2.\n\t\\end{align}\n\tUsing the above result and observing that \n\t\\begin{align}\n\t\t\\underset{j=1, \\dots, t+1}{\\min}\\:\\ip{\\grad \\phi(u^{j-1}), u^{j-1}-v^{j}} \\le & \n\t\t\\underset{j=1, \\dots, t+1}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x}\n\t\t\\\\\n\t\t\\le & \n\t\t\\underset{j=2, \\dots, t+1}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x},\\ \\forall t\\ge 1\n\t\\end{align}\n\twe conclude \\eqref{eq:termi_rate}. Moreover, from \\eqref{eq:termi_rate} and noting the choice of $\\delta^t$, the termination criterion \\eqref{eq:actual_termi} holds whenever\n    \\begin{align}\n        \\frac{6(\\sigma+1)\\beta D_X^2}{t-1} + \\frac{\\sigma \\beta D_X^2}{t} \\le \\eta.\n    \\end{align}\n    Noting the definition of $T$ \\eqref{def:T}, the above condition clearly holds for all $t\\ge T$.\n\\end{proof}","sha256_utf8":"abbc01967f5fb3209b72d5996481cb0c74f06da423e33243eaed8f362935b264"},{"start":79896,"end":80379,"start_line":887,"end_line":896,"raw_tex":"\\begin{lemma}\\label{lem:LkLB}\nFor any $\\delta >0$ and any $L$ such that\n\\begin{align}\n    L \\geq \\left( \\frac{1-\\nu}{1+\\nu}\\cdot\\frac{1}{\\tau}\\right)^{\\frac{1-\\nu}{1+\\nu}} M_\\nu ^ \\frac{2}{1+\\nu},\n\\end{align} \nwhere $\\nu$ and $M_\\nu$ are the H\\\"older continuity exponent and constant in \\eqref{def:holderCont}, we have\n\\begin{equation}\\label{eq:LkUB}\n    f(y)\\leq f(x) + \\ip{\\grad f(x), y-x} + \\frac{L}{2}\\norm{y-x}^2 + \\frac{\\tau}{2}, \\ \\forall x,y \\in X.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"1934c84b3ab21be3fc01a47eb23bdedafca16001dce93e73a4f64cca56376881"},{"start":80380,"end":80454,"start_line":897,"end_line":899,"raw_tex":"\\begin{proof}\n    See Lemma 1 of \\cite{nesterov2015universal}.\n\\end{proof}","sha256_utf8":"cc690004858fc9b612aa4202bac60d7242bf3ce591eca42ef791bcf5c0fea474"},{"start":80721,"end":81034,"start_line":901,"end_line":906,"raw_tex":"\\begin{corollary}\\label{coro:LkLB}\n    Any $L_k>0$ chosen by Algorithm \\ref{alg:UCGS} according to \\eqref{def:lk} must also satisfy\n    \\begin{align}\n        L_k \\leq 2\\left(\\frac{1-\\nu}{1+\\nu}\\cdot\\frac{1}{\\varepsilon \\gamma_k}\\right)^{\\frac{1-\\nu}{1+\\nu}} M_\\nu^{\\frac{2}{1+\\nu}}\n    \\end{align}\n\\end{corollary}","sha256_utf8":"952e9aebe7a4a4a85f00c8a7c721e0b5280c9a4682b478468d996c37c31f08d9"},{"start":81035,"end":81462,"start_line":907,"end_line":909,"raw_tex":"\\begin{proof}\n    Suppose that $L_k$ does not satisfy \\eqref{eq:LkUB}.  Then applying Proposition \\ref{prop:flkDiff}  with $\\tau = \\varepsilon \\gamma_k$ implies that $L_k/2$ satisfies \\eqref{def:lk}, contradicting the fact that $L_k$ was chosen at step $k$ following the proposed backtracking linesearch implementation (see the remark on practical implementation after the description of Algorithm \\ref{alg:UCGS}). \n\\end{proof}","sha256_utf8":"101a15558559ae502cb11f322da2fb398570cb9a423598d24cbe0bb3b3a80a35"},{"start":82513,"end":83061,"start_line":917,"end_line":927,"raw_tex":"\\begin{lemma}\\label{lem:lkgammak}\n    Let $L_k>0$ be chosen by \\eqref{def:lk} of Algorithm \\ref{alg:UCGS} at step $k$, then\n    \\begin{align}\n        L_k\\gamma_k^2 \\leq \\frac{C_\\nu M_\\nu^{\\frac{2}{1+\\nu}}}{k^{\\frac{1+3\\nu}{1+\\nu}}\\varepsilon^{\\frac{1-\\nu}{1+\\nu}}} \n    \\end{align}\n    where \n    \\begin{align}\\label{eq:c}\n        C_\\nu := \\left( \\frac{1+2\\nu}{1+3\\nu}\\right)^{\\frac{1+3\\nu}{1+\\nu}}\\left(\\frac{1-\\nu}{1+\\nu}\\right)^\\frac{1-\\nu}{1+\\nu}2^{\\frac{4+10\\nu}{1+\\nu}} \n    \\end{align}\n    is a constant depending only on $\\nu$. \n\\end{lemma}","sha256_utf8":"e3d102a54bbc485f1b92083dd44e5514905e1c4138f146c2497da805145c8c8b"},{"start":83063,"end":89276,"start_line":929,"end_line":1001,"raw_tex":"\\begin{proof}\n    The case when $k=1$ is immediate from Corollary \\ref{coro:LkLB}. Therefore, throughout the proof we will assume that $k\\ge 2$.\n    Since we set $\\Gamma_k = L_k\\gamma_k^2/k$ in Algorithm \\ref{alg:UCGS}, we can prove this proposition by bounding $\\Gamma_k$. \n    Set $s:= (1+\\nu)/(1+3\\nu)$.  Since $\\nu \\in [0,1]$, we have $s\\in [{1}/{2},1]$.\n    % \\begin{align}\n    %     \\frac{1}{2} = \\frac{1}{3}\\left(1+\\frac{2}{1+3}\\right) \\leq \\frac{1}{3}\\left(1+\\frac{2}{1+3\\nu}\\right) = \\frac{1}{3}\\left(\\frac{3+3\\nu}{1+3\\nu}\\right) = s < 1.\n    % \\end{align}\n    We will study the quantity ${1}/{\\Gamma_k^s} - {1}/{\\Gamma_{k-1}^s}$, which can be rewritten as\n    \\begin{align}\n        \\frac{1}{\\Gamma_k^s} - \\frac{1}{\\Gamma_{k-1}^s} &= \\frac{\\left( \\frac{1}{\\Gamma_k^s} - \\frac{1}{\\Gamma_{k-1}^s}\\right)\\left( \\frac{1}{\\Gamma_k^{1-s}} + \\frac{1}{\\Gamma_{k-1}^{1-s}}\\right)}{ \\frac{1}{\\Gamma_k^{1-s}} + \\frac{1}{\\Gamma_{k-1}^{1-s}}}\\\\\n        &= \\frac{\\frac{1}{\\Gamma_k}-\\frac{1}{\\Gamma_{k-1}} - \\frac{1}{\\Gamma_k}\\left(\\frac{\\Gamma_k}{\\Gamma_{k-1}}\\right)^s + \\frac{1}{\\Gamma_{k-1}}\\left(\\frac{\\Gamma_{k-1}}{\\Gamma_k}\\right)^s}{\\frac{1}{\\Gamma_k^{1-s}} + \\frac{1}{\\Gamma_{k-1}^{1-s}}}.\n    \\end{align}\n    Here, noting from the relation of $\\Gamma_k$ and $\\Gamma_{k-1}$ in \\eqref{eq:GammaRelation} that $\\Gamma_k \\leq \\Gamma_{k-1}$ and recalling that $s \\in [1/2,1]$, we can make two observations. First, we have $\\Gamma_k^{2s-1} \\leq \\Gamma_{k-1}^{2s-1}$, and hence\n    \\begin{align}\n        % \\Gamma_k^{2s-1} &\\leq \\Gamma_{k-1}^{2s-1}\\\\\n        % \\frac{\\Gamma_k^s}{\\Gamma_{k-1}^{s-1}} &\\leq \\frac{\\Gamma_{k-1}^s}{\\Gamma_k^{s-1}}\\\\\n        % \\frac{1}{\\Gamma_{k}}\\left(\\frac{\\Gamma_k}{\\Gamma_{k-1}}\\right)^s &\\leq \\frac{1}{\\Gamma_{k-1}}\\left(\\frac{\\Gamma_{k-1}}{\\Gamma_k}\\right)^s\n    % \\end{align}\n    % and consequently, \n% \n    % \\begin{align}\n        - \\frac{1}{\\Gamma_k}\\left(\\frac{\\Gamma_k}{\\Gamma_{k-1}}\\right)^s + \\frac{1}{\\Gamma_{k-1}}\\left(\\frac{\\Gamma_{k-1}}{\\Gamma_k}\\right)^s \\geq 0.\n    \\end{align} \n    Second, we have $\\Gamma_{k-1}^{1-s}\\le \\Gamma_k^{1-s}$, and hence\n    \\begin{align}\n        \\frac{1}{\\Gamma_k^{1-s}} + \\frac{1}{\\Gamma_{k-1}^{1-s}}\\leq \\frac{2}{\\Gamma_{k-1}^{1-s}}.\n    \\end{align}\n    Combining the above two observations and recalling that $s=(1+\\nu)/(1+3\\nu)$ and the relations concerning $\\Gamma_k$ and $\\Gamma_{k-1}$ in  \\eqref{eq:GammaRelation}, we have that\n    \\begin{align}\n        \\label{eq:tmp2}\n        \\frac{1}{\\Gamma_k^s} - \\frac{1}{\\Gamma_{k-1}^s} \\geq \\frac{\\frac{1}{\\Gamma_k}-\\frac{1}{\\Gamma_{k-1}}}{\\frac{2}{\\Gamma_k^{1-s}}}\\nonumber\n        = \\frac{\\frac{\\gamma_k}{\\Gamma_k}}{\\frac{2}{\\Gamma_k^{1-s}}}\\label{eq:gammaRelation}\n        = \\frac{\\gamma_k}{2}\\Gamma_k^{-\\frac{1+\\nu}{1+3\\nu}}\\nonumber.\n    \\end{align}\n    We can further bound the last expression in the above relation.  Indeed, recalling that $\\Gamma_k = L_k\\gamma_k^2/k$ and applying Corollary \\ref{coro:LkLB}, we have the inequality\n    \\begin{align}\n        \\frac{\\gamma_k^2}{k\\Gamma_k} = \\frac{1}{L_k} \\geq \\frac{1}{2M_\\nu^{\\frac{2}{1+\\nu}}}\\left(\\frac{1+\\nu}{1-\\nu}\\cdot\\varepsilon \\gamma_k\\right)^{\\frac{1-\\nu}{1+\\nu}}.\n    \\end{align}\n    In the above recall that we can use a continuity argument for the $\\nu=1$ case since $(1-\\nu)^{-(1-\\nu)}\\to 1$ as $\\nu\\to 1$. Rearranging terms in the above relation, we have\n    \\begin{align}\n        \\gamma_k\\Gamma_k^{-\\frac{1+\\nu}{1+3\\nu}} \\geq \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu}\\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}k^\\frac{1+\\nu}{1+3\\nu}}{2^\\frac{1+\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}}.\n    \\end{align}\n    Applying the above bound to \\eqref{eq:tmp2}, it follows that\n    \\begin{align}\n        \\frac{1}{\\Gamma_k^s} - \\frac{1}{\\Gamma_{k-1}^s} \\geq \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu} \\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}k^\\frac{1+\\nu}{1+3\\nu}}{2^\\frac{2+4\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}}.\n    \\end{align}\n    Summing the above from $i = 2$ to $k$ and using the fact that\n    \\begin{align}\n        \\sum_{i=2}^k i^{\\frac{1+\\nu}{1+3\\nu}} \\ge \\int_1^k u^{\\frac{1+\\nu}{1+3\\nu}}du = \\frac{1+3\\nu}{2+4\\nu}\\cdot\\left(k^{\\frac{2+4\\nu}{1+3\\nu}}-1\\right) \\ge \\frac{1+3\\nu}{4+8\\nu}k^{\\frac{2+4\\nu}{1+3\\nu}},\\ \\forall k\\ge 2\n    \\end{align}\n    and the definition of $C_\\nu$ in \\eqref{eq:c}, we obtain\n    \\begin{align}\n        \\frac{1}{\\Gamma_k^s} \\geq \\frac{1}{\\Gamma_k^s} - \\frac{1}{\\Gamma_{1}^s} \n        &\n        \\geq \n        % \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu}\\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}}{2^\\frac{2+4\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}}\\frac{1+3\\nu}{4+8\\nu}k^{\\frac{2+4\\nu}{1+3\\nu}}\n        % \\\\\n        % = & \n        \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu}\\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}}{2^\\frac{4+10\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}}\\frac{1+3\\nu}{1+2\\nu}k^{\\frac{2+4\\nu}{1+3\\nu}}\n    \\end{align}\n    Recalling that $s=(1+\\nu)/(1+3\\nu)$ and $\\Gamma_k = L_k\\gamma_k^2/k$, we conclude the proposition immediately from the above result.\n    % \\begin{align}\n    %     \\frac{L_k\\gamma_k^2}{k} \\le \\left(\\frac{1-\\nu}{1+\\nu}\\right)^\\frac{1-\\nu}{1+\\nu}\\frac{2^\\frac{4+10\\nu}{1+\\nu}M_\\nu^\\frac{2}{1+\\nu}}{\\varepsilon^\\frac{1-\\nu}{1+\\nu}}\\left(\\frac{1+2\\nu}{1+3\\nu}\\right)^{\\frac{1+3\\nu}{1+\\nu}}k^{-\\frac{2+4\\nu}{1+\\nu}}\n    % \\end{align}\n    % \n    % Noting that for $k \\geq 2$, $k^{\\frac{2+4\\nu}{1+3\\nu}}-1 \\geq \\frac{1}{2}k^{\\frac{2+4\\nu}{1+3\\nu}}$, we continue with \n    % \\begin{align}\n    %     \\frac{1}{\\Gamma_k^s} &\\geq \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu}\\left(\\frac{1+3\\nu}{2+4\\nu}\\right)\\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}(k^{\\frac{2+4\\nu}{1+3\\nu}}-1)}{2^\\frac{2+4\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}} \\\\\n    %     &\\geq \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu}\\left(\\frac{1+3\\nu}{2+4\\nu}\\right)\\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}k^{\\frac{2+4\\nu}{1+3\\nu}}}{2^\\frac{3+7\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}}\\\\\n    %     &= \\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}k^{\\frac{2+4\\nu}{1+3\\nu}}}{M_\\nu^\\frac{2}{1+3\\nu} C_\\nu^\\frac{1+\\nu}{1+3\\nu}}.\n    % \\end{align}\n    % Solving for $\\Gamma_k$ above, recalling our definition of $s$, and noting the relationship in \\eqref{def:Gammak}, we obtain the desired result. \n\\end{proof}","sha256_utf8":"2f0917a1a30150b12d65d92ec253bc1678ce93fd2951e22dbe973d76a045a189"},{"start":89945,"end":90389,"start_line":1005,"end_line":1010,"raw_tex":"\\begin{proposition}\\label{prop:flkDiff}\n    Suppose that the parameters in Algorithm \\ref{alg:UCGS} satisfy $\\beta_k \\geq L_k \\gamma_k$ for all $k$. Then for any $x\\in X$, \n    \\begin{align}\n        f(y_k) - \\ell_k(x) \\leq \\frac{\\varepsilon}{2} + \\Gamma_k \\sum_{i=1}^k \\frac{\\gamma_i\\beta_i}{2\\Gamma_i}\\left(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2\\right)^2 + \\Gamma_k \\sum_{i=1}^k \\frac{\\gamma_i\\eta_i}{\\Gamma_i}.\n    \\end{align} \n\\end{proposition}","sha256_utf8":"39c8849535ba8935c0e12cb03a037278b36f6ff328bb2c4953909e88f3bbb788"},{"start":90391,"end":93167,"start_line":1012,"end_line":1047,"raw_tex":"\\begin{proof}\n    Fix any $x\\in X$. From the definitions of $\\ell_k(x)$ and $y_k$ in \\eqref{def:lk} and \\eqref{def:yk} respectively, we have\n    \\begin{align}\n        \\frac{1}{\\Gamma_k}\\ell_k(x) =& \\sum_{i=1}^k \\frac{1}{\\Gamma_i}\\left( \\gamma_i f(z_i) + \\gamma_i \\ip{\\grad f(z_i),x-x_i} + \\ip{\\grad f(z_i),\\gamma_i(x_i-z_i)}\\right)\\\\\n        =& \\sum_{i=1}^k \\frac{1}{\\Gamma_i}\\left( f(z_i) + \\ip{\\grad f(z_i), y_i-z_i}\\right)+ \\frac{\\gamma_i}{\\Gamma_i} \\ip{\\grad f(z_i),x-x_i}\n        \\\\\n        &\\qquad- \\frac{1-\\gamma_i}{\\Gamma_i}(f(z_i) + \\ip{\\grad f(z_i), y_{i-1}-z_i})\n    \\end{align}\n    We will now bound three terms in the above relation.  First, by convexity of $f$, \n    \\begin{align}\n        -(f(z_i) + \\ip{\\grad f(z_i),y_{i-1}-z_i}) \\ge -f(y_{i-1}).\n    \\end{align}\n    Second, by our choice of $L_k$ in \\eqref{def:lk} and the definitions of $y_k$ and $z_k$ in \\eqref{def:yk} and \\eqref{def:zk} respectively, we have\n    \\begin{align}\n        f(z_i) + \\ip{\\grad f(z_i),y_i-z_i} \\ge & f(y_i) - \\frac{L_i}{2}\\norm{y_i-z_i}^2 - \\frac{\\varepsilon}{2}\\gamma_i \n        \\\\\n        = & f(y_i) - \\frac{L_i\\gamma_i^2}{2}\\norm{x_i-x_{i-1}}^2 - \\frac{\\varepsilon}{2}\\gamma_i.\n    \\end{align}\n    Lastly, using the result \\eqref{eq:innerIterUCGS_OC} in Lemma \\ref{lem:innerIterUCGS} and noting the definition of $\\phi(x)$ in \\eqref{def:phi}, we obtain the following result during the termination of the ACGM procedure in computing $x_i$:\n    \\begin{align}\n         \\ip{\\grad f(z_i), x-x_i} \\geq&\\: \\beta_i\\ip{x_i-x_{i-1},x_i-x} - \\eta_i\\\\\n        =& -\\frac{\\beta_i}{2}\\left(\\norm{x-x_{i-1}}^2 - \\norm{x_i-x_{i-1}}^2 - \\norm{x-x_i}^2\\right) - \\eta_i\\\\\n        \\geq& -\\frac{\\beta_i}{2}\\left(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2\\right) -\\eta_i + \\frac{L_i^2}{2}\\norm{x_i-x_{i-1}}^2.\n    \\end{align}\n    In the last inequality above we use our assumption that $\\beta_k\\geq L_k\\gamma_k$ for all $k$. Based on the above three observations and rearranging terms we obtain that\n    \\begin{align}\n        \\frac{1}{\\Gamma_k}\\ell_k(x) \\geq& \\sum_{i=1}^k \\frac{1}{\\Gamma_i}\\left( f(y_i) - (1-\\gamma_i)f(y_{i-1}) - \\frac{\\gamma_i\\beta_i}{2}(\\norm{x-x_{i-1}}^2-\\norm{x-x_i}^2)\\right)\n        \\\\\n        &\\qquad -\\frac{1}{\\Gamma_i}\\left(\\frac{\\varepsilon}{2}\\gamma_i + \\gamma_i\\eta_i\\right)\n        \\\\\n        = & \n        \\frac{f(y_k)}{\\Gamma_k} -\n        \\sum_{i=1}^k \\frac{\\gamma_i\\beta_i}{2\\Gamma_i}(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2) -\\sum_{i=1}^k \\frac{\\gamma_i\\eta_i}{\\Gamma_i} - \\frac{\\varepsilon}{2\\Gamma_k}.\n    \\end{align}\n    Here in the last equality we use the relations \\eqref{eq:GammaRelation} and \\eqref{eq:gammaSum} and the fact that $\\gamma_1=1$ in its definition \\eqref{def:gammak}. We conclude the result by multiplying by $\\Gamma_k$ and rearranging  terms.\n\\end{proof}","sha256_utf8":"7258156cff2cfbccfdbbf72e3c08188bd07e3e86d13891873ff9ff78bb844435"},{"start":95051,"end":101429,"start_line":1075,"end_line":1153,"raw_tex":"\\begin{proof}\n    From the definition of $s_k$ in Algorithm \\ref{alg:UCGS}, we have that if the termination criterion of UCGS in \\eqref{eq:stopCrit} holds, then $y_k$ is an $\\varepsilon$-solution to problem \\eqref{problem:CP}. Let us evaluate the number of gradient evaluations, or equivalently, the number of outer iterations of UCGS in order to compute an $\\varepsilon$-solution $y_k$.\n    % By the choice of parameters in \\eqref{eq:param}, we have $\\beta_k \\geq L_k\\gamma_k$.  Thus, applying Proposition\n    Applying Proposition \\ref{prop:flkDiff} with our choice of parameters we have\n    \\begin{align}\n        \\label{eq:flkStep1}\n        \\begin{aligned}\n        % & \n        f(y_k) - \\ell_k(x) \n        % \\\\\n        % \\leq& \\:\\frac{\\varepsilon}{2} + \\Gamma_k \\sum_{i=1}^k \\frac{L_i\\gamma_i^2}{2L_i\\gamma_i^2} i\\left(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2\\right) + \\Gamma_k \\sum_{i=1}^k \\frac{L_i\\gamma_i^2}{L_i\\gamma_i^2}D_X^2\n        % \\\\\n        =&\\: \\frac{\\varepsilon}{2} + \\frac{\\Gamma_k}{2}\\sum_{i=1}^k i(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2) + \\Gamma_kkD_X^2,\\ \\forall x\\in X.\n        \\end{aligned}\n    \\end{align}\n    The second term above can be further simplified by noting from the compactness of $X$ and the definition of the diameter $D_X$ in \\eqref{eq:D}. Indeed, we have\n    \\begin{align}\n    \\label{eq:telescope}\n    \\begin{aligned}\n        & \\sum_{i=1}^k i\\left(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2\\right)\n        \\\\\n        =& \\norm{x-x_0}^2 + \\sum_{i=2}^k (i-(i-1))\\norm{x-x_{i-1}}^2 - k\\norm{x-x_k}^2 \n        \\\\\n        \\leq&\\: D_X^2 + \\sum_{i=2}^k D_X^2 =\\: kD_X^2,\\ \\forall x\\in X.\n    \\end{aligned}\n    \\end{align}\n    Thus, we may continue by applying \\eqref{eq:telescope} to \\eqref{eq:flkStep1} with $x = s_k$ to conclude that\n    \\begin{align}\n        f(y_k) - \\ell_k(s_k) \\leq \\frac{\\varepsilon}{2} + \\frac{\\Gamma_k}{2}k D_X^2  + \\Gamma_kkD_X^2 = \\frac{\\varepsilon}{2} + \\frac{3L_k\\gamma_k^2}{2} D_X^2.\n    \\end{align}\n    Here we make use of the description of $\\Gamma_k$ in \\eqref{def:Gammakk} for the last equality. In view of the above result and the value of parameter $\\varepsilon_k$ in \\eqref{eq:param}, $y_k$ satisfies the termination criterion \\eqref{eq:stopCrit} of UCGS and hence becomes an $\\varepsilon$-solution whenever $k$ satisfies the relation \n    % \\begin{align}\n    %     \\frac{\\varepsilon}{2} + \\frac{3L_k\\gamma_k^2}{2} D_X^2 + \\frac{\\sigma L_k\\gamma_kD_X^2}{2} \\le \\varepsilon,\\text{ i.e., }(3+\\sigma)L_k\\gamma_k^2 D_X^2 \\leq \\varepsilon\n    % \\end{align}\n    $(3+\\sigma)L_k\\gamma_k^2 D_X^2 \\leq \\varepsilon$. Applying Lemma \\ref{lem:lkgammak}, it follows that such relation holds whenever\n    \\begin{align}\n        \\frac{(3+\\sigma)C_\\nu D_X^2  M_\\nu^{\\frac{2}{1+\\nu}}}{k^{\\frac{1+3\\nu}{1+\\nu}}\\varepsilon^{\\frac{1-\\nu}{1+\\nu}}} \\leq \\varepsilon,\\text{ i.e., }k\\ge C_\\nu^\\frac{1+\\nu}{1+3\\nu}\\left(\\frac{(3+\\sigma)^{\\frac{1+\\nu}{2}}M_\\nu D_X^{1+\\nu}}{\\varepsilon}\\right)^\\frac{2}{1+3\\nu}.\n    \\end{align}\n    Noting that $C_\\nu$ defined in \\eqref{eq:c} is a constant that depends only on $\\nu\\in[0,1]$ and observing that $C_\\nu^\\frac{1+\\nu}{1+3\\nu} \\leq 16$ for all $\\nu \\in [0,1]$, we conclude that whenever $k \\geq N_{\\text{grad}}$, $y_k$ is an $\\varepsilon$-solution. Therefore, UCGS requires at most $N_{\\text{grad}}$ gradient evaluations of $\\grad f$ to compute an $\\varepsilon$-solution.\n    % \\begin{align}\n    %     k \\geq 24\\left(\\frac{D_X^{1+\\nu}M_\\nu^\\frac{2}{1+\\nu}}{\\varepsilon}\\right)^\\frac{2}{1+3\\nu} = N_{grad}\n    % \\end{align}\n    \n    It suffices to compute the number of linear objective optimizations that UCGS requires for computing an $\\varepsilon$-solution. This is equivalent to estimating the total number of inner iterations that UCGS requires.\n    % Note first that whenever the termination criterion \\eqref{eq:actual_termi} is satisfied, then we have\n    % \\begin{align}\n    %     \\max_{x\\in X}\\ip{\\grad \\phi(u^{t-1}), u^{t-1}-x} &= \\max_{x\\in X}\\ip{\\grad \\phi(u^{t-1}), u^{t-1}-v^t} + \\ip{\\grad \\phi(u^{t-1}), v^t - x}\n    %     \\\\\n    %     &\\le  (\\eta - \\delta^t) + \\delta^t = \\eta.\n    % \\end{align}\n    % Here the inequality is from the termination criterion \\eqref{eq:actual_termi} and the fact that $v^t$ is an approximate solution that satisfies \\eqref{eq:cndgsubproblem}. \n    Let us estimate the maximal number of inner iterations required before the termination criterion \\eqref{eq:actual_termi} is satisfied.\n    Recall from the remark after \\eqref{def:alpha} that $\\alpha^t$ is the best linesearch parameter and hence satisfies assumption \\eqref{eq:alpha_req_2t1} in Proposition \\ref{prop:innerIterUCGS}. Applying Proposition \\ref{prop:innerIterUCGS} and noting the definition of approximate solution $v^t$ in \\eqref{eq:cndgsubproblem}, we have that\n    % \\begin{align}\n    %     \\underset{j=1, \\dots, t}{\\min}\\:\\ip{\\grad \\phi(u^{j-1}), u^{j-1}-v^{j}} \n    %     \\le & \\underset{j=1, \\dots, t}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x}\n    %     \\\\\n    %     = & \\underset{j=0, \\dots, t-1}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j}), u^{j}-x}\n    %     \\\\\n    %     \\le & \\frac{6(\\sigma+1)\\beta D_X^2}{t-1}\n    %     ,\\ \\forall t\\ge 2.\n    % \\end{align}\n    % From the above and noting the choice of $\\delta^t$, the termination criterion \\eqref{eq:actual_termi} holds whenever\n    % \\begin{align}\n    %     \\frac{6(\\sigma+1)\\beta D_X^2}{t-1} + \\frac{\\sigma \\beta D_X^2}{t} \\le \\eta.\n    % \\end{align}\n    % The above condition holds whenever\n    % \\begin{align}\n    %     t \\ge 1 + \\frac{(7\\sigma + 6)\\beta D_X^2}{\\eta}.\n    % \\end{align}\n    the maximal number of linear objective optimizations performed at the $k$-th call to the ACGM procedure is at most \n    \\begin{equation}\n        T_k := 1 + \\left\\lceil \\frac{(7\\sigma+6)\\beta_k D_X^2}{\\eta_k}\\right\\rceil = 1 + \\lceil (7\\sigma+6)k \\rceil.\n    \\end{equation}\n    Adding one linear objective optimization problem in \\eqref{def:lkproblem} in each other iteration concerning the termination criterion of UCGS, we conclude that the total number of linear objective optimizations for UCGS to compute an $\\varepsilon$-solution is bounded above by\n    \\begin{align}\n        \\sum_{k=1}^{N_{\\text{grad}}} (T_k + 1) \\le  \\sum_{k=1}^{N_{\\text{grad}}} 3 + (7\\sigma+6)k = \\left(\\frac{7}{2}\\sigma + 3\\right)N_{\\text{grad}}^2 + \\left(\\frac{7}{2}\\sigma + 6\\right)N_{\\text{grad}}.\n    \\end{align} \n\\end{proof}","sha256_utf8":"74b79e295795e4e4a03bbe23709aa3e2bc0987ff0d1a5a3040d852c2e959c8de"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"凸优化问题与集合直径","source_units":[{"start":4785,"end":5028,"start_line":119,"end_line":122,"raw_tex":"In this paper, we study first-order projection-free methods for computing $\\varepsilon$-approximation solutions to convex optimization problems of form\n\\begin{equation}\\label{problem:CP}\n    f^* = \\underset{x\\in X}{\\min}\\: f(x).\n\\end{equation}","sha256_utf8":"230f4a13004569898d7b8bfa6534359f5be26efa77dcd0e5986acf0eb9092e95"},{"start":5029,"end":5485,"start_line":123,"end_line":127,"raw_tex":"Here $X\\subset \\R^n$ is a high-dimensional compact convex set, $f$ is a convex function, and our goal is to compute an $\\varepsilon$-solution $y\\in X$ such that $f(y)-f^*\\le\\varepsilon$. We make the following assumptions concerning the set $X$ and the objective function $f$. Under the Euclidean norm $\\|\\cdot\\|$, we assume that $X$ is compact with diameter\n\\begin{align}\n\t\\label{eq:D}\n\tD_X := & \\underset{x,y \\in X}{\\max}\\: \\norm{x-y} < \\infty\n\\end{align}","sha256_utf8":"5521fb8ad9bbfaa8220f58b7e326920d5ae0b746af7becf1d4f21ec130f22a50"}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","title":"霍尔德上模型条件","source_units":[{"start":5486,"end":5728,"start_line":128,"end_line":132,"raw_tex":"and that there exists H\\\"older exponent $\\nu \\in (0,1]$ and constant $M_\\nu>0$ such that\n\\begin{equation}\n\t\\label{def:holderCont}\n\tf(y) \\leq f(x) + \\ip{\\grad f(x),y-x} + \\frac{M_\\nu}{1+\\nu}\\norm{x-y}^{1+\\nu},\\ \\forall x,y\\in X.\n\\end{equation}","sha256_utf8":"c92fddb8f844645c2874950e82e32fdc75c0112cd55cd73f0f72f997e9116402"}]},{"id":"S003","semantic_type":"algorithm","kind":"algorithm","title":"通用条件梯度滑动算法","source_units":[{"start":58543,"end":61513,"start_line":581,"end_line":653,"raw_tex":"\\begin{algorithm}[t]\n\t\\caption{\\label{alg:UCGS} Universal conditional gradient sliding (UCGS) method}\n\t\\footnotesize\n\t\\begin{algorithmic}\n\t\t\\State\n\t\tStart: Choose tolerance $\\varepsilon > 0$ and initial iteration $x_0 \\in X$.  Set $y_0 = x_0$. \n\t\t\\For {$k=1,2,\\ldots,$}\n\t\t\\State Decide $L_k >0$ such that\n\t\t\\begin{align}\n\t\t\tf(y_k) \\leq f(z_k) + \\ip{\\grad f(z_k),y_k-z_k} + \\frac{L_k}{2}\\norm{y_k-z_k}^2 + \\frac{\\varepsilon}{2}\\gamma_k\\label{def:lk}\n\t\t\\end{align}\n\t\t\\State where \n\t\t\\begin{align}\n\t\t\t\\label{def:gammak} \\gamma_k &:= \\begin{cases} 1, &k = 1\\\\ \\text{positive solution to } \\Gamma_{k-1}(1-\\gamma_k)=\\frac{L_k\\gamma_k^2}{k}, &k > 1 \\end{cases}\\\\\n\t\t\t\\label{def:zk} z_k &:= (1-\\gamma_k)y_{k-1} + \\gamma_k x_{k-1}\\\\\n\t\t\t\\label{def:xk} x_k &:= \\text{ACGM}(\\grad f(z_k), x_{k-1},\\beta_k,\\eta_k)\\\\\n\t\t\t\\label{def:yk} y_k &:= (1-\\gamma_k)y_{k-1} + \\gamma_k x_k\\\\\n\t\t\t\\label{def:Gammakk}\\Gamma_k &:= \\frac{L_k\\gamma_k^2}{k}.\n\t\t\\end{align}\n\t\t\n\t\t\\State \n\t\tCompute an approximate solution $s_k$ to the problem \n\t\t\\begin{align}\n\t\t    \\label{def:lkproblem}\n\t\t    \\min_{x\\in X}\\ell_k(x) := \\Gamma_k \\sum_{i=1}^k \\frac{\\gamma_i}{\\Gamma_i}\\left( f(z_i) + \\ip{\\grad f(z_i),x-z_i}\\right)\n\t\t\\end{align}\n\t\tsuch that $\\ell_k(s_k) - \\min_{x\\in X}\\ell_k(x)\\le \\varepsilon_k$. Terminate and output $y_k$ as an approximate solution if\n\t\t\\begin{align}\\label{eq:stopCrit}\n\t\t\tf(y_k) - \\ell_k(s_k) + \\varepsilon_k \\leq \\varepsilon.\n\t\t\\end{align}\n\t\t\\EndFor\n\t\t\\\\\n\t\t\n\t\t\n\t\t\\Procedure{$u^+ =$ ACGM}{$g,u,\\beta, \\eta$} \n\t\t\\State Goal: Compute $u^+$ such that $\\max_{x\\in X}\\ip{\\grad\\phi(u^+), u^+-x}\\le \\eta$, where\n\t\t\\begin{align}\n\t\t    \\label{def:phi}\n\t\t    \\phi(x):=\\ip{g, x} + \\frac{\\beta}{2}\\|x - u\\|^2.\n\t\t\\end{align}\n\t\t\\State Start: Set $u^0 = u$. \n\t\t\\For {$t=1,2,\\ldots,$}\n\t\t\\State Compute a $\\delta^t$-approximate solution $v^t$ to the problem $\\min_{x\\in X}\\ip{\\nabla\\phi(u^{t-1}),x}$ such that\n\t\t\\begin{equation}\\label{eq:cndgsubproblem}\n\t\t\t\\ip{g + \\beta(u^{t-1}-u),v^t} - \\min_{x\\in X}\\ip{g + \\beta(u^{t-1}-u), x}  \\leq \\delta^t.\n\t\t\\end{equation}\n\t\t\n\t\t\\State Terminate with $u^+:=u^{t-1}$ if\n\t\t\\begin{align}\n\t\t    \\label{eq:actual_termi}\n\t\t    \\ip{g+\\beta(u^{t-1}-u),u^{t-1}-v^t} + \\delta^t \\leq \\eta.\n\t\t\\end{align}\n\t\t\\State Otherwise, compute\n% \t\t\\begin{align}\n            $\n\t\t\tu^{t}=(1-\\alpha^t)u^{t-1}+\\alpha^tv^t\n% \t\t\t, \\text{ where }\n% \t\t\t\\alpha^t := \\min\\: \\left\\{1,\\frac{\\ip{g-\\beta(u-u^{t-1}),u^{t-1}-v^t}}{\\beta\\norm{v^t-u^{t-1}}^2}\\right\\}\n\t\t\t.\n\t\t\t$\n% \t\t\\end{align}\n\t\t% \t\t\t\\begin{enumerate}\n\t\t% \t\t\t    \\item Set $u_1 = u$ and $t=1$.\n\t\t% \t\t\t    \\item Let $v_t$ be the optimal solution for the subproblem of\n\t\t\n\t\t% \t\t\t    \\item If $V_{g,u,\\beta}(u_t) \\leq \\eta$, set $u^+ = u_t$ and terminate.\n\t\t% \t\t\t    \\item Set $u_{t+1}=(1-\\alpha_t)u_t+\\alpha_tv_t$ where $\\alpha_t = \\min\\: \\{1,\\ip{\\beta(u-u_t)-g,v_t-u_t}/(\\beta\\norm{v_t-u_t}^2)\\}$.\n\t\t% \t\t\t    \\item Set $t\\leftarrow t+1$ and go to step 2. \n\t\t% \t\t\t\\end{enumerate}\n\t\t\\EndFor\n\t\t\\EndProcedure\n\t\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"06238334a17241bea76e825c1603b54c14e225e778cceb8e3dd5478d12a6fd20"}]},{"id":"S004","semantic_type":"derived_fact","kind":"derived_fact","title":"内层精确线搜索","source_units":[{"start":61943,"end":62095,"start_line":658,"end_line":661,"raw_tex":"\\begin{align}\n\t\\label{def:alpha}\n\t\\alpha^t := \\min\\: \\left\\{1,\\frac{\\ip{g-\\beta(u-u^{t-1}),u^{t-1}-v^t}}{\\beta\\norm{v^t-u^{t-1}}^2}\\right\\}.\n\\end{align}","sha256_utf8":"5663aacb29781d9f61e7b67dec37791e68743236898a666a324ce1887d04bbbe"},{"start":62096,"end":62261,"start_line":662,"end_line":662,"raw_tex":"It is easy to observe that the above $\\alpha^t$ is the optimal solution to the exact linesearch problem $\\min_{\\alpha\\in[0,1]}\\phi((1-\\alpha)u^{t-1} + \\alpha v^t)$.\n","sha256_utf8":"262570bd3e6f559d55d726c4057a362f7af261ca88920818829b9847866572e9"}]},{"id":"S005","semantic_type":"derived_fact","kind":"derived_fact","title":"外层权重递推关系","source_units":[{"start":63113,"end":63254,"start_line":669,"end_line":671,"raw_tex":"\\begin{equation}\\label{eq:GammaRelation}\n    \\Gamma_k = \\begin{cases} 1, &k = 1\\\\ \\Gamma_{k-1}(1-\\gamma_k), &k > 1\\end{cases}.\n\\end{equation}","sha256_utf8":"9ff5920435913df8722482a931d473362eba31abaeace1febf18960299e43782"}]},{"id":"S006","semantic_type":"derived_fact","kind":"derived_fact","title":"光滑端点的连续延拓","source_units":[{"start":80455,"end":80721,"start_line":900,"end_line":900,"raw_tex":"Note that for $\\nu = 1$, the term $\\left(\\frac{1-\\nu}{1+\\nu}\\right)^\\frac{1-\\nu}{1+\\nu}$ can be handled using a continuity argument $\\lim_{\\nu\\to 1}\\left(\\frac{1-\\nu}{1+\\nu}\\right)^\\frac{1-\\nu}{1+\\nu} = 1$. We state an immediate corollary of the above lemma below. \n","sha256_utf8":"4a77d5759a0af24477717ee60e64ee8a721fdbc9da6afddb6dd3c6c08f336a56"}]},{"id":"S007","semantic_type":"theorem","kind":"theorem","title":"通用滑动算法的双重复杂度","source_units":[{"start":93369,"end":95049,"start_line":1051,"end_line":1073,"raw_tex":"\\begin{theorem}\\label{thm:flkDiff}\n    Suppose that we apply UCGS described in Algorithm \\ref{alg:UCGS} with parameters\n    \\begin{equation}\\label{eq:param}\n        \\beta_k = L_k\\gamma_k,\\ \\eta_k = \\frac{L_k\\gamma_kD_X^2}{k},\\text{ and }\\varepsilon_k = \\frac{\\sigma L_k\\gamma_kD_X^2}{2},\n    \\end{equation}\n    and $\\alpha^t$ in \\eqref{def:alpha} and $\\delta^t = \\sigma\\beta D_X^2/t$ in the ACGM procedure, where $\\sigma\\ge 0$ is a parameter related to the accuracy of approximately solving linear objective optimization subproblems.\n    Then \n    % we have\n    % \\begin{align}\n    %     f(y_k) - \\ell_k(x) \\leq \\frac{\\varepsilon}{2} + \\frac{3}{2}L_k\\gamma_k^2 D_X^2,\\ \\forall x\\in X.\n    % \\end{align}\n    % Consequently, \n    Algorithm \\ref{alg:UCGS} terminates with an $\\varepsilon$-solution after at most $N_{\\text{grad}}$ gradient evaluations and $N_{\\text{lin}}$ linear objective optimizations, where\n    \\begin{align}\\label{eq:evaluations}\n    \\begin{aligned}\n        N_{\\text{grad}} :=&\\left\\lceil16\\left(\\frac{(3+\\sigma)^{\\frac{1+\\nu}{2}}M_\\nu D_X^{1+\\nu}}{\\varepsilon}\\right)^\\frac{2}{1+3\\nu}\\right\\rceil\n        \\text{ and } \n        \\\\\n        N_{\\text{lin}} :=& \\left\\lceil\\left(\\frac{7}{2}\\sigma + 3\\right)N_{\\text{grad}}^2 + \\left(\\frac{7}{2}\\sigma + 6\\right)N_{\\text{grad}}\\right\\rceil.\n    \\end{aligned} \n    \\end{align}\n    % As a consequence, the total number of gradient evaluations and linear objective optimizations performed by UCGS to find an $\\varepsilon$ solution of \\eqref{problem:CP} can be bounded by $\\cO((M_{\\nu}D_X^{1+\\nu}/\\varepsilon)^{\\frac{2}{1+3\\nu}})$ and $\\cO((M_{\\nu}D_X^{1+\\nu}/\\varepsilon)^{\\frac{4}{1+3\\nu}})$ respectively. \n\\end{theorem}","sha256_utf8":"140498d20838d27c867698735f73f8d90156e3a4acae9a244489c4f1d49568be"}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","title":"主定理的参数与精度设置","source_units":[{"start":93404,"end":93903,"start_line":1052,"end_line":1056,"raw_tex":"    Suppose that we apply UCGS described in Algorithm \\ref{alg:UCGS} with parameters\n    \\begin{equation}\\label{eq:param}\n        \\beta_k = L_k\\gamma_k,\\ \\eta_k = \\frac{L_k\\gamma_kD_X^2}{k},\\text{ and }\\varepsilon_k = \\frac{\\sigma L_k\\gamma_kD_X^2}{2},\n    \\end{equation}\n    and $\\alpha^t$ in \\eqref{def:alpha} and $\\delta^t = \\sigma\\beta D_X^2/t$ in the ACGM procedure, where $\\sigma\\ge 0$ is a parameter related to the accuracy of approximately solving linear objective optimization subproblems.\n","sha256_utf8":"1dd0202d9e1a43859117a47931b4e9365d4721411ee2ddca6daf4807d97d2f6e"}]},{"id":"S009","semantic_type":"proof","kind":"proof","title":"主复杂度定理完整证明","source_units":[{"start":95051,"end":101429,"start_line":1075,"end_line":1153,"raw_tex":"\\begin{proof}\n    From the definition of $s_k$ in Algorithm \\ref{alg:UCGS}, we have that if the termination criterion of UCGS in \\eqref{eq:stopCrit} holds, then $y_k$ is an $\\varepsilon$-solution to problem \\eqref{problem:CP}. Let us evaluate the number of gradient evaluations, or equivalently, the number of outer iterations of UCGS in order to compute an $\\varepsilon$-solution $y_k$.\n    % By the choice of parameters in \\eqref{eq:param}, we have $\\beta_k \\geq L_k\\gamma_k$.  Thus, applying Proposition\n    Applying Proposition \\ref{prop:flkDiff} with our choice of parameters we have\n    \\begin{align}\n        \\label{eq:flkStep1}\n        \\begin{aligned}\n        % & \n        f(y_k) - \\ell_k(x) \n        % \\\\\n        % \\leq& \\:\\frac{\\varepsilon}{2} + \\Gamma_k \\sum_{i=1}^k \\frac{L_i\\gamma_i^2}{2L_i\\gamma_i^2} i\\left(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2\\right) + \\Gamma_k \\sum_{i=1}^k \\frac{L_i\\gamma_i^2}{L_i\\gamma_i^2}D_X^2\n        % \\\\\n        =&\\: \\frac{\\varepsilon}{2} + \\frac{\\Gamma_k}{2}\\sum_{i=1}^k i(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2) + \\Gamma_kkD_X^2,\\ \\forall x\\in X.\n        \\end{aligned}\n    \\end{align}\n    The second term above can be further simplified by noting from the compactness of $X$ and the definition of the diameter $D_X$ in \\eqref{eq:D}. Indeed, we have\n    \\begin{align}\n    \\label{eq:telescope}\n    \\begin{aligned}\n        & \\sum_{i=1}^k i\\left(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2\\right)\n        \\\\\n        =& \\norm{x-x_0}^2 + \\sum_{i=2}^k (i-(i-1))\\norm{x-x_{i-1}}^2 - k\\norm{x-x_k}^2 \n        \\\\\n        \\leq&\\: D_X^2 + \\sum_{i=2}^k D_X^2 =\\: kD_X^2,\\ \\forall x\\in X.\n    \\end{aligned}\n    \\end{align}\n    Thus, we may continue by applying \\eqref{eq:telescope} to \\eqref{eq:flkStep1} with $x = s_k$ to conclude that\n    \\begin{align}\n        f(y_k) - \\ell_k(s_k) \\leq \\frac{\\varepsilon}{2} + \\frac{\\Gamma_k}{2}k D_X^2  + \\Gamma_kkD_X^2 = \\frac{\\varepsilon}{2} + \\frac{3L_k\\gamma_k^2}{2} D_X^2.\n    \\end{align}\n    Here we make use of the description of $\\Gamma_k$ in \\eqref{def:Gammakk} for the last equality. In view of the above result and the value of parameter $\\varepsilon_k$ in \\eqref{eq:param}, $y_k$ satisfies the termination criterion \\eqref{eq:stopCrit} of UCGS and hence becomes an $\\varepsilon$-solution whenever $k$ satisfies the relation \n    % \\begin{align}\n    %     \\frac{\\varepsilon}{2} + \\frac{3L_k\\gamma_k^2}{2} D_X^2 + \\frac{\\sigma L_k\\gamma_kD_X^2}{2} \\le \\varepsilon,\\text{ i.e., }(3+\\sigma)L_k\\gamma_k^2 D_X^2 \\leq \\varepsilon\n    % \\end{align}\n    $(3+\\sigma)L_k\\gamma_k^2 D_X^2 \\leq \\varepsilon$. Applying Lemma \\ref{lem:lkgammak}, it follows that such relation holds whenever\n    \\begin{align}\n        \\frac{(3+\\sigma)C_\\nu D_X^2  M_\\nu^{\\frac{2}{1+\\nu}}}{k^{\\frac{1+3\\nu}{1+\\nu}}\\varepsilon^{\\frac{1-\\nu}{1+\\nu}}} \\leq \\varepsilon,\\text{ i.e., }k\\ge C_\\nu^\\frac{1+\\nu}{1+3\\nu}\\left(\\frac{(3+\\sigma)^{\\frac{1+\\nu}{2}}M_\\nu D_X^{1+\\nu}}{\\varepsilon}\\right)^\\frac{2}{1+3\\nu}.\n    \\end{align}\n    Noting that $C_\\nu$ defined in \\eqref{eq:c} is a constant that depends only on $\\nu\\in[0,1]$ and observing that $C_\\nu^\\frac{1+\\nu}{1+3\\nu} \\leq 16$ for all $\\nu \\in [0,1]$, we conclude that whenever $k \\geq N_{\\text{grad}}$, $y_k$ is an $\\varepsilon$-solution. Therefore, UCGS requires at most $N_{\\text{grad}}$ gradient evaluations of $\\grad f$ to compute an $\\varepsilon$-solution.\n    % \\begin{align}\n    %     k \\geq 24\\left(\\frac{D_X^{1+\\nu}M_\\nu^\\frac{2}{1+\\nu}}{\\varepsilon}\\right)^\\frac{2}{1+3\\nu} = N_{grad}\n    % \\end{align}\n    \n    It suffices to compute the number of linear objective optimizations that UCGS requires for computing an $\\varepsilon$-solution. This is equivalent to estimating the total number of inner iterations that UCGS requires.\n    % Note first that whenever the termination criterion \\eqref{eq:actual_termi} is satisfied, then we have\n    % \\begin{align}\n    %     \\max_{x\\in X}\\ip{\\grad \\phi(u^{t-1}), u^{t-1}-x} &= \\max_{x\\in X}\\ip{\\grad \\phi(u^{t-1}), u^{t-1}-v^t} + \\ip{\\grad \\phi(u^{t-1}), v^t - x}\n    %     \\\\\n    %     &\\le  (\\eta - \\delta^t) + \\delta^t = \\eta.\n    % \\end{align}\n    % Here the inequality is from the termination criterion \\eqref{eq:actual_termi} and the fact that $v^t$ is an approximate solution that satisfies \\eqref{eq:cndgsubproblem}. \n    Let us estimate the maximal number of inner iterations required before the termination criterion \\eqref{eq:actual_termi} is satisfied.\n    Recall from the remark after \\eqref{def:alpha} that $\\alpha^t$ is the best linesearch parameter and hence satisfies assumption \\eqref{eq:alpha_req_2t1} in Proposition \\ref{prop:innerIterUCGS}. Applying Proposition \\ref{prop:innerIterUCGS} and noting the definition of approximate solution $v^t$ in \\eqref{eq:cndgsubproblem}, we have that\n    % \\begin{align}\n    %     \\underset{j=1, \\dots, t}{\\min}\\:\\ip{\\grad \\phi(u^{j-1}), u^{j-1}-v^{j}} \n    %     \\le & \\underset{j=1, \\dots, t}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x}\n    %     \\\\\n    %     = & \\underset{j=0, \\dots, t-1}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j}), u^{j}-x}\n    %     \\\\\n    %     \\le & \\frac{6(\\sigma+1)\\beta D_X^2}{t-1}\n    %     ,\\ \\forall t\\ge 2.\n    % \\end{align}\n    % From the above and noting the choice of $\\delta^t$, the termination criterion \\eqref{eq:actual_termi} holds whenever\n    % \\begin{align}\n    %     \\frac{6(\\sigma+1)\\beta D_X^2}{t-1} + \\frac{\\sigma \\beta D_X^2}{t} \\le \\eta.\n    % \\end{align}\n    % The above condition holds whenever\n    % \\begin{align}\n    %     t \\ge 1 + \\frac{(7\\sigma + 6)\\beta D_X^2}{\\eta}.\n    % \\end{align}\n    the maximal number of linear objective optimizations performed at the $k$-th call to the ACGM procedure is at most \n    \\begin{equation}\n        T_k := 1 + \\left\\lceil \\frac{(7\\sigma+6)\\beta_k D_X^2}{\\eta_k}\\right\\rceil = 1 + \\lceil (7\\sigma+6)k \\rceil.\n    \\end{equation}\n    Adding one linear objective optimization problem in \\eqref{def:lkproblem} in each other iteration concerning the termination criterion of UCGS, we conclude that the total number of linear objective optimizations for UCGS to compute an $\\varepsilon$-solution is bounded above by\n    \\begin{align}\n        \\sum_{k=1}^{N_{\\text{grad}}} (T_k + 1) \\le  \\sum_{k=1}^{N_{\\text{grad}}} 3 + (7\\sigma+6)k = \\left(\\frac{7}{2}\\sigma + 3\\right)N_{\\text{grad}}^2 + \\left(\\frac{7}{2}\\sigma + 6\\right)N_{\\text{grad}}.\n    \\end{align} \n\\end{proof}","sha256_utf8":"74b79e295795e4e4a03bbe23709aa3e2bc0987ff0d1a5a3040d852c2e959c8de"}]},{"id":"S010","semantic_type":"proof_step","kind":"proof_step","title":"验证停止证书并代入参数","source_units":[{"start":95051,"end":96173,"start_line":1075,"end_line":1089,"raw_tex":"\\begin{proof}\n    From the definition of $s_k$ in Algorithm \\ref{alg:UCGS}, we have that if the termination criterion of UCGS in \\eqref{eq:stopCrit} holds, then $y_k$ is an $\\varepsilon$-solution to problem \\eqref{problem:CP}. Let us evaluate the number of gradient evaluations, or equivalently, the number of outer iterations of UCGS in order to compute an $\\varepsilon$-solution $y_k$.\n    % By the choice of parameters in \\eqref{eq:param}, we have $\\beta_k \\geq L_k\\gamma_k$.  Thus, applying Proposition\n    Applying Proposition \\ref{prop:flkDiff} with our choice of parameters we have\n    \\begin{align}\n        \\label{eq:flkStep1}\n        \\begin{aligned}\n        % & \n        f(y_k) - \\ell_k(x) \n        % \\\\\n        % \\leq& \\:\\frac{\\varepsilon}{2} + \\Gamma_k \\sum_{i=1}^k \\frac{L_i\\gamma_i^2}{2L_i\\gamma_i^2} i\\left(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2\\right) + \\Gamma_k \\sum_{i=1}^k \\frac{L_i\\gamma_i^2}{L_i\\gamma_i^2}D_X^2\n        % \\\\\n        =&\\: \\frac{\\varepsilon}{2} + \\frac{\\Gamma_k}{2}\\sum_{i=1}^k i(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2) + \\Gamma_kkD_X^2,\\ \\forall x\\in X.\n        \\end{aligned}\n    \\end{align}\n","sha256_utf8":"0d8ba306b1d942f7d45aa25d6c583f8b1add610e003c31db9458855833f82aa5"}]},{"id":"S011","semantic_type":"proof_step","kind":"proof_step","title":"直径控制加权望远镜和","source_units":[{"start":96173,"end":96689,"start_line":1090,"end_line":1100,"raw_tex":"    The second term above can be further simplified by noting from the compactness of $X$ and the definition of the diameter $D_X$ in \\eqref{eq:D}. Indeed, we have\n    \\begin{align}\n    \\label{eq:telescope}\n    \\begin{aligned}\n        & \\sum_{i=1}^k i\\left(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2\\right)\n        \\\\\n        =& \\norm{x-x_0}^2 + \\sum_{i=2}^k (i-(i-1))\\norm{x-x_{i-1}}^2 - k\\norm{x-x_k}^2 \n        \\\\\n        \\leq&\\: D_X^2 + \\sum_{i=2}^k D_X^2 =\\: kD_X^2,\\ \\forall x\\in X.\n    \\end{aligned}\n    \\end{align}\n","sha256_utf8":"eb7255d3f71475cb1832c0339f84cfe674fcb5d2b019c672d58539b203d8d930"}]},{"id":"S012","semantic_type":"proof_step","kind":"proof_step","title":"得到停止模型差上界","source_units":[{"start":96689,"end":96997,"start_line":1101,"end_line":1104,"raw_tex":"    Thus, we may continue by applying \\eqref{eq:telescope} to \\eqref{eq:flkStep1} with $x = s_k$ to conclude that\n    \\begin{align}\n        f(y_k) - \\ell_k(s_k) \\leq \\frac{\\varepsilon}{2} + \\frac{\\Gamma_k}{2}k D_X^2  + \\Gamma_kkD_X^2 = \\frac{\\varepsilon}{2} + \\frac{3L_k\\gamma_k^2}{2} D_X^2.\n    \\end{align}\n","sha256_utf8":"813ab49a291f969656bd45055e58ffb470df0baa01ec70e5f9ae0527dc25ed46"}]},{"id":"S013","semantic_type":"proof_step","kind":"proof_step","title":"推出梯度评价次数界","source_units":[{"start":96997,"end":98549,"start_line":1105,"end_line":1117,"raw_tex":"    Here we make use of the description of $\\Gamma_k$ in \\eqref{def:Gammakk} for the last equality. In view of the above result and the value of parameter $\\varepsilon_k$ in \\eqref{eq:param}, $y_k$ satisfies the termination criterion \\eqref{eq:stopCrit} of UCGS and hence becomes an $\\varepsilon$-solution whenever $k$ satisfies the relation \n    % \\begin{align}\n    %     \\frac{\\varepsilon}{2} + \\frac{3L_k\\gamma_k^2}{2} D_X^2 + \\frac{\\sigma L_k\\gamma_kD_X^2}{2} \\le \\varepsilon,\\text{ i.e., }(3+\\sigma)L_k\\gamma_k^2 D_X^2 \\leq \\varepsilon\n    % \\end{align}\n    $(3+\\sigma)L_k\\gamma_k^2 D_X^2 \\leq \\varepsilon$. Applying Lemma \\ref{lem:lkgammak}, it follows that such relation holds whenever\n    \\begin{align}\n        \\frac{(3+\\sigma)C_\\nu D_X^2  M_\\nu^{\\frac{2}{1+\\nu}}}{k^{\\frac{1+3\\nu}{1+\\nu}}\\varepsilon^{\\frac{1-\\nu}{1+\\nu}}} \\leq \\varepsilon,\\text{ i.e., }k\\ge C_\\nu^\\frac{1+\\nu}{1+3\\nu}\\left(\\frac{(3+\\sigma)^{\\frac{1+\\nu}{2}}M_\\nu D_X^{1+\\nu}}{\\varepsilon}\\right)^\\frac{2}{1+3\\nu}.\n    \\end{align}\n    Noting that $C_\\nu$ defined in \\eqref{eq:c} is a constant that depends only on $\\nu\\in[0,1]$ and observing that $C_\\nu^\\frac{1+\\nu}{1+3\\nu} \\leq 16$ for all $\\nu \\in [0,1]$, we conclude that whenever $k \\geq N_{\\text{grad}}$, $y_k$ is an $\\varepsilon$-solution. Therefore, UCGS requires at most $N_{\\text{grad}}$ gradient evaluations of $\\grad f$ to compute an $\\varepsilon$-solution.\n    % \\begin{align}\n    %     k \\geq 24\\left(\\frac{D_X^{1+\\nu}M_\\nu^\\frac{2}{1+\\nu}}{\\varepsilon}\\right)^\\frac{2}{1+3\\nu} = N_{grad}\n    % \\end{align}\n    \n","sha256_utf8":"a7c5812aa1ee988031e4fe6c789340523d218b6a17bc301bcf67fcabd65c630f"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"转向内层迭代计数","source_units":[{"start":98549,"end":99449,"start_line":1118,"end_line":1126,"raw_tex":"    It suffices to compute the number of linear objective optimizations that UCGS requires for computing an $\\varepsilon$-solution. This is equivalent to estimating the total number of inner iterations that UCGS requires.\n    % Note first that whenever the termination criterion \\eqref{eq:actual_termi} is satisfied, then we have\n    % \\begin{align}\n    %     \\max_{x\\in X}\\ip{\\grad \\phi(u^{t-1}), u^{t-1}-x} &= \\max_{x\\in X}\\ip{\\grad \\phi(u^{t-1}), u^{t-1}-v^t} + \\ip{\\grad \\phi(u^{t-1}), v^t - x}\n    %     \\\\\n    %     &\\le  (\\eta - \\delta^t) + \\delta^t = \\eta.\n    % \\end{align}\n    % Here the inequality is from the termination criterion \\eqref{eq:actual_termi} and the fact that $v^t$ is an approximate solution that satisfies \\eqref{eq:cndgsubproblem}. \n    Let us estimate the maximal number of inner iterations required before the termination criterion \\eqref{eq:actual_termi} is satisfied.\n","sha256_utf8":"18c0e6977e1aede187e59a9a97aef73a998d0b41d3eb35c9276ee0a5d08d239d"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"估计每次内层调用成本","source_units":[{"start":99449,"end":100896,"start_line":1127,"end_line":1148,"raw_tex":"    Recall from the remark after \\eqref{def:alpha} that $\\alpha^t$ is the best linesearch parameter and hence satisfies assumption \\eqref{eq:alpha_req_2t1} in Proposition \\ref{prop:innerIterUCGS}. Applying Proposition \\ref{prop:innerIterUCGS} and noting the definition of approximate solution $v^t$ in \\eqref{eq:cndgsubproblem}, we have that\n    % \\begin{align}\n    %     \\underset{j=1, \\dots, t}{\\min}\\:\\ip{\\grad \\phi(u^{j-1}), u^{j-1}-v^{j}} \n    %     \\le & \\underset{j=1, \\dots, t}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x}\n    %     \\\\\n    %     = & \\underset{j=0, \\dots, t-1}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j}), u^{j}-x}\n    %     \\\\\n    %     \\le & \\frac{6(\\sigma+1)\\beta D_X^2}{t-1}\n    %     ,\\ \\forall t\\ge 2.\n    % \\end{align}\n    % From the above and noting the choice of $\\delta^t$, the termination criterion \\eqref{eq:actual_termi} holds whenever\n    % \\begin{align}\n    %     \\frac{6(\\sigma+1)\\beta D_X^2}{t-1} + \\frac{\\sigma \\beta D_X^2}{t} \\le \\eta.\n    % \\end{align}\n    % The above condition holds whenever\n    % \\begin{align}\n    %     t \\ge 1 + \\frac{(7\\sigma + 6)\\beta D_X^2}{\\eta}.\n    % \\end{align}\n    the maximal number of linear objective optimizations performed at the $k$-th call to the ACGM procedure is at most \n    \\begin{equation}\n        T_k := 1 + \\left\\lceil \\frac{(7\\sigma+6)\\beta_k D_X^2}{\\eta_k}\\right\\rceil = 1 + \\lceil (7\\sigma+6)k \\rceil.\n    \\end{equation}\n","sha256_utf8":"609f4fa123667955e0e65c0fe63511b68a8b5c2dcd99dc74f7de1fc26d600bdb"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"汇总线性优化总次数","source_units":[{"start":100896,"end":101429,"start_line":1149,"end_line":1153,"raw_tex":"    Adding one linear objective optimization problem in \\eqref{def:lkproblem} in each other iteration concerning the termination criterion of UCGS, we conclude that the total number of linear objective optimizations for UCGS to compute an $\\varepsilon$-solution is bounded above by\n    \\begin{align}\n        \\sum_{k=1}^{N_{\\text{grad}}} (T_k + 1) \\le  \\sum_{k=1}^{N_{\\text{grad}}} 3 + (7\\sigma+6)k = \\left(\\frac{7}{2}\\sigma + 3\\right)N_{\\text{grad}}^2 + \\left(\\frac{7}{2}\\sigma + 6\\right)N_{\\text{grad}}.\n    \\end{align} \n\\end{proof}","sha256_utf8":"eb83b9f2cb0d2b1115251fecceb9246ea9963bb48309f98533067b3f6d4f2ff5"}]},{"id":"S017","semantic_type":"lemma","kind":"lemma","title":"递推不等式的求和引理","source_units":[{"start":37019,"end":37565,"start_line":366,"end_line":382,"raw_tex":"\\begin{lemma}\n\t\\label{lem:tele}\n\tSuppose that $\\{a_k\\},\\{b_k\\}\\subset\\R$ and $\\{\\gamma_k\\}\\subset [0,1]$ are sequences that satisfy $\\gamma_1=1$ and\n\t\\begin{align}\n\t\t\\label{eq:tele_assum}\n\t\ta_k \\le (1-\\gamma_k)a_{k-1} + \\gamma_k b_k,\\ \\forall k\\ge 1.\n\t\\end{align}\n\tThen we have\n\t\\begin{align}\n\t\t\\label{eq:tele_res}\n\t\ta_k\\le \\Gamma_k\\sum_{i=1}^{k}\\frac{\\gamma_i}{\\Gamma_i}b_i,\\ \\forall k\\ge 1,\\text{ where }\\Gamma_k:=\\begin{cases}\n\t\t\t1 & \\text{ when }k=1\n\t\t\t\\\\\n\t\t\t\\Gamma_{k-1}(1-\\gamma_k) & \\text{ when }k>1.\n\t\t\\end{cases}\n\t\\end{align}\n\\end{lemma}","sha256_utf8":"d290f0c73dd3ff2c60e13e8d86bcae8ad94d9cb9c4c0338fd4b3d41d365b12d1"}]},{"id":"S018","semantic_type":"proof","kind":"proof","title":"递推求和引理证明","source_units":[{"start":37566,"end":37791,"start_line":383,"end_line":385,"raw_tex":"\\begin{proof}\n\tDividing both sides of \\eqref{eq:tele_assum} by $\\Gamma_k$ we obtain a series of inequalities with telescoping terms concerning sequence $\\{a_k/\\Gamma_k\\}$. Summing up we obtain \\eqref{eq:tele_res}.\n\\end{proof}","sha256_utf8":"a995989e283d3d147ef119e543670b9c6d67df4e5fff852b4efc1b4a66964867"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"归一化递推并望远镜求和","source_units":[{"start":37566,"end":37791,"start_line":383,"end_line":385,"raw_tex":"\\begin{proof}\n\tDividing both sides of \\eqref{eq:tele_assum} by $\\Gamma_k$ we obtain a series of inequalities with telescoping terms concerning sequence $\\{a_k/\\Gamma_k\\}$. Summing up we obtain \\eqref{eq:tele_res}.\n\\end{proof}","sha256_utf8":"a995989e283d3d147ef119e543670b9c6d67df4e5fff852b4efc1b4a66964867"}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","title":"内层间隙的加权估计","source_units":[{"start":68146,"end":69559,"start_line":714,"end_line":740,"raw_tex":"\\begin{lemma}\\label{lem:innerIterUCGS}\n    Suppose that $\\{\\lambda^t\\}\\in [0,1]$ is any predetermined sequence satisfying $\\lambda^1= 1$. In the ACGM procedure, if $\\alpha^t$ is chosen such that \n    \\begin{align}\n        \\label{eq:alpha_req}\n        \\phi(u^t)\\le \\phi((1-\\lambda^t)u^{t-1} + \\lambda^tv^t),\\ \\forall t\\ge 1,\n    \\end{align}\n    then we have\n    \\begin{align}\n        \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j}\\: \\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}),u^{j-1} - x}\n        % \\\\\n%        \\leq & \\delta^1 + \\frac{\\beta D_X^2}{2} + \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j}\\left(\\delta^j + \\frac{\\beta D_X^2\\lambda^j}{2} +  \\Lambda^{j-1}\\sum_{i=1}^{j-1}\\frac{\\lambda^i}{\\Lambda^i}\\left(\\delta^i + \\frac{\\beta D_X^2\\lambda^i}{2}\\right)\\right)\n        \\leq &\n        \\left[\\delta^1 + \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j}\\left(\\delta^j + \\Lambda^{j-1}\\sum_{i=1}^{j-1}\\frac{\\lambda^i}{\\Lambda^i}\\delta^i\\right)\\right] \n        \\\\\n        & +\n        \\frac{\\beta D_X^2}{2}\\left[1+\\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j} \\left( \\lambda^j + \\Lambda^{j-1}\\sum_{i=1}^{j-1}\\frac{(\\lambda^i)^2}{\\Lambda^i}\\right)\\right]\n    \\end{align}\n    for all $t \\geq 2$, where\n    \\begin{align}\n        \\label{def:Lambda}\n        \\Lambda^t:=\\begin{cases}\n            1 & \\text{ when }t=1\n            \\\\\n            \\Lambda^{t-1}(1-\\lambda^t) & \\text{ when } t>1.\n        \\end{cases}\n    \\end{align}\n\\end{lemma}","sha256_utf8":"7abc49e2e7b3dbb1c4335b6cdd0e44b3e69a6ab007607a56c5007ccf54a994d8"}]},{"id":"S021","semantic_type":"proof","kind":"proof","title":"内层加权估计完整证明","source_units":[{"start":69561,"end":73402,"start_line":742,"end_line":795,"raw_tex":"\\begin{proof}\n    Observing that the function $\\phi(x)$ in \\eqref{def:phi} is a strongly convex function with Lipschitz continuous (with constant $\\beta$) gradient, using the assumption \\eqref{eq:alpha_req}, and noting the definition of approximate solution $v^t$ in \\eqref{eq:cndgsubproblem}, we have\n    \\begin{align}\n    \t\\label{eq:phiConvex}\n    \t\\begin{aligned}\n        &\\phi(u^t) - (1-\\lambda^t)\\phi(u^{t-1}) - \\lambda^t(\\phi(u^{t-1}) + \\ip{\\grad \\phi(u^{t-1}),x-u^{t-1}})\n        \\\\\n        \\leq& \\phi((1-\\lambda^t)u^{t-1} + \\lambda^tv^t) - \\phi(u^{t-1}) - \\lambda^t\\ip{\\grad\\phi(u^{t-1}),x-u^{t-1}}\n        \\\\\n        \\le & \\lambda^t\\ip{\\grad \\phi(u^{t-1}), v^t - u^{t-1}} + \\frac{\\beta(\\lambda^t)^2}{2}\\norm{v^t-u^{t-1}}^2 - \\lambda^t\\ip{\\grad \\phi(u^{t-1}),x-u^{t-1}}\n        \\\\\n        =& \\lambda^t\\ip{\\grad\\phi(u^{t-1}),v^t-x} + \\frac{\\beta(\\lambda^t)^2}{2}\\norm{v^t-u^{t-1}}^2\\\\\n        \\leq& \\lambda^t\\delta^t +\\frac{\\beta D_X^2(\\lambda^t)^2}{2},\\ \\forall x\\in X, t\\ge 1.\n\t    \\end{aligned}\n    \\end{align}\n\tDefining $x^*:=\\operatorname{argmin}_{x\\in X}\\phi(x)$, from the above relation we have for any $t\\ge 2$ that\n\t\\begin{align}\n\t\t\\label{eq:tmp1}\n\t\t\\begin{aligned}\n\t\t& \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j} \\underset{x\\in X}{\\max}\\:\\ip{\\grad\\phi(u^{j-1}), u^{j-1}-x}\n\t\t\\\\\n\t\t\\leq& \\sum_{j=2}^t \\frac{1}{\\Lambda^j}[\\phi(u^{j-1}) - \\phi(x^*)] - \\frac{1}{\\Lambda^j}[\\phi(u^j) - \\phi(x^*)] + \\frac{\\lambda^j}{\\Lambda^j}\\delta^j + \\frac{\\beta D_X^2 (\\lambda^j)^2}{2\\Lambda^j}\n\t\t\\\\\n\t\t= & [\\phi(u^1) - \\phi(x^*)] - \\frac{1}{\\Lambda^t}[\\phi(u^t) - \\phi(x^*)] \n\t\t\\\\\n\t\t& + \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j}[\\phi(u^{j-1}) - \\phi(x^*)] + \\frac{\\lambda^j}{\\Lambda^j}\\delta^j + \\frac{\\beta D_X^2(\\lambda^j)^2}{2\\Lambda^j}\n\t\t\\\\\n\t\t\\le & [\\phi(u^1) - \\phi(x^*)] + \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j}[\\phi(u^{j-1}) - \\phi(x^*)] + \\frac{\\lambda^j}{\\Lambda^j}\\delta^j + \\frac{\\beta D_X^2(\\lambda^j)^2}{2\\Lambda^j}.\n\t\t\\end{aligned}\n\t\\end{align}\n\tHere in the equality we use the following observations from the definition of $\\Lambda^t$ in \\eqref{def:Lambda}: $1/\\Lambda^1 = 1$ and $1/\\Lambda^j = 1/\\Lambda^{j-1} + \\lambda^j/\\Lambda^j$ for all $j\\ge 2$, \n%\t\\begin{align}\n%\t\t\\frac{1}{\\Lambda^j} = \\frac{1}{\\Lambda^{j-1}} + \\frac{\\lambda^j}{\\Lambda^{j}},\\ \\forall j\\ge 2,\n%\t\\end{align}\n\n    To finish the proof it suffices to bound $\\phi(u^{j-1})-\\phi(x^*)$ for any $j\\ge 2$. Observing that $\\phi(x)$ in \\eqref{def:phi} is strongly convex and quadratic with\n    \\begin{equation}\n        \\frac{\\beta}{2}\\norm{x-u^{t-1}}^2 = \\phi(x) - (\\phi(u^{t-1})+\\ip{\\grad \\phi(u^{t-1}), x-u^{t-1}}),\\ \\forall x\\in X, t\\ge 1,\n    \\end{equation}\n    we have from \\eqref{eq:phiConvex} (with $x = x^*$) that\n    \\begin{equation}\\label{eq:telescopingBound}\n        [\\phi(u^t)-\\phi(x^*)] - (1-\\lambda^t)[\\phi(u^{t-1})-\\phi(x^*)] \\leq \\lambda^t\\delta^t + \\frac{\\beta D_X^2(\\lambda^t)^2}{2} - \\frac{\\beta\\lambda^t}{2}\\norm{x^*-u^{t-1}}^2.\n    \\end{equation}\n    Applying Lemma \\ref{lem:tele} to the above recurrence relation and ignoring negative terms at the right hand side, we have\n    \\begin{align}\\label{eq:simplePhiBound}\n        \\phi(u^t) - \\phi(x^*) \\leq \\Lambda^t\\sum_{i=1}^t \\frac{\\lambda^i}{\\Lambda^i}\\delta^i + \\frac{\\beta D_X^2(\\lambda^i)^2}{2\\Lambda^i},\\ \\forall t\\ge 1.\n    \\end{align}\n\tWe conclude the lemma immediately by applying the above bound to \\eqref{eq:tmp1} and rearranging terms.\n%\t\\begin{align}\n%\t\t\t& \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j} \\underset{x\\in X}{\\max}\\:\\ip{\\grad\\phi(u^{j-1}), u^{j-1}-x}\n%\t\t\t\\\\\n%\t\t\t\\le & \\delta^1 + \\frac{\\beta D_X^2}{2} + \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j}\\Lambda^{j-1}\\left(\\sum_{i=1}^{j-1} \\frac{\\lambda^i}{\\Lambda^i}\\delta^i + \\frac{\\beta D_X^2(\\lambda^i)^2}{2\\Lambda^i}\\right) + \\frac{\\lambda^j}{\\Lambda^j}\\delta^j + \\frac{\\beta D_X^2(\\lambda^j)^2}{2\\Lambda^j}.\n%\t\\end{align}\n\\end{proof}","sha256_utf8":"79566cc6c60fb0a9ed1cdbcf98b3762c61d59e4f7480f9e3b368ed1d03b4718b"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"二次模型导出单步下降界","source_units":[{"start":69561,"end":70582,"start_line":742,"end_line":756,"raw_tex":"\\begin{proof}\n    Observing that the function $\\phi(x)$ in \\eqref{def:phi} is a strongly convex function with Lipschitz continuous (with constant $\\beta$) gradient, using the assumption \\eqref{eq:alpha_req}, and noting the definition of approximate solution $v^t$ in \\eqref{eq:cndgsubproblem}, we have\n    \\begin{align}\n    \t\\label{eq:phiConvex}\n    \t\\begin{aligned}\n        &\\phi(u^t) - (1-\\lambda^t)\\phi(u^{t-1}) - \\lambda^t(\\phi(u^{t-1}) + \\ip{\\grad \\phi(u^{t-1}),x-u^{t-1}})\n        \\\\\n        \\leq& \\phi((1-\\lambda^t)u^{t-1} + \\lambda^tv^t) - \\phi(u^{t-1}) - \\lambda^t\\ip{\\grad\\phi(u^{t-1}),x-u^{t-1}}\n        \\\\\n        \\le & \\lambda^t\\ip{\\grad \\phi(u^{t-1}), v^t - u^{t-1}} + \\frac{\\beta(\\lambda^t)^2}{2}\\norm{v^t-u^{t-1}}^2 - \\lambda^t\\ip{\\grad \\phi(u^{t-1}),x-u^{t-1}}\n        \\\\\n        =& \\lambda^t\\ip{\\grad\\phi(u^{t-1}),v^t-x} + \\frac{\\beta(\\lambda^t)^2}{2}\\norm{v^t-u^{t-1}}^2\\\\\n        \\leq& \\lambda^t\\delta^t +\\frac{\\beta D_X^2(\\lambda^t)^2}{2},\\ \\forall x\\in X, t\\ge 1.\n\t    \\end{aligned}\n    \\end{align}\n","sha256_utf8":"dd5206d2dd77bedbd045029130d312d54927381c6ca08f20ed8395c43ea9d9ef"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"加权累加内层最优性间隙","source_units":[{"start":70582,"end":71855,"start_line":757,"end_line":776,"raw_tex":"\tDefining $x^*:=\\operatorname{argmin}_{x\\in X}\\phi(x)$, from the above relation we have for any $t\\ge 2$ that\n\t\\begin{align}\n\t\t\\label{eq:tmp1}\n\t\t\\begin{aligned}\n\t\t& \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j} \\underset{x\\in X}{\\max}\\:\\ip{\\grad\\phi(u^{j-1}), u^{j-1}-x}\n\t\t\\\\\n\t\t\\leq& \\sum_{j=2}^t \\frac{1}{\\Lambda^j}[\\phi(u^{j-1}) - \\phi(x^*)] - \\frac{1}{\\Lambda^j}[\\phi(u^j) - \\phi(x^*)] + \\frac{\\lambda^j}{\\Lambda^j}\\delta^j + \\frac{\\beta D_X^2 (\\lambda^j)^2}{2\\Lambda^j}\n\t\t\\\\\n\t\t= & [\\phi(u^1) - \\phi(x^*)] - \\frac{1}{\\Lambda^t}[\\phi(u^t) - \\phi(x^*)] \n\t\t\\\\\n\t\t& + \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j}[\\phi(u^{j-1}) - \\phi(x^*)] + \\frac{\\lambda^j}{\\Lambda^j}\\delta^j + \\frac{\\beta D_X^2(\\lambda^j)^2}{2\\Lambda^j}\n\t\t\\\\\n\t\t\\le & [\\phi(u^1) - \\phi(x^*)] + \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j}[\\phi(u^{j-1}) - \\phi(x^*)] + \\frac{\\lambda^j}{\\Lambda^j}\\delta^j + \\frac{\\beta D_X^2(\\lambda^j)^2}{2\\Lambda^j}.\n\t\t\\end{aligned}\n\t\\end{align}\n\tHere in the equality we use the following observations from the definition of $\\Lambda^t$ in \\eqref{def:Lambda}: $1/\\Lambda^1 = 1$ and $1/\\Lambda^j = 1/\\Lambda^{j-1} + \\lambda^j/\\Lambda^j$ for all $j\\ge 2$, \n%\t\\begin{align}\n%\t\t\\frac{1}{\\Lambda^j} = \\frac{1}{\\Lambda^{j-1}} + \\frac{\\lambda^j}{\\Lambda^{j}},\\ \\forall j\\ge 2,\n%\t\\end{align}\n\n","sha256_utf8":"195935fe1fe25a3f29d7237fe824d1839b332e2c218bacff6cb15eacb17981be"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"建立二次目标误差递推","source_units":[{"start":71855,"end":72504,"start_line":777,"end_line":784,"raw_tex":"    To finish the proof it suffices to bound $\\phi(u^{j-1})-\\phi(x^*)$ for any $j\\ge 2$. Observing that $\\phi(x)$ in \\eqref{def:phi} is strongly convex and quadratic with\n    \\begin{equation}\n        \\frac{\\beta}{2}\\norm{x-u^{t-1}}^2 = \\phi(x) - (\\phi(u^{t-1})+\\ip{\\grad \\phi(u^{t-1}), x-u^{t-1}}),\\ \\forall x\\in X, t\\ge 1,\n    \\end{equation}\n    we have from \\eqref{eq:phiConvex} (with $x = x^*$) that\n    \\begin{equation}\\label{eq:telescopingBound}\n        [\\phi(u^t)-\\phi(x^*)] - (1-\\lambda^t)[\\phi(u^{t-1})-\\phi(x^*)] \\leq \\lambda^t\\delta^t + \\frac{\\beta D_X^2(\\lambda^t)^2}{2} - \\frac{\\beta\\lambda^t}{2}\\norm{x^*-u^{t-1}}^2.\n    \\end{equation}\n","sha256_utf8":"f727e69f7a42687209bca4a9c5829e331c844618f2d2e88a3be6390112fab901"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"求和误差递推并完成估计","source_units":[{"start":72504,"end":73402,"start_line":785,"end_line":795,"raw_tex":"    Applying Lemma \\ref{lem:tele} to the above recurrence relation and ignoring negative terms at the right hand side, we have\n    \\begin{align}\\label{eq:simplePhiBound}\n        \\phi(u^t) - \\phi(x^*) \\leq \\Lambda^t\\sum_{i=1}^t \\frac{\\lambda^i}{\\Lambda^i}\\delta^i + \\frac{\\beta D_X^2(\\lambda^i)^2}{2\\Lambda^i},\\ \\forall t\\ge 1.\n    \\end{align}\n\tWe conclude the lemma immediately by applying the above bound to \\eqref{eq:tmp1} and rearranging terms.\n%\t\\begin{align}\n%\t\t\t& \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j} \\underset{x\\in X}{\\max}\\:\\ip{\\grad\\phi(u^{j-1}), u^{j-1}-x}\n%\t\t\t\\\\\n%\t\t\t\\le & \\delta^1 + \\frac{\\beta D_X^2}{2} + \\sum_{j=2}^t \\frac{\\lambda^j}{\\Lambda^j}\\Lambda^{j-1}\\left(\\sum_{i=1}^{j-1} \\frac{\\lambda^i}{\\Lambda^i}\\delta^i + \\frac{\\beta D_X^2(\\lambda^i)^2}{2\\Lambda^i}\\right) + \\frac{\\lambda^j}{\\Lambda^j}\\delta^j + \\frac{\\beta D_X^2(\\lambda^j)^2}{2\\Lambda^j}.\n%\t\\end{align}\n\\end{proof}","sha256_utf8":"a4311b82670bec3d1b681b50157940e39d6a6299e034ef09e2166dfc3e65acfe"}]},{"id":"S026","semantic_type":"proposition","kind":"proposition","title":"内层停止保证与迭代复杂度","source_units":[{"start":73599,"end":74685,"start_line":799,"end_line":825,"raw_tex":"\\begin{proposition}\\label{prop:innerIterUCGS}\n\tIn the ACGM procedure, at termination we have\n\t\\begin{align}\n\t    \\label{eq:innerIterUCGS_OC}\n\t    \\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^+), u^+-x} \\le \\eta.\n\t\\end{align}\n\tMoreover, if $\\delta^t = \\sigma\\beta D_X^2/t$ for certain $\\sigma\\ge 0$ and $\\alpha^t$ is chosen such that \n\t\\begin{align}\n\t\t\\label{eq:alpha_req_2t1}\n\t\t\\phi(u^t) \\le \\phi\\left(\\frac{t-1}{t+1}u^{t-1} + \\frac{2}{t+1}v^t\\right),\n\t\\end{align}\n\tthen we have for any $t\\ge 1$ that\n    \\begin{align}\n        \\label{eq:termi_rate}\n        \\begin{aligned}\n        \\underset{j=1, \\dots, t+1}{\\min}\\:\\ip{\\grad \\phi(u^{j-1}), u^{j-1}-v^{j}}\\le & \\underset{j=1, \\dots, t+1}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x}\n        \\\\\n        \\le & \\frac{6(\\sigma+1)\\beta D_X^2}{t}.\n        \\end{aligned}\n    \\end{align}\n    Specially, it takes at most\n    \\begin{align}\n        \\label{def:T}\n        T := 1 + \\left\\lceil \\frac{(7\\sigma+6)\\beta D_X^2}{\\eta}\\right\\rceil \n    \\end{align}\n    iterations for the ACGM procedure to terminate.\n\\end{proposition}","sha256_utf8":"d985ae7f9ee055db54afb672269e0305ae5846deb6b05ca6f17bef3179010a07"}]},{"id":"S027","semantic_type":"proof","kind":"proof","title":"内层复杂度完整证明","source_units":[{"start":74687,"end":77877,"start_line":827,"end_line":873,"raw_tex":"\\begin{proof}\n    From the definition of the approximate solution $v^t$ in \\eqref{eq:cndgsubproblem}, if the termination criterion in \\eqref{eq:actual_termi} of the ACGM procedure is satisfied, then the output $u^+=u^{t-1}$ satisfies\n    \\begin{align}\n        \\max_{x\\in X}\\ip{\\grad \\phi(u^{t-1}), u^{t-1}-x} &= \\max_{x\\in X}\\ip{\\grad \\phi(u^{t-1}), u^{t-1}-v^t} + \\ip{\\grad \\phi(u^{t-1}), v^t - x}\n        \\\\\n        &\\le  (\\eta - \\delta^t) + \\delta^t = \\eta.\n    \\end{align}\n    Therefore \\eqref{eq:innerIterUCGS_OC} holds. To conclude the proposition it suffices to estimate the rate of convergence of $\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{t-1}), u^{t-1}-x}$. To analyze the rate, let us choose $\\lambda^t = 2/(t+1)$ and apply Lemma \\ref{lem:innerIterUCGS}. Then $\\Lambda^t = 2/(t(t+1))$ and\n    \\begin{align}\n        & \\sum_{j=2}^t j\\cdot\\:\\underset{x\\in X}{\\max}\\ip{\\phi(u^{j-1}),u^{j-1}-x}\n        \\\\\n%        \\leq& \\left[\\delta^1 + \\sum_{j=2}^t \\left(j\\delta^j + \\frac{2}{j-1}\\sum_{i=1}^{j-1} i \\delta^i\\right)\\right] + \\frac{\\beta D_X^2}{2}\\left[1+\\sum_{j=2}^t\\left( \\frac{2j}{j+1} + \\frac{2}{j-1}\\sum_{i=1}^{j-1} \\frac{2i}{i+1}\\right)\\right]\n        \\leq& \\sigma\\beta D_X^2\\left[1 + \\sum_{j=2}^t \\left(1 + \\frac{2}{j-1}\\sum_{i=1}^{j-1} 1\\right)\\right] + \\frac{\\beta D_X^2}{2}\\left[1+\\sum_{j=2}^t\\left( \\frac{2j}{j+1} + \\frac{2}{j-1}\\sum_{i=1}^{j-1} \\frac{2i}{i+1}\\right)\\right]\n        \\\\\n        < &  \\sigma\\beta D_X^2(3t-2)+ \\frac{\\beta D_X^2}{2}(6t-5),\\ \\forall t\\ge 2.\n    \\end{align}\n    Noting that $\\sum_{j=2}^t j = (t+2)(t-1)/2$, we have\n%    , after rearranging, we have \n%    \\begin{align}\n%        \\underset{j =2, \\dots, t}{\\min}\\:\\underset{x\\in X}{\\max}&\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x}\\\\\n%        \\leq& \\frac{2}{(t-1)(t+2)} \\left(\\sum_{j=2}^t j\\delta^j + \\frac{2}{j-1}\\sum_{i=1}^{j-1}i \\delta^i\\right) + \\frac{\\beta D_X^2(6t-5)}{(t-1)(t+2)}.\n%    \\end{align}\n%    A change of indexing allows us to conclude that\n\t\\begin{align}\n\t\t\\underset{j=2, \\dots, t}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x} \\le & \\frac{2}{(t+2)(t-1)}\\sum_{j=2}^t j\\cdot\\:\\underset{x\\in X}{\\max}\\ip{\\phi(u^{j-1}),u^{j-1}-x}\n\t\t\\\\\n%\t\t\\le & \\frac{2\\sigma\\beta D_X^2(3t-2)}{(t+2)(t-1)}+ \\frac{\\beta D_X^2(6t-5)}{(t+2)(t-1)}\n%\t\t\\\\\n%\t\t< & \\frac{6\\sigma\\beta D_X^2}{t-1}+ \\frac{6\\beta D_X^2}{t-1}\n%\t\t\\\\\n\t\t< & \\frac{6(\\sigma+1)\\beta D_X^2}{t-1}\n\t\t,\\ \\forall t\\ge 2.\n\t\\end{align}\n\tUsing the above result and observing that \n\t\\begin{align}\n\t\t\\underset{j=1, \\dots, t+1}{\\min}\\:\\ip{\\grad \\phi(u^{j-1}), u^{j-1}-v^{j}} \\le & \n\t\t\\underset{j=1, \\dots, t+1}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x}\n\t\t\\\\\n\t\t\\le & \n\t\t\\underset{j=2, \\dots, t+1}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x},\\ \\forall t\\ge 1\n\t\\end{align}\n\twe conclude \\eqref{eq:termi_rate}. Moreover, from \\eqref{eq:termi_rate} and noting the choice of $\\delta^t$, the termination criterion \\eqref{eq:actual_termi} holds whenever\n    \\begin{align}\n        \\frac{6(\\sigma+1)\\beta D_X^2}{t-1} + \\frac{\\sigma \\beta D_X^2}{t} \\le \\eta.\n    \\end{align}\n    Noting the definition of $T$ \\eqref{def:T}, the above condition clearly holds for all $t\\ge T$.\n\\end{proof}","sha256_utf8":"abbc01967f5fb3209b72d5996481cb0c74f06da423e33243eaed8f362935b264"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"近似线性解保证停止精度","source_units":[{"start":74687,"end":75164,"start_line":827,"end_line":833,"raw_tex":"\\begin{proof}\n    From the definition of the approximate solution $v^t$ in \\eqref{eq:cndgsubproblem}, if the termination criterion in \\eqref{eq:actual_termi} of the ACGM procedure is satisfied, then the output $u^+=u^{t-1}$ satisfies\n    \\begin{align}\n        \\max_{x\\in X}\\ip{\\grad \\phi(u^{t-1}), u^{t-1}-x} &= \\max_{x\\in X}\\ip{\\grad \\phi(u^{t-1}), u^{t-1}-v^t} + \\ip{\\grad \\phi(u^{t-1}), v^t - x}\n        \\\\\n        &\\le  (\\eta - \\delta^t) + \\delta^t = \\eta.\n    \\end{align}\n","sha256_utf8":"485594cd4a5ab5b6472175c6424a0378945c554b9e3c6347793d177e1bb46504"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"选取权重并估计加权间隙","source_units":[{"start":75164,"end":76192,"start_line":834,"end_line":842,"raw_tex":"    Therefore \\eqref{eq:innerIterUCGS_OC} holds. To conclude the proposition it suffices to estimate the rate of convergence of $\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{t-1}), u^{t-1}-x}$. To analyze the rate, let us choose $\\lambda^t = 2/(t+1)$ and apply Lemma \\ref{lem:innerIterUCGS}. Then $\\Lambda^t = 2/(t(t+1))$ and\n    \\begin{align}\n        & \\sum_{j=2}^t j\\cdot\\:\\underset{x\\in X}{\\max}\\ip{\\phi(u^{j-1}),u^{j-1}-x}\n        \\\\\n%        \\leq& \\left[\\delta^1 + \\sum_{j=2}^t \\left(j\\delta^j + \\frac{2}{j-1}\\sum_{i=1}^{j-1} i \\delta^i\\right)\\right] + \\frac{\\beta D_X^2}{2}\\left[1+\\sum_{j=2}^t\\left( \\frac{2j}{j+1} + \\frac{2}{j-1}\\sum_{i=1}^{j-1} \\frac{2i}{i+1}\\right)\\right]\n        \\leq& \\sigma\\beta D_X^2\\left[1 + \\sum_{j=2}^t \\left(1 + \\frac{2}{j-1}\\sum_{i=1}^{j-1} 1\\right)\\right] + \\frac{\\beta D_X^2}{2}\\left[1+\\sum_{j=2}^t\\left( \\frac{2j}{j+1} + \\frac{2}{j-1}\\sum_{i=1}^{j-1} \\frac{2i}{i+1}\\right)\\right]\n        \\\\\n        < &  \\sigma\\beta D_X^2(3t-2)+ \\frac{\\beta D_X^2}{2}(6t-5),\\ \\forall t\\ge 2.\n    \\end{align}\n","sha256_utf8":"d15d02554d0d931c7ee48d086aba921c2a2cb5d487f62ec0e9eaf919fe4151d6"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"由加权平均界控制最小间隙","source_units":[{"start":76192,"end":77091,"start_line":843,"end_line":859,"raw_tex":"    Noting that $\\sum_{j=2}^t j = (t+2)(t-1)/2$, we have\n%    , after rearranging, we have \n%    \\begin{align}\n%        \\underset{j =2, \\dots, t}{\\min}\\:\\underset{x\\in X}{\\max}&\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x}\\\\\n%        \\leq& \\frac{2}{(t-1)(t+2)} \\left(\\sum_{j=2}^t j\\delta^j + \\frac{2}{j-1}\\sum_{i=1}^{j-1}i \\delta^i\\right) + \\frac{\\beta D_X^2(6t-5)}{(t-1)(t+2)}.\n%    \\end{align}\n%    A change of indexing allows us to conclude that\n\t\\begin{align}\n\t\t\\underset{j=2, \\dots, t}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x} \\le & \\frac{2}{(t+2)(t-1)}\\sum_{j=2}^t j\\cdot\\:\\underset{x\\in X}{\\max}\\ip{\\phi(u^{j-1}),u^{j-1}-x}\n\t\t\\\\\n%\t\t\\le & \\frac{2\\sigma\\beta D_X^2(3t-2)}{(t+2)(t-1)}+ \\frac{\\beta D_X^2(6t-5)}{(t+2)(t-1)}\n%\t\t\\\\\n%\t\t< & \\frac{6\\sigma\\beta D_X^2}{t-1}+ \\frac{6\\beta D_X^2}{t-1}\n%\t\t\\\\\n\t\t< & \\frac{6(\\sigma+1)\\beta D_X^2}{t-1}\n\t\t,\\ \\forall t\\ge 2.\n\t\\end{align}\n","sha256_utf8":"0b4d328d97558a89e6836e8fbffcc280233c620589dc2c66cfd6b8524c0b54a6"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"转换索引得到间隙收敛率","source_units":[{"start":77091,"end":77473,"start_line":860,"end_line":867,"raw_tex":"\tUsing the above result and observing that \n\t\\begin{align}\n\t\t\\underset{j=1, \\dots, t+1}{\\min}\\:\\ip{\\grad \\phi(u^{j-1}), u^{j-1}-v^{j}} \\le & \n\t\t\\underset{j=1, \\dots, t+1}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x}\n\t\t\\\\\n\t\t\\le & \n\t\t\\underset{j=2, \\dots, t+1}{\\min}\\:\\underset{x\\in X}{\\max}\\: \\ip{\\grad \\phi(u^{j-1}), u^{j-1}-x},\\ \\forall t\\ge 1\n\t\\end{align}\n","sha256_utf8":"8bba5803b2adc9c9d10335391356385b8065dc0eb6670ace57e575a5013d22b5"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"结合线性误差推出停止次数","source_units":[{"start":77473,"end":77877,"start_line":868,"end_line":873,"raw_tex":"\twe conclude \\eqref{eq:termi_rate}. Moreover, from \\eqref{eq:termi_rate} and noting the choice of $\\delta^t$, the termination criterion \\eqref{eq:actual_termi} holds whenever\n    \\begin{align}\n        \\frac{6(\\sigma+1)\\beta D_X^2}{t-1} + \\frac{\\sigma \\beta D_X^2}{t} \\le \\eta.\n    \\end{align}\n    Noting the definition of $T$ \\eqref{def:T}, the above condition clearly holds for all $t\\ge T$.\n\\end{proof}","sha256_utf8":"ac09e61793802490d59922f3343fe5198e675bba066b92615b0eba0d560c0f4a"}]},{"id":"S033","semantic_type":"lemma","kind":"lemma","title":"霍尔德条件的近似二次上界","source_units":[{"start":79896,"end":80379,"start_line":887,"end_line":896,"raw_tex":"\\begin{lemma}\\label{lem:LkLB}\nFor any $\\delta >0$ and any $L$ such that\n\\begin{align}\n    L \\geq \\left( \\frac{1-\\nu}{1+\\nu}\\cdot\\frac{1}{\\tau}\\right)^{\\frac{1-\\nu}{1+\\nu}} M_\\nu ^ \\frac{2}{1+\\nu},\n\\end{align} \nwhere $\\nu$ and $M_\\nu$ are the H\\\"older continuity exponent and constant in \\eqref{def:holderCont}, we have\n\\begin{equation}\\label{eq:LkUB}\n    f(y)\\leq f(x) + \\ip{\\grad f(x), y-x} + \\frac{L}{2}\\norm{y-x}^2 + \\frac{\\tau}{2}, \\ \\forall x,y \\in X.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"1934c84b3ab21be3fc01a47eb23bdedafca16001dce93e73a4f64cca56376881"}]},{"id":"S034","semantic_type":"proof","kind":"proof","title":"引用标准二次上界引理","source_units":[{"start":80380,"end":80454,"start_line":897,"end_line":899,"raw_tex":"\\begin{proof}\n    See Lemma 1 of \\cite{nesterov2015universal}.\n\\end{proof}","sha256_utf8":"cc690004858fc9b612aa4202bac60d7242bf3ce591eca42ef791bcf5c0fea474"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"援引已有霍尔德模型结论","source_units":[{"start":80380,"end":80454,"start_line":897,"end_line":899,"raw_tex":"\\begin{proof}\n    See Lemma 1 of \\cite{nesterov2015universal}.\n\\end{proof}","sha256_utf8":"cc690004858fc9b612aa4202bac60d7242bf3ce591eca42ef791bcf5c0fea474"}]},{"id":"S036","semantic_type":"external_fact","kind":"external_fact","title":"标准霍尔德二次模型引理","source_units":[],"declared_external":true,"declaration_text":["Nesterov (2015), Lemma 1: under the Hölder upper-model condition with exponent ν∈(0,1] and constant Mν, for τ>0 and L≥[((1−ν)/(1+ν))/τ]^((1−ν)/(1+ν)) Mν^(2/(1+ν)), one has f(y)≤f(x)+〈∇f(x),y−x〉+(L/2)‖y−x‖²+τ/2 for all x,y∈X; at ν=1 the coefficient is interpreted by continuity."]},{"id":"S037","semantic_type":"corollary","kind":"corollary","title":"回溯曲率估计上界","source_units":[{"start":80721,"end":81034,"start_line":901,"end_line":906,"raw_tex":"\\begin{corollary}\\label{coro:LkLB}\n    Any $L_k>0$ chosen by Algorithm \\ref{alg:UCGS} according to \\eqref{def:lk} must also satisfy\n    \\begin{align}\n        L_k \\leq 2\\left(\\frac{1-\\nu}{1+\\nu}\\cdot\\frac{1}{\\varepsilon \\gamma_k}\\right)^{\\frac{1-\\nu}{1+\\nu}} M_\\nu^{\\frac{2}{1+\\nu}}\n    \\end{align}\n\\end{corollary}","sha256_utf8":"952e9aebe7a4a4a85f00c8a7c721e0b5280c9a4682b478468d996c37c31f08d9"}]},{"id":"S038","semantic_type":"proof","kind":"proof","title":"回溯曲率上界证明","source_units":[{"start":81035,"end":81462,"start_line":907,"end_line":909,"raw_tex":"\\begin{proof}\n    Suppose that $L_k$ does not satisfy \\eqref{eq:LkUB}.  Then applying Proposition \\ref{prop:flkDiff}  with $\\tau = \\varepsilon \\gamma_k$ implies that $L_k/2$ satisfies \\eqref{def:lk}, contradicting the fact that $L_k$ was chosen at step $k$ following the proposed backtracking linesearch implementation (see the remark on practical implementation after the description of Algorithm \\ref{alg:UCGS}). \n\\end{proof}","sha256_utf8":"101a15558559ae502cb11f322da2fb398570cb9a423598d24cbe0bb3b3a80a35"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"利用减半可行性反证曲率界","source_units":[{"start":81035,"end":81462,"start_line":907,"end_line":909,"raw_tex":"\\begin{proof}\n    Suppose that $L_k$ does not satisfy \\eqref{eq:LkUB}.  Then applying Proposition \\ref{prop:flkDiff}  with $\\tau = \\varepsilon \\gamma_k$ implies that $L_k/2$ satisfies \\eqref{def:lk}, contradicting the fact that $L_k$ was chosen at step $k$ following the proposed backtracking linesearch implementation (see the remark on practical implementation after the description of Algorithm \\ref{alg:UCGS}). \n\\end{proof}","sha256_utf8":"101a15558559ae502cb11f322da2fb398570cb9a423598d24cbe0bb3b3a80a35"}]},{"id":"S040","semantic_type":"lemma","kind":"lemma","title":"曲率与外层步长的衰减界","source_units":[{"start":82513,"end":83061,"start_line":917,"end_line":927,"raw_tex":"\\begin{lemma}\\label{lem:lkgammak}\n    Let $L_k>0$ be chosen by \\eqref{def:lk} of Algorithm \\ref{alg:UCGS} at step $k$, then\n    \\begin{align}\n        L_k\\gamma_k^2 \\leq \\frac{C_\\nu M_\\nu^{\\frac{2}{1+\\nu}}}{k^{\\frac{1+3\\nu}{1+\\nu}}\\varepsilon^{\\frac{1-\\nu}{1+\\nu}}} \n    \\end{align}\n    where \n    \\begin{align}\\label{eq:c}\n        C_\\nu := \\left( \\frac{1+2\\nu}{1+3\\nu}\\right)^{\\frac{1+3\\nu}{1+\\nu}}\\left(\\frac{1-\\nu}{1+\\nu}\\right)^\\frac{1-\\nu}{1+\\nu}2^{\\frac{4+10\\nu}{1+\\nu}} \n    \\end{align}\n    is a constant depending only on $\\nu$. \n\\end{lemma}","sha256_utf8":"e3d102a54bbc485f1b92083dd44e5514905e1c4138f146c2497da805145c8c8b"}]},{"id":"S041","semantic_type":"proof","kind":"proof","title":"曲率步长衰减完整证明","source_units":[{"start":83063,"end":89276,"start_line":929,"end_line":1001,"raw_tex":"\\begin{proof}\n    The case when $k=1$ is immediate from Corollary \\ref{coro:LkLB}. Therefore, throughout the proof we will assume that $k\\ge 2$.\n    Since we set $\\Gamma_k = L_k\\gamma_k^2/k$ in Algorithm \\ref{alg:UCGS}, we can prove this proposition by bounding $\\Gamma_k$. \n    Set $s:= (1+\\nu)/(1+3\\nu)$.  Since $\\nu \\in [0,1]$, we have $s\\in [{1}/{2},1]$.\n    % \\begin{align}\n    %     \\frac{1}{2} = \\frac{1}{3}\\left(1+\\frac{2}{1+3}\\right) \\leq \\frac{1}{3}\\left(1+\\frac{2}{1+3\\nu}\\right) = \\frac{1}{3}\\left(\\frac{3+3\\nu}{1+3\\nu}\\right) = s < 1.\n    % \\end{align}\n    We will study the quantity ${1}/{\\Gamma_k^s} - {1}/{\\Gamma_{k-1}^s}$, which can be rewritten as\n    \\begin{align}\n        \\frac{1}{\\Gamma_k^s} - \\frac{1}{\\Gamma_{k-1}^s} &= \\frac{\\left( \\frac{1}{\\Gamma_k^s} - \\frac{1}{\\Gamma_{k-1}^s}\\right)\\left( \\frac{1}{\\Gamma_k^{1-s}} + \\frac{1}{\\Gamma_{k-1}^{1-s}}\\right)}{ \\frac{1}{\\Gamma_k^{1-s}} + \\frac{1}{\\Gamma_{k-1}^{1-s}}}\\\\\n        &= \\frac{\\frac{1}{\\Gamma_k}-\\frac{1}{\\Gamma_{k-1}} - \\frac{1}{\\Gamma_k}\\left(\\frac{\\Gamma_k}{\\Gamma_{k-1}}\\right)^s + \\frac{1}{\\Gamma_{k-1}}\\left(\\frac{\\Gamma_{k-1}}{\\Gamma_k}\\right)^s}{\\frac{1}{\\Gamma_k^{1-s}} + \\frac{1}{\\Gamma_{k-1}^{1-s}}}.\n    \\end{align}\n    Here, noting from the relation of $\\Gamma_k$ and $\\Gamma_{k-1}$ in \\eqref{eq:GammaRelation} that $\\Gamma_k \\leq \\Gamma_{k-1}$ and recalling that $s \\in [1/2,1]$, we can make two observations. First, we have $\\Gamma_k^{2s-1} \\leq \\Gamma_{k-1}^{2s-1}$, and hence\n    \\begin{align}\n        % \\Gamma_k^{2s-1} &\\leq \\Gamma_{k-1}^{2s-1}\\\\\n        % \\frac{\\Gamma_k^s}{\\Gamma_{k-1}^{s-1}} &\\leq \\frac{\\Gamma_{k-1}^s}{\\Gamma_k^{s-1}}\\\\\n        % \\frac{1}{\\Gamma_{k}}\\left(\\frac{\\Gamma_k}{\\Gamma_{k-1}}\\right)^s &\\leq \\frac{1}{\\Gamma_{k-1}}\\left(\\frac{\\Gamma_{k-1}}{\\Gamma_k}\\right)^s\n    % \\end{align}\n    % and consequently, \n% \n    % \\begin{align}\n        - \\frac{1}{\\Gamma_k}\\left(\\frac{\\Gamma_k}{\\Gamma_{k-1}}\\right)^s + \\frac{1}{\\Gamma_{k-1}}\\left(\\frac{\\Gamma_{k-1}}{\\Gamma_k}\\right)^s \\geq 0.\n    \\end{align} \n    Second, we have $\\Gamma_{k-1}^{1-s}\\le \\Gamma_k^{1-s}$, and hence\n    \\begin{align}\n        \\frac{1}{\\Gamma_k^{1-s}} + \\frac{1}{\\Gamma_{k-1}^{1-s}}\\leq \\frac{2}{\\Gamma_{k-1}^{1-s}}.\n    \\end{align}\n    Combining the above two observations and recalling that $s=(1+\\nu)/(1+3\\nu)$ and the relations concerning $\\Gamma_k$ and $\\Gamma_{k-1}$ in  \\eqref{eq:GammaRelation}, we have that\n    \\begin{align}\n        \\label{eq:tmp2}\n        \\frac{1}{\\Gamma_k^s} - \\frac{1}{\\Gamma_{k-1}^s} \\geq \\frac{\\frac{1}{\\Gamma_k}-\\frac{1}{\\Gamma_{k-1}}}{\\frac{2}{\\Gamma_k^{1-s}}}\\nonumber\n        = \\frac{\\frac{\\gamma_k}{\\Gamma_k}}{\\frac{2}{\\Gamma_k^{1-s}}}\\label{eq:gammaRelation}\n        = \\frac{\\gamma_k}{2}\\Gamma_k^{-\\frac{1+\\nu}{1+3\\nu}}\\nonumber.\n    \\end{align}\n    We can further bound the last expression in the above relation.  Indeed, recalling that $\\Gamma_k = L_k\\gamma_k^2/k$ and applying Corollary \\ref{coro:LkLB}, we have the inequality\n    \\begin{align}\n        \\frac{\\gamma_k^2}{k\\Gamma_k} = \\frac{1}{L_k} \\geq \\frac{1}{2M_\\nu^{\\frac{2}{1+\\nu}}}\\left(\\frac{1+\\nu}{1-\\nu}\\cdot\\varepsilon \\gamma_k\\right)^{\\frac{1-\\nu}{1+\\nu}}.\n    \\end{align}\n    In the above recall that we can use a continuity argument for the $\\nu=1$ case since $(1-\\nu)^{-(1-\\nu)}\\to 1$ as $\\nu\\to 1$. Rearranging terms in the above relation, we have\n    \\begin{align}\n        \\gamma_k\\Gamma_k^{-\\frac{1+\\nu}{1+3\\nu}} \\geq \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu}\\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}k^\\frac{1+\\nu}{1+3\\nu}}{2^\\frac{1+\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}}.\n    \\end{align}\n    Applying the above bound to \\eqref{eq:tmp2}, it follows that\n    \\begin{align}\n        \\frac{1}{\\Gamma_k^s} - \\frac{1}{\\Gamma_{k-1}^s} \\geq \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu} \\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}k^\\frac{1+\\nu}{1+3\\nu}}{2^\\frac{2+4\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}}.\n    \\end{align}\n    Summing the above from $i = 2$ to $k$ and using the fact that\n    \\begin{align}\n        \\sum_{i=2}^k i^{\\frac{1+\\nu}{1+3\\nu}} \\ge \\int_1^k u^{\\frac{1+\\nu}{1+3\\nu}}du = \\frac{1+3\\nu}{2+4\\nu}\\cdot\\left(k^{\\frac{2+4\\nu}{1+3\\nu}}-1\\right) \\ge \\frac{1+3\\nu}{4+8\\nu}k^{\\frac{2+4\\nu}{1+3\\nu}},\\ \\forall k\\ge 2\n    \\end{align}\n    and the definition of $C_\\nu$ in \\eqref{eq:c}, we obtain\n    \\begin{align}\n        \\frac{1}{\\Gamma_k^s} \\geq \\frac{1}{\\Gamma_k^s} - \\frac{1}{\\Gamma_{1}^s} \n        &\n        \\geq \n        % \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu}\\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}}{2^\\frac{2+4\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}}\\frac{1+3\\nu}{4+8\\nu}k^{\\frac{2+4\\nu}{1+3\\nu}}\n        % \\\\\n        % = & \n        \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu}\\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}}{2^\\frac{4+10\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}}\\frac{1+3\\nu}{1+2\\nu}k^{\\frac{2+4\\nu}{1+3\\nu}}\n    \\end{align}\n    Recalling that $s=(1+\\nu)/(1+3\\nu)$ and $\\Gamma_k = L_k\\gamma_k^2/k$, we conclude the proposition immediately from the above result.\n    % \\begin{align}\n    %     \\frac{L_k\\gamma_k^2}{k} \\le \\left(\\frac{1-\\nu}{1+\\nu}\\right)^\\frac{1-\\nu}{1+\\nu}\\frac{2^\\frac{4+10\\nu}{1+\\nu}M_\\nu^\\frac{2}{1+\\nu}}{\\varepsilon^\\frac{1-\\nu}{1+\\nu}}\\left(\\frac{1+2\\nu}{1+3\\nu}\\right)^{\\frac{1+3\\nu}{1+\\nu}}k^{-\\frac{2+4\\nu}{1+\\nu}}\n    % \\end{align}\n    % \n    % Noting that for $k \\geq 2$, $k^{\\frac{2+4\\nu}{1+3\\nu}}-1 \\geq \\frac{1}{2}k^{\\frac{2+4\\nu}{1+3\\nu}}$, we continue with \n    % \\begin{align}\n    %     \\frac{1}{\\Gamma_k^s} &\\geq \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu}\\left(\\frac{1+3\\nu}{2+4\\nu}\\right)\\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}(k^{\\frac{2+4\\nu}{1+3\\nu}}-1)}{2^\\frac{2+4\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}} \\\\\n    %     &\\geq \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu}\\left(\\frac{1+3\\nu}{2+4\\nu}\\right)\\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}k^{\\frac{2+4\\nu}{1+3\\nu}}}{2^\\frac{3+7\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}}\\\\\n    %     &= \\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}k^{\\frac{2+4\\nu}{1+3\\nu}}}{M_\\nu^\\frac{2}{1+3\\nu} C_\\nu^\\frac{1+\\nu}{1+3\\nu}}.\n    % \\end{align}\n    % Solving for $\\Gamma_k$ above, recalling our definition of $s$, and noting the relationship in \\eqref{def:Gammak}, we obtain the desired result. \n\\end{proof}","sha256_utf8":"2f0917a1a30150b12d65d92ec253bc1678ce93fd2951e22dbe973d76a045a189"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"处理初始情形并改写权重增量","source_units":[{"start":83063,"end":84272,"start_line":929,"end_line":940,"raw_tex":"\\begin{proof}\n    The case when $k=1$ is immediate from Corollary \\ref{coro:LkLB}. Therefore, throughout the proof we will assume that $k\\ge 2$.\n    Since we set $\\Gamma_k = L_k\\gamma_k^2/k$ in Algorithm \\ref{alg:UCGS}, we can prove this proposition by bounding $\\Gamma_k$. \n    Set $s:= (1+\\nu)/(1+3\\nu)$.  Since $\\nu \\in [0,1]$, we have $s\\in [{1}/{2},1]$.\n    % \\begin{align}\n    %     \\frac{1}{2} = \\frac{1}{3}\\left(1+\\frac{2}{1+3}\\right) \\leq \\frac{1}{3}\\left(1+\\frac{2}{1+3\\nu}\\right) = \\frac{1}{3}\\left(\\frac{3+3\\nu}{1+3\\nu}\\right) = s < 1.\n    % \\end{align}\n    We will study the quantity ${1}/{\\Gamma_k^s} - {1}/{\\Gamma_{k-1}^s}$, which can be rewritten as\n    \\begin{align}\n        \\frac{1}{\\Gamma_k^s} - \\frac{1}{\\Gamma_{k-1}^s} &= \\frac{\\left( \\frac{1}{\\Gamma_k^s} - \\frac{1}{\\Gamma_{k-1}^s}\\right)\\left( \\frac{1}{\\Gamma_k^{1-s}} + \\frac{1}{\\Gamma_{k-1}^{1-s}}\\right)}{ \\frac{1}{\\Gamma_k^{1-s}} + \\frac{1}{\\Gamma_{k-1}^{1-s}}}\\\\\n        &= \\frac{\\frac{1}{\\Gamma_k}-\\frac{1}{\\Gamma_{k-1}} - \\frac{1}{\\Gamma_k}\\left(\\frac{\\Gamma_k}{\\Gamma_{k-1}}\\right)^s + \\frac{1}{\\Gamma_{k-1}}\\left(\\frac{\\Gamma_{k-1}}{\\Gamma_k}\\right)^s}{\\frac{1}{\\Gamma_k^{1-s}} + \\frac{1}{\\Gamma_{k-1}^{1-s}}}.\n    \\end{align}\n","sha256_utf8":"f1211b40db7310248304171b9773284a82e510c58eb4b310a5552e39760f2f34"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"估计增量分子的交叉项","source_units":[{"start":84272,"end":85084,"start_line":941,"end_line":951,"raw_tex":"    Here, noting from the relation of $\\Gamma_k$ and $\\Gamma_{k-1}$ in \\eqref{eq:GammaRelation} that $\\Gamma_k \\leq \\Gamma_{k-1}$ and recalling that $s \\in [1/2,1]$, we can make two observations. First, we have $\\Gamma_k^{2s-1} \\leq \\Gamma_{k-1}^{2s-1}$, and hence\n    \\begin{align}\n        % \\Gamma_k^{2s-1} &\\leq \\Gamma_{k-1}^{2s-1}\\\\\n        % \\frac{\\Gamma_k^s}{\\Gamma_{k-1}^{s-1}} &\\leq \\frac{\\Gamma_{k-1}^s}{\\Gamma_k^{s-1}}\\\\\n        % \\frac{1}{\\Gamma_{k}}\\left(\\frac{\\Gamma_k}{\\Gamma_{k-1}}\\right)^s &\\leq \\frac{1}{\\Gamma_{k-1}}\\left(\\frac{\\Gamma_{k-1}}{\\Gamma_k}\\right)^s\n    % \\end{align}\n    % and consequently, \n% \n    % \\begin{align}\n        - \\frac{1}{\\Gamma_k}\\left(\\frac{\\Gamma_k}{\\Gamma_{k-1}}\\right)^s + \\frac{1}{\\Gamma_{k-1}}\\left(\\frac{\\Gamma_{k-1}}{\\Gamma_k}\\right)^s \\geq 0.\n    \\end{align} \n","sha256_utf8":"290eabe3507c3530f980f3de933fb00322b7d9531fd872f49addb0d4eb6a4231"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"估计增量表达式的分母","source_units":[{"start":85084,"end":85286,"start_line":952,"end_line":955,"raw_tex":"    Second, we have $\\Gamma_{k-1}^{1-s}\\le \\Gamma_k^{1-s}$, and hence\n    \\begin{align}\n        \\frac{1}{\\Gamma_k^{1-s}} + \\frac{1}{\\Gamma_{k-1}^{1-s}}\\leq \\frac{2}{\\Gamma_{k-1}^{1-s}}.\n    \\end{align}\n","sha256_utf8":"00fce673f089a43f3b84347a5ed15aa46316e06df7d40ce9b385a18d2052c5a3"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"导出归一化权重增量下界","source_units":[{"start":85286,"end":85836,"start_line":956,"end_line":962,"raw_tex":"    Combining the above two observations and recalling that $s=(1+\\nu)/(1+3\\nu)$ and the relations concerning $\\Gamma_k$ and $\\Gamma_{k-1}$ in  \\eqref{eq:GammaRelation}, we have that\n    \\begin{align}\n        \\label{eq:tmp2}\n        \\frac{1}{\\Gamma_k^s} - \\frac{1}{\\Gamma_{k-1}^s} \\geq \\frac{\\frac{1}{\\Gamma_k}-\\frac{1}{\\Gamma_{k-1}}}{\\frac{2}{\\Gamma_k^{1-s}}}\\nonumber\n        = \\frac{\\frac{\\gamma_k}{\\Gamma_k}}{\\frac{2}{\\Gamma_k^{1-s}}}\\label{eq:gammaRelation}\n        = \\frac{\\gamma_k}{2}\\Gamma_k^{-\\frac{1+\\nu}{1+3\\nu}}\\nonumber.\n    \\end{align}\n","sha256_utf8":"22625dd12613d89509738dfb0f5319a7a1090e38089c36397902f68e8ff9e121"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"用回溯曲率界消去自适应参数","source_units":[{"start":85836,"end":86656,"start_line":963,"end_line":970,"raw_tex":"    We can further bound the last expression in the above relation.  Indeed, recalling that $\\Gamma_k = L_k\\gamma_k^2/k$ and applying Corollary \\ref{coro:LkLB}, we have the inequality\n    \\begin{align}\n        \\frac{\\gamma_k^2}{k\\Gamma_k} = \\frac{1}{L_k} \\geq \\frac{1}{2M_\\nu^{\\frac{2}{1+\\nu}}}\\left(\\frac{1+\\nu}{1-\\nu}\\cdot\\varepsilon \\gamma_k\\right)^{\\frac{1-\\nu}{1+\\nu}}.\n    \\end{align}\n    In the above recall that we can use a continuity argument for the $\\nu=1$ case since $(1-\\nu)^{-(1-\\nu)}\\to 1$ as $\\nu\\to 1$. Rearranging terms in the above relation, we have\n    \\begin{align}\n        \\gamma_k\\Gamma_k^{-\\frac{1+\\nu}{1+3\\nu}} \\geq \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu}\\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}k^\\frac{1+\\nu}{1+3\\nu}}{2^\\frac{1+\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}}.\n    \\end{align}\n","sha256_utf8":"4a23f17097354864c38a91894ffd42329d6ca42c758d8c5c0ab15dde9c80590f"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"累加增量并作积分比较","source_units":[{"start":86656,"end":87303,"start_line":971,"end_line":978,"raw_tex":"    Applying the above bound to \\eqref{eq:tmp2}, it follows that\n    \\begin{align}\n        \\frac{1}{\\Gamma_k^s} - \\frac{1}{\\Gamma_{k-1}^s} \\geq \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu} \\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}k^\\frac{1+\\nu}{1+3\\nu}}{2^\\frac{2+4\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}}.\n    \\end{align}\n    Summing the above from $i = 2$ to $k$ and using the fact that\n    \\begin{align}\n        \\sum_{i=2}^k i^{\\frac{1+\\nu}{1+3\\nu}} \\ge \\int_1^k u^{\\frac{1+\\nu}{1+3\\nu}}du = \\frac{1+3\\nu}{2+4\\nu}\\cdot\\left(k^{\\frac{2+4\\nu}{1+3\\nu}}-1\\right) \\ge \\frac{1+3\\nu}{4+8\\nu}k^{\\frac{2+4\\nu}{1+3\\nu}},\\ \\forall k\\ge 2\n    \\end{align}\n","sha256_utf8":"e885f707a847fa6154c4a2c4e40f9beb2b033de27ff39f7139ed5332af7510fa"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"整理常数得到曲率步长界","source_units":[{"start":87303,"end":89276,"start_line":979,"end_line":1001,"raw_tex":"    and the definition of $C_\\nu$ in \\eqref{eq:c}, we obtain\n    \\begin{align}\n        \\frac{1}{\\Gamma_k^s} \\geq \\frac{1}{\\Gamma_k^s} - \\frac{1}{\\Gamma_{1}^s} \n        &\n        \\geq \n        % \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu}\\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}}{2^\\frac{2+4\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}}\\frac{1+3\\nu}{4+8\\nu}k^{\\frac{2+4\\nu}{1+3\\nu}}\n        % \\\\\n        % = & \n        \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu}\\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}}{2^\\frac{4+10\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}}\\frac{1+3\\nu}{1+2\\nu}k^{\\frac{2+4\\nu}{1+3\\nu}}\n    \\end{align}\n    Recalling that $s=(1+\\nu)/(1+3\\nu)$ and $\\Gamma_k = L_k\\gamma_k^2/k$, we conclude the proposition immediately from the above result.\n    % \\begin{align}\n    %     \\frac{L_k\\gamma_k^2}{k} \\le \\left(\\frac{1-\\nu}{1+\\nu}\\right)^\\frac{1-\\nu}{1+\\nu}\\frac{2^\\frac{4+10\\nu}{1+\\nu}M_\\nu^\\frac{2}{1+\\nu}}{\\varepsilon^\\frac{1-\\nu}{1+\\nu}}\\left(\\frac{1+2\\nu}{1+3\\nu}\\right)^{\\frac{1+3\\nu}{1+\\nu}}k^{-\\frac{2+4\\nu}{1+\\nu}}\n    % \\end{align}\n    % \n    % Noting that for $k \\geq 2$, $k^{\\frac{2+4\\nu}{1+3\\nu}}-1 \\geq \\frac{1}{2}k^{\\frac{2+4\\nu}{1+3\\nu}}$, we continue with \n    % \\begin{align}\n    %     \\frac{1}{\\Gamma_k^s} &\\geq \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu}\\left(\\frac{1+3\\nu}{2+4\\nu}\\right)\\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}(k^{\\frac{2+4\\nu}{1+3\\nu}}-1)}{2^\\frac{2+4\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}} \\\\\n    %     &\\geq \\left(\\frac{1+\\nu}{1-\\nu}\\right)^\\frac{1-\\nu}{1+3\\nu}\\left(\\frac{1+3\\nu}{2+4\\nu}\\right)\\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}k^{\\frac{2+4\\nu}{1+3\\nu}}}{2^\\frac{3+7\\nu}{1+3\\nu}M_\\nu^\\frac{2}{1+3\\nu}}\\\\\n    %     &= \\frac{\\varepsilon^\\frac{1-\\nu}{1+3\\nu}k^{\\frac{2+4\\nu}{1+3\\nu}}}{M_\\nu^\\frac{2}{1+3\\nu} C_\\nu^\\frac{1+\\nu}{1+3\\nu}}.\n    % \\end{align}\n    % Solving for $\\Gamma_k$ above, recalling our definition of $s$, and noting the relationship in \\eqref{def:Gammak}, we obtain the desired result. \n\\end{proof}","sha256_utf8":"4707656bdacef2425dedea2785ec322db8bfb83a1eed3ca1f039a49e2aa46b62"}]},{"id":"S049","semantic_type":"proposition","kind":"proposition","title":"外层目标与下模型差估计","source_units":[{"start":89945,"end":90389,"start_line":1005,"end_line":1010,"raw_tex":"\\begin{proposition}\\label{prop:flkDiff}\n    Suppose that the parameters in Algorithm \\ref{alg:UCGS} satisfy $\\beta_k \\geq L_k \\gamma_k$ for all $k$. Then for any $x\\in X$, \n    \\begin{align}\n        f(y_k) - \\ell_k(x) \\leq \\frac{\\varepsilon}{2} + \\Gamma_k \\sum_{i=1}^k \\frac{\\gamma_i\\beta_i}{2\\Gamma_i}\\left(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2\\right)^2 + \\Gamma_k \\sum_{i=1}^k \\frac{\\gamma_i\\eta_i}{\\Gamma_i}.\n    \\end{align} \n\\end{proposition}","sha256_utf8":"39c8849535ba8935c0e12cb03a037278b36f6ff328bb2c4953909e88f3bbb788"}]},{"id":"S050","semantic_type":"proof","kind":"proof","title":"外层模型差完整证明","source_units":[{"start":90391,"end":93167,"start_line":1012,"end_line":1047,"raw_tex":"\\begin{proof}\n    Fix any $x\\in X$. From the definitions of $\\ell_k(x)$ and $y_k$ in \\eqref{def:lk} and \\eqref{def:yk} respectively, we have\n    \\begin{align}\n        \\frac{1}{\\Gamma_k}\\ell_k(x) =& \\sum_{i=1}^k \\frac{1}{\\Gamma_i}\\left( \\gamma_i f(z_i) + \\gamma_i \\ip{\\grad f(z_i),x-x_i} + \\ip{\\grad f(z_i),\\gamma_i(x_i-z_i)}\\right)\\\\\n        =& \\sum_{i=1}^k \\frac{1}{\\Gamma_i}\\left( f(z_i) + \\ip{\\grad f(z_i), y_i-z_i}\\right)+ \\frac{\\gamma_i}{\\Gamma_i} \\ip{\\grad f(z_i),x-x_i}\n        \\\\\n        &\\qquad- \\frac{1-\\gamma_i}{\\Gamma_i}(f(z_i) + \\ip{\\grad f(z_i), y_{i-1}-z_i})\n    \\end{align}\n    We will now bound three terms in the above relation.  First, by convexity of $f$, \n    \\begin{align}\n        -(f(z_i) + \\ip{\\grad f(z_i),y_{i-1}-z_i}) \\ge -f(y_{i-1}).\n    \\end{align}\n    Second, by our choice of $L_k$ in \\eqref{def:lk} and the definitions of $y_k$ and $z_k$ in \\eqref{def:yk} and \\eqref{def:zk} respectively, we have\n    \\begin{align}\n        f(z_i) + \\ip{\\grad f(z_i),y_i-z_i} \\ge & f(y_i) - \\frac{L_i}{2}\\norm{y_i-z_i}^2 - \\frac{\\varepsilon}{2}\\gamma_i \n        \\\\\n        = & f(y_i) - \\frac{L_i\\gamma_i^2}{2}\\norm{x_i-x_{i-1}}^2 - \\frac{\\varepsilon}{2}\\gamma_i.\n    \\end{align}\n    Lastly, using the result \\eqref{eq:innerIterUCGS_OC} in Lemma \\ref{lem:innerIterUCGS} and noting the definition of $\\phi(x)$ in \\eqref{def:phi}, we obtain the following result during the termination of the ACGM procedure in computing $x_i$:\n    \\begin{align}\n         \\ip{\\grad f(z_i), x-x_i} \\geq&\\: \\beta_i\\ip{x_i-x_{i-1},x_i-x} - \\eta_i\\\\\n        =& -\\frac{\\beta_i}{2}\\left(\\norm{x-x_{i-1}}^2 - \\norm{x_i-x_{i-1}}^2 - \\norm{x-x_i}^2\\right) - \\eta_i\\\\\n        \\geq& -\\frac{\\beta_i}{2}\\left(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2\\right) -\\eta_i + \\frac{L_i^2}{2}\\norm{x_i-x_{i-1}}^2.\n    \\end{align}\n    In the last inequality above we use our assumption that $\\beta_k\\geq L_k\\gamma_k$ for all $k$. Based on the above three observations and rearranging terms we obtain that\n    \\begin{align}\n        \\frac{1}{\\Gamma_k}\\ell_k(x) \\geq& \\sum_{i=1}^k \\frac{1}{\\Gamma_i}\\left( f(y_i) - (1-\\gamma_i)f(y_{i-1}) - \\frac{\\gamma_i\\beta_i}{2}(\\norm{x-x_{i-1}}^2-\\norm{x-x_i}^2)\\right)\n        \\\\\n        &\\qquad -\\frac{1}{\\Gamma_i}\\left(\\frac{\\varepsilon}{2}\\gamma_i + \\gamma_i\\eta_i\\right)\n        \\\\\n        = & \n        \\frac{f(y_k)}{\\Gamma_k} -\n        \\sum_{i=1}^k \\frac{\\gamma_i\\beta_i}{2\\Gamma_i}(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2) -\\sum_{i=1}^k \\frac{\\gamma_i\\eta_i}{\\Gamma_i} - \\frac{\\varepsilon}{2\\Gamma_k}.\n    \\end{align}\n    Here in the last equality we use the relations \\eqref{eq:GammaRelation} and \\eqref{eq:gammaSum} and the fact that $\\gamma_1=1$ in its definition \\eqref{def:gammak}. We conclude the result by multiplying by $\\Gamma_k$ and rearranging  terms.\n\\end{proof}","sha256_utf8":"7258156cff2cfbccfdbbf72e3c08188bd07e3e86d13891873ff9ff78bb844435"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"展开加权线性下模型","source_units":[{"start":90391,"end":90981,"start_line":1012,"end_line":1019,"raw_tex":"\\begin{proof}\n    Fix any $x\\in X$. From the definitions of $\\ell_k(x)$ and $y_k$ in \\eqref{def:lk} and \\eqref{def:yk} respectively, we have\n    \\begin{align}\n        \\frac{1}{\\Gamma_k}\\ell_k(x) =& \\sum_{i=1}^k \\frac{1}{\\Gamma_i}\\left( \\gamma_i f(z_i) + \\gamma_i \\ip{\\grad f(z_i),x-x_i} + \\ip{\\grad f(z_i),\\gamma_i(x_i-z_i)}\\right)\\\\\n        =& \\sum_{i=1}^k \\frac{1}{\\Gamma_i}\\left( f(z_i) + \\ip{\\grad f(z_i), y_i-z_i}\\right)+ \\frac{\\gamma_i}{\\Gamma_i} \\ip{\\grad f(z_i),x-x_i}\n        \\\\\n        &\\qquad- \\frac{1-\\gamma_i}{\\Gamma_i}(f(z_i) + \\ip{\\grad f(z_i), y_{i-1}-z_i})\n    \\end{align}\n","sha256_utf8":"90824917c8b9582cabeb0341ea02355797f120c1fce116e703cd3b51758f32f5"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"凸性控制上一轮线性项","source_units":[{"start":90981,"end":91169,"start_line":1020,"end_line":1023,"raw_tex":"    We will now bound three terms in the above relation.  First, by convexity of $f$, \n    \\begin{align}\n        -(f(z_i) + \\ip{\\grad f(z_i),y_{i-1}-z_i}) \\ge -f(y_{i-1}).\n    \\end{align}\n","sha256_utf8":"d5bfc158cea9f95848ab6bb621203b6a4344256342d3007a8f016d5146a18c97"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"外层线搜索控制目标上模型","source_units":[{"start":91169,"end":91584,"start_line":1024,"end_line":1029,"raw_tex":"    Second, by our choice of $L_k$ in \\eqref{def:lk} and the definitions of $y_k$ and $z_k$ in \\eqref{def:yk} and \\eqref{def:zk} respectively, we have\n    \\begin{align}\n        f(z_i) + \\ip{\\grad f(z_i),y_i-z_i} \\ge & f(y_i) - \\frac{L_i}{2}\\norm{y_i-z_i}^2 - \\frac{\\varepsilon}{2}\\gamma_i \n        \\\\\n        = & f(y_i) - \\frac{L_i\\gamma_i^2}{2}\\norm{x_i-x_{i-1}}^2 - \\frac{\\varepsilon}{2}\\gamma_i.\n    \\end{align}\n","sha256_utf8":"2c90c4d8b15c6eca1821a17393e312058af88a7775e85325b0a8618f404d6e83"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"内层停止条件控制投影误差","source_units":[{"start":91584,"end":92186,"start_line":1030,"end_line":1035,"raw_tex":"    Lastly, using the result \\eqref{eq:innerIterUCGS_OC} in Lemma \\ref{lem:innerIterUCGS} and noting the definition of $\\phi(x)$ in \\eqref{def:phi}, we obtain the following result during the termination of the ACGM procedure in computing $x_i$:\n    \\begin{align}\n         \\ip{\\grad f(z_i), x-x_i} \\geq&\\: \\beta_i\\ip{x_i-x_{i-1},x_i-x} - \\eta_i\\\\\n        =& -\\frac{\\beta_i}{2}\\left(\\norm{x-x_{i-1}}^2 - \\norm{x_i-x_{i-1}}^2 - \\norm{x-x_i}^2\\right) - \\eta_i\\\\\n        \\geq& -\\frac{\\beta_i}{2}\\left(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2\\right) -\\eta_i + \\frac{L_i^2}{2}\\norm{x_i-x_{i-1}}^2.\n    \\end{align}\n","sha256_utf8":"8affe4622fc12f79f483db11ebb25ff6b365adeee37190d3471f3feefcec1e93"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"合并三项并望远镜求和","source_units":[{"start":92186,"end":93167,"start_line":1036,"end_line":1047,"raw_tex":"    In the last inequality above we use our assumption that $\\beta_k\\geq L_k\\gamma_k$ for all $k$. Based on the above three observations and rearranging terms we obtain that\n    \\begin{align}\n        \\frac{1}{\\Gamma_k}\\ell_k(x) \\geq& \\sum_{i=1}^k \\frac{1}{\\Gamma_i}\\left( f(y_i) - (1-\\gamma_i)f(y_{i-1}) - \\frac{\\gamma_i\\beta_i}{2}(\\norm{x-x_{i-1}}^2-\\norm{x-x_i}^2)\\right)\n        \\\\\n        &\\qquad -\\frac{1}{\\Gamma_i}\\left(\\frac{\\varepsilon}{2}\\gamma_i + \\gamma_i\\eta_i\\right)\n        \\\\\n        = & \n        \\frac{f(y_k)}{\\Gamma_k} -\n        \\sum_{i=1}^k \\frac{\\gamma_i\\beta_i}{2\\Gamma_i}(\\norm{x-x_{i-1}}^2 - \\norm{x-x_i}^2) -\\sum_{i=1}^k \\frac{\\gamma_i\\eta_i}{\\Gamma_i} - \\frac{\\varepsilon}{2\\Gamma_k}.\n    \\end{align}\n    Here in the last equality we use the relations \\eqref{eq:GammaRelation} and \\eqref{eq:gammaSum} and the fact that $\\gamma_1=1$ in its definition \\eqref{def:gammak}. We conclude the result by multiplying by $\\Gamma_k$ and rearranging  terms.\n\\end{proof}","sha256_utf8":"c5d416a2229e1c112140cf54a803ad9ebacef8f68550fb25a6bef0d4a8c4b8d7"}]},{"id":"S056","semantic_type":"proof","kind":"proof","title":"回溯规则与停止证书论证","source_units":[{"start":65053,"end":66513,"start_line":679,"end_line":687,"raw_tex":"We proposed to search for such $L_k$ through a backtracking linesearch strategy. In particular, we initialize with any $L_0 \\in \\R$ and choose $L_1 = 2^i L_0$ where $i$ is the smallest integer such that \\eqref{def:lk} is satisfied. At the start of the $k$-th outer iteration where $k>1$, we set $L_k = L_{k-1}/2$ and assess the validity of $L_k$. If it does not satisfy $\\eqref{def:lk}$, we keep backtracking and replacing $L_k$ to $2L_k$ until \\eqref{def:lk} is satisfies. Through this backtracking linesearch strategy, we ensure that our choice of $L_k$ is adaptive and that performance is independent of the choice of $L_0$. Previous literature \\cite{nazari2020backtracking} on backtracking linesearch strategy of CGS require monotonicity of $L_k$ and may suffer from a poorly chosen $L_0$. Second, our termination criterion is based on \\eqref{eq:stopCrit}. We can observe immediately that if the parameter $\\varepsilon_k\\equiv 0$, i.e., $s_k$ is the exact solution to problem \\eqref{def:lkproblem}, then when \\eqref{eq:stopCrit} is satisfied, $y_k$ will be $\\varepsilon$-approximation solution to problem \\eqref{problem:CP}.  To see this, note from \\eqref{eq:GammaRelation} that\n    \\begin{equation}\\label{eq:gammaSum}\n        \\Gamma_k \\sum_{i=1}^k \\frac{\\gamma_i}{\\Gamma_i} = 1\n    \\end{equation}\n    and consequently\n    % \\begin{equation}\\label{eq:lkBound}\n    $$\n        f(y_k) - f^* \\le f(y_k) - \\min_{x\\in X}\\ell_k(x) =  f(y_k) - \\ell_k(s_k).\n    $$\n","sha256_utf8":"d10cfb3592cb78573b68bcb6add6e04a10a1d54d99206f1b3362ca0df6d46297"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"说明回溯规则及权重归一性","source_units":[{"start":65053,"end":66355,"start_line":679,"end_line":682,"raw_tex":"We proposed to search for such $L_k$ through a backtracking linesearch strategy. In particular, we initialize with any $L_0 \\in \\R$ and choose $L_1 = 2^i L_0$ where $i$ is the smallest integer such that \\eqref{def:lk} is satisfied. At the start of the $k$-th outer iteration where $k>1$, we set $L_k = L_{k-1}/2$ and assess the validity of $L_k$. If it does not satisfy $\\eqref{def:lk}$, we keep backtracking and replacing $L_k$ to $2L_k$ until \\eqref{def:lk} is satisfies. Through this backtracking linesearch strategy, we ensure that our choice of $L_k$ is adaptive and that performance is independent of the choice of $L_0$. Previous literature \\cite{nazari2020backtracking} on backtracking linesearch strategy of CGS require monotonicity of $L_k$ and may suffer from a poorly chosen $L_0$. Second, our termination criterion is based on \\eqref{eq:stopCrit}. We can observe immediately that if the parameter $\\varepsilon_k\\equiv 0$, i.e., $s_k$ is the exact solution to problem \\eqref{def:lkproblem}, then when \\eqref{eq:stopCrit} is satisfied, $y_k$ will be $\\varepsilon$-approximation solution to problem \\eqref{problem:CP}.  To see this, note from \\eqref{eq:GammaRelation} that\n    \\begin{equation}\\label{eq:gammaSum}\n        \\Gamma_k \\sum_{i=1}^k \\frac{\\gamma_i}{\\Gamma_i} = 1\n    \\end{equation}\n","sha256_utf8":"0482312d33657490e22945571bb17d913dfbeb4084330ac6fe829f3ceba2b0fe"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"线性下模型给出停止证书","source_units":[{"start":66355,"end":66513,"start_line":683,"end_line":687,"raw_tex":"    and consequently\n    % \\begin{equation}\\label{eq:lkBound}\n    $$\n        f(y_k) - f^* \\le f(y_k) - \\min_{x\\in X}\\ell_k(x) =  f(y_k) - \\ell_k(s_k).\n    $$\n","sha256_utf8":"ebf1e81b06730823bda79844a900fe31c2fc8c875971c603b04c16c8eef71b0f"}]},{"id":"S059","semantic_type":"assumption","kind":"assumption","title":"内层比较序列与下降假设","source_units":[{"start":68185,"end":68486,"start_line":715,"end_line":719,"raw_tex":"    Suppose that $\\{\\lambda^t\\}\\in [0,1]$ is any predetermined sequence satisfying $\\lambda^1= 1$. In the ACGM procedure, if $\\alpha^t$ is chosen such that \n    \\begin{align}\n        \\label{eq:alpha_req}\n        \\phi(u^t)\\le \\phi((1-\\lambda^t)u^{t-1} + \\lambda^tv^t),\\ \\forall t\\ge 1,\n    \\end{align}\n","sha256_utf8":"f7cb863f458b9dcee74fbb27f0407b07b08891824a31e9d0a9f0556efb7d5e0b"}]},{"id":"S060","semantic_type":"assumption","kind":"assumption","title":"内层误差衰减与步长假设","source_units":[{"start":73822,"end":74062,"start_line":805,"end_line":809,"raw_tex":"\tMoreover, if $\\delta^t = \\sigma\\beta D_X^2/t$ for certain $\\sigma\\ge 0$ and $\\alpha^t$ is chosen such that \n\t\\begin{align}\n\t\t\\label{eq:alpha_req_2t1}\n\t\t\\phi(u^t) \\le \\phi\\left(\\frac{t-1}{t+1}u^{t-1} + \\frac{2}{t+1}v^t\\right),\n\t\\end{align}\n","sha256_utf8":"f4f3b7f2ce8a1607ae2fe4307a25cffb82f12638848c84a3089118b460a78167"}]},{"id":"S061","semantic_type":"assumption","kind":"assumption","title":"外层正则系数支配曲率","source_units":[{"start":89985,"end":90118,"start_line":1006,"end_line":1006,"raw_tex":"    Suppose that the parameters in Algorithm \\ref{alg:UCGS} satisfy $\\beta_k \\geq L_k \\gamma_k$ for all $k$. Then for any $x\\in X$, \n","sha256_utf8":"6f247e5b6eb2c847f1835ed8a1c41be3a8661d53edf79be593e4a5657ec7dad3"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S007","edges":[{"source_id":"S008","target_id":"S010","kind":"supports"},{"source_id":"S003","target_id":"S010","kind":"supports"},{"source_id":"S049","target_id":"S010","kind":"supports"},{"source_id":"S058","target_id":"S010","kind":"supports"},{"source_id":"S001","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S003","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S008","target_id":"S013","kind":"supports"},{"source_id":"S040","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S004","target_id":"S015","kind":"supports"},{"source_id":"S026","target_id":"S015","kind":"supports"},{"source_id":"S008","target_id":"S015","kind":"supports"},{"source_id":"S003","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S013","target_id":"S016","kind":"supports"},{"source_id":"S003","target_id":"S016","kind":"supports"},{"source_id":"S010","target_id":"S009","kind":"supports"},{"source_id":"S011","target_id":"S009","kind":"supports"},{"source_id":"S012","target_id":"S009","kind":"supports"},{"source_id":"S013","target_id":"S009","kind":"supports"},{"source_id":"S014","target_id":"S009","kind":"supports"},{"source_id":"S015","target_id":"S009","kind":"supports"},{"source_id":"S016","target_id":"S009","kind":"supports"},{"source_id":"S009","target_id":"S007","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S003","target_id":"S022","kind":"supports"},{"source_id":"S001","target_id":"S022","kind":"supports"},{"source_id":"S059","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S059","target_id":"S023","kind":"supports"},{"source_id":"S003","target_id":"S024","kind":"supports"},{"source_id":"S022","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S017","target_id":"S025","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S023","target_id":"S021","kind":"supports"},{"source_id":"S024","target_id":"S021","kind":"supports"},{"source_id":"S025","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S003","target_id":"S028","kind":"supports"},{"source_id":"S020","target_id":"S029","kind":"supports"},{"source_id":"S060","target_id":"S029","kind":"supports"},{"source_id":"S001","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S003","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S060","target_id":"S032","kind":"supports"},{"source_id":"S003","target_id":"S032","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S027","kind":"supports"},{"source_id":"S031","target_id":"S027","kind":"supports"},{"source_id":"S032","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S002","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S039","kind":"supports"},{"source_id":"S057","target_id":"S039","kind":"supports"},{"source_id":"S003","target_id":"S039","kind":"supports"},{"source_id":"S006","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S042","kind":"supports"},{"source_id":"S003","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S005","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S044","kind":"supports"},{"source_id":"S005","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S005","target_id":"S045","kind":"supports"},{"source_id":"S037","target_id":"S046","kind":"supports"},{"source_id":"S003","target_id":"S046","kind":"supports"},{"source_id":"S006","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S003","target_id":"S048","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S043","target_id":"S041","kind":"supports"},{"source_id":"S044","target_id":"S041","kind":"supports"},{"source_id":"S045","target_id":"S041","kind":"supports"},{"source_id":"S046","target_id":"S041","kind":"supports"},{"source_id":"S047","target_id":"S041","kind":"supports"},{"source_id":"S048","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S003","target_id":"S051","kind":"supports"},{"source_id":"S001","target_id":"S052","kind":"supports"},{"source_id":"S003","target_id":"S053","kind":"supports"},{"source_id":"S026","target_id":"S054","kind":"supports"},{"source_id":"S003","target_id":"S054","kind":"supports"},{"source_id":"S061","target_id":"S054","kind":"supports"},{"source_id":"S051","target_id":"S055","kind":"supports"},{"source_id":"S052","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S005","target_id":"S055","kind":"supports"},{"source_id":"S057","target_id":"S055","kind":"supports"},{"source_id":"S003","target_id":"S055","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S050","kind":"supports"},{"source_id":"S054","target_id":"S050","kind":"supports"},{"source_id":"S055","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S005","target_id":"S057","kind":"supports"},{"source_id":"S003","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S001","target_id":"S058","kind":"supports"},{"source_id":"S003","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S058","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S010","kind":"supports"}]}}
{"record_id":"2103.14987v4","paper":{"title":"Quadratic Convergence of Smoothing Newton's Method for 0/1 Loss Optimization","authors":["Shenglong Zhou","Lili Pan","Naihua Xiu","Houduo Qi"],"arxiv_id":"2103.14987","version":"v4","journal":"SIAM Journal on Optimization","doi":"10.1137/21m1409445","abstract_url":"https://arxiv.org/abs/2103.14987v4","pdf_url":"https://arxiv.org/pdf/2103.14987v4"},"context":{"raw_tex":"This paper is concerned with the $0/1$-loss optimization:\n\\begin{equation}\\label{ell-0-1}\n\\underset{\\bfx\\in\\R^n}{\\min}~ f(\\bfx)+\\lambda \\|( A\\bfx +\\bfb )_+\\|_0, \n\\end{equation}\n\nwhere $f:\\R^n\\rightarrow\\R$ is  twice continuously differentiable, \n$\\lambda>0 $ is a penalty parameter and $A\\in \\R^{m\\times n}, \\bfb\\in \\R^m$. \nMoreover, ${\\bfz}_+:=((z_1)_+, \\ldots, (z_m)_+)^\\top $ with $z_+:=\\max\\{z,{0}\\}$ \nand $\\|\\bfz\\|_0$ is the $\\ell_0$ norm of $\\bfz$, counting the number of its non-zero entries. \nHence, $\\|\\bfz_+\\|_0$ counts the number of positive entries of $\\bfz$,\ni.e., $\\|\\bfz_+\\|_0=\\sum_{i=1}^m \\ell_{0/1}(z_i)$, where\n\\begin{eqnarray*}\n\\ell_{0/1}(z)=\n\\begin{cases}\n1, &z>0,\\\\\n0& z\\leq 0.\n\\end{cases}\n\\end{eqnarray*}\n\nGiven a subset $T\\subseteq\\N_m:=\\{1,2,\\ldots,m\\}$, its cardinality and complementary set are $|T|$ and $\\overline T$.   The neighbourhood of $\\bfx\\in\\R^n$ with a radius $\\delta>0$  is denoted by $N( \\bfx,\\delta)=\\{\\bfv\\in\\R^n: \\|\\bfv- \\bfx \\|< \\delta\\}$. \nMoreover, $\\bfx_{T}$ (resp. $A_{T}$) represents the sub-vector (resp. sub-matrix) contains elements (resp. rows) of $\\bfx$ indexed on $T$.   Particularly, $A_i$ is the $i$th row of $A$. \nWe combine two vectors as $(\\bfx;\\bfy):=(\\bfx^\\top~\\bfy^\\top)^\\top$. The $i$th   largest singular value of $H\\in\\R^{n\\times n}$ is written $\\sigma_{i}(H)$, namely $\\sigma_{1}(H)\\geq\\sigma_{2}(H)\\geq\\cdots\\geq\\sigma_{n}(H).$\nParticularly, we write $\\|H\\|:=\\sigma_{1}(H)$ and $\\sigma_{\\min}(H):=\\sigma_{n}(H).$ Finally, let $I$ be the identity matrix and $\\bfe$ be the vector with all entries being ones. \n\n\nNext we describe the formula for computing the subdifferential of \n$\\|\\bfz_+\\|_0$ and its proximal operator.\nWe note that $\\|\\cdot\\|_0$ is lower semi-continuous (lsc)\nand $\\bfz_+$ is obviously continuous, \nthe composition $\\|\\bfz_+\\|_0$ is also lsc.\nFor a proper and lsc function $g: \\R^m \\mapsto \\R$, its\nsubdifferential $\\partial g(\\cdot)$ is well defined as in \\cite[Definition 8.3]{RW1998}. \n\n\nThe assumption in (\\ref{chain-rule-cond}) can be regarded as a constraint qualification for the chain rule in (\\ref{chain-rule-sub}) to hold. We finish this section with a formula to compute the proximal operator of  $\\|(\\cdot)_+\\|_0$.  Let $\\alpha>0$, the proximal operator of $\\alpha \\|(\\cdot)_+\\|_0$ at  {$\\bfo$ is defined by \n\\begin{eqnarray*} %\\label{proximal-operator-0}\n\\prox(\\bfo)=\\underset{\\bfy\\in\\R^m}{\\rm argmin} ~\\frac{1}{2}\\|\\bfy-\\bfo\\|^2+ \\alpha \\|\\bfy_+\\|_0.\n\\end{eqnarray*}\n\nAs shown in \\cite[Lemma 2.2]{wang2019support}, the proximal operator admits a closed form as\n\n\n\\begin{eqnarray} \\label{proximal-operator}\n\\Big[\\prox(\\bfo)\\Big]_i=\n%&=&\n%\\begin{cases}\n%0,&    \\nu_i \\in [0, \\sqrt{2\\alpha}),\\\\\n%0~ \\text{or}~ \\nu_i ,& \\nu_i=\\sqrt{2\\alpha},\\\\\n%\\nu_i,&\\nu_i\\in(-\\infty,0)\\cup (\\sqrt{2\\alpha},\\infty),\n%\\end{cases} \\nonumber\\\\\n%&=&\n\\begin{cases}\n0,&    \\nu_i \\in (0, \\sqrt{2\\alpha}),\\\\\n0~ \\text{or}~ \\nu_i ,& \\nu_i\\in\\{0,\\sqrt{2\\alpha}\\},\\\\\n\\nu_i,&\\nu_i\\in(-\\infty,0)\\cup (\\sqrt{2\\alpha},\\infty).\n\\end{cases} \n\\end{eqnarray}\n\n\\begin{eqnarray}\\label{T-*}\n  \\Gamma_*&:=&\\{i\\in\\N_m: A_i\\bfx^*+b_i=0\\}.\n\\end{eqnarray}\n\n\\begin{definition}\\label{def-p-sta} \n\tA point $\\bfx^*$ is called a P-stationary point of the problem (\\ref{ell-0-1})  if there exist a constant $\\tau > 0$ and\n\ta point $\\bfz^*\\in\\R^m$  such that\n\\begin{eqnarray} \\label{p-stationary}\n \\begin{cases}\n\\hspace{21mm}\\nabla f(\\bfx^*)  + A^\\top \\bfz^* &=~ 0 \\\\\n\\pro (A\\bfx^*+\\bfb+\\tau \\bfz^*) &\\ni~A\\bfx^*+\\bfb.\n\\end{cases}\n\\end{eqnarray}\n\\end{definition}\n\nWe also say a point $(\\bfx^*;\\bfz^*)$ is a P-stationary point of the problem (\\ref{ell-0-1}) if it satisfies the conditions in \\eqref{p-stationary}. \nFor a point $\\bfx^*$, we denote two constants by \n\n\n\\begin{eqnarray}\n\\label{tau-*} \n~~~~\\tau_1 := \n\\begin{cases}\n+\\infty, & \\bfy^*\\leq0,\\\\\n{\\min} \\left\\{\\frac{(y^*_i)^2}{2\\lambda}:y_i^*>0\\right\\},& {\\rm otherwise},\\\\\n\\end{cases}~~\n\\tau_2:=\\begin{cases}\n+\\infty, & \\Gamma_*=\\emptyset,\\\\\n \\frac{2\\lambda}{\\max_i|p^*_i|^2},& {\\rm otherwise},\n\\end{cases}\n\\end{eqnarray}\n\nwhere $\\bfy^*:=A\\bfx^*+\\bfb$ and $\\bfp^*:=-(A_{\\Gamma_*}A_{\\Gamma_*}^{\\top})^{-1}A_{\\Gamma_*}\\nabla f(  \\bfx^*)$. Clearly, both $\\tau_1>0$ and $\\tau_2>0$. Based on these notation, we have the following main result of this section.\n\n\n\\begin{assumption} \\label{ass}\n\t Suppose $f$ is twice continuously differentiable, $\\nabla^2 f(\\bfx^*)$ is {positive definite} and $A_{\\Gamma_*}$ is full row rank, where $\\Gamma_*$ is given by (\\ref{T-*}).\n \\end{assumption}\n\n\\begin{assumption} \\label{ass2} \nSuppose $\\nabla^2 f$ is locally Lipschitz continuous around $\\bfx^*$ with a constant $L_*>0$, namely \n\\begin{eqnarray*} \\|\\nabla^2 f(\\bfx)-\\nabla^2 f(\\bfx')\\|\\leq L_* \\|   \\bfx -\\bfx' \\| \\end{eqnarray*}\nfor any $\\bfx$ and $\\bfx'$ in the neighbourhood of $\\bfx^*$.\n \\end{assumption}\n\n For a point  $\\bfw:=(\\bfx; \\bfz)$, we define the sets {\n\\begin{eqnarray}\\label{p-stationary-T}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n \\begin{array}{rcll}\n\\S&:=& \\left\\{i\\in\\N_m:  A_i\\bfx+b_i+\\tau z_i   \\in (0, \\sqrt{2\\tau\\lambda}) \\right\\}, \\\\\n \\E&:=& \\left\\{i\\in\\N_m: A_i\\bfx+b_i +\\tau z_i \\in\\{0,\\sqrt{2\\tau\\lambda}\\} \\right\\},\\\\\n\\O&:=& \\left\\{i\\in\\N_m:  A_i\\bfx+b_i+ \\tau z_i \\in(-\\infty,0)\\cup ( \\sqrt{2\\tau\\lambda},\\infty) \\right\\},\\\\\n\\E^o&:=& \\left\\{i\\in\\N_m: A_i\\bfx+b_i =0, \\tau z_i   \\in\\{0,\\sqrt{2\\tau\\lambda}\\} \\right\\}, \n\\end{array}\n\\end{eqnarray}\n\nfor a given $\\tau>0$. \n%The first three sets stand for index sets on which the value $|A_i\\bfx+b_i+\\tau z_i-\\sqrt{\\tau\\lambda/2}|$ is smaller than, equal to and over than $\\sqrt{\\tau\\lambda/2}$, respectively. \nObviously, $\\E^o\\subseteq \\E.$ \nIt is worth mentioning that all sets depend on $\\bfw$. \nFor simplicity, we drop their dependence whenever \nthere is no confusion to be caused. \nSame rules are also applied into $\\S_*,\\E_*,\\O_*$ and $\\E_*^o$ for  $\\bfw^*:=(\\bfx^*; \\bfz^*)$. \n\n\n\\begin{eqnarray}\n\\label{sta-eq-1}\n\\hspace{8mm} F(\\bfw; \\Gamma) =\\left[\\begin{array}{rrl}\n\\nabla f(\\bfx)  + A_{\\Gamma}^\\top \\bfz_{\\Gamma}  \\\\\nA_{\\Gamma} \\bfx + \\bfb_{\\Gamma} \\\\\n\\bfz_{\\overline \\Gamma} \n\\end{array}\n\\right] = 0, \\quad \n%\\label{def-Jacobian}\n\\nabla F_{\\mu}(\\bfw;\\Gamma) =\\left[\\begin{array}{ccc}\n\\nabla^2 f(\\bfx)   & A_{\\Gamma} ^\\top &0\\\\\nA_{\\Gamma} &-\\mu I&0 \\\\\n0&0&I\n\\end{array}\n\\right].\n\\end{eqnarray}\n\nWe note that the matrix $\\nabla F_{\\mu}$ is a slight perturbation of\nthe Jacobian matrix of $F$ and \n$\\nabla F(\\bfw;\\Gamma)=\\nabla F_{0}(\\bfw;\\Gamma)$,\n\n\nLet $\\bfw^k:=(\\bfx^k; \\bfz^k)$ be the current iterate.\nWe define $\\S_k$ and $\\E_k^o$ by \\eqref{p-stationary-T} with $\\bfw$ being replaced by $\\bfw^k$ and let  \n\n\n\\beq \\label{T-k}\n      T_k := \\S_k\\cup \\E_k^o.  \n \\eeq\n\nLet $\\bfd^k=(\\bfu^k;\\bfv^k)$ with $\\bfu^k\\in\\R^n$ and $\\bfv^k\\in\\R^{m}$. For \nsuch a defined $T_k$, a Newton direction $\\bfd^k$ for the equation \\eqref{sta-eq-1} solves the following linear equations:\n\n\n\\[ %\\beq \\label{newton-d}\n \\nabla F(\\bfw^k;T_k)~\\bfd = - F(\\bfw^k;T_k). \n \\]\n\nTo improve the nonsingularity of the Jacobian matrix $\\nabla F(\\bfw^k;T_k)$,\nwe replace it with $\\nabla F_{\\mu_k}(\\bfw^k;T_k)$. \nThat is, at $\\bfw^k$, we solve the equation:\n\n\n\\beq \\label{newton-dir-0}\n \\nabla F_{\\mu_k}(\\bfw^k;T_k)~\\bfd = - F(\\bfw^k;T_k), \n \\eeq\n\nwhere $\\nabla F_{\\mu_k}(\\bfw^k;T_k)$ is  defined in \\eqref{sta-eq-1}.\nThe Newton direction $\\bfd^k$ satisfies \n\n\n\\beq \\label{newton-dir} \n \\left[  \n     \\begin{array}{ccc}\n     {\\nabla^2  f(\\bfx^k)}& A_{T_k}^\\top&0\\\\\n   A_{T_k}&-\\mu_k I&0\\\\\n    0&0&I\n     \\end{array}\n     \\right]  \n     \\left[  \n     \\begin{array}{l}\n    \\bfu^k\\\\\n    \\bfv^k_{T_k}\\\\\n    \\bfv^k_{\\overline T_k}\n     \\end{array}\n     \\right]\n     =-\\left[  \n     \\begin{array}{r}\n    \\nabla  f(\\bfx^k)+A_{T_k}^\\top\\bfz^k_{T_k}\\\\\n   A_{T_k}\\bfx^k+\\bfb_{T_k} \\\\\n    \\bfz^k_{\\overline T_k}\n     \\end{array}\n     \\right].\n \\eeq\n\nHere, the rule to update $\\mu_k$ is as follows:\n\n\n\\beq \\label{update-mu} \n\\mu_{k}=\n\\min\\{\\alpha \\mu_{k-1}, \\rho\\|F(\\bfw^k;T_k)\\|\\},\n \\eeq\n\n where $\\alpha\\in(0,1)$ and $\\rho>0$. Now we summarize   the proposed method in  \\cref{Alg-NM01}. \n\\begin{algorithm}[H]\n\t\\caption{\\nhs: Newton's method for $0/1$-loss optimization}\n\t\\begin{algorithmic}[1] \\label{Alg-NM01}\n\t\t%\\scriptsize\n\t\t\\STATE Initialize $\\bfw^0=(\\bfx^0;\\bfz^0)$ and $\\mu_{-1}>0$. Set the parameter $\\tau,\\lambda,\\rho>0, \\alpha\\in(0,1)$.\\\\\n\t\t Compute $T_0$ by \\eqref{T-k} and set $k :=0$.\n\\IF{ $\\|F(\\bfw^k;T_k)\\|>0$}\t\t \n         \\STATE  Update $\\mu_{k}$ by \\eqref{update-mu}.\t\n\t\t\\STATE  Update $\\bfd^k$ by solving \\eqref{newton-dir}.\n\t\t\\STATE  Update $\\bfw^{k+1}~=~\\bfw^k+\\bfd^k$.\n\t\t\\STATE  Update $T_{k+1}$ by \\eqref{T-k} and  set $k := k+1$.   \t\t\n\\ENDIF\t\t\n\\RETURN $\\bfw^{k}$.\n\t\\end{algorithmic}\n\\end{algorithm}\n\nThe first technical result is about extending the stationarity equation to some indices that are given in a neighborhood of $\\bfw^*$. In the proof, we recall $\\bfy = A\\bfx+\\bfb$ and $\\bfy^* = A\\bfx^*+\\bfb$. \n","units":[{"start":5999,"end":6175,"start_line":179,"end_line":182,"raw_tex":"This paper is concerned with the $0/1$-loss optimization:\n\\begin{equation}\\label{ell-0-1}\n\\underset{\\bfx\\in\\R^n}{\\min}~ f(\\bfx)+\\lambda \\|( A\\bfx +\\bfb )_+\\|_0, \n\\end{equation}","sha256_utf8":"ba110f3ca8c4f2298ab52251fd0bc97bf516e9d287a9fcdfeaf088758a064885"},{"start":6177,"end":6725,"start_line":183,"end_line":195,"raw_tex":"where $f:\\R^n\\rightarrow\\R$ is  twice continuously differentiable, \n$\\lambda>0 $ is a penalty parameter and $A\\in \\R^{m\\times n}, \\bfb\\in \\R^m$. \nMoreover, ${\\bfz}_+:=((z_1)_+, \\ldots, (z_m)_+)^\\top $ with $z_+:=\\max\\{z,{0}\\}$ \nand $\\|\\bfz\\|_0$ is the $\\ell_0$ norm of $\\bfz$, counting the number of its non-zero entries. \nHence, $\\|\\bfz_+\\|_0$ counts the number of positive entries of $\\bfz$,\ni.e., $\\|\\bfz_+\\|_0=\\sum_{i=1}^m \\ell_{0/1}(z_i)$, where\n\\begin{eqnarray*}\n\\ell_{0/1}(z)=\n\\begin{cases}\n1, &z>0,\\\\\n0& z\\leq 0.\n\\end{cases}\n\\end{eqnarray*}","sha256_utf8":"b82baa1c8440987bca2ce1aa2b9b4532f9a793e6aa12d01935c31ae5c30a0f29"},{"start":17269,"end":18116,"start_line":306,"end_line":309,"raw_tex":"Given a subset $T\\subseteq\\N_m:=\\{1,2,\\ldots,m\\}$, its cardinality and complementary set are $|T|$ and $\\overline T$.   The neighbourhood of $\\bfx\\in\\R^n$ with a radius $\\delta>0$  is denoted by $N( \\bfx,\\delta)=\\{\\bfv\\in\\R^n: \\|\\bfv- \\bfx \\|< \\delta\\}$. \nMoreover, $\\bfx_{T}$ (resp. $A_{T}$) represents the sub-vector (resp. sub-matrix) contains elements (resp. rows) of $\\bfx$ indexed on $T$.   Particularly, $A_i$ is the $i$th row of $A$. \nWe combine two vectors as $(\\bfx;\\bfy):=(\\bfx^\\top~\\bfy^\\top)^\\top$. The $i$th   largest singular value of $H\\in\\R^{n\\times n}$ is written $\\sigma_{i}(H)$, namely $\\sigma_{1}(H)\\geq\\sigma_{2}(H)\\geq\\cdots\\geq\\sigma_{n}(H).$\nParticularly, we write $\\|H\\|:=\\sigma_{1}(H)$ and $\\sigma_{\\min}(H):=\\sigma_{n}(H).$ Finally, let $I$ be the identity matrix and $\\bfe$ be the vector with all entries being ones. \n","sha256_utf8":"af3af743d0258c4e088252f05dce341101b2fd03df7b1266a18cd069ce917085"},{"start":18117,"end":18513,"start_line":311,"end_line":317,"raw_tex":"Next we describe the formula for computing the subdifferential of \n$\\|\\bfz_+\\|_0$ and its proximal operator.\nWe note that $\\|\\cdot\\|_0$ is lower semi-continuous (lsc)\nand $\\bfz_+$ is obviously continuous, \nthe composition $\\|\\bfz_+\\|_0$ is also lsc.\nFor a proper and lsc function $g: \\R^m \\mapsto \\R$, its\nsubdifferential $\\partial g(\\cdot)$ is well defined as in \\cite[Definition 8.3]{RW1998}. \n","sha256_utf8":"d8472fd2fd20251991103462b7fbbbeb2023bc56a0b3fbb1602c249d962e837a"},{"start":22236,"end":22725,"start_line":386,"end_line":389,"raw_tex":"The assumption in (\\ref{chain-rule-cond}) can be regarded as a constraint qualification for the chain rule in (\\ref{chain-rule-sub}) to hold. We finish this section with a formula to compute the proximal operator of  $\\|(\\cdot)_+\\|_0$.  Let $\\alpha>0$, the proximal operator of $\\alpha \\|(\\cdot)_+\\|_0$ at  {$\\bfo$ is defined by \n\\begin{eqnarray*} %\\label{proximal-operator-0}\n\\prox(\\bfo)=\\underset{\\bfy\\in\\R^m}{\\rm argmin} ~\\frac{1}{2}\\|\\bfy-\\bfo\\|^2+ \\alpha \\|\\bfy_+\\|_0.\n\\end{eqnarray*}","sha256_utf8":"4b11f676da20d1519638110ea4bb53f85a694437d8e09ad03b159c18b0fecfbc"},{"start":22726,"end":22819,"start_line":390,"end_line":390,"raw_tex":"As shown in \\cite[Lemma 2.2]{wang2019support}, the proximal operator admits a closed form as\n","sha256_utf8":"71d047baba76d0fd440bfbf734865f631521f2fdf85759ff2d9c62e8bc6a9e7d"},{"start":23089,"end":23545,"start_line":399,"end_line":413,"raw_tex":"\\begin{eqnarray} \\label{proximal-operator}\n\\Big[\\prox(\\bfo)\\Big]_i=\n%&=&\n%\\begin{cases}\n%0,&    \\nu_i \\in [0, \\sqrt{2\\alpha}),\\\\\n%0~ \\text{or}~ \\nu_i ,& \\nu_i=\\sqrt{2\\alpha},\\\\\n%\\nu_i,&\\nu_i\\in(-\\infty,0)\\cup (\\sqrt{2\\alpha},\\infty),\n%\\end{cases} \\nonumber\\\\\n%&=&\n\\begin{cases}\n0,&    \\nu_i \\in (0, \\sqrt{2\\alpha}),\\\\\n0~ \\text{or}~ \\nu_i ,& \\nu_i\\in\\{0,\\sqrt{2\\alpha}\\},\\\\\n\\nu_i,&\\nu_i\\in(-\\infty,0)\\cup (\\sqrt{2\\alpha},\\infty).\n\\end{cases} \n\\end{eqnarray}","sha256_utf8":"8cc880e1ce567fdc38b3cbc36dd6d5fcb0e638a92bb6734e6b61da5bab5e3d91"},{"start":23971,"end":24058,"start_line":418,"end_line":420,"raw_tex":"\\begin{eqnarray}\\label{T-*}\n  \\Gamma_*&:=&\\{i\\in\\N_m: A_i\\bfx^*+b_i=0\\}.\n\\end{eqnarray}","sha256_utf8":"4aa14cb36359027d6f7e14e5c9cbb2b014d351b2f199ac9569827932b13b8d86"},{"start":31124,"end":31522,"start_line":535,"end_line":544,"raw_tex":"\\begin{definition}\\label{def-p-sta} \n\tA point $\\bfx^*$ is called a P-stationary point of the problem (\\ref{ell-0-1})  if there exist a constant $\\tau > 0$ and\n\ta point $\\bfz^*\\in\\R^m$  such that\n\\begin{eqnarray} \\label{p-stationary}\n \\begin{cases}\n\\hspace{21mm}\\nabla f(\\bfx^*)  + A^\\top \\bfz^* &=~ 0 \\\\\n\\pro (A\\bfx^*+\\bfb+\\tau \\bfz^*) &\\ni~A\\bfx^*+\\bfb.\n\\end{cases}\n\\end{eqnarray}\n\\end{definition}","sha256_utf8":"1f076a8322359eadb58c9a56d85a4d81fd1714a192472aa8809602a2a9e4d53d"},{"start":31524,"end":31724,"start_line":546,"end_line":547,"raw_tex":"We also say a point $(\\bfx^*;\\bfz^*)$ is a P-stationary point of the problem (\\ref{ell-0-1}) if it satisfies the conditions in \\eqref{p-stationary}. \nFor a point $\\bfx^*$, we denote two constants by \n","sha256_utf8":"b7622282fe31f10b1532eefa5dd9102cdfa5d21044dd53778c1c5f7d402d89f0"},{"start":31725,"end":32038,"start_line":548,"end_line":559,"raw_tex":"\\begin{eqnarray}\n\\label{tau-*} \n~~~~\\tau_1 := \n\\begin{cases}\n+\\infty, & \\bfy^*\\leq0,\\\\\n{\\min} \\left\\{\\frac{(y^*_i)^2}{2\\lambda}:y_i^*>0\\right\\},& {\\rm otherwise},\\\\\n\\end{cases}~~\n\\tau_2:=\\begin{cases}\n+\\infty, & \\Gamma_*=\\emptyset,\\\\\n \\frac{2\\lambda}{\\max_i|p^*_i|^2},& {\\rm otherwise},\n\\end{cases}\n\\end{eqnarray}","sha256_utf8":"3b99a7debbc9002bc8054760d5b9b09a0c985855f9a626179a53fd191d3a9b6f"},{"start":32039,"end":32270,"start_line":560,"end_line":560,"raw_tex":"where $\\bfy^*:=A\\bfx^*+\\bfb$ and $\\bfp^*:=-(A_{\\Gamma_*}A_{\\Gamma_*}^{\\top})^{-1}A_{\\Gamma_*}\\nabla f(  \\bfx^*)$. Clearly, both $\\tau_1>0$ and $\\tau_2>0$. Based on these notation, we have the following main result of this section.\n","sha256_utf8":"380afcde2a5421aff9e374c9411d36be7834cf6a5d28c0696e140f48987d3ed2"},{"start":37239,"end":37462,"start_line":642,"end_line":644,"raw_tex":"\\begin{assumption} \\label{ass}\n\t Suppose $f$ is twice continuously differentiable, $\\nabla^2 f(\\bfx^*)$ is {positive definite} and $A_{\\Gamma_*}$ is full row rank, where $\\Gamma_*$ is given by (\\ref{T-*}).\n \\end{assumption}","sha256_utf8":"697f653165091330a8c09380437b6e1d90ffccb392c0722313ddfc9911ccd3d7"},{"start":37465,"end":37779,"start_line":646,"end_line":650,"raw_tex":"\\begin{assumption} \\label{ass2} \nSuppose $\\nabla^2 f$ is locally Lipschitz continuous around $\\bfx^*$ with a constant $L_*>0$, namely \n\\begin{eqnarray*} \\|\\nabla^2 f(\\bfx)-\\nabla^2 f(\\bfx')\\|\\leq L_* \\|   \\bfx -\\bfx' \\| \\end{eqnarray*}\nfor any $\\bfx$ and $\\bfx'$ in the neighbourhood of $\\bfx^*$.\n \\end{assumption}","sha256_utf8":"abd5a06c17a3c93e3c6c746fe1ef3183981fc99abb24fecb9361193d9bde8484"},{"start":37816,"end":38379,"start_line":653,"end_line":662,"raw_tex":" For a point  $\\bfw:=(\\bfx; \\bfz)$, we define the sets {\n\\begin{eqnarray}\\label{p-stationary-T}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n \\begin{array}{rcll}\n\\S&:=& \\left\\{i\\in\\N_m:  A_i\\bfx+b_i+\\tau z_i   \\in (0, \\sqrt{2\\tau\\lambda}) \\right\\}, \\\\\n \\E&:=& \\left\\{i\\in\\N_m: A_i\\bfx+b_i +\\tau z_i \\in\\{0,\\sqrt{2\\tau\\lambda}\\} \\right\\},\\\\\n\\O&:=& \\left\\{i\\in\\N_m:  A_i\\bfx+b_i+ \\tau z_i \\in(-\\infty,0)\\cup ( \\sqrt{2\\tau\\lambda},\\infty) \\right\\},\\\\\n\\E^o&:=& \\left\\{i\\in\\N_m: A_i\\bfx+b_i =0, \\tau z_i   \\in\\{0,\\sqrt{2\\tau\\lambda}\\} \\right\\}, \n\\end{array}\n\\end{eqnarray}","sha256_utf8":"1fab1783d2b19b7d16a198d82c2db385cef85101ff7e2885ff6ed3f3ee2727ea"},{"start":38381,"end":38864,"start_line":663,"end_line":669,"raw_tex":"for a given $\\tau>0$. \n%The first three sets stand for index sets on which the value $|A_i\\bfx+b_i+\\tau z_i-\\sqrt{\\tau\\lambda/2}|$ is smaller than, equal to and over than $\\sqrt{\\tau\\lambda/2}$, respectively. \nObviously, $\\E^o\\subseteq \\E.$ \nIt is worth mentioning that all sets depend on $\\bfw$. \nFor simplicity, we drop their dependence whenever \nthere is no confusion to be caused. \nSame rules are also applied into $\\S_*,\\E_*,\\O_*$ and $\\E_*^o$ for  $\\bfw^*:=(\\bfx^*; \\bfz^*)$. \n","sha256_utf8":"4ceefdda8ea75787a23082d7dab69162e1f8ff0a02cb9f886e2b4d766caeec27"},{"start":39088,"end":39505,"start_line":674,"end_line":689,"raw_tex":"\\begin{eqnarray}\n\\label{sta-eq-1}\n\\hspace{8mm} F(\\bfw; \\Gamma) =\\left[\\begin{array}{rrl}\n\\nabla f(\\bfx)  + A_{\\Gamma}^\\top \\bfz_{\\Gamma}  \\\\\nA_{\\Gamma} \\bfx + \\bfb_{\\Gamma} \\\\\n\\bfz_{\\overline \\Gamma} \n\\end{array}\n\\right] = 0, \\quad \n%\\label{def-Jacobian}\n\\nabla F_{\\mu}(\\bfw;\\Gamma) =\\left[\\begin{array}{ccc}\n\\nabla^2 f(\\bfx)   & A_{\\Gamma} ^\\top &0\\\\\nA_{\\Gamma} &-\\mu I&0 \\\\\n0&0&I\n\\end{array}\n\\right].\n\\end{eqnarray}","sha256_utf8":"fbef5a38d62a94ede83d8c49a05fc040e8c265115c1d8f76a2759693546096c1"},{"start":39507,"end":39659,"start_line":690,"end_line":692,"raw_tex":"We note that the matrix $\\nabla F_{\\mu}$ is a slight perturbation of\nthe Jacobian matrix of $F$ and \n$\\nabla F(\\bfw;\\Gamma)=\\nabla F_{0}(\\bfw;\\Gamma)$,\n","sha256_utf8":"c21e9b2719da5d3eb2da1702f2448c846098c6efe61890b1e2efdb2e06440140"},{"start":44520,"end":44680,"start_line":786,"end_line":787,"raw_tex":"Let $\\bfw^k:=(\\bfx^k; \\bfz^k)$ be the current iterate.\nWe define $\\S_k$ and $\\E_k^o$ by \\eqref{p-stationary-T} with $\\bfw$ being replaced by $\\bfw^k$ and let  \n","sha256_utf8":"beb96114d22b84cf0f4b77df43284cdd183a059236e09a535c2e693f2196f21d"},{"start":44682,"end":44736,"start_line":788,"end_line":790,"raw_tex":"\\beq \\label{T-k}\n      T_k := \\S_k\\cup \\E_k^o.  \n \\eeq","sha256_utf8":"25845fed3810904a682d346ebd078570eeed2e9ab1bf80ff8496430f2dd32cbc"},{"start":44737,"end":44938,"start_line":791,"end_line":792,"raw_tex":"Let $\\bfd^k=(\\bfu^k;\\bfv^k)$ with $\\bfu^k\\in\\R^n$ and $\\bfv^k\\in\\R^{m}$. For \nsuch a defined $T_k$, a Newton direction $\\bfd^k$ for the equation \\eqref{sta-eq-1} solves the following linear equations:\n","sha256_utf8":"820cc6957ab6c9e6b52987f32c8d7aedb06feb352fdb676223576da99dbc39fb"},{"start":44939,"end":45015,"start_line":793,"end_line":795,"raw_tex":"\\[ %\\beq \\label{newton-d}\n \\nabla F(\\bfw^k;T_k)~\\bfd = - F(\\bfw^k;T_k). \n \\]","sha256_utf8":"7fc948b8017ca5c144349adc1edbfea572d816e836987cb0a5377ab49963e24b"},{"start":45022,"end":45196,"start_line":796,"end_line":798,"raw_tex":"To improve the nonsingularity of the Jacobian matrix $\\nabla F(\\bfw^k;T_k)$,\nwe replace it with $\\nabla F_{\\mu_k}(\\bfw^k;T_k)$. \nThat is, at $\\bfw^k$, we solve the equation:\n","sha256_utf8":"c9e55c48ba36c61a3b81774e40a9b8e3905062bf86d852ef611f2b7c1d2db2c9"},{"start":45197,"end":45283,"start_line":799,"end_line":801,"raw_tex":"\\beq \\label{newton-dir-0}\n \\nabla F_{\\mu_k}(\\bfw^k;T_k)~\\bfd = - F(\\bfw^k;T_k), \n \\eeq","sha256_utf8":"21c8603a17f704d008dd736b3b0226d2fe2e9b694702b779d004bbdc16106741"},{"start":45284,"end":45395,"start_line":802,"end_line":803,"raw_tex":"where $\\nabla F_{\\mu_k}(\\bfw^k;T_k)$ is  defined in \\eqref{sta-eq-1}.\nThe Newton direction $\\bfd^k$ satisfies \n","sha256_utf8":"faa58111bbf845ad031405097f60654116f1f4d99592334e31e3f6b07c75e5d2"},{"start":45397,"end":45873,"start_line":804,"end_line":826,"raw_tex":"\\beq \\label{newton-dir} \n \\left[  \n     \\begin{array}{ccc}\n     {\\nabla^2  f(\\bfx^k)}& A_{T_k}^\\top&0\\\\\n   A_{T_k}&-\\mu_k I&0\\\\\n    0&0&I\n     \\end{array}\n     \\right]  \n     \\left[  \n     \\begin{array}{l}\n    \\bfu^k\\\\\n    \\bfv^k_{T_k}\\\\\n    \\bfv^k_{\\overline T_k}\n     \\end{array}\n     \\right]\n     =-\\left[  \n     \\begin{array}{r}\n    \\nabla  f(\\bfx^k)+A_{T_k}^\\top\\bfz^k_{T_k}\\\\\n   A_{T_k}\\bfx^k+\\bfb_{T_k} \\\\\n    \\bfz^k_{\\overline T_k}\n     \\end{array}\n     \\right].\n \\eeq","sha256_utf8":"3de70cb6ac628dff24b0f0edd426a4c97a9b678a66c88addcdae46c57cfdb656"},{"start":46145,"end":46193,"start_line":828,"end_line":828,"raw_tex":"Here, the rule to update $\\mu_k$ is as follows:\n","sha256_utf8":"2db38478d11411a49e25592e70b68e11f0a6b9673ff53e803219a5fc003c3301"},{"start":46196,"end":46283,"start_line":829,"end_line":832,"raw_tex":"\\beq \\label{update-mu} \n\\mu_{k}=\n\\min\\{\\alpha \\mu_{k-1}, \\rho\\|F(\\bfw^k;T_k)\\|\\},\n \\eeq","sha256_utf8":"550f0cf1ca264fed85820183d660639a26a34f0fa560f9e747ee4df89ab11ac1"},{"start":46284,"end":47015,"start_line":833,"end_line":848,"raw_tex":" where $\\alpha\\in(0,1)$ and $\\rho>0$. Now we summarize   the proposed method in  \\cref{Alg-NM01}. \n\\begin{algorithm}[H]\n\t\\caption{\\nhs: Newton's method for $0/1$-loss optimization}\n\t\\begin{algorithmic}[1] \\label{Alg-NM01}\n\t\t%\\scriptsize\n\t\t\\STATE Initialize $\\bfw^0=(\\bfx^0;\\bfz^0)$ and $\\mu_{-1}>0$. Set the parameter $\\tau,\\lambda,\\rho>0, \\alpha\\in(0,1)$.\\\\\n\t\t Compute $T_0$ by \\eqref{T-k} and set $k :=0$.\n\\IF{ $\\|F(\\bfw^k;T_k)\\|>0$}\t\t \n         \\STATE  Update $\\mu_{k}$ by \\eqref{update-mu}.\t\n\t\t\\STATE  Update $\\bfd^k$ by solving \\eqref{newton-dir}.\n\t\t\\STATE  Update $\\bfw^{k+1}~=~\\bfw^k+\\bfd^k$.\n\t\t\\STATE  Update $T_{k+1}$ by \\eqref{T-k} and  set $k := k+1$.   \t\t\n\\ENDIF\t\t\n\\RETURN $\\bfw^{k}$.\n\t\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"0e3b0deb009143cd023795fab14368a6723a577a62a175bdccbe7702989952e1"},{"start":48940,"end":49148,"start_line":860,"end_line":860,"raw_tex":"The first technical result is about extending the stationarity equation to some indices that are given in a neighborhood of $\\bfw^*$. In the proof, we recall $\\bfy = A\\bfx+\\bfb$ and $\\bfy^* = A\\bfx^*+\\bfb$. \n","sha256_utf8":"448b596bc35ba52306ef4c89ea9f8e9f44a31aede5417695660c885a02cca3c2"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{quadratic-convergence} \nLet $\\bfw^*$ be any {\\ts} with $0<\\tau<\\tau_*$ of  (\\ref{ell-0-1}), $\\tau_*$ and $\\delta_2^*, c_*, C_*$ be given by (\\ref{tau-*}) and (\\ref{radius-2}). Assume  \\cref{ass} and \\cref{ass2}. Let $\\{\\bfw^k\\}$ be the  sequence generated by  \\cref{Alg-NM01} and $0\\leq\\mu_{-1}\\leq c_*/2$. If the initial point  satisfies $\\bfw^0\\in N(\\bfw^*,\\delta_*)$, where\n\\begin{eqnarray}\\label{delta-*}\n\\delta_*:=\\min\\left\\{\\delta_2^*,~{c_*}/{(2 (L_*+2\\rho C_*))}\\right\\},\n\\end{eqnarray}\n then the following results hold.\n\\begin{itemize}\n\\item[a)] The sequence $\\{\\bfd^k\\}_{k\\geq0}$ is well defined and $\\lim_{k\\rightarrow\\infty}\\bfd^k=0$. \n\\item[b)] The whole sequence $\\{\\bfw^k\\}$ converges to $\\bfw^*$ quadratically, namely,\n\\beq\n\\|\\bfw^{k+1}-\\bfw^{*}\\|  &\\leq& \n  ((L_*+2\\rho C_*)/c_*) \\|\\bfw^{k}-\\bfw^{*} \\|^2. \\nonumber \n\\eeq\n\\item[c)] The halting condition satisfies\n\\beq\n\\|F(\\bfw^{k+1};T_{k+1})\\| \n&\\leq& (L_*+2\\rho C_*)(C_*/c_*^{3})  \\|F(\\bfw^{k};T_k)\\|^2  \\nonumber \n\\eeq\nand  \\Cref{Alg-NM01} reaches $\\|F(\\bfw^{k};T_k)\\| <  \\epsilon$ for a given tolerance $\\epsilon>0$ when \\beq\\label{k-tol}\nk\\geq \\Big\\lceil \\log_2\\Big(2 \\sqrt{(L_*+2\\rho C_*)(C_*/c_*)^3  }\\|\\bfw^{0}-\\bfw^{*} \\|\\Big)-\\log_2(\\sqrt{\\epsilon})\\Big\\rceil.\\eeq  \n\\end{itemize}\n\\end{theorem}","units":[{"start":60565,"end":61857,"start_line":1085,"end_line":1106,"raw_tex":"\\begin{theorem}\\label{quadratic-convergence} \nLet $\\bfw^*$ be any {\\ts} with $0<\\tau<\\tau_*$ of  (\\ref{ell-0-1}), $\\tau_*$ and $\\delta_2^*, c_*, C_*$ be given by (\\ref{tau-*}) and (\\ref{radius-2}). Assume  \\cref{ass} and \\cref{ass2}. Let $\\{\\bfw^k\\}$ be the  sequence generated by  \\cref{Alg-NM01} and $0\\leq\\mu_{-1}\\leq c_*/2$. If the initial point  satisfies $\\bfw^0\\in N(\\bfw^*,\\delta_*)$, where\n\\begin{eqnarray}\\label{delta-*}\n\\delta_*:=\\min\\left\\{\\delta_2^*,~{c_*}/{(2 (L_*+2\\rho C_*))}\\right\\},\n\\end{eqnarray}\n then the following results hold.\n\\begin{itemize}\n\\item[a)] The sequence $\\{\\bfd^k\\}_{k\\geq0}$ is well defined and $\\lim_{k\\rightarrow\\infty}\\bfd^k=0$. \n\\item[b)] The whole sequence $\\{\\bfw^k\\}$ converges to $\\bfw^*$ quadratically, namely,\n\\beq\n\\|\\bfw^{k+1}-\\bfw^{*}\\|  &\\leq& \n  ((L_*+2\\rho C_*)/c_*) \\|\\bfw^{k}-\\bfw^{*} \\|^2. \\nonumber \n\\eeq\n\\item[c)] The halting condition satisfies\n\\beq\n\\|F(\\bfw^{k+1};T_{k+1})\\| \n&\\leq& (L_*+2\\rho C_*)(C_*/c_*^{3})  \\|F(\\bfw^{k};T_k)\\|^2  \\nonumber \n\\eeq\nand  \\Cref{Alg-NM01} reaches $\\|F(\\bfw^{k};T_k)\\| <  \\epsilon$ for a given tolerance $\\epsilon>0$ when \\beq\\label{k-tol}\nk\\geq \\Big\\lceil \\log_2\\Big(2 \\sqrt{(L_*+2\\rho C_*)(C_*/c_*)^3  }\\|\\bfw^{0}-\\bfw^{*} \\|\\Big)-\\log_2(\\sqrt{\\epsilon})\\Big\\rceil.\\eeq  \n\\end{itemize}\n\\end{theorem}","sha256_utf8":"177b766cb0e6a2c21c3394e9e2497f6722f235850b637c2f2d351c127f17f8ea"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma} \\label{subdifferentials}\n\n\\begin{itemize}\n\t\n \\item[(i)] We have\n \\begin{eqnarray}\\label{subfi}\n  \\partial\\|{\\bfy}_+\\|_0\n  =\\left\\{{\\bfv}\\in {\\R}^m: v_i\\left\\{\\begin{array}{ll}\n \\geq0,& y_i=0,\\\\\n =0,& y_i\\neq0,\n \\end{array}\n \\right. i\\in{\\N}_m\\right\\}. \n \\end{eqnarray}\n \n \\item[(ii)] Let $g(\\bfx):=\\|(h(\\bfx))_+\\|_0$, where $h:\\R^n\\rightarrow\\R^m$ is differentiable,  and $\\Gamma:=\\{i\\in\\N_m: h_i(  \\bfx)=0\\}$ for a given a point $ \\bfx\\in\\R^n$.  If \n \\begin{eqnarray}\n \\label{chain-rule-cond}\n \\left.\n \\begin{array}{r}\n \\forall \\bfz\\in\\R^{|\\Gamma|},~~\\bfz\\geq 0,~~\n (\\nabla h( \\bfx))_{\\Gamma}^\\top \\bfz = 0\n \\end{array}\\right.~~\\Longrightarrow~~\\bfz = 0,\n \\end{eqnarray}\n then the subdifferential of $g(\\bfx)$  at $  \\bfx$ is  \n \\begin{eqnarray}\n \\label{chain-rule-sub} \n \\partial g(  \\bfx)=\\nabla h( \\bfx)^\\top \\partial \\|(h(  \\bfx))_+\\|_0.\n \\end{eqnarray}\n\n\\end{itemize}\n\\end{lemma}\n\n\\begin{proof}\ni)   It is straight to check that the regular subdifferential $\\widehat{\\partial}\\|{\\bfy}_+\\|_0$ (see \\cite[Definition 8.3]{RW1998})  of $\\|{\\bfy}_+\\|_0$ at ${\\bfy}$ takes the following form,\n\\begin{eqnarray}\\label{subfi-0}\n \\widehat{\\partial}\\|{\\bfy}_+\\|_0=\\left\\{{\\bfv}\\in {\\R}^m: v_i\\left\\{\\begin{array}{ll}\n \\geq0,& y_i=0,\\\\\n =0,& y_i\\neq0,\n \\end{array}\n \\right. i\\in{\\N}_m\\right\\}=:\\Omega(\\bfy). \n\\end{eqnarray}\nWe next verify $ \\partial\\|{\\bfy}_+\\|_0= \\Omega({\\bfy}) $. By letting $\\varphi(\\cdot):=\\|(\\cdot)_+\\|_0$, we have\n  \\begin{eqnarray}\\label{sub-lim-reg}\n  \\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n  \\begin{array}{rclll}\n\\partial\\|{\\bfy}_+\\|_0&=&\\limsup_{{\\bfz}\\stackrel{\\varphi}{\\rightarrow}{\\bfy}}~ \\widehat{\\partial}\\|{\\bfz}_+\\|_0 ~~~( \\text{by  \\cite[Equation 8(5)]{RW1998}})\\\\\n&=&\\limsup_{{\\bfz}\\stackrel{\\varphi}{\\rightarrow}{\\bfy}}~\\Omega({\\bfz}) ~~~~~~( \\text{by \\eqref{subfi-0}})\\\\\n&=&\\{\\bfv\\in \\R^m:\\exists~\\bfz^k\\stackrel{\\varphi}{\\rightarrow} \\bfy,~\\bfv^k\\rightarrow \\bfv ~\\text{with}~\\bfv^k\\in \\Omega(\\bfz^k) \\}=:\\Theta,\n\\end{array}\n \\end{eqnarray}\nwhere  $\\bfz\\stackrel{\\varphi}{\\rightarrow} \\bfy$ represents $\\bfz\\rightarrow \\bfy, \\varphi(\\bfz)\\rightarrow \\varphi(\\bfy)$. Clearly, $\\Omega({\\bfy})\\subseteq\\Theta$. On the other hand, $\\Theta \\subseteq \\Omega({\\bfy})$  follows from that $\\Omega(\\bfz^k)\\subseteq \\Omega(\\bfy)$ for any $\\bfz^k\\stackrel{\\varphi}{\\rightarrow} \\bfy$ and $\\Omega(\\cdot)$ is closed.\n \n ii) Direct verifications yield the following chain of equations,\n \\begin{eqnarray*}  \\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n  \\begin{array}{rclll}\n \\partial^\\infty \\|\\bfy_+\\|_0&=& {\\limsup}_{\\sigma\\downarrow0,~{\\bfz}\\stackrel{\\varphi}{\\rightarrow} {\\bfy}} \\sigma \\widehat{\\partial}\\|\\bfz_+\\|_0&~~( \\text{by \\cite[Equation 8(5)]{RW1998}})\\\\ \n&=&\n {\\limsup}_{{\\bfz}\\stackrel{\\varphi}{\\rightarrow} {\\bfy}}~ \\widehat{\\partial}\\|\\bfz_+\\|_0&~~( \\text{by \\eqref{subfi-0}})\\\\\n & =& \\partial\\|{\\bfy}_+\\|_0,&~~( \\text{by \\eqref{sub-lim-reg}})\\\\\n \\end{array}\n \\end{eqnarray*}\n where $ \\partial^\\infty \\|{\\bfy}_+\\|_0$ is the horizon subdifferential of $\\|{\\bfy}_+\\|_0$. Therefore, we derive that $\\widehat{\\partial}\\varphi=\\partial \\varphi=\\partial^\\infty \\varphi$.  One can easily prove that the horizon cone $\\widehat{\\partial}\\varphi(\\bfy)^\\infty$ (see \\cite[Definition 3.3]{RW1998}) of $ \\widehat{\\partial}\\varphi(\\bfy)$ satisfies $\\widehat{\\partial}\\varphi(\\bfy)^\\infty = \\partial^ \\infty\\varphi(\\bfy)$. These conditions indicate that  the function $\\varphi$ is regular by \\cite[Corollary 8.11]{RW1998}, which together with \\eqref{chain-rule-cond}, $g(\\bfx)=\\varphi(h(\\bfx))$ and \\cite[Theorem 10.6]{RW1998} derives \\eqref{chain-rule-sub} immediately. \n\\end{proof}\n\n\\begin{lemma}\\label{first-order-nec}\n\tThe following relationships hold for the problem (\\ref{ell-0-1}).\n\\begin{itemize}\n\t\n\\item[i)] A local minimizer $\\bfx^*$   satisfies the following condition \n         if  $A_{\\Gamma_*}$ is full row rank,\n\\begin{eqnarray} \\label{lemma-1st-order-nec}\n    -\\nabla f(\\bfx^*) \\in A^\\top \\partial \\|(A\\bfx^*+\\bfb)_+\\|_0.\n\\end{eqnarray}\n \n\\item[ii)] A point $\\bfx^*$ satisfying (\\ref{lemma-1st-order-nec})  is a local minimizer if  the function $f$ is locally convex around $\\bfx^*$.\n\\end{itemize}\n\\end{lemma}\n\n\\begin{proof}\ni) It follows from \\cite[Theorem 10.1]{RW1998} that a local minimizer of \\eqref{ell-0-1} must satisfy $-\\nabla f(\\bfx^*)\\in\\lambda\\partial g(\\bfx^*)$, where $g(\\bfx):=\\|(A\\bfx+\\bfb)_+\\|_0$. This together with \\cref{chain-rule-sub}  and $\\lambda \\partial\\|(\\cdot)_+\\|_0 = \\partial \\|(\\cdot)_+\\|_0$ by \\eqref{subfi} derives the result immediately.  \n\nii)   Since the problem \\eqref{ell-0-1} is equivalent to the following problem, \n\\begin{eqnarray}\\label{ell-0-1-equ-0}\n&\\underset{\\bfx\\in\\R^n,\\bfy\\in\\R^m}{\\min}& f(\\bfx)+\\lambda \\|\\bfy_+\\|_0,\\\\\n  &{\\rm s.t.}& A\\bfx+\\bfb-\\bfy = 0,\\nonumber\n\\end{eqnarray}\nit suffices to show that  $(\\bfx^*;\\bfy^*)$  is a local minimizer of the   problem \\eqref{ell-0-1-equ-0}, where $\\bfx^*$ satisfies \\eqref{lemma-1st-order-nec} and $\\bfy^*=A \\bfx^*+\\bfb$, namely, there is a $\\bfz^*$ such that\n\\begin{eqnarray}\n\\label{lemma-1st-order-nec-10}\n\\nabla f(\\bfx^*) +A^\\top\\bfz^*=0,~~ \nA \\bfx^*+\\bfb-\\bfy^*=0, ~~\n\\partial\\|\\bfy^*_+\\|_0\\ni \\bfz^*. \n\\end{eqnarray} \nIt follows from $\\bfy^*=A\\bfx^*+\\bfb$ and \\eqref{T-*} that $\\Gamma_*=\\{i\\in\\N_m: y^*_i=0\\}$. This together with $\\partial\\|\\bfy^*_+\\|_0 \\ni \\bfz^*$ and the expression of the $\\partial\\|\\bfy^*_+\\|_0$ in  \\eqref{subfi} indicates \n\\begin{eqnarray}\\label{fact-11}\n\\bfy_{\\Gamma_*}^*=0,~~\\bfz_{\\Gamma_*}^* \\geq 0, ~~~~\\bfy_{\\overline \\Gamma_*}^*\\neq 0,~ ~\\bfz_{\\overline \\Gamma_*}^* = 0. \n\\end{eqnarray} \n Define a radius $\\delta:=\\min\\{\\delta_1,\\delta_2\\}$, where\n\\begin{eqnarray}\\label{delta-2}\n~~~~~~\\delta_1 := \n\\begin{cases}\n+\\infty,&A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}=0\\\\\n\\frac{\\lambda}{ \\|A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}\\|},& \\text{otherwise}, \n\\end{cases}~~~~\n\\delta_2 := \n\\begin{cases}\n+\\infty,&\\bfy^*\\leq {\\bf0},\\\\\n\\min_i\\{y_i^*:y_i^*>0\\},&\\text{otherwise}, \n\\end{cases}\n\\end{eqnarray}\nand consider a local region of ${\\bfw}^* := (\\bfx^*;\\bfy^*)$ by\n\\begin{eqnarray}\\label{local-area-1}\n~~N( {\\bfw}^*,\\delta)=\\left\\{ \\left(\n\\bfx; \n  {\\bfy}\\right)\\in\\R^{n+m}   \n:~A\\bfx+\\bfb - \\bfy=0, ~\n \\|{\\bfw}- {\\bfw}^*\\|<\\delta\n\\right\\}.\n\\end{eqnarray}\nIndeed, $N( {\\bfw}^*,\\delta)$ is a neighbourhood of ${\\bfw}^*$ since $A\\bfx^*+\\bfb - \\bfy^*=0$ from \\eqref{lemma-1st-order-nec-10}. Next we show that, for any $\\bfw\\in N( {\\bfw}^*,\\delta)$,\n\\begin{eqnarray}\\label{y-y*-0}\n\\|\\bfy^*_+\\|_0\\leq \\|\\bfy_+\\|_0.\n\\end{eqnarray}\nObviously, it is true if $\\bfy^*\\leq0$ as $\\|\\bfy^*_+\\|_0=0$. For $\\bfy^*\\nleq0$,  to guarantee \\eqref{y-y*-0}, it suffices to show that for any $i$, $y_i^*>0~\\Longrightarrow~y_i>0.$ \nSuppose there is a $j\\in\\N_m$ such that $y_j^*>0$ but $y_j\\leq0$.  This incurs the following contradiction \n\\begin{eqnarray*}\\delta_2 \\geq \\delta\n& {>}&\\|\\bfw-\\bfw^*\\| \\hspace{3.5cm} ({\\rm by}~ \\eqref{local-area-1})\\\\\n&\\geq& |y_j-y_j^*|=y_j^*-y_j  \\geq y_j^*\\geq\\delta_2.\\hspace{.5cm} ({\\rm by}~ \\eqref{delta-2}) \\end{eqnarray*}\nAgain, for any $\\bfw\\in N( {\\bfw}^*,\\delta)$, we have $A\\bfx+\\bfb - \\bfy=0$, which and \\eqref{lemma-1st-order-nec-10} generate\n \\begin{eqnarray}\\label{y-y*-h-h*}\n \\bfy-\\bfy^*=A (\\bfx- \\bfx^*).\n  \\end{eqnarray}\nNext, the convexity of $f$ gives rise to  \n \\begin{eqnarray}\\label{fx-fx*}\n\\hspace{15mm} f(\\bfx)-f(\\bfx^*)&\\geq & \\langle \\nabla f(\\bfx^*), \\bfx - \\bfx^*\\rangle~~= -\\langle  A^\\top \\bfz^* , \\bfx - \\bfx^*\\rangle \\hspace{1.6cm} ({\\rm by}~ \\eqref{lemma-1st-order-nec-10})\\nonumber\\\\\n% &=& -\\langle  A_{\\Gamma_*}  (\\bfx - \\bfx^*),\\bfz^* _{\\Gamma_*}\\rangle-\\langle   A_{\\overline  \\Gamma_*} (\\bfx - \\bfx^*),\\bfz^* _{\\overline  \\Gamma_*}\\rangle\\nonumber\\\\\n &=& -\\langle  A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*},\\bfx - \\bfx^*\\rangle= -\\langle  A_{\\Gamma_*}(\\bfx - \\bfx^*),\\bfz^* _{\\Gamma_*}\\rangle=:\\phi\\hspace{0.3cm} ({\\rm by}~ \\eqref{fact-11})\n  \\end{eqnarray} \n Now we make the conclusion by two cases.  \n If $\\|\\bfy^*_+\\|_0= \\|\\bfy_+\\|_0$, then   $\\bfy_{\\Gamma_*}\\leq 0$ due to $\\bfy^*_{\\Gamma_*}=0$. This and \\eqref{y-y*-h-h*} yield that\n  \\begin{eqnarray}\\label{y-y*-h-h*-0}\n0\\geq  \\bfy_{\\Gamma_*}= \\bfy_{\\Gamma_*}-\\bfy^*_{\\Gamma_*}  = A_{\\Gamma_*} (\\bfx-\\bfx^*),\n  \\end{eqnarray}\nwhich together with $\\bfz^* _{\\Gamma_*}\\geq0$ from \\eqref{fact-11} indicates $\\phi\\geq0$, namely $f(\\bfx)\\geq f(\\bfx^*)$. So $$f(\\bfx) +\\lambda \\|\\bfy_+\\|_0 \\geq f(\\bfx^*) + \\lambda\\|\\bfy_+^*\\|_0.$$\nIf $\\|\\bfy^*_+\\|_0 \\neq \\|\\bfy_+\\|_0$, we must have $ \\|\\bfy_+\\|_0 \\geq 1+\\|\\bfy^*_+\\|_0$ by \\eqref{y-y*-0}. If  $A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}=0$, then $\\phi=0>-\\lambda$. Otherwise, it follows\n  \\begin{eqnarray*}\n\\phi&\\geq&- \\|A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}\\|\\|\\bfx - \\bfx^*\\| \\geq - \\|A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}\\|\\|\\bfw - \\bfw^*\\|\\\\\n & \\geq&- \\|A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}\\|\\delta \n  \\geq - \\|A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}\\|\\delta_1=-\\lambda.\\hspace{.5cm} ({\\rm by}~ \\eqref{delta-2})\n  \\end{eqnarray*}\nBoth lead to $\\phi\\geq-\\lambda$, which results in\n  \\begin{eqnarray*}\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n  \\begin{array}{rclll}\n  f(\\bfx) +\\lambda \\|\\bfy_+\\|_0 &\\geq& f(\\bfx^*) + \\phi +\\lambda \\|\\bfy_+\\|_0 &\\ ({\\rm by}~ \\eqref{fx-fx*} )\\\\\n&\\geq& f(\\bfx^*) -\\lambda + \\lambda\\|\\bfy_+\\|_0 &\\ ({\\rm by}~ \\phi\\geq-\\lambda)\\\\\n&\\geq& f(\\bfx^*) + \\lambda\\|\\bfy_+^*\\|_0.&\\ ({\\rm by}~ \\|\\bfy_+\\|_0 \\geq 1+\\|\\bfy^*_+\\|_0 )\n \\end{array} \\end{eqnarray*}\nOverall, the two cases show that $(\\bfx^*;\\bfy^*)$ is  a local minimizer to \\eqref{ell-0-1-equ-0}. Namely, $\\bfx^*$ is  a local minimizer to \\eqref{ell-0-1}. \n\\end{proof}\n\n\\begin{theorem} \\label{Stationary-Thm}\n   The following relationships hold for the problem (\\ref{ell-0-1}).\n\\begin{itemize}\n\\item[i)]  A local minimizer $\\bfx^*$ is a  P-stationary point for any $0<\\tau<\\tau_*:=\\min\\{\\tau_1,\\tau_2\\}$ if  $A_{\\Gamma_*}$ is full row rank. \n\\item[ii)] A P-stationary point with $ \\tau>0$  is a local minimizer if  the function $f$ is locally convex around $\\bfx^*$. \n\\item[iii)] A P-stationary point with $ \\tau\\geq\\|A\\|^2/c_f$  is   a global minimizer if  the function $f$ is strongly convex with a constant $c_f>0$.\n\\end{itemize}\n\\end{theorem}\n\n\\begin{proof} i) As  $\\bfx^*$ is a local minimizer of \\eqref{ell-0-1},  condition  \\eqref{lemma-1st-order-nec} is valid by \\cref{first-order-nec} if $A_{\\Gamma_*}$ is  full row rank. In other words, there is a $\\bfz^*$ such that\n\\begin{eqnarray}\n\\label{lemma-1st-order-nec-1}\n\\nabla f(\\bfx^*) +A^\\top\\bfz^*=0,~~\nA \\bfx^*+\\bfb-\\bfy^*=0,~~\n\\bfz^*\\in\\partial\\|\\bfy^*_+\\|_0. \n\\end{eqnarray} \nTherefore, to show \\eqref{p-stationary}, we only need to verify that,  for any $0<\\tau<\\tau^*$, $$\\bfz^*\\in \\partial\\|\\bfy^*_+\\|_0~ \\Rightarrow~\\bfy^*\\in\\P :=\\pro (\\bfy^*+\\tau \\bfz^*). $$ \nRecall the definition of $\\Gamma_*$ in \\eqref{T-*} and the second condition in \\eqref{lemma-1st-order-nec-1}, we have $\\bfy_{\\Gamma_*}=(A \\bfx^*+\\bfb)_{\\Gamma_*}=0$. Same reasoning also allows for obtaining \\eqref{fact-11} due to $\\bfz^*\\in \\partial\\|\\bfy^*_+\\|_0$. \nAs $A_{\\Gamma_*}$ is full row rank, the first condition in \\eqref{lemma-1st-order-nec-1} and  $A^\\top\\bfz^*  =A_{\\Gamma_*}^\\top\\bfz^*_{\\Gamma_*}$ derive  that\n$$\\bfz^*_{\\Gamma_*}=\\bfp^*.$$\nNow, $0<\\tau<\\tau_*= \\min\\{\\tau_1,\\tau_2\\}$ in \\eqref{tau-*} results in \n\\begin{eqnarray}\\label{non-negative}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{lllllllllll}\n y_i^*  & \\geq & \\underset{i:y^*_i>0}{\\min} y^*_i &=&  \\sqrt{2\\tau_1\\lambda}&\\geq&\\sqrt{2\\tau_*\\lambda} &>&\\sqrt{2\\tau \\lambda}  &~~\\text{if}~~& y_i^*>0,\\\\ \n   z_i^* & \\leq &  {\\underset{i}{\\max}|p^*_i|} &= &\\sqrt{2\\lambda/\\tau_2}&\\leq&\\sqrt{2\\lambda/\\tau_*}&< &\\sqrt{2\\lambda/\\tau}& ~~ \\text{if}~~ &z_i^*>0.\n\\end{array}\n\\end{eqnarray}\nThese and \\eqref{fact-11}  yield the following condition,\n\\begin{eqnarray*} \ny_i^*\\begin{cases}\n=0, &   z_i^*=0 ~{\\rm or}~ 0< z_i^* < \\sqrt{2\\lambda/\\tau},\\\\\n<0 ~{\\rm or}~>\\sqrt{2\\tau\\lambda },& z_i^*=0. \n\\end{cases}\n\\end{eqnarray*}\n  {It is easy to see that the above condition satisfies that \n\\begin{eqnarray}\\label{ugTT-1}\ny_i^*  \\in \\P_i=\n\\begin{cases}\n0, & y_i^*+\\tau z_i^*\\in(0, \\sqrt{2\\tau\\lambda}),\\\\\n0~{\\rm or}~ y_i^*+\\tau z_i^*, &y_i^*+\\tau z_i^*\\in\\{0, \\sqrt{2\\tau\\lambda}\\},\\\\\ny_i^*+\\tau z_i^*,& y_i^*+\\tau z_i^*\\in(-\\infty,0)\\cup ( \\sqrt{2\\tau\\lambda},\\infty).\n\\end{cases}\n\\end{eqnarray}}\n\n\n\nii) Note that the second condition in \\eqref{p-stationary} means $\\bfy^*\\in\\P$, which by   \\eqref{ugTT-1} implies $z_i^*=0$ if $y_i^*\\neq 0$ and  $z_i^*\\geq0$ if $y_i^*= 0$, resulting in $\\bfz^*\\in\\partial\\|\\bfy^*_+\\|_0$ by \\eqref{subfi}. Consequently, we obtain \\eqref{lemma-1st-order-nec-1} and \\eqref{lemma-1st-order-nec}.\n% In other words, a  P-stationary point satisfies \\eqref{lemma-1st-order-nec} and thus is a local minimizer  if  $f$ is locally convex around $\\bfx^*$ by\nThe claim follows from  \\cref{first-order-nec} ii).\n\n\niii) Let $(\\bfx^*;\\bfz^*)$ be a  P-stationary point with $ \\tau\\geq\\|A\\|^2/c_f$. Then the second condition in \\eqref{p-stationary} indicates that\n\\beq\n&&({1}/{2\\tau})\\|A\\bfx^*+\\bfb-(A\\bfx^*+\\bfb+\\tau \\bfz^*)\\|^2+\\lambda\\|(A\\bfx^*+\\bfb)_+\\|_0 \\nonumber \\\\\n&\\leq& ({1}/{2\\tau})\\|A\\bfx+\\bfb-(A\\bfx^*+\\bfb+\\tau \\bfz^*)\\|^2+\\lambda\\|(A\\bfx+\\bfb)_+\\|_0\\nonumber\n\\eeq\nfor any $\\bfx\\in\\R^n$, which after simplifying leads to \n\\beq\\label{Ax*b-Axb}\n&& \\lambda\\|(A\\bfx^*+\\bfb)_+\\|_0 - \\|A(\\bfx-\\bfx^*)\\|^2 /(2\\tau) \\nonumber\\\\\n&\\leq& \\lambda\\|(A\\bfx+\\bfb)_+\\|_0 - \\langle  \\bfz^*, A(\\bfx-\\bfx^*) \\rangle\\nonumber\\\\\n&=& \\lambda\\|(A\\bfx+\\bfb)_+\\|_0 - \\langle  A^\\top \\bfz^*, \\bfx-\\bfx^*    \\rangle\\nonumber\\\\\n&=& \\lambda\\|(A\\bfx+\\bfb)_+\\|_0 + \\langle  \\nabla f(\\bfx^*), \\bfx-\\bfx^*  \\rangle. \\hspace{.5cm} ({\\rm by}~  \\eqref{p-stationary}) \n\\eeq\nThe strong convexity of $f$ implies\n\\begin{eqnarray*} \n&&f(\\bfx)+\\lambda\\|(A\\bfx+\\bfb)_+\\|_0-f(\\bfx^*)-\\lambda\\|(A\\bfx^*+\\bfb)_+\\|_0\\\\\n&\\geq&({c_f}/{2})\\|\\bfx-\\bfx^*\\|^2+\\langle  \\nabla f(\\bfx^*), \\bfx-\\bfx^*   \\rangle+\\lambda\\|(A\\bfx+\\bfb)_+\\|_0-\\lambda\\|(A\\bfx^*+\\bfb)_+\\|_0\\\\\n&\\geq&({c_f}/{2})\\|\\bfx-\\bfx^*\\|^2-1/({2\\tau})\\|A(\\bfx-\\bfx^*)\\|^2    \\hspace{.25cm} ({\\rm by}~  \\eqref{Ax*b-Axb})\\\\\n&\\geq&({c_f/2-\\|A\\|^2/(2\\tau}))\\|\\bfx-\\bfx^*\\|^2   \\geq 0. \\hspace{1cm} ({\\rm by}~  \\tau\\geq\\|A\\|^2/c_f) \n\\end{eqnarray*}\nThis shows the global optimality of $\\bfx^*$ to the problem \\eqref{ell-0-1}.\n\\end{proof}\n\n\\begin{theorem}\\label{cor-p-staionary}\nA point $\\bfw^*=(\\bfx^*;\\bfz^*)$ is a  P-stationary point with $\\tau>0$ of the problem (\\ref{ell-0-1}) if and only if   $F(\\bfw^*; \\Gamma_*)=0$ and $ \\Gamma_*=(\\S_* \\cup \\E^o_*)$, where $\\Gamma_*$ is defined by (\\ref{T-*}).\n%the following equations hold\n%\\begin{eqnarray}\n%\\label{sta-eq-1}\n%F(\\bfw^*; \\Gamma_*)=\\left[\\begin{array}{rrl}\n%\\nabla f(\\bfx^*)  + A_{\\Gamma_*}^\\top \\bfz_{\\Gamma_*}^*  \\\\\n%A_{\\Gamma_*} \\bfx^* + \\bfb_{\\Gamma_*} \\\\\n%\\bfz^*_{\\overline \\Gamma_*} \n%\\end{array}\n%\\right]=0. \n%\\end{eqnarray} \nThe Jacobian  $\\nabla F(\\bfw^*;\\Gamma_*)$ is   non-singular if \\cref{ass} holds.\n\\end{theorem}\n\n\\begin{proof} \nThe second claim is obvious due to Assumption \\ref{ass}.\n We only need to prove the first claim. \nWe start with the sufficiency.\n The definitions  in \\eqref{p-stationary-T} show the following relationships,\n\\beq\\label{TSE}\n\\Gamma_*= \\S_* \\cup \\E^o_*,~~~~\\overline{\\Gamma}_*=\\O_*\\cup(\\E_* \\setminus \\E^o_*).\n\\eeq\nWe recall $\\bfy^*=A\\bfx^*+\\bfb$ and $\\P:=\\text{Prox}_{\\tau\\lambda\\|({\\cdot})_{+}\\|_{0}}( \\bfy^*+\\tau \\bfz^*)$.\nIt follows that \n \\allowdisplaybreaks \\begin{eqnarray*} \n \\P_i \n=\n\\begin{cases}\n0, & i\\in \\S,\\\\\n0~{\\rm or}~y^*_i+\\tau z^*_i, &i\\in\\E_*,\\\\\n y_i+\\tau z^*_i,& i\\in \\O_*,\n\\end{cases} \n=\n\\begin{cases}\n0, & i\\in \\S_*\\subseteq \\Gamma_*,\\\\\n0~{\\rm or}~\\tau z^*_i, &i\\in \\E^o_* \\subseteq \\Gamma_*,\\\\\n0~{\\rm or}~ y^*_i, &i\\in(\\E_*\\setminus \\E^o_*) \\subseteq \\overline \\Gamma_*,\\\\\ny^*_i,& i\\in \\O_*\\subseteq \\overline \\Gamma_*,\n\\end{cases} \n\\end{eqnarray*}\nwhere the first equation is by \\eqref{proximal-operator} and second one is by \\eqref{sta-eq-1} and \\eqref{TSE}, which indicates $y^*_i \\in \\P_i$ due to $ \\bfy^*_{\\Gamma_*}=0$ from \\eqref{sta-eq-1}. Moreover,   \nthe first and third equations in \\eqref{sta-eq-1} suffice to $\\nabla f(\\bfx^*)  + A^\\top \\bfz^*  =  0$, showing \\eqref{p-stationary}. Namely,  $ \\bfw^*$ is a  P-stationary point. \n\n{Necessity.}  Let $ \\bfw^*$ be a  P-stationary point satisfying \\eqref{p-stationary} and $T_*:= \\S_* \\cup \\E^o_*$.  Then $\\overline{T}_*=\\O_*\\cup(\\E_* \\setminus \\E^o_*)$. It follows from $y^*_i\\in \\P_i$ and \\eqref{proximal-operator} that\n\\begin{eqnarray*} \ny^*_i\\in \n%\\Big[\\text{Prox}_{\\gamma C\\|({\\cdot})_{+}\\|_{0}}(  \\bfy^*+\\tau \\bfz^*)\\Big]_i \\\\\n\\begin{cases}\n0, & i\\in \\S_*,\\\\\n0~{\\rm or}~ y^*_i+\\tau z^*_i, &i\\in\\E_*,\\\\\ny^*_i+\\tau z^*_i,& i\\in \\O_*,\n\\end{cases} =\n\\begin{cases}\n0, & i\\in \\S_*,\\\\\n0~{\\rm or}~\\tau z^*_i, &i\\in\\E^o_*,\\\\\n0~{\\rm or}~ y^*_i+\\tau z^*_i, &i\\in\\E_*\\setminus \\E^o_*,\\\\\ny^*_i+\\tau z^*_i,& i\\in \\O_*,\n\\end{cases} \n\\end{eqnarray*}\nwhere the equality is by the definition  of $\\E^o_*$, which together with $ y^*_i= 0 , i\\in \\E^o_*$ and $y^*_i\\neq 0, i\\in \\E_*\\setminus \\E^o_*$ suffices to\n\\begin{eqnarray*} \n\\begin{cases}\ny^*_i=0, & i\\in \\S_*\\cup\\E^o_*=T_*,\\\\\ny^*_i = y^*_i+\\tau z^*_i, &i\\in\\E_*\\setminus \\E^o_*,\\\\\ny^*_i= y^*_i+\\tau z^*_i,& i\\in \\O_*,\n\\end{cases} ~~\\Longleftrightarrow~~\n\\begin{cases}\ny^*_i &=0, ~ i\\in T_*,\\\\\n z^*_i&=0, ~i\\in\\overline{T}_*.\n\\end{cases} \n\\end{eqnarray*}\nThis gives rise to the last two conditions $\\bfy^*_{T_*}=0,~\n\\bfz^*_{\\overline T_*}=0$  in \\eqref{sta-eq-1}. \nFurthermore, the first condition in \\eqref{p-stationary} and $\\bfz^*_{\\overline{T}_*}=0$ derive the first condition of \\eqref{sta-eq-1}.\nOverall, we have $F(\\bfw^*; T_*)=0$. \nNow we show $T_*=\\Gamma_*$. By  (\\ref{T-*}) that $\\Gamma_*=\\{i\\in\\N_m: y^*_i=0\\}$, it follows $T_*\\subseteq \\Gamma_*$. Suppose, there is a $j\\in \\Gamma_*$ but $j\\notin T_*$, then\nwe have $y_j^*=z_j^*=0$ and thus $j\\in \\E^o_* \\subseteq T_*$  by \\eqref{p-stationary-T}, a contradiction. Therefore, $T_*=\\Gamma_*$, finishing the proof. \\end{proof}\n\n\\begin{lemma}\\label{lemma-neighbour-1}\n\t Let $\\bfw^*$ be a \\ts\\  with $0<\\tau<\\tau_*:=\\min\\{\\tau_1,\\tau_2\\}$ of the problem (\\ref{ell-0-1}),  $\\tau_1,\\tau_2$ and $\\Gamma_*$  be given by (\\ref{tau-*}) and (\\ref{T-*}).  Then there is a $\\delta_1^*>0$ such that, for any $\\bfw\\in N(\\bfw^*,\\delta_1^*)$ with its associated indices $\\S$ and $\\E^o$, it holds \n \\beq\\label{gw*-0} \n F(\\bfw^*;T)=0 ~~~~{\\rm and}~~~T:=(\\S\\cup\\E^o)\\subseteq \\Gamma_*.\n \\eeq\n\\end{lemma}\n\n\\begin{proof} i) \\cref{cor-p-staionary} states that the \\ts\\ $\\bfw^*$ of (\\ref{ell-0-1}) satisfies\n\\begin{eqnarray}\n\\label{sta-eq-1-*}\n\\nabla f(\\bfx^*)  + A_{\\Gamma_*} \\bfz^*_{\\Gamma_*}=0,~~\n \\bfy^*_{\\Gamma_*}=0, ~~\n\\bfz^*_{\\overline \\Gamma_*}  =0 \n\\end{eqnarray} \n for $0<\\tau<\\tau_*$, where $\\Gamma_*=\\S_*\\cup\\E^o_*$. Note that $\\E_*\\setminus\\E^o_*\\subseteq \\overline{\\Gamma}_*$ which by \\eqref{sta-eq-1-*} leads to\n\\begin{eqnarray} \\label{sta-eq-1-*-z}\n   \\bfz^*_{\\E_*\\setminus\\E^o_*}=0.\n\\end{eqnarray} \nUsing the same reasoning for proving \\eqref{non-negative},  \nwe can prove for $ 0<\\tau<\\tau_*= \\min\\{\\tau_1,\\tau_2\\}$ in \\eqref{tau-*} that\n\\begin{eqnarray} \\label{yz-tau-lambda} \n y_i^*  >\\sqrt{2\\tau\\lambda}  ~~{\\rm if}~~y_i^*> 0,~~~~~~ \\tau z^*_i <   \\sqrt{2\\tau \\lambda   } ~~{\\rm if}~~z^*_i> 0.\n\\end{eqnarray}\nTherefore,  we have the following facts {\n\\allowdisplaybreaks\n\\begin{eqnarray*}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n\\E^o_*&=& \\{i\\in\\N_m: y_i^*=0, \\tau z_i^* \\in \\{0, \\sqrt{2\\tau\\lambda}\\} \\}&~~(\\text{by} ~\\eqref{p-stationary-T})\\\\\n&=& \\{i\\in\\N_m: y_i^*=0, z_i^*=0 \\}, &~~(\\text{by}~\\eqref{yz-tau-lambda})\\\\\n\\E_*\\setminus\\E^o_*&=& \\{i\\in\\N_m: y_i^*\\neq0, y_i^*+\\tau z_i^* = \\sqrt{2\\tau\\lambda } \\}&~~(\\text{by} ~\\eqref{p-stationary-T})\\\\\n &=& \\{i\\in\\N_m: y_i^* = \\sqrt{2\\tau\\lambda} \\}&~~(\\text{by}~\\eqref{sta-eq-1-*-z})\\\\\n&=&\\emptyset, &~~(\\text{by}~\\eqref{yz-tau-lambda})\n \\end{array}\\end{eqnarray*}}\nThese facts lead to\n\\begin{eqnarray}\\label{E*0}\n\\E_*=\\E^o_*\\cup(\\E_*\\setminus\\E^o_*)=\\E^o_*\n=\\Big\\{i\\in\\N_m: y_i^*= z_i^*=0\\Big\\},\n\\end{eqnarray}\nwhich yields the following relations\n\\begin{eqnarray}\\label{TSETO}\n\\Gamma_*=\\S_*\\cup \\E_*^o=\\S_*\\cup \\E_*,~~~~\\overline \\Gamma_*= \\O_*. \n\\end{eqnarray}   \n {For a sufficiently small $\\delta_1^*$, any $\\bfw\\in N(\\bfw^*,\\delta_1^*)$ satisfies,\n\\allowdisplaybreaks\\begin{eqnarray}\\label{hz-h*z*}\n&&  {| y_i +\\tau z_i - y_i^* -\\tau z^*_i |\\leq c\\delta_1^*},~~\\forall~i\\in\\N_m,   \n\\end{eqnarray} \n {where $c>0$ is a constant relied on $A$ and $\\tau$.} The definitions of $\\S$ and $\\S_*$  in \\eqref{p-stationary-T} mean that, for any $i\\in\\S$ or $i\\in\\S_*$,\n\\begin{eqnarray} \\label{facts-S}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n y_i+\\tau z_i\\in (0, \\sqrt{2\\tau\\lambda})~~ &\\Longleftrightarrow&~~  |y_i+\\tau z_i-\\sqrt{\\tau\\lambda/2} |< \\sqrt{\\tau\\lambda/2},\\\\\ny_i^*+\\tau z_i^*\\in (0, \\sqrt{2\\tau\\lambda})~~ &\\Longleftrightarrow&~~  |y_i^*+\\tau z_i^*-\\sqrt{\\tau\\lambda/2} |< \\sqrt{\\tau\\lambda/2}.\n\\end{array}\n\\end{eqnarray} \nUsing this fact, if $\\S_* \\nsubseteq \\S$, then there is an $i\\in \\S_*$ but $i\\notin\\S$ such that\n\\allowdisplaybreaks\n\\begin{eqnarray*}  \n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n|y_i +\\tau z_i - y_i^* -\\tau z^*_i|\n&=&|y_i+\\tau z_i-\\sqrt{\\tau\\lambda/2}  - y_i^*-\\tau z^*_i+\\sqrt{\\tau\\lambda/2}|\\nonumber\\\\\n&\\geq&   \\sqrt{\\tau\\lambda/2} -|y_i^*+\\tau z^*_i-\\sqrt{\\tau\\lambda/2}| ~~~~ (\\text{by}~i\\notin\\S~\\text{and}~\\eqref{facts-S})\\\\\n  &\\geq&   \\sqrt{\\tau\\lambda/2} -\\max_{i\\in\\S_*} |y_i^*+\\tau z^*_i-\\sqrt{\\tau\\lambda/2}| =:\\delta_s\\\\\n&>&0, ~~~~(\\text{by}~\\eqref{facts-S})\n\\end{array}\n\\end{eqnarray*}  {Since $c\\delta_1^*$} can be smaller than $\\delta_s$, the above fact contradicts with \\eqref{hz-h*z*}.  }Hence, it holds $\\S_* \\subseteq \\S$. Similar reasoning also derives $\\O_* \\subseteq \\O$. These allow us to obtain $$\\E = \\N_m \\setminus (\\S\\cup\\O)\\subseteq \\N_m \\setminus (\\S_*\\cup\\O_*)= \\E_*.$$\nOverall, for any $\\bfw\\in N(\\bfw^*,\\delta_1^*)$, it holds\n\\begin{eqnarray}\\label{SSOOEE}\n\\S_* \\subseteq \\S,~~ \\O_* \\subseteq \\O,~~ \\E\\subseteq \\E_*.\n\\end{eqnarray} \nThe above relations enable us to claim that \n\\begin{eqnarray}\\label{SS*E*}\n(\\S\\setminus \\S_*) \\subseteq \\E_*,~~~~(\\O\\setminus \\O_*) \\subseteq \\E_*.\n\\end{eqnarray}\nNow, we can show\n\\allowdisplaybreaks\n\\begin{eqnarray*} \n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n\\bfy^*_{ \\S_*}&=&0,&~~(\\text{by}~\\S_*\\subseteq \\Gamma_*~\\text{and}~\\eqref{sta-eq-1-*})\\\\\n\\bfy^*_{\\S\\setminus \\S_*} &=&0, &~~(\\text{by}~ \\S\\setminus \\S_*\\subseteq\\E_*~\\text{from}~\\eqref{SS*E*}~\\text{and}~\\eqref{E*0})\\\\\n\\bfy^*_{\\E^o} &=&0,&~~(\\text{by}~ \\E^o\\subseteq\\E\\subseteq\\E_*~\\text{from}~\\eqref{SSOOEE}~\\text{and}~\\eqref{E*0})\\\\\n\\bfz^*_{ \\O_*}  &=&0, &~~(\\text{by}~\\O_*\\subseteq \\overline \\Gamma_*~\\text{and}~\\eqref{sta-eq-1-*})\\\\\n\\bfz^*_{\\O\\setminus \\O_*} &=&0, &~~(\\text{by}~ \\O\\setminus \\O_*\\subseteq\\E_*~\\text{from}~\\eqref{SS*E*}~\\text{and}~\\eqref{E*0})\\\\\n\\bfz^*_{\\E\\setminus\\E^o} &=&0.&~~(\\text{by}~ \\E\\setminus\\E^o\\subseteq\\E\\subseteq\\E_*~\\text{from}~\\eqref{SSOOEE}~\\text{and}~\\eqref{E*0})\n\\end{array}\n\\end{eqnarray*}\nThese conditions combining with\n\\begin{eqnarray*}T&=&\\S\\cup\\E^o=\\S_*\\cup(\\S\\setminus \\S_*)\\cup\\E^o,\\\\\n \\overline T&=&\\O\\cup(\\E\\setminus\\E^o)=\\O_*\\cup(\\O\\setminus \\O_*)\\cup(\\E\\setminus\\E^o),\n \\end{eqnarray*}\nimply $\\bfy^*_{ T} =0$ and $\\bfz^*_{\\overline T}=0$. \nAs a consequence of this and $\\bfz^*_{\\overline \\Gamma_*}=0$ from \\eqref{sta-eq-1-*}, \n\\begin{eqnarray*}\n  0= f(\\bfx^*)+ A_{\\Gamma_*} ^\\top\\bfz^*_{\\Gamma_*}=f(\\bfx^*)+ A^\\top\\bfz^*= f(\\bfx^*)+ A_{T} ^\\top\\bfz^*_{T}.\n \\end{eqnarray*}\nOverall, we verify $F(\\bfw^*;T)=0$,  as desired. \nFinally, we observe that\n \\beq \n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n \\begin{array}{rcll}\nT=\\S\\cup\\E^o&=&\\S_*\\cup(\\S\\setminus \\S_*)\\cup\\E^o\\nonumber\\\\\n&\\subseteq&   \\S_*\\cup\\E_* \\cup\\E^o&(\\text{by}~ \\S\\setminus \\S_*\\subseteq\\E_*~\\text{from}~\\eqref{SS*E*})\\nonumber\\\\\n&\\subseteq&   \\S_*\\cup\\E_* \\cup \\E_*  &(\\text{by}~\\E^o\\subseteq\\E\\subseteq\\E_*~\\text{from}~\\eqref{SSOOEE})\\nonumber\\\\\n&=&   \\Gamma_*.  &(\\text{by}~\\eqref{TSETO})\\nonumber\n\\end{array}\n \\eeq\nThe whole proof is completed.   \\end{proof}\n\n\\begin{lemma}\\label{lemma-neighbour-2} Let $\\bfw^*$ be a \\ts\\  with $0<\\tau<\\tau_*$ of the problem (\\ref{ell-0-1}) and $\\tau_*$ be given by (\\ref{tau-*}).   Assume  \\cref{ass} and \\cref{ass2}. It holds \n \\beq \\label{bd-jacobian-F-w-T}\n  && C_*  \\geq \\|\\nabla F_{\\mu}(\\bfw;T)\\|\\geq\\sigma_{\\min}( \\nabla F_{\\mu}(\\bfw;T)) \\geq   c_*>0, \n\\eeq\nfor any $\\bfw\\in N(\\bfw^*,\\delta_2^*)$ and any $0\\leq\\mu\\leq c_*/2$, where $T=\\S\\cup\\E^o$ and\n \\beq\\label{radius-2}\nC_*&:=&2\\max\\{1, \\|H(\\Gamma_*)\\|\\} ,~~~~ c_* := 0.5{\\min\\{1,\\min_{\\Gamma\\subseteq\\Gamma_*}\\sigma_{\\min}(H(\\Gamma))\\}}, \\\\\n\\label{H*}\\delta_2^* &:=& \\min\\left\\{\\delta_1^*,   \\frac{c_*}{ 2L_*}     \\right\\},~~~~H(\\Gamma):=\\left[  \n     \\begin{array}{ccc}\n    \\nabla^2 f(\\bfx^*)& A^\\top_{\\Gamma}\\\\\n     A_{\\Gamma}&0\n     \\end{array}\n     \\right]. \n\\end{eqnarray}\n\\end{lemma}\n\n\\begin{proof}  \nSince  $\\nabla^2 f(\\bfx^*)$ is  {positive definite} and $A_{\\Gamma_*}$ is full row rank by \\cref{ass}, $A_\\Gamma$ is full row rank for any $ \\Gamma \\subseteq   \\Gamma_*$ and thus $H(\\Gamma)$ is non-singular. \nWe have $\\sigma_{\\min}(H(\\Gamma)) > 0$ for any \n$ \\Gamma \\subseteq   \\Gamma_*$ \nand  $c_*>0$. \nNow we build the bounds of $\\nabla F_{\\mu}(\\bfw;T)$. \nFor any given two matrices $D'$ and $D$, we have the first fact\n\\begin{eqnarray} \\label{lower-bd-HH}\n \\|D'-D\\|\n&\\geq& \\max_{i}|\\sigma_i(D')-\\sigma_i(D )|   \\geq |\\sigma_{i_0}( D')-\\sigma_{i_0}(D)|\\nonumber\\\\\n&\\geq& \\sigma_{i_0}(D')-\\sigma_{\\min}(D) \\geq \\sigma_{\\min}(D')-\\sigma_{\\min}(D), \n\\end{eqnarray}\nwhere the first inequality is from \\cite[Reminder (2), on Page 76]{lutkepohl1996handbook} and  $i_0$ satisfies that $\\sigma_{i_0}(D)=\\sigma_{\\min}(D)$.  Recall that\n \\begin{eqnarray*}\n \\label{FH*-FG}H(\\Gamma)\\overset{\\eqref{H*}}{=}\\left[  \n     \\begin{array}{ccc}\n    \\nabla^2 f(\\bfx^*)& A^\\top_{\\Gamma }\\\\\n     A_{\\Gamma }&0\n     \\end{array}\n     \\right],~~~~ \\nabla F(\\bfw^*;T) =\\left[  \n     \\begin{array}{cc}\n    H(T)&0\\\\\n     0&I  \n     \\end{array}\n     \\right].\n    \\end{eqnarray*}\nFor any $\\bfw\\in N(\\bfw^*,\\delta_2^*)$ and $\\delta_2^*\\leq\\delta_1^*$,   \\cref{lemma-neighbour-1} contributes to $T \\subseteq   \\Gamma_*$. So $H(T)$ is a submatrix of $H(\\Gamma_*)$. Hence, %It follows from \\cite[Theorem 1]{thompson1972principal} that the maximum  singular value of a  matrix is no less than the maximum singular value of its sub-matrix. Namely, \n\\beq \n\\|H(T)\\| \\leq \\|H(\\Gamma_*)\\|,~~ \\sigma_{\\min}(H(T)) \\geq \\min_{\\Gamma \\subseteq \\Gamma_*} \\sigma_{\\min} (H(\\Gamma)),\\nonumber\n\\eeq\nwhere the latter is by $T \\subseteq   \\Gamma_*$, which gives us the second fact \n\\beq\\label{JJogG0}\n\\sigma_{\\min}(\\nabla F(\\bfw^*;T))&=&\\min\\{1,\\sigma_{\\min}(H(T))\\}\\\\\n& \\geq&\\min\\{1, \\min_{\\Gamma \\subseteq \\Gamma_*} \\sigma_{\\min} (H(\\Gamma))\\}=2c_*, \\nonumber\\\\\n\\label{JJogG1}\n\\|\\nabla F(\\bfw^*;T)\\| &=& \\max\\{1, \\|H(T)\\|\\} \\leq \\max\\{1, \\|H(\\Gamma_*)\\|\\} =C_*/2. \n\\eeq\nThe locally Lipschitz continuity of $\\nabla^2 f$  around $\\bfx^*$ with $L_*$  yields the third fact,\n\\begin{eqnarray}\n\\label{HH*2}\n\\|\\nabla F (\\bfw^*;T)-\\nabla F (\\bfw;T)\\|&=&\\|\\nabla^2  f (\\bfx)-\\nabla^2  f (\\bfx^*)\\|  \\leq\n   L_* \\| \\bfx -\\bfx^* \\| \\nonumber\\\\\n   &\\leq& L_* \\| \\bfw -\\bfw^* \\| \n \\leq   L_* \\delta_2^*\n\\leq c_*/2.   \\hspace{.5cm}(\\text{by}~\\eqref{H*})\n\\end{eqnarray} \nNow these three facts allow us to derive\n\\allowdisplaybreaks \n\\begin{eqnarray}\\label{lower-bd-FF}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rclr}\n&& \\sigma_{\\min}(\\nabla F_{\\mu}(\\bfw;T))\\\\\n &\\geq& \\sigma_{\\min}(\\nabla F (\\bfw ;T))- \\|\\nabla F(\\bfw;T)-\\nabla F_{\\mu}  (\\bfw;T)\\| &(\\text{by}~\\eqref{lower-bd-HH})  \\\\\n&=& \\sigma_{\\min}(\\nabla F(\\bfw ;T))- \\mu &(\\text{by}~\\eqref{sta-eq-1}) \\\\\n&\\geq& \\sigma_{\\min}(\\nabla F(\\bfw^*;T))- \\|\\nabla F (\\bfw^*;T)-\\nabla F (\\bfw;T)\\|- \\mu   &(\\text{by}~\\eqref{lower-bd-HH}) \\\\\n&\\geq&\\sigma_{\\min}(\\nabla F(\\bfw^*;T))- c_*/2 - \\mu&(\\text{by}~\\eqref{HH*2}) \\\\\n&\\geq&\\sigma_{\\min}(\\nabla F(\\bfw^*;T))-  c_* &(\\text{by}~\\mu\\leq c_*/2)   \\\\\n&\\geq&c_*. &(\\text{by}~\\eqref{JJogG0})  \n\\end{array}\n\\end{eqnarray}\nSimilarly, we also have\n\\allowdisplaybreaks \n\\begin{eqnarray}\\label{upper-bd-FF}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rclr}\n&&\\|\\nabla F_{\\mu}(\\bfw;T)\\|\\\\\n &\\leq& \\|\\nabla F (\\bfw ;T)\\|+ \\|\\nabla F(\\bfw;T)-\\nabla F_{\\mu}  (\\bfw;T)\\| \\\\\n&=& \\|\\nabla F (\\bfw ;T)\\|+ \\mu &(\\text{by}~\\eqref{sta-eq-1})\\\\\n&\\leq& \\|\\nabla F(\\bfw^*;T)\\|+ \\|\\nabla F (\\bfw^*;T)-\\nabla F (\\bfw;T)\\|+ \\mu   &(\\text{by}~\\eqref{lower-bd-HH})  \\\\\n&\\leq& \\|\\nabla F(\\bfw^*;T)\\|+ c_*/2 + \\mu&(\\text{by}~\\eqref{HH*2}) \\\\\n&\\leq& \\|\\nabla F(\\bfw^*;T)\\|+ c_* &(\\text{by}~\\mu\\leq c_*/2)   \\\\\n&\\leq& C_*/2+c_* &(\\text{by}~\\eqref{JJogG1}) \\\\\n&\\leq& C_*. &(\\text{by}~c_*\\leq C_*/2)   \n\\end{array}\n\\end{eqnarray}\nThe whole proof is completed. \n\\end{proof}\n\n\\begin{proof} a) It is easily observed   \n\\begin{eqnarray}\\label{mu-k}\n0< \\mu_{k} \\leq \\mu_{k-1}, ~k=1,2,3,\\ldots, ~~~~{\\rm and}~~~~ \\lim_{k\\rightarrow\\infty}\\mu_k=0.\n\\end{eqnarray} \nIt follows from \\cref{lemma-neighbour-1} and the facts\n$\\delta_*\\leq \\delta_2^*\\leq\\delta_1^*$ and $\\bfw^0\\in N(\\bfw^*,\\delta_*)$ that     \n\\beq\\label{F-w*-T0-0}F (\\bfw^{*};T_0) =0  \n\\eeq\nwith $T_0=\\S_0\\cup\\E_0^o$ and from \\cref{lemma-neighbour-2} that     \n\\beq\\label{grad-F-w*-T0-c} C_*  \\geq \\|\\nabla F_{\\mu_0}(\\bfw^0;T_0)\\|\\geq\\sigma_{\\min}( \\nabla F_{\\mu_0}(\\bfw^0;T_0)) \\geq  c_*\n\\eeq\nHere, we used the fact that $\\mu_0\\leq \\mu_{-1}\\leq c_*/2$ by  \\eqref{mu-k}. \nFrom \\eqref{newton-dir-0}, we have\n\\beq\\label{dkt}\n\\nabla F_{\\mu_0} (\\bfw^{0};T_0)~\\bfd^{0}=- F (\\bfw^{0};T_0).\n\\eeq\n  \\cref{lemma-neighbour-2} states that $\\nabla F_{\\mu_0} (\\bfw^{0};T_0)$ is non-singular and thus $\\bfd^{0}$ is well defined.\nLet \n\\beq\\label{w0-beta-w*}\n\\bfw^{0}_\\beta= \\bfw^{*} + \\beta(\\bfw^{0}-\\bfw^{*})=(\\bfx^{0}_\\beta;\\bfz^{0}_\\beta)\n\\eeq\nwhere $\\beta\\in[0,1].$ One can easily check that $\\bfw^{0}_\\beta\\in N(\\bfw^*,\\delta_*)$ as $$\\|\\bfw^{0}_\\beta-\\bfw^{*}\\|= \\beta\\|\\bfw^{0}-\\bfw^{*}\\|\\leq  \\delta_*.$$\nThe definition  in \\eqref{sta-eq-1} enables us to obtain\n\\allowdisplaybreaks \n\\begin{eqnarray*}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n&&\\|\\nabla F_{\\mu_0} (\\bfw^{0};T_0)- \\nabla F (\\bfw^{0}_\\beta;T_0) \\|  \\\\\n&\\leq&\\|\\nabla^2 f(\\bfx^0)- \\nabla^2 f(\\bfx^0_\\beta)\\| + \\mu_0&(\\text{by}~\\eqref{sta-eq-1})  \\\\\n& \\leq& L_* \\|\\bfx^0-\\bfx^0_\\beta\\|+ \\rho  \\|F(\\bfw^0;T_0)\\|&(\\text{by}~\\eqref{update-mu}) \\\\ \n& \\leq& L_* \\|\\bfw^0-\\bfw^0_\\beta\\|+ \\rho \\|\\nabla F_{\\mu_0} (\\bfw^{0};T_0)\\| \\|\\bfd^0\\|&(\\text{by}~\\eqref{newton-dir-0}) \\\\ \n& \\leq& L_* \\|\\bfw^0-\\bfw^0_\\beta\\|+ \\rho C_* \\|\\bfw^1-\\bfw^0\\|&(\\text{by}~\\eqref{grad-F-w*-T0-c}) \\\\ \n& \\leq& L_*(1-\\beta)\\|\\bfw^0-\\bfw^*\\|+ \\rho C_* (\\|\\bfw^1-\\bfw^*\\|+\\|\\bfw^0-\\bfw^*\\|) &(\\text{by}~\\eqref{w0-beta-w*})  \\\\ \n& =& (L_*(1-\\beta)+\\rho C_*)\\|\\bfw^0-\\bfw^*\\|+ \\rho C_* \\|\\bfw^1-\\bfw^*\\|.\n\\end{array} \\end{eqnarray*}\nDenote $\\Theta_{\\mu_0}:=\\nabla F_{\\mu_0} (\\bfw^{0};T_0)$ and $\\Theta(\\beta):= \\nabla F (\\bfw^{0}_\\beta;T_0)$.  The above condition yields\n  \\beq\\label{lipschitz}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n \\int_0^1 \\|\\Theta_{\\mu_0}  - \\Theta(\\beta)\\|d\\beta\n\\leq (L_*/2+\\rho C_*)\\|\\bfw^0-\\bfw^*\\|+ \\rho C_* \\|\\bfw^1-\\bfw^*\\|.\n\\end{array} \\eeq\nFor the fixed $T_0$, the function $F(\\cdot,T_0)$ is differentiable, which by \\eqref{F-w*-T0-0} derives\n\\beq\\label{mean-value}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\nF (\\bfw^{0};T_0)=F (\\bfw^{*};T_0)+\\int_0^1 \\Theta(\\beta)(\\bfw^{0}-\\bfw^{*})d\\beta=\\int_0^1 \\Theta(\\beta) (\\bfw^{0}-\\bfw^{*})d\\beta.\n\\end{array}\\eeq\nNow the following chain of inequalities holds.\n\\begin{eqnarray*}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\nc_*\\|\\bfw^{1}-\\bfw^{*}\\| &=& c_*\\|\\bfw^{0}+\\bfd^{0}-\\bfw^{*}\\|\\\\\n& {=} &c_*\\|\\bfw^{0}-\\bfw^{*}-\\Theta_{\\mu_0}^{-1}   F(\\bfw^{0};T_0) \\|&~~(\\text{by}~\\eqref{dkt})\\\\\n& {\\leq} & \\|\\Theta_{\\mu_0} (\\bfw^{0}-\\bfw^{*})-   F(\\bfw^{0};T_0) \\|&~~(\\text{by}~\\eqref{grad-F-w*-T0-c})\\\\\n&= & \\|\\Theta_{\\mu_0}  (\\bfw^{0}-\\bfw^{*})-  \\int_0^1 \\Theta(\\beta)(\\bfw^{0}-\\bfw^{*})d\\beta \\| &~~(\\text{by}~\\eqref{mean-value}) \\\\\n& \\leq& \\int_0^1 \\|\\Theta_{\\mu_0} -\\Theta(\\beta) \\|\\|\\bfw^{0}-\\bfw^{*}\\|d\\beta \\\\\n& =& ({\\theta_*}/2) \\|\\bfw^{0} -\\bfw^{*}\\|^2+   {\\rho C_*}  \\|\\bfw^1-\\bfw^*\\|\\|\\bfw^{0}-\\bfw^{*}\\| &~~(\\text{by}~\\eqref{lipschitz}) \\\\\n&\\leq& ({\\theta_*}/2) \\|\\bfw^{0} -\\bfw^{*}\\|^2+ {\\rho C_*\\delta_*}  \\|\\bfw^1-\\bfw^*\\|   \\\\\n&\\leq& ({\\theta_*}/2) \\|\\bfw^{0} -\\bfw^{*}\\|^2+ (c_*/2) \\|\\bfw^1-\\bfw^*\\|, \n\\end{array}\n\\end{eqnarray*}\nwhere $\\theta_*:=L_* +2\\rho C_*$ and the last inequality is from $\\delta_*\\leq  {c_*}/(2 \\theta_*)$ by \\eqref{delta-*} and\n\\beq\\label{w0-w*-delta*}\n\\|\\bfw^{0}-\\bfw^{*}\\|<\\delta_*\\leq  {c_*}/(2 \\theta_*)<  {c_*}/({2\\rho  C_*}).\n\\eeq\n The above chain of inequalities  suffices to the following fact\n\\beq\\label{quadratic-chain}\n\\|\\bfw^{1}-\\bfw^{*}\\|\\leq  (\\theta_*/c_*) \\|\\bfw^{0} -\\bfw^{*}\\|^2.\n\\eeq\nThis together with $\\|\\bfw^{0}-\\bfw^{*}\\|<\\delta_*$ and \\eqref{w0-w*-delta*} derives\n\\beq \n\\|\\bfw^{1}-\\bfw^{*}\\|  &\\leq& \n   (\\theta_*/c_*)  \\delta_* \\|\\bfw^{0}-\\bfw^{*} \\|\\leq (1/2)\\|\\bfw^{0}-\\bfw^{*} \\|< \\delta_*,  \\nonumber\n \\eeq\nwhich means $\\bfw^1\\in N(\\bfw^*,\\delta_*)$. In addition, $\\mu_1\\leq\\mu_0\\leq c_*/2$ by  \\eqref{mu-k}. Hence, replacing $T_0 $ by $T_1$, the same reasoning allows us to show that $\\bfd^{1}$ is well defined and $$\\|\\bfw^{2}-\\bfw^{*}\\|  \\leq (\\theta_*/c_*)   \\|\\bfw^{1}-\\bfw^{*} \\|^2 .$$ By the induction, we can conclude that $\\bfw^k\\in N(\\bfw^*,\\delta_*)$, $\\bfd^{k}$ is well defined and \n\\beq\\label{quadratic-rate}\n\\hspace{10mm}\\|\\bfw^{k+1}-\\bfw^{*}\\|  &\\leq& (\\theta_*/c_*) \\|\\bfw^{k}-\\bfw^{*} \\|^2, \\\\\n\\label{quadratic-rate-1}&\\leq& (\\theta_*/c_*)  \\|\\bfw^{k}-\\bfw^{*} \\|\\delta_* \\leq (1/2) \\|\\bfw^{k}-\\bfw^{*} \\|.\\hspace{.2cm}(\\text{by}~\\eqref{w0-w*-delta*})\n\\eeq\nTherefore, \\eqref{quadratic-rate} claims b). The conclusion of a) can be made by \\eqref{quadratic-rate-1}  that   $$\\bfw^{k}\\rightarrow\\bfw^{*},~~ \\bfd^k=\\bfw^{k+1}-\\bfw^{k}=\\bfw^{k+1}-\\bfw^{*}+\\bfw^{*}-\\bfw^{k}\\rightarrow0.$$ \n\nc)  The  above  proof shows $\\bfw^k\\in N(\\bfw^*,\\delta_*)$ and hence \\eqref{gw*-0} results in\n\\beq\\label{wk-in-N}  F(\\bfw^{*};T_k){=}0,\\eeq\nwhere $T_k=\\S_k\\cup\\E_k^o$.\nBy letting $\\bfw^{k}_\\beta= \\bfw^{*} + \\beta(\\bfw^{k}-\\bfw^{*}),$ where $\\beta\\in[0,1]$, we have $\\bfw^k_\\beta\\in N(\\bfw^*,\\delta_*)$. To show \\eqref{bd-jacobian-F-w-T} in \\cref{lemma-neighbour-2}, we verified the lower and upper bounds by \\eqref{lower-bd-FF} and \\eqref{upper-bd-FF}. Similarly, we can prove these bounds hold for $\\nabla F (\\bfw^{k}_\\beta;T_k)$. (In fact, since $\\bfw^k_\\beta\\in N(\\bfw^*,\\delta_*)\\subseteq N(\\bfw^*,\\delta_2^*)$ by $\\delta_*\\leq \\delta_2^*$, one just needs to set $\\mu=0$ and $\\bfw= \\bfw^{k}_\\beta$ in  \\eqref{lower-bd-FF} and \\eqref{upper-bd-FF}. Therefore,\n\\beq\\label{C-F-w-beta-c}\nC_*  \\geq \\|\\nabla F (\\bfw^{k}_\\beta;T_k)\\|\\geq\\sigma_{\\min}( \\nabla F (\\bfw^{k}_\\beta;T_k)) \\geq  c_*, \n\\eeq\nAgain, the function $F(\\cdot,T_k)$ is differentiable for the fixed $T_k$, so the Mean-value theorem states that there is a $\\beta_0\\in(0,1)$ satisfying\n\\beq\\label{FkTk}\n\\|F(\\bfw^{k},T_k)\\|&=&\\|F(\\bfw^{*};T_k)+  \\nabla F (\\bfw^{k}_{\\beta_0};T_k) (\\bfw^{k}-\\bfw^{*})\\|\\nonumber\\\\\n&=&\\| \\nabla F (\\bfw^{k}_{\\beta_0};T_k) (\\bfw^{k}-\\bfw^{*}) \\|\\hspace{1.7cm}(\\text{by}~\\eqref{wk-in-N})\\nonumber\\\\\n&\\in&  [~c_*\\|\\bfw^{k}-\\bfw^{*} \\|, C_*\\|\\bfw^{k}-\\bfw^{*} \\|~], \\hspace{0.8cm}(\\text{by}~\\eqref{C-F-w-beta-c})\n\\eeq\nThis contributes to \n\\allowdisplaybreaks\n\\begin{eqnarray*}\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rclr}\n\\|F(\\bfw^{k};T_{k})\\| \\leq  C_*\\|\\bfw^{k}-\\bfw^{*}\\|  \n & {\\leq} & (\\theta_*C_*/c_*)   \\|\\bfw^{k-1}-\\bfw^{*} \\|^2 &(\\text{by}~\\eqref{quadratic-rate}) \\\\\n & {\\leq}& (\\theta_*C_*/c_*^{3})  \\|F(\\bfw^{k-1};T_{k-1})\\|^2&(\\text{by}~\\eqref{FkTk})\\\\\n &{\\leq}& (\\theta_*C_*^3/c_*^{3})      \\|\\bfw^{k-1}-\\bfw^{*} \\|^2&(\\text{by}~\\eqref{FkTk})\\\\\n& {\\leq}& (\\theta_*C_*^3/c_*^{3})  2^{-2}  \\|\\bfw^{k-2}-\\bfw^{*} \\|^2&~~(\\text{by}~\\eqref{quadratic-rate-1})\\\\\n    &\\vdots&\\\\\n & \\leq& (\\theta_*C_*^3/c_*^{3})   2^{2-2k} \\|\\bfw^{0}-\\bfw^{*} \\|^2,&(\\text{by}~\\eqref{quadratic-rate-1})\n \\end{array}  \\end{eqnarray*}\nwhere the third inequality yields the first conclusion in c). This also enable to verify that $\\|F(\\bfw^{k};T_k)\\|$ $< {\\epsilon}$ if $k$ satisfies \\eqref{k-tol}.\n%$$k\\geq  \\lceil \\log_2 (2 \\sqrt{\\theta_* (C_* /c_*)^3}\\|\\bfw^{0}-\\bfw^{*} \\| )-\\log_2(\\sqrt{\\epsilon}) \\rceil.$$ \nThe whole proof is completed.\n\\end{proof}","units":[{"start":18619,"end":19518,"start_line":322,"end_line":352,"raw_tex":"\\begin{lemma} \\label{subdifferentials}\n\n\\begin{itemize}\n\t\n \\item[(i)] We have\n \\begin{eqnarray}\\label{subfi}\n  \\partial\\|{\\bfy}_+\\|_0\n  =\\left\\{{\\bfv}\\in {\\R}^m: v_i\\left\\{\\begin{array}{ll}\n \\geq0,& y_i=0,\\\\\n =0,& y_i\\neq0,\n \\end{array}\n \\right. i\\in{\\N}_m\\right\\}. \n \\end{eqnarray}\n \n \\item[(ii)] Let $g(\\bfx):=\\|(h(\\bfx))_+\\|_0$, where $h:\\R^n\\rightarrow\\R^m$ is differentiable,  and $\\Gamma:=\\{i\\in\\N_m: h_i(  \\bfx)=0\\}$ for a given a point $ \\bfx\\in\\R^n$.  If \n \\begin{eqnarray}\n \\label{chain-rule-cond}\n \\left.\n \\begin{array}{r}\n \\forall \\bfz\\in\\R^{|\\Gamma|},~~\\bfz\\geq 0,~~\n (\\nabla h( \\bfx))_{\\Gamma}^\\top \\bfz = 0\n \\end{array}\\right.~~\\Longrightarrow~~\\bfz = 0,\n \\end{eqnarray}\n then the subdifferential of $g(\\bfx)$  at $  \\bfx$ is  \n \\begin{eqnarray}\n \\label{chain-rule-sub} \n \\partial g(  \\bfx)=\\nabla h( \\bfx)^\\top \\partial \\|(h(  \\bfx))_+\\|_0.\n \\end{eqnarray}\n\n\\end{itemize}\n\\end{lemma}","sha256_utf8":"0219cc4baee5412543f6d4fc134ca133340bca31a072d0a1ba7918e5293c779c"},{"start":19520,"end":22233,"start_line":354,"end_line":384,"raw_tex":"\\begin{proof}\ni)   It is straight to check that the regular subdifferential $\\widehat{\\partial}\\|{\\bfy}_+\\|_0$ (see \\cite[Definition 8.3]{RW1998})  of $\\|{\\bfy}_+\\|_0$ at ${\\bfy}$ takes the following form,\n\\begin{eqnarray}\\label{subfi-0}\n \\widehat{\\partial}\\|{\\bfy}_+\\|_0=\\left\\{{\\bfv}\\in {\\R}^m: v_i\\left\\{\\begin{array}{ll}\n \\geq0,& y_i=0,\\\\\n =0,& y_i\\neq0,\n \\end{array}\n \\right. i\\in{\\N}_m\\right\\}=:\\Omega(\\bfy). \n\\end{eqnarray}\nWe next verify $ \\partial\\|{\\bfy}_+\\|_0= \\Omega({\\bfy}) $. By letting $\\varphi(\\cdot):=\\|(\\cdot)_+\\|_0$, we have\n  \\begin{eqnarray}\\label{sub-lim-reg}\n  \\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n  \\begin{array}{rclll}\n\\partial\\|{\\bfy}_+\\|_0&=&\\limsup_{{\\bfz}\\stackrel{\\varphi}{\\rightarrow}{\\bfy}}~ \\widehat{\\partial}\\|{\\bfz}_+\\|_0 ~~~( \\text{by  \\cite[Equation 8(5)]{RW1998}})\\\\\n&=&\\limsup_{{\\bfz}\\stackrel{\\varphi}{\\rightarrow}{\\bfy}}~\\Omega({\\bfz}) ~~~~~~( \\text{by \\eqref{subfi-0}})\\\\\n&=&\\{\\bfv\\in \\R^m:\\exists~\\bfz^k\\stackrel{\\varphi}{\\rightarrow} \\bfy,~\\bfv^k\\rightarrow \\bfv ~\\text{with}~\\bfv^k\\in \\Omega(\\bfz^k) \\}=:\\Theta,\n\\end{array}\n \\end{eqnarray}\nwhere  $\\bfz\\stackrel{\\varphi}{\\rightarrow} \\bfy$ represents $\\bfz\\rightarrow \\bfy, \\varphi(\\bfz)\\rightarrow \\varphi(\\bfy)$. Clearly, $\\Omega({\\bfy})\\subseteq\\Theta$. On the other hand, $\\Theta \\subseteq \\Omega({\\bfy})$  follows from that $\\Omega(\\bfz^k)\\subseteq \\Omega(\\bfy)$ for any $\\bfz^k\\stackrel{\\varphi}{\\rightarrow} \\bfy$ and $\\Omega(\\cdot)$ is closed.\n \n ii) Direct verifications yield the following chain of equations,\n \\begin{eqnarray*}  \\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n  \\begin{array}{rclll}\n \\partial^\\infty \\|\\bfy_+\\|_0&=& {\\limsup}_{\\sigma\\downarrow0,~{\\bfz}\\stackrel{\\varphi}{\\rightarrow} {\\bfy}} \\sigma \\widehat{\\partial}\\|\\bfz_+\\|_0&~~( \\text{by \\cite[Equation 8(5)]{RW1998}})\\\\ \n&=&\n {\\limsup}_{{\\bfz}\\stackrel{\\varphi}{\\rightarrow} {\\bfy}}~ \\widehat{\\partial}\\|\\bfz_+\\|_0&~~( \\text{by \\eqref{subfi-0}})\\\\\n & =& \\partial\\|{\\bfy}_+\\|_0,&~~( \\text{by \\eqref{sub-lim-reg}})\\\\\n \\end{array}\n \\end{eqnarray*}\n where $ \\partial^\\infty \\|{\\bfy}_+\\|_0$ is the horizon subdifferential of $\\|{\\bfy}_+\\|_0$. Therefore, we derive that $\\widehat{\\partial}\\varphi=\\partial \\varphi=\\partial^\\infty \\varphi$.  One can easily prove that the horizon cone $\\widehat{\\partial}\\varphi(\\bfy)^\\infty$ (see \\cite[Definition 3.3]{RW1998}) of $ \\widehat{\\partial}\\varphi(\\bfy)$ satisfies $\\widehat{\\partial}\\varphi(\\bfy)^\\infty = \\partial^ \\infty\\varphi(\\bfy)$. These conditions indicate that  the function $\\varphi$ is regular by \\cite[Corollary 8.11]{RW1998}, which together with \\eqref{chain-rule-cond}, $g(\\bfx)=\\varphi(h(\\bfx))$ and \\cite[Theorem 10.6]{RW1998} derives \\eqref{chain-rule-sub} immediately. \n\\end{proof}","sha256_utf8":"cfde96f254c45f36f8556154d7ce75fc11b729e81d0bd10c8b8876d164cbeeaf"},{"start":24225,"end":24765,"start_line":425,"end_line":437,"raw_tex":"\\begin{lemma}\\label{first-order-nec}\n\tThe following relationships hold for the problem (\\ref{ell-0-1}).\n\\begin{itemize}\n\t\n\\item[i)] A local minimizer $\\bfx^*$   satisfies the following condition \n         if  $A_{\\Gamma_*}$ is full row rank,\n\\begin{eqnarray} \\label{lemma-1st-order-nec}\n    -\\nabla f(\\bfx^*) \\in A^\\top \\partial \\|(A\\bfx^*+\\bfb)_+\\|_0.\n\\end{eqnarray}\n \n\\item[ii)] A point $\\bfx^*$ satisfying (\\ref{lemma-1st-order-nec})  is a local minimizer if  the function $f$ is locally convex around $\\bfx^*$.\n\\end{itemize}\n\\end{lemma}","sha256_utf8":"baf90a44ff00a9cac72d08f8aa2feec7c1b31e5b5375e4c8bf85bd1bce63647a"},{"start":24828,"end":30204,"start_line":439,"end_line":519,"raw_tex":"\\begin{proof}\ni) It follows from \\cite[Theorem 10.1]{RW1998} that a local minimizer of \\eqref{ell-0-1} must satisfy $-\\nabla f(\\bfx^*)\\in\\lambda\\partial g(\\bfx^*)$, where $g(\\bfx):=\\|(A\\bfx+\\bfb)_+\\|_0$. This together with \\cref{chain-rule-sub}  and $\\lambda \\partial\\|(\\cdot)_+\\|_0 = \\partial \\|(\\cdot)_+\\|_0$ by \\eqref{subfi} derives the result immediately.  \n\nii)   Since the problem \\eqref{ell-0-1} is equivalent to the following problem, \n\\begin{eqnarray}\\label{ell-0-1-equ-0}\n&\\underset{\\bfx\\in\\R^n,\\bfy\\in\\R^m}{\\min}& f(\\bfx)+\\lambda \\|\\bfy_+\\|_0,\\\\\n  &{\\rm s.t.}& A\\bfx+\\bfb-\\bfy = 0,\\nonumber\n\\end{eqnarray}\nit suffices to show that  $(\\bfx^*;\\bfy^*)$  is a local minimizer of the   problem \\eqref{ell-0-1-equ-0}, where $\\bfx^*$ satisfies \\eqref{lemma-1st-order-nec} and $\\bfy^*=A \\bfx^*+\\bfb$, namely, there is a $\\bfz^*$ such that\n\\begin{eqnarray}\n\\label{lemma-1st-order-nec-10}\n\\nabla f(\\bfx^*) +A^\\top\\bfz^*=0,~~ \nA \\bfx^*+\\bfb-\\bfy^*=0, ~~\n\\partial\\|\\bfy^*_+\\|_0\\ni \\bfz^*. \n\\end{eqnarray} \nIt follows from $\\bfy^*=A\\bfx^*+\\bfb$ and \\eqref{T-*} that $\\Gamma_*=\\{i\\in\\N_m: y^*_i=0\\}$. This together with $\\partial\\|\\bfy^*_+\\|_0 \\ni \\bfz^*$ and the expression of the $\\partial\\|\\bfy^*_+\\|_0$ in  \\eqref{subfi} indicates \n\\begin{eqnarray}\\label{fact-11}\n\\bfy_{\\Gamma_*}^*=0,~~\\bfz_{\\Gamma_*}^* \\geq 0, ~~~~\\bfy_{\\overline \\Gamma_*}^*\\neq 0,~ ~\\bfz_{\\overline \\Gamma_*}^* = 0. \n\\end{eqnarray} \n Define a radius $\\delta:=\\min\\{\\delta_1,\\delta_2\\}$, where\n\\begin{eqnarray}\\label{delta-2}\n~~~~~~\\delta_1 := \n\\begin{cases}\n+\\infty,&A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}=0\\\\\n\\frac{\\lambda}{ \\|A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}\\|},& \\text{otherwise}, \n\\end{cases}~~~~\n\\delta_2 := \n\\begin{cases}\n+\\infty,&\\bfy^*\\leq {\\bf0},\\\\\n\\min_i\\{y_i^*:y_i^*>0\\},&\\text{otherwise}, \n\\end{cases}\n\\end{eqnarray}\nand consider a local region of ${\\bfw}^* := (\\bfx^*;\\bfy^*)$ by\n\\begin{eqnarray}\\label{local-area-1}\n~~N( {\\bfw}^*,\\delta)=\\left\\{ \\left(\n\\bfx; \n  {\\bfy}\\right)\\in\\R^{n+m}   \n:~A\\bfx+\\bfb - \\bfy=0, ~\n \\|{\\bfw}- {\\bfw}^*\\|<\\delta\n\\right\\}.\n\\end{eqnarray}\nIndeed, $N( {\\bfw}^*,\\delta)$ is a neighbourhood of ${\\bfw}^*$ since $A\\bfx^*+\\bfb - \\bfy^*=0$ from \\eqref{lemma-1st-order-nec-10}. Next we show that, for any $\\bfw\\in N( {\\bfw}^*,\\delta)$,\n\\begin{eqnarray}\\label{y-y*-0}\n\\|\\bfy^*_+\\|_0\\leq \\|\\bfy_+\\|_0.\n\\end{eqnarray}\nObviously, it is true if $\\bfy^*\\leq0$ as $\\|\\bfy^*_+\\|_0=0$. For $\\bfy^*\\nleq0$,  to guarantee \\eqref{y-y*-0}, it suffices to show that for any $i$, $y_i^*>0~\\Longrightarrow~y_i>0.$ \nSuppose there is a $j\\in\\N_m$ such that $y_j^*>0$ but $y_j\\leq0$.  This incurs the following contradiction \n\\begin{eqnarray*}\\delta_2 \\geq \\delta\n& {>}&\\|\\bfw-\\bfw^*\\| \\hspace{3.5cm} ({\\rm by}~ \\eqref{local-area-1})\\\\\n&\\geq& |y_j-y_j^*|=y_j^*-y_j  \\geq y_j^*\\geq\\delta_2.\\hspace{.5cm} ({\\rm by}~ \\eqref{delta-2}) \\end{eqnarray*}\nAgain, for any $\\bfw\\in N( {\\bfw}^*,\\delta)$, we have $A\\bfx+\\bfb - \\bfy=0$, which and \\eqref{lemma-1st-order-nec-10} generate\n \\begin{eqnarray}\\label{y-y*-h-h*}\n \\bfy-\\bfy^*=A (\\bfx- \\bfx^*).\n  \\end{eqnarray}\nNext, the convexity of $f$ gives rise to  \n \\begin{eqnarray}\\label{fx-fx*}\n\\hspace{15mm} f(\\bfx)-f(\\bfx^*)&\\geq & \\langle \\nabla f(\\bfx^*), \\bfx - \\bfx^*\\rangle~~= -\\langle  A^\\top \\bfz^* , \\bfx - \\bfx^*\\rangle \\hspace{1.6cm} ({\\rm by}~ \\eqref{lemma-1st-order-nec-10})\\nonumber\\\\\n% &=& -\\langle  A_{\\Gamma_*}  (\\bfx - \\bfx^*),\\bfz^* _{\\Gamma_*}\\rangle-\\langle   A_{\\overline  \\Gamma_*} (\\bfx - \\bfx^*),\\bfz^* _{\\overline  \\Gamma_*}\\rangle\\nonumber\\\\\n &=& -\\langle  A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*},\\bfx - \\bfx^*\\rangle= -\\langle  A_{\\Gamma_*}(\\bfx - \\bfx^*),\\bfz^* _{\\Gamma_*}\\rangle=:\\phi\\hspace{0.3cm} ({\\rm by}~ \\eqref{fact-11})\n  \\end{eqnarray} \n Now we make the conclusion by two cases.  \n If $\\|\\bfy^*_+\\|_0= \\|\\bfy_+\\|_0$, then   $\\bfy_{\\Gamma_*}\\leq 0$ due to $\\bfy^*_{\\Gamma_*}=0$. This and \\eqref{y-y*-h-h*} yield that\n  \\begin{eqnarray}\\label{y-y*-h-h*-0}\n0\\geq  \\bfy_{\\Gamma_*}= \\bfy_{\\Gamma_*}-\\bfy^*_{\\Gamma_*}  = A_{\\Gamma_*} (\\bfx-\\bfx^*),\n  \\end{eqnarray}\nwhich together with $\\bfz^* _{\\Gamma_*}\\geq0$ from \\eqref{fact-11} indicates $\\phi\\geq0$, namely $f(\\bfx)\\geq f(\\bfx^*)$. So $$f(\\bfx) +\\lambda \\|\\bfy_+\\|_0 \\geq f(\\bfx^*) + \\lambda\\|\\bfy_+^*\\|_0.$$\nIf $\\|\\bfy^*_+\\|_0 \\neq \\|\\bfy_+\\|_0$, we must have $ \\|\\bfy_+\\|_0 \\geq 1+\\|\\bfy^*_+\\|_0$ by \\eqref{y-y*-0}. If  $A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}=0$, then $\\phi=0>-\\lambda$. Otherwise, it follows\n  \\begin{eqnarray*}\n\\phi&\\geq&- \\|A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}\\|\\|\\bfx - \\bfx^*\\| \\geq - \\|A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}\\|\\|\\bfw - \\bfw^*\\|\\\\\n & \\geq&- \\|A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}\\|\\delta \n  \\geq - \\|A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}\\|\\delta_1=-\\lambda.\\hspace{.5cm} ({\\rm by}~ \\eqref{delta-2})\n  \\end{eqnarray*}\nBoth lead to $\\phi\\geq-\\lambda$, which results in\n  \\begin{eqnarray*}\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n  \\begin{array}{rclll}\n  f(\\bfx) +\\lambda \\|\\bfy_+\\|_0 &\\geq& f(\\bfx^*) + \\phi +\\lambda \\|\\bfy_+\\|_0 &\\ ({\\rm by}~ \\eqref{fx-fx*} )\\\\\n&\\geq& f(\\bfx^*) -\\lambda + \\lambda\\|\\bfy_+\\|_0 &\\ ({\\rm by}~ \\phi\\geq-\\lambda)\\\\\n&\\geq& f(\\bfx^*) + \\lambda\\|\\bfy_+^*\\|_0.&\\ ({\\rm by}~ \\|\\bfy_+\\|_0 \\geq 1+\\|\\bfy^*_+\\|_0 )\n \\end{array} \\end{eqnarray*}\nOverall, the two cases show that $(\\bfx^*;\\bfy^*)$ is  a local minimizer to \\eqref{ell-0-1-equ-0}. Namely, $\\bfx^*$ is  a local minimizer to \\eqref{ell-0-1}. \n\\end{proof}","sha256_utf8":"19ceb2643ac9e1855fde8dd5cac1d19669d72c4015e5313f3a858d4cb763b120"},{"start":32270,"end":32846,"start_line":561,"end_line":568,"raw_tex":"\\begin{theorem} \\label{Stationary-Thm}\n   The following relationships hold for the problem (\\ref{ell-0-1}).\n\\begin{itemize}\n\\item[i)]  A local minimizer $\\bfx^*$ is a  P-stationary point for any $0<\\tau<\\tau_*:=\\min\\{\\tau_1,\\tau_2\\}$ if  $A_{\\Gamma_*}$ is full row rank. \n\\item[ii)] A P-stationary point with $ \\tau>0$  is a local minimizer if  the function $f$ is locally convex around $\\bfx^*$. \n\\item[iii)] A P-stationary point with $ \\tau\\geq\\|A\\|^2/c_f$  is   a global minimizer if  the function $f$ is strongly convex with a constant $c_f>0$.\n\\end{itemize}\n\\end{theorem}","sha256_utf8":"2ba033af37388852f072c2d9655a0b0d4a2c7dc12743374a2d212062d649efb0"},{"start":32848,"end":36973,"start_line":570,"end_line":633,"raw_tex":"\\begin{proof} i) As  $\\bfx^*$ is a local minimizer of \\eqref{ell-0-1},  condition  \\eqref{lemma-1st-order-nec} is valid by \\cref{first-order-nec} if $A_{\\Gamma_*}$ is  full row rank. In other words, there is a $\\bfz^*$ such that\n\\begin{eqnarray}\n\\label{lemma-1st-order-nec-1}\n\\nabla f(\\bfx^*) +A^\\top\\bfz^*=0,~~\nA \\bfx^*+\\bfb-\\bfy^*=0,~~\n\\bfz^*\\in\\partial\\|\\bfy^*_+\\|_0. \n\\end{eqnarray} \nTherefore, to show \\eqref{p-stationary}, we only need to verify that,  for any $0<\\tau<\\tau^*$, $$\\bfz^*\\in \\partial\\|\\bfy^*_+\\|_0~ \\Rightarrow~\\bfy^*\\in\\P :=\\pro (\\bfy^*+\\tau \\bfz^*). $$ \nRecall the definition of $\\Gamma_*$ in \\eqref{T-*} and the second condition in \\eqref{lemma-1st-order-nec-1}, we have $\\bfy_{\\Gamma_*}=(A \\bfx^*+\\bfb)_{\\Gamma_*}=0$. Same reasoning also allows for obtaining \\eqref{fact-11} due to $\\bfz^*\\in \\partial\\|\\bfy^*_+\\|_0$. \nAs $A_{\\Gamma_*}$ is full row rank, the first condition in \\eqref{lemma-1st-order-nec-1} and  $A^\\top\\bfz^*  =A_{\\Gamma_*}^\\top\\bfz^*_{\\Gamma_*}$ derive  that\n$$\\bfz^*_{\\Gamma_*}=\\bfp^*.$$\nNow, $0<\\tau<\\tau_*= \\min\\{\\tau_1,\\tau_2\\}$ in \\eqref{tau-*} results in \n\\begin{eqnarray}\\label{non-negative}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{lllllllllll}\n y_i^*  & \\geq & \\underset{i:y^*_i>0}{\\min} y^*_i &=&  \\sqrt{2\\tau_1\\lambda}&\\geq&\\sqrt{2\\tau_*\\lambda} &>&\\sqrt{2\\tau \\lambda}  &~~\\text{if}~~& y_i^*>0,\\\\ \n   z_i^* & \\leq &  {\\underset{i}{\\max}|p^*_i|} &= &\\sqrt{2\\lambda/\\tau_2}&\\leq&\\sqrt{2\\lambda/\\tau_*}&< &\\sqrt{2\\lambda/\\tau}& ~~ \\text{if}~~ &z_i^*>0.\n\\end{array}\n\\end{eqnarray}\nThese and \\eqref{fact-11}  yield the following condition,\n\\begin{eqnarray*} \ny_i^*\\begin{cases}\n=0, &   z_i^*=0 ~{\\rm or}~ 0< z_i^* < \\sqrt{2\\lambda/\\tau},\\\\\n<0 ~{\\rm or}~>\\sqrt{2\\tau\\lambda },& z_i^*=0. \n\\end{cases}\n\\end{eqnarray*}\n  {It is easy to see that the above condition satisfies that \n\\begin{eqnarray}\\label{ugTT-1}\ny_i^*  \\in \\P_i=\n\\begin{cases}\n0, & y_i^*+\\tau z_i^*\\in(0, \\sqrt{2\\tau\\lambda}),\\\\\n0~{\\rm or}~ y_i^*+\\tau z_i^*, &y_i^*+\\tau z_i^*\\in\\{0, \\sqrt{2\\tau\\lambda}\\},\\\\\ny_i^*+\\tau z_i^*,& y_i^*+\\tau z_i^*\\in(-\\infty,0)\\cup ( \\sqrt{2\\tau\\lambda},\\infty).\n\\end{cases}\n\\end{eqnarray}}\n\n\n\nii) Note that the second condition in \\eqref{p-stationary} means $\\bfy^*\\in\\P$, which by   \\eqref{ugTT-1} implies $z_i^*=0$ if $y_i^*\\neq 0$ and  $z_i^*\\geq0$ if $y_i^*= 0$, resulting in $\\bfz^*\\in\\partial\\|\\bfy^*_+\\|_0$ by \\eqref{subfi}. Consequently, we obtain \\eqref{lemma-1st-order-nec-1} and \\eqref{lemma-1st-order-nec}.\n% In other words, a  P-stationary point satisfies \\eqref{lemma-1st-order-nec} and thus is a local minimizer  if  $f$ is locally convex around $\\bfx^*$ by\nThe claim follows from  \\cref{first-order-nec} ii).\n\n\niii) Let $(\\bfx^*;\\bfz^*)$ be a  P-stationary point with $ \\tau\\geq\\|A\\|^2/c_f$. Then the second condition in \\eqref{p-stationary} indicates that\n\\beq\n&&({1}/{2\\tau})\\|A\\bfx^*+\\bfb-(A\\bfx^*+\\bfb+\\tau \\bfz^*)\\|^2+\\lambda\\|(A\\bfx^*+\\bfb)_+\\|_0 \\nonumber \\\\\n&\\leq& ({1}/{2\\tau})\\|A\\bfx+\\bfb-(A\\bfx^*+\\bfb+\\tau \\bfz^*)\\|^2+\\lambda\\|(A\\bfx+\\bfb)_+\\|_0\\nonumber\n\\eeq\nfor any $\\bfx\\in\\R^n$, which after simplifying leads to \n\\beq\\label{Ax*b-Axb}\n&& \\lambda\\|(A\\bfx^*+\\bfb)_+\\|_0 - \\|A(\\bfx-\\bfx^*)\\|^2 /(2\\tau) \\nonumber\\\\\n&\\leq& \\lambda\\|(A\\bfx+\\bfb)_+\\|_0 - \\langle  \\bfz^*, A(\\bfx-\\bfx^*) \\rangle\\nonumber\\\\\n&=& \\lambda\\|(A\\bfx+\\bfb)_+\\|_0 - \\langle  A^\\top \\bfz^*, \\bfx-\\bfx^*    \\rangle\\nonumber\\\\\n&=& \\lambda\\|(A\\bfx+\\bfb)_+\\|_0 + \\langle  \\nabla f(\\bfx^*), \\bfx-\\bfx^*  \\rangle. \\hspace{.5cm} ({\\rm by}~  \\eqref{p-stationary}) \n\\eeq\nThe strong convexity of $f$ implies\n\\begin{eqnarray*} \n&&f(\\bfx)+\\lambda\\|(A\\bfx+\\bfb)_+\\|_0-f(\\bfx^*)-\\lambda\\|(A\\bfx^*+\\bfb)_+\\|_0\\\\\n&\\geq&({c_f}/{2})\\|\\bfx-\\bfx^*\\|^2+\\langle  \\nabla f(\\bfx^*), \\bfx-\\bfx^*   \\rangle+\\lambda\\|(A\\bfx+\\bfb)_+\\|_0-\\lambda\\|(A\\bfx^*+\\bfb)_+\\|_0\\\\\n&\\geq&({c_f}/{2})\\|\\bfx-\\bfx^*\\|^2-1/({2\\tau})\\|A(\\bfx-\\bfx^*)\\|^2    \\hspace{.25cm} ({\\rm by}~  \\eqref{Ax*b-Axb})\\\\\n&\\geq&({c_f/2-\\|A\\|^2/(2\\tau}))\\|\\bfx-\\bfx^*\\|^2   \\geq 0. \\hspace{1cm} ({\\rm by}~  \\tau\\geq\\|A\\|^2/c_f) \n\\end{eqnarray*}\nThis shows the global optimality of $\\bfx^*$ to the problem \\eqref{ell-0-1}.\n\\end{proof}","sha256_utf8":"dbbc7c1173ec0b579331a44018ed4a9beac0601af56213e8648a0be8455433af"},{"start":39816,"end":40461,"start_line":696,"end_line":709,"raw_tex":"\\begin{theorem}\\label{cor-p-staionary}\nA point $\\bfw^*=(\\bfx^*;\\bfz^*)$ is a  P-stationary point with $\\tau>0$ of the problem (\\ref{ell-0-1}) if and only if   $F(\\bfw^*; \\Gamma_*)=0$ and $ \\Gamma_*=(\\S_* \\cup \\E^o_*)$, where $\\Gamma_*$ is defined by (\\ref{T-*}).\n%the following equations hold\n%\\begin{eqnarray}\n%\\label{sta-eq-1}\n%F(\\bfw^*; \\Gamma_*)=\\left[\\begin{array}{rrl}\n%\\nabla f(\\bfx^*)  + A_{\\Gamma_*}^\\top \\bfz_{\\Gamma_*}^*  \\\\\n%A_{\\Gamma_*} \\bfx^* + \\bfb_{\\Gamma_*} \\\\\n%\\bfz^*_{\\overline \\Gamma_*} \n%\\end{array}\n%\\right]=0. \n%\\end{eqnarray} \nThe Jacobian  $\\nabla F(\\bfw^*;\\Gamma_*)$ is   non-singular if \\cref{ass} holds.\n\\end{theorem}","sha256_utf8":"55b573a06df65cdb98b1c5eeb6dd1a81b7ce4492152edaef3b1049aad2ab4f0d"},{"start":40463,"end":43479,"start_line":711,"end_line":773,"raw_tex":"\\begin{proof} \nThe second claim is obvious due to Assumption \\ref{ass}.\n We only need to prove the first claim. \nWe start with the sufficiency.\n The definitions  in \\eqref{p-stationary-T} show the following relationships,\n\\beq\\label{TSE}\n\\Gamma_*= \\S_* \\cup \\E^o_*,~~~~\\overline{\\Gamma}_*=\\O_*\\cup(\\E_* \\setminus \\E^o_*).\n\\eeq\nWe recall $\\bfy^*=A\\bfx^*+\\bfb$ and $\\P:=\\text{Prox}_{\\tau\\lambda\\|({\\cdot})_{+}\\|_{0}}( \\bfy^*+\\tau \\bfz^*)$.\nIt follows that \n \\allowdisplaybreaks \\begin{eqnarray*} \n \\P_i \n=\n\\begin{cases}\n0, & i\\in \\S,\\\\\n0~{\\rm or}~y^*_i+\\tau z^*_i, &i\\in\\E_*,\\\\\n y_i+\\tau z^*_i,& i\\in \\O_*,\n\\end{cases} \n=\n\\begin{cases}\n0, & i\\in \\S_*\\subseteq \\Gamma_*,\\\\\n0~{\\rm or}~\\tau z^*_i, &i\\in \\E^o_* \\subseteq \\Gamma_*,\\\\\n0~{\\rm or}~ y^*_i, &i\\in(\\E_*\\setminus \\E^o_*) \\subseteq \\overline \\Gamma_*,\\\\\ny^*_i,& i\\in \\O_*\\subseteq \\overline \\Gamma_*,\n\\end{cases} \n\\end{eqnarray*}\nwhere the first equation is by \\eqref{proximal-operator} and second one is by \\eqref{sta-eq-1} and \\eqref{TSE}, which indicates $y^*_i \\in \\P_i$ due to $ \\bfy^*_{\\Gamma_*}=0$ from \\eqref{sta-eq-1}. Moreover,   \nthe first and third equations in \\eqref{sta-eq-1} suffice to $\\nabla f(\\bfx^*)  + A^\\top \\bfz^*  =  0$, showing \\eqref{p-stationary}. Namely,  $ \\bfw^*$ is a  P-stationary point. \n\n{Necessity.}  Let $ \\bfw^*$ be a  P-stationary point satisfying \\eqref{p-stationary} and $T_*:= \\S_* \\cup \\E^o_*$.  Then $\\overline{T}_*=\\O_*\\cup(\\E_* \\setminus \\E^o_*)$. It follows from $y^*_i\\in \\P_i$ and \\eqref{proximal-operator} that\n\\begin{eqnarray*} \ny^*_i\\in \n%\\Big[\\text{Prox}_{\\gamma C\\|({\\cdot})_{+}\\|_{0}}(  \\bfy^*+\\tau \\bfz^*)\\Big]_i \\\\\n\\begin{cases}\n0, & i\\in \\S_*,\\\\\n0~{\\rm or}~ y^*_i+\\tau z^*_i, &i\\in\\E_*,\\\\\ny^*_i+\\tau z^*_i,& i\\in \\O_*,\n\\end{cases} =\n\\begin{cases}\n0, & i\\in \\S_*,\\\\\n0~{\\rm or}~\\tau z^*_i, &i\\in\\E^o_*,\\\\\n0~{\\rm or}~ y^*_i+\\tau z^*_i, &i\\in\\E_*\\setminus \\E^o_*,\\\\\ny^*_i+\\tau z^*_i,& i\\in \\O_*,\n\\end{cases} \n\\end{eqnarray*}\nwhere the equality is by the definition  of $\\E^o_*$, which together with $ y^*_i= 0 , i\\in \\E^o_*$ and $y^*_i\\neq 0, i\\in \\E_*\\setminus \\E^o_*$ suffices to\n\\begin{eqnarray*} \n\\begin{cases}\ny^*_i=0, & i\\in \\S_*\\cup\\E^o_*=T_*,\\\\\ny^*_i = y^*_i+\\tau z^*_i, &i\\in\\E_*\\setminus \\E^o_*,\\\\\ny^*_i= y^*_i+\\tau z^*_i,& i\\in \\O_*,\n\\end{cases} ~~\\Longleftrightarrow~~\n\\begin{cases}\ny^*_i &=0, ~ i\\in T_*,\\\\\n z^*_i&=0, ~i\\in\\overline{T}_*.\n\\end{cases} \n\\end{eqnarray*}\nThis gives rise to the last two conditions $\\bfy^*_{T_*}=0,~\n\\bfz^*_{\\overline T_*}=0$  in \\eqref{sta-eq-1}. \nFurthermore, the first condition in \\eqref{p-stationary} and $\\bfz^*_{\\overline{T}_*}=0$ derive the first condition of \\eqref{sta-eq-1}.\nOverall, we have $F(\\bfw^*; T_*)=0$. \nNow we show $T_*=\\Gamma_*$. By  (\\ref{T-*}) that $\\Gamma_*=\\{i\\in\\N_m: y^*_i=0\\}$, it follows $T_*\\subseteq \\Gamma_*$. Suppose, there is a $j\\in \\Gamma_*$ but $j\\notin T_*$, then\nwe have $y_j^*=z_j^*=0$ and thus $j\\in \\E^o_* \\subseteq T_*$  by \\eqref{p-stationary-T}, a contradiction. Therefore, $T_*=\\Gamma_*$, finishing the proof. \\end{proof}","sha256_utf8":"e984ad07fc9a2f07b745d1ea27644ea04450f460fc763e8aba170ec6cc47364a"},{"start":49149,"end":49606,"start_line":862,"end_line":867,"raw_tex":"\\begin{lemma}\\label{lemma-neighbour-1}\n\t Let $\\bfw^*$ be a \\ts\\  with $0<\\tau<\\tau_*:=\\min\\{\\tau_1,\\tau_2\\}$ of the problem (\\ref{ell-0-1}),  $\\tau_1,\\tau_2$ and $\\Gamma_*$  be given by (\\ref{tau-*}) and (\\ref{T-*}).  Then there is a $\\delta_1^*>0$ such that, for any $\\bfw\\in N(\\bfw^*,\\delta_1^*)$ with its associated indices $\\S$ and $\\E^o$, it holds \n \\beq\\label{gw*-0} \n F(\\bfw^*;T)=0 ~~~~{\\rm and}~~~T:=(\\S\\cup\\E^o)\\subseteq \\Gamma_*.\n \\eeq\n\\end{lemma}","sha256_utf8":"5189c2cb8bc8753087d8cc4e5a935293dfc020b21d4c5be346e473699bbf365b"},{"start":49608,"end":55286,"start_line":869,"end_line":970,"raw_tex":"\\begin{proof} i) \\cref{cor-p-staionary} states that the \\ts\\ $\\bfw^*$ of (\\ref{ell-0-1}) satisfies\n\\begin{eqnarray}\n\\label{sta-eq-1-*}\n\\nabla f(\\bfx^*)  + A_{\\Gamma_*} \\bfz^*_{\\Gamma_*}=0,~~\n \\bfy^*_{\\Gamma_*}=0, ~~\n\\bfz^*_{\\overline \\Gamma_*}  =0 \n\\end{eqnarray} \n for $0<\\tau<\\tau_*$, where $\\Gamma_*=\\S_*\\cup\\E^o_*$. Note that $\\E_*\\setminus\\E^o_*\\subseteq \\overline{\\Gamma}_*$ which by \\eqref{sta-eq-1-*} leads to\n\\begin{eqnarray} \\label{sta-eq-1-*-z}\n   \\bfz^*_{\\E_*\\setminus\\E^o_*}=0.\n\\end{eqnarray} \nUsing the same reasoning for proving \\eqref{non-negative},  \nwe can prove for $ 0<\\tau<\\tau_*= \\min\\{\\tau_1,\\tau_2\\}$ in \\eqref{tau-*} that\n\\begin{eqnarray} \\label{yz-tau-lambda} \n y_i^*  >\\sqrt{2\\tau\\lambda}  ~~{\\rm if}~~y_i^*> 0,~~~~~~ \\tau z^*_i <   \\sqrt{2\\tau \\lambda   } ~~{\\rm if}~~z^*_i> 0.\n\\end{eqnarray}\nTherefore,  we have the following facts {\n\\allowdisplaybreaks\n\\begin{eqnarray*}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n\\E^o_*&=& \\{i\\in\\N_m: y_i^*=0, \\tau z_i^* \\in \\{0, \\sqrt{2\\tau\\lambda}\\} \\}&~~(\\text{by} ~\\eqref{p-stationary-T})\\\\\n&=& \\{i\\in\\N_m: y_i^*=0, z_i^*=0 \\}, &~~(\\text{by}~\\eqref{yz-tau-lambda})\\\\\n\\E_*\\setminus\\E^o_*&=& \\{i\\in\\N_m: y_i^*\\neq0, y_i^*+\\tau z_i^* = \\sqrt{2\\tau\\lambda } \\}&~~(\\text{by} ~\\eqref{p-stationary-T})\\\\\n &=& \\{i\\in\\N_m: y_i^* = \\sqrt{2\\tau\\lambda} \\}&~~(\\text{by}~\\eqref{sta-eq-1-*-z})\\\\\n&=&\\emptyset, &~~(\\text{by}~\\eqref{yz-tau-lambda})\n \\end{array}\\end{eqnarray*}}\nThese facts lead to\n\\begin{eqnarray}\\label{E*0}\n\\E_*=\\E^o_*\\cup(\\E_*\\setminus\\E^o_*)=\\E^o_*\n=\\Big\\{i\\in\\N_m: y_i^*= z_i^*=0\\Big\\},\n\\end{eqnarray}\nwhich yields the following relations\n\\begin{eqnarray}\\label{TSETO}\n\\Gamma_*=\\S_*\\cup \\E_*^o=\\S_*\\cup \\E_*,~~~~\\overline \\Gamma_*= \\O_*. \n\\end{eqnarray}   \n {For a sufficiently small $\\delta_1^*$, any $\\bfw\\in N(\\bfw^*,\\delta_1^*)$ satisfies,\n\\allowdisplaybreaks\\begin{eqnarray}\\label{hz-h*z*}\n&&  {| y_i +\\tau z_i - y_i^* -\\tau z^*_i |\\leq c\\delta_1^*},~~\\forall~i\\in\\N_m,   \n\\end{eqnarray} \n {where $c>0$ is a constant relied on $A$ and $\\tau$.} The definitions of $\\S$ and $\\S_*$  in \\eqref{p-stationary-T} mean that, for any $i\\in\\S$ or $i\\in\\S_*$,\n\\begin{eqnarray} \\label{facts-S}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n y_i+\\tau z_i\\in (0, \\sqrt{2\\tau\\lambda})~~ &\\Longleftrightarrow&~~  |y_i+\\tau z_i-\\sqrt{\\tau\\lambda/2} |< \\sqrt{\\tau\\lambda/2},\\\\\ny_i^*+\\tau z_i^*\\in (0, \\sqrt{2\\tau\\lambda})~~ &\\Longleftrightarrow&~~  |y_i^*+\\tau z_i^*-\\sqrt{\\tau\\lambda/2} |< \\sqrt{\\tau\\lambda/2}.\n\\end{array}\n\\end{eqnarray} \nUsing this fact, if $\\S_* \\nsubseteq \\S$, then there is an $i\\in \\S_*$ but $i\\notin\\S$ such that\n\\allowdisplaybreaks\n\\begin{eqnarray*}  \n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n|y_i +\\tau z_i - y_i^* -\\tau z^*_i|\n&=&|y_i+\\tau z_i-\\sqrt{\\tau\\lambda/2}  - y_i^*-\\tau z^*_i+\\sqrt{\\tau\\lambda/2}|\\nonumber\\\\\n&\\geq&   \\sqrt{\\tau\\lambda/2} -|y_i^*+\\tau z^*_i-\\sqrt{\\tau\\lambda/2}| ~~~~ (\\text{by}~i\\notin\\S~\\text{and}~\\eqref{facts-S})\\\\\n  &\\geq&   \\sqrt{\\tau\\lambda/2} -\\max_{i\\in\\S_*} |y_i^*+\\tau z^*_i-\\sqrt{\\tau\\lambda/2}| =:\\delta_s\\\\\n&>&0, ~~~~(\\text{by}~\\eqref{facts-S})\n\\end{array}\n\\end{eqnarray*}  {Since $c\\delta_1^*$} can be smaller than $\\delta_s$, the above fact contradicts with \\eqref{hz-h*z*}.  }Hence, it holds $\\S_* \\subseteq \\S$. Similar reasoning also derives $\\O_* \\subseteq \\O$. These allow us to obtain $$\\E = \\N_m \\setminus (\\S\\cup\\O)\\subseteq \\N_m \\setminus (\\S_*\\cup\\O_*)= \\E_*.$$\nOverall, for any $\\bfw\\in N(\\bfw^*,\\delta_1^*)$, it holds\n\\begin{eqnarray}\\label{SSOOEE}\n\\S_* \\subseteq \\S,~~ \\O_* \\subseteq \\O,~~ \\E\\subseteq \\E_*.\n\\end{eqnarray} \nThe above relations enable us to claim that \n\\begin{eqnarray}\\label{SS*E*}\n(\\S\\setminus \\S_*) \\subseteq \\E_*,~~~~(\\O\\setminus \\O_*) \\subseteq \\E_*.\n\\end{eqnarray}\nNow, we can show\n\\allowdisplaybreaks\n\\begin{eqnarray*} \n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n\\bfy^*_{ \\S_*}&=&0,&~~(\\text{by}~\\S_*\\subseteq \\Gamma_*~\\text{and}~\\eqref{sta-eq-1-*})\\\\\n\\bfy^*_{\\S\\setminus \\S_*} &=&0, &~~(\\text{by}~ \\S\\setminus \\S_*\\subseteq\\E_*~\\text{from}~\\eqref{SS*E*}~\\text{and}~\\eqref{E*0})\\\\\n\\bfy^*_{\\E^o} &=&0,&~~(\\text{by}~ \\E^o\\subseteq\\E\\subseteq\\E_*~\\text{from}~\\eqref{SSOOEE}~\\text{and}~\\eqref{E*0})\\\\\n\\bfz^*_{ \\O_*}  &=&0, &~~(\\text{by}~\\O_*\\subseteq \\overline \\Gamma_*~\\text{and}~\\eqref{sta-eq-1-*})\\\\\n\\bfz^*_{\\O\\setminus \\O_*} &=&0, &~~(\\text{by}~ \\O\\setminus \\O_*\\subseteq\\E_*~\\text{from}~\\eqref{SS*E*}~\\text{and}~\\eqref{E*0})\\\\\n\\bfz^*_{\\E\\setminus\\E^o} &=&0.&~~(\\text{by}~ \\E\\setminus\\E^o\\subseteq\\E\\subseteq\\E_*~\\text{from}~\\eqref{SSOOEE}~\\text{and}~\\eqref{E*0})\n\\end{array}\n\\end{eqnarray*}\nThese conditions combining with\n\\begin{eqnarray*}T&=&\\S\\cup\\E^o=\\S_*\\cup(\\S\\setminus \\S_*)\\cup\\E^o,\\\\\n \\overline T&=&\\O\\cup(\\E\\setminus\\E^o)=\\O_*\\cup(\\O\\setminus \\O_*)\\cup(\\E\\setminus\\E^o),\n \\end{eqnarray*}\nimply $\\bfy^*_{ T} =0$ and $\\bfz^*_{\\overline T}=0$. \nAs a consequence of this and $\\bfz^*_{\\overline \\Gamma_*}=0$ from \\eqref{sta-eq-1-*}, \n\\begin{eqnarray*}\n  0= f(\\bfx^*)+ A_{\\Gamma_*} ^\\top\\bfz^*_{\\Gamma_*}=f(\\bfx^*)+ A^\\top\\bfz^*= f(\\bfx^*)+ A_{T} ^\\top\\bfz^*_{T}.\n \\end{eqnarray*}\nOverall, we verify $F(\\bfw^*;T)=0$,  as desired. \nFinally, we observe that\n \\beq \n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n \\begin{array}{rcll}\nT=\\S\\cup\\E^o&=&\\S_*\\cup(\\S\\setminus \\S_*)\\cup\\E^o\\nonumber\\\\\n&\\subseteq&   \\S_*\\cup\\E_* \\cup\\E^o&(\\text{by}~ \\S\\setminus \\S_*\\subseteq\\E_*~\\text{from}~\\eqref{SS*E*})\\nonumber\\\\\n&\\subseteq&   \\S_*\\cup\\E_* \\cup \\E_*  &(\\text{by}~\\E^o\\subseteq\\E\\subseteq\\E_*~\\text{from}~\\eqref{SSOOEE})\\nonumber\\\\\n&=&   \\Gamma_*.  &(\\text{by}~\\eqref{TSETO})\\nonumber\n\\end{array}\n \\eeq\nThe whole proof is completed.   \\end{proof}","sha256_utf8":"b6239d26a6a4c556eca351175ce8165147aec5ef5bae9ef88132b42ca376488c"},{"start":55445,"end":56270,"start_line":976,"end_line":990,"raw_tex":"\\begin{lemma}\\label{lemma-neighbour-2} Let $\\bfw^*$ be a \\ts\\  with $0<\\tau<\\tau_*$ of the problem (\\ref{ell-0-1}) and $\\tau_*$ be given by (\\ref{tau-*}).   Assume  \\cref{ass} and \\cref{ass2}. It holds \n \\beq \\label{bd-jacobian-F-w-T}\n  && C_*  \\geq \\|\\nabla F_{\\mu}(\\bfw;T)\\|\\geq\\sigma_{\\min}( \\nabla F_{\\mu}(\\bfw;T)) \\geq   c_*>0, \n\\eeq\nfor any $\\bfw\\in N(\\bfw^*,\\delta_2^*)$ and any $0\\leq\\mu\\leq c_*/2$, where $T=\\S\\cup\\E^o$ and\n \\beq\\label{radius-2}\nC_*&:=&2\\max\\{1, \\|H(\\Gamma_*)\\|\\} ,~~~~ c_* := 0.5{\\min\\{1,\\min_{\\Gamma\\subseteq\\Gamma_*}\\sigma_{\\min}(H(\\Gamma))\\}}, \\\\\n\\label{H*}\\delta_2^* &:=& \\min\\left\\{\\delta_1^*,   \\frac{c_*}{ 2L_*}     \\right\\},~~~~H(\\Gamma):=\\left[  \n     \\begin{array}{ccc}\n    \\nabla^2 f(\\bfx^*)& A^\\top_{\\Gamma}\\\\\n     A_{\\Gamma}&0\n     \\end{array}\n     \\right]. \n\\end{eqnarray}\n\\end{lemma}","sha256_utf8":"5e972dd9811e1a3b24dede66de05313d51240a9ccc9a9f517ae60baab21244af"},{"start":56272,"end":60187,"start_line":992,"end_line":1068,"raw_tex":"\\begin{proof}  \nSince  $\\nabla^2 f(\\bfx^*)$ is  {positive definite} and $A_{\\Gamma_*}$ is full row rank by \\cref{ass}, $A_\\Gamma$ is full row rank for any $ \\Gamma \\subseteq   \\Gamma_*$ and thus $H(\\Gamma)$ is non-singular. \nWe have $\\sigma_{\\min}(H(\\Gamma)) > 0$ for any \n$ \\Gamma \\subseteq   \\Gamma_*$ \nand  $c_*>0$. \nNow we build the bounds of $\\nabla F_{\\mu}(\\bfw;T)$. \nFor any given two matrices $D'$ and $D$, we have the first fact\n\\begin{eqnarray} \\label{lower-bd-HH}\n \\|D'-D\\|\n&\\geq& \\max_{i}|\\sigma_i(D')-\\sigma_i(D )|   \\geq |\\sigma_{i_0}( D')-\\sigma_{i_0}(D)|\\nonumber\\\\\n&\\geq& \\sigma_{i_0}(D')-\\sigma_{\\min}(D) \\geq \\sigma_{\\min}(D')-\\sigma_{\\min}(D), \n\\end{eqnarray}\nwhere the first inequality is from \\cite[Reminder (2), on Page 76]{lutkepohl1996handbook} and  $i_0$ satisfies that $\\sigma_{i_0}(D)=\\sigma_{\\min}(D)$.  Recall that\n \\begin{eqnarray*}\n \\label{FH*-FG}H(\\Gamma)\\overset{\\eqref{H*}}{=}\\left[  \n     \\begin{array}{ccc}\n    \\nabla^2 f(\\bfx^*)& A^\\top_{\\Gamma }\\\\\n     A_{\\Gamma }&0\n     \\end{array}\n     \\right],~~~~ \\nabla F(\\bfw^*;T) =\\left[  \n     \\begin{array}{cc}\n    H(T)&0\\\\\n     0&I  \n     \\end{array}\n     \\right].\n    \\end{eqnarray*}\nFor any $\\bfw\\in N(\\bfw^*,\\delta_2^*)$ and $\\delta_2^*\\leq\\delta_1^*$,   \\cref{lemma-neighbour-1} contributes to $T \\subseteq   \\Gamma_*$. So $H(T)$ is a submatrix of $H(\\Gamma_*)$. Hence, %It follows from \\cite[Theorem 1]{thompson1972principal} that the maximum  singular value of a  matrix is no less than the maximum singular value of its sub-matrix. Namely, \n\\beq \n\\|H(T)\\| \\leq \\|H(\\Gamma_*)\\|,~~ \\sigma_{\\min}(H(T)) \\geq \\min_{\\Gamma \\subseteq \\Gamma_*} \\sigma_{\\min} (H(\\Gamma)),\\nonumber\n\\eeq\nwhere the latter is by $T \\subseteq   \\Gamma_*$, which gives us the second fact \n\\beq\\label{JJogG0}\n\\sigma_{\\min}(\\nabla F(\\bfw^*;T))&=&\\min\\{1,\\sigma_{\\min}(H(T))\\}\\\\\n& \\geq&\\min\\{1, \\min_{\\Gamma \\subseteq \\Gamma_*} \\sigma_{\\min} (H(\\Gamma))\\}=2c_*, \\nonumber\\\\\n\\label{JJogG1}\n\\|\\nabla F(\\bfw^*;T)\\| &=& \\max\\{1, \\|H(T)\\|\\} \\leq \\max\\{1, \\|H(\\Gamma_*)\\|\\} =C_*/2. \n\\eeq\nThe locally Lipschitz continuity of $\\nabla^2 f$  around $\\bfx^*$ with $L_*$  yields the third fact,\n\\begin{eqnarray}\n\\label{HH*2}\n\\|\\nabla F (\\bfw^*;T)-\\nabla F (\\bfw;T)\\|&=&\\|\\nabla^2  f (\\bfx)-\\nabla^2  f (\\bfx^*)\\|  \\leq\n   L_* \\| \\bfx -\\bfx^* \\| \\nonumber\\\\\n   &\\leq& L_* \\| \\bfw -\\bfw^* \\| \n \\leq   L_* \\delta_2^*\n\\leq c_*/2.   \\hspace{.5cm}(\\text{by}~\\eqref{H*})\n\\end{eqnarray} \nNow these three facts allow us to derive\n\\allowdisplaybreaks \n\\begin{eqnarray}\\label{lower-bd-FF}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rclr}\n&& \\sigma_{\\min}(\\nabla F_{\\mu}(\\bfw;T))\\\\\n &\\geq& \\sigma_{\\min}(\\nabla F (\\bfw ;T))- \\|\\nabla F(\\bfw;T)-\\nabla F_{\\mu}  (\\bfw;T)\\| &(\\text{by}~\\eqref{lower-bd-HH})  \\\\\n&=& \\sigma_{\\min}(\\nabla F(\\bfw ;T))- \\mu &(\\text{by}~\\eqref{sta-eq-1}) \\\\\n&\\geq& \\sigma_{\\min}(\\nabla F(\\bfw^*;T))- \\|\\nabla F (\\bfw^*;T)-\\nabla F (\\bfw;T)\\|- \\mu   &(\\text{by}~\\eqref{lower-bd-HH}) \\\\\n&\\geq&\\sigma_{\\min}(\\nabla F(\\bfw^*;T))- c_*/2 - \\mu&(\\text{by}~\\eqref{HH*2}) \\\\\n&\\geq&\\sigma_{\\min}(\\nabla F(\\bfw^*;T))-  c_* &(\\text{by}~\\mu\\leq c_*/2)   \\\\\n&\\geq&c_*. &(\\text{by}~\\eqref{JJogG0})  \n\\end{array}\n\\end{eqnarray}\nSimilarly, we also have\n\\allowdisplaybreaks \n\\begin{eqnarray}\\label{upper-bd-FF}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rclr}\n&&\\|\\nabla F_{\\mu}(\\bfw;T)\\|\\\\\n &\\leq& \\|\\nabla F (\\bfw ;T)\\|+ \\|\\nabla F(\\bfw;T)-\\nabla F_{\\mu}  (\\bfw;T)\\| \\\\\n&=& \\|\\nabla F (\\bfw ;T)\\|+ \\mu &(\\text{by}~\\eqref{sta-eq-1})\\\\\n&\\leq& \\|\\nabla F(\\bfw^*;T)\\|+ \\|\\nabla F (\\bfw^*;T)-\\nabla F (\\bfw;T)\\|+ \\mu   &(\\text{by}~\\eqref{lower-bd-HH})  \\\\\n&\\leq& \\|\\nabla F(\\bfw^*;T)\\|+ c_*/2 + \\mu&(\\text{by}~\\eqref{HH*2}) \\\\\n&\\leq& \\|\\nabla F(\\bfw^*;T)\\|+ c_* &(\\text{by}~\\mu\\leq c_*/2)   \\\\\n&\\leq& C_*/2+c_* &(\\text{by}~\\eqref{JJogG1}) \\\\\n&\\leq& C_*. &(\\text{by}~c_*\\leq C_*/2)   \n\\end{array}\n\\end{eqnarray}\nThe whole proof is completed. \n\\end{proof}","sha256_utf8":"721b0283870775bccde6a431b453d92a26f6f8a37f3c896a06aa2b4da9300398"},{"start":61858,"end":69522,"start_line":1107,"end_line":1217,"raw_tex":"\\begin{proof} a) It is easily observed   \n\\begin{eqnarray}\\label{mu-k}\n0< \\mu_{k} \\leq \\mu_{k-1}, ~k=1,2,3,\\ldots, ~~~~{\\rm and}~~~~ \\lim_{k\\rightarrow\\infty}\\mu_k=0.\n\\end{eqnarray} \nIt follows from \\cref{lemma-neighbour-1} and the facts\n$\\delta_*\\leq \\delta_2^*\\leq\\delta_1^*$ and $\\bfw^0\\in N(\\bfw^*,\\delta_*)$ that     \n\\beq\\label{F-w*-T0-0}F (\\bfw^{*};T_0) =0  \n\\eeq\nwith $T_0=\\S_0\\cup\\E_0^o$ and from \\cref{lemma-neighbour-2} that     \n\\beq\\label{grad-F-w*-T0-c} C_*  \\geq \\|\\nabla F_{\\mu_0}(\\bfw^0;T_0)\\|\\geq\\sigma_{\\min}( \\nabla F_{\\mu_0}(\\bfw^0;T_0)) \\geq  c_*\n\\eeq\nHere, we used the fact that $\\mu_0\\leq \\mu_{-1}\\leq c_*/2$ by  \\eqref{mu-k}. \nFrom \\eqref{newton-dir-0}, we have\n\\beq\\label{dkt}\n\\nabla F_{\\mu_0} (\\bfw^{0};T_0)~\\bfd^{0}=- F (\\bfw^{0};T_0).\n\\eeq\n  \\cref{lemma-neighbour-2} states that $\\nabla F_{\\mu_0} (\\bfw^{0};T_0)$ is non-singular and thus $\\bfd^{0}$ is well defined.\nLet \n\\beq\\label{w0-beta-w*}\n\\bfw^{0}_\\beta= \\bfw^{*} + \\beta(\\bfw^{0}-\\bfw^{*})=(\\bfx^{0}_\\beta;\\bfz^{0}_\\beta)\n\\eeq\nwhere $\\beta\\in[0,1].$ One can easily check that $\\bfw^{0}_\\beta\\in N(\\bfw^*,\\delta_*)$ as $$\\|\\bfw^{0}_\\beta-\\bfw^{*}\\|= \\beta\\|\\bfw^{0}-\\bfw^{*}\\|\\leq  \\delta_*.$$\nThe definition  in \\eqref{sta-eq-1} enables us to obtain\n\\allowdisplaybreaks \n\\begin{eqnarray*}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n&&\\|\\nabla F_{\\mu_0} (\\bfw^{0};T_0)- \\nabla F (\\bfw^{0}_\\beta;T_0) \\|  \\\\\n&\\leq&\\|\\nabla^2 f(\\bfx^0)- \\nabla^2 f(\\bfx^0_\\beta)\\| + \\mu_0&(\\text{by}~\\eqref{sta-eq-1})  \\\\\n& \\leq& L_* \\|\\bfx^0-\\bfx^0_\\beta\\|+ \\rho  \\|F(\\bfw^0;T_0)\\|&(\\text{by}~\\eqref{update-mu}) \\\\ \n& \\leq& L_* \\|\\bfw^0-\\bfw^0_\\beta\\|+ \\rho \\|\\nabla F_{\\mu_0} (\\bfw^{0};T_0)\\| \\|\\bfd^0\\|&(\\text{by}~\\eqref{newton-dir-0}) \\\\ \n& \\leq& L_* \\|\\bfw^0-\\bfw^0_\\beta\\|+ \\rho C_* \\|\\bfw^1-\\bfw^0\\|&(\\text{by}~\\eqref{grad-F-w*-T0-c}) \\\\ \n& \\leq& L_*(1-\\beta)\\|\\bfw^0-\\bfw^*\\|+ \\rho C_* (\\|\\bfw^1-\\bfw^*\\|+\\|\\bfw^0-\\bfw^*\\|) &(\\text{by}~\\eqref{w0-beta-w*})  \\\\ \n& =& (L_*(1-\\beta)+\\rho C_*)\\|\\bfw^0-\\bfw^*\\|+ \\rho C_* \\|\\bfw^1-\\bfw^*\\|.\n\\end{array} \\end{eqnarray*}\nDenote $\\Theta_{\\mu_0}:=\\nabla F_{\\mu_0} (\\bfw^{0};T_0)$ and $\\Theta(\\beta):= \\nabla F (\\bfw^{0}_\\beta;T_0)$.  The above condition yields\n  \\beq\\label{lipschitz}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n \\int_0^1 \\|\\Theta_{\\mu_0}  - \\Theta(\\beta)\\|d\\beta\n\\leq (L_*/2+\\rho C_*)\\|\\bfw^0-\\bfw^*\\|+ \\rho C_* \\|\\bfw^1-\\bfw^*\\|.\n\\end{array} \\eeq\nFor the fixed $T_0$, the function $F(\\cdot,T_0)$ is differentiable, which by \\eqref{F-w*-T0-0} derives\n\\beq\\label{mean-value}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\nF (\\bfw^{0};T_0)=F (\\bfw^{*};T_0)+\\int_0^1 \\Theta(\\beta)(\\bfw^{0}-\\bfw^{*})d\\beta=\\int_0^1 \\Theta(\\beta) (\\bfw^{0}-\\bfw^{*})d\\beta.\n\\end{array}\\eeq\nNow the following chain of inequalities holds.\n\\begin{eqnarray*}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\nc_*\\|\\bfw^{1}-\\bfw^{*}\\| &=& c_*\\|\\bfw^{0}+\\bfd^{0}-\\bfw^{*}\\|\\\\\n& {=} &c_*\\|\\bfw^{0}-\\bfw^{*}-\\Theta_{\\mu_0}^{-1}   F(\\bfw^{0};T_0) \\|&~~(\\text{by}~\\eqref{dkt})\\\\\n& {\\leq} & \\|\\Theta_{\\mu_0} (\\bfw^{0}-\\bfw^{*})-   F(\\bfw^{0};T_0) \\|&~~(\\text{by}~\\eqref{grad-F-w*-T0-c})\\\\\n&= & \\|\\Theta_{\\mu_0}  (\\bfw^{0}-\\bfw^{*})-  \\int_0^1 \\Theta(\\beta)(\\bfw^{0}-\\bfw^{*})d\\beta \\| &~~(\\text{by}~\\eqref{mean-value}) \\\\\n& \\leq& \\int_0^1 \\|\\Theta_{\\mu_0} -\\Theta(\\beta) \\|\\|\\bfw^{0}-\\bfw^{*}\\|d\\beta \\\\\n& =& ({\\theta_*}/2) \\|\\bfw^{0} -\\bfw^{*}\\|^2+   {\\rho C_*}  \\|\\bfw^1-\\bfw^*\\|\\|\\bfw^{0}-\\bfw^{*}\\| &~~(\\text{by}~\\eqref{lipschitz}) \\\\\n&\\leq& ({\\theta_*}/2) \\|\\bfw^{0} -\\bfw^{*}\\|^2+ {\\rho C_*\\delta_*}  \\|\\bfw^1-\\bfw^*\\|   \\\\\n&\\leq& ({\\theta_*}/2) \\|\\bfw^{0} -\\bfw^{*}\\|^2+ (c_*/2) \\|\\bfw^1-\\bfw^*\\|, \n\\end{array}\n\\end{eqnarray*}\nwhere $\\theta_*:=L_* +2\\rho C_*$ and the last inequality is from $\\delta_*\\leq  {c_*}/(2 \\theta_*)$ by \\eqref{delta-*} and\n\\beq\\label{w0-w*-delta*}\n\\|\\bfw^{0}-\\bfw^{*}\\|<\\delta_*\\leq  {c_*}/(2 \\theta_*)<  {c_*}/({2\\rho  C_*}).\n\\eeq\n The above chain of inequalities  suffices to the following fact\n\\beq\\label{quadratic-chain}\n\\|\\bfw^{1}-\\bfw^{*}\\|\\leq  (\\theta_*/c_*) \\|\\bfw^{0} -\\bfw^{*}\\|^2.\n\\eeq\nThis together with $\\|\\bfw^{0}-\\bfw^{*}\\|<\\delta_*$ and \\eqref{w0-w*-delta*} derives\n\\beq \n\\|\\bfw^{1}-\\bfw^{*}\\|  &\\leq& \n   (\\theta_*/c_*)  \\delta_* \\|\\bfw^{0}-\\bfw^{*} \\|\\leq (1/2)\\|\\bfw^{0}-\\bfw^{*} \\|< \\delta_*,  \\nonumber\n \\eeq\nwhich means $\\bfw^1\\in N(\\bfw^*,\\delta_*)$. In addition, $\\mu_1\\leq\\mu_0\\leq c_*/2$ by  \\eqref{mu-k}. Hence, replacing $T_0 $ by $T_1$, the same reasoning allows us to show that $\\bfd^{1}$ is well defined and $$\\|\\bfw^{2}-\\bfw^{*}\\|  \\leq (\\theta_*/c_*)   \\|\\bfw^{1}-\\bfw^{*} \\|^2 .$$ By the induction, we can conclude that $\\bfw^k\\in N(\\bfw^*,\\delta_*)$, $\\bfd^{k}$ is well defined and \n\\beq\\label{quadratic-rate}\n\\hspace{10mm}\\|\\bfw^{k+1}-\\bfw^{*}\\|  &\\leq& (\\theta_*/c_*) \\|\\bfw^{k}-\\bfw^{*} \\|^2, \\\\\n\\label{quadratic-rate-1}&\\leq& (\\theta_*/c_*)  \\|\\bfw^{k}-\\bfw^{*} \\|\\delta_* \\leq (1/2) \\|\\bfw^{k}-\\bfw^{*} \\|.\\hspace{.2cm}(\\text{by}~\\eqref{w0-w*-delta*})\n\\eeq\nTherefore, \\eqref{quadratic-rate} claims b). The conclusion of a) can be made by \\eqref{quadratic-rate-1}  that   $$\\bfw^{k}\\rightarrow\\bfw^{*},~~ \\bfd^k=\\bfw^{k+1}-\\bfw^{k}=\\bfw^{k+1}-\\bfw^{*}+\\bfw^{*}-\\bfw^{k}\\rightarrow0.$$ \n\nc)  The  above  proof shows $\\bfw^k\\in N(\\bfw^*,\\delta_*)$ and hence \\eqref{gw*-0} results in\n\\beq\\label{wk-in-N}  F(\\bfw^{*};T_k){=}0,\\eeq\nwhere $T_k=\\S_k\\cup\\E_k^o$.\nBy letting $\\bfw^{k}_\\beta= \\bfw^{*} + \\beta(\\bfw^{k}-\\bfw^{*}),$ where $\\beta\\in[0,1]$, we have $\\bfw^k_\\beta\\in N(\\bfw^*,\\delta_*)$. To show \\eqref{bd-jacobian-F-w-T} in \\cref{lemma-neighbour-2}, we verified the lower and upper bounds by \\eqref{lower-bd-FF} and \\eqref{upper-bd-FF}. Similarly, we can prove these bounds hold for $\\nabla F (\\bfw^{k}_\\beta;T_k)$. (In fact, since $\\bfw^k_\\beta\\in N(\\bfw^*,\\delta_*)\\subseteq N(\\bfw^*,\\delta_2^*)$ by $\\delta_*\\leq \\delta_2^*$, one just needs to set $\\mu=0$ and $\\bfw= \\bfw^{k}_\\beta$ in  \\eqref{lower-bd-FF} and \\eqref{upper-bd-FF}. Therefore,\n\\beq\\label{C-F-w-beta-c}\nC_*  \\geq \\|\\nabla F (\\bfw^{k}_\\beta;T_k)\\|\\geq\\sigma_{\\min}( \\nabla F (\\bfw^{k}_\\beta;T_k)) \\geq  c_*, \n\\eeq\nAgain, the function $F(\\cdot,T_k)$ is differentiable for the fixed $T_k$, so the Mean-value theorem states that there is a $\\beta_0\\in(0,1)$ satisfying\n\\beq\\label{FkTk}\n\\|F(\\bfw^{k},T_k)\\|&=&\\|F(\\bfw^{*};T_k)+  \\nabla F (\\bfw^{k}_{\\beta_0};T_k) (\\bfw^{k}-\\bfw^{*})\\|\\nonumber\\\\\n&=&\\| \\nabla F (\\bfw^{k}_{\\beta_0};T_k) (\\bfw^{k}-\\bfw^{*}) \\|\\hspace{1.7cm}(\\text{by}~\\eqref{wk-in-N})\\nonumber\\\\\n&\\in&  [~c_*\\|\\bfw^{k}-\\bfw^{*} \\|, C_*\\|\\bfw^{k}-\\bfw^{*} \\|~], \\hspace{0.8cm}(\\text{by}~\\eqref{C-F-w-beta-c})\n\\eeq\nThis contributes to \n\\allowdisplaybreaks\n\\begin{eqnarray*}\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rclr}\n\\|F(\\bfw^{k};T_{k})\\| \\leq  C_*\\|\\bfw^{k}-\\bfw^{*}\\|  \n & {\\leq} & (\\theta_*C_*/c_*)   \\|\\bfw^{k-1}-\\bfw^{*} \\|^2 &(\\text{by}~\\eqref{quadratic-rate}) \\\\\n & {\\leq}& (\\theta_*C_*/c_*^{3})  \\|F(\\bfw^{k-1};T_{k-1})\\|^2&(\\text{by}~\\eqref{FkTk})\\\\\n &{\\leq}& (\\theta_*C_*^3/c_*^{3})      \\|\\bfw^{k-1}-\\bfw^{*} \\|^2&(\\text{by}~\\eqref{FkTk})\\\\\n& {\\leq}& (\\theta_*C_*^3/c_*^{3})  2^{-2}  \\|\\bfw^{k-2}-\\bfw^{*} \\|^2&~~(\\text{by}~\\eqref{quadratic-rate-1})\\\\\n    &\\vdots&\\\\\n & \\leq& (\\theta_*C_*^3/c_*^{3})   2^{2-2k} \\|\\bfw^{0}-\\bfw^{*} \\|^2,&(\\text{by}~\\eqref{quadratic-rate-1})\n \\end{array}  \\end{eqnarray*}\nwhere the third inequality yields the first conclusion in c). This also enable to verify that $\\|F(\\bfw^{k};T_k)\\|$ $< {\\epsilon}$ if $k$ satisfies \\eqref{k-tol}.\n%$$k\\geq  \\lceil \\log_2 (2 \\sqrt{\\theta_* (C_* /c_*)^3}\\|\\bfw^{0}-\\bfw^{*} \\| )-\\log_2(\\sqrt{\\epsilon}) \\rceil.$$ \nThe whole proof is completed.\n\\end{proof}","sha256_utf8":"2362e9f5969718e16d8585f09010dfd8e6f032c35948cae9b191d85de89e1719"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"零一损失优化模型","source_units":[{"start":5999,"end":6175,"start_line":179,"end_line":182,"raw_tex":"This paper is concerned with the $0/1$-loss optimization:\n\\begin{equation}\\label{ell-0-1}\n\\underset{\\bfx\\in\\R^n}{\\min}~ f(\\bfx)+\\lambda \\|( A\\bfx +\\bfb )_+\\|_0, \n\\end{equation}","sha256_utf8":"ba110f3ca8c4f2298ab52251fd0bc97bf516e9d287a9fcdfeaf088758a064885"},{"start":6177,"end":6725,"start_line":183,"end_line":195,"raw_tex":"where $f:\\R^n\\rightarrow\\R$ is  twice continuously differentiable, \n$\\lambda>0 $ is a penalty parameter and $A\\in \\R^{m\\times n}, \\bfb\\in \\R^m$. \nMoreover, ${\\bfz}_+:=((z_1)_+, \\ldots, (z_m)_+)^\\top $ with $z_+:=\\max\\{z,{0}\\}$ \nand $\\|\\bfz\\|_0$ is the $\\ell_0$ norm of $\\bfz$, counting the number of its non-zero entries. \nHence, $\\|\\bfz_+\\|_0$ counts the number of positive entries of $\\bfz$,\ni.e., $\\|\\bfz_+\\|_0=\\sum_{i=1}^m \\ell_{0/1}(z_i)$, where\n\\begin{eqnarray*}\n\\ell_{0/1}(z)=\n\\begin{cases}\n1, &z>0,\\\\\n0& z\\leq 0.\n\\end{cases}\n\\end{eqnarray*}","sha256_utf8":"b82baa1c8440987bca2ce1aa2b9b4532f9a793e6aa12d01935c31ae5c30a0f29"}]},{"id":"S002","semantic_type":"notation","kind":"notation","title":"集合邻域与矩阵范数","source_units":[{"start":17269,"end":18116,"start_line":306,"end_line":309,"raw_tex":"Given a subset $T\\subseteq\\N_m:=\\{1,2,\\ldots,m\\}$, its cardinality and complementary set are $|T|$ and $\\overline T$.   The neighbourhood of $\\bfx\\in\\R^n$ with a radius $\\delta>0$  is denoted by $N( \\bfx,\\delta)=\\{\\bfv\\in\\R^n: \\|\\bfv- \\bfx \\|< \\delta\\}$. \nMoreover, $\\bfx_{T}$ (resp. $A_{T}$) represents the sub-vector (resp. sub-matrix) contains elements (resp. rows) of $\\bfx$ indexed on $T$.   Particularly, $A_i$ is the $i$th row of $A$. \nWe combine two vectors as $(\\bfx;\\bfy):=(\\bfx^\\top~\\bfy^\\top)^\\top$. The $i$th   largest singular value of $H\\in\\R^{n\\times n}$ is written $\\sigma_{i}(H)$, namely $\\sigma_{1}(H)\\geq\\sigma_{2}(H)\\geq\\cdots\\geq\\sigma_{n}(H).$\nParticularly, we write $\\|H\\|:=\\sigma_{1}(H)$ and $\\sigma_{\\min}(H):=\\sigma_{n}(H).$ Finally, let $I$ be the identity matrix and $\\bfe$ be the vector with all entries being ones. \n","sha256_utf8":"af3af743d0258c4e088252f05dce341101b2fd03df7b1266a18cd069ce917085"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"下半连续性与次微分","source_units":[{"start":18117,"end":18513,"start_line":311,"end_line":317,"raw_tex":"Next we describe the formula for computing the subdifferential of \n$\\|\\bfz_+\\|_0$ and its proximal operator.\nWe note that $\\|\\cdot\\|_0$ is lower semi-continuous (lsc)\nand $\\bfz_+$ is obviously continuous, \nthe composition $\\|\\bfz_+\\|_0$ is also lsc.\nFor a proper and lsc function $g: \\R^m \\mapsto \\R$, its\nsubdifferential $\\partial g(\\cdot)$ is well defined as in \\cite[Definition 8.3]{RW1998}. \n","sha256_utf8":"d8472fd2fd20251991103462b7fbbbeb2023bc56a0b3fbb1602c249d962e837a"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"近端算子及显式公式","source_units":[{"start":22236,"end":22725,"start_line":386,"end_line":389,"raw_tex":"The assumption in (\\ref{chain-rule-cond}) can be regarded as a constraint qualification for the chain rule in (\\ref{chain-rule-sub}) to hold. We finish this section with a formula to compute the proximal operator of  $\\|(\\cdot)_+\\|_0$.  Let $\\alpha>0$, the proximal operator of $\\alpha \\|(\\cdot)_+\\|_0$ at  {$\\bfo$ is defined by \n\\begin{eqnarray*} %\\label{proximal-operator-0}\n\\prox(\\bfo)=\\underset{\\bfy\\in\\R^m}{\\rm argmin} ~\\frac{1}{2}\\|\\bfy-\\bfo\\|^2+ \\alpha \\|\\bfy_+\\|_0.\n\\end{eqnarray*}","sha256_utf8":"4b11f676da20d1519638110ea4bb53f85a694437d8e09ad03b159c18b0fecfbc"},{"start":22726,"end":22819,"start_line":390,"end_line":390,"raw_tex":"As shown in \\cite[Lemma 2.2]{wang2019support}, the proximal operator admits a closed form as\n","sha256_utf8":"71d047baba76d0fd440bfbf734865f631521f2fdf85759ff2d9c62e8bc6a9e7d"},{"start":23089,"end":23545,"start_line":399,"end_line":413,"raw_tex":"\\begin{eqnarray} \\label{proximal-operator}\n\\Big[\\prox(\\bfo)\\Big]_i=\n%&=&\n%\\begin{cases}\n%0,&    \\nu_i \\in [0, \\sqrt{2\\alpha}),\\\\\n%0~ \\text{or}~ \\nu_i ,& \\nu_i=\\sqrt{2\\alpha},\\\\\n%\\nu_i,&\\nu_i\\in(-\\infty,0)\\cup (\\sqrt{2\\alpha},\\infty),\n%\\end{cases} \\nonumber\\\\\n%&=&\n\\begin{cases}\n0,&    \\nu_i \\in (0, \\sqrt{2\\alpha}),\\\\\n0~ \\text{or}~ \\nu_i ,& \\nu_i\\in\\{0,\\sqrt{2\\alpha}\\},\\\\\n\\nu_i,&\\nu_i\\in(-\\infty,0)\\cup (\\sqrt{2\\alpha},\\infty).\n\\end{cases} \n\\end{eqnarray}","sha256_utf8":"8cc880e1ce567fdc38b3cbc36dd6d5fcb0e638a92bb6734e6b61da5bab5e3d91"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"零残差指标集","source_units":[{"start":23971,"end":24058,"start_line":418,"end_line":420,"raw_tex":"\\begin{eqnarray}\\label{T-*}\n  \\Gamma_*&:=&\\{i\\in\\N_m: A_i\\bfx^*+b_i=0\\}.\n\\end{eqnarray}","sha256_utf8":"4aa14cb36359027d6f7e14e5c9cbb2b014d351b2f199ac9569827932b13b8d86"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"近端平稳点","source_units":[{"start":31124,"end":31522,"start_line":535,"end_line":544,"raw_tex":"\\begin{definition}\\label{def-p-sta} \n\tA point $\\bfx^*$ is called a P-stationary point of the problem (\\ref{ell-0-1})  if there exist a constant $\\tau > 0$ and\n\ta point $\\bfz^*\\in\\R^m$  such that\n\\begin{eqnarray} \\label{p-stationary}\n \\begin{cases}\n\\hspace{21mm}\\nabla f(\\bfx^*)  + A^\\top \\bfz^* &=~ 0 \\\\\n\\pro (A\\bfx^*+\\bfb+\\tau \\bfz^*) &\\ni~A\\bfx^*+\\bfb.\n\\end{cases}\n\\end{eqnarray}\n\\end{definition}","sha256_utf8":"1f076a8322359eadb58c9a56d85a4d81fd1714a192472aa8809602a2a9e4d53d"},{"start":31524,"end":31674,"start_line":546,"end_line":546,"raw_tex":"We also say a point $(\\bfx^*;\\bfz^*)$ is a P-stationary point of the problem (\\ref{ell-0-1}) if it satisfies the conditions in \\eqref{p-stationary}. \n","sha256_utf8":"b235c5c6ea8121049556e22d3bd346ba4c9f11900b88ed8a1c367ac8460602ca"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"平稳参数阈值","source_units":[{"start":31674,"end":31724,"start_line":547,"end_line":547,"raw_tex":"For a point $\\bfx^*$, we denote two constants by \n","sha256_utf8":"7d42981f08420bfe888440ea00e0ade9dc40dd8295c846048095ea6adb51e4f3"},{"start":31725,"end":32038,"start_line":548,"end_line":559,"raw_tex":"\\begin{eqnarray}\n\\label{tau-*} \n~~~~\\tau_1 := \n\\begin{cases}\n+\\infty, & \\bfy^*\\leq0,\\\\\n{\\min} \\left\\{\\frac{(y^*_i)^2}{2\\lambda}:y_i^*>0\\right\\},& {\\rm otherwise},\\\\\n\\end{cases}~~\n\\tau_2:=\\begin{cases}\n+\\infty, & \\Gamma_*=\\emptyset,\\\\\n \\frac{2\\lambda}{\\max_i|p^*_i|^2},& {\\rm otherwise},\n\\end{cases}\n\\end{eqnarray}","sha256_utf8":"3b99a7debbc9002bc8054760d5b9b09a0c985855f9a626179a53fd191d3a9b6f"},{"start":32039,"end":32270,"start_line":560,"end_line":560,"raw_tex":"where $\\bfy^*:=A\\bfx^*+\\bfb$ and $\\bfp^*:=-(A_{\\Gamma_*}A_{\\Gamma_*}^{\\top})^{-1}A_{\\Gamma_*}\\nabla f(  \\bfx^*)$. Clearly, both $\\tau_1>0$ and $\\tau_2>0$. Based on these notation, we have the following main result of this section.\n","sha256_utf8":"380afcde2a5421aff9e374c9411d36be7834cf6a5d28c0696e140f48987d3ed2"}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","title":"正定性与满行秩假设","source_units":[{"start":37239,"end":37462,"start_line":642,"end_line":644,"raw_tex":"\\begin{assumption} \\label{ass}\n\t Suppose $f$ is twice continuously differentiable, $\\nabla^2 f(\\bfx^*)$ is {positive definite} and $A_{\\Gamma_*}$ is full row rank, where $\\Gamma_*$ is given by (\\ref{T-*}).\n \\end{assumption}","sha256_utf8":"697f653165091330a8c09380437b6e1d90ffccb392c0722313ddfc9911ccd3d7"}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","title":"海森矩阵局部利普希茨性","source_units":[{"start":37465,"end":37779,"start_line":646,"end_line":650,"raw_tex":"\\begin{assumption} \\label{ass2} \nSuppose $\\nabla^2 f$ is locally Lipschitz continuous around $\\bfx^*$ with a constant $L_*>0$, namely \n\\begin{eqnarray*} \\|\\nabla^2 f(\\bfx)-\\nabla^2 f(\\bfx')\\|\\leq L_* \\|   \\bfx -\\bfx' \\| \\end{eqnarray*}\nfor any $\\bfx$ and $\\bfx'$ in the neighbourhood of $\\bfx^*$.\n \\end{assumption}","sha256_utf8":"abd5a06c17a3c93e3c6c746fe1ef3183981fc99abb24fecb9361193d9bde8484"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"阈值指标集划分","source_units":[{"start":37816,"end":38379,"start_line":653,"end_line":662,"raw_tex":" For a point  $\\bfw:=(\\bfx; \\bfz)$, we define the sets {\n\\begin{eqnarray}\\label{p-stationary-T}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n \\begin{array}{rcll}\n\\S&:=& \\left\\{i\\in\\N_m:  A_i\\bfx+b_i+\\tau z_i   \\in (0, \\sqrt{2\\tau\\lambda}) \\right\\}, \\\\\n \\E&:=& \\left\\{i\\in\\N_m: A_i\\bfx+b_i +\\tau z_i \\in\\{0,\\sqrt{2\\tau\\lambda}\\} \\right\\},\\\\\n\\O&:=& \\left\\{i\\in\\N_m:  A_i\\bfx+b_i+ \\tau z_i \\in(-\\infty,0)\\cup ( \\sqrt{2\\tau\\lambda},\\infty) \\right\\},\\\\\n\\E^o&:=& \\left\\{i\\in\\N_m: A_i\\bfx+b_i =0, \\tau z_i   \\in\\{0,\\sqrt{2\\tau\\lambda}\\} \\right\\}, \n\\end{array}\n\\end{eqnarray}","sha256_utf8":"1fab1783d2b19b7d16a198d82c2db385cef85101ff7e2885ff6ed3f3ee2727ea"},{"start":38381,"end":38864,"start_line":663,"end_line":669,"raw_tex":"for a given $\\tau>0$. \n%The first three sets stand for index sets on which the value $|A_i\\bfx+b_i+\\tau z_i-\\sqrt{\\tau\\lambda/2}|$ is smaller than, equal to and over than $\\sqrt{\\tau\\lambda/2}$, respectively. \nObviously, $\\E^o\\subseteq \\E.$ \nIt is worth mentioning that all sets depend on $\\bfw$. \nFor simplicity, we drop their dependence whenever \nthere is no confusion to be caused. \nSame rules are also applied into $\\S_*,\\E_*,\\O_*$ and $\\E_*^o$ for  $\\bfw^*:=(\\bfx^*; \\bfz^*)$. \n","sha256_utf8":"4ceefdda8ea75787a23082d7dab69162e1f8ff0a02cb9f886e2b4d766caeec27"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"平稳方程与扰动雅可比","source_units":[{"start":39088,"end":39505,"start_line":674,"end_line":689,"raw_tex":"\\begin{eqnarray}\n\\label{sta-eq-1}\n\\hspace{8mm} F(\\bfw; \\Gamma) =\\left[\\begin{array}{rrl}\n\\nabla f(\\bfx)  + A_{\\Gamma}^\\top \\bfz_{\\Gamma}  \\\\\nA_{\\Gamma} \\bfx + \\bfb_{\\Gamma} \\\\\n\\bfz_{\\overline \\Gamma} \n\\end{array}\n\\right] = 0, \\quad \n%\\label{def-Jacobian}\n\\nabla F_{\\mu}(\\bfw;\\Gamma) =\\left[\\begin{array}{ccc}\n\\nabla^2 f(\\bfx)   & A_{\\Gamma} ^\\top &0\\\\\nA_{\\Gamma} &-\\mu I&0 \\\\\n0&0&I\n\\end{array}\n\\right].\n\\end{eqnarray}","sha256_utf8":"fbef5a38d62a94ede83d8c49a05fc040e8c265115c1d8f76a2759693546096c1"},{"start":39507,"end":39659,"start_line":690,"end_line":692,"raw_tex":"We note that the matrix $\\nabla F_{\\mu}$ is a slight perturbation of\nthe Jacobian matrix of $F$ and \n$\\nabla F(\\bfw;\\Gamma)=\\nabla F_{0}(\\bfw;\\Gamma)$,\n","sha256_utf8":"c21e9b2719da5d3eb2da1702f2448c846098c6efe61890b1e2efdb2e06440140"}]},{"id":"S012","semantic_type":"algorithm","kind":"algorithm","title":"迭代指标集选择","source_units":[{"start":44520,"end":44680,"start_line":786,"end_line":787,"raw_tex":"Let $\\bfw^k:=(\\bfx^k; \\bfz^k)$ be the current iterate.\nWe define $\\S_k$ and $\\E_k^o$ by \\eqref{p-stationary-T} with $\\bfw$ being replaced by $\\bfw^k$ and let  \n","sha256_utf8":"beb96114d22b84cf0f4b77df43284cdd183a059236e09a535c2e693f2196f21d"},{"start":44682,"end":44736,"start_line":788,"end_line":790,"raw_tex":"\\beq \\label{T-k}\n      T_k := \\S_k\\cup \\E_k^o.  \n \\eeq","sha256_utf8":"25845fed3810904a682d346ebd078570eeed2e9ab1bf80ff8496430f2dd32cbc"}]},{"id":"S013","semantic_type":"algorithm","kind":"algorithm","title":"平滑牛顿方向","source_units":[{"start":44737,"end":44938,"start_line":791,"end_line":792,"raw_tex":"Let $\\bfd^k=(\\bfu^k;\\bfv^k)$ with $\\bfu^k\\in\\R^n$ and $\\bfv^k\\in\\R^{m}$. For \nsuch a defined $T_k$, a Newton direction $\\bfd^k$ for the equation \\eqref{sta-eq-1} solves the following linear equations:\n","sha256_utf8":"820cc6957ab6c9e6b52987f32c8d7aedb06feb352fdb676223576da99dbc39fb"},{"start":44939,"end":45015,"start_line":793,"end_line":795,"raw_tex":"\\[ %\\beq \\label{newton-d}\n \\nabla F(\\bfw^k;T_k)~\\bfd = - F(\\bfw^k;T_k). \n \\]","sha256_utf8":"7fc948b8017ca5c144349adc1edbfea572d816e836987cb0a5377ab49963e24b"},{"start":45022,"end":45196,"start_line":796,"end_line":798,"raw_tex":"To improve the nonsingularity of the Jacobian matrix $\\nabla F(\\bfw^k;T_k)$,\nwe replace it with $\\nabla F_{\\mu_k}(\\bfw^k;T_k)$. \nThat is, at $\\bfw^k$, we solve the equation:\n","sha256_utf8":"c9e55c48ba36c61a3b81774e40a9b8e3905062bf86d852ef611f2b7c1d2db2c9"},{"start":45197,"end":45283,"start_line":799,"end_line":801,"raw_tex":"\\beq \\label{newton-dir-0}\n \\nabla F_{\\mu_k}(\\bfw^k;T_k)~\\bfd = - F(\\bfw^k;T_k), \n \\eeq","sha256_utf8":"21c8603a17f704d008dd736b3b0226d2fe2e9b694702b779d004bbdc16106741"},{"start":45284,"end":45395,"start_line":802,"end_line":803,"raw_tex":"where $\\nabla F_{\\mu_k}(\\bfw^k;T_k)$ is  defined in \\eqref{sta-eq-1}.\nThe Newton direction $\\bfd^k$ satisfies \n","sha256_utf8":"faa58111bbf845ad031405097f60654116f1f4d99592334e31e3f6b07c75e5d2"},{"start":45397,"end":45873,"start_line":804,"end_line":826,"raw_tex":"\\beq \\label{newton-dir} \n \\left[  \n     \\begin{array}{ccc}\n     {\\nabla^2  f(\\bfx^k)}& A_{T_k}^\\top&0\\\\\n   A_{T_k}&-\\mu_k I&0\\\\\n    0&0&I\n     \\end{array}\n     \\right]  \n     \\left[  \n     \\begin{array}{l}\n    \\bfu^k\\\\\n    \\bfv^k_{T_k}\\\\\n    \\bfv^k_{\\overline T_k}\n     \\end{array}\n     \\right]\n     =-\\left[  \n     \\begin{array}{r}\n    \\nabla  f(\\bfx^k)+A_{T_k}^\\top\\bfz^k_{T_k}\\\\\n   A_{T_k}\\bfx^k+\\bfb_{T_k} \\\\\n    \\bfz^k_{\\overline T_k}\n     \\end{array}\n     \\right].\n \\eeq","sha256_utf8":"3de70cb6ac628dff24b0f0edd426a4c97a9b678a66c88addcdae46c57cfdb656"}]},{"id":"S014","semantic_type":"algorithm","kind":"algorithm","title":"平滑参数更新","source_units":[{"start":46145,"end":46193,"start_line":828,"end_line":828,"raw_tex":"Here, the rule to update $\\mu_k$ is as follows:\n","sha256_utf8":"2db38478d11411a49e25592e70b68e11f0a6b9673ff53e803219a5fc003c3301"},{"start":46196,"end":46283,"start_line":829,"end_line":832,"raw_tex":"\\beq \\label{update-mu} \n\\mu_{k}=\n\\min\\{\\alpha \\mu_{k-1}, \\rho\\|F(\\bfw^k;T_k)\\|\\},\n \\eeq","sha256_utf8":"550f0cf1ca264fed85820183d660639a26a34f0fa560f9e747ee4df89ab11ac1"},{"start":46284,"end":46383,"start_line":833,"end_line":833,"raw_tex":" where $\\alpha\\in(0,1)$ and $\\rho>0$. Now we summarize   the proposed method in  \\cref{Alg-NM01}. \n","sha256_utf8":"d5a6d66da032727f86ae1ce310f2d4bcfe612d99744e97c9e48b23671da656ff"}]},{"id":"S015","semantic_type":"algorithm","kind":"algorithm","title":"零一损失牛顿算法","source_units":[{"start":46383,"end":47015,"start_line":834,"end_line":848,"raw_tex":"\\begin{algorithm}[H]\n\t\\caption{\\nhs: Newton's method for $0/1$-loss optimization}\n\t\\begin{algorithmic}[1] \\label{Alg-NM01}\n\t\t%\\scriptsize\n\t\t\\STATE Initialize $\\bfw^0=(\\bfx^0;\\bfz^0)$ and $\\mu_{-1}>0$. Set the parameter $\\tau,\\lambda,\\rho>0, \\alpha\\in(0,1)$.\\\\\n\t\t Compute $T_0$ by \\eqref{T-k} and set $k :=0$.\n\\IF{ $\\|F(\\bfw^k;T_k)\\|>0$}\t\t \n         \\STATE  Update $\\mu_{k}$ by \\eqref{update-mu}.\t\n\t\t\\STATE  Update $\\bfd^k$ by solving \\eqref{newton-dir}.\n\t\t\\STATE  Update $\\bfw^{k+1}~=~\\bfw^k+\\bfd^k$.\n\t\t\\STATE  Update $T_{k+1}$ by \\eqref{T-k} and  set $k := k+1$.   \t\t\n\\ENDIF\t\t\n\\RETURN $\\bfw^{k}$.\n\t\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"2c7a2f3e73b338dc81bd1f06292c70f1735ce69a73470c63a578f06956b2ff2e"}]},{"id":"S016","semantic_type":"notation","kind":"notation","title":"原始残差记号","source_units":[{"start":48940,"end":49148,"start_line":860,"end_line":860,"raw_tex":"The first technical result is about extending the stationarity equation to some indices that are given in a neighborhood of $\\bfw^*$. In the proof, we recall $\\bfy = A\\bfx+\\bfb$ and $\\bfy^* = A\\bfx^*+\\bfb$. \n","sha256_utf8":"448b596bc35ba52306ef4c89ea9f8e9f44a31aede5417695660c885a02cca3c2"}]},{"id":"S017","semantic_type":"lemma","kind":"lemma","title":"次微分公式与链式法则","source_units":[{"start":18619,"end":19518,"start_line":322,"end_line":352,"raw_tex":"\\begin{lemma} \\label{subdifferentials}\n\n\\begin{itemize}\n\t\n \\item[(i)] We have\n \\begin{eqnarray}\\label{subfi}\n  \\partial\\|{\\bfy}_+\\|_0\n  =\\left\\{{\\bfv}\\in {\\R}^m: v_i\\left\\{\\begin{array}{ll}\n \\geq0,& y_i=0,\\\\\n =0,& y_i\\neq0,\n \\end{array}\n \\right. i\\in{\\N}_m\\right\\}. \n \\end{eqnarray}\n \n \\item[(ii)] Let $g(\\bfx):=\\|(h(\\bfx))_+\\|_0$, where $h:\\R^n\\rightarrow\\R^m$ is differentiable,  and $\\Gamma:=\\{i\\in\\N_m: h_i(  \\bfx)=0\\}$ for a given a point $ \\bfx\\in\\R^n$.  If \n \\begin{eqnarray}\n \\label{chain-rule-cond}\n \\left.\n \\begin{array}{r}\n \\forall \\bfz\\in\\R^{|\\Gamma|},~~\\bfz\\geq 0,~~\n (\\nabla h( \\bfx))_{\\Gamma}^\\top \\bfz = 0\n \\end{array}\\right.~~\\Longrightarrow~~\\bfz = 0,\n \\end{eqnarray}\n then the subdifferential of $g(\\bfx)$  at $  \\bfx$ is  \n \\begin{eqnarray}\n \\label{chain-rule-sub} \n \\partial g(  \\bfx)=\\nabla h( \\bfx)^\\top \\partial \\|(h(  \\bfx))_+\\|_0.\n \\end{eqnarray}\n\n\\end{itemize}\n\\end{lemma}","sha256_utf8":"0219cc4baee5412543f6d4fc134ca133340bca31a072d0a1ba7918e5293c779c"}]},{"id":"S018","semantic_type":"proof","kind":"proof","title":"次微分引理完整证明","source_units":[{"start":19520,"end":22233,"start_line":354,"end_line":384,"raw_tex":"\\begin{proof}\ni)   It is straight to check that the regular subdifferential $\\widehat{\\partial}\\|{\\bfy}_+\\|_0$ (see \\cite[Definition 8.3]{RW1998})  of $\\|{\\bfy}_+\\|_0$ at ${\\bfy}$ takes the following form,\n\\begin{eqnarray}\\label{subfi-0}\n \\widehat{\\partial}\\|{\\bfy}_+\\|_0=\\left\\{{\\bfv}\\in {\\R}^m: v_i\\left\\{\\begin{array}{ll}\n \\geq0,& y_i=0,\\\\\n =0,& y_i\\neq0,\n \\end{array}\n \\right. i\\in{\\N}_m\\right\\}=:\\Omega(\\bfy). \n\\end{eqnarray}\nWe next verify $ \\partial\\|{\\bfy}_+\\|_0= \\Omega({\\bfy}) $. By letting $\\varphi(\\cdot):=\\|(\\cdot)_+\\|_0$, we have\n  \\begin{eqnarray}\\label{sub-lim-reg}\n  \\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n  \\begin{array}{rclll}\n\\partial\\|{\\bfy}_+\\|_0&=&\\limsup_{{\\bfz}\\stackrel{\\varphi}{\\rightarrow}{\\bfy}}~ \\widehat{\\partial}\\|{\\bfz}_+\\|_0 ~~~( \\text{by  \\cite[Equation 8(5)]{RW1998}})\\\\\n&=&\\limsup_{{\\bfz}\\stackrel{\\varphi}{\\rightarrow}{\\bfy}}~\\Omega({\\bfz}) ~~~~~~( \\text{by \\eqref{subfi-0}})\\\\\n&=&\\{\\bfv\\in \\R^m:\\exists~\\bfz^k\\stackrel{\\varphi}{\\rightarrow} \\bfy,~\\bfv^k\\rightarrow \\bfv ~\\text{with}~\\bfv^k\\in \\Omega(\\bfz^k) \\}=:\\Theta,\n\\end{array}\n \\end{eqnarray}\nwhere  $\\bfz\\stackrel{\\varphi}{\\rightarrow} \\bfy$ represents $\\bfz\\rightarrow \\bfy, \\varphi(\\bfz)\\rightarrow \\varphi(\\bfy)$. Clearly, $\\Omega({\\bfy})\\subseteq\\Theta$. On the other hand, $\\Theta \\subseteq \\Omega({\\bfy})$  follows from that $\\Omega(\\bfz^k)\\subseteq \\Omega(\\bfy)$ for any $\\bfz^k\\stackrel{\\varphi}{\\rightarrow} \\bfy$ and $\\Omega(\\cdot)$ is closed.\n \n ii) Direct verifications yield the following chain of equations,\n \\begin{eqnarray*}  \\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n  \\begin{array}{rclll}\n \\partial^\\infty \\|\\bfy_+\\|_0&=& {\\limsup}_{\\sigma\\downarrow0,~{\\bfz}\\stackrel{\\varphi}{\\rightarrow} {\\bfy}} \\sigma \\widehat{\\partial}\\|\\bfz_+\\|_0&~~( \\text{by \\cite[Equation 8(5)]{RW1998}})\\\\ \n&=&\n {\\limsup}_{{\\bfz}\\stackrel{\\varphi}{\\rightarrow} {\\bfy}}~ \\widehat{\\partial}\\|\\bfz_+\\|_0&~~( \\text{by \\eqref{subfi-0}})\\\\\n & =& \\partial\\|{\\bfy}_+\\|_0,&~~( \\text{by \\eqref{sub-lim-reg}})\\\\\n \\end{array}\n \\end{eqnarray*}\n where $ \\partial^\\infty \\|{\\bfy}_+\\|_0$ is the horizon subdifferential of $\\|{\\bfy}_+\\|_0$. Therefore, we derive that $\\widehat{\\partial}\\varphi=\\partial \\varphi=\\partial^\\infty \\varphi$.  One can easily prove that the horizon cone $\\widehat{\\partial}\\varphi(\\bfy)^\\infty$ (see \\cite[Definition 3.3]{RW1998}) of $ \\widehat{\\partial}\\varphi(\\bfy)$ satisfies $\\widehat{\\partial}\\varphi(\\bfy)^\\infty = \\partial^ \\infty\\varphi(\\bfy)$. These conditions indicate that  the function $\\varphi$ is regular by \\cite[Corollary 8.11]{RW1998}, which together with \\eqref{chain-rule-cond}, $g(\\bfx)=\\varphi(h(\\bfx))$ and \\cite[Theorem 10.6]{RW1998} derives \\eqref{chain-rule-sub} immediately. \n\\end{proof}","sha256_utf8":"cfde96f254c45f36f8556154d7ce75fc11b729e81d0bd10c8b8876d164cbeeaf"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"计算正则次微分","source_units":[{"start":19520,"end":19951,"start_line":354,"end_line":362,"raw_tex":"\\begin{proof}\ni)   It is straight to check that the regular subdifferential $\\widehat{\\partial}\\|{\\bfy}_+\\|_0$ (see \\cite[Definition 8.3]{RW1998})  of $\\|{\\bfy}_+\\|_0$ at ${\\bfy}$ takes the following form,\n\\begin{eqnarray}\\label{subfi-0}\n \\widehat{\\partial}\\|{\\bfy}_+\\|_0=\\left\\{{\\bfv}\\in {\\R}^m: v_i\\left\\{\\begin{array}{ll}\n \\geq0,& y_i=0,\\\\\n =0,& y_i\\neq0,\n \\end{array}\n \\right. i\\in{\\N}_m\\right\\}=:\\Omega(\\bfy). \n\\end{eqnarray}\n","sha256_utf8":"c47511ca3cdd29c22e78a53d3b6d6562dd0dbe4fd8a5d7f375644d66fb702383"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"辨认极限次微分","source_units":[{"start":19951,"end":20971,"start_line":363,"end_line":372,"raw_tex":"We next verify $ \\partial\\|{\\bfy}_+\\|_0= \\Omega({\\bfy}) $. By letting $\\varphi(\\cdot):=\\|(\\cdot)_+\\|_0$, we have\n  \\begin{eqnarray}\\label{sub-lim-reg}\n  \\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n  \\begin{array}{rclll}\n\\partial\\|{\\bfy}_+\\|_0&=&\\limsup_{{\\bfz}\\stackrel{\\varphi}{\\rightarrow}{\\bfy}}~ \\widehat{\\partial}\\|{\\bfz}_+\\|_0 ~~~( \\text{by  \\cite[Equation 8(5)]{RW1998}})\\\\\n&=&\\limsup_{{\\bfz}\\stackrel{\\varphi}{\\rightarrow}{\\bfy}}~\\Omega({\\bfz}) ~~~~~~( \\text{by \\eqref{subfi-0}})\\\\\n&=&\\{\\bfv\\in \\R^m:\\exists~\\bfz^k\\stackrel{\\varphi}{\\rightarrow} \\bfy,~\\bfv^k\\rightarrow \\bfv ~\\text{with}~\\bfv^k\\in \\Omega(\\bfz^k) \\}=:\\Theta,\n\\end{array}\n \\end{eqnarray}\nwhere  $\\bfz\\stackrel{\\varphi}{\\rightarrow} \\bfy$ represents $\\bfz\\rightarrow \\bfy, \\varphi(\\bfz)\\rightarrow \\varphi(\\bfy)$. Clearly, $\\Omega({\\bfy})\\subseteq\\Theta$. On the other hand, $\\Theta \\subseteq \\Omega({\\bfy})$  follows from that $\\Omega(\\bfz^k)\\subseteq \\Omega(\\bfy)$ for any $\\bfz^k\\stackrel{\\varphi}{\\rightarrow} \\bfy$ and $\\Omega(\\cdot)$ is closed.\n","sha256_utf8":"ef3d0daf82f82e5b6b972a1be5cde7ba53f113db41f7dae9704cfc1e2f4012e5"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"验证正则性并应用链式法则","source_units":[{"start":20973,"end":22233,"start_line":374,"end_line":384,"raw_tex":" ii) Direct verifications yield the following chain of equations,\n \\begin{eqnarray*}  \\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n  \\begin{array}{rclll}\n \\partial^\\infty \\|\\bfy_+\\|_0&=& {\\limsup}_{\\sigma\\downarrow0,~{\\bfz}\\stackrel{\\varphi}{\\rightarrow} {\\bfy}} \\sigma \\widehat{\\partial}\\|\\bfz_+\\|_0&~~( \\text{by \\cite[Equation 8(5)]{RW1998}})\\\\ \n&=&\n {\\limsup}_{{\\bfz}\\stackrel{\\varphi}{\\rightarrow} {\\bfy}}~ \\widehat{\\partial}\\|\\bfz_+\\|_0&~~( \\text{by \\eqref{subfi-0}})\\\\\n & =& \\partial\\|{\\bfy}_+\\|_0,&~~( \\text{by \\eqref{sub-lim-reg}})\\\\\n \\end{array}\n \\end{eqnarray*}\n where $ \\partial^\\infty \\|{\\bfy}_+\\|_0$ is the horizon subdifferential of $\\|{\\bfy}_+\\|_0$. Therefore, we derive that $\\widehat{\\partial}\\varphi=\\partial \\varphi=\\partial^\\infty \\varphi$.  One can easily prove that the horizon cone $\\widehat{\\partial}\\varphi(\\bfy)^\\infty$ (see \\cite[Definition 3.3]{RW1998}) of $ \\widehat{\\partial}\\varphi(\\bfy)$ satisfies $\\widehat{\\partial}\\varphi(\\bfy)^\\infty = \\partial^ \\infty\\varphi(\\bfy)$. These conditions indicate that  the function $\\varphi$ is regular by \\cite[Corollary 8.11]{RW1998}, which together with \\eqref{chain-rule-cond}, $g(\\bfx)=\\varphi(h(\\bfx))$ and \\cite[Theorem 10.6]{RW1998} derives \\eqref{chain-rule-sub} immediately. \n\\end{proof}","sha256_utf8":"478c7ffadb6cb4106a8ecc300458e45f6dec98be7a8c9a047dd3cf5925cbba19"}]},{"id":"S022","semantic_type":"lemma","kind":"lemma","title":"局部最优性的一阶刻画","source_units":[{"start":24225,"end":24765,"start_line":425,"end_line":437,"raw_tex":"\\begin{lemma}\\label{first-order-nec}\n\tThe following relationships hold for the problem (\\ref{ell-0-1}).\n\\begin{itemize}\n\t\n\\item[i)] A local minimizer $\\bfx^*$   satisfies the following condition \n         if  $A_{\\Gamma_*}$ is full row rank,\n\\begin{eqnarray} \\label{lemma-1st-order-nec}\n    -\\nabla f(\\bfx^*) \\in A^\\top \\partial \\|(A\\bfx^*+\\bfb)_+\\|_0.\n\\end{eqnarray}\n \n\\item[ii)] A point $\\bfx^*$ satisfying (\\ref{lemma-1st-order-nec})  is a local minimizer if  the function $f$ is locally convex around $\\bfx^*$.\n\\end{itemize}\n\\end{lemma}","sha256_utf8":"baf90a44ff00a9cac72d08f8aa2feec7c1b31e5b5375e4c8bf85bd1bce63647a"}]},{"id":"S023","semantic_type":"proof","kind":"proof","title":"一阶最优性引理完整证明","source_units":[{"start":24828,"end":30204,"start_line":439,"end_line":519,"raw_tex":"\\begin{proof}\ni) It follows from \\cite[Theorem 10.1]{RW1998} that a local minimizer of \\eqref{ell-0-1} must satisfy $-\\nabla f(\\bfx^*)\\in\\lambda\\partial g(\\bfx^*)$, where $g(\\bfx):=\\|(A\\bfx+\\bfb)_+\\|_0$. This together with \\cref{chain-rule-sub}  and $\\lambda \\partial\\|(\\cdot)_+\\|_0 = \\partial \\|(\\cdot)_+\\|_0$ by \\eqref{subfi} derives the result immediately.  \n\nii)   Since the problem \\eqref{ell-0-1} is equivalent to the following problem, \n\\begin{eqnarray}\\label{ell-0-1-equ-0}\n&\\underset{\\bfx\\in\\R^n,\\bfy\\in\\R^m}{\\min}& f(\\bfx)+\\lambda \\|\\bfy_+\\|_0,\\\\\n  &{\\rm s.t.}& A\\bfx+\\bfb-\\bfy = 0,\\nonumber\n\\end{eqnarray}\nit suffices to show that  $(\\bfx^*;\\bfy^*)$  is a local minimizer of the   problem \\eqref{ell-0-1-equ-0}, where $\\bfx^*$ satisfies \\eqref{lemma-1st-order-nec} and $\\bfy^*=A \\bfx^*+\\bfb$, namely, there is a $\\bfz^*$ such that\n\\begin{eqnarray}\n\\label{lemma-1st-order-nec-10}\n\\nabla f(\\bfx^*) +A^\\top\\bfz^*=0,~~ \nA \\bfx^*+\\bfb-\\bfy^*=0, ~~\n\\partial\\|\\bfy^*_+\\|_0\\ni \\bfz^*. \n\\end{eqnarray} \nIt follows from $\\bfy^*=A\\bfx^*+\\bfb$ and \\eqref{T-*} that $\\Gamma_*=\\{i\\in\\N_m: y^*_i=0\\}$. This together with $\\partial\\|\\bfy^*_+\\|_0 \\ni \\bfz^*$ and the expression of the $\\partial\\|\\bfy^*_+\\|_0$ in  \\eqref{subfi} indicates \n\\begin{eqnarray}\\label{fact-11}\n\\bfy_{\\Gamma_*}^*=0,~~\\bfz_{\\Gamma_*}^* \\geq 0, ~~~~\\bfy_{\\overline \\Gamma_*}^*\\neq 0,~ ~\\bfz_{\\overline \\Gamma_*}^* = 0. \n\\end{eqnarray} \n Define a radius $\\delta:=\\min\\{\\delta_1,\\delta_2\\}$, where\n\\begin{eqnarray}\\label{delta-2}\n~~~~~~\\delta_1 := \n\\begin{cases}\n+\\infty,&A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}=0\\\\\n\\frac{\\lambda}{ \\|A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}\\|},& \\text{otherwise}, \n\\end{cases}~~~~\n\\delta_2 := \n\\begin{cases}\n+\\infty,&\\bfy^*\\leq {\\bf0},\\\\\n\\min_i\\{y_i^*:y_i^*>0\\},&\\text{otherwise}, \n\\end{cases}\n\\end{eqnarray}\nand consider a local region of ${\\bfw}^* := (\\bfx^*;\\bfy^*)$ by\n\\begin{eqnarray}\\label{local-area-1}\n~~N( {\\bfw}^*,\\delta)=\\left\\{ \\left(\n\\bfx; \n  {\\bfy}\\right)\\in\\R^{n+m}   \n:~A\\bfx+\\bfb - \\bfy=0, ~\n \\|{\\bfw}- {\\bfw}^*\\|<\\delta\n\\right\\}.\n\\end{eqnarray}\nIndeed, $N( {\\bfw}^*,\\delta)$ is a neighbourhood of ${\\bfw}^*$ since $A\\bfx^*+\\bfb - \\bfy^*=0$ from \\eqref{lemma-1st-order-nec-10}. Next we show that, for any $\\bfw\\in N( {\\bfw}^*,\\delta)$,\n\\begin{eqnarray}\\label{y-y*-0}\n\\|\\bfy^*_+\\|_0\\leq \\|\\bfy_+\\|_0.\n\\end{eqnarray}\nObviously, it is true if $\\bfy^*\\leq0$ as $\\|\\bfy^*_+\\|_0=0$. For $\\bfy^*\\nleq0$,  to guarantee \\eqref{y-y*-0}, it suffices to show that for any $i$, $y_i^*>0~\\Longrightarrow~y_i>0.$ \nSuppose there is a $j\\in\\N_m$ such that $y_j^*>0$ but $y_j\\leq0$.  This incurs the following contradiction \n\\begin{eqnarray*}\\delta_2 \\geq \\delta\n& {>}&\\|\\bfw-\\bfw^*\\| \\hspace{3.5cm} ({\\rm by}~ \\eqref{local-area-1})\\\\\n&\\geq& |y_j-y_j^*|=y_j^*-y_j  \\geq y_j^*\\geq\\delta_2.\\hspace{.5cm} ({\\rm by}~ \\eqref{delta-2}) \\end{eqnarray*}\nAgain, for any $\\bfw\\in N( {\\bfw}^*,\\delta)$, we have $A\\bfx+\\bfb - \\bfy=0$, which and \\eqref{lemma-1st-order-nec-10} generate\n \\begin{eqnarray}\\label{y-y*-h-h*}\n \\bfy-\\bfy^*=A (\\bfx- \\bfx^*).\n  \\end{eqnarray}\nNext, the convexity of $f$ gives rise to  \n \\begin{eqnarray}\\label{fx-fx*}\n\\hspace{15mm} f(\\bfx)-f(\\bfx^*)&\\geq & \\langle \\nabla f(\\bfx^*), \\bfx - \\bfx^*\\rangle~~= -\\langle  A^\\top \\bfz^* , \\bfx - \\bfx^*\\rangle \\hspace{1.6cm} ({\\rm by}~ \\eqref{lemma-1st-order-nec-10})\\nonumber\\\\\n% &=& -\\langle  A_{\\Gamma_*}  (\\bfx - \\bfx^*),\\bfz^* _{\\Gamma_*}\\rangle-\\langle   A_{\\overline  \\Gamma_*} (\\bfx - \\bfx^*),\\bfz^* _{\\overline  \\Gamma_*}\\rangle\\nonumber\\\\\n &=& -\\langle  A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*},\\bfx - \\bfx^*\\rangle= -\\langle  A_{\\Gamma_*}(\\bfx - \\bfx^*),\\bfz^* _{\\Gamma_*}\\rangle=:\\phi\\hspace{0.3cm} ({\\rm by}~ \\eqref{fact-11})\n  \\end{eqnarray} \n Now we make the conclusion by two cases.  \n If $\\|\\bfy^*_+\\|_0= \\|\\bfy_+\\|_0$, then   $\\bfy_{\\Gamma_*}\\leq 0$ due to $\\bfy^*_{\\Gamma_*}=0$. This and \\eqref{y-y*-h-h*} yield that\n  \\begin{eqnarray}\\label{y-y*-h-h*-0}\n0\\geq  \\bfy_{\\Gamma_*}= \\bfy_{\\Gamma_*}-\\bfy^*_{\\Gamma_*}  = A_{\\Gamma_*} (\\bfx-\\bfx^*),\n  \\end{eqnarray}\nwhich together with $\\bfz^* _{\\Gamma_*}\\geq0$ from \\eqref{fact-11} indicates $\\phi\\geq0$, namely $f(\\bfx)\\geq f(\\bfx^*)$. So $$f(\\bfx) +\\lambda \\|\\bfy_+\\|_0 \\geq f(\\bfx^*) + \\lambda\\|\\bfy_+^*\\|_0.$$\nIf $\\|\\bfy^*_+\\|_0 \\neq \\|\\bfy_+\\|_0$, we must have $ \\|\\bfy_+\\|_0 \\geq 1+\\|\\bfy^*_+\\|_0$ by \\eqref{y-y*-0}. If  $A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}=0$, then $\\phi=0>-\\lambda$. Otherwise, it follows\n  \\begin{eqnarray*}\n\\phi&\\geq&- \\|A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}\\|\\|\\bfx - \\bfx^*\\| \\geq - \\|A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}\\|\\|\\bfw - \\bfw^*\\|\\\\\n & \\geq&- \\|A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}\\|\\delta \n  \\geq - \\|A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}\\|\\delta_1=-\\lambda.\\hspace{.5cm} ({\\rm by}~ \\eqref{delta-2})\n  \\end{eqnarray*}\nBoth lead to $\\phi\\geq-\\lambda$, which results in\n  \\begin{eqnarray*}\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n  \\begin{array}{rclll}\n  f(\\bfx) +\\lambda \\|\\bfy_+\\|_0 &\\geq& f(\\bfx^*) + \\phi +\\lambda \\|\\bfy_+\\|_0 &\\ ({\\rm by}~ \\eqref{fx-fx*} )\\\\\n&\\geq& f(\\bfx^*) -\\lambda + \\lambda\\|\\bfy_+\\|_0 &\\ ({\\rm by}~ \\phi\\geq-\\lambda)\\\\\n&\\geq& f(\\bfx^*) + \\lambda\\|\\bfy_+^*\\|_0.&\\ ({\\rm by}~ \\|\\bfy_+\\|_0 \\geq 1+\\|\\bfy^*_+\\|_0 )\n \\end{array} \\end{eqnarray*}\nOverall, the two cases show that $(\\bfx^*;\\bfy^*)$ is  a local minimizer to \\eqref{ell-0-1-equ-0}. Namely, $\\bfx^*$ is  a local minimizer to \\eqref{ell-0-1}. \n\\end{proof}","sha256_utf8":"19ceb2643ac9e1855fde8dd5cac1d19669d72c4015e5313f3a858d4cb763b120"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"由费马条件推出必要性","source_units":[{"start":24828,"end":25190,"start_line":439,"end_line":440,"raw_tex":"\\begin{proof}\ni) It follows from \\cite[Theorem 10.1]{RW1998} that a local minimizer of \\eqref{ell-0-1} must satisfy $-\\nabla f(\\bfx^*)\\in\\lambda\\partial g(\\bfx^*)$, where $g(\\bfx):=\\|(A\\bfx+\\bfb)_+\\|_0$. This together with \\cref{chain-rule-sub}  and $\\lambda \\partial\\|(\\cdot)_+\\|_0 = \\partial \\|(\\cdot)_+\\|_0$ by \\eqref{subfi} derives the result immediately.  \n","sha256_utf8":"e6e5fbd0233b3d81b8154e91a4a628f8f63cd5a459d427eefa309e3945f54c0d"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"引入残差变量与乘子结构","source_units":[{"start":25191,"end":26232,"start_line":442,"end_line":457,"raw_tex":"ii)   Since the problem \\eqref{ell-0-1} is equivalent to the following problem, \n\\begin{eqnarray}\\label{ell-0-1-equ-0}\n&\\underset{\\bfx\\in\\R^n,\\bfy\\in\\R^m}{\\min}& f(\\bfx)+\\lambda \\|\\bfy_+\\|_0,\\\\\n  &{\\rm s.t.}& A\\bfx+\\bfb-\\bfy = 0,\\nonumber\n\\end{eqnarray}\nit suffices to show that  $(\\bfx^*;\\bfy^*)$  is a local minimizer of the   problem \\eqref{ell-0-1-equ-0}, where $\\bfx^*$ satisfies \\eqref{lemma-1st-order-nec} and $\\bfy^*=A \\bfx^*+\\bfb$, namely, there is a $\\bfz^*$ such that\n\\begin{eqnarray}\n\\label{lemma-1st-order-nec-10}\n\\nabla f(\\bfx^*) +A^\\top\\bfz^*=0,~~ \nA \\bfx^*+\\bfb-\\bfy^*=0, ~~\n\\partial\\|\\bfy^*_+\\|_0\\ni \\bfz^*. \n\\end{eqnarray} \nIt follows from $\\bfy^*=A\\bfx^*+\\bfb$ and \\eqref{T-*} that $\\Gamma_*=\\{i\\in\\N_m: y^*_i=0\\}$. This together with $\\partial\\|\\bfy^*_+\\|_0 \\ni \\bfz^*$ and the expression of the $\\partial\\|\\bfy^*_+\\|_0$ in  \\eqref{subfi} indicates \n\\begin{eqnarray}\\label{fact-11}\n\\bfy_{\\Gamma_*}^*=0,~~\\bfz_{\\Gamma_*}^* \\geq 0, ~~~~\\bfy_{\\overline \\Gamma_*}^*\\neq 0,~ ~\\bfz_{\\overline \\Gamma_*}^* = 0. \n\\end{eqnarray} \n","sha256_utf8":"2b5f8afdc8a1f37e03368e7c973044e5df2152d71582f8e1c4c14504f9e9f2f3"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"构造邻域并保持正坐标","source_units":[{"start":26232,"end":27664,"start_line":458,"end_line":488,"raw_tex":" Define a radius $\\delta:=\\min\\{\\delta_1,\\delta_2\\}$, where\n\\begin{eqnarray}\\label{delta-2}\n~~~~~~\\delta_1 := \n\\begin{cases}\n+\\infty,&A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}=0\\\\\n\\frac{\\lambda}{ \\|A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}\\|},& \\text{otherwise}, \n\\end{cases}~~~~\n\\delta_2 := \n\\begin{cases}\n+\\infty,&\\bfy^*\\leq {\\bf0},\\\\\n\\min_i\\{y_i^*:y_i^*>0\\},&\\text{otherwise}, \n\\end{cases}\n\\end{eqnarray}\nand consider a local region of ${\\bfw}^* := (\\bfx^*;\\bfy^*)$ by\n\\begin{eqnarray}\\label{local-area-1}\n~~N( {\\bfw}^*,\\delta)=\\left\\{ \\left(\n\\bfx; \n  {\\bfy}\\right)\\in\\R^{n+m}   \n:~A\\bfx+\\bfb - \\bfy=0, ~\n \\|{\\bfw}- {\\bfw}^*\\|<\\delta\n\\right\\}.\n\\end{eqnarray}\nIndeed, $N( {\\bfw}^*,\\delta)$ is a neighbourhood of ${\\bfw}^*$ since $A\\bfx^*+\\bfb - \\bfy^*=0$ from \\eqref{lemma-1st-order-nec-10}. Next we show that, for any $\\bfw\\in N( {\\bfw}^*,\\delta)$,\n\\begin{eqnarray}\\label{y-y*-0}\n\\|\\bfy^*_+\\|_0\\leq \\|\\bfy_+\\|_0.\n\\end{eqnarray}\nObviously, it is true if $\\bfy^*\\leq0$ as $\\|\\bfy^*_+\\|_0=0$. For $\\bfy^*\\nleq0$,  to guarantee \\eqref{y-y*-0}, it suffices to show that for any $i$, $y_i^*>0~\\Longrightarrow~y_i>0.$ \nSuppose there is a $j\\in\\N_m$ such that $y_j^*>0$ but $y_j\\leq0$.  This incurs the following contradiction \n\\begin{eqnarray*}\\delta_2 \\geq \\delta\n& {>}&\\|\\bfw-\\bfw^*\\| \\hspace{3.5cm} ({\\rm by}~ \\eqref{local-area-1})\\\\\n&\\geq& |y_j-y_j^*|=y_j^*-y_j  \\geq y_j^*\\geq\\delta_2.\\hspace{.5cm} ({\\rm by}~ \\eqref{delta-2}) \\end{eqnarray*}\n","sha256_utf8":"884dc2a460aea985ed38d8fcb372f888f519bea56b4222962dce92122b963f2e"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"建立光滑目标的凸性下界","source_units":[{"start":27664,"end":28527,"start_line":489,"end_line":498,"raw_tex":"Again, for any $\\bfw\\in N( {\\bfw}^*,\\delta)$, we have $A\\bfx+\\bfb - \\bfy=0$, which and \\eqref{lemma-1st-order-nec-10} generate\n \\begin{eqnarray}\\label{y-y*-h-h*}\n \\bfy-\\bfy^*=A (\\bfx- \\bfx^*).\n  \\end{eqnarray}\nNext, the convexity of $f$ gives rise to  \n \\begin{eqnarray}\\label{fx-fx*}\n\\hspace{15mm} f(\\bfx)-f(\\bfx^*)&\\geq & \\langle \\nabla f(\\bfx^*), \\bfx - \\bfx^*\\rangle~~= -\\langle  A^\\top \\bfz^* , \\bfx - \\bfx^*\\rangle \\hspace{1.6cm} ({\\rm by}~ \\eqref{lemma-1st-order-nec-10})\\nonumber\\\\\n% &=& -\\langle  A_{\\Gamma_*}  (\\bfx - \\bfx^*),\\bfz^* _{\\Gamma_*}\\rangle-\\langle   A_{\\overline  \\Gamma_*} (\\bfx - \\bfx^*),\\bfz^* _{\\overline  \\Gamma_*}\\rangle\\nonumber\\\\\n &=& -\\langle  A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*},\\bfx - \\bfx^*\\rangle= -\\langle  A_{\\Gamma_*}(\\bfx - \\bfx^*),\\bfz^* _{\\Gamma_*}\\rangle=:\\phi\\hspace{0.3cm} ({\\rm by}~ \\eqref{fact-11})\n  \\end{eqnarray} \n","sha256_utf8":"64ab1d77097e18cfa733b10e3838b32616c01a0f8ba473176cad1405823d5766"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"处理正坐标数不变情形","source_units":[{"start":28527,"end":29049,"start_line":499,"end_line":504,"raw_tex":" Now we make the conclusion by two cases.  \n If $\\|\\bfy^*_+\\|_0= \\|\\bfy_+\\|_0$, then   $\\bfy_{\\Gamma_*}\\leq 0$ due to $\\bfy^*_{\\Gamma_*}=0$. This and \\eqref{y-y*-h-h*} yield that\n  \\begin{eqnarray}\\label{y-y*-h-h*-0}\n0\\geq  \\bfy_{\\Gamma_*}= \\bfy_{\\Gamma_*}-\\bfy^*_{\\Gamma_*}  = A_{\\Gamma_*} (\\bfx-\\bfx^*),\n  \\end{eqnarray}\nwhich together with $\\bfz^* _{\\Gamma_*}\\geq0$ from \\eqref{fact-11} indicates $\\phi\\geq0$, namely $f(\\bfx)\\geq f(\\bfx^*)$. So $$f(\\bfx) +\\lambda \\|\\bfy_+\\|_0 \\geq f(\\bfx^*) + \\lambda\\|\\bfy_+^*\\|_0.$$\n","sha256_utf8":"0287b1d17f9b1f04e04ef76ca21bb43a0b37cef470d38472923f933cf05ed84b"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"处理正坐标数增加情形","source_units":[{"start":29049,"end":30204,"start_line":505,"end_line":519,"raw_tex":"If $\\|\\bfy^*_+\\|_0 \\neq \\|\\bfy_+\\|_0$, we must have $ \\|\\bfy_+\\|_0 \\geq 1+\\|\\bfy^*_+\\|_0$ by \\eqref{y-y*-0}. If  $A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}=0$, then $\\phi=0>-\\lambda$. Otherwise, it follows\n  \\begin{eqnarray*}\n\\phi&\\geq&- \\|A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}\\|\\|\\bfx - \\bfx^*\\| \\geq - \\|A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}\\|\\|\\bfw - \\bfw^*\\|\\\\\n & \\geq&- \\|A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}\\|\\delta \n  \\geq - \\|A_{\\Gamma_*}^\\top\\bfz^* _{\\Gamma_*}\\|\\delta_1=-\\lambda.\\hspace{.5cm} ({\\rm by}~ \\eqref{delta-2})\n  \\end{eqnarray*}\nBoth lead to $\\phi\\geq-\\lambda$, which results in\n  \\begin{eqnarray*}\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n  \\begin{array}{rclll}\n  f(\\bfx) +\\lambda \\|\\bfy_+\\|_0 &\\geq& f(\\bfx^*) + \\phi +\\lambda \\|\\bfy_+\\|_0 &\\ ({\\rm by}~ \\eqref{fx-fx*} )\\\\\n&\\geq& f(\\bfx^*) -\\lambda + \\lambda\\|\\bfy_+\\|_0 &\\ ({\\rm by}~ \\phi\\geq-\\lambda)\\\\\n&\\geq& f(\\bfx^*) + \\lambda\\|\\bfy_+^*\\|_0.&\\ ({\\rm by}~ \\|\\bfy_+\\|_0 \\geq 1+\\|\\bfy^*_+\\|_0 )\n \\end{array} \\end{eqnarray*}\nOverall, the two cases show that $(\\bfx^*;\\bfy^*)$ is  a local minimizer to \\eqref{ell-0-1-equ-0}. Namely, $\\bfx^*$ is  a local minimizer to \\eqref{ell-0-1}. \n\\end{proof}","sha256_utf8":"1a8cc630f1763fef317451e747a5d4dba2119cb27c6853ffbc38d9ca308dabc0"}]},{"id":"S030","semantic_type":"theorem","kind":"theorem","title":"近端平稳性与最优性","source_units":[{"start":32270,"end":32846,"start_line":561,"end_line":568,"raw_tex":"\\begin{theorem} \\label{Stationary-Thm}\n   The following relationships hold for the problem (\\ref{ell-0-1}).\n\\begin{itemize}\n\\item[i)]  A local minimizer $\\bfx^*$ is a  P-stationary point for any $0<\\tau<\\tau_*:=\\min\\{\\tau_1,\\tau_2\\}$ if  $A_{\\Gamma_*}$ is full row rank. \n\\item[ii)] A P-stationary point with $ \\tau>0$  is a local minimizer if  the function $f$ is locally convex around $\\bfx^*$. \n\\item[iii)] A P-stationary point with $ \\tau\\geq\\|A\\|^2/c_f$  is   a global minimizer if  the function $f$ is strongly convex with a constant $c_f>0$.\n\\end{itemize}\n\\end{theorem}","sha256_utf8":"2ba033af37388852f072c2d9655a0b0d4a2c7dc12743374a2d212062d649efb0"}]},{"id":"S031","semantic_type":"proof","kind":"proof","title":"平稳最优性定理完整证明","source_units":[{"start":32848,"end":36973,"start_line":570,"end_line":633,"raw_tex":"\\begin{proof} i) As  $\\bfx^*$ is a local minimizer of \\eqref{ell-0-1},  condition  \\eqref{lemma-1st-order-nec} is valid by \\cref{first-order-nec} if $A_{\\Gamma_*}$ is  full row rank. In other words, there is a $\\bfz^*$ such that\n\\begin{eqnarray}\n\\label{lemma-1st-order-nec-1}\n\\nabla f(\\bfx^*) +A^\\top\\bfz^*=0,~~\nA \\bfx^*+\\bfb-\\bfy^*=0,~~\n\\bfz^*\\in\\partial\\|\\bfy^*_+\\|_0. \n\\end{eqnarray} \nTherefore, to show \\eqref{p-stationary}, we only need to verify that,  for any $0<\\tau<\\tau^*$, $$\\bfz^*\\in \\partial\\|\\bfy^*_+\\|_0~ \\Rightarrow~\\bfy^*\\in\\P :=\\pro (\\bfy^*+\\tau \\bfz^*). $$ \nRecall the definition of $\\Gamma_*$ in \\eqref{T-*} and the second condition in \\eqref{lemma-1st-order-nec-1}, we have $\\bfy_{\\Gamma_*}=(A \\bfx^*+\\bfb)_{\\Gamma_*}=0$. Same reasoning also allows for obtaining \\eqref{fact-11} due to $\\bfz^*\\in \\partial\\|\\bfy^*_+\\|_0$. \nAs $A_{\\Gamma_*}$ is full row rank, the first condition in \\eqref{lemma-1st-order-nec-1} and  $A^\\top\\bfz^*  =A_{\\Gamma_*}^\\top\\bfz^*_{\\Gamma_*}$ derive  that\n$$\\bfz^*_{\\Gamma_*}=\\bfp^*.$$\nNow, $0<\\tau<\\tau_*= \\min\\{\\tau_1,\\tau_2\\}$ in \\eqref{tau-*} results in \n\\begin{eqnarray}\\label{non-negative}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{lllllllllll}\n y_i^*  & \\geq & \\underset{i:y^*_i>0}{\\min} y^*_i &=&  \\sqrt{2\\tau_1\\lambda}&\\geq&\\sqrt{2\\tau_*\\lambda} &>&\\sqrt{2\\tau \\lambda}  &~~\\text{if}~~& y_i^*>0,\\\\ \n   z_i^* & \\leq &  {\\underset{i}{\\max}|p^*_i|} &= &\\sqrt{2\\lambda/\\tau_2}&\\leq&\\sqrt{2\\lambda/\\tau_*}&< &\\sqrt{2\\lambda/\\tau}& ~~ \\text{if}~~ &z_i^*>0.\n\\end{array}\n\\end{eqnarray}\nThese and \\eqref{fact-11}  yield the following condition,\n\\begin{eqnarray*} \ny_i^*\\begin{cases}\n=0, &   z_i^*=0 ~{\\rm or}~ 0< z_i^* < \\sqrt{2\\lambda/\\tau},\\\\\n<0 ~{\\rm or}~>\\sqrt{2\\tau\\lambda },& z_i^*=0. \n\\end{cases}\n\\end{eqnarray*}\n  {It is easy to see that the above condition satisfies that \n\\begin{eqnarray}\\label{ugTT-1}\ny_i^*  \\in \\P_i=\n\\begin{cases}\n0, & y_i^*+\\tau z_i^*\\in(0, \\sqrt{2\\tau\\lambda}),\\\\\n0~{\\rm or}~ y_i^*+\\tau z_i^*, &y_i^*+\\tau z_i^*\\in\\{0, \\sqrt{2\\tau\\lambda}\\},\\\\\ny_i^*+\\tau z_i^*,& y_i^*+\\tau z_i^*\\in(-\\infty,0)\\cup ( \\sqrt{2\\tau\\lambda},\\infty).\n\\end{cases}\n\\end{eqnarray}}\n\n\n\nii) Note that the second condition in \\eqref{p-stationary} means $\\bfy^*\\in\\P$, which by   \\eqref{ugTT-1} implies $z_i^*=0$ if $y_i^*\\neq 0$ and  $z_i^*\\geq0$ if $y_i^*= 0$, resulting in $\\bfz^*\\in\\partial\\|\\bfy^*_+\\|_0$ by \\eqref{subfi}. Consequently, we obtain \\eqref{lemma-1st-order-nec-1} and \\eqref{lemma-1st-order-nec}.\n% In other words, a  P-stationary point satisfies \\eqref{lemma-1st-order-nec} and thus is a local minimizer  if  $f$ is locally convex around $\\bfx^*$ by\nThe claim follows from  \\cref{first-order-nec} ii).\n\n\niii) Let $(\\bfx^*;\\bfz^*)$ be a  P-stationary point with $ \\tau\\geq\\|A\\|^2/c_f$. Then the second condition in \\eqref{p-stationary} indicates that\n\\beq\n&&({1}/{2\\tau})\\|A\\bfx^*+\\bfb-(A\\bfx^*+\\bfb+\\tau \\bfz^*)\\|^2+\\lambda\\|(A\\bfx^*+\\bfb)_+\\|_0 \\nonumber \\\\\n&\\leq& ({1}/{2\\tau})\\|A\\bfx+\\bfb-(A\\bfx^*+\\bfb+\\tau \\bfz^*)\\|^2+\\lambda\\|(A\\bfx+\\bfb)_+\\|_0\\nonumber\n\\eeq\nfor any $\\bfx\\in\\R^n$, which after simplifying leads to \n\\beq\\label{Ax*b-Axb}\n&& \\lambda\\|(A\\bfx^*+\\bfb)_+\\|_0 - \\|A(\\bfx-\\bfx^*)\\|^2 /(2\\tau) \\nonumber\\\\\n&\\leq& \\lambda\\|(A\\bfx+\\bfb)_+\\|_0 - \\langle  \\bfz^*, A(\\bfx-\\bfx^*) \\rangle\\nonumber\\\\\n&=& \\lambda\\|(A\\bfx+\\bfb)_+\\|_0 - \\langle  A^\\top \\bfz^*, \\bfx-\\bfx^*    \\rangle\\nonumber\\\\\n&=& \\lambda\\|(A\\bfx+\\bfb)_+\\|_0 + \\langle  \\nabla f(\\bfx^*), \\bfx-\\bfx^*  \\rangle. \\hspace{.5cm} ({\\rm by}~  \\eqref{p-stationary}) \n\\eeq\nThe strong convexity of $f$ implies\n\\begin{eqnarray*} \n&&f(\\bfx)+\\lambda\\|(A\\bfx+\\bfb)_+\\|_0-f(\\bfx^*)-\\lambda\\|(A\\bfx^*+\\bfb)_+\\|_0\\\\\n&\\geq&({c_f}/{2})\\|\\bfx-\\bfx^*\\|^2+\\langle  \\nabla f(\\bfx^*), \\bfx-\\bfx^*   \\rangle+\\lambda\\|(A\\bfx+\\bfb)_+\\|_0-\\lambda\\|(A\\bfx^*+\\bfb)_+\\|_0\\\\\n&\\geq&({c_f}/{2})\\|\\bfx-\\bfx^*\\|^2-1/({2\\tau})\\|A(\\bfx-\\bfx^*)\\|^2    \\hspace{.25cm} ({\\rm by}~  \\eqref{Ax*b-Axb})\\\\\n&\\geq&({c_f/2-\\|A\\|^2/(2\\tau}))\\|\\bfx-\\bfx^*\\|^2   \\geq 0. \\hspace{1cm} ({\\rm by}~  \\tau\\geq\\|A\\|^2/c_f) \n\\end{eqnarray*}\nThis shows the global optimality of $\\bfx^*$ to the problem \\eqref{ell-0-1}.\n\\end{proof}","sha256_utf8":"dbbc7c1173ec0b579331a44018ed4a9beac0601af56213e8648a0be8455433af"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"取得乘子并辨认阈值向量","source_units":[{"start":32848,"end":33881,"start_line":570,"end_line":580,"raw_tex":"\\begin{proof} i) As  $\\bfx^*$ is a local minimizer of \\eqref{ell-0-1},  condition  \\eqref{lemma-1st-order-nec} is valid by \\cref{first-order-nec} if $A_{\\Gamma_*}$ is  full row rank. In other words, there is a $\\bfz^*$ such that\n\\begin{eqnarray}\n\\label{lemma-1st-order-nec-1}\n\\nabla f(\\bfx^*) +A^\\top\\bfz^*=0,~~\nA \\bfx^*+\\bfb-\\bfy^*=0,~~\n\\bfz^*\\in\\partial\\|\\bfy^*_+\\|_0. \n\\end{eqnarray} \nTherefore, to show \\eqref{p-stationary}, we only need to verify that,  for any $0<\\tau<\\tau^*$, $$\\bfz^*\\in \\partial\\|\\bfy^*_+\\|_0~ \\Rightarrow~\\bfy^*\\in\\P :=\\pro (\\bfy^*+\\tau \\bfz^*). $$ \nRecall the definition of $\\Gamma_*$ in \\eqref{T-*} and the second condition in \\eqref{lemma-1st-order-nec-1}, we have $\\bfy_{\\Gamma_*}=(A \\bfx^*+\\bfb)_{\\Gamma_*}=0$. Same reasoning also allows for obtaining \\eqref{fact-11} due to $\\bfz^*\\in \\partial\\|\\bfy^*_+\\|_0$. \nAs $A_{\\Gamma_*}$ is full row rank, the first condition in \\eqref{lemma-1st-order-nec-1} and  $A^\\top\\bfz^*  =A_{\\Gamma_*}^\\top\\bfz^*_{\\Gamma_*}$ derive  that\n$$\\bfz^*_{\\Gamma_*}=\\bfp^*.$$\n","sha256_utf8":"c93ae4d38b4cf7c47ff9a2bca7ac0f73d54b140ec160868d341f6b99c30ddb47"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"阈值分离与近端包含关系","source_units":[{"start":33881,"end":34997,"start_line":581,"end_line":604,"raw_tex":"Now, $0<\\tau<\\tau_*= \\min\\{\\tau_1,\\tau_2\\}$ in \\eqref{tau-*} results in \n\\begin{eqnarray}\\label{non-negative}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{lllllllllll}\n y_i^*  & \\geq & \\underset{i:y^*_i>0}{\\min} y^*_i &=&  \\sqrt{2\\tau_1\\lambda}&\\geq&\\sqrt{2\\tau_*\\lambda} &>&\\sqrt{2\\tau \\lambda}  &~~\\text{if}~~& y_i^*>0,\\\\ \n   z_i^* & \\leq &  {\\underset{i}{\\max}|p^*_i|} &= &\\sqrt{2\\lambda/\\tau_2}&\\leq&\\sqrt{2\\lambda/\\tau_*}&< &\\sqrt{2\\lambda/\\tau}& ~~ \\text{if}~~ &z_i^*>0.\n\\end{array}\n\\end{eqnarray}\nThese and \\eqref{fact-11}  yield the following condition,\n\\begin{eqnarray*} \ny_i^*\\begin{cases}\n=0, &   z_i^*=0 ~{\\rm or}~ 0< z_i^* < \\sqrt{2\\lambda/\\tau},\\\\\n<0 ~{\\rm or}~>\\sqrt{2\\tau\\lambda },& z_i^*=0. \n\\end{cases}\n\\end{eqnarray*}\n  {It is easy to see that the above condition satisfies that \n\\begin{eqnarray}\\label{ugTT-1}\ny_i^*  \\in \\P_i=\n\\begin{cases}\n0, & y_i^*+\\tau z_i^*\\in(0, \\sqrt{2\\tau\\lambda}),\\\\\n0~{\\rm or}~ y_i^*+\\tau z_i^*, &y_i^*+\\tau z_i^*\\in\\{0, \\sqrt{2\\tau\\lambda}\\},\\\\\ny_i^*+\\tau z_i^*,& y_i^*+\\tau z_i^*\\in(-\\infty,0)\\cup ( \\sqrt{2\\tau\\lambda},\\infty).\n\\end{cases}\n\\end{eqnarray}}\n","sha256_utf8":"7dfa59f919a019705891c1d0369cba4c52d8569234a12343ccc150d08795ee92"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"由近端平稳性取得局部最优性","source_units":[{"start":35000,"end":35532,"start_line":608,"end_line":610,"raw_tex":"ii) Note that the second condition in \\eqref{p-stationary} means $\\bfy^*\\in\\P$, which by   \\eqref{ugTT-1} implies $z_i^*=0$ if $y_i^*\\neq 0$ and  $z_i^*\\geq0$ if $y_i^*= 0$, resulting in $\\bfz^*\\in\\partial\\|\\bfy^*_+\\|_0$ by \\eqref{subfi}. Consequently, we obtain \\eqref{lemma-1st-order-nec-1} and \\eqref{lemma-1st-order-nec}.\n% In other words, a  P-stationary point satisfies \\eqref{lemma-1st-order-nec} and thus is a local minimizer  if  $f$ is locally convex around $\\bfx^*$ by\nThe claim follows from  \\cref{first-order-nec} ii).\n","sha256_utf8":"506d8e9d41763f29ef73e42e253611531576b995afcb715549a9f74c93182be5"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"由近端极小性比较损失","source_units":[{"start":35534,"end":36367,"start_line":613,"end_line":624,"raw_tex":"iii) Let $(\\bfx^*;\\bfz^*)$ be a  P-stationary point with $ \\tau\\geq\\|A\\|^2/c_f$. Then the second condition in \\eqref{p-stationary} indicates that\n\\beq\n&&({1}/{2\\tau})\\|A\\bfx^*+\\bfb-(A\\bfx^*+\\bfb+\\tau \\bfz^*)\\|^2+\\lambda\\|(A\\bfx^*+\\bfb)_+\\|_0 \\nonumber \\\\\n&\\leq& ({1}/{2\\tau})\\|A\\bfx+\\bfb-(A\\bfx^*+\\bfb+\\tau \\bfz^*)\\|^2+\\lambda\\|(A\\bfx+\\bfb)_+\\|_0\\nonumber\n\\eeq\nfor any $\\bfx\\in\\R^n$, which after simplifying leads to \n\\beq\\label{Ax*b-Axb}\n&& \\lambda\\|(A\\bfx^*+\\bfb)_+\\|_0 - \\|A(\\bfx-\\bfx^*)\\|^2 /(2\\tau) \\nonumber\\\\\n&\\leq& \\lambda\\|(A\\bfx+\\bfb)_+\\|_0 - \\langle  \\bfz^*, A(\\bfx-\\bfx^*) \\rangle\\nonumber\\\\\n&=& \\lambda\\|(A\\bfx+\\bfb)_+\\|_0 - \\langle  A^\\top \\bfz^*, \\bfx-\\bfx^*    \\rangle\\nonumber\\\\\n&=& \\lambda\\|(A\\bfx+\\bfb)_+\\|_0 + \\langle  \\nabla f(\\bfx^*), \\bfx-\\bfx^*  \\rangle. \\hspace{.5cm} ({\\rm by}~  \\eqref{p-stationary}) \n\\eeq\n","sha256_utf8":"519d8abd13fceb4f2c845e214ae2c2b9af443f5432eac6780d6251696a4bd5e8"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"结合强凸性取得全局最优性","source_units":[{"start":36367,"end":36973,"start_line":625,"end_line":633,"raw_tex":"The strong convexity of $f$ implies\n\\begin{eqnarray*} \n&&f(\\bfx)+\\lambda\\|(A\\bfx+\\bfb)_+\\|_0-f(\\bfx^*)-\\lambda\\|(A\\bfx^*+\\bfb)_+\\|_0\\\\\n&\\geq&({c_f}/{2})\\|\\bfx-\\bfx^*\\|^2+\\langle  \\nabla f(\\bfx^*), \\bfx-\\bfx^*   \\rangle+\\lambda\\|(A\\bfx+\\bfb)_+\\|_0-\\lambda\\|(A\\bfx^*+\\bfb)_+\\|_0\\\\\n&\\geq&({c_f}/{2})\\|\\bfx-\\bfx^*\\|^2-1/({2\\tau})\\|A(\\bfx-\\bfx^*)\\|^2    \\hspace{.25cm} ({\\rm by}~  \\eqref{Ax*b-Axb})\\\\\n&\\geq&({c_f/2-\\|A\\|^2/(2\\tau}))\\|\\bfx-\\bfx^*\\|^2   \\geq 0. \\hspace{1cm} ({\\rm by}~  \\tau\\geq\\|A\\|^2/c_f) \n\\end{eqnarray*}\nThis shows the global optimality of $\\bfx^*$ to the problem \\eqref{ell-0-1}.\n\\end{proof}","sha256_utf8":"7546a3e97ecfe7f0463e74ca502c04e692f4056654e4d1cda416373a3a370d93"}]},{"id":"S037","semantic_type":"theorem","kind":"theorem","title":"近端平稳方程等价性","source_units":[{"start":39816,"end":40461,"start_line":696,"end_line":709,"raw_tex":"\\begin{theorem}\\label{cor-p-staionary}\nA point $\\bfw^*=(\\bfx^*;\\bfz^*)$ is a  P-stationary point with $\\tau>0$ of the problem (\\ref{ell-0-1}) if and only if   $F(\\bfw^*; \\Gamma_*)=0$ and $ \\Gamma_*=(\\S_* \\cup \\E^o_*)$, where $\\Gamma_*$ is defined by (\\ref{T-*}).\n%the following equations hold\n%\\begin{eqnarray}\n%\\label{sta-eq-1}\n%F(\\bfw^*; \\Gamma_*)=\\left[\\begin{array}{rrl}\n%\\nabla f(\\bfx^*)  + A_{\\Gamma_*}^\\top \\bfz_{\\Gamma_*}^*  \\\\\n%A_{\\Gamma_*} \\bfx^* + \\bfb_{\\Gamma_*} \\\\\n%\\bfz^*_{\\overline \\Gamma_*} \n%\\end{array}\n%\\right]=0. \n%\\end{eqnarray} \nThe Jacobian  $\\nabla F(\\bfw^*;\\Gamma_*)$ is   non-singular if \\cref{ass} holds.\n\\end{theorem}","sha256_utf8":"55b573a06df65cdb98b1c5eeb6dd1a81b7ce4492152edaef3b1049aad2ab4f0d"}]},{"id":"S038","semantic_type":"proof","kind":"proof","title":"平稳方程定理完整证明","source_units":[{"start":40463,"end":43479,"start_line":711,"end_line":773,"raw_tex":"\\begin{proof} \nThe second claim is obvious due to Assumption \\ref{ass}.\n We only need to prove the first claim. \nWe start with the sufficiency.\n The definitions  in \\eqref{p-stationary-T} show the following relationships,\n\\beq\\label{TSE}\n\\Gamma_*= \\S_* \\cup \\E^o_*,~~~~\\overline{\\Gamma}_*=\\O_*\\cup(\\E_* \\setminus \\E^o_*).\n\\eeq\nWe recall $\\bfy^*=A\\bfx^*+\\bfb$ and $\\P:=\\text{Prox}_{\\tau\\lambda\\|({\\cdot})_{+}\\|_{0}}( \\bfy^*+\\tau \\bfz^*)$.\nIt follows that \n \\allowdisplaybreaks \\begin{eqnarray*} \n \\P_i \n=\n\\begin{cases}\n0, & i\\in \\S,\\\\\n0~{\\rm or}~y^*_i+\\tau z^*_i, &i\\in\\E_*,\\\\\n y_i+\\tau z^*_i,& i\\in \\O_*,\n\\end{cases} \n=\n\\begin{cases}\n0, & i\\in \\S_*\\subseteq \\Gamma_*,\\\\\n0~{\\rm or}~\\tau z^*_i, &i\\in \\E^o_* \\subseteq \\Gamma_*,\\\\\n0~{\\rm or}~ y^*_i, &i\\in(\\E_*\\setminus \\E^o_*) \\subseteq \\overline \\Gamma_*,\\\\\ny^*_i,& i\\in \\O_*\\subseteq \\overline \\Gamma_*,\n\\end{cases} \n\\end{eqnarray*}\nwhere the first equation is by \\eqref{proximal-operator} and second one is by \\eqref{sta-eq-1} and \\eqref{TSE}, which indicates $y^*_i \\in \\P_i$ due to $ \\bfy^*_{\\Gamma_*}=0$ from \\eqref{sta-eq-1}. Moreover,   \nthe first and third equations in \\eqref{sta-eq-1} suffice to $\\nabla f(\\bfx^*)  + A^\\top \\bfz^*  =  0$, showing \\eqref{p-stationary}. Namely,  $ \\bfw^*$ is a  P-stationary point. \n\n{Necessity.}  Let $ \\bfw^*$ be a  P-stationary point satisfying \\eqref{p-stationary} and $T_*:= \\S_* \\cup \\E^o_*$.  Then $\\overline{T}_*=\\O_*\\cup(\\E_* \\setminus \\E^o_*)$. It follows from $y^*_i\\in \\P_i$ and \\eqref{proximal-operator} that\n\\begin{eqnarray*} \ny^*_i\\in \n%\\Big[\\text{Prox}_{\\gamma C\\|({\\cdot})_{+}\\|_{0}}(  \\bfy^*+\\tau \\bfz^*)\\Big]_i \\\\\n\\begin{cases}\n0, & i\\in \\S_*,\\\\\n0~{\\rm or}~ y^*_i+\\tau z^*_i, &i\\in\\E_*,\\\\\ny^*_i+\\tau z^*_i,& i\\in \\O_*,\n\\end{cases} =\n\\begin{cases}\n0, & i\\in \\S_*,\\\\\n0~{\\rm or}~\\tau z^*_i, &i\\in\\E^o_*,\\\\\n0~{\\rm or}~ y^*_i+\\tau z^*_i, &i\\in\\E_*\\setminus \\E^o_*,\\\\\ny^*_i+\\tau z^*_i,& i\\in \\O_*,\n\\end{cases} \n\\end{eqnarray*}\nwhere the equality is by the definition  of $\\E^o_*$, which together with $ y^*_i= 0 , i\\in \\E^o_*$ and $y^*_i\\neq 0, i\\in \\E_*\\setminus \\E^o_*$ suffices to\n\\begin{eqnarray*} \n\\begin{cases}\ny^*_i=0, & i\\in \\S_*\\cup\\E^o_*=T_*,\\\\\ny^*_i = y^*_i+\\tau z^*_i, &i\\in\\E_*\\setminus \\E^o_*,\\\\\ny^*_i= y^*_i+\\tau z^*_i,& i\\in \\O_*,\n\\end{cases} ~~\\Longleftrightarrow~~\n\\begin{cases}\ny^*_i &=0, ~ i\\in T_*,\\\\\n z^*_i&=0, ~i\\in\\overline{T}_*.\n\\end{cases} \n\\end{eqnarray*}\nThis gives rise to the last two conditions $\\bfy^*_{T_*}=0,~\n\\bfz^*_{\\overline T_*}=0$  in \\eqref{sta-eq-1}. \nFurthermore, the first condition in \\eqref{p-stationary} and $\\bfz^*_{\\overline{T}_*}=0$ derive the first condition of \\eqref{sta-eq-1}.\nOverall, we have $F(\\bfw^*; T_*)=0$. \nNow we show $T_*=\\Gamma_*$. By  (\\ref{T-*}) that $\\Gamma_*=\\{i\\in\\N_m: y^*_i=0\\}$, it follows $T_*\\subseteq \\Gamma_*$. Suppose, there is a $j\\in \\Gamma_*$ but $j\\notin T_*$, then\nwe have $y_j^*=z_j^*=0$ and thus $j\\in \\E^o_* \\subseteq T_*$  by \\eqref{p-stationary-T}, a contradiction. Therefore, $T_*=\\Gamma_*$, finishing the proof. \\end{proof}","sha256_utf8":"e984ad07fc9a2f07b745d1ea27644ea04450f460fc763e8aba170ec6cc47364a"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"说明雅可比非奇异性","source_units":[{"start":40463,"end":40607,"start_line":711,"end_line":714,"raw_tex":"\\begin{proof} \nThe second claim is obvious due to Assumption \\ref{ass}.\n We only need to prove the first claim. \nWe start with the sufficiency.\n","sha256_utf8":"5a7048c3d2fb9367e037cb9e5b3716546d9ab6a7fe4165b621f495e936c6f3ea"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"平稳方程推出近端平稳性","source_units":[{"start":40607,"end":41737,"start_line":715,"end_line":738,"raw_tex":" The definitions  in \\eqref{p-stationary-T} show the following relationships,\n\\beq\\label{TSE}\n\\Gamma_*= \\S_* \\cup \\E^o_*,~~~~\\overline{\\Gamma}_*=\\O_*\\cup(\\E_* \\setminus \\E^o_*).\n\\eeq\nWe recall $\\bfy^*=A\\bfx^*+\\bfb$ and $\\P:=\\text{Prox}_{\\tau\\lambda\\|({\\cdot})_{+}\\|_{0}}( \\bfy^*+\\tau \\bfz^*)$.\nIt follows that \n \\allowdisplaybreaks \\begin{eqnarray*} \n \\P_i \n=\n\\begin{cases}\n0, & i\\in \\S,\\\\\n0~{\\rm or}~y^*_i+\\tau z^*_i, &i\\in\\E_*,\\\\\n y_i+\\tau z^*_i,& i\\in \\O_*,\n\\end{cases} \n=\n\\begin{cases}\n0, & i\\in \\S_*\\subseteq \\Gamma_*,\\\\\n0~{\\rm or}~\\tau z^*_i, &i\\in \\E^o_* \\subseteq \\Gamma_*,\\\\\n0~{\\rm or}~ y^*_i, &i\\in(\\E_*\\setminus \\E^o_*) \\subseteq \\overline \\Gamma_*,\\\\\ny^*_i,& i\\in \\O_*\\subseteq \\overline \\Gamma_*,\n\\end{cases} \n\\end{eqnarray*}\nwhere the first equation is by \\eqref{proximal-operator} and second one is by \\eqref{sta-eq-1} and \\eqref{TSE}, which indicates $y^*_i \\in \\P_i$ due to $ \\bfy^*_{\\Gamma_*}=0$ from \\eqref{sta-eq-1}. Moreover,   \nthe first and third equations in \\eqref{sta-eq-1} suffice to $\\nabla f(\\bfx^*)  + A^\\top \\bfz^*  =  0$, showing \\eqref{p-stationary}. Namely,  $ \\bfw^*$ is a  P-stationary point. \n","sha256_utf8":"dd1ccc0149461c71ac807c5fdb7efdddf31ff75abf2cb715dad77df09b62777f"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"近端包含关系给出零坐标结构","source_units":[{"start":41738,"end":42850,"start_line":740,"end_line":767,"raw_tex":"{Necessity.}  Let $ \\bfw^*$ be a  P-stationary point satisfying \\eqref{p-stationary} and $T_*:= \\S_* \\cup \\E^o_*$.  Then $\\overline{T}_*=\\O_*\\cup(\\E_* \\setminus \\E^o_*)$. It follows from $y^*_i\\in \\P_i$ and \\eqref{proximal-operator} that\n\\begin{eqnarray*} \ny^*_i\\in \n%\\Big[\\text{Prox}_{\\gamma C\\|({\\cdot})_{+}\\|_{0}}(  \\bfy^*+\\tau \\bfz^*)\\Big]_i \\\\\n\\begin{cases}\n0, & i\\in \\S_*,\\\\\n0~{\\rm or}~ y^*_i+\\tau z^*_i, &i\\in\\E_*,\\\\\ny^*_i+\\tau z^*_i,& i\\in \\O_*,\n\\end{cases} =\n\\begin{cases}\n0, & i\\in \\S_*,\\\\\n0~{\\rm or}~\\tau z^*_i, &i\\in\\E^o_*,\\\\\n0~{\\rm or}~ y^*_i+\\tau z^*_i, &i\\in\\E_*\\setminus \\E^o_*,\\\\\ny^*_i+\\tau z^*_i,& i\\in \\O_*,\n\\end{cases} \n\\end{eqnarray*}\nwhere the equality is by the definition  of $\\E^o_*$, which together with $ y^*_i= 0 , i\\in \\E^o_*$ and $y^*_i\\neq 0, i\\in \\E_*\\setminus \\E^o_*$ suffices to\n\\begin{eqnarray*} \n\\begin{cases}\ny^*_i=0, & i\\in \\S_*\\cup\\E^o_*=T_*,\\\\\ny^*_i = y^*_i+\\tau z^*_i, &i\\in\\E_*\\setminus \\E^o_*,\\\\\ny^*_i= y^*_i+\\tau z^*_i,& i\\in \\O_*,\n\\end{cases} ~~\\Longleftrightarrow~~\n\\begin{cases}\ny^*_i &=0, ~ i\\in T_*,\\\\\n z^*_i&=0, ~i\\in\\overline{T}_*.\n\\end{cases} \n\\end{eqnarray*}\n","sha256_utf8":"6e006dd8c687716ca1665d478b9e7cfe4be31862b120d2ff0c57b5478c3b7488"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"恢复平稳方程及真实指标集","source_units":[{"start":42850,"end":43479,"start_line":768,"end_line":773,"raw_tex":"This gives rise to the last two conditions $\\bfy^*_{T_*}=0,~\n\\bfz^*_{\\overline T_*}=0$  in \\eqref{sta-eq-1}. \nFurthermore, the first condition in \\eqref{p-stationary} and $\\bfz^*_{\\overline{T}_*}=0$ derive the first condition of \\eqref{sta-eq-1}.\nOverall, we have $F(\\bfw^*; T_*)=0$. \nNow we show $T_*=\\Gamma_*$. By  (\\ref{T-*}) that $\\Gamma_*=\\{i\\in\\N_m: y^*_i=0\\}$, it follows $T_*\\subseteq \\Gamma_*$. Suppose, there is a $j\\in \\Gamma_*$ but $j\\notin T_*$, then\nwe have $y_j^*=z_j^*=0$ and thus $j\\in \\E^o_* \\subseteq T_*$  by \\eqref{p-stationary-T}, a contradiction. Therefore, $T_*=\\Gamma_*$, finishing the proof. \\end{proof}","sha256_utf8":"a9ae1b7c2333c6069c8266de36e4caa3b3db2c132ed1a15474f183cd11bdf0c8"}]},{"id":"S043","semantic_type":"lemma","kind":"lemma","title":"邻域指标的共同零点","source_units":[{"start":49149,"end":49606,"start_line":862,"end_line":867,"raw_tex":"\\begin{lemma}\\label{lemma-neighbour-1}\n\t Let $\\bfw^*$ be a \\ts\\  with $0<\\tau<\\tau_*:=\\min\\{\\tau_1,\\tau_2\\}$ of the problem (\\ref{ell-0-1}),  $\\tau_1,\\tau_2$ and $\\Gamma_*$  be given by (\\ref{tau-*}) and (\\ref{T-*}).  Then there is a $\\delta_1^*>0$ such that, for any $\\bfw\\in N(\\bfw^*,\\delta_1^*)$ with its associated indices $\\S$ and $\\E^o$, it holds \n \\beq\\label{gw*-0} \n F(\\bfw^*;T)=0 ~~~~{\\rm and}~~~T:=(\\S\\cup\\E^o)\\subseteq \\Gamma_*.\n \\eeq\n\\end{lemma}","sha256_utf8":"5189c2cb8bc8753087d8cc4e5a935293dfc020b21d4c5be346e473699bbf365b"}]},{"id":"S044","semantic_type":"proof","kind":"proof","title":"共同零点引理完整证明","source_units":[{"start":49608,"end":55286,"start_line":869,"end_line":970,"raw_tex":"\\begin{proof} i) \\cref{cor-p-staionary} states that the \\ts\\ $\\bfw^*$ of (\\ref{ell-0-1}) satisfies\n\\begin{eqnarray}\n\\label{sta-eq-1-*}\n\\nabla f(\\bfx^*)  + A_{\\Gamma_*} \\bfz^*_{\\Gamma_*}=0,~~\n \\bfy^*_{\\Gamma_*}=0, ~~\n\\bfz^*_{\\overline \\Gamma_*}  =0 \n\\end{eqnarray} \n for $0<\\tau<\\tau_*$, where $\\Gamma_*=\\S_*\\cup\\E^o_*$. Note that $\\E_*\\setminus\\E^o_*\\subseteq \\overline{\\Gamma}_*$ which by \\eqref{sta-eq-1-*} leads to\n\\begin{eqnarray} \\label{sta-eq-1-*-z}\n   \\bfz^*_{\\E_*\\setminus\\E^o_*}=0.\n\\end{eqnarray} \nUsing the same reasoning for proving \\eqref{non-negative},  \nwe can prove for $ 0<\\tau<\\tau_*= \\min\\{\\tau_1,\\tau_2\\}$ in \\eqref{tau-*} that\n\\begin{eqnarray} \\label{yz-tau-lambda} \n y_i^*  >\\sqrt{2\\tau\\lambda}  ~~{\\rm if}~~y_i^*> 0,~~~~~~ \\tau z^*_i <   \\sqrt{2\\tau \\lambda   } ~~{\\rm if}~~z^*_i> 0.\n\\end{eqnarray}\nTherefore,  we have the following facts {\n\\allowdisplaybreaks\n\\begin{eqnarray*}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n\\E^o_*&=& \\{i\\in\\N_m: y_i^*=0, \\tau z_i^* \\in \\{0, \\sqrt{2\\tau\\lambda}\\} \\}&~~(\\text{by} ~\\eqref{p-stationary-T})\\\\\n&=& \\{i\\in\\N_m: y_i^*=0, z_i^*=0 \\}, &~~(\\text{by}~\\eqref{yz-tau-lambda})\\\\\n\\E_*\\setminus\\E^o_*&=& \\{i\\in\\N_m: y_i^*\\neq0, y_i^*+\\tau z_i^* = \\sqrt{2\\tau\\lambda } \\}&~~(\\text{by} ~\\eqref{p-stationary-T})\\\\\n &=& \\{i\\in\\N_m: y_i^* = \\sqrt{2\\tau\\lambda} \\}&~~(\\text{by}~\\eqref{sta-eq-1-*-z})\\\\\n&=&\\emptyset, &~~(\\text{by}~\\eqref{yz-tau-lambda})\n \\end{array}\\end{eqnarray*}}\nThese facts lead to\n\\begin{eqnarray}\\label{E*0}\n\\E_*=\\E^o_*\\cup(\\E_*\\setminus\\E^o_*)=\\E^o_*\n=\\Big\\{i\\in\\N_m: y_i^*= z_i^*=0\\Big\\},\n\\end{eqnarray}\nwhich yields the following relations\n\\begin{eqnarray}\\label{TSETO}\n\\Gamma_*=\\S_*\\cup \\E_*^o=\\S_*\\cup \\E_*,~~~~\\overline \\Gamma_*= \\O_*. \n\\end{eqnarray}   \n {For a sufficiently small $\\delta_1^*$, any $\\bfw\\in N(\\bfw^*,\\delta_1^*)$ satisfies,\n\\allowdisplaybreaks\\begin{eqnarray}\\label{hz-h*z*}\n&&  {| y_i +\\tau z_i - y_i^* -\\tau z^*_i |\\leq c\\delta_1^*},~~\\forall~i\\in\\N_m,   \n\\end{eqnarray} \n {where $c>0$ is a constant relied on $A$ and $\\tau$.} The definitions of $\\S$ and $\\S_*$  in \\eqref{p-stationary-T} mean that, for any $i\\in\\S$ or $i\\in\\S_*$,\n\\begin{eqnarray} \\label{facts-S}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n y_i+\\tau z_i\\in (0, \\sqrt{2\\tau\\lambda})~~ &\\Longleftrightarrow&~~  |y_i+\\tau z_i-\\sqrt{\\tau\\lambda/2} |< \\sqrt{\\tau\\lambda/2},\\\\\ny_i^*+\\tau z_i^*\\in (0, \\sqrt{2\\tau\\lambda})~~ &\\Longleftrightarrow&~~  |y_i^*+\\tau z_i^*-\\sqrt{\\tau\\lambda/2} |< \\sqrt{\\tau\\lambda/2}.\n\\end{array}\n\\end{eqnarray} \nUsing this fact, if $\\S_* \\nsubseteq \\S$, then there is an $i\\in \\S_*$ but $i\\notin\\S$ such that\n\\allowdisplaybreaks\n\\begin{eqnarray*}  \n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n|y_i +\\tau z_i - y_i^* -\\tau z^*_i|\n&=&|y_i+\\tau z_i-\\sqrt{\\tau\\lambda/2}  - y_i^*-\\tau z^*_i+\\sqrt{\\tau\\lambda/2}|\\nonumber\\\\\n&\\geq&   \\sqrt{\\tau\\lambda/2} -|y_i^*+\\tau z^*_i-\\sqrt{\\tau\\lambda/2}| ~~~~ (\\text{by}~i\\notin\\S~\\text{and}~\\eqref{facts-S})\\\\\n  &\\geq&   \\sqrt{\\tau\\lambda/2} -\\max_{i\\in\\S_*} |y_i^*+\\tau z^*_i-\\sqrt{\\tau\\lambda/2}| =:\\delta_s\\\\\n&>&0, ~~~~(\\text{by}~\\eqref{facts-S})\n\\end{array}\n\\end{eqnarray*}  {Since $c\\delta_1^*$} can be smaller than $\\delta_s$, the above fact contradicts with \\eqref{hz-h*z*}.  }Hence, it holds $\\S_* \\subseteq \\S$. Similar reasoning also derives $\\O_* \\subseteq \\O$. These allow us to obtain $$\\E = \\N_m \\setminus (\\S\\cup\\O)\\subseteq \\N_m \\setminus (\\S_*\\cup\\O_*)= \\E_*.$$\nOverall, for any $\\bfw\\in N(\\bfw^*,\\delta_1^*)$, it holds\n\\begin{eqnarray}\\label{SSOOEE}\n\\S_* \\subseteq \\S,~~ \\O_* \\subseteq \\O,~~ \\E\\subseteq \\E_*.\n\\end{eqnarray} \nThe above relations enable us to claim that \n\\begin{eqnarray}\\label{SS*E*}\n(\\S\\setminus \\S_*) \\subseteq \\E_*,~~~~(\\O\\setminus \\O_*) \\subseteq \\E_*.\n\\end{eqnarray}\nNow, we can show\n\\allowdisplaybreaks\n\\begin{eqnarray*} \n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n\\bfy^*_{ \\S_*}&=&0,&~~(\\text{by}~\\S_*\\subseteq \\Gamma_*~\\text{and}~\\eqref{sta-eq-1-*})\\\\\n\\bfy^*_{\\S\\setminus \\S_*} &=&0, &~~(\\text{by}~ \\S\\setminus \\S_*\\subseteq\\E_*~\\text{from}~\\eqref{SS*E*}~\\text{and}~\\eqref{E*0})\\\\\n\\bfy^*_{\\E^o} &=&0,&~~(\\text{by}~ \\E^o\\subseteq\\E\\subseteq\\E_*~\\text{from}~\\eqref{SSOOEE}~\\text{and}~\\eqref{E*0})\\\\\n\\bfz^*_{ \\O_*}  &=&0, &~~(\\text{by}~\\O_*\\subseteq \\overline \\Gamma_*~\\text{and}~\\eqref{sta-eq-1-*})\\\\\n\\bfz^*_{\\O\\setminus \\O_*} &=&0, &~~(\\text{by}~ \\O\\setminus \\O_*\\subseteq\\E_*~\\text{from}~\\eqref{SS*E*}~\\text{and}~\\eqref{E*0})\\\\\n\\bfz^*_{\\E\\setminus\\E^o} &=&0.&~~(\\text{by}~ \\E\\setminus\\E^o\\subseteq\\E\\subseteq\\E_*~\\text{from}~\\eqref{SSOOEE}~\\text{and}~\\eqref{E*0})\n\\end{array}\n\\end{eqnarray*}\nThese conditions combining with\n\\begin{eqnarray*}T&=&\\S\\cup\\E^o=\\S_*\\cup(\\S\\setminus \\S_*)\\cup\\E^o,\\\\\n \\overline T&=&\\O\\cup(\\E\\setminus\\E^o)=\\O_*\\cup(\\O\\setminus \\O_*)\\cup(\\E\\setminus\\E^o),\n \\end{eqnarray*}\nimply $\\bfy^*_{ T} =0$ and $\\bfz^*_{\\overline T}=0$. \nAs a consequence of this and $\\bfz^*_{\\overline \\Gamma_*}=0$ from \\eqref{sta-eq-1-*}, \n\\begin{eqnarray*}\n  0= f(\\bfx^*)+ A_{\\Gamma_*} ^\\top\\bfz^*_{\\Gamma_*}=f(\\bfx^*)+ A^\\top\\bfz^*= f(\\bfx^*)+ A_{T} ^\\top\\bfz^*_{T}.\n \\end{eqnarray*}\nOverall, we verify $F(\\bfw^*;T)=0$,  as desired. \nFinally, we observe that\n \\beq \n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n \\begin{array}{rcll}\nT=\\S\\cup\\E^o&=&\\S_*\\cup(\\S\\setminus \\S_*)\\cup\\E^o\\nonumber\\\\\n&\\subseteq&   \\S_*\\cup\\E_* \\cup\\E^o&(\\text{by}~ \\S\\setminus \\S_*\\subseteq\\E_*~\\text{from}~\\eqref{SS*E*})\\nonumber\\\\\n&\\subseteq&   \\S_*\\cup\\E_* \\cup \\E_*  &(\\text{by}~\\E^o\\subseteq\\E\\subseteq\\E_*~\\text{from}~\\eqref{SSOOEE})\\nonumber\\\\\n&=&   \\Gamma_*.  &(\\text{by}~\\eqref{TSETO})\\nonumber\n\\end{array}\n \\eeq\nThe whole proof is completed.   \\end{proof}","sha256_utf8":"b6239d26a6a4c556eca351175ce8165147aec5ef5bae9ef88132b42ca376488c"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"写出极限点的平稳结构","source_units":[{"start":49608,"end":50115,"start_line":869,"end_line":879,"raw_tex":"\\begin{proof} i) \\cref{cor-p-staionary} states that the \\ts\\ $\\bfw^*$ of (\\ref{ell-0-1}) satisfies\n\\begin{eqnarray}\n\\label{sta-eq-1-*}\n\\nabla f(\\bfx^*)  + A_{\\Gamma_*} \\bfz^*_{\\Gamma_*}=0,~~\n \\bfy^*_{\\Gamma_*}=0, ~~\n\\bfz^*_{\\overline \\Gamma_*}  =0 \n\\end{eqnarray} \n for $0<\\tau<\\tau_*$, where $\\Gamma_*=\\S_*\\cup\\E^o_*$. Note that $\\E_*\\setminus\\E^o_*\\subseteq \\overline{\\Gamma}_*$ which by \\eqref{sta-eq-1-*} leads to\n\\begin{eqnarray} \\label{sta-eq-1-*-z}\n   \\bfz^*_{\\E_*\\setminus\\E^o_*}=0.\n\\end{eqnarray} \n","sha256_utf8":"85d8c2ef0c4aeea3a25214e6fd6d1d76a20ec2eb9853ac3c1b84de9bf914eabd"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"严格阈值分离消除边界歧义","source_units":[{"start":50115,"end":51358,"start_line":880,"end_line":904,"raw_tex":"Using the same reasoning for proving \\eqref{non-negative},  \nwe can prove for $ 0<\\tau<\\tau_*= \\min\\{\\tau_1,\\tau_2\\}$ in \\eqref{tau-*} that\n\\begin{eqnarray} \\label{yz-tau-lambda} \n y_i^*  >\\sqrt{2\\tau\\lambda}  ~~{\\rm if}~~y_i^*> 0,~~~~~~ \\tau z^*_i <   \\sqrt{2\\tau \\lambda   } ~~{\\rm if}~~z^*_i> 0.\n\\end{eqnarray}\nTherefore,  we have the following facts {\n\\allowdisplaybreaks\n\\begin{eqnarray*}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n\\E^o_*&=& \\{i\\in\\N_m: y_i^*=0, \\tau z_i^* \\in \\{0, \\sqrt{2\\tau\\lambda}\\} \\}&~~(\\text{by} ~\\eqref{p-stationary-T})\\\\\n&=& \\{i\\in\\N_m: y_i^*=0, z_i^*=0 \\}, &~~(\\text{by}~\\eqref{yz-tau-lambda})\\\\\n\\E_*\\setminus\\E^o_*&=& \\{i\\in\\N_m: y_i^*\\neq0, y_i^*+\\tau z_i^* = \\sqrt{2\\tau\\lambda } \\}&~~(\\text{by} ~\\eqref{p-stationary-T})\\\\\n &=& \\{i\\in\\N_m: y_i^* = \\sqrt{2\\tau\\lambda} \\}&~~(\\text{by}~\\eqref{sta-eq-1-*-z})\\\\\n&=&\\emptyset, &~~(\\text{by}~\\eqref{yz-tau-lambda})\n \\end{array}\\end{eqnarray*}}\nThese facts lead to\n\\begin{eqnarray}\\label{E*0}\n\\E_*=\\E^o_*\\cup(\\E_*\\setminus\\E^o_*)=\\E^o_*\n=\\Big\\{i\\in\\N_m: y_i^*= z_i^*=0\\Big\\},\n\\end{eqnarray}\nwhich yields the following relations\n\\begin{eqnarray}\\label{TSETO}\n\\Gamma_*=\\S_*\\cup \\E_*^o=\\S_*\\cup \\E_*,~~~~\\overline \\Gamma_*= \\O_*. \n\\end{eqnarray}   \n","sha256_utf8":"29c2ae8e3d1d86d2b25a7a9cd1d509e6b4b516b57de5c4045f2e492e7e2f3dea"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"连续扰动保持内部指标","source_units":[{"start":51358,"end":53065,"start_line":905,"end_line":928,"raw_tex":" {For a sufficiently small $\\delta_1^*$, any $\\bfw\\in N(\\bfw^*,\\delta_1^*)$ satisfies,\n\\allowdisplaybreaks\\begin{eqnarray}\\label{hz-h*z*}\n&&  {| y_i +\\tau z_i - y_i^* -\\tau z^*_i |\\leq c\\delta_1^*},~~\\forall~i\\in\\N_m,   \n\\end{eqnarray} \n {where $c>0$ is a constant relied on $A$ and $\\tau$.} The definitions of $\\S$ and $\\S_*$  in \\eqref{p-stationary-T} mean that, for any $i\\in\\S$ or $i\\in\\S_*$,\n\\begin{eqnarray} \\label{facts-S}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n y_i+\\tau z_i\\in (0, \\sqrt{2\\tau\\lambda})~~ &\\Longleftrightarrow&~~  |y_i+\\tau z_i-\\sqrt{\\tau\\lambda/2} |< \\sqrt{\\tau\\lambda/2},\\\\\ny_i^*+\\tau z_i^*\\in (0, \\sqrt{2\\tau\\lambda})~~ &\\Longleftrightarrow&~~  |y_i^*+\\tau z_i^*-\\sqrt{\\tau\\lambda/2} |< \\sqrt{\\tau\\lambda/2}.\n\\end{array}\n\\end{eqnarray} \nUsing this fact, if $\\S_* \\nsubseteq \\S$, then there is an $i\\in \\S_*$ but $i\\notin\\S$ such that\n\\allowdisplaybreaks\n\\begin{eqnarray*}  \n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n|y_i +\\tau z_i - y_i^* -\\tau z^*_i|\n&=&|y_i+\\tau z_i-\\sqrt{\\tau\\lambda/2}  - y_i^*-\\tau z^*_i+\\sqrt{\\tau\\lambda/2}|\\nonumber\\\\\n&\\geq&   \\sqrt{\\tau\\lambda/2} -|y_i^*+\\tau z^*_i-\\sqrt{\\tau\\lambda/2}| ~~~~ (\\text{by}~i\\notin\\S~\\text{and}~\\eqref{facts-S})\\\\\n  &\\geq&   \\sqrt{\\tau\\lambda/2} -\\max_{i\\in\\S_*} |y_i^*+\\tau z^*_i-\\sqrt{\\tau\\lambda/2}| =:\\delta_s\\\\\n&>&0, ~~~~(\\text{by}~\\eqref{facts-S})\n\\end{array}\n\\end{eqnarray*}  {Since $c\\delta_1^*$} can be smaller than $\\delta_s$, the above fact contradicts with \\eqref{hz-h*z*}.  }Hence, it holds $\\S_* \\subseteq \\S$. Similar reasoning also derives $\\O_* \\subseteq \\O$. These allow us to obtain $$\\E = \\N_m \\setminus (\\S\\cup\\O)\\subseteq \\N_m \\setminus (\\S_*\\cup\\O_*)= \\E_*.$$\n","sha256_utf8":"b8933d9e9973aca1f4fdb6d35ce06fa0204ee7b7b0758500a2fb123e8c38eb34"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"指标变化限于边界集合","source_units":[{"start":53065,"end":53393,"start_line":929,"end_line":936,"raw_tex":"Overall, for any $\\bfw\\in N(\\bfw^*,\\delta_1^*)$, it holds\n\\begin{eqnarray}\\label{SSOOEE}\n\\S_* \\subseteq \\S,~~ \\O_* \\subseteq \\O,~~ \\E\\subseteq \\E_*.\n\\end{eqnarray} \nThe above relations enable us to claim that \n\\begin{eqnarray}\\label{SS*E*}\n(\\S\\setminus \\S_*) \\subseteq \\E_*,~~~~(\\O\\setminus \\O_*) \\subseteq \\E_*.\n\\end{eqnarray}\n","sha256_utf8":"c5ca5875b64e76880787a2e1cdf294c0be65342c4534bbc6218d316e3d772afc"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"验证所选指标的零坐标","source_units":[{"start":53393,"end":54500,"start_line":937,"end_line":954,"raw_tex":"Now, we can show\n\\allowdisplaybreaks\n\\begin{eqnarray*} \n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n\\bfy^*_{ \\S_*}&=&0,&~~(\\text{by}~\\S_*\\subseteq \\Gamma_*~\\text{and}~\\eqref{sta-eq-1-*})\\\\\n\\bfy^*_{\\S\\setminus \\S_*} &=&0, &~~(\\text{by}~ \\S\\setminus \\S_*\\subseteq\\E_*~\\text{from}~\\eqref{SS*E*}~\\text{and}~\\eqref{E*0})\\\\\n\\bfy^*_{\\E^o} &=&0,&~~(\\text{by}~ \\E^o\\subseteq\\E\\subseteq\\E_*~\\text{from}~\\eqref{SSOOEE}~\\text{and}~\\eqref{E*0})\\\\\n\\bfz^*_{ \\O_*}  &=&0, &~~(\\text{by}~\\O_*\\subseteq \\overline \\Gamma_*~\\text{and}~\\eqref{sta-eq-1-*})\\\\\n\\bfz^*_{\\O\\setminus \\O_*} &=&0, &~~(\\text{by}~ \\O\\setminus \\O_*\\subseteq\\E_*~\\text{from}~\\eqref{SS*E*}~\\text{and}~\\eqref{E*0})\\\\\n\\bfz^*_{\\E\\setminus\\E^o} &=&0.&~~(\\text{by}~ \\E\\setminus\\E^o\\subseteq\\E\\subseteq\\E_*~\\text{from}~\\eqref{SSOOEE}~\\text{and}~\\eqref{E*0})\n\\end{array}\n\\end{eqnarray*}\nThese conditions combining with\n\\begin{eqnarray*}T&=&\\S\\cup\\E^o=\\S_*\\cup(\\S\\setminus \\S_*)\\cup\\E^o,\\\\\n \\overline T&=&\\O\\cup(\\E\\setminus\\E^o)=\\O_*\\cup(\\O\\setminus \\O_*)\\cup(\\E\\setminus\\E^o),\n \\end{eqnarray*}\nimply $\\bfy^*_{ T} =0$ and $\\bfz^*_{\\overline T}=0$. \n","sha256_utf8":"62615f46ed1aa26d7e7ffb9246a2f2b745747d7dff2eca4b8529705f89d40913"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"验证共同平稳方程","source_units":[{"start":54500,"end":54783,"start_line":955,"end_line":959,"raw_tex":"As a consequence of this and $\\bfz^*_{\\overline \\Gamma_*}=0$ from \\eqref{sta-eq-1-*}, \n\\begin{eqnarray*}\n  0= f(\\bfx^*)+ A_{\\Gamma_*} ^\\top\\bfz^*_{\\Gamma_*}=f(\\bfx^*)+ A^\\top\\bfz^*= f(\\bfx^*)+ A_{T} ^\\top\\bfz^*_{T}.\n \\end{eqnarray*}\nOverall, we verify $F(\\bfw^*;T)=0$,  as desired. \n","sha256_utf8":"7d5960515650bdd3747f547c81866659f0ea8941eaf292f837f92ff3215e962a"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"证明所选指标属于真实指标集","source_units":[{"start":54783,"end":55286,"start_line":960,"end_line":970,"raw_tex":"Finally, we observe that\n \\beq \n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n \\begin{array}{rcll}\nT=\\S\\cup\\E^o&=&\\S_*\\cup(\\S\\setminus \\S_*)\\cup\\E^o\\nonumber\\\\\n&\\subseteq&   \\S_*\\cup\\E_* \\cup\\E^o&(\\text{by}~ \\S\\setminus \\S_*\\subseteq\\E_*~\\text{from}~\\eqref{SS*E*})\\nonumber\\\\\n&\\subseteq&   \\S_*\\cup\\E_* \\cup \\E_*  &(\\text{by}~\\E^o\\subseteq\\E\\subseteq\\E_*~\\text{from}~\\eqref{SSOOEE})\\nonumber\\\\\n&=&   \\Gamma_*.  &(\\text{by}~\\eqref{TSETO})\\nonumber\n\\end{array}\n \\eeq\nThe whole proof is completed.   \\end{proof}","sha256_utf8":"767876ea118392a65a34f8ada3504da00242c0ac8aac928aebb08413d9e90a94"}]},{"id":"S052","semantic_type":"lemma","kind":"lemma","title":"扰动雅可比的一致界","source_units":[{"start":55445,"end":56270,"start_line":976,"end_line":990,"raw_tex":"\\begin{lemma}\\label{lemma-neighbour-2} Let $\\bfw^*$ be a \\ts\\  with $0<\\tau<\\tau_*$ of the problem (\\ref{ell-0-1}) and $\\tau_*$ be given by (\\ref{tau-*}).   Assume  \\cref{ass} and \\cref{ass2}. It holds \n \\beq \\label{bd-jacobian-F-w-T}\n  && C_*  \\geq \\|\\nabla F_{\\mu}(\\bfw;T)\\|\\geq\\sigma_{\\min}( \\nabla F_{\\mu}(\\bfw;T)) \\geq   c_*>0, \n\\eeq\nfor any $\\bfw\\in N(\\bfw^*,\\delta_2^*)$ and any $0\\leq\\mu\\leq c_*/2$, where $T=\\S\\cup\\E^o$ and\n \\beq\\label{radius-2}\nC_*&:=&2\\max\\{1, \\|H(\\Gamma_*)\\|\\} ,~~~~ c_* := 0.5{\\min\\{1,\\min_{\\Gamma\\subseteq\\Gamma_*}\\sigma_{\\min}(H(\\Gamma))\\}}, \\\\\n\\label{H*}\\delta_2^* &:=& \\min\\left\\{\\delta_1^*,   \\frac{c_*}{ 2L_*}     \\right\\},~~~~H(\\Gamma):=\\left[  \n     \\begin{array}{ccc}\n    \\nabla^2 f(\\bfx^*)& A^\\top_{\\Gamma}\\\\\n     A_{\\Gamma}&0\n     \\end{array}\n     \\right]. \n\\end{eqnarray}\n\\end{lemma}","sha256_utf8":"5e972dd9811e1a3b24dede66de05313d51240a9ccc9a9f517ae60baab21244af"}]},{"id":"S053","semantic_type":"proof","kind":"proof","title":"一致非奇异性引理完整证明","source_units":[{"start":56272,"end":60187,"start_line":992,"end_line":1068,"raw_tex":"\\begin{proof}  \nSince  $\\nabla^2 f(\\bfx^*)$ is  {positive definite} and $A_{\\Gamma_*}$ is full row rank by \\cref{ass}, $A_\\Gamma$ is full row rank for any $ \\Gamma \\subseteq   \\Gamma_*$ and thus $H(\\Gamma)$ is non-singular. \nWe have $\\sigma_{\\min}(H(\\Gamma)) > 0$ for any \n$ \\Gamma \\subseteq   \\Gamma_*$ \nand  $c_*>0$. \nNow we build the bounds of $\\nabla F_{\\mu}(\\bfw;T)$. \nFor any given two matrices $D'$ and $D$, we have the first fact\n\\begin{eqnarray} \\label{lower-bd-HH}\n \\|D'-D\\|\n&\\geq& \\max_{i}|\\sigma_i(D')-\\sigma_i(D )|   \\geq |\\sigma_{i_0}( D')-\\sigma_{i_0}(D)|\\nonumber\\\\\n&\\geq& \\sigma_{i_0}(D')-\\sigma_{\\min}(D) \\geq \\sigma_{\\min}(D')-\\sigma_{\\min}(D), \n\\end{eqnarray}\nwhere the first inequality is from \\cite[Reminder (2), on Page 76]{lutkepohl1996handbook} and  $i_0$ satisfies that $\\sigma_{i_0}(D)=\\sigma_{\\min}(D)$.  Recall that\n \\begin{eqnarray*}\n \\label{FH*-FG}H(\\Gamma)\\overset{\\eqref{H*}}{=}\\left[  \n     \\begin{array}{ccc}\n    \\nabla^2 f(\\bfx^*)& A^\\top_{\\Gamma }\\\\\n     A_{\\Gamma }&0\n     \\end{array}\n     \\right],~~~~ \\nabla F(\\bfw^*;T) =\\left[  \n     \\begin{array}{cc}\n    H(T)&0\\\\\n     0&I  \n     \\end{array}\n     \\right].\n    \\end{eqnarray*}\nFor any $\\bfw\\in N(\\bfw^*,\\delta_2^*)$ and $\\delta_2^*\\leq\\delta_1^*$,   \\cref{lemma-neighbour-1} contributes to $T \\subseteq   \\Gamma_*$. So $H(T)$ is a submatrix of $H(\\Gamma_*)$. Hence, %It follows from \\cite[Theorem 1]{thompson1972principal} that the maximum  singular value of a  matrix is no less than the maximum singular value of its sub-matrix. Namely, \n\\beq \n\\|H(T)\\| \\leq \\|H(\\Gamma_*)\\|,~~ \\sigma_{\\min}(H(T)) \\geq \\min_{\\Gamma \\subseteq \\Gamma_*} \\sigma_{\\min} (H(\\Gamma)),\\nonumber\n\\eeq\nwhere the latter is by $T \\subseteq   \\Gamma_*$, which gives us the second fact \n\\beq\\label{JJogG0}\n\\sigma_{\\min}(\\nabla F(\\bfw^*;T))&=&\\min\\{1,\\sigma_{\\min}(H(T))\\}\\\\\n& \\geq&\\min\\{1, \\min_{\\Gamma \\subseteq \\Gamma_*} \\sigma_{\\min} (H(\\Gamma))\\}=2c_*, \\nonumber\\\\\n\\label{JJogG1}\n\\|\\nabla F(\\bfw^*;T)\\| &=& \\max\\{1, \\|H(T)\\|\\} \\leq \\max\\{1, \\|H(\\Gamma_*)\\|\\} =C_*/2. \n\\eeq\nThe locally Lipschitz continuity of $\\nabla^2 f$  around $\\bfx^*$ with $L_*$  yields the third fact,\n\\begin{eqnarray}\n\\label{HH*2}\n\\|\\nabla F (\\bfw^*;T)-\\nabla F (\\bfw;T)\\|&=&\\|\\nabla^2  f (\\bfx)-\\nabla^2  f (\\bfx^*)\\|  \\leq\n   L_* \\| \\bfx -\\bfx^* \\| \\nonumber\\\\\n   &\\leq& L_* \\| \\bfw -\\bfw^* \\| \n \\leq   L_* \\delta_2^*\n\\leq c_*/2.   \\hspace{.5cm}(\\text{by}~\\eqref{H*})\n\\end{eqnarray} \nNow these three facts allow us to derive\n\\allowdisplaybreaks \n\\begin{eqnarray}\\label{lower-bd-FF}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rclr}\n&& \\sigma_{\\min}(\\nabla F_{\\mu}(\\bfw;T))\\\\\n &\\geq& \\sigma_{\\min}(\\nabla F (\\bfw ;T))- \\|\\nabla F(\\bfw;T)-\\nabla F_{\\mu}  (\\bfw;T)\\| &(\\text{by}~\\eqref{lower-bd-HH})  \\\\\n&=& \\sigma_{\\min}(\\nabla F(\\bfw ;T))- \\mu &(\\text{by}~\\eqref{sta-eq-1}) \\\\\n&\\geq& \\sigma_{\\min}(\\nabla F(\\bfw^*;T))- \\|\\nabla F (\\bfw^*;T)-\\nabla F (\\bfw;T)\\|- \\mu   &(\\text{by}~\\eqref{lower-bd-HH}) \\\\\n&\\geq&\\sigma_{\\min}(\\nabla F(\\bfw^*;T))- c_*/2 - \\mu&(\\text{by}~\\eqref{HH*2}) \\\\\n&\\geq&\\sigma_{\\min}(\\nabla F(\\bfw^*;T))-  c_* &(\\text{by}~\\mu\\leq c_*/2)   \\\\\n&\\geq&c_*. &(\\text{by}~\\eqref{JJogG0})  \n\\end{array}\n\\end{eqnarray}\nSimilarly, we also have\n\\allowdisplaybreaks \n\\begin{eqnarray}\\label{upper-bd-FF}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rclr}\n&&\\|\\nabla F_{\\mu}(\\bfw;T)\\|\\\\\n &\\leq& \\|\\nabla F (\\bfw ;T)\\|+ \\|\\nabla F(\\bfw;T)-\\nabla F_{\\mu}  (\\bfw;T)\\| \\\\\n&=& \\|\\nabla F (\\bfw ;T)\\|+ \\mu &(\\text{by}~\\eqref{sta-eq-1})\\\\\n&\\leq& \\|\\nabla F(\\bfw^*;T)\\|+ \\|\\nabla F (\\bfw^*;T)-\\nabla F (\\bfw;T)\\|+ \\mu   &(\\text{by}~\\eqref{lower-bd-HH})  \\\\\n&\\leq& \\|\\nabla F(\\bfw^*;T)\\|+ c_*/2 + \\mu&(\\text{by}~\\eqref{HH*2}) \\\\\n&\\leq& \\|\\nabla F(\\bfw^*;T)\\|+ c_* &(\\text{by}~\\mu\\leq c_*/2)   \\\\\n&\\leq& C_*/2+c_* &(\\text{by}~\\eqref{JJogG1}) \\\\\n&\\leq& C_*. &(\\text{by}~c_*\\leq C_*/2)   \n\\end{array}\n\\end{eqnarray}\nThe whole proof is completed. \n\\end{proof}","sha256_utf8":"721b0283870775bccde6a431b453d92a26f6f8a37f3c896a06aa2b4da9300398"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"子指标鞍点矩阵非奇异","source_units":[{"start":56272,"end":56592,"start_line":992,"end_line":996,"raw_tex":"\\begin{proof}  \nSince  $\\nabla^2 f(\\bfx^*)$ is  {positive definite} and $A_{\\Gamma_*}$ is full row rank by \\cref{ass}, $A_\\Gamma$ is full row rank for any $ \\Gamma \\subseteq   \\Gamma_*$ and thus $H(\\Gamma)$ is non-singular. \nWe have $\\sigma_{\\min}(H(\\Gamma)) > 0$ for any \n$ \\Gamma \\subseteq   \\Gamma_*$ \nand  $c_*>0$. \n","sha256_utf8":"53c9171d8e11f5c3d9eec053ee0a2cc98f4761d6c2f7b8697b3c25e39c6c3d09"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"建立奇异值扰动估计","source_units":[{"start":56592,"end":57117,"start_line":997,"end_line":1004,"raw_tex":"Now we build the bounds of $\\nabla F_{\\mu}(\\bfw;T)$. \nFor any given two matrices $D'$ and $D$, we have the first fact\n\\begin{eqnarray} \\label{lower-bd-HH}\n \\|D'-D\\|\n&\\geq& \\max_{i}|\\sigma_i(D')-\\sigma_i(D )|   \\geq |\\sigma_{i_0}( D')-\\sigma_{i_0}(D)|\\nonumber\\\\\n&\\geq& \\sigma_{i_0}(D')-\\sigma_{\\min}(D) \\geq \\sigma_{\\min}(D')-\\sigma_{\\min}(D), \n\\end{eqnarray}\nwhere the first inequality is from \\cite[Reminder (2), on Page 76]{lutkepohl1996handbook} and  $i_0$ satisfies that $\\sigma_{i_0}(D)=\\sigma_{\\min}(D)$.  Recall that\n","sha256_utf8":"c7de8b3b3ce44dae304c8ca69359810c851bd61366596ef5a25276d2698971b1"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"极限雅可比的一致谱界","source_units":[{"start":57117,"end":58312,"start_line":1005,"end_line":1028,"raw_tex":" \\begin{eqnarray*}\n \\label{FH*-FG}H(\\Gamma)\\overset{\\eqref{H*}}{=}\\left[  \n     \\begin{array}{ccc}\n    \\nabla^2 f(\\bfx^*)& A^\\top_{\\Gamma }\\\\\n     A_{\\Gamma }&0\n     \\end{array}\n     \\right],~~~~ \\nabla F(\\bfw^*;T) =\\left[  \n     \\begin{array}{cc}\n    H(T)&0\\\\\n     0&I  \n     \\end{array}\n     \\right].\n    \\end{eqnarray*}\nFor any $\\bfw\\in N(\\bfw^*,\\delta_2^*)$ and $\\delta_2^*\\leq\\delta_1^*$,   \\cref{lemma-neighbour-1} contributes to $T \\subseteq   \\Gamma_*$. So $H(T)$ is a submatrix of $H(\\Gamma_*)$. Hence, %It follows from \\cite[Theorem 1]{thompson1972principal} that the maximum  singular value of a  matrix is no less than the maximum singular value of its sub-matrix. Namely, \n\\beq \n\\|H(T)\\| \\leq \\|H(\\Gamma_*)\\|,~~ \\sigma_{\\min}(H(T)) \\geq \\min_{\\Gamma \\subseteq \\Gamma_*} \\sigma_{\\min} (H(\\Gamma)),\\nonumber\n\\eeq\nwhere the latter is by $T \\subseteq   \\Gamma_*$, which gives us the second fact \n\\beq\\label{JJogG0}\n\\sigma_{\\min}(\\nabla F(\\bfw^*;T))&=&\\min\\{1,\\sigma_{\\min}(H(T))\\}\\\\\n& \\geq&\\min\\{1, \\min_{\\Gamma \\subseteq \\Gamma_*} \\sigma_{\\min} (H(\\Gamma))\\}=2c_*, \\nonumber\\\\\n\\label{JJogG1}\n\\|\\nabla F(\\bfw^*;T)\\| &=& \\max\\{1, \\|H(T)\\|\\} \\leq \\max\\{1, \\|H(\\Gamma_*)\\|\\} =C_*/2. \n\\eeq\n","sha256_utf8":"0d5fcca1c8bd20fae01804d827016f221221f11131a33e76623081a35a5d193e"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"控制邻域雅可比扰动","source_units":[{"start":58312,"end":58698,"start_line":1029,"end_line":1037,"raw_tex":"The locally Lipschitz continuity of $\\nabla^2 f$  around $\\bfx^*$ with $L_*$  yields the third fact,\n\\begin{eqnarray}\n\\label{HH*2}\n\\|\\nabla F (\\bfw^*;T)-\\nabla F (\\bfw;T)\\|&=&\\|\\nabla^2  f (\\bfx)-\\nabla^2  f (\\bfx^*)\\|  \\leq\n   L_* \\| \\bfx -\\bfx^* \\| \\nonumber\\\\\n   &\\leq& L_* \\| \\bfw -\\bfw^* \\| \n \\leq   L_* \\delta_2^*\n\\leq c_*/2.   \\hspace{.5cm}(\\text{by}~\\eqref{H*})\n\\end{eqnarray} \n","sha256_utf8":"17417d7b495e4caf25a3cfe2306fd00d01cc8695dd6ca6c3da7c21ed41381b61"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"取得最小奇异值下界","source_units":[{"start":58698,"end":59455,"start_line":1038,"end_line":1051,"raw_tex":"Now these three facts allow us to derive\n\\allowdisplaybreaks \n\\begin{eqnarray}\\label{lower-bd-FF}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rclr}\n&& \\sigma_{\\min}(\\nabla F_{\\mu}(\\bfw;T))\\\\\n &\\geq& \\sigma_{\\min}(\\nabla F (\\bfw ;T))- \\|\\nabla F(\\bfw;T)-\\nabla F_{\\mu}  (\\bfw;T)\\| &(\\text{by}~\\eqref{lower-bd-HH})  \\\\\n&=& \\sigma_{\\min}(\\nabla F(\\bfw ;T))- \\mu &(\\text{by}~\\eqref{sta-eq-1}) \\\\\n&\\geq& \\sigma_{\\min}(\\nabla F(\\bfw^*;T))- \\|\\nabla F (\\bfw^*;T)-\\nabla F (\\bfw;T)\\|- \\mu   &(\\text{by}~\\eqref{lower-bd-HH}) \\\\\n&\\geq&\\sigma_{\\min}(\\nabla F(\\bfw^*;T))- c_*/2 - \\mu&(\\text{by}~\\eqref{HH*2}) \\\\\n&\\geq&\\sigma_{\\min}(\\nabla F(\\bfw^*;T))-  c_* &(\\text{by}~\\mu\\leq c_*/2)   \\\\\n&\\geq&c_*. &(\\text{by}~\\eqref{JJogG0})  \n\\end{array}\n\\end{eqnarray}\n","sha256_utf8":"e50ab3ac02a36bcfd57354c5e389011b09e58817de5daf98ec599271223fe831"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"取得算子范数上界","source_units":[{"start":59455,"end":60187,"start_line":1052,"end_line":1068,"raw_tex":"Similarly, we also have\n\\allowdisplaybreaks \n\\begin{eqnarray}\\label{upper-bd-FF}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rclr}\n&&\\|\\nabla F_{\\mu}(\\bfw;T)\\|\\\\\n &\\leq& \\|\\nabla F (\\bfw ;T)\\|+ \\|\\nabla F(\\bfw;T)-\\nabla F_{\\mu}  (\\bfw;T)\\| \\\\\n&=& \\|\\nabla F (\\bfw ;T)\\|+ \\mu &(\\text{by}~\\eqref{sta-eq-1})\\\\\n&\\leq& \\|\\nabla F(\\bfw^*;T)\\|+ \\|\\nabla F (\\bfw^*;T)-\\nabla F (\\bfw;T)\\|+ \\mu   &(\\text{by}~\\eqref{lower-bd-HH})  \\\\\n&\\leq& \\|\\nabla F(\\bfw^*;T)\\|+ c_*/2 + \\mu&(\\text{by}~\\eqref{HH*2}) \\\\\n&\\leq& \\|\\nabla F(\\bfw^*;T)\\|+ c_* &(\\text{by}~\\mu\\leq c_*/2)   \\\\\n&\\leq& C_*/2+c_* &(\\text{by}~\\eqref{JJogG1}) \\\\\n&\\leq& C_*. &(\\text{by}~c_*\\leq C_*/2)   \n\\end{array}\n\\end{eqnarray}\nThe whole proof is completed. \n\\end{proof}","sha256_utf8":"e67650c998f8465aec002216ab08395f565078363b536be3d9d62ab3f9944904"}]},{"id":"S060","semantic_type":"theorem","kind":"theorem","title":"平滑牛顿法二次收敛与停机界","source_units":[{"start":60565,"end":61857,"start_line":1085,"end_line":1106,"raw_tex":"\\begin{theorem}\\label{quadratic-convergence} \nLet $\\bfw^*$ be any {\\ts} with $0<\\tau<\\tau_*$ of  (\\ref{ell-0-1}), $\\tau_*$ and $\\delta_2^*, c_*, C_*$ be given by (\\ref{tau-*}) and (\\ref{radius-2}). Assume  \\cref{ass} and \\cref{ass2}. Let $\\{\\bfw^k\\}$ be the  sequence generated by  \\cref{Alg-NM01} and $0\\leq\\mu_{-1}\\leq c_*/2$. If the initial point  satisfies $\\bfw^0\\in N(\\bfw^*,\\delta_*)$, where\n\\begin{eqnarray}\\label{delta-*}\n\\delta_*:=\\min\\left\\{\\delta_2^*,~{c_*}/{(2 (L_*+2\\rho C_*))}\\right\\},\n\\end{eqnarray}\n then the following results hold.\n\\begin{itemize}\n\\item[a)] The sequence $\\{\\bfd^k\\}_{k\\geq0}$ is well defined and $\\lim_{k\\rightarrow\\infty}\\bfd^k=0$. \n\\item[b)] The whole sequence $\\{\\bfw^k\\}$ converges to $\\bfw^*$ quadratically, namely,\n\\beq\n\\|\\bfw^{k+1}-\\bfw^{*}\\|  &\\leq& \n  ((L_*+2\\rho C_*)/c_*) \\|\\bfw^{k}-\\bfw^{*} \\|^2. \\nonumber \n\\eeq\n\\item[c)] The halting condition satisfies\n\\beq\n\\|F(\\bfw^{k+1};T_{k+1})\\| \n&\\leq& (L_*+2\\rho C_*)(C_*/c_*^{3})  \\|F(\\bfw^{k};T_k)\\|^2  \\nonumber \n\\eeq\nand  \\Cref{Alg-NM01} reaches $\\|F(\\bfw^{k};T_k)\\| <  \\epsilon$ for a given tolerance $\\epsilon>0$ when \\beq\\label{k-tol}\nk\\geq \\Big\\lceil \\log_2\\Big(2 \\sqrt{(L_*+2\\rho C_*)(C_*/c_*)^3  }\\|\\bfw^{0}-\\bfw^{*} \\|\\Big)-\\log_2(\\sqrt{\\epsilon})\\Big\\rceil.\\eeq  \n\\end{itemize}\n\\end{theorem}","sha256_utf8":"177b766cb0e6a2c21c3394e9e2497f6722f235850b637c2f2d351c127f17f8ea"}]},{"id":"S061","semantic_type":"assumption","kind":"assumption","title":"收敛定理的局部初始条件","source_units":[{"start":60611,"end":61081,"start_line":1086,"end_line":1089,"raw_tex":"Let $\\bfw^*$ be any {\\ts} with $0<\\tau<\\tau_*$ of  (\\ref{ell-0-1}), $\\tau_*$ and $\\delta_2^*, c_*, C_*$ be given by (\\ref{tau-*}) and (\\ref{radius-2}). Assume  \\cref{ass} and \\cref{ass2}. Let $\\{\\bfw^k\\}$ be the  sequence generated by  \\cref{Alg-NM01} and $0\\leq\\mu_{-1}\\leq c_*/2$. If the initial point  satisfies $\\bfw^0\\in N(\\bfw^*,\\delta_*)$, where\n\\begin{eqnarray}\\label{delta-*}\n\\delta_*:=\\min\\left\\{\\delta_2^*,~{c_*}/{(2 (L_*+2\\rho C_*))}\\right\\},\n\\end{eqnarray}\n","sha256_utf8":"8084a938ffde8d2ba9f6e342106a1177cbbd8dad7d8339a80028a91404ac3a84"}]},{"id":"S062","semantic_type":"proof","kind":"proof","title":"二次收敛定理完整证明","source_units":[{"start":61858,"end":69522,"start_line":1107,"end_line":1217,"raw_tex":"\\begin{proof} a) It is easily observed   \n\\begin{eqnarray}\\label{mu-k}\n0< \\mu_{k} \\leq \\mu_{k-1}, ~k=1,2,3,\\ldots, ~~~~{\\rm and}~~~~ \\lim_{k\\rightarrow\\infty}\\mu_k=0.\n\\end{eqnarray} \nIt follows from \\cref{lemma-neighbour-1} and the facts\n$\\delta_*\\leq \\delta_2^*\\leq\\delta_1^*$ and $\\bfw^0\\in N(\\bfw^*,\\delta_*)$ that     \n\\beq\\label{F-w*-T0-0}F (\\bfw^{*};T_0) =0  \n\\eeq\nwith $T_0=\\S_0\\cup\\E_0^o$ and from \\cref{lemma-neighbour-2} that     \n\\beq\\label{grad-F-w*-T0-c} C_*  \\geq \\|\\nabla F_{\\mu_0}(\\bfw^0;T_0)\\|\\geq\\sigma_{\\min}( \\nabla F_{\\mu_0}(\\bfw^0;T_0)) \\geq  c_*\n\\eeq\nHere, we used the fact that $\\mu_0\\leq \\mu_{-1}\\leq c_*/2$ by  \\eqref{mu-k}. \nFrom \\eqref{newton-dir-0}, we have\n\\beq\\label{dkt}\n\\nabla F_{\\mu_0} (\\bfw^{0};T_0)~\\bfd^{0}=- F (\\bfw^{0};T_0).\n\\eeq\n  \\cref{lemma-neighbour-2} states that $\\nabla F_{\\mu_0} (\\bfw^{0};T_0)$ is non-singular and thus $\\bfd^{0}$ is well defined.\nLet \n\\beq\\label{w0-beta-w*}\n\\bfw^{0}_\\beta= \\bfw^{*} + \\beta(\\bfw^{0}-\\bfw^{*})=(\\bfx^{0}_\\beta;\\bfz^{0}_\\beta)\n\\eeq\nwhere $\\beta\\in[0,1].$ One can easily check that $\\bfw^{0}_\\beta\\in N(\\bfw^*,\\delta_*)$ as $$\\|\\bfw^{0}_\\beta-\\bfw^{*}\\|= \\beta\\|\\bfw^{0}-\\bfw^{*}\\|\\leq  \\delta_*.$$\nThe definition  in \\eqref{sta-eq-1} enables us to obtain\n\\allowdisplaybreaks \n\\begin{eqnarray*}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n&&\\|\\nabla F_{\\mu_0} (\\bfw^{0};T_0)- \\nabla F (\\bfw^{0}_\\beta;T_0) \\|  \\\\\n&\\leq&\\|\\nabla^2 f(\\bfx^0)- \\nabla^2 f(\\bfx^0_\\beta)\\| + \\mu_0&(\\text{by}~\\eqref{sta-eq-1})  \\\\\n& \\leq& L_* \\|\\bfx^0-\\bfx^0_\\beta\\|+ \\rho  \\|F(\\bfw^0;T_0)\\|&(\\text{by}~\\eqref{update-mu}) \\\\ \n& \\leq& L_* \\|\\bfw^0-\\bfw^0_\\beta\\|+ \\rho \\|\\nabla F_{\\mu_0} (\\bfw^{0};T_0)\\| \\|\\bfd^0\\|&(\\text{by}~\\eqref{newton-dir-0}) \\\\ \n& \\leq& L_* \\|\\bfw^0-\\bfw^0_\\beta\\|+ \\rho C_* \\|\\bfw^1-\\bfw^0\\|&(\\text{by}~\\eqref{grad-F-w*-T0-c}) \\\\ \n& \\leq& L_*(1-\\beta)\\|\\bfw^0-\\bfw^*\\|+ \\rho C_* (\\|\\bfw^1-\\bfw^*\\|+\\|\\bfw^0-\\bfw^*\\|) &(\\text{by}~\\eqref{w0-beta-w*})  \\\\ \n& =& (L_*(1-\\beta)+\\rho C_*)\\|\\bfw^0-\\bfw^*\\|+ \\rho C_* \\|\\bfw^1-\\bfw^*\\|.\n\\end{array} \\end{eqnarray*}\nDenote $\\Theta_{\\mu_0}:=\\nabla F_{\\mu_0} (\\bfw^{0};T_0)$ and $\\Theta(\\beta):= \\nabla F (\\bfw^{0}_\\beta;T_0)$.  The above condition yields\n  \\beq\\label{lipschitz}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n \\int_0^1 \\|\\Theta_{\\mu_0}  - \\Theta(\\beta)\\|d\\beta\n\\leq (L_*/2+\\rho C_*)\\|\\bfw^0-\\bfw^*\\|+ \\rho C_* \\|\\bfw^1-\\bfw^*\\|.\n\\end{array} \\eeq\nFor the fixed $T_0$, the function $F(\\cdot,T_0)$ is differentiable, which by \\eqref{F-w*-T0-0} derives\n\\beq\\label{mean-value}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\nF (\\bfw^{0};T_0)=F (\\bfw^{*};T_0)+\\int_0^1 \\Theta(\\beta)(\\bfw^{0}-\\bfw^{*})d\\beta=\\int_0^1 \\Theta(\\beta) (\\bfw^{0}-\\bfw^{*})d\\beta.\n\\end{array}\\eeq\nNow the following chain of inequalities holds.\n\\begin{eqnarray*}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\nc_*\\|\\bfw^{1}-\\bfw^{*}\\| &=& c_*\\|\\bfw^{0}+\\bfd^{0}-\\bfw^{*}\\|\\\\\n& {=} &c_*\\|\\bfw^{0}-\\bfw^{*}-\\Theta_{\\mu_0}^{-1}   F(\\bfw^{0};T_0) \\|&~~(\\text{by}~\\eqref{dkt})\\\\\n& {\\leq} & \\|\\Theta_{\\mu_0} (\\bfw^{0}-\\bfw^{*})-   F(\\bfw^{0};T_0) \\|&~~(\\text{by}~\\eqref{grad-F-w*-T0-c})\\\\\n&= & \\|\\Theta_{\\mu_0}  (\\bfw^{0}-\\bfw^{*})-  \\int_0^1 \\Theta(\\beta)(\\bfw^{0}-\\bfw^{*})d\\beta \\| &~~(\\text{by}~\\eqref{mean-value}) \\\\\n& \\leq& \\int_0^1 \\|\\Theta_{\\mu_0} -\\Theta(\\beta) \\|\\|\\bfw^{0}-\\bfw^{*}\\|d\\beta \\\\\n& =& ({\\theta_*}/2) \\|\\bfw^{0} -\\bfw^{*}\\|^2+   {\\rho C_*}  \\|\\bfw^1-\\bfw^*\\|\\|\\bfw^{0}-\\bfw^{*}\\| &~~(\\text{by}~\\eqref{lipschitz}) \\\\\n&\\leq& ({\\theta_*}/2) \\|\\bfw^{0} -\\bfw^{*}\\|^2+ {\\rho C_*\\delta_*}  \\|\\bfw^1-\\bfw^*\\|   \\\\\n&\\leq& ({\\theta_*}/2) \\|\\bfw^{0} -\\bfw^{*}\\|^2+ (c_*/2) \\|\\bfw^1-\\bfw^*\\|, \n\\end{array}\n\\end{eqnarray*}\nwhere $\\theta_*:=L_* +2\\rho C_*$ and the last inequality is from $\\delta_*\\leq  {c_*}/(2 \\theta_*)$ by \\eqref{delta-*} and\n\\beq\\label{w0-w*-delta*}\n\\|\\bfw^{0}-\\bfw^{*}\\|<\\delta_*\\leq  {c_*}/(2 \\theta_*)<  {c_*}/({2\\rho  C_*}).\n\\eeq\n The above chain of inequalities  suffices to the following fact\n\\beq\\label{quadratic-chain}\n\\|\\bfw^{1}-\\bfw^{*}\\|\\leq  (\\theta_*/c_*) \\|\\bfw^{0} -\\bfw^{*}\\|^2.\n\\eeq\nThis together with $\\|\\bfw^{0}-\\bfw^{*}\\|<\\delta_*$ and \\eqref{w0-w*-delta*} derives\n\\beq \n\\|\\bfw^{1}-\\bfw^{*}\\|  &\\leq& \n   (\\theta_*/c_*)  \\delta_* \\|\\bfw^{0}-\\bfw^{*} \\|\\leq (1/2)\\|\\bfw^{0}-\\bfw^{*} \\|< \\delta_*,  \\nonumber\n \\eeq\nwhich means $\\bfw^1\\in N(\\bfw^*,\\delta_*)$. In addition, $\\mu_1\\leq\\mu_0\\leq c_*/2$ by  \\eqref{mu-k}. Hence, replacing $T_0 $ by $T_1$, the same reasoning allows us to show that $\\bfd^{1}$ is well defined and $$\\|\\bfw^{2}-\\bfw^{*}\\|  \\leq (\\theta_*/c_*)   \\|\\bfw^{1}-\\bfw^{*} \\|^2 .$$ By the induction, we can conclude that $\\bfw^k\\in N(\\bfw^*,\\delta_*)$, $\\bfd^{k}$ is well defined and \n\\beq\\label{quadratic-rate}\n\\hspace{10mm}\\|\\bfw^{k+1}-\\bfw^{*}\\|  &\\leq& (\\theta_*/c_*) \\|\\bfw^{k}-\\bfw^{*} \\|^2, \\\\\n\\label{quadratic-rate-1}&\\leq& (\\theta_*/c_*)  \\|\\bfw^{k}-\\bfw^{*} \\|\\delta_* \\leq (1/2) \\|\\bfw^{k}-\\bfw^{*} \\|.\\hspace{.2cm}(\\text{by}~\\eqref{w0-w*-delta*})\n\\eeq\nTherefore, \\eqref{quadratic-rate} claims b). The conclusion of a) can be made by \\eqref{quadratic-rate-1}  that   $$\\bfw^{k}\\rightarrow\\bfw^{*},~~ \\bfd^k=\\bfw^{k+1}-\\bfw^{k}=\\bfw^{k+1}-\\bfw^{*}+\\bfw^{*}-\\bfw^{k}\\rightarrow0.$$ \n\nc)  The  above  proof shows $\\bfw^k\\in N(\\bfw^*,\\delta_*)$ and hence \\eqref{gw*-0} results in\n\\beq\\label{wk-in-N}  F(\\bfw^{*};T_k){=}0,\\eeq\nwhere $T_k=\\S_k\\cup\\E_k^o$.\nBy letting $\\bfw^{k}_\\beta= \\bfw^{*} + \\beta(\\bfw^{k}-\\bfw^{*}),$ where $\\beta\\in[0,1]$, we have $\\bfw^k_\\beta\\in N(\\bfw^*,\\delta_*)$. To show \\eqref{bd-jacobian-F-w-T} in \\cref{lemma-neighbour-2}, we verified the lower and upper bounds by \\eqref{lower-bd-FF} and \\eqref{upper-bd-FF}. Similarly, we can prove these bounds hold for $\\nabla F (\\bfw^{k}_\\beta;T_k)$. (In fact, since $\\bfw^k_\\beta\\in N(\\bfw^*,\\delta_*)\\subseteq N(\\bfw^*,\\delta_2^*)$ by $\\delta_*\\leq \\delta_2^*$, one just needs to set $\\mu=0$ and $\\bfw= \\bfw^{k}_\\beta$ in  \\eqref{lower-bd-FF} and \\eqref{upper-bd-FF}. Therefore,\n\\beq\\label{C-F-w-beta-c}\nC_*  \\geq \\|\\nabla F (\\bfw^{k}_\\beta;T_k)\\|\\geq\\sigma_{\\min}( \\nabla F (\\bfw^{k}_\\beta;T_k)) \\geq  c_*, \n\\eeq\nAgain, the function $F(\\cdot,T_k)$ is differentiable for the fixed $T_k$, so the Mean-value theorem states that there is a $\\beta_0\\in(0,1)$ satisfying\n\\beq\\label{FkTk}\n\\|F(\\bfw^{k},T_k)\\|&=&\\|F(\\bfw^{*};T_k)+  \\nabla F (\\bfw^{k}_{\\beta_0};T_k) (\\bfw^{k}-\\bfw^{*})\\|\\nonumber\\\\\n&=&\\| \\nabla F (\\bfw^{k}_{\\beta_0};T_k) (\\bfw^{k}-\\bfw^{*}) \\|\\hspace{1.7cm}(\\text{by}~\\eqref{wk-in-N})\\nonumber\\\\\n&\\in&  [~c_*\\|\\bfw^{k}-\\bfw^{*} \\|, C_*\\|\\bfw^{k}-\\bfw^{*} \\|~], \\hspace{0.8cm}(\\text{by}~\\eqref{C-F-w-beta-c})\n\\eeq\nThis contributes to \n\\allowdisplaybreaks\n\\begin{eqnarray*}\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rclr}\n\\|F(\\bfw^{k};T_{k})\\| \\leq  C_*\\|\\bfw^{k}-\\bfw^{*}\\|  \n & {\\leq} & (\\theta_*C_*/c_*)   \\|\\bfw^{k-1}-\\bfw^{*} \\|^2 &(\\text{by}~\\eqref{quadratic-rate}) \\\\\n & {\\leq}& (\\theta_*C_*/c_*^{3})  \\|F(\\bfw^{k-1};T_{k-1})\\|^2&(\\text{by}~\\eqref{FkTk})\\\\\n &{\\leq}& (\\theta_*C_*^3/c_*^{3})      \\|\\bfw^{k-1}-\\bfw^{*} \\|^2&(\\text{by}~\\eqref{FkTk})\\\\\n& {\\leq}& (\\theta_*C_*^3/c_*^{3})  2^{-2}  \\|\\bfw^{k-2}-\\bfw^{*} \\|^2&~~(\\text{by}~\\eqref{quadratic-rate-1})\\\\\n    &\\vdots&\\\\\n & \\leq& (\\theta_*C_*^3/c_*^{3})   2^{2-2k} \\|\\bfw^{0}-\\bfw^{*} \\|^2,&(\\text{by}~\\eqref{quadratic-rate-1})\n \\end{array}  \\end{eqnarray*}\nwhere the third inequality yields the first conclusion in c). This also enable to verify that $\\|F(\\bfw^{k};T_k)\\|$ $< {\\epsilon}$ if $k$ satisfies \\eqref{k-tol}.\n%$$k\\geq  \\lceil \\log_2 (2 \\sqrt{\\theta_* (C_* /c_*)^3}\\|\\bfw^{0}-\\bfw^{*} \\| )-\\log_2(\\sqrt{\\epsilon}) \\rceil.$$ \nThe whole proof is completed.\n\\end{proof}","sha256_utf8":"2362e9f5969718e16d8585f09010dfd8e6f032c35948cae9b191d85de89e1719"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"平滑参数单调趋零","source_units":[{"start":61858,"end":62041,"start_line":1107,"end_line":1110,"raw_tex":"\\begin{proof} a) It is easily observed   \n\\begin{eqnarray}\\label{mu-k}\n0< \\mu_{k} \\leq \\mu_{k-1}, ~k=1,2,3,\\ldots, ~~~~{\\rm and}~~~~ \\lim_{k\\rightarrow\\infty}\\mu_k=0.\n\\end{eqnarray} \n","sha256_utf8":"cc80bf6d0c643190cc7833c88c829a802d477ca654347c7fbbc0ba4f76910d2f"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"首步共同零点与方向存在性","source_units":[{"start":62041,"end":62753,"start_line":1111,"end_line":1123,"raw_tex":"It follows from \\cref{lemma-neighbour-1} and the facts\n$\\delta_*\\leq \\delta_2^*\\leq\\delta_1^*$ and $\\bfw^0\\in N(\\bfw^*,\\delta_*)$ that     \n\\beq\\label{F-w*-T0-0}F (\\bfw^{*};T_0) =0  \n\\eeq\nwith $T_0=\\S_0\\cup\\E_0^o$ and from \\cref{lemma-neighbour-2} that     \n\\beq\\label{grad-F-w*-T0-c} C_*  \\geq \\|\\nabla F_{\\mu_0}(\\bfw^0;T_0)\\|\\geq\\sigma_{\\min}( \\nabla F_{\\mu_0}(\\bfw^0;T_0)) \\geq  c_*\n\\eeq\nHere, we used the fact that $\\mu_0\\leq \\mu_{-1}\\leq c_*/2$ by  \\eqref{mu-k}. \nFrom \\eqref{newton-dir-0}, we have\n\\beq\\label{dkt}\n\\nabla F_{\\mu_0} (\\bfw^{0};T_0)~\\bfd^{0}=- F (\\bfw^{0};T_0).\n\\eeq\n  \\cref{lemma-neighbour-2} states that $\\nabla F_{\\mu_0} (\\bfw^{0};T_0)$ is non-singular and thus $\\bfd^{0}$ is well defined.\n","sha256_utf8":"3034b705354de6942a58cec1caec516158cf435987c7ffd6941bf66ac0799597"}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","title":"连接初始点与极限点","source_units":[{"start":62753,"end":63036,"start_line":1124,"end_line":1128,"raw_tex":"Let \n\\beq\\label{w0-beta-w*}\n\\bfw^{0}_\\beta= \\bfw^{*} + \\beta(\\bfw^{0}-\\bfw^{*})=(\\bfx^{0}_\\beta;\\bfz^{0}_\\beta)\n\\eeq\nwhere $\\beta\\in[0,1].$ One can easily check that $\\bfw^{0}_\\beta\\in N(\\bfw^*,\\delta_*)$ as $$\\|\\bfw^{0}_\\beta-\\bfw^{*}\\|= \\beta\\|\\bfw^{0}-\\bfw^{*}\\|\\leq  \\delta_*.$$\n","sha256_utf8":"f7f0fc3169cce6f2f702cf89737b44914379e23a69dc6865b01488da7bbd9621"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"积分估计雅可比差异","source_units":[{"start":63036,"end":64273,"start_line":1129,"end_line":1148,"raw_tex":"The definition  in \\eqref{sta-eq-1} enables us to obtain\n\\allowdisplaybreaks \n\\begin{eqnarray*}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n&&\\|\\nabla F_{\\mu_0} (\\bfw^{0};T_0)- \\nabla F (\\bfw^{0}_\\beta;T_0) \\|  \\\\\n&\\leq&\\|\\nabla^2 f(\\bfx^0)- \\nabla^2 f(\\bfx^0_\\beta)\\| + \\mu_0&(\\text{by}~\\eqref{sta-eq-1})  \\\\\n& \\leq& L_* \\|\\bfx^0-\\bfx^0_\\beta\\|+ \\rho  \\|F(\\bfw^0;T_0)\\|&(\\text{by}~\\eqref{update-mu}) \\\\ \n& \\leq& L_* \\|\\bfw^0-\\bfw^0_\\beta\\|+ \\rho \\|\\nabla F_{\\mu_0} (\\bfw^{0};T_0)\\| \\|\\bfd^0\\|&(\\text{by}~\\eqref{newton-dir-0}) \\\\ \n& \\leq& L_* \\|\\bfw^0-\\bfw^0_\\beta\\|+ \\rho C_* \\|\\bfw^1-\\bfw^0\\|&(\\text{by}~\\eqref{grad-F-w*-T0-c}) \\\\ \n& \\leq& L_*(1-\\beta)\\|\\bfw^0-\\bfw^*\\|+ \\rho C_* (\\|\\bfw^1-\\bfw^*\\|+\\|\\bfw^0-\\bfw^*\\|) &(\\text{by}~\\eqref{w0-beta-w*})  \\\\ \n& =& (L_*(1-\\beta)+\\rho C_*)\\|\\bfw^0-\\bfw^*\\|+ \\rho C_* \\|\\bfw^1-\\bfw^*\\|.\n\\end{array} \\end{eqnarray*}\nDenote $\\Theta_{\\mu_0}:=\\nabla F_{\\mu_0} (\\bfw^{0};T_0)$ and $\\Theta(\\beta):= \\nabla F (\\bfw^{0}_\\beta;T_0)$.  The above condition yields\n  \\beq\\label{lipschitz}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\n \\int_0^1 \\|\\Theta_{\\mu_0}  - \\Theta(\\beta)\\|d\\beta\n\\leq (L_*/2+\\rho C_*)\\|\\bfw^0-\\bfw^*\\|+ \\rho C_* \\|\\bfw^1-\\bfw^*\\|.\n\\end{array} \\eeq\n","sha256_utf8":"1b124926b73cbb04177c3e833140d271e139ee1e7d574ab4158c8a4ac5f4bc59"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"固定指标下展开残差","source_units":[{"start":64273,"end":64608,"start_line":1149,"end_line":1154,"raw_tex":"For the fixed $T_0$, the function $F(\\cdot,T_0)$ is differentiable, which by \\eqref{F-w*-T0-0} derives\n\\beq\\label{mean-value}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\nF (\\bfw^{0};T_0)=F (\\bfw^{*};T_0)+\\int_0^1 \\Theta(\\beta)(\\bfw^{0}-\\bfw^{*})d\\beta=\\int_0^1 \\Theta(\\beta) (\\bfw^{0}-\\bfw^{*})d\\beta.\n\\end{array}\\eeq\n","sha256_utf8":"0df972c55ead93fca4a95db625ff9aaf2c3f03398d239ea98f6c48797ac36b4a"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"吸收误差项建立首步估计","source_units":[{"start":64608,"end":65784,"start_line":1155,"end_line":1172,"raw_tex":"Now the following chain of inequalities holds.\n\\begin{eqnarray*}\n\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rcll}\nc_*\\|\\bfw^{1}-\\bfw^{*}\\| &=& c_*\\|\\bfw^{0}+\\bfd^{0}-\\bfw^{*}\\|\\\\\n& {=} &c_*\\|\\bfw^{0}-\\bfw^{*}-\\Theta_{\\mu_0}^{-1}   F(\\bfw^{0};T_0) \\|&~~(\\text{by}~\\eqref{dkt})\\\\\n& {\\leq} & \\|\\Theta_{\\mu_0} (\\bfw^{0}-\\bfw^{*})-   F(\\bfw^{0};T_0) \\|&~~(\\text{by}~\\eqref{grad-F-w*-T0-c})\\\\\n&= & \\|\\Theta_{\\mu_0}  (\\bfw^{0}-\\bfw^{*})-  \\int_0^1 \\Theta(\\beta)(\\bfw^{0}-\\bfw^{*})d\\beta \\| &~~(\\text{by}~\\eqref{mean-value}) \\\\\n& \\leq& \\int_0^1 \\|\\Theta_{\\mu_0} -\\Theta(\\beta) \\|\\|\\bfw^{0}-\\bfw^{*}\\|d\\beta \\\\\n& =& ({\\theta_*}/2) \\|\\bfw^{0} -\\bfw^{*}\\|^2+   {\\rho C_*}  \\|\\bfw^1-\\bfw^*\\|\\|\\bfw^{0}-\\bfw^{*}\\| &~~(\\text{by}~\\eqref{lipschitz}) \\\\\n&\\leq& ({\\theta_*}/2) \\|\\bfw^{0} -\\bfw^{*}\\|^2+ {\\rho C_*\\delta_*}  \\|\\bfw^1-\\bfw^*\\|   \\\\\n&\\leq& ({\\theta_*}/2) \\|\\bfw^{0} -\\bfw^{*}\\|^2+ (c_*/2) \\|\\bfw^1-\\bfw^*\\|, \n\\end{array}\n\\end{eqnarray*}\nwhere $\\theta_*:=L_* +2\\rho C_*$ and the last inequality is from $\\delta_*\\leq  {c_*}/(2 \\theta_*)$ by \\eqref{delta-*} and\n\\beq\\label{w0-w*-delta*}\n\\|\\bfw^{0}-\\bfw^{*}\\|<\\delta_*\\leq  {c_*}/(2 \\theta_*)<  {c_*}/({2\\rho  C_*}).\n\\eeq\n","sha256_utf8":"e694177bfce07ca8b8b438da805a559e9edb05f28bcfeab44c1890edb730d2dd"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"首步二次误差与邻域保持","source_units":[{"start":65784,"end":66183,"start_line":1173,"end_line":1181,"raw_tex":" The above chain of inequalities  suffices to the following fact\n\\beq\\label{quadratic-chain}\n\\|\\bfw^{1}-\\bfw^{*}\\|\\leq  (\\theta_*/c_*) \\|\\bfw^{0} -\\bfw^{*}\\|^2.\n\\eeq\nThis together with $\\|\\bfw^{0}-\\bfw^{*}\\|<\\delta_*$ and \\eqref{w0-w*-delta*} derives\n\\beq \n\\|\\bfw^{1}-\\bfw^{*}\\|  &\\leq& \n   (\\theta_*/c_*)  \\delta_* \\|\\bfw^{0}-\\bfw^{*} \\|\\leq (1/2)\\|\\bfw^{0}-\\bfw^{*} \\|< \\delta_*,  \\nonumber\n \\eeq\n","sha256_utf8":"7eee584eb3376cd761cce2df022ccb1f4b942c9c16f8ba700b626da1acb86642"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"归纳二次收敛及方向趋零","source_units":[{"start":66183,"end":67078,"start_line":1182,"end_line":1187,"raw_tex":"which means $\\bfw^1\\in N(\\bfw^*,\\delta_*)$. In addition, $\\mu_1\\leq\\mu_0\\leq c_*/2$ by  \\eqref{mu-k}. Hence, replacing $T_0 $ by $T_1$, the same reasoning allows us to show that $\\bfd^{1}$ is well defined and $$\\|\\bfw^{2}-\\bfw^{*}\\|  \\leq (\\theta_*/c_*)   \\|\\bfw^{1}-\\bfw^{*} \\|^2 .$$ By the induction, we can conclude that $\\bfw^k\\in N(\\bfw^*,\\delta_*)$, $\\bfd^{k}$ is well defined and \n\\beq\\label{quadratic-rate}\n\\hspace{10mm}\\|\\bfw^{k+1}-\\bfw^{*}\\|  &\\leq& (\\theta_*/c_*) \\|\\bfw^{k}-\\bfw^{*} \\|^2, \\\\\n\\label{quadratic-rate-1}&\\leq& (\\theta_*/c_*)  \\|\\bfw^{k}-\\bfw^{*} \\|\\delta_* \\leq (1/2) \\|\\bfw^{k}-\\bfw^{*} \\|.\\hspace{.2cm}(\\text{by}~\\eqref{w0-w*-delta*})\n\\eeq\nTherefore, \\eqref{quadratic-rate} claims b). The conclusion of a) can be made by \\eqref{quadratic-rate-1}  that   $$\\bfw^{k}\\rightarrow\\bfw^{*},~~ \\bfd^k=\\bfw^{k+1}-\\bfw^{k}=\\bfw^{k+1}-\\bfw^{*}+\\bfw^{*}-\\bfw^{k}\\rightarrow0.$$ \n","sha256_utf8":"4abca73d8ee4067da93380c276baa1f3a84ccba17c7f23f206f39a7e3a9099cc"}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","title":"沿迭代线段控制雅可比","source_units":[{"start":67079,"end":67976,"start_line":1189,"end_line":1195,"raw_tex":"c)  The  above  proof shows $\\bfw^k\\in N(\\bfw^*,\\delta_*)$ and hence \\eqref{gw*-0} results in\n\\beq\\label{wk-in-N}  F(\\bfw^{*};T_k){=}0,\\eeq\nwhere $T_k=\\S_k\\cup\\E_k^o$.\nBy letting $\\bfw^{k}_\\beta= \\bfw^{*} + \\beta(\\bfw^{k}-\\bfw^{*}),$ where $\\beta\\in[0,1]$, we have $\\bfw^k_\\beta\\in N(\\bfw^*,\\delta_*)$. To show \\eqref{bd-jacobian-F-w-T} in \\cref{lemma-neighbour-2}, we verified the lower and upper bounds by \\eqref{lower-bd-FF} and \\eqref{upper-bd-FF}. Similarly, we can prove these bounds hold for $\\nabla F (\\bfw^{k}_\\beta;T_k)$. (In fact, since $\\bfw^k_\\beta\\in N(\\bfw^*,\\delta_*)\\subseteq N(\\bfw^*,\\delta_2^*)$ by $\\delta_*\\leq \\delta_2^*$, one just needs to set $\\mu=0$ and $\\bfw= \\bfw^{k}_\\beta$ in  \\eqref{lower-bd-FF} and \\eqref{upper-bd-FF}. Therefore,\n\\beq\\label{C-F-w-beta-c}\nC_*  \\geq \\|\\nabla F (\\bfw^{k}_\\beta;T_k)\\|\\geq\\sigma_{\\min}( \\nabla F (\\bfw^{k}_\\beta;T_k)) \\geq  c_*, \n\\eeq\n","sha256_utf8":"12503d3c86d26598402c55061dbb02a7238a342cde2cf07effb76ff5087756fb"}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","title":"比较残差范数与点误差","source_units":[{"start":67976,"end":68486,"start_line":1196,"end_line":1201,"raw_tex":"Again, the function $F(\\cdot,T_k)$ is differentiable for the fixed $T_k$, so the Mean-value theorem states that there is a $\\beta_0\\in(0,1)$ satisfying\n\\beq\\label{FkTk}\n\\|F(\\bfw^{k},T_k)\\|&=&\\|F(\\bfw^{*};T_k)+  \\nabla F (\\bfw^{k}_{\\beta_0};T_k) (\\bfw^{k}-\\bfw^{*})\\|\\nonumber\\\\\n&=&\\| \\nabla F (\\bfw^{k}_{\\beta_0};T_k) (\\bfw^{k}-\\bfw^{*}) \\|\\hspace{1.7cm}(\\text{by}~\\eqref{wk-in-N})\\nonumber\\\\\n&\\in&  [~c_*\\|\\bfw^{k}-\\bfw^{*} \\|, C_*\\|\\bfw^{k}-\\bfw^{*} \\|~], \\hspace{0.8cm}(\\text{by}~\\eqref{C-F-w-beta-c})\n\\eeq\n","sha256_utf8":"9ec944ce769703908876539436f453289ff2ccc84fd046d686fd4be980359872"}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","title":"残差二次下降与停机次数","source_units":[{"start":68486,"end":69522,"start_line":1202,"end_line":1217,"raw_tex":"This contributes to \n\\allowdisplaybreaks\n\\begin{eqnarray*}\\arraycolsep=1.4pt\\def\\arraystretch{1.5}\n\\begin{array}{rclr}\n\\|F(\\bfw^{k};T_{k})\\| \\leq  C_*\\|\\bfw^{k}-\\bfw^{*}\\|  \n & {\\leq} & (\\theta_*C_*/c_*)   \\|\\bfw^{k-1}-\\bfw^{*} \\|^2 &(\\text{by}~\\eqref{quadratic-rate}) \\\\\n & {\\leq}& (\\theta_*C_*/c_*^{3})  \\|F(\\bfw^{k-1};T_{k-1})\\|^2&(\\text{by}~\\eqref{FkTk})\\\\\n &{\\leq}& (\\theta_*C_*^3/c_*^{3})      \\|\\bfw^{k-1}-\\bfw^{*} \\|^2&(\\text{by}~\\eqref{FkTk})\\\\\n& {\\leq}& (\\theta_*C_*^3/c_*^{3})  2^{-2}  \\|\\bfw^{k-2}-\\bfw^{*} \\|^2&~~(\\text{by}~\\eqref{quadratic-rate-1})\\\\\n    &\\vdots&\\\\\n & \\leq& (\\theta_*C_*^3/c_*^{3})   2^{2-2k} \\|\\bfw^{0}-\\bfw^{*} \\|^2,&(\\text{by}~\\eqref{quadratic-rate-1})\n \\end{array}  \\end{eqnarray*}\nwhere the third inequality yields the first conclusion in c). This also enable to verify that $\\|F(\\bfw^{k};T_k)\\|$ $< {\\epsilon}$ if $k$ satisfies \\eqref{k-tol}.\n%$$k\\geq  \\lceil \\log_2 (2 \\sqrt{\\theta_* (C_* /c_*)^3}\\|\\bfw^{0}-\\bfw^{*} \\| )-\\log_2(\\sqrt{\\epsilon}) \\rceil.$$ \nThe whole proof is completed.\n\\end{proof}","sha256_utf8":"18bdb1a97a0e0fb6cb6ab1fa8666096a782989ba1e88058f6cd8d298b5c2f9c0"}]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","title":"广义次微分的标准定义","source_units":[],"declared_external":true,"declaration_text":["RW1998 Definition 8.3 and Equation 8(5) characterize the regular subdifferential by the first-order lower supporting inequality, the limiting subdifferential by limits of regular subgradients along function-value-convergent sequences, and the horizon subdifferential by corresponding limits of positively scaled regular subgradients with scale tending to zero. Definition 3.3 supplies the horizon-cone definition."]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","title":"正则性判据与复合链式法则","source_units":[],"declared_external":true,"declaration_text":["RW1998 Corollary 8.11 provides the subdifferential regularity criterion invoked in B0014. The qualified smooth-composition rule of Theorem 10.6 gives the subdifferential of a regular outer function composed with a smooth inner map when the transpose derivative has no nonzero annihilated horizon subgradient."]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","title":"费马必要条件","source_units":[],"declared_external":true,"declaration_text":["At a local minimum of a proper lower semicontinuous function, zero belongs to its limiting subdifferential. Adding a continuously differentiable function translates the limiting subdifferential by its gradient, as used with RW1998 Theorem 10.1."]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","title":"零一损失近端公式","source_units":[],"declared_external":true,"declaration_text":["For alpha > 0, the proximal minimization of alpha times the number of positive coordinates separates coordinatewise: its ith output is zero for 0 < nu_i < sqrt(2 alpha), either zero or nu_i at nu_i in {0, sqrt(2 alpha)}, and nu_i outside the closed threshold interval. This is the cited wang2019support Lemma 2.2 reproduced in B0019."]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","title":"奇异值扰动不等式","source_units":[],"declared_external":true,"declaration_text":["For equally sized matrices D and D', their ordered singular values satisfy max_i |sigma_i(D') - sigma_i(D)| <= ||D' - D|| in the spectral norm, as cited from lutkepohl1996handbook, Reminder (2), page 76."]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","title":"基础线性代数与微积分","source_units":[],"declared_external":true,"declaration_text":["Standard boundary facts used by the author include Cauchy-Schwarz and triangle inequalities; convex and strongly convex first-order inequalities; nonsingularity of a saddle-point matrix with positive definite primal block and full-row-rank constraint block; spectral norms and singular values of block diagonal matrices; norm bounds for submatrices; the integral fundamental theorem of calculus for continuously differentiable finite-dimensional maps; the ordinary real-valued mean-value theorem; induction and geometric-decay estimates. These facts do not assert the validity of every application made in the extracted author proof."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S060","edges":[{"source_id":"S001","target_id":"S006","kind":"supports"},{"source_id":"S001","target_id":"S024","kind":"supports"},{"source_id":"S001","target_id":"S025","kind":"supports"},{"source_id":"S002","target_id":"S026","kind":"supports"},{"source_id":"S002","target_id":"S047","kind":"supports"},{"source_id":"S002","target_id":"S055","kind":"supports"},{"source_id":"S003","target_id":"S019","kind":"supports"},{"source_id":"S074","target_id":"S019","kind":"supports"},{"source_id":"S074","target_id":"S020","kind":"supports"},{"source_id":"S074","target_id":"S021","kind":"supports"},{"source_id":"S075","target_id":"S021","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S020","target_id":"S018","kind":"supports"},{"source_id":"S021","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S076","target_id":"S024","kind":"supports"},{"source_id":"S017","target_id":"S024","kind":"supports"},{"source_id":"S008","target_id":"S024","kind":"supports"},{"source_id":"S017","target_id":"S025","kind":"supports"},{"source_id":"S005","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S026","target_id":"S029","kind":"supports"},{"source_id":"S027","target_id":"S029","kind":"supports"},{"source_id":"S079","target_id":"S027","kind":"supports"},{"source_id":"S079","target_id":"S029","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S023","kind":"supports"},{"source_id":"S026","target_id":"S023","kind":"supports"},{"source_id":"S027","target_id":"S023","kind":"supports"},{"source_id":"S028","target_id":"S023","kind":"supports"},{"source_id":"S029","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S077","target_id":"S004","kind":"supports"},{"source_id":"S004","target_id":"S033","kind":"supports"},{"source_id":"S004","target_id":"S034","kind":"supports"},{"source_id":"S022","target_id":"S032","kind":"supports"},{"source_id":"S005","target_id":"S032","kind":"supports"},{"source_id":"S008","target_id":"S032","kind":"supports"},{"source_id":"S007","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S007","target_id":"S033","kind":"supports"},{"source_id":"S006","target_id":"S034","kind":"supports"},{"source_id":"S017","target_id":"S034","kind":"supports"},{"source_id":"S022","target_id":"S034","kind":"supports"},{"source_id":"S006","target_id":"S035","kind":"supports"},{"source_id":"S004","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S079","target_id":"S036","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S034","target_id":"S031","kind":"supports"},{"source_id":"S035","target_id":"S031","kind":"supports"},{"source_id":"S036","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S008","target_id":"S039","kind":"supports"},{"source_id":"S011","target_id":"S039","kind":"supports"},{"source_id":"S079","target_id":"S039","kind":"supports"},{"source_id":"S010","target_id":"S040","kind":"supports"},{"source_id":"S004","target_id":"S040","kind":"supports"},{"source_id":"S006","target_id":"S040","kind":"supports"},{"source_id":"S011","target_id":"S040","kind":"supports"},{"source_id":"S010","target_id":"S041","kind":"supports"},{"source_id":"S004","target_id":"S041","kind":"supports"},{"source_id":"S006","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S005","target_id":"S042","kind":"supports"},{"source_id":"S006","target_id":"S042","kind":"supports"},{"source_id":"S010","target_id":"S042","kind":"supports"},{"source_id":"S011","target_id":"S042","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S041","target_id":"S038","kind":"supports"},{"source_id":"S042","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S045","kind":"supports"},{"source_id":"S016","target_id":"S045","kind":"supports"},{"source_id":"S061","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S031","target_id":"S046","kind":"supports"},{"source_id":"S007","target_id":"S046","kind":"supports"},{"source_id":"S010","target_id":"S046","kind":"supports"},{"source_id":"S008","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S010","target_id":"S047","kind":"supports"},{"source_id":"S079","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S045","target_id":"S049","kind":"supports"},{"source_id":"S046","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S045","target_id":"S050","kind":"supports"},{"source_id":"S011","target_id":"S050","kind":"supports"},{"source_id":"S046","target_id":"S051","kind":"supports"},{"source_id":"S048","target_id":"S051","kind":"supports"},{"source_id":"S010","target_id":"S051","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S044","kind":"supports"},{"source_id":"S049","target_id":"S044","kind":"supports"},{"source_id":"S050","target_id":"S044","kind":"supports"},{"source_id":"S051","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S008","target_id":"S054","kind":"supports"},{"source_id":"S079","target_id":"S054","kind":"supports"},{"source_id":"S078","target_id":"S055","kind":"supports"},{"source_id":"S043","target_id":"S056","kind":"supports"},{"source_id":"S054","target_id":"S056","kind":"supports"},{"source_id":"S011","target_id":"S056","kind":"supports"},{"source_id":"S079","target_id":"S056","kind":"supports"},{"source_id":"S009","target_id":"S057","kind":"supports"},{"source_id":"S011","target_id":"S057","kind":"supports"},{"source_id":"S055","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S011","target_id":"S058","kind":"supports"},{"source_id":"S055","target_id":"S059","kind":"supports"},{"source_id":"S056","target_id":"S059","kind":"supports"},{"source_id":"S057","target_id":"S059","kind":"supports"},{"source_id":"S011","target_id":"S059","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S055","target_id":"S053","kind":"supports"},{"source_id":"S056","target_id":"S053","kind":"supports"},{"source_id":"S057","target_id":"S053","kind":"supports"},{"source_id":"S058","target_id":"S053","kind":"supports"},{"source_id":"S059","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S014","target_id":"S063","kind":"supports"},{"source_id":"S061","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S043","target_id":"S064","kind":"supports"},{"source_id":"S052","target_id":"S064","kind":"supports"},{"source_id":"S061","target_id":"S064","kind":"supports"},{"source_id":"S012","target_id":"S064","kind":"supports"},{"source_id":"S013","target_id":"S064","kind":"supports"},{"source_id":"S015","target_id":"S064","kind":"supports"},{"source_id":"S061","target_id":"S065","kind":"supports"},{"source_id":"S079","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S064","target_id":"S066","kind":"supports"},{"source_id":"S009","target_id":"S066","kind":"supports"},{"source_id":"S011","target_id":"S066","kind":"supports"},{"source_id":"S013","target_id":"S066","kind":"supports"},{"source_id":"S014","target_id":"S066","kind":"supports"},{"source_id":"S015","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S067","kind":"supports"},{"source_id":"S064","target_id":"S067","kind":"supports"},{"source_id":"S011","target_id":"S067","kind":"supports"},{"source_id":"S079","target_id":"S067","kind":"supports"},{"source_id":"S064","target_id":"S068","kind":"supports"},{"source_id":"S066","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S061","target_id":"S068","kind":"supports"},{"source_id":"S013","target_id":"S068","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S061","target_id":"S069","kind":"supports"},{"source_id":"S069","target_id":"S070","kind":"supports"},{"source_id":"S063","target_id":"S070","kind":"supports"},{"source_id":"S043","target_id":"S070","kind":"supports"},{"source_id":"S052","target_id":"S070","kind":"supports"},{"source_id":"S012","target_id":"S070","kind":"supports"},{"source_id":"S015","target_id":"S070","kind":"supports"},{"source_id":"S079","target_id":"S070","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S043","target_id":"S071","kind":"supports"},{"source_id":"S052","target_id":"S071","kind":"supports"},{"source_id":"S053","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S011","target_id":"S072","kind":"supports"},{"source_id":"S079","target_id":"S072","kind":"supports"},{"source_id":"S070","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S079","target_id":"S073","kind":"supports"},{"source_id":"S063","target_id":"S062","kind":"supports"},{"source_id":"S064","target_id":"S062","kind":"supports"},{"source_id":"S065","target_id":"S062","kind":"supports"},{"source_id":"S066","target_id":"S062","kind":"supports"},{"source_id":"S067","target_id":"S062","kind":"supports"},{"source_id":"S068","target_id":"S062","kind":"supports"},{"source_id":"S069","target_id":"S062","kind":"supports"},{"source_id":"S070","target_id":"S062","kind":"supports"},{"source_id":"S071","target_id":"S062","kind":"supports"},{"source_id":"S072","target_id":"S062","kind":"supports"},{"source_id":"S073","target_id":"S062","kind":"supports"},{"source_id":"S062","target_id":"S060","kind":"supports"}]}}
{"record_id":"2105.09716v1","paper":{"title":"A Stochastic Composite Augmented Lagrangian Method For Reinforcement Learning","authors":["Yongfeng Li","Mingming Zhao","Weijie Chen","Zaiwen Wen"],"arxiv_id":"2105.09716","version":"v1","journal":"SIAM Journal on Optimization","doi":"10.1137/21m1421726","abstract_url":"https://arxiv.org/abs/2105.09716v1","pdf_url":"https://arxiv.org/pdf/2105.09716v1"},"context":{"raw_tex":"We consider the standard reinforcement learning setting characterized by an infinite-horizon discounted MDP. Formally, a MDP is described by the tuple $(\\mathcal{S},\\mathcal{A},P,r,\\rho_0, \\gamma)$. We assume that $\\mathcal{S}$ and $\\mathcal{A}$ represent the finite state and action spaces, respectively. $P: \\mathcal{S}\\times \\mathcal{A}\\times \\mathcal{S}\\rightarrow \\mathbf{R}$ indicates the transition dynamics by a conditional distribution over the state and action spaces, $r: \\mathcal{S} \\times \\mathcal{A} \\rightarrow \\mathbf{R}$ is the bounded reward function. The initial state $s_0$ of the MDP obeys the distribution $\\rho_0: \\mathcal{S}\\rightarrow \\mathbf{R}$, and $\\gamma \\in (0,1)$ is the discount factor.\n\n\n\\begin{equation}\\label{eq:rllp-primal}\n\\begin{split}\n\\mathrm{(P):}\\quad\\min_V& \\sum_{s} \\rho_0(s)V(s), \\\\\n\\st & V(s)\\geq  r(s,a)+  \\gamma \\mathbb{E}_{s'|s,a}[ V(s')] , \\forall s, a.\n\\end{split}\n\\end{equation}\n\n\\begin{equation}\\label{eq:rllp-dual}\n\\begin{split}\n\\mathrm{(D):}\\quad\\max_{x\\geq0}& \\sum_{s,a} r(s,a)w(s,a)x(s,a), \\\\\n\\st& \\sum_{a}w(s',a)x(s',a)- \\sum_{s,a} \\gamma P(s'|s,a) w(s,a)x(s,a) = \\rho_0(s'), \\forall s',\n\\end{split}\n\\end{equation}\n\nwhere the weight function $w$ is a distribution with respect to the state and action spaces, i.e., \n\\bee\n\\sum\\limits_{s\\in\\mathcal{S},a\\in\\mathcal{A}}w(s,a)=1.\n\\eee\nThe only difference from the standard dual formulation is that the variables $x(s,a)$ carries the weight $w(s,a)$. Actually, $x$ can be viewed as the multipliers using a weighted inner product other than the standard case in Euclidean space. Moreover, the weight function $w(s,a)$ enables us to perform a sampling process from a replay buffer in large and complex environments. In practice, the explicit form of $w(s,a)$ is not required as it simply characterizes the sampling behavior.\n\n\nTo simplify the following derivation, we consider an equivalent primal LP formulation with equality constraints by introducing an additional slack variable. Namely, we reformulate \\eqref{eq:rllp-primal} as \n\\begin{equation}\\label{eq:rllp-primal-eq}\n\\begin{split}\n\\min_{h\\geq0,V}&\\  \\sum_{s} \\rho_0(s)V(s), \\\\\n\\st & V(s)=  r(s,a)+  \\gamma  \\mathbb{E}_{s'|s,a}[ V(s')]+h(s,a), \\forall s, a.\n\\end{split}\n\\end{equation}\n\nWe denote $x(s,a)$ as the multiplier of the constraints in \\eqref{eq:rllp-primal-eq}. For simplicity, we introduce a $Z$-function as\n\\bee\nZ_\\mu(V,h,x,s,a) =  x(s,a)+  \\mu \\left(h(s,a)+r(s,a)+\\gamma \\mathbb{E}_{s'|s,a}[ V(s')] - V(s)\\right),\\label{zfunc}\n\\eee\n\nand define\n\\bee\nz_\\mu(V,h,x,s,a,s') =  x(s,a)+  \\mu \\left(h(s,a)+r(s,a)+\\gamma V(s') - V(s)\\right),\\label{zfunc1}\n\\eee\n\nwhere $\\mu>0$ is the penalty parameter. Since \n\\bea\\label{sumP=1}\n\\sum_{s'} P(s'|s,a) = 1,\n\\eea\n\nwe have\n\\bea\\label{Z-z}\nZ_\\mu(V,h,x,s,a) = \\mathbb{E}_{s'|s,a} [z_\\mu(V,h,x,s,a,s')].\n\\eea\n\nThat is to say, $Z$-function is a conditional expectation of $s'$. For the LP problem \\eqref{eq:rllp-primal-eq}, the weighted augmented Lagrangian function is defined as \n\\begin{equation}\\label{randALM}\n\\mathcal{L}_{\\mu}(V,h,x) = \\sum_{s} \\rho_0(s)V(s) +\\frac{1}{2\\mu}\\sum_{s,a}w(s,a)[Z_\\mu(V,h,x,s,a)]^2.\n\\end{equation}\n\nWe next parametrize the variables by the following two-layer neural networks:\n\\bea\nV_{\\phi} (s) &=& \\frac{1}{\\sqrt{m_V}} \\sum_{i=1}^{m_V} \\phi_{1,i} \\cdot \\sigma(\\phi_{2,i}^Ts), \\\\ \nh_{\\psi} (s,a) &=& \\frac{1}{\\sqrt{m_h}} \\sum_{i=1}^{m_h} \\psi_{1,i} \\cdot \\sigma(\\psi_{2,i}^T(s,a)), \\\\\nx_{\\theta} (s,a) &=& \\frac{1}{\\sqrt{m_x}} \\sum_{i=1}^{m_x} \\theta_{1,i} \\cdot \\sigma(\\theta_{2,i}^T(s,a)),\n\\eea\n\nwhere $m_V, m_h, m_x$ are integers, $\\sigma =  \\max(0, x)$ is the ReLU function. The parameters are randomly initialized by either the uniform distribution or normal distribution, i.e.,\n\\be\\label{eq:dist}\n\\begin{split}\n&\\phi_{1,i}, \\theta_{1,i} \\sim \\mathcal{U}(\\{1,-1\\}), \\quad \\psi_{1,i} = 1, \\\\\n&\\phi_{2,i} \\sim \\mathcal{N}(0, I_{d_{s}}/d_s),\\quad \\psi_{2,i},\\theta_{1,i} \\sim \\mathcal{N}(0, I_{d_{sa}}/d_{sa}), \n\\end{split}\n\\ee\n\nwhere $d_{s}$ is the dimension of $s$ and $d_{sa}$ is the dimension of $(s,a)$. \nIn particular, $\\phi_{1,i}, \\psi_{1,i}, \\theta_{1,i}$ are fixed and we only optimize the parameter $\\phi_{2,i}, \\psi_{2,i}, \\theta_{2,i}$.\nLet $\\phi = (\\phi_{2,i})_{i=1}^{m_V}$, $\\psi = (\\psi_{2,i})_{i=1}^{m_V}$ and $\\theta = (\\theta_{2,i})_{i=1}^{m_V}$.\nThe corresponding random initialization parameters are denoted by $\\phi^0, \\psi^0, \\theta^0$. The expectation under the distribution \\eqref{eq:dist} is denoted by $E_0$.\n\nDefine the following function classes \n\\beaa\n\\mathcal{F}^V_{R_V, m_V} &=& \\{\\frac{1}{\\sqrt{m_V}} \\sum_{i=1}^{m_V} \\phi_{1,i} \\cdot 1_{\\{(\\phi_{2,i}^0)^Ts \\geq 0 \\}} \\phi_{2,i}^Ts : \\ \\|\\phi-\\phi^0\\| \\leq R_V\\},\\\\\n\\mathcal{F}^h_{R_h, m_h} &=& \\{\\frac{1}{\\sqrt{m_h}} \\sum_{i=1}^{m_h} \\psi_{1,i} \\cdot 1_{\\{(\\psi_{2,i}^0)^T(s,a) \\geq 0 \\}} \\psi_{2,i}^T(s,a): \\ \\|\\psi-\\psi^0\\| \\leq R_h\\}, \\\\\n\\mathcal{F}^x_{R_x, m_x} &=& \\{\\frac{1}{\\sqrt{m_h}} \\sum_{i=1}^{m_h} \\theta_{1,i} \\cdot 1_{\\{(\\theta_{2,i}^0)^T(s,a) \\geq 0 \\}} \\theta_{2,i}^T(s,a): \\ \\|\\theta-\\theta^0\\| \\leq R_x\\},\n\\eeaa\n\nand let \n\\bee\n\\Xi = \\{\\zeta: \\|\\phi-\\phi^0\\| \\leq R_V, \\|\\psi-\\psi^0\\| \\leq R_h \\text{ and } \\|\\theta-\\theta^0\\| \\leq R_x\\}.\n\\eee\n\nFor simplicity, we assume  that $\\|(s,a)\\| \\leq 1$, which follows that $\\|s\\| \\leq 1$. On $\\Xi$, we have\n\\be\\label{eq:bv}\n\\begin{split}\n&|V_\\phi(s) - V_{\\phi^0}(s)| \\leq \\frac{1}{\\sqrt{m_V}} \\sum_{i=1}^{m_V} |\\phi_{1,i}| \\cdot |\\sigma(\\phi_{2,i}^Ts) - \\sigma((\\phi_{2,i}^0)^Ts)| \\\\\n \\leq& \\frac{1}{\\sqrt{m_V}} \\sum_{i=1}^{m_V} |\\phi_{1,i}| \\cdot |\\phi_{2,i}^Ts - (\\phi_{2,i}^0)^Ts| \\leq \\frac{1}{\\sqrt{m_V}} \\sum_{i=1}^{m_V} \\|\\phi_{2,i} - (\\phi_{2,i}^0)\\| \\leq R_V^{1/2}.\n \\end{split}\n\\ee\n\nSimilarly, we have $|h_\\psi(s) - h_{\\psi^0}(s)| \\leq R_h^{1/2}$ and $|x_\\theta(s) - x_{\\theta^0}(s)| \\leq R_x^{1/2}$.\nWe can also obtain that\n\\beaa\n\\|\\nabla_\\phi V_\\phi(s) \\|^2 = \\frac{1}{m_V} \\sum_{i=1}^{m_V} 1_{\\{(\\phi_{2,i})^Ts \\geq 0 \\}} \\cdot \\|s\\|^2 \\leq 1.\n\\eeaa\n\nWe also yields $\\|\\nabla_\\psi h_\\psi(s,a) \\| \\leq  1$ and $\\|\\nabla_\\theta x_\\theta(s,a) \\| \\leq  1$.\nConsider that $(V,h,x)$ is in the Hilbert space $\\mathcal{H}$ with inner product\n\\[\n\\iprod{(V,h,x)}{(V',h',x')}_{\\mathcal{H}} = \\sum_{s,a}w(s,a)\\left(V(s)^2+ x(s,a)^2 + h(s,a)^2 \\right).\n\\]\n\nThe corresponding norm in $\\mathcal{H}$ is denoted by $\\|\\cdot\\|_{\\mathcal{H}}$.\nDefine\n\\begin{equation}\\label{comp-func-non}\n \\begin{split}\n \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x) \n&:= \\sum_{s} \\rho_0(s)V(s) +\\frac{1}{2\\mu}\\sum_{s,a}w(s,a)x(s,a)Z_\\mu(V,h,x_{\\theta^{k}},s,a)\\\\\n&+\\frac{\\beta}{2}\\sum_{s,a} w(s,a)\\left[x(s,a) - \\left(Z_\\mu(V,h,x_{\\theta^{k}},s,a)\\right)\\right]^2.\n\\end{split}\n\\end{equation}\n\nHere, we use the same notation as \\eqref{comp-func}. It is easy to distinguish them according to variables. Let $\\tilde{V}^k, \\tilde{h}^k, \\tilde{x}^k = \\arg\\min_{h\\geq 0, V, x} \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$. The following lemma shows properties of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$. \n\n\n\\begin{assumption}\\label{assum:fc}\n    \\begin{enumerate}[(i)]\n    \\item It holds that $\\tilde{V}^k \\in \\mathcal{F}^V_{R_V, m_V}, \\tilde{h}^k \\in \\mathcal{F}^h_{R_h, m_h}, \\tilde{x}^k \\in \\mathcal{F}^x_{R_x, m_x}$ for any $k$.\n    \\item For any unit vector $u, v$ and nonnegtive scalar $\\tau$, $E_w 1_{\\{|u^T(s,a)| < \\tau\\}} \\leq c\\tau$ and $E_w 1_{\\{|v^Ts| < \\tau\\}} \\leq c\\tau$ holds, where $c$ is a constant and $E_w f = \\sum_{s,a} w(s,a)f(s,a)$.  \n\\end{enumerate}\n\\end{assumption}\n\nAssumption \\ref{assum:fc} (i) is commonly used in the literature and Assumption \\ref{assum:fc} (ii) is on the regularity of the distribution $w$. For convenience, let $\\zeta = (\\phi, \\psi, \\theta)$, $\\Gamma = (V,h,x)$, $R = \\max\\{R_V, R_h, R_x\\}$ and $m = \\min\\{m_V, m_h, m_x\\}$. The following method is used to minimize \\eqref{comp-func}.\nLet $\\zeta^{k,0} = \\zeta^{k}$ and \n\\be\\label{eq:pjg}\n\\begin{split}\n    \\zeta^{k,t+1} &= \\Pi_{\\zeta \\in \\Xi}\\left(\\zeta^{k,t} - \\alpha_t \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}},h_{\\psi^{k,t}},x_{\\theta^{k,t}} )\\right), \\\\\n    \\zeta^{k+1} &= \\frac{1}{T}\\sum_{t=1}^T \\zeta^{k,t}.\n\\end{split}\n\\ee\n\nThe scheme \\eqref{eq:pjg} is actually a projected gradient descent method. Comparing with the algorithm in Section \\ref{The Augmented Lagrangian Method}, the scheme \\eqref{eq:pjg} has an additional projection and average operator, which is easy  for analysis. We also ignore the stochastic part, which is not hard to analyze by the method in \\cite{liu2019neural}.\nIn the following part, we show the convergence of the scheme \\eqref{eq:pjg}. Define the linearization function\n\\beaa\nV_{\\phi}^0 (s) &=& \\frac{1}{\\sqrt{m_V}} \\sum_{i=1}^{m_V} \\phi_{1,i} \\cdot 1_{\\{(\\phi_{2,i}^0)^Ts \\geq 0 \\}} \\phi_{2,i}^Ts, \\\\ \nh_{\\psi}^0 (s,a) &=& \\frac{1}{\\sqrt{m_h}} \\sum_{i=1}^{m_h} \\psi_{1,i} \\cdot 1_{\\{(\\psi_{2,i}^0)^T(s,a) \\geq 0 \\}} \\psi_{2,i}^T(s,a), \\\\\nx_{\\theta}^0 (s,a) &=& \\frac{1}{\\sqrt{m_x}} \\sum_{i=1}^{m_x} \\theta_{1,i} \\cdot 1_{\\{(\\theta_{2,i}^0)^T(s,a) \\geq 0 \\}} \\theta_{2,i}^T(s,a).\n\\eeaa\n\nLet $\\mathcal{S}$ be the subgradient operator of $-\\sum_{s,a} r(s,a)w(s,a)x(s,a) + 1_\\Omega(x)$ with respect to $x$, where\n\\bee\n\\Omega = \\{x :  \\sum_{s,a} (\\delta_{s'}(s) - \\gamma P(s'|s,a)) w(s,a)x(s,a) = \\rho_0(s'), \\forall s', \nx(s,a) \\geq 0, \\forall s,a\\}.\n\\eee\n\nThen $\\mathcal{S}$  is a maximal monotone operator. Lemma \\ref{lem:lp} means that \n\\[\nZ_\\mu(\\hat{V}^{k}, \\hat{h}^k, x_{\\theta^k}, s,a) = \\mathcal{T}x_{\\theta^k}:= (I + \\mu \\mathcal{S})^{-1}x_{\\theta^k} ,\n\\]\n\nwhere $\\hat{V}^{k},\\hat{h}^{k} = \\arg\\min_{h\\geq 0,V}  \\mathcal{L}_{\\mu}(V,h, x_{\\theta^k}) $.\n","units":[{"start":22019,"end":22739,"start_line":244,"end_line":244,"raw_tex":"We consider the standard reinforcement learning setting characterized by an infinite-horizon discounted MDP. Formally, a MDP is described by the tuple $(\\mathcal{S},\\mathcal{A},P,r,\\rho_0, \\gamma)$. We assume that $\\mathcal{S}$ and $\\mathcal{A}$ represent the finite state and action spaces, respectively. $P: \\mathcal{S}\\times \\mathcal{A}\\times \\mathcal{S}\\rightarrow \\mathbf{R}$ indicates the transition dynamics by a conditional distribution over the state and action spaces, $r: \\mathcal{S} \\times \\mathcal{A} \\rightarrow \\mathbf{R}$ is the bounded reward function. The initial state $s_0$ of the MDP obeys the distribution $\\rho_0: \\mathcal{S}\\rightarrow \\mathbf{R}$, and $\\gamma \\in (0,1)$ is the discount factor.\n","sha256_utf8":"97f9b3d29d225955491af8cc5fbbcc07daac862a90307306e313221261678de3"},{"start":24144,"end":24352,"start_line":270,"end_line":275,"raw_tex":"\\begin{equation}\\label{eq:rllp-primal}\n\\begin{split}\n\\mathrm{(P):}\\quad\\min_V& \\sum_{s} \\rho_0(s)V(s), \\\\\n\\st & V(s)\\geq  r(s,a)+  \\gamma \\mathbb{E}_{s'|s,a}[ V(s')] , \\forall s, a.\n\\end{split}\n\\end{equation}","sha256_utf8":"59e65272ab4229d80d52c2abd10397c2a334ef0cb84b763d568dafde63946605"},{"start":24952,"end":25192,"start_line":285,"end_line":290,"raw_tex":"\\begin{equation}\\label{eq:rllp-dual}\n\\begin{split}\n\\mathrm{(D):}\\quad\\max_{x\\geq0}& \\sum_{s,a} r(s,a)w(s,a)x(s,a), \\\\\n\\st& \\sum_{a}w(s',a)x(s',a)- \\sum_{s,a} \\gamma P(s'|s,a) w(s,a)x(s,a) = \\rho_0(s'), \\forall s',\n\\end{split}\n\\end{equation}","sha256_utf8":"1aaf23244b5c348d541b32e4ab4e2f5e499ece63bc1fd54cdab29e634274f1b7"},{"start":25193,"end":25845,"start_line":291,"end_line":295,"raw_tex":"where the weight function $w$ is a distribution with respect to the state and action spaces, i.e., \n\\bee\n\\sum\\limits_{s\\in\\mathcal{S},a\\in\\mathcal{A}}w(s,a)=1.\n\\eee\nThe only difference from the standard dual formulation is that the variables $x(s,a)$ carries the weight $w(s,a)$. Actually, $x$ can be viewed as the multipliers using a weighted inner product other than the standard case in Euclidean space. Moreover, the weight function $w(s,a)$ enables us to perform a sampling process from a replay buffer in large and complex environments. In practice, the explicit form of $w(s,a)$ is not required as it simply characterizes the sampling behavior.\n","sha256_utf8":"f9673b85d6851f2a81426b385f853b065c0fbd208601c212f5ae68fd395ae656"},{"start":26845,"end":27260,"start_line":312,"end_line":318,"raw_tex":"To simplify the following derivation, we consider an equivalent primal LP formulation with equality constraints by introducing an additional slack variable. Namely, we reformulate \\eqref{eq:rllp-primal} as \n\\begin{equation}\\label{eq:rllp-primal-eq}\n\\begin{split}\n\\min_{h\\geq0,V}&\\  \\sum_{s} \\rho_0(s)V(s), \\\\\n\\st & V(s)=  r(s,a)+  \\gamma  \\mathbb{E}_{s'|s,a}[ V(s')]+h(s,a), \\forall s, a.\n\\end{split}\n\\end{equation}","sha256_utf8":"56d810e727e3c52666fe3843c20341dc71c19af1b1b44c14f4332cb504ae1ee5"},{"start":27517,"end":27775,"start_line":326,"end_line":329,"raw_tex":"We denote $x(s,a)$ as the multiplier of the constraints in \\eqref{eq:rllp-primal-eq}. For simplicity, we introduce a $Z$-function as\n\\bee\nZ_\\mu(V,h,x,s,a) =  x(s,a)+  \\mu \\left(h(s,a)+r(s,a)+\\gamma \\mathbb{E}_{s'|s,a}[ V(s')] - V(s)\\right),\\label{zfunc}\n\\eee","sha256_utf8":"616c0a057aa9009d1caaf3b9f2c44b37c73413a2c48f68dc24581fbae84f415c"},{"start":27776,"end":27894,"start_line":330,"end_line":333,"raw_tex":"and define\n\\bee\nz_\\mu(V,h,x,s,a,s') =  x(s,a)+  \\mu \\left(h(s,a)+r(s,a)+\\gamma V(s') - V(s)\\right),\\label{zfunc1}\n\\eee","sha256_utf8":"da40aa622cfd1f2439163f5013c124fa0296e4f1fc2f10f75920b01b253f09cb"},{"start":27895,"end":27990,"start_line":334,"end_line":337,"raw_tex":"where $\\mu>0$ is the penalty parameter. Since \n\\bea\\label{sumP=1}\n\\sum_{s'} P(s'|s,a) = 1,\n\\eea","sha256_utf8":"5e560d047e427aee5b96bca869b35897c64382db4dd64f5feb9bb1f29e4281b2"},{"start":27991,"end":28081,"start_line":338,"end_line":341,"raw_tex":"we have\n\\bea\\label{Z-z}\nZ_\\mu(V,h,x,s,a) = \\mathbb{E}_{s'|s,a} [z_\\mu(V,h,x,s,a,s')].\n\\eea","sha256_utf8":"f2fb5d7249a2450b46120daf635dd03cb392d5ad25efeefbb783c9c58d8e46ad"},{"start":28082,"end":28402,"start_line":342,"end_line":345,"raw_tex":"That is to say, $Z$-function is a conditional expectation of $s'$. For the LP problem \\eqref{eq:rllp-primal-eq}, the weighted augmented Lagrangian function is defined as \n\\begin{equation}\\label{randALM}\n\\mathcal{L}_{\\mu}(V,h,x) = \\sum_{s} \\rho_0(s)V(s) +\\frac{1}{2\\mu}\\sum_{s,a}w(s,a)[Z_\\mu(V,h,x,s,a)]^2.\n\\end{equation}","sha256_utf8":"a123bccf03a06e999fb475a8f15de8150ac4094c4dbdfcf1a8d6b1224c8a922a"},{"start":73112,"end":73509,"start_line":894,"end_line":899,"raw_tex":"We next parametrize the variables by the following two-layer neural networks:\n\\bea\nV_{\\phi} (s) &=& \\frac{1}{\\sqrt{m_V}} \\sum_{i=1}^{m_V} \\phi_{1,i} \\cdot \\sigma(\\phi_{2,i}^Ts), \\\\ \nh_{\\psi} (s,a) &=& \\frac{1}{\\sqrt{m_h}} \\sum_{i=1}^{m_h} \\psi_{1,i} \\cdot \\sigma(\\psi_{2,i}^T(s,a)), \\\\\nx_{\\theta} (s,a) &=& \\frac{1}{\\sqrt{m_x}} \\sum_{i=1}^{m_x} \\theta_{1,i} \\cdot \\sigma(\\theta_{2,i}^T(s,a)),\n\\eea","sha256_utf8":"e96f6106b36dcb80904af12e29e4ff9722643b06ce6f30bc0b1fa9975cdb4b07"},{"start":73510,"end":73941,"start_line":900,"end_line":906,"raw_tex":"where $m_V, m_h, m_x$ are integers, $\\sigma =  \\max(0, x)$ is the ReLU function. The parameters are randomly initialized by either the uniform distribution or normal distribution, i.e.,\n\\be\\label{eq:dist}\n\\begin{split}\n&\\phi_{1,i}, \\theta_{1,i} \\sim \\mathcal{U}(\\{1,-1\\}), \\quad \\psi_{1,i} = 1, \\\\\n&\\phi_{2,i} \\sim \\mathcal{N}(0, I_{d_{s}}/d_s),\\quad \\psi_{2,i},\\theta_{1,i} \\sim \\mathcal{N}(0, I_{d_{sa}}/d_{sa}), \n\\end{split}\n\\ee","sha256_utf8":"18bab2a6c72589ef48a2f684456b944c72458150dbd6547b4a2a892bf060d2bf"},{"start":73942,"end":75026,"start_line":907,"end_line":917,"raw_tex":"where $d_{s}$ is the dimension of $s$ and $d_{sa}$ is the dimension of $(s,a)$. \nIn particular, $\\phi_{1,i}, \\psi_{1,i}, \\theta_{1,i}$ are fixed and we only optimize the parameter $\\phi_{2,i}, \\psi_{2,i}, \\theta_{2,i}$.\nLet $\\phi = (\\phi_{2,i})_{i=1}^{m_V}$, $\\psi = (\\psi_{2,i})_{i=1}^{m_V}$ and $\\theta = (\\theta_{2,i})_{i=1}^{m_V}$.\nThe corresponding random initialization parameters are denoted by $\\phi^0, \\psi^0, \\theta^0$. The expectation under the distribution \\eqref{eq:dist} is denoted by $E_0$.\n\nDefine the following function classes \n\\beaa\n\\mathcal{F}^V_{R_V, m_V} &=& \\{\\frac{1}{\\sqrt{m_V}} \\sum_{i=1}^{m_V} \\phi_{1,i} \\cdot 1_{\\{(\\phi_{2,i}^0)^Ts \\geq 0 \\}} \\phi_{2,i}^Ts : \\ \\|\\phi-\\phi^0\\| \\leq R_V\\},\\\\\n\\mathcal{F}^h_{R_h, m_h} &=& \\{\\frac{1}{\\sqrt{m_h}} \\sum_{i=1}^{m_h} \\psi_{1,i} \\cdot 1_{\\{(\\psi_{2,i}^0)^T(s,a) \\geq 0 \\}} \\psi_{2,i}^T(s,a): \\ \\|\\psi-\\psi^0\\| \\leq R_h\\}, \\\\\n\\mathcal{F}^x_{R_x, m_x} &=& \\{\\frac{1}{\\sqrt{m_h}} \\sum_{i=1}^{m_h} \\theta_{1,i} \\cdot 1_{\\{(\\theta_{2,i}^0)^T(s,a) \\geq 0 \\}} \\theta_{2,i}^T(s,a): \\ \\|\\theta-\\theta^0\\| \\leq R_x\\},\n\\eeaa","sha256_utf8":"275383926381a187503d80ee315d37c61036cfc082d6015af46adb9bb86b94c9"},{"start":75027,"end":75156,"start_line":918,"end_line":921,"raw_tex":"and let \n\\bee\n\\Xi = \\{\\zeta: \\|\\phi-\\phi^0\\| \\leq R_V, \\|\\psi-\\psi^0\\| \\leq R_h \\text{ and } \\|\\theta-\\theta^0\\| \\leq R_x\\}.\n\\eee","sha256_utf8":"a653351dd5ba409e497ce38f44edf32d692116b1f6735c2f009269e8531361aa"},{"start":75157,"end":75646,"start_line":922,"end_line":928,"raw_tex":"For simplicity, we assume  that $\\|(s,a)\\| \\leq 1$, which follows that $\\|s\\| \\leq 1$. On $\\Xi$, we have\n\\be\\label{eq:bv}\n\\begin{split}\n&|V_\\phi(s) - V_{\\phi^0}(s)| \\leq \\frac{1}{\\sqrt{m_V}} \\sum_{i=1}^{m_V} |\\phi_{1,i}| \\cdot |\\sigma(\\phi_{2,i}^Ts) - \\sigma((\\phi_{2,i}^0)^Ts)| \\\\\n \\leq& \\frac{1}{\\sqrt{m_V}} \\sum_{i=1}^{m_V} |\\phi_{1,i}| \\cdot |\\phi_{2,i}^Ts - (\\phi_{2,i}^0)^Ts| \\leq \\frac{1}{\\sqrt{m_V}} \\sum_{i=1}^{m_V} \\|\\phi_{2,i} - (\\phi_{2,i}^0)\\| \\leq R_V^{1/2}.\n \\end{split}\n\\ee","sha256_utf8":"30bb1b78f4c260363531a28a07f30f5affd9dd4575b5448b3375a77b765c7d0b"},{"start":75647,"end":75916,"start_line":929,"end_line":933,"raw_tex":"Similarly, we have $|h_\\psi(s) - h_{\\psi^0}(s)| \\leq R_h^{1/2}$ and $|x_\\theta(s) - x_{\\theta^0}(s)| \\leq R_x^{1/2}$.\nWe can also obtain that\n\\beaa\n\\|\\nabla_\\phi V_\\phi(s) \\|^2 = \\frac{1}{m_V} \\sum_{i=1}^{m_V} 1_{\\{(\\phi_{2,i})^Ts \\geq 0 \\}} \\cdot \\|s\\|^2 \\leq 1.\n\\eeaa","sha256_utf8":"98aec0bfc3419e7b311be47fa3add0510b940fa7f179bc0960adb8443c0dacba"},{"start":75917,"end":76208,"start_line":934,"end_line":938,"raw_tex":"We also yields $\\|\\nabla_\\psi h_\\psi(s,a) \\| \\leq  1$ and $\\|\\nabla_\\theta x_\\theta(s,a) \\| \\leq  1$.\nConsider that $(V,h,x)$ is in the Hilbert space $\\mathcal{H}$ with inner product\n\\[\n\\iprod{(V,h,x)}{(V',h',x')}_{\\mathcal{H}} = \\sum_{s,a}w(s,a)\\left(V(s)^2+ x(s,a)^2 + h(s,a)^2 \\right).\n\\]","sha256_utf8":"cd832a31e2cdfd474c258833ace2efe4f2fa3d9cc8e90d328a00a169ab06a956"},{"start":76209,"end":76617,"start_line":939,"end_line":947,"raw_tex":"The corresponding norm in $\\mathcal{H}$ is denoted by $\\|\\cdot\\|_{\\mathcal{H}}$.\nDefine\n\\begin{equation}\\label{comp-func-non}\n \\begin{split}\n \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x) \n&:= \\sum_{s} \\rho_0(s)V(s) +\\frac{1}{2\\mu}\\sum_{s,a}w(s,a)x(s,a)Z_\\mu(V,h,x_{\\theta^{k}},s,a)\\\\\n&+\\frac{\\beta}{2}\\sum_{s,a} w(s,a)\\left[x(s,a) - \\left(Z_\\mu(V,h,x_{\\theta^{k}},s,a)\\right)\\right]^2.\n\\end{split}\n\\end{equation}","sha256_utf8":"06666e0b31300a6cdcd8b1d1f5f99723595b1902fd379df2f9676b22c9672a7e"},{"start":76618,"end":76924,"start_line":948,"end_line":948,"raw_tex":"Here, we use the same notation as \\eqref{comp-func}. It is easy to distinguish them according to variables. Let $\\tilde{V}^k, \\tilde{h}^k, \\tilde{x}^k = \\arg\\min_{h\\geq 0, V, x} \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$. The following lemma shows properties of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$. \n","sha256_utf8":"e0d9fae59ac683b022785ee5eab6362998c7ba1004a42dbd4de4357648f6e21d"},{"start":79220,"end":79703,"start_line":978,"end_line":983,"raw_tex":"\\begin{assumption}\\label{assum:fc}\n    \\begin{enumerate}[(i)]\n    \\item It holds that $\\tilde{V}^k \\in \\mathcal{F}^V_{R_V, m_V}, \\tilde{h}^k \\in \\mathcal{F}^h_{R_h, m_h}, \\tilde{x}^k \\in \\mathcal{F}^x_{R_x, m_x}$ for any $k$.\n    \\item For any unit vector $u, v$ and nonnegtive scalar $\\tau$, $E_w 1_{\\{|u^T(s,a)| < \\tau\\}} \\leq c\\tau$ and $E_w 1_{\\{|v^Ts| < \\tau\\}} \\leq c\\tau$ holds, where $c$ is a constant and $E_w f = \\sum_{s,a} w(s,a)f(s,a)$.  \n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"8f9ea213a2a60391c5643afe06b897d178d703a54ac405338b4fcdb72c03cfb3"},{"start":79704,"end":80358,"start_line":984,"end_line":991,"raw_tex":"Assumption \\ref{assum:fc} (i) is commonly used in the literature and Assumption \\ref{assum:fc} (ii) is on the regularity of the distribution $w$. For convenience, let $\\zeta = (\\phi, \\psi, \\theta)$, $\\Gamma = (V,h,x)$, $R = \\max\\{R_V, R_h, R_x\\}$ and $m = \\min\\{m_V, m_h, m_x\\}$. The following method is used to minimize \\eqref{comp-func}.\nLet $\\zeta^{k,0} = \\zeta^{k}$ and \n\\be\\label{eq:pjg}\n\\begin{split}\n    \\zeta^{k,t+1} &= \\Pi_{\\zeta \\in \\Xi}\\left(\\zeta^{k,t} - \\alpha_t \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}},h_{\\psi^{k,t}},x_{\\theta^{k,t}} )\\right), \\\\\n    \\zeta^{k+1} &= \\frac{1}{T}\\sum_{t=1}^T \\zeta^{k,t}.\n\\end{split}\n\\ee","sha256_utf8":"4f3ee3bd6268be7a389877249da8ee4a73f36e8496d8633b4198dff451d73b5f"},{"start":80359,"end":81249,"start_line":992,"end_line":998,"raw_tex":"The scheme \\eqref{eq:pjg} is actually a projected gradient descent method. Comparing with the algorithm in Section \\ref{The Augmented Lagrangian Method}, the scheme \\eqref{eq:pjg} has an additional projection and average operator, which is easy  for analysis. We also ignore the stochastic part, which is not hard to analyze by the method in \\cite{liu2019neural}.\nIn the following part, we show the convergence of the scheme \\eqref{eq:pjg}. Define the linearization function\n\\beaa\nV_{\\phi}^0 (s) &=& \\frac{1}{\\sqrt{m_V}} \\sum_{i=1}^{m_V} \\phi_{1,i} \\cdot 1_{\\{(\\phi_{2,i}^0)^Ts \\geq 0 \\}} \\phi_{2,i}^Ts, \\\\ \nh_{\\psi}^0 (s,a) &=& \\frac{1}{\\sqrt{m_h}} \\sum_{i=1}^{m_h} \\psi_{1,i} \\cdot 1_{\\{(\\psi_{2,i}^0)^T(s,a) \\geq 0 \\}} \\psi_{2,i}^T(s,a), \\\\\nx_{\\theta}^0 (s,a) &=& \\frac{1}{\\sqrt{m_x}} \\sum_{i=1}^{m_x} \\theta_{1,i} \\cdot 1_{\\{(\\theta_{2,i}^0)^T(s,a) \\geq 0 \\}} \\theta_{2,i}^T(s,a).\n\\eeaa","sha256_utf8":"c284730d9dd8792d6a6df967d9e12b1725a222141973f06b8d5a670547202d85"},{"start":97941,"end":98206,"start_line":1247,"end_line":1251,"raw_tex":"Let $\\mathcal{S}$ be the subgradient operator of $-\\sum_{s,a} r(s,a)w(s,a)x(s,a) + 1_\\Omega(x)$ with respect to $x$, where\n\\bee\n\\Omega = \\{x :  \\sum_{s,a} (\\delta_{s'}(s) - \\gamma P(s'|s,a)) w(s,a)x(s,a) = \\rho_0(s'), \\forall s', \nx(s,a) \\geq 0, \\forall s,a\\}.\n\\eee","sha256_utf8":"a689b04b65be95bdce1c91e2dd8be90605d538a482af85fa1521ff423f227477"},{"start":98207,"end":98413,"start_line":1252,"end_line":1255,"raw_tex":"Then $\\mathcal{S}$  is a maximal monotone operator. Lemma \\ref{lem:lp} means that \n\\[\nZ_\\mu(\\hat{V}^{k}, \\hat{h}^k, x_{\\theta^k}, s,a) = \\mathcal{T}x_{\\theta^k}:= (I + \\mu \\mathcal{S})^{-1}x_{\\theta^k} ,\n\\]","sha256_utf8":"b62a5d7d0caf2bcdf23614d15b8d4de0f1d5b01b1dfe5538e9d18edcbb12544b"},{"start":98414,"end":98509,"start_line":1256,"end_line":1256,"raw_tex":"where $\\hat{V}^{k},\\hat{h}^{k} = \\arg\\min_{h\\geq 0,V}  \\mathcal{L}_{\\mu}(V,h, x_{\\theta^k}) $.\n","sha256_utf8":"0c055b056108370c98b5a67a79922d020e986fa0aff20b473743f574c10fce44"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\n    Suppose that Assumption \\ref{assum:fc} holds and $\\beta > \\frac{1}{4\\mu}$. Let  $\\zeta^{k+1} = (\\phi^{k+1}, \\psi^{k+1}, \\theta^{k+1})$ be a sequence generated by \\eqref{eq:pjg}. \n%   Let $\\hat{V}^{k},\\hat{h}^{k} = \\arg\\min_{h\\geq 0,V}  \\mathcal{L}_{\\mu}(V,h, x_{\\theta^k}) $ and  \n%    $\n%        \\hat{x}^{k+1}(s,a) = [Z_\\mu(x_{\\theta^k}, \\hat{V}^{k}, \\hat{h}^k, s,a)].\n%    $\n    Then we have that\n    \\bee\n    \\begin{split}\n       & \\frac{1}{K} \\sum_{k=1}^K E_{0,w} \\left({x}_{\\theta^{k}}(s,a) - \\mathcal{T}{x}_{\\theta^{k}}(s,a)\\right)^2 \\\\\n        =&  O\\left(R^{3/2}T^{-1/2} + R^{7/4}m^{-1/8} + R^{2}m^{-1/4} + \\beta^{-1}R + K^{-1}R\\right),\n        \\end{split}\n    \\eee\n    and \n    \\bee\n\\begin{split}\n &  E_{0,w}\\left(x_{\\theta^{K}}(s,a) - \\mathcal{T}x_{\\theta^{K}}(s,a)\\right)^2  \\\\\n= &O\\left(KR^{3/2}T^{-1/2} + KR^{7/4}m^{-1/8} + KR^{2}m^{-1/4} + K\\beta^{-1}R+ K^{-1}R \\right).\n\\end{split}\n\\eee\n\\end{theorem}","units":[{"start":98626,"end":99560,"start_line":1258,"end_line":1278,"raw_tex":"\\begin{theorem}\n    Suppose that Assumption \\ref{assum:fc} holds and $\\beta > \\frac{1}{4\\mu}$. Let  $\\zeta^{k+1} = (\\phi^{k+1}, \\psi^{k+1}, \\theta^{k+1})$ be a sequence generated by \\eqref{eq:pjg}. \n%   Let $\\hat{V}^{k},\\hat{h}^{k} = \\arg\\min_{h\\geq 0,V}  \\mathcal{L}_{\\mu}(V,h, x_{\\theta^k}) $ and  \n%    $\n%        \\hat{x}^{k+1}(s,a) = [Z_\\mu(x_{\\theta^k}, \\hat{V}^{k}, \\hat{h}^k, s,a)].\n%    $\n    Then we have that\n    \\bee\n    \\begin{split}\n       & \\frac{1}{K} \\sum_{k=1}^K E_{0,w} \\left({x}_{\\theta^{k}}(s,a) - \\mathcal{T}{x}_{\\theta^{k}}(s,a)\\right)^2 \\\\\n        =&  O\\left(R^{3/2}T^{-1/2} + R^{7/4}m^{-1/8} + R^{2}m^{-1/4} + \\beta^{-1}R + K^{-1}R\\right),\n        \\end{split}\n    \\eee\n    and \n    \\bee\n\\begin{split}\n &  E_{0,w}\\left(x_{\\theta^{K}}(s,a) - \\mathcal{T}x_{\\theta^{K}}(s,a)\\right)^2  \\\\\n= &O\\left(KR^{3/2}T^{-1/2} + KR^{7/4}m^{-1/8} + KR^{2}m^{-1/4} + K\\beta^{-1}R+ K^{-1}R \\right).\n\\end{split}\n\\eee\n\\end{theorem}","sha256_utf8":"8673bdb552c5d1137ce6a5a315c1413ef705ce1eb901e8b9f160d92a99990dab"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\\label{lem:lp}\n    If $V^*,h^* = \\arg\\min_{h \\geq 0,V}  \\mathcal{L}_{\\mu}(V,h, x')$, then the term $Z_\\mu(V^*,h^*,x', s,a)$ is an optimal solution of the following problem\n\\begin{equation}\\label{eq:prox-lp}\n\\begin{split}\n\\max_x& \\sum_{s,a} r(s,a)w(s,a)x(s,a) - \\frac{1}{2\\mu}\\sum_{s,a} w(s,a)(x(s,a) - x'(s,a))^2  \\\\\n\\st&  \\sum_{s,a} (\\delta_{s'}(s) - \\gamma P(s'|s,a)) w(s,a)x(s,a) = \\rho_0(s'), \\forall s' \\\\\n&x(s,a) \\geq 0, \\forall s,a.\n\\end{split}\n\\end{equation}\nThe optimal value of problem \\eqref{eq:prox-lp} is equal to \n\\be\\label{eq:Fmu}\n\\mathcal{F}_{\\mu}(V^*,h^*,x') := \\mathcal{L}_{\\mu}(V^*,h^*, x') - \\frac{1}{2\\mu}\\sum_{s,a} w(s,a)x'(s,a)^2.\n\\ee\n\n\\end{lemma}\n\n\\begin{proof}\nThe KKT condition of the problem \\eqref{eq:prox-lp} can be written as  \n\\bea\n\\begin{split}\n&x(s,a) = x'(s,a) + \\mu \\left(h(s,a) + r(s,a)+\\sum_{s'} \\gamma P(s'|s,a) V(s') - V(s)\\right), \\forall s,a,\\\\\n&\\sum_{s,a} (\\delta_{s'}(s) - \\gamma P(s'|s,a)) w(s,a)x(s,a) = \\rho_0(s'), \\forall s', \\label{eq:kkt} \\\\\n&x(s,a) \\geq 0, h(s,a) \\geq 0, \\forall s,a,\n\\end{split}\n\\eea\nwhere $V$ and $h$ are the dual variables. Then, $x^*$ is an optimal solution if and only if there exists variables $V$ and $h$ such that $x^*, V$ and $h$ satisfy the condition \\eqref{eq:kkt}. Suppose that $V^*,h^* = \\arg\\min_{h\\geq 0,V}  \\mathcal{L}_{\\mu}(V,h, x')$. We have the optimality condition\n\\be\\label{eq:kkt-lg}\n    \\begin{split}\n    &\\rho_0(s') - \\sum_{s,a}w(s,a)Z_\\mu(V^*,h^*,x',s,a)(\\delta_{s'}(s) - \\gamma P(s'|s,a)) = 0, \\forall s' \\\\\n    & \\mu h(x,a) = \\left[- x'(s,a) - \\mu\\left(r(s,a)+\\sum_{s'} \\gamma P(s'|s,a) V(s') - V(s)\\right)\\right]_+.\n\\end{split}\n\\ee\nLet \n\\bee\n    \\begin{split}\n    x(s,a) &= Z_\\mu(V^*,h^*,x', s,a)\\\\ \n    &= \\left[x'(s,a) + \\mu \\left(r(s,a)+\\sum_{s'} \\gamma P(s'|s,a) V(s') - V(s)\\right)\\right]_+ \\geq 0.\n    \\end{split}\n\\eee\nThen the KKT condition \\eqref{eq:kkt} holds, i.e., $x(s,a) = Z_\\mu(V^*,h^*,x',s,a)$ is an optimal solution of \\eqref{eq:prox-lp}.\n\nLet $x^* = Z_\\mu(V^*,h^*,x',s,a)$, then we obtain\n\\bee\n\\begin{split}\n&\\mathcal{F}_{\\mu}(V^*,h^*,x') \\\\\n= & \\sum_s \\rho_0(s) V(s) + \\frac{1}{2\\mu}\\sum_{s,a}w(s,a) x^*(s,a)^2 - \\frac{1}{2\\mu}\\sum_{s,a} w(s,a)x'(s,a)^2 \\\\\n = & \\sum_s \\rho_0(s) V(s) + \\frac{1}{\\mu}\\sum_{s,a}w(s,a) x^*(s,a)(x^*(s,a)-x'(s,a)) \\\\\n& - \\frac{1}{2\\mu}\\sum_{s,a} w(s,a)(x^*(s,a)-x'(s,a))^2 \\\\\n= & \\sum_{s,a} r(s,a)w(s,a)x^*(s,a) - \\frac{1}{2\\mu}\\sum_{s,a} w(s,a)(x^*(s,a)-x'(s,a))^2, \n\\end{split}\n\\eee\nwhere the last inequality is due to the first equation in \\eqref{eq:kkt-lg} and the definition of $Z_\\mu(V^*,h^*,x',s,a)$, which completes the proof.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:cvx}\n    If $\\beta > \\frac{1}{4\\mu}$, the following claims hold.\n    \\begin{enumerate}[(i)]\n        \\item $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is a restricted strongly convex function with respect to $V,h,x$, i.e., there exits a constant $c$ such that\n        \\bee\n            \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^k, \\tilde{h}^k, \\tilde{x}^k) \\geq \\frac{c}{2}\\dist\\left((V,h,x),\\mathcal{X}^*\\right)^2,  \n        \\eee\n        where $\\mathcal{X}^*$ is the set of optimal solutions of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$.\n        The gradient of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is $L$-Lipschitz continuous. \n        \\item $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is a $\\frac{1}{2\\mu}(1-\\frac{1}{4\\mu\\beta})$-stongly convex function with respect to $x$ for fixed $V,h$.\n    \\end{enumerate}\n\\end{lemma}\n\n\\begin{proof}\n    The function $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is a quadratic function. Hence, the gradient of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is $L$-Lipschitz continuous. To show the convexity of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$, we only need to show that the quadratic part is nonnegtive.  If $\\beta > \\frac{1}{4\\mu}$, by rearranging the quadratic term in $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$, we obtain\n    \\bee\n \\begin{split}\n& \\frac{1}{2\\mu}\\sum_{s,a}w(s,a)x(s,a)Z_\\mu(V,h,x_{\\theta^{k}},s,a)\n+\\frac{\\beta}{2}\\sum_{s,a} w(s,a)\\left[x(s,a) - \\left(Z_\\mu(V,h,x_{\\theta^{k}},s,a)\\right)\\right]^2 \\\\\n=& \\frac{\\beta}{2}\\sum_{s,a}w(s,a) \\left\\{ \\left[(1-\\frac{1}{2\\mu \\beta})x(s,a) -  Z_\\mu(V,h,x_{\\theta^{k}},s,a) \\right]^2 + \\frac{1}{4\\mu\\beta}(1-\\frac{1}{4\\mu\\beta})x(s,a)^2 \\right\\} \\\\\n\\geq & \\frac{1}{2\\mu}(1-\\frac{1}{4\\mu\\beta})\\sum_{s,a}w(s,a)x(s,a)^2 \\geq 0.\n\\end{split}\n\\eee\nHence, $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is restricted strongly convex and $c$ is the smallest positive eigenvalue of the Hessian of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$. It also implies that $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is $\\frac{1}{2\\mu}(1-\\frac{1}{4\\mu\\beta})$-strongly convex with respect to $x$.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:lr}\n    Suppose that Assumption \\ref{assum:fc} holds and $\\zeta \\in \\Xi$. \n    Then we have the following estimation:\n    \\beaa\n    E_{0,w}(V_{\\phi}(s) - V_{\\phi}^0(s))^2 &=& O\\left(R_V^3m_V^{-1/2}\\right), \\\\\n    E_{0,w}(h_{\\psi}(s,a) - h_{\\psi}^0(s.a))^2 &=& O\\left(R_h^3m_h^{-1/2}\\right),\\\\\n    E_{0,w}(x_{\\theta}(s,a) - x_{\\theta}^0(s,a))^2 &=& O\\left(R_x^3m_x^{-1/2}\\right), \\\\\n     E_{0}\\left| \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) -  \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\right| &=& O\\left(R^3m^{-1/2} + R^2m^{-1/4}\\right), \\\\ \n     E_{0}\\|\\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) - \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_2^2 &=& O\\left(R^3m^{-1/2}\\right),\n     \\eeaa\n     where $E_{0,w}$ represent the expectation under the distribution \\eqref{eq:dist} and $w$.\n\\end{lemma}\n\n\\begin{proof}\nThe first three estimations follows from \\cite[Lemma D.1]{liu2019neural}. The\nmain difference is that $\\psi_{1,i}$ is always equal to $1$, but it does not change the arguments.  By Lemma \\ref{lem:cvx}, it yields\n\\be\\label{eq:est-g0}\n\\begin{split}\n&  \\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_{\\mathcal{H}}^2 \\\\\n\\leq &  2\\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2\n+ 2\\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) - \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2 \\\\\n\\leq & 2\\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2+ \n2L\\| (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) - (V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2 \\\\\n\\leq & 4L\\| (V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2+ \n4\\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(0,0,0)\\|_{\\mathcal{H}}^2\n+ 6LR,\n\\end{split}\n\\ee\nwhere the last inequality uses the bound of $V, h, x$ in \\eqref{eq:bv}.\nIt implies\n\\[\nE_0  \\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_{\\mathcal{H}}^2 = O\\left(R\\right).\n\\]\nIt follows from Lemma \\ref{lem:cvx} and \\cite[Theorem 18.15]{bauschke2011convex} that\n\\beaa\n&& E_{0}\\left| \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) -  \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\right| \\\\\n&\\leq & E_0\\left| \\iprod{ \\nabla_\\Gamma\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)}{(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)}_{\\mathcal{H}}\\right| \\\\\n&& + \\frac{L}{2}E_0  \\|(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|^2_{\\mathcal{H}} \\\\\n&\\leq & E_0 \\| \\nabla_\\Gamma\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|\\|(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_{\\mathcal{H}} \\\\\n&&+ \\frac{L}{2}E_0 \\|(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|^2_{\\mathcal{H}} \\\\\n&\\leq &  \\left(E_0\\| \\nabla_\\Gamma\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|^2_{\\mathcal{H}} \\right)^{\\frac{1}{2}}\n\\left( E_0\\|(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|^2_{\\mathcal{H}} \\right)^{\\frac{1}{2}}\\\\\n&&+ \\frac{L}{2}E_0 \\|(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|^2_{\\mathcal{H}} \\\\\n&=& O\\left(R^3m^{-1/2} + R^2m^{-1/4}\\right).\n\\eeaa\nWe next prove the last estimation. We can obtain\n\\beaa\n && E_{0}\\|\\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) - \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_2^2  \\\\\n &\\leq & 2E_{0}\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta})^T\\cdot \\left( \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) \n - \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \\right)\\|_2^2  \\\\\n && + 2E_{0}\\|\\left( \\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta}) \n - \\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \\right)^T\\cdot\\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_2^2  \\\\\n  &\\leq & 2E_{0}\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta})\\|_*^2\n  \\|\\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) \n - \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_\\mathcal{H}^2  \\\\\n && + 2E_{0}\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta}) - \\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_*^2\n \\|\\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_\\mathcal{H}^2,  \\\\\n &:=& I_1 + I_2,\n\\eeaa\nwhere the second inequality is due to the triangle inequality and the H\\\"older inequality and $\\|\\cdot\\|_*$ is defined by\n\\be\\label{eq:est-g1}\n\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta})\\|_*^2 = \\sum_{s,a}w(s,a)\\left[ \\|\\nabla_\\phi V_\\phi(s)\\|^2 \n+ \\|\\nabla_\\psi h_\\psi(s)\\|^2 + \\|\\nabla_\\theta x_\\theta(s)\\|^2\\right] \\leq 3.\n\\ee\nUsing the relationship \\eqref{eq:est-g1}, we get \n\\beaa\nI_1 &\\leq& 6E_{0}\\|\\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) \n - \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_\\mathcal{H}^2 \\\\\n &\\leq& 6L E_{0}\\|(V_{\\phi},h_{\\psi},x_{\\theta})  - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_\\mathcal{H}^2 \n =  O\\left(R^3m^{-1/2}\\right).\n\\eeaa\nAccording to the arguments in  \\cite[Lemma D.2]{liu2019neural}, we have\n\\be\\label{eq:est-g2}\n\\begin{split}\n&E_{0}\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta}) - \\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_*^2\n=  O\\left(Rm^{-1/2}\\right), \\\\\n&E_{0}\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta}) - \\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_*^2\n\\| (V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2 \n=  O\\left(Rm^{-1/2}\\right).\n\\end{split}\n\\ee\nCombining \\eqref{eq:est-g0} and \\eqref{eq:est-g2} yields\n\\beaa\nI_2 \n =  O\\left(R^2m^{-1/2}\\right).\n\\eeaa\n which completes the proof.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:conv1}\n    Suppose that Assumption \\ref{assum:fc} holds and $\\beta > \\frac{1}{4\\mu}$. Let  $\\zeta^{k,t} = (\\phi^{k,t}, \\psi^{k,t}, \\theta^{k,t})$ be a sequence generated by the projected gradient descent method \\eqref{eq:pjg} with $\\alpha_t = \\alpha \\leq \\frac{1}{\\sqrt{3}L}$.\n    Then we have\n    \\be\\label{eq:conv1}\n    \\begin{split}\n   & E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}},h_{\\psi^{k+1}},x_{\\theta^{k+1}}) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})\\right) \\\\\n    = &O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2}\\right).\n    \\end{split}\n    \\ee\n    Furthermore, it holds that\n    \\be\\label{eq:conv2}\n    E_{0,w}(x_{\\theta^{k+1}}(s,a) - \\tilde{x}^{k+1}(s,a))^2 = O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2}\\right). \n    \\ee\n%    \\beaa\n%        E_w(x_{\\theta^{k,T}}(s,a) - x_{\\theta^*}^0(s,a))^2 &=& O(R^3m^{-1/2}+R^2/T), \\\\\n%        E_w(Z_\\mu(V_{\\phi^{k,T}},h_{\\psi^{k,T}},x_{\\theta^k},s,a) - Z_\\mu(V_{\\phi^*},h_{\\psi^*},x_{\\theta^k},s,a))^2 &=& O(R^3m^{-1/2}+R^2/T).\n%    \\eeaa\n\\end{lemma}\n\n\\begin{proof}\n    Since $V_{\\phi}^0 (s), h_{\\psi}^0 (s,a), x_{\\theta}^0 (s,a)$ is linear with respect to $\\phi,\\psi,\\theta$, respectively, $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)$ is convex for $\\zeta$ by Lemma \\ref{lem:cvx}. We also have\n    \\be\n    \\begin{split}\n   &  \\| \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) - \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi'}^0,h_{\\psi'}^0,x_{\\theta'}^0)  \\|^2 \\\\\n    \\leq & \\|\\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)^T\\cdot \n    \\left( \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \n - \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi'}^0,h_{\\psi'}^0,x_{\\theta'}^0) \\right)\\|^2 \\\\\n     \\leq &\\|\\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \\|_*^2\n     \\| \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \n - \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi'}^0,h_{\\psi'}^0,x_{\\theta'}^0) \\|_\\mathcal{H}^2 \\\\\n \\leq & 3L^2 \\| (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \n - (V_{\\phi'}^0,h_{\\psi'}^0,x_{\\theta'}^0) \\|_\\mathcal{H}^2 \\leq 3L^2 \\| \\zeta - \\zeta' \\|_2^2.\n    \\end{split}\n    \\ee\n    Hence, $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)$ has  $\\sqrt{3}L$-Lipschitz continuous gradient. According to \\cite[Theorem 18.15]{bauschke2011convex}, it implies \n\\be\\label{eq:cvx1}\n\\begin{split}\n&\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t+1}}^0,h_{\\psi^{k,t+1}}^0,x_{\\theta^{k,t+1}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) \\\\\n\\leq & \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0)^T(\\zeta^{k,t+1} - \\zeta^{k,t} ) + \\frac{\\sqrt{3}L}{2}\\|\\zeta^{k,t+1} - \\zeta^{k,t} \\|_2^2.\n\\end{split}\n\\ee\nBy the convexity of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)$, we have\n\\be\\label{eq:cvx2}\n\\begin{split}\n    & \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{*}}^0,h_{\\psi^{*}}^0,x_{\\theta^{*}}^0)  \\\\\n\\leq & \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0)^T(\\zeta^{k,t} - \\zeta^{*} ),\n\\end{split}\n\\ee\nwhere $\\zeta^* = (\\phi^*,\\psi^*,\\theta^*)$ is an optimal solution, i.e., $(V_{\\phi^{*}}^0,h_{\\psi^{*}}^0,x_{\\theta^{*}}^0) = (\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})$.\nAccording to the property of projection operator, we get\n\\be\\label{eq:cvx3}\n({\\zeta^{*} - \\zeta^{k,t+1}})^T\\left(\\zeta^{k,t} - \\alpha_t \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}},h_{\\psi^{k,t}},x_{\\theta^{k,t}} ) - \\zeta^{k,t+1}\\right) \\geq 0\n\\ee\nCombining \\eqref{eq:cvx1}, \\eqref{eq:cvx2} and \\eqref{eq:cvx3} yields\n\\be\\label{eq:cvx4}\n\\begin{split}\n&\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t+1}}^0,h_{\\psi^{k,t+1}}^0,x_{\\theta^{k,t+1}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{*}}^0,h_{\\psi^{*}}^0,x_{\\theta^{*}}^0) \\\\\n\\leq & \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0)^T(\\zeta^{k,t+1} - \\zeta^{*} ) + \\frac{\\sqrt{3}L}{2}\\|\\zeta^{k,t+1} - \\zeta^{k,t} \\|_2^2 \\\\\n\\leq & \\frac{1}{\\alpha_t}(\\zeta^{k,t}  - \\zeta^{k,t+1})^T(\\zeta^{k,t+1} - \\zeta^{*} ) + \\frac{\\sqrt{3}L}{2}\\|\\zeta^{k,t+1} - \\zeta^{k,t} \\|_2^2 \\\\ \n&+ \\left(\\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) -  \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}},h_{\\psi^{k,t}},x_{\\theta^{k,t}} )  \\right)^T(\\zeta^{k,t+1} - \\zeta^{*} ) \\\\\n\\leq& \\frac{1}{\\alpha}\\left(\\|\\zeta^{k,t} - \\zeta^{*}\\|^2 - \\|\\zeta^{k,t+1} - \\zeta^{*}\\|^2\\right) \\\\\n&+ \\|\\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) -  \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}},h_{\\psi^{k,t}},x_{\\theta^{k,t}} )\\|\\|\\zeta^{k,t+1} - \\zeta^{*}\\|.\n\\end{split}\n\\ee\nBy the convexity of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t+1}}^0,h_{\\psi^{k,t+1}}^0,x_{\\theta^{k,t+1}}^0)$ and Lemma \\ref{lem:lr}, we obtain\n\\bee\n    \\begin{split}\n     & E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}}^0,h_{\\psi^{k+1}}^0,x_{\\theta^{k+1}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})\\right) \\\\\n     \\leq & \\frac{1}{T} \\sum_{t=1}^T E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})\\right) \\\\\n \\leq &  \\frac{4R}{T} \\sum_{t=1}^T \\left( E_{0}\\|\\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) -  \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}},h_{\\psi^{k,t}},x_{\\theta^{k,t}} )\\|^2\\right)^{\\frac{1}{2}}  \\\\\n &+ \\frac{1}{\\alpha T} \\|\\zeta^{k,0} - \\zeta^{*}\\|^2\\\\\n = & O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} \\right),\n\\end{split}\n\\eee\nwhere the second inequality is due to \\eqref{eq:cvx4} and H\\\"older inequality.\nIt follows from Lemma \\ref{lem:lr} that\n\\bee\n    \\begin{split}\n&E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}},h_{\\psi^{k+1}},x_{\\theta^{k+1}}) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})\\right) \\\\\n\\leq & E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}}^0,h_{\\psi^{k+1}}^0,x_{\\theta^{k+1}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})\\right) \\\\\n&+ E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}},h_{\\psi^{k+1}},x_{\\theta^{k+1}}) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}}^0,h_{\\psi^{k+1}}^0,x_{\\theta^{k+1}}^0)\\right) \\\\\n&= O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2}\\right),\n\\end{split}\n\\eee\nwhich completes the proof of \\eqref{eq:conv1}.\n\nBy Lemma \\ref{lem:cvx} (i), we have\n\\bee\n    E_0\\dist\\left(\\left(V_{\\phi^{k+1}},h_{\\psi^{k+1}},x_{\\theta^{k+1}}\\right),\\mathcal{X}^*\\right)^2 = O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2}\\right).\n\\eee\nIt means that there exists an optimal solution $(V^*, h^*, x^*) \\in \\mathcal{X}^*$ such that\n\\bee\n    E_{0,w}(x_{\\theta^{k+1}}(s,a) - {x}^*(s,a))^2 = O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2}\\right).\n\\eee\nBy Lemma \\ref{lem:cvx} (ii), $x^*$ is unique in $\\mathcal{X}^*$, i.e. $x^* = \\tilde{x}^{k+1}$, which completes the proof of \\eqref{eq:conv2}.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:err-2}\n    %Let $\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1} = \\arg\\min_{h\\geq 0,V, x} \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$. \n    Suppose that Assumption \\ref{assum:fc} holds and $\\beta > \\frac{1}{4\\mu}$.\n    Define $\\hat{V}^{k},\\hat{h}^{k} = \\arg\\min_{h\\geq 0,V}  \\mathcal{L}_{\\mu}(V,h, x_{\\theta^k})$ and let \n    $\n        \\hat{x}^{k+1}(s,a) = [Z_\\mu( \\hat{V}^{k}, \\hat{h}^k, x_{\\theta^k},s,a)].\n    $\n    Then we have\n    \\bea\n    Z_\\mu(\\tilde{V}^{k+1}, \\tilde{h}^k, x_{\\theta^{k}}, s,a) &=& \\xi_1 Z_\\mu( \\hat{V}^{k+1}, \\hat{h}^{k+1}, x_{\\theta^k},s,a), \\\\\n    \\tilde{x}^{k+1} &=& \\xi_2 \\hat{x}^{k+1},\n    \\eea\n    where $\\xi_1 = \\frac{4\\mu\\beta}{4\\mu\\beta-1}$ and $\\xi_2 = \\frac{4\\mu\\beta - 2}{4\\mu\\beta-1}$.\n    Furthermore, it implies\n    \\[\n        E_{0,w}(x_{\\theta^{k+1}}(s,a) - \\hat{x}^{k+1}(s,a))^2 = O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2} + \\beta^{-2}R\\right).\n    \\]\n\\end{lemma}\n\n\\begin{proof}\n    Let $x^*_{V,h} = \\arg\\min_x \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$. The first order optimality condition gives \n    \\[\n        \\frac{1}{2\\mu}Z_\\mu(V,h,x_{\\theta^{k}},s,a)\n        + \\beta\\left[x^*_{V,h}(s,a) - \\left(Z_\\mu(V,h,x_{\\theta^{k}},s,a)\\right)\\right] = 0.\n    \\]\n    It implies that \n    \\be\\label{eq:xvh}\n    x^*_{V,h}(s,a) = (1-\\frac{1}{2\\mu\\beta})Z_\\mu(V,h,x_{\\theta^{k}},s,a).\n\\ee\nDefine\n    \\[\n    \\mathcal{M}(V,h) := \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x^*_{V,h}) \n    = \\sum_{s} \\rho_0(s)V(s) +\\frac{1}{2\\mu\\xi_1}\\sum_{s,a}w(s,a)[Z_\\mu(V,h,x_{\\theta^{k}},s,a)]^2.\n    \\]\n    The optimality condition of  $\\min_{h \\geq 0, V} \\mathcal{M}(V,h)$ is\n\\bee\n    \\begin{split}\n        &\\xi_1 \\rho_0(s') - \\sum_{s,a}w(s,a)Z_\\mu(V^*,h^*,x',s,a)(\\delta_{s'}(s) - \\gamma P(s'|s,a)) = 0, \\forall s' \\\\\n    & \\mu h(x,a) = \\left[- x'(s,a) - \\mu\\left(r(s,a)+\\sum_{s'} \\gamma P(s'|s,a) V(s') - V(s)\\right)\\right]_+.\n\\end{split}\n\\eee\nTogether with \\eqref{eq:kkt-lg}, we have \n\\beaa\nZ_\\mu(\\tilde{V}^{k+1}, \\tilde{h}^k, x_{\\theta^{k}}, s,a) &=& \\xi_1 Z_\\mu(\\hat{V}^{k+1}, \\hat{h}^{k+1},x_{\\theta^k},  s,a). \n    \\eeaa\n    Combining with \\eqref{eq:xvh}, we have\n\\[\n    \\tilde{x}^{k+1}(s,a) = x^*_{\\tilde{V}^{k+1},\\tilde{h}^{k+1}}(s,a) = (1-\\frac{1}{2\\mu\\beta})Z_\\mu(\\tilde{V}^{k+1},\\tilde{h}^{k+1},x_{\\theta^{k}},s,a) = \\xi_2 \\hat{x}^{k+1}(s,a).\n\\]\nAccording to Lemma \\ref{lem:conv1}, we have\n\\beaa\n   &&  E_{0,w}(x_{\\theta^{k+1}}(s,a) - \\hat{x}^{k+1}(s,a))^2 \n\\\\\n   &\\leq &  2E_{0,w}(x_{\\theta^{k+1}}(s,a) - \\tilde{x}^{k+1}(s,a))^2 +  2E_{0,w}(\\tilde{x}^{k+1}(s,a) - \\hat{x}^{k+1}(s,a))^2 \\\\\n   &= &  2E_{0,w}(x_{\\theta^{k+1}}(s,a) - \\tilde{x}^{k+1}(s,a))^2 +  2(1-\\xi_2)^2E_{0,w}( \\hat{x}^{k+1}(s,a))^2 \\\\\n   & = & O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2} + \\beta^{-2}R\\right)\n.\n\\eeaa\nThis completes the proof of Lemma \\ref{lem:err-2}.\n\\end{proof}\n\n\\begin{proof}\nAccording to Lemmas \\ref{lem:lp} and \\ref{lem:err-2}, we have   \n\\bee\n    E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 = O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2} + \\beta^{-2}R\\right). \n\\eee\nLet $x^*$ be the solution of  the equation $x^* = \\mathcal{T}x^*$, i.e., the optimal solution of the LP \\eqref{eq:rllp-dual}. It follows from \\cite[Proposition 1]{rockafellar1976augmented} that \n\\bee\n\\begin{split}\n& E_{0,w}\\left(x^*(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 + E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n\\leq & E_{0,w}\\left(x_{\\theta^{k}}(s,a) - x^*(s,a)\\right)^2.\n\\end{split}\n\\eee\nIt implies\n\\bee\n\\begin{split}\n&  E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n\\leq & E_{0,w}\\left(x_{\\theta^{k}}(s,a) - x^*(s,a)\\right)^2 - E_{0,w}\\left(x^*(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n= & E_{0,w}\\left(x_{\\theta^{k}}(s,a) - x^*(s,a)\\right)^2  - E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - x^*(s,a)\\right)^2\\\\\n& +E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - x^*(s,a)\\right)^2 - E_{0,w}\\left(x^*(s,a) - Tx_{\\theta^{k}}(s,a)\\right)^2 \\\\\n= & E_{0,w}\\left(x_{\\theta^{k}}(s,a) - x^*(s,a)\\right)^2  - E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - x^*(s,a)\\right)^2\\\\\n& +E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)\\left(x_{\\theta^{k+1}}(s,a) +  \\mathcal{T}x_{\\theta^{k}}(s,a) - 2x^*(s,a)\\right) \\\\\n\\leq & E_{0,w}\\left(x_{\\theta^{k}}(s,a) - x^*(s,a)\\right)^2  - E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - x^*(s,a)\\right)^2\\\\\n& + cR^{1/2} \\cdot (E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2)^{\\frac{1}{2}}, \n\\end{split}\n\\eee\nwhere $c$ is a constant.\nTaking summation over $k$, we obtain\n\\bee\n\\begin{split}\n& \\frac{1}{K} \\sum_{k=1}^K E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\ \n\\leq &  \\frac{1}{K} E_{0,w}\\left(x_{\\theta^{1}}(s,a) - x^*(s,a)\\right)^2 + cR^{1/2} \\cdot \\frac{1}{K} \\sum_{k=1}^K  (E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2)^{\\frac{1}{2}} \\\\\n = & O\\left(R^{3/2}T^{-1/2} + R^{7/4}m^{-1/8} + R^{2}m^{-1/4} + \\beta^{-1}R + K^{-1}R\\right).\n \\end{split}\n\\eee\nFurthermore, by the same arguments, we have\n\\bee\n\\begin{split}\n& E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - \\mathcal{T}x_{\\theta^{k+1}}(s,a)\\right)^2 -  E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n= & O\\left(R^{3/2}T^{-1/2} + R^{7/4}m^{-1/8} + R^{2}m^{-1/4} + \\beta^{-1}R \\right).\n\\end{split}\n\\eee\nIt implies that\n\\bee\n\\begin{split}\n & K \\cdot E_{0,w}\\left(x_{\\theta^{K}}(s,a) - \\mathcal{T}x_{\\theta^{K}}(s,a)\\right)^2  \\\\\n\\leq & \\sum_{k=1}^K E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n&+  \\sum_{k=1}^K (K-k) \\cdot O\\left(R^{3/2}T^{-1/2} + R^{7/4}m^{-1/8} + R^{2}m^{-1/4} + \\beta^{-1}R \\right).\n\\end{split}\n\\eee\nHence, we have\n\\bee\n\\begin{split}\n &  E_{0,w}\\left(x_{\\theta^{K}}(s,a) - \\mathcal{T}x_{\\theta^{K}}(s,a)\\right)^2  \\\\\n\\leq & \\frac{1}{K}\\sum_{k=1}^K E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n& + K \\cdot O\\left(R^{3/2}T^{-1/2} + R^{7/4}m^{-1/8} + R^{2}m^{-1/4} + \\beta^{-1}R \\right) \\\\\n= & O\\left(KR^{3/2}T^{-1/2} + KR^{7/4}m^{-1/8} + KR^{2}m^{-1/4} + K\\beta^{-1}R+ K^{-1}R \\right),\n\\end{split}\n\\eee\nwhich completes the proof.\n\\end{proof}","units":[{"start":63355,"end":64038,"start_line":726,"end_line":740,"raw_tex":"\\begin{lemma}\\label{lem:lp}\n    If $V^*,h^* = \\arg\\min_{h \\geq 0,V}  \\mathcal{L}_{\\mu}(V,h, x')$, then the term $Z_\\mu(V^*,h^*,x', s,a)$ is an optimal solution of the following problem\n\\begin{equation}\\label{eq:prox-lp}\n\\begin{split}\n\\max_x& \\sum_{s,a} r(s,a)w(s,a)x(s,a) - \\frac{1}{2\\mu}\\sum_{s,a} w(s,a)(x(s,a) - x'(s,a))^2  \\\\\n\\st&  \\sum_{s,a} (\\delta_{s'}(s) - \\gamma P(s'|s,a)) w(s,a)x(s,a) = \\rho_0(s'), \\forall s' \\\\\n&x(s,a) \\geq 0, \\forall s,a.\n\\end{split}\n\\end{equation}\nThe optimal value of problem \\eqref{eq:prox-lp} is equal to \n\\be\\label{eq:Fmu}\n\\mathcal{F}_{\\mu}(V^*,h^*,x') := \\mathcal{L}_{\\mu}(V^*,h^*, x') - \\frac{1}{2\\mu}\\sum_{s,a} w(s,a)x'(s,a)^2.\n\\ee\n\n\\end{lemma}","sha256_utf8":"9f22d23093c4ae1344730cd6a7c3bb062a4b610961ab9ef4f4b8dd67e27c33c4"},{"start":64039,"end":65955,"start_line":741,"end_line":777,"raw_tex":"\\begin{proof}\nThe KKT condition of the problem \\eqref{eq:prox-lp} can be written as  \n\\bea\n\\begin{split}\n&x(s,a) = x'(s,a) + \\mu \\left(h(s,a) + r(s,a)+\\sum_{s'} \\gamma P(s'|s,a) V(s') - V(s)\\right), \\forall s,a,\\\\\n&\\sum_{s,a} (\\delta_{s'}(s) - \\gamma P(s'|s,a)) w(s,a)x(s,a) = \\rho_0(s'), \\forall s', \\label{eq:kkt} \\\\\n&x(s,a) \\geq 0, h(s,a) \\geq 0, \\forall s,a,\n\\end{split}\n\\eea\nwhere $V$ and $h$ are the dual variables. Then, $x^*$ is an optimal solution if and only if there exists variables $V$ and $h$ such that $x^*, V$ and $h$ satisfy the condition \\eqref{eq:kkt}. Suppose that $V^*,h^* = \\arg\\min_{h\\geq 0,V}  \\mathcal{L}_{\\mu}(V,h, x')$. We have the optimality condition\n\\be\\label{eq:kkt-lg}\n    \\begin{split}\n    &\\rho_0(s') - \\sum_{s,a}w(s,a)Z_\\mu(V^*,h^*,x',s,a)(\\delta_{s'}(s) - \\gamma P(s'|s,a)) = 0, \\forall s' \\\\\n    & \\mu h(x,a) = \\left[- x'(s,a) - \\mu\\left(r(s,a)+\\sum_{s'} \\gamma P(s'|s,a) V(s') - V(s)\\right)\\right]_+.\n\\end{split}\n\\ee\nLet \n\\bee\n    \\begin{split}\n    x(s,a) &= Z_\\mu(V^*,h^*,x', s,a)\\\\ \n    &= \\left[x'(s,a) + \\mu \\left(r(s,a)+\\sum_{s'} \\gamma P(s'|s,a) V(s') - V(s)\\right)\\right]_+ \\geq 0.\n    \\end{split}\n\\eee\nThen the KKT condition \\eqref{eq:kkt} holds, i.e., $x(s,a) = Z_\\mu(V^*,h^*,x',s,a)$ is an optimal solution of \\eqref{eq:prox-lp}.\n\nLet $x^* = Z_\\mu(V^*,h^*,x',s,a)$, then we obtain\n\\bee\n\\begin{split}\n&\\mathcal{F}_{\\mu}(V^*,h^*,x') \\\\\n= & \\sum_s \\rho_0(s) V(s) + \\frac{1}{2\\mu}\\sum_{s,a}w(s,a) x^*(s,a)^2 - \\frac{1}{2\\mu}\\sum_{s,a} w(s,a)x'(s,a)^2 \\\\\n = & \\sum_s \\rho_0(s) V(s) + \\frac{1}{\\mu}\\sum_{s,a}w(s,a) x^*(s,a)(x^*(s,a)-x'(s,a)) \\\\\n& - \\frac{1}{2\\mu}\\sum_{s,a} w(s,a)(x^*(s,a)-x'(s,a))^2 \\\\\n= & \\sum_{s,a} r(s,a)w(s,a)x^*(s,a) - \\frac{1}{2\\mu}\\sum_{s,a} w(s,a)(x^*(s,a)-x'(s,a))^2, \n\\end{split}\n\\eee\nwhere the last inequality is due to the first equation in \\eqref{eq:kkt-lg} and the definition of $Z_\\mu(V^*,h^*,x',s,a)$, which completes the proof.\n\\end{proof}","sha256_utf8":"ad6e0d1016971ebe208f0f35e9a512eea78a3260cf62a48b74e2616499dc6938"},{"start":76924,"end":77821,"start_line":949,"end_line":960,"raw_tex":"\\begin{lemma}\\label{lem:cvx}\n    If $\\beta > \\frac{1}{4\\mu}$, the following claims hold.\n    \\begin{enumerate}[(i)]\n        \\item $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is a restricted strongly convex function with respect to $V,h,x$, i.e., there exits a constant $c$ such that\n        \\bee\n            \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^k, \\tilde{h}^k, \\tilde{x}^k) \\geq \\frac{c}{2}\\dist\\left((V,h,x),\\mathcal{X}^*\\right)^2,  \n        \\eee\n        where $\\mathcal{X}^*$ is the set of optimal solutions of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$.\n        The gradient of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is $L$-Lipschitz continuous. \n        \\item $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is a $\\frac{1}{2\\mu}(1-\\frac{1}{4\\mu\\beta})$-stongly convex function with respect to $x$ for fixed $V,h$.\n    \\end{enumerate}\n\\end{lemma}","sha256_utf8":"63df3e5cf8291c53eccab0b5fe30591a2f74dd0e2f58d0b406db4d3546c68f73"},{"start":77822,"end":79084,"start_line":961,"end_line":972,"raw_tex":"\\begin{proof}\n    The function $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is a quadratic function. Hence, the gradient of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is $L$-Lipschitz continuous. To show the convexity of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$, we only need to show that the quadratic part is nonnegtive.  If $\\beta > \\frac{1}{4\\mu}$, by rearranging the quadratic term in $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$, we obtain\n    \\bee\n \\begin{split}\n& \\frac{1}{2\\mu}\\sum_{s,a}w(s,a)x(s,a)Z_\\mu(V,h,x_{\\theta^{k}},s,a)\n+\\frac{\\beta}{2}\\sum_{s,a} w(s,a)\\left[x(s,a) - \\left(Z_\\mu(V,h,x_{\\theta^{k}},s,a)\\right)\\right]^2 \\\\\n=& \\frac{\\beta}{2}\\sum_{s,a}w(s,a) \\left\\{ \\left[(1-\\frac{1}{2\\mu \\beta})x(s,a) -  Z_\\mu(V,h,x_{\\theta^{k}},s,a) \\right]^2 + \\frac{1}{4\\mu\\beta}(1-\\frac{1}{4\\mu\\beta})x(s,a)^2 \\right\\} \\\\\n\\geq & \\frac{1}{2\\mu}(1-\\frac{1}{4\\mu\\beta})\\sum_{s,a}w(s,a)x(s,a)^2 \\geq 0.\n\\end{split}\n\\eee\nHence, $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is restricted strongly convex and $c$ is the smallest positive eigenvalue of the Hessian of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$. It also implies that $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is $\\frac{1}{2\\mu}(1-\\frac{1}{4\\mu\\beta})$-strongly convex with respect to $x$.\n\\end{proof}","sha256_utf8":"35a4f9bf4040e6718d65b576d9666f115f093ed6c018522ad10523d9c5918c07"},{"start":81311,"end":82255,"start_line":1000,"end_line":1011,"raw_tex":"\\begin{lemma}\\label{lem:lr}\n    Suppose that Assumption \\ref{assum:fc} holds and $\\zeta \\in \\Xi$. \n    Then we have the following estimation:\n    \\beaa\n    E_{0,w}(V_{\\phi}(s) - V_{\\phi}^0(s))^2 &=& O\\left(R_V^3m_V^{-1/2}\\right), \\\\\n    E_{0,w}(h_{\\psi}(s,a) - h_{\\psi}^0(s.a))^2 &=& O\\left(R_h^3m_h^{-1/2}\\right),\\\\\n    E_{0,w}(x_{\\theta}(s,a) - x_{\\theta}^0(s,a))^2 &=& O\\left(R_x^3m_x^{-1/2}\\right), \\\\\n     E_{0}\\left| \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) -  \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\right| &=& O\\left(R^3m^{-1/2} + R^2m^{-1/4}\\right), \\\\ \n     E_{0}\\|\\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) - \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_2^2 &=& O\\left(R^3m^{-1/2}\\right),\n     \\eeaa\n     where $E_{0,w}$ represent the expectation under the distribution \\eqref{eq:dist} and $w$.\n\\end{lemma}","sha256_utf8":"26671a94969a922f10a7bf65fc52ca08cdc3555ab203ee063b3c235aeeaac5a1"},{"start":82256,"end":87414,"start_line":1012,"end_line":1086,"raw_tex":"\\begin{proof}\nThe first three estimations follows from \\cite[Lemma D.1]{liu2019neural}. The\nmain difference is that $\\psi_{1,i}$ is always equal to $1$, but it does not change the arguments.  By Lemma \\ref{lem:cvx}, it yields\n\\be\\label{eq:est-g0}\n\\begin{split}\n&  \\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_{\\mathcal{H}}^2 \\\\\n\\leq &  2\\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2\n+ 2\\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) - \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2 \\\\\n\\leq & 2\\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2+ \n2L\\| (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) - (V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2 \\\\\n\\leq & 4L\\| (V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2+ \n4\\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(0,0,0)\\|_{\\mathcal{H}}^2\n+ 6LR,\n\\end{split}\n\\ee\nwhere the last inequality uses the bound of $V, h, x$ in \\eqref{eq:bv}.\nIt implies\n\\[\nE_0  \\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_{\\mathcal{H}}^2 = O\\left(R\\right).\n\\]\nIt follows from Lemma \\ref{lem:cvx} and \\cite[Theorem 18.15]{bauschke2011convex} that\n\\beaa\n&& E_{0}\\left| \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) -  \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\right| \\\\\n&\\leq & E_0\\left| \\iprod{ \\nabla_\\Gamma\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)}{(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)}_{\\mathcal{H}}\\right| \\\\\n&& + \\frac{L}{2}E_0  \\|(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|^2_{\\mathcal{H}} \\\\\n&\\leq & E_0 \\| \\nabla_\\Gamma\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|\\|(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_{\\mathcal{H}} \\\\\n&&+ \\frac{L}{2}E_0 \\|(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|^2_{\\mathcal{H}} \\\\\n&\\leq &  \\left(E_0\\| \\nabla_\\Gamma\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|^2_{\\mathcal{H}} \\right)^{\\frac{1}{2}}\n\\left( E_0\\|(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|^2_{\\mathcal{H}} \\right)^{\\frac{1}{2}}\\\\\n&&+ \\frac{L}{2}E_0 \\|(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|^2_{\\mathcal{H}} \\\\\n&=& O\\left(R^3m^{-1/2} + R^2m^{-1/4}\\right).\n\\eeaa\nWe next prove the last estimation. We can obtain\n\\beaa\n && E_{0}\\|\\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) - \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_2^2  \\\\\n &\\leq & 2E_{0}\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta})^T\\cdot \\left( \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) \n - \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \\right)\\|_2^2  \\\\\n && + 2E_{0}\\|\\left( \\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta}) \n - \\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \\right)^T\\cdot\\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_2^2  \\\\\n  &\\leq & 2E_{0}\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta})\\|_*^2\n  \\|\\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) \n - \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_\\mathcal{H}^2  \\\\\n && + 2E_{0}\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta}) - \\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_*^2\n \\|\\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_\\mathcal{H}^2,  \\\\\n &:=& I_1 + I_2,\n\\eeaa\nwhere the second inequality is due to the triangle inequality and the H\\\"older inequality and $\\|\\cdot\\|_*$ is defined by\n\\be\\label{eq:est-g1}\n\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta})\\|_*^2 = \\sum_{s,a}w(s,a)\\left[ \\|\\nabla_\\phi V_\\phi(s)\\|^2 \n+ \\|\\nabla_\\psi h_\\psi(s)\\|^2 + \\|\\nabla_\\theta x_\\theta(s)\\|^2\\right] \\leq 3.\n\\ee\nUsing the relationship \\eqref{eq:est-g1}, we get \n\\beaa\nI_1 &\\leq& 6E_{0}\\|\\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) \n - \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_\\mathcal{H}^2 \\\\\n &\\leq& 6L E_{0}\\|(V_{\\phi},h_{\\psi},x_{\\theta})  - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_\\mathcal{H}^2 \n =  O\\left(R^3m^{-1/2}\\right).\n\\eeaa\nAccording to the arguments in  \\cite[Lemma D.2]{liu2019neural}, we have\n\\be\\label{eq:est-g2}\n\\begin{split}\n&E_{0}\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta}) - \\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_*^2\n=  O\\left(Rm^{-1/2}\\right), \\\\\n&E_{0}\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta}) - \\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_*^2\n\\| (V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2 \n=  O\\left(Rm^{-1/2}\\right).\n\\end{split}\n\\ee\nCombining \\eqref{eq:est-g0} and \\eqref{eq:est-g2} yields\n\\beaa\nI_2 \n =  O\\left(R^2m^{-1/2}\\right).\n\\eeaa\n which completes the proof.\n\\end{proof}","sha256_utf8":"841746001f9eb49ab7e7a1279e84ca07aad85a79b68b07faee78e5567fa4f223"},{"start":87485,"end":88579,"start_line":1089,"end_line":1106,"raw_tex":"\\begin{lemma}\\label{lem:conv1}\n    Suppose that Assumption \\ref{assum:fc} holds and $\\beta > \\frac{1}{4\\mu}$. Let  $\\zeta^{k,t} = (\\phi^{k,t}, \\psi^{k,t}, \\theta^{k,t})$ be a sequence generated by the projected gradient descent method \\eqref{eq:pjg} with $\\alpha_t = \\alpha \\leq \\frac{1}{\\sqrt{3}L}$.\n    Then we have\n    \\be\\label{eq:conv1}\n    \\begin{split}\n   & E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}},h_{\\psi^{k+1}},x_{\\theta^{k+1}}) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})\\right) \\\\\n    = &O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2}\\right).\n    \\end{split}\n    \\ee\n    Furthermore, it holds that\n    \\be\\label{eq:conv2}\n    E_{0,w}(x_{\\theta^{k+1}}(s,a) - \\tilde{x}^{k+1}(s,a))^2 = O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2}\\right). \n    \\ee\n%    \\beaa\n%        E_w(x_{\\theta^{k,T}}(s,a) - x_{\\theta^*}^0(s,a))^2 &=& O(R^3m^{-1/2}+R^2/T), \\\\\n%        E_w(Z_\\mu(V_{\\phi^{k,T}},h_{\\psi^{k,T}},x_{\\theta^k},s,a) - Z_\\mu(V_{\\phi^*},h_{\\psi^*},x_{\\theta^k},s,a))^2 &=& O(R^3m^{-1/2}+R^2/T).\n%    \\eeaa\n\\end{lemma}","sha256_utf8":"ed6d6d6eafc13c44d33afaa1d2471b30fd390e21dff6420916498fbf171b1fd3"},{"start":88580,"end":95000,"start_line":1107,"end_line":1183,"raw_tex":"\\begin{proof}\n    Since $V_{\\phi}^0 (s), h_{\\psi}^0 (s,a), x_{\\theta}^0 (s,a)$ is linear with respect to $\\phi,\\psi,\\theta$, respectively, $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)$ is convex for $\\zeta$ by Lemma \\ref{lem:cvx}. We also have\n    \\be\n    \\begin{split}\n   &  \\| \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) - \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi'}^0,h_{\\psi'}^0,x_{\\theta'}^0)  \\|^2 \\\\\n    \\leq & \\|\\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)^T\\cdot \n    \\left( \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \n - \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi'}^0,h_{\\psi'}^0,x_{\\theta'}^0) \\right)\\|^2 \\\\\n     \\leq &\\|\\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \\|_*^2\n     \\| \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \n - \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi'}^0,h_{\\psi'}^0,x_{\\theta'}^0) \\|_\\mathcal{H}^2 \\\\\n \\leq & 3L^2 \\| (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \n - (V_{\\phi'}^0,h_{\\psi'}^0,x_{\\theta'}^0) \\|_\\mathcal{H}^2 \\leq 3L^2 \\| \\zeta - \\zeta' \\|_2^2.\n    \\end{split}\n    \\ee\n    Hence, $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)$ has  $\\sqrt{3}L$-Lipschitz continuous gradient. According to \\cite[Theorem 18.15]{bauschke2011convex}, it implies \n\\be\\label{eq:cvx1}\n\\begin{split}\n&\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t+1}}^0,h_{\\psi^{k,t+1}}^0,x_{\\theta^{k,t+1}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) \\\\\n\\leq & \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0)^T(\\zeta^{k,t+1} - \\zeta^{k,t} ) + \\frac{\\sqrt{3}L}{2}\\|\\zeta^{k,t+1} - \\zeta^{k,t} \\|_2^2.\n\\end{split}\n\\ee\nBy the convexity of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)$, we have\n\\be\\label{eq:cvx2}\n\\begin{split}\n    & \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{*}}^0,h_{\\psi^{*}}^0,x_{\\theta^{*}}^0)  \\\\\n\\leq & \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0)^T(\\zeta^{k,t} - \\zeta^{*} ),\n\\end{split}\n\\ee\nwhere $\\zeta^* = (\\phi^*,\\psi^*,\\theta^*)$ is an optimal solution, i.e., $(V_{\\phi^{*}}^0,h_{\\psi^{*}}^0,x_{\\theta^{*}}^0) = (\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})$.\nAccording to the property of projection operator, we get\n\\be\\label{eq:cvx3}\n({\\zeta^{*} - \\zeta^{k,t+1}})^T\\left(\\zeta^{k,t} - \\alpha_t \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}},h_{\\psi^{k,t}},x_{\\theta^{k,t}} ) - \\zeta^{k,t+1}\\right) \\geq 0\n\\ee\nCombining \\eqref{eq:cvx1}, \\eqref{eq:cvx2} and \\eqref{eq:cvx3} yields\n\\be\\label{eq:cvx4}\n\\begin{split}\n&\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t+1}}^0,h_{\\psi^{k,t+1}}^0,x_{\\theta^{k,t+1}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{*}}^0,h_{\\psi^{*}}^0,x_{\\theta^{*}}^0) \\\\\n\\leq & \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0)^T(\\zeta^{k,t+1} - \\zeta^{*} ) + \\frac{\\sqrt{3}L}{2}\\|\\zeta^{k,t+1} - \\zeta^{k,t} \\|_2^2 \\\\\n\\leq & \\frac{1}{\\alpha_t}(\\zeta^{k,t}  - \\zeta^{k,t+1})^T(\\zeta^{k,t+1} - \\zeta^{*} ) + \\frac{\\sqrt{3}L}{2}\\|\\zeta^{k,t+1} - \\zeta^{k,t} \\|_2^2 \\\\ \n&+ \\left(\\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) -  \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}},h_{\\psi^{k,t}},x_{\\theta^{k,t}} )  \\right)^T(\\zeta^{k,t+1} - \\zeta^{*} ) \\\\\n\\leq& \\frac{1}{\\alpha}\\left(\\|\\zeta^{k,t} - \\zeta^{*}\\|^2 - \\|\\zeta^{k,t+1} - \\zeta^{*}\\|^2\\right) \\\\\n&+ \\|\\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) -  \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}},h_{\\psi^{k,t}},x_{\\theta^{k,t}} )\\|\\|\\zeta^{k,t+1} - \\zeta^{*}\\|.\n\\end{split}\n\\ee\nBy the convexity of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t+1}}^0,h_{\\psi^{k,t+1}}^0,x_{\\theta^{k,t+1}}^0)$ and Lemma \\ref{lem:lr}, we obtain\n\\bee\n    \\begin{split}\n     & E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}}^0,h_{\\psi^{k+1}}^0,x_{\\theta^{k+1}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})\\right) \\\\\n     \\leq & \\frac{1}{T} \\sum_{t=1}^T E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})\\right) \\\\\n \\leq &  \\frac{4R}{T} \\sum_{t=1}^T \\left( E_{0}\\|\\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) -  \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}},h_{\\psi^{k,t}},x_{\\theta^{k,t}} )\\|^2\\right)^{\\frac{1}{2}}  \\\\\n &+ \\frac{1}{\\alpha T} \\|\\zeta^{k,0} - \\zeta^{*}\\|^2\\\\\n = & O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} \\right),\n\\end{split}\n\\eee\nwhere the second inequality is due to \\eqref{eq:cvx4} and H\\\"older inequality.\nIt follows from Lemma \\ref{lem:lr} that\n\\bee\n    \\begin{split}\n&E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}},h_{\\psi^{k+1}},x_{\\theta^{k+1}}) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})\\right) \\\\\n\\leq & E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}}^0,h_{\\psi^{k+1}}^0,x_{\\theta^{k+1}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})\\right) \\\\\n&+ E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}},h_{\\psi^{k+1}},x_{\\theta^{k+1}}) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}}^0,h_{\\psi^{k+1}}^0,x_{\\theta^{k+1}}^0)\\right) \\\\\n&= O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2}\\right),\n\\end{split}\n\\eee\nwhich completes the proof of \\eqref{eq:conv1}.\n\nBy Lemma \\ref{lem:cvx} (i), we have\n\\bee\n    E_0\\dist\\left(\\left(V_{\\phi^{k+1}},h_{\\psi^{k+1}},x_{\\theta^{k+1}}\\right),\\mathcal{X}^*\\right)^2 = O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2}\\right).\n\\eee\nIt means that there exists an optimal solution $(V^*, h^*, x^*) \\in \\mathcal{X}^*$ such that\n\\bee\n    E_{0,w}(x_{\\theta^{k+1}}(s,a) - {x}^*(s,a))^2 = O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2}\\right).\n\\eee\nBy Lemma \\ref{lem:cvx} (ii), $x^*$ is unique in $\\mathcal{X}^*$, i.e. $x^* = \\tilde{x}^{k+1}$, which completes the proof of \\eqref{eq:conv2}.\n\\end{proof}","sha256_utf8":"764d111217e5a24ce60fdb2dac754f32ef68b51ca8c340fd0243a8e5c238bd13"},{"start":95123,"end":96062,"start_line":1186,"end_line":1203,"raw_tex":"\\begin{lemma}\\label{lem:err-2}\n    %Let $\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1} = \\arg\\min_{h\\geq 0,V, x} \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$. \n    Suppose that Assumption \\ref{assum:fc} holds and $\\beta > \\frac{1}{4\\mu}$.\n    Define $\\hat{V}^{k},\\hat{h}^{k} = \\arg\\min_{h\\geq 0,V}  \\mathcal{L}_{\\mu}(V,h, x_{\\theta^k})$ and let \n    $\n        \\hat{x}^{k+1}(s,a) = [Z_\\mu( \\hat{V}^{k}, \\hat{h}^k, x_{\\theta^k},s,a)].\n    $\n    Then we have\n    \\bea\n    Z_\\mu(\\tilde{V}^{k+1}, \\tilde{h}^k, x_{\\theta^{k}}, s,a) &=& \\xi_1 Z_\\mu( \\hat{V}^{k+1}, \\hat{h}^{k+1}, x_{\\theta^k},s,a), \\\\\n    \\tilde{x}^{k+1} &=& \\xi_2 \\hat{x}^{k+1},\n    \\eea\n    where $\\xi_1 = \\frac{4\\mu\\beta}{4\\mu\\beta-1}$ and $\\xi_2 = \\frac{4\\mu\\beta - 2}{4\\mu\\beta-1}$.\n    Furthermore, it implies\n    \\[\n        E_{0,w}(x_{\\theta^{k+1}}(s,a) - \\hat{x}^{k+1}(s,a))^2 = O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2} + \\beta^{-2}R\\right).\n    \\]\n\\end{lemma}","sha256_utf8":"0a4e31827d0222c329148e2507febd863d6612e36784f437f59557ebb4661665"},{"start":96064,"end":97939,"start_line":1205,"end_line":1245,"raw_tex":"\\begin{proof}\n    Let $x^*_{V,h} = \\arg\\min_x \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$. The first order optimality condition gives \n    \\[\n        \\frac{1}{2\\mu}Z_\\mu(V,h,x_{\\theta^{k}},s,a)\n        + \\beta\\left[x^*_{V,h}(s,a) - \\left(Z_\\mu(V,h,x_{\\theta^{k}},s,a)\\right)\\right] = 0.\n    \\]\n    It implies that \n    \\be\\label{eq:xvh}\n    x^*_{V,h}(s,a) = (1-\\frac{1}{2\\mu\\beta})Z_\\mu(V,h,x_{\\theta^{k}},s,a).\n\\ee\nDefine\n    \\[\n    \\mathcal{M}(V,h) := \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x^*_{V,h}) \n    = \\sum_{s} \\rho_0(s)V(s) +\\frac{1}{2\\mu\\xi_1}\\sum_{s,a}w(s,a)[Z_\\mu(V,h,x_{\\theta^{k}},s,a)]^2.\n    \\]\n    The optimality condition of  $\\min_{h \\geq 0, V} \\mathcal{M}(V,h)$ is\n\\bee\n    \\begin{split}\n        &\\xi_1 \\rho_0(s') - \\sum_{s,a}w(s,a)Z_\\mu(V^*,h^*,x',s,a)(\\delta_{s'}(s) - \\gamma P(s'|s,a)) = 0, \\forall s' \\\\\n    & \\mu h(x,a) = \\left[- x'(s,a) - \\mu\\left(r(s,a)+\\sum_{s'} \\gamma P(s'|s,a) V(s') - V(s)\\right)\\right]_+.\n\\end{split}\n\\eee\nTogether with \\eqref{eq:kkt-lg}, we have \n\\beaa\nZ_\\mu(\\tilde{V}^{k+1}, \\tilde{h}^k, x_{\\theta^{k}}, s,a) &=& \\xi_1 Z_\\mu(\\hat{V}^{k+1}, \\hat{h}^{k+1},x_{\\theta^k},  s,a). \n    \\eeaa\n    Combining with \\eqref{eq:xvh}, we have\n\\[\n    \\tilde{x}^{k+1}(s,a) = x^*_{\\tilde{V}^{k+1},\\tilde{h}^{k+1}}(s,a) = (1-\\frac{1}{2\\mu\\beta})Z_\\mu(\\tilde{V}^{k+1},\\tilde{h}^{k+1},x_{\\theta^{k}},s,a) = \\xi_2 \\hat{x}^{k+1}(s,a).\n\\]\nAccording to Lemma \\ref{lem:conv1}, we have\n\\beaa\n   &&  E_{0,w}(x_{\\theta^{k+1}}(s,a) - \\hat{x}^{k+1}(s,a))^2 \n\\\\\n   &\\leq &  2E_{0,w}(x_{\\theta^{k+1}}(s,a) - \\tilde{x}^{k+1}(s,a))^2 +  2E_{0,w}(\\tilde{x}^{k+1}(s,a) - \\hat{x}^{k+1}(s,a))^2 \\\\\n   &= &  2E_{0,w}(x_{\\theta^{k+1}}(s,a) - \\tilde{x}^{k+1}(s,a))^2 +  2(1-\\xi_2)^2E_{0,w}( \\hat{x}^{k+1}(s,a))^2 \\\\\n   & = & O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2} + \\beta^{-2}R\\right)\n.\n\\eeaa\nThis completes the proof of Lemma \\ref{lem:err-2}.\n\\end{proof}","sha256_utf8":"ef7f20f569755bd2d11f35e65fa632937a19121428dac185ba7ce361fa86fd53"},{"start":99561,"end":102906,"start_line":1279,"end_line":1338,"raw_tex":"\\begin{proof}\nAccording to Lemmas \\ref{lem:lp} and \\ref{lem:err-2}, we have   \n\\bee\n    E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 = O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2} + \\beta^{-2}R\\right). \n\\eee\nLet $x^*$ be the solution of  the equation $x^* = \\mathcal{T}x^*$, i.e., the optimal solution of the LP \\eqref{eq:rllp-dual}. It follows from \\cite[Proposition 1]{rockafellar1976augmented} that \n\\bee\n\\begin{split}\n& E_{0,w}\\left(x^*(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 + E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n\\leq & E_{0,w}\\left(x_{\\theta^{k}}(s,a) - x^*(s,a)\\right)^2.\n\\end{split}\n\\eee\nIt implies\n\\bee\n\\begin{split}\n&  E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n\\leq & E_{0,w}\\left(x_{\\theta^{k}}(s,a) - x^*(s,a)\\right)^2 - E_{0,w}\\left(x^*(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n= & E_{0,w}\\left(x_{\\theta^{k}}(s,a) - x^*(s,a)\\right)^2  - E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - x^*(s,a)\\right)^2\\\\\n& +E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - x^*(s,a)\\right)^2 - E_{0,w}\\left(x^*(s,a) - Tx_{\\theta^{k}}(s,a)\\right)^2 \\\\\n= & E_{0,w}\\left(x_{\\theta^{k}}(s,a) - x^*(s,a)\\right)^2  - E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - x^*(s,a)\\right)^2\\\\\n& +E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)\\left(x_{\\theta^{k+1}}(s,a) +  \\mathcal{T}x_{\\theta^{k}}(s,a) - 2x^*(s,a)\\right) \\\\\n\\leq & E_{0,w}\\left(x_{\\theta^{k}}(s,a) - x^*(s,a)\\right)^2  - E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - x^*(s,a)\\right)^2\\\\\n& + cR^{1/2} \\cdot (E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2)^{\\frac{1}{2}}, \n\\end{split}\n\\eee\nwhere $c$ is a constant.\nTaking summation over $k$, we obtain\n\\bee\n\\begin{split}\n& \\frac{1}{K} \\sum_{k=1}^K E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\ \n\\leq &  \\frac{1}{K} E_{0,w}\\left(x_{\\theta^{1}}(s,a) - x^*(s,a)\\right)^2 + cR^{1/2} \\cdot \\frac{1}{K} \\sum_{k=1}^K  (E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2)^{\\frac{1}{2}} \\\\\n = & O\\left(R^{3/2}T^{-1/2} + R^{7/4}m^{-1/8} + R^{2}m^{-1/4} + \\beta^{-1}R + K^{-1}R\\right).\n \\end{split}\n\\eee\nFurthermore, by the same arguments, we have\n\\bee\n\\begin{split}\n& E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - \\mathcal{T}x_{\\theta^{k+1}}(s,a)\\right)^2 -  E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n= & O\\left(R^{3/2}T^{-1/2} + R^{7/4}m^{-1/8} + R^{2}m^{-1/4} + \\beta^{-1}R \\right).\n\\end{split}\n\\eee\nIt implies that\n\\bee\n\\begin{split}\n & K \\cdot E_{0,w}\\left(x_{\\theta^{K}}(s,a) - \\mathcal{T}x_{\\theta^{K}}(s,a)\\right)^2  \\\\\n\\leq & \\sum_{k=1}^K E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n&+  \\sum_{k=1}^K (K-k) \\cdot O\\left(R^{3/2}T^{-1/2} + R^{7/4}m^{-1/8} + R^{2}m^{-1/4} + \\beta^{-1}R \\right).\n\\end{split}\n\\eee\nHence, we have\n\\bee\n\\begin{split}\n &  E_{0,w}\\left(x_{\\theta^{K}}(s,a) - \\mathcal{T}x_{\\theta^{K}}(s,a)\\right)^2  \\\\\n\\leq & \\frac{1}{K}\\sum_{k=1}^K E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n& + K \\cdot O\\left(R^{3/2}T^{-1/2} + R^{7/4}m^{-1/8} + R^{2}m^{-1/4} + \\beta^{-1}R \\right) \\\\\n= & O\\left(KR^{3/2}T^{-1/2} + KR^{7/4}m^{-1/8} + KR^{2}m^{-1/4} + K\\beta^{-1}R+ K^{-1}R \\right),\n\\end{split}\n\\eee\nwhich completes the proof.\n\\end{proof}","sha256_utf8":"e65e56a26cae5c65e22daf7d60dcc7233d1c94fea42fe7e985ef05e0663c5cbc"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"神经切线核残差界","source_units":[{"start":98626,"end":99560,"start_line":1258,"end_line":1278,"raw_tex":"\\begin{theorem}\n    Suppose that Assumption \\ref{assum:fc} holds and $\\beta > \\frac{1}{4\\mu}$. Let  $\\zeta^{k+1} = (\\phi^{k+1}, \\psi^{k+1}, \\theta^{k+1})$ be a sequence generated by \\eqref{eq:pjg}. \n%   Let $\\hat{V}^{k},\\hat{h}^{k} = \\arg\\min_{h\\geq 0,V}  \\mathcal{L}_{\\mu}(V,h, x_{\\theta^k}) $ and  \n%    $\n%        \\hat{x}^{k+1}(s,a) = [Z_\\mu(x_{\\theta^k}, \\hat{V}^{k}, \\hat{h}^k, s,a)].\n%    $\n    Then we have that\n    \\bee\n    \\begin{split}\n       & \\frac{1}{K} \\sum_{k=1}^K E_{0,w} \\left({x}_{\\theta^{k}}(s,a) - \\mathcal{T}{x}_{\\theta^{k}}(s,a)\\right)^2 \\\\\n        =&  O\\left(R^{3/2}T^{-1/2} + R^{7/4}m^{-1/8} + R^{2}m^{-1/4} + \\beta^{-1}R + K^{-1}R\\right),\n        \\end{split}\n    \\eee\n    and \n    \\bee\n\\begin{split}\n &  E_{0,w}\\left(x_{\\theta^{K}}(s,a) - \\mathcal{T}x_{\\theta^{K}}(s,a)\\right)^2  \\\\\n= &O\\left(KR^{3/2}T^{-1/2} + KR^{7/4}m^{-1/8} + KR^{2}m^{-1/4} + K\\beta^{-1}R+ K^{-1}R \\right).\n\\end{split}\n\\eee\n\\end{theorem}","sha256_utf8":"8673bdb552c5d1137ce6a5a315c1413ef705ce1eb901e8b9f160d92a99990dab"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"主定理完整证明","source_units":[{"start":99561,"end":102906,"start_line":1279,"end_line":1338,"raw_tex":"\\begin{proof}\nAccording to Lemmas \\ref{lem:lp} and \\ref{lem:err-2}, we have   \n\\bee\n    E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 = O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2} + \\beta^{-2}R\\right). \n\\eee\nLet $x^*$ be the solution of  the equation $x^* = \\mathcal{T}x^*$, i.e., the optimal solution of the LP \\eqref{eq:rllp-dual}. It follows from \\cite[Proposition 1]{rockafellar1976augmented} that \n\\bee\n\\begin{split}\n& E_{0,w}\\left(x^*(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 + E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n\\leq & E_{0,w}\\left(x_{\\theta^{k}}(s,a) - x^*(s,a)\\right)^2.\n\\end{split}\n\\eee\nIt implies\n\\bee\n\\begin{split}\n&  E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n\\leq & E_{0,w}\\left(x_{\\theta^{k}}(s,a) - x^*(s,a)\\right)^2 - E_{0,w}\\left(x^*(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n= & E_{0,w}\\left(x_{\\theta^{k}}(s,a) - x^*(s,a)\\right)^2  - E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - x^*(s,a)\\right)^2\\\\\n& +E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - x^*(s,a)\\right)^2 - E_{0,w}\\left(x^*(s,a) - Tx_{\\theta^{k}}(s,a)\\right)^2 \\\\\n= & E_{0,w}\\left(x_{\\theta^{k}}(s,a) - x^*(s,a)\\right)^2  - E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - x^*(s,a)\\right)^2\\\\\n& +E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)\\left(x_{\\theta^{k+1}}(s,a) +  \\mathcal{T}x_{\\theta^{k}}(s,a) - 2x^*(s,a)\\right) \\\\\n\\leq & E_{0,w}\\left(x_{\\theta^{k}}(s,a) - x^*(s,a)\\right)^2  - E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - x^*(s,a)\\right)^2\\\\\n& + cR^{1/2} \\cdot (E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2)^{\\frac{1}{2}}, \n\\end{split}\n\\eee\nwhere $c$ is a constant.\nTaking summation over $k$, we obtain\n\\bee\n\\begin{split}\n& \\frac{1}{K} \\sum_{k=1}^K E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\ \n\\leq &  \\frac{1}{K} E_{0,w}\\left(x_{\\theta^{1}}(s,a) - x^*(s,a)\\right)^2 + cR^{1/2} \\cdot \\frac{1}{K} \\sum_{k=1}^K  (E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2)^{\\frac{1}{2}} \\\\\n = & O\\left(R^{3/2}T^{-1/2} + R^{7/4}m^{-1/8} + R^{2}m^{-1/4} + \\beta^{-1}R + K^{-1}R\\right).\n \\end{split}\n\\eee\nFurthermore, by the same arguments, we have\n\\bee\n\\begin{split}\n& E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - \\mathcal{T}x_{\\theta^{k+1}}(s,a)\\right)^2 -  E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n= & O\\left(R^{3/2}T^{-1/2} + R^{7/4}m^{-1/8} + R^{2}m^{-1/4} + \\beta^{-1}R \\right).\n\\end{split}\n\\eee\nIt implies that\n\\bee\n\\begin{split}\n & K \\cdot E_{0,w}\\left(x_{\\theta^{K}}(s,a) - \\mathcal{T}x_{\\theta^{K}}(s,a)\\right)^2  \\\\\n\\leq & \\sum_{k=1}^K E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n&+  \\sum_{k=1}^K (K-k) \\cdot O\\left(R^{3/2}T^{-1/2} + R^{7/4}m^{-1/8} + R^{2}m^{-1/4} + \\beta^{-1}R \\right).\n\\end{split}\n\\eee\nHence, we have\n\\bee\n\\begin{split}\n &  E_{0,w}\\left(x_{\\theta^{K}}(s,a) - \\mathcal{T}x_{\\theta^{K}}(s,a)\\right)^2  \\\\\n\\leq & \\frac{1}{K}\\sum_{k=1}^K E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n& + K \\cdot O\\left(R^{3/2}T^{-1/2} + R^{7/4}m^{-1/8} + R^{2}m^{-1/4} + \\beta^{-1}R \\right) \\\\\n= & O\\left(KR^{3/2}T^{-1/2} + KR^{7/4}m^{-1/8} + KR^{2}m^{-1/4} + K\\beta^{-1}R+ K^{-1}R \\right),\n\\end{split}\n\\eee\nwhich completes the proof.\n\\end{proof}","sha256_utf8":"e65e56a26cae5c65e22daf7d60dcc7233d1c94fea42fe7e985ef05e0663c5cbc"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"有限折扣决策过程","source_units":[{"start":22019,"end":22739,"start_line":244,"end_line":244,"raw_tex":"We consider the standard reinforcement learning setting characterized by an infinite-horizon discounted MDP. Formally, a MDP is described by the tuple $(\\mathcal{S},\\mathcal{A},P,r,\\rho_0, \\gamma)$. We assume that $\\mathcal{S}$ and $\\mathcal{A}$ represent the finite state and action spaces, respectively. $P: \\mathcal{S}\\times \\mathcal{A}\\times \\mathcal{S}\\rightarrow \\mathbf{R}$ indicates the transition dynamics by a conditional distribution over the state and action spaces, $r: \\mathcal{S} \\times \\mathcal{A} \\rightarrow \\mathbf{R}$ is the bounded reward function. The initial state $s_0$ of the MDP obeys the distribution $\\rho_0: \\mathcal{S}\\rightarrow \\mathbf{R}$, and $\\gamma \\in (0,1)$ is the discount factor.\n","sha256_utf8":"97f9b3d29d225955491af8cc5fbbcc07daac862a90307306e313221261678de3"}]},{"id":"S004","semantic_type":"problem_setting","kind":"problem_setting","title":"原始对偶线性规划","source_units":[{"start":24144,"end":24352,"start_line":270,"end_line":275,"raw_tex":"\\begin{equation}\\label{eq:rllp-primal}\n\\begin{split}\n\\mathrm{(P):}\\quad\\min_V& \\sum_{s} \\rho_0(s)V(s), \\\\\n\\st & V(s)\\geq  r(s,a)+  \\gamma \\mathbb{E}_{s'|s,a}[ V(s')] , \\forall s, a.\n\\end{split}\n\\end{equation}","sha256_utf8":"59e65272ab4229d80d52c2abd10397c2a334ef0cb84b763d568dafde63946605"},{"start":24952,"end":25192,"start_line":285,"end_line":290,"raw_tex":"\\begin{equation}\\label{eq:rllp-dual}\n\\begin{split}\n\\mathrm{(D):}\\quad\\max_{x\\geq0}& \\sum_{s,a} r(s,a)w(s,a)x(s,a), \\\\\n\\st& \\sum_{a}w(s',a)x(s',a)- \\sum_{s,a} \\gamma P(s'|s,a) w(s,a)x(s,a) = \\rho_0(s'), \\forall s',\n\\end{split}\n\\end{equation}","sha256_utf8":"1aaf23244b5c348d541b32e4ab4e2f5e499ece63bc1fd54cdab29e634274f1b7"},{"start":25193,"end":25845,"start_line":291,"end_line":295,"raw_tex":"where the weight function $w$ is a distribution with respect to the state and action spaces, i.e., \n\\bee\n\\sum\\limits_{s\\in\\mathcal{S},a\\in\\mathcal{A}}w(s,a)=1.\n\\eee\nThe only difference from the standard dual formulation is that the variables $x(s,a)$ carries the weight $w(s,a)$. Actually, $x$ can be viewed as the multipliers using a weighted inner product other than the standard case in Euclidean space. Moreover, the weight function $w(s,a)$ enables us to perform a sampling process from a replay buffer in large and complex environments. In practice, the explicit form of $w(s,a)$ is not required as it simply characterizes the sampling behavior.\n","sha256_utf8":"f9673b85d6851f2a81426b385f853b065c0fbd208601c212f5ae68fd395ae656"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"非负松弛变量","source_units":[{"start":26845,"end":27260,"start_line":312,"end_line":318,"raw_tex":"To simplify the following derivation, we consider an equivalent primal LP formulation with equality constraints by introducing an additional slack variable. Namely, we reformulate \\eqref{eq:rllp-primal} as \n\\begin{equation}\\label{eq:rllp-primal-eq}\n\\begin{split}\n\\min_{h\\geq0,V}&\\  \\sum_{s} \\rho_0(s)V(s), \\\\\n\\st & V(s)=  r(s,a)+  \\gamma  \\mathbb{E}_{s'|s,a}[ V(s')]+h(s,a), \\forall s, a.\n\\end{split}\n\\end{equation}","sha256_utf8":"56d810e727e3c52666fe3843c20341dc71c19af1b1b44c14f4332cb504ae1ee5"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"乘子更新与条件期望","source_units":[{"start":27517,"end":27775,"start_line":326,"end_line":329,"raw_tex":"We denote $x(s,a)$ as the multiplier of the constraints in \\eqref{eq:rllp-primal-eq}. For simplicity, we introduce a $Z$-function as\n\\bee\nZ_\\mu(V,h,x,s,a) =  x(s,a)+  \\mu \\left(h(s,a)+r(s,a)+\\gamma \\mathbb{E}_{s'|s,a}[ V(s')] - V(s)\\right),\\label{zfunc}\n\\eee","sha256_utf8":"616c0a057aa9009d1caaf3b9f2c44b37c73413a2c48f68dc24581fbae84f415c"},{"start":27776,"end":27894,"start_line":330,"end_line":333,"raw_tex":"and define\n\\bee\nz_\\mu(V,h,x,s,a,s') =  x(s,a)+  \\mu \\left(h(s,a)+r(s,a)+\\gamma V(s') - V(s)\\right),\\label{zfunc1}\n\\eee","sha256_utf8":"da40aa622cfd1f2439163f5013c124fa0296e4f1fc2f10f75920b01b253f09cb"},{"start":27895,"end":27990,"start_line":334,"end_line":337,"raw_tex":"where $\\mu>0$ is the penalty parameter. Since \n\\bea\\label{sumP=1}\n\\sum_{s'} P(s'|s,a) = 1,\n\\eea","sha256_utf8":"5e560d047e427aee5b96bca869b35897c64382db4dd64f5feb9bb1f29e4281b2"},{"start":27991,"end":28081,"start_line":338,"end_line":341,"raw_tex":"we have\n\\bea\\label{Z-z}\nZ_\\mu(V,h,x,s,a) = \\mathbb{E}_{s'|s,a} [z_\\mu(V,h,x,s,a,s')].\n\\eea","sha256_utf8":"f2fb5d7249a2450b46120daf635dd03cb392d5ad25efeefbb783c9c58d8e46ad"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"加权增广拉格朗日函数","source_units":[{"start":28082,"end":28402,"start_line":342,"end_line":345,"raw_tex":"That is to say, $Z$-function is a conditional expectation of $s'$. For the LP problem \\eqref{eq:rllp-primal-eq}, the weighted augmented Lagrangian function is defined as \n\\begin{equation}\\label{randALM}\n\\mathcal{L}_{\\mu}(V,h,x) = \\sum_{s} \\rho_0(s)V(s) +\\frac{1}{2\\mu}\\sum_{s,a}w(s,a)[Z_\\mu(V,h,x,s,a)]^2.\n\\end{equation}","sha256_utf8":"a123bccf03a06e999fb475a8f15de8150ac4094c4dbdfcf1a8d6b1224c8a922a"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"双层神经网络","source_units":[{"start":73112,"end":73509,"start_line":894,"end_line":899,"raw_tex":"We next parametrize the variables by the following two-layer neural networks:\n\\bea\nV_{\\phi} (s) &=& \\frac{1}{\\sqrt{m_V}} \\sum_{i=1}^{m_V} \\phi_{1,i} \\cdot \\sigma(\\phi_{2,i}^Ts), \\\\ \nh_{\\psi} (s,a) &=& \\frac{1}{\\sqrt{m_h}} \\sum_{i=1}^{m_h} \\psi_{1,i} \\cdot \\sigma(\\psi_{2,i}^T(s,a)), \\\\\nx_{\\theta} (s,a) &=& \\frac{1}{\\sqrt{m_x}} \\sum_{i=1}^{m_x} \\theta_{1,i} \\cdot \\sigma(\\theta_{2,i}^T(s,a)),\n\\eea","sha256_utf8":"e96f6106b36dcb80904af12e29e4ff9722643b06ce6f30bc0b1fa9975cdb4b07"},{"start":73510,"end":73696,"start_line":900,"end_line":900,"raw_tex":"where $m_V, m_h, m_x$ are integers, $\\sigma =  \\max(0, x)$ is the ReLU function. The parameters are randomly initialized by either the uniform distribution or normal distribution, i.e.,\n","sha256_utf8":"cf8b1394cd303a1cc95b3797742d5251cbe3e397282f26407bcd51c776a55d31"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"随机初始化与训练参数","source_units":[{"start":73696,"end":73941,"start_line":901,"end_line":906,"raw_tex":"\\be\\label{eq:dist}\n\\begin{split}\n&\\phi_{1,i}, \\theta_{1,i} \\sim \\mathcal{U}(\\{1,-1\\}), \\quad \\psi_{1,i} = 1, \\\\\n&\\phi_{2,i} \\sim \\mathcal{N}(0, I_{d_{s}}/d_s),\\quad \\psi_{2,i},\\theta_{1,i} \\sim \\mathcal{N}(0, I_{d_{sa}}/d_{sa}), \n\\end{split}\n\\ee","sha256_utf8":"2e00229375184cd470dbd389b1c36fcc50a995cc017ee263498b43cf6b96f5a5"},{"start":73942,"end":74448,"start_line":907,"end_line":910,"raw_tex":"where $d_{s}$ is the dimension of $s$ and $d_{sa}$ is the dimension of $(s,a)$. \nIn particular, $\\phi_{1,i}, \\psi_{1,i}, \\theta_{1,i}$ are fixed and we only optimize the parameter $\\phi_{2,i}, \\psi_{2,i}, \\theta_{2,i}$.\nLet $\\phi = (\\phi_{2,i})_{i=1}^{m_V}$, $\\psi = (\\psi_{2,i})_{i=1}^{m_V}$ and $\\theta = (\\theta_{2,i})_{i=1}^{m_V}$.\nThe corresponding random initialization parameters are denoted by $\\phi^0, \\psi^0, \\theta^0$. The expectation under the distribution \\eqref{eq:dist} is denoted by $E_0$.\n","sha256_utf8":"98d2065d644f937a99ee0f1c0bc7978c3cbf66604887c5aa045fbdbe7f2c04c9"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"线性函数类与参数域","source_units":[{"start":74449,"end":75026,"start_line":912,"end_line":917,"raw_tex":"Define the following function classes \n\\beaa\n\\mathcal{F}^V_{R_V, m_V} &=& \\{\\frac{1}{\\sqrt{m_V}} \\sum_{i=1}^{m_V} \\phi_{1,i} \\cdot 1_{\\{(\\phi_{2,i}^0)^Ts \\geq 0 \\}} \\phi_{2,i}^Ts : \\ \\|\\phi-\\phi^0\\| \\leq R_V\\},\\\\\n\\mathcal{F}^h_{R_h, m_h} &=& \\{\\frac{1}{\\sqrt{m_h}} \\sum_{i=1}^{m_h} \\psi_{1,i} \\cdot 1_{\\{(\\psi_{2,i}^0)^T(s,a) \\geq 0 \\}} \\psi_{2,i}^T(s,a): \\ \\|\\psi-\\psi^0\\| \\leq R_h\\}, \\\\\n\\mathcal{F}^x_{R_x, m_x} &=& \\{\\frac{1}{\\sqrt{m_h}} \\sum_{i=1}^{m_h} \\theta_{1,i} \\cdot 1_{\\{(\\theta_{2,i}^0)^T(s,a) \\geq 0 \\}} \\theta_{2,i}^T(s,a): \\ \\|\\theta-\\theta^0\\| \\leq R_x\\},\n\\eeaa","sha256_utf8":"28e7152ee1ab77a0eeb066fabb736398e7e78ba34c220a05b5e3a54fcc6c2ca0"},{"start":75027,"end":75156,"start_line":918,"end_line":921,"raw_tex":"and let \n\\bee\n\\Xi = \\{\\zeta: \\|\\phi-\\phi^0\\| \\leq R_V, \\|\\psi-\\psi^0\\| \\leq R_h \\text{ and } \\|\\theta-\\theta^0\\| \\leq R_x\\}.\n\\eee","sha256_utf8":"a653351dd5ba409e497ce38f44edf32d692116b1f6735c2f009269e8531361aa"}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","title":"输入范数有界","source_units":[{"start":75157,"end":75262,"start_line":922,"end_line":922,"raw_tex":"For simplicity, we assume  that $\\|(s,a)\\| \\leq 1$, which follows that $\\|s\\| \\leq 1$. On $\\Xi$, we have\n","sha256_utf8":"2931d76a343043743aad8e8fcd05f7c6ad84047ad06412684af49c0cb857d4ac"}]},{"id":"S012","semantic_type":"derived_fact","kind":"derived_fact","title":"网络偏移与梯度界","source_units":[{"start":75262,"end":75646,"start_line":923,"end_line":928,"raw_tex":"\\be\\label{eq:bv}\n\\begin{split}\n&|V_\\phi(s) - V_{\\phi^0}(s)| \\leq \\frac{1}{\\sqrt{m_V}} \\sum_{i=1}^{m_V} |\\phi_{1,i}| \\cdot |\\sigma(\\phi_{2,i}^Ts) - \\sigma((\\phi_{2,i}^0)^Ts)| \\\\\n \\leq& \\frac{1}{\\sqrt{m_V}} \\sum_{i=1}^{m_V} |\\phi_{1,i}| \\cdot |\\phi_{2,i}^Ts - (\\phi_{2,i}^0)^Ts| \\leq \\frac{1}{\\sqrt{m_V}} \\sum_{i=1}^{m_V} \\|\\phi_{2,i} - (\\phi_{2,i}^0)\\| \\leq R_V^{1/2}.\n \\end{split}\n\\ee","sha256_utf8":"1726360829ddc609b44b982a15b7105b79b43e0df223a8e14641f276fe9318c2"},{"start":75647,"end":75916,"start_line":929,"end_line":933,"raw_tex":"Similarly, we have $|h_\\psi(s) - h_{\\psi^0}(s)| \\leq R_h^{1/2}$ and $|x_\\theta(s) - x_{\\theta^0}(s)| \\leq R_x^{1/2}$.\nWe can also obtain that\n\\beaa\n\\|\\nabla_\\phi V_\\phi(s) \\|^2 = \\frac{1}{m_V} \\sum_{i=1}^{m_V} 1_{\\{(\\phi_{2,i})^Ts \\geq 0 \\}} \\cdot \\|s\\|^2 \\leq 1.\n\\eeaa","sha256_utf8":"98aec0bfc3419e7b311be47fa3add0510b940fa7f179bc0960adb8443c0dacba"},{"start":75917,"end":76019,"start_line":934,"end_line":934,"raw_tex":"We also yields $\\|\\nabla_\\psi h_\\psi(s,a) \\| \\leq  1$ and $\\|\\nabla_\\theta x_\\theta(s,a) \\| \\leq  1$.\n","sha256_utf8":"fbbca7a06bab374a7d023c95fd82209661b710afd67246053632009b2e26af3b"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"加权函数空间","source_units":[{"start":76019,"end":76208,"start_line":935,"end_line":938,"raw_tex":"Consider that $(V,h,x)$ is in the Hilbert space $\\mathcal{H}$ with inner product\n\\[\n\\iprod{(V,h,x)}{(V',h',x')}_{\\mathcal{H}} = \\sum_{s,a}w(s,a)\\left(V(s)^2+ x(s,a)^2 + h(s,a)^2 \\right).\n\\]","sha256_utf8":"edccc09ab7f33e176e51611d482b170b1d638d0ade31d4da5ca205ba46c5197a"},{"start":76209,"end":76290,"start_line":939,"end_line":939,"raw_tex":"The corresponding norm in $\\mathcal{H}$ is denoted by $\\|\\cdot\\|_{\\mathcal{H}}$.\n","sha256_utf8":"ba92af0adce7ba0b3373ddc4cb1ad298e9c021e387a376c88069f6e0de704434"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"罚函数与最优点","source_units":[{"start":76290,"end":76617,"start_line":940,"end_line":947,"raw_tex":"Define\n\\begin{equation}\\label{comp-func-non}\n \\begin{split}\n \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x) \n&:= \\sum_{s} \\rho_0(s)V(s) +\\frac{1}{2\\mu}\\sum_{s,a}w(s,a)x(s,a)Z_\\mu(V,h,x_{\\theta^{k}},s,a)\\\\\n&+\\frac{\\beta}{2}\\sum_{s,a} w(s,a)\\left[x(s,a) - \\left(Z_\\mu(V,h,x_{\\theta^{k}},s,a)\\right)\\right]^2.\n\\end{split}\n\\end{equation}","sha256_utf8":"88a92b4f319791603e4f3498d89e3342825344d1bcb598971311c977453bc756"},{"start":76618,"end":76924,"start_line":948,"end_line":948,"raw_tex":"Here, we use the same notation as \\eqref{comp-func}. It is easy to distinguish them according to variables. Let $\\tilde{V}^k, \\tilde{h}^k, \\tilde{x}^k = \\arg\\min_{h\\geq 0, V, x} \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$. The following lemma shows properties of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$. \n","sha256_utf8":"e0d9fae59ac683b022785ee5eab6362998c7ba1004a42dbd4de4357648f6e21d"}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","title":"可表示性与分布正则性","source_units":[{"start":79220,"end":79703,"start_line":978,"end_line":983,"raw_tex":"\\begin{assumption}\\label{assum:fc}\n    \\begin{enumerate}[(i)]\n    \\item It holds that $\\tilde{V}^k \\in \\mathcal{F}^V_{R_V, m_V}, \\tilde{h}^k \\in \\mathcal{F}^h_{R_h, m_h}, \\tilde{x}^k \\in \\mathcal{F}^x_{R_x, m_x}$ for any $k$.\n    \\item For any unit vector $u, v$ and nonnegtive scalar $\\tau$, $E_w 1_{\\{|u^T(s,a)| < \\tau\\}} \\leq c\\tau$ and $E_w 1_{\\{|v^Ts| < \\tau\\}} \\leq c\\tau$ holds, where $c$ is a constant and $E_w f = \\sum_{s,a} w(s,a)f(s,a)$.  \n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"8f9ea213a2a60391c5643afe06b897d178d703a54ac405338b4fcdb72c03cfb3"}]},{"id":"S016","semantic_type":"notation","kind":"notation","title":"联合参数与尺度","source_units":[{"start":79704,"end":80044,"start_line":984,"end_line":984,"raw_tex":"Assumption \\ref{assum:fc} (i) is commonly used in the literature and Assumption \\ref{assum:fc} (ii) is on the regularity of the distribution $w$. For convenience, let $\\zeta = (\\phi, \\psi, \\theta)$, $\\Gamma = (V,h,x)$, $R = \\max\\{R_V, R_h, R_x\\}$ and $m = \\min\\{m_V, m_h, m_x\\}$. The following method is used to minimize \\eqref{comp-func}.\n","sha256_utf8":"8c204055435211685db7a4de43ce8b68fdc5ff2b9f14e892a6105ce0c4e4cd06"}]},{"id":"S017","semantic_type":"algorithm","kind":"algorithm","title":"投影梯度与迭代平均","source_units":[{"start":80044,"end":80358,"start_line":985,"end_line":991,"raw_tex":"Let $\\zeta^{k,0} = \\zeta^{k}$ and \n\\be\\label{eq:pjg}\n\\begin{split}\n    \\zeta^{k,t+1} &= \\Pi_{\\zeta \\in \\Xi}\\left(\\zeta^{k,t} - \\alpha_t \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}},h_{\\psi^{k,t}},x_{\\theta^{k,t}} )\\right), \\\\\n    \\zeta^{k+1} &= \\frac{1}{T}\\sum_{t=1}^T \\zeta^{k,t}.\n\\end{split}\n\\ee","sha256_utf8":"9ac53900a5ffa7e6f6dfb744ad63acbb595bd709725e9e789c209094288deb11"},{"start":80359,"end":80834,"start_line":992,"end_line":993,"raw_tex":"The scheme \\eqref{eq:pjg} is actually a projected gradient descent method. Comparing with the algorithm in Section \\ref{The Augmented Lagrangian Method}, the scheme \\eqref{eq:pjg} has an additional projection and average operator, which is easy  for analysis. We also ignore the stochastic part, which is not hard to analyze by the method in \\cite{liu2019neural}.\nIn the following part, we show the convergence of the scheme \\eqref{eq:pjg}. Define the linearization function\n","sha256_utf8":"7edfb9ef283801b9f575e7fa42a1776dec836f657e5b59b0a58c2e24a67daa51"}]},{"id":"S018","semantic_type":"definition","kind":"definition","title":"冻结激活线性化","source_units":[{"start":80834,"end":81249,"start_line":994,"end_line":998,"raw_tex":"\\beaa\nV_{\\phi}^0 (s) &=& \\frac{1}{\\sqrt{m_V}} \\sum_{i=1}^{m_V} \\phi_{1,i} \\cdot 1_{\\{(\\phi_{2,i}^0)^Ts \\geq 0 \\}} \\phi_{2,i}^Ts, \\\\ \nh_{\\psi}^0 (s,a) &=& \\frac{1}{\\sqrt{m_h}} \\sum_{i=1}^{m_h} \\psi_{1,i} \\cdot 1_{\\{(\\psi_{2,i}^0)^T(s,a) \\geq 0 \\}} \\psi_{2,i}^T(s,a), \\\\\nx_{\\theta}^0 (s,a) &=& \\frac{1}{\\sqrt{m_x}} \\sum_{i=1}^{m_x} \\theta_{1,i} \\cdot 1_{\\{(\\theta_{2,i}^0)^T(s,a) \\geq 0 \\}} \\theta_{2,i}^T(s,a).\n\\eeaa","sha256_utf8":"cc94156d1cc99dc2648c46ec1d9520e352ee86777feba2ecde98d58ccafbc47a"}]},{"id":"S019","semantic_type":"definition","kind":"definition","title":"对偶次梯度与预解式","source_units":[{"start":97941,"end":98206,"start_line":1247,"end_line":1251,"raw_tex":"Let $\\mathcal{S}$ be the subgradient operator of $-\\sum_{s,a} r(s,a)w(s,a)x(s,a) + 1_\\Omega(x)$ with respect to $x$, where\n\\bee\n\\Omega = \\{x :  \\sum_{s,a} (\\delta_{s'}(s) - \\gamma P(s'|s,a)) w(s,a)x(s,a) = \\rho_0(s'), \\forall s', \nx(s,a) \\geq 0, \\forall s,a\\}.\n\\eee","sha256_utf8":"a689b04b65be95bdce1c91e2dd8be90605d538a482af85fa1521ff423f227477"},{"start":98207,"end":98413,"start_line":1252,"end_line":1255,"raw_tex":"Then $\\mathcal{S}$  is a maximal monotone operator. Lemma \\ref{lem:lp} means that \n\\[\nZ_\\mu(\\hat{V}^{k}, \\hat{h}^k, x_{\\theta^k}, s,a) = \\mathcal{T}x_{\\theta^k}:= (I + \\mu \\mathcal{S})^{-1}x_{\\theta^k} ,\n\\]","sha256_utf8":"b62a5d7d0caf2bcdf23614d15b8d4de0f1d5b01b1dfe5538e9d18edcbb12544b"},{"start":98414,"end":98509,"start_line":1256,"end_line":1256,"raw_tex":"where $\\hat{V}^{k},\\hat{h}^{k} = \\arg\\min_{h\\geq 0,V}  \\mathcal{L}_{\\mu}(V,h, x_{\\theta^k}) $.\n","sha256_utf8":"0c055b056108370c98b5a67a79922d020e986fa0aff20b473743f574c10fce44"}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","title":"增广更新的近端表示","source_units":[{"start":63355,"end":64038,"start_line":726,"end_line":740,"raw_tex":"\\begin{lemma}\\label{lem:lp}\n    If $V^*,h^* = \\arg\\min_{h \\geq 0,V}  \\mathcal{L}_{\\mu}(V,h, x')$, then the term $Z_\\mu(V^*,h^*,x', s,a)$ is an optimal solution of the following problem\n\\begin{equation}\\label{eq:prox-lp}\n\\begin{split}\n\\max_x& \\sum_{s,a} r(s,a)w(s,a)x(s,a) - \\frac{1}{2\\mu}\\sum_{s,a} w(s,a)(x(s,a) - x'(s,a))^2  \\\\\n\\st&  \\sum_{s,a} (\\delta_{s'}(s) - \\gamma P(s'|s,a)) w(s,a)x(s,a) = \\rho_0(s'), \\forall s' \\\\\n&x(s,a) \\geq 0, \\forall s,a.\n\\end{split}\n\\end{equation}\nThe optimal value of problem \\eqref{eq:prox-lp} is equal to \n\\be\\label{eq:Fmu}\n\\mathcal{F}_{\\mu}(V^*,h^*,x') := \\mathcal{L}_{\\mu}(V^*,h^*, x') - \\frac{1}{2\\mu}\\sum_{s,a} w(s,a)x'(s,a)^2.\n\\ee\n\n\\end{lemma}","sha256_utf8":"9f22d23093c4ae1344730cd6a7c3bb062a4b610961ab9ef4f4b8dd67e27c33c4"}]},{"id":"S021","semantic_type":"proof","kind":"proof","title":"近端表示完整证明","source_units":[{"start":64039,"end":65955,"start_line":741,"end_line":777,"raw_tex":"\\begin{proof}\nThe KKT condition of the problem \\eqref{eq:prox-lp} can be written as  \n\\bea\n\\begin{split}\n&x(s,a) = x'(s,a) + \\mu \\left(h(s,a) + r(s,a)+\\sum_{s'} \\gamma P(s'|s,a) V(s') - V(s)\\right), \\forall s,a,\\\\\n&\\sum_{s,a} (\\delta_{s'}(s) - \\gamma P(s'|s,a)) w(s,a)x(s,a) = \\rho_0(s'), \\forall s', \\label{eq:kkt} \\\\\n&x(s,a) \\geq 0, h(s,a) \\geq 0, \\forall s,a,\n\\end{split}\n\\eea\nwhere $V$ and $h$ are the dual variables. Then, $x^*$ is an optimal solution if and only if there exists variables $V$ and $h$ such that $x^*, V$ and $h$ satisfy the condition \\eqref{eq:kkt}. Suppose that $V^*,h^* = \\arg\\min_{h\\geq 0,V}  \\mathcal{L}_{\\mu}(V,h, x')$. We have the optimality condition\n\\be\\label{eq:kkt-lg}\n    \\begin{split}\n    &\\rho_0(s') - \\sum_{s,a}w(s,a)Z_\\mu(V^*,h^*,x',s,a)(\\delta_{s'}(s) - \\gamma P(s'|s,a)) = 0, \\forall s' \\\\\n    & \\mu h(x,a) = \\left[- x'(s,a) - \\mu\\left(r(s,a)+\\sum_{s'} \\gamma P(s'|s,a) V(s') - V(s)\\right)\\right]_+.\n\\end{split}\n\\ee\nLet \n\\bee\n    \\begin{split}\n    x(s,a) &= Z_\\mu(V^*,h^*,x', s,a)\\\\ \n    &= \\left[x'(s,a) + \\mu \\left(r(s,a)+\\sum_{s'} \\gamma P(s'|s,a) V(s') - V(s)\\right)\\right]_+ \\geq 0.\n    \\end{split}\n\\eee\nThen the KKT condition \\eqref{eq:kkt} holds, i.e., $x(s,a) = Z_\\mu(V^*,h^*,x',s,a)$ is an optimal solution of \\eqref{eq:prox-lp}.\n\nLet $x^* = Z_\\mu(V^*,h^*,x',s,a)$, then we obtain\n\\bee\n\\begin{split}\n&\\mathcal{F}_{\\mu}(V^*,h^*,x') \\\\\n= & \\sum_s \\rho_0(s) V(s) + \\frac{1}{2\\mu}\\sum_{s,a}w(s,a) x^*(s,a)^2 - \\frac{1}{2\\mu}\\sum_{s,a} w(s,a)x'(s,a)^2 \\\\\n = & \\sum_s \\rho_0(s) V(s) + \\frac{1}{\\mu}\\sum_{s,a}w(s,a) x^*(s,a)(x^*(s,a)-x'(s,a)) \\\\\n& - \\frac{1}{2\\mu}\\sum_{s,a} w(s,a)(x^*(s,a)-x'(s,a))^2 \\\\\n= & \\sum_{s,a} r(s,a)w(s,a)x^*(s,a) - \\frac{1}{2\\mu}\\sum_{s,a} w(s,a)(x^*(s,a)-x'(s,a))^2, \n\\end{split}\n\\eee\nwhere the last inequality is due to the first equation in \\eqref{eq:kkt-lg} and the definition of $Z_\\mu(V^*,h^*,x',s,a)$, which completes the proof.\n\\end{proof}","sha256_utf8":"ad6e0d1016971ebe208f0f35e9a512eea78a3260cf62a48b74e2616499dc6938"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"近端问题最优性条件","source_units":[{"start":64053,"end":64719,"start_line":742,"end_line":750,"raw_tex":"The KKT condition of the problem \\eqref{eq:prox-lp} can be written as  \n\\bea\n\\begin{split}\n&x(s,a) = x'(s,a) + \\mu \\left(h(s,a) + r(s,a)+\\sum_{s'} \\gamma P(s'|s,a) V(s') - V(s)\\right), \\forall s,a,\\\\\n&\\sum_{s,a} (\\delta_{s'}(s) - \\gamma P(s'|s,a)) w(s,a)x(s,a) = \\rho_0(s'), \\forall s', \\label{eq:kkt} \\\\\n&x(s,a) \\geq 0, h(s,a) \\geq 0, \\forall s,a,\n\\end{split}\n\\eea\nwhere $V$ and $h$ are the dual variables. Then, $x^*$ is an optimal solution if and only if there exists variables $V$ and $h$ such that $x^*, V$ and $h$ satisfy the condition \\eqref{eq:kkt}. Suppose that $V^*,h^* = \\arg\\min_{h\\geq 0,V}  \\mathcal{L}_{\\mu}(V,h, x')$. We have the optimality condition\n","sha256_utf8":"df1f8696f723613ba0cfdfb39a20d4155eefb98443068682a8658c040fc5cfcf"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"增广最优点满足近端条件","source_units":[{"start":64719,"end":65317,"start_line":751,"end_line":764,"raw_tex":"\\be\\label{eq:kkt-lg}\n    \\begin{split}\n    &\\rho_0(s') - \\sum_{s,a}w(s,a)Z_\\mu(V^*,h^*,x',s,a)(\\delta_{s'}(s) - \\gamma P(s'|s,a)) = 0, \\forall s' \\\\\n    & \\mu h(x,a) = \\left[- x'(s,a) - \\mu\\left(r(s,a)+\\sum_{s'} \\gamma P(s'|s,a) V(s') - V(s)\\right)\\right]_+.\n\\end{split}\n\\ee\nLet \n\\bee\n    \\begin{split}\n    x(s,a) &= Z_\\mu(V^*,h^*,x', s,a)\\\\ \n    &= \\left[x'(s,a) + \\mu \\left(r(s,a)+\\sum_{s'} \\gamma P(s'|s,a) V(s') - V(s)\\right)\\right]_+ \\geq 0.\n    \\end{split}\n\\eee\nThen the KKT condition \\eqref{eq:kkt} holds, i.e., $x(s,a) = Z_\\mu(V^*,h^*,x',s,a)$ is an optimal solution of \\eqref{eq:prox-lp}.\n","sha256_utf8":"7e4f6fd7c85058bae10b897e06c9812810f5fdc696966e5134950d4ad9bf9c68"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"计算近端最优值","source_units":[{"start":65318,"end":65944,"start_line":766,"end_line":776,"raw_tex":"Let $x^* = Z_\\mu(V^*,h^*,x',s,a)$, then we obtain\n\\bee\n\\begin{split}\n&\\mathcal{F}_{\\mu}(V^*,h^*,x') \\\\\n= & \\sum_s \\rho_0(s) V(s) + \\frac{1}{2\\mu}\\sum_{s,a}w(s,a) x^*(s,a)^2 - \\frac{1}{2\\mu}\\sum_{s,a} w(s,a)x'(s,a)^2 \\\\\n = & \\sum_s \\rho_0(s) V(s) + \\frac{1}{\\mu}\\sum_{s,a}w(s,a) x^*(s,a)(x^*(s,a)-x'(s,a)) \\\\\n& - \\frac{1}{2\\mu}\\sum_{s,a} w(s,a)(x^*(s,a)-x'(s,a))^2 \\\\\n= & \\sum_{s,a} r(s,a)w(s,a)x^*(s,a) - \\frac{1}{2\\mu}\\sum_{s,a} w(s,a)(x^*(s,a)-x'(s,a))^2, \n\\end{split}\n\\eee\nwhere the last inequality is due to the first equation in \\eqref{eq:kkt-lg} and the definition of $Z_\\mu(V^*,h^*,x',s,a)$, which completes the proof.\n","sha256_utf8":"ff8ad6e77c217a61e28f102cd70e29a595ab68792dd9c0abc739a84ec75466bc"}]},{"id":"S025","semantic_type":"lemma","kind":"lemma","title":"罚函数凸性与光滑性","source_units":[{"start":76924,"end":77821,"start_line":949,"end_line":960,"raw_tex":"\\begin{lemma}\\label{lem:cvx}\n    If $\\beta > \\frac{1}{4\\mu}$, the following claims hold.\n    \\begin{enumerate}[(i)]\n        \\item $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is a restricted strongly convex function with respect to $V,h,x$, i.e., there exits a constant $c$ such that\n        \\bee\n            \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^k, \\tilde{h}^k, \\tilde{x}^k) \\geq \\frac{c}{2}\\dist\\left((V,h,x),\\mathcal{X}^*\\right)^2,  \n        \\eee\n        where $\\mathcal{X}^*$ is the set of optimal solutions of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$.\n        The gradient of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is $L$-Lipschitz continuous. \n        \\item $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is a $\\frac{1}{2\\mu}(1-\\frac{1}{4\\mu\\beta})$-stongly convex function with respect to $x$ for fixed $V,h$.\n    \\end{enumerate}\n\\end{lemma}","sha256_utf8":"63df3e5cf8291c53eccab0b5fe30591a2f74dd0e2f58d0b406db4d3546c68f73"}]},{"id":"S026","semantic_type":"proof","kind":"proof","title":"罚函数性质完整证明","source_units":[{"start":77822,"end":79084,"start_line":961,"end_line":972,"raw_tex":"\\begin{proof}\n    The function $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is a quadratic function. Hence, the gradient of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is $L$-Lipschitz continuous. To show the convexity of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$, we only need to show that the quadratic part is nonnegtive.  If $\\beta > \\frac{1}{4\\mu}$, by rearranging the quadratic term in $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$, we obtain\n    \\bee\n \\begin{split}\n& \\frac{1}{2\\mu}\\sum_{s,a}w(s,a)x(s,a)Z_\\mu(V,h,x_{\\theta^{k}},s,a)\n+\\frac{\\beta}{2}\\sum_{s,a} w(s,a)\\left[x(s,a) - \\left(Z_\\mu(V,h,x_{\\theta^{k}},s,a)\\right)\\right]^2 \\\\\n=& \\frac{\\beta}{2}\\sum_{s,a}w(s,a) \\left\\{ \\left[(1-\\frac{1}{2\\mu \\beta})x(s,a) -  Z_\\mu(V,h,x_{\\theta^{k}},s,a) \\right]^2 + \\frac{1}{4\\mu\\beta}(1-\\frac{1}{4\\mu\\beta})x(s,a)^2 \\right\\} \\\\\n\\geq & \\frac{1}{2\\mu}(1-\\frac{1}{4\\mu\\beta})\\sum_{s,a}w(s,a)x(s,a)^2 \\geq 0.\n\\end{split}\n\\eee\nHence, $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is restricted strongly convex and $c$ is the smallest positive eigenvalue of the Hessian of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$. It also implies that $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is $\\frac{1}{2\\mu}(1-\\frac{1}{4\\mu\\beta})$-strongly convex with respect to $x$.\n\\end{proof}","sha256_utf8":"35a4f9bf4040e6718d65b576d9666f115f093ed6c018522ad10523d9c5918c07"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"二次项配方与非负性","source_units":[{"start":77836,"end":78743,"start_line":962,"end_line":970,"raw_tex":"    The function $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is a quadratic function. Hence, the gradient of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is $L$-Lipschitz continuous. To show the convexity of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$, we only need to show that the quadratic part is nonnegtive.  If $\\beta > \\frac{1}{4\\mu}$, by rearranging the quadratic term in $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$, we obtain\n    \\bee\n \\begin{split}\n& \\frac{1}{2\\mu}\\sum_{s,a}w(s,a)x(s,a)Z_\\mu(V,h,x_{\\theta^{k}},s,a)\n+\\frac{\\beta}{2}\\sum_{s,a} w(s,a)\\left[x(s,a) - \\left(Z_\\mu(V,h,x_{\\theta^{k}},s,a)\\right)\\right]^2 \\\\\n=& \\frac{\\beta}{2}\\sum_{s,a}w(s,a) \\left\\{ \\left[(1-\\frac{1}{2\\mu \\beta})x(s,a) -  Z_\\mu(V,h,x_{\\theta^{k}},s,a) \\right]^2 + \\frac{1}{4\\mu\\beta}(1-\\frac{1}{4\\mu\\beta})x(s,a)^2 \\right\\} \\\\\n\\geq & \\frac{1}{2\\mu}(1-\\frac{1}{4\\mu\\beta})\\sum_{s,a}w(s,a)x(s,a)^2 \\geq 0.\n\\end{split}\n\\eee\n","sha256_utf8":"e0f0e29424b678e1bfaa1bd199a7d253e2614699d4b6cc0e6c7313af339176a9"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"限制强凸性与乘子曲率","source_units":[{"start":78743,"end":79073,"start_line":971,"end_line":971,"raw_tex":"Hence, $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is restricted strongly convex and $c$ is the smallest positive eigenvalue of the Hessian of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$. It also implies that $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$ is $\\frac{1}{2\\mu}(1-\\frac{1}{4\\mu\\beta})$-strongly convex with respect to $x$.\n","sha256_utf8":"81ec8728e2513632da95b2e7868c01a0867764e26eb355ea09e4445c39d49f85"}]},{"id":"S029","semantic_type":"lemma","kind":"lemma","title":"局部线性化误差界","source_units":[{"start":81311,"end":82255,"start_line":1000,"end_line":1011,"raw_tex":"\\begin{lemma}\\label{lem:lr}\n    Suppose that Assumption \\ref{assum:fc} holds and $\\zeta \\in \\Xi$. \n    Then we have the following estimation:\n    \\beaa\n    E_{0,w}(V_{\\phi}(s) - V_{\\phi}^0(s))^2 &=& O\\left(R_V^3m_V^{-1/2}\\right), \\\\\n    E_{0,w}(h_{\\psi}(s,a) - h_{\\psi}^0(s.a))^2 &=& O\\left(R_h^3m_h^{-1/2}\\right),\\\\\n    E_{0,w}(x_{\\theta}(s,a) - x_{\\theta}^0(s,a))^2 &=& O\\left(R_x^3m_x^{-1/2}\\right), \\\\\n     E_{0}\\left| \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) -  \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\right| &=& O\\left(R^3m^{-1/2} + R^2m^{-1/4}\\right), \\\\ \n     E_{0}\\|\\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) - \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_2^2 &=& O\\left(R^3m^{-1/2}\\right),\n     \\eeaa\n     where $E_{0,w}$ represent the expectation under the distribution \\eqref{eq:dist} and $w$.\n\\end{lemma}","sha256_utf8":"26671a94969a922f10a7bf65fc52ca08cdc3555ab203ee063b3c235aeeaac5a1"}]},{"id":"S030","semantic_type":"proof","kind":"proof","title":"线性化误差完整证明","source_units":[{"start":82256,"end":87414,"start_line":1012,"end_line":1086,"raw_tex":"\\begin{proof}\nThe first three estimations follows from \\cite[Lemma D.1]{liu2019neural}. The\nmain difference is that $\\psi_{1,i}$ is always equal to $1$, but it does not change the arguments.  By Lemma \\ref{lem:cvx}, it yields\n\\be\\label{eq:est-g0}\n\\begin{split}\n&  \\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_{\\mathcal{H}}^2 \\\\\n\\leq &  2\\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2\n+ 2\\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) - \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2 \\\\\n\\leq & 2\\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2+ \n2L\\| (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) - (V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2 \\\\\n\\leq & 4L\\| (V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2+ \n4\\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(0,0,0)\\|_{\\mathcal{H}}^2\n+ 6LR,\n\\end{split}\n\\ee\nwhere the last inequality uses the bound of $V, h, x$ in \\eqref{eq:bv}.\nIt implies\n\\[\nE_0  \\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_{\\mathcal{H}}^2 = O\\left(R\\right).\n\\]\nIt follows from Lemma \\ref{lem:cvx} and \\cite[Theorem 18.15]{bauschke2011convex} that\n\\beaa\n&& E_{0}\\left| \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) -  \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\right| \\\\\n&\\leq & E_0\\left| \\iprod{ \\nabla_\\Gamma\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)}{(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)}_{\\mathcal{H}}\\right| \\\\\n&& + \\frac{L}{2}E_0  \\|(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|^2_{\\mathcal{H}} \\\\\n&\\leq & E_0 \\| \\nabla_\\Gamma\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|\\|(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_{\\mathcal{H}} \\\\\n&&+ \\frac{L}{2}E_0 \\|(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|^2_{\\mathcal{H}} \\\\\n&\\leq &  \\left(E_0\\| \\nabla_\\Gamma\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|^2_{\\mathcal{H}} \\right)^{\\frac{1}{2}}\n\\left( E_0\\|(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|^2_{\\mathcal{H}} \\right)^{\\frac{1}{2}}\\\\\n&&+ \\frac{L}{2}E_0 \\|(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|^2_{\\mathcal{H}} \\\\\n&=& O\\left(R^3m^{-1/2} + R^2m^{-1/4}\\right).\n\\eeaa\nWe next prove the last estimation. We can obtain\n\\beaa\n && E_{0}\\|\\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) - \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_2^2  \\\\\n &\\leq & 2E_{0}\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta})^T\\cdot \\left( \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) \n - \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \\right)\\|_2^2  \\\\\n && + 2E_{0}\\|\\left( \\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta}) \n - \\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \\right)^T\\cdot\\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_2^2  \\\\\n  &\\leq & 2E_{0}\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta})\\|_*^2\n  \\|\\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) \n - \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_\\mathcal{H}^2  \\\\\n && + 2E_{0}\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta}) - \\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_*^2\n \\|\\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_\\mathcal{H}^2,  \\\\\n &:=& I_1 + I_2,\n\\eeaa\nwhere the second inequality is due to the triangle inequality and the H\\\"older inequality and $\\|\\cdot\\|_*$ is defined by\n\\be\\label{eq:est-g1}\n\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta})\\|_*^2 = \\sum_{s,a}w(s,a)\\left[ \\|\\nabla_\\phi V_\\phi(s)\\|^2 \n+ \\|\\nabla_\\psi h_\\psi(s)\\|^2 + \\|\\nabla_\\theta x_\\theta(s)\\|^2\\right] \\leq 3.\n\\ee\nUsing the relationship \\eqref{eq:est-g1}, we get \n\\beaa\nI_1 &\\leq& 6E_{0}\\|\\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) \n - \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_\\mathcal{H}^2 \\\\\n &\\leq& 6L E_{0}\\|(V_{\\phi},h_{\\psi},x_{\\theta})  - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_\\mathcal{H}^2 \n =  O\\left(R^3m^{-1/2}\\right).\n\\eeaa\nAccording to the arguments in  \\cite[Lemma D.2]{liu2019neural}, we have\n\\be\\label{eq:est-g2}\n\\begin{split}\n&E_{0}\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta}) - \\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_*^2\n=  O\\left(Rm^{-1/2}\\right), \\\\\n&E_{0}\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta}) - \\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_*^2\n\\| (V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2 \n=  O\\left(Rm^{-1/2}\\right).\n\\end{split}\n\\ee\nCombining \\eqref{eq:est-g0} and \\eqref{eq:est-g2} yields\n\\beaa\nI_2 \n =  O\\left(R^2m^{-1/2}\\right).\n\\eeaa\n which completes the proof.\n\\end{proof}","sha256_utf8":"841746001f9eb49ab7e7a1279e84ca07aad85a79b68b07faee78e5567fa4f223"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"引用网络线性化估计","source_units":[{"start":82270,"end":82482,"start_line":1013,"end_line":1014,"raw_tex":"The first three estimations follows from \\cite[Lemma D.1]{liu2019neural}. The\nmain difference is that $\\psi_{1,i}$ is always equal to $1$, but it does not change the arguments.  By Lemma \\ref{lem:cvx}, it yields\n","sha256_utf8":"57463ba9bd23682697a0f36eac043cb3532de8fcc1d48f236be9065668ea7149"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"线性化目标梯度矩界","source_units":[{"start":82482,"end":83550,"start_line":1015,"end_line":1031,"raw_tex":"\\be\\label{eq:est-g0}\n\\begin{split}\n&  \\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_{\\mathcal{H}}^2 \\\\\n\\leq &  2\\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2\n+ 2\\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) - \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2 \\\\\n\\leq & 2\\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2+ \n2L\\| (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) - (V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2 \\\\\n\\leq & 4L\\| (V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2+ \n4\\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(0,0,0)\\|_{\\mathcal{H}}^2\n+ 6LR,\n\\end{split}\n\\ee\nwhere the last inequality uses the bound of $V, h, x$ in \\eqref{eq:bv}.\nIt implies\n\\[\nE_0  \\| \\nabla_{\\Gamma}\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_{\\mathcal{H}}^2 = O\\left(R\\right).\n\\]\n","sha256_utf8":"9ec10872d40cad3bb77e3191a9aec0a222f9a9e19b938c545d06982e39f2ecdf"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"控制目标函数差","source_units":[{"start":83550,"end":84856,"start_line":1032,"end_line":1043,"raw_tex":"It follows from Lemma \\ref{lem:cvx} and \\cite[Theorem 18.15]{bauschke2011convex} that\n\\beaa\n&& E_{0}\\left| \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) -  \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\right| \\\\\n&\\leq & E_0\\left| \\iprod{ \\nabla_\\Gamma\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)}{(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)}_{\\mathcal{H}}\\right| \\\\\n&& + \\frac{L}{2}E_0  \\|(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|^2_{\\mathcal{H}} \\\\\n&\\leq & E_0 \\| \\nabla_\\Gamma\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|\\|(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_{\\mathcal{H}} \\\\\n&&+ \\frac{L}{2}E_0 \\|(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|^2_{\\mathcal{H}} \\\\\n&\\leq &  \\left(E_0\\| \\nabla_\\Gamma\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|^2_{\\mathcal{H}} \\right)^{\\frac{1}{2}}\n\\left( E_0\\|(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|^2_{\\mathcal{H}} \\right)^{\\frac{1}{2}}\\\\\n&&+ \\frac{L}{2}E_0 \\|(V_{\\phi},h_{\\psi},x_{\\theta}) - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|^2_{\\mathcal{H}} \\\\\n&=& O\\left(R^3m^{-1/2} + R^2m^{-1/4}\\right).\n\\eeaa\n","sha256_utf8":"5dc794e57f0b24eb9c6864da5c8ceda94885acc300526c1b84b673f92b664947"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"分解参数梯度误差","source_units":[{"start":84856,"end":86410,"start_line":1044,"end_line":1062,"raw_tex":"We next prove the last estimation. We can obtain\n\\beaa\n && E_{0}\\|\\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) - \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_2^2  \\\\\n &\\leq & 2E_{0}\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta})^T\\cdot \\left( \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) \n - \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \\right)\\|_2^2  \\\\\n && + 2E_{0}\\|\\left( \\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta}) \n - \\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \\right)^T\\cdot\\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_2^2  \\\\\n  &\\leq & 2E_{0}\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta})\\|_*^2\n  \\|\\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) \n - \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_\\mathcal{H}^2  \\\\\n && + 2E_{0}\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta}) - \\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_*^2\n \\|\\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_\\mathcal{H}^2,  \\\\\n &:=& I_1 + I_2,\n\\eeaa\nwhere the second inequality is due to the triangle inequality and the H\\\"older inequality and $\\|\\cdot\\|_*$ is defined by\n\\be\\label{eq:est-g1}\n\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta})\\|_*^2 = \\sum_{s,a}w(s,a)\\left[ \\|\\nabla_\\phi V_\\phi(s)\\|^2 \n+ \\|\\nabla_\\psi h_\\psi(s)\\|^2 + \\|\\nabla_\\theta x_\\theta(s)\\|^2\\right] \\leq 3.\n\\ee\n","sha256_utf8":"a2c287edce32bdf2a90a7c822296d53ca88cf442e6c905589725bf68830d5100"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"控制函数梯度差项","source_units":[{"start":86410,"end":86813,"start_line":1063,"end_line":1069,"raw_tex":"Using the relationship \\eqref{eq:est-g1}, we get \n\\beaa\nI_1 &\\leq& 6E_{0}\\|\\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi},h_{\\psi},x_{\\theta}) \n - \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_\\mathcal{H}^2 \\\\\n &\\leq& 6L E_{0}\\|(V_{\\phi},h_{\\psi},x_{\\theta})  - (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_\\mathcal{H}^2 \n =  O\\left(R^3m^{-1/2}\\right).\n\\eeaa\n","sha256_utf8":"d371836f65d370680f1bcaff16147d570f6bb2694536b8e57d196c23585b52af"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"引用激活变化矩界","source_units":[{"start":86813,"end":87270,"start_line":1070,"end_line":1079,"raw_tex":"According to the arguments in  \\cite[Lemma D.2]{liu2019neural}, we have\n\\be\\label{eq:est-g2}\n\\begin{split}\n&E_{0}\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta}) - \\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_*^2\n=  O\\left(Rm^{-1/2}\\right), \\\\\n&E_{0}\\|\\nabla_\\zeta(V_{\\phi},h_{\\psi},x_{\\theta}) - \\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)\\|_*^2\n\\| (V_{\\phi^0},h_{\\psi^0},x_{\\theta^0})\\|_{\\mathcal{H}}^2 \n=  O\\left(Rm^{-1/2}\\right).\n\\end{split}\n\\ee\n","sha256_utf8":"88089e85b1a8c24a0b6632cad3eba30009aac503e97863e6c282c0184aec32d1"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"控制雅可比差项并收束","source_units":[{"start":87270,"end":87403,"start_line":1080,"end_line":1085,"raw_tex":"Combining \\eqref{eq:est-g0} and \\eqref{eq:est-g2} yields\n\\beaa\nI_2 \n =  O\\left(R^2m^{-1/2}\\right).\n\\eeaa\n which completes the proof.\n","sha256_utf8":"a7bc11eb22f3f3b20808c10bdec7caa368f201669fb66c17b29e5addc0346f2a"}]},{"id":"S038","semantic_type":"lemma","kind":"lemma","title":"内层投影梯度收敛界","source_units":[{"start":87485,"end":88579,"start_line":1089,"end_line":1106,"raw_tex":"\\begin{lemma}\\label{lem:conv1}\n    Suppose that Assumption \\ref{assum:fc} holds and $\\beta > \\frac{1}{4\\mu}$. Let  $\\zeta^{k,t} = (\\phi^{k,t}, \\psi^{k,t}, \\theta^{k,t})$ be a sequence generated by the projected gradient descent method \\eqref{eq:pjg} with $\\alpha_t = \\alpha \\leq \\frac{1}{\\sqrt{3}L}$.\n    Then we have\n    \\be\\label{eq:conv1}\n    \\begin{split}\n   & E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}},h_{\\psi^{k+1}},x_{\\theta^{k+1}}) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})\\right) \\\\\n    = &O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2}\\right).\n    \\end{split}\n    \\ee\n    Furthermore, it holds that\n    \\be\\label{eq:conv2}\n    E_{0,w}(x_{\\theta^{k+1}}(s,a) - \\tilde{x}^{k+1}(s,a))^2 = O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2}\\right). \n    \\ee\n%    \\beaa\n%        E_w(x_{\\theta^{k,T}}(s,a) - x_{\\theta^*}^0(s,a))^2 &=& O(R^3m^{-1/2}+R^2/T), \\\\\n%        E_w(Z_\\mu(V_{\\phi^{k,T}},h_{\\psi^{k,T}},x_{\\theta^k},s,a) - Z_\\mu(V_{\\phi^*},h_{\\psi^*},x_{\\theta^k},s,a))^2 &=& O(R^3m^{-1/2}+R^2/T).\n%    \\eeaa\n\\end{lemma}","sha256_utf8":"ed6d6d6eafc13c44d33afaa1d2471b30fd390e21dff6420916498fbf171b1fd3"}]},{"id":"S039","semantic_type":"proof","kind":"proof","title":"内层收敛完整证明","source_units":[{"start":88580,"end":95000,"start_line":1107,"end_line":1183,"raw_tex":"\\begin{proof}\n    Since $V_{\\phi}^0 (s), h_{\\psi}^0 (s,a), x_{\\theta}^0 (s,a)$ is linear with respect to $\\phi,\\psi,\\theta$, respectively, $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)$ is convex for $\\zeta$ by Lemma \\ref{lem:cvx}. We also have\n    \\be\n    \\begin{split}\n   &  \\| \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) - \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi'}^0,h_{\\psi'}^0,x_{\\theta'}^0)  \\|^2 \\\\\n    \\leq & \\|\\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)^T\\cdot \n    \\left( \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \n - \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi'}^0,h_{\\psi'}^0,x_{\\theta'}^0) \\right)\\|^2 \\\\\n     \\leq &\\|\\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \\|_*^2\n     \\| \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \n - \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi'}^0,h_{\\psi'}^0,x_{\\theta'}^0) \\|_\\mathcal{H}^2 \\\\\n \\leq & 3L^2 \\| (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \n - (V_{\\phi'}^0,h_{\\psi'}^0,x_{\\theta'}^0) \\|_\\mathcal{H}^2 \\leq 3L^2 \\| \\zeta - \\zeta' \\|_2^2.\n    \\end{split}\n    \\ee\n    Hence, $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)$ has  $\\sqrt{3}L$-Lipschitz continuous gradient. According to \\cite[Theorem 18.15]{bauschke2011convex}, it implies \n\\be\\label{eq:cvx1}\n\\begin{split}\n&\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t+1}}^0,h_{\\psi^{k,t+1}}^0,x_{\\theta^{k,t+1}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) \\\\\n\\leq & \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0)^T(\\zeta^{k,t+1} - \\zeta^{k,t} ) + \\frac{\\sqrt{3}L}{2}\\|\\zeta^{k,t+1} - \\zeta^{k,t} \\|_2^2.\n\\end{split}\n\\ee\nBy the convexity of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)$, we have\n\\be\\label{eq:cvx2}\n\\begin{split}\n    & \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{*}}^0,h_{\\psi^{*}}^0,x_{\\theta^{*}}^0)  \\\\\n\\leq & \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0)^T(\\zeta^{k,t} - \\zeta^{*} ),\n\\end{split}\n\\ee\nwhere $\\zeta^* = (\\phi^*,\\psi^*,\\theta^*)$ is an optimal solution, i.e., $(V_{\\phi^{*}}^0,h_{\\psi^{*}}^0,x_{\\theta^{*}}^0) = (\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})$.\nAccording to the property of projection operator, we get\n\\be\\label{eq:cvx3}\n({\\zeta^{*} - \\zeta^{k,t+1}})^T\\left(\\zeta^{k,t} - \\alpha_t \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}},h_{\\psi^{k,t}},x_{\\theta^{k,t}} ) - \\zeta^{k,t+1}\\right) \\geq 0\n\\ee\nCombining \\eqref{eq:cvx1}, \\eqref{eq:cvx2} and \\eqref{eq:cvx3} yields\n\\be\\label{eq:cvx4}\n\\begin{split}\n&\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t+1}}^0,h_{\\psi^{k,t+1}}^0,x_{\\theta^{k,t+1}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{*}}^0,h_{\\psi^{*}}^0,x_{\\theta^{*}}^0) \\\\\n\\leq & \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0)^T(\\zeta^{k,t+1} - \\zeta^{*} ) + \\frac{\\sqrt{3}L}{2}\\|\\zeta^{k,t+1} - \\zeta^{k,t} \\|_2^2 \\\\\n\\leq & \\frac{1}{\\alpha_t}(\\zeta^{k,t}  - \\zeta^{k,t+1})^T(\\zeta^{k,t+1} - \\zeta^{*} ) + \\frac{\\sqrt{3}L}{2}\\|\\zeta^{k,t+1} - \\zeta^{k,t} \\|_2^2 \\\\ \n&+ \\left(\\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) -  \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}},h_{\\psi^{k,t}},x_{\\theta^{k,t}} )  \\right)^T(\\zeta^{k,t+1} - \\zeta^{*} ) \\\\\n\\leq& \\frac{1}{\\alpha}\\left(\\|\\zeta^{k,t} - \\zeta^{*}\\|^2 - \\|\\zeta^{k,t+1} - \\zeta^{*}\\|^2\\right) \\\\\n&+ \\|\\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) -  \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}},h_{\\psi^{k,t}},x_{\\theta^{k,t}} )\\|\\|\\zeta^{k,t+1} - \\zeta^{*}\\|.\n\\end{split}\n\\ee\nBy the convexity of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t+1}}^0,h_{\\psi^{k,t+1}}^0,x_{\\theta^{k,t+1}}^0)$ and Lemma \\ref{lem:lr}, we obtain\n\\bee\n    \\begin{split}\n     & E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}}^0,h_{\\psi^{k+1}}^0,x_{\\theta^{k+1}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})\\right) \\\\\n     \\leq & \\frac{1}{T} \\sum_{t=1}^T E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})\\right) \\\\\n \\leq &  \\frac{4R}{T} \\sum_{t=1}^T \\left( E_{0}\\|\\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) -  \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}},h_{\\psi^{k,t}},x_{\\theta^{k,t}} )\\|^2\\right)^{\\frac{1}{2}}  \\\\\n &+ \\frac{1}{\\alpha T} \\|\\zeta^{k,0} - \\zeta^{*}\\|^2\\\\\n = & O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} \\right),\n\\end{split}\n\\eee\nwhere the second inequality is due to \\eqref{eq:cvx4} and H\\\"older inequality.\nIt follows from Lemma \\ref{lem:lr} that\n\\bee\n    \\begin{split}\n&E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}},h_{\\psi^{k+1}},x_{\\theta^{k+1}}) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})\\right) \\\\\n\\leq & E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}}^0,h_{\\psi^{k+1}}^0,x_{\\theta^{k+1}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})\\right) \\\\\n&+ E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}},h_{\\psi^{k+1}},x_{\\theta^{k+1}}) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}}^0,h_{\\psi^{k+1}}^0,x_{\\theta^{k+1}}^0)\\right) \\\\\n&= O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2}\\right),\n\\end{split}\n\\eee\nwhich completes the proof of \\eqref{eq:conv1}.\n\nBy Lemma \\ref{lem:cvx} (i), we have\n\\bee\n    E_0\\dist\\left(\\left(V_{\\phi^{k+1}},h_{\\psi^{k+1}},x_{\\theta^{k+1}}\\right),\\mathcal{X}^*\\right)^2 = O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2}\\right).\n\\eee\nIt means that there exists an optimal solution $(V^*, h^*, x^*) \\in \\mathcal{X}^*$ such that\n\\bee\n    E_{0,w}(x_{\\theta^{k+1}}(s,a) - {x}^*(s,a))^2 = O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2}\\right).\n\\eee\nBy Lemma \\ref{lem:cvx} (ii), $x^*$ is unique in $\\mathcal{X}^*$, i.e. $x^* = \\tilde{x}^{k+1}$, which completes the proof of \\eqref{eq:conv2}.\n\\end{proof}","sha256_utf8":"764d111217e5a24ce60fdb2dac754f32ef68b51ca8c340fd0243a8e5c238bd13"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"线性化目标凸性与光滑性","source_units":[{"start":88594,"end":89980,"start_line":1108,"end_line":1122,"raw_tex":"    Since $V_{\\phi}^0 (s), h_{\\psi}^0 (s,a), x_{\\theta}^0 (s,a)$ is linear with respect to $\\phi,\\psi,\\theta$, respectively, $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)$ is convex for $\\zeta$ by Lemma \\ref{lem:cvx}. We also have\n    \\be\n    \\begin{split}\n   &  \\| \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) - \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi'}^0,h_{\\psi'}^0,x_{\\theta'}^0)  \\|^2 \\\\\n    \\leq & \\|\\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)^T\\cdot \n    \\left( \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \n - \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi'}^0,h_{\\psi'}^0,x_{\\theta'}^0) \\right)\\|^2 \\\\\n     \\leq &\\|\\nabla_\\zeta(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \\|_*^2\n     \\| \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \n - \\nabla_\\Gamma \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi'}^0,h_{\\psi'}^0,x_{\\theta'}^0) \\|_\\mathcal{H}^2 \\\\\n \\leq & 3L^2 \\| (V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0) \n - (V_{\\phi'}^0,h_{\\psi'}^0,x_{\\theta'}^0) \\|_\\mathcal{H}^2 \\leq 3L^2 \\| \\zeta - \\zeta' \\|_2^2.\n    \\end{split}\n    \\ee\n    Hence, $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)$ has  $\\sqrt{3}L$-Lipschitz continuous gradient. According to \\cite[Theorem 18.15]{bauschke2011convex}, it implies \n","sha256_utf8":"0d1d5c7b00116221dc959b45c9f9e0f5297250e3ed15d7b910797871beaf1e7f"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"下降估计与凸性估计","source_units":[{"start":89980,"end":91064,"start_line":1123,"end_line":1136,"raw_tex":"\\be\\label{eq:cvx1}\n\\begin{split}\n&\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t+1}}^0,h_{\\psi^{k,t+1}}^0,x_{\\theta^{k,t+1}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) \\\\\n\\leq & \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0)^T(\\zeta^{k,t+1} - \\zeta^{k,t} ) + \\frac{\\sqrt{3}L}{2}\\|\\zeta^{k,t+1} - \\zeta^{k,t} \\|_2^2.\n\\end{split}\n\\ee\nBy the convexity of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi}^0,h_{\\psi}^0,x_{\\theta}^0)$, we have\n\\be\\label{eq:cvx2}\n\\begin{split}\n    & \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{*}}^0,h_{\\psi^{*}}^0,x_{\\theta^{*}}^0)  \\\\\n\\leq & \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0)^T(\\zeta^{k,t} - \\zeta^{*} ),\n\\end{split}\n\\ee\nwhere $\\zeta^* = (\\phi^*,\\psi^*,\\theta^*)$ is an optimal solution, i.e., $(V_{\\phi^{*}}^0,h_{\\psi^{*}}^0,x_{\\theta^{*}}^0) = (\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})$.\n","sha256_utf8":"ee2861cf9ce78c8779ffab0209fee5d447470d68458deac8d44908f2c18974ee"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"投影最优性不等式","source_units":[{"start":91064,"end":91330,"start_line":1137,"end_line":1140,"raw_tex":"According to the property of projection operator, we get\n\\be\\label{eq:cvx3}\n({\\zeta^{*} - \\zeta^{k,t+1}})^T\\left(\\zeta^{k,t} - \\alpha_t \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}},h_{\\psi^{k,t}},x_{\\theta^{k,t}} ) - \\zeta^{k,t+1}\\right) \\geq 0\n\\ee\n","sha256_utf8":"b0f18ba6b58201aad80783d872fb402f882a5c9a87c0d624986888a1d0008b4a"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"含线性化误差的单步界","source_units":[{"start":91330,"end":92568,"start_line":1141,"end_line":1151,"raw_tex":"Combining \\eqref{eq:cvx1}, \\eqref{eq:cvx2} and \\eqref{eq:cvx3} yields\n\\be\\label{eq:cvx4}\n\\begin{split}\n&\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t+1}}^0,h_{\\psi^{k,t+1}}^0,x_{\\theta^{k,t+1}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{*}}^0,h_{\\psi^{*}}^0,x_{\\theta^{*}}^0) \\\\\n\\leq & \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0)^T(\\zeta^{k,t+1} - \\zeta^{*} ) + \\frac{\\sqrt{3}L}{2}\\|\\zeta^{k,t+1} - \\zeta^{k,t} \\|_2^2 \\\\\n\\leq & \\frac{1}{\\alpha_t}(\\zeta^{k,t}  - \\zeta^{k,t+1})^T(\\zeta^{k,t+1} - \\zeta^{*} ) + \\frac{\\sqrt{3}L}{2}\\|\\zeta^{k,t+1} - \\zeta^{k,t} \\|_2^2 \\\\ \n&+ \\left(\\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) -  \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}},h_{\\psi^{k,t}},x_{\\theta^{k,t}} )  \\right)^T(\\zeta^{k,t+1} - \\zeta^{*} ) \\\\\n\\leq& \\frac{1}{\\alpha}\\left(\\|\\zeta^{k,t} - \\zeta^{*}\\|^2 - \\|\\zeta^{k,t+1} - \\zeta^{*}\\|^2\\right) \\\\\n&+ \\|\\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) -  \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}},h_{\\psi^{k,t}},x_{\\theta^{k,t}} )\\|\\|\\zeta^{k,t+1} - \\zeta^{*}\\|.\n\\end{split}\n\\ee\n","sha256_utf8":"bd73c78031ccc6e9be7173fb1a31a51b6736260aeb1044fb7650bf145ede81b9"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"平均迭代的目标误差","source_units":[{"start":92568,"end":93651,"start_line":1152,"end_line":1162,"raw_tex":"By the convexity of $\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t+1}}^0,h_{\\psi^{k,t+1}}^0,x_{\\theta^{k,t+1}}^0)$ and Lemma \\ref{lem:lr}, we obtain\n\\bee\n    \\begin{split}\n     & E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}}^0,h_{\\psi^{k+1}}^0,x_{\\theta^{k+1}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})\\right) \\\\\n     \\leq & \\frac{1}{T} \\sum_{t=1}^T E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})\\right) \\\\\n \\leq &  \\frac{4R}{T} \\sum_{t=1}^T \\left( E_{0}\\|\\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}}^0,h_{\\psi^{k,t}}^0,x_{\\theta^{k,t}}^0) -  \\nabla_\\zeta \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k,t}},h_{\\psi^{k,t}},x_{\\theta^{k,t}} )\\|^2\\right)^{\\frac{1}{2}}  \\\\\n &+ \\frac{1}{\\alpha T} \\|\\zeta^{k,0} - \\zeta^{*}\\|^2\\\\\n = & O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} \\right),\n\\end{split}\n\\eee\nwhere the second inequality is due to \\eqref{eq:cvx4} and H\\\"older inequality.\n","sha256_utf8":"96e68a746bd6f88f8af49a5f93fd6ec0b504812649776570c0229b5830ef6762"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"转回非线性网络目标","source_units":[{"start":93651,"end":94424,"start_line":1163,"end_line":1172,"raw_tex":"It follows from Lemma \\ref{lem:lr} that\n\\bee\n    \\begin{split}\n&E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}},h_{\\psi^{k+1}},x_{\\theta^{k+1}}) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})\\right) \\\\\n\\leq & E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}}^0,h_{\\psi^{k+1}}^0,x_{\\theta^{k+1}}^0) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1})\\right) \\\\\n&+ E_0\\left(\\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}},h_{\\psi^{k+1}},x_{\\theta^{k+1}}) - \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V_{\\phi^{k+1}}^0,h_{\\psi^{k+1}}^0,x_{\\theta^{k+1}}^0)\\right) \\\\\n&= O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2}\\right),\n\\end{split}\n\\eee\nwhich completes the proof of \\eqref{eq:conv1}.\n","sha256_utf8":"d5e41e9101b5a44cfba5e2d6b8785870fb2da8daaea4556ebb50beff22c15991"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"目标误差转为乘子误差","source_units":[{"start":94425,"end":94989,"start_line":1174,"end_line":1182,"raw_tex":"By Lemma \\ref{lem:cvx} (i), we have\n\\bee\n    E_0\\dist\\left(\\left(V_{\\phi^{k+1}},h_{\\psi^{k+1}},x_{\\theta^{k+1}}\\right),\\mathcal{X}^*\\right)^2 = O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2}\\right).\n\\eee\nIt means that there exists an optimal solution $(V^*, h^*, x^*) \\in \\mathcal{X}^*$ such that\n\\bee\n    E_{0,w}(x_{\\theta^{k+1}}(s,a) - {x}^*(s,a))^2 = O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2}\\right).\n\\eee\nBy Lemma \\ref{lem:cvx} (ii), $x^*$ is unique in $\\mathcal{X}^*$, i.e. $x^* = \\tilde{x}^{k+1}$, which completes the proof of \\eqref{eq:conv2}.\n","sha256_utf8":"99283a4e90219ac008ad43e79dd3fdad81653c3ae4bf6f10204c00131277eff0"}]},{"id":"S047","semantic_type":"lemma","kind":"lemma","title":"罚方法与精确增广更新差","source_units":[{"start":95123,"end":96062,"start_line":1186,"end_line":1203,"raw_tex":"\\begin{lemma}\\label{lem:err-2}\n    %Let $\\tilde{V}^{k+1}, \\tilde{h}^{k+1}, \\tilde{x}^{k+1} = \\arg\\min_{h\\geq 0,V, x} \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$. \n    Suppose that Assumption \\ref{assum:fc} holds and $\\beta > \\frac{1}{4\\mu}$.\n    Define $\\hat{V}^{k},\\hat{h}^{k} = \\arg\\min_{h\\geq 0,V}  \\mathcal{L}_{\\mu}(V,h, x_{\\theta^k})$ and let \n    $\n        \\hat{x}^{k+1}(s,a) = [Z_\\mu( \\hat{V}^{k}, \\hat{h}^k, x_{\\theta^k},s,a)].\n    $\n    Then we have\n    \\bea\n    Z_\\mu(\\tilde{V}^{k+1}, \\tilde{h}^k, x_{\\theta^{k}}, s,a) &=& \\xi_1 Z_\\mu( \\hat{V}^{k+1}, \\hat{h}^{k+1}, x_{\\theta^k},s,a), \\\\\n    \\tilde{x}^{k+1} &=& \\xi_2 \\hat{x}^{k+1},\n    \\eea\n    where $\\xi_1 = \\frac{4\\mu\\beta}{4\\mu\\beta-1}$ and $\\xi_2 = \\frac{4\\mu\\beta - 2}{4\\mu\\beta-1}$.\n    Furthermore, it implies\n    \\[\n        E_{0,w}(x_{\\theta^{k+1}}(s,a) - \\hat{x}^{k+1}(s,a))^2 = O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2} + \\beta^{-2}R\\right).\n    \\]\n\\end{lemma}","sha256_utf8":"0a4e31827d0222c329148e2507febd863d6612e36784f437f59557ebb4661665"}]},{"id":"S048","semantic_type":"proof","kind":"proof","title":"增广更新误差完整证明","source_units":[{"start":96064,"end":97939,"start_line":1205,"end_line":1245,"raw_tex":"\\begin{proof}\n    Let $x^*_{V,h} = \\arg\\min_x \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$. The first order optimality condition gives \n    \\[\n        \\frac{1}{2\\mu}Z_\\mu(V,h,x_{\\theta^{k}},s,a)\n        + \\beta\\left[x^*_{V,h}(s,a) - \\left(Z_\\mu(V,h,x_{\\theta^{k}},s,a)\\right)\\right] = 0.\n    \\]\n    It implies that \n    \\be\\label{eq:xvh}\n    x^*_{V,h}(s,a) = (1-\\frac{1}{2\\mu\\beta})Z_\\mu(V,h,x_{\\theta^{k}},s,a).\n\\ee\nDefine\n    \\[\n    \\mathcal{M}(V,h) := \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x^*_{V,h}) \n    = \\sum_{s} \\rho_0(s)V(s) +\\frac{1}{2\\mu\\xi_1}\\sum_{s,a}w(s,a)[Z_\\mu(V,h,x_{\\theta^{k}},s,a)]^2.\n    \\]\n    The optimality condition of  $\\min_{h \\geq 0, V} \\mathcal{M}(V,h)$ is\n\\bee\n    \\begin{split}\n        &\\xi_1 \\rho_0(s') - \\sum_{s,a}w(s,a)Z_\\mu(V^*,h^*,x',s,a)(\\delta_{s'}(s) - \\gamma P(s'|s,a)) = 0, \\forall s' \\\\\n    & \\mu h(x,a) = \\left[- x'(s,a) - \\mu\\left(r(s,a)+\\sum_{s'} \\gamma P(s'|s,a) V(s') - V(s)\\right)\\right]_+.\n\\end{split}\n\\eee\nTogether with \\eqref{eq:kkt-lg}, we have \n\\beaa\nZ_\\mu(\\tilde{V}^{k+1}, \\tilde{h}^k, x_{\\theta^{k}}, s,a) &=& \\xi_1 Z_\\mu(\\hat{V}^{k+1}, \\hat{h}^{k+1},x_{\\theta^k},  s,a). \n    \\eeaa\n    Combining with \\eqref{eq:xvh}, we have\n\\[\n    \\tilde{x}^{k+1}(s,a) = x^*_{\\tilde{V}^{k+1},\\tilde{h}^{k+1}}(s,a) = (1-\\frac{1}{2\\mu\\beta})Z_\\mu(\\tilde{V}^{k+1},\\tilde{h}^{k+1},x_{\\theta^{k}},s,a) = \\xi_2 \\hat{x}^{k+1}(s,a).\n\\]\nAccording to Lemma \\ref{lem:conv1}, we have\n\\beaa\n   &&  E_{0,w}(x_{\\theta^{k+1}}(s,a) - \\hat{x}^{k+1}(s,a))^2 \n\\\\\n   &\\leq &  2E_{0,w}(x_{\\theta^{k+1}}(s,a) - \\tilde{x}^{k+1}(s,a))^2 +  2E_{0,w}(\\tilde{x}^{k+1}(s,a) - \\hat{x}^{k+1}(s,a))^2 \\\\\n   &= &  2E_{0,w}(x_{\\theta^{k+1}}(s,a) - \\tilde{x}^{k+1}(s,a))^2 +  2(1-\\xi_2)^2E_{0,w}( \\hat{x}^{k+1}(s,a))^2 \\\\\n   & = & O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2} + \\beta^{-2}R\\right)\n.\n\\eeaa\nThis completes the proof of Lemma \\ref{lem:err-2}.\n\\end{proof}","sha256_utf8":"ef7f20f569755bd2d11f35e65fa632937a19121428dac185ba7ce361fa86fd53"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"消去乘子得到显式解","source_units":[{"start":96078,"end":96478,"start_line":1206,"end_line":1214,"raw_tex":"    Let $x^*_{V,h} = \\arg\\min_x \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x)$. The first order optimality condition gives \n    \\[\n        \\frac{1}{2\\mu}Z_\\mu(V,h,x_{\\theta^{k}},s,a)\n        + \\beta\\left[x^*_{V,h}(s,a) - \\left(Z_\\mu(V,h,x_{\\theta^{k}},s,a)\\right)\\right] = 0.\n    \\]\n    It implies that \n    \\be\\label{eq:xvh}\n    x^*_{V,h}(s,a) = (1-\\frac{1}{2\\mu\\beta})Z_\\mu(V,h,x_{\\theta^{k}},s,a).\n\\ee\n","sha256_utf8":"3c92c5b9a5f38d7b75e23b4a8a535a221f4c4ee842c51669f5f1a75d86af1d8f"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"约化罚函数最优性条件","source_units":[{"start":96478,"end":97017,"start_line":1215,"end_line":1226,"raw_tex":"Define\n    \\[\n    \\mathcal{M}(V,h) := \\tilde{\\mathcal{L}}^k_{\\beta,\\mu}(V,h,x^*_{V,h}) \n    = \\sum_{s} \\rho_0(s)V(s) +\\frac{1}{2\\mu\\xi_1}\\sum_{s,a}w(s,a)[Z_\\mu(V,h,x_{\\theta^{k}},s,a)]^2.\n    \\]\n    The optimality condition of  $\\min_{h \\geq 0, V} \\mathcal{M}(V,h)$ is\n\\bee\n    \\begin{split}\n        &\\xi_1 \\rho_0(s') - \\sum_{s,a}w(s,a)Z_\\mu(V^*,h^*,x',s,a)(\\delta_{s'}(s) - \\gamma P(s'|s,a)) = 0, \\forall s' \\\\\n    & \\mu h(x,a) = \\left[- x'(s,a) - \\mu\\left(r(s,a)+\\sum_{s'} \\gamma P(s'|s,a) V(s') - V(s)\\right)\\right]_+.\n\\end{split}\n\\eee\n","sha256_utf8":"fee98293081331827fefa821740d1d87c1064e4d77e418564fcc71224ee5c247"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"比较罚解与增广解的比例","source_units":[{"start":97017,"end":97429,"start_line":1227,"end_line":1234,"raw_tex":"Together with \\eqref{eq:kkt-lg}, we have \n\\beaa\nZ_\\mu(\\tilde{V}^{k+1}, \\tilde{h}^k, x_{\\theta^{k}}, s,a) &=& \\xi_1 Z_\\mu(\\hat{V}^{k+1}, \\hat{h}^{k+1},x_{\\theta^k},  s,a). \n    \\eeaa\n    Combining with \\eqref{eq:xvh}, we have\n\\[\n    \\tilde{x}^{k+1}(s,a) = x^*_{\\tilde{V}^{k+1},\\tilde{h}^{k+1}}(s,a) = (1-\\frac{1}{2\\mu\\beta})Z_\\mu(\\tilde{V}^{k+1},\\tilde{h}^{k+1},x_{\\theta^{k}},s,a) = \\xi_2 \\hat{x}^{k+1}(s,a).\n\\]\n","sha256_utf8":"c45033c6d2c8dd1b73ac1c8010d8d856f8e298378687f0c4e97a800d5ee1ab69"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"合并优化误差与罚参数误差","source_units":[{"start":97429,"end":97928,"start_line":1235,"end_line":1244,"raw_tex":"According to Lemma \\ref{lem:conv1}, we have\n\\beaa\n   &&  E_{0,w}(x_{\\theta^{k+1}}(s,a) - \\hat{x}^{k+1}(s,a))^2 \n\\\\\n   &\\leq &  2E_{0,w}(x_{\\theta^{k+1}}(s,a) - \\tilde{x}^{k+1}(s,a))^2 +  2E_{0,w}(\\tilde{x}^{k+1}(s,a) - \\hat{x}^{k+1}(s,a))^2 \\\\\n   &= &  2E_{0,w}(x_{\\theta^{k+1}}(s,a) - \\tilde{x}^{k+1}(s,a))^2 +  2(1-\\xi_2)^2E_{0,w}( \\hat{x}^{k+1}(s,a))^2 \\\\\n   & = & O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2} + \\beta^{-2}R\\right)\n.\n\\eeaa\nThis completes the proof of Lemma \\ref{lem:err-2}.\n","sha256_utf8":"db2857602cda954353ee4a0519bea1e398741b18cc6bfaba062a6b99b9f35407"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"获得非精确预解式误差","source_units":[{"start":99575,"end":99807,"start_line":1280,"end_line":1283,"raw_tex":"According to Lemmas \\ref{lem:lp} and \\ref{lem:err-2}, we have   \n\\bee\n    E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 = O\\left(R^2T^{-1} + R^{5/2}m^{-1/4} + R^{3}m^{-1/2} + \\beta^{-2}R\\right). \n\\eee\n","sha256_utf8":"133b99ab6d6d61091f69f5dc74feef59f8947518fdf52dd83b691d9d77d38223"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"固定点与坚定非扩张估计","source_units":[{"start":99807,"end":100245,"start_line":1284,"end_line":1290,"raw_tex":"Let $x^*$ be the solution of  the equation $x^* = \\mathcal{T}x^*$, i.e., the optimal solution of the LP \\eqref{eq:rllp-dual}. It follows from \\cite[Proposition 1]{rockafellar1976augmented} that \n\\bee\n\\begin{split}\n& E_{0,w}\\left(x^*(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 + E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n\\leq & E_{0,w}\\left(x_{\\theta^{k}}(s,a) - x^*(s,a)\\right)^2.\n\\end{split}\n\\eee\n","sha256_utf8":"ad9af67ab3893b2da160ef5675568baa620b224a0abeb93efbe8335ac0b69e02"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"残差的距离递减不等式","source_units":[{"start":100245,"end":101273,"start_line":1291,"end_line":1304,"raw_tex":"It implies\n\\bee\n\\begin{split}\n&  E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n\\leq & E_{0,w}\\left(x_{\\theta^{k}}(s,a) - x^*(s,a)\\right)^2 - E_{0,w}\\left(x^*(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n= & E_{0,w}\\left(x_{\\theta^{k}}(s,a) - x^*(s,a)\\right)^2  - E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - x^*(s,a)\\right)^2\\\\\n& +E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - x^*(s,a)\\right)^2 - E_{0,w}\\left(x^*(s,a) - Tx_{\\theta^{k}}(s,a)\\right)^2 \\\\\n= & E_{0,w}\\left(x_{\\theta^{k}}(s,a) - x^*(s,a)\\right)^2  - E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - x^*(s,a)\\right)^2\\\\\n& +E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)\\left(x_{\\theta^{k+1}}(s,a) +  \\mathcal{T}x_{\\theta^{k}}(s,a) - 2x^*(s,a)\\right) \\\\\n\\leq & E_{0,w}\\left(x_{\\theta^{k}}(s,a) - x^*(s,a)\\right)^2  - E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - x^*(s,a)\\right)^2\\\\\n& + cR^{1/2} \\cdot (E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2)^{\\frac{1}{2}}, \n\\end{split}\n\\eee\nwhere $c$ is a constant.\n","sha256_utf8":"bf007b92558a379b64b667159a8c6f39ee9b87c905e157329ffab9c118a1c0c6"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"求和得到平均残差界","source_units":[{"start":101273,"end":101759,"start_line":1305,"end_line":1312,"raw_tex":"Taking summation over $k$, we obtain\n\\bee\n\\begin{split}\n& \\frac{1}{K} \\sum_{k=1}^K E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\ \n\\leq &  \\frac{1}{K} E_{0,w}\\left(x_{\\theta^{1}}(s,a) - x^*(s,a)\\right)^2 + cR^{1/2} \\cdot \\frac{1}{K} \\sum_{k=1}^K  (E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2)^{\\frac{1}{2}} \\\\\n = & O\\left(R^{3/2}T^{-1/2} + R^{7/4}m^{-1/8} + R^{2}m^{-1/4} + \\beta^{-1}R + K^{-1}R\\right).\n \\end{split}\n\\eee\n","sha256_utf8":"7266cc8ad72965efd51d8c424fc3393110f42bc5462b5c6d4276937a8e862747"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"相邻迭代残差变化界","source_units":[{"start":101759,"end":102085,"start_line":1313,"end_line":1319,"raw_tex":"Furthermore, by the same arguments, we have\n\\bee\n\\begin{split}\n& E_{0,w}\\left(x_{\\theta^{k+1}}(s,a) - \\mathcal{T}x_{\\theta^{k+1}}(s,a)\\right)^2 -  E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n= & O\\left(R^{3/2}T^{-1/2} + R^{7/4}m^{-1/8} + R^{2}m^{-1/4} + \\beta^{-1}R \\right).\n\\end{split}\n\\eee\n","sha256_utf8":"0a94886c000e039d3c798f48510df3830517f83dda35a58d0495c4389c0be449"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"加权累加残差变化","source_units":[{"start":102085,"end":102434,"start_line":1320,"end_line":1327,"raw_tex":"It implies that\n\\bee\n\\begin{split}\n & K \\cdot E_{0,w}\\left(x_{\\theta^{K}}(s,a) - \\mathcal{T}x_{\\theta^{K}}(s,a)\\right)^2  \\\\\n\\leq & \\sum_{k=1}^K E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n&+  \\sum_{k=1}^K (K-k) \\cdot O\\left(R^{3/2}T^{-1/2} + R^{7/4}m^{-1/8} + R^{2}m^{-1/4} + \\beta^{-1}R \\right).\n\\end{split}\n\\eee\n","sha256_utf8":"ae06864b4160483255c904edeb26cfa64365eb075ab0fdbba99734bd54a4012d"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"得到末次迭代残差界","source_units":[{"start":102434,"end":102895,"start_line":1328,"end_line":1337,"raw_tex":"Hence, we have\n\\bee\n\\begin{split}\n &  E_{0,w}\\left(x_{\\theta^{K}}(s,a) - \\mathcal{T}x_{\\theta^{K}}(s,a)\\right)^2  \\\\\n\\leq & \\frac{1}{K}\\sum_{k=1}^K E_{0,w}\\left(x_{\\theta^{k}}(s,a) - \\mathcal{T}x_{\\theta^{k}}(s,a)\\right)^2 \\\\\n& + K \\cdot O\\left(R^{3/2}T^{-1/2} + R^{7/4}m^{-1/8} + R^{2}m^{-1/4} + \\beta^{-1}R \\right) \\\\\n= & O\\left(KR^{3/2}T^{-1/2} + KR^{7/4}m^{-1/8} + KR^{2}m^{-1/4} + K\\beta^{-1}R+ K^{-1}R \\right),\n\\end{split}\n\\eee\nwhich completes the proof.\n","sha256_utf8":"27301c383afee10f5832db772596870145cfd4071ecca258d631208a9e1e76a3"}]},{"id":"S060","semantic_type":"external_fact","kind":"external_fact","title":"网络线性化标准估计","source_units":[],"declared_external":true,"declaration_text":["liu2019neural, Lemma D.1 supplies the expected squared two-layer ReLU linearization error O(R^3 m^{-1/2}) under the relevant initialization and distribution regularity. The author states that fixing the slack network output weights to one leaves the argument unchanged."]},{"id":"S061","semantic_type":"external_fact","kind":"external_fact","title":"激活变化标准矩界","source_units":[],"declared_external":true,"declaration_text":["The argument of liu2019neural, Lemma D.2 supplies E_0||J-J^0||_*^2=O(R m^{-1/2}) and E_0[||J-J^0||_*^2 ||Gamma_initial||_H^2]=O(R m^{-1/2}), as invoked by the author."]},{"id":"S062","semantic_type":"external_fact","kind":"external_fact","title":"光滑函数二次上界","source_units":[],"declared_external":true,"declaration_text":["bauschke2011convex, Theorem 18.15: L-Lipschitz continuity of the gradient implies f(y)<=f(x)+<grad f(x),y-x>+(L/2)||y-x||^2."]},{"id":"S063","semantic_type":"external_fact","kind":"external_fact","title":"单调算子预解式性质","source_units":[],"declared_external":true,"declaration_text":["A proper lower-semicontinuous convex function has a maximal monotone subdifferential. Its resolvent is firmly nonexpansive, and its fixed points are the function's minimizers. In particular, for T=(I+mu S)^{-1} and Tx*=x*, ||Tx-x*||^2+||x-Tx||^2<=||x-x*||^2; the author cites rockafellar1976augmented, Proposition 1."]},{"id":"S064","semantic_type":"external_fact","kind":"external_fact","title":"凸最优性与投影性质","source_units":[],"declared_external":true,"declaration_text":["Standard convex quadratic KKT optimality and first-order optimality conditions; the convex first-order inequality; the projection characterization <z-P_C(y),y-P_C(y)><=0 for z in a closed convex set C; Jensen's inequality and Cauchy-Schwarz. The extended-valued indicator of C equals zero on C and positive infinity outside C."]},{"id":"S065","semantic_type":"assumption","kind":"assumption","title":"主定理局部假设","source_units":[{"start":98642,"end":98825,"start_line":1259,"end_line":1259,"raw_tex":"    Suppose that Assumption \\ref{assum:fc} holds and $\\beta > \\frac{1}{4\\mu}$. Let  $\\zeta^{k+1} = (\\phi^{k+1}, \\psi^{k+1}, \\theta^{k+1})$ be a sequence generated by \\eqref{eq:pjg}. \n","sha256_utf8":"7a4f8feaaf3fc8eacefc3a3b44f64d2f6085b3f09692cfece36c9da959f2b367"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S022","kind":"supports"},{"source_id":"S004","target_id":"S054","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S023","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S023","kind":"supports"},{"source_id":"S006","target_id":"S027","kind":"supports"},{"source_id":"S006","target_id":"S049","kind":"supports"},{"source_id":"S007","target_id":"S023","kind":"supports"},{"source_id":"S007","target_id":"S024","kind":"supports"},{"source_id":"S008","target_id":"S012","kind":"supports"},{"source_id":"S008","target_id":"S018","kind":"supports"},{"source_id":"S008","target_id":"S031","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S018","kind":"supports"},{"source_id":"S009","target_id":"S031","kind":"supports"},{"source_id":"S009","target_id":"S032","kind":"supports"},{"source_id":"S009","target_id":"S036","kind":"supports"},{"source_id":"S010","target_id":"S015","kind":"supports"},{"source_id":"S010","target_id":"S017","kind":"supports"},{"source_id":"S010","target_id":"S041","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S032","kind":"supports"},{"source_id":"S012","target_id":"S034","kind":"supports"},{"source_id":"S012","target_id":"S040","kind":"supports"},{"source_id":"S012","target_id":"S052","kind":"supports"},{"source_id":"S012","target_id":"S055","kind":"supports"},{"source_id":"S013","target_id":"S027","kind":"supports"},{"source_id":"S013","target_id":"S032","kind":"supports"},{"source_id":"S013","target_id":"S034","kind":"supports"},{"source_id":"S013","target_id":"S040","kind":"supports"},{"source_id":"S014","target_id":"S027","kind":"supports"},{"source_id":"S014","target_id":"S032","kind":"supports"},{"source_id":"S014","target_id":"S049","kind":"supports"},{"source_id":"S014","target_id":"S050","kind":"supports"},{"source_id":"S015","target_id":"S031","kind":"supports"},{"source_id":"S015","target_id":"S036","kind":"supports"},{"source_id":"S015","target_id":"S041","kind":"supports"},{"source_id":"S016","target_id":"S032","kind":"supports"},{"source_id":"S016","target_id":"S044","kind":"supports"},{"source_id":"S016","target_id":"S052","kind":"supports"},{"source_id":"S017","target_id":"S042","kind":"supports"},{"source_id":"S017","target_id":"S044","kind":"supports"},{"source_id":"S018","target_id":"S031","kind":"supports"},{"source_id":"S018","target_id":"S034","kind":"supports"},{"source_id":"S018","target_id":"S040","kind":"supports"},{"source_id":"S019","target_id":"S053","kind":"supports"},{"source_id":"S019","target_id":"S054","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S023","target_id":"S021","kind":"supports"},{"source_id":"S024","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S020","target_id":"S053","kind":"supports"},{"source_id":"S023","target_id":"S051","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S032","kind":"supports"},{"source_id":"S025","target_id":"S033","kind":"supports"},{"source_id":"S025","target_id":"S035","kind":"supports"},{"source_id":"S025","target_id":"S040","kind":"supports"},{"source_id":"S025","target_id":"S046","kind":"supports"},{"source_id":"S031","target_id":"S033","kind":"supports"},{"source_id":"S031","target_id":"S035","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S032","target_id":"S037","kind":"supports"},{"source_id":"S034","target_id":"S037","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S030","kind":"supports"},{"source_id":"S033","target_id":"S030","kind":"supports"},{"source_id":"S034","target_id":"S030","kind":"supports"},{"source_id":"S035","target_id":"S030","kind":"supports"},{"source_id":"S036","target_id":"S030","kind":"supports"},{"source_id":"S037","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S029","target_id":"S044","kind":"supports"},{"source_id":"S029","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S041","target_id":"S039","kind":"supports"},{"source_id":"S042","target_id":"S039","kind":"supports"},{"source_id":"S043","target_id":"S039","kind":"supports"},{"source_id":"S044","target_id":"S039","kind":"supports"},{"source_id":"S045","target_id":"S039","kind":"supports"},{"source_id":"S046","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S051","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S038","target_id":"S052","kind":"supports"},{"source_id":"S049","target_id":"S048","kind":"supports"},{"source_id":"S050","target_id":"S048","kind":"supports"},{"source_id":"S051","target_id":"S048","kind":"supports"},{"source_id":"S052","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S053","target_id":"S057","kind":"supports"},{"source_id":"S054","target_id":"S057","kind":"supports"},{"source_id":"S055","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S053","target_id":"S002","kind":"supports"},{"source_id":"S054","target_id":"S002","kind":"supports"},{"source_id":"S055","target_id":"S002","kind":"supports"},{"source_id":"S056","target_id":"S002","kind":"supports"},{"source_id":"S057","target_id":"S002","kind":"supports"},{"source_id":"S058","target_id":"S002","kind":"supports"},{"source_id":"S059","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S060","target_id":"S031","kind":"supports"},{"source_id":"S061","target_id":"S036","kind":"supports"},{"source_id":"S062","target_id":"S033","kind":"supports"},{"source_id":"S062","target_id":"S041","kind":"supports"},{"source_id":"S063","target_id":"S019","kind":"supports"},{"source_id":"S063","target_id":"S054","kind":"supports"},{"source_id":"S063","target_id":"S057","kind":"supports"},{"source_id":"S064","target_id":"S022","kind":"supports"},{"source_id":"S064","target_id":"S023","kind":"supports"},{"source_id":"S064","target_id":"S033","kind":"supports"},{"source_id":"S064","target_id":"S034","kind":"supports"},{"source_id":"S064","target_id":"S041","kind":"supports"},{"source_id":"S064","target_id":"S042","kind":"supports"},{"source_id":"S064","target_id":"S044","kind":"supports"},{"source_id":"S064","target_id":"S049","kind":"supports"},{"source_id":"S064","target_id":"S050","kind":"supports"},{"source_id":"S064","target_id":"S055","kind":"supports"},{"source_id":"S015","target_id":"S065","kind":"supports"},{"source_id":"S017","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S053","kind":"supports"}]}}
{"record_id":"2105.14033v2","paper":{"title":"An Inexact Projected Gradient Method with Rounding and Lifting by Nonlinear Programming for Solving Rank-One Semidefinite Relaxation of Polynomial Optimization","authors":["Heng Yang","Ling Liang","Luca Carlone","Kim-Chuan Toh"],"arxiv_id":"2105.14033","version":"v2","journal":"Mathematical Programming","doi":"10.1007/s10107-022-01912-6","abstract_url":"https://arxiv.org/abs/2105.14033v2","pdf_url":"https://arxiv.org/pdf/2105.14033v2"},"context":{"raw_tex":"Let $p, h_1,\\dots, h_l \\in \\Real{}[x]$ be real-valued multivariate polynomials in $x \\in \\Real{d}$, we consider the following equality constrained polynomial optimization problem\\footnote{Although our framework can be generalized to problems with inequality constraints, here we focus on equality constraints to simplify the presentation.}\n\\begin{equation} \\label{eq:pop}\n\\min_{x \\in \\Real{d}} \\cbrace{p(x)\\ |\\ h_i(x)=0, i=1,\\dots,l}. \\tag{POP}\n\\end{equation}\n\n{\\bf SDP Relaxations for Polynomial Optimization Problems}. Towards this goal, let us first give an overview of the celebrated Lasserre's moment/sums-of-squares (SOS) semidefinite relaxation hierarchy~\\cite{Lasserre01siopt-LasserreHierarchy,Parrilo03MP-SOS}. Let $\\kappa \\geq 1$ be an integer such that $2\\kappa$ is equal or greater than the maximum degree of $p,h_i,i=1,\\dots,l$, and denote \n\n\n$v := [x]_\\kappa$ as the standard vector of monomials in $x$ of degree up to $\\kappa$. We build the \\emph{moment matrix} $X := [x]_\\kappa [x]_\\kappa\\tran$ that contains the standard monomials in $x$ of degree up to $2\\kappa$. \nAs a result,\n$p$ and $h_i$'s can be written as linear functions in $X$.\n\n\n\\begin{equation}\\label{eq:primalSDP}\n\\min_{X \\in \\sym{n}} \\cbrace{\\inprod{C}{X}\\ |\\ \\calA(X) = b, \\ X \\succeq 0},\\tag{P}\n\\end{equation}\n\nwhere $n := \\bard_\\kappa \\triangleq \\nchoosek{d+\\kappa}{\\kappa}$ is the dimension of $[x]_\\kappa$, $b \\in \\Real{m}$, $\\calA: \\sym{n} \\rightarrow \\Real{m}$ is a linear map $\\calA(X) \\triangleq (\\inprod{A_i}{X})_{i=1}^m$ with $A_i \\in \\sym{n}$, that is onto and collects all independent linear constraints generated by the relaxation \\blue{of $X =[x]_\\kappa [x]_\\kappa\\tran$ to $X \\succeq 0$}. Let $\\calAadj: \\Real{m} \\rightarrow \\sym{n}$ be the adjoint of $\\calA$ defined as $\\calAadj y \\triangleq \\sum_{i=1}^m y_i A_i$, then the Lagrangian dual of problem~\\eqref{eq:primalSDP} reads\n\\begin{equation}\\label{eq:dualSDP}\n\\max_{y \\in \\Real{m}, S \\in \\sym{n}} \\cbrace{\\inprod{b}{y}\\ |\\ \\calAadj y + S = C,\\ S \\succeq 0},\\tag{D}\n\\end{equation}\n\nThroughout this paper, we assume that strong duality holds for \\eqref{eq:primalSDP} and \\eqref{eq:dualSDP}, and both \\eqref{eq:primalSDP} and \\eqref{eq:dualSDP} admit at least one solution.\\footnote{Particularly, if a redundant ball constraint is included in the relaxation, then dual Slater and strong duality hold for Lasserre's moment relaxations of all orders \\cite[Theorem 1]{Josz16OL-strongdualityLasserre}.}  We denote $\\Xstar$ and $(\\ystar,\\Sstar)$ as an optimal solution for \\eqref{eq:primalSDP} and \\eqref{eq:dualSDP}, respectively. Then, the KKT condition for \\eqref{eq:primalSDP} and \\eqref{eq:dualSDP} given as\n\n\n\\begin{equation}\n\t\\label{eq-kkt-sdp}\n\t\t\\cA(X) - b = 0,\\quad \\cA^*y + S - C = 0,\\quad \\inprod{X}{S} = 0,\\quad X, S\\succeq 0,\t\n\t\\end{equation}\n\n\thas $(\\Xstar,\\ystar,\\Sstar)$ as one of its solutions. Moreover, it follows that $\\fpopstar \\geq \\fpsdpstar = \\fdsdpstar$, where $\\fpsdpstar$ and $\\fdsdpstar$ are the optimal values of~\\eqref{eq:primalSDP} and~\\eqref{eq:dualSDP}, respectively. The SDP relaxation is said to be \\emph{tight} if $\\fpopstar = \\fpsdpstar$. In such cases, for any global minimizer $\\xstar$ of \\eqref{eq:pop}, the rank one lifting $X = [\\xstar]_\\kappa [\\xstar]_\\kappa\\tran$ is a minimizer of~\\eqref{eq:primalSDP}, \n\t{and any rank one optimal solution  $\\Xstar$ of \\eqref{eq:primalSDP} corresponds to a global minimizer of \\eqref{eq:pop}.}\n\n\nWe use $\\mathcal{X}$ and $\\mathcal{Y}$ to denote finite dimensional Euclidean spaces. Let $\\sym{n}$ be the space of real symmetric $n\\times n$ matrices, and $\\psd{n}$ (resp. $\\pd{n}$) be the set of positive semidefinite (resp. definite) matrices. We also write $X \\succeq 0$ (resp. $X \\succ 0$) to indicate that $X$ is positive semidefinite (resp. definite) when the dimension of $X$ is clear. We use $\\usphere{d-1}:=\\{ x \\in \\Real{d}\\mid \\norm{x} = 1 \\}$ to denote the $d$-dimensional unit sphere. For $x \\in \\Real{n}$, $\\norm{x} = \\sqrt{x\\tran x}$ is the standard $\\ell_2$ norm. For $X \\in \\Real{m \\times n}$, $\\norm{X} = \\sqrt{\\trace{X\\tran X}}$ denotes the Frobenius norm. $\\norm{x}_{\\calH} := \\sqrt{\\inprod{x}{\\calH x}}$ denotes the $\\calH$-weighted norm for a positive semidefinite mapping $\\calH$.\n\n\n\\begin{equation*}\n\t\\mathcal{F}_{\\mathrm{P}} := \\left\\{ X\\in \\mathbb{S}^n\\;\\middle |\\; \\mathcal{A}(X) = b,\\; X\\in \\mathbb{S}_+^n\\right\\},\n\\end{equation*}\n\n\\begin{algorithm}\n\t\\caption{An inexact projected gradient method (\\ipgm) for SDP \\eqref{eq:primalSDP}.}\n\t\\label{alg-iPGM}\n\t\\begin{algorithmic}\n\t\t\\item[{\\bf Input}:] Initial points $(X^0, y^0, S^0)\\in \\mathbb{S}^n_+ \\times \\mathbb{R}^m \\times \\mathbb{S}^n_+$, a nondecreasing positive sequence $\\{\\sigma_k\\}$ and a nonnegative sequence $\\{\\varepsilon_k\\}$ such that $\\{k\\epsilon_k\\}$ is summable.\n\t\t\\item[{\\bf Repeat} For $k \\geq 1$:] \n\t\t\\item[\\quad] Compute \n\t\t\\begin{equation}\n\t\t\\label{eq-iPGM-proj}\t\n\t\tX^{k} \\approx \\Pi_{\\mathcal{F}_{\\mathrm{P}}}\\left(X^{k-1} - \\sigma_k C\\right),\\quad X^{k} \\in \\mathbb{S}_+^n\n\t\t\\end{equation}\n\t\twith associated dual pair $(y^{k}, S^{k})\\in \\mathbb{R}^m\\times \\mathbb{S}_+^n$ such that the following conditions hold:\n\t\t\\begin{equation}\n\t\t\\label{eq-iPGM-inexact-conditions}\n\t\t\\begin{aligned}\n\t\t\t\\norm{\\mathcal{A}(X^{k}) - b} \\leq &\\; \\varepsilon_k, \\\\\n\t\t\t\t\\mathcal{A}^*y^k + S^k - C - \\frac{1}{\\sigma_k}\\left(X^{k} - X^{k-1}\\right)= &\\; 0, \\\\\n\t\t\t\t\\inprod{X^{k}}{S^{k}} = &\\;0.\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\t\\item[{\\bf Until}:] Termination conditions are met (\\cf Section \\ref{sec:numexps}).\n\t\t\\item[{\\bf Output}:] $(X^k,y^k,S^k)$.\n\t\\end{algorithmic}\n\\end{algorithm}\n\n\t\\item\\label{item:rounding} {\\bf (Rounding)}. Let $\\barX^{k} = \\sum_{i=1}^n \\lambda_i v_i v_i\\tran$ be the spectral decomposition of $\\barX^{k}$ with $\\lambda_1 \\geq \\dots \\geq \\lambda_n$ in nonincreasing order. Compute $r \\geq 1$ hypotheses for the \\eqref{eq:pop} from the leading $r$ eigenvectors $v_1,\\dots, v_r$   \n\t\\bea\n\t\t\\bar{x}^{k}_{i} = \\rounding(v_i), \\quad i = 1,\\dots,r,\n\t\\eea\n\twhere the function~\\rounding~can be problem-dependent and we provide examples in the numerical experiments in Section~\\ref{sec:numexps}. Generally, if the SDP~\\eqref{eq:primalSDP} comes from a dense relaxation, and the feasible set of the original~\\eqref{eq:pop}, denoted as $\\setpop$, is simple to project, then we can design~\\rounding~to be\n\t\\bea\n\tv_i \\leftarrow \\frac{v_i}{v_i[1]}, \\quad \\bar{x}^{k}_{i} \\leftarrow \\Pi_{\\setpop}(v_i[x]),\n\t\\eea\n\twhere one first normalizes $v_i$ such that its leading entry $v_i[1]$ is equal to $1$, and then projects its entries corresponding to order-one monomials,~\\ie~$v_i[x]$, to the feasible set of~\\eqref{eq:pop}. Note that the rationale for designing this rounding method is that practical POP applications often involve simple constraints such as binary~\\cite{Goemans95JACM-maxcut}, unit sphere~\\cite{Yang19ICCV-wahba}, and orthogonality~\\cite{Yang20neurips-onering,Briales17CVPR-3Dregistration}, whose projection maps are simple (\\eg if $x$ is binary, then $\\Pi_{\\setpop}(v_i[x]) = \\sign\\parentheses{v_i[x]}$ just takes the sign of each entry of $v_i[x]$). If $\\setpop$ is not easy to project, then we omit the projection, in which case the local search will start from an infeasible initialization.\n\t\\item {\\bf (Local Search)}. Apply a local search method for the POP (as an NLP) with the initial point chosen as $ \\bar{x}^{k}_i $ for each hypothesis $i=1,\\dots,r$. Denote the solution of each local search as $ \\hat{x}^{k}_i $, with associated objective value $p(\\hatx^{k}_i)$, choose the best local solution with minimum objective value. Formally, we have\n\t\\bea\n\t\\hatx_i^{k} = \\nlp(\\barx^{k}_i),\\quad i=1,\\dots,r, \\quad \\hatx^{k} = \\argmin_{\\hatx^{k}_i, i=1\\dots,r} p(\\hatx^{k}_i).\n\t\\eea\n\n\t\\item\\label{item:lifting} {\\bf (Lifting)}. Perform a rank-one lifting of the best local solution according to the SDP relaxation scheme\n\t\\bea\n\t\\hatX^{k} = v\\parentheses{\\hatx^{k}} v\\parentheses{\\hatx^{k}}\\tran,\n\t\\eea\n\twhere $v: \\Real{d} \\rightarrow \\Real{n}$ is a dense or sparse monomial lifting (\\eg $v(x) = [x]_\\kappa$ is the full set of monomials up to degree $\\kappa$ in the case of dense Lasserre's hierarchy at order $\\kappa$).\n\n\n\\begin{algorithm}\n\t\\caption{An inexact projected gradient method accelerated \\blue{via rounding and lifting by} NLP for solving tight SDP relaxations of POPs (\\name).}\n\t\\label{alg-iPGMnlp}\n\t\\begin{algorithmic}\n\t\t\\item[{\\bf Input}:] Initial points $ (X^{0}, y^0, S^0)\\in \\mathbb{S}_+^n\\times \\mathbb{R}^{m}\\times\\mathbb{S}_+^n $, a nondecreasing positive sequences $ \\{\\sigma_k\\}$, a nonnegative sequence $\\{\\varepsilon_k\\}$ such that $\\{k\\varepsilon_k\\}$ is summable, a stopping criterion $\\eta:\\mathbb{S}^n_+\\times \\mathbb{R}^m\\times \\mathbb{S}_+^n\\rightarrow \\mathbb{R}_+$ with a tolerance $ {\\tt Tol} > 0 $. Initialize $\\calV = \\{ X^0 \\}$ if $X^0\\in \\setsdpp$ and $\\calV = \\emptyset$ otherwise. Choose a positive integer $r \\in [1,n]$, and a positive constant $\\epsilon>0$. \n\t\t\\item[{\\bf Iterate} the following steps for $ k = 1,\\dots $:]\n\t\t\\item[\\quad {\\bf Step 1 (Projection)}.] Compute $(\\overline{X}^k, y^k, S^{k})$ satisfying \\eqref{eq-iPGM-proj} and \\eqref{eq-iPGM-inexact-conditions} (see Section \\ref{sec:apgsubproblem}).\n\t\t\\item[\\quad {\\bf Step 2 (Certification)}.] If $ \\eta(\\barX^{k}, y^{k}, S^{k}) < {\\tt Tol} $, {\\bf output} $(\\barX^{k}, y^{k}, S^{k})$ and {\\bf stop}.\n\t\t\\item[\\quad {\\bf Step 3 (Acceleration)}.] Compute $ \\widehat{X}^{k} $ as in items~\\ref{item:rounding}-\\ref{item:lifting}:\n\t\t\\bea\n\t\t\t& \\barX^{k} = \\sum_{i=1}^n \\lambda_i v_i v_i\\tran, \\quad \\bar{x}^{k}_{i} = \\rounding(v_i), \\quad i = 1,\\dots,r, \\nonumber \\\\\n\t\t\t& \\hatx_i^{k} = \\nlp(\\barx^{k}_i), \\quad i=1,\\dots,r, \\quad \\hatx^{k} = \\argmin_{\\hatx^{k}_i, i=1\\dots,r} p(\\hatx^{k}_i), \\nonumber \\\\\n\t\t\t& \\hatX^{k} = v\\parentheses{\\hatx^{k}} v\\parentheses{\\hatx^{k}}\\tran. \\nonumber \n\t\t\\eea\n\t\t\\item[\\quad {\\bf Step 4 (Policy)}.] Update $X^{k}$ according to \n\t\t\\begin{equation}\n\t\t\t\\label{eq:accept-reject}\n\t\t\tX^{k} = \n\t\t\t\t\\begin{cases}\n\t\t\t\t\t\\hatX^{k} & {\\rm if }\\; \\inprod{C}{\\hatX^k} < \n\t\t\t\t\t\\blue{\\min \\left(\\inprod{C}{\\barX^k}, \\min_{X\\in \\calV} \\left\\{\\inprod{C}{X}\\;\\middle|\\; X\\in \\calV\\right\\}\\right)} - \\epsilon,\\; \\widehat{X}^{k} \\in \\setsdpp  \\\\\n\t\t\t\t\t\\barX^{k} & {\\rm otherwise }\n\t\t\t\t\\end{cases}.\n\t\t\\end{equation}\n\t\tIf $X^{k} = \\hatX^{k}$, set $\\calV \\leftarrow \\calV \\cup \\{ \\hatX^k \\}$.\n\t\\end{algorithmic}\n\\end{algorithm}","units":[{"start":31103,"end":31562,"start_line":913,"end_line":916,"raw_tex":"Let $p, h_1,\\dots, h_l \\in \\Real{}[x]$ be real-valued multivariate polynomials in $x \\in \\Real{d}$, we consider the following equality constrained polynomial optimization problem\\footnote{Although our framework can be generalized to problems with inequality constraints, here we focus on equality constraints to simplify the presentation.}\n\\begin{equation} \\label{eq:pop}\n\\min_{x \\in \\Real{d}} \\cbrace{p(x)\\ |\\ h_i(x)=0, i=1,\\dots,l}. \\tag{POP}\n\\end{equation}","sha256_utf8":"8f3678e080d57e1b5c31e86088ad432ba17f02833b064b5cc494232db53910f5"},{"start":32555,"end":32948,"start_line":924,"end_line":924,"raw_tex":"{\\bf SDP Relaxations for Polynomial Optimization Problems}. Towards this goal, let us first give an overview of the celebrated Lasserre's moment/sums-of-squares (SOS) semidefinite relaxation hierarchy~\\cite{Lasserre01siopt-LasserreHierarchy,Parrilo03MP-SOS}. Let $\\kappa \\geq 1$ be an integer such that $2\\kappa$ is equal or greater than the maximum degree of $p,h_i,i=1,\\dots,l$, and denote \n","sha256_utf8":"c671364d39fb2fa0de3234ae9548dc48b12f9a68834f7ed5f41294c6a5e4bb84"},{"start":35116,"end":35415,"start_line":931,"end_line":933,"raw_tex":"$v := [x]_\\kappa$ as the standard vector of monomials in $x$ of degree up to $\\kappa$. We build the \\emph{moment matrix} $X := [x]_\\kappa [x]_\\kappa\\tran$ that contains the standard monomials in $x$ of degree up to $2\\kappa$. \nAs a result,\n$p$ and $h_i$'s can be written as linear functions in $X$.\n","sha256_utf8":"e29df74be3a10b86635b9b5dee145c8840d207ea0b40a26a96622fe541f4625c"},{"start":37006,"end":37141,"start_line":950,"end_line":952,"raw_tex":"\\begin{equation}\\label{eq:primalSDP}\n\\min_{X \\in \\sym{n}} \\cbrace{\\inprod{C}{X}\\ |\\ \\calA(X) = b, \\ X \\succeq 0},\\tag{P}\n\\end{equation}","sha256_utf8":"0af9bab08025fdddbd412ce72d8d5e8c2e4e81973fd3e80d76b51636cb263701"},{"start":37142,"end":37879,"start_line":953,"end_line":956,"raw_tex":"where $n := \\bard_\\kappa \\triangleq \\nchoosek{d+\\kappa}{\\kappa}$ is the dimension of $[x]_\\kappa$, $b \\in \\Real{m}$, $\\calA: \\sym{n} \\rightarrow \\Real{m}$ is a linear map $\\calA(X) \\triangleq (\\inprod{A_i}{X})_{i=1}^m$ with $A_i \\in \\sym{n}$, that is onto and collects all independent linear constraints generated by the relaxation \\blue{of $X =[x]_\\kappa [x]_\\kappa\\tran$ to $X \\succeq 0$}. Let $\\calAadj: \\Real{m} \\rightarrow \\sym{n}$ be the adjoint of $\\calA$ defined as $\\calAadj y \\triangleq \\sum_{i=1}^m y_i A_i$, then the Lagrangian dual of problem~\\eqref{eq:primalSDP} reads\n\\begin{equation}\\label{eq:dualSDP}\n\\max_{y \\in \\Real{m}, S \\in \\sym{n}} \\cbrace{\\inprod{b}{y}\\ |\\ \\calAadj y + S = C,\\ S \\succeq 0},\\tag{D}\n\\end{equation}","sha256_utf8":"a5076b00eeb9c5a038f36d3f7198576af498cc3fbf306a5abeca9e9ea1da83c0"},{"start":38859,"end":39483,"start_line":962,"end_line":962,"raw_tex":"Throughout this paper, we assume that strong duality holds for \\eqref{eq:primalSDP} and \\eqref{eq:dualSDP}, and both \\eqref{eq:primalSDP} and \\eqref{eq:dualSDP} admit at least one solution.\\footnote{Particularly, if a redundant ball constraint is included in the relaxation, then dual Slater and strong duality hold for Lasserre's moment relaxations of all orders \\cite[Theorem 1]{Josz16OL-strongdualityLasserre}.}  We denote $\\Xstar$ and $(\\ystar,\\Sstar)$ as an optimal solution for \\eqref{eq:primalSDP} and \\eqref{eq:dualSDP}, respectively. Then, the KKT condition for \\eqref{eq:primalSDP} and \\eqref{eq:dualSDP} given as\n","sha256_utf8":"b300024aebc4be2199ca0e4d35973d4e19a18a0b76fa153dfb471921162110e6"},{"start":39484,"end":39624,"start_line":963,"end_line":966,"raw_tex":"\\begin{equation}\n\t\\label{eq-kkt-sdp}\n\t\t\\cA(X) - b = 0,\\quad \\cA^*y + S - C = 0,\\quad \\inprod{X}{S} = 0,\\quad X, S\\succeq 0,\t\n\t\\end{equation}","sha256_utf8":"e63182667a76c9ac30e91e088349e214c04dade97b7efdfe4b4de06be3d8fed8"},{"start":39625,"end":40241,"start_line":967,"end_line":968,"raw_tex":"\thas $(\\Xstar,\\ystar,\\Sstar)$ as one of its solutions. Moreover, it follows that $\\fpopstar \\geq \\fpsdpstar = \\fdsdpstar$, where $\\fpsdpstar$ and $\\fdsdpstar$ are the optimal values of~\\eqref{eq:primalSDP} and~\\eqref{eq:dualSDP}, respectively. The SDP relaxation is said to be \\emph{tight} if $\\fpopstar = \\fpsdpstar$. In such cases, for any global minimizer $\\xstar$ of \\eqref{eq:pop}, the rank one lifting $X = [\\xstar]_\\kappa [\\xstar]_\\kappa\\tran$ is a minimizer of~\\eqref{eq:primalSDP}, \n\t{and any rank one optimal solution  $\\Xstar$ of \\eqref{eq:primalSDP} corresponds to a global minimizer of \\eqref{eq:pop}.}\n","sha256_utf8":"d96cca4cbff47237917c17ce3221dfb9246ed53f82e50943c98732ffeed23621"},{"start":64633,"end":65438,"start_line":1139,"end_line":1139,"raw_tex":"We use $\\mathcal{X}$ and $\\mathcal{Y}$ to denote finite dimensional Euclidean spaces. Let $\\sym{n}$ be the space of real symmetric $n\\times n$ matrices, and $\\psd{n}$ (resp. $\\pd{n}$) be the set of positive semidefinite (resp. definite) matrices. We also write $X \\succeq 0$ (resp. $X \\succ 0$) to indicate that $X$ is positive semidefinite (resp. definite) when the dimension of $X$ is clear. We use $\\usphere{d-1}:=\\{ x \\in \\Real{d}\\mid \\norm{x} = 1 \\}$ to denote the $d$-dimensional unit sphere. For $x \\in \\Real{n}$, $\\norm{x} = \\sqrt{x\\tran x}$ is the standard $\\ell_2$ norm. For $X \\in \\Real{m \\times n}$, $\\norm{X} = \\sqrt{\\trace{X\\tran X}}$ denotes the Frobenius norm. $\\norm{x}_{\\calH} := \\sqrt{\\inprod{x}{\\calH x}}$ denotes the $\\calH$-weighted norm for a positive semidefinite mapping $\\calH$.\n","sha256_utf8":"174993dba71d594013cc2bfb83aef5ca92135db91a9d2be84ae12af8655e4802"},{"start":67831,"end":67983,"start_line":1173,"end_line":1175,"raw_tex":"\\begin{equation*}\n\t\\mathcal{F}_{\\mathrm{P}} := \\left\\{ X\\in \\mathbb{S}^n\\;\\middle |\\; \\mathcal{A}(X) = b,\\; X\\in \\mathbb{S}_+^n\\right\\},\n\\end{equation*}","sha256_utf8":"5ac5e940ed10a0b4a90e4e0602b93cfa2846995f85e694f3828420e187ebb1f3"},{"start":68494,"end":69700,"start_line":1178,"end_line":1201,"raw_tex":"\\begin{algorithm}\n\t\\caption{An inexact projected gradient method (\\ipgm) for SDP \\eqref{eq:primalSDP}.}\n\t\\label{alg-iPGM}\n\t\\begin{algorithmic}\n\t\t\\item[{\\bf Input}:] Initial points $(X^0, y^0, S^0)\\in \\mathbb{S}^n_+ \\times \\mathbb{R}^m \\times \\mathbb{S}^n_+$, a nondecreasing positive sequence $\\{\\sigma_k\\}$ and a nonnegative sequence $\\{\\varepsilon_k\\}$ such that $\\{k\\epsilon_k\\}$ is summable.\n\t\t\\item[{\\bf Repeat} For $k \\geq 1$:] \n\t\t\\item[\\quad] Compute \n\t\t\\begin{equation}\n\t\t\\label{eq-iPGM-proj}\t\n\t\tX^{k} \\approx \\Pi_{\\mathcal{F}_{\\mathrm{P}}}\\left(X^{k-1} - \\sigma_k C\\right),\\quad X^{k} \\in \\mathbb{S}_+^n\n\t\t\\end{equation}\n\t\twith associated dual pair $(y^{k}, S^{k})\\in \\mathbb{R}^m\\times \\mathbb{S}_+^n$ such that the following conditions hold:\n\t\t\\begin{equation}\n\t\t\\label{eq-iPGM-inexact-conditions}\n\t\t\\begin{aligned}\n\t\t\t\\norm{\\mathcal{A}(X^{k}) - b} \\leq &\\; \\varepsilon_k, \\\\\n\t\t\t\t\\mathcal{A}^*y^k + S^k - C - \\frac{1}{\\sigma_k}\\left(X^{k} - X^{k-1}\\right)= &\\; 0, \\\\\n\t\t\t\t\\inprod{X^{k}}{S^{k}} = &\\;0.\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\t\\item[{\\bf Until}:] Termination conditions are met (\\cf Section \\ref{sec:numexps}).\n\t\t\\item[{\\bf Output}:] $(X^k,y^k,S^k)$.\n\t\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"55de25917b27c162acf9f2478688ed9b4605ff58183d6016ed5dacb3a0c4e115"},{"start":76009,"end":78571,"start_line":1274,"end_line":1292,"raw_tex":"\t\\item\\label{item:rounding} {\\bf (Rounding)}. Let $\\barX^{k} = \\sum_{i=1}^n \\lambda_i v_i v_i\\tran$ be the spectral decomposition of $\\barX^{k}$ with $\\lambda_1 \\geq \\dots \\geq \\lambda_n$ in nonincreasing order. Compute $r \\geq 1$ hypotheses for the \\eqref{eq:pop} from the leading $r$ eigenvectors $v_1,\\dots, v_r$   \n\t\\bea\n\t\t\\bar{x}^{k}_{i} = \\rounding(v_i), \\quad i = 1,\\dots,r,\n\t\\eea\n\twhere the function~\\rounding~can be problem-dependent and we provide examples in the numerical experiments in Section~\\ref{sec:numexps}. Generally, if the SDP~\\eqref{eq:primalSDP} comes from a dense relaxation, and the feasible set of the original~\\eqref{eq:pop}, denoted as $\\setpop$, is simple to project, then we can design~\\rounding~to be\n\t\\bea\n\tv_i \\leftarrow \\frac{v_i}{v_i[1]}, \\quad \\bar{x}^{k}_{i} \\leftarrow \\Pi_{\\setpop}(v_i[x]),\n\t\\eea\n\twhere one first normalizes $v_i$ such that its leading entry $v_i[1]$ is equal to $1$, and then projects its entries corresponding to order-one monomials,~\\ie~$v_i[x]$, to the feasible set of~\\eqref{eq:pop}. Note that the rationale for designing this rounding method is that practical POP applications often involve simple constraints such as binary~\\cite{Goemans95JACM-maxcut}, unit sphere~\\cite{Yang19ICCV-wahba}, and orthogonality~\\cite{Yang20neurips-onering,Briales17CVPR-3Dregistration}, whose projection maps are simple (\\eg if $x$ is binary, then $\\Pi_{\\setpop}(v_i[x]) = \\sign\\parentheses{v_i[x]}$ just takes the sign of each entry of $v_i[x]$). If $\\setpop$ is not easy to project, then we omit the projection, in which case the local search will start from an infeasible initialization.\n\t\\item {\\bf (Local Search)}. Apply a local search method for the POP (as an NLP) with the initial point chosen as $ \\bar{x}^{k}_i $ for each hypothesis $i=1,\\dots,r$. Denote the solution of each local search as $ \\hat{x}^{k}_i $, with associated objective value $p(\\hatx^{k}_i)$, choose the best local solution with minimum objective value. Formally, we have\n\t\\bea\n\t\\hatx_i^{k} = \\nlp(\\barx^{k}_i),\\quad i=1,\\dots,r, \\quad \\hatx^{k} = \\argmin_{\\hatx^{k}_i, i=1\\dots,r} p(\\hatx^{k}_i).\n\t\\eea\n\n\t\\item\\label{item:lifting} {\\bf (Lifting)}. Perform a rank-one lifting of the best local solution according to the SDP relaxation scheme\n\t\\bea\n\t\\hatX^{k} = v\\parentheses{\\hatx^{k}} v\\parentheses{\\hatx^{k}}\\tran,\n\t\\eea\n\twhere $v: \\Real{d} \\rightarrow \\Real{n}$ is a dense or sparse monomial lifting (\\eg $v(x) = [x]_\\kappa$ is the full set of monomials up to degree $\\kappa$ in the case of dense Lasserre's hierarchy at order $\\kappa$).\n","sha256_utf8":"bc5b6aee3fee5a8eb61cee2be7c8ec67a609fd7b1c310a006f277f3b0e4aa739"},{"start":79977,"end":82191,"start_line":1302,"end_line":1328,"raw_tex":"\\begin{algorithm}\n\t\\caption{An inexact projected gradient method accelerated \\blue{via rounding and lifting by} NLP for solving tight SDP relaxations of POPs (\\name).}\n\t\\label{alg-iPGMnlp}\n\t\\begin{algorithmic}\n\t\t\\item[{\\bf Input}:] Initial points $ (X^{0}, y^0, S^0)\\in \\mathbb{S}_+^n\\times \\mathbb{R}^{m}\\times\\mathbb{S}_+^n $, a nondecreasing positive sequences $ \\{\\sigma_k\\}$, a nonnegative sequence $\\{\\varepsilon_k\\}$ such that $\\{k\\varepsilon_k\\}$ is summable, a stopping criterion $\\eta:\\mathbb{S}^n_+\\times \\mathbb{R}^m\\times \\mathbb{S}_+^n\\rightarrow \\mathbb{R}_+$ with a tolerance $ {\\tt Tol} > 0 $. Initialize $\\calV = \\{ X^0 \\}$ if $X^0\\in \\setsdpp$ and $\\calV = \\emptyset$ otherwise. Choose a positive integer $r \\in [1,n]$, and a positive constant $\\epsilon>0$. \n\t\t\\item[{\\bf Iterate} the following steps for $ k = 1,\\dots $:]\n\t\t\\item[\\quad {\\bf Step 1 (Projection)}.] Compute $(\\overline{X}^k, y^k, S^{k})$ satisfying \\eqref{eq-iPGM-proj} and \\eqref{eq-iPGM-inexact-conditions} (see Section \\ref{sec:apgsubproblem}).\n\t\t\\item[\\quad {\\bf Step 2 (Certification)}.] If $ \\eta(\\barX^{k}, y^{k}, S^{k}) < {\\tt Tol} $, {\\bf output} $(\\barX^{k}, y^{k}, S^{k})$ and {\\bf stop}.\n\t\t\\item[\\quad {\\bf Step 3 (Acceleration)}.] Compute $ \\widehat{X}^{k} $ as in items~\\ref{item:rounding}-\\ref{item:lifting}:\n\t\t\\bea\n\t\t\t& \\barX^{k} = \\sum_{i=1}^n \\lambda_i v_i v_i\\tran, \\quad \\bar{x}^{k}_{i} = \\rounding(v_i), \\quad i = 1,\\dots,r, \\nonumber \\\\\n\t\t\t& \\hatx_i^{k} = \\nlp(\\barx^{k}_i), \\quad i=1,\\dots,r, \\quad \\hatx^{k} = \\argmin_{\\hatx^{k}_i, i=1\\dots,r} p(\\hatx^{k}_i), \\nonumber \\\\\n\t\t\t& \\hatX^{k} = v\\parentheses{\\hatx^{k}} v\\parentheses{\\hatx^{k}}\\tran. \\nonumber \n\t\t\\eea\n\t\t\\item[\\quad {\\bf Step 4 (Policy)}.] Update $X^{k}$ according to \n\t\t\\begin{equation}\n\t\t\t\\label{eq:accept-reject}\n\t\t\tX^{k} = \n\t\t\t\t\\begin{cases}\n\t\t\t\t\t\\hatX^{k} & {\\rm if }\\; \\inprod{C}{\\hatX^k} < \n\t\t\t\t\t\\blue{\\min \\left(\\inprod{C}{\\barX^k}, \\min_{X\\in \\calV} \\left\\{\\inprod{C}{X}\\;\\middle|\\; X\\in \\calV\\right\\}\\right)} - \\epsilon,\\; \\widehat{X}^{k} \\in \\setsdpp  \\\\\n\t\t\t\t\t\\barX^{k} & {\\rm otherwise }\n\t\t\t\t\\end{cases}.\n\t\t\\end{equation}\n\t\tIf $X^{k} = \\hatX^{k}$, set $\\calV \\leftarrow \\calV \\cup \\{ \\hatX^k \\}$.\n\t\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"ace96af0b28136ea59143d9db975f7188d579daf897b189d469ada3f51f1a880"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\n\t\\label{thm-ipgmnlp}\n\tLet $\\{(\\barX^k, y^k, S^k)\\}$ be any sequence generated by Algorithm \\ref{alg-iPGMnlp}, and suppose that there exists a constant $M>0$ such that $\\norm{y^k}\\leq M$ for all $k\\geq 0$. \n\t% \\red{Assuming Slater's conditions hold for SDP \\eqref{eq:primalSDP}-\\eqref{eq:dualSDP}} \\blue{(LIANG: I think we do not need Slater's conditions here for global convergence, but only need them for solving subproblem. See also the red part in Page 3.)}, \n\tIf one chooses $\\sigma_k \\geq O(\\sqrt{k})$ for all $k\\geq 1$, then, $\\left\\{\\inprod{C}{\\barX^k}\\right\\}$ converges to the optimal value of the SDP problem \\eqref{eq:primalSDP}. \n\\end{theorem}","units":[{"start":84489,"end":85160,"start_line":1363,"end_line":1368,"raw_tex":"\\begin{theorem}\n\t\\label{thm-ipgmnlp}\n\tLet $\\{(\\barX^k, y^k, S^k)\\}$ be any sequence generated by Algorithm \\ref{alg-iPGMnlp}, and suppose that there exists a constant $M>0$ such that $\\norm{y^k}\\leq M$ for all $k\\geq 0$. \n\t% \\red{Assuming Slater's conditions hold for SDP \\eqref{eq:primalSDP}-\\eqref{eq:dualSDP}} \\blue{(LIANG: I think we do not need Slater's conditions here for global convergence, but only need them for solving subproblem. See also the red part in Page 3.)}, \n\tIf one chooses $\\sigma_k \\geq O(\\sqrt{k})$ for all $k\\geq 1$, then, $\\left\\{\\inprod{C}{\\barX^k}\\right\\}$ converges to the optimal value of the SDP problem \\eqref{eq:primalSDP}. \n\\end{theorem}","sha256_utf8":"64104c6e50ca464f0ba72c28ac3254dd5c1ad81acabfd74bccdc39cee2ab2aa9"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{theorem}\n\t\\label{thm-iPGM}\n\tLet $\\{(X^k, y^k, S^k)\\}$ be any sequence generated by Algorithm \\ref{alg-iPGM}, and suppose that there exists a constant $M>0$ such that $\\norm{y^k}\\leq M$ for all $k\\geq 0$. Then, for all $k\\geq 1$, it holds that \n\t\\begin{equation*}\n\t\t\\left\\{\n\t\t\t\\begin{array}{l}\n\t\t\t\t\\displaystyle -\\frac{\\norm{y^*}k\\varepsilon_k}{k} \\leq \\inprod{C}{X^k - \\Xstar} \\leq \\frac{1}{k}\\left( \\frac{1}{2\\sigma_0}\\norm{X^0 - \\Xstar}^2 + 2M\\sum_{i = 1}^k i\\varepsilon_i\\right), \\\\\n\t\t\t\t\\displaystyle \\norm{\\mathcal{A}(X^k) - b} \\leq \\varepsilon_k \\leq O\\left(\\frac{1}{k}\\right), \\\\ \t\n\t\t\t\t\\displaystyle \\norm{\\mathcal{A}^*y^k + S^k - C} \\leq O\\left( \\frac{1}{\\sqrt{k\\sigma_k}}\\right),\n\t\t\t\\end{array}\n\t\t\\right.\n\t\\end{equation*}\n\twhere $\\Xstar$ is an optimal solution of the primal problem \\eqref{eq:primalSDP} and $(\\ystar, \\Sstar)$ is an optimal solution of the dual problem \\eqref{eq:dualSDP}. In particular, if one chooses $\\sigma_k \\geq O(\\sqrt{k}) > 0$ for all $k\\geq 1$, then we have \n\t\\begin{equation*}\n\t\t\\max\\left\\{ \\left\\lvert \\inprod{C}{X^k - \\Xstar}\\right\\rvert, \\norm{\\mathcal{A}(X^k) - b}, \\norm{\\mathcal{A}^*y^k + S^k - C}\\right\\} \\leq O\\left(\\frac{1}{k}\\right).\n\t\\end{equation*}\n\\end{theorem}\n\n\\begin{proof}\nLet us only consider the case with $X^0 \\in \\setsdpp$ such that $\\calV$ is initialized as $\\{ X^0\\}$ (the case with $\\calV$ initialized as $\\emptyset$ can be argued in a similar manner). By the construction of the set $\\calV$, it contains a sequence of feasible points of~\\eqref{eq:primalSDP}, and the objective value along this sequence is \\emph{strictly} decreasing according to the acceptance policy~\\eqref{eq:accept-reject}. Hence, we have the following relation:\n\\begin{equation*}\n\t\\inf\\left\\{\\inprod{C}{X}\\;\\middle|\\; X\\in \\calV\\right\\} \\leq \\inprod{C}{X^0} - (|\\calV|-1) \\epsilon,\n\\end{equation*}\nIf $|\\calV|=\\infty$, then there exists an infinite sequence of feasible points to problem \\eqref{eq:primalSDP} whose objective function value converges to $-\\infty$. In this case, problem \\eqref{eq:primalSDP} is unbounded, a contradiction (recall that we assumed strong duality throughout this paper in Section \\ref{sec:introduction}). Hence, $|\\calV|< \\infty$. If $|\\calV| = 1$, then none of the $\\hatX^{k}$ generated by {\\bf Step 3} has been accepted, in which case Algorithm \\ref{alg-iPGMnlp} reduces to Algorithm \\ref{alg-iPGM} iPGM. Therefore, by Theorem~\\ref{thm-iPGM}, Algorithm \\ref{alg-iPGMnlp} converges as $O(1/k)$ if one chooses $\\sigma_k \\geq O(\\sqrt{k})$ for all $k\\geq 1$. If $|\\calV| > 1$, then some of the $\\hatX^{k}$ generated by {\\bf Step 3} have been accepted. Denote the last element of $\\calV$ as $\\hatX_{\\calV}$ with associated dual variables $(y_\\calV,S_\\calV)$ (from {\\bf Step 1}), then Algorithm \\ref{alg-iPGMnlp} essentially reduces to \\ipgm~with a new initial point at $(\\hatX_{\\calV}, y_\\calV, S_\\calV)$. Again, by Theorem~\\ref{thm-iPGM}, Algorithm \\ref{alg-iPGMnlp} converges as $O(1/k)$. This completes the proof.\\qed\n\\end{proof}\n\n\\begin{lemma}\n\\label{lemma-fk}\n\tUnder the inexactness conditions in \\eqref{eq-iPGM-inexact-conditions}, for any $X\\in \\mathbb{S}_+^n$, it holds that\n\t\\[\n\t\t\\inprod{C}{X - X^k} \\geq \\frac{1}{\\sigma_k}\\norm{X^{k} - X^{k-1}}^2 + \\frac{1}{\\sigma_k}\\inprod{X^{k-1} - X}{X^{k} - X^{k-1}} + \\inprod{\\mathcal{A}^*y^k}{X - X^k}.\n\t\\]\n\\end{lemma}\n\n\\begin{proof}\n\tFrom the last two conditions in \\eqref{eq-iPGM-inexact-conditions}, $\\mathcal{A}^*y^k + S^k - C = \\frac{1}{\\sigma_k}\\left(X^k - X^{k-1}\\right)$ and $\\inprod{X^k}{S^k} = 0$, we have that\n\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\inprod{C}{X - X^k} = &\\; \\inprod{\\mathcal{A}^*y^k + S^k - \\frac{1}{\\sigma_k}\\left(X^k - X^{k-1}\\right)}{X - X^k} \\\\\n\t\t\t= &\\; \\frac{1}{\\sigma_k} \\inprod{X^k - X^{k-1}}{X^k - X} + \\inprod{\\mathcal{A}^*y^k}{X - X^k} + \\underbrace{\\inprod{S^k}{X}}_{\\geq 0} - \\underbrace{\\inprod{S^k}{X^k}}_{ = 0} \\\\\n\t\t\t\\geq &\\; \\frac{1}{\\sigma_k}\\norm{X^{k} - X^{k-1}}^2 + \\frac{1}{\\sigma_k}\\inprod{X^{k-1} - X}{X^{k} - X^{k-1}} + \\inprod{\\mathcal{A}^*y^k}{X - X^k},\n\t\t\\end{aligned}\n\t\\]\n\twhere the last inequality is due to the following fact:\n\t\\bea\n\t\t\\frac{1}{\\sigma_k} \\inprod{X^k - X^{k-1}}{X^k - X} = &  \\displaystyle \\frac{1}{\\sigma_k} \\inprod{X^k- X^{k-1}}{X^k -X^{k-1} + X^{k-1} - X} \\nonumber \\\\\n\t\t = & \\displaystyle \\frac{1}{\\sigma_k}\\norm{X^{k} - X^{k-1}}^2 + \\frac{1}{\\sigma_k}\\inprod{X^{k-1} - X}{X^{k} - X^{k-1}}. \\nonumber \n\t\\eea\n\tThus, the proof is completed. \\qed\n\\end{proof}\n\n\\begin{proof}\n\tFor notational simplicity, denote $r^k = \\mathcal{A}(X^k) - b$ and $e^k = X^k - \\Xstar$ for all $k\\geq 0$. Then, $\\mathcal{A}(e^k) = r^k$ since $\\mathcal{A}(\\Xstar) = b$. Moreover, we have $\\norm{r^k}\\leq \\varepsilon_k $ from the first condition in \\eqref{eq-iPGM-inexact-conditions}. Therefore, the convergence of $\\left\\{\\norm{r^k}\\right\\}$ is obvious, since $\\norm{r^k}\\leq \\varepsilon_k$ and $\\{k\\varepsilon_k\\}$ is summable. Recall that $\\Xstar$ is an optimal solution to problem \\eqref{eq:primalSDP}. Let $X = X^{k-1}$ in Lemma \\ref{lemma-fk}, we get \n\t\\begin{equation}\n\t\\label{eq-thm-iPGM-1}\n\t\t\\inprod{C}{X^{k-1} - X^k} \\geq \\frac{1}{\\sigma_k}\\norm{e^k - e^{k-1}}^2 + \\inprod{\\mathcal{A}^*y^k}{e^{k-1} - e^{k}}\t\n\t\\end{equation}\n\tOn the other hand, let $X = \\Xstar$ in Lemma \\ref{lemma-fk}, we obtain\n\t\\begin{equation}\n\t\\label{eq-thm-iPGM-2}\n\t\t\\inprod{C}{\\Xstar - X^k} \\geq \\frac{1}{\\sigma_k}\\norm{e^k - e^{k-1}}^2 + \\frac{1}{\\sigma_k}\\inprod{e^{k-1}}{e^k - e^{k-1}} - \\inprod{A^*y^k}{e^k}.\t\n\t\\end{equation}\n\tMultiplying $k-1$ to \\eqref{eq-thm-iPGM-1} and add the resulting inequality to \\eqref{eq-thm-iPGM-2} yields\n\t\\begin{equation}\n\t\t\\label{eq-thm-iPGM-3}\n\t\t\\begin{aligned}\n\t\t\t&\\; \\inprod{C}{(k-1)\\left(X^{k-1} - \\Xstar \\right) - k\\left(X^k - \\Xstar\\right)  } \\\\ \n\t\t\t\\geq &\\; \\frac{k}{\\sigma_k}\\norm{e^k - e^{k-1}}^2 + \\frac{1}{\\sigma_k}\\inprod{e^{k-1}}{e^k - e^{k-1}} + \\inprod{\\mathcal{A}^*y^k}{(k-1)e^{k-1} - ke^k} \\\\\n\t\t\t= &\\; \\frac{1}{\\sigma_k}\\left(k\\norm{e^k}^2 + (k-1)\\norm{e^{k-1}}^2 - (2k -1)\\inprod{e^k}{e^{k-1}}\\right) + \\inprod{y^k}{(k-1)r^{k-1} - k r^k} \\\\\n\t\t\t\\geq &\\; \\frac{1}{\\sigma_k}\\left(k\\norm{e^k}^2 + (k-1)\\norm{e^{k-1}}^2 - \\frac{2k-1}{2}\\left(\\norm{e^k}^2 + \\norm{e^{k-1}}^2\\right)\\right) + \\inprod{y^k}{(k-1)r^{k-1} - k r^k} \\\\\n\t\t\t = &\\; \\frac{1}{2\\sigma_k}\\norm{e^k}^2 - \\frac{1}{2\\sigma_k}\\norm{e^{k-1}}^2 + \\inprod{y^k}{(k-1)r^{k-1} - k r^k} \\\\\n\t\t\t \\geq &\\; \\frac{1}{2\\sigma_k}\\norm{e^k}^2 - \\frac{1}{2\\sigma_{k-1}}\\norm{e^{k-1}}^2 + \\inprod{y^k}{(k-1)r^{k-1} - k r^k},\n\t\t\\end{aligned} \n\t\\end{equation}\n\twhere in the last inequality, we use the fact that $\\sigma_k \\geq \\sigma_{k-1} > 0$ for all $k\\geq 1$. By summing the inequality in \\eqref{eq-thm-iPGM-3} for $k=1,\\dots,k$, we get (recall that $\\norm{y^k}\\leq M$ for all $k\\geq 0$)\n\t\\begin{equation*}\n\t\\begin{aligned}\n\t\t\t-k \\inprod{C}{X^k - \\Xstar} \\geq &\\; \\frac{1}{2\\sigma_k}\\norm{e^k}^2 - \\frac{1}{2\\sigma_0}\\norm{e_0}^2 + \\sum_{i=1}^k\\inprod{y^i}{(i-1)r^{i-1} - i r^i} \\\\\n\t\t\t\\geq &\\; \\frac{1}{2\\sigma_k}\\norm{e^k}^2 - \\frac{1}{2\\sigma_0}\\norm{e_0}^2 - 2M\\sum_{i=1}^k i\\varepsilon_i.\n\t\t\\end{aligned}\t\n\t\\end{equation*}\n\tThe above inequality leads to the following upper bound on $\\inprod{C}{X^k - \\Xstar}$:\n\t\\begin{equation}\n\t\\label{eq-thm-iPGM-4}\n\t\\inprod{C}{X^k - \\Xstar} \\leq \\frac{1}{k}\\left(\\frac{1}{2\\sigma_0}\\norm{e^0}^2 + 2M\\sum_{i=1}^k i\\varepsilon_i\\right).\t\n\t\\end{equation}\n\t\n\tNext, we shall provide a lower bound for $\\inprod{C}{X^k - \\Xstar}$ for $k\\geq 1$. To this end, let us define the set $\\mathcal{F}_k$, which is an enlargement of the primal feasible set $\\mathcal{F}_{\\mathrm{P}}$ with respect to $\\varepsilon_k \\geq 0$, as follows:\n\t\\[\n\t\t\\mathcal{F}_k:= \\left\\{X\\in \\mathbb{S}^n\\; \\middle | \\; \\norm{\\mathcal{A}(X) - b}\\leq \\varepsilon_k,\\; X\\in \\mathbb{S}_+^n\\right\\},\\quad \\forall\\, k\\geq 1. \n\t\\]\n\tMoreover, denote\n\t\\[\n\t\t\\Xstar_k:=\\argmin\\left\\{\\inprod{C}{X}\\; \\middle | \\; X\\in \\mathcal{F}_k\\right\\},\\quad \\forall\\, k\\geq 1.\n\t\\]\n\tSince $(\\ystar, \\Sstar)\\in \\mathbb{R}^m\\times \\mathbb{S}_+^n$ is a dual optimal solution, by \\cite[Lemma 3.4]{Jiang2012}, we have that \n\t\\[\n\t\t0\\leq \\inprod{C}{\\Xstar - \\Xstar_k} \\leq \\norm{\\ystar}\\varepsilon_k,\\quad \\forall\\, k\\geq 1.\n\t\\]\n\tAs a consequence, because $X^k \\in \\calF_k$ and $\\inprod{C}{X^k} \\geq \\inprod{C}{\\Xstar_k}$, it holds that \n\t\\begin{equation}\n\t\\label{eq-thm-iPGM-5}\n\t\t\\inprod{C}{X^k - \\Xstar} \\geq \\inprod{C}{\\Xstar_k - \\Xstar} \\geq -\\norm{\\ystar}\\varepsilon_k = -\\frac{\\norm{\\ystar}k\\varepsilon_k}{k}.\t\n\t\\end{equation}\n\tCombining \\eqref{eq-thm-iPGM-4} and \\eqref{eq-thm-iPGM-5}, we see that \n\t\\[\n\t\t-\\frac{\\norm{\\ystar}k\\varepsilon_k}{k} \\leq \\inprod{C}{X^k - \\Xstar} \\leq  \\frac{1}{k}\\left(\\frac{1}{2\\sigma_0}\\norm{e^0}^2 + 2M\\sum_{i=1}^k i\\varepsilon_i\\right).\n\t\\]\n\t\n\tFinally, let us estimate the dual infeasibility. To this end, we get from \\eqref{eq-thm-iPGM-1} that\n\t\\[\n\t\t\\frac{1}{\\sigma_k}\\norm{X^k - X^{k-1}}^2 \\leq  \\inprod{C}{X^{k-1} - \\Xstar} - \\inprod{C}{X^{k} - \\Xstar} +  \\norm{y^k}\\left(\\norm{r^{k-1}} + \\norm{r^k}\\right)\n\t\\]\n\twhich implies that \n\t\\[\n\t\t\\begin{aligned}\n\t\t\t&\\; \\norm{\\mathcal{A}^*y^k + S^k - C} =  \\frac{1}{\\sigma_k}\\norm{X^{k} - X^{k-1}} \\\\\n\t\t\t\\leq &\\; \\frac{1}{\\sqrt{\\sigma_k}}\\sqrt{ \\frac{1}{k-1}\\left(\\frac{1}{2\\sigma_0}\\norm{e^0}^2 + 2M\\sum_{i=1}^{k-1} i\\varepsilon_i\\right) + \\norm{\\ystar}\\varepsilon_k + M\\left(\\varepsilon_{k-1} + \\varepsilon_k\\right)} \\\\ \n\t\t\t\\leq &\\; O\\left(\\frac{1}{\\sqrt{k\\sigma_k}}\\right),\n\t\t\\end{aligned} \n\t\\]\n\tand establishes the convergence of the dual infeasibility. Therefore, the proof is completed. \\qed\n\\end{proof}","units":[{"start":71017,"end":72233,"start_line":1232,"end_line":1248,"raw_tex":"\\begin{theorem}\n\t\\label{thm-iPGM}\n\tLet $\\{(X^k, y^k, S^k)\\}$ be any sequence generated by Algorithm \\ref{alg-iPGM}, and suppose that there exists a constant $M>0$ such that $\\norm{y^k}\\leq M$ for all $k\\geq 0$. Then, for all $k\\geq 1$, it holds that \n\t\\begin{equation*}\n\t\t\\left\\{\n\t\t\t\\begin{array}{l}\n\t\t\t\t\\displaystyle -\\frac{\\norm{y^*}k\\varepsilon_k}{k} \\leq \\inprod{C}{X^k - \\Xstar} \\leq \\frac{1}{k}\\left( \\frac{1}{2\\sigma_0}\\norm{X^0 - \\Xstar}^2 + 2M\\sum_{i = 1}^k i\\varepsilon_i\\right), \\\\\n\t\t\t\t\\displaystyle \\norm{\\mathcal{A}(X^k) - b} \\leq \\varepsilon_k \\leq O\\left(\\frac{1}{k}\\right), \\\\ \t\n\t\t\t\t\\displaystyle \\norm{\\mathcal{A}^*y^k + S^k - C} \\leq O\\left( \\frac{1}{\\sqrt{k\\sigma_k}}\\right),\n\t\t\t\\end{array}\n\t\t\\right.\n\t\\end{equation*}\n\twhere $\\Xstar$ is an optimal solution of the primal problem \\eqref{eq:primalSDP} and $(\\ystar, \\Sstar)$ is an optimal solution of the dual problem \\eqref{eq:dualSDP}. In particular, if one chooses $\\sigma_k \\geq O(\\sqrt{k}) > 0$ for all $k\\geq 1$, then we have \n\t\\begin{equation*}\n\t\t\\max\\left\\{ \\left\\lvert \\inprod{C}{X^k - \\Xstar}\\right\\rvert, \\norm{\\mathcal{A}(X^k) - b}, \\norm{\\mathcal{A}^*y^k + S^k - C}\\right\\} \\leq O\\left(\\frac{1}{k}\\right).\n\t\\end{equation*}\n\\end{theorem}","sha256_utf8":"5b19ecbc691a98e7fe2fa7ebc47107ff5baf287e9fde3baaadcc5248adec309d"},{"start":85161,"end":86939,"start_line":1369,"end_line":1375,"raw_tex":"\\begin{proof}\nLet us only consider the case with $X^0 \\in \\setsdpp$ such that $\\calV$ is initialized as $\\{ X^0\\}$ (the case with $\\calV$ initialized as $\\emptyset$ can be argued in a similar manner). By the construction of the set $\\calV$, it contains a sequence of feasible points of~\\eqref{eq:primalSDP}, and the objective value along this sequence is \\emph{strictly} decreasing according to the acceptance policy~\\eqref{eq:accept-reject}. Hence, we have the following relation:\n\\begin{equation*}\n\t\\inf\\left\\{\\inprod{C}{X}\\;\\middle|\\; X\\in \\calV\\right\\} \\leq \\inprod{C}{X^0} - (|\\calV|-1) \\epsilon,\n\\end{equation*}\nIf $|\\calV|=\\infty$, then there exists an infinite sequence of feasible points to problem \\eqref{eq:primalSDP} whose objective function value converges to $-\\infty$. In this case, problem \\eqref{eq:primalSDP} is unbounded, a contradiction (recall that we assumed strong duality throughout this paper in Section \\ref{sec:introduction}). Hence, $|\\calV|< \\infty$. If $|\\calV| = 1$, then none of the $\\hatX^{k}$ generated by {\\bf Step 3} has been accepted, in which case Algorithm \\ref{alg-iPGMnlp} reduces to Algorithm \\ref{alg-iPGM} iPGM. Therefore, by Theorem~\\ref{thm-iPGM}, Algorithm \\ref{alg-iPGMnlp} converges as $O(1/k)$ if one chooses $\\sigma_k \\geq O(\\sqrt{k})$ for all $k\\geq 1$. If $|\\calV| > 1$, then some of the $\\hatX^{k}$ generated by {\\bf Step 3} have been accepted. Denote the last element of $\\calV$ as $\\hatX_{\\calV}$ with associated dual variables $(y_\\calV,S_\\calV)$ (from {\\bf Step 1}), then Algorithm \\ref{alg-iPGMnlp} essentially reduces to \\ipgm~with a new initial point at $(\\hatX_{\\calV}, y_\\calV, S_\\calV)$. Again, by Theorem~\\ref{thm-iPGM}, Algorithm \\ref{alg-iPGMnlp} converges as $O(1/k)$. This completes the proof.\\qed\n\\end{proof}","sha256_utf8":"3c0933b21f51cbfc45978285ffbbccb51ecef9957eb49f53b6d261906a7f30ea"},{"start":218313,"end":218647,"start_line":2792,"end_line":2798,"raw_tex":"\\begin{lemma}\n\\label{lemma-fk}\n\tUnder the inexactness conditions in \\eqref{eq-iPGM-inexact-conditions}, for any $X\\in \\mathbb{S}_+^n$, it holds that\n\t\\[\n\t\t\\inprod{C}{X - X^k} \\geq \\frac{1}{\\sigma_k}\\norm{X^{k} - X^{k-1}}^2 + \\frac{1}{\\sigma_k}\\inprod{X^{k-1} - X}{X^{k} - X^{k-1}} + \\inprod{\\mathcal{A}^*y^k}{X - X^k}.\n\t\\]\n\\end{lemma}","sha256_utf8":"4ede6e53d8568ab959e4588f5aa243adc0788384fdea76a78195aae95f9df345"},{"start":218648,"end":219746,"start_line":2799,"end_line":2814,"raw_tex":"\\begin{proof}\n\tFrom the last two conditions in \\eqref{eq-iPGM-inexact-conditions}, $\\mathcal{A}^*y^k + S^k - C = \\frac{1}{\\sigma_k}\\left(X^k - X^{k-1}\\right)$ and $\\inprod{X^k}{S^k} = 0$, we have that\n\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\inprod{C}{X - X^k} = &\\; \\inprod{\\mathcal{A}^*y^k + S^k - \\frac{1}{\\sigma_k}\\left(X^k - X^{k-1}\\right)}{X - X^k} \\\\\n\t\t\t= &\\; \\frac{1}{\\sigma_k} \\inprod{X^k - X^{k-1}}{X^k - X} + \\inprod{\\mathcal{A}^*y^k}{X - X^k} + \\underbrace{\\inprod{S^k}{X}}_{\\geq 0} - \\underbrace{\\inprod{S^k}{X^k}}_{ = 0} \\\\\n\t\t\t\\geq &\\; \\frac{1}{\\sigma_k}\\norm{X^{k} - X^{k-1}}^2 + \\frac{1}{\\sigma_k}\\inprod{X^{k-1} - X}{X^{k} - X^{k-1}} + \\inprod{\\mathcal{A}^*y^k}{X - X^k},\n\t\t\\end{aligned}\n\t\\]\n\twhere the last inequality is due to the following fact:\n\t\\bea\n\t\t\\frac{1}{\\sigma_k} \\inprod{X^k - X^{k-1}}{X^k - X} = &  \\displaystyle \\frac{1}{\\sigma_k} \\inprod{X^k- X^{k-1}}{X^k -X^{k-1} + X^{k-1} - X} \\nonumber \\\\\n\t\t = & \\displaystyle \\frac{1}{\\sigma_k}\\norm{X^{k} - X^{k-1}}^2 + \\frac{1}{\\sigma_k}\\inprod{X^{k-1} - X}{X^{k} - X^{k-1}}. \\nonumber \n\t\\eea\n\tThus, the proof is completed. \\qed\n\\end{proof}","sha256_utf8":"0e0ca63eb4d4083964027f95307949a0ea00ebb857a03e19f47a241818685288"},{"start":219820,"end":224882,"start_line":2818,"end_line":2889,"raw_tex":"\\begin{proof}\n\tFor notational simplicity, denote $r^k = \\mathcal{A}(X^k) - b$ and $e^k = X^k - \\Xstar$ for all $k\\geq 0$. Then, $\\mathcal{A}(e^k) = r^k$ since $\\mathcal{A}(\\Xstar) = b$. Moreover, we have $\\norm{r^k}\\leq \\varepsilon_k $ from the first condition in \\eqref{eq-iPGM-inexact-conditions}. Therefore, the convergence of $\\left\\{\\norm{r^k}\\right\\}$ is obvious, since $\\norm{r^k}\\leq \\varepsilon_k$ and $\\{k\\varepsilon_k\\}$ is summable. Recall that $\\Xstar$ is an optimal solution to problem \\eqref{eq:primalSDP}. Let $X = X^{k-1}$ in Lemma \\ref{lemma-fk}, we get \n\t\\begin{equation}\n\t\\label{eq-thm-iPGM-1}\n\t\t\\inprod{C}{X^{k-1} - X^k} \\geq \\frac{1}{\\sigma_k}\\norm{e^k - e^{k-1}}^2 + \\inprod{\\mathcal{A}^*y^k}{e^{k-1} - e^{k}}\t\n\t\\end{equation}\n\tOn the other hand, let $X = \\Xstar$ in Lemma \\ref{lemma-fk}, we obtain\n\t\\begin{equation}\n\t\\label{eq-thm-iPGM-2}\n\t\t\\inprod{C}{\\Xstar - X^k} \\geq \\frac{1}{\\sigma_k}\\norm{e^k - e^{k-1}}^2 + \\frac{1}{\\sigma_k}\\inprod{e^{k-1}}{e^k - e^{k-1}} - \\inprod{A^*y^k}{e^k}.\t\n\t\\end{equation}\n\tMultiplying $k-1$ to \\eqref{eq-thm-iPGM-1} and add the resulting inequality to \\eqref{eq-thm-iPGM-2} yields\n\t\\begin{equation}\n\t\t\\label{eq-thm-iPGM-3}\n\t\t\\begin{aligned}\n\t\t\t&\\; \\inprod{C}{(k-1)\\left(X^{k-1} - \\Xstar \\right) - k\\left(X^k - \\Xstar\\right)  } \\\\ \n\t\t\t\\geq &\\; \\frac{k}{\\sigma_k}\\norm{e^k - e^{k-1}}^2 + \\frac{1}{\\sigma_k}\\inprod{e^{k-1}}{e^k - e^{k-1}} + \\inprod{\\mathcal{A}^*y^k}{(k-1)e^{k-1} - ke^k} \\\\\n\t\t\t= &\\; \\frac{1}{\\sigma_k}\\left(k\\norm{e^k}^2 + (k-1)\\norm{e^{k-1}}^2 - (2k -1)\\inprod{e^k}{e^{k-1}}\\right) + \\inprod{y^k}{(k-1)r^{k-1} - k r^k} \\\\\n\t\t\t\\geq &\\; \\frac{1}{\\sigma_k}\\left(k\\norm{e^k}^2 + (k-1)\\norm{e^{k-1}}^2 - \\frac{2k-1}{2}\\left(\\norm{e^k}^2 + \\norm{e^{k-1}}^2\\right)\\right) + \\inprod{y^k}{(k-1)r^{k-1} - k r^k} \\\\\n\t\t\t = &\\; \\frac{1}{2\\sigma_k}\\norm{e^k}^2 - \\frac{1}{2\\sigma_k}\\norm{e^{k-1}}^2 + \\inprod{y^k}{(k-1)r^{k-1} - k r^k} \\\\\n\t\t\t \\geq &\\; \\frac{1}{2\\sigma_k}\\norm{e^k}^2 - \\frac{1}{2\\sigma_{k-1}}\\norm{e^{k-1}}^2 + \\inprod{y^k}{(k-1)r^{k-1} - k r^k},\n\t\t\\end{aligned} \n\t\\end{equation}\n\twhere in the last inequality, we use the fact that $\\sigma_k \\geq \\sigma_{k-1} > 0$ for all $k\\geq 1$. By summing the inequality in \\eqref{eq-thm-iPGM-3} for $k=1,\\dots,k$, we get (recall that $\\norm{y^k}\\leq M$ for all $k\\geq 0$)\n\t\\begin{equation*}\n\t\\begin{aligned}\n\t\t\t-k \\inprod{C}{X^k - \\Xstar} \\geq &\\; \\frac{1}{2\\sigma_k}\\norm{e^k}^2 - \\frac{1}{2\\sigma_0}\\norm{e_0}^2 + \\sum_{i=1}^k\\inprod{y^i}{(i-1)r^{i-1} - i r^i} \\\\\n\t\t\t\\geq &\\; \\frac{1}{2\\sigma_k}\\norm{e^k}^2 - \\frac{1}{2\\sigma_0}\\norm{e_0}^2 - 2M\\sum_{i=1}^k i\\varepsilon_i.\n\t\t\\end{aligned}\t\n\t\\end{equation*}\n\tThe above inequality leads to the following upper bound on $\\inprod{C}{X^k - \\Xstar}$:\n\t\\begin{equation}\n\t\\label{eq-thm-iPGM-4}\n\t\\inprod{C}{X^k - \\Xstar} \\leq \\frac{1}{k}\\left(\\frac{1}{2\\sigma_0}\\norm{e^0}^2 + 2M\\sum_{i=1}^k i\\varepsilon_i\\right).\t\n\t\\end{equation}\n\t\n\tNext, we shall provide a lower bound for $\\inprod{C}{X^k - \\Xstar}$ for $k\\geq 1$. To this end, let us define the set $\\mathcal{F}_k$, which is an enlargement of the primal feasible set $\\mathcal{F}_{\\mathrm{P}}$ with respect to $\\varepsilon_k \\geq 0$, as follows:\n\t\\[\n\t\t\\mathcal{F}_k:= \\left\\{X\\in \\mathbb{S}^n\\; \\middle | \\; \\norm{\\mathcal{A}(X) - b}\\leq \\varepsilon_k,\\; X\\in \\mathbb{S}_+^n\\right\\},\\quad \\forall\\, k\\geq 1. \n\t\\]\n\tMoreover, denote\n\t\\[\n\t\t\\Xstar_k:=\\argmin\\left\\{\\inprod{C}{X}\\; \\middle | \\; X\\in \\mathcal{F}_k\\right\\},\\quad \\forall\\, k\\geq 1.\n\t\\]\n\tSince $(\\ystar, \\Sstar)\\in \\mathbb{R}^m\\times \\mathbb{S}_+^n$ is a dual optimal solution, by \\cite[Lemma 3.4]{Jiang2012}, we have that \n\t\\[\n\t\t0\\leq \\inprod{C}{\\Xstar - \\Xstar_k} \\leq \\norm{\\ystar}\\varepsilon_k,\\quad \\forall\\, k\\geq 1.\n\t\\]\n\tAs a consequence, because $X^k \\in \\calF_k$ and $\\inprod{C}{X^k} \\geq \\inprod{C}{\\Xstar_k}$, it holds that \n\t\\begin{equation}\n\t\\label{eq-thm-iPGM-5}\n\t\t\\inprod{C}{X^k - \\Xstar} \\geq \\inprod{C}{\\Xstar_k - \\Xstar} \\geq -\\norm{\\ystar}\\varepsilon_k = -\\frac{\\norm{\\ystar}k\\varepsilon_k}{k}.\t\n\t\\end{equation}\n\tCombining \\eqref{eq-thm-iPGM-4} and \\eqref{eq-thm-iPGM-5}, we see that \n\t\\[\n\t\t-\\frac{\\norm{\\ystar}k\\varepsilon_k}{k} \\leq \\inprod{C}{X^k - \\Xstar} \\leq  \\frac{1}{k}\\left(\\frac{1}{2\\sigma_0}\\norm{e^0}^2 + 2M\\sum_{i=1}^k i\\varepsilon_i\\right).\n\t\\]\n\t\n\tFinally, let us estimate the dual infeasibility. To this end, we get from \\eqref{eq-thm-iPGM-1} that\n\t\\[\n\t\t\\frac{1}{\\sigma_k}\\norm{X^k - X^{k-1}}^2 \\leq  \\inprod{C}{X^{k-1} - \\Xstar} - \\inprod{C}{X^{k} - \\Xstar} +  \\norm{y^k}\\left(\\norm{r^{k-1}} + \\norm{r^k}\\right)\n\t\\]\n\twhich implies that \n\t\\[\n\t\t\\begin{aligned}\n\t\t\t&\\; \\norm{\\mathcal{A}^*y^k + S^k - C} =  \\frac{1}{\\sigma_k}\\norm{X^{k} - X^{k-1}} \\\\\n\t\t\t\\leq &\\; \\frac{1}{\\sqrt{\\sigma_k}}\\sqrt{ \\frac{1}{k-1}\\left(\\frac{1}{2\\sigma_0}\\norm{e^0}^2 + 2M\\sum_{i=1}^{k-1} i\\varepsilon_i\\right) + \\norm{\\ystar}\\varepsilon_k + M\\left(\\varepsilon_{k-1} + \\varepsilon_k\\right)} \\\\ \n\t\t\t\\leq &\\; O\\left(\\frac{1}{\\sqrt{k\\sigma_k}}\\right),\n\t\t\\end{aligned} \n\t\\]\n\tand establishes the convergence of the dual infeasibility. Therefore, the proof is completed. \\qed\n\\end{proof}","sha256_utf8":"c343072e029273de6aea4da7c44223f1b16604cfb8fe3ab53e8d149d86f3a2d4"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"多项式优化问题","source_units":[{"start":31103,"end":31562,"start_line":913,"end_line":916,"raw_tex":"Let $p, h_1,\\dots, h_l \\in \\Real{}[x]$ be real-valued multivariate polynomials in $x \\in \\Real{d}$, we consider the following equality constrained polynomial optimization problem\\footnote{Although our framework can be generalized to problems with inequality constraints, here we focus on equality constraints to simplify the presentation.}\n\\begin{equation} \\label{eq:pop}\n\\min_{x \\in \\Real{d}} \\cbrace{p(x)\\ |\\ h_i(x)=0, i=1,\\dots,l}. \\tag{POP}\n\\end{equation}","sha256_utf8":"8f3678e080d57e1b5c31e86088ad432ba17f02833b064b5cc494232db53910f5"}]},{"id":"S002","semantic_type":"definition","kind":"definition","title":"单项式提升","source_units":[{"start":32555,"end":32948,"start_line":924,"end_line":924,"raw_tex":"{\\bf SDP Relaxations for Polynomial Optimization Problems}. Towards this goal, let us first give an overview of the celebrated Lasserre's moment/sums-of-squares (SOS) semidefinite relaxation hierarchy~\\cite{Lasserre01siopt-LasserreHierarchy,Parrilo03MP-SOS}. Let $\\kappa \\geq 1$ be an integer such that $2\\kappa$ is equal or greater than the maximum degree of $p,h_i,i=1,\\dots,l$, and denote \n","sha256_utf8":"c671364d39fb2fa0de3234ae9548dc48b12f9a68834f7ed5f41294c6a5e4bb84"},{"start":35116,"end":35415,"start_line":931,"end_line":933,"raw_tex":"$v := [x]_\\kappa$ as the standard vector of monomials in $x$ of degree up to $\\kappa$. We build the \\emph{moment matrix} $X := [x]_\\kappa [x]_\\kappa\\tran$ that contains the standard monomials in $x$ of degree up to $2\\kappa$. \nAs a result,\n$p$ and $h_i$'s can be written as linear functions in $X$.\n","sha256_utf8":"e29df74be3a10b86635b9b5dee145c8840d207ea0b40a26a96622fe541f4625c"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"原始对偶半定规划","source_units":[{"start":37006,"end":37141,"start_line":950,"end_line":952,"raw_tex":"\\begin{equation}\\label{eq:primalSDP}\n\\min_{X \\in \\sym{n}} \\cbrace{\\inprod{C}{X}\\ |\\ \\calA(X) = b, \\ X \\succeq 0},\\tag{P}\n\\end{equation}","sha256_utf8":"0af9bab08025fdddbd412ce72d8d5e8c2e4e81973fd3e80d76b51636cb263701"},{"start":37142,"end":37879,"start_line":953,"end_line":956,"raw_tex":"where $n := \\bard_\\kappa \\triangleq \\nchoosek{d+\\kappa}{\\kappa}$ is the dimension of $[x]_\\kappa$, $b \\in \\Real{m}$, $\\calA: \\sym{n} \\rightarrow \\Real{m}$ is a linear map $\\calA(X) \\triangleq (\\inprod{A_i}{X})_{i=1}^m$ with $A_i \\in \\sym{n}$, that is onto and collects all independent linear constraints generated by the relaxation \\blue{of $X =[x]_\\kappa [x]_\\kappa\\tran$ to $X \\succeq 0$}. Let $\\calAadj: \\Real{m} \\rightarrow \\sym{n}$ be the adjoint of $\\calA$ defined as $\\calAadj y \\triangleq \\sum_{i=1}^m y_i A_i$, then the Lagrangian dual of problem~\\eqref{eq:primalSDP} reads\n\\begin{equation}\\label{eq:dualSDP}\n\\max_{y \\in \\Real{m}, S \\in \\sym{n}} \\cbrace{\\inprod{b}{y}\\ |\\ \\calAadj y + S = C,\\ S \\succeq 0},\\tag{D}\n\\end{equation}","sha256_utf8":"a5076b00eeb9c5a038f36d3f7198576af498cc3fbf306a5abeca9e9ea1da83c0"}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","title":"强对偶与最优解存在","source_units":[{"start":38859,"end":39483,"start_line":962,"end_line":962,"raw_tex":"Throughout this paper, we assume that strong duality holds for \\eqref{eq:primalSDP} and \\eqref{eq:dualSDP}, and both \\eqref{eq:primalSDP} and \\eqref{eq:dualSDP} admit at least one solution.\\footnote{Particularly, if a redundant ball constraint is included in the relaxation, then dual Slater and strong duality hold for Lasserre's moment relaxations of all orders \\cite[Theorem 1]{Josz16OL-strongdualityLasserre}.}  We denote $\\Xstar$ and $(\\ystar,\\Sstar)$ as an optimal solution for \\eqref{eq:primalSDP} and \\eqref{eq:dualSDP}, respectively. Then, the KKT condition for \\eqref{eq:primalSDP} and \\eqref{eq:dualSDP} given as\n","sha256_utf8":"b300024aebc4be2199ca0e4d35973d4e19a18a0b76fa153dfb471921162110e6"}]},{"id":"S005","semantic_type":"derived_fact","kind":"derived_fact","title":"最优性条件与秩一对应","source_units":[{"start":39484,"end":39624,"start_line":963,"end_line":966,"raw_tex":"\\begin{equation}\n\t\\label{eq-kkt-sdp}\n\t\t\\cA(X) - b = 0,\\quad \\cA^*y + S - C = 0,\\quad \\inprod{X}{S} = 0,\\quad X, S\\succeq 0,\t\n\t\\end{equation}","sha256_utf8":"e63182667a76c9ac30e91e088349e214c04dade97b7efdfe4b4de06be3d8fed8"},{"start":39625,"end":40241,"start_line":967,"end_line":968,"raw_tex":"\thas $(\\Xstar,\\ystar,\\Sstar)$ as one of its solutions. Moreover, it follows that $\\fpopstar \\geq \\fpsdpstar = \\fdsdpstar$, where $\\fpsdpstar$ and $\\fdsdpstar$ are the optimal values of~\\eqref{eq:primalSDP} and~\\eqref{eq:dualSDP}, respectively. The SDP relaxation is said to be \\emph{tight} if $\\fpopstar = \\fpsdpstar$. In such cases, for any global minimizer $\\xstar$ of \\eqref{eq:pop}, the rank one lifting $X = [\\xstar]_\\kappa [\\xstar]_\\kappa\\tran$ is a minimizer of~\\eqref{eq:primalSDP}, \n\t{and any rank one optimal solution  $\\Xstar$ of \\eqref{eq:primalSDP} corresponds to a global minimizer of \\eqref{eq:pop}.}\n","sha256_utf8":"d96cca4cbff47237917c17ce3221dfb9246ed53f82e50943c98732ffeed23621"}]},{"id":"S006","semantic_type":"notation","kind":"notation","title":"矩阵空间与范数","source_units":[{"start":64633,"end":65438,"start_line":1139,"end_line":1139,"raw_tex":"We use $\\mathcal{X}$ and $\\mathcal{Y}$ to denote finite dimensional Euclidean spaces. Let $\\sym{n}$ be the space of real symmetric $n\\times n$ matrices, and $\\psd{n}$ (resp. $\\pd{n}$) be the set of positive semidefinite (resp. definite) matrices. We also write $X \\succeq 0$ (resp. $X \\succ 0$) to indicate that $X$ is positive semidefinite (resp. definite) when the dimension of $X$ is clear. We use $\\usphere{d-1}:=\\{ x \\in \\Real{d}\\mid \\norm{x} = 1 \\}$ to denote the $d$-dimensional unit sphere. For $x \\in \\Real{n}$, $\\norm{x} = \\sqrt{x\\tran x}$ is the standard $\\ell_2$ norm. For $X \\in \\Real{m \\times n}$, $\\norm{X} = \\sqrt{\\trace{X\\tran X}}$ denotes the Frobenius norm. $\\norm{x}_{\\calH} := \\sqrt{\\inprod{x}{\\calH x}}$ denotes the $\\calH$-weighted norm for a positive semidefinite mapping $\\calH$.\n","sha256_utf8":"174993dba71d594013cc2bfb83aef5ca92135db91a9d2be84ae12af8655e4802"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"原始可行集","source_units":[{"start":67831,"end":67983,"start_line":1173,"end_line":1175,"raw_tex":"\\begin{equation*}\n\t\\mathcal{F}_{\\mathrm{P}} := \\left\\{ X\\in \\mathbb{S}^n\\;\\middle |\\; \\mathcal{A}(X) = b,\\; X\\in \\mathbb{S}_+^n\\right\\},\n\\end{equation*}","sha256_utf8":"5ac5e940ed10a0b4a90e4e0602b93cfa2846995f85e694f3828420e187ebb1f3"}]},{"id":"S008","semantic_type":"algorithm","kind":"algorithm","title":"非精确投影梯度法","source_units":[{"start":68494,"end":69700,"start_line":1178,"end_line":1201,"raw_tex":"\\begin{algorithm}\n\t\\caption{An inexact projected gradient method (\\ipgm) for SDP \\eqref{eq:primalSDP}.}\n\t\\label{alg-iPGM}\n\t\\begin{algorithmic}\n\t\t\\item[{\\bf Input}:] Initial points $(X^0, y^0, S^0)\\in \\mathbb{S}^n_+ \\times \\mathbb{R}^m \\times \\mathbb{S}^n_+$, a nondecreasing positive sequence $\\{\\sigma_k\\}$ and a nonnegative sequence $\\{\\varepsilon_k\\}$ such that $\\{k\\epsilon_k\\}$ is summable.\n\t\t\\item[{\\bf Repeat} For $k \\geq 1$:] \n\t\t\\item[\\quad] Compute \n\t\t\\begin{equation}\n\t\t\\label{eq-iPGM-proj}\t\n\t\tX^{k} \\approx \\Pi_{\\mathcal{F}_{\\mathrm{P}}}\\left(X^{k-1} - \\sigma_k C\\right),\\quad X^{k} \\in \\mathbb{S}_+^n\n\t\t\\end{equation}\n\t\twith associated dual pair $(y^{k}, S^{k})\\in \\mathbb{R}^m\\times \\mathbb{S}_+^n$ such that the following conditions hold:\n\t\t\\begin{equation}\n\t\t\\label{eq-iPGM-inexact-conditions}\n\t\t\\begin{aligned}\n\t\t\t\\norm{\\mathcal{A}(X^{k}) - b} \\leq &\\; \\varepsilon_k, \\\\\n\t\t\t\t\\mathcal{A}^*y^k + S^k - C - \\frac{1}{\\sigma_k}\\left(X^{k} - X^{k-1}\\right)= &\\; 0, \\\\\n\t\t\t\t\\inprod{X^{k}}{S^{k}} = &\\;0.\n\t\t\t\\end{aligned}\n\t\t\\end{equation}\n\t\t\\item[{\\bf Until}:] Termination conditions are met (\\cf Section \\ref{sec:numexps}).\n\t\t\\item[{\\bf Output}:] $(X^k,y^k,S^k)$.\n\t\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"55de25917b27c162acf9f2478688ed9b4605ff58183d6016ed5dacb3a0c4e115"}]},{"id":"S009","semantic_type":"algorithm","kind":"algorithm","title":"舍入与候选生成","source_units":[{"start":76009,"end":77643,"start_line":1274,"end_line":1282,"raw_tex":"\t\\item\\label{item:rounding} {\\bf (Rounding)}. Let $\\barX^{k} = \\sum_{i=1}^n \\lambda_i v_i v_i\\tran$ be the spectral decomposition of $\\barX^{k}$ with $\\lambda_1 \\geq \\dots \\geq \\lambda_n$ in nonincreasing order. Compute $r \\geq 1$ hypotheses for the \\eqref{eq:pop} from the leading $r$ eigenvectors $v_1,\\dots, v_r$   \n\t\\bea\n\t\t\\bar{x}^{k}_{i} = \\rounding(v_i), \\quad i = 1,\\dots,r,\n\t\\eea\n\twhere the function~\\rounding~can be problem-dependent and we provide examples in the numerical experiments in Section~\\ref{sec:numexps}. Generally, if the SDP~\\eqref{eq:primalSDP} comes from a dense relaxation, and the feasible set of the original~\\eqref{eq:pop}, denoted as $\\setpop$, is simple to project, then we can design~\\rounding~to be\n\t\\bea\n\tv_i \\leftarrow \\frac{v_i}{v_i[1]}, \\quad \\bar{x}^{k}_{i} \\leftarrow \\Pi_{\\setpop}(v_i[x]),\n\t\\eea\n\twhere one first normalizes $v_i$ such that its leading entry $v_i[1]$ is equal to $1$, and then projects its entries corresponding to order-one monomials,~\\ie~$v_i[x]$, to the feasible set of~\\eqref{eq:pop}. Note that the rationale for designing this rounding method is that practical POP applications often involve simple constraints such as binary~\\cite{Goemans95JACM-maxcut}, unit sphere~\\cite{Yang19ICCV-wahba}, and orthogonality~\\cite{Yang20neurips-onering,Briales17CVPR-3Dregistration}, whose projection maps are simple (\\eg if $x$ is binary, then $\\Pi_{\\setpop}(v_i[x]) = \\sign\\parentheses{v_i[x]}$ just takes the sign of each entry of $v_i[x]$). If $\\setpop$ is not easy to project, then we omit the projection, in which case the local search will start from an infeasible initialization.\n","sha256_utf8":"ae37069e1275ebc39a807b2ef055b7e4c690bbaea377b0c20d3c657257dd0a75"}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","title":"局部搜索与秩一提升","source_units":[{"start":77643,"end":78571,"start_line":1283,"end_line":1292,"raw_tex":"\t\\item {\\bf (Local Search)}. Apply a local search method for the POP (as an NLP) with the initial point chosen as $ \\bar{x}^{k}_i $ for each hypothesis $i=1,\\dots,r$. Denote the solution of each local search as $ \\hat{x}^{k}_i $, with associated objective value $p(\\hatx^{k}_i)$, choose the best local solution with minimum objective value. Formally, we have\n\t\\bea\n\t\\hatx_i^{k} = \\nlp(\\barx^{k}_i),\\quad i=1,\\dots,r, \\quad \\hatx^{k} = \\argmin_{\\hatx^{k}_i, i=1\\dots,r} p(\\hatx^{k}_i).\n\t\\eea\n\n\t\\item\\label{item:lifting} {\\bf (Lifting)}. Perform a rank-one lifting of the best local solution according to the SDP relaxation scheme\n\t\\bea\n\t\\hatX^{k} = v\\parentheses{\\hatx^{k}} v\\parentheses{\\hatx^{k}}\\tran,\n\t\\eea\n\twhere $v: \\Real{d} \\rightarrow \\Real{n}$ is a dense or sparse monomial lifting (\\eg $v(x) = [x]_\\kappa$ is the full set of monomials up to degree $\\kappa$ in the case of dense Lasserre's hierarchy at order $\\kappa$).\n","sha256_utf8":"0bc96be9c3d92f3e638f4438031de47ccb5d2677d3023a57df2ae87d7b53afd9"}]},{"id":"S011","semantic_type":"algorithm","kind":"algorithm","title":"带保护策略的STRIDE","source_units":[{"start":79977,"end":82191,"start_line":1302,"end_line":1328,"raw_tex":"\\begin{algorithm}\n\t\\caption{An inexact projected gradient method accelerated \\blue{via rounding and lifting by} NLP for solving tight SDP relaxations of POPs (\\name).}\n\t\\label{alg-iPGMnlp}\n\t\\begin{algorithmic}\n\t\t\\item[{\\bf Input}:] Initial points $ (X^{0}, y^0, S^0)\\in \\mathbb{S}_+^n\\times \\mathbb{R}^{m}\\times\\mathbb{S}_+^n $, a nondecreasing positive sequences $ \\{\\sigma_k\\}$, a nonnegative sequence $\\{\\varepsilon_k\\}$ such that $\\{k\\varepsilon_k\\}$ is summable, a stopping criterion $\\eta:\\mathbb{S}^n_+\\times \\mathbb{R}^m\\times \\mathbb{S}_+^n\\rightarrow \\mathbb{R}_+$ with a tolerance $ {\\tt Tol} > 0 $. Initialize $\\calV = \\{ X^0 \\}$ if $X^0\\in \\setsdpp$ and $\\calV = \\emptyset$ otherwise. Choose a positive integer $r \\in [1,n]$, and a positive constant $\\epsilon>0$. \n\t\t\\item[{\\bf Iterate} the following steps for $ k = 1,\\dots $:]\n\t\t\\item[\\quad {\\bf Step 1 (Projection)}.] Compute $(\\overline{X}^k, y^k, S^{k})$ satisfying \\eqref{eq-iPGM-proj} and \\eqref{eq-iPGM-inexact-conditions} (see Section \\ref{sec:apgsubproblem}).\n\t\t\\item[\\quad {\\bf Step 2 (Certification)}.] If $ \\eta(\\barX^{k}, y^{k}, S^{k}) < {\\tt Tol} $, {\\bf output} $(\\barX^{k}, y^{k}, S^{k})$ and {\\bf stop}.\n\t\t\\item[\\quad {\\bf Step 3 (Acceleration)}.] Compute $ \\widehat{X}^{k} $ as in items~\\ref{item:rounding}-\\ref{item:lifting}:\n\t\t\\bea\n\t\t\t& \\barX^{k} = \\sum_{i=1}^n \\lambda_i v_i v_i\\tran, \\quad \\bar{x}^{k}_{i} = \\rounding(v_i), \\quad i = 1,\\dots,r, \\nonumber \\\\\n\t\t\t& \\hatx_i^{k} = \\nlp(\\barx^{k}_i), \\quad i=1,\\dots,r, \\quad \\hatx^{k} = \\argmin_{\\hatx^{k}_i, i=1\\dots,r} p(\\hatx^{k}_i), \\nonumber \\\\\n\t\t\t& \\hatX^{k} = v\\parentheses{\\hatx^{k}} v\\parentheses{\\hatx^{k}}\\tran. \\nonumber \n\t\t\\eea\n\t\t\\item[\\quad {\\bf Step 4 (Policy)}.] Update $X^{k}$ according to \n\t\t\\begin{equation}\n\t\t\t\\label{eq:accept-reject}\n\t\t\tX^{k} = \n\t\t\t\t\\begin{cases}\n\t\t\t\t\t\\hatX^{k} & {\\rm if }\\; \\inprod{C}{\\hatX^k} < \n\t\t\t\t\t\\blue{\\min \\left(\\inprod{C}{\\barX^k}, \\min_{X\\in \\calV} \\left\\{\\inprod{C}{X}\\;\\middle|\\; X\\in \\calV\\right\\}\\right)} - \\epsilon,\\; \\widehat{X}^{k} \\in \\setsdpp  \\\\\n\t\t\t\t\t\\barX^{k} & {\\rm otherwise }\n\t\t\t\t\\end{cases}.\n\t\t\\end{equation}\n\t\tIf $X^{k} = \\hatX^{k}$, set $\\calV \\leftarrow \\calV \\cup \\{ \\hatX^k \\}$.\n\t\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"ace96af0b28136ea59143d9db975f7188d579daf897b189d469ada3f51f1a880"}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","title":"STRIDE乘子有界","source_units":[{"start":84526,"end":84711,"start_line":1365,"end_line":1365,"raw_tex":"\tLet $\\{(\\barX^k, y^k, S^k)\\}$ be any sequence generated by Algorithm \\ref{alg-iPGMnlp}, and suppose that there exists a constant $M>0$ such that $\\norm{y^k}\\leq M$ for all $k\\geq 0$. \n","sha256_utf8":"306fdd66b46c4534c79773a3cdb2c3ec65e63e1b18d053f9e806a8ee264918c7"}]},{"id":"S013","semantic_type":"theorem","kind":"theorem","title":"STRIDE目标值全局收敛","source_units":[{"start":84489,"end":85160,"start_line":1363,"end_line":1368,"raw_tex":"\\begin{theorem}\n\t\\label{thm-ipgmnlp}\n\tLet $\\{(\\barX^k, y^k, S^k)\\}$ be any sequence generated by Algorithm \\ref{alg-iPGMnlp}, and suppose that there exists a constant $M>0$ such that $\\norm{y^k}\\leq M$ for all $k\\geq 0$. \n\t% \\red{Assuming Slater's conditions hold for SDP \\eqref{eq:primalSDP}-\\eqref{eq:dualSDP}} \\blue{(LIANG: I think we do not need Slater's conditions here for global convergence, but only need them for solving subproblem. See also the red part in Page 3.)}, \n\tIf one chooses $\\sigma_k \\geq O(\\sqrt{k})$ for all $k\\geq 1$, then, $\\left\\{\\inprod{C}{\\barX^k}\\right\\}$ converges to the optimal value of the SDP problem \\eqref{eq:primalSDP}. \n\\end{theorem}","sha256_utf8":"64104c6e50ca464f0ba72c28ac3254dd5c1ad81acabfd74bccdc39cee2ab2aa9"}]},{"id":"S014","semantic_type":"proof","kind":"proof","title":"STRIDE收敛完整证明","source_units":[{"start":85161,"end":86939,"start_line":1369,"end_line":1375,"raw_tex":"\\begin{proof}\nLet us only consider the case with $X^0 \\in \\setsdpp$ such that $\\calV$ is initialized as $\\{ X^0\\}$ (the case with $\\calV$ initialized as $\\emptyset$ can be argued in a similar manner). By the construction of the set $\\calV$, it contains a sequence of feasible points of~\\eqref{eq:primalSDP}, and the objective value along this sequence is \\emph{strictly} decreasing according to the acceptance policy~\\eqref{eq:accept-reject}. Hence, we have the following relation:\n\\begin{equation*}\n\t\\inf\\left\\{\\inprod{C}{X}\\;\\middle|\\; X\\in \\calV\\right\\} \\leq \\inprod{C}{X^0} - (|\\calV|-1) \\epsilon,\n\\end{equation*}\nIf $|\\calV|=\\infty$, then there exists an infinite sequence of feasible points to problem \\eqref{eq:primalSDP} whose objective function value converges to $-\\infty$. In this case, problem \\eqref{eq:primalSDP} is unbounded, a contradiction (recall that we assumed strong duality throughout this paper in Section \\ref{sec:introduction}). Hence, $|\\calV|< \\infty$. If $|\\calV| = 1$, then none of the $\\hatX^{k}$ generated by {\\bf Step 3} has been accepted, in which case Algorithm \\ref{alg-iPGMnlp} reduces to Algorithm \\ref{alg-iPGM} iPGM. Therefore, by Theorem~\\ref{thm-iPGM}, Algorithm \\ref{alg-iPGMnlp} converges as $O(1/k)$ if one chooses $\\sigma_k \\geq O(\\sqrt{k})$ for all $k\\geq 1$. If $|\\calV| > 1$, then some of the $\\hatX^{k}$ generated by {\\bf Step 3} have been accepted. Denote the last element of $\\calV$ as $\\hatX_{\\calV}$ with associated dual variables $(y_\\calV,S_\\calV)$ (from {\\bf Step 1}), then Algorithm \\ref{alg-iPGMnlp} essentially reduces to \\ipgm~with a new initial point at $(\\hatX_{\\calV}, y_\\calV, S_\\calV)$. Again, by Theorem~\\ref{thm-iPGM}, Algorithm \\ref{alg-iPGMnlp} converges as $O(1/k)$. This completes the proof.\\qed\n\\end{proof}","sha256_utf8":"3c0933b21f51cbfc45978285ffbbccb51ecef9957eb49f53b6d261906a7f30ea"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"接受顶点的定量下降","source_units":[{"start":85175,"end":85779,"start_line":1370,"end_line":1373,"raw_tex":"Let us only consider the case with $X^0 \\in \\setsdpp$ such that $\\calV$ is initialized as $\\{ X^0\\}$ (the case with $\\calV$ initialized as $\\emptyset$ can be argued in a similar manner). By the construction of the set $\\calV$, it contains a sequence of feasible points of~\\eqref{eq:primalSDP}, and the objective value along this sequence is \\emph{strictly} decreasing according to the acceptance policy~\\eqref{eq:accept-reject}. Hence, we have the following relation:\n\\begin{equation*}\n\t\\inf\\left\\{\\inprod{C}{X}\\;\\middle|\\; X\\in \\calV\\right\\} \\leq \\inprod{C}{X^0} - (|\\calV|-1) \\epsilon,\n\\end{equation*}\n","sha256_utf8":"fe930c0d8b939b21e8f10b0497fbd355dc0a354167a00cb26510256d6ce7cb07"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"有限次接受后归约iPGM","source_units":[{"start":85779,"end":86928,"start_line":1374,"end_line":1374,"raw_tex":"If $|\\calV|=\\infty$, then there exists an infinite sequence of feasible points to problem \\eqref{eq:primalSDP} whose objective function value converges to $-\\infty$. In this case, problem \\eqref{eq:primalSDP} is unbounded, a contradiction (recall that we assumed strong duality throughout this paper in Section \\ref{sec:introduction}). Hence, $|\\calV|< \\infty$. If $|\\calV| = 1$, then none of the $\\hatX^{k}$ generated by {\\bf Step 3} has been accepted, in which case Algorithm \\ref{alg-iPGMnlp} reduces to Algorithm \\ref{alg-iPGM} iPGM. Therefore, by Theorem~\\ref{thm-iPGM}, Algorithm \\ref{alg-iPGMnlp} converges as $O(1/k)$ if one chooses $\\sigma_k \\geq O(\\sqrt{k})$ for all $k\\geq 1$. If $|\\calV| > 1$, then some of the $\\hatX^{k}$ generated by {\\bf Step 3} have been accepted. Denote the last element of $\\calV$ as $\\hatX_{\\calV}$ with associated dual variables $(y_\\calV,S_\\calV)$ (from {\\bf Step 1}), then Algorithm \\ref{alg-iPGMnlp} essentially reduces to \\ipgm~with a new initial point at $(\\hatX_{\\calV}, y_\\calV, S_\\calV)$. Again, by Theorem~\\ref{thm-iPGM}, Algorithm \\ref{alg-iPGMnlp} converges as $O(1/k)$. This completes the proof.\\qed\n","sha256_utf8":"08fab998a1242268f2419058aa263ce9a6926fe7402bc8d48f8a04f3391162c5"}]},{"id":"S017","semantic_type":"theorem","kind":"theorem","title":"iPGM收敛与残差复杂度","source_units":[{"start":71017,"end":72233,"start_line":1232,"end_line":1248,"raw_tex":"\\begin{theorem}\n\t\\label{thm-iPGM}\n\tLet $\\{(X^k, y^k, S^k)\\}$ be any sequence generated by Algorithm \\ref{alg-iPGM}, and suppose that there exists a constant $M>0$ such that $\\norm{y^k}\\leq M$ for all $k\\geq 0$. Then, for all $k\\geq 1$, it holds that \n\t\\begin{equation*}\n\t\t\\left\\{\n\t\t\t\\begin{array}{l}\n\t\t\t\t\\displaystyle -\\frac{\\norm{y^*}k\\varepsilon_k}{k} \\leq \\inprod{C}{X^k - \\Xstar} \\leq \\frac{1}{k}\\left( \\frac{1}{2\\sigma_0}\\norm{X^0 - \\Xstar}^2 + 2M\\sum_{i = 1}^k i\\varepsilon_i\\right), \\\\\n\t\t\t\t\\displaystyle \\norm{\\mathcal{A}(X^k) - b} \\leq \\varepsilon_k \\leq O\\left(\\frac{1}{k}\\right), \\\\ \t\n\t\t\t\t\\displaystyle \\norm{\\mathcal{A}^*y^k + S^k - C} \\leq O\\left( \\frac{1}{\\sqrt{k\\sigma_k}}\\right),\n\t\t\t\\end{array}\n\t\t\\right.\n\t\\end{equation*}\n\twhere $\\Xstar$ is an optimal solution of the primal problem \\eqref{eq:primalSDP} and $(\\ystar, \\Sstar)$ is an optimal solution of the dual problem \\eqref{eq:dualSDP}. In particular, if one chooses $\\sigma_k \\geq O(\\sqrt{k}) > 0$ for all $k\\geq 1$, then we have \n\t\\begin{equation*}\n\t\t\\max\\left\\{ \\left\\lvert \\inprod{C}{X^k - \\Xstar}\\right\\rvert, \\norm{\\mathcal{A}(X^k) - b}, \\norm{\\mathcal{A}^*y^k + S^k - C}\\right\\} \\leq O\\left(\\frac{1}{k}\\right).\n\t\\end{equation*}\n\\end{theorem}","sha256_utf8":"5b19ecbc691a98e7fe2fa7ebc47107ff5baf287e9fde3baaadcc5248adec309d"}]},{"id":"S018","semantic_type":"assumption","kind":"assumption","title":"iPGM乘子有界","source_units":[{"start":71051,"end":71268,"start_line":1234,"end_line":1234,"raw_tex":"\tLet $\\{(X^k, y^k, S^k)\\}$ be any sequence generated by Algorithm \\ref{alg-iPGM}, and suppose that there exists a constant $M>0$ such that $\\norm{y^k}\\leq M$ for all $k\\geq 0$. Then, for all $k\\geq 1$, it holds that \n","sha256_utf8":"88f8e8d9d76c4298ea0030324f7ddbcfdcd6ce5345964c265a38df9c43b6c0cd"}]},{"id":"S019","semantic_type":"proof","kind":"proof","title":"iPGM完整收敛证明","source_units":[{"start":219820,"end":224882,"start_line":2818,"end_line":2889,"raw_tex":"\\begin{proof}\n\tFor notational simplicity, denote $r^k = \\mathcal{A}(X^k) - b$ and $e^k = X^k - \\Xstar$ for all $k\\geq 0$. Then, $\\mathcal{A}(e^k) = r^k$ since $\\mathcal{A}(\\Xstar) = b$. Moreover, we have $\\norm{r^k}\\leq \\varepsilon_k $ from the first condition in \\eqref{eq-iPGM-inexact-conditions}. Therefore, the convergence of $\\left\\{\\norm{r^k}\\right\\}$ is obvious, since $\\norm{r^k}\\leq \\varepsilon_k$ and $\\{k\\varepsilon_k\\}$ is summable. Recall that $\\Xstar$ is an optimal solution to problem \\eqref{eq:primalSDP}. Let $X = X^{k-1}$ in Lemma \\ref{lemma-fk}, we get \n\t\\begin{equation}\n\t\\label{eq-thm-iPGM-1}\n\t\t\\inprod{C}{X^{k-1} - X^k} \\geq \\frac{1}{\\sigma_k}\\norm{e^k - e^{k-1}}^2 + \\inprod{\\mathcal{A}^*y^k}{e^{k-1} - e^{k}}\t\n\t\\end{equation}\n\tOn the other hand, let $X = \\Xstar$ in Lemma \\ref{lemma-fk}, we obtain\n\t\\begin{equation}\n\t\\label{eq-thm-iPGM-2}\n\t\t\\inprod{C}{\\Xstar - X^k} \\geq \\frac{1}{\\sigma_k}\\norm{e^k - e^{k-1}}^2 + \\frac{1}{\\sigma_k}\\inprod{e^{k-1}}{e^k - e^{k-1}} - \\inprod{A^*y^k}{e^k}.\t\n\t\\end{equation}\n\tMultiplying $k-1$ to \\eqref{eq-thm-iPGM-1} and add the resulting inequality to \\eqref{eq-thm-iPGM-2} yields\n\t\\begin{equation}\n\t\t\\label{eq-thm-iPGM-3}\n\t\t\\begin{aligned}\n\t\t\t&\\; \\inprod{C}{(k-1)\\left(X^{k-1} - \\Xstar \\right) - k\\left(X^k - \\Xstar\\right)  } \\\\ \n\t\t\t\\geq &\\; \\frac{k}{\\sigma_k}\\norm{e^k - e^{k-1}}^2 + \\frac{1}{\\sigma_k}\\inprod{e^{k-1}}{e^k - e^{k-1}} + \\inprod{\\mathcal{A}^*y^k}{(k-1)e^{k-1} - ke^k} \\\\\n\t\t\t= &\\; \\frac{1}{\\sigma_k}\\left(k\\norm{e^k}^2 + (k-1)\\norm{e^{k-1}}^2 - (2k -1)\\inprod{e^k}{e^{k-1}}\\right) + \\inprod{y^k}{(k-1)r^{k-1} - k r^k} \\\\\n\t\t\t\\geq &\\; \\frac{1}{\\sigma_k}\\left(k\\norm{e^k}^2 + (k-1)\\norm{e^{k-1}}^2 - \\frac{2k-1}{2}\\left(\\norm{e^k}^2 + \\norm{e^{k-1}}^2\\right)\\right) + \\inprod{y^k}{(k-1)r^{k-1} - k r^k} \\\\\n\t\t\t = &\\; \\frac{1}{2\\sigma_k}\\norm{e^k}^2 - \\frac{1}{2\\sigma_k}\\norm{e^{k-1}}^2 + \\inprod{y^k}{(k-1)r^{k-1} - k r^k} \\\\\n\t\t\t \\geq &\\; \\frac{1}{2\\sigma_k}\\norm{e^k}^2 - \\frac{1}{2\\sigma_{k-1}}\\norm{e^{k-1}}^2 + \\inprod{y^k}{(k-1)r^{k-1} - k r^k},\n\t\t\\end{aligned} \n\t\\end{equation}\n\twhere in the last inequality, we use the fact that $\\sigma_k \\geq \\sigma_{k-1} > 0$ for all $k\\geq 1$. By summing the inequality in \\eqref{eq-thm-iPGM-3} for $k=1,\\dots,k$, we get (recall that $\\norm{y^k}\\leq M$ for all $k\\geq 0$)\n\t\\begin{equation*}\n\t\\begin{aligned}\n\t\t\t-k \\inprod{C}{X^k - \\Xstar} \\geq &\\; \\frac{1}{2\\sigma_k}\\norm{e^k}^2 - \\frac{1}{2\\sigma_0}\\norm{e_0}^2 + \\sum_{i=1}^k\\inprod{y^i}{(i-1)r^{i-1} - i r^i} \\\\\n\t\t\t\\geq &\\; \\frac{1}{2\\sigma_k}\\norm{e^k}^2 - \\frac{1}{2\\sigma_0}\\norm{e_0}^2 - 2M\\sum_{i=1}^k i\\varepsilon_i.\n\t\t\\end{aligned}\t\n\t\\end{equation*}\n\tThe above inequality leads to the following upper bound on $\\inprod{C}{X^k - \\Xstar}$:\n\t\\begin{equation}\n\t\\label{eq-thm-iPGM-4}\n\t\\inprod{C}{X^k - \\Xstar} \\leq \\frac{1}{k}\\left(\\frac{1}{2\\sigma_0}\\norm{e^0}^2 + 2M\\sum_{i=1}^k i\\varepsilon_i\\right).\t\n\t\\end{equation}\n\t\n\tNext, we shall provide a lower bound for $\\inprod{C}{X^k - \\Xstar}$ for $k\\geq 1$. To this end, let us define the set $\\mathcal{F}_k$, which is an enlargement of the primal feasible set $\\mathcal{F}_{\\mathrm{P}}$ with respect to $\\varepsilon_k \\geq 0$, as follows:\n\t\\[\n\t\t\\mathcal{F}_k:= \\left\\{X\\in \\mathbb{S}^n\\; \\middle | \\; \\norm{\\mathcal{A}(X) - b}\\leq \\varepsilon_k,\\; X\\in \\mathbb{S}_+^n\\right\\},\\quad \\forall\\, k\\geq 1. \n\t\\]\n\tMoreover, denote\n\t\\[\n\t\t\\Xstar_k:=\\argmin\\left\\{\\inprod{C}{X}\\; \\middle | \\; X\\in \\mathcal{F}_k\\right\\},\\quad \\forall\\, k\\geq 1.\n\t\\]\n\tSince $(\\ystar, \\Sstar)\\in \\mathbb{R}^m\\times \\mathbb{S}_+^n$ is a dual optimal solution, by \\cite[Lemma 3.4]{Jiang2012}, we have that \n\t\\[\n\t\t0\\leq \\inprod{C}{\\Xstar - \\Xstar_k} \\leq \\norm{\\ystar}\\varepsilon_k,\\quad \\forall\\, k\\geq 1.\n\t\\]\n\tAs a consequence, because $X^k \\in \\calF_k$ and $\\inprod{C}{X^k} \\geq \\inprod{C}{\\Xstar_k}$, it holds that \n\t\\begin{equation}\n\t\\label{eq-thm-iPGM-5}\n\t\t\\inprod{C}{X^k - \\Xstar} \\geq \\inprod{C}{\\Xstar_k - \\Xstar} \\geq -\\norm{\\ystar}\\varepsilon_k = -\\frac{\\norm{\\ystar}k\\varepsilon_k}{k}.\t\n\t\\end{equation}\n\tCombining \\eqref{eq-thm-iPGM-4} and \\eqref{eq-thm-iPGM-5}, we see that \n\t\\[\n\t\t-\\frac{\\norm{\\ystar}k\\varepsilon_k}{k} \\leq \\inprod{C}{X^k - \\Xstar} \\leq  \\frac{1}{k}\\left(\\frac{1}{2\\sigma_0}\\norm{e^0}^2 + 2M\\sum_{i=1}^k i\\varepsilon_i\\right).\n\t\\]\n\t\n\tFinally, let us estimate the dual infeasibility. To this end, we get from \\eqref{eq-thm-iPGM-1} that\n\t\\[\n\t\t\\frac{1}{\\sigma_k}\\norm{X^k - X^{k-1}}^2 \\leq  \\inprod{C}{X^{k-1} - \\Xstar} - \\inprod{C}{X^{k} - \\Xstar} +  \\norm{y^k}\\left(\\norm{r^{k-1}} + \\norm{r^k}\\right)\n\t\\]\n\twhich implies that \n\t\\[\n\t\t\\begin{aligned}\n\t\t\t&\\; \\norm{\\mathcal{A}^*y^k + S^k - C} =  \\frac{1}{\\sigma_k}\\norm{X^{k} - X^{k-1}} \\\\\n\t\t\t\\leq &\\; \\frac{1}{\\sqrt{\\sigma_k}}\\sqrt{ \\frac{1}{k-1}\\left(\\frac{1}{2\\sigma_0}\\norm{e^0}^2 + 2M\\sum_{i=1}^{k-1} i\\varepsilon_i\\right) + \\norm{\\ystar}\\varepsilon_k + M\\left(\\varepsilon_{k-1} + \\varepsilon_k\\right)} \\\\ \n\t\t\t\\leq &\\; O\\left(\\frac{1}{\\sqrt{k\\sigma_k}}\\right),\n\t\t\\end{aligned} \n\t\\]\n\tand establishes the convergence of the dual infeasibility. Therefore, the proof is completed. \\qed\n\\end{proof}","sha256_utf8":"c343072e029273de6aea4da7c44223f1b16604cfb8fe3ab53e8d149d86f3a2d4"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"残差记号与单步下降","source_units":[{"start":219834,"end":220570,"start_line":2819,"end_line":2823,"raw_tex":"\tFor notational simplicity, denote $r^k = \\mathcal{A}(X^k) - b$ and $e^k = X^k - \\Xstar$ for all $k\\geq 0$. Then, $\\mathcal{A}(e^k) = r^k$ since $\\mathcal{A}(\\Xstar) = b$. Moreover, we have $\\norm{r^k}\\leq \\varepsilon_k $ from the first condition in \\eqref{eq-iPGM-inexact-conditions}. Therefore, the convergence of $\\left\\{\\norm{r^k}\\right\\}$ is obvious, since $\\norm{r^k}\\leq \\varepsilon_k$ and $\\{k\\varepsilon_k\\}$ is summable. Recall that $\\Xstar$ is an optimal solution to problem \\eqref{eq:primalSDP}. Let $X = X^{k-1}$ in Lemma \\ref{lemma-fk}, we get \n\t\\begin{equation}\n\t\\label{eq-thm-iPGM-1}\n\t\t\\inprod{C}{X^{k-1} - X^k} \\geq \\frac{1}{\\sigma_k}\\norm{e^k - e^{k-1}}^2 + \\inprod{\\mathcal{A}^*y^k}{e^{k-1} - e^{k}}\t\n\t\\end{equation}\n","sha256_utf8":"af8a51f1f289064c6666e3ca69c8c9c00a3fe81267bccfc9d21044e0b9f52f31"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"相对最优解的下降式","source_units":[{"start":220570,"end":220849,"start_line":2824,"end_line":2828,"raw_tex":"\tOn the other hand, let $X = \\Xstar$ in Lemma \\ref{lemma-fk}, we obtain\n\t\\begin{equation}\n\t\\label{eq-thm-iPGM-2}\n\t\t\\inprod{C}{\\Xstar - X^k} \\geq \\frac{1}{\\sigma_k}\\norm{e^k - e^{k-1}}^2 + \\frac{1}{\\sigma_k}\\inprod{e^{k-1}}{e^k - e^{k-1}} - \\inprod{A^*y^k}{e^k}.\t\n\t\\end{equation}\n","sha256_utf8":"7113845e7f913579d628ecbcf2b0d18fd663d2e70d9cf6d25b2a386804a60fcf"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"加权合并与势函数差","source_units":[{"start":220849,"end":221874,"start_line":2829,"end_line":2840,"raw_tex":"\tMultiplying $k-1$ to \\eqref{eq-thm-iPGM-1} and add the resulting inequality to \\eqref{eq-thm-iPGM-2} yields\n\t\\begin{equation}\n\t\t\\label{eq-thm-iPGM-3}\n\t\t\\begin{aligned}\n\t\t\t&\\; \\inprod{C}{(k-1)\\left(X^{k-1} - \\Xstar \\right) - k\\left(X^k - \\Xstar\\right)  } \\\\ \n\t\t\t\\geq &\\; \\frac{k}{\\sigma_k}\\norm{e^k - e^{k-1}}^2 + \\frac{1}{\\sigma_k}\\inprod{e^{k-1}}{e^k - e^{k-1}} + \\inprod{\\mathcal{A}^*y^k}{(k-1)e^{k-1} - ke^k} \\\\\n\t\t\t= &\\; \\frac{1}{\\sigma_k}\\left(k\\norm{e^k}^2 + (k-1)\\norm{e^{k-1}}^2 - (2k -1)\\inprod{e^k}{e^{k-1}}\\right) + \\inprod{y^k}{(k-1)r^{k-1} - k r^k} \\\\\n\t\t\t\\geq &\\; \\frac{1}{\\sigma_k}\\left(k\\norm{e^k}^2 + (k-1)\\norm{e^{k-1}}^2 - \\frac{2k-1}{2}\\left(\\norm{e^k}^2 + \\norm{e^{k-1}}^2\\right)\\right) + \\inprod{y^k}{(k-1)r^{k-1} - k r^k} \\\\\n\t\t\t = &\\; \\frac{1}{2\\sigma_k}\\norm{e^k}^2 - \\frac{1}{2\\sigma_k}\\norm{e^{k-1}}^2 + \\inprod{y^k}{(k-1)r^{k-1} - k r^k} \\\\\n\t\t\t \\geq &\\; \\frac{1}{2\\sigma_k}\\norm{e^k}^2 - \\frac{1}{2\\sigma_{k-1}}\\norm{e^{k-1}}^2 + \\inprod{y^k}{(k-1)r^{k-1} - k r^k},\n\t\t\\end{aligned} \n\t\\end{equation}\n","sha256_utf8":"00d323760cdd4c3a84537354bb666d2c76ac3943ff0782a694f76cb9cc2ee39b"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"望远镜求和控制误差","source_units":[{"start":221874,"end":222445,"start_line":2841,"end_line":2847,"raw_tex":"\twhere in the last inequality, we use the fact that $\\sigma_k \\geq \\sigma_{k-1} > 0$ for all $k\\geq 1$. By summing the inequality in \\eqref{eq-thm-iPGM-3} for $k=1,\\dots,k$, we get (recall that $\\norm{y^k}\\leq M$ for all $k\\geq 0$)\n\t\\begin{equation*}\n\t\\begin{aligned}\n\t\t\t-k \\inprod{C}{X^k - \\Xstar} \\geq &\\; \\frac{1}{2\\sigma_k}\\norm{e^k}^2 - \\frac{1}{2\\sigma_0}\\norm{e_0}^2 + \\sum_{i=1}^k\\inprod{y^i}{(i-1)r^{i-1} - i r^i} \\\\\n\t\t\t\\geq &\\; \\frac{1}{2\\sigma_k}\\norm{e^k}^2 - \\frac{1}{2\\sigma_0}\\norm{e_0}^2 - 2M\\sum_{i=1}^k i\\varepsilon_i.\n\t\t\\end{aligned}\t\n\t\\end{equation*}\n","sha256_utf8":"33b1b4ade63202f60c6da049add4efc9a92a3432158202f558891dccdf027d88"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"目标误差上界","source_units":[{"start":222445,"end":222711,"start_line":2848,"end_line":2852,"raw_tex":"\tThe above inequality leads to the following upper bound on $\\inprod{C}{X^k - \\Xstar}$:\n\t\\begin{equation}\n\t\\label{eq-thm-iPGM-4}\n\t\\inprod{C}{X^k - \\Xstar} \\leq \\frac{1}{k}\\left(\\frac{1}{2\\sigma_0}\\norm{e^0}^2 + 2M\\sum_{i=1}^k i\\varepsilon_i\\right).\t\n\t\\end{equation}\n","sha256_utf8":"4a65e54810a4ed773931e67f63f666a53ac45ae7131d3e3deed438c02bf6cc35"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"扩大可行集构造","source_units":[{"start":222713,"end":223279,"start_line":2854,"end_line":2861,"raw_tex":"\tNext, we shall provide a lower bound for $\\inprod{C}{X^k - \\Xstar}$ for $k\\geq 1$. To this end, let us define the set $\\mathcal{F}_k$, which is an enlargement of the primal feasible set $\\mathcal{F}_{\\mathrm{P}}$ with respect to $\\varepsilon_k \\geq 0$, as follows:\n\t\\[\n\t\t\\mathcal{F}_k:= \\left\\{X\\in \\mathbb{S}^n\\; \\middle | \\; \\norm{\\mathcal{A}(X) - b}\\leq \\varepsilon_k,\\; X\\in \\mathbb{S}_+^n\\right\\},\\quad \\forall\\, k\\geq 1. \n\t\\]\n\tMoreover, denote\n\t\\[\n\t\t\\Xstar_k:=\\argmin\\left\\{\\inprod{C}{X}\\; \\middle | \\; X\\in \\mathcal{F}_k\\right\\},\\quad \\forall\\, k\\geq 1.\n\t\\]\n","sha256_utf8":"0bd8eedcdedc0d93195f297c43019e83d48c3758b2a4ea444050bca272c0f3cf"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"应用对偶扰动界","source_units":[{"start":223279,"end":223519,"start_line":2862,"end_line":2865,"raw_tex":"\tSince $(\\ystar, \\Sstar)\\in \\mathbb{R}^m\\times \\mathbb{S}_+^n$ is a dual optimal solution, by \\cite[Lemma 3.4]{Jiang2012}, we have that \n\t\\[\n\t\t0\\leq \\inprod{C}{\\Xstar - \\Xstar_k} \\leq \\norm{\\ystar}\\varepsilon_k,\\quad \\forall\\, k\\geq 1.\n\t\\]\n","sha256_utf8":"3ad0caa29938449f30b17c0241ed5fb24cc68aef1d0f156f9e447bda0d99e4e7"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"目标误差双侧界","source_units":[{"start":223519,"end":224070,"start_line":2866,"end_line":2874,"raw_tex":"\tAs a consequence, because $X^k \\in \\calF_k$ and $\\inprod{C}{X^k} \\geq \\inprod{C}{\\Xstar_k}$, it holds that \n\t\\begin{equation}\n\t\\label{eq-thm-iPGM-5}\n\t\t\\inprod{C}{X^k - \\Xstar} \\geq \\inprod{C}{\\Xstar_k - \\Xstar} \\geq -\\norm{\\ystar}\\varepsilon_k = -\\frac{\\norm{\\ystar}k\\varepsilon_k}{k}.\t\n\t\\end{equation}\n\tCombining \\eqref{eq-thm-iPGM-4} and \\eqref{eq-thm-iPGM-5}, we see that \n\t\\[\n\t\t-\\frac{\\norm{\\ystar}k\\varepsilon_k}{k} \\leq \\inprod{C}{X^k - \\Xstar} \\leq  \\frac{1}{k}\\left(\\frac{1}{2\\sigma_0}\\norm{e^0}^2 + 2M\\sum_{i=1}^k i\\varepsilon_i\\right).\n\t\\]\n","sha256_utf8":"3158398d16422810c1d7b4b95f632c26f2d8d63bb37a4f57fad9b05a74c50dfd"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"迭代增量估计","source_units":[{"start":224072,"end":224343,"start_line":2876,"end_line":2879,"raw_tex":"\tFinally, let us estimate the dual infeasibility. To this end, we get from \\eqref{eq-thm-iPGM-1} that\n\t\\[\n\t\t\\frac{1}{\\sigma_k}\\norm{X^k - X^{k-1}}^2 \\leq  \\inprod{C}{X^{k-1} - \\Xstar} - \\inprod{C}{X^{k} - \\Xstar} +  \\norm{y^k}\\left(\\norm{r^{k-1}} + \\norm{r^k}\\right)\n\t\\]\n","sha256_utf8":"d02a3c2ca1e9c4ecb2413b563104317adc29a6368cf28e3601f64e1ce54aa49b"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"对偶不可行性收敛率","source_units":[{"start":224343,"end":224871,"start_line":2880,"end_line":2888,"raw_tex":"\twhich implies that \n\t\\[\n\t\t\\begin{aligned}\n\t\t\t&\\; \\norm{\\mathcal{A}^*y^k + S^k - C} =  \\frac{1}{\\sigma_k}\\norm{X^{k} - X^{k-1}} \\\\\n\t\t\t\\leq &\\; \\frac{1}{\\sqrt{\\sigma_k}}\\sqrt{ \\frac{1}{k-1}\\left(\\frac{1}{2\\sigma_0}\\norm{e^0}^2 + 2M\\sum_{i=1}^{k-1} i\\varepsilon_i\\right) + \\norm{\\ystar}\\varepsilon_k + M\\left(\\varepsilon_{k-1} + \\varepsilon_k\\right)} \\\\ \n\t\t\t\\leq &\\; O\\left(\\frac{1}{\\sqrt{k\\sigma_k}}\\right),\n\t\t\\end{aligned} \n\t\\]\n\tand establishes the convergence of the dual infeasibility. Therefore, the proof is completed. \\qed\n","sha256_utf8":"84bc9ab8577673978c808e653e8740d3324fec7b0a4e55c6139de828196c422b"}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","title":"非精确投影基本不等式","source_units":[{"start":218313,"end":218647,"start_line":2792,"end_line":2798,"raw_tex":"\\begin{lemma}\n\\label{lemma-fk}\n\tUnder the inexactness conditions in \\eqref{eq-iPGM-inexact-conditions}, for any $X\\in \\mathbb{S}_+^n$, it holds that\n\t\\[\n\t\t\\inprod{C}{X - X^k} \\geq \\frac{1}{\\sigma_k}\\norm{X^{k} - X^{k-1}}^2 + \\frac{1}{\\sigma_k}\\inprod{X^{k-1} - X}{X^{k} - X^{k-1}} + \\inprod{\\mathcal{A}^*y^k}{X - X^k}.\n\t\\]\n\\end{lemma}","sha256_utf8":"4ede6e53d8568ab959e4588f5aa243adc0788384fdea76a78195aae95f9df345"}]},{"id":"S031","semantic_type":"proof","kind":"proof","title":"基本不等式完整证明","source_units":[{"start":218648,"end":219746,"start_line":2799,"end_line":2814,"raw_tex":"\\begin{proof}\n\tFrom the last two conditions in \\eqref{eq-iPGM-inexact-conditions}, $\\mathcal{A}^*y^k + S^k - C = \\frac{1}{\\sigma_k}\\left(X^k - X^{k-1}\\right)$ and $\\inprod{X^k}{S^k} = 0$, we have that\n\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\inprod{C}{X - X^k} = &\\; \\inprod{\\mathcal{A}^*y^k + S^k - \\frac{1}{\\sigma_k}\\left(X^k - X^{k-1}\\right)}{X - X^k} \\\\\n\t\t\t= &\\; \\frac{1}{\\sigma_k} \\inprod{X^k - X^{k-1}}{X^k - X} + \\inprod{\\mathcal{A}^*y^k}{X - X^k} + \\underbrace{\\inprod{S^k}{X}}_{\\geq 0} - \\underbrace{\\inprod{S^k}{X^k}}_{ = 0} \\\\\n\t\t\t\\geq &\\; \\frac{1}{\\sigma_k}\\norm{X^{k} - X^{k-1}}^2 + \\frac{1}{\\sigma_k}\\inprod{X^{k-1} - X}{X^{k} - X^{k-1}} + \\inprod{\\mathcal{A}^*y^k}{X - X^k},\n\t\t\\end{aligned}\n\t\\]\n\twhere the last inequality is due to the following fact:\n\t\\bea\n\t\t\\frac{1}{\\sigma_k} \\inprod{X^k - X^{k-1}}{X^k - X} = &  \\displaystyle \\frac{1}{\\sigma_k} \\inprod{X^k- X^{k-1}}{X^k -X^{k-1} + X^{k-1} - X} \\nonumber \\\\\n\t\t = & \\displaystyle \\frac{1}{\\sigma_k}\\norm{X^{k} - X^{k-1}}^2 + \\frac{1}{\\sigma_k}\\inprod{X^{k-1} - X}{X^{k} - X^{k-1}}. \\nonumber \n\t\\eea\n\tThus, the proof is completed. \\qed\n\\end{proof}","sha256_utf8":"0e0ca63eb4d4083964027f95307949a0ea00ebb857a03e19f47a241818685288"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"代入平稳性与互补性","source_units":[{"start":218662,"end":219342,"start_line":2800,"end_line":2807,"raw_tex":"\tFrom the last two conditions in \\eqref{eq-iPGM-inexact-conditions}, $\\mathcal{A}^*y^k + S^k - C = \\frac{1}{\\sigma_k}\\left(X^k - X^{k-1}\\right)$ and $\\inprod{X^k}{S^k} = 0$, we have that\n\t\\[\n\t\t\\begin{aligned}\n\t\t\t\\inprod{C}{X - X^k} = &\\; \\inprod{\\mathcal{A}^*y^k + S^k - \\frac{1}{\\sigma_k}\\left(X^k - X^{k-1}\\right)}{X - X^k} \\\\\n\t\t\t= &\\; \\frac{1}{\\sigma_k} \\inprod{X^k - X^{k-1}}{X^k - X} + \\inprod{\\mathcal{A}^*y^k}{X - X^k} + \\underbrace{\\inprod{S^k}{X}}_{\\geq 0} - \\underbrace{\\inprod{S^k}{X^k}}_{ = 0} \\\\\n\t\t\t\\geq &\\; \\frac{1}{\\sigma_k}\\norm{X^{k} - X^{k-1}}^2 + \\frac{1}{\\sigma_k}\\inprod{X^{k-1} - X}{X^{k} - X^{k-1}} + \\inprod{\\mathcal{A}^*y^k}{X - X^k},\n\t\t\\end{aligned}\n\t\\]\n","sha256_utf8":"8cfccc027df9c3d05e17246dcc662a85d959a4518cd373fe3264a1724825fc2b"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"展开增量内积完成引理","source_units":[{"start":219342,"end":219735,"start_line":2808,"end_line":2813,"raw_tex":"\twhere the last inequality is due to the following fact:\n\t\\bea\n\t\t\\frac{1}{\\sigma_k} \\inprod{X^k - X^{k-1}}{X^k - X} = &  \\displaystyle \\frac{1}{\\sigma_k} \\inprod{X^k- X^{k-1}}{X^k -X^{k-1} + X^{k-1} - X} \\nonumber \\\\\n\t\t = & \\displaystyle \\frac{1}{\\sigma_k}\\norm{X^{k} - X^{k-1}}^2 + \\frac{1}{\\sigma_k}\\inprod{X^{k-1} - X}{X^{k} - X^{k-1}}. \\nonumber \n\t\\eea\n\tThus, the proof is completed. \\qed\n","sha256_utf8":"bd90d93352727101bcaabff874c6d49b38bb07af2551ce77dc60ff92c63ac32e"}]},{"id":"S034","semantic_type":"external_fact","kind":"external_fact","title":"标准对偶扰动界","source_units":[],"declared_external":true,"declaration_text":["Jiang2012, Lemma 3.4: Let X* and (y*,S*) be attained primal and dual SDP optima with equal objective values. If Fε={X⪰0: ||A(X)-b||≤ε} and X*ε minimizes <C,X> over Fε, then 0≤<C,X*−X*ε>≤||y*||ε."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S013","edges":[{"source_id":"S001","target_id":"S009","kind":"supports"},{"source_id":"S001","target_id":"S010","kind":"supports"},{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S010","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S003","target_id":"S007","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S006","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S011","kind":"supports"},{"source_id":"S009","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S011","target_id":"S015","kind":"supports"},{"source_id":"S007","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S004","target_id":"S016","kind":"supports"},{"source_id":"S012","target_id":"S016","kind":"supports"},{"source_id":"S011","target_id":"S016","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S016","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S008","target_id":"S032","kind":"supports"},{"source_id":"S006","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S020","kind":"supports"},{"source_id":"S030","target_id":"S021","kind":"supports"},{"source_id":"S005","target_id":"S020","kind":"supports"},{"source_id":"S008","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S008","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S018","target_id":"S023","kind":"supports"},{"source_id":"S008","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S003","target_id":"S025","kind":"supports"},{"source_id":"S007","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S004","target_id":"S026","kind":"supports"},{"source_id":"S005","target_id":"S026","kind":"supports"},{"source_id":"S034","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S024","target_id":"S027","kind":"supports"},{"source_id":"S020","target_id":"S027","kind":"supports"},{"source_id":"S020","target_id":"S028","kind":"supports"},{"source_id":"S018","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S027","target_id":"S029","kind":"supports"},{"source_id":"S008","target_id":"S029","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S019","kind":"supports"},{"source_id":"S022","target_id":"S019","kind":"supports"},{"source_id":"S023","target_id":"S019","kind":"supports"},{"source_id":"S024","target_id":"S019","kind":"supports"},{"source_id":"S025","target_id":"S019","kind":"supports"},{"source_id":"S026","target_id":"S019","kind":"supports"},{"source_id":"S027","target_id":"S019","kind":"supports"},{"source_id":"S028","target_id":"S019","kind":"supports"},{"source_id":"S029","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S017","kind":"supports"}]}}
{"record_id":"2106.08020v3","paper":{"title":"A note on the optimal convergence rate of descent methods with fixed step sizes for smooth strongly convex functions","authors":["André Uschmajew","Bart Vandereycken"],"arxiv_id":"2106.08020","version":"v3","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-022-02032-z","abstract_url":"https://arxiv.org/abs/2106.08020v3","pdf_url":"https://arxiv.org/pdf/2106.08020v3"},"context":{"raw_tex":"An $L$-smooth and $\\mu$-strongly convex function $f \\colon \\R^n \\to \\R$ is  characterized by the two properties\n\\begin{equation*}\\label{eq: Lipschitz condition}\n\\| \\nabla f(x) - \\nabla f(y) \\| \\le \\Lip \\| x - y \\|\n\\end{equation*}\n\nand\n\\begin{equation*}\\label{eq: strong convexity}\n f(x) \\ge f(y) + \\langle \\nabla f(y), x - y \\rangle + \\frac{\\mu}{2} \\| x - y \\|^2 \n\\end{equation*}\n\nfor some constants $0 < \\mu \\le L$ and all $x,y \\in \\R^n$. Here, $\\langle \\ , \\  \\rangle$ can be any inner product on $\\R^n$ with corresponding norm $\\| \\cdot \\|$, and $\\nabla f$ denotes the gradient with respect to this inner product. Note that the  constants $\\mu$ and $L$ depend on the chosen inner product. The class of such functions plays a main role in the convergence theory of the gradient method and related descent methods for finding the unique global minimum $x^*$ of a given $f$. The update rule of the gradient method is\n\\[\n x^+ = x - h \\nabla f(x),\n\\]\n\nIn a more general setting of proximal gradient methods, it has recently been shown by Taylor, Hendrickx, and Glineur~\\cite[Theorem~3.3 with $h=0$]{Taylor2018} that the same rate is also valid for the error in function value. Specifically, for any \n\\begin{equation}\\label{eq: feasible step sizes}\n 0 \\le h \\le \\frac{2}{L + \\mu}\n\\end{equation}\n\nit holds that\n\\begin{equation}\\label{eq: function values general h}\n f(x^+) - f(x^*) \\le (1 - h\\mu)^2 (f(x) - f(x^*)).\n\\end{equation}\n\nWe first consider the variable metric method. Here the update rule reads\n\\begin{equation}\\label{eq: variable metric method}\n x^+ = x - h A^{-1} \\nabla f(x),\n\\end{equation}\n\nwhere $A$ is a symmetric (with respect to the given inner product) and positive definite matrix. It is well known that such an update step can also be interpreted as a gradient step with respect to a modified inner product. This leads to the following result that will be the basis for our further considerations.\n\n\nNext we provide error estimates for gradient related descent methods under angle and scaling conditions. Specifically, we consider the update rule\n\\begin{equation}\\label{eq: perturbed GD}\n x^+ = x - h d,\n\\end{equation}\n\nwhere $-d$ is a descent direction, that is, $d$ satisfies\n\\begin{equation}\\label{eq:def of cos theta}\n\\langle \\nabla f(x), d \\rangle = \\cos \\theta \\| \\nabla f(x) \\| \\| d \\|, \\quad \\cos \\theta > 0,\n\\end{equation}\n\nfor some $\\theta \\in [0, \\pi/2)$. This condition is very natural since it guarantees the convergence of~\\eqref{eq: perturbed GD}; see, e.g., \\cite[Chapter 3.2]{nocedalNumericalOptimization2006}. In particular, for  the case of exact line search, it has been shown in~\\cite[Theorem~5.1]{deKlerk2017} that\n\\begin{equation}\\label{eq:optimal rate angle condition}\nf(x^+) - f(x^*) \\le \\left(\\frac{\\kappa_{f,\\theta} - 1}{\\kappa_{f,\\theta} + 1} \\right)^2 (f(x) - f(x^*)), \\quad \\kappa_{f,\\theta} = \\frac{L}{\\mu} \\left(\\frac{1 + \\sin \\theta}{1 - \\sin \\theta}\\right),\n\\end{equation}","units":[{"start":1978,"end":2207,"start_line":57,"end_line":60,"raw_tex":"An $L$-smooth and $\\mu$-strongly convex function $f \\colon \\R^n \\to \\R$ is  characterized by the two properties\n\\begin{equation*}\\label{eq: Lipschitz condition}\n\\| \\nabla f(x) - \\nabla f(y) \\| \\le \\Lip \\| x - y \\|\n\\end{equation*}","sha256_utf8":"d1b8c72a6ad83ef90acc51b0bc6a46c2d14a5d507252c91817b75029bd698edb"},{"start":2208,"end":2356,"start_line":61,"end_line":64,"raw_tex":"and\n\\begin{equation*}\\label{eq: strong convexity}\n f(x) \\ge f(y) + \\langle \\nabla f(y), x - y \\rangle + \\frac{\\mu}{2} \\| x - y \\|^2 \n\\end{equation*}","sha256_utf8":"0b4f9ba50ac943f057ecd5c2d394c1de9e59226a93898fc6cccf01ebe11685a3"},{"start":2357,"end":2924,"start_line":65,"end_line":68,"raw_tex":"for some constants $0 < \\mu \\le L$ and all $x,y \\in \\R^n$. Here, $\\langle \\ , \\  \\rangle$ can be any inner product on $\\R^n$ with corresponding norm $\\| \\cdot \\|$, and $\\nabla f$ denotes the gradient with respect to this inner product. Note that the  constants $\\mu$ and $L$ depend on the chosen inner product. The class of such functions plays a main role in the convergence theory of the gradient method and related descent methods for finding the unique global minimum $x^*$ of a given $f$. The update rule of the gradient method is\n\\[\n x^+ = x - h \\nabla f(x),\n\\]","sha256_utf8":"218a87f0779c48a013dccfe267f2ad410ab54fcf85b832856adfd568f0a0e1d8"},{"start":3421,"end":3762,"start_line":79,"end_line":82,"raw_tex":"In a more general setting of proximal gradient methods, it has recently been shown by Taylor, Hendrickx, and Glineur~\\cite[Theorem~3.3 with $h=0$]{Taylor2018} that the same rate is also valid for the error in function value. Specifically, for any \n\\begin{equation}\\label{eq: feasible step sizes}\n 0 \\le h \\le \\frac{2}{L + \\mu}\n\\end{equation}","sha256_utf8":"ef1c06a3cb9a781a92a763f8a651dac59a387598b68cfe242be766ddc61d9fac"},{"start":3763,"end":3896,"start_line":83,"end_line":86,"raw_tex":"it holds that\n\\begin{equation}\\label{eq: function values general h}\n f(x^+) - f(x^*) \\le (1 - h\\mu)^2 (f(x) - f(x^*)).\n\\end{equation}","sha256_utf8":"1233cf9e5c7c6ce72d339a21e5f26e8cc075902c3ad4192725922d4bd3b2bb5b"},{"start":6711,"end":6882,"start_line":102,"end_line":105,"raw_tex":"We first consider the variable metric method. Here the update rule reads\n\\begin{equation}\\label{eq: variable metric method}\n x^+ = x - h A^{-1} \\nabla f(x),\n\\end{equation}","sha256_utf8":"eda52f36c29919fe52f1824d9335b81bc2e280ab31797141fd90604d854e973b"},{"start":6883,"end":7197,"start_line":106,"end_line":106,"raw_tex":"where $A$ is a symmetric (with respect to the given inner product) and positive definite matrix. It is well known that such an update step can also be interpreted as a gradient step with respect to a modified inner product. This leads to the following result that will be the basis for our further considerations.\n","sha256_utf8":"cc08b9f59c1dcf619614c7ba170c28a0e60c2aac56ea585ebf66ed1e84b01e4e"},{"start":11021,"end":11239,"start_line":141,"end_line":144,"raw_tex":"Next we provide error estimates for gradient related descent methods under angle and scaling conditions. Specifically, we consider the update rule\n\\begin{equation}\\label{eq: perturbed GD}\n x^+ = x - h d,\n\\end{equation}","sha256_utf8":"161f0bd0c30bda5559871f1d9b024dcf7f7abd1eba45dc61f8cc241954767999"},{"start":11240,"end":11451,"start_line":145,"end_line":148,"raw_tex":"where $-d$ is a descent direction, that is, $d$ satisfies\n\\begin{equation}\\label{eq:def of cos theta}\n\\langle \\nabla f(x), d \\rangle = \\cos \\theta \\| \\nabla f(x) \\| \\| d \\|, \\quad \\cos \\theta > 0,\n\\end{equation}","sha256_utf8":"c4075031f04e146c11cd7c64a8155a7ab88c8db6bc6e3fd05feff6dfef2c3a10"},{"start":11452,"end":12025,"start_line":149,"end_line":152,"raw_tex":"for some $\\theta \\in [0, \\pi/2)$. This condition is very natural since it guarantees the convergence of~\\eqref{eq: perturbed GD}; see, e.g., \\cite[Chapter 3.2]{nocedalNumericalOptimization2006}. In particular, for  the case of exact line search, it has been shown in~\\cite[Theorem~5.1]{deKlerk2017} that\n\\begin{equation}\\label{eq:optimal rate angle condition}\nf(x^+) - f(x^*) \\le \\left(\\frac{\\kappa_{f,\\theta} - 1}{\\kappa_{f,\\theta} + 1} \\right)^2 (f(x) - f(x^*)), \\quad \\kappa_{f,\\theta} = \\frac{L}{\\mu} \\left(\\frac{1 + \\sin \\theta}{1 - \\sin \\theta}\\right),\n\\end{equation}","sha256_utf8":"2bc55f513709c6bdf0decfd1a8242a901a9ae0c86cbc0b36418d5280da8cdbd2"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{thm: gradient related}\nAssume~\\eqref{eq:def of cos theta} and \n\\begin{equation}\\label{eq: scaling condition}\n \\| d \\| = c \\| \\nabla f(x) \\|\n\\end{equation}\nfor some $c > 0$. Define\n\\[\n \\bar h = \\frac{2 \\cos \\theta}{L c (1 + \\sin \\theta) + \\mu c (1 - \\sin \\theta)}.\n\\]\nThen $x^+$ in~\\eqref{eq: perturbed GD} with $0 \\le h \\le \\bar h$ satisfies\n\\[\n f(x^+) - f(x^*) \\le \\left(1 - \\frac{h \\mu c (1-\\sin \\theta)}{ \\cos \\theta} \\right)^2 (f(x) - f(x^*)).\n\\]\nIn particular, the step size $h=\\bar h$ yields\n\\[\n f(x^+) - f(x^*) \\le \\left(\\frac{\\kappa_{f,\\theta} - 1}{\\kappa_{f,\\theta} + 1} \\right)^2 (f(x) - f(x^*)).\n\\]\n\\end{theorem}","units":[{"start":14571,"end":15216,"start_line":183,"end_line":200,"raw_tex":"\\begin{theorem}\\label{thm: gradient related}\nAssume~\\eqref{eq:def of cos theta} and \n\\begin{equation}\\label{eq: scaling condition}\n \\| d \\| = c \\| \\nabla f(x) \\|\n\\end{equation}\nfor some $c > 0$. Define\n\\[\n \\bar h = \\frac{2 \\cos \\theta}{L c (1 + \\sin \\theta) + \\mu c (1 - \\sin \\theta)}.\n\\]\nThen $x^+$ in~\\eqref{eq: perturbed GD} with $0 \\le h \\le \\bar h$ satisfies\n\\[\n f(x^+) - f(x^*) \\le \\left(1 - \\frac{h \\mu c (1-\\sin \\theta)}{ \\cos \\theta} \\right)^2 (f(x) - f(x^*)).\n\\]\nIn particular, the step size $h=\\bar h$ yields\n\\[\n f(x^+) - f(x^*) \\le \\left(\\frac{\\kappa_{f,\\theta} - 1}{\\kappa_{f,\\theta} + 1} \\right)^2 (f(x) - f(x^*)).\n\\]\n\\end{theorem}","sha256_utf8":"48f26ae2859f89460c6b33a42e5532eb659db8d566712a81a8cb94921726d582"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{theorem}\\label{thm: variable metric method}\nAssume the eigenvalues of $A$ are in the positive interval $[\\lambda,\\Lambda]$ and define\n\\[\n \\bar h = \\frac{2}{L/\\lambda + \\mu/\\Lambda}.\n\\]\nThen $x^+$ in~\\eqref{eq: variable metric method} with $0 \\le h \\le \\bar h$ satisfies\n\\[\n f(x^+) - f(x^*) \\le \\left(1 - \\frac{h \\mu}{\\Lambda} \\right)^2 (f(x) - f(x^*)).\n\\]\nIn particular, the step size $h = \\bar h$ yields\n\\begin{equation}\\label{eq: estimate for function values variable metric}\n f(x^+) - f(x^*) \\le \\left(\\frac{\\kappa_{f,A} - 1}{\\kappa_{f,A} + 1} \\right)^2 (f(x) - f(x^*)), \\quad \\kappa_{f,A} = \\frac{L }{\\mu} \\, \\frac{\\Lambda }{\\lambda}. \n\\end{equation}\n\\end{theorem}\n\n\\begin{proof}\nThe result is obtained from~\\eqref{eq: function values general h} by noting that $\\nabla_A f(x) =   A^{-1}\\nabla f(x)$ is the gradient of $f$ with respect to the $A$-inner product $\\langle x, y \\rangle_A = \\langle x, Ay \\rangle$. We have\n\\[\n\\langle \\nabla_A f(x) - \\nabla_A f(y), x - y \\rangle_A \\le L \\| x - y \\|^2 \\le \\frac{L}{\\lambda} \\| x - y \\|_A^2\n\\]\nas well as\n\\[\n\\langle \\nabla_A f(x) - \\nabla_A f(y), x - y \\rangle_A \\ge \\mu \\| x - y \\|^2 \\ge \\frac{\\mu}{\\Lambda} \\| x - y \\|_A^2\n\\]\nfor all $x,y$. These two conditions are equivalent to $f$ being $(L/\\lambda)$-smooth and $(\\mu/\\Lambda)$-strongly convex in that $A$-inner product; see, e.g.,~\\cite[Theorems~2.1.5~\\&~2.1.9]{Nesterov2004}. Thus in~\\eqref{eq: feasible step sizes} and~\\eqref{eq: function values general h}, we can replace $\\mu$ with $\\mu/\\Lambda$ and $L$ by $L/\\lambda$, which is exactly the statement of the theorem.\n\\end{proof}\n\n\\begin{lemma}\\label{lem: SR1 update}\nLet $u, v \\in \\R^n$ such that $\\|u \\| = \\| v \\| = 1$ and $\\langle u, v \\rangle = \\cos \\theta$. Then the matrix \n\\[\n B = \\frac{1}{\\alpha} \\left( I - \\frac{r r^*}{\\langle r, u \\rangle} \\right), \\quad r = u - \\alpha v, \\quad \\alpha = \\frac{1 - \\sin \\theta}{\\cos \\theta} = \\frac{\\cos \\theta}{1+\\sin \\theta}\n\\]\nis symmetric (for the given inner product), satisfies $Bu=v$, and has\n\\[\n\\lambda_{\\min}(B) = \\frac{\\cos \\theta}{1 + \\sin \\theta}, \\quad \\lambda_{\\max}(B) = \\frac{\\cos \\theta}{1 - \\sin \\theta},\n\\]\nas its smallest and largest eigenvalues, respectively. Here, $rr^*$ denotes the rank-one matrix satisfying $rr^* x = r \\langle r,x\\rangle$ for all $x \\in \\R^n$.\n\\end{lemma}\n\n\\begin{proof}\nThis is checked by a straightforward calculation. Obviously, the matrix $I - \\frac{\\ r^{} r^*}{\\langle r, u \\rangle}$ equals the identity on the orthogonal complement of $r$. Its eigenvalue belonging to the  eigenvector $r$ is\n\\[\n1 - \\frac{\\| r \\|^2}{\\langle r, u \\rangle} = 1 - \\frac{1 - 2 \\alpha \\cos \\theta + \\alpha^2}{1 - \\alpha \\cos \\theta} = \\frac{1 - \\sin \\theta - \\alpha^2}{\\sin \\theta} = \\alpha^2,\n\\]\nwhere one uses $1 - \\alpha \\cos \\theta = \\sin \\theta$ and $\\alpha^2 = (1 - \\sin \\theta)/(1 + \\sin \\theta)$. Therefore, the largest eigenvalue of $B$ is $1/\\alpha$ (with multiplicity $n-1$), and the smallest eigenvalue is $\\alpha$.\n\\end{proof}\n\n\\begin{proof}\nIf $d = 0$, the assertion is trivially true. Let $d \\neq 0$. By Lemma~\\ref{lem: SR1 update}, there exists a symmetric and positive definite matrix of the form $A = \\frac{\\| \\nabla f(x) \\|}{\\| d \\|}B = \\frac{1}{c}B$ such that $Ad = \\nabla f(x)$ and\n\\[\n\\lambda_{\\min}(A) = \\frac{1}{c} \\left(\\frac{\\cos \\theta}{1 + \\sin \\theta} \\right), \\quad \\lambda_{\\max}(A) = \\frac{1}{c} \\left(\\frac{\\cos \\theta}{1 - \\sin \\theta} \\right).\n\\]\nThe assertion follows therefore directly from Theorem~\\ref{thm: variable metric method}.\n\\end{proof}","units":[{"start":7198,"end":7873,"start_line":108,"end_line":121,"raw_tex":"\\begin{theorem}\\label{thm: variable metric method}\nAssume the eigenvalues of $A$ are in the positive interval $[\\lambda,\\Lambda]$ and define\n\\[\n \\bar h = \\frac{2}{L/\\lambda + \\mu/\\Lambda}.\n\\]\nThen $x^+$ in~\\eqref{eq: variable metric method} with $0 \\le h \\le \\bar h$ satisfies\n\\[\n f(x^+) - f(x^*) \\le \\left(1 - \\frac{h \\mu}{\\Lambda} \\right)^2 (f(x) - f(x^*)).\n\\]\nIn particular, the step size $h = \\bar h$ yields\n\\begin{equation}\\label{eq: estimate for function values variable metric}\n f(x^+) - f(x^*) \\le \\left(\\frac{\\kappa_{f,A} - 1}{\\kappa_{f,A} + 1} \\right)^2 (f(x) - f(x^*)), \\quad \\kappa_{f,A} = \\frac{L }{\\mu} \\, \\frac{\\Lambda }{\\lambda}. \n\\end{equation}\n\\end{theorem}","sha256_utf8":"6b694a6e5d622f0db82b4c412d4f98a03b2fb8d6247bd16e400b73fb30901e3d"},{"start":7875,"end":8790,"start_line":123,"end_line":133,"raw_tex":"\\begin{proof}\nThe result is obtained from~\\eqref{eq: function values general h} by noting that $\\nabla_A f(x) =   A^{-1}\\nabla f(x)$ is the gradient of $f$ with respect to the $A$-inner product $\\langle x, y \\rangle_A = \\langle x, Ay \\rangle$. We have\n\\[\n\\langle \\nabla_A f(x) - \\nabla_A f(y), x - y \\rangle_A \\le L \\| x - y \\|^2 \\le \\frac{L}{\\lambda} \\| x - y \\|_A^2\n\\]\nas well as\n\\[\n\\langle \\nabla_A f(x) - \\nabla_A f(y), x - y \\rangle_A \\ge \\mu \\| x - y \\|^2 \\ge \\frac{\\mu}{\\Lambda} \\| x - y \\|_A^2\n\\]\nfor all $x,y$. These two conditions are equivalent to $f$ being $(L/\\lambda)$-smooth and $(\\mu/\\Lambda)$-strongly convex in that $A$-inner product; see, e.g.,~\\cite[Theorems~2.1.5~\\&~2.1.9]{Nesterov2004}. Thus in~\\eqref{eq: feasible step sizes} and~\\eqref{eq: function values general h}, we can replace $\\mu$ with $\\mu/\\Lambda$ and $L$ by $L/\\lambda$, which is exactly the statement of the theorem.\n\\end{proof}","sha256_utf8":"04e70a395e7e0da6d76f14eb770d80c4def2c9b65112fdf3d1e580a157477cc2"},{"start":13063,"end":13774,"start_line":161,"end_line":171,"raw_tex":"\\begin{lemma}\\label{lem: SR1 update}\nLet $u, v \\in \\R^n$ such that $\\|u \\| = \\| v \\| = 1$ and $\\langle u, v \\rangle = \\cos \\theta$. Then the matrix \n\\[\n B = \\frac{1}{\\alpha} \\left( I - \\frac{r r^*}{\\langle r, u \\rangle} \\right), \\quad r = u - \\alpha v, \\quad \\alpha = \\frac{1 - \\sin \\theta}{\\cos \\theta} = \\frac{\\cos \\theta}{1+\\sin \\theta}\n\\]\nis symmetric (for the given inner product), satisfies $Bu=v$, and has\n\\[\n\\lambda_{\\min}(B) = \\frac{\\cos \\theta}{1 + \\sin \\theta}, \\quad \\lambda_{\\max}(B) = \\frac{\\cos \\theta}{1 - \\sin \\theta},\n\\]\nas its smallest and largest eigenvalues, respectively. Here, $rr^*$ denotes the rank-one matrix satisfying $rr^* x = r \\langle r,x\\rangle$ for all $x \\in \\R^n$.\n\\end{lemma}","sha256_utf8":"c8b7ff78b67dcab1541946434c73066e30637cc321140714e0861880a6912e61"},{"start":13776,"end":14442,"start_line":173,"end_line":179,"raw_tex":"\\begin{proof}\nThis is checked by a straightforward calculation. Obviously, the matrix $I - \\frac{\\ r^{} r^*}{\\langle r, u \\rangle}$ equals the identity on the orthogonal complement of $r$. Its eigenvalue belonging to the  eigenvector $r$ is\n\\[\n1 - \\frac{\\| r \\|^2}{\\langle r, u \\rangle} = 1 - \\frac{1 - 2 \\alpha \\cos \\theta + \\alpha^2}{1 - \\alpha \\cos \\theta} = \\frac{1 - \\sin \\theta - \\alpha^2}{\\sin \\theta} = \\alpha^2,\n\\]\nwhere one uses $1 - \\alpha \\cos \\theta = \\sin \\theta$ and $\\alpha^2 = (1 - \\sin \\theta)/(1 + \\sin \\theta)$. Therefore, the largest eigenvalue of $B$ is $1/\\alpha$ (with multiplicity $n-1$), and the smallest eigenvalue is $\\alpha$.\n\\end{proof}","sha256_utf8":"4df25912c762ae53dc6abea8b031762f784bcefdb5bc4c7cc23f01666b28c11f"},{"start":15219,"end":15759,"start_line":202,"end_line":208,"raw_tex":"\\begin{proof}\nIf $d = 0$, the assertion is trivially true. Let $d \\neq 0$. By Lemma~\\ref{lem: SR1 update}, there exists a symmetric and positive definite matrix of the form $A = \\frac{\\| \\nabla f(x) \\|}{\\| d \\|}B = \\frac{1}{c}B$ such that $Ad = \\nabla f(x)$ and\n\\[\n\\lambda_{\\min}(A) = \\frac{1}{c} \\left(\\frac{\\cos \\theta}{1 + \\sin \\theta} \\right), \\quad \\lambda_{\\max}(A) = \\frac{1}{c} \\left(\\frac{\\cos \\theta}{1 - \\sin \\theta} \\right).\n\\]\nThe assertion follows therefore directly from Theorem~\\ref{thm: variable metric method}.\n\\end{proof}","sha256_utf8":"cab206ec8ae2b54b128f1755885e643833dc95ad79fbf5d688b06e379b0f95fc"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"光滑强凸函数与内积","source_units":[{"start":1978,"end":2207,"start_line":57,"end_line":60,"raw_tex":"An $L$-smooth and $\\mu$-strongly convex function $f \\colon \\R^n \\to \\R$ is  characterized by the two properties\n\\begin{equation*}\\label{eq: Lipschitz condition}\n\\| \\nabla f(x) - \\nabla f(y) \\| \\le \\Lip \\| x - y \\|\n\\end{equation*}","sha256_utf8":"d1b8c72a6ad83ef90acc51b0bc6a46c2d14a5d507252c91817b75029bd698edb"},{"start":2208,"end":2356,"start_line":61,"end_line":64,"raw_tex":"and\n\\begin{equation*}\\label{eq: strong convexity}\n f(x) \\ge f(y) + \\langle \\nabla f(y), x - y \\rangle + \\frac{\\mu}{2} \\| x - y \\|^2 \n\\end{equation*}","sha256_utf8":"0b4f9ba50ac943f057ecd5c2d394c1de9e59226a93898fc6cccf01ebe11685a3"},{"start":2357,"end":2893,"start_line":65,"end_line":65,"raw_tex":"for some constants $0 < \\mu \\le L$ and all $x,y \\in \\R^n$. Here, $\\langle \\ , \\  \\rangle$ can be any inner product on $\\R^n$ with corresponding norm $\\| \\cdot \\|$, and $\\nabla f$ denotes the gradient with respect to this inner product. Note that the  constants $\\mu$ and $L$ depend on the chosen inner product. The class of such functions plays a main role in the convergence theory of the gradient method and related descent methods for finding the unique global minimum $x^*$ of a given $f$. The update rule of the gradient method is\n","sha256_utf8":"87b57f581a5a26c13821e4409c8441f978aaf917008068ce41492720d8bbc41e"}]},{"id":"S002","semantic_type":"algorithm","kind":"algorithm","title":"标准梯度更新","source_units":[{"start":2893,"end":2924,"start_line":66,"end_line":68,"raw_tex":"\\[\n x^+ = x - h \\nabla f(x),\n\\]","sha256_utf8":"25aae190c9304d0c14c7c7e987a7f0a63df07cdc1d496df1c27e92ecce058784"}]},{"id":"S003","semantic_type":"derived_fact","kind":"derived_fact","title":"标准梯度函数值收缩界","source_units":[{"start":3421,"end":3762,"start_line":79,"end_line":82,"raw_tex":"In a more general setting of proximal gradient methods, it has recently been shown by Taylor, Hendrickx, and Glineur~\\cite[Theorem~3.3 with $h=0$]{Taylor2018} that the same rate is also valid for the error in function value. Specifically, for any \n\\begin{equation}\\label{eq: feasible step sizes}\n 0 \\le h \\le \\frac{2}{L + \\mu}\n\\end{equation}","sha256_utf8":"ef1c06a3cb9a781a92a763f8a651dac59a387598b68cfe242be766ddc61d9fac"},{"start":3763,"end":3896,"start_line":83,"end_line":86,"raw_tex":"it holds that\n\\begin{equation}\\label{eq: function values general h}\n f(x^+) - f(x^*) \\le (1 - h\\mu)^2 (f(x) - f(x^*)).\n\\end{equation}","sha256_utf8":"1233cf9e5c7c6ce72d339a21e5f26e8cc075902c3ad4192725922d4bd3b2bb5b"}]},{"id":"S004","semantic_type":"algorithm","kind":"algorithm","title":"变度量更新及矩阵条件","source_units":[{"start":6711,"end":6882,"start_line":102,"end_line":105,"raw_tex":"We first consider the variable metric method. Here the update rule reads\n\\begin{equation}\\label{eq: variable metric method}\n x^+ = x - h A^{-1} \\nabla f(x),\n\\end{equation}","sha256_utf8":"eda52f36c29919fe52f1824d9335b81bc2e280ab31797141fd90604d854e973b"},{"start":6883,"end":7197,"start_line":106,"end_line":106,"raw_tex":"where $A$ is a symmetric (with respect to the given inner product) and positive definite matrix. It is well known that such an update step can also be interpreted as a gradient step with respect to a modified inner product. This leads to the following result that will be the basis for our further considerations.\n","sha256_utf8":"cc08b9f59c1dcf619614c7ba170c28a0e60c2aac56ea585ebf66ed1e84b01e4e"}]},{"id":"S005","semantic_type":"algorithm","kind":"algorithm","title":"一般下降方向更新","source_units":[{"start":11021,"end":11239,"start_line":141,"end_line":144,"raw_tex":"Next we provide error estimates for gradient related descent methods under angle and scaling conditions. Specifically, we consider the update rule\n\\begin{equation}\\label{eq: perturbed GD}\n x^+ = x - h d,\n\\end{equation}","sha256_utf8":"161f0bd0c30bda5559871f1d9b024dcf7f7abd1eba45dc61f8cc241954767999"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"下降方向夹角","source_units":[{"start":11240,"end":11451,"start_line":145,"end_line":148,"raw_tex":"where $-d$ is a descent direction, that is, $d$ satisfies\n\\begin{equation}\\label{eq:def of cos theta}\n\\langle \\nabla f(x), d \\rangle = \\cos \\theta \\| \\nabla f(x) \\| \\| d \\|, \\quad \\cos \\theta > 0,\n\\end{equation}","sha256_utf8":"c4075031f04e146c11cd7c64a8155a7ab88c8db6bc6e3fd05feff6dfef2c3a10"},{"start":11452,"end":11756,"start_line":149,"end_line":149,"raw_tex":"for some $\\theta \\in [0, \\pi/2)$. This condition is very natural since it guarantees the convergence of~\\eqref{eq: perturbed GD}; see, e.g., \\cite[Chapter 3.2]{nocedalNumericalOptimization2006}. In particular, for  the case of exact line search, it has been shown in~\\cite[Theorem~5.1]{deKlerk2017} that\n","sha256_utf8":"55cf5b52f34dba6edee6ed8d8105e54ea79c6d62608375a7608ea549d0bbe234"}]},{"id":"S007","semantic_type":"notation","kind":"notation","title":"夹角条件数及已有收缩界","source_units":[{"start":11756,"end":12025,"start_line":150,"end_line":152,"raw_tex":"\\begin{equation}\\label{eq:optimal rate angle condition}\nf(x^+) - f(x^*) \\le \\left(\\frac{\\kappa_{f,\\theta} - 1}{\\kappa_{f,\\theta} + 1} \\right)^2 (f(x) - f(x^*)), \\quad \\kappa_{f,\\theta} = \\frac{L}{\\mu} \\left(\\frac{1 + \\sin \\theta}{1 - \\sin \\theta}\\right),\n\\end{equation}","sha256_utf8":"6faf5984ea3465e8bebf7e2865be861b17528d19d1d7123d3e7bc673e784aec4"}]},{"id":"S008","semantic_type":"theorem","kind":"theorem","title":"夹角与缩放条件下的固定步长收敛率","source_units":[{"start":14571,"end":15216,"start_line":183,"end_line":200,"raw_tex":"\\begin{theorem}\\label{thm: gradient related}\nAssume~\\eqref{eq:def of cos theta} and \n\\begin{equation}\\label{eq: scaling condition}\n \\| d \\| = c \\| \\nabla f(x) \\|\n\\end{equation}\nfor some $c > 0$. Define\n\\[\n \\bar h = \\frac{2 \\cos \\theta}{L c (1 + \\sin \\theta) + \\mu c (1 - \\sin \\theta)}.\n\\]\nThen $x^+$ in~\\eqref{eq: perturbed GD} with $0 \\le h \\le \\bar h$ satisfies\n\\[\n f(x^+) - f(x^*) \\le \\left(1 - \\frac{h \\mu c (1-\\sin \\theta)}{ \\cos \\theta} \\right)^2 (f(x) - f(x^*)).\n\\]\nIn particular, the step size $h=\\bar h$ yields\n\\[\n f(x^+) - f(x^*) \\le \\left(\\frac{\\kappa_{f,\\theta} - 1}{\\kappa_{f,\\theta} + 1} \\right)^2 (f(x) - f(x^*)).\n\\]\n\\end{theorem}","sha256_utf8":"48f26ae2859f89460c6b33a42e5532eb659db8d566712a81a8cb94921726d582"}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","title":"主定理的夹角与缩放假设","source_units":[{"start":14616,"end":14773,"start_line":184,"end_line":188,"raw_tex":"Assume~\\eqref{eq:def of cos theta} and \n\\begin{equation}\\label{eq: scaling condition}\n \\| d \\| = c \\| \\nabla f(x) \\|\n\\end{equation}\nfor some $c > 0$. Define\n","sha256_utf8":"231dce820148f3e88f41241788ee298d35c23b2e9a01083af44c80c239708e0c"}]},{"id":"S010","semantic_type":"theorem","kind":"theorem","title":"变度量方法的函数值收缩","source_units":[{"start":7198,"end":7873,"start_line":108,"end_line":121,"raw_tex":"\\begin{theorem}\\label{thm: variable metric method}\nAssume the eigenvalues of $A$ are in the positive interval $[\\lambda,\\Lambda]$ and define\n\\[\n \\bar h = \\frac{2}{L/\\lambda + \\mu/\\Lambda}.\n\\]\nThen $x^+$ in~\\eqref{eq: variable metric method} with $0 \\le h \\le \\bar h$ satisfies\n\\[\n f(x^+) - f(x^*) \\le \\left(1 - \\frac{h \\mu}{\\Lambda} \\right)^2 (f(x) - f(x^*)).\n\\]\nIn particular, the step size $h = \\bar h$ yields\n\\begin{equation}\\label{eq: estimate for function values variable metric}\n f(x^+) - f(x^*) \\le \\left(\\frac{\\kappa_{f,A} - 1}{\\kappa_{f,A} + 1} \\right)^2 (f(x) - f(x^*)), \\quad \\kappa_{f,A} = \\frac{L }{\\mu} \\, \\frac{\\Lambda }{\\lambda}. \n\\end{equation}\n\\end{theorem}","sha256_utf8":"6b694a6e5d622f0db82b4c412d4f98a03b2fb8d6247bd16e400b73fb30901e3d"}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","title":"度量矩阵的正谱区间","source_units":[{"start":7249,"end":7339,"start_line":109,"end_line":109,"raw_tex":"Assume the eigenvalues of $A$ are in the positive interval $[\\lambda,\\Lambda]$ and define\n","sha256_utf8":"3656e65e1b24d5bd95aee8ed0443f8e9281aa396805616fd59fe0d585d3e948a"}]},{"id":"S012","semantic_type":"lemma","kind":"lemma","title":"映射单位向量的秩一矩阵构造","source_units":[{"start":13063,"end":13774,"start_line":161,"end_line":171,"raw_tex":"\\begin{lemma}\\label{lem: SR1 update}\nLet $u, v \\in \\R^n$ such that $\\|u \\| = \\| v \\| = 1$ and $\\langle u, v \\rangle = \\cos \\theta$. Then the matrix \n\\[\n B = \\frac{1}{\\alpha} \\left( I - \\frac{r r^*}{\\langle r, u \\rangle} \\right), \\quad r = u - \\alpha v, \\quad \\alpha = \\frac{1 - \\sin \\theta}{\\cos \\theta} = \\frac{\\cos \\theta}{1+\\sin \\theta}\n\\]\nis symmetric (for the given inner product), satisfies $Bu=v$, and has\n\\[\n\\lambda_{\\min}(B) = \\frac{\\cos \\theta}{1 + \\sin \\theta}, \\quad \\lambda_{\\max}(B) = \\frac{\\cos \\theta}{1 - \\sin \\theta},\n\\]\nas its smallest and largest eigenvalues, respectively. Here, $rr^*$ denotes the rank-one matrix satisfying $rr^* x = r \\langle r,x\\rangle$ for all $x \\in \\R^n$.\n\\end{lemma}","sha256_utf8":"c8b7ff78b67dcab1541946434c73066e30637cc321140714e0861880a6912e61"}]},{"id":"S013","semantic_type":"assumption","kind":"assumption","title":"单位向量及夹角假设","source_units":[{"start":13100,"end":13212,"start_line":162,"end_line":162,"raw_tex":"Let $u, v \\in \\R^n$ such that $\\|u \\| = \\| v \\| = 1$ and $\\langle u, v \\rangle = \\cos \\theta$. Then the matrix \n","sha256_utf8":"8f331df087c075cae6a01a8dea2e9a800bc95ce51fc1a08621078c3d4c01e12c"}]},{"id":"S014","semantic_type":"proof","kind":"proof","title":"主定理完整证明","source_units":[{"start":15219,"end":15759,"start_line":202,"end_line":208,"raw_tex":"\\begin{proof}\nIf $d = 0$, the assertion is trivially true. Let $d \\neq 0$. By Lemma~\\ref{lem: SR1 update}, there exists a symmetric and positive definite matrix of the form $A = \\frac{\\| \\nabla f(x) \\|}{\\| d \\|}B = \\frac{1}{c}B$ such that $Ad = \\nabla f(x)$ and\n\\[\n\\lambda_{\\min}(A) = \\frac{1}{c} \\left(\\frac{\\cos \\theta}{1 + \\sin \\theta} \\right), \\quad \\lambda_{\\max}(A) = \\frac{1}{c} \\left(\\frac{\\cos \\theta}{1 - \\sin \\theta} \\right).\n\\]\nThe assertion follows therefore directly from Theorem~\\ref{thm: variable metric method}.\n\\end{proof}","sha256_utf8":"cab206ec8ae2b54b128f1755885e643833dc95ad79fbf5d688b06e379b0f95fc"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"处理零方向并构造度量矩阵","source_units":[{"start":15233,"end":15481,"start_line":203,"end_line":203,"raw_tex":"If $d = 0$, the assertion is trivially true. Let $d \\neq 0$. By Lemma~\\ref{lem: SR1 update}, there exists a symmetric and positive definite matrix of the form $A = \\frac{\\| \\nabla f(x) \\|}{\\| d \\|}B = \\frac{1}{c}B$ such that $Ad = \\nabla f(x)$ and\n","sha256_utf8":"8b917b0302360cc2a9295813bbf27cba30ec19b234d3efcdf01911da6a96f970"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"确定度量矩阵的极端特征值","source_units":[{"start":15481,"end":15659,"start_line":204,"end_line":206,"raw_tex":"\\[\n\\lambda_{\\min}(A) = \\frac{1}{c} \\left(\\frac{\\cos \\theta}{1 + \\sin \\theta} \\right), \\quad \\lambda_{\\max}(A) = \\frac{1}{c} \\left(\\frac{\\cos \\theta}{1 - \\sin \\theta} \\right).\n\\]\n","sha256_utf8":"c4338cf6f32d0892549c2360461a9ce859a7e3e049ad3a1112ea87ef40fa5f17"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"应用变度量收缩定理","source_units":[{"start":15659,"end":15748,"start_line":207,"end_line":207,"raw_tex":"The assertion follows therefore directly from Theorem~\\ref{thm: variable metric method}.\n","sha256_utf8":"06fcf1e6f2c4cbf7a14edccf415ac6bd7ed6c59314c4f1055bae52f5cff960c6"}]},{"id":"S018","semantic_type":"proof","kind":"proof","title":"变度量收缩定理完整证明","source_units":[{"start":7875,"end":8790,"start_line":123,"end_line":133,"raw_tex":"\\begin{proof}\nThe result is obtained from~\\eqref{eq: function values general h} by noting that $\\nabla_A f(x) =   A^{-1}\\nabla f(x)$ is the gradient of $f$ with respect to the $A$-inner product $\\langle x, y \\rangle_A = \\langle x, Ay \\rangle$. We have\n\\[\n\\langle \\nabla_A f(x) - \\nabla_A f(y), x - y \\rangle_A \\le L \\| x - y \\|^2 \\le \\frac{L}{\\lambda} \\| x - y \\|_A^2\n\\]\nas well as\n\\[\n\\langle \\nabla_A f(x) - \\nabla_A f(y), x - y \\rangle_A \\ge \\mu \\| x - y \\|^2 \\ge \\frac{\\mu}{\\Lambda} \\| x - y \\|_A^2\n\\]\nfor all $x,y$. These two conditions are equivalent to $f$ being $(L/\\lambda)$-smooth and $(\\mu/\\Lambda)$-strongly convex in that $A$-inner product; see, e.g.,~\\cite[Theorems~2.1.5~\\&~2.1.9]{Nesterov2004}. Thus in~\\eqref{eq: feasible step sizes} and~\\eqref{eq: function values general h}, we can replace $\\mu$ with $\\mu/\\Lambda$ and $L$ by $L/\\lambda$, which is exactly the statement of the theorem.\n\\end{proof}","sha256_utf8":"04e70a395e7e0da6d76f14eb770d80c4def2c9b65112fdf3d1e580a157477cc2"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"识别新梯度并建立光滑上界","source_units":[{"start":7889,"end":8246,"start_line":124,"end_line":127,"raw_tex":"The result is obtained from~\\eqref{eq: function values general h} by noting that $\\nabla_A f(x) =   A^{-1}\\nabla f(x)$ is the gradient of $f$ with respect to the $A$-inner product $\\langle x, y \\rangle_A = \\langle x, Ay \\rangle$. We have\n\\[\n\\langle \\nabla_A f(x) - \\nabla_A f(y), x - y \\rangle_A \\le L \\| x - y \\|^2 \\le \\frac{L}{\\lambda} \\| x - y \\|_A^2\n\\]\n","sha256_utf8":"d8f50eb74b8c85419906ac796fe4d0368de628573fd843e00a75e74a2d07ddde"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"建立新内积下的强凸下界","source_units":[{"start":8246,"end":8380,"start_line":128,"end_line":131,"raw_tex":"as well as\n\\[\n\\langle \\nabla_A f(x) - \\nabla_A f(y), x - y \\rangle_A \\ge \\mu \\| x - y \\|^2 \\ge \\frac{\\mu}{\\Lambda} \\| x - y \\|_A^2\n\\]\n","sha256_utf8":"24f2e175630bd9d29c90ccce8b212df7e2be480d7e528c5795796fead07ba395"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"替换光滑强凸常数得到收缩界","source_units":[{"start":8380,"end":8779,"start_line":132,"end_line":132,"raw_tex":"for all $x,y$. These two conditions are equivalent to $f$ being $(L/\\lambda)$-smooth and $(\\mu/\\Lambda)$-strongly convex in that $A$-inner product; see, e.g.,~\\cite[Theorems~2.1.5~\\&~2.1.9]{Nesterov2004}. Thus in~\\eqref{eq: feasible step sizes} and~\\eqref{eq: function values general h}, we can replace $\\mu$ with $\\mu/\\Lambda$ and $L$ by $L/\\lambda$, which is exactly the statement of the theorem.\n","sha256_utf8":"1670fe5c53af38ce42d3c8b50b0d9e964cf716e86561f128d5578e98798f97bb"}]},{"id":"S022","semantic_type":"proof","kind":"proof","title":"秩一构造引理完整证明","source_units":[{"start":13776,"end":14442,"start_line":173,"end_line":179,"raw_tex":"\\begin{proof}\nThis is checked by a straightforward calculation. Obviously, the matrix $I - \\frac{\\ r^{} r^*}{\\langle r, u \\rangle}$ equals the identity on the orthogonal complement of $r$. Its eigenvalue belonging to the  eigenvector $r$ is\n\\[\n1 - \\frac{\\| r \\|^2}{\\langle r, u \\rangle} = 1 - \\frac{1 - 2 \\alpha \\cos \\theta + \\alpha^2}{1 - \\alpha \\cos \\theta} = \\frac{1 - \\sin \\theta - \\alpha^2}{\\sin \\theta} = \\alpha^2,\n\\]\nwhere one uses $1 - \\alpha \\cos \\theta = \\sin \\theta$ and $\\alpha^2 = (1 - \\sin \\theta)/(1 + \\sin \\theta)$. Therefore, the largest eigenvalue of $B$ is $1/\\alpha$ (with multiplicity $n-1$), and the smallest eigenvalue is $\\alpha$.\n\\end{proof}","sha256_utf8":"4df25912c762ae53dc6abea8b031762f784bcefdb5bc4c7cc23f01666b28c11f"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"计算正交补与秩一方向的特征值","source_units":[{"start":13790,"end":14200,"start_line":174,"end_line":177,"raw_tex":"This is checked by a straightforward calculation. Obviously, the matrix $I - \\frac{\\ r^{} r^*}{\\langle r, u \\rangle}$ equals the identity on the orthogonal complement of $r$. Its eigenvalue belonging to the  eigenvector $r$ is\n\\[\n1 - \\frac{\\| r \\|^2}{\\langle r, u \\rangle} = 1 - \\frac{1 - 2 \\alpha \\cos \\theta + \\alpha^2}{1 - \\alpha \\cos \\theta} = \\frac{1 - \\sin \\theta - \\alpha^2}{\\sin \\theta} = \\alpha^2,\n\\]\n","sha256_utf8":"a6243432253dab9742ada94f1f2b7a94b7651116ddae76f2530d64e395f8961f"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"利用参数恒等式确定极端谱值","source_units":[{"start":14200,"end":14431,"start_line":178,"end_line":178,"raw_tex":"where one uses $1 - \\alpha \\cos \\theta = \\sin \\theta$ and $\\alpha^2 = (1 - \\sin \\theta)/(1 + \\sin \\theta)$. Therefore, the largest eigenvalue of $B$ is $1/\\alpha$ (with multiplicity $n-1$), and the smallest eigenvalue is $\\alpha$.\n","sha256_utf8":"a726a8c55ec93db92f038243edf89dbe6cb09bf14a1248beaae399fe1d670c97"}]},{"id":"S025","semantic_type":"external_fact","kind":"external_fact","title":"标准梯度法的函数值收缩定理","source_units":[],"declared_external":true,"declaration_text":["Taylor2018, Theorem 3.3 as invoked at lines 79–86: for an L-smooth, μ-strongly convex function in the chosen inner product, x⁺ = x − h∇f(x) and 0 ≤ h ≤ 2/(L+μ) imply f(x⁺)−f(x*) ≤ (1−hμ)²(f(x)−f(x*))."]},{"id":"S026","semantic_type":"external_fact","kind":"external_fact","title":"梯度单调性与光滑强凸性的等价刻画","source_units":[],"declared_external":true,"declaration_text":["Nesterov2004, Theorems 2.1.5 and 2.1.9 as invoked at line 132: for a differentiable convex function, an upper bound ⟨∇f(x)−∇f(y),x−y⟩ ≤ L‖x−y‖² characterizes L-smoothness, and the corresponding lower bound with μ characterizes μ-strong convexity, using the relevant inner product and its gradient."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S008","edges":[{"source_id":"S001","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S025","target_id":"S003","kind":"supports"},{"source_id":"S001","target_id":"S019","kind":"supports"},{"source_id":"S001","target_id":"S020","kind":"supports"},{"source_id":"S004","target_id":"S019","kind":"supports"},{"source_id":"S011","target_id":"S019","kind":"supports"},{"source_id":"S011","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S026","target_id":"S021","kind":"supports"},{"source_id":"S003","target_id":"S021","kind":"supports"},{"source_id":"S004","target_id":"S021","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S020","target_id":"S018","kind":"supports"},{"source_id":"S021","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S010","kind":"supports"},{"source_id":"S013","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S012","kind":"supports"},{"source_id":"S006","target_id":"S015","kind":"supports"},{"source_id":"S009","target_id":"S015","kind":"supports"},{"source_id":"S012","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S012","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S005","target_id":"S017","kind":"supports"},{"source_id":"S010","target_id":"S017","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S016","target_id":"S014","kind":"supports"},{"source_id":"S017","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"}]}}
{"record_id":"2106.09286v1","paper":{"title":"Sub-linear convergence of a tamed stochastic gradient descent method in Hilbert space","authors":["Monika Eisenmann","Tony Stillfjord"],"arxiv_id":"2106.09286","version":"v1","journal":"SIAM Journal on Optimization","doi":"10.1137/21m1427450","abstract_url":"https://arxiv.org/abs/2106.09286v1","pdf_url":"https://arxiv.org/pdf/2106.09286v1"},"context":{"raw_tex":"In the following, let $(H, \\dual{\\cdot}{\\cdot}, \\|\\cdot \\|)$ be a real Hilbert space. Its \ndual space is denoted\nby $(H^*, \\dual[H^*]{\\cdot}{\\cdot}, \\| \\cdot \\|_{H^*} )$. \nSince $H$ is a Hilbert space, there exists an isometric isomorphism $\\iota \\colon \nH^* \\to H$ such that $\\iota^{-1} \\colon H \\to H^*$ with $\\iota^{-1}: v \\mapsto \n\\dual{v}{\\cdot}$. \nWe denote by $\\N$ the natural numbers, not including $0$.\n\nLet $(\\Omega, \\F, \\P )$ be a complete probability space and let $\\{\\xi_n\\}_{n \\in \\N}$ be \na family of jointly independent random variables on $\\Omega$.\nFor a random variable $X \\colon \\Omega \\to H$, let $\\E_{\\xi}[X]$ denote the \nexpectation with respect to the probability distribution of $\\xi$. \nWe are mainly interested in the total expectation\n\\begin{equation*}\n  \\E_n \\big[ \\|X\\|^2 \\big] = \\E_{\\xi_1}\\big[ \\E_{\\xi_2}\\big[ \\cdots \n  \\E_{\\xi_n} \\big[ \\|X \\|^2 \\big] \\cdots \\big]\\big].\n\\end{equation*}\nSince the random variables $\\{\\xi_n\\}_{n \\in \\N}$ are jointly independent, this expectation \ncoincides with the expectation with respect to the joint probability distribution of $\\xi_1, \n\\ldots, \\xi_n$. We also note here that if one of the following statements does not involve an expectation but does contain a random variable, then it is assumed to hold almost surely (a.s.) even if this is not explicitly stated.\n\nFor a random variable $\\xi$ on $\\Omega$, we consider the function $f(\\xi, \\cdot ) \\colon \n\\Omega \\times H \\to \\R$ such that\n\\begin{equation*}\n  F(w) = \\E_{\\xi} \\big[ f(\\xi, w)\\big],\n\\end{equation*}\nand aim to find\n\\begin{equation*}\n  w^* = \\argmin_{w} F(w).\n\\end{equation*}\nThe  existence of such a minimum will be guaranteed by a strong convexity assumption below. We note that this means that $\\nabla F(w^*) = 0$.\n\n\n\\begin{assumption} \\label{ass:fStoch}\n  Let $f(\\xi, \\cdot ) \\colon \\Omega \\times H \\to \\R$ be given such that\n  \\begin{itemize}\n    \\item $\\dual{\\iota \\nabla f (v)}{w} = \\lim_{h \\to 0} \\frac{f(v + hw) - f(v)}{h}$ a.s. \n      for all $v,w \\in H$, i.e., $f$ is G\\^{a}teaux differentiable a.s.;\n    \\item there exists \n      $\\mu_{\\xi} \\colon \\Omega \\to [0,\\infty)$ with $\\E_{\\xi} [\\mu_{\\xi}] =: \\mu \\in (0,\\infty)$ \n      such that\n      \\begin{equation*}\n        \\dual{\\iota \\nabla f(\\xi,v) - \\iota \\nabla f(\\xi,w)}{v - w} \\geq \\mu_{\\xi} \\|v - w\\|^2 \\quad \n        \\text{a.s. for all } v,w \\in H;\n      \\end{equation*}\n    \\item there exists \n    $L_{\\xi} \\colon \\Omega \\to [0,\\infty)$ with $\\big(\\E_{\\xi}\\big[ L_{\\xi}^2 \n    \\big]\\big)^{\\frac{1}{2}} =: L \\in (0,\\infty)$ such that\n    \\begin{equation*}\n      \\| \\iota \\nabla f(\\xi, v) - \\iota \\nabla f(\\xi, w) \\| \\leq L_{\\xi} \\| v - w\\| \\quad \n      \\text{a.s. for all } v,w \\in H;\n    \\end{equation*}\n    \\item $\\big(\\E_{\\xi} \\big[\\| \\iota \\nabla f(\\xi,w^*) \\|^2 \\big]\\big)^{\\frac{1}{2}} =: \n    \\sigma \\in [0,\\infty)$.\n  \\end{itemize}\n\\end{assumption}\n\n\\begin{assumption}\\label{ass:fHigherIntegrability}\n  Let $f$ be given such that Assumption~\\ref{ass:fStoch} is fulfilled. Further, assume that \n  for all $v,w \\in H$\n  \\begin{itemize}\n    \\item $\\dual{\\iota \\nabla f(\\xi,v) - \\iota \\nabla f(\\xi,w)}{v - w} \\geq \\mu_{\\xi} \\|v - w\\|^2$ \n    with $\\big(\\E_{\\xi} \\big[\\| \\mu_{\\xi}^2 \\big]\\big)^{\\frac{1}{2}} =: \n    \\mu_2 \\in (0,\\infty)$.\n    \\item $ \\| \\iota \\nabla f(\\xi, v) - \\iota \\nabla f(\\xi, w) \\| \\leq L_{\\xi} \\| v - w\\|$ with \n    $\\big(\\E_{\\xi}\\big[ L_{\\xi}^4 \\big]\\big)^{\\frac{1}{4}} =: L_4 \\in (0,\\infty)$;\n    \\item $\\big(\\E_{\\xi} \\big[\\| \\iota \\nabla f(\\xi,w^*) \\|^4 \\big]\\big)^{\\frac{1}{4}} =: \n    \\sigma_4 \\in [0,\\infty)$.\n  \\end{itemize}\n\\end{assumption}\n\nThroughout the paper, we will assume that $\\{\\xi_n\\}_{n \\in \\N}$ is a given a family of \njointly independent random variables and we will abbreviate $f_n(w) = f(\\xi_n,w)$ for $n \n\\in \\N$. The $\\xi_n$ typically correspond to what batches we choose in each iteration, \ni.e.\\ on which part of the data we evaluate the gradient.\nLet $\\{\\alpha_n\\}_{n \\in \\N}$ be a sequence of of positive real numbers.\nWe then consider the stochastic tamed Euler scheme\n\\begin{align}\\label{eq:stochTamedEuler}\n  w^{n+1} = w^n - \\frac{\\alpha_n \\iota \\nabla f_n(w^n)}{1 + \\alpha_n \\|\\iota \n  \\nabla f_n(w^n)\\|} \\quad \\text{for }n \\in \\N, \n  \\quad w^1 = w_1.\n\\end{align}\n\nNote that it is also possible to choose a random initial value $w^1$, and our convergence statements can be \nextended to this setting in a straightforward way. For simplicity, we restrict ourselves to a fixed initial \nvalue $w_1 \\in H$ in the following.\n\n\nGiven  $z \\in H$ and $\\alpha > 0$, we define $T_{\\alpha f_n, z}(w) \\colon \\Omega \\times \nH \\to H$ by\n\\begin{align*}\n  T_{\\alpha f_n, z}(w)\n  = w - \\frac{\\alpha \\iota \\nabla f_n(w)}{1 + \\alpha \\|\\iota \\nabla f_n(z)\\|}.\n\\end{align*}\nThis implies that the next iterate $w^{n+1}$ is given by $T_{\\alpha_n f_n, w^n}(w^n)$. \n","units":[{"start":13691,"end":15441,"start_line":274,"end_line":304,"raw_tex":"In the following, let $(H, \\dual{\\cdot}{\\cdot}, \\|\\cdot \\|)$ be a real Hilbert space. Its \ndual space is denoted\nby $(H^*, \\dual[H^*]{\\cdot}{\\cdot}, \\| \\cdot \\|_{H^*} )$. \nSince $H$ is a Hilbert space, there exists an isometric isomorphism $\\iota \\colon \nH^* \\to H$ such that $\\iota^{-1} \\colon H \\to H^*$ with $\\iota^{-1}: v \\mapsto \n\\dual{v}{\\cdot}$. \nWe denote by $\\N$ the natural numbers, not including $0$.\n\nLet $(\\Omega, \\F, \\P )$ be a complete probability space and let $\\{\\xi_n\\}_{n \\in \\N}$ be \na family of jointly independent random variables on $\\Omega$.\nFor a random variable $X \\colon \\Omega \\to H$, let $\\E_{\\xi}[X]$ denote the \nexpectation with respect to the probability distribution of $\\xi$. \nWe are mainly interested in the total expectation\n\\begin{equation*}\n  \\E_n \\big[ \\|X\\|^2 \\big] = \\E_{\\xi_1}\\big[ \\E_{\\xi_2}\\big[ \\cdots \n  \\E_{\\xi_n} \\big[ \\|X \\|^2 \\big] \\cdots \\big]\\big].\n\\end{equation*}\nSince the random variables $\\{\\xi_n\\}_{n \\in \\N}$ are jointly independent, this expectation \ncoincides with the expectation with respect to the joint probability distribution of $\\xi_1, \n\\ldots, \\xi_n$. We also note here that if one of the following statements does not involve an expectation but does contain a random variable, then it is assumed to hold almost surely (a.s.) even if this is not explicitly stated.\n\nFor a random variable $\\xi$ on $\\Omega$, we consider the function $f(\\xi, \\cdot ) \\colon \n\\Omega \\times H \\to \\R$ such that\n\\begin{equation*}\n  F(w) = \\E_{\\xi} \\big[ f(\\xi, w)\\big],\n\\end{equation*}\nand aim to find\n\\begin{equation*}\n  w^* = \\argmin_{w} F(w).\n\\end{equation*}\nThe  existence of such a minimum will be guaranteed by a strong convexity assumption below. We note that this means that $\\nabla F(w^*) = 0$.\n","sha256_utf8":"66844bc9dbab7cf3fa30a41c9660ff9c0ab2b4d419a963eae319f61b907e308a"},{"start":15752,"end":16855,"start_line":312,"end_line":334,"raw_tex":"\\begin{assumption} \\label{ass:fStoch}\n  Let $f(\\xi, \\cdot ) \\colon \\Omega \\times H \\to \\R$ be given such that\n  \\begin{itemize}\n    \\item $\\dual{\\iota \\nabla f (v)}{w} = \\lim_{h \\to 0} \\frac{f(v + hw) - f(v)}{h}$ a.s. \n      for all $v,w \\in H$, i.e., $f$ is G\\^{a}teaux differentiable a.s.;\n    \\item there exists \n      $\\mu_{\\xi} \\colon \\Omega \\to [0,\\infty)$ with $\\E_{\\xi} [\\mu_{\\xi}] =: \\mu \\in (0,\\infty)$ \n      such that\n      \\begin{equation*}\n        \\dual{\\iota \\nabla f(\\xi,v) - \\iota \\nabla f(\\xi,w)}{v - w} \\geq \\mu_{\\xi} \\|v - w\\|^2 \\quad \n        \\text{a.s. for all } v,w \\in H;\n      \\end{equation*}\n    \\item there exists \n    $L_{\\xi} \\colon \\Omega \\to [0,\\infty)$ with $\\big(\\E_{\\xi}\\big[ L_{\\xi}^2 \n    \\big]\\big)^{\\frac{1}{2}} =: L \\in (0,\\infty)$ such that\n    \\begin{equation*}\n      \\| \\iota \\nabla f(\\xi, v) - \\iota \\nabla f(\\xi, w) \\| \\leq L_{\\xi} \\| v - w\\| \\quad \n      \\text{a.s. for all } v,w \\in H;\n    \\end{equation*}\n    \\item $\\big(\\E_{\\xi} \\big[\\| \\iota \\nabla f(\\xi,w^*) \\|^2 \\big]\\big)^{\\frac{1}{2}} =: \n    \\sigma \\in [0,\\infty)$.\n  \\end{itemize}\n\\end{assumption}","sha256_utf8":"0310756367ecf98e20fe33efad81d95e42d3e5e931625b826e8043b097479675"},{"start":17089,"end":17806,"start_line":341,"end_line":353,"raw_tex":"\\begin{assumption}\\label{ass:fHigherIntegrability}\n  Let $f$ be given such that Assumption~\\ref{ass:fStoch} is fulfilled. Further, assume that \n  for all $v,w \\in H$\n  \\begin{itemize}\n    \\item $\\dual{\\iota \\nabla f(\\xi,v) - \\iota \\nabla f(\\xi,w)}{v - w} \\geq \\mu_{\\xi} \\|v - w\\|^2$ \n    with $\\big(\\E_{\\xi} \\big[\\| \\mu_{\\xi}^2 \\big]\\big)^{\\frac{1}{2}} =: \n    \\mu_2 \\in (0,\\infty)$.\n    \\item $ \\| \\iota \\nabla f(\\xi, v) - \\iota \\nabla f(\\xi, w) \\| \\leq L_{\\xi} \\| v - w\\|$ with \n    $\\big(\\E_{\\xi}\\big[ L_{\\xi}^4 \\big]\\big)^{\\frac{1}{4}} =: L_4 \\in (0,\\infty)$;\n    \\item $\\big(\\E_{\\xi} \\big[\\| \\iota \\nabla f(\\xi,w^*) \\|^4 \\big]\\big)^{\\frac{1}{4}} =: \n    \\sigma_4 \\in [0,\\infty)$.\n  \\end{itemize}\n\\end{assumption}","sha256_utf8":"14974d412093126f74585e1cad76b986781cbb6d9b6bd8c2b489e9a946bbdb83"},{"start":21069,"end":21715,"start_line":417,"end_line":427,"raw_tex":"Throughout the paper, we will assume that $\\{\\xi_n\\}_{n \\in \\N}$ is a given a family of \njointly independent random variables and we will abbreviate $f_n(w) = f(\\xi_n,w)$ for $n \n\\in \\N$. The $\\xi_n$ typically correspond to what batches we choose in each iteration, \ni.e.\\ on which part of the data we evaluate the gradient.\nLet $\\{\\alpha_n\\}_{n \\in \\N}$ be a sequence of of positive real numbers.\nWe then consider the stochastic tamed Euler scheme\n\\begin{align}\\label{eq:stochTamedEuler}\n  w^{n+1} = w^n - \\frac{\\alpha_n \\iota \\nabla f_n(w^n)}{1 + \\alpha_n \\|\\iota \n  \\nabla f_n(w^n)\\|} \\quad \\text{for }n \\in \\N, \n  \\quad w^1 = w_1.\n\\end{align}","sha256_utf8":"3d2675efc09dc0ac713eed273b900263b87083f28bdfacd49e1bce8c17cb293c"},{"start":21716,"end":21970,"start_line":428,"end_line":430,"raw_tex":"Note that it is also possible to choose a random initial value $w^1$, and our convergence statements can be \nextended to this setting in a straightforward way. For simplicity, we restrict ourselves to a fixed initial \nvalue $w_1 \\in H$ in the following.\n","sha256_utf8":"53fa36d4d84c5cce8cc4c2d8b1d73f9a3524e4ec4cfed09bb7479ebacc691a15"},{"start":40861,"end":41180,"start_line":801,"end_line":807,"raw_tex":"Given  $z \\in H$ and $\\alpha > 0$, we define $T_{\\alpha f_n, z}(w) \\colon \\Omega \\times \nH \\to H$ by\n\\begin{align*}\n  T_{\\alpha f_n, z}(w)\n  = w - \\frac{\\alpha \\iota \\nabla f_n(w)}{1 + \\alpha \\|\\iota \\nabla f_n(z)\\|}.\n\\end{align*}\nThis implies that the next iterate $w^{n+1}$ is given by $T_{\\alpha_n f_n, w^n}(w^n)$. \n","sha256_utf8":"8d0d03081d9b1dddba91ee6892d432593602fe36779cb7edfa903a77e5bd93aa"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{thm:ConvNorm1}\n  Let Assumption~\\ref{ass:fHigherIntegrability} be fulfilled.\n  For $\\alpha_n = \\frac{\\vartheta}{n + \\gamma}$, $n \\in \\N$, with $\\gamma \\in [0,\\infty)$, \n  $\\vartheta \\in (0,\\frac{1 + \\gamma}{2\\mu}]$ and\n  \\begin{align*}\n    K = 2 \\vartheta^2 L \\mu_2 M_4^{\\frac{3}{4}} \n    + \\big(2 \\vartheta^2 L^2 + 2 \\vartheta^2 L \\sigma + 2 \\vartheta^2 \\mu_2 \\sigma \n    \\big) M_2\n    + 2 \\vartheta^2 \\sigma^2 M_2^{\\frac{1}{2}} + 2 \\vartheta^2 \\sigma^2,\n  \\end{align*}\n  it follows that\n  \\begin{align*}\n    &\\E_n \\big[\\| w^{n+1} - w^* \\|^2\\big]\\\\\n    &\\leq \\| w_1 - w^*\\|^2 (1+\\gamma)^{2 \\vartheta \\mu}(n+1+\\gamma)^{-2 \\vartheta \\mu} \\\\\n    &\\quad + \\exptextB{\\frac{2 \\vartheta \\mu}{1+\\gamma}} K \\begin{cases}\n      (n+1+\\gamma)^{-1} \\frac{1}{2 \\vartheta \\mu-1}, &2 \\vartheta \\mu \\in (1,\\infty),\\\\\n      (n+1+\\gamma)^{-1} (1 + \\ln{(n + \\gamma)}), &2 \\vartheta \\mu = 1,\\\\\n      (n+1+\\gamma)^{-2 \\vartheta \\mu} \\frac{(1+\\gamma)^{2 \\vartheta \\mu-2}(2 \\vartheta \n      \\mu-2-\\gamma)}{2 \\vartheta \\mu-1}, &2 \\vartheta \\mu \\in [0,1),\n    \\end{cases}\n  \\end{align*}\n  for every $n \\in \\N$.\n\\end{theorem}","units":[{"start":48984,"end":50105,"start_line":976,"end_line":998,"raw_tex":"\\begin{theorem}\\label{thm:ConvNorm1}\n  Let Assumption~\\ref{ass:fHigherIntegrability} be fulfilled.\n  For $\\alpha_n = \\frac{\\vartheta}{n + \\gamma}$, $n \\in \\N$, with $\\gamma \\in [0,\\infty)$, \n  $\\vartheta \\in (0,\\frac{1 + \\gamma}{2\\mu}]$ and\n  \\begin{align*}\n    K = 2 \\vartheta^2 L \\mu_2 M_4^{\\frac{3}{4}} \n    + \\big(2 \\vartheta^2 L^2 + 2 \\vartheta^2 L \\sigma + 2 \\vartheta^2 \\mu_2 \\sigma \n    \\big) M_2\n    + 2 \\vartheta^2 \\sigma^2 M_2^{\\frac{1}{2}} + 2 \\vartheta^2 \\sigma^2,\n  \\end{align*}\n  it follows that\n  \\begin{align*}\n    &\\E_n \\big[\\| w^{n+1} - w^* \\|^2\\big]\\\\\n    &\\leq \\| w_1 - w^*\\|^2 (1+\\gamma)^{2 \\vartheta \\mu}(n+1+\\gamma)^{-2 \\vartheta \\mu} \\\\\n    &\\quad + \\exptextB{\\frac{2 \\vartheta \\mu}{1+\\gamma}} K \\begin{cases}\n      (n+1+\\gamma)^{-1} \\frac{1}{2 \\vartheta \\mu-1}, &2 \\vartheta \\mu \\in (1,\\infty),\\\\\n      (n+1+\\gamma)^{-1} (1 + \\ln{(n + \\gamma)}), &2 \\vartheta \\mu = 1,\\\\\n      (n+1+\\gamma)^{-2 \\vartheta \\mu} \\frac{(1+\\gamma)^{2 \\vartheta \\mu-2}(2 \\vartheta \n      \\mu-2-\\gamma)}{2 \\vartheta \\mu-1}, &2 \\vartheta \\mu \\in [0,1),\n    \\end{cases}\n  \\end{align*}\n  for every $n \\in \\N$.\n\\end{theorem}","sha256_utf8":"f919a59b470e017b93c469d56f251e53f9ebb72cda3c51f294c0858577f0b450"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma} \\label{lem:Fconsequences}\n  Let Assumption~\\ref{ass:fStoch} be fulfilled. \n  Then $F$ is strongly convex with convexity constant $\\mu$ and $\\nabla F$ is Lipschitz \n  continuous with Lipschitz constant $L_F \\le L$, i.e.\\ for all $v,w \\in H$ it holds that\n  \\begin{align*}\n    \\| \\nabla F(v) - \\nabla F(w) \\|_{H^*} \n    &= \\| \\iota \\nabla F(v) - \\iota \\nabla F(w) \\| \n    \\leq L_F \\| v - w\\| \n    \\leq L \\| v - w\\| \\quad \\text{and} \\\\\n    F(v) &\\geq F(w) + \\dual{\\iota \\nabla F(w) }{ v - w }+ \\frac{\\mu}{2} \\|v - w\\|^2.\n  \\end{align*}\n  Further, the first inequality implies that\n  \\begin{align*}\n    F(v) \n    &\\leq F(w)+\\dual{\\iota \\nabla F(w) }{ v - w }+ \\frac{L}{2} \\|v - w\\|^2.\n  \\end{align*}\n  Finally, there exists a unique $w^* \\in H$ such that $F(w^*) = \\min_{w \\in H} F(w)$.\n\\end{lemma}\n\n\\begin{proof}\n  Using $\\nabla F(w) = \\E_{\\xi} \\big[ \\nabla f(\\xi, w)\\big]$, we obtain\n  \\begin{align*}\n    \\dual{\\iota \\nabla F(v) - \\iota \\nabla F(w)}{v - w} \\geq \\mu \\|v - w\\|^2 \\quad \n    \\text{for all } v,w \\in H.\n  \\end{align*}\n  Thus, the function $v\\mapsto \\nabla F(v) - \\mu \\iota^{-1} v$ is monotone. Applying \n  \\cite[Proposition~25.10]{ZeidlerIIB.1990}, it follows that $v \\mapsto F(v) - \\frac{\\mu}{2} \n  \\|v\\|^2$ is convex such that $F$ is strongly convex and the variational inequality stated \n  in the lemma is fulfilled. The Lipschitz continuity of $\\iota \\nabla F$ \n  similarly follows from the Lipschitz continuity of $\\iota \\nabla f(\\xi, \\cdot)$ by the \n  identification provided in Lemma~\\ref{lem:swap_diff_and_expectation}. The final \n  inequality follows by expanding $F$ in a zeroth-order Taylor expansion \n  around $w$ and using the Lipschitz continuity. See e.g.\\ \n  \\cite[Appendix~B]{BottouCurtisNocedal.2018} for more details.\n %\n  We note that $F$ is coercive since it is strongly convex. Combined with the G\\^{a}teaux \n  differentiability, this guarantees the existence of a unique global \n  minimum, see e.g.~\\cite[Theorem 25.D, Proposition~25.20 and \n  Corollary~25.15]{ZeidlerIIB.1990}.\n\\end{proof}\n\n\\begin{lemma} \\label{lem:apriori2}\n  Let Assumption~\\ref{ass:fStoch} be fulfilled and let $\\{\\alpha_n\\}_{n \\in \\N}$ be a \n  sequence of positive real numbers such that $\\sum_{n =1}^{\\infty} \\alpha_n^2 < \n  \\infty$.\n  For $\\Phi \\in [0,\\infty)$ the a priori bound\n  \\begin{align*}\n    &\\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \\\\\n    &\\leq \\| w_1 - w^*\\|^2 \\exptextB{ \\sum_{i=1}^{\\infty} \\big(2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n      \\alpha_i^2 \\big\\} + 4 \\alpha_i^2 L^2 m_i\\big)} \\\\\n    &\\quad + \\sum_{i=1}^{\\infty} \\big(\\Phi^2 \\min \\big\\{ \\E_i \\big[ \\|\\iota \\nabla \n    f_i(w^i)\\|^{-2} \\big], \n    \\alpha_i^2 \\big\\} + 2 ( 1- m_i) + 2 \\min \\big\\{1, 2 \\alpha_i^2 \\sigma^2 \\big\\}\\big)\\\\\n    &\\qquad \\times \\exptextB{\\sum_{j=i+1}^{\\infty} \\big(2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n      \\alpha_j^2 \\big\\} + 4 \\alpha_j^2 L^2 m_j\\big)} =: M_2\n  \\end{align*}\n  is fulfilled, where $\\min\\big\\{\\Phi^{-2}, x \\big\\} = x$ for $\\Phi = 0$ and every $x \\in \\R$ \n  and \n  \\begin{align*}\n    m_i =\n    \\begin{cases}\n      1, & \\text{\\emph{if }} 2 \\alpha_i^2 L^2 \\E_{i-1} \\big[ \\| w^i - w^*\\|^2\\big] \\le 1, \\\\\n      0, & \\text{\\emph{otherwise}}.\n    \\end{cases}\n  \\end{align*}\n  Furthermore, there exists $n_0 \\in \\N$ such that $m_i = 1$ for all $i \\geq n_0$.\n\\end{lemma}\n\n\\begin{proof}[Proof of Lemma~\\ref{lem:apriori2}]\n  We test the scheme defined by~\\eqref{eq:stochTamedEuler}\n  with $w^n - w^*$, in order to obtain that\n  \\begin{align}\\label{eq:proofApriori1}\n    \\begin{split}\n      &\\dual{w^{n+1} - w^* - (w^n - w^*)}{w^n - w^*} \\\\\n      &\\quad + \\frac{\\alpha_n \\dual{\\iota \\nabla f_n(w^n) - \\iota \\nabla \n          f_n(w^*)}{w^n - w^*}}{1 \n        + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\|} \n      = - \\frac{\\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*}}{1 + \\alpha_n \\|\\iota \n        \\nabla f_n(w^n)\\|}.\n    \\end{split}\n  \\end{align}\n  Using the identity $\\dual{u-v}{u} = \\frac{1}{2} (\\|u\\|^2 - \\|v\\|^2 + \\|u-v\\|^2)$, $u,v \\in H$, the \n  first summand on the left-hand side can be written as\n  \\begin{align*}\n    &- \\dual{w^n - w^* - (w^{n+1} - w^*)}{w^n - w^*} \\\\\n    &= - \\frac{1}{2} \\big( \\|w^n - w^*\\|^2 - \\|w^{n+1} - w^*\\|^2 + \\|w^{n+1} - w^n\\|^2 \\big)\\\\\n    &= \\frac{1}{2} \\Big(\\|w^{n+1} - w^*\\|^2 - \\|w^n - w^*\\|^2 - \\frac{\\alpha_n^2 \\| \\iota \n      \\nabla  f_n(w^n)\\|^2 }{(1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\|)^2}\\Big), \n  \\end{align*}\n  where we inserted the scheme in the last step. \n  Thus, inserting the monotonicity condition for $f_n$ into \n  \\eqref{eq:proofApriori1} and multiplying the inequality with the factor two, it follows that\n  \\begin{align*}\n    &\\|w^{n+1} - w^*\\|^2 - \\|w^n - w^*\\|^2 \\\\\n    &\\leq - \\frac{2\\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*}}{1 + \\alpha_n \n      \\|\\iota \\nabla  \n      f_n(w^n)\\|} \n    + \\frac{\\alpha_n^2 \\| \\iota \\nabla f_n(w^n)\\|^2 }{(1 + \\alpha_n \\|\\iota \\nabla  \n      f_n(w^n)\\|)^2} \n    =: I_1 + I_2.\n  \\end{align*}\n  Since the tamed Euler scheme is the forward Euler scheme with a second order \n  perturbation, it follows that\n  \\begin{align*}\n    \\frac{\\alpha_n \\iota \\nabla f_n(w^*)}{1 + \\alpha_n \\|\\iota \\nabla  f_n(w^n)\\|}\n    = \\frac{\\alpha_n \\iota \\nabla f_n(w^*)}{1 + \\alpha_n \\Phi}\n    + \\frac{ \\alpha_n^2 \\big(\\Phi - \\|\\iota \\nabla f_n(w^n)\\|\\big) \\iota \\nabla f_n(w^*) \n    }{(1 + \\alpha_n \\|\\iota \\nabla  f_n(w^n)\\|) (1 + \\alpha_n \\Phi )}\n  \\end{align*}\n  for $\\Phi \\in [0,\\infty)$. Note that we have $w^*$ in the numerator of the left-hand-side \n  but $w^n$ in the denominator. We insert this equality into $I_1$ and use the \n  Cauchy--Schwarz inequality and \n  Young's  inequality for products in order to obtain\n  \\begin{align*}\n    I_1 \n    &= - \\frac{\\alpha_n \\dual{2 \\iota \\nabla f_n(w^*)}{w^n - w^*}} {1 + \\alpha_n \\|\\iota \\nabla  \n      f_n(w^n)\\|} \\\\\n    &\\leq - \\frac{2 \\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*} }{1 + \\alpha_n \\Phi}\n    + \\frac{2 \\alpha_n^2 \\Phi \\| \\iota \\nabla f_n(w^*)\\| \\| w^n - w^*\\| }{(1 + \\alpha_n \\|\\iota \n    \\nabla  f_n(w^n)\\|) (1 + \\alpha_n \\Phi )} \\\\\n    &\\quad + \\frac{2 \\alpha_n^2 \\|\\iota \\nabla f_n(w^n)\\| \\| \\iota \\nabla f_n(w^*)\\| \\| w^n - \n    w^*\\| }{(1 + \\alpha_n \\|\\iota \\nabla  f_n(w^n)\\|) (1 + \\alpha_n \\Phi )} \\\\\n    &\\leq - \\frac{2 \\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*} }{1 + \\alpha_n \\Phi}\n    + \\frac{ \\alpha_n^2 \\big(\\Phi^2 + \\|\\iota \\nabla f_n(w^n)\\|^2\\big)}{(1 + \\alpha_n \\|\\iota \n    \\nabla  f_n(w^n)\\|)^2} \\\\\n    &\\quad + \\frac{2 \\alpha_n^2 \\| \\iota \\nabla f_n(w^*)\\|^2 \\| w^n - w^*\\|^2 }{(1 + \\alpha_n \n    \\Phi )^2}\n    =: I_{1,1} + I_{1,2} + I_{1,3}.\n  \\end{align*}\n  For $I_{1,1} = - \\frac{2 \\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*} }{1 + \\alpha_n \n  \\Phi}$, we notice that $\\E_{\\xi_n} [I_{1,1}] = 0$. Moreover, for $I_{1,2} = \\frac{\\alpha_n^2 \n  ( \n  \\Phi^2 + \\|\\iota \\nabla f_n(w^n)\\|^2 )}{(1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\|)^2}$, we get\n  \\begin{align*}\n    I_{1,2}\n    &\\leq \\Phi^2 \\min \\big\\{ \\|\\iota \\nabla f_n(w^n)\\|^{-2} , \\alpha_n^2 \\big\\}\n    + \\min \\big\\{1, \\alpha_n^2 \\|\\iota \\nabla f_n(w^n)\\|^2 \\big\\} \\\\\n    &\\leq \\Phi^2 \\min \\big\\{ \\|\\iota \\nabla f_n(w^n)\\|^{-2} , \\alpha_n^2 \\big\\}\n    + \\min \\big\\{1, 2 \\alpha_n^2 L_{\\xi_n}^2 \\|w^n - w^*\\|^2\\big\\}\\\\ \n    &\\quad + \\min \\big\\{1,2 \\alpha_n^2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\big\\}\n  \\end{align*}\n  and\n  \\begin{align*}\n    I_{1,3}\n    = \\frac{2\\alpha_n^2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\|w^n - w^*\\|^2}{(1 + \\alpha_n \\Phi )^2}\n    \\leq 2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\min \\big\\{\\Phi^{-2}, \\alpha_n^2 \\big\\} \\|w^n - w^*\\|^2.\n  \\end{align*}\n  A bound for $I_2$ is given by\n  \\begin{align*}\n    I_2 &= \\frac{\\alpha_n^2 \\| \\iota \\nabla f_n(w^n)\\|^2 }{(1 + \\alpha_n \\|\\iota \\nabla \n    f_n(w^n)\\| )^2}\n    \\leq \\min \\big\\{1, \\alpha_n^2 \\| \\iota \\nabla f_n(w^n)\\|^2 \\big\\}\\\\\n    &\\leq \\min \\big\\{1, 2 \\alpha_n^2 L_{\\xi_n}^2 \\| w^n - w^*\\|^2 \\big\\}\n    + \\min \\big\\{1, 2 \\alpha_n^2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\big\\}.\n  \\end{align*}  \n  Then it follows\n  \\begin{align}\n    \\label{eq:aprioriBoundMn}\n    \\begin{split}\n      &\\|w^{n+1} - w^*\\|^2 - \\|w^n - w^*\\|^2 \\leq  I_1 + I_2\\\\\n      &\\leq - \\frac{2 \\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*} }{1 + \\alpha_n \\Phi}\n      + \\Phi^2 \\min \\big\\{ \\|\\iota \\nabla f_n(w^n)\\|^{-2} , \\alpha_n^2 \\big\\}\\\\\n      &\\quad + 2 \\min \\big\\{1, 2 \\alpha_n^2 L_{\\xi_n}^2 \\|w^n - w^*\\|^2\\big\\} + 2 \\min \n      \\big\\{1,2 \\alpha_n^2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\big\\}\\\\\n      &\\quad + 2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\min \\big\\{\\Phi^{-2}, \\alpha_n^2 \\big\\} \\|w^n - \n      w^*\\|^2.\n    \\end{split}\n  \\end{align}\n  Taking the $\\E_n$-expectation, we then obtain\n  \\begin{equation}\n    \\label{eq:aprioriBoundMnExp}\n    \\begin{aligned}\n      &\\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \\\\\n      &\\quad \\leq \\big(1 + 2 \\sigma^2 \\min\\big\\{ \\Phi^{-2}, \\alpha_n^2 \\big\\} \\big) \\E_{n-1} \n      \\big[ \n      \\|w^n - w^*\\|^2 \\big]\\\\\n      &\\qquad + \\Phi^2 \\min \\big\\{ \\E_n \\big[\\|\\iota \\nabla f_n(w^n)\\|^{-2}\\big], \\alpha_n^2 \n      \\big\\}\\\\\n      &\\qquad + 2 \\min \\big\\{1, 2 \\alpha_n^2 L^2 \\E_{n-1} \\big[ \\|w^n - w^*\\|^2 \\big]\\big\\} \n      + 2 \\min \\big\\{1,2 \\alpha_n^2 \\sigma^2\\big\\}\\\\\n      &\\quad= \\big(1 + 2 \\min\\big\\{ \\Phi^{-2}, \\alpha_n^2 \\big\\} \\sigma^2+ 4 \\alpha_n^2 L^2 \n      m_n\\big) \\E_{n-1} \\big[ \\|w^n - w^*\\|^2 \\big]\\\\\n      &\\qquad + \\Phi^2 \\min \\big\\{ \\E_n \\big[\\|\\iota \\nabla f_n(w^n)\\|^{-2}\\big], \n      \\alpha_n^2 \\big\\}\n      + 2 (1-m_n) \n      + 2 \\min \\big\\{1,2 \\alpha_n^2 \\sigma^2\\big\\},\n    \\end{aligned}\n  \\end{equation}\n  with $m_n$ defined as in the lemma statement. Reinserting the bound \n  repeatedly thus yields\n  \\begin{align*}\n    &\\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \\\\\n    &\\leq \\| w_1 - w^*\\|^2 \\prod_{i=1}^{n} \\big(1 + 2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n    \\alpha_i^2 \\big\\} + 4 \\alpha_i^2 L^2 m_i\\big) \\\\\n    &\\quad + \\sum_{i=1}^{n} \\big(\\Phi^2 \\min \\big\\{ \\E_i \\big[ \\|\\iota \\nabla \n    f_i(w^i)\\|^{-2} \\big], \n    \\alpha_i^2 \\big\\} + 2 ( 1- m_i) + 2 \\min \\big\\{1, 2 \\alpha_i^2 \\sigma^2 \\big\\}\\big)\\\\\n    &\\qquad \\times \\prod_{j=i+1}^{n} \\big(1 + 2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n    \\alpha_j^2 \\big\\} + 4 \\alpha_j^2 L^2 m_j\\big).\n  \\end{align*}\n  Finally, we apply the inequality $1 + x \\leq \\exptext{x}$, $x \\in \\R$, and make \n  the bound independent of $n$ by bounding the final sums by the corresponding infinite \n  sums, in order to obtain\n  \\begin{align*}\n    &\\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \\\\\n    &\\leq \\| w_1 - w^*\\|^2 \\exptextB{ \\sum_{i=1}^{\\infty} \\big(2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n      \\alpha_i^2 \\big\\} + 4 \\alpha_i^2 L^2 m_i\\big)} \\\\\n    &\\quad + \\sum_{i=1}^{\\infty} \\big(\\Phi^2 \\min \\big\\{ \\E_i \\big[ \\|\\iota \\nabla \n    f_i(w^i)\\|^{-2} \\big], \n    \\alpha_i^2 \\big\\} + 2 ( 1- m_i) + 2 \\min \\big\\{1, 2 \\alpha_i^2 \\sigma^2 \\big\\}\\big)\\\\\n    &\\qquad \\times \\exptextB{\\sum_{j=i+1}^{\\infty} \\big(2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n      \\alpha_j^2 \\big\\} + 4 \\alpha_j^2 L^2 m_j\\big)}.\n  \\end{align*}\n  It remains to verify, that there exists $n_0 \\in \\N$ such that $m_n = 1$ for all $n \\geq \n  n_0$. This can be done by estimating \\eqref{eq:aprioriBoundMnExp} and following a \n  similar line of argumentation as before. First, we can write\n  \\begin{align*}\n    \\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \n    \\leq \\big(1 + 2 \\alpha_n^2 \\sigma^2 + 4 \\alpha_n^2 L^2 \\big) \\E_{n-1} \\big[ \\|w^n - \n    w^*\\|^2\\big] + \\Phi^2 \\alpha_n^2 + 4 \\alpha_n^2 \\sigma^2.\n  \\end{align*}\n  Reinserting the inequality $n -1$ times, it follows that\n  \\begin{align*}\n    &\\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \\\\\n    &\\leq \\| w_1 - w^*\\|^2 \\prod_{i=1}^{n} \\big(1 + 2 \\sigma^2 \\alpha_i^2 + 4 \\alpha_i^2 L^2 \n    \\big) \n    + \\sum_{i=1}^{n} 4 \\alpha_i^2 \\sigma^2 \\prod_{j=i+1}^{n} \\big(1 + 2 \\sigma^2 \n    \\alpha_j^2 + 4 \\alpha_j^2 L^2 \\big)\\\\\n    &\\leq \\| w_1 - w^*\\|^2 \\exptextB{ \\big(2 \\sigma^2 + 4 L^2 \\big) \\sum_{i=1}^{\\infty} \n    \\alpha_i^2 }\n    + \\sum_{i=1}^{\\infty} 4 \\alpha_i^2 \\sigma^2 \\exptextB{ \\big(2 \\sigma^2 + 4 L^2 \\big) \n    \\sum_{j=i+1}^{\\infty} \\alpha_j^2 }.\n  \\end{align*}\n  Since $\\sum_{n =1}^{\\infty} \\alpha_n^2 < \\infty$ there is thus a $n_0 \\in \\N$ such that $2 \\alpha_n^2 L^2 \\E_{n-1} \\big[ \\| w^n - w^*\\|^2\\big] \\leq 1$ for all $n \\geq n_0$.\n\\end{proof}\n\n\\begin{lemma} \\label{lem:apriori4}\n  Let Assumption~\\ref{ass:fHigherIntegrability} be fulfilled and \n  let $\\{\\alpha_n\\}_{n \\in \\N}$ be a sequence of positive real numbers such that $\\sum_{n \n  =1}^{\\infty} \\alpha_n^2 < \\infty$.\n  Then the a priori bound  \n  \\begin{align*}\n    \\E_n \\big[\\|w^{n+1} - w^*\\|^4\\big] \n    &\\leq \\|w^1 - w^*\\|^4 \\exptextB{\\sum_{i = 1}^{\\infty} c_1^i(\\alpha_i)} \\\\\n    &\\quad + \\sum_{i=1}^{\\infty} \\big(c_2^i(\\alpha_i) M_2 + c_3^i(\\alpha_i)\\big) \n    \\exptextB{\\sum_{j = i+ 1}^{\\infty} c_1^j(\\alpha_j) } =: M_4\n  \\end{align*}\n  is fulfilled for $c_1^i, c_2^i, c_3^i \\colon (0,\\infty) \\to (0,\\infty)$ such that there exist \n  $C_1^k, C_2^k, C_3^k, C_4^k \\in (0,\\infty)$ with $c_k^i(\\alpha) \\leq C_1^i \\min \\{C_2^i, \n  \\alpha^4\\} + C_3^i \\min \\{C_4^i, \\alpha^2\\}$ for all $\\alpha \\in (0,\\infty)$, $k \\in \\{1,2,3\\}$ \n  and $i \\in \\N$.\n\\end{lemma}\n\n\\begin{proof}\n  Within the proof of Lemma~\\ref{lem:apriori2}, we verified the inequality \n  \\eqref{eq:aprioriBoundMn}. Starting from this point, we find\n  \\begin{equation}\n    \\label{eq:apriori4}\n    \\|w^{n+1} - w^*\\|^2 - \\|w^n - w^*\\|^2 \\leq A_n + B_n,\n  \\end{equation}\n  where\n  \\begin{align*}\n    A_n &= -\\frac{2 \\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - \n    w^*} }{1 + \\alpha_n \\Phi} \\qquad \\text{and} \\\\\n    B_n &= \\Phi^2 \\min \\big\\{ \\|\\iota \\nabla f_n(w^n)\\|^{-2} , \\alpha_n^2 \\big\\}\n     + 2 \\min \\big\\{1, 2 \\alpha_n^2 L_{\\xi_n}^2 \\|w^n - w^*\\|^2\\big\\}\\\\\n      &\\quad + 2 \\min \\big\\{1,2 \\alpha_n^2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\big\\}\n      + 2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\min \\big\\{\\Phi^{-2}, \\alpha_n^2 \\big\\} \\|w^n - w^*\\|^2 \n  \\end{align*}\n  for $\\Phi \\in [0,\\infty)$. We note that the parameter $\\Phi$ can be chosen such that \n  $\\E_n[B_n]$ is as small as possible.\n  From this it follows that \n  \\begin{align*}\n    \\E_{\\xi_n} [A_n] &= 0,\\\\\n    \\E_{\\xi_n} [B_n] &\\leq  \\Phi^2 \\min \\big\\{ \\E_{\\xi_n} \\big[ \\|\\iota \\nabla f_n(w^n)\\|^{-2}\\big] \n    , \n    \\alpha_n^2 \\big\\}\n    + 2 \\min \\big\\{1, 2 \\alpha_n^2 L^2 \\|w^n - w^*\\|^2\\big\\}\\\\\n    &\\quad + 2 \\min \\big\\{1,2 \\alpha_n^2 \\sigma^2 \\big\\}\n    + 2 \\sigma^2 \\min \\big\\{\\Phi^{-2}, \\alpha_n^2 \\big\\} \\|w^n - w^*\\|^2, \\\\\n    \\E_{\\xi_n} [A_n^2] \n    &\\leq 4 \\sigma^2 \\min\\{ \\Phi^{-2}, \\alpha_n^2 \\} \\| w^n - w^*\\|^2 \\qquad \\text{and}\\\\\n    \\E_{\\xi_n} [B_n^2] \n    &\\leq 4 \\Phi^4 \\min \\big\\{ \\big(\\E_{\\xi_n} \\big[ \\|\\iota \\nabla f_n(w^n)\\|^{-2}\\big]\\big)^2 , \n    \\alpha_n^4 \\big\\}\n    + 16 \\min \\big\\{1, 4 \\alpha_n^4 L_4^4 \\|w^n - w^*\\|^4\\big\\}\\\\\n    &\\quad + 16 \\min \\big\\{1,4 \\alpha_n^4 \\sigma_4^4 \\big\\}\n    + 16  \\sigma_4^4 \\min \\big\\{\\Phi^{-4}, \\alpha_n^4 \\big\\} \\|w^n - w^*\\|^4.\n  \\end{align*}\n   We note that for $a,b \\in \\R$ we have the identity $(a - b)a = \\frac{1}{2} \\big( |a|^2 - |b|^2 \n   + |a - b|^2 \\big)$, and thus $|a|^2 - |b|^2 \\le 2(a-b)a$. By multiplying the inequality from \n   \\eqref{eq:apriori4} with the factor $2\\|w^{n+1} - w^*\\|^2$, we therefore obtain\n  \\begin{align*}\n    \\|w^{n+1} - w^*\\|^4 - \\|w^n - w^*\\|^4\n    &\\leq 2\\big( A_n + B_n \\big) \\|w^{n+1} - w^*\\|^2\\\\\n    &\\leq 2\\big( A_n + B_n \\big) \\big( \\|w^n - w^*\\|^2 + A_n + B_n  \\big)\\\\\n    &\\leq 2(A_n + B_n) \\|w^n - w^*\\|^2  + 4 A_n^2 + 4 B_n^2 \n  \\end{align*}\n  and in $\\E_{\\xi_n}$-expectation\n  \\begin{align*}\n    & \\E_{\\xi_n} \\big[ \\|w^{n+1} - w^*\\|^4\\big] - \\|w^n - w^*\\|^4\\\\\n    &\\leq 4 \\Big(2 L^2 \\min \\big\\{\\tfrac{1}{2} L^{-2} \\|w^n - w^*\\|^{-2}, \\alpha_n^2 \\big\\}\n    + \\sigma^2 \\min \\big\\{\\Phi^{-2}, \\alpha_n^2 \\big\\}\\\\\n    &\\qquad + 64 L_4^4 \\min \\big\\{ \\tfrac{1}{4} L_4^{-4} \\|w^n - w^*\\|^{-4}, \\alpha_n^4 \n    \\big\\} + 16 \\sigma_4^4 \\min \\big\\{\\Phi^{-4}, \\alpha_n^4 \\big\\} \\Big)\\|w^n - w^*\\|^4 \\\\\n    &\\quad + 2 \\Big( \\Phi^2 \\min \\big\\{ \\E_{\\xi_n} \\big[ \\|\\iota \\nabla f_n(w^n)\\|^{-2}\\big] , \n    \\alpha_n^2 \\big\\} \n    + 4 \\sigma^2\\min \\big\\{\\tfrac{1}{2} \\sigma^{-2} , \\alpha_n^2 \\big\\} \\\\\n    &\\qquad + 8  \\sigma^2 \\min\\{ \\Phi^{-2}, \\alpha_n^2 \\} \\Big) \\|w^n - w^*\\|^2 \\\\\n    &\\quad + 16 \\Phi^4 \\min \\big\\{ \\big(\\E_{\\xi_n} \\big[ \\|\\iota \\nabla \n    f_n(w^n)\\|^{-2}\\big]\\big)^2 \n    , \\alpha_n^4 \\big\\}\n    + 256 \\sigma_4^4 \\min \\big\\{ \\tfrac{1}{4}\\sigma_4^{-4},\\alpha_n^4 \\big\\}\\\\\n    %\n    &=: c_1^n (\\alpha_n) \\|w^n - w^*\\|^4 + c_2^n(\\alpha_n) \\|w^n - w^*\\|^2 \n    + c_3^n(\\alpha_n).\n  \\end{align*}\n  Adding $\\|w^n - w^*\\|^4$ to both sides of the inequality, taking the \n  $\\E_{n-1}$-expectation and reinserting the bound, we obtain\n  \\begin{align*}\n    & \\E_n \\big[ \\|w^{n+1} - w^*\\|^4\\big] \\\\\n    &\\leq \\big(1 + c_1^n(\\alpha_n) \\big) \\E_{n-1} \\big[ \\|w^n - w^*\\|^4 \\big]\n    + c_2^n(\\alpha_n) M_2 + c_3^n(\\alpha_n)\\\\\n    &\\leq \\|w^1 - w^*\\|^4 \\prod_{i = 1}^{n} \\big(1 + c_1^i(\\alpha_i) \\big) \n    + \\sum_{i=1}^{n} \\big(c_2^i(\\alpha_i) M_2 + c_3^i(\\alpha_i)\\big) \\prod_{j = i+ \n    1}^{n} \\big(1 + c_1^j(\\alpha_j) \\big) \\\\\n    &\\leq \\|w^1 - w^*\\|^4 \\exptextB{\\sum_{i = 1}^{\\infty} c_1^i(\\alpha_i) } \n    + \\sum_{i=1}^{\\infty} \\big(c_2^i(\\alpha_i) M_2 + c_3^i(\\alpha_i)\\big) \n    \\exptextB{\\sum_{j = i+ 1}^{\\infty} c_1^j(\\alpha_j)} .\n  \\end{align*}\n  Finally, this is finite due to the assumption $\\sum_{n  =1}^{\\infty} \\alpha_n^2 < \\infty$.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:SecondLipT}\n  Let Assumption~\\ref{ass:fStoch} be fulfilled and let $z \\in H$, $\\alpha \n  \\in (0,\\infty)$ and $\\Xi \\in [0,\\infty)$ be given.\n  It then follows that\n  \\begin{align*}\n    \\Big\\| T_{\\alpha f_n, z}(w) - w + \\frac{\\alpha \n      \\iota \\nabla f_n(w)}{1 + \\alpha \\Xi} \\Big\\|\n    = \\frac{ \\alpha^2 \\big| \\Xi - \\|\\iota \\nabla f_n(z)\\|\\big| \\|\\iota \\nabla f_n(w)\\|}\n    {(1 + \\alpha \\|\\iota \\nabla f_n(z)\\| ) (1 + \\alpha \\Xi )}\n  \\end{align*}\n  for all $w \\in H$.\n\\end{lemma}\n\n\\begin{proof}\n  Inserting the definition of $T_{\\alpha f_n, z}$, it follows that\n  \\begin{align*}\n    &\\Big\\| T_{\\alpha f_n, z}(w) - w + \\frac{\\alpha \\iota \\nabla f_n(w)}{1 + \\alpha \n    \\Xi} \\Big\\|\n  = \\Big\\| \\frac{\\alpha \\iota \\nabla f_n(w)}{1 + \\alpha \\|\\iota \\nabla f_n(z)\\|} - \n    \\frac{\\alpha \\iota \\nabla f_n(w)}{1 + \\alpha \\Xi } \\Big\\|\\\\\n    &= \\Big\\| \\frac{\\alpha \\iota \\nabla f_n(w) + \\alpha^2 \\Xi \\iota \\nabla f_n(w) \n    }{(1 + \\alpha \\|\\iota \\nabla f_n(z)\\| ) (1 + \\alpha \\Xi )}\n    - \\frac{\\alpha \\iota \\nabla f_n(w) + \\alpha^2 \\|\\iota \\nabla f_n(z)\\| \\iota \\nabla \n    f_n(w) }{ (1 + \\alpha \\|\\iota \\nabla f_n(z)\\| ) (1 + \\alpha \\Xi ) } \\Big\\|\\\\\n    &= \\frac{ \\alpha^2 \\big| \\Xi - \\|\\iota \\nabla f_n(z)\\|\\big| \\|\\iota \\nabla f_n(w)\\|}\n    {(1 + \\alpha \\|\\iota \\nabla f_n(z)\\| ) (1 + \\alpha \\Xi )},\n  \\end{align*}\n  for all $w \\in H$, which proves the claim.\n\\end{proof}\n\n\\begin{lemma} \\label{lem:boundForTheo}\n  Let Assumption~\\ref{ass:fStoch} be fulfilled and let \n  $\\{\\alpha_n\\}_{n \n  \\in \\N}$  be a sequence of positive real numbers.\n   For any $\\Xi \\in [0,\\infty)$ it then follows that\n  \\begin{align*}\n    \\E_{\\xi_n} \\big[\\| w^{n+1} - w^* \\|^2\\big]\n    &\\leq \\Big( 1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \\nabla \n    f_n(w^n)\\| } \\Big] \\Big) \\| w^n - w^*\\|^2\\\\\n    &\\quad + 2 \\E_{\\xi_n} \\Big[ \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla \n    f_n(w^*)\\|^2}{(1 + \\alpha_n \\|\\iota \\nabla \n      f_n(w^n)\\| )^2} \\Big]\\\\\n    &\\quad + 2 \\E_{\\xi_n} \\Big[ \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n (w^*)\\|^2 }{( 1 \n    + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| )^2} \\Big]\\\\\n    &\\quad + 2\\E_{\\xi_n} \\Big[ \\frac{ \\alpha_n^2 \\big| \\Xi - \\|\\iota  \\nabla f_n(w^n)\\| \\big| \n    \\|\\iota \\nabla f_n(w^*)\\|}\n    {(1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| ) (1 + \\alpha_n \\Xi )} \\Big] \n    \\| w^n - w^* \\| \n  \\end{align*}\n  for every $n \\in \\N$.\n\\end{lemma}\n\n\\begin{proof}\n  From $w^{n+1} = T_{\\alpha_n f_n, w^n}(w^n)$ we obtain that\n  \\begin{align*}\n    \\| w^{n+1} - w^* \\|^2\n    &= \\| T_{\\alpha_n f_n, w^n}(w^n) - T_{\\alpha_n f_n, w^n}(w^*) + T_{\\alpha_n f_n, \n      w^n}(w^*) - w^*\\|^2\\\\\n    &= \\| T_{\\alpha_n f_n, w^n}(w^n) - T_{\\alpha_n f_n, w^n}(w^*)\\|^2 + \\|T_{\\alpha_n f_n, \n      w^n}(w^*) - w^*\\|^2\\\\\n    &\\quad + 2 \\dual{T_{\\alpha_n f_n, w^n}(w^n) - T_{\\alpha_n f_n, w^n}(w^*)}{T_{\\alpha_n \n        f_n, w^n}(w^*) - w^*}\\\\\n    &=: I_1 + I_2 + 2 I_3.\n  \\end{align*}\n  For $I_1$, we can write\n  \\begin{align*}\n    I_1&= \\| T_{\\alpha_n f_n, w^n}(w^n) - T_{\\alpha_n f_n, w^n}(w^*)\\|^2\\\\\n    &= \\big\\| w^n - w^* + (T_{\\alpha_n f_n, w^n} - I)(w^n) -\n    (T_{\\alpha_n f_n, w^n} - I)(w^*)\\big\\|^2\\\\\n    &= \\| w^n - w^*\\|^2\n    + 2 \\dualb{w^n - w^*}{(T_{\\alpha_n f_n, w^n} - I)(w^n) - (T_{\\alpha_n f_n, w^n} - I)(w^*) } \n    \\\\\n    &\\quad + \\| (T_{\\alpha_n f_n, w^n} - I)(w^n) - (T_{\\alpha_n f_n, w^n} - I)(w^*) \\|^2\n    =: I_{1,1} + 2I_{1,2} + I_{1,3}.\n  \\end{align*}\n  The term $I_{1,2}$ can be estimated by applying the Cauchy--Schwarz inequality:\n  \\begin{align*}\n    I_{1,2}&= \\dualb{w^n - w^*}{ (T_{\\alpha_n f_n, w^n} - I)(w^n) - (T_{\\alpha_n f_n, w^n} - \n      I)(w^*) }\\\\\n    &= - \\dualB{w^n - w^*}{\\frac{\\alpha_n \\iota \\nabla f_n(w^n)}{1 + \\alpha_n \\|\\iota \n    \\nabla f_n(w^n)\\|} \n      - \\frac{\\alpha_n \\iota \\nabla f_n(w^*)}{1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\|}}\\\\\n    &\\leq - \\frac{\\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| } \\|w^n - \n    w^*\\|^2.\n  \\end{align*}\n  For $I_{1,3}$, we insert the definition of $T_{\\alpha_n f_n, w^n}$ and find\n  \\begin{align*}\n    I_{1,3} \n    &= \\| (T_{\\alpha_n f_n, w^n} - I)(w^n) - (T_{\\alpha_n f_n, w^n} - I)(w^*) \\|^2\\\\\n    &=  \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla f_n(w^*)\\|^2 }{ (1 + \\alpha \n    \\|\\iota \\nabla f_n(w^n)\\| )^2}.\n  \\end{align*}\n  Thus, for $I_1$, we have\n  \\begin{align*}\n    I_1\n    \\leq \\Big( 1 - \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| } \n    \\Big)\\| w^n - w^*\\|^2\n    + \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla f_n(w^*)\\|^2 }{ (1 + \\alpha \n      \\|\\iota \\nabla f_n(w^n)\\| )^2}.\n  \\end{align*}\n  Further, $I_2$ can be written as\n  \\begin{align*}\n    I_2\n    = \\|T_{\\alpha_n f_n, w^n}(w^*) - w^*\\|^2\n    = \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n (w^*)\\|^2 }{( 1 + \\alpha_n \\|\\iota \\nabla \n      f_n(w^n)\\| )^2}.\n  \\end{align*}\n  Finally, $I_3$ can be rewritten as\n  \\begin{align*}\n    I_3 &= \\dual{T_{\\alpha_n f_n, w^n}(w^n) - T_{\\alpha_n f_n, w^n}(w^*)}{T_{\\alpha_n f_n, \n        w^n}(w^*) - w^*}\\\\\n    &= \\dual{(T_{\\alpha_n f_n, w^n} - I) (w^n) - (T_{\\alpha_n f_n, w^n} - I) \n      (w^*)}{(T_{\\alpha_n f_n, w^n} - I) (w^*)}\\\\\n    &\\quad + \\dual{w^n - w^*}{(T_{\\alpha_n f_n, w^n} - I) (w^*)}\n    =: I_{3,1} + I_{3,2}.\n  \\end{align*}\n  Then for $I_{3,1}$, we insert the definition of $T_{\\alpha_n f_n, w^n}$ and obtain\n  \\begin{align*}\n    I_{3,1} &= \\dual{(T_{\\alpha_n f_n, w^n} - I) (w^n) - (T_{\\alpha_n f_n, w^n} - I) \n      (w^*)}{(T_{\\alpha_n f_n, w^n} - I) (w^*)}\\\\\n    &\\leq \\frac{\\alpha_n \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla f_n(w^*)\\|}{1 + \\alpha_n \\|\\iota \n    \\nabla f_n(w^n)\\|} \\cdot\n    \\frac{\\alpha_n \\| \\iota \\nabla f_n (w^*)\\|}{1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\|} \\\\\n    &\\leq \\frac{1}{2}\\frac{\\alpha_n^2 \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla f_n(w^*)\\|^2}{ (1 + \n    \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| )^2} \n    + \\frac{1}{2} \\frac{\\alpha_n^2 \\| \\iota \\nabla f_n (w^*)\\|^2}{(1 + \\alpha_n \\|\\iota \n      \\nabla f_n(w^n)\\| )^2},\n  \\end{align*}\n  where we applied the Cauchy--Schwarz inequality and Young's inequality.\n  To estimate $I_{3,2}$, we add and subtract an additional summand, so that\n  \\begin{align*}\n    I_{3,2} &= \\dual{w^n - w^*}{(T_{\\alpha_n f_n, w^n} - I) (w^*)}\\\\\n    &= \\dualB{w^n - w^*}{(T_{\\alpha_n f_n, w^n} - I) (w^*) + \\frac{\\alpha_n \\iota \\nabla \n    f_n(w^*)}{1 + \\alpha_n \\Xi }}\n    - \\dualB{w^n - w^*}{\\frac{\\alpha_n \\iota \\nabla f_n(w^*)}{1 + \\alpha_n \\Xi }},\n  \\end{align*}\n  where $\\E_{\\xi_n}\\big[ \\dualb{w^n - w^*}{\\frac{\\alpha_n \\iota \\nabla f_n(w^*)}{1 \n  + \\alpha_n \\Xi }} \\big] = 0$ is fulfilled. Moreover, applying \n  Lemma~\\ref{lem:SecondLipT} and the Cauchy--Schwarz inequality, we obtain\n  \\begin{align*}\n    &\\dualB{w^n - w^*}{(T_{\\alpha_n f_n, w^n} - I) (w^*) + \\frac{\\alpha_n \\iota \\nabla \n        f_n(w^*)}{1 + \\alpha_n \\Xi }}\\\\\n    &\\leq \\frac{ \\alpha_n^2 \\big|\\Xi - \\|\\iota \\nabla f_n(w^n)\\| \\big| \\|\\iota \\nabla f_n(w^*)\\|}\n    {(1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| ) (1 + \\alpha_n \\Xi )}  \\| w^n \n    - w^* \\|.\n  \\end{align*}\n  Thus, the expectation of $I_3 = I_{3,1} + I_{3,2}$ can be bounded by\n  \\begin{align*}\n    \\E_{\\xi_n} [I_3 ]\n    &\\leq \\frac{1}{2} \\E_{\\xi_n} \\Big[ \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla \n    f_n(w^*)\\|^2 }{\\big(1 + \\alpha_n \\|\\iota \\nabla \n      f_n(w^n)\\|\\big)^2} + \\frac{\\alpha_n^2 \\| \\iota \\nabla f_n \n      (w^*)\\|^2}{ (1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| )^2}\\Big]\\\\\n    &\\quad + \\E_{\\xi_n} \\Big[ \\frac{ \\alpha_n^2 \\big| \\Xi - \\|\\iota \\nabla f_n(w^n)\\| \\big| \\|\\iota \n    \\nabla f_n(w^*)\\| }\n    {(1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| ) (1 + \\alpha_n \\Xi )}\\Big]  \\| w^n - w^* \\| .\n  \\end{align*}\n  Inserting the bounds for $I_1$, $I_2$ and $I_3$ into $\\E_{\\xi_n} \\big[\\| w^{n+1} - w^* \n  \\|^2\\big] = \\E_{\\xi_n} \\big[ I_1 + I_2 + 2 I_3\\big]$ finishes the proof.\n\\end{proof}\n\n\\begin{proof}[Proof of Theorem~\\ref{thm:ConvNorm1}]\n  The main idea of the proof is to apply the bound from Lemma~\\ref{lem:boundForTheo} \n  with $\\Xi = 0$ and bound the denominators of the last three \n  summands from below by one. We then get\n  \\begin{align*}\n    &\\E_{\\xi_n} \\big[\\| w^{n+1} - w^* \\|^2\\big]\\\\\n    &\\leq  \\Big( 1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \\nabla \n      f_n(w^n)\\| } \\Big] \\Big) \\| w^n - w^*\\|^2\n    + 2 \\alpha_n^2 L^2 \\| w^n - w^*\\|^2\n    + 2 \\alpha_n^2 \\sigma^2 \\\\\n    &\\quad + 2 \\alpha_n^2 \\big(\\E_{\\xi_n} \\big[ \\|\\iota \\nabla f_n(w^n)\\|^2 \n    \\big]\\big)^{\\frac{1}{2}} \\sigma \\| w^n - w^* \\| \\\\\n    %\n    &\\leq \\Big(1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n L_{\\xi_n}\n    \\|w^n - w^n\\| + \\alpha_n \\|\\iota \\nabla f_n(w^*)\\|} \\Big] \\Big) \\| w^n - w^*\\|^2\\\\\n    &\\quad + \\big(2 \\alpha_n^2 L^2 + 2 \\alpha_n^2 L \\sigma \\big)\\| w^n - w^*\\|^2 \n    + 2 \\alpha_n^2 \\sigma^2 \\| w^n - w^* \\| + 2 \\alpha_n^2 \\sigma^2 \\\\\n    %\n    &\\leq \\Big(1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \n      \\nabla f_n(w^*)\\|} + \\frac{2 \\alpha_n^2 L_{\\xi_n} \\mu_{\\xi_n} }{(1 + \\alpha_n \\|\\iota \n      \\nabla f_n(w^*)\\|)^2} \\| w^n - w^*\\| \\Big]\\Big) \\| w^n - w^*\\|^2\\\\\n    &\\quad + \\alpha_n^2 \\big(\\big(2 L^2 + 2 L \\sigma \\big) \\| w^n - w^*\\|^2 \n    + 2 \\sigma^2 \\| w^n - w^* \\| + 2 \\sigma^2\\big) \\\\  \n    %\n    &\\leq \\Big(1 - \\E_{\\xi_n} \\Big[  \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \n      \\nabla f_n(w^*)\\|} \\Big]\\Big) \\| w^n - w^*\\|^2\\\\\n    &\\quad +  \\alpha_n^2 \\big(2 L \\mu_2 \\| w^n - w^*\\|^3 + \\big(2 \n    L^2 + 2 L \\sigma \\big) \\| w^n - w^*\\|^2\n    + 2 \\sigma^2 \\| w^n - w^* \\| + 2 \\sigma^2 \\big),\n  \\end{align*}\n  where we added and subtracted $\\iota \\nabla f_n(w^*)$ and applied Minkowski's \n  inequality in the second step and Lemma~\\ref{lem:TaylorEx} in the third.\n  For the first summand of the previous inequality, we apply Lemma~\\ref{lem:TaylorEx} \n  once more and find\n  \\begin{align*}\n    & \\Big(1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \n      \\nabla f_n(w^*)\\|} \\Big]\\Big) \\| w^n - w^*\\|^2\\\\\n    &= \\Big(1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\vartheta \\mu_{\\xi_n} }{n + \\gamma + \\vartheta \\|\\iota \n    \\nabla f_n(w^*)\\|} \\Big]\\Big) \\| w^n - w^*\\|^2\\\\\n    &\\leq \\Big(1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\vartheta \\mu_{\\xi_n}}{n + \\gamma}\\Big]\n    + \\E_{\\xi_n} \\Big[ \\frac{2 \\vartheta \\mu_{\\xi_n}}{(n + \\gamma)^2} \\vartheta \\|\\iota \\nabla \n    f_n(w^*)\\| \\Big]\\Big)\n    \\| w^n - w^*\\|^2\\\\\n    &\\leq \\Big(1 - \\frac{2 \\vartheta \\mu }{n + \\gamma} \\Big) \\| w^n - w^*\\|^2\n    + 2 \\alpha_n^2 \\mu_2 \\sigma \\| w^n - w^*\\|^2.\n  \\end{align*}\n  Taking the $\\E_{n-1}$-expectation and applying the a priori bounds from \n  Lemma~\\ref{lem:apriori2} and Lemma~\\ref{lem:apriori4}, \n  we find that\n  \\begin{align*}\n    &\\E_n \\big[\\| w^{n+1} - w^* \\|^2\\big]\\\\\n    &\\leq \\Big(1 - \\frac{2 \\vartheta \\mu }{n + \\gamma} \\Big) \\E_{n-1} \\big[ \\| w^n - w^*\\|^2\\big]\n    + 2 \\alpha_n^2 L \\mu_2 \\E_{n-1} \\big[\\| w^n - w^*\\|^3 \\big]\\\\\n    &\\quad  + \\big(2 \\alpha_n^2 L^2 + 2 \\alpha_n^2 \n    L \\sigma + 2 \\alpha_n^2 \\mu_2 \\sigma \\big) \\E_{n-1}\\big[\\| w^n - \n    w^*\\|^2\\big]\\\\\n    &\\quad + 2 \\alpha_n^2 \\sigma^2 \\E_{n-1}\\big[ \\| w^n - w^* \\|\\big] + 2 \\alpha_n^2 \n    \\sigma^2\\\\\n    &\\leq \\Big(1 - \\frac{2 \\vartheta \\mu }{n + \\gamma} \\Big) \\E_{n-1} \\big[ \\| w^n - w^*\\|^2\\big]\n    + 2 \\alpha_n^2 L \\mu_2 M_4^{\\frac{3}{4}} \\\\\n    &\\quad + \\big(2 \\alpha_n^2 L^2 + 2 \\alpha_n^2 L \\sigma + 2 \\alpha_n^2 \\mu_2 \\sigma \n    \\big) M_2\n     + 2 \\alpha_n^2 \\sigma^2 M_2^{\\frac{1}{2}} + 2 \\alpha_n^2 \\sigma^2\\\\\n    &= \\Big(1 - \\frac{2 \\vartheta \\mu }{n + \\gamma} \\Big) \\E_{n-1} \\big[ \\| w^n - w^*\\|^2\\big]\n    + \\frac{K}{(n + \\gamma)^2}.\n  \\end{align*}\n  Reinserting the inequality $n-1$ times yields\n  \\begin{align*}\n    &\\E_n \\big[\\| w^{n+1} - w^* \\|^2\\big]\\\\\n    &\\leq \\| w_1 - w^*\\|^2\\prod_{i=1}^{n} \\Big( 1 - \\frac{2 \\vartheta \\mu}{\\gamma + i} \\Big)\n    + K \\sum_{i=1}^{n} \\frac{1}{(\\gamma+ i)^2} \\prod_{j=i+1}^{n} \\Big( 1 - \\frac{2 \\vartheta \n    \\mu}{\\gamma + j} \\Big).\n  \\end{align*}\n  Due to the assumption $\\vartheta \\in (0,\\frac{1 + \\gamma}{2 \\mu }]$, we can apply \n  Lemma~\\ref{lem:algebraic_inequalities} with $x=2 \\vartheta \\mu$ and \n  $y=\\gamma$ and obtain the claimed error bound.\n\\end{proof}\n\n\\begin{lemma} \\label{lem:algebraic_inequalities}\n  Let $x, y \\in (0,\\infty)$ and $n,m \\in \\N$ be given such that $\\frac{x}{1 + y} \\leq 1$. Then \n  the following inequalities \n  are satisfied:\n  \\begin{enumerate}[label=(\\roman*)] \n    \\item $\\prod_{i= m}^n \\big(1-\\frac{x}{i + y}\\big) \\le \n    \\big(\\frac{n+1+ y}{m+y}\\big)^{-x}$,\\\\\n    \\item \n    $\\sum_{i=1}^n \\frac{1}{(i+y)^2} \\prod_{j =i+1}^n \\big(1-\\frac{x}{j+y} \\big)\\\\\n    \\leq \\exptextb{\\frac{x}{1+y}}\n    \\begin{cases}\n      (n+1+y)^{-1}\\frac{1}{x-1}, &x \\in (1,\\infty),\\\\\n      (n+1+y)^{-1} \\big(1 + \\ln{(n+y)}\\big), &x = 1,\\\\\n      (n+1+y)^{-x} \\frac{(1+y)^{x-2}(x-2-y)}{x-1},\n      &x \\in [0,1).\n    \\end{cases}$\n  \\end{enumerate}\n\\end{lemma}\n\n\\begin{proof}\n  In this proof, we apply the following basic inequalities involving (generalized) \n  harmonic numbers\n  \\begin{align*}\n    \\sum_{i=m}^n (i+y)^{-1} &\\geq \\ln{(n+1+y)} - \\ln{(m+y)}, \n    \\quad m \\in \\{1,\\dots,n\\},\\\\\n    \\sum_{i=1}^n{ (i+y)^{p}}\n    &\\leq \n    \\begin{cases}\n      \\frac{(n+1+y)^{p+1}}{p+1}, &p \\in [0,\\infty) ,\\\\\n      \\frac{(n+y)^{p+1}}{p+1}, &p \\in (-1,0),\\\\\n      1 + \\ln{(n+y)}, &p = -1,\\\\\n      \\frac{(1+y)^{p}(p-y)}{p+1}, &p \\in (-\\infty,-1),\n    \\end{cases}\n  \\end{align*}\n  for $y \\in (0,\\infty)$.\n  These inequalities follow by treating the sums as a lower or upper Riemann \n  sums approximating the integral $\\int {(u+y)^p \\diff{u}}$ over the intervals $[0,n]$, $[1,n]$ \n  or $[0,n+1]$. \n  \n  Using the inequality $1 + u \\le \\exp{u}$ for $u \\in [-1, \\infty)$, it follows that $0\\leq \n  1-\\frac{x}{i + y} \\le \\exptext{-\\frac{x}{i+y}}$ is fulfilled for every $i \\in \\N$ since \n  $\\frac{x}{1+y} \\leq 1$. It then follows that\n  \\begin{align*}\n    \\prod_{i= m}^n \\Big(1- \\frac{x }{i + y}\\Big) \n    &\\leq \\exptextB{- x \\sum_{i=m}^{n} (i + y)^{-1}} \\\\\n    &\\leq \\exptextb{- x \\big(\\ln{(n+1+y)} - \\ln(m+y)\\big)} \\\\\n    &= \\exptextB{- x \\ln{\\Big(\\frac{n+1+y}{m+y} \\Big)} } \n    = \\Big(\\frac{n+1+y}{m+y} \\Big)^{-x}\n  \\end{align*}\n  from which the first claim follows directly. \n  For the second claim, we use the fact that $\\frac{i+1+y}{i+y} = 1 + \\frac{1}{i+y} \\leq  1 + \n  \\frac{1}{1+y}\n  \\leq \\exptext{\\frac{1}{1+y}}$ for all $i \\in \\N$ and find that\n  \\begin{align*}\n    &\\sum_{i=1}^n \\frac{1}{(i+y)^2} \\prod_{j = i+1}^n \\Big(1-\\frac{x}{j+y} \n    \\Big)\n    \\le \\sum_{i=1}^n { \\frac{1}{(i+y)^2} \\Big(\\frac{n+1+y}{i+1+y}\\Big)^{-x} } \\\\\n    &\\le (n+1+y)^{-x} \\sum_{i=1}^n{ \\Big(\\frac{i+1+y}{i+y}\\Big)^{x} (i+y)^{x-2} }\\\\\n    &\\le  \\exptextB{\\frac{x}{1+y}} (n+1+y)^{-x} \n    \\sum_{i=1}^n{ (i+y)^{x-2} }\\\\\n    &\\leq \\exptextB{\\frac{x}{1+y}}\n    \\begin{cases}\n      (n+1+y)^{-1}\\frac{1}{x-1}, &x \\in (1,\\infty) ,\\\\\n      (n+1+y)^{-1} \\big(1 + \\ln{(n+y)}\\big), &x = 1,\\\\\n      (n+1+y)^{-x}\\frac{(1+y)^{x-2}(x-2-y)}{x-1}, \n      &x \\in [0,1),\n    \\end{cases}\n  \\end{align*}\n  where we applied the basic inequalities from the beginning of the proof.\n\\end{proof}\n\n\\begin{lemma} \\label{lem:TaylorEx}\n  Given $a,b \\in (0,\\infty)$, $\\frac{-1}{a x + b} \\leq - \\frac{1}{b} + \\frac{a}{b^2} \n  x$ for every $x \\in (0,\\infty)$. \n\\end{lemma}\n\n\\begin{proof}\n  We consider the function $f \\colon [0,\\infty) \\to \\R$ with $f(x) = \\frac{-1}{a x + b}$. \n  Then the first and second derivative of $f$ are given by $f'(x) = \\frac{a}{(a x + b)^2}$ and \n  $f''(x) = \\frac{-2a^2}{(a x + b)^3}$. Using a first-order Taylor expansion of $f$ then shows \n  that\n  \\begin{align*}\n    f(x) = - \\frac{1}{b} + \\frac{a}{b^2} x - \\frac{a^2}{(a \\xi + b)^3} x^2\n    \\leq - \\frac{1}{b} + \\frac{a}{b^2} x,\n  \\end{align*}\n  where $\\xi \\in (0,x)$. \n\\end{proof}\n\n\\begin{lemma} \\label{lem:swap_diff_and_expectation}\n  Let Assumption~\\ref{ass:fStoch} be fulfilled.\n  Then $F = \\E_{\\xi}[f(\\xi, \\cdot)]$ is G\\^{a}teaux differentiable and its derivative is \n  given by\n  \\begin{align*}\n    \\dual{\\iota \\nabla F (v)}{w} = \\E_{\\xi}[\\dual{\\iota \\nabla f (\\xi, v)}{w}].\n  \\end{align*}\n\\end{lemma}\n\n\\begin{proof}\n  See e.g.~\\cite[Lemma 6]{RyuBoyd.2016}.\n\\end{proof}","units":[{"start":18964,"end":19772,"start_line":375,"end_line":392,"raw_tex":"\\begin{lemma} \\label{lem:Fconsequences}\n  Let Assumption~\\ref{ass:fStoch} be fulfilled. \n  Then $F$ is strongly convex with convexity constant $\\mu$ and $\\nabla F$ is Lipschitz \n  continuous with Lipschitz constant $L_F \\le L$, i.e.\\ for all $v,w \\in H$ it holds that\n  \\begin{align*}\n    \\| \\nabla F(v) - \\nabla F(w) \\|_{H^*} \n    &= \\| \\iota \\nabla F(v) - \\iota \\nabla F(w) \\| \n    \\leq L_F \\| v - w\\| \n    \\leq L \\| v - w\\| \\quad \\text{and} \\\\\n    F(v) &\\geq F(w) + \\dual{\\iota \\nabla F(w) }{ v - w }+ \\frac{\\mu}{2} \\|v - w\\|^2.\n  \\end{align*}\n  Further, the first inequality implies that\n  \\begin{align*}\n    F(v) \n    &\\leq F(w)+\\dual{\\iota \\nabla F(w) }{ v - w }+ \\frac{L}{2} \\|v - w\\|^2.\n  \\end{align*}\n  Finally, there exists a unique $w^* \\in H$ such that $F(w^*) = \\min_{w \\in H} F(w)$.\n\\end{lemma}","sha256_utf8":"9e8bc5ef547e4e4231b3353c3f7ca899883266c60a5ca01ce1aa7961e8b5aa40"},{"start":19774,"end":21002,"start_line":394,"end_line":414,"raw_tex":"\\begin{proof}\n  Using $\\nabla F(w) = \\E_{\\xi} \\big[ \\nabla f(\\xi, w)\\big]$, we obtain\n  \\begin{align*}\n    \\dual{\\iota \\nabla F(v) - \\iota \\nabla F(w)}{v - w} \\geq \\mu \\|v - w\\|^2 \\quad \n    \\text{for all } v,w \\in H.\n  \\end{align*}\n  Thus, the function $v\\mapsto \\nabla F(v) - \\mu \\iota^{-1} v$ is monotone. Applying \n  \\cite[Proposition~25.10]{ZeidlerIIB.1990}, it follows that $v \\mapsto F(v) - \\frac{\\mu}{2} \n  \\|v\\|^2$ is convex such that $F$ is strongly convex and the variational inequality stated \n  in the lemma is fulfilled. The Lipschitz continuity of $\\iota \\nabla F$ \n  similarly follows from the Lipschitz continuity of $\\iota \\nabla f(\\xi, \\cdot)$ by the \n  identification provided in Lemma~\\ref{lem:swap_diff_and_expectation}. The final \n  inequality follows by expanding $F$ in a zeroth-order Taylor expansion \n  around $w$ and using the Lipschitz continuity. See e.g.\\ \n  \\cite[Appendix~B]{BottouCurtisNocedal.2018} for more details.\n %\n  We note that $F$ is coercive since it is strongly convex. Combined with the G\\^{a}teaux \n  differentiability, this guarantees the existence of a unique global \n  minimum, see e.g.~\\cite[Theorem 25.D, Proposition~25.20 and \n  Corollary~25.15]{ZeidlerIIB.1990}.\n\\end{proof}","sha256_utf8":"10df7a67785a0db0140db083a1f9277c6c373a1c0afbd81be35e0b82d4bfe066"},{"start":23514,"end":24755,"start_line":458,"end_line":483,"raw_tex":"\\begin{lemma} \\label{lem:apriori2}\n  Let Assumption~\\ref{ass:fStoch} be fulfilled and let $\\{\\alpha_n\\}_{n \\in \\N}$ be a \n  sequence of positive real numbers such that $\\sum_{n =1}^{\\infty} \\alpha_n^2 < \n  \\infty$.\n  For $\\Phi \\in [0,\\infty)$ the a priori bound\n  \\begin{align*}\n    &\\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \\\\\n    &\\leq \\| w_1 - w^*\\|^2 \\exptextB{ \\sum_{i=1}^{\\infty} \\big(2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n      \\alpha_i^2 \\big\\} + 4 \\alpha_i^2 L^2 m_i\\big)} \\\\\n    &\\quad + \\sum_{i=1}^{\\infty} \\big(\\Phi^2 \\min \\big\\{ \\E_i \\big[ \\|\\iota \\nabla \n    f_i(w^i)\\|^{-2} \\big], \n    \\alpha_i^2 \\big\\} + 2 ( 1- m_i) + 2 \\min \\big\\{1, 2 \\alpha_i^2 \\sigma^2 \\big\\}\\big)\\\\\n    &\\qquad \\times \\exptextB{\\sum_{j=i+1}^{\\infty} \\big(2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n      \\alpha_j^2 \\big\\} + 4 \\alpha_j^2 L^2 m_j\\big)} =: M_2\n  \\end{align*}\n  is fulfilled, where $\\min\\big\\{\\Phi^{-2}, x \\big\\} = x$ for $\\Phi = 0$ and every $x \\in \\R$ \n  and \n  \\begin{align*}\n    m_i =\n    \\begin{cases}\n      1, & \\text{\\emph{if }} 2 \\alpha_i^2 L^2 \\E_{i-1} \\big[ \\| w^i - w^*\\|^2\\big] \\le 1, \\\\\n      0, & \\text{\\emph{otherwise}}.\n    \\end{cases}\n  \\end{align*}\n  Furthermore, there exists $n_0 \\in \\N$ such that $m_i = 1$ for all $i \\geq n_0$.\n\\end{lemma}","sha256_utf8":"8d753cbfdcdb4d4a652eb6d35be830e6c813624904aa49daed93acf78aa81930"},{"start":25525,"end":34293,"start_line":499,"end_line":665,"raw_tex":"\\begin{proof}[Proof of Lemma~\\ref{lem:apriori2}]\n  We test the scheme defined by~\\eqref{eq:stochTamedEuler}\n  with $w^n - w^*$, in order to obtain that\n  \\begin{align}\\label{eq:proofApriori1}\n    \\begin{split}\n      &\\dual{w^{n+1} - w^* - (w^n - w^*)}{w^n - w^*} \\\\\n      &\\quad + \\frac{\\alpha_n \\dual{\\iota \\nabla f_n(w^n) - \\iota \\nabla \n          f_n(w^*)}{w^n - w^*}}{1 \n        + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\|} \n      = - \\frac{\\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*}}{1 + \\alpha_n \\|\\iota \n        \\nabla f_n(w^n)\\|}.\n    \\end{split}\n  \\end{align}\n  Using the identity $\\dual{u-v}{u} = \\frac{1}{2} (\\|u\\|^2 - \\|v\\|^2 + \\|u-v\\|^2)$, $u,v \\in H$, the \n  first summand on the left-hand side can be written as\n  \\begin{align*}\n    &- \\dual{w^n - w^* - (w^{n+1} - w^*)}{w^n - w^*} \\\\\n    &= - \\frac{1}{2} \\big( \\|w^n - w^*\\|^2 - \\|w^{n+1} - w^*\\|^2 + \\|w^{n+1} - w^n\\|^2 \\big)\\\\\n    &= \\frac{1}{2} \\Big(\\|w^{n+1} - w^*\\|^2 - \\|w^n - w^*\\|^2 - \\frac{\\alpha_n^2 \\| \\iota \n      \\nabla  f_n(w^n)\\|^2 }{(1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\|)^2}\\Big), \n  \\end{align*}\n  where we inserted the scheme in the last step. \n  Thus, inserting the monotonicity condition for $f_n$ into \n  \\eqref{eq:proofApriori1} and multiplying the inequality with the factor two, it follows that\n  \\begin{align*}\n    &\\|w^{n+1} - w^*\\|^2 - \\|w^n - w^*\\|^2 \\\\\n    &\\leq - \\frac{2\\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*}}{1 + \\alpha_n \n      \\|\\iota \\nabla  \n      f_n(w^n)\\|} \n    + \\frac{\\alpha_n^2 \\| \\iota \\nabla f_n(w^n)\\|^2 }{(1 + \\alpha_n \\|\\iota \\nabla  \n      f_n(w^n)\\|)^2} \n    =: I_1 + I_2.\n  \\end{align*}\n  Since the tamed Euler scheme is the forward Euler scheme with a second order \n  perturbation, it follows that\n  \\begin{align*}\n    \\frac{\\alpha_n \\iota \\nabla f_n(w^*)}{1 + \\alpha_n \\|\\iota \\nabla  f_n(w^n)\\|}\n    = \\frac{\\alpha_n \\iota \\nabla f_n(w^*)}{1 + \\alpha_n \\Phi}\n    + \\frac{ \\alpha_n^2 \\big(\\Phi - \\|\\iota \\nabla f_n(w^n)\\|\\big) \\iota \\nabla f_n(w^*) \n    }{(1 + \\alpha_n \\|\\iota \\nabla  f_n(w^n)\\|) (1 + \\alpha_n \\Phi )}\n  \\end{align*}\n  for $\\Phi \\in [0,\\infty)$. Note that we have $w^*$ in the numerator of the left-hand-side \n  but $w^n$ in the denominator. We insert this equality into $I_1$ and use the \n  Cauchy--Schwarz inequality and \n  Young's  inequality for products in order to obtain\n  \\begin{align*}\n    I_1 \n    &= - \\frac{\\alpha_n \\dual{2 \\iota \\nabla f_n(w^*)}{w^n - w^*}} {1 + \\alpha_n \\|\\iota \\nabla  \n      f_n(w^n)\\|} \\\\\n    &\\leq - \\frac{2 \\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*} }{1 + \\alpha_n \\Phi}\n    + \\frac{2 \\alpha_n^2 \\Phi \\| \\iota \\nabla f_n(w^*)\\| \\| w^n - w^*\\| }{(1 + \\alpha_n \\|\\iota \n    \\nabla  f_n(w^n)\\|) (1 + \\alpha_n \\Phi )} \\\\\n    &\\quad + \\frac{2 \\alpha_n^2 \\|\\iota \\nabla f_n(w^n)\\| \\| \\iota \\nabla f_n(w^*)\\| \\| w^n - \n    w^*\\| }{(1 + \\alpha_n \\|\\iota \\nabla  f_n(w^n)\\|) (1 + \\alpha_n \\Phi )} \\\\\n    &\\leq - \\frac{2 \\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*} }{1 + \\alpha_n \\Phi}\n    + \\frac{ \\alpha_n^2 \\big(\\Phi^2 + \\|\\iota \\nabla f_n(w^n)\\|^2\\big)}{(1 + \\alpha_n \\|\\iota \n    \\nabla  f_n(w^n)\\|)^2} \\\\\n    &\\quad + \\frac{2 \\alpha_n^2 \\| \\iota \\nabla f_n(w^*)\\|^2 \\| w^n - w^*\\|^2 }{(1 + \\alpha_n \n    \\Phi )^2}\n    =: I_{1,1} + I_{1,2} + I_{1,3}.\n  \\end{align*}\n  For $I_{1,1} = - \\frac{2 \\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*} }{1 + \\alpha_n \n  \\Phi}$, we notice that $\\E_{\\xi_n} [I_{1,1}] = 0$. Moreover, for $I_{1,2} = \\frac{\\alpha_n^2 \n  ( \n  \\Phi^2 + \\|\\iota \\nabla f_n(w^n)\\|^2 )}{(1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\|)^2}$, we get\n  \\begin{align*}\n    I_{1,2}\n    &\\leq \\Phi^2 \\min \\big\\{ \\|\\iota \\nabla f_n(w^n)\\|^{-2} , \\alpha_n^2 \\big\\}\n    + \\min \\big\\{1, \\alpha_n^2 \\|\\iota \\nabla f_n(w^n)\\|^2 \\big\\} \\\\\n    &\\leq \\Phi^2 \\min \\big\\{ \\|\\iota \\nabla f_n(w^n)\\|^{-2} , \\alpha_n^2 \\big\\}\n    + \\min \\big\\{1, 2 \\alpha_n^2 L_{\\xi_n}^2 \\|w^n - w^*\\|^2\\big\\}\\\\ \n    &\\quad + \\min \\big\\{1,2 \\alpha_n^2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\big\\}\n  \\end{align*}\n  and\n  \\begin{align*}\n    I_{1,3}\n    = \\frac{2\\alpha_n^2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\|w^n - w^*\\|^2}{(1 + \\alpha_n \\Phi )^2}\n    \\leq 2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\min \\big\\{\\Phi^{-2}, \\alpha_n^2 \\big\\} \\|w^n - w^*\\|^2.\n  \\end{align*}\n  A bound for $I_2$ is given by\n  \\begin{align*}\n    I_2 &= \\frac{\\alpha_n^2 \\| \\iota \\nabla f_n(w^n)\\|^2 }{(1 + \\alpha_n \\|\\iota \\nabla \n    f_n(w^n)\\| )^2}\n    \\leq \\min \\big\\{1, \\alpha_n^2 \\| \\iota \\nabla f_n(w^n)\\|^2 \\big\\}\\\\\n    &\\leq \\min \\big\\{1, 2 \\alpha_n^2 L_{\\xi_n}^2 \\| w^n - w^*\\|^2 \\big\\}\n    + \\min \\big\\{1, 2 \\alpha_n^2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\big\\}.\n  \\end{align*}  \n  Then it follows\n  \\begin{align}\n    \\label{eq:aprioriBoundMn}\n    \\begin{split}\n      &\\|w^{n+1} - w^*\\|^2 - \\|w^n - w^*\\|^2 \\leq  I_1 + I_2\\\\\n      &\\leq - \\frac{2 \\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*} }{1 + \\alpha_n \\Phi}\n      + \\Phi^2 \\min \\big\\{ \\|\\iota \\nabla f_n(w^n)\\|^{-2} , \\alpha_n^2 \\big\\}\\\\\n      &\\quad + 2 \\min \\big\\{1, 2 \\alpha_n^2 L_{\\xi_n}^2 \\|w^n - w^*\\|^2\\big\\} + 2 \\min \n      \\big\\{1,2 \\alpha_n^2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\big\\}\\\\\n      &\\quad + 2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\min \\big\\{\\Phi^{-2}, \\alpha_n^2 \\big\\} \\|w^n - \n      w^*\\|^2.\n    \\end{split}\n  \\end{align}\n  Taking the $\\E_n$-expectation, we then obtain\n  \\begin{equation}\n    \\label{eq:aprioriBoundMnExp}\n    \\begin{aligned}\n      &\\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \\\\\n      &\\quad \\leq \\big(1 + 2 \\sigma^2 \\min\\big\\{ \\Phi^{-2}, \\alpha_n^2 \\big\\} \\big) \\E_{n-1} \n      \\big[ \n      \\|w^n - w^*\\|^2 \\big]\\\\\n      &\\qquad + \\Phi^2 \\min \\big\\{ \\E_n \\big[\\|\\iota \\nabla f_n(w^n)\\|^{-2}\\big], \\alpha_n^2 \n      \\big\\}\\\\\n      &\\qquad + 2 \\min \\big\\{1, 2 \\alpha_n^2 L^2 \\E_{n-1} \\big[ \\|w^n - w^*\\|^2 \\big]\\big\\} \n      + 2 \\min \\big\\{1,2 \\alpha_n^2 \\sigma^2\\big\\}\\\\\n      &\\quad= \\big(1 + 2 \\min\\big\\{ \\Phi^{-2}, \\alpha_n^2 \\big\\} \\sigma^2+ 4 \\alpha_n^2 L^2 \n      m_n\\big) \\E_{n-1} \\big[ \\|w^n - w^*\\|^2 \\big]\\\\\n      &\\qquad + \\Phi^2 \\min \\big\\{ \\E_n \\big[\\|\\iota \\nabla f_n(w^n)\\|^{-2}\\big], \n      \\alpha_n^2 \\big\\}\n      + 2 (1-m_n) \n      + 2 \\min \\big\\{1,2 \\alpha_n^2 \\sigma^2\\big\\},\n    \\end{aligned}\n  \\end{equation}\n  with $m_n$ defined as in the lemma statement. Reinserting the bound \n  repeatedly thus yields\n  \\begin{align*}\n    &\\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \\\\\n    &\\leq \\| w_1 - w^*\\|^2 \\prod_{i=1}^{n} \\big(1 + 2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n    \\alpha_i^2 \\big\\} + 4 \\alpha_i^2 L^2 m_i\\big) \\\\\n    &\\quad + \\sum_{i=1}^{n} \\big(\\Phi^2 \\min \\big\\{ \\E_i \\big[ \\|\\iota \\nabla \n    f_i(w^i)\\|^{-2} \\big], \n    \\alpha_i^2 \\big\\} + 2 ( 1- m_i) + 2 \\min \\big\\{1, 2 \\alpha_i^2 \\sigma^2 \\big\\}\\big)\\\\\n    &\\qquad \\times \\prod_{j=i+1}^{n} \\big(1 + 2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n    \\alpha_j^2 \\big\\} + 4 \\alpha_j^2 L^2 m_j\\big).\n  \\end{align*}\n  Finally, we apply the inequality $1 + x \\leq \\exptext{x}$, $x \\in \\R$, and make \n  the bound independent of $n$ by bounding the final sums by the corresponding infinite \n  sums, in order to obtain\n  \\begin{align*}\n    &\\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \\\\\n    &\\leq \\| w_1 - w^*\\|^2 \\exptextB{ \\sum_{i=1}^{\\infty} \\big(2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n      \\alpha_i^2 \\big\\} + 4 \\alpha_i^2 L^2 m_i\\big)} \\\\\n    &\\quad + \\sum_{i=1}^{\\infty} \\big(\\Phi^2 \\min \\big\\{ \\E_i \\big[ \\|\\iota \\nabla \n    f_i(w^i)\\|^{-2} \\big], \n    \\alpha_i^2 \\big\\} + 2 ( 1- m_i) + 2 \\min \\big\\{1, 2 \\alpha_i^2 \\sigma^2 \\big\\}\\big)\\\\\n    &\\qquad \\times \\exptextB{\\sum_{j=i+1}^{\\infty} \\big(2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n      \\alpha_j^2 \\big\\} + 4 \\alpha_j^2 L^2 m_j\\big)}.\n  \\end{align*}\n  It remains to verify, that there exists $n_0 \\in \\N$ such that $m_n = 1$ for all $n \\geq \n  n_0$. This can be done by estimating \\eqref{eq:aprioriBoundMnExp} and following a \n  similar line of argumentation as before. First, we can write\n  \\begin{align*}\n    \\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \n    \\leq \\big(1 + 2 \\alpha_n^2 \\sigma^2 + 4 \\alpha_n^2 L^2 \\big) \\E_{n-1} \\big[ \\|w^n - \n    w^*\\|^2\\big] + \\Phi^2 \\alpha_n^2 + 4 \\alpha_n^2 \\sigma^2.\n  \\end{align*}\n  Reinserting the inequality $n -1$ times, it follows that\n  \\begin{align*}\n    &\\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \\\\\n    &\\leq \\| w_1 - w^*\\|^2 \\prod_{i=1}^{n} \\big(1 + 2 \\sigma^2 \\alpha_i^2 + 4 \\alpha_i^2 L^2 \n    \\big) \n    + \\sum_{i=1}^{n} 4 \\alpha_i^2 \\sigma^2 \\prod_{j=i+1}^{n} \\big(1 + 2 \\sigma^2 \n    \\alpha_j^2 + 4 \\alpha_j^2 L^2 \\big)\\\\\n    &\\leq \\| w_1 - w^*\\|^2 \\exptextB{ \\big(2 \\sigma^2 + 4 L^2 \\big) \\sum_{i=1}^{\\infty} \n    \\alpha_i^2 }\n    + \\sum_{i=1}^{\\infty} 4 \\alpha_i^2 \\sigma^2 \\exptextB{ \\big(2 \\sigma^2 + 4 L^2 \\big) \n    \\sum_{j=i+1}^{\\infty} \\alpha_j^2 }.\n  \\end{align*}\n  Since $\\sum_{n =1}^{\\infty} \\alpha_n^2 < \\infty$ there is thus a $n_0 \\in \\N$ such that $2 \\alpha_n^2 L^2 \\E_{n-1} \\big[ \\| w^n - w^*\\|^2\\big] \\leq 1$ for all $n \\geq n_0$.\n\\end{proof}","sha256_utf8":"244a2b9cb735081711b2f7ebc699e08ea5aa4251c52c7c01c1d74a123169d45a"},{"start":34296,"end":35166,"start_line":668,"end_line":683,"raw_tex":"\\begin{lemma} \\label{lem:apriori4}\n  Let Assumption~\\ref{ass:fHigherIntegrability} be fulfilled and \n  let $\\{\\alpha_n\\}_{n \\in \\N}$ be a sequence of positive real numbers such that $\\sum_{n \n  =1}^{\\infty} \\alpha_n^2 < \\infty$.\n  Then the a priori bound  \n  \\begin{align*}\n    \\E_n \\big[\\|w^{n+1} - w^*\\|^4\\big] \n    &\\leq \\|w^1 - w^*\\|^4 \\exptextB{\\sum_{i = 1}^{\\infty} c_1^i(\\alpha_i)} \\\\\n    &\\quad + \\sum_{i=1}^{\\infty} \\big(c_2^i(\\alpha_i) M_2 + c_3^i(\\alpha_i)\\big) \n    \\exptextB{\\sum_{j = i+ 1}^{\\infty} c_1^j(\\alpha_j) } =: M_4\n  \\end{align*}\n  is fulfilled for $c_1^i, c_2^i, c_3^i \\colon (0,\\infty) \\to (0,\\infty)$ such that there exist \n  $C_1^k, C_2^k, C_3^k, C_4^k \\in (0,\\infty)$ with $c_k^i(\\alpha) \\leq C_1^i \\min \\{C_2^i, \n  \\alpha^4\\} + C_3^i \\min \\{C_4^i, \\alpha^2\\}$ for all $\\alpha \\in (0,\\infty)$, $k \\in \\{1,2,3\\}$ \n  and $i \\in \\N$.\n\\end{lemma}","sha256_utf8":"3ae02ad76679e85e9ce6164f85487cf383a2520ea5a6e9b97aed47f92b2e47d1"},{"start":35169,"end":39382,"start_line":686,"end_line":764,"raw_tex":"\\begin{proof}\n  Within the proof of Lemma~\\ref{lem:apriori2}, we verified the inequality \n  \\eqref{eq:aprioriBoundMn}. Starting from this point, we find\n  \\begin{equation}\n    \\label{eq:apriori4}\n    \\|w^{n+1} - w^*\\|^2 - \\|w^n - w^*\\|^2 \\leq A_n + B_n,\n  \\end{equation}\n  where\n  \\begin{align*}\n    A_n &= -\\frac{2 \\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - \n    w^*} }{1 + \\alpha_n \\Phi} \\qquad \\text{and} \\\\\n    B_n &= \\Phi^2 \\min \\big\\{ \\|\\iota \\nabla f_n(w^n)\\|^{-2} , \\alpha_n^2 \\big\\}\n     + 2 \\min \\big\\{1, 2 \\alpha_n^2 L_{\\xi_n}^2 \\|w^n - w^*\\|^2\\big\\}\\\\\n      &\\quad + 2 \\min \\big\\{1,2 \\alpha_n^2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\big\\}\n      + 2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\min \\big\\{\\Phi^{-2}, \\alpha_n^2 \\big\\} \\|w^n - w^*\\|^2 \n  \\end{align*}\n  for $\\Phi \\in [0,\\infty)$. We note that the parameter $\\Phi$ can be chosen such that \n  $\\E_n[B_n]$ is as small as possible.\n  From this it follows that \n  \\begin{align*}\n    \\E_{\\xi_n} [A_n] &= 0,\\\\\n    \\E_{\\xi_n} [B_n] &\\leq  \\Phi^2 \\min \\big\\{ \\E_{\\xi_n} \\big[ \\|\\iota \\nabla f_n(w^n)\\|^{-2}\\big] \n    , \n    \\alpha_n^2 \\big\\}\n    + 2 \\min \\big\\{1, 2 \\alpha_n^2 L^2 \\|w^n - w^*\\|^2\\big\\}\\\\\n    &\\quad + 2 \\min \\big\\{1,2 \\alpha_n^2 \\sigma^2 \\big\\}\n    + 2 \\sigma^2 \\min \\big\\{\\Phi^{-2}, \\alpha_n^2 \\big\\} \\|w^n - w^*\\|^2, \\\\\n    \\E_{\\xi_n} [A_n^2] \n    &\\leq 4 \\sigma^2 \\min\\{ \\Phi^{-2}, \\alpha_n^2 \\} \\| w^n - w^*\\|^2 \\qquad \\text{and}\\\\\n    \\E_{\\xi_n} [B_n^2] \n    &\\leq 4 \\Phi^4 \\min \\big\\{ \\big(\\E_{\\xi_n} \\big[ \\|\\iota \\nabla f_n(w^n)\\|^{-2}\\big]\\big)^2 , \n    \\alpha_n^4 \\big\\}\n    + 16 \\min \\big\\{1, 4 \\alpha_n^4 L_4^4 \\|w^n - w^*\\|^4\\big\\}\\\\\n    &\\quad + 16 \\min \\big\\{1,4 \\alpha_n^4 \\sigma_4^4 \\big\\}\n    + 16  \\sigma_4^4 \\min \\big\\{\\Phi^{-4}, \\alpha_n^4 \\big\\} \\|w^n - w^*\\|^4.\n  \\end{align*}\n   We note that for $a,b \\in \\R$ we have the identity $(a - b)a = \\frac{1}{2} \\big( |a|^2 - |b|^2 \n   + |a - b|^2 \\big)$, and thus $|a|^2 - |b|^2 \\le 2(a-b)a$. By multiplying the inequality from \n   \\eqref{eq:apriori4} with the factor $2\\|w^{n+1} - w^*\\|^2$, we therefore obtain\n  \\begin{align*}\n    \\|w^{n+1} - w^*\\|^4 - \\|w^n - w^*\\|^4\n    &\\leq 2\\big( A_n + B_n \\big) \\|w^{n+1} - w^*\\|^2\\\\\n    &\\leq 2\\big( A_n + B_n \\big) \\big( \\|w^n - w^*\\|^2 + A_n + B_n  \\big)\\\\\n    &\\leq 2(A_n + B_n) \\|w^n - w^*\\|^2  + 4 A_n^2 + 4 B_n^2 \n  \\end{align*}\n  and in $\\E_{\\xi_n}$-expectation\n  \\begin{align*}\n    & \\E_{\\xi_n} \\big[ \\|w^{n+1} - w^*\\|^4\\big] - \\|w^n - w^*\\|^4\\\\\n    &\\leq 4 \\Big(2 L^2 \\min \\big\\{\\tfrac{1}{2} L^{-2} \\|w^n - w^*\\|^{-2}, \\alpha_n^2 \\big\\}\n    + \\sigma^2 \\min \\big\\{\\Phi^{-2}, \\alpha_n^2 \\big\\}\\\\\n    &\\qquad + 64 L_4^4 \\min \\big\\{ \\tfrac{1}{4} L_4^{-4} \\|w^n - w^*\\|^{-4}, \\alpha_n^4 \n    \\big\\} + 16 \\sigma_4^4 \\min \\big\\{\\Phi^{-4}, \\alpha_n^4 \\big\\} \\Big)\\|w^n - w^*\\|^4 \\\\\n    &\\quad + 2 \\Big( \\Phi^2 \\min \\big\\{ \\E_{\\xi_n} \\big[ \\|\\iota \\nabla f_n(w^n)\\|^{-2}\\big] , \n    \\alpha_n^2 \\big\\} \n    + 4 \\sigma^2\\min \\big\\{\\tfrac{1}{2} \\sigma^{-2} , \\alpha_n^2 \\big\\} \\\\\n    &\\qquad + 8  \\sigma^2 \\min\\{ \\Phi^{-2}, \\alpha_n^2 \\} \\Big) \\|w^n - w^*\\|^2 \\\\\n    &\\quad + 16 \\Phi^4 \\min \\big\\{ \\big(\\E_{\\xi_n} \\big[ \\|\\iota \\nabla \n    f_n(w^n)\\|^{-2}\\big]\\big)^2 \n    , \\alpha_n^4 \\big\\}\n    + 256 \\sigma_4^4 \\min \\big\\{ \\tfrac{1}{4}\\sigma_4^{-4},\\alpha_n^4 \\big\\}\\\\\n    %\n    &=: c_1^n (\\alpha_n) \\|w^n - w^*\\|^4 + c_2^n(\\alpha_n) \\|w^n - w^*\\|^2 \n    + c_3^n(\\alpha_n).\n  \\end{align*}\n  Adding $\\|w^n - w^*\\|^4$ to both sides of the inequality, taking the \n  $\\E_{n-1}$-expectation and reinserting the bound, we obtain\n  \\begin{align*}\n    & \\E_n \\big[ \\|w^{n+1} - w^*\\|^4\\big] \\\\\n    &\\leq \\big(1 + c_1^n(\\alpha_n) \\big) \\E_{n-1} \\big[ \\|w^n - w^*\\|^4 \\big]\n    + c_2^n(\\alpha_n) M_2 + c_3^n(\\alpha_n)\\\\\n    &\\leq \\|w^1 - w^*\\|^4 \\prod_{i = 1}^{n} \\big(1 + c_1^i(\\alpha_i) \\big) \n    + \\sum_{i=1}^{n} \\big(c_2^i(\\alpha_i) M_2 + c_3^i(\\alpha_i)\\big) \\prod_{j = i+ \n    1}^{n} \\big(1 + c_1^j(\\alpha_j) \\big) \\\\\n    &\\leq \\|w^1 - w^*\\|^4 \\exptextB{\\sum_{i = 1}^{\\infty} c_1^i(\\alpha_i) } \n    + \\sum_{i=1}^{\\infty} \\big(c_2^i(\\alpha_i) M_2 + c_3^i(\\alpha_i)\\big) \n    \\exptextB{\\sum_{j = i+ 1}^{\\infty} c_1^j(\\alpha_j)} .\n  \\end{align*}\n  Finally, this is finite due to the assumption $\\sum_{n  =1}^{\\infty} \\alpha_n^2 < \\infty$.\n\\end{proof}","sha256_utf8":"93e202518d4d4bb8f91cbc1c2dc38e50cfa6060369f21dedb1dcae6eb53aa66b"},{"start":41183,"end":41686,"start_line":811,"end_line":822,"raw_tex":"\\begin{lemma}\\label{lem:SecondLipT}\n  Let Assumption~\\ref{ass:fStoch} be fulfilled and let $z \\in H$, $\\alpha \n  \\in (0,\\infty)$ and $\\Xi \\in [0,\\infty)$ be given.\n  It then follows that\n  \\begin{align*}\n    \\Big\\| T_{\\alpha f_n, z}(w) - w + \\frac{\\alpha \n      \\iota \\nabla f_n(w)}{1 + \\alpha \\Xi} \\Big\\|\n    = \\frac{ \\alpha^2 \\big| \\Xi - \\|\\iota \\nabla f_n(z)\\|\\big| \\|\\iota \\nabla f_n(w)\\|}\n    {(1 + \\alpha \\|\\iota \\nabla f_n(z)\\| ) (1 + \\alpha \\Xi )}\n  \\end{align*}\n  for all $w \\in H$.\n\\end{lemma}","sha256_utf8":"93071b9583353288a35373b78c00e750f714f989064c78f63ac9bab1e4f3bdaf"},{"start":41688,"end":42573,"start_line":824,"end_line":839,"raw_tex":"\\begin{proof}\n  Inserting the definition of $T_{\\alpha f_n, z}$, it follows that\n  \\begin{align*}\n    &\\Big\\| T_{\\alpha f_n, z}(w) - w + \\frac{\\alpha \\iota \\nabla f_n(w)}{1 + \\alpha \n    \\Xi} \\Big\\|\n  = \\Big\\| \\frac{\\alpha \\iota \\nabla f_n(w)}{1 + \\alpha \\|\\iota \\nabla f_n(z)\\|} - \n    \\frac{\\alpha \\iota \\nabla f_n(w)}{1 + \\alpha \\Xi } \\Big\\|\\\\\n    &= \\Big\\| \\frac{\\alpha \\iota \\nabla f_n(w) + \\alpha^2 \\Xi \\iota \\nabla f_n(w) \n    }{(1 + \\alpha \\|\\iota \\nabla f_n(z)\\| ) (1 + \\alpha \\Xi )}\n    - \\frac{\\alpha \\iota \\nabla f_n(w) + \\alpha^2 \\|\\iota \\nabla f_n(z)\\| \\iota \\nabla \n    f_n(w) }{ (1 + \\alpha \\|\\iota \\nabla f_n(z)\\| ) (1 + \\alpha \\Xi ) } \\Big\\|\\\\\n    &= \\frac{ \\alpha^2 \\big| \\Xi - \\|\\iota \\nabla f_n(z)\\|\\big| \\|\\iota \\nabla f_n(w)\\|}\n    {(1 + \\alpha \\|\\iota \\nabla f_n(z)\\| ) (1 + \\alpha \\Xi )},\n  \\end{align*}\n  for all $w \\in H$, which proves the claim.\n\\end{proof}","sha256_utf8":"a80fe403c46a6cba885223c1d035d021bc33783e3c4d0bf320a339c662626897"},{"start":42576,"end":43583,"start_line":842,"end_line":862,"raw_tex":"\\begin{lemma} \\label{lem:boundForTheo}\n  Let Assumption~\\ref{ass:fStoch} be fulfilled and let \n  $\\{\\alpha_n\\}_{n \n  \\in \\N}$  be a sequence of positive real numbers.\n   For any $\\Xi \\in [0,\\infty)$ it then follows that\n  \\begin{align*}\n    \\E_{\\xi_n} \\big[\\| w^{n+1} - w^* \\|^2\\big]\n    &\\leq \\Big( 1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \\nabla \n    f_n(w^n)\\| } \\Big] \\Big) \\| w^n - w^*\\|^2\\\\\n    &\\quad + 2 \\E_{\\xi_n} \\Big[ \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla \n    f_n(w^*)\\|^2}{(1 + \\alpha_n \\|\\iota \\nabla \n      f_n(w^n)\\| )^2} \\Big]\\\\\n    &\\quad + 2 \\E_{\\xi_n} \\Big[ \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n (w^*)\\|^2 }{( 1 \n    + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| )^2} \\Big]\\\\\n    &\\quad + 2\\E_{\\xi_n} \\Big[ \\frac{ \\alpha_n^2 \\big| \\Xi - \\|\\iota  \\nabla f_n(w^n)\\| \\big| \n    \\|\\iota \\nabla f_n(w^*)\\|}\n    {(1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| ) (1 + \\alpha_n \\Xi )} \\Big] \n    \\| w^n - w^* \\| \n  \\end{align*}\n  for every $n \\in \\N$.\n\\end{lemma}","sha256_utf8":"5c49f1f9f1eaf094f8020f00e47b23f39bdb43f73b80b1bc1fd3b58afb62c3cf"},{"start":43586,"end":48980,"start_line":865,"end_line":972,"raw_tex":"\\begin{proof}\n  From $w^{n+1} = T_{\\alpha_n f_n, w^n}(w^n)$ we obtain that\n  \\begin{align*}\n    \\| w^{n+1} - w^* \\|^2\n    &= \\| T_{\\alpha_n f_n, w^n}(w^n) - T_{\\alpha_n f_n, w^n}(w^*) + T_{\\alpha_n f_n, \n      w^n}(w^*) - w^*\\|^2\\\\\n    &= \\| T_{\\alpha_n f_n, w^n}(w^n) - T_{\\alpha_n f_n, w^n}(w^*)\\|^2 + \\|T_{\\alpha_n f_n, \n      w^n}(w^*) - w^*\\|^2\\\\\n    &\\quad + 2 \\dual{T_{\\alpha_n f_n, w^n}(w^n) - T_{\\alpha_n f_n, w^n}(w^*)}{T_{\\alpha_n \n        f_n, w^n}(w^*) - w^*}\\\\\n    &=: I_1 + I_2 + 2 I_3.\n  \\end{align*}\n  For $I_1$, we can write\n  \\begin{align*}\n    I_1&= \\| T_{\\alpha_n f_n, w^n}(w^n) - T_{\\alpha_n f_n, w^n}(w^*)\\|^2\\\\\n    &= \\big\\| w^n - w^* + (T_{\\alpha_n f_n, w^n} - I)(w^n) -\n    (T_{\\alpha_n f_n, w^n} - I)(w^*)\\big\\|^2\\\\\n    &= \\| w^n - w^*\\|^2\n    + 2 \\dualb{w^n - w^*}{(T_{\\alpha_n f_n, w^n} - I)(w^n) - (T_{\\alpha_n f_n, w^n} - I)(w^*) } \n    \\\\\n    &\\quad + \\| (T_{\\alpha_n f_n, w^n} - I)(w^n) - (T_{\\alpha_n f_n, w^n} - I)(w^*) \\|^2\n    =: I_{1,1} + 2I_{1,2} + I_{1,3}.\n  \\end{align*}\n  The term $I_{1,2}$ can be estimated by applying the Cauchy--Schwarz inequality:\n  \\begin{align*}\n    I_{1,2}&= \\dualb{w^n - w^*}{ (T_{\\alpha_n f_n, w^n} - I)(w^n) - (T_{\\alpha_n f_n, w^n} - \n      I)(w^*) }\\\\\n    &= - \\dualB{w^n - w^*}{\\frac{\\alpha_n \\iota \\nabla f_n(w^n)}{1 + \\alpha_n \\|\\iota \n    \\nabla f_n(w^n)\\|} \n      - \\frac{\\alpha_n \\iota \\nabla f_n(w^*)}{1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\|}}\\\\\n    &\\leq - \\frac{\\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| } \\|w^n - \n    w^*\\|^2.\n  \\end{align*}\n  For $I_{1,3}$, we insert the definition of $T_{\\alpha_n f_n, w^n}$ and find\n  \\begin{align*}\n    I_{1,3} \n    &= \\| (T_{\\alpha_n f_n, w^n} - I)(w^n) - (T_{\\alpha_n f_n, w^n} - I)(w^*) \\|^2\\\\\n    &=  \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla f_n(w^*)\\|^2 }{ (1 + \\alpha \n    \\|\\iota \\nabla f_n(w^n)\\| )^2}.\n  \\end{align*}\n  Thus, for $I_1$, we have\n  \\begin{align*}\n    I_1\n    \\leq \\Big( 1 - \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| } \n    \\Big)\\| w^n - w^*\\|^2\n    + \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla f_n(w^*)\\|^2 }{ (1 + \\alpha \n      \\|\\iota \\nabla f_n(w^n)\\| )^2}.\n  \\end{align*}\n  Further, $I_2$ can be written as\n  \\begin{align*}\n    I_2\n    = \\|T_{\\alpha_n f_n, w^n}(w^*) - w^*\\|^2\n    = \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n (w^*)\\|^2 }{( 1 + \\alpha_n \\|\\iota \\nabla \n      f_n(w^n)\\| )^2}.\n  \\end{align*}\n  Finally, $I_3$ can be rewritten as\n  \\begin{align*}\n    I_3 &= \\dual{T_{\\alpha_n f_n, w^n}(w^n) - T_{\\alpha_n f_n, w^n}(w^*)}{T_{\\alpha_n f_n, \n        w^n}(w^*) - w^*}\\\\\n    &= \\dual{(T_{\\alpha_n f_n, w^n} - I) (w^n) - (T_{\\alpha_n f_n, w^n} - I) \n      (w^*)}{(T_{\\alpha_n f_n, w^n} - I) (w^*)}\\\\\n    &\\quad + \\dual{w^n - w^*}{(T_{\\alpha_n f_n, w^n} - I) (w^*)}\n    =: I_{3,1} + I_{3,2}.\n  \\end{align*}\n  Then for $I_{3,1}$, we insert the definition of $T_{\\alpha_n f_n, w^n}$ and obtain\n  \\begin{align*}\n    I_{3,1} &= \\dual{(T_{\\alpha_n f_n, w^n} - I) (w^n) - (T_{\\alpha_n f_n, w^n} - I) \n      (w^*)}{(T_{\\alpha_n f_n, w^n} - I) (w^*)}\\\\\n    &\\leq \\frac{\\alpha_n \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla f_n(w^*)\\|}{1 + \\alpha_n \\|\\iota \n    \\nabla f_n(w^n)\\|} \\cdot\n    \\frac{\\alpha_n \\| \\iota \\nabla f_n (w^*)\\|}{1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\|} \\\\\n    &\\leq \\frac{1}{2}\\frac{\\alpha_n^2 \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla f_n(w^*)\\|^2}{ (1 + \n    \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| )^2} \n    + \\frac{1}{2} \\frac{\\alpha_n^2 \\| \\iota \\nabla f_n (w^*)\\|^2}{(1 + \\alpha_n \\|\\iota \n      \\nabla f_n(w^n)\\| )^2},\n  \\end{align*}\n  where we applied the Cauchy--Schwarz inequality and Young's inequality.\n  To estimate $I_{3,2}$, we add and subtract an additional summand, so that\n  \\begin{align*}\n    I_{3,2} &= \\dual{w^n - w^*}{(T_{\\alpha_n f_n, w^n} - I) (w^*)}\\\\\n    &= \\dualB{w^n - w^*}{(T_{\\alpha_n f_n, w^n} - I) (w^*) + \\frac{\\alpha_n \\iota \\nabla \n    f_n(w^*)}{1 + \\alpha_n \\Xi }}\n    - \\dualB{w^n - w^*}{\\frac{\\alpha_n \\iota \\nabla f_n(w^*)}{1 + \\alpha_n \\Xi }},\n  \\end{align*}\n  where $\\E_{\\xi_n}\\big[ \\dualb{w^n - w^*}{\\frac{\\alpha_n \\iota \\nabla f_n(w^*)}{1 \n  + \\alpha_n \\Xi }} \\big] = 0$ is fulfilled. Moreover, applying \n  Lemma~\\ref{lem:SecondLipT} and the Cauchy--Schwarz inequality, we obtain\n  \\begin{align*}\n    &\\dualB{w^n - w^*}{(T_{\\alpha_n f_n, w^n} - I) (w^*) + \\frac{\\alpha_n \\iota \\nabla \n        f_n(w^*)}{1 + \\alpha_n \\Xi }}\\\\\n    &\\leq \\frac{ \\alpha_n^2 \\big|\\Xi - \\|\\iota \\nabla f_n(w^n)\\| \\big| \\|\\iota \\nabla f_n(w^*)\\|}\n    {(1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| ) (1 + \\alpha_n \\Xi )}  \\| w^n \n    - w^* \\|.\n  \\end{align*}\n  Thus, the expectation of $I_3 = I_{3,1} + I_{3,2}$ can be bounded by\n  \\begin{align*}\n    \\E_{\\xi_n} [I_3 ]\n    &\\leq \\frac{1}{2} \\E_{\\xi_n} \\Big[ \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla \n    f_n(w^*)\\|^2 }{\\big(1 + \\alpha_n \\|\\iota \\nabla \n      f_n(w^n)\\|\\big)^2} + \\frac{\\alpha_n^2 \\| \\iota \\nabla f_n \n      (w^*)\\|^2}{ (1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| )^2}\\Big]\\\\\n    &\\quad + \\E_{\\xi_n} \\Big[ \\frac{ \\alpha_n^2 \\big| \\Xi - \\|\\iota \\nabla f_n(w^n)\\| \\big| \\|\\iota \n    \\nabla f_n(w^*)\\| }\n    {(1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| ) (1 + \\alpha_n \\Xi )}\\Big]  \\| w^n - w^* \\| .\n  \\end{align*}\n  Inserting the bounds for $I_1$, $I_2$ and $I_3$ into $\\E_{\\xi_n} \\big[\\| w^{n+1} - w^* \n  \\|^2\\big] = \\E_{\\xi_n} \\big[ I_1 + I_2 + 2 I_3\\big]$ finishes the proof.\n\\end{proof}","sha256_utf8":"75a6e181c77c82a357b86fc8f21ce3aa78f67f438803940853d88c321dbd9cbe"},{"start":50928,"end":55223,"start_line":1014,"end_line":1090,"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm:ConvNorm1}]\n  The main idea of the proof is to apply the bound from Lemma~\\ref{lem:boundForTheo} \n  with $\\Xi = 0$ and bound the denominators of the last three \n  summands from below by one. We then get\n  \\begin{align*}\n    &\\E_{\\xi_n} \\big[\\| w^{n+1} - w^* \\|^2\\big]\\\\\n    &\\leq  \\Big( 1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \\nabla \n      f_n(w^n)\\| } \\Big] \\Big) \\| w^n - w^*\\|^2\n    + 2 \\alpha_n^2 L^2 \\| w^n - w^*\\|^2\n    + 2 \\alpha_n^2 \\sigma^2 \\\\\n    &\\quad + 2 \\alpha_n^2 \\big(\\E_{\\xi_n} \\big[ \\|\\iota \\nabla f_n(w^n)\\|^2 \n    \\big]\\big)^{\\frac{1}{2}} \\sigma \\| w^n - w^* \\| \\\\\n    %\n    &\\leq \\Big(1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n L_{\\xi_n}\n    \\|w^n - w^n\\| + \\alpha_n \\|\\iota \\nabla f_n(w^*)\\|} \\Big] \\Big) \\| w^n - w^*\\|^2\\\\\n    &\\quad + \\big(2 \\alpha_n^2 L^2 + 2 \\alpha_n^2 L \\sigma \\big)\\| w^n - w^*\\|^2 \n    + 2 \\alpha_n^2 \\sigma^2 \\| w^n - w^* \\| + 2 \\alpha_n^2 \\sigma^2 \\\\\n    %\n    &\\leq \\Big(1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \n      \\nabla f_n(w^*)\\|} + \\frac{2 \\alpha_n^2 L_{\\xi_n} \\mu_{\\xi_n} }{(1 + \\alpha_n \\|\\iota \n      \\nabla f_n(w^*)\\|)^2} \\| w^n - w^*\\| \\Big]\\Big) \\| w^n - w^*\\|^2\\\\\n    &\\quad + \\alpha_n^2 \\big(\\big(2 L^2 + 2 L \\sigma \\big) \\| w^n - w^*\\|^2 \n    + 2 \\sigma^2 \\| w^n - w^* \\| + 2 \\sigma^2\\big) \\\\  \n    %\n    &\\leq \\Big(1 - \\E_{\\xi_n} \\Big[  \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \n      \\nabla f_n(w^*)\\|} \\Big]\\Big) \\| w^n - w^*\\|^2\\\\\n    &\\quad +  \\alpha_n^2 \\big(2 L \\mu_2 \\| w^n - w^*\\|^3 + \\big(2 \n    L^2 + 2 L \\sigma \\big) \\| w^n - w^*\\|^2\n    + 2 \\sigma^2 \\| w^n - w^* \\| + 2 \\sigma^2 \\big),\n  \\end{align*}\n  where we added and subtracted $\\iota \\nabla f_n(w^*)$ and applied Minkowski's \n  inequality in the second step and Lemma~\\ref{lem:TaylorEx} in the third.\n  For the first summand of the previous inequality, we apply Lemma~\\ref{lem:TaylorEx} \n  once more and find\n  \\begin{align*}\n    & \\Big(1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \n      \\nabla f_n(w^*)\\|} \\Big]\\Big) \\| w^n - w^*\\|^2\\\\\n    &= \\Big(1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\vartheta \\mu_{\\xi_n} }{n + \\gamma + \\vartheta \\|\\iota \n    \\nabla f_n(w^*)\\|} \\Big]\\Big) \\| w^n - w^*\\|^2\\\\\n    &\\leq \\Big(1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\vartheta \\mu_{\\xi_n}}{n + \\gamma}\\Big]\n    + \\E_{\\xi_n} \\Big[ \\frac{2 \\vartheta \\mu_{\\xi_n}}{(n + \\gamma)^2} \\vartheta \\|\\iota \\nabla \n    f_n(w^*)\\| \\Big]\\Big)\n    \\| w^n - w^*\\|^2\\\\\n    &\\leq \\Big(1 - \\frac{2 \\vartheta \\mu }{n + \\gamma} \\Big) \\| w^n - w^*\\|^2\n    + 2 \\alpha_n^2 \\mu_2 \\sigma \\| w^n - w^*\\|^2.\n  \\end{align*}\n  Taking the $\\E_{n-1}$-expectation and applying the a priori bounds from \n  Lemma~\\ref{lem:apriori2} and Lemma~\\ref{lem:apriori4}, \n  we find that\n  \\begin{align*}\n    &\\E_n \\big[\\| w^{n+1} - w^* \\|^2\\big]\\\\\n    &\\leq \\Big(1 - \\frac{2 \\vartheta \\mu }{n + \\gamma} \\Big) \\E_{n-1} \\big[ \\| w^n - w^*\\|^2\\big]\n    + 2 \\alpha_n^2 L \\mu_2 \\E_{n-1} \\big[\\| w^n - w^*\\|^3 \\big]\\\\\n    &\\quad  + \\big(2 \\alpha_n^2 L^2 + 2 \\alpha_n^2 \n    L \\sigma + 2 \\alpha_n^2 \\mu_2 \\sigma \\big) \\E_{n-1}\\big[\\| w^n - \n    w^*\\|^2\\big]\\\\\n    &\\quad + 2 \\alpha_n^2 \\sigma^2 \\E_{n-1}\\big[ \\| w^n - w^* \\|\\big] + 2 \\alpha_n^2 \n    \\sigma^2\\\\\n    &\\leq \\Big(1 - \\frac{2 \\vartheta \\mu }{n + \\gamma} \\Big) \\E_{n-1} \\big[ \\| w^n - w^*\\|^2\\big]\n    + 2 \\alpha_n^2 L \\mu_2 M_4^{\\frac{3}{4}} \\\\\n    &\\quad + \\big(2 \\alpha_n^2 L^2 + 2 \\alpha_n^2 L \\sigma + 2 \\alpha_n^2 \\mu_2 \\sigma \n    \\big) M_2\n     + 2 \\alpha_n^2 \\sigma^2 M_2^{\\frac{1}{2}} + 2 \\alpha_n^2 \\sigma^2\\\\\n    &= \\Big(1 - \\frac{2 \\vartheta \\mu }{n + \\gamma} \\Big) \\E_{n-1} \\big[ \\| w^n - w^*\\|^2\\big]\n    + \\frac{K}{(n + \\gamma)^2}.\n  \\end{align*}\n  Reinserting the inequality $n-1$ times yields\n  \\begin{align*}\n    &\\E_n \\big[\\| w^{n+1} - w^* \\|^2\\big]\\\\\n    &\\leq \\| w_1 - w^*\\|^2\\prod_{i=1}^{n} \\Big( 1 - \\frac{2 \\vartheta \\mu}{\\gamma + i} \\Big)\n    + K \\sum_{i=1}^{n} \\frac{1}{(\\gamma+ i)^2} \\prod_{j=i+1}^{n} \\Big( 1 - \\frac{2 \\vartheta \n    \\mu}{\\gamma + j} \\Big).\n  \\end{align*}\n  Due to the assumption $\\vartheta \\in (0,\\frac{1 + \\gamma}{2 \\mu }]$, we can apply \n  Lemma~\\ref{lem:algebraic_inequalities} with $x=2 \\vartheta \\mu$ and \n  $y=\\gamma$ and obtain the claimed error bound.\n\\end{proof}","sha256_utf8":"dafde01eb31e268ff79db063ba68741b4cbcefb22e6fe041fefcf2c2eb34743c"},{"start":81325,"end":82027,"start_line":1567,"end_line":1584,"raw_tex":"\\begin{lemma} \\label{lem:algebraic_inequalities}\n  Let $x, y \\in (0,\\infty)$ and $n,m \\in \\N$ be given such that $\\frac{x}{1 + y} \\leq 1$. Then \n  the following inequalities \n  are satisfied:\n  \\begin{enumerate}[label=(\\roman*)] \n    \\item $\\prod_{i= m}^n \\big(1-\\frac{x}{i + y}\\big) \\le \n    \\big(\\frac{n+1+ y}{m+y}\\big)^{-x}$,\\\\\n    \\item \n    $\\sum_{i=1}^n \\frac{1}{(i+y)^2} \\prod_{j =i+1}^n \\big(1-\\frac{x}{j+y} \\big)\\\\\n    \\leq \\exptextb{\\frac{x}{1+y}}\n    \\begin{cases}\n      (n+1+y)^{-1}\\frac{1}{x-1}, &x \\in (1,\\infty),\\\\\n      (n+1+y)^{-1} \\big(1 + \\ln{(n+y)}\\big), &x = 1,\\\\\n      (n+1+y)^{-x} \\frac{(1+y)^{x-2}(x-2-y)}{x-1},\n      &x \\in [0,1).\n    \\end{cases}$\n  \\end{enumerate}\n\\end{lemma}","sha256_utf8":"88dd459a0888b1506a866808f02cd6a3fa0d01af1e8b9005165c0b7c938db303"},{"start":82029,"end":84214,"start_line":1586,"end_line":1636,"raw_tex":"\\begin{proof}\n  In this proof, we apply the following basic inequalities involving (generalized) \n  harmonic numbers\n  \\begin{align*}\n    \\sum_{i=m}^n (i+y)^{-1} &\\geq \\ln{(n+1+y)} - \\ln{(m+y)}, \n    \\quad m \\in \\{1,\\dots,n\\},\\\\\n    \\sum_{i=1}^n{ (i+y)^{p}}\n    &\\leq \n    \\begin{cases}\n      \\frac{(n+1+y)^{p+1}}{p+1}, &p \\in [0,\\infty) ,\\\\\n      \\frac{(n+y)^{p+1}}{p+1}, &p \\in (-1,0),\\\\\n      1 + \\ln{(n+y)}, &p = -1,\\\\\n      \\frac{(1+y)^{p}(p-y)}{p+1}, &p \\in (-\\infty,-1),\n    \\end{cases}\n  \\end{align*}\n  for $y \\in (0,\\infty)$.\n  These inequalities follow by treating the sums as a lower or upper Riemann \n  sums approximating the integral $\\int {(u+y)^p \\diff{u}}$ over the intervals $[0,n]$, $[1,n]$ \n  or $[0,n+1]$. \n  \n  Using the inequality $1 + u \\le \\exp{u}$ for $u \\in [-1, \\infty)$, it follows that $0\\leq \n  1-\\frac{x}{i + y} \\le \\exptext{-\\frac{x}{i+y}}$ is fulfilled for every $i \\in \\N$ since \n  $\\frac{x}{1+y} \\leq 1$. It then follows that\n  \\begin{align*}\n    \\prod_{i= m}^n \\Big(1- \\frac{x }{i + y}\\Big) \n    &\\leq \\exptextB{- x \\sum_{i=m}^{n} (i + y)^{-1}} \\\\\n    &\\leq \\exptextb{- x \\big(\\ln{(n+1+y)} - \\ln(m+y)\\big)} \\\\\n    &= \\exptextB{- x \\ln{\\Big(\\frac{n+1+y}{m+y} \\Big)} } \n    = \\Big(\\frac{n+1+y}{m+y} \\Big)^{-x}\n  \\end{align*}\n  from which the first claim follows directly. \n  For the second claim, we use the fact that $\\frac{i+1+y}{i+y} = 1 + \\frac{1}{i+y} \\leq  1 + \n  \\frac{1}{1+y}\n  \\leq \\exptext{\\frac{1}{1+y}}$ for all $i \\in \\N$ and find that\n  \\begin{align*}\n    &\\sum_{i=1}^n \\frac{1}{(i+y)^2} \\prod_{j = i+1}^n \\Big(1-\\frac{x}{j+y} \n    \\Big)\n    \\le \\sum_{i=1}^n { \\frac{1}{(i+y)^2} \\Big(\\frac{n+1+y}{i+1+y}\\Big)^{-x} } \\\\\n    &\\le (n+1+y)^{-x} \\sum_{i=1}^n{ \\Big(\\frac{i+1+y}{i+y}\\Big)^{x} (i+y)^{x-2} }\\\\\n    &\\le  \\exptextB{\\frac{x}{1+y}} (n+1+y)^{-x} \n    \\sum_{i=1}^n{ (i+y)^{x-2} }\\\\\n    &\\leq \\exptextB{\\frac{x}{1+y}}\n    \\begin{cases}\n      (n+1+y)^{-1}\\frac{1}{x-1}, &x \\in (1,\\infty) ,\\\\\n      (n+1+y)^{-1} \\big(1 + \\ln{(n+y)}\\big), &x = 1,\\\\\n      (n+1+y)^{-x}\\frac{(1+y)^{x-2}(x-2-y)}{x-1}, \n      &x \\in [0,1),\n    \\end{cases}\n  \\end{align*}\n  where we applied the basic inequalities from the beginning of the proof.\n\\end{proof}","sha256_utf8":"bd02aa6f01e43c5ff23b5724ae1c95ded415ed1e13e10700e3c0331a00306687"},{"start":84216,"end":84384,"start_line":1638,"end_line":1641,"raw_tex":"\\begin{lemma} \\label{lem:TaylorEx}\n  Given $a,b \\in (0,\\infty)$, $\\frac{-1}{a x + b} \\leq - \\frac{1}{b} + \\frac{a}{b^2} \n  x$ for every $x \\in (0,\\infty)$. \n\\end{lemma}","sha256_utf8":"f1a201e0354facc08495ac9a45e294acd20fff4c2a27491f7a781ce77f8f9468"},{"start":84386,"end":84876,"start_line":1643,"end_line":1653,"raw_tex":"\\begin{proof}\n  We consider the function $f \\colon [0,\\infty) \\to \\R$ with $f(x) = \\frac{-1}{a x + b}$. \n  Then the first and second derivative of $f$ are given by $f'(x) = \\frac{a}{(a x + b)^2}$ and \n  $f''(x) = \\frac{-2a^2}{(a x + b)^3}$. Using a first-order Taylor expansion of $f$ then shows \n  that\n  \\begin{align*}\n    f(x) = - \\frac{1}{b} + \\frac{a}{b^2} x - \\frac{a^2}{(a \\xi + b)^3} x^2\n    \\leq - \\frac{1}{b} + \\frac{a}{b^2} x,\n  \\end{align*}\n  where $\\xi \\in (0,x)$. \n\\end{proof}","sha256_utf8":"f19b51fd6a3ca1c7fff3cef37d9842b0e7706075fc43bd39db42951dac58d0c1"},{"start":84991,"end":85315,"start_line":1658,"end_line":1665,"raw_tex":"\\begin{lemma} \\label{lem:swap_diff_and_expectation}\n  Let Assumption~\\ref{ass:fStoch} be fulfilled.\n  Then $F = \\E_{\\xi}[f(\\xi, \\cdot)]$ is G\\^{a}teaux differentiable and its derivative is \n  given by\n  \\begin{align*}\n    \\dual{\\iota \\nabla F (v)}{w} = \\E_{\\xi}[\\dual{\\iota \\nabla f (\\xi, v)}{w}].\n  \\end{align*}\n\\end{lemma}","sha256_utf8":"81bbcfe4234aa907410e770c82389b03053e921ecb941f194621aae274ea92e6"},{"start":85316,"end":85382,"start_line":1666,"end_line":1668,"raw_tex":"\\begin{proof}\n  See e.g.~\\cite[Lemma 6]{RyuBoyd.2016}.\n\\end{proof}","sha256_utf8":"6b82bf242fb6043ceb73b5b4c1acb99665eebe1e8a776ce160b600554d359660"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"TSGD均方误差收敛率","source_units":[{"start":48984,"end":50105,"start_line":976,"end_line":998,"raw_tex":"\\begin{theorem}\\label{thm:ConvNorm1}\n  Let Assumption~\\ref{ass:fHigherIntegrability} be fulfilled.\n  For $\\alpha_n = \\frac{\\vartheta}{n + \\gamma}$, $n \\in \\N$, with $\\gamma \\in [0,\\infty)$, \n  $\\vartheta \\in (0,\\frac{1 + \\gamma}{2\\mu}]$ and\n  \\begin{align*}\n    K = 2 \\vartheta^2 L \\mu_2 M_4^{\\frac{3}{4}} \n    + \\big(2 \\vartheta^2 L^2 + 2 \\vartheta^2 L \\sigma + 2 \\vartheta^2 \\mu_2 \\sigma \n    \\big) M_2\n    + 2 \\vartheta^2 \\sigma^2 M_2^{\\frac{1}{2}} + 2 \\vartheta^2 \\sigma^2,\n  \\end{align*}\n  it follows that\n  \\begin{align*}\n    &\\E_n \\big[\\| w^{n+1} - w^* \\|^2\\big]\\\\\n    &\\leq \\| w_1 - w^*\\|^2 (1+\\gamma)^{2 \\vartheta \\mu}(n+1+\\gamma)^{-2 \\vartheta \\mu} \\\\\n    &\\quad + \\exptextB{\\frac{2 \\vartheta \\mu}{1+\\gamma}} K \\begin{cases}\n      (n+1+\\gamma)^{-1} \\frac{1}{2 \\vartheta \\mu-1}, &2 \\vartheta \\mu \\in (1,\\infty),\\\\\n      (n+1+\\gamma)^{-1} (1 + \\ln{(n + \\gamma)}), &2 \\vartheta \\mu = 1,\\\\\n      (n+1+\\gamma)^{-2 \\vartheta \\mu} \\frac{(1+\\gamma)^{2 \\vartheta \\mu-2}(2 \\vartheta \n      \\mu-2-\\gamma)}{2 \\vartheta \\mu-1}, &2 \\vartheta \\mu \\in [0,1),\n    \\end{cases}\n  \\end{align*}\n  for every $n \\in \\N$.\n\\end{theorem}","sha256_utf8":"f919a59b470e017b93c469d56f251e53f9ebb72cda3c51f294c0858577f0b450"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"收敛率定理完整证明","source_units":[{"start":50928,"end":55223,"start_line":1014,"end_line":1090,"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm:ConvNorm1}]\n  The main idea of the proof is to apply the bound from Lemma~\\ref{lem:boundForTheo} \n  with $\\Xi = 0$ and bound the denominators of the last three \n  summands from below by one. We then get\n  \\begin{align*}\n    &\\E_{\\xi_n} \\big[\\| w^{n+1} - w^* \\|^2\\big]\\\\\n    &\\leq  \\Big( 1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \\nabla \n      f_n(w^n)\\| } \\Big] \\Big) \\| w^n - w^*\\|^2\n    + 2 \\alpha_n^2 L^2 \\| w^n - w^*\\|^2\n    + 2 \\alpha_n^2 \\sigma^2 \\\\\n    &\\quad + 2 \\alpha_n^2 \\big(\\E_{\\xi_n} \\big[ \\|\\iota \\nabla f_n(w^n)\\|^2 \n    \\big]\\big)^{\\frac{1}{2}} \\sigma \\| w^n - w^* \\| \\\\\n    %\n    &\\leq \\Big(1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n L_{\\xi_n}\n    \\|w^n - w^n\\| + \\alpha_n \\|\\iota \\nabla f_n(w^*)\\|} \\Big] \\Big) \\| w^n - w^*\\|^2\\\\\n    &\\quad + \\big(2 \\alpha_n^2 L^2 + 2 \\alpha_n^2 L \\sigma \\big)\\| w^n - w^*\\|^2 \n    + 2 \\alpha_n^2 \\sigma^2 \\| w^n - w^* \\| + 2 \\alpha_n^2 \\sigma^2 \\\\\n    %\n    &\\leq \\Big(1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \n      \\nabla f_n(w^*)\\|} + \\frac{2 \\alpha_n^2 L_{\\xi_n} \\mu_{\\xi_n} }{(1 + \\alpha_n \\|\\iota \n      \\nabla f_n(w^*)\\|)^2} \\| w^n - w^*\\| \\Big]\\Big) \\| w^n - w^*\\|^2\\\\\n    &\\quad + \\alpha_n^2 \\big(\\big(2 L^2 + 2 L \\sigma \\big) \\| w^n - w^*\\|^2 \n    + 2 \\sigma^2 \\| w^n - w^* \\| + 2 \\sigma^2\\big) \\\\  \n    %\n    &\\leq \\Big(1 - \\E_{\\xi_n} \\Big[  \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \n      \\nabla f_n(w^*)\\|} \\Big]\\Big) \\| w^n - w^*\\|^2\\\\\n    &\\quad +  \\alpha_n^2 \\big(2 L \\mu_2 \\| w^n - w^*\\|^3 + \\big(2 \n    L^2 + 2 L \\sigma \\big) \\| w^n - w^*\\|^2\n    + 2 \\sigma^2 \\| w^n - w^* \\| + 2 \\sigma^2 \\big),\n  \\end{align*}\n  where we added and subtracted $\\iota \\nabla f_n(w^*)$ and applied Minkowski's \n  inequality in the second step and Lemma~\\ref{lem:TaylorEx} in the third.\n  For the first summand of the previous inequality, we apply Lemma~\\ref{lem:TaylorEx} \n  once more and find\n  \\begin{align*}\n    & \\Big(1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \n      \\nabla f_n(w^*)\\|} \\Big]\\Big) \\| w^n - w^*\\|^2\\\\\n    &= \\Big(1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\vartheta \\mu_{\\xi_n} }{n + \\gamma + \\vartheta \\|\\iota \n    \\nabla f_n(w^*)\\|} \\Big]\\Big) \\| w^n - w^*\\|^2\\\\\n    &\\leq \\Big(1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\vartheta \\mu_{\\xi_n}}{n + \\gamma}\\Big]\n    + \\E_{\\xi_n} \\Big[ \\frac{2 \\vartheta \\mu_{\\xi_n}}{(n + \\gamma)^2} \\vartheta \\|\\iota \\nabla \n    f_n(w^*)\\| \\Big]\\Big)\n    \\| w^n - w^*\\|^2\\\\\n    &\\leq \\Big(1 - \\frac{2 \\vartheta \\mu }{n + \\gamma} \\Big) \\| w^n - w^*\\|^2\n    + 2 \\alpha_n^2 \\mu_2 \\sigma \\| w^n - w^*\\|^2.\n  \\end{align*}\n  Taking the $\\E_{n-1}$-expectation and applying the a priori bounds from \n  Lemma~\\ref{lem:apriori2} and Lemma~\\ref{lem:apriori4}, \n  we find that\n  \\begin{align*}\n    &\\E_n \\big[\\| w^{n+1} - w^* \\|^2\\big]\\\\\n    &\\leq \\Big(1 - \\frac{2 \\vartheta \\mu }{n + \\gamma} \\Big) \\E_{n-1} \\big[ \\| w^n - w^*\\|^2\\big]\n    + 2 \\alpha_n^2 L \\mu_2 \\E_{n-1} \\big[\\| w^n - w^*\\|^3 \\big]\\\\\n    &\\quad  + \\big(2 \\alpha_n^2 L^2 + 2 \\alpha_n^2 \n    L \\sigma + 2 \\alpha_n^2 \\mu_2 \\sigma \\big) \\E_{n-1}\\big[\\| w^n - \n    w^*\\|^2\\big]\\\\\n    &\\quad + 2 \\alpha_n^2 \\sigma^2 \\E_{n-1}\\big[ \\| w^n - w^* \\|\\big] + 2 \\alpha_n^2 \n    \\sigma^2\\\\\n    &\\leq \\Big(1 - \\frac{2 \\vartheta \\mu }{n + \\gamma} \\Big) \\E_{n-1} \\big[ \\| w^n - w^*\\|^2\\big]\n    + 2 \\alpha_n^2 L \\mu_2 M_4^{\\frac{3}{4}} \\\\\n    &\\quad + \\big(2 \\alpha_n^2 L^2 + 2 \\alpha_n^2 L \\sigma + 2 \\alpha_n^2 \\mu_2 \\sigma \n    \\big) M_2\n     + 2 \\alpha_n^2 \\sigma^2 M_2^{\\frac{1}{2}} + 2 \\alpha_n^2 \\sigma^2\\\\\n    &= \\Big(1 - \\frac{2 \\vartheta \\mu }{n + \\gamma} \\Big) \\E_{n-1} \\big[ \\| w^n - w^*\\|^2\\big]\n    + \\frac{K}{(n + \\gamma)^2}.\n  \\end{align*}\n  Reinserting the inequality $n-1$ times yields\n  \\begin{align*}\n    &\\E_n \\big[\\| w^{n+1} - w^* \\|^2\\big]\\\\\n    &\\leq \\| w_1 - w^*\\|^2\\prod_{i=1}^{n} \\Big( 1 - \\frac{2 \\vartheta \\mu}{\\gamma + i} \\Big)\n    + K \\sum_{i=1}^{n} \\frac{1}{(\\gamma+ i)^2} \\prod_{j=i+1}^{n} \\Big( 1 - \\frac{2 \\vartheta \n    \\mu}{\\gamma + j} \\Big).\n  \\end{align*}\n  Due to the assumption $\\vartheta \\in (0,\\frac{1 + \\gamma}{2 \\mu }]$, we can apply \n  Lemma~\\ref{lem:algebraic_inequalities} with $x=2 \\vartheta \\mu$ and \n  $y=\\gamma$ and obtain the claimed error bound.\n\\end{proof}","sha256_utf8":"dafde01eb31e268ff79db063ba68741b4cbcefb22e6fe041fefcf2c2eb34743c"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"估计单步误差与三次项","source_units":[{"start":50928,"end":52803,"start_line":1014,"end_line":1045,"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm:ConvNorm1}]\n  The main idea of the proof is to apply the bound from Lemma~\\ref{lem:boundForTheo} \n  with $\\Xi = 0$ and bound the denominators of the last three \n  summands from below by one. We then get\n  \\begin{align*}\n    &\\E_{\\xi_n} \\big[\\| w^{n+1} - w^* \\|^2\\big]\\\\\n    &\\leq  \\Big( 1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \\nabla \n      f_n(w^n)\\| } \\Big] \\Big) \\| w^n - w^*\\|^2\n    + 2 \\alpha_n^2 L^2 \\| w^n - w^*\\|^2\n    + 2 \\alpha_n^2 \\sigma^2 \\\\\n    &\\quad + 2 \\alpha_n^2 \\big(\\E_{\\xi_n} \\big[ \\|\\iota \\nabla f_n(w^n)\\|^2 \n    \\big]\\big)^{\\frac{1}{2}} \\sigma \\| w^n - w^* \\| \\\\\n    %\n    &\\leq \\Big(1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n L_{\\xi_n}\n    \\|w^n - w^n\\| + \\alpha_n \\|\\iota \\nabla f_n(w^*)\\|} \\Big] \\Big) \\| w^n - w^*\\|^2\\\\\n    &\\quad + \\big(2 \\alpha_n^2 L^2 + 2 \\alpha_n^2 L \\sigma \\big)\\| w^n - w^*\\|^2 \n    + 2 \\alpha_n^2 \\sigma^2 \\| w^n - w^* \\| + 2 \\alpha_n^2 \\sigma^2 \\\\\n    %\n    &\\leq \\Big(1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \n      \\nabla f_n(w^*)\\|} + \\frac{2 \\alpha_n^2 L_{\\xi_n} \\mu_{\\xi_n} }{(1 + \\alpha_n \\|\\iota \n      \\nabla f_n(w^*)\\|)^2} \\| w^n - w^*\\| \\Big]\\Big) \\| w^n - w^*\\|^2\\\\\n    &\\quad + \\alpha_n^2 \\big(\\big(2 L^2 + 2 L \\sigma \\big) \\| w^n - w^*\\|^2 \n    + 2 \\sigma^2 \\| w^n - w^* \\| + 2 \\sigma^2\\big) \\\\  \n    %\n    &\\leq \\Big(1 - \\E_{\\xi_n} \\Big[  \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \n      \\nabla f_n(w^*)\\|} \\Big]\\Big) \\| w^n - w^*\\|^2\\\\\n    &\\quad +  \\alpha_n^2 \\big(2 L \\mu_2 \\| w^n - w^*\\|^3 + \\big(2 \n    L^2 + 2 L \\sigma \\big) \\| w^n - w^*\\|^2\n    + 2 \\sigma^2 \\| w^n - w^* \\| + 2 \\sigma^2 \\big),\n  \\end{align*}\n  where we added and subtracted $\\iota \\nabla f_n(w^*)$ and applied Minkowski's \n  inequality in the second step and Lemma~\\ref{lem:TaylorEx} in the third.\n","sha256_utf8":"64eaa0356aeb097afca03d2856c495ce7a5c3d6230e58157e6f62d0d46bea1c1"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"展开随机收缩系数","source_units":[{"start":52803,"end":53590,"start_line":1046,"end_line":1059,"raw_tex":"  For the first summand of the previous inequality, we apply Lemma~\\ref{lem:TaylorEx} \n  once more and find\n  \\begin{align*}\n    & \\Big(1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \n      \\nabla f_n(w^*)\\|} \\Big]\\Big) \\| w^n - w^*\\|^2\\\\\n    &= \\Big(1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\vartheta \\mu_{\\xi_n} }{n + \\gamma + \\vartheta \\|\\iota \n    \\nabla f_n(w^*)\\|} \\Big]\\Big) \\| w^n - w^*\\|^2\\\\\n    &\\leq \\Big(1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\vartheta \\mu_{\\xi_n}}{n + \\gamma}\\Big]\n    + \\E_{\\xi_n} \\Big[ \\frac{2 \\vartheta \\mu_{\\xi_n}}{(n + \\gamma)^2} \\vartheta \\|\\iota \\nabla \n    f_n(w^*)\\| \\Big]\\Big)\n    \\| w^n - w^*\\|^2\\\\\n    &\\leq \\Big(1 - \\frac{2 \\vartheta \\mu }{n + \\gamma} \\Big) \\| w^n - w^*\\|^2\n    + 2 \\alpha_n^2 \\mu_2 \\sigma \\| w^n - w^*\\|^2.\n  \\end{align*}\n","sha256_utf8":"79156425195968d7bc6478953a40f43b0a0ec86ed527c5e4ff34efccab3e7d31"}]},{"id":"S005","semantic_type":"proof_step","kind":"proof_step","title":"利用矩界得到误差递推","source_units":[{"start":53590,"end":54668,"start_line":1060,"end_line":1079,"raw_tex":"  Taking the $\\E_{n-1}$-expectation and applying the a priori bounds from \n  Lemma~\\ref{lem:apriori2} and Lemma~\\ref{lem:apriori4}, \n  we find that\n  \\begin{align*}\n    &\\E_n \\big[\\| w^{n+1} - w^* \\|^2\\big]\\\\\n    &\\leq \\Big(1 - \\frac{2 \\vartheta \\mu }{n + \\gamma} \\Big) \\E_{n-1} \\big[ \\| w^n - w^*\\|^2\\big]\n    + 2 \\alpha_n^2 L \\mu_2 \\E_{n-1} \\big[\\| w^n - w^*\\|^3 \\big]\\\\\n    &\\quad  + \\big(2 \\alpha_n^2 L^2 + 2 \\alpha_n^2 \n    L \\sigma + 2 \\alpha_n^2 \\mu_2 \\sigma \\big) \\E_{n-1}\\big[\\| w^n - \n    w^*\\|^2\\big]\\\\\n    &\\quad + 2 \\alpha_n^2 \\sigma^2 \\E_{n-1}\\big[ \\| w^n - w^* \\|\\big] + 2 \\alpha_n^2 \n    \\sigma^2\\\\\n    &\\leq \\Big(1 - \\frac{2 \\vartheta \\mu }{n + \\gamma} \\Big) \\E_{n-1} \\big[ \\| w^n - w^*\\|^2\\big]\n    + 2 \\alpha_n^2 L \\mu_2 M_4^{\\frac{3}{4}} \\\\\n    &\\quad + \\big(2 \\alpha_n^2 L^2 + 2 \\alpha_n^2 L \\sigma + 2 \\alpha_n^2 \\mu_2 \\sigma \n    \\big) M_2\n     + 2 \\alpha_n^2 \\sigma^2 M_2^{\\frac{1}{2}} + 2 \\alpha_n^2 \\sigma^2\\\\\n    &= \\Big(1 - \\frac{2 \\vartheta \\mu }{n + \\gamma} \\Big) \\E_{n-1} \\big[ \\| w^n - w^*\\|^2\\big]\n    + \\frac{K}{(n + \\gamma)^2}.\n  \\end{align*}\n","sha256_utf8":"9881c502886075c5472cb50117a154322cd00247b8dc652981001c33fc7efbbc"}]},{"id":"S006","semantic_type":"proof_step","kind":"proof_step","title":"迭代展开误差递推","source_units":[{"start":54668,"end":55007,"start_line":1080,"end_line":1086,"raw_tex":"  Reinserting the inequality $n-1$ times yields\n  \\begin{align*}\n    &\\E_n \\big[\\| w^{n+1} - w^* \\|^2\\big]\\\\\n    &\\leq \\| w_1 - w^*\\|^2\\prod_{i=1}^{n} \\Big( 1 - \\frac{2 \\vartheta \\mu}{\\gamma + i} \\Big)\n    + K \\sum_{i=1}^{n} \\frac{1}{(\\gamma+ i)^2} \\prod_{j=i+1}^{n} \\Big( 1 - \\frac{2 \\vartheta \n    \\mu}{\\gamma + j} \\Big).\n  \\end{align*}\n","sha256_utf8":"f8a3635b3861b958a79b63e5834d6965f2f3f0a363bc7c6c77186ae75bfb484c"}]},{"id":"S007","semantic_type":"proof_step","kind":"proof_step","title":"应用乘积求和界","source_units":[{"start":55007,"end":55223,"start_line":1087,"end_line":1090,"raw_tex":"  Due to the assumption $\\vartheta \\in (0,\\frac{1 + \\gamma}{2 \\mu }]$, we can apply \n  Lemma~\\ref{lem:algebraic_inequalities} with $x=2 \\vartheta \\mu$ and \n  $y=\\gamma$ and obtain the claimed error bound.\n\\end{proof}","sha256_utf8":"d380f6acc6fc528204f097de69cdd9c1eda9b3c32f68812d238bf6d07599b06a"}]},{"id":"S008","semantic_type":"notation","kind":"notation","title":"希尔伯特空间与对偶记号","source_units":[{"start":13691,"end":14103,"start_line":274,"end_line":280,"raw_tex":"In the following, let $(H, \\dual{\\cdot}{\\cdot}, \\|\\cdot \\|)$ be a real Hilbert space. Its \ndual space is denoted\nby $(H^*, \\dual[H^*]{\\cdot}{\\cdot}, \\| \\cdot \\|_{H^*} )$. \nSince $H$ is a Hilbert space, there exists an isometric isomorphism $\\iota \\colon \nH^* \\to H$ such that $\\iota^{-1} \\colon H \\to H^*$ with $\\iota^{-1}: v \\mapsto \n\\dual{v}{\\cdot}$. \nWe denote by $\\N$ the natural numbers, not including $0$.\n","sha256_utf8":"68d6363556b3480c7cea428d9d7761a94206715da483d4dc025aceaeac5c5ef9"}]},{"id":"S009","semantic_type":"notation","kind":"notation","title":"独立采样与迭代期望","source_units":[{"start":14104,"end":15024,"start_line":282,"end_line":293,"raw_tex":"Let $(\\Omega, \\F, \\P )$ be a complete probability space and let $\\{\\xi_n\\}_{n \\in \\N}$ be \na family of jointly independent random variables on $\\Omega$.\nFor a random variable $X \\colon \\Omega \\to H$, let $\\E_{\\xi}[X]$ denote the \nexpectation with respect to the probability distribution of $\\xi$. \nWe are mainly interested in the total expectation\n\\begin{equation*}\n  \\E_n \\big[ \\|X\\|^2 \\big] = \\E_{\\xi_1}\\big[ \\E_{\\xi_2}\\big[ \\cdots \n  \\E_{\\xi_n} \\big[ \\|X \\|^2 \\big] \\cdots \\big]\\big].\n\\end{equation*}\nSince the random variables $\\{\\xi_n\\}_{n \\in \\N}$ are jointly independent, this expectation \ncoincides with the expectation with respect to the joint probability distribution of $\\xi_1, \n\\ldots, \\xi_n$. We also note here that if one of the following statements does not involve an expectation but does contain a random variable, then it is assumed to hold almost surely (a.s.) even if this is not explicitly stated.\n","sha256_utf8":"3e4a0b8e48a1441dbfba1e9c8e7a754b73bc6485bd3e80242aa0ae76b6902221"}]},{"id":"S010","semantic_type":"problem_setting","kind":"problem_setting","title":"期望目标与最优解","source_units":[{"start":15025,"end":15441,"start_line":295,"end_line":304,"raw_tex":"For a random variable $\\xi$ on $\\Omega$, we consider the function $f(\\xi, \\cdot ) \\colon \n\\Omega \\times H \\to \\R$ such that\n\\begin{equation*}\n  F(w) = \\E_{\\xi} \\big[ f(\\xi, w)\\big],\n\\end{equation*}\nand aim to find\n\\begin{equation*}\n  w^* = \\argmin_{w} F(w).\n\\end{equation*}\nThe  existence of such a minimum will be guaranteed by a strong convexity assumption below. We note that this means that $\\nabla F(w^*) = 0$.\n","sha256_utf8":"ee51590004d308b6917c56976c93fd9672f61493cab39fc33da0993e5e33a82c"}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","title":"随机梯度基本假设","source_units":[{"start":15752,"end":16855,"start_line":312,"end_line":334,"raw_tex":"\\begin{assumption} \\label{ass:fStoch}\n  Let $f(\\xi, \\cdot ) \\colon \\Omega \\times H \\to \\R$ be given such that\n  \\begin{itemize}\n    \\item $\\dual{\\iota \\nabla f (v)}{w} = \\lim_{h \\to 0} \\frac{f(v + hw) - f(v)}{h}$ a.s. \n      for all $v,w \\in H$, i.e., $f$ is G\\^{a}teaux differentiable a.s.;\n    \\item there exists \n      $\\mu_{\\xi} \\colon \\Omega \\to [0,\\infty)$ with $\\E_{\\xi} [\\mu_{\\xi}] =: \\mu \\in (0,\\infty)$ \n      such that\n      \\begin{equation*}\n        \\dual{\\iota \\nabla f(\\xi,v) - \\iota \\nabla f(\\xi,w)}{v - w} \\geq \\mu_{\\xi} \\|v - w\\|^2 \\quad \n        \\text{a.s. for all } v,w \\in H;\n      \\end{equation*}\n    \\item there exists \n    $L_{\\xi} \\colon \\Omega \\to [0,\\infty)$ with $\\big(\\E_{\\xi}\\big[ L_{\\xi}^2 \n    \\big]\\big)^{\\frac{1}{2}} =: L \\in (0,\\infty)$ such that\n    \\begin{equation*}\n      \\| \\iota \\nabla f(\\xi, v) - \\iota \\nabla f(\\xi, w) \\| \\leq L_{\\xi} \\| v - w\\| \\quad \n      \\text{a.s. for all } v,w \\in H;\n    \\end{equation*}\n    \\item $\\big(\\E_{\\xi} \\big[\\| \\iota \\nabla f(\\xi,w^*) \\|^2 \\big]\\big)^{\\frac{1}{2}} =: \n    \\sigma \\in [0,\\infty)$.\n  \\end{itemize}\n\\end{assumption}","sha256_utf8":"0310756367ecf98e20fe33efad81d95e42d3e5e931625b826e8043b097479675"}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","title":"随机参数高阶矩假设","source_units":[{"start":17089,"end":17806,"start_line":341,"end_line":353,"raw_tex":"\\begin{assumption}\\label{ass:fHigherIntegrability}\n  Let $f$ be given such that Assumption~\\ref{ass:fStoch} is fulfilled. Further, assume that \n  for all $v,w \\in H$\n  \\begin{itemize}\n    \\item $\\dual{\\iota \\nabla f(\\xi,v) - \\iota \\nabla f(\\xi,w)}{v - w} \\geq \\mu_{\\xi} \\|v - w\\|^2$ \n    with $\\big(\\E_{\\xi} \\big[\\| \\mu_{\\xi}^2 \\big]\\big)^{\\frac{1}{2}} =: \n    \\mu_2 \\in (0,\\infty)$.\n    \\item $ \\| \\iota \\nabla f(\\xi, v) - \\iota \\nabla f(\\xi, w) \\| \\leq L_{\\xi} \\| v - w\\|$ with \n    $\\big(\\E_{\\xi}\\big[ L_{\\xi}^4 \\big]\\big)^{\\frac{1}{4}} =: L_4 \\in (0,\\infty)$;\n    \\item $\\big(\\E_{\\xi} \\big[\\| \\iota \\nabla f(\\xi,w^*) \\|^4 \\big]\\big)^{\\frac{1}{4}} =: \n    \\sigma_4 \\in [0,\\infty)$.\n  \\end{itemize}\n\\end{assumption}","sha256_utf8":"14974d412093126f74585e1cad76b986781cbb6d9b6bd8c2b489e9a946bbdb83"}]},{"id":"S013","semantic_type":"algorithm","kind":"algorithm","title":"TSGD迭代与初值","source_units":[{"start":21069,"end":21715,"start_line":417,"end_line":427,"raw_tex":"Throughout the paper, we will assume that $\\{\\xi_n\\}_{n \\in \\N}$ is a given a family of \njointly independent random variables and we will abbreviate $f_n(w) = f(\\xi_n,w)$ for $n \n\\in \\N$. The $\\xi_n$ typically correspond to what batches we choose in each iteration, \ni.e.\\ on which part of the data we evaluate the gradient.\nLet $\\{\\alpha_n\\}_{n \\in \\N}$ be a sequence of of positive real numbers.\nWe then consider the stochastic tamed Euler scheme\n\\begin{align}\\label{eq:stochTamedEuler}\n  w^{n+1} = w^n - \\frac{\\alpha_n \\iota \\nabla f_n(w^n)}{1 + \\alpha_n \\|\\iota \n  \\nabla f_n(w^n)\\|} \\quad \\text{for }n \\in \\N, \n  \\quad w^1 = w_1.\n\\end{align}","sha256_utf8":"3d2675efc09dc0ac713eed273b900263b87083f28bdfacd49e1bce8c17cb293c"},{"start":21716,"end":21970,"start_line":428,"end_line":430,"raw_tex":"Note that it is also possible to choose a random initial value $w^1$, and our convergence statements can be \nextended to this setting in a straightforward way. For simplicity, we restrict ourselves to a fixed initial \nvalue $w_1 \\in H$ in the following.\n","sha256_utf8":"53fa36d4d84c5cce8cc4c2d8b1d73f9a3524e4ec4cfed09bb7479ebacc691a15"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"冻结分母迭代映射","source_units":[{"start":40861,"end":41180,"start_line":801,"end_line":807,"raw_tex":"Given  $z \\in H$ and $\\alpha > 0$, we define $T_{\\alpha f_n, z}(w) \\colon \\Omega \\times \nH \\to H$ by\n\\begin{align*}\n  T_{\\alpha f_n, z}(w)\n  = w - \\frac{\\alpha \\iota \\nabla f_n(w)}{1 + \\alpha \\|\\iota \\nabla f_n(z)\\|}.\n\\end{align*}\nThis implies that the next iterate $w^{n+1}$ is given by $T_{\\alpha_n f_n, w^n}(w^n)$. \n","sha256_utf8":"8d0d03081d9b1dddba91ee6892d432593602fe36779cb7edfa903a77e5bd93aa"}]},{"id":"S015","semantic_type":"lemma","kind":"lemma","title":"目标函数性质与唯一极小点","source_units":[{"start":18964,"end":19772,"start_line":375,"end_line":392,"raw_tex":"\\begin{lemma} \\label{lem:Fconsequences}\n  Let Assumption~\\ref{ass:fStoch} be fulfilled. \n  Then $F$ is strongly convex with convexity constant $\\mu$ and $\\nabla F$ is Lipschitz \n  continuous with Lipschitz constant $L_F \\le L$, i.e.\\ for all $v,w \\in H$ it holds that\n  \\begin{align*}\n    \\| \\nabla F(v) - \\nabla F(w) \\|_{H^*} \n    &= \\| \\iota \\nabla F(v) - \\iota \\nabla F(w) \\| \n    \\leq L_F \\| v - w\\| \n    \\leq L \\| v - w\\| \\quad \\text{and} \\\\\n    F(v) &\\geq F(w) + \\dual{\\iota \\nabla F(w) }{ v - w }+ \\frac{\\mu}{2} \\|v - w\\|^2.\n  \\end{align*}\n  Further, the first inequality implies that\n  \\begin{align*}\n    F(v) \n    &\\leq F(w)+\\dual{\\iota \\nabla F(w) }{ v - w }+ \\frac{L}{2} \\|v - w\\|^2.\n  \\end{align*}\n  Finally, there exists a unique $w^* \\in H$ such that $F(w^*) = \\min_{w \\in H} F(w)$.\n\\end{lemma}","sha256_utf8":"9e8bc5ef547e4e4231b3353c3f7ca899883266c60a5ca01ce1aa7961e8b5aa40"}]},{"id":"S016","semantic_type":"proof","kind":"proof","title":"目标函数性质完整证明","source_units":[{"start":19774,"end":21002,"start_line":394,"end_line":414,"raw_tex":"\\begin{proof}\n  Using $\\nabla F(w) = \\E_{\\xi} \\big[ \\nabla f(\\xi, w)\\big]$, we obtain\n  \\begin{align*}\n    \\dual{\\iota \\nabla F(v) - \\iota \\nabla F(w)}{v - w} \\geq \\mu \\|v - w\\|^2 \\quad \n    \\text{for all } v,w \\in H.\n  \\end{align*}\n  Thus, the function $v\\mapsto \\nabla F(v) - \\mu \\iota^{-1} v$ is monotone. Applying \n  \\cite[Proposition~25.10]{ZeidlerIIB.1990}, it follows that $v \\mapsto F(v) - \\frac{\\mu}{2} \n  \\|v\\|^2$ is convex such that $F$ is strongly convex and the variational inequality stated \n  in the lemma is fulfilled. The Lipschitz continuity of $\\iota \\nabla F$ \n  similarly follows from the Lipschitz continuity of $\\iota \\nabla f(\\xi, \\cdot)$ by the \n  identification provided in Lemma~\\ref{lem:swap_diff_and_expectation}. The final \n  inequality follows by expanding $F$ in a zeroth-order Taylor expansion \n  around $w$ and using the Lipschitz continuity. See e.g.\\ \n  \\cite[Appendix~B]{BottouCurtisNocedal.2018} for more details.\n %\n  We note that $F$ is coercive since it is strongly convex. Combined with the G\\^{a}teaux \n  differentiability, this guarantees the existence of a unique global \n  minimum, see e.g.~\\cite[Theorem 25.D, Proposition~25.20 and \n  Corollary~25.15]{ZeidlerIIB.1990}.\n\\end{proof}","sha256_utf8":"10df7a67785a0db0140db083a1f9277c6c373a1c0afbd81be35e0b82d4bfe066"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"由平均单调性推出强凸性","source_units":[{"start":19774,"end":20355,"start_line":394,"end_line":403,"raw_tex":"\\begin{proof}\n  Using $\\nabla F(w) = \\E_{\\xi} \\big[ \\nabla f(\\xi, w)\\big]$, we obtain\n  \\begin{align*}\n    \\dual{\\iota \\nabla F(v) - \\iota \\nabla F(w)}{v - w} \\geq \\mu \\|v - w\\|^2 \\quad \n    \\text{for all } v,w \\in H.\n  \\end{align*}\n  Thus, the function $v\\mapsto \\nabla F(v) - \\mu \\iota^{-1} v$ is monotone. Applying \n  \\cite[Proposition~25.10]{ZeidlerIIB.1990}, it follows that $v \\mapsto F(v) - \\frac{\\mu}{2} \n  \\|v\\|^2$ is convex such that $F$ is strongly convex and the variational inequality stated \n  in the lemma is fulfilled. The Lipschitz continuity of $\\iota \\nabla F$ \n","sha256_utf8":"f340e1ac43f09bcbe352e126106745a41757428a1d99b88190b3f2f0a558ef3a"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"平均光滑性与二次上界","source_units":[{"start":20355,"end":20726,"start_line":404,"end_line":408,"raw_tex":"  similarly follows from the Lipschitz continuity of $\\iota \\nabla f(\\xi, \\cdot)$ by the \n  identification provided in Lemma~\\ref{lem:swap_diff_and_expectation}. The final \n  inequality follows by expanding $F$ in a zeroth-order Taylor expansion \n  around $w$ and using the Lipschitz continuity. See e.g.\\ \n  \\cite[Appendix~B]{BottouCurtisNocedal.2018} for more details.\n","sha256_utf8":"2bef0fe9ba9ec008eb99657c5f9e659213e91cf393cb2943dcadce3bc9716586"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"强制性保证唯一极小点","source_units":[{"start":20726,"end":21002,"start_line":409,"end_line":414,"raw_tex":" %\n  We note that $F$ is coercive since it is strongly convex. Combined with the G\\^{a}teaux \n  differentiability, this guarantees the existence of a unique global \n  minimum, see e.g.~\\cite[Theorem 25.D, Proposition~25.20 and \n  Corollary~25.15]{ZeidlerIIB.1990}.\n\\end{proof}","sha256_utf8":"fa0f1b40098c5fa95b14915321b0d36609c7b8212c6c1415269a83744c366805"}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","title":"微分与期望交换","source_units":[{"start":84991,"end":85315,"start_line":1658,"end_line":1665,"raw_tex":"\\begin{lemma} \\label{lem:swap_diff_and_expectation}\n  Let Assumption~\\ref{ass:fStoch} be fulfilled.\n  Then $F = \\E_{\\xi}[f(\\xi, \\cdot)]$ is G\\^{a}teaux differentiable and its derivative is \n  given by\n  \\begin{align*}\n    \\dual{\\iota \\nabla F (v)}{w} = \\E_{\\xi}[\\dual{\\iota \\nabla f (\\xi, v)}{w}].\n  \\end{align*}\n\\end{lemma}","sha256_utf8":"81bbcfe4234aa907410e770c82389b03053e921ecb941f194621aae274ea92e6"}]},{"id":"S021","semantic_type":"proof","kind":"proof","title":"交换引理完整引用证明","source_units":[{"start":85316,"end":85382,"start_line":1666,"end_line":1668,"raw_tex":"\\begin{proof}\n  See e.g.~\\cite[Lemma 6]{RyuBoyd.2016}.\n\\end{proof}","sha256_utf8":"6b82bf242fb6043ceb73b5b4c1acb99665eebe1e8a776ce160b600554d359660"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"援引微分交换标准引理","source_units":[{"start":85316,"end":85382,"start_line":1666,"end_line":1668,"raw_tex":"\\begin{proof}\n  See e.g.~\\cite[Lemma 6]{RyuBoyd.2016}.\n\\end{proof}","sha256_utf8":"6b82bf242fb6043ceb73b5b4c1acb99665eebe1e8a776ce160b600554d359660"}]},{"id":"S023","semantic_type":"lemma","kind":"lemma","title":"二阶矩一致先验界","source_units":[{"start":23514,"end":24755,"start_line":458,"end_line":483,"raw_tex":"\\begin{lemma} \\label{lem:apriori2}\n  Let Assumption~\\ref{ass:fStoch} be fulfilled and let $\\{\\alpha_n\\}_{n \\in \\N}$ be a \n  sequence of positive real numbers such that $\\sum_{n =1}^{\\infty} \\alpha_n^2 < \n  \\infty$.\n  For $\\Phi \\in [0,\\infty)$ the a priori bound\n  \\begin{align*}\n    &\\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \\\\\n    &\\leq \\| w_1 - w^*\\|^2 \\exptextB{ \\sum_{i=1}^{\\infty} \\big(2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n      \\alpha_i^2 \\big\\} + 4 \\alpha_i^2 L^2 m_i\\big)} \\\\\n    &\\quad + \\sum_{i=1}^{\\infty} \\big(\\Phi^2 \\min \\big\\{ \\E_i \\big[ \\|\\iota \\nabla \n    f_i(w^i)\\|^{-2} \\big], \n    \\alpha_i^2 \\big\\} + 2 ( 1- m_i) + 2 \\min \\big\\{1, 2 \\alpha_i^2 \\sigma^2 \\big\\}\\big)\\\\\n    &\\qquad \\times \\exptextB{\\sum_{j=i+1}^{\\infty} \\big(2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n      \\alpha_j^2 \\big\\} + 4 \\alpha_j^2 L^2 m_j\\big)} =: M_2\n  \\end{align*}\n  is fulfilled, where $\\min\\big\\{\\Phi^{-2}, x \\big\\} = x$ for $\\Phi = 0$ and every $x \\in \\R$ \n  and \n  \\begin{align*}\n    m_i =\n    \\begin{cases}\n      1, & \\text{\\emph{if }} 2 \\alpha_i^2 L^2 \\E_{i-1} \\big[ \\| w^i - w^*\\|^2\\big] \\le 1, \\\\\n      0, & \\text{\\emph{otherwise}}.\n    \\end{cases}\n  \\end{align*}\n  Furthermore, there exists $n_0 \\in \\N$ such that $m_i = 1$ for all $i \\geq n_0$.\n\\end{lemma}","sha256_utf8":"8d753cbfdcdb4d4a652eb6d35be830e6c813624904aa49daed93acf78aa81930"}]},{"id":"S024","semantic_type":"proof","kind":"proof","title":"二阶矩界完整证明","source_units":[{"start":25525,"end":34293,"start_line":499,"end_line":665,"raw_tex":"\\begin{proof}[Proof of Lemma~\\ref{lem:apriori2}]\n  We test the scheme defined by~\\eqref{eq:stochTamedEuler}\n  with $w^n - w^*$, in order to obtain that\n  \\begin{align}\\label{eq:proofApriori1}\n    \\begin{split}\n      &\\dual{w^{n+1} - w^* - (w^n - w^*)}{w^n - w^*} \\\\\n      &\\quad + \\frac{\\alpha_n \\dual{\\iota \\nabla f_n(w^n) - \\iota \\nabla \n          f_n(w^*)}{w^n - w^*}}{1 \n        + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\|} \n      = - \\frac{\\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*}}{1 + \\alpha_n \\|\\iota \n        \\nabla f_n(w^n)\\|}.\n    \\end{split}\n  \\end{align}\n  Using the identity $\\dual{u-v}{u} = \\frac{1}{2} (\\|u\\|^2 - \\|v\\|^2 + \\|u-v\\|^2)$, $u,v \\in H$, the \n  first summand on the left-hand side can be written as\n  \\begin{align*}\n    &- \\dual{w^n - w^* - (w^{n+1} - w^*)}{w^n - w^*} \\\\\n    &= - \\frac{1}{2} \\big( \\|w^n - w^*\\|^2 - \\|w^{n+1} - w^*\\|^2 + \\|w^{n+1} - w^n\\|^2 \\big)\\\\\n    &= \\frac{1}{2} \\Big(\\|w^{n+1} - w^*\\|^2 - \\|w^n - w^*\\|^2 - \\frac{\\alpha_n^2 \\| \\iota \n      \\nabla  f_n(w^n)\\|^2 }{(1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\|)^2}\\Big), \n  \\end{align*}\n  where we inserted the scheme in the last step. \n  Thus, inserting the monotonicity condition for $f_n$ into \n  \\eqref{eq:proofApriori1} and multiplying the inequality with the factor two, it follows that\n  \\begin{align*}\n    &\\|w^{n+1} - w^*\\|^2 - \\|w^n - w^*\\|^2 \\\\\n    &\\leq - \\frac{2\\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*}}{1 + \\alpha_n \n      \\|\\iota \\nabla  \n      f_n(w^n)\\|} \n    + \\frac{\\alpha_n^2 \\| \\iota \\nabla f_n(w^n)\\|^2 }{(1 + \\alpha_n \\|\\iota \\nabla  \n      f_n(w^n)\\|)^2} \n    =: I_1 + I_2.\n  \\end{align*}\n  Since the tamed Euler scheme is the forward Euler scheme with a second order \n  perturbation, it follows that\n  \\begin{align*}\n    \\frac{\\alpha_n \\iota \\nabla f_n(w^*)}{1 + \\alpha_n \\|\\iota \\nabla  f_n(w^n)\\|}\n    = \\frac{\\alpha_n \\iota \\nabla f_n(w^*)}{1 + \\alpha_n \\Phi}\n    + \\frac{ \\alpha_n^2 \\big(\\Phi - \\|\\iota \\nabla f_n(w^n)\\|\\big) \\iota \\nabla f_n(w^*) \n    }{(1 + \\alpha_n \\|\\iota \\nabla  f_n(w^n)\\|) (1 + \\alpha_n \\Phi )}\n  \\end{align*}\n  for $\\Phi \\in [0,\\infty)$. Note that we have $w^*$ in the numerator of the left-hand-side \n  but $w^n$ in the denominator. We insert this equality into $I_1$ and use the \n  Cauchy--Schwarz inequality and \n  Young's  inequality for products in order to obtain\n  \\begin{align*}\n    I_1 \n    &= - \\frac{\\alpha_n \\dual{2 \\iota \\nabla f_n(w^*)}{w^n - w^*}} {1 + \\alpha_n \\|\\iota \\nabla  \n      f_n(w^n)\\|} \\\\\n    &\\leq - \\frac{2 \\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*} }{1 + \\alpha_n \\Phi}\n    + \\frac{2 \\alpha_n^2 \\Phi \\| \\iota \\nabla f_n(w^*)\\| \\| w^n - w^*\\| }{(1 + \\alpha_n \\|\\iota \n    \\nabla  f_n(w^n)\\|) (1 + \\alpha_n \\Phi )} \\\\\n    &\\quad + \\frac{2 \\alpha_n^2 \\|\\iota \\nabla f_n(w^n)\\| \\| \\iota \\nabla f_n(w^*)\\| \\| w^n - \n    w^*\\| }{(1 + \\alpha_n \\|\\iota \\nabla  f_n(w^n)\\|) (1 + \\alpha_n \\Phi )} \\\\\n    &\\leq - \\frac{2 \\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*} }{1 + \\alpha_n \\Phi}\n    + \\frac{ \\alpha_n^2 \\big(\\Phi^2 + \\|\\iota \\nabla f_n(w^n)\\|^2\\big)}{(1 + \\alpha_n \\|\\iota \n    \\nabla  f_n(w^n)\\|)^2} \\\\\n    &\\quad + \\frac{2 \\alpha_n^2 \\| \\iota \\nabla f_n(w^*)\\|^2 \\| w^n - w^*\\|^2 }{(1 + \\alpha_n \n    \\Phi )^2}\n    =: I_{1,1} + I_{1,2} + I_{1,3}.\n  \\end{align*}\n  For $I_{1,1} = - \\frac{2 \\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*} }{1 + \\alpha_n \n  \\Phi}$, we notice that $\\E_{\\xi_n} [I_{1,1}] = 0$. Moreover, for $I_{1,2} = \\frac{\\alpha_n^2 \n  ( \n  \\Phi^2 + \\|\\iota \\nabla f_n(w^n)\\|^2 )}{(1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\|)^2}$, we get\n  \\begin{align*}\n    I_{1,2}\n    &\\leq \\Phi^2 \\min \\big\\{ \\|\\iota \\nabla f_n(w^n)\\|^{-2} , \\alpha_n^2 \\big\\}\n    + \\min \\big\\{1, \\alpha_n^2 \\|\\iota \\nabla f_n(w^n)\\|^2 \\big\\} \\\\\n    &\\leq \\Phi^2 \\min \\big\\{ \\|\\iota \\nabla f_n(w^n)\\|^{-2} , \\alpha_n^2 \\big\\}\n    + \\min \\big\\{1, 2 \\alpha_n^2 L_{\\xi_n}^2 \\|w^n - w^*\\|^2\\big\\}\\\\ \n    &\\quad + \\min \\big\\{1,2 \\alpha_n^2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\big\\}\n  \\end{align*}\n  and\n  \\begin{align*}\n    I_{1,3}\n    = \\frac{2\\alpha_n^2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\|w^n - w^*\\|^2}{(1 + \\alpha_n \\Phi )^2}\n    \\leq 2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\min \\big\\{\\Phi^{-2}, \\alpha_n^2 \\big\\} \\|w^n - w^*\\|^2.\n  \\end{align*}\n  A bound for $I_2$ is given by\n  \\begin{align*}\n    I_2 &= \\frac{\\alpha_n^2 \\| \\iota \\nabla f_n(w^n)\\|^2 }{(1 + \\alpha_n \\|\\iota \\nabla \n    f_n(w^n)\\| )^2}\n    \\leq \\min \\big\\{1, \\alpha_n^2 \\| \\iota \\nabla f_n(w^n)\\|^2 \\big\\}\\\\\n    &\\leq \\min \\big\\{1, 2 \\alpha_n^2 L_{\\xi_n}^2 \\| w^n - w^*\\|^2 \\big\\}\n    + \\min \\big\\{1, 2 \\alpha_n^2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\big\\}.\n  \\end{align*}  \n  Then it follows\n  \\begin{align}\n    \\label{eq:aprioriBoundMn}\n    \\begin{split}\n      &\\|w^{n+1} - w^*\\|^2 - \\|w^n - w^*\\|^2 \\leq  I_1 + I_2\\\\\n      &\\leq - \\frac{2 \\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*} }{1 + \\alpha_n \\Phi}\n      + \\Phi^2 \\min \\big\\{ \\|\\iota \\nabla f_n(w^n)\\|^{-2} , \\alpha_n^2 \\big\\}\\\\\n      &\\quad + 2 \\min \\big\\{1, 2 \\alpha_n^2 L_{\\xi_n}^2 \\|w^n - w^*\\|^2\\big\\} + 2 \\min \n      \\big\\{1,2 \\alpha_n^2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\big\\}\\\\\n      &\\quad + 2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\min \\big\\{\\Phi^{-2}, \\alpha_n^2 \\big\\} \\|w^n - \n      w^*\\|^2.\n    \\end{split}\n  \\end{align}\n  Taking the $\\E_n$-expectation, we then obtain\n  \\begin{equation}\n    \\label{eq:aprioriBoundMnExp}\n    \\begin{aligned}\n      &\\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \\\\\n      &\\quad \\leq \\big(1 + 2 \\sigma^2 \\min\\big\\{ \\Phi^{-2}, \\alpha_n^2 \\big\\} \\big) \\E_{n-1} \n      \\big[ \n      \\|w^n - w^*\\|^2 \\big]\\\\\n      &\\qquad + \\Phi^2 \\min \\big\\{ \\E_n \\big[\\|\\iota \\nabla f_n(w^n)\\|^{-2}\\big], \\alpha_n^2 \n      \\big\\}\\\\\n      &\\qquad + 2 \\min \\big\\{1, 2 \\alpha_n^2 L^2 \\E_{n-1} \\big[ \\|w^n - w^*\\|^2 \\big]\\big\\} \n      + 2 \\min \\big\\{1,2 \\alpha_n^2 \\sigma^2\\big\\}\\\\\n      &\\quad= \\big(1 + 2 \\min\\big\\{ \\Phi^{-2}, \\alpha_n^2 \\big\\} \\sigma^2+ 4 \\alpha_n^2 L^2 \n      m_n\\big) \\E_{n-1} \\big[ \\|w^n - w^*\\|^2 \\big]\\\\\n      &\\qquad + \\Phi^2 \\min \\big\\{ \\E_n \\big[\\|\\iota \\nabla f_n(w^n)\\|^{-2}\\big], \n      \\alpha_n^2 \\big\\}\n      + 2 (1-m_n) \n      + 2 \\min \\big\\{1,2 \\alpha_n^2 \\sigma^2\\big\\},\n    \\end{aligned}\n  \\end{equation}\n  with $m_n$ defined as in the lemma statement. Reinserting the bound \n  repeatedly thus yields\n  \\begin{align*}\n    &\\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \\\\\n    &\\leq \\| w_1 - w^*\\|^2 \\prod_{i=1}^{n} \\big(1 + 2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n    \\alpha_i^2 \\big\\} + 4 \\alpha_i^2 L^2 m_i\\big) \\\\\n    &\\quad + \\sum_{i=1}^{n} \\big(\\Phi^2 \\min \\big\\{ \\E_i \\big[ \\|\\iota \\nabla \n    f_i(w^i)\\|^{-2} \\big], \n    \\alpha_i^2 \\big\\} + 2 ( 1- m_i) + 2 \\min \\big\\{1, 2 \\alpha_i^2 \\sigma^2 \\big\\}\\big)\\\\\n    &\\qquad \\times \\prod_{j=i+1}^{n} \\big(1 + 2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n    \\alpha_j^2 \\big\\} + 4 \\alpha_j^2 L^2 m_j\\big).\n  \\end{align*}\n  Finally, we apply the inequality $1 + x \\leq \\exptext{x}$, $x \\in \\R$, and make \n  the bound independent of $n$ by bounding the final sums by the corresponding infinite \n  sums, in order to obtain\n  \\begin{align*}\n    &\\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \\\\\n    &\\leq \\| w_1 - w^*\\|^2 \\exptextB{ \\sum_{i=1}^{\\infty} \\big(2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n      \\alpha_i^2 \\big\\} + 4 \\alpha_i^2 L^2 m_i\\big)} \\\\\n    &\\quad + \\sum_{i=1}^{\\infty} \\big(\\Phi^2 \\min \\big\\{ \\E_i \\big[ \\|\\iota \\nabla \n    f_i(w^i)\\|^{-2} \\big], \n    \\alpha_i^2 \\big\\} + 2 ( 1- m_i) + 2 \\min \\big\\{1, 2 \\alpha_i^2 \\sigma^2 \\big\\}\\big)\\\\\n    &\\qquad \\times \\exptextB{\\sum_{j=i+1}^{\\infty} \\big(2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n      \\alpha_j^2 \\big\\} + 4 \\alpha_j^2 L^2 m_j\\big)}.\n  \\end{align*}\n  It remains to verify, that there exists $n_0 \\in \\N$ such that $m_n = 1$ for all $n \\geq \n  n_0$. This can be done by estimating \\eqref{eq:aprioriBoundMnExp} and following a \n  similar line of argumentation as before. First, we can write\n  \\begin{align*}\n    \\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \n    \\leq \\big(1 + 2 \\alpha_n^2 \\sigma^2 + 4 \\alpha_n^2 L^2 \\big) \\E_{n-1} \\big[ \\|w^n - \n    w^*\\|^2\\big] + \\Phi^2 \\alpha_n^2 + 4 \\alpha_n^2 \\sigma^2.\n  \\end{align*}\n  Reinserting the inequality $n -1$ times, it follows that\n  \\begin{align*}\n    &\\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \\\\\n    &\\leq \\| w_1 - w^*\\|^2 \\prod_{i=1}^{n} \\big(1 + 2 \\sigma^2 \\alpha_i^2 + 4 \\alpha_i^2 L^2 \n    \\big) \n    + \\sum_{i=1}^{n} 4 \\alpha_i^2 \\sigma^2 \\prod_{j=i+1}^{n} \\big(1 + 2 \\sigma^2 \n    \\alpha_j^2 + 4 \\alpha_j^2 L^2 \\big)\\\\\n    &\\leq \\| w_1 - w^*\\|^2 \\exptextB{ \\big(2 \\sigma^2 + 4 L^2 \\big) \\sum_{i=1}^{\\infty} \n    \\alpha_i^2 }\n    + \\sum_{i=1}^{\\infty} 4 \\alpha_i^2 \\sigma^2 \\exptextB{ \\big(2 \\sigma^2 + 4 L^2 \\big) \n    \\sum_{j=i+1}^{\\infty} \\alpha_j^2 }.\n  \\end{align*}\n  Since $\\sum_{n =1}^{\\infty} \\alpha_n^2 < \\infty$ there is thus a $n_0 \\in \\N$ such that $2 \\alpha_n^2 L^2 \\E_{n-1} \\big[ \\| w^n - w^*\\|^2\\big] \\leq 1$ for all $n \\geq n_0$.\n\\end{proof}","sha256_utf8":"244a2b9cb735081711b2f7ebc699e08ea5aa4251c52c7c01c1d74a123169d45a"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"测试迭代得到误差恒等式","source_units":[{"start":25525,"end":26093,"start_line":499,"end_line":511,"raw_tex":"\\begin{proof}[Proof of Lemma~\\ref{lem:apriori2}]\n  We test the scheme defined by~\\eqref{eq:stochTamedEuler}\n  with $w^n - w^*$, in order to obtain that\n  \\begin{align}\\label{eq:proofApriori1}\n    \\begin{split}\n      &\\dual{w^{n+1} - w^* - (w^n - w^*)}{w^n - w^*} \\\\\n      &\\quad + \\frac{\\alpha_n \\dual{\\iota \\nabla f_n(w^n) - \\iota \\nabla \n          f_n(w^*)}{w^n - w^*}}{1 \n        + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\|} \n      = - \\frac{\\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*}}{1 + \\alpha_n \\|\\iota \n        \\nabla f_n(w^n)\\|}.\n    \\end{split}\n  \\end{align}\n","sha256_utf8":"ac8615abb7270ece707f0cb463f527cbe068a5f1e4598158c40992d3274c4bf8"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"展开误差能量差","source_units":[{"start":26093,"end":26655,"start_line":512,"end_line":520,"raw_tex":"  Using the identity $\\dual{u-v}{u} = \\frac{1}{2} (\\|u\\|^2 - \\|v\\|^2 + \\|u-v\\|^2)$, $u,v \\in H$, the \n  first summand on the left-hand side can be written as\n  \\begin{align*}\n    &- \\dual{w^n - w^* - (w^{n+1} - w^*)}{w^n - w^*} \\\\\n    &= - \\frac{1}{2} \\big( \\|w^n - w^*\\|^2 - \\|w^{n+1} - w^*\\|^2 + \\|w^{n+1} - w^n\\|^2 \\big)\\\\\n    &= \\frac{1}{2} \\Big(\\|w^{n+1} - w^*\\|^2 - \\|w^n - w^*\\|^2 - \\frac{\\alpha_n^2 \\| \\iota \n      \\nabla  f_n(w^n)\\|^2 }{(1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\|)^2}\\Big), \n  \\end{align*}\n  where we inserted the scheme in the last step. \n","sha256_utf8":"1c5eeed251fe1a468536ccb867aef60d77f467cfa857ce32713f0c1b2d1c212c"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"单调性分离两个误差项","source_units":[{"start":26655,"end":27139,"start_line":521,"end_line":531,"raw_tex":"  Thus, inserting the monotonicity condition for $f_n$ into \n  \\eqref{eq:proofApriori1} and multiplying the inequality with the factor two, it follows that\n  \\begin{align*}\n    &\\|w^{n+1} - w^*\\|^2 - \\|w^n - w^*\\|^2 \\\\\n    &\\leq - \\frac{2\\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*}}{1 + \\alpha_n \n      \\|\\iota \\nabla  \n      f_n(w^n)\\|} \n    + \\frac{\\alpha_n^2 \\| \\iota \\nabla f_n(w^n)\\|^2 }{(1 + \\alpha_n \\|\\iota \\nabla  \n      f_n(w^n)\\|)^2} \n    =: I_1 + I_2.\n  \\end{align*}\n","sha256_utf8":"70e63f985ad4e0aaf2fdc2505c4c4e80635faf18ac56ebd8fcd7a8bcc16a4f0c"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"引入截断参数分解噪声","source_units":[{"start":27139,"end":28780,"start_line":532,"end_line":559,"raw_tex":"  Since the tamed Euler scheme is the forward Euler scheme with a second order \n  perturbation, it follows that\n  \\begin{align*}\n    \\frac{\\alpha_n \\iota \\nabla f_n(w^*)}{1 + \\alpha_n \\|\\iota \\nabla  f_n(w^n)\\|}\n    = \\frac{\\alpha_n \\iota \\nabla f_n(w^*)}{1 + \\alpha_n \\Phi}\n    + \\frac{ \\alpha_n^2 \\big(\\Phi - \\|\\iota \\nabla f_n(w^n)\\|\\big) \\iota \\nabla f_n(w^*) \n    }{(1 + \\alpha_n \\|\\iota \\nabla  f_n(w^n)\\|) (1 + \\alpha_n \\Phi )}\n  \\end{align*}\n  for $\\Phi \\in [0,\\infty)$. Note that we have $w^*$ in the numerator of the left-hand-side \n  but $w^n$ in the denominator. We insert this equality into $I_1$ and use the \n  Cauchy--Schwarz inequality and \n  Young's  inequality for products in order to obtain\n  \\begin{align*}\n    I_1 \n    &= - \\frac{\\alpha_n \\dual{2 \\iota \\nabla f_n(w^*)}{w^n - w^*}} {1 + \\alpha_n \\|\\iota \\nabla  \n      f_n(w^n)\\|} \\\\\n    &\\leq - \\frac{2 \\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*} }{1 + \\alpha_n \\Phi}\n    + \\frac{2 \\alpha_n^2 \\Phi \\| \\iota \\nabla f_n(w^*)\\| \\| w^n - w^*\\| }{(1 + \\alpha_n \\|\\iota \n    \\nabla  f_n(w^n)\\|) (1 + \\alpha_n \\Phi )} \\\\\n    &\\quad + \\frac{2 \\alpha_n^2 \\|\\iota \\nabla f_n(w^n)\\| \\| \\iota \\nabla f_n(w^*)\\| \\| w^n - \n    w^*\\| }{(1 + \\alpha_n \\|\\iota \\nabla  f_n(w^n)\\|) (1 + \\alpha_n \\Phi )} \\\\\n    &\\leq - \\frac{2 \\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*} }{1 + \\alpha_n \\Phi}\n    + \\frac{ \\alpha_n^2 \\big(\\Phi^2 + \\|\\iota \\nabla f_n(w^n)\\|^2\\big)}{(1 + \\alpha_n \\|\\iota \n    \\nabla  f_n(w^n)\\|)^2} \\\\\n    &\\quad + \\frac{2 \\alpha_n^2 \\| \\iota \\nabla f_n(w^*)\\|^2 \\| w^n - w^*\\|^2 }{(1 + \\alpha_n \n    \\Phi )^2}\n    =: I_{1,1} + I_{1,2} + I_{1,3}.\n  \\end{align*}\n","sha256_utf8":"2c94be49483556310d504408c6db4b993986c11e82bf07bcaabb58abb467d417"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"中心化噪声与截断估计","source_units":[{"start":28780,"end":29724,"start_line":560,"end_line":577,"raw_tex":"  For $I_{1,1} = - \\frac{2 \\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*} }{1 + \\alpha_n \n  \\Phi}$, we notice that $\\E_{\\xi_n} [I_{1,1}] = 0$. Moreover, for $I_{1,2} = \\frac{\\alpha_n^2 \n  ( \n  \\Phi^2 + \\|\\iota \\nabla f_n(w^n)\\|^2 )}{(1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\|)^2}$, we get\n  \\begin{align*}\n    I_{1,2}\n    &\\leq \\Phi^2 \\min \\big\\{ \\|\\iota \\nabla f_n(w^n)\\|^{-2} , \\alpha_n^2 \\big\\}\n    + \\min \\big\\{1, \\alpha_n^2 \\|\\iota \\nabla f_n(w^n)\\|^2 \\big\\} \\\\\n    &\\leq \\Phi^2 \\min \\big\\{ \\|\\iota \\nabla f_n(w^n)\\|^{-2} , \\alpha_n^2 \\big\\}\n    + \\min \\big\\{1, 2 \\alpha_n^2 L_{\\xi_n}^2 \\|w^n - w^*\\|^2\\big\\}\\\\ \n    &\\quad + \\min \\big\\{1,2 \\alpha_n^2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\big\\}\n  \\end{align*}\n  and\n  \\begin{align*}\n    I_{1,3}\n    = \\frac{2\\alpha_n^2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\|w^n - w^*\\|^2}{(1 + \\alpha_n \\Phi )^2}\n    \\leq 2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\min \\big\\{\\Phi^{-2}, \\alpha_n^2 \\big\\} \\|w^n - w^*\\|^2.\n  \\end{align*}\n","sha256_utf8":"5164fdaca5afdd33fe733f616de53e4ad5ebdac86c44936384a5c2e39932454e"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"估计迭代增量平方","source_units":[{"start":29724,"end":30113,"start_line":578,"end_line":585,"raw_tex":"  A bound for $I_2$ is given by\n  \\begin{align*}\n    I_2 &= \\frac{\\alpha_n^2 \\| \\iota \\nabla f_n(w^n)\\|^2 }{(1 + \\alpha_n \\|\\iota \\nabla \n    f_n(w^n)\\| )^2}\n    \\leq \\min \\big\\{1, \\alpha_n^2 \\| \\iota \\nabla f_n(w^n)\\|^2 \\big\\}\\\\\n    &\\leq \\min \\big\\{1, 2 \\alpha_n^2 L_{\\xi_n}^2 \\| w^n - w^*\\|^2 \\big\\}\n    + \\min \\big\\{1, 2 \\alpha_n^2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\big\\}.\n  \\end{align*}  \n","sha256_utf8":"30552a0fb8a74fc3c42a564d3d5643d5d3f50fa0f8691f6a3e57017030f387a9"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"汇总路径误差不等式","source_units":[{"start":30113,"end":30721,"start_line":586,"end_line":598,"raw_tex":"  Then it follows\n  \\begin{align}\n    \\label{eq:aprioriBoundMn}\n    \\begin{split}\n      &\\|w^{n+1} - w^*\\|^2 - \\|w^n - w^*\\|^2 \\leq  I_1 + I_2\\\\\n      &\\leq - \\frac{2 \\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - w^*} }{1 + \\alpha_n \\Phi}\n      + \\Phi^2 \\min \\big\\{ \\|\\iota \\nabla f_n(w^n)\\|^{-2} , \\alpha_n^2 \\big\\}\\\\\n      &\\quad + 2 \\min \\big\\{1, 2 \\alpha_n^2 L_{\\xi_n}^2 \\|w^n - w^*\\|^2\\big\\} + 2 \\min \n      \\big\\{1,2 \\alpha_n^2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\big\\}\\\\\n      &\\quad + 2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\min \\big\\{\\Phi^{-2}, \\alpha_n^2 \\big\\} \\|w^n - \n      w^*\\|^2.\n    \\end{split}\n  \\end{align}\n","sha256_utf8":"0dca8c5887a289dd45ac2c7d3af89426212e5719f03a4d3ad2cc745409792b3d"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"取期望引入截断指标","source_units":[{"start":30721,"end":31638,"start_line":599,"end_line":618,"raw_tex":"  Taking the $\\E_n$-expectation, we then obtain\n  \\begin{equation}\n    \\label{eq:aprioriBoundMnExp}\n    \\begin{aligned}\n      &\\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \\\\\n      &\\quad \\leq \\big(1 + 2 \\sigma^2 \\min\\big\\{ \\Phi^{-2}, \\alpha_n^2 \\big\\} \\big) \\E_{n-1} \n      \\big[ \n      \\|w^n - w^*\\|^2 \\big]\\\\\n      &\\qquad + \\Phi^2 \\min \\big\\{ \\E_n \\big[\\|\\iota \\nabla f_n(w^n)\\|^{-2}\\big], \\alpha_n^2 \n      \\big\\}\\\\\n      &\\qquad + 2 \\min \\big\\{1, 2 \\alpha_n^2 L^2 \\E_{n-1} \\big[ \\|w^n - w^*\\|^2 \\big]\\big\\} \n      + 2 \\min \\big\\{1,2 \\alpha_n^2 \\sigma^2\\big\\}\\\\\n      &\\quad= \\big(1 + 2 \\min\\big\\{ \\Phi^{-2}, \\alpha_n^2 \\big\\} \\sigma^2+ 4 \\alpha_n^2 L^2 \n      m_n\\big) \\E_{n-1} \\big[ \\|w^n - w^*\\|^2 \\big]\\\\\n      &\\qquad + \\Phi^2 \\min \\big\\{ \\E_n \\big[\\|\\iota \\nabla f_n(w^n)\\|^{-2}\\big], \n      \\alpha_n^2 \\big\\}\n      + 2 (1-m_n) \n      + 2 \\min \\big\\{1,2 \\alpha_n^2 \\sigma^2\\big\\},\n    \\end{aligned}\n  \\end{equation}\n","sha256_utf8":"903926e8a0a0c56068a364872ca466e1c939d9750f2ab890d9328dd9b8ee4488"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"展开二阶矩乘积递推","source_units":[{"start":31638,"end":32274,"start_line":619,"end_line":630,"raw_tex":"  with $m_n$ defined as in the lemma statement. Reinserting the bound \n  repeatedly thus yields\n  \\begin{align*}\n    &\\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \\\\\n    &\\leq \\| w_1 - w^*\\|^2 \\prod_{i=1}^{n} \\big(1 + 2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n    \\alpha_i^2 \\big\\} + 4 \\alpha_i^2 L^2 m_i\\big) \\\\\n    &\\quad + \\sum_{i=1}^{n} \\big(\\Phi^2 \\min \\big\\{ \\E_i \\big[ \\|\\iota \\nabla \n    f_i(w^i)\\|^{-2} \\big], \n    \\alpha_i^2 \\big\\} + 2 ( 1- m_i) + 2 \\min \\big\\{1, 2 \\alpha_i^2 \\sigma^2 \\big\\}\\big)\\\\\n    &\\qquad \\times \\prod_{j=i+1}^{n} \\big(1 + 2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n    \\alpha_j^2 \\big\\} + 4 \\alpha_j^2 L^2 m_j\\big).\n  \\end{align*}\n","sha256_utf8":"babc59584ffeaee75e00616651b01e1a329214484f919b41778dbab23ffe8b57"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"指数控制得到统一界","source_units":[{"start":32274,"end":33045,"start_line":631,"end_line":643,"raw_tex":"  Finally, we apply the inequality $1 + x \\leq \\exptext{x}$, $x \\in \\R$, and make \n  the bound independent of $n$ by bounding the final sums by the corresponding infinite \n  sums, in order to obtain\n  \\begin{align*}\n    &\\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \\\\\n    &\\leq \\| w_1 - w^*\\|^2 \\exptextB{ \\sum_{i=1}^{\\infty} \\big(2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n      \\alpha_i^2 \\big\\} + 4 \\alpha_i^2 L^2 m_i\\big)} \\\\\n    &\\quad + \\sum_{i=1}^{\\infty} \\big(\\Phi^2 \\min \\big\\{ \\E_i \\big[ \\|\\iota \\nabla \n    f_i(w^i)\\|^{-2} \\big], \n    \\alpha_i^2 \\big\\} + 2 ( 1- m_i) + 2 \\min \\big\\{1, 2 \\alpha_i^2 \\sigma^2 \\big\\}\\big)\\\\\n    &\\qquad \\times \\exptextB{\\sum_{j=i+1}^{\\infty} \\big(2 \\sigma^2 \\min\\big\\{\\Phi^{-2}, \n      \\alpha_j^2 \\big\\} + 4 \\alpha_j^2 L^2 m_j\\big)}.\n  \\end{align*}\n","sha256_utf8":"f256e713d5b041797483dcce697828142b8083e644e3defc146610a57b4c3383"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"建立去截断粗递推","source_units":[{"start":33045,"end":33508,"start_line":644,"end_line":651,"raw_tex":"  It remains to verify, that there exists $n_0 \\in \\N$ such that $m_n = 1$ for all $n \\geq \n  n_0$. This can be done by estimating \\eqref{eq:aprioriBoundMnExp} and following a \n  similar line of argumentation as before. First, we can write\n  \\begin{align*}\n    \\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \n    \\leq \\big(1 + 2 \\alpha_n^2 \\sigma^2 + 4 \\alpha_n^2 L^2 \\big) \\E_{n-1} \\big[ \\|w^n - \n    w^*\\|^2\\big] + \\Phi^2 \\alpha_n^2 + 4 \\alpha_n^2 \\sigma^2.\n  \\end{align*}\n","sha256_utf8":"d7d5bf0c0958ad8414b32694b3e22bcd906c42019eb9135aa7000abac1387565"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"证明截断指标最终为一","source_units":[{"start":33508,"end":34293,"start_line":652,"end_line":665,"raw_tex":"  Reinserting the inequality $n -1$ times, it follows that\n  \\begin{align*}\n    &\\E_n \\big[\\|w^{n+1} - w^*\\|^2\\big] \\\\\n    &\\leq \\| w_1 - w^*\\|^2 \\prod_{i=1}^{n} \\big(1 + 2 \\sigma^2 \\alpha_i^2 + 4 \\alpha_i^2 L^2 \n    \\big) \n    + \\sum_{i=1}^{n} 4 \\alpha_i^2 \\sigma^2 \\prod_{j=i+1}^{n} \\big(1 + 2 \\sigma^2 \n    \\alpha_j^2 + 4 \\alpha_j^2 L^2 \\big)\\\\\n    &\\leq \\| w_1 - w^*\\|^2 \\exptextB{ \\big(2 \\sigma^2 + 4 L^2 \\big) \\sum_{i=1}^{\\infty} \n    \\alpha_i^2 }\n    + \\sum_{i=1}^{\\infty} 4 \\alpha_i^2 \\sigma^2 \\exptextB{ \\big(2 \\sigma^2 + 4 L^2 \\big) \n    \\sum_{j=i+1}^{\\infty} \\alpha_j^2 }.\n  \\end{align*}\n  Since $\\sum_{n =1}^{\\infty} \\alpha_n^2 < \\infty$ there is thus a $n_0 \\in \\N$ such that $2 \\alpha_n^2 L^2 \\E_{n-1} \\big[ \\| w^n - w^*\\|^2\\big] \\leq 1$ for all $n \\geq n_0$.\n\\end{proof}","sha256_utf8":"43bca419c8af84c8dd862355a9c53904a1ec2c498f6fb808cf767d313b026d88"}]},{"id":"S037","semantic_type":"lemma","kind":"lemma","title":"四阶矩一致先验界","source_units":[{"start":34296,"end":35166,"start_line":668,"end_line":683,"raw_tex":"\\begin{lemma} \\label{lem:apriori4}\n  Let Assumption~\\ref{ass:fHigherIntegrability} be fulfilled and \n  let $\\{\\alpha_n\\}_{n \\in \\N}$ be a sequence of positive real numbers such that $\\sum_{n \n  =1}^{\\infty} \\alpha_n^2 < \\infty$.\n  Then the a priori bound  \n  \\begin{align*}\n    \\E_n \\big[\\|w^{n+1} - w^*\\|^4\\big] \n    &\\leq \\|w^1 - w^*\\|^4 \\exptextB{\\sum_{i = 1}^{\\infty} c_1^i(\\alpha_i)} \\\\\n    &\\quad + \\sum_{i=1}^{\\infty} \\big(c_2^i(\\alpha_i) M_2 + c_3^i(\\alpha_i)\\big) \n    \\exptextB{\\sum_{j = i+ 1}^{\\infty} c_1^j(\\alpha_j) } =: M_4\n  \\end{align*}\n  is fulfilled for $c_1^i, c_2^i, c_3^i \\colon (0,\\infty) \\to (0,\\infty)$ such that there exist \n  $C_1^k, C_2^k, C_3^k, C_4^k \\in (0,\\infty)$ with $c_k^i(\\alpha) \\leq C_1^i \\min \\{C_2^i, \n  \\alpha^4\\} + C_3^i \\min \\{C_4^i, \\alpha^2\\}$ for all $\\alpha \\in (0,\\infty)$, $k \\in \\{1,2,3\\}$ \n  and $i \\in \\N$.\n\\end{lemma}","sha256_utf8":"3ae02ad76679e85e9ce6164f85487cf383a2520ea5a6e9b97aed47f92b2e47d1"}]},{"id":"S038","semantic_type":"proof","kind":"proof","title":"四阶矩界完整证明","source_units":[{"start":35169,"end":39382,"start_line":686,"end_line":764,"raw_tex":"\\begin{proof}\n  Within the proof of Lemma~\\ref{lem:apriori2}, we verified the inequality \n  \\eqref{eq:aprioriBoundMn}. Starting from this point, we find\n  \\begin{equation}\n    \\label{eq:apriori4}\n    \\|w^{n+1} - w^*\\|^2 - \\|w^n - w^*\\|^2 \\leq A_n + B_n,\n  \\end{equation}\n  where\n  \\begin{align*}\n    A_n &= -\\frac{2 \\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - \n    w^*} }{1 + \\alpha_n \\Phi} \\qquad \\text{and} \\\\\n    B_n &= \\Phi^2 \\min \\big\\{ \\|\\iota \\nabla f_n(w^n)\\|^{-2} , \\alpha_n^2 \\big\\}\n     + 2 \\min \\big\\{1, 2 \\alpha_n^2 L_{\\xi_n}^2 \\|w^n - w^*\\|^2\\big\\}\\\\\n      &\\quad + 2 \\min \\big\\{1,2 \\alpha_n^2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\big\\}\n      + 2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\min \\big\\{\\Phi^{-2}, \\alpha_n^2 \\big\\} \\|w^n - w^*\\|^2 \n  \\end{align*}\n  for $\\Phi \\in [0,\\infty)$. We note that the parameter $\\Phi$ can be chosen such that \n  $\\E_n[B_n]$ is as small as possible.\n  From this it follows that \n  \\begin{align*}\n    \\E_{\\xi_n} [A_n] &= 0,\\\\\n    \\E_{\\xi_n} [B_n] &\\leq  \\Phi^2 \\min \\big\\{ \\E_{\\xi_n} \\big[ \\|\\iota \\nabla f_n(w^n)\\|^{-2}\\big] \n    , \n    \\alpha_n^2 \\big\\}\n    + 2 \\min \\big\\{1, 2 \\alpha_n^2 L^2 \\|w^n - w^*\\|^2\\big\\}\\\\\n    &\\quad + 2 \\min \\big\\{1,2 \\alpha_n^2 \\sigma^2 \\big\\}\n    + 2 \\sigma^2 \\min \\big\\{\\Phi^{-2}, \\alpha_n^2 \\big\\} \\|w^n - w^*\\|^2, \\\\\n    \\E_{\\xi_n} [A_n^2] \n    &\\leq 4 \\sigma^2 \\min\\{ \\Phi^{-2}, \\alpha_n^2 \\} \\| w^n - w^*\\|^2 \\qquad \\text{and}\\\\\n    \\E_{\\xi_n} [B_n^2] \n    &\\leq 4 \\Phi^4 \\min \\big\\{ \\big(\\E_{\\xi_n} \\big[ \\|\\iota \\nabla f_n(w^n)\\|^{-2}\\big]\\big)^2 , \n    \\alpha_n^4 \\big\\}\n    + 16 \\min \\big\\{1, 4 \\alpha_n^4 L_4^4 \\|w^n - w^*\\|^4\\big\\}\\\\\n    &\\quad + 16 \\min \\big\\{1,4 \\alpha_n^4 \\sigma_4^4 \\big\\}\n    + 16  \\sigma_4^4 \\min \\big\\{\\Phi^{-4}, \\alpha_n^4 \\big\\} \\|w^n - w^*\\|^4.\n  \\end{align*}\n   We note that for $a,b \\in \\R$ we have the identity $(a - b)a = \\frac{1}{2} \\big( |a|^2 - |b|^2 \n   + |a - b|^2 \\big)$, and thus $|a|^2 - |b|^2 \\le 2(a-b)a$. By multiplying the inequality from \n   \\eqref{eq:apriori4} with the factor $2\\|w^{n+1} - w^*\\|^2$, we therefore obtain\n  \\begin{align*}\n    \\|w^{n+1} - w^*\\|^4 - \\|w^n - w^*\\|^4\n    &\\leq 2\\big( A_n + B_n \\big) \\|w^{n+1} - w^*\\|^2\\\\\n    &\\leq 2\\big( A_n + B_n \\big) \\big( \\|w^n - w^*\\|^2 + A_n + B_n  \\big)\\\\\n    &\\leq 2(A_n + B_n) \\|w^n - w^*\\|^2  + 4 A_n^2 + 4 B_n^2 \n  \\end{align*}\n  and in $\\E_{\\xi_n}$-expectation\n  \\begin{align*}\n    & \\E_{\\xi_n} \\big[ \\|w^{n+1} - w^*\\|^4\\big] - \\|w^n - w^*\\|^4\\\\\n    &\\leq 4 \\Big(2 L^2 \\min \\big\\{\\tfrac{1}{2} L^{-2} \\|w^n - w^*\\|^{-2}, \\alpha_n^2 \\big\\}\n    + \\sigma^2 \\min \\big\\{\\Phi^{-2}, \\alpha_n^2 \\big\\}\\\\\n    &\\qquad + 64 L_4^4 \\min \\big\\{ \\tfrac{1}{4} L_4^{-4} \\|w^n - w^*\\|^{-4}, \\alpha_n^4 \n    \\big\\} + 16 \\sigma_4^4 \\min \\big\\{\\Phi^{-4}, \\alpha_n^4 \\big\\} \\Big)\\|w^n - w^*\\|^4 \\\\\n    &\\quad + 2 \\Big( \\Phi^2 \\min \\big\\{ \\E_{\\xi_n} \\big[ \\|\\iota \\nabla f_n(w^n)\\|^{-2}\\big] , \n    \\alpha_n^2 \\big\\} \n    + 4 \\sigma^2\\min \\big\\{\\tfrac{1}{2} \\sigma^{-2} , \\alpha_n^2 \\big\\} \\\\\n    &\\qquad + 8  \\sigma^2 \\min\\{ \\Phi^{-2}, \\alpha_n^2 \\} \\Big) \\|w^n - w^*\\|^2 \\\\\n    &\\quad + 16 \\Phi^4 \\min \\big\\{ \\big(\\E_{\\xi_n} \\big[ \\|\\iota \\nabla \n    f_n(w^n)\\|^{-2}\\big]\\big)^2 \n    , \\alpha_n^4 \\big\\}\n    + 256 \\sigma_4^4 \\min \\big\\{ \\tfrac{1}{4}\\sigma_4^{-4},\\alpha_n^4 \\big\\}\\\\\n    %\n    &=: c_1^n (\\alpha_n) \\|w^n - w^*\\|^4 + c_2^n(\\alpha_n) \\|w^n - w^*\\|^2 \n    + c_3^n(\\alpha_n).\n  \\end{align*}\n  Adding $\\|w^n - w^*\\|^4$ to both sides of the inequality, taking the \n  $\\E_{n-1}$-expectation and reinserting the bound, we obtain\n  \\begin{align*}\n    & \\E_n \\big[ \\|w^{n+1} - w^*\\|^4\\big] \\\\\n    &\\leq \\big(1 + c_1^n(\\alpha_n) \\big) \\E_{n-1} \\big[ \\|w^n - w^*\\|^4 \\big]\n    + c_2^n(\\alpha_n) M_2 + c_3^n(\\alpha_n)\\\\\n    &\\leq \\|w^1 - w^*\\|^4 \\prod_{i = 1}^{n} \\big(1 + c_1^i(\\alpha_i) \\big) \n    + \\sum_{i=1}^{n} \\big(c_2^i(\\alpha_i) M_2 + c_3^i(\\alpha_i)\\big) \\prod_{j = i+ \n    1}^{n} \\big(1 + c_1^j(\\alpha_j) \\big) \\\\\n    &\\leq \\|w^1 - w^*\\|^4 \\exptextB{\\sum_{i = 1}^{\\infty} c_1^i(\\alpha_i) } \n    + \\sum_{i=1}^{\\infty} \\big(c_2^i(\\alpha_i) M_2 + c_3^i(\\alpha_i)\\big) \n    \\exptextB{\\sum_{j = i+ 1}^{\\infty} c_1^j(\\alpha_j)} .\n  \\end{align*}\n  Finally, this is finite due to the assumption $\\sum_{n  =1}^{\\infty} \\alpha_n^2 < \\infty$.\n\\end{proof}","sha256_utf8":"93e202518d4d4bb8f91cbc1c2dc38e50cfa6060369f21dedb1dcae6eb53aa66b"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"分离中心化项与余项","source_units":[{"start":35169,"end":36049,"start_line":686,"end_line":703,"raw_tex":"\\begin{proof}\n  Within the proof of Lemma~\\ref{lem:apriori2}, we verified the inequality \n  \\eqref{eq:aprioriBoundMn}. Starting from this point, we find\n  \\begin{equation}\n    \\label{eq:apriori4}\n    \\|w^{n+1} - w^*\\|^2 - \\|w^n - w^*\\|^2 \\leq A_n + B_n,\n  \\end{equation}\n  where\n  \\begin{align*}\n    A_n &= -\\frac{2 \\alpha_n \\dual{\\iota \\nabla f_n(w^*)}{w^n - \n    w^*} }{1 + \\alpha_n \\Phi} \\qquad \\text{and} \\\\\n    B_n &= \\Phi^2 \\min \\big\\{ \\|\\iota \\nabla f_n(w^n)\\|^{-2} , \\alpha_n^2 \\big\\}\n     + 2 \\min \\big\\{1, 2 \\alpha_n^2 L_{\\xi_n}^2 \\|w^n - w^*\\|^2\\big\\}\\\\\n      &\\quad + 2 \\min \\big\\{1,2 \\alpha_n^2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\big\\}\n      + 2 \\|\\iota \\nabla f_n(w^*)\\|^2 \\min \\big\\{\\Phi^{-2}, \\alpha_n^2 \\big\\} \\|w^n - w^*\\|^2 \n  \\end{align*}\n  for $\\Phi \\in [0,\\infty)$. We note that the parameter $\\Phi$ can be chosen such that \n  $\\E_n[B_n]$ is as small as possible.\n","sha256_utf8":"0e55ae26b6f5e29093e7c67267dce8a903682e80d0e27a980b6bb886fa87d80d"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"估计两项条件矩","source_units":[{"start":36049,"end":36929,"start_line":704,"end_line":721,"raw_tex":"  From this it follows that \n  \\begin{align*}\n    \\E_{\\xi_n} [A_n] &= 0,\\\\\n    \\E_{\\xi_n} [B_n] &\\leq  \\Phi^2 \\min \\big\\{ \\E_{\\xi_n} \\big[ \\|\\iota \\nabla f_n(w^n)\\|^{-2}\\big] \n    , \n    \\alpha_n^2 \\big\\}\n    + 2 \\min \\big\\{1, 2 \\alpha_n^2 L^2 \\|w^n - w^*\\|^2\\big\\}\\\\\n    &\\quad + 2 \\min \\big\\{1,2 \\alpha_n^2 \\sigma^2 \\big\\}\n    + 2 \\sigma^2 \\min \\big\\{\\Phi^{-2}, \\alpha_n^2 \\big\\} \\|w^n - w^*\\|^2, \\\\\n    \\E_{\\xi_n} [A_n^2] \n    &\\leq 4 \\sigma^2 \\min\\{ \\Phi^{-2}, \\alpha_n^2 \\} \\| w^n - w^*\\|^2 \\qquad \\text{and}\\\\\n    \\E_{\\xi_n} [B_n^2] \n    &\\leq 4 \\Phi^4 \\min \\big\\{ \\big(\\E_{\\xi_n} \\big[ \\|\\iota \\nabla f_n(w^n)\\|^{-2}\\big]\\big)^2 , \n    \\alpha_n^4 \\big\\}\n    + 16 \\min \\big\\{1, 4 \\alpha_n^4 L_4^4 \\|w^n - w^*\\|^4\\big\\}\\\\\n    &\\quad + 16 \\min \\big\\{1,4 \\alpha_n^4 \\sigma_4^4 \\big\\}\n    + 16  \\sigma_4^4 \\min \\big\\{\\Phi^{-4}, \\alpha_n^4 \\big\\} \\|w^n - w^*\\|^4.\n  \\end{align*}\n","sha256_utf8":"4ef1983bec24047602cb98ee25fa099f162b8c708a002243e0fe451c884c3692"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"提升为四阶误差不等式","source_units":[{"start":36929,"end":37474,"start_line":722,"end_line":730,"raw_tex":"   We note that for $a,b \\in \\R$ we have the identity $(a - b)a = \\frac{1}{2} \\big( |a|^2 - |b|^2 \n   + |a - b|^2 \\big)$, and thus $|a|^2 - |b|^2 \\le 2(a-b)a$. By multiplying the inequality from \n   \\eqref{eq:apriori4} with the factor $2\\|w^{n+1} - w^*\\|^2$, we therefore obtain\n  \\begin{align*}\n    \\|w^{n+1} - w^*\\|^4 - \\|w^n - w^*\\|^4\n    &\\leq 2\\big( A_n + B_n \\big) \\|w^{n+1} - w^*\\|^2\\\\\n    &\\leq 2\\big( A_n + B_n \\big) \\big( \\|w^n - w^*\\|^2 + A_n + B_n  \\big)\\\\\n    &\\leq 2(A_n + B_n) \\|w^n - w^*\\|^2  + 4 A_n^2 + 4 B_n^2 \n  \\end{align*}\n","sha256_utf8":"1f059563fc0d28013a4ae9b3c1119bd2c860d0c3f42057a2ce67f01b355ae5de"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"建立四阶矩条件递推","source_units":[{"start":37474,"end":38528,"start_line":731,"end_line":749,"raw_tex":"  and in $\\E_{\\xi_n}$-expectation\n  \\begin{align*}\n    & \\E_{\\xi_n} \\big[ \\|w^{n+1} - w^*\\|^4\\big] - \\|w^n - w^*\\|^4\\\\\n    &\\leq 4 \\Big(2 L^2 \\min \\big\\{\\tfrac{1}{2} L^{-2} \\|w^n - w^*\\|^{-2}, \\alpha_n^2 \\big\\}\n    + \\sigma^2 \\min \\big\\{\\Phi^{-2}, \\alpha_n^2 \\big\\}\\\\\n    &\\qquad + 64 L_4^4 \\min \\big\\{ \\tfrac{1}{4} L_4^{-4} \\|w^n - w^*\\|^{-4}, \\alpha_n^4 \n    \\big\\} + 16 \\sigma_4^4 \\min \\big\\{\\Phi^{-4}, \\alpha_n^4 \\big\\} \\Big)\\|w^n - w^*\\|^4 \\\\\n    &\\quad + 2 \\Big( \\Phi^2 \\min \\big\\{ \\E_{\\xi_n} \\big[ \\|\\iota \\nabla f_n(w^n)\\|^{-2}\\big] , \n    \\alpha_n^2 \\big\\} \n    + 4 \\sigma^2\\min \\big\\{\\tfrac{1}{2} \\sigma^{-2} , \\alpha_n^2 \\big\\} \\\\\n    &\\qquad + 8  \\sigma^2 \\min\\{ \\Phi^{-2}, \\alpha_n^2 \\} \\Big) \\|w^n - w^*\\|^2 \\\\\n    &\\quad + 16 \\Phi^4 \\min \\big\\{ \\big(\\E_{\\xi_n} \\big[ \\|\\iota \\nabla \n    f_n(w^n)\\|^{-2}\\big]\\big)^2 \n    , \\alpha_n^4 \\big\\}\n    + 256 \\sigma_4^4 \\min \\big\\{ \\tfrac{1}{4}\\sigma_4^{-4},\\alpha_n^4 \\big\\}\\\\\n    %\n    &=: c_1^n (\\alpha_n) \\|w^n - w^*\\|^4 + c_2^n(\\alpha_n) \\|w^n - w^*\\|^2 \n    + c_3^n(\\alpha_n).\n  \\end{align*}\n","sha256_utf8":"b03d33b0d9506871aefd7e35ceb13810f21c008156b84c62d4d2f69a4d3e08b4"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"迭代并证明四阶界有限","source_units":[{"start":38528,"end":39382,"start_line":750,"end_line":764,"raw_tex":"  Adding $\\|w^n - w^*\\|^4$ to both sides of the inequality, taking the \n  $\\E_{n-1}$-expectation and reinserting the bound, we obtain\n  \\begin{align*}\n    & \\E_n \\big[ \\|w^{n+1} - w^*\\|^4\\big] \\\\\n    &\\leq \\big(1 + c_1^n(\\alpha_n) \\big) \\E_{n-1} \\big[ \\|w^n - w^*\\|^4 \\big]\n    + c_2^n(\\alpha_n) M_2 + c_3^n(\\alpha_n)\\\\\n    &\\leq \\|w^1 - w^*\\|^4 \\prod_{i = 1}^{n} \\big(1 + c_1^i(\\alpha_i) \\big) \n    + \\sum_{i=1}^{n} \\big(c_2^i(\\alpha_i) M_2 + c_3^i(\\alpha_i)\\big) \\prod_{j = i+ \n    1}^{n} \\big(1 + c_1^j(\\alpha_j) \\big) \\\\\n    &\\leq \\|w^1 - w^*\\|^4 \\exptextB{\\sum_{i = 1}^{\\infty} c_1^i(\\alpha_i) } \n    + \\sum_{i=1}^{\\infty} \\big(c_2^i(\\alpha_i) M_2 + c_3^i(\\alpha_i)\\big) \n    \\exptextB{\\sum_{j = i+ 1}^{\\infty} c_1^j(\\alpha_j)} .\n  \\end{align*}\n  Finally, this is finite due to the assumption $\\sum_{n  =1}^{\\infty} \\alpha_n^2 < \\infty$.\n\\end{proof}","sha256_utf8":"bc6dedb0276ebf695ab3175c5aab7bde28565c264a56b4a03ecc278958cfd23c"}]},{"id":"S044","semantic_type":"lemma","kind":"lemma","title":"冻结分母扰动恒等式","source_units":[{"start":41183,"end":41686,"start_line":811,"end_line":822,"raw_tex":"\\begin{lemma}\\label{lem:SecondLipT}\n  Let Assumption~\\ref{ass:fStoch} be fulfilled and let $z \\in H$, $\\alpha \n  \\in (0,\\infty)$ and $\\Xi \\in [0,\\infty)$ be given.\n  It then follows that\n  \\begin{align*}\n    \\Big\\| T_{\\alpha f_n, z}(w) - w + \\frac{\\alpha \n      \\iota \\nabla f_n(w)}{1 + \\alpha \\Xi} \\Big\\|\n    = \\frac{ \\alpha^2 \\big| \\Xi - \\|\\iota \\nabla f_n(z)\\|\\big| \\|\\iota \\nabla f_n(w)\\|}\n    {(1 + \\alpha \\|\\iota \\nabla f_n(z)\\| ) (1 + \\alpha \\Xi )}\n  \\end{align*}\n  for all $w \\in H$.\n\\end{lemma}","sha256_utf8":"93071b9583353288a35373b78c00e750f714f989064c78f63ac9bab1e4f3bdaf"}]},{"id":"S045","semantic_type":"proof","kind":"proof","title":"分母扰动完整证明","source_units":[{"start":41688,"end":42573,"start_line":824,"end_line":839,"raw_tex":"\\begin{proof}\n  Inserting the definition of $T_{\\alpha f_n, z}$, it follows that\n  \\begin{align*}\n    &\\Big\\| T_{\\alpha f_n, z}(w) - w + \\frac{\\alpha \\iota \\nabla f_n(w)}{1 + \\alpha \n    \\Xi} \\Big\\|\n  = \\Big\\| \\frac{\\alpha \\iota \\nabla f_n(w)}{1 + \\alpha \\|\\iota \\nabla f_n(z)\\|} - \n    \\frac{\\alpha \\iota \\nabla f_n(w)}{1 + \\alpha \\Xi } \\Big\\|\\\\\n    &= \\Big\\| \\frac{\\alpha \\iota \\nabla f_n(w) + \\alpha^2 \\Xi \\iota \\nabla f_n(w) \n    }{(1 + \\alpha \\|\\iota \\nabla f_n(z)\\| ) (1 + \\alpha \\Xi )}\n    - \\frac{\\alpha \\iota \\nabla f_n(w) + \\alpha^2 \\|\\iota \\nabla f_n(z)\\| \\iota \\nabla \n    f_n(w) }{ (1 + \\alpha \\|\\iota \\nabla f_n(z)\\| ) (1 + \\alpha \\Xi ) } \\Big\\|\\\\\n    &= \\frac{ \\alpha^2 \\big| \\Xi - \\|\\iota \\nabla f_n(z)\\|\\big| \\|\\iota \\nabla f_n(w)\\|}\n    {(1 + \\alpha \\|\\iota \\nabla f_n(z)\\| ) (1 + \\alpha \\Xi )},\n  \\end{align*}\n  for all $w \\in H$, which proves the claim.\n\\end{proof}","sha256_utf8":"a80fe403c46a6cba885223c1d035d021bc33783e3c4d0bf320a339c662626897"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"通分计算扰动范数","source_units":[{"start":41688,"end":42573,"start_line":824,"end_line":839,"raw_tex":"\\begin{proof}\n  Inserting the definition of $T_{\\alpha f_n, z}$, it follows that\n  \\begin{align*}\n    &\\Big\\| T_{\\alpha f_n, z}(w) - w + \\frac{\\alpha \\iota \\nabla f_n(w)}{1 + \\alpha \n    \\Xi} \\Big\\|\n  = \\Big\\| \\frac{\\alpha \\iota \\nabla f_n(w)}{1 + \\alpha \\|\\iota \\nabla f_n(z)\\|} - \n    \\frac{\\alpha \\iota \\nabla f_n(w)}{1 + \\alpha \\Xi } \\Big\\|\\\\\n    &= \\Big\\| \\frac{\\alpha \\iota \\nabla f_n(w) + \\alpha^2 \\Xi \\iota \\nabla f_n(w) \n    }{(1 + \\alpha \\|\\iota \\nabla f_n(z)\\| ) (1 + \\alpha \\Xi )}\n    - \\frac{\\alpha \\iota \\nabla f_n(w) + \\alpha^2 \\|\\iota \\nabla f_n(z)\\| \\iota \\nabla \n    f_n(w) }{ (1 + \\alpha \\|\\iota \\nabla f_n(z)\\| ) (1 + \\alpha \\Xi ) } \\Big\\|\\\\\n    &= \\frac{ \\alpha^2 \\big| \\Xi - \\|\\iota \\nabla f_n(z)\\|\\big| \\|\\iota \\nabla f_n(w)\\|}\n    {(1 + \\alpha \\|\\iota \\nabla f_n(z)\\| ) (1 + \\alpha \\Xi )},\n  \\end{align*}\n  for all $w \\in H$, which proves the claim.\n\\end{proof}","sha256_utf8":"a80fe403c46a6cba885223c1d035d021bc33783e3c4d0bf320a339c662626897"}]},{"id":"S047","semantic_type":"lemma","kind":"lemma","title":"条件均方误差单步界","source_units":[{"start":42576,"end":43583,"start_line":842,"end_line":862,"raw_tex":"\\begin{lemma} \\label{lem:boundForTheo}\n  Let Assumption~\\ref{ass:fStoch} be fulfilled and let \n  $\\{\\alpha_n\\}_{n \n  \\in \\N}$  be a sequence of positive real numbers.\n   For any $\\Xi \\in [0,\\infty)$ it then follows that\n  \\begin{align*}\n    \\E_{\\xi_n} \\big[\\| w^{n+1} - w^* \\|^2\\big]\n    &\\leq \\Big( 1 - \\E_{\\xi_n} \\Big[ \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \\nabla \n    f_n(w^n)\\| } \\Big] \\Big) \\| w^n - w^*\\|^2\\\\\n    &\\quad + 2 \\E_{\\xi_n} \\Big[ \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla \n    f_n(w^*)\\|^2}{(1 + \\alpha_n \\|\\iota \\nabla \n      f_n(w^n)\\| )^2} \\Big]\\\\\n    &\\quad + 2 \\E_{\\xi_n} \\Big[ \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n (w^*)\\|^2 }{( 1 \n    + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| )^2} \\Big]\\\\\n    &\\quad + 2\\E_{\\xi_n} \\Big[ \\frac{ \\alpha_n^2 \\big| \\Xi - \\|\\iota  \\nabla f_n(w^n)\\| \\big| \n    \\|\\iota \\nabla f_n(w^*)\\|}\n    {(1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| ) (1 + \\alpha_n \\Xi )} \\Big] \n    \\| w^n - w^* \\| \n  \\end{align*}\n  for every $n \\in \\N$.\n\\end{lemma}","sha256_utf8":"5c49f1f9f1eaf094f8020f00e47b23f39bdb43f73b80b1bc1fd3b58afb62c3cf"}]},{"id":"S048","semantic_type":"proof","kind":"proof","title":"单步误差界完整证明","source_units":[{"start":43586,"end":48980,"start_line":865,"end_line":972,"raw_tex":"\\begin{proof}\n  From $w^{n+1} = T_{\\alpha_n f_n, w^n}(w^n)$ we obtain that\n  \\begin{align*}\n    \\| w^{n+1} - w^* \\|^2\n    &= \\| T_{\\alpha_n f_n, w^n}(w^n) - T_{\\alpha_n f_n, w^n}(w^*) + T_{\\alpha_n f_n, \n      w^n}(w^*) - w^*\\|^2\\\\\n    &= \\| T_{\\alpha_n f_n, w^n}(w^n) - T_{\\alpha_n f_n, w^n}(w^*)\\|^2 + \\|T_{\\alpha_n f_n, \n      w^n}(w^*) - w^*\\|^2\\\\\n    &\\quad + 2 \\dual{T_{\\alpha_n f_n, w^n}(w^n) - T_{\\alpha_n f_n, w^n}(w^*)}{T_{\\alpha_n \n        f_n, w^n}(w^*) - w^*}\\\\\n    &=: I_1 + I_2 + 2 I_3.\n  \\end{align*}\n  For $I_1$, we can write\n  \\begin{align*}\n    I_1&= \\| T_{\\alpha_n f_n, w^n}(w^n) - T_{\\alpha_n f_n, w^n}(w^*)\\|^2\\\\\n    &= \\big\\| w^n - w^* + (T_{\\alpha_n f_n, w^n} - I)(w^n) -\n    (T_{\\alpha_n f_n, w^n} - I)(w^*)\\big\\|^2\\\\\n    &= \\| w^n - w^*\\|^2\n    + 2 \\dualb{w^n - w^*}{(T_{\\alpha_n f_n, w^n} - I)(w^n) - (T_{\\alpha_n f_n, w^n} - I)(w^*) } \n    \\\\\n    &\\quad + \\| (T_{\\alpha_n f_n, w^n} - I)(w^n) - (T_{\\alpha_n f_n, w^n} - I)(w^*) \\|^2\n    =: I_{1,1} + 2I_{1,2} + I_{1,3}.\n  \\end{align*}\n  The term $I_{1,2}$ can be estimated by applying the Cauchy--Schwarz inequality:\n  \\begin{align*}\n    I_{1,2}&= \\dualb{w^n - w^*}{ (T_{\\alpha_n f_n, w^n} - I)(w^n) - (T_{\\alpha_n f_n, w^n} - \n      I)(w^*) }\\\\\n    &= - \\dualB{w^n - w^*}{\\frac{\\alpha_n \\iota \\nabla f_n(w^n)}{1 + \\alpha_n \\|\\iota \n    \\nabla f_n(w^n)\\|} \n      - \\frac{\\alpha_n \\iota \\nabla f_n(w^*)}{1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\|}}\\\\\n    &\\leq - \\frac{\\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| } \\|w^n - \n    w^*\\|^2.\n  \\end{align*}\n  For $I_{1,3}$, we insert the definition of $T_{\\alpha_n f_n, w^n}$ and find\n  \\begin{align*}\n    I_{1,3} \n    &= \\| (T_{\\alpha_n f_n, w^n} - I)(w^n) - (T_{\\alpha_n f_n, w^n} - I)(w^*) \\|^2\\\\\n    &=  \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla f_n(w^*)\\|^2 }{ (1 + \\alpha \n    \\|\\iota \\nabla f_n(w^n)\\| )^2}.\n  \\end{align*}\n  Thus, for $I_1$, we have\n  \\begin{align*}\n    I_1\n    \\leq \\Big( 1 - \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| } \n    \\Big)\\| w^n - w^*\\|^2\n    + \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla f_n(w^*)\\|^2 }{ (1 + \\alpha \n      \\|\\iota \\nabla f_n(w^n)\\| )^2}.\n  \\end{align*}\n  Further, $I_2$ can be written as\n  \\begin{align*}\n    I_2\n    = \\|T_{\\alpha_n f_n, w^n}(w^*) - w^*\\|^2\n    = \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n (w^*)\\|^2 }{( 1 + \\alpha_n \\|\\iota \\nabla \n      f_n(w^n)\\| )^2}.\n  \\end{align*}\n  Finally, $I_3$ can be rewritten as\n  \\begin{align*}\n    I_3 &= \\dual{T_{\\alpha_n f_n, w^n}(w^n) - T_{\\alpha_n f_n, w^n}(w^*)}{T_{\\alpha_n f_n, \n        w^n}(w^*) - w^*}\\\\\n    &= \\dual{(T_{\\alpha_n f_n, w^n} - I) (w^n) - (T_{\\alpha_n f_n, w^n} - I) \n      (w^*)}{(T_{\\alpha_n f_n, w^n} - I) (w^*)}\\\\\n    &\\quad + \\dual{w^n - w^*}{(T_{\\alpha_n f_n, w^n} - I) (w^*)}\n    =: I_{3,1} + I_{3,2}.\n  \\end{align*}\n  Then for $I_{3,1}$, we insert the definition of $T_{\\alpha_n f_n, w^n}$ and obtain\n  \\begin{align*}\n    I_{3,1} &= \\dual{(T_{\\alpha_n f_n, w^n} - I) (w^n) - (T_{\\alpha_n f_n, w^n} - I) \n      (w^*)}{(T_{\\alpha_n f_n, w^n} - I) (w^*)}\\\\\n    &\\leq \\frac{\\alpha_n \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla f_n(w^*)\\|}{1 + \\alpha_n \\|\\iota \n    \\nabla f_n(w^n)\\|} \\cdot\n    \\frac{\\alpha_n \\| \\iota \\nabla f_n (w^*)\\|}{1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\|} \\\\\n    &\\leq \\frac{1}{2}\\frac{\\alpha_n^2 \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla f_n(w^*)\\|^2}{ (1 + \n    \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| )^2} \n    + \\frac{1}{2} \\frac{\\alpha_n^2 \\| \\iota \\nabla f_n (w^*)\\|^2}{(1 + \\alpha_n \\|\\iota \n      \\nabla f_n(w^n)\\| )^2},\n  \\end{align*}\n  where we applied the Cauchy--Schwarz inequality and Young's inequality.\n  To estimate $I_{3,2}$, we add and subtract an additional summand, so that\n  \\begin{align*}\n    I_{3,2} &= \\dual{w^n - w^*}{(T_{\\alpha_n f_n, w^n} - I) (w^*)}\\\\\n    &= \\dualB{w^n - w^*}{(T_{\\alpha_n f_n, w^n} - I) (w^*) + \\frac{\\alpha_n \\iota \\nabla \n    f_n(w^*)}{1 + \\alpha_n \\Xi }}\n    - \\dualB{w^n - w^*}{\\frac{\\alpha_n \\iota \\nabla f_n(w^*)}{1 + \\alpha_n \\Xi }},\n  \\end{align*}\n  where $\\E_{\\xi_n}\\big[ \\dualb{w^n - w^*}{\\frac{\\alpha_n \\iota \\nabla f_n(w^*)}{1 \n  + \\alpha_n \\Xi }} \\big] = 0$ is fulfilled. Moreover, applying \n  Lemma~\\ref{lem:SecondLipT} and the Cauchy--Schwarz inequality, we obtain\n  \\begin{align*}\n    &\\dualB{w^n - w^*}{(T_{\\alpha_n f_n, w^n} - I) (w^*) + \\frac{\\alpha_n \\iota \\nabla \n        f_n(w^*)}{1 + \\alpha_n \\Xi }}\\\\\n    &\\leq \\frac{ \\alpha_n^2 \\big|\\Xi - \\|\\iota \\nabla f_n(w^n)\\| \\big| \\|\\iota \\nabla f_n(w^*)\\|}\n    {(1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| ) (1 + \\alpha_n \\Xi )}  \\| w^n \n    - w^* \\|.\n  \\end{align*}\n  Thus, the expectation of $I_3 = I_{3,1} + I_{3,2}$ can be bounded by\n  \\begin{align*}\n    \\E_{\\xi_n} [I_3 ]\n    &\\leq \\frac{1}{2} \\E_{\\xi_n} \\Big[ \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla \n    f_n(w^*)\\|^2 }{\\big(1 + \\alpha_n \\|\\iota \\nabla \n      f_n(w^n)\\|\\big)^2} + \\frac{\\alpha_n^2 \\| \\iota \\nabla f_n \n      (w^*)\\|^2}{ (1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| )^2}\\Big]\\\\\n    &\\quad + \\E_{\\xi_n} \\Big[ \\frac{ \\alpha_n^2 \\big| \\Xi - \\|\\iota \\nabla f_n(w^n)\\| \\big| \\|\\iota \n    \\nabla f_n(w^*)\\| }\n    {(1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| ) (1 + \\alpha_n \\Xi )}\\Big]  \\| w^n - w^* \\| .\n  \\end{align*}\n  Inserting the bounds for $I_1$, $I_2$ and $I_3$ into $\\E_{\\xi_n} \\big[\\| w^{n+1} - w^* \n  \\|^2\\big] = \\E_{\\xi_n} \\big[ I_1 + I_2 + 2 I_3\\big]$ finishes the proof.\n\\end{proof}","sha256_utf8":"75a6e181c77c82a357b86fc8f21ce3aa78f67f438803940853d88c321dbd9cbe"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"误差平方分为三项","source_units":[{"start":43586,"end":44103,"start_line":865,"end_line":876,"raw_tex":"\\begin{proof}\n  From $w^{n+1} = T_{\\alpha_n f_n, w^n}(w^n)$ we obtain that\n  \\begin{align*}\n    \\| w^{n+1} - w^* \\|^2\n    &= \\| T_{\\alpha_n f_n, w^n}(w^n) - T_{\\alpha_n f_n, w^n}(w^*) + T_{\\alpha_n f_n, \n      w^n}(w^*) - w^*\\|^2\\\\\n    &= \\| T_{\\alpha_n f_n, w^n}(w^n) - T_{\\alpha_n f_n, w^n}(w^*)\\|^2 + \\|T_{\\alpha_n f_n, \n      w^n}(w^*) - w^*\\|^2\\\\\n    &\\quad + 2 \\dual{T_{\\alpha_n f_n, w^n}(w^n) - T_{\\alpha_n f_n, w^n}(w^*)}{T_{\\alpha_n \n        f_n, w^n}(w^*) - w^*}\\\\\n    &=: I_1 + I_2 + 2 I_3.\n  \\end{align*}\n","sha256_utf8":"b19d9a881afeeb8624e477cc06eaffecd444f35bf17923796b4ba35d032a2ee6"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"展开映射差平方","source_units":[{"start":44103,"end":44598,"start_line":877,"end_line":887,"raw_tex":"  For $I_1$, we can write\n  \\begin{align*}\n    I_1&= \\| T_{\\alpha_n f_n, w^n}(w^n) - T_{\\alpha_n f_n, w^n}(w^*)\\|^2\\\\\n    &= \\big\\| w^n - w^* + (T_{\\alpha_n f_n, w^n} - I)(w^n) -\n    (T_{\\alpha_n f_n, w^n} - I)(w^*)\\big\\|^2\\\\\n    &= \\| w^n - w^*\\|^2\n    + 2 \\dualb{w^n - w^*}{(T_{\\alpha_n f_n, w^n} - I)(w^n) - (T_{\\alpha_n f_n, w^n} - I)(w^*) } \n    \\\\\n    &\\quad + \\| (T_{\\alpha_n f_n, w^n} - I)(w^n) - (T_{\\alpha_n f_n, w^n} - I)(w^*) \\|^2\n    =: I_{1,1} + 2I_{1,2} + I_{1,3}.\n  \\end{align*}\n","sha256_utf8":"5638e30c95985a9337467cd35eeda668b276b09bc9bfebf480724a2438561cee"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"单调性给出负漂移","source_units":[{"start":44598,"end":45128,"start_line":888,"end_line":897,"raw_tex":"  The term $I_{1,2}$ can be estimated by applying the Cauchy--Schwarz inequality:\n  \\begin{align*}\n    I_{1,2}&= \\dualb{w^n - w^*}{ (T_{\\alpha_n f_n, w^n} - I)(w^n) - (T_{\\alpha_n f_n, w^n} - \n      I)(w^*) }\\\\\n    &= - \\dualB{w^n - w^*}{\\frac{\\alpha_n \\iota \\nabla f_n(w^n)}{1 + \\alpha_n \\|\\iota \n    \\nabla f_n(w^n)\\|} \n      - \\frac{\\alpha_n \\iota \\nabla f_n(w^*)}{1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\|}}\\\\\n    &\\leq - \\frac{\\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| } \\|w^n - \n    w^*\\|^2.\n  \\end{align*}\n","sha256_utf8":"8470b92b1a432d95f4e05e04b6eb10f273dfe664a14682b15f29b73bfd325639"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"计算梯度差平方项","source_units":[{"start":45128,"end":45465,"start_line":898,"end_line":904,"raw_tex":"  For $I_{1,3}$, we insert the definition of $T_{\\alpha_n f_n, w^n}$ and find\n  \\begin{align*}\n    I_{1,3} \n    &= \\| (T_{\\alpha_n f_n, w^n} - I)(w^n) - (T_{\\alpha_n f_n, w^n} - I)(w^*) \\|^2\\\\\n    &=  \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla f_n(w^*)\\|^2 }{ (1 + \\alpha \n    \\|\\iota \\nabla f_n(w^n)\\| )^2}.\n  \\end{align*}\n","sha256_utf8":"3fb826f83782d96166b937c58aeec2c025da9a4c7e8d93045528ba9adc9f57bf"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"汇总映射差收缩界","source_units":[{"start":45465,"end":45779,"start_line":905,"end_line":912,"raw_tex":"  Thus, for $I_1$, we have\n  \\begin{align*}\n    I_1\n    \\leq \\Big( 1 - \\frac{2 \\alpha_n \\mu_{\\xi_n} }{1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| } \n    \\Big)\\| w^n - w^*\\|^2\n    + \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla f_n(w^*)\\|^2 }{ (1 + \\alpha \n      \\|\\iota \\nabla f_n(w^n)\\| )^2}.\n  \\end{align*}\n","sha256_utf8":"670e5a5679833b87c1fd903f992180971b489faf1dc912567327ccef2af0289c"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"计算最优点噪声项","source_units":[{"start":45779,"end":46007,"start_line":913,"end_line":919,"raw_tex":"  Further, $I_2$ can be written as\n  \\begin{align*}\n    I_2\n    = \\|T_{\\alpha_n f_n, w^n}(w^*) - w^*\\|^2\n    = \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n (w^*)\\|^2 }{( 1 + \\alpha_n \\|\\iota \\nabla \n      f_n(w^n)\\| )^2}.\n  \\end{align*}\n","sha256_utf8":"2f6a05d2c49781cea146e1c81a6b3f0920a4d0059c5e08be36be2c96b870937c"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"拆分误差交叉项","source_units":[{"start":46007,"end":46414,"start_line":920,"end_line":928,"raw_tex":"  Finally, $I_3$ can be rewritten as\n  \\begin{align*}\n    I_3 &= \\dual{T_{\\alpha_n f_n, w^n}(w^n) - T_{\\alpha_n f_n, w^n}(w^*)}{T_{\\alpha_n f_n, \n        w^n}(w^*) - w^*}\\\\\n    &= \\dual{(T_{\\alpha_n f_n, w^n} - I) (w^n) - (T_{\\alpha_n f_n, w^n} - I) \n      (w^*)}{(T_{\\alpha_n f_n, w^n} - I) (w^*)}\\\\\n    &\\quad + \\dual{w^n - w^*}{(T_{\\alpha_n f_n, w^n} - I) (w^*)}\n    =: I_{3,1} + I_{3,2}.\n  \\end{align*}\n","sha256_utf8":"b64ca0da89252554b25d7a89f357d556e76ab26e882bee4f851037ca44efb721"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"杨氏不等式控制交叉项","source_units":[{"start":46414,"end":47221,"start_line":929,"end_line":941,"raw_tex":"  Then for $I_{3,1}$, we insert the definition of $T_{\\alpha_n f_n, w^n}$ and obtain\n  \\begin{align*}\n    I_{3,1} &= \\dual{(T_{\\alpha_n f_n, w^n} - I) (w^n) - (T_{\\alpha_n f_n, w^n} - I) \n      (w^*)}{(T_{\\alpha_n f_n, w^n} - I) (w^*)}\\\\\n    &\\leq \\frac{\\alpha_n \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla f_n(w^*)\\|}{1 + \\alpha_n \\|\\iota \n    \\nabla f_n(w^n)\\|} \\cdot\n    \\frac{\\alpha_n \\| \\iota \\nabla f_n (w^*)\\|}{1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\|} \\\\\n    &\\leq \\frac{1}{2}\\frac{\\alpha_n^2 \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla f_n(w^*)\\|^2}{ (1 + \n    \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| )^2} \n    + \\frac{1}{2} \\frac{\\alpha_n^2 \\| \\iota \\nabla f_n (w^*)\\|^2}{(1 + \\alpha_n \\|\\iota \n      \\nabla f_n(w^n)\\| )^2},\n  \\end{align*}\n  where we applied the Cauchy--Schwarz inequality and Young's inequality.\n","sha256_utf8":"dc557bb7421ed67154bc8413076d2a9289eb8d5250d8e8a3b12925607315c55d"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"中心化并估计分母扰动","source_units":[{"start":47221,"end":48178,"start_line":942,"end_line":958,"raw_tex":"  To estimate $I_{3,2}$, we add and subtract an additional summand, so that\n  \\begin{align*}\n    I_{3,2} &= \\dual{w^n - w^*}{(T_{\\alpha_n f_n, w^n} - I) (w^*)}\\\\\n    &= \\dualB{w^n - w^*}{(T_{\\alpha_n f_n, w^n} - I) (w^*) + \\frac{\\alpha_n \\iota \\nabla \n    f_n(w^*)}{1 + \\alpha_n \\Xi }}\n    - \\dualB{w^n - w^*}{\\frac{\\alpha_n \\iota \\nabla f_n(w^*)}{1 + \\alpha_n \\Xi }},\n  \\end{align*}\n  where $\\E_{\\xi_n}\\big[ \\dualb{w^n - w^*}{\\frac{\\alpha_n \\iota \\nabla f_n(w^*)}{1 \n  + \\alpha_n \\Xi }} \\big] = 0$ is fulfilled. Moreover, applying \n  Lemma~\\ref{lem:SecondLipT} and the Cauchy--Schwarz inequality, we obtain\n  \\begin{align*}\n    &\\dualB{w^n - w^*}{(T_{\\alpha_n f_n, w^n} - I) (w^*) + \\frac{\\alpha_n \\iota \\nabla \n        f_n(w^*)}{1 + \\alpha_n \\Xi }}\\\\\n    &\\leq \\frac{ \\alpha_n^2 \\big|\\Xi - \\|\\iota \\nabla f_n(w^n)\\| \\big| \\|\\iota \\nabla f_n(w^*)\\|}\n    {(1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| ) (1 + \\alpha_n \\Xi )}  \\| w^n \n    - w^* \\|.\n  \\end{align*}\n","sha256_utf8":"696ffc175383a1512d1ad5c16480e9d6f73541d1a08746db3fff7015e5681436"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"汇总条件期望误差界","source_units":[{"start":48178,"end":48980,"start_line":959,"end_line":972,"raw_tex":"  Thus, the expectation of $I_3 = I_{3,1} + I_{3,2}$ can be bounded by\n  \\begin{align*}\n    \\E_{\\xi_n} [I_3 ]\n    &\\leq \\frac{1}{2} \\E_{\\xi_n} \\Big[ \\frac{\\alpha_n^2 \\|\\iota \\nabla f_n(w^n) - \\iota \\nabla \n    f_n(w^*)\\|^2 }{\\big(1 + \\alpha_n \\|\\iota \\nabla \n      f_n(w^n)\\|\\big)^2} + \\frac{\\alpha_n^2 \\| \\iota \\nabla f_n \n      (w^*)\\|^2}{ (1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| )^2}\\Big]\\\\\n    &\\quad + \\E_{\\xi_n} \\Big[ \\frac{ \\alpha_n^2 \\big| \\Xi - \\|\\iota \\nabla f_n(w^n)\\| \\big| \\|\\iota \n    \\nabla f_n(w^*)\\| }\n    {(1 + \\alpha_n \\|\\iota \\nabla f_n(w^n)\\| ) (1 + \\alpha_n \\Xi )}\\Big]  \\| w^n - w^* \\| .\n  \\end{align*}\n  Inserting the bounds for $I_1$, $I_2$ and $I_3$ into $\\E_{\\xi_n} \\big[\\| w^{n+1} - w^* \n  \\|^2\\big] = \\E_{\\xi_n} \\big[ I_1 + I_2 + 2 I_3\\big]$ finishes the proof.\n\\end{proof}","sha256_utf8":"6f4f2f46fa37160ddf0b99084a04f003343610d8f5673f32a0ae5f9e4013ad3f"}]},{"id":"S059","semantic_type":"lemma","kind":"lemma","title":"收缩乘积与加权和界","source_units":[{"start":81325,"end":82027,"start_line":1567,"end_line":1584,"raw_tex":"\\begin{lemma} \\label{lem:algebraic_inequalities}\n  Let $x, y \\in (0,\\infty)$ and $n,m \\in \\N$ be given such that $\\frac{x}{1 + y} \\leq 1$. Then \n  the following inequalities \n  are satisfied:\n  \\begin{enumerate}[label=(\\roman*)] \n    \\item $\\prod_{i= m}^n \\big(1-\\frac{x}{i + y}\\big) \\le \n    \\big(\\frac{n+1+ y}{m+y}\\big)^{-x}$,\\\\\n    \\item \n    $\\sum_{i=1}^n \\frac{1}{(i+y)^2} \\prod_{j =i+1}^n \\big(1-\\frac{x}{j+y} \\big)\\\\\n    \\leq \\exptextb{\\frac{x}{1+y}}\n    \\begin{cases}\n      (n+1+y)^{-1}\\frac{1}{x-1}, &x \\in (1,\\infty),\\\\\n      (n+1+y)^{-1} \\big(1 + \\ln{(n+y)}\\big), &x = 1,\\\\\n      (n+1+y)^{-x} \\frac{(1+y)^{x-2}(x-2-y)}{x-1},\n      &x \\in [0,1).\n    \\end{cases}$\n  \\end{enumerate}\n\\end{lemma}","sha256_utf8":"88dd459a0888b1506a866808f02cd6a3fa0d01af1e8b9005165c0b7c938db303"}]},{"id":"S060","semantic_type":"proof","kind":"proof","title":"乘积求和界完整证明","source_units":[{"start":82029,"end":84214,"start_line":1586,"end_line":1636,"raw_tex":"\\begin{proof}\n  In this proof, we apply the following basic inequalities involving (generalized) \n  harmonic numbers\n  \\begin{align*}\n    \\sum_{i=m}^n (i+y)^{-1} &\\geq \\ln{(n+1+y)} - \\ln{(m+y)}, \n    \\quad m \\in \\{1,\\dots,n\\},\\\\\n    \\sum_{i=1}^n{ (i+y)^{p}}\n    &\\leq \n    \\begin{cases}\n      \\frac{(n+1+y)^{p+1}}{p+1}, &p \\in [0,\\infty) ,\\\\\n      \\frac{(n+y)^{p+1}}{p+1}, &p \\in (-1,0),\\\\\n      1 + \\ln{(n+y)}, &p = -1,\\\\\n      \\frac{(1+y)^{p}(p-y)}{p+1}, &p \\in (-\\infty,-1),\n    \\end{cases}\n  \\end{align*}\n  for $y \\in (0,\\infty)$.\n  These inequalities follow by treating the sums as a lower or upper Riemann \n  sums approximating the integral $\\int {(u+y)^p \\diff{u}}$ over the intervals $[0,n]$, $[1,n]$ \n  or $[0,n+1]$. \n  \n  Using the inequality $1 + u \\le \\exp{u}$ for $u \\in [-1, \\infty)$, it follows that $0\\leq \n  1-\\frac{x}{i + y} \\le \\exptext{-\\frac{x}{i+y}}$ is fulfilled for every $i \\in \\N$ since \n  $\\frac{x}{1+y} \\leq 1$. It then follows that\n  \\begin{align*}\n    \\prod_{i= m}^n \\Big(1- \\frac{x }{i + y}\\Big) \n    &\\leq \\exptextB{- x \\sum_{i=m}^{n} (i + y)^{-1}} \\\\\n    &\\leq \\exptextb{- x \\big(\\ln{(n+1+y)} - \\ln(m+y)\\big)} \\\\\n    &= \\exptextB{- x \\ln{\\Big(\\frac{n+1+y}{m+y} \\Big)} } \n    = \\Big(\\frac{n+1+y}{m+y} \\Big)^{-x}\n  \\end{align*}\n  from which the first claim follows directly. \n  For the second claim, we use the fact that $\\frac{i+1+y}{i+y} = 1 + \\frac{1}{i+y} \\leq  1 + \n  \\frac{1}{1+y}\n  \\leq \\exptext{\\frac{1}{1+y}}$ for all $i \\in \\N$ and find that\n  \\begin{align*}\n    &\\sum_{i=1}^n \\frac{1}{(i+y)^2} \\prod_{j = i+1}^n \\Big(1-\\frac{x}{j+y} \n    \\Big)\n    \\le \\sum_{i=1}^n { \\frac{1}{(i+y)^2} \\Big(\\frac{n+1+y}{i+1+y}\\Big)^{-x} } \\\\\n    &\\le (n+1+y)^{-x} \\sum_{i=1}^n{ \\Big(\\frac{i+1+y}{i+y}\\Big)^{x} (i+y)^{x-2} }\\\\\n    &\\le  \\exptextB{\\frac{x}{1+y}} (n+1+y)^{-x} \n    \\sum_{i=1}^n{ (i+y)^{x-2} }\\\\\n    &\\leq \\exptextB{\\frac{x}{1+y}}\n    \\begin{cases}\n      (n+1+y)^{-1}\\frac{1}{x-1}, &x \\in (1,\\infty) ,\\\\\n      (n+1+y)^{-1} \\big(1 + \\ln{(n+y)}\\big), &x = 1,\\\\\n      (n+1+y)^{-x}\\frac{(1+y)^{x-2}(x-2-y)}{x-1}, \n      &x \\in [0,1),\n    \\end{cases}\n  \\end{align*}\n  where we applied the basic inequalities from the beginning of the proof.\n\\end{proof}","sha256_utf8":"bd02aa6f01e43c5ff23b5724ae1c95ded415ed1e13e10700e3c0331a00306687"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"积分比较估计调和和","source_units":[{"start":82029,"end":82756,"start_line":1586,"end_line":1604,"raw_tex":"\\begin{proof}\n  In this proof, we apply the following basic inequalities involving (generalized) \n  harmonic numbers\n  \\begin{align*}\n    \\sum_{i=m}^n (i+y)^{-1} &\\geq \\ln{(n+1+y)} - \\ln{(m+y)}, \n    \\quad m \\in \\{1,\\dots,n\\},\\\\\n    \\sum_{i=1}^n{ (i+y)^{p}}\n    &\\leq \n    \\begin{cases}\n      \\frac{(n+1+y)^{p+1}}{p+1}, &p \\in [0,\\infty) ,\\\\\n      \\frac{(n+y)^{p+1}}{p+1}, &p \\in (-1,0),\\\\\n      1 + \\ln{(n+y)}, &p = -1,\\\\\n      \\frac{(1+y)^{p}(p-y)}{p+1}, &p \\in (-\\infty,-1),\n    \\end{cases}\n  \\end{align*}\n  for $y \\in (0,\\infty)$.\n  These inequalities follow by treating the sums as a lower or upper Riemann \n  sums approximating the integral $\\int {(u+y)^p \\diff{u}}$ over the intervals $[0,n]$, $[1,n]$ \n  or $[0,n+1]$. \n","sha256_utf8":"ea9904251d1cb280508619019668ce6c5c643decac4df869c623e55a1ab6ffef"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"指数估计收缩乘积","source_units":[{"start":82759,"end":83336,"start_line":1606,"end_line":1616,"raw_tex":"  Using the inequality $1 + u \\le \\exp{u}$ for $u \\in [-1, \\infty)$, it follows that $0\\leq \n  1-\\frac{x}{i + y} \\le \\exptext{-\\frac{x}{i+y}}$ is fulfilled for every $i \\in \\N$ since \n  $\\frac{x}{1+y} \\leq 1$. It then follows that\n  \\begin{align*}\n    \\prod_{i= m}^n \\Big(1- \\frac{x }{i + y}\\Big) \n    &\\leq \\exptextB{- x \\sum_{i=m}^{n} (i + y)^{-1}} \\\\\n    &\\leq \\exptextb{- x \\big(\\ln{(n+1+y)} - \\ln(m+y)\\big)} \\\\\n    &= \\exptextB{- x \\ln{\\Big(\\frac{n+1+y}{m+y} \\Big)} } \n    = \\Big(\\frac{n+1+y}{m+y} \\Big)^{-x}\n  \\end{align*}\n  from which the first claim follows directly. \n","sha256_utf8":"502b16a157f03580e6f4fb1da3c25fcec28590e4d2109d0c87d63dcd4cda6412"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"分情形估计加权和","source_units":[{"start":83336,"end":84214,"start_line":1617,"end_line":1636,"raw_tex":"  For the second claim, we use the fact that $\\frac{i+1+y}{i+y} = 1 + \\frac{1}{i+y} \\leq  1 + \n  \\frac{1}{1+y}\n  \\leq \\exptext{\\frac{1}{1+y}}$ for all $i \\in \\N$ and find that\n  \\begin{align*}\n    &\\sum_{i=1}^n \\frac{1}{(i+y)^2} \\prod_{j = i+1}^n \\Big(1-\\frac{x}{j+y} \n    \\Big)\n    \\le \\sum_{i=1}^n { \\frac{1}{(i+y)^2} \\Big(\\frac{n+1+y}{i+1+y}\\Big)^{-x} } \\\\\n    &\\le (n+1+y)^{-x} \\sum_{i=1}^n{ \\Big(\\frac{i+1+y}{i+y}\\Big)^{x} (i+y)^{x-2} }\\\\\n    &\\le  \\exptextB{\\frac{x}{1+y}} (n+1+y)^{-x} \n    \\sum_{i=1}^n{ (i+y)^{x-2} }\\\\\n    &\\leq \\exptextB{\\frac{x}{1+y}}\n    \\begin{cases}\n      (n+1+y)^{-1}\\frac{1}{x-1}, &x \\in (1,\\infty) ,\\\\\n      (n+1+y)^{-1} \\big(1 + \\ln{(n+y)}\\big), &x = 1,\\\\\n      (n+1+y)^{-x}\\frac{(1+y)^{x-2}(x-2-y)}{x-1}, \n      &x \\in [0,1),\n    \\end{cases}\n  \\end{align*}\n  where we applied the basic inequalities from the beginning of the proof.\n\\end{proof}","sha256_utf8":"03366c212f433758554835be1adaff096b16a27ddcd27f7a0ee99a4273f7aacb"}]},{"id":"S064","semantic_type":"lemma","kind":"lemma","title":"负倒数切线上界","source_units":[{"start":84216,"end":84384,"start_line":1638,"end_line":1641,"raw_tex":"\\begin{lemma} \\label{lem:TaylorEx}\n  Given $a,b \\in (0,\\infty)$, $\\frac{-1}{a x + b} \\leq - \\frac{1}{b} + \\frac{a}{b^2} \n  x$ for every $x \\in (0,\\infty)$. \n\\end{lemma}","sha256_utf8":"f1a201e0354facc08495ac9a45e294acd20fff4c2a27491f7a781ce77f8f9468"}]},{"id":"S065","semantic_type":"proof","kind":"proof","title":"负倒数界完整证明","source_units":[{"start":84386,"end":84876,"start_line":1643,"end_line":1653,"raw_tex":"\\begin{proof}\n  We consider the function $f \\colon [0,\\infty) \\to \\R$ with $f(x) = \\frac{-1}{a x + b}$. \n  Then the first and second derivative of $f$ are given by $f'(x) = \\frac{a}{(a x + b)^2}$ and \n  $f''(x) = \\frac{-2a^2}{(a x + b)^3}$. Using a first-order Taylor expansion of $f$ then shows \n  that\n  \\begin{align*}\n    f(x) = - \\frac{1}{b} + \\frac{a}{b^2} x - \\frac{a^2}{(a \\xi + b)^3} x^2\n    \\leq - \\frac{1}{b} + \\frac{a}{b^2} x,\n  \\end{align*}\n  where $\\xi \\in (0,x)$. \n\\end{proof}","sha256_utf8":"f19b51fd6a3ca1c7fff3cef37d9842b0e7706075fc43bd39db42951dac58d0c1"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"泰勒余项符号给出上界","source_units":[{"start":84386,"end":84876,"start_line":1643,"end_line":1653,"raw_tex":"\\begin{proof}\n  We consider the function $f \\colon [0,\\infty) \\to \\R$ with $f(x) = \\frac{-1}{a x + b}$. \n  Then the first and second derivative of $f$ are given by $f'(x) = \\frac{a}{(a x + b)^2}$ and \n  $f''(x) = \\frac{-2a^2}{(a x + b)^3}$. Using a first-order Taylor expansion of $f$ then shows \n  that\n  \\begin{align*}\n    f(x) = - \\frac{1}{b} + \\frac{a}{b^2} x - \\frac{a^2}{(a \\xi + b)^3} x^2\n    \\leq - \\frac{1}{b} + \\frac{a}{b^2} x,\n  \\end{align*}\n  where $\\xi \\in (0,x)$. \n\\end{proof}","sha256_utf8":"f19b51fd6a3ca1c7fff3cef37d9842b0e7706075fc43bd39db42951dac58d0c1"}]},{"id":"S067","semantic_type":"external_fact","kind":"external_fact","title":"微分交换标准定理","source_units":[],"declared_external":true,"declaration_text":["RyuBoyd.2016, Lemma 6 supplies the differentiation-under-expectation result invoked by the author in B0131 under the assumptions of B0129."]},{"id":"S068","semantic_type":"external_fact","kind":"external_fact","title":"单调导数刻画凸性","source_units":[],"declared_external":true,"declaration_text":["A differentiable functional with monotone derivative is convex. Applied to F(v) minus mu times the squared norm of v divided by two, this yields the strong convexity inequality, as cited from ZeidlerIIB.1990, Proposition 25.10."]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","title":"光滑函数二次上界","source_units":[],"declared_external":true,"declaration_text":["A differentiable functional whose gradient is L-Lipschitz satisfies F(v) <= F(w) + <iota grad F(w), v-w> + L ||v-w||^2 / 2, as invoked from BottouCurtisNocedal.2018, Appendix B."]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","title":"强凸强制函数极小点存在","source_units":[],"declared_external":true,"declaration_text":["In the Hilbert-space setting used here, a continuous strongly convex coercive functional attains its minimum uniquely; the author invokes ZeidlerIIB.1990, Theorem 25.D, Proposition 25.20 and Corollary 25.15."]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","title":"标准分析与期望运算","source_units":[],"declared_external":true,"declaration_text":["Use the Riesz identification, the vanishing derivative at a differentiable minimizer, Hilbert norm identities, Cauchy–Schwarz, Young, Minkowski, Hölder and Jensen inequalities, independence and iterated expectation, and elementary exponential and Taylor inequalities. In particular, independence and the mean-gradient identity make the optimum-gradient terms centered, and lower moments are bounded by the corresponding higher moments."]},{"id":"S072","semantic_type":"assumption","kind":"assumption","title":"主定理矩条件与步长限制","source_units":[{"start":49021,"end":49225,"start_line":977,"end_line":979,"raw_tex":"  Let Assumption~\\ref{ass:fHigherIntegrability} be fulfilled.\n  For $\\alpha_n = \\frac{\\vartheta}{n + \\gamma}$, $n \\in \\N$, with $\\gamma \\in [0,\\infty)$, \n  $\\vartheta \\in (0,\\frac{1 + \\gamma}{2\\mu}]$ and\n","sha256_utf8":"bcccbe249380c00bbd2c7fad13db21b9ee81895b36b69bf33a1543eeeadea0a8"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S006","target_id":"S002","kind":"supports"},{"source_id":"S007","target_id":"S002","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S047","target_id":"S003","kind":"supports"},{"source_id":"S011","target_id":"S003","kind":"supports"},{"source_id":"S012","target_id":"S003","kind":"supports"},{"source_id":"S064","target_id":"S003","kind":"supports"},{"source_id":"S064","target_id":"S004","kind":"supports"},{"source_id":"S012","target_id":"S004","kind":"supports"},{"source_id":"S023","target_id":"S005","kind":"supports"},{"source_id":"S037","target_id":"S005","kind":"supports"},{"source_id":"S009","target_id":"S005","kind":"supports"},{"source_id":"S072","target_id":"S004","kind":"supports"},{"source_id":"S072","target_id":"S005","kind":"supports"},{"source_id":"S072","target_id":"S006","kind":"supports"},{"source_id":"S072","target_id":"S007","kind":"supports"},{"source_id":"S059","target_id":"S007","kind":"supports"},{"source_id":"S071","target_id":"S003","kind":"supports"},{"source_id":"S071","target_id":"S004","kind":"supports"},{"source_id":"S071","target_id":"S005","kind":"supports"},{"source_id":"S008","target_id":"S010","kind":"supports"},{"source_id":"S008","target_id":"S011","kind":"supports"},{"source_id":"S009","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S072","kind":"supports"},{"source_id":"S009","target_id":"S013","kind":"supports"},{"source_id":"S008","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S016","kind":"supports"},{"source_id":"S019","target_id":"S016","kind":"supports"},{"source_id":"S011","target_id":"S017","kind":"supports"},{"source_id":"S020","target_id":"S017","kind":"supports"},{"source_id":"S068","target_id":"S017","kind":"supports"},{"source_id":"S011","target_id":"S018","kind":"supports"},{"source_id":"S020","target_id":"S018","kind":"supports"},{"source_id":"S069","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S019","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S070","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S011","target_id":"S022","kind":"supports"},{"source_id":"S010","target_id":"S022","kind":"supports"},{"source_id":"S067","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S027","target_id":"S024","kind":"supports"},{"source_id":"S028","target_id":"S024","kind":"supports"},{"source_id":"S029","target_id":"S024","kind":"supports"},{"source_id":"S030","target_id":"S024","kind":"supports"},{"source_id":"S031","target_id":"S024","kind":"supports"},{"source_id":"S032","target_id":"S024","kind":"supports"},{"source_id":"S033","target_id":"S024","kind":"supports"},{"source_id":"S034","target_id":"S024","kind":"supports"},{"source_id":"S035","target_id":"S024","kind":"supports"},{"source_id":"S036","target_id":"S024","kind":"supports"},{"source_id":"S013","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S027","kind":"supports"},{"source_id":"S013","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S011","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S011","target_id":"S029","kind":"supports"},{"source_id":"S015","target_id":"S029","kind":"supports"},{"source_id":"S020","target_id":"S029","kind":"supports"},{"source_id":"S009","target_id":"S029","kind":"supports"},{"source_id":"S027","target_id":"S030","kind":"supports"},{"source_id":"S011","target_id":"S030","kind":"supports"},{"source_id":"S027","target_id":"S031","kind":"supports"},{"source_id":"S028","target_id":"S031","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S011","target_id":"S032","kind":"supports"},{"source_id":"S009","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S032","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S072","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S034","kind":"supports"},{"source_id":"S071","target_id":"S026","kind":"supports"},{"source_id":"S071","target_id":"S028","kind":"supports"},{"source_id":"S071","target_id":"S029","kind":"supports"},{"source_id":"S071","target_id":"S032","kind":"supports"},{"source_id":"S071","target_id":"S034","kind":"supports"},{"source_id":"S071","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S041","target_id":"S038","kind":"supports"},{"source_id":"S042","target_id":"S038","kind":"supports"},{"source_id":"S043","target_id":"S038","kind":"supports"},{"source_id":"S031","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S012","target_id":"S040","kind":"supports"},{"source_id":"S015","target_id":"S040","kind":"supports"},{"source_id":"S020","target_id":"S040","kind":"supports"},{"source_id":"S009","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S023","target_id":"S043","kind":"supports"},{"source_id":"S072","target_id":"S043","kind":"supports"},{"source_id":"S009","target_id":"S043","kind":"supports"},{"source_id":"S071","target_id":"S040","kind":"supports"},{"source_id":"S071","target_id":"S041","kind":"supports"},{"source_id":"S071","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S014","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S049","target_id":"S048","kind":"supports"},{"source_id":"S050","target_id":"S048","kind":"supports"},{"source_id":"S051","target_id":"S048","kind":"supports"},{"source_id":"S052","target_id":"S048","kind":"supports"},{"source_id":"S053","target_id":"S048","kind":"supports"},{"source_id":"S054","target_id":"S048","kind":"supports"},{"source_id":"S055","target_id":"S048","kind":"supports"},{"source_id":"S056","target_id":"S048","kind":"supports"},{"source_id":"S057","target_id":"S048","kind":"supports"},{"source_id":"S058","target_id":"S048","kind":"supports"},{"source_id":"S013","target_id":"S049","kind":"supports"},{"source_id":"S014","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S011","target_id":"S051","kind":"supports"},{"source_id":"S014","target_id":"S051","kind":"supports"},{"source_id":"S014","target_id":"S052","kind":"supports"},{"source_id":"S050","target_id":"S053","kind":"supports"},{"source_id":"S051","target_id":"S053","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S014","target_id":"S054","kind":"supports"},{"source_id":"S049","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S014","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S057","kind":"supports"},{"source_id":"S044","target_id":"S057","kind":"supports"},{"source_id":"S015","target_id":"S057","kind":"supports"},{"source_id":"S020","target_id":"S057","kind":"supports"},{"source_id":"S009","target_id":"S057","kind":"supports"},{"source_id":"S049","target_id":"S058","kind":"supports"},{"source_id":"S053","target_id":"S058","kind":"supports"},{"source_id":"S054","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S071","target_id":"S049","kind":"supports"},{"source_id":"S071","target_id":"S056","kind":"supports"},{"source_id":"S071","target_id":"S057","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S060","kind":"supports"},{"source_id":"S063","target_id":"S060","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S063","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S071","target_id":"S061","kind":"supports"},{"source_id":"S071","target_id":"S062","kind":"supports"},{"source_id":"S071","target_id":"S063","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"},{"source_id":"S071","target_id":"S066","kind":"supports"}]}}
{"record_id":"2106.09340v2","paper":{"title":"A trust region-type normal map-based semismooth Newton method for nonsmooth nonconvex composite optimization","authors":["Wenqing Ouyang","Andre Milzarek"],"arxiv_id":"2106.09340","version":"v2","journal":"Mathematical Programming","doi":"10.1007/s10107-024-02110-2","abstract_url":"https://arxiv.org/abs/2106.09340v2","pdf_url":"https://arxiv.org/pdf/2106.09340v2"},"context":{"raw_tex":"In this paper, we develop and analyze a novel normal-map based second-order approach for the composite optimization problem\n%\n\\begin{align} \\label{eq1-1}\n\\min\\limits_{x\\in\\mathbb{R}^n}~\\psi(x):=f(x)+\\varphi(x),\n\\end{align}\n\nwhere $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ is a continuously differentiable (not necessarily convex) function and $\\varphi:\\mathbb{R}^n\\rightarrow(-\\infty,\\infty]$ is a convex, lower semicontinuous (lsc), and proper (not necessarily smooth) mapping. \n\n\nwhere $\\oprox : \\Rn \\to \\Rn$, $\\prox{x} := \\argmin_{y \\in \\Rn} \\varphi(y) + \\frac{1}{2\\lambda} \\|x-y\\|^2$ denotes the well-known proximity operator of $\\varphi$, \\cite{Mor65}, and $\\lambda > 0$ is a positive parameter. Alternatively, the forward-backward scheme in \\eqref{eq:fbs-intro} can also be interpreted as a fixed-point procedure applied to the nonsmooth equation:\n\n\nBy $\\iprod{\\cdot}{\\cdot}$ and $\\|\\cdot\\|$ we denote the standard Euclidean inner product and norm. For matrices, the norm $\\|\\cdot \\|$ is the standard spectral norm. The sets of symmetric and symmetric positive definite $n \\times n$ matrices are denoted by $\\mathbb S^n$ and $\\Spp$, respectively. For a given matrix $A \\in \\Spp$, we define the norm $\\|x\\|_A := \\sqrt{\\iprod{x}{Ax}}$. For two matrices $A, B \\in \\mathbb S^n$, we write $A\\succeq B$ if $A-B$ is positive semidefinite. We use $\\mathrm{ri}(S)$ to denote the relative interior of a convex set $S \\subset \\Rn$.\n\n\nAs mentioned, $\\oprox$ denotes the proximity operator of $\\varphi$ with respect to the scalar $\\lambda$. \nThe proximity operator is a {firmly nonexpansive mapping}, i.e.,\n%\n\\be \\label{eq:prox-nonexp} \\|\\prox{x}-\\prox{y}\\|^2\\leq\\langle x-y,\\prox{x}-\\prox{y} \\rangle \\quad \\forall~x,y \\in \\Rn. \\ee\n\nIn particular, $\\oprox$ is globally Lipschitz continuous with constant 1. Moreover, the proximity operator can be characterized by the associated optimality conditions of its underlying optimization problem:\n%\n\\begin{align}\\label{eq1-3}\n\\prox{x}\\in x-\\lambda \\partial\\varphi(\\prox{x}). \n\\end{align}\n\nWe will also work with the Moreau envelope $\\env{x}:=\\min_y\\varphi(y)+\\frac{1}{2\\lambda}\\|x-y\\|^2$. It is well known that $\\oenv$ is convex and continuously differentiable and its gradient is given by $\\nabla\\env{x}=\\frac{1}{\\lambda}(x-\\prox{x})$. Let us further note that due to the convexity of $\\vp$ and differentiability of $f$, Clarke's subdifferential $\\partial \\psi$ coincides with the regular and limiting subdifferential of $\\psi$. We refer to \\cite{Mor65,rockafellar2009variational,BauCom11} for additional details and background. \n\n\n\\begin{align} \\label{normal_eq} \\Fnor{z} := \\nabla f(\\prox{z})+{\\lambda^{-1}}(z-\\prox{z}) =0 \\end{align}\n\n\\begin{align}\\label{eq3-4}\n  \\min_{q}~m_k(q):=\\iprod{\\Fnor{z_k}}{D_kq} + \\frac{1}{2} \\iprod{M_kq}{D_kq} \\quad \\text{s.t.} \\quad \\|q\\|\\leq\\Delta_k,\n\\end{align}\n\nwhere $\\Delta_k$ is the trust region radius. %As mentioned, the Steihaug-CG method can be utilized to solve the subproblem \\eqref{eq3-4} inexactly. \nLet $\\bar q_k$ denote an approximate solution of the subproblem \\eqref{eq3-4} returned by the Steihaug-CG method. Motivated by our previous discussion, we can then generate a lifted and rescaled step $s_k$ via\n\n\n\\be \\label{eq:def-sk-bar-sk} \\bar s_k = \\bar q_k - \\lambda  (\\Fnor{z_k}+M_k\\bar q_k) \\quad \\text{and} \\quad s_k = \\min\\left\\{1,{\\Delta_k}/{\\|\\bar s_k\\|}\\right\\} \\bar s_k, \\ee\n\n\\begin{defn} \\label{def:mer}\nLet $\\lambda > 0$, $\\tau \\in (0,1)$ be given. We define the merit function\n$$ \\mer: \\Rn \\to \\R, \\quad \\mer(z):=\\psi(\\prox{z})+\\frac{\\tau\\lambda}{2}\\|\\Fnor{z}\\|^2. $$\n%where $\\tau \\in [0,1-\\nu)$, $\\nu \\in [0,1)$. %Furthermore, the criticality measure $\\chi$ is set to $\\chi(z):=\\|\\Fnor{z}\\|_{\\lambda }$.\n\\end{defn}\n\nBased on \\cref{def:mer}, we define the actual reduction term $\\mathrm{ared}_k$ via:  \n%\n\\[ \\mathrm{ared}_k := \\mer(z_k) - \\mer(z_k+s_k). \\]\n\n\nHere, $\\chi:\\mathbb{R}^n\\rightarrow\\mathbb{R}_{+}$ is a continuous criticality measure. Throughout this work, we will use with the criticality measure %$\\chi(z) := \\|\\Fnor{z}\\|$.\n%\n\\[ \\chi(z) := \\|\\Fnor{z}\\|. \\]\n\n\n\\begin{align}  \\label{eq:pred} \\mathrm{pred}(z,s,\\Delta,\\nu) & := \\frac{\\tau\\chi(z)}{2} \\min\\{\\lambda,\\Delta,\\lambda\\chi(z)\\} \\\\ & \\hspace{8ex} + \\frac{\\nu \\chi(z)}{\\min\\{\\Delta,\\lambda\\chi(z)\\}} \\|\\prox{z+s} - \\prox{z}\\|^2, \\nonumber \\end{align}\n\nwhere $\\tau \\in (0,1)$ and $\\nu \\in [0,1)$ are given. In the next sections, we show that this choice of $\\rho_k$, $\\mathrm{ared}_k$, and $\\mathrm{pred}_k$ meets all the mentioned requirements. The full details of the method are presented in \\cref{algo2}. Notice that we allow the usage of approximate Hessian information $B_k \\approx \\nabla^2 f(\\prox{z_k})$. We call iteration $k$ successful if $z_k + s_k$ is accepted as new iterate, i.e., $z_{k+1}=z_k+s_k$.\n\n\n\\begin{algorithm}[t]\n        \\caption{A Trust Region Normal Map Semismooth Newton Method}\n         \\label{algo2}\n        \\begin{algorithmic}[1]\n            \\Require Choose an initial point $z_0\\in\\mathbb{R}^n$, $B_0 \\in \\mathbb{S}^n$, $\\lambda>0$, and $\\{\\epsilon_k\\} \\subset \\R_{+}$. Set iteration $k=0$.\n            \\While{$F_{\\text{nor}}^{\\lambda}(z_k)\\neq0$}\n            \\State Choose $D_k\\in\\partial\\text{prox}_{\\varphi}^{\\lambda}(z_k)$ and set $M_k=B_kD_k+\\frac{1}{\\lambda}(I-D_k)$.\n            \\State Run the Steihaug-CG method with $S = D_k M_k$, $g = D_k \\Fnor{z_k}$, $\\Delta = \\Delta_k$, and $\\epsilon = \\epsilon_k \\geq 0$ returning $\\bar q_k = q$;\n            \\State Set $\\bar{s}_k=\\bar q_k-\\lambda (F_{\\text{nor}}^{\\lambda}(z_k)+M_k\\bar q_k)$ and $s_k=\\min\\{1,\\frac{\\Delta_k}{\\|\\bar{s}_k\\|}\\}\\bar{s}_k$;\n            %\\EndIf\n             \\If{$\\rho_k=\\frac{\\mer(z_k)-\\mer(z_k+s_k)}{\\mathrm{pred}(z_k,s_k,\\Delta_k,\\nu_k)} < \\eta_1$}\n             \\State Set $z_{k+1}=z_{k}$ and $B_{k+1} = B_k$;\n             \\Else\n             \\State Set $z_{k+1}=z_k+s_k$ and choose $B_{k+1} \\approx \\nabla^2 f(\\prox{z_{k+1}})$;\n             \\EndIf\n             \\State Set $\\Delta_{k+1}$ based on $\\rho_k$ by invoking \\cref{algo3};\n             \\State $k\\gets k+1$;\n            \\EndWhile\n        \\end{algorithmic}\n    \\end{algorithm}\n\n\\begin{algorithm}[t]\n        \\caption{Update of the Trust Region Radius}\n        \\label{algo3}\n        \\begin{algorithmic}[1]  \n            \\Require Input: $\\Delta_k,\\rho_k$. Let $\\eta_1\\leq\\eta_2<1$ and $0<\\gamma_0<\\gamma_1<1<\\gamma_2$, and $\\Delta_{\\min} \\geq 0$ be fixed. \\vspace{0.1ex}\n           \\State Select $\\Delta_{k+1}\\in \\begin{cases} (\\gamma_0\\Delta_k,\\gamma_1\\Delta_k] & \\text{if }  \\rho_k < \\eta_1, \\\\ [\\gamma_1\\Delta_k,\\max\\{\\Delta_{\\min},\\Delta_k\\}]\\cap[\\Delta_{\\min},\\infty) & \\text{if } \\rho_k\\in[\\eta_1,\\eta_2)\\\\ (\\Delta_k,\\max\\{\\Delta_{\\min},\\gamma_2\\Delta_k\\}]\\cap[\\Delta_{\\min},\\infty) & \\text{if } \\rho_k \\geq \\eta_2 \\end{cases}$;\n%           \\If{$\\rho_k < \\eta_1$}\n%           \\State Set $\\Delta_{k+1}\\in (\\gamma_0\\Delta_k,\\gamma_1\\Delta_k]$.\n%           \\ElsIf{$\\rho_k\\in[\\eta_1,\\eta_2)$}\n%           \\State Set $\\Delta_{k+1}\\in[\\gamma_1\\Delta_k,\\max\\{\\Delta_{\\min},\\Delta_k\\}]\\cap[\\Delta_{\\min},\\infty)$;\n%           \\Else\n%          \\State Set $\\Delta_{k+1}\\in(\\Delta_k,\\max\\{\\Delta_{\\min},\\gamma_2\\Delta_k\\}]\\cap[\\Delta_{\\min},\\infty)$;\n%           \\EndIf\n           \\State Return $\\Delta_{k+1}$;\n        \\end{algorithmic}\n\\end{algorithm}\n\n\\begin{algorithm}[th]\n        \\caption{The Steihaug-CG Method}\n          \\label{algo1}\n        \\begin{algorithmic}[1]  \n            \\Require Input: $S \\in \\R^{n\\times n}$, $g \\in \\Rn$, $\\epsilon, \\Delta \\geq 0$. Set $q_0=0$, $r_0=g$, $p_0 = -g$ and $i=0$.\n            \\If{$\\|r_0\\|<\\epsilon$}\n            \\State return $q = q_0$;\n            \\EndIf\n            \\While{$i\\leq n-1$}\n                    \\If{$\\iprod{p_i}{Sp_i}\\leq0$}\n                        \\State Compute $\\alpha_{i}$ such that $\\alpha_i=\\argmin\\limits_{\\|q_i+\\alpha_ip_i\\|=\\Delta}m(q_i+\\alpha_ip_i)$ and return $q = q_i+\\alpha_ip_i$;\n                        \\Else\n                        \\State Set $\\alpha_i=\\frac{\\langle r_i,r_i \\rangle}{\\langle p_i,Sp_i\\rangle}$ and $q_{i+1}=q_i+\\alpha_ip_i$;\n                        \\If{$\\|q_{i+1}\\|\\geq\\Delta$}\n                        \\State Reset $\\alpha_i$ such that $\\alpha_i\\geq0$ and $\\|q_i+\\alpha_ip_i\\|=\\Delta$ and return $q = q_i+\\alpha_ip_i$;\n                        \\EndIf\n                        \\State Set $r_{i+1}=r_i+\\alpha_iSp_i$;\n                        \\If{$\\|r_{i+1}\\|<\\epsilon$}\n                        \\State Return $q = q_{i+1}$;\n                        \\EndIf\n                        \\State Set $\\beta_{i+1}=\\frac{\\|r_{i+1}\\|^2}{\\|r_i\\|^2}$ and $p_{i+1}=-r_{i+1}+\\beta_{i+1}p_i$;\n                    \\EndIf\n                    \\State $i\\gets i+1$\n            \\EndWhile\n        \\end{algorithmic}\n      \\end{algorithm}\n\n\\begin{assumption}\\label{assum2-1} We consider the conditions:\n\\begin{enumerate}[label=\\textup{\\textrm{(A.\\arabic*)}},topsep=0pt,itemsep=0ex,partopsep=0ex,leftmargin=8ex]\n  \\item \\label{A1} The gradient $\\nabla f$ is Lipschitz continuous on $\\dom{\\vp}$ with modulus $L$.\n  \\item \\label{A2} The objective function $\\psi$ is lower bounded on $\\dom{\\partial\\varphi}$.\n\\end{enumerate}\n\\end{assumption}\n\n\\begin{assumption}\n\\label{assum2-2}  We assume: \n\\begin{enumerate}[label=\\textup{\\textrm{(B.\\arabic*)}},topsep=0pt,itemsep=0ex,partopsep=0ex,leftmargin=8ex]\n\\item \\label{B1} \nThe parameters $\\tau$ and $\\nu_k$ satisfy the conditions: \n%\n\\[ \\nu \\in [0,1),\\quad 0\\leq\\nu_k\\leq\\nu, \\quad \\text{and} \\quad (L^2\\lambda^2+2)\\tau< 2(1-\\nu). \\]\n%\n\\item \\label{B2} It holds that $\\sum_{k=0}^\\infty ({1 + \\|B_k\\|})^{-1} = \\infty$. \n\\item \\label{B3} There is $\\kappa_B > 0$ such that $\\|B_k\\| \\leq \\kappa_B$ for all $k \\in \\N$.\n\\end{enumerate}\n\\end{assumption}\n\nCondition \\ref{B2} holds, e.g., if the matrices $\\{B_k\\}$ satisfy $\\|B_k\\| \\leq c_{B_1} + c_{B_2} k$ for all $k$ and some $c_{B_1}, c_{B_2} > 0$. Hence, \\ref{B2} is generally weaker than \\ref{B3}. Let $\\{j_i\\}$ be an increasing sequence enumerating the indices of the accepted iterates in step 8 of \\cref{algo2}. We then define the set of all successful iterations as $\\mathcal S := \\{j_i : i \\geq 0\\} = \\{k \\in \\N : \\rho_k \\geq \\eta_1\\}$. We will also use the notations \n\n\n\\be \\label{eq:muk-xk} {\\mu}_k := \\nu_k \\chi(z_k) \\min\\{\\Delta_k,\\lambda\\chi(z_k)\\}^{-1} \\quad \\text{and} \\quad x_k := \\prox{z_k}, \\ee\n\nwhere $\\{z_k\\}$ and $\\{\\Delta_k\\}$ denote the sequences generated by \\cref{algo2}. %Furthermore, \n","units":[{"start":9218,"end":9440,"start_line":228,"end_line":232,"raw_tex":"In this paper, we develop and analyze a novel normal-map based second-order approach for the composite optimization problem\n%\n\\begin{align} \\label{eq1-1}\n\\min\\limits_{x\\in\\mathbb{R}^n}~\\psi(x):=f(x)+\\varphi(x),\n\\end{align}","sha256_utf8":"c3a96ea9eb5f526832788676c3e37d26c0c5ffb1adfec83214abb2565f2f5a64"},{"start":9443,"end":9695,"start_line":234,"end_line":234,"raw_tex":"where $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ is a continuously differentiable (not necessarily convex) function and $\\varphi:\\mathbb{R}^n\\rightarrow(-\\infty,\\infty]$ is a convex, lower semicontinuous (lsc), and proper (not necessarily smooth) mapping. \n","sha256_utf8":"9418c85803fb777d3350aca551874aa69f6941ae2b176b1286f83e3a74434e40"},{"start":10770,"end":11142,"start_line":242,"end_line":242,"raw_tex":"where $\\oprox : \\Rn \\to \\Rn$, $\\prox{x} := \\argmin_{y \\in \\Rn} \\varphi(y) + \\frac{1}{2\\lambda} \\|x-y\\|^2$ denotes the well-known proximity operator of $\\varphi$, \\cite{Mor65}, and $\\lambda > 0$ is a positive parameter. Alternatively, the forward-backward scheme in \\eqref{eq:fbs-intro} can also be interpreted as a fixed-point procedure applied to the nonsmooth equation:\n","sha256_utf8":"cd32297af9019f5d7659b619ff1ef0176d43884ce9e0b4f0176c84a00a511fe3"},{"start":27615,"end":28186,"start_line":346,"end_line":346,"raw_tex":"By $\\iprod{\\cdot}{\\cdot}$ and $\\|\\cdot\\|$ we denote the standard Euclidean inner product and norm. For matrices, the norm $\\|\\cdot \\|$ is the standard spectral norm. The sets of symmetric and symmetric positive definite $n \\times n$ matrices are denoted by $\\mathbb S^n$ and $\\Spp$, respectively. For a given matrix $A \\in \\Spp$, we define the norm $\\|x\\|_A := \\sqrt{\\iprod{x}{Ax}}$. For two matrices $A, B \\in \\mathbb S^n$, we write $A\\succeq B$ if $A-B$ is positive semidefinite. We use $\\mathrm{ri}(S)$ to denote the relative interior of a convex set $S \\subset \\Rn$.\n","sha256_utf8":"d5b9b98cc0d200070cca8d11958a4d19cb9f3062fee4c6635a65613876bf7ee5"},{"start":31457,"end":31752,"start_line":376,"end_line":379,"raw_tex":"As mentioned, $\\oprox$ denotes the proximity operator of $\\varphi$ with respect to the scalar $\\lambda$. \nThe proximity operator is a {firmly nonexpansive mapping}, i.e.,\n%\n\\be \\label{eq:prox-nonexp} \\|\\prox{x}-\\prox{y}\\|^2\\leq\\langle x-y,\\prox{x}-\\prox{y} \\rangle \\quad \\forall~x,y \\in \\Rn. \\ee","sha256_utf8":"ddc48bd1fb7e1ea005b3170e1c3fc2ace00f31c498562de1b7d8d86ac45ed9f3"},{"start":31755,"end":32053,"start_line":381,"end_line":385,"raw_tex":"In particular, $\\oprox$ is globally Lipschitz continuous with constant 1. Moreover, the proximity operator can be characterized by the associated optimality conditions of its underlying optimization problem:\n%\n\\begin{align}\\label{eq1-3}\n\\prox{x}\\in x-\\lambda \\partial\\varphi(\\prox{x}). \n\\end{align}","sha256_utf8":"3042047f13d24c9187752f3525def449444f60eb1546be7f10d5697191a39d79"},{"start":32056,"end":32598,"start_line":387,"end_line":387,"raw_tex":"We will also work with the Moreau envelope $\\env{x}:=\\min_y\\varphi(y)+\\frac{1}{2\\lambda}\\|x-y\\|^2$. It is well known that $\\oenv$ is convex and continuously differentiable and its gradient is given by $\\nabla\\env{x}=\\frac{1}{\\lambda}(x-\\prox{x})$. Let us further note that due to the convexity of $\\vp$ and differentiability of $f$, Clarke's subdifferential $\\partial \\psi$ coincides with the regular and limiting subdifferential of $\\psi$. We refer to \\cite{Mor65,rockafellar2009variational,BauCom11} for additional details and background. \n","sha256_utf8":"42b9f8133910ab602b4276e0f4fb4a1dd4ea14a87470e09aaa25f75486f9f533"},{"start":32811,"end":32915,"start_line":392,"end_line":392,"raw_tex":"\\begin{align} \\label{normal_eq} \\Fnor{z} := \\nabla f(\\prox{z})+{\\lambda^{-1}}(z-\\prox{z}) =0 \\end{align}","sha256_utf8":"804cf7ff3849ce24c591da0bf0e431f1d9d116a2db0b10f14c5d105961a3745a"},{"start":43453,"end":43611,"start_line":526,"end_line":528,"raw_tex":"\\begin{align}\\label{eq3-4}\n  \\min_{q}~m_k(q):=\\iprod{\\Fnor{z_k}}{D_kq} + \\frac{1}{2} \\iprod{M_kq}{D_kq} \\quad \\text{s.t.} \\quad \\|q\\|\\leq\\Delta_k,\n\\end{align}","sha256_utf8":"c146931df9c25e6b193082159c937b88277bc24cbc1b742ade059bf04b53cf09"},{"start":43614,"end":43973,"start_line":530,"end_line":531,"raw_tex":"where $\\Delta_k$ is the trust region radius. %As mentioned, the Steihaug-CG method can be utilized to solve the subproblem \\eqref{eq3-4} inexactly. \nLet $\\bar q_k$ denote an approximate solution of the subproblem \\eqref{eq3-4} returned by the Steihaug-CG method. Motivated by our previous discussion, we can then generate a lifted and rescaled step $s_k$ via\n","sha256_utf8":"b5544c653dcafa0050e413d1797ac00ce93fec9aa5d5385d5ccaed2d7cab82e1"},{"start":43975,"end":44149,"start_line":533,"end_line":533,"raw_tex":"\\be \\label{eq:def-sk-bar-sk} \\bar s_k = \\bar q_k - \\lambda  (\\Fnor{z_k}+M_k\\bar q_k) \\quad \\text{and} \\quad s_k = \\min\\left\\{1,{\\Delta_k}/{\\|\\bar s_k\\|}\\right\\} \\bar s_k, \\ee","sha256_utf8":"37d7deaa7d780cce18a46da6b3ee37ee147e472bb6cd70353122b7facee9581e"},{"start":49680,"end":50022,"start_line":573,"end_line":577,"raw_tex":"\\begin{defn} \\label{def:mer}\nLet $\\lambda > 0$, $\\tau \\in (0,1)$ be given. We define the merit function\n$$ \\mer: \\Rn \\to \\R, \\quad \\mer(z):=\\psi(\\prox{z})+\\frac{\\tau\\lambda}{2}\\|\\Fnor{z}\\|^2. $$\n%where $\\tau \\in [0,1-\\nu)$, $\\nu \\in [0,1)$. %Furthermore, the criticality measure $\\chi$ is set to $\\chi(z):=\\|\\Fnor{z}\\|_{\\lambda }$.\n\\end{defn}","sha256_utf8":"caa867d3a2cbbe284739924bd23e2759d65c992f7c3be2ef87a0e06ff19d03b5"},{"start":50391,"end":50531,"start_line":580,"end_line":582,"raw_tex":"Based on \\cref{def:mer}, we define the actual reduction term $\\mathrm{ared}_k$ via:  \n%\n\\[ \\mathrm{ared}_k := \\mer(z_k) - \\mer(z_k+s_k). \\]\n","sha256_utf8":"b58fba0dce2f93a904ca8c2dc6a03fa586656e1b28aafc83b04d10f85ebf9cc4"},{"start":50887,"end":51099,"start_line":589,"end_line":591,"raw_tex":"Here, $\\chi:\\mathbb{R}^n\\rightarrow\\mathbb{R}_{+}$ is a continuous criticality measure. Throughout this work, we will use with the criticality measure %$\\chi(z) := \\|\\Fnor{z}\\|$.\n%\n\\[ \\chi(z) := \\|\\Fnor{z}\\|. \\]\n","sha256_utf8":"7a951ffb4b18b650ff909c4143dc5cc6844a1471496a90d5c24c848264dcbd1e"},{"start":51325,"end":51571,"start_line":596,"end_line":596,"raw_tex":"\\begin{align}  \\label{eq:pred} \\mathrm{pred}(z,s,\\Delta,\\nu) & := \\frac{\\tau\\chi(z)}{2} \\min\\{\\lambda,\\Delta,\\lambda\\chi(z)\\} \\\\ & \\hspace{8ex} + \\frac{\\nu \\chi(z)}{\\min\\{\\Delta,\\lambda\\chi(z)\\}} \\|\\prox{z+s} - \\prox{z}\\|^2, \\nonumber \\end{align}","sha256_utf8":"3f048bb178b0d697670b7e23f78bb7418d65deb2f0eabc6c4c5a1b68decfbc4b"},{"start":51574,"end":52034,"start_line":598,"end_line":598,"raw_tex":"where $\\tau \\in (0,1)$ and $\\nu \\in [0,1)$ are given. In the next sections, we show that this choice of $\\rho_k$, $\\mathrm{ared}_k$, and $\\mathrm{pred}_k$ meets all the mentioned requirements. The full details of the method are presented in \\cref{algo2}. Notice that we allow the usage of approximate Hessian information $B_k \\approx \\nabla^2 f(\\prox{z_k})$. We call iteration $k$ successful if $z_k + s_k$ is accepted as new iterate, i.e., $z_{k+1}=z_k+s_k$.\n","sha256_utf8":"dfb2e5e95bd3716ed69ed1ea50eadd1bed8316573aaa7190f5d595f27822c141"},{"start":52036,"end":53361,"start_line":600,"end_line":619,"raw_tex":"\\begin{algorithm}[t]\n        \\caption{A Trust Region Normal Map Semismooth Newton Method}\n         \\label{algo2}\n        \\begin{algorithmic}[1]\n            \\Require Choose an initial point $z_0\\in\\mathbb{R}^n$, $B_0 \\in \\mathbb{S}^n$, $\\lambda>0$, and $\\{\\epsilon_k\\} \\subset \\R_{+}$. Set iteration $k=0$.\n            \\While{$F_{\\text{nor}}^{\\lambda}(z_k)\\neq0$}\n            \\State Choose $D_k\\in\\partial\\text{prox}_{\\varphi}^{\\lambda}(z_k)$ and set $M_k=B_kD_k+\\frac{1}{\\lambda}(I-D_k)$.\n            \\State Run the Steihaug-CG method with $S = D_k M_k$, $g = D_k \\Fnor{z_k}$, $\\Delta = \\Delta_k$, and $\\epsilon = \\epsilon_k \\geq 0$ returning $\\bar q_k = q$;\n            \\State Set $\\bar{s}_k=\\bar q_k-\\lambda (F_{\\text{nor}}^{\\lambda}(z_k)+M_k\\bar q_k)$ and $s_k=\\min\\{1,\\frac{\\Delta_k}{\\|\\bar{s}_k\\|}\\}\\bar{s}_k$;\n            %\\EndIf\n             \\If{$\\rho_k=\\frac{\\mer(z_k)-\\mer(z_k+s_k)}{\\mathrm{pred}(z_k,s_k,\\Delta_k,\\nu_k)} < \\eta_1$}\n             \\State Set $z_{k+1}=z_{k}$ and $B_{k+1} = B_k$;\n             \\Else\n             \\State Set $z_{k+1}=z_k+s_k$ and choose $B_{k+1} \\approx \\nabla^2 f(\\prox{z_{k+1}})$;\n             \\EndIf\n             \\State Set $\\Delta_{k+1}$ based on $\\rho_k$ by invoking \\cref{algo3};\n             \\State $k\\gets k+1$;\n            \\EndWhile\n        \\end{algorithmic}\n    \\end{algorithm}","sha256_utf8":"d63e1afe81fe95683e4dbcb623440685608cd2f9309dd92f61f5aeb649faefbf"},{"start":55682,"end":56848,"start_line":642,"end_line":657,"raw_tex":"\\begin{algorithm}[t]\n        \\caption{Update of the Trust Region Radius}\n        \\label{algo3}\n        \\begin{algorithmic}[1]  \n            \\Require Input: $\\Delta_k,\\rho_k$. Let $\\eta_1\\leq\\eta_2<1$ and $0<\\gamma_0<\\gamma_1<1<\\gamma_2$, and $\\Delta_{\\min} \\geq 0$ be fixed. \\vspace{0.1ex}\n           \\State Select $\\Delta_{k+1}\\in \\begin{cases} (\\gamma_0\\Delta_k,\\gamma_1\\Delta_k] & \\text{if }  \\rho_k < \\eta_1, \\\\ [\\gamma_1\\Delta_k,\\max\\{\\Delta_{\\min},\\Delta_k\\}]\\cap[\\Delta_{\\min},\\infty) & \\text{if } \\rho_k\\in[\\eta_1,\\eta_2)\\\\ (\\Delta_k,\\max\\{\\Delta_{\\min},\\gamma_2\\Delta_k\\}]\\cap[\\Delta_{\\min},\\infty) & \\text{if } \\rho_k \\geq \\eta_2 \\end{cases}$;\n%           \\If{$\\rho_k < \\eta_1$}\n%           \\State Set $\\Delta_{k+1}\\in (\\gamma_0\\Delta_k,\\gamma_1\\Delta_k]$.\n%           \\ElsIf{$\\rho_k\\in[\\eta_1,\\eta_2)$}\n%           \\State Set $\\Delta_{k+1}\\in[\\gamma_1\\Delta_k,\\max\\{\\Delta_{\\min},\\Delta_k\\}]\\cap[\\Delta_{\\min},\\infty)$;\n%           \\Else\n%          \\State Set $\\Delta_{k+1}\\in(\\Delta_k,\\max\\{\\Delta_{\\min},\\gamma_2\\Delta_k\\}]\\cap[\\Delta_{\\min},\\infty)$;\n%           \\EndIf\n           \\State Return $\\Delta_{k+1}$;\n        \\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"7abd34442a870b830596db76dbd3ff4d4d3ee84aa149c72671ba21138715c36f"},{"start":56858,"end":58300,"start_line":660,"end_line":685,"raw_tex":"\\begin{algorithm}[th]\n        \\caption{The Steihaug-CG Method}\n          \\label{algo1}\n        \\begin{algorithmic}[1]  \n            \\Require Input: $S \\in \\R^{n\\times n}$, $g \\in \\Rn$, $\\epsilon, \\Delta \\geq 0$. Set $q_0=0$, $r_0=g$, $p_0 = -g$ and $i=0$.\n            \\If{$\\|r_0\\|<\\epsilon$}\n            \\State return $q = q_0$;\n            \\EndIf\n            \\While{$i\\leq n-1$}\n                    \\If{$\\iprod{p_i}{Sp_i}\\leq0$}\n                        \\State Compute $\\alpha_{i}$ such that $\\alpha_i=\\argmin\\limits_{\\|q_i+\\alpha_ip_i\\|=\\Delta}m(q_i+\\alpha_ip_i)$ and return $q = q_i+\\alpha_ip_i$;\n                        \\Else\n                        \\State Set $\\alpha_i=\\frac{\\langle r_i,r_i \\rangle}{\\langle p_i,Sp_i\\rangle}$ and $q_{i+1}=q_i+\\alpha_ip_i$;\n                        \\If{$\\|q_{i+1}\\|\\geq\\Delta$}\n                        \\State Reset $\\alpha_i$ such that $\\alpha_i\\geq0$ and $\\|q_i+\\alpha_ip_i\\|=\\Delta$ and return $q = q_i+\\alpha_ip_i$;\n                        \\EndIf\n                        \\State Set $r_{i+1}=r_i+\\alpha_iSp_i$;\n                        \\If{$\\|r_{i+1}\\|<\\epsilon$}\n                        \\State Return $q = q_{i+1}$;\n                        \\EndIf\n                        \\State Set $\\beta_{i+1}=\\frac{\\|r_{i+1}\\|^2}{\\|r_i\\|^2}$ and $p_{i+1}=-r_{i+1}+\\beta_{i+1}p_i$;\n                    \\EndIf\n                    \\State $i\\gets i+1$\n            \\EndWhile\n        \\end{algorithmic}\n      \\end{algorithm}","sha256_utf8":"b3acd1542b4555b6ccbac2893b4affbd100f84ee62380123db6d3d43efe1a5b2"},{"start":60725,"end":61122,"start_line":726,"end_line":731,"raw_tex":"\\begin{assumption}\\label{assum2-1} We consider the conditions:\n\\begin{enumerate}[label=\\textup{\\textrm{(A.\\arabic*)}},topsep=0pt,itemsep=0ex,partopsep=0ex,leftmargin=8ex]\n  \\item \\label{A1} The gradient $\\nabla f$ is Lipschitz continuous on $\\dom{\\vp}$ with modulus $L$.\n  \\item \\label{A2} The objective function $\\psi$ is lower bounded on $\\dom{\\partial\\varphi}$.\n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"43118bd7928190dfd79e40e2281fa2dcd98f75e8f8d093394623fb8f7d858b4f"},{"start":61554,"end":62102,"start_line":737,"end_line":748,"raw_tex":"\\begin{assumption}\n\\label{assum2-2}  We assume: \n\\begin{enumerate}[label=\\textup{\\textrm{(B.\\arabic*)}},topsep=0pt,itemsep=0ex,partopsep=0ex,leftmargin=8ex]\n\\item \\label{B1} \nThe parameters $\\tau$ and $\\nu_k$ satisfy the conditions: \n%\n\\[ \\nu \\in [0,1),\\quad 0\\leq\\nu_k\\leq\\nu, \\quad \\text{and} \\quad (L^2\\lambda^2+2)\\tau< 2(1-\\nu). \\]\n%\n\\item \\label{B2} It holds that $\\sum_{k=0}^\\infty ({1 + \\|B_k\\|})^{-1} = \\infty$. \n\\item \\label{B3} There is $\\kappa_B > 0$ such that $\\|B_k\\| \\leq \\kappa_B$ for all $k \\in \\N$.\n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"f30985f68ecb5ab05a76e6cc4c437c6e788649abb2b2b79c21949ae1e6ba1999"},{"start":62105,"end":62577,"start_line":750,"end_line":750,"raw_tex":"Condition \\ref{B2} holds, e.g., if the matrices $\\{B_k\\}$ satisfy $\\|B_k\\| \\leq c_{B_1} + c_{B_2} k$ for all $k$ and some $c_{B_1}, c_{B_2} > 0$. Hence, \\ref{B2} is generally weaker than \\ref{B3}. Let $\\{j_i\\}$ be an increasing sequence enumerating the indices of the accepted iterates in step 8 of \\cref{algo2}. We then define the set of all successful iterations as $\\mathcal S := \\{j_i : i \\geq 0\\} = \\{k \\in \\N : \\rho_k \\geq \\eta_1\\}$. We will also use the notations \n","sha256_utf8":"5654edbced9b68dc1cc08354452004374f2352772210a91b6c97c1ae6c323216"},{"start":62579,"end":62712,"start_line":752,"end_line":752,"raw_tex":"\\be \\label{eq:muk-xk} {\\mu}_k := \\nu_k \\chi(z_k) \\min\\{\\Delta_k,\\lambda\\chi(z_k)\\}^{-1} \\quad \\text{and} \\quad x_k := \\prox{z_k}, \\ee","sha256_utf8":"561e883b6696b68d6d793406ed8ae036a11257f3880b223cacafc396bec0fed3"},{"start":62715,"end":62813,"start_line":754,"end_line":754,"raw_tex":"where $\\{z_k\\}$ and $\\{\\Delta_k\\}$ denote the sequences generated by \\cref{algo2}. %Furthermore, \n","sha256_utf8":"0f49c2234703f3e3c697a40fc39025b8496226d837a4474c4063fbd9f45996c1"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{thm}\n\\label{thm:global_conv}\nLet the conditions \\ref{A1}--\\ref{A2}, \\ref{B1}--\\ref{B2}, and $\\Delta_{\\min} > 0$ hold and assume that \\cref{algo2} does not terminate after finitely many steps. Then, we have\n%\n\\be \\label{eq:liminf-conv} \\lim_{k\\rightarrow\\infty}\\chi(z_k)=0 \\quad \\text{and} \\quad \\sum_{k=0}^\\infty {\\mu}_k \\|x_{k+1}-x_k\\|^2 < \\infty. \\ee\n%\n\\end{thm}","units":[{"start":78207,"end":78578,"start_line":902,"end_line":908,"raw_tex":"\\begin{thm}\n\\label{thm:global_conv}\nLet the conditions \\ref{A1}--\\ref{A2}, \\ref{B1}--\\ref{B2}, and $\\Delta_{\\min} > 0$ hold and assume that \\cref{algo2} does not terminate after finitely many steps. Then, we have\n%\n\\be \\label{eq:liminf-conv} \\lim_{k\\rightarrow\\infty}\\chi(z_k)=0 \\quad \\text{and} \\quad \\sum_{k=0}^\\infty {\\mu}_k \\|x_{k+1}-x_k\\|^2 < \\infty. \\ee\n%\n\\end{thm}","sha256_utf8":"8600de6acfef3a5be1e4415aba9bfba8bf3f2303bc650ade8aca25f34717a2ad"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma} \\label{prop2-5} \nLet $D\\in\\partial\\prox{x} \\subseteq \\R^{n \\times n}$, $x \\in \\Rn$, be an arbitrary generalized derivative. Then, it holds that: \n\\begin{itemize}\n\\item[\\rmn{(i)}] Both $D$ and $I-D$ are symmetric and positive semidefinite matrices.\n\\item[\\rmn{(ii)}] The matrix $D(I-D)$ is positive semidefinite.\n\\end{itemize}\n\\end{lemma}\n\n\\begin{lemma}\n\\label{lemma2-1} Let $B, D \\in \\mathbb{S}^n$ be symmetric matrices and let $r \\in \\Rn$ and $\\epsilon \\geq 0$ be given. %Suppose that $D$ is symmetric %and $\\Lambda(I-D)$ are symmetric and positive semidefinite \nLet us set $M := B D + \\frac{1}{\\lambda}(I-D)$ and assume that $y$ satisfies the condition $\\|D(My + r)\\| \\leq \\epsilon$. Then, setting $x = y - \\lambda (My + r)$, it follows $\\|Mx + r\\| \\leq \\|I-\\lambda B\\| \\epsilon$. \n\\end{lemma}\n\n\\begin{proof}\nApplying the definition of $M$, we directly obtain\n\\begin{align}\\label{eq3-3}\n\\lambda M =\\lambda BD +I-D=(\\lambda B -I)D+I\n\\end{align}\n%\nand\n%\n%\\begin{align*}\n$ \\|Mx+r\\|=\\|My-\\lambda M (My+r)+r\\| =\\|(I-\\lambda B)D(My+r)\\| \\leq  \\|I - \\lambda B \\| \\epsilon$. \n%\\end{align*}\n\\end{proof}\n\n\\begin{lemma}\n\\label{lemma2-5}\nSuppose that \\ref{A1} is satisfied and let $z, e \\in \\Rn$, $\\alpha \\in (0,1]$, {$\\nu \\in [0,1)$, and $\\tau \\in (0,1-\\nu)$} be given. Setting $d:=-\\Fnor{z}$ and $x := \\prox{z}$, it follows\n%\n\\begin{align*}\n\\mer(z+\\alpha\\lambda (d+e))-\\mer(z) & \\leq -\\frac{\\tau\\lambda\\alpha}{2}\\|\\Fnor{z}\\|^2 - \\frac{\\nu}{\\lambda\\alpha}\\|p_\\alpha - x\\|^2 + \\frac{\\tau\\lambda\\alpha}{2} \\|e\\|^2 \\\\ & \\hspace{4ex} + \\left[L\\tau\\lambda + 1 - \\tau\\right] \\alpha \\|e\\|\\|d+e\\| + C(\\alpha) \\|p_\\alpha-x\\|^2,\n%\\mer(z+\\alpha\\lambda (d+e))-\\mer(z) & \\\\ & \\hspace{-25ex} \\leq -\\frac{\\tau\\lambda\\alpha}{2}\\|\\Fnor{z}\\|^2 - \\frac{\\nu}{\\lambda\\alpha}\\|p_\\alpha - x\\|^2 + \\left[L\\tau + \\lambda(1 - \\tau)\\right] \\alpha \\|e\\|\\|d+e\\| \\\\ & \\hspace{-19ex} + \\frac{\\tau\\lambda\\alpha}{2} \\|e\\|^2 + C(\\alpha) \\|p_\\alpha-x\\|^2,\n\\end{align*}\nwhere $p_\\alpha := \\prox{z+\\alpha\\lambda (d+e)}$ and $C(\\alpha) := (\\frac{L^2\\lambda^2+2}{2\\lambda} \\cdot \\tau - \\frac{1-\\nu}{\\lambda})\\frac{1}{\\alpha} + L \\tau + \\frac{L}{2}-\\frac{\\tau}{2\\lambda}$.\n\\end{lemma}\n\n\\begin{proof} Applying the Lipschitz continuity of $\\nabla f$ on $\\dom{\\vp}$ and $\\nabla \\env{z+\\alpha \\lambda (d+e)} = \\frac{1}{\\lambda}(z+\\alpha \\lambda  (d+e) - p_\\alpha) \\in \\partial \\vp(p_\\alpha)$, see \\eqref{eq1-3}, we have\n%\n\\begin{align} \\label{eq:here-esti1} %\\psi(p_\\alpha) - \\psi(x) & \\\\ & \\hspace{-8ex} \n\\psi(p_\\alpha) - \\psi(x) & \\leq  \\iprod{\\nabla f(x)}{p_\\alpha - x}  + \\frac{L}{2} \\|p_\\alpha - x\\|^2 + \\frac{1}{\\lambda}\\iprod{z+\\alpha \\lambda  (d+e) - p_\\alpha}{p_\\alpha - x} \\nonumber \\\\ & = \\iprod{\\Fnor{z}+\\alpha (d+e)}{p_\\alpha - x} + \\left[\\frac{L}{2}-\\frac{1}{\\lambda}\\right] \\|p_\\alpha - x\\|^2 \\nonumber \\\\ & = (1-\\alpha) \\iprod{\\Fnor{z} - e}{p_\\alpha - x} + \\iprod{e}{p_\\alpha - x} + \\left[\\frac{L}{2}-\\frac{1}{\\lambda}\\right] \\|p_\\alpha - x\\|^2. \\end{align}\n%\nMoreover, it holds that\n%\n\\begingroup\n\\allowdisplaybreaks\n\\begin{align*} \\frac{\\lambda}{2} \\|\\Fnor{z+\\alpha\\lambda (d+e)}\\|^2 & = \\frac{\\lambda}{2} \\|\\nabla f(p_\\alpha) + \\lambda^{-1}(z+\\alpha\\lambda(d+e)-p_\\alpha)\\|^2 \\\\ & = \\frac{\\lambda}{2} \\|(1-\\alpha)\\Fnor{z} + \\nabla f(p_\\alpha) - \\nabla f(x) - {\\lambda^{-1}}(p_\\alpha - x) + \\alpha e\\|^2 \\\\ & = \\frac{\\lambda(1-\\alpha)^2}{2} \\|\\Fnor{z}\\|^2 + (1-\\alpha)\\iprod{\\Fnor{z}}{\\lambda [\\nabla f(p_\\alpha)- \\nabla f(x)]-[p_\\alpha-x]}  \\\\ & \\hspace{4ex} + (1-\\alpha)\\alpha \\iprod{\\Fnor{z}}{\\lambda e} +  \\iprod{\\nabla f(p_\\alpha)-\\nabla f(x)}{x-p_\\alpha +  \\lambda \\alpha e}  \\\\ & \\hspace{4ex} + \\frac{\\lambda}{2}\\|\\nabla f(p_\\alpha)-\\nabla f(x)\\|^2 + \\frac{\\lambda\\alpha^2}{2} \\|e\\|^2  - \\alpha \\iprod{e}{p_\\alpha - x}  + \\frac{1}{2\\lambda} \\|p_\\alpha - x\\|^2  \\\\ & \\hspace{0ex} \\leq \\frac{\\lambda(1-\\alpha)^2}{2} \\|\\Fnor{z}\\|^2 - (1-\\alpha)\\iprod{\\Fnor{z} - e}{p_\\alpha - x} - \\iprod{e}{p_\\alpha - x} \\\\ & \\hspace{4ex}  + (1-\\alpha)\\alpha \\iprod{\\Fnor{z}}{\\lambda e} + \\iprod{(1-\\alpha)\\Fnor{z}+\\alpha e}{\\lambda [\\nabla f(p_\\alpha)-\\nabla f(x)]}  \\\\ & \\hspace{4ex} + \\frac{\\lambda\\alpha^2}{2} \\|e\\|^2 + \\left[ \\frac{L^2\\lambda}{2} + L + \\frac{1}{2\\lambda}\\right] \\|p_\\alpha - x\\|^2 \\\\ & \\hspace{0ex} = \\frac{\\lambda}{2}(1-\\alpha) \\|\\Fnor{z}\\|^2 + (1-\\alpha) \\iprod{d+e}{p_\\alpha - x} - \\iprod{e}{p_\\alpha - x} + \\frac{\\lambda\\alpha}{2} \\|e\\|^2 \\\\ & \\hspace{4ex} - \\frac{\\lambda\\alpha}{2}(1-\\alpha) \\|d+e\\|^2  + \\left[ \\frac{L^2\\lambda}{2} + L + \\frac{1}{2\\lambda}\\right] \\|p_\\alpha - x\\|^2 \\\\ & \\hspace{4ex} + \\iprod{(1-\\alpha)\\Fnor{z}+\\alpha e}{\\lambda [\\nabla f(p_\\alpha)-\\nabla f(x)]}, \n\\end{align*}\n\\endgroup\n%\nwhere we used the Lipschitz continuity of $\\nabla f$ and $2 \\iprod{\\Fnor{z}}{e} =  \\|\\Fnor{z}\\|^2 + \\|e\\|^2 - \\|d+e\\|^2$ in the last step. Next, applying Young's inequality, it follows\n%\n\\begin{align*} \\vert\\iprod{d+e}{\\nabla f(p_\\alpha) - \\nabla f(x)}\\vert & \\leq \\|d+e\\| \\cdot L \\|p_\\alpha - x\\|  \\leq \\frac{\\alpha}{2}  \\|d+e\\|^2 + \\frac{L^2}{2\\alpha} \\|p_\\alpha - x\\|^2 \\end{align*}\n%\\begin{align*} \\vert\\iprod{d+e}{\\lambda [\\nabla f(p_\\alpha) - \\nabla f(x)]}\\vert & \\leq \\sqrt{\\lambda}\\|d+e\\| \\cdot L\\sqrt{\\lambda} \\|p_\\alpha - x\\|  \\leq \\frac{\\lambda\\alpha}{2}  \\|d+e\\|^2 + \\frac{L^2\\lambda}{2\\alpha} \\|p_\\alpha - x\\|^2 \\end{align*}\n%\nand we have $\\|p_\\alpha - x\\|\\leq \\|z+\\alpha\\lambda (d+e)-z\\| = \\lambda\\alpha \\|d+e\\|$, $\\vert\\iprod{e}{p_\\alpha - x}\\vert \\leq \\lambda\\alpha \\|e\\|\\|d+e\\|$, and $\\vert\\iprod{e}{\\nabla f(p_\\alpha) - \\nabla f(x)}\\vert \\leq  L\\lambda\\alpha \\|e\\| \\|d+e\\|$. Hence, upon writing $(1-\\alpha)\\Fnor{z}+\\alpha e = -(1-\\alpha)(d+e)+e$, this implies\n%\n\\begin{align} \\label{eq:here-esti2}\n \\frac{\\lambda}{2} \\|\\Fnor{z+\\alpha\\lambda (d+e)}\\|^2 & \\leq \\frac{\\lambda}{2}(1-\\alpha) \\|\\Fnor{z}\\|^2 + (1-\\alpha) \\iprod{d+e}{p_\\alpha - x}  \\\\ & \\hspace{4ex} - \\iprod{e}{p_\\alpha - x} + \\frac{\\lambda\\alpha}{2} \\|e\\|^2  + \\left[ \\frac{L^2\\lambda}{2\\alpha} + L + \\frac{1}{2\\lambda}\\right] \\|p_\\alpha - x\\|^2 + L\\lambda^2\\alpha  \\|e\\|\\|d+e\\|. \\nonumber\n \\end{align} \n%\\begin{align*}\n% \\frac{\\lambda}{2} \\|\\Fnor{z+\\alpha\\lambda (d+e)}\\|^2 & \\\\ & \\hspace{-28ex} \\leq \\frac{\\lambda}{2}(1-\\alpha) \\|\\Fnor{z}\\|^2 + (1-\\alpha) \\iprod{d+e}{p_\\alpha - x} - \\iprod{e}{p_\\alpha - x} + \\frac{\\alpha\\lambda}{2} \\|e\\|^2 \\\\ & \\hspace{-24ex}  + \\left[ \\frac{\\lambda L^2}{2\\alpha} + L\\right] \\|p_\\alpha - x\\|^2 + \\frac{1}{2\\lambda} \\|p_\\alpha - x\\|^2 + \\alpha\\lambda L \\|e\\|\\|d+e\\|.\n% \\end{align*} \nFurthermore, by the firm nonexpansiveness of the proximity operator, \\eqref{eq:prox-nonexp}, we have\n%\n\\begin{equation} \\label{eq:here-esti3} -\\iprod{d+e}{p_\\alpha - x} = - \\frac{1}{\\lambda\\alpha}\\iprod{z + \\lambda\\alpha (d+e) - z}{p_\\alpha - x} \\leq - \\frac{1}{\\lambda\\alpha} \\|p_\\alpha - x\\|^2. \\end{equation}\n%\nCombining the estimates \\eqref{eq:here-esti1}, \\eqref{eq:here-esti2}, and \\eqref{eq:here-esti3}, we finally obtain\n%\n\\begin{align*}\n\\mer(z+\\alpha\\lambda (d+e))-\\mer(z) + \\frac{\\tau\\lambda\\alpha}{2}\\|\\Fnor{z}\\|^2& \\\\ &  \\hspace{-40ex} =\\psi(p_\\alpha) - \\psi(x) +\\frac{\\tau\\lambda}{2}\\|\\Fnor{z+\\alpha\\lambda (d+e)}\\|^2-\\frac{\\tau\\lambda}{2}(1-\\alpha)\\|\\Fnor{z}\\|^2 \\\\ & \\hspace{-40ex} \\leq (1-\\tau)\\langle e,p_\\alpha-x\\rangle +  L\\tau\\lambda^2\\alpha \\|e\\|\\|d+e\\| +\\frac{\\tau\\lambda\\alpha}{2}\\|e\\|^2 \\\\ & \\hspace{-36ex} + \\left[ \\frac{L^2\\lambda \\tau}{2\\alpha} + L \\tau + \\frac{\\tau}{2\\lambda} + \\frac{L}{2}-\\frac{1}{\\lambda} - \\frac{(1-\\tau)(1-\\alpha)}{\\lambda\\alpha} \\right]\\|p_\\alpha -x\\|^2 \\\\ & \\hspace{-40ex} \\leq  - \\left[\\frac{\\nu}{\\lambda\\alpha} - C(\\alpha)\\right] \\|p_\\alpha-x\\|^2 + \\left[L\\tau\\lambda + 1-\\tau \\right]\\lambda\\alpha \\|e\\|\\|d+e\\| + \\frac{\\tau\\lambda\\alpha}{2} \\|e\\|^2,\n\\end{align*}\nas desired. \\end{proof}\n\n\\begin{remark} \\label{remark:lwb-step-size} In \\cref{lemma2-5}, suppose that condition \\ref{B1} is additionally satisfied. Then, the factor $\\frac{L^2\\lambda^2+2}{2\\lambda} \\cdot \\tau - \\frac{1-\\nu}{\\lambda}$ in $C(\\alpha)$ is negative. In particular, defining\n%\n\\be \\label{eq:lwb-step-size} \\bar\\alpha = \\bar\\alpha(L,\\lambda,\\tau,\\nu):= \\begin{cases} 1 & \\text{if $(1-2L\\lambda)\\tau \\geq L\\lambda$,} \\\\ \\min\\left\\{1,\\frac{2(1-\\nu)-(L^2\\lambda^2+2)\\tau}{(1+2\\tau)L\\lambda+\\tau}\\right\\} & \\text{otherwise,} \\end{cases} \\ee\n%\nand for all $\\alpha \\in [0,\\bar\\alpha]$, we can infer\n%\n\\begin{align*} \\mer(z+\\alpha\\lambda (d+e))-\\mer(z) & \\leq -\\frac{\\tau\\lambda\\alpha}{2} \\chi(z)^2 - \\frac{\\nu}{\\lambda\\alpha} \\|p_\\alpha - x\\|^2 + \\frac{\\tau\\lambda\\alpha}{2} \\|e\\|^2 \\\\ & \\hspace{4ex} +\\left[L\\tau\\lambda  + 1 - \\tau \\right] \\lambda\\alpha \\|e\\| \\|d+e\\|  \\end{align*}\n%\n%\n\\end{remark}\n\n\n\\begin{lemma} \\label{lemma2-6}\nLet $\\{z_k\\}$, $\\{s_k\\}$, and $\\{\\Delta_k\\}$ be generated by \\cref{algo2} and suppose that the assumptions \\ref{A1} and \\ref{B1} are satisfied. Then, there exists a constant $\\hat c > 0$ that only depends on $\\bar \\alpha$, $\\eta_2$, $\\tau$, $\\lambda$, $L$ such that every iteration $k$ with \n%\n\\be \\label{eq:bd-del} \\Delta_k \\leq \\hat c_k \\chi(z_k), \\quad \\hat c_k := \\frac{\\hat c}{1+\\lambda\\|B_k\\|}, \\ee\n%\nis very successful, i.e., the condition \\eqref{eq:bd-del} implies $\\rho_k \\geq \\eta_2$ and $k \\in \\mathcal S$.  \n\\end{lemma}\n\n\\begin{proof} First, by \\cref{prop2-5}, we have $\\|D\\|\\leq 1$ for all $D \\in \\partial \\prox{z}$ and $z \\in \\Rn$. Hence, using \\eqref{eq3-3}, we obtain\n%\n\\begin{equation} \n\\|I-\\lambda M_k\\|=\\|(I-\\lambda B_k)D_k\\|\\leq 1+\\lambda\\|B_k\\|. \\label{eq4-16}\n\\end{equation}\n%\nWe now define the constants $c_1 ={\\lambda\\bar{\\alpha}}/{(1+\\bar{\\alpha})}$, $c_2 = (1-\\eta_2)\\lambda$, \n%\n\\[  c_3 = \\frac{\\tau\\lambda(1-\\eta_2)}{8(1+\\lambda)(L\\tau\\lambda +1-\\tau)}, \\quad c_4 = \\frac{\\lambda\\sqrt{1-\\eta_2}}{2}, \\quad \\text{and} \\quad c_5=\\frac{(1-\\eta_2)\\lambda}{2\\eta_2}, \\]\n%\nwhere $\\bar{\\alpha}$ was introduced in \\cref{remark:lwb-step-size}. Furthermore, let us set \n%\n\\be \\label{eq:hat-ck} \\hat c := \\min\\left\\{\\min_{1\\leq i \\leq 5} c_i,1\\right\\} \\quad \\text{and} \\quad \\hat c_k := \\frac{\\hat c}{1+\\lambda\\|B_k\\|}. \\ee\n%\nWe consider an iteration $k$ with $\\Delta_k \\leq \\hat c_k \\chi(z_k)$. Let us define $s_k^d := -\\lambda d = \\lambda  \\Fnor{z_k}$. We have $\\|\\bar{s}_k\\| \\geq\\|s^d_k\\|-\\|(I-\\lambda B_k)D_k\\bar q_k\\|$ and by the algorithmic construction, it holds that $\\|\\bar q_k\\|\\leq\\Delta_k$. Then, applying $\\|s_k^d\\| = \\lambda\\chi(z_k)$, \\eqref{eq4-16}, and \\eqref{eq:hat-ck}, it follows\n%\n\\begin{align*}\n\\|\\bar{s}_k\\| & \\geq\\|s_k^d\\| - (1+\\lambda\\|B_k\\|)\\Delta_k \\geq \\left[ \\frac{\\lambda}{\\hat c_k}- (1+\\lambda\\|B_k\\|) \\right] \\Delta_k \\geq \\left[ \\frac{\\lambda}{c_1} - 1 \\right] (1+\\lambda\\|B_k\\|) \\Delta_k \\geq \\frac{1}{\\bar\\alpha} \\Delta_k, \\\\ \\|\\bar s_k\\| & \\geq \\lambda\\chi(z_k) - (1+\\lambda\\|B_k\\|)\\hat c_k \\chi(z_k) \\geq [\\lambda - c_2] \\chi(z_k)  = \\lambda\\eta_2\\chi(z_k),\n\\end{align*}\n%\nand $\\frac{\\Delta_k}{\\|\\bar{s}_k\\|}\\leq \\min\\{\\bar{\\alpha},  \\frac{\\Delta_k}{\\lambda\\eta_2\\chi(z_k)} \\} \\leq1$. This yields $s_k=\\frac{\\Delta_k}{\\|\\bar{s}_k\\|}\\lambda (-\\Fnor{z_k}+(\\frac{1}{\\lambda} I-M_k)\\bar q_k)$. Setting  $\\alpha={\\Delta_k}/\\|\\bar{s}_k\\| \\leq\\bar{\\alpha}$, $e=(\\frac{1}{\\lambda} I-M_k)\\bar q_k$, and using $\\lambda(d + e)=\\bar s_k$ (cf. \\eqref{eq:def-sk-bar-sk}), \\cref{lemma2-5} and \\cref{remark:lwb-step-size} imply\n%\n\\begin{align} \\label{eq4-17} \\nonumber\n\\mer(z_k+s_k)-\\mer(z_k) & \\leq -\\frac{\\tau\\lambda\\alpha}{2}\\cdot \\chi(z_k)^2 - \\frac{\\nu}{\\lambda\\alpha} \\|\\prox{z_k+s_k} - x_k\\|^2  \\\\ & \\hspace{9ex} + \\left[L \\tau\\lambda + 1-\\tau\\right] \\lambda\\alpha \\|e\\| \\|\\bar s_k\\| + \\frac{\\tau\\lambda\\alpha}{2} \\|e\\|^2.\n\\end{align}\n%\nNext, we provide additional estimates for $\\|e\\|$ and $\\|\\bar s_k\\|$. Invoking \\eqref{eq4-16}, it holds that $\\|e\\| = \\|(\\frac{1}{\\lambda} I-B_k)D_k\\bar q_k\\|\\leq \\frac{1}{\\lambda}(1+\\lambda\\|B_k\\|) \\Delta_k\\leq \\frac{1}{\\lambda} \\hat c\\chi(z_k)$ and\n%\n\\be \\label{eq:upp-sk}\n\\|\\bar s_k\\| \\leq \\|s_k^d\\| + \\lambda\\|e\\| \\leq ( \\lambda + \\hat c) \\chi(z_k). \n\\ee\n%\nThus, we obtain\n%\n\\begin{align*} \n \\left[ L\\tau\\lambda  + 1-\\tau\\right]\\lambda\\|e\\|\\|\\bar s_k\\| \\leq \\left[ L\\tau\\lambda  + 1-\\tau\\right]c_3(\\lambda+1)\\chi(z_k)^2  = \\frac{\\tau\\lambda}{2} \\frac{1-\\eta_2}{4} \\cdot \\chi(z_k)^2 \n\\end{align*} \n%\nand $\\|e\\|^2 \\leq \\frac{c_4^2}{\\lambda^2} \\chi(z_k)^2 = \\frac14{(1-\\eta_2)} \\chi(z_k)^2$. Using these estimates in \\eqref{eq4-17}, we have\n%\n\\begin{align*} \\mer(z_k+s_k)-\\mer(z_k) \\leq -\\frac{\\tau\\lambda\\alpha}{2}\\frac{1+\\eta_2}{2} \\chi(z_k)^2 - \\frac{\\nu}{\\lambda\\alpha} \\|\\prox{z_k+s_k} - x_k\\|^2. \\end{align*}\n%\nDue to $\\Delta_k \\leq c_2 \\chi(z_k) \\leq \\lambda\\chi(z_k)$ and $\\nu_k \\leq \\nu$, we can further infer \n%\n\\[ -\\frac{\\nu}{\\lambda\\alpha} \\leq -\\frac{\\nu_k}{\\lambda}\\cdot\\frac{\\eta_2 \\lambda\\chi(z_k)}{\\Delta_k} = - \\frac{\\eta_2 \\cdot \\nu_k\\chi(z_k)}{\\min\\{\\Delta_k,\\lambda\\chi(z_k)\\}} = -\\eta_2\\mu_k. \\] \n%have $\\alpha \\leq \\frac{\\Delta_k}{\\eta_2\\lambda\\chi(z_k)} = \\frac{\\min\\{\\Delta_k,\\lambda\\chi(z_k)}{\\eta_2\\lambda\\chi(z_k)} $. Hence, applying , it follows $-\\nu(\\lambda\\alpha)^{-1} \\leq \\eta_2\\mu_k$.}\n%\n%\\begin{align}\\label{eq4-18} \\nonumber \\mer(z_k+s_k)-\\mer(z_k)& \\\\ \\nonumber & \\hspace{-20ex} \\leq-\\frac{\\tau\\lambda\\alpha}{2} \\cdot \\chi(z_k)^2+\\frac{\\tau\\lambda\\alpha}{2}\\frac{1-\\eta_2}{2} \\cdot\\chi(z_k)^2 - \\frac{\\nu}{\\lambda\\alpha} \\|\\prox{z_k+s_k} - x_k\\|^2 \\\\ & \\hspace{-20ex} \\leq -\\frac{\\tau\\alpha\\lambda}{2}\\frac{\\eta_2+1}{2}\\cdot \\chi(z_k)^2 - \\frac{\\eta_2\\nu}{\\lambda\\min\\{1,{\\Delta_k}/{\\lambda/\\chi(z_k)}\\}} \\|\\prox{z_k+s_k} - x_k\\|^2. \\end{align}\n%\nBy \\eqref{eq:upp-sk}, we also have $\\|\\bar s_k\\|\\leq (\\lambda + c_5) \\chi(z_k) = (2\\eta_2)^{-1}(1+\\eta_2)\\lambda \\chi(z_k)$ which establishes $-\\alpha = - \\Delta_k/\\|\\bar s_k\\| \\leq - 2\\eta_2 \\Delta_k((1+\\eta_2)\\lambda\\chi(z_k))^{-1}$. Combining the last steps, it follows\n%\n\\begin{align*} \\mer(z_k+s_k)-\\mer(z_k) & \\leq-\\frac{\\eta_2\\tau}{2} \\Delta_k\\chi(z_k) - {\\eta_2{\\mu}_k} \\|\\prox{z_k+s_k}-x_k\\|^2 \\leq - \\eta_2 \\mathrm{pred}(z_k,s_k,\\Delta_k,\\nu_k). \\end{align*}\n%\nConsequently, we have $\\rho_k\\geq\\eta_2$ which concludes the proof. \\end{proof}\n\n\\begin{remark}\n\\label{remark:bound_chi_delta} \\cref{lemma2-6} implies that the condition $\\Delta_k \\leq \\hat c_k \\chi(z_k)$ can not hold for unsuccessful iterations $k \\notin \\mathcal S$. Let us further consider an iterate $z_k$ with $k\\in\\mathcal{S}$. If $k-1\\in\\mathcal{S}$, then we obtain $\\Delta_k\\geq \\Delta_{\\min}$. Otherwise, if $k-1\\notin\\mathcal{S}$, we have $B_k = B_{k-1}$ and we can infer $\\Delta_{k-1}>\\hat c_{k-1}\\chi(z_{k-1})=\\hat c_k\\chi(z_k)$. Thus, by the algorithmic construction, it follows $\\Delta_{k}\\geq\\gamma_0 \\Delta_{k-1}\\geq \\gamma_0\\hat c_k\\chi(z_k)$ and for all $k \\in \\mathcal S$, we have \n%\n\\[ \\Delta_k\\geq\\min\\{\\gamma_0\\hat c_k\\chi(z_k),\\Delta_{\\min}\\}.  \\]\n%\n%for all $k \\in \\mathcal S$.\n\\end{remark}\n\n\n\\begin{lemma}\n\\label{lemma2-7}\nUnder the assumptions \\ref{A1} and \\ref{B1}, \\cref{algo2} either terminates after finitely many iterations or it generates infinitely many successful steps.\n\\end{lemma}\n\n\\begin{proof} Conversely, assume that \\cref{algo2} generates an infinite sequence $\\{z_k\\}$ with only finitely many successful steps. Let $k'\\in\\mathbb{N}$ denote the last successful iteration, i.e., it holds that $z_k = z_{k'}$ for all $k\\geq k'$. The update rule for the trust region radius then implies $\\Delta_k\\rightarrow 0$. Furthermore, since the matrices $B_k$ are no longer updated for all $k \\geq k^\\prime$, we obtain $\\hat c_k = \\hat c_{k^\\prime}$ for all $k \\geq k^\\prime$ where the parameter $\\hat c_k$ is defined in \\eqref{eq:bd-del}. Due to $\\Delta_k \\to 0$ there then exists $\\ell > k^\\prime$ with $\\Delta_{\\ell} \\leq \\hat c_{k^\\prime} \\chi(z_{k^\\prime}) =  \\hat c_{\\ell}\\chi(z_{\\ell})$ which, by \\cref{lemma2-6}, yields ${\\ell} \\in \\mathcal S$. However, this is a contradiction to our assumption. %and finishes the proof of \\cref{lemma2-7}. \n\\end{proof}\n\n\\begin{proof} \\cref{lemma2-7} implies $\\vert{\\mathcal S}\\vert=\\infty$ and it holds that\n%\n\\[ {\\sum}_{k\\in\\mathcal{S}}\\mer(z_{k})-\\mer(z_{k+1})\\geq{\\sum}_{k\\in\\mathcal{S}}\\eta_1 \\mathrm{pred}(z_k,s_k,\\Delta_k,\\nu_k). \\]\n%\nFurthermore, by assumption \\ref{A2}, the merit function $\\mer$ is bounded from below on $\\Rn$. Hence, since the sequence $\\{\\mer(z_k)\\}$ is non-increasing, there exists $\\zeta \\in \\R$ with $\\lim_{k \\to \\infty} \\mer(z_k) = \\zeta$ and it follows\n%\n\\be \\label{eq:crit-sum} \\sum_{k \\in \\mathcal S} {\\mu}_k \\|x_{k+1}-x_k\\|^2 < \\infty \\quad \\text{and} \\quad \\sum_{k\\in\\mathcal{S}}\\chi(z_k)\\min\\{\\lambda,{\\Delta_k},\\lambda\\chi(z_k)\\}<\\infty. \\ee\n%\nWe now proceed as in \\cite{Pow84} and first show the weaker condition $\\liminf_{k \\to \\infty} \\chi(z_k) = 0$. {Assume that there is $\\epsilon \\in (0,\\min\\{1,\\lambda^{-1}\\Delta_{\\min}\\})$ with $\\chi(z_k) \\geq \\epsilon$ for all $k$ sufficiently large}. Then, utilizing \\cref{remark:bound_chi_delta} and \\eqref{eq:crit-sum}, we obtain\n%\n\\be \\label{eq:p-sum-lw} \\infty > \\sum_{k \\in \\mathcal S} {\\epsilon} \\min\\{\\lambda,\\Delta_k,\\lambda\\epsilon\\} \\geq {\\epsilon} \\sum_{k \\in \\mathcal S} \\min\\left\\{\\lambda\\epsilon,\\gamma_0\\hat c_k\\chi(z_k)\\right\\} \\geq {\\epsilon^2} \\sum_{k \\in \\mathcal S} \\frac{\\min\\{\\lambda,\\hat c \\gamma_0\\}}{1+\\lambda\\|B_k\\|}. \\ee % \\epsilon \\sum_{k \\in \\mathcal S} \\min\\left\\{\\epsilon,\\min\\left\\{\\frac{\\hat c \\gamma_0\\}{1+\\lambda\\|B_k\\|}, \\Delta_{\\min}\\right\\}\\right\\}  \\ee\n%\nDue to $\\vert\\mathcal S\\vert = \\infty$, we have $(1+\\lambda\\|B_k\\|)^{-1} \\to 0$ as $\\mathcal S \\ni k \\to \\infty$ and hence, it follows $\\sum_{k \\in \\mathcal S} (1+\\lambda\\|B_k\\|)^{-1} \\leq \\infty$. Let us recall the notation $\\mathcal S = \\{j_i : i \\geq 0\\}$ and let us consider an arbitrary unsuccessful iteration $ j_i+1 \\leq k \\leq j_{i+1}-1$. Using the trust region update mechanism and \\cref{lemma2-6}, this yields\n%\n\\[\\frac{\\hat c\\epsilon}{1+\\lambda\\|B_k\\|} \\leq \\hat c_k \\chi(z_k) \\leq \\Delta_k \\leq \\gamma_1^{k-j_i} \\Delta_{j_i}. \\]\n%\nSumming this estimate, we obtain\n%\n\\[ \\sum_{k=j_i+1}^{j_{i+1}-1} \\frac{1}{1+\\lambda\\|B_k\\|} \\leq \\frac{\\Delta_{j_i}}{\\hat c \\epsilon} \\sum_{k=j_i+1}^{j_{i+1}-1} \\gamma_1^{k-j_i} \\leq \\frac{\\Delta_{j_i}}{\\hat c \\epsilon} \\left[ \\sum_{k=0}^{\\infty} \\gamma_1^{k} - 1 \\right] = \\frac{\\gamma_1}{(1-\\gamma_1)\\hat c \\epsilon} \\cdot \\Delta_{j_i}. \\]\n%\nSumming this expression once more for all $i$, we can infer $\\sum_{k \\notin \\mathcal S}(1+\\lambda\\|B_k\\|^{-1}) \\leq \\frac{\\gamma_1}{(1-\\gamma_1)\\hat c\\epsilon} \\sum_{k \\in \\mathcal S} \\Delta_k$. As before, the summability condition in \\eqref{eq:p-sum-lw} implies $\\Delta_k \\to 0$ as $\\mathcal S \\ni k \\to \\infty$ and hence, combining the last steps, it follows \n%\n\\[ \\sum_{k=0}^\\infty \\frac{1}{1+\\lambda\\|B_k\\|} < \\infty \\quad \\implies \\quad \\sum_{k=0}^\\infty \\frac{1}{1+\\|B_k\\|}<\\infty. \\]\n%\nHowever, this contradicts \\ref{B2} and thus, we can establish $\\liminf_{k \\to \\infty} \\chi(z_k) = 0$. %\\revise{In order to show convergence of the full sequence $\\{\\chi(z_k)\\}$, %we can apply the Lipschitz continuity of $\\nabla f$ {and $\\chi$} and follow the strategies used in the convergence analysis of classical trust region-type methods, see, e.g., \n%\\cite[Theorem 6.4.6]{ConGouToi00}. We refer to \\cite[Theorem 4.8]{OuyMil21} for a full derivation. \n%\nLet us now suppose that $\\{\\chi(z_k)\\}$ does not converge. Then there exist $\\delta > 0$ and infinite, increasing sequences $\\{t_i\\}_i,\\{\\ell_i\\}_i \\subset \\mathcal S$ such that $t_{i+1} \\geq \\ell_i > t_i$ for all $i \\in \\N$ and\n%\n\\[ \\chi(z_{t_i}) \\geq 2\\delta, \\quad \\chi(z_{\\ell_i}) < \\delta, \\quad \\text{and} \\quad \\chi(z_k) \\geq \\delta \\quad k = t_i+1,...,\\ell_i-1. \\]\n%\nLet us define $\\mathcal K := \\{k \\in \\mathcal S: t_i \\leq k < \\ell_i\\}$. Due to $\\mathcal K \\subseteq \\mathcal S$, the second condition in \\eqref{eq:crit-sum} implies \n%\n\\[ \\infty > \\sum_{k \\in \\mathcal K} \\chi(z_k) \\min\\{\\lambda,\\Delta_k,\\lambda\\chi(z_k)\\} \\geq \\sum_{k \\in \\mathcal K} \\delta \\min\\{\\lambda,\\Delta_k,\\lambda\\delta\\}. \\]\n%\nConsequently, we have $\\Delta_k \\to 0$ as $\\mathcal K \\ni k \\to \\infty$ and utilizing $\\rho_k \\geq \\eta_1$ for all $k \\in \\mathcal K$, it follows $\\Delta_k \\leq \\frac{2}{\\tau\\eta_1\\delta}[\\mer(z_k)-\\mer(z_{k+1})]$ for all $k \\in \\mathcal K$ sufficiently large. Next, the Lipschitz continuity of $\\nabla f$ and $\\oprox$ yield\n%\n\\begin{align} \\nonumber \\vert\\chi(w)-\\chi(z)\\vert & \\leq \\|\\Fnor{w}-\\Fnor{z}\\| \\\\ \\nonumber & \\hspace{-12ex} \\leq \\|\\nabla f(\\prox{w})-\\nabla f(\\prox{z})\\| + {\\lambda^{-1}}\\|w-\\prox{w} - (z-\\prox{z})\\| \\\\ & \\hspace{-12ex}  \\leq L \\|\\prox{w}-\\prox{z}\\| + 2\\lambda^{-1}\\|w-z\\| \\leq (L+2\\lambda^{-1}) \\|w-z\\| \\label{eq:fnor-lipschitz} \\end{align}\n%\nfor all $w, z \\in \\Rn$. Thus, setting $L_F := L+2\\lambda^{-1}$, combining the previous estimates, and using \\eqref{eq:def-sk-bar-sk}, we obtain\n%\n\\begin{align*} \\delta & < \\vert\\chi(z_{t_i}) - \\chi(z_{\\ell_i})\\vert \\leq L_F {\\|z_{\\ell_i}-z_{t_i}\\|} \\\\ & \\leq L_F \\sum_{k = t_i, \\, k \\in \\mathcal K}^{\\ell_i-1} \\|s_k\\| \\leq L_F \\sum_{k = t_i, \\, k \\in \\mathcal K}^{\\ell_i-1} \\Delta_k \\leq {\\frac{2L_F}{\\tau\\eta_1\\delta}}[\\mer(z_{t_i}) - \\mer(z_{\\ell_i})]. \\end{align*}\n%\nDue to $\\mer(z_k) \\to \\zeta$, the right hand side of the last inequality has to converge to zero which is a contradiction.\n\\end{proof}","units":[{"start":38719,"end":39070,"start_line":457,"end_line":463,"raw_tex":"\\begin{lemma} \\label{prop2-5} \nLet $D\\in\\partial\\prox{x} \\subseteq \\R^{n \\times n}$, $x \\in \\Rn$, be an arbitrary generalized derivative. Then, it holds that: \n\\begin{itemize}\n\\item[\\rmn{(i)}] Both $D$ and $I-D$ are symmetric and positive semidefinite matrices.\n\\item[\\rmn{(ii)}] The matrix $D(I-D)$ is positive semidefinite.\n\\end{itemize}\n\\end{lemma}","sha256_utf8":"4599a3adef05e7e133109779edd456ee78889a50739e86d219ab42d004442009"},{"start":41968,"end":42424,"start_line":502,"end_line":505,"raw_tex":"\\begin{lemma}\n\\label{lemma2-1} Let $B, D \\in \\mathbb{S}^n$ be symmetric matrices and let $r \\in \\Rn$ and $\\epsilon \\geq 0$ be given. %Suppose that $D$ is symmetric %and $\\Lambda(I-D)$ are symmetric and positive semidefinite \nLet us set $M := B D + \\frac{1}{\\lambda}(I-D)$ and assume that $y$ satisfies the condition $\\|D(My + r)\\| \\leq \\epsilon$. Then, setting $x = y - \\lambda (My + r)$, it follows $\\|Mx + r\\| \\leq \\|I-\\lambda B\\| \\epsilon$. \n\\end{lemma}","sha256_utf8":"3f228539841cbb741fc0dfcdbecea2db94970c294aaad524eead23a940e8a39a"},{"start":42425,"end":42723,"start_line":506,"end_line":517,"raw_tex":"\\begin{proof}\nApplying the definition of $M$, we directly obtain\n\\begin{align}\\label{eq3-3}\n\\lambda M =\\lambda BD +I-D=(\\lambda B -I)D+I\n\\end{align}\n%\nand\n%\n%\\begin{align*}\n$ \\|Mx+r\\|=\\|My-\\lambda M (My+r)+r\\| =\\|(I-\\lambda B)D(My+r)\\| \\leq  \\|I - \\lambda B \\| \\epsilon$. \n%\\end{align*}\n\\end{proof}","sha256_utf8":"cf870412e28956afbfec3e8c1ca3049bd100f371b521bbe56479b06761a0d2de"},{"start":62883,"end":63921,"start_line":758,"end_line":767,"raw_tex":"\\begin{lemma}\n\\label{lemma2-5}\nSuppose that \\ref{A1} is satisfied and let $z, e \\in \\Rn$, $\\alpha \\in (0,1]$, {$\\nu \\in [0,1)$, and $\\tau \\in (0,1-\\nu)$} be given. Setting $d:=-\\Fnor{z}$ and $x := \\prox{z}$, it follows\n%\n\\begin{align*}\n\\mer(z+\\alpha\\lambda (d+e))-\\mer(z) & \\leq -\\frac{\\tau\\lambda\\alpha}{2}\\|\\Fnor{z}\\|^2 - \\frac{\\nu}{\\lambda\\alpha}\\|p_\\alpha - x\\|^2 + \\frac{\\tau\\lambda\\alpha}{2} \\|e\\|^2 \\\\ & \\hspace{4ex} + \\left[L\\tau\\lambda + 1 - \\tau\\right] \\alpha \\|e\\|\\|d+e\\| + C(\\alpha) \\|p_\\alpha-x\\|^2,\n%\\mer(z+\\alpha\\lambda (d+e))-\\mer(z) & \\\\ & \\hspace{-25ex} \\leq -\\frac{\\tau\\lambda\\alpha}{2}\\|\\Fnor{z}\\|^2 - \\frac{\\nu}{\\lambda\\alpha}\\|p_\\alpha - x\\|^2 + \\left[L\\tau + \\lambda(1 - \\tau)\\right] \\alpha \\|e\\|\\|d+e\\| \\\\ & \\hspace{-19ex} + \\frac{\\tau\\lambda\\alpha}{2} \\|e\\|^2 + C(\\alpha) \\|p_\\alpha-x\\|^2,\n\\end{align*}\nwhere $p_\\alpha := \\prox{z+\\alpha\\lambda (d+e)}$ and $C(\\alpha) := (\\frac{L^2\\lambda^2+2}{2\\lambda} \\cdot \\tau - \\frac{1-\\nu}{\\lambda})\\frac{1}{\\alpha} + L \\tau + \\frac{L}{2}-\\frac{\\tau}{2\\lambda}$.\n\\end{lemma}","sha256_utf8":"97346ffdc64d06f9834b13fdfcd707cf625cd152e2776d2d21f1b63b3487999c"},{"start":63924,"end":69416,"start_line":769,"end_line":804,"raw_tex":"\\begin{proof} Applying the Lipschitz continuity of $\\nabla f$ on $\\dom{\\vp}$ and $\\nabla \\env{z+\\alpha \\lambda (d+e)} = \\frac{1}{\\lambda}(z+\\alpha \\lambda  (d+e) - p_\\alpha) \\in \\partial \\vp(p_\\alpha)$, see \\eqref{eq1-3}, we have\n%\n\\begin{align} \\label{eq:here-esti1} %\\psi(p_\\alpha) - \\psi(x) & \\\\ & \\hspace{-8ex} \n\\psi(p_\\alpha) - \\psi(x) & \\leq  \\iprod{\\nabla f(x)}{p_\\alpha - x}  + \\frac{L}{2} \\|p_\\alpha - x\\|^2 + \\frac{1}{\\lambda}\\iprod{z+\\alpha \\lambda  (d+e) - p_\\alpha}{p_\\alpha - x} \\nonumber \\\\ & = \\iprod{\\Fnor{z}+\\alpha (d+e)}{p_\\alpha - x} + \\left[\\frac{L}{2}-\\frac{1}{\\lambda}\\right] \\|p_\\alpha - x\\|^2 \\nonumber \\\\ & = (1-\\alpha) \\iprod{\\Fnor{z} - e}{p_\\alpha - x} + \\iprod{e}{p_\\alpha - x} + \\left[\\frac{L}{2}-\\frac{1}{\\lambda}\\right] \\|p_\\alpha - x\\|^2. \\end{align}\n%\nMoreover, it holds that\n%\n\\begingroup\n\\allowdisplaybreaks\n\\begin{align*} \\frac{\\lambda}{2} \\|\\Fnor{z+\\alpha\\lambda (d+e)}\\|^2 & = \\frac{\\lambda}{2} \\|\\nabla f(p_\\alpha) + \\lambda^{-1}(z+\\alpha\\lambda(d+e)-p_\\alpha)\\|^2 \\\\ & = \\frac{\\lambda}{2} \\|(1-\\alpha)\\Fnor{z} + \\nabla f(p_\\alpha) - \\nabla f(x) - {\\lambda^{-1}}(p_\\alpha - x) + \\alpha e\\|^2 \\\\ & = \\frac{\\lambda(1-\\alpha)^2}{2} \\|\\Fnor{z}\\|^2 + (1-\\alpha)\\iprod{\\Fnor{z}}{\\lambda [\\nabla f(p_\\alpha)- \\nabla f(x)]-[p_\\alpha-x]}  \\\\ & \\hspace{4ex} + (1-\\alpha)\\alpha \\iprod{\\Fnor{z}}{\\lambda e} +  \\iprod{\\nabla f(p_\\alpha)-\\nabla f(x)}{x-p_\\alpha +  \\lambda \\alpha e}  \\\\ & \\hspace{4ex} + \\frac{\\lambda}{2}\\|\\nabla f(p_\\alpha)-\\nabla f(x)\\|^2 + \\frac{\\lambda\\alpha^2}{2} \\|e\\|^2  - \\alpha \\iprod{e}{p_\\alpha - x}  + \\frac{1}{2\\lambda} \\|p_\\alpha - x\\|^2  \\\\ & \\hspace{0ex} \\leq \\frac{\\lambda(1-\\alpha)^2}{2} \\|\\Fnor{z}\\|^2 - (1-\\alpha)\\iprod{\\Fnor{z} - e}{p_\\alpha - x} - \\iprod{e}{p_\\alpha - x} \\\\ & \\hspace{4ex}  + (1-\\alpha)\\alpha \\iprod{\\Fnor{z}}{\\lambda e} + \\iprod{(1-\\alpha)\\Fnor{z}+\\alpha e}{\\lambda [\\nabla f(p_\\alpha)-\\nabla f(x)]}  \\\\ & \\hspace{4ex} + \\frac{\\lambda\\alpha^2}{2} \\|e\\|^2 + \\left[ \\frac{L^2\\lambda}{2} + L + \\frac{1}{2\\lambda}\\right] \\|p_\\alpha - x\\|^2 \\\\ & \\hspace{0ex} = \\frac{\\lambda}{2}(1-\\alpha) \\|\\Fnor{z}\\|^2 + (1-\\alpha) \\iprod{d+e}{p_\\alpha - x} - \\iprod{e}{p_\\alpha - x} + \\frac{\\lambda\\alpha}{2} \\|e\\|^2 \\\\ & \\hspace{4ex} - \\frac{\\lambda\\alpha}{2}(1-\\alpha) \\|d+e\\|^2  + \\left[ \\frac{L^2\\lambda}{2} + L + \\frac{1}{2\\lambda}\\right] \\|p_\\alpha - x\\|^2 \\\\ & \\hspace{4ex} + \\iprod{(1-\\alpha)\\Fnor{z}+\\alpha e}{\\lambda [\\nabla f(p_\\alpha)-\\nabla f(x)]}, \n\\end{align*}\n\\endgroup\n%\nwhere we used the Lipschitz continuity of $\\nabla f$ and $2 \\iprod{\\Fnor{z}}{e} =  \\|\\Fnor{z}\\|^2 + \\|e\\|^2 - \\|d+e\\|^2$ in the last step. Next, applying Young's inequality, it follows\n%\n\\begin{align*} \\vert\\iprod{d+e}{\\nabla f(p_\\alpha) - \\nabla f(x)}\\vert & \\leq \\|d+e\\| \\cdot L \\|p_\\alpha - x\\|  \\leq \\frac{\\alpha}{2}  \\|d+e\\|^2 + \\frac{L^2}{2\\alpha} \\|p_\\alpha - x\\|^2 \\end{align*}\n%\\begin{align*} \\vert\\iprod{d+e}{\\lambda [\\nabla f(p_\\alpha) - \\nabla f(x)]}\\vert & \\leq \\sqrt{\\lambda}\\|d+e\\| \\cdot L\\sqrt{\\lambda} \\|p_\\alpha - x\\|  \\leq \\frac{\\lambda\\alpha}{2}  \\|d+e\\|^2 + \\frac{L^2\\lambda}{2\\alpha} \\|p_\\alpha - x\\|^2 \\end{align*}\n%\nand we have $\\|p_\\alpha - x\\|\\leq \\|z+\\alpha\\lambda (d+e)-z\\| = \\lambda\\alpha \\|d+e\\|$, $\\vert\\iprod{e}{p_\\alpha - x}\\vert \\leq \\lambda\\alpha \\|e\\|\\|d+e\\|$, and $\\vert\\iprod{e}{\\nabla f(p_\\alpha) - \\nabla f(x)}\\vert \\leq  L\\lambda\\alpha \\|e\\| \\|d+e\\|$. Hence, upon writing $(1-\\alpha)\\Fnor{z}+\\alpha e = -(1-\\alpha)(d+e)+e$, this implies\n%\n\\begin{align} \\label{eq:here-esti2}\n \\frac{\\lambda}{2} \\|\\Fnor{z+\\alpha\\lambda (d+e)}\\|^2 & \\leq \\frac{\\lambda}{2}(1-\\alpha) \\|\\Fnor{z}\\|^2 + (1-\\alpha) \\iprod{d+e}{p_\\alpha - x}  \\\\ & \\hspace{4ex} - \\iprod{e}{p_\\alpha - x} + \\frac{\\lambda\\alpha}{2} \\|e\\|^2  + \\left[ \\frac{L^2\\lambda}{2\\alpha} + L + \\frac{1}{2\\lambda}\\right] \\|p_\\alpha - x\\|^2 + L\\lambda^2\\alpha  \\|e\\|\\|d+e\\|. \\nonumber\n \\end{align} \n%\\begin{align*}\n% \\frac{\\lambda}{2} \\|\\Fnor{z+\\alpha\\lambda (d+e)}\\|^2 & \\\\ & \\hspace{-28ex} \\leq \\frac{\\lambda}{2}(1-\\alpha) \\|\\Fnor{z}\\|^2 + (1-\\alpha) \\iprod{d+e}{p_\\alpha - x} - \\iprod{e}{p_\\alpha - x} + \\frac{\\alpha\\lambda}{2} \\|e\\|^2 \\\\ & \\hspace{-24ex}  + \\left[ \\frac{\\lambda L^2}{2\\alpha} + L\\right] \\|p_\\alpha - x\\|^2 + \\frac{1}{2\\lambda} \\|p_\\alpha - x\\|^2 + \\alpha\\lambda L \\|e\\|\\|d+e\\|.\n% \\end{align*} \nFurthermore, by the firm nonexpansiveness of the proximity operator, \\eqref{eq:prox-nonexp}, we have\n%\n\\begin{equation} \\label{eq:here-esti3} -\\iprod{d+e}{p_\\alpha - x} = - \\frac{1}{\\lambda\\alpha}\\iprod{z + \\lambda\\alpha (d+e) - z}{p_\\alpha - x} \\leq - \\frac{1}{\\lambda\\alpha} \\|p_\\alpha - x\\|^2. \\end{equation}\n%\nCombining the estimates \\eqref{eq:here-esti1}, \\eqref{eq:here-esti2}, and \\eqref{eq:here-esti3}, we finally obtain\n%\n\\begin{align*}\n\\mer(z+\\alpha\\lambda (d+e))-\\mer(z) + \\frac{\\tau\\lambda\\alpha}{2}\\|\\Fnor{z}\\|^2& \\\\ &  \\hspace{-40ex} =\\psi(p_\\alpha) - \\psi(x) +\\frac{\\tau\\lambda}{2}\\|\\Fnor{z+\\alpha\\lambda (d+e)}\\|^2-\\frac{\\tau\\lambda}{2}(1-\\alpha)\\|\\Fnor{z}\\|^2 \\\\ & \\hspace{-40ex} \\leq (1-\\tau)\\langle e,p_\\alpha-x\\rangle +  L\\tau\\lambda^2\\alpha \\|e\\|\\|d+e\\| +\\frac{\\tau\\lambda\\alpha}{2}\\|e\\|^2 \\\\ & \\hspace{-36ex} + \\left[ \\frac{L^2\\lambda \\tau}{2\\alpha} + L \\tau + \\frac{\\tau}{2\\lambda} + \\frac{L}{2}-\\frac{1}{\\lambda} - \\frac{(1-\\tau)(1-\\alpha)}{\\lambda\\alpha} \\right]\\|p_\\alpha -x\\|^2 \\\\ & \\hspace{-40ex} \\leq  - \\left[\\frac{\\nu}{\\lambda\\alpha} - C(\\alpha)\\right] \\|p_\\alpha-x\\|^2 + \\left[L\\tau\\lambda + 1-\\tau \\right]\\lambda\\alpha \\|e\\|\\|d+e\\| + \\frac{\\tau\\lambda\\alpha}{2} \\|e\\|^2,\n\\end{align*}\nas desired. \\end{proof}","sha256_utf8":"1d76e46e03dab2a4cd6d6ca20f929f91be5834a6873961d6177f05b9dc946caf"},{"start":69418,"end":70297,"start_line":806,"end_line":815,"raw_tex":"\\begin{remark} \\label{remark:lwb-step-size} In \\cref{lemma2-5}, suppose that condition \\ref{B1} is additionally satisfied. Then, the factor $\\frac{L^2\\lambda^2+2}{2\\lambda} \\cdot \\tau - \\frac{1-\\nu}{\\lambda}$ in $C(\\alpha)$ is negative. In particular, defining\n%\n\\be \\label{eq:lwb-step-size} \\bar\\alpha = \\bar\\alpha(L,\\lambda,\\tau,\\nu):= \\begin{cases} 1 & \\text{if $(1-2L\\lambda)\\tau \\geq L\\lambda$,} \\\\ \\min\\left\\{1,\\frac{2(1-\\nu)-(L^2\\lambda^2+2)\\tau}{(1+2\\tau)L\\lambda+\\tau}\\right\\} & \\text{otherwise,} \\end{cases} \\ee\n%\nand for all $\\alpha \\in [0,\\bar\\alpha]$, we can infer\n%\n\\begin{align*} \\mer(z+\\alpha\\lambda (d+e))-\\mer(z) & \\leq -\\frac{\\tau\\lambda\\alpha}{2} \\chi(z)^2 - \\frac{\\nu}{\\lambda\\alpha} \\|p_\\alpha - x\\|^2 + \\frac{\\tau\\lambda\\alpha}{2} \\|e\\|^2 \\\\ & \\hspace{4ex} +\\left[L\\tau\\lambda  + 1 - \\tau \\right] \\lambda\\alpha \\|e\\| \\|d+e\\|  \\end{align*}\n%\n%\n\\end{remark}\n","sha256_utf8":"d66fbc31b4a0a9e6a608e32bce13103ea1d2c8c44ed76406cfe4e7be7ecd01a3"},{"start":70642,"end":71204,"start_line":819,"end_line":825,"raw_tex":"\\begin{lemma} \\label{lemma2-6}\nLet $\\{z_k\\}$, $\\{s_k\\}$, and $\\{\\Delta_k\\}$ be generated by \\cref{algo2} and suppose that the assumptions \\ref{A1} and \\ref{B1} are satisfied. Then, there exists a constant $\\hat c > 0$ that only depends on $\\bar \\alpha$, $\\eta_2$, $\\tau$, $\\lambda$, $L$ such that every iteration $k$ with \n%\n\\be \\label{eq:bd-del} \\Delta_k \\leq \\hat c_k \\chi(z_k), \\quad \\hat c_k := \\frac{\\hat c}{1+\\lambda\\|B_k\\|}, \\ee\n%\nis very successful, i.e., the condition \\eqref{eq:bd-del} implies $\\rho_k \\geq \\eta_2$ and $k \\in \\mathcal S$.  \n\\end{lemma}","sha256_utf8":"6adbc4af21339efd92dfe9dc3c87e66d4820ddf8e5c66a070b8d9f1a0c3e3e72"},{"start":71207,"end":75992,"start_line":827,"end_line":880,"raw_tex":"\\begin{proof} First, by \\cref{prop2-5}, we have $\\|D\\|\\leq 1$ for all $D \\in \\partial \\prox{z}$ and $z \\in \\Rn$. Hence, using \\eqref{eq3-3}, we obtain\n%\n\\begin{equation} \n\\|I-\\lambda M_k\\|=\\|(I-\\lambda B_k)D_k\\|\\leq 1+\\lambda\\|B_k\\|. \\label{eq4-16}\n\\end{equation}\n%\nWe now define the constants $c_1 ={\\lambda\\bar{\\alpha}}/{(1+\\bar{\\alpha})}$, $c_2 = (1-\\eta_2)\\lambda$, \n%\n\\[  c_3 = \\frac{\\tau\\lambda(1-\\eta_2)}{8(1+\\lambda)(L\\tau\\lambda +1-\\tau)}, \\quad c_4 = \\frac{\\lambda\\sqrt{1-\\eta_2}}{2}, \\quad \\text{and} \\quad c_5=\\frac{(1-\\eta_2)\\lambda}{2\\eta_2}, \\]\n%\nwhere $\\bar{\\alpha}$ was introduced in \\cref{remark:lwb-step-size}. Furthermore, let us set \n%\n\\be \\label{eq:hat-ck} \\hat c := \\min\\left\\{\\min_{1\\leq i \\leq 5} c_i,1\\right\\} \\quad \\text{and} \\quad \\hat c_k := \\frac{\\hat c}{1+\\lambda\\|B_k\\|}. \\ee\n%\nWe consider an iteration $k$ with $\\Delta_k \\leq \\hat c_k \\chi(z_k)$. Let us define $s_k^d := -\\lambda d = \\lambda  \\Fnor{z_k}$. We have $\\|\\bar{s}_k\\| \\geq\\|s^d_k\\|-\\|(I-\\lambda B_k)D_k\\bar q_k\\|$ and by the algorithmic construction, it holds that $\\|\\bar q_k\\|\\leq\\Delta_k$. Then, applying $\\|s_k^d\\| = \\lambda\\chi(z_k)$, \\eqref{eq4-16}, and \\eqref{eq:hat-ck}, it follows\n%\n\\begin{align*}\n\\|\\bar{s}_k\\| & \\geq\\|s_k^d\\| - (1+\\lambda\\|B_k\\|)\\Delta_k \\geq \\left[ \\frac{\\lambda}{\\hat c_k}- (1+\\lambda\\|B_k\\|) \\right] \\Delta_k \\geq \\left[ \\frac{\\lambda}{c_1} - 1 \\right] (1+\\lambda\\|B_k\\|) \\Delta_k \\geq \\frac{1}{\\bar\\alpha} \\Delta_k, \\\\ \\|\\bar s_k\\| & \\geq \\lambda\\chi(z_k) - (1+\\lambda\\|B_k\\|)\\hat c_k \\chi(z_k) \\geq [\\lambda - c_2] \\chi(z_k)  = \\lambda\\eta_2\\chi(z_k),\n\\end{align*}\n%\nand $\\frac{\\Delta_k}{\\|\\bar{s}_k\\|}\\leq \\min\\{\\bar{\\alpha},  \\frac{\\Delta_k}{\\lambda\\eta_2\\chi(z_k)} \\} \\leq1$. This yields $s_k=\\frac{\\Delta_k}{\\|\\bar{s}_k\\|}\\lambda (-\\Fnor{z_k}+(\\frac{1}{\\lambda} I-M_k)\\bar q_k)$. Setting  $\\alpha={\\Delta_k}/\\|\\bar{s}_k\\| \\leq\\bar{\\alpha}$, $e=(\\frac{1}{\\lambda} I-M_k)\\bar q_k$, and using $\\lambda(d + e)=\\bar s_k$ (cf. \\eqref{eq:def-sk-bar-sk}), \\cref{lemma2-5} and \\cref{remark:lwb-step-size} imply\n%\n\\begin{align} \\label{eq4-17} \\nonumber\n\\mer(z_k+s_k)-\\mer(z_k) & \\leq -\\frac{\\tau\\lambda\\alpha}{2}\\cdot \\chi(z_k)^2 - \\frac{\\nu}{\\lambda\\alpha} \\|\\prox{z_k+s_k} - x_k\\|^2  \\\\ & \\hspace{9ex} + \\left[L \\tau\\lambda + 1-\\tau\\right] \\lambda\\alpha \\|e\\| \\|\\bar s_k\\| + \\frac{\\tau\\lambda\\alpha}{2} \\|e\\|^2.\n\\end{align}\n%\nNext, we provide additional estimates for $\\|e\\|$ and $\\|\\bar s_k\\|$. Invoking \\eqref{eq4-16}, it holds that $\\|e\\| = \\|(\\frac{1}{\\lambda} I-B_k)D_k\\bar q_k\\|\\leq \\frac{1}{\\lambda}(1+\\lambda\\|B_k\\|) \\Delta_k\\leq \\frac{1}{\\lambda} \\hat c\\chi(z_k)$ and\n%\n\\be \\label{eq:upp-sk}\n\\|\\bar s_k\\| \\leq \\|s_k^d\\| + \\lambda\\|e\\| \\leq ( \\lambda + \\hat c) \\chi(z_k). \n\\ee\n%\nThus, we obtain\n%\n\\begin{align*} \n \\left[ L\\tau\\lambda  + 1-\\tau\\right]\\lambda\\|e\\|\\|\\bar s_k\\| \\leq \\left[ L\\tau\\lambda  + 1-\\tau\\right]c_3(\\lambda+1)\\chi(z_k)^2  = \\frac{\\tau\\lambda}{2} \\frac{1-\\eta_2}{4} \\cdot \\chi(z_k)^2 \n\\end{align*} \n%\nand $\\|e\\|^2 \\leq \\frac{c_4^2}{\\lambda^2} \\chi(z_k)^2 = \\frac14{(1-\\eta_2)} \\chi(z_k)^2$. Using these estimates in \\eqref{eq4-17}, we have\n%\n\\begin{align*} \\mer(z_k+s_k)-\\mer(z_k) \\leq -\\frac{\\tau\\lambda\\alpha}{2}\\frac{1+\\eta_2}{2} \\chi(z_k)^2 - \\frac{\\nu}{\\lambda\\alpha} \\|\\prox{z_k+s_k} - x_k\\|^2. \\end{align*}\n%\nDue to $\\Delta_k \\leq c_2 \\chi(z_k) \\leq \\lambda\\chi(z_k)$ and $\\nu_k \\leq \\nu$, we can further infer \n%\n\\[ -\\frac{\\nu}{\\lambda\\alpha} \\leq -\\frac{\\nu_k}{\\lambda}\\cdot\\frac{\\eta_2 \\lambda\\chi(z_k)}{\\Delta_k} = - \\frac{\\eta_2 \\cdot \\nu_k\\chi(z_k)}{\\min\\{\\Delta_k,\\lambda\\chi(z_k)\\}} = -\\eta_2\\mu_k. \\] \n%have $\\alpha \\leq \\frac{\\Delta_k}{\\eta_2\\lambda\\chi(z_k)} = \\frac{\\min\\{\\Delta_k,\\lambda\\chi(z_k)}{\\eta_2\\lambda\\chi(z_k)} $. Hence, applying , it follows $-\\nu(\\lambda\\alpha)^{-1} \\leq \\eta_2\\mu_k$.}\n%\n%\\begin{align}\\label{eq4-18} \\nonumber \\mer(z_k+s_k)-\\mer(z_k)& \\\\ \\nonumber & \\hspace{-20ex} \\leq-\\frac{\\tau\\lambda\\alpha}{2} \\cdot \\chi(z_k)^2+\\frac{\\tau\\lambda\\alpha}{2}\\frac{1-\\eta_2}{2} \\cdot\\chi(z_k)^2 - \\frac{\\nu}{\\lambda\\alpha} \\|\\prox{z_k+s_k} - x_k\\|^2 \\\\ & \\hspace{-20ex} \\leq -\\frac{\\tau\\alpha\\lambda}{2}\\frac{\\eta_2+1}{2}\\cdot \\chi(z_k)^2 - \\frac{\\eta_2\\nu}{\\lambda\\min\\{1,{\\Delta_k}/{\\lambda/\\chi(z_k)}\\}} \\|\\prox{z_k+s_k} - x_k\\|^2. \\end{align}\n%\nBy \\eqref{eq:upp-sk}, we also have $\\|\\bar s_k\\|\\leq (\\lambda + c_5) \\chi(z_k) = (2\\eta_2)^{-1}(1+\\eta_2)\\lambda \\chi(z_k)$ which establishes $-\\alpha = - \\Delta_k/\\|\\bar s_k\\| \\leq - 2\\eta_2 \\Delta_k((1+\\eta_2)\\lambda\\chi(z_k))^{-1}$. Combining the last steps, it follows\n%\n\\begin{align*} \\mer(z_k+s_k)-\\mer(z_k) & \\leq-\\frac{\\eta_2\\tau}{2} \\Delta_k\\chi(z_k) - {\\eta_2{\\mu}_k} \\|\\prox{z_k+s_k}-x_k\\|^2 \\leq - \\eta_2 \\mathrm{pred}(z_k,s_k,\\Delta_k,\\nu_k). \\end{align*}\n%\nConsequently, we have $\\rho_k\\geq\\eta_2$ which concludes the proof. \\end{proof}","sha256_utf8":"89bde93948e8d1da7e538b4d7e47adb3f19d62136ba3f82dab3699565783a38c"},{"start":75994,"end":76728,"start_line":882,"end_line":888,"raw_tex":"\\begin{remark}\n\\label{remark:bound_chi_delta} \\cref{lemma2-6} implies that the condition $\\Delta_k \\leq \\hat c_k \\chi(z_k)$ can not hold for unsuccessful iterations $k \\notin \\mathcal S$. Let us further consider an iterate $z_k$ with $k\\in\\mathcal{S}$. If $k-1\\in\\mathcal{S}$, then we obtain $\\Delta_k\\geq \\Delta_{\\min}$. Otherwise, if $k-1\\notin\\mathcal{S}$, we have $B_k = B_{k-1}$ and we can infer $\\Delta_{k-1}>\\hat c_{k-1}\\chi(z_{k-1})=\\hat c_k\\chi(z_k)$. Thus, by the algorithmic construction, it follows $\\Delta_{k}\\geq\\gamma_0 \\Delta_{k-1}\\geq \\gamma_0\\hat c_k\\chi(z_k)$ and for all $k \\in \\mathcal S$, we have \n%\n\\[ \\Delta_k\\geq\\min\\{\\gamma_0\\hat c_k\\chi(z_k),\\Delta_{\\min}\\}.  \\]\n%\n%for all $k \\in \\mathcal S$.\n\\end{remark}\n","sha256_utf8":"180532c955e18b15e424b29cd19c597c127d93bb3c88d859d4c1b2ea74276d99"},{"start":76855,"end":77054,"start_line":892,"end_line":895,"raw_tex":"\\begin{lemma}\n\\label{lemma2-7}\nUnder the assumptions \\ref{A1} and \\ref{B1}, \\cref{algo2} either terminates after finitely many iterations or it generates infinitely many successful steps.\n\\end{lemma}","sha256_utf8":"19b2337cf5a535ed01c4fe67f49b5560dc2a30d35c4042d1cbf3d808000ecf81"},{"start":77057,"end":77927,"start_line":897,"end_line":898,"raw_tex":"\\begin{proof} Conversely, assume that \\cref{algo2} generates an infinite sequence $\\{z_k\\}$ with only finitely many successful steps. Let $k'\\in\\mathbb{N}$ denote the last successful iteration, i.e., it holds that $z_k = z_{k'}$ for all $k\\geq k'$. The update rule for the trust region radius then implies $\\Delta_k\\rightarrow 0$. Furthermore, since the matrices $B_k$ are no longer updated for all $k \\geq k^\\prime$, we obtain $\\hat c_k = \\hat c_{k^\\prime}$ for all $k \\geq k^\\prime$ where the parameter $\\hat c_k$ is defined in \\eqref{eq:bd-del}. Due to $\\Delta_k \\to 0$ there then exists $\\ell > k^\\prime$ with $\\Delta_{\\ell} \\leq \\hat c_{k^\\prime} \\chi(z_{k^\\prime}) =  \\hat c_{\\ell}\\chi(z_{\\ell})$ which, by \\cref{lemma2-6}, yields ${\\ell} \\in \\mathcal S$. However, this is a contradiction to our assumption. %and finishes the proof of \\cref{lemma2-7}. \n\\end{proof}","sha256_utf8":"7ad89e85227076c97e8cb642b9645079fa642a06ba2551ef33eb19dfa357bbf3"},{"start":78581,"end":83866,"start_line":910,"end_line":954,"raw_tex":"\\begin{proof} \\cref{lemma2-7} implies $\\vert{\\mathcal S}\\vert=\\infty$ and it holds that\n%\n\\[ {\\sum}_{k\\in\\mathcal{S}}\\mer(z_{k})-\\mer(z_{k+1})\\geq{\\sum}_{k\\in\\mathcal{S}}\\eta_1 \\mathrm{pred}(z_k,s_k,\\Delta_k,\\nu_k). \\]\n%\nFurthermore, by assumption \\ref{A2}, the merit function $\\mer$ is bounded from below on $\\Rn$. Hence, since the sequence $\\{\\mer(z_k)\\}$ is non-increasing, there exists $\\zeta \\in \\R$ with $\\lim_{k \\to \\infty} \\mer(z_k) = \\zeta$ and it follows\n%\n\\be \\label{eq:crit-sum} \\sum_{k \\in \\mathcal S} {\\mu}_k \\|x_{k+1}-x_k\\|^2 < \\infty \\quad \\text{and} \\quad \\sum_{k\\in\\mathcal{S}}\\chi(z_k)\\min\\{\\lambda,{\\Delta_k},\\lambda\\chi(z_k)\\}<\\infty. \\ee\n%\nWe now proceed as in \\cite{Pow84} and first show the weaker condition $\\liminf_{k \\to \\infty} \\chi(z_k) = 0$. {Assume that there is $\\epsilon \\in (0,\\min\\{1,\\lambda^{-1}\\Delta_{\\min}\\})$ with $\\chi(z_k) \\geq \\epsilon$ for all $k$ sufficiently large}. Then, utilizing \\cref{remark:bound_chi_delta} and \\eqref{eq:crit-sum}, we obtain\n%\n\\be \\label{eq:p-sum-lw} \\infty > \\sum_{k \\in \\mathcal S} {\\epsilon} \\min\\{\\lambda,\\Delta_k,\\lambda\\epsilon\\} \\geq {\\epsilon} \\sum_{k \\in \\mathcal S} \\min\\left\\{\\lambda\\epsilon,\\gamma_0\\hat c_k\\chi(z_k)\\right\\} \\geq {\\epsilon^2} \\sum_{k \\in \\mathcal S} \\frac{\\min\\{\\lambda,\\hat c \\gamma_0\\}}{1+\\lambda\\|B_k\\|}. \\ee % \\epsilon \\sum_{k \\in \\mathcal S} \\min\\left\\{\\epsilon,\\min\\left\\{\\frac{\\hat c \\gamma_0\\}{1+\\lambda\\|B_k\\|}, \\Delta_{\\min}\\right\\}\\right\\}  \\ee\n%\nDue to $\\vert\\mathcal S\\vert = \\infty$, we have $(1+\\lambda\\|B_k\\|)^{-1} \\to 0$ as $\\mathcal S \\ni k \\to \\infty$ and hence, it follows $\\sum_{k \\in \\mathcal S} (1+\\lambda\\|B_k\\|)^{-1} \\leq \\infty$. Let us recall the notation $\\mathcal S = \\{j_i : i \\geq 0\\}$ and let us consider an arbitrary unsuccessful iteration $ j_i+1 \\leq k \\leq j_{i+1}-1$. Using the trust region update mechanism and \\cref{lemma2-6}, this yields\n%\n\\[\\frac{\\hat c\\epsilon}{1+\\lambda\\|B_k\\|} \\leq \\hat c_k \\chi(z_k) \\leq \\Delta_k \\leq \\gamma_1^{k-j_i} \\Delta_{j_i}. \\]\n%\nSumming this estimate, we obtain\n%\n\\[ \\sum_{k=j_i+1}^{j_{i+1}-1} \\frac{1}{1+\\lambda\\|B_k\\|} \\leq \\frac{\\Delta_{j_i}}{\\hat c \\epsilon} \\sum_{k=j_i+1}^{j_{i+1}-1} \\gamma_1^{k-j_i} \\leq \\frac{\\Delta_{j_i}}{\\hat c \\epsilon} \\left[ \\sum_{k=0}^{\\infty} \\gamma_1^{k} - 1 \\right] = \\frac{\\gamma_1}{(1-\\gamma_1)\\hat c \\epsilon} \\cdot \\Delta_{j_i}. \\]\n%\nSumming this expression once more for all $i$, we can infer $\\sum_{k \\notin \\mathcal S}(1+\\lambda\\|B_k\\|^{-1}) \\leq \\frac{\\gamma_1}{(1-\\gamma_1)\\hat c\\epsilon} \\sum_{k \\in \\mathcal S} \\Delta_k$. As before, the summability condition in \\eqref{eq:p-sum-lw} implies $\\Delta_k \\to 0$ as $\\mathcal S \\ni k \\to \\infty$ and hence, combining the last steps, it follows \n%\n\\[ \\sum_{k=0}^\\infty \\frac{1}{1+\\lambda\\|B_k\\|} < \\infty \\quad \\implies \\quad \\sum_{k=0}^\\infty \\frac{1}{1+\\|B_k\\|}<\\infty. \\]\n%\nHowever, this contradicts \\ref{B2} and thus, we can establish $\\liminf_{k \\to \\infty} \\chi(z_k) = 0$. %\\revise{In order to show convergence of the full sequence $\\{\\chi(z_k)\\}$, %we can apply the Lipschitz continuity of $\\nabla f$ {and $\\chi$} and follow the strategies used in the convergence analysis of classical trust region-type methods, see, e.g., \n%\\cite[Theorem 6.4.6]{ConGouToi00}. We refer to \\cite[Theorem 4.8]{OuyMil21} for a full derivation. \n%\nLet us now suppose that $\\{\\chi(z_k)\\}$ does not converge. Then there exist $\\delta > 0$ and infinite, increasing sequences $\\{t_i\\}_i,\\{\\ell_i\\}_i \\subset \\mathcal S$ such that $t_{i+1} \\geq \\ell_i > t_i$ for all $i \\in \\N$ and\n%\n\\[ \\chi(z_{t_i}) \\geq 2\\delta, \\quad \\chi(z_{\\ell_i}) < \\delta, \\quad \\text{and} \\quad \\chi(z_k) \\geq \\delta \\quad k = t_i+1,...,\\ell_i-1. \\]\n%\nLet us define $\\mathcal K := \\{k \\in \\mathcal S: t_i \\leq k < \\ell_i\\}$. Due to $\\mathcal K \\subseteq \\mathcal S$, the second condition in \\eqref{eq:crit-sum} implies \n%\n\\[ \\infty > \\sum_{k \\in \\mathcal K} \\chi(z_k) \\min\\{\\lambda,\\Delta_k,\\lambda\\chi(z_k)\\} \\geq \\sum_{k \\in \\mathcal K} \\delta \\min\\{\\lambda,\\Delta_k,\\lambda\\delta\\}. \\]\n%\nConsequently, we have $\\Delta_k \\to 0$ as $\\mathcal K \\ni k \\to \\infty$ and utilizing $\\rho_k \\geq \\eta_1$ for all $k \\in \\mathcal K$, it follows $\\Delta_k \\leq \\frac{2}{\\tau\\eta_1\\delta}[\\mer(z_k)-\\mer(z_{k+1})]$ for all $k \\in \\mathcal K$ sufficiently large. Next, the Lipschitz continuity of $\\nabla f$ and $\\oprox$ yield\n%\n\\begin{align} \\nonumber \\vert\\chi(w)-\\chi(z)\\vert & \\leq \\|\\Fnor{w}-\\Fnor{z}\\| \\\\ \\nonumber & \\hspace{-12ex} \\leq \\|\\nabla f(\\prox{w})-\\nabla f(\\prox{z})\\| + {\\lambda^{-1}}\\|w-\\prox{w} - (z-\\prox{z})\\| \\\\ & \\hspace{-12ex}  \\leq L \\|\\prox{w}-\\prox{z}\\| + 2\\lambda^{-1}\\|w-z\\| \\leq (L+2\\lambda^{-1}) \\|w-z\\| \\label{eq:fnor-lipschitz} \\end{align}\n%\nfor all $w, z \\in \\Rn$. Thus, setting $L_F := L+2\\lambda^{-1}$, combining the previous estimates, and using \\eqref{eq:def-sk-bar-sk}, we obtain\n%\n\\begin{align*} \\delta & < \\vert\\chi(z_{t_i}) - \\chi(z_{\\ell_i})\\vert \\leq L_F {\\|z_{\\ell_i}-z_{t_i}\\|} \\\\ & \\leq L_F \\sum_{k = t_i, \\, k \\in \\mathcal K}^{\\ell_i-1} \\|s_k\\| \\leq L_F \\sum_{k = t_i, \\, k \\in \\mathcal K}^{\\ell_i-1} \\Delta_k \\leq {\\frac{2L_F}{\\tau\\eta_1\\delta}}[\\mer(z_{t_i}) - \\mer(z_{\\ell_i})]. \\end{align*}\n%\nDue to $\\mer(z_k) \\to \\zeta$, the right hand side of the last inequality has to converge to zero which is a contradiction.\n\\end{proof}","sha256_utf8":"31b048eeea04543b9c0ba30d87d42602e13f6ef0a85a899e8858ad3df0ca016e"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"算法全局收敛","source_units":[{"start":78207,"end":78578,"start_line":902,"end_line":908,"raw_tex":"\\begin{thm}\n\\label{thm:global_conv}\nLet the conditions \\ref{A1}--\\ref{A2}, \\ref{B1}--\\ref{B2}, and $\\Delta_{\\min} > 0$ hold and assume that \\cref{algo2} does not terminate after finitely many steps. Then, we have\n%\n\\be \\label{eq:liminf-conv} \\lim_{k\\rightarrow\\infty}\\chi(z_k)=0 \\quad \\text{and} \\quad \\sum_{k=0}^\\infty {\\mu}_k \\|x_{k+1}-x_k\\|^2 < \\infty. \\ee\n%\n\\end{thm}","sha256_utf8":"8600de6acfef3a5be1e4415aba9bfba8bf3f2303bc650ade8aca25f34717a2ad"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"全局收敛完整证明","source_units":[{"start":78581,"end":83866,"start_line":910,"end_line":954,"raw_tex":"\\begin{proof} \\cref{lemma2-7} implies $\\vert{\\mathcal S}\\vert=\\infty$ and it holds that\n%\n\\[ {\\sum}_{k\\in\\mathcal{S}}\\mer(z_{k})-\\mer(z_{k+1})\\geq{\\sum}_{k\\in\\mathcal{S}}\\eta_1 \\mathrm{pred}(z_k,s_k,\\Delta_k,\\nu_k). \\]\n%\nFurthermore, by assumption \\ref{A2}, the merit function $\\mer$ is bounded from below on $\\Rn$. Hence, since the sequence $\\{\\mer(z_k)\\}$ is non-increasing, there exists $\\zeta \\in \\R$ with $\\lim_{k \\to \\infty} \\mer(z_k) = \\zeta$ and it follows\n%\n\\be \\label{eq:crit-sum} \\sum_{k \\in \\mathcal S} {\\mu}_k \\|x_{k+1}-x_k\\|^2 < \\infty \\quad \\text{and} \\quad \\sum_{k\\in\\mathcal{S}}\\chi(z_k)\\min\\{\\lambda,{\\Delta_k},\\lambda\\chi(z_k)\\}<\\infty. \\ee\n%\nWe now proceed as in \\cite{Pow84} and first show the weaker condition $\\liminf_{k \\to \\infty} \\chi(z_k) = 0$. {Assume that there is $\\epsilon \\in (0,\\min\\{1,\\lambda^{-1}\\Delta_{\\min}\\})$ with $\\chi(z_k) \\geq \\epsilon$ for all $k$ sufficiently large}. Then, utilizing \\cref{remark:bound_chi_delta} and \\eqref{eq:crit-sum}, we obtain\n%\n\\be \\label{eq:p-sum-lw} \\infty > \\sum_{k \\in \\mathcal S} {\\epsilon} \\min\\{\\lambda,\\Delta_k,\\lambda\\epsilon\\} \\geq {\\epsilon} \\sum_{k \\in \\mathcal S} \\min\\left\\{\\lambda\\epsilon,\\gamma_0\\hat c_k\\chi(z_k)\\right\\} \\geq {\\epsilon^2} \\sum_{k \\in \\mathcal S} \\frac{\\min\\{\\lambda,\\hat c \\gamma_0\\}}{1+\\lambda\\|B_k\\|}. \\ee % \\epsilon \\sum_{k \\in \\mathcal S} \\min\\left\\{\\epsilon,\\min\\left\\{\\frac{\\hat c \\gamma_0\\}{1+\\lambda\\|B_k\\|}, \\Delta_{\\min}\\right\\}\\right\\}  \\ee\n%\nDue to $\\vert\\mathcal S\\vert = \\infty$, we have $(1+\\lambda\\|B_k\\|)^{-1} \\to 0$ as $\\mathcal S \\ni k \\to \\infty$ and hence, it follows $\\sum_{k \\in \\mathcal S} (1+\\lambda\\|B_k\\|)^{-1} \\leq \\infty$. Let us recall the notation $\\mathcal S = \\{j_i : i \\geq 0\\}$ and let us consider an arbitrary unsuccessful iteration $ j_i+1 \\leq k \\leq j_{i+1}-1$. Using the trust region update mechanism and \\cref{lemma2-6}, this yields\n%\n\\[\\frac{\\hat c\\epsilon}{1+\\lambda\\|B_k\\|} \\leq \\hat c_k \\chi(z_k) \\leq \\Delta_k \\leq \\gamma_1^{k-j_i} \\Delta_{j_i}. \\]\n%\nSumming this estimate, we obtain\n%\n\\[ \\sum_{k=j_i+1}^{j_{i+1}-1} \\frac{1}{1+\\lambda\\|B_k\\|} \\leq \\frac{\\Delta_{j_i}}{\\hat c \\epsilon} \\sum_{k=j_i+1}^{j_{i+1}-1} \\gamma_1^{k-j_i} \\leq \\frac{\\Delta_{j_i}}{\\hat c \\epsilon} \\left[ \\sum_{k=0}^{\\infty} \\gamma_1^{k} - 1 \\right] = \\frac{\\gamma_1}{(1-\\gamma_1)\\hat c \\epsilon} \\cdot \\Delta_{j_i}. \\]\n%\nSumming this expression once more for all $i$, we can infer $\\sum_{k \\notin \\mathcal S}(1+\\lambda\\|B_k\\|^{-1}) \\leq \\frac{\\gamma_1}{(1-\\gamma_1)\\hat c\\epsilon} \\sum_{k \\in \\mathcal S} \\Delta_k$. As before, the summability condition in \\eqref{eq:p-sum-lw} implies $\\Delta_k \\to 0$ as $\\mathcal S \\ni k \\to \\infty$ and hence, combining the last steps, it follows \n%\n\\[ \\sum_{k=0}^\\infty \\frac{1}{1+\\lambda\\|B_k\\|} < \\infty \\quad \\implies \\quad \\sum_{k=0}^\\infty \\frac{1}{1+\\|B_k\\|}<\\infty. \\]\n%\nHowever, this contradicts \\ref{B2} and thus, we can establish $\\liminf_{k \\to \\infty} \\chi(z_k) = 0$. %\\revise{In order to show convergence of the full sequence $\\{\\chi(z_k)\\}$, %we can apply the Lipschitz continuity of $\\nabla f$ {and $\\chi$} and follow the strategies used in the convergence analysis of classical trust region-type methods, see, e.g., \n%\\cite[Theorem 6.4.6]{ConGouToi00}. We refer to \\cite[Theorem 4.8]{OuyMil21} for a full derivation. \n%\nLet us now suppose that $\\{\\chi(z_k)\\}$ does not converge. Then there exist $\\delta > 0$ and infinite, increasing sequences $\\{t_i\\}_i,\\{\\ell_i\\}_i \\subset \\mathcal S$ such that $t_{i+1} \\geq \\ell_i > t_i$ for all $i \\in \\N$ and\n%\n\\[ \\chi(z_{t_i}) \\geq 2\\delta, \\quad \\chi(z_{\\ell_i}) < \\delta, \\quad \\text{and} \\quad \\chi(z_k) \\geq \\delta \\quad k = t_i+1,...,\\ell_i-1. \\]\n%\nLet us define $\\mathcal K := \\{k \\in \\mathcal S: t_i \\leq k < \\ell_i\\}$. Due to $\\mathcal K \\subseteq \\mathcal S$, the second condition in \\eqref{eq:crit-sum} implies \n%\n\\[ \\infty > \\sum_{k \\in \\mathcal K} \\chi(z_k) \\min\\{\\lambda,\\Delta_k,\\lambda\\chi(z_k)\\} \\geq \\sum_{k \\in \\mathcal K} \\delta \\min\\{\\lambda,\\Delta_k,\\lambda\\delta\\}. \\]\n%\nConsequently, we have $\\Delta_k \\to 0$ as $\\mathcal K \\ni k \\to \\infty$ and utilizing $\\rho_k \\geq \\eta_1$ for all $k \\in \\mathcal K$, it follows $\\Delta_k \\leq \\frac{2}{\\tau\\eta_1\\delta}[\\mer(z_k)-\\mer(z_{k+1})]$ for all $k \\in \\mathcal K$ sufficiently large. Next, the Lipschitz continuity of $\\nabla f$ and $\\oprox$ yield\n%\n\\begin{align} \\nonumber \\vert\\chi(w)-\\chi(z)\\vert & \\leq \\|\\Fnor{w}-\\Fnor{z}\\| \\\\ \\nonumber & \\hspace{-12ex} \\leq \\|\\nabla f(\\prox{w})-\\nabla f(\\prox{z})\\| + {\\lambda^{-1}}\\|w-\\prox{w} - (z-\\prox{z})\\| \\\\ & \\hspace{-12ex}  \\leq L \\|\\prox{w}-\\prox{z}\\| + 2\\lambda^{-1}\\|w-z\\| \\leq (L+2\\lambda^{-1}) \\|w-z\\| \\label{eq:fnor-lipschitz} \\end{align}\n%\nfor all $w, z \\in \\Rn$. Thus, setting $L_F := L+2\\lambda^{-1}$, combining the previous estimates, and using \\eqref{eq:def-sk-bar-sk}, we obtain\n%\n\\begin{align*} \\delta & < \\vert\\chi(z_{t_i}) - \\chi(z_{\\ell_i})\\vert \\leq L_F {\\|z_{\\ell_i}-z_{t_i}\\|} \\\\ & \\leq L_F \\sum_{k = t_i, \\, k \\in \\mathcal K}^{\\ell_i-1} \\|s_k\\| \\leq L_F \\sum_{k = t_i, \\, k \\in \\mathcal K}^{\\ell_i-1} \\Delta_k \\leq {\\frac{2L_F}{\\tau\\eta_1\\delta}}[\\mer(z_{t_i}) - \\mer(z_{\\ell_i})]. \\end{align*}\n%\nDue to $\\mer(z_k) \\to \\zeta$, the right hand side of the last inequality has to converge to zero which is a contradiction.\n\\end{proof}","sha256_utf8":"31b048eeea04543b9c0ba30d87d42602e13f6ef0a85a899e8858ad3df0ca016e"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"复合优化问题","source_units":[{"start":9218,"end":9440,"start_line":228,"end_line":232,"raw_tex":"In this paper, we develop and analyze a novel normal-map based second-order approach for the composite optimization problem\n%\n\\begin{align} \\label{eq1-1}\n\\min\\limits_{x\\in\\mathbb{R}^n}~\\psi(x):=f(x)+\\varphi(x),\n\\end{align}","sha256_utf8":"c3a96ea9eb5f526832788676c3e37d26c0c5ffb1adfec83214abb2565f2f5a64"},{"start":9443,"end":9695,"start_line":234,"end_line":234,"raw_tex":"where $f:\\mathbb{R}^n\\rightarrow\\mathbb{R}$ is a continuously differentiable (not necessarily convex) function and $\\varphi:\\mathbb{R}^n\\rightarrow(-\\infty,\\infty]$ is a convex, lower semicontinuous (lsc), and proper (not necessarily smooth) mapping. \n","sha256_utf8":"9418c85803fb777d3350aca551874aa69f6941ae2b176b1286f83e3a74434e40"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"近端映射定义","source_units":[{"start":10770,"end":11142,"start_line":242,"end_line":242,"raw_tex":"where $\\oprox : \\Rn \\to \\Rn$, $\\prox{x} := \\argmin_{y \\in \\Rn} \\varphi(y) + \\frac{1}{2\\lambda} \\|x-y\\|^2$ denotes the well-known proximity operator of $\\varphi$, \\cite{Mor65}, and $\\lambda > 0$ is a positive parameter. Alternatively, the forward-backward scheme in \\eqref{eq:fbs-intro} can also be interpreted as a fixed-point procedure applied to the nonsmooth equation:\n","sha256_utf8":"cd32297af9019f5d7659b619ff1ef0176d43884ce9e0b4f0176c84a00a511fe3"}]},{"id":"S005","semantic_type":"notation","kind":"notation","title":"范数与矩阵序","source_units":[{"start":27615,"end":28186,"start_line":346,"end_line":346,"raw_tex":"By $\\iprod{\\cdot}{\\cdot}$ and $\\|\\cdot\\|$ we denote the standard Euclidean inner product and norm. For matrices, the norm $\\|\\cdot \\|$ is the standard spectral norm. The sets of symmetric and symmetric positive definite $n \\times n$ matrices are denoted by $\\mathbb S^n$ and $\\Spp$, respectively. For a given matrix $A \\in \\Spp$, we define the norm $\\|x\\|_A := \\sqrt{\\iprod{x}{Ax}}$. For two matrices $A, B \\in \\mathbb S^n$, we write $A\\succeq B$ if $A-B$ is positive semidefinite. We use $\\mathrm{ri}(S)$ to denote the relative interior of a convex set $S \\subset \\Rn$.\n","sha256_utf8":"d5b9b98cc0d200070cca8d11958a4d19cb9f3062fee4c6635a65613876bf7ee5"}]},{"id":"S006","semantic_type":"derived_fact","kind":"derived_fact","title":"近端映射与包络性质","source_units":[{"start":31457,"end":31752,"start_line":376,"end_line":379,"raw_tex":"As mentioned, $\\oprox$ denotes the proximity operator of $\\varphi$ with respect to the scalar $\\lambda$. \nThe proximity operator is a {firmly nonexpansive mapping}, i.e.,\n%\n\\be \\label{eq:prox-nonexp} \\|\\prox{x}-\\prox{y}\\|^2\\leq\\langle x-y,\\prox{x}-\\prox{y} \\rangle \\quad \\forall~x,y \\in \\Rn. \\ee","sha256_utf8":"ddc48bd1fb7e1ea005b3170e1c3fc2ace00f31c498562de1b7d8d86ac45ed9f3"},{"start":31755,"end":32053,"start_line":381,"end_line":385,"raw_tex":"In particular, $\\oprox$ is globally Lipschitz continuous with constant 1. Moreover, the proximity operator can be characterized by the associated optimality conditions of its underlying optimization problem:\n%\n\\begin{align}\\label{eq1-3}\n\\prox{x}\\in x-\\lambda \\partial\\varphi(\\prox{x}). \n\\end{align}","sha256_utf8":"3042047f13d24c9187752f3525def449444f60eb1546be7f10d5697191a39d79"},{"start":32056,"end":32598,"start_line":387,"end_line":387,"raw_tex":"We will also work with the Moreau envelope $\\env{x}:=\\min_y\\varphi(y)+\\frac{1}{2\\lambda}\\|x-y\\|^2$. It is well known that $\\oenv$ is convex and continuously differentiable and its gradient is given by $\\nabla\\env{x}=\\frac{1}{\\lambda}(x-\\prox{x})$. Let us further note that due to the convexity of $\\vp$ and differentiability of $f$, Clarke's subdifferential $\\partial \\psi$ coincides with the regular and limiting subdifferential of $\\psi$. We refer to \\cite{Mor65,rockafellar2009variational,BauCom11} for additional details and background. \n","sha256_utf8":"42b9f8133910ab602b4276e0f4fb4a1dd4ea14a87470e09aaa25f75486f9f533"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"正规映射定义","source_units":[{"start":32811,"end":32915,"start_line":392,"end_line":392,"raw_tex":"\\begin{align} \\label{normal_eq} \\Fnor{z} := \\nabla f(\\prox{z})+{\\lambda^{-1}}(z-\\prox{z}) =0 \\end{align}","sha256_utf8":"804cf7ff3849ce24c591da0bf0e431f1d9d116a2db0b10f14c5d105961a3745a"}]},{"id":"S008","semantic_type":"algorithm","kind":"algorithm","title":"信赖域模型与提升步","source_units":[{"start":43453,"end":43611,"start_line":526,"end_line":528,"raw_tex":"\\begin{align}\\label{eq3-4}\n  \\min_{q}~m_k(q):=\\iprod{\\Fnor{z_k}}{D_kq} + \\frac{1}{2} \\iprod{M_kq}{D_kq} \\quad \\text{s.t.} \\quad \\|q\\|\\leq\\Delta_k,\n\\end{align}","sha256_utf8":"c146931df9c25e6b193082159c937b88277bc24cbc1b742ade059bf04b53cf09"},{"start":43614,"end":43973,"start_line":530,"end_line":531,"raw_tex":"where $\\Delta_k$ is the trust region radius. %As mentioned, the Steihaug-CG method can be utilized to solve the subproblem \\eqref{eq3-4} inexactly. \nLet $\\bar q_k$ denote an approximate solution of the subproblem \\eqref{eq3-4} returned by the Steihaug-CG method. Motivated by our previous discussion, we can then generate a lifted and rescaled step $s_k$ via\n","sha256_utf8":"b5544c653dcafa0050e413d1797ac00ce93fec9aa5d5385d5ccaed2d7cab82e1"},{"start":43975,"end":44149,"start_line":533,"end_line":533,"raw_tex":"\\be \\label{eq:def-sk-bar-sk} \\bar s_k = \\bar q_k - \\lambda  (\\Fnor{z_k}+M_k\\bar q_k) \\quad \\text{and} \\quad s_k = \\min\\left\\{1,{\\Delta_k}/{\\|\\bar s_k\\|}\\right\\} \\bar s_k, \\ee","sha256_utf8":"37d7deaa7d780cce18a46da6b3ee37ee147e472bb6cd70353122b7facee9581e"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"评价函数定义","source_units":[{"start":49680,"end":50022,"start_line":573,"end_line":577,"raw_tex":"\\begin{defn} \\label{def:mer}\nLet $\\lambda > 0$, $\\tau \\in (0,1)$ be given. We define the merit function\n$$ \\mer: \\Rn \\to \\R, \\quad \\mer(z):=\\psi(\\prox{z})+\\frac{\\tau\\lambda}{2}\\|\\Fnor{z}\\|^2. $$\n%where $\\tau \\in [0,1-\\nu)$, $\\nu \\in [0,1)$. %Furthermore, the criticality measure $\\chi$ is set to $\\chi(z):=\\|\\Fnor{z}\\|_{\\lambda }$.\n\\end{defn}","sha256_utf8":"caa867d3a2cbbe284739924bd23e2759d65c992f7c3be2ef87a0e06ff19d03b5"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"临界性度量","source_units":[{"start":50887,"end":51099,"start_line":589,"end_line":591,"raw_tex":"Here, $\\chi:\\mathbb{R}^n\\rightarrow\\mathbb{R}_{+}$ is a continuous criticality measure. Throughout this work, we will use with the criticality measure %$\\chi(z) := \\|\\Fnor{z}\\|$.\n%\n\\[ \\chi(z) := \\|\\Fnor{z}\\|. \\]\n","sha256_utf8":"7a951ffb4b18b650ff909c4143dc5cc6844a1471496a90d5c24c848264dcbd1e"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"实际下降量","source_units":[{"start":50391,"end":50531,"start_line":580,"end_line":582,"raw_tex":"Based on \\cref{def:mer}, we define the actual reduction term $\\mathrm{ared}_k$ via:  \n%\n\\[ \\mathrm{ared}_k := \\mer(z_k) - \\mer(z_k+s_k). \\]\n","sha256_utf8":"b58fba0dce2f93a904ca8c2dc6a03fa586656e1b28aafc83b04d10f85ebf9cc4"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"预测下降量","source_units":[{"start":51325,"end":51571,"start_line":596,"end_line":596,"raw_tex":"\\begin{align}  \\label{eq:pred} \\mathrm{pred}(z,s,\\Delta,\\nu) & := \\frac{\\tau\\chi(z)}{2} \\min\\{\\lambda,\\Delta,\\lambda\\chi(z)\\} \\\\ & \\hspace{8ex} + \\frac{\\nu \\chi(z)}{\\min\\{\\Delta,\\lambda\\chi(z)\\}} \\|\\prox{z+s} - \\prox{z}\\|^2, \\nonumber \\end{align}","sha256_utf8":"3f048bb178b0d697670b7e23f78bb7418d65deb2f0eabc6c4c5a1b68decfbc4b"},{"start":51574,"end":52034,"start_line":598,"end_line":598,"raw_tex":"where $\\tau \\in (0,1)$ and $\\nu \\in [0,1)$ are given. In the next sections, we show that this choice of $\\rho_k$, $\\mathrm{ared}_k$, and $\\mathrm{pred}_k$ meets all the mentioned requirements. The full details of the method are presented in \\cref{algo2}. Notice that we allow the usage of approximate Hessian information $B_k \\approx \\nabla^2 f(\\prox{z_k})$. We call iteration $k$ successful if $z_k + s_k$ is accepted as new iterate, i.e., $z_{k+1}=z_k+s_k$.\n","sha256_utf8":"dfb2e5e95bd3716ed69ed1ea50eadd1bed8316573aaa7190f5d595f27822c141"}]},{"id":"S013","semantic_type":"algorithm","kind":"algorithm","title":"正规映射牛顿算法","source_units":[{"start":52036,"end":53361,"start_line":600,"end_line":619,"raw_tex":"\\begin{algorithm}[t]\n        \\caption{A Trust Region Normal Map Semismooth Newton Method}\n         \\label{algo2}\n        \\begin{algorithmic}[1]\n            \\Require Choose an initial point $z_0\\in\\mathbb{R}^n$, $B_0 \\in \\mathbb{S}^n$, $\\lambda>0$, and $\\{\\epsilon_k\\} \\subset \\R_{+}$. Set iteration $k=0$.\n            \\While{$F_{\\text{nor}}^{\\lambda}(z_k)\\neq0$}\n            \\State Choose $D_k\\in\\partial\\text{prox}_{\\varphi}^{\\lambda}(z_k)$ and set $M_k=B_kD_k+\\frac{1}{\\lambda}(I-D_k)$.\n            \\State Run the Steihaug-CG method with $S = D_k M_k$, $g = D_k \\Fnor{z_k}$, $\\Delta = \\Delta_k$, and $\\epsilon = \\epsilon_k \\geq 0$ returning $\\bar q_k = q$;\n            \\State Set $\\bar{s}_k=\\bar q_k-\\lambda (F_{\\text{nor}}^{\\lambda}(z_k)+M_k\\bar q_k)$ and $s_k=\\min\\{1,\\frac{\\Delta_k}{\\|\\bar{s}_k\\|}\\}\\bar{s}_k$;\n            %\\EndIf\n             \\If{$\\rho_k=\\frac{\\mer(z_k)-\\mer(z_k+s_k)}{\\mathrm{pred}(z_k,s_k,\\Delta_k,\\nu_k)} < \\eta_1$}\n             \\State Set $z_{k+1}=z_{k}$ and $B_{k+1} = B_k$;\n             \\Else\n             \\State Set $z_{k+1}=z_k+s_k$ and choose $B_{k+1} \\approx \\nabla^2 f(\\prox{z_{k+1}})$;\n             \\EndIf\n             \\State Set $\\Delta_{k+1}$ based on $\\rho_k$ by invoking \\cref{algo3};\n             \\State $k\\gets k+1$;\n            \\EndWhile\n        \\end{algorithmic}\n    \\end{algorithm}","sha256_utf8":"d63e1afe81fe95683e4dbcb623440685608cd2f9309dd92f61f5aeb649faefbf"}]},{"id":"S014","semantic_type":"algorithm","kind":"algorithm","title":"信赖域半径更新","source_units":[{"start":55682,"end":56848,"start_line":642,"end_line":657,"raw_tex":"\\begin{algorithm}[t]\n        \\caption{Update of the Trust Region Radius}\n        \\label{algo3}\n        \\begin{algorithmic}[1]  \n            \\Require Input: $\\Delta_k,\\rho_k$. Let $\\eta_1\\leq\\eta_2<1$ and $0<\\gamma_0<\\gamma_1<1<\\gamma_2$, and $\\Delta_{\\min} \\geq 0$ be fixed. \\vspace{0.1ex}\n           \\State Select $\\Delta_{k+1}\\in \\begin{cases} (\\gamma_0\\Delta_k,\\gamma_1\\Delta_k] & \\text{if }  \\rho_k < \\eta_1, \\\\ [\\gamma_1\\Delta_k,\\max\\{\\Delta_{\\min},\\Delta_k\\}]\\cap[\\Delta_{\\min},\\infty) & \\text{if } \\rho_k\\in[\\eta_1,\\eta_2)\\\\ (\\Delta_k,\\max\\{\\Delta_{\\min},\\gamma_2\\Delta_k\\}]\\cap[\\Delta_{\\min},\\infty) & \\text{if } \\rho_k \\geq \\eta_2 \\end{cases}$;\n%           \\If{$\\rho_k < \\eta_1$}\n%           \\State Set $\\Delta_{k+1}\\in (\\gamma_0\\Delta_k,\\gamma_1\\Delta_k]$.\n%           \\ElsIf{$\\rho_k\\in[\\eta_1,\\eta_2)$}\n%           \\State Set $\\Delta_{k+1}\\in[\\gamma_1\\Delta_k,\\max\\{\\Delta_{\\min},\\Delta_k\\}]\\cap[\\Delta_{\\min},\\infty)$;\n%           \\Else\n%          \\State Set $\\Delta_{k+1}\\in(\\Delta_k,\\max\\{\\Delta_{\\min},\\gamma_2\\Delta_k\\}]\\cap[\\Delta_{\\min},\\infty)$;\n%           \\EndIf\n           \\State Return $\\Delta_{k+1}$;\n        \\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"7abd34442a870b830596db76dbd3ff4d4d3ee84aa149c72671ba21138715c36f"}]},{"id":"S015","semantic_type":"algorithm","kind":"algorithm","title":"截断共轭梯度法","source_units":[{"start":56858,"end":58300,"start_line":660,"end_line":685,"raw_tex":"\\begin{algorithm}[th]\n        \\caption{The Steihaug-CG Method}\n          \\label{algo1}\n        \\begin{algorithmic}[1]  \n            \\Require Input: $S \\in \\R^{n\\times n}$, $g \\in \\Rn$, $\\epsilon, \\Delta \\geq 0$. Set $q_0=0$, $r_0=g$, $p_0 = -g$ and $i=0$.\n            \\If{$\\|r_0\\|<\\epsilon$}\n            \\State return $q = q_0$;\n            \\EndIf\n            \\While{$i\\leq n-1$}\n                    \\If{$\\iprod{p_i}{Sp_i}\\leq0$}\n                        \\State Compute $\\alpha_{i}$ such that $\\alpha_i=\\argmin\\limits_{\\|q_i+\\alpha_ip_i\\|=\\Delta}m(q_i+\\alpha_ip_i)$ and return $q = q_i+\\alpha_ip_i$;\n                        \\Else\n                        \\State Set $\\alpha_i=\\frac{\\langle r_i,r_i \\rangle}{\\langle p_i,Sp_i\\rangle}$ and $q_{i+1}=q_i+\\alpha_ip_i$;\n                        \\If{$\\|q_{i+1}\\|\\geq\\Delta$}\n                        \\State Reset $\\alpha_i$ such that $\\alpha_i\\geq0$ and $\\|q_i+\\alpha_ip_i\\|=\\Delta$ and return $q = q_i+\\alpha_ip_i$;\n                        \\EndIf\n                        \\State Set $r_{i+1}=r_i+\\alpha_iSp_i$;\n                        \\If{$\\|r_{i+1}\\|<\\epsilon$}\n                        \\State Return $q = q_{i+1}$;\n                        \\EndIf\n                        \\State Set $\\beta_{i+1}=\\frac{\\|r_{i+1}\\|^2}{\\|r_i\\|^2}$ and $p_{i+1}=-r_{i+1}+\\beta_{i+1}p_i$;\n                    \\EndIf\n                    \\State $i\\gets i+1$\n            \\EndWhile\n        \\end{algorithmic}\n      \\end{algorithm}","sha256_utf8":"b3acd1542b4555b6ccbac2893b4affbd100f84ee62380123db6d3d43efe1a5b2"}]},{"id":"S016","semantic_type":"assumption","kind":"assumption","title":"光滑性与下界假设","source_units":[{"start":60725,"end":61122,"start_line":726,"end_line":731,"raw_tex":"\\begin{assumption}\\label{assum2-1} We consider the conditions:\n\\begin{enumerate}[label=\\textup{\\textrm{(A.\\arabic*)}},topsep=0pt,itemsep=0ex,partopsep=0ex,leftmargin=8ex]\n  \\item \\label{A1} The gradient $\\nabla f$ is Lipschitz continuous on $\\dom{\\vp}$ with modulus $L$.\n  \\item \\label{A2} The objective function $\\psi$ is lower bounded on $\\dom{\\partial\\varphi}$.\n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"43118bd7928190dfd79e40e2281fa2dcd98f75e8f8d093394623fb8f7d858b4f"}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","title":"参数与曲率假设","source_units":[{"start":61554,"end":62102,"start_line":737,"end_line":748,"raw_tex":"\\begin{assumption}\n\\label{assum2-2}  We assume: \n\\begin{enumerate}[label=\\textup{\\textrm{(B.\\arabic*)}},topsep=0pt,itemsep=0ex,partopsep=0ex,leftmargin=8ex]\n\\item \\label{B1} \nThe parameters $\\tau$ and $\\nu_k$ satisfy the conditions: \n%\n\\[ \\nu \\in [0,1),\\quad 0\\leq\\nu_k\\leq\\nu, \\quad \\text{and} \\quad (L^2\\lambda^2+2)\\tau< 2(1-\\nu). \\]\n%\n\\item \\label{B2} It holds that $\\sum_{k=0}^\\infty ({1 + \\|B_k\\|})^{-1} = \\infty$. \n\\item \\label{B3} There is $\\kappa_B > 0$ such that $\\|B_k\\| \\leq \\kappa_B$ for all $k \\in \\N$.\n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"f30985f68ecb5ab05a76e6cc4c437c6e788649abb2b2b79c21949ae1e6ba1999"}]},{"id":"S018","semantic_type":"notation","kind":"notation","title":"成功指标与迭代记号","source_units":[{"start":62105,"end":62577,"start_line":750,"end_line":750,"raw_tex":"Condition \\ref{B2} holds, e.g., if the matrices $\\{B_k\\}$ satisfy $\\|B_k\\| \\leq c_{B_1} + c_{B_2} k$ for all $k$ and some $c_{B_1}, c_{B_2} > 0$. Hence, \\ref{B2} is generally weaker than \\ref{B3}. Let $\\{j_i\\}$ be an increasing sequence enumerating the indices of the accepted iterates in step 8 of \\cref{algo2}. We then define the set of all successful iterations as $\\mathcal S := \\{j_i : i \\geq 0\\} = \\{k \\in \\N : \\rho_k \\geq \\eta_1\\}$. We will also use the notations \n","sha256_utf8":"5654edbced9b68dc1cc08354452004374f2352772210a91b6c97c1ae6c323216"},{"start":62579,"end":62712,"start_line":752,"end_line":752,"raw_tex":"\\be \\label{eq:muk-xk} {\\mu}_k := \\nu_k \\chi(z_k) \\min\\{\\Delta_k,\\lambda\\chi(z_k)\\}^{-1} \\quad \\text{and} \\quad x_k := \\prox{z_k}, \\ee","sha256_utf8":"561e883b6696b68d6d793406ed8ae036a11257f3880b223cacafc396bec0fed3"},{"start":62715,"end":62813,"start_line":754,"end_line":754,"raw_tex":"where $\\{z_k\\}$ and $\\{\\Delta_k\\}$ denote the sequences generated by \\cref{algo2}. %Furthermore, \n","sha256_utf8":"0f49c2234703f3e3c697a40fc39025b8496226d837a4474c4063fbd9f45996c1"}]},{"id":"S019","semantic_type":"lemma","kind":"lemma","title":"广义导数矩阵结构","source_units":[{"start":38719,"end":39070,"start_line":457,"end_line":463,"raw_tex":"\\begin{lemma} \\label{prop2-5} \nLet $D\\in\\partial\\prox{x} \\subseteq \\R^{n \\times n}$, $x \\in \\Rn$, be an arbitrary generalized derivative. Then, it holds that: \n\\begin{itemize}\n\\item[\\rmn{(i)}] Both $D$ and $I-D$ are symmetric and positive semidefinite matrices.\n\\item[\\rmn{(ii)}] The matrix $D(I-D)$ is positive semidefinite.\n\\end{itemize}\n\\end{lemma}","sha256_utf8":"4599a3adef05e7e133109779edd456ee78889a50739e86d219ab42d004442009"}]},{"id":"S020","semantic_type":"external_fact","kind":"external_fact","title":"近端广义雅可比标准性质","source_units":[],"declared_external":true,"declaration_text":["For a proper lower semicontinuous convex function φ and λ>0, every Clarke generalized Jacobian D∈∂prox_{λφ}(x) has D and I−D symmetric positive semidefinite, and D(I−D) positive semidefinite. This is the standard result cited at line 454: MenSunZha05, Proposition 1; milzarek2016numerical, Lemma 3.3.5."]},{"id":"S021","semantic_type":"lemma","kind":"lemma","title":"提升步残差关系","source_units":[{"start":41968,"end":42424,"start_line":502,"end_line":505,"raw_tex":"\\begin{lemma}\n\\label{lemma2-1} Let $B, D \\in \\mathbb{S}^n$ be symmetric matrices and let $r \\in \\Rn$ and $\\epsilon \\geq 0$ be given. %Suppose that $D$ is symmetric %and $\\Lambda(I-D)$ are symmetric and positive semidefinite \nLet us set $M := B D + \\frac{1}{\\lambda}(I-D)$ and assume that $y$ satisfies the condition $\\|D(My + r)\\| \\leq \\epsilon$. Then, setting $x = y - \\lambda (My + r)$, it follows $\\|Mx + r\\| \\leq \\|I-\\lambda B\\| \\epsilon$. \n\\end{lemma}","sha256_utf8":"3f228539841cbb741fc0dfcdbecea2db94970c294aaad524eead23a940e8a39a"}]},{"id":"S022","semantic_type":"proof","kind":"proof","title":"提升步关系证明","source_units":[{"start":42425,"end":42723,"start_line":506,"end_line":517,"raw_tex":"\\begin{proof}\nApplying the definition of $M$, we directly obtain\n\\begin{align}\\label{eq3-3}\n\\lambda M =\\lambda BD +I-D=(\\lambda B -I)D+I\n\\end{align}\n%\nand\n%\n%\\begin{align*}\n$ \\|Mx+r\\|=\\|My-\\lambda M (My+r)+r\\| =\\|(I-\\lambda B)D(My+r)\\| \\leq  \\|I - \\lambda B \\| \\epsilon$. \n%\\end{align*}\n\\end{proof}","sha256_utf8":"cf870412e28956afbfec3e8c1ca3049bd100f371b521bbe56479b06761a0d2de"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"展开牛顿矩阵","source_units":[{"start":42425,"end":42574,"start_line":506,"end_line":510,"raw_tex":"\\begin{proof}\nApplying the definition of $M$, we directly obtain\n\\begin{align}\\label{eq3-3}\n\\lambda M =\\lambda BD +I-D=(\\lambda B -I)D+I\n\\end{align}\n","sha256_utf8":"4a2f15e5bda97fda5fc8517e357b5bb772d6e580cb9616c3a8b450e1d196b22c"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"估计提升步残差","source_units":[{"start":42576,"end":42723,"start_line":512,"end_line":517,"raw_tex":"and\n%\n%\\begin{align*}\n$ \\|Mx+r\\|=\\|My-\\lambda M (My+r)+r\\| =\\|(I-\\lambda B)D(My+r)\\| \\leq  \\|I - \\lambda B \\| \\epsilon$. \n%\\end{align*}\n\\end{proof}","sha256_utf8":"2a16f960245c78b053f781c7013c15d180d15007169ae9f474c0b4f5249e2267"}]},{"id":"S025","semantic_type":"lemma","kind":"lemma","title":"评价函数下降估计","source_units":[{"start":62883,"end":63921,"start_line":758,"end_line":767,"raw_tex":"\\begin{lemma}\n\\label{lemma2-5}\nSuppose that \\ref{A1} is satisfied and let $z, e \\in \\Rn$, $\\alpha \\in (0,1]$, {$\\nu \\in [0,1)$, and $\\tau \\in (0,1-\\nu)$} be given. Setting $d:=-\\Fnor{z}$ and $x := \\prox{z}$, it follows\n%\n\\begin{align*}\n\\mer(z+\\alpha\\lambda (d+e))-\\mer(z) & \\leq -\\frac{\\tau\\lambda\\alpha}{2}\\|\\Fnor{z}\\|^2 - \\frac{\\nu}{\\lambda\\alpha}\\|p_\\alpha - x\\|^2 + \\frac{\\tau\\lambda\\alpha}{2} \\|e\\|^2 \\\\ & \\hspace{4ex} + \\left[L\\tau\\lambda + 1 - \\tau\\right] \\alpha \\|e\\|\\|d+e\\| + C(\\alpha) \\|p_\\alpha-x\\|^2,\n%\\mer(z+\\alpha\\lambda (d+e))-\\mer(z) & \\\\ & \\hspace{-25ex} \\leq -\\frac{\\tau\\lambda\\alpha}{2}\\|\\Fnor{z}\\|^2 - \\frac{\\nu}{\\lambda\\alpha}\\|p_\\alpha - x\\|^2 + \\left[L\\tau + \\lambda(1 - \\tau)\\right] \\alpha \\|e\\|\\|d+e\\| \\\\ & \\hspace{-19ex} + \\frac{\\tau\\lambda\\alpha}{2} \\|e\\|^2 + C(\\alpha) \\|p_\\alpha-x\\|^2,\n\\end{align*}\nwhere $p_\\alpha := \\prox{z+\\alpha\\lambda (d+e)}$ and $C(\\alpha) := (\\frac{L^2\\lambda^2+2}{2\\lambda} \\cdot \\tau - \\frac{1-\\nu}{\\lambda})\\frac{1}{\\alpha} + L \\tau + \\frac{L}{2}-\\frac{\\tau}{2\\lambda}$.\n\\end{lemma}","sha256_utf8":"97346ffdc64d06f9834b13fdfcd707cf625cd152e2776d2d21f1b63b3487999c"}]},{"id":"S026","semantic_type":"proof","kind":"proof","title":"下降估计完整证明","source_units":[{"start":63924,"end":69416,"start_line":769,"end_line":804,"raw_tex":"\\begin{proof} Applying the Lipschitz continuity of $\\nabla f$ on $\\dom{\\vp}$ and $\\nabla \\env{z+\\alpha \\lambda (d+e)} = \\frac{1}{\\lambda}(z+\\alpha \\lambda  (d+e) - p_\\alpha) \\in \\partial \\vp(p_\\alpha)$, see \\eqref{eq1-3}, we have\n%\n\\begin{align} \\label{eq:here-esti1} %\\psi(p_\\alpha) - \\psi(x) & \\\\ & \\hspace{-8ex} \n\\psi(p_\\alpha) - \\psi(x) & \\leq  \\iprod{\\nabla f(x)}{p_\\alpha - x}  + \\frac{L}{2} \\|p_\\alpha - x\\|^2 + \\frac{1}{\\lambda}\\iprod{z+\\alpha \\lambda  (d+e) - p_\\alpha}{p_\\alpha - x} \\nonumber \\\\ & = \\iprod{\\Fnor{z}+\\alpha (d+e)}{p_\\alpha - x} + \\left[\\frac{L}{2}-\\frac{1}{\\lambda}\\right] \\|p_\\alpha - x\\|^2 \\nonumber \\\\ & = (1-\\alpha) \\iprod{\\Fnor{z} - e}{p_\\alpha - x} + \\iprod{e}{p_\\alpha - x} + \\left[\\frac{L}{2}-\\frac{1}{\\lambda}\\right] \\|p_\\alpha - x\\|^2. \\end{align}\n%\nMoreover, it holds that\n%\n\\begingroup\n\\allowdisplaybreaks\n\\begin{align*} \\frac{\\lambda}{2} \\|\\Fnor{z+\\alpha\\lambda (d+e)}\\|^2 & = \\frac{\\lambda}{2} \\|\\nabla f(p_\\alpha) + \\lambda^{-1}(z+\\alpha\\lambda(d+e)-p_\\alpha)\\|^2 \\\\ & = \\frac{\\lambda}{2} \\|(1-\\alpha)\\Fnor{z} + \\nabla f(p_\\alpha) - \\nabla f(x) - {\\lambda^{-1}}(p_\\alpha - x) + \\alpha e\\|^2 \\\\ & = \\frac{\\lambda(1-\\alpha)^2}{2} \\|\\Fnor{z}\\|^2 + (1-\\alpha)\\iprod{\\Fnor{z}}{\\lambda [\\nabla f(p_\\alpha)- \\nabla f(x)]-[p_\\alpha-x]}  \\\\ & \\hspace{4ex} + (1-\\alpha)\\alpha \\iprod{\\Fnor{z}}{\\lambda e} +  \\iprod{\\nabla f(p_\\alpha)-\\nabla f(x)}{x-p_\\alpha +  \\lambda \\alpha e}  \\\\ & \\hspace{4ex} + \\frac{\\lambda}{2}\\|\\nabla f(p_\\alpha)-\\nabla f(x)\\|^2 + \\frac{\\lambda\\alpha^2}{2} \\|e\\|^2  - \\alpha \\iprod{e}{p_\\alpha - x}  + \\frac{1}{2\\lambda} \\|p_\\alpha - x\\|^2  \\\\ & \\hspace{0ex} \\leq \\frac{\\lambda(1-\\alpha)^2}{2} \\|\\Fnor{z}\\|^2 - (1-\\alpha)\\iprod{\\Fnor{z} - e}{p_\\alpha - x} - \\iprod{e}{p_\\alpha - x} \\\\ & \\hspace{4ex}  + (1-\\alpha)\\alpha \\iprod{\\Fnor{z}}{\\lambda e} + \\iprod{(1-\\alpha)\\Fnor{z}+\\alpha e}{\\lambda [\\nabla f(p_\\alpha)-\\nabla f(x)]}  \\\\ & \\hspace{4ex} + \\frac{\\lambda\\alpha^2}{2} \\|e\\|^2 + \\left[ \\frac{L^2\\lambda}{2} + L + \\frac{1}{2\\lambda}\\right] \\|p_\\alpha - x\\|^2 \\\\ & \\hspace{0ex} = \\frac{\\lambda}{2}(1-\\alpha) \\|\\Fnor{z}\\|^2 + (1-\\alpha) \\iprod{d+e}{p_\\alpha - x} - \\iprod{e}{p_\\alpha - x} + \\frac{\\lambda\\alpha}{2} \\|e\\|^2 \\\\ & \\hspace{4ex} - \\frac{\\lambda\\alpha}{2}(1-\\alpha) \\|d+e\\|^2  + \\left[ \\frac{L^2\\lambda}{2} + L + \\frac{1}{2\\lambda}\\right] \\|p_\\alpha - x\\|^2 \\\\ & \\hspace{4ex} + \\iprod{(1-\\alpha)\\Fnor{z}+\\alpha e}{\\lambda [\\nabla f(p_\\alpha)-\\nabla f(x)]}, \n\\end{align*}\n\\endgroup\n%\nwhere we used the Lipschitz continuity of $\\nabla f$ and $2 \\iprod{\\Fnor{z}}{e} =  \\|\\Fnor{z}\\|^2 + \\|e\\|^2 - \\|d+e\\|^2$ in the last step. Next, applying Young's inequality, it follows\n%\n\\begin{align*} \\vert\\iprod{d+e}{\\nabla f(p_\\alpha) - \\nabla f(x)}\\vert & \\leq \\|d+e\\| \\cdot L \\|p_\\alpha - x\\|  \\leq \\frac{\\alpha}{2}  \\|d+e\\|^2 + \\frac{L^2}{2\\alpha} \\|p_\\alpha - x\\|^2 \\end{align*}\n%\\begin{align*} \\vert\\iprod{d+e}{\\lambda [\\nabla f(p_\\alpha) - \\nabla f(x)]}\\vert & \\leq \\sqrt{\\lambda}\\|d+e\\| \\cdot L\\sqrt{\\lambda} \\|p_\\alpha - x\\|  \\leq \\frac{\\lambda\\alpha}{2}  \\|d+e\\|^2 + \\frac{L^2\\lambda}{2\\alpha} \\|p_\\alpha - x\\|^2 \\end{align*}\n%\nand we have $\\|p_\\alpha - x\\|\\leq \\|z+\\alpha\\lambda (d+e)-z\\| = \\lambda\\alpha \\|d+e\\|$, $\\vert\\iprod{e}{p_\\alpha - x}\\vert \\leq \\lambda\\alpha \\|e\\|\\|d+e\\|$, and $\\vert\\iprod{e}{\\nabla f(p_\\alpha) - \\nabla f(x)}\\vert \\leq  L\\lambda\\alpha \\|e\\| \\|d+e\\|$. Hence, upon writing $(1-\\alpha)\\Fnor{z}+\\alpha e = -(1-\\alpha)(d+e)+e$, this implies\n%\n\\begin{align} \\label{eq:here-esti2}\n \\frac{\\lambda}{2} \\|\\Fnor{z+\\alpha\\lambda (d+e)}\\|^2 & \\leq \\frac{\\lambda}{2}(1-\\alpha) \\|\\Fnor{z}\\|^2 + (1-\\alpha) \\iprod{d+e}{p_\\alpha - x}  \\\\ & \\hspace{4ex} - \\iprod{e}{p_\\alpha - x} + \\frac{\\lambda\\alpha}{2} \\|e\\|^2  + \\left[ \\frac{L^2\\lambda}{2\\alpha} + L + \\frac{1}{2\\lambda}\\right] \\|p_\\alpha - x\\|^2 + L\\lambda^2\\alpha  \\|e\\|\\|d+e\\|. \\nonumber\n \\end{align} \n%\\begin{align*}\n% \\frac{\\lambda}{2} \\|\\Fnor{z+\\alpha\\lambda (d+e)}\\|^2 & \\\\ & \\hspace{-28ex} \\leq \\frac{\\lambda}{2}(1-\\alpha) \\|\\Fnor{z}\\|^2 + (1-\\alpha) \\iprod{d+e}{p_\\alpha - x} - \\iprod{e}{p_\\alpha - x} + \\frac{\\alpha\\lambda}{2} \\|e\\|^2 \\\\ & \\hspace{-24ex}  + \\left[ \\frac{\\lambda L^2}{2\\alpha} + L\\right] \\|p_\\alpha - x\\|^2 + \\frac{1}{2\\lambda} \\|p_\\alpha - x\\|^2 + \\alpha\\lambda L \\|e\\|\\|d+e\\|.\n% \\end{align*} \nFurthermore, by the firm nonexpansiveness of the proximity operator, \\eqref{eq:prox-nonexp}, we have\n%\n\\begin{equation} \\label{eq:here-esti3} -\\iprod{d+e}{p_\\alpha - x} = - \\frac{1}{\\lambda\\alpha}\\iprod{z + \\lambda\\alpha (d+e) - z}{p_\\alpha - x} \\leq - \\frac{1}{\\lambda\\alpha} \\|p_\\alpha - x\\|^2. \\end{equation}\n%\nCombining the estimates \\eqref{eq:here-esti1}, \\eqref{eq:here-esti2}, and \\eqref{eq:here-esti3}, we finally obtain\n%\n\\begin{align*}\n\\mer(z+\\alpha\\lambda (d+e))-\\mer(z) + \\frac{\\tau\\lambda\\alpha}{2}\\|\\Fnor{z}\\|^2& \\\\ &  \\hspace{-40ex} =\\psi(p_\\alpha) - \\psi(x) +\\frac{\\tau\\lambda}{2}\\|\\Fnor{z+\\alpha\\lambda (d+e)}\\|^2-\\frac{\\tau\\lambda}{2}(1-\\alpha)\\|\\Fnor{z}\\|^2 \\\\ & \\hspace{-40ex} \\leq (1-\\tau)\\langle e,p_\\alpha-x\\rangle +  L\\tau\\lambda^2\\alpha \\|e\\|\\|d+e\\| +\\frac{\\tau\\lambda\\alpha}{2}\\|e\\|^2 \\\\ & \\hspace{-36ex} + \\left[ \\frac{L^2\\lambda \\tau}{2\\alpha} + L \\tau + \\frac{\\tau}{2\\lambda} + \\frac{L}{2}-\\frac{1}{\\lambda} - \\frac{(1-\\tau)(1-\\alpha)}{\\lambda\\alpha} \\right]\\|p_\\alpha -x\\|^2 \\\\ & \\hspace{-40ex} \\leq  - \\left[\\frac{\\nu}{\\lambda\\alpha} - C(\\alpha)\\right] \\|p_\\alpha-x\\|^2 + \\left[L\\tau\\lambda + 1-\\tau \\right]\\lambda\\alpha \\|e\\|\\|d+e\\| + \\frac{\\tau\\lambda\\alpha}{2} \\|e\\|^2,\n\\end{align*}\nas desired. \\end{proof}","sha256_utf8":"1d76e46e03dab2a4cd6d6ca20f929f91be5834a6873961d6177f05b9dc946caf"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"估计复合目标变化","source_units":[{"start":63924,"end":64708,"start_line":769,"end_line":772,"raw_tex":"\\begin{proof} Applying the Lipschitz continuity of $\\nabla f$ on $\\dom{\\vp}$ and $\\nabla \\env{z+\\alpha \\lambda (d+e)} = \\frac{1}{\\lambda}(z+\\alpha \\lambda  (d+e) - p_\\alpha) \\in \\partial \\vp(p_\\alpha)$, see \\eqref{eq1-3}, we have\n%\n\\begin{align} \\label{eq:here-esti1} %\\psi(p_\\alpha) - \\psi(x) & \\\\ & \\hspace{-8ex} \n\\psi(p_\\alpha) - \\psi(x) & \\leq  \\iprod{\\nabla f(x)}{p_\\alpha - x}  + \\frac{L}{2} \\|p_\\alpha - x\\|^2 + \\frac{1}{\\lambda}\\iprod{z+\\alpha \\lambda  (d+e) - p_\\alpha}{p_\\alpha - x} \\nonumber \\\\ & = \\iprod{\\Fnor{z}+\\alpha (d+e)}{p_\\alpha - x} + \\left[\\frac{L}{2}-\\frac{1}{\\lambda}\\right] \\|p_\\alpha - x\\|^2 \\nonumber \\\\ & = (1-\\alpha) \\iprod{\\Fnor{z} - e}{p_\\alpha - x} + \\iprod{e}{p_\\alpha - x} + \\left[\\frac{L}{2}-\\frac{1}{\\lambda}\\right] \\|p_\\alpha - x\\|^2. \\end{align}\n","sha256_utf8":"5d377a339f2d154b3349d756d39681d249f36adbdbcd676f4d7e7e1834632cbb"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"展开正规残差平方","source_units":[{"start":64710,"end":66374,"start_line":774,"end_line":780,"raw_tex":"Moreover, it holds that\n%\n\\begingroup\n\\allowdisplaybreaks\n\\begin{align*} \\frac{\\lambda}{2} \\|\\Fnor{z+\\alpha\\lambda (d+e)}\\|^2 & = \\frac{\\lambda}{2} \\|\\nabla f(p_\\alpha) + \\lambda^{-1}(z+\\alpha\\lambda(d+e)-p_\\alpha)\\|^2 \\\\ & = \\frac{\\lambda}{2} \\|(1-\\alpha)\\Fnor{z} + \\nabla f(p_\\alpha) - \\nabla f(x) - {\\lambda^{-1}}(p_\\alpha - x) + \\alpha e\\|^2 \\\\ & = \\frac{\\lambda(1-\\alpha)^2}{2} \\|\\Fnor{z}\\|^2 + (1-\\alpha)\\iprod{\\Fnor{z}}{\\lambda [\\nabla f(p_\\alpha)- \\nabla f(x)]-[p_\\alpha-x]}  \\\\ & \\hspace{4ex} + (1-\\alpha)\\alpha \\iprod{\\Fnor{z}}{\\lambda e} +  \\iprod{\\nabla f(p_\\alpha)-\\nabla f(x)}{x-p_\\alpha +  \\lambda \\alpha e}  \\\\ & \\hspace{4ex} + \\frac{\\lambda}{2}\\|\\nabla f(p_\\alpha)-\\nabla f(x)\\|^2 + \\frac{\\lambda\\alpha^2}{2} \\|e\\|^2  - \\alpha \\iprod{e}{p_\\alpha - x}  + \\frac{1}{2\\lambda} \\|p_\\alpha - x\\|^2  \\\\ & \\hspace{0ex} \\leq \\frac{\\lambda(1-\\alpha)^2}{2} \\|\\Fnor{z}\\|^2 - (1-\\alpha)\\iprod{\\Fnor{z} - e}{p_\\alpha - x} - \\iprod{e}{p_\\alpha - x} \\\\ & \\hspace{4ex}  + (1-\\alpha)\\alpha \\iprod{\\Fnor{z}}{\\lambda e} + \\iprod{(1-\\alpha)\\Fnor{z}+\\alpha e}{\\lambda [\\nabla f(p_\\alpha)-\\nabla f(x)]}  \\\\ & \\hspace{4ex} + \\frac{\\lambda\\alpha^2}{2} \\|e\\|^2 + \\left[ \\frac{L^2\\lambda}{2} + L + \\frac{1}{2\\lambda}\\right] \\|p_\\alpha - x\\|^2 \\\\ & \\hspace{0ex} = \\frac{\\lambda}{2}(1-\\alpha) \\|\\Fnor{z}\\|^2 + (1-\\alpha) \\iprod{d+e}{p_\\alpha - x} - \\iprod{e}{p_\\alpha - x} + \\frac{\\lambda\\alpha}{2} \\|e\\|^2 \\\\ & \\hspace{4ex} - \\frac{\\lambda\\alpha}{2}(1-\\alpha) \\|d+e\\|^2  + \\left[ \\frac{L^2\\lambda}{2} + L + \\frac{1}{2\\lambda}\\right] \\|p_\\alpha - x\\|^2 \\\\ & \\hspace{4ex} + \\iprod{(1-\\alpha)\\Fnor{z}+\\alpha e}{\\lambda [\\nabla f(p_\\alpha)-\\nabla f(x)]}, \n\\end{align*}\n\\endgroup\n","sha256_utf8":"e327741fd8b7e979b14356232f67005a384128be8655f930b48c87ca26440157"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"控制残差展开交叉项","source_units":[{"start":66376,"end":67760,"start_line":782,"end_line":791,"raw_tex":"where we used the Lipschitz continuity of $\\nabla f$ and $2 \\iprod{\\Fnor{z}}{e} =  \\|\\Fnor{z}\\|^2 + \\|e\\|^2 - \\|d+e\\|^2$ in the last step. Next, applying Young's inequality, it follows\n%\n\\begin{align*} \\vert\\iprod{d+e}{\\nabla f(p_\\alpha) - \\nabla f(x)}\\vert & \\leq \\|d+e\\| \\cdot L \\|p_\\alpha - x\\|  \\leq \\frac{\\alpha}{2}  \\|d+e\\|^2 + \\frac{L^2}{2\\alpha} \\|p_\\alpha - x\\|^2 \\end{align*}\n%\\begin{align*} \\vert\\iprod{d+e}{\\lambda [\\nabla f(p_\\alpha) - \\nabla f(x)]}\\vert & \\leq \\sqrt{\\lambda}\\|d+e\\| \\cdot L\\sqrt{\\lambda} \\|p_\\alpha - x\\|  \\leq \\frac{\\lambda\\alpha}{2}  \\|d+e\\|^2 + \\frac{L^2\\lambda}{2\\alpha} \\|p_\\alpha - x\\|^2 \\end{align*}\n%\nand we have $\\|p_\\alpha - x\\|\\leq \\|z+\\alpha\\lambda (d+e)-z\\| = \\lambda\\alpha \\|d+e\\|$, $\\vert\\iprod{e}{p_\\alpha - x}\\vert \\leq \\lambda\\alpha \\|e\\|\\|d+e\\|$, and $\\vert\\iprod{e}{\\nabla f(p_\\alpha) - \\nabla f(x)}\\vert \\leq  L\\lambda\\alpha \\|e\\| \\|d+e\\|$. Hence, upon writing $(1-\\alpha)\\Fnor{z}+\\alpha e = -(1-\\alpha)(d+e)+e$, this implies\n%\n\\begin{align} \\label{eq:here-esti2}\n \\frac{\\lambda}{2} \\|\\Fnor{z+\\alpha\\lambda (d+e)}\\|^2 & \\leq \\frac{\\lambda}{2}(1-\\alpha) \\|\\Fnor{z}\\|^2 + (1-\\alpha) \\iprod{d+e}{p_\\alpha - x}  \\\\ & \\hspace{4ex} - \\iprod{e}{p_\\alpha - x} + \\frac{\\lambda\\alpha}{2} \\|e\\|^2  + \\left[ \\frac{L^2\\lambda}{2\\alpha} + L + \\frac{1}{2\\lambda}\\right] \\|p_\\alpha - x\\|^2 + L\\lambda^2\\alpha  \\|e\\|\\|d+e\\|. \\nonumber\n \\end{align} \n","sha256_utf8":"160540fcd7c4f0faf5b63a7a51fbd4a913cf290bf0faf2a5a8fde8f590066ef0"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"利用坚定非扩张性","source_units":[{"start":68176,"end":68488,"start_line":795,"end_line":797,"raw_tex":"Furthermore, by the firm nonexpansiveness of the proximity operator, \\eqref{eq:prox-nonexp}, we have\n%\n\\begin{equation} \\label{eq:here-esti3} -\\iprod{d+e}{p_\\alpha - x} = - \\frac{1}{\\lambda\\alpha}\\iprod{z + \\lambda\\alpha (d+e) - z}{p_\\alpha - x} \\leq - \\frac{1}{\\lambda\\alpha} \\|p_\\alpha - x\\|^2. \\end{equation}\n","sha256_utf8":"8786e549ef84bf67faceeae1f6a0046c038e0c8e2aa1aeb0258158bef4636d74"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"合并得到评价函数下降","source_units":[{"start":68490,"end":69416,"start_line":799,"end_line":804,"raw_tex":"Combining the estimates \\eqref{eq:here-esti1}, \\eqref{eq:here-esti2}, and \\eqref{eq:here-esti3}, we finally obtain\n%\n\\begin{align*}\n\\mer(z+\\alpha\\lambda (d+e))-\\mer(z) + \\frac{\\tau\\lambda\\alpha}{2}\\|\\Fnor{z}\\|^2& \\\\ &  \\hspace{-40ex} =\\psi(p_\\alpha) - \\psi(x) +\\frac{\\tau\\lambda}{2}\\|\\Fnor{z+\\alpha\\lambda (d+e)}\\|^2-\\frac{\\tau\\lambda}{2}(1-\\alpha)\\|\\Fnor{z}\\|^2 \\\\ & \\hspace{-40ex} \\leq (1-\\tau)\\langle e,p_\\alpha-x\\rangle +  L\\tau\\lambda^2\\alpha \\|e\\|\\|d+e\\| +\\frac{\\tau\\lambda\\alpha}{2}\\|e\\|^2 \\\\ & \\hspace{-36ex} + \\left[ \\frac{L^2\\lambda \\tau}{2\\alpha} + L \\tau + \\frac{\\tau}{2\\lambda} + \\frac{L}{2}-\\frac{1}{\\lambda} - \\frac{(1-\\tau)(1-\\alpha)}{\\lambda\\alpha} \\right]\\|p_\\alpha -x\\|^2 \\\\ & \\hspace{-40ex} \\leq  - \\left[\\frac{\\nu}{\\lambda\\alpha} - C(\\alpha)\\right] \\|p_\\alpha-x\\|^2 + \\left[L\\tau\\lambda + 1-\\tau \\right]\\lambda\\alpha \\|e\\|\\|d+e\\| + \\frac{\\tau\\lambda\\alpha}{2} \\|e\\|^2,\n\\end{align*}\nas desired. \\end{proof}","sha256_utf8":"e152e4b87bf9c4eb44949d4b99944953a1c6a6c7681148b7970cb7a27aa6c23b"}]},{"id":"S032","semantic_type":"proof","kind":"proof","title":"有效步长阈值推导","source_units":[{"start":69418,"end":70297,"start_line":806,"end_line":815,"raw_tex":"\\begin{remark} \\label{remark:lwb-step-size} In \\cref{lemma2-5}, suppose that condition \\ref{B1} is additionally satisfied. Then, the factor $\\frac{L^2\\lambda^2+2}{2\\lambda} \\cdot \\tau - \\frac{1-\\nu}{\\lambda}$ in $C(\\alpha)$ is negative. In particular, defining\n%\n\\be \\label{eq:lwb-step-size} \\bar\\alpha = \\bar\\alpha(L,\\lambda,\\tau,\\nu):= \\begin{cases} 1 & \\text{if $(1-2L\\lambda)\\tau \\geq L\\lambda$,} \\\\ \\min\\left\\{1,\\frac{2(1-\\nu)-(L^2\\lambda^2+2)\\tau}{(1+2\\tau)L\\lambda+\\tau}\\right\\} & \\text{otherwise,} \\end{cases} \\ee\n%\nand for all $\\alpha \\in [0,\\bar\\alpha]$, we can infer\n%\n\\begin{align*} \\mer(z+\\alpha\\lambda (d+e))-\\mer(z) & \\leq -\\frac{\\tau\\lambda\\alpha}{2} \\chi(z)^2 - \\frac{\\nu}{\\lambda\\alpha} \\|p_\\alpha - x\\|^2 + \\frac{\\tau\\lambda\\alpha}{2} \\|e\\|^2 \\\\ & \\hspace{4ex} +\\left[L\\tau\\lambda  + 1 - \\tau \\right] \\lambda\\alpha \\|e\\| \\|d+e\\|  \\end{align*}\n%\n%\n\\end{remark}\n","sha256_utf8":"d66fbc31b4a0a9e6a608e32bce13103ea1d2c8c44ed76406cfe4e7be7ecd01a3"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"选取有效步长阈值","source_units":[{"start":69418,"end":69940,"start_line":806,"end_line":808,"raw_tex":"\\begin{remark} \\label{remark:lwb-step-size} In \\cref{lemma2-5}, suppose that condition \\ref{B1} is additionally satisfied. Then, the factor $\\frac{L^2\\lambda^2+2}{2\\lambda} \\cdot \\tau - \\frac{1-\\nu}{\\lambda}$ in $C(\\alpha)$ is negative. In particular, defining\n%\n\\be \\label{eq:lwb-step-size} \\bar\\alpha = \\bar\\alpha(L,\\lambda,\\tau,\\nu):= \\begin{cases} 1 & \\text{if $(1-2L\\lambda)\\tau \\geq L\\lambda$,} \\\\ \\min\\left\\{1,\\frac{2(1-\\nu)-(L^2\\lambda^2+2)\\tau}{(1+2\\tau)L\\lambda+\\tau}\\right\\} & \\text{otherwise,} \\end{cases} \\ee\n","sha256_utf8":"add5f446200e8bf05ba76ed35a4e5bf9521ee6e6cd5164d27528dee3941fc0ee"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"消去非正余项","source_units":[{"start":69942,"end":70297,"start_line":810,"end_line":815,"raw_tex":"and for all $\\alpha \\in [0,\\bar\\alpha]$, we can infer\n%\n\\begin{align*} \\mer(z+\\alpha\\lambda (d+e))-\\mer(z) & \\leq -\\frac{\\tau\\lambda\\alpha}{2} \\chi(z)^2 - \\frac{\\nu}{\\lambda\\alpha} \\|p_\\alpha - x\\|^2 + \\frac{\\tau\\lambda\\alpha}{2} \\|e\\|^2 \\\\ & \\hspace{4ex} +\\left[L\\tau\\lambda  + 1 - \\tau \\right] \\lambda\\alpha \\|e\\| \\|d+e\\|  \\end{align*}\n%\n%\n\\end{remark}\n","sha256_utf8":"d22fca6310f29bcfbb9af11ad3e46685b79a35cb832781e6339e947e524d8d3b"}]},{"id":"S035","semantic_type":"lemma","kind":"lemma","title":"小半径保证步接受","source_units":[{"start":70642,"end":71204,"start_line":819,"end_line":825,"raw_tex":"\\begin{lemma} \\label{lemma2-6}\nLet $\\{z_k\\}$, $\\{s_k\\}$, and $\\{\\Delta_k\\}$ be generated by \\cref{algo2} and suppose that the assumptions \\ref{A1} and \\ref{B1} are satisfied. Then, there exists a constant $\\hat c > 0$ that only depends on $\\bar \\alpha$, $\\eta_2$, $\\tau$, $\\lambda$, $L$ such that every iteration $k$ with \n%\n\\be \\label{eq:bd-del} \\Delta_k \\leq \\hat c_k \\chi(z_k), \\quad \\hat c_k := \\frac{\\hat c}{1+\\lambda\\|B_k\\|}, \\ee\n%\nis very successful, i.e., the condition \\eqref{eq:bd-del} implies $\\rho_k \\geq \\eta_2$ and $k \\in \\mathcal S$.  \n\\end{lemma}","sha256_utf8":"6adbc4af21339efd92dfe9dc3c87e66d4820ddf8e5c66a070b8d9f1a0c3e3e72"}]},{"id":"S036","semantic_type":"proof","kind":"proof","title":"小半径接受性完整证明","source_units":[{"start":71207,"end":75992,"start_line":827,"end_line":880,"raw_tex":"\\begin{proof} First, by \\cref{prop2-5}, we have $\\|D\\|\\leq 1$ for all $D \\in \\partial \\prox{z}$ and $z \\in \\Rn$. Hence, using \\eqref{eq3-3}, we obtain\n%\n\\begin{equation} \n\\|I-\\lambda M_k\\|=\\|(I-\\lambda B_k)D_k\\|\\leq 1+\\lambda\\|B_k\\|. \\label{eq4-16}\n\\end{equation}\n%\nWe now define the constants $c_1 ={\\lambda\\bar{\\alpha}}/{(1+\\bar{\\alpha})}$, $c_2 = (1-\\eta_2)\\lambda$, \n%\n\\[  c_3 = \\frac{\\tau\\lambda(1-\\eta_2)}{8(1+\\lambda)(L\\tau\\lambda +1-\\tau)}, \\quad c_4 = \\frac{\\lambda\\sqrt{1-\\eta_2}}{2}, \\quad \\text{and} \\quad c_5=\\frac{(1-\\eta_2)\\lambda}{2\\eta_2}, \\]\n%\nwhere $\\bar{\\alpha}$ was introduced in \\cref{remark:lwb-step-size}. Furthermore, let us set \n%\n\\be \\label{eq:hat-ck} \\hat c := \\min\\left\\{\\min_{1\\leq i \\leq 5} c_i,1\\right\\} \\quad \\text{and} \\quad \\hat c_k := \\frac{\\hat c}{1+\\lambda\\|B_k\\|}. \\ee\n%\nWe consider an iteration $k$ with $\\Delta_k \\leq \\hat c_k \\chi(z_k)$. Let us define $s_k^d := -\\lambda d = \\lambda  \\Fnor{z_k}$. We have $\\|\\bar{s}_k\\| \\geq\\|s^d_k\\|-\\|(I-\\lambda B_k)D_k\\bar q_k\\|$ and by the algorithmic construction, it holds that $\\|\\bar q_k\\|\\leq\\Delta_k$. Then, applying $\\|s_k^d\\| = \\lambda\\chi(z_k)$, \\eqref{eq4-16}, and \\eqref{eq:hat-ck}, it follows\n%\n\\begin{align*}\n\\|\\bar{s}_k\\| & \\geq\\|s_k^d\\| - (1+\\lambda\\|B_k\\|)\\Delta_k \\geq \\left[ \\frac{\\lambda}{\\hat c_k}- (1+\\lambda\\|B_k\\|) \\right] \\Delta_k \\geq \\left[ \\frac{\\lambda}{c_1} - 1 \\right] (1+\\lambda\\|B_k\\|) \\Delta_k \\geq \\frac{1}{\\bar\\alpha} \\Delta_k, \\\\ \\|\\bar s_k\\| & \\geq \\lambda\\chi(z_k) - (1+\\lambda\\|B_k\\|)\\hat c_k \\chi(z_k) \\geq [\\lambda - c_2] \\chi(z_k)  = \\lambda\\eta_2\\chi(z_k),\n\\end{align*}\n%\nand $\\frac{\\Delta_k}{\\|\\bar{s}_k\\|}\\leq \\min\\{\\bar{\\alpha},  \\frac{\\Delta_k}{\\lambda\\eta_2\\chi(z_k)} \\} \\leq1$. This yields $s_k=\\frac{\\Delta_k}{\\|\\bar{s}_k\\|}\\lambda (-\\Fnor{z_k}+(\\frac{1}{\\lambda} I-M_k)\\bar q_k)$. Setting  $\\alpha={\\Delta_k}/\\|\\bar{s}_k\\| \\leq\\bar{\\alpha}$, $e=(\\frac{1}{\\lambda} I-M_k)\\bar q_k$, and using $\\lambda(d + e)=\\bar s_k$ (cf. \\eqref{eq:def-sk-bar-sk}), \\cref{lemma2-5} and \\cref{remark:lwb-step-size} imply\n%\n\\begin{align} \\label{eq4-17} \\nonumber\n\\mer(z_k+s_k)-\\mer(z_k) & \\leq -\\frac{\\tau\\lambda\\alpha}{2}\\cdot \\chi(z_k)^2 - \\frac{\\nu}{\\lambda\\alpha} \\|\\prox{z_k+s_k} - x_k\\|^2  \\\\ & \\hspace{9ex} + \\left[L \\tau\\lambda + 1-\\tau\\right] \\lambda\\alpha \\|e\\| \\|\\bar s_k\\| + \\frac{\\tau\\lambda\\alpha}{2} \\|e\\|^2.\n\\end{align}\n%\nNext, we provide additional estimates for $\\|e\\|$ and $\\|\\bar s_k\\|$. Invoking \\eqref{eq4-16}, it holds that $\\|e\\| = \\|(\\frac{1}{\\lambda} I-B_k)D_k\\bar q_k\\|\\leq \\frac{1}{\\lambda}(1+\\lambda\\|B_k\\|) \\Delta_k\\leq \\frac{1}{\\lambda} \\hat c\\chi(z_k)$ and\n%\n\\be \\label{eq:upp-sk}\n\\|\\bar s_k\\| \\leq \\|s_k^d\\| + \\lambda\\|e\\| \\leq ( \\lambda + \\hat c) \\chi(z_k). \n\\ee\n%\nThus, we obtain\n%\n\\begin{align*} \n \\left[ L\\tau\\lambda  + 1-\\tau\\right]\\lambda\\|e\\|\\|\\bar s_k\\| \\leq \\left[ L\\tau\\lambda  + 1-\\tau\\right]c_3(\\lambda+1)\\chi(z_k)^2  = \\frac{\\tau\\lambda}{2} \\frac{1-\\eta_2}{4} \\cdot \\chi(z_k)^2 \n\\end{align*} \n%\nand $\\|e\\|^2 \\leq \\frac{c_4^2}{\\lambda^2} \\chi(z_k)^2 = \\frac14{(1-\\eta_2)} \\chi(z_k)^2$. Using these estimates in \\eqref{eq4-17}, we have\n%\n\\begin{align*} \\mer(z_k+s_k)-\\mer(z_k) \\leq -\\frac{\\tau\\lambda\\alpha}{2}\\frac{1+\\eta_2}{2} \\chi(z_k)^2 - \\frac{\\nu}{\\lambda\\alpha} \\|\\prox{z_k+s_k} - x_k\\|^2. \\end{align*}\n%\nDue to $\\Delta_k \\leq c_2 \\chi(z_k) \\leq \\lambda\\chi(z_k)$ and $\\nu_k \\leq \\nu$, we can further infer \n%\n\\[ -\\frac{\\nu}{\\lambda\\alpha} \\leq -\\frac{\\nu_k}{\\lambda}\\cdot\\frac{\\eta_2 \\lambda\\chi(z_k)}{\\Delta_k} = - \\frac{\\eta_2 \\cdot \\nu_k\\chi(z_k)}{\\min\\{\\Delta_k,\\lambda\\chi(z_k)\\}} = -\\eta_2\\mu_k. \\] \n%have $\\alpha \\leq \\frac{\\Delta_k}{\\eta_2\\lambda\\chi(z_k)} = \\frac{\\min\\{\\Delta_k,\\lambda\\chi(z_k)}{\\eta_2\\lambda\\chi(z_k)} $. Hence, applying , it follows $-\\nu(\\lambda\\alpha)^{-1} \\leq \\eta_2\\mu_k$.}\n%\n%\\begin{align}\\label{eq4-18} \\nonumber \\mer(z_k+s_k)-\\mer(z_k)& \\\\ \\nonumber & \\hspace{-20ex} \\leq-\\frac{\\tau\\lambda\\alpha}{2} \\cdot \\chi(z_k)^2+\\frac{\\tau\\lambda\\alpha}{2}\\frac{1-\\eta_2}{2} \\cdot\\chi(z_k)^2 - \\frac{\\nu}{\\lambda\\alpha} \\|\\prox{z_k+s_k} - x_k\\|^2 \\\\ & \\hspace{-20ex} \\leq -\\frac{\\tau\\alpha\\lambda}{2}\\frac{\\eta_2+1}{2}\\cdot \\chi(z_k)^2 - \\frac{\\eta_2\\nu}{\\lambda\\min\\{1,{\\Delta_k}/{\\lambda/\\chi(z_k)}\\}} \\|\\prox{z_k+s_k} - x_k\\|^2. \\end{align}\n%\nBy \\eqref{eq:upp-sk}, we also have $\\|\\bar s_k\\|\\leq (\\lambda + c_5) \\chi(z_k) = (2\\eta_2)^{-1}(1+\\eta_2)\\lambda \\chi(z_k)$ which establishes $-\\alpha = - \\Delta_k/\\|\\bar s_k\\| \\leq - 2\\eta_2 \\Delta_k((1+\\eta_2)\\lambda\\chi(z_k))^{-1}$. Combining the last steps, it follows\n%\n\\begin{align*} \\mer(z_k+s_k)-\\mer(z_k) & \\leq-\\frac{\\eta_2\\tau}{2} \\Delta_k\\chi(z_k) - {\\eta_2{\\mu}_k} \\|\\prox{z_k+s_k}-x_k\\|^2 \\leq - \\eta_2 \\mathrm{pred}(z_k,s_k,\\Delta_k,\\nu_k). \\end{align*}\n%\nConsequently, we have $\\rho_k\\geq\\eta_2$ which concludes the proof. \\end{proof}","sha256_utf8":"89bde93948e8d1da7e538b4d7e47adb3f19d62136ba3f82dab3699565783a38c"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"控制牛顿矩阵偏差","source_units":[{"start":71207,"end":71471,"start_line":827,"end_line":831,"raw_tex":"\\begin{proof} First, by \\cref{prop2-5}, we have $\\|D\\|\\leq 1$ for all $D \\in \\partial \\prox{z}$ and $z \\in \\Rn$. Hence, using \\eqref{eq3-3}, we obtain\n%\n\\begin{equation} \n\\|I-\\lambda M_k\\|=\\|(I-\\lambda B_k)D_k\\|\\leq 1+\\lambda\\|B_k\\|. \\label{eq4-16}\n\\end{equation}\n","sha256_utf8":"3f1ce0fd5a1dfd1c716e6251bfe51cb9ce1213b68c08a24b9a2859dbbbef4162"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"构造接受性常数","source_units":[{"start":71473,"end":72015,"start_line":833,"end_line":839,"raw_tex":"We now define the constants $c_1 ={\\lambda\\bar{\\alpha}}/{(1+\\bar{\\alpha})}$, $c_2 = (1-\\eta_2)\\lambda$, \n%\n\\[  c_3 = \\frac{\\tau\\lambda(1-\\eta_2)}{8(1+\\lambda)(L\\tau\\lambda +1-\\tau)}, \\quad c_4 = \\frac{\\lambda\\sqrt{1-\\eta_2}}{2}, \\quad \\text{and} \\quad c_5=\\frac{(1-\\eta_2)\\lambda}{2\\eta_2}, \\]\n%\nwhere $\\bar{\\alpha}$ was introduced in \\cref{remark:lwb-step-size}. Furthermore, let us set \n%\n\\be \\label{eq:hat-ck} \\hat c := \\min\\left\\{\\min_{1\\leq i \\leq 5} c_i,1\\right\\} \\quad \\text{and} \\quad \\hat c_k := \\frac{\\hat c}{1+\\lambda\\|B_k\\|}. \\ee\n","sha256_utf8":"9c50525ab527249e98bf56c03f4ea61315097f7cbd2e32e7e5597d52f1352aba"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"界定提升步与缩放因子","source_units":[{"start":72017,"end":73240,"start_line":841,"end_line":847,"raw_tex":"We consider an iteration $k$ with $\\Delta_k \\leq \\hat c_k \\chi(z_k)$. Let us define $s_k^d := -\\lambda d = \\lambda  \\Fnor{z_k}$. We have $\\|\\bar{s}_k\\| \\geq\\|s^d_k\\|-\\|(I-\\lambda B_k)D_k\\bar q_k\\|$ and by the algorithmic construction, it holds that $\\|\\bar q_k\\|\\leq\\Delta_k$. Then, applying $\\|s_k^d\\| = \\lambda\\chi(z_k)$, \\eqref{eq4-16}, and \\eqref{eq:hat-ck}, it follows\n%\n\\begin{align*}\n\\|\\bar{s}_k\\| & \\geq\\|s_k^d\\| - (1+\\lambda\\|B_k\\|)\\Delta_k \\geq \\left[ \\frac{\\lambda}{\\hat c_k}- (1+\\lambda\\|B_k\\|) \\right] \\Delta_k \\geq \\left[ \\frac{\\lambda}{c_1} - 1 \\right] (1+\\lambda\\|B_k\\|) \\Delta_k \\geq \\frac{1}{\\bar\\alpha} \\Delta_k, \\\\ \\|\\bar s_k\\| & \\geq \\lambda\\chi(z_k) - (1+\\lambda\\|B_k\\|)\\hat c_k \\chi(z_k) \\geq [\\lambda - c_2] \\chi(z_k)  = \\lambda\\eta_2\\chi(z_k),\n\\end{align*}\n%\nand $\\frac{\\Delta_k}{\\|\\bar{s}_k\\|}\\leq \\min\\{\\bar{\\alpha},  \\frac{\\Delta_k}{\\lambda\\eta_2\\chi(z_k)} \\} \\leq1$. This yields $s_k=\\frac{\\Delta_k}{\\|\\bar{s}_k\\|}\\lambda (-\\Fnor{z_k}+(\\frac{1}{\\lambda} I-M_k)\\bar q_k)$. Setting  $\\alpha={\\Delta_k}/\\|\\bar{s}_k\\| \\leq\\bar{\\alpha}$, $e=(\\frac{1}{\\lambda} I-M_k)\\bar q_k$, and using $\\lambda(d + e)=\\bar s_k$ (cf. \\eqref{eq:def-sk-bar-sk}), \\cref{lemma2-5} and \\cref{remark:lwb-step-size} imply\n","sha256_utf8":"7a470fb7b119aeb6c869ceb874c97dc1146ee6c0cbf1697639d4171813a17bee"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"应用评价函数下降估计","source_units":[{"start":73242,"end":73554,"start_line":849,"end_line":851,"raw_tex":"\\begin{align} \\label{eq4-17} \\nonumber\n\\mer(z_k+s_k)-\\mer(z_k) & \\leq -\\frac{\\tau\\lambda\\alpha}{2}\\cdot \\chi(z_k)^2 - \\frac{\\nu}{\\lambda\\alpha} \\|\\prox{z_k+s_k} - x_k\\|^2  \\\\ & \\hspace{9ex} + \\left[L \\tau\\lambda + 1-\\tau\\right] \\lambda\\alpha \\|e\\| \\|\\bar s_k\\| + \\frac{\\tau\\lambda\\alpha}{2} \\|e\\|^2.\n\\end{align}\n","sha256_utf8":"d2b5d5ceb90976c696d191a150f1db389349096914035cda59684dddb538bd98"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"估计扰动与提升步上界","source_units":[{"start":73556,"end":73915,"start_line":853,"end_line":857,"raw_tex":"Next, we provide additional estimates for $\\|e\\|$ and $\\|\\bar s_k\\|$. Invoking \\eqref{eq4-16}, it holds that $\\|e\\| = \\|(\\frac{1}{\\lambda} I-B_k)D_k\\bar q_k\\|\\leq \\frac{1}{\\lambda}(1+\\lambda\\|B_k\\|) \\Delta_k\\leq \\frac{1}{\\lambda} \\hat c\\chi(z_k)$ and\n%\n\\be \\label{eq:upp-sk}\n\\|\\bar s_k\\| \\leq \\|s_k^d\\| + \\lambda\\|e\\| \\leq ( \\lambda + \\hat c) \\chi(z_k). \n\\ee\n","sha256_utf8":"d8e8b9c81679b3163eafb97a0f1f59142fa00fb58aafe6219eaa0388ee958197"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"吸收扰动误差","source_units":[{"start":73917,"end":74472,"start_line":859,"end_line":867,"raw_tex":"Thus, we obtain\n%\n\\begin{align*} \n \\left[ L\\tau\\lambda  + 1-\\tau\\right]\\lambda\\|e\\|\\|\\bar s_k\\| \\leq \\left[ L\\tau\\lambda  + 1-\\tau\\right]c_3(\\lambda+1)\\chi(z_k)^2  = \\frac{\\tau\\lambda}{2} \\frac{1-\\eta_2}{4} \\cdot \\chi(z_k)^2 \n\\end{align*} \n%\nand $\\|e\\|^2 \\leq \\frac{c_4^2}{\\lambda^2} \\chi(z_k)^2 = \\frac14{(1-\\eta_2)} \\chi(z_k)^2$. Using these estimates in \\eqref{eq4-17}, we have\n%\n\\begin{align*} \\mer(z_k+s_k)-\\mer(z_k) \\leq -\\frac{\\tau\\lambda\\alpha}{2}\\frac{1+\\eta_2}{2} \\chi(z_k)^2 - \\frac{\\nu}{\\lambda\\alpha} \\|\\prox{z_k+s_k} - x_k\\|^2. \\end{align*}\n","sha256_utf8":"6ed26fb12bef078f07040ae0d8ef3494cf757c7e2ea92ebeca20ddf47e669ca9"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"匹配近端位移预测项","source_units":[{"start":74474,"end":74776,"start_line":869,"end_line":871,"raw_tex":"Due to $\\Delta_k \\leq c_2 \\chi(z_k) \\leq \\lambda\\chi(z_k)$ and $\\nu_k \\leq \\nu$, we can further infer \n%\n\\[ -\\frac{\\nu}{\\lambda\\alpha} \\leq -\\frac{\\nu_k}{\\lambda}\\cdot\\frac{\\eta_2 \\lambda\\chi(z_k)}{\\Delta_k} = - \\frac{\\eta_2 \\cdot \\nu_k\\chi(z_k)}{\\min\\{\\Delta_k,\\lambda\\chi(z_k)\\}} = -\\eta_2\\mu_k. \\] \n","sha256_utf8":"9577cf33745acb487406f1e77837a49bd9e51003266f38e164d3a8a213656f6a"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"建立充分接受比率","source_units":[{"start":75442,"end":75992,"start_line":876,"end_line":880,"raw_tex":"By \\eqref{eq:upp-sk}, we also have $\\|\\bar s_k\\|\\leq (\\lambda + c_5) \\chi(z_k) = (2\\eta_2)^{-1}(1+\\eta_2)\\lambda \\chi(z_k)$ which establishes $-\\alpha = - \\Delta_k/\\|\\bar s_k\\| \\leq - 2\\eta_2 \\Delta_k((1+\\eta_2)\\lambda\\chi(z_k))^{-1}$. Combining the last steps, it follows\n%\n\\begin{align*} \\mer(z_k+s_k)-\\mer(z_k) & \\leq-\\frac{\\eta_2\\tau}{2} \\Delta_k\\chi(z_k) - {\\eta_2{\\mu}_k} \\|\\prox{z_k+s_k}-x_k\\|^2 \\leq - \\eta_2 \\mathrm{pred}(z_k,s_k,\\Delta_k,\\nu_k). \\end{align*}\n%\nConsequently, we have $\\rho_k\\geq\\eta_2$ which concludes the proof. \\end{proof}","sha256_utf8":"dfa03bf601449a907577d46b935cf188ba0cc7c196ee984add6a1b817a44ba30"}]},{"id":"S045","semantic_type":"proof","kind":"proof","title":"成功步半径下界推导","source_units":[{"start":75994,"end":76728,"start_line":882,"end_line":888,"raw_tex":"\\begin{remark}\n\\label{remark:bound_chi_delta} \\cref{lemma2-6} implies that the condition $\\Delta_k \\leq \\hat c_k \\chi(z_k)$ can not hold for unsuccessful iterations $k \\notin \\mathcal S$. Let us further consider an iterate $z_k$ with $k\\in\\mathcal{S}$. If $k-1\\in\\mathcal{S}$, then we obtain $\\Delta_k\\geq \\Delta_{\\min}$. Otherwise, if $k-1\\notin\\mathcal{S}$, we have $B_k = B_{k-1}$ and we can infer $\\Delta_{k-1}>\\hat c_{k-1}\\chi(z_{k-1})=\\hat c_k\\chi(z_k)$. Thus, by the algorithmic construction, it follows $\\Delta_{k}\\geq\\gamma_0 \\Delta_{k-1}\\geq \\gamma_0\\hat c_k\\chi(z_k)$ and for all $k \\in \\mathcal S$, we have \n%\n\\[ \\Delta_k\\geq\\min\\{\\gamma_0\\hat c_k\\chi(z_k),\\Delta_{\\min}\\}.  \\]\n%\n%for all $k \\in \\mathcal S$.\n\\end{remark}\n","sha256_utf8":"180532c955e18b15e424b29cd19c597c127d93bb3c88d859d4c1b2ea74276d99"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"按前一步成败估计半径","source_units":[{"start":75994,"end":76728,"start_line":882,"end_line":888,"raw_tex":"\\begin{remark}\n\\label{remark:bound_chi_delta} \\cref{lemma2-6} implies that the condition $\\Delta_k \\leq \\hat c_k \\chi(z_k)$ can not hold for unsuccessful iterations $k \\notin \\mathcal S$. Let us further consider an iterate $z_k$ with $k\\in\\mathcal{S}$. If $k-1\\in\\mathcal{S}$, then we obtain $\\Delta_k\\geq \\Delta_{\\min}$. Otherwise, if $k-1\\notin\\mathcal{S}$, we have $B_k = B_{k-1}$ and we can infer $\\Delta_{k-1}>\\hat c_{k-1}\\chi(z_{k-1})=\\hat c_k\\chi(z_k)$. Thus, by the algorithmic construction, it follows $\\Delta_{k}\\geq\\gamma_0 \\Delta_{k-1}\\geq \\gamma_0\\hat c_k\\chi(z_k)$ and for all $k \\in \\mathcal S$, we have \n%\n\\[ \\Delta_k\\geq\\min\\{\\gamma_0\\hat c_k\\chi(z_k),\\Delta_{\\min}\\}.  \\]\n%\n%for all $k \\in \\mathcal S$.\n\\end{remark}\n","sha256_utf8":"180532c955e18b15e424b29cd19c597c127d93bb3c88d859d4c1b2ea74276d99"}]},{"id":"S047","semantic_type":"lemma","kind":"lemma","title":"无限多个成功步","source_units":[{"start":76855,"end":77054,"start_line":892,"end_line":895,"raw_tex":"\\begin{lemma}\n\\label{lemma2-7}\nUnder the assumptions \\ref{A1} and \\ref{B1}, \\cref{algo2} either terminates after finitely many iterations or it generates infinitely many successful steps.\n\\end{lemma}","sha256_utf8":"19b2337cf5a535ed01c4fe67f49b5560dc2a30d35c4042d1cbf3d808000ecf81"}]},{"id":"S048","semantic_type":"proof","kind":"proof","title":"无限成功步证明","source_units":[{"start":77057,"end":77927,"start_line":897,"end_line":898,"raw_tex":"\\begin{proof} Conversely, assume that \\cref{algo2} generates an infinite sequence $\\{z_k\\}$ with only finitely many successful steps. Let $k'\\in\\mathbb{N}$ denote the last successful iteration, i.e., it holds that $z_k = z_{k'}$ for all $k\\geq k'$. The update rule for the trust region radius then implies $\\Delta_k\\rightarrow 0$. Furthermore, since the matrices $B_k$ are no longer updated for all $k \\geq k^\\prime$, we obtain $\\hat c_k = \\hat c_{k^\\prime}$ for all $k \\geq k^\\prime$ where the parameter $\\hat c_k$ is defined in \\eqref{eq:bd-del}. Due to $\\Delta_k \\to 0$ there then exists $\\ell > k^\\prime$ with $\\Delta_{\\ell} \\leq \\hat c_{k^\\prime} \\chi(z_{k^\\prime}) =  \\hat c_{\\ell}\\chi(z_{\\ell})$ which, by \\cref{lemma2-6}, yields ${\\ell} \\in \\mathcal S$. However, this is a contradiction to our assumption. %and finishes the proof of \\cref{lemma2-7}. \n\\end{proof}","sha256_utf8":"7ad89e85227076c97e8cb642b9645079fa642a06ba2551ef33eb19dfa357bbf3"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"半径趋零排除永久拒绝","source_units":[{"start":77057,"end":77927,"start_line":897,"end_line":898,"raw_tex":"\\begin{proof} Conversely, assume that \\cref{algo2} generates an infinite sequence $\\{z_k\\}$ with only finitely many successful steps. Let $k'\\in\\mathbb{N}$ denote the last successful iteration, i.e., it holds that $z_k = z_{k'}$ for all $k\\geq k'$. The update rule for the trust region radius then implies $\\Delta_k\\rightarrow 0$. Furthermore, since the matrices $B_k$ are no longer updated for all $k \\geq k^\\prime$, we obtain $\\hat c_k = \\hat c_{k^\\prime}$ for all $k \\geq k^\\prime$ where the parameter $\\hat c_k$ is defined in \\eqref{eq:bd-del}. Due to $\\Delta_k \\to 0$ there then exists $\\ell > k^\\prime$ with $\\Delta_{\\ell} \\leq \\hat c_{k^\\prime} \\chi(z_{k^\\prime}) =  \\hat c_{\\ell}\\chi(z_{\\ell})$ which, by \\cref{lemma2-6}, yields ${\\ell} \\in \\mathcal S$. However, this is a contradiction to our assumption. %and finishes the proof of \\cref{lemma2-7}. \n\\end{proof}","sha256_utf8":"7ad89e85227076c97e8cb642b9645079fa642a06ba2551ef33eb19dfa357bbf3"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"评价下降与两项可和性","source_units":[{"start":78581,"end":79241,"start_line":910,"end_line":916,"raw_tex":"\\begin{proof} \\cref{lemma2-7} implies $\\vert{\\mathcal S}\\vert=\\infty$ and it holds that\n%\n\\[ {\\sum}_{k\\in\\mathcal{S}}\\mer(z_{k})-\\mer(z_{k+1})\\geq{\\sum}_{k\\in\\mathcal{S}}\\eta_1 \\mathrm{pred}(z_k,s_k,\\Delta_k,\\nu_k). \\]\n%\nFurthermore, by assumption \\ref{A2}, the merit function $\\mer$ is bounded from below on $\\Rn$. Hence, since the sequence $\\{\\mer(z_k)\\}$ is non-increasing, there exists $\\zeta \\in \\R$ with $\\lim_{k \\to \\infty} \\mer(z_k) = \\zeta$ and it follows\n%\n\\be \\label{eq:crit-sum} \\sum_{k \\in \\mathcal S} {\\mu}_k \\|x_{k+1}-x_k\\|^2 < \\infty \\quad \\text{and} \\quad \\sum_{k\\in\\mathcal{S}}\\chi(z_k)\\min\\{\\lambda,{\\Delta_k},\\lambda\\chi(z_k)\\}<\\infty. \\ee\n","sha256_utf8":"348204cfb14f963d20803c6a5b7c39f1179e2fa60a1933da9468cdb9265a0535"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"反设残差具有正下界","source_units":[{"start":79243,"end":80035,"start_line":918,"end_line":920,"raw_tex":"We now proceed as in \\cite{Pow84} and first show the weaker condition $\\liminf_{k \\to \\infty} \\chi(z_k) = 0$. {Assume that there is $\\epsilon \\in (0,\\min\\{1,\\lambda^{-1}\\Delta_{\\min}\\})$ with $\\chi(z_k) \\geq \\epsilon$ for all $k$ sufficiently large}. Then, utilizing \\cref{remark:bound_chi_delta} and \\eqref{eq:crit-sum}, we obtain\n%\n\\be \\label{eq:p-sum-lw} \\infty > \\sum_{k \\in \\mathcal S} {\\epsilon} \\min\\{\\lambda,\\Delta_k,\\lambda\\epsilon\\} \\geq {\\epsilon} \\sum_{k \\in \\mathcal S} \\min\\left\\{\\lambda\\epsilon,\\gamma_0\\hat c_k\\chi(z_k)\\right\\} \\geq {\\epsilon^2} \\sum_{k \\in \\mathcal S} \\frac{\\min\\{\\lambda,\\hat c \\gamma_0\\}}{1+\\lambda\\|B_k\\|}. \\ee % \\epsilon \\sum_{k \\in \\mathcal S} \\min\\left\\{\\epsilon,\\min\\left\\{\\frac{\\hat c \\gamma_0\\}{1+\\lambda\\|B_k\\|}, \\Delta_{\\min}\\right\\}\\right\\}  \\ee\n","sha256_utf8":"61070426d8c9e3d9ba3cfd69c021836e37db424b138fec5d409d74e8001788d0"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"估计失败区间贡献","source_units":[{"start":80037,"end":80922,"start_line":922,"end_line":928,"raw_tex":"Due to $\\vert\\mathcal S\\vert = \\infty$, we have $(1+\\lambda\\|B_k\\|)^{-1} \\to 0$ as $\\mathcal S \\ni k \\to \\infty$ and hence, it follows $\\sum_{k \\in \\mathcal S} (1+\\lambda\\|B_k\\|)^{-1} \\leq \\infty$. Let us recall the notation $\\mathcal S = \\{j_i : i \\geq 0\\}$ and let us consider an arbitrary unsuccessful iteration $ j_i+1 \\leq k \\leq j_{i+1}-1$. Using the trust region update mechanism and \\cref{lemma2-6}, this yields\n%\n\\[\\frac{\\hat c\\epsilon}{1+\\lambda\\|B_k\\|} \\leq \\hat c_k \\chi(z_k) \\leq \\Delta_k \\leq \\gamma_1^{k-j_i} \\Delta_{j_i}. \\]\n%\nSumming this estimate, we obtain\n%\n\\[ \\sum_{k=j_i+1}^{j_{i+1}-1} \\frac{1}{1+\\lambda\\|B_k\\|} \\leq \\frac{\\Delta_{j_i}}{\\hat c \\epsilon} \\sum_{k=j_i+1}^{j_{i+1}-1} \\gamma_1^{k-j_i} \\leq \\frac{\\Delta_{j_i}}{\\hat c \\epsilon} \\left[ \\sum_{k=0}^{\\infty} \\gamma_1^{k} - 1 \\right] = \\frac{\\gamma_1}{(1-\\gamma_1)\\hat c \\epsilon} \\cdot \\Delta_{j_i}. \\]\n","sha256_utf8":"e1b3a4ef1176b39ca2f038fcc961637d7f7857b06b1e61ec8fc5b9acb6bb66e7"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"利用非可和假设得矛盾","source_units":[{"start":80924,"end":81772,"start_line":930,"end_line":934,"raw_tex":"Summing this expression once more for all $i$, we can infer $\\sum_{k \\notin \\mathcal S}(1+\\lambda\\|B_k\\|^{-1}) \\leq \\frac{\\gamma_1}{(1-\\gamma_1)\\hat c\\epsilon} \\sum_{k \\in \\mathcal S} \\Delta_k$. As before, the summability condition in \\eqref{eq:p-sum-lw} implies $\\Delta_k \\to 0$ as $\\mathcal S \\ni k \\to \\infty$ and hence, combining the last steps, it follows \n%\n\\[ \\sum_{k=0}^\\infty \\frac{1}{1+\\lambda\\|B_k\\|} < \\infty \\quad \\implies \\quad \\sum_{k=0}^\\infty \\frac{1}{1+\\|B_k\\|}<\\infty. \\]\n%\nHowever, this contradicts \\ref{B2} and thus, we can establish $\\liminf_{k \\to \\infty} \\chi(z_k) = 0$. %\\revise{In order to show convergence of the full sequence $\\{\\chi(z_k)\\}$, %we can apply the Lipschitz continuity of $\\nabla f$ {and $\\chi$} and follow the strategies used in the convergence analysis of classical trust region-type methods, see, e.g., \n","sha256_utf8":"fe75b3fe2a54443c337562b94acb0deabaded71f6c3c7e58c16fe870d00e6ef4"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"构造残差往返区间","source_units":[{"start":81875,"end":82587,"start_line":937,"end_line":943,"raw_tex":"Let us now suppose that $\\{\\chi(z_k)\\}$ does not converge. Then there exist $\\delta > 0$ and infinite, increasing sequences $\\{t_i\\}_i,\\{\\ell_i\\}_i \\subset \\mathcal S$ such that $t_{i+1} \\geq \\ell_i > t_i$ for all $i \\in \\N$ and\n%\n\\[ \\chi(z_{t_i}) \\geq 2\\delta, \\quad \\chi(z_{\\ell_i}) < \\delta, \\quad \\text{and} \\quad \\chi(z_k) \\geq \\delta \\quad k = t_i+1,...,\\ell_i-1. \\]\n%\nLet us define $\\mathcal K := \\{k \\in \\mathcal S: t_i \\leq k < \\ell_i\\}$. Due to $\\mathcal K \\subseteq \\mathcal S$, the second condition in \\eqref{eq:crit-sum} implies \n%\n\\[ \\infty > \\sum_{k \\in \\mathcal K} \\chi(z_k) \\min\\{\\lambda,\\Delta_k,\\lambda\\chi(z_k)\\} \\geq \\sum_{k \\in \\mathcal K} \\delta \\min\\{\\lambda,\\Delta_k,\\lambda\\delta\\}. \\]\n","sha256_utf8":"0406b80a5cfe29872535dc2d9fcbd536f70db87c1bc07e96864ffd2fae4fec2d"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"半径下降界与残差连续性","source_units":[{"start":82589,"end":83260,"start_line":945,"end_line":947,"raw_tex":"Consequently, we have $\\Delta_k \\to 0$ as $\\mathcal K \\ni k \\to \\infty$ and utilizing $\\rho_k \\geq \\eta_1$ for all $k \\in \\mathcal K$, it follows $\\Delta_k \\leq \\frac{2}{\\tau\\eta_1\\delta}[\\mer(z_k)-\\mer(z_{k+1})]$ for all $k \\in \\mathcal K$ sufficiently large. Next, the Lipschitz continuity of $\\nabla f$ and $\\oprox$ yield\n%\n\\begin{align} \\nonumber \\vert\\chi(w)-\\chi(z)\\vert & \\leq \\|\\Fnor{w}-\\Fnor{z}\\| \\\\ \\nonumber & \\hspace{-12ex} \\leq \\|\\nabla f(\\prox{w})-\\nabla f(\\prox{z})\\| + {\\lambda^{-1}}\\|w-\\prox{w} - (z-\\prox{z})\\| \\\\ & \\hspace{-12ex}  \\leq L \\|\\prox{w}-\\prox{z}\\| + 2\\lambda^{-1}\\|w-z\\| \\leq (L+2\\lambda^{-1}) \\|w-z\\| \\label{eq:fnor-lipschitz} \\end{align}\n","sha256_utf8":"70a1cb1be873b7fc208d20344f3749b581d302505fc65e9e69b009e79336af9d"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"望远镜求和排除残差振荡","source_units":[{"start":83262,"end":83866,"start_line":949,"end_line":954,"raw_tex":"for all $w, z \\in \\Rn$. Thus, setting $L_F := L+2\\lambda^{-1}$, combining the previous estimates, and using \\eqref{eq:def-sk-bar-sk}, we obtain\n%\n\\begin{align*} \\delta & < \\vert\\chi(z_{t_i}) - \\chi(z_{\\ell_i})\\vert \\leq L_F {\\|z_{\\ell_i}-z_{t_i}\\|} \\\\ & \\leq L_F \\sum_{k = t_i, \\, k \\in \\mathcal K}^{\\ell_i-1} \\|s_k\\| \\leq L_F \\sum_{k = t_i, \\, k \\in \\mathcal K}^{\\ell_i-1} \\Delta_k \\leq {\\frac{2L_F}{\\tau\\eta_1\\delta}}[\\mer(z_{t_i}) - \\mer(z_{\\ell_i})]. \\end{align*}\n%\nDue to $\\mer(z_k) \\to \\zeta$, the right hand side of the last inequality has to converge to zero which is a contradiction.\n\\end{proof}","sha256_utf8":"b46fb2caf6246b0416134c3bfc1c93b9d48b830a7a064a0f543e4063a96110d4"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S009","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S004","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S019","kind":"supports"},{"source_id":"S005","target_id":"S023","kind":"supports"},{"source_id":"S007","target_id":"S009","kind":"supports"},{"source_id":"S007","target_id":"S010","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S009","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S008","target_id":"S013","kind":"supports"},{"source_id":"S011","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S015","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S018","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S016","target_id":"S027","kind":"supports"},{"source_id":"S006","target_id":"S027","kind":"supports"},{"source_id":"S007","target_id":"S027","kind":"supports"},{"source_id":"S007","target_id":"S028","kind":"supports"},{"source_id":"S016","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S006","target_id":"S029","kind":"supports"},{"source_id":"S016","target_id":"S029","kind":"supports"},{"source_id":"S006","target_id":"S030","kind":"supports"},{"source_id":"S027","target_id":"S031","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S009","target_id":"S031","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S029","target_id":"S026","kind":"supports"},{"source_id":"S030","target_id":"S026","kind":"supports"},{"source_id":"S031","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S033","kind":"supports"},{"source_id":"S017","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S025","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S032","kind":"supports"},{"source_id":"S019","target_id":"S037","kind":"supports"},{"source_id":"S021","target_id":"S037","kind":"supports"},{"source_id":"S023","target_id":"S037","kind":"supports"},{"source_id":"S013","target_id":"S037","kind":"supports"},{"source_id":"S032","target_id":"S038","kind":"supports"},{"source_id":"S014","target_id":"S038","kind":"supports"},{"source_id":"S017","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S008","target_id":"S039","kind":"supports"},{"source_id":"S015","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S025","target_id":"S040","kind":"supports"},{"source_id":"S032","target_id":"S040","kind":"supports"},{"source_id":"S037","target_id":"S041","kind":"supports"},{"source_id":"S038","target_id":"S041","kind":"supports"},{"source_id":"S008","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S038","target_id":"S042","kind":"supports"},{"source_id":"S039","target_id":"S043","kind":"supports"},{"source_id":"S017","target_id":"S043","kind":"supports"},{"source_id":"S018","target_id":"S043","kind":"supports"},{"source_id":"S041","target_id":"S044","kind":"supports"},{"source_id":"S042","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S038","target_id":"S044","kind":"supports"},{"source_id":"S012","target_id":"S044","kind":"supports"},{"source_id":"S013","target_id":"S044","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S039","target_id":"S036","kind":"supports"},{"source_id":"S040","target_id":"S036","kind":"supports"},{"source_id":"S041","target_id":"S036","kind":"supports"},{"source_id":"S042","target_id":"S036","kind":"supports"},{"source_id":"S043","target_id":"S036","kind":"supports"},{"source_id":"S044","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S046","kind":"supports"},{"source_id":"S013","target_id":"S046","kind":"supports"},{"source_id":"S014","target_id":"S046","kind":"supports"},{"source_id":"S018","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S035","target_id":"S049","kind":"supports"},{"source_id":"S013","target_id":"S049","kind":"supports"},{"source_id":"S014","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S050","kind":"supports"},{"source_id":"S013","target_id":"S050","kind":"supports"},{"source_id":"S012","target_id":"S050","kind":"supports"},{"source_id":"S009","target_id":"S050","kind":"supports"},{"source_id":"S016","target_id":"S050","kind":"supports"},{"source_id":"S018","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S045","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S035","target_id":"S052","kind":"supports"},{"source_id":"S014","target_id":"S052","kind":"supports"},{"source_id":"S051","target_id":"S053","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S017","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S050","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S012","target_id":"S055","kind":"supports"},{"source_id":"S013","target_id":"S055","kind":"supports"},{"source_id":"S016","target_id":"S055","kind":"supports"},{"source_id":"S006","target_id":"S055","kind":"supports"},{"source_id":"S007","target_id":"S055","kind":"supports"},{"source_id":"S010","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S050","target_id":"S056","kind":"supports"},{"source_id":"S008","target_id":"S056","kind":"supports"},{"source_id":"S050","target_id":"S002","kind":"supports"},{"source_id":"S051","target_id":"S002","kind":"supports"},{"source_id":"S052","target_id":"S002","kind":"supports"},{"source_id":"S053","target_id":"S002","kind":"supports"},{"source_id":"S054","target_id":"S002","kind":"supports"},{"source_id":"S055","target_id":"S002","kind":"supports"},{"source_id":"S056","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S001","kind":"supports"}]}}
{"record_id":"2108.02129v2","paper":{"title":"Convergence results of a nested decentralized gradient method for non-strongly convex problems","authors":["Woocheol Choi","Doheon Kim","Seok-Bae Yun"],"arxiv_id":"2108.02129","version":"v2","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-022-02069-0","abstract_url":"https://arxiv.org/abs/2108.02129v2","pdf_url":"https://arxiv.org/pdf/2108.02129v2"},"context":{"raw_tex":"\\begin{align}\\label{1.1}\n\\min_{x\\in\\mathbb{R}^p} f(x):= \\frac{1}{n} \\sum_{i=1}^nf_i(x) ,\n\\end{align}\n\nHere $f_i:\\mathbb{R}^p\\rightarrow\\mathbb{R}$ is a local objective function (or cost function of agent $i$) and $n$ is the number of agents.\n\n\nwhere $F$ denotes the aggregate cost function in (\\ref{1.1}):\n\\[\nF({\\bf x}) =\\sum_{i=1}^nf_i(x_i).\n\\]\n\nHere $I_p$ is the $p\\times p$ identity matrix, ${\\bf x}$ denotes an $np$ dimensional column vector made by stacking up $x_i\\in\\mathbb{R}^p$ $(i=1,\\cdots,n)$:\n \n\n\n$${\\bf x}= (x_{1}^{\\top}, x_{2}^{\\top}, \\cdots, x_{n}^{\\top})^{\\top},$$\n\n  and the consensus matrix $W$ is the  $n\\times n$ matrix that contains the information on the connectivity of the underlying network:\n\\begin{equation}\\label{stack}\n%{\\bf x}=\\begin{pmatrix} x_1 \\\\  \\vdots \\\\ x_n\\end{pmatrix},\\quad \nW= \\begin{pmatrix} w_{11} & w_{12} & \\cdots & w_{1n} \\\\ w_{21} & w_{22} & \\cdots &w_{2n} \\\\ \\vdots &\\vdots &\\ddots &\\vdots \\\\ w_{n1} & w_{n2} &\\cdots &w_{nn}\n\\end{pmatrix}. \n\\end{equation}\n\nThroughout this paper, we assume the following properties for the consensus matrix:%\\newline\n%{\\bf Assumptions on the consensus matrix }$W$:\n%We assume the consensus matrix\n{\\color{black}\n\\begin{itemize}\n\\item $W$ is a doubly-stochastic $n\\times n$ matrix.\n\\item The directed graph associated to $W$ is strongly connected, and has a self-loop at every vertex.\n\\end{itemize}\n\n\nIn Lemma \\ref{newlemma}, we will show that the graph associated to $W^\\top W$ is connected, and so the eigenvalue $1$ of $W^\\top W$ is simple and all other eigenvalues have modulus strictly less than 1. We let $\\beta\\in[0,1)$ be the square root of the second largest eigenvalue of $W^\\top W$, which is equal to the spectral norm of $W-\\frac{1}{n}1_n1_n^\\top$, where $1_n$ is a column vector in $\\mathbb{R}^n$ whose entries are all $1$.} We let $f^*$ be the optimal objective value of \\eqref{1.1} and denote by ${X}^*$ the set of optimal solutions to \\eqref{1.1} which is assumed to be nonempty, i.e., \n\\begin{equation*}\n{X}^* = \\{x \\in \\mathbb{R}^p ~:~ f(x) =f^*\\}.\n\\end{equation*}\n\n\\noindent {\\bf Notation.} $\\|\\cdot\\|$ denotes the standard $\\ell^2$-norm in the Euclidean space, or the operator norm induced by the $\\ell^2$-norm, depending on the context.\n\n\n\\begin{defn}For given $\\alpha \\geq 0$, a function $\\mathbf{f}: \\mathbb{R}^p \\rightarrow \\mathbb{R}$ is called $\\alpha$-strongly convex (convex if $\\alpha =0$) if\n\\begin{equation*}\n\\mathbf{f}(y) \\geq \\mathbf{f}(x) + \\langle y-x, \\nabla \\mathbf{f}(x) \\rangle+ \\frac{\\alpha}{2} \\|y-x\\|^2\\quad \\forall~x,y \\in \\mathbb{R}^p.\n\\end{equation*}\n\\end{defn}\n\n\\begin{defn}We  say that a function $\\mathbf{f}: \\mathbb{R}^p \\rightarrow \\mathbb{R}$ is called $L$-smooth for given   $L >0$ if $f$ has Lipschitz continuous gradient with constant $L$, i.e., \n\\begin{equation*}\n\\|\\nabla \\mathbf{f}(x) - \\nabla \\mathbf{f}(y)\\| \\leq L \\|x-y\\|\\quad \\forall~x,y \\in\\mathbb{R}^p.\n\\end{equation*}\n\\end{defn}\n\n\\begin{equation}\\label{NEAR DGD}\n{\\bf {\\bf x}}_{k+1} = \\big(W^{t(k)}\\otimes I_p\\big)\\big({\\bf x}_k - \\mu \\nabla F({\\bf x}_k)\\big)\\quad \\forall~ k \\geq0,%\\in \\mathbb{N},\n\\end{equation}\n\nwhere ${\\bf x}_k= (x_{1,k}^{\\top}, x_{2,k}^{\\top}, \\cdots, x_{n,k}^{\\top})^{\\top}$ and $\\mu >0$. Here  $\\nabla F (\\mathbf{x}_k)\\in \\mathbb{R}^{np}$ denotes\n\\begin{equation*}\n\\nabla F({\\bf x}_k)=\\left(\n\\nabla f_1(x_{1,k})^{\\top},\\cdots\n, \\nabla f_n(x_{n,k})^{\\top}\n\\right)^{\\top}\n\\end{equation*}\n\nand the exponent $t(k) \\in \\mathbb{N}$ indicates the number of consensus step. \n\n\n To state convergence results, we introduce the averaging of any $np$-dimensional vector $\\bold x$:\n\\begin{align}\n\\bar{\\bf x}=\\frac{1}{n}\\big(1_n1^{\\top}_n\\!\\!\\otimes I_p\\big){\\bold x}\n\\end{align}\n\nwhere $1_n$ is a column vector in $\\mathbb{R}^n$ whose entries are all $1$. In particular, we  have\n\\begin{align}\\label{11t}\n\\bar{{\\bf x}}_k=\\frac{1}{n}\\big(1_n1^{\\top}_n\\!\\!\\otimes I_p\\big){\\bf x}_k~~\\mbox{  and  }~~ \n\\overline{\\nabla  F} (\\mathbf{x}_k)=\\frac{1}{n}\\big(1_n1^{\\top}_n\\!\\!\\otimes I_p\\big)\\nabla F (\\mathbf{x}_k).\n\\end{align}\n\nWe note that\n$$\n\\bar{{\\bf x}}_k=1_n\\otimes\\left(\\frac{1}{n}\\sum_{i=1}^nx_{i,k}\\right)=1_n\\otimes \\bar x_k= (\\bar x_{k}^{\\top},\\bar x_{k}^{\\top},\\cdots,\\bar x_{k}^{\\top})^{\\top}.\n$$\n\nUsing these notation for averaged quantities, we derive the following averaged version of (\\ref{NEAR DGD}) which plays an important role as a useful intermediate step in the convergence analysis:\n\\begin{equation}\\label{average}\n\\bar{{\\bf x}}_{k+1} = \\bar{{\\bf x}}_k - \\mu \\overline{\\nabla F}({\\bf x}_k),\n\\end{equation}\n\nwhere we used the property that $1_n^\\top W = 1_n^\\top$ coming from the column-stochasticity of the consensus matrix $W$.\n","units":[{"start":4123,"end":4223,"start_line":113,"end_line":115,"raw_tex":"\\begin{align}\\label{1.1}\n\\min_{x\\in\\mathbb{R}^p} f(x):= \\frac{1}{n} \\sum_{i=1}^nf_i(x) ,\n\\end{align}","sha256_utf8":"1b9d0a8dfa313ac5a20acf2af95d8812a19d330e16b9687bb322a072b6249489"},{"start":4304,"end":4444,"start_line":117,"end_line":117,"raw_tex":"Here $f_i:\\mathbb{R}^p\\rightarrow\\mathbb{R}$ is a local objective function (or cost function of agent $i$) and $n$ is the number of agents.\n","sha256_utf8":"d80134a7ac6ad276b6c9db6e3b6ab570c35f56b440dd3ff4ebfffb2ba863a15a"},{"start":5232,"end":5333,"start_line":137,"end_line":140,"raw_tex":"where $F$ denotes the aggregate cost function in (\\ref{1.1}):\n\\[\nF({\\bf x}) =\\sum_{i=1}^nf_i(x_i).\n\\]","sha256_utf8":"169f5fb5d86256addfe14f3e8dde4ee236af543946fdbb730a24aadd36c0bf57"},{"start":5334,"end":5494,"start_line":141,"end_line":142,"raw_tex":"Here $I_p$ is the $p\\times p$ identity matrix, ${\\bf x}$ denotes an $np$ dimensional column vector made by stacking up $x_i\\in\\mathbb{R}^p$ $(i=1,\\cdots,n)$:\n \n","sha256_utf8":"2abeb9a8435f9d9939fcfdee974fc8deee320fa257e0cf31988a06e244c7e6a1"},{"start":5495,"end":5566,"start_line":143,"end_line":143,"raw_tex":"$${\\bf x}= (x_{1}^{\\top}, x_{2}^{\\top}, \\cdots, x_{n}^{\\top})^{\\top},$$","sha256_utf8":"f257d17e6a46f8bf78eb139092274f7f5a5c70d0de1253050fef7feca16e9377"},{"start":5567,"end":5987,"start_line":144,"end_line":149,"raw_tex":"  and the consensus matrix $W$ is the  $n\\times n$ matrix that contains the information on the connectivity of the underlying network:\n\\begin{equation}\\label{stack}\n%{\\bf x}=\\begin{pmatrix} x_1 \\\\  \\vdots \\\\ x_n\\end{pmatrix},\\quad \nW= \\begin{pmatrix} w_{11} & w_{12} & \\cdots & w_{1n} \\\\ w_{21} & w_{22} & \\cdots &w_{2n} \\\\ \\vdots &\\vdots &\\ddots &\\vdots \\\\ w_{n1} & w_{n2} &\\cdots &w_{nn}\n\\end{pmatrix}. \n\\end{equation}","sha256_utf8":"72f9c2359b10393a9330c5e635551070f57540530da25aefde0f90dc885216fc"},{"start":6363,"end":6737,"start_line":159,"end_line":166,"raw_tex":"Throughout this paper, we assume the following properties for the consensus matrix:%\\newline\n%{\\bf Assumptions on the consensus matrix }$W$:\n%We assume the consensus matrix\n{\\color{black}\n\\begin{itemize}\n\\item $W$ is a doubly-stochastic $n\\times n$ matrix.\n\\item The directed graph associated to $W$ is strongly connected, and has a self-loop at every vertex.\n\\end{itemize}\n","sha256_utf8":"5dfab364bbf2ab86c5b2fe48e832384b53eecd6af5951d3c3e4a452b56323e00"},{"start":7448,"end":8129,"start_line":169,"end_line":172,"raw_tex":"In Lemma \\ref{newlemma}, we will show that the graph associated to $W^\\top W$ is connected, and so the eigenvalue $1$ of $W^\\top W$ is simple and all other eigenvalues have modulus strictly less than 1. We let $\\beta\\in[0,1)$ be the square root of the second largest eigenvalue of $W^\\top W$, which is equal to the spectral norm of $W-\\frac{1}{n}1_n1_n^\\top$, where $1_n$ is a column vector in $\\mathbb{R}^n$ whose entries are all $1$.} We let $f^*$ be the optimal objective value of \\eqref{1.1} and denote by ${X}^*$ the set of optimal solutions to \\eqref{1.1} which is assumed to be nonempty, i.e., \n\\begin{equation*}\n{X}^* = \\{x \\in \\mathbb{R}^p ~:~ f(x) =f^*\\}.\n\\end{equation*}","sha256_utf8":"5b3f221fc2b4845d90cd969a8d66e174e24f3c33f38ab882588004cb18e7a8a6"},{"start":12078,"end":12252,"start_line":195,"end_line":195,"raw_tex":"\\noindent {\\bf Notation.} $\\|\\cdot\\|$ denotes the standard $\\ell^2$-norm in the Euclidean space, or the operator norm induced by the $\\ell^2$-norm, depending on the context.\n","sha256_utf8":"f3c42b3446b326d3eb599c2a8c1c84d79a3bb6dcb8ea66a290c15ca4c58188e8"},{"start":12393,"end":12739,"start_line":202,"end_line":206,"raw_tex":"\\begin{defn}For given $\\alpha \\geq 0$, a function $\\mathbf{f}: \\mathbb{R}^p \\rightarrow \\mathbb{R}$ is called $\\alpha$-strongly convex (convex if $\\alpha =0$) if\n\\begin{equation*}\n\\mathbf{f}(y) \\geq \\mathbf{f}(x) + \\langle y-x, \\nabla \\mathbf{f}(x) \\rangle+ \\frac{\\alpha}{2} \\|y-x\\|^2\\quad \\forall~x,y \\in \\mathbb{R}^p.\n\\end{equation*}\n\\end{defn}","sha256_utf8":"c4ded88e4e869246e61f3886ee98b0e5819538b8772295715eb7359d0a38c23d"},{"start":12740,"end":13074,"start_line":207,"end_line":211,"raw_tex":"\\begin{defn}We  say that a function $\\mathbf{f}: \\mathbb{R}^p \\rightarrow \\mathbb{R}$ is called $L$-smooth for given   $L >0$ if $f$ has Lipschitz continuous gradient with constant $L$, i.e., \n\\begin{equation*}\n\\|\\nabla \\mathbf{f}(x) - \\nabla \\mathbf{f}(y)\\| \\leq L \\|x-y\\|\\quad \\forall~x,y \\in\\mathbb{R}^p.\n\\end{equation*}\n\\end{defn}","sha256_utf8":"df12912af46fcd2f6d1c367f40776e4f7f4f5f3ecbab9d137ffcb38cab3f323c"},{"start":16437,"end":16620,"start_line":263,"end_line":265,"raw_tex":"\\begin{equation}\\label{NEAR DGD}\n{\\bf {\\bf x}}_{k+1} = \\big(W^{t(k)}\\otimes I_p\\big)\\big({\\bf x}_k - \\mu \\nabla F({\\bf x}_k)\\big)\\quad \\forall~ k \\geq0,%\\in \\mathbb{N},\n\\end{equation}","sha256_utf8":"6f1b29b891a897d1544203fe52e6ba880e976ed2641e2ad7cef531e442030098"},{"start":16622,"end":16916,"start_line":266,"end_line":272,"raw_tex":"where ${\\bf x}_k= (x_{1,k}^{\\top}, x_{2,k}^{\\top}, \\cdots, x_{n,k}^{\\top})^{\\top}$ and $\\mu >0$. Here  $\\nabla F (\\mathbf{x}_k)\\in \\mathbb{R}^{np}$ denotes\n\\begin{equation*}\n\\nabla F({\\bf x}_k)=\\left(\n\\nabla f_1(x_{1,k})^{\\top},\\cdots\n, \\nabla f_n(x_{n,k})^{\\top}\n\\right)^{\\top}\n\\end{equation*}","sha256_utf8":"4d59ac0e5bb334bd5958584044f109ecc532cfed132c45f07d5a388d617ab30a"},{"start":16917,"end":16997,"start_line":273,"end_line":273,"raw_tex":"and the exponent $t(k) \\in \\mathbb{N}$ indicates the number of consensus step. \n","sha256_utf8":"bcf6e89ad297c09cb867730356fad4eeb04b0012818475bfb7cde2ab952d5d9a"},{"start":16998,"end":17194,"start_line":275,"end_line":278,"raw_tex":" To state convergence results, we introduce the averaging of any $np$-dimensional vector $\\bold x$:\n\\begin{align}\n\\bar{\\bf x}=\\frac{1}{n}\\big(1_n1^{\\top}_n\\!\\!\\otimes I_p\\big){\\bold x}\n\\end{align}","sha256_utf8":"ed5b57ff96d87a2d82b0364fed45863373cdf4eeb1114d741f87ff0de8e5f9ee"},{"start":17195,"end":17535,"start_line":279,"end_line":283,"raw_tex":"where $1_n$ is a column vector in $\\mathbb{R}^n$ whose entries are all $1$. In particular, we  have\n\\begin{align}\\label{11t}\n\\bar{{\\bf x}}_k=\\frac{1}{n}\\big(1_n1^{\\top}_n\\!\\!\\otimes I_p\\big){\\bf x}_k~~\\mbox{  and  }~~ \n\\overline{\\nabla  F} (\\mathbf{x}_k)=\\frac{1}{n}\\big(1_n1^{\\top}_n\\!\\!\\otimes I_p\\big)\\nabla F (\\mathbf{x}_k).\n\\end{align}","sha256_utf8":"e9df616b4b4f7836a5f84cfb98fd5ca64760c478dbbe009f3d70d0b3f603c650"},{"start":17536,"end":17716,"start_line":284,"end_line":287,"raw_tex":"We note that\n$$\n\\bar{{\\bf x}}_k=1_n\\otimes\\left(\\frac{1}{n}\\sum_{i=1}^nx_{i,k}\\right)=1_n\\otimes \\bar x_k= (\\bar x_{k}^{\\top},\\bar x_{k}^{\\top},\\cdots,\\bar x_{k}^{\\top})^{\\top}.\n$$","sha256_utf8":"99d12911b1d7cb58f51bab2aa335d4847170a097f1aa5dfe1a0434ab42510bd4"},{"start":17717,"end":18035,"start_line":288,"end_line":291,"raw_tex":"Using these notation for averaged quantities, we derive the following averaged version of (\\ref{NEAR DGD}) which plays an important role as a useful intermediate step in the convergence analysis:\n\\begin{equation}\\label{average}\n\\bar{{\\bf x}}_{k+1} = \\bar{{\\bf x}}_k - \\mu \\overline{\\nabla F}({\\bf x}_k),\n\\end{equation}","sha256_utf8":"6bafbd13a57df4daf852c76fa219d8029ad3a589a358d45cc0776ae2fa7bb405"},{"start":18036,"end":18158,"start_line":292,"end_line":292,"raw_tex":"where we used the property that $1_n^\\top W = 1_n^\\top$ coming from the column-stochasticity of the consensus matrix $W$.\n","sha256_utf8":"e679064930e15a0f4849f4d51f5e095d8fbdeb9ef2f435af1b9d6a530efde074"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{thm}\\label{thm-1-2} Suppose that the cost function $f$ is convex and each local cost $f_i$ is $L_i$-smooth for some $L_i>0$.  Assume that $\\mu \\in (0, 1/L]$ with $L = \\max_{1 \\leq i \\leq n} L_i$ and the sequence $\\{t(k)\\}_{k \\in \\mathbb{N}\\cup\\{0\\}}$ satisfies $\\sum_{k=0}^{\\infty}\\beta^{t(k)}<\\infty$. \n\t Then   we have\n\t \\begin{equation*}\n\t f\\Bigg(\\frac{1}{T} \\sum_{k=0}^{T-1}  \\bar{x}_k \\Bigg) - f^*= O\\Big( \\frac{1}{T}\\Big)\n\t \\end{equation*}\n\t for $T \\in \\mathbb{N}$.\n\\end{thm}","units":[{"start":21145,"end":21633,"start_line":326,"end_line":332,"raw_tex":"\\begin{thm}\\label{thm-1-2} Suppose that the cost function $f$ is convex and each local cost $f_i$ is $L_i$-smooth for some $L_i>0$.  Assume that $\\mu \\in (0, 1/L]$ with $L = \\max_{1 \\leq i \\leq n} L_i$ and the sequence $\\{t(k)\\}_{k \\in \\mathbb{N}\\cup\\{0\\}}$ satisfies $\\sum_{k=0}^{\\infty}\\beta^{t(k)}<\\infty$. \n\t Then   we have\n\t \\begin{equation*}\n\t f\\Bigg(\\frac{1}{T} \\sum_{k=0}^{T-1}  \\bar{x}_k \\Bigg) - f^*= O\\Big( \\frac{1}{T}\\Big)\n\t \\end{equation*}\n\t for $T \\in \\mathbb{N}$.\n\\end{thm}","sha256_utf8":"313a67b33ba183edf3c1ab2d851a89ffcae35534fbbe5ce67c96bae5a39fdc8e"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{thm}\\label{Main1} Suppose that the cost function $f$ is convex and each local cost $f_i$ is $L_i$-smooth \nfor some $L_i>0$. Assume that $\\mu \\in (0,2/L]$ with $L = \\max_{1 \\leq i \\leq n} L_i$ and the sequence $\\{t(k)\\}_{k \\in \\mathbb{N}\\cup\\{0\\}}$ satisfies $\\sum_{k=0}^\\infty \\beta^{t(k)} <\\infty$. Then, for any minimizer $x_*\\in X^*$ , we have%there exists a constant $R>0$ such that\n\\begin{equation}\\label{eq-1-1}\n\\|{\\bar{\\bf x}}_k -{\\bf x}_*\\|=O(1)\\quad \\textrm{and}\\quad\\|\\bar{\\bold x}_{k+1} -\\bold x_{k+1}\\| = O(\\beta^{t(k)}),\\quad k\\to\\infty,%\\forall~k \\in \\mathbb{N},\n\\end{equation}\nwhere %$Q = \\Big[ (1+2\\mu L) R + \\mu D \\Big]$ and \n$\\bold x_*:=1_n\\otimes x_* =(x_*^{\\top},x_*^{\\top},\\cdots,x_*^{\\top})^{\\top}$. \t\n%\t Then, there exists $\\bar\\mu$ such that if  $\\mu<\\bar\\mu$, then the discrepancy from the optimal solution $\\|{\\bf x}_k-{\\bf x}_*\\|$ is bounded uniformly in $k$.\n\\end{thm}\n\n\\begin{lem}\\label{lem-2-1} Assume that $f: \\mathbb{R}^n \\rightarrow \\mathbb{R}$ is $\\alpha$-strongly convex and $L$-smooth. Then, for any $x,y \\in \\mathbb{R}^n$ we have\n\\begin{equation*}\n\\langle \\nabla f(x) - \\nabla f(y), x-y \\rangle \\geq \\frac{L\\alpha}{L+\\alpha}\\|x-y\\|^2 + \\frac{1}{\\alpha +L} \\|\\nabla f(x) - \\nabla f(y)\\|^2.\n\\end{equation*}\n\\end{lem}\n\n\\begin{prop}\\label{prop-2-2} \nLet  $\\{X_k\\}_{k\\geq0}$ and $\\{Y_k\\}_{k\\geq0}$ be sequences of non-negative vectors in $\\mathbb{R}^2$ and $\\{M_k\\}_{k\\geq0}$ be a sequence of $2\\times 2$ matrices. Assume that $\\{X_k\\}_{k\\geq0}$, $\\{Y_k\\}_{k\\geq0}$ and $\\{M_k\\}_{k\\geq0}$ satisfy the following iterative relation:\n\t\\begin{equation*}\n\tX_{k+1} \\leq M_k X_k + Y_k.\n\t\\end{equation*}\n\tSuppose further that $\\{Y_k\\}_{k \\geq0}$ is a sequence of non-negative vectors such that $\\sum_{k=0}^{\\infty}\\big\\| Y_k \\big\\|< \\infty$, and $\\{M_k\\}_{k\\geq0}$ takes the following form:\n\t \\begin{equation*}\n\t M_k = \\begin{pmatrix} 1 & r \\\\ \\alpha_k & \\alpha_k \\end{pmatrix}\n\t \\end{equation*}\n\t for some fixed $r>0$ and a sequence of non-negative real numbers $\\{\\alpha_k\\}_{k\\geq0}$ satisfying  $\\sum_{k=0}^{\\infty} \\alpha_k <\\infty$. \n Then, we have\n \\begin{enumerate}\n \t \t\\item The matrix $M_b M_{b-1}\\dots M_a$   is uniformly bounded for $0 \\leq a < b< \\infty$.\n \t\\item  $\\{X_{k}\\}_{k\\geq0}$ is uniformly bounded.\n \t\\end{enumerate}\n\\end{prop}\n\n\\begin{proof}\n(1) We take $c = \\max\\{r,1\\}$ and define a weighted norm $\\|\\cdot\\|_{c}$ in $\\mathbb{R}^2$ by\n\\begin{equation*}\n\\|(x,y)^\\top\\|_c = |x| + c|y| \\quad \\textcolor{black}{\\mbox{for }x,y\\in\\mathbb{R}}.\n\\end{equation*}\nFor a matrix $M \\in \\mathbb{R}^{2\\times 2}$, we define the  operator norm by\n\\begin{equation*}\n\\|M\\|_{\\mathcal{L}} =\\sup_{v \\in \\mathbb{R}^2 \\setminus \\{0\\}} \\frac{\\|Mv\\|_{c}}{\\|v\\|_{c}}.\n\\end{equation*}\nWe claim that\n\\begin{equation*}\\label{eq-2-3}\n\\|M_k\\|_{\\mathcal{L}} \\leq 1+ c\\alpha_k.\n\\end{equation*}\nTo show this, we take an arbitrary $v = (x,y)^\\top \\in \\mathbb{R}^2$ and compute\n\\begin{equation*}\nM_kv=\\begin{pmatrix} 1 & r \\\\ \\alpha_k &\\alpha_k \\end{pmatrix} \\begin{pmatrix} x \\\\ y \\end{pmatrix} = \\begin{pmatrix} x + ry \\\\ \\alpha_k x + \\alpha_k y \\end{pmatrix},\n\\end{equation*}\nwhich has the weighted $L^1$ norm as\n\\begin{equation*}\n\\|M_kv\\|_{c} = |x+ry| + c |\\alpha_k x + \\alpha_k y| \\leq (1+c\\alpha_k)|x| + (r+c \\alpha_k )|y|.\n\\end{equation*}\nSince $c = \\max\\{r,1\\}$ we have\n\\begin{equation*}\nr+c \\alpha_k \\leq c (1+c\\alpha_k)\n\\end{equation*}\nso that\n\\begin{equation*}\n\\begin{split}\n\\|M_kv\\|_c&\\leq(1+c\\alpha_k) |x| + (r+c \\alpha_k ) |y|\n\\\\\n&\\leq (1+c\\alpha_k) (|x|+c|y|) = (1+c\\alpha_k) \\|v\\|_c.\n\\end{split}\n\\end{equation*}\nThis proves the claim. Using \\eqref{eq-2-3}, for any $1\\leq a<b<\\infty$ we  obtain\n\\begin{equation*}\n\\begin{split}\n\\Big\\| M_b M_{b-1}\\dots M_a \\Big\\|_{\\mathcal{L}}& \\leq \\prod_{k=a}^b \\|M_k \\|_{\\mathcal{L}}\n\\\\\n& \\leq \\prod_{k=a}^b (1+c\\alpha_k)\n\\\\\n& \\leq \\prod_{k=a}^b e^{c\\alpha_k} = \\exp \\Big(c\\sum_{k=a}^b \\alpha_k\\Big).\n\\end{split}\n\\end{equation*}\nCombining this with the assumption that $\\sum_{k=0}^{\\infty} \\alpha_k < \\infty$, we conclude that $\\Big\\| M_b M_{b-1}\\dots M_a \\Big\\|_{\\mathcal{L}} $ is uniformly bounded for $0\\leq a<b <\\infty$.\n\n\\noindent (2) By induction, the iteration implies that\n\\begin{equation}\\label{eq-2-10}\nX_{k+1} \\leq \\Big( M_k M_{k-1}\\dots M_0\\Big)X_0 + \\sum_{j=0}^{k-1} \\Big( M_k M_{k-1}\\dots M_{j+1}\\Big) Y_j + Y_k.\n\\end{equation}\nThis gives the desired uniform boundedness of $X_k$ since $M_b M_{b-1}\\dots M_a$ is uniformly bounded and $\\sum_{j=0}^{\\infty} \\big\\|Y_j \\big\\|< \\infty$. \n\\end{proof}\n\n\\begin{lem}\\label{newlemma}\\cite{QL}\nAssume that the consensus matrix $W$ satisfies the following:\n\\begin{itemize}\n\\item $W$ is a doubly-stochastic $n\\times n$ matrix.\n\\item The directed graph associated to $W$ is strongly connected, and has a self-loop at every vertex.\n\\end{itemize}\nThen the following assertions hold:\n\\begin{enumerate}\n\\item The graph associated to $W^\\top W$ is connected and aperiodic. (Since $W^\\top W$ is symmetric, we may either interpret its associated graph as a strongly connected directed graph, or a connected undirected graph, depending on preference.)\n\\item The largest eigenvalue of $W^\\top W$ is $1$, and it is simple.\n\\item The spectral norm $\\beta$ of $W-\\frac{1}{n}1_n1_n^\\top$ satisfies $\\beta\\in[0,1)$.\n\\end{enumerate}\n\\end{lem}\n\n\\begin{proof}\n(1) The assumption that the graph associated to $W$ has a self-loop at every vertex, is equivalent to $w_{ii}>0$ for all $1\\leq i\\leq n$. Let $w_0:=\\frac{1}{2}\\min\\limits_{1\\leq i\\leq n}w_{ii}>0$ and $\\widetilde W:=W-w_0 I_n$. Then all entries of $\\widetilde W$ are nonnegative, and the directed graph associated to $\\widetilde W$ is precisely the same as that of $W$. Note that\n\\[\nW^\\top W=(\\widetilde W+w_0I_n)^\\top(\\widetilde W+w_0I_n)\\geq w_0 \\widetilde W,\n\\]\nwith the inequality being in entrywise sense. Hence the graph associated to $W^\\top W$ contains that associated to $\\widetilde W$ (or $W$) as its subgraph, which is indeed strongly connected and aperiodc.\n\n(2) Since $W^\\top W$ is symmetric positive semidefinite, we can arrange its eigenvalues as $\\lambda_1\\geq \\lambda_2\\geq\\dots\\geq\\lambda_n\\geq0$. The Perron-Frobenius theorem combined with (1) yields $1=\\lambda_1>\\lambda_2$.\n\n(3) Since $W^\\top W$ is symmetric, it can be diagonalized by an orthogonal matrix, i.e.,\n\\[\nW^\\top W=Q \\operatorname{diag}(\\lambda_1,\\dots,\\lambda_n)Q^\\top,\\quad Q:=[u_1|\\dots|u_n],\n\\]\nwith $u_i$ being an eigenvector associated to $\\lambda_i$, $i=1,\\dots,n$. Note that $\\lambda_1=1$ and $u_1=\\frac{1}{\\sqrt{n}}1_n$, because $W$ is doubly-stochastic. Hence\n\\[\nW^\\top W=\\sum_{i=1}^n \\lambda_i u_iu_i^\\top=\\frac{1}{n}1_n1_n^\\top+\\sum_{i=2}^n \\lambda_i u_iu_i^\\top,\n\\]\nwhich yields\n\\begin{align*}\n\\left\\|W-\\frac{1}{n}1_n1_n^\\top\\right\\|&=\\sqrt{\\mbox{the largest eigenvalue of }(W-\\frac{1}{n}1_n1_n^\\top)^\\top(W-\\frac{1}{n}1_n1_n^\\top)}\\\\\n&=\\sqrt{\\mbox{the largest eigenvalue of }W^\\top W-\\frac{1}{n}1_n1_n^\\top}=\\sqrt{\\lambda_2}<1,\n\\end{align*}\nwhere the second equality follows using that $1_n 1_n^\\top W = 1_n 1_n^\\top$ and $W^\\top 1_n 1_n^\\top  = 1_n 1_n^\\top$ since $W$ is doubly-stochastic. The proof is finished.\n\\end{proof}\n\n\\begin{proof}[Proof of Theorem \\ref{Main1}]We define\n\\begin{equation}\\label{eq-3-10}\nA_k:= \\|\\bar{\\bf x}_k-{\\bf x}_*\\| = \\sqrt{n} \\|\\bar{x}_k - x_*\\|,\\quad B_k:=\\|{\\bf x}_k-\\bar{\\bf x}_k\\|\n\\end{equation}\nand proceed to obtain a recursive estimate for $A_k$ and $B_k$. We first estimate $B_{k+1}$ in terms of $A_k$ and $B_k$. For this, we use the algorithm \\eqref{NEAR DGD} along with \\eqref{average} and  \\eqref{11t} to find\n\\begin{equation}\\label{eq-3-30}\n\\begin{split}\n&\\|\\bar{{\\bf x}}_{k+1} - {\\bf x}_{k+1}\\|\\cr\n & = \\|(\\bar{{\\bf x}}_k - \\mu \\overline{\\nabla F}({\\bf x}_k)) - (W^{t(k)}\\otimes I_p) ({\\bf x}_k - \\mu \\nabla F({\\bf x}_k))\\|\n\\\\\n& = \\left\\| \\left[\\left( \\frac{1}{n} 1_n 1_n^\\top  - W^{t(k)}\\right)\\otimes I_p\\right] ({\\bf x}_k -\\mu \\nabla F({\\bf x}_k) \\right\\|\n\\\\\n&\\leq \\left\\| \\left[\\left( \\frac{1}{n} 1_n 1_n^\\top  - W^{t(k)}\\right)\\otimes I_p\\right] {\\bf x}_k \\right\\| + \\mu \\left\\| \\left[\\left( \\frac{1}{n} 1_n 1_n^\\top  - W^{t(k)}\\right)\\otimes I_p\\right] \\nabla F({\\bf x}_k)\\right\\|.\n\\end{split}\n\\end{equation}\nHere we can estimate the right hand side using \n\\begin{equation}\\label{eq-3-31}\n\\left\\{\\left( \\frac{1}{n} 1_n 1_n^\\top  - W^{t(k)}\\right)\\otimes I_p\\right\\} \\bar{\\bf x}_k=0 \n\t\\quad \\textrm{and}\\quad\n\t\\Big\\| \\frac{1}{n} 1_n 1_n^\\top  - W^{t(k)}\\Big\\|_2\\leq \\beta^{t(k)},\n\t\\end{equation}\nwhere the second inequality came from Lemma \\ref{newlemma} and the relation\n\\[\n W^{t(k)}-\\frac{1}{n} 1_n 1_n^\\top  = \\left(W-\\frac{1}{n} 1_n 1_n^\\top \\right)^{t(k)}.\n\\]\nUsing \\eqref{eq-3-31} in \\eqref{eq-3-30} and  the triangle inequality,  we  achieve\n\\begin{align}\\label{consensus estimate}\n\\begin{split}\n\\|\\bar{{\\bf x}}_{k+1} - {\\bf x}_{k+1}\\|&\\leq {}\\beta^{t(k)} \\|{\\bf x}_k - \\bar{\\bf x}_k\\| + {}\\mu \\beta^{t(k)} \\|\\nabla F({\\bf x}_k)\\|\\cr\n &\\leq {}\\beta^{t(k)} \\|{\\bf x}_k - \\bar{\\bf x}_k\\| + {}\\mu \\beta^{t(k)} \n \\left\\{\\|\\nabla F({\\bf x}_k)-\\nabla F(\\bar{\\bf x}_k)\\|\\right.\\cr\n &\\quad\\left.+\\|\\nabla F(\\bar{\\bf x}_k)-\\nabla F({\\bf x}_*)\\|\n +\\|\\nabla F({\\bf x}_*)\\|\n \\right\\}.\\cr\n \\end{split}\n \\end{align}\nWe note that the $L_j$-smoothness of $f_j$ yields that for $x, y \\in \\mathbb{R}^{np}$ we have\n\\begin{equation*}\n\\begin{split}\n\\| \\nabla F (x) - \\nabla F(y) \\| & = \\Bigg( \\sum_{j=1}^n \\|\\nabla f_j (x^j) - \\nabla f_j (y^j) \\|^2 \\Bigg)^{1/2} \n\\\\\n&\\leq \\Bigg(\\sum_{j=1}^n \\textcolor{black}{L_j^2 \\|x^j -y^j\\|^2} \\Bigg)^{1/2} \n\\\\\n& \\leq L \\|x-y\\|,\n\\end{split}\n\\end{equation*}\nwhere we denoted $x= ((x^1)^\\top, \\cdots, (x^n)^\\top)^\\top$ and $y= ((y^1)^\\top, \\cdots, (y^n)^\\top)^\\top$. Using this in \\eqref{consensus estimate} we obtain\n \\begin{equation}\\label{eq-3-32}\n \\begin{split}\n \\|\\bar{{\\bf x}}_{k+1} - {\\bf x}_{k+1}\\| &\\leq {}(1+\\mu L)\\beta^{t(k)} \n \\|{\\bf x}_k-\\bar{\\bf x}_k \\|\n +{}(\\mu L)\\beta^{t(k)}\\|\\bar{\\bf x}_k-{\\bf x}_*\\|\n +{}\\mu\\beta^{t(k)}D\n \\\\\n & ={}(1+\\mu L)\\beta^{t(k)} \n \\|{\\bf x}_k-\\bar{\\bf x}_k \\|\n + {}(\\mu \\sqrt{n} L)\\beta^{t(k)}\\|\\bar{x}_k-{x}_*\\|\n +{}\\mu\\beta^{t(k)}D,\n\\end{split}\n\\end{equation}\nwhere $D:=\\|\\nabla F({\\bf x}_*)\\|$. This gives\n\\begin{align}\\label{estimate A}\nB_{k+1} \\leq  {}(\\mu L)\\beta^{t(k)}A_k+{}(1+\\mu L)\\beta^{t(k)}B_k+{}\\mu\\beta^{t(k)} D. \n\\end{align}\nNext we estimate the error $A_{k+1}$ between the average and the optimizer $x_*$ in terms of $A_k$ and $B_k$. Using \\eqref{average} and the triangle inequality, we find\n\\begin{align}\\label{main comp}\n\\begin{split}\n\\|\\bar{{ x}}_{k+1} - { x}_*\\| \n &= \\bigg\\|\\bar{{x}}_k - {x}_* - \\frac{\\mu}{n} \\sum_{j=1}^n \\nabla f_j (x_{j,k})\\bigg\\|\\\\\n& = \\bigg\\|\\bar{{ x}}_k - {x}_* - \\frac{\\mu}{n} \\sum_{j=1}^n \\nabla f_j (\\bar{x}_k) + \\mu \\bigg( \\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (\\bar{x}_k)- \\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (x_{j,k})\\bigg)\\bigg\\|\\cr\n&\\leq \\|\\bar{{  x}}_k - {  x}_* - \\mu \\nabla f (\\bar{{ x}}_k) \\|+ \\mu \\bigg\\|\\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (\\bar{x}_k)- \\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (x_{j,k})\\bigg\\|.\n\\end{split}\n\\end{align}\nWe make use of $\\nabla f(x_*)=0$ to compute\n\\begin{equation}\\label{eq-3-1}\n\\begin{split}\n&\\|\\bar{{x}}_k- {x}_*-\\mu{\\nabla f}(\\bar{{x}}_k)\\|^2\\\\\n&\\quad= \\|\\bar{{ x}}_k-{ x}_*\\|^2\n-2\\mu\\langle  \\bar{{x}}_k-{x}_*,\\nabla f(\\bar{{ x}}_k)\\rangle\n+\\mu^2\\|\\nabla f(\\bar{{ x}}_k)\\|^2\\\\\n&\\quad=\\|\\bar{{x}}_k-{x}_*\\|^2\n-2\\mu\\langle  \\bar{{x}}_k-{x}_*, {\\nabla f}(\\bar{{x}}_k)\n- {\\nabla f}({{x}}_*)\\rangle\n+\\mu^2\\|{\\nabla f}(\\bar{{x}}_k)-{\\nabla f}({x}_*)\\|^2.\n\\end{split}\n\\end{equation}\nFrom Lemma \\ref{lem-2-1}, we find\n\\begin{equation}\\label{eq-3-2}\n\\langle \\bar{x}_k -{x}_*,  {\\nabla f}(\\bar{x}_k)- {\\nabla f}({ x}_*)\\rangle \\geq\\frac{1}{L}\\| \\textcolor{black}{{\\nabla f}(\\bar{ x}_k)}-{\\nabla f}({x}_*)\\|^2,\n\\end{equation}\nand apply it to \\eqref{eq-3-1} to obtain\n\\begin{align*}\n\\begin{split}\n& \\|\\bar{{x}}_k-{x}_*-\\mu{\\nabla f}(\\bar{{x}}_k)\\|^2\\\\\n&\\quad\\leq\\|\\bar{{x}}_k-{x}_*\\|^2+\\mu\\left(\\mu-\\frac{2}{L}\\right)\\|{\\nabla f}(\\bar{{x}}_k)-{\\nabla f}({{x}}_*)\\|^2\\cr\n&\\quad\\leq \\|\\bar{{x}}_k-{x}_*\\|^2\n\\end{split}\n\\end{align*}\nfor $\\mu \\in (0, 2/L]$.\nThis gives us the estimate\n\\begin{align}\\label{eq-3-6}\n&\\|\\bar{{x}}_k-{x}_*-\\mu {\\nabla f}(\\bar{{ x}}_k)\\|\n\\leq \\|\\bar{{x}}_k-{x}_*\\|.\n\\end{align}\nOn the other hand, we use the $L_j$-smoothness of $f_j$ and the triangle inequality to deduce\n\\begin{align}\\label{insert1}\n\\begin{split} \n \\bigg\\|\\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (\\bar{x}_k)- \\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (x_{j,k})\\bigg\\| & \\leq \\frac{1}{n} \\sum_{j=1}^n L_j \\|\\bar{x}_k - x_{j,k}\\|\n \\\\\n&\\leq \\frac{L}{\\sqrt{n}} \\|\\bar{{\\bf x}}_k - {\\bf x}_k\\|.\n\\end{split}\n\\end{align}\nCombining \\eqref{main comp},  \\eqref{eq-3-6} and \\eqref{insert1}, we get\n\\begin{align}\\label{line for A}\n\\begin{split}\n\\|\\bar{{x}}_{k+1}- {{x}}_*\\|\n\\leq \\|\\bar{{x}}_k- {{x}}_*\\|+ \\frac{\\mu L}{\\sqrt{n}}\\|{\\bf x}_k-\\bar{{\\bf x}}_k\\|.\n\\end{split}\n\\end{align}\nFrom \\eqref{estimate A} and \\eqref{line for A}, we get the following iterative relation:\n\\begin{equation}\\label{eq-3-50}\n\\left(\\begin{array}{c} A_{k+1}\\cr B_{k+1}\\end{array}\\right) \\leq\n\\left(\\begin{array}{cc} 1 &  \\mu L\\cr {}(\\mu L) \\beta^{t(k)}& {} (1+\\mu L)\\beta^{t(k)}\\end{array}\\right)\n\\left(\\begin{array}{c} A_{k}\\cr B_{k}\\end{array}\\right)\n+\n{}\\beta^{t(k)}\\left(\\begin{array}{c} 0\\cr \\mu D\\end{array}\\right),\n\\end{equation}\n\\textcolor{black}{\nwhich clearly implies\n\\begin{align*}\n\\left(\\begin{array}{c} A_{k+1}\\cr B_{k+1}\\end{array}\\right) \\leq\n\\left(\\begin{array}{cc} 1 &  \\mu L\\cr {}(1+\\mu L) \\beta^{t(k)}& {} (1+\\mu L)\\beta^{t(k)}\\end{array}\\right)\n\\left(\\begin{array}{c} A_{k}\\cr B_{k}\\end{array}\\right)\n+\n{}\\beta^{t(k)}\\left(\\begin{array}{c} 0\\cr \\mu D\\end{array}\\right). \n\\end{align*}\n}\nNow we apply Proposition \\ref{prop-2-2} to conclude that there exists $R>0$ such that\n\\begin{equation*}\n\\|\\bar{\\bf x}_k -\\bold x_*\\| \\leq R \\quad \\textrm{and}\\quad \\|\\bar{\\bold x}_k -\\bold x_k \\| \\leq R\n\\end{equation*}\nfor all $k \\in \\mathbb{N}$. By applying this bound to \\eqref{estimate A}, we obtain\n\\begin{equation*}\n\\|\\bar{\\bold x}_{k+1} -\\bold x_{k+1}\\| \\leq {}\\beta^{t(k)} \\Big[ (1+2\\mu L) R + \\mu D \\Big].\n\\end{equation*}\nThis completes the proof.\n\\end{proof}\n\n\\begin{proof}[Proof of Theorem \\ref{thm-1-2}]\nApplying \\eqref{eq-3-6} and \n\\eqref{insert1} to \\eqref{main comp}, we find\n\\begin{equation}\\label{eq-3-12}\n\\begin{split}\n\\|\\bar{x}_{k+1} -x_*\\|^2 & \\leq \\| \\bar{x}_k - x_* - \\mu {\\nabla f}(\\bar{x}_k) \\|^2\n\\\\\n&\\quad + \\frac{2 \\mu L }{\\sqrt{n}}\\|\\bar{x}_k - x_* \\| \\|\\bar{\\bold x}_k -\\bold x_k\\| + \\frac{\\mu^2 L^2}{n} \\|\\bar{\\bold x}_k -\\bold x_k\\|^2.\n\\end{split}\n\\end{equation}\nWe use {$\\nabla f(x_*)=0$} and apply \\eqref{eq-3-2} to obtain  \n\\begin{equation*}\n\t\\begin{split}\n\t\t- 2\\mu \\langle \\bar{{x}}_k -{x}_*,~ {\\nabla f}(\\bar{{ x}}_k) \\rangle\t\t& = - 2\\mu \\langle \\bar{{x}}_k -{x}_*,~  {\\nabla f}(\\bar{{x}}_k)-  {\\nabla f}( {x}_*) \\rangle \n\t\t \\\\\n\t\t& \\leq- \\mu \\langle \\bar{{x}}_k -{x}_*,~ {\\nabla f}(\\bar{{x}}_k) \\rangle - \\frac{\\mu}{L} \\|\\nabla%_{\\bar{\\bf x}} \nf(\\bar{{x}}_k)\\|^2\n\t\\\\\n\t\t& \\leq \\mu\\left\\{f({{x}}_*)-f(\\bar{{x}}_k)\\right\\}  - \\frac{\\mu}{L} \\|\\nabla%_{\\bar{\\bf x}}\n f(\\bar{{x}}_k)\\|^2,\n\t\\end{split}\n\\end{equation*}\nwhere we used \n%\\[\n%\\bar{{\\bf x}}_k -{\\bf x}_*=1_n\\otimes(\\bar{  x}_k -  x_*),\\quad \\overline{\\nabla F}(\\bar{{\\bf x}}_k)=\\frac{1}{n}\\big(1_n1^{\\top}_n\\!\\!\\otimes I_p\\big)\\nabla F (1_n\\otimes \\bar{  x}_k)=1_n\\otimes \\nabla f(  \\bar  x_k)\n%\\]\nthe convexity of $f$ in the second inequality. Using this we find\n\\begin{equation}\\label{insert2}\n\\begin{split}\n&\\|\\bar{{x}}_k - {x}_* - \\mu {\\nabla f}(\\bar{{x}}_k) \\|^2 \n\\\\\n&\\qquad = \\|\\bar{{x}}_k -{x}_*\\|^2 - 2\\mu \\langle \\bar{{x}}_k -{x}_*,~ {\\nabla f}(\\bar{{x}}_k) \\rangle + \\mu^2 \\|{\\nabla f}(\\bar{{x}}_k)\\|^2\n\\\\\n&\\qquad \\leq \\|\\bar{{x}}_k - {x}_*\\|^2 + \n\\mu\\left\\{f({{x}}_*)-f(\\bar{{x}}_k)\\right\\}\n + (\\mu^2 - \\mu/L) \\| {\\nabla f}(\\bar{{x}}_k)\\|^2.\n\\end{split}\n\\end{equation}\n\\iffalse\nIt also implies that\n\\begin{equation*}\n\\|\\bar{{x}}_k - {x}_* - \\mu  {\\nabla f}(\\bar{{x}}_k) \\| \\leq \\|\\bar{{x}}_k - {x}_*\\|.\n\\end{equation*}\n\\fi\nInserting (\\ref{insert2}) into \\eqref{eq-3-12}, we obtain\n\\begin{equation}\\label{above}\n\\begin{split}\n\\|\\bar{{x}}_{k+1} - {x}_*\\|^2 & \\leq \\|\\bar{{x}}_k - {x}_*\\|^2 +\n\\mu\\left\\{f({{x}}_*)-f(\\bar{{x}}_k)\\right\\}\n + (\\mu^2 - \\mu /L) \\|{\\nabla f}(\\bar{{x}}_k)\\|^2\n\\\\\n&\\quad +\\frac{2\\mu L}{\\sqrt{n}}\\|\\bar{{x}}_k -{x}_*\\| \\|\\bar{{\\bf x}}_k - {\\bf x}_k \\| + \\frac{\\mu^2 L^2}{n} \\|\\bar{{\\bf x}}_k - {\\bf x}_k \\|^2\n\\\\\n&\\leq \\|\\bar{{x}}_k - {x}_*\\|^2 + \\mu\\left\\{f({{x}}_*)-f(\\bar{{x}}_k)\\right\\}\n\\\\\n&\\quad + \\frac{2\\mu L}{\\sqrt{n}} \\|\\bar{{x}}_k -{x}_*\\| \\|\\bar{{\\bf x}}_k -{\\bf x}_k\\| + \\frac{\\mu^2 L^2}{n}\\|\\bar{{\\bf x}}_k - {\\bf x}_k\\|^2\n\\end{split}\n\\end{equation}\nprovided $\\mu\\in (0, 1/L]$.\n%This completes the proof of the averaged gradient error.\n%\\iffalse\n%Therefore,\n%\\begin{equation*}\n%\\begin{split}\n%\\|\\bar{{x}}_{k+1} - {x}_*\\|^2 \n%&\\leq (1+\\mu L\\delta)\\|\\bar{{\\bf x}}_k - {\\bf x}_*\\|^2 + \\mu\\left\\{F(\\bar{{\\bf x}}_*)-F(\\bar{{\\bf x}}_k)\\right\\}\n%\\\\\n%&\\quad + 2\\left(\\mu L+\\frac{\\mu}{\\delta}\\right) \\|\\bar{{\\bf x}}_k - {\\bf x}_k\\|^2\n%\\end{split}\n%\\end{equation*}\n%\\fi\n%\n%\\noindent$\\bullet$ Proof of Theorem \\ref{Main1}: \nCombining   \\eqref{eq-1-1} with (\\ref{above}), we obtain\n\\begin{equation*}\n\\|\\bar{{x}}_{k+1} - {x}_*\\|^2 \\leq \\|\\bar{{x}}_k - {x}_*\\|^2 + \\mu\\left\\{f({{x}}_*)-f(\\bar{{x}}_k)\\right\\} + %2\\mu L QR \nC\\beta^{t({k-1})}  + %\\mu^2 L^2 Q^2 \nC\\beta^{2t(k-1)}\n\\end{equation*}\nfor some constant $C>0$.\n%\\textcolor{red}{for all $k\\in\\mathbb N$. with $\\beta^{t(0)}$}\nSumming this up over $k={1},\\cdots T-1$, we get \n\\begin{equation*}\n\\begin{split}\n&\\mu\\sum_{k=1}^{T-1} \\left\\{f(\\bar{{x}}_k)-f({{x}}_*)\\right\\} + \\|\\bar{{x}}_{T} -{x}_*\\|^2 \n\\\\\n&\\quad \\leq \\|\\bar{{x}}_{1} -{x}_*\\|^2 +C \\sum_{k=0}^{{T-2}} \\beta^{t(k)}  + C\\sum_{k=0}^{{T-2}}  \\beta^{2t(k)}.\n\\end{split}\n\\end{equation*}\nDividing both sides by $T$ and applying the convexity of $f$ to the left-hand side, we obtain\n\\begin{equation*}\n\\begin{split}\n& f\\Bigg( \\frac{1}{T}\\sum_{k=0}^{T-1}  \\bar{x}_k \\Bigg) -f(\\bar{{x}}_*) \\\\\n&\\leq \\frac{1}{\\mu T}\\Bigg[{\\mu\\left\\{f(\\bar{{x}}_0)-f(\\bar{{x}}_*)\\right\\}}+\\|\\bar{{ x}}_{1} -{x}_*\\|^2 + C\\sum_{k=0}^{{T-1}} \\beta^{t(k)}  + C\\sum_{k=0}^{{T-1}} \\beta^{2t(k)}\\Bigg],\n\\end{split}\n\\end{equation*}\nwhich completes the proof.\n\\end{proof}","units":[{"start":20240,"end":21143,"start_line":316,"end_line":324,"raw_tex":"\\begin{thm}\\label{Main1} Suppose that the cost function $f$ is convex and each local cost $f_i$ is $L_i$-smooth \nfor some $L_i>0$. Assume that $\\mu \\in (0,2/L]$ with $L = \\max_{1 \\leq i \\leq n} L_i$ and the sequence $\\{t(k)\\}_{k \\in \\mathbb{N}\\cup\\{0\\}}$ satisfies $\\sum_{k=0}^\\infty \\beta^{t(k)} <\\infty$. Then, for any minimizer $x_*\\in X^*$ , we have%there exists a constant $R>0$ such that\n\\begin{equation}\\label{eq-1-1}\n\\|{\\bar{\\bf x}}_k -{\\bf x}_*\\|=O(1)\\quad \\textrm{and}\\quad\\|\\bar{\\bold x}_{k+1} -\\bold x_{k+1}\\| = O(\\beta^{t(k)}),\\quad k\\to\\infty,%\\forall~k \\in \\mathbb{N},\n\\end{equation}\nwhere %$Q = \\Big[ (1+2\\mu L) R + \\mu D \\Big]$ and \n$\\bold x_*:=1_n\\otimes x_* =(x_*^{\\top},x_*^{\\top},\\cdots,x_*^{\\top})^{\\top}$. \t\n%\t Then, there exists $\\bar\\mu$ such that if  $\\mu<\\bar\\mu$, then the discrepancy from the optimal solution $\\|{\\bf x}_k-{\\bf x}_*\\|$ is bounded uniformly in $k$.\n\\end{thm}","sha256_utf8":"4af633e42b3f684b339fd87c100915cf4e1ed0be4c4013ac9ed711b92d94a3c7"},{"start":37757,"end":38110,"start_line":549,"end_line":553,"raw_tex":"\\begin{lem}\\label{lem-2-1} Assume that $f: \\mathbb{R}^n \\rightarrow \\mathbb{R}$ is $\\alpha$-strongly convex and $L$-smooth. Then, for any $x,y \\in \\mathbb{R}^n$ we have\n\\begin{equation*}\n\\langle \\nabla f(x) - \\nabla f(y), x-y \\rangle \\geq \\frac{L\\alpha}{L+\\alpha}\\|x-y\\|^2 + \\frac{1}{\\alpha +L} \\|\\nabla f(x) - \\nabla f(y)\\|^2.\n\\end{equation*}\n\\end{lem}","sha256_utf8":"c7f0ef3619bd250ad62e456137b3f333931a726bf915df51d8f10970d9fb81cc"},{"start":38507,"end":39527,"start_line":560,"end_line":575,"raw_tex":"\\begin{prop}\\label{prop-2-2} \nLet  $\\{X_k\\}_{k\\geq0}$ and $\\{Y_k\\}_{k\\geq0}$ be sequences of non-negative vectors in $\\mathbb{R}^2$ and $\\{M_k\\}_{k\\geq0}$ be a sequence of $2\\times 2$ matrices. Assume that $\\{X_k\\}_{k\\geq0}$, $\\{Y_k\\}_{k\\geq0}$ and $\\{M_k\\}_{k\\geq0}$ satisfy the following iterative relation:\n\t\\begin{equation*}\n\tX_{k+1} \\leq M_k X_k + Y_k.\n\t\\end{equation*}\n\tSuppose further that $\\{Y_k\\}_{k \\geq0}$ is a sequence of non-negative vectors such that $\\sum_{k=0}^{\\infty}\\big\\| Y_k \\big\\|< \\infty$, and $\\{M_k\\}_{k\\geq0}$ takes the following form:\n\t \\begin{equation*}\n\t M_k = \\begin{pmatrix} 1 & r \\\\ \\alpha_k & \\alpha_k \\end{pmatrix}\n\t \\end{equation*}\n\t for some fixed $r>0$ and a sequence of non-negative real numbers $\\{\\alpha_k\\}_{k\\geq0}$ satisfying  $\\sum_{k=0}^{\\infty} \\alpha_k <\\infty$. \n Then, we have\n \\begin{enumerate}\n \t \t\\item The matrix $M_b M_{b-1}\\dots M_a$   is uniformly bounded for $0 \\leq a < b< \\infty$.\n \t\\item  $\\{X_{k}\\}_{k\\geq0}$ is uniformly bounded.\n \t\\end{enumerate}\n\\end{prop}","sha256_utf8":"0f5fef443be4c5fdaad76e8ef01287dc7ec74fbc745a46b67a5e7de2aa5826a7"},{"start":39530,"end":41725,"start_line":577,"end_line":627,"raw_tex":"\\begin{proof}\n(1) We take $c = \\max\\{r,1\\}$ and define a weighted norm $\\|\\cdot\\|_{c}$ in $\\mathbb{R}^2$ by\n\\begin{equation*}\n\\|(x,y)^\\top\\|_c = |x| + c|y| \\quad \\textcolor{black}{\\mbox{for }x,y\\in\\mathbb{R}}.\n\\end{equation*}\nFor a matrix $M \\in \\mathbb{R}^{2\\times 2}$, we define the  operator norm by\n\\begin{equation*}\n\\|M\\|_{\\mathcal{L}} =\\sup_{v \\in \\mathbb{R}^2 \\setminus \\{0\\}} \\frac{\\|Mv\\|_{c}}{\\|v\\|_{c}}.\n\\end{equation*}\nWe claim that\n\\begin{equation*}\\label{eq-2-3}\n\\|M_k\\|_{\\mathcal{L}} \\leq 1+ c\\alpha_k.\n\\end{equation*}\nTo show this, we take an arbitrary $v = (x,y)^\\top \\in \\mathbb{R}^2$ and compute\n\\begin{equation*}\nM_kv=\\begin{pmatrix} 1 & r \\\\ \\alpha_k &\\alpha_k \\end{pmatrix} \\begin{pmatrix} x \\\\ y \\end{pmatrix} = \\begin{pmatrix} x + ry \\\\ \\alpha_k x + \\alpha_k y \\end{pmatrix},\n\\end{equation*}\nwhich has the weighted $L^1$ norm as\n\\begin{equation*}\n\\|M_kv\\|_{c} = |x+ry| + c |\\alpha_k x + \\alpha_k y| \\leq (1+c\\alpha_k)|x| + (r+c \\alpha_k )|y|.\n\\end{equation*}\nSince $c = \\max\\{r,1\\}$ we have\n\\begin{equation*}\nr+c \\alpha_k \\leq c (1+c\\alpha_k)\n\\end{equation*}\nso that\n\\begin{equation*}\n\\begin{split}\n\\|M_kv\\|_c&\\leq(1+c\\alpha_k) |x| + (r+c \\alpha_k ) |y|\n\\\\\n&\\leq (1+c\\alpha_k) (|x|+c|y|) = (1+c\\alpha_k) \\|v\\|_c.\n\\end{split}\n\\end{equation*}\nThis proves the claim. Using \\eqref{eq-2-3}, for any $1\\leq a<b<\\infty$ we  obtain\n\\begin{equation*}\n\\begin{split}\n\\Big\\| M_b M_{b-1}\\dots M_a \\Big\\|_{\\mathcal{L}}& \\leq \\prod_{k=a}^b \\|M_k \\|_{\\mathcal{L}}\n\\\\\n& \\leq \\prod_{k=a}^b (1+c\\alpha_k)\n\\\\\n& \\leq \\prod_{k=a}^b e^{c\\alpha_k} = \\exp \\Big(c\\sum_{k=a}^b \\alpha_k\\Big).\n\\end{split}\n\\end{equation*}\nCombining this with the assumption that $\\sum_{k=0}^{\\infty} \\alpha_k < \\infty$, we conclude that $\\Big\\| M_b M_{b-1}\\dots M_a \\Big\\|_{\\mathcal{L}} $ is uniformly bounded for $0\\leq a<b <\\infty$.\n\n\\noindent (2) By induction, the iteration implies that\n\\begin{equation}\\label{eq-2-10}\nX_{k+1} \\leq \\Big( M_k M_{k-1}\\dots M_0\\Big)X_0 + \\sum_{j=0}^{k-1} \\Big( M_k M_{k-1}\\dots M_{j+1}\\Big) Y_j + Y_k.\n\\end{equation}\nThis gives the desired uniform boundedness of $X_k$ since $M_b M_{b-1}\\dots M_a$ is uniformly bounded and $\\sum_{j=0}^{\\infty} \\big\\|Y_j \\big\\|< \\infty$. \n\\end{proof}","sha256_utf8":"0b4c2d6de61ec4928dc84db57208525d4da14621e57ee3319d27e851702e4efc"},{"start":42005,"end":42772,"start_line":630,"end_line":642,"raw_tex":"\\begin{lem}\\label{newlemma}\\cite{QL}\nAssume that the consensus matrix $W$ satisfies the following:\n\\begin{itemize}\n\\item $W$ is a doubly-stochastic $n\\times n$ matrix.\n\\item The directed graph associated to $W$ is strongly connected, and has a self-loop at every vertex.\n\\end{itemize}\nThen the following assertions hold:\n\\begin{enumerate}\n\\item The graph associated to $W^\\top W$ is connected and aperiodic. (Since $W^\\top W$ is symmetric, we may either interpret its associated graph as a strongly connected directed graph, or a connected undirected graph, depending on preference.)\n\\item The largest eigenvalue of $W^\\top W$ is $1$, and it is simple.\n\\item The spectral norm $\\beta$ of $W-\\frac{1}{n}1_n1_n^\\top$ satisfies $\\beta\\in[0,1)$.\n\\end{enumerate}\n\\end{lem}","sha256_utf8":"394787773a0f5e08ae41a38143b1d813201b02c3143512183c4962f35d73167e"},{"start":42773,"end":44608,"start_line":643,"end_line":666,"raw_tex":"\\begin{proof}\n(1) The assumption that the graph associated to $W$ has a self-loop at every vertex, is equivalent to $w_{ii}>0$ for all $1\\leq i\\leq n$. Let $w_0:=\\frac{1}{2}\\min\\limits_{1\\leq i\\leq n}w_{ii}>0$ and $\\widetilde W:=W-w_0 I_n$. Then all entries of $\\widetilde W$ are nonnegative, and the directed graph associated to $\\widetilde W$ is precisely the same as that of $W$. Note that\n\\[\nW^\\top W=(\\widetilde W+w_0I_n)^\\top(\\widetilde W+w_0I_n)\\geq w_0 \\widetilde W,\n\\]\nwith the inequality being in entrywise sense. Hence the graph associated to $W^\\top W$ contains that associated to $\\widetilde W$ (or $W$) as its subgraph, which is indeed strongly connected and aperiodc.\n\n(2) Since $W^\\top W$ is symmetric positive semidefinite, we can arrange its eigenvalues as $\\lambda_1\\geq \\lambda_2\\geq\\dots\\geq\\lambda_n\\geq0$. The Perron-Frobenius theorem combined with (1) yields $1=\\lambda_1>\\lambda_2$.\n\n(3) Since $W^\\top W$ is symmetric, it can be diagonalized by an orthogonal matrix, i.e.,\n\\[\nW^\\top W=Q \\operatorname{diag}(\\lambda_1,\\dots,\\lambda_n)Q^\\top,\\quad Q:=[u_1|\\dots|u_n],\n\\]\nwith $u_i$ being an eigenvector associated to $\\lambda_i$, $i=1,\\dots,n$. Note that $\\lambda_1=1$ and $u_1=\\frac{1}{\\sqrt{n}}1_n$, because $W$ is doubly-stochastic. Hence\n\\[\nW^\\top W=\\sum_{i=1}^n \\lambda_i u_iu_i^\\top=\\frac{1}{n}1_n1_n^\\top+\\sum_{i=2}^n \\lambda_i u_iu_i^\\top,\n\\]\nwhich yields\n\\begin{align*}\n\\left\\|W-\\frac{1}{n}1_n1_n^\\top\\right\\|&=\\sqrt{\\mbox{the largest eigenvalue of }(W-\\frac{1}{n}1_n1_n^\\top)^\\top(W-\\frac{1}{n}1_n1_n^\\top)}\\\\\n&=\\sqrt{\\mbox{the largest eigenvalue of }W^\\top W-\\frac{1}{n}1_n1_n^\\top}=\\sqrt{\\lambda_2}<1,\n\\end{align*}\nwhere the second equality follows using that $1_n 1_n^\\top W = 1_n 1_n^\\top$ and $W^\\top 1_n 1_n^\\top  = 1_n 1_n^\\top$ since $W$ is doubly-stochastic. The proof is finished.\n\\end{proof}","sha256_utf8":"37ff477678398eabd81290dd6a47807eee745e0a6a2507822494f9be8c54ec28"},{"start":44810,"end":51790,"start_line":671,"end_line":817,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{Main1}]We define\n\\begin{equation}\\label{eq-3-10}\nA_k:= \\|\\bar{\\bf x}_k-{\\bf x}_*\\| = \\sqrt{n} \\|\\bar{x}_k - x_*\\|,\\quad B_k:=\\|{\\bf x}_k-\\bar{\\bf x}_k\\|\n\\end{equation}\nand proceed to obtain a recursive estimate for $A_k$ and $B_k$. We first estimate $B_{k+1}$ in terms of $A_k$ and $B_k$. For this, we use the algorithm \\eqref{NEAR DGD} along with \\eqref{average} and  \\eqref{11t} to find\n\\begin{equation}\\label{eq-3-30}\n\\begin{split}\n&\\|\\bar{{\\bf x}}_{k+1} - {\\bf x}_{k+1}\\|\\cr\n & = \\|(\\bar{{\\bf x}}_k - \\mu \\overline{\\nabla F}({\\bf x}_k)) - (W^{t(k)}\\otimes I_p) ({\\bf x}_k - \\mu \\nabla F({\\bf x}_k))\\|\n\\\\\n& = \\left\\| \\left[\\left( \\frac{1}{n} 1_n 1_n^\\top  - W^{t(k)}\\right)\\otimes I_p\\right] ({\\bf x}_k -\\mu \\nabla F({\\bf x}_k) \\right\\|\n\\\\\n&\\leq \\left\\| \\left[\\left( \\frac{1}{n} 1_n 1_n^\\top  - W^{t(k)}\\right)\\otimes I_p\\right] {\\bf x}_k \\right\\| + \\mu \\left\\| \\left[\\left( \\frac{1}{n} 1_n 1_n^\\top  - W^{t(k)}\\right)\\otimes I_p\\right] \\nabla F({\\bf x}_k)\\right\\|.\n\\end{split}\n\\end{equation}\nHere we can estimate the right hand side using \n\\begin{equation}\\label{eq-3-31}\n\\left\\{\\left( \\frac{1}{n} 1_n 1_n^\\top  - W^{t(k)}\\right)\\otimes I_p\\right\\} \\bar{\\bf x}_k=0 \n\t\\quad \\textrm{and}\\quad\n\t\\Big\\| \\frac{1}{n} 1_n 1_n^\\top  - W^{t(k)}\\Big\\|_2\\leq \\beta^{t(k)},\n\t\\end{equation}\nwhere the second inequality came from Lemma \\ref{newlemma} and the relation\n\\[\n W^{t(k)}-\\frac{1}{n} 1_n 1_n^\\top  = \\left(W-\\frac{1}{n} 1_n 1_n^\\top \\right)^{t(k)}.\n\\]\nUsing \\eqref{eq-3-31} in \\eqref{eq-3-30} and  the triangle inequality,  we  achieve\n\\begin{align}\\label{consensus estimate}\n\\begin{split}\n\\|\\bar{{\\bf x}}_{k+1} - {\\bf x}_{k+1}\\|&\\leq {}\\beta^{t(k)} \\|{\\bf x}_k - \\bar{\\bf x}_k\\| + {}\\mu \\beta^{t(k)} \\|\\nabla F({\\bf x}_k)\\|\\cr\n &\\leq {}\\beta^{t(k)} \\|{\\bf x}_k - \\bar{\\bf x}_k\\| + {}\\mu \\beta^{t(k)} \n \\left\\{\\|\\nabla F({\\bf x}_k)-\\nabla F(\\bar{\\bf x}_k)\\|\\right.\\cr\n &\\quad\\left.+\\|\\nabla F(\\bar{\\bf x}_k)-\\nabla F({\\bf x}_*)\\|\n +\\|\\nabla F({\\bf x}_*)\\|\n \\right\\}.\\cr\n \\end{split}\n \\end{align}\nWe note that the $L_j$-smoothness of $f_j$ yields that for $x, y \\in \\mathbb{R}^{np}$ we have\n\\begin{equation*}\n\\begin{split}\n\\| \\nabla F (x) - \\nabla F(y) \\| & = \\Bigg( \\sum_{j=1}^n \\|\\nabla f_j (x^j) - \\nabla f_j (y^j) \\|^2 \\Bigg)^{1/2} \n\\\\\n&\\leq \\Bigg(\\sum_{j=1}^n \\textcolor{black}{L_j^2 \\|x^j -y^j\\|^2} \\Bigg)^{1/2} \n\\\\\n& \\leq L \\|x-y\\|,\n\\end{split}\n\\end{equation*}\nwhere we denoted $x= ((x^1)^\\top, \\cdots, (x^n)^\\top)^\\top$ and $y= ((y^1)^\\top, \\cdots, (y^n)^\\top)^\\top$. Using this in \\eqref{consensus estimate} we obtain\n \\begin{equation}\\label{eq-3-32}\n \\begin{split}\n \\|\\bar{{\\bf x}}_{k+1} - {\\bf x}_{k+1}\\| &\\leq {}(1+\\mu L)\\beta^{t(k)} \n \\|{\\bf x}_k-\\bar{\\bf x}_k \\|\n +{}(\\mu L)\\beta^{t(k)}\\|\\bar{\\bf x}_k-{\\bf x}_*\\|\n +{}\\mu\\beta^{t(k)}D\n \\\\\n & ={}(1+\\mu L)\\beta^{t(k)} \n \\|{\\bf x}_k-\\bar{\\bf x}_k \\|\n + {}(\\mu \\sqrt{n} L)\\beta^{t(k)}\\|\\bar{x}_k-{x}_*\\|\n +{}\\mu\\beta^{t(k)}D,\n\\end{split}\n\\end{equation}\nwhere $D:=\\|\\nabla F({\\bf x}_*)\\|$. This gives\n\\begin{align}\\label{estimate A}\nB_{k+1} \\leq  {}(\\mu L)\\beta^{t(k)}A_k+{}(1+\\mu L)\\beta^{t(k)}B_k+{}\\mu\\beta^{t(k)} D. \n\\end{align}\nNext we estimate the error $A_{k+1}$ between the average and the optimizer $x_*$ in terms of $A_k$ and $B_k$. Using \\eqref{average} and the triangle inequality, we find\n\\begin{align}\\label{main comp}\n\\begin{split}\n\\|\\bar{{ x}}_{k+1} - { x}_*\\| \n &= \\bigg\\|\\bar{{x}}_k - {x}_* - \\frac{\\mu}{n} \\sum_{j=1}^n \\nabla f_j (x_{j,k})\\bigg\\|\\\\\n& = \\bigg\\|\\bar{{ x}}_k - {x}_* - \\frac{\\mu}{n} \\sum_{j=1}^n \\nabla f_j (\\bar{x}_k) + \\mu \\bigg( \\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (\\bar{x}_k)- \\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (x_{j,k})\\bigg)\\bigg\\|\\cr\n&\\leq \\|\\bar{{  x}}_k - {  x}_* - \\mu \\nabla f (\\bar{{ x}}_k) \\|+ \\mu \\bigg\\|\\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (\\bar{x}_k)- \\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (x_{j,k})\\bigg\\|.\n\\end{split}\n\\end{align}\nWe make use of $\\nabla f(x_*)=0$ to compute\n\\begin{equation}\\label{eq-3-1}\n\\begin{split}\n&\\|\\bar{{x}}_k- {x}_*-\\mu{\\nabla f}(\\bar{{x}}_k)\\|^2\\\\\n&\\quad= \\|\\bar{{ x}}_k-{ x}_*\\|^2\n-2\\mu\\langle  \\bar{{x}}_k-{x}_*,\\nabla f(\\bar{{ x}}_k)\\rangle\n+\\mu^2\\|\\nabla f(\\bar{{ x}}_k)\\|^2\\\\\n&\\quad=\\|\\bar{{x}}_k-{x}_*\\|^2\n-2\\mu\\langle  \\bar{{x}}_k-{x}_*, {\\nabla f}(\\bar{{x}}_k)\n- {\\nabla f}({{x}}_*)\\rangle\n+\\mu^2\\|{\\nabla f}(\\bar{{x}}_k)-{\\nabla f}({x}_*)\\|^2.\n\\end{split}\n\\end{equation}\nFrom Lemma \\ref{lem-2-1}, we find\n\\begin{equation}\\label{eq-3-2}\n\\langle \\bar{x}_k -{x}_*,  {\\nabla f}(\\bar{x}_k)- {\\nabla f}({ x}_*)\\rangle \\geq\\frac{1}{L}\\| \\textcolor{black}{{\\nabla f}(\\bar{ x}_k)}-{\\nabla f}({x}_*)\\|^2,\n\\end{equation}\nand apply it to \\eqref{eq-3-1} to obtain\n\\begin{align*}\n\\begin{split}\n& \\|\\bar{{x}}_k-{x}_*-\\mu{\\nabla f}(\\bar{{x}}_k)\\|^2\\\\\n&\\quad\\leq\\|\\bar{{x}}_k-{x}_*\\|^2+\\mu\\left(\\mu-\\frac{2}{L}\\right)\\|{\\nabla f}(\\bar{{x}}_k)-{\\nabla f}({{x}}_*)\\|^2\\cr\n&\\quad\\leq \\|\\bar{{x}}_k-{x}_*\\|^2\n\\end{split}\n\\end{align*}\nfor $\\mu \\in (0, 2/L]$.\nThis gives us the estimate\n\\begin{align}\\label{eq-3-6}\n&\\|\\bar{{x}}_k-{x}_*-\\mu {\\nabla f}(\\bar{{ x}}_k)\\|\n\\leq \\|\\bar{{x}}_k-{x}_*\\|.\n\\end{align}\nOn the other hand, we use the $L_j$-smoothness of $f_j$ and the triangle inequality to deduce\n\\begin{align}\\label{insert1}\n\\begin{split} \n \\bigg\\|\\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (\\bar{x}_k)- \\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (x_{j,k})\\bigg\\| & \\leq \\frac{1}{n} \\sum_{j=1}^n L_j \\|\\bar{x}_k - x_{j,k}\\|\n \\\\\n&\\leq \\frac{L}{\\sqrt{n}} \\|\\bar{{\\bf x}}_k - {\\bf x}_k\\|.\n\\end{split}\n\\end{align}\nCombining \\eqref{main comp},  \\eqref{eq-3-6} and \\eqref{insert1}, we get\n\\begin{align}\\label{line for A}\n\\begin{split}\n\\|\\bar{{x}}_{k+1}- {{x}}_*\\|\n\\leq \\|\\bar{{x}}_k- {{x}}_*\\|+ \\frac{\\mu L}{\\sqrt{n}}\\|{\\bf x}_k-\\bar{{\\bf x}}_k\\|.\n\\end{split}\n\\end{align}\nFrom \\eqref{estimate A} and \\eqref{line for A}, we get the following iterative relation:\n\\begin{equation}\\label{eq-3-50}\n\\left(\\begin{array}{c} A_{k+1}\\cr B_{k+1}\\end{array}\\right) \\leq\n\\left(\\begin{array}{cc} 1 &  \\mu L\\cr {}(\\mu L) \\beta^{t(k)}& {} (1+\\mu L)\\beta^{t(k)}\\end{array}\\right)\n\\left(\\begin{array}{c} A_{k}\\cr B_{k}\\end{array}\\right)\n+\n{}\\beta^{t(k)}\\left(\\begin{array}{c} 0\\cr \\mu D\\end{array}\\right),\n\\end{equation}\n\\textcolor{black}{\nwhich clearly implies\n\\begin{align*}\n\\left(\\begin{array}{c} A_{k+1}\\cr B_{k+1}\\end{array}\\right) \\leq\n\\left(\\begin{array}{cc} 1 &  \\mu L\\cr {}(1+\\mu L) \\beta^{t(k)}& {} (1+\\mu L)\\beta^{t(k)}\\end{array}\\right)\n\\left(\\begin{array}{c} A_{k}\\cr B_{k}\\end{array}\\right)\n+\n{}\\beta^{t(k)}\\left(\\begin{array}{c} 0\\cr \\mu D\\end{array}\\right). \n\\end{align*}\n}\nNow we apply Proposition \\ref{prop-2-2} to conclude that there exists $R>0$ such that\n\\begin{equation*}\n\\|\\bar{\\bf x}_k -\\bold x_*\\| \\leq R \\quad \\textrm{and}\\quad \\|\\bar{\\bold x}_k -\\bold x_k \\| \\leq R\n\\end{equation*}\nfor all $k \\in \\mathbb{N}$. By applying this bound to \\eqref{estimate A}, we obtain\n\\begin{equation*}\n\\|\\bar{\\bold x}_{k+1} -\\bold x_{k+1}\\| \\leq {}\\beta^{t(k)} \\Big[ (1+2\\mu L) R + \\mu D \\Big].\n\\end{equation*}\nThis completes the proof.\n\\end{proof}","sha256_utf8":"2751190c6952208a70be9939692eecd4368be1438249d28d29fd2d570c23edca"},{"start":51793,"end":55892,"start_line":820,"end_line":917,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{thm-1-2}]\nApplying \\eqref{eq-3-6} and \n\\eqref{insert1} to \\eqref{main comp}, we find\n\\begin{equation}\\label{eq-3-12}\n\\begin{split}\n\\|\\bar{x}_{k+1} -x_*\\|^2 & \\leq \\| \\bar{x}_k - x_* - \\mu {\\nabla f}(\\bar{x}_k) \\|^2\n\\\\\n&\\quad + \\frac{2 \\mu L }{\\sqrt{n}}\\|\\bar{x}_k - x_* \\| \\|\\bar{\\bold x}_k -\\bold x_k\\| + \\frac{\\mu^2 L^2}{n} \\|\\bar{\\bold x}_k -\\bold x_k\\|^2.\n\\end{split}\n\\end{equation}\nWe use {$\\nabla f(x_*)=0$} and apply \\eqref{eq-3-2} to obtain  \n\\begin{equation*}\n\t\\begin{split}\n\t\t- 2\\mu \\langle \\bar{{x}}_k -{x}_*,~ {\\nabla f}(\\bar{{ x}}_k) \\rangle\t\t& = - 2\\mu \\langle \\bar{{x}}_k -{x}_*,~  {\\nabla f}(\\bar{{x}}_k)-  {\\nabla f}( {x}_*) \\rangle \n\t\t \\\\\n\t\t& \\leq- \\mu \\langle \\bar{{x}}_k -{x}_*,~ {\\nabla f}(\\bar{{x}}_k) \\rangle - \\frac{\\mu}{L} \\|\\nabla%_{\\bar{\\bf x}} \nf(\\bar{{x}}_k)\\|^2\n\t\\\\\n\t\t& \\leq \\mu\\left\\{f({{x}}_*)-f(\\bar{{x}}_k)\\right\\}  - \\frac{\\mu}{L} \\|\\nabla%_{\\bar{\\bf x}}\n f(\\bar{{x}}_k)\\|^2,\n\t\\end{split}\n\\end{equation*}\nwhere we used \n%\\[\n%\\bar{{\\bf x}}_k -{\\bf x}_*=1_n\\otimes(\\bar{  x}_k -  x_*),\\quad \\overline{\\nabla F}(\\bar{{\\bf x}}_k)=\\frac{1}{n}\\big(1_n1^{\\top}_n\\!\\!\\otimes I_p\\big)\\nabla F (1_n\\otimes \\bar{  x}_k)=1_n\\otimes \\nabla f(  \\bar  x_k)\n%\\]\nthe convexity of $f$ in the second inequality. Using this we find\n\\begin{equation}\\label{insert2}\n\\begin{split}\n&\\|\\bar{{x}}_k - {x}_* - \\mu {\\nabla f}(\\bar{{x}}_k) \\|^2 \n\\\\\n&\\qquad = \\|\\bar{{x}}_k -{x}_*\\|^2 - 2\\mu \\langle \\bar{{x}}_k -{x}_*,~ {\\nabla f}(\\bar{{x}}_k) \\rangle + \\mu^2 \\|{\\nabla f}(\\bar{{x}}_k)\\|^2\n\\\\\n&\\qquad \\leq \\|\\bar{{x}}_k - {x}_*\\|^2 + \n\\mu\\left\\{f({{x}}_*)-f(\\bar{{x}}_k)\\right\\}\n + (\\mu^2 - \\mu/L) \\| {\\nabla f}(\\bar{{x}}_k)\\|^2.\n\\end{split}\n\\end{equation}\n\\iffalse\nIt also implies that\n\\begin{equation*}\n\\|\\bar{{x}}_k - {x}_* - \\mu  {\\nabla f}(\\bar{{x}}_k) \\| \\leq \\|\\bar{{x}}_k - {x}_*\\|.\n\\end{equation*}\n\\fi\nInserting (\\ref{insert2}) into \\eqref{eq-3-12}, we obtain\n\\begin{equation}\\label{above}\n\\begin{split}\n\\|\\bar{{x}}_{k+1} - {x}_*\\|^2 & \\leq \\|\\bar{{x}}_k - {x}_*\\|^2 +\n\\mu\\left\\{f({{x}}_*)-f(\\bar{{x}}_k)\\right\\}\n + (\\mu^2 - \\mu /L) \\|{\\nabla f}(\\bar{{x}}_k)\\|^2\n\\\\\n&\\quad +\\frac{2\\mu L}{\\sqrt{n}}\\|\\bar{{x}}_k -{x}_*\\| \\|\\bar{{\\bf x}}_k - {\\bf x}_k \\| + \\frac{\\mu^2 L^2}{n} \\|\\bar{{\\bf x}}_k - {\\bf x}_k \\|^2\n\\\\\n&\\leq \\|\\bar{{x}}_k - {x}_*\\|^2 + \\mu\\left\\{f({{x}}_*)-f(\\bar{{x}}_k)\\right\\}\n\\\\\n&\\quad + \\frac{2\\mu L}{\\sqrt{n}} \\|\\bar{{x}}_k -{x}_*\\| \\|\\bar{{\\bf x}}_k -{\\bf x}_k\\| + \\frac{\\mu^2 L^2}{n}\\|\\bar{{\\bf x}}_k - {\\bf x}_k\\|^2\n\\end{split}\n\\end{equation}\nprovided $\\mu\\in (0, 1/L]$.\n%This completes the proof of the averaged gradient error.\n%\\iffalse\n%Therefore,\n%\\begin{equation*}\n%\\begin{split}\n%\\|\\bar{{x}}_{k+1} - {x}_*\\|^2 \n%&\\leq (1+\\mu L\\delta)\\|\\bar{{\\bf x}}_k - {\\bf x}_*\\|^2 + \\mu\\left\\{F(\\bar{{\\bf x}}_*)-F(\\bar{{\\bf x}}_k)\\right\\}\n%\\\\\n%&\\quad + 2\\left(\\mu L+\\frac{\\mu}{\\delta}\\right) \\|\\bar{{\\bf x}}_k - {\\bf x}_k\\|^2\n%\\end{split}\n%\\end{equation*}\n%\\fi\n%\n%\\noindent$\\bullet$ Proof of Theorem \\ref{Main1}: \nCombining   \\eqref{eq-1-1} with (\\ref{above}), we obtain\n\\begin{equation*}\n\\|\\bar{{x}}_{k+1} - {x}_*\\|^2 \\leq \\|\\bar{{x}}_k - {x}_*\\|^2 + \\mu\\left\\{f({{x}}_*)-f(\\bar{{x}}_k)\\right\\} + %2\\mu L QR \nC\\beta^{t({k-1})}  + %\\mu^2 L^2 Q^2 \nC\\beta^{2t(k-1)}\n\\end{equation*}\nfor some constant $C>0$.\n%\\textcolor{red}{for all $k\\in\\mathbb N$. with $\\beta^{t(0)}$}\nSumming this up over $k={1},\\cdots T-1$, we get \n\\begin{equation*}\n\\begin{split}\n&\\mu\\sum_{k=1}^{T-1} \\left\\{f(\\bar{{x}}_k)-f({{x}}_*)\\right\\} + \\|\\bar{{x}}_{T} -{x}_*\\|^2 \n\\\\\n&\\quad \\leq \\|\\bar{{x}}_{1} -{x}_*\\|^2 +C \\sum_{k=0}^{{T-2}} \\beta^{t(k)}  + C\\sum_{k=0}^{{T-2}}  \\beta^{2t(k)}.\n\\end{split}\n\\end{equation*}\nDividing both sides by $T$ and applying the convexity of $f$ to the left-hand side, we obtain\n\\begin{equation*}\n\\begin{split}\n& f\\Bigg( \\frac{1}{T}\\sum_{k=0}^{T-1}  \\bar{x}_k \\Bigg) -f(\\bar{{x}}_*) \\\\\n&\\leq \\frac{1}{\\mu T}\\Bigg[{\\mu\\left\\{f(\\bar{{x}}_0)-f(\\bar{{x}}_*)\\right\\}}+\\|\\bar{{ x}}_{1} -{x}_*\\|^2 + C\\sum_{k=0}^{{T-1}} \\beta^{t(k)}  + C\\sum_{k=0}^{{T-1}} \\beta^{2t(k)}\\Bigg],\n\\end{split}\n\\end{equation*}\nwhich completes the proof.\n\\end{proof}","sha256_utf8":"676cbac4d9fe54b89a6eb2e076dbf1c97dd96bba089ab18b3c5cdd1d253ceca3"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"分布式目标与堆叠变量","source_units":[{"start":4123,"end":4223,"start_line":113,"end_line":115,"raw_tex":"\\begin{align}\\label{1.1}\n\\min_{x\\in\\mathbb{R}^p} f(x):= \\frac{1}{n} \\sum_{i=1}^nf_i(x) ,\n\\end{align}","sha256_utf8":"1b9d0a8dfa313ac5a20acf2af95d8812a19d330e16b9687bb322a072b6249489"},{"start":4304,"end":4444,"start_line":117,"end_line":117,"raw_tex":"Here $f_i:\\mathbb{R}^p\\rightarrow\\mathbb{R}$ is a local objective function (or cost function of agent $i$) and $n$ is the number of agents.\n","sha256_utf8":"d80134a7ac6ad276b6c9db6e3b6ab570c35f56b440dd3ff4ebfffb2ba863a15a"},{"start":5232,"end":5333,"start_line":137,"end_line":140,"raw_tex":"where $F$ denotes the aggregate cost function in (\\ref{1.1}):\n\\[\nF({\\bf x}) =\\sum_{i=1}^nf_i(x_i).\n\\]","sha256_utf8":"169f5fb5d86256addfe14f3e8dde4ee236af543946fdbb730a24aadd36c0bf57"},{"start":5334,"end":5494,"start_line":141,"end_line":142,"raw_tex":"Here $I_p$ is the $p\\times p$ identity matrix, ${\\bf x}$ denotes an $np$ dimensional column vector made by stacking up $x_i\\in\\mathbb{R}^p$ $(i=1,\\cdots,n)$:\n \n","sha256_utf8":"2abeb9a8435f9d9939fcfdee974fc8deee320fa257e0cf31988a06e244c7e6a1"},{"start":5495,"end":5566,"start_line":143,"end_line":143,"raw_tex":"$${\\bf x}= (x_{1}^{\\top}, x_{2}^{\\top}, \\cdots, x_{n}^{\\top})^{\\top},$$","sha256_utf8":"f257d17e6a46f8bf78eb139092274f7f5a5c70d0de1253050fef7feca16e9377"}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","title":"共识矩阵及网络假设","source_units":[{"start":5567,"end":5987,"start_line":144,"end_line":149,"raw_tex":"  and the consensus matrix $W$ is the  $n\\times n$ matrix that contains the information on the connectivity of the underlying network:\n\\begin{equation}\\label{stack}\n%{\\bf x}=\\begin{pmatrix} x_1 \\\\  \\vdots \\\\ x_n\\end{pmatrix},\\quad \nW= \\begin{pmatrix} w_{11} & w_{12} & \\cdots & w_{1n} \\\\ w_{21} & w_{22} & \\cdots &w_{2n} \\\\ \\vdots &\\vdots &\\ddots &\\vdots \\\\ w_{n1} & w_{n2} &\\cdots &w_{nn}\n\\end{pmatrix}. \n\\end{equation}","sha256_utf8":"72f9c2359b10393a9330c5e635551070f57540530da25aefde0f90dc885216fc"},{"start":6363,"end":6737,"start_line":159,"end_line":166,"raw_tex":"Throughout this paper, we assume the following properties for the consensus matrix:%\\newline\n%{\\bf Assumptions on the consensus matrix }$W$:\n%We assume the consensus matrix\n{\\color{black}\n\\begin{itemize}\n\\item $W$ is a doubly-stochastic $n\\times n$ matrix.\n\\item The directed graph associated to $W$ is strongly connected, and has a self-loop at every vertex.\n\\end{itemize}\n","sha256_utf8":"5dfab364bbf2ab86c5b2fe48e832384b53eecd6af5951d3c3e4a452b56323e00"}]},{"id":"S003","semantic_type":"notation","kind":"notation","title":"谱参数与非空最优集","source_units":[{"start":7448,"end":8129,"start_line":169,"end_line":172,"raw_tex":"In Lemma \\ref{newlemma}, we will show that the graph associated to $W^\\top W$ is connected, and so the eigenvalue $1$ of $W^\\top W$ is simple and all other eigenvalues have modulus strictly less than 1. We let $\\beta\\in[0,1)$ be the square root of the second largest eigenvalue of $W^\\top W$, which is equal to the spectral norm of $W-\\frac{1}{n}1_n1_n^\\top$, where $1_n$ is a column vector in $\\mathbb{R}^n$ whose entries are all $1$.} We let $f^*$ be the optimal objective value of \\eqref{1.1} and denote by ${X}^*$ the set of optimal solutions to \\eqref{1.1} which is assumed to be nonempty, i.e., \n\\begin{equation*}\n{X}^* = \\{x \\in \\mathbb{R}^p ~:~ f(x) =f^*\\}.\n\\end{equation*}","sha256_utf8":"5b3f221fc2b4845d90cd969a8d66e174e24f3c33f38ab882588004cb18e7a8a6"}]},{"id":"S004","semantic_type":"notation","kind":"notation","title":"欧氏范数约定","source_units":[{"start":12078,"end":12252,"start_line":195,"end_line":195,"raw_tex":"\\noindent {\\bf Notation.} $\\|\\cdot\\|$ denotes the standard $\\ell^2$-norm in the Euclidean space, or the operator norm induced by the $\\ell^2$-norm, depending on the context.\n","sha256_utf8":"f3c42b3446b326d3eb599c2a8c1c84d79a3bb6dcb8ea66a290c15ca4c58188e8"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"凸性与强凸性","source_units":[{"start":12393,"end":12739,"start_line":202,"end_line":206,"raw_tex":"\\begin{defn}For given $\\alpha \\geq 0$, a function $\\mathbf{f}: \\mathbb{R}^p \\rightarrow \\mathbb{R}$ is called $\\alpha$-strongly convex (convex if $\\alpha =0$) if\n\\begin{equation*}\n\\mathbf{f}(y) \\geq \\mathbf{f}(x) + \\langle y-x, \\nabla \\mathbf{f}(x) \\rangle+ \\frac{\\alpha}{2} \\|y-x\\|^2\\quad \\forall~x,y \\in \\mathbb{R}^p.\n\\end{equation*}\n\\end{defn}","sha256_utf8":"c4ded88e4e869246e61f3886ee98b0e5819538b8772295715eb7359d0a38c23d"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"梯度光滑性","source_units":[{"start":12740,"end":13074,"start_line":207,"end_line":211,"raw_tex":"\\begin{defn}We  say that a function $\\mathbf{f}: \\mathbb{R}^p \\rightarrow \\mathbb{R}$ is called $L$-smooth for given   $L >0$ if $f$ has Lipschitz continuous gradient with constant $L$, i.e., \n\\begin{equation*}\n\\|\\nabla \\mathbf{f}(x) - \\nabla \\mathbf{f}(y)\\| \\leq L \\|x-y\\|\\quad \\forall~x,y \\in\\mathbb{R}^p.\n\\end{equation*}\n\\end{defn}","sha256_utf8":"df12912af46fcd2f6d1c367f40776e4f7f4f5f3ecbab9d137ffcb38cab3f323c"}]},{"id":"S007","semantic_type":"algorithm","kind":"algorithm","title":"嵌套共识梯度迭代","source_units":[{"start":16437,"end":16620,"start_line":263,"end_line":265,"raw_tex":"\\begin{equation}\\label{NEAR DGD}\n{\\bf {\\bf x}}_{k+1} = \\big(W^{t(k)}\\otimes I_p\\big)\\big({\\bf x}_k - \\mu \\nabla F({\\bf x}_k)\\big)\\quad \\forall~ k \\geq0,%\\in \\mathbb{N},\n\\end{equation}","sha256_utf8":"6f1b29b891a897d1544203fe52e6ba880e976ed2641e2ad7cef531e442030098"},{"start":16622,"end":16916,"start_line":266,"end_line":272,"raw_tex":"where ${\\bf x}_k= (x_{1,k}^{\\top}, x_{2,k}^{\\top}, \\cdots, x_{n,k}^{\\top})^{\\top}$ and $\\mu >0$. Here  $\\nabla F (\\mathbf{x}_k)\\in \\mathbb{R}^{np}$ denotes\n\\begin{equation*}\n\\nabla F({\\bf x}_k)=\\left(\n\\nabla f_1(x_{1,k})^{\\top},\\cdots\n, \\nabla f_n(x_{n,k})^{\\top}\n\\right)^{\\top}\n\\end{equation*}","sha256_utf8":"4d59ac0e5bb334bd5958584044f109ecc532cfed132c45f07d5a388d617ab30a"},{"start":16917,"end":16997,"start_line":273,"end_line":273,"raw_tex":"and the exponent $t(k) \\in \\mathbb{N}$ indicates the number of consensus step. \n","sha256_utf8":"bcf6e89ad297c09cb867730356fad4eeb04b0012818475bfb7cde2ab952d5d9a"}]},{"id":"S008","semantic_type":"algorithm","kind":"algorithm","title":"平均量与平均更新式","source_units":[{"start":16998,"end":17194,"start_line":275,"end_line":278,"raw_tex":" To state convergence results, we introduce the averaging of any $np$-dimensional vector $\\bold x$:\n\\begin{align}\n\\bar{\\bf x}=\\frac{1}{n}\\big(1_n1^{\\top}_n\\!\\!\\otimes I_p\\big){\\bold x}\n\\end{align}","sha256_utf8":"ed5b57ff96d87a2d82b0364fed45863373cdf4eeb1114d741f87ff0de8e5f9ee"},{"start":17195,"end":17535,"start_line":279,"end_line":283,"raw_tex":"where $1_n$ is a column vector in $\\mathbb{R}^n$ whose entries are all $1$. In particular, we  have\n\\begin{align}\\label{11t}\n\\bar{{\\bf x}}_k=\\frac{1}{n}\\big(1_n1^{\\top}_n\\!\\!\\otimes I_p\\big){\\bf x}_k~~\\mbox{  and  }~~ \n\\overline{\\nabla  F} (\\mathbf{x}_k)=\\frac{1}{n}\\big(1_n1^{\\top}_n\\!\\!\\otimes I_p\\big)\\nabla F (\\mathbf{x}_k).\n\\end{align}","sha256_utf8":"e9df616b4b4f7836a5f84cfb98fd5ca64760c478dbbe009f3d70d0b3f603c650"},{"start":17536,"end":17716,"start_line":284,"end_line":287,"raw_tex":"We note that\n$$\n\\bar{{\\bf x}}_k=1_n\\otimes\\left(\\frac{1}{n}\\sum_{i=1}^nx_{i,k}\\right)=1_n\\otimes \\bar x_k= (\\bar x_{k}^{\\top},\\bar x_{k}^{\\top},\\cdots,\\bar x_{k}^{\\top})^{\\top}.\n$$","sha256_utf8":"99d12911b1d7cb58f51bab2aa335d4847170a097f1aa5dfe1a0434ab42510bd4"},{"start":17717,"end":18035,"start_line":288,"end_line":291,"raw_tex":"Using these notation for averaged quantities, we derive the following averaged version of (\\ref{NEAR DGD}) which plays an important role as a useful intermediate step in the convergence analysis:\n\\begin{equation}\\label{average}\n\\bar{{\\bf x}}_{k+1} = \\bar{{\\bf x}}_k - \\mu \\overline{\\nabla F}({\\bf x}_k),\n\\end{equation}","sha256_utf8":"6bafbd13a57df4daf852c76fa219d8029ad3a589a358d45cc0776ae2fa7bb405"},{"start":18036,"end":18158,"start_line":292,"end_line":292,"raw_tex":"where we used the property that $1_n^\\top W = 1_n^\\top$ coming from the column-stochasticity of the consensus matrix $W$.\n","sha256_utf8":"e679064930e15a0f4849f4d51f5e095d8fbdeb9ef2f435af1b9d6a530efde074"}]},{"id":"S009","semantic_type":"theorem","kind":"theorem","title":"凸目标的遍历收敛率","source_units":[{"start":21145,"end":21633,"start_line":326,"end_line":332,"raw_tex":"\\begin{thm}\\label{thm-1-2} Suppose that the cost function $f$ is convex and each local cost $f_i$ is $L_i$-smooth for some $L_i>0$.  Assume that $\\mu \\in (0, 1/L]$ with $L = \\max_{1 \\leq i \\leq n} L_i$ and the sequence $\\{t(k)\\}_{k \\in \\mathbb{N}\\cup\\{0\\}}$ satisfies $\\sum_{k=0}^{\\infty}\\beta^{t(k)}<\\infty$. \n\t Then   we have\n\t \\begin{equation*}\n\t f\\Bigg(\\frac{1}{T} \\sum_{k=0}^{T-1}  \\bar{x}_k \\Bigg) - f^*= O\\Big( \\frac{1}{T}\\Big)\n\t \\end{equation*}\n\t for $T \\in \\mathbb{N}$.\n\\end{thm}","sha256_utf8":"313a67b33ba183edf3c1ab2d851a89ffcae35534fbbe5ce67c96bae5a39fdc8e"}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","title":"主定理的光滑性与可和条件","source_units":[{"start":21145,"end":21456,"start_line":326,"end_line":326,"raw_tex":"\\begin{thm}\\label{thm-1-2} Suppose that the cost function $f$ is convex and each local cost $f_i$ is $L_i$-smooth for some $L_i>0$.  Assume that $\\mu \\in (0, 1/L]$ with $L = \\max_{1 \\leq i \\leq n} L_i$ and the sequence $\\{t(k)\\}_{k \\in \\mathbb{N}\\cup\\{0\\}}$ satisfies $\\sum_{k=0}^{\\infty}\\beta^{t(k)}<\\infty$. \n","sha256_utf8":"6494387a195e7d16e01442f46c8a14f3b21b2a57de3fbc8d9bc4649098ba1992"}]},{"id":"S011","semantic_type":"proof","kind":"proof","title":"遍历收敛率完整证明","source_units":[{"start":51793,"end":55892,"start_line":820,"end_line":917,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{thm-1-2}]\nApplying \\eqref{eq-3-6} and \n\\eqref{insert1} to \\eqref{main comp}, we find\n\\begin{equation}\\label{eq-3-12}\n\\begin{split}\n\\|\\bar{x}_{k+1} -x_*\\|^2 & \\leq \\| \\bar{x}_k - x_* - \\mu {\\nabla f}(\\bar{x}_k) \\|^2\n\\\\\n&\\quad + \\frac{2 \\mu L }{\\sqrt{n}}\\|\\bar{x}_k - x_* \\| \\|\\bar{\\bold x}_k -\\bold x_k\\| + \\frac{\\mu^2 L^2}{n} \\|\\bar{\\bold x}_k -\\bold x_k\\|^2.\n\\end{split}\n\\end{equation}\nWe use {$\\nabla f(x_*)=0$} and apply \\eqref{eq-3-2} to obtain  \n\\begin{equation*}\n\t\\begin{split}\n\t\t- 2\\mu \\langle \\bar{{x}}_k -{x}_*,~ {\\nabla f}(\\bar{{ x}}_k) \\rangle\t\t& = - 2\\mu \\langle \\bar{{x}}_k -{x}_*,~  {\\nabla f}(\\bar{{x}}_k)-  {\\nabla f}( {x}_*) \\rangle \n\t\t \\\\\n\t\t& \\leq- \\mu \\langle \\bar{{x}}_k -{x}_*,~ {\\nabla f}(\\bar{{x}}_k) \\rangle - \\frac{\\mu}{L} \\|\\nabla%_{\\bar{\\bf x}} \nf(\\bar{{x}}_k)\\|^2\n\t\\\\\n\t\t& \\leq \\mu\\left\\{f({{x}}_*)-f(\\bar{{x}}_k)\\right\\}  - \\frac{\\mu}{L} \\|\\nabla%_{\\bar{\\bf x}}\n f(\\bar{{x}}_k)\\|^2,\n\t\\end{split}\n\\end{equation*}\nwhere we used \n%\\[\n%\\bar{{\\bf x}}_k -{\\bf x}_*=1_n\\otimes(\\bar{  x}_k -  x_*),\\quad \\overline{\\nabla F}(\\bar{{\\bf x}}_k)=\\frac{1}{n}\\big(1_n1^{\\top}_n\\!\\!\\otimes I_p\\big)\\nabla F (1_n\\otimes \\bar{  x}_k)=1_n\\otimes \\nabla f(  \\bar  x_k)\n%\\]\nthe convexity of $f$ in the second inequality. Using this we find\n\\begin{equation}\\label{insert2}\n\\begin{split}\n&\\|\\bar{{x}}_k - {x}_* - \\mu {\\nabla f}(\\bar{{x}}_k) \\|^2 \n\\\\\n&\\qquad = \\|\\bar{{x}}_k -{x}_*\\|^2 - 2\\mu \\langle \\bar{{x}}_k -{x}_*,~ {\\nabla f}(\\bar{{x}}_k) \\rangle + \\mu^2 \\|{\\nabla f}(\\bar{{x}}_k)\\|^2\n\\\\\n&\\qquad \\leq \\|\\bar{{x}}_k - {x}_*\\|^2 + \n\\mu\\left\\{f({{x}}_*)-f(\\bar{{x}}_k)\\right\\}\n + (\\mu^2 - \\mu/L) \\| {\\nabla f}(\\bar{{x}}_k)\\|^2.\n\\end{split}\n\\end{equation}\n\\iffalse\nIt also implies that\n\\begin{equation*}\n\\|\\bar{{x}}_k - {x}_* - \\mu  {\\nabla f}(\\bar{{x}}_k) \\| \\leq \\|\\bar{{x}}_k - {x}_*\\|.\n\\end{equation*}\n\\fi\nInserting (\\ref{insert2}) into \\eqref{eq-3-12}, we obtain\n\\begin{equation}\\label{above}\n\\begin{split}\n\\|\\bar{{x}}_{k+1} - {x}_*\\|^2 & \\leq \\|\\bar{{x}}_k - {x}_*\\|^2 +\n\\mu\\left\\{f({{x}}_*)-f(\\bar{{x}}_k)\\right\\}\n + (\\mu^2 - \\mu /L) \\|{\\nabla f}(\\bar{{x}}_k)\\|^2\n\\\\\n&\\quad +\\frac{2\\mu L}{\\sqrt{n}}\\|\\bar{{x}}_k -{x}_*\\| \\|\\bar{{\\bf x}}_k - {\\bf x}_k \\| + \\frac{\\mu^2 L^2}{n} \\|\\bar{{\\bf x}}_k - {\\bf x}_k \\|^2\n\\\\\n&\\leq \\|\\bar{{x}}_k - {x}_*\\|^2 + \\mu\\left\\{f({{x}}_*)-f(\\bar{{x}}_k)\\right\\}\n\\\\\n&\\quad + \\frac{2\\mu L}{\\sqrt{n}} \\|\\bar{{x}}_k -{x}_*\\| \\|\\bar{{\\bf x}}_k -{\\bf x}_k\\| + \\frac{\\mu^2 L^2}{n}\\|\\bar{{\\bf x}}_k - {\\bf x}_k\\|^2\n\\end{split}\n\\end{equation}\nprovided $\\mu\\in (0, 1/L]$.\n%This completes the proof of the averaged gradient error.\n%\\iffalse\n%Therefore,\n%\\begin{equation*}\n%\\begin{split}\n%\\|\\bar{{x}}_{k+1} - {x}_*\\|^2 \n%&\\leq (1+\\mu L\\delta)\\|\\bar{{\\bf x}}_k - {\\bf x}_*\\|^2 + \\mu\\left\\{F(\\bar{{\\bf x}}_*)-F(\\bar{{\\bf x}}_k)\\right\\}\n%\\\\\n%&\\quad + 2\\left(\\mu L+\\frac{\\mu}{\\delta}\\right) \\|\\bar{{\\bf x}}_k - {\\bf x}_k\\|^2\n%\\end{split}\n%\\end{equation*}\n%\\fi\n%\n%\\noindent$\\bullet$ Proof of Theorem \\ref{Main1}: \nCombining   \\eqref{eq-1-1} with (\\ref{above}), we obtain\n\\begin{equation*}\n\\|\\bar{{x}}_{k+1} - {x}_*\\|^2 \\leq \\|\\bar{{x}}_k - {x}_*\\|^2 + \\mu\\left\\{f({{x}}_*)-f(\\bar{{x}}_k)\\right\\} + %2\\mu L QR \nC\\beta^{t({k-1})}  + %\\mu^2 L^2 Q^2 \nC\\beta^{2t(k-1)}\n\\end{equation*}\nfor some constant $C>0$.\n%\\textcolor{red}{for all $k\\in\\mathbb N$. with $\\beta^{t(0)}$}\nSumming this up over $k={1},\\cdots T-1$, we get \n\\begin{equation*}\n\\begin{split}\n&\\mu\\sum_{k=1}^{T-1} \\left\\{f(\\bar{{x}}_k)-f({{x}}_*)\\right\\} + \\|\\bar{{x}}_{T} -{x}_*\\|^2 \n\\\\\n&\\quad \\leq \\|\\bar{{x}}_{1} -{x}_*\\|^2 +C \\sum_{k=0}^{{T-2}} \\beta^{t(k)}  + C\\sum_{k=0}^{{T-2}}  \\beta^{2t(k)}.\n\\end{split}\n\\end{equation*}\nDividing both sides by $T$ and applying the convexity of $f$ to the left-hand side, we obtain\n\\begin{equation*}\n\\begin{split}\n& f\\Bigg( \\frac{1}{T}\\sum_{k=0}^{T-1}  \\bar{x}_k \\Bigg) -f(\\bar{{x}}_*) \\\\\n&\\leq \\frac{1}{\\mu T}\\Bigg[{\\mu\\left\\{f(\\bar{{x}}_0)-f(\\bar{{x}}_*)\\right\\}}+\\|\\bar{{ x}}_{1} -{x}_*\\|^2 + C\\sum_{k=0}^{{T-1}} \\beta^{t(k)}  + C\\sum_{k=0}^{{T-1}} \\beta^{2t(k)}\\Bigg],\n\\end{split}\n\\end{equation*}\nwhich completes the proof.\n\\end{proof}","sha256_utf8":"676cbac4d9fe54b89a6eb2e076dbf1c97dd96bba089ab18b3c5cdd1d253ceca3"}]},{"id":"S012","semantic_type":"proof_step","kind":"proof_step","title":"平均迭代的平方误差","source_units":[{"start":51793,"end":52216,"start_line":820,"end_line":829,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{thm-1-2}]\nApplying \\eqref{eq-3-6} and \n\\eqref{insert1} to \\eqref{main comp}, we find\n\\begin{equation}\\label{eq-3-12}\n\\begin{split}\n\\|\\bar{x}_{k+1} -x_*\\|^2 & \\leq \\| \\bar{x}_k - x_* - \\mu {\\nabla f}(\\bar{x}_k) \\|^2\n\\\\\n&\\quad + \\frac{2 \\mu L }{\\sqrt{n}}\\|\\bar{x}_k - x_* \\| \\|\\bar{\\bold x}_k -\\bold x_k\\| + \\frac{\\mu^2 L^2}{n} \\|\\bar{\\bold x}_k -\\bold x_k\\|^2.\n\\end{split}\n\\end{equation}\n","sha256_utf8":"f395db6078d65d5f343b1588c9d337a9fbfee04b436be98001b2eac5b1aac78a"}]},{"id":"S013","semantic_type":"proof_step","kind":"proof_step","title":"凸性给出目标下降项","source_units":[{"start":52216,"end":53492,"start_line":830,"end_line":857,"raw_tex":"We use {$\\nabla f(x_*)=0$} and apply \\eqref{eq-3-2} to obtain  \n\\begin{equation*}\n\t\\begin{split}\n\t\t- 2\\mu \\langle \\bar{{x}}_k -{x}_*,~ {\\nabla f}(\\bar{{ x}}_k) \\rangle\t\t& = - 2\\mu \\langle \\bar{{x}}_k -{x}_*,~  {\\nabla f}(\\bar{{x}}_k)-  {\\nabla f}( {x}_*) \\rangle \n\t\t \\\\\n\t\t& \\leq- \\mu \\langle \\bar{{x}}_k -{x}_*,~ {\\nabla f}(\\bar{{x}}_k) \\rangle - \\frac{\\mu}{L} \\|\\nabla%_{\\bar{\\bf x}} \nf(\\bar{{x}}_k)\\|^2\n\t\\\\\n\t\t& \\leq \\mu\\left\\{f({{x}}_*)-f(\\bar{{x}}_k)\\right\\}  - \\frac{\\mu}{L} \\|\\nabla%_{\\bar{\\bf x}}\n f(\\bar{{x}}_k)\\|^2,\n\t\\end{split}\n\\end{equation*}\nwhere we used \n%\\[\n%\\bar{{\\bf x}}_k -{\\bf x}_*=1_n\\otimes(\\bar{  x}_k -  x_*),\\quad \\overline{\\nabla F}(\\bar{{\\bf x}}_k)=\\frac{1}{n}\\big(1_n1^{\\top}_n\\!\\!\\otimes I_p\\big)\\nabla F (1_n\\otimes \\bar{  x}_k)=1_n\\otimes \\nabla f(  \\bar  x_k)\n%\\]\nthe convexity of $f$ in the second inequality. Using this we find\n\\begin{equation}\\label{insert2}\n\\begin{split}\n&\\|\\bar{{x}}_k - {x}_* - \\mu {\\nabla f}(\\bar{{x}}_k) \\|^2 \n\\\\\n&\\qquad = \\|\\bar{{x}}_k -{x}_*\\|^2 - 2\\mu \\langle \\bar{{x}}_k -{x}_*,~ {\\nabla f}(\\bar{{x}}_k) \\rangle + \\mu^2 \\|{\\nabla f}(\\bar{{x}}_k)\\|^2\n\\\\\n&\\qquad \\leq \\|\\bar{{x}}_k - {x}_*\\|^2 + \n\\mu\\left\\{f({{x}}_*)-f(\\bar{{x}}_k)\\right\\}\n + (\\mu^2 - \\mu/L) \\| {\\nabla f}(\\bar{{x}}_k)\\|^2.\n\\end{split}\n\\end{equation}\n","sha256_utf8":"cb222a9882732df1ceeef732f8221e47523ca142f1c6f94b382a9d7d040c985f"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"合并下降与共识误差","source_units":[{"start":53646,"end":54335,"start_line":864,"end_line":878,"raw_tex":"Inserting (\\ref{insert2}) into \\eqref{eq-3-12}, we obtain\n\\begin{equation}\\label{above}\n\\begin{split}\n\\|\\bar{{x}}_{k+1} - {x}_*\\|^2 & \\leq \\|\\bar{{x}}_k - {x}_*\\|^2 +\n\\mu\\left\\{f({{x}}_*)-f(\\bar{{x}}_k)\\right\\}\n + (\\mu^2 - \\mu /L) \\|{\\nabla f}(\\bar{{x}}_k)\\|^2\n\\\\\n&\\quad +\\frac{2\\mu L}{\\sqrt{n}}\\|\\bar{{x}}_k -{x}_*\\| \\|\\bar{{\\bf x}}_k - {\\bf x}_k \\| + \\frac{\\mu^2 L^2}{n} \\|\\bar{{\\bf x}}_k - {\\bf x}_k \\|^2\n\\\\\n&\\leq \\|\\bar{{x}}_k - {x}_*\\|^2 + \\mu\\left\\{f({{x}}_*)-f(\\bar{{x}}_k)\\right\\}\n\\\\\n&\\quad + \\frac{2\\mu L}{\\sqrt{n}} \\|\\bar{{x}}_k -{x}_*\\| \\|\\bar{{\\bf x}}_k -{\\bf x}_k\\| + \\frac{\\mu^2 L^2}{n}\\|\\bar{{\\bf x}}_k - {\\bf x}_k\\|^2\n\\end{split}\n\\end{equation}\nprovided $\\mu\\in (0, 1/L]$.\n","sha256_utf8":"140dd0dd6f141625f0b1c1f83387aacd678652e85acf9716a0766fa4746ed6c1"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"用有界性控制扰动","source_units":[{"start":54770,"end":55061,"start_line":893,"end_line":899,"raw_tex":"Combining   \\eqref{eq-1-1} with (\\ref{above}), we obtain\n\\begin{equation*}\n\\|\\bar{{x}}_{k+1} - {x}_*\\|^2 \\leq \\|\\bar{{x}}_k - {x}_*\\|^2 + \\mu\\left\\{f({{x}}_*)-f(\\bar{{x}}_k)\\right\\} + %2\\mu L QR \nC\\beta^{t({k-1})}  + %\\mu^2 L^2 Q^2 \nC\\beta^{2t(k-1)}\n\\end{equation*}\nfor some constant $C>0$.\n","sha256_utf8":"9e3ab680cb940cc3cd04f7a1380605bcabd0e54f63cb1573110a79419eac212a"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"求和得到望远镜界","source_units":[{"start":55124,"end":55441,"start_line":901,"end_line":908,"raw_tex":"Summing this up over $k={1},\\cdots T-1$, we get \n\\begin{equation*}\n\\begin{split}\n&\\mu\\sum_{k=1}^{T-1} \\left\\{f(\\bar{{x}}_k)-f({{x}}_*)\\right\\} + \\|\\bar{{x}}_{T} -{x}_*\\|^2 \n\\\\\n&\\quad \\leq \\|\\bar{{x}}_{1} -{x}_*\\|^2 +C \\sum_{k=0}^{{T-2}} \\beta^{t(k)}  + C\\sum_{k=0}^{{T-2}}  \\beta^{2t(k)}.\n\\end{split}\n\\end{equation*}\n","sha256_utf8":"38860bed87fed7f6770852f32efca101a69bd73c3a663c50472216195880d352"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"凸平均得到收敛率","source_units":[{"start":55441,"end":55892,"start_line":909,"end_line":917,"raw_tex":"Dividing both sides by $T$ and applying the convexity of $f$ to the left-hand side, we obtain\n\\begin{equation*}\n\\begin{split}\n& f\\Bigg( \\frac{1}{T}\\sum_{k=0}^{T-1}  \\bar{x}_k \\Bigg) -f(\\bar{{x}}_*) \\\\\n&\\leq \\frac{1}{\\mu T}\\Bigg[{\\mu\\left\\{f(\\bar{{x}}_0)-f(\\bar{{x}}_*)\\right\\}}+\\|\\bar{{ x}}_{1} -{x}_*\\|^2 + C\\sum_{k=0}^{{T-1}} \\beta^{t(k)}  + C\\sum_{k=0}^{{T-1}} \\beta^{2t(k)}\\Bigg],\n\\end{split}\n\\end{equation*}\nwhich completes the proof.\n\\end{proof}","sha256_utf8":"5687ddbc9fd892560513cbf01f39b6f3564b8ed95b66e1417b6d5574856eff1f"}]},{"id":"S018","semantic_type":"theorem","kind":"theorem","title":"迭代有界与共识误差","source_units":[{"start":20240,"end":21143,"start_line":316,"end_line":324,"raw_tex":"\\begin{thm}\\label{Main1} Suppose that the cost function $f$ is convex and each local cost $f_i$ is $L_i$-smooth \nfor some $L_i>0$. Assume that $\\mu \\in (0,2/L]$ with $L = \\max_{1 \\leq i \\leq n} L_i$ and the sequence $\\{t(k)\\}_{k \\in \\mathbb{N}\\cup\\{0\\}}$ satisfies $\\sum_{k=0}^\\infty \\beta^{t(k)} <\\infty$. Then, for any minimizer $x_*\\in X^*$ , we have%there exists a constant $R>0$ such that\n\\begin{equation}\\label{eq-1-1}\n\\|{\\bar{\\bf x}}_k -{\\bf x}_*\\|=O(1)\\quad \\textrm{and}\\quad\\|\\bar{\\bold x}_{k+1} -\\bold x_{k+1}\\| = O(\\beta^{t(k)}),\\quad k\\to\\infty,%\\forall~k \\in \\mathbb{N},\n\\end{equation}\nwhere %$Q = \\Big[ (1+2\\mu L) R + \\mu D \\Big]$ and \n$\\bold x_*:=1_n\\otimes x_* =(x_*^{\\top},x_*^{\\top},\\cdots,x_*^{\\top})^{\\top}$. \t\n%\t Then, there exists $\\bar\\mu$ such that if  $\\mu<\\bar\\mu$, then the discrepancy from the optimal solution $\\|{\\bf x}_k-{\\bf x}_*\\|$ is bounded uniformly in $k$.\n\\end{thm}","sha256_utf8":"4af633e42b3f684b339fd87c100915cf4e1ed0be4c4013ac9ed711b92d94a3c7"}]},{"id":"S019","semantic_type":"assumption","kind":"assumption","title":"有界性定理的局部假设","source_units":[{"start":20240,"end":20634,"start_line":316,"end_line":317,"raw_tex":"\\begin{thm}\\label{Main1} Suppose that the cost function $f$ is convex and each local cost $f_i$ is $L_i$-smooth \nfor some $L_i>0$. Assume that $\\mu \\in (0,2/L]$ with $L = \\max_{1 \\leq i \\leq n} L_i$ and the sequence $\\{t(k)\\}_{k \\in \\mathbb{N}\\cup\\{0\\}}$ satisfies $\\sum_{k=0}^\\infty \\beta^{t(k)} <\\infty$. Then, for any minimizer $x_*\\in X^*$ , we have%there exists a constant $R>0$ such that\n","sha256_utf8":"a619c035363a5a737e907c6f5811f256559ec2f8e238a61defc317a11bb8d820"}]},{"id":"S020","semantic_type":"proof","kind":"proof","title":"有界性与共识界完整证明","source_units":[{"start":44810,"end":51790,"start_line":671,"end_line":817,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{Main1}]We define\n\\begin{equation}\\label{eq-3-10}\nA_k:= \\|\\bar{\\bf x}_k-{\\bf x}_*\\| = \\sqrt{n} \\|\\bar{x}_k - x_*\\|,\\quad B_k:=\\|{\\bf x}_k-\\bar{\\bf x}_k\\|\n\\end{equation}\nand proceed to obtain a recursive estimate for $A_k$ and $B_k$. We first estimate $B_{k+1}$ in terms of $A_k$ and $B_k$. For this, we use the algorithm \\eqref{NEAR DGD} along with \\eqref{average} and  \\eqref{11t} to find\n\\begin{equation}\\label{eq-3-30}\n\\begin{split}\n&\\|\\bar{{\\bf x}}_{k+1} - {\\bf x}_{k+1}\\|\\cr\n & = \\|(\\bar{{\\bf x}}_k - \\mu \\overline{\\nabla F}({\\bf x}_k)) - (W^{t(k)}\\otimes I_p) ({\\bf x}_k - \\mu \\nabla F({\\bf x}_k))\\|\n\\\\\n& = \\left\\| \\left[\\left( \\frac{1}{n} 1_n 1_n^\\top  - W^{t(k)}\\right)\\otimes I_p\\right] ({\\bf x}_k -\\mu \\nabla F({\\bf x}_k) \\right\\|\n\\\\\n&\\leq \\left\\| \\left[\\left( \\frac{1}{n} 1_n 1_n^\\top  - W^{t(k)}\\right)\\otimes I_p\\right] {\\bf x}_k \\right\\| + \\mu \\left\\| \\left[\\left( \\frac{1}{n} 1_n 1_n^\\top  - W^{t(k)}\\right)\\otimes I_p\\right] \\nabla F({\\bf x}_k)\\right\\|.\n\\end{split}\n\\end{equation}\nHere we can estimate the right hand side using \n\\begin{equation}\\label{eq-3-31}\n\\left\\{\\left( \\frac{1}{n} 1_n 1_n^\\top  - W^{t(k)}\\right)\\otimes I_p\\right\\} \\bar{\\bf x}_k=0 \n\t\\quad \\textrm{and}\\quad\n\t\\Big\\| \\frac{1}{n} 1_n 1_n^\\top  - W^{t(k)}\\Big\\|_2\\leq \\beta^{t(k)},\n\t\\end{equation}\nwhere the second inequality came from Lemma \\ref{newlemma} and the relation\n\\[\n W^{t(k)}-\\frac{1}{n} 1_n 1_n^\\top  = \\left(W-\\frac{1}{n} 1_n 1_n^\\top \\right)^{t(k)}.\n\\]\nUsing \\eqref{eq-3-31} in \\eqref{eq-3-30} and  the triangle inequality,  we  achieve\n\\begin{align}\\label{consensus estimate}\n\\begin{split}\n\\|\\bar{{\\bf x}}_{k+1} - {\\bf x}_{k+1}\\|&\\leq {}\\beta^{t(k)} \\|{\\bf x}_k - \\bar{\\bf x}_k\\| + {}\\mu \\beta^{t(k)} \\|\\nabla F({\\bf x}_k)\\|\\cr\n &\\leq {}\\beta^{t(k)} \\|{\\bf x}_k - \\bar{\\bf x}_k\\| + {}\\mu \\beta^{t(k)} \n \\left\\{\\|\\nabla F({\\bf x}_k)-\\nabla F(\\bar{\\bf x}_k)\\|\\right.\\cr\n &\\quad\\left.+\\|\\nabla F(\\bar{\\bf x}_k)-\\nabla F({\\bf x}_*)\\|\n +\\|\\nabla F({\\bf x}_*)\\|\n \\right\\}.\\cr\n \\end{split}\n \\end{align}\nWe note that the $L_j$-smoothness of $f_j$ yields that for $x, y \\in \\mathbb{R}^{np}$ we have\n\\begin{equation*}\n\\begin{split}\n\\| \\nabla F (x) - \\nabla F(y) \\| & = \\Bigg( \\sum_{j=1}^n \\|\\nabla f_j (x^j) - \\nabla f_j (y^j) \\|^2 \\Bigg)^{1/2} \n\\\\\n&\\leq \\Bigg(\\sum_{j=1}^n \\textcolor{black}{L_j^2 \\|x^j -y^j\\|^2} \\Bigg)^{1/2} \n\\\\\n& \\leq L \\|x-y\\|,\n\\end{split}\n\\end{equation*}\nwhere we denoted $x= ((x^1)^\\top, \\cdots, (x^n)^\\top)^\\top$ and $y= ((y^1)^\\top, \\cdots, (y^n)^\\top)^\\top$. Using this in \\eqref{consensus estimate} we obtain\n \\begin{equation}\\label{eq-3-32}\n \\begin{split}\n \\|\\bar{{\\bf x}}_{k+1} - {\\bf x}_{k+1}\\| &\\leq {}(1+\\mu L)\\beta^{t(k)} \n \\|{\\bf x}_k-\\bar{\\bf x}_k \\|\n +{}(\\mu L)\\beta^{t(k)}\\|\\bar{\\bf x}_k-{\\bf x}_*\\|\n +{}\\mu\\beta^{t(k)}D\n \\\\\n & ={}(1+\\mu L)\\beta^{t(k)} \n \\|{\\bf x}_k-\\bar{\\bf x}_k \\|\n + {}(\\mu \\sqrt{n} L)\\beta^{t(k)}\\|\\bar{x}_k-{x}_*\\|\n +{}\\mu\\beta^{t(k)}D,\n\\end{split}\n\\end{equation}\nwhere $D:=\\|\\nabla F({\\bf x}_*)\\|$. This gives\n\\begin{align}\\label{estimate A}\nB_{k+1} \\leq  {}(\\mu L)\\beta^{t(k)}A_k+{}(1+\\mu L)\\beta^{t(k)}B_k+{}\\mu\\beta^{t(k)} D. \n\\end{align}\nNext we estimate the error $A_{k+1}$ between the average and the optimizer $x_*$ in terms of $A_k$ and $B_k$. Using \\eqref{average} and the triangle inequality, we find\n\\begin{align}\\label{main comp}\n\\begin{split}\n\\|\\bar{{ x}}_{k+1} - { x}_*\\| \n &= \\bigg\\|\\bar{{x}}_k - {x}_* - \\frac{\\mu}{n} \\sum_{j=1}^n \\nabla f_j (x_{j,k})\\bigg\\|\\\\\n& = \\bigg\\|\\bar{{ x}}_k - {x}_* - \\frac{\\mu}{n} \\sum_{j=1}^n \\nabla f_j (\\bar{x}_k) + \\mu \\bigg( \\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (\\bar{x}_k)- \\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (x_{j,k})\\bigg)\\bigg\\|\\cr\n&\\leq \\|\\bar{{  x}}_k - {  x}_* - \\mu \\nabla f (\\bar{{ x}}_k) \\|+ \\mu \\bigg\\|\\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (\\bar{x}_k)- \\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (x_{j,k})\\bigg\\|.\n\\end{split}\n\\end{align}\nWe make use of $\\nabla f(x_*)=0$ to compute\n\\begin{equation}\\label{eq-3-1}\n\\begin{split}\n&\\|\\bar{{x}}_k- {x}_*-\\mu{\\nabla f}(\\bar{{x}}_k)\\|^2\\\\\n&\\quad= \\|\\bar{{ x}}_k-{ x}_*\\|^2\n-2\\mu\\langle  \\bar{{x}}_k-{x}_*,\\nabla f(\\bar{{ x}}_k)\\rangle\n+\\mu^2\\|\\nabla f(\\bar{{ x}}_k)\\|^2\\\\\n&\\quad=\\|\\bar{{x}}_k-{x}_*\\|^2\n-2\\mu\\langle  \\bar{{x}}_k-{x}_*, {\\nabla f}(\\bar{{x}}_k)\n- {\\nabla f}({{x}}_*)\\rangle\n+\\mu^2\\|{\\nabla f}(\\bar{{x}}_k)-{\\nabla f}({x}_*)\\|^2.\n\\end{split}\n\\end{equation}\nFrom Lemma \\ref{lem-2-1}, we find\n\\begin{equation}\\label{eq-3-2}\n\\langle \\bar{x}_k -{x}_*,  {\\nabla f}(\\bar{x}_k)- {\\nabla f}({ x}_*)\\rangle \\geq\\frac{1}{L}\\| \\textcolor{black}{{\\nabla f}(\\bar{ x}_k)}-{\\nabla f}({x}_*)\\|^2,\n\\end{equation}\nand apply it to \\eqref{eq-3-1} to obtain\n\\begin{align*}\n\\begin{split}\n& \\|\\bar{{x}}_k-{x}_*-\\mu{\\nabla f}(\\bar{{x}}_k)\\|^2\\\\\n&\\quad\\leq\\|\\bar{{x}}_k-{x}_*\\|^2+\\mu\\left(\\mu-\\frac{2}{L}\\right)\\|{\\nabla f}(\\bar{{x}}_k)-{\\nabla f}({{x}}_*)\\|^2\\cr\n&\\quad\\leq \\|\\bar{{x}}_k-{x}_*\\|^2\n\\end{split}\n\\end{align*}\nfor $\\mu \\in (0, 2/L]$.\nThis gives us the estimate\n\\begin{align}\\label{eq-3-6}\n&\\|\\bar{{x}}_k-{x}_*-\\mu {\\nabla f}(\\bar{{ x}}_k)\\|\n\\leq \\|\\bar{{x}}_k-{x}_*\\|.\n\\end{align}\nOn the other hand, we use the $L_j$-smoothness of $f_j$ and the triangle inequality to deduce\n\\begin{align}\\label{insert1}\n\\begin{split} \n \\bigg\\|\\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (\\bar{x}_k)- \\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (x_{j,k})\\bigg\\| & \\leq \\frac{1}{n} \\sum_{j=1}^n L_j \\|\\bar{x}_k - x_{j,k}\\|\n \\\\\n&\\leq \\frac{L}{\\sqrt{n}} \\|\\bar{{\\bf x}}_k - {\\bf x}_k\\|.\n\\end{split}\n\\end{align}\nCombining \\eqref{main comp},  \\eqref{eq-3-6} and \\eqref{insert1}, we get\n\\begin{align}\\label{line for A}\n\\begin{split}\n\\|\\bar{{x}}_{k+1}- {{x}}_*\\|\n\\leq \\|\\bar{{x}}_k- {{x}}_*\\|+ \\frac{\\mu L}{\\sqrt{n}}\\|{\\bf x}_k-\\bar{{\\bf x}}_k\\|.\n\\end{split}\n\\end{align}\nFrom \\eqref{estimate A} and \\eqref{line for A}, we get the following iterative relation:\n\\begin{equation}\\label{eq-3-50}\n\\left(\\begin{array}{c} A_{k+1}\\cr B_{k+1}\\end{array}\\right) \\leq\n\\left(\\begin{array}{cc} 1 &  \\mu L\\cr {}(\\mu L) \\beta^{t(k)}& {} (1+\\mu L)\\beta^{t(k)}\\end{array}\\right)\n\\left(\\begin{array}{c} A_{k}\\cr B_{k}\\end{array}\\right)\n+\n{}\\beta^{t(k)}\\left(\\begin{array}{c} 0\\cr \\mu D\\end{array}\\right),\n\\end{equation}\n\\textcolor{black}{\nwhich clearly implies\n\\begin{align*}\n\\left(\\begin{array}{c} A_{k+1}\\cr B_{k+1}\\end{array}\\right) \\leq\n\\left(\\begin{array}{cc} 1 &  \\mu L\\cr {}(1+\\mu L) \\beta^{t(k)}& {} (1+\\mu L)\\beta^{t(k)}\\end{array}\\right)\n\\left(\\begin{array}{c} A_{k}\\cr B_{k}\\end{array}\\right)\n+\n{}\\beta^{t(k)}\\left(\\begin{array}{c} 0\\cr \\mu D\\end{array}\\right). \n\\end{align*}\n}\nNow we apply Proposition \\ref{prop-2-2} to conclude that there exists $R>0$ such that\n\\begin{equation*}\n\\|\\bar{\\bf x}_k -\\bold x_*\\| \\leq R \\quad \\textrm{and}\\quad \\|\\bar{\\bold x}_k -\\bold x_k \\| \\leq R\n\\end{equation*}\nfor all $k \\in \\mathbb{N}$. By applying this bound to \\eqref{estimate A}, we obtain\n\\begin{equation*}\n\\|\\bar{\\bold x}_{k+1} -\\bold x_{k+1}\\| \\leq {}\\beta^{t(k)} \\Big[ (1+2\\mu L) R + \\mu D \\Big].\n\\end{equation*}\nThis completes the proof.\n\\end{proof}","sha256_utf8":"2751190c6952208a70be9939692eecd4368be1438249d28d29fd2d570c23edca"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"定义两类误差序列","source_units":[{"start":44810,"end":45235,"start_line":671,"end_line":675,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{Main1}]We define\n\\begin{equation}\\label{eq-3-10}\nA_k:= \\|\\bar{\\bf x}_k-{\\bf x}_*\\| = \\sqrt{n} \\|\\bar{x}_k - x_*\\|,\\quad B_k:=\\|{\\bf x}_k-\\bar{\\bf x}_k\\|\n\\end{equation}\nand proceed to obtain a recursive estimate for $A_k$ and $B_k$. We first estimate $B_{k+1}$ in terms of $A_k$ and $B_k$. For this, we use the algorithm \\eqref{NEAR DGD} along with \\eqref{average} and  \\eqref{11t} to find\n","sha256_utf8":"b8d9b86233bbcbe8167a690e5a83493083c3665d98612680a7ddea979200afea"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"共识投影与矩阵幂界","source_units":[{"start":45235,"end":46297,"start_line":676,"end_line":695,"raw_tex":"\\begin{equation}\\label{eq-3-30}\n\\begin{split}\n&\\|\\bar{{\\bf x}}_{k+1} - {\\bf x}_{k+1}\\|\\cr\n & = \\|(\\bar{{\\bf x}}_k - \\mu \\overline{\\nabla F}({\\bf x}_k)) - (W^{t(k)}\\otimes I_p) ({\\bf x}_k - \\mu \\nabla F({\\bf x}_k))\\|\n\\\\\n& = \\left\\| \\left[\\left( \\frac{1}{n} 1_n 1_n^\\top  - W^{t(k)}\\right)\\otimes I_p\\right] ({\\bf x}_k -\\mu \\nabla F({\\bf x}_k) \\right\\|\n\\\\\n&\\leq \\left\\| \\left[\\left( \\frac{1}{n} 1_n 1_n^\\top  - W^{t(k)}\\right)\\otimes I_p\\right] {\\bf x}_k \\right\\| + \\mu \\left\\| \\left[\\left( \\frac{1}{n} 1_n 1_n^\\top  - W^{t(k)}\\right)\\otimes I_p\\right] \\nabla F({\\bf x}_k)\\right\\|.\n\\end{split}\n\\end{equation}\nHere we can estimate the right hand side using \n\\begin{equation}\\label{eq-3-31}\n\\left\\{\\left( \\frac{1}{n} 1_n 1_n^\\top  - W^{t(k)}\\right)\\otimes I_p\\right\\} \\bar{\\bf x}_k=0 \n\t\\quad \\textrm{and}\\quad\n\t\\Big\\| \\frac{1}{n} 1_n 1_n^\\top  - W^{t(k)}\\Big\\|_2\\leq \\beta^{t(k)},\n\t\\end{equation}\nwhere the second inequality came from Lemma \\ref{newlemma} and the relation\n\\[\n W^{t(k)}-\\frac{1}{n} 1_n 1_n^\\top  = \\left(W-\\frac{1}{n} 1_n 1_n^\\top \\right)^{t(k)}.\n\\]\n","sha256_utf8":"1a5106486e6b2a6918ebd5d19e18ed562f8417333a0825129603e8fa18b21b18"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"分解梯度引起的共识误差","source_units":[{"start":46297,"end":46841,"start_line":696,"end_line":706,"raw_tex":"Using \\eqref{eq-3-31} in \\eqref{eq-3-30} and  the triangle inequality,  we  achieve\n\\begin{align}\\label{consensus estimate}\n\\begin{split}\n\\|\\bar{{\\bf x}}_{k+1} - {\\bf x}_{k+1}\\|&\\leq {}\\beta^{t(k)} \\|{\\bf x}_k - \\bar{\\bf x}_k\\| + {}\\mu \\beta^{t(k)} \\|\\nabla F({\\bf x}_k)\\|\\cr\n &\\leq {}\\beta^{t(k)} \\|{\\bf x}_k - \\bar{\\bf x}_k\\| + {}\\mu \\beta^{t(k)} \n \\left\\{\\|\\nabla F({\\bf x}_k)-\\nabla F(\\bar{\\bf x}_k)\\|\\right.\\cr\n &\\quad\\left.+\\|\\nabla F(\\bar{\\bf x}_k)-\\nabla F({\\bf x}_*)\\|\n +\\|\\nabla F({\\bf x}_*)\\|\n \\right\\}.\\cr\n \\end{split}\n \\end{align}\n","sha256_utf8":"aed4a14afcc0db5a39d54afdd9d20864c11f20b8bcba225b108df10c8205a5f4"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"堆叠梯度的光滑性","source_units":[{"start":46841,"end":47371,"start_line":707,"end_line":717,"raw_tex":"We note that the $L_j$-smoothness of $f_j$ yields that for $x, y \\in \\mathbb{R}^{np}$ we have\n\\begin{equation*}\n\\begin{split}\n\\| \\nabla F (x) - \\nabla F(y) \\| & = \\Bigg( \\sum_{j=1}^n \\|\\nabla f_j (x^j) - \\nabla f_j (y^j) \\|^2 \\Bigg)^{1/2} \n\\\\\n&\\leq \\Bigg(\\sum_{j=1}^n \\textcolor{black}{L_j^2 \\|x^j -y^j\\|^2} \\Bigg)^{1/2} \n\\\\\n& \\leq L \\|x-y\\|,\n\\end{split}\n\\end{equation*}\nwhere we denoted $x= ((x^1)^\\top, \\cdots, (x^n)^\\top)^\\top$ and $y= ((y^1)^\\top, \\cdots, (y^n)^\\top)^\\top$. Using this in \\eqref{consensus estimate} we obtain\n","sha256_utf8":"02121998bb57cc751bea9ec88c373cef6ffe8d93b0a66bc02412b4ef617e6935"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"建立共识误差递推","source_units":[{"start":47371,"end":47937,"start_line":718,"end_line":734,"raw_tex":" \\begin{equation}\\label{eq-3-32}\n \\begin{split}\n \\|\\bar{{\\bf x}}_{k+1} - {\\bf x}_{k+1}\\| &\\leq {}(1+\\mu L)\\beta^{t(k)} \n \\|{\\bf x}_k-\\bar{\\bf x}_k \\|\n +{}(\\mu L)\\beta^{t(k)}\\|\\bar{\\bf x}_k-{\\bf x}_*\\|\n +{}\\mu\\beta^{t(k)}D\n \\\\\n & ={}(1+\\mu L)\\beta^{t(k)} \n \\|{\\bf x}_k-\\bar{\\bf x}_k \\|\n + {}(\\mu \\sqrt{n} L)\\beta^{t(k)}\\|\\bar{x}_k-{x}_*\\|\n +{}\\mu\\beta^{t(k)}D,\n\\end{split}\n\\end{equation}\nwhere $D:=\\|\\nabla F({\\bf x}_*)\\|$. This gives\n\\begin{align}\\label{estimate A}\nB_{k+1} \\leq  {}(\\mu L)\\beta^{t(k)}A_k+{}(1+\\mu L)\\beta^{t(k)}B_k+{}\\mu\\beta^{t(k)} D. \n\\end{align}\n","sha256_utf8":"d3ff94b08c3620ee7d56129a3c9bbf62dda8c6f361b1b3d131900baac02681e5"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"拆分平均迭代与梯度偏差","source_units":[{"start":47937,"end":48684,"start_line":735,"end_line":743,"raw_tex":"Next we estimate the error $A_{k+1}$ between the average and the optimizer $x_*$ in terms of $A_k$ and $B_k$. Using \\eqref{average} and the triangle inequality, we find\n\\begin{align}\\label{main comp}\n\\begin{split}\n\\|\\bar{{ x}}_{k+1} - { x}_*\\| \n &= \\bigg\\|\\bar{{x}}_k - {x}_* - \\frac{\\mu}{n} \\sum_{j=1}^n \\nabla f_j (x_{j,k})\\bigg\\|\\\\\n& = \\bigg\\|\\bar{{ x}}_k - {x}_* - \\frac{\\mu}{n} \\sum_{j=1}^n \\nabla f_j (\\bar{x}_k) + \\mu \\bigg( \\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (\\bar{x}_k)- \\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (x_{j,k})\\bigg)\\bigg\\|\\cr\n&\\leq \\|\\bar{{  x}}_k - {  x}_* - \\mu \\nabla f (\\bar{{ x}}_k) \\|+ \\mu \\bigg\\|\\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (\\bar{x}_k)- \\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (x_{j,k})\\bigg\\|.\n\\end{split}\n\\end{align}\n","sha256_utf8":"318cd2fb7973e6ca7ac19181da1c27a7de4276322e0935175f50ff5e824232a0"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"展开理想梯度步的误差","source_units":[{"start":48684,"end":49160,"start_line":744,"end_line":756,"raw_tex":"We make use of $\\nabla f(x_*)=0$ to compute\n\\begin{equation}\\label{eq-3-1}\n\\begin{split}\n&\\|\\bar{{x}}_k- {x}_*-\\mu{\\nabla f}(\\bar{{x}}_k)\\|^2\\\\\n&\\quad= \\|\\bar{{ x}}_k-{ x}_*\\|^2\n-2\\mu\\langle  \\bar{{x}}_k-{x}_*,\\nabla f(\\bar{{ x}}_k)\\rangle\n+\\mu^2\\|\\nabla f(\\bar{{ x}}_k)\\|^2\\\\\n&\\quad=\\|\\bar{{x}}_k-{x}_*\\|^2\n-2\\mu\\langle  \\bar{{x}}_k-{x}_*, {\\nabla f}(\\bar{{x}}_k)\n- {\\nabla f}({{x}}_*)\\rangle\n+\\mu^2\\|{\\nabla f}(\\bar{{x}}_k)-{\\nabla f}({x}_*)\\|^2.\n\\end{split}\n\\end{equation}\n","sha256_utf8":"44149986fa763ced47294a60bbcf4164d8f4d057699364ba700c44613a8c2d4d"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"余强单调性给出非扩张性","source_units":[{"start":49160,"end":49873,"start_line":757,"end_line":774,"raw_tex":"From Lemma \\ref{lem-2-1}, we find\n\\begin{equation}\\label{eq-3-2}\n\\langle \\bar{x}_k -{x}_*,  {\\nabla f}(\\bar{x}_k)- {\\nabla f}({ x}_*)\\rangle \\geq\\frac{1}{L}\\| \\textcolor{black}{{\\nabla f}(\\bar{ x}_k)}-{\\nabla f}({x}_*)\\|^2,\n\\end{equation}\nand apply it to \\eqref{eq-3-1} to obtain\n\\begin{align*}\n\\begin{split}\n& \\|\\bar{{x}}_k-{x}_*-\\mu{\\nabla f}(\\bar{{x}}_k)\\|^2\\\\\n&\\quad\\leq\\|\\bar{{x}}_k-{x}_*\\|^2+\\mu\\left(\\mu-\\frac{2}{L}\\right)\\|{\\nabla f}(\\bar{{x}}_k)-{\\nabla f}({{x}}_*)\\|^2\\cr\n&\\quad\\leq \\|\\bar{{x}}_k-{x}_*\\|^2\n\\end{split}\n\\end{align*}\nfor $\\mu \\in (0, 2/L]$.\nThis gives us the estimate\n\\begin{align}\\label{eq-3-6}\n&\\|\\bar{{x}}_k-{x}_*-\\mu {\\nabla f}(\\bar{{ x}}_k)\\|\n\\leq \\|\\bar{{x}}_k-{x}_*\\|.\n\\end{align}\n","sha256_utf8":"409085be09a9d341eab15ac773eb1b2e32238c8b79bd3c9972963abfec6508a1"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"共识误差控制平均梯度偏差","source_units":[{"start":49873,"end":50267,"start_line":775,"end_line":782,"raw_tex":"On the other hand, we use the $L_j$-smoothness of $f_j$ and the triangle inequality to deduce\n\\begin{align}\\label{insert1}\n\\begin{split} \n \\bigg\\|\\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (\\bar{x}_k)- \\frac{1}{n} \\sum_{j=1}^n \\nabla f_j (x_{j,k})\\bigg\\| & \\leq \\frac{1}{n} \\sum_{j=1}^n L_j \\|\\bar{x}_k - x_{j,k}\\|\n \\\\\n&\\leq \\frac{L}{\\sqrt{n}} \\|\\bar{{\\bf x}}_k - {\\bf x}_k\\|.\n\\end{split}\n\\end{align}\n","sha256_utf8":"eeec4eefef171eef7766a321f017956f69343cd8edac7e6d95e9cc8cfd17879f"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"建立平均误差递推","source_units":[{"start":50267,"end":50523,"start_line":783,"end_line":789,"raw_tex":"Combining \\eqref{main comp},  \\eqref{eq-3-6} and \\eqref{insert1}, we get\n\\begin{align}\\label{line for A}\n\\begin{split}\n\\|\\bar{{x}}_{k+1}- {{x}}_*\\|\n\\leq \\|\\bar{{x}}_k- {{x}}_*\\|+ \\frac{\\mu L}{\\sqrt{n}}\\|{\\bf x}_k-\\bar{{\\bf x}}_k\\|.\n\\end{split}\n\\end{align}\n","sha256_utf8":"5571607f1934a681693df79569a33b6fe8e47ecab3029b5f23bbc0181f3bf100"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"构造二维矩阵递推","source_units":[{"start":50523,"end":51323,"start_line":790,"end_line":807,"raw_tex":"From \\eqref{estimate A} and \\eqref{line for A}, we get the following iterative relation:\n\\begin{equation}\\label{eq-3-50}\n\\left(\\begin{array}{c} A_{k+1}\\cr B_{k+1}\\end{array}\\right) \\leq\n\\left(\\begin{array}{cc} 1 &  \\mu L\\cr {}(\\mu L) \\beta^{t(k)}& {} (1+\\mu L)\\beta^{t(k)}\\end{array}\\right)\n\\left(\\begin{array}{c} A_{k}\\cr B_{k}\\end{array}\\right)\n+\n{}\\beta^{t(k)}\\left(\\begin{array}{c} 0\\cr \\mu D\\end{array}\\right),\n\\end{equation}\n\\textcolor{black}{\nwhich clearly implies\n\\begin{align*}\n\\left(\\begin{array}{c} A_{k+1}\\cr B_{k+1}\\end{array}\\right) \\leq\n\\left(\\begin{array}{cc} 1 &  \\mu L\\cr {}(1+\\mu L) \\beta^{t(k)}& {} (1+\\mu L)\\beta^{t(k)}\\end{array}\\right)\n\\left(\\begin{array}{c} A_{k}\\cr B_{k}\\end{array}\\right)\n+\n{}\\beta^{t(k)}\\left(\\begin{array}{c} 0\\cr \\mu D\\end{array}\\right). \n\\end{align*}\n}\n","sha256_utf8":"df480b3214bafdc12365ab2f25eee30145001b722adacc5f04973961bd8167f5"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"矩阵乘积界推出有界性","source_units":[{"start":51323,"end":51790,"start_line":808,"end_line":817,"raw_tex":"Now we apply Proposition \\ref{prop-2-2} to conclude that there exists $R>0$ such that\n\\begin{equation*}\n\\|\\bar{\\bf x}_k -\\bold x_*\\| \\leq R \\quad \\textrm{and}\\quad \\|\\bar{\\bold x}_k -\\bold x_k \\| \\leq R\n\\end{equation*}\nfor all $k \\in \\mathbb{N}$. By applying this bound to \\eqref{estimate A}, we obtain\n\\begin{equation*}\n\\|\\bar{\\bold x}_{k+1} -\\bold x_{k+1}\\| \\leq {}\\beta^{t(k)} \\Big[ (1+2\\mu L) R + \\mu D \\Big].\n\\end{equation*}\nThis completes the proof.\n\\end{proof}","sha256_utf8":"e3e3ccd910f4af23d0025df65b1f45fa284f6119ae8300a9b818d8858bb5c097"}]},{"id":"S033","semantic_type":"proposition","kind":"proposition","title":"可和扰动下的矩阵递推界","source_units":[{"start":38507,"end":39527,"start_line":560,"end_line":575,"raw_tex":"\\begin{prop}\\label{prop-2-2} \nLet  $\\{X_k\\}_{k\\geq0}$ and $\\{Y_k\\}_{k\\geq0}$ be sequences of non-negative vectors in $\\mathbb{R}^2$ and $\\{M_k\\}_{k\\geq0}$ be a sequence of $2\\times 2$ matrices. Assume that $\\{X_k\\}_{k\\geq0}$, $\\{Y_k\\}_{k\\geq0}$ and $\\{M_k\\}_{k\\geq0}$ satisfy the following iterative relation:\n\t\\begin{equation*}\n\tX_{k+1} \\leq M_k X_k + Y_k.\n\t\\end{equation*}\n\tSuppose further that $\\{Y_k\\}_{k \\geq0}$ is a sequence of non-negative vectors such that $\\sum_{k=0}^{\\infty}\\big\\| Y_k \\big\\|< \\infty$, and $\\{M_k\\}_{k\\geq0}$ takes the following form:\n\t \\begin{equation*}\n\t M_k = \\begin{pmatrix} 1 & r \\\\ \\alpha_k & \\alpha_k \\end{pmatrix}\n\t \\end{equation*}\n\t for some fixed $r>0$ and a sequence of non-negative real numbers $\\{\\alpha_k\\}_{k\\geq0}$ satisfying  $\\sum_{k=0}^{\\infty} \\alpha_k <\\infty$. \n Then, we have\n \\begin{enumerate}\n \t \t\\item The matrix $M_b M_{b-1}\\dots M_a$   is uniformly bounded for $0 \\leq a < b< \\infty$.\n \t\\item  $\\{X_{k}\\}_{k\\geq0}$ is uniformly bounded.\n \t\\end{enumerate}\n\\end{prop}","sha256_utf8":"0f5fef443be4c5fdaad76e8ef01287dc7ec74fbc745a46b67a5e7de2aa5826a7"}]},{"id":"S034","semantic_type":"assumption","kind":"assumption","title":"非负递推与可和参数","source_units":[{"start":38537,"end":39318,"start_line":561,"end_line":569,"raw_tex":"Let  $\\{X_k\\}_{k\\geq0}$ and $\\{Y_k\\}_{k\\geq0}$ be sequences of non-negative vectors in $\\mathbb{R}^2$ and $\\{M_k\\}_{k\\geq0}$ be a sequence of $2\\times 2$ matrices. Assume that $\\{X_k\\}_{k\\geq0}$, $\\{Y_k\\}_{k\\geq0}$ and $\\{M_k\\}_{k\\geq0}$ satisfy the following iterative relation:\n\t\\begin{equation*}\n\tX_{k+1} \\leq M_k X_k + Y_k.\n\t\\end{equation*}\n\tSuppose further that $\\{Y_k\\}_{k \\geq0}$ is a sequence of non-negative vectors such that $\\sum_{k=0}^{\\infty}\\big\\| Y_k \\big\\|< \\infty$, and $\\{M_k\\}_{k\\geq0}$ takes the following form:\n\t \\begin{equation*}\n\t M_k = \\begin{pmatrix} 1 & r \\\\ \\alpha_k & \\alpha_k \\end{pmatrix}\n\t \\end{equation*}\n\t for some fixed $r>0$ and a sequence of non-negative real numbers $\\{\\alpha_k\\}_{k\\geq0}$ satisfying  $\\sum_{k=0}^{\\infty} \\alpha_k <\\infty$. \n","sha256_utf8":"d37c837ae97b93827e449d7681eafb8a4a56710729df177e86a7fd867778dce8"}]},{"id":"S035","semantic_type":"proof","kind":"proof","title":"矩阵递推界完整证明","source_units":[{"start":39530,"end":41725,"start_line":577,"end_line":627,"raw_tex":"\\begin{proof}\n(1) We take $c = \\max\\{r,1\\}$ and define a weighted norm $\\|\\cdot\\|_{c}$ in $\\mathbb{R}^2$ by\n\\begin{equation*}\n\\|(x,y)^\\top\\|_c = |x| + c|y| \\quad \\textcolor{black}{\\mbox{for }x,y\\in\\mathbb{R}}.\n\\end{equation*}\nFor a matrix $M \\in \\mathbb{R}^{2\\times 2}$, we define the  operator norm by\n\\begin{equation*}\n\\|M\\|_{\\mathcal{L}} =\\sup_{v \\in \\mathbb{R}^2 \\setminus \\{0\\}} \\frac{\\|Mv\\|_{c}}{\\|v\\|_{c}}.\n\\end{equation*}\nWe claim that\n\\begin{equation*}\\label{eq-2-3}\n\\|M_k\\|_{\\mathcal{L}} \\leq 1+ c\\alpha_k.\n\\end{equation*}\nTo show this, we take an arbitrary $v = (x,y)^\\top \\in \\mathbb{R}^2$ and compute\n\\begin{equation*}\nM_kv=\\begin{pmatrix} 1 & r \\\\ \\alpha_k &\\alpha_k \\end{pmatrix} \\begin{pmatrix} x \\\\ y \\end{pmatrix} = \\begin{pmatrix} x + ry \\\\ \\alpha_k x + \\alpha_k y \\end{pmatrix},\n\\end{equation*}\nwhich has the weighted $L^1$ norm as\n\\begin{equation*}\n\\|M_kv\\|_{c} = |x+ry| + c |\\alpha_k x + \\alpha_k y| \\leq (1+c\\alpha_k)|x| + (r+c \\alpha_k )|y|.\n\\end{equation*}\nSince $c = \\max\\{r,1\\}$ we have\n\\begin{equation*}\nr+c \\alpha_k \\leq c (1+c\\alpha_k)\n\\end{equation*}\nso that\n\\begin{equation*}\n\\begin{split}\n\\|M_kv\\|_c&\\leq(1+c\\alpha_k) |x| + (r+c \\alpha_k ) |y|\n\\\\\n&\\leq (1+c\\alpha_k) (|x|+c|y|) = (1+c\\alpha_k) \\|v\\|_c.\n\\end{split}\n\\end{equation*}\nThis proves the claim. Using \\eqref{eq-2-3}, for any $1\\leq a<b<\\infty$ we  obtain\n\\begin{equation*}\n\\begin{split}\n\\Big\\| M_b M_{b-1}\\dots M_a \\Big\\|_{\\mathcal{L}}& \\leq \\prod_{k=a}^b \\|M_k \\|_{\\mathcal{L}}\n\\\\\n& \\leq \\prod_{k=a}^b (1+c\\alpha_k)\n\\\\\n& \\leq \\prod_{k=a}^b e^{c\\alpha_k} = \\exp \\Big(c\\sum_{k=a}^b \\alpha_k\\Big).\n\\end{split}\n\\end{equation*}\nCombining this with the assumption that $\\sum_{k=0}^{\\infty} \\alpha_k < \\infty$, we conclude that $\\Big\\| M_b M_{b-1}\\dots M_a \\Big\\|_{\\mathcal{L}} $ is uniformly bounded for $0\\leq a<b <\\infty$.\n\n\\noindent (2) By induction, the iteration implies that\n\\begin{equation}\\label{eq-2-10}\nX_{k+1} \\leq \\Big( M_k M_{k-1}\\dots M_0\\Big)X_0 + \\sum_{j=0}^{k-1} \\Big( M_k M_{k-1}\\dots M_{j+1}\\Big) Y_j + Y_k.\n\\end{equation}\nThis gives the desired uniform boundedness of $X_k$ since $M_b M_{b-1}\\dots M_a$ is uniformly bounded and $\\sum_{j=0}^{\\infty} \\big\\|Y_j \\big\\|< \\infty$. \n\\end{proof}","sha256_utf8":"0b4c2d6de61ec4928dc84db57208525d4da14621e57ee3319d27e851702e4efc"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"引入加权范数","source_units":[{"start":39530,"end":40063,"start_line":577,"end_line":589,"raw_tex":"\\begin{proof}\n(1) We take $c = \\max\\{r,1\\}$ and define a weighted norm $\\|\\cdot\\|_{c}$ in $\\mathbb{R}^2$ by\n\\begin{equation*}\n\\|(x,y)^\\top\\|_c = |x| + c|y| \\quad \\textcolor{black}{\\mbox{for }x,y\\in\\mathbb{R}}.\n\\end{equation*}\nFor a matrix $M \\in \\mathbb{R}^{2\\times 2}$, we define the  operator norm by\n\\begin{equation*}\n\\|M\\|_{\\mathcal{L}} =\\sup_{v \\in \\mathbb{R}^2 \\setminus \\{0\\}} \\frac{\\|Mv\\|_{c}}{\\|v\\|_{c}}.\n\\end{equation*}\nWe claim that\n\\begin{equation*}\\label{eq-2-3}\n\\|M_k\\|_{\\mathcal{L}} \\leq 1+ c\\alpha_k.\n\\end{equation*}\n","sha256_utf8":"b2cfe2496a8a5e708495b250eaa1b9e3416e4d8264a1ca5e7ff80c8f8c148e0e"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"估计单步矩阵范数","source_units":[{"start":40063,"end":40794,"start_line":590,"end_line":609,"raw_tex":"To show this, we take an arbitrary $v = (x,y)^\\top \\in \\mathbb{R}^2$ and compute\n\\begin{equation*}\nM_kv=\\begin{pmatrix} 1 & r \\\\ \\alpha_k &\\alpha_k \\end{pmatrix} \\begin{pmatrix} x \\\\ y \\end{pmatrix} = \\begin{pmatrix} x + ry \\\\ \\alpha_k x + \\alpha_k y \\end{pmatrix},\n\\end{equation*}\nwhich has the weighted $L^1$ norm as\n\\begin{equation*}\n\\|M_kv\\|_{c} = |x+ry| + c |\\alpha_k x + \\alpha_k y| \\leq (1+c\\alpha_k)|x| + (r+c \\alpha_k )|y|.\n\\end{equation*}\nSince $c = \\max\\{r,1\\}$ we have\n\\begin{equation*}\nr+c \\alpha_k \\leq c (1+c\\alpha_k)\n\\end{equation*}\nso that\n\\begin{equation*}\n\\begin{split}\n\\|M_kv\\|_c&\\leq(1+c\\alpha_k) |x| + (r+c \\alpha_k ) |y|\n\\\\\n&\\leq (1+c\\alpha_k) (|x|+c|y|) = (1+c\\alpha_k) \\|v\\|_c.\n\\end{split}\n\\end{equation*}\n","sha256_utf8":"550f94aad903bea224b1f40505ca8652499964ab656aafb563f5bc8ff8d31053"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"可和性控制矩阵乘积","source_units":[{"start":40794,"end":41342,"start_line":610,"end_line":620,"raw_tex":"This proves the claim. Using \\eqref{eq-2-3}, for any $1\\leq a<b<\\infty$ we  obtain\n\\begin{equation*}\n\\begin{split}\n\\Big\\| M_b M_{b-1}\\dots M_a \\Big\\|_{\\mathcal{L}}& \\leq \\prod_{k=a}^b \\|M_k \\|_{\\mathcal{L}}\n\\\\\n& \\leq \\prod_{k=a}^b (1+c\\alpha_k)\n\\\\\n& \\leq \\prod_{k=a}^b e^{c\\alpha_k} = \\exp \\Big(c\\sum_{k=a}^b \\alpha_k\\Big).\n\\end{split}\n\\end{equation*}\nCombining this with the assumption that $\\sum_{k=0}^{\\infty} \\alpha_k < \\infty$, we conclude that $\\Big\\| M_b M_{b-1}\\dots M_a \\Big\\|_{\\mathcal{L}} $ is uniformly bounded for $0\\leq a<b <\\infty$.\n","sha256_utf8":"06645f25db3bcfb636b894c0dded00a5a2edf733e62bfc4a1fc76b87aae07f1b"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"展开非齐次递推","source_units":[{"start":41343,"end":41725,"start_line":622,"end_line":627,"raw_tex":"\\noindent (2) By induction, the iteration implies that\n\\begin{equation}\\label{eq-2-10}\nX_{k+1} \\leq \\Big( M_k M_{k-1}\\dots M_0\\Big)X_0 + \\sum_{j=0}^{k-1} \\Big( M_k M_{k-1}\\dots M_{j+1}\\Big) Y_j + Y_k.\n\\end{equation}\nThis gives the desired uniform boundedness of $X_k$ since $M_b M_{b-1}\\dots M_a$ is uniformly bounded and $\\sum_{j=0}^{\\infty} \\big\\|Y_j \\big\\|< \\infty$. \n\\end{proof}","sha256_utf8":"098fdc3701bdda44ad84dd418496f57908e17e5d105f7bd17850e6e78380efdd"}]},{"id":"S040","semantic_type":"lemma","kind":"lemma","title":"共识矩阵的严格谱收缩","source_units":[{"start":42005,"end":42772,"start_line":630,"end_line":642,"raw_tex":"\\begin{lem}\\label{newlemma}\\cite{QL}\nAssume that the consensus matrix $W$ satisfies the following:\n\\begin{itemize}\n\\item $W$ is a doubly-stochastic $n\\times n$ matrix.\n\\item The directed graph associated to $W$ is strongly connected, and has a self-loop at every vertex.\n\\end{itemize}\nThen the following assertions hold:\n\\begin{enumerate}\n\\item The graph associated to $W^\\top W$ is connected and aperiodic. (Since $W^\\top W$ is symmetric, we may either interpret its associated graph as a strongly connected directed graph, or a connected undirected graph, depending on preference.)\n\\item The largest eigenvalue of $W^\\top W$ is $1$, and it is simple.\n\\item The spectral norm $\\beta$ of $W-\\frac{1}{n}1_n1_n^\\top$ satisfies $\\beta\\in[0,1)$.\n\\end{enumerate}\n\\end{lem}","sha256_utf8":"394787773a0f5e08ae41a38143b1d813201b02c3143512183c4962f35d73167e"}]},{"id":"S041","semantic_type":"proof","kind":"proof","title":"谱收缩引理完整证明","source_units":[{"start":42773,"end":44608,"start_line":643,"end_line":666,"raw_tex":"\\begin{proof}\n(1) The assumption that the graph associated to $W$ has a self-loop at every vertex, is equivalent to $w_{ii}>0$ for all $1\\leq i\\leq n$. Let $w_0:=\\frac{1}{2}\\min\\limits_{1\\leq i\\leq n}w_{ii}>0$ and $\\widetilde W:=W-w_0 I_n$. Then all entries of $\\widetilde W$ are nonnegative, and the directed graph associated to $\\widetilde W$ is precisely the same as that of $W$. Note that\n\\[\nW^\\top W=(\\widetilde W+w_0I_n)^\\top(\\widetilde W+w_0I_n)\\geq w_0 \\widetilde W,\n\\]\nwith the inequality being in entrywise sense. Hence the graph associated to $W^\\top W$ contains that associated to $\\widetilde W$ (or $W$) as its subgraph, which is indeed strongly connected and aperiodc.\n\n(2) Since $W^\\top W$ is symmetric positive semidefinite, we can arrange its eigenvalues as $\\lambda_1\\geq \\lambda_2\\geq\\dots\\geq\\lambda_n\\geq0$. The Perron-Frobenius theorem combined with (1) yields $1=\\lambda_1>\\lambda_2$.\n\n(3) Since $W^\\top W$ is symmetric, it can be diagonalized by an orthogonal matrix, i.e.,\n\\[\nW^\\top W=Q \\operatorname{diag}(\\lambda_1,\\dots,\\lambda_n)Q^\\top,\\quad Q:=[u_1|\\dots|u_n],\n\\]\nwith $u_i$ being an eigenvector associated to $\\lambda_i$, $i=1,\\dots,n$. Note that $\\lambda_1=1$ and $u_1=\\frac{1}{\\sqrt{n}}1_n$, because $W$ is doubly-stochastic. Hence\n\\[\nW^\\top W=\\sum_{i=1}^n \\lambda_i u_iu_i^\\top=\\frac{1}{n}1_n1_n^\\top+\\sum_{i=2}^n \\lambda_i u_iu_i^\\top,\n\\]\nwhich yields\n\\begin{align*}\n\\left\\|W-\\frac{1}{n}1_n1_n^\\top\\right\\|&=\\sqrt{\\mbox{the largest eigenvalue of }(W-\\frac{1}{n}1_n1_n^\\top)^\\top(W-\\frac{1}{n}1_n1_n^\\top)}\\\\\n&=\\sqrt{\\mbox{the largest eigenvalue of }W^\\top W-\\frac{1}{n}1_n1_n^\\top}=\\sqrt{\\lambda_2}<1,\n\\end{align*}\nwhere the second equality follows using that $1_n 1_n^\\top W = 1_n 1_n^\\top$ and $W^\\top 1_n 1_n^\\top  = 1_n 1_n^\\top$ since $W$ is doubly-stochastic. The proof is finished.\n\\end{proof}","sha256_utf8":"37ff477678398eabd81290dd6a47807eee745e0a6a2507822494f9be8c54ec28"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"证明乘积矩阵图的连通性","source_units":[{"start":42773,"end":43456,"start_line":643,"end_line":648,"raw_tex":"\\begin{proof}\n(1) The assumption that the graph associated to $W$ has a self-loop at every vertex, is equivalent to $w_{ii}>0$ for all $1\\leq i\\leq n$. Let $w_0:=\\frac{1}{2}\\min\\limits_{1\\leq i\\leq n}w_{ii}>0$ and $\\widetilde W:=W-w_0 I_n$. Then all entries of $\\widetilde W$ are nonnegative, and the directed graph associated to $\\widetilde W$ is precisely the same as that of $W$. Note that\n\\[\nW^\\top W=(\\widetilde W+w_0I_n)^\\top(\\widetilde W+w_0I_n)\\geq w_0 \\widetilde W,\n\\]\nwith the inequality being in entrywise sense. Hence the graph associated to $W^\\top W$ contains that associated to $\\widetilde W$ (or $W$) as its subgraph, which is indeed strongly connected and aperiodc.\n","sha256_utf8":"c633e54196a5083024707fcb64fea9a852217ab0510c2a0422330741fc95f233"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"证明首特征值单重","source_units":[{"start":43457,"end":43681,"start_line":650,"end_line":650,"raw_tex":"(2) Since $W^\\top W$ is symmetric positive semidefinite, we can arrange its eigenvalues as $\\lambda_1\\geq \\lambda_2\\geq\\dots\\geq\\lambda_n\\geq0$. The Perron-Frobenius theorem combined with (1) yields $1=\\lambda_1>\\lambda_2$.\n","sha256_utf8":"299308a26610e3172eddf7b8e05821a25ca808d6a1b30ac0dd9e21119334b427"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"谱分解计算共识收缩范数","source_units":[{"start":43682,"end":44608,"start_line":652,"end_line":666,"raw_tex":"(3) Since $W^\\top W$ is symmetric, it can be diagonalized by an orthogonal matrix, i.e.,\n\\[\nW^\\top W=Q \\operatorname{diag}(\\lambda_1,\\dots,\\lambda_n)Q^\\top,\\quad Q:=[u_1|\\dots|u_n],\n\\]\nwith $u_i$ being an eigenvector associated to $\\lambda_i$, $i=1,\\dots,n$. Note that $\\lambda_1=1$ and $u_1=\\frac{1}{\\sqrt{n}}1_n$, because $W$ is doubly-stochastic. Hence\n\\[\nW^\\top W=\\sum_{i=1}^n \\lambda_i u_iu_i^\\top=\\frac{1}{n}1_n1_n^\\top+\\sum_{i=2}^n \\lambda_i u_iu_i^\\top,\n\\]\nwhich yields\n\\begin{align*}\n\\left\\|W-\\frac{1}{n}1_n1_n^\\top\\right\\|&=\\sqrt{\\mbox{the largest eigenvalue of }(W-\\frac{1}{n}1_n1_n^\\top)^\\top(W-\\frac{1}{n}1_n1_n^\\top)}\\\\\n&=\\sqrt{\\mbox{the largest eigenvalue of }W^\\top W-\\frac{1}{n}1_n1_n^\\top}=\\sqrt{\\lambda_2}<1,\n\\end{align*}\nwhere the second equality follows using that $1_n 1_n^\\top W = 1_n 1_n^\\top$ and $W^\\top 1_n 1_n^\\top  = 1_n 1_n^\\top$ since $W$ is doubly-stochastic. The proof is finished.\n\\end{proof}","sha256_utf8":"5d8a6ebff7f9ba21873199448badbf63547a57d12e9bdddb75b423df166819f8"}]},{"id":"S045","semantic_type":"lemma","kind":"lemma","title":"光滑强凸函数的梯度不等式","source_units":[{"start":37757,"end":38110,"start_line":549,"end_line":553,"raw_tex":"\\begin{lem}\\label{lem-2-1} Assume that $f: \\mathbb{R}^n \\rightarrow \\mathbb{R}$ is $\\alpha$-strongly convex and $L$-smooth. Then, for any $x,y \\in \\mathbb{R}^n$ we have\n\\begin{equation*}\n\\langle \\nabla f(x) - \\nabla f(y), x-y \\rangle \\geq \\frac{L\\alpha}{L+\\alpha}\\|x-y\\|^2 + \\frac{1}{\\alpha +L} \\|\\nabla f(x) - \\nabla f(y)\\|^2.\n\\end{equation*}\n\\end{lem}","sha256_utf8":"c7f0ef3619bd250ad62e456137b3f333931a726bf915df51d8f10970d9fb81cc"}]},{"id":"S046","semantic_type":"external_fact","kind":"external_fact","title":"标准梯度插值不等式","source_units":[],"declared_external":true,"declaration_text":["For an alpha-strongly convex, L-smooth differentiable function h, with alpha>=0 and L>0, <grad h(x)-grad h(y),x-y> >= (L alpha/(L+alpha))||x-y||^2 + (1/(L+alpha))||grad h(x)-grad h(y)||^2. In particular, for alpha=0 the gradient is 1/L-cocoercive. This is the standard fact recalled in B0067 and cited to Bubeck, Lemma 3.11."]},{"id":"S047","semantic_type":"external_fact","kind":"external_fact","title":"随机矩阵与对称矩阵谱理论","source_units":[],"declared_external":true,"declaration_text":["An irreducible aperiodic stochastic matrix has simple eigenvalue 1 and every other eigenvalue has modulus less than 1. A real symmetric positive semidefinite matrix has an orthonormal eigenbasis and nonnegative eigenvalues; ||A||_2^2 is the largest eigenvalue of A^T A."]},{"id":"S048","semantic_type":"external_fact","kind":"external_fact","title":"有限维范数的标准性质","source_units":[],"declared_external":true,"declaration_text":["Triangle and Cauchy–Schwarz inequalities hold for Euclidean vectors. Induced operator norms are submultiplicative, all norms on a fixed finite-dimensional vector space are equivalent, and ||A tensor I_p||_2=||A||_2. These facts also justify the weighted-norm product estimates."]},{"id":"S049","semantic_type":"external_fact","kind":"external_fact","title":"凸函数最优性与平均不等式","source_units":[],"declared_external":true,"declaration_text":["A differentiable function on Euclidean space has zero gradient at an unconstrained minimizer. For differentiable convex f, f(y)>=f(x)+<grad f(x),y-x>, and f((1/T) sum_k x_k)<=(1/T) sum_k f(x_k)."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S009","edges":[{"source_id":"S001","target_id":"S007","kind":"supports"},{"source_id":"S001","target_id":"S026","kind":"supports"},{"source_id":"S002","target_id":"S008","kind":"supports"},{"source_id":"S002","target_id":"S022","kind":"supports"},{"source_id":"S002","target_id":"S042","kind":"supports"},{"source_id":"S003","target_id":"S021","kind":"supports"},{"source_id":"S003","target_id":"S025","kind":"supports"},{"source_id":"S003","target_id":"S027","kind":"supports"},{"source_id":"S004","target_id":"S021","kind":"supports"},{"source_id":"S004","target_id":"S036","kind":"supports"},{"source_id":"S005","target_id":"S013","kind":"supports"},{"source_id":"S005","target_id":"S017","kind":"supports"},{"source_id":"S006","target_id":"S024","kind":"supports"},{"source_id":"S006","target_id":"S029","kind":"supports"},{"source_id":"S007","target_id":"S022","kind":"supports"},{"source_id":"S008","target_id":"S022","kind":"supports"},{"source_id":"S008","target_id":"S026","kind":"supports"},{"source_id":"S010","target_id":"S014","kind":"supports"},{"source_id":"S010","target_id":"S015","kind":"supports"},{"source_id":"S010","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S011","kind":"supports"},{"source_id":"S013","target_id":"S011","kind":"supports"},{"source_id":"S014","target_id":"S011","kind":"supports"},{"source_id":"S015","target_id":"S011","kind":"supports"},{"source_id":"S016","target_id":"S011","kind":"supports"},{"source_id":"S017","target_id":"S011","kind":"supports"},{"source_id":"S011","target_id":"S009","kind":"supports"},{"source_id":"S018","target_id":"S015","kind":"supports"},{"source_id":"S019","target_id":"S028","kind":"supports"},{"source_id":"S019","target_id":"S031","kind":"supports"},{"source_id":"S019","target_id":"S032","kind":"supports"},{"source_id":"S020","target_id":"S018","kind":"supports"},{"source_id":"S021","target_id":"S025","kind":"supports"},{"source_id":"S021","target_id":"S031","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S031","kind":"supports"},{"source_id":"S025","target_id":"S032","kind":"supports"},{"source_id":"S026","target_id":"S030","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S020","kind":"supports"},{"source_id":"S023","target_id":"S020","kind":"supports"},{"source_id":"S024","target_id":"S020","kind":"supports"},{"source_id":"S025","target_id":"S020","kind":"supports"},{"source_id":"S026","target_id":"S020","kind":"supports"},{"source_id":"S027","target_id":"S020","kind":"supports"},{"source_id":"S028","target_id":"S020","kind":"supports"},{"source_id":"S029","target_id":"S020","kind":"supports"},{"source_id":"S030","target_id":"S020","kind":"supports"},{"source_id":"S031","target_id":"S020","kind":"supports"},{"source_id":"S032","target_id":"S020","kind":"supports"},{"source_id":"S026","target_id":"S012","kind":"supports"},{"source_id":"S028","target_id":"S012","kind":"supports"},{"source_id":"S029","target_id":"S012","kind":"supports"},{"source_id":"S028","target_id":"S013","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S037","kind":"supports"},{"source_id":"S034","target_id":"S038","kind":"supports"},{"source_id":"S034","target_id":"S039","kind":"supports"},{"source_id":"S035","target_id":"S033","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S035","kind":"supports"},{"source_id":"S039","target_id":"S035","kind":"supports"},{"source_id":"S040","target_id":"S022","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S043","target_id":"S041","kind":"supports"},{"source_id":"S044","target_id":"S041","kind":"supports"},{"source_id":"S045","target_id":"S028","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S043","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S022","kind":"supports"},{"source_id":"S048","target_id":"S023","kind":"supports"},{"source_id":"S048","target_id":"S024","kind":"supports"},{"source_id":"S048","target_id":"S029","kind":"supports"},{"source_id":"S048","target_id":"S037","kind":"supports"},{"source_id":"S048","target_id":"S038","kind":"supports"},{"source_id":"S048","target_id":"S039","kind":"supports"},{"source_id":"S049","target_id":"S027","kind":"supports"},{"source_id":"S049","target_id":"S013","kind":"supports"},{"source_id":"S049","target_id":"S017","kind":"supports"}]}}
{"record_id":"2110.03079v2","paper":{"title":"Convex relaxations of integral variational problems: pointwise dual relaxation and sum-of-squares optimization","authors":["Alexander Chernyavsky","Jason J. Bramburger","Giovanni Fantuzzi","David Goluskin"],"arxiv_id":"2110.03079","version":"v2","journal":"SIAM Journal on Optimization","doi":"10.1137/21m1455127","abstract_url":"https://arxiv.org/abs/2110.03079v2","pdf_url":"https://arxiv.org/pdf/2110.03079v2"},"context":{"raw_tex":"\\begin{equation}\n\\mathcal{F}^*\n:= \\inf_{\\substack{u\\in W^{1,p} \\\\ +\\text{ constraints}}}\n\t\\int_\\Omega f(x,u,\\nabla u)\\,\\dx,\n\\label{eq:primal-vp}\n\\end{equation}\n\nwhere $\\Omega\\subset\\R^n$ is open and bounded, $W^{1,p}$ is the Sobolev space of functions $u:\\Omega\\to\\R^m$ such that all components of $u$ and their weak derivatives are integrable in the Lebesgue space $L^p$ for some $p\\ge1$, and $\\nabla u:\\Omega\\to\\R^{m\\times n}$ is the gradient tensor defined in the weak sense. The integrand function $f$ can be any such that the integral is well defined for all admissible $u$, and, as specified in the next section, the constraints can include pointwise or integral conditions over $\\Omega$ or its boundary $\\partial\\Omega$. In general the infimum $\\cF^*$ might not be attained, meaning no global minimizer $u$ exists.\n\n\n\\begin{equation}\n\\label{eq: D}\n\\cD\\varphi(x,y,z) := \\sum_{i=1}^n \\frac{\\partial\\varphi_i}{\\partial x_i}\n+ \\sum_{i=1}^n \\sum_{j=1}^m z_{ji}\\frac{\\partial\\varphi_i}{\\partial y_j}.\n\\end{equation}\n\nConsider the integral variational problem~\\cref{eq:primal-vp}, where $\\Omega$ is a bounded open Lipschitz domain. For simplicity we assume that the domain boundary is specified as a finite union, $\\partial\\Omega = \\cup_{i=1}^s \\partial\\Omega_i$, where each $\\partial\\Omega_i$ is a smooth compact manifold.\nWe assume the integrand function $f$ in~\\cref{eq:primal-vp} is continuous and gives a well defined integral over $\\Omega$ for all $u\\in W^{1,p}$ satisfying the constraints. For instance, in the absence of pointwise constraints that would ensure $u(x)$ and $\\nabla u(x)$ are uniformly bounded, we assume that $|f(x,y,z)|$ grows no faster than $|y|^p + |z|^p$ as $|y|,|z|\\to\\infty$.\n\n\nFor the purpose of illustration throughout this section, we suppose the constraints on $u$ in~\\cref{eq:primal-vp} include one integral inequality, one pointwise equality on $\\Omega$, and one equality on each boundary component $\\partial\\Omega_i$ (i.e, one boundary condition):\n\n\n\\begin{align}\n\t\\label{int-both}\n\t\\int_\\Omega a(x,u,\\nabla u) \\,\\dx &\\ge 0,\n\t\\\\\n\t\\label{pw-domain}\n\tc(x,u,\\nabla u)&= 0 \\quad x\\in\\Omega,\n\t\\\\\n\t\\label{pw-boundary}\n\td_i(x,u)&= 0 \\quad x\\in\\partial\\Omega_i,\\quad i=1,\\ldots,s.\n\t\\end{align}\n\nWe assume that the functions $a,c,d_i$ are continuous and that $a(x,u,\\nabla u)$ is integrable for all $u\\in W^{1,p}$.\n\n\nThe constraint $v=\\nabla u$ is imposed using the divergence theorem for vector-valued functions $x \\mapsto \\vphi(x,u(x))$, where $\\varphi: \\overline{\\Omega}\\times\\mathbb{R}^m \\to \\mathbb{R}^n$ is continuously differentiable:\n\\beq\n\\label{eq:div-theorem}\n\\int_\\Omega\\diver_x\\vphi(x,u(x))\\,\\dx - \\sum_{i=1}^s \\int_{\\partial\\Omega_i}\\varphi(x,u)\\cdot n_i(x) \\,\\dbound = 0.\n\\eeq\n\nHere, $n_i(x)$ is the outward unit normal vector on $\\partial\\Omega_i$. Let $\\Phi$ denote the set of $\\vphi$ for which the above identity holds and for which the map $x \\mapsto \\varphi(x,u(x))$ is differentiable via the chain rule for all $u\\in W^{1,p}(\\Omega,\\R^m)$. (See~\\cite{Fantuzzi2022} for a more explicit characterization of $\\Phi$.) Since $\\cD\\vphi$ in~\\cref{eq: D} is defined in order to satisfy $\\diver_x\\vphi(x,u(x))=\\cD\\vphi(x,u,\\nabla u)$, the left-hand integrand in~\\cref{eq:div-theorem} can be replaced by $\\cD\\vphi(x,u,\\nabla u)$. If $v=\\nabla u$ weakly, therefore, identity~\\cref{eq:div-theorem} holds for all $\\vphi\\in\\Phi$ with the left-hand integrand replaced by $\\cD\\vphi(x,u,v)$. The proof of~\\cite[Lemma~1]{Korda2018} implies that the converse is also true, so the divergence theorem characterizes the weak gradient $\\nabla u$. The next Lemma summarizes these observations.\n\n\n\\begin{align}\n\\tag{\\normalsize\\theequation a}\n\\cL^*&\\ge\n\\sup_{\\substack{\\eta \\in \\mathbb{R}_+\\\\\\varphi \\in \\Phi}}\n\t\\inf_{\\substack{u\\in W^{1,p}\\\\v\\in L^p\\\\ \\text{(\\ref{eq:F-div-dual}c,d)}}}\\!\\!\n\t\\left\\{\\!\\int\\limits_\\Omega \\![f-\\eta a + \\mathcal{D}\\vphi](x,u,v)\\,\\dx -\\sum_{i=1}^s \\int\\limits_{\\partial\\Omega_i}\\!\\!\\varphi(x,u)\\cdot n_i(x)\\,\\dbound \\right\\}\n\\label{eq:div-dual}\n\\\\\n\\tag{\\normalsize\\theequation b}\n&\\ge\n\\sup_{\\substack{\\eta \\in \\mathbb{R}_+\\\\\\varphi \\in \\Phi}}\n\\left\\{\\! \\int\\limits_{\\Omega}\\!\\!\\!\\! \\inf_{\\substack{ y\\in \\R^m \\\\ z \\in \\R^{m\\times n}\\\\c(x,y,z)=0}}\\!\\!\\![f-\\eta a + \\mathcal{D}\\vphi](x,y,z)\\, \\dx+ \\sum_{i=1}^s \\int\\limits_{\\partial\\Omega_i}\\!\\! \\inf_{\\substack{y\\in\\R^m\\\\d_i(x,y)=0}}\\!\\!\\!\\!-\\varphi(x,y)\\cdot n_i(x)\\dbound\\right\\}.\n\\label{eq:relaxed-dual}\n\\end{align}\n\nFinally, we bound the pointwise infima in~\\cref{eq:relaxed-dual} from below by the largest continuous functions $h\\in C(\\overline{\\Omega},\\R)$ and $\\ell_i\\in C(\\partial\\Omega_i,\\R)$ satisfying\n\n\n\\begin{align}\n\t\\label{eq:pw-inequality-bulk}\n\tf(x,y,z) - \\eta\\, a(x,y,z) + \\mathcal{D}\\varphi(x,y,z)  - h(x) &\\geq 0 \\; \\text { on } \\Gamma,\\\\\n\t-\\varphi(x,y)\\cdot n_i(x) - \\ell_i(x) &\\geq 0  \\; \\text { on } \\Lambda_i, \\quad i=1,\\ldots,s,\n\t\\label{eq:pw-inequality-boundary}\n\t\\end{align}\n\nwhere $\\Gamma$ and $\\Lambda_i$ are the sets on which the pointwise constraints are satisfied:\n\n\n\\begin{align}\n\\label{eq:Gamma-def}\n\\Gamma &:= \\{(x,y,z) \\in \\Omega \\times \\mathbb{R}^m \\times \\mathbb{R}^{m \\times n}\\colon\\; c(x,y,z)= 0\\},\\\\\n\\Lambda_i &:= \\{(x,y) \\in \\partial\\Omega_i \\times \\mathbb{R}^m \\colon \\; d_i(x,y)= 0\\}. \\label{eq:Lambda-def}\n\\end{align}\n\n\\begin{equation}\n\\label{eq:pdr-2}\n\\cF^* \\geq \\cL^* \\geq \\Lpdr :=\n\\sup_{\\substack{ \\eta \\in \\mathbb{R}_+,\\,\\varphi \\in \\Phi,\\\\ h \\in C(\\overline{\\Omega}),\\, \\ell_i \\in C(\\partial\\Omega_i)\\\\ \\text{\\rm ~(\\ref{eq:relaxed-inequalities-all}a,b)} }} \\;\n\\left\\{\n\\int_\\Omega h(x) \\,\\dx\n+\\sum_{i=1}^s \\int_{\\partial\\Omega_i} \\ell_i(x) \\,\\dbound\n\\right\\},\n\\end{equation}\n\nWe now consider the particular case in which the integral variational problem~\\cref{eq:primal-vp} or the integral-constrained optimization problem~\\cref{eq:integral-inequality-setup} has polynomial data, meaning that all functions appearing as integrands or in pointwise constraints are polynomial in $(x,u,\\nabla u)$, and that the domain $\\Omega$ can be specified by polynomials in $x$. Precisely, we assume that $\\overline\\Omega$ is a basic closed semialgebraic set defined by $s$ polynomial inequalities,\n\\begin{equation}\n\\overline\\Omega = \\{x \\in \\R^n:\\; g_1(x)\\ge 0,\\,\\ldots,\\,g_s(x)\\ge 0\\}.\n\\label{def-omega}\n\\end{equation}\n\nWe also assume that each $g_i$ vanishes identically on a single boundary component, so\n\\begin{align}\n\\partial\\Omega_i\n&= \\{x \\in \\R^n:\\;g_i(x)=0 \\text{ and } g_j(x)\\geq 0 \\text{ for } j\\neq i \\},\n\\end{align}\n\nand that $\\nabla g_i$ does not vanish anywhere on $\\partial\\Omega_i$, so the outward unit normal vector is well defined as $n_i(x) = -\\nabla g_i/|\\nabla g_i|$. Then, the sets $\\Gamma$ and $\\Lambda_i$ defined in~\\cref{eq:Gamma-def,eq:Lambda-def} admit the semialgebraic definitions\n\n\n\\begin{align}\n\\Gamma &= \\{(x,y,z) \\in \\R^n \\times \\R^m \\times \\R^{m\\times n}:\nc(x,y,z)= 0,\\,g_1(x)\\ge 0,\\ldots,g_s(x)\\geq 0\\},\n\\\\\n\\Lambda_i &= \\{(x,y) \\in \\R^n \\times \\R^m:\\,\nd_i(x,y) = 0,\\,g_i(x)=0,\\,\ng_j(x)\\geq 0 \\text{ for } j\\neq i\n\\}.\n\\end{align}\n\nFor any positive integer $\\nu$, we can bound $\\Lpdr$ from below by restricting the maximization in~\\cref{eq:pdr-2} to \n\\begin{align}\n\\vphi &\\in \\Phi\\cap\\R^n[x,y]_\\nu, &\nh &\\in \\R[x]_\\nu, &\n\\ell_i &= q_i |\\nabla g_i| \\; \\text{ with } \\; q_i \\in \\R[x]_\\nu,\n\\end{align}\n\nwhere $\\R^n[\\cdot]_\\nu$ is the space of $n$-dimensional vectors of polynomials in the bracketed variables whose degrees are at most $\\nu$. More generally, one can choose any finite vector space of polynomials, \nor use rational functions with fixed nonnegative denominators (see \\cref{rem:rational} below). The intersection with $\\Phi$\nexcludes polynomials $\\vphi(x,y)$ whose degree in $y$ is too large to guarantee integrability of $\\cD\\vphi(x,u,\\nabla u)$ for all $u\\in W^{1,p}$. Recalling that $n_i(x) = -\\nabla g_i/|\\nabla g_i|$ for each $i=1,\\ldots,s$, we find\n\\begin{equation}\n\\label{eq:vp-finite-dimensional}\n\\Lpdr\\ge\n\\Lnu :=\\!\\!\\!\\!\\!\\!\\!\n\\sup_{ \\substack{\\vphi \\in \\Phi \\cap \\R^n[x,y]_\\nu \\\\h, q_i \\in \\R[x]_\\nu, \\, \\eta \\in \\R_+} }\n\\!\\!\\!\\!\\!\\!L(h,q_1,\\ldots,q_s)\n\\;\\; \\text{s.t.}\n\\begin{array}[t]{l}\nf - \\eta a + \\mathcal{D}\\varphi - h \\geq 0  \\,\\text { on } \\Gamma,\\\\\n\\varphi\\cdot \\nabla g_i - q_i|\\nabla g_i|^2 \\geq 0  \\, \\text { on } \\Lambda_i, \n\\end{array}\n\\end{equation}\n\nwhere the maximization objective is\n\\beq\n\\label{eq:L-objective}\nL(h,q_1,\\ldots,q_s) : = \\int_{\\Omega} h(x) \\, \\dx + \\sum_{i=1}^s \\int_{\\partial\\Omega_i} q_i(x)|\\nabla g_i(x)| \\,\\dbound.\n\\eeq\n\nProblem~\\cref{eq:vp-finite-dimensional} is finite-dimensional but still intractable because pointwise polynomial inequalities are NP-hard to verify in general~\\cite{Murty1987}. For this reason, we follow a standard approach~\\cite{Parrilo2003,Lasserre2001} and strengthen all polynomial nonnegativity constraints into weighted SOS constraints. Letting $\\Sigma[\\cdot]$ denote the subset of nonnegative polynomials in the bracketed variables that can be written as sums of squares of other polynomials, we consider the sets\n\n\n\\begin{align}\n\t\\mathcal{Q}(\\Gamma) &:= \\bigg\\{\\rho c + \\sigma_0 + \\sum_{i=1}^s g_i \\sigma_i:\\; \\sigma_k \\in \\Sigma[x,y,z], \\,\\rho \\in \\R[x,y,z] \\bigg\\}, \\\\\n\t\\mathcal{Q}(\\Lambda_i) &:= \\bigg\\{\\rho_1 d_i + \\rho_2 g_i  + \\sigma_0 + \\sum_{j\\neq i} g_i \\sigma_i:\\; \\sigma_k \\in \\Sigma[x,y], \\,\\rho_j \\in \\R[x,y] \\bigg\\}.\n\t\\end{align}\n\n\\begin{align}\n\\label{eq:quad-mods-nu}\n\t\\mathcal{Q}_\\nu(\\Gamma) &= \\mathcal{Q}(\\Gamma) \\cap \\R[x,y,z]_\\nu, &\n\t\\mathcal{Q}_\\nu(\\Lambda_i) &= \\mathcal{Q}(\\Lambda_i) \\cap \\R[x,y]_\\nu.\n\\end{align}\n\nRequiring a polynomial to belong to a finite-degree set such as $\\cQ_\\nu(\\Gamma)$ or $\\cQ_\\nu(\\Lambda_i)$ is a \\emph{weighted SOS} condition, which can be included as a constraint in an SOS program.\n\n\nThe constraints of~\\cref{eq:vp-finite-dimensional}, which enforce polynomial nonnegativity on semialgebraic sets, can be strengthened into weighted SOS constraints to obtain\n\\begin{equation}\n\\label{eq:vp-SOS}\n\\Lnu\\ge\\Lsos :=\\!\\!\\!\\!\\!\\!\\!\n\\sup_{ \\substack{\\vphi \\in \\Phi \\cap \\R^n[x,y]_\\nu\\\\h, q_i \\in \\R[x]_\\nu, \\, \\eta \\in \\R_+} }\n\\!\\!\\!\\!\\!\\!L(h,q_1,\\ldots,q_s)\n\\;\\; \\text{s.t.}\n\\begin{array}[t]{l}\nf - \\eta a + \\mathcal{D}\\varphi - h \\in \\cQ_\\nu(\\Gamma),\\\\\n\\varphi\\cdot \\nabla g_i - q_i|\\nabla g_i|^2 \\in \\cQ_\\nu(\\Lambda_i). \n\\end{array}\n\\end{equation}","units":[{"start":10122,"end":10281,"start_line":225,"end_line":230,"raw_tex":"\\begin{equation}\n\\mathcal{F}^*\n:= \\inf_{\\substack{u\\in W^{1,p} \\\\ +\\text{ constraints}}}\n\t\\int_\\Omega f(x,u,\\nabla u)\\,\\dx,\n\\label{eq:primal-vp}\n\\end{equation}","sha256_utf8":"4a06fd017440d0741dd483aaa4ee27c46973474df87c2f097d5fe9765f09ef6b"},{"start":10282,"end":10943,"start_line":231,"end_line":231,"raw_tex":"where $\\Omega\\subset\\R^n$ is open and bounded, $W^{1,p}$ is the Sobolev space of functions $u:\\Omega\\to\\R^m$ such that all components of $u$ and their weak derivatives are integrable in the Lebesgue space $L^p$ for some $p\\ge1$, and $\\nabla u:\\Omega\\to\\R^{m\\times n}$ is the gradient tensor defined in the weak sense. The integrand function $f$ can be any such that the integral is well defined for all admissible $u$, and, as specified in the next section, the constraints can include pointwise or integral conditions over $\\Omega$ or its boundary $\\partial\\Omega$. In general the infimum $\\cF^*$ might not be attained, meaning no global minimizer $u$ exists.\n","sha256_utf8":"8ebe5e0a948aa7d46167a04c5084f19d22946e21a9682966c435aaf12eabaf8b"},{"start":13470,"end":13662,"start_line":246,"end_line":250,"raw_tex":"\\begin{equation}\n\\label{eq: D}\n\\cD\\varphi(x,y,z) := \\sum_{i=1}^n \\frac{\\partial\\varphi_i}{\\partial x_i}\n+ \\sum_{i=1}^n \\sum_{j=1}^m z_{ji}\\frac{\\partial\\varphi_i}{\\partial y_j}.\n\\end{equation}","sha256_utf8":"4602a9b28c0bbfac980615108c38a7eb06eb71413d314ac15eae902591d9b8e9"},{"start":26557,"end":27244,"start_line":321,"end_line":322,"raw_tex":"Consider the integral variational problem~\\cref{eq:primal-vp}, where $\\Omega$ is a bounded open Lipschitz domain. For simplicity we assume that the domain boundary is specified as a finite union, $\\partial\\Omega = \\cup_{i=1}^s \\partial\\Omega_i$, where each $\\partial\\Omega_i$ is a smooth compact manifold.\nWe assume the integrand function $f$ in~\\cref{eq:primal-vp} is continuous and gives a well defined integral over $\\Omega$ for all $u\\in W^{1,p}$ satisfying the constraints. For instance, in the absence of pointwise constraints that would ensure $u(x)$ and $\\nabla u(x)$ are uniformly bounded, we assume that $|f(x,y,z)|$ grows no faster than $|y|^p + |z|^p$ as $|y|,|z|\\to\\infty$.\n","sha256_utf8":"060013b67d3be985fa78129620b75fe6521206242ff62c0c933c3423a234aa86"},{"start":27676,"end":27953,"start_line":325,"end_line":325,"raw_tex":"For the purpose of illustration throughout this section, we suppose the constraints on $u$ in~\\cref{eq:primal-vp} include one integral inequality, one pointwise equality on $\\Omega$, and one equality on each boundary component $\\partial\\Omega_i$ (i.e, one boundary condition):\n","sha256_utf8":"09163329930276b31ad8b1cb3784c8f76e406f30cc7dfb17231bf42aaa60a8d6"},{"start":28005,"end":28240,"start_line":329,"end_line":338,"raw_tex":"\\begin{align}\n\t\\label{int-both}\n\t\\int_\\Omega a(x,u,\\nabla u) \\,\\dx &\\ge 0,\n\t\\\\\n\t\\label{pw-domain}\n\tc(x,u,\\nabla u)&= 0 \\quad x\\in\\Omega,\n\t\\\\\n\t\\label{pw-boundary}\n\td_i(x,u)&= 0 \\quad x\\in\\partial\\Omega_i,\\quad i=1,\\ldots,s.\n\t\\end{align}","sha256_utf8":"394088bb8eda0b10c94fba590d0b365f8587d9809a5aac3b990da50069b8dcac"},{"start":28262,"end":28381,"start_line":341,"end_line":341,"raw_tex":"We assume that the functions $a,c,d_i$ are continuous and that $a(x,u,\\nabla u)$ is integrable for all $u\\in W^{1,p}$.\n","sha256_utf8":"60ff425b146df358e088b3e81a28e864ae318ee08d5338100ae5391e3bb293c0"},{"start":31802,"end":32175,"start_line":376,"end_line":380,"raw_tex":"The constraint $v=\\nabla u$ is imposed using the divergence theorem for vector-valued functions $x \\mapsto \\vphi(x,u(x))$, where $\\varphi: \\overline{\\Omega}\\times\\mathbb{R}^m \\to \\mathbb{R}^n$ is continuously differentiable:\n\\beq\n\\label{eq:div-theorem}\n\\int_\\Omega\\diver_x\\vphi(x,u(x))\\,\\dx - \\sum_{i=1}^s \\int_{\\partial\\Omega_i}\\varphi(x,u)\\cdot n_i(x) \\,\\dbound = 0.\n\\eeq","sha256_utf8":"84be6cc1baa8ca636076b5b6c3996d7f5bf1bbe30dc87b81460537457545e44d"},{"start":32176,"end":33074,"start_line":381,"end_line":381,"raw_tex":"Here, $n_i(x)$ is the outward unit normal vector on $\\partial\\Omega_i$. Let $\\Phi$ denote the set of $\\vphi$ for which the above identity holds and for which the map $x \\mapsto \\varphi(x,u(x))$ is differentiable via the chain rule for all $u\\in W^{1,p}(\\Omega,\\R^m)$. (See~\\cite{Fantuzzi2022} for a more explicit characterization of $\\Phi$.) Since $\\cD\\vphi$ in~\\cref{eq: D} is defined in order to satisfy $\\diver_x\\vphi(x,u(x))=\\cD\\vphi(x,u,\\nabla u)$, the left-hand integrand in~\\cref{eq:div-theorem} can be replaced by $\\cD\\vphi(x,u,\\nabla u)$. If $v=\\nabla u$ weakly, therefore, identity~\\cref{eq:div-theorem} holds for all $\\vphi\\in\\Phi$ with the left-hand integrand replaced by $\\cD\\vphi(x,u,v)$. The proof of~\\cite[Lemma~1]{Korda2018} implies that the converse is also true, so the divergence theorem characterizes the weak gradient $\\nabla u$. The next Lemma summarizes these observations.\n","sha256_utf8":"cb1c96e723adef29975da860d735dcc220f733cafc89d9c5b6c0cb63b4c35833"},{"start":35515,"end":36315,"start_line":405,"end_line":418,"raw_tex":"\\begin{align}\n\\tag{\\normalsize\\theequation a}\n\\cL^*&\\ge\n\\sup_{\\substack{\\eta \\in \\mathbb{R}_+\\\\\\varphi \\in \\Phi}}\n\t\\inf_{\\substack{u\\in W^{1,p}\\\\v\\in L^p\\\\ \\text{(\\ref{eq:F-div-dual}c,d)}}}\\!\\!\n\t\\left\\{\\!\\int\\limits_\\Omega \\![f-\\eta a + \\mathcal{D}\\vphi](x,u,v)\\,\\dx -\\sum_{i=1}^s \\int\\limits_{\\partial\\Omega_i}\\!\\!\\varphi(x,u)\\cdot n_i(x)\\,\\dbound \\right\\}\n\\label{eq:div-dual}\n\\\\\n\\tag{\\normalsize\\theequation b}\n&\\ge\n\\sup_{\\substack{\\eta \\in \\mathbb{R}_+\\\\\\varphi \\in \\Phi}}\n\\left\\{\\! \\int\\limits_{\\Omega}\\!\\!\\!\\! \\inf_{\\substack{ y\\in \\R^m \\\\ z \\in \\R^{m\\times n}\\\\c(x,y,z)=0}}\\!\\!\\![f-\\eta a + \\mathcal{D}\\vphi](x,y,z)\\, \\dx+ \\sum_{i=1}^s \\int\\limits_{\\partial\\Omega_i}\\!\\! \\inf_{\\substack{y\\in\\R^m\\\\d_i(x,y)=0}}\\!\\!\\!\\!-\\varphi(x,y)\\cdot n_i(x)\\dbound\\right\\}.\n\\label{eq:relaxed-dual}\n\\end{align}","sha256_utf8":"50cec111b0bba00d1104e8dfbfc1280f0aa7b0cea752e4ddc19e5d74180fe334"},{"start":36348,"end":36541,"start_line":422,"end_line":422,"raw_tex":"Finally, we bound the pointwise infima in~\\cref{eq:relaxed-dual} from below by the largest continuous functions $h\\in C(\\overline{\\Omega},\\R)$ and $\\ell_i\\in C(\\partial\\Omega_i,\\R)$ satisfying\n","sha256_utf8":"39946146ee921d735a05ddc745cde262ad1b973e57cb6447b2a5ef6c746e2751"},{"start":36600,"end":36885,"start_line":425,"end_line":430,"raw_tex":"\\begin{align}\n\t\\label{eq:pw-inequality-bulk}\n\tf(x,y,z) - \\eta\\, a(x,y,z) + \\mathcal{D}\\varphi(x,y,z)  - h(x) &\\geq 0 \\; \\text { on } \\Gamma,\\\\\n\t-\\varphi(x,y)\\cdot n_i(x) - \\ell_i(x) &\\geq 0  \\; \\text { on } \\Lambda_i, \\quad i=1,\\ldots,s,\n\t\\label{eq:pw-inequality-boundary}\n\t\\end{align}","sha256_utf8":"1de19e7d8a4e29cfcfb0dbb3244aa78b60ad2aa9b93999f96a8a7b62598172e0"},{"start":36905,"end":36999,"start_line":432,"end_line":432,"raw_tex":"where $\\Gamma$ and $\\Lambda_i$ are the sets on which the pointwise constraints are satisfied:\n","sha256_utf8":"cbdca12de2a47fa2f08e80fdd43166f8586fc36f536468cb13ccd82f3e7405a8"},{"start":37044,"end":37308,"start_line":435,"end_line":439,"raw_tex":"\\begin{align}\n\\label{eq:Gamma-def}\n\\Gamma &:= \\{(x,y,z) \\in \\Omega \\times \\mathbb{R}^m \\times \\mathbb{R}^{m \\times n}\\colon\\; c(x,y,z)= 0\\},\\\\\n\\Lambda_i &:= \\{(x,y) \\in \\partial\\Omega_i \\times \\mathbb{R}^m \\colon \\; d_i(x,y)= 0\\}. \\label{eq:Lambda-def}\n\\end{align}","sha256_utf8":"f81cd4e4032020b06a0dc327f3375ffbd28327649735af35faaf1dc38ccad5c3"},{"start":37344,"end":37703,"start_line":444,"end_line":452,"raw_tex":"\\begin{equation}\n\\label{eq:pdr-2}\n\\cF^* \\geq \\cL^* \\geq \\Lpdr :=\n\\sup_{\\substack{ \\eta \\in \\mathbb{R}_+,\\,\\varphi \\in \\Phi,\\\\ h \\in C(\\overline{\\Omega}),\\, \\ell_i \\in C(\\partial\\Omega_i)\\\\ \\text{\\rm ~(\\ref{eq:relaxed-inequalities-all}a,b)} }} \\;\n\\left\\{\n\\int_\\Omega h(x) \\,\\dx\n+\\sum_{i=1}^s \\int_{\\partial\\Omega_i} \\ell_i(x) \\,\\dbound\n\\right\\},\n\\end{equation}","sha256_utf8":"07296c376b3e5291f827e067c47669693c94253d77723274310909ef9902eecf"},{"start":52096,"end":52725,"start_line":586,"end_line":590,"raw_tex":"We now consider the particular case in which the integral variational problem~\\cref{eq:primal-vp} or the integral-constrained optimization problem~\\cref{eq:integral-inequality-setup} has polynomial data, meaning that all functions appearing as integrands or in pointwise constraints are polynomial in $(x,u,\\nabla u)$, and that the domain $\\Omega$ can be specified by polynomials in $x$. Precisely, we assume that $\\overline\\Omega$ is a basic closed semialgebraic set defined by $s$ polynomial inequalities,\n\\begin{equation}\n\\overline\\Omega = \\{x \\in \\R^n:\\; g_1(x)\\ge 0,\\,\\ldots,\\,g_s(x)\\ge 0\\}.\n\\label{def-omega}\n\\end{equation}","sha256_utf8":"bb2abbba5a1a7324b8c3d63c15d2b74fa27135f76b276a1e8e4264b7d272d97a"},{"start":52726,"end":52933,"start_line":591,"end_line":595,"raw_tex":"We also assume that each $g_i$ vanishes identically on a single boundary component, so\n\\begin{align}\n\\partial\\Omega_i\n&= \\{x \\in \\R^n:\\;g_i(x)=0 \\text{ and } g_j(x)\\geq 0 \\text{ for } j\\neq i \\},\n\\end{align}","sha256_utf8":"41ce26ca826b52c7a4a7b51522975df6169e9d2296720aa60f9fd80e83ed5629"},{"start":52934,"end":53215,"start_line":596,"end_line":596,"raw_tex":"and that $\\nabla g_i$ does not vanish anywhere on $\\partial\\Omega_i$, so the outward unit normal vector is well defined as $n_i(x) = -\\nabla g_i/|\\nabla g_i|$. Then, the sets $\\Gamma$ and $\\Lambda_i$ defined in~\\cref{eq:Gamma-def,eq:Lambda-def} admit the semialgebraic definitions\n","sha256_utf8":"eadfb4a10f641c207fad36b05a315a3a95db0bedebfafcf8dd90283fe11f0ae2"},{"start":53270,"end":53521,"start_line":600,"end_line":608,"raw_tex":"\\begin{align}\n\\Gamma &= \\{(x,y,z) \\in \\R^n \\times \\R^m \\times \\R^{m\\times n}:\nc(x,y,z)= 0,\\,g_1(x)\\ge 0,\\ldots,g_s(x)\\geq 0\\},\n\\\\\n\\Lambda_i &= \\{(x,y) \\in \\R^n \\times \\R^m:\\,\nd_i(x,y) = 0,\\,g_i(x)=0,\\,\ng_j(x)\\geq 0 \\text{ for } j\\neq i\n\\}.\n\\end{align}","sha256_utf8":"6536f89559b6b7b977057bd34ee226636b33b0eaa58266048b69df9ccc8a2934"},{"start":53917,"end":54183,"start_line":614,"end_line":619,"raw_tex":"For any positive integer $\\nu$, we can bound $\\Lpdr$ from below by restricting the maximization in~\\cref{eq:pdr-2} to \n\\begin{align}\n\\vphi &\\in \\Phi\\cap\\R^n[x,y]_\\nu, &\nh &\\in \\R[x]_\\nu, &\n\\ell_i &= q_i |\\nabla g_i| \\; \\text{ with } \\; q_i \\in \\R[x]_\\nu,\n\\end{align}","sha256_utf8":"74a21b747acd2f88d228762234562ca49d71e4182a54100b8b7b09e65bb84769"},{"start":54184,"end":55169,"start_line":620,"end_line":634,"raw_tex":"where $\\R^n[\\cdot]_\\nu$ is the space of $n$-dimensional vectors of polynomials in the bracketed variables whose degrees are at most $\\nu$. More generally, one can choose any finite vector space of polynomials, \nor use rational functions with fixed nonnegative denominators (see \\cref{rem:rational} below). The intersection with $\\Phi$\nexcludes polynomials $\\vphi(x,y)$ whose degree in $y$ is too large to guarantee integrability of $\\cD\\vphi(x,u,\\nabla u)$ for all $u\\in W^{1,p}$. Recalling that $n_i(x) = -\\nabla g_i/|\\nabla g_i|$ for each $i=1,\\ldots,s$, we find\n\\begin{equation}\n\\label{eq:vp-finite-dimensional}\n\\Lpdr\\ge\n\\Lnu :=\\!\\!\\!\\!\\!\\!\\!\n\\sup_{ \\substack{\\vphi \\in \\Phi \\cap \\R^n[x,y]_\\nu \\\\h, q_i \\in \\R[x]_\\nu, \\, \\eta \\in \\R_+} }\n\\!\\!\\!\\!\\!\\!L(h,q_1,\\ldots,q_s)\n\\;\\; \\text{s.t.}\n\\begin{array}[t]{l}\nf - \\eta a + \\mathcal{D}\\varphi - h \\geq 0  \\,\\text { on } \\Gamma,\\\\\n\\varphi\\cdot \\nabla g_i - q_i|\\nabla g_i|^2 \\geq 0  \\, \\text { on } \\Lambda_i, \n\\end{array}\n\\end{equation}","sha256_utf8":"eb41d9dbbc4994f8dd78c51c99040f569a92ed9d2b55b84e999a1fb6b1ae04f3"},{"start":55170,"end":55360,"start_line":635,"end_line":639,"raw_tex":"where the maximization objective is\n\\beq\n\\label{eq:L-objective}\nL(h,q_1,\\ldots,q_s) : = \\int_{\\Omega} h(x) \\, \\dx + \\sum_{i=1}^s \\int_{\\partial\\Omega_i} q_i(x)|\\nabla g_i(x)| \\,\\dbound.\n\\eeq","sha256_utf8":"d62d2ff127f4899095015a69d5eceeba5e638173ca253e7b01b19976c72ab15a"},{"start":55906,"end":56427,"start_line":642,"end_line":642,"raw_tex":"Problem~\\cref{eq:vp-finite-dimensional} is finite-dimensional but still intractable because pointwise polynomial inequalities are NP-hard to verify in general~\\cite{Murty1987}. For this reason, we follow a standard approach~\\cite{Parrilo2003,Lasserre2001} and strengthen all polynomial nonnegativity constraints into weighted SOS constraints. Letting $\\Sigma[\\cdot]$ denote the subset of nonnegative polynomials in the bracketed variables that can be written as sums of squares of other polynomials, we consider the sets\n","sha256_utf8":"63c90c500716b18e1465d57bdd586b309d88f6af78d92f7b77bd59cf45bee390"},{"start":56470,"end":56798,"start_line":645,"end_line":648,"raw_tex":"\\begin{align}\n\t\\mathcal{Q}(\\Gamma) &:= \\bigg\\{\\rho c + \\sigma_0 + \\sum_{i=1}^s g_i \\sigma_i:\\; \\sigma_k \\in \\Sigma[x,y,z], \\,\\rho \\in \\R[x,y,z] \\bigg\\}, \\\\\n\t\\mathcal{Q}(\\Lambda_i) &:= \\bigg\\{\\rho_1 d_i + \\rho_2 g_i  + \\sigma_0 + \\sum_{j\\neq i} g_i \\sigma_i:\\; \\sigma_k \\in \\Sigma[x,y], \\,\\rho_j \\in \\R[x,y] \\bigg\\}.\n\t\\end{align}","sha256_utf8":"1232c204dc27fb2d220137fc7db6ec10f93d30c912dabac2d972e7fcdd935b5d"},{"start":57227,"end":57418,"start_line":651,"end_line":655,"raw_tex":"\\begin{align}\n\\label{eq:quad-mods-nu}\n\t\\mathcal{Q}_\\nu(\\Gamma) &= \\mathcal{Q}(\\Gamma) \\cap \\R[x,y,z]_\\nu, &\n\t\\mathcal{Q}_\\nu(\\Lambda_i) &= \\mathcal{Q}(\\Lambda_i) \\cap \\R[x,y]_\\nu.\n\\end{align}","sha256_utf8":"f5f6c8af06120755a89b8801871780df8ea5b02a0c1f2805bfe2023c9ed82616"},{"start":57419,"end":57618,"start_line":656,"end_line":656,"raw_tex":"Requiring a polynomial to belong to a finite-degree set such as $\\cQ_\\nu(\\Gamma)$ or $\\cQ_\\nu(\\Lambda_i)$ is a \\emph{weighted SOS} condition, which can be included as a constraint in an SOS program.\n","sha256_utf8":"e941bd7f3cee47fd4f3d6d7fe1c6a9ab9dd9455ba21e3bf1cbc3302eca51960a"},{"start":57619,"end":58175,"start_line":658,"end_line":669,"raw_tex":"The constraints of~\\cref{eq:vp-finite-dimensional}, which enforce polynomial nonnegativity on semialgebraic sets, can be strengthened into weighted SOS constraints to obtain\n\\begin{equation}\n\\label{eq:vp-SOS}\n\\Lnu\\ge\\Lsos :=\\!\\!\\!\\!\\!\\!\\!\n\\sup_{ \\substack{\\vphi \\in \\Phi \\cap \\R^n[x,y]_\\nu\\\\h, q_i \\in \\R[x]_\\nu, \\, \\eta \\in \\R_+} }\n\\!\\!\\!\\!\\!\\!L(h,q_1,\\ldots,q_s)\n\\;\\; \\text{s.t.}\n\\begin{array}[t]{l}\nf - \\eta a + \\mathcal{D}\\varphi - h \\in \\cQ_\\nu(\\Gamma),\\\\\n\\varphi\\cdot \\nabla g_i - q_i|\\nabla g_i|^2 \\in \\cQ_\\nu(\\Lambda_i). \n\\end{array}\n\\end{equation}","sha256_utf8":"fa3d5d9e15c680fc917b95f687e14385ab1aeae1c6d974a1d66075290b22a826"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\n\\label{th:sos-variational-problem}\nIf there exist $r_0,\\ldots,r_s \\in \\R$ such that $r_0^2 - |(x,y,z)|^2 \\in \\mathcal{Q}(\\Gamma)$ and $r_i^2 - |(x,y)|^2 \\in \\mathcal{Q}(\\Lambda_i)$ for $i=1,\\ldots,s$, then $\\Lsos\\nearrow\\Lpdr$ as $\\nu\\to\\infty$.\n\\end{theorem}","units":[{"start":67532,"end":67807,"start_line":738,"end_line":741,"raw_tex":"\\begin{theorem}\n\\label{th:sos-variational-problem}\nIf there exist $r_0,\\ldots,r_s \\in \\R$ such that $r_0^2 - |(x,y,z)|^2 \\in \\mathcal{Q}(\\Gamma)$ and $r_i^2 - |(x,y)|^2 \\in \\mathcal{Q}(\\Lambda_i)$ for $i=1,\\ldots,s$, then $\\Lsos\\nearrow\\Lpdr$ as $\\nu\\to\\infty$.\n\\end{theorem}","sha256_utf8":"4d367b2b4889c345cf83046f766152c96e583713ccc4e4ef73cf84455312fcbe"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\section{Proof of \\texorpdfstring{\\cref{th:sos-variational-problem}}{Theorem \\ref{th:sos-variational-problem}}}\n\\label{ss:proof-sos-variational}\n%The inequalities $\\mathbb{L}_{1,{\\rm sos}} \\leq \\mathbb{L}_{2,{\\rm sos}} \\leq \\cdots \\leq \\mathbb{L}_\\infty \\leq \\mathcal{F}^*$ follow upon combining  \\cref{th:variational-porblem-lower-bound} with $\\mathbb{L}_{\\nu,{\\rm sos}} \\leq \\mathbb{L}_\\nu \\leq \\mathbb{L}_\\infty$ and $\\mathbb{L}_{\\nu,{\\rm sos}} \\leq \\mathbb{L}_{\\nu+1,{\\rm sos}}$, which holds because truncated quadratic modules satisfy $\\mathcal{Q}_\\nu \\subset \\mathcal{Q}_{\\nu+1}$. \n%To establish convergence as $\\nu \\to \\infty$ \nIt suffices to prove that, for any $\\varepsilon>0$, there exists $\\nu$ such that \n%\\begin{equation}\n%\\label{eq:epsilon-suboptimality-target-VP}\n%\\mathbb{L}_{\\nu,{\\rm sos}} \\geq \\mathbb{L}_{\\infty} - 2\\varepsilon.\n%\\end{equation} \n$\\Lsos \\geq \\Lpdr - 2\\varepsilon.$\n%\nFix any feasible tuple $(\\eta^\\varepsilon, \\vphi^\\varepsilon,h^\\varepsilon,\\ell_1^\\varepsilon,\\ldots,\\ell_s^\\varepsilon)$ for~\\cref{eq:relaxed-dual} that satisfies \n%\n\\begin{equation}\n\\label{eq:epsilon-suboptimality-nonpoly-VP}\n\\int_{\\Omega} h^\\varepsilon(x) \\, \\dx + \\sum_{i=1}^s \\int_{\\partial\\Omega_i} \\ell_i^\\varepsilon(x) \\,\\dbound \\geq \\Lpdr - \\varepsilon.\n\\end{equation}\n%\nWe will use $\\vphi^\\varepsilon,h^\\varepsilon,\\ell_1^\\varepsilon,\\ldots,\\ell_s^\\varepsilon$ to construct polynomials $\\vphi,h,q_1,\\ldots,q_s$ such that the tuple $(\\eta^\\varepsilon, \\vphi,h,q_1,\\ldots,q_s)$ is feasible for~\\cref{eq:vp-SOS} for sufficiently large $\\nu$ and achieves an objective value larger than $\\Lpdr - 2\\varepsilon$. This will imply $\\Lsos \\geq \\Lpdr - 2\\varepsilon$.\n\nBy assumption, the domain $\\Omega$ and the boundary sections $\\partial\\Omega_i$ are compact and $\\abs{\\nabla g_i}$ is strictly positive on $\\partial\\Omega_i$. Moreover, the $(x,y)$ projection of the compact set $\\Gamma$, denoted $\\Gamma_{xy}$ below, must be a compact subset of $\\R^n\\times\\R^m$. Since polynomials are dense in the space of $t$-times continuously differentiable functions on compact sets for all $t$, we can find polynomial $\\vphi$, $h$, $q_1$, $\\ldots$, $q_s$ that satisfy\n%\n\\begin{subequations}\n\t\\begin{align}\n\t\\label{eq:poly-approx-1}\n\t\\max_{i=1,\\ldots,n} \\left\\| \\vphi_i - \\vphi^\\varepsilon_i \\right\\|_{C^1(\\Gamma_{xy})} &\\leq \\beta \\varepsilon,\\\\\n\t\\label{eq:poly-approx-2}\n\t\\left\\| h - h^\\varepsilon + \\alpha_0 \\varepsilon \\right\\|_{C(\\Omega)} &\\leq \\beta \\varepsilon,\\\\\n\t\\label{eq:poly-approx-3}\n%\t\\left\\| q_i - \\frac{\\ell^\\varepsilon_i}{\\abs{\\nabla g_i}} + \\frac{\\alpha_i\\varepsilon}{\\abs{\\nabla g_i}^2} \\right\\|_{C(\\partial\\Omega_i)} &\\leq \\beta \\varepsilon,\n\t\\left\\| q_i - \\ell^\\varepsilon_i \\abs{\\nabla g_i}^{-1} + \\alpha_i\\varepsilon \\abs{\\nabla g_i}^{-2} \\right\\|_{C(\\partial\\Omega_i)} &\\leq \\beta \\varepsilon,\n\t\\end{align}\n\\end{subequations}\n%\nwhere $\\alpha_0,\\ldots,\\alpha_s$ and $\\beta$ are strictly positive constants to be determined later.\n\nOmitting all function arguments to simplify the notation, we may estimate\n%\n\\begin{align}\n\\mathcal{D}\\vphi^\\varepsilon - \\mathcal{D}\\vphi\n&= \\sum_{i=1}^n \\frac{\\partial}{\\partial x_i} (\\vphi_i^\\varepsilon - \\vphi_i) +   \\sum_{i=1}^n \\sum_{j=1}^m \\frac{\\partial}{\\partial y_i} (\\vphi_i^\\varepsilon - \\vphi_i) z_{ji}\\\\\\nonumber\n&\\leq \\bigg( n + \\sum_{i=1}^n \\sum_{j=1}^m \\max_{\\Gamma} \\vert z_{ji} \\vert \\bigg) \\beta \\varepsilon \n\\qquad \\forall (x,y,z) \\in \\Gamma.\n\\end{align}\n%\nThus,\n%\n\\begin{align}\nf + \\mathcal{D}\\vphi - \\eta^\\varepsilon a - h\n&\\geq \nf + \\mathcal{D}\\vphi^\\varepsilon - \\eta^\\varepsilon a - h^\\varepsilon \n+ \\alpha_0\\varepsilon \n- \\bigg( 1 + n + \\sum_{i,j} \\max_{\\Gamma} |z_{ji}|\\bigg) \\beta\\varepsilon\n\\\\\\nonumber\n%&\\geq \\alpha_0 \\varepsilon - \\bigg( 1 + n + \\sum_{i,j} \\max_{\\Gamma} |z_{ji}|\\bigg) \\beta\\varepsilon\n%\\\\\n&\\geq  \\alpha_0 \\varepsilon - C_0 \\beta\\varepsilon\n\\qquad \\forall (x,y,z) \\in \\Gamma,\n\\end{align}\n%\nwhere $C_0:= 1 + n + \\sum_{i,j} \\max_{\\Gamma} |z_{ji}|$ is a positive constant that does not depend on $\\alpha_0$, $\\beta$ or $\\varepsilon$.\nThe second inequality holds because $(\\eta^\\varepsilon, \\vphi^\\varepsilon,h^\\varepsilon,\\ell_1^\\varepsilon,\\ldots,\\ell_s^\\varepsilon)$ is feasible for~\\cref{eq:relaxed-dual}. Similarly, on each $\\Lambda_i$ the inequality $\\vphi \\cdot \\nabla g_i - q_i \\abs{\\nabla g_i}^2 \\geq \n\\alpha_i \\varepsilon - C_i \\beta \\varepsilon$ holds for some positive constant $C_i$ that depends only on the particular $\\partial\\Omega_i$ and $g_i$ under consideration, not on $\\alpha_i$, $\\beta$ or $\\varepsilon$. The details are omitted for brevity.\n\nNow choose $\\alpha_0 = 2C_0 \\beta$ and $\\alpha_i = 2C_i \\beta$ for all $i=1,\\ldots,s$ to obtain\n%\n\\begin{subequations}\n\t\\begin{align}\n\tf + \\mathcal{D}\\vphi - \\eta^\\varepsilon a - h &\\geq C_0\\beta \\varepsilon \\quad \\text{ on } \\Gamma,\\\\\n\t\\vphi \\cdot \\nabla g_i - q_i \\abs{\\nabla g_i}^2 &\\geq C_i\\beta \\varepsilon \\quad \\text{ on } \\Lambda_i, \\quad i = 1,\\ldots,s.\n\t\\end{align}\n\\end{subequations}\n%\nThe right-hand sides of these inequalities are strictly positive and, by assumption, $r_0^2 - |(x,y,z)|^2 \\in \\mathcal{Q}(\\Gamma)$ and $r_i^2 - |(x,y)|^2 \\in \\mathcal{Q}(\\Lambda_i)$, therefore Putinar's Positivstellensatz~\\cite{Putinar} guarantees that $f + \\mathcal{D}\\vphi - \\eta^\\varepsilon a - h \\in \\mathcal{Q}_\\nu(\\Gamma)$ and $\\vphi \\cdot \\nabla g_i - q_i \\abs{\\nabla g_i}^2 \\in \\mathcal{Q}_\\nu(\\Lambda_i)$ for a sufficiently large integer $\\nu$. In other words, the tuple $(\\eta^\\varepsilon,\\vphi,h,q_1,\\ldots,q_s)$ is feasible for the SOS program~\\cref{eq:vp-SOS}. Then, we must have\n%\n\\ifx\\SIOPT\\undefined\n% not the SIOPT version, do nothing!\n\\else\n% We have defined the \\SIOPT variable, but is it really the SIOPT version? Another \\if statement...\n\\if\\SIOPT1\n\\enlargethispage{\\baselineskip}\n\\fi\n\\fi\n\\begin{align}\n\\label{eq:epsilon-suboptimality-final-estimate-VP}\n\\Lsos\n&\\geq \\int_{\\Omega} h \\, \\dx + \\sum_{i=1}^s \\int_{\\partial\\Omega_i} q_i \\abs{\\nabla g_i} \\,\\dbound\n\\\\\\nonumber\n&\\geq \\int_{\\Omega} h^\\varepsilon\\, \\dx \n- (\\alpha_0+\\beta)\\vert\\Omega\\vert \\varepsilon\n+ \\sum_{i=1}^s \\int_{\\partial\\Omega_i} \n\t\\ell_i^\\varepsilon \n\t-\\beta\\varepsilon \\abs{\\nabla g_i}\n\t- \\alpha_i \\varepsilon\\abs{\\nabla g_i}^{-1}\n\\,\\dbound\n\\\\\\nonumber\n&\\geq \\Lpdr - \\bigg[ 1+ (2C_0+1)\\vert\\Omega\\vert \\beta + \\beta \\sum_{i=1}^s \\int_{\\partial\\Omega_i} \n\\abs{\\nabla g_i}\n+2C_i \\abs{\\nabla g_i}^{-1}\n\\,\\dbound\n\\bigg] \\varepsilon.\n\\end{align}\nThe second inequality follows from elementary estimates based on~\\cref{eq:poly-approx-1,eq:poly-approx-2,eq:poly-approx-3}, while the last one follows from~\\cref{eq:epsilon-suboptimality-nonpoly-VP} and our choices for $\\alpha_0,\\ldots,\\alpha_s$. Choosing a sufficiently small positive value for $\\beta$ \n%yields~\\cref{eq:epsilon-suboptimality-target-VP}, as desired.\nyields $\\Lsos \\geq \\Lpdr - 2\\varepsilon$, as desired.\n\n%\\blue{[[AC: Is $\\Linf$ looking confusing here? Shall we replace it with $\\cL_{\\nu=\\infty}$ instead?]]}\n\n\n\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n","units":[{"start":113703,"end":120657,"start_line":1251,"end_line":1361,"raw_tex":"\\section{Proof of \\texorpdfstring{\\cref{th:sos-variational-problem}}{Theorem \\ref{th:sos-variational-problem}}}\n\\label{ss:proof-sos-variational}\n%The inequalities $\\mathbb{L}_{1,{\\rm sos}} \\leq \\mathbb{L}_{2,{\\rm sos}} \\leq \\cdots \\leq \\mathbb{L}_\\infty \\leq \\mathcal{F}^*$ follow upon combining  \\cref{th:variational-porblem-lower-bound} with $\\mathbb{L}_{\\nu,{\\rm sos}} \\leq \\mathbb{L}_\\nu \\leq \\mathbb{L}_\\infty$ and $\\mathbb{L}_{\\nu,{\\rm sos}} \\leq \\mathbb{L}_{\\nu+1,{\\rm sos}}$, which holds because truncated quadratic modules satisfy $\\mathcal{Q}_\\nu \\subset \\mathcal{Q}_{\\nu+1}$. \n%To establish convergence as $\\nu \\to \\infty$ \nIt suffices to prove that, for any $\\varepsilon>0$, there exists $\\nu$ such that \n%\\begin{equation}\n%\\label{eq:epsilon-suboptimality-target-VP}\n%\\mathbb{L}_{\\nu,{\\rm sos}} \\geq \\mathbb{L}_{\\infty} - 2\\varepsilon.\n%\\end{equation} \n$\\Lsos \\geq \\Lpdr - 2\\varepsilon.$\n%\nFix any feasible tuple $(\\eta^\\varepsilon, \\vphi^\\varepsilon,h^\\varepsilon,\\ell_1^\\varepsilon,\\ldots,\\ell_s^\\varepsilon)$ for~\\cref{eq:relaxed-dual} that satisfies \n%\n\\begin{equation}\n\\label{eq:epsilon-suboptimality-nonpoly-VP}\n\\int_{\\Omega} h^\\varepsilon(x) \\, \\dx + \\sum_{i=1}^s \\int_{\\partial\\Omega_i} \\ell_i^\\varepsilon(x) \\,\\dbound \\geq \\Lpdr - \\varepsilon.\n\\end{equation}\n%\nWe will use $\\vphi^\\varepsilon,h^\\varepsilon,\\ell_1^\\varepsilon,\\ldots,\\ell_s^\\varepsilon$ to construct polynomials $\\vphi,h,q_1,\\ldots,q_s$ such that the tuple $(\\eta^\\varepsilon, \\vphi,h,q_1,\\ldots,q_s)$ is feasible for~\\cref{eq:vp-SOS} for sufficiently large $\\nu$ and achieves an objective value larger than $\\Lpdr - 2\\varepsilon$. This will imply $\\Lsos \\geq \\Lpdr - 2\\varepsilon$.\n\nBy assumption, the domain $\\Omega$ and the boundary sections $\\partial\\Omega_i$ are compact and $\\abs{\\nabla g_i}$ is strictly positive on $\\partial\\Omega_i$. Moreover, the $(x,y)$ projection of the compact set $\\Gamma$, denoted $\\Gamma_{xy}$ below, must be a compact subset of $\\R^n\\times\\R^m$. Since polynomials are dense in the space of $t$-times continuously differentiable functions on compact sets for all $t$, we can find polynomial $\\vphi$, $h$, $q_1$, $\\ldots$, $q_s$ that satisfy\n%\n\\begin{subequations}\n\t\\begin{align}\n\t\\label{eq:poly-approx-1}\n\t\\max_{i=1,\\ldots,n} \\left\\| \\vphi_i - \\vphi^\\varepsilon_i \\right\\|_{C^1(\\Gamma_{xy})} &\\leq \\beta \\varepsilon,\\\\\n\t\\label{eq:poly-approx-2}\n\t\\left\\| h - h^\\varepsilon + \\alpha_0 \\varepsilon \\right\\|_{C(\\Omega)} &\\leq \\beta \\varepsilon,\\\\\n\t\\label{eq:poly-approx-3}\n%\t\\left\\| q_i - \\frac{\\ell^\\varepsilon_i}{\\abs{\\nabla g_i}} + \\frac{\\alpha_i\\varepsilon}{\\abs{\\nabla g_i}^2} \\right\\|_{C(\\partial\\Omega_i)} &\\leq \\beta \\varepsilon,\n\t\\left\\| q_i - \\ell^\\varepsilon_i \\abs{\\nabla g_i}^{-1} + \\alpha_i\\varepsilon \\abs{\\nabla g_i}^{-2} \\right\\|_{C(\\partial\\Omega_i)} &\\leq \\beta \\varepsilon,\n\t\\end{align}\n\\end{subequations}\n%\nwhere $\\alpha_0,\\ldots,\\alpha_s$ and $\\beta$ are strictly positive constants to be determined later.\n\nOmitting all function arguments to simplify the notation, we may estimate\n%\n\\begin{align}\n\\mathcal{D}\\vphi^\\varepsilon - \\mathcal{D}\\vphi\n&= \\sum_{i=1}^n \\frac{\\partial}{\\partial x_i} (\\vphi_i^\\varepsilon - \\vphi_i) +   \\sum_{i=1}^n \\sum_{j=1}^m \\frac{\\partial}{\\partial y_i} (\\vphi_i^\\varepsilon - \\vphi_i) z_{ji}\\\\\\nonumber\n&\\leq \\bigg( n + \\sum_{i=1}^n \\sum_{j=1}^m \\max_{\\Gamma} \\vert z_{ji} \\vert \\bigg) \\beta \\varepsilon \n\\qquad \\forall (x,y,z) \\in \\Gamma.\n\\end{align}\n%\nThus,\n%\n\\begin{align}\nf + \\mathcal{D}\\vphi - \\eta^\\varepsilon a - h\n&\\geq \nf + \\mathcal{D}\\vphi^\\varepsilon - \\eta^\\varepsilon a - h^\\varepsilon \n+ \\alpha_0\\varepsilon \n- \\bigg( 1 + n + \\sum_{i,j} \\max_{\\Gamma} |z_{ji}|\\bigg) \\beta\\varepsilon\n\\\\\\nonumber\n%&\\geq \\alpha_0 \\varepsilon - \\bigg( 1 + n + \\sum_{i,j} \\max_{\\Gamma} |z_{ji}|\\bigg) \\beta\\varepsilon\n%\\\\\n&\\geq  \\alpha_0 \\varepsilon - C_0 \\beta\\varepsilon\n\\qquad \\forall (x,y,z) \\in \\Gamma,\n\\end{align}\n%\nwhere $C_0:= 1 + n + \\sum_{i,j} \\max_{\\Gamma} |z_{ji}|$ is a positive constant that does not depend on $\\alpha_0$, $\\beta$ or $\\varepsilon$.\nThe second inequality holds because $(\\eta^\\varepsilon, \\vphi^\\varepsilon,h^\\varepsilon,\\ell_1^\\varepsilon,\\ldots,\\ell_s^\\varepsilon)$ is feasible for~\\cref{eq:relaxed-dual}. Similarly, on each $\\Lambda_i$ the inequality $\\vphi \\cdot \\nabla g_i - q_i \\abs{\\nabla g_i}^2 \\geq \n\\alpha_i \\varepsilon - C_i \\beta \\varepsilon$ holds for some positive constant $C_i$ that depends only on the particular $\\partial\\Omega_i$ and $g_i$ under consideration, not on $\\alpha_i$, $\\beta$ or $\\varepsilon$. The details are omitted for brevity.\n\nNow choose $\\alpha_0 = 2C_0 \\beta$ and $\\alpha_i = 2C_i \\beta$ for all $i=1,\\ldots,s$ to obtain\n%\n\\begin{subequations}\n\t\\begin{align}\n\tf + \\mathcal{D}\\vphi - \\eta^\\varepsilon a - h &\\geq C_0\\beta \\varepsilon \\quad \\text{ on } \\Gamma,\\\\\n\t\\vphi \\cdot \\nabla g_i - q_i \\abs{\\nabla g_i}^2 &\\geq C_i\\beta \\varepsilon \\quad \\text{ on } \\Lambda_i, \\quad i = 1,\\ldots,s.\n\t\\end{align}\n\\end{subequations}\n%\nThe right-hand sides of these inequalities are strictly positive and, by assumption, $r_0^2 - |(x,y,z)|^2 \\in \\mathcal{Q}(\\Gamma)$ and $r_i^2 - |(x,y)|^2 \\in \\mathcal{Q}(\\Lambda_i)$, therefore Putinar's Positivstellensatz~\\cite{Putinar} guarantees that $f + \\mathcal{D}\\vphi - \\eta^\\varepsilon a - h \\in \\mathcal{Q}_\\nu(\\Gamma)$ and $\\vphi \\cdot \\nabla g_i - q_i \\abs{\\nabla g_i}^2 \\in \\mathcal{Q}_\\nu(\\Lambda_i)$ for a sufficiently large integer $\\nu$. In other words, the tuple $(\\eta^\\varepsilon,\\vphi,h,q_1,\\ldots,q_s)$ is feasible for the SOS program~\\cref{eq:vp-SOS}. Then, we must have\n%\n\\ifx\\SIOPT\\undefined\n% not the SIOPT version, do nothing!\n\\else\n% We have defined the \\SIOPT variable, but is it really the SIOPT version? Another \\if statement...\n\\if\\SIOPT1\n\\enlargethispage{\\baselineskip}\n\\fi\n\\fi\n\\begin{align}\n\\label{eq:epsilon-suboptimality-final-estimate-VP}\n\\Lsos\n&\\geq \\int_{\\Omega} h \\, \\dx + \\sum_{i=1}^s \\int_{\\partial\\Omega_i} q_i \\abs{\\nabla g_i} \\,\\dbound\n\\\\\\nonumber\n&\\geq \\int_{\\Omega} h^\\varepsilon\\, \\dx \n- (\\alpha_0+\\beta)\\vert\\Omega\\vert \\varepsilon\n+ \\sum_{i=1}^s \\int_{\\partial\\Omega_i} \n\t\\ell_i^\\varepsilon \n\t-\\beta\\varepsilon \\abs{\\nabla g_i}\n\t- \\alpha_i \\varepsilon\\abs{\\nabla g_i}^{-1}\n\\,\\dbound\n\\\\\\nonumber\n&\\geq \\Lpdr - \\bigg[ 1+ (2C_0+1)\\vert\\Omega\\vert \\beta + \\beta \\sum_{i=1}^s \\int_{\\partial\\Omega_i} \n\\abs{\\nabla g_i}\n+2C_i \\abs{\\nabla g_i}^{-1}\n\\,\\dbound\n\\bigg] \\varepsilon.\n\\end{align}\nThe second inequality follows from elementary estimates based on~\\cref{eq:poly-approx-1,eq:poly-approx-2,eq:poly-approx-3}, while the last one follows from~\\cref{eq:epsilon-suboptimality-nonpoly-VP} and our choices for $\\alpha_0,\\ldots,\\alpha_s$. Choosing a sufficiently small positive value for $\\beta$ \n%yields~\\cref{eq:epsilon-suboptimality-target-VP}, as desired.\nyields $\\Lsos \\geq \\Lpdr - 2\\varepsilon$, as desired.\n\n%\\blue{[[AC: Is $\\Linf$ looking confusing here? Shall we replace it with $\\cL_{\\nu=\\infty}$ instead?]]}\n\n\n\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n","sha256_utf8":"ac11492ce23485be2f5ef150306f93839cf0be6dc873da7c9d68250ee2e86154"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"紧集上SOS松弛收敛","source_units":[{"start":67532,"end":67807,"start_line":738,"end_line":741,"raw_tex":"\\begin{theorem}\n\\label{th:sos-variational-problem}\nIf there exist $r_0,\\ldots,r_s \\in \\R$ such that $r_0^2 - |(x,y,z)|^2 \\in \\mathcal{Q}(\\Gamma)$ and $r_i^2 - |(x,y)|^2 \\in \\mathcal{Q}(\\Lambda_i)$ for $i=1,\\ldots,s$, then $\\Lsos\\nearrow\\Lpdr$ as $\\nu\\to\\infty$.\n\\end{theorem}","sha256_utf8":"4d367b2b4889c345cf83046f766152c96e583713ccc4e4ef73cf84455312fcbe"}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","title":"二次模的球约束","source_units":[{"start":67583,"end":67794,"start_line":740,"end_line":740,"raw_tex":"If there exist $r_0,\\ldots,r_s \\in \\R$ such that $r_0^2 - |(x,y,z)|^2 \\in \\mathcal{Q}(\\Gamma)$ and $r_i^2 - |(x,y)|^2 \\in \\mathcal{Q}(\\Lambda_i)$ for $i=1,\\ldots,s$, then $\\Lsos\\nearrow\\Lpdr$ as $\\nu\\to\\infty$.\n","sha256_utf8":"fa42c0341953121dcd549cc6046fd58c81740ade44babccb2c834680345365e6"}]},{"id":"S003","semantic_type":"proof","kind":"proof","title":"SOS收敛完整证明","source_units":[{"start":113703,"end":120657,"start_line":1251,"end_line":1361,"raw_tex":"\\section{Proof of \\texorpdfstring{\\cref{th:sos-variational-problem}}{Theorem \\ref{th:sos-variational-problem}}}\n\\label{ss:proof-sos-variational}\n%The inequalities $\\mathbb{L}_{1,{\\rm sos}} \\leq \\mathbb{L}_{2,{\\rm sos}} \\leq \\cdots \\leq \\mathbb{L}_\\infty \\leq \\mathcal{F}^*$ follow upon combining  \\cref{th:variational-porblem-lower-bound} with $\\mathbb{L}_{\\nu,{\\rm sos}} \\leq \\mathbb{L}_\\nu \\leq \\mathbb{L}_\\infty$ and $\\mathbb{L}_{\\nu,{\\rm sos}} \\leq \\mathbb{L}_{\\nu+1,{\\rm sos}}$, which holds because truncated quadratic modules satisfy $\\mathcal{Q}_\\nu \\subset \\mathcal{Q}_{\\nu+1}$. \n%To establish convergence as $\\nu \\to \\infty$ \nIt suffices to prove that, for any $\\varepsilon>0$, there exists $\\nu$ such that \n%\\begin{equation}\n%\\label{eq:epsilon-suboptimality-target-VP}\n%\\mathbb{L}_{\\nu,{\\rm sos}} \\geq \\mathbb{L}_{\\infty} - 2\\varepsilon.\n%\\end{equation} \n$\\Lsos \\geq \\Lpdr - 2\\varepsilon.$\n%\nFix any feasible tuple $(\\eta^\\varepsilon, \\vphi^\\varepsilon,h^\\varepsilon,\\ell_1^\\varepsilon,\\ldots,\\ell_s^\\varepsilon)$ for~\\cref{eq:relaxed-dual} that satisfies \n%\n\\begin{equation}\n\\label{eq:epsilon-suboptimality-nonpoly-VP}\n\\int_{\\Omega} h^\\varepsilon(x) \\, \\dx + \\sum_{i=1}^s \\int_{\\partial\\Omega_i} \\ell_i^\\varepsilon(x) \\,\\dbound \\geq \\Lpdr - \\varepsilon.\n\\end{equation}\n%\nWe will use $\\vphi^\\varepsilon,h^\\varepsilon,\\ell_1^\\varepsilon,\\ldots,\\ell_s^\\varepsilon$ to construct polynomials $\\vphi,h,q_1,\\ldots,q_s$ such that the tuple $(\\eta^\\varepsilon, \\vphi,h,q_1,\\ldots,q_s)$ is feasible for~\\cref{eq:vp-SOS} for sufficiently large $\\nu$ and achieves an objective value larger than $\\Lpdr - 2\\varepsilon$. This will imply $\\Lsos \\geq \\Lpdr - 2\\varepsilon$.\n\nBy assumption, the domain $\\Omega$ and the boundary sections $\\partial\\Omega_i$ are compact and $\\abs{\\nabla g_i}$ is strictly positive on $\\partial\\Omega_i$. Moreover, the $(x,y)$ projection of the compact set $\\Gamma$, denoted $\\Gamma_{xy}$ below, must be a compact subset of $\\R^n\\times\\R^m$. Since polynomials are dense in the space of $t$-times continuously differentiable functions on compact sets for all $t$, we can find polynomial $\\vphi$, $h$, $q_1$, $\\ldots$, $q_s$ that satisfy\n%\n\\begin{subequations}\n\t\\begin{align}\n\t\\label{eq:poly-approx-1}\n\t\\max_{i=1,\\ldots,n} \\left\\| \\vphi_i - \\vphi^\\varepsilon_i \\right\\|_{C^1(\\Gamma_{xy})} &\\leq \\beta \\varepsilon,\\\\\n\t\\label{eq:poly-approx-2}\n\t\\left\\| h - h^\\varepsilon + \\alpha_0 \\varepsilon \\right\\|_{C(\\Omega)} &\\leq \\beta \\varepsilon,\\\\\n\t\\label{eq:poly-approx-3}\n%\t\\left\\| q_i - \\frac{\\ell^\\varepsilon_i}{\\abs{\\nabla g_i}} + \\frac{\\alpha_i\\varepsilon}{\\abs{\\nabla g_i}^2} \\right\\|_{C(\\partial\\Omega_i)} &\\leq \\beta \\varepsilon,\n\t\\left\\| q_i - \\ell^\\varepsilon_i \\abs{\\nabla g_i}^{-1} + \\alpha_i\\varepsilon \\abs{\\nabla g_i}^{-2} \\right\\|_{C(\\partial\\Omega_i)} &\\leq \\beta \\varepsilon,\n\t\\end{align}\n\\end{subequations}\n%\nwhere $\\alpha_0,\\ldots,\\alpha_s$ and $\\beta$ are strictly positive constants to be determined later.\n\nOmitting all function arguments to simplify the notation, we may estimate\n%\n\\begin{align}\n\\mathcal{D}\\vphi^\\varepsilon - \\mathcal{D}\\vphi\n&= \\sum_{i=1}^n \\frac{\\partial}{\\partial x_i} (\\vphi_i^\\varepsilon - \\vphi_i) +   \\sum_{i=1}^n \\sum_{j=1}^m \\frac{\\partial}{\\partial y_i} (\\vphi_i^\\varepsilon - \\vphi_i) z_{ji}\\\\\\nonumber\n&\\leq \\bigg( n + \\sum_{i=1}^n \\sum_{j=1}^m \\max_{\\Gamma} \\vert z_{ji} \\vert \\bigg) \\beta \\varepsilon \n\\qquad \\forall (x,y,z) \\in \\Gamma.\n\\end{align}\n%\nThus,\n%\n\\begin{align}\nf + \\mathcal{D}\\vphi - \\eta^\\varepsilon a - h\n&\\geq \nf + \\mathcal{D}\\vphi^\\varepsilon - \\eta^\\varepsilon a - h^\\varepsilon \n+ \\alpha_0\\varepsilon \n- \\bigg( 1 + n + \\sum_{i,j} \\max_{\\Gamma} |z_{ji}|\\bigg) \\beta\\varepsilon\n\\\\\\nonumber\n%&\\geq \\alpha_0 \\varepsilon - \\bigg( 1 + n + \\sum_{i,j} \\max_{\\Gamma} |z_{ji}|\\bigg) \\beta\\varepsilon\n%\\\\\n&\\geq  \\alpha_0 \\varepsilon - C_0 \\beta\\varepsilon\n\\qquad \\forall (x,y,z) \\in \\Gamma,\n\\end{align}\n%\nwhere $C_0:= 1 + n + \\sum_{i,j} \\max_{\\Gamma} |z_{ji}|$ is a positive constant that does not depend on $\\alpha_0$, $\\beta$ or $\\varepsilon$.\nThe second inequality holds because $(\\eta^\\varepsilon, \\vphi^\\varepsilon,h^\\varepsilon,\\ell_1^\\varepsilon,\\ldots,\\ell_s^\\varepsilon)$ is feasible for~\\cref{eq:relaxed-dual}. Similarly, on each $\\Lambda_i$ the inequality $\\vphi \\cdot \\nabla g_i - q_i \\abs{\\nabla g_i}^2 \\geq \n\\alpha_i \\varepsilon - C_i \\beta \\varepsilon$ holds for some positive constant $C_i$ that depends only on the particular $\\partial\\Omega_i$ and $g_i$ under consideration, not on $\\alpha_i$, $\\beta$ or $\\varepsilon$. The details are omitted for brevity.\n\nNow choose $\\alpha_0 = 2C_0 \\beta$ and $\\alpha_i = 2C_i \\beta$ for all $i=1,\\ldots,s$ to obtain\n%\n\\begin{subequations}\n\t\\begin{align}\n\tf + \\mathcal{D}\\vphi - \\eta^\\varepsilon a - h &\\geq C_0\\beta \\varepsilon \\quad \\text{ on } \\Gamma,\\\\\n\t\\vphi \\cdot \\nabla g_i - q_i \\abs{\\nabla g_i}^2 &\\geq C_i\\beta \\varepsilon \\quad \\text{ on } \\Lambda_i, \\quad i = 1,\\ldots,s.\n\t\\end{align}\n\\end{subequations}\n%\nThe right-hand sides of these inequalities are strictly positive and, by assumption, $r_0^2 - |(x,y,z)|^2 \\in \\mathcal{Q}(\\Gamma)$ and $r_i^2 - |(x,y)|^2 \\in \\mathcal{Q}(\\Lambda_i)$, therefore Putinar's Positivstellensatz~\\cite{Putinar} guarantees that $f + \\mathcal{D}\\vphi - \\eta^\\varepsilon a - h \\in \\mathcal{Q}_\\nu(\\Gamma)$ and $\\vphi \\cdot \\nabla g_i - q_i \\abs{\\nabla g_i}^2 \\in \\mathcal{Q}_\\nu(\\Lambda_i)$ for a sufficiently large integer $\\nu$. In other words, the tuple $(\\eta^\\varepsilon,\\vphi,h,q_1,\\ldots,q_s)$ is feasible for the SOS program~\\cref{eq:vp-SOS}. Then, we must have\n%\n\\ifx\\SIOPT\\undefined\n% not the SIOPT version, do nothing!\n\\else\n% We have defined the \\SIOPT variable, but is it really the SIOPT version? Another \\if statement...\n\\if\\SIOPT1\n\\enlargethispage{\\baselineskip}\n\\fi\n\\fi\n\\begin{align}\n\\label{eq:epsilon-suboptimality-final-estimate-VP}\n\\Lsos\n&\\geq \\int_{\\Omega} h \\, \\dx + \\sum_{i=1}^s \\int_{\\partial\\Omega_i} q_i \\abs{\\nabla g_i} \\,\\dbound\n\\\\\\nonumber\n&\\geq \\int_{\\Omega} h^\\varepsilon\\, \\dx \n- (\\alpha_0+\\beta)\\vert\\Omega\\vert \\varepsilon\n+ \\sum_{i=1}^s \\int_{\\partial\\Omega_i} \n\t\\ell_i^\\varepsilon \n\t-\\beta\\varepsilon \\abs{\\nabla g_i}\n\t- \\alpha_i \\varepsilon\\abs{\\nabla g_i}^{-1}\n\\,\\dbound\n\\\\\\nonumber\n&\\geq \\Lpdr - \\bigg[ 1+ (2C_0+1)\\vert\\Omega\\vert \\beta + \\beta \\sum_{i=1}^s \\int_{\\partial\\Omega_i} \n\\abs{\\nabla g_i}\n+2C_i \\abs{\\nabla g_i}^{-1}\n\\,\\dbound\n\\bigg] \\varepsilon.\n\\end{align}\nThe second inequality follows from elementary estimates based on~\\cref{eq:poly-approx-1,eq:poly-approx-2,eq:poly-approx-3}, while the last one follows from~\\cref{eq:epsilon-suboptimality-nonpoly-VP} and our choices for $\\alpha_0,\\ldots,\\alpha_s$. Choosing a sufficiently small positive value for $\\beta$ \n%yields~\\cref{eq:epsilon-suboptimality-target-VP}, as desired.\nyields $\\Lsos \\geq \\Lpdr - 2\\varepsilon$, as desired.\n\n%\\blue{[[AC: Is $\\Linf$ looking confusing here? Shall we replace it with $\\cL_{\\nu=\\infty}$ instead?]]}\n\n\n\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n","sha256_utf8":"ac11492ce23485be2f5ef150306f93839cf0be6dc873da7c9d68250ee2e86154"}]},{"id":"S004","semantic_type":"problem_setting","kind":"problem_setting","title":"积分变分问题","source_units":[{"start":10122,"end":10281,"start_line":225,"end_line":230,"raw_tex":"\\begin{equation}\n\\mathcal{F}^*\n:= \\inf_{\\substack{u\\in W^{1,p} \\\\ +\\text{ constraints}}}\n\t\\int_\\Omega f(x,u,\\nabla u)\\,\\dx,\n\\label{eq:primal-vp}\n\\end{equation}","sha256_utf8":"4a06fd017440d0741dd483aaa4ee27c46973474df87c2f097d5fe9765f09ef6b"},{"start":10282,"end":10943,"start_line":231,"end_line":231,"raw_tex":"where $\\Omega\\subset\\R^n$ is open and bounded, $W^{1,p}$ is the Sobolev space of functions $u:\\Omega\\to\\R^m$ such that all components of $u$ and their weak derivatives are integrable in the Lebesgue space $L^p$ for some $p\\ge1$, and $\\nabla u:\\Omega\\to\\R^{m\\times n}$ is the gradient tensor defined in the weak sense. The integrand function $f$ can be any such that the integral is well defined for all admissible $u$, and, as specified in the next section, the constraints can include pointwise or integral conditions over $\\Omega$ or its boundary $\\partial\\Omega$. In general the infimum $\\cF^*$ might not be attained, meaning no global minimizer $u$ exists.\n","sha256_utf8":"8ebe5e0a948aa7d46167a04c5084f19d22946e21a9682966c435aaf12eabaf8b"}]},{"id":"S005","semantic_type":"assumption","kind":"assumption","title":"区域与约束数据","source_units":[{"start":26557,"end":27244,"start_line":321,"end_line":322,"raw_tex":"Consider the integral variational problem~\\cref{eq:primal-vp}, where $\\Omega$ is a bounded open Lipschitz domain. For simplicity we assume that the domain boundary is specified as a finite union, $\\partial\\Omega = \\cup_{i=1}^s \\partial\\Omega_i$, where each $\\partial\\Omega_i$ is a smooth compact manifold.\nWe assume the integrand function $f$ in~\\cref{eq:primal-vp} is continuous and gives a well defined integral over $\\Omega$ for all $u\\in W^{1,p}$ satisfying the constraints. For instance, in the absence of pointwise constraints that would ensure $u(x)$ and $\\nabla u(x)$ are uniformly bounded, we assume that $|f(x,y,z)|$ grows no faster than $|y|^p + |z|^p$ as $|y|,|z|\\to\\infty$.\n","sha256_utf8":"060013b67d3be985fa78129620b75fe6521206242ff62c0c933c3423a234aa86"},{"start":27676,"end":27953,"start_line":325,"end_line":325,"raw_tex":"For the purpose of illustration throughout this section, we suppose the constraints on $u$ in~\\cref{eq:primal-vp} include one integral inequality, one pointwise equality on $\\Omega$, and one equality on each boundary component $\\partial\\Omega_i$ (i.e, one boundary condition):\n","sha256_utf8":"09163329930276b31ad8b1cb3784c8f76e406f30cc7dfb17231bf42aaa60a8d6"},{"start":28005,"end":28240,"start_line":329,"end_line":338,"raw_tex":"\\begin{align}\n\t\\label{int-both}\n\t\\int_\\Omega a(x,u,\\nabla u) \\,\\dx &\\ge 0,\n\t\\\\\n\t\\label{pw-domain}\n\tc(x,u,\\nabla u)&= 0 \\quad x\\in\\Omega,\n\t\\\\\n\t\\label{pw-boundary}\n\td_i(x,u)&= 0 \\quad x\\in\\partial\\Omega_i,\\quad i=1,\\ldots,s.\n\t\\end{align}","sha256_utf8":"394088bb8eda0b10c94fba590d0b365f8587d9809a5aac3b990da50069b8dcac"},{"start":28262,"end":28381,"start_line":341,"end_line":341,"raw_tex":"We assume that the functions $a,c,d_i$ are continuous and that $a(x,u,\\nabla u)$ is integrable for all $u\\in W^{1,p}$.\n","sha256_utf8":"60ff425b146df358e088b3e81a28e864ae318ee08d5338100ae5391e3bb293c0"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"全散度算子","source_units":[{"start":13470,"end":13662,"start_line":246,"end_line":250,"raw_tex":"\\begin{equation}\n\\label{eq: D}\n\\cD\\varphi(x,y,z) := \\sum_{i=1}^n \\frac{\\partial\\varphi_i}{\\partial x_i}\n+ \\sum_{i=1}^n \\sum_{j=1}^m z_{ji}\\frac{\\partial\\varphi_i}{\\partial y_j}.\n\\end{equation}","sha256_utf8":"4602a9b28c0bbfac980615108c38a7eb06eb71413d314ac15eae902591d9b8e9"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"容许向量场与法向量","source_units":[{"start":31802,"end":32175,"start_line":376,"end_line":380,"raw_tex":"The constraint $v=\\nabla u$ is imposed using the divergence theorem for vector-valued functions $x \\mapsto \\vphi(x,u(x))$, where $\\varphi: \\overline{\\Omega}\\times\\mathbb{R}^m \\to \\mathbb{R}^n$ is continuously differentiable:\n\\beq\n\\label{eq:div-theorem}\n\\int_\\Omega\\diver_x\\vphi(x,u(x))\\,\\dx - \\sum_{i=1}^s \\int_{\\partial\\Omega_i}\\varphi(x,u)\\cdot n_i(x) \\,\\dbound = 0.\n\\eeq","sha256_utf8":"84be6cc1baa8ca636076b5b6c3996d7f5bf1bbe30dc87b81460537457545e44d"},{"start":32176,"end":33074,"start_line":381,"end_line":381,"raw_tex":"Here, $n_i(x)$ is the outward unit normal vector on $\\partial\\Omega_i$. Let $\\Phi$ denote the set of $\\vphi$ for which the above identity holds and for which the map $x \\mapsto \\varphi(x,u(x))$ is differentiable via the chain rule for all $u\\in W^{1,p}(\\Omega,\\R^m)$. (See~\\cite{Fantuzzi2022} for a more explicit characterization of $\\Phi$.) Since $\\cD\\vphi$ in~\\cref{eq: D} is defined in order to satisfy $\\diver_x\\vphi(x,u(x))=\\cD\\vphi(x,u,\\nabla u)$, the left-hand integrand in~\\cref{eq:div-theorem} can be replaced by $\\cD\\vphi(x,u,\\nabla u)$. If $v=\\nabla u$ weakly, therefore, identity~\\cref{eq:div-theorem} holds for all $\\vphi\\in\\Phi$ with the left-hand integrand replaced by $\\cD\\vphi(x,u,v)$. The proof of~\\cite[Lemma~1]{Korda2018} implies that the converse is also true, so the divergence theorem characterizes the weak gradient $\\nabla u$. The next Lemma summarizes these observations.\n","sha256_utf8":"cb1c96e723adef29975da860d735dcc220f733cafc89d9c5b6c0cb63b4c35833"}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","title":"逐点下确界松弛","source_units":[{"start":35515,"end":36315,"start_line":405,"end_line":418,"raw_tex":"\\begin{align}\n\\tag{\\normalsize\\theequation a}\n\\cL^*&\\ge\n\\sup_{\\substack{\\eta \\in \\mathbb{R}_+\\\\\\varphi \\in \\Phi}}\n\t\\inf_{\\substack{u\\in W^{1,p}\\\\v\\in L^p\\\\ \\text{(\\ref{eq:F-div-dual}c,d)}}}\\!\\!\n\t\\left\\{\\!\\int\\limits_\\Omega \\![f-\\eta a + \\mathcal{D}\\vphi](x,u,v)\\,\\dx -\\sum_{i=1}^s \\int\\limits_{\\partial\\Omega_i}\\!\\!\\varphi(x,u)\\cdot n_i(x)\\,\\dbound \\right\\}\n\\label{eq:div-dual}\n\\\\\n\\tag{\\normalsize\\theequation b}\n&\\ge\n\\sup_{\\substack{\\eta \\in \\mathbb{R}_+\\\\\\varphi \\in \\Phi}}\n\\left\\{\\! \\int\\limits_{\\Omega}\\!\\!\\!\\! \\inf_{\\substack{ y\\in \\R^m \\\\ z \\in \\R^{m\\times n}\\\\c(x,y,z)=0}}\\!\\!\\![f-\\eta a + \\mathcal{D}\\vphi](x,y,z)\\, \\dx+ \\sum_{i=1}^s \\int\\limits_{\\partial\\Omega_i}\\!\\! \\inf_{\\substack{y\\in\\R^m\\\\d_i(x,y)=0}}\\!\\!\\!\\!-\\varphi(x,y)\\cdot n_i(x)\\dbound\\right\\}.\n\\label{eq:relaxed-dual}\n\\end{align}","sha256_utf8":"50cec111b0bba00d1104e8dfbfc1280f0aa7b0cea752e4ddc19e5d74180fe334"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"逐点约束与可行集合","source_units":[{"start":36348,"end":36541,"start_line":422,"end_line":422,"raw_tex":"Finally, we bound the pointwise infima in~\\cref{eq:relaxed-dual} from below by the largest continuous functions $h\\in C(\\overline{\\Omega},\\R)$ and $\\ell_i\\in C(\\partial\\Omega_i,\\R)$ satisfying\n","sha256_utf8":"39946146ee921d735a05ddc745cde262ad1b973e57cb6447b2a5ef6c746e2751"},{"start":36600,"end":36885,"start_line":425,"end_line":430,"raw_tex":"\\begin{align}\n\t\\label{eq:pw-inequality-bulk}\n\tf(x,y,z) - \\eta\\, a(x,y,z) + \\mathcal{D}\\varphi(x,y,z)  - h(x) &\\geq 0 \\; \\text { on } \\Gamma,\\\\\n\t-\\varphi(x,y)\\cdot n_i(x) - \\ell_i(x) &\\geq 0  \\; \\text { on } \\Lambda_i, \\quad i=1,\\ldots,s,\n\t\\label{eq:pw-inequality-boundary}\n\t\\end{align}","sha256_utf8":"1de19e7d8a4e29cfcfb0dbb3244aa78b60ad2aa9b93999f96a8a7b62598172e0"},{"start":36905,"end":36999,"start_line":432,"end_line":432,"raw_tex":"where $\\Gamma$ and $\\Lambda_i$ are the sets on which the pointwise constraints are satisfied:\n","sha256_utf8":"cbdca12de2a47fa2f08e80fdd43166f8586fc36f536468cb13ccd82f3e7405a8"},{"start":37044,"end":37308,"start_line":435,"end_line":439,"raw_tex":"\\begin{align}\n\\label{eq:Gamma-def}\n\\Gamma &:= \\{(x,y,z) \\in \\Omega \\times \\mathbb{R}^m \\times \\mathbb{R}^{m \\times n}\\colon\\; c(x,y,z)= 0\\},\\\\\n\\Lambda_i &:= \\{(x,y) \\in \\partial\\Omega_i \\times \\mathbb{R}^m \\colon \\; d_i(x,y)= 0\\}. \\label{eq:Lambda-def}\n\\end{align}","sha256_utf8":"f81cd4e4032020b06a0dc327f3375ffbd28327649735af35faaf1dc38ccad5c3"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"逐点对偶松弛值","source_units":[{"start":37344,"end":37703,"start_line":444,"end_line":452,"raw_tex":"\\begin{equation}\n\\label{eq:pdr-2}\n\\cF^* \\geq \\cL^* \\geq \\Lpdr :=\n\\sup_{\\substack{ \\eta \\in \\mathbb{R}_+,\\,\\varphi \\in \\Phi,\\\\ h \\in C(\\overline{\\Omega}),\\, \\ell_i \\in C(\\partial\\Omega_i)\\\\ \\text{\\rm ~(\\ref{eq:relaxed-inequalities-all}a,b)} }} \\;\n\\left\\{\n\\int_\\Omega h(x) \\,\\dx\n+\\sum_{i=1}^s \\int_{\\partial\\Omega_i} \\ell_i(x) \\,\\dbound\n\\right\\},\n\\end{equation}","sha256_utf8":"07296c376b3e5291f827e067c47669693c94253d77723274310909ef9902eecf"}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","title":"多项式与半代数数据","source_units":[{"start":52096,"end":52725,"start_line":586,"end_line":590,"raw_tex":"We now consider the particular case in which the integral variational problem~\\cref{eq:primal-vp} or the integral-constrained optimization problem~\\cref{eq:integral-inequality-setup} has polynomial data, meaning that all functions appearing as integrands or in pointwise constraints are polynomial in $(x,u,\\nabla u)$, and that the domain $\\Omega$ can be specified by polynomials in $x$. Precisely, we assume that $\\overline\\Omega$ is a basic closed semialgebraic set defined by $s$ polynomial inequalities,\n\\begin{equation}\n\\overline\\Omega = \\{x \\in \\R^n:\\; g_1(x)\\ge 0,\\,\\ldots,\\,g_s(x)\\ge 0\\}.\n\\label{def-omega}\n\\end{equation}","sha256_utf8":"bb2abbba5a1a7324b8c3d63c15d2b74fa27135f76b276a1e8e4264b7d272d97a"},{"start":52726,"end":52933,"start_line":591,"end_line":595,"raw_tex":"We also assume that each $g_i$ vanishes identically on a single boundary component, so\n\\begin{align}\n\\partial\\Omega_i\n&= \\{x \\in \\R^n:\\;g_i(x)=0 \\text{ and } g_j(x)\\geq 0 \\text{ for } j\\neq i \\},\n\\end{align}","sha256_utf8":"41ce26ca826b52c7a4a7b51522975df6169e9d2296720aa60f9fd80e83ed5629"},{"start":52934,"end":53215,"start_line":596,"end_line":596,"raw_tex":"and that $\\nabla g_i$ does not vanish anywhere on $\\partial\\Omega_i$, so the outward unit normal vector is well defined as $n_i(x) = -\\nabla g_i/|\\nabla g_i|$. Then, the sets $\\Gamma$ and $\\Lambda_i$ defined in~\\cref{eq:Gamma-def,eq:Lambda-def} admit the semialgebraic definitions\n","sha256_utf8":"eadfb4a10f641c207fad36b05a315a3a95db0bedebfafcf8dd90283fe11f0ae2"},{"start":53270,"end":53521,"start_line":600,"end_line":608,"raw_tex":"\\begin{align}\n\\Gamma &= \\{(x,y,z) \\in \\R^n \\times \\R^m \\times \\R^{m\\times n}:\nc(x,y,z)= 0,\\,g_1(x)\\ge 0,\\ldots,g_s(x)\\geq 0\\},\n\\\\\n\\Lambda_i &= \\{(x,y) \\in \\R^n \\times \\R^m:\\,\nd_i(x,y) = 0,\\,g_i(x)=0,\\,\ng_j(x)\\geq 0 \\text{ for } j\\neq i\n\\}.\n\\end{align}","sha256_utf8":"6536f89559b6b7b977057bd34ee226636b33b0eaa58266048b69df9ccc8a2934"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"多项式搜索空间","source_units":[{"start":53917,"end":54183,"start_line":614,"end_line":619,"raw_tex":"For any positive integer $\\nu$, we can bound $\\Lpdr$ from below by restricting the maximization in~\\cref{eq:pdr-2} to \n\\begin{align}\n\\vphi &\\in \\Phi\\cap\\R^n[x,y]_\\nu, &\nh &\\in \\R[x]_\\nu, &\n\\ell_i &= q_i |\\nabla g_i| \\; \\text{ with } \\; q_i \\in \\R[x]_\\nu,\n\\end{align}","sha256_utf8":"74a21b747acd2f88d228762234562ca49d71e4182a54100b8b7b09e65bb84769"},{"start":54184,"end":54749,"start_line":620,"end_line":622,"raw_tex":"where $\\R^n[\\cdot]_\\nu$ is the space of $n$-dimensional vectors of polynomials in the bracketed variables whose degrees are at most $\\nu$. More generally, one can choose any finite vector space of polynomials, \nor use rational functions with fixed nonnegative denominators (see \\cref{rem:rational} below). The intersection with $\\Phi$\nexcludes polynomials $\\vphi(x,y)$ whose degree in $y$ is too large to guarantee integrability of $\\cD\\vphi(x,u,\\nabla u)$ for all $u\\in W^{1,p}$. Recalling that $n_i(x) = -\\nabla g_i/|\\nabla g_i|$ for each $i=1,\\ldots,s$, we find\n","sha256_utf8":"5226611a1bf4d2829b74ba53177f9218c9ae7f0f2fb45fa57810029ec5e56c86"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"有限维松弛与目标","source_units":[{"start":54749,"end":55169,"start_line":623,"end_line":634,"raw_tex":"\\begin{equation}\n\\label{eq:vp-finite-dimensional}\n\\Lpdr\\ge\n\\Lnu :=\\!\\!\\!\\!\\!\\!\\!\n\\sup_{ \\substack{\\vphi \\in \\Phi \\cap \\R^n[x,y]_\\nu \\\\h, q_i \\in \\R[x]_\\nu, \\, \\eta \\in \\R_+} }\n\\!\\!\\!\\!\\!\\!L(h,q_1,\\ldots,q_s)\n\\;\\; \\text{s.t.}\n\\begin{array}[t]{l}\nf - \\eta a + \\mathcal{D}\\varphi - h \\geq 0  \\,\\text { on } \\Gamma,\\\\\n\\varphi\\cdot \\nabla g_i - q_i|\\nabla g_i|^2 \\geq 0  \\, \\text { on } \\Lambda_i, \n\\end{array}\n\\end{equation}","sha256_utf8":"377d7e4e80733bc1c43d9486e30605dfbccd19126382d5e4c95a0a4bbac8cee4"},{"start":55170,"end":55360,"start_line":635,"end_line":639,"raw_tex":"where the maximization objective is\n\\beq\n\\label{eq:L-objective}\nL(h,q_1,\\ldots,q_s) : = \\int_{\\Omega} h(x) \\, \\dx + \\sum_{i=1}^s \\int_{\\partial\\Omega_i} q_i(x)|\\nabla g_i(x)| \\,\\dbound.\n\\eeq","sha256_utf8":"d62d2ff127f4899095015a69d5eceeba5e638173ca253e7b01b19976c72ab15a"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"平方和与二次模","source_units":[{"start":55906,"end":56427,"start_line":642,"end_line":642,"raw_tex":"Problem~\\cref{eq:vp-finite-dimensional} is finite-dimensional but still intractable because pointwise polynomial inequalities are NP-hard to verify in general~\\cite{Murty1987}. For this reason, we follow a standard approach~\\cite{Parrilo2003,Lasserre2001} and strengthen all polynomial nonnegativity constraints into weighted SOS constraints. Letting $\\Sigma[\\cdot]$ denote the subset of nonnegative polynomials in the bracketed variables that can be written as sums of squares of other polynomials, we consider the sets\n","sha256_utf8":"63c90c500716b18e1465d57bdd586b309d88f6af78d92f7b77bd59cf45bee390"},{"start":56470,"end":56798,"start_line":645,"end_line":648,"raw_tex":"\\begin{align}\n\t\\mathcal{Q}(\\Gamma) &:= \\bigg\\{\\rho c + \\sigma_0 + \\sum_{i=1}^s g_i \\sigma_i:\\; \\sigma_k \\in \\Sigma[x,y,z], \\,\\rho \\in \\R[x,y,z] \\bigg\\}, \\\\\n\t\\mathcal{Q}(\\Lambda_i) &:= \\bigg\\{\\rho_1 d_i + \\rho_2 g_i  + \\sigma_0 + \\sum_{j\\neq i} g_i \\sigma_i:\\; \\sigma_k \\in \\Sigma[x,y], \\,\\rho_j \\in \\R[x,y] \\bigg\\}.\n\t\\end{align}","sha256_utf8":"1232c204dc27fb2d220137fc7db6ec10f93d30c912dabac2d972e7fcdd935b5d"}]},{"id":"S015","semantic_type":"definition","kind":"definition","title":"有限次数二次模","source_units":[{"start":57227,"end":57418,"start_line":651,"end_line":655,"raw_tex":"\\begin{align}\n\\label{eq:quad-mods-nu}\n\t\\mathcal{Q}_\\nu(\\Gamma) &= \\mathcal{Q}(\\Gamma) \\cap \\R[x,y,z]_\\nu, &\n\t\\mathcal{Q}_\\nu(\\Lambda_i) &= \\mathcal{Q}(\\Lambda_i) \\cap \\R[x,y]_\\nu.\n\\end{align}","sha256_utf8":"f5f6c8af06120755a89b8801871780df8ea5b02a0c1f2805bfe2023c9ed82616"},{"start":57419,"end":57618,"start_line":656,"end_line":656,"raw_tex":"Requiring a polynomial to belong to a finite-degree set such as $\\cQ_\\nu(\\Gamma)$ or $\\cQ_\\nu(\\Lambda_i)$ is a \\emph{weighted SOS} condition, which can be included as a constraint in an SOS program.\n","sha256_utf8":"e941bd7f3cee47fd4f3d6d7fe1c6a9ab9dd9455ba21e3bf1cbc3302eca51960a"}]},{"id":"S016","semantic_type":"algorithm","kind":"algorithm","title":"SOS松弛层级","source_units":[{"start":57619,"end":58175,"start_line":658,"end_line":669,"raw_tex":"The constraints of~\\cref{eq:vp-finite-dimensional}, which enforce polynomial nonnegativity on semialgebraic sets, can be strengthened into weighted SOS constraints to obtain\n\\begin{equation}\n\\label{eq:vp-SOS}\n\\Lnu\\ge\\Lsos :=\\!\\!\\!\\!\\!\\!\\!\n\\sup_{ \\substack{\\vphi \\in \\Phi \\cap \\R^n[x,y]_\\nu\\\\h, q_i \\in \\R[x]_\\nu, \\, \\eta \\in \\R_+} }\n\\!\\!\\!\\!\\!\\!L(h,q_1,\\ldots,q_s)\n\\;\\; \\text{s.t.}\n\\begin{array}[t]{l}\nf - \\eta a + \\mathcal{D}\\varphi - h \\in \\cQ_\\nu(\\Gamma),\\\\\n\\varphi\\cdot \\nabla g_i - q_i|\\nabla g_i|^2 \\in \\cQ_\\nu(\\Lambda_i). \n\\end{array}\n\\end{equation}","sha256_utf8":"fa3d5d9e15c680fc917b95f687e14385ab1aeae1c6d974a1d66075290b22a826"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"选取近优对偶可行点","source_units":[{"start":113814,"end":115372,"start_line":1251,"end_line":1269,"raw_tex":"\n\\label{ss:proof-sos-variational}\n%The inequalities $\\mathbb{L}_{1,{\\rm sos}} \\leq \\mathbb{L}_{2,{\\rm sos}} \\leq \\cdots \\leq \\mathbb{L}_\\infty \\leq \\mathcal{F}^*$ follow upon combining  \\cref{th:variational-porblem-lower-bound} with $\\mathbb{L}_{\\nu,{\\rm sos}} \\leq \\mathbb{L}_\\nu \\leq \\mathbb{L}_\\infty$ and $\\mathbb{L}_{\\nu,{\\rm sos}} \\leq \\mathbb{L}_{\\nu+1,{\\rm sos}}$, which holds because truncated quadratic modules satisfy $\\mathcal{Q}_\\nu \\subset \\mathcal{Q}_{\\nu+1}$. \n%To establish convergence as $\\nu \\to \\infty$ \nIt suffices to prove that, for any $\\varepsilon>0$, there exists $\\nu$ such that \n%\\begin{equation}\n%\\label{eq:epsilon-suboptimality-target-VP}\n%\\mathbb{L}_{\\nu,{\\rm sos}} \\geq \\mathbb{L}_{\\infty} - 2\\varepsilon.\n%\\end{equation} \n$\\Lsos \\geq \\Lpdr - 2\\varepsilon.$\n%\nFix any feasible tuple $(\\eta^\\varepsilon, \\vphi^\\varepsilon,h^\\varepsilon,\\ell_1^\\varepsilon,\\ldots,\\ell_s^\\varepsilon)$ for~\\cref{eq:relaxed-dual} that satisfies \n%\n\\begin{equation}\n\\label{eq:epsilon-suboptimality-nonpoly-VP}\n\\int_{\\Omega} h^\\varepsilon(x) \\, \\dx + \\sum_{i=1}^s \\int_{\\partial\\Omega_i} \\ell_i^\\varepsilon(x) \\,\\dbound \\geq \\Lpdr - \\varepsilon.\n\\end{equation}\n%\nWe will use $\\vphi^\\varepsilon,h^\\varepsilon,\\ell_1^\\varepsilon,\\ldots,\\ell_s^\\varepsilon$ to construct polynomials $\\vphi,h,q_1,\\ldots,q_s$ such that the tuple $(\\eta^\\varepsilon, \\vphi,h,q_1,\\ldots,q_s)$ is feasible for~\\cref{eq:vp-SOS} for sufficiently large $\\nu$ and achieves an objective value larger than $\\Lpdr - 2\\varepsilon$. This will imply $\\Lsos \\geq \\Lpdr - 2\\varepsilon$.\n","sha256_utf8":"968f516248a11f5c6a70c6b865c48148d4b752148c7917475281faac649b7d97"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"构造带余量的多项式逼近","source_units":[{"start":115373,"end":116647,"start_line":1271,"end_line":1285,"raw_tex":"By assumption, the domain $\\Omega$ and the boundary sections $\\partial\\Omega_i$ are compact and $\\abs{\\nabla g_i}$ is strictly positive on $\\partial\\Omega_i$. Moreover, the $(x,y)$ projection of the compact set $\\Gamma$, denoted $\\Gamma_{xy}$ below, must be a compact subset of $\\R^n\\times\\R^m$. Since polynomials are dense in the space of $t$-times continuously differentiable functions on compact sets for all $t$, we can find polynomial $\\vphi$, $h$, $q_1$, $\\ldots$, $q_s$ that satisfy\n%\n\\begin{subequations}\n\t\\begin{align}\n\t\\label{eq:poly-approx-1}\n\t\\max_{i=1,\\ldots,n} \\left\\| \\vphi_i - \\vphi^\\varepsilon_i \\right\\|_{C^1(\\Gamma_{xy})} &\\leq \\beta \\varepsilon,\\\\\n\t\\label{eq:poly-approx-2}\n\t\\left\\| h - h^\\varepsilon + \\alpha_0 \\varepsilon \\right\\|_{C(\\Omega)} &\\leq \\beta \\varepsilon,\\\\\n\t\\label{eq:poly-approx-3}\n%\t\\left\\| q_i - \\frac{\\ell^\\varepsilon_i}{\\abs{\\nabla g_i}} + \\frac{\\alpha_i\\varepsilon}{\\abs{\\nabla g_i}^2} \\right\\|_{C(\\partial\\Omega_i)} &\\leq \\beta \\varepsilon,\n\t\\left\\| q_i - \\ell^\\varepsilon_i \\abs{\\nabla g_i}^{-1} + \\alpha_i\\varepsilon \\abs{\\nabla g_i}^{-2} \\right\\|_{C(\\partial\\Omega_i)} &\\leq \\beta \\varepsilon,\n\t\\end{align}\n\\end{subequations}\n%\nwhere $\\alpha_0,\\ldots,\\alpha_s$ and $\\beta$ are strictly positive constants to be determined later.\n","sha256_utf8":"1a002500cb953b1931a9f90ceabc732649010f31ec1d09f76e32a056b974cfd6"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"估计全散度逼近误差","source_units":[{"start":116648,"end":117123,"start_line":1287,"end_line":1294,"raw_tex":"Omitting all function arguments to simplify the notation, we may estimate\n%\n\\begin{align}\n\\mathcal{D}\\vphi^\\varepsilon - \\mathcal{D}\\vphi\n&= \\sum_{i=1}^n \\frac{\\partial}{\\partial x_i} (\\vphi_i^\\varepsilon - \\vphi_i) +   \\sum_{i=1}^n \\sum_{j=1}^m \\frac{\\partial}{\\partial y_i} (\\vphi_i^\\varepsilon - \\vphi_i) z_{ji}\\\\\\nonumber\n&\\leq \\bigg( n + \\sum_{i=1}^n \\sum_{j=1}^m \\max_{\\Gamma} \\vert z_{ji} \\vert \\bigg) \\beta \\varepsilon \n\\qquad \\forall (x,y,z) \\in \\Gamma.\n\\end{align}\n","sha256_utf8":"f1f9b94488a12c750cec56419af7f7e00c6a702cb446e8594387114796370bdc"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"控制内部与边界残差","source_units":[{"start":117125,"end":118256,"start_line":1296,"end_line":1313,"raw_tex":"Thus,\n%\n\\begin{align}\nf + \\mathcal{D}\\vphi - \\eta^\\varepsilon a - h\n&\\geq \nf + \\mathcal{D}\\vphi^\\varepsilon - \\eta^\\varepsilon a - h^\\varepsilon \n+ \\alpha_0\\varepsilon \n- \\bigg( 1 + n + \\sum_{i,j} \\max_{\\Gamma} |z_{ji}|\\bigg) \\beta\\varepsilon\n\\\\\\nonumber\n%&\\geq \\alpha_0 \\varepsilon - \\bigg( 1 + n + \\sum_{i,j} \\max_{\\Gamma} |z_{ji}|\\bigg) \\beta\\varepsilon\n%\\\\\n&\\geq  \\alpha_0 \\varepsilon - C_0 \\beta\\varepsilon\n\\qquad \\forall (x,y,z) \\in \\Gamma,\n\\end{align}\n%\nwhere $C_0:= 1 + n + \\sum_{i,j} \\max_{\\Gamma} |z_{ji}|$ is a positive constant that does not depend on $\\alpha_0$, $\\beta$ or $\\varepsilon$.\nThe second inequality holds because $(\\eta^\\varepsilon, \\vphi^\\varepsilon,h^\\varepsilon,\\ell_1^\\varepsilon,\\ldots,\\ell_s^\\varepsilon)$ is feasible for~\\cref{eq:relaxed-dual}. Similarly, on each $\\Lambda_i$ the inequality $\\vphi \\cdot \\nabla g_i - q_i \\abs{\\nabla g_i}^2 \\geq \n\\alpha_i \\varepsilon - C_i \\beta \\varepsilon$ holds for some positive constant $C_i$ that depends only on the particular $\\partial\\Omega_i$ and $g_i$ under consideration, not on $\\alpha_i$, $\\beta$ or $\\varepsilon$. The details are omitted for brevity.\n","sha256_utf8":"90f64b46a396e059866cb04e4886ca5084f91bc7415d81f62aa4f830cf98df8c"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"选择偏移确保严格正性","source_units":[{"start":118257,"end":118652,"start_line":1315,"end_line":1322,"raw_tex":"Now choose $\\alpha_0 = 2C_0 \\beta$ and $\\alpha_i = 2C_i \\beta$ for all $i=1,\\ldots,s$ to obtain\n%\n\\begin{subequations}\n\t\\begin{align}\n\tf + \\mathcal{D}\\vphi - \\eta^\\varepsilon a - h &\\geq C_0\\beta \\varepsilon \\quad \\text{ on } \\Gamma,\\\\\n\t\\vphi \\cdot \\nabla g_i - q_i \\abs{\\nabla g_i}^2 &\\geq C_i\\beta \\varepsilon \\quad \\text{ on } \\Lambda_i, \\quad i = 1,\\ldots,s.\n\t\\end{align}\n\\end{subequations}\n","sha256_utf8":"b8101e2a5c6fd04e1dbe34e3ec9a514b62aa58b19e6886b2125f04d8fff882a4"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"取得有限次数SOS证书","source_units":[{"start":118654,"end":119464,"start_line":1324,"end_line":1333,"raw_tex":"The right-hand sides of these inequalities are strictly positive and, by assumption, $r_0^2 - |(x,y,z)|^2 \\in \\mathcal{Q}(\\Gamma)$ and $r_i^2 - |(x,y)|^2 \\in \\mathcal{Q}(\\Lambda_i)$, therefore Putinar's Positivstellensatz~\\cite{Putinar} guarantees that $f + \\mathcal{D}\\vphi - \\eta^\\varepsilon a - h \\in \\mathcal{Q}_\\nu(\\Gamma)$ and $\\vphi \\cdot \\nabla g_i - q_i \\abs{\\nabla g_i}^2 \\in \\mathcal{Q}_\\nu(\\Lambda_i)$ for a sufficiently large integer $\\nu$. In other words, the tuple $(\\eta^\\varepsilon,\\vphi,h,q_1,\\ldots,q_s)$ is feasible for the SOS program~\\cref{eq:vp-SOS}. Then, we must have\n%\n\\ifx\\SIOPT\\undefined\n% not the SIOPT version, do nothing!\n\\else\n% We have defined the \\SIOPT variable, but is it really the SIOPT version? Another \\if statement...\n\\if\\SIOPT1\n\\enlargethispage{\\baselineskip}\n\\fi\n\\fi\n","sha256_utf8":"a948e728d9e7d329a7fccf99cd608e0f82429f4ae0f1eb3d022af1b04037de85"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"控制目标损失并完成收敛","source_units":[{"start":119464,"end":120508,"start_line":1334,"end_line":1355,"raw_tex":"\\begin{align}\n\\label{eq:epsilon-suboptimality-final-estimate-VP}\n\\Lsos\n&\\geq \\int_{\\Omega} h \\, \\dx + \\sum_{i=1}^s \\int_{\\partial\\Omega_i} q_i \\abs{\\nabla g_i} \\,\\dbound\n\\\\\\nonumber\n&\\geq \\int_{\\Omega} h^\\varepsilon\\, \\dx \n- (\\alpha_0+\\beta)\\vert\\Omega\\vert \\varepsilon\n+ \\sum_{i=1}^s \\int_{\\partial\\Omega_i} \n\t\\ell_i^\\varepsilon \n\t-\\beta\\varepsilon \\abs{\\nabla g_i}\n\t- \\alpha_i \\varepsilon\\abs{\\nabla g_i}^{-1}\n\\,\\dbound\n\\\\\\nonumber\n&\\geq \\Lpdr - \\bigg[ 1+ (2C_0+1)\\vert\\Omega\\vert \\beta + \\beta \\sum_{i=1}^s \\int_{\\partial\\Omega_i} \n\\abs{\\nabla g_i}\n+2C_i \\abs{\\nabla g_i}^{-1}\n\\,\\dbound\n\\bigg] \\varepsilon.\n\\end{align}\nThe second inequality follows from elementary estimates based on~\\cref{eq:poly-approx-1,eq:poly-approx-2,eq:poly-approx-3}, while the last one follows from~\\cref{eq:epsilon-suboptimality-nonpoly-VP} and our choices for $\\alpha_0,\\ldots,\\alpha_s$. Choosing a sufficiently small positive value for $\\beta$ \n%yields~\\cref{eq:epsilon-suboptimality-target-VP}, as desired.\nyields $\\Lsos \\geq \\Lpdr - 2\\varepsilon$, as desired.\n","sha256_utf8":"ef7be40aeba658c1aa0177b473847fb49346b214659ebcbdef467f5754708be4"}]},{"id":"S024","semantic_type":"external_fact","kind":"external_fact","title":"多项式一致逼近","source_units":[],"declared_external":true,"declaration_text":["Polynomial approximation: continuous functions on the relevant compact sets admit uniform polynomial approximation, and the continuously differentiable functions used in the proof admit polynomial approximation together with their first derivatives."]},{"id":"S025","semantic_type":"external_fact","kind":"external_fact","title":"Putinar正性定理","source_units":[],"declared_external":true,"declaration_text":["Putinar's Positivstellensatz: if a quadratic module is Archimedean, every polynomial strictly positive on its defining basic closed semialgebraic set belongs to that module; each resulting representation has some finite degree."]},{"id":"S026","semantic_type":"external_fact","kind":"external_fact","title":"嵌套上确界的收敛","source_units":[],"declared_external":true,"declaration_text":["Suprema over nested feasible sets are monotone. If all such suprema are bounded above by L and, for every epsilon>0, one of them is at least L-2epsilon, their limit is L."]},{"id":"S027","semantic_type":"external_fact","kind":"external_fact","title":"弱梯度的散度刻画","source_units":[],"declared_external":true,"declaration_text":["The Sobolev chain rule and divergence theorem apply to the admissible test fields defining Phi. The converse characterization of the weak gradient by these test identities is the cited boundary fact from Korda2018, Lemma 1."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S003","target_id":"S001","kind":"supports"},{"source_id":"S004","target_id":"S010","kind":"supports"},{"source_id":"S005","target_id":"S009","kind":"supports"},{"source_id":"S006","target_id":"S008","kind":"supports"},{"source_id":"S006","target_id":"S009","kind":"supports"},{"source_id":"S006","target_id":"S019","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S009","kind":"supports"},{"source_id":"S007","target_id":"S010","kind":"supports"},{"source_id":"S008","target_id":"S017","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S020","kind":"supports"},{"source_id":"S010","target_id":"S013","kind":"supports"},{"source_id":"S010","target_id":"S017","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S014","kind":"supports"},{"source_id":"S011","target_id":"S018","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S018","kind":"supports"},{"source_id":"S013","target_id":"S016","kind":"supports"},{"source_id":"S013","target_id":"S023","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S022","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S022","kind":"supports"},{"source_id":"S016","target_id":"S023","kind":"supports"},{"source_id":"S002","target_id":"S018","kind":"supports"},{"source_id":"S002","target_id":"S022","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S020","kind":"supports"},{"source_id":"S017","target_id":"S023","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S018","target_id":"S023","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S017","target_id":"S003","kind":"supports"},{"source_id":"S018","target_id":"S003","kind":"supports"},{"source_id":"S019","target_id":"S003","kind":"supports"},{"source_id":"S020","target_id":"S003","kind":"supports"},{"source_id":"S021","target_id":"S003","kind":"supports"},{"source_id":"S022","target_id":"S003","kind":"supports"},{"source_id":"S023","target_id":"S003","kind":"supports"},{"source_id":"S024","target_id":"S018","kind":"supports"},{"source_id":"S025","target_id":"S022","kind":"supports"},{"source_id":"S026","target_id":"S017","kind":"supports"},{"source_id":"S026","target_id":"S023","kind":"supports"},{"source_id":"S027","target_id":"S007","kind":"supports"}]}}
{"record_id":"2111.15645v5","paper":{"title":"Survey Descent: A Multipoint Generalization of Gradient Descent for Nonsmooth Optimization","authors":["X. Y. Han","Adrian S. Lewis"],"arxiv_id":"2111.15645","version":"v5","journal":"SIAM Journal on Optimization","doi":"10.1137/21m1468450","abstract_url":"https://arxiv.org/abs/2111.15645v5","pdf_url":"https://arxiv.org/pdf/2111.15645v5"},"context":{"raw_tex":"To approach our target---a fast local iteration for nonsmooth optimization---let us recall, as motivation, the classical foundations of gradient descent (GD). For some function $h$, denote the linear approximation of $h$ centered at some differentiable iterate $\\tx$ by \\[\\ell_{\\tx}^{h}(x)\\equiv h(\\tx) + \\nabla h(\\tx)^T(x-\\tx),\\]\n\n\n\\begin{defn} \\textbf{(Survey Descent Iteration $\\boldsymbol{\\left\\{\\left(P_i^\\S\\right)\\right\\}_{i=1}^k}$)\\label{def:SD_Method}} Given a \\textit{survey} of $k$ points, $\\S=\\left\\{ s_{i}\\right\\} _{i=1}^{k}$, at which the nonsmooth objective $h$ is differentiable and a step-control parameter $L$, for each $i=1,{\\dots},k$, define the \\textit{$i$-th subproblem} $\\left(P_i^\\S\\right)$ as follows:\n\\begin{align}\n\\min_{x}\\ & \\left\\| x-\\left(s_{i}-\\frac{1}{L}\\nabla h\\left(s_{i}\\right)\\right)\\right\\| _{2}^{2}\\label{eq:SD_objective}\\\\\n\\mathrm{s.t.}\\ & \\ell_{s_j}^{h}(x)+\\frac{L}{2}\\left\\| x-s_{j}\\right\\| _{2}^{2}\\leq \\ell_{s_i}^{h}(x)\\ \\forall\\ j\\neq i.\\label{eq:SD_constraints}\n\\end{align}\nWe refer to the solving of all subproblems $\\left\\{\\left(P_i^\\S\\right)\\right\\}_{i=1}^k$ as a \\textit{Survey Descent iteration}.\n\\end{defn}\n\nWhen $\\left(P_i^\\S\\right)$ is feasible, we denote its optimal solution as $s_i^+$. When all subproblems are feasible, we say that the entire Survey Descent iteration is \\textit{feasible} and call $\\S^+=\\left\\{s_i^+\\right\\}_{i=1}^k$ the \\textit{outputs} of the iteration; otherwise, we say the entire iteration is \\textit{infeasible}. After a feasible iteration, we would update $\\S{\\leftarrow}\\S^{+}$ and repeat the Survey Descent iteration on the updated survey.\n\n\n\\begin{equation}\n    \\bx = \\arg\\min_{x \\in \\R^n} f(x) \\text{ and } 0\\in\\Relint\\left( \\partial f(\\bx) \\right),\\label{eq:nondegen}\n\\end{equation}\n\nwhere $\\Relint(\\cdot)$ denotes the relative interior. We further assume the following structure around $\\bx$.\n\\begin{defn}\\label{def:strong_max}\nA function $f:\\R^n \\to \\R$ is a \\textit{strong $\\C^2$ max function} if it is locally expressible near $\\bx$ as\\begin{equation}\\label{eq:f_objective}\nf\\left(x\\right)=\\underset{i=1,...k}{\\max}f_{i}\\left(x\\right),\n\\end{equation}\nwhere $k$ is some finite number, the \\textit{components} $\\left\\{f_i\\right\\}_{i=1}^k$ are $\\C^2$-functions satisfying $f_i(\\bx)=f(\\bx)$ for all $i$, their gradients $\\left\\{\\nabla f_i(\\bx)\\right\\}_{i=1}^k$ are affinely independent, and their Hessians $\\nabla^2 f_i (\\bx)$ are positive definite. As a consequence, there exists constants $\\delta,L>0$ such that their Hessians satisfy\\begin{equation}\n    \\delta I \\preceq \\nabla^2 f_i(x) \\preceq L I \\ \\forall \\ i=1,\\dots,k, \\label{eq:hessian}\n\\end{equation} \nfor $x$ near $\\bx$---where $I$ is the $n{\\times}n$ identity matrix.\n\\end{defn}\n\nOur analysis in this paper is entirely local, but we \\underline{assume for the rest of this paper} that $f$ is a strong $\\C^2$ max function with nondegenerate minimizer $\\bx$. For simplicity, we also assume that the properties in Definition \\ref{def:strong_max} hold \\textit{globally} for all $x\\in \\R^n$. In this setting, we refer to $k$---which is necessarily unique---as the \\textit{degree} of $f$. Moreover, note that \\eqref{eq:hessian} implies all components $\\left\\{f_i\\right\\}_{i=1}^k$ are $L$-smooth, both $f$ and its components are $\\delta$-strongly convex, and $\\bx$ is $f$'s \\textit{unique} minimizer. \n\nTo study Survey Descent (Definition \\ref{def:SD_Method}) on the objective $f$, we choose the above-presented $L$ as the step-control parameter and define the following notions of \\textit{valid} and \\textit{minimizing} surveys.\n\n\n\\begin{defn}\n\\textbf{\\label{def:validS} ($\\S, \\bX$-Surveys and Validity)} For the objective $f$, define a \\textit{survey} as a matrix $\\S \\equiv \\left[s_{1}{,}\\dots{,}s_{k}\\right] \\in \\mathbb{\\R}^{n\\times k}$ consisting of $k$ columns\\footnote{We can equivalently consider $\\S$ as a set $\\{s_i\\}_{i=1}^k$. We will use the matrix and set interpretations interchangeably.} (the ``survey points''), where $k$ is the degree of $f$; \\textit{valid surveys} as surveys satisfying $f_{i}\\left(s_{i}\\right){>}f_{j}\\left(s_{i}\\right)$ for all $i{\\neq}j$; and the \\textit{minimizing survey} as $\\bX \\equiv [\\bx{,}\\dots{,}\\bx]$ whose survey points are all equal to $\\bx$.\n\\end{defn}\n\n\\begin{equation}\n\\left\\| \\S\\right\\| \\equiv\\left\\| \\left[s_{1},\\dots,s_{k}\\right]\\right\\| _{2,\\infty}=\\max_{i=1,\\dots,k}\\left\\| s_{i}\\right\\| _{2},\\label{eq:surv_norm}\n\\end{equation}\n\nGiven a valid $\\S$ sufficiently close to $\\bX$, Survey Descent exhibits many desirable properties when minimizing $f$. As a preliminary, note that \\eqref{eq:nondegen} and Definition \\ref{def:strong_max} implies there exists unique \\textit{critical weights} $\\left\\{ \\blam_{i}\\right\\}_{i=1}^{k}{>}0$ such that \n\\begin{align}\n\\sum_{i=1}^{k}\\blam_{i} & =1,\\label{eq:lam_sum1}\\\\\n\\sum_{i=1}^{k}\\blam_{i}\\nabla f_{i}(\\bx) & =0.\\label{eq:lam_sum0}\n\\end{align}\n\nAs we will see, the Lagrange multipliers of Survey Descent subproblems will lie ``close'' to $\\left\\{ \\blam_{i}\\right\\}_{i=1}^{k}{>}0$, while the subproblem outputs $\\left\\{s_i^+\\right\\}_{i=1}^k$ will lie near  $\\bx$. To characterize the order-of-magnitude of these distances, we adopt the ``Big-Oh'' notation. In particular, for a mapping $g:\\mathbf{E}{\\to}\\F$ between two Euclidean spaces and letting $\\left\\| \\cdot\\right\\|^p$ denote an arbitrary Euclidean norm raised to the $p$-th power, we use the notation \n$g\\left(x\\right)=O\\left(\\left\\| x\\right\\| ^{p}\\right)$\nto indicate the property that there exists a constant $K{>}0$ such that $ \\left\\| g\\left(x\\right)\\right\\|{\\leq}K\\left\\| x\\right\\| ^{p}$ holds for all small $x$. By itself, we let $\\mathrm{O}\\left(\\left\\| x\\right\\| ^{p}\\right)$ denote an element of the class of all functions with this property.\n\n\n\\begin{setting} \\label{setting:localSD} \\textbf{(Local Analysis of Survey Descent)}\nConsider an iteration of Survey Descent on a strong $\\C^2$ max function objective, $f$, with a survey \n$\\S$ that is valid and sufficiently close to $\\bX$. For $i{=}1{,}\\dots{,}k$, denote the output of the $i$-th Survey Descent subproblem by $s_i^+$.\n\\end{setting}\n\n\n\\begin{equation}\\label{eq:valS}\n\\nabla f\\left(s_{i}\\right)=\\nabla f_{i}\\left(s_{i}\\right) \\ \\forall \\ i=1,\\dots,k,\n\\end{equation}\n\n\\begin{observe}\n\\textbf{(Survey Descent Iterations on Max-of-Smooth Objectives)\\label{obs:SD_MOS}} Assume that $\\S$ is a valid survey for the objective $f$. Then, for all $i$, the $i$-th subproblem $\\left(P_{i}^{\\S}\\right)$ of Survey Descent (Definition \\ref{def:SD_Method}) is equivalent to \\begin{equation}\\begin{aligned}\\min_{x}\\ & \\left\\| x-\\left(s_{i}-\\frac{1}{L}\\nabla f_{i}\\left(s_{i}\\right)\\right)\\right\\| _{2}^{2}\\\\\n\\mathrm{s.t.}\\ & \\ell_{s_j}^{f_j}(x)+\\frac{L}{2}\\left\\| x-s_{j}\\right\\| _{2}^{2}\\leq \\ell_{s_i}^{f_i}(x)\\ \\forall\\ j\\neq i.\n\\end{aligned}\n\\label{eq:raw_form_analytic}\n\\end{equation}\n\\end{observe}\n\n\n\\begin{align}\n\\U & \\equiv \\bx + \\Span\\left\\{ \\nabla f_{i}(\\bx)-\\nabla f_{j}(\\bx):1\\leq i,j\\leq k\\right\\} ^{\\perp} \\label{eq:Udef1}\\\\\n & = \\bx +  \\left\\{ \\sum_{i}^k\\gamma_{i}\\nabla f_{i}(\\bx):\\sum_{i}^k\\gamma_{i}=0\\right\\} ^{\\perp} \\label{eq:Udef2}\\\\\n & = \\bx +  \\left\\{x\\in \\R^n: \\ \\nabla f_i(\\bx)^T x = \\nabla f_j(\\bx)^T x \\ \\forall \\ 1\\leq i,j\\leq k\\right\\}.\\label{eq:Udef3}\n\\end{align}\n\n\\begin{equation}\n    a_i \\equiv \\nabla f_i(\\bx) \\text{ and } A_i \\equiv \\nabla^2 f_i(\\bx) \\ \\forall \\  i=1,\\dots,k. \\label{eq:A_notation}\n\\end{equation}\n\n\\begin{equation}\n    f_{i}\\left(x\\right)= f(0) + a_{i}^{T}x+\\frac{1}{2}x^{T}A_{i}x+r_{i}\\left(x\\right) \\ \\forall \\ i=1,\\dots,k,\\label{eq:taylor}\n\\end{equation}\n\nwhere $r_i(x)$ is a residual function.\n\n\n\\begin{equation}\n    \\blam_{\\min} \\equiv \\min_{i=1,\\dots,k} \\blam_{i} > 0. \\label{eq:min_lam}\n\\end{equation}\n\n\\begin{procedure}\\textbf{(Repeated Iterations of Survey Descent) }\\label{procedure:SD}\nFor the strong $\\C^2$ max function objective $f$, assume $\\S^{0}$ is a valid initial survey sufficiently close to $\\bX$, and initialize $t=0$. Iterate the following steps:\n\\begin{enumerate}\n\\item Solve the Survey Descent iteration $\\left\\{ \\left(P_{i}^{\\S^{t}}\\right)\\right\\} _{i=1}^{k}$\nand denote the output $\\S^{t+1}\\equiv\\left(\\S^{t}\\right)^{+}$.\n\\item Increment $t\\leftarrow t+1$. \n\\end{enumerate}\n\\end{procedure}\n\n\n\\begin{observe}\\label{obs:allT}\nThe proof of Corollary \\ref{cor:viable} more generally shows that, for all $t \\in \\nonneg$, all surveys $\\S^{t}$ will remain sufficiently close to $\\bX$. Thus, the conclusions from Theorems \\ref{thm:SD_MBP}-\\ref{thm:s_monotone} apply to every iteration of Procedure \\ref{procedure:SD}.\n\\end{observe}\n\n\n\\begin{defn}\n    \\textbf{(Multiple Balls Problem, $\\MBP\\param$)\\label{def:MBP}} \\sloppy For some\n    target vector $y\\in\\R^{n}$, ball centers $Z=\\left[z_{1},\\dots,z_{\\bk}\\right]\\in\\R^{n\\times\\bk}$\n    where $\\bk\\leq n$, and non-negative radii $r=\\left[r_{1},\\dots,r_{\\bk}\\right]^{T}\\in\\R^{\\bk}_{+}$,\n    define the \\textit{Multiple Balls Problem (MBP)} as \n    \\begin{align}\n    \\min_{x\\in\\R^{n}}\\ & \\left\\| x-y\\right\\| _{2}^{2} \\nonumber\\\\\n    \\mathrm{s.t.}\\ & \\left\\| x-z_{j}\\right\\| _{2}^{2} = r_{j}^{2}\\ \\forall\\ j=1,\\dots,\\bk. \\label{eq:MBP_cons}\n    \\end{align}\n    Denote the problem as $\\MBP\\param$ and its \\textit{feasible region} as $\\B\\spec$. We will refer to the tuple $\\param$ as the \\textit{parameters} and the sub-tuple $\\spec$ as the \\textit{ball-specifications} of the MBP.\n    \\end{defn}\n\n    Since $y$, $Z$, and $r$ all reside in Euclidean spaces, we can also consider $\\spec$ and $\\param$ elements of some product space. Let $\\left\\Vert \\cdot\\right\\Vert $ denote an arbitrary Euclidean norm. This will allow us to formalize the notion of distance in of parameters and ball-specifications in their respective spaces. We are particularly interested in the behavior of $\\MBP\\param$ where $\\param$ is near some set of \\textit{base parameters}.\n\n\n\\begin{defn}\n    \\textbf{(Base Parameters)\\label{def:base}} We refer to a set of parameters\n    as \\textit{base parameters} and denote them as $\\bparam$\n    if they satisfy the following properties:\n    \\begin{itemize}\n    \\item The columns of $\\bZ=\\left[\\bz_{1},\\dots,\\bz_{\\bk}\\right]$ are linearly\n    independent.\n    \\item $\\br_{j}^{2}=\\left\\| \\bz_{j}\\right\\| _{2}^{2}$.\n    \\item There exist strictly-positive weights $\\clam=\\left[\\clam_{1},\\dots,\\clam_{\\bk}\\right]^{T}\\in\\R_{++}^{\\bk}$\n    such that $\\bar{y}=-\\sum_{j=1}^{\\bk}\\clam_{j}\\bar{z}_{j}.$\n    \\end{itemize}\n    \\end{defn}\n\n    \\begin{itemize}\n        \\item The \\textit{central affine subspace}\n            \\begin{equation}\n            \\A\\spec \\equiv \\left\\{ x\\in\\R^{n}:\\ 2z_{j}^{T}x-\\left\\| z_{j}\\right\\| _{2}^{2}+r_{j}^{2}\\text{ equal  for all } j=1,\\dots,\\bk\\right\\}. \\label{eq:CAS}\n            \\end{equation}\n        \\item The \\textit{core}\n            \\begin{equation}\n            c\\spec\\equiv\\projA{\\spec}\\left(z_{j}\\right)\\ \\text{ for any }\\ j=1,\\dots,\\bk, \\label{eq:core}\n            \\end{equation}\n            where $\\projA{\\spec}$ denotes the projection onto $\\A\\spec$.\n        \\item The \\textit{margin}\n            \\begin{equation}\n            \\rho\\spec\\equiv\\sqrt{r_{j}^{2}-\\left\\| z_{j}-c\\spec\\right\\| _{2}^{2}} \\ \\text{ for any }\\ j=1,\\dots,\\bk. \\label{eq:margin}\n            \\end{equation}\n    \\end{itemize}\n\n\n    In this section, we will show that the optimal solutions of MBPs are unchanged when the equality constraints \\eqref{eq:MBP_cons} are relaxed to inequalities. In particular, regardless of whether $\\MBP\\param$ is defined with equality or inequality constraints, note that the stationarity conditions remain the same. Therefore, there must exist some Lagrange multipliers, $\\lambda\\param\\in\\R^m$, such that\\begin{equation}\n    x\\param-y+\\sum_{j=1}^{\\bk}\\lambda_{j}\\param\\left[x\\param-z_{j}\\right]=0, \\label{eq:MBP_FOC}\n    \\end{equation}\n    where $\\lambda_{j}\\param$ denotes the $j$-th element of $\\lambda\\param$. The only difference is that the inequality version possesses additional dual feasibility conditions requiring the non-negativity of $\\lambda\\param$. Lemma \\ref{lem:MBP_lam} will show that the $\\lambda\\param$ are unique and strictly positive.\n\n\n    We can reformulate Survey Descent subproblems into inequality-constrained MBPs. In particular, for all $i$, we can rewrite the $i$-th Survey Descent subproblem in Observation \\ref{obs:SD_MOS} as\n    \n\n\n\\begin{gather}\\begin{aligned}\\min_{x}\\ & \\left\\| (x - s_{i}) - y_{i}\\left(\\S\\right) \\right\\| _{2}^{2}\\\\\n    \\mathrm{s.t.}\\ & \\left\\| (x-s_{i})-z_{ij}\\left(\\S\\right)\\right\\| _{2}^{2}\\leq r_{ij}^{2}\\left(\\S\\right)\\ \\forall\\ j\\neq i,\n    \\end{aligned}\n    \\label{eq:proj_form}\n    \\end{gather}\n\n    where, for all $i$ and $j$, we define the following mappings using the components $\\{f_i\\}_{i=1}^k$ from Definition \\ref{def:strong_max}:\n    \n\n\n\\begin{gather}\\begin{aligned}\n    y_{i}\\left(\\S\\right)& \\equiv-\\frac{1}{L}\\nabla f_{i}\\left(s_{i}\\right),\\\\\n    z_{ij}\\left(\\S\\right) & \\equiv s_j - s_i  + y_{j}\\left(\\S\\right)-y_{i}\\left(\\S\\right),\\\\\n    r_{ij}^{2}\\left(\\S\\right) & \\equiv\\frac{2}{L}\\left[f_{i}\\left(s_{i}\\right)-f_{j}\\left(s_{j}\\right)+\\nabla f_{i}\\left(s_{i}\\right)^{T}\\left(s_{j}-s_{i}\\right)\\right]+\\left\\|y_{j}\\left(\\S\\right)-y_{i}\\left(\\S\\right)\\right\\| _{2}^{2}.\n    \\end{aligned}\\label{eq:MBP_mappings}\n    \\end{gather}\n\n    \\begin{observe}\\textbf{(Survey Descent as Inequality-Constrained MBP)} \\label{obs:SD_as_MBP} Consider any $k$-point survey $\\S$ and any $i{=}1,\\dots,k$. After the changes-of-variables $x \\leftarrow x{-} s_i$, the minimization problem in \\eqref{eq:proj_form} is an inequality-constrained MBP  with $\\m = k{-}1$ and parameters \n    \n    $$\\param = \\left(y_{i}\\left(\\S\\right), \\left[ z_{ij}\\left(\\S\\right)\\right]_{j\\neq i}, \\left[r_{ij}\\left(\\S\\right)\\right]_{j\\neq i},\\left[ s_{j}\\right]_{j\\neq i}\\right),$$\n    where, for any $j$-indexed scalars $\\left\\{v_j\\right\\}_{j=1}^{k}$, we use the notation $\\left[v_j\\right]_{j \\neq i}{\\equiv} \\left[v_1,\\dots,v_{i-1},v_{i+1},\\dots,v_{k}\\right]^T$ to denote the length $k{-}1$ vector excluding $v_i$.\n    Furthermore, the mappings \\eqref{eq:MBP_mappings}  evaluated at the base survey\\footnote{Evaluating of the mappings \\eqref{eq:MBP_mappings} at $\\bX$ is a technical step. $\\bX$ is not a valid survey (Definition \\ref{def:validS}), so it does not correspond to an actual Survey Descent iteration.} $\\bX$ (Definition \\ref{def:validS}) form a set of base parameters\n    \\begin{equation}\n        \\bparam = \\left(y_{i}\\left(\\bX\\right), \\left[ z_{ij}\\left(\\bX\\right)\\right]_{j\\neq i}, \\left[r_{ij}\\left(\\bX\\right)\\right]_{j\\neq i}, [\\bx]_{j\\neq i}\\right),\\label{eq:MBP_SD_base}\n    \\end{equation}\n    with corresponding weights $\\{\\blam_j\\}_{j\\neq i}$ from \\eqref{eq:lam_sum1}-\\eqref{eq:lam_sum0}, and where $[\\bx]_{j\\neq i}$ is the $k{-}1$ length vector with all $\\bx$ entries.\n    \\end{observe}\n","units":[{"start":1861,"end":2192,"start_line":38,"end_line":38,"raw_tex":"To approach our target---a fast local iteration for nonsmooth optimization---let us recall, as motivation, the classical foundations of gradient descent (GD). For some function $h$, denote the linear approximation of $h$ centered at some differentiable iterate $\\tx$ by \\[\\ell_{\\tx}^{h}(x)\\equiv h(\\tx) + \\nabla h(\\tx)^T(x-\\tx),\\]\n","sha256_utf8":"d1500ef829c50fb34a7dcd3009032a6f63620172c3dc633ead8c25dc6d20de0e"},{"start":5027,"end":5850,"start_line":55,"end_line":61,"raw_tex":"\\begin{defn} \\textbf{(Survey Descent Iteration $\\boldsymbol{\\left\\{\\left(P_i^\\S\\right)\\right\\}_{i=1}^k}$)\\label{def:SD_Method}} Given a \\textit{survey} of $k$ points, $\\S=\\left\\{ s_{i}\\right\\} _{i=1}^{k}$, at which the nonsmooth objective $h$ is differentiable and a step-control parameter $L$, for each $i=1,{\\dots},k$, define the \\textit{$i$-th subproblem} $\\left(P_i^\\S\\right)$ as follows:\n\\begin{align}\n\\min_{x}\\ & \\left\\| x-\\left(s_{i}-\\frac{1}{L}\\nabla h\\left(s_{i}\\right)\\right)\\right\\| _{2}^{2}\\label{eq:SD_objective}\\\\\n\\mathrm{s.t.}\\ & \\ell_{s_j}^{h}(x)+\\frac{L}{2}\\left\\| x-s_{j}\\right\\| _{2}^{2}\\leq \\ell_{s_i}^{h}(x)\\ \\forall\\ j\\neq i.\\label{eq:SD_constraints}\n\\end{align}\nWe refer to the solving of all subproblems $\\left\\{\\left(P_i^\\S\\right)\\right\\}_{i=1}^k$ as a \\textit{Survey Descent iteration}.\n\\end{defn}","sha256_utf8":"9cff18d3dc4ae00d29f9738249198246c759e5a9dfa46bd090f037de070316be"},{"start":5852,"end":6316,"start_line":63,"end_line":63,"raw_tex":"When $\\left(P_i^\\S\\right)$ is feasible, we denote its optimal solution as $s_i^+$. When all subproblems are feasible, we say that the entire Survey Descent iteration is \\textit{feasible} and call $\\S^+=\\left\\{s_i^+\\right\\}_{i=1}^k$ the \\textit{outputs} of the iteration; otherwise, we say the entire iteration is \\textit{infeasible}. After a feasible iteration, we would update $\\S{\\leftarrow}\\S^{+}$ and repeat the Survey Descent iteration on the updated survey.\n","sha256_utf8":"a9d78312bb0603d717b1c48a6a8c9d6b1b6b47b051bdb9053b34098789a174ac"},{"start":39374,"end":39517,"start_line":279,"end_line":281,"raw_tex":"\\begin{equation}\n    \\bx = \\arg\\min_{x \\in \\R^n} f(x) \\text{ and } 0\\in\\Relint\\left( \\partial f(\\bx) \\right),\\label{eq:nondegen}\n\\end{equation}","sha256_utf8":"d3660dec29db7edc5b2a7048fe6ee6632adffbce8b449aff47e02d62fa2729c7"},{"start":39518,"end":40474,"start_line":282,"end_line":291,"raw_tex":"where $\\Relint(\\cdot)$ denotes the relative interior. We further assume the following structure around $\\bx$.\n\\begin{defn}\\label{def:strong_max}\nA function $f:\\R^n \\to \\R$ is a \\textit{strong $\\C^2$ max function} if it is locally expressible near $\\bx$ as\\begin{equation}\\label{eq:f_objective}\nf\\left(x\\right)=\\underset{i=1,...k}{\\max}f_{i}\\left(x\\right),\n\\end{equation}\nwhere $k$ is some finite number, the \\textit{components} $\\left\\{f_i\\right\\}_{i=1}^k$ are $\\C^2$-functions satisfying $f_i(\\bx)=f(\\bx)$ for all $i$, their gradients $\\left\\{\\nabla f_i(\\bx)\\right\\}_{i=1}^k$ are affinely independent, and their Hessians $\\nabla^2 f_i (\\bx)$ are positive definite. As a consequence, there exists constants $\\delta,L>0$ such that their Hessians satisfy\\begin{equation}\n    \\delta I \\preceq \\nabla^2 f_i(x) \\preceq L I \\ \\forall \\ i=1,\\dots,k, \\label{eq:hessian}\n\\end{equation} \nfor $x$ near $\\bx$---where $I$ is the $n{\\times}n$ identity matrix.\n\\end{defn}","sha256_utf8":"eee7849d0392e87b029e95df65377b9a717bf8761ea5d773f11c4e0d0472f4f8"},{"start":40476,"end":41318,"start_line":293,"end_line":295,"raw_tex":"Our analysis in this paper is entirely local, but we \\underline{assume for the rest of this paper} that $f$ is a strong $\\C^2$ max function with nondegenerate minimizer $\\bx$. For simplicity, we also assume that the properties in Definition \\ref{def:strong_max} hold \\textit{globally} for all $x\\in \\R^n$. In this setting, we refer to $k$---which is necessarily unique---as the \\textit{degree} of $f$. Moreover, note that \\eqref{eq:hessian} implies all components $\\left\\{f_i\\right\\}_{i=1}^k$ are $L$-smooth, both $f$ and its components are $\\delta$-strongly convex, and $\\bx$ is $f$'s \\textit{unique} minimizer. \n\nTo study Survey Descent (Definition \\ref{def:SD_Method}) on the objective $f$, we choose the above-presented $L$ as the step-control parameter and define the following notions of \\textit{valid} and \\textit{minimizing} surveys.\n","sha256_utf8":"deda749a251c5e5f3794b8e53d6a8ab9a0ee6785b67adba496bc57280f620838"},{"start":41319,"end":41989,"start_line":297,"end_line":299,"raw_tex":"\\begin{defn}\n\\textbf{\\label{def:validS} ($\\S, \\bX$-Surveys and Validity)} For the objective $f$, define a \\textit{survey} as a matrix $\\S \\equiv \\left[s_{1}{,}\\dots{,}s_{k}\\right] \\in \\mathbb{\\R}^{n\\times k}$ consisting of $k$ columns\\footnote{We can equivalently consider $\\S$ as a set $\\{s_i\\}_{i=1}^k$. We will use the matrix and set interpretations interchangeably.} (the ``survey points''), where $k$ is the degree of $f$; \\textit{valid surveys} as surveys satisfying $f_{i}\\left(s_{i}\\right){>}f_{j}\\left(s_{i}\\right)$ for all $i{\\neq}j$; and the \\textit{minimizing survey} as $\\bX \\equiv [\\bx{,}\\dots{,}\\bx]$ whose survey points are all equal to $\\bx$.\n\\end{defn}","sha256_utf8":"f8370b2d87ca4af39c7aa56687ca2b4a675d722a1d2ecf3f3aa1b17b6e9b4126"},{"start":42032,"end":42213,"start_line":300,"end_line":302,"raw_tex":"\\begin{equation}\n\\left\\| \\S\\right\\| \\equiv\\left\\| \\left[s_{1},\\dots,s_{k}\\right]\\right\\| _{2,\\infty}=\\max_{i=1,\\dots,k}\\left\\| s_{i}\\right\\| _{2},\\label{eq:surv_norm}\n\\end{equation}","sha256_utf8":"4d227f4b257a5e2cb04e7f14428e7bf9ee2ac0dc336ab0e1a8c01edfb37d24b3"},{"start":42465,"end":42917,"start_line":305,"end_line":309,"raw_tex":"Given a valid $\\S$ sufficiently close to $\\bX$, Survey Descent exhibits many desirable properties when minimizing $f$. As a preliminary, note that \\eqref{eq:nondegen} and Definition \\ref{def:strong_max} implies there exists unique \\textit{critical weights} $\\left\\{ \\blam_{i}\\right\\}_{i=1}^{k}{>}0$ such that \n\\begin{align}\n\\sum_{i=1}^{k}\\blam_{i} & =1,\\label{eq:lam_sum1}\\\\\n\\sum_{i=1}^{k}\\blam_{i}\\nabla f_{i}(\\bx) & =0.\\label{eq:lam_sum0}\n\\end{align}","sha256_utf8":"35cbb785618786f59686dfab6c2532434e3581c547d5a3c579f92d287de835de"},{"start":42918,"end":43781,"start_line":310,"end_line":312,"raw_tex":"As we will see, the Lagrange multipliers of Survey Descent subproblems will lie ``close'' to $\\left\\{ \\blam_{i}\\right\\}_{i=1}^{k}{>}0$, while the subproblem outputs $\\left\\{s_i^+\\right\\}_{i=1}^k$ will lie near  $\\bx$. To characterize the order-of-magnitude of these distances, we adopt the ``Big-Oh'' notation. In particular, for a mapping $g:\\mathbf{E}{\\to}\\F$ between two Euclidean spaces and letting $\\left\\| \\cdot\\right\\|^p$ denote an arbitrary Euclidean norm raised to the $p$-th power, we use the notation \n$g\\left(x\\right)=O\\left(\\left\\| x\\right\\| ^{p}\\right)$\nto indicate the property that there exists a constant $K{>}0$ such that $ \\left\\| g\\left(x\\right)\\right\\|{\\leq}K\\left\\| x\\right\\| ^{p}$ holds for all small $x$. By itself, we let $\\mathrm{O}\\left(\\left\\| x\\right\\| ^{p}\\right)$ denote an element of the class of all functions with this property.\n","sha256_utf8":"6606dc121599ede97ddd4da0f1eeb5c2a974d150cd7962d414b0bece5651ae56"},{"start":43878,"end":44226,"start_line":316,"end_line":319,"raw_tex":"\\begin{setting} \\label{setting:localSD} \\textbf{(Local Analysis of Survey Descent)}\nConsider an iteration of Survey Descent on a strong $\\C^2$ max function objective, $f$, with a survey \n$\\S$ that is valid and sufficiently close to $\\bX$. For $i{=}1{,}\\dots{,}k$, denote the output of the $i$-th Survey Descent subproblem by $s_i^+$.\n\\end{setting}\n","sha256_utf8":"353b4a3067deab5509bd1e4c66f72621ce42306c4d0603d2c2dbb97c0c55c53f"},{"start":45896,"end":46025,"start_line":330,"end_line":332,"raw_tex":"\\begin{equation}\\label{eq:valS}\n\\nabla f\\left(s_{i}\\right)=\\nabla f_{i}\\left(s_{i}\\right) \\ \\forall \\ i=1,\\dots,k,\n\\end{equation}","sha256_utf8":"46406626f54677188e6b60cc6b7ab2e52483cca8be0fc4f0b1f9e9108de7d91d"},{"start":46068,"end":46689,"start_line":334,"end_line":340,"raw_tex":"\\begin{observe}\n\\textbf{(Survey Descent Iterations on Max-of-Smooth Objectives)\\label{obs:SD_MOS}} Assume that $\\S$ is a valid survey for the objective $f$. Then, for all $i$, the $i$-th subproblem $\\left(P_{i}^{\\S}\\right)$ of Survey Descent (Definition \\ref{def:SD_Method}) is equivalent to \\begin{equation}\\begin{aligned}\\min_{x}\\ & \\left\\| x-\\left(s_{i}-\\frac{1}{L}\\nabla f_{i}\\left(s_{i}\\right)\\right)\\right\\| _{2}^{2}\\\\\n\\mathrm{s.t.}\\ & \\ell_{s_j}^{f_j}(x)+\\frac{L}{2}\\left\\| x-s_{j}\\right\\| _{2}^{2}\\leq \\ell_{s_i}^{f_i}(x)\\ \\forall\\ j\\neq i.\n\\end{aligned}\n\\label{eq:raw_form_analytic}\n\\end{equation}\n\\end{observe}\n","sha256_utf8":"189c4d60598030ab4b3adf58b00e3b48dbe61b1acc29be4536b2087fc956ba4f"},{"start":48888,"end":49276,"start_line":371,"end_line":375,"raw_tex":"\\begin{align}\n\\U & \\equiv \\bx + \\Span\\left\\{ \\nabla f_{i}(\\bx)-\\nabla f_{j}(\\bx):1\\leq i,j\\leq k\\right\\} ^{\\perp} \\label{eq:Udef1}\\\\\n & = \\bx +  \\left\\{ \\sum_{i}^k\\gamma_{i}\\nabla f_{i}(\\bx):\\sum_{i}^k\\gamma_{i}=0\\right\\} ^{\\perp} \\label{eq:Udef2}\\\\\n & = \\bx +  \\left\\{x\\in \\R^n: \\ \\nabla f_i(\\bx)^T x = \\nabla f_j(\\bx)^T x \\ \\forall \\ 1\\leq i,j\\leq k\\right\\}.\\label{eq:Udef3}\n\\end{align}","sha256_utf8":"150cfbc78bd9d98aa8abb7377946b0ec183725cb1e009a55f9cdfde30295bae5"},{"start":49984,"end":50136,"start_line":378,"end_line":380,"raw_tex":"\\begin{equation}\n    a_i \\equiv \\nabla f_i(\\bx) \\text{ and } A_i \\equiv \\nabla^2 f_i(\\bx) \\ \\forall \\  i=1,\\dots,k. \\label{eq:A_notation}\n\\end{equation}","sha256_utf8":"ee5ba6bad11a0a5ce9e383447c9497cdd26dd2560574337981f6430acb947ed2"},{"start":50302,"end":50461,"start_line":381,"end_line":383,"raw_tex":"\\begin{equation}\n    f_{i}\\left(x\\right)= f(0) + a_{i}^{T}x+\\frac{1}{2}x^{T}A_{i}x+r_{i}\\left(x\\right) \\ \\forall \\ i=1,\\dots,k,\\label{eq:taylor}\n\\end{equation}","sha256_utf8":"f486d1e713e52bff8aabeb9a0072f9f5b01e0fb94e8fb7000f642a2890b2acc9"},{"start":50462,"end":50501,"start_line":384,"end_line":384,"raw_tex":"where $r_i(x)$ is a residual function.\n","sha256_utf8":"4308aeb7c8b5676ed23904370f221855ea260ecdfb646a21b43390332be0650f"},{"start":62729,"end":62837,"start_line":535,"end_line":537,"raw_tex":"\\begin{equation}\n    \\blam_{\\min} \\equiv \\min_{i=1,\\dots,k} \\blam_{i} > 0. \\label{eq:min_lam}\n\\end{equation}","sha256_utf8":"46268577299e542f614861b685a6fffe3e8887e0bfbfadf78e9b7f276e760947"},{"start":66400,"end":66906,"start_line":575,"end_line":582,"raw_tex":"\\begin{procedure}\\textbf{(Repeated Iterations of Survey Descent) }\\label{procedure:SD}\nFor the strong $\\C^2$ max function objective $f$, assume $\\S^{0}$ is a valid initial survey sufficiently close to $\\bX$, and initialize $t=0$. Iterate the following steps:\n\\begin{enumerate}\n\\item Solve the Survey Descent iteration $\\left\\{ \\left(P_{i}^{\\S^{t}}\\right)\\right\\} _{i=1}^{k}$\nand denote the output $\\S^{t+1}\\equiv\\left(\\S^{t}\\right)^{+}$.\n\\item Increment $t\\leftarrow t+1$. \n\\end{enumerate}\n\\end{procedure}\n","sha256_utf8":"d123909f73be8cd5e1e33c3b7f4e1c29bfd829f27e60046b87036c5324af8e5d"},{"start":68016,"end":68348,"start_line":591,"end_line":593,"raw_tex":"\\begin{observe}\\label{obs:allT}\nThe proof of Corollary \\ref{cor:viable} more generally shows that, for all $t \\in \\nonneg$, all surveys $\\S^{t}$ will remain sufficiently close to $\\bX$. Thus, the conclusions from Theorems \\ref{thm:SD_MBP}-\\ref{thm:s_monotone} apply to every iteration of Procedure \\ref{procedure:SD}.\n\\end{observe}\n","sha256_utf8":"ab8921e6e5220fc0b33515f1b29fe95c28d50c1e7c32ab2ca820dea7bf555c85"},{"start":95766,"end":96571,"start_line":881,"end_line":891,"raw_tex":"\\begin{defn}\n    \\textbf{(Multiple Balls Problem, $\\MBP\\param$)\\label{def:MBP}} \\sloppy For some\n    target vector $y\\in\\R^{n}$, ball centers $Z=\\left[z_{1},\\dots,z_{\\bk}\\right]\\in\\R^{n\\times\\bk}$\n    where $\\bk\\leq n$, and non-negative radii $r=\\left[r_{1},\\dots,r_{\\bk}\\right]^{T}\\in\\R^{\\bk}_{+}$,\n    define the \\textit{Multiple Balls Problem (MBP)} as \n    \\begin{align}\n    \\min_{x\\in\\R^{n}}\\ & \\left\\| x-y\\right\\| _{2}^{2} \\nonumber\\\\\n    \\mathrm{s.t.}\\ & \\left\\| x-z_{j}\\right\\| _{2}^{2} = r_{j}^{2}\\ \\forall\\ j=1,\\dots,\\bk. \\label{eq:MBP_cons}\n    \\end{align}\n    Denote the problem as $\\MBP\\param$ and its \\textit{feasible region} as $\\B\\spec$. We will refer to the tuple $\\param$ as the \\textit{parameters} and the sub-tuple $\\spec$ as the \\textit{ball-specifications} of the MBP.\n    \\end{defn}","sha256_utf8":"d14640e18076985a7371fc03acf58fbbc0b0a63a515eccc9a5cbbf8a7a6c3b87"},{"start":96847,"end":97300,"start_line":895,"end_line":895,"raw_tex":"    Since $y$, $Z$, and $r$ all reside in Euclidean spaces, we can also consider $\\spec$ and $\\param$ elements of some product space. Let $\\left\\Vert \\cdot\\right\\Vert $ denote an arbitrary Euclidean norm. This will allow us to formalize the notion of distance in of parameters and ball-specifications in their respective spaces. We are particularly interested in the behavior of $\\MBP\\param$ where $\\param$ is near some set of \\textit{base parameters}.\n","sha256_utf8":"967f4efae6abcda2c6215c5ae023bd5f34062dcfe7d2ea5708d47c3408d7d3f8"},{"start":97309,"end":97896,"start_line":897,"end_line":908,"raw_tex":"\\begin{defn}\n    \\textbf{(Base Parameters)\\label{def:base}} We refer to a set of parameters\n    as \\textit{base parameters} and denote them as $\\bparam$\n    if they satisfy the following properties:\n    \\begin{itemize}\n    \\item The columns of $\\bZ=\\left[\\bz_{1},\\dots,\\bz_{\\bk}\\right]$ are linearly\n    independent.\n    \\item $\\br_{j}^{2}=\\left\\| \\bz_{j}\\right\\| _{2}^{2}$.\n    \\item There exist strictly-positive weights $\\clam=\\left[\\clam_{1},\\dots,\\clam_{\\bk}\\right]^{T}\\in\\R_{++}^{\\bk}$\n    such that $\\bar{y}=-\\sum_{j=1}^{\\bk}\\clam_{j}\\bar{z}_{j}.$\n    \\end{itemize}\n    \\end{defn}","sha256_utf8":"e33ea561f0eeda0430468d673c70890f1faa3219e998c4e751f705928cf9056d"},{"start":98933,"end":99732,"start_line":916,"end_line":930,"raw_tex":"    \\begin{itemize}\n        \\item The \\textit{central affine subspace}\n            \\begin{equation}\n            \\A\\spec \\equiv \\left\\{ x\\in\\R^{n}:\\ 2z_{j}^{T}x-\\left\\| z_{j}\\right\\| _{2}^{2}+r_{j}^{2}\\text{ equal  for all } j=1,\\dots,\\bk\\right\\}. \\label{eq:CAS}\n            \\end{equation}\n        \\item The \\textit{core}\n            \\begin{equation}\n            c\\spec\\equiv\\projA{\\spec}\\left(z_{j}\\right)\\ \\text{ for any }\\ j=1,\\dots,\\bk, \\label{eq:core}\n            \\end{equation}\n            where $\\projA{\\spec}$ denotes the projection onto $\\A\\spec$.\n        \\item The \\textit{margin}\n            \\begin{equation}\n            \\rho\\spec\\equiv\\sqrt{r_{j}^{2}-\\left\\| z_{j}-c\\spec\\right\\| _{2}^{2}} \\ \\text{ for any }\\ j=1,\\dots,\\bk. \\label{eq:margin}\n            \\end{equation}\n    \\end{itemize}\n","sha256_utf8":"645bf46f7500005c739ca022e8e933e81d7e95b490c7df19a7702feaa6f95dd8"},{"start":113031,"end":113889,"start_line":1102,"end_line":1105,"raw_tex":"    In this section, we will show that the optimal solutions of MBPs are unchanged when the equality constraints \\eqref{eq:MBP_cons} are relaxed to inequalities. In particular, regardless of whether $\\MBP\\param$ is defined with equality or inequality constraints, note that the stationarity conditions remain the same. Therefore, there must exist some Lagrange multipliers, $\\lambda\\param\\in\\R^m$, such that\\begin{equation}\n    x\\param-y+\\sum_{j=1}^{\\bk}\\lambda_{j}\\param\\left[x\\param-z_{j}\\right]=0, \\label{eq:MBP_FOC}\n    \\end{equation}\n    where $\\lambda_{j}\\param$ denotes the $j$-th element of $\\lambda\\param$. The only difference is that the inequality version possesses additional dual feasibility conditions requiring the non-negativity of $\\lambda\\param$. Lemma \\ref{lem:MBP_lam} will show that the $\\lambda\\param$ are unique and strictly positive.\n","sha256_utf8":"ce9ee413aa325f128c062efae374deebca0f97bfebff7e6f4b4aea290b40af19"},{"start":119620,"end":119824,"start_line":1167,"end_line":1168,"raw_tex":"    We can reformulate Survey Descent subproblems into inequality-constrained MBPs. In particular, for all $i$, we can rewrite the $i$-th Survey Descent subproblem in Observation \\ref{obs:SD_MOS} as\n    \n","sha256_utf8":"a6d772275b75cda52042f009be1e05ae74c7459759e32b92ceee3ec4cdaec9a4"},{"start":119828,"end":120118,"start_line":1169,"end_line":1173,"raw_tex":"\\begin{gather}\\begin{aligned}\\min_{x}\\ & \\left\\| (x - s_{i}) - y_{i}\\left(\\S\\right) \\right\\| _{2}^{2}\\\\\n    \\mathrm{s.t.}\\ & \\left\\| (x-s_{i})-z_{ij}\\left(\\S\\right)\\right\\| _{2}^{2}\\leq r_{ij}^{2}\\left(\\S\\right)\\ \\forall\\ j\\neq i,\n    \\end{aligned}\n    \\label{eq:proj_form}\n    \\end{gather}","sha256_utf8":"aa7d5c21a4d4d1da20d9410a1a6e5ee3e86c8aa1341ee40e857309dc1f931aa1"},{"start":120119,"end":120266,"start_line":1174,"end_line":1175,"raw_tex":"    where, for all $i$ and $j$, we define the following mappings using the components $\\{f_i\\}_{i=1}^k$ from Definition \\ref{def:strong_max}:\n    \n","sha256_utf8":"19fcc73533831e9abfc44138e8e44eb6a9919f16503fe2e1a93522d8aecd8003"},{"start":120270,"end":120764,"start_line":1176,"end_line":1181,"raw_tex":"\\begin{gather}\\begin{aligned}\n    y_{i}\\left(\\S\\right)& \\equiv-\\frac{1}{L}\\nabla f_{i}\\left(s_{i}\\right),\\\\\n    z_{ij}\\left(\\S\\right) & \\equiv s_j - s_i  + y_{j}\\left(\\S\\right)-y_{i}\\left(\\S\\right),\\\\\n    r_{ij}^{2}\\left(\\S\\right) & \\equiv\\frac{2}{L}\\left[f_{i}\\left(s_{i}\\right)-f_{j}\\left(s_{j}\\right)+\\nabla f_{i}\\left(s_{i}\\right)^{T}\\left(s_{j}-s_{i}\\right)\\right]+\\left\\|y_{j}\\left(\\S\\right)-y_{i}\\left(\\S\\right)\\right\\| _{2}^{2}.\n    \\end{aligned}\\label{eq:MBP_mappings}\n    \\end{gather}","sha256_utf8":"02adef7ab486fcf7e277f6e332e86cd6d1c166cf58500a5466507000646615e6"},{"start":120765,"end":122303,"start_line":1182,"end_line":1191,"raw_tex":"    \\begin{observe}\\textbf{(Survey Descent as Inequality-Constrained MBP)} \\label{obs:SD_as_MBP} Consider any $k$-point survey $\\S$ and any $i{=}1,\\dots,k$. After the changes-of-variables $x \\leftarrow x{-} s_i$, the minimization problem in \\eqref{eq:proj_form} is an inequality-constrained MBP  with $\\m = k{-}1$ and parameters \n    \n    $$\\param = \\left(y_{i}\\left(\\S\\right), \\left[ z_{ij}\\left(\\S\\right)\\right]_{j\\neq i}, \\left[r_{ij}\\left(\\S\\right)\\right]_{j\\neq i},\\left[ s_{j}\\right]_{j\\neq i}\\right),$$\n    where, for any $j$-indexed scalars $\\left\\{v_j\\right\\}_{j=1}^{k}$, we use the notation $\\left[v_j\\right]_{j \\neq i}{\\equiv} \\left[v_1,\\dots,v_{i-1},v_{i+1},\\dots,v_{k}\\right]^T$ to denote the length $k{-}1$ vector excluding $v_i$.\n    Furthermore, the mappings \\eqref{eq:MBP_mappings}  evaluated at the base survey\\footnote{Evaluating of the mappings \\eqref{eq:MBP_mappings} at $\\bX$ is a technical step. $\\bX$ is not a valid survey (Definition \\ref{def:validS}), so it does not correspond to an actual Survey Descent iteration.} $\\bX$ (Definition \\ref{def:validS}) form a set of base parameters\n    \\begin{equation}\n        \\bparam = \\left(y_{i}\\left(\\bX\\right), \\left[ z_{ij}\\left(\\bX\\right)\\right]_{j\\neq i}, \\left[r_{ij}\\left(\\bX\\right)\\right]_{j\\neq i}, [\\bx]_{j\\neq i}\\right),\\label{eq:MBP_SD_base}\n    \\end{equation}\n    with corresponding weights $\\{\\blam_j\\}_{j\\neq i}$ from \\eqref{eq:lam_sum1}-\\eqref{eq:lam_sum0}, and where $[\\bx]_{j\\neq i}$ is the $k{-}1$ length vector with all $\\bx$ entries.\n    \\end{observe}\n","sha256_utf8":"601f87fd665460571a5e1d43b83ff7aebcbce330faeeeb168ff8b653f2e11986"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{thm}\n\\textbf{(Q-Linear Convergence of Survey Points)\\label{thm:q_linear}} Procedure \\ref{procedure:SD} satisfies the following property:\\begin{equation}\n\\underset{t\\to\\infty}{\\limsup}\\frac{\\left\\| \\S^{t+1} - \\bX \\right\\| ^{2}}{\\left\\| \\S^{t}  - \\bX \\right\\| ^{2}}\\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right),\\label{eq:s_result}\n\\end{equation}\nwhere $\\blam_{\\min}$ is defined as in \\eqref{eq:min_lam}.\n\\end{thm}","units":[{"start":68514,"end":68934,"start_line":597,"end_line":602,"raw_tex":"\\begin{thm}\n\\textbf{(Q-Linear Convergence of Survey Points)\\label{thm:q_linear}} Procedure \\ref{procedure:SD} satisfies the following property:\\begin{equation}\n\\underset{t\\to\\infty}{\\limsup}\\frac{\\left\\| \\S^{t+1} - \\bX \\right\\| ^{2}}{\\left\\| \\S^{t}  - \\bX \\right\\| ^{2}}\\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right),\\label{eq:s_result}\n\\end{equation}\nwhere $\\blam_{\\min}$ is defined as in \\eqref{eq:min_lam}.\n\\end{thm}","sha256_utf8":"8156d01578966ac60bb468c4034f228742c56e40cfc531fc8348559f2c8a364c"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{restatable}{thm}{PBmbp}\\label{thm:SD_MBP} \\textbf{(Local Feasibility, Uniqueness, Tightness, and Smoothness of Survey Descent)} Assume Setting \\ref{setting:localSD}. Then for all $i$, the Survey Descent subproblem $\\left(P_{i}^{\\S}\\right)$ is feasible and has a unique solution $s_{i}^{+}$, which satisfies all the constraints with equality, with unique associated Lagrange multipliers $\\lambda_{j}^i$ \\edit{for the $j$-th constraint within the $i$-th subproblem}  (for $j\\neq i$). The solutions and multipliers depend smoothly on the input survey $\\S$, and satisfy\\begin{equation}\ns_{i}^{+}\\left(\\S\\right) ~=~\\bx+\\mathrm{O}\\left(\\left\\| \\S-\\bX\\right\\| \\right) \\qquad \\text{and} \\qquad \\lambda_{j}^i\\left(\\S\\right) ~=~ \\blam_{j}+\\mathrm{O}\\left(\\left\\| \\S-\\bX\\right\\| \\right)\\ \\forall\\ j\\neq i.\\label{eq:SD_forms}\n\\end{equation}\n\\end{restatable}\n\n\\begin{proof}\nFollows from a routine analysis of the subproblem's first-order optimality conditions combined with the implicit function theorem.\n\\end{proof}\n\n\\begin{thm} \\label{thm:strictval} \\textbf{(Preservation of Validity)} Assuming Setting \\ref{setting:localSD}, the output survey $\\S^{+}$ of a Survey Descent iteration is valid.\n\\end{thm}\n\n\\begin{proof}\nConsider any fixed $i$. If $s_{i}^{+}=s_{i}$, then the fact\nthat $f_{i}\\left(s_{i}^{+}\\right){>}f_{j}\\left(s_{i}^{+}\\right)$ for all $j\\neq i$ immediately follows since Theorem \\ref{thm:SD_MBP} assumes $\\S$ is valid.\n\nNow, consider the $s_{i}^{+}{\\neq}s_{i}$ case. By Theorem \\ref{thm:SD_MBP}, $s_{i}^{+}$ satisfies the constraints of $\\left(P_{i}^{\\S}\\right)$ with equality. Using Observation \\ref{obs:SD_MOS}, we express this as\\begin{equation}\n\\ell_{s_i}^{f_i}(s_i^+) = \\ell_{s_j}^{f_j}(s_i^+)+\\frac{L}{2}\\left\\| s_{i}^{+}-s_{j}\\right\\|_{2}^{2}\\ \\forall\\ j\\neq i.\\label{eq:left_valid}\n\\end{equation}\nThe $\\delta$-strong convexity of $f_{i}$ implies\n\\begin{equation*}\nf_{i}\\left(s_{i}^{+}\\right)\\geq\\ell_{s_i}^{f_i}(s_i^+)+\\frac{\\delta}{2}\\left\\| s_{i}^{+}-s_{i}\\right\\| _{2}^{2}>\\ell_{s_i}^{f_i}(s_i^+),\n\\end{equation*}\nwhere the strict inequality follows from $s_{i}^{+}{\\neq}s_{i}$. Next, by the $L$-smoothness of $\\left\\{f_j\\right\\}_{j=1}^k$, the right-hand side of \\eqref{eq:left_valid} is a $s_j$-centered quadratic upper bound of $f_{j}$ evaluated at $s_i^+$. Thus,\n\\begin{equation*}\n\\ell_{s_j}^{f_j}(s_i^+)+\\frac{L}{2}\\left\\| s_{i}^{+}-s_{j}\\right\\|_{2}^{2} \\geq f_{j}\\left(s_{i}^{+}\\right) \\ \\forall\\ j\\neq i.\n\\end{equation*}\n\n\\noindent Therefore, \n\\begin{equation*}\nf_{i}\\left(s_{i}^{+}\\right) > \\ell_{s_i}^{f_i}(s_i^+) = \\ell_{s_j}^{f_j}(s_i^+)+\\frac{L}{2}\\left\\| s_{i}^{+}-s_{j}\\right\\|_{2}^{2} \\geq f_{j}\\left(s_{i}^{+}\\right) \\ \\forall\\ j\\neq i.\n\\end{equation*}\nThis completes the proof.\n\\end{proof}\n\n\\begin{lem}\n\\label{lem:r_grad} For all $i$, the residual function $r_i(x)$ in \\eqref{eq:taylor} satisfies $r_i(x)=\\mathrm{O}\\left(\\|x\\|^2\\right)$ and $\\nabla r_{i}\\left(x\\right)=\\mathrm{O}\\left(\\left\\| x\\right\\|\\right)$.\n\\end{lem}\n\n\\begin{proof}\n$r_i(x)=\\mathrm{O}\\left(\\|x\\|^2\\right)$ follows directly from Taylor's Theorem. Next, observe that \n\\begin{equation*}\n    r_{i}\\left(x\\right)=f_{i}\\left(x\\right)-\\left(f(0) + a_{i}^{T}x+\\frac{1}{2}x^{T}A_{i}x\\right).\n\\end{equation*}\nThus, since $f_i$ is $\\C^2$, the residual $r_i$ must also be $\\C^2$, so its gradient and Hessian are well-defined. This implies $\\nabla r_{i}\\left(x\\right)$ is then $\\C^{1}$. Taylor expanding $\\nabla r_i$ around zero then gives the desired $\\nabla r_{i}\\left(x\\right)= \\mathrm{O}\\left(\\| x\\|\\right)$\nafter observing that $\\nabla r_i(0)$ is the zero-vector and $\\nabla^2 r_i(0)$ is the zero-matrix (since $r_i$ is the residual of the Taylor expansion of $f_i$ around zero).\n\\end{proof}\n\n\\begin{lem}\\label{lem:U_component} Assume Setting \\ref{setting:localSD} and suppose $\\bx{=}0$. Then, $s_{i}^{+}$ satisfies\n\\begin{equation*}\n\\proj\\left[s_{i}^{+}\\right]=\\frac{1}{2-\\blam_{i}}\\proj\\left[s_{i}-\\frac{\\blam_{i}}{L}\\nabla f_{i}\\left(s_{i}\\right)\\right]+\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\proj\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right]+\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right) \\ \\forall \\ i=1,\\dots,k.\n\\end{equation*}\nwhere $\\proj$ is the projection operator onto the $\\U$-subspace \\eqref{eq:Udef1}.\n\\end{lem}\n\n\\begin{proof}\nWithout loss of generality, assume $i=k$. Scale the objective of problem \\eqref{eq:raw_form_analytic} by $\\frac{L}{2}$ and consider the equivalent problem\n\\begin{equation}\\begin{aligned}\\min_{x}\\ & \\frac{L}{2} \\left\\| x-\\left(s_{k}-\\frac{1}{L}\\nabla f_{k}\\left(s_{k}\\right)\\right)\\right\\| _{2}^{2}\\\\\n\\mathrm{s.t.}\\ & \\ell_{s_j}^{f_j}(x)+\\frac{L}{2}\\left\\| x-s_{j}\\right\\| _{2}^{2}\\leq \\ell_{s_k}^{f_k}(x)\\ \\forall\\ j\\neq k.\n\\end{aligned}\\label{eq:scaledPk}\n\\end{equation}\nBy Theorem \\ref{thm:SD_MBP} and Observation \\ref{obs:SD_MOS}, there exists unique Lagrange multipliers $\\left\\{\\lambda_j\\right\\}_{j=1}^{k-1}$ (we omit the $k$-superscript for simplicity) satisfying the following first-order optimality condition for \\eqref{eq:scaledPk}:\n$$\n    0=L\\left(s_{k}^{+}-\\left(s_{k}-\\frac{1}{L} \\nabla f_{k}\\left(s_{k}\\right)\\right)\\right)+\\sum_{j=1}^{k-1} \\lambda_{j}\\left(\\edit{\\nabla f_{j}\\left(s_{j}\\right)}+L\\left(s_{k}^{+}-s_{j}\\right)-\\nabla f_{k}\\left(s_{k}\\right)\\right)\n    $$\nAfter rearranging this gives\\begin{equation}\n0= L\\left(1+\\sum_{j=1}^{k-1}\\lambda_{j}\\right)s_{k}^{+}-L\\left(s_{k} + \\sum_{j=1}^{k-1} \\lambda_{j} s_{j}\\right)+\\nabla f_k(s_k) +\\sum_{j=1}^{k-1}\\lambda_{j}\\left(\\nabla f_j(s_j) - \\nabla f_k(s_k)\\right).\n\\label{eq:FOC2}\n\\end{equation}\nDefine the following scalars:\n\\begin{align}\n    \\lambda_{k} & \\equiv  1-\\sum_{j=1}^{k-1}\\lambda_{j},\\label{eq:lambda}\\\\\n    \\mu_{j} & \\equiv  \\blam_{j}-\\lambda_{j} \\ \\forall \\ j=1,\\dots,k. \\label{eq:mu}\n\\end{align}\nFor $j\\neq k$, Theorem \\ref{thm:SD_MBP} gives\n$\\mu_{j}=\\mathrm{O}\\left(\\left\\| \\S\\right\\| \\right)$ .\nFor $j=k$, combining \\eqref{eq:lambda}-\\eqref{eq:mu} with equation \\eqref{eq:lam_sum1} implies\n\\begin{equation*}\n\\mu_{k} =\\blam_{k}-\\left(1-\\sum_{j=1}^{k-1}\\lambda_{j}\\right) =\\blam_{k}-\\left(1-\\sum_{j=1}^{k-1}\\blam_{j}\\right)+\\mathrm{O}\\left(\\left\\| \\S\\right\\| \\right) =\\mathrm{O}\\left(\\left\\| \\S\\right\\| \\right).\n\\end{equation*}\nThus,\\begin{equation}\n    \\mu_j = \\mathrm{O}(\\|\\S\\|) \\ \\forall \\ j=1,\\dots,k.\\label{eq:mu_size}\n\\end{equation}\nSubstituting \\eqref{eq:mu} into \\eqref{eq:FOC2}, applying identities \\eqref{eq:lam_sum1}-\\eqref{eq:lam_sum0}, collecting $\\mathrm{O}\\left(\\|\\S\\|^2\\right)$ terms, and solving for $s_k^{+}$ gives\\begin{equation}s_{k}^{+}= \\frac{1}{2-\\blam_{k}}\\left[s_{k}-\\frac{\\blam_{k}}{L}\\nabla f_{k}\\left(s_{k}\\right)\\right]+\\sum_{j=1}^{k-1}\\frac{\\blam_{j}}{2-\\blam_{k}}\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right]+\\frac{1}{L\\left(2-\\blam_{k}\\right)}\\sum_{j=1}^{k}\\mu_{j}\\nabla f_j(s_j) +\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right).\\label{eq:pre_sform}\n\\end{equation}\nAdopting the notation \\eqref{eq:A_notation} and applying the Taylor expansion \\eqref{eq:taylor}, observe\\begin{equation}\n    \\sum_{j=1}^{k}\\mu_{j}\\nabla f_j(s_j) = \\sum_{j=1}^{k}\\mu_{j}\\left(a_j + A_j s_j + \\nabla r_j (s_j)\\right) = \\sum_{j=1}^{k}\\mu_{j}a_j +\\mathrm{O}\\left(\\left\\| \\S\\right\\|^{2}\\right),\\label{eq:sum_muA}\n\\end{equation}\nwhere last equality follows from \\eqref{eq:mu_size} and Lemma \\ref{lem:r_grad}. Substituting \\eqref{eq:sum_muA} into \\eqref{eq:pre_sform} then leads to\\begin{equation}\ns_{k}^{+}=\\frac{1}{2-\\blam_{k}}\\left[s_{k}-\\frac{\\blam_{k}}{L}\\nabla f_{k}\\left(s_{k}\\right)\\right]+\\sum_{j=1}^{k-1}\\frac{\\blam_{j}}{2-\\blam_{k}}\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right]+\\frac{1}{L\\left(2-\\blam_{k}\\right)}\\underbrace{\\sum_{j=1}^{k}\\mu_{j}a_j}_{(\\star)} +\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right).\n\\label{eq:s_form}\n\\end{equation}\nNext, again combine \\eqref{eq:lambda}-\\eqref{eq:mu} with equation \\eqref{eq:lam_sum1} to deduce\n\\begin{equation*}\n\\sum_{j=1}^{k}\\mu_{j}=\\sum_{j=1}^{k}\\left(\\blam_{j}-\\lambda_{j}\\right)=0.\n\\end{equation*}\nThen, it follows from \\eqref{eq:Udef2} that $\\proj\\left[(\\star)\\right]=0$.\nThe desired result then follows from applying $\\proj$ to \\eqref{eq:s_form}.\n\\end{proof}\n\n\\begin{lem}\\label{lem:Up_component}\nAssume Setting \\ref{setting:localSD}, and let $\\bx{=}0$. Then, $s_{i}^{+}$ satisfies\n\\begin{equation*}\n\\projp\\left[s_{i}^{+}\\right]=\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right) \\ \\forall \\ i=1,\\dots,k.\n\\end{equation*}\nwhere $\\projp$ is the projection operator onto the orthogonal complement of the $\\U$-subspace \\eqref{eq:Udef1}.\n\\end{lem}\n\n\\begin{proof}\nConsider any fixed $i$. By Theorem \\ref{thm:SD_MBP}, the constraints of $\\left(P_{i}^{\\S}\\right)$\nare tight. Thus, by Observation \\ref{obs:SD_MOS}, $s_{i}^{+}$ satisfies\n\\begin{equation*}\n    f_{j}\\left(s_{j}\\right)+\\nabla f_{j}\\left(s_{j}\\right)^{T}\\left(s_i^+-s_{j}\\right)+\\frac{L}{2}\\left\\| s_i^+-s_{j}\\right\\| _{2}^{2} = f_{i}\\left(s_{i}\\right)+\\nabla f_{i}\\left(s_{i}\\right)^{T}\\left(s_i^+-s_{i}\\right) \\ \\forall \\ j\\neq i.\n\\end{equation*}\nUsing equation \\eqref{eq:taylor} in the above gives, for all $j\\neq i$,\n\\begin{multline*}\n    f(0) + a_{j}^{T}s_{j}+\\frac{1}{2}s_{j}^{T}A_{j}s_{j}+r_{j}\\left(s_j\\right)+\\left(a_{j}+A_{j}s_{j} + \\nabla r_j(s_j)\\right)^T\\left(s_{i}^{+}-s_{j}\\right)+\\frac{L}{2}\\left\\Vert s_{i}^{+}-s_{j}\\right\\Vert _{2}^{2}\\\\\n    =f(0) + a_{i}^{T}s_{i}+\\frac{1}{2}s_{i}^{T}A_{i}s_{i}+r_{i}\\left(s_i\\right)+\\left(a_{i}+A_{i}s_{i}+\\nabla r_i(s_i)\\right)^T\\left(s_{i}^{+}-s_{i}\\right).\n\\end{multline*}\nAfter simplification and rearrangement, we obtain\\begin{equation}\n\\left(a_{j}-a_{i}\\right)^{T}s_{i}^{+}=-\\left(A_{j}s_{j}-A_{i}s_{i}+\\nabla r_{j}\\left(s_{j}\\right)-\\nabla r_{i}\\left(s_{i}\\right)\\right)^{T}s_{i}^{+}-\\frac{L}{2}\\left\\| s_{i}^{+}-s_{j}\\right\\| _{2}^{2}-c_{j}\\left(s_{j}\\right)+c_{i}\\left(s_{i}\\right)  \\ \\forall \\ j\\neq i, \\label{eq:s_constr}\n\\end{equation}\nwhere we define the functions\n\\begin{equation*}\nc_{j}\\left(x\\right)\\equiv-\\frac{1}{2}x^{T}A_{j}x+r_{j}\\left(x\\right)-\\nabla r_{j}\\left(x\\right)^{T}x \\ \\forall \\ j=1,\\dots,i.\n\\end{equation*}\nBy Lemma \\ref{lem:r_grad}, $c_j(x)$ is $O\\left(\\|x\\|^2\\right)$. Since we assume $\\bx{=}0$, by Theorem \\ref{thm:SD_MBP}, $s_{i}^{+}=\\mathrm{O}\\left(\\left\\| \\S\\right\\| \\right)$.\nWe then deduce that all terms on the right-hand side of \\eqref{eq:s_constr} are $\\mathrm{O}\\left(\\left\\| \\S\\right\\|^2 \\right)$:\n\\begin{equation}\n\\left(a_{j}-a_{i}\\right)^{T}s_{i}^{+}=\\mathrm{O}\\left(\\left\\| \\S\\right\\|^2 \\right) \\ \\forall \\ j\\neq i.\\label{eq:projsp}\n\\end{equation}\n% Consider the following $QR$-decomposition \n% \\[\n% \\begin{bmatrix}\\left(a_{1}-a_{i}\\right), & \\dots, & \\left(a_{i-1}-a_{i}\\right)\\end{bmatrix}\\equiv\\bQ\\bR,\n% \\]\n% where $\\bQ\\in\\R^{n\\times\\left(k-1\\right)}$ is an orthonormal matrix,\n% and $\\bR\\in\\R^{\\left(k-1\\right)\\times\\left(k-1\\right)}$ is an upper-triangular\n% matrix. Then,\n% \\begin{equation}\n% \\bR^{T}\\bQ^{T}s_{i}^{+}=\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right).\\label{eq:s_constr2}\n% \\end{equation}\nBy \\eqref{eq:Udef1},\n\\begin{align*}\n\\U^{\\perp} & =\\Span\\left\\{ a_{i}-a_{j}:1\\leq i,j\\leq k\\right\\} =\\Span\\left\\{ a_{j}-a_{i}:\\ j \\neq i \\right\\}.\n\\end{align*}\nCombined with the affine independence of $\\left\\{ a_{j}\\right\\} _{j=1}^{k}$\n(Definition \\ref{def:strong_max}), the above implies $\\left\\{ a_{j}-a_{i}\\right\\} _{j\\neq i}$ forms a basis for $\\U^{\\perp}$. Therefore, we can express the projection onto $\\U^{\\perp}$ in terms $\\left\\{ a_{j}-a_{i}\\right\\} _{j\\neq i}$. The result now follows from equation \\eqref{eq:projsp} and standard linear algebra.\n% Thus, $\\bR$ is invertible,\n% and we can represent the projection onto $\\U^{\\perp}$ as multiplication by the matrix $\\bQ\\bQ^{T}.$ From this, we deduce\n% \\begin{equation*}\n% \\projp\\left[s_{i}^{+}\\right]=\\bQ\\bQ^{T}s_{i}^{+}=\\bQ\\left(\\bR^{T}\\right)^{-1}\\bR^{T}\\bQ^{T}s_{i}^{+}=\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right),\n% \\end{equation*}\n% where the last equality follows from \\eqref{eq:s_constr2} and the\n% fact that $\\bQ\\left(\\bR^{T}\\right)^{-1}$ is a constant-matrix independent\n% of $\\S$.\n\\end{proof}\n\n\\begin{thm}\n\\label{thm:approx_s} \\textbf{(Gradient Descent Approximation of Survey Updates)} Assume Setting \\ref{setting:localSD}.  Then the solution of the $i$-th Survey Descent subproblem satisfies, for all $i$, \\begin{equation}\ns_{i}^{+}= \\ts_{i}+\\mathrm{O}\\left(\\left\\| \\S - \\bX\\right\\| ^{2}\\right) \\label{eq:SD_iter}\n\\end{equation}\nwhere we define\\begin{equation}\n\\ts_{i}\\equiv\\frac{1}{2-\\blam_{i}}\\proj\\left[s_{i}-\\frac{\\blam_{i}}{L}\\nabla f_{i}\\left(s_{i}\\right)\\right]+\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\proj\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right], \\label{eq:s_tilde}\n\\end{equation}\nand $\\proj$ is the projection operator onto the $\\U$-subspace \\eqref{eq:Udef1}.\n\\end{thm}\n\n\\begin{proof}\nWithout loss of generality, assume that $\\bx{=}0$. Then, the desired result follows from decomposing $s_{k}^{+} = \\proj\\left[s_{k}^{+}\\right] + \\projp\\left[s_{k}^{+}\\right]$ and applying Lemmas \\ref{lem:U_component} and \\ref{lem:Up_component}. We can deduce the general $\\bx{\\neq}0$ case with the change of variables $x{\\leftarrow}(x-\\bx)$.\n\\end{proof}\n\n\\begin{thm}\n\\textbf{\\label{thm:PR}(Projected-GD Behavior)} Assume that $\\S$ is a valid survey for the strong $\\C^2$ max function objective $f$. Then, the following holds for all $i$ and any constant $\\tL \\geq L$:\n\\begin{align*}\n\\left\\| \\proj\\left[s_i - \\frac{1}{\\tL} \\nabla f_i(s_i)\\right] - \\bx\\right\\| _{2}^{2} &\\leq \\left(1-\\frac{\\delta}{\\tL}\\right)\\left\\| s_{i} - \\bx \\right\\| _{2}^{2},\\\\\nf_{i}\\left(\\proj\\left[s_i - \\frac{1}{\\tL} \\nabla f_i(s_i)\\right]\\right) - f_i(\\bx) &\\leq \\frac{\\tL}{2}\\left(1-\\frac{\\delta}{\\tL}\\right)\\left\\| s_{i} - \\bx\\right\\| ^{2}.\n\\end{align*}\n\\end{thm}\n\n\\begin{proof}\nConsider any fixed $i$. It is easy to check that $\\bx = \\arg\\min_{x \\in \\U} f_i(x)$ using first-order optimality conditions combined with \\eqref{eq:lam_sum1}-\\eqref{eq:lam_sum0} and \\eqref{eq:Udef3}. The desired result then follows from classical projected-GD convergence results (given in standard texts such as \\citet[Theorem 10.29]{beck2017}) after observing that $\\U$ is convex and $f_i$ is $L$-smooth and $\\delta$-strongly convex.\n\\end{proof}\n\n\\begin{lem}\n\\label{lem:st_bound} Assume Setting \\ref{setting:localSD}, and suppose $\\bx{=}0$. Then, the following must hold:\n\\begin{equation*}\n\\left\\| \\ts_{i}\\right\\| _{2}^{2}\\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)\\max_{j=1,....,k}\\left\\| s_{j}\\right\\| _{2}^{2} \\ \\forall \\ i=1,\\dots,k,\n\\end{equation*}\nwhere $\\ts_i$ and $\\blam_{\\min}$ are as defined in equations \\eqref{eq:s_tilde} and \\eqref{eq:min_lam}, respectively.\n\\end{lem}\n\n\\begin{proof} Consider any fixed $i$. Then,\n\\begin{align*}\n\\left\\| \\ts_{i}\\right\\| _{2}^{2} & =\\left\\| \\frac{1}{2-\\blam_{i}}\\proj\\left[s_{i}-\\frac{\\blam_{i}}{L}\\nabla f_{i}\\left(s_{i}\\right)\\right]+\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\proj\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right] \\right\\| _{2}^{2}  \\text{ (Substitute \\eqref{eq:s_tilde})}\\\\\n & \\leq\\frac{1}{2-\\blam_{i}}\\left\\| \\proj\\left[s_{i}-\\frac{\\blam_{i}}{L}\\nabla f_{i}\\left(s_{i}\\right)\\right]\\right\\| _{2}^{2}+\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\left\\| \\proj\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right]\\right\\| _{2}^{2}\\text{ (Convexity)}\\\\\n & \\leq\\frac{1}{2-\\blam_{i}}\\left(1-\\frac{\\blam_{i}\\delta}{L}\\right)\\left\\| s_{i}\\right\\| _{2}^{2}+\\left(1-\\frac{\\delta}{L}\\right)\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\left\\| s_{j}\\right\\| _{2}^{2} \\text{ (Theorem \\ref{thm:PR} where $\\tL=\\frac{L}{\\blam_i}$)}\\\\\n & \\leq\\left(1-\\frac{\\blam_{i}\\delta}{L}\\right)\\left[\\frac{1}{2-\\blam_{i}}\\left\\| s_{i}\\right\\| _{2}^{2}+\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\left\\| s_{j}\\right\\| _{2}^{2}\\right]\\\\\n & \\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)\\max_{j=1,....,k}\\left\\| s_{j}\\right\\| _{2}^{2}.\n\\end{align*}\nThis completes the proof.\n\\end{proof}\n\n\\begin{thm} \\textbf{(Survey-Norm Contraction)} \\label{thm:s_monotone} Assume Setting \\ref{setting:localSD}.  Then the following must hold:\n\\begin{equation*}\n\\max_{j=1,...,k}\\left\\| s_{j}^{+} - \\bx \\right\\| _{2}^{2}\\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)\\max_{j=1,...,k}\\left\\| s_{j} - \\bx \\right\\| _{2}^{2}+\\mathrm{O}\\left(\\left\\| \\S - \\bX\\right\\|^{3}\\right),\n\\end{equation*}\nwhere $\\blam_{\\min}$ is defined as in \\eqref{eq:min_lam}.\n\\end{thm}\n\n\\begin{proof}\nWithout loss of generality, assume $\\bx = 0$. Consider any fixed $i$. By Lemma \\ref{lem:st_bound}, $\\ts_{i}=\\mathrm{O}\\left(\\left\\| \\S\\right\\| \\right)$. Then,\\begin{align*}\n\\left\\| s_{i}^{+}\\right\\| _{2}^{2} & =\\left\\| \\ts_{i}+\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right)\\right\\| _{2}^{2} \\text{ (Theorem \\ref{thm:approx_s})}\\\\\n & =\\left\\| \\ts_{i}\\right\\| _{2}^{2}+2\\left\\langle \\ts_{i},\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right)\\right\\rangle +\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{4}\\right)\\\\\n & \\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)\\max_{j=1,....,k}\\left\\| s_{j}\\right\\| _{2}^{2}+\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{3}\\right) \\text{ (Lemma \\ref{lem:st_bound})}.\n\\end{align*}\nThe right-hand side is independent of $i$, so taking the max over $i$ on the left-hand side leads to the desired result. We can deduce the general $\\bx{\\neq}0$ case with the change of variables $x{\\leftarrow}(x-\\bx)$.\n\\end{proof}\n\n\\begin{cor}\\textbf{(Well-Definedness of Survey Descent Repetitions)}\\label{cor:viable} Procedure \\ref{procedure:SD} is well-defined:  for each $t=0,1,2\\ldots$, the Survey Descent iteration $\\left\\{ \\left(P_{i}^{\\S^{t}}\\right)\\right\\} _{i=1}^{k}$ is feasible and produces a valid output survey.\n\\end{cor}\n\n\\begin{proof}\nFor $\\S^{0}$ sufficiently close to $\\bX$, Theorem \\ref{thm:s_monotone} inductively implies that $\\left\\| \\S^{t+1} - \\bX \\right\\| <\\left\\| \\S^{t} - \\bX\\right\\| $ for all $t\\in\\nonneg$. In other words, output surveys will always move closer to $\\bX$, and Theorem \\ref{thm:SD_MBP} and \\ref{thm:strictval} continue to hold for Survey Descent iterations $\\left\\{\\left(P_{i}^{\\S^{t}}\\right)\\right\\}_{i=1}^{k}$ for all $t\\in \\nonneg$. Thus, Theorem \\ref{thm:SD_MBP} implies the feasibility all Survey Descent iterations while Theorem \\ref{thm:strictval} implies the validity of output surveys.\n% The viability of implementing the consecutive Survey Descent iterations with only a first-order oracle on $f$ follows directly from Remark \\ref{rem:SD_FOM} and Theorem \\ref{thm:strictval}.\n\\end{proof}\n\n\\begin{proof}\nBy Theorem \\ref{thm:s_monotone} and Observation \\ref{obs:allT}, there exists $K>0$ such that\n\\begin{equation*}\n\\left\\| \\S^{t+1} - \\bX \\right\\|^{2}\\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)\\left\\| \\S^{t} - \\bX \\right\\|^{2}+K\\left\\| \\S^{t} - \\bX \\right\\|^{3} \\ \\forall \\ t\\in \\nonneg.\n\\end{equation*}\nCorollary \\ref{cor:viable} implies that $\\S^t$ is a valid survey. By Definition \\ref{def:validS}, $\\bX$ is not a valid survey because $f_i(\\bx){=}f_j(\\bx)$ for all $i,j$. Thus, $\\S^t {\\neq}\\bX$, so $\\left\\| \\S^{t} - \\bX \\right\\|^2$ is nonzero and we can divide both sides by it.  We deduce\n\\begin{equation*}\n\\frac{\\left\\| \\S^{t+1} - \\bX \\right\\|^{2}}{\\left\\| \\S^{t} - \\bX \\right\\| ^{2}}\\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)+K\\left\\| \\S^{t} - \\bX\\right\\|  \\ \\forall \\ t\\in \\nonneg.\n\\end{equation*}\n\\sloppy \\edit{For $\\S^{0}$ sufficiently close to $\\bX$, this inequality guarantees that $\\left\\| \\S^{t}{-}\\bX\\right\\|$ monotonically decreases with $t$ and, in particular, $\\left\\| \\S^{t}{-}\\bX\\right\\|{<}\\frac{\\blam_{\\min}\\delta}{LK}$ for all $t\\in\\nonneg$.} Thus, $\\frac{\\left\\| \\S^{t+1} - \\bX \\right\\|}{\\left\\| \\S^{t} - \\bX\\right\\|}{<}1$ implying $\\underset{t\\to\\infty}{\\lim}\\left\\| \\S^{t}{-}\\bX\\right\\|{=}0$. Taking the lim-sup with $t{\\to}\\infty$ on both sides of the above inequality then gives the desired result.\n\\end{proof}\n\n\\begin{lem}\\textbf{(Local Properties of the Central Affine Subspace)}\\label{lem:A}\n    For any ball-specifications $\\spec$ sufficiently close to base ball-specifications $\\bspec$, the following properties hold:\n    \\begin{enumerate}\n        \\item $\\A\\spec$ is non-empty with dimension $n{-}\\bk{+}1$.\n        \\item For all $i\\neq j$, the direction $(z_i - z_j)$ is orthogonal to the linear subspace parallel to $\\A\\spec$.\n        \\item For $w\\in\\R^n$, the projection $\\projA{\\spec}(w)$ is well-defined and varies smoothly with $\\spec$ and $w$.\n    \\end{enumerate}\n    \\end{lem}\n\n\\begin{proof}\n    Observe that we can rewrite \\eqref{eq:CAS} as the following $\\left(\\bk{-}1\\right){\\times}n$ linear system:\\begin{equation}\n    \\A\\spec=\\left\\{ x\\in\\R^{n}:\\ M\\spec x=b\\spec \\right\\} ,\\label{eq:CAS_sys}\n    \\end{equation}\n    where \n    \\[\n    M\\spec\\equiv2\\begin{bmatrix}\\left(z_{2}-z_{1}\\right)^{T}\\\\\n    \\vdots\\\\\n    \\left(z_{\\m}-z_{1}\\right)^{T}\n    \\end{bmatrix}\n    \\text{ and }\n    b\\spec\\equiv\\begin{bmatrix}\\left(\\left\\| z_{2}\\right\\| _{2}^{2}-r_{2}^{2}\\right)-\\left\\| z_{1}\\right\\| _{2}^{2}+r_{1}^{2}\\\\\n    \\vdots\\\\\n    \\left(\\left\\| z_{\\bk}\\right\\| _{2}^{2}-r_{\\bk}^{2}\\right)-\\left\\| z_{1}\\right\\| _{2}^{2}+r_{1}^{2}\n    \\end{bmatrix}.\\\n    \\]\n    \n    Consider $\\A\\bspec$. Since $\\left\\{ \\bz_{j}\\right\\} _{j=1}^{\\bk}$ are linearly independent,\n    $\\left\\{ \\bz_{j}-\\bz_{1}\\right\\} _{j=2}^{\\bk}$---which are the rows of $M\\bspec$---are also linearly independent. For $\\spec$ sufficiently close to $\\bspec$, the rows of $M\\spec$ will remain linearly independent. In this case, the linear system \\eqref{eq:CAS_sys} is then necessarily solvable: $\\A\\spec$ is non-empty. Let $x_0$ be such a solution. Then, $\\A\\spec$ consists of all vectors $x \\in \\R^n$ such that $x{-}x_0$ is orthogonal to the $m{-}1$ linearly independent rows of $M\\spec$. Thus, $\\Dim\\left\\{\\A\\spec\\right\\}{=}n{-}\\bk{+}1$. \n    \n    The vectors $\\left\\{(z_i - z_j)\\right\\}_{j\\neq i}$ clearly all lie in the rowspace of $M\\spec$. Hence, the vectors$\\left\\{(z_i - z_j)\\right\\}_{j\\neq i}$ are orthogonal to all direction in $\\A\\spec$.\n    \n    Finally, write $M$ instead of $M\\spec$ for simplicity. Since $M$ has independent rows, the projection of $w$ onto $\\A\\spec$ is explicitly given by\n    \\begin{equation*}\n    \\projA{\\spec}\\left(w\\right)=\\left(I-M\\left(M^{T}M\\right)^{-1}M^{T}\\right)w+M\\left(M^{T}M\\right)^{-1}b,\n    \\end{equation*}\n    which varies smoothly with $\\spec$ and $w$.\n    \\end{proof}\n\n\\begin{lem}\n    \\textbf{\\label{lem:core} (Local Properties of Core)} For any ball-specifications $\\spec$ sufficiently close to base ball-specifications $\\bspec$, the core in \\eqref{eq:core} is well-defined and varies smoothly with $\\spec$.\n    \\end{lem}\n\n\\begin{proof} By the second fact in Lemma \\ref{lem:A}, for any pair $i\\neq j$, the difference $(z_i{-}z_j)$ is orthogonal to the linear subspace parallel to $\\A\\param$. Therefore, $\\projA{\\param}(z_i)$ and $\\projA{\\param}(z_j)$ are the same. Hence, \\eqref{eq:core} is well-defined. The smoothness of the $c\\spec$ with respect to $\\spec$ then follows from \\eqref{eq:core} and the third fact of Lemma \\ref{lem:A}.\n    \\end{proof}\n\n\\begin{lem}\n    \\textbf{(The Base Case)} \\label{lem:linA}\n    At base ball-specifications, the central affine subspace is a linear subspace and the core is non-zero.\n    \\end{lem}\n\n\\begin{proof}\n    By Definition \\ref{def:base}, $\\br_{j}^{2}=\\left\\| \\bz_{j}\\right\\| _{2}^{2}$\n    for all $j$. Thus, \\eqref{eq:CAS} simplifies to\n    \\begin{equation*}\n    \\A\\bspec=\\left\\{ x\\in\\R^{n}:\\ 2\\bz_{j}^{T}x\\text{ equal  for all } j=1,\\dots,\\bk\\right\\}.\n    \\end{equation*}\n    Clearly, the origin satisfies the RHS conditions above, so $0\\in\\A\\bspec$.\n    \n    \\sloppy Next, assume for contradiction that $0{=}c\\bspec$. By definition, $c\\bspec$ is the projection of $\\bz_j$ onto $\\A\\bspec$ for any $j$, so the vectors $\\left\\{ \\bz_{j}-c\\bspec\\right\\}_{j=1}^{\\bk}{=}\\left\\{ \\bz_{j}\\right\\} _{j=1}^{\\bk}$ must be orthogonal to $\\A\\bspec$. The linear independence of $\\left\\{ \\bz_{j}\\right\\}_{j=1}^{\\bk}$ implies $\\Dim\\left\\{ \\A\\bspec\\right\\}{\\leq}n{-}\\bk$, which contradicts the first fact of Lemma \\ref{lem:A}. Thus, $0{\\neq}c\\bspec$.\n    \\end{proof}\n\n\\begin{lem}\n    \\textbf{(Local Properties of the Margin)\\label{lem:margin}} \n    For ball-specifications $\\spec$ near some base ball-specifications $\\bspec$, the margin in \\eqref{eq:margin} is well-defined, strictly-positive, and varies smoothly with $\\spec$.\n    \\end{lem}\n\n\\begin{proof}\n    It suffices to show that the square of the margin,\\begin{equation}\n    r_{j}^{2}-\\left\\| z_{j}-c\\spec\\right\\| _{2}^{2}, \\label{eq:margin_sq}\n    \\end{equation}\n    does not depend on the index $j$, is strictly-positive, and varies smoothly with $\\param$. First, consider the core and central affine subspace exactly at base ball-specifications $\\bspec$. By Lemma\n    \\ref{lem:linA}, since $0 \\in \\A\\bspec$ and $\\projA{\\bspec}\\left(\\bz_{j}\\right){=}c\\bspec{\\neq}0$\n    for all $j$, the definition of a projection gives\n    \\begin{equation*}\n    \\left\\| \\bz_{j}-c\\bspec\\right\\| _{2}<\\left\\| \\bz_{j}-0\\right\\| _{2}=\\br_{j}\\ \\forall\\ j=1,\\dots,\\bk.\n    \\end{equation*}\n    Thus, \\eqref{eq:margin_sq} is strictly positive, which implies the argument of the square-root in \\eqref{eq:margin} is strictly positive. Next, observe that\n    \\begin{align*}\n    r_{j}^{2}-\\left\\| z_{j}-c\\spec\\right\\| _{2}^{2} & =-\\left\\| c\\spec\\right\\| _{2}^{2}+\\underbrace{2z_{j}^{T}c\\spec-\\left\\| z_{j}\\right\\| _{2}^{2}+r_{j}^{2}}_{(\\star)_j} \\ \\forall\\  j=1,\\dots,\\bk.\n    \\end{align*}\n    Since $c\\spec{\\in}\\A\\spec$, the term $(\\star)_j$ is equal for all $j$. Therefore, \\eqref{eq:margin_sq} is $j$-independent, and $\\rho\\spec$ in \\eqref{eq:margin} is well-defined. The smoothness of $\\rho\\spec$  with respect to $\\spec$ then follows directly from \\eqref{eq:margin} and the smoothness of the core (Lemma \\ref{lem:core}).\n    \\end{proof}\n\n\\begin{lem}\n    \\label{lem:B}\\textbf{(Local Properties of the Feasible Region)} For ball-specifications\n    $\\spec$ near some base ball-specifications $\\bspec$, the feasible region is nonempty and satisfies\\begin{equation}\n    \\B\\spec=\\left\\{ x\\in\\A\\spec:\\ \\left\\| x-c\\spec\\right\\| _{2}=\\rho\\spec\\right\\}.\\label{eq:Bdef2} \n    \\end{equation}\n    \\end{lem}\n\n\\begin{proof}\n    \\noindent\\textit{Equivalence of \\eqref{eq:MBP_cons} and \\eqref{eq:Bdef2}:}\n    Expanding and rearranging \\eqref{eq:MBP_cons} gives\n    \\begin{align*}\n    \\left\\| x-z_{j}\\right\\| _{2}=r_{j}\\ \\forall\\ j & \\iff\\left\\| x\\right\\| _{2}^{2}=2z_{j}^{T}x-\\left\\| z_{j}\\right\\| _{2}^{2}+r_{j}^{2}\\ \\forall\\ j\\\\\n    & \\implies2z_{j}^{T}x-\\left\\| z_{j}\\right\\| _{2}^{2}+r_{j}^{2}=2z_{i}^{T}x-\\left\\| z_{i}\\right\\| _{2}^{2}+r_{i}^{2}\\ \\forall\\ i\\neq j.\n    \\end{align*}\n    Hence, $\\B\\spec\\subseteq\\A\\spec$ implying\n    \\begin{equation*}\n    \\B\\spec=\\left\\{ x\\in\\A\\spec:\\ \\left\\| x-z_{j}\\right\\| _{2}=r_{j}\\text{  for all } j\\right\\}.\n    \\end{equation*}\n    Since $c\\left(r,Z\\right){=}\\projA{\\spec}\\left(z_{i}\\right)$, for any $x{\\in}\\A\\spec$, the vectors $\\left(x{-}c\\left(r,Z\\right)\\right)$ and $\\left(z_{j}{-}c\\spec\\right)$ are orthogonal. Then, applying the Pythagorean Theorem to the equality condition in \\eqref{eq:MBP_cons} gives\n    \\begin{equation*}\n    r_{j}^{2} = \\left\\| x-z_i\\right\\|_{2}^{2} =\\left\\| x-c\\left(r,Z\\right)\\right\\| _{2}^{2}+\\left\\| z_{j}-c\\spec\\right\\|_{2}^{2} \\ \\forall \\ x\\in\\A\\spec \\text{ and } j=1,\\dots,k.\n    \\end{equation*}\n    We then deduce \\eqref{eq:Bdef2} by subtracting $\\left\\| z_{j}{-}c\\spec\\right\\|_{2}^{2}$ from the left and right sides, and applying the definition of $\\rho\\spec$ in \\eqref{eq:margin}.\n    \n    \\noindent \\textit{Non-Emptiness of $\\B\\spec$:} By the first fact of Lemma \\ref{lem:A}, the affine subspace $\\A\\spec$ is $(n{-}m{+}1)$-dimensional, so $\\A\\spec$ is a non-empty space. Thus, there exists $a\\in\\A\\spec$ such that $a\\neq c\\spec$. Choose any such $a$ and consider the direction \n    \\begin{equation*}\n    d = \\frac{1}{\\left\\| a-c\\spec\\right\\|} \\left(a-c\\spec\\right)\n    \\end{equation*}\n    along with the vector\n    \\begin{equation*}\n    v = c\\spec+\\rho\\spec d.\n    \\end{equation*}\n    Then, by construction,\n    \\begin{equation*}\n    \\left\\| v-c\\spec\\right\\| _{2}=\\rho\\spec.\n    \\end{equation*}\n    Since $\\A\\spec$ contains both $c\\spec$ and $d$, we deduce $v\\in\\A\\spec$. Thus, $v$ satisfies all conditions in \\eqref{eq:Bdef2}, so $v\\in\\B\\spec$, and hence $\\B\\spec$ is non-empty.\n    \\end{proof}\n\n\\begin{thm}\\label{thm:MBP_sol}\n    \\textbf{(Local, Explicit Solution of $\\boldsymbol{MBP\\param}$)} For parameters $\\param$ near some base parameters $\\bparam$, the optimal solution to $\\MBP\\param$ is unique, varies smoothly with $\\param$, and is given by\\begin{equation}\\label{eq:x}\n    x\\param = c\\spec+\\rho\\spec u\\param,\n    \\end{equation}\n    where\\begin{equation}\\label{eq:u}\n    u\\param = \\frac{1}{\\left\\| \\projA{\\spec}\\left(y\\right)-c\\spec\\right\\| _{2}}\\left(\\projA{\\spec}\\left(y\\right)-c\\spec\\right).\n    \\end{equation}\n    In particular, $x\\bparam = 0$.\n    \\end{thm}\n\n\\begin{proof}\n    \\textit{The vector $u\\param$ is well-defined:} To show the well-definedness of the expression in \\eqref{eq:u}, observe the following at base parameters $\\bparam$:\n    \n    \\begin{align*}\n    \\projA{\\bspec}\\left(\\by\\right)-c\\bspec &= \\projA{\\bspec}\\left(-\\sum_{j=1}^{\\bk}\\clam_{j}\\bar{z}_{j}\\right)-c\\bspec &\\text{ (By Definition \\ref{def:base})}\\\\\n    & = -\\sum_{j=1}^{\\bk}\\clam_{j}\\projA{\\bspec}\\left(\\bar{z}_{j}\\right)-c\\bspec &\\text{ (By linearity of $\\A\\bspec$ from Lemma \\ref{lem:linA})}\\\\\n    & = -\\left(\\sum_{j=1}^{\\bk}\\clam_{j}+1\\right)c\\bspec &\\text{ (By definition of core \\eqref{eq:core})}\\\\\n    & \\neq 0. &\\text{ (By Definition \\ref{def:base} and Lemma \\ref{lem:linA})}.\n    \\end{align*}\n    By the smoothness of the projection (third fact of Lemma \\ref{lem:A}), we deduce $\\projA{\\spec}\\left(y\\right){-}c\\spec{\\neq}0$ for $\\param$ sufficiently close to $\\bparam$ as well. Thus, \\eqref{eq:u} is well-defined. Moreover, the smoothness of the projection and \\eqref{eq:u} further shows that $u\\param$ is smooth with respect to $\\param$.\n    \n    \\noindent \\textit{Derivation of $x\\param$:} By Lemma \\ref{lem:B}, $\\B\\spec\\subseteq\\A\\spec$ so\\begin{align*}\n    x\\param=\\projB{\\spec}\\left(y\\right) & =\\projB{\\spec}\\left(\\projA{\\spec}\\left(y\\right)\\right).\n    \\end{align*}\n    Lemma \\ref{lem:B} also states that $\\B\\spec$ is simply the $c\\spec$-centered ball with radius $\\rho\\spec$ restricted to $\\A\\spec$. Thus, $\\projB{\\spec}\\left(\\projA{\\spec}\\left(y\\right)\\right)$ must be the (necessarily unique) point that\n    \\begin{enumerate}\n        \\item lies on the straight line from $c\\spec$ to $\\projA{\\spec}\\left(y\\right)$, and\n        \\item is exactly the distance $\\rho\\spec$ from $c\\spec$.\n    \\end{enumerate}\n    Formalizing the above two criteria leads to the explicit-form\n    \\begin{equation*}\n    x\\param=c\\spec+\\frac{\\rho\\spec}{\\left\\| \\projA{\\spec}\\left(y\\right)-c\\spec\\right\\| _{2}}\\left(\\projA{\\spec}\\left(y\\right)-c\\spec\\right)=c\\spec+\\rho\\spec u\\param.\n    \\end{equation*}\n    The smoothness of $x\\param$ follows directly from the smoothness of the core (Lemma \\ref{lem:core}), the margin (Lemma \\ref{lem:margin}), and $u\\param$ (shown above).\n    \n    \\noindent \\textit{Demonstration of $x\\bparam{=}0$:} Observe that the Lagrangian of $\\MBP\\bparam$ is\n    \\[\n    \\L(x,\\lambda) = \\left\\| x-\\by\\right\\| _{2}^{2} + \\sum_{j=1}^\\bk \\lambda_j \\left\\| x-\\bz_{j}\\right\\| _{2}^{2},\n    \\]\n    where $\\lambda_j$ is the $j$-th element of a Lagrange multipliers vector $\\lambda \\in \\R^\\bk$. By Definitions \\ref{def:MBP}-\\ref{def:base}, the zero-vector is a feasible solution to $\\MBP\\bparam$. A feasible vector $x^*$ is the solution of $\\MBP\\bparam$ if and only if there exists $\\lambda^*$ such that $x^*$ minimizes $\\L(\\cdot,\\lambda^*)$. It is easy to check using Definition \\ref{def:base} and first-order optimality conditions that $x^*{=}0$ is the unique minimizer of $\\L(\\cdot,\\clam)$. Thus, $x\\bparam{=}0$ is the unique solution to $\\MBP\\bparam$.\n    \\end{proof}\n\n\\begin{lem}\\label{lem:MBP_lam}\\textbf{(Properties of MBP Lagrange Multipliers)} Suppose the parameters $\\param$ are sufficiently close to base parameters $\\bparam$ with corresponding base weights vector $\\clam$. Consider the optimal solution $x\\param$ of $\\MBP\\param$. Then, the following hold:\n    \\begin{enumerate}\n        \\item There exists a unique Lagrange multiplier vector, $\\lambda\\param\\in\\R_{++}^{\\bk}$, satisfying the stationarity condition \\eqref{eq:MBP_FOC}.\n        \\item The vector $\\lambda\\param$ varies smoothly with $\\param$.\n        \\item At the base parameters, $\\lambda\\bparam=\\clam$.\n    \\end{enumerate}\n    \\end{lem}\n\n\\begin{proof}\n    Since $x\\param$ is optimal for $\\MBP\\param$, there must exist a Lagrange multipliers vector $\\lambda\\param\\in\\R^{\\bk}$ satisfying \\eqref{eq:MBP_FOC}. We first show $\\lambda\\param$ is unique. Finding Lagrange multipliers satisfying \\eqref{eq:MBP_FOC} is equivalent to finding minimizers of the function\n    \\begin{equation*}\n    F\\left(\\lambda\\right)=\\left\\| x\\param-y+X\\lambda\\right\\| _{2}^{2},\n    \\end{equation*}\n    where we define the matrix\n    \\[X=\\begin{bmatrix}x\\param{-}z_{1}, \\dots, x\\param{-}z_{\\bk}\\end{bmatrix}\\in\\R^{n\\times\\bk}.\\]\n    Then, a vector $\\lambda$ minimizes $F$ if and only if it satisfies the below $\\bk\\times\\bk$ linear system:\n    \\begin{equation*}\n    X^{T}\\left(x\\param-y+X\\lambda\\right)=0\\iff X^{T}X\\lambda=X^{T}\\left(y-x\\param\\right).\n    \\end{equation*}\n    By Theorem \\ref{thm:MBP_sol}, for $\\param$ close to $\\bparam$, the MBP-solution $x\\param$ is close to zero. Therefore, columns of $X$ are close to $\\left\\{-\\bz_{j}\\right\\}_{j=1}^{\\bk}$, which are linearly independent (Definition \\ref{def:base}). This implies $X^{T}X$ is invertible, and the minimizer of $F$ is unique and explicitly given by \n    \\begin{equation*}\n    \\lambda\\param = \\argmin_\\lambda F\\left(\\lambda\\right)=\\left(X^{T}X\\right)^{-1}X^{T}\\left(y-x\\param\\right).\n    \\end{equation*}\n    The smoothness of $\\lambda\\param$ with respect to $\\param$ follows from the smoothness of $x\\param$ (Theorem \\ref{thm:MBP_sol}) and the smoothness of the above matrix operations. \n    \n    Finally, it follows directly from Definition \\ref{def:base} and Theorem \\ref{thm:MBP_sol} that\n    \\begin{equation*}\n    x\\bparam-\\by+\\sum_{j=1}^{\\bk}\\clam_{j}\\left[x\\bparam-\\bz_{j}\\right]=0,\n    \\end{equation*}\n    which implies the Lagrange multiplier vector at the base parameters is uniquely $\\lambda\\bparam=\\clam$.\n    \\end{proof}\n\n\\begin{thm}\\textbf{(Inequality-Constrained MBP)}\\label{thm:MBP_ineq}\n    Consider the following inequality-constrained MBP:\n    \\begin{align*}\n    %\\begin{aligned}\n    \\min_{x\\in\\R^{n}}\\ & \\left\\| x-y\\right\\| _{2}^{2}\\\\\n    \\mathrm{s.t.}\\ & \\left\\| x-z_{j}\\right\\| _{2}^{2} \\leq r_{j}^{2}\\ \\forall\\ j=1,\\dots,\\bk,\n    %\\end{aligned}%\\label{eq:MBP_ineqFull}\n    \\end{align*}\n    and denote it as $\\MBPl\\param$. For parameters $\\param$ sufficiently close to $\\bparam$, $\\MBPl\\param$ is feasible and possesses the same unique optimal solution, $x\\param$, and unique Lagrange multipliers, $\\lambda\\param$, as $\\MBP\\param$. Consequently, all constraints of $\\MBPl\\param$ are tight. Moreover, the optimal solution and multipliers satisfy\\begin{equation}\n    x\\param = \\mathrm{O}\\left(\\|\\param-\\bparam\\|\\right)\\text{ and } \\lambda\\param = \\clam + \\mathrm{O}\\left(\\|\\param-\\bparam\\|\\right).\\label{eq:MBP_Onotation}\n    \\end{equation}\n    \\end{thm}\n\n\\begin{proof}\n    The feasibility of $\\MBPl\\param$ follows directly from Lemma \\ref{lem:B} since $\\B\\param$ is a subset of the feasible region of $\\MBPl\\param$.\n    \n    $\\MBPl\\param$ possesses a unique minimizer since it minimizes a strongly convex objective on a convex feasible region. Moreover, checking the KKT conditions is sufficient for demonstrating optimality because the $\\|x - z_j\\|_2^2$-term in the constraints is convex and differentiable for all $j$. Therefore---denoting the unique pair of optimal solutions and Lagrange multipliers of $\\MBP\\param$ as $\\left(x\\param, \\lambda\\param\\right)$---it suffices to check that $\\left(x\\param, \\lambda\\param\\right)$ satisfies the KKT conditions of $\\MBPl\\param$:\n    \\begin{enumerate}\n        \\item \\textbf{(Stationarity)} Follows from $\\MBPl\\param$ and $\\MBP\\param$ having the same stationarity condition \\eqref{eq:MBP_FOC}.\n        \\item \\textbf{(Primal Feasibility)} Follows from $\\B\\param$ being a subset of the feasible region of $\\MBPl\\param$.\n        \\item \\textbf{(Dual Feasibility)} The inequality constraints of $\\MBPl\\param$ require the non-negativity of all Lagrange multipliers. By Definition \\ref{def:base}, the elements of $\\clam$ are strictly-positive. Thus, by Lemma \\ref{lem:MBP_lam}, the elements of $\\lambda\\param$ are also strictly-positive for $\\param$ sufficiently close to $\\bparam$.\n        \\item \\textbf{(Complementary Slackness)} \\sloppy By construction, $x\\param$ satisfies all the inequality constraints of $\\MBPl\\param$ with equality. Correspondingly, $\\lambda\\param$ are all strictly positive.\n    \\end{enumerate}\n    Therefore, $\\left(x\\param, \\lambda\\param\\right)$ is the unique pair of optimal solutions and Lagrange multipliers for $\\MBPl\\param$ as well. The characterizations in \\eqref{eq:MBP_Onotation} directly follows from Theorem \\ref{thm:MBP_sol} and Lemma \\ref{lem:MBP_lam} as well as recalling that smooth functions are locally Lipschitz.\n    \\end{proof}\n\n\\begin{proof}\n    It is easy to check from the mapping definitions \\eqref{eq:MBP_mappings} and the Taylor expansion \\eqref{eq:taylor} that \n    \n    $$\\left\\|\\left(y_{i}\\left(\\S\\right), \\left[ z_{ij}\\left(\\S\\right)\\right]_{j\\neq i}, \\left[r_{ij}\\left(\\S\\right)\\right]_{j\\neq i},\\left[ s_{j}\\right]_{j\\neq i}\\right) - \\left(y_{i}\\left(\\bX\\right), \\left[ z_{ij}\\left(\\bX\\right)\\right]_{j\\neq i}, \\left[r_{ij}\\left(\\bX\\right)\\right]_{j\\neq i}, [\\bx]_{j\\neq i}\\right)\\right\\| = \\mathrm{O}\\left(\\|\\S-\\bX\\|\\right).$$\n    The result then follows directly from Observation \\ref{obs:SD_MOS}, Observation \\ref{obs:SD_as_MBP}, and Theorem \\ref{thm:MBP_ineq}.\n    \\end{proof}","units":[{"start":44227,"end":45079,"start_line":321,"end_line":324,"raw_tex":"\\begin{restatable}{thm}{PBmbp}\\label{thm:SD_MBP} \\textbf{(Local Feasibility, Uniqueness, Tightness, and Smoothness of Survey Descent)} Assume Setting \\ref{setting:localSD}. Then for all $i$, the Survey Descent subproblem $\\left(P_{i}^{\\S}\\right)$ is feasible and has a unique solution $s_{i}^{+}$, which satisfies all the constraints with equality, with unique associated Lagrange multipliers $\\lambda_{j}^i$ \\edit{for the $j$-th constraint within the $i$-th subproblem}  (for $j\\neq i$). The solutions and multipliers depend smoothly on the input survey $\\S$, and satisfy\\begin{equation}\ns_{i}^{+}\\left(\\S\\right) ~=~\\bx+\\mathrm{O}\\left(\\left\\| \\S-\\bX\\right\\| \\right) \\qquad \\text{and} \\qquad \\lambda_{j}^i\\left(\\S\\right) ~=~ \\blam_{j}+\\mathrm{O}\\left(\\left\\| \\S-\\bX\\right\\| \\right)\\ \\forall\\ j\\neq i.\\label{eq:SD_forms}\n\\end{equation}\n\\end{restatable}","sha256_utf8":"dbded2cd688b822b30d1130c855753c6f635a45aee3a2264368f3089f21368f6"},{"start":45080,"end":45236,"start_line":325,"end_line":327,"raw_tex":"\\begin{proof}\nFollows from a routine analysis of the subproblem's first-order optimality conditions combined with the implicit function theorem.\n\\end{proof}","sha256_utf8":"64afb1672f0e40317fdb92623612b091e65b53f6c4fb0c96c714eb7ae69ece28"},{"start":46805,"end":46991,"start_line":344,"end_line":345,"raw_tex":"\\begin{thm} \\label{thm:strictval} \\textbf{(Preservation of Validity)} Assuming Setting \\ref{setting:localSD}, the output survey $\\S^{+}$ of a Survey Descent iteration is valid.\n\\end{thm}","sha256_utf8":"2289d998374779a1b937a646773208e770475785cb1976cb039ce897ee3fdfee"},{"start":46992,"end":48521,"start_line":346,"end_line":367,"raw_tex":"\\begin{proof}\nConsider any fixed $i$. If $s_{i}^{+}=s_{i}$, then the fact\nthat $f_{i}\\left(s_{i}^{+}\\right){>}f_{j}\\left(s_{i}^{+}\\right)$ for all $j\\neq i$ immediately follows since Theorem \\ref{thm:SD_MBP} assumes $\\S$ is valid.\n\nNow, consider the $s_{i}^{+}{\\neq}s_{i}$ case. By Theorem \\ref{thm:SD_MBP}, $s_{i}^{+}$ satisfies the constraints of $\\left(P_{i}^{\\S}\\right)$ with equality. Using Observation \\ref{obs:SD_MOS}, we express this as\\begin{equation}\n\\ell_{s_i}^{f_i}(s_i^+) = \\ell_{s_j}^{f_j}(s_i^+)+\\frac{L}{2}\\left\\| s_{i}^{+}-s_{j}\\right\\|_{2}^{2}\\ \\forall\\ j\\neq i.\\label{eq:left_valid}\n\\end{equation}\nThe $\\delta$-strong convexity of $f_{i}$ implies\n\\begin{equation*}\nf_{i}\\left(s_{i}^{+}\\right)\\geq\\ell_{s_i}^{f_i}(s_i^+)+\\frac{\\delta}{2}\\left\\| s_{i}^{+}-s_{i}\\right\\| _{2}^{2}>\\ell_{s_i}^{f_i}(s_i^+),\n\\end{equation*}\nwhere the strict inequality follows from $s_{i}^{+}{\\neq}s_{i}$. Next, by the $L$-smoothness of $\\left\\{f_j\\right\\}_{j=1}^k$, the right-hand side of \\eqref{eq:left_valid} is a $s_j$-centered quadratic upper bound of $f_{j}$ evaluated at $s_i^+$. Thus,\n\\begin{equation*}\n\\ell_{s_j}^{f_j}(s_i^+)+\\frac{L}{2}\\left\\| s_{i}^{+}-s_{j}\\right\\|_{2}^{2} \\geq f_{j}\\left(s_{i}^{+}\\right) \\ \\forall\\ j\\neq i.\n\\end{equation*}\n\n\\noindent Therefore, \n\\begin{equation*}\nf_{i}\\left(s_{i}^{+}\\right) > \\ell_{s_i}^{f_i}(s_i^+) = \\ell_{s_j}^{f_j}(s_i^+)+\\frac{L}{2}\\left\\| s_{i}^{+}-s_{j}\\right\\|_{2}^{2} \\geq f_{j}\\left(s_{i}^{+}\\right) \\ \\forall\\ j\\neq i.\n\\end{equation*}\nThis completes the proof.\n\\end{proof}","sha256_utf8":"d81d688c6d3dd2df89c6ea910f6e3f6f452c2e947224b46f6274ffeea215d589"},{"start":50502,"end":50732,"start_line":386,"end_line":388,"raw_tex":"\\begin{lem}\n\\label{lem:r_grad} For all $i$, the residual function $r_i(x)$ in \\eqref{eq:taylor} satisfies $r_i(x)=\\mathrm{O}\\left(\\|x\\|^2\\right)$ and $\\nabla r_{i}\\left(x\\right)=\\mathrm{O}\\left(\\left\\| x\\right\\|\\right)$.\n\\end{lem}","sha256_utf8":"21a0a351120583563935dba51a1865ed20d0d87cc8857d595d0fa0d7602f2574"},{"start":50733,"end":51464,"start_line":389,"end_line":396,"raw_tex":"\\begin{proof}\n$r_i(x)=\\mathrm{O}\\left(\\|x\\|^2\\right)$ follows directly from Taylor's Theorem. Next, observe that \n\\begin{equation*}\n    r_{i}\\left(x\\right)=f_{i}\\left(x\\right)-\\left(f(0) + a_{i}^{T}x+\\frac{1}{2}x^{T}A_{i}x\\right).\n\\end{equation*}\nThus, since $f_i$ is $\\C^2$, the residual $r_i$ must also be $\\C^2$, so its gradient and Hessian are well-defined. This implies $\\nabla r_{i}\\left(x\\right)$ is then $\\C^{1}$. Taylor expanding $\\nabla r_i$ around zero then gives the desired $\\nabla r_{i}\\left(x\\right)= \\mathrm{O}\\left(\\| x\\|\\right)$\nafter observing that $\\nabla r_i(0)$ is the zero-vector and $\\nabla^2 r_i(0)$ is the zero-matrix (since $r_i$ is the residual of the Taylor expansion of $f_i$ around zero).\n\\end{proof}","sha256_utf8":"f448e3387690ea899fd735775acb1047e7599e368b3d2d1772761c6a87879e2c"},{"start":51526,"end":52064,"start_line":400,"end_line":405,"raw_tex":"\\begin{lem}\\label{lem:U_component} Assume Setting \\ref{setting:localSD} and suppose $\\bx{=}0$. Then, $s_{i}^{+}$ satisfies\n\\begin{equation*}\n\\proj\\left[s_{i}^{+}\\right]=\\frac{1}{2-\\blam_{i}}\\proj\\left[s_{i}-\\frac{\\blam_{i}}{L}\\nabla f_{i}\\left(s_{i}\\right)\\right]+\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\proj\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right]+\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right) \\ \\forall \\ i=1,\\dots,k.\n\\end{equation*}\nwhere $\\proj$ is the projection operator onto the $\\U$-subspace \\eqref{eq:Udef1}.\n\\end{lem}","sha256_utf8":"c169c715a3c5400fbb9effeb89a0d7c0ccecb5e23a3214b3899e90a4e108a423"},{"start":52065,"end":55890,"start_line":406,"end_line":449,"raw_tex":"\\begin{proof}\nWithout loss of generality, assume $i=k$. Scale the objective of problem \\eqref{eq:raw_form_analytic} by $\\frac{L}{2}$ and consider the equivalent problem\n\\begin{equation}\\begin{aligned}\\min_{x}\\ & \\frac{L}{2} \\left\\| x-\\left(s_{k}-\\frac{1}{L}\\nabla f_{k}\\left(s_{k}\\right)\\right)\\right\\| _{2}^{2}\\\\\n\\mathrm{s.t.}\\ & \\ell_{s_j}^{f_j}(x)+\\frac{L}{2}\\left\\| x-s_{j}\\right\\| _{2}^{2}\\leq \\ell_{s_k}^{f_k}(x)\\ \\forall\\ j\\neq k.\n\\end{aligned}\\label{eq:scaledPk}\n\\end{equation}\nBy Theorem \\ref{thm:SD_MBP} and Observation \\ref{obs:SD_MOS}, there exists unique Lagrange multipliers $\\left\\{\\lambda_j\\right\\}_{j=1}^{k-1}$ (we omit the $k$-superscript for simplicity) satisfying the following first-order optimality condition for \\eqref{eq:scaledPk}:\n$$\n    0=L\\left(s_{k}^{+}-\\left(s_{k}-\\frac{1}{L} \\nabla f_{k}\\left(s_{k}\\right)\\right)\\right)+\\sum_{j=1}^{k-1} \\lambda_{j}\\left(\\edit{\\nabla f_{j}\\left(s_{j}\\right)}+L\\left(s_{k}^{+}-s_{j}\\right)-\\nabla f_{k}\\left(s_{k}\\right)\\right)\n    $$\nAfter rearranging this gives\\begin{equation}\n0= L\\left(1+\\sum_{j=1}^{k-1}\\lambda_{j}\\right)s_{k}^{+}-L\\left(s_{k} + \\sum_{j=1}^{k-1} \\lambda_{j} s_{j}\\right)+\\nabla f_k(s_k) +\\sum_{j=1}^{k-1}\\lambda_{j}\\left(\\nabla f_j(s_j) - \\nabla f_k(s_k)\\right).\n\\label{eq:FOC2}\n\\end{equation}\nDefine the following scalars:\n\\begin{align}\n    \\lambda_{k} & \\equiv  1-\\sum_{j=1}^{k-1}\\lambda_{j},\\label{eq:lambda}\\\\\n    \\mu_{j} & \\equiv  \\blam_{j}-\\lambda_{j} \\ \\forall \\ j=1,\\dots,k. \\label{eq:mu}\n\\end{align}\nFor $j\\neq k$, Theorem \\ref{thm:SD_MBP} gives\n$\\mu_{j}=\\mathrm{O}\\left(\\left\\| \\S\\right\\| \\right)$ .\nFor $j=k$, combining \\eqref{eq:lambda}-\\eqref{eq:mu} with equation \\eqref{eq:lam_sum1} implies\n\\begin{equation*}\n\\mu_{k} =\\blam_{k}-\\left(1-\\sum_{j=1}^{k-1}\\lambda_{j}\\right) =\\blam_{k}-\\left(1-\\sum_{j=1}^{k-1}\\blam_{j}\\right)+\\mathrm{O}\\left(\\left\\| \\S\\right\\| \\right) =\\mathrm{O}\\left(\\left\\| \\S\\right\\| \\right).\n\\end{equation*}\nThus,\\begin{equation}\n    \\mu_j = \\mathrm{O}(\\|\\S\\|) \\ \\forall \\ j=1,\\dots,k.\\label{eq:mu_size}\n\\end{equation}\nSubstituting \\eqref{eq:mu} into \\eqref{eq:FOC2}, applying identities \\eqref{eq:lam_sum1}-\\eqref{eq:lam_sum0}, collecting $\\mathrm{O}\\left(\\|\\S\\|^2\\right)$ terms, and solving for $s_k^{+}$ gives\\begin{equation}s_{k}^{+}= \\frac{1}{2-\\blam_{k}}\\left[s_{k}-\\frac{\\blam_{k}}{L}\\nabla f_{k}\\left(s_{k}\\right)\\right]+\\sum_{j=1}^{k-1}\\frac{\\blam_{j}}{2-\\blam_{k}}\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right]+\\frac{1}{L\\left(2-\\blam_{k}\\right)}\\sum_{j=1}^{k}\\mu_{j}\\nabla f_j(s_j) +\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right).\\label{eq:pre_sform}\n\\end{equation}\nAdopting the notation \\eqref{eq:A_notation} and applying the Taylor expansion \\eqref{eq:taylor}, observe\\begin{equation}\n    \\sum_{j=1}^{k}\\mu_{j}\\nabla f_j(s_j) = \\sum_{j=1}^{k}\\mu_{j}\\left(a_j + A_j s_j + \\nabla r_j (s_j)\\right) = \\sum_{j=1}^{k}\\mu_{j}a_j +\\mathrm{O}\\left(\\left\\| \\S\\right\\|^{2}\\right),\\label{eq:sum_muA}\n\\end{equation}\nwhere last equality follows from \\eqref{eq:mu_size} and Lemma \\ref{lem:r_grad}. Substituting \\eqref{eq:sum_muA} into \\eqref{eq:pre_sform} then leads to\\begin{equation}\ns_{k}^{+}=\\frac{1}{2-\\blam_{k}}\\left[s_{k}-\\frac{\\blam_{k}}{L}\\nabla f_{k}\\left(s_{k}\\right)\\right]+\\sum_{j=1}^{k-1}\\frac{\\blam_{j}}{2-\\blam_{k}}\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right]+\\frac{1}{L\\left(2-\\blam_{k}\\right)}\\underbrace{\\sum_{j=1}^{k}\\mu_{j}a_j}_{(\\star)} +\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right).\n\\label{eq:s_form}\n\\end{equation}\nNext, again combine \\eqref{eq:lambda}-\\eqref{eq:mu} with equation \\eqref{eq:lam_sum1} to deduce\n\\begin{equation*}\n\\sum_{j=1}^{k}\\mu_{j}=\\sum_{j=1}^{k}\\left(\\blam_{j}-\\lambda_{j}\\right)=0.\n\\end{equation*}\nThen, it follows from \\eqref{eq:Udef2} that $\\proj\\left[(\\star)\\right]=0$.\nThe desired result then follows from applying $\\proj$ to \\eqref{eq:s_form}.\n\\end{proof}","sha256_utf8":"b10b489203e777b061c41197d95326165e32a31670b55f2b3770e68af9a76e56"},{"start":55892,"end":56269,"start_line":451,"end_line":457,"raw_tex":"\\begin{lem}\\label{lem:Up_component}\nAssume Setting \\ref{setting:localSD}, and let $\\bx{=}0$. Then, $s_{i}^{+}$ satisfies\n\\begin{equation*}\n\\projp\\left[s_{i}^{+}\\right]=\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right) \\ \\forall \\ i=1,\\dots,k.\n\\end{equation*}\nwhere $\\projp$ is the projection operator onto the orthogonal complement of the $\\U$-subspace \\eqref{eq:Udef1}.\n\\end{lem}","sha256_utf8":"68c3ce37ead3237cb17043552fa526462dd7635488fb648d94bd1a8e97461db1"},{"start":56270,"end":59750,"start_line":458,"end_line":505,"raw_tex":"\\begin{proof}\nConsider any fixed $i$. By Theorem \\ref{thm:SD_MBP}, the constraints of $\\left(P_{i}^{\\S}\\right)$\nare tight. Thus, by Observation \\ref{obs:SD_MOS}, $s_{i}^{+}$ satisfies\n\\begin{equation*}\n    f_{j}\\left(s_{j}\\right)+\\nabla f_{j}\\left(s_{j}\\right)^{T}\\left(s_i^+-s_{j}\\right)+\\frac{L}{2}\\left\\| s_i^+-s_{j}\\right\\| _{2}^{2} = f_{i}\\left(s_{i}\\right)+\\nabla f_{i}\\left(s_{i}\\right)^{T}\\left(s_i^+-s_{i}\\right) \\ \\forall \\ j\\neq i.\n\\end{equation*}\nUsing equation \\eqref{eq:taylor} in the above gives, for all $j\\neq i$,\n\\begin{multline*}\n    f(0) + a_{j}^{T}s_{j}+\\frac{1}{2}s_{j}^{T}A_{j}s_{j}+r_{j}\\left(s_j\\right)+\\left(a_{j}+A_{j}s_{j} + \\nabla r_j(s_j)\\right)^T\\left(s_{i}^{+}-s_{j}\\right)+\\frac{L}{2}\\left\\Vert s_{i}^{+}-s_{j}\\right\\Vert _{2}^{2}\\\\\n    =f(0) + a_{i}^{T}s_{i}+\\frac{1}{2}s_{i}^{T}A_{i}s_{i}+r_{i}\\left(s_i\\right)+\\left(a_{i}+A_{i}s_{i}+\\nabla r_i(s_i)\\right)^T\\left(s_{i}^{+}-s_{i}\\right).\n\\end{multline*}\nAfter simplification and rearrangement, we obtain\\begin{equation}\n\\left(a_{j}-a_{i}\\right)^{T}s_{i}^{+}=-\\left(A_{j}s_{j}-A_{i}s_{i}+\\nabla r_{j}\\left(s_{j}\\right)-\\nabla r_{i}\\left(s_{i}\\right)\\right)^{T}s_{i}^{+}-\\frac{L}{2}\\left\\| s_{i}^{+}-s_{j}\\right\\| _{2}^{2}-c_{j}\\left(s_{j}\\right)+c_{i}\\left(s_{i}\\right)  \\ \\forall \\ j\\neq i, \\label{eq:s_constr}\n\\end{equation}\nwhere we define the functions\n\\begin{equation*}\nc_{j}\\left(x\\right)\\equiv-\\frac{1}{2}x^{T}A_{j}x+r_{j}\\left(x\\right)-\\nabla r_{j}\\left(x\\right)^{T}x \\ \\forall \\ j=1,\\dots,i.\n\\end{equation*}\nBy Lemma \\ref{lem:r_grad}, $c_j(x)$ is $O\\left(\\|x\\|^2\\right)$. Since we assume $\\bx{=}0$, by Theorem \\ref{thm:SD_MBP}, $s_{i}^{+}=\\mathrm{O}\\left(\\left\\| \\S\\right\\| \\right)$.\nWe then deduce that all terms on the right-hand side of \\eqref{eq:s_constr} are $\\mathrm{O}\\left(\\left\\| \\S\\right\\|^2 \\right)$:\n\\begin{equation}\n\\left(a_{j}-a_{i}\\right)^{T}s_{i}^{+}=\\mathrm{O}\\left(\\left\\| \\S\\right\\|^2 \\right) \\ \\forall \\ j\\neq i.\\label{eq:projsp}\n\\end{equation}\n% Consider the following $QR$-decomposition \n% \\[\n% \\begin{bmatrix}\\left(a_{1}-a_{i}\\right), & \\dots, & \\left(a_{i-1}-a_{i}\\right)\\end{bmatrix}\\equiv\\bQ\\bR,\n% \\]\n% where $\\bQ\\in\\R^{n\\times\\left(k-1\\right)}$ is an orthonormal matrix,\n% and $\\bR\\in\\R^{\\left(k-1\\right)\\times\\left(k-1\\right)}$ is an upper-triangular\n% matrix. Then,\n% \\begin{equation}\n% \\bR^{T}\\bQ^{T}s_{i}^{+}=\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right).\\label{eq:s_constr2}\n% \\end{equation}\nBy \\eqref{eq:Udef1},\n\\begin{align*}\n\\U^{\\perp} & =\\Span\\left\\{ a_{i}-a_{j}:1\\leq i,j\\leq k\\right\\} =\\Span\\left\\{ a_{j}-a_{i}:\\ j \\neq i \\right\\}.\n\\end{align*}\nCombined with the affine independence of $\\left\\{ a_{j}\\right\\} _{j=1}^{k}$\n(Definition \\ref{def:strong_max}), the above implies $\\left\\{ a_{j}-a_{i}\\right\\} _{j\\neq i}$ forms a basis for $\\U^{\\perp}$. Therefore, we can express the projection onto $\\U^{\\perp}$ in terms $\\left\\{ a_{j}-a_{i}\\right\\} _{j\\neq i}$. The result now follows from equation \\eqref{eq:projsp} and standard linear algebra.\n% Thus, $\\bR$ is invertible,\n% and we can represent the projection onto $\\U^{\\perp}$ as multiplication by the matrix $\\bQ\\bQ^{T}.$ From this, we deduce\n% \\begin{equation*}\n% \\projp\\left[s_{i}^{+}\\right]=\\bQ\\bQ^{T}s_{i}^{+}=\\bQ\\left(\\bR^{T}\\right)^{-1}\\bR^{T}\\bQ^{T}s_{i}^{+}=\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right),\n% \\end{equation*}\n% where the last equality follows from \\eqref{eq:s_constr2} and the\n% fact that $\\bQ\\left(\\bR^{T}\\right)^{-1}$ is a constant-matrix independent\n% of $\\S$.\n\\end{proof}","sha256_utf8":"d49c5783f48db11402f17bade5122204cb92563fbe7cf42f1a667a996c139209"},{"start":59752,"end":60448,"start_line":507,"end_line":515,"raw_tex":"\\begin{thm}\n\\label{thm:approx_s} \\textbf{(Gradient Descent Approximation of Survey Updates)} Assume Setting \\ref{setting:localSD}.  Then the solution of the $i$-th Survey Descent subproblem satisfies, for all $i$, \\begin{equation}\ns_{i}^{+}= \\ts_{i}+\\mathrm{O}\\left(\\left\\| \\S - \\bX\\right\\| ^{2}\\right) \\label{eq:SD_iter}\n\\end{equation}\nwhere we define\\begin{equation}\n\\ts_{i}\\equiv\\frac{1}{2-\\blam_{i}}\\proj\\left[s_{i}-\\frac{\\blam_{i}}{L}\\nabla f_{i}\\left(s_{i}\\right)\\right]+\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\proj\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right], \\label{eq:s_tilde}\n\\end{equation}\nand $\\proj$ is the projection operator onto the $\\U$-subspace \\eqref{eq:Udef1}.\n\\end{thm}","sha256_utf8":"51a6da9e6c20716b669b6830a3929774d96a0df08c386d564f33c9d0b523654c"},{"start":60449,"end":60815,"start_line":516,"end_line":518,"raw_tex":"\\begin{proof}\nWithout loss of generality, assume that $\\bx{=}0$. Then, the desired result follows from decomposing $s_{k}^{+} = \\proj\\left[s_{k}^{+}\\right] + \\projp\\left[s_{k}^{+}\\right]$ and applying Lemmas \\ref{lem:U_component} and \\ref{lem:Up_component}. We can deduce the general $\\bx{\\neq}0$ case with the change of variables $x{\\leftarrow}(x-\\bx)$.\n\\end{proof}","sha256_utf8":"268dda081f1ff1b9d10543ae5a199cc63bbfc85e87452d6f5ec02b14602543d2"},{"start":61486,"end":62070,"start_line":524,"end_line":530,"raw_tex":"\\begin{thm}\n\\textbf{\\label{thm:PR}(Projected-GD Behavior)} Assume that $\\S$ is a valid survey for the strong $\\C^2$ max function objective $f$. Then, the following holds for all $i$ and any constant $\\tL \\geq L$:\n\\begin{align*}\n\\left\\| \\proj\\left[s_i - \\frac{1}{\\tL} \\nabla f_i(s_i)\\right] - \\bx\\right\\| _{2}^{2} &\\leq \\left(1-\\frac{\\delta}{\\tL}\\right)\\left\\| s_{i} - \\bx \\right\\| _{2}^{2},\\\\\nf_{i}\\left(\\proj\\left[s_i - \\frac{1}{\\tL} \\nabla f_i(s_i)\\right]\\right) - f_i(\\bx) &\\leq \\frac{\\tL}{2}\\left(1-\\frac{\\delta}{\\tL}\\right)\\left\\| s_{i} - \\bx\\right\\| ^{2}.\n\\end{align*}\n\\end{thm}","sha256_utf8":"fc92c9604f8b8050d9b7deeb9a6db618bfdf99496a8727f9c0f5292a75199540"},{"start":62071,"end":62532,"start_line":531,"end_line":533,"raw_tex":"\\begin{proof}\nConsider any fixed $i$. It is easy to check that $\\bx = \\arg\\min_{x \\in \\U} f_i(x)$ using first-order optimality conditions combined with \\eqref{eq:lam_sum1}-\\eqref{eq:lam_sum0} and \\eqref{eq:Udef3}. The desired result then follows from classical projected-GD convergence results (given in standard texts such as \\citet[Theorem 10.29]{beck2017}) after observing that $\\U$ is convex and $f_i$ is $L$-smooth and $\\delta$-strongly convex.\n\\end{proof}","sha256_utf8":"e99b259058eb261006437a4bd25b5b7cc0e966ca234199173eb5fd608d3a36dc"},{"start":62839,"end":63277,"start_line":539,"end_line":545,"raw_tex":"\\begin{lem}\n\\label{lem:st_bound} Assume Setting \\ref{setting:localSD}, and suppose $\\bx{=}0$. Then, the following must hold:\n\\begin{equation*}\n\\left\\| \\ts_{i}\\right\\| _{2}^{2}\\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)\\max_{j=1,....,k}\\left\\| s_{j}\\right\\| _{2}^{2} \\ \\forall \\ i=1,\\dots,k,\n\\end{equation*}\nwhere $\\ts_i$ and $\\blam_{\\min}$ are as defined in equations \\eqref{eq:s_tilde} and \\eqref{eq:min_lam}, respectively.\n\\end{lem}","sha256_utf8":"cb77b8feb2e84d9a1314865a4d258db31e2b9d1da9f788f16f7ab0da06c813f7"},{"start":63278,"end":64504,"start_line":546,"end_line":555,"raw_tex":"\\begin{proof} Consider any fixed $i$. Then,\n\\begin{align*}\n\\left\\| \\ts_{i}\\right\\| _{2}^{2} & =\\left\\| \\frac{1}{2-\\blam_{i}}\\proj\\left[s_{i}-\\frac{\\blam_{i}}{L}\\nabla f_{i}\\left(s_{i}\\right)\\right]+\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\proj\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right] \\right\\| _{2}^{2}  \\text{ (Substitute \\eqref{eq:s_tilde})}\\\\\n & \\leq\\frac{1}{2-\\blam_{i}}\\left\\| \\proj\\left[s_{i}-\\frac{\\blam_{i}}{L}\\nabla f_{i}\\left(s_{i}\\right)\\right]\\right\\| _{2}^{2}+\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\left\\| \\proj\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right]\\right\\| _{2}^{2}\\text{ (Convexity)}\\\\\n & \\leq\\frac{1}{2-\\blam_{i}}\\left(1-\\frac{\\blam_{i}\\delta}{L}\\right)\\left\\| s_{i}\\right\\| _{2}^{2}+\\left(1-\\frac{\\delta}{L}\\right)\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\left\\| s_{j}\\right\\| _{2}^{2} \\text{ (Theorem \\ref{thm:PR} where $\\tL=\\frac{L}{\\blam_i}$)}\\\\\n & \\leq\\left(1-\\frac{\\blam_{i}\\delta}{L}\\right)\\left[\\frac{1}{2-\\blam_{i}}\\left\\| s_{i}\\right\\| _{2}^{2}+\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\left\\| s_{j}\\right\\| _{2}^{2}\\right]\\\\\n & \\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)\\max_{j=1,....,k}\\left\\| s_{j}\\right\\| _{2}^{2}.\n\\end{align*}\nThis completes the proof.\n\\end{proof}","sha256_utf8":"fa103e75dad6d5e80348ec05492fd575d66759c54b2195ba1da097076fb768e1"},{"start":64506,"end":64957,"start_line":557,"end_line":562,"raw_tex":"\\begin{thm} \\textbf{(Survey-Norm Contraction)} \\label{thm:s_monotone} Assume Setting \\ref{setting:localSD}.  Then the following must hold:\n\\begin{equation*}\n\\max_{j=1,...,k}\\left\\| s_{j}^{+} - \\bx \\right\\| _{2}^{2}\\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)\\max_{j=1,...,k}\\left\\| s_{j} - \\bx \\right\\| _{2}^{2}+\\mathrm{O}\\left(\\left\\| \\S - \\bX\\right\\|^{3}\\right),\n\\end{equation*}\nwhere $\\blam_{\\min}$ is defined as in \\eqref{eq:min_lam}.\n\\end{thm}","sha256_utf8":"3c1a0ae26b590fef71e49c9013a5895e7ece97475013071e8f34b9f9b1cb2100"},{"start":64958,"end":65894,"start_line":563,"end_line":570,"raw_tex":"\\begin{proof}\nWithout loss of generality, assume $\\bx = 0$. Consider any fixed $i$. By Lemma \\ref{lem:st_bound}, $\\ts_{i}=\\mathrm{O}\\left(\\left\\| \\S\\right\\| \\right)$. Then,\\begin{align*}\n\\left\\| s_{i}^{+}\\right\\| _{2}^{2} & =\\left\\| \\ts_{i}+\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right)\\right\\| _{2}^{2} \\text{ (Theorem \\ref{thm:approx_s})}\\\\\n & =\\left\\| \\ts_{i}\\right\\| _{2}^{2}+2\\left\\langle \\ts_{i},\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right)\\right\\rangle +\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{4}\\right)\\\\\n & \\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)\\max_{j=1,....,k}\\left\\| s_{j}\\right\\| _{2}^{2}+\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{3}\\right) \\text{ (Lemma \\ref{lem:st_bound})}.\n\\end{align*}\nThe right-hand side is independent of $i$, so taking the max over $i$ on the left-hand side leads to the desired result. We can deduce the general $\\bx{\\neq}0$ case with the change of variables $x{\\leftarrow}(x-\\bx)$.\n\\end{proof}","sha256_utf8":"851df233da05bfdb555febf297f586ac2c92b4cc7ff09c45456c138f9dfa9806"},{"start":66907,"end":67210,"start_line":584,"end_line":585,"raw_tex":"\\begin{cor}\\textbf{(Well-Definedness of Survey Descent Repetitions)}\\label{cor:viable} Procedure \\ref{procedure:SD} is well-defined:  for each $t=0,1,2\\ldots$, the Survey Descent iteration $\\left\\{ \\left(P_{i}^{\\S^{t}}\\right)\\right\\} _{i=1}^{k}$ is feasible and produces a valid output survey.\n\\end{cor}","sha256_utf8":"ffa5f82a1e833acdb4a8cea2b8d9cd91106feed90fb91a923f82b1616ba3301b"},{"start":67211,"end":68014,"start_line":586,"end_line":589,"raw_tex":"\\begin{proof}\nFor $\\S^{0}$ sufficiently close to $\\bX$, Theorem \\ref{thm:s_monotone} inductively implies that $\\left\\| \\S^{t+1} - \\bX \\right\\| <\\left\\| \\S^{t} - \\bX\\right\\| $ for all $t\\in\\nonneg$. In other words, output surveys will always move closer to $\\bX$, and Theorem \\ref{thm:SD_MBP} and \\ref{thm:strictval} continue to hold for Survey Descent iterations $\\left\\{\\left(P_{i}^{\\S^{t}}\\right)\\right\\}_{i=1}^{k}$ for all $t\\in \\nonneg$. Thus, Theorem \\ref{thm:SD_MBP} implies the feasibility all Survey Descent iterations while Theorem \\ref{thm:strictval} implies the validity of output surveys.\n% The viability of implementing the consecutive Survey Descent iterations with only a first-order oracle on $f$ follows directly from Remark \\ref{rem:SD_FOM} and Theorem \\ref{thm:strictval}.\n\\end{proof}","sha256_utf8":"d31879310570759ac825ead18ca13bb2ad303069591af4f86b5dc955c367da63"},{"start":68935,"end":70291,"start_line":603,"end_line":613,"raw_tex":"\\begin{proof}\nBy Theorem \\ref{thm:s_monotone} and Observation \\ref{obs:allT}, there exists $K>0$ such that\n\\begin{equation*}\n\\left\\| \\S^{t+1} - \\bX \\right\\|^{2}\\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)\\left\\| \\S^{t} - \\bX \\right\\|^{2}+K\\left\\| \\S^{t} - \\bX \\right\\|^{3} \\ \\forall \\ t\\in \\nonneg.\n\\end{equation*}\nCorollary \\ref{cor:viable} implies that $\\S^t$ is a valid survey. By Definition \\ref{def:validS}, $\\bX$ is not a valid survey because $f_i(\\bx){=}f_j(\\bx)$ for all $i,j$. Thus, $\\S^t {\\neq}\\bX$, so $\\left\\| \\S^{t} - \\bX \\right\\|^2$ is nonzero and we can divide both sides by it.  We deduce\n\\begin{equation*}\n\\frac{\\left\\| \\S^{t+1} - \\bX \\right\\|^{2}}{\\left\\| \\S^{t} - \\bX \\right\\| ^{2}}\\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)+K\\left\\| \\S^{t} - \\bX\\right\\|  \\ \\forall \\ t\\in \\nonneg.\n\\end{equation*}\n\\sloppy \\edit{For $\\S^{0}$ sufficiently close to $\\bX$, this inequality guarantees that $\\left\\| \\S^{t}{-}\\bX\\right\\|$ monotonically decreases with $t$ and, in particular, $\\left\\| \\S^{t}{-}\\bX\\right\\|{<}\\frac{\\blam_{\\min}\\delta}{LK}$ for all $t\\in\\nonneg$.} Thus, $\\frac{\\left\\| \\S^{t+1} - \\bX \\right\\|}{\\left\\| \\S^{t} - \\bX\\right\\|}{<}1$ implying $\\underset{t\\to\\infty}{\\lim}\\left\\| \\S^{t}{-}\\bX\\right\\|{=}0$. Taking the lim-sup with $t{\\to}\\infty$ on both sides of the above inequality then gives the desired result.\n\\end{proof}","sha256_utf8":"470e9ea0acae38f2cda06f2f133f615596df6ee254f46edda1831a674ee4adfc"},{"start":99811,"end":100387,"start_line":933,"end_line":940,"raw_tex":"\\begin{lem}\\textbf{(Local Properties of the Central Affine Subspace)}\\label{lem:A}\n    For any ball-specifications $\\spec$ sufficiently close to base ball-specifications $\\bspec$, the following properties hold:\n    \\begin{enumerate}\n        \\item $\\A\\spec$ is non-empty with dimension $n{-}\\bk{+}1$.\n        \\item For all $i\\neq j$, the direction $(z_i - z_j)$ is orthogonal to the linear subspace parallel to $\\A\\spec$.\n        \\item For $w\\in\\R^n$, the projection $\\projA{\\spec}(w)$ is well-defined and varies smoothly with $\\spec$ and $w$.\n    \\end{enumerate}\n    \\end{lem}","sha256_utf8":"fa992ea4457d46d91e97d6cb0bfee1e7ac4f828f71f3072e219be02043772943"},{"start":100392,"end":102285,"start_line":941,"end_line":968,"raw_tex":"\\begin{proof}\n    Observe that we can rewrite \\eqref{eq:CAS} as the following $\\left(\\bk{-}1\\right){\\times}n$ linear system:\\begin{equation}\n    \\A\\spec=\\left\\{ x\\in\\R^{n}:\\ M\\spec x=b\\spec \\right\\} ,\\label{eq:CAS_sys}\n    \\end{equation}\n    where \n    \\[\n    M\\spec\\equiv2\\begin{bmatrix}\\left(z_{2}-z_{1}\\right)^{T}\\\\\n    \\vdots\\\\\n    \\left(z_{\\m}-z_{1}\\right)^{T}\n    \\end{bmatrix}\n    \\text{ and }\n    b\\spec\\equiv\\begin{bmatrix}\\left(\\left\\| z_{2}\\right\\| _{2}^{2}-r_{2}^{2}\\right)-\\left\\| z_{1}\\right\\| _{2}^{2}+r_{1}^{2}\\\\\n    \\vdots\\\\\n    \\left(\\left\\| z_{\\bk}\\right\\| _{2}^{2}-r_{\\bk}^{2}\\right)-\\left\\| z_{1}\\right\\| _{2}^{2}+r_{1}^{2}\n    \\end{bmatrix}.\\\n    \\]\n    \n    Consider $\\A\\bspec$. Since $\\left\\{ \\bz_{j}\\right\\} _{j=1}^{\\bk}$ are linearly independent,\n    $\\left\\{ \\bz_{j}-\\bz_{1}\\right\\} _{j=2}^{\\bk}$---which are the rows of $M\\bspec$---are also linearly independent. For $\\spec$ sufficiently close to $\\bspec$, the rows of $M\\spec$ will remain linearly independent. In this case, the linear system \\eqref{eq:CAS_sys} is then necessarily solvable: $\\A\\spec$ is non-empty. Let $x_0$ be such a solution. Then, $\\A\\spec$ consists of all vectors $x \\in \\R^n$ such that $x{-}x_0$ is orthogonal to the $m{-}1$ linearly independent rows of $M\\spec$. Thus, $\\Dim\\left\\{\\A\\spec\\right\\}{=}n{-}\\bk{+}1$. \n    \n    The vectors $\\left\\{(z_i - z_j)\\right\\}_{j\\neq i}$ clearly all lie in the rowspace of $M\\spec$. Hence, the vectors$\\left\\{(z_i - z_j)\\right\\}_{j\\neq i}$ are orthogonal to all direction in $\\A\\spec$.\n    \n    Finally, write $M$ instead of $M\\spec$ for simplicity. Since $M$ has independent rows, the projection of $w$ onto $\\A\\spec$ is explicitly given by\n    \\begin{equation*}\n    \\projA{\\spec}\\left(w\\right)=\\left(I-M\\left(M^{T}M\\right)^{-1}M^{T}\\right)w+M\\left(M^{T}M\\right)^{-1}b,\n    \\end{equation*}\n    which varies smoothly with $\\spec$ and $w$.\n    \\end{proof}","sha256_utf8":"0b64b4a1919333c8221c74f2f368fc93ad86ec08e2c24f4594c8158f44356f24"},{"start":103359,"end":103612,"start_line":979,"end_line":981,"raw_tex":"\\begin{lem}\n    \\textbf{\\label{lem:core} (Local Properties of Core)} For any ball-specifications $\\spec$ sufficiently close to base ball-specifications $\\bspec$, the core in \\eqref{eq:core} is well-defined and varies smoothly with $\\spec$.\n    \\end{lem}","sha256_utf8":"576c3ad97a528ec57286bb0d4ed5edf4f0454ac99889bb4ce57747c10c3a5629"},{"start":103617,"end":104044,"start_line":982,"end_line":983,"raw_tex":"\\begin{proof} By the second fact in Lemma \\ref{lem:A}, for any pair $i\\neq j$, the difference $(z_i{-}z_j)$ is orthogonal to the linear subspace parallel to $\\A\\param$. Therefore, $\\projA{\\param}(z_i)$ and $\\projA{\\param}(z_j)$ are the same. Hence, \\eqref{eq:core} is well-defined. The smoothness of the $c\\spec$ with respect to $\\spec$ then follows from \\eqref{eq:core} and the third fact of Lemma \\ref{lem:A}.\n    \\end{proof}","sha256_utf8":"cd495af875972d840879b427062858296ece71740da6dcceddc18bfe1e73b883"},{"start":104054,"end":104233,"start_line":985,"end_line":988,"raw_tex":"\\begin{lem}\n    \\textbf{(The Base Case)} \\label{lem:linA}\n    At base ball-specifications, the central affine subspace is a linear subspace and the core is non-zero.\n    \\end{lem}","sha256_utf8":"98b434d6f5428c79dee990b39c6fadbe110747efb86bcb8cac95c120cf916dd8"},{"start":104238,"end":105097,"start_line":989,"end_line":998,"raw_tex":"\\begin{proof}\n    By Definition \\ref{def:base}, $\\br_{j}^{2}=\\left\\| \\bz_{j}\\right\\| _{2}^{2}$\n    for all $j$. Thus, \\eqref{eq:CAS} simplifies to\n    \\begin{equation*}\n    \\A\\bspec=\\left\\{ x\\in\\R^{n}:\\ 2\\bz_{j}^{T}x\\text{ equal  for all } j=1,\\dots,\\bk\\right\\}.\n    \\end{equation*}\n    Clearly, the origin satisfies the RHS conditions above, so $0\\in\\A\\bspec$.\n    \n    \\sloppy Next, assume for contradiction that $0{=}c\\bspec$. By definition, $c\\bspec$ is the projection of $\\bz_j$ onto $\\A\\bspec$ for any $j$, so the vectors $\\left\\{ \\bz_{j}-c\\bspec\\right\\}_{j=1}^{\\bk}{=}\\left\\{ \\bz_{j}\\right\\} _{j=1}^{\\bk}$ must be orthogonal to $\\A\\bspec$. The linear independence of $\\left\\{ \\bz_{j}\\right\\}_{j=1}^{\\bk}$ implies $\\Dim\\left\\{ \\A\\bspec\\right\\}{\\leq}n{-}\\bk$, which contradicts the first fact of Lemma \\ref{lem:A}. Thus, $0{\\neq}c\\bspec$.\n    \\end{proof}","sha256_utf8":"263a780f50f6475bfe53d847cfc6d35f39df134f0b5f58026259ff7f584f4f10"},{"start":105107,"end":105380,"start_line":1000,"end_line":1003,"raw_tex":"\\begin{lem}\n    \\textbf{(Local Properties of the Margin)\\label{lem:margin}} \n    For ball-specifications $\\spec$ near some base ball-specifications $\\bspec$, the margin in \\eqref{eq:margin} is well-defined, strictly-positive, and varies smoothly with $\\spec$.\n    \\end{lem}","sha256_utf8":"4e21c23bbcc8d172ae1ef0d5455fe0bc10a749d965ac11a0d29d7cd1447a1ee6"},{"start":105385,"end":106814,"start_line":1004,"end_line":1019,"raw_tex":"\\begin{proof}\n    It suffices to show that the square of the margin,\\begin{equation}\n    r_{j}^{2}-\\left\\| z_{j}-c\\spec\\right\\| _{2}^{2}, \\label{eq:margin_sq}\n    \\end{equation}\n    does not depend on the index $j$, is strictly-positive, and varies smoothly with $\\param$. First, consider the core and central affine subspace exactly at base ball-specifications $\\bspec$. By Lemma\n    \\ref{lem:linA}, since $0 \\in \\A\\bspec$ and $\\projA{\\bspec}\\left(\\bz_{j}\\right){=}c\\bspec{\\neq}0$\n    for all $j$, the definition of a projection gives\n    \\begin{equation*}\n    \\left\\| \\bz_{j}-c\\bspec\\right\\| _{2}<\\left\\| \\bz_{j}-0\\right\\| _{2}=\\br_{j}\\ \\forall\\ j=1,\\dots,\\bk.\n    \\end{equation*}\n    Thus, \\eqref{eq:margin_sq} is strictly positive, which implies the argument of the square-root in \\eqref{eq:margin} is strictly positive. Next, observe that\n    \\begin{align*}\n    r_{j}^{2}-\\left\\| z_{j}-c\\spec\\right\\| _{2}^{2} & =-\\left\\| c\\spec\\right\\| _{2}^{2}+\\underbrace{2z_{j}^{T}c\\spec-\\left\\| z_{j}\\right\\| _{2}^{2}+r_{j}^{2}}_{(\\star)_j} \\ \\forall\\  j=1,\\dots,\\bk.\n    \\end{align*}\n    Since $c\\spec{\\in}\\A\\spec$, the term $(\\star)_j$ is equal for all $j$. Therefore, \\eqref{eq:margin_sq} is $j$-independent, and $\\rho\\spec$ in \\eqref{eq:margin} is well-defined. The smoothness of $\\rho\\spec$  with respect to $\\spec$ then follows directly from \\eqref{eq:margin} and the smoothness of the core (Lemma \\ref{lem:core}).\n    \\end{proof}","sha256_utf8":"4c9ad765b2ca4533a42ebc82b379c3987fdd69b31aa4939fcde6d0f103b13359"},{"start":106824,"end":107179,"start_line":1021,"end_line":1026,"raw_tex":"\\begin{lem}\n    \\label{lem:B}\\textbf{(Local Properties of the Feasible Region)} For ball-specifications\n    $\\spec$ near some base ball-specifications $\\bspec$, the feasible region is nonempty and satisfies\\begin{equation}\n    \\B\\spec=\\left\\{ x\\in\\A\\spec:\\ \\left\\| x-c\\spec\\right\\| _{2}=\\rho\\spec\\right\\}.\\label{eq:Bdef2} \n    \\end{equation}\n    \\end{lem}","sha256_utf8":"455ab9c3ed90be5270c418f3928759ba51e8f80a92d621784aa502d7f1ad210e"},{"start":107184,"end":109351,"start_line":1027,"end_line":1057,"raw_tex":"\\begin{proof}\n    \\noindent\\textit{Equivalence of \\eqref{eq:MBP_cons} and \\eqref{eq:Bdef2}:}\n    Expanding and rearranging \\eqref{eq:MBP_cons} gives\n    \\begin{align*}\n    \\left\\| x-z_{j}\\right\\| _{2}=r_{j}\\ \\forall\\ j & \\iff\\left\\| x\\right\\| _{2}^{2}=2z_{j}^{T}x-\\left\\| z_{j}\\right\\| _{2}^{2}+r_{j}^{2}\\ \\forall\\ j\\\\\n    & \\implies2z_{j}^{T}x-\\left\\| z_{j}\\right\\| _{2}^{2}+r_{j}^{2}=2z_{i}^{T}x-\\left\\| z_{i}\\right\\| _{2}^{2}+r_{i}^{2}\\ \\forall\\ i\\neq j.\n    \\end{align*}\n    Hence, $\\B\\spec\\subseteq\\A\\spec$ implying\n    \\begin{equation*}\n    \\B\\spec=\\left\\{ x\\in\\A\\spec:\\ \\left\\| x-z_{j}\\right\\| _{2}=r_{j}\\text{  for all } j\\right\\}.\n    \\end{equation*}\n    Since $c\\left(r,Z\\right){=}\\projA{\\spec}\\left(z_{i}\\right)$, for any $x{\\in}\\A\\spec$, the vectors $\\left(x{-}c\\left(r,Z\\right)\\right)$ and $\\left(z_{j}{-}c\\spec\\right)$ are orthogonal. Then, applying the Pythagorean Theorem to the equality condition in \\eqref{eq:MBP_cons} gives\n    \\begin{equation*}\n    r_{j}^{2} = \\left\\| x-z_i\\right\\|_{2}^{2} =\\left\\| x-c\\left(r,Z\\right)\\right\\| _{2}^{2}+\\left\\| z_{j}-c\\spec\\right\\|_{2}^{2} \\ \\forall \\ x\\in\\A\\spec \\text{ and } j=1,\\dots,k.\n    \\end{equation*}\n    We then deduce \\eqref{eq:Bdef2} by subtracting $\\left\\| z_{j}{-}c\\spec\\right\\|_{2}^{2}$ from the left and right sides, and applying the definition of $\\rho\\spec$ in \\eqref{eq:margin}.\n    \n    \\noindent \\textit{Non-Emptiness of $\\B\\spec$:} By the first fact of Lemma \\ref{lem:A}, the affine subspace $\\A\\spec$ is $(n{-}m{+}1)$-dimensional, so $\\A\\spec$ is a non-empty space. Thus, there exists $a\\in\\A\\spec$ such that $a\\neq c\\spec$. Choose any such $a$ and consider the direction \n    \\begin{equation*}\n    d = \\frac{1}{\\left\\| a-c\\spec\\right\\|} \\left(a-c\\spec\\right)\n    \\end{equation*}\n    along with the vector\n    \\begin{equation*}\n    v = c\\spec+\\rho\\spec d.\n    \\end{equation*}\n    Then, by construction,\n    \\begin{equation*}\n    \\left\\| v-c\\spec\\right\\| _{2}=\\rho\\spec.\n    \\end{equation*}\n    Since $\\A\\spec$ contains both $c\\spec$ and $d$, we deduce $v\\in\\A\\spec$. Thus, $v$ satisfies all conditions in \\eqref{eq:Bdef2}, so $v\\in\\B\\spec$, and hence $\\B\\spec$ is non-empty.\n    \\end{proof}","sha256_utf8":"8caa0c52ace9a04fb9cde23d31b1f4f2f8978470ff5c244cf2fabc439764ee32"},{"start":109361,"end":109936,"start_line":1059,"end_line":1067,"raw_tex":"\\begin{thm}\\label{thm:MBP_sol}\n    \\textbf{(Local, Explicit Solution of $\\boldsymbol{MBP\\param}$)} For parameters $\\param$ near some base parameters $\\bparam$, the optimal solution to $\\MBP\\param$ is unique, varies smoothly with $\\param$, and is given by\\begin{equation}\\label{eq:x}\n    x\\param = c\\spec+\\rho\\spec u\\param,\n    \\end{equation}\n    where\\begin{equation}\\label{eq:u}\n    u\\param = \\frac{1}{\\left\\| \\projA{\\spec}\\left(y\\right)-c\\spec\\right\\| _{2}}\\left(\\projA{\\spec}\\left(y\\right)-c\\spec\\right).\n    \\end{equation}\n    In particular, $x\\bparam = 0$.\n    \\end{thm}","sha256_utf8":"0e0b69723f4d0443020ec9a3dddad4b3555d08dfb2b3ab7cd7b311452f2bbecb"},{"start":109941,"end":112935,"start_line":1068,"end_line":1098,"raw_tex":"\\begin{proof}\n    \\textit{The vector $u\\param$ is well-defined:} To show the well-definedness of the expression in \\eqref{eq:u}, observe the following at base parameters $\\bparam$:\n    \n    \\begin{align*}\n    \\projA{\\bspec}\\left(\\by\\right)-c\\bspec &= \\projA{\\bspec}\\left(-\\sum_{j=1}^{\\bk}\\clam_{j}\\bar{z}_{j}\\right)-c\\bspec &\\text{ (By Definition \\ref{def:base})}\\\\\n    & = -\\sum_{j=1}^{\\bk}\\clam_{j}\\projA{\\bspec}\\left(\\bar{z}_{j}\\right)-c\\bspec &\\text{ (By linearity of $\\A\\bspec$ from Lemma \\ref{lem:linA})}\\\\\n    & = -\\left(\\sum_{j=1}^{\\bk}\\clam_{j}+1\\right)c\\bspec &\\text{ (By definition of core \\eqref{eq:core})}\\\\\n    & \\neq 0. &\\text{ (By Definition \\ref{def:base} and Lemma \\ref{lem:linA})}.\n    \\end{align*}\n    By the smoothness of the projection (third fact of Lemma \\ref{lem:A}), we deduce $\\projA{\\spec}\\left(y\\right){-}c\\spec{\\neq}0$ for $\\param$ sufficiently close to $\\bparam$ as well. Thus, \\eqref{eq:u} is well-defined. Moreover, the smoothness of the projection and \\eqref{eq:u} further shows that $u\\param$ is smooth with respect to $\\param$.\n    \n    \\noindent \\textit{Derivation of $x\\param$:} By Lemma \\ref{lem:B}, $\\B\\spec\\subseteq\\A\\spec$ so\\begin{align*}\n    x\\param=\\projB{\\spec}\\left(y\\right) & =\\projB{\\spec}\\left(\\projA{\\spec}\\left(y\\right)\\right).\n    \\end{align*}\n    Lemma \\ref{lem:B} also states that $\\B\\spec$ is simply the $c\\spec$-centered ball with radius $\\rho\\spec$ restricted to $\\A\\spec$. Thus, $\\projB{\\spec}\\left(\\projA{\\spec}\\left(y\\right)\\right)$ must be the (necessarily unique) point that\n    \\begin{enumerate}\n        \\item lies on the straight line from $c\\spec$ to $\\projA{\\spec}\\left(y\\right)$, and\n        \\item is exactly the distance $\\rho\\spec$ from $c\\spec$.\n    \\end{enumerate}\n    Formalizing the above two criteria leads to the explicit-form\n    \\begin{equation*}\n    x\\param=c\\spec+\\frac{\\rho\\spec}{\\left\\| \\projA{\\spec}\\left(y\\right)-c\\spec\\right\\| _{2}}\\left(\\projA{\\spec}\\left(y\\right)-c\\spec\\right)=c\\spec+\\rho\\spec u\\param.\n    \\end{equation*}\n    The smoothness of $x\\param$ follows directly from the smoothness of the core (Lemma \\ref{lem:core}), the margin (Lemma \\ref{lem:margin}), and $u\\param$ (shown above).\n    \n    \\noindent \\textit{Demonstration of $x\\bparam{=}0$:} Observe that the Lagrangian of $\\MBP\\bparam$ is\n    \\[\n    \\L(x,\\lambda) = \\left\\| x-\\by\\right\\| _{2}^{2} + \\sum_{j=1}^\\bk \\lambda_j \\left\\| x-\\bz_{j}\\right\\| _{2}^{2},\n    \\]\n    where $\\lambda_j$ is the $j$-th element of a Lagrange multipliers vector $\\lambda \\in \\R^\\bk$. By Definitions \\ref{def:MBP}-\\ref{def:base}, the zero-vector is a feasible solution to $\\MBP\\bparam$. A feasible vector $x^*$ is the solution of $\\MBP\\bparam$ if and only if there exists $\\lambda^*$ such that $x^*$ minimizes $\\L(\\cdot,\\lambda^*)$. It is easy to check using Definition \\ref{def:base} and first-order optimality conditions that $x^*{=}0$ is the unique minimizer of $\\L(\\cdot,\\clam)$. Thus, $x\\bparam{=}0$ is the unique solution to $\\MBP\\bparam$.\n    \\end{proof}","sha256_utf8":"8a65ce290948216b13c3db7ee7552a09537453f46dc8d4c8be7117125f6a32d0"},{"start":114142,"end":114781,"start_line":1109,"end_line":1115,"raw_tex":"\\begin{lem}\\label{lem:MBP_lam}\\textbf{(Properties of MBP Lagrange Multipliers)} Suppose the parameters $\\param$ are sufficiently close to base parameters $\\bparam$ with corresponding base weights vector $\\clam$. Consider the optimal solution $x\\param$ of $\\MBP\\param$. Then, the following hold:\n    \\begin{enumerate}\n        \\item There exists a unique Lagrange multiplier vector, $\\lambda\\param\\in\\R_{++}^{\\bk}$, satisfying the stationarity condition \\eqref{eq:MBP_FOC}.\n        \\item The vector $\\lambda\\param$ varies smoothly with $\\param$.\n        \\item At the base parameters, $\\lambda\\bparam=\\clam$.\n    \\end{enumerate}\n    \\end{lem}","sha256_utf8":"b5d3e0496a3031488d3ad259a3ca5d59cafd76394945451faac64b1397b8c6e7"},{"start":114786,"end":116620,"start_line":1116,"end_line":1138,"raw_tex":"\\begin{proof}\n    Since $x\\param$ is optimal for $\\MBP\\param$, there must exist a Lagrange multipliers vector $\\lambda\\param\\in\\R^{\\bk}$ satisfying \\eqref{eq:MBP_FOC}. We first show $\\lambda\\param$ is unique. Finding Lagrange multipliers satisfying \\eqref{eq:MBP_FOC} is equivalent to finding minimizers of the function\n    \\begin{equation*}\n    F\\left(\\lambda\\right)=\\left\\| x\\param-y+X\\lambda\\right\\| _{2}^{2},\n    \\end{equation*}\n    where we define the matrix\n    \\[X=\\begin{bmatrix}x\\param{-}z_{1}, \\dots, x\\param{-}z_{\\bk}\\end{bmatrix}\\in\\R^{n\\times\\bk}.\\]\n    Then, a vector $\\lambda$ minimizes $F$ if and only if it satisfies the below $\\bk\\times\\bk$ linear system:\n    \\begin{equation*}\n    X^{T}\\left(x\\param-y+X\\lambda\\right)=0\\iff X^{T}X\\lambda=X^{T}\\left(y-x\\param\\right).\n    \\end{equation*}\n    By Theorem \\ref{thm:MBP_sol}, for $\\param$ close to $\\bparam$, the MBP-solution $x\\param$ is close to zero. Therefore, columns of $X$ are close to $\\left\\{-\\bz_{j}\\right\\}_{j=1}^{\\bk}$, which are linearly independent (Definition \\ref{def:base}). This implies $X^{T}X$ is invertible, and the minimizer of $F$ is unique and explicitly given by \n    \\begin{equation*}\n    \\lambda\\param = \\argmin_\\lambda F\\left(\\lambda\\right)=\\left(X^{T}X\\right)^{-1}X^{T}\\left(y-x\\param\\right).\n    \\end{equation*}\n    The smoothness of $\\lambda\\param$ with respect to $\\param$ follows from the smoothness of $x\\param$ (Theorem \\ref{thm:MBP_sol}) and the smoothness of the above matrix operations. \n    \n    Finally, it follows directly from Definition \\ref{def:base} and Theorem \\ref{thm:MBP_sol} that\n    \\begin{equation*}\n    x\\bparam-\\by+\\sum_{j=1}^{\\bk}\\clam_{j}\\left[x\\bparam-\\bz_{j}\\right]=0,\n    \\end{equation*}\n    which implies the Lagrange multiplier vector at the base parameters is uniquely $\\lambda\\bparam=\\clam$.\n    \\end{proof}","sha256_utf8":"41f403ee33f993f5610aded3895e63c5f469773ee4cc33829f9d058a749bd807"},{"start":116630,"end":117569,"start_line":1140,"end_line":1151,"raw_tex":"\\begin{thm}\\textbf{(Inequality-Constrained MBP)}\\label{thm:MBP_ineq}\n    Consider the following inequality-constrained MBP:\n    \\begin{align*}\n    %\\begin{aligned}\n    \\min_{x\\in\\R^{n}}\\ & \\left\\| x-y\\right\\| _{2}^{2}\\\\\n    \\mathrm{s.t.}\\ & \\left\\| x-z_{j}\\right\\| _{2}^{2} \\leq r_{j}^{2}\\ \\forall\\ j=1,\\dots,\\bk,\n    %\\end{aligned}%\\label{eq:MBP_ineqFull}\n    \\end{align*}\n    and denote it as $\\MBPl\\param$. For parameters $\\param$ sufficiently close to $\\bparam$, $\\MBPl\\param$ is feasible and possesses the same unique optimal solution, $x\\param$, and unique Lagrange multipliers, $\\lambda\\param$, as $\\MBP\\param$. Consequently, all constraints of $\\MBPl\\param$ are tight. Moreover, the optimal solution and multipliers satisfy\\begin{equation}\n    x\\param = \\mathrm{O}\\left(\\|\\param-\\bparam\\|\\right)\\text{ and } \\lambda\\param = \\clam + \\mathrm{O}\\left(\\|\\param-\\bparam\\|\\right).\\label{eq:MBP_Onotation}\n    \\end{equation}\n    \\end{thm}","sha256_utf8":"438725bd7f9f272a4abb311cd16054a60bdd3652391578b820fe940b0df053eb"},{"start":117574,"end":119527,"start_line":1152,"end_line":1163,"raw_tex":"\\begin{proof}\n    The feasibility of $\\MBPl\\param$ follows directly from Lemma \\ref{lem:B} since $\\B\\param$ is a subset of the feasible region of $\\MBPl\\param$.\n    \n    $\\MBPl\\param$ possesses a unique minimizer since it minimizes a strongly convex objective on a convex feasible region. Moreover, checking the KKT conditions is sufficient for demonstrating optimality because the $\\|x - z_j\\|_2^2$-term in the constraints is convex and differentiable for all $j$. Therefore---denoting the unique pair of optimal solutions and Lagrange multipliers of $\\MBP\\param$ as $\\left(x\\param, \\lambda\\param\\right)$---it suffices to check that $\\left(x\\param, \\lambda\\param\\right)$ satisfies the KKT conditions of $\\MBPl\\param$:\n    \\begin{enumerate}\n        \\item \\textbf{(Stationarity)} Follows from $\\MBPl\\param$ and $\\MBP\\param$ having the same stationarity condition \\eqref{eq:MBP_FOC}.\n        \\item \\textbf{(Primal Feasibility)} Follows from $\\B\\param$ being a subset of the feasible region of $\\MBPl\\param$.\n        \\item \\textbf{(Dual Feasibility)} The inequality constraints of $\\MBPl\\param$ require the non-negativity of all Lagrange multipliers. By Definition \\ref{def:base}, the elements of $\\clam$ are strictly-positive. Thus, by Lemma \\ref{lem:MBP_lam}, the elements of $\\lambda\\param$ are also strictly-positive for $\\param$ sufficiently close to $\\bparam$.\n        \\item \\textbf{(Complementary Slackness)} \\sloppy By construction, $x\\param$ satisfies all the inequality constraints of $\\MBPl\\param$ with equality. Correspondingly, $\\lambda\\param$ are all strictly positive.\n    \\end{enumerate}\n    Therefore, $\\left(x\\param, \\lambda\\param\\right)$ is the unique pair of optimal solutions and Lagrange multipliers for $\\MBPl\\param$ as well. The characterizations in \\eqref{eq:MBP_Onotation} directly follows from Theorem \\ref{thm:MBP_sol} and Lemma \\ref{lem:MBP_lam} as well as recalling that smooth functions are locally Lipschitz.\n    \\end{proof}","sha256_utf8":"a82a9bea8702908715d073b4bde281e0ad805f373703a985351d1ffb08a86da5"},{"start":122359,"end":123022,"start_line":1194,"end_line":1199,"raw_tex":"\\begin{proof}\n    It is easy to check from the mapping definitions \\eqref{eq:MBP_mappings} and the Taylor expansion \\eqref{eq:taylor} that \n    \n    $$\\left\\|\\left(y_{i}\\left(\\S\\right), \\left[ z_{ij}\\left(\\S\\right)\\right]_{j\\neq i}, \\left[r_{ij}\\left(\\S\\right)\\right]_{j\\neq i},\\left[ s_{j}\\right]_{j\\neq i}\\right) - \\left(y_{i}\\left(\\bX\\right), \\left[ z_{ij}\\left(\\bX\\right)\\right]_{j\\neq i}, \\left[r_{ij}\\left(\\bX\\right)\\right]_{j\\neq i}, [\\bx]_{j\\neq i}\\right)\\right\\| = \\mathrm{O}\\left(\\|\\S-\\bX\\|\\right).$$\n    The result then follows directly from Observation \\ref{obs:SD_MOS}, Observation \\ref{obs:SD_as_MBP}, and Theorem \\ref{thm:MBP_ineq}.\n    \\end{proof}","sha256_utf8":"ccfbe5da2a8493437b9aae62f72d6da069b93db249d6faaa4818bf70153a474d"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"调查点的Q线性收敛","source_units":[{"start":68514,"end":68934,"start_line":597,"end_line":602,"raw_tex":"\\begin{thm}\n\\textbf{(Q-Linear Convergence of Survey Points)\\label{thm:q_linear}} Procedure \\ref{procedure:SD} satisfies the following property:\\begin{equation}\n\\underset{t\\to\\infty}{\\limsup}\\frac{\\left\\| \\S^{t+1} - \\bX \\right\\| ^{2}}{\\left\\| \\S^{t}  - \\bX \\right\\| ^{2}}\\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right),\\label{eq:s_result}\n\\end{equation}\nwhere $\\blam_{\\min}$ is defined as in \\eqref{eq:min_lam}.\n\\end{thm}","sha256_utf8":"8156d01578966ac60bb468c4034f228742c56e40cfc531fc8348559f2c8a364c"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"Q线性收敛完整证明","source_units":[{"start":68935,"end":70291,"start_line":603,"end_line":613,"raw_tex":"\\begin{proof}\nBy Theorem \\ref{thm:s_monotone} and Observation \\ref{obs:allT}, there exists $K>0$ such that\n\\begin{equation*}\n\\left\\| \\S^{t+1} - \\bX \\right\\|^{2}\\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)\\left\\| \\S^{t} - \\bX \\right\\|^{2}+K\\left\\| \\S^{t} - \\bX \\right\\|^{3} \\ \\forall \\ t\\in \\nonneg.\n\\end{equation*}\nCorollary \\ref{cor:viable} implies that $\\S^t$ is a valid survey. By Definition \\ref{def:validS}, $\\bX$ is not a valid survey because $f_i(\\bx){=}f_j(\\bx)$ for all $i,j$. Thus, $\\S^t {\\neq}\\bX$, so $\\left\\| \\S^{t} - \\bX \\right\\|^2$ is nonzero and we can divide both sides by it.  We deduce\n\\begin{equation*}\n\\frac{\\left\\| \\S^{t+1} - \\bX \\right\\|^{2}}{\\left\\| \\S^{t} - \\bX \\right\\| ^{2}}\\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)+K\\left\\| \\S^{t} - \\bX\\right\\|  \\ \\forall \\ t\\in \\nonneg.\n\\end{equation*}\n\\sloppy \\edit{For $\\S^{0}$ sufficiently close to $\\bX$, this inequality guarantees that $\\left\\| \\S^{t}{-}\\bX\\right\\|$ monotonically decreases with $t$ and, in particular, $\\left\\| \\S^{t}{-}\\bX\\right\\|{<}\\frac{\\blam_{\\min}\\delta}{LK}$ for all $t\\in\\nonneg$.} Thus, $\\frac{\\left\\| \\S^{t+1} - \\bX \\right\\|}{\\left\\| \\S^{t} - \\bX\\right\\|}{<}1$ implying $\\underset{t\\to\\infty}{\\lim}\\left\\| \\S^{t}{-}\\bX\\right\\|{=}0$. Taking the lim-sup with $t{\\to}\\infty$ on both sides of the above inequality then gives the desired result.\n\\end{proof}","sha256_utf8":"470e9ea0acae38f2cda06f2f133f615596df6ee254f46edda1831a674ee4adfc"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"统一三阶余项递推","source_units":[{"start":68949,"end":69253,"start_line":604,"end_line":607,"raw_tex":"By Theorem \\ref{thm:s_monotone} and Observation \\ref{obs:allT}, there exists $K>0$ such that\n\\begin{equation*}\n\\left\\| \\S^{t+1} - \\bX \\right\\|^{2}\\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)\\left\\| \\S^{t} - \\bX \\right\\|^{2}+K\\left\\| \\S^{t} - \\bX \\right\\|^{3} \\ \\forall \\ t\\in \\nonneg.\n\\end{equation*}\n","sha256_utf8":"7c3457c8f92f806cf5aecce7be1e0cd14c019c5af394e15ae96af3de4066c129"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"有效性保证误差可除","source_units":[{"start":69253,"end":69760,"start_line":608,"end_line":611,"raw_tex":"Corollary \\ref{cor:viable} implies that $\\S^t$ is a valid survey. By Definition \\ref{def:validS}, $\\bX$ is not a valid survey because $f_i(\\bx){=}f_j(\\bx)$ for all $i,j$. Thus, $\\S^t {\\neq}\\bX$, so $\\left\\| \\S^{t} - \\bX \\right\\|^2$ is nonzero and we can divide both sides by it.  We deduce\n\\begin{equation*}\n\\frac{\\left\\| \\S^{t+1} - \\bX \\right\\|^{2}}{\\left\\| \\S^{t} - \\bX \\right\\| ^{2}}\\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)+K\\left\\| \\S^{t} - \\bX\\right\\|  \\ \\forall \\ t\\in \\nonneg.\n\\end{equation*}\n","sha256_utf8":"2ccbe86910aeccdd8e18ae89d038a3cc8ad54b7da942acadf8308eb18336a61b"}]},{"id":"S005","semantic_type":"proof_step","kind":"proof_step","title":"局部收敛并取上极限","source_units":[{"start":69760,"end":70280,"start_line":612,"end_line":612,"raw_tex":"\\sloppy \\edit{For $\\S^{0}$ sufficiently close to $\\bX$, this inequality guarantees that $\\left\\| \\S^{t}{-}\\bX\\right\\|$ monotonically decreases with $t$ and, in particular, $\\left\\| \\S^{t}{-}\\bX\\right\\|{<}\\frac{\\blam_{\\min}\\delta}{LK}$ for all $t\\in\\nonneg$.} Thus, $\\frac{\\left\\| \\S^{t+1} - \\bX \\right\\|}{\\left\\| \\S^{t} - \\bX\\right\\|}{<}1$ implying $\\underset{t\\to\\infty}{\\lim}\\left\\| \\S^{t}{-}\\bX\\right\\|{=}0$. Taking the lim-sup with $t{\\to}\\infty$ on both sides of the above inequality then gives the desired result.\n","sha256_utf8":"f3fd8ba800fcc4742fb74eb01afa25af57706e340d07c481d9658d83ad13cd1b"}]},{"id":"S006","semantic_type":"notation","kind":"notation","title":"局部线性近似记号","source_units":[{"start":1861,"end":2192,"start_line":38,"end_line":38,"raw_tex":"To approach our target---a fast local iteration for nonsmooth optimization---let us recall, as motivation, the classical foundations of gradient descent (GD). For some function $h$, denote the linear approximation of $h$ centered at some differentiable iterate $\\tx$ by \\[\\ell_{\\tx}^{h}(x)\\equiv h(\\tx) + \\nabla h(\\tx)^T(x-\\tx),\\]\n","sha256_utf8":"d1500ef829c50fb34a7dcd3009032a6f63620172c3dc633ead8c25dc6d20de0e"}]},{"id":"S007","semantic_type":"algorithm","kind":"algorithm","title":"调查下降子问题与输出","source_units":[{"start":5027,"end":5850,"start_line":55,"end_line":61,"raw_tex":"\\begin{defn} \\textbf{(Survey Descent Iteration $\\boldsymbol{\\left\\{\\left(P_i^\\S\\right)\\right\\}_{i=1}^k}$)\\label{def:SD_Method}} Given a \\textit{survey} of $k$ points, $\\S=\\left\\{ s_{i}\\right\\} _{i=1}^{k}$, at which the nonsmooth objective $h$ is differentiable and a step-control parameter $L$, for each $i=1,{\\dots},k$, define the \\textit{$i$-th subproblem} $\\left(P_i^\\S\\right)$ as follows:\n\\begin{align}\n\\min_{x}\\ & \\left\\| x-\\left(s_{i}-\\frac{1}{L}\\nabla h\\left(s_{i}\\right)\\right)\\right\\| _{2}^{2}\\label{eq:SD_objective}\\\\\n\\mathrm{s.t.}\\ & \\ell_{s_j}^{h}(x)+\\frac{L}{2}\\left\\| x-s_{j}\\right\\| _{2}^{2}\\leq \\ell_{s_i}^{h}(x)\\ \\forall\\ j\\neq i.\\label{eq:SD_constraints}\n\\end{align}\nWe refer to the solving of all subproblems $\\left\\{\\left(P_i^\\S\\right)\\right\\}_{i=1}^k$ as a \\textit{Survey Descent iteration}.\n\\end{defn}","sha256_utf8":"9cff18d3dc4ae00d29f9738249198246c759e5a9dfa46bd090f037de070316be"},{"start":5852,"end":6316,"start_line":63,"end_line":63,"raw_tex":"When $\\left(P_i^\\S\\right)$ is feasible, we denote its optimal solution as $s_i^+$. When all subproblems are feasible, we say that the entire Survey Descent iteration is \\textit{feasible} and call $\\S^+=\\left\\{s_i^+\\right\\}_{i=1}^k$ the \\textit{outputs} of the iteration; otherwise, we say the entire iteration is \\textit{infeasible}. After a feasible iteration, we would update $\\S{\\leftarrow}\\S^{+}$ and repeat the Survey Descent iteration on the updated survey.\n","sha256_utf8":"a9d78312bb0603d717b1c48a6a8c9d6b1b6b47b051bdb9053b34098789a174ac"}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","title":"凸目标的非退化极小点","source_units":[{"start":39374,"end":39517,"start_line":279,"end_line":281,"raw_tex":"\\begin{equation}\n    \\bx = \\arg\\min_{x \\in \\R^n} f(x) \\text{ and } 0\\in\\Relint\\left( \\partial f(\\bx) \\right),\\label{eq:nondegen}\n\\end{equation}","sha256_utf8":"d3660dec29db7edc5b2a7048fe6ee6632adffbce8b449aff47e02d62fa2729c7"},{"start":39518,"end":39628,"start_line":282,"end_line":282,"raw_tex":"where $\\Relint(\\cdot)$ denotes the relative interior. We further assume the following structure around $\\bx$.\n","sha256_utf8":"7c1ed219818d2c94533ce259325ff5bb619458cac0182bd3dad57db1aa154041"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"强二阶光滑最大值函数","source_units":[{"start":39628,"end":40474,"start_line":283,"end_line":291,"raw_tex":"\\begin{defn}\\label{def:strong_max}\nA function $f:\\R^n \\to \\R$ is a \\textit{strong $\\C^2$ max function} if it is locally expressible near $\\bx$ as\\begin{equation}\\label{eq:f_objective}\nf\\left(x\\right)=\\underset{i=1,...k}{\\max}f_{i}\\left(x\\right),\n\\end{equation}\nwhere $k$ is some finite number, the \\textit{components} $\\left\\{f_i\\right\\}_{i=1}^k$ are $\\C^2$-functions satisfying $f_i(\\bx)=f(\\bx)$ for all $i$, their gradients $\\left\\{\\nabla f_i(\\bx)\\right\\}_{i=1}^k$ are affinely independent, and their Hessians $\\nabla^2 f_i (\\bx)$ are positive definite. As a consequence, there exists constants $\\delta,L>0$ such that their Hessians satisfy\\begin{equation}\n    \\delta I \\preceq \\nabla^2 f_i(x) \\preceq L I \\ \\forall \\ i=1,\\dots,k, \\label{eq:hessian}\n\\end{equation} \nfor $x$ near $\\bx$---where $I$ is the $n{\\times}n$ identity matrix.\n\\end{defn}","sha256_utf8":"e2f87e6631a306f18b6d06b2bfc3ddf9dacdcc1e73a8649f739ef86b45227bce"}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","title":"全局结构与步长参数","source_units":[{"start":40476,"end":41318,"start_line":293,"end_line":295,"raw_tex":"Our analysis in this paper is entirely local, but we \\underline{assume for the rest of this paper} that $f$ is a strong $\\C^2$ max function with nondegenerate minimizer $\\bx$. For simplicity, we also assume that the properties in Definition \\ref{def:strong_max} hold \\textit{globally} for all $x\\in \\R^n$. In this setting, we refer to $k$---which is necessarily unique---as the \\textit{degree} of $f$. Moreover, note that \\eqref{eq:hessian} implies all components $\\left\\{f_i\\right\\}_{i=1}^k$ are $L$-smooth, both $f$ and its components are $\\delta$-strongly convex, and $\\bx$ is $f$'s \\textit{unique} minimizer. \n\nTo study Survey Descent (Definition \\ref{def:SD_Method}) on the objective $f$, we choose the above-presented $L$ as the step-control parameter and define the following notions of \\textit{valid} and \\textit{minimizing} surveys.\n","sha256_utf8":"deda749a251c5e5f3794b8e53d6a8ab9a0ee6785b67adba496bc57280f620838"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"有效调查与极小调查","source_units":[{"start":41319,"end":41989,"start_line":297,"end_line":299,"raw_tex":"\\begin{defn}\n\\textbf{\\label{def:validS} ($\\S, \\bX$-Surveys and Validity)} For the objective $f$, define a \\textit{survey} as a matrix $\\S \\equiv \\left[s_{1}{,}\\dots{,}s_{k}\\right] \\in \\mathbb{\\R}^{n\\times k}$ consisting of $k$ columns\\footnote{We can equivalently consider $\\S$ as a set $\\{s_i\\}_{i=1}^k$. We will use the matrix and set interpretations interchangeably.} (the ``survey points''), where $k$ is the degree of $f$; \\textit{valid surveys} as surveys satisfying $f_{i}\\left(s_{i}\\right){>}f_{j}\\left(s_{i}\\right)$ for all $i{\\neq}j$; and the \\textit{minimizing survey} as $\\bX \\equiv [\\bx{,}\\dots{,}\\bx]$ whose survey points are all equal to $\\bx$.\n\\end{defn}","sha256_utf8":"f8370b2d87ca4af39c7aa56687ca2b4a675d722a1d2ecf3f3aa1b17b6e9b4126"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"调查矩阵范数","source_units":[{"start":42032,"end":42213,"start_line":300,"end_line":302,"raw_tex":"\\begin{equation}\n\\left\\| \\S\\right\\| \\equiv\\left\\| \\left[s_{1},\\dots,s_{k}\\right]\\right\\| _{2,\\infty}=\\max_{i=1,\\dots,k}\\left\\| s_{i}\\right\\| _{2},\\label{eq:surv_norm}\n\\end{equation}","sha256_utf8":"4d227f4b257a5e2cb04e7f14428e7bf9ee2ac0dc336ab0e1a8c01edfb37d24b3"}]},{"id":"S013","semantic_type":"derived_fact","kind":"derived_fact","title":"唯一严格正临界权重","source_units":[{"start":42465,"end":42917,"start_line":305,"end_line":309,"raw_tex":"Given a valid $\\S$ sufficiently close to $\\bX$, Survey Descent exhibits many desirable properties when minimizing $f$. As a preliminary, note that \\eqref{eq:nondegen} and Definition \\ref{def:strong_max} implies there exists unique \\textit{critical weights} $\\left\\{ \\blam_{i}\\right\\}_{i=1}^{k}{>}0$ such that \n\\begin{align}\n\\sum_{i=1}^{k}\\blam_{i} & =1,\\label{eq:lam_sum1}\\\\\n\\sum_{i=1}^{k}\\blam_{i}\\nabla f_{i}(\\bx) & =0.\\label{eq:lam_sum0}\n\\end{align}","sha256_utf8":"35cbb785618786f59686dfab6c2532434e3581c547d5a3c579f92d287de835de"}]},{"id":"S014","semantic_type":"notation","kind":"notation","title":"大O余项记号","source_units":[{"start":42918,"end":43781,"start_line":310,"end_line":312,"raw_tex":"As we will see, the Lagrange multipliers of Survey Descent subproblems will lie ``close'' to $\\left\\{ \\blam_{i}\\right\\}_{i=1}^{k}{>}0$, while the subproblem outputs $\\left\\{s_i^+\\right\\}_{i=1}^k$ will lie near  $\\bx$. To characterize the order-of-magnitude of these distances, we adopt the ``Big-Oh'' notation. In particular, for a mapping $g:\\mathbf{E}{\\to}\\F$ between two Euclidean spaces and letting $\\left\\| \\cdot\\right\\|^p$ denote an arbitrary Euclidean norm raised to the $p$-th power, we use the notation \n$g\\left(x\\right)=O\\left(\\left\\| x\\right\\| ^{p}\\right)$\nto indicate the property that there exists a constant $K{>}0$ such that $ \\left\\| g\\left(x\\right)\\right\\|{\\leq}K\\left\\| x\\right\\| ^{p}$ holds for all small $x$. By itself, we let $\\mathrm{O}\\left(\\left\\| x\\right\\| ^{p}\\right)$ denote an element of the class of all functions with this property.\n","sha256_utf8":"6606dc121599ede97ddd4da0f1eeb5c2a974d150cd7962d414b0bece5651ae56"}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","title":"调查下降局部分析设定","source_units":[{"start":43878,"end":44226,"start_line":316,"end_line":319,"raw_tex":"\\begin{setting} \\label{setting:localSD} \\textbf{(Local Analysis of Survey Descent)}\nConsider an iteration of Survey Descent on a strong $\\C^2$ max function objective, $f$, with a survey \n$\\S$ that is valid and sufficiently close to $\\bX$. For $i{=}1{,}\\dots{,}k$, denote the output of the $i$-th Survey Descent subproblem by $s_i^+$.\n\\end{setting}\n","sha256_utf8":"353b4a3067deab5509bd1e4c66f72621ce42306c4d0603d2c2dbb97c0c55c53f"}]},{"id":"S016","semantic_type":"derived_fact","kind":"derived_fact","title":"有效点的活动分量梯度","source_units":[{"start":45896,"end":46025,"start_line":330,"end_line":332,"raw_tex":"\\begin{equation}\\label{eq:valS}\n\\nabla f\\left(s_{i}\\right)=\\nabla f_{i}\\left(s_{i}\\right) \\ \\forall \\ i=1,\\dots,k,\n\\end{equation}","sha256_utf8":"46406626f54677188e6b60cc6b7ab2e52483cca8be0fc4f0b1f9e9108de7d91d"}]},{"id":"S017","semantic_type":"algorithm","kind":"algorithm","title":"最大值目标的等价子问题","source_units":[{"start":46068,"end":46689,"start_line":334,"end_line":340,"raw_tex":"\\begin{observe}\n\\textbf{(Survey Descent Iterations on Max-of-Smooth Objectives)\\label{obs:SD_MOS}} Assume that $\\S$ is a valid survey for the objective $f$. Then, for all $i$, the $i$-th subproblem $\\left(P_{i}^{\\S}\\right)$ of Survey Descent (Definition \\ref{def:SD_Method}) is equivalent to \\begin{equation}\\begin{aligned}\\min_{x}\\ & \\left\\| x-\\left(s_{i}-\\frac{1}{L}\\nabla f_{i}\\left(s_{i}\\right)\\right)\\right\\| _{2}^{2}\\\\\n\\mathrm{s.t.}\\ & \\ell_{s_j}^{f_j}(x)+\\frac{L}{2}\\left\\| x-s_{j}\\right\\| _{2}^{2}\\leq \\ell_{s_i}^{f_i}(x)\\ \\forall\\ j\\neq i.\n\\end{aligned}\n\\label{eq:raw_form_analytic}\n\\end{equation}\n\\end{observe}\n","sha256_utf8":"189c4d60598030ab4b3adf58b00e3b48dbe61b1acc29be4536b2087fc956ba4f"}]},{"id":"S018","semantic_type":"definition","kind":"definition","title":"光滑方向仿射子空间","source_units":[{"start":48888,"end":49276,"start_line":371,"end_line":375,"raw_tex":"\\begin{align}\n\\U & \\equiv \\bx + \\Span\\left\\{ \\nabla f_{i}(\\bx)-\\nabla f_{j}(\\bx):1\\leq i,j\\leq k\\right\\} ^{\\perp} \\label{eq:Udef1}\\\\\n & = \\bx +  \\left\\{ \\sum_{i}^k\\gamma_{i}\\nabla f_{i}(\\bx):\\sum_{i}^k\\gamma_{i}=0\\right\\} ^{\\perp} \\label{eq:Udef2}\\\\\n & = \\bx +  \\left\\{x\\in \\R^n: \\ \\nabla f_i(\\bx)^T x = \\nabla f_j(\\bx)^T x \\ \\forall \\ 1\\leq i,j\\leq k\\right\\}.\\label{eq:Udef3}\n\\end{align}","sha256_utf8":"150cfbc78bd9d98aa8abb7377946b0ec183725cb1e009a55f9cdfde30295bae5"}]},{"id":"S019","semantic_type":"notation","kind":"notation","title":"梯度Hessian与Taylor余项","source_units":[{"start":49984,"end":50136,"start_line":378,"end_line":380,"raw_tex":"\\begin{equation}\n    a_i \\equiv \\nabla f_i(\\bx) \\text{ and } A_i \\equiv \\nabla^2 f_i(\\bx) \\ \\forall \\  i=1,\\dots,k. \\label{eq:A_notation}\n\\end{equation}","sha256_utf8":"ee5ba6bad11a0a5ce9e383447c9497cdd26dd2560574337981f6430acb947ed2"},{"start":50302,"end":50461,"start_line":381,"end_line":383,"raw_tex":"\\begin{equation}\n    f_{i}\\left(x\\right)= f(0) + a_{i}^{T}x+\\frac{1}{2}x^{T}A_{i}x+r_{i}\\left(x\\right) \\ \\forall \\ i=1,\\dots,k,\\label{eq:taylor}\n\\end{equation}","sha256_utf8":"f486d1e713e52bff8aabeb9a0072f9f5b01e0fb94e8fb7000f642a2890b2acc9"},{"start":50462,"end":50501,"start_line":384,"end_line":384,"raw_tex":"where $r_i(x)$ is a residual function.\n","sha256_utf8":"4308aeb7c8b5676ed23904370f221855ea260ecdfb646a21b43390332be0650f"}]},{"id":"S020","semantic_type":"definition","kind":"definition","title":"最小临界权重","source_units":[{"start":62729,"end":62837,"start_line":535,"end_line":537,"raw_tex":"\\begin{equation}\n    \\blam_{\\min} \\equiv \\min_{i=1,\\dots,k} \\blam_{i} > 0. \\label{eq:min_lam}\n\\end{equation}","sha256_utf8":"46268577299e542f614861b685a6fffe3e8887e0bfbfadf78e9b7f276e760947"}]},{"id":"S021","semantic_type":"algorithm","kind":"algorithm","title":"重复调查下降过程","source_units":[{"start":66400,"end":66906,"start_line":575,"end_line":582,"raw_tex":"\\begin{procedure}\\textbf{(Repeated Iterations of Survey Descent) }\\label{procedure:SD}\nFor the strong $\\C^2$ max function objective $f$, assume $\\S^{0}$ is a valid initial survey sufficiently close to $\\bX$, and initialize $t=0$. Iterate the following steps:\n\\begin{enumerate}\n\\item Solve the Survey Descent iteration $\\left\\{ \\left(P_{i}^{\\S^{t}}\\right)\\right\\} _{i=1}^{k}$\nand denote the output $\\S^{t+1}\\equiv\\left(\\S^{t}\\right)^{+}$.\n\\item Increment $t\\leftarrow t+1$. \n\\end{enumerate}\n\\end{procedure}\n","sha256_utf8":"d123909f73be8cd5e1e33c3b7f4e1c29bfd829f27e60046b87036c5324af8e5d"}]},{"id":"S022","semantic_type":"derived_fact","kind":"derived_fact","title":"所有迭代保持局部条件","source_units":[{"start":68016,"end":68348,"start_line":591,"end_line":593,"raw_tex":"\\begin{observe}\\label{obs:allT}\nThe proof of Corollary \\ref{cor:viable} more generally shows that, for all $t \\in \\nonneg$, all surveys $\\S^{t}$ will remain sufficiently close to $\\bX$. Thus, the conclusions from Theorems \\ref{thm:SD_MBP}-\\ref{thm:s_monotone} apply to every iteration of Procedure \\ref{procedure:SD}.\n\\end{observe}\n","sha256_utf8":"ab8921e6e5220fc0b33515f1b29fe95c28d50c1e7c32ab2ca820dea7bf555c85"}]},{"id":"S023","semantic_type":"definition","kind":"definition","title":"多球边界投影问题","source_units":[{"start":95766,"end":96571,"start_line":881,"end_line":891,"raw_tex":"\\begin{defn}\n    \\textbf{(Multiple Balls Problem, $\\MBP\\param$)\\label{def:MBP}} \\sloppy For some\n    target vector $y\\in\\R^{n}$, ball centers $Z=\\left[z_{1},\\dots,z_{\\bk}\\right]\\in\\R^{n\\times\\bk}$\n    where $\\bk\\leq n$, and non-negative radii $r=\\left[r_{1},\\dots,r_{\\bk}\\right]^{T}\\in\\R^{\\bk}_{+}$,\n    define the \\textit{Multiple Balls Problem (MBP)} as \n    \\begin{align}\n    \\min_{x\\in\\R^{n}}\\ & \\left\\| x-y\\right\\| _{2}^{2} \\nonumber\\\\\n    \\mathrm{s.t.}\\ & \\left\\| x-z_{j}\\right\\| _{2}^{2} = r_{j}^{2}\\ \\forall\\ j=1,\\dots,\\bk. \\label{eq:MBP_cons}\n    \\end{align}\n    Denote the problem as $\\MBP\\param$ and its \\textit{feasible region} as $\\B\\spec$. We will refer to the tuple $\\param$ as the \\textit{parameters} and the sub-tuple $\\spec$ as the \\textit{ball-specifications} of the MBP.\n    \\end{defn}","sha256_utf8":"d14640e18076985a7371fc03acf58fbbc0b0a63a515eccc9a5cbbf8a7a6c3b87"}]},{"id":"S024","semantic_type":"notation","kind":"notation","title":"多球参数空间与距离","source_units":[{"start":96847,"end":97300,"start_line":895,"end_line":895,"raw_tex":"    Since $y$, $Z$, and $r$ all reside in Euclidean spaces, we can also consider $\\spec$ and $\\param$ elements of some product space. Let $\\left\\Vert \\cdot\\right\\Vert $ denote an arbitrary Euclidean norm. This will allow us to formalize the notion of distance in of parameters and ball-specifications in their respective spaces. We are particularly interested in the behavior of $\\MBP\\param$ where $\\param$ is near some set of \\textit{base parameters}.\n","sha256_utf8":"967f4efae6abcda2c6215c5ae023bd5f34062dcfe7d2ea5708d47c3408d7d3f8"}]},{"id":"S025","semantic_type":"definition","kind":"definition","title":"多球问题基准参数","source_units":[{"start":97309,"end":97896,"start_line":897,"end_line":908,"raw_tex":"\\begin{defn}\n    \\textbf{(Base Parameters)\\label{def:base}} We refer to a set of parameters\n    as \\textit{base parameters} and denote them as $\\bparam$\n    if they satisfy the following properties:\n    \\begin{itemize}\n    \\item The columns of $\\bZ=\\left[\\bz_{1},\\dots,\\bz_{\\bk}\\right]$ are linearly\n    independent.\n    \\item $\\br_{j}^{2}=\\left\\| \\bz_{j}\\right\\| _{2}^{2}$.\n    \\item There exist strictly-positive weights $\\clam=\\left[\\clam_{1},\\dots,\\clam_{\\bk}\\right]^{T}\\in\\R_{++}^{\\bk}$\n    such that $\\bar{y}=-\\sum_{j=1}^{\\bk}\\clam_{j}\\bar{z}_{j}.$\n    \\end{itemize}\n    \\end{defn}","sha256_utf8":"e33ea561f0eeda0430468d673c70890f1faa3219e998c4e751f705928cf9056d"}]},{"id":"S026","semantic_type":"definition","kind":"definition","title":"中心仿射空间核心与余量","source_units":[{"start":98933,"end":99732,"start_line":916,"end_line":930,"raw_tex":"    \\begin{itemize}\n        \\item The \\textit{central affine subspace}\n            \\begin{equation}\n            \\A\\spec \\equiv \\left\\{ x\\in\\R^{n}:\\ 2z_{j}^{T}x-\\left\\| z_{j}\\right\\| _{2}^{2}+r_{j}^{2}\\text{ equal  for all } j=1,\\dots,\\bk\\right\\}. \\label{eq:CAS}\n            \\end{equation}\n        \\item The \\textit{core}\n            \\begin{equation}\n            c\\spec\\equiv\\projA{\\spec}\\left(z_{j}\\right)\\ \\text{ for any }\\ j=1,\\dots,\\bk, \\label{eq:core}\n            \\end{equation}\n            where $\\projA{\\spec}$ denotes the projection onto $\\A\\spec$.\n        \\item The \\textit{margin}\n            \\begin{equation}\n            \\rho\\spec\\equiv\\sqrt{r_{j}^{2}-\\left\\| z_{j}-c\\spec\\right\\| _{2}^{2}} \\ \\text{ for any }\\ j=1,\\dots,\\bk. \\label{eq:margin}\n            \\end{equation}\n    \\end{itemize}\n","sha256_utf8":"645bf46f7500005c739ca022e8e933e81d7e95b490c7df19a7702feaa6f95dd8"}]},{"id":"S027","semantic_type":"notation","kind":"notation","title":"多球问题乘子驻点方程","source_units":[{"start":113031,"end":113889,"start_line":1102,"end_line":1105,"raw_tex":"    In this section, we will show that the optimal solutions of MBPs are unchanged when the equality constraints \\eqref{eq:MBP_cons} are relaxed to inequalities. In particular, regardless of whether $\\MBP\\param$ is defined with equality or inequality constraints, note that the stationarity conditions remain the same. Therefore, there must exist some Lagrange multipliers, $\\lambda\\param\\in\\R^m$, such that\\begin{equation}\n    x\\param-y+\\sum_{j=1}^{\\bk}\\lambda_{j}\\param\\left[x\\param-z_{j}\\right]=0, \\label{eq:MBP_FOC}\n    \\end{equation}\n    where $\\lambda_{j}\\param$ denotes the $j$-th element of $\\lambda\\param$. The only difference is that the inequality version possesses additional dual feasibility conditions requiring the non-negativity of $\\lambda\\param$. Lemma \\ref{lem:MBP_lam} will show that the $\\lambda\\param$ are unique and strictly positive.\n","sha256_utf8":"ce9ee413aa325f128c062efae374deebca0f97bfebff7e6f4b4aea290b40af19"}]},{"id":"S028","semantic_type":"algorithm","kind":"algorithm","title":"调查子问题的多球重写","source_units":[{"start":119620,"end":119824,"start_line":1167,"end_line":1168,"raw_tex":"    We can reformulate Survey Descent subproblems into inequality-constrained MBPs. In particular, for all $i$, we can rewrite the $i$-th Survey Descent subproblem in Observation \\ref{obs:SD_MOS} as\n    \n","sha256_utf8":"a6d772275b75cda52042f009be1e05ae74c7459759e32b92ceee3ec4cdaec9a4"},{"start":119828,"end":120118,"start_line":1169,"end_line":1173,"raw_tex":"\\begin{gather}\\begin{aligned}\\min_{x}\\ & \\left\\| (x - s_{i}) - y_{i}\\left(\\S\\right) \\right\\| _{2}^{2}\\\\\n    \\mathrm{s.t.}\\ & \\left\\| (x-s_{i})-z_{ij}\\left(\\S\\right)\\right\\| _{2}^{2}\\leq r_{ij}^{2}\\left(\\S\\right)\\ \\forall\\ j\\neq i,\n    \\end{aligned}\n    \\label{eq:proj_form}\n    \\end{gather}","sha256_utf8":"aa7d5c21a4d4d1da20d9410a1a6e5ee3e86c8aa1341ee40e857309dc1f931aa1"},{"start":120119,"end":120266,"start_line":1174,"end_line":1175,"raw_tex":"    where, for all $i$ and $j$, we define the following mappings using the components $\\{f_i\\}_{i=1}^k$ from Definition \\ref{def:strong_max}:\n    \n","sha256_utf8":"19fcc73533831e9abfc44138e8e44eb6a9919f16503fe2e1a93522d8aecd8003"},{"start":120270,"end":120764,"start_line":1176,"end_line":1181,"raw_tex":"\\begin{gather}\\begin{aligned}\n    y_{i}\\left(\\S\\right)& \\equiv-\\frac{1}{L}\\nabla f_{i}\\left(s_{i}\\right),\\\\\n    z_{ij}\\left(\\S\\right) & \\equiv s_j - s_i  + y_{j}\\left(\\S\\right)-y_{i}\\left(\\S\\right),\\\\\n    r_{ij}^{2}\\left(\\S\\right) & \\equiv\\frac{2}{L}\\left[f_{i}\\left(s_{i}\\right)-f_{j}\\left(s_{j}\\right)+\\nabla f_{i}\\left(s_{i}\\right)^{T}\\left(s_{j}-s_{i}\\right)\\right]+\\left\\|y_{j}\\left(\\S\\right)-y_{i}\\left(\\S\\right)\\right\\| _{2}^{2}.\n    \\end{aligned}\\label{eq:MBP_mappings}\n    \\end{gather}","sha256_utf8":"02adef7ab486fcf7e277f6e332e86cd6d1c166cf58500a5466507000646615e6"}]},{"id":"S029","semantic_type":"derived_fact","kind":"derived_fact","title":"调查参数对应多球基准","source_units":[{"start":120765,"end":122303,"start_line":1182,"end_line":1191,"raw_tex":"    \\begin{observe}\\textbf{(Survey Descent as Inequality-Constrained MBP)} \\label{obs:SD_as_MBP} Consider any $k$-point survey $\\S$ and any $i{=}1,\\dots,k$. After the changes-of-variables $x \\leftarrow x{-} s_i$, the minimization problem in \\eqref{eq:proj_form} is an inequality-constrained MBP  with $\\m = k{-}1$ and parameters \n    \n    $$\\param = \\left(y_{i}\\left(\\S\\right), \\left[ z_{ij}\\left(\\S\\right)\\right]_{j\\neq i}, \\left[r_{ij}\\left(\\S\\right)\\right]_{j\\neq i},\\left[ s_{j}\\right]_{j\\neq i}\\right),$$\n    where, for any $j$-indexed scalars $\\left\\{v_j\\right\\}_{j=1}^{k}$, we use the notation $\\left[v_j\\right]_{j \\neq i}{\\equiv} \\left[v_1,\\dots,v_{i-1},v_{i+1},\\dots,v_{k}\\right]^T$ to denote the length $k{-}1$ vector excluding $v_i$.\n    Furthermore, the mappings \\eqref{eq:MBP_mappings}  evaluated at the base survey\\footnote{Evaluating of the mappings \\eqref{eq:MBP_mappings} at $\\bX$ is a technical step. $\\bX$ is not a valid survey (Definition \\ref{def:validS}), so it does not correspond to an actual Survey Descent iteration.} $\\bX$ (Definition \\ref{def:validS}) form a set of base parameters\n    \\begin{equation}\n        \\bparam = \\left(y_{i}\\left(\\bX\\right), \\left[ z_{ij}\\left(\\bX\\right)\\right]_{j\\neq i}, \\left[r_{ij}\\left(\\bX\\right)\\right]_{j\\neq i}, [\\bx]_{j\\neq i}\\right),\\label{eq:MBP_SD_base}\n    \\end{equation}\n    with corresponding weights $\\{\\blam_j\\}_{j\\neq i}$ from \\eqref{eq:lam_sum1}-\\eqref{eq:lam_sum0}, and where $[\\bx]_{j\\neq i}$ is the $k{-}1$ length vector with all $\\bx$ entries.\n    \\end{observe}\n","sha256_utf8":"601f87fd665460571a5e1d43b83ff7aebcbce330faeeeb168ff8b653f2e11986"}]},{"id":"S030","semantic_type":"theorem","kind":"theorem","title":"局部可行唯一紧约束性质","source_units":[{"start":44227,"end":45079,"start_line":321,"end_line":324,"raw_tex":"\\begin{restatable}{thm}{PBmbp}\\label{thm:SD_MBP} \\textbf{(Local Feasibility, Uniqueness, Tightness, and Smoothness of Survey Descent)} Assume Setting \\ref{setting:localSD}. Then for all $i$, the Survey Descent subproblem $\\left(P_{i}^{\\S}\\right)$ is feasible and has a unique solution $s_{i}^{+}$, which satisfies all the constraints with equality, with unique associated Lagrange multipliers $\\lambda_{j}^i$ \\edit{for the $j$-th constraint within the $i$-th subproblem}  (for $j\\neq i$). The solutions and multipliers depend smoothly on the input survey $\\S$, and satisfy\\begin{equation}\ns_{i}^{+}\\left(\\S\\right) ~=~\\bx+\\mathrm{O}\\left(\\left\\| \\S-\\bX\\right\\| \\right) \\qquad \\text{and} \\qquad \\lambda_{j}^i\\left(\\S\\right) ~=~ \\blam_{j}+\\mathrm{O}\\left(\\left\\| \\S-\\bX\\right\\| \\right)\\ \\forall\\ j\\neq i.\\label{eq:SD_forms}\n\\end{equation}\n\\end{restatable}","sha256_utf8":"dbded2cd688b822b30d1130c855753c6f635a45aee3a2264368f3089f21368f6"}]},{"id":"S031","semantic_type":"proof","kind":"proof","title":"子问题局部性质完整证明","source_units":[{"start":45080,"end":45236,"start_line":325,"end_line":327,"raw_tex":"\\begin{proof}\nFollows from a routine analysis of the subproblem's first-order optimality conditions combined with the implicit function theorem.\n\\end{proof}","sha256_utf8":"64afb1672f0e40317fdb92623612b091e65b53f6c4fb0c96c714eb7ae69ece28"},{"start":122359,"end":123022,"start_line":1194,"end_line":1199,"raw_tex":"\\begin{proof}\n    It is easy to check from the mapping definitions \\eqref{eq:MBP_mappings} and the Taylor expansion \\eqref{eq:taylor} that \n    \n    $$\\left\\|\\left(y_{i}\\left(\\S\\right), \\left[ z_{ij}\\left(\\S\\right)\\right]_{j\\neq i}, \\left[r_{ij}\\left(\\S\\right)\\right]_{j\\neq i},\\left[ s_{j}\\right]_{j\\neq i}\\right) - \\left(y_{i}\\left(\\bX\\right), \\left[ z_{ij}\\left(\\bX\\right)\\right]_{j\\neq i}, \\left[r_{ij}\\left(\\bX\\right)\\right]_{j\\neq i}, [\\bx]_{j\\neq i}\\right)\\right\\| = \\mathrm{O}\\left(\\|\\S-\\bX\\|\\right).$$\n    The result then follows directly from Observation \\ref{obs:SD_MOS}, Observation \\ref{obs:SD_as_MBP}, and Theorem \\ref{thm:MBP_ineq}.\n    \\end{proof}","sha256_utf8":"ccfbe5da2a8493437b9aae62f72d6da069b93db249d6faaa4818bf70153a474d"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"最优性条件与隐函数论证","source_units":[{"start":45094,"end":45225,"start_line":326,"end_line":326,"raw_tex":"Follows from a routine analysis of the subproblem's first-order optimality conditions combined with the implicit function theorem.\n","sha256_utf8":"1a9543a4ab08abcbb18bed3dd805fb498871c518c02f22d269954b3a318a7391"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"多球参数的一阶扰动界","source_units":[{"start":122373,"end":122870,"start_line":1195,"end_line":1197,"raw_tex":"    It is easy to check from the mapping definitions \\eqref{eq:MBP_mappings} and the Taylor expansion \\eqref{eq:taylor} that \n    \n    $$\\left\\|\\left(y_{i}\\left(\\S\\right), \\left[ z_{ij}\\left(\\S\\right)\\right]_{j\\neq i}, \\left[r_{ij}\\left(\\S\\right)\\right]_{j\\neq i},\\left[ s_{j}\\right]_{j\\neq i}\\right) - \\left(y_{i}\\left(\\bX\\right), \\left[ z_{ij}\\left(\\bX\\right)\\right]_{j\\neq i}, \\left[r_{ij}\\left(\\bX\\right)\\right]_{j\\neq i}, [\\bx]_{j\\neq i}\\right)\\right\\| = \\mathrm{O}\\left(\\|\\S-\\bX\\|\\right).$$\n","sha256_utf8":"54f86210be07362661b782967160ab38c85235060cabd6b8e7761ba1a04d670e"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"应用不等式多球定理","source_units":[{"start":122870,"end":123007,"start_line":1198,"end_line":1198,"raw_tex":"    The result then follows directly from Observation \\ref{obs:SD_MOS}, Observation \\ref{obs:SD_as_MBP}, and Theorem \\ref{thm:MBP_ineq}.\n","sha256_utf8":"2cfedceaf8de6e05fcf96d44055b8fea5af12255c2ea5a594a281473c16441d9"}]},{"id":"S035","semantic_type":"theorem","kind":"theorem","title":"调查有效性保持","source_units":[{"start":46805,"end":46991,"start_line":344,"end_line":345,"raw_tex":"\\begin{thm} \\label{thm:strictval} \\textbf{(Preservation of Validity)} Assuming Setting \\ref{setting:localSD}, the output survey $\\S^{+}$ of a Survey Descent iteration is valid.\n\\end{thm}","sha256_utf8":"2289d998374779a1b937a646773208e770475785cb1976cb039ce897ee3fdfee"}]},{"id":"S036","semantic_type":"proof","kind":"proof","title":"有效性保持完整证明","source_units":[{"start":46992,"end":48521,"start_line":346,"end_line":367,"raw_tex":"\\begin{proof}\nConsider any fixed $i$. If $s_{i}^{+}=s_{i}$, then the fact\nthat $f_{i}\\left(s_{i}^{+}\\right){>}f_{j}\\left(s_{i}^{+}\\right)$ for all $j\\neq i$ immediately follows since Theorem \\ref{thm:SD_MBP} assumes $\\S$ is valid.\n\nNow, consider the $s_{i}^{+}{\\neq}s_{i}$ case. By Theorem \\ref{thm:SD_MBP}, $s_{i}^{+}$ satisfies the constraints of $\\left(P_{i}^{\\S}\\right)$ with equality. Using Observation \\ref{obs:SD_MOS}, we express this as\\begin{equation}\n\\ell_{s_i}^{f_i}(s_i^+) = \\ell_{s_j}^{f_j}(s_i^+)+\\frac{L}{2}\\left\\| s_{i}^{+}-s_{j}\\right\\|_{2}^{2}\\ \\forall\\ j\\neq i.\\label{eq:left_valid}\n\\end{equation}\nThe $\\delta$-strong convexity of $f_{i}$ implies\n\\begin{equation*}\nf_{i}\\left(s_{i}^{+}\\right)\\geq\\ell_{s_i}^{f_i}(s_i^+)+\\frac{\\delta}{2}\\left\\| s_{i}^{+}-s_{i}\\right\\| _{2}^{2}>\\ell_{s_i}^{f_i}(s_i^+),\n\\end{equation*}\nwhere the strict inequality follows from $s_{i}^{+}{\\neq}s_{i}$. Next, by the $L$-smoothness of $\\left\\{f_j\\right\\}_{j=1}^k$, the right-hand side of \\eqref{eq:left_valid} is a $s_j$-centered quadratic upper bound of $f_{j}$ evaluated at $s_i^+$. Thus,\n\\begin{equation*}\n\\ell_{s_j}^{f_j}(s_i^+)+\\frac{L}{2}\\left\\| s_{i}^{+}-s_{j}\\right\\|_{2}^{2} \\geq f_{j}\\left(s_{i}^{+}\\right) \\ \\forall\\ j\\neq i.\n\\end{equation*}\n\n\\noindent Therefore, \n\\begin{equation*}\nf_{i}\\left(s_{i}^{+}\\right) > \\ell_{s_i}^{f_i}(s_i^+) = \\ell_{s_j}^{f_j}(s_i^+)+\\frac{L}{2}\\left\\| s_{i}^{+}-s_{j}\\right\\|_{2}^{2} \\geq f_{j}\\left(s_{i}^{+}\\right) \\ \\forall\\ j\\neq i.\n\\end{equation*}\nThis completes the proof.\n\\end{proof}","sha256_utf8":"d81d688c6d3dd2df89c6ea910f6e3f6f452c2e947224b46f6274ffeea215d589"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"未移动点保持有效","source_units":[{"start":47006,"end":47223,"start_line":347,"end_line":348,"raw_tex":"Consider any fixed $i$. If $s_{i}^{+}=s_{i}$, then the fact\nthat $f_{i}\\left(s_{i}^{+}\\right){>}f_{j}\\left(s_{i}^{+}\\right)$ for all $j\\neq i$ immediately follows since Theorem \\ref{thm:SD_MBP} assumes $\\S$ is valid.\n","sha256_utf8":"e8eeea9e15c7e184155897c38f57b3c01400fa381777f3d89e132b4cdfae578a"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"移动点满足紧约束","source_units":[{"start":47224,"end":47609,"start_line":350,"end_line":352,"raw_tex":"Now, consider the $s_{i}^{+}{\\neq}s_{i}$ case. By Theorem \\ref{thm:SD_MBP}, $s_{i}^{+}$ satisfies the constraints of $\\left(P_{i}^{\\S}\\right)$ with equality. Using Observation \\ref{obs:SD_MOS}, we express this as\\begin{equation}\n\\ell_{s_i}^{f_i}(s_i^+) = \\ell_{s_j}^{f_j}(s_i^+)+\\frac{L}{2}\\left\\| s_{i}^{+}-s_{j}\\right\\|_{2}^{2}\\ \\forall\\ j\\neq i.\\label{eq:left_valid}\n\\end{equation}\n","sha256_utf8":"0245c15c87bc4aa94faff8a0d49a5116072a6eb42af0f9816521638d981a608e"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"强凸下界与光滑上界","source_units":[{"start":47609,"end":48243,"start_line":353,"end_line":360,"raw_tex":"The $\\delta$-strong convexity of $f_{i}$ implies\n\\begin{equation*}\nf_{i}\\left(s_{i}^{+}\\right)\\geq\\ell_{s_i}^{f_i}(s_i^+)+\\frac{\\delta}{2}\\left\\| s_{i}^{+}-s_{i}\\right\\| _{2}^{2}>\\ell_{s_i}^{f_i}(s_i^+),\n\\end{equation*}\nwhere the strict inequality follows from $s_{i}^{+}{\\neq}s_{i}$. Next, by the $L$-smoothness of $\\left\\{f_j\\right\\}_{j=1}^k$, the right-hand side of \\eqref{eq:left_valid} is a $s_j$-centered quadratic upper bound of $f_{j}$ evaluated at $s_i^+$. Thus,\n\\begin{equation*}\n\\ell_{s_j}^{f_j}(s_i^+)+\\frac{L}{2}\\left\\| s_{i}^{+}-s_{j}\\right\\|_{2}^{2} \\geq f_{j}\\left(s_{i}^{+}\\right) \\ \\forall\\ j\\neq i.\n\\end{equation*}\n","sha256_utf8":"a39ba6adc7a2a522dbe4c45ccb2573a7d732813efaa8ecdadda3a2199f9a28c4"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"合并得到严格活动性","source_units":[{"start":48244,"end":48510,"start_line":362,"end_line":366,"raw_tex":"\\noindent Therefore, \n\\begin{equation*}\nf_{i}\\left(s_{i}^{+}\\right) > \\ell_{s_i}^{f_i}(s_i^+) = \\ell_{s_j}^{f_j}(s_i^+)+\\frac{L}{2}\\left\\| s_{i}^{+}-s_{j}\\right\\|_{2}^{2} \\geq f_{j}\\left(s_{i}^{+}\\right) \\ \\forall\\ j\\neq i.\n\\end{equation*}\nThis completes the proof.\n","sha256_utf8":"dd9ecaddcf59cd624a9bb3b1d9c2570a8c6076796130b978d5b71dbc9198433d"}]},{"id":"S041","semantic_type":"lemma","kind":"lemma","title":"Taylor余项与梯度阶数","source_units":[{"start":50502,"end":50732,"start_line":386,"end_line":388,"raw_tex":"\\begin{lem}\n\\label{lem:r_grad} For all $i$, the residual function $r_i(x)$ in \\eqref{eq:taylor} satisfies $r_i(x)=\\mathrm{O}\\left(\\|x\\|^2\\right)$ and $\\nabla r_{i}\\left(x\\right)=\\mathrm{O}\\left(\\left\\| x\\right\\|\\right)$.\n\\end{lem}","sha256_utf8":"21a0a351120583563935dba51a1865ed20d0d87cc8857d595d0fa0d7602f2574"}]},{"id":"S042","semantic_type":"proof","kind":"proof","title":"余项阶数完整证明","source_units":[{"start":50733,"end":51464,"start_line":389,"end_line":396,"raw_tex":"\\begin{proof}\n$r_i(x)=\\mathrm{O}\\left(\\|x\\|^2\\right)$ follows directly from Taylor's Theorem. Next, observe that \n\\begin{equation*}\n    r_{i}\\left(x\\right)=f_{i}\\left(x\\right)-\\left(f(0) + a_{i}^{T}x+\\frac{1}{2}x^{T}A_{i}x\\right).\n\\end{equation*}\nThus, since $f_i$ is $\\C^2$, the residual $r_i$ must also be $\\C^2$, so its gradient and Hessian are well-defined. This implies $\\nabla r_{i}\\left(x\\right)$ is then $\\C^{1}$. Taylor expanding $\\nabla r_i$ around zero then gives the desired $\\nabla r_{i}\\left(x\\right)= \\mathrm{O}\\left(\\| x\\|\\right)$\nafter observing that $\\nabla r_i(0)$ is the zero-vector and $\\nabla^2 r_i(0)$ is the zero-matrix (since $r_i$ is the residual of the Taylor expansion of $f_i$ around zero).\n\\end{proof}","sha256_utf8":"f448e3387690ea899fd735775acb1047e7599e368b3d2d1772761c6a87879e2c"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"函数余项的二阶界","source_units":[{"start":50747,"end":50980,"start_line":390,"end_line":393,"raw_tex":"$r_i(x)=\\mathrm{O}\\left(\\|x\\|^2\\right)$ follows directly from Taylor's Theorem. Next, observe that \n\\begin{equation*}\n    r_{i}\\left(x\\right)=f_{i}\\left(x\\right)-\\left(f(0) + a_{i}^{T}x+\\frac{1}{2}x^{T}A_{i}x\\right).\n\\end{equation*}\n","sha256_utf8":"5873c1d2c324fb568fb099b11acac0a375b40c1a44e0eb90ae16324e6d2a58c0"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"余项梯度的一阶界","source_units":[{"start":50980,"end":51453,"start_line":394,"end_line":395,"raw_tex":"Thus, since $f_i$ is $\\C^2$, the residual $r_i$ must also be $\\C^2$, so its gradient and Hessian are well-defined. This implies $\\nabla r_{i}\\left(x\\right)$ is then $\\C^{1}$. Taylor expanding $\\nabla r_i$ around zero then gives the desired $\\nabla r_{i}\\left(x\\right)= \\mathrm{O}\\left(\\| x\\|\\right)$\nafter observing that $\\nabla r_i(0)$ is the zero-vector and $\\nabla^2 r_i(0)$ is the zero-matrix (since $r_i$ is the residual of the Taylor expansion of $f_i$ around zero).\n","sha256_utf8":"d68fd71be1f8ecaaefebf96425855588d7205cf2b27a0394e64a2ffebe846b39"}]},{"id":"S045","semantic_type":"lemma","kind":"lemma","title":"更新的光滑方向分量","source_units":[{"start":51526,"end":52064,"start_line":400,"end_line":405,"raw_tex":"\\begin{lem}\\label{lem:U_component} Assume Setting \\ref{setting:localSD} and suppose $\\bx{=}0$. Then, $s_{i}^{+}$ satisfies\n\\begin{equation*}\n\\proj\\left[s_{i}^{+}\\right]=\\frac{1}{2-\\blam_{i}}\\proj\\left[s_{i}-\\frac{\\blam_{i}}{L}\\nabla f_{i}\\left(s_{i}\\right)\\right]+\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\proj\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right]+\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right) \\ \\forall \\ i=1,\\dots,k.\n\\end{equation*}\nwhere $\\proj$ is the projection operator onto the $\\U$-subspace \\eqref{eq:Udef1}.\n\\end{lem}","sha256_utf8":"c169c715a3c5400fbb9effeb89a0d7c0ccecb5e23a3214b3899e90a4e108a423"}]},{"id":"S046","semantic_type":"proof","kind":"proof","title":"光滑方向近似完整证明","source_units":[{"start":52065,"end":55890,"start_line":406,"end_line":449,"raw_tex":"\\begin{proof}\nWithout loss of generality, assume $i=k$. Scale the objective of problem \\eqref{eq:raw_form_analytic} by $\\frac{L}{2}$ and consider the equivalent problem\n\\begin{equation}\\begin{aligned}\\min_{x}\\ & \\frac{L}{2} \\left\\| x-\\left(s_{k}-\\frac{1}{L}\\nabla f_{k}\\left(s_{k}\\right)\\right)\\right\\| _{2}^{2}\\\\\n\\mathrm{s.t.}\\ & \\ell_{s_j}^{f_j}(x)+\\frac{L}{2}\\left\\| x-s_{j}\\right\\| _{2}^{2}\\leq \\ell_{s_k}^{f_k}(x)\\ \\forall\\ j\\neq k.\n\\end{aligned}\\label{eq:scaledPk}\n\\end{equation}\nBy Theorem \\ref{thm:SD_MBP} and Observation \\ref{obs:SD_MOS}, there exists unique Lagrange multipliers $\\left\\{\\lambda_j\\right\\}_{j=1}^{k-1}$ (we omit the $k$-superscript for simplicity) satisfying the following first-order optimality condition for \\eqref{eq:scaledPk}:\n$$\n    0=L\\left(s_{k}^{+}-\\left(s_{k}-\\frac{1}{L} \\nabla f_{k}\\left(s_{k}\\right)\\right)\\right)+\\sum_{j=1}^{k-1} \\lambda_{j}\\left(\\edit{\\nabla f_{j}\\left(s_{j}\\right)}+L\\left(s_{k}^{+}-s_{j}\\right)-\\nabla f_{k}\\left(s_{k}\\right)\\right)\n    $$\nAfter rearranging this gives\\begin{equation}\n0= L\\left(1+\\sum_{j=1}^{k-1}\\lambda_{j}\\right)s_{k}^{+}-L\\left(s_{k} + \\sum_{j=1}^{k-1} \\lambda_{j} s_{j}\\right)+\\nabla f_k(s_k) +\\sum_{j=1}^{k-1}\\lambda_{j}\\left(\\nabla f_j(s_j) - \\nabla f_k(s_k)\\right).\n\\label{eq:FOC2}\n\\end{equation}\nDefine the following scalars:\n\\begin{align}\n    \\lambda_{k} & \\equiv  1-\\sum_{j=1}^{k-1}\\lambda_{j},\\label{eq:lambda}\\\\\n    \\mu_{j} & \\equiv  \\blam_{j}-\\lambda_{j} \\ \\forall \\ j=1,\\dots,k. \\label{eq:mu}\n\\end{align}\nFor $j\\neq k$, Theorem \\ref{thm:SD_MBP} gives\n$\\mu_{j}=\\mathrm{O}\\left(\\left\\| \\S\\right\\| \\right)$ .\nFor $j=k$, combining \\eqref{eq:lambda}-\\eqref{eq:mu} with equation \\eqref{eq:lam_sum1} implies\n\\begin{equation*}\n\\mu_{k} =\\blam_{k}-\\left(1-\\sum_{j=1}^{k-1}\\lambda_{j}\\right) =\\blam_{k}-\\left(1-\\sum_{j=1}^{k-1}\\blam_{j}\\right)+\\mathrm{O}\\left(\\left\\| \\S\\right\\| \\right) =\\mathrm{O}\\left(\\left\\| \\S\\right\\| \\right).\n\\end{equation*}\nThus,\\begin{equation}\n    \\mu_j = \\mathrm{O}(\\|\\S\\|) \\ \\forall \\ j=1,\\dots,k.\\label{eq:mu_size}\n\\end{equation}\nSubstituting \\eqref{eq:mu} into \\eqref{eq:FOC2}, applying identities \\eqref{eq:lam_sum1}-\\eqref{eq:lam_sum0}, collecting $\\mathrm{O}\\left(\\|\\S\\|^2\\right)$ terms, and solving for $s_k^{+}$ gives\\begin{equation}s_{k}^{+}= \\frac{1}{2-\\blam_{k}}\\left[s_{k}-\\frac{\\blam_{k}}{L}\\nabla f_{k}\\left(s_{k}\\right)\\right]+\\sum_{j=1}^{k-1}\\frac{\\blam_{j}}{2-\\blam_{k}}\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right]+\\frac{1}{L\\left(2-\\blam_{k}\\right)}\\sum_{j=1}^{k}\\mu_{j}\\nabla f_j(s_j) +\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right).\\label{eq:pre_sform}\n\\end{equation}\nAdopting the notation \\eqref{eq:A_notation} and applying the Taylor expansion \\eqref{eq:taylor}, observe\\begin{equation}\n    \\sum_{j=1}^{k}\\mu_{j}\\nabla f_j(s_j) = \\sum_{j=1}^{k}\\mu_{j}\\left(a_j + A_j s_j + \\nabla r_j (s_j)\\right) = \\sum_{j=1}^{k}\\mu_{j}a_j +\\mathrm{O}\\left(\\left\\| \\S\\right\\|^{2}\\right),\\label{eq:sum_muA}\n\\end{equation}\nwhere last equality follows from \\eqref{eq:mu_size} and Lemma \\ref{lem:r_grad}. Substituting \\eqref{eq:sum_muA} into \\eqref{eq:pre_sform} then leads to\\begin{equation}\ns_{k}^{+}=\\frac{1}{2-\\blam_{k}}\\left[s_{k}-\\frac{\\blam_{k}}{L}\\nabla f_{k}\\left(s_{k}\\right)\\right]+\\sum_{j=1}^{k-1}\\frac{\\blam_{j}}{2-\\blam_{k}}\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right]+\\frac{1}{L\\left(2-\\blam_{k}\\right)}\\underbrace{\\sum_{j=1}^{k}\\mu_{j}a_j}_{(\\star)} +\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right).\n\\label{eq:s_form}\n\\end{equation}\nNext, again combine \\eqref{eq:lambda}-\\eqref{eq:mu} with equation \\eqref{eq:lam_sum1} to deduce\n\\begin{equation*}\n\\sum_{j=1}^{k}\\mu_{j}=\\sum_{j=1}^{k}\\left(\\blam_{j}-\\lambda_{j}\\right)=0.\n\\end{equation*}\nThen, it follows from \\eqref{eq:Udef2} that $\\proj\\left[(\\star)\\right]=0$.\nThe desired result then follows from applying $\\proj$ to \\eqref{eq:s_form}.\n\\end{proof}","sha256_utf8":"b10b489203e777b061c41197d95326165e32a31670b55f2b3770e68af9a76e56"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"缩放子问题并写驻点式","source_units":[{"start":52079,"end":53344,"start_line":407,"end_line":419,"raw_tex":"Without loss of generality, assume $i=k$. Scale the objective of problem \\eqref{eq:raw_form_analytic} by $\\frac{L}{2}$ and consider the equivalent problem\n\\begin{equation}\\begin{aligned}\\min_{x}\\ & \\frac{L}{2} \\left\\| x-\\left(s_{k}-\\frac{1}{L}\\nabla f_{k}\\left(s_{k}\\right)\\right)\\right\\| _{2}^{2}\\\\\n\\mathrm{s.t.}\\ & \\ell_{s_j}^{f_j}(x)+\\frac{L}{2}\\left\\| x-s_{j}\\right\\| _{2}^{2}\\leq \\ell_{s_k}^{f_k}(x)\\ \\forall\\ j\\neq k.\n\\end{aligned}\\label{eq:scaledPk}\n\\end{equation}\nBy Theorem \\ref{thm:SD_MBP} and Observation \\ref{obs:SD_MOS}, there exists unique Lagrange multipliers $\\left\\{\\lambda_j\\right\\}_{j=1}^{k-1}$ (we omit the $k$-superscript for simplicity) satisfying the following first-order optimality condition for \\eqref{eq:scaledPk}:\n$$\n    0=L\\left(s_{k}^{+}-\\left(s_{k}-\\frac{1}{L} \\nabla f_{k}\\left(s_{k}\\right)\\right)\\right)+\\sum_{j=1}^{k-1} \\lambda_{j}\\left(\\edit{\\nabla f_{j}\\left(s_{j}\\right)}+L\\left(s_{k}^{+}-s_{j}\\right)-\\nabla f_{k}\\left(s_{k}\\right)\\right)\n    $$\nAfter rearranging this gives\\begin{equation}\n0= L\\left(1+\\sum_{j=1}^{k-1}\\lambda_{j}\\right)s_{k}^{+}-L\\left(s_{k} + \\sum_{j=1}^{k-1} \\lambda_{j} s_{j}\\right)+\\nabla f_k(s_k) +\\sum_{j=1}^{k-1}\\lambda_{j}\\left(\\nabla f_j(s_j) - \\nabla f_k(s_k)\\right).\n\\label{eq:FOC2}\n\\end{equation}\n","sha256_utf8":"b1537fd9437165a92abe3345cca5db73fda3dbb52a53ac0e67b005ba508aa438"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"构造乘子偏差并估阶","source_units":[{"start":53344,"end":54102,"start_line":420,"end_line":433,"raw_tex":"Define the following scalars:\n\\begin{align}\n    \\lambda_{k} & \\equiv  1-\\sum_{j=1}^{k-1}\\lambda_{j},\\label{eq:lambda}\\\\\n    \\mu_{j} & \\equiv  \\blam_{j}-\\lambda_{j} \\ \\forall \\ j=1,\\dots,k. \\label{eq:mu}\n\\end{align}\nFor $j\\neq k$, Theorem \\ref{thm:SD_MBP} gives\n$\\mu_{j}=\\mathrm{O}\\left(\\left\\| \\S\\right\\| \\right)$ .\nFor $j=k$, combining \\eqref{eq:lambda}-\\eqref{eq:mu} with equation \\eqref{eq:lam_sum1} implies\n\\begin{equation*}\n\\mu_{k} =\\blam_{k}-\\left(1-\\sum_{j=1}^{k-1}\\lambda_{j}\\right) =\\blam_{k}-\\left(1-\\sum_{j=1}^{k-1}\\blam_{j}\\right)+\\mathrm{O}\\left(\\left\\| \\S\\right\\| \\right) =\\mathrm{O}\\left(\\left\\| \\S\\right\\| \\right).\n\\end{equation*}\nThus,\\begin{equation}\n    \\mu_j = \\mathrm{O}(\\|\\S\\|) \\ \\forall \\ j=1,\\dots,k.\\label{eq:mu_size}\n\\end{equation}\n","sha256_utf8":"ba7951bae9f702fd238ba219dda4c57beefcd5c3bb27180bb7cb4a5e8a09dc4d"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"展开更新并收集二阶项","source_units":[{"start":54102,"end":55524,"start_line":434,"end_line":442,"raw_tex":"Substituting \\eqref{eq:mu} into \\eqref{eq:FOC2}, applying identities \\eqref{eq:lam_sum1}-\\eqref{eq:lam_sum0}, collecting $\\mathrm{O}\\left(\\|\\S\\|^2\\right)$ terms, and solving for $s_k^{+}$ gives\\begin{equation}s_{k}^{+}= \\frac{1}{2-\\blam_{k}}\\left[s_{k}-\\frac{\\blam_{k}}{L}\\nabla f_{k}\\left(s_{k}\\right)\\right]+\\sum_{j=1}^{k-1}\\frac{\\blam_{j}}{2-\\blam_{k}}\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right]+\\frac{1}{L\\left(2-\\blam_{k}\\right)}\\sum_{j=1}^{k}\\mu_{j}\\nabla f_j(s_j) +\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right).\\label{eq:pre_sform}\n\\end{equation}\nAdopting the notation \\eqref{eq:A_notation} and applying the Taylor expansion \\eqref{eq:taylor}, observe\\begin{equation}\n    \\sum_{j=1}^{k}\\mu_{j}\\nabla f_j(s_j) = \\sum_{j=1}^{k}\\mu_{j}\\left(a_j + A_j s_j + \\nabla r_j (s_j)\\right) = \\sum_{j=1}^{k}\\mu_{j}a_j +\\mathrm{O}\\left(\\left\\| \\S\\right\\|^{2}\\right),\\label{eq:sum_muA}\n\\end{equation}\nwhere last equality follows from \\eqref{eq:mu_size} and Lemma \\ref{lem:r_grad}. Substituting \\eqref{eq:sum_muA} into \\eqref{eq:pre_sform} then leads to\\begin{equation}\ns_{k}^{+}=\\frac{1}{2-\\blam_{k}}\\left[s_{k}-\\frac{\\blam_{k}}{L}\\nabla f_{k}\\left(s_{k}\\right)\\right]+\\sum_{j=1}^{k-1}\\frac{\\blam_{j}}{2-\\blam_{k}}\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right]+\\frac{1}{L\\left(2-\\blam_{k}\\right)}\\underbrace{\\sum_{j=1}^{k}\\mu_{j}a_j}_{(\\star)} +\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right).\n\\label{eq:s_form}\n\\end{equation}\n","sha256_utf8":"f1533b7f3963a0e5587e05345efc1e6ddbd4dd8e2430ec2fdc5903403fee6a9e"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"投影消去乘子偏差项","source_units":[{"start":55524,"end":55879,"start_line":443,"end_line":448,"raw_tex":"Next, again combine \\eqref{eq:lambda}-\\eqref{eq:mu} with equation \\eqref{eq:lam_sum1} to deduce\n\\begin{equation*}\n\\sum_{j=1}^{k}\\mu_{j}=\\sum_{j=1}^{k}\\left(\\blam_{j}-\\lambda_{j}\\right)=0.\n\\end{equation*}\nThen, it follows from \\eqref{eq:Udef2} that $\\proj\\left[(\\star)\\right]=0$.\nThe desired result then follows from applying $\\proj$ to \\eqref{eq:s_form}.\n","sha256_utf8":"3a692743fd9c4ea34646ae7a521908275b0c4d419b0b9129f5e2c79054c1b0f7"}]},{"id":"S051","semantic_type":"lemma","kind":"lemma","title":"更新正交分量为二阶","source_units":[{"start":55892,"end":56269,"start_line":451,"end_line":457,"raw_tex":"\\begin{lem}\\label{lem:Up_component}\nAssume Setting \\ref{setting:localSD}, and let $\\bx{=}0$. Then, $s_{i}^{+}$ satisfies\n\\begin{equation*}\n\\projp\\left[s_{i}^{+}\\right]=\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right) \\ \\forall \\ i=1,\\dots,k.\n\\end{equation*}\nwhere $\\projp$ is the projection operator onto the orthogonal complement of the $\\U$-subspace \\eqref{eq:Udef1}.\n\\end{lem}","sha256_utf8":"68c3ce37ead3237cb17043552fa526462dd7635488fb648d94bd1a8e97461db1"}]},{"id":"S052","semantic_type":"proof","kind":"proof","title":"正交分量估计完整证明","source_units":[{"start":56270,"end":59750,"start_line":458,"end_line":505,"raw_tex":"\\begin{proof}\nConsider any fixed $i$. By Theorem \\ref{thm:SD_MBP}, the constraints of $\\left(P_{i}^{\\S}\\right)$\nare tight. Thus, by Observation \\ref{obs:SD_MOS}, $s_{i}^{+}$ satisfies\n\\begin{equation*}\n    f_{j}\\left(s_{j}\\right)+\\nabla f_{j}\\left(s_{j}\\right)^{T}\\left(s_i^+-s_{j}\\right)+\\frac{L}{2}\\left\\| s_i^+-s_{j}\\right\\| _{2}^{2} = f_{i}\\left(s_{i}\\right)+\\nabla f_{i}\\left(s_{i}\\right)^{T}\\left(s_i^+-s_{i}\\right) \\ \\forall \\ j\\neq i.\n\\end{equation*}\nUsing equation \\eqref{eq:taylor} in the above gives, for all $j\\neq i$,\n\\begin{multline*}\n    f(0) + a_{j}^{T}s_{j}+\\frac{1}{2}s_{j}^{T}A_{j}s_{j}+r_{j}\\left(s_j\\right)+\\left(a_{j}+A_{j}s_{j} + \\nabla r_j(s_j)\\right)^T\\left(s_{i}^{+}-s_{j}\\right)+\\frac{L}{2}\\left\\Vert s_{i}^{+}-s_{j}\\right\\Vert _{2}^{2}\\\\\n    =f(0) + a_{i}^{T}s_{i}+\\frac{1}{2}s_{i}^{T}A_{i}s_{i}+r_{i}\\left(s_i\\right)+\\left(a_{i}+A_{i}s_{i}+\\nabla r_i(s_i)\\right)^T\\left(s_{i}^{+}-s_{i}\\right).\n\\end{multline*}\nAfter simplification and rearrangement, we obtain\\begin{equation}\n\\left(a_{j}-a_{i}\\right)^{T}s_{i}^{+}=-\\left(A_{j}s_{j}-A_{i}s_{i}+\\nabla r_{j}\\left(s_{j}\\right)-\\nabla r_{i}\\left(s_{i}\\right)\\right)^{T}s_{i}^{+}-\\frac{L}{2}\\left\\| s_{i}^{+}-s_{j}\\right\\| _{2}^{2}-c_{j}\\left(s_{j}\\right)+c_{i}\\left(s_{i}\\right)  \\ \\forall \\ j\\neq i, \\label{eq:s_constr}\n\\end{equation}\nwhere we define the functions\n\\begin{equation*}\nc_{j}\\left(x\\right)\\equiv-\\frac{1}{2}x^{T}A_{j}x+r_{j}\\left(x\\right)-\\nabla r_{j}\\left(x\\right)^{T}x \\ \\forall \\ j=1,\\dots,i.\n\\end{equation*}\nBy Lemma \\ref{lem:r_grad}, $c_j(x)$ is $O\\left(\\|x\\|^2\\right)$. Since we assume $\\bx{=}0$, by Theorem \\ref{thm:SD_MBP}, $s_{i}^{+}=\\mathrm{O}\\left(\\left\\| \\S\\right\\| \\right)$.\nWe then deduce that all terms on the right-hand side of \\eqref{eq:s_constr} are $\\mathrm{O}\\left(\\left\\| \\S\\right\\|^2 \\right)$:\n\\begin{equation}\n\\left(a_{j}-a_{i}\\right)^{T}s_{i}^{+}=\\mathrm{O}\\left(\\left\\| \\S\\right\\|^2 \\right) \\ \\forall \\ j\\neq i.\\label{eq:projsp}\n\\end{equation}\n% Consider the following $QR$-decomposition \n% \\[\n% \\begin{bmatrix}\\left(a_{1}-a_{i}\\right), & \\dots, & \\left(a_{i-1}-a_{i}\\right)\\end{bmatrix}\\equiv\\bQ\\bR,\n% \\]\n% where $\\bQ\\in\\R^{n\\times\\left(k-1\\right)}$ is an orthonormal matrix,\n% and $\\bR\\in\\R^{\\left(k-1\\right)\\times\\left(k-1\\right)}$ is an upper-triangular\n% matrix. Then,\n% \\begin{equation}\n% \\bR^{T}\\bQ^{T}s_{i}^{+}=\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right).\\label{eq:s_constr2}\n% \\end{equation}\nBy \\eqref{eq:Udef1},\n\\begin{align*}\n\\U^{\\perp} & =\\Span\\left\\{ a_{i}-a_{j}:1\\leq i,j\\leq k\\right\\} =\\Span\\left\\{ a_{j}-a_{i}:\\ j \\neq i \\right\\}.\n\\end{align*}\nCombined with the affine independence of $\\left\\{ a_{j}\\right\\} _{j=1}^{k}$\n(Definition \\ref{def:strong_max}), the above implies $\\left\\{ a_{j}-a_{i}\\right\\} _{j\\neq i}$ forms a basis for $\\U^{\\perp}$. Therefore, we can express the projection onto $\\U^{\\perp}$ in terms $\\left\\{ a_{j}-a_{i}\\right\\} _{j\\neq i}$. The result now follows from equation \\eqref{eq:projsp} and standard linear algebra.\n% Thus, $\\bR$ is invertible,\n% and we can represent the projection onto $\\U^{\\perp}$ as multiplication by the matrix $\\bQ\\bQ^{T}.$ From this, we deduce\n% \\begin{equation*}\n% \\projp\\left[s_{i}^{+}\\right]=\\bQ\\bQ^{T}s_{i}^{+}=\\bQ\\left(\\bR^{T}\\right)^{-1}\\bR^{T}\\bQ^{T}s_{i}^{+}=\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right),\n% \\end{equation*}\n% where the last equality follows from \\eqref{eq:s_constr2} and the\n% fact that $\\bQ\\left(\\bR^{T}\\right)^{-1}$ is a constant-matrix independent\n% of $\\S$.\n\\end{proof}","sha256_utf8":"d49c5783f48db11402f17bade5122204cb92563fbe7cf42f1a667a996c139209"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"紧约束的Taylor展开","source_units":[{"start":56284,"end":57209,"start_line":459,"end_line":468,"raw_tex":"Consider any fixed $i$. By Theorem \\ref{thm:SD_MBP}, the constraints of $\\left(P_{i}^{\\S}\\right)$\nare tight. Thus, by Observation \\ref{obs:SD_MOS}, $s_{i}^{+}$ satisfies\n\\begin{equation*}\n    f_{j}\\left(s_{j}\\right)+\\nabla f_{j}\\left(s_{j}\\right)^{T}\\left(s_i^+-s_{j}\\right)+\\frac{L}{2}\\left\\| s_i^+-s_{j}\\right\\| _{2}^{2} = f_{i}\\left(s_{i}\\right)+\\nabla f_{i}\\left(s_{i}\\right)^{T}\\left(s_i^+-s_{i}\\right) \\ \\forall \\ j\\neq i.\n\\end{equation*}\nUsing equation \\eqref{eq:taylor} in the above gives, for all $j\\neq i$,\n\\begin{multline*}\n    f(0) + a_{j}^{T}s_{j}+\\frac{1}{2}s_{j}^{T}A_{j}s_{j}+r_{j}\\left(s_j\\right)+\\left(a_{j}+A_{j}s_{j} + \\nabla r_j(s_j)\\right)^T\\left(s_{i}^{+}-s_{j}\\right)+\\frac{L}{2}\\left\\Vert s_{i}^{+}-s_{j}\\right\\Vert _{2}^{2}\\\\\n    =f(0) + a_{i}^{T}s_{i}+\\frac{1}{2}s_{i}^{T}A_{i}s_{i}+r_{i}\\left(s_i\\right)+\\left(a_{i}+A_{i}s_{i}+\\nabla r_i(s_i)\\right)^T\\left(s_{i}^{+}-s_{i}\\right).\n\\end{multline*}\n","sha256_utf8":"393e47ee51a61f827c73c9ed52aa35b1e232537a930c7c5095816479c56cacde"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"梯度差内积的二阶界","source_units":[{"start":57209,"end":58688,"start_line":469,"end_line":490,"raw_tex":"After simplification and rearrangement, we obtain\\begin{equation}\n\\left(a_{j}-a_{i}\\right)^{T}s_{i}^{+}=-\\left(A_{j}s_{j}-A_{i}s_{i}+\\nabla r_{j}\\left(s_{j}\\right)-\\nabla r_{i}\\left(s_{i}\\right)\\right)^{T}s_{i}^{+}-\\frac{L}{2}\\left\\| s_{i}^{+}-s_{j}\\right\\| _{2}^{2}-c_{j}\\left(s_{j}\\right)+c_{i}\\left(s_{i}\\right)  \\ \\forall \\ j\\neq i, \\label{eq:s_constr}\n\\end{equation}\nwhere we define the functions\n\\begin{equation*}\nc_{j}\\left(x\\right)\\equiv-\\frac{1}{2}x^{T}A_{j}x+r_{j}\\left(x\\right)-\\nabla r_{j}\\left(x\\right)^{T}x \\ \\forall \\ j=1,\\dots,i.\n\\end{equation*}\nBy Lemma \\ref{lem:r_grad}, $c_j(x)$ is $O\\left(\\|x\\|^2\\right)$. Since we assume $\\bx{=}0$, by Theorem \\ref{thm:SD_MBP}, $s_{i}^{+}=\\mathrm{O}\\left(\\left\\| \\S\\right\\| \\right)$.\nWe then deduce that all terms on the right-hand side of \\eqref{eq:s_constr} are $\\mathrm{O}\\left(\\left\\| \\S\\right\\|^2 \\right)$:\n\\begin{equation}\n\\left(a_{j}-a_{i}\\right)^{T}s_{i}^{+}=\\mathrm{O}\\left(\\left\\| \\S\\right\\|^2 \\right) \\ \\forall \\ j\\neq i.\\label{eq:projsp}\n\\end{equation}\n% Consider the following $QR$-decomposition \n% \\[\n% \\begin{bmatrix}\\left(a_{1}-a_{i}\\right), & \\dots, & \\left(a_{i-1}-a_{i}\\right)\\end{bmatrix}\\equiv\\bQ\\bR,\n% \\]\n% where $\\bQ\\in\\R^{n\\times\\left(k-1\\right)}$ is an orthonormal matrix,\n% and $\\bR\\in\\R^{\\left(k-1\\right)\\times\\left(k-1\\right)}$ is an upper-triangular\n% matrix. Then,\n% \\begin{equation}\n% \\bR^{T}\\bQ^{T}s_{i}^{+}=\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right).\\label{eq:s_constr2}\n% \\end{equation}\n","sha256_utf8":"c9c9d121bb6655a17a8d5abdbfdf493b5a8255a649f8014d83ab85e97ae0e0f5"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"利用正交补基控制投影","source_units":[{"start":58688,"end":59739,"start_line":491,"end_line":504,"raw_tex":"By \\eqref{eq:Udef1},\n\\begin{align*}\n\\U^{\\perp} & =\\Span\\left\\{ a_{i}-a_{j}:1\\leq i,j\\leq k\\right\\} =\\Span\\left\\{ a_{j}-a_{i}:\\ j \\neq i \\right\\}.\n\\end{align*}\nCombined with the affine independence of $\\left\\{ a_{j}\\right\\} _{j=1}^{k}$\n(Definition \\ref{def:strong_max}), the above implies $\\left\\{ a_{j}-a_{i}\\right\\} _{j\\neq i}$ forms a basis for $\\U^{\\perp}$. Therefore, we can express the projection onto $\\U^{\\perp}$ in terms $\\left\\{ a_{j}-a_{i}\\right\\} _{j\\neq i}$. The result now follows from equation \\eqref{eq:projsp} and standard linear algebra.\n% Thus, $\\bR$ is invertible,\n% and we can represent the projection onto $\\U^{\\perp}$ as multiplication by the matrix $\\bQ\\bQ^{T}.$ From this, we deduce\n% \\begin{equation*}\n% \\projp\\left[s_{i}^{+}\\right]=\\bQ\\bQ^{T}s_{i}^{+}=\\bQ\\left(\\bR^{T}\\right)^{-1}\\bR^{T}\\bQ^{T}s_{i}^{+}=\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right),\n% \\end{equation*}\n% where the last equality follows from \\eqref{eq:s_constr2} and the\n% fact that $\\bQ\\left(\\bR^{T}\\right)^{-1}$ is a constant-matrix independent\n% of $\\S$.\n","sha256_utf8":"afe64599d272ca4626615521470bf6237f43e5f08bfced6af94f6a4db0f9ec4d"}]},{"id":"S056","semantic_type":"theorem","kind":"theorem","title":"更新的投影梯度近似","source_units":[{"start":59752,"end":60448,"start_line":507,"end_line":515,"raw_tex":"\\begin{thm}\n\\label{thm:approx_s} \\textbf{(Gradient Descent Approximation of Survey Updates)} Assume Setting \\ref{setting:localSD}.  Then the solution of the $i$-th Survey Descent subproblem satisfies, for all $i$, \\begin{equation}\ns_{i}^{+}= \\ts_{i}+\\mathrm{O}\\left(\\left\\| \\S - \\bX\\right\\| ^{2}\\right) \\label{eq:SD_iter}\n\\end{equation}\nwhere we define\\begin{equation}\n\\ts_{i}\\equiv\\frac{1}{2-\\blam_{i}}\\proj\\left[s_{i}-\\frac{\\blam_{i}}{L}\\nabla f_{i}\\left(s_{i}\\right)\\right]+\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\proj\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right], \\label{eq:s_tilde}\n\\end{equation}\nand $\\proj$ is the projection operator onto the $\\U$-subspace \\eqref{eq:Udef1}.\n\\end{thm}","sha256_utf8":"51a6da9e6c20716b669b6830a3929774d96a0df08c386d564f33c9d0b523654c"}]},{"id":"S057","semantic_type":"proof","kind":"proof","title":"梯度近似完整证明","source_units":[{"start":60449,"end":60815,"start_line":516,"end_line":518,"raw_tex":"\\begin{proof}\nWithout loss of generality, assume that $\\bx{=}0$. Then, the desired result follows from decomposing $s_{k}^{+} = \\proj\\left[s_{k}^{+}\\right] + \\projp\\left[s_{k}^{+}\\right]$ and applying Lemmas \\ref{lem:U_component} and \\ref{lem:Up_component}. We can deduce the general $\\bx{\\neq}0$ case with the change of variables $x{\\leftarrow}(x-\\bx)$.\n\\end{proof}","sha256_utf8":"268dda081f1ff1b9d10543ae5a199cc63bbfc85e87452d6f5ec02b14602543d2"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"合并正交分量并平移","source_units":[{"start":60463,"end":60804,"start_line":517,"end_line":517,"raw_tex":"Without loss of generality, assume that $\\bx{=}0$. Then, the desired result follows from decomposing $s_{k}^{+} = \\proj\\left[s_{k}^{+}\\right] + \\projp\\left[s_{k}^{+}\\right]$ and applying Lemmas \\ref{lem:U_component} and \\ref{lem:Up_component}. We can deduce the general $\\bx{\\neq}0$ case with the change of variables $x{\\leftarrow}(x-\\bx)$.\n","sha256_utf8":"818ffa3fbe18a0c9d5664181350062702cc3c3a3d993a6951eee1a47dcd91511"}]},{"id":"S059","semantic_type":"theorem","kind":"theorem","title":"投影梯度步收缩估计","source_units":[{"start":61486,"end":62070,"start_line":524,"end_line":530,"raw_tex":"\\begin{thm}\n\\textbf{\\label{thm:PR}(Projected-GD Behavior)} Assume that $\\S$ is a valid survey for the strong $\\C^2$ max function objective $f$. Then, the following holds for all $i$ and any constant $\\tL \\geq L$:\n\\begin{align*}\n\\left\\| \\proj\\left[s_i - \\frac{1}{\\tL} \\nabla f_i(s_i)\\right] - \\bx\\right\\| _{2}^{2} &\\leq \\left(1-\\frac{\\delta}{\\tL}\\right)\\left\\| s_{i} - \\bx \\right\\| _{2}^{2},\\\\\nf_{i}\\left(\\proj\\left[s_i - \\frac{1}{\\tL} \\nabla f_i(s_i)\\right]\\right) - f_i(\\bx) &\\leq \\frac{\\tL}{2}\\left(1-\\frac{\\delta}{\\tL}\\right)\\left\\| s_{i} - \\bx\\right\\| ^{2}.\n\\end{align*}\n\\end{thm}","sha256_utf8":"fc92c9604f8b8050d9b7deeb9a6db618bfdf99496a8727f9c0f5292a75199540"}]},{"id":"S060","semantic_type":"proof","kind":"proof","title":"投影梯度性质完整证明","source_units":[{"start":62071,"end":62532,"start_line":531,"end_line":533,"raw_tex":"\\begin{proof}\nConsider any fixed $i$. It is easy to check that $\\bx = \\arg\\min_{x \\in \\U} f_i(x)$ using first-order optimality conditions combined with \\eqref{eq:lam_sum1}-\\eqref{eq:lam_sum0} and \\eqref{eq:Udef3}. The desired result then follows from classical projected-GD convergence results (given in standard texts such as \\citet[Theorem 10.29]{beck2017}) after observing that $\\U$ is convex and $f_i$ is $L$-smooth and $\\delta$-strongly convex.\n\\end{proof}","sha256_utf8":"e99b259058eb261006437a4bd25b5b7cc0e966ca234199173eb5fd608d3a36dc"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"识别约束极小点并引用估计","source_units":[{"start":62085,"end":62521,"start_line":532,"end_line":532,"raw_tex":"Consider any fixed $i$. It is easy to check that $\\bx = \\arg\\min_{x \\in \\U} f_i(x)$ using first-order optimality conditions combined with \\eqref{eq:lam_sum1}-\\eqref{eq:lam_sum0} and \\eqref{eq:Udef3}. The desired result then follows from classical projected-GD convergence results (given in standard texts such as \\citet[Theorem 10.29]{beck2017}) after observing that $\\U$ is convex and $f_i$ is $L$-smooth and $\\delta$-strongly convex.\n","sha256_utf8":"bbaf63d4d70f50f4365c69bc70388afaf9a63a79184d713ec8ba7fa4d909f6f9"}]},{"id":"S062","semantic_type":"lemma","kind":"lemma","title":"参考更新的统一收缩界","source_units":[{"start":62839,"end":63277,"start_line":539,"end_line":545,"raw_tex":"\\begin{lem}\n\\label{lem:st_bound} Assume Setting \\ref{setting:localSD}, and suppose $\\bx{=}0$. Then, the following must hold:\n\\begin{equation*}\n\\left\\| \\ts_{i}\\right\\| _{2}^{2}\\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)\\max_{j=1,....,k}\\left\\| s_{j}\\right\\| _{2}^{2} \\ \\forall \\ i=1,\\dots,k,\n\\end{equation*}\nwhere $\\ts_i$ and $\\blam_{\\min}$ are as defined in equations \\eqref{eq:s_tilde} and \\eqref{eq:min_lam}, respectively.\n\\end{lem}","sha256_utf8":"cb77b8feb2e84d9a1314865a4d258db31e2b9d1da9f788f16f7ab0da06c813f7"}]},{"id":"S063","semantic_type":"proof","kind":"proof","title":"参考更新收缩完整证明","source_units":[{"start":63278,"end":64504,"start_line":546,"end_line":555,"raw_tex":"\\begin{proof} Consider any fixed $i$. Then,\n\\begin{align*}\n\\left\\| \\ts_{i}\\right\\| _{2}^{2} & =\\left\\| \\frac{1}{2-\\blam_{i}}\\proj\\left[s_{i}-\\frac{\\blam_{i}}{L}\\nabla f_{i}\\left(s_{i}\\right)\\right]+\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\proj\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right] \\right\\| _{2}^{2}  \\text{ (Substitute \\eqref{eq:s_tilde})}\\\\\n & \\leq\\frac{1}{2-\\blam_{i}}\\left\\| \\proj\\left[s_{i}-\\frac{\\blam_{i}}{L}\\nabla f_{i}\\left(s_{i}\\right)\\right]\\right\\| _{2}^{2}+\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\left\\| \\proj\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right]\\right\\| _{2}^{2}\\text{ (Convexity)}\\\\\n & \\leq\\frac{1}{2-\\blam_{i}}\\left(1-\\frac{\\blam_{i}\\delta}{L}\\right)\\left\\| s_{i}\\right\\| _{2}^{2}+\\left(1-\\frac{\\delta}{L}\\right)\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\left\\| s_{j}\\right\\| _{2}^{2} \\text{ (Theorem \\ref{thm:PR} where $\\tL=\\frac{L}{\\blam_i}$)}\\\\\n & \\leq\\left(1-\\frac{\\blam_{i}\\delta}{L}\\right)\\left[\\frac{1}{2-\\blam_{i}}\\left\\| s_{i}\\right\\| _{2}^{2}+\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\left\\| s_{j}\\right\\| _{2}^{2}\\right]\\\\\n & \\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)\\max_{j=1,....,k}\\left\\| s_{j}\\right\\| _{2}^{2}.\n\\end{align*}\nThis completes the proof.\n\\end{proof}","sha256_utf8":"fa103e75dad6d5e80348ec05492fd575d66759c54b2195ba1da097076fb768e1"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"凸组合结合投影步收缩","source_units":[{"start":63278,"end":64493,"start_line":546,"end_line":554,"raw_tex":"\\begin{proof} Consider any fixed $i$. Then,\n\\begin{align*}\n\\left\\| \\ts_{i}\\right\\| _{2}^{2} & =\\left\\| \\frac{1}{2-\\blam_{i}}\\proj\\left[s_{i}-\\frac{\\blam_{i}}{L}\\nabla f_{i}\\left(s_{i}\\right)\\right]+\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\proj\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right] \\right\\| _{2}^{2}  \\text{ (Substitute \\eqref{eq:s_tilde})}\\\\\n & \\leq\\frac{1}{2-\\blam_{i}}\\left\\| \\proj\\left[s_{i}-\\frac{\\blam_{i}}{L}\\nabla f_{i}\\left(s_{i}\\right)\\right]\\right\\| _{2}^{2}+\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\left\\| \\proj\\left[s_j - \\frac{1}{L} \\nabla f_j(s_j)\\right]\\right\\| _{2}^{2}\\text{ (Convexity)}\\\\\n & \\leq\\frac{1}{2-\\blam_{i}}\\left(1-\\frac{\\blam_{i}\\delta}{L}\\right)\\left\\| s_{i}\\right\\| _{2}^{2}+\\left(1-\\frac{\\delta}{L}\\right)\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\left\\| s_{j}\\right\\| _{2}^{2} \\text{ (Theorem \\ref{thm:PR} where $\\tL=\\frac{L}{\\blam_i}$)}\\\\\n & \\leq\\left(1-\\frac{\\blam_{i}\\delta}{L}\\right)\\left[\\frac{1}{2-\\blam_{i}}\\left\\| s_{i}\\right\\| _{2}^{2}+\\sum_{j\\neq i}\\frac{\\blam_{j}}{2-\\blam_{i}}\\left\\| s_{j}\\right\\| _{2}^{2}\\right]\\\\\n & \\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)\\max_{j=1,....,k}\\left\\| s_{j}\\right\\| _{2}^{2}.\n\\end{align*}\nThis completes the proof.\n","sha256_utf8":"1f041b0d92c4c1a9b1847a767a3b1d118ba2e0c2b02d4f6d6284ac8866c183ac"}]},{"id":"S065","semantic_type":"theorem","kind":"theorem","title":"调查范数单步收缩","source_units":[{"start":64506,"end":64957,"start_line":557,"end_line":562,"raw_tex":"\\begin{thm} \\textbf{(Survey-Norm Contraction)} \\label{thm:s_monotone} Assume Setting \\ref{setting:localSD}.  Then the following must hold:\n\\begin{equation*}\n\\max_{j=1,...,k}\\left\\| s_{j}^{+} - \\bx \\right\\| _{2}^{2}\\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)\\max_{j=1,...,k}\\left\\| s_{j} - \\bx \\right\\| _{2}^{2}+\\mathrm{O}\\left(\\left\\| \\S - \\bX\\right\\|^{3}\\right),\n\\end{equation*}\nwhere $\\blam_{\\min}$ is defined as in \\eqref{eq:min_lam}.\n\\end{thm}","sha256_utf8":"3c1a0ae26b590fef71e49c9013a5895e7ece97475013071e8f34b9f9b1cb2100"}]},{"id":"S066","semantic_type":"proof","kind":"proof","title":"范数收缩完整证明","source_units":[{"start":64958,"end":65894,"start_line":563,"end_line":570,"raw_tex":"\\begin{proof}\nWithout loss of generality, assume $\\bx = 0$. Consider any fixed $i$. By Lemma \\ref{lem:st_bound}, $\\ts_{i}=\\mathrm{O}\\left(\\left\\| \\S\\right\\| \\right)$. Then,\\begin{align*}\n\\left\\| s_{i}^{+}\\right\\| _{2}^{2} & =\\left\\| \\ts_{i}+\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right)\\right\\| _{2}^{2} \\text{ (Theorem \\ref{thm:approx_s})}\\\\\n & =\\left\\| \\ts_{i}\\right\\| _{2}^{2}+2\\left\\langle \\ts_{i},\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right)\\right\\rangle +\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{4}\\right)\\\\\n & \\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)\\max_{j=1,....,k}\\left\\| s_{j}\\right\\| _{2}^{2}+\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{3}\\right) \\text{ (Lemma \\ref{lem:st_bound})}.\n\\end{align*}\nThe right-hand side is independent of $i$, so taking the max over $i$ on the left-hand side leads to the desired result. We can deduce the general $\\bx{\\neq}0$ case with the change of variables $x{\\leftarrow}(x-\\bx)$.\n\\end{proof}","sha256_utf8":"851df233da05bfdb555febf297f586ac2c92b4cc7ff09c45456c138f9dfa9806"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"平方展开控制三阶误差","source_units":[{"start":64972,"end":65665,"start_line":564,"end_line":568,"raw_tex":"Without loss of generality, assume $\\bx = 0$. Consider any fixed $i$. By Lemma \\ref{lem:st_bound}, $\\ts_{i}=\\mathrm{O}\\left(\\left\\| \\S\\right\\| \\right)$. Then,\\begin{align*}\n\\left\\| s_{i}^{+}\\right\\| _{2}^{2} & =\\left\\| \\ts_{i}+\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right)\\right\\| _{2}^{2} \\text{ (Theorem \\ref{thm:approx_s})}\\\\\n & =\\left\\| \\ts_{i}\\right\\| _{2}^{2}+2\\left\\langle \\ts_{i},\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{2}\\right)\\right\\rangle +\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{4}\\right)\\\\\n & \\leq\\left(1-\\frac{\\blam_{\\min}\\delta}{L}\\right)\\max_{j=1,....,k}\\left\\| s_{j}\\right\\| _{2}^{2}+\\mathrm{O}\\left(\\left\\| \\S\\right\\| ^{3}\\right) \\text{ (Lemma \\ref{lem:st_bound})}.\n\\end{align*}\n","sha256_utf8":"b57672ab96d50c1fccc9a1174bbff49c14d22e070be180c21e6620083160ffe6"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"取点间最大值并平移","source_units":[{"start":65665,"end":65883,"start_line":569,"end_line":569,"raw_tex":"The right-hand side is independent of $i$, so taking the max over $i$ on the left-hand side leads to the desired result. We can deduce the general $\\bx{\\neq}0$ case with the change of variables $x{\\leftarrow}(x-\\bx)$.\n","sha256_utf8":"c6734456df5b2ff951820c72100fa205c6a162611e466e6c1aa8a89a0707e1bf"}]},{"id":"S069","semantic_type":"corollary","kind":"corollary","title":"重复迭代良定义","source_units":[{"start":66907,"end":67210,"start_line":584,"end_line":585,"raw_tex":"\\begin{cor}\\textbf{(Well-Definedness of Survey Descent Repetitions)}\\label{cor:viable} Procedure \\ref{procedure:SD} is well-defined:  for each $t=0,1,2\\ldots$, the Survey Descent iteration $\\left\\{ \\left(P_{i}^{\\S^{t}}\\right)\\right\\} _{i=1}^{k}$ is feasible and produces a valid output survey.\n\\end{cor}","sha256_utf8":"ffa5f82a1e833acdb4a8cea2b8d9cd91106feed90fb91a923f82b1616ba3301b"}]},{"id":"S070","semantic_type":"proof","kind":"proof","title":"重复迭代良定义完整证明","source_units":[{"start":67211,"end":68014,"start_line":586,"end_line":589,"raw_tex":"\\begin{proof}\nFor $\\S^{0}$ sufficiently close to $\\bX$, Theorem \\ref{thm:s_monotone} inductively implies that $\\left\\| \\S^{t+1} - \\bX \\right\\| <\\left\\| \\S^{t} - \\bX\\right\\| $ for all $t\\in\\nonneg$. In other words, output surveys will always move closer to $\\bX$, and Theorem \\ref{thm:SD_MBP} and \\ref{thm:strictval} continue to hold for Survey Descent iterations $\\left\\{\\left(P_{i}^{\\S^{t}}\\right)\\right\\}_{i=1}^{k}$ for all $t\\in \\nonneg$. Thus, Theorem \\ref{thm:SD_MBP} implies the feasibility all Survey Descent iterations while Theorem \\ref{thm:strictval} implies the validity of output surveys.\n% The viability of implementing the consecutive Survey Descent iterations with only a first-order oracle on $f$ follows directly from Remark \\ref{rem:SD_FOM} and Theorem \\ref{thm:strictval}.\n\\end{proof}","sha256_utf8":"d31879310570759ac825ead18ca13bb2ad303069591af4f86b5dc955c367da63"}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","title":"归纳保持邻域可行与有效","source_units":[{"start":67225,"end":68003,"start_line":587,"end_line":588,"raw_tex":"For $\\S^{0}$ sufficiently close to $\\bX$, Theorem \\ref{thm:s_monotone} inductively implies that $\\left\\| \\S^{t+1} - \\bX \\right\\| <\\left\\| \\S^{t} - \\bX\\right\\| $ for all $t\\in\\nonneg$. In other words, output surveys will always move closer to $\\bX$, and Theorem \\ref{thm:SD_MBP} and \\ref{thm:strictval} continue to hold for Survey Descent iterations $\\left\\{\\left(P_{i}^{\\S^{t}}\\right)\\right\\}_{i=1}^{k}$ for all $t\\in \\nonneg$. Thus, Theorem \\ref{thm:SD_MBP} implies the feasibility all Survey Descent iterations while Theorem \\ref{thm:strictval} implies the validity of output surveys.\n% The viability of implementing the consecutive Survey Descent iterations with only a first-order oracle on $f$ follows directly from Remark \\ref{rem:SD_FOM} and Theorem \\ref{thm:strictval}.\n","sha256_utf8":"cb6fbe9e6a3de17e0e11923f63e62ae0b190548d740a18d228601e5722b99e8c"}]},{"id":"S072","semantic_type":"lemma","kind":"lemma","title":"中心仿射空间局部性质","source_units":[{"start":99811,"end":100387,"start_line":933,"end_line":940,"raw_tex":"\\begin{lem}\\textbf{(Local Properties of the Central Affine Subspace)}\\label{lem:A}\n    For any ball-specifications $\\spec$ sufficiently close to base ball-specifications $\\bspec$, the following properties hold:\n    \\begin{enumerate}\n        \\item $\\A\\spec$ is non-empty with dimension $n{-}\\bk{+}1$.\n        \\item For all $i\\neq j$, the direction $(z_i - z_j)$ is orthogonal to the linear subspace parallel to $\\A\\spec$.\n        \\item For $w\\in\\R^n$, the projection $\\projA{\\spec}(w)$ is well-defined and varies smoothly with $\\spec$ and $w$.\n    \\end{enumerate}\n    \\end{lem}","sha256_utf8":"fa992ea4457d46d91e97d6cb0bfee1e7ac4f828f71f3072e219be02043772943"}]},{"id":"S073","semantic_type":"proof","kind":"proof","title":"中心仿射空间完整证明","source_units":[{"start":100392,"end":102285,"start_line":941,"end_line":968,"raw_tex":"\\begin{proof}\n    Observe that we can rewrite \\eqref{eq:CAS} as the following $\\left(\\bk{-}1\\right){\\times}n$ linear system:\\begin{equation}\n    \\A\\spec=\\left\\{ x\\in\\R^{n}:\\ M\\spec x=b\\spec \\right\\} ,\\label{eq:CAS_sys}\n    \\end{equation}\n    where \n    \\[\n    M\\spec\\equiv2\\begin{bmatrix}\\left(z_{2}-z_{1}\\right)^{T}\\\\\n    \\vdots\\\\\n    \\left(z_{\\m}-z_{1}\\right)^{T}\n    \\end{bmatrix}\n    \\text{ and }\n    b\\spec\\equiv\\begin{bmatrix}\\left(\\left\\| z_{2}\\right\\| _{2}^{2}-r_{2}^{2}\\right)-\\left\\| z_{1}\\right\\| _{2}^{2}+r_{1}^{2}\\\\\n    \\vdots\\\\\n    \\left(\\left\\| z_{\\bk}\\right\\| _{2}^{2}-r_{\\bk}^{2}\\right)-\\left\\| z_{1}\\right\\| _{2}^{2}+r_{1}^{2}\n    \\end{bmatrix}.\\\n    \\]\n    \n    Consider $\\A\\bspec$. Since $\\left\\{ \\bz_{j}\\right\\} _{j=1}^{\\bk}$ are linearly independent,\n    $\\left\\{ \\bz_{j}-\\bz_{1}\\right\\} _{j=2}^{\\bk}$---which are the rows of $M\\bspec$---are also linearly independent. For $\\spec$ sufficiently close to $\\bspec$, the rows of $M\\spec$ will remain linearly independent. In this case, the linear system \\eqref{eq:CAS_sys} is then necessarily solvable: $\\A\\spec$ is non-empty. Let $x_0$ be such a solution. Then, $\\A\\spec$ consists of all vectors $x \\in \\R^n$ such that $x{-}x_0$ is orthogonal to the $m{-}1$ linearly independent rows of $M\\spec$. Thus, $\\Dim\\left\\{\\A\\spec\\right\\}{=}n{-}\\bk{+}1$. \n    \n    The vectors $\\left\\{(z_i - z_j)\\right\\}_{j\\neq i}$ clearly all lie in the rowspace of $M\\spec$. Hence, the vectors$\\left\\{(z_i - z_j)\\right\\}_{j\\neq i}$ are orthogonal to all direction in $\\A\\spec$.\n    \n    Finally, write $M$ instead of $M\\spec$ for simplicity. Since $M$ has independent rows, the projection of $w$ onto $\\A\\spec$ is explicitly given by\n    \\begin{equation*}\n    \\projA{\\spec}\\left(w\\right)=\\left(I-M\\left(M^{T}M\\right)^{-1}M^{T}\\right)w+M\\left(M^{T}M\\right)^{-1}b,\n    \\end{equation*}\n    which varies smoothly with $\\spec$ and $w$.\n    \\end{proof}","sha256_utf8":"0b64b4a1919333c8221c74f2f368fc93ad86ec08e2c24f4594c8158f44356f24"}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","title":"中心空间改写为线性系统","source_units":[{"start":100406,"end":101064,"start_line":942,"end_line":956,"raw_tex":"    Observe that we can rewrite \\eqref{eq:CAS} as the following $\\left(\\bk{-}1\\right){\\times}n$ linear system:\\begin{equation}\n    \\A\\spec=\\left\\{ x\\in\\R^{n}:\\ M\\spec x=b\\spec \\right\\} ,\\label{eq:CAS_sys}\n    \\end{equation}\n    where \n    \\[\n    M\\spec\\equiv2\\begin{bmatrix}\\left(z_{2}-z_{1}\\right)^{T}\\\\\n    \\vdots\\\\\n    \\left(z_{\\m}-z_{1}\\right)^{T}\n    \\end{bmatrix}\n    \\text{ and }\n    b\\spec\\equiv\\begin{bmatrix}\\left(\\left\\| z_{2}\\right\\| _{2}^{2}-r_{2}^{2}\\right)-\\left\\| z_{1}\\right\\| _{2}^{2}+r_{1}^{2}\\\\\n    \\vdots\\\\\n    \\left(\\left\\| z_{\\bk}\\right\\| _{2}^{2}-r_{\\bk}^{2}\\right)-\\left\\| z_{1}\\right\\| _{2}^{2}+r_{1}^{2}\n    \\end{bmatrix}.\\\n    \\]\n","sha256_utf8":"03044926a8ce957cc30600a6c1acb81bd0bf708ff631508b394f078a887f800c"}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","title":"行满秩保证非空及维数","source_units":[{"start":101069,"end":101709,"start_line":958,"end_line":959,"raw_tex":"    Consider $\\A\\bspec$. Since $\\left\\{ \\bz_{j}\\right\\} _{j=1}^{\\bk}$ are linearly independent,\n    $\\left\\{ \\bz_{j}-\\bz_{1}\\right\\} _{j=2}^{\\bk}$---which are the rows of $M\\bspec$---are also linearly independent. For $\\spec$ sufficiently close to $\\bspec$, the rows of $M\\spec$ will remain linearly independent. In this case, the linear system \\eqref{eq:CAS_sys} is then necessarily solvable: $\\A\\spec$ is non-empty. Let $x_0$ be such a solution. Then, $\\A\\spec$ consists of all vectors $x \\in \\R^n$ such that $x{-}x_0$ is orthogonal to the $m{-}1$ linearly independent rows of $M\\spec$. Thus, $\\Dim\\left\\{\\A\\spec\\right\\}{=}n{-}\\bk{+}1$. \n","sha256_utf8":"b5e9b69d807f9cde92639918f5d7fb94e5d76b28b7d39035cfbc5916a254d927"}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","title":"中心差与空间方向正交","source_units":[{"start":101714,"end":101917,"start_line":961,"end_line":961,"raw_tex":"    The vectors $\\left\\{(z_i - z_j)\\right\\}_{j\\neq i}$ clearly all lie in the rowspace of $M\\spec$. Hence, the vectors$\\left\\{(z_i - z_j)\\right\\}_{j\\neq i}$ are orthogonal to all direction in $\\A\\spec$.\n","sha256_utf8":"9e69796c5ea2f22e9aac51bf636a3ee6e888b08295a9c568f1eb94016a3b6d82"}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","title":"显式投影与参数光滑性","source_units":[{"start":101922,"end":102270,"start_line":963,"end_line":967,"raw_tex":"    Finally, write $M$ instead of $M\\spec$ for simplicity. Since $M$ has independent rows, the projection of $w$ onto $\\A\\spec$ is explicitly given by\n    \\begin{equation*}\n    \\projA{\\spec}\\left(w\\right)=\\left(I-M\\left(M^{T}M\\right)^{-1}M^{T}\\right)w+M\\left(M^{T}M\\right)^{-1}b,\n    \\end{equation*}\n    which varies smoothly with $\\spec$ and $w$.\n","sha256_utf8":"4e405046ebf3aeeee664653b61af4f04f676bcfb693ee096917b77cea08c651b"}]},{"id":"S078","semantic_type":"lemma","kind":"lemma","title":"核心的局部良定义性","source_units":[{"start":103359,"end":103612,"start_line":979,"end_line":981,"raw_tex":"\\begin{lem}\n    \\textbf{\\label{lem:core} (Local Properties of Core)} For any ball-specifications $\\spec$ sufficiently close to base ball-specifications $\\bspec$, the core in \\eqref{eq:core} is well-defined and varies smoothly with $\\spec$.\n    \\end{lem}","sha256_utf8":"576c3ad97a528ec57286bb0d4ed5edf4f0454ac99889bb4ce57747c10c3a5629"}]},{"id":"S079","semantic_type":"proof","kind":"proof","title":"核心性质完整证明","source_units":[{"start":103617,"end":104044,"start_line":982,"end_line":983,"raw_tex":"\\begin{proof} By the second fact in Lemma \\ref{lem:A}, for any pair $i\\neq j$, the difference $(z_i{-}z_j)$ is orthogonal to the linear subspace parallel to $\\A\\param$. Therefore, $\\projA{\\param}(z_i)$ and $\\projA{\\param}(z_j)$ are the same. Hence, \\eqref{eq:core} is well-defined. The smoothness of the $c\\spec$ with respect to $\\spec$ then follows from \\eqref{eq:core} and the third fact of Lemma \\ref{lem:A}.\n    \\end{proof}","sha256_utf8":"cd495af875972d840879b427062858296ece71740da6dcceddc18bfe1e73b883"}]},{"id":"S080","semantic_type":"proof_step","kind":"proof_step","title":"各中心投影相同且光滑","source_units":[{"start":103617,"end":104029,"start_line":982,"end_line":982,"raw_tex":"\\begin{proof} By the second fact in Lemma \\ref{lem:A}, for any pair $i\\neq j$, the difference $(z_i{-}z_j)$ is orthogonal to the linear subspace parallel to $\\A\\param$. Therefore, $\\projA{\\param}(z_i)$ and $\\projA{\\param}(z_j)$ are the same. Hence, \\eqref{eq:core} is well-defined. The smoothness of the $c\\spec$ with respect to $\\spec$ then follows from \\eqref{eq:core} and the third fact of Lemma \\ref{lem:A}.\n","sha256_utf8":"8c3fea32556ca6da220055f9187c76422f91f96b0aa698a5ddc1262a65d7139c"}]},{"id":"S081","semantic_type":"lemma","kind":"lemma","title":"基准空间线性且核心非零","source_units":[{"start":104054,"end":104233,"start_line":985,"end_line":988,"raw_tex":"\\begin{lem}\n    \\textbf{(The Base Case)} \\label{lem:linA}\n    At base ball-specifications, the central affine subspace is a linear subspace and the core is non-zero.\n    \\end{lem}","sha256_utf8":"98b434d6f5428c79dee990b39c6fadbe110747efb86bcb8cac95c120cf916dd8"}]},{"id":"S082","semantic_type":"proof","kind":"proof","title":"基准几何性质完整证明","source_units":[{"start":104238,"end":105097,"start_line":989,"end_line":998,"raw_tex":"\\begin{proof}\n    By Definition \\ref{def:base}, $\\br_{j}^{2}=\\left\\| \\bz_{j}\\right\\| _{2}^{2}$\n    for all $j$. Thus, \\eqref{eq:CAS} simplifies to\n    \\begin{equation*}\n    \\A\\bspec=\\left\\{ x\\in\\R^{n}:\\ 2\\bz_{j}^{T}x\\text{ equal  for all } j=1,\\dots,\\bk\\right\\}.\n    \\end{equation*}\n    Clearly, the origin satisfies the RHS conditions above, so $0\\in\\A\\bspec$.\n    \n    \\sloppy Next, assume for contradiction that $0{=}c\\bspec$. By definition, $c\\bspec$ is the projection of $\\bz_j$ onto $\\A\\bspec$ for any $j$, so the vectors $\\left\\{ \\bz_{j}-c\\bspec\\right\\}_{j=1}^{\\bk}{=}\\left\\{ \\bz_{j}\\right\\} _{j=1}^{\\bk}$ must be orthogonal to $\\A\\bspec$. The linear independence of $\\left\\{ \\bz_{j}\\right\\}_{j=1}^{\\bk}$ implies $\\Dim\\left\\{ \\A\\bspec\\right\\}{\\leq}n{-}\\bk$, which contradicts the first fact of Lemma \\ref{lem:A}. Thus, $0{\\neq}c\\bspec$.\n    \\end{proof}","sha256_utf8":"263a780f50f6475bfe53d847cfc6d35f39df134f0b5f58026259ff7f584f4f10"}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","title":"基准半径使空间包含原点","source_units":[{"start":104252,"end":104600,"start_line":990,"end_line":995,"raw_tex":"    By Definition \\ref{def:base}, $\\br_{j}^{2}=\\left\\| \\bz_{j}\\right\\| _{2}^{2}$\n    for all $j$. Thus, \\eqref{eq:CAS} simplifies to\n    \\begin{equation*}\n    \\A\\bspec=\\left\\{ x\\in\\R^{n}:\\ 2\\bz_{j}^{T}x\\text{ equal  for all } j=1,\\dots,\\bk\\right\\}.\n    \\end{equation*}\n    Clearly, the origin satisfies the RHS conditions above, so $0\\in\\A\\bspec$.\n","sha256_utf8":"3242f7d589994e1cdd843fc3e961ac7d6499031827d09ea7f8ea66c1ddaf7caf"}]},{"id":"S084","semantic_type":"proof_step","kind":"proof_step","title":"维数反证核心非零","source_units":[{"start":104605,"end":105082,"start_line":997,"end_line":997,"raw_tex":"    \\sloppy Next, assume for contradiction that $0{=}c\\bspec$. By definition, $c\\bspec$ is the projection of $\\bz_j$ onto $\\A\\bspec$ for any $j$, so the vectors $\\left\\{ \\bz_{j}-c\\bspec\\right\\}_{j=1}^{\\bk}{=}\\left\\{ \\bz_{j}\\right\\} _{j=1}^{\\bk}$ must be orthogonal to $\\A\\bspec$. The linear independence of $\\left\\{ \\bz_{j}\\right\\}_{j=1}^{\\bk}$ implies $\\Dim\\left\\{ \\A\\bspec\\right\\}{\\leq}n{-}\\bk$, which contradicts the first fact of Lemma \\ref{lem:A}. Thus, $0{\\neq}c\\bspec$.\n","sha256_utf8":"9bb8f08107dd3a645976d56cf76e9af284818ad27d6d79e01bfb77fee4daa311"}]},{"id":"S085","semantic_type":"lemma","kind":"lemma","title":"余量严格正且光滑","source_units":[{"start":105107,"end":105380,"start_line":1000,"end_line":1003,"raw_tex":"\\begin{lem}\n    \\textbf{(Local Properties of the Margin)\\label{lem:margin}} \n    For ball-specifications $\\spec$ near some base ball-specifications $\\bspec$, the margin in \\eqref{eq:margin} is well-defined, strictly-positive, and varies smoothly with $\\spec$.\n    \\end{lem}","sha256_utf8":"4e21c23bbcc8d172ae1ef0d5455fe0bc10a749d965ac11a0d29d7cd1447a1ee6"}]},{"id":"S086","semantic_type":"proof","kind":"proof","title":"余量性质完整证明","source_units":[{"start":105385,"end":106814,"start_line":1004,"end_line":1019,"raw_tex":"\\begin{proof}\n    It suffices to show that the square of the margin,\\begin{equation}\n    r_{j}^{2}-\\left\\| z_{j}-c\\spec\\right\\| _{2}^{2}, \\label{eq:margin_sq}\n    \\end{equation}\n    does not depend on the index $j$, is strictly-positive, and varies smoothly with $\\param$. First, consider the core and central affine subspace exactly at base ball-specifications $\\bspec$. By Lemma\n    \\ref{lem:linA}, since $0 \\in \\A\\bspec$ and $\\projA{\\bspec}\\left(\\bz_{j}\\right){=}c\\bspec{\\neq}0$\n    for all $j$, the definition of a projection gives\n    \\begin{equation*}\n    \\left\\| \\bz_{j}-c\\bspec\\right\\| _{2}<\\left\\| \\bz_{j}-0\\right\\| _{2}=\\br_{j}\\ \\forall\\ j=1,\\dots,\\bk.\n    \\end{equation*}\n    Thus, \\eqref{eq:margin_sq} is strictly positive, which implies the argument of the square-root in \\eqref{eq:margin} is strictly positive. Next, observe that\n    \\begin{align*}\n    r_{j}^{2}-\\left\\| z_{j}-c\\spec\\right\\| _{2}^{2} & =-\\left\\| c\\spec\\right\\| _{2}^{2}+\\underbrace{2z_{j}^{T}c\\spec-\\left\\| z_{j}\\right\\| _{2}^{2}+r_{j}^{2}}_{(\\star)_j} \\ \\forall\\  j=1,\\dots,\\bk.\n    \\end{align*}\n    Since $c\\spec{\\in}\\A\\spec$, the term $(\\star)_j$ is equal for all $j$. Therefore, \\eqref{eq:margin_sq} is $j$-independent, and $\\rho\\spec$ in \\eqref{eq:margin} is well-defined. The smoothness of $\\rho\\spec$  with respect to $\\spec$ then follows directly from \\eqref{eq:margin} and the smoothness of the core (Lemma \\ref{lem:core}).\n    \\end{proof}","sha256_utf8":"4c9ad765b2ca4533a42ebc82b379c3987fdd69b31aa4939fcde6d0f103b13359"}]},{"id":"S087","semantic_type":"proof_step","kind":"proof_step","title":"基准投影距离保证正余量","source_units":[{"start":105399,"end":106229,"start_line":1005,"end_line":1014,"raw_tex":"    It suffices to show that the square of the margin,\\begin{equation}\n    r_{j}^{2}-\\left\\| z_{j}-c\\spec\\right\\| _{2}^{2}, \\label{eq:margin_sq}\n    \\end{equation}\n    does not depend on the index $j$, is strictly-positive, and varies smoothly with $\\param$. First, consider the core and central affine subspace exactly at base ball-specifications $\\bspec$. By Lemma\n    \\ref{lem:linA}, since $0 \\in \\A\\bspec$ and $\\projA{\\bspec}\\left(\\bz_{j}\\right){=}c\\bspec{\\neq}0$\n    for all $j$, the definition of a projection gives\n    \\begin{equation*}\n    \\left\\| \\bz_{j}-c\\bspec\\right\\| _{2}<\\left\\| \\bz_{j}-0\\right\\| _{2}=\\br_{j}\\ \\forall\\ j=1,\\dots,\\bk.\n    \\end{equation*}\n    Thus, \\eqref{eq:margin_sq} is strictly positive, which implies the argument of the square-root in \\eqref{eq:margin} is strictly positive. Next, observe that\n","sha256_utf8":"e52b8e2d66e1fe5d30cfb252f8cc2afcc84aa16d4b5fb1bcfa5e6b59560bd036"}]},{"id":"S088","semantic_type":"proof_step","kind":"proof_step","title":"余量与索引无关且光滑","source_units":[{"start":106229,"end":106799,"start_line":1015,"end_line":1018,"raw_tex":"    \\begin{align*}\n    r_{j}^{2}-\\left\\| z_{j}-c\\spec\\right\\| _{2}^{2} & =-\\left\\| c\\spec\\right\\| _{2}^{2}+\\underbrace{2z_{j}^{T}c\\spec-\\left\\| z_{j}\\right\\| _{2}^{2}+r_{j}^{2}}_{(\\star)_j} \\ \\forall\\  j=1,\\dots,\\bk.\n    \\end{align*}\n    Since $c\\spec{\\in}\\A\\spec$, the term $(\\star)_j$ is equal for all $j$. Therefore, \\eqref{eq:margin_sq} is $j$-independent, and $\\rho\\spec$ in \\eqref{eq:margin} is well-defined. The smoothness of $\\rho\\spec$  with respect to $\\spec$ then follows directly from \\eqref{eq:margin} and the smoothness of the core (Lemma \\ref{lem:core}).\n","sha256_utf8":"8da6a4972a60df013e044d1fe3b40cc1d927d4cedb21ad7afb5995234fd67437"}]},{"id":"S089","semantic_type":"lemma","kind":"lemma","title":"可行域的球面表示","source_units":[{"start":106824,"end":107179,"start_line":1021,"end_line":1026,"raw_tex":"\\begin{lem}\n    \\label{lem:B}\\textbf{(Local Properties of the Feasible Region)} For ball-specifications\n    $\\spec$ near some base ball-specifications $\\bspec$, the feasible region is nonempty and satisfies\\begin{equation}\n    \\B\\spec=\\left\\{ x\\in\\A\\spec:\\ \\left\\| x-c\\spec\\right\\| _{2}=\\rho\\spec\\right\\}.\\label{eq:Bdef2} \n    \\end{equation}\n    \\end{lem}","sha256_utf8":"455ab9c3ed90be5270c418f3928759ba51e8f80a92d621784aa502d7f1ad210e"}]},{"id":"S090","semantic_type":"proof","kind":"proof","title":"可行域表示完整证明","source_units":[{"start":107184,"end":109351,"start_line":1027,"end_line":1057,"raw_tex":"\\begin{proof}\n    \\noindent\\textit{Equivalence of \\eqref{eq:MBP_cons} and \\eqref{eq:Bdef2}:}\n    Expanding and rearranging \\eqref{eq:MBP_cons} gives\n    \\begin{align*}\n    \\left\\| x-z_{j}\\right\\| _{2}=r_{j}\\ \\forall\\ j & \\iff\\left\\| x\\right\\| _{2}^{2}=2z_{j}^{T}x-\\left\\| z_{j}\\right\\| _{2}^{2}+r_{j}^{2}\\ \\forall\\ j\\\\\n    & \\implies2z_{j}^{T}x-\\left\\| z_{j}\\right\\| _{2}^{2}+r_{j}^{2}=2z_{i}^{T}x-\\left\\| z_{i}\\right\\| _{2}^{2}+r_{i}^{2}\\ \\forall\\ i\\neq j.\n    \\end{align*}\n    Hence, $\\B\\spec\\subseteq\\A\\spec$ implying\n    \\begin{equation*}\n    \\B\\spec=\\left\\{ x\\in\\A\\spec:\\ \\left\\| x-z_{j}\\right\\| _{2}=r_{j}\\text{  for all } j\\right\\}.\n    \\end{equation*}\n    Since $c\\left(r,Z\\right){=}\\projA{\\spec}\\left(z_{i}\\right)$, for any $x{\\in}\\A\\spec$, the vectors $\\left(x{-}c\\left(r,Z\\right)\\right)$ and $\\left(z_{j}{-}c\\spec\\right)$ are orthogonal. Then, applying the Pythagorean Theorem to the equality condition in \\eqref{eq:MBP_cons} gives\n    \\begin{equation*}\n    r_{j}^{2} = \\left\\| x-z_i\\right\\|_{2}^{2} =\\left\\| x-c\\left(r,Z\\right)\\right\\| _{2}^{2}+\\left\\| z_{j}-c\\spec\\right\\|_{2}^{2} \\ \\forall \\ x\\in\\A\\spec \\text{ and } j=1,\\dots,k.\n    \\end{equation*}\n    We then deduce \\eqref{eq:Bdef2} by subtracting $\\left\\| z_{j}{-}c\\spec\\right\\|_{2}^{2}$ from the left and right sides, and applying the definition of $\\rho\\spec$ in \\eqref{eq:margin}.\n    \n    \\noindent \\textit{Non-Emptiness of $\\B\\spec$:} By the first fact of Lemma \\ref{lem:A}, the affine subspace $\\A\\spec$ is $(n{-}m{+}1)$-dimensional, so $\\A\\spec$ is a non-empty space. Thus, there exists $a\\in\\A\\spec$ such that $a\\neq c\\spec$. Choose any such $a$ and consider the direction \n    \\begin{equation*}\n    d = \\frac{1}{\\left\\| a-c\\spec\\right\\|} \\left(a-c\\spec\\right)\n    \\end{equation*}\n    along with the vector\n    \\begin{equation*}\n    v = c\\spec+\\rho\\spec d.\n    \\end{equation*}\n    Then, by construction,\n    \\begin{equation*}\n    \\left\\| v-c\\spec\\right\\| _{2}=\\rho\\spec.\n    \\end{equation*}\n    Since $\\A\\spec$ contains both $c\\spec$ and $d$, we deduce $v\\in\\A\\spec$. Thus, $v$ satisfies all conditions in \\eqref{eq:Bdef2}, so $v\\in\\B\\spec$, and hence $\\B\\spec$ is non-empty.\n    \\end{proof}","sha256_utf8":"8caa0c52ace9a04fb9cde23d31b1f4f2f8978470ff5c244cf2fabc439764ee32"}]},{"id":"S091","semantic_type":"proof_step","kind":"proof_step","title":"球面交包含于中心空间","source_units":[{"start":107198,"end":107844,"start_line":1028,"end_line":1037,"raw_tex":"    \\noindent\\textit{Equivalence of \\eqref{eq:MBP_cons} and \\eqref{eq:Bdef2}:}\n    Expanding and rearranging \\eqref{eq:MBP_cons} gives\n    \\begin{align*}\n    \\left\\| x-z_{j}\\right\\| _{2}=r_{j}\\ \\forall\\ j & \\iff\\left\\| x\\right\\| _{2}^{2}=2z_{j}^{T}x-\\left\\| z_{j}\\right\\| _{2}^{2}+r_{j}^{2}\\ \\forall\\ j\\\\\n    & \\implies2z_{j}^{T}x-\\left\\| z_{j}\\right\\| _{2}^{2}+r_{j}^{2}=2z_{i}^{T}x-\\left\\| z_{i}\\right\\| _{2}^{2}+r_{i}^{2}\\ \\forall\\ i\\neq j.\n    \\end{align*}\n    Hence, $\\B\\spec\\subseteq\\A\\spec$ implying\n    \\begin{equation*}\n    \\B\\spec=\\left\\{ x\\in\\A\\spec:\\ \\left\\| x-z_{j}\\right\\| _{2}=r_{j}\\text{  for all } j\\right\\}.\n    \\end{equation*}\n","sha256_utf8":"f3ba644914ace053d2ee2263e89f41c3438ad875919c35f9df02937f0f996fb3"}]},{"id":"S092","semantic_type":"proof_step","kind":"proof_step","title":"勾股分解得到核心球面","source_units":[{"start":107844,"end":108536,"start_line":1038,"end_line":1042,"raw_tex":"    Since $c\\left(r,Z\\right){=}\\projA{\\spec}\\left(z_{i}\\right)$, for any $x{\\in}\\A\\spec$, the vectors $\\left(x{-}c\\left(r,Z\\right)\\right)$ and $\\left(z_{j}{-}c\\spec\\right)$ are orthogonal. Then, applying the Pythagorean Theorem to the equality condition in \\eqref{eq:MBP_cons} gives\n    \\begin{equation*}\n    r_{j}^{2} = \\left\\| x-z_i\\right\\|_{2}^{2} =\\left\\| x-c\\left(r,Z\\right)\\right\\| _{2}^{2}+\\left\\| z_{j}-c\\spec\\right\\|_{2}^{2} \\ \\forall \\ x\\in\\A\\spec \\text{ and } j=1,\\dots,k.\n    \\end{equation*}\n    We then deduce \\eqref{eq:Bdef2} by subtracting $\\left\\| z_{j}{-}c\\spec\\right\\|_{2}^{2}$ from the left and right sides, and applying the definition of $\\rho\\spec$ in \\eqref{eq:margin}.\n","sha256_utf8":"5f253ced8879ba5f434178d31958528e9c37eafca22f3953edf69801b65c4fbc"}]},{"id":"S093","semantic_type":"proof_step","kind":"proof_step","title":"构造球面上的可行点","source_units":[{"start":108541,"end":109336,"start_line":1044,"end_line":1056,"raw_tex":"    \\noindent \\textit{Non-Emptiness of $\\B\\spec$:} By the first fact of Lemma \\ref{lem:A}, the affine subspace $\\A\\spec$ is $(n{-}m{+}1)$-dimensional, so $\\A\\spec$ is a non-empty space. Thus, there exists $a\\in\\A\\spec$ such that $a\\neq c\\spec$. Choose any such $a$ and consider the direction \n    \\begin{equation*}\n    d = \\frac{1}{\\left\\| a-c\\spec\\right\\|} \\left(a-c\\spec\\right)\n    \\end{equation*}\n    along with the vector\n    \\begin{equation*}\n    v = c\\spec+\\rho\\spec d.\n    \\end{equation*}\n    Then, by construction,\n    \\begin{equation*}\n    \\left\\| v-c\\spec\\right\\| _{2}=\\rho\\spec.\n    \\end{equation*}\n    Since $\\A\\spec$ contains both $c\\spec$ and $d$, we deduce $v\\in\\A\\spec$. Thus, $v$ satisfies all conditions in \\eqref{eq:Bdef2}, so $v\\in\\B\\spec$, and hence $\\B\\spec$ is non-empty.\n","sha256_utf8":"b2da66c53ec6e8069748d5b2c014e2855ed3135990546eae56bc5e808323afca"}]},{"id":"S094","semantic_type":"theorem","kind":"theorem","title":"多球问题局部显式解","source_units":[{"start":109361,"end":109936,"start_line":1059,"end_line":1067,"raw_tex":"\\begin{thm}\\label{thm:MBP_sol}\n    \\textbf{(Local, Explicit Solution of $\\boldsymbol{MBP\\param}$)} For parameters $\\param$ near some base parameters $\\bparam$, the optimal solution to $\\MBP\\param$ is unique, varies smoothly with $\\param$, and is given by\\begin{equation}\\label{eq:x}\n    x\\param = c\\spec+\\rho\\spec u\\param,\n    \\end{equation}\n    where\\begin{equation}\\label{eq:u}\n    u\\param = \\frac{1}{\\left\\| \\projA{\\spec}\\left(y\\right)-c\\spec\\right\\| _{2}}\\left(\\projA{\\spec}\\left(y\\right)-c\\spec\\right).\n    \\end{equation}\n    In particular, $x\\bparam = 0$.\n    \\end{thm}","sha256_utf8":"0e0b69723f4d0443020ec9a3dddad4b3555d08dfb2b3ab7cd7b311452f2bbecb"}]},{"id":"S095","semantic_type":"proof","kind":"proof","title":"多球显式解完整证明","source_units":[{"start":109941,"end":112935,"start_line":1068,"end_line":1098,"raw_tex":"\\begin{proof}\n    \\textit{The vector $u\\param$ is well-defined:} To show the well-definedness of the expression in \\eqref{eq:u}, observe the following at base parameters $\\bparam$:\n    \n    \\begin{align*}\n    \\projA{\\bspec}\\left(\\by\\right)-c\\bspec &= \\projA{\\bspec}\\left(-\\sum_{j=1}^{\\bk}\\clam_{j}\\bar{z}_{j}\\right)-c\\bspec &\\text{ (By Definition \\ref{def:base})}\\\\\n    & = -\\sum_{j=1}^{\\bk}\\clam_{j}\\projA{\\bspec}\\left(\\bar{z}_{j}\\right)-c\\bspec &\\text{ (By linearity of $\\A\\bspec$ from Lemma \\ref{lem:linA})}\\\\\n    & = -\\left(\\sum_{j=1}^{\\bk}\\clam_{j}+1\\right)c\\bspec &\\text{ (By definition of core \\eqref{eq:core})}\\\\\n    & \\neq 0. &\\text{ (By Definition \\ref{def:base} and Lemma \\ref{lem:linA})}.\n    \\end{align*}\n    By the smoothness of the projection (third fact of Lemma \\ref{lem:A}), we deduce $\\projA{\\spec}\\left(y\\right){-}c\\spec{\\neq}0$ for $\\param$ sufficiently close to $\\bparam$ as well. Thus, \\eqref{eq:u} is well-defined. Moreover, the smoothness of the projection and \\eqref{eq:u} further shows that $u\\param$ is smooth with respect to $\\param$.\n    \n    \\noindent \\textit{Derivation of $x\\param$:} By Lemma \\ref{lem:B}, $\\B\\spec\\subseteq\\A\\spec$ so\\begin{align*}\n    x\\param=\\projB{\\spec}\\left(y\\right) & =\\projB{\\spec}\\left(\\projA{\\spec}\\left(y\\right)\\right).\n    \\end{align*}\n    Lemma \\ref{lem:B} also states that $\\B\\spec$ is simply the $c\\spec$-centered ball with radius $\\rho\\spec$ restricted to $\\A\\spec$. Thus, $\\projB{\\spec}\\left(\\projA{\\spec}\\left(y\\right)\\right)$ must be the (necessarily unique) point that\n    \\begin{enumerate}\n        \\item lies on the straight line from $c\\spec$ to $\\projA{\\spec}\\left(y\\right)$, and\n        \\item is exactly the distance $\\rho\\spec$ from $c\\spec$.\n    \\end{enumerate}\n    Formalizing the above two criteria leads to the explicit-form\n    \\begin{equation*}\n    x\\param=c\\spec+\\frac{\\rho\\spec}{\\left\\| \\projA{\\spec}\\left(y\\right)-c\\spec\\right\\| _{2}}\\left(\\projA{\\spec}\\left(y\\right)-c\\spec\\right)=c\\spec+\\rho\\spec u\\param.\n    \\end{equation*}\n    The smoothness of $x\\param$ follows directly from the smoothness of the core (Lemma \\ref{lem:core}), the margin (Lemma \\ref{lem:margin}), and $u\\param$ (shown above).\n    \n    \\noindent \\textit{Demonstration of $x\\bparam{=}0$:} Observe that the Lagrangian of $\\MBP\\bparam$ is\n    \\[\n    \\L(x,\\lambda) = \\left\\| x-\\by\\right\\| _{2}^{2} + \\sum_{j=1}^\\bk \\lambda_j \\left\\| x-\\bz_{j}\\right\\| _{2}^{2},\n    \\]\n    where $\\lambda_j$ is the $j$-th element of a Lagrange multipliers vector $\\lambda \\in \\R^\\bk$. By Definitions \\ref{def:MBP}-\\ref{def:base}, the zero-vector is a feasible solution to $\\MBP\\bparam$. A feasible vector $x^*$ is the solution of $\\MBP\\bparam$ if and only if there exists $\\lambda^*$ such that $x^*$ minimizes $\\L(\\cdot,\\lambda^*)$. It is easy to check using Definition \\ref{def:base} and first-order optimality conditions that $x^*{=}0$ is the unique minimizer of $\\L(\\cdot,\\clam)$. Thus, $x\\bparam{=}0$ is the unique solution to $\\MBP\\bparam$.\n    \\end{proof}","sha256_utf8":"8a65ce290948216b13c3db7ee7552a09537453f46dc8d4c8be7117125f6a32d0"}]},{"id":"S096","semantic_type":"proof_step","kind":"proof_step","title":"归一化方向非零且光滑","source_units":[{"start":109955,"end":111005,"start_line":1069,"end_line":1077,"raw_tex":"    \\textit{The vector $u\\param$ is well-defined:} To show the well-definedness of the expression in \\eqref{eq:u}, observe the following at base parameters $\\bparam$:\n    \n    \\begin{align*}\n    \\projA{\\bspec}\\left(\\by\\right)-c\\bspec &= \\projA{\\bspec}\\left(-\\sum_{j=1}^{\\bk}\\clam_{j}\\bar{z}_{j}\\right)-c\\bspec &\\text{ (By Definition \\ref{def:base})}\\\\\n    & = -\\sum_{j=1}^{\\bk}\\clam_{j}\\projA{\\bspec}\\left(\\bar{z}_{j}\\right)-c\\bspec &\\text{ (By linearity of $\\A\\bspec$ from Lemma \\ref{lem:linA})}\\\\\n    & = -\\left(\\sum_{j=1}^{\\bk}\\clam_{j}+1\\right)c\\bspec &\\text{ (By definition of core \\eqref{eq:core})}\\\\\n    & \\neq 0. &\\text{ (By Definition \\ref{def:base} and Lemma \\ref{lem:linA})}.\n    \\end{align*}\n    By the smoothness of the projection (third fact of Lemma \\ref{lem:A}), we deduce $\\projA{\\spec}\\left(y\\right){-}c\\spec{\\neq}0$ for $\\param$ sufficiently close to $\\bparam$ as well. Thus, \\eqref{eq:u} is well-defined. Moreover, the smoothness of the projection and \\eqref{eq:u} further shows that $u\\param$ is smooth with respect to $\\param$.\n","sha256_utf8":"951aa36aa1e9e1f307ccc88851ee4dc99223f487ef81a24c95f77207fa79da91"}]},{"id":"S097","semantic_type":"proof_step","kind":"proof_step","title":"球面投影导出唯一显式解","source_units":[{"start":111010,"end":112123,"start_line":1079,"end_line":1091,"raw_tex":"    \\noindent \\textit{Derivation of $x\\param$:} By Lemma \\ref{lem:B}, $\\B\\spec\\subseteq\\A\\spec$ so\\begin{align*}\n    x\\param=\\projB{\\spec}\\left(y\\right) & =\\projB{\\spec}\\left(\\projA{\\spec}\\left(y\\right)\\right).\n    \\end{align*}\n    Lemma \\ref{lem:B} also states that $\\B\\spec$ is simply the $c\\spec$-centered ball with radius $\\rho\\spec$ restricted to $\\A\\spec$. Thus, $\\projB{\\spec}\\left(\\projA{\\spec}\\left(y\\right)\\right)$ must be the (necessarily unique) point that\n    \\begin{enumerate}\n        \\item lies on the straight line from $c\\spec$ to $\\projA{\\spec}\\left(y\\right)$, and\n        \\item is exactly the distance $\\rho\\spec$ from $c\\spec$.\n    \\end{enumerate}\n    Formalizing the above two criteria leads to the explicit-form\n    \\begin{equation*}\n    x\\param=c\\spec+\\frac{\\rho\\spec}{\\left\\| \\projA{\\spec}\\left(y\\right)-c\\spec\\right\\| _{2}}\\left(\\projA{\\spec}\\left(y\\right)-c\\spec\\right)=c\\spec+\\rho\\spec u\\param.\n    \\end{equation*}\n    The smoothness of $x\\param$ follows directly from the smoothness of the core (Lemma \\ref{lem:core}), the margin (Lemma \\ref{lem:margin}), and $u\\param$ (shown above).\n","sha256_utf8":"acf971e1c12a00d291ee866bf5f9e0ca4bdf2a31509c54d3baec43cac9aa05bd"}]},{"id":"S098","semantic_type":"proof_step","kind":"proof_step","title":"基准Lagrangian识别零解","source_units":[{"start":112128,"end":112920,"start_line":1093,"end_line":1097,"raw_tex":"    \\noindent \\textit{Demonstration of $x\\bparam{=}0$:} Observe that the Lagrangian of $\\MBP\\bparam$ is\n    \\[\n    \\L(x,\\lambda) = \\left\\| x-\\by\\right\\| _{2}^{2} + \\sum_{j=1}^\\bk \\lambda_j \\left\\| x-\\bz_{j}\\right\\| _{2}^{2},\n    \\]\n    where $\\lambda_j$ is the $j$-th element of a Lagrange multipliers vector $\\lambda \\in \\R^\\bk$. By Definitions \\ref{def:MBP}-\\ref{def:base}, the zero-vector is a feasible solution to $\\MBP\\bparam$. A feasible vector $x^*$ is the solution of $\\MBP\\bparam$ if and only if there exists $\\lambda^*$ such that $x^*$ minimizes $\\L(\\cdot,\\lambda^*)$. It is easy to check using Definition \\ref{def:base} and first-order optimality conditions that $x^*{=}0$ is the unique minimizer of $\\L(\\cdot,\\clam)$. Thus, $x\\bparam{=}0$ is the unique solution to $\\MBP\\bparam$.\n","sha256_utf8":"faefe9a8ad3b0926d5a664380dc0f8338aa7d2f3105aa83cd0cbe72509d74a55"}]},{"id":"S099","semantic_type":"lemma","kind":"lemma","title":"多球乘子唯一正且光滑","source_units":[{"start":114142,"end":114781,"start_line":1109,"end_line":1115,"raw_tex":"\\begin{lem}\\label{lem:MBP_lam}\\textbf{(Properties of MBP Lagrange Multipliers)} Suppose the parameters $\\param$ are sufficiently close to base parameters $\\bparam$ with corresponding base weights vector $\\clam$. Consider the optimal solution $x\\param$ of $\\MBP\\param$. Then, the following hold:\n    \\begin{enumerate}\n        \\item There exists a unique Lagrange multiplier vector, $\\lambda\\param\\in\\R_{++}^{\\bk}$, satisfying the stationarity condition \\eqref{eq:MBP_FOC}.\n        \\item The vector $\\lambda\\param$ varies smoothly with $\\param$.\n        \\item At the base parameters, $\\lambda\\bparam=\\clam$.\n    \\end{enumerate}\n    \\end{lem}","sha256_utf8":"b5d3e0496a3031488d3ad259a3ca5d59cafd76394945451faac64b1397b8c6e7"}]},{"id":"S100","semantic_type":"proof","kind":"proof","title":"多球乘子性质完整证明","source_units":[{"start":114786,"end":116620,"start_line":1116,"end_line":1138,"raw_tex":"\\begin{proof}\n    Since $x\\param$ is optimal for $\\MBP\\param$, there must exist a Lagrange multipliers vector $\\lambda\\param\\in\\R^{\\bk}$ satisfying \\eqref{eq:MBP_FOC}. We first show $\\lambda\\param$ is unique. Finding Lagrange multipliers satisfying \\eqref{eq:MBP_FOC} is equivalent to finding minimizers of the function\n    \\begin{equation*}\n    F\\left(\\lambda\\right)=\\left\\| x\\param-y+X\\lambda\\right\\| _{2}^{2},\n    \\end{equation*}\n    where we define the matrix\n    \\[X=\\begin{bmatrix}x\\param{-}z_{1}, \\dots, x\\param{-}z_{\\bk}\\end{bmatrix}\\in\\R^{n\\times\\bk}.\\]\n    Then, a vector $\\lambda$ minimizes $F$ if and only if it satisfies the below $\\bk\\times\\bk$ linear system:\n    \\begin{equation*}\n    X^{T}\\left(x\\param-y+X\\lambda\\right)=0\\iff X^{T}X\\lambda=X^{T}\\left(y-x\\param\\right).\n    \\end{equation*}\n    By Theorem \\ref{thm:MBP_sol}, for $\\param$ close to $\\bparam$, the MBP-solution $x\\param$ is close to zero. Therefore, columns of $X$ are close to $\\left\\{-\\bz_{j}\\right\\}_{j=1}^{\\bk}$, which are linearly independent (Definition \\ref{def:base}). This implies $X^{T}X$ is invertible, and the minimizer of $F$ is unique and explicitly given by \n    \\begin{equation*}\n    \\lambda\\param = \\argmin_\\lambda F\\left(\\lambda\\right)=\\left(X^{T}X\\right)^{-1}X^{T}\\left(y-x\\param\\right).\n    \\end{equation*}\n    The smoothness of $\\lambda\\param$ with respect to $\\param$ follows from the smoothness of $x\\param$ (Theorem \\ref{thm:MBP_sol}) and the smoothness of the above matrix operations. \n    \n    Finally, it follows directly from Definition \\ref{def:base} and Theorem \\ref{thm:MBP_sol} that\n    \\begin{equation*}\n    x\\bparam-\\by+\\sum_{j=1}^{\\bk}\\clam_{j}\\left[x\\bparam-\\bz_{j}\\right]=0,\n    \\end{equation*}\n    which implies the Lagrange multiplier vector at the base parameters is uniquely $\\lambda\\bparam=\\clam$.\n    \\end{proof}","sha256_utf8":"41f403ee33f993f5610aded3895e63c5f469773ee4cc33829f9d058a749bd807"}]},{"id":"S101","semantic_type":"proof_step","kind":"proof_step","title":"驻点乘子转为最小二乘","source_units":[{"start":114800,"end":115592,"start_line":1117,"end_line":1126,"raw_tex":"    Since $x\\param$ is optimal for $\\MBP\\param$, there must exist a Lagrange multipliers vector $\\lambda\\param\\in\\R^{\\bk}$ satisfying \\eqref{eq:MBP_FOC}. We first show $\\lambda\\param$ is unique. Finding Lagrange multipliers satisfying \\eqref{eq:MBP_FOC} is equivalent to finding minimizers of the function\n    \\begin{equation*}\n    F\\left(\\lambda\\right)=\\left\\| x\\param-y+X\\lambda\\right\\| _{2}^{2},\n    \\end{equation*}\n    where we define the matrix\n    \\[X=\\begin{bmatrix}x\\param{-}z_{1}, \\dots, x\\param{-}z_{\\bk}\\end{bmatrix}\\in\\R^{n\\times\\bk}.\\]\n    Then, a vector $\\lambda$ minimizes $F$ if and only if it satisfies the below $\\bk\\times\\bk$ linear system:\n    \\begin{equation*}\n    X^{T}\\left(x\\param-y+X\\lambda\\right)=0\\iff X^{T}X\\lambda=X^{T}\\left(y-x\\param\\right).\n    \\end{equation*}\n","sha256_utf8":"1148dcee91df706a7f9fa0ca473e8fb719e754ee20ee0212396bd117a07b0c21"}]},{"id":"S102","semantic_type":"proof_step","kind":"proof_step","title":"独立约束给出唯一光滑乘子","source_units":[{"start":115592,"end":116276,"start_line":1127,"end_line":1131,"raw_tex":"    By Theorem \\ref{thm:MBP_sol}, for $\\param$ close to $\\bparam$, the MBP-solution $x\\param$ is close to zero. Therefore, columns of $X$ are close to $\\left\\{-\\bz_{j}\\right\\}_{j=1}^{\\bk}$, which are linearly independent (Definition \\ref{def:base}). This implies $X^{T}X$ is invertible, and the minimizer of $F$ is unique and explicitly given by \n    \\begin{equation*}\n    \\lambda\\param = \\argmin_\\lambda F\\left(\\lambda\\right)=\\left(X^{T}X\\right)^{-1}X^{T}\\left(y-x\\param\\right).\n    \\end{equation*}\n    The smoothness of $\\lambda\\param$ with respect to $\\param$ follows from the smoothness of $x\\param$ (Theorem \\ref{thm:MBP_sol}) and the smoothness of the above matrix operations. \n","sha256_utf8":"c687f536876abaa5eaac9c3b8ab02fd46c4f6d564b958c591937ca4f78ea0296"}]},{"id":"S103","semantic_type":"proof_step","kind":"proof_step","title":"基准乘子等于正权重","source_units":[{"start":116281,"end":116605,"start_line":1133,"end_line":1137,"raw_tex":"    Finally, it follows directly from Definition \\ref{def:base} and Theorem \\ref{thm:MBP_sol} that\n    \\begin{equation*}\n    x\\bparam-\\by+\\sum_{j=1}^{\\bk}\\clam_{j}\\left[x\\bparam-\\bz_{j}\\right]=0,\n    \\end{equation*}\n    which implies the Lagrange multiplier vector at the base parameters is uniquely $\\lambda\\bparam=\\clam$.\n","sha256_utf8":"eb29349a08a7fe26e77e44c7fefc2381485c0e1b1cf1df6690a21760c94ba29c"}]},{"id":"S104","semantic_type":"theorem","kind":"theorem","title":"不等式多球问题局部性质","source_units":[{"start":116630,"end":117569,"start_line":1140,"end_line":1151,"raw_tex":"\\begin{thm}\\textbf{(Inequality-Constrained MBP)}\\label{thm:MBP_ineq}\n    Consider the following inequality-constrained MBP:\n    \\begin{align*}\n    %\\begin{aligned}\n    \\min_{x\\in\\R^{n}}\\ & \\left\\| x-y\\right\\| _{2}^{2}\\\\\n    \\mathrm{s.t.}\\ & \\left\\| x-z_{j}\\right\\| _{2}^{2} \\leq r_{j}^{2}\\ \\forall\\ j=1,\\dots,\\bk,\n    %\\end{aligned}%\\label{eq:MBP_ineqFull}\n    \\end{align*}\n    and denote it as $\\MBPl\\param$. For parameters $\\param$ sufficiently close to $\\bparam$, $\\MBPl\\param$ is feasible and possesses the same unique optimal solution, $x\\param$, and unique Lagrange multipliers, $\\lambda\\param$, as $\\MBP\\param$. Consequently, all constraints of $\\MBPl\\param$ are tight. Moreover, the optimal solution and multipliers satisfy\\begin{equation}\n    x\\param = \\mathrm{O}\\left(\\|\\param-\\bparam\\|\\right)\\text{ and } \\lambda\\param = \\clam + \\mathrm{O}\\left(\\|\\param-\\bparam\\|\\right).\\label{eq:MBP_Onotation}\n    \\end{equation}\n    \\end{thm}","sha256_utf8":"438725bd7f9f272a4abb311cd16054a60bdd3652391578b820fe940b0df053eb"}]},{"id":"S105","semantic_type":"proof","kind":"proof","title":"不等式多球问题完整证明","source_units":[{"start":117574,"end":119527,"start_line":1152,"end_line":1163,"raw_tex":"\\begin{proof}\n    The feasibility of $\\MBPl\\param$ follows directly from Lemma \\ref{lem:B} since $\\B\\param$ is a subset of the feasible region of $\\MBPl\\param$.\n    \n    $\\MBPl\\param$ possesses a unique minimizer since it minimizes a strongly convex objective on a convex feasible region. Moreover, checking the KKT conditions is sufficient for demonstrating optimality because the $\\|x - z_j\\|_2^2$-term in the constraints is convex and differentiable for all $j$. Therefore---denoting the unique pair of optimal solutions and Lagrange multipliers of $\\MBP\\param$ as $\\left(x\\param, \\lambda\\param\\right)$---it suffices to check that $\\left(x\\param, \\lambda\\param\\right)$ satisfies the KKT conditions of $\\MBPl\\param$:\n    \\begin{enumerate}\n        \\item \\textbf{(Stationarity)} Follows from $\\MBPl\\param$ and $\\MBP\\param$ having the same stationarity condition \\eqref{eq:MBP_FOC}.\n        \\item \\textbf{(Primal Feasibility)} Follows from $\\B\\param$ being a subset of the feasible region of $\\MBPl\\param$.\n        \\item \\textbf{(Dual Feasibility)} The inequality constraints of $\\MBPl\\param$ require the non-negativity of all Lagrange multipliers. By Definition \\ref{def:base}, the elements of $\\clam$ are strictly-positive. Thus, by Lemma \\ref{lem:MBP_lam}, the elements of $\\lambda\\param$ are also strictly-positive for $\\param$ sufficiently close to $\\bparam$.\n        \\item \\textbf{(Complementary Slackness)} \\sloppy By construction, $x\\param$ satisfies all the inequality constraints of $\\MBPl\\param$ with equality. Correspondingly, $\\lambda\\param$ are all strictly positive.\n    \\end{enumerate}\n    Therefore, $\\left(x\\param, \\lambda\\param\\right)$ is the unique pair of optimal solutions and Lagrange multipliers for $\\MBPl\\param$ as well. The characterizations in \\eqref{eq:MBP_Onotation} directly follows from Theorem \\ref{thm:MBP_sol} and Lemma \\ref{lem:MBP_lam} as well as recalling that smooth functions are locally Lipschitz.\n    \\end{proof}","sha256_utf8":"a82a9bea8702908715d073b4bde281e0ad805f373703a985351d1ffb08a86da5"}]},{"id":"S106","semantic_type":"proof_step","kind":"proof_step","title":"等式可行域保证可行性","source_units":[{"start":117588,"end":117735,"start_line":1153,"end_line":1153,"raw_tex":"    The feasibility of $\\MBPl\\param$ follows directly from Lemma \\ref{lem:B} since $\\B\\param$ is a subset of the feasible region of $\\MBPl\\param$.\n","sha256_utf8":"2c4fc461820ce51c97485bd746bcf805cb042de2983fdcc70e4e2b71c0919ddd"}]},{"id":"S107","semantic_type":"proof_step","kind":"proof_step","title":"验证凸问题全部KKT条件","source_units":[{"start":117740,"end":119175,"start_line":1155,"end_line":1161,"raw_tex":"    $\\MBPl\\param$ possesses a unique minimizer since it minimizes a strongly convex objective on a convex feasible region. Moreover, checking the KKT conditions is sufficient for demonstrating optimality because the $\\|x - z_j\\|_2^2$-term in the constraints is convex and differentiable for all $j$. Therefore---denoting the unique pair of optimal solutions and Lagrange multipliers of $\\MBP\\param$ as $\\left(x\\param, \\lambda\\param\\right)$---it suffices to check that $\\left(x\\param, \\lambda\\param\\right)$ satisfies the KKT conditions of $\\MBPl\\param$:\n    \\begin{enumerate}\n        \\item \\textbf{(Stationarity)} Follows from $\\MBPl\\param$ and $\\MBP\\param$ having the same stationarity condition \\eqref{eq:MBP_FOC}.\n        \\item \\textbf{(Primal Feasibility)} Follows from $\\B\\param$ being a subset of the feasible region of $\\MBPl\\param$.\n        \\item \\textbf{(Dual Feasibility)} The inequality constraints of $\\MBPl\\param$ require the non-negativity of all Lagrange multipliers. By Definition \\ref{def:base}, the elements of $\\clam$ are strictly-positive. Thus, by Lemma \\ref{lem:MBP_lam}, the elements of $\\lambda\\param$ are also strictly-positive for $\\param$ sufficiently close to $\\bparam$.\n        \\item \\textbf{(Complementary Slackness)} \\sloppy By construction, $x\\param$ satisfies all the inequality constraints of $\\MBPl\\param$ with equality. Correspondingly, $\\lambda\\param$ are all strictly positive.\n    \\end{enumerate}\n","sha256_utf8":"166f7ed4754332fee29d0fb89dcb909c97077680a4a3aa66f430cd02a6715a54"}]},{"id":"S108","semantic_type":"proof_step","kind":"proof_step","title":"唯一解乘子及局部扰动界","source_units":[{"start":119175,"end":119512,"start_line":1162,"end_line":1162,"raw_tex":"    Therefore, $\\left(x\\param, \\lambda\\param\\right)$ is the unique pair of optimal solutions and Lagrange multipliers for $\\MBPl\\param$ as well. The characterizations in \\eqref{eq:MBP_Onotation} directly follows from Theorem \\ref{thm:MBP_sol} and Lemma \\ref{lem:MBP_lam} as well as recalling that smooth functions are locally Lipschitz.\n","sha256_utf8":"0a24d59af1cb1b78e4367e7d55cb56af53e1271fce2221a4a150c5c89080eb09"}]},{"id":"S109","semantic_type":"external_fact","kind":"external_fact","title":"凸最大值次微分与重心","source_units":[],"declared_external":true,"declaration_text":["有限个可微凸函数的最大值，其凸次微分为活动分量梯度的凸包；在仿射独立情形，凸包相对内部的点具有唯一严格正重心坐标。"]},{"id":"S110","semantic_type":"external_fact","kind":"external_fact","title":"Taylor定理与局部光滑运算","source_units":[],"declared_external":true,"declaration_text":["C²函数满足二阶Taylor展开，其梯度满足一阶Taylor展开；连续可微映射局部Lipschitz，正数上的平方根为光滑函数。"]},{"id":"S111","semantic_type":"external_fact","kind":"external_fact","title":"正交投影与矩阵光滑性","source_units":[],"declared_external":true,"declaration_text":["有限维线性代数中的秩稳定性、正交分解、仿射子空间投影、勾股定理，以及可逆矩阵的逆对矩阵参数光滑依赖。"]},{"id":"S112","semantic_type":"external_fact","kind":"external_fact","title":"凸估计与KKT理论","source_units":[],"declared_external":true,"declaration_text":["凸函数的Jensen不等式、强凸函数的一阶下界、光滑函数的二次上界；约束梯度线性无关时的Lagrange乘子必要条件，以及凸可微规划的KKT充分条件。"]},{"id":"S113","semantic_type":"external_fact","kind":"external_fact","title":"经典投影梯度收缩估计","source_units":[],"declared_external":true,"declaration_text":["若g为L光滑且δ强凸，C为闭凸集，x*为g在C上的最小点，则对τ≥L及z=P_C(s−∇g(s)/τ)，有||z−x*||²≤(1−δ/τ)||s−x*||²及g(z)−g(x*)≤(τ/2)(1−δ/τ)||s−x*||²；对应文中beck2017定理10.29。"]},{"id":"S114","semantic_type":"external_fact","kind":"external_fact","title":"隐函数定理","source_units":[],"declared_external":true,"declaration_text":["连续可微方程组在解处关于未知量的Jacobian可逆时，附近解唯一并连续可微地依赖参数。"]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S006","target_id":"S017","kind":"supports"},{"source_id":"S007","target_id":"S017","kind":"supports"},{"source_id":"S007","target_id":"S021","kind":"supports"},{"source_id":"S008","target_id":"S013","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S013","kind":"supports"},{"source_id":"S109","target_id":"S013","kind":"supports"},{"source_id":"S010","target_id":"S015","kind":"supports"},{"source_id":"S011","target_id":"S015","kind":"supports"},{"source_id":"S011","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S003","kind":"supports"},{"source_id":"S013","target_id":"S020","kind":"supports"},{"source_id":"S014","target_id":"S003","kind":"supports"},{"source_id":"S015","target_id":"S032","kind":"supports"},{"source_id":"S015","target_id":"S034","kind":"supports"},{"source_id":"S017","target_id":"S028","kind":"supports"},{"source_id":"S018","target_id":"S050","kind":"supports"},{"source_id":"S018","target_id":"S055","kind":"supports"},{"source_id":"S018","target_id":"S061","kind":"supports"},{"source_id":"S019","target_id":"S033","kind":"supports"},{"source_id":"S019","target_id":"S043","kind":"supports"},{"source_id":"S019","target_id":"S044","kind":"supports"},{"source_id":"S020","target_id":"S064","kind":"supports"},{"source_id":"S020","target_id":"S005","kind":"supports"},{"source_id":"S021","target_id":"S071","kind":"supports"},{"source_id":"S022","target_id":"S003","kind":"supports"},{"source_id":"S023","target_id":"S091","kind":"supports"},{"source_id":"S023","target_id":"S098","kind":"supports"},{"source_id":"S024","target_id":"S033","kind":"supports"},{"source_id":"S025","target_id":"S029","kind":"supports"},{"source_id":"S025","target_id":"S075","kind":"supports"},{"source_id":"S025","target_id":"S083","kind":"supports"},{"source_id":"S025","target_id":"S096","kind":"supports"},{"source_id":"S025","target_id":"S098","kind":"supports"},{"source_id":"S026","target_id":"S074","kind":"supports"},{"source_id":"S026","target_id":"S080","kind":"supports"},{"source_id":"S026","target_id":"S083","kind":"supports"},{"source_id":"S026","target_id":"S087","kind":"supports"},{"source_id":"S026","target_id":"S088","kind":"supports"},{"source_id":"S026","target_id":"S092","kind":"supports"},{"source_id":"S027","target_id":"S101","kind":"supports"},{"source_id":"S027","target_id":"S107","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S009","target_id":"S029","kind":"supports"},{"source_id":"S013","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S033","kind":"supports"},{"source_id":"S029","target_id":"S034","kind":"supports"},{"source_id":"S017","target_id":"S034","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S034","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S114","target_id":"S032","kind":"supports"},{"source_id":"S112","target_id":"S032","kind":"supports"},{"source_id":"S110","target_id":"S033","kind":"supports"},{"source_id":"S104","target_id":"S034","kind":"supports"},{"source_id":"S011","target_id":"S037","kind":"supports"},{"source_id":"S030","target_id":"S038","kind":"supports"},{"source_id":"S017","target_id":"S038","kind":"supports"},{"source_id":"S010","target_id":"S039","kind":"supports"},{"source_id":"S112","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S039","target_id":"S036","kind":"supports"},{"source_id":"S040","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S009","target_id":"S044","kind":"supports"},{"source_id":"S110","target_id":"S043","kind":"supports"},{"source_id":"S110","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S044","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S017","target_id":"S047","kind":"supports"},{"source_id":"S030","target_id":"S047","kind":"supports"},{"source_id":"S112","target_id":"S047","kind":"supports"},{"source_id":"S030","target_id":"S048","kind":"supports"},{"source_id":"S013","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S013","target_id":"S049","kind":"supports"},{"source_id":"S019","target_id":"S049","kind":"supports"},{"source_id":"S041","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S013","target_id":"S050","kind":"supports"},{"source_id":"S111","target_id":"S050","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S049","target_id":"S046","kind":"supports"},{"source_id":"S050","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S030","target_id":"S053","kind":"supports"},{"source_id":"S017","target_id":"S053","kind":"supports"},{"source_id":"S019","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S030","target_id":"S054","kind":"supports"},{"source_id":"S041","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S009","target_id":"S055","kind":"supports"},{"source_id":"S111","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S055","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S045","target_id":"S058","kind":"supports"},{"source_id":"S051","target_id":"S058","kind":"supports"},{"source_id":"S111","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S013","target_id":"S061","kind":"supports"},{"source_id":"S010","target_id":"S061","kind":"supports"},{"source_id":"S112","target_id":"S061","kind":"supports"},{"source_id":"S113","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S056","target_id":"S064","kind":"supports"},{"source_id":"S013","target_id":"S064","kind":"supports"},{"source_id":"S059","target_id":"S064","kind":"supports"},{"source_id":"S112","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S062","kind":"supports"},{"source_id":"S062","target_id":"S067","kind":"supports"},{"source_id":"S056","target_id":"S067","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S012","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S066","kind":"supports"},{"source_id":"S068","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S071","kind":"supports"},{"source_id":"S030","target_id":"S071","kind":"supports"},{"source_id":"S035","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S070","kind":"supports"},{"source_id":"S070","target_id":"S069","kind":"supports"},{"source_id":"S070","target_id":"S022","kind":"supports"},{"source_id":"S065","target_id":"S003","kind":"supports"},{"source_id":"S069","target_id":"S004","kind":"supports"},{"source_id":"S011","target_id":"S004","kind":"supports"},{"source_id":"S009","target_id":"S004","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S076","kind":"supports"},{"source_id":"S075","target_id":"S077","kind":"supports"},{"source_id":"S111","target_id":"S075","kind":"supports"},{"source_id":"S111","target_id":"S076","kind":"supports"},{"source_id":"S111","target_id":"S077","kind":"supports"},{"source_id":"S074","target_id":"S073","kind":"supports"},{"source_id":"S075","target_id":"S073","kind":"supports"},{"source_id":"S076","target_id":"S073","kind":"supports"},{"source_id":"S077","target_id":"S073","kind":"supports"},{"source_id":"S073","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S080","kind":"supports"},{"source_id":"S111","target_id":"S080","kind":"supports"},{"source_id":"S080","target_id":"S079","kind":"supports"},{"source_id":"S079","target_id":"S078","kind":"supports"},{"source_id":"S083","target_id":"S084","kind":"supports"},{"source_id":"S025","target_id":"S084","kind":"supports"},{"source_id":"S072","target_id":"S084","kind":"supports"},{"source_id":"S078","target_id":"S084","kind":"supports"},{"source_id":"S111","target_id":"S084","kind":"supports"},{"source_id":"S083","target_id":"S082","kind":"supports"},{"source_id":"S084","target_id":"S082","kind":"supports"},{"source_id":"S082","target_id":"S081","kind":"supports"},{"source_id":"S081","target_id":"S087","kind":"supports"},{"source_id":"S025","target_id":"S087","kind":"supports"},{"source_id":"S111","target_id":"S087","kind":"supports"},{"source_id":"S087","target_id":"S088","kind":"supports"},{"source_id":"S078","target_id":"S088","kind":"supports"},{"source_id":"S110","target_id":"S088","kind":"supports"},{"source_id":"S087","target_id":"S086","kind":"supports"},{"source_id":"S088","target_id":"S086","kind":"supports"},{"source_id":"S086","target_id":"S085","kind":"supports"},{"source_id":"S026","target_id":"S091","kind":"supports"},{"source_id":"S091","target_id":"S092","kind":"supports"},{"source_id":"S078","target_id":"S092","kind":"supports"},{"source_id":"S085","target_id":"S092","kind":"supports"},{"source_id":"S111","target_id":"S092","kind":"supports"},{"source_id":"S072","target_id":"S093","kind":"supports"},{"source_id":"S085","target_id":"S093","kind":"supports"},{"source_id":"S092","target_id":"S093","kind":"supports"},{"source_id":"S091","target_id":"S090","kind":"supports"},{"source_id":"S092","target_id":"S090","kind":"supports"},{"source_id":"S093","target_id":"S090","kind":"supports"},{"source_id":"S090","target_id":"S089","kind":"supports"},{"source_id":"S081","target_id":"S096","kind":"supports"},{"source_id":"S078","target_id":"S096","kind":"supports"},{"source_id":"S072","target_id":"S096","kind":"supports"},{"source_id":"S096","target_id":"S097","kind":"supports"},{"source_id":"S089","target_id":"S097","kind":"supports"},{"source_id":"S078","target_id":"S097","kind":"supports"},{"source_id":"S085","target_id":"S097","kind":"supports"},{"source_id":"S111","target_id":"S097","kind":"supports"},{"source_id":"S112","target_id":"S098","kind":"supports"},{"source_id":"S096","target_id":"S095","kind":"supports"},{"source_id":"S097","target_id":"S095","kind":"supports"},{"source_id":"S098","target_id":"S095","kind":"supports"},{"source_id":"S095","target_id":"S094","kind":"supports"},{"source_id":"S112","target_id":"S101","kind":"supports"},{"source_id":"S101","target_id":"S102","kind":"supports"},{"source_id":"S094","target_id":"S102","kind":"supports"},{"source_id":"S025","target_id":"S102","kind":"supports"},{"source_id":"S111","target_id":"S102","kind":"supports"},{"source_id":"S094","target_id":"S103","kind":"supports"},{"source_id":"S025","target_id":"S103","kind":"supports"},{"source_id":"S102","target_id":"S103","kind":"supports"},{"source_id":"S101","target_id":"S100","kind":"supports"},{"source_id":"S102","target_id":"S100","kind":"supports"},{"source_id":"S103","target_id":"S100","kind":"supports"},{"source_id":"S100","target_id":"S099","kind":"supports"},{"source_id":"S089","target_id":"S106","kind":"supports"},{"source_id":"S106","target_id":"S107","kind":"supports"},{"source_id":"S094","target_id":"S107","kind":"supports"},{"source_id":"S099","target_id":"S107","kind":"supports"},{"source_id":"S025","target_id":"S107","kind":"supports"},{"source_id":"S112","target_id":"S107","kind":"supports"},{"source_id":"S107","target_id":"S108","kind":"supports"},{"source_id":"S094","target_id":"S108","kind":"supports"},{"source_id":"S099","target_id":"S108","kind":"supports"},{"source_id":"S110","target_id":"S108","kind":"supports"},{"source_id":"S106","target_id":"S105","kind":"supports"},{"source_id":"S107","target_id":"S105","kind":"supports"},{"source_id":"S108","target_id":"S105","kind":"supports"},{"source_id":"S105","target_id":"S104","kind":"supports"}]}}
{"record_id":"2112.01798v2","paper":{"title":"Convergence Properties of Monotone and Nonmonotone Proximal Gradient Methods Revisited","authors":["Christian Kanzow","Patrick Mehlitz"],"arxiv_id":"2112.01798","version":"v2","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-022-02101-3","abstract_url":"https://arxiv.org/abs/2112.01798v2","pdf_url":"https://arxiv.org/pdf/2112.01798v2"},"context":{"raw_tex":"We consider the \\emph{composite} optimization problem\n\\begin{equation}\\label{Eq:P}\\tag{P}\n   \\min_x \\ \\psi(x):=f(x) + \\phi (x), \\quad \\quad x \\in \\mathbb X,\n\\end{equation}\nwhere $ f\\colon \\mathbb{X} \\to \\mathbb{R} $ is\ncontinuously differentiable, $ \\phi\\colon \\mathbb{X} \\to \\overline{\\mathbb{R}}:= \\R\\cup\\{\\infty\\} $ \nis lower semicontinuous (possibly infinite-valued and nondifferentiable), \nand $ \\mathbb{X} $ denotes a\nEuclidean space, i.e., a real and finite-dimensional Hilbert space.\nWe assume that the domain $\\dom\\phi:=\\{x\\in\\mathbb X\\,|\\,\\phi(x)<\\infty\\}$ of $ \\phi $\nis nonempty to rule out trivial situations.\n\n\nThroughout the paper, the Euclidean space $\\mathbb X$ \nwill be equipped with the inner product $\\langle\\cdot,\\cdot\\rangle\\colon \\mathbb X\\times\\mathbb X\\to\\R$\nand the associated norm $\\norm{\\cdot}$.\nFor some set $A\\subset\\mathbb X$ and some point $x\\in\\mathbb X$, we make use of\n$A+x=x+A:=\\{x+a\\,|\\,a\\in A\\}$ for the purpose of simplicity.\nFor some sequence $\\{x^k\\}\\subset\\mathbb X$ and $x\\in\\mathbb X$, $x^k\\to_\\phi x$ means that\n$x^k\\to x$ and $\\phi(x^k)\\to\\phi(x)$.\nThe continuous linear operator\n$f'(x)\\colon\\mathbb X\\to\\R$ denotes the derivative of $f$ at $x\\in\\mathbb X$, and we will\nmake use of $\\nabla f(x):=f'(x)^*1$ where $f'(x)^*\\colon\\R\\to\\mathbb X$ is the adjoint of $f'(x)$.\nThis way, $\\nabla f$ is a mapping from $\\mathbb X$ to $\\mathbb X$.\nFurthermore, we find $f'(x)d=\\langle\\nabla f(x),d\\rangle$ for each $d\\in\\mathbb X$.\n \nThe following concepts are standard in variational analysis, see e.g.\\ \\cite{Mordukhovich2018,RockafellarWets2009}.\nLet us fix some point $x\\in\\dom\\phi$. Then\n\\[\n\t\\widehat\\partial\\phi(x)\n\t:=\n\t\\left\\{\n\t\t\\eta\\in\\mathbb X\\,\\middle|\\,\n\t\t\\liminf\\limits_{y\\to x}\\frac{\\phi(y)-\\phi(x)-\\langle \\eta,y-x\\rangle}{\\norm{y-x}}\\geq 0\n\t\\right\\}\n\\] \nis called the {\\em regular} (or {\\em Fr\\'{e}chet}) {\\em subdifferential} of $\\phi$ at $x$.\nFurthermore, the set\n\\[\n\t\\partial\\phi(x)\n\t:=\n\t\\left\\{\n\t\t\\eta\\in\\mathbb X\\,\\middle|\\,\n\t\t\\exists\\{x^k\\},\\{\\eta^k\\}\\subset\\mathbb X\\colon x^k\\to_\\phi x,\\,\\eta^k\\to\\eta,\\,\\eta^k\\in\\widehat{\\partial}\\phi(x^k)\\,\\forall k\\in\\N\n\t\\right\\}\n\\]\nis well known as the {\\em limiting} (or {\\em Mordukhovich}) \n{\\em subdifferential} of $\\phi$ at $x$.\nClearly, we always have $\\widehat{\\partial}\\phi(x)\\subset\\partial\\phi(x)$ by construction.\n\n\nIt can be seen right from the definition that whenever $x^*\\in\\dom\\phi$ is a local minimizer\nof $\\phi$, then $0\\in\\widehat\\partial\\phi(x^*)$, which is referred to as Fermat's rule,\nsee \\cite[Proposition~1.30(i)]{Mordukhovich2018}.\n\nGiven $ x \\in \\dom\\phi $, the limiting subdifferential has the important robustness\nproperty \n\\begin{equation}\\label{Eq:robustness}\n   \\left\\{ \n   \t\\eta\\in\\mathbb X\\,\\middle|\\,\n   \t\\exists\\{x^k\\},\\{\\eta^k\\}\\subset\\mathbb X\\colon\\,x^k\\to_\\phi x,\\,\\eta^k\\to\\eta,\\,\\eta^k\\in\\partial\\phi(x^k)\\,\\forall k\\in\\N\n   \\right\\}\n   \\subset\n   \\partial\\phi(x),\n\\end{equation}\n\nsee \\cite[Proposition~1.20]{Mordukhovich2018}. \nClearly, the converse inclusion $\\supset$ is also valid by definition of the limiting subdifferential.\n\n\nWhenever $x\\in\\dom\\phi$ is fixed, the sum rule\n\\begin{equation}\n\t\t\\label{eq:regular_sum_rule}\n\t\t\\widehat{\\partial}(f+\\phi)(x)\n\t\t=\n\t\t\\nabla f(x)+\\widehat{\\partial}\\phi(x)\n\\end{equation}\n\nholds, see \\cite[Proposition~1.30(ii)]{Mordukhovich2018}.\n\n\nThroughout the paper, a point $x^*\\in\\dom\\phi$ satisfying\n$0\\in \\nabla f (x^*)+\\partial\\phi(x^*)$ will be called a \n{\\em Mordukhovich-stationary} ({\\em M-stationary} for short) point of \n\\eqref{Eq:P} due to the appearance of the limiting subdifferential. In the literature, the name \n\\emph{limiting critical point} is used as well. We close this section with two special instances\nof problem \\eqref{Eq:P} and comment on the corresponding M-stationary conditions.\n\n\n\\begin{algorithm}[Monotone Proximal Gradient Method]\\leavevmode\n\t\\label{Alg:MonotoneProxGrad}\n\t\\begin{algorithmic}[1]\n\t\t\\REQUIRE $ \\tau > 1, 0 < \\gamma_{\\min} \\leq  \\gamma_{\\max} < \\infty, \\delta \\in (0,1), x^0 \\in \\dom\\phi $\n\t\t\\STATE Set $k := 0$.\n\t\t\\WHILE{A suitable termination criterion is violated at iteration $ k $}\n\t\t\\STATE Choose $ \\gamma_k^0 \\in [ \\gamma_{\\min}, \\gamma_{\\max}] $.\n\t\t\\STATE\\label{step:subproblem_solve_MonotoneProxGrad} \n\t\t\tFor $ i = 0, 1, 2, \\ldots $, compute a solution $ x^{k,i} $ of\n      \t\t\\begin{equation}\\label{Eq:Subki}\n         \t\t\\min_x \\ f (x^k) + \\langle\\nabla f(x^k), x - x^k \\rangle + \\frac{\\gamma_{k,i}}{2} \\| x - x^k \\|^2 + \\phi (x), \n         \t\t\\quad x \\in \\mathbb X\n      \t\t\\end{equation}\n      \t\twith $ \\gamma_{k,i} := \\tau^i \\gamma_k^0 $, until the acceptance criterion\n      \t\t\\begin{equation}\\label{Eq:StepCrit}\n         \t\t\\psi (x^{k,i}) \\leq \n         \t\t\\psi (x^k) - \\delta \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2 \n      \t\t\\end{equation}\n      \t\tholds.\n\t\t\\STATE Denote by $ i_k := i $ the terminal value, and set $ \\gamma_k := \n      \t\t\t\\gamma_{k,i_k} $ and $ x^{k+1} := x^{k,i_k} $.\n      \t\\STATE Set $ k \\leftarrow k + 1 $.\n\t\t\\ENDWHILE\n\t\t\\RETURN $x^k$\n\t\\end{algorithmic}\n\\end{algorithm}\n\n\\begin{assumption}\\label{Ass:ProxGradMonotone}\n\\begin{enumerate}\n   \\item \\label{item:psi_bounded} The function $ \\psi $ is bounded from below on $ \\dom\\phi $.\n   \\item \\label{item:phi_bounded_affine} The function  $ \\phi $ is bounded from below by an affine function.\n\\end{enumerate}\n\\end{assumption}\n\nThe subsequent convergence theory assumes implicitly that \\Cref{Alg:MonotoneProxGrad}\ngenerates an infinite sequence. \n","units":[{"start":16050,"end":16673,"start_line":381,"end_line":391,"raw_tex":"We consider the \\emph{composite} optimization problem\n\\begin{equation}\\label{Eq:P}\\tag{P}\n   \\min_x \\ \\psi(x):=f(x) + \\phi (x), \\quad \\quad x \\in \\mathbb X,\n\\end{equation}\nwhere $ f\\colon \\mathbb{X} \\to \\mathbb{R} $ is\ncontinuously differentiable, $ \\phi\\colon \\mathbb{X} \\to \\overline{\\mathbb{R}}:= \\R\\cup\\{\\infty\\} $ \nis lower semicontinuous (possibly infinite-valued and nondifferentiable), \nand $ \\mathbb{X} $ denotes a\nEuclidean space, i.e., a real and finite-dimensional Hilbert space.\nWe assume that the domain $\\dom\\phi:=\\{x\\in\\mathbb X\\,|\\,\\phi(x)<\\infty\\}$ of $ \\phi $\nis nonempty to rule out trivial situations.\n","sha256_utf8":"6619dda46c80b9a42d4c40215d5c4a33f680eb5dc94f79826c69fcc0bda35f9f"},{"start":17010,"end":18704,"start_line":397,"end_line":432,"raw_tex":"Throughout the paper, the Euclidean space $\\mathbb X$ \nwill be equipped with the inner product $\\langle\\cdot,\\cdot\\rangle\\colon \\mathbb X\\times\\mathbb X\\to\\R$\nand the associated norm $\\norm{\\cdot}$.\nFor some set $A\\subset\\mathbb X$ and some point $x\\in\\mathbb X$, we make use of\n$A+x=x+A:=\\{x+a\\,|\\,a\\in A\\}$ for the purpose of simplicity.\nFor some sequence $\\{x^k\\}\\subset\\mathbb X$ and $x\\in\\mathbb X$, $x^k\\to_\\phi x$ means that\n$x^k\\to x$ and $\\phi(x^k)\\to\\phi(x)$.\nThe continuous linear operator\n$f'(x)\\colon\\mathbb X\\to\\R$ denotes the derivative of $f$ at $x\\in\\mathbb X$, and we will\nmake use of $\\nabla f(x):=f'(x)^*1$ where $f'(x)^*\\colon\\R\\to\\mathbb X$ is the adjoint of $f'(x)$.\nThis way, $\\nabla f$ is a mapping from $\\mathbb X$ to $\\mathbb X$.\nFurthermore, we find $f'(x)d=\\langle\\nabla f(x),d\\rangle$ for each $d\\in\\mathbb X$.\n \nThe following concepts are standard in variational analysis, see e.g.\\ \\cite{Mordukhovich2018,RockafellarWets2009}.\nLet us fix some point $x\\in\\dom\\phi$. Then\n\\[\n\t\\widehat\\partial\\phi(x)\n\t:=\n\t\\left\\{\n\t\t\\eta\\in\\mathbb X\\,\\middle|\\,\n\t\t\\liminf\\limits_{y\\to x}\\frac{\\phi(y)-\\phi(x)-\\langle \\eta,y-x\\rangle}{\\norm{y-x}}\\geq 0\n\t\\right\\}\n\\] \nis called the {\\em regular} (or {\\em Fr\\'{e}chet}) {\\em subdifferential} of $\\phi$ at $x$.\nFurthermore, the set\n\\[\n\t\\partial\\phi(x)\n\t:=\n\t\\left\\{\n\t\t\\eta\\in\\mathbb X\\,\\middle|\\,\n\t\t\\exists\\{x^k\\},\\{\\eta^k\\}\\subset\\mathbb X\\colon x^k\\to_\\phi x,\\,\\eta^k\\to\\eta,\\,\\eta^k\\in\\widehat{\\partial}\\phi(x^k)\\,\\forall k\\in\\N\n\t\\right\\}\n\\]\nis well known as the {\\em limiting} (or {\\em Mordukhovich}) \n{\\em subdifferential} of $\\phi$ at $x$.\nClearly, we always have $\\widehat{\\partial}\\phi(x)\\subset\\partial\\phi(x)$ by construction.\n","sha256_utf8":"329be898bf52b317fe8d9993b3d42a6cd613d15e2d16906c4fd0fec1626a7441"},{"start":19016,"end":19610,"start_line":445,"end_line":458,"raw_tex":"It can be seen right from the definition that whenever $x^*\\in\\dom\\phi$ is a local minimizer\nof $\\phi$, then $0\\in\\widehat\\partial\\phi(x^*)$, which is referred to as Fermat's rule,\nsee \\cite[Proposition~1.30(i)]{Mordukhovich2018}.\n\nGiven $ x \\in \\dom\\phi $, the limiting subdifferential has the important robustness\nproperty \n\\begin{equation}\\label{Eq:robustness}\n   \\left\\{ \n   \t\\eta\\in\\mathbb X\\,\\middle|\\,\n   \t\\exists\\{x^k\\},\\{\\eta^k\\}\\subset\\mathbb X\\colon\\,x^k\\to_\\phi x,\\,\\eta^k\\to\\eta,\\,\\eta^k\\in\\partial\\phi(x^k)\\,\\forall k\\in\\N\n   \\right\\}\n   \\subset\n   \\partial\\phi(x),\n\\end{equation}","sha256_utf8":"e6b3ea1879d207bae99b65bbdd7d23bd66eab60bf85ca9e6f1a70417bb2ae948"},{"start":19611,"end":19762,"start_line":459,"end_line":460,"raw_tex":"see \\cite[Proposition~1.20]{Mordukhovich2018}. \nClearly, the converse inclusion $\\supset$ is also valid by definition of the limiting subdifferential.\n","sha256_utf8":"bebaa80fd7315daa4652ac293daedb28042f84e6cd9ae8241a7258e4b63f7ccd"},{"start":20383,"end":20567,"start_line":474,"end_line":480,"raw_tex":"Whenever $x\\in\\dom\\phi$ is fixed, the sum rule\n\\begin{equation}\n\t\t\\label{eq:regular_sum_rule}\n\t\t\\widehat{\\partial}(f+\\phi)(x)\n\t\t=\n\t\t\\nabla f(x)+\\widehat{\\partial}\\phi(x)\n\\end{equation}","sha256_utf8":"1af312283b215935659dac26ff6e8c24e9de0d1c08e11a551edd1ea8d871befe"},{"start":20568,"end":20626,"start_line":481,"end_line":481,"raw_tex":"holds, see \\cite[Proposition~1.30(ii)]{Mordukhovich2018}.\n","sha256_utf8":"47e4477042db2ea38c0a1379e475d48284a966d6859e3772470914453e814a55"},{"start":21208,"end":21671,"start_line":492,"end_line":497,"raw_tex":"Throughout the paper, a point $x^*\\in\\dom\\phi$ satisfying\n$0\\in \\nabla f (x^*)+\\partial\\phi(x^*)$ will be called a \n{\\em Mordukhovich-stationary} ({\\em M-stationary} for short) point of \n\\eqref{Eq:P} due to the appearance of the limiting subdifferential. In the literature, the name \n\\emph{limiting critical point} is used as well. We close this section with two special instances\nof problem \\eqref{Eq:P} and comment on the corresponding M-stationary conditions.\n","sha256_utf8":"742b126866cd3b52dea4bc27b66101cbc84cd22de20972e35c3a5d556c53ec61"},{"start":24720,"end":25961,"start_line":570,"end_line":595,"raw_tex":"\\begin{algorithm}[Monotone Proximal Gradient Method]\\leavevmode\n\t\\label{Alg:MonotoneProxGrad}\n\t\\begin{algorithmic}[1]\n\t\t\\REQUIRE $ \\tau > 1, 0 < \\gamma_{\\min} \\leq  \\gamma_{\\max} < \\infty, \\delta \\in (0,1), x^0 \\in \\dom\\phi $\n\t\t\\STATE Set $k := 0$.\n\t\t\\WHILE{A suitable termination criterion is violated at iteration $ k $}\n\t\t\\STATE Choose $ \\gamma_k^0 \\in [ \\gamma_{\\min}, \\gamma_{\\max}] $.\n\t\t\\STATE\\label{step:subproblem_solve_MonotoneProxGrad} \n\t\t\tFor $ i = 0, 1, 2, \\ldots $, compute a solution $ x^{k,i} $ of\n      \t\t\\begin{equation}\\label{Eq:Subki}\n         \t\t\\min_x \\ f (x^k) + \\langle\\nabla f(x^k), x - x^k \\rangle + \\frac{\\gamma_{k,i}}{2} \\| x - x^k \\|^2 + \\phi (x), \n         \t\t\\quad x \\in \\mathbb X\n      \t\t\\end{equation}\n      \t\twith $ \\gamma_{k,i} := \\tau^i \\gamma_k^0 $, until the acceptance criterion\n      \t\t\\begin{equation}\\label{Eq:StepCrit}\n         \t\t\\psi (x^{k,i}) \\leq \n         \t\t\\psi (x^k) - \\delta \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2 \n      \t\t\\end{equation}\n      \t\tholds.\n\t\t\\STATE Denote by $ i_k := i $ the terminal value, and set $ \\gamma_k := \n      \t\t\t\\gamma_{k,i_k} $ and $ x^{k+1} := x^{k,i_k} $.\n      \t\\STATE Set $ k \\leftarrow k + 1 $.\n\t\t\\ENDWHILE\n\t\t\\RETURN $x^k$\n\t\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"92313ca355e1634f4fd6862a92f8638934ded4ffcc312815829f2b8adae2bec3"},{"start":26025,"end":26326,"start_line":600,"end_line":605,"raw_tex":"\\begin{assumption}\\label{Ass:ProxGradMonotone}\n\\begin{enumerate}\n   \\item \\label{item:psi_bounded} The function $ \\psi $ is bounded from below on $ \\dom\\phi $.\n   \\item \\label{item:phi_bounded_affine} The function  $ \\phi $ is bounded from below by an affine function.\n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"7632b8834e3c1780b165a44b205492faf728c7df6cfafba92b4fcdb47ad63b5a"},{"start":26886,"end":27005,"start_line":614,"end_line":615,"raw_tex":"The subsequent convergence theory assumes implicitly that \\Cref{Alg:MonotoneProxGrad}\ngenerates an infinite sequence. \n","sha256_utf8":"53b7daa50b86720ae93ae70c0dff099c07ece1adce0bd5163d0c52e5c3354f99"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{Thm:ConvProxGrad}\nAssume that \\Cref{Ass:ProxGradMonotone} holds while either $ \\phi $ is continuous\non $\\dom\\phi$ or $ \\nabla f\\colon\\mathbb X\\to\\mathbb X $ is locally Lipschitz continuous.\nThen each accumulation point\n$ x^* $ of a sequence $ \\{ x^k \\} $ generated by \\Cref{Alg:MonotoneProxGrad}\nis an M-stationary point of \\eqref{Eq:P}.\n\\end{theorem}","units":[{"start":40798,"end":41171,"start_line":909,"end_line":915,"raw_tex":"\\begin{theorem}\\label{Thm:ConvProxGrad}\nAssume that \\Cref{Ass:ProxGradMonotone} holds while either $ \\phi $ is continuous\non $\\dom\\phi$ or $ \\nabla f\\colon\\mathbb X\\to\\mathbb X $ is locally Lipschitz continuous.\nThen each accumulation point\n$ x^* $ of a sequence $ \\{ x^k \\} $ generated by \\Cref{Alg:MonotoneProxGrad}\nis an M-stationary point of \\eqref{Eq:P}.\n\\end{theorem}","sha256_utf8":"9f71e373f83213c321aed1f8a4b09305a2b6acd76b6b6a82a8a4066d3678dc32"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\Cref{Ass:ProxGradMonotone}~\\ref{item:psi_bounded} is a reasonable condition regarding the\ngiven composite optimization problem, whereas \\Cref{Ass:ProxGradMonotone}~\\ref{item:phi_bounded_affine}\nis essentially a statement relevant for the subproblems from \\eqref{Eq:Subki}. In particular,\n\\Cref{Ass:ProxGradMonotone}~\\ref{item:phi_bounded_affine} implies that the quadratic objective function\nof the subproblems \\eqref{Eq:Subki} are, for fixed $ k, i \\in\\N$, coercive, and therefore always\nattain a solution $ x^{k,i} $ (which, however, may not be unique).\n\n\n\\begin{lemma}\\label{Lem:StepsizeFinite}\nConsider a fixed iteration $ k $ of \\cref{Alg:MonotoneProxGrad}, assume that $ x^k $ is not an M-stationary point of\n\\eqref{Eq:P}, and suppose that \\Cref{Ass:ProxGradMonotone}~\\ref{item:phi_bounded_affine} holds. Then the inner \nloop in \\cref{step:subproblem_solve_MonotoneProxGrad} of \\Cref{Alg:MonotoneProxGrad} is finite, i.e., we have $ \\gamma_k = \\gamma_{k,i_k} $\nfor some finite index $ i_k \\in \\{ 0, 1, 2, \\ldots \\} $.\n\\end{lemma}\n\n\\begin{proof}\n\tSuppose that the inner loop of \\cref{Alg:MonotoneProxGrad} does not\n\tterminate after a finite number of steps in iteration $k$.\nRecall that $ x^{k,i} $ is a solution of \\eqref{Eq:Subki}. Therefore, we get\n\\begin{equation}\\label{Eq:Sub1i}\n   \\langle \\nabla f(x^k),x^{k,i} - x^k\\rangle +    \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2 + \\phi (x^{k,i}) \\leq \\phi (x^k).\n\\end{equation}\nNoting that $ \\gamma_{k,i} \\to \\infty $ for $ i \\to \\infty $ and using \n\\Cref{Ass:ProxGradMonotone}~\\ref{item:phi_bounded_affine}, we obtain $ x^{k,i} \\to x^k $ for $ i \\to \\infty $. \nTaking the limit $ i \\to \\infty $ therefore yields\n\\begin{equation*}\n   \\phi (x^k) \\leq \\liminf_{i \\to \\infty} \\phi (x^{k,i}) \\leq \n   \\limsup_{i \\to \\infty} \\phi (x^{k,i}) \\leq \\phi (x^k),\n\\end{equation*}\nwhere the first estimate follows from the lower semicontinuity of $ \\phi $\nand the final inequality is a consequence of \\eqref{Eq:Sub1i}.\nTherefore, we have\n\\begin{equation}\\label{eq:robustsum}\n   \\phi (x^{k,i}) \\to \\phi (x^k) \\quad\n   \\text{for } i \\to \\infty .\n\\end{equation}\nWe claim that\n\\begin{equation}\\label{Eq:Sub2i}\n   \\liminf_{i \\to \\infty} \\gamma_{k,i} \\| x^{k,i} - x^k \\| > 0.\n\\end{equation}\nAssume, by contradiction, that there is a subsequence $ i_l \\to \\infty $ such that \n\\begin{equation}\\label{Eq:Sub3i}\n   \\liminf_{l \\to \\infty} \\gamma_{k, i_l} \\| x^{k, i_l} - x^k \\| = 0. \n\\end{equation}\nSince $ x^{k,i_l} $ is optimal for \\eqref{Eq:Subki}, Fermat's rule and the sum rule\n\\eqref{eq:regular_sum_rule} yield\n\\begin{equation}\\label{eq:optimality_condition_subproblem}\n   0 \\in  \\nabla f (x^k) + \\gamma_{k, i_l} ( x^{k, i_l} - x^k ) +\n   \\widehat\\partial \\phi (x^{k,i_l})\n\\end{equation}\nfor all $l\\in\\N$.\nTaking the limit $ l \\to \\infty $ while using \\eqref{eq:robustsum} and \\eqref{Eq:Sub3i},\nwe obtain\n\\begin{equation*}\n   0 \\in  \\nabla f (x^k) + \\partial \\phi (x^k),\n\\end{equation*}\nwhich means that $ x^k $ is already an M-stationary point of \\eqref{Eq:P}. This \ncontradiction shows that \\eqref{Eq:Sub2i} holds. Hence, there is a constant $ c > 0 $\nsuch that\n\\begin{equation*}\n   \\gamma_{k,i} \\| x^{k,i} - x^k \\| \\geq c \n\\end{equation*}\nholds for all large enough $i\\in\\N$.\nIn particular, this implies\n\\begin{equation}\\label{Eq:Sub4i}\n   ( 1- \\delta ) \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2 \\geq \n   \\frac{1- \\delta}{2} c \\| x^{k,i} - x^k \\| \\geq o \\big( \\| x^{k,i} - x^k \\| \\big)\n\\end{equation}\nfor all sufficiently large $i\\in\\N$.\nFurthermore, \\eqref{Eq:Sub1i} shows that\n\\begin{equation}\\label{Eq:Sub5i}\n\t\\langle \\nabla f (x^k),x^{k,i} - x^k\\rangle +\n   \\phi \\big( x^{k,i} \\big) - \\phi (x^k)\n   \\leq\n   - \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2 .\n\\end{equation}\nUsing a Taylor expansion of the \n\tfunction $ f $ \nand exploiting \\eqref{Eq:Sub4i},\n\\eqref{Eq:Sub5i}, we obtain \n\\begin{align*}\n   \\psi ( x^{k,i}) - \\psi (x^k)\n   & =  \n   f ( x^{k,i}) + \\phi (x^{k,i}) - f (x^k) - \\phi (x^k) \\\\\n   & =  \n   \\langle \\nabla f (x^k),x^{k,i}-x^k\\rangle + \\phi ( x^{k,i} ) - \\phi (x^k) +\n   \t\to \\big( \\| x^{k,i} - x^k \\| \\big) \\\\\n   & \\leq  \n   - \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2 + \n   \t\to \\big( \\| x^{k,i} - x^k \\| \\big) \\\\\n   & \\leq  \n   - \\delta \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2\n\\end{align*}\nfor all $ i \\in \\N $ sufficiently large.\n\tThis, however, means that the acceptance criterion \n\t\\eqref{Eq:StepCrit} is valid for sufficiently\n\tlarge $i\\in\\N$,\n\tcontradicting our assumption.\n\tThis completes the proof.\n\\end{proof}\n\n\\begin{proposition}\\label{Prop:xdiff}\nLet \\Cref{Ass:ProxGradMonotone} hold. Then each sequence \n$ \\{ x^k \\} $ generated by \\Cref{Alg:MonotoneProxGrad} satisfies\n$ \\| x^{k+1} - x^k \\| \\to 0 $.\n\\end{proposition}\n\n\\begin{proof}\nFirst recall that the sequence $ \\{ x^k \\} $ is well-defined by \\Cref{Lem:StepsizeFinite}.\nUsing the acceptance criterion \\eqref{Eq:StepCrit}, we get\n\\begin{equation}\\label{Eq:xdiff-1}\n   \\psi (x^{k+1}) \\leq \\psi (x^k) - \\delta \\frac{\\gamma_k}{2} \\| x^{k+1} - x^k \\|^2 \n   \\leq \\psi (x^k)\n\\end{equation}\nfor all $ k \\in \\mathbb{{N}} $. Hence, the sequence $ \\{ \\psi (x^k) \\} $ is \nmonotonically decreasing. Since $ \\psi $ is bounded from below on $ \\dom\\phi $\nby \\Cref{Ass:ProxGradMonotone}~\\ref{item:psi_bounded} and $ \\{ x^k \\} \\subset \\dom\\phi $, it follows that\nthis sequence is convergent. Therefore, \\eqref{Eq:xdiff-1} implies\n\\begin{equation*}\n   \\gamma_k \\| x^{k+1} - x^k \\|^2 \\to 0 \\quad \\text{for } k \\to \\infty .\n\\end{equation*}\nHence the assertion follows from the fact that, by \nconstruction, we have $ \\gamma_k \\geq \\gamma_{\\min} > 0 $ for all $ k \\in \\mathbb{N} $.\n\\end{proof}\n\n\\begin{proposition}\\label{Prop:gammaxdiff}\nLet \\Cref{Ass:ProxGradMonotone} hold, let $\\{x^k\\}$ be a sequence generated by \\cref{Alg:MonotoneProxGrad}, \nand let $ \\{ x^k \\}_K $ be a subsequence converging\nto some point $ x^* $. Then $ \\gamma_k \\| x^{k+1} - x^k \\| \\to_K 0 $ holds.\n\\end{proposition}\n\n\\begin{proof}\nIf the subsequence $ \\{ \\gamma_k \\}_K $ is bounded, the statement follows\nimmediately from \\Cref{Prop:xdiff}. The remaining part of this proof\ntherefore assumes that this subsequence is unbounded. Without loss of \ngenerality, we may assume that $ \\gamma_k  \\to_K \\infty $\n\tand that the acceptance criterion \\eqref{Eq:StepCrit} is violated\n\tin the first iteration of the inner loop for each $k\\in K$.\nThen, for $ \\hat{\\gamma}_k := \\gamma_k / \\tau $, $k\\in K$, we also have $\\hat\\gamma_k\\to_K\\infty$, but the \ncorresponding vector $ \\hat x^k := x^{k, i_k-1} $ does not satisfy the stepsize\ncondition from \\eqref{Eq:StepCrit}, i.e., we have\n\\begin{equation}\\label{Prop:gammaxdiff-1}\n   \\psi (\\hat x^k) > \n   \\psi (x^k) - \\delta \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2 \n   \\quad \\forall k \\in K.\n\\end{equation}\nOn the other hand, since $ \\hat{x}^k $ solves the corresponding \nsubproblem \\eqref{Eq:Subki} with $\\hat{\\gamma}_k = \\gamma_{k,i_k-1}$, we have\n\\begin{equation}\\label{Prop:gammaxdiff-2}\n    \\langle \\nabla f (x^k), \\hat{x}^k - x^k\\rangle + \\frac{\\hat{\\gamma}_k}{2}\n   \\| \\hat{x}^k - x^k \\|^2 + \\phi ( \\hat{x}^k ) - \\phi (x^k) \\leq 0 .\n\\end{equation}\nWe claim that this, in particular, implies $ \\hat{x}^k \\to_K x^* $. In fact, using\n\\eqref{Prop:gammaxdiff-2}, the Cauchy-Schwarz inequality, and the monotonicity of\n$ \\{ \\psi (x^k) \\} $, we obtain\n\\begin{align*}\n   \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2 \n   & \\leq \n   \\| \\nabla f (x^k) \\| \\| \\hat{x}^k - x^k \\| + \\phi (x^k) - \\phi ( \\hat{x}^k ) \\\\\n   & = \n   \\| \\nabla f (x^k) \\| \\| \\hat{x}^k - x^k \\| + \\psi (x^k) - f (x^k) - \\phi ( \\hat{x}^k ) \\\\\n   & \\leq \n   \\| \\nabla f (x^k) \\| \\| \\hat{x}^k - x^k \\| + \\psi (x^0) - f (x^k) - \\phi ( \\hat{x}^k ) .\n\\end{align*}\nSince $ f $ is continuously differentiable and $ - \\phi $ is bounded from above by \nan affine function in view of \\Cref{Ass:ProxGradMonotone}~\\ref{item:phi_bounded_affine}, this implies \n$ \\| \\hat{x}^k - x^k \\| \\to_K 0 $.\nIn fact, if $ \\{ \\| \\hat{x}^k - x^k \\| \\}_K $ would\nbe unbounded, then the left-hand side would grow more rapidly than the right-hand side,\nand if $ \\{ \\| \\hat{x}^k - x^k \\| \\}_K $ would be bounded, but staying away, at least\non a subsequence, from zero by a positive number, the right-hand side would be bounded, \nwhereas the left-hand side would be unbounded on the corresponding subsequence.\n\nNow, by the mean-value theorem, there exists $ \\xi^k $ on the line segment connecting\n$ x^k $ with $ \\hat{x}^k $ such that\n\\begin{equation}\\label{Prop:gammaxdiff-3}\n\t\\begin{aligned}\n   \t\t\\psi ( \\hat{x}^k ) - \\psi (x^k) \n   \t\t&= \n   \t\tf ( \\hat{x}^k ) + \\phi (\\hat{x}^k) -   f (x^k) - \\phi (x^k) \\\\\n   \t\t&= \n   \t\t\\langle  \\nabla f (\\xi^k),\\hat{x}^k - x^k \\rangle + \\phi (\\hat{x}^k) - \\phi (x^k) .\n   \\end{aligned}\n\\end{equation}\nSubstituting the expression $ \\phi (\\hat{x}^k) - \\phi (x^k) $ from \\eqref{Prop:gammaxdiff-3}\ninto \\eqref{Prop:gammaxdiff-2} yields \n\\begin{equation*}\n   \\langle \\nabla f(x^k) - \\nabla f ( \\xi^k) ,\\hat x^k - x^k \\rangle +\n   \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2 + \\psi ( \\hat{x}^k ) - \\psi (x^k) \\leq 0.\n\\end{equation*}\nExploiting \\eqref{Prop:gammaxdiff-1}, we therefore obtain \n\\begin{align*}\n   \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2\n   & \\leq \n   - \\langle \\nabla f(x^k) - \\nabla f ( \\xi^k), \\hat x^k - x^k \\rangle +  \\psi (x^k) - \\psi ( \\hat{x}^k ) \\\\\n   & \\leq\n   \\| \\nabla f (x^k) - \\nabla f ( \\xi^k) \\| \\| \\hat x^k - x^k \\| + \\delta \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2 ,\n\\end{align*}\nwhich can be rewritten as\n\\begin{equation}\\label{Prop:gammaxdiff-4}\n   ( 1 - \\delta ) \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\| \\leq \n   \\| \\nabla f (x^k) - \\nabla f ( \\xi^k) \\| \n\\end{equation}\n(note that $ \\hat{x}^k \\neq x^k $ in view of \\eqref{Prop:gammaxdiff-1}).\nSince $ x^k\\to_K x^* $ (by assumption) and $ \\hat{x}^k \\to_K x^* $ (by the\nprevious part of this proof), we also get $ \\xi^k \\to_K x^* $. Using $ \\delta \\in (0,1) $\nand the continuous differentiability of $ f $, it follows from \\eqref{Prop:gammaxdiff-4}\nthat $ \\hat{\\gamma}_k \\| \\hat{x}^k - x^k \\| \\to_K 0 $.\n\nFinally, exploiting the fact that $ x^{k+1} $ and $ \\hat{x}^k $ are solutions of\nthe subproblems \\eqref{Eq:Subki} with parameters $ \\gamma_k $ and $ \\hat{\\gamma}_k $,\nrespectively, we find\n\\begin{align*}\n   \\langle \\nabla f(x^k),x^{k+1} - x^k\\rangle \n   &+\n   \\frac{\\gamma_k}{2} \\| x^{k+1} - x^k \\|^2 + \\phi (x^{k+1})\n   \\\\\n   &\n   \\leq \\langle \\nabla f (x^k),  \\hat{x}^k - x^k \\rangle +\n   \\frac{\\gamma_k}{2} \\| \\hat{x}^k - x^k \\|^2 + \\phi (\\hat{x}^k),\n   \\\\\n    \\langle \\nabla f(x^k), \\hat{x}^k - x^k \\rangle \n    &+\n   \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2 + \\phi (\\hat{x}^k)\n   \\\\\n   &\n   \\leq  \\langle \\nabla f(x^k),  x^{k+1} - x^k \\rangle +\n   \\frac{\\hat{\\gamma}_k}{2} \\| x^{k+1} - x^k \\|^2 + \\phi (x^{k+1}).\n\\end{align*}\nAdding these two inequalities and noting that $ \\gamma_k = \\tau \\hat{\\gamma}_k >\n\\hat{\\gamma}_k $ yields $ \\| x^{k+1} - x^k \\| \\leq \\| \\hat{x}^k - x^k \\| $ and,\ntherefore,\n\\begin{equation*}\n   \\gamma_k \\| x^{k+1}- x^k \\| = \\tau \\hat{\\gamma}_k \\| x^{k+1} - x^k \\| \\leq\n   \\tau \\hat{\\gamma}_k \\| \\hat{x}^k - x^k \\| \\to_K 0.\n\\end{equation*}\nThis completes the proof.\n\\end{proof}\n\n\\begin{corollary}\\label{Cor:LipschitzCase}\nLet \\Cref{Ass:ProxGradMonotone} hold, let $\\{x^k\\}$ be a sequence generated by \\Cref{Alg:MonotoneProxGrad}, let $ \\{ x^k \\}_K $ be a subsequence converging\nto some point $ x^* $, and assume that $ \\nabla f \\colon \\mathbb X\\to\\mathbb X$ is locally Lipschitz continuous\naround $ x^* $. Then the corresponding subsequence $ \\{ \\gamma_k \\}_K $ is\nbounded.\n\\end{corollary}\n\n\\begin{proof}\nWe may argue as in the proof of \\cref{Prop:gammaxdiff}. \nHence, \n\ton the contrary, assume\nthat $ \\gamma_k \\to_K \\infty $. For each $k\\in K$, define $ \\hat{\\gamma}_k $ and $ \\hat{x}^k $ as in\nthat proof, and let $ L > 0 $ denote the local Lipschitz constant of $ \\nabla f $ around\n$ x^* $. Recall that $ x^k \\to_K x^* $ (by assumption) and $ \\hat{x}^k \\to_K x^* $\n(from the proof of \\cref{Prop:gammaxdiff}). Exploiting \\eqref{Prop:gammaxdiff-4},\nwe therefore obtain\n\\begin{equation*}\n   ( 1 - \\delta ) \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\| \\leq L \\| \\hat{x}^k - \\xi^k \\|\n   \\leq L \\| \\hat{x}^k - x^k \\|\n\\end{equation*}\nfor all $ k \\in K $ sufficiently large, using the fact that $ \\xi^k $ is on the \nline segment between $ x^k $ and $ \\hat{x}^k $. Since $ \\hat{\\gamma}_k \\to_K \\infty $ and \n$ \\hat{x}^k \\neq x^k $, see once again \\eqref{Prop:gammaxdiff-1}, this gives a contradiction. \nHence, $ \\{ \\gamma_k \\}_K $ stays bounded.\n\\end{proof}\n\n\\begin{proof}\nLet $ \\{ x^k \\}_K $ be a subsequence converging to $ x^* $. In view of\n\\Cref{Prop:xdiff}, it follows that also the subsequence $ \\{ x^{k+1} \\}_K $\nconverges to $ x^* $. Furthermore, \\Cref{Prop:gammaxdiff} yields\n$ \\gamma_k \\| x^{k+1} - x^k \\| \\to_K 0 $.\nThe minimizing property of $ x^{k+1} $, Fermat's rule, and the sum rule \\eqref{eq:regular_sum_rule} imply that\n\\begin{equation}\\label{Eq:OptCondSub}\n   0 \\in  \\nabla f (x^{k}) + \\gamma_k (x^{k+1}-x^k) \n   + \\widehat\\partial \\phi (x^{k+1})\n\\end{equation}\nholds for each $k\\in K$. Hence, if we can show $ \\phi (x^{k+1}) \\to_K \\phi (x^*) $, we can\ntake the limit $ k \\to_K \\infty $ in \\eqref{Eq:OptCondSub} \nto obtain the desired statement $ 0 \\in  \\nabla f (x^*) + \\partial \\phi (x^*) $.\n\n\tDue to \\eqref{Eq:xdiff-1}, we find $\\psi(x^{k+1})\\leq \\psi(x^0)$\n\tfor each $k\\in K$. Taking the limit $k\\to_K\\infty$ while respecting\n\tthe lower semicontinuity of $\\phi$ gives $\\psi(x^*)\\leq\\psi(x^0)$,\n\tand due to $x^0\\in\\dom\\phi$, we find $x^*\\in\\dom\\phi$.\nThus, the condition $ \\phi (x^{k+1}) \\to_K \\phi (x^*) $ obviously holds if\n$ \\phi $ is continuous on its domain since all iterates $ x^k $ generated by\n\\Cref{Alg:MonotoneProxGrad} \n\tas well as $x^*$\nbelong to $\\dom\\phi$.\n\nHence, it remains to consider the situation where $ \\phi $ is only lower semicontinuous, but\n$ \\nabla f $ is locally Lipschitz continuous. \nFrom $ x^{k+1} \\to_K x^* $ and the lower semicontinuity of $\\phi$, we find\n\\begin{equation*}\n   \\phi (x^*) \\leq \\liminf_{k \\in K} \\phi (x^{k+1}) \\leq \\limsup_{k \\in K} \\phi (x^{k+1}).\n\\end{equation*}\nIt therefore suffices to show that $ \\limsup_{k \\in K} \\phi (x^{k+1}) \\leq \\phi (x^*) $ holds.\nSince $ x^{k+1} $ solves the subproblem \\eqref{Eq:Subki} with parameter $ \\gamma_k $, we obtain\n\\begin{align*}\n    \\langle \\nabla f (x^k),x^{k+1}-x^k \\rangle \n    &+ \n    \\frac{\\gamma_k}{2} \\| x^{k+1} - x^k \\|^2 +  \\phi (x^{k+1} ) \n    \\\\\n    &\n    \\leq  \n    \\langle \\nabla f(x^k),x^* - x^k \\rangle \n   \t+ \n   \t\\frac{\\gamma_k}{2} \\| x^* - x^k \\|^2 + \\phi (x^*)\n\\end{align*}\nfor each $k\\in K$.\nWe now take the upper limit over $K$ on both sides. \nUsing the continuity of $ \\nabla f $, \nthe convergences $ x^{k+1} - x^k \\to_K 0 $ \nas well as $ \\gamma_k \\| x^{k+1} - x^k \\|^2 \\to_K 0 $ (see \\Cref{Prop:xdiff,Prop:gammaxdiff}),\nand taking into account that $ \\gamma_k \\| x^k - x^* \\|^2 \\to_K 0 $ due to the boundedness\nof the subsequence $ \\{ \\gamma_k \\}_K $ in this situation, see \n\\cref{Cor:LipschitzCase}, we obtain \n$ \\limsup_{k \\in K} \\phi (x^{k+1}) \\leq \\phi (x^*) $. Altogether, we\ntherefore get $ \\phi (x^{k+1}) \\to_K \\phi (x^*) $, and this completes the proof.\n\\end{proof}","units":[{"start":26328,"end":26885,"start_line":607,"end_line":612,"raw_tex":"\\Cref{Ass:ProxGradMonotone}~\\ref{item:psi_bounded} is a reasonable condition regarding the\ngiven composite optimization problem, whereas \\Cref{Ass:ProxGradMonotone}~\\ref{item:phi_bounded_affine}\nis essentially a statement relevant for the subproblems from \\eqref{Eq:Subki}. In particular,\n\\Cref{Ass:ProxGradMonotone}~\\ref{item:phi_bounded_affine} implies that the quadratic objective function\nof the subproblems \\eqref{Eq:Subki} are, for fixed $ k, i \\in\\N$, coercive, and therefore always\nattain a solution $ x^{k,i} $ (which, however, may not be unique).\n","sha256_utf8":"21339d11278cef93ff86aff867eb55c76d7aa41a0a25e2603b1cae430af2e261"},{"start":27147,"end":27624,"start_line":619,"end_line":624,"raw_tex":"\\begin{lemma}\\label{Lem:StepsizeFinite}\nConsider a fixed iteration $ k $ of \\cref{Alg:MonotoneProxGrad}, assume that $ x^k $ is not an M-stationary point of\n\\eqref{Eq:P}, and suppose that \\Cref{Ass:ProxGradMonotone}~\\ref{item:phi_bounded_affine} holds. Then the inner \nloop in \\cref{step:subproblem_solve_MonotoneProxGrad} of \\Cref{Alg:MonotoneProxGrad} is finite, i.e., we have $ \\gamma_k = \\gamma_{k,i_k} $\nfor some finite index $ i_k \\in \\{ 0, 1, 2, \\ldots \\} $.\n\\end{lemma}","sha256_utf8":"7157e4613e2fbfa961118a791bfd453c2d77db114d039153ade6a1d44a84654b"},{"start":27626,"end":31082,"start_line":626,"end_line":710,"raw_tex":"\\begin{proof}\n\tSuppose that the inner loop of \\cref{Alg:MonotoneProxGrad} does not\n\tterminate after a finite number of steps in iteration $k$.\nRecall that $ x^{k,i} $ is a solution of \\eqref{Eq:Subki}. Therefore, we get\n\\begin{equation}\\label{Eq:Sub1i}\n   \\langle \\nabla f(x^k),x^{k,i} - x^k\\rangle +    \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2 + \\phi (x^{k,i}) \\leq \\phi (x^k).\n\\end{equation}\nNoting that $ \\gamma_{k,i} \\to \\infty $ for $ i \\to \\infty $ and using \n\\Cref{Ass:ProxGradMonotone}~\\ref{item:phi_bounded_affine}, we obtain $ x^{k,i} \\to x^k $ for $ i \\to \\infty $. \nTaking the limit $ i \\to \\infty $ therefore yields\n\\begin{equation*}\n   \\phi (x^k) \\leq \\liminf_{i \\to \\infty} \\phi (x^{k,i}) \\leq \n   \\limsup_{i \\to \\infty} \\phi (x^{k,i}) \\leq \\phi (x^k),\n\\end{equation*}\nwhere the first estimate follows from the lower semicontinuity of $ \\phi $\nand the final inequality is a consequence of \\eqref{Eq:Sub1i}.\nTherefore, we have\n\\begin{equation}\\label{eq:robustsum}\n   \\phi (x^{k,i}) \\to \\phi (x^k) \\quad\n   \\text{for } i \\to \\infty .\n\\end{equation}\nWe claim that\n\\begin{equation}\\label{Eq:Sub2i}\n   \\liminf_{i \\to \\infty} \\gamma_{k,i} \\| x^{k,i} - x^k \\| > 0.\n\\end{equation}\nAssume, by contradiction, that there is a subsequence $ i_l \\to \\infty $ such that \n\\begin{equation}\\label{Eq:Sub3i}\n   \\liminf_{l \\to \\infty} \\gamma_{k, i_l} \\| x^{k, i_l} - x^k \\| = 0. \n\\end{equation}\nSince $ x^{k,i_l} $ is optimal for \\eqref{Eq:Subki}, Fermat's rule and the sum rule\n\\eqref{eq:regular_sum_rule} yield\n\\begin{equation}\\label{eq:optimality_condition_subproblem}\n   0 \\in  \\nabla f (x^k) + \\gamma_{k, i_l} ( x^{k, i_l} - x^k ) +\n   \\widehat\\partial \\phi (x^{k,i_l})\n\\end{equation}\nfor all $l\\in\\N$.\nTaking the limit $ l \\to \\infty $ while using \\eqref{eq:robustsum} and \\eqref{Eq:Sub3i},\nwe obtain\n\\begin{equation*}\n   0 \\in  \\nabla f (x^k) + \\partial \\phi (x^k),\n\\end{equation*}\nwhich means that $ x^k $ is already an M-stationary point of \\eqref{Eq:P}. This \ncontradiction shows that \\eqref{Eq:Sub2i} holds. Hence, there is a constant $ c > 0 $\nsuch that\n\\begin{equation*}\n   \\gamma_{k,i} \\| x^{k,i} - x^k \\| \\geq c \n\\end{equation*}\nholds for all large enough $i\\in\\N$.\nIn particular, this implies\n\\begin{equation}\\label{Eq:Sub4i}\n   ( 1- \\delta ) \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2 \\geq \n   \\frac{1- \\delta}{2} c \\| x^{k,i} - x^k \\| \\geq o \\big( \\| x^{k,i} - x^k \\| \\big)\n\\end{equation}\nfor all sufficiently large $i\\in\\N$.\nFurthermore, \\eqref{Eq:Sub1i} shows that\n\\begin{equation}\\label{Eq:Sub5i}\n\t\\langle \\nabla f (x^k),x^{k,i} - x^k\\rangle +\n   \\phi \\big( x^{k,i} \\big) - \\phi (x^k)\n   \\leq\n   - \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2 .\n\\end{equation}\nUsing a Taylor expansion of the \n\tfunction $ f $ \nand exploiting \\eqref{Eq:Sub4i},\n\\eqref{Eq:Sub5i}, we obtain \n\\begin{align*}\n   \\psi ( x^{k,i}) - \\psi (x^k)\n   & =  \n   f ( x^{k,i}) + \\phi (x^{k,i}) - f (x^k) - \\phi (x^k) \\\\\n   & =  \n   \\langle \\nabla f (x^k),x^{k,i}-x^k\\rangle + \\phi ( x^{k,i} ) - \\phi (x^k) +\n   \t\to \\big( \\| x^{k,i} - x^k \\| \\big) \\\\\n   & \\leq  \n   - \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2 + \n   \t\to \\big( \\| x^{k,i} - x^k \\| \\big) \\\\\n   & \\leq  \n   - \\delta \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2\n\\end{align*}\nfor all $ i \\in \\N $ sufficiently large.\n\tThis, however, means that the acceptance criterion \n\t\\eqref{Eq:StepCrit} is valid for sufficiently\n\tlarge $i\\in\\N$,\n\tcontradicting our assumption.\n\tThis completes the proof.\n\\end{proof}","sha256_utf8":"2297bcadd4a3b193bc907a29a9f0d66cc246fd8d78800a09ea402b7ead1ddb1a"},{"start":32301,"end":32510,"start_line":736,"end_line":740,"raw_tex":"\\begin{proposition}\\label{Prop:xdiff}\nLet \\Cref{Ass:ProxGradMonotone} hold. Then each sequence \n$ \\{ x^k \\} $ generated by \\Cref{Alg:MonotoneProxGrad} satisfies\n$ \\| x^{k+1} - x^k \\| \\to 0 $.\n\\end{proposition}","sha256_utf8":"792d2db8cc7192bf608af00f247e62869b9a80cf51fa31492b13d2067aea5e10"},{"start":32512,"end":33417,"start_line":742,"end_line":758,"raw_tex":"\\begin{proof}\nFirst recall that the sequence $ \\{ x^k \\} $ is well-defined by \\Cref{Lem:StepsizeFinite}.\nUsing the acceptance criterion \\eqref{Eq:StepCrit}, we get\n\\begin{equation}\\label{Eq:xdiff-1}\n   \\psi (x^{k+1}) \\leq \\psi (x^k) - \\delta \\frac{\\gamma_k}{2} \\| x^{k+1} - x^k \\|^2 \n   \\leq \\psi (x^k)\n\\end{equation}\nfor all $ k \\in \\mathbb{{N}} $. Hence, the sequence $ \\{ \\psi (x^k) \\} $ is \nmonotonically decreasing. Since $ \\psi $ is bounded from below on $ \\dom\\phi $\nby \\Cref{Ass:ProxGradMonotone}~\\ref{item:psi_bounded} and $ \\{ x^k \\} \\subset \\dom\\phi $, it follows that\nthis sequence is convergent. Therefore, \\eqref{Eq:xdiff-1} implies\n\\begin{equation*}\n   \\gamma_k \\| x^{k+1} - x^k \\|^2 \\to 0 \\quad \\text{for } k \\to \\infty .\n\\end{equation*}\nHence the assertion follows from the fact that, by \nconstruction, we have $ \\gamma_k \\geq \\gamma_{\\min} > 0 $ for all $ k \\in \\mathbb{N} $.\n\\end{proof}","sha256_utf8":"1d8229ded74d13ef93f297b78cf6044d65de9d36e6095c809e3d90c74f1ff9a7"},{"start":33467,"end":33764,"start_line":762,"end_line":766,"raw_tex":"\\begin{proposition}\\label{Prop:gammaxdiff}\nLet \\Cref{Ass:ProxGradMonotone} hold, let $\\{x^k\\}$ be a sequence generated by \\cref{Alg:MonotoneProxGrad}, \nand let $ \\{ x^k \\}_K $ be a subsequence converging\nto some point $ x^* $. Then $ \\gamma_k \\| x^{k+1} - x^k \\| \\to_K 0 $ holds.\n\\end{proposition}","sha256_utf8":"8e67accfe8616e42959bfb5792621a203686526c03b15f1a2cf824d5feccb5c4"},{"start":33766,"end":38986,"start_line":768,"end_line":874,"raw_tex":"\\begin{proof}\nIf the subsequence $ \\{ \\gamma_k \\}_K $ is bounded, the statement follows\nimmediately from \\Cref{Prop:xdiff}. The remaining part of this proof\ntherefore assumes that this subsequence is unbounded. Without loss of \ngenerality, we may assume that $ \\gamma_k  \\to_K \\infty $\n\tand that the acceptance criterion \\eqref{Eq:StepCrit} is violated\n\tin the first iteration of the inner loop for each $k\\in K$.\nThen, for $ \\hat{\\gamma}_k := \\gamma_k / \\tau $, $k\\in K$, we also have $\\hat\\gamma_k\\to_K\\infty$, but the \ncorresponding vector $ \\hat x^k := x^{k, i_k-1} $ does not satisfy the stepsize\ncondition from \\eqref{Eq:StepCrit}, i.e., we have\n\\begin{equation}\\label{Prop:gammaxdiff-1}\n   \\psi (\\hat x^k) > \n   \\psi (x^k) - \\delta \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2 \n   \\quad \\forall k \\in K.\n\\end{equation}\nOn the other hand, since $ \\hat{x}^k $ solves the corresponding \nsubproblem \\eqref{Eq:Subki} with $\\hat{\\gamma}_k = \\gamma_{k,i_k-1}$, we have\n\\begin{equation}\\label{Prop:gammaxdiff-2}\n    \\langle \\nabla f (x^k), \\hat{x}^k - x^k\\rangle + \\frac{\\hat{\\gamma}_k}{2}\n   \\| \\hat{x}^k - x^k \\|^2 + \\phi ( \\hat{x}^k ) - \\phi (x^k) \\leq 0 .\n\\end{equation}\nWe claim that this, in particular, implies $ \\hat{x}^k \\to_K x^* $. In fact, using\n\\eqref{Prop:gammaxdiff-2}, the Cauchy-Schwarz inequality, and the monotonicity of\n$ \\{ \\psi (x^k) \\} $, we obtain\n\\begin{align*}\n   \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2 \n   & \\leq \n   \\| \\nabla f (x^k) \\| \\| \\hat{x}^k - x^k \\| + \\phi (x^k) - \\phi ( \\hat{x}^k ) \\\\\n   & = \n   \\| \\nabla f (x^k) \\| \\| \\hat{x}^k - x^k \\| + \\psi (x^k) - f (x^k) - \\phi ( \\hat{x}^k ) \\\\\n   & \\leq \n   \\| \\nabla f (x^k) \\| \\| \\hat{x}^k - x^k \\| + \\psi (x^0) - f (x^k) - \\phi ( \\hat{x}^k ) .\n\\end{align*}\nSince $ f $ is continuously differentiable and $ - \\phi $ is bounded from above by \nan affine function in view of \\Cref{Ass:ProxGradMonotone}~\\ref{item:phi_bounded_affine}, this implies \n$ \\| \\hat{x}^k - x^k \\| \\to_K 0 $.\nIn fact, if $ \\{ \\| \\hat{x}^k - x^k \\| \\}_K $ would\nbe unbounded, then the left-hand side would grow more rapidly than the right-hand side,\nand if $ \\{ \\| \\hat{x}^k - x^k \\| \\}_K $ would be bounded, but staying away, at least\non a subsequence, from zero by a positive number, the right-hand side would be bounded, \nwhereas the left-hand side would be unbounded on the corresponding subsequence.\n\nNow, by the mean-value theorem, there exists $ \\xi^k $ on the line segment connecting\n$ x^k $ with $ \\hat{x}^k $ such that\n\\begin{equation}\\label{Prop:gammaxdiff-3}\n\t\\begin{aligned}\n   \t\t\\psi ( \\hat{x}^k ) - \\psi (x^k) \n   \t\t&= \n   \t\tf ( \\hat{x}^k ) + \\phi (\\hat{x}^k) -   f (x^k) - \\phi (x^k) \\\\\n   \t\t&= \n   \t\t\\langle  \\nabla f (\\xi^k),\\hat{x}^k - x^k \\rangle + \\phi (\\hat{x}^k) - \\phi (x^k) .\n   \\end{aligned}\n\\end{equation}\nSubstituting the expression $ \\phi (\\hat{x}^k) - \\phi (x^k) $ from \\eqref{Prop:gammaxdiff-3}\ninto \\eqref{Prop:gammaxdiff-2} yields \n\\begin{equation*}\n   \\langle \\nabla f(x^k) - \\nabla f ( \\xi^k) ,\\hat x^k - x^k \\rangle +\n   \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2 + \\psi ( \\hat{x}^k ) - \\psi (x^k) \\leq 0.\n\\end{equation*}\nExploiting \\eqref{Prop:gammaxdiff-1}, we therefore obtain \n\\begin{align*}\n   \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2\n   & \\leq \n   - \\langle \\nabla f(x^k) - \\nabla f ( \\xi^k), \\hat x^k - x^k \\rangle +  \\psi (x^k) - \\psi ( \\hat{x}^k ) \\\\\n   & \\leq\n   \\| \\nabla f (x^k) - \\nabla f ( \\xi^k) \\| \\| \\hat x^k - x^k \\| + \\delta \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2 ,\n\\end{align*}\nwhich can be rewritten as\n\\begin{equation}\\label{Prop:gammaxdiff-4}\n   ( 1 - \\delta ) \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\| \\leq \n   \\| \\nabla f (x^k) - \\nabla f ( \\xi^k) \\| \n\\end{equation}\n(note that $ \\hat{x}^k \\neq x^k $ in view of \\eqref{Prop:gammaxdiff-1}).\nSince $ x^k\\to_K x^* $ (by assumption) and $ \\hat{x}^k \\to_K x^* $ (by the\nprevious part of this proof), we also get $ \\xi^k \\to_K x^* $. Using $ \\delta \\in (0,1) $\nand the continuous differentiability of $ f $, it follows from \\eqref{Prop:gammaxdiff-4}\nthat $ \\hat{\\gamma}_k \\| \\hat{x}^k - x^k \\| \\to_K 0 $.\n\nFinally, exploiting the fact that $ x^{k+1} $ and $ \\hat{x}^k $ are solutions of\nthe subproblems \\eqref{Eq:Subki} with parameters $ \\gamma_k $ and $ \\hat{\\gamma}_k $,\nrespectively, we find\n\\begin{align*}\n   \\langle \\nabla f(x^k),x^{k+1} - x^k\\rangle \n   &+\n   \\frac{\\gamma_k}{2} \\| x^{k+1} - x^k \\|^2 + \\phi (x^{k+1})\n   \\\\\n   &\n   \\leq \\langle \\nabla f (x^k),  \\hat{x}^k - x^k \\rangle +\n   \\frac{\\gamma_k}{2} \\| \\hat{x}^k - x^k \\|^2 + \\phi (\\hat{x}^k),\n   \\\\\n    \\langle \\nabla f(x^k), \\hat{x}^k - x^k \\rangle \n    &+\n   \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2 + \\phi (\\hat{x}^k)\n   \\\\\n   &\n   \\leq  \\langle \\nabla f(x^k),  x^{k+1} - x^k \\rangle +\n   \\frac{\\hat{\\gamma}_k}{2} \\| x^{k+1} - x^k \\|^2 + \\phi (x^{k+1}).\n\\end{align*}\nAdding these two inequalities and noting that $ \\gamma_k = \\tau \\hat{\\gamma}_k >\n\\hat{\\gamma}_k $ yields $ \\| x^{k+1} - x^k \\| \\leq \\| \\hat{x}^k - x^k \\| $ and,\ntherefore,\n\\begin{equation*}\n   \\gamma_k \\| x^{k+1}- x^k \\| = \\tau \\hat{\\gamma}_k \\| x^{k+1} - x^k \\| \\leq\n   \\tau \\hat{\\gamma}_k \\| \\hat{x}^k - x^k \\| \\to_K 0.\n\\end{equation*}\nThis completes the proof.\n\\end{proof}","sha256_utf8":"442c39c9657cf1ddb56ff854e2d10614a020ce11e2415c7dad4d55af17f13978"},{"start":39257,"end":39667,"start_line":880,"end_line":885,"raw_tex":"\\begin{corollary}\\label{Cor:LipschitzCase}\nLet \\Cref{Ass:ProxGradMonotone} hold, let $\\{x^k\\}$ be a sequence generated by \\Cref{Alg:MonotoneProxGrad}, let $ \\{ x^k \\}_K $ be a subsequence converging\nto some point $ x^* $, and assume that $ \\nabla f \\colon \\mathbb X\\to\\mathbb X$ is locally Lipschitz continuous\naround $ x^* $. Then the corresponding subsequence $ \\{ \\gamma_k \\}_K $ is\nbounded.\n\\end{corollary}","sha256_utf8":"31ae025e7a6383c682d1b9569d2a1432dc974a2699ee44bc693bb6f8df6bc4db"},{"start":39669,"end":40631,"start_line":887,"end_line":904,"raw_tex":"\\begin{proof}\nWe may argue as in the proof of \\cref{Prop:gammaxdiff}. \nHence, \n\ton the contrary, assume\nthat $ \\gamma_k \\to_K \\infty $. For each $k\\in K$, define $ \\hat{\\gamma}_k $ and $ \\hat{x}^k $ as in\nthat proof, and let $ L > 0 $ denote the local Lipschitz constant of $ \\nabla f $ around\n$ x^* $. Recall that $ x^k \\to_K x^* $ (by assumption) and $ \\hat{x}^k \\to_K x^* $\n(from the proof of \\cref{Prop:gammaxdiff}). Exploiting \\eqref{Prop:gammaxdiff-4},\nwe therefore obtain\n\\begin{equation*}\n   ( 1 - \\delta ) \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\| \\leq L \\| \\hat{x}^k - \\xi^k \\|\n   \\leq L \\| \\hat{x}^k - x^k \\|\n\\end{equation*}\nfor all $ k \\in K $ sufficiently large, using the fact that $ \\xi^k $ is on the \nline segment between $ x^k $ and $ \\hat{x}^k $. Since $ \\hat{\\gamma}_k \\to_K \\infty $ and \n$ \\hat{x}^k \\neq x^k $, see once again \\eqref{Prop:gammaxdiff-1}, this gives a contradiction. \nHence, $ \\{ \\gamma_k \\}_K $ stays bounded.\n\\end{proof}","sha256_utf8":"962130ce6a0f9ba228a75a747bf587455a48282c345299878e47bff454413738"},{"start":41173,"end":43822,"start_line":917,"end_line":970,"raw_tex":"\\begin{proof}\nLet $ \\{ x^k \\}_K $ be a subsequence converging to $ x^* $. In view of\n\\Cref{Prop:xdiff}, it follows that also the subsequence $ \\{ x^{k+1} \\}_K $\nconverges to $ x^* $. Furthermore, \\Cref{Prop:gammaxdiff} yields\n$ \\gamma_k \\| x^{k+1} - x^k \\| \\to_K 0 $.\nThe minimizing property of $ x^{k+1} $, Fermat's rule, and the sum rule \\eqref{eq:regular_sum_rule} imply that\n\\begin{equation}\\label{Eq:OptCondSub}\n   0 \\in  \\nabla f (x^{k}) + \\gamma_k (x^{k+1}-x^k) \n   + \\widehat\\partial \\phi (x^{k+1})\n\\end{equation}\nholds for each $k\\in K$. Hence, if we can show $ \\phi (x^{k+1}) \\to_K \\phi (x^*) $, we can\ntake the limit $ k \\to_K \\infty $ in \\eqref{Eq:OptCondSub} \nto obtain the desired statement $ 0 \\in  \\nabla f (x^*) + \\partial \\phi (x^*) $.\n\n\tDue to \\eqref{Eq:xdiff-1}, we find $\\psi(x^{k+1})\\leq \\psi(x^0)$\n\tfor each $k\\in K$. Taking the limit $k\\to_K\\infty$ while respecting\n\tthe lower semicontinuity of $\\phi$ gives $\\psi(x^*)\\leq\\psi(x^0)$,\n\tand due to $x^0\\in\\dom\\phi$, we find $x^*\\in\\dom\\phi$.\nThus, the condition $ \\phi (x^{k+1}) \\to_K \\phi (x^*) $ obviously holds if\n$ \\phi $ is continuous on its domain since all iterates $ x^k $ generated by\n\\Cref{Alg:MonotoneProxGrad} \n\tas well as $x^*$\nbelong to $\\dom\\phi$.\n\nHence, it remains to consider the situation where $ \\phi $ is only lower semicontinuous, but\n$ \\nabla f $ is locally Lipschitz continuous. \nFrom $ x^{k+1} \\to_K x^* $ and the lower semicontinuity of $\\phi$, we find\n\\begin{equation*}\n   \\phi (x^*) \\leq \\liminf_{k \\in K} \\phi (x^{k+1}) \\leq \\limsup_{k \\in K} \\phi (x^{k+1}).\n\\end{equation*}\nIt therefore suffices to show that $ \\limsup_{k \\in K} \\phi (x^{k+1}) \\leq \\phi (x^*) $ holds.\nSince $ x^{k+1} $ solves the subproblem \\eqref{Eq:Subki} with parameter $ \\gamma_k $, we obtain\n\\begin{align*}\n    \\langle \\nabla f (x^k),x^{k+1}-x^k \\rangle \n    &+ \n    \\frac{\\gamma_k}{2} \\| x^{k+1} - x^k \\|^2 +  \\phi (x^{k+1} ) \n    \\\\\n    &\n    \\leq  \n    \\langle \\nabla f(x^k),x^* - x^k \\rangle \n   \t+ \n   \t\\frac{\\gamma_k}{2} \\| x^* - x^k \\|^2 + \\phi (x^*)\n\\end{align*}\nfor each $k\\in K$.\nWe now take the upper limit over $K$ on both sides. \nUsing the continuity of $ \\nabla f $, \nthe convergences $ x^{k+1} - x^k \\to_K 0 $ \nas well as $ \\gamma_k \\| x^{k+1} - x^k \\|^2 \\to_K 0 $ (see \\Cref{Prop:xdiff,Prop:gammaxdiff}),\nand taking into account that $ \\gamma_k \\| x^k - x^* \\|^2 \\to_K 0 $ due to the boundedness\nof the subsequence $ \\{ \\gamma_k \\}_K $ in this situation, see \n\\cref{Cor:LipschitzCase}, we obtain \n$ \\limsup_{k \\in K} \\phi (x^{k+1}) \\leq \\phi (x^*) $. Altogether, we\ntherefore get $ \\phi (x^{k+1}) \\to_K \\phi (x^*) $, and this completes the proof.\n\\end{proof}","sha256_utf8":"9c437ee71db4c2b8b83b72ae134c5132600b12f3cba80fa49d644482d8985f22"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"复合优化问题","source_units":[{"start":16050,"end":16673,"start_line":381,"end_line":391,"raw_tex":"We consider the \\emph{composite} optimization problem\n\\begin{equation}\\label{Eq:P}\\tag{P}\n   \\min_x \\ \\psi(x):=f(x) + \\phi (x), \\quad \\quad x \\in \\mathbb X,\n\\end{equation}\nwhere $ f\\colon \\mathbb{X} \\to \\mathbb{R} $ is\ncontinuously differentiable, $ \\phi\\colon \\mathbb{X} \\to \\overline{\\mathbb{R}}:= \\R\\cup\\{\\infty\\} $ \nis lower semicontinuous (possibly infinite-valued and nondifferentiable), \nand $ \\mathbb{X} $ denotes a\nEuclidean space, i.e., a real and finite-dimensional Hilbert space.\nWe assume that the domain $\\dom\\phi:=\\{x\\in\\mathbb X\\,|\\,\\phi(x)<\\infty\\}$ of $ \\phi $\nis nonempty to rule out trivial situations.\n","sha256_utf8":"6619dda46c80b9a42d4c40215d5c4a33f680eb5dc94f79826c69fcc0bda35f9f"}]},{"id":"S002","semantic_type":"notation","kind":"notation","title":"空间与收敛记号","source_units":[{"start":17010,"end":17851,"start_line":397,"end_line":408,"raw_tex":"Throughout the paper, the Euclidean space $\\mathbb X$ \nwill be equipped with the inner product $\\langle\\cdot,\\cdot\\rangle\\colon \\mathbb X\\times\\mathbb X\\to\\R$\nand the associated norm $\\norm{\\cdot}$.\nFor some set $A\\subset\\mathbb X$ and some point $x\\in\\mathbb X$, we make use of\n$A+x=x+A:=\\{x+a\\,|\\,a\\in A\\}$ for the purpose of simplicity.\nFor some sequence $\\{x^k\\}\\subset\\mathbb X$ and $x\\in\\mathbb X$, $x^k\\to_\\phi x$ means that\n$x^k\\to x$ and $\\phi(x^k)\\to\\phi(x)$.\nThe continuous linear operator\n$f'(x)\\colon\\mathbb X\\to\\R$ denotes the derivative of $f$ at $x\\in\\mathbb X$, and we will\nmake use of $\\nabla f(x):=f'(x)^*1$ where $f'(x)^*\\colon\\R\\to\\mathbb X$ is the adjoint of $f'(x)$.\nThis way, $\\nabla f$ is a mapping from $\\mathbb X$ to $\\mathbb X$.\nFurthermore, we find $f'(x)d=\\langle\\nabla f(x),d\\rangle$ for each $d\\in\\mathbb X$.\n","sha256_utf8":"621b001604e40eb779d75ac6ecaec6ea98a9632d9b5ea58caa704cbd60c037c7"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"正则与极限次微分","source_units":[{"start":17853,"end":18704,"start_line":410,"end_line":432,"raw_tex":"The following concepts are standard in variational analysis, see e.g.\\ \\cite{Mordukhovich2018,RockafellarWets2009}.\nLet us fix some point $x\\in\\dom\\phi$. Then\n\\[\n\t\\widehat\\partial\\phi(x)\n\t:=\n\t\\left\\{\n\t\t\\eta\\in\\mathbb X\\,\\middle|\\,\n\t\t\\liminf\\limits_{y\\to x}\\frac{\\phi(y)-\\phi(x)-\\langle \\eta,y-x\\rangle}{\\norm{y-x}}\\geq 0\n\t\\right\\}\n\\] \nis called the {\\em regular} (or {\\em Fr\\'{e}chet}) {\\em subdifferential} of $\\phi$ at $x$.\nFurthermore, the set\n\\[\n\t\\partial\\phi(x)\n\t:=\n\t\\left\\{\n\t\t\\eta\\in\\mathbb X\\,\\middle|\\,\n\t\t\\exists\\{x^k\\},\\{\\eta^k\\}\\subset\\mathbb X\\colon x^k\\to_\\phi x,\\,\\eta^k\\to\\eta,\\,\\eta^k\\in\\widehat{\\partial}\\phi(x^k)\\,\\forall k\\in\\N\n\t\\right\\}\n\\]\nis well known as the {\\em limiting} (or {\\em Mordukhovich}) \n{\\em subdifferential} of $\\phi$ at $x$.\nClearly, we always have $\\widehat{\\partial}\\phi(x)\\subset\\partial\\phi(x)$ by construction.\n","sha256_utf8":"49872e0fdaeef34be00bb72215ba706085612df22e13a9b3311504fa6fc04aa2"}]},{"id":"S004","semantic_type":"derived_fact","kind":"derived_fact","title":"费马必要条件","source_units":[{"start":19016,"end":19247,"start_line":445,"end_line":447,"raw_tex":"It can be seen right from the definition that whenever $x^*\\in\\dom\\phi$ is a local minimizer\nof $\\phi$, then $0\\in\\widehat\\partial\\phi(x^*)$, which is referred to as Fermat's rule,\nsee \\cite[Proposition~1.30(i)]{Mordukhovich2018}.\n","sha256_utf8":"2e749ef61db371db290c6bc84a63457de4d39df8643eeb4f33f0b344a020457d"}]},{"id":"S005","semantic_type":"derived_fact","kind":"derived_fact","title":"极限次微分稳健性","source_units":[{"start":19248,"end":19610,"start_line":449,"end_line":458,"raw_tex":"Given $ x \\in \\dom\\phi $, the limiting subdifferential has the important robustness\nproperty \n\\begin{equation}\\label{Eq:robustness}\n   \\left\\{ \n   \t\\eta\\in\\mathbb X\\,\\middle|\\,\n   \t\\exists\\{x^k\\},\\{\\eta^k\\}\\subset\\mathbb X\\colon\\,x^k\\to_\\phi x,\\,\\eta^k\\to\\eta,\\,\\eta^k\\in\\partial\\phi(x^k)\\,\\forall k\\in\\N\n   \\right\\}\n   \\subset\n   \\partial\\phi(x),\n\\end{equation}","sha256_utf8":"ea2376846e1b0d6863e7f11b498e3aa29ebe625530fc3868cf3b82f0da24e9ac"},{"start":19611,"end":19762,"start_line":459,"end_line":460,"raw_tex":"see \\cite[Proposition~1.20]{Mordukhovich2018}. \nClearly, the converse inclusion $\\supset$ is also valid by definition of the limiting subdifferential.\n","sha256_utf8":"bebaa80fd7315daa4652ac293daedb28042f84e6cd9ae8241a7258e4b63f7ccd"}]},{"id":"S006","semantic_type":"derived_fact","kind":"derived_fact","title":"光滑项求和法则","source_units":[{"start":20383,"end":20567,"start_line":474,"end_line":480,"raw_tex":"Whenever $x\\in\\dom\\phi$ is fixed, the sum rule\n\\begin{equation}\n\t\t\\label{eq:regular_sum_rule}\n\t\t\\widehat{\\partial}(f+\\phi)(x)\n\t\t=\n\t\t\\nabla f(x)+\\widehat{\\partial}\\phi(x)\n\\end{equation}","sha256_utf8":"1af312283b215935659dac26ff6e8c24e9de0d1c08e11a551edd1ea8d871befe"},{"start":20568,"end":20626,"start_line":481,"end_line":481,"raw_tex":"holds, see \\cite[Proposition~1.30(ii)]{Mordukhovich2018}.\n","sha256_utf8":"47e4477042db2ea38c0a1379e475d48284a966d6859e3772470914453e814a55"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"极限驻点定义","source_units":[{"start":21208,"end":21671,"start_line":492,"end_line":497,"raw_tex":"Throughout the paper, a point $x^*\\in\\dom\\phi$ satisfying\n$0\\in \\nabla f (x^*)+\\partial\\phi(x^*)$ will be called a \n{\\em Mordukhovich-stationary} ({\\em M-stationary} for short) point of \n\\eqref{Eq:P} due to the appearance of the limiting subdifferential. In the literature, the name \n\\emph{limiting critical point} is used as well. We close this section with two special instances\nof problem \\eqref{Eq:P} and comment on the corresponding M-stationary conditions.\n","sha256_utf8":"742b126866cd3b52dea4bc27b66101cbc84cd22de20972e35c3a5d556c53ec61"}]},{"id":"S008","semantic_type":"algorithm","kind":"algorithm","title":"单调近端梯度算法","source_units":[{"start":24720,"end":25961,"start_line":570,"end_line":595,"raw_tex":"\\begin{algorithm}[Monotone Proximal Gradient Method]\\leavevmode\n\t\\label{Alg:MonotoneProxGrad}\n\t\\begin{algorithmic}[1]\n\t\t\\REQUIRE $ \\tau > 1, 0 < \\gamma_{\\min} \\leq  \\gamma_{\\max} < \\infty, \\delta \\in (0,1), x^0 \\in \\dom\\phi $\n\t\t\\STATE Set $k := 0$.\n\t\t\\WHILE{A suitable termination criterion is violated at iteration $ k $}\n\t\t\\STATE Choose $ \\gamma_k^0 \\in [ \\gamma_{\\min}, \\gamma_{\\max}] $.\n\t\t\\STATE\\label{step:subproblem_solve_MonotoneProxGrad} \n\t\t\tFor $ i = 0, 1, 2, \\ldots $, compute a solution $ x^{k,i} $ of\n      \t\t\\begin{equation}\\label{Eq:Subki}\n         \t\t\\min_x \\ f (x^k) + \\langle\\nabla f(x^k), x - x^k \\rangle + \\frac{\\gamma_{k,i}}{2} \\| x - x^k \\|^2 + \\phi (x), \n         \t\t\\quad x \\in \\mathbb X\n      \t\t\\end{equation}\n      \t\twith $ \\gamma_{k,i} := \\tau^i \\gamma_k^0 $, until the acceptance criterion\n      \t\t\\begin{equation}\\label{Eq:StepCrit}\n         \t\t\\psi (x^{k,i}) \\leq \n         \t\t\\psi (x^k) - \\delta \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2 \n      \t\t\\end{equation}\n      \t\tholds.\n\t\t\\STATE Denote by $ i_k := i $ the terminal value, and set $ \\gamma_k := \n      \t\t\t\\gamma_{k,i_k} $ and $ x^{k+1} := x^{k,i_k} $.\n      \t\\STATE Set $ k \\leftarrow k + 1 $.\n\t\t\\ENDWHILE\n\t\t\\RETURN $x^k$\n\t\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"92313ca355e1634f4fd6862a92f8638934ded4ffcc312815829f2b8adae2bec3"}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","title":"目标与非光滑项下界","source_units":[{"start":26025,"end":26326,"start_line":600,"end_line":605,"raw_tex":"\\begin{assumption}\\label{Ass:ProxGradMonotone}\n\\begin{enumerate}\n   \\item \\label{item:psi_bounded} The function $ \\psi $ is bounded from below on $ \\dom\\phi $.\n   \\item \\label{item:phi_bounded_affine} The function  $ \\phi $ is bounded from below by an affine function.\n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"7632b8834e3c1780b165a44b205492faf728c7df6cfafba92b4fcdb47ad63b5a"}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","title":"无限迭代约定","source_units":[{"start":26886,"end":27005,"start_line":614,"end_line":615,"raw_tex":"The subsequent convergence theory assumes implicitly that \\Cref{Alg:MonotoneProxGrad}\ngenerates an infinite sequence. \n","sha256_utf8":"53b7daa50b86720ae93ae70c0dff099c07ece1adce0bd5163d0c52e5c3354f99"}]},{"id":"S011","semantic_type":"external_fact","kind":"external_fact","title":"强制函数取到极小值","source_units":[],"declared_external":true,"declaration_text":["Every proper lower semicontinuous coercive extended-real-valued function on a finite-dimensional Euclidean space attains a global minimum."]},{"id":"S012","semantic_type":"external_fact","kind":"external_fact","title":"基础分析工具","source_units":[],"declared_external":true,"declaration_text":["A continuously differentiable real-valued function admits a first-order Taylor expansion with an o(norm of increment) remainder and satisfies the scalar mean-value theorem on line segments. Cauchy–Schwarz holds in Euclidean spaces, and every bounded monotone real sequence converges."]},{"id":"S013","semantic_type":"lemma","kind":"lemma","title":"非驻点处回溯有限","source_units":[{"start":27147,"end":27624,"start_line":619,"end_line":624,"raw_tex":"\\begin{lemma}\\label{Lem:StepsizeFinite}\nConsider a fixed iteration $ k $ of \\cref{Alg:MonotoneProxGrad}, assume that $ x^k $ is not an M-stationary point of\n\\eqref{Eq:P}, and suppose that \\Cref{Ass:ProxGradMonotone}~\\ref{item:phi_bounded_affine} holds. Then the inner \nloop in \\cref{step:subproblem_solve_MonotoneProxGrad} of \\Cref{Alg:MonotoneProxGrad} is finite, i.e., we have $ \\gamma_k = \\gamma_{k,i_k} $\nfor some finite index $ i_k \\in \\{ 0, 1, 2, \\ldots \\} $.\n\\end{lemma}","sha256_utf8":"7157e4613e2fbfa961118a791bfd453c2d77db114d039153ade6a1d44a84654b"}]},{"id":"S014","semantic_type":"proof","kind":"proof","title":"回溯有限性完整证明","source_units":[{"start":27626,"end":31082,"start_line":626,"end_line":710,"raw_tex":"\\begin{proof}\n\tSuppose that the inner loop of \\cref{Alg:MonotoneProxGrad} does not\n\tterminate after a finite number of steps in iteration $k$.\nRecall that $ x^{k,i} $ is a solution of \\eqref{Eq:Subki}. Therefore, we get\n\\begin{equation}\\label{Eq:Sub1i}\n   \\langle \\nabla f(x^k),x^{k,i} - x^k\\rangle +    \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2 + \\phi (x^{k,i}) \\leq \\phi (x^k).\n\\end{equation}\nNoting that $ \\gamma_{k,i} \\to \\infty $ for $ i \\to \\infty $ and using \n\\Cref{Ass:ProxGradMonotone}~\\ref{item:phi_bounded_affine}, we obtain $ x^{k,i} \\to x^k $ for $ i \\to \\infty $. \nTaking the limit $ i \\to \\infty $ therefore yields\n\\begin{equation*}\n   \\phi (x^k) \\leq \\liminf_{i \\to \\infty} \\phi (x^{k,i}) \\leq \n   \\limsup_{i \\to \\infty} \\phi (x^{k,i}) \\leq \\phi (x^k),\n\\end{equation*}\nwhere the first estimate follows from the lower semicontinuity of $ \\phi $\nand the final inequality is a consequence of \\eqref{Eq:Sub1i}.\nTherefore, we have\n\\begin{equation}\\label{eq:robustsum}\n   \\phi (x^{k,i}) \\to \\phi (x^k) \\quad\n   \\text{for } i \\to \\infty .\n\\end{equation}\nWe claim that\n\\begin{equation}\\label{Eq:Sub2i}\n   \\liminf_{i \\to \\infty} \\gamma_{k,i} \\| x^{k,i} - x^k \\| > 0.\n\\end{equation}\nAssume, by contradiction, that there is a subsequence $ i_l \\to \\infty $ such that \n\\begin{equation}\\label{Eq:Sub3i}\n   \\liminf_{l \\to \\infty} \\gamma_{k, i_l} \\| x^{k, i_l} - x^k \\| = 0. \n\\end{equation}\nSince $ x^{k,i_l} $ is optimal for \\eqref{Eq:Subki}, Fermat's rule and the sum rule\n\\eqref{eq:regular_sum_rule} yield\n\\begin{equation}\\label{eq:optimality_condition_subproblem}\n   0 \\in  \\nabla f (x^k) + \\gamma_{k, i_l} ( x^{k, i_l} - x^k ) +\n   \\widehat\\partial \\phi (x^{k,i_l})\n\\end{equation}\nfor all $l\\in\\N$.\nTaking the limit $ l \\to \\infty $ while using \\eqref{eq:robustsum} and \\eqref{Eq:Sub3i},\nwe obtain\n\\begin{equation*}\n   0 \\in  \\nabla f (x^k) + \\partial \\phi (x^k),\n\\end{equation*}\nwhich means that $ x^k $ is already an M-stationary point of \\eqref{Eq:P}. This \ncontradiction shows that \\eqref{Eq:Sub2i} holds. Hence, there is a constant $ c > 0 $\nsuch that\n\\begin{equation*}\n   \\gamma_{k,i} \\| x^{k,i} - x^k \\| \\geq c \n\\end{equation*}\nholds for all large enough $i\\in\\N$.\nIn particular, this implies\n\\begin{equation}\\label{Eq:Sub4i}\n   ( 1- \\delta ) \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2 \\geq \n   \\frac{1- \\delta}{2} c \\| x^{k,i} - x^k \\| \\geq o \\big( \\| x^{k,i} - x^k \\| \\big)\n\\end{equation}\nfor all sufficiently large $i\\in\\N$.\nFurthermore, \\eqref{Eq:Sub1i} shows that\n\\begin{equation}\\label{Eq:Sub5i}\n\t\\langle \\nabla f (x^k),x^{k,i} - x^k\\rangle +\n   \\phi \\big( x^{k,i} \\big) - \\phi (x^k)\n   \\leq\n   - \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2 .\n\\end{equation}\nUsing a Taylor expansion of the \n\tfunction $ f $ \nand exploiting \\eqref{Eq:Sub4i},\n\\eqref{Eq:Sub5i}, we obtain \n\\begin{align*}\n   \\psi ( x^{k,i}) - \\psi (x^k)\n   & =  \n   f ( x^{k,i}) + \\phi (x^{k,i}) - f (x^k) - \\phi (x^k) \\\\\n   & =  \n   \\langle \\nabla f (x^k),x^{k,i}-x^k\\rangle + \\phi ( x^{k,i} ) - \\phi (x^k) +\n   \t\to \\big( \\| x^{k,i} - x^k \\| \\big) \\\\\n   & \\leq  \n   - \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2 + \n   \t\to \\big( \\| x^{k,i} - x^k \\| \\big) \\\\\n   & \\leq  \n   - \\delta \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2\n\\end{align*}\nfor all $ i \\in \\N $ sufficiently large.\n\tThis, however, means that the acceptance criterion \n\t\\eqref{Eq:StepCrit} is valid for sufficiently\n\tlarge $i\\in\\N$,\n\tcontradicting our assumption.\n\tThis completes the proof.\n\\end{proof}","sha256_utf8":"2297bcadd4a3b193bc907a29a9f0d66cc246fd8d78800a09ea402b7ead1ddb1a"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"无限回溯与模型比较","source_units":[{"start":27626,"end":28024,"start_line":626,"end_line":632,"raw_tex":"\\begin{proof}\n\tSuppose that the inner loop of \\cref{Alg:MonotoneProxGrad} does not\n\tterminate after a finite number of steps in iteration $k$.\nRecall that $ x^{k,i} $ is a solution of \\eqref{Eq:Subki}. Therefore, we get\n\\begin{equation}\\label{Eq:Sub1i}\n   \\langle \\nabla f(x^k),x^{k,i} - x^k\\rangle +    \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2 + \\phi (x^{k,i}) \\leq \\phi (x^k).\n\\end{equation}\n","sha256_utf8":"3a85caafa423498b47664df41a666b7c7fa37f46ad67a6771d715ee3fc13515f"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"试探点及函数值收敛","source_units":[{"start":28024,"end":28692,"start_line":633,"end_line":646,"raw_tex":"Noting that $ \\gamma_{k,i} \\to \\infty $ for $ i \\to \\infty $ and using \n\\Cref{Ass:ProxGradMonotone}~\\ref{item:phi_bounded_affine}, we obtain $ x^{k,i} \\to x^k $ for $ i \\to \\infty $. \nTaking the limit $ i \\to \\infty $ therefore yields\n\\begin{equation*}\n   \\phi (x^k) \\leq \\liminf_{i \\to \\infty} \\phi (x^{k,i}) \\leq \n   \\limsup_{i \\to \\infty} \\phi (x^{k,i}) \\leq \\phi (x^k),\n\\end{equation*}\nwhere the first estimate follows from the lower semicontinuity of $ \\phi $\nand the final inequality is a consequence of \\eqref{Eq:Sub1i}.\nTherefore, we have\n\\begin{equation}\\label{eq:robustsum}\n   \\phi (x^{k,i}) \\to \\phi (x^k) \\quad\n   \\text{for } i \\to \\infty .\n\\end{equation}\n","sha256_utf8":"7e8c25caa8a663a7cd91067d0fbf30c5b9ad2bb43253c1c3c833dfa9b546c9ea"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"非驻点给出残差下界","source_units":[{"start":28692,"end":29807,"start_line":647,"end_line":673,"raw_tex":"We claim that\n\\begin{equation}\\label{Eq:Sub2i}\n   \\liminf_{i \\to \\infty} \\gamma_{k,i} \\| x^{k,i} - x^k \\| > 0.\n\\end{equation}\nAssume, by contradiction, that there is a subsequence $ i_l \\to \\infty $ such that \n\\begin{equation}\\label{Eq:Sub3i}\n   \\liminf_{l \\to \\infty} \\gamma_{k, i_l} \\| x^{k, i_l} - x^k \\| = 0. \n\\end{equation}\nSince $ x^{k,i_l} $ is optimal for \\eqref{Eq:Subki}, Fermat's rule and the sum rule\n\\eqref{eq:regular_sum_rule} yield\n\\begin{equation}\\label{eq:optimality_condition_subproblem}\n   0 \\in  \\nabla f (x^k) + \\gamma_{k, i_l} ( x^{k, i_l} - x^k ) +\n   \\widehat\\partial \\phi (x^{k,i_l})\n\\end{equation}\nfor all $l\\in\\N$.\nTaking the limit $ l \\to \\infty $ while using \\eqref{eq:robustsum} and \\eqref{Eq:Sub3i},\nwe obtain\n\\begin{equation*}\n   0 \\in  \\nabla f (x^k) + \\partial \\phi (x^k),\n\\end{equation*}\nwhich means that $ x^k $ is already an M-stationary point of \\eqref{Eq:P}. This \ncontradiction shows that \\eqref{Eq:Sub2i} holds. Hence, there is a constant $ c > 0 $\nsuch that\n\\begin{equation*}\n   \\gamma_{k,i} \\| x^{k,i} - x^k \\| \\geq c \n\\end{equation*}\nholds for all large enough $i\\in\\N$.\n","sha256_utf8":"93982d905a310c3110261bda134dc2b547db06c67145c871f83e1a327e7afc54"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"二次下降控制余项","source_units":[{"start":29807,"end":30309,"start_line":674,"end_line":686,"raw_tex":"In particular, this implies\n\\begin{equation}\\label{Eq:Sub4i}\n   ( 1- \\delta ) \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2 \\geq \n   \\frac{1- \\delta}{2} c \\| x^{k,i} - x^k \\| \\geq o \\big( \\| x^{k,i} - x^k \\| \\big)\n\\end{equation}\nfor all sufficiently large $i\\in\\N$.\nFurthermore, \\eqref{Eq:Sub1i} shows that\n\\begin{equation}\\label{Eq:Sub5i}\n\t\\langle \\nabla f (x^k),x^{k,i} - x^k\\rangle +\n   \\phi \\big( x^{k,i} \\big) - \\phi (x^k)\n   \\leq\n   - \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2 .\n\\end{equation}\n","sha256_utf8":"7bb4ffea677f649449250354fec5a71e8d03532f3dd27ceea43de30a90672514"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"泰勒展开推出接受矛盾","source_units":[{"start":30309,"end":31082,"start_line":687,"end_line":710,"raw_tex":"Using a Taylor expansion of the \n\tfunction $ f $ \nand exploiting \\eqref{Eq:Sub4i},\n\\eqref{Eq:Sub5i}, we obtain \n\\begin{align*}\n   \\psi ( x^{k,i}) - \\psi (x^k)\n   & =  \n   f ( x^{k,i}) + \\phi (x^{k,i}) - f (x^k) - \\phi (x^k) \\\\\n   & =  \n   \\langle \\nabla f (x^k),x^{k,i}-x^k\\rangle + \\phi ( x^{k,i} ) - \\phi (x^k) +\n   \t\to \\big( \\| x^{k,i} - x^k \\| \\big) \\\\\n   & \\leq  \n   - \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2 + \n   \t\to \\big( \\| x^{k,i} - x^k \\| \\big) \\\\\n   & \\leq  \n   - \\delta \\frac{\\gamma_{k,i}}{2} \\| x^{k,i} - x^k \\|^2\n\\end{align*}\nfor all $ i \\in \\N $ sufficiently large.\n\tThis, however, means that the acceptance criterion \n\t\\eqref{Eq:StepCrit} is valid for sufficiently\n\tlarge $i\\in\\N$,\n\tcontradicting our assumption.\n\tThis completes the proof.\n\\end{proof}","sha256_utf8":"3efa286ed9b12cb3fa761fb1374d62b4dacc1a9b9f31c19e7d9602dae143a101"}]},{"id":"S020","semantic_type":"proposition","kind":"proposition","title":"相邻迭代差趋零","source_units":[{"start":32301,"end":32510,"start_line":736,"end_line":740,"raw_tex":"\\begin{proposition}\\label{Prop:xdiff}\nLet \\Cref{Ass:ProxGradMonotone} hold. Then each sequence \n$ \\{ x^k \\} $ generated by \\Cref{Alg:MonotoneProxGrad} satisfies\n$ \\| x^{k+1} - x^k \\| \\to 0 $.\n\\end{proposition}","sha256_utf8":"792d2db8cc7192bf608af00f247e62869b9a80cf51fa31492b13d2067aea5e10"}]},{"id":"S021","semantic_type":"proof","kind":"proof","title":"迭代差趋零完整证明","source_units":[{"start":32512,"end":33417,"start_line":742,"end_line":758,"raw_tex":"\\begin{proof}\nFirst recall that the sequence $ \\{ x^k \\} $ is well-defined by \\Cref{Lem:StepsizeFinite}.\nUsing the acceptance criterion \\eqref{Eq:StepCrit}, we get\n\\begin{equation}\\label{Eq:xdiff-1}\n   \\psi (x^{k+1}) \\leq \\psi (x^k) - \\delta \\frac{\\gamma_k}{2} \\| x^{k+1} - x^k \\|^2 \n   \\leq \\psi (x^k)\n\\end{equation}\nfor all $ k \\in \\mathbb{{N}} $. Hence, the sequence $ \\{ \\psi (x^k) \\} $ is \nmonotonically decreasing. Since $ \\psi $ is bounded from below on $ \\dom\\phi $\nby \\Cref{Ass:ProxGradMonotone}~\\ref{item:psi_bounded} and $ \\{ x^k \\} \\subset \\dom\\phi $, it follows that\nthis sequence is convergent. Therefore, \\eqref{Eq:xdiff-1} implies\n\\begin{equation*}\n   \\gamma_k \\| x^{k+1} - x^k \\|^2 \\to 0 \\quad \\text{for } k \\to \\infty .\n\\end{equation*}\nHence the assertion follows from the fact that, by \nconstruction, we have $ \\gamma_k \\geq \\gamma_{\\min} > 0 $ for all $ k \\in \\mathbb{N} $.\n\\end{proof}","sha256_utf8":"1d8229ded74d13ef93f297b78cf6044d65de9d36e6095c809e3d90c74f1ff9a7"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"单调下降与参数下界","source_units":[{"start":32512,"end":33417,"start_line":742,"end_line":758,"raw_tex":"\\begin{proof}\nFirst recall that the sequence $ \\{ x^k \\} $ is well-defined by \\Cref{Lem:StepsizeFinite}.\nUsing the acceptance criterion \\eqref{Eq:StepCrit}, we get\n\\begin{equation}\\label{Eq:xdiff-1}\n   \\psi (x^{k+1}) \\leq \\psi (x^k) - \\delta \\frac{\\gamma_k}{2} \\| x^{k+1} - x^k \\|^2 \n   \\leq \\psi (x^k)\n\\end{equation}\nfor all $ k \\in \\mathbb{{N}} $. Hence, the sequence $ \\{ \\psi (x^k) \\} $ is \nmonotonically decreasing. Since $ \\psi $ is bounded from below on $ \\dom\\phi $\nby \\Cref{Ass:ProxGradMonotone}~\\ref{item:psi_bounded} and $ \\{ x^k \\} \\subset \\dom\\phi $, it follows that\nthis sequence is convergent. Therefore, \\eqref{Eq:xdiff-1} implies\n\\begin{equation*}\n   \\gamma_k \\| x^{k+1} - x^k \\|^2 \\to 0 \\quad \\text{for } k \\to \\infty .\n\\end{equation*}\nHence the assertion follows from the fact that, by \nconstruction, we have $ \\gamma_k \\geq \\gamma_{\\min} > 0 $ for all $ k \\in \\mathbb{N} $.\n\\end{proof}","sha256_utf8":"1d8229ded74d13ef93f297b78cf6044d65de9d36e6095c809e3d90c74f1ff9a7"}]},{"id":"S023","semantic_type":"proposition","kind":"proposition","title":"聚点子列加权残差趋零","source_units":[{"start":33467,"end":33764,"start_line":762,"end_line":766,"raw_tex":"\\begin{proposition}\\label{Prop:gammaxdiff}\nLet \\Cref{Ass:ProxGradMonotone} hold, let $\\{x^k\\}$ be a sequence generated by \\cref{Alg:MonotoneProxGrad}, \nand let $ \\{ x^k \\}_K $ be a subsequence converging\nto some point $ x^* $. Then $ \\gamma_k \\| x^{k+1} - x^k \\| \\to_K 0 $ holds.\n\\end{proposition}","sha256_utf8":"8e67accfe8616e42959bfb5792621a203686526c03b15f1a2cf824d5feccb5c4"}]},{"id":"S024","semantic_type":"proof","kind":"proof","title":"加权残差完整证明","source_units":[{"start":33766,"end":38986,"start_line":768,"end_line":874,"raw_tex":"\\begin{proof}\nIf the subsequence $ \\{ \\gamma_k \\}_K $ is bounded, the statement follows\nimmediately from \\Cref{Prop:xdiff}. The remaining part of this proof\ntherefore assumes that this subsequence is unbounded. Without loss of \ngenerality, we may assume that $ \\gamma_k  \\to_K \\infty $\n\tand that the acceptance criterion \\eqref{Eq:StepCrit} is violated\n\tin the first iteration of the inner loop for each $k\\in K$.\nThen, for $ \\hat{\\gamma}_k := \\gamma_k / \\tau $, $k\\in K$, we also have $\\hat\\gamma_k\\to_K\\infty$, but the \ncorresponding vector $ \\hat x^k := x^{k, i_k-1} $ does not satisfy the stepsize\ncondition from \\eqref{Eq:StepCrit}, i.e., we have\n\\begin{equation}\\label{Prop:gammaxdiff-1}\n   \\psi (\\hat x^k) > \n   \\psi (x^k) - \\delta \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2 \n   \\quad \\forall k \\in K.\n\\end{equation}\nOn the other hand, since $ \\hat{x}^k $ solves the corresponding \nsubproblem \\eqref{Eq:Subki} with $\\hat{\\gamma}_k = \\gamma_{k,i_k-1}$, we have\n\\begin{equation}\\label{Prop:gammaxdiff-2}\n    \\langle \\nabla f (x^k), \\hat{x}^k - x^k\\rangle + \\frac{\\hat{\\gamma}_k}{2}\n   \\| \\hat{x}^k - x^k \\|^2 + \\phi ( \\hat{x}^k ) - \\phi (x^k) \\leq 0 .\n\\end{equation}\nWe claim that this, in particular, implies $ \\hat{x}^k \\to_K x^* $. In fact, using\n\\eqref{Prop:gammaxdiff-2}, the Cauchy-Schwarz inequality, and the monotonicity of\n$ \\{ \\psi (x^k) \\} $, we obtain\n\\begin{align*}\n   \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2 \n   & \\leq \n   \\| \\nabla f (x^k) \\| \\| \\hat{x}^k - x^k \\| + \\phi (x^k) - \\phi ( \\hat{x}^k ) \\\\\n   & = \n   \\| \\nabla f (x^k) \\| \\| \\hat{x}^k - x^k \\| + \\psi (x^k) - f (x^k) - \\phi ( \\hat{x}^k ) \\\\\n   & \\leq \n   \\| \\nabla f (x^k) \\| \\| \\hat{x}^k - x^k \\| + \\psi (x^0) - f (x^k) - \\phi ( \\hat{x}^k ) .\n\\end{align*}\nSince $ f $ is continuously differentiable and $ - \\phi $ is bounded from above by \nan affine function in view of \\Cref{Ass:ProxGradMonotone}~\\ref{item:phi_bounded_affine}, this implies \n$ \\| \\hat{x}^k - x^k \\| \\to_K 0 $.\nIn fact, if $ \\{ \\| \\hat{x}^k - x^k \\| \\}_K $ would\nbe unbounded, then the left-hand side would grow more rapidly than the right-hand side,\nand if $ \\{ \\| \\hat{x}^k - x^k \\| \\}_K $ would be bounded, but staying away, at least\non a subsequence, from zero by a positive number, the right-hand side would be bounded, \nwhereas the left-hand side would be unbounded on the corresponding subsequence.\n\nNow, by the mean-value theorem, there exists $ \\xi^k $ on the line segment connecting\n$ x^k $ with $ \\hat{x}^k $ such that\n\\begin{equation}\\label{Prop:gammaxdiff-3}\n\t\\begin{aligned}\n   \t\t\\psi ( \\hat{x}^k ) - \\psi (x^k) \n   \t\t&= \n   \t\tf ( \\hat{x}^k ) + \\phi (\\hat{x}^k) -   f (x^k) - \\phi (x^k) \\\\\n   \t\t&= \n   \t\t\\langle  \\nabla f (\\xi^k),\\hat{x}^k - x^k \\rangle + \\phi (\\hat{x}^k) - \\phi (x^k) .\n   \\end{aligned}\n\\end{equation}\nSubstituting the expression $ \\phi (\\hat{x}^k) - \\phi (x^k) $ from \\eqref{Prop:gammaxdiff-3}\ninto \\eqref{Prop:gammaxdiff-2} yields \n\\begin{equation*}\n   \\langle \\nabla f(x^k) - \\nabla f ( \\xi^k) ,\\hat x^k - x^k \\rangle +\n   \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2 + \\psi ( \\hat{x}^k ) - \\psi (x^k) \\leq 0.\n\\end{equation*}\nExploiting \\eqref{Prop:gammaxdiff-1}, we therefore obtain \n\\begin{align*}\n   \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2\n   & \\leq \n   - \\langle \\nabla f(x^k) - \\nabla f ( \\xi^k), \\hat x^k - x^k \\rangle +  \\psi (x^k) - \\psi ( \\hat{x}^k ) \\\\\n   & \\leq\n   \\| \\nabla f (x^k) - \\nabla f ( \\xi^k) \\| \\| \\hat x^k - x^k \\| + \\delta \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2 ,\n\\end{align*}\nwhich can be rewritten as\n\\begin{equation}\\label{Prop:gammaxdiff-4}\n   ( 1 - \\delta ) \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\| \\leq \n   \\| \\nabla f (x^k) - \\nabla f ( \\xi^k) \\| \n\\end{equation}\n(note that $ \\hat{x}^k \\neq x^k $ in view of \\eqref{Prop:gammaxdiff-1}).\nSince $ x^k\\to_K x^* $ (by assumption) and $ \\hat{x}^k \\to_K x^* $ (by the\nprevious part of this proof), we also get $ \\xi^k \\to_K x^* $. Using $ \\delta \\in (0,1) $\nand the continuous differentiability of $ f $, it follows from \\eqref{Prop:gammaxdiff-4}\nthat $ \\hat{\\gamma}_k \\| \\hat{x}^k - x^k \\| \\to_K 0 $.\n\nFinally, exploiting the fact that $ x^{k+1} $ and $ \\hat{x}^k $ are solutions of\nthe subproblems \\eqref{Eq:Subki} with parameters $ \\gamma_k $ and $ \\hat{\\gamma}_k $,\nrespectively, we find\n\\begin{align*}\n   \\langle \\nabla f(x^k),x^{k+1} - x^k\\rangle \n   &+\n   \\frac{\\gamma_k}{2} \\| x^{k+1} - x^k \\|^2 + \\phi (x^{k+1})\n   \\\\\n   &\n   \\leq \\langle \\nabla f (x^k),  \\hat{x}^k - x^k \\rangle +\n   \\frac{\\gamma_k}{2} \\| \\hat{x}^k - x^k \\|^2 + \\phi (\\hat{x}^k),\n   \\\\\n    \\langle \\nabla f(x^k), \\hat{x}^k - x^k \\rangle \n    &+\n   \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2 + \\phi (\\hat{x}^k)\n   \\\\\n   &\n   \\leq  \\langle \\nabla f(x^k),  x^{k+1} - x^k \\rangle +\n   \\frac{\\hat{\\gamma}_k}{2} \\| x^{k+1} - x^k \\|^2 + \\phi (x^{k+1}).\n\\end{align*}\nAdding these two inequalities and noting that $ \\gamma_k = \\tau \\hat{\\gamma}_k >\n\\hat{\\gamma}_k $ yields $ \\| x^{k+1} - x^k \\| \\leq \\| \\hat{x}^k - x^k \\| $ and,\ntherefore,\n\\begin{equation*}\n   \\gamma_k \\| x^{k+1}- x^k \\| = \\tau \\hat{\\gamma}_k \\| x^{k+1} - x^k \\| \\leq\n   \\tau \\hat{\\gamma}_k \\| \\hat{x}^k - x^k \\| \\to_K 0.\n\\end{equation*}\nThis completes the proof.\n\\end{proof}","sha256_utf8":"442c39c9657cf1ddb56ff854e2d10614a020ce11e2415c7dad4d55af17f13978"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"参数分情形与前次拒绝","source_units":[{"start":33766,"end":34944,"start_line":768,"end_line":788,"raw_tex":"\\begin{proof}\nIf the subsequence $ \\{ \\gamma_k \\}_K $ is bounded, the statement follows\nimmediately from \\Cref{Prop:xdiff}. The remaining part of this proof\ntherefore assumes that this subsequence is unbounded. Without loss of \ngenerality, we may assume that $ \\gamma_k  \\to_K \\infty $\n\tand that the acceptance criterion \\eqref{Eq:StepCrit} is violated\n\tin the first iteration of the inner loop for each $k\\in K$.\nThen, for $ \\hat{\\gamma}_k := \\gamma_k / \\tau $, $k\\in K$, we also have $\\hat\\gamma_k\\to_K\\infty$, but the \ncorresponding vector $ \\hat x^k := x^{k, i_k-1} $ does not satisfy the stepsize\ncondition from \\eqref{Eq:StepCrit}, i.e., we have\n\\begin{equation}\\label{Prop:gammaxdiff-1}\n   \\psi (\\hat x^k) > \n   \\psi (x^k) - \\delta \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2 \n   \\quad \\forall k \\in K.\n\\end{equation}\nOn the other hand, since $ \\hat{x}^k $ solves the corresponding \nsubproblem \\eqref{Eq:Subki} with $\\hat{\\gamma}_k = \\gamma_{k,i_k-1}$, we have\n\\begin{equation}\\label{Prop:gammaxdiff-2}\n    \\langle \\nabla f (x^k), \\hat{x}^k - x^k\\rangle + \\frac{\\hat{\\gamma}_k}{2}\n   \\| \\hat{x}^k - x^k \\|^2 + \\phi ( \\hat{x}^k ) - \\phi (x^k) \\leq 0 .\n\\end{equation}\n","sha256_utf8":"eb66f8857572e7cda884c4cf2ec95f2f16b4152b3f0c3d5ae0b4a3ba25e6ffa1"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"前次试探点趋向聚点","source_units":[{"start":34944,"end":36137,"start_line":789,"end_line":808,"raw_tex":"We claim that this, in particular, implies $ \\hat{x}^k \\to_K x^* $. In fact, using\n\\eqref{Prop:gammaxdiff-2}, the Cauchy-Schwarz inequality, and the monotonicity of\n$ \\{ \\psi (x^k) \\} $, we obtain\n\\begin{align*}\n   \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2 \n   & \\leq \n   \\| \\nabla f (x^k) \\| \\| \\hat{x}^k - x^k \\| + \\phi (x^k) - \\phi ( \\hat{x}^k ) \\\\\n   & = \n   \\| \\nabla f (x^k) \\| \\| \\hat{x}^k - x^k \\| + \\psi (x^k) - f (x^k) - \\phi ( \\hat{x}^k ) \\\\\n   & \\leq \n   \\| \\nabla f (x^k) \\| \\| \\hat{x}^k - x^k \\| + \\psi (x^0) - f (x^k) - \\phi ( \\hat{x}^k ) .\n\\end{align*}\nSince $ f $ is continuously differentiable and $ - \\phi $ is bounded from above by \nan affine function in view of \\Cref{Ass:ProxGradMonotone}~\\ref{item:phi_bounded_affine}, this implies \n$ \\| \\hat{x}^k - x^k \\| \\to_K 0 $.\nIn fact, if $ \\{ \\| \\hat{x}^k - x^k \\| \\}_K $ would\nbe unbounded, then the left-hand side would grow more rapidly than the right-hand side,\nand if $ \\{ \\| \\hat{x}^k - x^k \\| \\}_K $ would be bounded, but staying away, at least\non a subsequence, from zero by a positive number, the right-hand side would be bounded, \nwhereas the left-hand side would be unbounded on the corresponding subsequence.\n","sha256_utf8":"861f40fc419ea6a9968ebcb3e1201f27d89bce91807a04ce912e29cfd97c13d9"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"中值公式改写模型估计","source_units":[{"start":36138,"end":36896,"start_line":810,"end_line":826,"raw_tex":"Now, by the mean-value theorem, there exists $ \\xi^k $ on the line segment connecting\n$ x^k $ with $ \\hat{x}^k $ such that\n\\begin{equation}\\label{Prop:gammaxdiff-3}\n\t\\begin{aligned}\n   \t\t\\psi ( \\hat{x}^k ) - \\psi (x^k) \n   \t\t&= \n   \t\tf ( \\hat{x}^k ) + \\phi (\\hat{x}^k) -   f (x^k) - \\phi (x^k) \\\\\n   \t\t&= \n   \t\t\\langle  \\nabla f (\\xi^k),\\hat{x}^k - x^k \\rangle + \\phi (\\hat{x}^k) - \\phi (x^k) .\n   \\end{aligned}\n\\end{equation}\nSubstituting the expression $ \\phi (\\hat{x}^k) - \\phi (x^k) $ from \\eqref{Prop:gammaxdiff-3}\ninto \\eqref{Prop:gammaxdiff-2} yields \n\\begin{equation*}\n   \\langle \\nabla f(x^k) - \\nabla f ( \\xi^k) ,\\hat x^k - x^k \\rangle +\n   \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2 + \\psi ( \\hat{x}^k ) - \\psi (x^k) \\leq 0.\n\\end{equation*}\n","sha256_utf8":"6f84c598a6a6e6be75b061dcf097f89db1ae2a51141d0ac56d35210b66971458"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"拒绝条件迫使试探残差趋零","source_units":[{"start":36896,"end":37871,"start_line":827,"end_line":844,"raw_tex":"Exploiting \\eqref{Prop:gammaxdiff-1}, we therefore obtain \n\\begin{align*}\n   \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2\n   & \\leq \n   - \\langle \\nabla f(x^k) - \\nabla f ( \\xi^k), \\hat x^k - x^k \\rangle +  \\psi (x^k) - \\psi ( \\hat{x}^k ) \\\\\n   & \\leq\n   \\| \\nabla f (x^k) - \\nabla f ( \\xi^k) \\| \\| \\hat x^k - x^k \\| + \\delta \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2 ,\n\\end{align*}\nwhich can be rewritten as\n\\begin{equation}\\label{Prop:gammaxdiff-4}\n   ( 1 - \\delta ) \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\| \\leq \n   \\| \\nabla f (x^k) - \\nabla f ( \\xi^k) \\| \n\\end{equation}\n(note that $ \\hat{x}^k \\neq x^k $ in view of \\eqref{Prop:gammaxdiff-1}).\nSince $ x^k\\to_K x^* $ (by assumption) and $ \\hat{x}^k \\to_K x^* $ (by the\nprevious part of this proof), we also get $ \\xi^k \\to_K x^* $. Using $ \\delta \\in (0,1) $\nand the continuous differentiability of $ f $, it follows from \\eqref{Prop:gammaxdiff-4}\nthat $ \\hat{\\gamma}_k \\| \\hat{x}^k - x^k \\| \\to_K 0 $.\n","sha256_utf8":"b1126b8c2d6c2de03a542a2377f3a8bba42a8eb6b8bbdf863e96db327e86af97"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"双重最优比较传递残差","source_units":[{"start":37872,"end":38986,"start_line":846,"end_line":874,"raw_tex":"Finally, exploiting the fact that $ x^{k+1} $ and $ \\hat{x}^k $ are solutions of\nthe subproblems \\eqref{Eq:Subki} with parameters $ \\gamma_k $ and $ \\hat{\\gamma}_k $,\nrespectively, we find\n\\begin{align*}\n   \\langle \\nabla f(x^k),x^{k+1} - x^k\\rangle \n   &+\n   \\frac{\\gamma_k}{2} \\| x^{k+1} - x^k \\|^2 + \\phi (x^{k+1})\n   \\\\\n   &\n   \\leq \\langle \\nabla f (x^k),  \\hat{x}^k - x^k \\rangle +\n   \\frac{\\gamma_k}{2} \\| \\hat{x}^k - x^k \\|^2 + \\phi (\\hat{x}^k),\n   \\\\\n    \\langle \\nabla f(x^k), \\hat{x}^k - x^k \\rangle \n    &+\n   \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\|^2 + \\phi (\\hat{x}^k)\n   \\\\\n   &\n   \\leq  \\langle \\nabla f(x^k),  x^{k+1} - x^k \\rangle +\n   \\frac{\\hat{\\gamma}_k}{2} \\| x^{k+1} - x^k \\|^2 + \\phi (x^{k+1}).\n\\end{align*}\nAdding these two inequalities and noting that $ \\gamma_k = \\tau \\hat{\\gamma}_k >\n\\hat{\\gamma}_k $ yields $ \\| x^{k+1} - x^k \\| \\leq \\| \\hat{x}^k - x^k \\| $ and,\ntherefore,\n\\begin{equation*}\n   \\gamma_k \\| x^{k+1}- x^k \\| = \\tau \\hat{\\gamma}_k \\| x^{k+1} - x^k \\| \\leq\n   \\tau \\hat{\\gamma}_k \\| \\hat{x}^k - x^k \\| \\to_K 0.\n\\end{equation*}\nThis completes the proof.\n\\end{proof}","sha256_utf8":"0d2a499e506d6a362e87b062e091996a3adcef2ff51a0c12d1a1bb238cc94d24"}]},{"id":"S030","semantic_type":"corollary","kind":"corollary","title":"局部利普希茨保证参数有界","source_units":[{"start":39257,"end":39667,"start_line":880,"end_line":885,"raw_tex":"\\begin{corollary}\\label{Cor:LipschitzCase}\nLet \\Cref{Ass:ProxGradMonotone} hold, let $\\{x^k\\}$ be a sequence generated by \\Cref{Alg:MonotoneProxGrad}, let $ \\{ x^k \\}_K $ be a subsequence converging\nto some point $ x^* $, and assume that $ \\nabla f \\colon \\mathbb X\\to\\mathbb X$ is locally Lipschitz continuous\naround $ x^* $. Then the corresponding subsequence $ \\{ \\gamma_k \\}_K $ is\nbounded.\n\\end{corollary}","sha256_utf8":"31ae025e7a6383c682d1b9569d2a1432dc974a2699ee44bc693bb6f8df6bc4db"}]},{"id":"S031","semantic_type":"proof","kind":"proof","title":"参数有界性完整证明","source_units":[{"start":39669,"end":40631,"start_line":887,"end_line":904,"raw_tex":"\\begin{proof}\nWe may argue as in the proof of \\cref{Prop:gammaxdiff}. \nHence, \n\ton the contrary, assume\nthat $ \\gamma_k \\to_K \\infty $. For each $k\\in K$, define $ \\hat{\\gamma}_k $ and $ \\hat{x}^k $ as in\nthat proof, and let $ L > 0 $ denote the local Lipschitz constant of $ \\nabla f $ around\n$ x^* $. Recall that $ x^k \\to_K x^* $ (by assumption) and $ \\hat{x}^k \\to_K x^* $\n(from the proof of \\cref{Prop:gammaxdiff}). Exploiting \\eqref{Prop:gammaxdiff-4},\nwe therefore obtain\n\\begin{equation*}\n   ( 1 - \\delta ) \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\| \\leq L \\| \\hat{x}^k - \\xi^k \\|\n   \\leq L \\| \\hat{x}^k - x^k \\|\n\\end{equation*}\nfor all $ k \\in K $ sufficiently large, using the fact that $ \\xi^k $ is on the \nline segment between $ x^k $ and $ \\hat{x}^k $. Since $ \\hat{\\gamma}_k \\to_K \\infty $ and \n$ \\hat{x}^k \\neq x^k $, see once again \\eqref{Prop:gammaxdiff-1}, this gives a contradiction. \nHence, $ \\{ \\gamma_k \\}_K $ stays bounded.\n\\end{proof}","sha256_utf8":"962130ce6a0f9ba228a75a747bf587455a48282c345299878e47bff454413738"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"局部梯度估计排除参数发散","source_units":[{"start":39669,"end":40631,"start_line":887,"end_line":904,"raw_tex":"\\begin{proof}\nWe may argue as in the proof of \\cref{Prop:gammaxdiff}. \nHence, \n\ton the contrary, assume\nthat $ \\gamma_k \\to_K \\infty $. For each $k\\in K$, define $ \\hat{\\gamma}_k $ and $ \\hat{x}^k $ as in\nthat proof, and let $ L > 0 $ denote the local Lipschitz constant of $ \\nabla f $ around\n$ x^* $. Recall that $ x^k \\to_K x^* $ (by assumption) and $ \\hat{x}^k \\to_K x^* $\n(from the proof of \\cref{Prop:gammaxdiff}). Exploiting \\eqref{Prop:gammaxdiff-4},\nwe therefore obtain\n\\begin{equation*}\n   ( 1 - \\delta ) \\frac{\\hat{\\gamma}_k}{2} \\| \\hat{x}^k - x^k \\| \\leq L \\| \\hat{x}^k - \\xi^k \\|\n   \\leq L \\| \\hat{x}^k - x^k \\|\n\\end{equation*}\nfor all $ k \\in K $ sufficiently large, using the fact that $ \\xi^k $ is on the \nline segment between $ x^k $ and $ \\hat{x}^k $. Since $ \\hat{\\gamma}_k \\to_K \\infty $ and \n$ \\hat{x}^k \\neq x^k $, see once again \\eqref{Prop:gammaxdiff-1}, this gives a contradiction. \nHence, $ \\{ \\gamma_k \\}_K $ stays bounded.\n\\end{proof}","sha256_utf8":"962130ce6a0f9ba228a75a747bf587455a48282c345299878e47bff454413738"}]},{"id":"S033","semantic_type":"theorem","kind":"theorem","title":"单调近端梯度聚点驻定性","source_units":[{"start":40798,"end":41171,"start_line":909,"end_line":915,"raw_tex":"\\begin{theorem}\\label{Thm:ConvProxGrad}\nAssume that \\Cref{Ass:ProxGradMonotone} holds while either $ \\phi $ is continuous\non $\\dom\\phi$ or $ \\nabla f\\colon\\mathbb X\\to\\mathbb X $ is locally Lipschitz continuous.\nThen each accumulation point\n$ x^* $ of a sequence $ \\{ x^k \\} $ generated by \\Cref{Alg:MonotoneProxGrad}\nis an M-stationary point of \\eqref{Eq:P}.\n\\end{theorem}","sha256_utf8":"9f71e373f83213c321aed1f8a4b09305a2b6acd76b6b6a82a8a4066d3678dc32"}]},{"id":"S034","semantic_type":"proof","kind":"proof","title":"聚点驻定性完整证明","source_units":[{"start":41173,"end":43822,"start_line":917,"end_line":970,"raw_tex":"\\begin{proof}\nLet $ \\{ x^k \\}_K $ be a subsequence converging to $ x^* $. In view of\n\\Cref{Prop:xdiff}, it follows that also the subsequence $ \\{ x^{k+1} \\}_K $\nconverges to $ x^* $. Furthermore, \\Cref{Prop:gammaxdiff} yields\n$ \\gamma_k \\| x^{k+1} - x^k \\| \\to_K 0 $.\nThe minimizing property of $ x^{k+1} $, Fermat's rule, and the sum rule \\eqref{eq:regular_sum_rule} imply that\n\\begin{equation}\\label{Eq:OptCondSub}\n   0 \\in  \\nabla f (x^{k}) + \\gamma_k (x^{k+1}-x^k) \n   + \\widehat\\partial \\phi (x^{k+1})\n\\end{equation}\nholds for each $k\\in K$. Hence, if we can show $ \\phi (x^{k+1}) \\to_K \\phi (x^*) $, we can\ntake the limit $ k \\to_K \\infty $ in \\eqref{Eq:OptCondSub} \nto obtain the desired statement $ 0 \\in  \\nabla f (x^*) + \\partial \\phi (x^*) $.\n\n\tDue to \\eqref{Eq:xdiff-1}, we find $\\psi(x^{k+1})\\leq \\psi(x^0)$\n\tfor each $k\\in K$. Taking the limit $k\\to_K\\infty$ while respecting\n\tthe lower semicontinuity of $\\phi$ gives $\\psi(x^*)\\leq\\psi(x^0)$,\n\tand due to $x^0\\in\\dom\\phi$, we find $x^*\\in\\dom\\phi$.\nThus, the condition $ \\phi (x^{k+1}) \\to_K \\phi (x^*) $ obviously holds if\n$ \\phi $ is continuous on its domain since all iterates $ x^k $ generated by\n\\Cref{Alg:MonotoneProxGrad} \n\tas well as $x^*$\nbelong to $\\dom\\phi$.\n\nHence, it remains to consider the situation where $ \\phi $ is only lower semicontinuous, but\n$ \\nabla f $ is locally Lipschitz continuous. \nFrom $ x^{k+1} \\to_K x^* $ and the lower semicontinuity of $\\phi$, we find\n\\begin{equation*}\n   \\phi (x^*) \\leq \\liminf_{k \\in K} \\phi (x^{k+1}) \\leq \\limsup_{k \\in K} \\phi (x^{k+1}).\n\\end{equation*}\nIt therefore suffices to show that $ \\limsup_{k \\in K} \\phi (x^{k+1}) \\leq \\phi (x^*) $ holds.\nSince $ x^{k+1} $ solves the subproblem \\eqref{Eq:Subki} with parameter $ \\gamma_k $, we obtain\n\\begin{align*}\n    \\langle \\nabla f (x^k),x^{k+1}-x^k \\rangle \n    &+ \n    \\frac{\\gamma_k}{2} \\| x^{k+1} - x^k \\|^2 +  \\phi (x^{k+1} ) \n    \\\\\n    &\n    \\leq  \n    \\langle \\nabla f(x^k),x^* - x^k \\rangle \n   \t+ \n   \t\\frac{\\gamma_k}{2} \\| x^* - x^k \\|^2 + \\phi (x^*)\n\\end{align*}\nfor each $k\\in K$.\nWe now take the upper limit over $K$ on both sides. \nUsing the continuity of $ \\nabla f $, \nthe convergences $ x^{k+1} - x^k \\to_K 0 $ \nas well as $ \\gamma_k \\| x^{k+1} - x^k \\|^2 \\to_K 0 $ (see \\Cref{Prop:xdiff,Prop:gammaxdiff}),\nand taking into account that $ \\gamma_k \\| x^k - x^* \\|^2 \\to_K 0 $ due to the boundedness\nof the subsequence $ \\{ \\gamma_k \\}_K $ in this situation, see \n\\cref{Cor:LipschitzCase}, we obtain \n$ \\limsup_{k \\in K} \\phi (x^{k+1}) \\leq \\phi (x^*) $. Altogether, we\ntherefore get $ \\phi (x^{k+1}) \\to_K \\phi (x^*) $, and this completes the proof.\n\\end{proof}","sha256_utf8":"9c437ee71db4c2b8b83b72ae134c5132600b12f3cba80fa49d644482d8985f22"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"子列收敛与最优性极限目标","source_units":[{"start":41173,"end":41927,"start_line":917,"end_line":929,"raw_tex":"\\begin{proof}\nLet $ \\{ x^k \\}_K $ be a subsequence converging to $ x^* $. In view of\n\\Cref{Prop:xdiff}, it follows that also the subsequence $ \\{ x^{k+1} \\}_K $\nconverges to $ x^* $. Furthermore, \\Cref{Prop:gammaxdiff} yields\n$ \\gamma_k \\| x^{k+1} - x^k \\| \\to_K 0 $.\nThe minimizing property of $ x^{k+1} $, Fermat's rule, and the sum rule \\eqref{eq:regular_sum_rule} imply that\n\\begin{equation}\\label{Eq:OptCondSub}\n   0 \\in  \\nabla f (x^{k}) + \\gamma_k (x^{k+1}-x^k) \n   + \\widehat\\partial \\phi (x^{k+1})\n\\end{equation}\nholds for each $k\\in K$. Hence, if we can show $ \\phi (x^{k+1}) \\to_K \\phi (x^*) $, we can\ntake the limit $ k \\to_K \\infty $ in \\eqref{Eq:OptCondSub} \nto obtain the desired statement $ 0 \\in  \\nabla f (x^*) + \\partial \\phi (x^*) $.\n","sha256_utf8":"23b5895da652f26b4565f090499cc8bab26eff255e25cb6eb1004dc1de62023d"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"聚点可行与连续情形","source_units":[{"start":41928,"end":42408,"start_line":931,"end_line":939,"raw_tex":"\tDue to \\eqref{Eq:xdiff-1}, we find $\\psi(x^{k+1})\\leq \\psi(x^0)$\n\tfor each $k\\in K$. Taking the limit $k\\to_K\\infty$ while respecting\n\tthe lower semicontinuity of $\\phi$ gives $\\psi(x^*)\\leq\\psi(x^0)$,\n\tand due to $x^0\\in\\dom\\phi$, we find $x^*\\in\\dom\\phi$.\nThus, the condition $ \\phi (x^{k+1}) \\to_K \\phi (x^*) $ obviously holds if\n$ \\phi $ is continuous on its domain since all iterates $ x^k $ generated by\n\\Cref{Alg:MonotoneProxGrad} \n\tas well as $x^*$\nbelong to $\\dom\\phi$.\n","sha256_utf8":"338378a51b83cd555caa5abeec65f5c03c7dbe9a9f94d774b4fda8f5ab2bcf1f"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"半连续情形的模型比较","source_units":[{"start":42409,"end":43238,"start_line":941,"end_line":960,"raw_tex":"Hence, it remains to consider the situation where $ \\phi $ is only lower semicontinuous, but\n$ \\nabla f $ is locally Lipschitz continuous. \nFrom $ x^{k+1} \\to_K x^* $ and the lower semicontinuity of $\\phi$, we find\n\\begin{equation*}\n   \\phi (x^*) \\leq \\liminf_{k \\in K} \\phi (x^{k+1}) \\leq \\limsup_{k \\in K} \\phi (x^{k+1}).\n\\end{equation*}\nIt therefore suffices to show that $ \\limsup_{k \\in K} \\phi (x^{k+1}) \\leq \\phi (x^*) $ holds.\nSince $ x^{k+1} $ solves the subproblem \\eqref{Eq:Subki} with parameter $ \\gamma_k $, we obtain\n\\begin{align*}\n    \\langle \\nabla f (x^k),x^{k+1}-x^k \\rangle \n    &+ \n    \\frac{\\gamma_k}{2} \\| x^{k+1} - x^k \\|^2 +  \\phi (x^{k+1} ) \n    \\\\\n    &\n    \\leq  \n    \\langle \\nabla f(x^k),x^* - x^k \\rangle \n   \t+ \n   \t\\frac{\\gamma_k}{2} \\| x^* - x^k \\|^2 + \\phi (x^*)\n\\end{align*}\nfor each $k\\in K$.\n","sha256_utf8":"4daf1fa55ede72bd98bdee16ce7e944980a70677c8ca2948da147ba4471f7bc9"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"有界参数完成函数值收敛","source_units":[{"start":43238,"end":43822,"start_line":961,"end_line":970,"raw_tex":"We now take the upper limit over $K$ on both sides. \nUsing the continuity of $ \\nabla f $, \nthe convergences $ x^{k+1} - x^k \\to_K 0 $ \nas well as $ \\gamma_k \\| x^{k+1} - x^k \\|^2 \\to_K 0 $ (see \\Cref{Prop:xdiff,Prop:gammaxdiff}),\nand taking into account that $ \\gamma_k \\| x^k - x^* \\|^2 \\to_K 0 $ due to the boundedness\nof the subsequence $ \\{ \\gamma_k \\}_K $ in this situation, see \n\\cref{Cor:LipschitzCase}, we obtain \n$ \\limsup_{k \\in K} \\phi (x^{k+1}) \\leq \\phi (x^*) $. Altogether, we\ntherefore get $ \\phi (x^{k+1}) \\to_K \\phi (x^*) $, and this completes the proof.\n\\end{proof}","sha256_utf8":"03fbe5dd76faadb511658df111bc43cf9081c589d2d717c05c982a61bf59d8a9"}]},{"id":"S039","semantic_type":"assumption","kind":"assumption","title":"备选正则性假设","source_units":[{"start":40838,"end":41010,"start_line":910,"end_line":911,"raw_tex":"Assume that \\Cref{Ass:ProxGradMonotone} holds while either $ \\phi $ is continuous\non $\\dom\\phi$ or $ \\nabla f\\colon\\mathbb X\\to\\mathbb X $ is locally Lipschitz continuous.\n","sha256_utf8":"e89711fd000dd967feaa4317e9aefcbbc1f3d7457d08ffd93888908fd87be604"}]},{"id":"S040","semantic_type":"proof","kind":"proof","title":"子问题可解性论证","source_units":[{"start":26328,"end":26885,"start_line":607,"end_line":612,"raw_tex":"\\Cref{Ass:ProxGradMonotone}~\\ref{item:psi_bounded} is a reasonable condition regarding the\ngiven composite optimization problem, whereas \\Cref{Ass:ProxGradMonotone}~\\ref{item:phi_bounded_affine}\nis essentially a statement relevant for the subproblems from \\eqref{Eq:Subki}. In particular,\n\\Cref{Ass:ProxGradMonotone}~\\ref{item:phi_bounded_affine} implies that the quadratic objective function\nof the subproblems \\eqref{Eq:Subki} are, for fixed $ k, i \\in\\N$, coercive, and therefore always\nattain a solution $ x^{k,i} $ (which, however, may not be unique).\n","sha256_utf8":"21339d11278cef93ff86aff867eb55c76d7aa41a0a25e2603b1cae430af2e261"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"仿射下界推出强制与取解","source_units":[{"start":26328,"end":26885,"start_line":607,"end_line":612,"raw_tex":"\\Cref{Ass:ProxGradMonotone}~\\ref{item:psi_bounded} is a reasonable condition regarding the\ngiven composite optimization problem, whereas \\Cref{Ass:ProxGradMonotone}~\\ref{item:phi_bounded_affine}\nis essentially a statement relevant for the subproblems from \\eqref{Eq:Subki}. In particular,\n\\Cref{Ass:ProxGradMonotone}~\\ref{item:phi_bounded_affine} implies that the quadratic objective function\nof the subproblems \\eqref{Eq:Subki} are, for fixed $ k, i \\in\\N$, coercive, and therefore always\nattain a solution $ x^{k,i} $ (which, however, may not be unique).\n","sha256_utf8":"21339d11278cef93ff86aff867eb55c76d7aa41a0a25e2603b1cae430af2e261"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S033","edges":[{"source_id":"S001","target_id":"S041","kind":"supports"},{"source_id":"S001","target_id":"S016","kind":"supports"},{"source_id":"S001","target_id":"S026","kind":"supports"},{"source_id":"S001","target_id":"S036","kind":"supports"},{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S005","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S007","kind":"supports"},{"source_id":"S003","target_id":"S017","kind":"supports"},{"source_id":"S004","target_id":"S017","kind":"supports"},{"source_id":"S004","target_id":"S035","kind":"supports"},{"source_id":"S005","target_id":"S017","kind":"supports"},{"source_id":"S005","target_id":"S035","kind":"supports"},{"source_id":"S006","target_id":"S017","kind":"supports"},{"source_id":"S006","target_id":"S035","kind":"supports"},{"source_id":"S007","target_id":"S017","kind":"supports"},{"source_id":"S007","target_id":"S035","kind":"supports"},{"source_id":"S008","target_id":"S041","kind":"supports"},{"source_id":"S008","target_id":"S015","kind":"supports"},{"source_id":"S008","target_id":"S022","kind":"supports"},{"source_id":"S008","target_id":"S025","kind":"supports"},{"source_id":"S008","target_id":"S029","kind":"supports"},{"source_id":"S008","target_id":"S035","kind":"supports"},{"source_id":"S008","target_id":"S037","kind":"supports"},{"source_id":"S009","target_id":"S041","kind":"supports"},{"source_id":"S009","target_id":"S016","kind":"supports"},{"source_id":"S009","target_id":"S022","kind":"supports"},{"source_id":"S009","target_id":"S026","kind":"supports"},{"source_id":"S010","target_id":"S035","kind":"supports"},{"source_id":"S011","target_id":"S041","kind":"supports"},{"source_id":"S012","target_id":"S019","kind":"supports"},{"source_id":"S012","target_id":"S022","kind":"supports"},{"source_id":"S012","target_id":"S026","kind":"supports"},{"source_id":"S012","target_id":"S027","kind":"supports"},{"source_id":"S012","target_id":"S028","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S015","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S016","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S016","target_id":"S014","kind":"supports"},{"source_id":"S017","target_id":"S014","kind":"supports"},{"source_id":"S018","target_id":"S014","kind":"supports"},{"source_id":"S019","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S025","kind":"supports"},{"source_id":"S022","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S027","kind":"supports"},{"source_id":"S025","target_id":"S028","kind":"supports"},{"source_id":"S026","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S027","target_id":"S024","kind":"supports"},{"source_id":"S028","target_id":"S024","kind":"supports"},{"source_id":"S029","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S032","kind":"supports"},{"source_id":"S026","target_id":"S032","kind":"supports"},{"source_id":"S028","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S020","target_id":"S035","kind":"supports"},{"source_id":"S023","target_id":"S035","kind":"supports"},{"source_id":"S022","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S036","kind":"supports"},{"source_id":"S039","target_id":"S037","kind":"supports"},{"source_id":"S020","target_id":"S038","kind":"supports"},{"source_id":"S023","target_id":"S038","kind":"supports"},{"source_id":"S030","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S034","kind":"supports"},{"source_id":"S037","target_id":"S034","kind":"supports"},{"source_id":"S038","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S015","kind":"supports"}]}}
{"record_id":"2112.13000v2","paper":{"title":"Proximal Gradient Algorithms under Local Lipschitz Gradient Continuity: A Convergence and Robustness Analysis of PANOC","authors":["Alberto De Marchi","Andreas Themelis"],"arxiv_id":"2112.13000","version":"v2","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-022-02048-5","abstract_url":"https://arxiv.org/abs/2112.13000v2","pdf_url":"https://arxiv.org/pdf/2112.13000v2"},"context":{"raw_tex":"\\[\\tag{P}\\label{eq:P}\n\t\\minimize_{x\\in\\R^n}\\varphi(x)\\coloneqq f(x)+g(x),\n\\]\n\n\\begin{ass}\n\t\tThe following hold in problem \\eqref{eq:P}:\n\t\t\\begin{enumeratass}\n\t\t\\item\\label{ass:f}%\n\t\t\t\\(\\func{f}{\\R^n}{\\R}\\) has locally Lipschitz-continuous gradient.\n\t\t\\item\\label{ass:g}%\n\t\t\t\\(\\func{g}{\\R^n}{\\R\\cup\\set\\infty}\\) is proper, lsc, and \\(\\gamma_g\\)-prox-bounded.\n\t\t\\item\\label{ass:phi}%\n\t\t\t\\(\\inf\\varphi>-\\infty\\).\n\t\t\\end{enumeratass}\n\t\\end{ass}\n\n\t\t\t\tWith \\(\\R\\) and \\(\\Rinf\\coloneqq\\R\\cup\\set\\infty\\) we denote the real and extended-real line, and by \\(\\N=\\set{0,1,\\dots}\\) the set of natural numbers.\nThe effective domain of an extended-real-valued function \\(\\func{h}{\\R^n}{\\Rinf}\\) is denoted by \\(\\dom h\\coloneqq\\set{x\\in\\R^n}[h(x)<\\infty]\\), and we say that \\(h\\) is: proper if \\(\\dom h\\neq\\emptyset\\); lower semicontinuous (lsc) if \\(h(\\bar x)\\leq\\liminf_{x\\to\\bar x}h(x)\\) for all \\(\\bar x\\in\\R^n\\); coercive if \\(h(x)\\to\\infty\\) as \\(\\|x\\|\\to\\infty\\).\nFor $\\alpha\\in\\R$, the $\\alpha$-\\emph{sublevel set} of $h$ is $\\lev_{\\leq\\alpha} h \\coloneqq \\set{x\\in\\R^n : h(x) \\leq \\alpha}$.\n\n\nGiven a parameter value \\(\\gamma>0\\), the \\emph{Moreau envelope} function \\(h^\\gamma\\) and the \\emph{proximal mapping} \\(\\prox_{\\gamma h}\\) are defined by\n\\begin{align}\n\t\\label{eq:moreauenvelope}\n\th^\\gamma(x)\n\t{}\\coloneqq{} &\n\t\\inf_{z\\in\\R^n}\\set{\n\t\th(z)\n\t\t{}+{}\n\t\t\\tfrac{1}{2\\gamma}\\|z-x\\|^2\n\t} ,\n\t\\\\\n\t\\label{eq:proxoperator}\n\t\\prox_{\\gamma h}(x)\n\t{}\\coloneqq{} &\n\t\\argmin_{z\\in\\R^n}\\set{\n\t\th(z)\n\t\t{}+{}\n\t\t\\tfrac{1}{2\\gamma}\\|z-x\\|^2\n\t},\n\\end{align}\n% Prox-boundedness\nand we say that \\(h\\) is \\emph{prox-bounded} if it is proper and \\(h + \\tfrac{1}{2\\gamma}\\|\\cdot\\|^2\\) is bounded below on \\(\\R^n\\) for some \\(\\gamma>0\\).\nThe supremum of all such \\(\\gamma\\) is the threshold \\(\\gamma_h\\) of prox-boundedness for \\(h\\).\n\n\n\t\t\t\tGiven a point \\(x \\in \\R^n\\), one iteration of the proximal gradient (PG) method for problem \\eqref{eq:P} consists in selecting\n\\begin{equation}\\label{eq:FBSiter}\n\t\\def\\iter{}\n\t\\bar x\n\t{}\\in{}\n\t\\T(x)\n\t{}\\coloneqq{}\n\t\\FB{x},\n\\end{equation}\n\nwhere \\(\\gamma\\in(0,\\gamma_g)\\) is a stepsize parameter.\n\n\n\\begin{defin}[forward-backward envelope]%\n\t\\begin{subequations}\\label{subeq:FBE}%\n\t\tRelative to \\eqref{eq:P}, the FBE with stepsize \\(\\gamma\\in(0,\\gamma_g)\\) is\n\t\t\\begin{align}\n\t\t\\label{eq:FBE}\n\t\t\t\\FBE(x)\n\t\t{}\\coloneqq{} &\n\t\t\t\\min_{w\\in\\R^n}\\set{\n\t\t\t\tf(x)\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla f(x)}{w-x}\n\t\t\t\t{}+{}\n\t\t\t\tg(w)\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\\|w-x\\|^2\n\t\t\t}\n\t\t\\\\\n\t\t\\label{eq:FBEMoreau}\n\t\t{}={} &\n\t\t\tf(x)\n\t\t\t{}-{}\n\t\t\t\\tfrac\\gamma2\\|\\nabla f(x)\\|^2\n\t\t\t{}+{}\n\t\t\tg^\\gamma(\\Fw x)\n\t\t\\shortintertext{%\n\t\t\tor, equivalently, letting \\(\\bar x\\) be any element of \\(\\T(x)\\),\n\t\t}\n\t\t\\label{eq:FBEmin}\n\t\t{}={} &\n\t\t\tf(x)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla f(x)}{\\bar x-x}\n\t\t\t{}+{}\n\t\t\tg(\\bar x)\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2\\gamma}\\|\\bar x-x\\|^2.\n\t\t\\end{align}\n\t\\end{subequations}\n\\end{defin}\n\n\\begin{algorithm}[tb!]\n\t\\caption{\\panoc+: the ``good'' adaptive \\(\\gamma\\)-stepsize rule}%\n\t\\label{alg:PANOC+}%\n\t\\begin{algorithmic}[1]%\n\\linespread{1.4}\\selectfont%\n%\n%\n%\n\\Require\n\t\\(x^0\\in\\R^n\\);~~\n\t\\(\\gamma_0\\in(0,\\gamma_g)\\);~~\n\t\\(D\\geq0\\);~~\n\t\\(\\alpha,\\beta\\in(0,1)\\)%\n\\Initialize\n\t\\(k\\gets0\\),~~\n\tand start from \\cref{state:PANOC+:barx}\n%\n%\n\\vspace*{1ex}%\n\\hrule\n\\vspace*{1ex}%\n%\n%\n%\n\\def\\iter{k}%\n%\n%\n%\n\t\\State\n\t\t\\label{state:PANOC+:init}%\n\t\t\\(\\gamma_k\\gets\\gamma_{k-1}\\)\n\t\\State\n\t\t\\label{state:PANOC+:d}%\n\t\tSelect an update direction \\(d^k\\in\\R^n\\) with \\(\\|d^k\\|\\leq D\\|\\bar x^{k-1}-x^{k-1}\\|\\) and set \\(\\tau_k=1\\)%\n\t\\State\n\t\t\\label{state:PANOC+:x+}%\n\t\t\\(x^k=(1-\\tau_k)\\bar x^{k-1}+\\tau_k(x^{k-1}+d^k)\\)\n\t\\State\n\t\t\\label{state:PANOC+:barx}%\n\t\tCompute \\(\\bar x^k\\in\\T(x^k)\\)\n\t\tand use it to evaluate \\(\\Phi_k\\coloneqq\\FBE(x^k)\\) as in \\eqref{eq:FBEmin}%\n\t\\If{\n\t\t\\(\n\t\t\tf(\\bar x^k)\n\t\t{}>{}\n\t\t\tf(x^k)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla f(x^k)}{\\bar x^k-x^k}\n\t\t\t{}+{}\n\t\t\t\\tfrac{\\alpha}{2\\gamma_k}\\|\\bar x^k-x^k\\|^2\n\t\t\\)\n\t}%\n\t\\label{state:PANOC+:gammaLS}%\n\t\t\\Statex*%\n\t\t\t\\(\\gamma_k\\gets\\nicefrac{\\gamma_k}{2}\\),~\n\t\t\tand go back to \\cref{state:PANOC+:d} if \\(k>0\\), or \\cref{state:PANOC+:barx} if \\(k=0\\)\n\t\\EndIf\n\t\\If{ \\(k>0\\) ~{\\sc and}~ \\(\n\t\t\\Phi_k\n\t\t{}>{}\n\t\t\\Phi_{k-1}\n\t\t{}-{}\n\t\t\\beta\\tfrac{1-\\alpha}{2\\gamma_{k-1}}\n\t\t\\|\\bar x^{k-1}-x^{k-1}\\|^2\n\t\\) }\\label{state:PANOC+:tauLS}%\n\t\t\\Statex*\n\t\t\t\\(\\tau_k\\gets\\nicefrac{\\tau_k}{2}\\)~\n\t\t\tand go back to \\cref{state:PANOC+:x+}\n\t\\EndIf\n\t\\State\n\t\t\\(k\\gets k+1\\)~\n\t\tand start the next iteration at \\cref{state:PANOC+:init}\n\\end{algorithmic}\n%\n\\end{algorithm}\n\n\\begin{rem}[Algorithm notation]\\label{rem:notation}%\n\t\\Cref{alg:PANOC+} operates two linesearch steps within each iteration, one on the ``proximal'' stepsize \\(\\gamma_k\\) at \\cref{state:PANOC+:gammaLS} and one on the ``direction'' stepsize \\(\\tau_k\\) at \\cref{state:PANOC+:tauLS}.\n\tWhenever the respective needed conditions are violated, either \\(\\gamma_k\\) or \\(\\tau_k\\) is reduced and the iteration restarted from a previous step.\n\tAs a consequence, variables may be \\emph{overwritten} within each iteration before being accepted.\n\tTo avoid a heavy double-index notation, used only within proofs out of full rigor, the sub- and superscript notation is designed to differentiate temporary and permanent variables; specifically, within iteration \\(k\\) only variables indexed with \\(k\\) are updated, whereas those indexed with \\(k-1\\) remain untouched.\n\tSimilar considerations apply to \\cref{alg:PANOC}.\n\\end{rem}\n\n\\begin{align}\\label{eq:L}\n\t\t\\LL_\\beta(x,z,y)\n\t{}\\coloneqq{} &\n\t\tf(x)+g(z)+\\innprod{y}{x-z}+\\tfrac\\beta2\\|x-z\\|^2,\n\t\\shortintertext{we remark that}\\label{eq:L:FBE}\n\t\t\\FBE(x)\n\t{}={} &\n\t\t\\LL(x,\\bar x,-\\nabla f(x)),\n\t\\shortintertext{where}\\label{eq:L:barx}\n\t\t\\bar x\n\t{}\\in{} &\n\t\t\\T(x)\n\t\t{}={}\n\t\t\\argmin\\LL(x,{}\\cdot{},-\\nabla f(x))\n\t\\end{align}\n\n\\begin{algorithmic}[1]%\n\t\\makeatletter\n\t\t\\renewcommand{\\alglinenumber}[1]{\\footnotesize\\fillwidthof[l]{\\oldstylenums{88}:}{{\\bf 2}.4$'$:}~}%\n\t\t\\renewcommand{\\theALG@line}{2.\\oldstylenums{4}$'$}%\n\t\\makeatother\n\t\\State\\label{state:NoProx:barx}%\n\t\tLet \\(\\bar x^k\\) be a suboptimal minimizer of \\(\\LL(x^k,{}\\cdot{},-\\nabla f(x^k))\\) such that\n\t\t\\begin{equation}\\label{eq:prox:better}\n\t\t\t\\Phi_k\n\t\t{}\\coloneqq{}\n\t\t\t\\LL(x^k,\\bar x^k,-\\nabla f(x^k))\n\t\t{}\\leq{}\n\t\t\t\\LL(x^k,\\bar x^{k-1},-\\nabla f(x^k)).\n\t\t\\end{equation}\n\t\\end{algorithmic}\n\nTo avoid trivialities, in what follows we assume that \\(x^k\\neq\\bar x^k\\) always holds.\n","units":[{"start":15098,"end":15174,"start_line":256,"end_line":258,"raw_tex":"\\[\\tag{P}\\label{eq:P}\n\t\\minimize_{x\\in\\R^n}\\varphi(x)\\coloneqq f(x)+g(x),\n\\]","sha256_utf8":"a7638b298f19040d7025505cdec8a01a301a6d8a5a1a1ffdeccbf4674df924b4"},{"start":15307,"end":15669,"start_line":262,"end_line":272,"raw_tex":"\\begin{ass}\n\t\tThe following hold in problem \\eqref{eq:P}:\n\t\t\\begin{enumeratass}\n\t\t\\item\\label{ass:f}%\n\t\t\t\\(\\func{f}{\\R^n}{\\R}\\) has locally Lipschitz-continuous gradient.\n\t\t\\item\\label{ass:g}%\n\t\t\t\\(\\func{g}{\\R^n}{\\R\\cup\\set\\infty}\\) is proper, lsc, and \\(\\gamma_g\\)-prox-bounded.\n\t\t\\item\\label{ass:phi}%\n\t\t\t\\(\\inf\\varphi>-\\infty\\).\n\t\t\\end{enumeratass}\n\t\\end{ass}","sha256_utf8":"316eec3423f62ed1176b9a962f22f1ad5cc82cf3fa3120299c9cc0932b8aacfb"},{"start":16252,"end":16895,"start_line":280,"end_line":282,"raw_tex":"\t\t\t\tWith \\(\\R\\) and \\(\\Rinf\\coloneqq\\R\\cup\\set\\infty\\) we denote the real and extended-real line, and by \\(\\N=\\set{0,1,\\dots}\\) the set of natural numbers.\nThe effective domain of an extended-real-valued function \\(\\func{h}{\\R^n}{\\Rinf}\\) is denoted by \\(\\dom h\\coloneqq\\set{x\\in\\R^n}[h(x)<\\infty]\\), and we say that \\(h\\) is: proper if \\(\\dom h\\neq\\emptyset\\); lower semicontinuous (lsc) if \\(h(\\bar x)\\leq\\liminf_{x\\to\\bar x}h(x)\\) for all \\(\\bar x\\in\\R^n\\); coercive if \\(h(x)\\to\\infty\\) as \\(\\|x\\|\\to\\infty\\).\nFor $\\alpha\\in\\R$, the $\\alpha$-\\emph{sublevel set} of $h$ is $\\lev_{\\leq\\alpha} h \\coloneqq \\set{x\\in\\R^n : h(x) \\leq \\alpha}$.\n","sha256_utf8":"fab821b87e9c5bbb7f41366774cea2c1ef26f5b14e618ff1d28251504f40c7c3"},{"start":18394,"end":19116,"start_line":301,"end_line":323,"raw_tex":"Given a parameter value \\(\\gamma>0\\), the \\emph{Moreau envelope} function \\(h^\\gamma\\) and the \\emph{proximal mapping} \\(\\prox_{\\gamma h}\\) are defined by\n\\begin{align}\n\t\\label{eq:moreauenvelope}\n\th^\\gamma(x)\n\t{}\\coloneqq{} &\n\t\\inf_{z\\in\\R^n}\\set{\n\t\th(z)\n\t\t{}+{}\n\t\t\\tfrac{1}{2\\gamma}\\|z-x\\|^2\n\t} ,\n\t\\\\\n\t\\label{eq:proxoperator}\n\t\\prox_{\\gamma h}(x)\n\t{}\\coloneqq{} &\n\t\\argmin_{z\\in\\R^n}\\set{\n\t\th(z)\n\t\t{}+{}\n\t\t\\tfrac{1}{2\\gamma}\\|z-x\\|^2\n\t},\n\\end{align}\n% Prox-boundedness\nand we say that \\(h\\) is \\emph{prox-bounded} if it is proper and \\(h + \\tfrac{1}{2\\gamma}\\|\\cdot\\|^2\\) is bounded below on \\(\\R^n\\) for some \\(\\gamma>0\\).\nThe supremum of all such \\(\\gamma\\) is the threshold \\(\\gamma_h\\) of prox-boundedness for \\(h\\).\n","sha256_utf8":"b0abcbc2574aaccd3db0db490a5d15b522124d356080dc61c9f44914af539103"},{"start":19523,"end":19765,"start_line":329,"end_line":337,"raw_tex":"\t\t\t\tGiven a point \\(x \\in \\R^n\\), one iteration of the proximal gradient (PG) method for problem \\eqref{eq:P} consists in selecting\n\\begin{equation}\\label{eq:FBSiter}\n\t\\def\\iter{}\n\t\\bar x\n\t{}\\in{}\n\t\\T(x)\n\t{}\\coloneqq{}\n\t\\FB{x},\n\\end{equation}","sha256_utf8":"4275dd10c99103d5d91131713ea1ea11a3f3ae9d0a4396651804b08312a40a83"},{"start":19766,"end":19823,"start_line":338,"end_line":338,"raw_tex":"where \\(\\gamma\\in(0,\\gamma_g)\\) is a stepsize parameter.\n","sha256_utf8":"202bb4708d7701aed148ac89ffdc7ac4f3f83f479c077c987946be8289986807"},{"start":21252,"end":22020,"start_line":362,"end_line":400,"raw_tex":"\\begin{defin}[forward-backward envelope]%\n\t\\begin{subequations}\\label{subeq:FBE}%\n\t\tRelative to \\eqref{eq:P}, the FBE with stepsize \\(\\gamma\\in(0,\\gamma_g)\\) is\n\t\t\\begin{align}\n\t\t\\label{eq:FBE}\n\t\t\t\\FBE(x)\n\t\t{}\\coloneqq{} &\n\t\t\t\\min_{w\\in\\R^n}\\set{\n\t\t\t\tf(x)\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla f(x)}{w-x}\n\t\t\t\t{}+{}\n\t\t\t\tg(w)\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\\|w-x\\|^2\n\t\t\t}\n\t\t\\\\\n\t\t\\label{eq:FBEMoreau}\n\t\t{}={} &\n\t\t\tf(x)\n\t\t\t{}-{}\n\t\t\t\\tfrac\\gamma2\\|\\nabla f(x)\\|^2\n\t\t\t{}+{}\n\t\t\tg^\\gamma(\\Fw x)\n\t\t\\shortintertext{%\n\t\t\tor, equivalently, letting \\(\\bar x\\) be any element of \\(\\T(x)\\),\n\t\t}\n\t\t\\label{eq:FBEmin}\n\t\t{}={} &\n\t\t\tf(x)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla f(x)}{\\bar x-x}\n\t\t\t{}+{}\n\t\t\tg(\\bar x)\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2\\gamma}\\|\\bar x-x\\|^2.\n\t\t\\end{align}\n\t\\end{subequations}\n\\end{defin}","sha256_utf8":"8e8f54c6b4136c7afb49fbf9152210231d3f6d4450aeb5cb215f850db57293ac"},{"start":29805,"end":31402,"start_line":557,"end_line":631,"raw_tex":"\\begin{algorithm}[tb!]\n\t\\caption{\\panoc+: the ``good'' adaptive \\(\\gamma\\)-stepsize rule}%\n\t\\label{alg:PANOC+}%\n\t\\begin{algorithmic}[1]%\n\\linespread{1.4}\\selectfont%\n%\n%\n%\n\\Require\n\t\\(x^0\\in\\R^n\\);~~\n\t\\(\\gamma_0\\in(0,\\gamma_g)\\);~~\n\t\\(D\\geq0\\);~~\n\t\\(\\alpha,\\beta\\in(0,1)\\)%\n\\Initialize\n\t\\(k\\gets0\\),~~\n\tand start from \\cref{state:PANOC+:barx}\n%\n%\n\\vspace*{1ex}%\n\\hrule\n\\vspace*{1ex}%\n%\n%\n%\n\\def\\iter{k}%\n%\n%\n%\n\t\\State\n\t\t\\label{state:PANOC+:init}%\n\t\t\\(\\gamma_k\\gets\\gamma_{k-1}\\)\n\t\\State\n\t\t\\label{state:PANOC+:d}%\n\t\tSelect an update direction \\(d^k\\in\\R^n\\) with \\(\\|d^k\\|\\leq D\\|\\bar x^{k-1}-x^{k-1}\\|\\) and set \\(\\tau_k=1\\)%\n\t\\State\n\t\t\\label{state:PANOC+:x+}%\n\t\t\\(x^k=(1-\\tau_k)\\bar x^{k-1}+\\tau_k(x^{k-1}+d^k)\\)\n\t\\State\n\t\t\\label{state:PANOC+:barx}%\n\t\tCompute \\(\\bar x^k\\in\\T(x^k)\\)\n\t\tand use it to evaluate \\(\\Phi_k\\coloneqq\\FBE(x^k)\\) as in \\eqref{eq:FBEmin}%\n\t\\If{\n\t\t\\(\n\t\t\tf(\\bar x^k)\n\t\t{}>{}\n\t\t\tf(x^k)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla f(x^k)}{\\bar x^k-x^k}\n\t\t\t{}+{}\n\t\t\t\\tfrac{\\alpha}{2\\gamma_k}\\|\\bar x^k-x^k\\|^2\n\t\t\\)\n\t}%\n\t\\label{state:PANOC+:gammaLS}%\n\t\t\\Statex*%\n\t\t\t\\(\\gamma_k\\gets\\nicefrac{\\gamma_k}{2}\\),~\n\t\t\tand go back to \\cref{state:PANOC+:d} if \\(k>0\\), or \\cref{state:PANOC+:barx} if \\(k=0\\)\n\t\\EndIf\n\t\\If{ \\(k>0\\) ~{\\sc and}~ \\(\n\t\t\\Phi_k\n\t\t{}>{}\n\t\t\\Phi_{k-1}\n\t\t{}-{}\n\t\t\\beta\\tfrac{1-\\alpha}{2\\gamma_{k-1}}\n\t\t\\|\\bar x^{k-1}-x^{k-1}\\|^2\n\t\\) }\\label{state:PANOC+:tauLS}%\n\t\t\\Statex*\n\t\t\t\\(\\tau_k\\gets\\nicefrac{\\tau_k}{2}\\)~\n\t\t\tand go back to \\cref{state:PANOC+:x+}\n\t\\EndIf\n\t\\State\n\t\t\\(k\\gets k+1\\)~\n\t\tand start the next iteration at \\cref{state:PANOC+:init}\n\\end{algorithmic}\n%\n\\end{algorithm}","sha256_utf8":"1695c10379656f3a3509fd996ff1ebe2afae71ddef4ba323fd6a352cc3fbaa75"},{"start":34005,"end":34917,"start_line":649,"end_line":655,"raw_tex":"\\begin{rem}[Algorithm notation]\\label{rem:notation}%\n\t\\Cref{alg:PANOC+} operates two linesearch steps within each iteration, one on the ``proximal'' stepsize \\(\\gamma_k\\) at \\cref{state:PANOC+:gammaLS} and one on the ``direction'' stepsize \\(\\tau_k\\) at \\cref{state:PANOC+:tauLS}.\n\tWhenever the respective needed conditions are violated, either \\(\\gamma_k\\) or \\(\\tau_k\\) is reduced and the iteration restarted from a previous step.\n\tAs a consequence, variables may be \\emph{overwritten} within each iteration before being accepted.\n\tTo avoid a heavy double-index notation, used only within proofs out of full rigor, the sub- and superscript notation is designed to differentiate temporary and permanent variables; specifically, within iteration \\(k\\) only variables indexed with \\(k\\) are updated, whereas those indexed with \\(k-1\\) remain untouched.\n\tSimilar considerations apply to \\cref{alg:PANOC}.\n\\end{rem}","sha256_utf8":"76856da9b14adfcffa8d115277e51f6adfd4bf7aa975051cc766a0875bb57ee2"},{"start":49658,"end":49998,"start_line":960,"end_line":974,"raw_tex":"\\begin{align}\\label{eq:L}\n\t\t\\LL_\\beta(x,z,y)\n\t{}\\coloneqq{} &\n\t\tf(x)+g(z)+\\innprod{y}{x-z}+\\tfrac\\beta2\\|x-z\\|^2,\n\t\\shortintertext{we remark that}\\label{eq:L:FBE}\n\t\t\\FBE(x)\n\t{}={} &\n\t\t\\LL(x,\\bar x,-\\nabla f(x)),\n\t\\shortintertext{where}\\label{eq:L:barx}\n\t\t\\bar x\n\t{}\\in{} &\n\t\t\\T(x)\n\t\t{}={}\n\t\t\\argmin\\LL(x,{}\\cdot{},-\\nabla f(x))\n\t\\end{align}","sha256_utf8":"332620e858bedaf34909afe0ae2bf6b88ad650908723b5edfc8912c7b7f61740"},{"start":51673,"end":52202,"start_line":991,"end_line":1005,"raw_tex":"\\begin{algorithmic}[1]%\n\t\\makeatletter\n\t\t\\renewcommand{\\alglinenumber}[1]{\\footnotesize\\fillwidthof[l]{\\oldstylenums{88}:}{{\\bf 2}.4$'$:}~}%\n\t\t\\renewcommand{\\theALG@line}{2.\\oldstylenums{4}$'$}%\n\t\\makeatother\n\t\\State\\label{state:NoProx:barx}%\n\t\tLet \\(\\bar x^k\\) be a suboptimal minimizer of \\(\\LL(x^k,{}\\cdot{},-\\nabla f(x^k))\\) such that\n\t\t\\begin{equation}\\label{eq:prox:better}\n\t\t\t\\Phi_k\n\t\t{}\\coloneqq{}\n\t\t\t\\LL(x^k,\\bar x^k,-\\nabla f(x^k))\n\t\t{}\\leq{}\n\t\t\t\\LL(x^k,\\bar x^{k-1},-\\nabla f(x^k)).\n\t\t\\end{equation}\n\t\\end{algorithmic}","sha256_utf8":"86b0e4a2f9c8391f1d6345867a3e9e69719bf63ded15ae7b5935b366bf749d8a"},{"start":54085,"end":54173,"start_line":1039,"end_line":1039,"raw_tex":"To avoid trivialities, in what follows we assume that \\(x^k\\neq\\bar x^k\\) always holds.\n","sha256_utf8":"1790a1c2a9bf38b1c4343333fedd337b090f329da33857fa9393dcefd9dce7f1"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{thm}[Asymptotic analysis of the ``good'' (inexact) \\panoc+]\\label{thm:PANOC+:asymp}%\n\tConsider the iterates generated by \\cref{alg:PANOC+} with inexact proximal evaluation at \\cref{state:PANOC+:barx} as given in \\eqref{eq:prox:better}.\n\tThe following hold:\n\t\\begin{enumerate}\n\t\\item\\label{thm:PANOC+:asymp:cost}%\n\t\t\\(\\seq{\\Phi_k}\\) converges to a finite value \\(\\varphi_\\star\\geq\\inf\\varphi\\) from above.\n\t\\item\\label{thm:PANOC+:asymp:summable}%\n\t\t\\(\n\t\t\t\\sum_{k\\in\\N}\\frac{1}{\\gamma_k}\\|\\bar x^k-x^k\\|^2\n\t\t{}<{}\n\t\t\t\\infty\n\t\t\\).\n\t\\item\\label{thm:PANOC+:asymp:fpr}%\n\t\t\\(\n\t\t\t\\lim_{k\\to\\infty}\\|x^k-\\bar x^k\\|\n\t\t{}={}\n\t\t\t\\lim_{k\\to\\infty}\\|x^k-x^{k-1}\\|\n\t\t{}={}\n\t\t\t\\lim_{k\\to\\infty}\\|\\bar x^k-\\bar x^{k-1}\\|\n\t\t{}={}\n\t\t\t0\n\t\t\\),\n\t\tand in particular the set of limit points of \\(\\seq{x^k}\\) is closed and connected, and coincides with that of \\(\\seq{\\bar x^k}\\).\n\t\\item\\label{thm:PANOC+:asymp:gamma}%\n\t\t\\(\\sum_{k\\in\\N}\\gamma_k=\\infty\\).\n\t\\item\\label{thm:PANOC+:asymp:res}%\n\t\t\\(\n\t\t\t\\liminf_{k\\to\\infty}\\frac{1}{\\gamma_k}\\|x^k-\\bar x^k\\|\n\t\t{}={}\n\t\t\t0\n\t\t\\).\n\t\\item\\label{thm:PANOC+:asymp:gammaconstant}%\n\t\tConsider the following assertions:\n\t\t\\begin{enumerate}[label={(\\oldstylenums{\\arabic*})},ref={(\\oldstylenums{\\arabic*})}]\n\t\t\\item\\label{thm:PANOC+:gammaconstant:lb}%\n\t\t\t\\(\\varphi\\) is level bounded;\n\t\t\\item\\label{thm:PANOC+:gammaconstant:xk}%\n\t\t\t\\(\\seq{\\bar x^k}\\) is bounded;\n\t\t\\item\\label{thm:PANOC+:gammaconstant:barxk}%\n\t\t\t\\(\\seq{x^k}\\) is bounded;\n\t\t\\item\\label{thm:PANOC+:gammaconstant:gamma}%\n\t\t\t\\(\\seq{\\gamma_k}\\) is asymptotically constant, \\ie, there exists \\(\\kappa\\in\\N\\) such that \\(\\gamma_k=\\gamma_\\kappa\\) for every \\(k\\geq\\kappa\\);\n\t\t\\item\\label{thm:PANOC+:gammaconstant:Lf}%\n\t\t\t\\(f\\) has globally Lipschitz-continuous gradient.\n\t\t\\end{enumerate}\n\t\tOne has~\n\t\t\\ref{thm:PANOC+:gammaconstant:lb}\n\t\t~\\(\\Rightarrow\\)~\n\t\t\\ref{thm:PANOC+:gammaconstant:xk}\n\t\t~\\(\\Leftrightarrow\\)~\n\t\t\\ref{thm:PANOC+:gammaconstant:barxk}\n\t\t~\\(\\Rightarrow\\)~\n\t\t\\ref{thm:PANOC+:gammaconstant:gamma}\n\t\t~\\(\\Leftarrow\\)~\n\t\t\\ref{thm:PANOC+:gammaconstant:Lf}.\n\t\\end{enumerate}\n\t\n\n\n\\end{thm}","units":[{"start":59431,"end":61496,"start_line":1156,"end_line":1212,"raw_tex":"\\begin{thm}[Asymptotic analysis of the ``good'' (inexact) \\panoc+]\\label{thm:PANOC+:asymp}%\n\tConsider the iterates generated by \\cref{alg:PANOC+} with inexact proximal evaluation at \\cref{state:PANOC+:barx} as given in \\eqref{eq:prox:better}.\n\tThe following hold:\n\t\\begin{enumerate}\n\t\\item\\label{thm:PANOC+:asymp:cost}%\n\t\t\\(\\seq{\\Phi_k}\\) converges to a finite value \\(\\varphi_\\star\\geq\\inf\\varphi\\) from above.\n\t\\item\\label{thm:PANOC+:asymp:summable}%\n\t\t\\(\n\t\t\t\\sum_{k\\in\\N}\\frac{1}{\\gamma_k}\\|\\bar x^k-x^k\\|^2\n\t\t{}<{}\n\t\t\t\\infty\n\t\t\\).\n\t\\item\\label{thm:PANOC+:asymp:fpr}%\n\t\t\\(\n\t\t\t\\lim_{k\\to\\infty}\\|x^k-\\bar x^k\\|\n\t\t{}={}\n\t\t\t\\lim_{k\\to\\infty}\\|x^k-x^{k-1}\\|\n\t\t{}={}\n\t\t\t\\lim_{k\\to\\infty}\\|\\bar x^k-\\bar x^{k-1}\\|\n\t\t{}={}\n\t\t\t0\n\t\t\\),\n\t\tand in particular the set of limit points of \\(\\seq{x^k}\\) is closed and connected, and coincides with that of \\(\\seq{\\bar x^k}\\).\n\t\\item\\label{thm:PANOC+:asymp:gamma}%\n\t\t\\(\\sum_{k\\in\\N}\\gamma_k=\\infty\\).\n\t\\item\\label{thm:PANOC+:asymp:res}%\n\t\t\\(\n\t\t\t\\liminf_{k\\to\\infty}\\frac{1}{\\gamma_k}\\|x^k-\\bar x^k\\|\n\t\t{}={}\n\t\t\t0\n\t\t\\).\n\t\\item\\label{thm:PANOC+:asymp:gammaconstant}%\n\t\tConsider the following assertions:\n\t\t\\begin{enumerate}[label={(\\oldstylenums{\\arabic*})},ref={(\\oldstylenums{\\arabic*})}]\n\t\t\\item\\label{thm:PANOC+:gammaconstant:lb}%\n\t\t\t\\(\\varphi\\) is level bounded;\n\t\t\\item\\label{thm:PANOC+:gammaconstant:xk}%\n\t\t\t\\(\\seq{\\bar x^k}\\) is bounded;\n\t\t\\item\\label{thm:PANOC+:gammaconstant:barxk}%\n\t\t\t\\(\\seq{x^k}\\) is bounded;\n\t\t\\item\\label{thm:PANOC+:gammaconstant:gamma}%\n\t\t\t\\(\\seq{\\gamma_k}\\) is asymptotically constant, \\ie, there exists \\(\\kappa\\in\\N\\) such that \\(\\gamma_k=\\gamma_\\kappa\\) for every \\(k\\geq\\kappa\\);\n\t\t\\item\\label{thm:PANOC+:gammaconstant:Lf}%\n\t\t\t\\(f\\) has globally Lipschitz-continuous gradient.\n\t\t\\end{enumerate}\n\t\tOne has~\n\t\t\\ref{thm:PANOC+:gammaconstant:lb}\n\t\t~\\(\\Rightarrow\\)~\n\t\t\\ref{thm:PANOC+:gammaconstant:xk}\n\t\t~\\(\\Leftrightarrow\\)~\n\t\t\\ref{thm:PANOC+:gammaconstant:barxk}\n\t\t~\\(\\Rightarrow\\)~\n\t\t\\ref{thm:PANOC+:gammaconstant:gamma}\n\t\t~\\(\\Leftarrow\\)~\n\t\t\\ref{thm:PANOC+:gammaconstant:Lf}.\n\t\\end{enumerate}\n\t","sha256_utf8":"b73de2e00008eea7adddc0f7cba1382f9d0749d09e946d9d61f3eeabad9ee8e4"},{"start":64610,"end":64620,"start_line":1307,"end_line":1308,"raw_tex":"\n\\end{thm}","sha256_utf8":"27661e277ceb78e4dd676b4ea4b5c8d9a4d3c607a1a758eac095282a4f311f77"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lem}\\label{thm:inexact:lb}%\n\tLet a constant \\(c\\in\\R\\), a sequence \\(\\seq{\\gamma_j}[j\\in\\N]\\searrow0\\), and two bounded sequences \\(\\seq{u^j,z^j}[j\\in\\N]\\) in \\(\\R^n\\) be fixed, and for every \\(j\\in\\N\\) let \\(\\bar z^j\\) be such that\n\t\\[\n\t\tg(\\bar z^j)+\\innprod*{u^j}{\\bar z^j-z^j}+\\tfrac{1}{2\\gamma_j}\\|\\bar z^j-z^j\\|^2\n\t\t{}\\leq{}\n\t\t\\tfrac{c}{2\\gamma_j}.\n\t\\]\n\tThen, \\(\\seq{\\bar z^j}[j\\in\\N]\\) is bounded.\n\t\\begin{proof}\n\t\tAn application of Young's inequality on the inner product yields\n\t\t\\[\n\t\t2\\gamma_jg(\\bar z^j)\n\t\t{}\\leq{}\n\t\tc\n\t\t{}+{}\n\t\t\\gamma_j\\|u_j\\|^2\n\t\t{}-{}\n\t\t(1-\\gamma_j)\\|\\bar z^j-z^j\\|^2.\n\t\t\\]\n\t\tTo arrive to a contradiction, up to extracting if necessary, suppose that \\(0<\\|\\bar z^j\\|\\to\\infty\\).\n\t\tSince \\(\\liminf_{j\\to\\infty}{g(\\bar z^j)}/{\\|\\bar z^j\\|^2}>-\\infty\\) by \\cite[Ex. 1.24]{rockafellar1998variational}, dividing by \\(\\|\\bar z^j\\|^2\\) and passing to the limit leads to the contradiction \\(0\\leq -1\\).\n\t\\end{proof}\n\\end{lem}\n\n\\begin{lem}[Well definedness of the ``good'' (inexact) \\panoc+]\\label{thm:PANOC+:finite}%\n\tConsider the iterates generated by \\cref{alg:PANOC+} with inexact proximal evaluation at \\cref{state:PANOC+:barx} as given in \\eqref{eq:prox:better}.\n\tThe following hold:\n\t\\begin{enumerate}\n\t\\item\\label{thm:finite:LS}%\n\t\tWell definedness: at every iteration, the number of backtrackings at \\cref{state:PANOC+:gammaLS,state:PANOC+:tauLS} is finite.\n\t\\item\\label{thm:finite:descent}%\n\t\tAt the end of the \\(k\\)-th iteration (\\(k\\geq1\\)), one has\n\t\t\\begin{equation}\\label{eq:SD}\n\t\t\t\t\\varphi(\\bar x^k)\n\t\t\t\t{}+{}\n\t\t\t\t\\delta_k\n\t\t\t{}\\leq{}\n\t\t\t\t\\Phi_k\n\t\t\t{}\\leq{}\n\t\t\t\t\\def\\iter{k-1}\n\t\t\t\t\\Phi_{k-1}\n\t\t\t\t{}-{}\n\t\t\t\t\\beta\\delta_{k-1}\n\t\t\t\\quad\\text{where}\\quad\n\t\t\t\t\\delta_k\n\t\t\t{}\\coloneqq{}\n\t\t\t\t\\tfrac{1-\\alpha}{2\\gamma_k}\n\t\t\t\t\\|\\bar x^k-x^k\\|^2.\n\t\t\t\\end{equation}\n\t\\item\\label{thm:finite:sublevel}%\n\t\tEvery iterate \\(\\bar x^k\\) remains within \\(\\lev_{\\leq c}\\varphi\\), where \\(c=\\Phi_0<\\infty\\).\n\t\\end{enumerate}\n\t\\begin{proof}\n\t\tAs observed in \\cref{rem:notation}, each iteration \\(k\\) defines or updates only variables indexed with a \\(k\\) sub/superscript, while those defined in previous iterations are untouched.\n\t\tIn what follows, let us index by \\({k,j}\\) the variables defined at the \\(j\\)-th attempt within iteration \\(k\\).\n\t\tNote further that \\(\\gamma_{k,j}L_{k,j}=\\alpha\\in(0,1)\\) holds for every attempt \\(j\\) within every iteration \\(k\\), since every time \\(\\gamma_k\\) is halvened the estimate \\(L_k\\) is doubled (cf. \\cref{state:PANOC+:gammaLS}).\n\t\t\\begin{proofitemize}\n\t\t\\item\\ref{thm:finite:LS}~\n\t\t\tWe proceed by induction on \\(k\\).\n\t\t\tIf \\(k=0\\), there is no backtracking on \\(\\tau\\), and from \\cref{thm:inexact:lb} we conclude that all the trials \\(\\bar x^{0,j}\\) remain confined in a bounded set \\(\\Omega_0\\), and therefore any stepsize \\(\\gamma_{0,j}<\\nicefrac{1}{L_{f,\\Omega_0}}\\) is accepted.\n\n\t\t\tSuppose now that \\(k>0\\) and observe that, by the definition of \\(\\Phi_k\\) in \\eqref{eq:prox:better} and the failure of the condition at \\cref{state:PANOC+:gammaLS}, the inequality\n\t\t\t\\begin{equation}\\label{eq:lphi}\n\t\t\t\t\\varphi(\\bar x^{k-1})\n\t\t\t\t{}\\leq{}\n\t\t\t\t\\Phi_{k-1}\n\t\t\t\t{}-{}\n\t\t\t\t\\tfrac{1-\\alpha}{2\\gamma_{k-1}}\\|x^{k-1}-\\bar x^{k-1}\\|^2\n\t\t\t\\end{equation}\n\t\t\tholds.\n\t\t\tSince \\(\\|d^{k,j}\\|\\leq D\\|\\bar x^{k-1}-x^{k-1}\\|\\) and \\(\\tau_{k,j}\\in[0,1]\\), any attempt \\(x^{k,j}\\) defined at \\cref{state:PANOC+:x+} during the \\(k\\)-th iteration satisfies\n\t\t\t\\[\n\t\t\t\t\\|x^{k,j}-\\bar x^{k-1}\\|\n\t\t\t{}={}\n\t\t\t\t\\tau_{k,j}\\|\n\t\t\t\t\tx^{k-1}-\\bar x^{k-1}\n\t\t\t\t\t{}+{}\n\t\t\t\t\td^{k,j}\n\t\t\t\t\\|\n\t\t\t{}\\leq{}\n\t\t\t\t(1+D)\\|\\bar x^{k-1}-x^{k-1}\\|\n\t\t\t\\]\n\t\t\tand thus remains in a bounded set, be it \\(\\Omega_k\\).\n\t\t\tTo arrive to a contradiction, suppose that \\(\\gamma_{k,j}\\searrow0\\) as \\(j\\to\\infty\\).\n\t\t\tObserve that condition \\eqref{eq:prox:better} reads\n\t\t\t\\begin{Align*}\n\t\t\t\tg(\\bar x^{k,j})\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod*{\\nabla f(x^{k,j})}{\\bar x^{k,j}-\\bar x^{k-1}}\\quad\n\t\t\t&\n\t\t\t\\\\\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma_{k,j}}\\|x^{k,j}-\\bar x^{k,j}\\|^2\n\t\t\t& {}\\leq{}\n\t\t\t\tg(\\bar x^{k-1})\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma_{k,j}}\\|x^{k,j}-\\bar x^{k-1}\\|^2.\n\t\t\t\\end{Align*}\n\t\t\tSince \\(\\seq{x^{k,j}}[j\\in\\N]\\) is bounded, an application of \\cref{thm:inexact:lb} reveals that \\(\\seq{\\bar x^{k,j}}\\) too is bounded.\n\t\t\tUp to possibly enlarging the set, both sequences remain confined in the bounded set \\(\\Omega_k\\), implying that the condition at \\cref{state:PANOC+:gammaLS} should have terminated in finite time, whence the sought contradiction.\n\t\t\t\n\t\t\tHence, \\(\\gamma_{k,j}\\) is backtracked finitely many times within iteration \\(k\\);\n\t\t\tup to discarding early attempts, we may denote \\(\\gamma_{k,j}=\\gamma_k\\).\n\t\t\tCondition \\eqref{eq:prox:better} reads\n\t\t\t\\begin{align*}\n\t\t\t\t\\LL(x^{k,j},\\bar x^{k,j},-\\nabla f(x^{k,j}))\n\t\t\t{}\\leq{} &\n\t\t\t\t\\LL(x^{k,j},\\bar x^{k-1},-\\nabla f(x^{k,j}))\n\t\t\t\\\\\n\t\t\t{}={} &\n\t\t\t\tf(x^{k,j})+g(\\bar x^{k-1})+\\innprod*{\\nabla f(x^{k,j})}{\\bar x^{k-1}-x^{k,j}}\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t{+{}}\\tfrac{1}{2\\gamma}\\|x^{k,j}-\\bar x^{k-1}\\|^2.\n\t\t\t\\end{align*}\n\t\t\tAs \\(\\tau_{k,j}\\searrow0\\), one has that \\(x^{k,j}\\to\\bar x^{k-1}\\).\n\t\t\tSince \\(f\\) and \\(\\nabla f\\) are continuous, the right-hand side of the inequality converges to \\(\\varphi(\\bar x^{k-1})\\), overall resulting in\n\t\t\t\\[\n\t\t\t\\limsup_{j\\to\\infty}\\LL(x^{k,j},\\bar x^{k,j},-\\nabla f(x^{k,j}))\n\t\t\t{}\\leq{}\n\t\t\t\\varphi(\\bar x^{k-1})\n\t\t\t{}\\overrel[\\leq]{\\eqref{eq:lphi}}{}\n\t\t\t\\Phi_{k-1}-\\tfrac{1-\\alpha}{2\\gamma_{k-1}}\\|x^{k-1}-\\bar x^{k-1}\\|^2.\n\t\t\t\\]\n\t\t\tSince \\(\\|x^{k-1}-\\bar x^{k-1}\\|>0\\) and \\(\\beta<1\\), for \\(j\\) large enough the condition at \\cref{state:PANOC+:tauLS} will be violated and therefore the \\(k\\)-th iteration successfully terminated.\n\t\t%%\n\t\t%%\n\t\t\\item\\ref{thm:finite:descent}\n\t\t\tFollows by combining \\eqref{eq:lphi} with the failure of the condition at \\cref{state:PANOC+:tauLS} at the end of the iteration.\n\t\t\\item\\ref{thm:finite:sublevel}~\n\t\t\tDirect consequence of assertion \\ref{thm:finite:descent}.\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}\n\\end{lem}\n\n\\begin{proof}\n\t\t\\begin{proofitemize}\n\t\t\\item\\ref{thm:PANOC+:asymp:cost}~\n\t\t\tFollows from \\eqref{eq:SD}.\n\t\t\\item\\ref{thm:PANOC+:asymp:summable}~\n\t\t\tA telescoping argument on \\eqref{eq:SD} yields\n\t\t\t\\begin{equation}\\label{eq:summable}\n\t\t\t\t\\beta(1-\\alpha)\n\t\t\t\t\\sum_{k\\in\\N}{\n\t\t\t\t\t\\tfrac{1}{2\\gamma_k}\n\t\t\t\t\t\\|\\bar x^k-x^k\\|^2\n\t\t\t\t}\n\t\t\t{}\\leq{}\n\t\t\t\t\\Phi_0\n\t\t\t\t{}-{}\n\t\t\t\t\\inf\\varphi\n\t\t\t{}={}\n\t\t\t\t\\def\\iter{0}\n\t\t\t\t\\FBE(x^0)-\\inf\\varphi,\n\t\t\t\\end{equation}\n\t\t\twhence the claimed finite sum.\n\t%%%%\n\t%%%%\n\t%%%%\n\t\t\\item\\ref{thm:PANOC+:asymp:fpr}~\n\t\t\tThat \\(\\|x^k-\\bar x^k\\|\\to0\\) follows from assertion \\ref{thm:PANOC+:asymp:summable}, since \\(\\gamma_k\\) is upper bounded.\n\t\t\tNext, by the conditions at \\cref{state:PANOC+:x+,,state:PANOC+:d}, observe that\n\t\t\t\\begin{align*}\n\t\t\t\\numberthis\\label{eq:xkdiff}\n\t\t\t\t\\|x^k-x^{k-1}\\|\n\t\t\t{}={} &\n\t\t\t\t\\bigl\\|\n\t\t\t\t\t(1-\\tau_k)(\\bar x^{k-1}-x^{k-1})\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tau_kd^k\n\t\t\t\t\\bigr\\|\n\t\t\t{}\\leq{}\n\t\t\t\t(1+D)\\|\\bar x^{k-1}-x^{k-1}\\|\n\t\t\t\\shortintertext{and thus \\(\\|x^k-x^{k-1}\\|\\) vanishes, and in turn so does \\(\\|\\bar x^k-\\bar x^{k-1}\\|\\) since}\n\t\t\t\t\\|\\bar x^k-\\bar x^{k-1}\\|\n\t\t\t{}\\leq{} &\n\t\t\t\t\\|x^k-\\bar x^k\\|\n\t\t\t\t{}+{}\n\t\t\t\t\\|\\bar x^{k-1}-x^{k-1}\\|\n\t\t\t\t{}+{}\n\t\t\t\t\\|x^k-x^{k-1}\\|.\n\t\t\t\\end{align*}\n\t\t\\item\\ref{thm:PANOC+:asymp:gammaconstant}~\n\t\t\tThe first implication follows from \\cref{thm:finite:sublevel}, and the second one from assertion \\ref{thm:PANOC+:asymp:summable}.\n\t\t\tIf \\(\\seq{x^k}\\) is bounded, and thus so is \\(\\seq{\\bar x^k}\\), the set \\(\\Omega_k\\) in the proof of \\cref{thm:finite:LS} can be taken independent of \\(k\\), and asymptotic constancy of \\(\\gamma_k\\) follows from the same arguments therein.\n\t\t\tFinally, if \\(\\nabla f\\) is $L_f$-Lipschitz continuous the condition at \\cref{state:PANOC+:gammaLS} fails to hold as soon as $\\gamma_k\\leq\\nicefrac{\\alpha}{L_f}$ \\cite[Prop. A.24]{bertsekas1999nonlinear}, and \\(\\gamma_k\\) is thus asymptotically constant.\n\t%%%%\n\t%%%%\n\t%%%%\n\t\t\\item\\ref{thm:PANOC+:asymp:gamma}~\n\t\t\tBy iteratively applying inequality \\eqref{eq:xkdiff}, we obtain that\n\t\t\t\\begin{align*}\n\t\t\t\t\\|x^k-x^0\\|\n\t\t\t{}\\leq{} &\n\t\t\t\t(1+D)\\sum_{j=0}^{k-1}\\|\\bar x^j-x^j\\|\n\t\t\t\\\\\n\t\t\t{}={} &\n\t\t\t\t(1+D)\\sum_{j=0}^{k-1}\\gamma_j^{-\\nicefrac12}\\|\\bar x^j-x^j\\|\\gamma_j^{\\nicefrac12}\n\t\t\t\\\\\n\t\t\t{}\\leq{} &\n\t\t\t\t\\textstyle\n\t\t\t\t(1+D)\n\t\t\t\t\\sqrt{\n\t\t\t\t\t\\sum_{j=0}^{k-1}\n\t\t\t\t\t\\gamma_j^{-1}\\|\\bar x^j-x^j\\|^2\n\t\t\t\t}\n\t\t\t\t\\sqrt{\n\t\t\t\t\t\\sum_{j=0}^{k-1}\\gamma_j\n\t\t\t\t}\n\t\t\t\\\\\n\t\t\t{}\\overrel*[\\leq]{\\eqref{eq:summable}}{} &\n\t\t\t\t\\textstyle\n\t\t\t\t(1+D)\n\t\t\t\t\\sqrt{\n\t\t\t\t\t2\n\t\t\t\t\t\\frac{\\def\\iter{0}\\FBE(x^0)-\\inf\\varphi}{\\beta(1-\\alpha)}\n\t\t\t\t}\n\t\t\t\t\\sqrt{\n\t\t\t\t\t\\sum_{j=0}^{k-1}\\gamma_j\n\t\t\t\t}.\n\t\t\t\\end{align*}\n\t\t\tContrary to the claim, if \\(\\sum_{k\\in\\N}\\gamma_k<\\infty\\) holds, then \\(\\seq{x^k}\\) is bounded.\n\t\t\tFrom assertion \\ref{thm:PANOC+:asymp:gammaconstant} proven above we then infer that \\(\\gamma_k\\) is asymptotically constant, thus contradicting the finiteness of \\(\\sum_{k\\in\\N}\\gamma_k\\).\n\t%%%%\n\t%%%%\n\t%%%%\n\t\t\\item\\ref{thm:PANOC+:asymp:res}~\n\t\t\tImmediate consequence of assertions \\ref{thm:PANOC+:asymp:summable} and \\ref{thm:PANOC+:asymp:gamma}.\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}","units":[{"start":53129,"end":54083,"start_line":1015,"end_line":1037,"raw_tex":"\\begin{lem}\\label{thm:inexact:lb}%\n\tLet a constant \\(c\\in\\R\\), a sequence \\(\\seq{\\gamma_j}[j\\in\\N]\\searrow0\\), and two bounded sequences \\(\\seq{u^j,z^j}[j\\in\\N]\\) in \\(\\R^n\\) be fixed, and for every \\(j\\in\\N\\) let \\(\\bar z^j\\) be such that\n\t\\[\n\t\tg(\\bar z^j)+\\innprod*{u^j}{\\bar z^j-z^j}+\\tfrac{1}{2\\gamma_j}\\|\\bar z^j-z^j\\|^2\n\t\t{}\\leq{}\n\t\t\\tfrac{c}{2\\gamma_j}.\n\t\\]\n\tThen, \\(\\seq{\\bar z^j}[j\\in\\N]\\) is bounded.\n\t\\begin{proof}\n\t\tAn application of Young's inequality on the inner product yields\n\t\t\\[\n\t\t2\\gamma_jg(\\bar z^j)\n\t\t{}\\leq{}\n\t\tc\n\t\t{}+{}\n\t\t\\gamma_j\\|u_j\\|^2\n\t\t{}-{}\n\t\t(1-\\gamma_j)\\|\\bar z^j-z^j\\|^2.\n\t\t\\]\n\t\tTo arrive to a contradiction, up to extracting if necessary, suppose that \\(0<\\|\\bar z^j\\|\\to\\infty\\).\n\t\tSince \\(\\liminf_{j\\to\\infty}{g(\\bar z^j)}/{\\|\\bar z^j\\|^2}>-\\infty\\) by \\cite[Ex. 1.24]{rockafellar1998variational}, dividing by \\(\\|\\bar z^j\\|^2\\) and passing to the limit leads to the contradiction \\(0\\leq -1\\).\n\t\\end{proof}\n\\end{lem}","sha256_utf8":"feb82eabd7c653190c757e2c10771138dc09c32b39d70ed94357863235f2599f"},{"start":54380,"end":59370,"start_line":1042,"end_line":1152,"raw_tex":"\\begin{lem}[Well definedness of the ``good'' (inexact) \\panoc+]\\label{thm:PANOC+:finite}%\n\tConsider the iterates generated by \\cref{alg:PANOC+} with inexact proximal evaluation at \\cref{state:PANOC+:barx} as given in \\eqref{eq:prox:better}.\n\tThe following hold:\n\t\\begin{enumerate}\n\t\\item\\label{thm:finite:LS}%\n\t\tWell definedness: at every iteration, the number of backtrackings at \\cref{state:PANOC+:gammaLS,state:PANOC+:tauLS} is finite.\n\t\\item\\label{thm:finite:descent}%\n\t\tAt the end of the \\(k\\)-th iteration (\\(k\\geq1\\)), one has\n\t\t\\begin{equation}\\label{eq:SD}\n\t\t\t\t\\varphi(\\bar x^k)\n\t\t\t\t{}+{}\n\t\t\t\t\\delta_k\n\t\t\t{}\\leq{}\n\t\t\t\t\\Phi_k\n\t\t\t{}\\leq{}\n\t\t\t\t\\def\\iter{k-1}\n\t\t\t\t\\Phi_{k-1}\n\t\t\t\t{}-{}\n\t\t\t\t\\beta\\delta_{k-1}\n\t\t\t\\quad\\text{where}\\quad\n\t\t\t\t\\delta_k\n\t\t\t{}\\coloneqq{}\n\t\t\t\t\\tfrac{1-\\alpha}{2\\gamma_k}\n\t\t\t\t\\|\\bar x^k-x^k\\|^2.\n\t\t\t\\end{equation}\n\t\\item\\label{thm:finite:sublevel}%\n\t\tEvery iterate \\(\\bar x^k\\) remains within \\(\\lev_{\\leq c}\\varphi\\), where \\(c=\\Phi_0<\\infty\\).\n\t\\end{enumerate}\n\t\\begin{proof}\n\t\tAs observed in \\cref{rem:notation}, each iteration \\(k\\) defines or updates only variables indexed with a \\(k\\) sub/superscript, while those defined in previous iterations are untouched.\n\t\tIn what follows, let us index by \\({k,j}\\) the variables defined at the \\(j\\)-th attempt within iteration \\(k\\).\n\t\tNote further that \\(\\gamma_{k,j}L_{k,j}=\\alpha\\in(0,1)\\) holds for every attempt \\(j\\) within every iteration \\(k\\), since every time \\(\\gamma_k\\) is halvened the estimate \\(L_k\\) is doubled (cf. \\cref{state:PANOC+:gammaLS}).\n\t\t\\begin{proofitemize}\n\t\t\\item\\ref{thm:finite:LS}~\n\t\t\tWe proceed by induction on \\(k\\).\n\t\t\tIf \\(k=0\\), there is no backtracking on \\(\\tau\\), and from \\cref{thm:inexact:lb} we conclude that all the trials \\(\\bar x^{0,j}\\) remain confined in a bounded set \\(\\Omega_0\\), and therefore any stepsize \\(\\gamma_{0,j}<\\nicefrac{1}{L_{f,\\Omega_0}}\\) is accepted.\n\n\t\t\tSuppose now that \\(k>0\\) and observe that, by the definition of \\(\\Phi_k\\) in \\eqref{eq:prox:better} and the failure of the condition at \\cref{state:PANOC+:gammaLS}, the inequality\n\t\t\t\\begin{equation}\\label{eq:lphi}\n\t\t\t\t\\varphi(\\bar x^{k-1})\n\t\t\t\t{}\\leq{}\n\t\t\t\t\\Phi_{k-1}\n\t\t\t\t{}-{}\n\t\t\t\t\\tfrac{1-\\alpha}{2\\gamma_{k-1}}\\|x^{k-1}-\\bar x^{k-1}\\|^2\n\t\t\t\\end{equation}\n\t\t\tholds.\n\t\t\tSince \\(\\|d^{k,j}\\|\\leq D\\|\\bar x^{k-1}-x^{k-1}\\|\\) and \\(\\tau_{k,j}\\in[0,1]\\), any attempt \\(x^{k,j}\\) defined at \\cref{state:PANOC+:x+} during the \\(k\\)-th iteration satisfies\n\t\t\t\\[\n\t\t\t\t\\|x^{k,j}-\\bar x^{k-1}\\|\n\t\t\t{}={}\n\t\t\t\t\\tau_{k,j}\\|\n\t\t\t\t\tx^{k-1}-\\bar x^{k-1}\n\t\t\t\t\t{}+{}\n\t\t\t\t\td^{k,j}\n\t\t\t\t\\|\n\t\t\t{}\\leq{}\n\t\t\t\t(1+D)\\|\\bar x^{k-1}-x^{k-1}\\|\n\t\t\t\\]\n\t\t\tand thus remains in a bounded set, be it \\(\\Omega_k\\).\n\t\t\tTo arrive to a contradiction, suppose that \\(\\gamma_{k,j}\\searrow0\\) as \\(j\\to\\infty\\).\n\t\t\tObserve that condition \\eqref{eq:prox:better} reads\n\t\t\t\\begin{Align*}\n\t\t\t\tg(\\bar x^{k,j})\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod*{\\nabla f(x^{k,j})}{\\bar x^{k,j}-\\bar x^{k-1}}\\quad\n\t\t\t&\n\t\t\t\\\\\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma_{k,j}}\\|x^{k,j}-\\bar x^{k,j}\\|^2\n\t\t\t& {}\\leq{}\n\t\t\t\tg(\\bar x^{k-1})\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma_{k,j}}\\|x^{k,j}-\\bar x^{k-1}\\|^2.\n\t\t\t\\end{Align*}\n\t\t\tSince \\(\\seq{x^{k,j}}[j\\in\\N]\\) is bounded, an application of \\cref{thm:inexact:lb} reveals that \\(\\seq{\\bar x^{k,j}}\\) too is bounded.\n\t\t\tUp to possibly enlarging the set, both sequences remain confined in the bounded set \\(\\Omega_k\\), implying that the condition at \\cref{state:PANOC+:gammaLS} should have terminated in finite time, whence the sought contradiction.\n\t\t\t\n\t\t\tHence, \\(\\gamma_{k,j}\\) is backtracked finitely many times within iteration \\(k\\);\n\t\t\tup to discarding early attempts, we may denote \\(\\gamma_{k,j}=\\gamma_k\\).\n\t\t\tCondition \\eqref{eq:prox:better} reads\n\t\t\t\\begin{align*}\n\t\t\t\t\\LL(x^{k,j},\\bar x^{k,j},-\\nabla f(x^{k,j}))\n\t\t\t{}\\leq{} &\n\t\t\t\t\\LL(x^{k,j},\\bar x^{k-1},-\\nabla f(x^{k,j}))\n\t\t\t\\\\\n\t\t\t{}={} &\n\t\t\t\tf(x^{k,j})+g(\\bar x^{k-1})+\\innprod*{\\nabla f(x^{k,j})}{\\bar x^{k-1}-x^{k,j}}\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t{+{}}\\tfrac{1}{2\\gamma}\\|x^{k,j}-\\bar x^{k-1}\\|^2.\n\t\t\t\\end{align*}\n\t\t\tAs \\(\\tau_{k,j}\\searrow0\\), one has that \\(x^{k,j}\\to\\bar x^{k-1}\\).\n\t\t\tSince \\(f\\) and \\(\\nabla f\\) are continuous, the right-hand side of the inequality converges to \\(\\varphi(\\bar x^{k-1})\\), overall resulting in\n\t\t\t\\[\n\t\t\t\\limsup_{j\\to\\infty}\\LL(x^{k,j},\\bar x^{k,j},-\\nabla f(x^{k,j}))\n\t\t\t{}\\leq{}\n\t\t\t\\varphi(\\bar x^{k-1})\n\t\t\t{}\\overrel[\\leq]{\\eqref{eq:lphi}}{}\n\t\t\t\\Phi_{k-1}-\\tfrac{1-\\alpha}{2\\gamma_{k-1}}\\|x^{k-1}-\\bar x^{k-1}\\|^2.\n\t\t\t\\]\n\t\t\tSince \\(\\|x^{k-1}-\\bar x^{k-1}\\|>0\\) and \\(\\beta<1\\), for \\(j\\) large enough the condition at \\cref{state:PANOC+:tauLS} will be violated and therefore the \\(k\\)-th iteration successfully terminated.\n\t\t%%\n\t\t%%\n\t\t\\item\\ref{thm:finite:descent}\n\t\t\tFollows by combining \\eqref{eq:lphi} with the failure of the condition at \\cref{state:PANOC+:tauLS} at the end of the iteration.\n\t\t\\item\\ref{thm:finite:sublevel}~\n\t\t\tDirect consequence of assertion \\ref{thm:finite:descent}.\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}\n\\end{lem}","sha256_utf8":"15d5c51809bef0272cbe7168ff916d04dd7447df7bc3588b3e956e548f8146c5"},{"start":61496,"end":64610,"start_line":1212,"end_line":1307,"raw_tex":"\\begin{proof}\n\t\t\\begin{proofitemize}\n\t\t\\item\\ref{thm:PANOC+:asymp:cost}~\n\t\t\tFollows from \\eqref{eq:SD}.\n\t\t\\item\\ref{thm:PANOC+:asymp:summable}~\n\t\t\tA telescoping argument on \\eqref{eq:SD} yields\n\t\t\t\\begin{equation}\\label{eq:summable}\n\t\t\t\t\\beta(1-\\alpha)\n\t\t\t\t\\sum_{k\\in\\N}{\n\t\t\t\t\t\\tfrac{1}{2\\gamma_k}\n\t\t\t\t\t\\|\\bar x^k-x^k\\|^2\n\t\t\t\t}\n\t\t\t{}\\leq{}\n\t\t\t\t\\Phi_0\n\t\t\t\t{}-{}\n\t\t\t\t\\inf\\varphi\n\t\t\t{}={}\n\t\t\t\t\\def\\iter{0}\n\t\t\t\t\\FBE(x^0)-\\inf\\varphi,\n\t\t\t\\end{equation}\n\t\t\twhence the claimed finite sum.\n\t%%%%\n\t%%%%\n\t%%%%\n\t\t\\item\\ref{thm:PANOC+:asymp:fpr}~\n\t\t\tThat \\(\\|x^k-\\bar x^k\\|\\to0\\) follows from assertion \\ref{thm:PANOC+:asymp:summable}, since \\(\\gamma_k\\) is upper bounded.\n\t\t\tNext, by the conditions at \\cref{state:PANOC+:x+,,state:PANOC+:d}, observe that\n\t\t\t\\begin{align*}\n\t\t\t\\numberthis\\label{eq:xkdiff}\n\t\t\t\t\\|x^k-x^{k-1}\\|\n\t\t\t{}={} &\n\t\t\t\t\\bigl\\|\n\t\t\t\t\t(1-\\tau_k)(\\bar x^{k-1}-x^{k-1})\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tau_kd^k\n\t\t\t\t\\bigr\\|\n\t\t\t{}\\leq{}\n\t\t\t\t(1+D)\\|\\bar x^{k-1}-x^{k-1}\\|\n\t\t\t\\shortintertext{and thus \\(\\|x^k-x^{k-1}\\|\\) vanishes, and in turn so does \\(\\|\\bar x^k-\\bar x^{k-1}\\|\\) since}\n\t\t\t\t\\|\\bar x^k-\\bar x^{k-1}\\|\n\t\t\t{}\\leq{} &\n\t\t\t\t\\|x^k-\\bar x^k\\|\n\t\t\t\t{}+{}\n\t\t\t\t\\|\\bar x^{k-1}-x^{k-1}\\|\n\t\t\t\t{}+{}\n\t\t\t\t\\|x^k-x^{k-1}\\|.\n\t\t\t\\end{align*}\n\t\t\\item\\ref{thm:PANOC+:asymp:gammaconstant}~\n\t\t\tThe first implication follows from \\cref{thm:finite:sublevel}, and the second one from assertion \\ref{thm:PANOC+:asymp:summable}.\n\t\t\tIf \\(\\seq{x^k}\\) is bounded, and thus so is \\(\\seq{\\bar x^k}\\), the set \\(\\Omega_k\\) in the proof of \\cref{thm:finite:LS} can be taken independent of \\(k\\), and asymptotic constancy of \\(\\gamma_k\\) follows from the same arguments therein.\n\t\t\tFinally, if \\(\\nabla f\\) is $L_f$-Lipschitz continuous the condition at \\cref{state:PANOC+:gammaLS} fails to hold as soon as $\\gamma_k\\leq\\nicefrac{\\alpha}{L_f}$ \\cite[Prop. A.24]{bertsekas1999nonlinear}, and \\(\\gamma_k\\) is thus asymptotically constant.\n\t%%%%\n\t%%%%\n\t%%%%\n\t\t\\item\\ref{thm:PANOC+:asymp:gamma}~\n\t\t\tBy iteratively applying inequality \\eqref{eq:xkdiff}, we obtain that\n\t\t\t\\begin{align*}\n\t\t\t\t\\|x^k-x^0\\|\n\t\t\t{}\\leq{} &\n\t\t\t\t(1+D)\\sum_{j=0}^{k-1}\\|\\bar x^j-x^j\\|\n\t\t\t\\\\\n\t\t\t{}={} &\n\t\t\t\t(1+D)\\sum_{j=0}^{k-1}\\gamma_j^{-\\nicefrac12}\\|\\bar x^j-x^j\\|\\gamma_j^{\\nicefrac12}\n\t\t\t\\\\\n\t\t\t{}\\leq{} &\n\t\t\t\t\\textstyle\n\t\t\t\t(1+D)\n\t\t\t\t\\sqrt{\n\t\t\t\t\t\\sum_{j=0}^{k-1}\n\t\t\t\t\t\\gamma_j^{-1}\\|\\bar x^j-x^j\\|^2\n\t\t\t\t}\n\t\t\t\t\\sqrt{\n\t\t\t\t\t\\sum_{j=0}^{k-1}\\gamma_j\n\t\t\t\t}\n\t\t\t\\\\\n\t\t\t{}\\overrel*[\\leq]{\\eqref{eq:summable}}{} &\n\t\t\t\t\\textstyle\n\t\t\t\t(1+D)\n\t\t\t\t\\sqrt{\n\t\t\t\t\t2\n\t\t\t\t\t\\frac{\\def\\iter{0}\\FBE(x^0)-\\inf\\varphi}{\\beta(1-\\alpha)}\n\t\t\t\t}\n\t\t\t\t\\sqrt{\n\t\t\t\t\t\\sum_{j=0}^{k-1}\\gamma_j\n\t\t\t\t}.\n\t\t\t\\end{align*}\n\t\t\tContrary to the claim, if \\(\\sum_{k\\in\\N}\\gamma_k<\\infty\\) holds, then \\(\\seq{x^k}\\) is bounded.\n\t\t\tFrom assertion \\ref{thm:PANOC+:asymp:gammaconstant} proven above we then infer that \\(\\gamma_k\\) is asymptotically constant, thus contradicting the finiteness of \\(\\sum_{k\\in\\N}\\gamma_k\\).\n\t%%%%\n\t%%%%\n\t%%%%\n\t\t\\item\\ref{thm:PANOC+:asymp:res}~\n\t\t\tImmediate consequence of assertions \\ref{thm:PANOC+:asymp:summable} and \\ref{thm:PANOC+:asymp:gamma}.\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}","sha256_utf8":"12c0ddf4b40642d020fbbad2a0e1e64be8a839415bbe6f1f40d9d1d6ac3b0c46"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"非精确PANOC+的渐近收敛","source_units":[{"start":59431,"end":61496,"start_line":1156,"end_line":1212,"raw_tex":"\\begin{thm}[Asymptotic analysis of the ``good'' (inexact) \\panoc+]\\label{thm:PANOC+:asymp}%\n\tConsider the iterates generated by \\cref{alg:PANOC+} with inexact proximal evaluation at \\cref{state:PANOC+:barx} as given in \\eqref{eq:prox:better}.\n\tThe following hold:\n\t\\begin{enumerate}\n\t\\item\\label{thm:PANOC+:asymp:cost}%\n\t\t\\(\\seq{\\Phi_k}\\) converges to a finite value \\(\\varphi_\\star\\geq\\inf\\varphi\\) from above.\n\t\\item\\label{thm:PANOC+:asymp:summable}%\n\t\t\\(\n\t\t\t\\sum_{k\\in\\N}\\frac{1}{\\gamma_k}\\|\\bar x^k-x^k\\|^2\n\t\t{}<{}\n\t\t\t\\infty\n\t\t\\).\n\t\\item\\label{thm:PANOC+:asymp:fpr}%\n\t\t\\(\n\t\t\t\\lim_{k\\to\\infty}\\|x^k-\\bar x^k\\|\n\t\t{}={}\n\t\t\t\\lim_{k\\to\\infty}\\|x^k-x^{k-1}\\|\n\t\t{}={}\n\t\t\t\\lim_{k\\to\\infty}\\|\\bar x^k-\\bar x^{k-1}\\|\n\t\t{}={}\n\t\t\t0\n\t\t\\),\n\t\tand in particular the set of limit points of \\(\\seq{x^k}\\) is closed and connected, and coincides with that of \\(\\seq{\\bar x^k}\\).\n\t\\item\\label{thm:PANOC+:asymp:gamma}%\n\t\t\\(\\sum_{k\\in\\N}\\gamma_k=\\infty\\).\n\t\\item\\label{thm:PANOC+:asymp:res}%\n\t\t\\(\n\t\t\t\\liminf_{k\\to\\infty}\\frac{1}{\\gamma_k}\\|x^k-\\bar x^k\\|\n\t\t{}={}\n\t\t\t0\n\t\t\\).\n\t\\item\\label{thm:PANOC+:asymp:gammaconstant}%\n\t\tConsider the following assertions:\n\t\t\\begin{enumerate}[label={(\\oldstylenums{\\arabic*})},ref={(\\oldstylenums{\\arabic*})}]\n\t\t\\item\\label{thm:PANOC+:gammaconstant:lb}%\n\t\t\t\\(\\varphi\\) is level bounded;\n\t\t\\item\\label{thm:PANOC+:gammaconstant:xk}%\n\t\t\t\\(\\seq{\\bar x^k}\\) is bounded;\n\t\t\\item\\label{thm:PANOC+:gammaconstant:barxk}%\n\t\t\t\\(\\seq{x^k}\\) is bounded;\n\t\t\\item\\label{thm:PANOC+:gammaconstant:gamma}%\n\t\t\t\\(\\seq{\\gamma_k}\\) is asymptotically constant, \\ie, there exists \\(\\kappa\\in\\N\\) such that \\(\\gamma_k=\\gamma_\\kappa\\) for every \\(k\\geq\\kappa\\);\n\t\t\\item\\label{thm:PANOC+:gammaconstant:Lf}%\n\t\t\t\\(f\\) has globally Lipschitz-continuous gradient.\n\t\t\\end{enumerate}\n\t\tOne has~\n\t\t\\ref{thm:PANOC+:gammaconstant:lb}\n\t\t~\\(\\Rightarrow\\)~\n\t\t\\ref{thm:PANOC+:gammaconstant:xk}\n\t\t~\\(\\Leftrightarrow\\)~\n\t\t\\ref{thm:PANOC+:gammaconstant:barxk}\n\t\t~\\(\\Rightarrow\\)~\n\t\t\\ref{thm:PANOC+:gammaconstant:gamma}\n\t\t~\\(\\Leftarrow\\)~\n\t\t\\ref{thm:PANOC+:gammaconstant:Lf}.\n\t\\end{enumerate}\n\t","sha256_utf8":"b73de2e00008eea7adddc0f7cba1382f9d0749d09e946d9d61f3eeabad9ee8e4"},{"start":64610,"end":64620,"start_line":1307,"end_line":1308,"raw_tex":"\n\\end{thm}","sha256_utf8":"27661e277ceb78e4dd676b4ea4b5c8d9a4d3c607a1a758eac095282a4f311f77"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"渐近收敛定理完整证明","source_units":[{"start":61496,"end":64610,"start_line":1212,"end_line":1307,"raw_tex":"\\begin{proof}\n\t\t\\begin{proofitemize}\n\t\t\\item\\ref{thm:PANOC+:asymp:cost}~\n\t\t\tFollows from \\eqref{eq:SD}.\n\t\t\\item\\ref{thm:PANOC+:asymp:summable}~\n\t\t\tA telescoping argument on \\eqref{eq:SD} yields\n\t\t\t\\begin{equation}\\label{eq:summable}\n\t\t\t\t\\beta(1-\\alpha)\n\t\t\t\t\\sum_{k\\in\\N}{\n\t\t\t\t\t\\tfrac{1}{2\\gamma_k}\n\t\t\t\t\t\\|\\bar x^k-x^k\\|^2\n\t\t\t\t}\n\t\t\t{}\\leq{}\n\t\t\t\t\\Phi_0\n\t\t\t\t{}-{}\n\t\t\t\t\\inf\\varphi\n\t\t\t{}={}\n\t\t\t\t\\def\\iter{0}\n\t\t\t\t\\FBE(x^0)-\\inf\\varphi,\n\t\t\t\\end{equation}\n\t\t\twhence the claimed finite sum.\n\t%%%%\n\t%%%%\n\t%%%%\n\t\t\\item\\ref{thm:PANOC+:asymp:fpr}~\n\t\t\tThat \\(\\|x^k-\\bar x^k\\|\\to0\\) follows from assertion \\ref{thm:PANOC+:asymp:summable}, since \\(\\gamma_k\\) is upper bounded.\n\t\t\tNext, by the conditions at \\cref{state:PANOC+:x+,,state:PANOC+:d}, observe that\n\t\t\t\\begin{align*}\n\t\t\t\\numberthis\\label{eq:xkdiff}\n\t\t\t\t\\|x^k-x^{k-1}\\|\n\t\t\t{}={} &\n\t\t\t\t\\bigl\\|\n\t\t\t\t\t(1-\\tau_k)(\\bar x^{k-1}-x^{k-1})\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tau_kd^k\n\t\t\t\t\\bigr\\|\n\t\t\t{}\\leq{}\n\t\t\t\t(1+D)\\|\\bar x^{k-1}-x^{k-1}\\|\n\t\t\t\\shortintertext{and thus \\(\\|x^k-x^{k-1}\\|\\) vanishes, and in turn so does \\(\\|\\bar x^k-\\bar x^{k-1}\\|\\) since}\n\t\t\t\t\\|\\bar x^k-\\bar x^{k-1}\\|\n\t\t\t{}\\leq{} &\n\t\t\t\t\\|x^k-\\bar x^k\\|\n\t\t\t\t{}+{}\n\t\t\t\t\\|\\bar x^{k-1}-x^{k-1}\\|\n\t\t\t\t{}+{}\n\t\t\t\t\\|x^k-x^{k-1}\\|.\n\t\t\t\\end{align*}\n\t\t\\item\\ref{thm:PANOC+:asymp:gammaconstant}~\n\t\t\tThe first implication follows from \\cref{thm:finite:sublevel}, and the second one from assertion \\ref{thm:PANOC+:asymp:summable}.\n\t\t\tIf \\(\\seq{x^k}\\) is bounded, and thus so is \\(\\seq{\\bar x^k}\\), the set \\(\\Omega_k\\) in the proof of \\cref{thm:finite:LS} can be taken independent of \\(k\\), and asymptotic constancy of \\(\\gamma_k\\) follows from the same arguments therein.\n\t\t\tFinally, if \\(\\nabla f\\) is $L_f$-Lipschitz continuous the condition at \\cref{state:PANOC+:gammaLS} fails to hold as soon as $\\gamma_k\\leq\\nicefrac{\\alpha}{L_f}$ \\cite[Prop. A.24]{bertsekas1999nonlinear}, and \\(\\gamma_k\\) is thus asymptotically constant.\n\t%%%%\n\t%%%%\n\t%%%%\n\t\t\\item\\ref{thm:PANOC+:asymp:gamma}~\n\t\t\tBy iteratively applying inequality \\eqref{eq:xkdiff}, we obtain that\n\t\t\t\\begin{align*}\n\t\t\t\t\\|x^k-x^0\\|\n\t\t\t{}\\leq{} &\n\t\t\t\t(1+D)\\sum_{j=0}^{k-1}\\|\\bar x^j-x^j\\|\n\t\t\t\\\\\n\t\t\t{}={} &\n\t\t\t\t(1+D)\\sum_{j=0}^{k-1}\\gamma_j^{-\\nicefrac12}\\|\\bar x^j-x^j\\|\\gamma_j^{\\nicefrac12}\n\t\t\t\\\\\n\t\t\t{}\\leq{} &\n\t\t\t\t\\textstyle\n\t\t\t\t(1+D)\n\t\t\t\t\\sqrt{\n\t\t\t\t\t\\sum_{j=0}^{k-1}\n\t\t\t\t\t\\gamma_j^{-1}\\|\\bar x^j-x^j\\|^2\n\t\t\t\t}\n\t\t\t\t\\sqrt{\n\t\t\t\t\t\\sum_{j=0}^{k-1}\\gamma_j\n\t\t\t\t}\n\t\t\t\\\\\n\t\t\t{}\\overrel*[\\leq]{\\eqref{eq:summable}}{} &\n\t\t\t\t\\textstyle\n\t\t\t\t(1+D)\n\t\t\t\t\\sqrt{\n\t\t\t\t\t2\n\t\t\t\t\t\\frac{\\def\\iter{0}\\FBE(x^0)-\\inf\\varphi}{\\beta(1-\\alpha)}\n\t\t\t\t}\n\t\t\t\t\\sqrt{\n\t\t\t\t\t\\sum_{j=0}^{k-1}\\gamma_j\n\t\t\t\t}.\n\t\t\t\\end{align*}\n\t\t\tContrary to the claim, if \\(\\sum_{k\\in\\N}\\gamma_k<\\infty\\) holds, then \\(\\seq{x^k}\\) is bounded.\n\t\t\tFrom assertion \\ref{thm:PANOC+:asymp:gammaconstant} proven above we then infer that \\(\\gamma_k\\) is asymptotically constant, thus contradicting the finiteness of \\(\\sum_{k\\in\\N}\\gamma_k\\).\n\t%%%%\n\t%%%%\n\t%%%%\n\t\t\\item\\ref{thm:PANOC+:asymp:res}~\n\t\t\tImmediate consequence of assertions \\ref{thm:PANOC+:asymp:summable} and \\ref{thm:PANOC+:asymp:gamma}.\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}","sha256_utf8":"12c0ddf4b40642d020fbbad2a0e1e64be8a839415bbe6f1f40d9d1d6ac3b0c46"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"复合非凸最小化问题","source_units":[{"start":15098,"end":15174,"start_line":256,"end_line":258,"raw_tex":"\\[\\tag{P}\\label{eq:P}\n\t\\minimize_{x\\in\\R^n}\\varphi(x)\\coloneqq f(x)+g(x),\n\\]","sha256_utf8":"a7638b298f19040d7025505cdec8a01a301a6d8a5a1a1ffdeccbf4674df924b4"}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","title":"局部光滑与近端有界假设","source_units":[{"start":15307,"end":15669,"start_line":262,"end_line":272,"raw_tex":"\\begin{ass}\n\t\tThe following hold in problem \\eqref{eq:P}:\n\t\t\\begin{enumeratass}\n\t\t\\item\\label{ass:f}%\n\t\t\t\\(\\func{f}{\\R^n}{\\R}\\) has locally Lipschitz-continuous gradient.\n\t\t\\item\\label{ass:g}%\n\t\t\t\\(\\func{g}{\\R^n}{\\R\\cup\\set\\infty}\\) is proper, lsc, and \\(\\gamma_g\\)-prox-bounded.\n\t\t\\item\\label{ass:phi}%\n\t\t\t\\(\\inf\\varphi>-\\infty\\).\n\t\t\\end{enumeratass}\n\t\\end{ass}","sha256_utf8":"316eec3423f62ed1176b9a962f22f1ad5cc82cf3fa3120299c9cc0932b8aacfb"}]},{"id":"S005","semantic_type":"notation","kind":"notation","title":"定义域与下水平集记号","source_units":[{"start":16252,"end":16895,"start_line":280,"end_line":282,"raw_tex":"\t\t\t\tWith \\(\\R\\) and \\(\\Rinf\\coloneqq\\R\\cup\\set\\infty\\) we denote the real and extended-real line, and by \\(\\N=\\set{0,1,\\dots}\\) the set of natural numbers.\nThe effective domain of an extended-real-valued function \\(\\func{h}{\\R^n}{\\Rinf}\\) is denoted by \\(\\dom h\\coloneqq\\set{x\\in\\R^n}[h(x)<\\infty]\\), and we say that \\(h\\) is: proper if \\(\\dom h\\neq\\emptyset\\); lower semicontinuous (lsc) if \\(h(\\bar x)\\leq\\liminf_{x\\to\\bar x}h(x)\\) for all \\(\\bar x\\in\\R^n\\); coercive if \\(h(x)\\to\\infty\\) as \\(\\|x\\|\\to\\infty\\).\nFor $\\alpha\\in\\R$, the $\\alpha$-\\emph{sublevel set} of $h$ is $\\lev_{\\leq\\alpha} h \\coloneqq \\set{x\\in\\R^n : h(x) \\leq \\alpha}$.\n","sha256_utf8":"fab821b87e9c5bbb7f41366774cea2c1ef26f5b14e618ff1d28251504f40c7c3"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"近端映射与近端有界性","source_units":[{"start":18394,"end":19116,"start_line":301,"end_line":323,"raw_tex":"Given a parameter value \\(\\gamma>0\\), the \\emph{Moreau envelope} function \\(h^\\gamma\\) and the \\emph{proximal mapping} \\(\\prox_{\\gamma h}\\) are defined by\n\\begin{align}\n\t\\label{eq:moreauenvelope}\n\th^\\gamma(x)\n\t{}\\coloneqq{} &\n\t\\inf_{z\\in\\R^n}\\set{\n\t\th(z)\n\t\t{}+{}\n\t\t\\tfrac{1}{2\\gamma}\\|z-x\\|^2\n\t} ,\n\t\\\\\n\t\\label{eq:proxoperator}\n\t\\prox_{\\gamma h}(x)\n\t{}\\coloneqq{} &\n\t\\argmin_{z\\in\\R^n}\\set{\n\t\th(z)\n\t\t{}+{}\n\t\t\\tfrac{1}{2\\gamma}\\|z-x\\|^2\n\t},\n\\end{align}\n% Prox-boundedness\nand we say that \\(h\\) is \\emph{prox-bounded} if it is proper and \\(h + \\tfrac{1}{2\\gamma}\\|\\cdot\\|^2\\) is bounded below on \\(\\R^n\\) for some \\(\\gamma>0\\).\nThe supremum of all such \\(\\gamma\\) is the threshold \\(\\gamma_h\\) of prox-boundedness for \\(h\\).\n","sha256_utf8":"b0abcbc2574aaccd3db0db490a5d15b522124d356080dc61c9f44914af539103"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"近端梯度迭代映射","source_units":[{"start":19523,"end":19765,"start_line":329,"end_line":337,"raw_tex":"\t\t\t\tGiven a point \\(x \\in \\R^n\\), one iteration of the proximal gradient (PG) method for problem \\eqref{eq:P} consists in selecting\n\\begin{equation}\\label{eq:FBSiter}\n\t\\def\\iter{}\n\t\\bar x\n\t{}\\in{}\n\t\\T(x)\n\t{}\\coloneqq{}\n\t\\FB{x},\n\\end{equation}","sha256_utf8":"4275dd10c99103d5d91131713ea1ea11a3f3ae9d0a4396651804b08312a40a83"},{"start":19766,"end":19823,"start_line":338,"end_line":338,"raw_tex":"where \\(\\gamma\\in(0,\\gamma_g)\\) is a stepsize parameter.\n","sha256_utf8":"202bb4708d7701aed148ac89ffdc7ac4f3f83f479c077c987946be8289986807"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"前向后向包络","source_units":[{"start":21252,"end":22020,"start_line":362,"end_line":400,"raw_tex":"\\begin{defin}[forward-backward envelope]%\n\t\\begin{subequations}\\label{subeq:FBE}%\n\t\tRelative to \\eqref{eq:P}, the FBE with stepsize \\(\\gamma\\in(0,\\gamma_g)\\) is\n\t\t\\begin{align}\n\t\t\\label{eq:FBE}\n\t\t\t\\FBE(x)\n\t\t{}\\coloneqq{} &\n\t\t\t\\min_{w\\in\\R^n}\\set{\n\t\t\t\tf(x)\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod{\\nabla f(x)}{w-x}\n\t\t\t\t{}+{}\n\t\t\t\tg(w)\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma}\\|w-x\\|^2\n\t\t\t}\n\t\t\\\\\n\t\t\\label{eq:FBEMoreau}\n\t\t{}={} &\n\t\t\tf(x)\n\t\t\t{}-{}\n\t\t\t\\tfrac\\gamma2\\|\\nabla f(x)\\|^2\n\t\t\t{}+{}\n\t\t\tg^\\gamma(\\Fw x)\n\t\t\\shortintertext{%\n\t\t\tor, equivalently, letting \\(\\bar x\\) be any element of \\(\\T(x)\\),\n\t\t}\n\t\t\\label{eq:FBEmin}\n\t\t{}={} &\n\t\t\tf(x)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla f(x)}{\\bar x-x}\n\t\t\t{}+{}\n\t\t\tg(\\bar x)\n\t\t\t{}+{}\n\t\t\t\\tfrac{1}{2\\gamma}\\|\\bar x-x\\|^2.\n\t\t\\end{align}\n\t\\end{subequations}\n\\end{defin}","sha256_utf8":"8e8f54c6b4136c7afb49fbf9152210231d3f6d4450aeb5cb215f850db57293ac"}]},{"id":"S009","semantic_type":"algorithm","kind":"algorithm","title":"PANOC+嵌套回溯算法","source_units":[{"start":29805,"end":31402,"start_line":557,"end_line":631,"raw_tex":"\\begin{algorithm}[tb!]\n\t\\caption{\\panoc+: the ``good'' adaptive \\(\\gamma\\)-stepsize rule}%\n\t\\label{alg:PANOC+}%\n\t\\begin{algorithmic}[1]%\n\\linespread{1.4}\\selectfont%\n%\n%\n%\n\\Require\n\t\\(x^0\\in\\R^n\\);~~\n\t\\(\\gamma_0\\in(0,\\gamma_g)\\);~~\n\t\\(D\\geq0\\);~~\n\t\\(\\alpha,\\beta\\in(0,1)\\)%\n\\Initialize\n\t\\(k\\gets0\\),~~\n\tand start from \\cref{state:PANOC+:barx}\n%\n%\n\\vspace*{1ex}%\n\\hrule\n\\vspace*{1ex}%\n%\n%\n%\n\\def\\iter{k}%\n%\n%\n%\n\t\\State\n\t\t\\label{state:PANOC+:init}%\n\t\t\\(\\gamma_k\\gets\\gamma_{k-1}\\)\n\t\\State\n\t\t\\label{state:PANOC+:d}%\n\t\tSelect an update direction \\(d^k\\in\\R^n\\) with \\(\\|d^k\\|\\leq D\\|\\bar x^{k-1}-x^{k-1}\\|\\) and set \\(\\tau_k=1\\)%\n\t\\State\n\t\t\\label{state:PANOC+:x+}%\n\t\t\\(x^k=(1-\\tau_k)\\bar x^{k-1}+\\tau_k(x^{k-1}+d^k)\\)\n\t\\State\n\t\t\\label{state:PANOC+:barx}%\n\t\tCompute \\(\\bar x^k\\in\\T(x^k)\\)\n\t\tand use it to evaluate \\(\\Phi_k\\coloneqq\\FBE(x^k)\\) as in \\eqref{eq:FBEmin}%\n\t\\If{\n\t\t\\(\n\t\t\tf(\\bar x^k)\n\t\t{}>{}\n\t\t\tf(x^k)\n\t\t\t{}+{}\n\t\t\t\\innprod{\\nabla f(x^k)}{\\bar x^k-x^k}\n\t\t\t{}+{}\n\t\t\t\\tfrac{\\alpha}{2\\gamma_k}\\|\\bar x^k-x^k\\|^2\n\t\t\\)\n\t}%\n\t\\label{state:PANOC+:gammaLS}%\n\t\t\\Statex*%\n\t\t\t\\(\\gamma_k\\gets\\nicefrac{\\gamma_k}{2}\\),~\n\t\t\tand go back to \\cref{state:PANOC+:d} if \\(k>0\\), or \\cref{state:PANOC+:barx} if \\(k=0\\)\n\t\\EndIf\n\t\\If{ \\(k>0\\) ~{\\sc and}~ \\(\n\t\t\\Phi_k\n\t\t{}>{}\n\t\t\\Phi_{k-1}\n\t\t{}-{}\n\t\t\\beta\\tfrac{1-\\alpha}{2\\gamma_{k-1}}\n\t\t\\|\\bar x^{k-1}-x^{k-1}\\|^2\n\t\\) }\\label{state:PANOC+:tauLS}%\n\t\t\\Statex*\n\t\t\t\\(\\tau_k\\gets\\nicefrac{\\tau_k}{2}\\)~\n\t\t\tand go back to \\cref{state:PANOC+:x+}\n\t\\EndIf\n\t\\State\n\t\t\\(k\\gets k+1\\)~\n\t\tand start the next iteration at \\cref{state:PANOC+:init}\n\\end{algorithmic}\n%\n\\end{algorithm}","sha256_utf8":"1695c10379656f3a3509fd996ff1ebe2afae71ddef4ba323fd6a352cc3fbaa75"}]},{"id":"S010","semantic_type":"notation","kind":"notation","title":"试探变量与接受变量","source_units":[{"start":34005,"end":34917,"start_line":649,"end_line":655,"raw_tex":"\\begin{rem}[Algorithm notation]\\label{rem:notation}%\n\t\\Cref{alg:PANOC+} operates two linesearch steps within each iteration, one on the ``proximal'' stepsize \\(\\gamma_k\\) at \\cref{state:PANOC+:gammaLS} and one on the ``direction'' stepsize \\(\\tau_k\\) at \\cref{state:PANOC+:tauLS}.\n\tWhenever the respective needed conditions are violated, either \\(\\gamma_k\\) or \\(\\tau_k\\) is reduced and the iteration restarted from a previous step.\n\tAs a consequence, variables may be \\emph{overwritten} within each iteration before being accepted.\n\tTo avoid a heavy double-index notation, used only within proofs out of full rigor, the sub- and superscript notation is designed to differentiate temporary and permanent variables; specifically, within iteration \\(k\\) only variables indexed with \\(k\\) are updated, whereas those indexed with \\(k-1\\) remain untouched.\n\tSimilar considerations apply to \\cref{alg:PANOC}.\n\\end{rem}","sha256_utf8":"76856da9b14adfcffa8d115277e51f6adfd4bf7aa975051cc766a0875bb57ee2"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"增广拉格朗日与近端子问题","source_units":[{"start":49658,"end":49998,"start_line":960,"end_line":974,"raw_tex":"\\begin{align}\\label{eq:L}\n\t\t\\LL_\\beta(x,z,y)\n\t{}\\coloneqq{} &\n\t\tf(x)+g(z)+\\innprod{y}{x-z}+\\tfrac\\beta2\\|x-z\\|^2,\n\t\\shortintertext{we remark that}\\label{eq:L:FBE}\n\t\t\\FBE(x)\n\t{}={} &\n\t\t\\LL(x,\\bar x,-\\nabla f(x)),\n\t\\shortintertext{where}\\label{eq:L:barx}\n\t\t\\bar x\n\t{}\\in{} &\n\t\t\\T(x)\n\t\t{}={}\n\t\t\\argmin\\LL(x,{}\\cdot{},-\\nabla f(x))\n\t\\end{align}","sha256_utf8":"332620e858bedaf34909afe0ae2bf6b88ad650908723b5edfc8912c7b7f61740"}]},{"id":"S012","semantic_type":"algorithm","kind":"algorithm","title":"非精确近端下降要求","source_units":[{"start":51673,"end":52202,"start_line":991,"end_line":1005,"raw_tex":"\\begin{algorithmic}[1]%\n\t\\makeatletter\n\t\t\\renewcommand{\\alglinenumber}[1]{\\footnotesize\\fillwidthof[l]{\\oldstylenums{88}:}{{\\bf 2}.4$'$:}~}%\n\t\t\\renewcommand{\\theALG@line}{2.\\oldstylenums{4}$'$}%\n\t\\makeatother\n\t\\State\\label{state:NoProx:barx}%\n\t\tLet \\(\\bar x^k\\) be a suboptimal minimizer of \\(\\LL(x^k,{}\\cdot{},-\\nabla f(x^k))\\) such that\n\t\t\\begin{equation}\\label{eq:prox:better}\n\t\t\t\\Phi_k\n\t\t{}\\coloneqq{}\n\t\t\t\\LL(x^k,\\bar x^k,-\\nabla f(x^k))\n\t\t{}\\leq{}\n\t\t\t\\LL(x^k,\\bar x^{k-1},-\\nabla f(x^k)).\n\t\t\\end{equation}\n\t\\end{algorithmic}","sha256_utf8":"86b0e4a2f9c8391f1d6345867a3e9e69719bf63ded15ae7b5935b366bf749d8a"}]},{"id":"S013","semantic_type":"assumption","kind":"assumption","title":"排除零近端位移","source_units":[{"start":54085,"end":54173,"start_line":1039,"end_line":1039,"raw_tex":"To avoid trivialities, in what follows we assume that \\(x^k\\neq\\bar x^k\\) always holds.\n","sha256_utf8":"1790a1c2a9bf38b1c4343333fedd337b090f329da33857fa9393dcefd9dce7f1"}]},{"id":"S014","semantic_type":"lemma","kind":"lemma","title":"非精确近端试探点有界性","source_units":[{"start":53129,"end":53541,"start_line":1015,"end_line":1023,"raw_tex":"\\begin{lem}\\label{thm:inexact:lb}%\n\tLet a constant \\(c\\in\\R\\), a sequence \\(\\seq{\\gamma_j}[j\\in\\N]\\searrow0\\), and two bounded sequences \\(\\seq{u^j,z^j}[j\\in\\N]\\) in \\(\\R^n\\) be fixed, and for every \\(j\\in\\N\\) let \\(\\bar z^j\\) be such that\n\t\\[\n\t\tg(\\bar z^j)+\\innprod*{u^j}{\\bar z^j-z^j}+\\tfrac{1}{2\\gamma_j}\\|\\bar z^j-z^j\\|^2\n\t\t{}\\leq{}\n\t\t\\tfrac{c}{2\\gamma_j}.\n\t\\]\n\tThen, \\(\\seq{\\bar z^j}[j\\in\\N]\\) is bounded.\n\t","sha256_utf8":"ead354a85d07c32e1f299ddc0ab96e829ead9cf0d77b44b7092f0e97b6455cb0"},{"start":54073,"end":54083,"start_line":1036,"end_line":1037,"raw_tex":"\n\\end{lem}","sha256_utf8":"401658723a1e057444124cf77a6be177a3c9cb2efee9af44c2830fc4a15ddaba"}]},{"id":"S015","semantic_type":"proof","kind":"proof","title":"近端试探点有界性完整证明","source_units":[{"start":53541,"end":54073,"start_line":1023,"end_line":1036,"raw_tex":"\\begin{proof}\n\t\tAn application of Young's inequality on the inner product yields\n\t\t\\[\n\t\t2\\gamma_jg(\\bar z^j)\n\t\t{}\\leq{}\n\t\tc\n\t\t{}+{}\n\t\t\\gamma_j\\|u_j\\|^2\n\t\t{}-{}\n\t\t(1-\\gamma_j)\\|\\bar z^j-z^j\\|^2.\n\t\t\\]\n\t\tTo arrive to a contradiction, up to extracting if necessary, suppose that \\(0<\\|\\bar z^j\\|\\to\\infty\\).\n\t\tSince \\(\\liminf_{j\\to\\infty}{g(\\bar z^j)}/{\\|\\bar z^j\\|^2}>-\\infty\\) by \\cite[Ex. 1.24]{rockafellar1998variational}, dividing by \\(\\|\\bar z^j\\|^2\\) and passing to the limit leads to the contradiction \\(0\\leq -1\\).\n\t\\end{proof}","sha256_utf8":"5ccccd8405b00da1bcf79e600c6f1c354a61ae754b1bd4224c13600669f5f521"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"用Young不等式控制内积","source_units":[{"start":53555,"end":53740,"start_line":1024,"end_line":1033,"raw_tex":"\t\tAn application of Young's inequality on the inner product yields\n\t\t\\[\n\t\t2\\gamma_jg(\\bar z^j)\n\t\t{}\\leq{}\n\t\tc\n\t\t{}+{}\n\t\t\\gamma_j\\|u_j\\|^2\n\t\t{}-{}\n\t\t(1-\\gamma_j)\\|\\bar z^j-z^j\\|^2.\n\t\t\\]\n","sha256_utf8":"0f086bd330732fc0a479e4f05a00fe20c0056dd173f41ee6a4b6a3e794f28f6a"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"用二次下界排除无界试探点","source_units":[{"start":53740,"end":54061,"start_line":1034,"end_line":1035,"raw_tex":"\t\tTo arrive to a contradiction, up to extracting if necessary, suppose that \\(0<\\|\\bar z^j\\|\\to\\infty\\).\n\t\tSince \\(\\liminf_{j\\to\\infty}{g(\\bar z^j)}/{\\|\\bar z^j\\|^2}>-\\infty\\) by \\cite[Ex. 1.24]{rockafellar1998variational}, dividing by \\(\\|\\bar z^j\\|^2\\) and passing to the limit leads to the contradiction \\(0\\leq -1\\).\n","sha256_utf8":"73b549463cbd65074cf2b3e259b532752e289f3edf6ce8054434bee5bb8e6d26"}]},{"id":"S018","semantic_type":"lemma","kind":"lemma","title":"有限回溯与充分下降","source_units":[{"start":54380,"end":55372,"start_line":1042,"end_line":1070,"raw_tex":"\\begin{lem}[Well definedness of the ``good'' (inexact) \\panoc+]\\label{thm:PANOC+:finite}%\n\tConsider the iterates generated by \\cref{alg:PANOC+} with inexact proximal evaluation at \\cref{state:PANOC+:barx} as given in \\eqref{eq:prox:better}.\n\tThe following hold:\n\t\\begin{enumerate}\n\t\\item\\label{thm:finite:LS}%\n\t\tWell definedness: at every iteration, the number of backtrackings at \\cref{state:PANOC+:gammaLS,state:PANOC+:tauLS} is finite.\n\t\\item\\label{thm:finite:descent}%\n\t\tAt the end of the \\(k\\)-th iteration (\\(k\\geq1\\)), one has\n\t\t\\begin{equation}\\label{eq:SD}\n\t\t\t\t\\varphi(\\bar x^k)\n\t\t\t\t{}+{}\n\t\t\t\t\\delta_k\n\t\t\t{}\\leq{}\n\t\t\t\t\\Phi_k\n\t\t\t{}\\leq{}\n\t\t\t\t\\def\\iter{k-1}\n\t\t\t\t\\Phi_{k-1}\n\t\t\t\t{}-{}\n\t\t\t\t\\beta\\delta_{k-1}\n\t\t\t\\quad\\text{where}\\quad\n\t\t\t\t\\delta_k\n\t\t\t{}\\coloneqq{}\n\t\t\t\t\\tfrac{1-\\alpha}{2\\gamma_k}\n\t\t\t\t\\|\\bar x^k-x^k\\|^2.\n\t\t\t\\end{equation}\n\t\\item\\label{thm:finite:sublevel}%\n\t\tEvery iterate \\(\\bar x^k\\) remains within \\(\\lev_{\\leq c}\\varphi\\), where \\(c=\\Phi_0<\\infty\\).\n\t\\end{enumerate}\n\t","sha256_utf8":"1edc224b63981275b26cc49098333fc53ca6b14169855e9d14e27124ebd38326"},{"start":59360,"end":59370,"start_line":1151,"end_line":1152,"raw_tex":"\n\\end{lem}","sha256_utf8":"401658723a1e057444124cf77a6be177a3c9cb2efee9af44c2830fc4a15ddaba"}]},{"id":"S019","semantic_type":"proof","kind":"proof","title":"算法良定义性完整证明","source_units":[{"start":55372,"end":59360,"start_line":1070,"end_line":1151,"raw_tex":"\\begin{proof}\n\t\tAs observed in \\cref{rem:notation}, each iteration \\(k\\) defines or updates only variables indexed with a \\(k\\) sub/superscript, while those defined in previous iterations are untouched.\n\t\tIn what follows, let us index by \\({k,j}\\) the variables defined at the \\(j\\)-th attempt within iteration \\(k\\).\n\t\tNote further that \\(\\gamma_{k,j}L_{k,j}=\\alpha\\in(0,1)\\) holds for every attempt \\(j\\) within every iteration \\(k\\), since every time \\(\\gamma_k\\) is halvened the estimate \\(L_k\\) is doubled (cf. \\cref{state:PANOC+:gammaLS}).\n\t\t\\begin{proofitemize}\n\t\t\\item\\ref{thm:finite:LS}~\n\t\t\tWe proceed by induction on \\(k\\).\n\t\t\tIf \\(k=0\\), there is no backtracking on \\(\\tau\\), and from \\cref{thm:inexact:lb} we conclude that all the trials \\(\\bar x^{0,j}\\) remain confined in a bounded set \\(\\Omega_0\\), and therefore any stepsize \\(\\gamma_{0,j}<\\nicefrac{1}{L_{f,\\Omega_0}}\\) is accepted.\n\n\t\t\tSuppose now that \\(k>0\\) and observe that, by the definition of \\(\\Phi_k\\) in \\eqref{eq:prox:better} and the failure of the condition at \\cref{state:PANOC+:gammaLS}, the inequality\n\t\t\t\\begin{equation}\\label{eq:lphi}\n\t\t\t\t\\varphi(\\bar x^{k-1})\n\t\t\t\t{}\\leq{}\n\t\t\t\t\\Phi_{k-1}\n\t\t\t\t{}-{}\n\t\t\t\t\\tfrac{1-\\alpha}{2\\gamma_{k-1}}\\|x^{k-1}-\\bar x^{k-1}\\|^2\n\t\t\t\\end{equation}\n\t\t\tholds.\n\t\t\tSince \\(\\|d^{k,j}\\|\\leq D\\|\\bar x^{k-1}-x^{k-1}\\|\\) and \\(\\tau_{k,j}\\in[0,1]\\), any attempt \\(x^{k,j}\\) defined at \\cref{state:PANOC+:x+} during the \\(k\\)-th iteration satisfies\n\t\t\t\\[\n\t\t\t\t\\|x^{k,j}-\\bar x^{k-1}\\|\n\t\t\t{}={}\n\t\t\t\t\\tau_{k,j}\\|\n\t\t\t\t\tx^{k-1}-\\bar x^{k-1}\n\t\t\t\t\t{}+{}\n\t\t\t\t\td^{k,j}\n\t\t\t\t\\|\n\t\t\t{}\\leq{}\n\t\t\t\t(1+D)\\|\\bar x^{k-1}-x^{k-1}\\|\n\t\t\t\\]\n\t\t\tand thus remains in a bounded set, be it \\(\\Omega_k\\).\n\t\t\tTo arrive to a contradiction, suppose that \\(\\gamma_{k,j}\\searrow0\\) as \\(j\\to\\infty\\).\n\t\t\tObserve that condition \\eqref{eq:prox:better} reads\n\t\t\t\\begin{Align*}\n\t\t\t\tg(\\bar x^{k,j})\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod*{\\nabla f(x^{k,j})}{\\bar x^{k,j}-\\bar x^{k-1}}\\quad\n\t\t\t&\n\t\t\t\\\\\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma_{k,j}}\\|x^{k,j}-\\bar x^{k,j}\\|^2\n\t\t\t& {}\\leq{}\n\t\t\t\tg(\\bar x^{k-1})\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma_{k,j}}\\|x^{k,j}-\\bar x^{k-1}\\|^2.\n\t\t\t\\end{Align*}\n\t\t\tSince \\(\\seq{x^{k,j}}[j\\in\\N]\\) is bounded, an application of \\cref{thm:inexact:lb} reveals that \\(\\seq{\\bar x^{k,j}}\\) too is bounded.\n\t\t\tUp to possibly enlarging the set, both sequences remain confined in the bounded set \\(\\Omega_k\\), implying that the condition at \\cref{state:PANOC+:gammaLS} should have terminated in finite time, whence the sought contradiction.\n\t\t\t\n\t\t\tHence, \\(\\gamma_{k,j}\\) is backtracked finitely many times within iteration \\(k\\);\n\t\t\tup to discarding early attempts, we may denote \\(\\gamma_{k,j}=\\gamma_k\\).\n\t\t\tCondition \\eqref{eq:prox:better} reads\n\t\t\t\\begin{align*}\n\t\t\t\t\\LL(x^{k,j},\\bar x^{k,j},-\\nabla f(x^{k,j}))\n\t\t\t{}\\leq{} &\n\t\t\t\t\\LL(x^{k,j},\\bar x^{k-1},-\\nabla f(x^{k,j}))\n\t\t\t\\\\\n\t\t\t{}={} &\n\t\t\t\tf(x^{k,j})+g(\\bar x^{k-1})+\\innprod*{\\nabla f(x^{k,j})}{\\bar x^{k-1}-x^{k,j}}\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t{+{}}\\tfrac{1}{2\\gamma}\\|x^{k,j}-\\bar x^{k-1}\\|^2.\n\t\t\t\\end{align*}\n\t\t\tAs \\(\\tau_{k,j}\\searrow0\\), one has that \\(x^{k,j}\\to\\bar x^{k-1}\\).\n\t\t\tSince \\(f\\) and \\(\\nabla f\\) are continuous, the right-hand side of the inequality converges to \\(\\varphi(\\bar x^{k-1})\\), overall resulting in\n\t\t\t\\[\n\t\t\t\\limsup_{j\\to\\infty}\\LL(x^{k,j},\\bar x^{k,j},-\\nabla f(x^{k,j}))\n\t\t\t{}\\leq{}\n\t\t\t\\varphi(\\bar x^{k-1})\n\t\t\t{}\\overrel[\\leq]{\\eqref{eq:lphi}}{}\n\t\t\t\\Phi_{k-1}-\\tfrac{1-\\alpha}{2\\gamma_{k-1}}\\|x^{k-1}-\\bar x^{k-1}\\|^2.\n\t\t\t\\]\n\t\t\tSince \\(\\|x^{k-1}-\\bar x^{k-1}\\|>0\\) and \\(\\beta<1\\), for \\(j\\) large enough the condition at \\cref{state:PANOC+:tauLS} will be violated and therefore the \\(k\\)-th iteration successfully terminated.\n\t\t%%\n\t\t%%\n\t\t\\item\\ref{thm:finite:descent}\n\t\t\tFollows by combining \\eqref{eq:lphi} with the failure of the condition at \\cref{state:PANOC+:tauLS} at the end of the iteration.\n\t\t\\item\\ref{thm:finite:sublevel}~\n\t\t\tDirect consequence of assertion \\ref{thm:finite:descent}.\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}","sha256_utf8":"c9083988fba297554905c80a46c687a8ee7fb3edd52389f69e81ac6be87b27b3"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"引入双重试探指标","source_units":[{"start":55386,"end":55941,"start_line":1071,"end_line":1074,"raw_tex":"\t\tAs observed in \\cref{rem:notation}, each iteration \\(k\\) defines or updates only variables indexed with a \\(k\\) sub/superscript, while those defined in previous iterations are untouched.\n\t\tIn what follows, let us index by \\({k,j}\\) the variables defined at the \\(j\\)-th attempt within iteration \\(k\\).\n\t\tNote further that \\(\\gamma_{k,j}L_{k,j}=\\alpha\\in(0,1)\\) holds for every attempt \\(j\\) within every iteration \\(k\\), since every time \\(\\gamma_k\\) is halvened the estimate \\(L_k\\) is doubled (cf. \\cref{state:PANOC+:gammaLS}).\n\t\t\\begin{proofitemize}\n","sha256_utf8":"6811cc4e02048fc56237f17c339857b03dab17dbf635dcb37c9de03bc61a1440"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"初始迭代有限回溯","source_units":[{"start":55941,"end":56272,"start_line":1075,"end_line":1077,"raw_tex":"\t\t\\item\\ref{thm:finite:LS}~\n\t\t\tWe proceed by induction on \\(k\\).\n\t\t\tIf \\(k=0\\), there is no backtracking on \\(\\tau\\), and from \\cref{thm:inexact:lb} we conclude that all the trials \\(\\bar x^{0,j}\\) remain confined in a bounded set \\(\\Omega_0\\), and therefore any stepsize \\(\\gamma_{0,j}<\\nicefrac{1}{L_{f,\\Omega_0}}\\) is accepted.\n","sha256_utf8":"ce16d0f8db8df17410e1206fba551abaf33444c093af3c0b1da27dd507eab5c6"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"上一迭代的目标下降界","source_units":[{"start":56273,"end":56646,"start_line":1079,"end_line":1087,"raw_tex":"\t\t\tSuppose now that \\(k>0\\) and observe that, by the definition of \\(\\Phi_k\\) in \\eqref{eq:prox:better} and the failure of the condition at \\cref{state:PANOC+:gammaLS}, the inequality\n\t\t\t\\begin{equation}\\label{eq:lphi}\n\t\t\t\t\\varphi(\\bar x^{k-1})\n\t\t\t\t{}\\leq{}\n\t\t\t\t\\Phi_{k-1}\n\t\t\t\t{}-{}\n\t\t\t\t\\tfrac{1-\\alpha}{2\\gamma_{k-1}}\\|x^{k-1}-\\bar x^{k-1}\\|^2\n\t\t\t\\end{equation}\n\t\t\tholds.\n","sha256_utf8":"a7249775af1817f803ed775aa6c4058fd8722b91c20b2ee21212f93e17f6ad1d"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"方向约束保证候选点有界","source_units":[{"start":56646,"end":57055,"start_line":1088,"end_line":1100,"raw_tex":"\t\t\tSince \\(\\|d^{k,j}\\|\\leq D\\|\\bar x^{k-1}-x^{k-1}\\|\\) and \\(\\tau_{k,j}\\in[0,1]\\), any attempt \\(x^{k,j}\\) defined at \\cref{state:PANOC+:x+} during the \\(k\\)-th iteration satisfies\n\t\t\t\\[\n\t\t\t\t\\|x^{k,j}-\\bar x^{k-1}\\|\n\t\t\t{}={}\n\t\t\t\t\\tau_{k,j}\\|\n\t\t\t\t\tx^{k-1}-\\bar x^{k-1}\n\t\t\t\t\t{}+{}\n\t\t\t\t\td^{k,j}\n\t\t\t\t\\|\n\t\t\t{}\\leq{}\n\t\t\t\t(1+D)\\|\\bar x^{k-1}-x^{k-1}\\|\n\t\t\t\\]\n\t\t\tand thus remains in a bounded set, be it \\(\\Omega_k\\).\n","sha256_utf8":"040a512037db20be65c6f50621afc0e036c4c91bb63f8751d6c54af358124f96"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"排除近端步长无限减半","source_units":[{"start":57055,"end":57877,"start_line":1101,"end_line":1117,"raw_tex":"\t\t\tTo arrive to a contradiction, suppose that \\(\\gamma_{k,j}\\searrow0\\) as \\(j\\to\\infty\\).\n\t\t\tObserve that condition \\eqref{eq:prox:better} reads\n\t\t\t\\begin{Align*}\n\t\t\t\tg(\\bar x^{k,j})\n\t\t\t\t{}+{}\n\t\t\t\t\\innprod*{\\nabla f(x^{k,j})}{\\bar x^{k,j}-\\bar x^{k-1}}\\quad\n\t\t\t&\n\t\t\t\\\\\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma_{k,j}}\\|x^{k,j}-\\bar x^{k,j}\\|^2\n\t\t\t& {}\\leq{}\n\t\t\t\tg(\\bar x^{k-1})\n\t\t\t\t{}+{}\n\t\t\t\t\\tfrac{1}{2\\gamma_{k,j}}\\|x^{k,j}-\\bar x^{k-1}\\|^2.\n\t\t\t\\end{Align*}\n\t\t\tSince \\(\\seq{x^{k,j}}[j\\in\\N]\\) is bounded, an application of \\cref{thm:inexact:lb} reveals that \\(\\seq{\\bar x^{k,j}}\\) too is bounded.\n\t\t\tUp to possibly enlarging the set, both sequences remain confined in the bounded set \\(\\Omega_k\\), implying that the condition at \\cref{state:PANOC+:gammaLS} should have terminated in finite time, whence the sought contradiction.\n","sha256_utf8":"08d3b6f5b228261bf1fb451dc46d9114cb1e658b0bbbeb62f3ac6d7265912f24"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"固定近端步长后的比较界","source_units":[{"start":57881,"end":58397,"start_line":1119,"end_line":1132,"raw_tex":"\t\t\tHence, \\(\\gamma_{k,j}\\) is backtracked finitely many times within iteration \\(k\\);\n\t\t\tup to discarding early attempts, we may denote \\(\\gamma_{k,j}=\\gamma_k\\).\n\t\t\tCondition \\eqref{eq:prox:better} reads\n\t\t\t\\begin{align*}\n\t\t\t\t\\LL(x^{k,j},\\bar x^{k,j},-\\nabla f(x^{k,j}))\n\t\t\t{}\\leq{} &\n\t\t\t\t\\LL(x^{k,j},\\bar x^{k-1},-\\nabla f(x^{k,j}))\n\t\t\t\\\\\n\t\t\t{}={} &\n\t\t\t\tf(x^{k,j})+g(\\bar x^{k-1})+\\innprod*{\\nabla f(x^{k,j})}{\\bar x^{k-1}-x^{k,j}}\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t{+{}}\\tfrac{1}{2\\gamma}\\|x^{k,j}-\\bar x^{k-1}\\|^2.\n\t\t\t\\end{align*}\n","sha256_utf8":"521f83ffb2c5b3caf035dcc498356d211c8d234e1036ae56bf1efac60809d01e"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"方向步长回溯有限终止","source_units":[{"start":58397,"end":59047,"start_line":1133,"end_line":1142,"raw_tex":"\t\t\tAs \\(\\tau_{k,j}\\searrow0\\), one has that \\(x^{k,j}\\to\\bar x^{k-1}\\).\n\t\t\tSince \\(f\\) and \\(\\nabla f\\) are continuous, the right-hand side of the inequality converges to \\(\\varphi(\\bar x^{k-1})\\), overall resulting in\n\t\t\t\\[\n\t\t\t\\limsup_{j\\to\\infty}\\LL(x^{k,j},\\bar x^{k,j},-\\nabla f(x^{k,j}))\n\t\t\t{}\\leq{}\n\t\t\t\\varphi(\\bar x^{k-1})\n\t\t\t{}\\overrel[\\leq]{\\eqref{eq:lphi}}{}\n\t\t\t\\Phi_{k-1}-\\tfrac{1-\\alpha}{2\\gamma_{k-1}}\\|x^{k-1}-\\bar x^{k-1}\\|^2.\n\t\t\t\\]\n\t\t\tSince \\(\\|x^{k-1}-\\bar x^{k-1}\\|>0\\) and \\(\\beta<1\\), for \\(j\\) large enough the condition at \\cref{state:PANOC+:tauLS} will be violated and therefore the \\(k\\)-th iteration successfully terminated.\n","sha256_utf8":"b94957d6a4121ef9352c4165d3a7a2032427cf8b185cdab0f19678c5635b53f3"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"组合接受条件得到充分下降","source_units":[{"start":59057,"end":59221,"start_line":1145,"end_line":1146,"raw_tex":"\t\t\\item\\ref{thm:finite:descent}\n\t\t\tFollows by combining \\eqref{eq:lphi} with the failure of the condition at \\cref{state:PANOC+:tauLS} at the end of the iteration.\n","sha256_utf8":"cfcd6f0b28fdd42fb69c6682c616ff374548b180f8dc6cbc061ae61273f7b56f"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"近端点保持在初始下水平集","source_units":[{"start":59221,"end":59349,"start_line":1147,"end_line":1151,"raw_tex":"\t\t\\item\\ref{thm:finite:sublevel}~\n\t\t\tDirect consequence of assertion \\ref{thm:finite:descent}.\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t","sha256_utf8":"b8a04999acb51caf329e3dc1a698ca4a25e63127a2e12520b8334c03a7eee79d"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"势函数单调收敛","source_units":[{"start":61509,"end":61600,"start_line":1212,"end_line":1215,"raw_tex":"\n\t\t\\begin{proofitemize}\n\t\t\\item\\ref{thm:PANOC+:asymp:cost}~\n\t\t\tFollows from \\eqref{eq:SD}.\n","sha256_utf8":"cb4888aee639df6159060b81cc3259f9ddfd51fc682e6529fdfcee9ecf7acc21"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"望远镜求和得到加权可和性","source_units":[{"start":61600,"end":61978,"start_line":1216,"end_line":1232,"raw_tex":"\t\t\\item\\ref{thm:PANOC+:asymp:summable}~\n\t\t\tA telescoping argument on \\eqref{eq:SD} yields\n\t\t\t\\begin{equation}\\label{eq:summable}\n\t\t\t\t\\beta(1-\\alpha)\n\t\t\t\t\\sum_{k\\in\\N}{\n\t\t\t\t\t\\tfrac{1}{2\\gamma_k}\n\t\t\t\t\t\\|\\bar x^k-x^k\\|^2\n\t\t\t\t}\n\t\t\t{}\\leq{}\n\t\t\t\t\\Phi_0\n\t\t\t\t{}-{}\n\t\t\t\t\\inf\\varphi\n\t\t\t{}={}\n\t\t\t\t\\def\\iter{0}\n\t\t\t\t\\FBE(x^0)-\\inf\\varphi,\n\t\t\t\\end{equation}\n\t\t\twhence the claimed finite sum.\n","sha256_utf8":"f162aef5ff45b9c508349902cde0e6f64d879b5f24e09332369b86bacf0356ef"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"近端位移与相邻迭代差消失","source_units":[{"start":61996,"end":62721,"start_line":1236,"end_line":1258,"raw_tex":"\t\t\\item\\ref{thm:PANOC+:asymp:fpr}~\n\t\t\tThat \\(\\|x^k-\\bar x^k\\|\\to0\\) follows from assertion \\ref{thm:PANOC+:asymp:summable}, since \\(\\gamma_k\\) is upper bounded.\n\t\t\tNext, by the conditions at \\cref{state:PANOC+:x+,,state:PANOC+:d}, observe that\n\t\t\t\\begin{align*}\n\t\t\t\\numberthis\\label{eq:xkdiff}\n\t\t\t\t\\|x^k-x^{k-1}\\|\n\t\t\t{}={} &\n\t\t\t\t\\bigl\\|\n\t\t\t\t\t(1-\\tau_k)(\\bar x^{k-1}-x^{k-1})\n\t\t\t\t\t{}+{}\n\t\t\t\t\t\\tau_kd^k\n\t\t\t\t\\bigr\\|\n\t\t\t{}\\leq{}\n\t\t\t\t(1+D)\\|\\bar x^{k-1}-x^{k-1}\\|\n\t\t\t\\shortintertext{and thus \\(\\|x^k-x^{k-1}\\|\\) vanishes, and in turn so does \\(\\|\\bar x^k-\\bar x^{k-1}\\|\\) since}\n\t\t\t\t\\|\\bar x^k-\\bar x^{k-1}\\|\n\t\t\t{}\\leq{} &\n\t\t\t\t\\|x^k-\\bar x^k\\|\n\t\t\t\t{}+{}\n\t\t\t\t\\|\\bar x^{k-1}-x^{k-1}\\|\n\t\t\t\t{}+{}\n\t\t\t\t\\|x^k-x^{k-1}\\|.\n\t\t\t\\end{align*}\n","sha256_utf8":"195e4b9ea3eee45a6fb93c280f7240567998e217a2a387a55d28fb138b5d0226"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"有界性与步长最终恒定","source_units":[{"start":62721,"end":63399,"start_line":1259,"end_line":1262,"raw_tex":"\t\t\\item\\ref{thm:PANOC+:asymp:gammaconstant}~\n\t\t\tThe first implication follows from \\cref{thm:finite:sublevel}, and the second one from assertion \\ref{thm:PANOC+:asymp:summable}.\n\t\t\tIf \\(\\seq{x^k}\\) is bounded, and thus so is \\(\\seq{\\bar x^k}\\), the set \\(\\Omega_k\\) in the proof of \\cref{thm:finite:LS} can be taken independent of \\(k\\), and asymptotic constancy of \\(\\gamma_k\\) follows from the same arguments therein.\n\t\t\tFinally, if \\(\\nabla f\\) is $L_f$-Lipschitz continuous the condition at \\cref{state:PANOC+:gammaLS} fails to hold as soon as $\\gamma_k\\leq\\nicefrac{\\alpha}{L_f}$ \\cite[Prop. A.24]{bertsekas1999nonlinear}, and \\(\\gamma_k\\) is thus asymptotically constant.\n","sha256_utf8":"ed7a92fb0772d13c3aa4442f4ced0fa46c476acee37a6c9e8f8b808f2c059fd6"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"反证步长总和必然发散","source_units":[{"start":63417,"end":64408,"start_line":1266,"end_line":1299,"raw_tex":"\t\t\\item\\ref{thm:PANOC+:asymp:gamma}~\n\t\t\tBy iteratively applying inequality \\eqref{eq:xkdiff}, we obtain that\n\t\t\t\\begin{align*}\n\t\t\t\t\\|x^k-x^0\\|\n\t\t\t{}\\leq{} &\n\t\t\t\t(1+D)\\sum_{j=0}^{k-1}\\|\\bar x^j-x^j\\|\n\t\t\t\\\\\n\t\t\t{}={} &\n\t\t\t\t(1+D)\\sum_{j=0}^{k-1}\\gamma_j^{-\\nicefrac12}\\|\\bar x^j-x^j\\|\\gamma_j^{\\nicefrac12}\n\t\t\t\\\\\n\t\t\t{}\\leq{} &\n\t\t\t\t\\textstyle\n\t\t\t\t(1+D)\n\t\t\t\t\\sqrt{\n\t\t\t\t\t\\sum_{j=0}^{k-1}\n\t\t\t\t\t\\gamma_j^{-1}\\|\\bar x^j-x^j\\|^2\n\t\t\t\t}\n\t\t\t\t\\sqrt{\n\t\t\t\t\t\\sum_{j=0}^{k-1}\\gamma_j\n\t\t\t\t}\n\t\t\t\\\\\n\t\t\t{}\\overrel*[\\leq]{\\eqref{eq:summable}}{} &\n\t\t\t\t\\textstyle\n\t\t\t\t(1+D)\n\t\t\t\t\\sqrt{\n\t\t\t\t\t2\n\t\t\t\t\t\\frac{\\def\\iter{0}\\FBE(x^0)-\\inf\\varphi}{\\beta(1-\\alpha)}\n\t\t\t\t}\n\t\t\t\t\\sqrt{\n\t\t\t\t\t\\sum_{j=0}^{k-1}\\gamma_j\n\t\t\t\t}.\n\t\t\t\\end{align*}\n\t\t\tContrary to the claim, if \\(\\sum_{k\\in\\N}\\gamma_k<\\infty\\) holds, then \\(\\seq{x^k}\\) is bounded.\n\t\t\tFrom assertion \\ref{thm:PANOC+:asymp:gammaconstant} proven above we then infer that \\(\\gamma_k\\) is asymptotically constant, thus contradicting the finiteness of \\(\\sum_{k\\in\\N}\\gamma_k\\).\n","sha256_utf8":"c71fa39b6ec39a3deba0ce1e2e6b2b2e9eeddb6a8c42f0d8d86017eb107e8096"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"残差下极限为零","source_units":[{"start":64426,"end":64599,"start_line":1303,"end_line":1307,"raw_tex":"\t\t\\item\\ref{thm:PANOC+:asymp:res}~\n\t\t\tImmediate consequence of assertions \\ref{thm:PANOC+:asymp:summable} and \\ref{thm:PANOC+:asymp:gamma}.\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t","sha256_utf8":"d685a55993761c31662fa1110cf52ea4750295b122dfd26444eb4095344315a7"}]},{"id":"S035","semantic_type":"external_fact","kind":"external_fact","title":"近端有界函数的二次下界","source_units":[],"declared_external":true,"declaration_text":["A prox-bounded function g admits finite constants a,b with g(z) >= -a||z||^2-b. Thus liminf g(z_j)/||z_j||^2 > -infinity whenever ||z_j|| tends to infinity; see Rockafellar–Wets, Variational Analysis, Exercise 1.24."]},{"id":"S036","semantic_type":"external_fact","kind":"external_fact","title":"局部Lipschitz梯度的下降引理","source_units":[],"declared_external":true,"declaration_text":["If ∇f is L-Lipschitz on a convex neighborhood containing x and y, then f(y) <= f(x)+<∇f(x),y-x>+(L/2)||y-x||^2. A locally Lipschitz gradient admits a uniform such constant on each fixed compact convex neighborhood. The global version is cited as Bertsekas, Nonlinear Programming, Proposition A.24."]},{"id":"S037","semantic_type":"external_fact","kind":"external_fact","title":"Young与Cauchy–Schwarz不等式","source_units":[],"declared_external":true,"declaration_text":["Young's inequality gives 2|<u,v>| <= ||u||^2+||v||^2. Cauchy–Schwarz gives sum_j a_j b_j <= sqrt(sum_j a_j^2) sqrt(sum_j b_j^2) for finite real sequences."]},{"id":"S038","semantic_type":"external_fact","kind":"external_fact","title":"单调序列与加权级数事实","source_units":[],"declared_external":true,"declaration_text":["A nonincreasing real sequence bounded below converges, and summable nonnegative terms tend to zero. If a_k >= 0, gamma_k > 0, sum_k a_k^2/gamma_k < infinity, and sum_k gamma_k = infinity, then liminf_k a_k/gamma_k = 0. Cluster sets are closed, and sequences whose pointwise difference tends to zero have the same cluster set."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S003","target_id":"S011","kind":"supports"},{"source_id":"S004","target_id":"S017","kind":"supports"},{"source_id":"S004","target_id":"S021","kind":"supports"},{"source_id":"S004","target_id":"S024","kind":"supports"},{"source_id":"S004","target_id":"S026","kind":"supports"},{"source_id":"S004","target_id":"S029","kind":"supports"},{"source_id":"S004","target_id":"S032","kind":"supports"},{"source_id":"S005","target_id":"S028","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S008","kind":"supports"},{"source_id":"S006","target_id":"S017","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S030","kind":"supports"},{"source_id":"S009","target_id":"S020","kind":"supports"},{"source_id":"S009","target_id":"S021","kind":"supports"},{"source_id":"S009","target_id":"S023","kind":"supports"},{"source_id":"S009","target_id":"S024","kind":"supports"},{"source_id":"S009","target_id":"S026","kind":"supports"},{"source_id":"S009","target_id":"S027","kind":"supports"},{"source_id":"S009","target_id":"S031","kind":"supports"},{"source_id":"S009","target_id":"S032","kind":"supports"},{"source_id":"S010","target_id":"S020","kind":"supports"},{"source_id":"S011","target_id":"S024","kind":"supports"},{"source_id":"S011","target_id":"S025","kind":"supports"},{"source_id":"S012","target_id":"S021","kind":"supports"},{"source_id":"S012","target_id":"S022","kind":"supports"},{"source_id":"S012","target_id":"S024","kind":"supports"},{"source_id":"S012","target_id":"S025","kind":"supports"},{"source_id":"S013","target_id":"S026","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S021","kind":"supports"},{"source_id":"S014","target_id":"S024","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S022","kind":"supports"},{"source_id":"S020","target_id":"S023","kind":"supports"},{"source_id":"S020","target_id":"S024","kind":"supports"},{"source_id":"S020","target_id":"S025","kind":"supports"},{"source_id":"S020","target_id":"S026","kind":"supports"},{"source_id":"S022","target_id":"S026","kind":"supports"},{"source_id":"S022","target_id":"S027","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S026","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S019","kind":"supports"},{"source_id":"S022","target_id":"S019","kind":"supports"},{"source_id":"S023","target_id":"S019","kind":"supports"},{"source_id":"S024","target_id":"S019","kind":"supports"},{"source_id":"S025","target_id":"S019","kind":"supports"},{"source_id":"S026","target_id":"S019","kind":"supports"},{"source_id":"S027","target_id":"S019","kind":"supports"},{"source_id":"S028","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S029","kind":"supports"},{"source_id":"S018","target_id":"S030","kind":"supports"},{"source_id":"S018","target_id":"S032","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S030","target_id":"S033","kind":"supports"},{"source_id":"S030","target_id":"S034","kind":"supports"},{"source_id":"S031","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S029","target_id":"S002","kind":"supports"},{"source_id":"S030","target_id":"S002","kind":"supports"},{"source_id":"S031","target_id":"S002","kind":"supports"},{"source_id":"S032","target_id":"S002","kind":"supports"},{"source_id":"S033","target_id":"S002","kind":"supports"},{"source_id":"S034","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S035","target_id":"S017","kind":"supports"},{"source_id":"S036","target_id":"S021","kind":"supports"},{"source_id":"S036","target_id":"S024","kind":"supports"},{"source_id":"S036","target_id":"S032","kind":"supports"},{"source_id":"S037","target_id":"S016","kind":"supports"},{"source_id":"S037","target_id":"S033","kind":"supports"},{"source_id":"S038","target_id":"S029","kind":"supports"},{"source_id":"S038","target_id":"S031","kind":"supports"},{"source_id":"S038","target_id":"S034","kind":"supports"}]}}
{"record_id":"2202.07581v2","paper":{"title":"Bayesian Stochastic Gradient Descent for Stochastic Optimization with Streaming Input Data","authors":["Tianyi Liu","Yifan Lin","Enlu Zhou"],"arxiv_id":"2202.07581","version":"v2","journal":"SIAM Journal on Optimization","doi":"10.1137/22m1478951","abstract_url":"https://arxiv.org/abs/2202.07581v2","pdf_url":"https://arxiv.org/pdf/2202.07581v2"},"context":{"raw_tex":"\\begin{align}\\label{eq: obj-independent}\n    \\min _{x \\in \\mathcal{X}} H(x):=\\mathbb{E}_{f(\\cdot;\\theta^c)}[h(x, \\xi)] ~~~\\text{(decision-independent uncertainty)}\n\\end{align}\n\nwhere $x \\in \\mathcal{X} \\subset \\mathbb{R}^d$ is the decision vector, $\\xi \\in \\Xi \\subset \\mathbb{R}^m$ is a random vector, $h: \\mathbb{R}^d \\times \\mathbb{R}^m \\to \\mathbb{R}$ is a deterministic function. The expectation is taken with respect to (w.r.t.) the distribution of $\\xi$, which is denoted as $f(\\cdot;\\theta^c)$ in the decision-independent case, and as $f(\\cdot;x,\\theta^c)$ in the decision-dependent case. The density function $f(\\cdot;x,\\theta^c)$ takes a general form, where the parameter $\\theta^c$ does not depend on $x$. For example, $f(\\xi;x,\\theta^c)=\\theta^c x \\exp (-\\theta^c x \\xi)$ is the density function of the exponential distribution with rate $\\theta^c x$. More assumptions on the density function will be discussed in \\cref{sec: convergence}. We assume the distribution of $\\xi$ belongs to a parameterized family of distributions with parameter set $\\Theta\\subset \\mathbb{R}^l$, and let $\\theta^c$ be the true parameter value of the distribution.\n\n\nWe now discuss the details of these two steps in the following. Let's first focus on the decision-independent case. Suppose at each time stage $t$ we observe a batch of data $\\mathbf{y}_t = \\{y_{t,j}, j=1,\\ldots,D\\}$, where $\\{y_{t,j}\\}$ are i.i.d. according to $f(\\cdot;\\theta^c)$ and $D$ is the batch size. By viewing the unknown distributional parameter as a random vector $\\theta$ and assuming a prior distribution $\\pi_0$ on $\\theta$, the posterior distribution of $\\theta$ is updated by the Bayes rule as follows:\n\\begin{align}\\label{eq:independent-posterior}\n    \\pi_t(\\theta) = \\frac{\\pi_{t-1}(\\theta)f(\\mathbf{y}_t;\\theta)}{\\int\\pi_{t-1}(\\theta)f(\\mathbf{y}_t;\\theta)d\\theta} =  \\frac{\\pi_{t-1}(\\theta) \\prod_{j=1}^{D} f(y_{t,j};\\theta)}{\\int\\pi_{t-1}(\\theta) \\prod_{j=1}^{D} f(y_{t,j};\\theta)d\\theta}.\n\\end{align}\n\nThe objective function \\eqref{eq: obj-independent} can be viewed as a function of $\\theta$, so we define the following function\n$$\nH(x,\\theta):= \\mathbb{E}_{f(\\cdot;\\theta)}[h(x, \\xi)]. \n$$\n\nTo estimate the true objective function \\eqref{eq: obj-independent}, we consider the Bayesian average of the objective function:\n\\begin{align}\n    \\min_{x \\in \\mathcal{X}} \\mathbb{E}_{\\pi_{t}}\\left[H(x, \\theta)\\right],\n\\label{eq: independent}\n\\end{align}\n\n\\begin{algorithm}\n\\caption{Bayesian-SGD (decision-independent uncertainty)}\n\\label{algorithm: independent}\n\\begin{algorithmic}\n\\STATE{\\textbf{input}: data batch size $D$, number of SGD iterations $K$, step size sequence $\\{a_{t,j}, t=1,2,\\ldots; j=0,\\ldots,K-1\\}$, time horizon $T$.}\n\\STATE{\\textbf{initialization}: choose an initial decision $x_1$ and prior distribution $\\pi_0(\\theta)$.}\n\\FOR{$t =1: T$}\n    \\STATE{-A batch of data $y_{t,1},\\cdots,y_{t,D} \\iid f(\\cdot; \\theta^c)$ arrives;}\n    \\STATE{-Posterior Update: compute $\\pi_t(\\theta)$ according to \\eqref{eq:independent-posterior}. }\n    \\STATE{-Decision Update:\n        \\begin{itemize}\n            \\item set $x_{t,0}:=x_{t}$;\n            \\item for $j=0,\\cdots,K-1$, draw sample $\\theta_{t,j} \\sim \\pi_t(\\theta)$ and $\\xi_{t,j} \\sim f(\\cdot;\\theta_{t,j})$, and carry out SGD iteration:\n        \\end{itemize}\n        \\begin{align}\\label{eq: SGD_independent}\n            x_{t,j+1}:=\\operatorname{Proj}_{\\mathcal{X}}\\left\\{x_{t,j}-a_{t,j}\\nabla_x h(x_{t,j},\\xi_{t,j})\\right\\},\n        \\end{align}\n        ~~~~~~~~~~~~~~where $\\operatorname{Proj}_{\\mathcal{X}}$ is a projection operator that projects the iterate  to the  set $\\mathcal{X}$. \n        \\begin{itemize}\n            \\item set the updated decision as $x_{t+1}:=x_{t,K}$;\n        \\end{itemize}\n    }\n\\ENDFOR\n\\RETURN $x_{T+1}$\n\\end{algorithmic}\n\\end{algorithm}\n\nLet's first consider the decision-independent case. The probability space is constructed as follows. Define the Bayesian prior $\\pi_0$ on $(\\Theta, \\mathcal{B}_{\\Theta})$, where $\\mathcal{B}_{\\Theta}$ is the Borel $\\sigma$-algebra on $\\Theta$. Let $\\mathcal{Y} \\subset \\mathbb{R}^{m}$ denote the data (observation) space. The data $y$ takes value in $\\mathcal{Y}$ equipped with a Borel $\\sigma$-algebra $\\mathcal{B}_{\\mathcal{Y}}$ and a probability measure $\\left\\{\\mathbb{P}_{\\theta^c}\\right\\}$, such that $\\mathbb{P}_{\\theta^c}(y \\in A) = \\int_{A} f\\left(y; \\theta^c\\right) d y, \\forall A \\in \\mathcal{B}(\\mathcal{Y})$. For the sequence $y_1,y_2,\\cdots,y_n \\iid f(\\cdot;\\theta^c)$, the probability measure is denoted by $\\mathbb{P}_{\\theta^c}^{n}$. As for the infinite sequence $\\{y_1,y_2,\\ldots\\}$, the probability measure $\\mathbb{P}_{\\theta^c}^{\\infty}$ can be constructed by Kolmogorov's extension theorem (cf. Theorem A.3.1 in \\cite{durrett2019probability}). In the following, w.p.1 (or almost surely) means that the considered property holds with probability one w.r.t. the probability measure $\\mathbb{P}_{\\theta^c}^\\infty$. Finally, let $\\mathcal{F}_{t}:=\\sigma\\left\\{\\left(y_{\\tau} \\right), \\tau \\leq t\\right\\}$ be the $\\sigma$-filtration generated by the data. We have the convergence of the posterior distribution $\\{\\pi_t\\}$ that is updated according to \\cref{eq:independent-posterior} under the following assumptions.\n\n\n\\begin{assumption}[\\cite{shapiro2021bayesian}, Assumption 3.1] (i) The set $\\Theta$ is convex and compact with non-empty interior. (ii) $\\ln \\pi_0(\\theta)$ is bounded on $\\Theta$. (iii) $f(\\xi|\\theta) > 0$ for all $\\xi \\in \\Xi$ and $\\theta \\in \\Theta$. (iv) $f(\\xi|\\theta)$ is continuous in $\\theta \\in \\Theta$. (v) $\\ln f(\\xi|\\theta), \\theta \\in \\Theta$ is dominated by an integrable (w.r.t. $\\xi \\sim f(\\cdot;\\theta^c)$) function. (vi) The data batches are i.i.d. over time from the fixed distribution $f(\\cdot;\\theta^c)$. \n\\label{ass: Alex2022}\n\\end{assumption}\n\nWe then study the asymptotic behavior of \\cref{algorithm: independent} by the ordinary differential equation (ODE) method (please refer to \\cite{Kushner2003} for a detailed exposition on the ODE method for stochastic approximation). The main idea is that SGD can be viewed as a noisy discretization of an ODE. Under certain conditions, the noise in SGD averages out asymptotically, such that the SGD iterates converge to the solution trajectory of the ODE. For simplicity, we consider the case where $K=1$ and rewrite the SGD iteration \\cref{eq: SGD_independent} as\n\\begin{equation}\\label{eq: SGD_independent_2}\n    x_{t+1}=x_{t}-a_{t} \\nabla_{x} h\\left(x_{t}, \\xi_{t}\\right) + a_t z_t,\n\\end{equation}\n\nwhere $a_t z_t$ is the projection term, i.e., the vector of shortest Euclidean length needed to keep the decision $x_{t+1}$ from leaving the decision space $\\mathcal{X}$. We first show that under certain mild conditions, the proposed gradient estimator in \\cref{eq: SGD_independent_2} is unbiased. \n\n\n\\begin{assumption}\\label{ass: independent_DCT}\n$h(x,\\xi)$ is $C^{1}$-smooth in $x$ for all $\\xi \\in \\Xi$, and the map $\\xi \\to \\nabla_x h(x,\\xi)$ is $L_h$-Lipschitz continuous for any $x \\in \\mathcal{X}$. \n\\end{assumption}\n\n\\begin{assumption}\\label{ass: independent_step}\n~\n\\begin{itemize}\n    \\item The step size $\\{a_t\\}$ satisfies $\\sum_{t=1}^{\\infty} a_{t}^{2}<\\infty$, $\\sum_{t=1}^{\\infty} a_{t}=\\infty$, $\\lim_{t \\to \\infty} a_{t} = 0$, $a_t > 0, \\forall t > 0$.\n    \\item The decision space $\\mathcal{X} \\subset \\mathbb{R}^{d}$ is compact and convex.\n\\end{itemize}\n\\end{assumption}\n\nNext, we investigate the convergence rate of \\cref{algorithm: independent} for the unconstrained case, i.e., without the projection term $a_t z_t$ under the following additional assumptions.\n\n\n\\begin{assumption}\\label{ass: independent_rate}\n~\n\\begin{itemize}\n    \\item The parameter space $\\Theta$ is finite, i.e., $\\Theta = \\{\\theta_1,\\ldots,\\theta_k\\}$. Moreover, $\\theta^c \\in \\Theta$.\n    \\item There exists $0<L_{H}<\\infty$ such that $||\\nabla_x H(x,\\theta_1)-\\nabla_x H(x,\\theta_2)||_2 \\leq L_{H}||\\theta_1-\\theta_2||_2$ for all $\\theta_1,\\theta_2 \\in \\Theta$ and for all $x \\in \\mathcal{X}$.\n    \\item Sampling variance is bounded by $\\sigma^2$, i.e., $\\mathbb{E}[||\\nabla_x h(x,\\xi) - \\nabla_x H(x,\\theta)||_2^2|\\theta] \\leq \\sigma^2$, for all $\\theta \\in \\Theta$.\n\\end{itemize}\n\\end{assumption}\n\nThe proof of \\cref{lemma: independent_concentration} can be found in \\cref{appendix: A}. Next, we show the convergence rate of \\cref{algorithm: independent}. To simplify the analysis and also be consistent with the convergence analysis of smooth non-convex SGD, we consider a variant of SGD where the final output is randomly chosen as follows: let $z_T = x_{t}$ with probability $\\frac{a_t}{\\sum_{t=1}^{T}a_t}$, $t=1,\\cdots,T$. The randomization scheme helps with the analysis of the expected gradient of the final output under the true parameter $\\theta^c$, and has been widely used in the smooth non-convex SGD literature (cf. \\cite{ghadimi2013stochastic}). We then have the following theorem giving the convergence rate of the randomized output algorithm under different step sizes.\n","units":[{"start":18865,"end":19040,"start_line":133,"end_line":135,"raw_tex":"\\begin{align}\\label{eq: obj-independent}\n    \\min _{x \\in \\mathcal{X}} H(x):=\\mathbb{E}_{f(\\cdot;\\theta^c)}[h(x, \\xi)] ~~~\\text{(decision-independent uncertainty)}\n\\end{align}","sha256_utf8":"f393888613ac4c7c6c1a171817bd18c12dc50a43e7f071a2e510e6e8c947619f"},{"start":19216,"end":20194,"start_line":139,"end_line":139,"raw_tex":"where $x \\in \\mathcal{X} \\subset \\mathbb{R}^d$ is the decision vector, $\\xi \\in \\Xi \\subset \\mathbb{R}^m$ is a random vector, $h: \\mathbb{R}^d \\times \\mathbb{R}^m \\to \\mathbb{R}$ is a deterministic function. The expectation is taken with respect to (w.r.t.) the distribution of $\\xi$, which is denoted as $f(\\cdot;\\theta^c)$ in the decision-independent case, and as $f(\\cdot;x,\\theta^c)$ in the decision-dependent case. The density function $f(\\cdot;x,\\theta^c)$ takes a general form, where the parameter $\\theta^c$ does not depend on $x$. For example, $f(\\xi;x,\\theta^c)=\\theta^c x \\exp (-\\theta^c x \\xi)$ is the density function of the exponential distribution with rate $\\theta^c x$. More assumptions on the density function will be discussed in \\cref{sec: convergence}. We assume the distribution of $\\xi$ belongs to a parameterized family of distributions with parameter set $\\Theta\\subset \\mathbb{R}^l$, and let $\\theta^c$ be the true parameter value of the distribution.\n","sha256_utf8":"bc6363f5497884322584df1e1ab2342457b337f8e6b9f6044d2452b4a832fd87"},{"start":21431,"end":22254,"start_line":148,"end_line":151,"raw_tex":"We now discuss the details of these two steps in the following. Let's first focus on the decision-independent case. Suppose at each time stage $t$ we observe a batch of data $\\mathbf{y}_t = \\{y_{t,j}, j=1,\\ldots,D\\}$, where $\\{y_{t,j}\\}$ are i.i.d. according to $f(\\cdot;\\theta^c)$ and $D$ is the batch size. By viewing the unknown distributional parameter as a random vector $\\theta$ and assuming a prior distribution $\\pi_0$ on $\\theta$, the posterior distribution of $\\theta$ is updated by the Bayes rule as follows:\n\\begin{align}\\label{eq:independent-posterior}\n    \\pi_t(\\theta) = \\frac{\\pi_{t-1}(\\theta)f(\\mathbf{y}_t;\\theta)}{\\int\\pi_{t-1}(\\theta)f(\\mathbf{y}_t;\\theta)d\\theta} =  \\frac{\\pi_{t-1}(\\theta) \\prod_{j=1}^{D} f(y_{t,j};\\theta)}{\\int\\pi_{t-1}(\\theta) \\prod_{j=1}^{D} f(y_{t,j};\\theta)d\\theta}.\n\\end{align}","sha256_utf8":"b2a586a467d71f70852dbc568be8f1b8c770e04e07982b975eced97a80d2da5b"},{"start":22255,"end":22444,"start_line":152,"end_line":155,"raw_tex":"The objective function \\eqref{eq: obj-independent} can be viewed as a function of $\\theta$, so we define the following function\n$$\nH(x,\\theta):= \\mathbb{E}_{f(\\cdot;\\theta)}[h(x, \\xi)]. \n$$","sha256_utf8":"eed4acb283dcb3c58e519be571fa99ced4932b992882bfb38c8e1becb75163de"},{"start":22445,"end":22699,"start_line":156,"end_line":160,"raw_tex":"To estimate the true objective function \\eqref{eq: obj-independent}, we consider the Bayesian average of the objective function:\n\\begin{align}\n    \\min_{x \\in \\mathcal{X}} \\mathbb{E}_{\\pi_{t}}\\left[H(x, \\theta)\\right],\n\\label{eq: independent}\n\\end{align}","sha256_utf8":"66ce8605f2f7470e417895359802b3cef713d1aaefdd73d7f9faa9042a80d270"},{"start":26938,"end":28315,"start_line":187,"end_line":212,"raw_tex":"\\begin{algorithm}\n\\caption{Bayesian-SGD (decision-independent uncertainty)}\n\\label{algorithm: independent}\n\\begin{algorithmic}\n\\STATE{\\textbf{input}: data batch size $D$, number of SGD iterations $K$, step size sequence $\\{a_{t,j}, t=1,2,\\ldots; j=0,\\ldots,K-1\\}$, time horizon $T$.}\n\\STATE{\\textbf{initialization}: choose an initial decision $x_1$ and prior distribution $\\pi_0(\\theta)$.}\n\\FOR{$t =1: T$}\n    \\STATE{-A batch of data $y_{t,1},\\cdots,y_{t,D} \\iid f(\\cdot; \\theta^c)$ arrives;}\n    \\STATE{-Posterior Update: compute $\\pi_t(\\theta)$ according to \\eqref{eq:independent-posterior}. }\n    \\STATE{-Decision Update:\n        \\begin{itemize}\n            \\item set $x_{t,0}:=x_{t}$;\n            \\item for $j=0,\\cdots,K-1$, draw sample $\\theta_{t,j} \\sim \\pi_t(\\theta)$ and $\\xi_{t,j} \\sim f(\\cdot;\\theta_{t,j})$, and carry out SGD iteration:\n        \\end{itemize}\n        \\begin{align}\\label{eq: SGD_independent}\n            x_{t,j+1}:=\\operatorname{Proj}_{\\mathcal{X}}\\left\\{x_{t,j}-a_{t,j}\\nabla_x h(x_{t,j},\\xi_{t,j})\\right\\},\n        \\end{align}\n        ~~~~~~~~~~~~~~where $\\operatorname{Proj}_{\\mathcal{X}}$ is a projection operator that projects the iterate  to the  set $\\mathcal{X}$. \n        \\begin{itemize}\n            \\item set the updated decision as $x_{t+1}:=x_{t,K}$;\n        \\end{itemize}\n    }\n\\ENDFOR\n\\RETURN $x_{T+1}$\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"d9f2a5896b213f5ac495ff6023e022639b03ff30cbe468e3578eb57235e3f2d6"},{"start":30476,"end":31909,"start_line":248,"end_line":248,"raw_tex":"Let's first consider the decision-independent case. The probability space is constructed as follows. Define the Bayesian prior $\\pi_0$ on $(\\Theta, \\mathcal{B}_{\\Theta})$, where $\\mathcal{B}_{\\Theta}$ is the Borel $\\sigma$-algebra on $\\Theta$. Let $\\mathcal{Y} \\subset \\mathbb{R}^{m}$ denote the data (observation) space. The data $y$ takes value in $\\mathcal{Y}$ equipped with a Borel $\\sigma$-algebra $\\mathcal{B}_{\\mathcal{Y}}$ and a probability measure $\\left\\{\\mathbb{P}_{\\theta^c}\\right\\}$, such that $\\mathbb{P}_{\\theta^c}(y \\in A) = \\int_{A} f\\left(y; \\theta^c\\right) d y, \\forall A \\in \\mathcal{B}(\\mathcal{Y})$. For the sequence $y_1,y_2,\\cdots,y_n \\iid f(\\cdot;\\theta^c)$, the probability measure is denoted by $\\mathbb{P}_{\\theta^c}^{n}$. As for the infinite sequence $\\{y_1,y_2,\\ldots\\}$, the probability measure $\\mathbb{P}_{\\theta^c}^{\\infty}$ can be constructed by Kolmogorov's extension theorem (cf. Theorem A.3.1 in \\cite{durrett2019probability}). In the following, w.p.1 (or almost surely) means that the considered property holds with probability one w.r.t. the probability measure $\\mathbb{P}_{\\theta^c}^\\infty$. Finally, let $\\mathcal{F}_{t}:=\\sigma\\left\\{\\left(y_{\\tau} \\right), \\tau \\leq t\\right\\}$ be the $\\sigma$-filtration generated by the data. We have the convergence of the posterior distribution $\\{\\pi_t\\}$ that is updated according to \\cref{eq:independent-posterior} under the following assumptions.\n","sha256_utf8":"a923ba99c9a3abbf8873d64ff060e17d344ede27598bf5eccc09c4ebb107919f"},{"start":31910,"end":32474,"start_line":250,"end_line":252,"raw_tex":"\\begin{assumption}[\\cite{shapiro2021bayesian}, Assumption 3.1] (i) The set $\\Theta$ is convex and compact with non-empty interior. (ii) $\\ln \\pi_0(\\theta)$ is bounded on $\\Theta$. (iii) $f(\\xi|\\theta) > 0$ for all $\\xi \\in \\Xi$ and $\\theta \\in \\Theta$. (iv) $f(\\xi|\\theta)$ is continuous in $\\theta \\in \\Theta$. (v) $\\ln f(\\xi|\\theta), \\theta \\in \\Theta$ is dominated by an integrable (w.r.t. $\\xi \\sim f(\\cdot;\\theta^c)$) function. (vi) The data batches are i.i.d. over time from the fixed distribution $f(\\cdot;\\theta^c)$. \n\\label{ass: Alex2022}\n\\end{assumption}","sha256_utf8":"1e6af3fb0b10b54f3bfe7e0e904d5eb256af45269a5371cbb1486f727893cfa2"},{"start":33249,"end":33950,"start_line":265,"end_line":268,"raw_tex":"We then study the asymptotic behavior of \\cref{algorithm: independent} by the ordinary differential equation (ODE) method (please refer to \\cite{Kushner2003} for a detailed exposition on the ODE method for stochastic approximation). The main idea is that SGD can be viewed as a noisy discretization of an ODE. Under certain conditions, the noise in SGD averages out asymptotically, such that the SGD iterates converge to the solution trajectory of the ODE. For simplicity, we consider the case where $K=1$ and rewrite the SGD iteration \\cref{eq: SGD_independent} as\n\\begin{equation}\\label{eq: SGD_independent_2}\n    x_{t+1}=x_{t}-a_{t} \\nabla_{x} h\\left(x_{t}, \\xi_{t}\\right) + a_t z_t,\n\\end{equation}","sha256_utf8":"47c5325daa4cc0fb9ae7b1bdb6d53811ec657b8c2afdf1930ea5918336685490"},{"start":33951,"end":34250,"start_line":269,"end_line":269,"raw_tex":"where $a_t z_t$ is the projection term, i.e., the vector of shortest Euclidean length needed to keep the decision $x_{t+1}$ from leaving the decision space $\\mathcal{X}$. We first show that under certain mild conditions, the proposed gradient estimator in \\cref{eq: SGD_independent_2} is unbiased. \n","sha256_utf8":"0eee459d57278856aff91263b1e32a826c1a756a5c51d0c6713e0fea6161b58c"},{"start":34251,"end":34473,"start_line":271,"end_line":273,"raw_tex":"\\begin{assumption}\\label{ass: independent_DCT}\n$h(x,\\xi)$ is $C^{1}$-smooth in $x$ for all $\\xi \\in \\Xi$, and the map $\\xi \\to \\nabla_x h(x,\\xi)$ is $L_h$-Lipschitz continuous for any $x \\in \\mathcal{X}$. \n\\end{assumption}","sha256_utf8":"c272fb2f2ee43eca9e341cb067a5c3dbc4bf4b203cb00169c7cb5924ff64f0dc"},{"start":35867,"end":36231,"start_line":290,"end_line":296,"raw_tex":"\\begin{assumption}\\label{ass: independent_step}\n~\n\\begin{itemize}\n    \\item The step size $\\{a_t\\}$ satisfies $\\sum_{t=1}^{\\infty} a_{t}^{2}<\\infty$, $\\sum_{t=1}^{\\infty} a_{t}=\\infty$, $\\lim_{t \\to \\infty} a_{t} = 0$, $a_t > 0, \\forall t > 0$.\n    \\item The decision space $\\mathcal{X} \\subset \\mathbb{R}^{d}$ is compact and convex.\n\\end{itemize}\n\\end{assumption}","sha256_utf8":"2336a68eb91a04989b5b40d3ddac6d4f2b26c9641857769e3a0d6a002410df6d"},{"start":41040,"end":41231,"start_line":328,"end_line":328,"raw_tex":"Next, we investigate the convergence rate of \\cref{algorithm: independent} for the unconstrained case, i.e., without the projection term $a_t z_t$ under the following additional assumptions.\n","sha256_utf8":"49adbab7936fcd526b0041280554e96f8c8088bceed7296e5e9ccc0255c0cc4f"},{"start":41232,"end":41842,"start_line":330,"end_line":337,"raw_tex":"\\begin{assumption}\\label{ass: independent_rate}\n~\n\\begin{itemize}\n    \\item The parameter space $\\Theta$ is finite, i.e., $\\Theta = \\{\\theta_1,\\ldots,\\theta_k\\}$. Moreover, $\\theta^c \\in \\Theta$.\n    \\item There exists $0<L_{H}<\\infty$ such that $||\\nabla_x H(x,\\theta_1)-\\nabla_x H(x,\\theta_2)||_2 \\leq L_{H}||\\theta_1-\\theta_2||_2$ for all $\\theta_1,\\theta_2 \\in \\Theta$ and for all $x \\in \\mathcal{X}$.\n    \\item Sampling variance is bounded by $\\sigma^2$, i.e., $\\mathbb{E}[||\\nabla_x h(x,\\xi) - \\nabla_x H(x,\\theta)||_2^2|\\theta] \\leq \\sigma^2$, for all $\\theta \\in \\Theta$.\n\\end{itemize}\n\\end{assumption}","sha256_utf8":"e363e793d60671f486b1f7d5e3ade900f3505275daf87b270cfb8f901448eb23"},{"start":43222,"end":44009,"start_line":350,"end_line":350,"raw_tex":"The proof of \\cref{lemma: independent_concentration} can be found in \\cref{appendix: A}. Next, we show the convergence rate of \\cref{algorithm: independent}. To simplify the analysis and also be consistent with the convergence analysis of smooth non-convex SGD, we consider a variant of SGD where the final output is randomly chosen as follows: let $z_T = x_{t}$ with probability $\\frac{a_t}{\\sum_{t=1}^{T}a_t}$, $t=1,\\cdots,T$. The randomization scheme helps with the analysis of the expected gradient of the final output under the true parameter $\\theta^c$, and has been widely used in the smooth non-convex SGD literature (cf. \\cite{ghadimi2013stochastic}). We then have the following theorem giving the convergence rate of the randomized output algorithm under different step sizes.\n","sha256_utf8":"db401b7c42a3c6d0adefc1a1073dc582eeb48a7f380b85ccccf63c4bb7960ad4"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{thm: independent_rate}\nUnder \\cref{ass: Alex2022}, \\cref{ass: independent_DCT}, \\cref{ass: independent_step}, and \\cref{ass: independent_rate}, for any $\\delta>0$, we have with probability at least $1-\\delta,$ for any $T>0$, the following bound on the expected gradient of the final output under the true parameter $\\theta^c$  \n\\begin{enumerate}\n    \\item[(i)] If the step size satisfies $a_t=\\frac{a}{\\sqrt{T}}$, $\\forall t \\leq T$, for some constant $a<\\frac{\\sqrt{T}}{L_h}$, then \n    {\\small\n    \\begin{align*}\n        & \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2] \\\\\n\t    &\\leq \\left[\\frac{2(H(x_1,\\theta^c)- \\min_{x\\in\\mathcal{X}}H(x,\\theta^c))}{a\\sqrt{T}}\\right] + \\left[\\frac{A_1}{T} + \\frac{A_2 \\log T}{T} + \\frac{A_3 \\log^2 T}{T}\\right] + \\frac{L_h a\\sigma^2}{\\sqrt{T}},\n    \\end{align*}\n    }\n    where $A_1=\\frac{C_1 (\\log D - \\log \\delta)}{L_h D}$, $A_2=\\frac{C_1 (\\log D - \\log \\delta)}{L_h D} + \\frac{C_1}{L_h D}$, $A_3=\\frac{C_1}{L_h D}$.\n    \\item[(ii)]  If the step size satisfies $a_t = \\frac{a}{t}$, $\\forall t \\leq T$, for some constant $a < \\frac{1}{L_h}$, then\n    {\\small\n    \\begin{align*}\n\t& \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2]\\\\\n\t&\\leq \\left[\\frac{2(H(x_1,\\theta^c)- \\min_{x\\in\\mathcal{X}}H(x,\\theta^c))}{a} +\\frac{6C_1 + \\pi^2C_1 (\\log D - \\log \\delta)}{6D}+\\frac{\\pi^2 L_h a \\sigma^2}{6}\\right]\\frac{1}{\\log T}.\n    \\end{align*}\n    }\n    \\item[(iii)]  If the step size satisfies $a_t = \\frac{a}{\\sqrt{t}}$, $\\forall t \\leq T$, for some constant $a < \\frac{1}{L_h}$, then\n    {\\small\n    \\begin{align*}\n\t& \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2]\\\\\n\t&\\leq [\\frac{2(H(x_1,\\theta^c)- \\min_{x\\in\\mathcal{X}}H(x,\\theta^c))}{a\\sqrt{T}} + \\frac{3C_1(\\log D - \\log \\delta) + 4C_1}{D\\sqrt{T}} + \\frac{L_h a \\sigma^2}{\\sqrt{T}}] + \\frac{L_h a \\sigma^2 \\log T}{\\sqrt{T}}.\n    \\end{align*}\n    }\n\\end{enumerate}\n\\end{theorem}","units":[{"start":44010,"end":45892,"start_line":352,"end_line":378,"raw_tex":"\\begin{theorem}\\label{thm: independent_rate}\nUnder \\cref{ass: Alex2022}, \\cref{ass: independent_DCT}, \\cref{ass: independent_step}, and \\cref{ass: independent_rate}, for any $\\delta>0$, we have with probability at least $1-\\delta,$ for any $T>0$, the following bound on the expected gradient of the final output under the true parameter $\\theta^c$  \n\\begin{enumerate}\n    \\item[(i)] If the step size satisfies $a_t=\\frac{a}{\\sqrt{T}}$, $\\forall t \\leq T$, for some constant $a<\\frac{\\sqrt{T}}{L_h}$, then \n    {\\small\n    \\begin{align*}\n        & \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2] \\\\\n\t    &\\leq \\left[\\frac{2(H(x_1,\\theta^c)- \\min_{x\\in\\mathcal{X}}H(x,\\theta^c))}{a\\sqrt{T}}\\right] + \\left[\\frac{A_1}{T} + \\frac{A_2 \\log T}{T} + \\frac{A_3 \\log^2 T}{T}\\right] + \\frac{L_h a\\sigma^2}{\\sqrt{T}},\n    \\end{align*}\n    }\n    where $A_1=\\frac{C_1 (\\log D - \\log \\delta)}{L_h D}$, $A_2=\\frac{C_1 (\\log D - \\log \\delta)}{L_h D} + \\frac{C_1}{L_h D}$, $A_3=\\frac{C_1}{L_h D}$.\n    \\item[(ii)]  If the step size satisfies $a_t = \\frac{a}{t}$, $\\forall t \\leq T$, for some constant $a < \\frac{1}{L_h}$, then\n    {\\small\n    \\begin{align*}\n\t& \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2]\\\\\n\t&\\leq \\left[\\frac{2(H(x_1,\\theta^c)- \\min_{x\\in\\mathcal{X}}H(x,\\theta^c))}{a} +\\frac{6C_1 + \\pi^2C_1 (\\log D - \\log \\delta)}{6D}+\\frac{\\pi^2 L_h a \\sigma^2}{6}\\right]\\frac{1}{\\log T}.\n    \\end{align*}\n    }\n    \\item[(iii)]  If the step size satisfies $a_t = \\frac{a}{\\sqrt{t}}$, $\\forall t \\leq T$, for some constant $a < \\frac{1}{L_h}$, then\n    {\\small\n    \\begin{align*}\n\t& \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2]\\\\\n\t&\\leq [\\frac{2(H(x_1,\\theta^c)- \\min_{x\\in\\mathcal{X}}H(x,\\theta^c))}{a\\sqrt{T}} + \\frac{3C_1(\\log D - \\log \\delta) + 4C_1}{D\\sqrt{T}} + \\frac{L_h a \\sigma^2}{\\sqrt{T}}] + \\frac{L_h a \\sigma^2 \\log T}{\\sqrt{T}}.\n    \\end{align*}\n    }\n\\end{enumerate}\n\\end{theorem}","sha256_utf8":"2a91377f60ab9ccf6afb03a2b3c62a3a2973626b4861bcd5c5c3e049aacec410"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\\label{lemma: independent_unbiasedness}\nUnder \\cref{ass: independent_DCT}, $\\nabla_{x} h(x, \\xi)$ with $\\xi \\sim f(\\cdot;\\theta)$ and $\\theta \\sim \\pi_t$ is an unbiased gradient estimator of the objective function in \\cref{eq: independent}.\n\\end{lemma}\n\n\\begin{proof}\nFor every fixed $x\\in\\mathcal{X}$, \n\\begin{align*}\n    \\mathbb{E}_{\\pi_{t}}\\left[\\mathbb{E}_{f(\\cdot;\\theta)} [\\nabla_x h(x, \\xi)]\\right] & = \\mathbb{E}_{\\pi_{t}}\\left[\\nabla_{x} \\mathbb{E}_{f(\\cdot;\\theta)} [h(x, \\xi)]\\right]\\\\\n    & = \\nabla_{x} \\mathbb{E}_{\\pi_{t}}\\left[\\mathbb{E}_{f(\\cdot;\\theta)} [h(x, \\xi)]\\right],\n\\end{align*}\nwhere the first equality holds because the gradient $\\nabla_x h(x,\\xi)$ is Lipschitz continuous, and the interchange between expectation and differentiation is justified by dominated convergence theorem (DCT). Similarly, the second equality above is again justified by DCT. Therefore, the proposed estimator in \\cref{eq: SGD_independent_2} is unbiased gradient estimator of the objective function in \\cref{eq: independent}.\n\\end{proof}\n\n\\begin{lemma}\\label{lemma: independent_concentration}\nUnder \\cref{ass: independent_rate}, there exists a constant $C_1>0$ such that for any $\\delta>0$, with probability at least $1-\\delta$ we have\n\\begin{align*}\n\t||\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta)-\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta^c)||_2^2\\leq C_1 \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt}, \\forall x\\in \\mathcal{X}, \\forall t>0. \n\\end{align*}\n\\end{lemma}\n\n\\begin{proof}\nDefine the Hellinger distance between $\\theta_1$ and $\\theta_2$ as\n\\begin{align*}\n    d(\\theta_1,\\theta_2) =\\sqrt{\\frac{1}{2}\\int_\\mathcal{Y} (\\sqrt{f(y;\\theta_1)}-\\sqrt{f(y;\\theta_2)})^2}.\n\\end{align*}\nOne can easily verify that there exist a constant $A$ such that $\\|\\theta_1-\\theta_2\\|\\leq Ad(\\theta_1,\\theta_2)$, where $||\\cdot||$ is the Euclidean norm. Let $B_k^t = B(\\theta^c, k/\\sqrt{Dt})$ be a ball centered at $\\theta^c$ with radius $k/\\sqrt{Dt}$ under distance $d.$ Since $\\Theta$ is finite, we can directly apply Proposition 1 in \\cite{birge2015non}. Then for $t\\leq T, \\epsilon,\\delta\\in(0,1)$ with probability at least $1-\\frac{6\\delta}{\\pi^2 t^2}$ with respect to $\\mathbb{P}_{\\theta^c}^{t}$, we have\n\\begin{align*}\n\t\\pi_t(B_{k(t)}^t) \\geq 1-\\epsilon,\n\\end{align*}\nwhere \n\\begin{align*}\n    k(t) =\\inf\\left\\{j\\geq 1\\Big| \\sum_{i\\geq j}|\\Theta|e^{-i^2}\\leq \\frac{6\\delta}{\\pi^2 t^2}\\sqrt{\\epsilon \\pi_0(\\theta^c)}\\right\\}.\n\\end{align*}\nNote that $\\sum_{i\\geq j}e^{-i^2}\\leq \\frac{e}{e-1} e^{-j^2}$, we can set $k(t)$ to be the solution of next equation. \n\\begin{align*}\n    \\frac{e}{e-1}|\\Theta| e^{-k(t)^2} =\\frac{6\\delta}{\\pi^2 t^2}\\sqrt{\\epsilon \\pi_0(\\theta^c)}.\n\\end{align*}\nBy simple calculation, we have $k(t) = \\sqrt{\\log \\frac{e|\\Theta|\\pi^2 t^2}{6\\delta(e-1)\\sqrt{\\epsilon\\pi_0(\\theta^c)}}}$. Now we are ready to bound the bias in the gradient estimator.\n\\begin{align*}\n\t& \\|\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta)-\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta^c)\\|_2^2 \\\\\n\t& = \\left\\|\\int (\\nabla_x H(x, \\theta)- \\nabla_x H(x, \\theta^c))\\pi_t(\\theta) d\\theta\\right\\|_2^2\\\\\n\t&\\leq \\int  \\left\\|(\\nabla_x H(x, \\theta)- \\nabla_x H(x, \\theta^c))\\right\\|_2^2\\pi_t(\\theta) d\\theta\\\\\n\t&\\leq  \\int L_{H}^2||\\theta-\\theta^c||_2^2 \\pi_t(\\theta) d\\theta\\\\\n\t&=  \\int_{B_{k(t)}^t} L_{H}^2||\\theta-\\theta^c||_2^2 \\pi_t(\\theta) d\\theta + \\int_{(B_{k(t)}^t)^c} L_{H}^2||\\theta-\\theta^c||_2^2 \\pi_t(\\theta) d\\theta\\\\\n\t&\\leq A^2 L_{H}^2 \\frac{k(t)^2}{Dt}  \\int_{B_{k(t)}^t} \\pi_t(\\theta) d\\theta + L_{H}^2 \\max_{\\theta\\in\\Theta}\\|\\theta-\\theta^c\\|^2_2\\int_{(B_{k(t)}^t)^c}  \\pi_t(\\theta) d\\theta\\\\\n\t&\\leq  A^2 L_{H}^2 \\frac{k(t)^2}{Dt} +  L_{H}^2 \\max_{\\theta\\in\\Theta}\\|\\theta-\\theta^c\\|^2_2\\epsilon.\n\\end{align*}\nRecall that $D$ is the data batch size. Take $\\epsilon =\\frac{1}{Dt}$, note that $k(t) =\\sqrt{\\log \\frac{e|\\Theta|\\pi^2 t^2\\sqrt{Dt}}{6\\delta(e-1)\\sqrt{\\pi_0(\\theta^c)}}}$. We further have\n\\begin{align*}\n\t\\|\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta)-\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta^c)\\|_2^2\n\t&\\leq  A^2 L_{H}^2 \\frac{k(t)^2}{Dt} +  L_{H}^2 \\max_{\\theta\\in\\Theta}\\|\\theta-\\theta^c\\|^2_2\\epsilon\\\\\n\t&\\leq 2A^2 L_{H}^2\\max_{\\theta\\in\\Theta}\\|\\theta-\\theta^c\\|^2_2\\frac{\\log \\frac{e|\\Theta|\\pi^2 t^2\\sqrt{Dt}}{6\\delta(e-1)\\sqrt{\\pi_0(\\theta^c)} }}{Dt}\\\\\n\t&= O(\\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt}).\n\\end{align*}\n\nLet $\\mathcal{E}_t$ denote the event that the above inequality holds, and $\\mathcal{E}_t^c$ denote the complement event. Then we have $\\mathbb{P}(\\mathcal{E}_t^c) \\leq \\frac{6\\delta}{\\pi^2 t^2}$. Therefore,\n\\begin{align*}\n    \\mathbb{P}(\\cap_{t=1}^\\infty \\mathcal{E}_t) & = 1- \\mathbb{P}(\\bigcup_{t=1}^{\\infty} \\mathcal{E}^c_t) \\\\\n    & \\geq 1 - \\sum_{t=1}^{\\infty} \\mathbb{P}(\\mathcal{E}_t^c) \\quad (\\text{union bound}) \\\\\n    & \\geq 1 - \\sum_{t=1}^{\\infty} \\frac{6\\delta}{\\pi^2 t^2} \\\\\n    & = 1 - \\delta.\n\\end{align*}\n\\end{proof}\n\n\\begin{proof}\nBy the update \\cref{eq: SGD_independent_2}, we know that for any $t\\leq T$,\n\\begin{align*}\n    x_{t+1} &= x_{t} - a_{t} \\nabla_x H(x_{t},\\theta^c) -  a_{t}[\\mathbb{E}_{\\pi_t}\\nabla_x H(x_{t}, \\theta)-\\nabla_x H(x_{t}, \\theta^c)]\t\\\\\n    &~~~~~~~~- a_{t}[\\nabla_x h(x_{t}, \\xi_{t})-\\mathbb{E}_{\\pi_t}\\nabla_x H(x_{t}, \\theta)]\\\\\n    & =  x_{t} - a_{t}\t\\nabla_x H(x_{t},\\theta^c) -  a_{t} B_{t} - a_{t} N_{t},\n\\end{align*}\nwhere $B_{t}$ is the bias and $N_{t}$ is the noise. By \\cref{lemma: independent_concentration}, we know $\\mathbb{E}[\\|B_{t}\\|_2^2]\\leq  C_1 \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt}.$ From \\cref{ass: independent_rate} we have $\\mathbb{E}[\\|N_{t}\\|_2^2]\\leq \\sigma^2.$ By the proof of Lemma 2 in \\cite{ajalloeian2020analysis}, we know that \n\\begin{align*}\n    \\mathbb{E}[H(x_{t+1},\\theta^c)] - H(x_{t},\\theta^c)\\leq -\\frac{a_{t}}{2}\\|\\nabla_x H(x_{t},\\theta^c) \\|_2^2 +\\frac{a_{t}}{2}C_1 \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt} +\\frac{a_{t}^2}{2}L_{h} \\sigma^2,\n\\end{align*} \nRearranging the terms in the inequality above, summing over $t$ from 1 to $T$, and noting that $H(x_{t}, \\theta^c) \\leq \\min_{x \\in \\mathcal{X}} H(x, \\theta^c), \\forall t$, we have \n{\\small\n\\begin{align*}\n    \\sum_{t=1}^{T} a_t \\mathbb{E}[\\|\\nabla_x H(x_{t},\\theta^c) \\|_2^2]\\leq 2\n    (H(x_1,\\theta^c) - \\min_{x \\in \\mathcal{X}} H(x,\\theta^c))\n    + C_1 \\sum_{t=1}^{T} a_t \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt} + L_{h}\\sigma^2 \\sum_{t=1}^{T} a_t^2,\n\\end{align*}\n}\nDividing both sides of the above inequality by $\\sum_{t=1}^{T} a_t$, and noting that \n\\begin{align*}\n    \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2] = \\frac{1}{\\sum_{t=1}^{T} a_t} \\sum_{t=1}^{T} a_t \\mathbb{E}[\\|\\nabla_x H(x_{t},\\theta^c) \\|_2^2],\n\\end{align*}\nwe have\n{\\small\n\\begin{align*}\n    \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2] \\leq \\frac{1}{\\sum_{t=1}^{T} a_t}\\left[2 (H(x_1,\\theta^c) - \\min_{x \\in \\mathcal{X}} H(x,\\theta^c)) +C_1 \\sum_{t=1}^{T} a_t \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt} + L_{h} \\sigma^2\\sum_{t=1}^{T}a_t^2.\n    \\right]\n\\end{align*}\n}\n(i) $a_t=\\frac{a}{\\sqrt{T}}$, $\\forall t \\leq T$, for some constant $a<\\frac{\\sqrt{T}}{L_{h}}$. Note that $\\sum_{t=1}^{T} \\frac{1}{t} \\leq \\log T+1$ and $\\sum_{t=1}^{T} \\frac{\\log t}{t} \\leq \\log (\\log T+1)$.\nThen\n{\\small\n\\begin{align*}\n    & \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2]\\\\\n    &\\leq \\frac{2(H(x_1,\\theta^c)- \\min_{x}H(x,\\theta^c))}{a\\sqrt{T}} + \\frac{C_1 (\\log D - \\log \\delta) (\\log T + 1)}{L_{h} D T} + \\frac{C_1 \\log T (\\log T + 1)}{L_{h} D T} \\\\ \n    & = \\frac{2(H(x_1,\\theta^c)- \\min_{x}H(x,\\theta^c))}{a\\sqrt{T}} + \\frac{C_1 (\\log D - \\log \\delta)}{L_{h} D T} + \\frac{C_1 (\\log D - \\log \\delta) \\log T}{L_{h} D T} + \\frac{C_1 \\log^2 T}{L_{h} D T} + \\frac{L_{h} a\\sigma^2}{\\sqrt{T}}\n\\end{align*}\n}\n(ii) $a_t = \\frac{a}{t}$, $\\forall t \\leq T$, for some constant $a < \\frac{1}{L_{h}}$. Let $M_T=\\sum_{t=1}^{T}\\frac{1}{t}$. Note that \n\\begin{align*}\n    \\sum_{t=1}^{T}\\frac{\\log t}{t^2} < \\sum_{t=1}^{\\infty}\\frac{\\log t}{t^2} = \\frac{\\pi^2}{6}(12 \\ln A - \\gamma - \\ln 2\\pi) < 1,\n\\end{align*}\nwhere $A \\approx 1.28$ is the Glaisher-Kinkelin constant and $\\gamma \\approx 0.58$ is the Euler-Mascheroni constant. Then we have\n\\begin{align*}\n\t& \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2]\\\\\n\t&\\leq \\frac{2(H(x_1,\\theta^c)- \\min_{x\\in\\mathcal{X}}H(x,\\theta^c))}{a M_T} +\\frac{C_1}{M_T}\\sum_{t=1}^T \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt^2} +\\sum_{t=1}^T \\frac{L_{h} a \\sigma^2}{M_T t^2}\\\\\n\t& \\leq \\left[\\frac{2(H(x_1,\\theta^c)- \\min_{x\\in\\mathcal{X}}H(x,\\theta^c))}{a} +\\frac{6C_1 + \\pi^2C_1 (\\log D - \\log \\delta)}{6D}+\\frac{\\pi^2 L_{h} a \\sigma^2}{6}\\right]\\frac{1}{\\log T}.\n\\end{align*}\n(iii) $a_t = \\frac{a}{\\sqrt{t}}$, $\\forall t \\leq T$, for some constant $a < \\frac{1}{L_{h}}$. Let $Q_t=\\sum_{t=1}^{T}\\frac{1}{\\sqrt{t}}$. Note that $\\sum_{t=1}^{\\infty}\\frac{1}{t\\sqrt{t}}=\\zeta(1.5)\\approx2.61<3$, $\\sum_{t=1}^{\\infty} \\frac{\\log t}{t\\sqrt{t}} < 4$, $\\sum_{t=1}^{T}\\frac{1}{\\sqrt{t}} \\geq \\sqrt{T}$, where $\\zeta(\\cdot)$ is the Riemann’s zeta function. Then we have\n{\\small\n\\begin{align*}\n\t& \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2]\\\\\n\t&\\leq \\frac{2(H(x_1,\\theta^c)- \\min_{x}H(x,\\theta^c))}{a Q_T} +\\frac{C_1(\\log D - \\log \\delta)}{D Q_T}\\sum_{t=1}^T \\frac{1}{t\\sqrt{t}} + \\frac{C_1}{D Q_T}\\sum_{t=1}^{T}\\frac{\\log t}{t \\sqrt{t}} +  \\frac{L_{h} a \\sigma^2}{Q_T} \\sum_{t=1}^T \\frac{1}{t}\\\\\n\t&\\leq \\left[\\frac{2(H(x_1,\\theta^c)- \\min_{x}H(x,\\theta^c))}{a\\sqrt{T}} + \\frac{3C_1(\\log D - \\log \\delta) + 4C_1}{D\\sqrt{T}} + \\frac{L_{h} a \\sigma^2}{\\sqrt{T}}\\right] + \\frac{L_{h} a \\sigma^2 \\log T}{\\sqrt{T}}.\n\\end{align*}\n}\n\\end{proof}","units":[{"start":34813,"end":35078,"start_line":277,"end_line":279,"raw_tex":"\\begin{lemma}\\label{lemma: independent_unbiasedness}\nUnder \\cref{ass: independent_DCT}, $\\nabla_{x} h(x, \\xi)$ with $\\xi \\sim f(\\cdot;\\theta)$ and $\\theta \\sim \\pi_t$ is an unbiased gradient estimator of the objective function in \\cref{eq: independent}.\n\\end{lemma}","sha256_utf8":"ca69a3ebc12078386cc119337948f9b81fcd743487ff5f3955f2415444b12c93"},{"start":35080,"end":35865,"start_line":281,"end_line":288,"raw_tex":"\\begin{proof}\nFor every fixed $x\\in\\mathcal{X}$, \n\\begin{align*}\n    \\mathbb{E}_{\\pi_{t}}\\left[\\mathbb{E}_{f(\\cdot;\\theta)} [\\nabla_x h(x, \\xi)]\\right] & = \\mathbb{E}_{\\pi_{t}}\\left[\\nabla_{x} \\mathbb{E}_{f(\\cdot;\\theta)} [h(x, \\xi)]\\right]\\\\\n    & = \\nabla_{x} \\mathbb{E}_{\\pi_{t}}\\left[\\mathbb{E}_{f(\\cdot;\\theta)} [h(x, \\xi)]\\right],\n\\end{align*}\nwhere the first equality holds because the gradient $\\nabla_x h(x,\\xi)$ is Lipschitz continuous, and the interchange between expectation and differentiation is justified by dominated convergence theorem (DCT). Similarly, the second equality above is again justified by DCT. Therefore, the proposed estimator in \\cref{eq: SGD_independent_2} is unbiased gradient estimator of the objective function in \\cref{eq: independent}.\n\\end{proof}","sha256_utf8":"860b262f44508f936642df8aaed976795bc0465a41aeb61b686e4326f3932031"},{"start":42801,"end":43220,"start_line":343,"end_line":348,"raw_tex":"\\begin{lemma}\\label{lemma: independent_concentration}\nUnder \\cref{ass: independent_rate}, there exists a constant $C_1>0$ such that for any $\\delta>0$, with probability at least $1-\\delta$ we have\n\\begin{align*}\n\t||\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta)-\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta^c)||_2^2\\leq C_1 \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt}, \\forall x\\in \\mathcal{X}, \\forall t>0. \n\\end{align*}\n\\end{lemma}","sha256_utf8":"d7427c66a5b94e4ee0a55f1a81e84d2b13f4f6df9cdcf39522020a069f941058"},{"start":87357,"end":90737,"start_line":681,"end_line":723,"raw_tex":"\\begin{proof}\nDefine the Hellinger distance between $\\theta_1$ and $\\theta_2$ as\n\\begin{align*}\n    d(\\theta_1,\\theta_2) =\\sqrt{\\frac{1}{2}\\int_\\mathcal{Y} (\\sqrt{f(y;\\theta_1)}-\\sqrt{f(y;\\theta_2)})^2}.\n\\end{align*}\nOne can easily verify that there exist a constant $A$ such that $\\|\\theta_1-\\theta_2\\|\\leq Ad(\\theta_1,\\theta_2)$, where $||\\cdot||$ is the Euclidean norm. Let $B_k^t = B(\\theta^c, k/\\sqrt{Dt})$ be a ball centered at $\\theta^c$ with radius $k/\\sqrt{Dt}$ under distance $d.$ Since $\\Theta$ is finite, we can directly apply Proposition 1 in \\cite{birge2015non}. Then for $t\\leq T, \\epsilon,\\delta\\in(0,1)$ with probability at least $1-\\frac{6\\delta}{\\pi^2 t^2}$ with respect to $\\mathbb{P}_{\\theta^c}^{t}$, we have\n\\begin{align*}\n\t\\pi_t(B_{k(t)}^t) \\geq 1-\\epsilon,\n\\end{align*}\nwhere \n\\begin{align*}\n    k(t) =\\inf\\left\\{j\\geq 1\\Big| \\sum_{i\\geq j}|\\Theta|e^{-i^2}\\leq \\frac{6\\delta}{\\pi^2 t^2}\\sqrt{\\epsilon \\pi_0(\\theta^c)}\\right\\}.\n\\end{align*}\nNote that $\\sum_{i\\geq j}e^{-i^2}\\leq \\frac{e}{e-1} e^{-j^2}$, we can set $k(t)$ to be the solution of next equation. \n\\begin{align*}\n    \\frac{e}{e-1}|\\Theta| e^{-k(t)^2} =\\frac{6\\delta}{\\pi^2 t^2}\\sqrt{\\epsilon \\pi_0(\\theta^c)}.\n\\end{align*}\nBy simple calculation, we have $k(t) = \\sqrt{\\log \\frac{e|\\Theta|\\pi^2 t^2}{6\\delta(e-1)\\sqrt{\\epsilon\\pi_0(\\theta^c)}}}$. Now we are ready to bound the bias in the gradient estimator.\n\\begin{align*}\n\t& \\|\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta)-\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta^c)\\|_2^2 \\\\\n\t& = \\left\\|\\int (\\nabla_x H(x, \\theta)- \\nabla_x H(x, \\theta^c))\\pi_t(\\theta) d\\theta\\right\\|_2^2\\\\\n\t&\\leq \\int  \\left\\|(\\nabla_x H(x, \\theta)- \\nabla_x H(x, \\theta^c))\\right\\|_2^2\\pi_t(\\theta) d\\theta\\\\\n\t&\\leq  \\int L_{H}^2||\\theta-\\theta^c||_2^2 \\pi_t(\\theta) d\\theta\\\\\n\t&=  \\int_{B_{k(t)}^t} L_{H}^2||\\theta-\\theta^c||_2^2 \\pi_t(\\theta) d\\theta + \\int_{(B_{k(t)}^t)^c} L_{H}^2||\\theta-\\theta^c||_2^2 \\pi_t(\\theta) d\\theta\\\\\n\t&\\leq A^2 L_{H}^2 \\frac{k(t)^2}{Dt}  \\int_{B_{k(t)}^t} \\pi_t(\\theta) d\\theta + L_{H}^2 \\max_{\\theta\\in\\Theta}\\|\\theta-\\theta^c\\|^2_2\\int_{(B_{k(t)}^t)^c}  \\pi_t(\\theta) d\\theta\\\\\n\t&\\leq  A^2 L_{H}^2 \\frac{k(t)^2}{Dt} +  L_{H}^2 \\max_{\\theta\\in\\Theta}\\|\\theta-\\theta^c\\|^2_2\\epsilon.\n\\end{align*}\nRecall that $D$ is the data batch size. Take $\\epsilon =\\frac{1}{Dt}$, note that $k(t) =\\sqrt{\\log \\frac{e|\\Theta|\\pi^2 t^2\\sqrt{Dt}}{6\\delta(e-1)\\sqrt{\\pi_0(\\theta^c)}}}$. We further have\n\\begin{align*}\n\t\\|\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta)-\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta^c)\\|_2^2\n\t&\\leq  A^2 L_{H}^2 \\frac{k(t)^2}{Dt} +  L_{H}^2 \\max_{\\theta\\in\\Theta}\\|\\theta-\\theta^c\\|^2_2\\epsilon\\\\\n\t&\\leq 2A^2 L_{H}^2\\max_{\\theta\\in\\Theta}\\|\\theta-\\theta^c\\|^2_2\\frac{\\log \\frac{e|\\Theta|\\pi^2 t^2\\sqrt{Dt}}{6\\delta(e-1)\\sqrt{\\pi_0(\\theta^c)} }}{Dt}\\\\\n\t&= O(\\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt}).\n\\end{align*}\n\nLet $\\mathcal{E}_t$ denote the event that the above inequality holds, and $\\mathcal{E}_t^c$ denote the complement event. Then we have $\\mathbb{P}(\\mathcal{E}_t^c) \\leq \\frac{6\\delta}{\\pi^2 t^2}$. Therefore,\n\\begin{align*}\n    \\mathbb{P}(\\cap_{t=1}^\\infty \\mathcal{E}_t) & = 1- \\mathbb{P}(\\bigcup_{t=1}^{\\infty} \\mathcal{E}^c_t) \\\\\n    & \\geq 1 - \\sum_{t=1}^{\\infty} \\mathbb{P}(\\mathcal{E}_t^c) \\quad (\\text{union bound}) \\\\\n    & \\geq 1 - \\sum_{t=1}^{\\infty} \\frac{6\\delta}{\\pi^2 t^2} \\\\\n    & = 1 - \\delta.\n\\end{align*}\n\\end{proof}","sha256_utf8":"4a27bd1a8ca4e2700fb976b77c9b2144b5780ff251973c24ba5f954c2c530f3c"},{"start":90807,"end":95427,"start_line":727,"end_line":784,"raw_tex":"\\begin{proof}\nBy the update \\cref{eq: SGD_independent_2}, we know that for any $t\\leq T$,\n\\begin{align*}\n    x_{t+1} &= x_{t} - a_{t} \\nabla_x H(x_{t},\\theta^c) -  a_{t}[\\mathbb{E}_{\\pi_t}\\nabla_x H(x_{t}, \\theta)-\\nabla_x H(x_{t}, \\theta^c)]\t\\\\\n    &~~~~~~~~- a_{t}[\\nabla_x h(x_{t}, \\xi_{t})-\\mathbb{E}_{\\pi_t}\\nabla_x H(x_{t}, \\theta)]\\\\\n    & =  x_{t} - a_{t}\t\\nabla_x H(x_{t},\\theta^c) -  a_{t} B_{t} - a_{t} N_{t},\n\\end{align*}\nwhere $B_{t}$ is the bias and $N_{t}$ is the noise. By \\cref{lemma: independent_concentration}, we know $\\mathbb{E}[\\|B_{t}\\|_2^2]\\leq  C_1 \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt}.$ From \\cref{ass: independent_rate} we have $\\mathbb{E}[\\|N_{t}\\|_2^2]\\leq \\sigma^2.$ By the proof of Lemma 2 in \\cite{ajalloeian2020analysis}, we know that \n\\begin{align*}\n    \\mathbb{E}[H(x_{t+1},\\theta^c)] - H(x_{t},\\theta^c)\\leq -\\frac{a_{t}}{2}\\|\\nabla_x H(x_{t},\\theta^c) \\|_2^2 +\\frac{a_{t}}{2}C_1 \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt} +\\frac{a_{t}^2}{2}L_{h} \\sigma^2,\n\\end{align*} \nRearranging the terms in the inequality above, summing over $t$ from 1 to $T$, and noting that $H(x_{t}, \\theta^c) \\leq \\min_{x \\in \\mathcal{X}} H(x, \\theta^c), \\forall t$, we have \n{\\small\n\\begin{align*}\n    \\sum_{t=1}^{T} a_t \\mathbb{E}[\\|\\nabla_x H(x_{t},\\theta^c) \\|_2^2]\\leq 2\n    (H(x_1,\\theta^c) - \\min_{x \\in \\mathcal{X}} H(x,\\theta^c))\n    + C_1 \\sum_{t=1}^{T} a_t \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt} + L_{h}\\sigma^2 \\sum_{t=1}^{T} a_t^2,\n\\end{align*}\n}\nDividing both sides of the above inequality by $\\sum_{t=1}^{T} a_t$, and noting that \n\\begin{align*}\n    \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2] = \\frac{1}{\\sum_{t=1}^{T} a_t} \\sum_{t=1}^{T} a_t \\mathbb{E}[\\|\\nabla_x H(x_{t},\\theta^c) \\|_2^2],\n\\end{align*}\nwe have\n{\\small\n\\begin{align*}\n    \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2] \\leq \\frac{1}{\\sum_{t=1}^{T} a_t}\\left[2 (H(x_1,\\theta^c) - \\min_{x \\in \\mathcal{X}} H(x,\\theta^c)) +C_1 \\sum_{t=1}^{T} a_t \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt} + L_{h} \\sigma^2\\sum_{t=1}^{T}a_t^2.\n    \\right]\n\\end{align*}\n}\n(i) $a_t=\\frac{a}{\\sqrt{T}}$, $\\forall t \\leq T$, for some constant $a<\\frac{\\sqrt{T}}{L_{h}}$. Note that $\\sum_{t=1}^{T} \\frac{1}{t} \\leq \\log T+1$ and $\\sum_{t=1}^{T} \\frac{\\log t}{t} \\leq \\log (\\log T+1)$.\nThen\n{\\small\n\\begin{align*}\n    & \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2]\\\\\n    &\\leq \\frac{2(H(x_1,\\theta^c)- \\min_{x}H(x,\\theta^c))}{a\\sqrt{T}} + \\frac{C_1 (\\log D - \\log \\delta) (\\log T + 1)}{L_{h} D T} + \\frac{C_1 \\log T (\\log T + 1)}{L_{h} D T} \\\\ \n    & = \\frac{2(H(x_1,\\theta^c)- \\min_{x}H(x,\\theta^c))}{a\\sqrt{T}} + \\frac{C_1 (\\log D - \\log \\delta)}{L_{h} D T} + \\frac{C_1 (\\log D - \\log \\delta) \\log T}{L_{h} D T} + \\frac{C_1 \\log^2 T}{L_{h} D T} + \\frac{L_{h} a\\sigma^2}{\\sqrt{T}}\n\\end{align*}\n}\n(ii) $a_t = \\frac{a}{t}$, $\\forall t \\leq T$, for some constant $a < \\frac{1}{L_{h}}$. Let $M_T=\\sum_{t=1}^{T}\\frac{1}{t}$. Note that \n\\begin{align*}\n    \\sum_{t=1}^{T}\\frac{\\log t}{t^2} < \\sum_{t=1}^{\\infty}\\frac{\\log t}{t^2} = \\frac{\\pi^2}{6}(12 \\ln A - \\gamma - \\ln 2\\pi) < 1,\n\\end{align*}\nwhere $A \\approx 1.28$ is the Glaisher-Kinkelin constant and $\\gamma \\approx 0.58$ is the Euler-Mascheroni constant. Then we have\n\\begin{align*}\n\t& \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2]\\\\\n\t&\\leq \\frac{2(H(x_1,\\theta^c)- \\min_{x\\in\\mathcal{X}}H(x,\\theta^c))}{a M_T} +\\frac{C_1}{M_T}\\sum_{t=1}^T \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt^2} +\\sum_{t=1}^T \\frac{L_{h} a \\sigma^2}{M_T t^2}\\\\\n\t& \\leq \\left[\\frac{2(H(x_1,\\theta^c)- \\min_{x\\in\\mathcal{X}}H(x,\\theta^c))}{a} +\\frac{6C_1 + \\pi^2C_1 (\\log D - \\log \\delta)}{6D}+\\frac{\\pi^2 L_{h} a \\sigma^2}{6}\\right]\\frac{1}{\\log T}.\n\\end{align*}\n(iii) $a_t = \\frac{a}{\\sqrt{t}}$, $\\forall t \\leq T$, for some constant $a < \\frac{1}{L_{h}}$. Let $Q_t=\\sum_{t=1}^{T}\\frac{1}{\\sqrt{t}}$. Note that $\\sum_{t=1}^{\\infty}\\frac{1}{t\\sqrt{t}}=\\zeta(1.5)\\approx2.61<3$, $\\sum_{t=1}^{\\infty} \\frac{\\log t}{t\\sqrt{t}} < 4$, $\\sum_{t=1}^{T}\\frac{1}{\\sqrt{t}} \\geq \\sqrt{T}$, where $\\zeta(\\cdot)$ is the Riemann’s zeta function. Then we have\n{\\small\n\\begin{align*}\n\t& \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2]\\\\\n\t&\\leq \\frac{2(H(x_1,\\theta^c)- \\min_{x}H(x,\\theta^c))}{a Q_T} +\\frac{C_1(\\log D - \\log \\delta)}{D Q_T}\\sum_{t=1}^T \\frac{1}{t\\sqrt{t}} + \\frac{C_1}{D Q_T}\\sum_{t=1}^{T}\\frac{\\log t}{t \\sqrt{t}} +  \\frac{L_{h} a \\sigma^2}{Q_T} \\sum_{t=1}^T \\frac{1}{t}\\\\\n\t&\\leq \\left[\\frac{2(H(x_1,\\theta^c)- \\min_{x}H(x,\\theta^c))}{a\\sqrt{T}} + \\frac{3C_1(\\log D - \\log \\delta) + 4C_1}{D\\sqrt{T}} + \\frac{L_{h} a \\sigma^2}{\\sqrt{T}}\\right] + \\frac{L_{h} a \\sigma^2 \\log T}{\\sqrt{T}}.\n\\end{align*}\n}\n\\end{proof}","sha256_utf8":"6815777f787baceac5101626eb2d41ae90ff5000ffc4039738acbd0caf1e79ad"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"贝叶斯随机梯度收敛率","source_units":[{"start":44010,"end":45892,"start_line":352,"end_line":378,"raw_tex":"\\begin{theorem}\\label{thm: independent_rate}\nUnder \\cref{ass: Alex2022}, \\cref{ass: independent_DCT}, \\cref{ass: independent_step}, and \\cref{ass: independent_rate}, for any $\\delta>0$, we have with probability at least $1-\\delta,$ for any $T>0$, the following bound on the expected gradient of the final output under the true parameter $\\theta^c$  \n\\begin{enumerate}\n    \\item[(i)] If the step size satisfies $a_t=\\frac{a}{\\sqrt{T}}$, $\\forall t \\leq T$, for some constant $a<\\frac{\\sqrt{T}}{L_h}$, then \n    {\\small\n    \\begin{align*}\n        & \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2] \\\\\n\t    &\\leq \\left[\\frac{2(H(x_1,\\theta^c)- \\min_{x\\in\\mathcal{X}}H(x,\\theta^c))}{a\\sqrt{T}}\\right] + \\left[\\frac{A_1}{T} + \\frac{A_2 \\log T}{T} + \\frac{A_3 \\log^2 T}{T}\\right] + \\frac{L_h a\\sigma^2}{\\sqrt{T}},\n    \\end{align*}\n    }\n    where $A_1=\\frac{C_1 (\\log D - \\log \\delta)}{L_h D}$, $A_2=\\frac{C_1 (\\log D - \\log \\delta)}{L_h D} + \\frac{C_1}{L_h D}$, $A_3=\\frac{C_1}{L_h D}$.\n    \\item[(ii)]  If the step size satisfies $a_t = \\frac{a}{t}$, $\\forall t \\leq T$, for some constant $a < \\frac{1}{L_h}$, then\n    {\\small\n    \\begin{align*}\n\t& \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2]\\\\\n\t&\\leq \\left[\\frac{2(H(x_1,\\theta^c)- \\min_{x\\in\\mathcal{X}}H(x,\\theta^c))}{a} +\\frac{6C_1 + \\pi^2C_1 (\\log D - \\log \\delta)}{6D}+\\frac{\\pi^2 L_h a \\sigma^2}{6}\\right]\\frac{1}{\\log T}.\n    \\end{align*}\n    }\n    \\item[(iii)]  If the step size satisfies $a_t = \\frac{a}{\\sqrt{t}}$, $\\forall t \\leq T$, for some constant $a < \\frac{1}{L_h}$, then\n    {\\small\n    \\begin{align*}\n\t& \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2]\\\\\n\t&\\leq [\\frac{2(H(x_1,\\theta^c)- \\min_{x\\in\\mathcal{X}}H(x,\\theta^c))}{a\\sqrt{T}} + \\frac{3C_1(\\log D - \\log \\delta) + 4C_1}{D\\sqrt{T}} + \\frac{L_h a \\sigma^2}{\\sqrt{T}}] + \\frac{L_h a \\sigma^2 \\log T}{\\sqrt{T}}.\n    \\end{align*}\n    }\n\\end{enumerate}\n\\end{theorem}","sha256_utf8":"2a91377f60ab9ccf6afb03a2b3c62a3a2973626b4861bcd5c5c3e049aacec410"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"收敛率定理完整证明","source_units":[{"start":90807,"end":95427,"start_line":727,"end_line":784,"raw_tex":"\\begin{proof}\nBy the update \\cref{eq: SGD_independent_2}, we know that for any $t\\leq T$,\n\\begin{align*}\n    x_{t+1} &= x_{t} - a_{t} \\nabla_x H(x_{t},\\theta^c) -  a_{t}[\\mathbb{E}_{\\pi_t}\\nabla_x H(x_{t}, \\theta)-\\nabla_x H(x_{t}, \\theta^c)]\t\\\\\n    &~~~~~~~~- a_{t}[\\nabla_x h(x_{t}, \\xi_{t})-\\mathbb{E}_{\\pi_t}\\nabla_x H(x_{t}, \\theta)]\\\\\n    & =  x_{t} - a_{t}\t\\nabla_x H(x_{t},\\theta^c) -  a_{t} B_{t} - a_{t} N_{t},\n\\end{align*}\nwhere $B_{t}$ is the bias and $N_{t}$ is the noise. By \\cref{lemma: independent_concentration}, we know $\\mathbb{E}[\\|B_{t}\\|_2^2]\\leq  C_1 \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt}.$ From \\cref{ass: independent_rate} we have $\\mathbb{E}[\\|N_{t}\\|_2^2]\\leq \\sigma^2.$ By the proof of Lemma 2 in \\cite{ajalloeian2020analysis}, we know that \n\\begin{align*}\n    \\mathbb{E}[H(x_{t+1},\\theta^c)] - H(x_{t},\\theta^c)\\leq -\\frac{a_{t}}{2}\\|\\nabla_x H(x_{t},\\theta^c) \\|_2^2 +\\frac{a_{t}}{2}C_1 \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt} +\\frac{a_{t}^2}{2}L_{h} \\sigma^2,\n\\end{align*} \nRearranging the terms in the inequality above, summing over $t$ from 1 to $T$, and noting that $H(x_{t}, \\theta^c) \\leq \\min_{x \\in \\mathcal{X}} H(x, \\theta^c), \\forall t$, we have \n{\\small\n\\begin{align*}\n    \\sum_{t=1}^{T} a_t \\mathbb{E}[\\|\\nabla_x H(x_{t},\\theta^c) \\|_2^2]\\leq 2\n    (H(x_1,\\theta^c) - \\min_{x \\in \\mathcal{X}} H(x,\\theta^c))\n    + C_1 \\sum_{t=1}^{T} a_t \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt} + L_{h}\\sigma^2 \\sum_{t=1}^{T} a_t^2,\n\\end{align*}\n}\nDividing both sides of the above inequality by $\\sum_{t=1}^{T} a_t$, and noting that \n\\begin{align*}\n    \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2] = \\frac{1}{\\sum_{t=1}^{T} a_t} \\sum_{t=1}^{T} a_t \\mathbb{E}[\\|\\nabla_x H(x_{t},\\theta^c) \\|_2^2],\n\\end{align*}\nwe have\n{\\small\n\\begin{align*}\n    \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2] \\leq \\frac{1}{\\sum_{t=1}^{T} a_t}\\left[2 (H(x_1,\\theta^c) - \\min_{x \\in \\mathcal{X}} H(x,\\theta^c)) +C_1 \\sum_{t=1}^{T} a_t \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt} + L_{h} \\sigma^2\\sum_{t=1}^{T}a_t^2.\n    \\right]\n\\end{align*}\n}\n(i) $a_t=\\frac{a}{\\sqrt{T}}$, $\\forall t \\leq T$, for some constant $a<\\frac{\\sqrt{T}}{L_{h}}$. Note that $\\sum_{t=1}^{T} \\frac{1}{t} \\leq \\log T+1$ and $\\sum_{t=1}^{T} \\frac{\\log t}{t} \\leq \\log (\\log T+1)$.\nThen\n{\\small\n\\begin{align*}\n    & \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2]\\\\\n    &\\leq \\frac{2(H(x_1,\\theta^c)- \\min_{x}H(x,\\theta^c))}{a\\sqrt{T}} + \\frac{C_1 (\\log D - \\log \\delta) (\\log T + 1)}{L_{h} D T} + \\frac{C_1 \\log T (\\log T + 1)}{L_{h} D T} \\\\ \n    & = \\frac{2(H(x_1,\\theta^c)- \\min_{x}H(x,\\theta^c))}{a\\sqrt{T}} + \\frac{C_1 (\\log D - \\log \\delta)}{L_{h} D T} + \\frac{C_1 (\\log D - \\log \\delta) \\log T}{L_{h} D T} + \\frac{C_1 \\log^2 T}{L_{h} D T} + \\frac{L_{h} a\\sigma^2}{\\sqrt{T}}\n\\end{align*}\n}\n(ii) $a_t = \\frac{a}{t}$, $\\forall t \\leq T$, for some constant $a < \\frac{1}{L_{h}}$. Let $M_T=\\sum_{t=1}^{T}\\frac{1}{t}$. Note that \n\\begin{align*}\n    \\sum_{t=1}^{T}\\frac{\\log t}{t^2} < \\sum_{t=1}^{\\infty}\\frac{\\log t}{t^2} = \\frac{\\pi^2}{6}(12 \\ln A - \\gamma - \\ln 2\\pi) < 1,\n\\end{align*}\nwhere $A \\approx 1.28$ is the Glaisher-Kinkelin constant and $\\gamma \\approx 0.58$ is the Euler-Mascheroni constant. Then we have\n\\begin{align*}\n\t& \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2]\\\\\n\t&\\leq \\frac{2(H(x_1,\\theta^c)- \\min_{x\\in\\mathcal{X}}H(x,\\theta^c))}{a M_T} +\\frac{C_1}{M_T}\\sum_{t=1}^T \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt^2} +\\sum_{t=1}^T \\frac{L_{h} a \\sigma^2}{M_T t^2}\\\\\n\t& \\leq \\left[\\frac{2(H(x_1,\\theta^c)- \\min_{x\\in\\mathcal{X}}H(x,\\theta^c))}{a} +\\frac{6C_1 + \\pi^2C_1 (\\log D - \\log \\delta)}{6D}+\\frac{\\pi^2 L_{h} a \\sigma^2}{6}\\right]\\frac{1}{\\log T}.\n\\end{align*}\n(iii) $a_t = \\frac{a}{\\sqrt{t}}$, $\\forall t \\leq T$, for some constant $a < \\frac{1}{L_{h}}$. Let $Q_t=\\sum_{t=1}^{T}\\frac{1}{\\sqrt{t}}$. Note that $\\sum_{t=1}^{\\infty}\\frac{1}{t\\sqrt{t}}=\\zeta(1.5)\\approx2.61<3$, $\\sum_{t=1}^{\\infty} \\frac{\\log t}{t\\sqrt{t}} < 4$, $\\sum_{t=1}^{T}\\frac{1}{\\sqrt{t}} \\geq \\sqrt{T}$, where $\\zeta(\\cdot)$ is the Riemann’s zeta function. Then we have\n{\\small\n\\begin{align*}\n\t& \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2]\\\\\n\t&\\leq \\frac{2(H(x_1,\\theta^c)- \\min_{x}H(x,\\theta^c))}{a Q_T} +\\frac{C_1(\\log D - \\log \\delta)}{D Q_T}\\sum_{t=1}^T \\frac{1}{t\\sqrt{t}} + \\frac{C_1}{D Q_T}\\sum_{t=1}^{T}\\frac{\\log t}{t \\sqrt{t}} +  \\frac{L_{h} a \\sigma^2}{Q_T} \\sum_{t=1}^T \\frac{1}{t}\\\\\n\t&\\leq \\left[\\frac{2(H(x_1,\\theta^c)- \\min_{x}H(x,\\theta^c))}{a\\sqrt{T}} + \\frac{3C_1(\\log D - \\log \\delta) + 4C_1}{D\\sqrt{T}} + \\frac{L_{h} a \\sigma^2}{\\sqrt{T}}\\right] + \\frac{L_{h} a \\sigma^2 \\log T}{\\sqrt{T}}.\n\\end{align*}\n}\n\\end{proof}","sha256_utf8":"6815777f787baceac5101626eb2d41ae90ff5000ffc4039738acbd0caf1e79ad"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"分解真实梯度与误差","source_units":[{"start":90807,"end":91241,"start_line":727,"end_line":733,"raw_tex":"\\begin{proof}\nBy the update \\cref{eq: SGD_independent_2}, we know that for any $t\\leq T$,\n\\begin{align*}\n    x_{t+1} &= x_{t} - a_{t} \\nabla_x H(x_{t},\\theta^c) -  a_{t}[\\mathbb{E}_{\\pi_t}\\nabla_x H(x_{t}, \\theta)-\\nabla_x H(x_{t}, \\theta^c)]\t\\\\\n    &~~~~~~~~- a_{t}[\\nabla_x h(x_{t}, \\xi_{t})-\\mathbb{E}_{\\pi_t}\\nabla_x H(x_{t}, \\theta)]\\\\\n    & =  x_{t} - a_{t}\t\\nabla_x H(x_{t},\\theta^c) -  a_{t} B_{t} - a_{t} N_{t},\n\\end{align*}\n","sha256_utf8":"084ecc9421ce4995332ae1783e29aeb7a09968eb8e1e21144f12846570627f42"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"建立单步期望下降界","source_units":[{"start":91241,"end":91820,"start_line":734,"end_line":737,"raw_tex":"where $B_{t}$ is the bias and $N_{t}$ is the noise. By \\cref{lemma: independent_concentration}, we know $\\mathbb{E}[\\|B_{t}\\|_2^2]\\leq  C_1 \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt}.$ From \\cref{ass: independent_rate} we have $\\mathbb{E}[\\|N_{t}\\|_2^2]\\leq \\sigma^2.$ By the proof of Lemma 2 in \\cite{ajalloeian2020analysis}, we know that \n\\begin{align*}\n    \\mathbb{E}[H(x_{t+1},\\theta^c)] - H(x_{t},\\theta^c)\\leq -\\frac{a_{t}}{2}\\|\\nabla_x H(x_{t},\\theta^c) \\|_2^2 +\\frac{a_{t}}{2}C_1 \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt} +\\frac{a_{t}^2}{2}L_{h} \\sigma^2,\n\\end{align*} \n","sha256_utf8":"d495258afa9eb3b360ba2192fefba775cd23f5c9238c610e584f626950a15475"}]},{"id":"S005","semantic_type":"proof_step","kind":"proof_step","title":"累加得到梯度加权界","source_units":[{"start":91820,"end":92290,"start_line":738,"end_line":745,"raw_tex":"Rearranging the terms in the inequality above, summing over $t$ from 1 to $T$, and noting that $H(x_{t}, \\theta^c) \\leq \\min_{x \\in \\mathcal{X}} H(x, \\theta^c), \\forall t$, we have \n{\\small\n\\begin{align*}\n    \\sum_{t=1}^{T} a_t \\mathbb{E}[\\|\\nabla_x H(x_{t},\\theta^c) \\|_2^2]\\leq 2\n    (H(x_1,\\theta^c) - \\min_{x \\in \\mathcal{X}} H(x,\\theta^c))\n    + C_1 \\sum_{t=1}^{T} a_t \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt} + L_{h}\\sigma^2 \\sum_{t=1}^{T} a_t^2,\n\\end{align*}\n}\n","sha256_utf8":"3892421cdd28eafddeb95aa5569c10006000e3878274ca353059a212ea282bb9"}]},{"id":"S006","semantic_type":"proof_step","kind":"proof_step","title":"转换为随机输出梯度界","source_units":[{"start":92290,"end":92865,"start_line":746,"end_line":756,"raw_tex":"Dividing both sides of the above inequality by $\\sum_{t=1}^{T} a_t$, and noting that \n\\begin{align*}\n    \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2] = \\frac{1}{\\sum_{t=1}^{T} a_t} \\sum_{t=1}^{T} a_t \\mathbb{E}[\\|\\nabla_x H(x_{t},\\theta^c) \\|_2^2],\n\\end{align*}\nwe have\n{\\small\n\\begin{align*}\n    \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2] \\leq \\frac{1}{\\sum_{t=1}^{T} a_t}\\left[2 (H(x_1,\\theta^c) - \\min_{x \\in \\mathcal{X}} H(x,\\theta^c)) +C_1 \\sum_{t=1}^{T} a_t \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt} + L_{h} \\sigma^2\\sum_{t=1}^{T}a_t^2.\n    \\right]\n\\end{align*}\n}\n","sha256_utf8":"5ca89bc4f46374aabe751af194191c0c01f4b81d8a25c8bee08c4475e1e41251"}]},{"id":"S007","semantic_type":"proof_step","kind":"proof_step","title":"计算固定步长收敛界","source_units":[{"start":92865,"end":93586,"start_line":757,"end_line":765,"raw_tex":"(i) $a_t=\\frac{a}{\\sqrt{T}}$, $\\forall t \\leq T$, for some constant $a<\\frac{\\sqrt{T}}{L_{h}}$. Note that $\\sum_{t=1}^{T} \\frac{1}{t} \\leq \\log T+1$ and $\\sum_{t=1}^{T} \\frac{\\log t}{t} \\leq \\log (\\log T+1)$.\nThen\n{\\small\n\\begin{align*}\n    & \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2]\\\\\n    &\\leq \\frac{2(H(x_1,\\theta^c)- \\min_{x}H(x,\\theta^c))}{a\\sqrt{T}} + \\frac{C_1 (\\log D - \\log \\delta) (\\log T + 1)}{L_{h} D T} + \\frac{C_1 \\log T (\\log T + 1)}{L_{h} D T} \\\\ \n    & = \\frac{2(H(x_1,\\theta^c)- \\min_{x}H(x,\\theta^c))}{a\\sqrt{T}} + \\frac{C_1 (\\log D - \\log \\delta)}{L_{h} D T} + \\frac{C_1 (\\log D - \\log \\delta) \\log T}{L_{h} D T} + \\frac{C_1 \\log^2 T}{L_{h} D T} + \\frac{L_{h} a\\sigma^2}{\\sqrt{T}}\n\\end{align*}\n}\n","sha256_utf8":"45fcbacb377f8ebda4386b2060463b00f13642c7d2bb3f42122670a2f9dcd6f9"}]},{"id":"S008","semantic_type":"proof_step","kind":"proof_step","title":"计算倒数步长收敛界","source_units":[{"start":93586,"end":94476,"start_line":766,"end_line":775,"raw_tex":"(ii) $a_t = \\frac{a}{t}$, $\\forall t \\leq T$, for some constant $a < \\frac{1}{L_{h}}$. Let $M_T=\\sum_{t=1}^{T}\\frac{1}{t}$. Note that \n\\begin{align*}\n    \\sum_{t=1}^{T}\\frac{\\log t}{t^2} < \\sum_{t=1}^{\\infty}\\frac{\\log t}{t^2} = \\frac{\\pi^2}{6}(12 \\ln A - \\gamma - \\ln 2\\pi) < 1,\n\\end{align*}\nwhere $A \\approx 1.28$ is the Glaisher-Kinkelin constant and $\\gamma \\approx 0.58$ is the Euler-Mascheroni constant. Then we have\n\\begin{align*}\n\t& \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2]\\\\\n\t&\\leq \\frac{2(H(x_1,\\theta^c)- \\min_{x\\in\\mathcal{X}}H(x,\\theta^c))}{a M_T} +\\frac{C_1}{M_T}\\sum_{t=1}^T \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt^2} +\\sum_{t=1}^T \\frac{L_{h} a \\sigma^2}{M_T t^2}\\\\\n\t& \\leq \\left[\\frac{2(H(x_1,\\theta^c)- \\min_{x\\in\\mathcal{X}}H(x,\\theta^c))}{a} +\\frac{6C_1 + \\pi^2C_1 (\\log D - \\log \\delta)}{6D}+\\frac{\\pi^2 L_{h} a \\sigma^2}{6}\\right]\\frac{1}{\\log T}.\n\\end{align*}\n","sha256_utf8":"36b90cab3d458c7cde3e8ed467407b0883595c140c64618766d4c2951558d9b1"}]},{"id":"S009","semantic_type":"proof_step","kind":"proof_step","title":"计算平方根步长收敛界","source_units":[{"start":94476,"end":95427,"start_line":776,"end_line":784,"raw_tex":"(iii) $a_t = \\frac{a}{\\sqrt{t}}$, $\\forall t \\leq T$, for some constant $a < \\frac{1}{L_{h}}$. Let $Q_t=\\sum_{t=1}^{T}\\frac{1}{\\sqrt{t}}$. Note that $\\sum_{t=1}^{\\infty}\\frac{1}{t\\sqrt{t}}=\\zeta(1.5)\\approx2.61<3$, $\\sum_{t=1}^{\\infty} \\frac{\\log t}{t\\sqrt{t}} < 4$, $\\sum_{t=1}^{T}\\frac{1}{\\sqrt{t}} \\geq \\sqrt{T}$, where $\\zeta(\\cdot)$ is the Riemann’s zeta function. Then we have\n{\\small\n\\begin{align*}\n\t& \\mathbb{E}[\\|\\nabla_x H(z_T,\\theta^c) \\|_2^2]\\\\\n\t&\\leq \\frac{2(H(x_1,\\theta^c)- \\min_{x}H(x,\\theta^c))}{a Q_T} +\\frac{C_1(\\log D - \\log \\delta)}{D Q_T}\\sum_{t=1}^T \\frac{1}{t\\sqrt{t}} + \\frac{C_1}{D Q_T}\\sum_{t=1}^{T}\\frac{\\log t}{t \\sqrt{t}} +  \\frac{L_{h} a \\sigma^2}{Q_T} \\sum_{t=1}^T \\frac{1}{t}\\\\\n\t&\\leq \\left[\\frac{2(H(x_1,\\theta^c)- \\min_{x}H(x,\\theta^c))}{a\\sqrt{T}} + \\frac{3C_1(\\log D - \\log \\delta) + 4C_1}{D\\sqrt{T}} + \\frac{L_{h} a \\sigma^2}{\\sqrt{T}}\\right] + \\frac{L_{h} a \\sigma^2 \\log T}{\\sqrt{T}}.\n\\end{align*}\n}\n\\end{proof}","sha256_utf8":"ccf83868f88220112f55f5ceb412c5b7079755cdf009e78f851f67d15c87a33a"}]},{"id":"S010","semantic_type":"lemma","kind":"lemma","title":"后验梯度偏差集中界","source_units":[{"start":42801,"end":43220,"start_line":343,"end_line":348,"raw_tex":"\\begin{lemma}\\label{lemma: independent_concentration}\nUnder \\cref{ass: independent_rate}, there exists a constant $C_1>0$ such that for any $\\delta>0$, with probability at least $1-\\delta$ we have\n\\begin{align*}\n\t||\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta)-\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta^c)||_2^2\\leq C_1 \\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt}, \\forall x\\in \\mathcal{X}, \\forall t>0. \n\\end{align*}\n\\end{lemma}","sha256_utf8":"d7427c66a5b94e4ee0a55f1a81e84d2b13f4f6df9cdcf39522020a069f941058"}]},{"id":"S011","semantic_type":"proof","kind":"proof","title":"偏差集中引理完整证明","source_units":[{"start":87357,"end":90737,"start_line":681,"end_line":723,"raw_tex":"\\begin{proof}\nDefine the Hellinger distance between $\\theta_1$ and $\\theta_2$ as\n\\begin{align*}\n    d(\\theta_1,\\theta_2) =\\sqrt{\\frac{1}{2}\\int_\\mathcal{Y} (\\sqrt{f(y;\\theta_1)}-\\sqrt{f(y;\\theta_2)})^2}.\n\\end{align*}\nOne can easily verify that there exist a constant $A$ such that $\\|\\theta_1-\\theta_2\\|\\leq Ad(\\theta_1,\\theta_2)$, where $||\\cdot||$ is the Euclidean norm. Let $B_k^t = B(\\theta^c, k/\\sqrt{Dt})$ be a ball centered at $\\theta^c$ with radius $k/\\sqrt{Dt}$ under distance $d.$ Since $\\Theta$ is finite, we can directly apply Proposition 1 in \\cite{birge2015non}. Then for $t\\leq T, \\epsilon,\\delta\\in(0,1)$ with probability at least $1-\\frac{6\\delta}{\\pi^2 t^2}$ with respect to $\\mathbb{P}_{\\theta^c}^{t}$, we have\n\\begin{align*}\n\t\\pi_t(B_{k(t)}^t) \\geq 1-\\epsilon,\n\\end{align*}\nwhere \n\\begin{align*}\n    k(t) =\\inf\\left\\{j\\geq 1\\Big| \\sum_{i\\geq j}|\\Theta|e^{-i^2}\\leq \\frac{6\\delta}{\\pi^2 t^2}\\sqrt{\\epsilon \\pi_0(\\theta^c)}\\right\\}.\n\\end{align*}\nNote that $\\sum_{i\\geq j}e^{-i^2}\\leq \\frac{e}{e-1} e^{-j^2}$, we can set $k(t)$ to be the solution of next equation. \n\\begin{align*}\n    \\frac{e}{e-1}|\\Theta| e^{-k(t)^2} =\\frac{6\\delta}{\\pi^2 t^2}\\sqrt{\\epsilon \\pi_0(\\theta^c)}.\n\\end{align*}\nBy simple calculation, we have $k(t) = \\sqrt{\\log \\frac{e|\\Theta|\\pi^2 t^2}{6\\delta(e-1)\\sqrt{\\epsilon\\pi_0(\\theta^c)}}}$. Now we are ready to bound the bias in the gradient estimator.\n\\begin{align*}\n\t& \\|\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta)-\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta^c)\\|_2^2 \\\\\n\t& = \\left\\|\\int (\\nabla_x H(x, \\theta)- \\nabla_x H(x, \\theta^c))\\pi_t(\\theta) d\\theta\\right\\|_2^2\\\\\n\t&\\leq \\int  \\left\\|(\\nabla_x H(x, \\theta)- \\nabla_x H(x, \\theta^c))\\right\\|_2^2\\pi_t(\\theta) d\\theta\\\\\n\t&\\leq  \\int L_{H}^2||\\theta-\\theta^c||_2^2 \\pi_t(\\theta) d\\theta\\\\\n\t&=  \\int_{B_{k(t)}^t} L_{H}^2||\\theta-\\theta^c||_2^2 \\pi_t(\\theta) d\\theta + \\int_{(B_{k(t)}^t)^c} L_{H}^2||\\theta-\\theta^c||_2^2 \\pi_t(\\theta) d\\theta\\\\\n\t&\\leq A^2 L_{H}^2 \\frac{k(t)^2}{Dt}  \\int_{B_{k(t)}^t} \\pi_t(\\theta) d\\theta + L_{H}^2 \\max_{\\theta\\in\\Theta}\\|\\theta-\\theta^c\\|^2_2\\int_{(B_{k(t)}^t)^c}  \\pi_t(\\theta) d\\theta\\\\\n\t&\\leq  A^2 L_{H}^2 \\frac{k(t)^2}{Dt} +  L_{H}^2 \\max_{\\theta\\in\\Theta}\\|\\theta-\\theta^c\\|^2_2\\epsilon.\n\\end{align*}\nRecall that $D$ is the data batch size. Take $\\epsilon =\\frac{1}{Dt}$, note that $k(t) =\\sqrt{\\log \\frac{e|\\Theta|\\pi^2 t^2\\sqrt{Dt}}{6\\delta(e-1)\\sqrt{\\pi_0(\\theta^c)}}}$. We further have\n\\begin{align*}\n\t\\|\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta)-\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta^c)\\|_2^2\n\t&\\leq  A^2 L_{H}^2 \\frac{k(t)^2}{Dt} +  L_{H}^2 \\max_{\\theta\\in\\Theta}\\|\\theta-\\theta^c\\|^2_2\\epsilon\\\\\n\t&\\leq 2A^2 L_{H}^2\\max_{\\theta\\in\\Theta}\\|\\theta-\\theta^c\\|^2_2\\frac{\\log \\frac{e|\\Theta|\\pi^2 t^2\\sqrt{Dt}}{6\\delta(e-1)\\sqrt{\\pi_0(\\theta^c)} }}{Dt}\\\\\n\t&= O(\\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt}).\n\\end{align*}\n\nLet $\\mathcal{E}_t$ denote the event that the above inequality holds, and $\\mathcal{E}_t^c$ denote the complement event. Then we have $\\mathbb{P}(\\mathcal{E}_t^c) \\leq \\frac{6\\delta}{\\pi^2 t^2}$. Therefore,\n\\begin{align*}\n    \\mathbb{P}(\\cap_{t=1}^\\infty \\mathcal{E}_t) & = 1- \\mathbb{P}(\\bigcup_{t=1}^{\\infty} \\mathcal{E}^c_t) \\\\\n    & \\geq 1 - \\sum_{t=1}^{\\infty} \\mathbb{P}(\\mathcal{E}_t^c) \\quad (\\text{union bound}) \\\\\n    & \\geq 1 - \\sum_{t=1}^{\\infty} \\frac{6\\delta}{\\pi^2 t^2} \\\\\n    & = 1 - \\delta.\n\\end{align*}\n\\end{proof}","sha256_utf8":"4a27bd1a8ca4e2700fb976b77c9b2144b5780ff251973c24ba5f954c2c530f3c"}]},{"id":"S012","semantic_type":"proof_step","kind":"proof_step","title":"定义参数间海林格距离","source_units":[{"start":87357,"end":87574,"start_line":681,"end_line":685,"raw_tex":"\\begin{proof}\nDefine the Hellinger distance between $\\theta_1$ and $\\theta_2$ as\n\\begin{align*}\n    d(\\theta_1,\\theta_2) =\\sqrt{\\frac{1}{2}\\int_\\mathcal{Y} (\\sqrt{f(y;\\theta_1)}-\\sqrt{f(y;\\theta_2)})^2}.\n\\end{align*}\n","sha256_utf8":"4e731b42f8ea641968858fae6782ff40701e98871d7652c1b18ee36b4158356f"}]},{"id":"S013","semantic_type":"proof_step","kind":"proof_step","title":"调用有限模型后验集中","source_units":[{"start":87574,"end":88321,"start_line":686,"end_line":693,"raw_tex":"One can easily verify that there exist a constant $A$ such that $\\|\\theta_1-\\theta_2\\|\\leq Ad(\\theta_1,\\theta_2)$, where $||\\cdot||$ is the Euclidean norm. Let $B_k^t = B(\\theta^c, k/\\sqrt{Dt})$ be a ball centered at $\\theta^c$ with radius $k/\\sqrt{Dt}$ under distance $d.$ Since $\\Theta$ is finite, we can directly apply Proposition 1 in \\cite{birge2015non}. Then for $t\\leq T, \\epsilon,\\delta\\in(0,1)$ with probability at least $1-\\frac{6\\delta}{\\pi^2 t^2}$ with respect to $\\mathbb{P}_{\\theta^c}^{t}$, we have\n\\begin{align*}\n\t\\pi_t(B_{k(t)}^t) \\geq 1-\\epsilon,\n\\end{align*}\nwhere \n\\begin{align*}\n    k(t) =\\inf\\left\\{j\\geq 1\\Big| \\sum_{i\\geq j}|\\Theta|e^{-i^2}\\leq \\frac{6\\delta}{\\pi^2 t^2}\\sqrt{\\epsilon \\pi_0(\\theta^c)}\\right\\}.\n\\end{align*}\n","sha256_utf8":"161bbd260e4d87e6be896a5bdb3185662388b76e8ec14d3f9411b58dfe8128d8"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"确定集中球半径","source_units":[{"start":88321,"end":88750,"start_line":694,"end_line":698,"raw_tex":"Note that $\\sum_{i\\geq j}e^{-i^2}\\leq \\frac{e}{e-1} e^{-j^2}$, we can set $k(t)$ to be the solution of next equation. \n\\begin{align*}\n    \\frac{e}{e-1}|\\Theta| e^{-k(t)^2} =\\frac{6\\delta}{\\pi^2 t^2}\\sqrt{\\epsilon \\pi_0(\\theta^c)}.\n\\end{align*}\nBy simple calculation, we have $k(t) = \\sqrt{\\log \\frac{e|\\Theta|\\pi^2 t^2}{6\\delta(e-1)\\sqrt{\\epsilon\\pi_0(\\theta^c)}}}$. Now we are ready to bound the bias in the gradient estimator.\n","sha256_utf8":"b223353339548adf42759d37c1aa6ebe687c97397a7d02670fcfad17aa8e20f3"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"分球内外控制梯度偏差","source_units":[{"start":88750,"end":89586,"start_line":699,"end_line":707,"raw_tex":"\\begin{align*}\n\t& \\|\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta)-\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta^c)\\|_2^2 \\\\\n\t& = \\left\\|\\int (\\nabla_x H(x, \\theta)- \\nabla_x H(x, \\theta^c))\\pi_t(\\theta) d\\theta\\right\\|_2^2\\\\\n\t&\\leq \\int  \\left\\|(\\nabla_x H(x, \\theta)- \\nabla_x H(x, \\theta^c))\\right\\|_2^2\\pi_t(\\theta) d\\theta\\\\\n\t&\\leq  \\int L_{H}^2||\\theta-\\theta^c||_2^2 \\pi_t(\\theta) d\\theta\\\\\n\t&=  \\int_{B_{k(t)}^t} L_{H}^2||\\theta-\\theta^c||_2^2 \\pi_t(\\theta) d\\theta + \\int_{(B_{k(t)}^t)^c} L_{H}^2||\\theta-\\theta^c||_2^2 \\pi_t(\\theta) d\\theta\\\\\n\t&\\leq A^2 L_{H}^2 \\frac{k(t)^2}{Dt}  \\int_{B_{k(t)}^t} \\pi_t(\\theta) d\\theta + L_{H}^2 \\max_{\\theta\\in\\Theta}\\|\\theta-\\theta^c\\|^2_2\\int_{(B_{k(t)}^t)^c}  \\pi_t(\\theta) d\\theta\\\\\n\t&\\leq  A^2 L_{H}^2 \\frac{k(t)^2}{Dt} +  L_{H}^2 \\max_{\\theta\\in\\Theta}\\|\\theta-\\theta^c\\|^2_2\\epsilon.\n\\end{align*}\n","sha256_utf8":"9eeeb7614a03e5295c2be7d210ac37be62cb65206eea5f6d94bfdcf8e14fbf7f"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"选择尾质量获得偏差阶","source_units":[{"start":89586,"end":90204,"start_line":708,"end_line":714,"raw_tex":"Recall that $D$ is the data batch size. Take $\\epsilon =\\frac{1}{Dt}$, note that $k(t) =\\sqrt{\\log \\frac{e|\\Theta|\\pi^2 t^2\\sqrt{Dt}}{6\\delta(e-1)\\sqrt{\\pi_0(\\theta^c)}}}$. We further have\n\\begin{align*}\n\t\\|\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta)-\\mathbb{E}_{\\pi_t}\\nabla_x H(x, \\theta^c)\\|_2^2\n\t&\\leq  A^2 L_{H}^2 \\frac{k(t)^2}{Dt} +  L_{H}^2 \\max_{\\theta\\in\\Theta}\\|\\theta-\\theta^c\\|^2_2\\epsilon\\\\\n\t&\\leq 2A^2 L_{H}^2\\max_{\\theta\\in\\Theta}\\|\\theta-\\theta^c\\|^2_2\\frac{\\log \\frac{e|\\Theta|\\pi^2 t^2\\sqrt{Dt}}{6\\delta(e-1)\\sqrt{\\pi_0(\\theta^c)} }}{Dt}\\\\\n\t&= O(\\frac{\\log Dt + \\log \\frac{1}{\\delta}}{Dt}).\n\\end{align*}\n","sha256_utf8":"b4dbca629520c272ea0c02346e34d02d834ce103be5fc8c2da1ce4e82c439b36"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"联合界保证全时域成立","source_units":[{"start":90204,"end":90737,"start_line":715,"end_line":723,"raw_tex":"\nLet $\\mathcal{E}_t$ denote the event that the above inequality holds, and $\\mathcal{E}_t^c$ denote the complement event. Then we have $\\mathbb{P}(\\mathcal{E}_t^c) \\leq \\frac{6\\delta}{\\pi^2 t^2}$. Therefore,\n\\begin{align*}\n    \\mathbb{P}(\\cap_{t=1}^\\infty \\mathcal{E}_t) & = 1- \\mathbb{P}(\\bigcup_{t=1}^{\\infty} \\mathcal{E}^c_t) \\\\\n    & \\geq 1 - \\sum_{t=1}^{\\infty} \\mathbb{P}(\\mathcal{E}_t^c) \\quad (\\text{union bound}) \\\\\n    & \\geq 1 - \\sum_{t=1}^{\\infty} \\frac{6\\delta}{\\pi^2 t^2} \\\\\n    & = 1 - \\delta.\n\\end{align*}\n\\end{proof}","sha256_utf8":"50d18b8243b1527ad271741b170d7f32e6a2d55ac2bfb574d0c5ee101a59ecbc"}]},{"id":"S018","semantic_type":"lemma","kind":"lemma","title":"后验目标梯度估计无偏","source_units":[{"start":34813,"end":35078,"start_line":277,"end_line":279,"raw_tex":"\\begin{lemma}\\label{lemma: independent_unbiasedness}\nUnder \\cref{ass: independent_DCT}, $\\nabla_{x} h(x, \\xi)$ with $\\xi \\sim f(\\cdot;\\theta)$ and $\\theta \\sim \\pi_t$ is an unbiased gradient estimator of the objective function in \\cref{eq: independent}.\n\\end{lemma}","sha256_utf8":"ca69a3ebc12078386cc119337948f9b81fcd743487ff5f3955f2415444b12c93"}]},{"id":"S019","semantic_type":"proof","kind":"proof","title":"梯度无偏性完整证明","source_units":[{"start":35080,"end":35865,"start_line":281,"end_line":288,"raw_tex":"\\begin{proof}\nFor every fixed $x\\in\\mathcal{X}$, \n\\begin{align*}\n    \\mathbb{E}_{\\pi_{t}}\\left[\\mathbb{E}_{f(\\cdot;\\theta)} [\\nabla_x h(x, \\xi)]\\right] & = \\mathbb{E}_{\\pi_{t}}\\left[\\nabla_{x} \\mathbb{E}_{f(\\cdot;\\theta)} [h(x, \\xi)]\\right]\\\\\n    & = \\nabla_{x} \\mathbb{E}_{\\pi_{t}}\\left[\\mathbb{E}_{f(\\cdot;\\theta)} [h(x, \\xi)]\\right],\n\\end{align*}\nwhere the first equality holds because the gradient $\\nabla_x h(x,\\xi)$ is Lipschitz continuous, and the interchange between expectation and differentiation is justified by dominated convergence theorem (DCT). Similarly, the second equality above is again justified by DCT. Therefore, the proposed estimator in \\cref{eq: SGD_independent_2} is unbiased gradient estimator of the objective function in \\cref{eq: independent}.\n\\end{proof}","sha256_utf8":"860b262f44508f936642df8aaed976795bc0465a41aeb61b686e4326f3932031"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"交换两层期望与微分","source_units":[{"start":35080,"end":35430,"start_line":281,"end_line":286,"raw_tex":"\\begin{proof}\nFor every fixed $x\\in\\mathcal{X}$, \n\\begin{align*}\n    \\mathbb{E}_{\\pi_{t}}\\left[\\mathbb{E}_{f(\\cdot;\\theta)} [\\nabla_x h(x, \\xi)]\\right] & = \\mathbb{E}_{\\pi_{t}}\\left[\\nabla_{x} \\mathbb{E}_{f(\\cdot;\\theta)} [h(x, \\xi)]\\right]\\\\\n    & = \\nabla_{x} \\mathbb{E}_{\\pi_{t}}\\left[\\mathbb{E}_{f(\\cdot;\\theta)} [h(x, \\xi)]\\right],\n\\end{align*}\n","sha256_utf8":"23159d36091bffaa169b823a246bcfd42ce9ce097f65751e8a06fc6b5c025f89"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"支配收敛论证无偏性","source_units":[{"start":35430,"end":35865,"start_line":287,"end_line":288,"raw_tex":"where the first equality holds because the gradient $\\nabla_x h(x,\\xi)$ is Lipschitz continuous, and the interchange between expectation and differentiation is justified by dominated convergence theorem (DCT). Similarly, the second equality above is again justified by DCT. Therefore, the proposed estimator in \\cref{eq: SGD_independent_2} is unbiased gradient estimator of the objective function in \\cref{eq: independent}.\n\\end{proof}","sha256_utf8":"61bf876f6d978381316c37f17b17626d9a71b98f8b15a2d41b7148d07357fa01"}]},{"id":"S022","semantic_type":"problem_setting","kind":"problem_setting","title":"真实随机优化问题与变量","source_units":[{"start":18865,"end":19040,"start_line":133,"end_line":135,"raw_tex":"\\begin{align}\\label{eq: obj-independent}\n    \\min _{x \\in \\mathcal{X}} H(x):=\\mathbb{E}_{f(\\cdot;\\theta^c)}[h(x, \\xi)] ~~~\\text{(decision-independent uncertainty)}\n\\end{align}","sha256_utf8":"f393888613ac4c7c6c1a171817bd18c12dc50a43e7f071a2e510e6e8c947619f"},{"start":19216,"end":20194,"start_line":139,"end_line":139,"raw_tex":"where $x \\in \\mathcal{X} \\subset \\mathbb{R}^d$ is the decision vector, $\\xi \\in \\Xi \\subset \\mathbb{R}^m$ is a random vector, $h: \\mathbb{R}^d \\times \\mathbb{R}^m \\to \\mathbb{R}$ is a deterministic function. The expectation is taken with respect to (w.r.t.) the distribution of $\\xi$, which is denoted as $f(\\cdot;\\theta^c)$ in the decision-independent case, and as $f(\\cdot;x,\\theta^c)$ in the decision-dependent case. The density function $f(\\cdot;x,\\theta^c)$ takes a general form, where the parameter $\\theta^c$ does not depend on $x$. For example, $f(\\xi;x,\\theta^c)=\\theta^c x \\exp (-\\theta^c x \\xi)$ is the density function of the exponential distribution with rate $\\theta^c x$. More assumptions on the density function will be discussed in \\cref{sec: convergence}. We assume the distribution of $\\xi$ belongs to a parameterized family of distributions with parameter set $\\Theta\\subset \\mathbb{R}^l$, and let $\\theta^c$ be the true parameter value of the distribution.\n","sha256_utf8":"bc6363f5497884322584df1e1ab2342457b337f8e6b9f6044d2452b4a832fd87"}]},{"id":"S023","semantic_type":"definition","kind":"definition","title":"独立数据与贝叶斯更新","source_units":[{"start":21431,"end":22254,"start_line":148,"end_line":151,"raw_tex":"We now discuss the details of these two steps in the following. Let's first focus on the decision-independent case. Suppose at each time stage $t$ we observe a batch of data $\\mathbf{y}_t = \\{y_{t,j}, j=1,\\ldots,D\\}$, where $\\{y_{t,j}\\}$ are i.i.d. according to $f(\\cdot;\\theta^c)$ and $D$ is the batch size. By viewing the unknown distributional parameter as a random vector $\\theta$ and assuming a prior distribution $\\pi_0$ on $\\theta$, the posterior distribution of $\\theta$ is updated by the Bayes rule as follows:\n\\begin{align}\\label{eq:independent-posterior}\n    \\pi_t(\\theta) = \\frac{\\pi_{t-1}(\\theta)f(\\mathbf{y}_t;\\theta)}{\\int\\pi_{t-1}(\\theta)f(\\mathbf{y}_t;\\theta)d\\theta} =  \\frac{\\pi_{t-1}(\\theta) \\prod_{j=1}^{D} f(y_{t,j};\\theta)}{\\int\\pi_{t-1}(\\theta) \\prod_{j=1}^{D} f(y_{t,j};\\theta)d\\theta}.\n\\end{align}","sha256_utf8":"b2a586a467d71f70852dbc568be8f1b8c770e04e07982b975eced97a80d2da5b"}]},{"id":"S024","semantic_type":"definition","kind":"definition","title":"参数化期望目标函数","source_units":[{"start":22255,"end":22444,"start_line":152,"end_line":155,"raw_tex":"The objective function \\eqref{eq: obj-independent} can be viewed as a function of $\\theta$, so we define the following function\n$$\nH(x,\\theta):= \\mathbb{E}_{f(\\cdot;\\theta)}[h(x, \\xi)]. \n$$","sha256_utf8":"eed4acb283dcb3c58e519be571fa99ced4932b992882bfb38c8e1becb75163de"}]},{"id":"S025","semantic_type":"definition","kind":"definition","title":"后验平均优化目标","source_units":[{"start":22445,"end":22699,"start_line":156,"end_line":160,"raw_tex":"To estimate the true objective function \\eqref{eq: obj-independent}, we consider the Bayesian average of the objective function:\n\\begin{align}\n    \\min_{x \\in \\mathcal{X}} \\mathbb{E}_{\\pi_{t}}\\left[H(x, \\theta)\\right],\n\\label{eq: independent}\n\\end{align}","sha256_utf8":"66ce8605f2f7470e417895359802b3cef713d1aaefdd73d7f9faa9042a80d270"}]},{"id":"S026","semantic_type":"algorithm","kind":"algorithm","title":"独立数据贝叶斯随机梯度法","source_units":[{"start":26938,"end":28315,"start_line":187,"end_line":212,"raw_tex":"\\begin{algorithm}\n\\caption{Bayesian-SGD (decision-independent uncertainty)}\n\\label{algorithm: independent}\n\\begin{algorithmic}\n\\STATE{\\textbf{input}: data batch size $D$, number of SGD iterations $K$, step size sequence $\\{a_{t,j}, t=1,2,\\ldots; j=0,\\ldots,K-1\\}$, time horizon $T$.}\n\\STATE{\\textbf{initialization}: choose an initial decision $x_1$ and prior distribution $\\pi_0(\\theta)$.}\n\\FOR{$t =1: T$}\n    \\STATE{-A batch of data $y_{t,1},\\cdots,y_{t,D} \\iid f(\\cdot; \\theta^c)$ arrives;}\n    \\STATE{-Posterior Update: compute $\\pi_t(\\theta)$ according to \\eqref{eq:independent-posterior}. }\n    \\STATE{-Decision Update:\n        \\begin{itemize}\n            \\item set $x_{t,0}:=x_{t}$;\n            \\item for $j=0,\\cdots,K-1$, draw sample $\\theta_{t,j} \\sim \\pi_t(\\theta)$ and $\\xi_{t,j} \\sim f(\\cdot;\\theta_{t,j})$, and carry out SGD iteration:\n        \\end{itemize}\n        \\begin{align}\\label{eq: SGD_independent}\n            x_{t,j+1}:=\\operatorname{Proj}_{\\mathcal{X}}\\left\\{x_{t,j}-a_{t,j}\\nabla_x h(x_{t,j},\\xi_{t,j})\\right\\},\n        \\end{align}\n        ~~~~~~~~~~~~~~where $\\operatorname{Proj}_{\\mathcal{X}}$ is a projection operator that projects the iterate  to the  set $\\mathcal{X}$. \n        \\begin{itemize}\n            \\item set the updated decision as $x_{t+1}:=x_{t,K}$;\n        \\end{itemize}\n    }\n\\ENDFOR\n\\RETURN $x_{T+1}$\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"d9f2a5896b213f5ac495ff6023e022639b03ff30cbe468e3578eb57235e3f2d6"}]},{"id":"S027","semantic_type":"notation","kind":"notation","title":"观测概率空间与滤过","source_units":[{"start":30476,"end":31909,"start_line":248,"end_line":248,"raw_tex":"Let's first consider the decision-independent case. The probability space is constructed as follows. Define the Bayesian prior $\\pi_0$ on $(\\Theta, \\mathcal{B}_{\\Theta})$, where $\\mathcal{B}_{\\Theta}$ is the Borel $\\sigma$-algebra on $\\Theta$. Let $\\mathcal{Y} \\subset \\mathbb{R}^{m}$ denote the data (observation) space. The data $y$ takes value in $\\mathcal{Y}$ equipped with a Borel $\\sigma$-algebra $\\mathcal{B}_{\\mathcal{Y}}$ and a probability measure $\\left\\{\\mathbb{P}_{\\theta^c}\\right\\}$, such that $\\mathbb{P}_{\\theta^c}(y \\in A) = \\int_{A} f\\left(y; \\theta^c\\right) d y, \\forall A \\in \\mathcal{B}(\\mathcal{Y})$. For the sequence $y_1,y_2,\\cdots,y_n \\iid f(\\cdot;\\theta^c)$, the probability measure is denoted by $\\mathbb{P}_{\\theta^c}^{n}$. As for the infinite sequence $\\{y_1,y_2,\\ldots\\}$, the probability measure $\\mathbb{P}_{\\theta^c}^{\\infty}$ can be constructed by Kolmogorov's extension theorem (cf. Theorem A.3.1 in \\cite{durrett2019probability}). In the following, w.p.1 (or almost surely) means that the considered property holds with probability one w.r.t. the probability measure $\\mathbb{P}_{\\theta^c}^\\infty$. Finally, let $\\mathcal{F}_{t}:=\\sigma\\left\\{\\left(y_{\\tau} \\right), \\tau \\leq t\\right\\}$ be the $\\sigma$-filtration generated by the data. We have the convergence of the posterior distribution $\\{\\pi_t\\}$ that is updated according to \\cref{eq:independent-posterior} under the following assumptions.\n","sha256_utf8":"a923ba99c9a3abbf8873d64ff060e17d344ede27598bf5eccc09c4ebb107919f"}]},{"id":"S028","semantic_type":"assumption","kind":"assumption","title":"先验与观测模型正则条件","source_units":[{"start":31910,"end":32474,"start_line":250,"end_line":252,"raw_tex":"\\begin{assumption}[\\cite{shapiro2021bayesian}, Assumption 3.1] (i) The set $\\Theta$ is convex and compact with non-empty interior. (ii) $\\ln \\pi_0(\\theta)$ is bounded on $\\Theta$. (iii) $f(\\xi|\\theta) > 0$ for all $\\xi \\in \\Xi$ and $\\theta \\in \\Theta$. (iv) $f(\\xi|\\theta)$ is continuous in $\\theta \\in \\Theta$. (v) $\\ln f(\\xi|\\theta), \\theta \\in \\Theta$ is dominated by an integrable (w.r.t. $\\xi \\sim f(\\cdot;\\theta^c)$) function. (vi) The data batches are i.i.d. over time from the fixed distribution $f(\\cdot;\\theta^c)$. \n\\label{ass: Alex2022}\n\\end{assumption}","sha256_utf8":"1e6af3fb0b10b54f3bfe7e0e904d5eb256af45269a5371cbb1486f727893cfa2"}]},{"id":"S029","semantic_type":"algorithm","kind":"algorithm","title":"单次梯度更新与投影项","source_units":[{"start":33249,"end":33950,"start_line":265,"end_line":268,"raw_tex":"We then study the asymptotic behavior of \\cref{algorithm: independent} by the ordinary differential equation (ODE) method (please refer to \\cite{Kushner2003} for a detailed exposition on the ODE method for stochastic approximation). The main idea is that SGD can be viewed as a noisy discretization of an ODE. Under certain conditions, the noise in SGD averages out asymptotically, such that the SGD iterates converge to the solution trajectory of the ODE. For simplicity, we consider the case where $K=1$ and rewrite the SGD iteration \\cref{eq: SGD_independent} as\n\\begin{equation}\\label{eq: SGD_independent_2}\n    x_{t+1}=x_{t}-a_{t} \\nabla_{x} h\\left(x_{t}, \\xi_{t}\\right) + a_t z_t,\n\\end{equation}","sha256_utf8":"47c5325daa4cc0fb9ae7b1bdb6d53811ec657b8c2afdf1930ea5918336685490"},{"start":33951,"end":34250,"start_line":269,"end_line":269,"raw_tex":"where $a_t z_t$ is the projection term, i.e., the vector of shortest Euclidean length needed to keep the decision $x_{t+1}$ from leaving the decision space $\\mathcal{X}$. We first show that under certain mild conditions, the proposed gradient estimator in \\cref{eq: SGD_independent_2} is unbiased. \n","sha256_utf8":"0eee459d57278856aff91263b1e32a826c1a756a5c51d0c6713e0fea6161b58c"}]},{"id":"S030","semantic_type":"assumption","kind":"assumption","title":"目标光滑性与梯度条件","source_units":[{"start":34251,"end":34473,"start_line":271,"end_line":273,"raw_tex":"\\begin{assumption}\\label{ass: independent_DCT}\n$h(x,\\xi)$ is $C^{1}$-smooth in $x$ for all $\\xi \\in \\Xi$, and the map $\\xi \\to \\nabla_x h(x,\\xi)$ is $L_h$-Lipschitz continuous for any $x \\in \\mathcal{X}$. \n\\end{assumption}","sha256_utf8":"c272fb2f2ee43eca9e341cb067a5c3dbc4bf4b203cb00169c7cb5924ff64f0dc"}]},{"id":"S031","semantic_type":"assumption","kind":"assumption","title":"步长与决策空间条件","source_units":[{"start":35867,"end":36231,"start_line":290,"end_line":296,"raw_tex":"\\begin{assumption}\\label{ass: independent_step}\n~\n\\begin{itemize}\n    \\item The step size $\\{a_t\\}$ satisfies $\\sum_{t=1}^{\\infty} a_{t}^{2}<\\infty$, $\\sum_{t=1}^{\\infty} a_{t}=\\infty$, $\\lim_{t \\to \\infty} a_{t} = 0$, $a_t > 0, \\forall t > 0$.\n    \\item The decision space $\\mathcal{X} \\subset \\mathbb{R}^{d}$ is compact and convex.\n\\end{itemize}\n\\end{assumption}","sha256_utf8":"2336a68eb91a04989b5b40d3ddac6d4f2b26c9641857769e3a0d6a002410df6d"}]},{"id":"S032","semantic_type":"assumption","kind":"assumption","title":"有限参数与梯度方差条件","source_units":[{"start":41232,"end":41842,"start_line":330,"end_line":337,"raw_tex":"\\begin{assumption}\\label{ass: independent_rate}\n~\n\\begin{itemize}\n    \\item The parameter space $\\Theta$ is finite, i.e., $\\Theta = \\{\\theta_1,\\ldots,\\theta_k\\}$. Moreover, $\\theta^c \\in \\Theta$.\n    \\item There exists $0<L_{H}<\\infty$ such that $||\\nabla_x H(x,\\theta_1)-\\nabla_x H(x,\\theta_2)||_2 \\leq L_{H}||\\theta_1-\\theta_2||_2$ for all $\\theta_1,\\theta_2 \\in \\Theta$ and for all $x \\in \\mathcal{X}$.\n    \\item Sampling variance is bounded by $\\sigma^2$, i.e., $\\mathbb{E}[||\\nabla_x h(x,\\xi) - \\nabla_x H(x,\\theta)||_2^2|\\theta] \\leq \\sigma^2$, for all $\\theta \\in \\Theta$.\n\\end{itemize}\n\\end{assumption}","sha256_utf8":"e363e793d60671f486b1f7d5e3ade900f3505275daf87b270cfb8f901448eb23"}]},{"id":"S033","semantic_type":"algorithm","kind":"algorithm","title":"按步长随机选取输出","source_units":[{"start":43222,"end":44009,"start_line":350,"end_line":350,"raw_tex":"The proof of \\cref{lemma: independent_concentration} can be found in \\cref{appendix: A}. Next, we show the convergence rate of \\cref{algorithm: independent}. To simplify the analysis and also be consistent with the convergence analysis of smooth non-convex SGD, we consider a variant of SGD where the final output is randomly chosen as follows: let $z_T = x_{t}$ with probability $\\frac{a_t}{\\sum_{t=1}^{T}a_t}$, $t=1,\\cdots,T$. The randomization scheme helps with the analysis of the expected gradient of the final output under the true parameter $\\theta^c$, and has been widely used in the smooth non-convex SGD literature (cf. \\cite{ghadimi2013stochastic}). We then have the following theorem giving the convergence rate of the randomized output algorithm under different step sizes.\n","sha256_utf8":"db401b7c42a3c6d0adefc1a1073dc582eeb48a7f380b85ccccf63c4bb7960ad4"}]},{"id":"S034","semantic_type":"assumption","kind":"assumption","title":"收敛率分析取消投影","source_units":[{"start":41040,"end":41231,"start_line":328,"end_line":328,"raw_tex":"Next, we investigate the convergence rate of \\cref{algorithm: independent} for the unconstrained case, i.e., without the projection term $a_t z_t$ under the following additional assumptions.\n","sha256_utf8":"49adbab7936fcd526b0041280554e96f8c8088bceed7296e5e9ccc0255c0cc4f"}]},{"id":"S035","semantic_type":"external_fact","kind":"external_fact","title":"支配收敛与微分积分交换","source_units":[],"declared_external":true,"declaration_text":["在相应可积支配条件下，支配收敛定理允许交换极限与积分，并可用于论证作者 L284–L287 中的期望与微分交换。"]},{"id":"S036","semantic_type":"external_fact","kind":"external_fact","title":"有限模型后验集中估计","source_units":[],"declared_external":true,"declaration_text":["birge2015non 的 Proposition 1 提供有限参数模型的后验集中估计；作者在 L686–L693 调用它，以概率至少 1−6δ/(π²t²) 保证 π_t(B_{k(t)}^t)≥1−ε，其中 k(t) 按 L691–L693 定义。"]},{"id":"S037","semantic_type":"external_fact","kind":"external_fact","title":"带偏差随机梯度下降估计","source_units":[],"declared_external":true,"declaration_text":["ajalloeian2020analysis 的 Lemma 2 证明提供带偏差随机梯度的单步期望下降估计；作者在 L734–L737 调用的形式包含 −a_t/2 倍真实梯度平方、a_t/2 倍偏差平方上界，以及 a_t²L_h/2 倍采样方差。"]},{"id":"S038","semantic_type":"external_fact","kind":"external_fact","title":"无限观测序列概率测度","source_units":[],"declared_external":true,"declaration_text":["Kolmogorov 扩张定理保证相容的有限维观测分布可以扩张为无限观测序列上的概率测度；作者在 L248 引用 durrett2019probability 的 Theorem A.3.1。"]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S008","kind":"supports"},{"source_id":"S006","target_id":"S009","kind":"supports"},{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S006","target_id":"S002","kind":"supports"},{"source_id":"S007","target_id":"S002","kind":"supports"},{"source_id":"S008","target_id":"S002","kind":"supports"},{"source_id":"S009","target_id":"S002","kind":"supports"},{"source_id":"S010","target_id":"S004","kind":"supports"},{"source_id":"S011","target_id":"S010","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S011","kind":"supports"},{"source_id":"S013","target_id":"S011","kind":"supports"},{"source_id":"S014","target_id":"S011","kind":"supports"},{"source_id":"S015","target_id":"S011","kind":"supports"},{"source_id":"S016","target_id":"S011","kind":"supports"},{"source_id":"S017","target_id":"S011","kind":"supports"},{"source_id":"S018","target_id":"S004","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S019","kind":"supports"},{"source_id":"S022","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S026","kind":"supports"},{"source_id":"S023","target_id":"S013","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S020","kind":"supports"},{"source_id":"S024","target_id":"S003","kind":"supports"},{"source_id":"S024","target_id":"S015","kind":"supports"},{"source_id":"S025","target_id":"S020","kind":"supports"},{"source_id":"S026","target_id":"S029","kind":"supports"},{"source_id":"S026","target_id":"S003","kind":"supports"},{"source_id":"S026","target_id":"S004","kind":"supports"},{"source_id":"S027","target_id":"S013","kind":"supports"},{"source_id":"S027","target_id":"S017","kind":"supports"},{"source_id":"S028","target_id":"S013","kind":"supports"},{"source_id":"S028","target_id":"S001","kind":"supports"},{"source_id":"S029","target_id":"S003","kind":"supports"},{"source_id":"S029","target_id":"S021","kind":"supports"},{"source_id":"S030","target_id":"S021","kind":"supports"},{"source_id":"S030","target_id":"S004","kind":"supports"},{"source_id":"S031","target_id":"S001","kind":"supports"},{"source_id":"S032","target_id":"S013","kind":"supports"},{"source_id":"S032","target_id":"S015","kind":"supports"},{"source_id":"S032","target_id":"S004","kind":"supports"},{"source_id":"S033","target_id":"S006","kind":"supports"},{"source_id":"S034","target_id":"S003","kind":"supports"},{"source_id":"S035","target_id":"S021","kind":"supports"},{"source_id":"S036","target_id":"S013","kind":"supports"},{"source_id":"S037","target_id":"S004","kind":"supports"},{"source_id":"S038","target_id":"S027","kind":"supports"}]}}
{"record_id":"2203.05276v4","paper":{"title":"Constrained composite optimization and augmented Lagrangian methods","authors":["Alberto De Marchi","Xiaoxi Jia","Christian Kanzow","Patrick Mehlitz"],"arxiv_id":"2203.05276","version":"v4","journal":"Mathematical Programming","doi":"10.1007/s10107-022-01922-4","abstract_url":"https://arxiv.org/abs/2203.05276v4","pdf_url":"https://arxiv.org/pdf/2203.05276v4"},"context":{"raw_tex":"\\newcommand{\\XX}{\\R^n}\n\\newcommand{\\YY}{\\R^m}\n\n\n\\begin{equation}\n    \\minimize_{x}\n    {}\\quad{}\n    q(x) \\coloneqq f(x) + g(x)\n    {}\\qquad{}\n    \\stt\n    {}\\quad{}\n    c(x) \\in D ,\n    \\tag{P}\\label{eq:P}\n\\end{equation}\n\nwhere $x$ is the decision variable, $f$ and $c$ are smooth functions, $g$ is proper and lower semicontinuous, and $D$ is a nonempty closed set.\n\n\n\\begin{ass}\\label{ass:P}\n\t\tThe following hold in \\eqref{eq:P}:\n\t\t\\begin{enumerate}\n\t\t\t\\item\\label{ass:f}%\n\t\t\t\t$\\func{f}{\\XX}{\\R}$ and $\\func{c}{\\XX}{\\YY}$ are continuously differentiable with locally Lipschitz continuous derivatives;\n\t\t\t\\item\\label{ass:g}%\n\t\t\t\t\\(\\func{g}{\\XX}{\\Rinf}\\) is proper, lower semicontinuous and prox-bounded;\n\t\t\t\\item\\label{ass:set}%\n\t\t\t\t\\(D\\subset \\YY\\) is a nonempty and closed set.\n\t\t\\end{enumerate}\n\t\\end{ass}\n\nThe effective domain of an extended real-valued function $\\func{h}{\\R^n}{\\Rinf}$ is denoted by $\\dom h \\coloneqq \\{x\\in\\R^n \\suchthat h(x) < \\infty \\}$.\nWe say that $h$ is \\emph{proper} if $\\dom h \\neq \\emptyset$\nand \\emph{lower semicontinuous} (lsc) if $h(\\bar{x}) \\leq \\liminf_{x\\to\\bar{x}} h(x)$ for all $\\bar{x} \\in \\R^n$.\n\n\n% subgradient\nGiven a proper and lsc function $\\func{h}{\\XX}{\\Rinf}$ and a point $\\bar{x}\\in\\dom h$, we may avoid to assume $h$ continuous and instead appeal \nto $h$-\\emph{attentive} convergence of a sequence $\\{ x^k \\}$:\n\\begin{equation}\nx^k \\toattentive{h} \\bar{x}\n{}\\quad:\\Leftrightarrow\\quad{}\nx^k \\to \\bar{x}\n{}\\quad\\text{with}\\quad{}\nh(x^k) \\to h(\\bar{x}) .\n\\end{equation}\nFollowing \\cite[Def. 8.3]{rockafellar1998variational}, we denote by $\\ffunc{\\hat{\\partial} h}{\\XX}{\\XX}$ the \\emph{regular subdifferential} of $h$, where\n\\begin{equation}\nv \\in \\hat{\\partial} h(\\bar{x})\n{}\\quad:\\Leftrightarrow\\quad{}\n\\liminf_{\\substack{x\\to\\bar{x}\\\\x\\neq\\bar{x}}} \\frac{h(x) - h(\\bar{x}) - \\langle v, x-\\bar{x}\\rangle}{\\|x-\\bar{x}\\|} \\geq 0 .\n\\end{equation}\nThe (limiting) \\emph{subdifferential} of $h$ is $\\ffunc{\\partial h}{\\XX}{\\XX}$, where $v \\in \\partial h(\\bar{x})$ if and only if there exist sequences $\\{x^k\\}$ and $\\{v^k\\}$ such that $x^k \\toattentive{h} \\bar{x}$ and $v^k \\in \\hat{\\partial} h(x^k)$ with $v^k \\to v$.\nThe subdifferential of $h$ at $\\bar{x}$ satisfies $\\partial(h+h_0)(\\bar{x}) = \\partial h(\\bar{x}) + \\nabla h_0(\\bar{x})$ for any $\\func{h_0}{\\XX}{\\Rinf}$ continuously differentiable around $\\bar{x}$ \\cite[Ex. 8.8]{rockafellar1998variational}.\nFor formal completeness, we set $\\hat{\\partial} h(\\bar{x})\\coloneqq\\partial h(\\bar{x})\\coloneqq\\emptyset$ for each $\\bar{x}\\notin\\dom h$.\n\n\nFurthermore, we say that $x^\\ast \\in \\XX$ is \\emph{$\\varepsilon$-stationary} for some $\\varepsilon \\geq 0$ if\n\\begin{equation}\n\t\\exists \\eta \\in \\partial h(x^\\ast) \\colon \\|\\eta\\| \\leq \\varepsilon .\n\t\\label{eq:epsstationary}\n\\end{equation}\n\n\nIf $S(\\bar{x})$ is nonempty, we define the \\emph{outer limit} of $S$ at $\\bar{x}$\nby means of\n\\begin{align*}\n\t\\limsup\\limits_{x\\to\\bar{x}} S(x)\n\t\\coloneqq\n\t\\{\n\t\ty\\in\\YY\n\t\t\\suchthat\n\t\t\\exists x^k\\to\\bar{x},\\,\n\t\t\\exists y^k\\to y,\\,\n\t\ty^k\\in S(x^k)\\,\\forall k\\in\\N\n\t\\}\n\\end{align*}\nand note that this is a closed superset of $S(\\bar{x})$ by definition.\n\n\n% Prox-boundedness\nand we say that $h$ is \\emph{prox-bounded} if it is proper and $h + \\|\\cdot\\|^2 / (2\\gamma)$ is bounded below on $\\XX$ for some $\\gamma > 0$.\nThe supremum of all such $\\gamma$ is the threshold $\\gamma_h$ of prox-boundedness for $h$.\n\n\nThe \\emph{projection} mapping $\\proj_D$ and the \\emph{distance} function $\\dist_D$ are defined by\n\\begin{equation*}\n\t\\proj_D(v)\n\t{}\\coloneqq{}\n\t\\argmin_{z \\in D} \\|z - v\\|\n\t{}\\quad\\text{and}\\quad{}\n\t\\dist_D(v)\n\t{}\\coloneqq{}\n\t\\inf_{z \\in D} \\|z - v\\| .\n\\end{equation*}\nThe former is a set-valued mapping whenever $D$ is nonconvex, whereas the latter is always single-valued.\n\n\nThe \\emph{indicator} function of a set $D \\subset \\YY$ is the function $\\func{\\indicator_D}{\\YY}{\\Rinf}$ defined as $\\indicator_D(v) = 0$ if $v \\in D$, and $\\indicator_D(v) = \\infty$ otherwise.\nIf $D$ is nonempty and closed, then $\\indicator_D$ is proper and lsc.\n\n\nGiven $z \\in D$, the \\emph{limiting normal cone} to $D$ at $z$ is the closed cone\n\\begin{equation*}\n\t\\limnormalcone_D(z) \\coloneqq \\limsup_{v \\to z} \\; \\cone \\left( v - \\proj_D(v) \\right) .\n\\end{equation*}\nFor $\\tilde z\\notin D$, we formally set $\\limnormalcone_D(\\tilde z):=\\emptyset$.\n\n\n\\begin{defin}[Feasibility]\\label{def:feasible}\n\t\tA point $x^\\ast \\in \\XX$ is called \\emph{feasible} for \\eqref{eq:P} \n\t\tif $x^\\ast \\in \\dom q$ and $c(x^\\ast) \\in D$.\n\t\\end{defin}\n\n\\begin{defin}[AM-stationarity]\\label{def:AMstationary}\n\t\tLet $x^\\ast \\in \\XX$ be a feasible point for \\eqref{eq:P}.\n\t\tThen, $x^\\ast$ is called an \\emph{asymptotically M-stationary} point of \\eqref{eq:P} \n\t\tif there exist sequences $\\{x^k\\}, \\{\\eta^k\\} \\subset \\XX$ and \n\t\t$\\{y^k\\}, \\{\\zeta^k\\} \\subset \\YY$ such that \n\t\t$x^k \\toattentive{q} x^\\ast$, $\\eta^k \\to 0$, $\\zeta^k \\to 0$ and\n\t\t\\begin{subequations}\n\t\t\t\\begin{align}\n\t\t\t\t-\\nabla c(x^k)^\\top y^k + \\eta^k\n\t\t\t\t{}\\in{}&\n\t\t\t\t\\partial q(x^k) ,\n\t\t\t\t\\label{eq:AMstationary:x} \\\\\n\t\t\t\ty^k \n\t\t\t\t{}\\in{}&\n\t\t\t\t\\limnormalcone_D( c(x^k) + \\zeta^k )\n\t\t\t\t\\label{eq:AMstationary:y}\n\t\t\t\\end{align}\n\t\t\t\\label{eq:AMstationary}\n\t\t\\end{subequations}\n\t\tfor all $k \\in \\N$.\n\t\\end{defin}\n\n\\begin{equation}\n\t\\minimize_{x ,\\; s}\n\t{}\\quad{}\n\tq(x)\n\t{}\\qquad{}\n\t\\stt\n\t{}\\quad{}\n\tc(x) - s = 0\n\t,\\quad\n\ts \\in D .\n\t\\tag{P$_{\\text{S}}$}\\label{eq:ALMz:P}\n\\end{equation}\n\nFor some penalty parameter $\\mu>0$,\nlet us define the $\\mu$-augmented Lagrangian function $\\func{\\LLslack_\\mu}{\\XX \\times \\YY \\times \\YY}{\\Rinf}$ associated to \\eqref{eq:ALMz:P} as\n\\begin{align}\n\t\\LLslack_\\mu(x,s,y)\n\t{}\\coloneqq{}&\n\tq(x) + \\indicator_D(s) + \\innprod{y}{c(x) - s} + \\frac{1}{2\\mu} \\| c(x) - s \\|^2 \\nonumber \\\\\n\t{}={}&\n\tq(x) + \\indicator_D(s) + \\frac{1}{2\\mu} \\| c(x) + \\mu y - s \\|^2 - \\frac{\\mu}{2}\\|y\\|^2 .\n\t\\label{eq:Lz}\n\\end{align}\n\nFor later use, let us compute the subdifferential of $\\LLslack_\\mu$ with respect to the\nvariables $x$ and $s$:\n\\begin{subequations}\n\t\\label{eq:dLS}\n\n\n\\begin{align}\n\t\t\\partial_x \\LLslack_\\mu(x,s,y) \n\t\t{}={}&\n\t\t\\partial q(x) + \\frac{1}{\\mu} \\nabla c(x)^\\top [c(x) + \\mu y - s] ,\n\t\t\\label{eq:dxLS} \\\\\n\t\t\\partial_s \\LLslack_\\mu(x,s,y) \n\t\t{}={}&\n\t\t\\limnormalcone_D(s) - \\frac{1}{\\mu} [c(x) + \\mu y - s] .\n\t\t\\label{eq:dzLS}\n\t\\end{align}\n\n\\begin{algorithm}[t]\n\t\\caption{Augmented Lagrangian method for \\eqref{eq:P}}%\n\t\\label{alg:ALM}%\n\t\\begin{algorithmic}[1]%\n\t\\linespread{1.2}\\selectfont%\n\t%\n\t%\n%\t\\Require $x^0 \\in \\XX$, $y^0 \\in \\YY$ %$\\varepsilon_{\\text{tol}} \\geq 0$\n\t\\Initialize Select $\\mu_0 > 0$, $\\theta,\\kappa \\in(0,1)$ and $\\Ybounded \\subset \\YY$ nonempty bounded\n\t\\item[For \\(k=0,1,2\\ldots\\)]\n\t\\State\\label{state:ALM:init}\\label{state:ALM:ysafe}%\n\t\tSelect $\\hat{y}^k \\in \\Ybounded$ and $\\varepsilon_k \\geq 0$\n\t\\State\\label{state:ALM:subproblem}%\\label{state:ALM:z}%\n\t\tCompute an $\\varepsilon_k$-stationary point $(x^k,s^k)\\in\\XX\\times D$ of $\\LL_{\\mu_k}^{\\text{S}}(\\cdot,\\cdot,\\hat{y}^k)$\n\t\\State\\label{state:ALM:y}%\n\t\tSet $y^k \\gets \\hat{y}^k + [ c(x^k) - s^k ] / \\mu_k$\n%\t\\If{$\\varepsilon_{k+1} \\leq \\varepsilon_{\\text{tol}}$ and $\\|c(x^{k+1}) - z^{k+1}\\| \\leq \\varepsilon_{\\text{tol}}$}\\label{state:ALM:ifend}%\n%\t\t\\State\\label{state:ALM:end}%\n%\t\t\t\\algfont{return} $(x^{k+1}, y^{k+1})$\n%\t\\EndIf\n\t\\If{$k = 0$ or $\\| c(x^k) - s^k \\| \\leq \\theta \\, \\| c(x^{k-1}) - s^{k-1} \\|$}\\label{state:ALM:if}\n\t\t\\State\\label{state:ALM:ifpassed}%\n\t\t\tSet $\\mu_{k+1} \\gets \\mu_k$\n\t\\Else\n\t\t\\State\\label{state:ALM:iffailed}%\n\t\t\tSelect $\\mu_{k+1} \\in (0, \\kappa \\mu_k]$\n\t\\EndIf\n\\end{algorithmic}\n%\n\\end{algorithm}\n\n\tThroughout our convergence analysis, we assume that \\cref{alg:ALM} is well-defined, \nthus requiring that each subproblem at \\cref{state:ALM:subproblem} admits an approximate stationary point.\nMoreover, the following statements assume the existence of some accumulation point $x^\\ast$ \nor $(x^\\ast,s^\\ast)$\nfor a sequence $\\{ x^k \\}$ or $\\{(x^k,s^k)\\}$, respectively, generated by \\cref{alg:ALM}.\n","units":[{"start":1907,"end":1953,"start_line":86,"end_line":87,"raw_tex":"\\newcommand{\\XX}{\\R^n}\n\\newcommand{\\YY}{\\R^m}\n","sha256_utf8":"9648c600decab7777b5c70354c474cf936b1a4f0d7c48ffc62e8d3f3222f6dd8"},{"start":6650,"end":6823,"start_line":195,"end_line":204,"raw_tex":"\\begin{equation}\n    \\minimize_{x}\n    {}\\quad{}\n    q(x) \\coloneqq f(x) + g(x)\n    {}\\qquad{}\n    \\stt\n    {}\\quad{}\n    c(x) \\in D ,\n    \\tag{P}\\label{eq:P}\n\\end{equation}","sha256_utf8":"4b3443a2e4575b394988dfe74c26f87c6a10bfde26b3599ece9eae41459fecac"},{"start":6824,"end":6968,"start_line":205,"end_line":205,"raw_tex":"where $x$ is the decision variable, $f$ and $c$ are smooth functions, $g$ is proper and lower semicontinuous, and $D$ is a nonempty closed set.\n","sha256_utf8":"2a2dc03d76c6d55e9223cecf8f97a4dbe64ffca3e6070a47d5422fe55866f4dd"},{"start":12806,"end":13246,"start_line":267,"end_line":277,"raw_tex":"\\begin{ass}\\label{ass:P}\n\t\tThe following hold in \\eqref{eq:P}:\n\t\t\\begin{enumerate}\n\t\t\t\\item\\label{ass:f}%\n\t\t\t\t$\\func{f}{\\XX}{\\R}$ and $\\func{c}{\\XX}{\\YY}$ are continuously differentiable with locally Lipschitz continuous derivatives;\n\t\t\t\\item\\label{ass:g}%\n\t\t\t\t\\(\\func{g}{\\XX}{\\Rinf}\\) is proper, lower semicontinuous and prox-bounded;\n\t\t\t\\item\\label{ass:set}%\n\t\t\t\t\\(D\\subset \\YY\\) is a nonempty and closed set.\n\t\t\\end{enumerate}\n\t\\end{ass}","sha256_utf8":"b758255e08012fb2fb1a0a35729c0bb01cc1dbd9aa771aa49d01534563b8eef2"},{"start":19466,"end":19793,"start_line":345,"end_line":347,"raw_tex":"The effective domain of an extended real-valued function $\\func{h}{\\R^n}{\\Rinf}$ is denoted by $\\dom h \\coloneqq \\{x\\in\\R^n \\suchthat h(x) < \\infty \\}$.\nWe say that $h$ is \\emph{proper} if $\\dom h \\neq \\emptyset$\nand \\emph{lower semicontinuous} (lsc) if $h(\\bar{x}) \\leq \\liminf_{x\\to\\bar{x}} h(x)$ for all $\\bar{x} \\in \\R^n$.\n","sha256_utf8":"f9e509b7e85e3c18359c9344a6eeb269c28158faacf7060cb416b2b11f73b603"},{"start":19794,"end":21198,"start_line":349,"end_line":367,"raw_tex":"% subgradient\nGiven a proper and lsc function $\\func{h}{\\XX}{\\Rinf}$ and a point $\\bar{x}\\in\\dom h$, we may avoid to assume $h$ continuous and instead appeal \nto $h$-\\emph{attentive} convergence of a sequence $\\{ x^k \\}$:\n\\begin{equation}\nx^k \\toattentive{h} \\bar{x}\n{}\\quad:\\Leftrightarrow\\quad{}\nx^k \\to \\bar{x}\n{}\\quad\\text{with}\\quad{}\nh(x^k) \\to h(\\bar{x}) .\n\\end{equation}\nFollowing \\cite[Def. 8.3]{rockafellar1998variational}, we denote by $\\ffunc{\\hat{\\partial} h}{\\XX}{\\XX}$ the \\emph{regular subdifferential} of $h$, where\n\\begin{equation}\nv \\in \\hat{\\partial} h(\\bar{x})\n{}\\quad:\\Leftrightarrow\\quad{}\n\\liminf_{\\substack{x\\to\\bar{x}\\\\x\\neq\\bar{x}}} \\frac{h(x) - h(\\bar{x}) - \\langle v, x-\\bar{x}\\rangle}{\\|x-\\bar{x}\\|} \\geq 0 .\n\\end{equation}\nThe (limiting) \\emph{subdifferential} of $h$ is $\\ffunc{\\partial h}{\\XX}{\\XX}$, where $v \\in \\partial h(\\bar{x})$ if and only if there exist sequences $\\{x^k\\}$ and $\\{v^k\\}$ such that $x^k \\toattentive{h} \\bar{x}$ and $v^k \\in \\hat{\\partial} h(x^k)$ with $v^k \\to v$.\nThe subdifferential of $h$ at $\\bar{x}$ satisfies $\\partial(h+h_0)(\\bar{x}) = \\partial h(\\bar{x}) + \\nabla h_0(\\bar{x})$ for any $\\func{h_0}{\\XX}{\\Rinf}$ continuously differentiable around $\\bar{x}$ \\cite[Ex. 8.8]{rockafellar1998variational}.\nFor formal completeness, we set $\\hat{\\partial} h(\\bar{x})\\coloneqq\\partial h(\\bar{x})\\coloneqq\\emptyset$ for each $\\bar{x}\\notin\\dom h$.\n","sha256_utf8":"e322f8ca976b726212d993ee81f1882563ba80b684d0ce8bc5e3060db334f1dc"},{"start":21454,"end":21694,"start_line":370,"end_line":374,"raw_tex":"Furthermore, we say that $x^\\ast \\in \\XX$ is \\emph{$\\varepsilon$-stationary} for some $\\varepsilon \\geq 0$ if\n\\begin{equation}\n\t\\exists \\eta \\in \\partial h(x^\\ast) \\colon \\|\\eta\\| \\leq \\varepsilon .\n\t\\label{eq:epsstationary}\n\\end{equation}\n","sha256_utf8":"f7f799d21b7e36a1eae86f954819d50efece386e1f22a2afe0142289f06ed003"},{"start":22027,"end":22377,"start_line":378,"end_line":391,"raw_tex":"If $S(\\bar{x})$ is nonempty, we define the \\emph{outer limit} of $S$ at $\\bar{x}$\nby means of\n\\begin{align*}\n\t\\limsup\\limits_{x\\to\\bar{x}} S(x)\n\t\\coloneqq\n\t\\{\n\t\ty\\in\\YY\n\t\t\\suchthat\n\t\t\\exists x^k\\to\\bar{x},\\,\n\t\t\\exists y^k\\to y,\\,\n\t\ty^k\\in S(x^k)\\,\\forall k\\in\\N\n\t\\}\n\\end{align*}\nand note that this is a closed superset of $S(\\bar{x})$ by definition.\n","sha256_utf8":"01bdc03ea573f7885d9bff9f0ffa021bf148bfceca11db075e99fed35cf300e0"},{"start":22632,"end":22884,"start_line":400,"end_line":402,"raw_tex":"% Prox-boundedness\nand we say that $h$ is \\emph{prox-bounded} if it is proper and $h + \\|\\cdot\\|^2 / (2\\gamma)$ is bounded below on $\\XX$ for some $\\gamma > 0$.\nThe supremum of all such $\\gamma$ is the threshold $\\gamma_h$ of prox-boundedness for $h$.\n","sha256_utf8":"74ca58db0f804b36f76772e146614d82cb49be11088cab800f37c51d7619efa4"},{"start":23390,"end":23765,"start_line":408,"end_line":418,"raw_tex":"The \\emph{projection} mapping $\\proj_D$ and the \\emph{distance} function $\\dist_D$ are defined by\n\\begin{equation*}\n\t\\proj_D(v)\n\t{}\\coloneqq{}\n\t\\argmin_{z \\in D} \\|z - v\\|\n\t{}\\quad\\text{and}\\quad{}\n\t\\dist_D(v)\n\t{}\\coloneqq{}\n\t\\inf_{z \\in D} \\|z - v\\| .\n\\end{equation*}\nThe former is a set-valued mapping whenever $D$ is nonconvex, whereas the latter is always single-valued.\n","sha256_utf8":"0e33b485f3402dfe3d4ad3c895b8f7b17dcbdd0a1604b869c400b244e96ec9ff"},{"start":23777,"end":24041,"start_line":420,"end_line":421,"raw_tex":"The \\emph{indicator} function of a set $D \\subset \\YY$ is the function $\\func{\\indicator_D}{\\YY}{\\Rinf}$ defined as $\\indicator_D(v) = 0$ if $v \\in D$, and $\\indicator_D(v) = \\infty$ otherwise.\nIf $D$ is nonempty and closed, then $\\indicator_D$ is proper and lsc.\n","sha256_utf8":"c0b8509d84dc100f86cceb8223af72bbb1fa2f8362d142612618c891fef2fe64"},{"start":24168,"end":24455,"start_line":424,"end_line":428,"raw_tex":"Given $z \\in D$, the \\emph{limiting normal cone} to $D$ at $z$ is the closed cone\n\\begin{equation*}\n\t\\limnormalcone_D(z) \\coloneqq \\limsup_{v \\to z} \\; \\cone \\left( v - \\proj_D(v) \\right) .\n\\end{equation*}\nFor $\\tilde z\\notin D$, we formally set $\\limnormalcone_D(\\tilde z):=\\emptyset$.\n","sha256_utf8":"2b46142ba037da3ac3c18bde799a8cc0b79d1ed346389985ee81e3552609a7a2"},{"start":26560,"end":26738,"start_line":485,"end_line":488,"raw_tex":"\\begin{defin}[Feasibility]\\label{def:feasible}\n\t\tA point $x^\\ast \\in \\XX$ is called \\emph{feasible} for \\eqref{eq:P} \n\t\tif $x^\\ast \\in \\dom q$ and $c(x^\\ast) \\in D$.\n\t\\end{defin}","sha256_utf8":"83610e92391b10f082473459fe9661a44b962f179ee9b8cf7101c3c8af26cbd8"},{"start":28261,"end":28982,"start_line":526,"end_line":546,"raw_tex":"\\begin{defin}[AM-stationarity]\\label{def:AMstationary}\n\t\tLet $x^\\ast \\in \\XX$ be a feasible point for \\eqref{eq:P}.\n\t\tThen, $x^\\ast$ is called an \\emph{asymptotically M-stationary} point of \\eqref{eq:P} \n\t\tif there exist sequences $\\{x^k\\}, \\{\\eta^k\\} \\subset \\XX$ and \n\t\t$\\{y^k\\}, \\{\\zeta^k\\} \\subset \\YY$ such that \n\t\t$x^k \\toattentive{q} x^\\ast$, $\\eta^k \\to 0$, $\\zeta^k \\to 0$ and\n\t\t\\begin{subequations}\n\t\t\t\\begin{align}\n\t\t\t\t-\\nabla c(x^k)^\\top y^k + \\eta^k\n\t\t\t\t{}\\in{}&\n\t\t\t\t\\partial q(x^k) ,\n\t\t\t\t\\label{eq:AMstationary:x} \\\\\n\t\t\t\ty^k \n\t\t\t\t{}\\in{}&\n\t\t\t\t\\limnormalcone_D( c(x^k) + \\zeta^k )\n\t\t\t\t\\label{eq:AMstationary:y}\n\t\t\t\\end{align}\n\t\t\t\\label{eq:AMstationary}\n\t\t\\end{subequations}\n\t\tfor all $k \\in \\N$.\n\t\\end{defin}","sha256_utf8":"54fb08dae5c387db810e276ffc75b4b17d9df778af454e22452a8b9a2101f240"},{"start":38742,"end":38912,"start_line":766,"end_line":777,"raw_tex":"\\begin{equation}\n\t\\minimize_{x ,\\; s}\n\t{}\\quad{}\n\tq(x)\n\t{}\\qquad{}\n\t\\stt\n\t{}\\quad{}\n\tc(x) - s = 0\n\t,\\quad\n\ts \\in D .\n\t\\tag{P$_{\\text{S}}$}\\label{eq:ALMz:P}\n\\end{equation}","sha256_utf8":"a3862533ae61c11836ab69e2c01c3568b752736f810b12af5d7ed339f7c169bc"},{"start":41987,"end":42439,"start_line":826,"end_line":835,"raw_tex":"For some penalty parameter $\\mu>0$,\nlet us define the $\\mu$-augmented Lagrangian function $\\func{\\LLslack_\\mu}{\\XX \\times \\YY \\times \\YY}{\\Rinf}$ associated to \\eqref{eq:ALMz:P} as\n\\begin{align}\n\t\\LLslack_\\mu(x,s,y)\n\t{}\\coloneqq{}&\n\tq(x) + \\indicator_D(s) + \\innprod{y}{c(x) - s} + \\frac{1}{2\\mu} \\| c(x) - s \\|^2 \\nonumber \\\\\n\t{}={}&\n\tq(x) + \\indicator_D(s) + \\frac{1}{2\\mu} \\| c(x) + \\mu y - s \\|^2 - \\frac{\\mu}{2}\\|y\\|^2 .\n\t\\label{eq:Lz}\n\\end{align}","sha256_utf8":"4f6c79a7f43cf6b3f8d0eae7fbd5dfd461fea3f7904664dc6703a2ed0e8337f3"},{"start":42753,"end":42901,"start_line":838,"end_line":841,"raw_tex":"For later use, let us compute the subdifferential of $\\LLslack_\\mu$ with respect to the\nvariables $x$ and $s$:\n\\begin{subequations}\n\t\\label{eq:dLS}\n","sha256_utf8":"d45e3d51dae94b69679f62d923c66d440ff49975f7cee3016fded3bcf94ab635"},{"start":42902,"end":43182,"start_line":842,"end_line":851,"raw_tex":"\\begin{align}\n\t\t\\partial_x \\LLslack_\\mu(x,s,y) \n\t\t{}={}&\n\t\t\\partial q(x) + \\frac{1}{\\mu} \\nabla c(x)^\\top [c(x) + \\mu y - s] ,\n\t\t\\label{eq:dxLS} \\\\\n\t\t\\partial_s \\LLslack_\\mu(x,s,y) \n\t\t{}={}&\n\t\t\\limnormalcone_D(s) - \\frac{1}{\\mu} [c(x) + \\mu y - s] .\n\t\t\\label{eq:dzLS}\n\t\\end{align}","sha256_utf8":"9211706ca575b3cb3204b118168a7edde9fa11fe0b2b17924f30db2c3c904a3a"},{"start":48317,"end":49583,"start_line":932,"end_line":961,"raw_tex":"\\begin{algorithm}[t]\n\t\\caption{Augmented Lagrangian method for \\eqref{eq:P}}%\n\t\\label{alg:ALM}%\n\t\\begin{algorithmic}[1]%\n\t\\linespread{1.2}\\selectfont%\n\t%\n\t%\n%\t\\Require $x^0 \\in \\XX$, $y^0 \\in \\YY$ %$\\varepsilon_{\\text{tol}} \\geq 0$\n\t\\Initialize Select $\\mu_0 > 0$, $\\theta,\\kappa \\in(0,1)$ and $\\Ybounded \\subset \\YY$ nonempty bounded\n\t\\item[For \\(k=0,1,2\\ldots\\)]\n\t\\State\\label{state:ALM:init}\\label{state:ALM:ysafe}%\n\t\tSelect $\\hat{y}^k \\in \\Ybounded$ and $\\varepsilon_k \\geq 0$\n\t\\State\\label{state:ALM:subproblem}%\\label{state:ALM:z}%\n\t\tCompute an $\\varepsilon_k$-stationary point $(x^k,s^k)\\in\\XX\\times D$ of $\\LL_{\\mu_k}^{\\text{S}}(\\cdot,\\cdot,\\hat{y}^k)$\n\t\\State\\label{state:ALM:y}%\n\t\tSet $y^k \\gets \\hat{y}^k + [ c(x^k) - s^k ] / \\mu_k$\n%\t\\If{$\\varepsilon_{k+1} \\leq \\varepsilon_{\\text{tol}}$ and $\\|c(x^{k+1}) - z^{k+1}\\| \\leq \\varepsilon_{\\text{tol}}$}\\label{state:ALM:ifend}%\n%\t\t\\State\\label{state:ALM:end}%\n%\t\t\t\\algfont{return} $(x^{k+1}, y^{k+1})$\n%\t\\EndIf\n\t\\If{$k = 0$ or $\\| c(x^k) - s^k \\| \\leq \\theta \\, \\| c(x^{k-1}) - s^{k-1} \\|$}\\label{state:ALM:if}\n\t\t\\State\\label{state:ALM:ifpassed}%\n\t\t\tSet $\\mu_{k+1} \\gets \\mu_k$\n\t\\Else\n\t\t\\State\\label{state:ALM:iffailed}%\n\t\t\tSelect $\\mu_{k+1} \\in (0, \\kappa \\mu_k]$\n\t\\EndIf\n\\end{algorithmic}\n%\n\\end{algorithm}","sha256_utf8":"fee140ed71cf6ba8564aaef6d6aca8d83dd16d29c433c66c2002f3f30a345d67"},{"start":53367,"end":53764,"start_line":1038,"end_line":1042,"raw_tex":"\tThroughout our convergence analysis, we assume that \\cref{alg:ALM} is well-defined, \nthus requiring that each subproblem at \\cref{state:ALM:subproblem} admits an approximate stationary point.\nMoreover, the following statements assume the existence of some accumulation point $x^\\ast$ \nor $(x^\\ast,s^\\ast)$\nfor a sequence $\\{ x^k \\}$ or $\\{(x^k,s^k)\\}$, respectively, generated by \\cref{alg:ALM}.\n","sha256_utf8":"2bcdaf43625b602562b756c1117e4d0579073591bc25101dd29885ba26b5f789"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{thm}\\label{thm:AMstationary}\n\t\tLet \\cref{ass:P} hold and consider a sequence $\\{ (x^k,s^k) \\}$ of iterates \n\t\tgenerated by \\cref{alg:ALM} with $\\varepsilon_k \\to 0$.\n\t\tLet $(x^\\ast,c(x^\\ast))$ be an accumulation point  of $\\{ (x^k,s^k) \\}$ \n\t\tfeasible to \\eqref{eq:ALMz:P} and\n\t\t$\\{ (x^k,s^k)\\}_K$ a subsequence such that $x^k \\toattentive{q}_K x^\\ast$ and $s^k \\to_K c(x^\\ast)$.\n\t\tThen, $x^\\ast$ is an AM-stationary point for \\eqref{eq:P}.\n\t\\end{thm}","units":[{"start":58214,"end":58672,"start_line":1150,"end_line":1157,"raw_tex":"\\begin{thm}\\label{thm:AMstationary}\n\t\tLet \\cref{ass:P} hold and consider a sequence $\\{ (x^k,s^k) \\}$ of iterates \n\t\tgenerated by \\cref{alg:ALM} with $\\varepsilon_k \\to 0$.\n\t\tLet $(x^\\ast,c(x^\\ast))$ be an accumulation point  of $\\{ (x^k,s^k) \\}$ \n\t\tfeasible to \\eqref{eq:ALMz:P} and\n\t\t$\\{ (x^k,s^k)\\}_K$ a subsequence such that $x^k \\toattentive{q}_K x^\\ast$ and $s^k \\to_K c(x^\\ast)$.\n\t\tThen, $x^\\ast$ is an AM-stationary point for \\eqref{eq:P}.\n\t\\end{thm}","sha256_utf8":"af519f93af9fe306a2408bebe3f96eefa788db361a85bd1a878d67a35252a6e9"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{proof}\n\tDefine $\\zeta^k \\coloneqq s^k - c(x^k)$ for all $k \\in \\N$.\n\tThen, from \\cref{state:ALM:subproblem,state:ALM:y} of \\cref{alg:ALM}, we have that\n\t\\begin{align}\n\t\t- \\nabla c(x^k)^\\top y^k + \\xi^k {}\\in{}& \\partial q(x^k) ,\n\t\t\\label{thm:AMstationary:x} \\\\\n\t\ty^k + \\nu^k {}\\in{}& \\limnormalcone_D(c(x^k) + \\zeta^k)\n\t\t\\label{thm:AMstationary:y}\n\t\\end{align}\n\tfor some $\\xi^k \\in \\XX$, $\\| \\xi^k \\| \\leq \\varepsilon_k$,\n\tand $\\nu^k \\in \\YY$, $\\| \\nu^k \\| \\leq \\varepsilon_k$; cf.\\ \\eqref{eq:epsstationary} and \\eqref{eq:dLS}.\n\tSet $\\lambda^k \\coloneqq y^k+\\nu^k$ and $\\eta^k \\coloneqq \\nabla c(x^k)^\\top\\nu^k+\\xi^k$ for all $k \\in \\N$.\n\t\n\tWe claim that the four subsequences $\\{x^k\\}_K$, $\\{\\eta^k\\}_K$, \n\t$\\{\\lambda^k\\}_K$\n\tand $\\{\\zeta^k\\}_K$ satisfy the properties \n\tin \\cref{def:AMstationary} and therefore show that $x^\\ast$ is an AM-stationary point \n\tfor \\eqref{eq:P}.\n\t\n\tBy construction, we have $x^k \\toattentive{q}_K x^\\ast$ \n\tas well as $-\\nabla c(x^k)^\\top \\lambda^k+\\eta^k\\in\\partial q(x^k)$ and\n\t$\\lambda^k\\in\\limnormalcone_D(c(x^k)+\\zeta^k)$ for each $k \\in \\N$.\n\tContinuous differentiability of $c$ and $\\|\\xi^k\\|,\\|\\nu^k\\|\\leq\\varepsilon_k$ give $\\|\\eta^k\\| \\to_K 0$.\n\tFinally, $\\zeta^k \\to_K 0$ follows from $s^k \\to_K c(x^\\ast)$, $x^k \\to_K x^\\ast$ and continuity of $c$.\n\t\n\tOverall, this proves that $x^\\ast$ is an AM-stationary point for \\eqref{eq:P}.\n\\end{proof}","units":[{"start":58685,"end":60078,"start_line":1159,"end_line":1185,"raw_tex":"\\begin{proof}\n\tDefine $\\zeta^k \\coloneqq s^k - c(x^k)$ for all $k \\in \\N$.\n\tThen, from \\cref{state:ALM:subproblem,state:ALM:y} of \\cref{alg:ALM}, we have that\n\t\\begin{align}\n\t\t- \\nabla c(x^k)^\\top y^k + \\xi^k {}\\in{}& \\partial q(x^k) ,\n\t\t\\label{thm:AMstationary:x} \\\\\n\t\ty^k + \\nu^k {}\\in{}& \\limnormalcone_D(c(x^k) + \\zeta^k)\n\t\t\\label{thm:AMstationary:y}\n\t\\end{align}\n\tfor some $\\xi^k \\in \\XX$, $\\| \\xi^k \\| \\leq \\varepsilon_k$,\n\tand $\\nu^k \\in \\YY$, $\\| \\nu^k \\| \\leq \\varepsilon_k$; cf.\\ \\eqref{eq:epsstationary} and \\eqref{eq:dLS}.\n\tSet $\\lambda^k \\coloneqq y^k+\\nu^k$ and $\\eta^k \\coloneqq \\nabla c(x^k)^\\top\\nu^k+\\xi^k$ for all $k \\in \\N$.\n\t\n\tWe claim that the four subsequences $\\{x^k\\}_K$, $\\{\\eta^k\\}_K$, \n\t$\\{\\lambda^k\\}_K$\n\tand $\\{\\zeta^k\\}_K$ satisfy the properties \n\tin \\cref{def:AMstationary} and therefore show that $x^\\ast$ is an AM-stationary point \n\tfor \\eqref{eq:P}.\n\t\n\tBy construction, we have $x^k \\toattentive{q}_K x^\\ast$ \n\tas well as $-\\nabla c(x^k)^\\top \\lambda^k+\\eta^k\\in\\partial q(x^k)$ and\n\t$\\lambda^k\\in\\limnormalcone_D(c(x^k)+\\zeta^k)$ for each $k \\in \\N$.\n\tContinuous differentiability of $c$ and $\\|\\xi^k\\|,\\|\\nu^k\\|\\leq\\varepsilon_k$ give $\\|\\eta^k\\| \\to_K 0$.\n\tFinally, $\\zeta^k \\to_K 0$ follows from $s^k \\to_K c(x^\\ast)$, $x^k \\to_K x^\\ast$ and continuity of $c$.\n\t\n\tOverall, this proves that $x^\\ast$ is an AM-stationary point for \\eqref{eq:P}.\n\\end{proof}","sha256_utf8":"95d27d9a2b78c20d2b6e1ea268bffc770e1a2c69fd387a299dc31f77836fe4bf"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"增广拉格朗日迭代的渐近驻点收敛","source_units":[{"start":58214,"end":58672,"start_line":1150,"end_line":1157,"raw_tex":"\\begin{thm}\\label{thm:AMstationary}\n\t\tLet \\cref{ass:P} hold and consider a sequence $\\{ (x^k,s^k) \\}$ of iterates \n\t\tgenerated by \\cref{alg:ALM} with $\\varepsilon_k \\to 0$.\n\t\tLet $(x^\\ast,c(x^\\ast))$ be an accumulation point  of $\\{ (x^k,s^k) \\}$ \n\t\tfeasible to \\eqref{eq:ALMz:P} and\n\t\t$\\{ (x^k,s^k)\\}_K$ a subsequence such that $x^k \\toattentive{q}_K x^\\ast$ and $s^k \\to_K c(x^\\ast)$.\n\t\tThen, $x^\\ast$ is an AM-stationary point for \\eqref{eq:P}.\n\t\\end{thm}","sha256_utf8":"af519f93af9fe306a2408bebe3f96eefa788db361a85bd1a878d67a35252a6e9"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"渐近驻点收敛的完整证明","source_units":[{"start":58685,"end":60078,"start_line":1159,"end_line":1185,"raw_tex":"\\begin{proof}\n\tDefine $\\zeta^k \\coloneqq s^k - c(x^k)$ for all $k \\in \\N$.\n\tThen, from \\cref{state:ALM:subproblem,state:ALM:y} of \\cref{alg:ALM}, we have that\n\t\\begin{align}\n\t\t- \\nabla c(x^k)^\\top y^k + \\xi^k {}\\in{}& \\partial q(x^k) ,\n\t\t\\label{thm:AMstationary:x} \\\\\n\t\ty^k + \\nu^k {}\\in{}& \\limnormalcone_D(c(x^k) + \\zeta^k)\n\t\t\\label{thm:AMstationary:y}\n\t\\end{align}\n\tfor some $\\xi^k \\in \\XX$, $\\| \\xi^k \\| \\leq \\varepsilon_k$,\n\tand $\\nu^k \\in \\YY$, $\\| \\nu^k \\| \\leq \\varepsilon_k$; cf.\\ \\eqref{eq:epsstationary} and \\eqref{eq:dLS}.\n\tSet $\\lambda^k \\coloneqq y^k+\\nu^k$ and $\\eta^k \\coloneqq \\nabla c(x^k)^\\top\\nu^k+\\xi^k$ for all $k \\in \\N$.\n\t\n\tWe claim that the four subsequences $\\{x^k\\}_K$, $\\{\\eta^k\\}_K$, \n\t$\\{\\lambda^k\\}_K$\n\tand $\\{\\zeta^k\\}_K$ satisfy the properties \n\tin \\cref{def:AMstationary} and therefore show that $x^\\ast$ is an AM-stationary point \n\tfor \\eqref{eq:P}.\n\t\n\tBy construction, we have $x^k \\toattentive{q}_K x^\\ast$ \n\tas well as $-\\nabla c(x^k)^\\top \\lambda^k+\\eta^k\\in\\partial q(x^k)$ and\n\t$\\lambda^k\\in\\limnormalcone_D(c(x^k)+\\zeta^k)$ for each $k \\in \\N$.\n\tContinuous differentiability of $c$ and $\\|\\xi^k\\|,\\|\\nu^k\\|\\leq\\varepsilon_k$ give $\\|\\eta^k\\| \\to_K 0$.\n\tFinally, $\\zeta^k \\to_K 0$ follows from $s^k \\to_K c(x^\\ast)$, $x^k \\to_K x^\\ast$ and continuity of $c$.\n\t\n\tOverall, this proves that $x^\\ast$ is an AM-stationary point for \\eqref{eq:P}.\n\\end{proof}","sha256_utf8":"95d27d9a2b78c20d2b6e1ea268bffc770e1a2c69fd387a299dc31f77836fe4bf"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"由子问题驻点条件取得误差包含式","source_units":[{"start":58699,"end":59220,"start_line":1160,"end_line":1169,"raw_tex":"\tDefine $\\zeta^k \\coloneqq s^k - c(x^k)$ for all $k \\in \\N$.\n\tThen, from \\cref{state:ALM:subproblem,state:ALM:y} of \\cref{alg:ALM}, we have that\n\t\\begin{align}\n\t\t- \\nabla c(x^k)^\\top y^k + \\xi^k {}\\in{}& \\partial q(x^k) ,\n\t\t\\label{thm:AMstationary:x} \\\\\n\t\ty^k + \\nu^k {}\\in{}& \\limnormalcone_D(c(x^k) + \\zeta^k)\n\t\t\\label{thm:AMstationary:y}\n\t\\end{align}\n\tfor some $\\xi^k \\in \\XX$, $\\| \\xi^k \\| \\leq \\varepsilon_k$,\n\tand $\\nu^k \\in \\YY$, $\\| \\nu^k \\| \\leq \\varepsilon_k$; cf.\\ \\eqref{eq:epsstationary} and \\eqref{eq:dLS}.\n","sha256_utf8":"d121b93fc04e2cca97798a333f0954669a99db24f9203c3a5fa71cdeb5863dcc"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"构造修正乘子与驻点误差序列","source_units":[{"start":59220,"end":59570,"start_line":1170,"end_line":1176,"raw_tex":"\tSet $\\lambda^k \\coloneqq y^k+\\nu^k$ and $\\eta^k \\coloneqq \\nabla c(x^k)^\\top\\nu^k+\\xi^k$ for all $k \\in \\N$.\n\t\n\tWe claim that the four subsequences $\\{x^k\\}_K$, $\\{\\eta^k\\}_K$, \n\t$\\{\\lambda^k\\}_K$\n\tand $\\{\\zeta^k\\}_K$ satisfy the properties \n\tin \\cref{def:AMstationary} and therefore show that $x^\\ast$ is an AM-stationary point \n\tfor \\eqref{eq:P}.\n","sha256_utf8":"6d3d1cb70b53c89c4b3cf7ef9e968fdd1261f7b4679285cc9d6a5da66e818d6e"}]},{"id":"S005","semantic_type":"proof_step","kind":"proof_step","title":"验证包含式与误差消失并完成证明","source_units":[{"start":59572,"end":60067,"start_line":1178,"end_line":1184,"raw_tex":"\tBy construction, we have $x^k \\toattentive{q}_K x^\\ast$ \n\tas well as $-\\nabla c(x^k)^\\top \\lambda^k+\\eta^k\\in\\partial q(x^k)$ and\n\t$\\lambda^k\\in\\limnormalcone_D(c(x^k)+\\zeta^k)$ for each $k \\in \\N$.\n\tContinuous differentiability of $c$ and $\\|\\xi^k\\|,\\|\\nu^k\\|\\leq\\varepsilon_k$ give $\\|\\eta^k\\| \\to_K 0$.\n\tFinally, $\\zeta^k \\to_K 0$ follows from $s^k \\to_K c(x^\\ast)$, $x^k \\to_K x^\\ast$ and continuity of $c$.\n\t\n\tOverall, this proves that $x^\\ast$ is an AM-stationary point for \\eqref{eq:P}.\n","sha256_utf8":"aacc1694721431d209d963c19996309a1ae738d0617d02c7928da18bd9fbc217"}]},{"id":"S006","semantic_type":"problem_setting","kind":"problem_setting","title":"约束复合优化问题","source_units":[{"start":6650,"end":6823,"start_line":195,"end_line":204,"raw_tex":"\\begin{equation}\n    \\minimize_{x}\n    {}\\quad{}\n    q(x) \\coloneqq f(x) + g(x)\n    {}\\qquad{}\n    \\stt\n    {}\\quad{}\n    c(x) \\in D ,\n    \\tag{P}\\label{eq:P}\n\\end{equation}","sha256_utf8":"4b3443a2e4575b394988dfe74c26f87c6a10bfde26b3599ece9eae41459fecac"},{"start":6824,"end":6968,"start_line":205,"end_line":205,"raw_tex":"where $x$ is the decision variable, $f$ and $c$ are smooth functions, $g$ is proper and lower semicontinuous, and $D$ is a nonempty closed set.\n","sha256_utf8":"2a2dc03d76c6d55e9223cecf8f97a4dbe64ffca3e6070a47d5422fe55866f4dd"}]},{"id":"S007","semantic_type":"assumption","kind":"assumption","title":"问题数据的全局基本假设","source_units":[{"start":12806,"end":13246,"start_line":267,"end_line":277,"raw_tex":"\\begin{ass}\\label{ass:P}\n\t\tThe following hold in \\eqref{eq:P}:\n\t\t\\begin{enumerate}\n\t\t\t\\item\\label{ass:f}%\n\t\t\t\t$\\func{f}{\\XX}{\\R}$ and $\\func{c}{\\XX}{\\YY}$ are continuously differentiable with locally Lipschitz continuous derivatives;\n\t\t\t\\item\\label{ass:g}%\n\t\t\t\t\\(\\func{g}{\\XX}{\\Rinf}\\) is proper, lower semicontinuous and prox-bounded;\n\t\t\t\\item\\label{ass:set}%\n\t\t\t\t\\(D\\subset \\YY\\) is a nonempty and closed set.\n\t\t\\end{enumerate}\n\t\\end{ass}","sha256_utf8":"b758255e08012fb2fb1a0a35729c0bb01cc1dbd9aa771aa49d01534563b8eef2"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"函数值伴随收敛","source_units":[{"start":19794,"end":20173,"start_line":349,"end_line":358,"raw_tex":"% subgradient\nGiven a proper and lsc function $\\func{h}{\\XX}{\\Rinf}$ and a point $\\bar{x}\\in\\dom h$, we may avoid to assume $h$ continuous and instead appeal \nto $h$-\\emph{attentive} convergence of a sequence $\\{ x^k \\}$:\n\\begin{equation}\nx^k \\toattentive{h} \\bar{x}\n{}\\quad:\\Leftrightarrow\\quad{}\nx^k \\to \\bar{x}\n{}\\quad\\text{with}\\quad{}\nh(x^k) \\to h(\\bar{x}) .\n\\end{equation}\n","sha256_utf8":"58d488d72819fa246a9aa903f22332bd3f4b12b5b6cba3a8f5311bb4f2485b00"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"正则与极限次微分及光滑扰动规则","source_units":[{"start":20173,"end":21198,"start_line":359,"end_line":367,"raw_tex":"Following \\cite[Def. 8.3]{rockafellar1998variational}, we denote by $\\ffunc{\\hat{\\partial} h}{\\XX}{\\XX}$ the \\emph{regular subdifferential} of $h$, where\n\\begin{equation}\nv \\in \\hat{\\partial} h(\\bar{x})\n{}\\quad:\\Leftrightarrow\\quad{}\n\\liminf_{\\substack{x\\to\\bar{x}\\\\x\\neq\\bar{x}}} \\frac{h(x) - h(\\bar{x}) - \\langle v, x-\\bar{x}\\rangle}{\\|x-\\bar{x}\\|} \\geq 0 .\n\\end{equation}\nThe (limiting) \\emph{subdifferential} of $h$ is $\\ffunc{\\partial h}{\\XX}{\\XX}$, where $v \\in \\partial h(\\bar{x})$ if and only if there exist sequences $\\{x^k\\}$ and $\\{v^k\\}$ such that $x^k \\toattentive{h} \\bar{x}$ and $v^k \\in \\hat{\\partial} h(x^k)$ with $v^k \\to v$.\nThe subdifferential of $h$ at $\\bar{x}$ satisfies $\\partial(h+h_0)(\\bar{x}) = \\partial h(\\bar{x}) + \\nabla h_0(\\bar{x})$ for any $\\func{h_0}{\\XX}{\\Rinf}$ continuously differentiable around $\\bar{x}$ \\cite[Ex. 8.8]{rockafellar1998variational}.\nFor formal completeness, we set $\\hat{\\partial} h(\\bar{x})\\coloneqq\\partial h(\\bar{x})\\coloneqq\\emptyset$ for each $\\bar{x}\\notin\\dom h$.\n","sha256_utf8":"eac3834398198cf551145a4505dd9ed61f25059a02d01f26ea26884f29ccce29"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"近似驻点","source_units":[{"start":21454,"end":21694,"start_line":370,"end_line":374,"raw_tex":"Furthermore, we say that $x^\\ast \\in \\XX$ is \\emph{$\\varepsilon$-stationary} for some $\\varepsilon \\geq 0$ if\n\\begin{equation}\n\t\\exists \\eta \\in \\partial h(x^\\ast) \\colon \\|\\eta\\| \\leq \\varepsilon .\n\t\\label{eq:epsstationary}\n\\end{equation}\n","sha256_utf8":"f7f799d21b7e36a1eae86f954819d50efece386e1f22a2afe0142289f06ed003"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"近端有界性及其阈值","source_units":[{"start":22632,"end":22884,"start_line":400,"end_line":402,"raw_tex":"% Prox-boundedness\nand we say that $h$ is \\emph{prox-bounded} if it is proper and $h + \\|\\cdot\\|^2 / (2\\gamma)$ is bounded below on $\\XX$ for some $\\gamma > 0$.\nThe supremum of all such $\\gamma$ is the threshold $\\gamma_h$ of prox-boundedness for $h$.\n","sha256_utf8":"74ca58db0f804b36f76772e146614d82cb49be11088cab800f37c51d7619efa4"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"集合投影与距离","source_units":[{"start":23390,"end":23765,"start_line":408,"end_line":418,"raw_tex":"The \\emph{projection} mapping $\\proj_D$ and the \\emph{distance} function $\\dist_D$ are defined by\n\\begin{equation*}\n\t\\proj_D(v)\n\t{}\\coloneqq{}\n\t\\argmin_{z \\in D} \\|z - v\\|\n\t{}\\quad\\text{and}\\quad{}\n\t\\dist_D(v)\n\t{}\\coloneqq{}\n\t\\inf_{z \\in D} \\|z - v\\| .\n\\end{equation*}\nThe former is a set-valued mapping whenever $D$ is nonconvex, whereas the latter is always single-valued.\n","sha256_utf8":"0e33b485f3402dfe3d4ad3c895b8f7b17dcbdd0a1604b869c400b244e96ec9ff"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"集合值映射的外极限","source_units":[{"start":22027,"end":22377,"start_line":378,"end_line":391,"raw_tex":"If $S(\\bar{x})$ is nonempty, we define the \\emph{outer limit} of $S$ at $\\bar{x}$\nby means of\n\\begin{align*}\n\t\\limsup\\limits_{x\\to\\bar{x}} S(x)\n\t\\coloneqq\n\t\\{\n\t\ty\\in\\YY\n\t\t\\suchthat\n\t\t\\exists x^k\\to\\bar{x},\\,\n\t\t\\exists y^k\\to y,\\,\n\t\ty^k\\in S(x^k)\\,\\forall k\\in\\N\n\t\\}\n\\end{align*}\nand note that this is a closed superset of $S(\\bar{x})$ by definition.\n","sha256_utf8":"01bdc03ea573f7885d9bff9f0ffa021bf148bfceca11db075e99fed35cf300e0"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"集合指示函数","source_units":[{"start":23777,"end":24041,"start_line":420,"end_line":421,"raw_tex":"The \\emph{indicator} function of a set $D \\subset \\YY$ is the function $\\func{\\indicator_D}{\\YY}{\\Rinf}$ defined as $\\indicator_D(v) = 0$ if $v \\in D$, and $\\indicator_D(v) = \\infty$ otherwise.\nIf $D$ is nonempty and closed, then $\\indicator_D$ is proper and lsc.\n","sha256_utf8":"c0b8509d84dc100f86cceb8223af72bbb1fa2f8362d142612618c891fef2fe64"}]},{"id":"S015","semantic_type":"definition","kind":"definition","title":"极限法锥","source_units":[{"start":24168,"end":24455,"start_line":424,"end_line":428,"raw_tex":"Given $z \\in D$, the \\emph{limiting normal cone} to $D$ at $z$ is the closed cone\n\\begin{equation*}\n\t\\limnormalcone_D(z) \\coloneqq \\limsup_{v \\to z} \\; \\cone \\left( v - \\proj_D(v) \\right) .\n\\end{equation*}\nFor $\\tilde z\\notin D$, we formally set $\\limnormalcone_D(\\tilde z):=\\emptyset$.\n","sha256_utf8":"2b46142ba037da3ac3c18bde799a8cc0b79d1ed346389985ee81e3552609a7a2"}]},{"id":"S016","semantic_type":"definition","kind":"definition","title":"复合问题的可行性","source_units":[{"start":26560,"end":26738,"start_line":485,"end_line":488,"raw_tex":"\\begin{defin}[Feasibility]\\label{def:feasible}\n\t\tA point $x^\\ast \\in \\XX$ is called \\emph{feasible} for \\eqref{eq:P} \n\t\tif $x^\\ast \\in \\dom q$ and $c(x^\\ast) \\in D$.\n\t\\end{defin}","sha256_utf8":"83610e92391b10f082473459fe9661a44b962f179ee9b8cf7101c3c8af26cbd8"}]},{"id":"S017","semantic_type":"definition","kind":"definition","title":"渐近M驻点","source_units":[{"start":28261,"end":28982,"start_line":526,"end_line":546,"raw_tex":"\\begin{defin}[AM-stationarity]\\label{def:AMstationary}\n\t\tLet $x^\\ast \\in \\XX$ be a feasible point for \\eqref{eq:P}.\n\t\tThen, $x^\\ast$ is called an \\emph{asymptotically M-stationary} point of \\eqref{eq:P} \n\t\tif there exist sequences $\\{x^k\\}, \\{\\eta^k\\} \\subset \\XX$ and \n\t\t$\\{y^k\\}, \\{\\zeta^k\\} \\subset \\YY$ such that \n\t\t$x^k \\toattentive{q} x^\\ast$, $\\eta^k \\to 0$, $\\zeta^k \\to 0$ and\n\t\t\\begin{subequations}\n\t\t\t\\begin{align}\n\t\t\t\t-\\nabla c(x^k)^\\top y^k + \\eta^k\n\t\t\t\t{}\\in{}&\n\t\t\t\t\\partial q(x^k) ,\n\t\t\t\t\\label{eq:AMstationary:x} \\\\\n\t\t\t\ty^k \n\t\t\t\t{}\\in{}&\n\t\t\t\t\\limnormalcone_D( c(x^k) + \\zeta^k )\n\t\t\t\t\\label{eq:AMstationary:y}\n\t\t\t\\end{align}\n\t\t\t\\label{eq:AMstationary}\n\t\t\\end{subequations}\n\t\tfor all $k \\in \\N$.\n\t\\end{defin}","sha256_utf8":"54fb08dae5c387db810e276ffc75b4b17d9df778af454e22452a8b9a2101f240"}]},{"id":"S018","semantic_type":"problem_setting","kind":"problem_setting","title":"显式松弛变量重构","source_units":[{"start":38742,"end":38912,"start_line":766,"end_line":777,"raw_tex":"\\begin{equation}\n\t\\minimize_{x ,\\; s}\n\t{}\\quad{}\n\tq(x)\n\t{}\\qquad{}\n\t\\stt\n\t{}\\quad{}\n\tc(x) - s = 0\n\t,\\quad\n\ts \\in D .\n\t\\tag{P$_{\\text{S}}$}\\label{eq:ALMz:P}\n\\end{equation}","sha256_utf8":"a3862533ae61c11836ab69e2c01c3568b752736f810b12af5d7ed339f7c169bc"}]},{"id":"S019","semantic_type":"definition","kind":"definition","title":"松弛形式的增广拉格朗日函数","source_units":[{"start":41987,"end":42439,"start_line":826,"end_line":835,"raw_tex":"For some penalty parameter $\\mu>0$,\nlet us define the $\\mu$-augmented Lagrangian function $\\func{\\LLslack_\\mu}{\\XX \\times \\YY \\times \\YY}{\\Rinf}$ associated to \\eqref{eq:ALMz:P} as\n\\begin{align}\n\t\\LLslack_\\mu(x,s,y)\n\t{}\\coloneqq{}&\n\tq(x) + \\indicator_D(s) + \\innprod{y}{c(x) - s} + \\frac{1}{2\\mu} \\| c(x) - s \\|^2 \\nonumber \\\\\n\t{}={}&\n\tq(x) + \\indicator_D(s) + \\frac{1}{2\\mu} \\| c(x) + \\mu y - s \\|^2 - \\frac{\\mu}{2}\\|y\\|^2 .\n\t\\label{eq:Lz}\n\\end{align}","sha256_utf8":"4f6c79a7f43cf6b3f8d0eae7fbd5dfd461fea3f7904664dc6703a2ed0e8337f3"}]},{"id":"S020","semantic_type":"derived_fact","kind":"derived_fact","title":"增广拉格朗日函数的偏次微分","source_units":[{"start":42753,"end":42901,"start_line":838,"end_line":841,"raw_tex":"For later use, let us compute the subdifferential of $\\LLslack_\\mu$ with respect to the\nvariables $x$ and $s$:\n\\begin{subequations}\n\t\\label{eq:dLS}\n","sha256_utf8":"d45e3d51dae94b69679f62d923c66d440ff49975f7cee3016fded3bcf94ab635"},{"start":42902,"end":43182,"start_line":842,"end_line":851,"raw_tex":"\\begin{align}\n\t\t\\partial_x \\LLslack_\\mu(x,s,y) \n\t\t{}={}&\n\t\t\\partial q(x) + \\frac{1}{\\mu} \\nabla c(x)^\\top [c(x) + \\mu y - s] ,\n\t\t\\label{eq:dxLS} \\\\\n\t\t\\partial_s \\LLslack_\\mu(x,s,y) \n\t\t{}={}&\n\t\t\\limnormalcone_D(s) - \\frac{1}{\\mu} [c(x) + \\mu y - s] .\n\t\t\\label{eq:dzLS}\n\t\\end{align}","sha256_utf8":"9211706ca575b3cb3204b118168a7edde9fa11fe0b2b17924f30db2c3c904a3a"}]},{"id":"S021","semantic_type":"algorithm","kind":"algorithm","title":"带乘子保护的增广拉格朗日算法","source_units":[{"start":48317,"end":49583,"start_line":932,"end_line":961,"raw_tex":"\\begin{algorithm}[t]\n\t\\caption{Augmented Lagrangian method for \\eqref{eq:P}}%\n\t\\label{alg:ALM}%\n\t\\begin{algorithmic}[1]%\n\t\\linespread{1.2}\\selectfont%\n\t%\n\t%\n%\t\\Require $x^0 \\in \\XX$, $y^0 \\in \\YY$ %$\\varepsilon_{\\text{tol}} \\geq 0$\n\t\\Initialize Select $\\mu_0 > 0$, $\\theta,\\kappa \\in(0,1)$ and $\\Ybounded \\subset \\YY$ nonempty bounded\n\t\\item[For \\(k=0,1,2\\ldots\\)]\n\t\\State\\label{state:ALM:init}\\label{state:ALM:ysafe}%\n\t\tSelect $\\hat{y}^k \\in \\Ybounded$ and $\\varepsilon_k \\geq 0$\n\t\\State\\label{state:ALM:subproblem}%\\label{state:ALM:z}%\n\t\tCompute an $\\varepsilon_k$-stationary point $(x^k,s^k)\\in\\XX\\times D$ of $\\LL_{\\mu_k}^{\\text{S}}(\\cdot,\\cdot,\\hat{y}^k)$\n\t\\State\\label{state:ALM:y}%\n\t\tSet $y^k \\gets \\hat{y}^k + [ c(x^k) - s^k ] / \\mu_k$\n%\t\\If{$\\varepsilon_{k+1} \\leq \\varepsilon_{\\text{tol}}$ and $\\|c(x^{k+1}) - z^{k+1}\\| \\leq \\varepsilon_{\\text{tol}}$}\\label{state:ALM:ifend}%\n%\t\t\\State\\label{state:ALM:end}%\n%\t\t\t\\algfont{return} $(x^{k+1}, y^{k+1})$\n%\t\\EndIf\n\t\\If{$k = 0$ or $\\| c(x^k) - s^k \\| \\leq \\theta \\, \\| c(x^{k-1}) - s^{k-1} \\|$}\\label{state:ALM:if}\n\t\t\\State\\label{state:ALM:ifpassed}%\n\t\t\tSet $\\mu_{k+1} \\gets \\mu_k$\n\t\\Else\n\t\t\\State\\label{state:ALM:iffailed}%\n\t\t\tSelect $\\mu_{k+1} \\in (0, \\kappa \\mu_k]$\n\t\\EndIf\n\\end{algorithmic}\n%\n\\end{algorithm}","sha256_utf8":"fee140ed71cf6ba8564aaef6d6aca8d83dd16d29c433c66c2002f3f30a345d67"}]},{"id":"S022","semantic_type":"assumption","kind":"assumption","title":"子问题可解与聚点存在的分析前提","source_units":[{"start":53367,"end":53764,"start_line":1038,"end_line":1042,"raw_tex":"\tThroughout our convergence analysis, we assume that \\cref{alg:ALM} is well-defined, \nthus requiring that each subproblem at \\cref{state:ALM:subproblem} admits an approximate stationary point.\nMoreover, the following statements assume the existence of some accumulation point $x^\\ast$ \nor $(x^\\ast,s^\\ast)$\nfor a sequence $\\{ x^k \\}$ or $\\{(x^k,s^k)\\}$, respectively, generated by \\cref{alg:ALM}.\n","sha256_utf8":"2bcdaf43625b602562b756c1117e4d0579073591bc25101dd29885ba26b5f789"}]},{"id":"S023","semantic_type":"notation","kind":"notation","title":"变量空间与函数基本术语","source_units":[{"start":1907,"end":1953,"start_line":86,"end_line":87,"raw_tex":"\\newcommand{\\XX}{\\R^n}\n\\newcommand{\\YY}{\\R^m}\n","sha256_utf8":"9648c600decab7777b5c70354c474cf936b1a4f0d7c48ffc62e8d3f3222f6dd8"},{"start":19466,"end":19793,"start_line":345,"end_line":347,"raw_tex":"The effective domain of an extended real-valued function $\\func{h}{\\R^n}{\\Rinf}$ is denoted by $\\dom h \\coloneqq \\{x\\in\\R^n \\suchthat h(x) < \\infty \\}$.\nWe say that $h$ is \\emph{proper} if $\\dom h \\neq \\emptyset$\nand \\emph{lower semicontinuous} (lsc) if $h(\\bar{x}) \\leq \\liminf_{x\\to\\bar{x}} h(x)$ for all $\\bar{x} \\in \\R^n$.\n","sha256_utf8":"f9e509b7e85e3c18359c9344a6eeb269c28158faacf7060cb416b2b11f73b603"}]},{"id":"S024","semantic_type":"assumption","kind":"assumption","title":"误差趋零与可行子序列收敛假设","source_units":[{"start":58250,"end":58601,"start_line":1151,"end_line":1155,"raw_tex":"\t\tLet \\cref{ass:P} hold and consider a sequence $\\{ (x^k,s^k) \\}$ of iterates \n\t\tgenerated by \\cref{alg:ALM} with $\\varepsilon_k \\to 0$.\n\t\tLet $(x^\\ast,c(x^\\ast))$ be an accumulation point  of $\\{ (x^k,s^k) \\}$ \n\t\tfeasible to \\eqref{eq:ALMz:P} and\n\t\t$\\{ (x^k,s^k)\\}_K$ a subsequence such that $x^k \\toattentive{q}_K x^\\ast$ and $s^k \\to_K c(x^\\ast)$.\n","sha256_utf8":"b41241a684f3361036a30edb95f306cd3e2b0cfe0014accd690fa735bd81cb0f"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S023","target_id":"S006","kind":"supports"},{"source_id":"S023","target_id":"S007","kind":"supports"},{"source_id":"S023","target_id":"S009","kind":"supports"},{"source_id":"S011","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S016","kind":"supports"},{"source_id":"S006","target_id":"S018","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S017","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S015","kind":"supports"},{"source_id":"S013","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S014","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S009","target_id":"S020","kind":"supports"},{"source_id":"S015","target_id":"S020","kind":"supports"},{"source_id":"S007","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S010","target_id":"S021","kind":"supports"},{"source_id":"S022","target_id":"S003","kind":"supports"},{"source_id":"S021","target_id":"S003","kind":"supports"},{"source_id":"S020","target_id":"S003","kind":"supports"},{"source_id":"S010","target_id":"S003","kind":"supports"},{"source_id":"S024","target_id":"S003","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S024","target_id":"S005","kind":"supports"},{"source_id":"S007","target_id":"S005","kind":"supports"},{"source_id":"S008","target_id":"S005","kind":"supports"},{"source_id":"S017","target_id":"S005","kind":"supports"},{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S001","kind":"supports"}]}}
{"record_id":"2203.10015v2","paper":{"title":"Second-order optimality conditions for general nonconvex optimization problems and variational analysis of disjunctive systems","authors":["Matus Benko","Helmut Gfrerer","Jane Ye","Jin Zhang","Jinchuan Zhou"],"arxiv_id":"2203.10015","version":"v2","journal":"SIAM Journal on Optimization","doi":"10.1137/22m1484742","abstract_url":"https://arxiv.org/abs/2203.10015v2","pdf_url":"https://arxiv.org/pdf/2203.10015v2"},"context":{"raw_tex":"\\begin{flalign*}\n\\begin{split}\n\\mbox{(GP)} \\hspace{49mm} \\min & \\ \\  f(x)\\ \\ \\ {\\rm s.t. } \\ \\  g(x)\\in C.\n\\end{split}&\n\\end{flalign*}\n\nHere $C \\subset \\mathbb{R}^m$ is a closed set and $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ and $g:\\mathbb{R}^n\\rightarrow \\mathbb{R}^m$ are twice continuously differentiable functions unless otherwise specified.\n\n\nThe open unit ball is denoted by $\\mathbb{B}$ and the open ball centred at $z$ with radius $\\delta$ is denoted by $ \\mathbb{B}(z,\\delta)$.\n\n\nThe indicator function $\\delta_S:\\R^n \\to \\bar \\R := [-\\infty,+\\infty]$ of $S$\nis given as $\\delta_S(z)=0$ for $z \\in S$ and $\\delta_S(z)=+\\infty$ if $z \\notin S$.\n\n\n Let $o:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ stand for a mapping with the property that $o(t)/t \\rightarrow 0$ when $t \\downarrow 0$.\n The symbol $z^{\\prime} \\overset{S}{\\to} z$ indicates that $z^{\\prime}\\in S$\n and $z^{\\prime} \\rightarrow z$.\n\n\n  For a mapping $\\psi:\\mathbb{R}^n\\to \\mathbb{R}^d$ and $z \\in \\R^n$, we denote by\n $\\nabla \\psi(z)\\in \\mathbb{R}^{d\\times n}$ its Jacobian at $z$\n and by $\\nabla^2 \\psi(z)$ its second derivative at $z$ as defined by\n $$w^T\\nabla ^2 \\psi(z) :=\\lim_{t\\rightarrow 0} \\frac{\\nabla \\psi(z+ tw)-\\nabla \\psi( z)}{t} \\quad \\forall w \\in \\mathbb{R}^n.$$\n Hence\n %when $d=1$,  $\\nabla^2 \\psi(z)$ can be identified with the Hessian and\n $$\\nabla^2 \\psi(z)(w,w):=w^T \\nabla^2 \\psi(z) w=(w^T \\nabla^2 \\psi_1(z)w, \\dots, w^T \\nabla^2 \\psi_d(z)w)^T \\quad \\forall w\\in \\mathbb{R}^n.$$\n\n\n \\begin{definition}[Tangent and normal cones, \\cite{RoWe98}]%(\\cite{RoWe98}).\n Given $S\\subset \\mathbb{R}^n$, $z\\in S$, the\n regular/Clarke tangent cone and\n tangent/contingent cone to $S$ at $z$ are defined, respectively, by\n \\begin{eqnarray*}\n  \\widehat{T}_S(z)&:=&\n  \\liminf\\limits_{{z' \\stackrel{S}{\\to}z} \\atop {t\\downarrow 0}}\\frac{S-z'}{t}=\n  \\Big\\{ w\\in \\R^n\\,\\Big|\\, \\forall \\, t_k\\downarrow 0,\\, {z_k \\stackrel{S}{\\to}z}, \\; \\exists w_k\\to w \\\n  \\ {\\rm with}\\ \\ z_k+t_k w_k\\in S  \\Big\\},\\\\\n T_S(z)&:=&\n %\\limsup\\limits_{t\\downarrow 0}\\frac{S-x}{t}\n  \\big\\{w \\in\\R^n \\, \\big| \\, \\exists \\ t_k\\downarrow 0,\\; w_k\\to w \\ \\ {\\rm with}\n \\ \\ z+t_k w_k\\in S \\big\\}.\n \\end{eqnarray*}\n\n\n\\begin{definition}[Second subderivative, {\\cite[Defintion 13.3]{RoWe98}}]\nLet $\\varphi: \\R^n\\to \\bar \\R$, $\\varphi(z)$ be finite and $z^*\\in  \\mathbb{R}^n$. The second subderivative of $\\varphi$ at $z$ for $z^*$  is a function defined by\n \\begin{eqnarray*}\n {\\rm d}^2\\varphi(z;z^*)(w):= \\liminf\\limits_{{t\\downarrow 0} \\atop {w'\\to w}} \\frac{\\varphi(z+tw')-\\varphi(z)-t\\langle z^*, w'\\rangle}{\\frac{1}{2}t^2}\n % \\bigtriangleup_{t}^2 \\varphi (\\bar x;z^*)(w'),\n \\qquad \\forall w \\in  \\mathbb{R}^n.\n %\\\\ &&  \\mbox{ where }  \\bigtriangleup_{t}^2 \\varphi (\\bar x;z^*)(w'):=\\frac{\\varphi(\\bar{x}+tw')-\\varphi(\\bar{x})-t\\langle \\bar{v}, w'\\rangle}{\\frac{1}{2}t^2}  .\n\\end{eqnarray*}\n \\end{definition}\n\nBy definition, the second subderivative of the indicator function $\\delta_S$ of a set\n$S$ at $z\\in S$ for $z^*$ is\n\\begin{eqnarray}\n{\\rm d}^2\\delta_{S}(z;z^*)(w)\n =\\liminf\\limits_{{t\\downarrow 0} \\atop w'\\to w}\\frac{\\delta_{S}(z+tw')-\\delta_{S}(z)-t\\langle z^*, w' \\rangle }{\\frac{1}{2}t^2}\n=\\liminf\\limits_{{t\\downarrow 0, w'\\to w} \\atop { z+tw' \\in S }}\\frac{-2\\langle z^*, w' \\rangle }{t}. \\label{indicatorf}\n \\end{eqnarray}\n\nfrom the introduction. At a feasible point $\\xb$ of (GP), the critical cone is defined as\n\n\n\\[ \\mathcal{C}(\\xb):=\\{u \\in \\R^n \\mv \\nabla g(\\xb)u\\in T_C(g(\\xb)), \\nabla f(\\xb) u \\leq 0\\}\\]\n\n     and the generalized Lagrangian $L^\\alpha:\\R^n\\times\\R^m\\to \\R$ with $\\alpha\\geq 0$ is given as\n\n\n\\[\n\tL^\\alpha(x,\\lambda):=\\alpha f(x)+ g(x)^T \\lambda,\n     \\]\n\n\twhere for $\\alpha = 1$ we get the standard Lagrangian $L:=L^1$.\n\n\n\\begin{definition}[Essential local minimizer of second order]\n A point $\\bar{x}$ is said to {be} an essential local minimizer of second order for problem (GP) if $\\bar x$ is feasible and there exist $\\varepsilon>0$ and $\\delta>0$ such that\n \\[\n \\max\\{f(x)-f(\\bar{x}), {\\rm dist}(g(x),C) \\}\\geq \\varepsilon \\|x-\\bar{x}\\|^2, \\ \\ \\forall x\\in \\mathbb{B}(\\bar{x},\\delta).\n \\]\n \\end{definition}","units":[{"start":7401,"end":7535,"start_line":162,"end_line":166,"raw_tex":"\\begin{flalign*}\n\\begin{split}\n\\mbox{(GP)} \\hspace{49mm} \\min & \\ \\  f(x)\\ \\ \\ {\\rm s.t. } \\ \\  g(x)\\in C.\n\\end{split}&\n\\end{flalign*}","sha256_utf8":"c1b6a509a8807d9cebbf9a471012494f1b3b9ae0a23fc77d4f5a33182725fe1a"},{"start":7536,"end":7746,"start_line":167,"end_line":167,"raw_tex":"Here $C \\subset \\mathbb{R}^m$ is a closed set and $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ and $g:\\mathbb{R}^n\\rightarrow \\mathbb{R}^m$ are twice continuously differentiable functions unless otherwise specified.\n","sha256_utf8":"a73622847f5892c9fb15fa73ea30b2fe9f6112d4c7aa5b9621e94327a056e9bd"},{"start":21188,"end":21327,"start_line":272,"end_line":272,"raw_tex":"The open unit ball is denoted by $\\mathbb{B}$ and the open ball centred at $z$ with radius $\\delta$ is denoted by $ \\mathbb{B}(z,\\delta)$.\n","sha256_utf8":"850261cbef54048eef6f420717934786c28c64eb5dcdef5ea97e0e795223138b"},{"start":21765,"end":21929,"start_line":278,"end_line":279,"raw_tex":"The indicator function $\\delta_S:\\R^n \\to \\bar \\R := [-\\infty,+\\infty]$ of $S$\nis given as $\\delta_S(z)=0$ for $z \\in S$ and $\\delta_S(z)=+\\infty$ if $z \\notin S$.\n","sha256_utf8":"4ff04037ebdff50fba915e820cd8e8e972906675dae94fdb7be5a872c1eed5bc"},{"start":22492,"end":22737,"start_line":284,"end_line":286,"raw_tex":" Let $o:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ stand for a mapping with the property that $o(t)/t \\rightarrow 0$ when $t \\downarrow 0$.\n The symbol $z^{\\prime} \\overset{S}{\\to} z$ indicates that $z^{\\prime}\\in S$\n and $z^{\\prime} \\rightarrow z$.\n","sha256_utf8":"c7d9db651f3830083fb3b9bacd95f1e619900681fd21f4fb65ddfb79601bc0c9"},{"start":22993,"end":23563,"start_line":290,"end_line":296,"raw_tex":"  For a mapping $\\psi:\\mathbb{R}^n\\to \\mathbb{R}^d$ and $z \\in \\R^n$, we denote by\n $\\nabla \\psi(z)\\in \\mathbb{R}^{d\\times n}$ its Jacobian at $z$\n and by $\\nabla^2 \\psi(z)$ its second derivative at $z$ as defined by\n $$w^T\\nabla ^2 \\psi(z) :=\\lim_{t\\rightarrow 0} \\frac{\\nabla \\psi(z+ tw)-\\nabla \\psi( z)}{t} \\quad \\forall w \\in \\mathbb{R}^n.$$\n Hence\n %when $d=1$,  $\\nabla^2 \\psi(z)$ can be identified with the Hessian and\n $$\\nabla^2 \\psi(z)(w,w):=w^T \\nabla^2 \\psi(z) w=(w^T \\nabla^2 \\psi_1(z)w, \\dots, w^T \\nabla^2 \\psi_d(z)w)^T \\quad \\forall w\\in \\mathbb{R}^n.$$\n","sha256_utf8":"8dea54eb006d5fccd6804c03047849eb9295a37e3d31c8dae5aa88b0b8afd633"},{"start":23759,"end":24445,"start_line":303,"end_line":316,"raw_tex":" \\begin{definition}[Tangent and normal cones, \\cite{RoWe98}]%(\\cite{RoWe98}).\n Given $S\\subset \\mathbb{R}^n$, $z\\in S$, the\n regular/Clarke tangent cone and\n tangent/contingent cone to $S$ at $z$ are defined, respectively, by\n \\begin{eqnarray*}\n  \\widehat{T}_S(z)&:=&\n  \\liminf\\limits_{{z' \\stackrel{S}{\\to}z} \\atop {t\\downarrow 0}}\\frac{S-z'}{t}=\n  \\Big\\{ w\\in \\R^n\\,\\Big|\\, \\forall \\, t_k\\downarrow 0,\\, {z_k \\stackrel{S}{\\to}z}, \\; \\exists w_k\\to w \\\n  \\ {\\rm with}\\ \\ z_k+t_k w_k\\in S  \\Big\\},\\\\\n T_S(z)&:=&\n %\\limsup\\limits_{t\\downarrow 0}\\frac{S-x}{t}\n  \\big\\{w \\in\\R^n \\, \\big| \\, \\exists \\ t_k\\downarrow 0,\\; w_k\\to w \\ \\ {\\rm with}\n \\ \\ z+t_k w_k\\in S \\big\\}.\n \\end{eqnarray*}\n","sha256_utf8":"8a4b645340e839be48746dc43b892a23dc39ed39389ace35c8c2a28e1cdcf336"},{"start":54627,"end":55320,"start_line":781,"end_line":789,"raw_tex":"\\begin{definition}[Second subderivative, {\\cite[Defintion 13.3]{RoWe98}}]\nLet $\\varphi: \\R^n\\to \\bar \\R$, $\\varphi(z)$ be finite and $z^*\\in  \\mathbb{R}^n$. The second subderivative of $\\varphi$ at $z$ for $z^*$  is a function defined by\n \\begin{eqnarray*}\n {\\rm d}^2\\varphi(z;z^*)(w):= \\liminf\\limits_{{t\\downarrow 0} \\atop {w'\\to w}} \\frac{\\varphi(z+tw')-\\varphi(z)-t\\langle z^*, w'\\rangle}{\\frac{1}{2}t^2}\n % \\bigtriangleup_{t}^2 \\varphi (\\bar x;z^*)(w'),\n \\qquad \\forall w \\in  \\mathbb{R}^n.\n %\\\\ &&  \\mbox{ where }  \\bigtriangleup_{t}^2 \\varphi (\\bar x;z^*)(w'):=\\frac{\\varphi(\\bar{x}+tw')-\\varphi(\\bar{x})-t\\langle \\bar{v}, w'\\rangle}{\\frac{1}{2}t^2}  .\n\\end{eqnarray*}\n \\end{definition}","sha256_utf8":"67386a0bf774e74e42fa84e25ff8f6abff447bee27eb7d54df2162e2d21d87c0"},{"start":55504,"end":55931,"start_line":792,"end_line":798,"raw_tex":"By definition, the second subderivative of the indicator function $\\delta_S$ of a set\n$S$ at $z\\in S$ for $z^*$ is\n\\begin{eqnarray}\n{\\rm d}^2\\delta_{S}(z;z^*)(w)\n =\\liminf\\limits_{{t\\downarrow 0} \\atop w'\\to w}\\frac{\\delta_{S}(z+tw')-\\delta_{S}(z)-t\\langle z^*, w' \\rangle }{\\frac{1}{2}t^2}\n=\\liminf\\limits_{{t\\downarrow 0, w'\\to w} \\atop { z+tw' \\in S }}\\frac{-2\\langle z^*, w' \\rangle }{t}. \\label{indicatorf}\n \\end{eqnarray}","sha256_utf8":"6024eacdbe58c02f26c3d4c2fdebeeaa74ade160b784d0ccc9d9ab15b95dd95e"},{"start":62374,"end":62464,"start_line":894,"end_line":894,"raw_tex":"from the introduction. At a feasible point $\\xb$ of (GP), the critical cone is defined as\n","sha256_utf8":"60d4412a6497966024253f19927c16b21ec86fa89069cc8ab3c0381d9ad90b28"},{"start":62468,"end":62563,"start_line":895,"end_line":895,"raw_tex":"\\[ \\mathcal{C}(\\xb):=\\{u \\in \\R^n \\mv \\nabla g(\\xb)u\\in T_C(g(\\xb)), \\nabla f(\\xb) u \\leq 0\\}\\]","sha256_utf8":"e4ff8b39ad5a18cd96bd56a3920e6acb613f19d4f26dc113ef2573737e6cb1e3"},{"start":62564,"end":62664,"start_line":896,"end_line":896,"raw_tex":"     and the generalized Lagrangian $L^\\alpha:\\R^n\\times\\R^m\\to \\R$ with $\\alpha\\geq 0$ is given as\n","sha256_utf8":"8db35aa3a4af8a7ee39b38429d1f22b901f910446ccb6c5cba08b27232a5f1ac"},{"start":62669,"end":62730,"start_line":897,"end_line":899,"raw_tex":"\\[\n\tL^\\alpha(x,\\lambda):=\\alpha f(x)+ g(x)^T \\lambda,\n     \\]","sha256_utf8":"dc30e0f1a2c506d9e93dfb98d34d5c6a0819081bb5149b0ec4227a7504e4042d"},{"start":62731,"end":62796,"start_line":900,"end_line":900,"raw_tex":"\twhere for $\\alpha = 1$ we get the standard Lagrangian $L:=L^1$.\n","sha256_utf8":"64af5cf129ac413599d04cfbc0fb346a94bc53d20723a7983a6378526e4dd421"},{"start":65933,"end":66322,"start_line":939,"end_line":944,"raw_tex":"\\begin{definition}[Essential local minimizer of second order]\n A point $\\bar{x}$ is said to {be} an essential local minimizer of second order for problem (GP) if $\\bar x$ is feasible and there exist $\\varepsilon>0$ and $\\delta>0$ such that\n \\[\n \\max\\{f(x)-f(\\bar{x}), {\\rm dist}(g(x),C) \\}\\geq \\varepsilon \\|x-\\bar{x}\\|^2, \\ \\ \\forall x\\in \\mathbb{B}(\\bar{x},\\delta).\n \\]\n \\end{definition}","sha256_utf8":"ba5b938b27dedc2aae007574f03f009ae2d67c1ebb8703df056d02431b50ace4"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{Theorem5.9}\n Let $\\bar x$ be a feasible point of problem (GP).\n Suppose that for every $u\\in \\mathcal{C}(\\bar x ) \\backslash \\{0\\}$ there is $\\alpha\\geq 0$ and $\\lambda\\in \\R^m$ such that\n \\begin{equation}\\label{Alpha}\n \\nabla_x L^\\alpha (\\bar x,\\lambda)=0\\end{equation}\n  and\n  \\begin{equation}\\label{alpha-sufficient}\n   \\nabla^2_{xx}L^\\alpha(\\bar{x},\\lambda)(u,u)+{\\rm d}^2\\delta_{C}(g(\\bar{x});\\lambda)(\\nabla g(\\bar{x})u)>0.\n  \\end{equation}\n  Then $\\bar{x}$ is an essential local minimizer of second order.\n \\end{theorem}","units":[{"start":66323,"end":66872,"start_line":945,"end_line":955,"raw_tex":"\\begin{theorem}\\label{Theorem5.9}\n Let $\\bar x$ be a feasible point of problem (GP).\n Suppose that for every $u\\in \\mathcal{C}(\\bar x ) \\backslash \\{0\\}$ there is $\\alpha\\geq 0$ and $\\lambda\\in \\R^m$ such that\n \\begin{equation}\\label{Alpha}\n \\nabla_x L^\\alpha (\\bar x,\\lambda)=0\\end{equation}\n  and\n  \\begin{equation}\\label{alpha-sufficient}\n   \\nabla^2_{xx}L^\\alpha(\\bar{x},\\lambda)(u,u)+{\\rm d}^2\\delta_{C}(g(\\bar{x});\\lambda)(\\nabla g(\\bar{x})u)>0.\n  \\end{equation}\n  Then $\\bar{x}$ is an essential local minimizer of second order.\n \\end{theorem}","sha256_utf8":"c0e5e06299152070e48c4dd407cd498c17ae76d098e906b66c98043bccd6b938"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{proof}\n By contradiction, if $\\bar{x}$ is not an essential local minimizer of second order, then there exists a sequence $x_k$ converging to $\\xb$ such that\n\\begin{eqnarray}\n&&  f(x_k)-f(\\bar{x})\\leq o(\\|x_k-\\bar{x}\\|^2), \\label{cqg}\\\\\n&&  {\\rm dist}(g(x_k),C)\\leq o(\\|x_k-\\bar{x}\\|^2).\\label{distance}\n \\end{eqnarray}\n Let $t_k:=\\|x_k-\\bar{x}\\|$ and $u_k:=(x_k-\\bar{x})/t_k$.\n We assume without loss of generality that $u_k$ is converging to $u$.\n From equation \\eqref{cqg} it readily follows that $\\nabla f(\\xb)u\\leq 0$ and\n  \\begin{equation}\\label{f-inequality}\n \\liminf\\limits_{k}-\\frac{f(x_k)-f(\\bar{x})}{\\frac{1}{2}t_k^2}\\geq 0.\n \\end{equation}\n By (\\ref{distance}), there exists {$r_k\\in \\R^d$} such that $\\|r_k\\| \\to 0$ and\n $g(x_k)+t^2_k r_k\\in C$.\n By Taylor's expansion, since $x_k=\\bar x+t_ku_k$, we have\n \\[\nv_k:= \\frac{g(x_k)+ t^2_kr_k - g(\\xb)}{t_k} = \\frac{\\nabla g(\\bar{x})t_ku_k +o(t_k) +  t^2_kr_k}{t_k} = \\nabla g(\\bar{x})u_k +\\frac{o(t_k)}{t_k}{+t_kr_k} \\ \\to \\ \\nabla g(\\bar{x})u.\\]\nMoreover, $g(\\bar{x})+t_kv_k=g(x_k)+ t^2_kr_k\\in C$ and so $\\nabla g(\\bar{x})u\\in T_C(g(\\xb))$ follows.\nThus $u\\in \\mathcal{C}(\\xb)\\setminus\\{0\\}$ and the assumption of the theorem yields the existence of $\\alpha\\geq 0$ and $\\lambda\\in\\R^m$ satisfying \\eqref{Alpha} and \\eqref{alpha-sufficient}.\nUsing \\eqref{indicatorf}, \\eqref{f-inequality}, $\\|r_k\\| \\to 0$, and \\eqref{Alpha}, however, we obtain\n \\begin{eqnarray*} %\\nonumber\n\t{\\rm d}^2\\delta_{C}(g(\\bar{x});\\lambda)(\\nabla g(\\bar{x})u)\n\t&=&\n\t\\liminf\\limits_{{t\\downarrow 0, v'\\to \\nabla g(\\bar{x})u} \\atop { g(\\bar{x})+tv'\\in C }}\\frac{-\\langle \\lambda, v' \\rangle }{\\frac{1}{2}t}\n\t \\leq\n\t\\liminf\\limits_{k}\\frac{-t_k\\langle \\lambda, v_k \\rangle }{\\frac{1}{2}t_k^2} \\\\\n\t&=&\n\t\\liminf\\limits_{k}\\frac{-\\langle \\lambda, g(x_k)+ t^2_kr_k-g(\\bar{x})  \\rangle }{\\frac{1}{2}t_k^2} \\\\%\\label{g-inequality}\n\t&\\leq&\n\t\\liminf\\limits_{k}-\\alpha\\frac{f(x_k)-f(\\bar{x})}{\\frac{1}{2}t^2_k}+ \\liminf\\limits_{k}-\\frac{\\langle \\lambda, g(x_k)-g(\\bar{x}) \\rangle }{\\frac{1}{2}t^2_k}\\\\\n\t&\\leq&\n\t\\liminf\\limits_{k}-\\frac{L^\\alpha (x_k,\\lambda)-L^\\alpha(\\bar{x},\\lambda)}{\\frac{1}{2}t_k^2}\\\\\n\t&=&\n\t\\liminf\\limits_{k}-\\frac{\\nabla_x L^\\alpha(\\bar{x},\\lambda)(t_ku_k)+\\frac 12 \\nabla^2_{xx} L^\\alpha(\\bar{x},\\lambda)(t_ku_k,t_ku_k)+\\oo(t_k^2)}{\\frac{1}{2}t_k^2}\\\\\n\t&=&\n\t\\liminf\\limits_{k}-\\nabla^2_{xx} L^\\alpha(\\bar{x},\\lambda)(u_k,u_k)=-\\nabla^2_{xx}L^\\alpha(\\bar{x},\\lambda)(u,u),\n \\end{eqnarray*}\nwhich contradicts \\eqref{alpha-sufficient}. This completes the proof.\n\\end{proof}","units":[{"start":66874,"end":69398,"start_line":956,"end_line":994,"raw_tex":"\\begin{proof}\n By contradiction, if $\\bar{x}$ is not an essential local minimizer of second order, then there exists a sequence $x_k$ converging to $\\xb$ such that\n\\begin{eqnarray}\n&&  f(x_k)-f(\\bar{x})\\leq o(\\|x_k-\\bar{x}\\|^2), \\label{cqg}\\\\\n&&  {\\rm dist}(g(x_k),C)\\leq o(\\|x_k-\\bar{x}\\|^2).\\label{distance}\n \\end{eqnarray}\n Let $t_k:=\\|x_k-\\bar{x}\\|$ and $u_k:=(x_k-\\bar{x})/t_k$.\n We assume without loss of generality that $u_k$ is converging to $u$.\n From equation \\eqref{cqg} it readily follows that $\\nabla f(\\xb)u\\leq 0$ and\n  \\begin{equation}\\label{f-inequality}\n \\liminf\\limits_{k}-\\frac{f(x_k)-f(\\bar{x})}{\\frac{1}{2}t_k^2}\\geq 0.\n \\end{equation}\n By (\\ref{distance}), there exists {$r_k\\in \\R^d$} such that $\\|r_k\\| \\to 0$ and\n $g(x_k)+t^2_k r_k\\in C$.\n By Taylor's expansion, since $x_k=\\bar x+t_ku_k$, we have\n \\[\nv_k:= \\frac{g(x_k)+ t^2_kr_k - g(\\xb)}{t_k} = \\frac{\\nabla g(\\bar{x})t_ku_k +o(t_k) +  t^2_kr_k}{t_k} = \\nabla g(\\bar{x})u_k +\\frac{o(t_k)}{t_k}{+t_kr_k} \\ \\to \\ \\nabla g(\\bar{x})u.\\]\nMoreover, $g(\\bar{x})+t_kv_k=g(x_k)+ t^2_kr_k\\in C$ and so $\\nabla g(\\bar{x})u\\in T_C(g(\\xb))$ follows.\nThus $u\\in \\mathcal{C}(\\xb)\\setminus\\{0\\}$ and the assumption of the theorem yields the existence of $\\alpha\\geq 0$ and $\\lambda\\in\\R^m$ satisfying \\eqref{Alpha} and \\eqref{alpha-sufficient}.\nUsing \\eqref{indicatorf}, \\eqref{f-inequality}, $\\|r_k\\| \\to 0$, and \\eqref{Alpha}, however, we obtain\n \\begin{eqnarray*} %\\nonumber\n\t{\\rm d}^2\\delta_{C}(g(\\bar{x});\\lambda)(\\nabla g(\\bar{x})u)\n\t&=&\n\t\\liminf\\limits_{{t\\downarrow 0, v'\\to \\nabla g(\\bar{x})u} \\atop { g(\\bar{x})+tv'\\in C }}\\frac{-\\langle \\lambda, v' \\rangle }{\\frac{1}{2}t}\n\t \\leq\n\t\\liminf\\limits_{k}\\frac{-t_k\\langle \\lambda, v_k \\rangle }{\\frac{1}{2}t_k^2} \\\\\n\t&=&\n\t\\liminf\\limits_{k}\\frac{-\\langle \\lambda, g(x_k)+ t^2_kr_k-g(\\bar{x})  \\rangle }{\\frac{1}{2}t_k^2} \\\\%\\label{g-inequality}\n\t&\\leq&\n\t\\liminf\\limits_{k}-\\alpha\\frac{f(x_k)-f(\\bar{x})}{\\frac{1}{2}t^2_k}+ \\liminf\\limits_{k}-\\frac{\\langle \\lambda, g(x_k)-g(\\bar{x}) \\rangle }{\\frac{1}{2}t^2_k}\\\\\n\t&\\leq&\n\t\\liminf\\limits_{k}-\\frac{L^\\alpha (x_k,\\lambda)-L^\\alpha(\\bar{x},\\lambda)}{\\frac{1}{2}t_k^2}\\\\\n\t&=&\n\t\\liminf\\limits_{k}-\\frac{\\nabla_x L^\\alpha(\\bar{x},\\lambda)(t_ku_k)+\\frac 12 \\nabla^2_{xx} L^\\alpha(\\bar{x},\\lambda)(t_ku_k,t_ku_k)+\\oo(t_k^2)}{\\frac{1}{2}t_k^2}\\\\\n\t&=&\n\t\\liminf\\limits_{k}-\\nabla^2_{xx} L^\\alpha(\\bar{x},\\lambda)(u_k,u_k)=-\\nabla^2_{xx}L^\\alpha(\\bar{x},\\lambda)(u,u),\n \\end{eqnarray*}\nwhich contradicts \\eqref{alpha-sufficient}. This completes the proof.\n\\end{proof}","sha256_utf8":"7b68f7160c3ba306db705536ef95692d170fa8289ef8a0ec9d2d3026156d3448"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"一般约束优化问题","source_units":[{"start":7401,"end":7535,"start_line":162,"end_line":166,"raw_tex":"\\begin{flalign*}\n\\begin{split}\n\\mbox{(GP)} \\hspace{49mm} \\min & \\ \\  f(x)\\ \\ \\ {\\rm s.t. } \\ \\  g(x)\\in C.\n\\end{split}&\n\\end{flalign*}","sha256_utf8":"c1b6a509a8807d9cebbf9a471012494f1b3b9ae0a23fc77d4f5a33182725fe1a"},{"start":7536,"end":7746,"start_line":167,"end_line":167,"raw_tex":"Here $C \\subset \\mathbb{R}^m$ is a closed set and $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ and $g:\\mathbb{R}^n\\rightarrow \\mathbb{R}^m$ are twice continuously differentiable functions unless otherwise specified.\n","sha256_utf8":"a73622847f5892c9fb15fa73ea30b2fe9f6112d4c7aa5b9621e94327a056e9bd"}]},{"id":"S002","semantic_type":"notation","kind":"notation","title":"邻域与微分记号","source_units":[{"start":21188,"end":21327,"start_line":272,"end_line":272,"raw_tex":"The open unit ball is denoted by $\\mathbb{B}$ and the open ball centred at $z$ with radius $\\delta$ is denoted by $ \\mathbb{B}(z,\\delta)$.\n","sha256_utf8":"850261cbef54048eef6f420717934786c28c64eb5dcdef5ea97e0e795223138b"},{"start":22492,"end":22737,"start_line":284,"end_line":286,"raw_tex":" Let $o:\\mathbb{R}_+\\rightarrow \\mathbb{R}^n$ stand for a mapping with the property that $o(t)/t \\rightarrow 0$ when $t \\downarrow 0$.\n The symbol $z^{\\prime} \\overset{S}{\\to} z$ indicates that $z^{\\prime}\\in S$\n and $z^{\\prime} \\rightarrow z$.\n","sha256_utf8":"c7d9db651f3830083fb3b9bacd95f1e619900681fd21f4fb65ddfb79601bc0c9"},{"start":22993,"end":23563,"start_line":290,"end_line":296,"raw_tex":"  For a mapping $\\psi:\\mathbb{R}^n\\to \\mathbb{R}^d$ and $z \\in \\R^n$, we denote by\n $\\nabla \\psi(z)\\in \\mathbb{R}^{d\\times n}$ its Jacobian at $z$\n and by $\\nabla^2 \\psi(z)$ its second derivative at $z$ as defined by\n $$w^T\\nabla ^2 \\psi(z) :=\\lim_{t\\rightarrow 0} \\frac{\\nabla \\psi(z+ tw)-\\nabla \\psi( z)}{t} \\quad \\forall w \\in \\mathbb{R}^n.$$\n Hence\n %when $d=1$,  $\\nabla^2 \\psi(z)$ can be identified with the Hessian and\n $$\\nabla^2 \\psi(z)(w,w):=w^T \\nabla^2 \\psi(z) w=(w^T \\nabla^2 \\psi_1(z)w, \\dots, w^T \\nabla^2 \\psi_d(z)w)^T \\quad \\forall w\\in \\mathbb{R}^n.$$\n","sha256_utf8":"8dea54eb006d5fccd6804c03047849eb9295a37e3d31c8dae5aa88b0b8afd633"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"集合示性函数","source_units":[{"start":21765,"end":21929,"start_line":278,"end_line":279,"raw_tex":"The indicator function $\\delta_S:\\R^n \\to \\bar \\R := [-\\infty,+\\infty]$ of $S$\nis given as $\\delta_S(z)=0$ for $z \\in S$ and $\\delta_S(z)=+\\infty$ if $z \\notin S$.\n","sha256_utf8":"4ff04037ebdff50fba915e820cd8e8e972906675dae94fdb7be5a872c1eed5bc"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"切锥的序列定义","source_units":[{"start":23759,"end":24445,"start_line":303,"end_line":316,"raw_tex":" \\begin{definition}[Tangent and normal cones, \\cite{RoWe98}]%(\\cite{RoWe98}).\n Given $S\\subset \\mathbb{R}^n$, $z\\in S$, the\n regular/Clarke tangent cone and\n tangent/contingent cone to $S$ at $z$ are defined, respectively, by\n \\begin{eqnarray*}\n  \\widehat{T}_S(z)&:=&\n  \\liminf\\limits_{{z' \\stackrel{S}{\\to}z} \\atop {t\\downarrow 0}}\\frac{S-z'}{t}=\n  \\Big\\{ w\\in \\R^n\\,\\Big|\\, \\forall \\, t_k\\downarrow 0,\\, {z_k \\stackrel{S}{\\to}z}, \\; \\exists w_k\\to w \\\n  \\ {\\rm with}\\ \\ z_k+t_k w_k\\in S  \\Big\\},\\\\\n T_S(z)&:=&\n %\\limsup\\limits_{t\\downarrow 0}\\frac{S-x}{t}\n  \\big\\{w \\in\\R^n \\, \\big| \\, \\exists \\ t_k\\downarrow 0,\\; w_k\\to w \\ \\ {\\rm with}\n \\ \\ z+t_k w_k\\in S \\big\\}.\n \\end{eqnarray*}\n","sha256_utf8":"8a4b645340e839be48746dc43b892a23dc39ed39389ace35c8c2a28e1cdcf336"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"二阶次导数","source_units":[{"start":54627,"end":55320,"start_line":781,"end_line":789,"raw_tex":"\\begin{definition}[Second subderivative, {\\cite[Defintion 13.3]{RoWe98}}]\nLet $\\varphi: \\R^n\\to \\bar \\R$, $\\varphi(z)$ be finite and $z^*\\in  \\mathbb{R}^n$. The second subderivative of $\\varphi$ at $z$ for $z^*$  is a function defined by\n \\begin{eqnarray*}\n {\\rm d}^2\\varphi(z;z^*)(w):= \\liminf\\limits_{{t\\downarrow 0} \\atop {w'\\to w}} \\frac{\\varphi(z+tw')-\\varphi(z)-t\\langle z^*, w'\\rangle}{\\frac{1}{2}t^2}\n % \\bigtriangleup_{t}^2 \\varphi (\\bar x;z^*)(w'),\n \\qquad \\forall w \\in  \\mathbb{R}^n.\n %\\\\ &&  \\mbox{ where }  \\bigtriangleup_{t}^2 \\varphi (\\bar x;z^*)(w'):=\\frac{\\varphi(\\bar{x}+tw')-\\varphi(\\bar{x})-t\\langle \\bar{v}, w'\\rangle}{\\frac{1}{2}t^2}  .\n\\end{eqnarray*}\n \\end{definition}","sha256_utf8":"67386a0bf774e74e42fa84e25ff8f6abff447bee27eb7d54df2162e2d21d87c0"}]},{"id":"S006","semantic_type":"derived_fact","kind":"derived_fact","title":"示性函数的二阶次导数表达","source_units":[{"start":55504,"end":55931,"start_line":792,"end_line":798,"raw_tex":"By definition, the second subderivative of the indicator function $\\delta_S$ of a set\n$S$ at $z\\in S$ for $z^*$ is\n\\begin{eqnarray}\n{\\rm d}^2\\delta_{S}(z;z^*)(w)\n =\\liminf\\limits_{{t\\downarrow 0} \\atop w'\\to w}\\frac{\\delta_{S}(z+tw')-\\delta_{S}(z)-t\\langle z^*, w' \\rangle }{\\frac{1}{2}t^2}\n=\\liminf\\limits_{{t\\downarrow 0, w'\\to w} \\atop { z+tw' \\in S }}\\frac{-2\\langle z^*, w' \\rangle }{t}. \\label{indicatorf}\n \\end{eqnarray}","sha256_utf8":"6024eacdbe58c02f26c3d4c2fdebeeaa74ade160b784d0ccc9d9ab15b95dd95e"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"临界方向锥","source_units":[{"start":62374,"end":62464,"start_line":894,"end_line":894,"raw_tex":"from the introduction. At a feasible point $\\xb$ of (GP), the critical cone is defined as\n","sha256_utf8":"60d4412a6497966024253f19927c16b21ec86fa89069cc8ab3c0381d9ad90b28"},{"start":62468,"end":62563,"start_line":895,"end_line":895,"raw_tex":"\\[ \\mathcal{C}(\\xb):=\\{u \\in \\R^n \\mv \\nabla g(\\xb)u\\in T_C(g(\\xb)), \\nabla f(\\xb) u \\leq 0\\}\\]","sha256_utf8":"e4ff8b39ad5a18cd96bd56a3920e6acb613f19d4f26dc113ef2573737e6cb1e3"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"广义拉格朗日函数","source_units":[{"start":62564,"end":62664,"start_line":896,"end_line":896,"raw_tex":"     and the generalized Lagrangian $L^\\alpha:\\R^n\\times\\R^m\\to \\R$ with $\\alpha\\geq 0$ is given as\n","sha256_utf8":"8db35aa3a4af8a7ee39b38429d1f22b901f910446ccb6c5cba08b27232a5f1ac"},{"start":62669,"end":62730,"start_line":897,"end_line":899,"raw_tex":"\\[\n\tL^\\alpha(x,\\lambda):=\\alpha f(x)+ g(x)^T \\lambda,\n     \\]","sha256_utf8":"dc30e0f1a2c506d9e93dfb98d34d5c6a0819081bb5149b0ec4227a7504e4042d"},{"start":62731,"end":62796,"start_line":900,"end_line":900,"raw_tex":"\twhere for $\\alpha = 1$ we get the standard Lagrangian $L:=L^1$.\n","sha256_utf8":"64af5cf129ac413599d04cfbc0fb346a94bc53d20723a7983a6378526e4dd421"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"二阶本质局部极小点","source_units":[{"start":65933,"end":66322,"start_line":939,"end_line":944,"raw_tex":"\\begin{definition}[Essential local minimizer of second order]\n A point $\\bar{x}$ is said to {be} an essential local minimizer of second order for problem (GP) if $\\bar x$ is feasible and there exist $\\varepsilon>0$ and $\\delta>0$ such that\n \\[\n \\max\\{f(x)-f(\\bar{x}), {\\rm dist}(g(x),C) \\}\\geq \\varepsilon \\|x-\\bar{x}\\|^2, \\ \\ \\forall x\\in \\mathbb{B}(\\bar{x},\\delta).\n \\]\n \\end{definition}","sha256_utf8":"ba5b938b27dedc2aae007574f03f009ae2d67c1ebb8703df056d02431b50ace4"}]},{"id":"S010","semantic_type":"theorem","kind":"theorem","title":"一般非凸问题的二阶充分条件","source_units":[{"start":66323,"end":66872,"start_line":945,"end_line":955,"raw_tex":"\\begin{theorem}\\label{Theorem5.9}\n Let $\\bar x$ be a feasible point of problem (GP).\n Suppose that for every $u\\in \\mathcal{C}(\\bar x ) \\backslash \\{0\\}$ there is $\\alpha\\geq 0$ and $\\lambda\\in \\R^m$ such that\n \\begin{equation}\\label{Alpha}\n \\nabla_x L^\\alpha (\\bar x,\\lambda)=0\\end{equation}\n  and\n  \\begin{equation}\\label{alpha-sufficient}\n   \\nabla^2_{xx}L^\\alpha(\\bar{x},\\lambda)(u,u)+{\\rm d}^2\\delta_{C}(g(\\bar{x});\\lambda)(\\nabla g(\\bar{x})u)>0.\n  \\end{equation}\n  Then $\\bar{x}$ is an essential local minimizer of second order.\n \\end{theorem}","sha256_utf8":"c0e5e06299152070e48c4dd407cd498c17ae76d098e906b66c98043bccd6b938"}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","title":"可行性与逐方向严格二阶条件","source_units":[{"start":66357,"end":66792,"start_line":946,"end_line":953,"raw_tex":" Let $\\bar x$ be a feasible point of problem (GP).\n Suppose that for every $u\\in \\mathcal{C}(\\bar x ) \\backslash \\{0\\}$ there is $\\alpha\\geq 0$ and $\\lambda\\in \\R^m$ such that\n \\begin{equation}\\label{Alpha}\n \\nabla_x L^\\alpha (\\bar x,\\lambda)=0\\end{equation}\n  and\n  \\begin{equation}\\label{alpha-sufficient}\n   \\nabla^2_{xx}L^\\alpha(\\bar{x},\\lambda)(u,u)+{\\rm d}^2\\delta_{C}(g(\\bar{x});\\lambda)(\\nabla g(\\bar{x})u)>0.\n  \\end{equation}\n","sha256_utf8":"cb830105b4226b347717553784c3c0b494a5f5e96aa950aa6045079c0cb20e7c"}]},{"id":"S012","semantic_type":"proof","kind":"proof","title":"二阶充分条件的完整证明","source_units":[{"start":66874,"end":69398,"start_line":956,"end_line":994,"raw_tex":"\\begin{proof}\n By contradiction, if $\\bar{x}$ is not an essential local minimizer of second order, then there exists a sequence $x_k$ converging to $\\xb$ such that\n\\begin{eqnarray}\n&&  f(x_k)-f(\\bar{x})\\leq o(\\|x_k-\\bar{x}\\|^2), \\label{cqg}\\\\\n&&  {\\rm dist}(g(x_k),C)\\leq o(\\|x_k-\\bar{x}\\|^2).\\label{distance}\n \\end{eqnarray}\n Let $t_k:=\\|x_k-\\bar{x}\\|$ and $u_k:=(x_k-\\bar{x})/t_k$.\n We assume without loss of generality that $u_k$ is converging to $u$.\n From equation \\eqref{cqg} it readily follows that $\\nabla f(\\xb)u\\leq 0$ and\n  \\begin{equation}\\label{f-inequality}\n \\liminf\\limits_{k}-\\frac{f(x_k)-f(\\bar{x})}{\\frac{1}{2}t_k^2}\\geq 0.\n \\end{equation}\n By (\\ref{distance}), there exists {$r_k\\in \\R^d$} such that $\\|r_k\\| \\to 0$ and\n $g(x_k)+t^2_k r_k\\in C$.\n By Taylor's expansion, since $x_k=\\bar x+t_ku_k$, we have\n \\[\nv_k:= \\frac{g(x_k)+ t^2_kr_k - g(\\xb)}{t_k} = \\frac{\\nabla g(\\bar{x})t_ku_k +o(t_k) +  t^2_kr_k}{t_k} = \\nabla g(\\bar{x})u_k +\\frac{o(t_k)}{t_k}{+t_kr_k} \\ \\to \\ \\nabla g(\\bar{x})u.\\]\nMoreover, $g(\\bar{x})+t_kv_k=g(x_k)+ t^2_kr_k\\in C$ and so $\\nabla g(\\bar{x})u\\in T_C(g(\\xb))$ follows.\nThus $u\\in \\mathcal{C}(\\xb)\\setminus\\{0\\}$ and the assumption of the theorem yields the existence of $\\alpha\\geq 0$ and $\\lambda\\in\\R^m$ satisfying \\eqref{Alpha} and \\eqref{alpha-sufficient}.\nUsing \\eqref{indicatorf}, \\eqref{f-inequality}, $\\|r_k\\| \\to 0$, and \\eqref{Alpha}, however, we obtain\n \\begin{eqnarray*} %\\nonumber\n\t{\\rm d}^2\\delta_{C}(g(\\bar{x});\\lambda)(\\nabla g(\\bar{x})u)\n\t&=&\n\t\\liminf\\limits_{{t\\downarrow 0, v'\\to \\nabla g(\\bar{x})u} \\atop { g(\\bar{x})+tv'\\in C }}\\frac{-\\langle \\lambda, v' \\rangle }{\\frac{1}{2}t}\n\t \\leq\n\t\\liminf\\limits_{k}\\frac{-t_k\\langle \\lambda, v_k \\rangle }{\\frac{1}{2}t_k^2} \\\\\n\t&=&\n\t\\liminf\\limits_{k}\\frac{-\\langle \\lambda, g(x_k)+ t^2_kr_k-g(\\bar{x})  \\rangle }{\\frac{1}{2}t_k^2} \\\\%\\label{g-inequality}\n\t&\\leq&\n\t\\liminf\\limits_{k}-\\alpha\\frac{f(x_k)-f(\\bar{x})}{\\frac{1}{2}t^2_k}+ \\liminf\\limits_{k}-\\frac{\\langle \\lambda, g(x_k)-g(\\bar{x}) \\rangle }{\\frac{1}{2}t^2_k}\\\\\n\t&\\leq&\n\t\\liminf\\limits_{k}-\\frac{L^\\alpha (x_k,\\lambda)-L^\\alpha(\\bar{x},\\lambda)}{\\frac{1}{2}t_k^2}\\\\\n\t&=&\n\t\\liminf\\limits_{k}-\\frac{\\nabla_x L^\\alpha(\\bar{x},\\lambda)(t_ku_k)+\\frac 12 \\nabla^2_{xx} L^\\alpha(\\bar{x},\\lambda)(t_ku_k,t_ku_k)+\\oo(t_k^2)}{\\frac{1}{2}t_k^2}\\\\\n\t&=&\n\t\\liminf\\limits_{k}-\\nabla^2_{xx} L^\\alpha(\\bar{x},\\lambda)(u_k,u_k)=-\\nabla^2_{xx}L^\\alpha(\\bar{x},\\lambda)(u,u),\n \\end{eqnarray*}\nwhich contradicts \\eqref{alpha-sufficient}. This completes the proof.\n\\end{proof}","sha256_utf8":"7b68f7160c3ba306db705536ef95692d170fa8289ef8a0ec9d2d3026156d3448"}]},{"id":"S013","semantic_type":"proof_step","kind":"proof_step","title":"反设并构造二阶退化序列","source_units":[{"start":66874,"end":67200,"start_line":956,"end_line":961,"raw_tex":"\\begin{proof}\n By contradiction, if $\\bar{x}$ is not an essential local minimizer of second order, then there exists a sequence $x_k$ converging to $\\xb$ such that\n\\begin{eqnarray}\n&&  f(x_k)-f(\\bar{x})\\leq o(\\|x_k-\\bar{x}\\|^2), \\label{cqg}\\\\\n&&  {\\rm dist}(g(x_k),C)\\leq o(\\|x_k-\\bar{x}\\|^2).\\label{distance}\n \\end{eqnarray}\n","sha256_utf8":"7044a580d115ee41313b413192c7c980fbb696ccd207d59c7df7f03f09c05646"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"归一化方向与目标函数估计","source_units":[{"start":67200,"end":67532,"start_line":962,"end_line":967,"raw_tex":" Let $t_k:=\\|x_k-\\bar{x}\\|$ and $u_k:=(x_k-\\bar{x})/t_k$.\n We assume without loss of generality that $u_k$ is converging to $u$.\n From equation \\eqref{cqg} it readily follows that $\\nabla f(\\xb)u\\leq 0$ and\n  \\begin{equation}\\label{f-inequality}\n \\liminf\\limits_{k}-\\frac{f(x_k)-f(\\bar{x})}{\\frac{1}{2}t_k^2}\\geq 0.\n \\end{equation}\n","sha256_utf8":"1c074ea6c7eb5263c463575b23cf60875d92cfe5ca7a5d2cc6f97b56944f281e"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"修正约束并得到切方向","source_units":[{"start":67532,"end":67990,"start_line":968,"end_line":973,"raw_tex":" By (\\ref{distance}), there exists {$r_k\\in \\R^d$} such that $\\|r_k\\| \\to 0$ and\n $g(x_k)+t^2_k r_k\\in C$.\n By Taylor's expansion, since $x_k=\\bar x+t_ku_k$, we have\n \\[\nv_k:= \\frac{g(x_k)+ t^2_kr_k - g(\\xb)}{t_k} = \\frac{\\nabla g(\\bar{x})t_ku_k +o(t_k) +  t^2_kr_k}{t_k} = \\nabla g(\\bar{x})u_k +\\frac{o(t_k)}{t_k}{+t_kr_k} \\ \\to \\ \\nabla g(\\bar{x})u.\\]\nMoreover, $g(\\bar{x})+t_kv_k=g(x_k)+ t^2_kr_k\\in C$ and so $\\nabla g(\\bar{x})u\\in T_C(g(\\xb))$ follows.\n","sha256_utf8":"94d0d8a373a651ff0dee9202259ee7b0c40a7766b0942646f789513477b388cb"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"确认非零临界方向并选取乘子","source_units":[{"start":67990,"end":68285,"start_line":974,"end_line":975,"raw_tex":"Thus $u\\in \\mathcal{C}(\\xb)\\setminus\\{0\\}$ and the assumption of the theorem yields the existence of $\\alpha\\geq 0$ and $\\lambda\\in\\R^m$ satisfying \\eqref{Alpha} and \\eqref{alpha-sufficient}.\nUsing \\eqref{indicatorf}, \\eqref{f-inequality}, $\\|r_k\\| \\to 0$, and \\eqref{Alpha}, however, we obtain\n","sha256_utf8":"8bdf67f9b07f04198c1a2b34676c5176f9cd87a33350b4f9301b96983edbb232"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"次导数估计与拉格朗日泰勒展开","source_units":[{"start":68285,"end":69317,"start_line":976,"end_line":992,"raw_tex":" \\begin{eqnarray*} %\\nonumber\n\t{\\rm d}^2\\delta_{C}(g(\\bar{x});\\lambda)(\\nabla g(\\bar{x})u)\n\t&=&\n\t\\liminf\\limits_{{t\\downarrow 0, v'\\to \\nabla g(\\bar{x})u} \\atop { g(\\bar{x})+tv'\\in C }}\\frac{-\\langle \\lambda, v' \\rangle }{\\frac{1}{2}t}\n\t \\leq\n\t\\liminf\\limits_{k}\\frac{-t_k\\langle \\lambda, v_k \\rangle }{\\frac{1}{2}t_k^2} \\\\\n\t&=&\n\t\\liminf\\limits_{k}\\frac{-\\langle \\lambda, g(x_k)+ t^2_kr_k-g(\\bar{x})  \\rangle }{\\frac{1}{2}t_k^2} \\\\%\\label{g-inequality}\n\t&\\leq&\n\t\\liminf\\limits_{k}-\\alpha\\frac{f(x_k)-f(\\bar{x})}{\\frac{1}{2}t^2_k}+ \\liminf\\limits_{k}-\\frac{\\langle \\lambda, g(x_k)-g(\\bar{x}) \\rangle }{\\frac{1}{2}t^2_k}\\\\\n\t&\\leq&\n\t\\liminf\\limits_{k}-\\frac{L^\\alpha (x_k,\\lambda)-L^\\alpha(\\bar{x},\\lambda)}{\\frac{1}{2}t_k^2}\\\\\n\t&=&\n\t\\liminf\\limits_{k}-\\frac{\\nabla_x L^\\alpha(\\bar{x},\\lambda)(t_ku_k)+\\frac 12 \\nabla^2_{xx} L^\\alpha(\\bar{x},\\lambda)(t_ku_k,t_ku_k)+\\oo(t_k^2)}{\\frac{1}{2}t_k^2}\\\\\n\t&=&\n\t\\liminf\\limits_{k}-\\nabla^2_{xx} L^\\alpha(\\bar{x},\\lambda)(u_k,u_k)=-\\nabla^2_{xx}L^\\alpha(\\bar{x},\\lambda)(u,u),\n \\end{eqnarray*}\n","sha256_utf8":"067b9a373b49ea87712bdff882bf25ef379adf04f411a537ef928d99f6037f1f"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"严格正性导致矛盾","source_units":[{"start":69317,"end":69398,"start_line":993,"end_line":994,"raw_tex":"which contradicts \\eqref{alpha-sufficient}. This completes the proof.\n\\end{proof}","sha256_utf8":"72532838c90c78d0cf35e0ec66ddc9c3880dcff1851b07a8c31d2ab911b46e66"}]},{"id":"S019","semantic_type":"external_fact","kind":"external_fact","title":"单位球面的紧性","source_units":[],"declared_external":true,"declaration_text":["有限维欧氏空间中的单位球面是紧的，因此单位向量序列存在收敛子序列，且极限仍为单位向量。"]},{"id":"S020","semantic_type":"external_fact","kind":"external_fact","title":"闭集最近点存在性","source_units":[],"declared_external":true,"declaration_text":["非空闭集上的欧氏距离可以由最近点实现。"]},{"id":"S021","semantic_type":"external_fact","kind":"external_fact","title":"光滑映射的泰勒展开","source_units":[],"declared_external":true,"declaration_text":["二阶连续可微映射满足一阶与二阶泰勒展开；其二阶导数连续。"]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S010","edges":[{"source_id":"S001","target_id":"S007","kind":"supports"},{"source_id":"S001","target_id":"S008","kind":"supports"},{"source_id":"S001","target_id":"S009","kind":"supports"},{"source_id":"S001","target_id":"S014","kind":"supports"},{"source_id":"S001","target_id":"S015","kind":"supports"},{"source_id":"S001","target_id":"S017","kind":"supports"},{"source_id":"S002","target_id":"S009","kind":"supports"},{"source_id":"S002","target_id":"S013","kind":"supports"},{"source_id":"S002","target_id":"S014","kind":"supports"},{"source_id":"S002","target_id":"S015","kind":"supports"},{"source_id":"S002","target_id":"S017","kind":"supports"},{"source_id":"S003","target_id":"S006","kind":"supports"},{"source_id":"S004","target_id":"S007","kind":"supports"},{"source_id":"S004","target_id":"S015","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S006","target_id":"S017","kind":"supports"},{"source_id":"S007","target_id":"S016","kind":"supports"},{"source_id":"S008","target_id":"S017","kind":"supports"},{"source_id":"S009","target_id":"S013","kind":"supports"},{"source_id":"S011","target_id":"S016","kind":"supports"},{"source_id":"S011","target_id":"S017","kind":"supports"},{"source_id":"S011","target_id":"S018","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S016","kind":"supports"},{"source_id":"S014","target_id":"S017","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S014","target_id":"S012","kind":"supports"},{"source_id":"S015","target_id":"S012","kind":"supports"},{"source_id":"S016","target_id":"S012","kind":"supports"},{"source_id":"S017","target_id":"S012","kind":"supports"},{"source_id":"S018","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S010","kind":"supports"},{"source_id":"S019","target_id":"S014","kind":"supports"},{"source_id":"S020","target_id":"S015","kind":"supports"},{"source_id":"S021","target_id":"S014","kind":"supports"},{"source_id":"S021","target_id":"S015","kind":"supports"},{"source_id":"S021","target_id":"S017","kind":"supports"}]}}
{"record_id":"2203.10605v2","paper":{"title":"Convergence rates of the stochastic alternating algorithm for bi-objective optimization","authors":["Suyun Liu","Luis Nunes Vicente"],"arxiv_id":"2203.10605","version":"v2","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-023-02253-w","abstract_url":"https://arxiv.org/abs/2203.10605v2","pdf_url":"https://arxiv.org/pdf/2203.10605v2"},"context":{"raw_tex":"\\begin{equation}\n\\label{biobjective_smooth}\n    \\min~F(x) = (f^a(x), f^b(x))^\\top, \\quad x \\in \\mathcal{X},\n\\end{equation}\n\nwhere $\\mathcal{X}$ is a nonempty bounded closed convex region. \nWe say that the multi-objective problem is smooth if all objective functions~$f^i$ are continuously differentiable. In the convergence theory of our algorithms, we will assume that both functions are convex. Furthermore, both objectives involve randomness in its parameters, in which case we expect that a true gradient or subgradient is not available or too expensive to compute. Instead, one can access a unbiased estimator $g^i(x, \\xi), i \\in \\{a, b\\}$, of a true gradient or subgradient of $f^i$, where $\\xi$ denotes some random variable.\n\n\nThe key idea of the alternating bi-objective optimization for solving~\\eqref{biobjective_smooth} consists of iteratively taking $n_a$ gradient (or subgradient) descent steps for the first objective and then $n_b$ gradient (or subgradient) descent steps for the second objective. For simplicity, we denote such $n_a + n_b$ steps as one single iteration of the algorithm. The stochastic alternating bi-objective gradient or subgradient algorithm (SA2GD) is formally described in Algorithm~\\ref{alg1_SA2GD}. At every iteration $t$, starting at $x_t = y^a_{0, t}$, the algorithm computes two sequences of intermediate iterates, $\\{y^a_{r, t}\\}_{r=1}^{n_a}$ and $\\{y^b_{r, t}\\}_{r=1}^{n_b}$. Let $P_{\\mathcal{X}}$ denote the orthogonal projection operator that projects \nthe new iterate back to the feasible region. Such an operator is well-defined because we assume a compact and convex feasible region.\n\n\n\\begin{algorithm}[ht]\n\\caption{Stochastic alternating bi-objective gradient (or subgradient) algorithm (SA2GD)} \n\\label{alg1_SA2GD} \n\\begin{algorithmic}[1]\n\\item \\textbf{Input:} Initial point $x_0 = y^a_{0, 0}$ and a step size sequence $\\{\\alpha_t\\}$.\n\\item \\textbf{Output:} A likely non-dominated or Pareto solution $x_{T} = y^a_{0, T}$.\n\\item {\\bf for} $t = 0, 1, \\ldots, T-1$ {\\bf do}\n\\item \\quad {\\bf for} $r = 0, \\ldots, n_a-1$ {\\bf do}\n\\item \\quad \\quad Generate a stochastic gradient $g^a(y^a_{r, t}, \\xi^r_{t})$.\n\\item \\quad \\quad Update $y^a_{r+1, t} = y^a_{r, t} - \\alpha_t g^a(y^a_{r, t}, \\xi^r_{t})$. \n\\item \\quad Set $y^b_{0, t} = y^a_{n_a, t}$.\n\\item \\quad {\\bf for} $r = 0, \\ldots, n_b - 1$ {\\bf do}\n\\item \\quad \\quad Generate a stochastic gradient $g^b(y^b_{r, t}, \\xi^{n_a + r}_{t})$.\n\\item \\quad \\quad Update $y^b_{r+1, t} = y^b_{r, t} - \\alpha_t g^b(y^b_{r, t}, \\xi^{n_a + r}_{t})$. \n\\item \\quad Set $x_{t+1} = y^a_{0, t+1} = P_{\\mathcal{X}}(y^b_{n_b, t})$.\n\\par\\vspace*{0.1cm}\n\\end{algorithmic}\n\\end{algorithm}\n\n\\begin{assumption}\n\\label{ass:limit_pts}\n\\textbf{(Bounded feasible region)}\nThe feasible region $\\mathcal{X}$ is compact, and in particular there exists a positive constant $\\Theta$ such that\n\\begin{equation*}\n\\displaystyle \\max_{x, \\bar{x} \\in \\mathcal{X}} \\| x - \\bar{x}\\| \\;\\leq\\; \\textstyle \\Theta \\;<\\; \\infty.\n\\end{equation*}\n\\end{assumption}\n\n\\begin{assumption}\n\\label{ass:Lipschitz}\n\\textbf{(Lipschitz continuous gradients)} The individual true gradients are Lipschitz continuous with Lipschitz constants $L_i > 0, i \\in \\{a, b\\}$, i.e., \n\\begin{equation*}\n    % \\label{eq:Lipschitz}\n    \\begin{split}\n        \\|\\nabla f^i(x) - \\nabla f^i(\\bar{x}) \\| \\;\\leq\\;& L_i \\| x - \\bar{x} \\|, \\quad \\forall (x, \\bar{x}) \\in \\mathcal{X} \\times \\mathcal{X}.\n    \\end{split}\n\\end{equation*}\n\\end{assumption}\n\nNote that any continuous function  attains a maximum on a compact set, and thus one can assume the existence of positive constants~$M_\\nabla^i$, $i \\in \\{a,b\\}$, such that\n\\begin{equation} \\label{Mnabla}\n\\| \\nabla f^i(x) \\|^2 \\; \\leq \\; M_\\nabla^i,  \\quad \\forall x \\in \\mathcal{X}.  \n\\end{equation}\n\n\\begin{assumption}\n\\label{ass:strongconv}\n\\textbf{(Strong convexity)} For all $i = \\{a, b\\}$, there exists a scalar $c_i > 0$ such that\n\n\\begin{equation*}\nf^i(\\bar{x}) \\;\\geq\\; f^i(x) + \\nabla f^i(x)^\\top(\\bar{x} - x) + \\frac{c_i}{2} \\| \\bar{x} - x \\|^2, \\quad \\forall (x,\\bar{x}) \\in \\mathcal{X} \\times \\mathcal{X}.\n\\end{equation*}\n\\end{assumption}\n\nNote that based on the above assumption, the weighted-sum function $S(x, \\lambda) = \\lambda f^a(x) + (1-\\lambda)f^b(x), \\lambda \\in [0, 1]$, is also strongly convex with constant $c = \\min(c_a, c_b)$. Given the individual stochastic gradients $g^i(x_t, \\xi_t), \\forall i \\in \\{a, b\\}$, generated with random variable~$\\xi_t$, we use $\\mathbb{E}_{\\xi_t}[\\cdot]$ to denote the conditional expectation taken with respect to $\\xi_t$. We also impose the following two classical assumptions of stochastic gradients. \n\n\n\\begin{assumption}\n\\label{ass:gradient}\nFor both objective functions $i \\in \\{a, b\\}$, and all iterates $t \\in \\mathbb{N}$, the individual stochastic gradients~$g^i(x_t, \\xi_t)$ satisfy the following:\n\\begin{enumerate}\n    \\item[(a)]\\textbf{(Unbiased gradient estimation)} \n    $\\mathbb{E}_{\\xi_t}[g^i(x_t, \\xi_t)] \\;=\\; \\nabla f^i(x_t), ~\\forall i \\in \\{a, b\\}$. \n    \\item[(b)]\\textbf{(Bound on the second moment)}\nThere exist positive scalars $G_i > 0$ and $\\bar{G}_i > 0$ such that\n\\begin{equation*}\n\\begin{split}\n\\mathbb{E}_{\\xi_t}[ g^i(x_t, \\xi_t)\\|^2] \\;\\leq\\; G_i + \\bar{G}_i\\| \\nabla f^i(x_t)\\|^2, \\quad \\forall i \\in \\{a, b\\}.\n\\end{split}\n\\end{equation*}\n\\end{enumerate}\n\\end{assumption}","units":[{"start":6856,"end":6978,"start_line":162,"end_line":165,"raw_tex":"\\begin{equation}\n\\label{biobjective_smooth}\n    \\min~F(x) = (f^a(x), f^b(x))^\\top, \\quad x \\in \\mathcal{X},\n\\end{equation}","sha256_utf8":"9b019f8376edc3103224de298dffb9c5b882767f1d801a129bb607e472f0ec99"},{"start":6979,"end":7587,"start_line":166,"end_line":167,"raw_tex":"where $\\mathcal{X}$ is a nonempty bounded closed convex region. \nWe say that the multi-objective problem is smooth if all objective functions~$f^i$ are continuously differentiable. In the convergence theory of our algorithms, we will assume that both functions are convex. Furthermore, both objectives involve randomness in its parameters, in which case we expect that a true gradient or subgradient is not available or too expensive to compute. Instead, one can access a unbiased estimator $g^i(x, \\xi), i \\in \\{a, b\\}$, of a true gradient or subgradient of $f^i$, where $\\xi$ denotes some random variable.\n","sha256_utf8":"ea13fe48a78fa0851dd0132330834cdab40b36fc66163e86cd8995dbb29da94c"},{"start":16493,"end":17393,"start_line":203,"end_line":204,"raw_tex":"The key idea of the alternating bi-objective optimization for solving~\\eqref{biobjective_smooth} consists of iteratively taking $n_a$ gradient (or subgradient) descent steps for the first objective and then $n_b$ gradient (or subgradient) descent steps for the second objective. For simplicity, we denote such $n_a + n_b$ steps as one single iteration of the algorithm. The stochastic alternating bi-objective gradient or subgradient algorithm (SA2GD) is formally described in Algorithm~\\ref{alg1_SA2GD}. At every iteration $t$, starting at $x_t = y^a_{0, t}$, the algorithm computes two sequences of intermediate iterates, $\\{y^a_{r, t}\\}_{r=1}^{n_a}$ and $\\{y^b_{r, t}\\}_{r=1}^{n_b}$. Let $P_{\\mathcal{X}}$ denote the orthogonal projection operator that projects \nthe new iterate back to the feasible region. Such an operator is well-defined because we assume a compact and convex feasible region.\n","sha256_utf8":"6d333ce8cc0d5fac1fdd7b73e5a8ecb1e5fdef6d14e74c224fe6df3eba10cd83"},{"start":17480,"end":18510,"start_line":208,"end_line":225,"raw_tex":"\\begin{algorithm}[ht]\n\\caption{Stochastic alternating bi-objective gradient (or subgradient) algorithm (SA2GD)} \n\\label{alg1_SA2GD} \n\\begin{algorithmic}[1]\n\\item \\textbf{Input:} Initial point $x_0 = y^a_{0, 0}$ and a step size sequence $\\{\\alpha_t\\}$.\n\\item \\textbf{Output:} A likely non-dominated or Pareto solution $x_{T} = y^a_{0, T}$.\n\\item {\\bf for} $t = 0, 1, \\ldots, T-1$ {\\bf do}\n\\item \\quad {\\bf for} $r = 0, \\ldots, n_a-1$ {\\bf do}\n\\item \\quad \\quad Generate a stochastic gradient $g^a(y^a_{r, t}, \\xi^r_{t})$.\n\\item \\quad \\quad Update $y^a_{r+1, t} = y^a_{r, t} - \\alpha_t g^a(y^a_{r, t}, \\xi^r_{t})$. \n\\item \\quad Set $y^b_{0, t} = y^a_{n_a, t}$.\n\\item \\quad {\\bf for} $r = 0, \\ldots, n_b - 1$ {\\bf do}\n\\item \\quad \\quad Generate a stochastic gradient $g^b(y^b_{r, t}, \\xi^{n_a + r}_{t})$.\n\\item \\quad \\quad Update $y^b_{r+1, t} = y^b_{r, t} - \\alpha_t g^b(y^b_{r, t}, \\xi^{n_a + r}_{t})$. \n\\item \\quad Set $x_{t+1} = y^a_{0, t+1} = P_{\\mathcal{X}}(y^b_{n_b, t})$.\n\\par\\vspace*{0.1cm}\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"4d4373104be200c57f5dc63abb3a08c7d6e791a7027fbb5ef316fd49e000aec3"},{"start":20948,"end":21296,"start_line":242,"end_line":249,"raw_tex":"\\begin{assumption}\n\\label{ass:limit_pts}\n\\textbf{(Bounded feasible region)}\nThe feasible region $\\mathcal{X}$ is compact, and in particular there exists a positive constant $\\Theta$ such that\n\\begin{equation*}\n\\displaystyle \\max_{x, \\bar{x} \\in \\mathcal{X}} \\| x - \\bar{x}\\| \\;\\leq\\; \\textstyle \\Theta \\;<\\; \\infty.\n\\end{equation*}\n\\end{assumption}","sha256_utf8":"e17e6d5a5d2137677e3fbaf0af2a241a6bc709d1eddfa0f82c4457dd5a5acc3f"},{"start":21396,"end":21849,"start_line":253,"end_line":262,"raw_tex":"\\begin{assumption}\n\\label{ass:Lipschitz}\n\\textbf{(Lipschitz continuous gradients)} The individual true gradients are Lipschitz continuous with Lipschitz constants $L_i > 0, i \\in \\{a, b\\}$, i.e., \n\\begin{equation*}\n    % \\label{eq:Lipschitz}\n    \\begin{split}\n        \\|\\nabla f^i(x) - \\nabla f^i(\\bar{x}) \\| \\;\\leq\\;& L_i \\| x - \\bar{x} \\|, \\quad \\forall (x, \\bar{x}) \\in \\mathcal{X} \\times \\mathcal{X}.\n    \\end{split}\n\\end{equation*}\n\\end{assumption}","sha256_utf8":"69a772d0b1df100b37fd1c942163435a1de4cd6b12d9a16df4488f9a4f2f0a07"},{"start":21851,"end":22150,"start_line":264,"end_line":267,"raw_tex":"Note that any continuous function  attains a maximum on a compact set, and thus one can assume the existence of positive constants~$M_\\nabla^i$, $i \\in \\{a,b\\}$, such that\n\\begin{equation} \\label{Mnabla}\n\\| \\nabla f^i(x) \\|^2 \\; \\leq \\; M_\\nabla^i,  \\quad \\forall x \\in \\mathcal{X}.  \n\\end{equation}","sha256_utf8":"123269cd4babc3f1582e9054c922c4842a902ebd933014e8b53e059888c16388"},{"start":22255,"end":22604,"start_line":270,"end_line":277,"raw_tex":"\\begin{assumption}\n\\label{ass:strongconv}\n\\textbf{(Strong convexity)} For all $i = \\{a, b\\}$, there exists a scalar $c_i > 0$ such that\n\n\\begin{equation*}\nf^i(\\bar{x}) \\;\\geq\\; f^i(x) + \\nabla f^i(x)^\\top(\\bar{x} - x) + \\frac{c_i}{2} \\| \\bar{x} - x \\|^2, \\quad \\forall (x,\\bar{x}) \\in \\mathcal{X} \\times \\mathcal{X}.\n\\end{equation*}\n\\end{assumption}","sha256_utf8":"8afa2871d92d32c7a30b99b7bd83c13412e8f4f723cf2aecf84ea3a3d3589f9b"},{"start":22606,"end":23117,"start_line":279,"end_line":279,"raw_tex":"Note that based on the above assumption, the weighted-sum function $S(x, \\lambda) = \\lambda f^a(x) + (1-\\lambda)f^b(x), \\lambda \\in [0, 1]$, is also strongly convex with constant $c = \\min(c_a, c_b)$. Given the individual stochastic gradients $g^i(x_t, \\xi_t), \\forall i \\in \\{a, b\\}$, generated with random variable~$\\xi_t$, we use $\\mathbb{E}_{\\xi_t}[\\cdot]$ to denote the conditional expectation taken with respect to $\\xi_t$. We also impose the following two classical assumptions of stochastic gradients. \n","sha256_utf8":"32c1a4e10a23220c07c7a7fe8ddbceb8e64ce7a34c4ae468b785a69f8d9575c7"},{"start":23118,"end":23815,"start_line":281,"end_line":295,"raw_tex":"\\begin{assumption}\n\\label{ass:gradient}\nFor both objective functions $i \\in \\{a, b\\}$, and all iterates $t \\in \\mathbb{N}$, the individual stochastic gradients~$g^i(x_t, \\xi_t)$ satisfy the following:\n\\begin{enumerate}\n    \\item[(a)]\\textbf{(Unbiased gradient estimation)} \n    $\\mathbb{E}_{\\xi_t}[g^i(x_t, \\xi_t)] \\;=\\; \\nabla f^i(x_t), ~\\forall i \\in \\{a, b\\}$. \n    \\item[(b)]\\textbf{(Bound on the second moment)}\nThere exist positive scalars $G_i > 0$ and $\\bar{G}_i > 0$ such that\n\\begin{equation*}\n\\begin{split}\n\\mathbb{E}_{\\xi_t}[ g^i(x_t, \\xi_t)\\|^2] \\;\\leq\\; G_i + \\bar{G}_i\\| \\nabla f^i(x_t)\\|^2, \\quad \\forall i \\in \\{a, b\\}.\n\\end{split}\n\\end{equation*}\n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"e9b4494cdc1c8b059daaeec3613f69b5a5c7f67d70401300922a8d22a172980b"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem} (\\textbf{Sublinear convergence rate of SA2GD in the smooth and strongly convex case})\n\\label{th:smooth_strongcov}\nLet Assumptions~\\ref{ass:limit_pts}-\\ref{ass:gradient} hold and $x_*$ be the unique minimizer of the weighted function $S(\\cdot, \\lambda_*)$ in~$\\mathcal{X}$, where $\\lambda_* = \\lambda(n_a, n_b) = n_a/(n_a + n_b)$. Choosing a diminishing step size sequence\n$\\alpha_t = \\frac{2}{c(t+1)(n_a + n_b)}$, the sequence of iterates generated by the SA2GD algorithm satisfies\n\\begin{equation*}\n    \\min_{t = 1, \\ldots,T} \\mathbb{E}[S(x_t, \\lambda_*)] - S(x_*, \\lambda_*) \\;\\leq\\; \\frac{4}{c(T + 1)} \\left(\\hat{G}^2 +  L\\Theta\\hat{G}\\right).   \n\\end{equation*}\n where $\\hat{G} = \\sqrt{G + \\bar{G}M_{\\nabla}}$, $L = \\max(L_a, L_b)$, $G = \\max(G_a, G_b)$, $\\bar{G} = \\max(\\bar{G}_a, \\bar{G}_b)$, and $M_\\nabla = \\max(M_\\nabla^a, M_\\nabla^b)$.\n\\end{theorem}","units":[{"start":24492,"end":25367,"start_line":301,"end_line":309,"raw_tex":"\\begin{theorem} (\\textbf{Sublinear convergence rate of SA2GD in the smooth and strongly convex case})\n\\label{th:smooth_strongcov}\nLet Assumptions~\\ref{ass:limit_pts}-\\ref{ass:gradient} hold and $x_*$ be the unique minimizer of the weighted function $S(\\cdot, \\lambda_*)$ in~$\\mathcal{X}$, where $\\lambda_* = \\lambda(n_a, n_b) = n_a/(n_a + n_b)$. Choosing a diminishing step size sequence\n$\\alpha_t = \\frac{2}{c(t+1)(n_a + n_b)}$, the sequence of iterates generated by the SA2GD algorithm satisfies\n\\begin{equation*}\n    \\min_{t = 1, \\ldots,T} \\mathbb{E}[S(x_t, \\lambda_*)] - S(x_*, \\lambda_*) \\;\\leq\\; \\frac{4}{c(T + 1)} \\left(\\hat{G}^2 +  L\\Theta\\hat{G}\\right).   \n\\end{equation*}\n where $\\hat{G} = \\sqrt{G + \\bar{G}M_{\\nabla}}$, $L = \\max(L_a, L_b)$, $G = \\max(G_a, G_b)$, $\\bar{G} = \\max(\\bar{G}_a, \\bar{G}_b)$, and $M_\\nabla = \\max(M_\\nabla^a, M_\\nabla^b)$.\n\\end{theorem}","sha256_utf8":"bdf89f1fe2b568ef28d1d8ecc3625a30ddb28f8dc8a0d569a53b01fdb99af340"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{proof}\nThe proof is divided in three parts for better organization and understanding. In the first part, one obtains an upper bound on the norm of the iterates, $\\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2]$. Strong convexity of the weighted-sum function is applied in the second part. The third part concludes the proof using standard arguments. For simplicity, we let $z_t = y^b_{0, t} = y^a_{n_a, t}$ be the intermediate point at each iteration.  \\vspace{1ex}\n\n\\noindent {\\bf Part I: Bound on the iterates error using the Intermediate Value Theorem.} \nAt any iteration $t$, the sequence of stochastic gradients is computed from drawing the sequence of random variables $\\xi_t = \\{\\xi^0_t, \\ldots, \\xi^{n_a + n_b - 1}_t \\}$. We have\n\\begin{align*}\n    x_{t+1} - x_* & \\;=\\; P_{\\mathcal{X}}\\left(z_t - \\alpha_t \\sum_{r = 0}^{n_b - 1} g^b(y^b_{r, t}, \\xi^r_t)\\right) - x_*, \\\\\n    &\\;=\\;P_{\\mathcal{X}}\\left(\n    x_t - \\alpha_t \\sum_{r = 0}^{n_a - 1}  g^a(y^a_{r, t}, \\xi^r_t) - \\alpha_t \\sum_{r = 0}^{n_b - 1}  g^b(y^b_{r, t}, \\xi^{n_a + r}_t) \\right) - x_*.\n\\end{align*}\n\nSince the sequence $\\xi_t$ is drawn independently, using Assumption~\\ref{ass:gradient} (a) one has\n\\[\\mathbb{E}_{\\xi_t}[g^a(y_{r, t}^a, \\xi_{t}^r)] \\;=\\; \\mathbb{E}_{\\xi_t^0, \\ldots, \\xi_{t}^{r-1}}[\\mathbb{E}_{\\xi_t^r}[g^a(y_{r, t}^a, \\xi_{t}^r)]] \\;=\\; \\mathbb{E}_{\\xi_t^0, \\ldots, \\xi_{t}^{r-1}}[\\nabla f^a(y_{r, t}^a)] \\;=\\; \\mathbb{E}_{\\xi_t}[\\nabla f^a(y_{r, t}^a)],\\]\nwhere the last equality holds due to the independence between $y_{r, t}^a$ and $\\{\\xi_t^{r}, \\ldots, \\xi_t^{n_a + n_b - 1}\\}$. Similarly, we have $\\mathbb{E}_{\\xi_t}[g^b(y_{r, t}^b, \\xi_{t}^{n_a + r})] = \\mathbb{E}_{\\xi_t}[\\nabla f^b(y_{r, t}^b)]$.\nThen, taking square norms and expectations over the random variables $\\xi_t$ on both sides yields\n\\begin{equation}\n\\label{theorem2_eq1}\n    \\begin{split}\n    \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2]  \n     \\;\\leq\\; & \\|x_t - x_*\\|^2 + \\alpha_t^2\\mathbb{E}_{\\xi_t}[\\|\\sum_{r = 0}^{n_a - 1} g^a(y^a_{r, t}, \\xi^r_t) + \\sum_{r = 0}^{n_b - 1} g^b(y^b_{r, t}, \\xi^{n_a + r}_t)\\|^2] \\\\\n     & - \\mathbb{E}_{\\xi_t}[2\\alpha_t(x_t - x_*)^\\top (\\sum_{r = 0}^{n_a - 1} \\nabla f^a(y^a_{r, t}) + \\sum_{r = 0}^{n_b - 1} \\nabla f^b(y^b_{r, t}))],\n    \\end{split}\n\\end{equation}\nwhich holds by the non-expansiveness property of the orthogonal projection operator.\n\nWe now claim, by applying a version of the Intermediate Value Theorem given in Proposition~\\ref{prop_ivt}, that the last term of the right-hand side in~\\eqref{theorem2_eq1} can be written as $- 2\\alpha_t(x_t - x_*)^\\top \\mathbb{E}_{\\xi_t}[n_a \\nabla f^a(w_t^a) + n_b \\nabla f^b(w_t^b)]$ for some $w_t^a$ and $w_t^b$. In fact, we apply Proposition~\\ref{prop_ivt} to the real continuous function $\\phi^i(y) = -2\\alpha_t(x_t - x_*)^\\top \\nabla f^i(y)$, from which we then know that $w_t^i$ is a convex linear combination of a sequence of points $\\{y_{r, t}^i\\}_{r = 0}^{n_i - 1}$ for both $i \\in \\{a, b\\}$. \n\nUsing a combination of~(\\ref{Mnabla}) and Assumption~\\ref{ass:gradient}~(b), the bound for the second moment of the stochastic gradients is given by\n\\begin{equation} \\label{SGbound}\n\\mathbb{E}_{\\xi_t}[\\|g^i(y^i_{r, t}, \\xi_t^r)\\|^2] \\; \\leq \\; G + \\bar{G}M_{\\nabla}, \\quad\n\\mathbb{E}_{\\xi_t}[\\|g^i(y^i_{r, t}, \\xi_t^r)\\|] \\; \\leq \\; \\sqrt{G + \\bar{G}M_{\\nabla}},\n\\end{equation} \nwhere the second bound results from applying Jensen's inequality to the first one.\nHence, the second term in the right-hand side of~\\eqref{theorem2_eq1} can be bounded by\n\\begin{equation*}\n\\begin{split}\n   & \\mathbb{E}_{\\xi_t}[\\|\\sum_{r = 0}^{n_a - 1} g^a(y^a_{r, t}, \\xi^r_t) + \\sum_{r = 0}^{n_b - 1}  g^b(y^b_{r, t}, \\xi^{n_a + r}_t)\\|^2] \\\\\n   &  \\;\\leq\\; 2\\mathbb{E}_{\\xi_t}[\\|\\sum_{r = 0}^{n_a - 1}  g^a(y^a_{r, t}, \\xi^r_t)\\|^2] + 2\\mathbb{E}_{\\xi_t}[\\|\\sum_{r = 0}^{n_b - 1}  g^b(y^b_{r, t}, \\xi^{n_a+r}_t)\\|^2] \\\\\n& \\;\\leq\\;  2n_a\\sum_{r = 0}^{n_a - 1} \\mathbb{E}_{\\xi_t}[\\| g^a(y^a_{r, t}, \\xi^r_t)\\|^2] + 2n_b\\sum_{r = 0}^{n_b - 1} \\mathbb{E}_{\\xi_t}[\\| g^b(y^b_{r, t}, \\xi^{n_a + r}_t)\\|^2] \\\\\n& \\;\\leq\\;  2(n_a + n_b)^2(G + \\bar{G}M_{\\nabla}).\n\\end{split}\n\\end{equation*}\nWe thus arrive at\n\\begin{equation}\n    \\label{theorem2_eq2}\n    \\begin{split}\n    \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2]\n    \\;\\leq\\; &  \\|x_t - x_*\\|^2 + 2\\alpha_t^2(n_a + n_b)^2(G + \\bar{G}M_{\\nabla}) \\\\\n     & - 2\\alpha_t(x_t - x_*)^\\top \\mathbb{E}_{\\xi_t}[n_a \\nabla f^a(w_t^a) + n_b \\nabla f^b(w_t^b)].\n    \\end{split}\n\\end{equation}\n\nBy adding and subtracting $2\\alpha_t(x_t - x_*)^\\top(n_a\\nabla f^a(x_t) + n_b \\nabla f^b(x_t))$ in the right-hand side of~\\eqref{theorem2_eq2}, we further rewrite it as\n\\begin{equation}\n\\label{theorem2_eq22}\n    \\begin{split}\n    \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2]\n    \\;\\leq\\; &  \\|x_t - x_*\\|^2 + 2\\alpha_t^2(n_a + n_b)^2(G + \\bar{G}M_{\\nabla}) \\\\\n     & - 2\\alpha_t(x_t - x_*)^\\top (n_a \\nabla f^a(x_t) + n_b \\nabla f^b(x_t)) \\\\\n     & + 2\\alpha_t\\|x_t - x_*\\|\\mathbb{E}_{\\xi_t}[\\|n_a \\nabla f^a(w_t^a) - n_a \\nabla f^a(x_t)\\|] \\\\\n     & + 2\\alpha_t\\|x_t - x_*\\|\\mathbb{E}_{\\xi_t}[\\|n_b \\nabla f^b(w^b_t) - n_b \\nabla f^b(x_t)\\|]. \n    \\end{split}\n\\end{equation}\nNote that the last two terms are derived by the Cauchy–Schwarz and Jensen's inequalities.\n\n\\vspace{2ex}\n\\noindent {\\bf Part II: Using strong convexity.} \nSelecting $\\lambda_* = \\lambda(n_a, n_b) = n_a/(n_a + n_b)$, by the strong convexity of the weighted-sum function, one has\n\\begin{equation*}\n \\nabla_x S(x_t, \\lambda_*)^\\top(x_t - x_*)  \\;\\geq\\; S(x_t, \\lambda_*) - S(x_*, \\lambda_*) + \\frac{c}{2}\\| x_t - x_*\\|^2,\n\\end{equation*}\nwhich is equivalent to\n\\begin{equation}\n\\label{theorm2_eq3}\n (x_t - x_*)^\\top(n_a \\nabla f^a(x_t) + n_b \\nabla f^b(x_t))  \\;\\geq\\; (n_a + n_b)(S(x_t, \\lambda_*) - S(x_*, \\lambda_*) + \\frac{c}{2}\\| x_t - x_*\\|^2).\n\\end{equation}\n\nFrom Assumption~\\ref{ass:Lipschitz}, we obtain a bound for the last two terms of~\\eqref{theorem2_eq22} in the form \n\\begin{equation}\n    \\label{theorem2_eq4}\n    2\\alpha_t\\|x_t - x_*\\|\\mathbb{E}_{\\xi_t}[\\|n_i \\nabla f^i(w_t^i) - n_i \\nabla f^i(x_t)\\|] \\;\\leq\\;  2\\alpha_tLn_i\\|x_t - x_*\\|\\mathbb{E}_{\\xi_t}[\\|x_t - w_t^i\\|], \\forall i \\in \\{a, b\\}.\n\\end{equation}\nAccording to Proposition~\\ref{prop_ivt}, $w_t^i$ is a convex linear combination of a sequence of points $\\{y_{r, t}^i\\}_{r = 0}^{n_i -1}$ for $i \\in \\{a, b\\}$. One can write $w_t^i = \\sum_{r=0}^{n_i - 1} \\beta_r y_{r, t}^i$ with $\\beta_r \\geq 0$, $r = 0, \\ldots, n_i - 1$ , and $\\sum_{r=0}^{n_i - 1} \\beta_r = 1$. An explicit upper bound of $\\|x_t - w_t^i\\|$ can then be derived as follows\n\\begin{equation}\n\\label{part2_eq2}\n    \\begin{split} \n        \\|x_t - \\sum_{r=0}^{n_i - 1} \\beta_r y_{r, t}^i\\| & \\;=\\; \\|\\sum_{r=0}^{n_i - 1} \\beta_r (x_t - y_{r, t}^i)\\| \\;\\leq\\; \\sum_{r=0}^{n_i - 1} \\beta_r \\|x_t - y_{r, t}^i\\|.\n    \\end{split}\n\\end{equation}\nUsing $y_{r, t}^i = y_{0, t}^i - \\sum_{j = 0}^{r-1}\\alpha_t g^i(y_{j, t}^i, \\xi_{t}^j)$ and applying the triangle inequality, we have\n\\begin{align}\n    \\|x_t - y_{r, t}^a\\| \\;\\leq\\; \\alpha_t \\sum_{j = 0}^{r-1} \\|g^a(y^a_{j, t}, \\xi_{t}^j)\\|,\\label{part2_eq3}\n\\end{align}  \nand\n\\begin{equation}\n\\label{part2_eq4}\n\\begin{split}\n    \\|x_t - y_{r, t}^b\\| & \\;=\\; \\|x_t - y_{n_a, t}^a + \\sum_{j = 0}^{r-1}\\alpha_t g^b(y_{j, t}^b, \\xi_{t}^{n_a + j})\\| \\\\\n    & \\;\\leq\\; \\alpha_t \\sum_{j = 0}^{n_a-1} \\|g^a(y^a_{j, t}, \\xi_{t}^j)\\| +  \\alpha_t \\sum_{j = 0}^{r-1} \\|g^b(y^b_{j, t}, \\xi_{t}^{n_a + j})\\|.\n\\end{split}\n\\end{equation}\nPlugging~\\eqref{part2_eq3} into~\\eqref{part2_eq2} with $i = a$ results in (note that $y^a_{0,t}=x_t$)\n\\begin{equation}\\label{part2_eq5}\n\\begin{split}\n       \\|x_t - w_t^a\\| & \\;\\leq\\; \\alpha_t \\sum_{r=1}^{n_a - 1} \\beta_r \\sum_{j = 0}^{r-1} \\|g^a(y^a_{j, t}, \\xi_{t}^j)\\| \\\\ & \\;  = \\;  \\alpha_t \n       \\sum_{j=0}^{n_a - 1} \\|g^a(y^a_{j, t}, \\xi_{t}^j)\\|\n      \\sum_{r=j+1}^{n_a-1} \\beta_r\n       \\;\\leq \\; \\alpha_t  \\sum_{j=0}^{n_a - 1} \\|g^a(y^a_{j,t}, \\xi_{t}^{j})\\|.\n\\end{split}\n\\end{equation}\n\n\nApplying expectations on both sides of~\\eqref{part2_eq5}, followed by the bound~(\\ref{SGbound}) on the second moment of the stochastic gradients,\n\\begin{equation} \\label{part2_eq6}\n\\mathbb{E}_{\\xi_t}[\\|x_t - w_t^a\\|] \\; \\leq \\; \\alpha_t n_a\\sqrt{G + \\bar{G}M_{\\nabla}}.\n\\end{equation}\nSimilarly, merging~\\eqref{part2_eq4} into~\\eqref{part2_eq2} with $i = b$ leads to \n\\begin{equation}\n\\label{part2_eq6.5}\n        \\|x_t - w_t^b\\| \\;\\leq \\; \\alpha_t \\sum_{j=0}^{n_a - 1} \\|g^a(y^a_{j,t}, \\xi_{t}^{j})\\| + \\alpha_t  \\sum_{j=0}^{n_b - 1} \\|g^b(y^b_{j,t}, \\xi_{t}^{n_a+j})\\|,\n\\end{equation}\nand applying expectations\n\\begin{equation}\n\\label{part2_eq7}\n        \\mathbb{E}_{\\xi_t}[\\|x_t - w_t^b\\|] \\;\\leq \\; \\alpha_t (n_a+n_b) \\sqrt{G + \\bar{G}M_{\\nabla}}.\n\\end{equation}\nFinally, combining~\\eqref{theorem2_eq4}, \\eqref{part2_eq6}, and~\\eqref{part2_eq7} yields\n\\begin{equation}\n\\label{part2_eq8}\n    \\begin{split}\n        2\\alpha_t\\|x_t - x_*\\|\\sum_{i \\in \\{a, b\\}} \\mathbb{E}_{\\xi_t}[\\|n_i \\nabla f^i(w_t^i) - n_i \\nabla f^i(x_t)\\|] \\;\\leq\\; 2\\alpha_t^2L\\Theta (n_a + n_b)^2\\sqrt{G + \\bar{G}M_{\\nabla}}.\n    \\end{split}\n\\end{equation}\n\n\\vspace{2ex}\n\\noindent {\\bf Part III: Bound on the optimality gap in terms of weighted-sum function.}\nApplying inequalities \\eqref{theorm2_eq3} and \\eqref{part2_eq8} to \\eqref{theorem2_eq22} leads to \n\\begin{equation}\n\\label{smooth_sc_part3_eq1}\n    \\begin{split}\n     \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2]  \\;\\leq\\; &  (1-\\alpha_t(n_a + n_b)c) \\|x_t - x_*\\|^2 -  2\\alpha_t(n_a + n_b)(S(x_t, \\lambda_*) - S(x_*, \\lambda_*)) \\\\\n     & + \\alpha^2_t M,\n    \\end{split}\n\\end{equation}\nwhere we let $M = 2(n_a + n_b)^2 (G + \\bar{G}M_{\\nabla} +  L\\Theta\\sqrt{G + \\bar{G}M_{\\nabla}})$. Plugging in $\\alpha_t = \\frac{2}{c(n_a+n_b)(t+1)}$ and rearranging the last inequality result in \n\\begin{equation*}\n    \\begin{split}\n        S(x_t, \\lambda_*) - S(x_*, \\lambda_*)  \\;\\leq\\; & \\frac{(1-\\alpha_t(n_a + n_b)c) \\|x_t - x_*\\|^2 - \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2] + \\alpha^2_t M}{2\\alpha_t(n_a + n_b)} \\\\\n        \\;\\leq\\; & \\frac{c(t-1)}{4}\\|x_t - x_*\\|^2  - \\frac{c(t+1)}{4} \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2] + \\frac{\\tilde{M}}{c(t+1)},\n    \\end{split}\n\\end{equation*}\nwhere $\\tilde{M} = \\frac{M}{(n_a + n_b)^2}$. By taking total expectation over $\\{\\xi_t\\}$, multiplying both sides by $t$, and summing over $t = 1, \\ldots, T$, one obtains\n\\begin{equation*}\n    \\begin{split}\n       \\sum_{t=1}^T \n       t(\\mathbb{E}[S(x_t, \\lambda_*)] - S(x_*, \\lambda_*)) \\;\\leq\\; & \\sum_{t=1}^T \\left ( \\frac{ct(t-1)}{4}\\mathbb{E}[\\|x_t - x_*\\|^2]  - \\frac{ct(t+1)}{4} \\mathbb{E}[\\|x_{t+1} - x_*\\|^2]\\right) \\\\\n       & + \\sum_{t=1}^T \\frac{\\tilde{M}t}{c(t+1)} \\\\\n       \\;\\leq\\; & -\\frac{cT(T+1)}{4} \\mathbb{E}[\\|x_{T+1} - x_*\\|^2] + \\sum_{t=1}^T \\frac{\\tilde{M}t}{c(t+1)}  \\;\\leq\\; \\frac{T}{c} \\tilde{M}. \n    \\end{split}\n\\end{equation*}\nDividing both sides of the last inequality by $\\sum_{t=1}^T t$ yields\n\\begin{equation*}\n    \\min_{t = 1, \\ldots,T} \\mathbb{E}[S(x_t, \\lambda_*)] - S(x_*, \\lambda_*) \\;\\leq\\; \\frac{2}{c(T + 1)} \\tilde{M},   \n\\end{equation*}\nwhich concludes the proof.\n\\end{proof}\n\n\\begin{proposition} \\label{prop_ivt}\nGiven a continuous real function $\\phi(x): \\mathbb{R}^n \\to \\mathbb{R}$ and a set of points $\\{x_j\\}_{j =1}^m$, there exists $w \\in \\mathbb{R}^n$ such that \n\\[m\\phi(w) = \\sum_{j = 1}^m \\phi(x_j),\\]\nwhere~$w = \\sum_{j = 1}^m \\mu_j x_j,\\mbox{ with }\\sum_{j = 1}^m \\mu_j = 1, \\mu_j \\geq 0$, $i = 1, \\ldots, m$, is a convex linear combination of $\\{x_j\\}_{j =1}^m$. \n\\end{proposition}\n\n\\begin{proof}\nThe proposition is obtained by consecutively applying the Intermediate Value Theorem to $\\phi(x)$. First, for the pair of points $x_1$ and $x_2$, there exists a point $w_{12} = \\mu_{12} x_1 + (1-\\mu_{12}) x_2, \\mu_{12} \\in [0, 1]$, such that~$\\phi(w_{12}) = (\\phi(x_1) + \\phi(x_2))/2$ according to the Intermediate Value Theorem, which implies that $\\sum_{j = 1}^m \\phi(x_j) = 2\\phi(w_{12}) + \\sum_{j = 3}^m \\phi(x_j)$.  Then, there exists $w_{13} = \\mu_{13} w_{12} + (1-\\mu_{13}) x_3, \\mu_{13} \\geq 0$, such that $\\phi(w_{13}) = (2\\phi(w_{12}) + \\phi(x_3))/3$ holds given that the average function value $(2\\phi(w_{12}) + \\phi(x_3))/3$ lies between $\\phi(w_{12})$ and $\\phi(x_3)$. Notice that~$w_{13}$ can also be written as convex linear combination of~$\\{x_1, x_2, x_3\\}$. The proof is concluded by continuing this process until $x_m$ is reached. \n\\end{proof}","units":[{"start":25369,"end":36423,"start_line":311,"end_line":476,"raw_tex":"\\begin{proof}\nThe proof is divided in three parts for better organization and understanding. In the first part, one obtains an upper bound on the norm of the iterates, $\\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2]$. Strong convexity of the weighted-sum function is applied in the second part. The third part concludes the proof using standard arguments. For simplicity, we let $z_t = y^b_{0, t} = y^a_{n_a, t}$ be the intermediate point at each iteration.  \\vspace{1ex}\n\n\\noindent {\\bf Part I: Bound on the iterates error using the Intermediate Value Theorem.} \nAt any iteration $t$, the sequence of stochastic gradients is computed from drawing the sequence of random variables $\\xi_t = \\{\\xi^0_t, \\ldots, \\xi^{n_a + n_b - 1}_t \\}$. We have\n\\begin{align*}\n    x_{t+1} - x_* & \\;=\\; P_{\\mathcal{X}}\\left(z_t - \\alpha_t \\sum_{r = 0}^{n_b - 1} g^b(y^b_{r, t}, \\xi^r_t)\\right) - x_*, \\\\\n    &\\;=\\;P_{\\mathcal{X}}\\left(\n    x_t - \\alpha_t \\sum_{r = 0}^{n_a - 1}  g^a(y^a_{r, t}, \\xi^r_t) - \\alpha_t \\sum_{r = 0}^{n_b - 1}  g^b(y^b_{r, t}, \\xi^{n_a + r}_t) \\right) - x_*.\n\\end{align*}\n\nSince the sequence $\\xi_t$ is drawn independently, using Assumption~\\ref{ass:gradient} (a) one has\n\\[\\mathbb{E}_{\\xi_t}[g^a(y_{r, t}^a, \\xi_{t}^r)] \\;=\\; \\mathbb{E}_{\\xi_t^0, \\ldots, \\xi_{t}^{r-1}}[\\mathbb{E}_{\\xi_t^r}[g^a(y_{r, t}^a, \\xi_{t}^r)]] \\;=\\; \\mathbb{E}_{\\xi_t^0, \\ldots, \\xi_{t}^{r-1}}[\\nabla f^a(y_{r, t}^a)] \\;=\\; \\mathbb{E}_{\\xi_t}[\\nabla f^a(y_{r, t}^a)],\\]\nwhere the last equality holds due to the independence between $y_{r, t}^a$ and $\\{\\xi_t^{r}, \\ldots, \\xi_t^{n_a + n_b - 1}\\}$. Similarly, we have $\\mathbb{E}_{\\xi_t}[g^b(y_{r, t}^b, \\xi_{t}^{n_a + r})] = \\mathbb{E}_{\\xi_t}[\\nabla f^b(y_{r, t}^b)]$.\nThen, taking square norms and expectations over the random variables $\\xi_t$ on both sides yields\n\\begin{equation}\n\\label{theorem2_eq1}\n    \\begin{split}\n    \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2]  \n     \\;\\leq\\; & \\|x_t - x_*\\|^2 + \\alpha_t^2\\mathbb{E}_{\\xi_t}[\\|\\sum_{r = 0}^{n_a - 1} g^a(y^a_{r, t}, \\xi^r_t) + \\sum_{r = 0}^{n_b - 1} g^b(y^b_{r, t}, \\xi^{n_a + r}_t)\\|^2] \\\\\n     & - \\mathbb{E}_{\\xi_t}[2\\alpha_t(x_t - x_*)^\\top (\\sum_{r = 0}^{n_a - 1} \\nabla f^a(y^a_{r, t}) + \\sum_{r = 0}^{n_b - 1} \\nabla f^b(y^b_{r, t}))],\n    \\end{split}\n\\end{equation}\nwhich holds by the non-expansiveness property of the orthogonal projection operator.\n\nWe now claim, by applying a version of the Intermediate Value Theorem given in Proposition~\\ref{prop_ivt}, that the last term of the right-hand side in~\\eqref{theorem2_eq1} can be written as $- 2\\alpha_t(x_t - x_*)^\\top \\mathbb{E}_{\\xi_t}[n_a \\nabla f^a(w_t^a) + n_b \\nabla f^b(w_t^b)]$ for some $w_t^a$ and $w_t^b$. In fact, we apply Proposition~\\ref{prop_ivt} to the real continuous function $\\phi^i(y) = -2\\alpha_t(x_t - x_*)^\\top \\nabla f^i(y)$, from which we then know that $w_t^i$ is a convex linear combination of a sequence of points $\\{y_{r, t}^i\\}_{r = 0}^{n_i - 1}$ for both $i \\in \\{a, b\\}$. \n\nUsing a combination of~(\\ref{Mnabla}) and Assumption~\\ref{ass:gradient}~(b), the bound for the second moment of the stochastic gradients is given by\n\\begin{equation} \\label{SGbound}\n\\mathbb{E}_{\\xi_t}[\\|g^i(y^i_{r, t}, \\xi_t^r)\\|^2] \\; \\leq \\; G + \\bar{G}M_{\\nabla}, \\quad\n\\mathbb{E}_{\\xi_t}[\\|g^i(y^i_{r, t}, \\xi_t^r)\\|] \\; \\leq \\; \\sqrt{G + \\bar{G}M_{\\nabla}},\n\\end{equation} \nwhere the second bound results from applying Jensen's inequality to the first one.\nHence, the second term in the right-hand side of~\\eqref{theorem2_eq1} can be bounded by\n\\begin{equation*}\n\\begin{split}\n   & \\mathbb{E}_{\\xi_t}[\\|\\sum_{r = 0}^{n_a - 1} g^a(y^a_{r, t}, \\xi^r_t) + \\sum_{r = 0}^{n_b - 1}  g^b(y^b_{r, t}, \\xi^{n_a + r}_t)\\|^2] \\\\\n   &  \\;\\leq\\; 2\\mathbb{E}_{\\xi_t}[\\|\\sum_{r = 0}^{n_a - 1}  g^a(y^a_{r, t}, \\xi^r_t)\\|^2] + 2\\mathbb{E}_{\\xi_t}[\\|\\sum_{r = 0}^{n_b - 1}  g^b(y^b_{r, t}, \\xi^{n_a+r}_t)\\|^2] \\\\\n& \\;\\leq\\;  2n_a\\sum_{r = 0}^{n_a - 1} \\mathbb{E}_{\\xi_t}[\\| g^a(y^a_{r, t}, \\xi^r_t)\\|^2] + 2n_b\\sum_{r = 0}^{n_b - 1} \\mathbb{E}_{\\xi_t}[\\| g^b(y^b_{r, t}, \\xi^{n_a + r}_t)\\|^2] \\\\\n& \\;\\leq\\;  2(n_a + n_b)^2(G + \\bar{G}M_{\\nabla}).\n\\end{split}\n\\end{equation*}\nWe thus arrive at\n\\begin{equation}\n    \\label{theorem2_eq2}\n    \\begin{split}\n    \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2]\n    \\;\\leq\\; &  \\|x_t - x_*\\|^2 + 2\\alpha_t^2(n_a + n_b)^2(G + \\bar{G}M_{\\nabla}) \\\\\n     & - 2\\alpha_t(x_t - x_*)^\\top \\mathbb{E}_{\\xi_t}[n_a \\nabla f^a(w_t^a) + n_b \\nabla f^b(w_t^b)].\n    \\end{split}\n\\end{equation}\n\nBy adding and subtracting $2\\alpha_t(x_t - x_*)^\\top(n_a\\nabla f^a(x_t) + n_b \\nabla f^b(x_t))$ in the right-hand side of~\\eqref{theorem2_eq2}, we further rewrite it as\n\\begin{equation}\n\\label{theorem2_eq22}\n    \\begin{split}\n    \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2]\n    \\;\\leq\\; &  \\|x_t - x_*\\|^2 + 2\\alpha_t^2(n_a + n_b)^2(G + \\bar{G}M_{\\nabla}) \\\\\n     & - 2\\alpha_t(x_t - x_*)^\\top (n_a \\nabla f^a(x_t) + n_b \\nabla f^b(x_t)) \\\\\n     & + 2\\alpha_t\\|x_t - x_*\\|\\mathbb{E}_{\\xi_t}[\\|n_a \\nabla f^a(w_t^a) - n_a \\nabla f^a(x_t)\\|] \\\\\n     & + 2\\alpha_t\\|x_t - x_*\\|\\mathbb{E}_{\\xi_t}[\\|n_b \\nabla f^b(w^b_t) - n_b \\nabla f^b(x_t)\\|]. \n    \\end{split}\n\\end{equation}\nNote that the last two terms are derived by the Cauchy–Schwarz and Jensen's inequalities.\n\n\\vspace{2ex}\n\\noindent {\\bf Part II: Using strong convexity.} \nSelecting $\\lambda_* = \\lambda(n_a, n_b) = n_a/(n_a + n_b)$, by the strong convexity of the weighted-sum function, one has\n\\begin{equation*}\n \\nabla_x S(x_t, \\lambda_*)^\\top(x_t - x_*)  \\;\\geq\\; S(x_t, \\lambda_*) - S(x_*, \\lambda_*) + \\frac{c}{2}\\| x_t - x_*\\|^2,\n\\end{equation*}\nwhich is equivalent to\n\\begin{equation}\n\\label{theorm2_eq3}\n (x_t - x_*)^\\top(n_a \\nabla f^a(x_t) + n_b \\nabla f^b(x_t))  \\;\\geq\\; (n_a + n_b)(S(x_t, \\lambda_*) - S(x_*, \\lambda_*) + \\frac{c}{2}\\| x_t - x_*\\|^2).\n\\end{equation}\n\nFrom Assumption~\\ref{ass:Lipschitz}, we obtain a bound for the last two terms of~\\eqref{theorem2_eq22} in the form \n\\begin{equation}\n    \\label{theorem2_eq4}\n    2\\alpha_t\\|x_t - x_*\\|\\mathbb{E}_{\\xi_t}[\\|n_i \\nabla f^i(w_t^i) - n_i \\nabla f^i(x_t)\\|] \\;\\leq\\;  2\\alpha_tLn_i\\|x_t - x_*\\|\\mathbb{E}_{\\xi_t}[\\|x_t - w_t^i\\|], \\forall i \\in \\{a, b\\}.\n\\end{equation}\nAccording to Proposition~\\ref{prop_ivt}, $w_t^i$ is a convex linear combination of a sequence of points $\\{y_{r, t}^i\\}_{r = 0}^{n_i -1}$ for $i \\in \\{a, b\\}$. One can write $w_t^i = \\sum_{r=0}^{n_i - 1} \\beta_r y_{r, t}^i$ with $\\beta_r \\geq 0$, $r = 0, \\ldots, n_i - 1$ , and $\\sum_{r=0}^{n_i - 1} \\beta_r = 1$. An explicit upper bound of $\\|x_t - w_t^i\\|$ can then be derived as follows\n\\begin{equation}\n\\label{part2_eq2}\n    \\begin{split} \n        \\|x_t - \\sum_{r=0}^{n_i - 1} \\beta_r y_{r, t}^i\\| & \\;=\\; \\|\\sum_{r=0}^{n_i - 1} \\beta_r (x_t - y_{r, t}^i)\\| \\;\\leq\\; \\sum_{r=0}^{n_i - 1} \\beta_r \\|x_t - y_{r, t}^i\\|.\n    \\end{split}\n\\end{equation}\nUsing $y_{r, t}^i = y_{0, t}^i - \\sum_{j = 0}^{r-1}\\alpha_t g^i(y_{j, t}^i, \\xi_{t}^j)$ and applying the triangle inequality, we have\n\\begin{align}\n    \\|x_t - y_{r, t}^a\\| \\;\\leq\\; \\alpha_t \\sum_{j = 0}^{r-1} \\|g^a(y^a_{j, t}, \\xi_{t}^j)\\|,\\label{part2_eq3}\n\\end{align}  \nand\n\\begin{equation}\n\\label{part2_eq4}\n\\begin{split}\n    \\|x_t - y_{r, t}^b\\| & \\;=\\; \\|x_t - y_{n_a, t}^a + \\sum_{j = 0}^{r-1}\\alpha_t g^b(y_{j, t}^b, \\xi_{t}^{n_a + j})\\| \\\\\n    & \\;\\leq\\; \\alpha_t \\sum_{j = 0}^{n_a-1} \\|g^a(y^a_{j, t}, \\xi_{t}^j)\\| +  \\alpha_t \\sum_{j = 0}^{r-1} \\|g^b(y^b_{j, t}, \\xi_{t}^{n_a + j})\\|.\n\\end{split}\n\\end{equation}\nPlugging~\\eqref{part2_eq3} into~\\eqref{part2_eq2} with $i = a$ results in (note that $y^a_{0,t}=x_t$)\n\\begin{equation}\\label{part2_eq5}\n\\begin{split}\n       \\|x_t - w_t^a\\| & \\;\\leq\\; \\alpha_t \\sum_{r=1}^{n_a - 1} \\beta_r \\sum_{j = 0}^{r-1} \\|g^a(y^a_{j, t}, \\xi_{t}^j)\\| \\\\ & \\;  = \\;  \\alpha_t \n       \\sum_{j=0}^{n_a - 1} \\|g^a(y^a_{j, t}, \\xi_{t}^j)\\|\n      \\sum_{r=j+1}^{n_a-1} \\beta_r\n       \\;\\leq \\; \\alpha_t  \\sum_{j=0}^{n_a - 1} \\|g^a(y^a_{j,t}, \\xi_{t}^{j})\\|.\n\\end{split}\n\\end{equation}\n\n\nApplying expectations on both sides of~\\eqref{part2_eq5}, followed by the bound~(\\ref{SGbound}) on the second moment of the stochastic gradients,\n\\begin{equation} \\label{part2_eq6}\n\\mathbb{E}_{\\xi_t}[\\|x_t - w_t^a\\|] \\; \\leq \\; \\alpha_t n_a\\sqrt{G + \\bar{G}M_{\\nabla}}.\n\\end{equation}\nSimilarly, merging~\\eqref{part2_eq4} into~\\eqref{part2_eq2} with $i = b$ leads to \n\\begin{equation}\n\\label{part2_eq6.5}\n        \\|x_t - w_t^b\\| \\;\\leq \\; \\alpha_t \\sum_{j=0}^{n_a - 1} \\|g^a(y^a_{j,t}, \\xi_{t}^{j})\\| + \\alpha_t  \\sum_{j=0}^{n_b - 1} \\|g^b(y^b_{j,t}, \\xi_{t}^{n_a+j})\\|,\n\\end{equation}\nand applying expectations\n\\begin{equation}\n\\label{part2_eq7}\n        \\mathbb{E}_{\\xi_t}[\\|x_t - w_t^b\\|] \\;\\leq \\; \\alpha_t (n_a+n_b) \\sqrt{G + \\bar{G}M_{\\nabla}}.\n\\end{equation}\nFinally, combining~\\eqref{theorem2_eq4}, \\eqref{part2_eq6}, and~\\eqref{part2_eq7} yields\n\\begin{equation}\n\\label{part2_eq8}\n    \\begin{split}\n        2\\alpha_t\\|x_t - x_*\\|\\sum_{i \\in \\{a, b\\}} \\mathbb{E}_{\\xi_t}[\\|n_i \\nabla f^i(w_t^i) - n_i \\nabla f^i(x_t)\\|] \\;\\leq\\; 2\\alpha_t^2L\\Theta (n_a + n_b)^2\\sqrt{G + \\bar{G}M_{\\nabla}}.\n    \\end{split}\n\\end{equation}\n\n\\vspace{2ex}\n\\noindent {\\bf Part III: Bound on the optimality gap in terms of weighted-sum function.}\nApplying inequalities \\eqref{theorm2_eq3} and \\eqref{part2_eq8} to \\eqref{theorem2_eq22} leads to \n\\begin{equation}\n\\label{smooth_sc_part3_eq1}\n    \\begin{split}\n     \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2]  \\;\\leq\\; &  (1-\\alpha_t(n_a + n_b)c) \\|x_t - x_*\\|^2 -  2\\alpha_t(n_a + n_b)(S(x_t, \\lambda_*) - S(x_*, \\lambda_*)) \\\\\n     & + \\alpha^2_t M,\n    \\end{split}\n\\end{equation}\nwhere we let $M = 2(n_a + n_b)^2 (G + \\bar{G}M_{\\nabla} +  L\\Theta\\sqrt{G + \\bar{G}M_{\\nabla}})$. Plugging in $\\alpha_t = \\frac{2}{c(n_a+n_b)(t+1)}$ and rearranging the last inequality result in \n\\begin{equation*}\n    \\begin{split}\n        S(x_t, \\lambda_*) - S(x_*, \\lambda_*)  \\;\\leq\\; & \\frac{(1-\\alpha_t(n_a + n_b)c) \\|x_t - x_*\\|^2 - \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2] + \\alpha^2_t M}{2\\alpha_t(n_a + n_b)} \\\\\n        \\;\\leq\\; & \\frac{c(t-1)}{4}\\|x_t - x_*\\|^2  - \\frac{c(t+1)}{4} \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2] + \\frac{\\tilde{M}}{c(t+1)},\n    \\end{split}\n\\end{equation*}\nwhere $\\tilde{M} = \\frac{M}{(n_a + n_b)^2}$. By taking total expectation over $\\{\\xi_t\\}$, multiplying both sides by $t$, and summing over $t = 1, \\ldots, T$, one obtains\n\\begin{equation*}\n    \\begin{split}\n       \\sum_{t=1}^T \n       t(\\mathbb{E}[S(x_t, \\lambda_*)] - S(x_*, \\lambda_*)) \\;\\leq\\; & \\sum_{t=1}^T \\left ( \\frac{ct(t-1)}{4}\\mathbb{E}[\\|x_t - x_*\\|^2]  - \\frac{ct(t+1)}{4} \\mathbb{E}[\\|x_{t+1} - x_*\\|^2]\\right) \\\\\n       & + \\sum_{t=1}^T \\frac{\\tilde{M}t}{c(t+1)} \\\\\n       \\;\\leq\\; & -\\frac{cT(T+1)}{4} \\mathbb{E}[\\|x_{T+1} - x_*\\|^2] + \\sum_{t=1}^T \\frac{\\tilde{M}t}{c(t+1)}  \\;\\leq\\; \\frac{T}{c} \\tilde{M}. \n    \\end{split}\n\\end{equation*}\nDividing both sides of the last inequality by $\\sum_{t=1}^T t$ yields\n\\begin{equation*}\n    \\min_{t = 1, \\ldots,T} \\mathbb{E}[S(x_t, \\lambda_*)] - S(x_*, \\lambda_*) \\;\\leq\\; \\frac{2}{c(T + 1)} \\tilde{M},   \n\\end{equation*}\nwhich concludes the proof.\n\\end{proof}","sha256_utf8":"1519b9880459bcb39bd3aa172bf0be7cd83e801252685a5b13daa51b6cd8f14f"},{"start":64237,"end":64654,"start_line":824,"end_line":828,"raw_tex":"\\begin{proposition} \\label{prop_ivt}\nGiven a continuous real function $\\phi(x): \\mathbb{R}^n \\to \\mathbb{R}$ and a set of points $\\{x_j\\}_{j =1}^m$, there exists $w \\in \\mathbb{R}^n$ such that \n\\[m\\phi(w) = \\sum_{j = 1}^m \\phi(x_j),\\]\nwhere~$w = \\sum_{j = 1}^m \\mu_j x_j,\\mbox{ with }\\sum_{j = 1}^m \\mu_j = 1, \\mu_j \\geq 0$, $i = 1, \\ldots, m$, is a convex linear combination of $\\{x_j\\}_{j =1}^m$. \n\\end{proposition}","sha256_utf8":"e43baba5fee93b6d52b8dc31ee176d85289368e09a8fd45d8982733ccb4987f7"},{"start":64656,"end":65532,"start_line":830,"end_line":832,"raw_tex":"\\begin{proof}\nThe proposition is obtained by consecutively applying the Intermediate Value Theorem to $\\phi(x)$. First, for the pair of points $x_1$ and $x_2$, there exists a point $w_{12} = \\mu_{12} x_1 + (1-\\mu_{12}) x_2, \\mu_{12} \\in [0, 1]$, such that~$\\phi(w_{12}) = (\\phi(x_1) + \\phi(x_2))/2$ according to the Intermediate Value Theorem, which implies that $\\sum_{j = 1}^m \\phi(x_j) = 2\\phi(w_{12}) + \\sum_{j = 3}^m \\phi(x_j)$.  Then, there exists $w_{13} = \\mu_{13} w_{12} + (1-\\mu_{13}) x_3, \\mu_{13} \\geq 0$, such that $\\phi(w_{13}) = (2\\phi(w_{12}) + \\phi(x_3))/3$ holds given that the average function value $(2\\phi(w_{12}) + \\phi(x_3))/3$ lies between $\\phi(w_{12})$ and $\\phi(x_3)$. Notice that~$w_{13}$ can also be written as convex linear combination of~$\\{x_1, x_2, x_3\\}$. The proof is concluded by continuing this process until $x_m$ is reached. \n\\end{proof}","sha256_utf8":"8153cb1c16f203d799ccb6daa0a8e1f24bcc88606b8faf4656a2d6f4d5f2c39d"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"随机双目标问题","source_units":[{"start":6856,"end":6978,"start_line":162,"end_line":165,"raw_tex":"\\begin{equation}\n\\label{biobjective_smooth}\n    \\min~F(x) = (f^a(x), f^b(x))^\\top, \\quad x \\in \\mathcal{X},\n\\end{equation}","sha256_utf8":"9b019f8376edc3103224de298dffb9c5b882767f1d801a129bb607e472f0ec99"},{"start":6979,"end":7587,"start_line":166,"end_line":167,"raw_tex":"where $\\mathcal{X}$ is a nonempty bounded closed convex region. \nWe say that the multi-objective problem is smooth if all objective functions~$f^i$ are continuously differentiable. In the convergence theory of our algorithms, we will assume that both functions are convex. Furthermore, both objectives involve randomness in its parameters, in which case we expect that a true gradient or subgradient is not available or too expensive to compute. Instead, one can access a unbiased estimator $g^i(x, \\xi), i \\in \\{a, b\\}$, of a true gradient or subgradient of $f^i$, where $\\xi$ denotes some random variable.\n","sha256_utf8":"ea13fe48a78fa0851dd0132330834cdab40b36fc66163e86cd8995dbb29da94c"}]},{"id":"S002","semantic_type":"algorithm","kind":"algorithm","title":"随机交替梯度算法","source_units":[{"start":16493,"end":17393,"start_line":203,"end_line":204,"raw_tex":"The key idea of the alternating bi-objective optimization for solving~\\eqref{biobjective_smooth} consists of iteratively taking $n_a$ gradient (or subgradient) descent steps for the first objective and then $n_b$ gradient (or subgradient) descent steps for the second objective. For simplicity, we denote such $n_a + n_b$ steps as one single iteration of the algorithm. The stochastic alternating bi-objective gradient or subgradient algorithm (SA2GD) is formally described in Algorithm~\\ref{alg1_SA2GD}. At every iteration $t$, starting at $x_t = y^a_{0, t}$, the algorithm computes two sequences of intermediate iterates, $\\{y^a_{r, t}\\}_{r=1}^{n_a}$ and $\\{y^b_{r, t}\\}_{r=1}^{n_b}$. Let $P_{\\mathcal{X}}$ denote the orthogonal projection operator that projects \nthe new iterate back to the feasible region. Such an operator is well-defined because we assume a compact and convex feasible region.\n","sha256_utf8":"6d333ce8cc0d5fac1fdd7b73e5a8ecb1e5fdef6d14e74c224fe6df3eba10cd83"},{"start":17480,"end":18510,"start_line":208,"end_line":225,"raw_tex":"\\begin{algorithm}[ht]\n\\caption{Stochastic alternating bi-objective gradient (or subgradient) algorithm (SA2GD)} \n\\label{alg1_SA2GD} \n\\begin{algorithmic}[1]\n\\item \\textbf{Input:} Initial point $x_0 = y^a_{0, 0}$ and a step size sequence $\\{\\alpha_t\\}$.\n\\item \\textbf{Output:} A likely non-dominated or Pareto solution $x_{T} = y^a_{0, T}$.\n\\item {\\bf for} $t = 0, 1, \\ldots, T-1$ {\\bf do}\n\\item \\quad {\\bf for} $r = 0, \\ldots, n_a-1$ {\\bf do}\n\\item \\quad \\quad Generate a stochastic gradient $g^a(y^a_{r, t}, \\xi^r_{t})$.\n\\item \\quad \\quad Update $y^a_{r+1, t} = y^a_{r, t} - \\alpha_t g^a(y^a_{r, t}, \\xi^r_{t})$. \n\\item \\quad Set $y^b_{0, t} = y^a_{n_a, t}$.\n\\item \\quad {\\bf for} $r = 0, \\ldots, n_b - 1$ {\\bf do}\n\\item \\quad \\quad Generate a stochastic gradient $g^b(y^b_{r, t}, \\xi^{n_a + r}_{t})$.\n\\item \\quad \\quad Update $y^b_{r+1, t} = y^b_{r, t} - \\alpha_t g^b(y^b_{r, t}, \\xi^{n_a + r}_{t})$. \n\\item \\quad Set $x_{t+1} = y^a_{0, t+1} = P_{\\mathcal{X}}(y^b_{n_b, t})$.\n\\par\\vspace*{0.1cm}\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"4d4373104be200c57f5dc63abb3a08c7d6e791a7027fbb5ef316fd49e000aec3"}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","title":"可行域直径有界","source_units":[{"start":20948,"end":21296,"start_line":242,"end_line":249,"raw_tex":"\\begin{assumption}\n\\label{ass:limit_pts}\n\\textbf{(Bounded feasible region)}\nThe feasible region $\\mathcal{X}$ is compact, and in particular there exists a positive constant $\\Theta$ such that\n\\begin{equation*}\n\\displaystyle \\max_{x, \\bar{x} \\in \\mathcal{X}} \\| x - \\bar{x}\\| \\;\\leq\\; \\textstyle \\Theta \\;<\\; \\infty.\n\\end{equation*}\n\\end{assumption}","sha256_utf8":"e17e6d5a5d2137677e3fbaf0af2a241a6bc709d1eddfa0f82c4457dd5a5acc3f"}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","title":"梯度利普希茨连续","source_units":[{"start":21396,"end":21849,"start_line":253,"end_line":262,"raw_tex":"\\begin{assumption}\n\\label{ass:Lipschitz}\n\\textbf{(Lipschitz continuous gradients)} The individual true gradients are Lipschitz continuous with Lipschitz constants $L_i > 0, i \\in \\{a, b\\}$, i.e., \n\\begin{equation*}\n    % \\label{eq:Lipschitz}\n    \\begin{split}\n        \\|\\nabla f^i(x) - \\nabla f^i(\\bar{x}) \\| \\;\\leq\\;& L_i \\| x - \\bar{x} \\|, \\quad \\forall (x, \\bar{x}) \\in \\mathcal{X} \\times \\mathcal{X}.\n    \\end{split}\n\\end{equation*}\n\\end{assumption}","sha256_utf8":"69a772d0b1df100b37fd1c942163435a1de4cd6b12d9a16df4488f9a4f2f0a07"}]},{"id":"S005","semantic_type":"derived_fact","kind":"derived_fact","title":"真梯度统一有界","source_units":[{"start":21851,"end":22150,"start_line":264,"end_line":267,"raw_tex":"Note that any continuous function  attains a maximum on a compact set, and thus one can assume the existence of positive constants~$M_\\nabla^i$, $i \\in \\{a,b\\}$, such that\n\\begin{equation} \\label{Mnabla}\n\\| \\nabla f^i(x) \\|^2 \\; \\leq \\; M_\\nabla^i,  \\quad \\forall x \\in \\mathcal{X}.  \n\\end{equation}","sha256_utf8":"123269cd4babc3f1582e9054c922c4842a902ebd933014e8b53e059888c16388"}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","title":"两个目标强凸","source_units":[{"start":22255,"end":22604,"start_line":270,"end_line":277,"raw_tex":"\\begin{assumption}\n\\label{ass:strongconv}\n\\textbf{(Strong convexity)} For all $i = \\{a, b\\}$, there exists a scalar $c_i > 0$ such that\n\n\\begin{equation*}\nf^i(\\bar{x}) \\;\\geq\\; f^i(x) + \\nabla f^i(x)^\\top(\\bar{x} - x) + \\frac{c_i}{2} \\| \\bar{x} - x \\|^2, \\quad \\forall (x,\\bar{x}) \\in \\mathcal{X} \\times \\mathcal{X}.\n\\end{equation*}\n\\end{assumption}","sha256_utf8":"8afa2871d92d32c7a30b99b7bd83c13412e8f4f723cf2aecf84ea3a3d3589f9b"}]},{"id":"S007","semantic_type":"notation","kind":"notation","title":"加权目标与条件期望","source_units":[{"start":22606,"end":23117,"start_line":279,"end_line":279,"raw_tex":"Note that based on the above assumption, the weighted-sum function $S(x, \\lambda) = \\lambda f^a(x) + (1-\\lambda)f^b(x), \\lambda \\in [0, 1]$, is also strongly convex with constant $c = \\min(c_a, c_b)$. Given the individual stochastic gradients $g^i(x_t, \\xi_t), \\forall i \\in \\{a, b\\}$, generated with random variable~$\\xi_t$, we use $\\mathbb{E}_{\\xi_t}[\\cdot]$ to denote the conditional expectation taken with respect to $\\xi_t$. We also impose the following two classical assumptions of stochastic gradients. \n","sha256_utf8":"32c1a4e10a23220c07c7a7fe8ddbceb8e64ce7a34c4ae468b785a69f8d9575c7"}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","title":"随机梯度无偏及矩界","source_units":[{"start":23118,"end":23815,"start_line":281,"end_line":295,"raw_tex":"\\begin{assumption}\n\\label{ass:gradient}\nFor both objective functions $i \\in \\{a, b\\}$, and all iterates $t \\in \\mathbb{N}$, the individual stochastic gradients~$g^i(x_t, \\xi_t)$ satisfy the following:\n\\begin{enumerate}\n    \\item[(a)]\\textbf{(Unbiased gradient estimation)} \n    $\\mathbb{E}_{\\xi_t}[g^i(x_t, \\xi_t)] \\;=\\; \\nabla f^i(x_t), ~\\forall i \\in \\{a, b\\}$. \n    \\item[(b)]\\textbf{(Bound on the second moment)}\nThere exist positive scalars $G_i > 0$ and $\\bar{G}_i > 0$ such that\n\\begin{equation*}\n\\begin{split}\n\\mathbb{E}_{\\xi_t}[ g^i(x_t, \\xi_t)\\|^2] \\;\\leq\\; G_i + \\bar{G}_i\\| \\nabla f^i(x_t)\\|^2, \\quad \\forall i \\in \\{a, b\\}.\n\\end{split}\n\\end{equation*}\n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"e9b4494cdc1c8b059daaeec3613f69b5a5c7f67d70401300922a8d22a172980b"}]},{"id":"S009","semantic_type":"theorem","kind":"theorem","title":"随机交替算法的一阶次线性速率","source_units":[{"start":24492,"end":25367,"start_line":301,"end_line":309,"raw_tex":"\\begin{theorem} (\\textbf{Sublinear convergence rate of SA2GD in the smooth and strongly convex case})\n\\label{th:smooth_strongcov}\nLet Assumptions~\\ref{ass:limit_pts}-\\ref{ass:gradient} hold and $x_*$ be the unique minimizer of the weighted function $S(\\cdot, \\lambda_*)$ in~$\\mathcal{X}$, where $\\lambda_* = \\lambda(n_a, n_b) = n_a/(n_a + n_b)$. Choosing a diminishing step size sequence\n$\\alpha_t = \\frac{2}{c(t+1)(n_a + n_b)}$, the sequence of iterates generated by the SA2GD algorithm satisfies\n\\begin{equation*}\n    \\min_{t = 1, \\ldots,T} \\mathbb{E}[S(x_t, \\lambda_*)] - S(x_*, \\lambda_*) \\;\\leq\\; \\frac{4}{c(T + 1)} \\left(\\hat{G}^2 +  L\\Theta\\hat{G}\\right).   \n\\end{equation*}\n where $\\hat{G} = \\sqrt{G + \\bar{G}M_{\\nabla}}$, $L = \\max(L_a, L_b)$, $G = \\max(G_a, G_b)$, $\\bar{G} = \\max(\\bar{G}_a, \\bar{G}_b)$, and $M_\\nabla = \\max(M_\\nabla^a, M_\\nabla^b)$.\n\\end{theorem}","sha256_utf8":"bdf89f1fe2b568ef28d1d8ecc3625a30ddb28f8dc8a0d569a53b01fdb99af340"}]},{"id":"S010","semantic_type":"proof","kind":"proof","title":"光滑强凸收敛定理完整证明","source_units":[{"start":25369,"end":36423,"start_line":311,"end_line":476,"raw_tex":"\\begin{proof}\nThe proof is divided in three parts for better organization and understanding. In the first part, one obtains an upper bound on the norm of the iterates, $\\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2]$. Strong convexity of the weighted-sum function is applied in the second part. The third part concludes the proof using standard arguments. For simplicity, we let $z_t = y^b_{0, t} = y^a_{n_a, t}$ be the intermediate point at each iteration.  \\vspace{1ex}\n\n\\noindent {\\bf Part I: Bound on the iterates error using the Intermediate Value Theorem.} \nAt any iteration $t$, the sequence of stochastic gradients is computed from drawing the sequence of random variables $\\xi_t = \\{\\xi^0_t, \\ldots, \\xi^{n_a + n_b - 1}_t \\}$. We have\n\\begin{align*}\n    x_{t+1} - x_* & \\;=\\; P_{\\mathcal{X}}\\left(z_t - \\alpha_t \\sum_{r = 0}^{n_b - 1} g^b(y^b_{r, t}, \\xi^r_t)\\right) - x_*, \\\\\n    &\\;=\\;P_{\\mathcal{X}}\\left(\n    x_t - \\alpha_t \\sum_{r = 0}^{n_a - 1}  g^a(y^a_{r, t}, \\xi^r_t) - \\alpha_t \\sum_{r = 0}^{n_b - 1}  g^b(y^b_{r, t}, \\xi^{n_a + r}_t) \\right) - x_*.\n\\end{align*}\n\nSince the sequence $\\xi_t$ is drawn independently, using Assumption~\\ref{ass:gradient} (a) one has\n\\[\\mathbb{E}_{\\xi_t}[g^a(y_{r, t}^a, \\xi_{t}^r)] \\;=\\; \\mathbb{E}_{\\xi_t^0, \\ldots, \\xi_{t}^{r-1}}[\\mathbb{E}_{\\xi_t^r}[g^a(y_{r, t}^a, \\xi_{t}^r)]] \\;=\\; \\mathbb{E}_{\\xi_t^0, \\ldots, \\xi_{t}^{r-1}}[\\nabla f^a(y_{r, t}^a)] \\;=\\; \\mathbb{E}_{\\xi_t}[\\nabla f^a(y_{r, t}^a)],\\]\nwhere the last equality holds due to the independence between $y_{r, t}^a$ and $\\{\\xi_t^{r}, \\ldots, \\xi_t^{n_a + n_b - 1}\\}$. Similarly, we have $\\mathbb{E}_{\\xi_t}[g^b(y_{r, t}^b, \\xi_{t}^{n_a + r})] = \\mathbb{E}_{\\xi_t}[\\nabla f^b(y_{r, t}^b)]$.\nThen, taking square norms and expectations over the random variables $\\xi_t$ on both sides yields\n\\begin{equation}\n\\label{theorem2_eq1}\n    \\begin{split}\n    \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2]  \n     \\;\\leq\\; & \\|x_t - x_*\\|^2 + \\alpha_t^2\\mathbb{E}_{\\xi_t}[\\|\\sum_{r = 0}^{n_a - 1} g^a(y^a_{r, t}, \\xi^r_t) + \\sum_{r = 0}^{n_b - 1} g^b(y^b_{r, t}, \\xi^{n_a + r}_t)\\|^2] \\\\\n     & - \\mathbb{E}_{\\xi_t}[2\\alpha_t(x_t - x_*)^\\top (\\sum_{r = 0}^{n_a - 1} \\nabla f^a(y^a_{r, t}) + \\sum_{r = 0}^{n_b - 1} \\nabla f^b(y^b_{r, t}))],\n    \\end{split}\n\\end{equation}\nwhich holds by the non-expansiveness property of the orthogonal projection operator.\n\nWe now claim, by applying a version of the Intermediate Value Theorem given in Proposition~\\ref{prop_ivt}, that the last term of the right-hand side in~\\eqref{theorem2_eq1} can be written as $- 2\\alpha_t(x_t - x_*)^\\top \\mathbb{E}_{\\xi_t}[n_a \\nabla f^a(w_t^a) + n_b \\nabla f^b(w_t^b)]$ for some $w_t^a$ and $w_t^b$. In fact, we apply Proposition~\\ref{prop_ivt} to the real continuous function $\\phi^i(y) = -2\\alpha_t(x_t - x_*)^\\top \\nabla f^i(y)$, from which we then know that $w_t^i$ is a convex linear combination of a sequence of points $\\{y_{r, t}^i\\}_{r = 0}^{n_i - 1}$ for both $i \\in \\{a, b\\}$. \n\nUsing a combination of~(\\ref{Mnabla}) and Assumption~\\ref{ass:gradient}~(b), the bound for the second moment of the stochastic gradients is given by\n\\begin{equation} \\label{SGbound}\n\\mathbb{E}_{\\xi_t}[\\|g^i(y^i_{r, t}, \\xi_t^r)\\|^2] \\; \\leq \\; G + \\bar{G}M_{\\nabla}, \\quad\n\\mathbb{E}_{\\xi_t}[\\|g^i(y^i_{r, t}, \\xi_t^r)\\|] \\; \\leq \\; \\sqrt{G + \\bar{G}M_{\\nabla}},\n\\end{equation} \nwhere the second bound results from applying Jensen's inequality to the first one.\nHence, the second term in the right-hand side of~\\eqref{theorem2_eq1} can be bounded by\n\\begin{equation*}\n\\begin{split}\n   & \\mathbb{E}_{\\xi_t}[\\|\\sum_{r = 0}^{n_a - 1} g^a(y^a_{r, t}, \\xi^r_t) + \\sum_{r = 0}^{n_b - 1}  g^b(y^b_{r, t}, \\xi^{n_a + r}_t)\\|^2] \\\\\n   &  \\;\\leq\\; 2\\mathbb{E}_{\\xi_t}[\\|\\sum_{r = 0}^{n_a - 1}  g^a(y^a_{r, t}, \\xi^r_t)\\|^2] + 2\\mathbb{E}_{\\xi_t}[\\|\\sum_{r = 0}^{n_b - 1}  g^b(y^b_{r, t}, \\xi^{n_a+r}_t)\\|^2] \\\\\n& \\;\\leq\\;  2n_a\\sum_{r = 0}^{n_a - 1} \\mathbb{E}_{\\xi_t}[\\| g^a(y^a_{r, t}, \\xi^r_t)\\|^2] + 2n_b\\sum_{r = 0}^{n_b - 1} \\mathbb{E}_{\\xi_t}[\\| g^b(y^b_{r, t}, \\xi^{n_a + r}_t)\\|^2] \\\\\n& \\;\\leq\\;  2(n_a + n_b)^2(G + \\bar{G}M_{\\nabla}).\n\\end{split}\n\\end{equation*}\nWe thus arrive at\n\\begin{equation}\n    \\label{theorem2_eq2}\n    \\begin{split}\n    \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2]\n    \\;\\leq\\; &  \\|x_t - x_*\\|^2 + 2\\alpha_t^2(n_a + n_b)^2(G + \\bar{G}M_{\\nabla}) \\\\\n     & - 2\\alpha_t(x_t - x_*)^\\top \\mathbb{E}_{\\xi_t}[n_a \\nabla f^a(w_t^a) + n_b \\nabla f^b(w_t^b)].\n    \\end{split}\n\\end{equation}\n\nBy adding and subtracting $2\\alpha_t(x_t - x_*)^\\top(n_a\\nabla f^a(x_t) + n_b \\nabla f^b(x_t))$ in the right-hand side of~\\eqref{theorem2_eq2}, we further rewrite it as\n\\begin{equation}\n\\label{theorem2_eq22}\n    \\begin{split}\n    \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2]\n    \\;\\leq\\; &  \\|x_t - x_*\\|^2 + 2\\alpha_t^2(n_a + n_b)^2(G + \\bar{G}M_{\\nabla}) \\\\\n     & - 2\\alpha_t(x_t - x_*)^\\top (n_a \\nabla f^a(x_t) + n_b \\nabla f^b(x_t)) \\\\\n     & + 2\\alpha_t\\|x_t - x_*\\|\\mathbb{E}_{\\xi_t}[\\|n_a \\nabla f^a(w_t^a) - n_a \\nabla f^a(x_t)\\|] \\\\\n     & + 2\\alpha_t\\|x_t - x_*\\|\\mathbb{E}_{\\xi_t}[\\|n_b \\nabla f^b(w^b_t) - n_b \\nabla f^b(x_t)\\|]. \n    \\end{split}\n\\end{equation}\nNote that the last two terms are derived by the Cauchy–Schwarz and Jensen's inequalities.\n\n\\vspace{2ex}\n\\noindent {\\bf Part II: Using strong convexity.} \nSelecting $\\lambda_* = \\lambda(n_a, n_b) = n_a/(n_a + n_b)$, by the strong convexity of the weighted-sum function, one has\n\\begin{equation*}\n \\nabla_x S(x_t, \\lambda_*)^\\top(x_t - x_*)  \\;\\geq\\; S(x_t, \\lambda_*) - S(x_*, \\lambda_*) + \\frac{c}{2}\\| x_t - x_*\\|^2,\n\\end{equation*}\nwhich is equivalent to\n\\begin{equation}\n\\label{theorm2_eq3}\n (x_t - x_*)^\\top(n_a \\nabla f^a(x_t) + n_b \\nabla f^b(x_t))  \\;\\geq\\; (n_a + n_b)(S(x_t, \\lambda_*) - S(x_*, \\lambda_*) + \\frac{c}{2}\\| x_t - x_*\\|^2).\n\\end{equation}\n\nFrom Assumption~\\ref{ass:Lipschitz}, we obtain a bound for the last two terms of~\\eqref{theorem2_eq22} in the form \n\\begin{equation}\n    \\label{theorem2_eq4}\n    2\\alpha_t\\|x_t - x_*\\|\\mathbb{E}_{\\xi_t}[\\|n_i \\nabla f^i(w_t^i) - n_i \\nabla f^i(x_t)\\|] \\;\\leq\\;  2\\alpha_tLn_i\\|x_t - x_*\\|\\mathbb{E}_{\\xi_t}[\\|x_t - w_t^i\\|], \\forall i \\in \\{a, b\\}.\n\\end{equation}\nAccording to Proposition~\\ref{prop_ivt}, $w_t^i$ is a convex linear combination of a sequence of points $\\{y_{r, t}^i\\}_{r = 0}^{n_i -1}$ for $i \\in \\{a, b\\}$. One can write $w_t^i = \\sum_{r=0}^{n_i - 1} \\beta_r y_{r, t}^i$ with $\\beta_r \\geq 0$, $r = 0, \\ldots, n_i - 1$ , and $\\sum_{r=0}^{n_i - 1} \\beta_r = 1$. An explicit upper bound of $\\|x_t - w_t^i\\|$ can then be derived as follows\n\\begin{equation}\n\\label{part2_eq2}\n    \\begin{split} \n        \\|x_t - \\sum_{r=0}^{n_i - 1} \\beta_r y_{r, t}^i\\| & \\;=\\; \\|\\sum_{r=0}^{n_i - 1} \\beta_r (x_t - y_{r, t}^i)\\| \\;\\leq\\; \\sum_{r=0}^{n_i - 1} \\beta_r \\|x_t - y_{r, t}^i\\|.\n    \\end{split}\n\\end{equation}\nUsing $y_{r, t}^i = y_{0, t}^i - \\sum_{j = 0}^{r-1}\\alpha_t g^i(y_{j, t}^i, \\xi_{t}^j)$ and applying the triangle inequality, we have\n\\begin{align}\n    \\|x_t - y_{r, t}^a\\| \\;\\leq\\; \\alpha_t \\sum_{j = 0}^{r-1} \\|g^a(y^a_{j, t}, \\xi_{t}^j)\\|,\\label{part2_eq3}\n\\end{align}  \nand\n\\begin{equation}\n\\label{part2_eq4}\n\\begin{split}\n    \\|x_t - y_{r, t}^b\\| & \\;=\\; \\|x_t - y_{n_a, t}^a + \\sum_{j = 0}^{r-1}\\alpha_t g^b(y_{j, t}^b, \\xi_{t}^{n_a + j})\\| \\\\\n    & \\;\\leq\\; \\alpha_t \\sum_{j = 0}^{n_a-1} \\|g^a(y^a_{j, t}, \\xi_{t}^j)\\| +  \\alpha_t \\sum_{j = 0}^{r-1} \\|g^b(y^b_{j, t}, \\xi_{t}^{n_a + j})\\|.\n\\end{split}\n\\end{equation}\nPlugging~\\eqref{part2_eq3} into~\\eqref{part2_eq2} with $i = a$ results in (note that $y^a_{0,t}=x_t$)\n\\begin{equation}\\label{part2_eq5}\n\\begin{split}\n       \\|x_t - w_t^a\\| & \\;\\leq\\; \\alpha_t \\sum_{r=1}^{n_a - 1} \\beta_r \\sum_{j = 0}^{r-1} \\|g^a(y^a_{j, t}, \\xi_{t}^j)\\| \\\\ & \\;  = \\;  \\alpha_t \n       \\sum_{j=0}^{n_a - 1} \\|g^a(y^a_{j, t}, \\xi_{t}^j)\\|\n      \\sum_{r=j+1}^{n_a-1} \\beta_r\n       \\;\\leq \\; \\alpha_t  \\sum_{j=0}^{n_a - 1} \\|g^a(y^a_{j,t}, \\xi_{t}^{j})\\|.\n\\end{split}\n\\end{equation}\n\n\nApplying expectations on both sides of~\\eqref{part2_eq5}, followed by the bound~(\\ref{SGbound}) on the second moment of the stochastic gradients,\n\\begin{equation} \\label{part2_eq6}\n\\mathbb{E}_{\\xi_t}[\\|x_t - w_t^a\\|] \\; \\leq \\; \\alpha_t n_a\\sqrt{G + \\bar{G}M_{\\nabla}}.\n\\end{equation}\nSimilarly, merging~\\eqref{part2_eq4} into~\\eqref{part2_eq2} with $i = b$ leads to \n\\begin{equation}\n\\label{part2_eq6.5}\n        \\|x_t - w_t^b\\| \\;\\leq \\; \\alpha_t \\sum_{j=0}^{n_a - 1} \\|g^a(y^a_{j,t}, \\xi_{t}^{j})\\| + \\alpha_t  \\sum_{j=0}^{n_b - 1} \\|g^b(y^b_{j,t}, \\xi_{t}^{n_a+j})\\|,\n\\end{equation}\nand applying expectations\n\\begin{equation}\n\\label{part2_eq7}\n        \\mathbb{E}_{\\xi_t}[\\|x_t - w_t^b\\|] \\;\\leq \\; \\alpha_t (n_a+n_b) \\sqrt{G + \\bar{G}M_{\\nabla}}.\n\\end{equation}\nFinally, combining~\\eqref{theorem2_eq4}, \\eqref{part2_eq6}, and~\\eqref{part2_eq7} yields\n\\begin{equation}\n\\label{part2_eq8}\n    \\begin{split}\n        2\\alpha_t\\|x_t - x_*\\|\\sum_{i \\in \\{a, b\\}} \\mathbb{E}_{\\xi_t}[\\|n_i \\nabla f^i(w_t^i) - n_i \\nabla f^i(x_t)\\|] \\;\\leq\\; 2\\alpha_t^2L\\Theta (n_a + n_b)^2\\sqrt{G + \\bar{G}M_{\\nabla}}.\n    \\end{split}\n\\end{equation}\n\n\\vspace{2ex}\n\\noindent {\\bf Part III: Bound on the optimality gap in terms of weighted-sum function.}\nApplying inequalities \\eqref{theorm2_eq3} and \\eqref{part2_eq8} to \\eqref{theorem2_eq22} leads to \n\\begin{equation}\n\\label{smooth_sc_part3_eq1}\n    \\begin{split}\n     \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2]  \\;\\leq\\; &  (1-\\alpha_t(n_a + n_b)c) \\|x_t - x_*\\|^2 -  2\\alpha_t(n_a + n_b)(S(x_t, \\lambda_*) - S(x_*, \\lambda_*)) \\\\\n     & + \\alpha^2_t M,\n    \\end{split}\n\\end{equation}\nwhere we let $M = 2(n_a + n_b)^2 (G + \\bar{G}M_{\\nabla} +  L\\Theta\\sqrt{G + \\bar{G}M_{\\nabla}})$. Plugging in $\\alpha_t = \\frac{2}{c(n_a+n_b)(t+1)}$ and rearranging the last inequality result in \n\\begin{equation*}\n    \\begin{split}\n        S(x_t, \\lambda_*) - S(x_*, \\lambda_*)  \\;\\leq\\; & \\frac{(1-\\alpha_t(n_a + n_b)c) \\|x_t - x_*\\|^2 - \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2] + \\alpha^2_t M}{2\\alpha_t(n_a + n_b)} \\\\\n        \\;\\leq\\; & \\frac{c(t-1)}{4}\\|x_t - x_*\\|^2  - \\frac{c(t+1)}{4} \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2] + \\frac{\\tilde{M}}{c(t+1)},\n    \\end{split}\n\\end{equation*}\nwhere $\\tilde{M} = \\frac{M}{(n_a + n_b)^2}$. By taking total expectation over $\\{\\xi_t\\}$, multiplying both sides by $t$, and summing over $t = 1, \\ldots, T$, one obtains\n\\begin{equation*}\n    \\begin{split}\n       \\sum_{t=1}^T \n       t(\\mathbb{E}[S(x_t, \\lambda_*)] - S(x_*, \\lambda_*)) \\;\\leq\\; & \\sum_{t=1}^T \\left ( \\frac{ct(t-1)}{4}\\mathbb{E}[\\|x_t - x_*\\|^2]  - \\frac{ct(t+1)}{4} \\mathbb{E}[\\|x_{t+1} - x_*\\|^2]\\right) \\\\\n       & + \\sum_{t=1}^T \\frac{\\tilde{M}t}{c(t+1)} \\\\\n       \\;\\leq\\; & -\\frac{cT(T+1)}{4} \\mathbb{E}[\\|x_{T+1} - x_*\\|^2] + \\sum_{t=1}^T \\frac{\\tilde{M}t}{c(t+1)}  \\;\\leq\\; \\frac{T}{c} \\tilde{M}. \n    \\end{split}\n\\end{equation*}\nDividing both sides of the last inequality by $\\sum_{t=1}^T t$ yields\n\\begin{equation*}\n    \\min_{t = 1, \\ldots,T} \\mathbb{E}[S(x_t, \\lambda_*)] - S(x_*, \\lambda_*) \\;\\leq\\; \\frac{2}{c(T + 1)} \\tilde{M},   \n\\end{equation*}\nwhich concludes the proof.\n\\end{proof}","sha256_utf8":"1519b9880459bcb39bd3aa172bf0be7cd83e801252685a5b13daa51b6cd8f14f"}]},{"id":"S011","semantic_type":"proposition","kind":"proposition","title":"函数均值的凸包实现","source_units":[{"start":64237,"end":64654,"start_line":824,"end_line":828,"raw_tex":"\\begin{proposition} \\label{prop_ivt}\nGiven a continuous real function $\\phi(x): \\mathbb{R}^n \\to \\mathbb{R}$ and a set of points $\\{x_j\\}_{j =1}^m$, there exists $w \\in \\mathbb{R}^n$ such that \n\\[m\\phi(w) = \\sum_{j = 1}^m \\phi(x_j),\\]\nwhere~$w = \\sum_{j = 1}^m \\mu_j x_j,\\mbox{ with }\\sum_{j = 1}^m \\mu_j = 1, \\mu_j \\geq 0$, $i = 1, \\ldots, m$, is a convex linear combination of $\\{x_j\\}_{j =1}^m$. \n\\end{proposition}","sha256_utf8":"e43baba5fee93b6d52b8dc31ee176d85289368e09a8fd45d8982733ccb4987f7"}]},{"id":"S012","semantic_type":"proof","kind":"proof","title":"凸包均值命题完整证明","source_units":[{"start":64656,"end":65532,"start_line":830,"end_line":832,"raw_tex":"\\begin{proof}\nThe proposition is obtained by consecutively applying the Intermediate Value Theorem to $\\phi(x)$. First, for the pair of points $x_1$ and $x_2$, there exists a point $w_{12} = \\mu_{12} x_1 + (1-\\mu_{12}) x_2, \\mu_{12} \\in [0, 1]$, such that~$\\phi(w_{12}) = (\\phi(x_1) + \\phi(x_2))/2$ according to the Intermediate Value Theorem, which implies that $\\sum_{j = 1}^m \\phi(x_j) = 2\\phi(w_{12}) + \\sum_{j = 3}^m \\phi(x_j)$.  Then, there exists $w_{13} = \\mu_{13} w_{12} + (1-\\mu_{13}) x_3, \\mu_{13} \\geq 0$, such that $\\phi(w_{13}) = (2\\phi(w_{12}) + \\phi(x_3))/3$ holds given that the average function value $(2\\phi(w_{12}) + \\phi(x_3))/3$ lies between $\\phi(w_{12})$ and $\\phi(x_3)$. Notice that~$w_{13}$ can also be written as convex linear combination of~$\\{x_1, x_2, x_3\\}$. The proof is concluded by continuing this process until $x_m$ is reached. \n\\end{proof}","sha256_utf8":"8153cb1c16f203d799ccb6daa0a8e1f24bcc88606b8faf4656a2d6f4d5f2c39d"}]},{"id":"S013","semantic_type":"proof_step","kind":"proof_step","title":"逐次应用中间值定理","source_units":[{"start":64656,"end":65532,"start_line":830,"end_line":832,"raw_tex":"\\begin{proof}\nThe proposition is obtained by consecutively applying the Intermediate Value Theorem to $\\phi(x)$. First, for the pair of points $x_1$ and $x_2$, there exists a point $w_{12} = \\mu_{12} x_1 + (1-\\mu_{12}) x_2, \\mu_{12} \\in [0, 1]$, such that~$\\phi(w_{12}) = (\\phi(x_1) + \\phi(x_2))/2$ according to the Intermediate Value Theorem, which implies that $\\sum_{j = 1}^m \\phi(x_j) = 2\\phi(w_{12}) + \\sum_{j = 3}^m \\phi(x_j)$.  Then, there exists $w_{13} = \\mu_{13} w_{12} + (1-\\mu_{13}) x_3, \\mu_{13} \\geq 0$, such that $\\phi(w_{13}) = (2\\phi(w_{12}) + \\phi(x_3))/3$ holds given that the average function value $(2\\phi(w_{12}) + \\phi(x_3))/3$ lies between $\\phi(w_{12})$ and $\\phi(x_3)$. Notice that~$w_{13}$ can also be written as convex linear combination of~$\\{x_1, x_2, x_3\\}$. The proof is concluded by continuing this process until $x_m$ is reached. \n\\end{proof}","sha256_utf8":"8153cb1c16f203d799ccb6daa0a8e1f24bcc88606b8faf4656a2d6f4d5f2c39d"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"证明分段与中间点记号","source_units":[{"start":25369,"end":25834,"start_line":311,"end_line":312,"raw_tex":"\\begin{proof}\nThe proof is divided in three parts for better organization and understanding. In the first part, one obtains an upper bound on the norm of the iterates, $\\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2]$. Strong convexity of the weighted-sum function is applied in the second part. The third part concludes the proof using standard arguments. For simplicity, we let $z_t = y^b_{0, t} = y^a_{n_a, t}$ be the intermediate point at each iteration.  \\vspace{1ex}\n","sha256_utf8":"65534ca27010609af482df3bc256b43bba2d6c7c4dfe11794ed7e73630c0bff7"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"展开一次交替更新","source_units":[{"start":25835,"end":26444,"start_line":314,"end_line":320,"raw_tex":"\\noindent {\\bf Part I: Bound on the iterates error using the Intermediate Value Theorem.} \nAt any iteration $t$, the sequence of stochastic gradients is computed from drawing the sequence of random variables $\\xi_t = \\{\\xi^0_t, \\ldots, \\xi^{n_a + n_b - 1}_t \\}$. We have\n\\begin{align*}\n    x_{t+1} - x_* & \\;=\\; P_{\\mathcal{X}}\\left(z_t - \\alpha_t \\sum_{r = 0}^{n_b - 1} g^b(y^b_{r, t}, \\xi^r_t)\\right) - x_*, \\\\\n    &\\;=\\;P_{\\mathcal{X}}\\left(\n    x_t - \\alpha_t \\sum_{r = 0}^{n_a - 1}  g^a(y^a_{r, t}, \\xi^r_t) - \\alpha_t \\sum_{r = 0}^{n_b - 1}  g^b(y^b_{r, t}, \\xi^{n_a + r}_t) \\right) - x_*.\n\\end{align*}\n","sha256_utf8":"115d0f7cd69c6dbc4ee00412df1f01876bb60f698c56fa9995da0c9dd423facd"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"无偏性与投影误差递推","source_units":[{"start":26445,"end":27715,"start_line":322,"end_line":334,"raw_tex":"Since the sequence $\\xi_t$ is drawn independently, using Assumption~\\ref{ass:gradient} (a) one has\n\\[\\mathbb{E}_{\\xi_t}[g^a(y_{r, t}^a, \\xi_{t}^r)] \\;=\\; \\mathbb{E}_{\\xi_t^0, \\ldots, \\xi_{t}^{r-1}}[\\mathbb{E}_{\\xi_t^r}[g^a(y_{r, t}^a, \\xi_{t}^r)]] \\;=\\; \\mathbb{E}_{\\xi_t^0, \\ldots, \\xi_{t}^{r-1}}[\\nabla f^a(y_{r, t}^a)] \\;=\\; \\mathbb{E}_{\\xi_t}[\\nabla f^a(y_{r, t}^a)],\\]\nwhere the last equality holds due to the independence between $y_{r, t}^a$ and $\\{\\xi_t^{r}, \\ldots, \\xi_t^{n_a + n_b - 1}\\}$. Similarly, we have $\\mathbb{E}_{\\xi_t}[g^b(y_{r, t}^b, \\xi_{t}^{n_a + r})] = \\mathbb{E}_{\\xi_t}[\\nabla f^b(y_{r, t}^b)]$.\nThen, taking square norms and expectations over the random variables $\\xi_t$ on both sides yields\n\\begin{equation}\n\\label{theorem2_eq1}\n    \\begin{split}\n    \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2]  \n     \\;\\leq\\; & \\|x_t - x_*\\|^2 + \\alpha_t^2\\mathbb{E}_{\\xi_t}[\\|\\sum_{r = 0}^{n_a - 1} g^a(y^a_{r, t}, \\xi^r_t) + \\sum_{r = 0}^{n_b - 1} g^b(y^b_{r, t}, \\xi^{n_a + r}_t)\\|^2] \\\\\n     & - \\mathbb{E}_{\\xi_t}[2\\alpha_t(x_t - x_*)^\\top (\\sum_{r = 0}^{n_a - 1} \\nabla f^a(y^a_{r, t}) + \\sum_{r = 0}^{n_b - 1} \\nabla f^b(y^b_{r, t}))],\n    \\end{split}\n\\end{equation}\nwhich holds by the non-expansiveness property of the orthogonal projection operator.\n","sha256_utf8":"bad1dbaa891a8f3950793697c57d515b831929cecbb8dd28b03784bd336ebd75"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"中间值定理聚合梯度内积","source_units":[{"start":27716,"end":28321,"start_line":336,"end_line":336,"raw_tex":"We now claim, by applying a version of the Intermediate Value Theorem given in Proposition~\\ref{prop_ivt}, that the last term of the right-hand side in~\\eqref{theorem2_eq1} can be written as $- 2\\alpha_t(x_t - x_*)^\\top \\mathbb{E}_{\\xi_t}[n_a \\nabla f^a(w_t^a) + n_b \\nabla f^b(w_t^b)]$ for some $w_t^a$ and $w_t^b$. In fact, we apply Proposition~\\ref{prop_ivt} to the real continuous function $\\phi^i(y) = -2\\alpha_t(x_t - x_*)^\\top \\nabla f^i(y)$, from which we then know that $w_t^i$ is a convex linear combination of a sequence of points $\\{y_{r, t}^i\\}_{r = 0}^{n_i - 1}$ for both $i \\in \\{a, b\\}$. \n","sha256_utf8":"3389da98d077768eb8796ff430881dbdb483f6448d221ef1405a6ea0eb1ccfb5"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"随机梯度一二阶矩界","source_units":[{"start":28322,"end":28784,"start_line":338,"end_line":343,"raw_tex":"Using a combination of~(\\ref{Mnabla}) and Assumption~\\ref{ass:gradient}~(b), the bound for the second moment of the stochastic gradients is given by\n\\begin{equation} \\label{SGbound}\n\\mathbb{E}_{\\xi_t}[\\|g^i(y^i_{r, t}, \\xi_t^r)\\|^2] \\; \\leq \\; G + \\bar{G}M_{\\nabla}, \\quad\n\\mathbb{E}_{\\xi_t}[\\|g^i(y^i_{r, t}, \\xi_t^r)\\|] \\; \\leq \\; \\sqrt{G + \\bar{G}M_{\\nabla}},\n\\end{equation} \nwhere the second bound results from applying Jensen's inequality to the first one.\n","sha256_utf8":"ab0108ee064ff8b8a873bc55043b2db2b91991cc648cbf285e4fed11b367581f"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"梯度和矩界及聚合递推","source_units":[{"start":28784,"end":29825,"start_line":344,"end_line":361,"raw_tex":"Hence, the second term in the right-hand side of~\\eqref{theorem2_eq1} can be bounded by\n\\begin{equation*}\n\\begin{split}\n   & \\mathbb{E}_{\\xi_t}[\\|\\sum_{r = 0}^{n_a - 1} g^a(y^a_{r, t}, \\xi^r_t) + \\sum_{r = 0}^{n_b - 1}  g^b(y^b_{r, t}, \\xi^{n_a + r}_t)\\|^2] \\\\\n   &  \\;\\leq\\; 2\\mathbb{E}_{\\xi_t}[\\|\\sum_{r = 0}^{n_a - 1}  g^a(y^a_{r, t}, \\xi^r_t)\\|^2] + 2\\mathbb{E}_{\\xi_t}[\\|\\sum_{r = 0}^{n_b - 1}  g^b(y^b_{r, t}, \\xi^{n_a+r}_t)\\|^2] \\\\\n& \\;\\leq\\;  2n_a\\sum_{r = 0}^{n_a - 1} \\mathbb{E}_{\\xi_t}[\\| g^a(y^a_{r, t}, \\xi^r_t)\\|^2] + 2n_b\\sum_{r = 0}^{n_b - 1} \\mathbb{E}_{\\xi_t}[\\| g^b(y^b_{r, t}, \\xi^{n_a + r}_t)\\|^2] \\\\\n& \\;\\leq\\;  2(n_a + n_b)^2(G + \\bar{G}M_{\\nabla}).\n\\end{split}\n\\end{equation*}\nWe thus arrive at\n\\begin{equation}\n    \\label{theorem2_eq2}\n    \\begin{split}\n    \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2]\n    \\;\\leq\\; &  \\|x_t - x_*\\|^2 + 2\\alpha_t^2(n_a + n_b)^2(G + \\bar{G}M_{\\nabla}) \\\\\n     & - 2\\alpha_t(x_t - x_*)^\\top \\mathbb{E}_{\\xi_t}[n_a \\nabla f^a(w_t^a) + n_b \\nabla f^b(w_t^b)].\n    \\end{split}\n\\end{equation}\n","sha256_utf8":"dd07fedd5f8346570cafe25d17cf60f411fd906fb740a372b0d2d3f6f252a393"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"分离当前梯度与聚合偏差","source_units":[{"start":29826,"end":30587,"start_line":363,"end_line":374,"raw_tex":"By adding and subtracting $2\\alpha_t(x_t - x_*)^\\top(n_a\\nabla f^a(x_t) + n_b \\nabla f^b(x_t))$ in the right-hand side of~\\eqref{theorem2_eq2}, we further rewrite it as\n\\begin{equation}\n\\label{theorem2_eq22}\n    \\begin{split}\n    \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2]\n    \\;\\leq\\; &  \\|x_t - x_*\\|^2 + 2\\alpha_t^2(n_a + n_b)^2(G + \\bar{G}M_{\\nabla}) \\\\\n     & - 2\\alpha_t(x_t - x_*)^\\top (n_a \\nabla f^a(x_t) + n_b \\nabla f^b(x_t)) \\\\\n     & + 2\\alpha_t\\|x_t - x_*\\|\\mathbb{E}_{\\xi_t}[\\|n_a \\nabla f^a(w_t^a) - n_a \\nabla f^a(x_t)\\|] \\\\\n     & + 2\\alpha_t\\|x_t - x_*\\|\\mathbb{E}_{\\xi_t}[\\|n_b \\nabla f^b(w^b_t) - n_b \\nabla f^b(x_t)\\|]. \n    \\end{split}\n\\end{equation}\nNote that the last two terms are derived by the Cauchy–Schwarz and Jensen's inequalities.\n","sha256_utf8":"5f4b30b87c0366a3bd92e68ee5a296c849404665667305fb5c0179d12672cb99"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"强凸性给出目标差下界","source_units":[{"start":30588,"end":31159,"start_line":376,"end_line":386,"raw_tex":"\\vspace{2ex}\n\\noindent {\\bf Part II: Using strong convexity.} \nSelecting $\\lambda_* = \\lambda(n_a, n_b) = n_a/(n_a + n_b)$, by the strong convexity of the weighted-sum function, one has\n\\begin{equation*}\n \\nabla_x S(x_t, \\lambda_*)^\\top(x_t - x_*)  \\;\\geq\\; S(x_t, \\lambda_*) - S(x_*, \\lambda_*) + \\frac{c}{2}\\| x_t - x_*\\|^2,\n\\end{equation*}\nwhich is equivalent to\n\\begin{equation}\n\\label{theorm2_eq3}\n (x_t - x_*)^\\top(n_a \\nabla f^a(x_t) + n_b \\nabla f^b(x_t))  \\;\\geq\\; (n_a + n_b)(S(x_t, \\lambda_*) - S(x_*, \\lambda_*) + \\frac{c}{2}\\| x_t - x_*\\|^2).\n\\end{equation}\n","sha256_utf8":"de4adb3c3913082c234b3ee24c0feefc8984331492ae950351cc10c6049406b1"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"梯度偏差转化为点距离","source_units":[{"start":31160,"end":31524,"start_line":388,"end_line":392,"raw_tex":"From Assumption~\\ref{ass:Lipschitz}, we obtain a bound for the last two terms of~\\eqref{theorem2_eq22} in the form \n\\begin{equation}\n    \\label{theorem2_eq4}\n    2\\alpha_t\\|x_t - x_*\\|\\mathbb{E}_{\\xi_t}[\\|n_i \\nabla f^i(w_t^i) - n_i \\nabla f^i(x_t)\\|] \\;\\leq\\;  2\\alpha_tLn_i\\|x_t - x_*\\|\\mathbb{E}_{\\xi_t}[\\|x_t - w_t^i\\|], \\forall i \\in \\{a, b\\}.\n\\end{equation}\n","sha256_utf8":"07c8637f648b086067b9e142fdd60a1371d5b11462a7318281bde1483fb214f6"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"凸包表示控制聚合点距离","source_units":[{"start":31524,"end":32177,"start_line":393,"end_line":399,"raw_tex":"According to Proposition~\\ref{prop_ivt}, $w_t^i$ is a convex linear combination of a sequence of points $\\{y_{r, t}^i\\}_{r = 0}^{n_i -1}$ for $i \\in \\{a, b\\}$. One can write $w_t^i = \\sum_{r=0}^{n_i - 1} \\beta_r y_{r, t}^i$ with $\\beta_r \\geq 0$, $r = 0, \\ldots, n_i - 1$ , and $\\sum_{r=0}^{n_i - 1} \\beta_r = 1$. An explicit upper bound of $\\|x_t - w_t^i\\|$ can then be derived as follows\n\\begin{equation}\n\\label{part2_eq2}\n    \\begin{split} \n        \\|x_t - \\sum_{r=0}^{n_i - 1} \\beta_r y_{r, t}^i\\| & \\;=\\; \\|\\sum_{r=0}^{n_i - 1} \\beta_r (x_t - y_{r, t}^i)\\| \\;\\leq\\; \\sum_{r=0}^{n_i - 1} \\beta_r \\|x_t - y_{r, t}^i\\|.\n    \\end{split}\n\\end{equation}\n","sha256_utf8":"ed46a6cdb19063e3f390e8996c999cebf99f371b9aa3323942fadb1891c6c313"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"第一阶段中间迭代距离","source_units":[{"start":32177,"end":32450,"start_line":400,"end_line":403,"raw_tex":"Using $y_{r, t}^i = y_{0, t}^i - \\sum_{j = 0}^{r-1}\\alpha_t g^i(y_{j, t}^i, \\xi_{t}^j)$ and applying the triangle inequality, we have\n\\begin{align}\n    \\|x_t - y_{r, t}^a\\| \\;\\leq\\; \\alpha_t \\sum_{j = 0}^{r-1} \\|g^a(y^a_{j, t}, \\xi_{t}^j)\\|,\\label{part2_eq3}\n\\end{align}  \n","sha256_utf8":"67d867921aec4002879ecef7ea581c81e516029514125bea6568338842dbc30e"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"第二阶段中间迭代距离","source_units":[{"start":32450,"end":32800,"start_line":404,"end_line":411,"raw_tex":"and\n\\begin{equation}\n\\label{part2_eq4}\n\\begin{split}\n    \\|x_t - y_{r, t}^b\\| & \\;=\\; \\|x_t - y_{n_a, t}^a + \\sum_{j = 0}^{r-1}\\alpha_t g^b(y_{j, t}^b, \\xi_{t}^{n_a + j})\\| \\\\\n    & \\;\\leq\\; \\alpha_t \\sum_{j = 0}^{n_a-1} \\|g^a(y^a_{j, t}, \\xi_{t}^j)\\| +  \\alpha_t \\sum_{j = 0}^{r-1} \\|g^b(y^b_{j, t}, \\xi_{t}^{n_a + j})\\|.\n\\end{split}\n\\end{equation}\n","sha256_utf8":"b9ddd5086cea4ffa49f001db43c731e9706f3609b612fa8f7728881b4acabb50"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"第一聚合点的路径距离界","source_units":[{"start":32800,"end":33299,"start_line":412,"end_line":420,"raw_tex":"Plugging~\\eqref{part2_eq3} into~\\eqref{part2_eq2} with $i = a$ results in (note that $y^a_{0,t}=x_t$)\n\\begin{equation}\\label{part2_eq5}\n\\begin{split}\n       \\|x_t - w_t^a\\| & \\;\\leq\\; \\alpha_t \\sum_{r=1}^{n_a - 1} \\beta_r \\sum_{j = 0}^{r-1} \\|g^a(y^a_{j, t}, \\xi_{t}^j)\\| \\\\ & \\;  = \\;  \\alpha_t \n       \\sum_{j=0}^{n_a - 1} \\|g^a(y^a_{j, t}, \\xi_{t}^j)\\|\n      \\sum_{r=j+1}^{n_a-1} \\beta_r\n       \\;\\leq \\; \\alpha_t  \\sum_{j=0}^{n_a - 1} \\|g^a(y^a_{j,t}, \\xi_{t}^{j})\\|.\n\\end{split}\n\\end{equation}\n","sha256_utf8":"23b5db35d23c461bfdb3f6de35fef56a3d058a1046c19d3db1ffdb5ef028f164"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"第一聚合点的期望距离界","source_units":[{"start":33301,"end":33586,"start_line":423,"end_line":426,"raw_tex":"Applying expectations on both sides of~\\eqref{part2_eq5}, followed by the bound~(\\ref{SGbound}) on the second moment of the stochastic gradients,\n\\begin{equation} \\label{part2_eq6}\n\\mathbb{E}_{\\xi_t}[\\|x_t - w_t^a\\|] \\; \\leq \\; \\alpha_t n_a\\sqrt{G + \\bar{G}M_{\\nabla}}.\n\\end{equation}\n","sha256_utf8":"664d9ff475853605f32b8ed42b7bd6e88714154639131d01d4f6ee010be6d570"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"第二聚合点的距离与期望界","source_units":[{"start":33586,"end":34066,"start_line":427,"end_line":436,"raw_tex":"Similarly, merging~\\eqref{part2_eq4} into~\\eqref{part2_eq2} with $i = b$ leads to \n\\begin{equation}\n\\label{part2_eq6.5}\n        \\|x_t - w_t^b\\| \\;\\leq \\; \\alpha_t \\sum_{j=0}^{n_a - 1} \\|g^a(y^a_{j,t}, \\xi_{t}^{j})\\| + \\alpha_t  \\sum_{j=0}^{n_b - 1} \\|g^b(y^b_{j,t}, \\xi_{t}^{n_a+j})\\|,\n\\end{equation}\nand applying expectations\n\\begin{equation}\n\\label{part2_eq7}\n        \\mathbb{E}_{\\xi_t}[\\|x_t - w_t^b\\|] \\;\\leq \\; \\alpha_t (n_a+n_b) \\sqrt{G + \\bar{G}M_{\\nabla}}.\n\\end{equation}\n","sha256_utf8":"418be6c3f1375c0b4d7e44e638c25af7cd18dbec4fe4380041ccf8784c6cf508"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"聚合偏差的二阶步长界","source_units":[{"start":34066,"end":34430,"start_line":437,"end_line":443,"raw_tex":"Finally, combining~\\eqref{theorem2_eq4}, \\eqref{part2_eq6}, and~\\eqref{part2_eq7} yields\n\\begin{equation}\n\\label{part2_eq8}\n    \\begin{split}\n        2\\alpha_t\\|x_t - x_*\\|\\sum_{i \\in \\{a, b\\}} \\mathbb{E}_{\\xi_t}[\\|n_i \\nabla f^i(w_t^i) - n_i \\nabla f^i(x_t)\\|] \\;\\leq\\; 2\\alpha_t^2L\\Theta (n_a + n_b)^2\\sqrt{G + \\bar{G}M_{\\nabla}}.\n    \\end{split}\n\\end{equation}\n","sha256_utf8":"04e98a74c08bc91588b4507731d491816f6c131396d319a52132a299cb2ee3d1"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"合并递推并代入递减步长","source_units":[{"start":34431,"end":35505,"start_line":445,"end_line":461,"raw_tex":"\\vspace{2ex}\n\\noindent {\\bf Part III: Bound on the optimality gap in terms of weighted-sum function.}\nApplying inequalities \\eqref{theorm2_eq3} and \\eqref{part2_eq8} to \\eqref{theorem2_eq22} leads to \n\\begin{equation}\n\\label{smooth_sc_part3_eq1}\n    \\begin{split}\n     \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2]  \\;\\leq\\; &  (1-\\alpha_t(n_a + n_b)c) \\|x_t - x_*\\|^2 -  2\\alpha_t(n_a + n_b)(S(x_t, \\lambda_*) - S(x_*, \\lambda_*)) \\\\\n     & + \\alpha^2_t M,\n    \\end{split}\n\\end{equation}\nwhere we let $M = 2(n_a + n_b)^2 (G + \\bar{G}M_{\\nabla} +  L\\Theta\\sqrt{G + \\bar{G}M_{\\nabla}})$. Plugging in $\\alpha_t = \\frac{2}{c(n_a+n_b)(t+1)}$ and rearranging the last inequality result in \n\\begin{equation*}\n    \\begin{split}\n        S(x_t, \\lambda_*) - S(x_*, \\lambda_*)  \\;\\leq\\; & \\frac{(1-\\alpha_t(n_a + n_b)c) \\|x_t - x_*\\|^2 - \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2] + \\alpha^2_t M}{2\\alpha_t(n_a + n_b)} \\\\\n        \\;\\leq\\; & \\frac{c(t-1)}{4}\\|x_t - x_*\\|^2  - \\frac{c(t+1)}{4} \\mathbb{E}_{\\xi_t}[\\|x_{t+1} - x_*\\|^2] + \\frac{\\tilde{M}}{c(t+1)},\n    \\end{split}\n\\end{equation*}\n","sha256_utf8":"189837af0a752db0c0946756976f77b998cfa89fba0df74db8ec3b4a73d9524b"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"加权求和与望远镜消去","source_units":[{"start":35505,"end":36162,"start_line":462,"end_line":470,"raw_tex":"where $\\tilde{M} = \\frac{M}{(n_a + n_b)^2}$. By taking total expectation over $\\{\\xi_t\\}$, multiplying both sides by $t$, and summing over $t = 1, \\ldots, T$, one obtains\n\\begin{equation*}\n    \\begin{split}\n       \\sum_{t=1}^T \n       t(\\mathbb{E}[S(x_t, \\lambda_*)] - S(x_*, \\lambda_*)) \\;\\leq\\; & \\sum_{t=1}^T \\left ( \\frac{ct(t-1)}{4}\\mathbb{E}[\\|x_t - x_*\\|^2]  - \\frac{ct(t+1)}{4} \\mathbb{E}[\\|x_{t+1} - x_*\\|^2]\\right) \\\\\n       & + \\sum_{t=1}^T \\frac{\\tilde{M}t}{c(t+1)} \\\\\n       \\;\\leq\\; & -\\frac{cT(T+1)}{4} \\mathbb{E}[\\|x_{T+1} - x_*\\|^2] + \\sum_{t=1}^T \\frac{\\tilde{M}t}{c(t+1)}  \\;\\leq\\; \\frac{T}{c} \\tilde{M}. \n    \\end{split}\n\\end{equation*}\n","sha256_utf8":"0d2a2a123555dd07b9dcd8964566d18a6a62eb556246c510d62ffea41f71bdda"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"归一化得到最优目标差界","source_units":[{"start":36162,"end":36423,"start_line":471,"end_line":476,"raw_tex":"Dividing both sides of the last inequality by $\\sum_{t=1}^T t$ yields\n\\begin{equation*}\n    \\min_{t = 1, \\ldots,T} \\mathbb{E}[S(x_t, \\lambda_*)] - S(x_*, \\lambda_*) \\;\\leq\\; \\frac{2}{c(T + 1)} \\tilde{M},   \n\\end{equation*}\nwhich concludes the proof.\n\\end{proof}","sha256_utf8":"b2e92e8b4e96d148015d493558810dff8fddf3638e6e73f5557f7e744441ea53"}]},{"id":"S033","semantic_type":"external_fact","kind":"external_fact","title":"中间值定理","source_units":[],"declared_external":true,"declaration_text":["连续实值函数在线段上取得两个端点函数值之间的每个值。"]},{"id":"S034","semantic_type":"external_fact","kind":"external_fact","title":"正交投影非扩张性","source_units":[],"declared_external":true,"declaration_text":["非空闭凸集X上的欧氏正交投影满足||P_X(u)-P_X(v)||≤||u-v||，且当v属于X时P_X(v)=v。"]},{"id":"S035","semantic_type":"external_fact","kind":"external_fact","title":"范数与期望基本不等式","source_units":[],"declared_external":true,"declaration_text":["三角不等式、柯西–施瓦茨不等式和詹森不等式成立；特别地，E||Z||≤sqrt(E||Z||²)，且||sum_{j=1}^m v_j||²≤m sum_{j=1}^m ||v_j||²。"]},{"id":"S036","semantic_type":"external_fact","kind":"external_fact","title":"条件期望塔式法则","source_units":[],"declared_external":true,"declaration_text":["可积随机变量的条件期望满足塔式法则；独立抽样允许按已生成的迭代历史逐次取条件期望。"]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S009","edges":[{"source_id":"S001","target_id":"S016","kind":"supports"},{"source_id":"S002","target_id":"S015","kind":"supports"},{"source_id":"S002","target_id":"S024","kind":"supports"},{"source_id":"S002","target_id":"S025","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S017","kind":"supports"},{"source_id":"S004","target_id":"S022","kind":"supports"},{"source_id":"S005","target_id":"S018","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S007","target_id":"S021","kind":"supports"},{"source_id":"S007","target_id":"S030","kind":"supports"},{"source_id":"S008","target_id":"S016","kind":"supports"},{"source_id":"S008","target_id":"S018","kind":"supports"},{"source_id":"S003","target_id":"S029","kind":"supports"},{"source_id":"S033","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S011","kind":"supports"},{"source_id":"S011","target_id":"S017","kind":"supports"},{"source_id":"S011","target_id":"S023","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S034","target_id":"S016","kind":"supports"},{"source_id":"S036","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S019","kind":"supports"},{"source_id":"S017","target_id":"S019","kind":"supports"},{"source_id":"S035","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S035","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S035","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S030","kind":"supports"},{"source_id":"S021","target_id":"S030","kind":"supports"},{"source_id":"S035","target_id":"S023","kind":"supports"},{"source_id":"S035","target_id":"S024","kind":"supports"},{"source_id":"S035","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S026","kind":"supports"},{"source_id":"S024","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S018","target_id":"S027","kind":"supports"},{"source_id":"S023","target_id":"S028","kind":"supports"},{"source_id":"S025","target_id":"S028","kind":"supports"},{"source_id":"S018","target_id":"S028","kind":"supports"},{"source_id":"S022","target_id":"S029","kind":"supports"},{"source_id":"S027","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S036","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S014","target_id":"S010","kind":"supports"},{"source_id":"S015","target_id":"S010","kind":"supports"},{"source_id":"S016","target_id":"S010","kind":"supports"},{"source_id":"S017","target_id":"S010","kind":"supports"},{"source_id":"S018","target_id":"S010","kind":"supports"},{"source_id":"S019","target_id":"S010","kind":"supports"},{"source_id":"S020","target_id":"S010","kind":"supports"},{"source_id":"S021","target_id":"S010","kind":"supports"},{"source_id":"S022","target_id":"S010","kind":"supports"},{"source_id":"S023","target_id":"S010","kind":"supports"},{"source_id":"S024","target_id":"S010","kind":"supports"},{"source_id":"S025","target_id":"S010","kind":"supports"},{"source_id":"S026","target_id":"S010","kind":"supports"},{"source_id":"S027","target_id":"S010","kind":"supports"},{"source_id":"S028","target_id":"S010","kind":"supports"},{"source_id":"S029","target_id":"S010","kind":"supports"},{"source_id":"S030","target_id":"S010","kind":"supports"},{"source_id":"S031","target_id":"S010","kind":"supports"},{"source_id":"S032","target_id":"S010","kind":"supports"},{"source_id":"S010","target_id":"S009","kind":"supports"}]}}
{"record_id":"2204.01806v2","paper":{"title":"Inexact reduced gradient methods in nonconvex optimization","authors":["Pham Duy Khanh","Boris S. Mordukhovich","Dat Ba Tran"],"arxiv_id":"2204.01806","version":"v2","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-023-02319-9","abstract_url":"https://arxiv.org/abs/2204.01806v2","pdf_url":"https://arxiv.org/pdf/2204.01806v2"},"context":{"raw_tex":"\\begin{align}\\label{optim prob}\n{\\rm minimize}\\quad f(x)\\quad\\text{ subject to }\\; x\\in\\R^n\n\\end{align}\n\nwith a continuously differentiable ($\\mathcal{C}^1$-smooth) objective function $f:\\R^n\\rightarrow\\R$. One of the most natural and classical approaches to solve (\\ref{optim prob}) is by using \\textit{linesearch methods}; see, e.g., \\cite{beck,bertsekasbook,solodovbook,nesterovbook18,nocedalbook,polyakbook}. Given a starting point $x^1\\in\\R^n$, such methods construct the iterative procedure \n\\begin{align}\\label{intro: stepsize + descent direction}\nx^{k+1}:=x^k+t_kd^k\\;\\text{ for all }\\;k\\in\\N,\n\\end{align}\n\nFirst we recall some notions and notation frequently used in what follows. All our considerations are given in the space $\\R^n$ with the Euclidean norm $\\|\\cdot\\|$ and scalar/inner product $\\dotproduct{\\cdot,\\cdot}$. We use $\\N:=\\{1,2,\\ldots\\}$, $\\R_+$, and $\\overline{\\R}:=\\R\\cup\\set{\\infty}$ to denote the collections of natural numbers, positive numbers, and the extended real line, respectively. The symbol $x^k\\st{J}{\\to}\\ox$ means that $x^k\\to\\ox$ as $k\\to\\infty$ with $k\\in J\\subset\\N$. For a $\\mathcal{C}^1$-smooth function  $f:\\R^n\\rightarrow\\R$, $\\bar x$ is a \\textit{stationary point} of $f$ if $\\nabla f(\\bar x)=0$. The function $f$ is said to satisfy the {\\em $L$-descent condition} with some $L>0$ if\n\\begin{align}\\label{descent condition}\nf(y)\\le f(x)+\\dotproduct{\\nabla f(x),y-x}+\\dfrac{L}{2}\\norm{y-x}^2\\;\\text{ for all }\\;x,y\\in\\R^n.\n\\end{align}\n\n\\begin{Definition}\\rm\\label{defi: approximate gradient}\nLet $\\set{x^k}$ be a sequence in $\\R^n$. The direction sequence $\\set{d^k}$ is called \\textit{gradient associated} with $\\set{x^k}$ if we have the implication\n\\begin{align}\\label{condition for stationarity}\n\\text{ whenever }\\;d^k\\overset{J}{\\rightarrow}0\\;\\text{ for some infinite set }\\;J\\subset\\N,\\;\\text{ it holds that }\\;\\nabla f(x^k)\\overset{J}{\\rightarrow}0.\n\\end{align}\n\\end{Definition}\n\n\\begin{Definition}\\rm \\label{KL ine}\\rm\nLet $f:\\R^n\\rightarrow\\R$ be a differentiable function. We say that $f$ satisfies the \\textit{KL property} at $\\bar{x}\\in\\R^n$ if there exist a number $\\eta>0$, a neighborhood $U$ of $\\bar{x}$, and a nondecreasing function $\\psi:(0,\\eta)\\rightarrow(0,\\infty)$ such that the function $1/\\psi$ is integrable over $(0,\\eta)$ and we have\n\\begin{align}\\label{kl absil}\n \\norm{\\nabla f(x)}\\ge\\psi\\big(f(x)-f(\\bar{x})\\big)\n\\end{align}\nfor all $x\\in U$ with $f(\\bar x)<f(x)<f(\\bar x)+\\eta$.\n\\end{Definition}\n\n\\begin{align}\\label{series is finite}\n\\sum_{k=1}^\\infty\\norm{x^{k+1}-x^k}\\cdot\\norm{d^k}<\\infty.\n\\end{align}\n\nAlthough the focus of this paper is on smooth optimization problems, we first recall the {\\em perturbed} notions of generalized differentiation for nonsmooth functions and their smooth specifications employed in \\cite{blo02,kiwiel07} as well as different ones used in our algorithms. In what follows, the notation $\\B(\\ox,\\ve)$ stands for the closed ball centered at $\\ox\\in\\R^n$ with radius $\\ve>0$, by ${\\rm co}\\,\\O$ we understand the convex hull of a set $\\O\\subset\\R^n$, and the symbol $x\\st{\\O}{\\to}\\ox$ indicates that $x\\to\\ox$ with $x\\in\\O$.\\vspace*{0.05in} \n\n\nPrior to introducing our IRG methods, let us briefly describe the GS scheme based on the $\\ve$-generalized gradient in the smooth case \\eqref{egg-smooth}. Recall that the \n{\\em projection} ${\\rm Proj}(x,\\O)$ of $x\\in\\R^n$ onto a closed and convex set $\\O\\subset\\R^n$ is a singleton that is characterized by \n\\begin{align}\\label{proj}\nu={\\rm Proj}(x,\\O)\\;\\text{ if and only if $u\\in\\O$ and  }\\;\\dotproduct{y-u,x-u}\\le 0\\;\\text{ for all }\\;y\\in\\O.\n\\end{align}\n\n\\begin{Algorithm}[\\bf general framework for IRG methods]\\rm \\label{al gd inexact}\n\\quad\n\\setcounter{Step}{-1}\n\\begin{Step}[initialization]\\rm Select an initial point $x^1\\in \\R^n$, initial radii $\\varepsilon_1,r_1> 0$, radius reduction factors $\\mu,\\theta\\in(0,1)$, a sequence of manually controlled errors $\\set{\\rho_k}\\subset \\R_+.$\n\\end{Step}\n\\begin{Step}[inexact gradient and stopping criterion]\\rm \\label{Step 1}\nChoose $g^k$ such that \n\\begin{align}\\label{calculate approx grad}\n\\norm{g^k-\\nabla f(x^k)}\\le \\min\\set{\\varepsilon_k,\\rho_k}.\n\\end{align}\n\\end{Step}\n\\begin{Step}[radius update]\\rm \\label{Step 2}\nIf $\\norm{g^k}\\le r_{k}+\\varepsilon_k$, then set $r_{k+1}:=\\mu r_k,\\;\\varepsilon_{k+1}:=\\theta\\varepsilon_k$, $d^k:=0$, and go to~Step~3. Otherwise, set $r_{k+1}:=r_k$, $\\varepsilon_{k+1}:=\\varepsilon_k$, and \n\\begin{align}\\label{choose dk}\nd^k:=-\\dfrac{\\norm{g^k}-\\varepsilon_k}{\\norm{g^k}}g^k.\n\\end{align}\n\\end{Step}\n\\begin{Step}[stepsize]\\rm\\label{Step: 3}\nChoose $t_k>0$ by a specific rule.\n\\end{Step}\n\\begin{Step}[iteration update]\\rm\\label{step 4}\nSet $x^{k+1}:=x^{k}+t_kd^k$.\n\\end{Step}\n\\begin{Step}\\rm\nIncrease $k$ by $1$ and go back to Step $1.$\n\\end{Step}\n\\end{Algorithm}\n\n\\item[\\bf(iii)] For each $k\\in\\N$, we have from Step~\\ref{Step 2} and Step~\\ref{Step: 3} the equivalences\n\\begin{align}\\label{alter condi for null}\nx^{k+1}=x^k\\Longleftrightarrow d^k=0\\Longleftrightarrow r_{k+1}=\\mu r_k\\Longleftrightarrow\\varepsilon_{k+1}=\\theta\\varepsilon_k\\Longleftrightarrow\\norm{g^k}\\le r_k+\\varepsilon_k.\n\\end{align}\n\n\\begin{Definition}\\rm\\label{nul iter}\nThe $k^{\\rm th}$ iteration of Algorithm~{\\rm\\ref{al gd inexact}} is called a {\\em null iteration} if $x^{k+1}=x^k$. The set of all null iterations is denoted by\n$$\n\\mathcal{N}:=\\big\\{k\\in\\N\\;\\big|\\;x^{k+1}=x^k\\big\\}.\n$$\n\\end{Definition}\n\nIn this section, we develop novel IRG methods with the following selections of stepsize rules: {\\em backtracking stepsize}, {\\em constant stepsize}, and {\\em diminishing stepsize}. Address first an IRG method with the {\\em backtracking line search}. Choose a line search scalar $\\beta\\in(0,1)$, a reduction factor $\\gamma\\in(0,1)$, and an artificial stepsize of null iterations $\\tau\\in(0,1)$. Consider the {\\em Master Algorithm}~\\ref{al gd inexact} with the stepsize sequence $\\set{t_k}$ in Step~3 calculated as follows. If $d^k=0$, then put $t_k:=\\tau$. Otherwise, we set\n\\begin{align}\\label{defi: tk}\nt_k:=\\max\\big\\{t\\;\\big|\\;f(x^k+td^k)\\le f(x^k)-\\beta t\\big\\|d^k\\big\\|^2,\\;t=1,\\;\\gamma,\\;\\gamma^2,\\ldots\\big\\}.\n\\end{align}\n\nThe last two theorems establish efficient conditions ensuring the {\\em convergence rates} in Algorithm~\\ref{al gd inexact} under different stepsize selections. Having the sequence of iterations $\\set{x^k}$ generated by this algorithm, we obtain first from Proposition~\\ref{null properties}(iii) that if $\\N\\setminus\\mathcal{N}$ is finite, then $\\set{x^k}$ stops after a finite number of iterations. Thus we consider the case where the set $\\N\\setminus\\mathcal{N}$ is infinite and can be numerated as $\\set{j_1,j_2,\\ldots}$. Construct the sequence $\\set{z^k}$ by\n\\begin{align}\\label{z_k}\n z^k:=x^{j_k}\\;\\text{ for all }\\;k\\in\\N.\n\\end{align}","units":[{"start":6028,"end":6131,"start_line":153,"end_line":155,"raw_tex":"\\begin{align}\\label{optim prob}\n{\\rm minimize}\\quad f(x)\\quad\\text{ subject to }\\; x\\in\\R^n\n\\end{align}","sha256_utf8":"047de3103461676b3488a0933f586b78b7aa37c9446dbddc97f622a7941ddab4"},{"start":6132,"end":6640,"start_line":156,"end_line":159,"raw_tex":"with a continuously differentiable ($\\mathcal{C}^1$-smooth) objective function $f:\\R^n\\rightarrow\\R$. One of the most natural and classical approaches to solve (\\ref{optim prob}) is by using \\textit{linesearch methods}; see, e.g., \\cite{beck,bertsekasbook,solodovbook,nesterovbook18,nocedalbook,polyakbook}. Given a starting point $x^1\\in\\R^n$, such methods construct the iterative procedure \n\\begin{align}\\label{intro: stepsize + descent direction}\nx^{k+1}:=x^k+t_kd^k\\;\\text{ for all }\\;k\\in\\N,\n\\end{align}","sha256_utf8":"ad9a5dc3f9506283974bbd7f26623ee1d7956842ba38f4a747b8da2421d1a44b"},{"start":19953,"end":20816,"start_line":212,"end_line":215,"raw_tex":"First we recall some notions and notation frequently used in what follows. All our considerations are given in the space $\\R^n$ with the Euclidean norm $\\|\\cdot\\|$ and scalar/inner product $\\dotproduct{\\cdot,\\cdot}$. We use $\\N:=\\{1,2,\\ldots\\}$, $\\R_+$, and $\\overline{\\R}:=\\R\\cup\\set{\\infty}$ to denote the collections of natural numbers, positive numbers, and the extended real line, respectively. The symbol $x^k\\st{J}{\\to}\\ox$ means that $x^k\\to\\ox$ as $k\\to\\infty$ with $k\\in J\\subset\\N$. For a $\\mathcal{C}^1$-smooth function  $f:\\R^n\\rightarrow\\R$, $\\bar x$ is a \\textit{stationary point} of $f$ if $\\nabla f(\\bar x)=0$. The function $f$ is said to satisfy the {\\em $L$-descent condition} with some $L>0$ if\n\\begin{align}\\label{descent condition}\nf(y)\\le f(x)+\\dotproduct{\\nabla f(x),y-x}+\\dfrac{L}{2}\\norm{y-x}^2\\;\\text{ for all }\\;x,y\\in\\R^n.\n\\end{align}","sha256_utf8":"072aafc5728d50ea612377c94cdda333bdb574dfbb66a244c0097405cdaf33a0"},{"start":25540,"end":25989,"start_line":263,"end_line":268,"raw_tex":"\\begin{Definition}\\rm\\label{defi: approximate gradient}\nLet $\\set{x^k}$ be a sequence in $\\R^n$. The direction sequence $\\set{d^k}$ is called \\textit{gradient associated} with $\\set{x^k}$ if we have the implication\n\\begin{align}\\label{condition for stationarity}\n\\text{ whenever }\\;d^k\\overset{J}{\\rightarrow}0\\;\\text{ for some infinite set }\\;J\\subset\\N,\\;\\text{ it holds that }\\;\\nabla f(x^k)\\overset{J}{\\rightarrow}0.\n\\end{align}\n\\end{Definition}","sha256_utf8":"f4ef2766cafca555af056867e5be13237aeeb61e2d1edef869ecf57fb326c2fb"},{"start":27173,"end":27712,"start_line":283,"end_line":289,"raw_tex":"\\begin{Definition}\\rm \\label{KL ine}\\rm\nLet $f:\\R^n\\rightarrow\\R$ be a differentiable function. We say that $f$ satisfies the \\textit{KL property} at $\\bar{x}\\in\\R^n$ if there exist a number $\\eta>0$, a neighborhood $U$ of $\\bar{x}$, and a nondecreasing function $\\psi:(0,\\eta)\\rightarrow(0,\\infty)$ such that the function $1/\\psi$ is integrable over $(0,\\eta)$ and we have\n\\begin{align}\\label{kl absil}\n \\norm{\\nabla f(x)}\\ge\\psi\\big(f(x)-f(\\bar{x})\\big)\n\\end{align}\nfor all $x\\in U$ with $f(\\bar x)<f(x)<f(\\bar x)+\\eta$.\n\\end{Definition}","sha256_utf8":"bc30f7942490dc64983ef717785879c85cf710ce8e53e603cefe2cba64c598fa"},{"start":31437,"end":31545,"start_line":324,"end_line":326,"raw_tex":"\\begin{align}\\label{series is finite}\n\\sum_{k=1}^\\infty\\norm{x^{k+1}-x^k}\\cdot\\norm{d^k}<\\infty.\n\\end{align}","sha256_utf8":"084913f86a6acf768e30d5650b4f3a5a910a4a58a4ccff7aeb24c3215e452d79"},{"start":57562,"end":58128,"start_line":664,"end_line":664,"raw_tex":"Although the focus of this paper is on smooth optimization problems, we first recall the {\\em perturbed} notions of generalized differentiation for nonsmooth functions and their smooth specifications employed in \\cite{blo02,kiwiel07} as well as different ones used in our algorithms. In what follows, the notation $\\B(\\ox,\\ve)$ stands for the closed ball centered at $\\ox\\in\\R^n$ with radius $\\ve>0$, by ${\\rm co}\\,\\O$ we understand the convex hull of a set $\\O\\subset\\R^n$, and the symbol $x\\st{\\O}{\\to}\\ox$ indicates that $x\\to\\ox$ with $x\\in\\O$.\\vspace*{0.05in} \n","sha256_utf8":"8dd195c28493b4450743f2d80cb10800dfab4056431c7c297dd05e649532d0c8"},{"start":59280,"end":59737,"start_line":681,"end_line":685,"raw_tex":"Prior to introducing our IRG methods, let us briefly describe the GS scheme based on the $\\ve$-generalized gradient in the smooth case \\eqref{egg-smooth}. Recall that the \n{\\em projection} ${\\rm Proj}(x,\\O)$ of $x\\in\\R^n$ onto a closed and convex set $\\O\\subset\\R^n$ is a singleton that is characterized by \n\\begin{align}\\label{proj}\nu={\\rm Proj}(x,\\O)\\;\\text{ if and only if $u\\in\\O$ and  }\\;\\dotproduct{y-u,x-u}\\le 0\\;\\text{ for all }\\;y\\in\\O.\n\\end{align}","sha256_utf8":"a40bf8dbcaf2ce11c9e1c050eb356cc22fe35f9cc42fabc3540bf30ce45427c3"},{"start":67429,"end":68624,"start_line":767,"end_line":793,"raw_tex":"\\begin{Algorithm}[\\bf general framework for IRG methods]\\rm \\label{al gd inexact}\n\\quad\n\\setcounter{Step}{-1}\n\\begin{Step}[initialization]\\rm Select an initial point $x^1\\in \\R^n$, initial radii $\\varepsilon_1,r_1> 0$, radius reduction factors $\\mu,\\theta\\in(0,1)$, a sequence of manually controlled errors $\\set{\\rho_k}\\subset \\R_+.$\n\\end{Step}\n\\begin{Step}[inexact gradient and stopping criterion]\\rm \\label{Step 1}\nChoose $g^k$ such that \n\\begin{align}\\label{calculate approx grad}\n\\norm{g^k-\\nabla f(x^k)}\\le \\min\\set{\\varepsilon_k,\\rho_k}.\n\\end{align}\n\\end{Step}\n\\begin{Step}[radius update]\\rm \\label{Step 2}\nIf $\\norm{g^k}\\le r_{k}+\\varepsilon_k$, then set $r_{k+1}:=\\mu r_k,\\;\\varepsilon_{k+1}:=\\theta\\varepsilon_k$, $d^k:=0$, and go to~Step~3. Otherwise, set $r_{k+1}:=r_k$, $\\varepsilon_{k+1}:=\\varepsilon_k$, and \n\\begin{align}\\label{choose dk}\nd^k:=-\\dfrac{\\norm{g^k}-\\varepsilon_k}{\\norm{g^k}}g^k.\n\\end{align}\n\\end{Step}\n\\begin{Step}[stepsize]\\rm\\label{Step: 3}\nChoose $t_k>0$ by a specific rule.\n\\end{Step}\n\\begin{Step}[iteration update]\\rm\\label{step 4}\nSet $x^{k+1}:=x^{k}+t_kd^k$.\n\\end{Step}\n\\begin{Step}\\rm\nIncrease $k$ by $1$ and go back to Step $1.$\n\\end{Step}\n\\end{Algorithm}","sha256_utf8":"2c61ec34fc02acb22a6669cb2cb22d10912869d82da462b138491314f93410c3"},{"start":72666,"end":73004,"start_line":837,"end_line":840,"raw_tex":"\\item[\\bf(iii)] For each $k\\in\\N$, we have from Step~\\ref{Step 2} and Step~\\ref{Step: 3} the equivalences\n\\begin{align}\\label{alter condi for null}\nx^{k+1}=x^k\\Longleftrightarrow d^k=0\\Longleftrightarrow r_{k+1}=\\mu r_k\\Longleftrightarrow\\varepsilon_{k+1}=\\theta\\varepsilon_k\\Longleftrightarrow\\norm{g^k}\\le r_k+\\varepsilon_k.\n\\end{align}","sha256_utf8":"f03f63e7d89eced47c8aba1e629b302135a150c6d70b74318bc3e7a20459ae15"},{"start":74306,"end":74580,"start_line":871,"end_line":876,"raw_tex":"\\begin{Definition}\\rm\\label{nul iter}\nThe $k^{\\rm th}$ iteration of Algorithm~{\\rm\\ref{al gd inexact}} is called a {\\em null iteration} if $x^{k+1}=x^k$. The set of all null iterations is denoted by\n$$\n\\mathcal{N}:=\\big\\{k\\in\\N\\;\\big|\\;x^{k+1}=x^k\\big\\}.\n$$\n\\end{Definition}","sha256_utf8":"a8fa19a555a07c2089ad08fa87a3591621bb526e49fe8fcec5c4e71d0bb3e206"},{"start":81831,"end":82558,"start_line":961,"end_line":964,"raw_tex":"In this section, we develop novel IRG methods with the following selections of stepsize rules: {\\em backtracking stepsize}, {\\em constant stepsize}, and {\\em diminishing stepsize}. Address first an IRG method with the {\\em backtracking line search}. Choose a line search scalar $\\beta\\in(0,1)$, a reduction factor $\\gamma\\in(0,1)$, and an artificial stepsize of null iterations $\\tau\\in(0,1)$. Consider the {\\em Master Algorithm}~\\ref{al gd inexact} with the stepsize sequence $\\set{t_k}$ in Step~3 calculated as follows. If $d^k=0$, then put $t_k:=\\tau$. Otherwise, we set\n\\begin{align}\\label{defi: tk}\nt_k:=\\max\\big\\{t\\;\\big|\\;f(x^k+td^k)\\le f(x^k)-\\beta t\\big\\|d^k\\big\\|^2,\\;t=1,\\;\\gamma,\\;\\gamma^2,\\ldots\\big\\}.\n\\end{align}","sha256_utf8":"afc5e2d0ed3d16eb839ad554e4f711ead807b1473ea484cf5a8879880f9d6086"},{"start":102456,"end":103095,"start_line":1177,"end_line":1180,"raw_tex":"The last two theorems establish efficient conditions ensuring the {\\em convergence rates} in Algorithm~\\ref{al gd inexact} under different stepsize selections. Having the sequence of iterations $\\set{x^k}$ generated by this algorithm, we obtain first from Proposition~\\ref{null properties}(iii) that if $\\N\\setminus\\mathcal{N}$ is finite, then $\\set{x^k}$ stops after a finite number of iterations. Thus we consider the case where the set $\\N\\setminus\\mathcal{N}$ is infinite and can be numerated as $\\set{j_1,j_2,\\ldots}$. Construct the sequence $\\set{z^k}$ by\n\\begin{align}\\label{z_k}\n z^k:=x^{j_k}\\;\\text{ for all }\\;k\\in\\N.\n\\end{align}","sha256_utf8":"cd0675d5d62f9ec86a85b021a1c2ce70c4dd44c225b47a9b3d0874cda9a42eb3"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{Theorem} \\label{rate} \nConsider Algorithm~{\\rm\\ref{al gd inexact}} with the backtracking stepsize selections under the condition $\\theta<\\mu$. Let $\\set{x^k}$ be the iterative sequence generated by this algorithm. Suppose that $\\rho_k\\rightarrow0$ as $k\\rightarrow\\infty$. Assume further that $\\set{x^k}$ has an accumulation point $\\bar x$, that $f$ satisfies the KL property at $\\bar x$ with $\\psi(t)=Mt^{q}$ for some $M>0$ and $q\\in(0,1)$, and that $\\nabla f$ is locally Lipschitzian around $\\bar x$.  The following convergence rates are guaranteed for the sequence $\\set{z^k}$ defined in \\eqref{z_k}:\n\\begin{itemize}\n\\item[\\bf(i)] If $q\\in(0,1/2]$, then the sequence $\\set{z^k}$ converges linearly to $\\bar x$.\n\\item[\\bf(ii)] If $q\\in(1/2,1)$, then there exists a positive constant $\\varrho$ such that \n\\begin{align*}\n \\norm{z^k-\\bar x}\\le\\varrho k^{-\\frac{1-q}{2q-1}}\\;\\text{ for all large }\\;k\\in\\N.\n\\end{align*}\n\\end{itemize}\n\\end{Theorem}","units":[{"start":103905,"end":104857,"start_line":1193,"end_line":1202,"raw_tex":"\\begin{Theorem} \\label{rate} \nConsider Algorithm~{\\rm\\ref{al gd inexact}} with the backtracking stepsize selections under the condition $\\theta<\\mu$. Let $\\set{x^k}$ be the iterative sequence generated by this algorithm. Suppose that $\\rho_k\\rightarrow0$ as $k\\rightarrow\\infty$. Assume further that $\\set{x^k}$ has an accumulation point $\\bar x$, that $f$ satisfies the KL property at $\\bar x$ with $\\psi(t)=Mt^{q}$ for some $M>0$ and $q\\in(0,1)$, and that $\\nabla f$ is locally Lipschitzian around $\\bar x$.  The following convergence rates are guaranteed for the sequence $\\set{z^k}$ defined in \\eqref{z_k}:\n\\begin{itemize}\n\\item[\\bf(i)] If $q\\in(0,1/2]$, then the sequence $\\set{z^k}$ converges linearly to $\\bar x$.\n\\item[\\bf(ii)] If $q\\in(1/2,1)$, then there exists a positive constant $\\varrho$ such that \n\\begin{align*}\n \\norm{z^k-\\bar x}\\le\\varrho k^{-\\frac{1-q}{2q-1}}\\;\\text{ for all large }\\;k\\in\\N.\n\\end{align*}\n\\end{itemize}\n\\end{Theorem}","sha256_utf8":"e4aade4159db7da07a8051102dba29243cacdecf01509b1054d2816c2d614a34"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{Lemma}\\label{stationary point lemma}\nLet $\\set{x^k}$ and $\\set{d^k}$ be sequences satisfying \n\\eqref{series is finite}. If $\\bar x$ is an accumulation point of $\\set{x^k}$ and if $0$ is an accumulation point of $\\set{d^k}$, then there exists an infinite set $J\\subset\\N$ such that\n\\begin{align}\\label{relation dk xk 3}\nx^k\\overset{J}{\\rightarrow}\\bar x\\;\\mbox{ and }\\;d^k\\overset{J}{\\rightarrow}0.\n\\end{align}\n\\end{Lemma}\n\n\\begin{proof}\nIf $x^k\\rightarrow\\bar x$ as $k\\to\\infty$, the conclusion obviously holds, so suppose that $x^k\\not\\rightarrow \\bar x$. It suffices to show that for any $\\delta>0$ sufficiently small and any $N\\in\\N$ there is a number $k_N\\ge N$ such that\n\\begin{align*}\n\\|x^{k_N}-\\bar x\\|<\\delta\\;\\text{ and }\\;\\|d^{k_N}\\|<\\delta.\n\\end{align*}\nFix such $\\delta>0$ and $N\\in\\N$. Since $\\delta$ is sufficiently small and $x^k\\not\\rightarrow\\bar x$, suppose that the set\n\\begin{align*}\nA_1:=\\big\\{k\\ge N\\;\\big|\\;\\|x^k-\\bar x\\|\\ge\\delta\\big\\}\\;\\text{ is infinite}.\n\\end{align*}\nAs $\\bar x$ is an accumulation point of $\\set{x^k}$ and $0$ is an accumulation point of $\\set{d^k}$, we get that\n\\begin{align*}\nA_2:&=\\big\\{k\\ge N\\;\\big|\\;\\|x^k-\\bar x\\|<\\delta\\big\\}\\;\\text{ is infinite},\\\\\nA_3:&=\\big\\{k\\ge N\\;\\big|\\;\\|x^k-\\bar x\\|<\\delta/2\\big\\}\\;\\text{ is infinite},\\\\\n A_4:&=\\big\\{k\\ge N\\;\\big|\\;\\|d^k\\|<\\delta\\big\\}\\;\\text{ is infinite}.\n\\end{align*}\nIt suffices to verify that $A_2\\cap A_4\\ne\\emptyset.$ Suppose on the contrary that $A_2\\cap A_4=\\emptyset$, i.e.,\n\\begin{align*}\n\\|d^k\\|\\ge\\delta\\;\\text{ for all }\\;k\\in A_2.\n\\end{align*}\nBy (\\ref{series is finite}), we have the estimates\n\\begin{align*}\n\\infty> \\sum_{k=1}^\\infty\\norm{x^{k+1}-x^k}\\cdot\\norm{d^k}&\\ge \\sum_{k\\in A_2} \\norm{x^{k+1}-x^k}\\cdot\\norm{d^k}\\\\\n&\\ge \\delta\\sum_{k\\in A_2}\\norm{x^{k+1}-x^k},\n\\end{align*}\nwhich ensure the series convergence\n\\begin{align}\\label{sum bounded 1}\n\\sum_{k\\in A_2}\\norm{x^{k+1}-x^k}<\\infty.\n\\end{align}\nTaking any number $K\\in A_3$, we also have $K\\in A_2.$ Since $A_1$ is infinite and  $A_1,A_2$ form a partition of the set $\\set{N,N+1,\\ldots}$ including $K$, there exists a number $\\widehat{K}\\in A_1$ with $\\widehat{K}>K$ such that $K,K+1,\\ldots,\\widehat{K}-1\\in A_2$. Then we have the estimates\n\\begin{align*}\n\\norm{x^{\\widehat{K}}-x^K}\\ge\\norm{x^{\\widehat{K}}-\\bar x}-\\norm{x^K-\\bar x}\\ge\\delta-\\delta/2=\\delta/2.\n\\end{align*}\nUsing the triangle inequality and (\\ref{sum bounded 1}) gives us\n\\begin{align*}\n\\delta/2\\le\\norm{x^{\\widehat{K}}-x^K}\\le \\sum_{i=K}^{\\widehat{K}-1}\\norm{x^{i+1}-x^i}\\le\\sum_{i\\ge K,\\;i\\in A_2}\\norm{x^{i+1}-x^i}\\xrightarrow[K\\in A_3]{K\\rightarrow\\infty}0,\n\\end{align*}\nwhich brings us to a contradiction that completes the proof of the lemma.\n\\end{proof}\n\n\\begin{Lemma}\\label{lemma: ostrowski}\nLet $\\set{x^k}$ be a sequence satisfying the Ostrowski condition\n\\begin{align}\\label{8.3.8}\n\\lim_{k\\rightarrow\\infty}\\|x^{k+1}-x^k\\|=0.\n\\end{align}\nThen the following assertions hold:\n\\begin{itemize}\n\\item[\\bf(i)] If $\\set{x^k}$ is bounded, then the set of accumulation points of $\\set{x^k}$ is nonempty, compact, and connected in $\\R^n$.\n\n\\item[\\bf(ii)] If $\\set{x^k}$ has an isolated accumulation point, then this sequence converges to it.\n\\end{itemize}\n\\end{Lemma}\n\n\\begin{Theorem}\\label{stationary point theorem}\nLet $\\set{x^k}$ be a sequence generated by a linesearch method \\eqref{intro: stepsize + descent direction} such that:\n\\begin{itemize}\n\\item[\\bf(a)] $\\set{d^k}$ is gradient associated with $\\set{x^k}$;\n\\item[\\bf(b)] $0$ is an accumulation point of $\\set{d^k};$\n\\item[\\bf(c)] $\\displaystyle{\\sum_{k=1}^\\infty}t_k\\|d^k\\|^2<\\infty.$\n\\end{itemize}\nThen every accumulation point of $\\set{x^k}$ is a stationary point of $f$. Moreover, if $\\set{t_k}$ is bounded from above, then the following assertions hold:\n\\begin{itemize}\n\\item[\\bf(i)] If $\\set{x^k}$ is bounded, then the set of accumulation points of $\\set{x^k}$ is nonempty, compact, and connected.\n\\item[\\bf(ii)] If $\\set{x^k}$ has an isolated accumulation point, then this sequence converges to it.\n\\end{itemize}\n\\end{Theorem}\n\n\\begin{proof}\nLet $\\bar x$ be an accumulation point of $\\set{x^k}$.  Note that (c) is equivalent to \\eqref{series is finite} under the linesearch relationship $x^{k+1}=x^k+t_kd^k$ in (\\ref{intro: stepsize + descent direction}). Applying Lemma~\\ref{stationary point lemma} with taking into account (b) and (c), we can find an infinite set $J\\subset\\N$ such that $x^k\\overset{J}{\\rightarrow }\\bar x$ and $d^k\\overset{J}{\\rightarrow}0$. Then (a) implies that $\\nabla f(x^k)\\overset{J}{\\rightarrow}0$. Employing the continuity of $\\nabla f$, we have \n\\begin{align*}\n\\nabla f(\\bar x)=\\lim_{k\\overset{J}{\\rightarrow}\\infty}\\nabla f(x^k)=0,\n\\end{align*}\nwhich tells us that $\\bar x$ is a stationary point of $f$. Suppose now that $\\set{t_k}$ is bounded from above by some $\\tau>0$. Using (c), we immediately get\n\\begin{align*}\n\\sum_{k=1}^\\infty\\|x^{k+1}-x^k\\|^2=\\sum_{k=1}^\\infty t_k^2\\|d^k\\|^2\\le\\tau\\sum_{k=1}^\\infty t_k\\|d^k\\|^2<\\infty.\n\\end{align*}\nThis leads us to\n$\\norm{x^{k+1}-x^k}\\rightarrow 0$ and verifies assertions (i) and (ii) by applying Lemma~\\ref{lemma: ostrowski}.\n\\end{proof}\n\n\\begin{Proposition}\\label{general convergence under KL}\nLet $f:\\R^n\\rightarrow\\R$ be a $\\mathcal{C}^1$-smooth function, and let the sequence of iterations $\\set{x^k}\\subset\\R^n$ satisfy the following conditions:\n\\begin{itemize}\n\\item[\\bf(H1)] {\\rm(primary descent condition)}. There exists $\\sigma>0$ such that for sufficiently large $k\\in\\N$ we have \n\\begin{align*}\n f(x^k)-f(x^{k+1})\\ge\\sigma\\norm{\\nabla f(x^k)}\\cdot\\norm{x^{k+1}-x^k}.\n\\end{align*}\n\\item[\\bf(H2)] {\\rm(complementary descent condition)}. For sufficiently large $k\\in\\N$, we have\n\\begin{align*}\n \\big[f(x^{k+1})=f(x^k)\\big]\\Longrightarrow [x^{k+1}=x^k].\n\\end{align*}\n\\end{itemize}\nIf $\\bar x$ is an accumulation point of $\\set{x^k}$  and $f$ satisfies the KL property at $\\bar x$, then $x^k\\rightarrow\\bar x$ as $k\\to\\infty$.\n\\end{Proposition}\n\nNote that \\cite[Theorem~3.2]{absil05} provides a detailed proof for Proposition~\\ref{general convergence under KL} when $f$ satisfies the KL property at $\\bar x$ with $\\psi(t)=Mt^q$ for some $M>0$ and $q\\in [0,1)$. The key ingredient of the proof is the existence of some $k_1\\in\\mathbb{N}$ such that\n\\begin{align}\\label{kl induction}\n \\norm{x^{k+1}-x^k}&\\le  \\dfrac{1}{\\sigma M(1-q )}\\sbrac{(f(x^k)-f(\\bar x))^{1-q }-(f(x^{k+1})-f(\\bar x))^{1-q }}\\text{ for all }k\\ge k_1.\n\\end{align}\nUsing this, we can deduce that\n\\begin{align}\\label{sum series cauchy}\n\\sum_{k=1}^\\infty \\norm{x^{k+1}-x^k}<\\infty,\n\\end{align}\nand so the sequence $\\set{x^k}$ is convergent.\n\n\n\\begin{Lemma}\\label{lemma KL}\nLet $\\set{s_k}$ be a nonnegative sequence in $\\R$, and let $\\alpha,\\beta$ be some positive constants. Suppose that $s_k\\rightarrow 0$ as $k\\to\\infty$ and that this sequence satisfies the condition\n\\begin{align*}\ns_k^\\alpha\\le\\beta(s_k-s_{k-1})\\;\\text{ for all }\\;k\\in\\N\\;\\text{ sufficiently large}.\n\\end{align*}\nThe following assertions hold:\n\\begin{enumerate}\n\\item[\\bf (i)] If $\\alpha\\in(0,1]$, then $s_{k+1}\\le\\brac{1-\\dfrac{1}{\\beta}}s_k$ for all $k\\in\\N$ sufficiently large, i.e., $\\set{s_k}$ converges linearly to $0$ with rank $1-\\dfrac{1}{\\beta}.$\n\\item[\\bf (ii)] If $\\alpha>1$, then there exists a number $\\varrho>0$ such that \n\\begin{align*}\n s_k\\le\\varrho k^{-\\frac{1}{\\alpha-1}}\\;\\text{ for all }\\;k\\in\\N\\;\\text{ sufficiently large}.\n\\end{align*}\n\\end{enumerate}\n\\end{Lemma}\n\n\\begin{Theorem}\\label{general rate}\nLet the sequences $\\set{x^k}\\subset\\R^n$, $\\set{t_k}\\subset\\R_+$ and the numbers $\\beta>0,\\;c>0$ be such that $x^{k+1}\\ne x^k$ for all $k\\in\\N$, and that we have\n\\begin{align}\\label{two conditions}\nf(x^k)-f(x^{k+1})\\ge \\dfrac{\\beta}{t_k}\\norm{x^{k+1}-x^k}^2\\;\\text{ and }\\;\\norm{\\nabla f(x^k)}\\le\\dfrac{c}{t_k}\\norm{x^{k+1}-x^k}\n\\end{align}\nfor sufficiently large $k\\in\\N$. Suppose that the sequence $\\set{t_k}$ is bounded away from $0$, that $\\bar x$ is an accumulation point of $\\set{x^k}$, and that $f$ satisfies the KL property at $\\bar x$ with $\\psi(t)=Mt^{q}$ for some $M>0$ and $q\\in(0,1)$. The following convergence rates are guaranteed:\n\\begin{itemize}\\vspace*{-0.05in}\n\\item[\\bf(i)] If $q\\in(0,1/2]$, then the sequence $\\set{x^k}$ converges linearly to $\\bar x$.\\vspace*{-0.05in}\n\\item[\\bf(ii)]If $q\\in(1/2,1)$, then there exists a positive constant $\\varrho$ such that \n\\begin{align*}\n\\norm{x^k-\\bar x}\\le \\varrho k^{-\\frac{1-q}{2q-1}}\\;\\text{ for sufficiently large }\\;k\\in\\N.\n\\end{align*}\n\\end{itemize}\n\\end{Theorem}\n\n\\begin{proof}\nThe conditions imposed in (\\ref{two conditions}) imply that\n\\begin{align*}\nf(x^k)-f(x^{k+1})\\ge \\dfrac{\\beta}{c}\\norm{\\nabla f(x^k)}\\cdot\\norm{x^{k+1}-x^k}\\;\\text{ for sufficiently large }\\;k\\in\\N,\n\\end{align*}\nwhich ensures that assumption (H1) in Proposition~\\ref{general convergence under KL} holds with $\\sigma:=\\dfrac{\\beta}{c}$. By\n$t_k>0$ for all $k\\in\\N$, the first condition in (\\ref{two conditions}) yields also assumption (H2) of this proposition. Let $k_1\\in\\N$ be such that \\eqref{kl induction} is satisfied for all $k\\ge k_1$, which implies in turn inequality \\eqref{sum series cauchy} and thus the convergence of $\\set{x^k}$ to $\\ox$. Moreover, the first condition in (\\ref{two conditions}) together with $x^{k+1}\\ne x^k$ as $k\\in\\N$ implies that \n\\begin{align}\\label{f(xk) converges}\nf(x^k)\\downarrow f(\\bar x):=\\bar f\\;\\text{ and }\\;f(x^k)>\\bar f\\;\\text{ for all }\\;k\\in\\N.\n\\end{align}\nIt follows from (\\ref{sum series cauchy}) and the triangle inequality that\n\\begin{align*}\n\\norm{x^i-\\bar x}\\le\\sum_{k=i}^\\infty\\norm{x^{k+1}-x^k}=:s_i<\\infty\\;\\text{ whenever }\\;i\\in\\N.\n\\end{align*}\nTherefore, the convergence rate of $x^i$ to $\\bar x$ can be deduced from the convergence rate of $s_i$ to $0$. Let $k_1$ be defined in (\\ref{kl induction}) and pick $i\\ge k_1$. Summing up (\\ref{kl induction}) from $i$ to any $p>i$ gives us\n\\begin{align*}\n\\sum_{k=i}^{p}\\norm{x^{k+1}-x^k}\\le\\dfrac{1}{\\sigma M(1-q)}\\big(f(x^i)-\\bar f\\big)^{1-q}.\n\\end{align*}\nLetting $p\\rightarrow\\infty$, we get the estimate\n\\begin{align}\\label{sikm}\ns_i\\le\\dfrac{1}{\\sigma M(1-q )}\\big(f(x^i)-\\bar f\\big)^{1-q}\\;\\text{ for all }\\;i\\ge k_1.\n\\end{align}\nThe KL property of $f$ at $\\bar x$ with $\\psi(t)=Mt^{q }$ allows us to find a positive number $\\eta$ and a neighborhood $U$ of $\\bar x$ ensuring the condition\n\\begin{align}\\label{KL M gradient}\n\\norm{\\nabla f(x)}\\ge M\\big(f(x)-\\bar f\\big)^q\\;\\mbox{ whenever }\\;x\\in U,\\;\\bar f<f(x)<\\bar f+\\eta.\n\\end{align} \nUsing (\\ref{KL M gradient}), $x^k\\rightarrow\\bar x$, and (\\ref{f(xk) converges}) gives us a natural number $k_2>k_1$ such that\n\\begin{align}\\label{M(1-theta)}\n\\norm{\\nabla f(x^i)}\\ge M\\big(f(x^i)-\\bar f\\big)^q\\;\\text{ for all }\\;i\\ge k_2.\n\\end{align}\nLet $\\bar{t}>0$ be a lower bound of $\\set{t_k}$. Combining this with (\\ref{M(1-theta)}) and the second inequality in (\\ref{two conditions}), we have the estimates\n\\begin{align}\n\\big(f(x^i)-\\bar f\\big)^q &\\le\\dfrac{1}{M}\\norm{\\nabla f(x^i)}\\le\\dfrac{c}{Mt_i}\\norm{x^{i+1}-x^i}\\nonumber\\\\\n&\\le\\dfrac{c}{M\\bar{t}}\\norm{x^{i+1}-x^i}\\;\\text{ for all }\\;i\\ge k_2.\\label{3.220}\n\\end{align}\nInvolving further (\\ref{sikm}) and (\\ref{3.220}) ensures that\n\\begin{align*}\ns_i^{\\frac{q}{1-q }}&\\le\\brac{\\dfrac{1}{\\sigma M(1-q)}}^{\\frac{q }{1-q}}\\big(f(x^i)-\\bar f\\big)^q\\\\\n&\\le\\brac{\\dfrac{1}{\\sigma M(1-q)}}^{\\frac{q }{1-q}}\\dfrac{c}{M\\bar{t}}\\norm{x^{i+1}-x^i}.\n\\end{align*}\nDefine $\\alpha:=\\dfrac{q }{1-q }$ and $\\beta:=\\brac{\\dfrac{1}{\\sigma M(1-q )}}^{\\frac{q }{1-q }}\\dfrac{c}{M\\bar{t}}$, and then rewrite the latter estimate as\n\\begin{align*}\ns_i^\\alpha\\le\\beta(s_i-s_{i-1})\\;\\text{ for all }\\;i\\ge k_2.\n\\end{align*}\nApplying finally Lemma~\\ref{lemma KL}, we verify the convergence rates in (i), (ii), and (iii). \n\\end{proof}\n\n\\begin{Proposition}\\label{prop sufi holds}\nIn Algorithm~{\\rm\\ref{al gd inexact}}, $\\set{d^k}$ satisfies the sufficient descent condition with constant~$1,$ i.e., \n\\begin{align}\\label{sufi des holds}\n\\dotproduct{\\nabla f(x^k),d^k}\\le -\\norm{d^k}^2\\;\\text{ for all }\\;k\\in\\N.\n\\end{align}\n\\end{Proposition}\n\n\\begin{proof}\nNote that (\\ref{sufi des holds}) automatically holds if $d^k=0$. Supposing that $d^k\\ne 0$ and using the construction of $d^k$ in Step~\\ref{Step 2}, we have the expression\n\\begin{align*}\n-d^k={\\rm Proj}\\big(0,\\mathbb{B}(g^k,\\varepsilon_k)\\big).\n\\end{align*}\nIt follows from (\\ref{calculate approx grad}) that $\\norm{g^k-\\nabla f(x^k)}\\le \\varepsilon_k$, which means that $\\nabla f(x^k)\\in \\mathbb{B}(g^k,\\varepsilon_k)$. Invoking the projection description for convex sets \\eqref{proj} yields\n\\begin{align*}\n\\dotproduct{0+d^k,\\nabla f(x^k)+d^k}\\le 0,\n\\end{align*}\nwhich is in turn equivalent to\n\\begin{align*}\n\\dotproduct{\\nabla f(x^k),d^k}\\le-\\norm{d^k}^2\n\\end{align*}\nand thus verifies the claim in \\eqref{sufi des holds}.\n\\end{proof}\n\n\\begin{Proposition}\\label{null properties}\nLet $\\set{x^k},\\set{g^k},\\set{d^k},\\set{\\varepsilon_k}$, and $\\set{r_k}$ be sequences generated by Algorithm~{\\rm\\ref{al gd inexact}}. The following assertions hold:\n\\begin{itemize}\n\\item[\\bf(i)] $k\\in\\mathcal{N}$ if and only if either one of the equivalent conditions in \\eqref{alter condi for null} holds.\\vspace*{-0.05in}\n\n\\item[\\bf(ii)] $\\varepsilon_k\\downarrow 0$ if and only if $r_k\\downarrow 0$, which is equivalent to the set $\\mathcal{N}$ being infinite.\n\n\\item[\\bf(iii)] If $\\mathcal{N}$ is finite, then we have\n\\begin{align*}\n \\norm{g^k}>r_N+\\varepsilon_N\\;\\text{ and }\\;\\norm{d^k}>r_N\n\\end{align*}\nfor all $k\\ge N$, where $N:=\\max\\mathcal{N}+1$.\n\n\\item[\\bf(iv)] If $\\N\\setminus\\mathcal{N}$ is finite, then $\\nabla f(x^K)=0$ and $\\set{x^k}_{k\\ge K}$ is a constant sequence, where we denote $K:=\\max\\{\\N\\setminus\\mathcal{N}\\}+1$. Otherwise, $\\bar x$ is an accumulation point of $\\set{x^k}$ if and only if it is an accumulation point of $\\set{x^k}_{k\\in \\N\\setminus\\mathcal{N}}$, and therefore $x^k\\rightarrow\\bar x$ if and only if $x^k\\overset{\\N\\setminus\\mathcal{N}}{\\longrightarrow}\\bar x$.\n\\end{itemize}\n\\end{Proposition}\n\n\\begin{proof} Assertions (i) and (ii) follow directly from Definition~\\ref{nul iter}. To verify (iii), observe that for any natural number $k\\ge N$, the $k^{\\rm th}$ iteration is not a null one and then deduce from (i) that\n\\begin{align*}\n\\varepsilon_{k+1}=\\varepsilon_k=\\varepsilon_N,\\quad r_{k+1}=r_k=r_N,\\;\\mbox{ and }\\;\\norm{g^k}>r_k+\\varepsilon_k=r_N+\\varepsilon_N.\n\\end{align*}\nTogether with Step~\\ref{Step 2} in Algorithm~\\ref{al gd inexact}, this ensures that   \n\\begin{align*}\n\\norm{d^k}=\\norm{g^k}-\\varepsilon_k>r_k=r_N.\n\\end{align*}\nwhich readily justifies assertion (i).\n\nThe proof of (iv) is a bit more involved. Supposing that the set $\\N\\setminus\\mathcal{N}$ is finite, we have that $k\\in \\mathcal{N}$ for all $k\\ge K$. This means by (i) that\n\\begin{align*}\nx^{k+1}=x^{k},\\;\\varepsilon_{k+1}=\\theta\\varepsilon_k,\\;r_{k+1}=\\mu r_k,\\;\\text{ and }\\;\\norm{g^k}\\le r_k+\\varepsilon_k\\;\\text{ whenever }\\;k\\ge K.\n\\end{align*}\nThis tells us that $x^k=x^{K}$ for all $k\\ge K$, and that $\\varepsilon_k\\downarrow 0,\\;r_k\\downarrow 0$, and $g^k\\rightarrow 0$ as $k\\to\\infty$. Taking the limit as $k\\rightarrow \\infty$ in $\\norm{g^k-\\nabla f(x^k)}\\le \\varepsilon_k$ gives us $\\nabla f(x^k)\\rightarrow0$, which yields $\\nabla f(x^{K})=0$.\n\nSupposing otherwise that the set $\\N\\setminus\\mathcal{N}$ is infinite, we obviously get that every accumulation point of $\\set{x^k}_{k\\in \\N\\setminus \\mathcal{N}}$ is an accumulation point of $\\set{x^k}$. Conversely, taking any accumulation point $\\bar x$ of $\\set{ x^k}$, it suffices to show that \n\\begin{align*}\n\\mbox{for any }\\;\\delta>0,\\;N\\in\\N\\;\\mbox{ there exists }\\;k_N\\in\\N\\setminus\\mathcal{N},\\;k_N\\ge N\\;\\mbox{ with }\\;\\norm{x^{k_N}-\\bar x}<\\delta.\n\\end{align*}\nTo verify this, fixing $\\delta >0$ and $N\\in\\N$ and remembering that $\\bar x$ is an accumulation point of $\\set{x^k}$, we find $K\\ge N$ such that $\\norm{x^K-\\bar x}<\\delta$. If $K\\in\\N\\setminus\\mathcal{N}$, choose $k_N:=K$. Otherwise, using that $\\N\\setminus\\mathcal{N}$ is infinite allows us to find $\\widehat{K}\\in\\N\\setminus\\mathcal{N}$ for which $K,K+1,\\ldots,\\widehat{K}-1\\in\\mathcal{N}$. This ensures that \n\\begin{align*}\n{x^{\\widehat{K}}}={x^{\\widehat{K}-1}}=\\ldots=x^{K+1}=x^K,\n\\end{align*}\nand therefore, with $k_N:=\\widehat{K}$, we get that $\\norm{x^{k_N}-\\bar x}<\\delta$. Since $\\delta$ was chosen arbitrarily, this clearly shows that $\\bar x$ is an accumulation point of $\\set{x^k}_{k\\in\\N\\setminus\\mathcal{N}}$ and thus completes the proof. \n\\end{proof}\n\n\\begin{Proposition}\\label{goes to 0} Let $\\set{g^k},\\set{d^k},\\;\\set{\\varepsilon_k}$, and $\\set{r_k}$ be sequences generated by Algorithm~{\\rm\\ref{al gd inexact}}. Then for any $k\\in\\N$ we have the estimates\n\\begin{align}\\label{nx1}\n\\norm{d^k}\\le\\norm{g^k}\\le\\norm{d^k}+\\varepsilon_k+r_k.\n\\end{align}\nConsequently, the following assertions hold:\n\\begin{itemize}\n\\item[\\bf(i)] $\\varepsilon_k\\downarrow0$ if and only if there is an infinite set $J\\subset\\N$ such that $g^k\\overset{J}{\\rightarrow}0$.\n\n\\item[\\bf(ii)] For any infinite set $J\\subset\\N$, we have the equivalence\n\\begin{align*}\ng^k\\overset{J}{\\rightarrow}0\\Longleftrightarrow d^k\\overset{J}{\\rightarrow}0.\n\\end{align*}\n\\end{itemize}\n\\end{Proposition}\n\n\\begin{proof}\nFix any $k\\in\\N$. If $k\\in\\mathcal{N}$, then we get by Proposition~\\ref{null properties}(i) that $d^k=0$ and $\\norm{g^k}\\le \\varepsilon_k+r_k$. Otherwise, Step~\\ref{Step 2} in Algorithm~\\ref{al gd inexact} yields $\\norm{d^k}=\\norm{g^k}-\\varepsilon_k\\le \\norm{g^k}$. In both cases, (\\ref{nx1}) holds. \n\nTo deduce (i) from (\\ref{nx1}), suppose that $\\varepsilon_k\\downarrow 0$ as $k\\to\\infty$. By Proposition~\\ref{null properties}(ii) we have that $r_k\\downarrow 0$ and the set $\\mathcal{N}$ is infinite. Then for any $\\delta>0$ and $N\\in\\N$ there is $k\\ge N$ with $k\\in\\mathcal{N}$ such that\n\\begin{align*}\n\\norm{g^{k}}\\le r_k+\\varepsilon_k<\\delta,\n\\end{align*}\nwhere the first inequality follows from (\\ref{alter condi for null}). Thus we can construct an infinite set $J\\subset\\N$ such that $g^k\\overset{J}{\\longrightarrow}0$. If conversely the sequence $\\set{\\varepsilon_k}$ does not converge to $0$, then the set $\\mathcal{N}$ is finite by Proposition~\\ref{null properties}(ii). Using Proposition~\\ref{null properties}(iii) confirms that $\\set{g^k}$ is bounded away from $0$, which tells us that such an index set $J$ does not exist.\n\nTo verify now assertion (ii), observe that assuming $g^k\\overset{J}{\\rightarrow}0$, implies by the first inequality in (\\ref{nx1}) that $d^k\\overset{J}{\\rightarrow}0$. Conversely, suppose that $d^k\\overset{J}{\\rightarrow}0$ and deduce from Proposition~\\ref{null properties}(i) that the set $\\mathcal{N}$ is infinite. Then it follows from Proposition~\\ref{null properties}(ii) that  $\\varepsilon_k\\downarrow0$ and $r_k\\downarrow0$ as $k\\to\\infty$. Using the second inequality in (\\ref{nx1}), we arrive at $g^k\\overset{J}{\\rightarrow}0$ and thus complete the proof of the proposition.\n\\end{proof}\n\n\\begin{Corollary}\\label{overall} The sequence $\\set{d^k}$ in Algorithms~{\\rm\\ref{al gd inexact}} is gradient associated with $\\set{x^k}$.\n\\end{Corollary}\n\n\\begin{proof} It follows from Proposition \\ref{goes to 0} that the convergence $d^k\\overset{J}{\\rightarrow}0$ yields $g^k\\overset{J}{\\rightarrow}0$ and $\\varepsilon_k\\downarrow0.$ Thus we get  $\\nabla f(x^k)\\overset{J}{\\rightarrow}0$ by taking into account $\\norm{g^k-\\nabla f(x^k)}\\le \\varepsilon_k$ from \\eqref{calculate approx grad}. This shows therefore that the sequence $\\set{d^k}$ is gradient associated with $\\set{x^k}$.\n\\end{proof}\n\n\\begin{Proposition}\\label{descent} If $d^k\\ne 0$, then there exists a positive number $\\bar t$ such that \n\\begin{align*}\nf(x^k+td^k)\\le f(x^k)-\\beta t\\big\\|d^k\\big\\|^2\\;\\text{ for all }\\;t\\in [0,\\bar t],\n\\end{align*}\nwhich ensures the existence of the stepsize sequence $\\{t_k\\}$ in \\eqref{defi: tk}. \n\\end{Proposition}\n\n\\begin{proof} Suppose that $d^k\\ne 0$ and get by Proposition~\\ref{prop sufi holds} that $\\dotproduct{\\nabla f(x^k),d^k}\\le-\\norm{d^k}^2$. By the differentiability of $f$ at $x^k$, for each $t>0$ sufficiently small we have\n\\begin{align*}\nf(x^k+td^k)-f(x^k)&=t\\dotproduct{\\nabla f(x^k),d^k}+o(t)\\le-t\\norm{d^k}^2+o(t)\\\\\n&=-\\beta t\\norm{d^k}^2+t\\brac{\n(\\beta-1)\\norm{d^k}^2+ \\dfrac{o(t)}{t}}.\n\\end{align*}\nSince $o(t)/t\\rightarrow0$ as $t\\dn 0$ and since $(\\beta-1)\\norm{d^k}^2<0$, there exists $\\bar t>0$ such that \n\\begin{align*}\nf(x^k+td^k)\\le f(x^k)-\\beta t\\norm{d^k}^2\\;\\text{ for all }\\;t\\in(0,\\bar t].\n\\end{align*}\nTherefore, all the numbers $t_k$ introduced in (\\ref{defi: tk}) are well-defined.\n\\end{proof}\n\n\\begin{Theorem}\\label{convergence 1}\nLet $\\set{x^k}$ be the sequence of iterations generated by Algorithm~{\\rm\\ref{al gd inexact}} with the stepsize sequences $t_k$ from \\eqref{defi: tk} of the backtracking line search. Assume that $\\inf f(x^k)>-\\infty$ and that $\\rho_k\\rightarrow0$ as $k\\rightarrow\\infty$. Then the following assertions hold:\n\\begin{itemize}\n\\item[\\bf(i)] $\\varepsilon_k\\downarrow0$ and $r_k\\downarrow 0$ as $k\\to\\infty$.\n\n\\item[\\bf(ii)] Every accumulation point of $\\set{x^k}$ is a stationary point of $f$.\n\n\\item[\\bf(iii)] If the sequence $\\set{x^k}$ is bounded, then the set of accumulation points of $\\set{x^k}$ is nonempty, compact, and connected.\n\n\\item[\\bf(iv)] If $\\set{x^k}$ has an isolated accumulation point, then the entire sequence $\\set{x^k}$ converges to this point.\n\\end{itemize}\n\\end{Theorem}\n\n\\begin{proof} From the choice of $\\set{t_k}$ and Step~4 in Algorithm~\\ref{al gd inexact}, for every $k\\in\\N$ we have\n\\begin{align}\\label{tk dk}\n\\beta t_k\\norm{d^k}^2\\le f(x^k)-f(x^{k+1}).\n\\end{align}\nSince $\\inf f(x^k)>-\\infty$, summing up  on both sides of (\\ref{tk dk}) over $k=1,2,\\ldots$ and using the relation $x^{k+1}=x^k+t_kd^k$, we get that\n\\begin{align}\\label{taking sum 1}\n\\sum_{k=1}^\\infty t_k\\norm{d^k}^2<\\infty\\quad\\text{ and }\\quad \\sum_{k=1}^\\infty\\norm{x^{k+1}-x^k}\\cdot\\norm{d^k}<\\infty.\n\\end{align}\n\nTo verify assertion (i), recall by Proposition~\\ref{null properties}(ii) that the convergence $\\varepsilon_k\\downarrow0$ is equivalent to $r_k\\downarrow 0$ and to the fact that the set of null iterations $\\mathcal{N}$ is infinite. Assume on the contrary that $\\mathcal{N}$ is finite. By Proposition~\\ref{null properties}(iii) with $N=\\max\\mathcal{N}+1$, we have\n\\begin{align}\\label{dk bounded}\n\\norm{g^k}>r_N+\\varepsilon_N\\;\\text{ and }\\;\\norm{d^k}>  r_N\\;\\text{ for all }\\;k\\ge N.\n\\end{align}\nThen (\\ref{taking sum 1}) gives us $\\sum_{k=1}^\\infty t_k<\\infty$ and thus $t_k\\downarrow0$ as $k\\to\\infty$. Choosing a larger number $N$ if necessary, we get that $t_k<1$ for all $k\\ge N$. For such $k$, it follows from the exit condition of the algorithm that\n\\begin{align}\\label{exit inexact}\n-\\gamma^{-1}\\beta t_k\\norm{d^k}^2< f(x^{k}+\\gamma^{-1}t_kd^k)-f(x^k).\n\\end{align}\nBy the classical mean value theorem, there exists some $\\tilde{x}^k\\in[x^k,x^k+\\gamma^{-1}t_kd^k]$ such that \n\\begin{align*}\nf(x^{k}+\\gamma^{-1}t_kd^k)-f(x^k)= \\gamma^{-1}t_k\\dotproduct{d^k,\\nabla f(\\tilde{x}^k)}.\n\\end{align*}\nCombining the latter with (\\ref{exit inexact}) tells us that\n\\begin{align}\\label{contra 1}\n\\dotproduct{-d^k,\\nabla f(\\tilde{x}^k)}\\le\\beta \\norm{d^k}^2\\;\\text{ for all }\\;k\\ge N.\n\\end{align}\nUsing (\\ref{taking sum 1}) and (\\ref{dk bounded}), we have $\\sum_{k=1}^\\infty\\norm{x^{k+1}-x^k}<\\infty$, and thus $\\set{x^k}$ converges to some $\\bar x\\in\\R^n$. The continuity of $\\nabla f$ ensures that $\\nabla f(x^k)\\rightarrow\\nabla f(\\bar x)$. Then employing $\\norm{g^k-\\nabla f(x^k)}\\le \\rho_k\\rightarrow0$ yields $g^k\\rightarrow\\nabla f(\\bar x)$ as $k\\rightarrow\\infty$.\nIt follows from Step~\\ref{Step 2} that\n\\begin{align*}\n-d^k&=\\dfrac{\\norm{g^k}-\\varepsilon_k}{\\norm{g^k}}g^k=\\dfrac{\\norm{g^k}-\\varepsilon_N}{\\norm{g^k}}g^k\\;\\text{ for all \\;}k\\ge N.\n\\end{align*}\nLetting $k\\rightarrow\\infty$ leads us to the equalities\n\\begin{align}\\label{1.8 1}\n-d^k\\rightarrow\\bar g:=\\dfrac{\\norm{\\nabla f(\\bar x)}- \\varepsilon_N }{\\norm{\\nabla f(\\bar x)}}\\nabla f(\\bar x)={\\rm Proj}\\big(0,\\mathbb{B}(\\nabla f(\\bar x),\\varepsilon_N)\\big).\n\\end{align}\nUsing $t_k\\downarrow0$, we get that $\\tilde{x}^k\\rightarrow\\bar x$, and thus $\\nabla f(\\tilde{x}^k)\\rightarrow\\nabla f(\\bar x)$ as $k\\to\\infty$. Combining the latter with (\\ref{contra 1}), (\\ref{1.8 1}), and the projection characterization in \\eqref{proj} verifies the estimates\n\\begin{align}\n\\norm{\\bar g}^2\\le\\dotproduct{\\bar g,\\nabla f(\\bar x)}\\le\\beta \\norm{\\bar g}^2.\n\\end{align}\nThis tells us that $\\bar g=0$, which contradicts the condition $\\norm{\\bar g}\\ge r_N$ by (\\ref{dk bounded}). Therefore, we arrive at $\\varepsilon_k\\downarrow 0$ and $r_k\\downarrow 0$ as $k\\to\\infty$, which completes the proof of assertion (i).\n\nTo justify assertions (ii)--(iv), recall from Corollary~\\ref{overall} that $\\set{d^k}$ is \\textit{gradient associated} with $\\set{x^k}$. Since $\\varepsilon_k\\downarrow 0$, we deduce from Proposition~\\ref{goes to 0} that \n$0$ is an accumulation point of $\\set{d^k}$. Combining these facts with (\\ref{taking sum 1}) and $t_k\\le 1$ whenever $k\\in\\N$ ensures that all the assumptions of Theorem~\\ref{stationary point theorem} are satisfied. Therefore, we verify assertions (ii)--(iv) and finish the proof of the theorem.\n\\end{proof}\n\n\\begin{Lemma}\\label{gradien 3 dk}\nLet $\\set{x^k}$ be the sequence generated by Algorithm~{\\rm\\ref{al gd inexact}} with $\\theta<\\mu$. Assume that  $\\varepsilon_k\\downarrow 0$ and $r_k\\downarrow0$ as $k\\to\\infty$. Then there exists some $N\\in\\N$ such that\n\\begin{align}\\label{equation grad 3k}\n\\norm{\\nabla f(x^k)}\\le 3\\norm{d^k}\\;\\text{ for all }\\;k\\notin\\mathcal{N},\\;k\\ge N,\n\\end{align}\nwhere the set $\\mathcal{N}$ is taken from Definition~{\\rm\\ref{nul iter}}.\n\\end{Lemma}\n\n\\begin{proof} It follows directly from the assumptions of the lemma that there exists a natural number $N$ such that $\\varepsilon_k\\le r_k$ for all $k\\ge N$. By Step~\\ref{Step 2} of the algorithm, for any $k\\ge N$ with $k\\notin\\mathcal{N}$ we have $\\norm{g^k}>r_k+\\varepsilon_k$ with the direction $d^k$ calculated in (\\ref{choose dk}). Thus for such $k$ we get the estimates\n\\begin{align}\\label{66}\n\\norm{d^k}=\\norm{g^k}-\\varepsilon_k>r_k+\\varepsilon_k-\\varepsilon_k= r_k\\ge\\varepsilon_k.\n\\end{align}\nIt follows from (\\ref{calculate approx grad}) in Step~\\ref{Step 1} and from (\\ref{66}) that\n\\begin{align*}\n\\norm{\\nabla f(x^k)}\\le \\norm{g^k}+\\varepsilon_k=\\norm{d^k}+2\\varepsilon_k\\le 3\\norm{d^k},\n\\end{align*}\nwhich verifies the conclusion of the lemma.\n\\end{proof}\n\n\\begin{Theorem}\\label{convergence under KL}\nLet $\\set{x^k}$ be the iterative sequence generated by Algorithm~{\\rm\\ref{al gd inexact}} with the backtracking linesearch such that $\\theta<\\mu$. Assume that $\\rho_k\\rightarrow0$ as $k\\rightarrow\\infty$, $\\set{x^k}$ has an accumulation point $\\bar x$, and $f$ satisfies the KL property at $\\bar x$. Then $\\bar x$ is a stationary point of $f$, and we have $x^k\\rightarrow\\bar x$ as $k\\to\\infty$.\n\\end{Theorem}\n\n\\begin{proof}\nSince $\\bar x$ is an accumulation point of $\\set{x^k}$, we can find some infinite set $J\\subset\\N$ such that $x^k\\overset{J}{\\rightarrow}\\bar x$. It follows from the choice of $\\set{t_k}$ in \\eqref{defi: tk} that $\\set{f(x^k)}$ is nonincreasing, which implies that \n\\begin{align*}\n\\inf_{k\\in \\N} f(x^k)= \\inf_{k\\in J} f(x^k)=f(\\bar x)>-\\infty.\n\\end{align*}\nTherefore, the results of Theorem~\\ref{convergence 1} tell us that $\\bar x$ is a stationary point of $f$ and that $\\varepsilon_k\\dn 0$, $r_k\\downarrow 0$ as $k\\to\\infty$. We employ Proposition~\\ref{general convergence under KL} to verify that $x^k\\rightarrow\\bar x$ along the entire sequence of iterations. Indeed, the imposed assumptions and the convergence $\\varepsilon_k\\dn 0$, $r_k\\downarrow 0$ as $k\\to\\infty$ guarantee that all the requirements  of Lemma~\\ref{gradien 3 dk} are satisfied. Pick $N\\in\\N$ such that (\\ref{equation grad 3k}) holds. The choice of $\\set{t_k}$ in (\\ref{defi: tk}) ensures the lower estimate\n\\begin{align}\\label{5.133}\nf(x^{k})-f(x^{k+1})&\\ge\\beta t_k\\big\\|d^k\\big\\|^2\\;\\text{ for all }\\;k\\in\\N.\n\\end{align}\nCombining this with (\\ref{equation grad 3k}) and the relation $x^{k+1}=x^k+t_kd^k$ yields\n\\begin{align}\\label{5.13}\nf(x^{k})-f(x^{k+1})\\ge\\dfrac{\\beta}{3}\\norm{\\nabla f(x^k)}\\cdot\\norm{x^{k+1}-x^k}\\;\\text{ for all }\\;k\\notin\\mathcal{N},\\;k\\ge N.\n\\end{align}\nObserve that when $k\\in\\mathcal{N}$, both sides of\n(\\ref{5.13}) reduce to zero, and so (\\ref{5.13}) is satisfied. Therefore, assumption (H1) in Proposition~\\ref{general convergence under KL} holds. Moreover, for $k\\ge N$ the conditions $f(x^{k+1})=f(x^k)$ and (\\ref{5.133}) imply that $d^k=0$, and hence $x^{k+1}=x^k$. Thus assumption (H2) in  Proposition~\\ref{general convergence under KL} is satisfied as well. Applying the latter proposition, we arrive at $x^k\\rightarrow\\bar x$ as $k\\to\\infty$ and complete the proof.\n\\end{proof}\n\n\\begin{Proposition}\\label{bounded away}\nLet $\\set{x^k}$ be a sequence generated by Algorithm~{\\rm\\ref{al gd inexact}} with the backtracking stepsize. Suppose that $\\rho_k\\rightarrow0$ as $k\\rightarrow\\infty$ and that there exists an infinite set $J\\subset\\N$ such that $\\set{x^k}_{k\\in J}$ converges to some $\\bar x$. Assume further that $\\nabla f$ is locally Lipschitzian around $\\bar x$. Then the stepsize sequence $\\set{t_k}_{k\\in J}$ is bounded away from zero.\n\\end{Proposition}\n\n\\begin{proof}\n Assume on the contrary that $\\set{t_k}_{k\\in J}$ is not bounded away from zero. Then we find an infinite set $\\bar J\\subset J$ such that $t_k\\overset{\\bar J}{\\longrightarrow}0$. Let $\\tau\\in(0,1)$ be an artificial stepsize of null iterations. Since $t_k\\overset{\\bar J}{\\longrightarrow}0$, there exists a number $N\\in\\N$ such that \n \\begin{align}\\label{tk < tau}\nt_k<\\tau<1\\;\\text{ for all }\\;k\\ge N,\\;k\\in\\bar J.\n \\end{align}\n This means that $k\\notin\\mathcal{N}$ whenever $k\\ge N,\\;k\\in\\bar J$. By Proposition~\\ref{null properties}(i), we have $d^k\\ne 0$ for all $k\\ge N,\\;k\\in\\bar J$. Then condition (\\ref{choose dk}) in Step~\\ref{Step 2} leads us to\n\\begin{align}\\label{norm dk < norm gk}\n\\norm{d^k}=\\norm{g^k}-\\varepsilon_k\\le\\norm{g^k}\\;\\mbox{ for all }\\;k\\ge N,\\;k\\in\\bar J.\n\\end{align}\nSince $x^k\\overset{\\bar J}{\\rightarrow}\\bar x$, the continuity of $\\nabla f$ and the estimate $\\norm{\\nabla f(x^k)-g^k}\\le \\rho_k\\rightarrow0$ yield that $g^k\\overset{\\bar J}{\\rightarrow}\\nabla f(\\bar x)$. Using (\\ref{norm dk < norm gk}), we get that the sequence $\\set{d^k}_{k\\in\\bar J}$ is bounded, and thus \n\\begin{align}\\label{bound of norm dk}\nx^k+\\gamma^{-1} t_kd^k\\rightarrow\\bar x\\;\\mbox{ as }\\;k\\overset{\\bar J}{\\rightarrow}\\infty.\n\\end{align}\nSince $\\nabla f$ is locally Lipschitzian around $\\bar x$, there exists a positive number $\\delta$ such that $\\nabla f$ is Lipschitz continuous on $\\mathbb{B}(\\bar x,\\delta)$ with some modulus $L>0$. By (\\ref{bound of norm dk}) and $x^k\\overset{\\bar J}{\\rightarrow}\\bar x$, we find $N_1\\ge N$ with $x^k,x^k+\\gamma^{-1}t_kd^k\\in\\mathbb{B}(\\bar x,\\delta)$ for all $k\\ge N_1,\\;k\\in\\bar J$. The Lipschitz continuity of $\\nabla f$ on $\\mathbb{B}(\\bar x,\\delta)$ with modulus $L$ yields by \\cite[Proposition~A.24]{bertsekasbook} the $L$-descent condition, i.e.,\n\\begin{align}\\label{lipschitz}\nf(x)-f(y)\\le\\dotproduct{\\nabla f(y),x-y}+\\dfrac{L}{2}\\norm{x-y}^2\\;\\text{ for all }\\;x,y\\in \\mathbb{B}(\\bar x,\\delta).\n\\end{align}\nFixing $k\\in\\bar J,\\;k\\ge N_1$, we deduce from the above that $d^k\\ne 0$, and $t_k<1$. The exit condition for the backtracking line search implies that\n\\begin{align}\\label{exit 2}\n-\\gamma^{-1}\\beta t_k\\norm{d^k}^2<f(x^{k}+\\gamma^{-1}t_kd^k)-f(x^k).\n\\end{align}\nApplying (\\ref{lipschitz}) for $x=x^{k}+\\gamma^{-1}t_kd^k$ and $y=x^k$ we have that\n\\begin{align*}\nf(x^{k}+\\gamma^{-1}t_kd^k)-f(x^k)\\le\\gamma^{-1}t_k\\dotproduct{\\nabla f(x^k),d^k}+\\dfrac{L\\gamma^{-2}t_k^2}{2}\\norm{d^k}^2.\n\\end{align*}\nCombining this with (\\ref{exit 2}) leads us to \n\\begin{align*}\n-\\gamma^{-1}\\beta t_k\\norm{d^k}^2<\\gamma^{-1}t_k\\dotproduct{\\nabla f(x^k),d^k}+\\dfrac{L\\gamma^{-2}t_k^2}{2}\\norm{d^k}^2,\n\\end{align*}\nor equivalently to the inequality \n\\begin{align}\\label{0< gamma-1}\n0<\\gamma^{-1}\\beta t_k\\norm{d^k}^2+\\gamma^{-1}t_k\\dotproduct{\\nabla f(x^k),d^k}+\\dfrac{L\\gamma^{-2}t_k^2}{2}\\norm{d^k}^2.\n\\end{align}\nProposition~\\ref{prop sufi holds} and $d^k\\ne 0$ tell us that $0<\\norm{d^k}^2\\le\\dotproduct{\\nabla f(x^k),-d^k}$. Then we deduce from (\\ref{0< gamma-1}) the fulfillment of the estimate\n\\begin{align*}\n0<\\gamma^{-1}\\beta t_k\\dotproduct{\\nabla f(x^k),-d^k}+\\gamma^{-1}t_k\\dotproduct{\\nabla f(x^k),d^k}+\\dfrac{L\\gamma^{-2}t_k^2}{2}\\dotproduct{\\nabla f(x^k),-d^k}.\n\\end{align*}\nDividing both sides above by $\\gamma^{-1}t_k\\dotproduct{\\nabla f(x^k),-d^k}>0$, we get $0<\\beta-1+\\dfrac{L\\gamma^{-1}t_k}{2}$.\nLetting $k\\overset{\\bar J}{\\longrightarrow}\\infty$ yields $\\beta\\ge 1$, which contradicts the choice of $\\beta\\in(0,1)$. Thus we verify that the sequence $\\set{t_k}_{k\\in J}$ is bounded away from zero, which completes the proof of the proposition.\n\\end{proof}\n\nWe have $j_{k+1}\\ge j_{k}+1$ whenever $k\\in\\N$. If the equality holds therein, then $z^{k+1}=z^{j_k+1}$. Otherwise, by taking into account that the indices $j_{k}+1,\\ldots,j_{k+1}-1$ correspond to null iterations, we get that\n\\begin{align}\\label{5 23}\nx^{j_k+1}=x^{j_{k}+2}=\\ldots=x^{j_{k+1}-1}=x^{j_{k+1}}=z^{k+1}.\n\\end{align}\nTherefore, it follows from $j_k\\notin\\mathcal{N}$ that\n\\begin{align}\\label{differ}\nz^{k+1}=x^{j_k+1}\\ne x^{j_k}=z^k\\;\\text{ for all }\\;k\\in\\N.\n\\end{align}\nFurthermore, Proposition~\\ref{null properties}(iv) tells us that $\\bar x$ is an accumulation point of $\\set{z^k}$ if and only if $\\bar x$ is also an accumulation point of $\\set{x^k}$.\\vspace*{0.05in}\n\n\n\\begin{proof}\nThe imposed assumptions yield the fulfillment of those in Theorem~\\ref{convergence under KL}, and so lead us to the convergence $x^k\\rightarrow\\bar x$ as $k\\to\\infty$. Then the local Lipschitz continuity of $\\nabla f$ around $\\bar x$ and Proposition~\\ref{bounded away} ensure that the sequence $\\set{t_k}$ is bounded away from zero.\n \n To deduce now the claimed convergence rates in (i)--(iii) from Theorem~\\ref{general rate}, define $\\tau_k:=t_{j_k}$ for all $k\\in\\N$. Then $\\set{\\tau_k}$ is also bounded away from zero as a subsequence of $\\set{t_k}$. Furthermore, using (\\ref{5 23}) and the line search conditions, we have\n\\begin{align}\\label{beta tau k}\n f(z^k)-f(z^{k+1})&=f(x^{j_{k}})-f(x^{j_{k}+1})\\ge\\beta t_{j_k}\\norm{d^{j_k}}^2\\nonumber\\\\\n &=\\dfrac{\\beta}{t_{j_k}}\\norm{x^{j_k+1}-x^{j_k}}^2=\\dfrac{\\beta}{\\tau_k}\\norm{z^{k+1}-z^k}^2\n\\end{align}\nfor all $k\\in\\N$. Note that all the assumptions of Theorem~\\ref{convergence 1} are satisfied, and so Lemma~\\ref{gradien 3 dk} holds. Pick any $N\\in\\N$ from (\\ref{equation grad 3k}) and fix $k\\ge N$. Then using (\\ref{5 23}) and (\\ref{equation grad 3k}) with taking into account that $j_k\\notin\\mathcal{N}$ for $j_k\\ge k$ leads us to\n\\begin{align*}\n\\norm{\\nabla f(z^k)}=\\norm{\\nabla f(x^{j_k})}&\\le \\dfrac{3}{t_{j_k}}\\norm{x^{j_k+1}-x^{j_k}}=\\dfrac{3}{\\tau_k}\\norm{z^{k+1}-z^k}.\n\\end{align*}\nApply finally Theorem~\\ref{general rate} to $\\set{z^k}$ and $\\set{\\tau_k}$ while remembering that $z^{k+1}\\ne z^k$ for all $k\\in\\N$ from (\\ref{differ}). This verifies the convergence rates (i)--(iii) claimed in the theorem.\n\\end{proof}","units":[{"start":31646,"end":32074,"start_line":329,"end_line":335,"raw_tex":"\\begin{Lemma}\\label{stationary point lemma}\nLet $\\set{x^k}$ and $\\set{d^k}$ be sequences satisfying \n\\eqref{series is finite}. If $\\bar x$ is an accumulation point of $\\set{x^k}$ and if $0$ is an accumulation point of $\\set{d^k}$, then there exists an infinite set $J\\subset\\N$ such that\n\\begin{align}\\label{relation dk xk 3}\nx^k\\overset{J}{\\rightarrow}\\bar x\\;\\mbox{ and }\\;d^k\\overset{J}{\\rightarrow}0.\n\\end{align}\n\\end{Lemma}","sha256_utf8":"1cff84fb18b74b13bc7a997e7510767c8068d024a93d664f4be1e5440f303173"},{"start":32075,"end":34354,"start_line":336,"end_line":373,"raw_tex":"\\begin{proof}\nIf $x^k\\rightarrow\\bar x$ as $k\\to\\infty$, the conclusion obviously holds, so suppose that $x^k\\not\\rightarrow \\bar x$. It suffices to show that for any $\\delta>0$ sufficiently small and any $N\\in\\N$ there is a number $k_N\\ge N$ such that\n\\begin{align*}\n\\|x^{k_N}-\\bar x\\|<\\delta\\;\\text{ and }\\;\\|d^{k_N}\\|<\\delta.\n\\end{align*}\nFix such $\\delta>0$ and $N\\in\\N$. Since $\\delta$ is sufficiently small and $x^k\\not\\rightarrow\\bar x$, suppose that the set\n\\begin{align*}\nA_1:=\\big\\{k\\ge N\\;\\big|\\;\\|x^k-\\bar x\\|\\ge\\delta\\big\\}\\;\\text{ is infinite}.\n\\end{align*}\nAs $\\bar x$ is an accumulation point of $\\set{x^k}$ and $0$ is an accumulation point of $\\set{d^k}$, we get that\n\\begin{align*}\nA_2:&=\\big\\{k\\ge N\\;\\big|\\;\\|x^k-\\bar x\\|<\\delta\\big\\}\\;\\text{ is infinite},\\\\\nA_3:&=\\big\\{k\\ge N\\;\\big|\\;\\|x^k-\\bar x\\|<\\delta/2\\big\\}\\;\\text{ is infinite},\\\\\n A_4:&=\\big\\{k\\ge N\\;\\big|\\;\\|d^k\\|<\\delta\\big\\}\\;\\text{ is infinite}.\n\\end{align*}\nIt suffices to verify that $A_2\\cap A_4\\ne\\emptyset.$ Suppose on the contrary that $A_2\\cap A_4=\\emptyset$, i.e.,\n\\begin{align*}\n\\|d^k\\|\\ge\\delta\\;\\text{ for all }\\;k\\in A_2.\n\\end{align*}\nBy (\\ref{series is finite}), we have the estimates\n\\begin{align*}\n\\infty> \\sum_{k=1}^\\infty\\norm{x^{k+1}-x^k}\\cdot\\norm{d^k}&\\ge \\sum_{k\\in A_2} \\norm{x^{k+1}-x^k}\\cdot\\norm{d^k}\\\\\n&\\ge \\delta\\sum_{k\\in A_2}\\norm{x^{k+1}-x^k},\n\\end{align*}\nwhich ensure the series convergence\n\\begin{align}\\label{sum bounded 1}\n\\sum_{k\\in A_2}\\norm{x^{k+1}-x^k}<\\infty.\n\\end{align}\nTaking any number $K\\in A_3$, we also have $K\\in A_2.$ Since $A_1$ is infinite and  $A_1,A_2$ form a partition of the set $\\set{N,N+1,\\ldots}$ including $K$, there exists a number $\\widehat{K}\\in A_1$ with $\\widehat{K}>K$ such that $K,K+1,\\ldots,\\widehat{K}-1\\in A_2$. Then we have the estimates\n\\begin{align*}\n\\norm{x^{\\widehat{K}}-x^K}\\ge\\norm{x^{\\widehat{K}}-\\bar x}-\\norm{x^K-\\bar x}\\ge\\delta-\\delta/2=\\delta/2.\n\\end{align*}\nUsing the triangle inequality and (\\ref{sum bounded 1}) gives us\n\\begin{align*}\n\\delta/2\\le\\norm{x^{\\widehat{K}}-x^K}\\le \\sum_{i=K}^{\\widehat{K}-1}\\norm{x^{i+1}-x^i}\\le\\sum_{i\\ge K,\\;i\\in A_2}\\norm{x^{i+1}-x^i}\\xrightarrow[K\\in A_3]{K\\rightarrow\\infty}0,\n\\end{align*}\nwhich brings us to a contradiction that completes the proof of the lemma.\n\\end{proof}","sha256_utf8":"8f4c240dd8f74530f0444705713eab9330c0d739ef87d5dbe14b45f0b6e89c72"},{"start":35770,"end":36275,"start_line":389,"end_line":400,"raw_tex":"\\begin{Lemma}\\label{lemma: ostrowski}\nLet $\\set{x^k}$ be a sequence satisfying the Ostrowski condition\n\\begin{align}\\label{8.3.8}\n\\lim_{k\\rightarrow\\infty}\\|x^{k+1}-x^k\\|=0.\n\\end{align}\nThen the following assertions hold:\n\\begin{itemize}\n\\item[\\bf(i)] If $\\set{x^k}$ is bounded, then the set of accumulation points of $\\set{x^k}$ is nonempty, compact, and connected in $\\R^n$.\n\n\\item[\\bf(ii)] If $\\set{x^k}$ has an isolated accumulation point, then this sequence converges to it.\n\\end{itemize}\n\\end{Lemma}","sha256_utf8":"1a1bc6df7483a9c51f183a7ebd43a86d5754701bdd798fa21d120a941273f5be"},{"start":36421,"end":37245,"start_line":404,"end_line":416,"raw_tex":"\\begin{Theorem}\\label{stationary point theorem}\nLet $\\set{x^k}$ be a sequence generated by a linesearch method \\eqref{intro: stepsize + descent direction} such that:\n\\begin{itemize}\n\\item[\\bf(a)] $\\set{d^k}$ is gradient associated with $\\set{x^k}$;\n\\item[\\bf(b)] $0$ is an accumulation point of $\\set{d^k};$\n\\item[\\bf(c)] $\\displaystyle{\\sum_{k=1}^\\infty}t_k\\|d^k\\|^2<\\infty.$\n\\end{itemize}\nThen every accumulation point of $\\set{x^k}$ is a stationary point of $f$. Moreover, if $\\set{t_k}$ is bounded from above, then the following assertions hold:\n\\begin{itemize}\n\\item[\\bf(i)] If $\\set{x^k}$ is bounded, then the set of accumulation points of $\\set{x^k}$ is nonempty, compact, and connected.\n\\item[\\bf(ii)] If $\\set{x^k}$ has an isolated accumulation point, then this sequence converges to it.\n\\end{itemize}\n\\end{Theorem}","sha256_utf8":"7df1cf41467e6992f439de522427cc1de8d07ae4195c502941b2824f7b13803d"},{"start":37246,"end":38333,"start_line":417,"end_line":428,"raw_tex":"\\begin{proof}\nLet $\\bar x$ be an accumulation point of $\\set{x^k}$.  Note that (c) is equivalent to \\eqref{series is finite} under the linesearch relationship $x^{k+1}=x^k+t_kd^k$ in (\\ref{intro: stepsize + descent direction}). Applying Lemma~\\ref{stationary point lemma} with taking into account (b) and (c), we can find an infinite set $J\\subset\\N$ such that $x^k\\overset{J}{\\rightarrow }\\bar x$ and $d^k\\overset{J}{\\rightarrow}0$. Then (a) implies that $\\nabla f(x^k)\\overset{J}{\\rightarrow}0$. Employing the continuity of $\\nabla f$, we have \n\\begin{align*}\n\\nabla f(\\bar x)=\\lim_{k\\overset{J}{\\rightarrow}\\infty}\\nabla f(x^k)=0,\n\\end{align*}\nwhich tells us that $\\bar x$ is a stationary point of $f$. Suppose now that $\\set{t_k}$ is bounded from above by some $\\tau>0$. Using (c), we immediately get\n\\begin{align*}\n\\sum_{k=1}^\\infty\\|x^{k+1}-x^k\\|^2=\\sum_{k=1}^\\infty t_k^2\\|d^k\\|^2\\le\\tau\\sum_{k=1}^\\infty t_k\\|d^k\\|^2<\\infty.\n\\end{align*}\nThis leads us to\n$\\norm{x^{k+1}-x^k}\\rightarrow 0$ and verifies assertions (i) and (ii) by applying Lemma~\\ref{lemma: ostrowski}.\n\\end{proof}","sha256_utf8":"ea09f77cd6c634a9bb13f3b13e8d9a31996c47031bff74fef7c04d63cd53bcca"},{"start":48902,"end":49713,"start_line":547,"end_line":560,"raw_tex":"\\begin{Proposition}\\label{general convergence under KL}\nLet $f:\\R^n\\rightarrow\\R$ be a $\\mathcal{C}^1$-smooth function, and let the sequence of iterations $\\set{x^k}\\subset\\R^n$ satisfy the following conditions:\n\\begin{itemize}\n\\item[\\bf(H1)] {\\rm(primary descent condition)}. There exists $\\sigma>0$ such that for sufficiently large $k\\in\\N$ we have \n\\begin{align*}\n f(x^k)-f(x^{k+1})\\ge\\sigma\\norm{\\nabla f(x^k)}\\cdot\\norm{x^{k+1}-x^k}.\n\\end{align*}\n\\item[\\bf(H2)] {\\rm(complementary descent condition)}. For sufficiently large $k\\in\\N$, we have\n\\begin{align*}\n \\big[f(x^{k+1})=f(x^k)\\big]\\Longrightarrow [x^{k+1}=x^k].\n\\end{align*}\n\\end{itemize}\nIf $\\bar x$ is an accumulation point of $\\set{x^k}$  and $f$ satisfies the KL property at $\\bar x$, then $x^k\\rightarrow\\bar x$ as $k\\to\\infty$.\n\\end{Proposition}","sha256_utf8":"b9cfe5fce787fe97f851030f052418309b4da846abe3ee90a9ac9fff63aa1178"},{"start":49715,"end":50375,"start_line":562,"end_line":570,"raw_tex":"Note that \\cite[Theorem~3.2]{absil05} provides a detailed proof for Proposition~\\ref{general convergence under KL} when $f$ satisfies the KL property at $\\bar x$ with $\\psi(t)=Mt^q$ for some $M>0$ and $q\\in [0,1)$. The key ingredient of the proof is the existence of some $k_1\\in\\mathbb{N}$ such that\n\\begin{align}\\label{kl induction}\n \\norm{x^{k+1}-x^k}&\\le  \\dfrac{1}{\\sigma M(1-q )}\\sbrac{(f(x^k)-f(\\bar x))^{1-q }-(f(x^{k+1})-f(\\bar x))^{1-q }}\\text{ for all }k\\ge k_1.\n\\end{align}\nUsing this, we can deduce that\n\\begin{align}\\label{sum series cauchy}\n\\sum_{k=1}^\\infty \\norm{x^{k+1}-x^k}<\\infty,\n\\end{align}\nand so the sequence $\\set{x^k}$ is convergent.\n","sha256_utf8":"4b18e73a7df85c66c8e7f3ddd0ebb4dfb271332f21dfb8970c28a46d068a9f5d"},{"start":51279,"end":52095,"start_line":575,"end_line":588,"raw_tex":"\\begin{Lemma}\\label{lemma KL}\nLet $\\set{s_k}$ be a nonnegative sequence in $\\R$, and let $\\alpha,\\beta$ be some positive constants. Suppose that $s_k\\rightarrow 0$ as $k\\to\\infty$ and that this sequence satisfies the condition\n\\begin{align*}\ns_k^\\alpha\\le\\beta(s_k-s_{k-1})\\;\\text{ for all }\\;k\\in\\N\\;\\text{ sufficiently large}.\n\\end{align*}\nThe following assertions hold:\n\\begin{enumerate}\n\\item[\\bf (i)] If $\\alpha\\in(0,1]$, then $s_{k+1}\\le\\brac{1-\\dfrac{1}{\\beta}}s_k$ for all $k\\in\\N$ sufficiently large, i.e., $\\set{s_k}$ converges linearly to $0$ with rank $1-\\dfrac{1}{\\beta}.$\n\\item[\\bf (ii)] If $\\alpha>1$, then there exists a number $\\varrho>0$ such that \n\\begin{align*}\n s_k\\le\\varrho k^{-\\frac{1}{\\alpha-1}}\\;\\text{ for all }\\;k\\in\\N\\;\\text{ sufficiently large}.\n\\end{align*}\n\\end{enumerate}\n\\end{Lemma}","sha256_utf8":"6937521b5b2a1039607d5d4b72a430f8ebf4caa960fa3975a2569ccd7032a41a"},{"start":52162,"end":53227,"start_line":592,"end_line":605,"raw_tex":"\\begin{Theorem}\\label{general rate}\nLet the sequences $\\set{x^k}\\subset\\R^n$, $\\set{t_k}\\subset\\R_+$ and the numbers $\\beta>0,\\;c>0$ be such that $x^{k+1}\\ne x^k$ for all $k\\in\\N$, and that we have\n\\begin{align}\\label{two conditions}\nf(x^k)-f(x^{k+1})\\ge \\dfrac{\\beta}{t_k}\\norm{x^{k+1}-x^k}^2\\;\\text{ and }\\;\\norm{\\nabla f(x^k)}\\le\\dfrac{c}{t_k}\\norm{x^{k+1}-x^k}\n\\end{align}\nfor sufficiently large $k\\in\\N$. Suppose that the sequence $\\set{t_k}$ is bounded away from $0$, that $\\bar x$ is an accumulation point of $\\set{x^k}$, and that $f$ satisfies the KL property at $\\bar x$ with $\\psi(t)=Mt^{q}$ for some $M>0$ and $q\\in(0,1)$. The following convergence rates are guaranteed:\n\\begin{itemize}\\vspace*{-0.05in}\n\\item[\\bf(i)] If $q\\in(0,1/2]$, then the sequence $\\set{x^k}$ converges linearly to $\\bar x$.\\vspace*{-0.05in}\n\\item[\\bf(ii)]If $q\\in(1/2,1)$, then there exists a positive constant $\\varrho$ such that \n\\begin{align*}\n\\norm{x^k-\\bar x}\\le \\varrho k^{-\\frac{1-q}{2q-1}}\\;\\text{ for sufficiently large }\\;k\\in\\N.\n\\end{align*}\n\\end{itemize}\n\\end{Theorem}","sha256_utf8":"07b0982c1c03173a4757472344788690aeac7db3e60bde360934cc2e04cd725e"},{"start":53245,"end":56475,"start_line":606,"end_line":651,"raw_tex":"\\begin{proof}\nThe conditions imposed in (\\ref{two conditions}) imply that\n\\begin{align*}\nf(x^k)-f(x^{k+1})\\ge \\dfrac{\\beta}{c}\\norm{\\nabla f(x^k)}\\cdot\\norm{x^{k+1}-x^k}\\;\\text{ for sufficiently large }\\;k\\in\\N,\n\\end{align*}\nwhich ensures that assumption (H1) in Proposition~\\ref{general convergence under KL} holds with $\\sigma:=\\dfrac{\\beta}{c}$. By\n$t_k>0$ for all $k\\in\\N$, the first condition in (\\ref{two conditions}) yields also assumption (H2) of this proposition. Let $k_1\\in\\N$ be such that \\eqref{kl induction} is satisfied for all $k\\ge k_1$, which implies in turn inequality \\eqref{sum series cauchy} and thus the convergence of $\\set{x^k}$ to $\\ox$. Moreover, the first condition in (\\ref{two conditions}) together with $x^{k+1}\\ne x^k$ as $k\\in\\N$ implies that \n\\begin{align}\\label{f(xk) converges}\nf(x^k)\\downarrow f(\\bar x):=\\bar f\\;\\text{ and }\\;f(x^k)>\\bar f\\;\\text{ for all }\\;k\\in\\N.\n\\end{align}\nIt follows from (\\ref{sum series cauchy}) and the triangle inequality that\n\\begin{align*}\n\\norm{x^i-\\bar x}\\le\\sum_{k=i}^\\infty\\norm{x^{k+1}-x^k}=:s_i<\\infty\\;\\text{ whenever }\\;i\\in\\N.\n\\end{align*}\nTherefore, the convergence rate of $x^i$ to $\\bar x$ can be deduced from the convergence rate of $s_i$ to $0$. Let $k_1$ be defined in (\\ref{kl induction}) and pick $i\\ge k_1$. Summing up (\\ref{kl induction}) from $i$ to any $p>i$ gives us\n\\begin{align*}\n\\sum_{k=i}^{p}\\norm{x^{k+1}-x^k}\\le\\dfrac{1}{\\sigma M(1-q)}\\big(f(x^i)-\\bar f\\big)^{1-q}.\n\\end{align*}\nLetting $p\\rightarrow\\infty$, we get the estimate\n\\begin{align}\\label{sikm}\ns_i\\le\\dfrac{1}{\\sigma M(1-q )}\\big(f(x^i)-\\bar f\\big)^{1-q}\\;\\text{ for all }\\;i\\ge k_1.\n\\end{align}\nThe KL property of $f$ at $\\bar x$ with $\\psi(t)=Mt^{q }$ allows us to find a positive number $\\eta$ and a neighborhood $U$ of $\\bar x$ ensuring the condition\n\\begin{align}\\label{KL M gradient}\n\\norm{\\nabla f(x)}\\ge M\\big(f(x)-\\bar f\\big)^q\\;\\mbox{ whenever }\\;x\\in U,\\;\\bar f<f(x)<\\bar f+\\eta.\n\\end{align} \nUsing (\\ref{KL M gradient}), $x^k\\rightarrow\\bar x$, and (\\ref{f(xk) converges}) gives us a natural number $k_2>k_1$ such that\n\\begin{align}\\label{M(1-theta)}\n\\norm{\\nabla f(x^i)}\\ge M\\big(f(x^i)-\\bar f\\big)^q\\;\\text{ for all }\\;i\\ge k_2.\n\\end{align}\nLet $\\bar{t}>0$ be a lower bound of $\\set{t_k}$. Combining this with (\\ref{M(1-theta)}) and the second inequality in (\\ref{two conditions}), we have the estimates\n\\begin{align}\n\\big(f(x^i)-\\bar f\\big)^q &\\le\\dfrac{1}{M}\\norm{\\nabla f(x^i)}\\le\\dfrac{c}{Mt_i}\\norm{x^{i+1}-x^i}\\nonumber\\\\\n&\\le\\dfrac{c}{M\\bar{t}}\\norm{x^{i+1}-x^i}\\;\\text{ for all }\\;i\\ge k_2.\\label{3.220}\n\\end{align}\nInvolving further (\\ref{sikm}) and (\\ref{3.220}) ensures that\n\\begin{align*}\ns_i^{\\frac{q}{1-q }}&\\le\\brac{\\dfrac{1}{\\sigma M(1-q)}}^{\\frac{q }{1-q}}\\big(f(x^i)-\\bar f\\big)^q\\\\\n&\\le\\brac{\\dfrac{1}{\\sigma M(1-q)}}^{\\frac{q }{1-q}}\\dfrac{c}{M\\bar{t}}\\norm{x^{i+1}-x^i}.\n\\end{align*}\nDefine $\\alpha:=\\dfrac{q }{1-q }$ and $\\beta:=\\brac{\\dfrac{1}{\\sigma M(1-q )}}^{\\frac{q }{1-q }}\\dfrac{c}{M\\bar{t}}$, and then rewrite the latter estimate as\n\\begin{align*}\ns_i^\\alpha\\le\\beta(s_i-s_{i-1})\\;\\text{ for all }\\;i\\ge k_2.\n\\end{align*}\nApplying finally Lemma~\\ref{lemma KL}, we verify the convergence rates in (i), (ii), and (iii). \n\\end{proof}","sha256_utf8":"3c21024f07b44a632286bf988eb6f29bc7b82311ff1a1856401027e49e0a1399"},{"start":73124,"end":73427,"start_line":846,"end_line":851,"raw_tex":"\\begin{Proposition}\\label{prop sufi holds}\nIn Algorithm~{\\rm\\ref{al gd inexact}}, $\\set{d^k}$ satisfies the sufficient descent condition with constant~$1,$ i.e., \n\\begin{align}\\label{sufi des holds}\n\\dotproduct{\\nabla f(x^k),d^k}\\le -\\norm{d^k}^2\\;\\text{ for all }\\;k\\in\\N.\n\\end{align}\n\\end{Proposition}","sha256_utf8":"d6fd56e527652171a66264eb9d459c67619025bd126205239fc45d8c857306d1"},{"start":73429,"end":74179,"start_line":853,"end_line":867,"raw_tex":"\\begin{proof}\nNote that (\\ref{sufi des holds}) automatically holds if $d^k=0$. Supposing that $d^k\\ne 0$ and using the construction of $d^k$ in Step~\\ref{Step 2}, we have the expression\n\\begin{align*}\n-d^k={\\rm Proj}\\big(0,\\mathbb{B}(g^k,\\varepsilon_k)\\big).\n\\end{align*}\nIt follows from (\\ref{calculate approx grad}) that $\\norm{g^k-\\nabla f(x^k)}\\le \\varepsilon_k$, which means that $\\nabla f(x^k)\\in \\mathbb{B}(g^k,\\varepsilon_k)$. Invoking the projection description for convex sets \\eqref{proj} yields\n\\begin{align*}\n\\dotproduct{0+d^k,\\nabla f(x^k)+d^k}\\le 0,\n\\end{align*}\nwhich is in turn equivalent to\n\\begin{align*}\n\\dotproduct{\\nabla f(x^k),d^k}\\le-\\norm{d^k}^2\n\\end{align*}\nand thus verifies the claim in \\eqref{sufi des holds}.\n\\end{proof}","sha256_utf8":"26b47bb2a93937ceba6cad7eece0e54c4edcb0b0857e58bc3b0f38d4c5ea5a36"},{"start":74671,"end":75848,"start_line":880,"end_line":895,"raw_tex":"\\begin{Proposition}\\label{null properties}\nLet $\\set{x^k},\\set{g^k},\\set{d^k},\\set{\\varepsilon_k}$, and $\\set{r_k}$ be sequences generated by Algorithm~{\\rm\\ref{al gd inexact}}. The following assertions hold:\n\\begin{itemize}\n\\item[\\bf(i)] $k\\in\\mathcal{N}$ if and only if either one of the equivalent conditions in \\eqref{alter condi for null} holds.\\vspace*{-0.05in}\n\n\\item[\\bf(ii)] $\\varepsilon_k\\downarrow 0$ if and only if $r_k\\downarrow 0$, which is equivalent to the set $\\mathcal{N}$ being infinite.\n\n\\item[\\bf(iii)] If $\\mathcal{N}$ is finite, then we have\n\\begin{align*}\n \\norm{g^k}>r_N+\\varepsilon_N\\;\\text{ and }\\;\\norm{d^k}>r_N\n\\end{align*}\nfor all $k\\ge N$, where $N:=\\max\\mathcal{N}+1$.\n\n\\item[\\bf(iv)] If $\\N\\setminus\\mathcal{N}$ is finite, then $\\nabla f(x^K)=0$ and $\\set{x^k}_{k\\ge K}$ is a constant sequence, where we denote $K:=\\max\\{\\N\\setminus\\mathcal{N}\\}+1$. Otherwise, $\\bar x$ is an accumulation point of $\\set{x^k}$ if and only if it is an accumulation point of $\\set{x^k}_{k\\in \\N\\setminus\\mathcal{N}}$, and therefore $x^k\\rightarrow\\bar x$ if and only if $x^k\\overset{\\N\\setminus\\mathcal{N}}{\\longrightarrow}\\bar x$.\n\\end{itemize}\n\\end{Proposition}","sha256_utf8":"c15c378533592dfa1592f6f747c3609e6590d5fac115299124fe478b3fb94e6f"},{"start":75849,"end":78328,"start_line":896,"end_line":921,"raw_tex":"\\begin{proof} Assertions (i) and (ii) follow directly from Definition~\\ref{nul iter}. To verify (iii), observe that for any natural number $k\\ge N$, the $k^{\\rm th}$ iteration is not a null one and then deduce from (i) that\n\\begin{align*}\n\\varepsilon_{k+1}=\\varepsilon_k=\\varepsilon_N,\\quad r_{k+1}=r_k=r_N,\\;\\mbox{ and }\\;\\norm{g^k}>r_k+\\varepsilon_k=r_N+\\varepsilon_N.\n\\end{align*}\nTogether with Step~\\ref{Step 2} in Algorithm~\\ref{al gd inexact}, this ensures that   \n\\begin{align*}\n\\norm{d^k}=\\norm{g^k}-\\varepsilon_k>r_k=r_N.\n\\end{align*}\nwhich readily justifies assertion (i).\n\nThe proof of (iv) is a bit more involved. Supposing that the set $\\N\\setminus\\mathcal{N}$ is finite, we have that $k\\in \\mathcal{N}$ for all $k\\ge K$. This means by (i) that\n\\begin{align*}\nx^{k+1}=x^{k},\\;\\varepsilon_{k+1}=\\theta\\varepsilon_k,\\;r_{k+1}=\\mu r_k,\\;\\text{ and }\\;\\norm{g^k}\\le r_k+\\varepsilon_k\\;\\text{ whenever }\\;k\\ge K.\n\\end{align*}\nThis tells us that $x^k=x^{K}$ for all $k\\ge K$, and that $\\varepsilon_k\\downarrow 0,\\;r_k\\downarrow 0$, and $g^k\\rightarrow 0$ as $k\\to\\infty$. Taking the limit as $k\\rightarrow \\infty$ in $\\norm{g^k-\\nabla f(x^k)}\\le \\varepsilon_k$ gives us $\\nabla f(x^k)\\rightarrow0$, which yields $\\nabla f(x^{K})=0$.\n\nSupposing otherwise that the set $\\N\\setminus\\mathcal{N}$ is infinite, we obviously get that every accumulation point of $\\set{x^k}_{k\\in \\N\\setminus \\mathcal{N}}$ is an accumulation point of $\\set{x^k}$. Conversely, taking any accumulation point $\\bar x$ of $\\set{ x^k}$, it suffices to show that \n\\begin{align*}\n\\mbox{for any }\\;\\delta>0,\\;N\\in\\N\\;\\mbox{ there exists }\\;k_N\\in\\N\\setminus\\mathcal{N},\\;k_N\\ge N\\;\\mbox{ with }\\;\\norm{x^{k_N}-\\bar x}<\\delta.\n\\end{align*}\nTo verify this, fixing $\\delta >0$ and $N\\in\\N$ and remembering that $\\bar x$ is an accumulation point of $\\set{x^k}$, we find $K\\ge N$ such that $\\norm{x^K-\\bar x}<\\delta$. If $K\\in\\N\\setminus\\mathcal{N}$, choose $k_N:=K$. Otherwise, using that $\\N\\setminus\\mathcal{N}$ is infinite allows us to find $\\widehat{K}\\in\\N\\setminus\\mathcal{N}$ for which $K,K+1,\\ldots,\\widehat{K}-1\\in\\mathcal{N}$. This ensures that \n\\begin{align*}\n{x^{\\widehat{K}}}={x^{\\widehat{K}-1}}=\\ldots=x^{K+1}=x^K,\n\\end{align*}\nand therefore, with $k_N:=\\widehat{K}$, we get that $\\norm{x^{k_N}-\\bar x}<\\delta$. Since $\\delta$ was chosen arbitrarily, this clearly shows that $\\bar x$ is an accumulation point of $\\set{x^k}_{k\\in\\N\\setminus\\mathcal{N}}$ and thus completes the proof. \n\\end{proof}","sha256_utf8":"781c37ea86e09d7e66c1f4b88d9c60f6cc516c7f67c2cc75bdc5dca1bc517e03"},{"start":78493,"end":79203,"start_line":925,"end_line":938,"raw_tex":"\\begin{Proposition}\\label{goes to 0} Let $\\set{g^k},\\set{d^k},\\;\\set{\\varepsilon_k}$, and $\\set{r_k}$ be sequences generated by Algorithm~{\\rm\\ref{al gd inexact}}. Then for any $k\\in\\N$ we have the estimates\n\\begin{align}\\label{nx1}\n\\norm{d^k}\\le\\norm{g^k}\\le\\norm{d^k}+\\varepsilon_k+r_k.\n\\end{align}\nConsequently, the following assertions hold:\n\\begin{itemize}\n\\item[\\bf(i)] $\\varepsilon_k\\downarrow0$ if and only if there is an infinite set $J\\subset\\N$ such that $g^k\\overset{J}{\\rightarrow}0$.\n\n\\item[\\bf(ii)] For any infinite set $J\\subset\\N$, we have the equivalence\n\\begin{align*}\ng^k\\overset{J}{\\rightarrow}0\\Longleftrightarrow d^k\\overset{J}{\\rightarrow}0.\n\\end{align*}\n\\end{itemize}\n\\end{Proposition}","sha256_utf8":"c70d94cc40ad9e013a3d95ddc3a5a61a6d0176bbcfe5644621fc5578e3fd985e"},{"start":79205,"end":80951,"start_line":940,"end_line":950,"raw_tex":"\\begin{proof}\nFix any $k\\in\\N$. If $k\\in\\mathcal{N}$, then we get by Proposition~\\ref{null properties}(i) that $d^k=0$ and $\\norm{g^k}\\le \\varepsilon_k+r_k$. Otherwise, Step~\\ref{Step 2} in Algorithm~\\ref{al gd inexact} yields $\\norm{d^k}=\\norm{g^k}-\\varepsilon_k\\le \\norm{g^k}$. In both cases, (\\ref{nx1}) holds. \n\nTo deduce (i) from (\\ref{nx1}), suppose that $\\varepsilon_k\\downarrow 0$ as $k\\to\\infty$. By Proposition~\\ref{null properties}(ii) we have that $r_k\\downarrow 0$ and the set $\\mathcal{N}$ is infinite. Then for any $\\delta>0$ and $N\\in\\N$ there is $k\\ge N$ with $k\\in\\mathcal{N}$ such that\n\\begin{align*}\n\\norm{g^{k}}\\le r_k+\\varepsilon_k<\\delta,\n\\end{align*}\nwhere the first inequality follows from (\\ref{alter condi for null}). Thus we can construct an infinite set $J\\subset\\N$ such that $g^k\\overset{J}{\\longrightarrow}0$. If conversely the sequence $\\set{\\varepsilon_k}$ does not converge to $0$, then the set $\\mathcal{N}$ is finite by Proposition~\\ref{null properties}(ii). Using Proposition~\\ref{null properties}(iii) confirms that $\\set{g^k}$ is bounded away from $0$, which tells us that such an index set $J$ does not exist.\n\nTo verify now assertion (ii), observe that assuming $g^k\\overset{J}{\\rightarrow}0$, implies by the first inequality in (\\ref{nx1}) that $d^k\\overset{J}{\\rightarrow}0$. Conversely, suppose that $d^k\\overset{J}{\\rightarrow}0$ and deduce from Proposition~\\ref{null properties}(i) that the set $\\mathcal{N}$ is infinite. Then it follows from Proposition~\\ref{null properties}(ii) that  $\\varepsilon_k\\downarrow0$ and $r_k\\downarrow0$ as $k\\to\\infty$. Using the second inequality in (\\ref{nx1}), we arrive at $g^k\\overset{J}{\\rightarrow}0$ and thus complete the proof of the proposition.\n\\end{proof}","sha256_utf8":"28579b488121af6df9adebd23c31f767c896f9316ca64a9a566e7ca20e8fdbe1"},{"start":81120,"end":81273,"start_line":954,"end_line":955,"raw_tex":"\\begin{Corollary}\\label{overall} The sequence $\\set{d^k}$ in Algorithms~{\\rm\\ref{al gd inexact}} is gradient associated with $\\set{x^k}$.\n\\end{Corollary}","sha256_utf8":"f11320b4b23128716b51070c7e942e73edbb483745e31a3443f097719f4d7682"},{"start":81290,"end":81730,"start_line":956,"end_line":957,"raw_tex":"\\begin{proof} It follows from Proposition \\ref{goes to 0} that the convergence $d^k\\overset{J}{\\rightarrow}0$ yields $g^k\\overset{J}{\\rightarrow}0$ and $\\varepsilon_k\\downarrow0.$ Thus we get  $\\nabla f(x^k)\\overset{J}{\\rightarrow}0$ by taking into account $\\norm{g^k-\\nabla f(x^k)}\\le \\varepsilon_k$ from \\eqref{calculate approx grad}. This shows therefore that the sequence $\\set{d^k}$ is gradient associated with $\\set{x^k}$.\n\\end{proof}","sha256_utf8":"c9116f4f7ce8e4419745ca50e9c60c11f332232cc5f79d1fc78b5952d7a5b8f2"},{"start":82645,"end":82964,"start_line":968,"end_line":973,"raw_tex":"\\begin{Proposition}\\label{descent} If $d^k\\ne 0$, then there exists a positive number $\\bar t$ such that \n\\begin{align*}\nf(x^k+td^k)\\le f(x^k)-\\beta t\\big\\|d^k\\big\\|^2\\;\\text{ for all }\\;t\\in [0,\\bar t],\n\\end{align*}\nwhich ensures the existence of the stepsize sequence $\\{t_k\\}$ in \\eqref{defi: tk}. \n\\end{Proposition}","sha256_utf8":"1ae02f1261fdb40f0b326d2d3d54f449e972adc8a47c898f7dca29f2eb55e031"},{"start":82965,"end":83677,"start_line":974,"end_line":985,"raw_tex":"\\begin{proof} Suppose that $d^k\\ne 0$ and get by Proposition~\\ref{prop sufi holds} that $\\dotproduct{\\nabla f(x^k),d^k}\\le-\\norm{d^k}^2$. By the differentiability of $f$ at $x^k$, for each $t>0$ sufficiently small we have\n\\begin{align*}\nf(x^k+td^k)-f(x^k)&=t\\dotproduct{\\nabla f(x^k),d^k}+o(t)\\le-t\\norm{d^k}^2+o(t)\\\\\n&=-\\beta t\\norm{d^k}^2+t\\brac{\n(\\beta-1)\\norm{d^k}^2+ \\dfrac{o(t)}{t}}.\n\\end{align*}\nSince $o(t)/t\\rightarrow0$ as $t\\dn 0$ and since $(\\beta-1)\\norm{d^k}^2<0$, there exists $\\bar t>0$ such that \n\\begin{align*}\nf(x^k+td^k)\\le f(x^k)-\\beta t\\norm{d^k}^2\\;\\text{ for all }\\;t\\in(0,\\bar t].\n\\end{align*}\nTherefore, all the numbers $t_k$ introduced in (\\ref{defi: tk}) are well-defined.\n\\end{proof}","sha256_utf8":"4482283482f942bf008e25418ff78356720b226c980c135ec887cb54d53fe6bd"},{"start":83881,"end":84709,"start_line":989,"end_line":1000,"raw_tex":"\\begin{Theorem}\\label{convergence 1}\nLet $\\set{x^k}$ be the sequence of iterations generated by Algorithm~{\\rm\\ref{al gd inexact}} with the stepsize sequences $t_k$ from \\eqref{defi: tk} of the backtracking line search. Assume that $\\inf f(x^k)>-\\infty$ and that $\\rho_k\\rightarrow0$ as $k\\rightarrow\\infty$. Then the following assertions hold:\n\\begin{itemize}\n\\item[\\bf(i)] $\\varepsilon_k\\downarrow0$ and $r_k\\downarrow 0$ as $k\\to\\infty$.\n\n\\item[\\bf(ii)] Every accumulation point of $\\set{x^k}$ is a stationary point of $f$.\n\n\\item[\\bf(iii)] If the sequence $\\set{x^k}$ is bounded, then the set of accumulation points of $\\set{x^k}$ is nonempty, compact, and connected.\n\n\\item[\\bf(iv)] If $\\set{x^k}$ has an isolated accumulation point, then the entire sequence $\\set{x^k}$ converges to this point.\n\\end{itemize}\n\\end{Theorem}","sha256_utf8":"e8a7c1869e0d032b33f347089b052eca858dcac4abf8430a78963d3ebc68cb25"},{"start":84710,"end":88520,"start_line":1001,"end_line":1043,"raw_tex":"\\begin{proof} From the choice of $\\set{t_k}$ and Step~4 in Algorithm~\\ref{al gd inexact}, for every $k\\in\\N$ we have\n\\begin{align}\\label{tk dk}\n\\beta t_k\\norm{d^k}^2\\le f(x^k)-f(x^{k+1}).\n\\end{align}\nSince $\\inf f(x^k)>-\\infty$, summing up  on both sides of (\\ref{tk dk}) over $k=1,2,\\ldots$ and using the relation $x^{k+1}=x^k+t_kd^k$, we get that\n\\begin{align}\\label{taking sum 1}\n\\sum_{k=1}^\\infty t_k\\norm{d^k}^2<\\infty\\quad\\text{ and }\\quad \\sum_{k=1}^\\infty\\norm{x^{k+1}-x^k}\\cdot\\norm{d^k}<\\infty.\n\\end{align}\n\nTo verify assertion (i), recall by Proposition~\\ref{null properties}(ii) that the convergence $\\varepsilon_k\\downarrow0$ is equivalent to $r_k\\downarrow 0$ and to the fact that the set of null iterations $\\mathcal{N}$ is infinite. Assume on the contrary that $\\mathcal{N}$ is finite. By Proposition~\\ref{null properties}(iii) with $N=\\max\\mathcal{N}+1$, we have\n\\begin{align}\\label{dk bounded}\n\\norm{g^k}>r_N+\\varepsilon_N\\;\\text{ and }\\;\\norm{d^k}>  r_N\\;\\text{ for all }\\;k\\ge N.\n\\end{align}\nThen (\\ref{taking sum 1}) gives us $\\sum_{k=1}^\\infty t_k<\\infty$ and thus $t_k\\downarrow0$ as $k\\to\\infty$. Choosing a larger number $N$ if necessary, we get that $t_k<1$ for all $k\\ge N$. For such $k$, it follows from the exit condition of the algorithm that\n\\begin{align}\\label{exit inexact}\n-\\gamma^{-1}\\beta t_k\\norm{d^k}^2< f(x^{k}+\\gamma^{-1}t_kd^k)-f(x^k).\n\\end{align}\nBy the classical mean value theorem, there exists some $\\tilde{x}^k\\in[x^k,x^k+\\gamma^{-1}t_kd^k]$ such that \n\\begin{align*}\nf(x^{k}+\\gamma^{-1}t_kd^k)-f(x^k)= \\gamma^{-1}t_k\\dotproduct{d^k,\\nabla f(\\tilde{x}^k)}.\n\\end{align*}\nCombining the latter with (\\ref{exit inexact}) tells us that\n\\begin{align}\\label{contra 1}\n\\dotproduct{-d^k,\\nabla f(\\tilde{x}^k)}\\le\\beta \\norm{d^k}^2\\;\\text{ for all }\\;k\\ge N.\n\\end{align}\nUsing (\\ref{taking sum 1}) and (\\ref{dk bounded}), we have $\\sum_{k=1}^\\infty\\norm{x^{k+1}-x^k}<\\infty$, and thus $\\set{x^k}$ converges to some $\\bar x\\in\\R^n$. The continuity of $\\nabla f$ ensures that $\\nabla f(x^k)\\rightarrow\\nabla f(\\bar x)$. Then employing $\\norm{g^k-\\nabla f(x^k)}\\le \\rho_k\\rightarrow0$ yields $g^k\\rightarrow\\nabla f(\\bar x)$ as $k\\rightarrow\\infty$.\nIt follows from Step~\\ref{Step 2} that\n\\begin{align*}\n-d^k&=\\dfrac{\\norm{g^k}-\\varepsilon_k}{\\norm{g^k}}g^k=\\dfrac{\\norm{g^k}-\\varepsilon_N}{\\norm{g^k}}g^k\\;\\text{ for all \\;}k\\ge N.\n\\end{align*}\nLetting $k\\rightarrow\\infty$ leads us to the equalities\n\\begin{align}\\label{1.8 1}\n-d^k\\rightarrow\\bar g:=\\dfrac{\\norm{\\nabla f(\\bar x)}- \\varepsilon_N }{\\norm{\\nabla f(\\bar x)}}\\nabla f(\\bar x)={\\rm Proj}\\big(0,\\mathbb{B}(\\nabla f(\\bar x),\\varepsilon_N)\\big).\n\\end{align}\nUsing $t_k\\downarrow0$, we get that $\\tilde{x}^k\\rightarrow\\bar x$, and thus $\\nabla f(\\tilde{x}^k)\\rightarrow\\nabla f(\\bar x)$ as $k\\to\\infty$. Combining the latter with (\\ref{contra 1}), (\\ref{1.8 1}), and the projection characterization in \\eqref{proj} verifies the estimates\n\\begin{align}\n\\norm{\\bar g}^2\\le\\dotproduct{\\bar g,\\nabla f(\\bar x)}\\le\\beta \\norm{\\bar g}^2.\n\\end{align}\nThis tells us that $\\bar g=0$, which contradicts the condition $\\norm{\\bar g}\\ge r_N$ by (\\ref{dk bounded}). Therefore, we arrive at $\\varepsilon_k\\downarrow 0$ and $r_k\\downarrow 0$ as $k\\to\\infty$, which completes the proof of assertion (i).\n\nTo justify assertions (ii)--(iv), recall from Corollary~\\ref{overall} that $\\set{d^k}$ is \\textit{gradient associated} with $\\set{x^k}$. Since $\\varepsilon_k\\downarrow 0$, we deduce from Proposition~\\ref{goes to 0} that \n$0$ is an accumulation point of $\\set{d^k}$. Combining these facts with (\\ref{taking sum 1}) and $t_k\\le 1$ whenever $k\\in\\N$ ensures that all the assumptions of Theorem~\\ref{stationary point theorem} are satisfied. Therefore, we verify assertions (ii)--(iv) and finish the proof of the theorem.\n\\end{proof}","sha256_utf8":"16dd8afb57712273a7aba7efa9829b08584d02555a047c7a5acaba4507c15e27"},{"start":91370,"end":91843,"start_line":1073,"end_line":1079,"raw_tex":"\\begin{Lemma}\\label{gradien 3 dk}\nLet $\\set{x^k}$ be the sequence generated by Algorithm~{\\rm\\ref{al gd inexact}} with $\\theta<\\mu$. Assume that  $\\varepsilon_k\\downarrow 0$ and $r_k\\downarrow0$ as $k\\to\\infty$. Then there exists some $N\\in\\N$ such that\n\\begin{align}\\label{equation grad 3k}\n\\norm{\\nabla f(x^k)}\\le 3\\norm{d^k}\\;\\text{ for all }\\;k\\notin\\mathcal{N},\\;k\\ge N,\n\\end{align}\nwhere the set $\\mathcal{N}$ is taken from Definition~{\\rm\\ref{nul iter}}.\n\\end{Lemma}","sha256_utf8":"1091dc743d62a4bb0a50b246a9136a05aead4853df98fa7e0d9edf9e19ba1ac8"},{"start":91844,"end":92612,"start_line":1080,"end_line":1089,"raw_tex":"\\begin{proof} It follows directly from the assumptions of the lemma that there exists a natural number $N$ such that $\\varepsilon_k\\le r_k$ for all $k\\ge N$. By Step~\\ref{Step 2} of the algorithm, for any $k\\ge N$ with $k\\notin\\mathcal{N}$ we have $\\norm{g^k}>r_k+\\varepsilon_k$ with the direction $d^k$ calculated in (\\ref{choose dk}). Thus for such $k$ we get the estimates\n\\begin{align}\\label{66}\n\\norm{d^k}=\\norm{g^k}-\\varepsilon_k>r_k+\\varepsilon_k-\\varepsilon_k= r_k\\ge\\varepsilon_k.\n\\end{align}\nIt follows from (\\ref{calculate approx grad}) in Step~\\ref{Step 1} and from (\\ref{66}) that\n\\begin{align*}\n\\norm{\\nabla f(x^k)}\\le \\norm{g^k}+\\varepsilon_k=\\norm{d^k}+2\\varepsilon_k\\le 3\\norm{d^k},\n\\end{align*}\nwhich verifies the conclusion of the lemma.\n\\end{proof}","sha256_utf8":"749341fd30c70222c28ec6b4f4fe539b1b2bcc5dc88b02cc93f4295d40ca747e"},{"start":92905,"end":93358,"start_line":1093,"end_line":1095,"raw_tex":"\\begin{Theorem}\\label{convergence under KL}\nLet $\\set{x^k}$ be the iterative sequence generated by Algorithm~{\\rm\\ref{al gd inexact}} with the backtracking linesearch such that $\\theta<\\mu$. Assume that $\\rho_k\\rightarrow0$ as $k\\rightarrow\\infty$, $\\set{x^k}$ has an accumulation point $\\bar x$, and $f$ satisfies the KL property at $\\bar x$. Then $\\bar x$ is a stationary point of $f$, and we have $x^k\\rightarrow\\bar x$ as $k\\to\\infty$.\n\\end{Theorem}","sha256_utf8":"fa9c92dc7a4e2ee1ad6573ce8aaed81263026d011334ee220213084de5e08850"},{"start":93359,"end":95261,"start_line":1096,"end_line":1111,"raw_tex":"\\begin{proof}\nSince $\\bar x$ is an accumulation point of $\\set{x^k}$, we can find some infinite set $J\\subset\\N$ such that $x^k\\overset{J}{\\rightarrow}\\bar x$. It follows from the choice of $\\set{t_k}$ in \\eqref{defi: tk} that $\\set{f(x^k)}$ is nonincreasing, which implies that \n\\begin{align*}\n\\inf_{k\\in \\N} f(x^k)= \\inf_{k\\in J} f(x^k)=f(\\bar x)>-\\infty.\n\\end{align*}\nTherefore, the results of Theorem~\\ref{convergence 1} tell us that $\\bar x$ is a stationary point of $f$ and that $\\varepsilon_k\\dn 0$, $r_k\\downarrow 0$ as $k\\to\\infty$. We employ Proposition~\\ref{general convergence under KL} to verify that $x^k\\rightarrow\\bar x$ along the entire sequence of iterations. Indeed, the imposed assumptions and the convergence $\\varepsilon_k\\dn 0$, $r_k\\downarrow 0$ as $k\\to\\infty$ guarantee that all the requirements  of Lemma~\\ref{gradien 3 dk} are satisfied. Pick $N\\in\\N$ such that (\\ref{equation grad 3k}) holds. The choice of $\\set{t_k}$ in (\\ref{defi: tk}) ensures the lower estimate\n\\begin{align}\\label{5.133}\nf(x^{k})-f(x^{k+1})&\\ge\\beta t_k\\big\\|d^k\\big\\|^2\\;\\text{ for all }\\;k\\in\\N.\n\\end{align}\nCombining this with (\\ref{equation grad 3k}) and the relation $x^{k+1}=x^k+t_kd^k$ yields\n\\begin{align}\\label{5.13}\nf(x^{k})-f(x^{k+1})\\ge\\dfrac{\\beta}{3}\\norm{\\nabla f(x^k)}\\cdot\\norm{x^{k+1}-x^k}\\;\\text{ for all }\\;k\\notin\\mathcal{N},\\;k\\ge N.\n\\end{align}\nObserve that when $k\\in\\mathcal{N}$, both sides of\n(\\ref{5.13}) reduce to zero, and so (\\ref{5.13}) is satisfied. Therefore, assumption (H1) in Proposition~\\ref{general convergence under KL} holds. Moreover, for $k\\ge N$ the conditions $f(x^{k+1})=f(x^k)$ and (\\ref{5.133}) imply that $d^k=0$, and hence $x^{k+1}=x^k$. Thus assumption (H2) in  Proposition~\\ref{general convergence under KL} is satisfied as well. Applying the latter proposition, we arrive at $x^k\\rightarrow\\bar x$ as $k\\to\\infty$ and complete the proof.\n\\end{proof}","sha256_utf8":"c51f235bc5207fca8b6e0105d1cfbf5545617517f1a52b3f1ce5eae37f27af7e"},{"start":98338,"end":98820,"start_line":1132,"end_line":1134,"raw_tex":"\\begin{Proposition}\\label{bounded away}\nLet $\\set{x^k}$ be a sequence generated by Algorithm~{\\rm\\ref{al gd inexact}} with the backtracking stepsize. Suppose that $\\rho_k\\rightarrow0$ as $k\\rightarrow\\infty$ and that there exists an infinite set $J\\subset\\N$ such that $\\set{x^k}_{k\\in J}$ converges to some $\\bar x$. Assume further that $\\nabla f$ is locally Lipschitzian around $\\bar x$. Then the stepsize sequence $\\set{t_k}_{k\\in J}$ is bounded away from zero.\n\\end{Proposition}","sha256_utf8":"4cee4f43d9be017fb2bfc2699664acb681291a4aea7926e4034113d3b20c4949"},{"start":98822,"end":102454,"start_line":1136,"end_line":1175,"raw_tex":"\\begin{proof}\n Assume on the contrary that $\\set{t_k}_{k\\in J}$ is not bounded away from zero. Then we find an infinite set $\\bar J\\subset J$ such that $t_k\\overset{\\bar J}{\\longrightarrow}0$. Let $\\tau\\in(0,1)$ be an artificial stepsize of null iterations. Since $t_k\\overset{\\bar J}{\\longrightarrow}0$, there exists a number $N\\in\\N$ such that \n \\begin{align}\\label{tk < tau}\nt_k<\\tau<1\\;\\text{ for all }\\;k\\ge N,\\;k\\in\\bar J.\n \\end{align}\n This means that $k\\notin\\mathcal{N}$ whenever $k\\ge N,\\;k\\in\\bar J$. By Proposition~\\ref{null properties}(i), we have $d^k\\ne 0$ for all $k\\ge N,\\;k\\in\\bar J$. Then condition (\\ref{choose dk}) in Step~\\ref{Step 2} leads us to\n\\begin{align}\\label{norm dk < norm gk}\n\\norm{d^k}=\\norm{g^k}-\\varepsilon_k\\le\\norm{g^k}\\;\\mbox{ for all }\\;k\\ge N,\\;k\\in\\bar J.\n\\end{align}\nSince $x^k\\overset{\\bar J}{\\rightarrow}\\bar x$, the continuity of $\\nabla f$ and the estimate $\\norm{\\nabla f(x^k)-g^k}\\le \\rho_k\\rightarrow0$ yield that $g^k\\overset{\\bar J}{\\rightarrow}\\nabla f(\\bar x)$. Using (\\ref{norm dk < norm gk}), we get that the sequence $\\set{d^k}_{k\\in\\bar J}$ is bounded, and thus \n\\begin{align}\\label{bound of norm dk}\nx^k+\\gamma^{-1} t_kd^k\\rightarrow\\bar x\\;\\mbox{ as }\\;k\\overset{\\bar J}{\\rightarrow}\\infty.\n\\end{align}\nSince $\\nabla f$ is locally Lipschitzian around $\\bar x$, there exists a positive number $\\delta$ such that $\\nabla f$ is Lipschitz continuous on $\\mathbb{B}(\\bar x,\\delta)$ with some modulus $L>0$. By (\\ref{bound of norm dk}) and $x^k\\overset{\\bar J}{\\rightarrow}\\bar x$, we find $N_1\\ge N$ with $x^k,x^k+\\gamma^{-1}t_kd^k\\in\\mathbb{B}(\\bar x,\\delta)$ for all $k\\ge N_1,\\;k\\in\\bar J$. The Lipschitz continuity of $\\nabla f$ on $\\mathbb{B}(\\bar x,\\delta)$ with modulus $L$ yields by \\cite[Proposition~A.24]{bertsekasbook} the $L$-descent condition, i.e.,\n\\begin{align}\\label{lipschitz}\nf(x)-f(y)\\le\\dotproduct{\\nabla f(y),x-y}+\\dfrac{L}{2}\\norm{x-y}^2\\;\\text{ for all }\\;x,y\\in \\mathbb{B}(\\bar x,\\delta).\n\\end{align}\nFixing $k\\in\\bar J,\\;k\\ge N_1$, we deduce from the above that $d^k\\ne 0$, and $t_k<1$. The exit condition for the backtracking line search implies that\n\\begin{align}\\label{exit 2}\n-\\gamma^{-1}\\beta t_k\\norm{d^k}^2<f(x^{k}+\\gamma^{-1}t_kd^k)-f(x^k).\n\\end{align}\nApplying (\\ref{lipschitz}) for $x=x^{k}+\\gamma^{-1}t_kd^k$ and $y=x^k$ we have that\n\\begin{align*}\nf(x^{k}+\\gamma^{-1}t_kd^k)-f(x^k)\\le\\gamma^{-1}t_k\\dotproduct{\\nabla f(x^k),d^k}+\\dfrac{L\\gamma^{-2}t_k^2}{2}\\norm{d^k}^2.\n\\end{align*}\nCombining this with (\\ref{exit 2}) leads us to \n\\begin{align*}\n-\\gamma^{-1}\\beta t_k\\norm{d^k}^2<\\gamma^{-1}t_k\\dotproduct{\\nabla f(x^k),d^k}+\\dfrac{L\\gamma^{-2}t_k^2}{2}\\norm{d^k}^2,\n\\end{align*}\nor equivalently to the inequality \n\\begin{align}\\label{0< gamma-1}\n0<\\gamma^{-1}\\beta t_k\\norm{d^k}^2+\\gamma^{-1}t_k\\dotproduct{\\nabla f(x^k),d^k}+\\dfrac{L\\gamma^{-2}t_k^2}{2}\\norm{d^k}^2.\n\\end{align}\nProposition~\\ref{prop sufi holds} and $d^k\\ne 0$ tell us that $0<\\norm{d^k}^2\\le\\dotproduct{\\nabla f(x^k),-d^k}$. Then we deduce from (\\ref{0< gamma-1}) the fulfillment of the estimate\n\\begin{align*}\n0<\\gamma^{-1}\\beta t_k\\dotproduct{\\nabla f(x^k),-d^k}+\\gamma^{-1}t_k\\dotproduct{\\nabla f(x^k),d^k}+\\dfrac{L\\gamma^{-2}t_k^2}{2}\\dotproduct{\\nabla f(x^k),-d^k}.\n\\end{align*}\nDividing both sides above by $\\gamma^{-1}t_k\\dotproduct{\\nabla f(x^k),-d^k}>0$, we get $0<\\beta-1+\\dfrac{L\\gamma^{-1}t_k}{2}$.\nLetting $k\\overset{\\bar J}{\\longrightarrow}\\infty$ yields $\\beta\\ge 1$, which contradicts the choice of $\\beta\\in(0,1)$. Thus we verify that the sequence $\\set{t_k}_{k\\in J}$ is bounded away from zero, which completes the proof of the proposition.\n\\end{proof}","sha256_utf8":"8e0162ade836a97b78f4eafbef68ae0436508156a0d4b3144f8473ada7c3b40f"},{"start":103096,"end":103779,"start_line":1181,"end_line":1189,"raw_tex":"We have $j_{k+1}\\ge j_{k}+1$ whenever $k\\in\\N$. If the equality holds therein, then $z^{k+1}=z^{j_k+1}$. Otherwise, by taking into account that the indices $j_{k}+1,\\ldots,j_{k+1}-1$ correspond to null iterations, we get that\n\\begin{align}\\label{5 23}\nx^{j_k+1}=x^{j_{k}+2}=\\ldots=x^{j_{k+1}-1}=x^{j_{k+1}}=z^{k+1}.\n\\end{align}\nTherefore, it follows from $j_k\\notin\\mathcal{N}$ that\n\\begin{align}\\label{differ}\nz^{k+1}=x^{j_k+1}\\ne x^{j_k}=z^k\\;\\text{ for all }\\;k\\in\\N.\n\\end{align}\nFurthermore, Proposition~\\ref{null properties}(iv) tells us that $\\bar x$ is an accumulation point of $\\set{z^k}$ if and only if $\\bar x$ is also an accumulation point of $\\set{x^k}$.\\vspace*{0.05in}\n","sha256_utf8":"2ec670912505ddf4467585c9bd36df150aea29e9d7d4eb9d0b8d25423216705a"},{"start":104858,"end":106452,"start_line":1203,"end_line":1216,"raw_tex":"\\begin{proof}\nThe imposed assumptions yield the fulfillment of those in Theorem~\\ref{convergence under KL}, and so lead us to the convergence $x^k\\rightarrow\\bar x$ as $k\\to\\infty$. Then the local Lipschitz continuity of $\\nabla f$ around $\\bar x$ and Proposition~\\ref{bounded away} ensure that the sequence $\\set{t_k}$ is bounded away from zero.\n \n To deduce now the claimed convergence rates in (i)--(iii) from Theorem~\\ref{general rate}, define $\\tau_k:=t_{j_k}$ for all $k\\in\\N$. Then $\\set{\\tau_k}$ is also bounded away from zero as a subsequence of $\\set{t_k}$. Furthermore, using (\\ref{5 23}) and the line search conditions, we have\n\\begin{align}\\label{beta tau k}\n f(z^k)-f(z^{k+1})&=f(x^{j_{k}})-f(x^{j_{k}+1})\\ge\\beta t_{j_k}\\norm{d^{j_k}}^2\\nonumber\\\\\n &=\\dfrac{\\beta}{t_{j_k}}\\norm{x^{j_k+1}-x^{j_k}}^2=\\dfrac{\\beta}{\\tau_k}\\norm{z^{k+1}-z^k}^2\n\\end{align}\nfor all $k\\in\\N$. Note that all the assumptions of Theorem~\\ref{convergence 1} are satisfied, and so Lemma~\\ref{gradien 3 dk} holds. Pick any $N\\in\\N$ from (\\ref{equation grad 3k}) and fix $k\\ge N$. Then using (\\ref{5 23}) and (\\ref{equation grad 3k}) with taking into account that $j_k\\notin\\mathcal{N}$ for $j_k\\ge k$ leads us to\n\\begin{align*}\n\\norm{\\nabla f(z^k)}=\\norm{\\nabla f(x^{j_k})}&\\le \\dfrac{3}{t_{j_k}}\\norm{x^{j_k+1}-x^{j_k}}=\\dfrac{3}{\\tau_k}\\norm{z^{k+1}-z^k}.\n\\end{align*}\nApply finally Theorem~\\ref{general rate} to $\\set{z^k}$ and $\\set{\\tau_k}$ while remembering that $z^{k+1}\\ne z^k$ for all $k\\in\\N$ from (\\ref{differ}). This verifies the convergence rates (i)--(iii) claimed in the theorem.\n\\end{proof}","sha256_utf8":"ab8d997d15cf8c61d4ef775f9af797cdb4054dcf2e116d5e6016d9f5ffefdef1"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"光滑无约束优化","source_units":[{"start":6028,"end":6131,"start_line":153,"end_line":155,"raw_tex":"\\begin{align}\\label{optim prob}\n{\\rm minimize}\\quad f(x)\\quad\\text{ subject to }\\; x\\in\\R^n\n\\end{align}","sha256_utf8":"047de3103461676b3488a0933f586b78b7aa37c9446dbddc97f622a7941ddab4"},{"start":6132,"end":6525,"start_line":156,"end_line":156,"raw_tex":"with a continuously differentiable ($\\mathcal{C}^1$-smooth) objective function $f:\\R^n\\rightarrow\\R$. One of the most natural and classical approaches to solve (\\ref{optim prob}) is by using \\textit{linesearch methods}; see, e.g., \\cite{beck,bertsekasbook,solodovbook,nesterovbook18,nocedalbook,polyakbook}. Given a starting point $x^1\\in\\R^n$, such methods construct the iterative procedure \n","sha256_utf8":"daf78c9aa64028b3da3381636da6791463bd216612f254d31026f8c79fa977fd"}]},{"id":"S002","semantic_type":"notation","kind":"notation","title":"空间与基本记号","source_units":[{"start":19953,"end":20668,"start_line":212,"end_line":212,"raw_tex":"First we recall some notions and notation frequently used in what follows. All our considerations are given in the space $\\R^n$ with the Euclidean norm $\\|\\cdot\\|$ and scalar/inner product $\\dotproduct{\\cdot,\\cdot}$. We use $\\N:=\\{1,2,\\ldots\\}$, $\\R_+$, and $\\overline{\\R}:=\\R\\cup\\set{\\infty}$ to denote the collections of natural numbers, positive numbers, and the extended real line, respectively. The symbol $x^k\\st{J}{\\to}\\ox$ means that $x^k\\to\\ox$ as $k\\to\\infty$ with $k\\in J\\subset\\N$. For a $\\mathcal{C}^1$-smooth function  $f:\\R^n\\rightarrow\\R$, $\\bar x$ is a \\textit{stationary point} of $f$ if $\\nabla f(\\bar x)=0$. The function $f$ is said to satisfy the {\\em $L$-descent condition} with some $L>0$ if\n","sha256_utf8":"c082eac9b0f329093f10da397df04fea128d3ada3a0c7f2c12dae33282846e70"},{"start":57562,"end":58128,"start_line":664,"end_line":664,"raw_tex":"Although the focus of this paper is on smooth optimization problems, we first recall the {\\em perturbed} notions of generalized differentiation for nonsmooth functions and their smooth specifications employed in \\cite{blo02,kiwiel07} as well as different ones used in our algorithms. In what follows, the notation $\\B(\\ox,\\ve)$ stands for the closed ball centered at $\\ox\\in\\R^n$ with radius $\\ve>0$, by ${\\rm co}\\,\\O$ we understand the convex hull of a set $\\O\\subset\\R^n$, and the symbol $x\\st{\\O}{\\to}\\ox$ indicates that $x\\to\\ox$ with $x\\in\\O$.\\vspace*{0.05in} \n","sha256_utf8":"8dd195c28493b4450743f2d80cb10800dfab4056431c7c297dd05e649532d0c8"}]},{"id":"S003","semantic_type":"notation","kind":"notation","title":"线搜索迭代关系","source_units":[{"start":6525,"end":6640,"start_line":157,"end_line":159,"raw_tex":"\\begin{align}\\label{intro: stepsize + descent direction}\nx^{k+1}:=x^k+t_kd^k\\;\\text{ for all }\\;k\\in\\N,\n\\end{align}","sha256_utf8":"a898dfa23e7aed8435036c28cd8b1cbcdb87726c15e8ac3b922df6b4500a51f2"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"二次下降条件","source_units":[{"start":20668,"end":20816,"start_line":213,"end_line":215,"raw_tex":"\\begin{align}\\label{descent condition}\nf(y)\\le f(x)+\\dotproduct{\\nabla f(x),y-x}+\\dfrac{L}{2}\\norm{y-x}^2\\;\\text{ for all }\\;x,y\\in\\R^n.\n\\end{align}","sha256_utf8":"606ee03f4aff56324f9609027ddfd6c0116d726d5950de3abe6f45b4d4c50bb5"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"梯度关联方向","source_units":[{"start":25540,"end":25989,"start_line":263,"end_line":268,"raw_tex":"\\begin{Definition}\\rm\\label{defi: approximate gradient}\nLet $\\set{x^k}$ be a sequence in $\\R^n$. The direction sequence $\\set{d^k}$ is called \\textit{gradient associated} with $\\set{x^k}$ if we have the implication\n\\begin{align}\\label{condition for stationarity}\n\\text{ whenever }\\;d^k\\overset{J}{\\rightarrow}0\\;\\text{ for some infinite set }\\;J\\subset\\N,\\;\\text{ it holds that }\\;\\nabla f(x^k)\\overset{J}{\\rightarrow}0.\n\\end{align}\n\\end{Definition}","sha256_utf8":"f4ef2766cafca555af056867e5be13237aeeb61e2d1edef869ecf57fb326c2fb"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"KL性质","source_units":[{"start":27173,"end":27712,"start_line":283,"end_line":289,"raw_tex":"\\begin{Definition}\\rm \\label{KL ine}\\rm\nLet $f:\\R^n\\rightarrow\\R$ be a differentiable function. We say that $f$ satisfies the \\textit{KL property} at $\\bar{x}\\in\\R^n$ if there exist a number $\\eta>0$, a neighborhood $U$ of $\\bar{x}$, and a nondecreasing function $\\psi:(0,\\eta)\\rightarrow(0,\\infty)$ such that the function $1/\\psi$ is integrable over $(0,\\eta)$ and we have\n\\begin{align}\\label{kl absil}\n \\norm{\\nabla f(x)}\\ge\\psi\\big(f(x)-f(\\bar{x})\\big)\n\\end{align}\nfor all $x\\in U$ with $f(\\bar x)<f(x)<f(\\bar x)+\\eta$.\n\\end{Definition}","sha256_utf8":"bc30f7942490dc64983ef717785879c85cf710ce8e53e603cefe2cba64c598fa"}]},{"id":"S007","semantic_type":"notation","kind":"notation","title":"加权增量可和条件","source_units":[{"start":31437,"end":31545,"start_line":324,"end_line":326,"raw_tex":"\\begin{align}\\label{series is finite}\n\\sum_{k=1}^\\infty\\norm{x^{k+1}-x^k}\\cdot\\norm{d^k}<\\infty.\n\\end{align}","sha256_utf8":"084913f86a6acf768e30d5650b4f3a5a910a4a58a4ccff7aeb24c3215e452d79"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"闭凸集投影刻画","source_units":[{"start":59280,"end":59737,"start_line":681,"end_line":685,"raw_tex":"Prior to introducing our IRG methods, let us briefly describe the GS scheme based on the $\\ve$-generalized gradient in the smooth case \\eqref{egg-smooth}. Recall that the \n{\\em projection} ${\\rm Proj}(x,\\O)$ of $x\\in\\R^n$ onto a closed and convex set $\\O\\subset\\R^n$ is a singleton that is characterized by \n\\begin{align}\\label{proj}\nu={\\rm Proj}(x,\\O)\\;\\text{ if and only if $u\\in\\O$ and  }\\;\\dotproduct{y-u,x-u}\\le 0\\;\\text{ for all }\\;y\\in\\O.\n\\end{align}","sha256_utf8":"a40bf8dbcaf2ce11c9e1c050eb356cc22fe35f9cc42fabc3540bf30ce45427c3"}]},{"id":"S009","semantic_type":"algorithm","kind":"algorithm","title":"IRG主算法","source_units":[{"start":67429,"end":68624,"start_line":767,"end_line":793,"raw_tex":"\\begin{Algorithm}[\\bf general framework for IRG methods]\\rm \\label{al gd inexact}\n\\quad\n\\setcounter{Step}{-1}\n\\begin{Step}[initialization]\\rm Select an initial point $x^1\\in \\R^n$, initial radii $\\varepsilon_1,r_1> 0$, radius reduction factors $\\mu,\\theta\\in(0,1)$, a sequence of manually controlled errors $\\set{\\rho_k}\\subset \\R_+.$\n\\end{Step}\n\\begin{Step}[inexact gradient and stopping criterion]\\rm \\label{Step 1}\nChoose $g^k$ such that \n\\begin{align}\\label{calculate approx grad}\n\\norm{g^k-\\nabla f(x^k)}\\le \\min\\set{\\varepsilon_k,\\rho_k}.\n\\end{align}\n\\end{Step}\n\\begin{Step}[radius update]\\rm \\label{Step 2}\nIf $\\norm{g^k}\\le r_{k}+\\varepsilon_k$, then set $r_{k+1}:=\\mu r_k,\\;\\varepsilon_{k+1}:=\\theta\\varepsilon_k$, $d^k:=0$, and go to~Step~3. Otherwise, set $r_{k+1}:=r_k$, $\\varepsilon_{k+1}:=\\varepsilon_k$, and \n\\begin{align}\\label{choose dk}\nd^k:=-\\dfrac{\\norm{g^k}-\\varepsilon_k}{\\norm{g^k}}g^k.\n\\end{align}\n\\end{Step}\n\\begin{Step}[stepsize]\\rm\\label{Step: 3}\nChoose $t_k>0$ by a specific rule.\n\\end{Step}\n\\begin{Step}[iteration update]\\rm\\label{step 4}\nSet $x^{k+1}:=x^{k}+t_kd^k$.\n\\end{Step}\n\\begin{Step}\\rm\nIncrease $k$ by $1$ and go back to Step $1.$\n\\end{Step}\n\\end{Algorithm}","sha256_utf8":"2c61ec34fc02acb22a6669cb2cb22d10912869d82da462b138491314f93410c3"}]},{"id":"S010","semantic_type":"derived_fact","kind":"derived_fact","title":"空迭代等价条件","source_units":[{"start":72666,"end":73004,"start_line":837,"end_line":840,"raw_tex":"\\item[\\bf(iii)] For each $k\\in\\N$, we have from Step~\\ref{Step 2} and Step~\\ref{Step: 3} the equivalences\n\\begin{align}\\label{alter condi for null}\nx^{k+1}=x^k\\Longleftrightarrow d^k=0\\Longleftrightarrow r_{k+1}=\\mu r_k\\Longleftrightarrow\\varepsilon_{k+1}=\\theta\\varepsilon_k\\Longleftrightarrow\\norm{g^k}\\le r_k+\\varepsilon_k.\n\\end{align}","sha256_utf8":"f03f63e7d89eced47c8aba1e629b302135a150c6d70b74318bc3e7a20459ae15"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"空迭代指标集","source_units":[{"start":74306,"end":74580,"start_line":871,"end_line":876,"raw_tex":"\\begin{Definition}\\rm\\label{nul iter}\nThe $k^{\\rm th}$ iteration of Algorithm~{\\rm\\ref{al gd inexact}} is called a {\\em null iteration} if $x^{k+1}=x^k$. The set of all null iterations is denoted by\n$$\n\\mathcal{N}:=\\big\\{k\\in\\N\\;\\big|\\;x^{k+1}=x^k\\big\\}.\n$$\n\\end{Definition}","sha256_utf8":"a8fa19a555a07c2089ad08fa87a3591621bb526e49fe8fcec5c4e71d0bb3e206"}]},{"id":"S012","semantic_type":"algorithm","kind":"algorithm","title":"回溯步长规则","source_units":[{"start":81831,"end":82558,"start_line":961,"end_line":964,"raw_tex":"In this section, we develop novel IRG methods with the following selections of stepsize rules: {\\em backtracking stepsize}, {\\em constant stepsize}, and {\\em diminishing stepsize}. Address first an IRG method with the {\\em backtracking line search}. Choose a line search scalar $\\beta\\in(0,1)$, a reduction factor $\\gamma\\in(0,1)$, and an artificial stepsize of null iterations $\\tau\\in(0,1)$. Consider the {\\em Master Algorithm}~\\ref{al gd inexact} with the stepsize sequence $\\set{t_k}$ in Step~3 calculated as follows. If $d^k=0$, then put $t_k:=\\tau$. Otherwise, we set\n\\begin{align}\\label{defi: tk}\nt_k:=\\max\\big\\{t\\;\\big|\\;f(x^k+td^k)\\le f(x^k)-\\beta t\\big\\|d^k\\big\\|^2,\\;t=1,\\;\\gamma,\\;\\gamma^2,\\ldots\\big\\}.\n\\end{align}","sha256_utf8":"afc5e2d0ed3d16eb839ad554e4f711ead807b1473ea484cf5a8879880f9d6086"}]},{"id":"S013","semantic_type":"notation","kind":"notation","title":"非空迭代子序列","source_units":[{"start":102456,"end":103095,"start_line":1177,"end_line":1180,"raw_tex":"The last two theorems establish efficient conditions ensuring the {\\em convergence rates} in Algorithm~\\ref{al gd inexact} under different stepsize selections. Having the sequence of iterations $\\set{x^k}$ generated by this algorithm, we obtain first from Proposition~\\ref{null properties}(iii) that if $\\N\\setminus\\mathcal{N}$ is finite, then $\\set{x^k}$ stops after a finite number of iterations. Thus we consider the case where the set $\\N\\setminus\\mathcal{N}$ is infinite and can be numerated as $\\set{j_1,j_2,\\ldots}$. Construct the sequence $\\set{z^k}$ by\n\\begin{align}\\label{z_k}\n z^k:=x^{j_k}\\;\\text{ for all }\\;k\\in\\N.\n\\end{align}","sha256_utf8":"cd0675d5d62f9ec86a85b021a1c2ce70c4dd44c225b47a9b3d0874cda9a42eb3"}]},{"id":"S014","semantic_type":"theorem","kind":"theorem","title":"回溯IRG收敛速率","source_units":[{"start":103905,"end":104857,"start_line":1193,"end_line":1202,"raw_tex":"\\begin{Theorem} \\label{rate} \nConsider Algorithm~{\\rm\\ref{al gd inexact}} with the backtracking stepsize selections under the condition $\\theta<\\mu$. Let $\\set{x^k}$ be the iterative sequence generated by this algorithm. Suppose that $\\rho_k\\rightarrow0$ as $k\\rightarrow\\infty$. Assume further that $\\set{x^k}$ has an accumulation point $\\bar x$, that $f$ satisfies the KL property at $\\bar x$ with $\\psi(t)=Mt^{q}$ for some $M>0$ and $q\\in(0,1)$, and that $\\nabla f$ is locally Lipschitzian around $\\bar x$.  The following convergence rates are guaranteed for the sequence $\\set{z^k}$ defined in \\eqref{z_k}:\n\\begin{itemize}\n\\item[\\bf(i)] If $q\\in(0,1/2]$, then the sequence $\\set{z^k}$ converges linearly to $\\bar x$.\n\\item[\\bf(ii)] If $q\\in(1/2,1)$, then there exists a positive constant $\\varrho$ such that \n\\begin{align*}\n \\norm{z^k-\\bar x}\\le\\varrho k^{-\\frac{1-q}{2q-1}}\\;\\text{ for all large }\\;k\\in\\N.\n\\end{align*}\n\\end{itemize}\n\\end{Theorem}","sha256_utf8":"e4aade4159db7da07a8051102dba29243cacdecf01509b1054d2816c2d614a34"}]},{"id":"S015","semantic_type":"proof","kind":"proof","title":"回溯速率完整证明","source_units":[{"start":104858,"end":106452,"start_line":1203,"end_line":1216,"raw_tex":"\\begin{proof}\nThe imposed assumptions yield the fulfillment of those in Theorem~\\ref{convergence under KL}, and so lead us to the convergence $x^k\\rightarrow\\bar x$ as $k\\to\\infty$. Then the local Lipschitz continuity of $\\nabla f$ around $\\bar x$ and Proposition~\\ref{bounded away} ensure that the sequence $\\set{t_k}$ is bounded away from zero.\n \n To deduce now the claimed convergence rates in (i)--(iii) from Theorem~\\ref{general rate}, define $\\tau_k:=t_{j_k}$ for all $k\\in\\N$. Then $\\set{\\tau_k}$ is also bounded away from zero as a subsequence of $\\set{t_k}$. Furthermore, using (\\ref{5 23}) and the line search conditions, we have\n\\begin{align}\\label{beta tau k}\n f(z^k)-f(z^{k+1})&=f(x^{j_{k}})-f(x^{j_{k}+1})\\ge\\beta t_{j_k}\\norm{d^{j_k}}^2\\nonumber\\\\\n &=\\dfrac{\\beta}{t_{j_k}}\\norm{x^{j_k+1}-x^{j_k}}^2=\\dfrac{\\beta}{\\tau_k}\\norm{z^{k+1}-z^k}^2\n\\end{align}\nfor all $k\\in\\N$. Note that all the assumptions of Theorem~\\ref{convergence 1} are satisfied, and so Lemma~\\ref{gradien 3 dk} holds. Pick any $N\\in\\N$ from (\\ref{equation grad 3k}) and fix $k\\ge N$. Then using (\\ref{5 23}) and (\\ref{equation grad 3k}) with taking into account that $j_k\\notin\\mathcal{N}$ for $j_k\\ge k$ leads us to\n\\begin{align*}\n\\norm{\\nabla f(z^k)}=\\norm{\\nabla f(x^{j_k})}&\\le \\dfrac{3}{t_{j_k}}\\norm{x^{j_k+1}-x^{j_k}}=\\dfrac{3}{\\tau_k}\\norm{z^{k+1}-z^k}.\n\\end{align*}\nApply finally Theorem~\\ref{general rate} to $\\set{z^k}$ and $\\set{\\tau_k}$ while remembering that $z^{k+1}\\ne z^k$ for all $k\\in\\N$ from (\\ref{differ}). This verifies the convergence rates (i)--(iii) claimed in the theorem.\n\\end{proof}","sha256_utf8":"ab8d997d15cf8c61d4ef775f9af797cdb4054dcf2e116d5e6016d9f5ffefdef1"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"全序列收敛与步长下界","source_units":[{"start":104872,"end":105205,"start_line":1204,"end_line":1204,"raw_tex":"The imposed assumptions yield the fulfillment of those in Theorem~\\ref{convergence under KL}, and so lead us to the convergence $x^k\\rightarrow\\bar x$ as $k\\to\\infty$. Then the local Lipschitz continuity of $\\nabla f$ around $\\bar x$ and Proposition~\\ref{bounded away} ensure that the sequence $\\set{t_k}$ is bounded away from zero.\n","sha256_utf8":"707250ba21a83831608f71db403702f970e4218b8676076d6a1b7435069efbb0"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"子序列平方下降估计","source_units":[{"start":105207,"end":105727,"start_line":1206,"end_line":1210,"raw_tex":" To deduce now the claimed convergence rates in (i)--(iii) from Theorem~\\ref{general rate}, define $\\tau_k:=t_{j_k}$ for all $k\\in\\N$. Then $\\set{\\tau_k}$ is also bounded away from zero as a subsequence of $\\set{t_k}$. Furthermore, using (\\ref{5 23}) and the line search conditions, we have\n\\begin{align}\\label{beta tau k}\n f(z^k)-f(z^{k+1})&=f(x^{j_{k}})-f(x^{j_{k}+1})\\ge\\beta t_{j_k}\\norm{d^{j_k}}^2\\nonumber\\\\\n &=\\dfrac{\\beta}{t_{j_k}}\\norm{x^{j_k+1}-x^{j_k}}^2=\\dfrac{\\beta}{\\tau_k}\\norm{z^{k+1}-z^k}^2\n\\end{align}\n","sha256_utf8":"e7c298a0ad53de1700d592a92f69e1e94a4bb01bfbda1f04d6c79b4a4156acbf"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"子序列梯度增量估计","source_units":[{"start":105727,"end":106217,"start_line":1211,"end_line":1214,"raw_tex":"for all $k\\in\\N$. Note that all the assumptions of Theorem~\\ref{convergence 1} are satisfied, and so Lemma~\\ref{gradien 3 dk} holds. Pick any $N\\in\\N$ from (\\ref{equation grad 3k}) and fix $k\\ge N$. Then using (\\ref{5 23}) and (\\ref{equation grad 3k}) with taking into account that $j_k\\notin\\mathcal{N}$ for $j_k\\ge k$ leads us to\n\\begin{align*}\n\\norm{\\nabla f(z^k)}=\\norm{\\nabla f(x^{j_k})}&\\le \\dfrac{3}{t_{j_k}}\\norm{x^{j_k+1}-x^{j_k}}=\\dfrac{3}{\\tau_k}\\norm{z^{k+1}-z^k}.\n\\end{align*}\n","sha256_utf8":"47d07d835293c0f9694ae3572a255453d573d0c40c6c1da0a84c7a7b1dd02d71"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"应用抽象速率定理","source_units":[{"start":106217,"end":106441,"start_line":1215,"end_line":1215,"raw_tex":"Apply finally Theorem~\\ref{general rate} to $\\set{z^k}$ and $\\set{\\tau_k}$ while remembering that $z^{k+1}\\ne z^k$ for all $k\\in\\N$ from (\\ref{differ}). This verifies the convergence rates (i)--(iii) claimed in the theorem.\n","sha256_utf8":"aea5f39db2246b4e54b2f6390d287504e1fb8e61d1520298fee846d19e1efeea"}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","title":"聚点与零方向同步","source_units":[{"start":31646,"end":32074,"start_line":329,"end_line":335,"raw_tex":"\\begin{Lemma}\\label{stationary point lemma}\nLet $\\set{x^k}$ and $\\set{d^k}$ be sequences satisfying \n\\eqref{series is finite}. If $\\bar x$ is an accumulation point of $\\set{x^k}$ and if $0$ is an accumulation point of $\\set{d^k}$, then there exists an infinite set $J\\subset\\N$ such that\n\\begin{align}\\label{relation dk xk 3}\nx^k\\overset{J}{\\rightarrow}\\bar x\\;\\mbox{ and }\\;d^k\\overset{J}{\\rightarrow}0.\n\\end{align}\n\\end{Lemma}","sha256_utf8":"1cff84fb18b74b13bc7a997e7510767c8068d024a93d664f4be1e5440f303173"}]},{"id":"S021","semantic_type":"proof","kind":"proof","title":"同步子序列完整证明","source_units":[{"start":32075,"end":34354,"start_line":336,"end_line":373,"raw_tex":"\\begin{proof}\nIf $x^k\\rightarrow\\bar x$ as $k\\to\\infty$, the conclusion obviously holds, so suppose that $x^k\\not\\rightarrow \\bar x$. It suffices to show that for any $\\delta>0$ sufficiently small and any $N\\in\\N$ there is a number $k_N\\ge N$ such that\n\\begin{align*}\n\\|x^{k_N}-\\bar x\\|<\\delta\\;\\text{ and }\\;\\|d^{k_N}\\|<\\delta.\n\\end{align*}\nFix such $\\delta>0$ and $N\\in\\N$. Since $\\delta$ is sufficiently small and $x^k\\not\\rightarrow\\bar x$, suppose that the set\n\\begin{align*}\nA_1:=\\big\\{k\\ge N\\;\\big|\\;\\|x^k-\\bar x\\|\\ge\\delta\\big\\}\\;\\text{ is infinite}.\n\\end{align*}\nAs $\\bar x$ is an accumulation point of $\\set{x^k}$ and $0$ is an accumulation point of $\\set{d^k}$, we get that\n\\begin{align*}\nA_2:&=\\big\\{k\\ge N\\;\\big|\\;\\|x^k-\\bar x\\|<\\delta\\big\\}\\;\\text{ is infinite},\\\\\nA_3:&=\\big\\{k\\ge N\\;\\big|\\;\\|x^k-\\bar x\\|<\\delta/2\\big\\}\\;\\text{ is infinite},\\\\\n A_4:&=\\big\\{k\\ge N\\;\\big|\\;\\|d^k\\|<\\delta\\big\\}\\;\\text{ is infinite}.\n\\end{align*}\nIt suffices to verify that $A_2\\cap A_4\\ne\\emptyset.$ Suppose on the contrary that $A_2\\cap A_4=\\emptyset$, i.e.,\n\\begin{align*}\n\\|d^k\\|\\ge\\delta\\;\\text{ for all }\\;k\\in A_2.\n\\end{align*}\nBy (\\ref{series is finite}), we have the estimates\n\\begin{align*}\n\\infty> \\sum_{k=1}^\\infty\\norm{x^{k+1}-x^k}\\cdot\\norm{d^k}&\\ge \\sum_{k\\in A_2} \\norm{x^{k+1}-x^k}\\cdot\\norm{d^k}\\\\\n&\\ge \\delta\\sum_{k\\in A_2}\\norm{x^{k+1}-x^k},\n\\end{align*}\nwhich ensure the series convergence\n\\begin{align}\\label{sum bounded 1}\n\\sum_{k\\in A_2}\\norm{x^{k+1}-x^k}<\\infty.\n\\end{align}\nTaking any number $K\\in A_3$, we also have $K\\in A_2.$ Since $A_1$ is infinite and  $A_1,A_2$ form a partition of the set $\\set{N,N+1,\\ldots}$ including $K$, there exists a number $\\widehat{K}\\in A_1$ with $\\widehat{K}>K$ such that $K,K+1,\\ldots,\\widehat{K}-1\\in A_2$. Then we have the estimates\n\\begin{align*}\n\\norm{x^{\\widehat{K}}-x^K}\\ge\\norm{x^{\\widehat{K}}-\\bar x}-\\norm{x^K-\\bar x}\\ge\\delta-\\delta/2=\\delta/2.\n\\end{align*}\nUsing the triangle inequality and (\\ref{sum bounded 1}) gives us\n\\begin{align*}\n\\delta/2\\le\\norm{x^{\\widehat{K}}-x^K}\\le \\sum_{i=K}^{\\widehat{K}-1}\\norm{x^{i+1}-x^i}\\le\\sum_{i\\ge K,\\;i\\in A_2}\\norm{x^{i+1}-x^i}\\xrightarrow[K\\in A_3]{K\\rightarrow\\infty}0,\n\\end{align*}\nwhich brings us to a contradiction that completes the proof of the lemma.\n\\end{proof}","sha256_utf8":"8f4c240dd8f74530f0444705713eab9330c0d739ef87d5dbe14b45f0b6e89c72"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"化为同时邻近条件","source_units":[{"start":32089,"end":32417,"start_line":337,"end_line":340,"raw_tex":"If $x^k\\rightarrow\\bar x$ as $k\\to\\infty$, the conclusion obviously holds, so suppose that $x^k\\not\\rightarrow \\bar x$. It suffices to show that for any $\\delta>0$ sufficiently small and any $N\\in\\N$ there is a number $k_N\\ge N$ such that\n\\begin{align*}\n\\|x^{k_N}-\\bar x\\|<\\delta\\;\\text{ and }\\;\\|d^{k_N}\\|<\\delta.\n\\end{align*}\n","sha256_utf8":"89954aa5b5b7e51ff78ad683ec63f21088db0c948852198258c35159570c5123"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"构造四类无限指标","source_units":[{"start":32417,"end":33019,"start_line":341,"end_line":350,"raw_tex":"Fix such $\\delta>0$ and $N\\in\\N$. Since $\\delta$ is sufficiently small and $x^k\\not\\rightarrow\\bar x$, suppose that the set\n\\begin{align*}\nA_1:=\\big\\{k\\ge N\\;\\big|\\;\\|x^k-\\bar x\\|\\ge\\delta\\big\\}\\;\\text{ is infinite}.\n\\end{align*}\nAs $\\bar x$ is an accumulation point of $\\set{x^k}$ and $0$ is an accumulation point of $\\set{d^k}$, we get that\n\\begin{align*}\nA_2:&=\\big\\{k\\ge N\\;\\big|\\;\\|x^k-\\bar x\\|<\\delta\\big\\}\\;\\text{ is infinite},\\\\\nA_3:&=\\big\\{k\\ge N\\;\\big|\\;\\|x^k-\\bar x\\|<\\delta/2\\big\\}\\;\\text{ is infinite},\\\\\n A_4:&=\\big\\{k\\ge N\\;\\big|\\;\\|d^k\\|<\\delta\\big\\}\\;\\text{ is infinite}.\n\\end{align*}\n","sha256_utf8":"8a48f26edfd64d34193d450b3cc7e311da840bd45a32e2e281d113092eb1690b"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"反设导出局部有限长度","source_units":[{"start":33019,"end":33572,"start_line":351,"end_line":363,"raw_tex":"It suffices to verify that $A_2\\cap A_4\\ne\\emptyset.$ Suppose on the contrary that $A_2\\cap A_4=\\emptyset$, i.e.,\n\\begin{align*}\n\\|d^k\\|\\ge\\delta\\;\\text{ for all }\\;k\\in A_2.\n\\end{align*}\nBy (\\ref{series is finite}), we have the estimates\n\\begin{align*}\n\\infty> \\sum_{k=1}^\\infty\\norm{x^{k+1}-x^k}\\cdot\\norm{d^k}&\\ge \\sum_{k\\in A_2} \\norm{x^{k+1}-x^k}\\cdot\\norm{d^k}\\\\\n&\\ge \\delta\\sum_{k\\in A_2}\\norm{x^{k+1}-x^k},\n\\end{align*}\nwhich ensure the series convergence\n\\begin{align}\\label{sum bounded 1}\n\\sum_{k\\in A_2}\\norm{x^{k+1}-x^k}<\\infty.\n\\end{align}\n","sha256_utf8":"341d445c41ec2843c05a0261956e7c391413af5beaf5a663994829f79b68d4fc"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"离开邻域与尾和矛盾","source_units":[{"start":33572,"end":34343,"start_line":364,"end_line":372,"raw_tex":"Taking any number $K\\in A_3$, we also have $K\\in A_2.$ Since $A_1$ is infinite and  $A_1,A_2$ form a partition of the set $\\set{N,N+1,\\ldots}$ including $K$, there exists a number $\\widehat{K}\\in A_1$ with $\\widehat{K}>K$ such that $K,K+1,\\ldots,\\widehat{K}-1\\in A_2$. Then we have the estimates\n\\begin{align*}\n\\norm{x^{\\widehat{K}}-x^K}\\ge\\norm{x^{\\widehat{K}}-\\bar x}-\\norm{x^K-\\bar x}\\ge\\delta-\\delta/2=\\delta/2.\n\\end{align*}\nUsing the triangle inequality and (\\ref{sum bounded 1}) gives us\n\\begin{align*}\n\\delta/2\\le\\norm{x^{\\widehat{K}}-x^K}\\le \\sum_{i=K}^{\\widehat{K}-1}\\norm{x^{i+1}-x^i}\\le\\sum_{i\\ge K,\\;i\\in A_2}\\norm{x^{i+1}-x^i}\\xrightarrow[K\\in A_3]{K\\rightarrow\\infty}0,\n\\end{align*}\nwhich brings us to a contradiction that completes the proof of the lemma.\n","sha256_utf8":"51fc31f39f84cad8b5218222db69f2438f0f36e812fbdac18fab44161d1540eb"}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","title":"Ostrowski聚点性质","source_units":[{"start":35770,"end":36275,"start_line":389,"end_line":400,"raw_tex":"\\begin{Lemma}\\label{lemma: ostrowski}\nLet $\\set{x^k}$ be a sequence satisfying the Ostrowski condition\n\\begin{align}\\label{8.3.8}\n\\lim_{k\\rightarrow\\infty}\\|x^{k+1}-x^k\\|=0.\n\\end{align}\nThen the following assertions hold:\n\\begin{itemize}\n\\item[\\bf(i)] If $\\set{x^k}$ is bounded, then the set of accumulation points of $\\set{x^k}$ is nonempty, compact, and connected in $\\R^n$.\n\n\\item[\\bf(ii)] If $\\set{x^k}$ has an isolated accumulation point, then this sequence converges to it.\n\\end{itemize}\n\\end{Lemma}","sha256_utf8":"1a1bc6df7483a9c51f183a7ebd43a86d5754701bdd798fa21d120a941273f5be"}]},{"id":"S027","semantic_type":"external_fact","kind":"external_fact","title":"经典聚点集定理","source_units":[],"declared_external":true,"declaration_text":["相邻差趋零的有限维有界序列具有非空、紧、连通的聚点集；相邻差趋零且具有孤立聚点的序列收敛至该点。此为第388行所引述的Ostrowski及Facchinei结果。"]},{"id":"S028","semantic_type":"theorem","kind":"theorem","title":"抽象线搜索驻点收敛","source_units":[{"start":36421,"end":37245,"start_line":404,"end_line":416,"raw_tex":"\\begin{Theorem}\\label{stationary point theorem}\nLet $\\set{x^k}$ be a sequence generated by a linesearch method \\eqref{intro: stepsize + descent direction} such that:\n\\begin{itemize}\n\\item[\\bf(a)] $\\set{d^k}$ is gradient associated with $\\set{x^k}$;\n\\item[\\bf(b)] $0$ is an accumulation point of $\\set{d^k};$\n\\item[\\bf(c)] $\\displaystyle{\\sum_{k=1}^\\infty}t_k\\|d^k\\|^2<\\infty.$\n\\end{itemize}\nThen every accumulation point of $\\set{x^k}$ is a stationary point of $f$. Moreover, if $\\set{t_k}$ is bounded from above, then the following assertions hold:\n\\begin{itemize}\n\\item[\\bf(i)] If $\\set{x^k}$ is bounded, then the set of accumulation points of $\\set{x^k}$ is nonempty, compact, and connected.\n\\item[\\bf(ii)] If $\\set{x^k}$ has an isolated accumulation point, then this sequence converges to it.\n\\end{itemize}\n\\end{Theorem}","sha256_utf8":"7df1cf41467e6992f439de522427cc1de8d07ae4195c502941b2824f7b13803d"}]},{"id":"S029","semantic_type":"proof","kind":"proof","title":"抽象驻点定理完整证明","source_units":[{"start":37246,"end":38333,"start_line":417,"end_line":428,"raw_tex":"\\begin{proof}\nLet $\\bar x$ be an accumulation point of $\\set{x^k}$.  Note that (c) is equivalent to \\eqref{series is finite} under the linesearch relationship $x^{k+1}=x^k+t_kd^k$ in (\\ref{intro: stepsize + descent direction}). Applying Lemma~\\ref{stationary point lemma} with taking into account (b) and (c), we can find an infinite set $J\\subset\\N$ such that $x^k\\overset{J}{\\rightarrow }\\bar x$ and $d^k\\overset{J}{\\rightarrow}0$. Then (a) implies that $\\nabla f(x^k)\\overset{J}{\\rightarrow}0$. Employing the continuity of $\\nabla f$, we have \n\\begin{align*}\n\\nabla f(\\bar x)=\\lim_{k\\overset{J}{\\rightarrow}\\infty}\\nabla f(x^k)=0,\n\\end{align*}\nwhich tells us that $\\bar x$ is a stationary point of $f$. Suppose now that $\\set{t_k}$ is bounded from above by some $\\tau>0$. Using (c), we immediately get\n\\begin{align*}\n\\sum_{k=1}^\\infty\\|x^{k+1}-x^k\\|^2=\\sum_{k=1}^\\infty t_k^2\\|d^k\\|^2\\le\\tau\\sum_{k=1}^\\infty t_k\\|d^k\\|^2<\\infty.\n\\end{align*}\nThis leads us to\n$\\norm{x^{k+1}-x^k}\\rightarrow 0$ and verifies assertions (i) and (ii) by applying Lemma~\\ref{lemma: ostrowski}.\n\\end{proof}","sha256_utf8":"ea09f77cd6c634a9bb13f3b13e8d9a31996c47031bff74fef7c04d63cd53bcca"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"同步子序列推出驻点","source_units":[{"start":37260,"end":38051,"start_line":418,"end_line":422,"raw_tex":"Let $\\bar x$ be an accumulation point of $\\set{x^k}$.  Note that (c) is equivalent to \\eqref{series is finite} under the linesearch relationship $x^{k+1}=x^k+t_kd^k$ in (\\ref{intro: stepsize + descent direction}). Applying Lemma~\\ref{stationary point lemma} with taking into account (b) and (c), we can find an infinite set $J\\subset\\N$ such that $x^k\\overset{J}{\\rightarrow }\\bar x$ and $d^k\\overset{J}{\\rightarrow}0$. Then (a) implies that $\\nabla f(x^k)\\overset{J}{\\rightarrow}0$. Employing the continuity of $\\nabla f$, we have \n\\begin{align*}\n\\nabla f(\\bar x)=\\lim_{k\\overset{J}{\\rightarrow}\\infty}\\nabla f(x^k)=0,\n\\end{align*}\nwhich tells us that $\\bar x$ is a stationary point of $f$. Suppose now that $\\set{t_k}$ is bounded from above by some $\\tau>0$. Using (c), we immediately get\n","sha256_utf8":"b1af2d49da309f7457bb97bf01edc8a863f95059a601bee5c31bd4dc7e64b817"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"平方增量可和与聚点结构","source_units":[{"start":38051,"end":38322,"start_line":423,"end_line":427,"raw_tex":"\\begin{align*}\n\\sum_{k=1}^\\infty\\|x^{k+1}-x^k\\|^2=\\sum_{k=1}^\\infty t_k^2\\|d^k\\|^2\\le\\tau\\sum_{k=1}^\\infty t_k\\|d^k\\|^2<\\infty.\n\\end{align*}\nThis leads us to\n$\\norm{x^{k+1}-x^k}\\rightarrow 0$ and verifies assertions (i) and (ii) by applying Lemma~\\ref{lemma: ostrowski}.\n","sha256_utf8":"c6587c669adf0dd3c09198c794f736a15730625c9ab96e239a12bd4bef4128a7"}]},{"id":"S032","semantic_type":"external_fact","kind":"external_fact","title":"KL下降收敛与长度估计","source_units":[],"declared_external":true,"declaration_text":["Absil等的Theorems 3.4、3.2给出：满足B0065的主下降及补充下降条件且在聚点满足KL性质的序列收敛至该聚点；当ψ(t)=Mt^q时，最终有‖x^{k+1}−x^k‖≤[(f(x^k)−f(x̄))^{1−q}−(f(x^{k+1})−f(x̄))^{1−q}]/[σM(1−q)]，即B0068。"]},{"id":"S033","semantic_type":"proposition","kind":"proposition","title":"KL下降序列全局收敛","source_units":[{"start":48902,"end":49713,"start_line":547,"end_line":560,"raw_tex":"\\begin{Proposition}\\label{general convergence under KL}\nLet $f:\\R^n\\rightarrow\\R$ be a $\\mathcal{C}^1$-smooth function, and let the sequence of iterations $\\set{x^k}\\subset\\R^n$ satisfy the following conditions:\n\\begin{itemize}\n\\item[\\bf(H1)] {\\rm(primary descent condition)}. There exists $\\sigma>0$ such that for sufficiently large $k\\in\\N$ we have \n\\begin{align*}\n f(x^k)-f(x^{k+1})\\ge\\sigma\\norm{\\nabla f(x^k)}\\cdot\\norm{x^{k+1}-x^k}.\n\\end{align*}\n\\item[\\bf(H2)] {\\rm(complementary descent condition)}. For sufficiently large $k\\in\\N$, we have\n\\begin{align*}\n \\big[f(x^{k+1})=f(x^k)\\big]\\Longrightarrow [x^{k+1}=x^k].\n\\end{align*}\n\\end{itemize}\nIf $\\bar x$ is an accumulation point of $\\set{x^k}$  and $f$ satisfies the KL property at $\\bar x$, then $x^k\\rightarrow\\bar x$ as $k\\to\\infty$.\n\\end{Proposition}","sha256_utf8":"b9cfe5fce787fe97f851030f052418309b4da846abe3ee90a9ac9fff63aa1178"}]},{"id":"S034","semantic_type":"proof","kind":"proof","title":"KL有限长度补充论证","source_units":[{"start":49715,"end":50375,"start_line":562,"end_line":570,"raw_tex":"Note that \\cite[Theorem~3.2]{absil05} provides a detailed proof for Proposition~\\ref{general convergence under KL} when $f$ satisfies the KL property at $\\bar x$ with $\\psi(t)=Mt^q$ for some $M>0$ and $q\\in [0,1)$. The key ingredient of the proof is the existence of some $k_1\\in\\mathbb{N}$ such that\n\\begin{align}\\label{kl induction}\n \\norm{x^{k+1}-x^k}&\\le  \\dfrac{1}{\\sigma M(1-q )}\\sbrac{(f(x^k)-f(\\bar x))^{1-q }-(f(x^{k+1})-f(\\bar x))^{1-q }}\\text{ for all }k\\ge k_1.\n\\end{align}\nUsing this, we can deduce that\n\\begin{align}\\label{sum series cauchy}\n\\sum_{k=1}^\\infty \\norm{x^{k+1}-x^k}<\\infty,\n\\end{align}\nand so the sequence $\\set{x^k}$ is convergent.\n","sha256_utf8":"4b18e73a7df85c66c8e7f3ddd0ebb4dfb271332f21dfb8970c28a46d068a9f5d"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"调用幂型KL长度估计","source_units":[{"start":49715,"end":50201,"start_line":562,"end_line":565,"raw_tex":"Note that \\cite[Theorem~3.2]{absil05} provides a detailed proof for Proposition~\\ref{general convergence under KL} when $f$ satisfies the KL property at $\\bar x$ with $\\psi(t)=Mt^q$ for some $M>0$ and $q\\in [0,1)$. The key ingredient of the proof is the existence of some $k_1\\in\\mathbb{N}$ such that\n\\begin{align}\\label{kl induction}\n \\norm{x^{k+1}-x^k}&\\le  \\dfrac{1}{\\sigma M(1-q )}\\sbrac{(f(x^k)-f(\\bar x))^{1-q }-(f(x^{k+1})-f(\\bar x))^{1-q }}\\text{ for all }k\\ge k_1.\n\\end{align}\n","sha256_utf8":"1bf50b2b46ff86931c97a2402a9688b8b0a4ce00bfb84c27a31b808ca4845b32"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"求和得到有限长度","source_units":[{"start":50201,"end":50375,"start_line":566,"end_line":570,"raw_tex":"Using this, we can deduce that\n\\begin{align}\\label{sum series cauchy}\n\\sum_{k=1}^\\infty \\norm{x^{k+1}-x^k}<\\infty,\n\\end{align}\nand so the sequence $\\set{x^k}$ is convergent.\n","sha256_utf8":"c71d78e6d381969adf9611ee3afe1616b544b292798223a2483c71a2f7159e72"}]},{"id":"S037","semantic_type":"external_fact","kind":"external_fact","title":"标量递推速率引理","source_units":[],"declared_external":true,"declaration_text":["采用第573行所引Aragón Artacho等Lemma 1及B0070的原文转述：非负趋零序列满足所述幂型差分控制时，α∈(0,1]给出线性收敛，α>1给出s_k≤ϱk^{-1/(α−1)}。原文递推式和索引保留在所选作者文本中。"]},{"id":"S038","semantic_type":"lemma","kind":"lemma","title":"幂型差分递推速率","source_units":[{"start":51279,"end":52095,"start_line":575,"end_line":588,"raw_tex":"\\begin{Lemma}\\label{lemma KL}\nLet $\\set{s_k}$ be a nonnegative sequence in $\\R$, and let $\\alpha,\\beta$ be some positive constants. Suppose that $s_k\\rightarrow 0$ as $k\\to\\infty$ and that this sequence satisfies the condition\n\\begin{align*}\ns_k^\\alpha\\le\\beta(s_k-s_{k-1})\\;\\text{ for all }\\;k\\in\\N\\;\\text{ sufficiently large}.\n\\end{align*}\nThe following assertions hold:\n\\begin{enumerate}\n\\item[\\bf (i)] If $\\alpha\\in(0,1]$, then $s_{k+1}\\le\\brac{1-\\dfrac{1}{\\beta}}s_k$ for all $k\\in\\N$ sufficiently large, i.e., $\\set{s_k}$ converges linearly to $0$ with rank $1-\\dfrac{1}{\\beta}.$\n\\item[\\bf (ii)] If $\\alpha>1$, then there exists a number $\\varrho>0$ such that \n\\begin{align*}\n s_k\\le\\varrho k^{-\\frac{1}{\\alpha-1}}\\;\\text{ for all }\\;k\\in\\N\\;\\text{ sufficiently large}.\n\\end{align*}\n\\end{enumerate}\n\\end{Lemma}","sha256_utf8":"6937521b5b2a1039607d5d4b72a430f8ebf4caa960fa3975a2569ccd7032a41a"}]},{"id":"S039","semantic_type":"theorem","kind":"theorem","title":"抽象KL收敛速率","source_units":[{"start":52162,"end":53227,"start_line":592,"end_line":605,"raw_tex":"\\begin{Theorem}\\label{general rate}\nLet the sequences $\\set{x^k}\\subset\\R^n$, $\\set{t_k}\\subset\\R_+$ and the numbers $\\beta>0,\\;c>0$ be such that $x^{k+1}\\ne x^k$ for all $k\\in\\N$, and that we have\n\\begin{align}\\label{two conditions}\nf(x^k)-f(x^{k+1})\\ge \\dfrac{\\beta}{t_k}\\norm{x^{k+1}-x^k}^2\\;\\text{ and }\\;\\norm{\\nabla f(x^k)}\\le\\dfrac{c}{t_k}\\norm{x^{k+1}-x^k}\n\\end{align}\nfor sufficiently large $k\\in\\N$. Suppose that the sequence $\\set{t_k}$ is bounded away from $0$, that $\\bar x$ is an accumulation point of $\\set{x^k}$, and that $f$ satisfies the KL property at $\\bar x$ with $\\psi(t)=Mt^{q}$ for some $M>0$ and $q\\in(0,1)$. The following convergence rates are guaranteed:\n\\begin{itemize}\\vspace*{-0.05in}\n\\item[\\bf(i)] If $q\\in(0,1/2]$, then the sequence $\\set{x^k}$ converges linearly to $\\bar x$.\\vspace*{-0.05in}\n\\item[\\bf(ii)]If $q\\in(1/2,1)$, then there exists a positive constant $\\varrho$ such that \n\\begin{align*}\n\\norm{x^k-\\bar x}\\le \\varrho k^{-\\frac{1-q}{2q-1}}\\;\\text{ for sufficiently large }\\;k\\in\\N.\n\\end{align*}\n\\end{itemize}\n\\end{Theorem}","sha256_utf8":"07b0982c1c03173a4757472344788690aeac7db3e60bde360934cc2e04cd725e"}]},{"id":"S040","semantic_type":"proof","kind":"proof","title":"抽象速率完整证明","source_units":[{"start":53245,"end":56475,"start_line":606,"end_line":651,"raw_tex":"\\begin{proof}\nThe conditions imposed in (\\ref{two conditions}) imply that\n\\begin{align*}\nf(x^k)-f(x^{k+1})\\ge \\dfrac{\\beta}{c}\\norm{\\nabla f(x^k)}\\cdot\\norm{x^{k+1}-x^k}\\;\\text{ for sufficiently large }\\;k\\in\\N,\n\\end{align*}\nwhich ensures that assumption (H1) in Proposition~\\ref{general convergence under KL} holds with $\\sigma:=\\dfrac{\\beta}{c}$. By\n$t_k>0$ for all $k\\in\\N$, the first condition in (\\ref{two conditions}) yields also assumption (H2) of this proposition. Let $k_1\\in\\N$ be such that \\eqref{kl induction} is satisfied for all $k\\ge k_1$, which implies in turn inequality \\eqref{sum series cauchy} and thus the convergence of $\\set{x^k}$ to $\\ox$. Moreover, the first condition in (\\ref{two conditions}) together with $x^{k+1}\\ne x^k$ as $k\\in\\N$ implies that \n\\begin{align}\\label{f(xk) converges}\nf(x^k)\\downarrow f(\\bar x):=\\bar f\\;\\text{ and }\\;f(x^k)>\\bar f\\;\\text{ for all }\\;k\\in\\N.\n\\end{align}\nIt follows from (\\ref{sum series cauchy}) and the triangle inequality that\n\\begin{align*}\n\\norm{x^i-\\bar x}\\le\\sum_{k=i}^\\infty\\norm{x^{k+1}-x^k}=:s_i<\\infty\\;\\text{ whenever }\\;i\\in\\N.\n\\end{align*}\nTherefore, the convergence rate of $x^i$ to $\\bar x$ can be deduced from the convergence rate of $s_i$ to $0$. Let $k_1$ be defined in (\\ref{kl induction}) and pick $i\\ge k_1$. Summing up (\\ref{kl induction}) from $i$ to any $p>i$ gives us\n\\begin{align*}\n\\sum_{k=i}^{p}\\norm{x^{k+1}-x^k}\\le\\dfrac{1}{\\sigma M(1-q)}\\big(f(x^i)-\\bar f\\big)^{1-q}.\n\\end{align*}\nLetting $p\\rightarrow\\infty$, we get the estimate\n\\begin{align}\\label{sikm}\ns_i\\le\\dfrac{1}{\\sigma M(1-q )}\\big(f(x^i)-\\bar f\\big)^{1-q}\\;\\text{ for all }\\;i\\ge k_1.\n\\end{align}\nThe KL property of $f$ at $\\bar x$ with $\\psi(t)=Mt^{q }$ allows us to find a positive number $\\eta$ and a neighborhood $U$ of $\\bar x$ ensuring the condition\n\\begin{align}\\label{KL M gradient}\n\\norm{\\nabla f(x)}\\ge M\\big(f(x)-\\bar f\\big)^q\\;\\mbox{ whenever }\\;x\\in U,\\;\\bar f<f(x)<\\bar f+\\eta.\n\\end{align} \nUsing (\\ref{KL M gradient}), $x^k\\rightarrow\\bar x$, and (\\ref{f(xk) converges}) gives us a natural number $k_2>k_1$ such that\n\\begin{align}\\label{M(1-theta)}\n\\norm{\\nabla f(x^i)}\\ge M\\big(f(x^i)-\\bar f\\big)^q\\;\\text{ for all }\\;i\\ge k_2.\n\\end{align}\nLet $\\bar{t}>0$ be a lower bound of $\\set{t_k}$. Combining this with (\\ref{M(1-theta)}) and the second inequality in (\\ref{two conditions}), we have the estimates\n\\begin{align}\n\\big(f(x^i)-\\bar f\\big)^q &\\le\\dfrac{1}{M}\\norm{\\nabla f(x^i)}\\le\\dfrac{c}{Mt_i}\\norm{x^{i+1}-x^i}\\nonumber\\\\\n&\\le\\dfrac{c}{M\\bar{t}}\\norm{x^{i+1}-x^i}\\;\\text{ for all }\\;i\\ge k_2.\\label{3.220}\n\\end{align}\nInvolving further (\\ref{sikm}) and (\\ref{3.220}) ensures that\n\\begin{align*}\ns_i^{\\frac{q}{1-q }}&\\le\\brac{\\dfrac{1}{\\sigma M(1-q)}}^{\\frac{q }{1-q}}\\big(f(x^i)-\\bar f\\big)^q\\\\\n&\\le\\brac{\\dfrac{1}{\\sigma M(1-q)}}^{\\frac{q }{1-q}}\\dfrac{c}{M\\bar{t}}\\norm{x^{i+1}-x^i}.\n\\end{align*}\nDefine $\\alpha:=\\dfrac{q }{1-q }$ and $\\beta:=\\brac{\\dfrac{1}{\\sigma M(1-q )}}^{\\frac{q }{1-q }}\\dfrac{c}{M\\bar{t}}$, and then rewrite the latter estimate as\n\\begin{align*}\ns_i^\\alpha\\le\\beta(s_i-s_{i-1})\\;\\text{ for all }\\;i\\ge k_2.\n\\end{align*}\nApplying finally Lemma~\\ref{lemma KL}, we verify the convergence rates in (i), (ii), and (iii). \n\\end{proof}","sha256_utf8":"3c21024f07b44a632286bf988eb6f29bc7b82311ff1a1856401027e49e0a1399"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"验证KL下降并得到收敛","source_units":[{"start":53259,"end":54162,"start_line":607,"end_line":615,"raw_tex":"The conditions imposed in (\\ref{two conditions}) imply that\n\\begin{align*}\nf(x^k)-f(x^{k+1})\\ge \\dfrac{\\beta}{c}\\norm{\\nabla f(x^k)}\\cdot\\norm{x^{k+1}-x^k}\\;\\text{ for sufficiently large }\\;k\\in\\N,\n\\end{align*}\nwhich ensures that assumption (H1) in Proposition~\\ref{general convergence under KL} holds with $\\sigma:=\\dfrac{\\beta}{c}$. By\n$t_k>0$ for all $k\\in\\N$, the first condition in (\\ref{two conditions}) yields also assumption (H2) of this proposition. Let $k_1\\in\\N$ be such that \\eqref{kl induction} is satisfied for all $k\\ge k_1$, which implies in turn inequality \\eqref{sum series cauchy} and thus the convergence of $\\set{x^k}$ to $\\ox$. Moreover, the first condition in (\\ref{two conditions}) together with $x^{k+1}\\ne x^k$ as $k\\in\\N$ implies that \n\\begin{align}\\label{f(xk) converges}\nf(x^k)\\downarrow f(\\bar x):=\\bar f\\;\\text{ and }\\;f(x^k)>\\bar f\\;\\text{ for all }\\;k\\in\\N.\n\\end{align}\n","sha256_utf8":"d7ba1861c632273374b786a3b0970b1034075f329d3a080c5f05ffa20cea9d87"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"尾长度控制迭代误差","source_units":[{"start":54162,"end":54897,"start_line":616,"end_line":627,"raw_tex":"It follows from (\\ref{sum series cauchy}) and the triangle inequality that\n\\begin{align*}\n\\norm{x^i-\\bar x}\\le\\sum_{k=i}^\\infty\\norm{x^{k+1}-x^k}=:s_i<\\infty\\;\\text{ whenever }\\;i\\in\\N.\n\\end{align*}\nTherefore, the convergence rate of $x^i$ to $\\bar x$ can be deduced from the convergence rate of $s_i$ to $0$. Let $k_1$ be defined in (\\ref{kl induction}) and pick $i\\ge k_1$. Summing up (\\ref{kl induction}) from $i$ to any $p>i$ gives us\n\\begin{align*}\n\\sum_{k=i}^{p}\\norm{x^{k+1}-x^k}\\le\\dfrac{1}{\\sigma M(1-q)}\\big(f(x^i)-\\bar f\\big)^{1-q}.\n\\end{align*}\nLetting $p\\rightarrow\\infty$, we get the estimate\n\\begin{align}\\label{sikm}\ns_i\\le\\dfrac{1}{\\sigma M(1-q )}\\big(f(x^i)-\\bar f\\big)^{1-q}\\;\\text{ for all }\\;i\\ge k_1.\n\\end{align}\n","sha256_utf8":"15f37132544265cfc5ef62b47d148543a42fae88a6dc3b0faeabbe9ae94d5c8a"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"最终进入KL有效邻域","source_units":[{"start":54897,"end":55456,"start_line":628,"end_line":635,"raw_tex":"The KL property of $f$ at $\\bar x$ with $\\psi(t)=Mt^{q }$ allows us to find a positive number $\\eta$ and a neighborhood $U$ of $\\bar x$ ensuring the condition\n\\begin{align}\\label{KL M gradient}\n\\norm{\\nabla f(x)}\\ge M\\big(f(x)-\\bar f\\big)^q\\;\\mbox{ whenever }\\;x\\in U,\\;\\bar f<f(x)<\\bar f+\\eta.\n\\end{align} \nUsing (\\ref{KL M gradient}), $x^k\\rightarrow\\bar x$, and (\\ref{f(xk) converges}) gives us a natural number $k_2>k_1$ such that\n\\begin{align}\\label{M(1-theta)}\n\\norm{\\nabla f(x^i)}\\ge M\\big(f(x^i)-\\bar f\\big)^q\\;\\text{ for all }\\;i\\ge k_2.\n\\end{align}\n","sha256_utf8":"d075f4935af1949b0452ba772cb60ec2296a8378d6c1c4ff8ffea18b0611a82c"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"步长下界控制函数差","source_units":[{"start":55456,"end":55839,"start_line":636,"end_line":640,"raw_tex":"Let $\\bar{t}>0$ be a lower bound of $\\set{t_k}$. Combining this with (\\ref{M(1-theta)}) and the second inequality in (\\ref{two conditions}), we have the estimates\n\\begin{align}\n\\big(f(x^i)-\\bar f\\big)^q &\\le\\dfrac{1}{M}\\norm{\\nabla f(x^i)}\\le\\dfrac{c}{Mt_i}\\norm{x^{i+1}-x^i}\\nonumber\\\\\n&\\le\\dfrac{c}{M\\bar{t}}\\norm{x^{i+1}-x^i}\\;\\text{ for all }\\;i\\ge k_2.\\label{3.220}\n\\end{align}\n","sha256_utf8":"8347f317bd30466846b230001bc7d2095164d06d3474b3b15a6c1f8cc58ff272"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"构造标量递推并取速率","source_units":[{"start":55839,"end":56464,"start_line":641,"end_line":650,"raw_tex":"Involving further (\\ref{sikm}) and (\\ref{3.220}) ensures that\n\\begin{align*}\ns_i^{\\frac{q}{1-q }}&\\le\\brac{\\dfrac{1}{\\sigma M(1-q)}}^{\\frac{q }{1-q}}\\big(f(x^i)-\\bar f\\big)^q\\\\\n&\\le\\brac{\\dfrac{1}{\\sigma M(1-q)}}^{\\frac{q }{1-q}}\\dfrac{c}{M\\bar{t}}\\norm{x^{i+1}-x^i}.\n\\end{align*}\nDefine $\\alpha:=\\dfrac{q }{1-q }$ and $\\beta:=\\brac{\\dfrac{1}{\\sigma M(1-q )}}^{\\frac{q }{1-q }}\\dfrac{c}{M\\bar{t}}$, and then rewrite the latter estimate as\n\\begin{align*}\ns_i^\\alpha\\le\\beta(s_i-s_{i-1})\\;\\text{ for all }\\;i\\ge k_2.\n\\end{align*}\nApplying finally Lemma~\\ref{lemma KL}, we verify the convergence rates in (i), (ii), and (iii). \n","sha256_utf8":"4a00368ea2645ec56ce56f201c0bba76387c7ec034e8fa22bba53d411f918a6f"}]},{"id":"S046","semantic_type":"proposition","kind":"proposition","title":"IRG充分下降方向","source_units":[{"start":73124,"end":73427,"start_line":846,"end_line":851,"raw_tex":"\\begin{Proposition}\\label{prop sufi holds}\nIn Algorithm~{\\rm\\ref{al gd inexact}}, $\\set{d^k}$ satisfies the sufficient descent condition with constant~$1,$ i.e., \n\\begin{align}\\label{sufi des holds}\n\\dotproduct{\\nabla f(x^k),d^k}\\le -\\norm{d^k}^2\\;\\text{ for all }\\;k\\in\\N.\n\\end{align}\n\\end{Proposition}","sha256_utf8":"d6fd56e527652171a66264eb9d459c67619025bd126205239fc45d8c857306d1"}]},{"id":"S047","semantic_type":"proof","kind":"proof","title":"充分下降完整证明","source_units":[{"start":73429,"end":74179,"start_line":853,"end_line":867,"raw_tex":"\\begin{proof}\nNote that (\\ref{sufi des holds}) automatically holds if $d^k=0$. Supposing that $d^k\\ne 0$ and using the construction of $d^k$ in Step~\\ref{Step 2}, we have the expression\n\\begin{align*}\n-d^k={\\rm Proj}\\big(0,\\mathbb{B}(g^k,\\varepsilon_k)\\big).\n\\end{align*}\nIt follows from (\\ref{calculate approx grad}) that $\\norm{g^k-\\nabla f(x^k)}\\le \\varepsilon_k$, which means that $\\nabla f(x^k)\\in \\mathbb{B}(g^k,\\varepsilon_k)$. Invoking the projection description for convex sets \\eqref{proj} yields\n\\begin{align*}\n\\dotproduct{0+d^k,\\nabla f(x^k)+d^k}\\le 0,\n\\end{align*}\nwhich is in turn equivalent to\n\\begin{align*}\n\\dotproduct{\\nabla f(x^k),d^k}\\le-\\norm{d^k}^2\n\\end{align*}\nand thus verifies the claim in \\eqref{sufi des holds}.\n\\end{proof}","sha256_utf8":"26b47bb2a93937ceba6cad7eece0e54c4edcb0b0857e58bc3b0f38d4c5ea5a36"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"方向表示为球上投影","source_units":[{"start":73443,"end":73701,"start_line":854,"end_line":857,"raw_tex":"Note that (\\ref{sufi des holds}) automatically holds if $d^k=0$. Supposing that $d^k\\ne 0$ and using the construction of $d^k$ in Step~\\ref{Step 2}, we have the expression\n\\begin{align*}\n-d^k={\\rm Proj}\\big(0,\\mathbb{B}(g^k,\\varepsilon_k)\\big).\n\\end{align*}\n","sha256_utf8":"fb5e4dade66929ebf4db3e3f345d11d32a0505551790af077d2c67160da6757e"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"投影不等式推出下降","source_units":[{"start":73701,"end":74168,"start_line":858,"end_line":866,"raw_tex":"It follows from (\\ref{calculate approx grad}) that $\\norm{g^k-\\nabla f(x^k)}\\le \\varepsilon_k$, which means that $\\nabla f(x^k)\\in \\mathbb{B}(g^k,\\varepsilon_k)$. Invoking the projection description for convex sets \\eqref{proj} yields\n\\begin{align*}\n\\dotproduct{0+d^k,\\nabla f(x^k)+d^k}\\le 0,\n\\end{align*}\nwhich is in turn equivalent to\n\\begin{align*}\n\\dotproduct{\\nabla f(x^k),d^k}\\le-\\norm{d^k}^2\n\\end{align*}\nand thus verifies the claim in \\eqref{sufi des holds}.\n","sha256_utf8":"0e6920221126873021b7e83cb154cfffccd1fd57305556d6b0d2f876c6d45df7"}]},{"id":"S050","semantic_type":"proposition","kind":"proposition","title":"空迭代基本性质","source_units":[{"start":74671,"end":75848,"start_line":880,"end_line":895,"raw_tex":"\\begin{Proposition}\\label{null properties}\nLet $\\set{x^k},\\set{g^k},\\set{d^k},\\set{\\varepsilon_k}$, and $\\set{r_k}$ be sequences generated by Algorithm~{\\rm\\ref{al gd inexact}}. The following assertions hold:\n\\begin{itemize}\n\\item[\\bf(i)] $k\\in\\mathcal{N}$ if and only if either one of the equivalent conditions in \\eqref{alter condi for null} holds.\\vspace*{-0.05in}\n\n\\item[\\bf(ii)] $\\varepsilon_k\\downarrow 0$ if and only if $r_k\\downarrow 0$, which is equivalent to the set $\\mathcal{N}$ being infinite.\n\n\\item[\\bf(iii)] If $\\mathcal{N}$ is finite, then we have\n\\begin{align*}\n \\norm{g^k}>r_N+\\varepsilon_N\\;\\text{ and }\\;\\norm{d^k}>r_N\n\\end{align*}\nfor all $k\\ge N$, where $N:=\\max\\mathcal{N}+1$.\n\n\\item[\\bf(iv)] If $\\N\\setminus\\mathcal{N}$ is finite, then $\\nabla f(x^K)=0$ and $\\set{x^k}_{k\\ge K}$ is a constant sequence, where we denote $K:=\\max\\{\\N\\setminus\\mathcal{N}\\}+1$. Otherwise, $\\bar x$ is an accumulation point of $\\set{x^k}$ if and only if it is an accumulation point of $\\set{x^k}_{k\\in \\N\\setminus\\mathcal{N}}$, and therefore $x^k\\rightarrow\\bar x$ if and only if $x^k\\overset{\\N\\setminus\\mathcal{N}}{\\longrightarrow}\\bar x$.\n\\end{itemize}\n\\end{Proposition}","sha256_utf8":"c15c378533592dfa1592f6f747c3609e6590d5fac115299124fe478b3fb94e6f"}]},{"id":"S051","semantic_type":"proof","kind":"proof","title":"空迭代性质完整证明","source_units":[{"start":75849,"end":78328,"start_line":896,"end_line":921,"raw_tex":"\\begin{proof} Assertions (i) and (ii) follow directly from Definition~\\ref{nul iter}. To verify (iii), observe that for any natural number $k\\ge N$, the $k^{\\rm th}$ iteration is not a null one and then deduce from (i) that\n\\begin{align*}\n\\varepsilon_{k+1}=\\varepsilon_k=\\varepsilon_N,\\quad r_{k+1}=r_k=r_N,\\;\\mbox{ and }\\;\\norm{g^k}>r_k+\\varepsilon_k=r_N+\\varepsilon_N.\n\\end{align*}\nTogether with Step~\\ref{Step 2} in Algorithm~\\ref{al gd inexact}, this ensures that   \n\\begin{align*}\n\\norm{d^k}=\\norm{g^k}-\\varepsilon_k>r_k=r_N.\n\\end{align*}\nwhich readily justifies assertion (i).\n\nThe proof of (iv) is a bit more involved. Supposing that the set $\\N\\setminus\\mathcal{N}$ is finite, we have that $k\\in \\mathcal{N}$ for all $k\\ge K$. This means by (i) that\n\\begin{align*}\nx^{k+1}=x^{k},\\;\\varepsilon_{k+1}=\\theta\\varepsilon_k,\\;r_{k+1}=\\mu r_k,\\;\\text{ and }\\;\\norm{g^k}\\le r_k+\\varepsilon_k\\;\\text{ whenever }\\;k\\ge K.\n\\end{align*}\nThis tells us that $x^k=x^{K}$ for all $k\\ge K$, and that $\\varepsilon_k\\downarrow 0,\\;r_k\\downarrow 0$, and $g^k\\rightarrow 0$ as $k\\to\\infty$. Taking the limit as $k\\rightarrow \\infty$ in $\\norm{g^k-\\nabla f(x^k)}\\le \\varepsilon_k$ gives us $\\nabla f(x^k)\\rightarrow0$, which yields $\\nabla f(x^{K})=0$.\n\nSupposing otherwise that the set $\\N\\setminus\\mathcal{N}$ is infinite, we obviously get that every accumulation point of $\\set{x^k}_{k\\in \\N\\setminus \\mathcal{N}}$ is an accumulation point of $\\set{x^k}$. Conversely, taking any accumulation point $\\bar x$ of $\\set{ x^k}$, it suffices to show that \n\\begin{align*}\n\\mbox{for any }\\;\\delta>0,\\;N\\in\\N\\;\\mbox{ there exists }\\;k_N\\in\\N\\setminus\\mathcal{N},\\;k_N\\ge N\\;\\mbox{ with }\\;\\norm{x^{k_N}-\\bar x}<\\delta.\n\\end{align*}\nTo verify this, fixing $\\delta >0$ and $N\\in\\N$ and remembering that $\\bar x$ is an accumulation point of $\\set{x^k}$, we find $K\\ge N$ such that $\\norm{x^K-\\bar x}<\\delta$. If $K\\in\\N\\setminus\\mathcal{N}$, choose $k_N:=K$. Otherwise, using that $\\N\\setminus\\mathcal{N}$ is infinite allows us to find $\\widehat{K}\\in\\N\\setminus\\mathcal{N}$ for which $K,K+1,\\ldots,\\widehat{K}-1\\in\\mathcal{N}$. This ensures that \n\\begin{align*}\n{x^{\\widehat{K}}}={x^{\\widehat{K}-1}}=\\ldots=x^{K+1}=x^K,\n\\end{align*}\nand therefore, with $k_N:=\\widehat{K}$, we get that $\\norm{x^{k_N}-\\bar x}<\\delta$. Since $\\delta$ was chosen arbitrarily, this clearly shows that $\\bar x$ is an accumulation point of $\\set{x^k}_{k\\in\\N\\setminus\\mathcal{N}}$ and thus completes the proof. \n\\end{proof}","sha256_utf8":"781c37ea86e09d7e66c1f4b88d9c60f6cc516c7f67c2cc75bdc5dca1bc517e03"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"半径收缩及非空方向下界","source_units":[{"start":75849,"end":76432,"start_line":896,"end_line":904,"raw_tex":"\\begin{proof} Assertions (i) and (ii) follow directly from Definition~\\ref{nul iter}. To verify (iii), observe that for any natural number $k\\ge N$, the $k^{\\rm th}$ iteration is not a null one and then deduce from (i) that\n\\begin{align*}\n\\varepsilon_{k+1}=\\varepsilon_k=\\varepsilon_N,\\quad r_{k+1}=r_k=r_N,\\;\\mbox{ and }\\;\\norm{g^k}>r_k+\\varepsilon_k=r_N+\\varepsilon_N.\n\\end{align*}\nTogether with Step~\\ref{Step 2} in Algorithm~\\ref{al gd inexact}, this ensures that   \n\\begin{align*}\n\\norm{d^k}=\\norm{g^k}-\\varepsilon_k>r_k=r_N.\n\\end{align*}\nwhich readily justifies assertion (i).\n","sha256_utf8":"fcdec80af9210325b3acb713f0f94c5f92e89182a4092848ef68eb9597e2b275"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"最终全为空迭代时驻点","source_units":[{"start":76433,"end":77089,"start_line":906,"end_line":910,"raw_tex":"The proof of (iv) is a bit more involved. Supposing that the set $\\N\\setminus\\mathcal{N}$ is finite, we have that $k\\in \\mathcal{N}$ for all $k\\ge K$. This means by (i) that\n\\begin{align*}\nx^{k+1}=x^{k},\\;\\varepsilon_{k+1}=\\theta\\varepsilon_k,\\;r_{k+1}=\\mu r_k,\\;\\text{ and }\\;\\norm{g^k}\\le r_k+\\varepsilon_k\\;\\text{ whenever }\\;k\\ge K.\n\\end{align*}\nThis tells us that $x^k=x^{K}$ for all $k\\ge K$, and that $\\varepsilon_k\\downarrow 0,\\;r_k\\downarrow 0$, and $g^k\\rightarrow 0$ as $k\\to\\infty$. Taking the limit as $k\\rightarrow \\infty$ in $\\norm{g^k-\\nabla f(x^k)}\\le \\varepsilon_k$ gives us $\\nabla f(x^k)\\rightarrow0$, which yields $\\nabla f(x^{K})=0$.\n","sha256_utf8":"1aedee3a93156d11284f5ed45e0676509805d941774ceb9ce11f10ab1dc38dc5"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"全序列与非空子列聚点相同","source_units":[{"start":77090,"end":78317,"start_line":912,"end_line":920,"raw_tex":"Supposing otherwise that the set $\\N\\setminus\\mathcal{N}$ is infinite, we obviously get that every accumulation point of $\\set{x^k}_{k\\in \\N\\setminus \\mathcal{N}}$ is an accumulation point of $\\set{x^k}$. Conversely, taking any accumulation point $\\bar x$ of $\\set{ x^k}$, it suffices to show that \n\\begin{align*}\n\\mbox{for any }\\;\\delta>0,\\;N\\in\\N\\;\\mbox{ there exists }\\;k_N\\in\\N\\setminus\\mathcal{N},\\;k_N\\ge N\\;\\mbox{ with }\\;\\norm{x^{k_N}-\\bar x}<\\delta.\n\\end{align*}\nTo verify this, fixing $\\delta >0$ and $N\\in\\N$ and remembering that $\\bar x$ is an accumulation point of $\\set{x^k}$, we find $K\\ge N$ such that $\\norm{x^K-\\bar x}<\\delta$. If $K\\in\\N\\setminus\\mathcal{N}$, choose $k_N:=K$. Otherwise, using that $\\N\\setminus\\mathcal{N}$ is infinite allows us to find $\\widehat{K}\\in\\N\\setminus\\mathcal{N}$ for which $K,K+1,\\ldots,\\widehat{K}-1\\in\\mathcal{N}$. This ensures that \n\\begin{align*}\n{x^{\\widehat{K}}}={x^{\\widehat{K}-1}}=\\ldots=x^{K+1}=x^K,\n\\end{align*}\nand therefore, with $k_N:=\\widehat{K}$, we get that $\\norm{x^{k_N}-\\bar x}<\\delta$. Since $\\delta$ was chosen arbitrarily, this clearly shows that $\\bar x$ is an accumulation point of $\\set{x^k}_{k\\in\\N\\setminus\\mathcal{N}}$ and thus completes the proof. \n","sha256_utf8":"4e9a0bd5fa630cee5d7c23ff13da8150608ffc392d1e7a2f22e0019cb6dc865f"}]},{"id":"S055","semantic_type":"proposition","kind":"proposition","title":"近似梯度与方向趋零关系","source_units":[{"start":78493,"end":79203,"start_line":925,"end_line":938,"raw_tex":"\\begin{Proposition}\\label{goes to 0} Let $\\set{g^k},\\set{d^k},\\;\\set{\\varepsilon_k}$, and $\\set{r_k}$ be sequences generated by Algorithm~{\\rm\\ref{al gd inexact}}. Then for any $k\\in\\N$ we have the estimates\n\\begin{align}\\label{nx1}\n\\norm{d^k}\\le\\norm{g^k}\\le\\norm{d^k}+\\varepsilon_k+r_k.\n\\end{align}\nConsequently, the following assertions hold:\n\\begin{itemize}\n\\item[\\bf(i)] $\\varepsilon_k\\downarrow0$ if and only if there is an infinite set $J\\subset\\N$ such that $g^k\\overset{J}{\\rightarrow}0$.\n\n\\item[\\bf(ii)] For any infinite set $J\\subset\\N$, we have the equivalence\n\\begin{align*}\ng^k\\overset{J}{\\rightarrow}0\\Longleftrightarrow d^k\\overset{J}{\\rightarrow}0.\n\\end{align*}\n\\end{itemize}\n\\end{Proposition}","sha256_utf8":"c70d94cc40ad9e013a3d95ddc3a5a61a6d0176bbcfe5644621fc5578e3fd985e"}]},{"id":"S056","semantic_type":"proof","kind":"proof","title":"趋零关系完整证明","source_units":[{"start":79205,"end":80951,"start_line":940,"end_line":950,"raw_tex":"\\begin{proof}\nFix any $k\\in\\N$. If $k\\in\\mathcal{N}$, then we get by Proposition~\\ref{null properties}(i) that $d^k=0$ and $\\norm{g^k}\\le \\varepsilon_k+r_k$. Otherwise, Step~\\ref{Step 2} in Algorithm~\\ref{al gd inexact} yields $\\norm{d^k}=\\norm{g^k}-\\varepsilon_k\\le \\norm{g^k}$. In both cases, (\\ref{nx1}) holds. \n\nTo deduce (i) from (\\ref{nx1}), suppose that $\\varepsilon_k\\downarrow 0$ as $k\\to\\infty$. By Proposition~\\ref{null properties}(ii) we have that $r_k\\downarrow 0$ and the set $\\mathcal{N}$ is infinite. Then for any $\\delta>0$ and $N\\in\\N$ there is $k\\ge N$ with $k\\in\\mathcal{N}$ such that\n\\begin{align*}\n\\norm{g^{k}}\\le r_k+\\varepsilon_k<\\delta,\n\\end{align*}\nwhere the first inequality follows from (\\ref{alter condi for null}). Thus we can construct an infinite set $J\\subset\\N$ such that $g^k\\overset{J}{\\longrightarrow}0$. If conversely the sequence $\\set{\\varepsilon_k}$ does not converge to $0$, then the set $\\mathcal{N}$ is finite by Proposition~\\ref{null properties}(ii). Using Proposition~\\ref{null properties}(iii) confirms that $\\set{g^k}$ is bounded away from $0$, which tells us that such an index set $J$ does not exist.\n\nTo verify now assertion (ii), observe that assuming $g^k\\overset{J}{\\rightarrow}0$, implies by the first inequality in (\\ref{nx1}) that $d^k\\overset{J}{\\rightarrow}0$. Conversely, suppose that $d^k\\overset{J}{\\rightarrow}0$ and deduce from Proposition~\\ref{null properties}(i) that the set $\\mathcal{N}$ is infinite. Then it follows from Proposition~\\ref{null properties}(ii) that  $\\varepsilon_k\\downarrow0$ and $r_k\\downarrow0$ as $k\\to\\infty$. Using the second inequality in (\\ref{nx1}), we arrive at $g^k\\overset{J}{\\rightarrow}0$ and thus complete the proof of the proposition.\n\\end{proof}","sha256_utf8":"28579b488121af6df9adebd23c31f767c896f9316ca64a9a566e7ca20e8fdbe1"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"分空迭代建立范数夹界","source_units":[{"start":79219,"end":79520,"start_line":941,"end_line":941,"raw_tex":"Fix any $k\\in\\N$. If $k\\in\\mathcal{N}$, then we get by Proposition~\\ref{null properties}(i) that $d^k=0$ and $\\norm{g^k}\\le \\varepsilon_k+r_k$. Otherwise, Step~\\ref{Step 2} in Algorithm~\\ref{al gd inexact} yields $\\norm{d^k}=\\norm{g^k}-\\varepsilon_k\\le \\norm{g^k}$. In both cases, (\\ref{nx1}) holds. \n","sha256_utf8":"0d3fc7c9db0c063d2d4be437a3d9602c2595abe7f08988f99fa3d2ffdd4b8996"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"误差趋零等价于梯度子列趋零","source_units":[{"start":79521,"end":80356,"start_line":943,"end_line":947,"raw_tex":"To deduce (i) from (\\ref{nx1}), suppose that $\\varepsilon_k\\downarrow 0$ as $k\\to\\infty$. By Proposition~\\ref{null properties}(ii) we have that $r_k\\downarrow 0$ and the set $\\mathcal{N}$ is infinite. Then for any $\\delta>0$ and $N\\in\\N$ there is $k\\ge N$ with $k\\in\\mathcal{N}$ such that\n\\begin{align*}\n\\norm{g^{k}}\\le r_k+\\varepsilon_k<\\delta,\n\\end{align*}\nwhere the first inequality follows from (\\ref{alter condi for null}). Thus we can construct an infinite set $J\\subset\\N$ such that $g^k\\overset{J}{\\longrightarrow}0$. If conversely the sequence $\\set{\\varepsilon_k}$ does not converge to $0$, then the set $\\mathcal{N}$ is finite by Proposition~\\ref{null properties}(ii). Using Proposition~\\ref{null properties}(iii) confirms that $\\set{g^k}$ is bounded away from $0$, which tells us that such an index set $J$ does not exist.\n","sha256_utf8":"ac1f9f111282c328012bb02da7499fd42e9b8fd0edcc48c25d9b7ffc16f96901"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"方向与近似梯度同步趋零","source_units":[{"start":80357,"end":80940,"start_line":949,"end_line":949,"raw_tex":"To verify now assertion (ii), observe that assuming $g^k\\overset{J}{\\rightarrow}0$, implies by the first inequality in (\\ref{nx1}) that $d^k\\overset{J}{\\rightarrow}0$. Conversely, suppose that $d^k\\overset{J}{\\rightarrow}0$ and deduce from Proposition~\\ref{null properties}(i) that the set $\\mathcal{N}$ is infinite. Then it follows from Proposition~\\ref{null properties}(ii) that  $\\varepsilon_k\\downarrow0$ and $r_k\\downarrow0$ as $k\\to\\infty$. Using the second inequality in (\\ref{nx1}), we arrive at $g^k\\overset{J}{\\rightarrow}0$ and thus complete the proof of the proposition.\n","sha256_utf8":"6885ba8e1d1920ce656a08ed5a91758b01ba641d7093b965cbab7369bf566fb6"}]},{"id":"S060","semantic_type":"corollary","kind":"corollary","title":"IRG方向梯度关联","source_units":[{"start":81120,"end":81273,"start_line":954,"end_line":955,"raw_tex":"\\begin{Corollary}\\label{overall} The sequence $\\set{d^k}$ in Algorithms~{\\rm\\ref{al gd inexact}} is gradient associated with $\\set{x^k}$.\n\\end{Corollary}","sha256_utf8":"f11320b4b23128716b51070c7e942e73edbb483745e31a3443f097719f4d7682"}]},{"id":"S061","semantic_type":"proof","kind":"proof","title":"梯度关联完整证明","source_units":[{"start":81290,"end":81730,"start_line":956,"end_line":957,"raw_tex":"\\begin{proof} It follows from Proposition \\ref{goes to 0} that the convergence $d^k\\overset{J}{\\rightarrow}0$ yields $g^k\\overset{J}{\\rightarrow}0$ and $\\varepsilon_k\\downarrow0.$ Thus we get  $\\nabla f(x^k)\\overset{J}{\\rightarrow}0$ by taking into account $\\norm{g^k-\\nabla f(x^k)}\\le \\varepsilon_k$ from \\eqref{calculate approx grad}. This shows therefore that the sequence $\\set{d^k}$ is gradient associated with $\\set{x^k}$.\n\\end{proof}","sha256_utf8":"c9116f4f7ce8e4419745ca50e9c60c11f332232cc5f79d1fc78b5952d7a5b8f2"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"近似误差传递至真实梯度","source_units":[{"start":81290,"end":81719,"start_line":956,"end_line":956,"raw_tex":"\\begin{proof} It follows from Proposition \\ref{goes to 0} that the convergence $d^k\\overset{J}{\\rightarrow}0$ yields $g^k\\overset{J}{\\rightarrow}0$ and $\\varepsilon_k\\downarrow0.$ Thus we get  $\\nabla f(x^k)\\overset{J}{\\rightarrow}0$ by taking into account $\\norm{g^k-\\nabla f(x^k)}\\le \\varepsilon_k$ from \\eqref{calculate approx grad}. This shows therefore that the sequence $\\set{d^k}$ is gradient associated with $\\set{x^k}$.\n","sha256_utf8":"2991af71450fb061abe96b18d733dad5ee307e45d1941de0e43a064d7d8e82bc"}]},{"id":"S063","semantic_type":"proposition","kind":"proposition","title":"回溯步长存在","source_units":[{"start":82645,"end":82964,"start_line":968,"end_line":973,"raw_tex":"\\begin{Proposition}\\label{descent} If $d^k\\ne 0$, then there exists a positive number $\\bar t$ such that \n\\begin{align*}\nf(x^k+td^k)\\le f(x^k)-\\beta t\\big\\|d^k\\big\\|^2\\;\\text{ for all }\\;t\\in [0,\\bar t],\n\\end{align*}\nwhich ensures the existence of the stepsize sequence $\\{t_k\\}$ in \\eqref{defi: tk}. \n\\end{Proposition}","sha256_utf8":"1ae02f1261fdb40f0b326d2d3d54f449e972adc8a47c898f7dca29f2eb55e031"}]},{"id":"S064","semantic_type":"proof","kind":"proof","title":"步长存在完整证明","source_units":[{"start":82965,"end":83677,"start_line":974,"end_line":985,"raw_tex":"\\begin{proof} Suppose that $d^k\\ne 0$ and get by Proposition~\\ref{prop sufi holds} that $\\dotproduct{\\nabla f(x^k),d^k}\\le-\\norm{d^k}^2$. By the differentiability of $f$ at $x^k$, for each $t>0$ sufficiently small we have\n\\begin{align*}\nf(x^k+td^k)-f(x^k)&=t\\dotproduct{\\nabla f(x^k),d^k}+o(t)\\le-t\\norm{d^k}^2+o(t)\\\\\n&=-\\beta t\\norm{d^k}^2+t\\brac{\n(\\beta-1)\\norm{d^k}^2+ \\dfrac{o(t)}{t}}.\n\\end{align*}\nSince $o(t)/t\\rightarrow0$ as $t\\dn 0$ and since $(\\beta-1)\\norm{d^k}^2<0$, there exists $\\bar t>0$ such that \n\\begin{align*}\nf(x^k+td^k)\\le f(x^k)-\\beta t\\norm{d^k}^2\\;\\text{ for all }\\;t\\in(0,\\bar t].\n\\end{align*}\nTherefore, all the numbers $t_k$ introduced in (\\ref{defi: tk}) are well-defined.\n\\end{proof}","sha256_utf8":"4482283482f942bf008e25418ff78356720b226c980c135ec887cb54d53fe6bd"}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","title":"一阶展开结合充分下降","source_units":[{"start":82965,"end":83368,"start_line":974,"end_line":979,"raw_tex":"\\begin{proof} Suppose that $d^k\\ne 0$ and get by Proposition~\\ref{prop sufi holds} that $\\dotproduct{\\nabla f(x^k),d^k}\\le-\\norm{d^k}^2$. By the differentiability of $f$ at $x^k$, for each $t>0$ sufficiently small we have\n\\begin{align*}\nf(x^k+td^k)-f(x^k)&=t\\dotproduct{\\nabla f(x^k),d^k}+o(t)\\le-t\\norm{d^k}^2+o(t)\\\\\n&=-\\beta t\\norm{d^k}^2+t\\brac{\n(\\beta-1)\\norm{d^k}^2+ \\dfrac{o(t)}{t}}.\n\\end{align*}\n","sha256_utf8":"11ccd95b9de674e55612fbc3484c4e8b6425081c96e4b9096547c1d0550de046"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"小步长满足回溯检验","source_units":[{"start":83368,"end":83666,"start_line":980,"end_line":984,"raw_tex":"Since $o(t)/t\\rightarrow0$ as $t\\dn 0$ and since $(\\beta-1)\\norm{d^k}^2<0$, there exists $\\bar t>0$ such that \n\\begin{align*}\nf(x^k+td^k)\\le f(x^k)-\\beta t\\norm{d^k}^2\\;\\text{ for all }\\;t\\in(0,\\bar t].\n\\end{align*}\nTherefore, all the numbers $t_k$ introduced in (\\ref{defi: tk}) are well-defined.\n","sha256_utf8":"ba623f37bcc3f187272df1b8cb52ef3e7851c7363c4478eebd806cf90abe59e2"}]},{"id":"S067","semantic_type":"theorem","kind":"theorem","title":"回溯IRG驻点聚点收敛","source_units":[{"start":83881,"end":84709,"start_line":989,"end_line":1000,"raw_tex":"\\begin{Theorem}\\label{convergence 1}\nLet $\\set{x^k}$ be the sequence of iterations generated by Algorithm~{\\rm\\ref{al gd inexact}} with the stepsize sequences $t_k$ from \\eqref{defi: tk} of the backtracking line search. Assume that $\\inf f(x^k)>-\\infty$ and that $\\rho_k\\rightarrow0$ as $k\\rightarrow\\infty$. Then the following assertions hold:\n\\begin{itemize}\n\\item[\\bf(i)] $\\varepsilon_k\\downarrow0$ and $r_k\\downarrow 0$ as $k\\to\\infty$.\n\n\\item[\\bf(ii)] Every accumulation point of $\\set{x^k}$ is a stationary point of $f$.\n\n\\item[\\bf(iii)] If the sequence $\\set{x^k}$ is bounded, then the set of accumulation points of $\\set{x^k}$ is nonempty, compact, and connected.\n\n\\item[\\bf(iv)] If $\\set{x^k}$ has an isolated accumulation point, then the entire sequence $\\set{x^k}$ converges to this point.\n\\end{itemize}\n\\end{Theorem}","sha256_utf8":"e8a7c1869e0d032b33f347089b052eca858dcac4abf8430a78963d3ebc68cb25"}]},{"id":"S068","semantic_type":"proof","kind":"proof","title":"回溯驻点收敛完整证明","source_units":[{"start":84710,"end":88520,"start_line":1001,"end_line":1043,"raw_tex":"\\begin{proof} From the choice of $\\set{t_k}$ and Step~4 in Algorithm~\\ref{al gd inexact}, for every $k\\in\\N$ we have\n\\begin{align}\\label{tk dk}\n\\beta t_k\\norm{d^k}^2\\le f(x^k)-f(x^{k+1}).\n\\end{align}\nSince $\\inf f(x^k)>-\\infty$, summing up  on both sides of (\\ref{tk dk}) over $k=1,2,\\ldots$ and using the relation $x^{k+1}=x^k+t_kd^k$, we get that\n\\begin{align}\\label{taking sum 1}\n\\sum_{k=1}^\\infty t_k\\norm{d^k}^2<\\infty\\quad\\text{ and }\\quad \\sum_{k=1}^\\infty\\norm{x^{k+1}-x^k}\\cdot\\norm{d^k}<\\infty.\n\\end{align}\n\nTo verify assertion (i), recall by Proposition~\\ref{null properties}(ii) that the convergence $\\varepsilon_k\\downarrow0$ is equivalent to $r_k\\downarrow 0$ and to the fact that the set of null iterations $\\mathcal{N}$ is infinite. Assume on the contrary that $\\mathcal{N}$ is finite. By Proposition~\\ref{null properties}(iii) with $N=\\max\\mathcal{N}+1$, we have\n\\begin{align}\\label{dk bounded}\n\\norm{g^k}>r_N+\\varepsilon_N\\;\\text{ and }\\;\\norm{d^k}>  r_N\\;\\text{ for all }\\;k\\ge N.\n\\end{align}\nThen (\\ref{taking sum 1}) gives us $\\sum_{k=1}^\\infty t_k<\\infty$ and thus $t_k\\downarrow0$ as $k\\to\\infty$. Choosing a larger number $N$ if necessary, we get that $t_k<1$ for all $k\\ge N$. For such $k$, it follows from the exit condition of the algorithm that\n\\begin{align}\\label{exit inexact}\n-\\gamma^{-1}\\beta t_k\\norm{d^k}^2< f(x^{k}+\\gamma^{-1}t_kd^k)-f(x^k).\n\\end{align}\nBy the classical mean value theorem, there exists some $\\tilde{x}^k\\in[x^k,x^k+\\gamma^{-1}t_kd^k]$ such that \n\\begin{align*}\nf(x^{k}+\\gamma^{-1}t_kd^k)-f(x^k)= \\gamma^{-1}t_k\\dotproduct{d^k,\\nabla f(\\tilde{x}^k)}.\n\\end{align*}\nCombining the latter with (\\ref{exit inexact}) tells us that\n\\begin{align}\\label{contra 1}\n\\dotproduct{-d^k,\\nabla f(\\tilde{x}^k)}\\le\\beta \\norm{d^k}^2\\;\\text{ for all }\\;k\\ge N.\n\\end{align}\nUsing (\\ref{taking sum 1}) and (\\ref{dk bounded}), we have $\\sum_{k=1}^\\infty\\norm{x^{k+1}-x^k}<\\infty$, and thus $\\set{x^k}$ converges to some $\\bar x\\in\\R^n$. The continuity of $\\nabla f$ ensures that $\\nabla f(x^k)\\rightarrow\\nabla f(\\bar x)$. Then employing $\\norm{g^k-\\nabla f(x^k)}\\le \\rho_k\\rightarrow0$ yields $g^k\\rightarrow\\nabla f(\\bar x)$ as $k\\rightarrow\\infty$.\nIt follows from Step~\\ref{Step 2} that\n\\begin{align*}\n-d^k&=\\dfrac{\\norm{g^k}-\\varepsilon_k}{\\norm{g^k}}g^k=\\dfrac{\\norm{g^k}-\\varepsilon_N}{\\norm{g^k}}g^k\\;\\text{ for all \\;}k\\ge N.\n\\end{align*}\nLetting $k\\rightarrow\\infty$ leads us to the equalities\n\\begin{align}\\label{1.8 1}\n-d^k\\rightarrow\\bar g:=\\dfrac{\\norm{\\nabla f(\\bar x)}- \\varepsilon_N }{\\norm{\\nabla f(\\bar x)}}\\nabla f(\\bar x)={\\rm Proj}\\big(0,\\mathbb{B}(\\nabla f(\\bar x),\\varepsilon_N)\\big).\n\\end{align}\nUsing $t_k\\downarrow0$, we get that $\\tilde{x}^k\\rightarrow\\bar x$, and thus $\\nabla f(\\tilde{x}^k)\\rightarrow\\nabla f(\\bar x)$ as $k\\to\\infty$. Combining the latter with (\\ref{contra 1}), (\\ref{1.8 1}), and the projection characterization in \\eqref{proj} verifies the estimates\n\\begin{align}\n\\norm{\\bar g}^2\\le\\dotproduct{\\bar g,\\nabla f(\\bar x)}\\le\\beta \\norm{\\bar g}^2.\n\\end{align}\nThis tells us that $\\bar g=0$, which contradicts the condition $\\norm{\\bar g}\\ge r_N$ by (\\ref{dk bounded}). Therefore, we arrive at $\\varepsilon_k\\downarrow 0$ and $r_k\\downarrow 0$ as $k\\to\\infty$, which completes the proof of assertion (i).\n\nTo justify assertions (ii)--(iv), recall from Corollary~\\ref{overall} that $\\set{d^k}$ is \\textit{gradient associated} with $\\set{x^k}$. Since $\\varepsilon_k\\downarrow 0$, we deduce from Proposition~\\ref{goes to 0} that \n$0$ is an accumulation point of $\\set{d^k}$. Combining these facts with (\\ref{taking sum 1}) and $t_k\\le 1$ whenever $k\\in\\N$ ensures that all the assumptions of Theorem~\\ref{stationary point theorem} are satisfied. Therefore, we verify assertions (ii)--(iv) and finish the proof of the theorem.\n\\end{proof}","sha256_utf8":"16dd8afb57712273a7aba7efa9829b08584d02555a047c7a5acaba4507c15e27"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"下降求和得到加权可和","source_units":[{"start":84710,"end":85227,"start_line":1001,"end_line":1008,"raw_tex":"\\begin{proof} From the choice of $\\set{t_k}$ and Step~4 in Algorithm~\\ref{al gd inexact}, for every $k\\in\\N$ we have\n\\begin{align}\\label{tk dk}\n\\beta t_k\\norm{d^k}^2\\le f(x^k)-f(x^{k+1}).\n\\end{align}\nSince $\\inf f(x^k)>-\\infty$, summing up  on both sides of (\\ref{tk dk}) over $k=1,2,\\ldots$ and using the relation $x^{k+1}=x^k+t_kd^k$, we get that\n\\begin{align}\\label{taking sum 1}\n\\sum_{k=1}^\\infty t_k\\norm{d^k}^2<\\infty\\quad\\text{ and }\\quad \\sum_{k=1}^\\infty\\norm{x^{k+1}-x^k}\\cdot\\norm{d^k}<\\infty.\n\\end{align}\n","sha256_utf8":"b1337ffcd5969eaae7fd3ce6215336a302579af7876451143117d541442338c6"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"有限空迭代迫使步长趋零","source_units":[{"start":85228,"end":86099,"start_line":1010,"end_line":1017,"raw_tex":"To verify assertion (i), recall by Proposition~\\ref{null properties}(ii) that the convergence $\\varepsilon_k\\downarrow0$ is equivalent to $r_k\\downarrow 0$ and to the fact that the set of null iterations $\\mathcal{N}$ is infinite. Assume on the contrary that $\\mathcal{N}$ is finite. By Proposition~\\ref{null properties}(iii) with $N=\\max\\mathcal{N}+1$, we have\n\\begin{align}\\label{dk bounded}\n\\norm{g^k}>r_N+\\varepsilon_N\\;\\text{ and }\\;\\norm{d^k}>  r_N\\;\\text{ for all }\\;k\\ge N.\n\\end{align}\nThen (\\ref{taking sum 1}) gives us $\\sum_{k=1}^\\infty t_k<\\infty$ and thus $t_k\\downarrow0$ as $k\\to\\infty$. Choosing a larger number $N$ if necessary, we get that $t_k<1$ for all $k\\ge N$. For such $k$, it follows from the exit condition of the algorithm that\n\\begin{align}\\label{exit inexact}\n-\\gamma^{-1}\\beta t_k\\norm{d^k}^2< f(x^{k}+\\gamma^{-1}t_kd^k)-f(x^k).\n\\end{align}\n","sha256_utf8":"1eabdfb40de79e100e4d92e9220ac1cf829a6f226284a65582e99161c7219b45"}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","title":"均值定理转化失败检验","source_units":[{"start":86099,"end":86517,"start_line":1018,"end_line":1025,"raw_tex":"By the classical mean value theorem, there exists some $\\tilde{x}^k\\in[x^k,x^k+\\gamma^{-1}t_kd^k]$ such that \n\\begin{align*}\nf(x^{k}+\\gamma^{-1}t_kd^k)-f(x^k)= \\gamma^{-1}t_k\\dotproduct{d^k,\\nabla f(\\tilde{x}^k)}.\n\\end{align*}\nCombining the latter with (\\ref{exit inexact}) tells us that\n\\begin{align}\\label{contra 1}\n\\dotproduct{-d^k,\\nabla f(\\tilde{x}^k)}\\le\\beta \\norm{d^k}^2\\;\\text{ for all }\\;k\\ge N.\n\\end{align}\n","sha256_utf8":"c37636a0c361b996c73f9a3a47808ac833a91be1d9c08b0d2ebe75ebdb08b2bf"}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","title":"有限长度得到方向投影极限","source_units":[{"start":86517,"end":87362,"start_line":1026,"end_line":1034,"raw_tex":"Using (\\ref{taking sum 1}) and (\\ref{dk bounded}), we have $\\sum_{k=1}^\\infty\\norm{x^{k+1}-x^k}<\\infty$, and thus $\\set{x^k}$ converges to some $\\bar x\\in\\R^n$. The continuity of $\\nabla f$ ensures that $\\nabla f(x^k)\\rightarrow\\nabla f(\\bar x)$. Then employing $\\norm{g^k-\\nabla f(x^k)}\\le \\rho_k\\rightarrow0$ yields $g^k\\rightarrow\\nabla f(\\bar x)$ as $k\\rightarrow\\infty$.\nIt follows from Step~\\ref{Step 2} that\n\\begin{align*}\n-d^k&=\\dfrac{\\norm{g^k}-\\varepsilon_k}{\\norm{g^k}}g^k=\\dfrac{\\norm{g^k}-\\varepsilon_N}{\\norm{g^k}}g^k\\;\\text{ for all \\;}k\\ge N.\n\\end{align*}\nLetting $k\\rightarrow\\infty$ leads us to the equalities\n\\begin{align}\\label{1.8 1}\n-d^k\\rightarrow\\bar g:=\\dfrac{\\norm{\\nabla f(\\bar x)}- \\varepsilon_N }{\\norm{\\nabla f(\\bar x)}}\\nabla f(\\bar x)={\\rm Proj}\\big(0,\\mathbb{B}(\\nabla f(\\bar x),\\varepsilon_N)\\big).\n\\end{align}\n","sha256_utf8":"92292be66448be29d5fbfbc54f4ffeda5daca065fb0fb3bcb3ca03e6528cdc96"}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","title":"投影极限矛盾迫使半径趋零","source_units":[{"start":87362,"end":87991,"start_line":1035,"end_line":1039,"raw_tex":"Using $t_k\\downarrow0$, we get that $\\tilde{x}^k\\rightarrow\\bar x$, and thus $\\nabla f(\\tilde{x}^k)\\rightarrow\\nabla f(\\bar x)$ as $k\\to\\infty$. Combining the latter with (\\ref{contra 1}), (\\ref{1.8 1}), and the projection characterization in \\eqref{proj} verifies the estimates\n\\begin{align}\n\\norm{\\bar g}^2\\le\\dotproduct{\\bar g,\\nabla f(\\bar x)}\\le\\beta \\norm{\\bar g}^2.\n\\end{align}\nThis tells us that $\\bar g=0$, which contradicts the condition $\\norm{\\bar g}\\ge r_N$ by (\\ref{dk bounded}). Therefore, we arrive at $\\varepsilon_k\\downarrow 0$ and $r_k\\downarrow 0$ as $k\\to\\infty$, which completes the proof of assertion (i).\n","sha256_utf8":"bf693da650790e84ef4ced6508cc2153761a6deeb95cead542c56b198780f471"}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","title":"调用抽象驻点与聚点定理","source_units":[{"start":87992,"end":88509,"start_line":1041,"end_line":1042,"raw_tex":"To justify assertions (ii)--(iv), recall from Corollary~\\ref{overall} that $\\set{d^k}$ is \\textit{gradient associated} with $\\set{x^k}$. Since $\\varepsilon_k\\downarrow 0$, we deduce from Proposition~\\ref{goes to 0} that \n$0$ is an accumulation point of $\\set{d^k}$. Combining these facts with (\\ref{taking sum 1}) and $t_k\\le 1$ whenever $k\\in\\N$ ensures that all the assumptions of Theorem~\\ref{stationary point theorem} are satisfied. Therefore, we verify assertions (ii)--(iv) and finish the proof of the theorem.\n","sha256_utf8":"483756382f9c78ccf9eb90ecfde64756ccb3982331bb60dcd122106583e805a9"}]},{"id":"S075","semantic_type":"lemma","kind":"lemma","title":"真实梯度由方向控制","source_units":[{"start":91370,"end":91843,"start_line":1073,"end_line":1079,"raw_tex":"\\begin{Lemma}\\label{gradien 3 dk}\nLet $\\set{x^k}$ be the sequence generated by Algorithm~{\\rm\\ref{al gd inexact}} with $\\theta<\\mu$. Assume that  $\\varepsilon_k\\downarrow 0$ and $r_k\\downarrow0$ as $k\\to\\infty$. Then there exists some $N\\in\\N$ such that\n\\begin{align}\\label{equation grad 3k}\n\\norm{\\nabla f(x^k)}\\le 3\\norm{d^k}\\;\\text{ for all }\\;k\\notin\\mathcal{N},\\;k\\ge N,\n\\end{align}\nwhere the set $\\mathcal{N}$ is taken from Definition~{\\rm\\ref{nul iter}}.\n\\end{Lemma}","sha256_utf8":"1091dc743d62a4bb0a50b246a9136a05aead4853df98fa7e0d9edf9e19ba1ac8"}]},{"id":"S076","semantic_type":"proof","kind":"proof","title":"梯度控制完整证明","source_units":[{"start":91844,"end":92612,"start_line":1080,"end_line":1089,"raw_tex":"\\begin{proof} It follows directly from the assumptions of the lemma that there exists a natural number $N$ such that $\\varepsilon_k\\le r_k$ for all $k\\ge N$. By Step~\\ref{Step 2} of the algorithm, for any $k\\ge N$ with $k\\notin\\mathcal{N}$ we have $\\norm{g^k}>r_k+\\varepsilon_k$ with the direction $d^k$ calculated in (\\ref{choose dk}). Thus for such $k$ we get the estimates\n\\begin{align}\\label{66}\n\\norm{d^k}=\\norm{g^k}-\\varepsilon_k>r_k+\\varepsilon_k-\\varepsilon_k= r_k\\ge\\varepsilon_k.\n\\end{align}\nIt follows from (\\ref{calculate approx grad}) in Step~\\ref{Step 1} and from (\\ref{66}) that\n\\begin{align*}\n\\norm{\\nabla f(x^k)}\\le \\norm{g^k}+\\varepsilon_k=\\norm{d^k}+2\\varepsilon_k\\le 3\\norm{d^k},\n\\end{align*}\nwhich verifies the conclusion of the lemma.\n\\end{proof}","sha256_utf8":"749341fd30c70222c28ec6b4f4fe539b1b2bcc5dc88b02cc93f4295d40ca747e"}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","title":"误差半径最终小于方向","source_units":[{"start":91844,"end":92346,"start_line":1080,"end_line":1083,"raw_tex":"\\begin{proof} It follows directly from the assumptions of the lemma that there exists a natural number $N$ such that $\\varepsilon_k\\le r_k$ for all $k\\ge N$. By Step~\\ref{Step 2} of the algorithm, for any $k\\ge N$ with $k\\notin\\mathcal{N}$ we have $\\norm{g^k}>r_k+\\varepsilon_k$ with the direction $d^k$ calculated in (\\ref{choose dk}). Thus for such $k$ we get the estimates\n\\begin{align}\\label{66}\n\\norm{d^k}=\\norm{g^k}-\\varepsilon_k>r_k+\\varepsilon_k-\\varepsilon_k= r_k\\ge\\varepsilon_k.\n\\end{align}\n","sha256_utf8":"9acd05cdf08af73aca50e39b7439105143f80f24873fe40fc4d83810cd1a8062"}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","title":"三角不等式给出三倍界","source_units":[{"start":92346,"end":92601,"start_line":1084,"end_line":1088,"raw_tex":"It follows from (\\ref{calculate approx grad}) in Step~\\ref{Step 1} and from (\\ref{66}) that\n\\begin{align*}\n\\norm{\\nabla f(x^k)}\\le \\norm{g^k}+\\varepsilon_k=\\norm{d^k}+2\\varepsilon_k\\le 3\\norm{d^k},\n\\end{align*}\nwhich verifies the conclusion of the lemma.\n","sha256_utf8":"f23d9927952f1e3d599c7d7d5ce7eccaaafee440f67ac7bd7a17c73443e6587e"}]},{"id":"S079","semantic_type":"theorem","kind":"theorem","title":"回溯IRG的KL全局收敛","source_units":[{"start":92905,"end":93358,"start_line":1093,"end_line":1095,"raw_tex":"\\begin{Theorem}\\label{convergence under KL}\nLet $\\set{x^k}$ be the iterative sequence generated by Algorithm~{\\rm\\ref{al gd inexact}} with the backtracking linesearch such that $\\theta<\\mu$. Assume that $\\rho_k\\rightarrow0$ as $k\\rightarrow\\infty$, $\\set{x^k}$ has an accumulation point $\\bar x$, and $f$ satisfies the KL property at $\\bar x$. Then $\\bar x$ is a stationary point of $f$, and we have $x^k\\rightarrow\\bar x$ as $k\\to\\infty$.\n\\end{Theorem}","sha256_utf8":"fa9c92dc7a4e2ee1ad6573ce8aaed81263026d011334ee220213084de5e08850"}]},{"id":"S080","semantic_type":"proof","kind":"proof","title":"KL全局收敛完整证明","source_units":[{"start":93359,"end":95261,"start_line":1096,"end_line":1111,"raw_tex":"\\begin{proof}\nSince $\\bar x$ is an accumulation point of $\\set{x^k}$, we can find some infinite set $J\\subset\\N$ such that $x^k\\overset{J}{\\rightarrow}\\bar x$. It follows from the choice of $\\set{t_k}$ in \\eqref{defi: tk} that $\\set{f(x^k)}$ is nonincreasing, which implies that \n\\begin{align*}\n\\inf_{k\\in \\N} f(x^k)= \\inf_{k\\in J} f(x^k)=f(\\bar x)>-\\infty.\n\\end{align*}\nTherefore, the results of Theorem~\\ref{convergence 1} tell us that $\\bar x$ is a stationary point of $f$ and that $\\varepsilon_k\\dn 0$, $r_k\\downarrow 0$ as $k\\to\\infty$. We employ Proposition~\\ref{general convergence under KL} to verify that $x^k\\rightarrow\\bar x$ along the entire sequence of iterations. Indeed, the imposed assumptions and the convergence $\\varepsilon_k\\dn 0$, $r_k\\downarrow 0$ as $k\\to\\infty$ guarantee that all the requirements  of Lemma~\\ref{gradien 3 dk} are satisfied. Pick $N\\in\\N$ such that (\\ref{equation grad 3k}) holds. The choice of $\\set{t_k}$ in (\\ref{defi: tk}) ensures the lower estimate\n\\begin{align}\\label{5.133}\nf(x^{k})-f(x^{k+1})&\\ge\\beta t_k\\big\\|d^k\\big\\|^2\\;\\text{ for all }\\;k\\in\\N.\n\\end{align}\nCombining this with (\\ref{equation grad 3k}) and the relation $x^{k+1}=x^k+t_kd^k$ yields\n\\begin{align}\\label{5.13}\nf(x^{k})-f(x^{k+1})\\ge\\dfrac{\\beta}{3}\\norm{\\nabla f(x^k)}\\cdot\\norm{x^{k+1}-x^k}\\;\\text{ for all }\\;k\\notin\\mathcal{N},\\;k\\ge N.\n\\end{align}\nObserve that when $k\\in\\mathcal{N}$, both sides of\n(\\ref{5.13}) reduce to zero, and so (\\ref{5.13}) is satisfied. Therefore, assumption (H1) in Proposition~\\ref{general convergence under KL} holds. Moreover, for $k\\ge N$ the conditions $f(x^{k+1})=f(x^k)$ and (\\ref{5.133}) imply that $d^k=0$, and hence $x^{k+1}=x^k$. Thus assumption (H2) in  Proposition~\\ref{general convergence under KL} is satisfied as well. Applying the latter proposition, we arrive at $x^k\\rightarrow\\bar x$ as $k\\to\\infty$ and complete the proof.\n\\end{proof}","sha256_utf8":"c51f235bc5207fca8b6e0105d1cfbf5545617517f1a52b3f1ce5eae37f27af7e"}]},{"id":"S081","semantic_type":"proof_step","kind":"proof_step","title":"聚点保证函数下界与驻点","source_units":[{"start":93373,"end":94354,"start_line":1097,"end_line":1101,"raw_tex":"Since $\\bar x$ is an accumulation point of $\\set{x^k}$, we can find some infinite set $J\\subset\\N$ such that $x^k\\overset{J}{\\rightarrow}\\bar x$. It follows from the choice of $\\set{t_k}$ in \\eqref{defi: tk} that $\\set{f(x^k)}$ is nonincreasing, which implies that \n\\begin{align*}\n\\inf_{k\\in \\N} f(x^k)= \\inf_{k\\in J} f(x^k)=f(\\bar x)>-\\infty.\n\\end{align*}\nTherefore, the results of Theorem~\\ref{convergence 1} tell us that $\\bar x$ is a stationary point of $f$ and that $\\varepsilon_k\\dn 0$, $r_k\\downarrow 0$ as $k\\to\\infty$. We employ Proposition~\\ref{general convergence under KL} to verify that $x^k\\rightarrow\\bar x$ along the entire sequence of iterations. Indeed, the imposed assumptions and the convergence $\\varepsilon_k\\dn 0$, $r_k\\downarrow 0$ as $k\\to\\infty$ guarantee that all the requirements  of Lemma~\\ref{gradien 3 dk} are satisfied. Pick $N\\in\\N$ such that (\\ref{equation grad 3k}) holds. The choice of $\\set{t_k}$ in (\\ref{defi: tk}) ensures the lower estimate\n","sha256_utf8":"7e6837c1f86a50a261f5f8d181f438ff018c34741ff2ebef3c69c0923f73a8ac"}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","title":"建立KL主下降条件","source_units":[{"start":94354,"end":94728,"start_line":1102,"end_line":1108,"raw_tex":"\\begin{align}\\label{5.133}\nf(x^{k})-f(x^{k+1})&\\ge\\beta t_k\\big\\|d^k\\big\\|^2\\;\\text{ for all }\\;k\\in\\N.\n\\end{align}\nCombining this with (\\ref{equation grad 3k}) and the relation $x^{k+1}=x^k+t_kd^k$ yields\n\\begin{align}\\label{5.13}\nf(x^{k})-f(x^{k+1})\\ge\\dfrac{\\beta}{3}\\norm{\\nabla f(x^k)}\\cdot\\norm{x^{k+1}-x^k}\\;\\text{ for all }\\;k\\notin\\mathcal{N},\\;k\\ge N.\n\\end{align}\n","sha256_utf8":"5396cffdb086fa71ae41d68c84ce5e873e25c25eff00d9b5158383174b62f058"}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","title":"处理空迭代并应用KL定理","source_units":[{"start":94728,"end":95250,"start_line":1109,"end_line":1110,"raw_tex":"Observe that when $k\\in\\mathcal{N}$, both sides of\n(\\ref{5.13}) reduce to zero, and so (\\ref{5.13}) is satisfied. Therefore, assumption (H1) in Proposition~\\ref{general convergence under KL} holds. Moreover, for $k\\ge N$ the conditions $f(x^{k+1})=f(x^k)$ and (\\ref{5.133}) imply that $d^k=0$, and hence $x^{k+1}=x^k$. Thus assumption (H2) in  Proposition~\\ref{general convergence under KL} is satisfied as well. Applying the latter proposition, we arrive at $x^k\\rightarrow\\bar x$ as $k\\to\\infty$ and complete the proof.\n","sha256_utf8":"ee7e9ae7e84dcbf67f1e2963cc6c647f3c517b356d5c478e1df8d4642e0af074"}]},{"id":"S084","semantic_type":"proposition","kind":"proposition","title":"回溯步长远离零","source_units":[{"start":98338,"end":98820,"start_line":1132,"end_line":1134,"raw_tex":"\\begin{Proposition}\\label{bounded away}\nLet $\\set{x^k}$ be a sequence generated by Algorithm~{\\rm\\ref{al gd inexact}} with the backtracking stepsize. Suppose that $\\rho_k\\rightarrow0$ as $k\\rightarrow\\infty$ and that there exists an infinite set $J\\subset\\N$ such that $\\set{x^k}_{k\\in J}$ converges to some $\\bar x$. Assume further that $\\nabla f$ is locally Lipschitzian around $\\bar x$. Then the stepsize sequence $\\set{t_k}_{k\\in J}$ is bounded away from zero.\n\\end{Proposition}","sha256_utf8":"4cee4f43d9be017fb2bfc2699664acb681291a4aea7926e4034113d3b20c4949"}]},{"id":"S085","semantic_type":"proof","kind":"proof","title":"步长下界完整证明","source_units":[{"start":98822,"end":102454,"start_line":1136,"end_line":1175,"raw_tex":"\\begin{proof}\n Assume on the contrary that $\\set{t_k}_{k\\in J}$ is not bounded away from zero. Then we find an infinite set $\\bar J\\subset J$ such that $t_k\\overset{\\bar J}{\\longrightarrow}0$. Let $\\tau\\in(0,1)$ be an artificial stepsize of null iterations. Since $t_k\\overset{\\bar J}{\\longrightarrow}0$, there exists a number $N\\in\\N$ such that \n \\begin{align}\\label{tk < tau}\nt_k<\\tau<1\\;\\text{ for all }\\;k\\ge N,\\;k\\in\\bar J.\n \\end{align}\n This means that $k\\notin\\mathcal{N}$ whenever $k\\ge N,\\;k\\in\\bar J$. By Proposition~\\ref{null properties}(i), we have $d^k\\ne 0$ for all $k\\ge N,\\;k\\in\\bar J$. Then condition (\\ref{choose dk}) in Step~\\ref{Step 2} leads us to\n\\begin{align}\\label{norm dk < norm gk}\n\\norm{d^k}=\\norm{g^k}-\\varepsilon_k\\le\\norm{g^k}\\;\\mbox{ for all }\\;k\\ge N,\\;k\\in\\bar J.\n\\end{align}\nSince $x^k\\overset{\\bar J}{\\rightarrow}\\bar x$, the continuity of $\\nabla f$ and the estimate $\\norm{\\nabla f(x^k)-g^k}\\le \\rho_k\\rightarrow0$ yield that $g^k\\overset{\\bar J}{\\rightarrow}\\nabla f(\\bar x)$. Using (\\ref{norm dk < norm gk}), we get that the sequence $\\set{d^k}_{k\\in\\bar J}$ is bounded, and thus \n\\begin{align}\\label{bound of norm dk}\nx^k+\\gamma^{-1} t_kd^k\\rightarrow\\bar x\\;\\mbox{ as }\\;k\\overset{\\bar J}{\\rightarrow}\\infty.\n\\end{align}\nSince $\\nabla f$ is locally Lipschitzian around $\\bar x$, there exists a positive number $\\delta$ such that $\\nabla f$ is Lipschitz continuous on $\\mathbb{B}(\\bar x,\\delta)$ with some modulus $L>0$. By (\\ref{bound of norm dk}) and $x^k\\overset{\\bar J}{\\rightarrow}\\bar x$, we find $N_1\\ge N$ with $x^k,x^k+\\gamma^{-1}t_kd^k\\in\\mathbb{B}(\\bar x,\\delta)$ for all $k\\ge N_1,\\;k\\in\\bar J$. The Lipschitz continuity of $\\nabla f$ on $\\mathbb{B}(\\bar x,\\delta)$ with modulus $L$ yields by \\cite[Proposition~A.24]{bertsekasbook} the $L$-descent condition, i.e.,\n\\begin{align}\\label{lipschitz}\nf(x)-f(y)\\le\\dotproduct{\\nabla f(y),x-y}+\\dfrac{L}{2}\\norm{x-y}^2\\;\\text{ for all }\\;x,y\\in \\mathbb{B}(\\bar x,\\delta).\n\\end{align}\nFixing $k\\in\\bar J,\\;k\\ge N_1$, we deduce from the above that $d^k\\ne 0$, and $t_k<1$. The exit condition for the backtracking line search implies that\n\\begin{align}\\label{exit 2}\n-\\gamma^{-1}\\beta t_k\\norm{d^k}^2<f(x^{k}+\\gamma^{-1}t_kd^k)-f(x^k).\n\\end{align}\nApplying (\\ref{lipschitz}) for $x=x^{k}+\\gamma^{-1}t_kd^k$ and $y=x^k$ we have that\n\\begin{align*}\nf(x^{k}+\\gamma^{-1}t_kd^k)-f(x^k)\\le\\gamma^{-1}t_k\\dotproduct{\\nabla f(x^k),d^k}+\\dfrac{L\\gamma^{-2}t_k^2}{2}\\norm{d^k}^2.\n\\end{align*}\nCombining this with (\\ref{exit 2}) leads us to \n\\begin{align*}\n-\\gamma^{-1}\\beta t_k\\norm{d^k}^2<\\gamma^{-1}t_k\\dotproduct{\\nabla f(x^k),d^k}+\\dfrac{L\\gamma^{-2}t_k^2}{2}\\norm{d^k}^2,\n\\end{align*}\nor equivalently to the inequality \n\\begin{align}\\label{0< gamma-1}\n0<\\gamma^{-1}\\beta t_k\\norm{d^k}^2+\\gamma^{-1}t_k\\dotproduct{\\nabla f(x^k),d^k}+\\dfrac{L\\gamma^{-2}t_k^2}{2}\\norm{d^k}^2.\n\\end{align}\nProposition~\\ref{prop sufi holds} and $d^k\\ne 0$ tell us that $0<\\norm{d^k}^2\\le\\dotproduct{\\nabla f(x^k),-d^k}$. Then we deduce from (\\ref{0< gamma-1}) the fulfillment of the estimate\n\\begin{align*}\n0<\\gamma^{-1}\\beta t_k\\dotproduct{\\nabla f(x^k),-d^k}+\\gamma^{-1}t_k\\dotproduct{\\nabla f(x^k),d^k}+\\dfrac{L\\gamma^{-2}t_k^2}{2}\\dotproduct{\\nabla f(x^k),-d^k}.\n\\end{align*}\nDividing both sides above by $\\gamma^{-1}t_k\\dotproduct{\\nabla f(x^k),-d^k}>0$, we get $0<\\beta-1+\\dfrac{L\\gamma^{-1}t_k}{2}$.\nLetting $k\\overset{\\bar J}{\\longrightarrow}\\infty$ yields $\\beta\\ge 1$, which contradicts the choice of $\\beta\\in(0,1)$. Thus we verify that the sequence $\\set{t_k}_{k\\in J}$ is bounded away from zero, which completes the proof of the proposition.\n\\end{proof}","sha256_utf8":"8e0162ade836a97b78f4eafbef68ae0436508156a0d4b3144f8473ada7c3b40f"}]},{"id":"S086","semantic_type":"proof_step","kind":"proof_step","title":"反设抽取非空小步长子列","source_units":[{"start":98836,"end":99631,"start_line":1137,"end_line":1144,"raw_tex":" Assume on the contrary that $\\set{t_k}_{k\\in J}$ is not bounded away from zero. Then we find an infinite set $\\bar J\\subset J$ such that $t_k\\overset{\\bar J}{\\longrightarrow}0$. Let $\\tau\\in(0,1)$ be an artificial stepsize of null iterations. Since $t_k\\overset{\\bar J}{\\longrightarrow}0$, there exists a number $N\\in\\N$ such that \n \\begin{align}\\label{tk < tau}\nt_k<\\tau<1\\;\\text{ for all }\\;k\\ge N,\\;k\\in\\bar J.\n \\end{align}\n This means that $k\\notin\\mathcal{N}$ whenever $k\\ge N,\\;k\\in\\bar J$. By Proposition~\\ref{null properties}(i), we have $d^k\\ne 0$ for all $k\\ge N,\\;k\\in\\bar J$. Then condition (\\ref{choose dk}) in Step~\\ref{Step 2} leads us to\n\\begin{align}\\label{norm dk < norm gk}\n\\norm{d^k}=\\norm{g^k}-\\varepsilon_k\\le\\norm{g^k}\\;\\mbox{ for all }\\;k\\ge N,\\;k\\in\\bar J.\n\\end{align}\n","sha256_utf8":"3ea9cc9b18ae38c56e36c944680efdb40862a2bd049a19a84042a3a3db02d6fd"}]},{"id":"S087","semantic_type":"proof_step","kind":"proof_step","title":"方向有界使试探点趋近聚点","source_units":[{"start":99631,"end":100084,"start_line":1145,"end_line":1148,"raw_tex":"Since $x^k\\overset{\\bar J}{\\rightarrow}\\bar x$, the continuity of $\\nabla f$ and the estimate $\\norm{\\nabla f(x^k)-g^k}\\le \\rho_k\\rightarrow0$ yield that $g^k\\overset{\\bar J}{\\rightarrow}\\nabla f(\\bar x)$. Using (\\ref{norm dk < norm gk}), we get that the sequence $\\set{d^k}_{k\\in\\bar J}$ is bounded, and thus \n\\begin{align}\\label{bound of norm dk}\nx^k+\\gamma^{-1} t_kd^k\\rightarrow\\bar x\\;\\mbox{ as }\\;k\\overset{\\bar J}{\\rightarrow}\\infty.\n\\end{align}\n","sha256_utf8":"e819a812a31b9a93f382ce494c497d526b6f6a2284dda930f618ac54abbe0051"}]},{"id":"S088","semantic_type":"proof_step","kind":"proof_step","title":"局部Lipschitz给出下降引理","source_units":[{"start":100084,"end":100801,"start_line":1149,"end_line":1152,"raw_tex":"Since $\\nabla f$ is locally Lipschitzian around $\\bar x$, there exists a positive number $\\delta$ such that $\\nabla f$ is Lipschitz continuous on $\\mathbb{B}(\\bar x,\\delta)$ with some modulus $L>0$. By (\\ref{bound of norm dk}) and $x^k\\overset{\\bar J}{\\rightarrow}\\bar x$, we find $N_1\\ge N$ with $x^k,x^k+\\gamma^{-1}t_kd^k\\in\\mathbb{B}(\\bar x,\\delta)$ for all $k\\ge N_1,\\;k\\in\\bar J$. The Lipschitz continuity of $\\nabla f$ on $\\mathbb{B}(\\bar x,\\delta)$ with modulus $L$ yields by \\cite[Proposition~A.24]{bertsekasbook} the $L$-descent condition, i.e.,\n\\begin{align}\\label{lipschitz}\nf(x)-f(y)\\le\\dotproduct{\\nabla f(y),x-y}+\\dfrac{L}{2}\\norm{x-y}^2\\;\\text{ for all }\\;x,y\\in \\mathbb{B}(\\bar x,\\delta).\n\\end{align}\n","sha256_utf8":"7192f7360eb7ff8f267865c26eb1196ed5a83b9a9ce8da273b08e5608bd639c8"}]},{"id":"S089","semantic_type":"proof_step","kind":"proof_step","title":"失败回溯检验结合二次上界","source_units":[{"start":100801,"end":101695,"start_line":1153,"end_line":1168,"raw_tex":"Fixing $k\\in\\bar J,\\;k\\ge N_1$, we deduce from the above that $d^k\\ne 0$, and $t_k<1$. The exit condition for the backtracking line search implies that\n\\begin{align}\\label{exit 2}\n-\\gamma^{-1}\\beta t_k\\norm{d^k}^2<f(x^{k}+\\gamma^{-1}t_kd^k)-f(x^k).\n\\end{align}\nApplying (\\ref{lipschitz}) for $x=x^{k}+\\gamma^{-1}t_kd^k$ and $y=x^k$ we have that\n\\begin{align*}\nf(x^{k}+\\gamma^{-1}t_kd^k)-f(x^k)\\le\\gamma^{-1}t_k\\dotproduct{\\nabla f(x^k),d^k}+\\dfrac{L\\gamma^{-2}t_k^2}{2}\\norm{d^k}^2.\n\\end{align*}\nCombining this with (\\ref{exit 2}) leads us to \n\\begin{align*}\n-\\gamma^{-1}\\beta t_k\\norm{d^k}^2<\\gamma^{-1}t_k\\dotproduct{\\nabla f(x^k),d^k}+\\dfrac{L\\gamma^{-2}t_k^2}{2}\\norm{d^k}^2,\n\\end{align*}\nor equivalently to the inequality \n\\begin{align}\\label{0< gamma-1}\n0<\\gamma^{-1}\\beta t_k\\norm{d^k}^2+\\gamma^{-1}t_k\\dotproduct{\\nabla f(x^k),d^k}+\\dfrac{L\\gamma^{-2}t_k^2}{2}\\norm{d^k}^2.\n\\end{align}\n","sha256_utf8":"a3a9874cb2ac1773cc747c36a64111d992f25a709f4f4b788b4b7bdbc753034f"}]},{"id":"S090","semantic_type":"proof_step","kind":"proof_step","title":"充分下降导出参数矛盾","source_units":[{"start":101695,"end":102443,"start_line":1169,"end_line":1174,"raw_tex":"Proposition~\\ref{prop sufi holds} and $d^k\\ne 0$ tell us that $0<\\norm{d^k}^2\\le\\dotproduct{\\nabla f(x^k),-d^k}$. Then we deduce from (\\ref{0< gamma-1}) the fulfillment of the estimate\n\\begin{align*}\n0<\\gamma^{-1}\\beta t_k\\dotproduct{\\nabla f(x^k),-d^k}+\\gamma^{-1}t_k\\dotproduct{\\nabla f(x^k),d^k}+\\dfrac{L\\gamma^{-2}t_k^2}{2}\\dotproduct{\\nabla f(x^k),-d^k}.\n\\end{align*}\nDividing both sides above by $\\gamma^{-1}t_k\\dotproduct{\\nabla f(x^k),-d^k}>0$, we get $0<\\beta-1+\\dfrac{L\\gamma^{-1}t_k}{2}$.\nLetting $k\\overset{\\bar J}{\\longrightarrow}\\infty$ yields $\\beta\\ge 1$, which contradicts the choice of $\\beta\\in(0,1)$. Thus we verify that the sequence $\\set{t_k}_{k\\in J}$ is bounded away from zero, which completes the proof of the proposition.\n","sha256_utf8":"bec0243a8179baef2945cbec406c01493fde7fe489341c4a112fc5f72194a1a9"}]},{"id":"S091","semantic_type":"external_fact","kind":"external_fact","title":"局部光滑下降引理","source_units":[],"declared_external":true,"declaration_text":["若∇f在凸球上以常数L Lipschitz连续，则球内任意x、y满足f(x)−f(y)≤⟨∇f(y),x−y⟩+(L/2)‖x−y‖²；作者第1149行引用Bertsekas, Proposition A.24。"]},{"id":"S092","semantic_type":"proof","kind":"proof","title":"非空子序列关系论证","source_units":[{"start":103096,"end":103779,"start_line":1181,"end_line":1189,"raw_tex":"We have $j_{k+1}\\ge j_{k}+1$ whenever $k\\in\\N$. If the equality holds therein, then $z^{k+1}=z^{j_k+1}$. Otherwise, by taking into account that the indices $j_{k}+1,\\ldots,j_{k+1}-1$ correspond to null iterations, we get that\n\\begin{align}\\label{5 23}\nx^{j_k+1}=x^{j_{k}+2}=\\ldots=x^{j_{k+1}-1}=x^{j_{k+1}}=z^{k+1}.\n\\end{align}\nTherefore, it follows from $j_k\\notin\\mathcal{N}$ that\n\\begin{align}\\label{differ}\nz^{k+1}=x^{j_k+1}\\ne x^{j_k}=z^k\\;\\text{ for all }\\;k\\in\\N.\n\\end{align}\nFurthermore, Proposition~\\ref{null properties}(iv) tells us that $\\bar x$ is an accumulation point of $\\set{z^k}$ if and only if $\\bar x$ is also an accumulation point of $\\set{x^k}$.\\vspace*{0.05in}\n","sha256_utf8":"2ec670912505ddf4467585c9bd36df150aea29e9d7d4eb9d0b8d25423216705a"}]},{"id":"S093","semantic_type":"proof_step","kind":"proof_step","title":"跨越空迭代保持同一点","source_units":[{"start":103096,"end":103424,"start_line":1181,"end_line":1184,"raw_tex":"We have $j_{k+1}\\ge j_{k}+1$ whenever $k\\in\\N$. If the equality holds therein, then $z^{k+1}=z^{j_k+1}$. Otherwise, by taking into account that the indices $j_{k}+1,\\ldots,j_{k+1}-1$ correspond to null iterations, we get that\n\\begin{align}\\label{5 23}\nx^{j_k+1}=x^{j_{k}+2}=\\ldots=x^{j_{k+1}-1}=x^{j_{k+1}}=z^{k+1}.\n\\end{align}\n","sha256_utf8":"789256afbe71a313f40a755abafbb6d2c9ceac3aa0763283a7e49749ed188558"}]},{"id":"S094","semantic_type":"proof_step","kind":"proof_step","title":"非零增量及聚点等价","source_units":[{"start":103424,"end":103779,"start_line":1185,"end_line":1189,"raw_tex":"Therefore, it follows from $j_k\\notin\\mathcal{N}$ that\n\\begin{align}\\label{differ}\nz^{k+1}=x^{j_k+1}\\ne x^{j_k}=z^k\\;\\text{ for all }\\;k\\in\\N.\n\\end{align}\nFurthermore, Proposition~\\ref{null properties}(iv) tells us that $\\bar x$ is an accumulation point of $\\set{z^k}$ if and only if $\\bar x$ is also an accumulation point of $\\set{x^k}$.\\vspace*{0.05in}\n","sha256_utf8":"5b4ba5afbecbfa9ebaca7e201b3221dcbcf4b4902a74e3f455c0270c8a2385e6"}]},{"id":"S095","semantic_type":"external_fact","kind":"external_fact","title":"微积分与序列基本事实","source_units":[],"declared_external":true,"declaration_text":["使用连续映射保持极限、C¹函数在线段上的均值定理、有限维空间中绝对可和增量蕴含序列收敛、非负收敛级数尾和趋零，以及下降不等式的望远镜求和。"]},{"id":"S096","semantic_type":"external_fact","kind":"external_fact","title":"欧氏投影变分原理","source_units":[],"declared_external":true,"declaration_text":["非空闭凸集Ω上的欧氏投影唯一，且u=Proj(x,Ω)当且仅当u∈Ω并对所有y∈Ω有⟨y−u,x−u⟩≤0，即B0092中回顾的标准投影刻画。"]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S014","edges":[{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S015","kind":"supports"},{"source_id":"S018","target_id":"S015","kind":"supports"},{"source_id":"S019","target_id":"S015","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S019","kind":"supports"},{"source_id":"S017","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S079","target_id":"S016","kind":"supports"},{"source_id":"S084","target_id":"S016","kind":"supports"},{"source_id":"S009","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S017","kind":"supports"},{"source_id":"S013","target_id":"S017","kind":"supports"},{"source_id":"S092","target_id":"S017","kind":"supports"},{"source_id":"S067","target_id":"S018","kind":"supports"},{"source_id":"S075","target_id":"S018","kind":"supports"},{"source_id":"S009","target_id":"S018","kind":"supports"},{"source_id":"S013","target_id":"S018","kind":"supports"},{"source_id":"S092","target_id":"S018","kind":"supports"},{"source_id":"S039","target_id":"S019","kind":"supports"},{"source_id":"S006","target_id":"S019","kind":"supports"},{"source_id":"S094","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S023","target_id":"S021","kind":"supports"},{"source_id":"S024","target_id":"S021","kind":"supports"},{"source_id":"S025","target_id":"S021","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S007","target_id":"S024","kind":"supports"},{"source_id":"S002","target_id":"S023","kind":"supports"},{"source_id":"S095","target_id":"S025","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S020","target_id":"S030","kind":"supports"},{"source_id":"S003","target_id":"S030","kind":"supports"},{"source_id":"S005","target_id":"S030","kind":"supports"},{"source_id":"S007","target_id":"S030","kind":"supports"},{"source_id":"S001","target_id":"S030","kind":"supports"},{"source_id":"S002","target_id":"S030","kind":"supports"},{"source_id":"S095","target_id":"S030","kind":"supports"},{"source_id":"S003","target_id":"S031","kind":"supports"},{"source_id":"S026","target_id":"S031","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S006","target_id":"S033","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S034","kind":"supports"},{"source_id":"S032","target_id":"S035","kind":"supports"},{"source_id":"S006","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S095","target_id":"S036","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S042","target_id":"S040","kind":"supports"},{"source_id":"S043","target_id":"S040","kind":"supports"},{"source_id":"S044","target_id":"S040","kind":"supports"},{"source_id":"S045","target_id":"S040","kind":"supports"},{"source_id":"S033","target_id":"S041","kind":"supports"},{"source_id":"S034","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S034","target_id":"S042","kind":"supports"},{"source_id":"S006","target_id":"S043","kind":"supports"},{"source_id":"S041","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S042","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S038","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S049","target_id":"S047","kind":"supports"},{"source_id":"S009","target_id":"S048","kind":"supports"},{"source_id":"S008","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S009","target_id":"S049","kind":"supports"},{"source_id":"S008","target_id":"S049","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S051","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S010","target_id":"S052","kind":"supports"},{"source_id":"S011","target_id":"S052","kind":"supports"},{"source_id":"S009","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S009","target_id":"S053","kind":"supports"},{"source_id":"S011","target_id":"S053","kind":"supports"},{"source_id":"S095","target_id":"S053","kind":"supports"},{"source_id":"S010","target_id":"S054","kind":"supports"},{"source_id":"S011","target_id":"S054","kind":"supports"},{"source_id":"S009","target_id":"S054","kind":"supports"},{"source_id":"S056","target_id":"S055","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S058","target_id":"S056","kind":"supports"},{"source_id":"S059","target_id":"S056","kind":"supports"},{"source_id":"S050","target_id":"S057","kind":"supports"},{"source_id":"S009","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S050","target_id":"S058","kind":"supports"},{"source_id":"S010","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S059","kind":"supports"},{"source_id":"S050","target_id":"S059","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S055","target_id":"S062","kind":"supports"},{"source_id":"S009","target_id":"S062","kind":"supports"},{"source_id":"S005","target_id":"S062","kind":"supports"},{"source_id":"S064","target_id":"S063","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S064","kind":"supports"},{"source_id":"S046","target_id":"S065","kind":"supports"},{"source_id":"S001","target_id":"S065","kind":"supports"},{"source_id":"S012","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S012","target_id":"S066","kind":"supports"},{"source_id":"S068","target_id":"S067","kind":"supports"},{"source_id":"S069","target_id":"S068","kind":"supports"},{"source_id":"S070","target_id":"S068","kind":"supports"},{"source_id":"S071","target_id":"S068","kind":"supports"},{"source_id":"S072","target_id":"S068","kind":"supports"},{"source_id":"S073","target_id":"S068","kind":"supports"},{"source_id":"S074","target_id":"S068","kind":"supports"},{"source_id":"S063","target_id":"S069","kind":"supports"},{"source_id":"S012","target_id":"S069","kind":"supports"},{"source_id":"S009","target_id":"S069","kind":"supports"},{"source_id":"S095","target_id":"S069","kind":"supports"},{"source_id":"S069","target_id":"S070","kind":"supports"},{"source_id":"S050","target_id":"S070","kind":"supports"},{"source_id":"S012","target_id":"S070","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S095","target_id":"S071","kind":"supports"},{"source_id":"S069","target_id":"S072","kind":"supports"},{"source_id":"S070","target_id":"S072","kind":"supports"},{"source_id":"S050","target_id":"S072","kind":"supports"},{"source_id":"S009","target_id":"S072","kind":"supports"},{"source_id":"S008","target_id":"S072","kind":"supports"},{"source_id":"S095","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S070","target_id":"S073","kind":"supports"},{"source_id":"S008","target_id":"S073","kind":"supports"},{"source_id":"S095","target_id":"S073","kind":"supports"},{"source_id":"S060","target_id":"S074","kind":"supports"},{"source_id":"S055","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S069","target_id":"S074","kind":"supports"},{"source_id":"S012","target_id":"S074","kind":"supports"},{"source_id":"S028","target_id":"S074","kind":"supports"},{"source_id":"S076","target_id":"S075","kind":"supports"},{"source_id":"S077","target_id":"S076","kind":"supports"},{"source_id":"S078","target_id":"S076","kind":"supports"},{"source_id":"S009","target_id":"S077","kind":"supports"},{"source_id":"S011","target_id":"S077","kind":"supports"},{"source_id":"S077","target_id":"S078","kind":"supports"},{"source_id":"S009","target_id":"S078","kind":"supports"},{"source_id":"S080","target_id":"S079","kind":"supports"},{"source_id":"S081","target_id":"S080","kind":"supports"},{"source_id":"S082","target_id":"S080","kind":"supports"},{"source_id":"S083","target_id":"S080","kind":"supports"},{"source_id":"S012","target_id":"S081","kind":"supports"},{"source_id":"S001","target_id":"S081","kind":"supports"},{"source_id":"S067","target_id":"S081","kind":"supports"},{"source_id":"S095","target_id":"S081","kind":"supports"},{"source_id":"S081","target_id":"S082","kind":"supports"},{"source_id":"S075","target_id":"S082","kind":"supports"},{"source_id":"S012","target_id":"S082","kind":"supports"},{"source_id":"S009","target_id":"S082","kind":"supports"},{"source_id":"S082","target_id":"S083","kind":"supports"},{"source_id":"S011","target_id":"S083","kind":"supports"},{"source_id":"S009","target_id":"S083","kind":"supports"},{"source_id":"S033","target_id":"S083","kind":"supports"},{"source_id":"S006","target_id":"S083","kind":"supports"},{"source_id":"S081","target_id":"S083","kind":"supports"},{"source_id":"S085","target_id":"S084","kind":"supports"},{"source_id":"S086","target_id":"S085","kind":"supports"},{"source_id":"S087","target_id":"S085","kind":"supports"},{"source_id":"S088","target_id":"S085","kind":"supports"},{"source_id":"S089","target_id":"S085","kind":"supports"},{"source_id":"S090","target_id":"S085","kind":"supports"},{"source_id":"S012","target_id":"S086","kind":"supports"},{"source_id":"S050","target_id":"S086","kind":"supports"},{"source_id":"S009","target_id":"S086","kind":"supports"},{"source_id":"S086","target_id":"S087","kind":"supports"},{"source_id":"S009","target_id":"S087","kind":"supports"},{"source_id":"S095","target_id":"S087","kind":"supports"},{"source_id":"S087","target_id":"S088","kind":"supports"},{"source_id":"S091","target_id":"S088","kind":"supports"},{"source_id":"S004","target_id":"S088","kind":"supports"},{"source_id":"S088","target_id":"S089","kind":"supports"},{"source_id":"S086","target_id":"S089","kind":"supports"},{"source_id":"S012","target_id":"S089","kind":"supports"},{"source_id":"S089","target_id":"S090","kind":"supports"},{"source_id":"S046","target_id":"S090","kind":"supports"},{"source_id":"S086","target_id":"S090","kind":"supports"},{"source_id":"S012","target_id":"S090","kind":"supports"},{"source_id":"S050","target_id":"S013","kind":"supports"},{"source_id":"S011","target_id":"S013","kind":"supports"},{"source_id":"S093","target_id":"S092","kind":"supports"},{"source_id":"S094","target_id":"S092","kind":"supports"},{"source_id":"S013","target_id":"S093","kind":"supports"},{"source_id":"S011","target_id":"S093","kind":"supports"},{"source_id":"S009","target_id":"S093","kind":"supports"},{"source_id":"S093","target_id":"S094","kind":"supports"},{"source_id":"S050","target_id":"S094","kind":"supports"},{"source_id":"S011","target_id":"S094","kind":"supports"},{"source_id":"S096","target_id":"S008","kind":"supports"}]}}
{"record_id":"2204.09266v2","paper":{"title":"Hessian Averaging in Stochastic Newton Methods Achieves Superlinear Convergence","authors":["Sen Na","Michał Dereziński","Michael W. Mahoney"],"arxiv_id":"2204.09266","version":"v2","journal":"Mathematical Programming","doi":"10.1007/s10107-022-01913-5","abstract_url":"https://arxiv.org/abs/2204.09266v2","pdf_url":"https://arxiv.org/pdf/2204.09266v2"},"context":{"raw_tex":"We consider minimizing a smooth and strongly convex objective function:\n\\begin{equation}\\label{pro:1}\n\\min_{\\xb\\in \\mR^d}\\; f(\\xb),\n\\end{equation}\nwhere $f:\\mR^d\\rightarrow \\mR$ is twice continuously differentiable with $\\Hb(\\xb) = \\nabla^2 f(\\xb) \\in \\mR^{d\\times d}$ being its Hessian matrix. We suppose the Hessian $\\Hb(\\xb)$ satisfies\n\\begin{equation*}\n\\lambda_{\\min}\\cdot \\Ib \\preceq \\Hb(\\xb) \\preceq\\lambda_{\\max}\\cdot\\Ib, \\quad\\;  \\forall \\xb\\in\\mR^d,\n\\end{equation*}\nfor some constants $0<\\lambda_{\\min}\\leq\\lambda_{\\max}$, and we let $\\kappa = \\lambda_{\\max}/\\lambda_{\\min}$ denote its~condition number. We also let $\\xb^\\star = \\arg\\min_{\\xb\\in \\mR^d} f(\\xb)$ be the unique global~solution.\n\n\n\\begin{equation}\\label{equ:Newton}\n\\xb_{t+1} = \\xb_t - \\mu_t \\Hb_t^{-1}\\nabla f_t,\n\\end{equation}\n\nwhere $\\nabla f_t = \\nabla f(\\xb_t)$, $\\Hb_t = \\Hb(\\xb_t)$, and $\\mu_t$ is selected by passing a line search condition. Classical results indicate that Newton's method in \\eqref{equ:Newton} \\mbox{exploits}~a~global $Q$-linear convergence in the error of function value $f(\\xb_t)-f(\\tx)$; and a local $Q$-quadratic convergence in the iterate error $\\|\\xb_t-\\tx\\|$. More precisely, Newton's method has two phases, separated by a neighborhood \n\n\nA parallel line of research explores stochastic Newton methods, where a stochastic approximation $\\hHb_t$ is used in place of $\\Hb_t$ in \\eqref{equ:Newton}. The stochasticity of $\\hHb_t$ may come from evaluating the Hessian on a random subset of data points~(i.e., subsampling), or from projecting the Hessian onto a random subspace to~achieve~the dimension reduction (i.e., sketching). To unify different approaches, we consider in this paper a general Hessian approximation given by a \\textit{stochastic oracle}. In particular, we express the approximation $\\hHb(\\xb)$ at any point $\\xb$ by\n\\begin{equation}\\label{equ:error}\n\\hHb(\\xb) = \\Hb(\\xb) + \\Eb(\\xb),\n\\end{equation}\n\nwhere $\\Eb(\\xb)\\in \\mS^{d\\times d}$ is a symmetric random noise matrix following a certain distribution (conditional on $\\xb$) with mean zero. At iterate $\\xb_t$, we query the oracle to obtain an approximation $\\hHb_t = \\hHb(\\xb_t)$, which (implicitly) comes from generating a realization of the random matrix $\\Eb_t = \\Eb(\\xb_t)$, and then adding $\\Eb_t$ to the true Hessian $\\Hb_t$. We do not assume $\\Eb_t$ and $\\Hb_t$ are accessible to us.\n\n\nWe consider an online averaging scheme (we let $\\tHb_{-1} = \\0$):\n\\begin{equation}\\label{equ:ave:Hessian}\n\\tHb_t  = \\frac{w_{t-1}}{w_t} \\tHb_{t-1} + \\rbr{1 - \\frac{w_{t-1}}{w_t}}\\hHb_t, \\quad t = 0,1,2,\\ldots,\\\\\n\\end{equation}\n\nwhere $\\{w_t\\}_{t=-1}^{\\infty}$ is a prespecified increasing non-negative weight sequence~starting with $w_{-1} = 0$. By \\emph{online} we mean that we only keep an average Hessian $\\tHb_t$ in each iteration, and update it as \\eqref{equ:ave:Hessian} when we obtain a new Hessian estimate. This scheme is in contrast to keeping all the past Hessian estimates. By the scheme \\eqref{equ:ave:Hessian}, we note that, at iteration $t$, we re-scale the weights of $\\{\\hHb_i\\}_{t=0}^{t-1}$ equally by a factor of $w_{t-1}/w_t$, instead of completely redefining all the weights. We note that such an online averaging scheme is memory and computation efficient: compared to stochastic Newton methods without averaging, we only require an extra $O(d^2)$ space to save $\\tHb_t$ and $O(d^2)$ flops to update it, which is negligible in the setting of stochastic Newton methods where the Hessian vector product requires $O(d^2)$ flops and solving the exact Newton system requires $O(d^3)$ flops. In \\eqref{equ:ave:Hessian}, we use the ratio factors $w_{t-1}/w_t$ instead of direct weights merely to simplify our later presentation. Furthermore, \\eqref{equ:ave:Hessian} can be reformulated as a general weighted average as follows:\\vskip-0.32cm\n\\begin{equation}\\label{equ:z}\n\\tHb_t =\\sum_{i=0}^tz_{i,t}\\hHb_i,\\qquad\\text{with}\\quad\n  z_{i,t}\\coloneqq(w_i-w_{i-1})/w_t.\n\\end{equation}\n\n\\noindent In particular, by setting $w_t=t+1$, we obtain $z_{i,t} = 1/(t+1)$ and further have $\\tHb_t = \\frac1{t+1}\\sum_{i=0}^{t}\\hHb_i$. Thus, we recover simple uniform averaging. If we let the sequence $w_t$ grow faster-than-linearly,~this results in a weighted average that is skewed towards the more recent Hessian estimates. \n\n\n\\begin{algorithm}[!t]\n\\caption{Stochastic Newton Method with Hessian Averaging}\n\\label{alg:1}\n\\begin{algorithmic}[1]\n\\State \\textbf{Input}: iterate $\\xb_0$, weights $\\{w_t\\}_{t=-1}^{\\infty}$, scalars $\\beta\\in(0, 1/2)$, $\\rho\\in(0, 1)$, and $\\tHb_{-1} = \\0$;\n\\For{ $t = 0,1,2,\\ldots$}\n\\State Obtain a stochastic Hessian approximation $\\hHb_t = \\Hb_t + \\Eb_t$;\n\\State Compute the Hessian average via \\eqref{equ:ave:Hessian}: $\\tHb_t  = \\frac{w_{t-1}}{w_t} \\tHb_{t-1} + \\rbr{1 - \\frac{w_{t-1}}{w_t}}\\hHb_t$;\n\\State Compute the Newton direction $\\pb_t$ by solving: $\\tHb_t\\pb_t = -\\nabla f_t$;\\vspace{1mm}\n\\State \\textbf{if} $\\tHb_t\\pb_t = -\\nabla f_t$ is not solvable \\textbf{or} $\\nabla f_t^\\top\\pb_t\\geq 0$, \\ \\textbf{then} \\ skip iteration with~$\\xb_{t+1}=\\xb_t$; \n\\State Compute a stepsize $\\mu_t = \\rho^{j_t}$, where $j_t$ is the smallest nonnegative integer such that\n\\begin{equation*}\n\\text{Armijo condition}:\\hskip 0.5cm f(\\xb_t+\\mu_t\\pb_t) \\leq f(\\xb_t) + \\mu_t\\beta \\nabla f_t^\\top\\pb_t\n\\end{equation*}\n\\State Update $\\xb_{t+1} = \\xb_t +\\mu_t\\pb_t$;\n\\EndFor\n\\end{algorithmic}\n\\end{algorithm}\n\n\\noindent We mention that the superlinear rate is measured with respect to the error $\\|\\xb_t-\\ttxb\\|_{\\ttHb} \\coloneqq \\sqrt{(\\xb_t-\\ttxb)^\\top\\ttHb(\\xb_t-\\xb_0)}$ where~$\\ttHb = \\Hb(\\ttxb)$. Before introducing the main results in Theorems \\ref{thm:main-1} and \\ref{thm:main-2}, we summarize them in Table \\ref{tab:1}, showing the transition points for~two weight sequences. The transitions for general weights are provided~in Section~\\ref{sec:4}. We use $\\Upsilon$ to denote the \\textit{noise level} of stochastic Hessian oracle (see Assumption \\ref{ass:3} for a formal definition; typically $\\Upsilon = O(\\kappa)$). We also use $O(\\cdot)$ to suppress the logarithmic factors and the dependence on other constants except $\\Upsilon$ and $\\kappa$. We emphasize that all results of the~paper require that the (weighted) average of the errors $\\{\\Eb_i\\}_{i=0}^t$ is sufficiently concentrated; thus, they hold \\textit{with high probability}.\n\n\n\\noindent{\\bf Notation}: Throughout the paper, we use $\\Ib$ to denote the identity matrix,~and~$\\0$ to denote the zero vector or matrix. Their dimensions are clear from the~context. We use $\\|\\cdot\\|$ to denote the $\\ell_2$ norm for vectors and spectral norm for matrices.  For a positive semidefinite matrix $\\Ab$, we let $\\|\\xb\\|_{\\Ab} = \\sqrt{\\xb^\\top\\Ab\\xb}$. For two scalars $a, b$, $a\\vee b = \\max(a,b)$ and $a\\wedge b = \\min(a, b)$. For two matrices $\\Ab, \\Bb$, $\\Ab \\prec (\\preceq)\\Bb$ if $\\Bb - \\Ab$ is a positive (semi)definite matrix. Recall that we reserve the notation $\\lambda_{\\min}, \\lambda_{\\max}$ to denote the lower and upper bounds of the true~Hessian, and~$\\kappa = \\lambda_{\\max}/\\lambda_{\\min}$ is the condition number. \n\n\nGiven a sequence  of stochastic iterates $\\{\\xb_t\\}_{t=0}^{\\infty}$, we let $\\mF_0\\subseteq\\mF_1\\subseteq\\mF_2\\subseteq\\ldots$ be a filtration where $\\mF_t = \\sigma(\\xb_{0:t})$, $\\forall t\\geq 0$, is the $\\sigma$-algebra~generated~by~the randomness from $\\xb_0$ to $\\xb_t$. With such a filtration, we denote the~conditional~expectation by~$\\mE_t[\\cdot] = \\mE[\\cdot \\mid \\mF_t]$ and the conditional probability by $\\mP_t(\\cdot) = \\mP(\\cdot \\mid \\mF_t)$. We suppose $\\xb_0$ is deterministic, so that $\\mF_0$ is a trivial $\\sigma$-algebra. \n\n\nFor a given weight sequence $\\{w_t\\}_{t=-1}^\\infty$ with $w_{-1}=0$, the scheme \\eqref{equ:ave:Hessian} leads~to \\vskip-0.4cm\n\\begin{equation}\\label{equ:simple}\n\\tHb_t\\;\\; \\stackrel{\\mathclap{\\eqref{equ:z}}}{=}\\; \\sum_{i=0}^{t}z_{i,t}\\hHb_i \\ \\stackrel{\\mathclap{\\eqref{equ:error}}}{=} \\!\\!\\underbrace{\\sum_{i=0}^{t}z_{i,t}\\Hb_i}_{\\text{Hessian averaging}}\\!\\!+\\!\\!\\quad\\barEb_t\\qquad\\quad\\text{for} \\quad \\barEb_t\\coloneqq\\!\\!\\underbrace{\\sum_{i=0}^{t}z_{i,t}\\Eb_i }_{\\text{noise averaging}}\n\\end{equation}\n\n\\noindent where $z_{i,t} = (w_i - w_{i-1})/w_t$. Note that $\\sum_{i=0}^t z_{i,t}=1$ and $z_{i,t}\\propto z_i\\coloneqq w_i-w_{i-1}$, i.e., $z_{i,t}$ is proportional to an un-normalized weight $z_i$. We see from \\eqref{equ:simple} that~$\\tHb_t$ consists of the Hessian averaging and noise averaging with the same weights. In principle, the Hessian averaging $\\sum_{i=0}^{t}z_{i,t}\\Hb_i\\rightarrow \\ttHb$ as $\\xb_t\\rightarrow \\tx$, while the noise averaging $\\sum_{i=0}^{t}z_{i,t}\\Eb_i\\rightarrow \\0$ due to the central limit~theorem. We will show that the Hessian averaging (eventually) converges faster than the noise averaging.\n\n\nTo study the concentration of noise averaging $\\barEb_t$, we use the fact that~$\\{\\Eb_t\\}_{t=0}^\\infty$ is a martingale difference sequence, and rely on concentration inequalities for matrix martingales. These concentration inequalities require a sub-exponential tail condition on the noise.  We say that a random variable $X$ is $K$-sub-exponential if $\\mE[|X|^p]\\leq p!\\cdot K^p/2$ for all $p=2,3,...\\,$, which is consistent (up to constants) with all standard notions of sub-exponentiality \\citep[see Section 2.7 in][]{Vershynin2018High}.\n\n\n\\begin{assumption}[Sub-exponential noise]\\label{ass:3}\nWe assume that  $\\Eb(\\xb)$ is mean zero and $\\|\\Eb(\\xb)\\|$ is $\\Upsilon_E$-sub-exponential for all $\\xb$. Also, we define $\\Upsilon\\coloneqq\\Upsilon_E/\\lambda_{\\min}$ to be the scale-invariant noise level. \n\\end{assumption}\n\n\\begin{assumption}[Lipschitz Hessian]\\label{ass:4}\nWe assume $\\Hb(\\xb)$ is $L$-Lipschitz continuous. That is $\\|\\Hb(\\xb_1) - \\Hb(\\xb_2)\\|\\leq L\\|\\xb_1 - \\xb_2\\|$ for any $\\xb_1, \\xb_2\\in \\mR^d$.\n\\end{assumption}","units":[{"start":4450,"end":5151,"start_line":99,"end_line":107,"raw_tex":"We consider minimizing a smooth and strongly convex objective function:\n\\begin{equation}\\label{pro:1}\n\\min_{\\xb\\in \\mR^d}\\; f(\\xb),\n\\end{equation}\nwhere $f:\\mR^d\\rightarrow \\mR$ is twice continuously differentiable with $\\Hb(\\xb) = \\nabla^2 f(\\xb) \\in \\mR^{d\\times d}$ being its Hessian matrix. We suppose the Hessian $\\Hb(\\xb)$ satisfies\n\\begin{equation*}\n\\lambda_{\\min}\\cdot \\Ib \\preceq \\Hb(\\xb) \\preceq\\lambda_{\\max}\\cdot\\Ib, \\quad\\;  \\forall \\xb\\in\\mR^d,\n\\end{equation*}\nfor some constants $0<\\lambda_{\\min}\\leq\\lambda_{\\max}$, and we let $\\kappa = \\lambda_{\\max}/\\lambda_{\\min}$ denote its~condition number. We also let $\\xb^\\star = \\arg\\min_{\\xb\\in \\mR^d} f(\\xb)$ be the unique global~solution.\n","sha256_utf8":"001cbb4de324145ce1ae439b592bd025806ee66ff0b729c2e426fc8f833aec81"},{"start":5733,"end":5830,"start_line":111,"end_line":113,"raw_tex":"\\begin{equation}\\label{equ:Newton}\n\\xb_{t+1} = \\xb_t - \\mu_t \\Hb_t^{-1}\\nabla f_t,\n\\end{equation}","sha256_utf8":"49caad6e922d7bc26767e1140ae8058948f78d57744381e6d6258ee6b1a3098e"},{"start":5831,"end":6272,"start_line":114,"end_line":114,"raw_tex":"where $\\nabla f_t = \\nabla f(\\xb_t)$, $\\Hb_t = \\Hb(\\xb_t)$, and $\\mu_t$ is selected by passing a line search condition. Classical results indicate that Newton's method in \\eqref{equ:Newton} \\mbox{exploits}~a~global $Q$-linear convergence in the error of function value $f(\\xb_t)-f(\\tx)$; and a local $Q$-quadratic convergence in the iterate error $\\|\\xb_t-\\tx\\|$. More precisely, Newton's method has two phases, separated by a neighborhood \n","sha256_utf8":"1c3cb30ecaf98d7e79c32f85da76943a5221e01692133e17080b8856b4c1c9d0"},{"start":9446,"end":10120,"start_line":133,"end_line":136,"raw_tex":"A parallel line of research explores stochastic Newton methods, where a stochastic approximation $\\hHb_t$ is used in place of $\\Hb_t$ in \\eqref{equ:Newton}. The stochasticity of $\\hHb_t$ may come from evaluating the Hessian on a random subset of data points~(i.e., subsampling), or from projecting the Hessian onto a random subspace to~achieve~the dimension reduction (i.e., sketching). To unify different approaches, we consider in this paper a general Hessian approximation given by a \\textit{stochastic oracle}. In particular, we express the approximation $\\hHb(\\xb)$ at any point $\\xb$ by\n\\begin{equation}\\label{equ:error}\n\\hHb(\\xb) = \\Hb(\\xb) + \\Eb(\\xb),\n\\end{equation}","sha256_utf8":"844efd468e353c9913c57f3ae5a2f43af9c226f2938d731bcadb801fca718dfb"},{"start":10121,"end":10565,"start_line":137,"end_line":137,"raw_tex":"where $\\Eb(\\xb)\\in \\mS^{d\\times d}$ is a symmetric random noise matrix following a certain distribution (conditional on $\\xb$) with mean zero. At iterate $\\xb_t$, we query the oracle to obtain an approximation $\\hHb_t = \\hHb(\\xb_t)$, which (implicitly) comes from generating a realization of the random matrix $\\Eb_t = \\Eb(\\xb_t)$, and then adding $\\Eb_t$ to the true Hessian $\\Hb_t$. We do not assume $\\Eb_t$ and $\\Hb_t$ are accessible to us.\n","sha256_utf8":"99b061de9d268b9208fdf9fe4deed7dfae3aa7cd9214e78e0f133541f71090d9"},{"start":20751,"end":20977,"start_line":206,"end_line":209,"raw_tex":"We consider an online averaging scheme (we let $\\tHb_{-1} = \\0$):\n\\begin{equation}\\label{equ:ave:Hessian}\n\\tHb_t  = \\frac{w_{t-1}}{w_t} \\tHb_{t-1} + \\rbr{1 - \\frac{w_{t-1}}{w_t}}\\hHb_t, \\quad t = 0,1,2,\\ldots,\\\\\n\\end{equation}","sha256_utf8":"8ffa6c349d5d924fa668363fffcf11e62e98c7abcefba574df54d1e1ea4f8771"},{"start":20978,"end":22344,"start_line":210,"end_line":214,"raw_tex":"where $\\{w_t\\}_{t=-1}^{\\infty}$ is a prespecified increasing non-negative weight sequence~starting with $w_{-1} = 0$. By \\emph{online} we mean that we only keep an average Hessian $\\tHb_t$ in each iteration, and update it as \\eqref{equ:ave:Hessian} when we obtain a new Hessian estimate. This scheme is in contrast to keeping all the past Hessian estimates. By the scheme \\eqref{equ:ave:Hessian}, we note that, at iteration $t$, we re-scale the weights of $\\{\\hHb_i\\}_{t=0}^{t-1}$ equally by a factor of $w_{t-1}/w_t$, instead of completely redefining all the weights. We note that such an online averaging scheme is memory and computation efficient: compared to stochastic Newton methods without averaging, we only require an extra $O(d^2)$ space to save $\\tHb_t$ and $O(d^2)$ flops to update it, which is negligible in the setting of stochastic Newton methods where the Hessian vector product requires $O(d^2)$ flops and solving the exact Newton system requires $O(d^3)$ flops. In \\eqref{equ:ave:Hessian}, we use the ratio factors $w_{t-1}/w_t$ instead of direct weights merely to simplify our later presentation. Furthermore, \\eqref{equ:ave:Hessian} can be reformulated as a general weighted average as follows:\\vskip-0.32cm\n\\begin{equation}\\label{equ:z}\n\\tHb_t =\\sum_{i=0}^tz_{i,t}\\hHb_i,\\qquad\\text{with}\\quad\n  z_{i,t}\\coloneqq(w_i-w_{i-1})/w_t.\n\\end{equation}","sha256_utf8":"08f4d79a53db8506df87be0bf78f4ae730cd67bc95d8a35221724b93aa1714bc"},{"start":22357,"end":22688,"start_line":215,"end_line":215,"raw_tex":"\\noindent In particular, by setting $w_t=t+1$, we obtain $z_{i,t} = 1/(t+1)$ and further have $\\tHb_t = \\frac1{t+1}\\sum_{i=0}^{t}\\hHb_i$. Thus, we recover simple uniform averaging. If we let the sequence $w_t$ grow faster-than-linearly,~this results in a weighted average that is skewed towards the more recent Hessian estimates. \n","sha256_utf8":"3438d5b3ec1a4dcf7d148bc0a4c998e1513b8428b4c8afce3158ffd9bc31cc72"},{"start":22691,"end":23790,"start_line":219,"end_line":236,"raw_tex":"\\begin{algorithm}[!t]\n\\caption{Stochastic Newton Method with Hessian Averaging}\n\\label{alg:1}\n\\begin{algorithmic}[1]\n\\State \\textbf{Input}: iterate $\\xb_0$, weights $\\{w_t\\}_{t=-1}^{\\infty}$, scalars $\\beta\\in(0, 1/2)$, $\\rho\\in(0, 1)$, and $\\tHb_{-1} = \\0$;\n\\For{ $t = 0,1,2,\\ldots$}\n\\State Obtain a stochastic Hessian approximation $\\hHb_t = \\Hb_t + \\Eb_t$;\n\\State Compute the Hessian average via \\eqref{equ:ave:Hessian}: $\\tHb_t  = \\frac{w_{t-1}}{w_t} \\tHb_{t-1} + \\rbr{1 - \\frac{w_{t-1}}{w_t}}\\hHb_t$;\n\\State Compute the Newton direction $\\pb_t$ by solving: $\\tHb_t\\pb_t = -\\nabla f_t$;\\vspace{1mm}\n\\State \\textbf{if} $\\tHb_t\\pb_t = -\\nabla f_t$ is not solvable \\textbf{or} $\\nabla f_t^\\top\\pb_t\\geq 0$, \\ \\textbf{then} \\ skip iteration with~$\\xb_{t+1}=\\xb_t$; \n\\State Compute a stepsize $\\mu_t = \\rho^{j_t}$, where $j_t$ is the smallest nonnegative integer such that\n\\begin{equation*}\n\\text{Armijo condition}:\\hskip 0.5cm f(\\xb_t+\\mu_t\\pb_t) \\leq f(\\xb_t) + \\mu_t\\beta \\nabla f_t^\\top\\pb_t\n\\end{equation*}\n\\State Update $\\xb_{t+1} = \\xb_t +\\mu_t\\pb_t$;\n\\EndFor\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"e5ed1828ddfcb3cdd24c100b105322c8147a257d1f95d3fbab572666120bff72"},{"start":27003,"end":27942,"start_line":262,"end_line":262,"raw_tex":"\\noindent We mention that the superlinear rate is measured with respect to the error $\\|\\xb_t-\\ttxb\\|_{\\ttHb} \\coloneqq \\sqrt{(\\xb_t-\\ttxb)^\\top\\ttHb(\\xb_t-\\xb_0)}$ where~$\\ttHb = \\Hb(\\ttxb)$. Before introducing the main results in Theorems \\ref{thm:main-1} and \\ref{thm:main-2}, we summarize them in Table \\ref{tab:1}, showing the transition points for~two weight sequences. The transitions for general weights are provided~in Section~\\ref{sec:4}. We use $\\Upsilon$ to denote the \\textit{noise level} of stochastic Hessian oracle (see Assumption \\ref{ass:3} for a formal definition; typically $\\Upsilon = O(\\kappa)$). We also use $O(\\cdot)$ to suppress the logarithmic factors and the dependence on other constants except $\\Upsilon$ and $\\kappa$. We emphasize that all results of the~paper require that the (weighted) average of the errors $\\{\\Eb_i\\}_{i=0}^t$ is sufficiently concentrated; thus, they hold \\textit{with high probability}.\n","sha256_utf8":"5a7ea5ff53b7b8d8854d00aa6ad1fe0e40fe797e1069cd210f11efacadfce0ee"},{"start":43624,"end":44368,"start_line":346,"end_line":346,"raw_tex":"\\noindent{\\bf Notation}: Throughout the paper, we use $\\Ib$ to denote the identity matrix,~and~$\\0$ to denote the zero vector or matrix. Their dimensions are clear from the~context. We use $\\|\\cdot\\|$ to denote the $\\ell_2$ norm for vectors and spectral norm for matrices.  For a positive semidefinite matrix $\\Ab$, we let $\\|\\xb\\|_{\\Ab} = \\sqrt{\\xb^\\top\\Ab\\xb}$. For two scalars $a, b$, $a\\vee b = \\max(a,b)$ and $a\\wedge b = \\min(a, b)$. For two matrices $\\Ab, \\Bb$, $\\Ab \\prec (\\preceq)\\Bb$ if $\\Bb - \\Ab$ is a positive (semi)definite matrix. Recall that we reserve the notation $\\lambda_{\\min}, \\lambda_{\\max}$ to denote the lower and upper bounds of the true~Hessian, and~$\\kappa = \\lambda_{\\max}/\\lambda_{\\min}$ is the condition number. \n","sha256_utf8":"743f083b742b238d7873e6ac8c1c1dff3055bc176e361c49efb28661ee09ea5a"},{"start":45298,"end":45836,"start_line":357,"end_line":357,"raw_tex":"Given a sequence  of stochastic iterates $\\{\\xb_t\\}_{t=0}^{\\infty}$, we let $\\mF_0\\subseteq\\mF_1\\subseteq\\mF_2\\subseteq\\ldots$ be a filtration where $\\mF_t = \\sigma(\\xb_{0:t})$, $\\forall t\\geq 0$, is the $\\sigma$-algebra~generated~by~the randomness from $\\xb_0$ to $\\xb_t$. With such a filtration, we denote the~conditional~expectation by~$\\mE_t[\\cdot] = \\mE[\\cdot \\mid \\mF_t]$ and the conditional probability by $\\mP_t(\\cdot) = \\mP(\\cdot \\mid \\mF_t)$. We suppose $\\xb_0$ is deterministic, so that $\\mF_0$ is a trivial $\\sigma$-algebra. \n","sha256_utf8":"200add066c7ee79697846f94b2e6cf1647cb8ade6b63219da7be40ac4180bb43"},{"start":45838,"end":46345,"start_line":360,"end_line":363,"raw_tex":"For a given weight sequence $\\{w_t\\}_{t=-1}^\\infty$ with $w_{-1}=0$, the scheme \\eqref{equ:ave:Hessian} leads~to \\vskip-0.4cm\n\\begin{equation}\\label{equ:simple}\n\\tHb_t\\;\\; \\stackrel{\\mathclap{\\eqref{equ:z}}}{=}\\; \\sum_{i=0}^{t}z_{i,t}\\hHb_i \\ \\stackrel{\\mathclap{\\eqref{equ:error}}}{=} \\!\\!\\underbrace{\\sum_{i=0}^{t}z_{i,t}\\Hb_i}_{\\text{Hessian averaging}}\\!\\!+\\!\\!\\quad\\barEb_t\\qquad\\quad\\text{for} \\quad \\barEb_t\\coloneqq\\!\\!\\underbrace{\\sum_{i=0}^{t}z_{i,t}\\Eb_i }_{\\text{noise averaging}}\n\\end{equation}","sha256_utf8":"01cf80e05a6386bbbb2824c17fd0520bcd4f1acfc94292f16f58f341179c6062"},{"start":46358,"end":46985,"start_line":364,"end_line":364,"raw_tex":"\\noindent where $z_{i,t} = (w_i - w_{i-1})/w_t$. Note that $\\sum_{i=0}^t z_{i,t}=1$ and $z_{i,t}\\propto z_i\\coloneqq w_i-w_{i-1}$, i.e., $z_{i,t}$ is proportional to an un-normalized weight $z_i$. We see from \\eqref{equ:simple} that~$\\tHb_t$ consists of the Hessian averaging and noise averaging with the same weights. In principle, the Hessian averaging $\\sum_{i=0}^{t}z_{i,t}\\Hb_i\\rightarrow \\ttHb$ as $\\xb_t\\rightarrow \\tx$, while the noise averaging $\\sum_{i=0}^{t}z_{i,t}\\Eb_i\\rightarrow \\0$ due to the central limit~theorem. We will show that the Hessian averaging (eventually) converges faster than the noise averaging.\n","sha256_utf8":"eb6695e078e39d4022213c6f208603c963d567af1b4d9706ab4990600d054fb6"},{"start":46987,"end":47529,"start_line":367,"end_line":367,"raw_tex":"To study the concentration of noise averaging $\\barEb_t$, we use the fact that~$\\{\\Eb_t\\}_{t=0}^\\infty$ is a martingale difference sequence, and rely on concentration inequalities for matrix martingales. These concentration inequalities require a sub-exponential tail condition on the noise.  We say that a random variable $X$ is $K$-sub-exponential if $\\mE[|X|^p]\\leq p!\\cdot K^p/2$ for all $p=2,3,...\\,$, which is consistent (up to constants) with all standard notions of sub-exponentiality \\citep[see Section 2.7 in][]{Vershynin2018High}.\n","sha256_utf8":"e42794037fdfa816f2e48be73de3d976ee82b3ef9930924f36c4b09a58e181e2"},{"start":47530,"end":47808,"start_line":369,"end_line":371,"raw_tex":"\\begin{assumption}[Sub-exponential noise]\\label{ass:3}\nWe assume that  $\\Eb(\\xb)$ is mean zero and $\\|\\Eb(\\xb)\\|$ is $\\Upsilon_E$-sub-exponential for all $\\xb$. Also, we define $\\Upsilon\\coloneqq\\Upsilon_E/\\lambda_{\\min}$ to be the scale-invariant noise level. \n\\end{assumption}","sha256_utf8":"29266f1d1ddceb482cc2228625bd470b4681e41f52db4a67d9eca6ad19d19ac8"},{"start":57445,"end":57656,"start_line":491,"end_line":493,"raw_tex":"\\begin{assumption}[Lipschitz Hessian]\\label{ass:4}\nWe assume $\\Hb(\\xb)$ is $L$-Lipschitz continuous. That is $\\|\\Hb(\\xb_1) - \\Hb(\\xb_2)\\|\\leq L\\|\\xb_1 - \\xb_2\\|$ for any $\\xb_1, \\xb_2\\in \\mR^d$.\n\\end{assumption}","sha256_utf8":"b4bbf97df1476ff0ac29d6f4b9d87ca8fd935c6cb00936ec5c2d0df1423ade26"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{thm:3}\t\nConsider Algorithm \\ref{alg:1} with $w_t = t+1$, $\\forall t\\geq 0$. Under Assumptions \\ref{ass:3}, \\ref{ass:4}, we let $\\delta \\in(0, 1)$ satisfy $d/\\delta\\geq e$, and let $\\epsilon, \\nu\\in(0, 1)$ satisfy\n\\begin{equation}\\label{cond:epsnu}\n\\epsilon \\vee \\nu \\leq \\frac{1}{3}\\frac{0.5-\\beta}{1.5-\\beta}\\wedge \\frac{1}{48}.\n\\end{equation}\nDefine the neighborhood $\\N_{\\nu}$ as in \\eqref{Nnu}, and define $\\T = \\T_1 +\\T_2$ with $\\T_1$ given~by \\eqref{T1} and $\\T_2$ given by \\eqref{T2}. We also let $\\J = 4\\T\\kappa/\\nu$. Then, with probability $1-\\delta\\pi^2/6$, we have that $\\xb_{\\T_1:\\T+\\J}$ converges $R$-linearly, $\\xb_{\\T:\\T+\\J}\\in \\N_{\\nu}$, and\n\\begin{equation*}\n\\|\\xb_{\\T+\\J +t +1} - \\ttxb\\|_{\\ttHb} \\leq 12 \\rho_t\\|\\xb_{\\T+\\J +t} - \\ttxb\\|_{\\ttHb}, \\quad \\forall t\\geq 0\n\\end{equation*}\nwith\n\\begin{equation*}\n\\rho_t = \\frac{4\\T\\kappa}{\\T+\\J+t+1} + 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}}\n\\end{equation*}\nsatisfying $24\\rho_t\\leq 1$, $\\forall t\\geq 0$.\n\n\\end{theorem}","units":[{"start":60492,"end":61516,"start_line":556,"end_line":571,"raw_tex":"\\begin{theorem}\\label{thm:3}\t\nConsider Algorithm \\ref{alg:1} with $w_t = t+1$, $\\forall t\\geq 0$. Under Assumptions \\ref{ass:3}, \\ref{ass:4}, we let $\\delta \\in(0, 1)$ satisfy $d/\\delta\\geq e$, and let $\\epsilon, \\nu\\in(0, 1)$ satisfy\n\\begin{equation}\\label{cond:epsnu}\n\\epsilon \\vee \\nu \\leq \\frac{1}{3}\\frac{0.5-\\beta}{1.5-\\beta}\\wedge \\frac{1}{48}.\n\\end{equation}\nDefine the neighborhood $\\N_{\\nu}$ as in \\eqref{Nnu}, and define $\\T = \\T_1 +\\T_2$ with $\\T_1$ given~by \\eqref{T1} and $\\T_2$ given by \\eqref{T2}. We also let $\\J = 4\\T\\kappa/\\nu$. Then, with probability $1-\\delta\\pi^2/6$, we have that $\\xb_{\\T_1:\\T+\\J}$ converges $R$-linearly, $\\xb_{\\T:\\T+\\J}\\in \\N_{\\nu}$, and\n\\begin{equation*}\n\\|\\xb_{\\T+\\J +t +1} - \\ttxb\\|_{\\ttHb} \\leq 12 \\rho_t\\|\\xb_{\\T+\\J +t} - \\ttxb\\|_{\\ttHb}, \\quad \\forall t\\geq 0\n\\end{equation*}\nwith\n\\begin{equation*}\n\\rho_t = \\frac{4\\T\\kappa}{\\T+\\J+t+1} + 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}}\n\\end{equation*}\nsatisfying $24\\rho_t\\leq 1$, $\\forall t\\geq 0$.\n\n\\end{theorem}","sha256_utf8":"7a874d6a41c1461869f8c30050f3e62ae7f47b11f4c9fdd19acabf6d8d3fdf1e"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}[Adapted from Theorem 2.3 in \\cite{Tropp2011Freedmans}]\\label{lem:2}\nLet $t\\geq 0$ be a fixed integer. Consider a $d$-dimensional martingale difference $\\{\\Eb_i\\}_{i=0}^{t}$ (i.e., $\\mE_i[\\Eb_i] = \\0$). Suppose there exists a function $g_t:\\Theta_t\\rightarrow [0, \\infty]$ with $\\Theta_t\\subseteq (0, \\infty)$, and~a~sequence of matrices $\\{\\Ub_i\\}_{i=0}^{t}$, such that for any $i = 0,1,\\ldots,t$,\\footnote{The matrix exponential is defined by power series expansion: $\\exp(\\Ab) = \\Ib + \\sum_{i=1}^{\\infty} \\Ab^i/ i!$.}\n\\begin{equation}\\label{equ:exp:cond}\n\\mE_i\\sbr{\\exp\\rbr{\\theta\\Eb_i}}\\preceq \\exp\\rbr{g_t(\\theta)\\Ub_i} \\quad \\text{ almost surely for each } \\theta\\in \\Theta_t.\n\\end{equation}\nThen, we have for any scalars $\\eta \\geq 0$ and $\\sigma^2>0$,\n\\begin{equation*}\n\\mP\\rbr{\\Big\\|\\sum_{i=0}^{t}\\Eb_i\\Big\\| \\geq \\eta\\; \\text{ and } \\; \\Big\\|\\sum_{i=0}^{t}\\Ub_i\\Big\\|\\leq \\sigma^2} \\leq 2d\\cdot \\inf_{\\theta\\in \\Theta_t}\\exp\\rbr{-\\theta \\eta + g_t(\\theta)\\sigma^2}.\n\\end{equation*}\t\t\n\\end{lemma}\n\n\\begin{theorem}[Concentration of sub-exponential martingale difference]\\label{thm:2}\nUnder Assumption \\ref{ass:3}, for any integer $t\\geq 0$ and scalar $\\eta\\geq 0$, $\\barEb_t$ in \\eqref{equ:simple} satisfies\n\\begin{equation}\\label{equ:subE}\n\\mP\\rbr{\\nbr{\\barEb_t} \\geq \\eta} \\leq 2d\\cdot\\exp\\rbr{-\\frac{\\eta^2/2}{\\Upsilon_E^2\\sum_{i=0}^{t}z_{i,t}^2 + z_t^{(\\max)}\\Upsilon_E\\eta } }\n\\end{equation}\nwhere $z_t^{(\\max)} = \\max_{i\\in \\{0,\\ldots,t\\}}z_{i,t}$.\n\\end{theorem}\n\n\\begin{proof}\nSee Appendix \\ref{pf:thm:2}.\n\\end{proof}\n\nThe martingale concentration in Theorem \\ref{thm:2} matches the matrix Bernstein results for independent noises $\\{\\Eb_i\\}_{i=0}^t$ \\citep[cf. Theorems 6.1, 6.2 in][]{Tropp2011User}. For any~$\\delta\\in(0, 1)$, if we let the right hand side of \\eqref{equ:subE} be $\\delta/(t+1)^2$, then we obtain that, with probability at least $1 - \\delta/(t+1)^2$,  \n\\begin{equation}\\label{equ:sub:concen}\n\\small \\nbr{\\barEb_t} \\leq 8\\Upsilon_E\\sqrt{\\log\\rbr{\\frac{d(t+1)}{\\delta}}}\\rbr{\\sqrt{\\sum_{i=0}^{t}z_{i,t}^2} \\; \\vee \\; \\sqrt{\\log\\rbr{\\frac{d(t+1)}{\\delta}}} \\cdot z_t^{(\\max)}}.\n\\end{equation}\n\n\nTo achieve the fastest concentration rate, we minimize the right~hand side of \\eqref{equ:sub:concen} under the~restriction $\\sum_{i=0}^t z_{i,t} = 1$. Note that the minimum of both $\\sum_{i=0}^t z_{i,t}^2$ and $z_t^{(\\max)}$ is attained with equal weights, that is $z_{i,t} = 1/(t+1)$, for $ i = 0,1,\\ldots, t$. Thus, the fastest concentration rate is attained~with~equal~weights. Furthermore, a union bound over $t$ leads to\n\\begin{equation}\\label{equ:equal:con}\n\\mP\\rbr{\\forall t:\\ \\nbr{\\barEb_t} \\leq\n8\\Upsilon_E\\bigg(\\sqrt{\\frac{\\log(d(t+1)/\\delta)}{t+1}} \\vee\n  \\frac{\\log(d(t+1)/\\delta)}{t+1}\\bigg)} \n\\geq 1 - \\sum_{t=0}^{\\infty}\n\\frac{\\delta}{(t+1)^2}= 1-\\frac{\\pi^2\\delta}{6}.\n\\end{equation}\n\n\n\\begin{lemma}\\label{lem:3}\nConsider Algorithm \\ref{alg:1} with $w_t = t+1$, $\\forall t\\geq 0$. Under Assumption~\\ref{ass:3}, we let $\\delta, \\epsilon\\in(0, 1)$ with $d/\\delta\\geq e$. We also let\n\\begin{equation}\\label{T1}\n\\T_1 =4\\rbr{1\\vee (8\\Upsilon/\\epsilon)}^2 \\log\\rbr{d/\\delta\\cdot\\cbr{1\\vee(8\\Upsilon/\\epsilon)}}.\n\\end{equation}\nThen, with probability $1-\\delta\\pi^2/6$, the event\n\\begin{equation}\\label{event:EE}\n\\EE = \\bigcap_{t=\\T_1}^{\\infty}\\cbr{\\|\\barEb_t\\| \\leq 8\\Upsilon_E\\sqrt{\\frac{\\log(d(t+1)/\\delta)}{t+1}}}\n\\end{equation}\noccurs, which implies $(1-\\epsilon)\\lambda_{\\min}\\cdot\\Ib\\preceq \\tHb_t \\preceq (1+\\epsilon)\\lambda_{\\max}\\cdot\\Ib$, $\\forall t\\geq \\T_1$.\n\\end{lemma}\n\n\\begin{proof}\nSee Appendix \\ref{pf:lem:3}. \n\\end{proof}\n\n\\begin{lemma}\\label{lem:4}\nConditioning on the event \\eqref{event:EE}, we let\n\\begin{equation*}\n\\phi = \\frac{4\\rho\\beta(1-\\beta)(1-\\epsilon)}{\\kappa^2(1+\\epsilon)}\n\\end{equation*}\nand have $f(\\xb_{t+1})-f(\\ttxb) \\leq (1-\\phi)(f(\\xb_t) - f(\\ttxb))$, $\\forall t\\geq \\T_1$, which implies $R$-linear convergence of the iterate error,\n\\begin{align*}\n\\|\\xb_t - \\ttxb\\| &\\leq \\cbr{\\frac{2}{\\lambda_{\\min}}(f(\\xb_0) - f(\\ttxb))(1-\\phi)^{t-\\T_1}}^{1/2}, \\quad t\\geq \\T_1,\\\\\n\\|\\xb_t - \\ttxb\\|_{\\ttHb} &\\leq \\cbr{2\\kappa(f(\\xb_0) - f(\\ttxb))(1-\\phi)^{t-\\T_1}}^{1/2}, \\quad\\quad\\; t\\geq \\T_1.\n\\end{align*}\n\\end{lemma}\n\n\\begin{proof}\nSee Appendix \\ref{pf:lem:4}.\n\\end{proof}\n\n\\begin{corollary}\\label{cor:1}\nConsider Algorithm \\ref{alg:1} with $w_t=t+1$, $\\forall t\\geq 0$. Under Assumptions \\ref{ass:3} and \\ref{ass:4}, we let $\\delta, \\epsilon\\in(0, 1)$ with $d/\\delta \\geq e$, and define the neighborhood $\\N_{\\nu}$ as \n\\begin{equation}\\label{Nnu}\n\\N_\\nu = \\{\\xb: \\|\\xb - \\ttxb\\|_{\\ttHb} \\leq \\nu \\cdot \\lambda_{\\min}^{3/2}/L\\}\\quad \\text{ for } \\nu \\in (0, 1].\n\\end{equation}\nThen, with probability $1 - \\delta\\pi^2/6$, \nwe have $\\xb_t\\in \\N_\\nu$, for all $t\\geq \\T$ where~$\\T = \\T_1 + \\T_2$ with $\\T_1$ defined in \\eqref{T1} and\n\\begin{equation}\\label{T2}\n\\T_2 =  \\frac {\\kappa^2(1+\\epsilon)}{4\\rho\\beta(1-\\beta)(1-\\epsilon)}\\log\\rbr{\\frac{3L^2(f(\\xb_0) - f(\\ttxb))}{\\nu^2\\lambda_{\\min}^3}}.\n\\end{equation}\t\n\\end{corollary}\n\n\\begin{proof}\nSee Appendix \\ref{pf:cor:1}.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:5}\nUnder Assumption \\ref{ass:4}, suppose $\\pb_t = -(\\tHb_t)^{-1}\\nabla f_t$. Then $\\mu_t = 1$ if $\\xb_t \\in \\N_{\\nu}$ and $(1-\\psi)\\Hb_t \\preceq \\tHb_t \\preceq(1+\\psi)\\Hb_t$ with $\\nu, \\psi$ satisfying\n\\begin{equation}\\label{cond:nupsi}\n0<\\nu \\leq \\frac{2}{3}(1/2-\\beta), \\quad\\quad 0<\\psi\\leq \\frac{1/2-\\beta}{3/2-\\beta}.\n\\end{equation}\n\\end{lemma}\n\n\\begin{proof}\nSee Appendix \\ref{pf:lem:5}.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:6}\nUnder Assumption \\ref{ass:4} and suppose $\\pb_t = -(\\tHb_t)^{-1}\\nabla f_t$, $\\xb_t \\in \\N_{\\nu}$, and $(1-\\psi)\\Hb_t \\preceq \\tHb_t \\preceq(1+\\psi)\\Hb_t$ with $\\nu, \\psi$ satisfying \\eqref{cond:nupsi}. Then, we have\n\\begin{equation*}\n\\nbr{\\xb_{t+1} - \\ttxb}_{\\ttHb} \\leq 3\\cbr{ \\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_t-\\ttxb\\|_{\\ttHb}^2 + \\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| \\cdot \\|\\xb_t - \\ttxb\\|_{\\ttHb}}.\n\\end{equation*}\n\\end{lemma}\n\n\\begin{proof}\nSee Appendix \\ref{pf:lem:6}.\n\\end{proof}\n\n\\begin{proof}\nSee Appendix \\ref{pf:thm:3}.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:pre}\nLet $d\\geq 1$ and $\\delta\\in(0,1)$. If $d/\\delta\\geq e$, then for any $0<a\\leq 1$,\n\\begin{equation*}\nt\\geq \\frac {2\\log(d/(\\delta a))}{a}\\quad\\Longrightarrow\\quad\\frac{\\log(dt/\\delta)}{t} \\leq a,\n\\end{equation*}\nand moreover, the right hand side inequality fails if $t= \\log(d/(\\delta a))/a$ and $d/\\delta >e$.\n\\end{lemma}\n\n\\begin{proof}\nLet $t = 2\\log(d/(\\delta a))/a$. We will show for this $t$ that $t\\geq \\log(dt/\\delta)/a$. We have\n\\begin{align*}\nt - \\frac{\\log(dt/\\delta)}{a}\n& =  \\frac{2\\log(d/(\\delta a))}{a} - \\frac{\\log(d/(\\delta a)) + \\log(2\\log(d/(\\delta a)))}{a}\n\\\\\n&=\\frac{\\log(d/(\\delta a)) - \\log(2\\log(d/(\\delta a)))}{a}\\\\\n&= \\frac1a\\log\\Big(\\frac{x}{2\\log(x)}\\Big)\\geq 0,\\qquad\\text{for}\\quad x=\\frac{d}{\\delta a},\n\\end{align*}\nwhere the last step is due to the fact that $x\\geq 2\\log(x)$ for all $x>0$. Now, we consider the function $f(t) = \\log(dt/\\delta)/t$. For any $t\\geq1$, its derivative is bounded as\n\\begin{align*}\n\\frac{\\partial f(t)}{\\partial t} = \\frac{1-\\log(dt/\\delta)}{t^2}\\leq 0,\n\\end{align*}\nwhich implies that for $t\\geq2\\log(d/(\\delta a))/a \\geq 1$ we get $\\log(dt/\\delta)/t =f(t)\\leq f(2\\log(d/(\\delta a))/a)\\leq a$. Finally, if $t=\\log(d/(\\delta a))/a$ and $d/\\delta >e$, then $\\log(dt/\\delta)/a=\\log(d/(\\delta a))/a + \\log(\\log(d/(\\delta a)))/a >t$, which completes the proof.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:pre:1}\nDefine the neighborhood $\\bar{\\N}_{\\nu} = \\{\\xb: f(\\xb) \\leq f(\\ttxb) + \\nu\\lambda_{\\min}^3/L^2\\}$ for $\\nu>0$. Suppose Assumption \\ref{ass:4} holds, then the following relation holds\n\\begin{equation*}\n\\bar{\\N}_{\\nu^2/3} \\subseteq \\N_{\\nu} \\subseteq \\bar{\\N}_{\\nu^2}, \\quad \\text{for any } \\nu \\in (0, 1]\n\\end{equation*}\nwhere $\\N_{\\nu}$ is defined in \\eqref{Nnu}.\t\n\\end{lemma}\n\n\\begin{proof}\nBy Assumption \\ref{ass:4}, we have\n\\begin{equation}\\label{pequ:A3}\n\\abr{f(\\xb) - f(\\ttxb) - \\frac{1}{2}\\nbr{\\xb - \\ttxb}_{\\ttHb}^2} \\leq \\frac{L}{6}\\nbr{\\xb - \\ttxb}^3.\n\\end{equation}\nSuppose $\\xb \\in \\bar{\\N}_{\\nu^2/3}$ for $\\nu\\in(0, 1]$, we have\n\\begin{equation}\\label{pequ:A4}\nf(\\ttxb) + \\frac{\\lambda_{\\min}}{2}\\|\\xb - \\ttxb\\|^2\\leq f(\\xb) \\leq f(\\ttxb) + \\frac{\\nu^2\\lambda_{\\min}^3}{3L^2} \\Longrightarrow \\|\\xb - \\ttxb\\| \\leq \\frac{\\sqrt{2}\\nu\\lambda_{\\min}}{\\sqrt{3}L}.\n\\end{equation}\nCombining \\eqref{pequ:A3} and \\eqref{pequ:A4}, we know that $\\xb \\in \\bar{\\N}_{\\nu^2/3}$ for $\\nu\\in(0, 1]$ leads to\n\\begin{align*}\nf(\\ttxb) + \\frac{\\nu^2\\lambda_{\\min}^3}{3L^2} \\geq f(\\xb) & \\stackrel{\\mathclap{\\eqref{pequ:A3}}}{\\geq} f(\\ttxb) + \\frac{1}{2}\\|\\xb - \\ttxb\\|_{\\ttHb}^2 - \\frac{L}{6}\\|\\xb-\\ttxb\\|^3\t\\\\\n& \\geq f(\\ttxb) + \\frac{1}{2}\\nbr{\\xb - \\ttxb}_{\\ttHb}^2 - \\frac{L}{6}\\nbr{\\xb - \\ttxb}\\cdot \\frac{\\nbr{\\xb - \\ttxb}_{\\ttHb}^2}{\\lambda_{\\min}}\\\\\n& = f(\\ttxb) + \\rbr{\\frac{1}{2}-\\frac{L\\|\\xb - \\ttxb\\|}{6\\lambda_{\\min}}}\\|\\xb - \\ttxb\\|_{\\ttHb}^2\\\\\n& \\stackrel{\\mathclap{\\eqref{pequ:A4}}}{\\geq} f(\\ttxb) + \\rbr{\\frac{1}{2} - \\frac{\\sqrt{2}\\nu}{6\\sqrt{3}}}\\|\\xb - \\ttxb\\|_{\\ttHb}^2 \\geq f(\\ttxb) + \\frac{\\|\\xb - \\ttxb\\|_{\\ttHb}^2}{3},\n\\end{align*}\nwhich further implies $\\|\\xb - \\ttxb\\|_{\\ttHb} \\leq \\nu\\lambda_{\\min}^{3/2}/L$ and $\\xb\\in \\N_{\\nu}$. On the other hand, suppose $\\xb \\in \\N_{\\nu}$, we have\n\\begin{align*}\nf(\\xb)  &\\stackrel{\\mathclap{\\eqref{pequ:A3}}}{\\leq} f(\\ttxb) + \\frac{1}{2}\\nbr{\\xb - \\ttxb}_{\\ttHb}^2 + \\frac{L}{6}\\nbr{\\xb - \\ttxb}^3 \\leq f(\\ttxb) + \\rbr{ \\frac{1}{2} + \\frac{L\\nbr{\\xb - \\ttxb}_{\\ttHb}}{6\\lambda_{\\min}^{3/2}}} \\nbr{\\xb - \\ttxb}_{\\ttHb}^2\\\\\n& \\leq f(\\ttxb) + \\rbr{ \\frac{1}{2} + \\frac{\\nu}{6}} \\nbr{\\xb - \\ttxb}_{\\ttHb}^2\\leq f(\\ttxb)  + \\nbr{\\xb - \\ttxb}_{\\ttHb}^2 \\leq f(\\ttxb) + \\frac{\\nu^2\\lambda_{\\min}^3}{L^2}.\n\\end{align*}\nThus, we have $\\xb \\in \\bar{\\N}_{\\nu^2}$. This completes the proof.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:pre:2}\n\tSuppose Assumption \\ref{ass:4} holds, then we have\n\t\\begin{equation*}\n\t\t\\|\\Hb(\\xb_1)^{-1/2}\\Hb(\\xb_2)\\Hb(\\xb_1)^{-1/2} - \\Ib\\| \\leq \\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_1 - \\xb_2\\|_{\\ttHb}, \\quad \\forall \\xb_1, \\xb_2\\in \\mR^d.\n\t\\end{equation*}\n\\end{lemma}\n\n\\begin{proof}\nWe note that\n\\begin{equation*}\n\\|\\Hb(\\xb_1)^{-1/2}\\Hb(\\xb_2)\\Hb(\\xb_1)^{-1/2} - \\Ib\\|  \\leq \\frac{1}{\\lambda_{\\min}}\\|\\Hb(\\xb_1) - \\Hb(\\xb_2)\\| \\leq \\frac{L}{\\lambda_{\\min}}\\|\\xb_1 - \\xb_2\\| \\leq \\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_1 - \\xb_2\\|_{\\ttHb}.\n\\end{equation*}\nThis completes the proof.\n\\end{proof}\n\n\\subsection{Proof of Theorem \\ref{thm:2}}\\label{pf:thm:2}\n\nFor any $t\\geq 0$ and scalar $\\theta>0$, we have from Assumption \\ref{ass:3} that\n\\begin{align*}\n\\mE_i\\sbr{\\exp\\rbr{\\theta z_{i,t}\\Eb_i}}  & = \\Ib + \\theta z_{i,t}\\mE_i[\\Eb_i] + \\sum_{j=2}^{\\infty} \\frac{(\\theta z_{i,t})^j\\mE_i[\\Eb_i^j]}{j!}\n\\preceq \\Ib + \\sum_{j=2}^{\\infty}(\\theta z_{i,t}\\Upsilon_E)^{j-2}\\cdot \\frac{(\\theta z_{i,t}\\Upsilon_E)^2}{2}\\cdot\\Ib \\\\\n& = \\Ib + \\frac{(\\theta z_{i,t}\\Upsilon_E)^2}{2(1 - \\theta z_{i,t} \\Upsilon_E)}\\cdot \\Ib \\preceq\\exp\\rbr{ \\frac{(\\theta z_{i,t}\\Upsilon_E)^2}{2(1 - \\theta z_{i,t}\\Upsilon_E)} \\cdot \\Ib}, \\quad \\forall i = 0,1,\\ldots,t,\n\\end{align*}\nwhere the second equality holds for $\\cbr{\\theta: \\theta z_{i,t}\\Upsilon_E < 1}$. Therefore, we let $\\Theta_t \\coloneqq\\{\\theta: \\theta < 1/(z_t^{(\\max)}\\Upsilon_E )\\}$, and have for all $i = 0,1,\\ldots, t$,\n\\begin{equation*}\n\\mE_i\\sbr{\\exp\\rbr{\\theta z_{i,t}\\Eb_i}}  \\preceq \\exp\\rbr{\\frac{(\\theta z_{i,t}\\Upsilon_E)^2}{2(1 - \\theta z_t^{(\\max)} \\Upsilon_E) } \\cdot \\Ib}, \\quad \\forall\\theta\\in \\Theta_t.\n\\end{equation*}\nThe above inequality suggests that the condition \\eqref{equ:exp:cond} in Lemma \\ref{lem:2} holds with\n\\begin{equation*}\ng_t(\\theta) = \\frac{\\theta^2}{2(1 - \\theta z_t^{(\\max)} \\Upsilon_E) }\\quad \\text{ and }\\quad \\Ub_i = z_{i,t}^2\\Upsilon_E^2\\cdot \\Ib.\n\\end{equation*}\nLet $\\sigma^2 = \\|\\sum_{i=0}^{t}\\Ub_i\\| = \\Upsilon_E^2\\sum_{i=0}^{t}z_{i,t}^2$ in Lemma \\ref{lem:2}. Then we have for any $\\eta\\geq 0$,\n\\begin{equation*}\n\\mP\\rbr{\\nbr{\\barEb_t} \\geq \\eta} \\leq 2d\\cdot \\inf_{\\theta \\in \\Theta_t}\\exp\\rbr{-\\theta\\eta + g_t(\\theta)\\sigma^2}.\n\\end{equation*}\nPlugging $\\theta = \\eta/(z_t^{(\\max)} \\Upsilon_E \\eta + \\sigma^2) \\in \\Theta_t$ into the above right hand side, we obtain\n\\begin{equation*}\n\\mP\\rbr{\\|\\barEb_t\\| \\geq \\eta} \\leq 2d\\cdot\\exp\\rbr{-\\frac{\\eta^2/2}{ \\sigma^2 + z_t^{(\\max)} \\Upsilon_E \\eta}  }.\n\\end{equation*}\nThis completes the proof.\n\n\n\\subsection{Proof of Lemma \\ref{lem:3}}\\label{pf:lem:3}\n\nLet us define the event\n\\begin{equation*}\n\\bar{\\EE} = \\bigcap_{t=0}^{\\infty}\\cbr{\\nbr{\\barEb_t} \\leq 8\\Upsilon_E\\rbr{\\sqrt{\\frac{\\log(d(t+1)/\\delta)}{t+1}}\\vee \\frac{\\log(d(t+1)/\\delta)}{t+1} } }.\n\\end{equation*}\nBy Theorem \\ref{thm:2} and \\eqref{equ:equal:con}, we know $\\mP(\\bar{\\EE}) \\geq 1-\\delta\\pi^2/6$. We also note that\n\\begin{align}\\label{nequ:1}\n\\|\\barEb_t\\|\\leq \\epsilon\\lambda_{\\min} & \\Longleftarrow 8\\Upsilon_E\\rbr{\\sqrt{\\frac{\\log(d(t+1)/\\delta)}{t+1}}\\vee \\frac{\\log(d(t+1)/\\delta)}{t+1} } \\leq \\epsilon\\lambda_{\\min} \\nonumber\\\\\n& \\Longleftarrow \\frac{\\log(d(t+1)/\\delta)}{t+1} \\leq \\frac{\\epsilon^2}{64\\Upsilon^2} \\wedge1 = \\rbr{\\frac{\\epsilon}{8\\Upsilon}\\wedge 1}^2 \\nonumber\\\\\n&\\Longleftarrow t \\geq 4\\rbr{\\frac{8\\Upsilon}{\\epsilon}\\vee 1}^2 \\log\\cbr{\\frac{d}{\\delta}\\cdot\\rbr{\\frac{8\\Upsilon}{\\epsilon}\\vee 1}}=\\T_1. \\quad (\\text{Lemma \\ref{lem:pre}})\n\\end{align}\nTherefore, with probability at least $1-\\delta\\pi^2/6$, we have for any $t\\geq \\T_1$ that $\\|\\barEb_t\\|\\leq \\epsilon\\lambda_{\\min}$. This shows that the event $\\EE$ defined in \\eqref{event:EE} happens, and\n\\begin{equation*}\n(1-\\epsilon)\\lambda_{\\min}\\cdot\\Ib\\preceq\\tHb_t \\stackrel{\\eqref{equ:simple}}{=} \\frac{1}{t+1}\\sum_{i=0}^{t}\\Hb_i + \\barEb_t \\preceq (\\lambda_{\\max} + \\epsilon\\lambda_{\\min})\\cdot\\Ib\\preceq (1+\\epsilon)\\lambda_{\\max} \\cdot\\Ib.\n\\end{equation*}\nThis completes the proof.\n\n\n\\subsection{Proof of Lemma \\ref{lem:4}}\\label{pf:lem:4}\n\nBy Lemma \\ref{lem:3}, we know that $\\pb_t = -(\\tHb_t)^{-1}\\nabla f_t$ for $t\\geq \\T_1$. We apply Taylor's expansion:\n\\begin{align*}\n\tf(\\xb_t - \\mu_t(\\tHb_t)^{-1}\\nabla f_t ) & \\leq f(\\xb_t) - \\mu_t \\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t + \\frac{\\lambda_{\\max}\\mu_t^2}{2}\\nabla f_t^\\top(\\tHb_t)^{-2}\\nabla f_t \\\\\n\t& \\leq f(\\xb_t) - \\mu_t \\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t +\\frac{\\kappa\\mu_t^2}{2(1-\\epsilon)}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t \\quad (\\text{Lemma \\ref{lem:3}})\\\\\n\t& =  f(\\xb_t) - \\mu_t\\rbr{1 - \\frac{\\kappa\\mu_t}{2(1-\\epsilon)}}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t.\n\\end{align*}\nThus, the Armijo condition is satisfied if\n\\begin{equation*}\n1 - \\frac{\\kappa\\mu_t}{2(1-\\epsilon)} \\geq \\beta \\Longleftrightarrow \\mu_t \\leq \\frac{2(1-\\beta)(1-\\epsilon)}{\\kappa}.\n\\end{equation*}\nTherefore, the backtracking line search on Line 7 leads to a stepsize satisfying\n\\begin{equation}\\label{npequ:1}\n\\mu_t\\geq \\frac{2\\rho(1-\\beta)(1-\\epsilon)}{\\kappa}.\n\\end{equation}\nMoreover, we apply Lemma \\ref{lem:3} and strong convexity of $f$, and have\n\\begin{equation}\\label{npequ:2}\n\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t \\geq \\frac{1}{(1+\\epsilon)\\lambda_{\\max}}\\|\\nabla f_t\\|^2 \\geq \\frac{2}{(1+\\epsilon)\\kappa}(f(\\xb_t) - f(\\ttxb)).\n\\end{equation}\nCombining \\eqref{npequ:1}, \\eqref{npequ:2} with the Armijo condition, we have\n\\begin{align}\\label{pequ:B1}\nf(\\xb_{t+1}) - f(\\ttxb) &\\leq f(\\xb_t) - f(\\ttxb) - \\mu_t \\beta \\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t \\nonumber\\\\\n& \\leq f(\\xb_t) - f(\\ttxb) - \\frac{2\\rho\\beta(1-\\beta)(1-\\epsilon)}{\\kappa}\\cdot \\frac{2}{(1+\\epsilon)\\kappa}(f(\\xb_t) - f(\\ttxb)) \\nonumber\\\\\n& = (1-\\phi)(f(\\xb_t) - f(\\ttxb)),\n\\end{align}\nwhich shows the first part of the statement. Furthermore, we apply \\eqref{pequ:B1} recursively, apply strong convexity, and have\n\\begin{equation*}\n\\|\\xb_t - \\ttxb\\|^2\\leq \\frac{2(f(\\xb_t)-f(\\ttxb))}{\\lambda_{\\min}} \\\\\n\\stackrel{\\eqref{pequ:B1}}{\\leq} \\frac{2(1-\\phi)^{t-\\T_1}}{\\lambda_{\\min}}(f(\\xb_{\\T_1}) - f(\\ttxb)) \\leq \\frac{2(1-\\phi)^{t-\\T_1}}{\\lambda_{\\min}}(f(\\xb_0) - f(\\ttxb)).\n\\end{equation*}\nThe last statement follows from $\\|\\xb_t-\\ttxb\\|_{\\ttHb}^2\\leq \\lambda_{\\max}\\|\\xb_t - \\ttxb\\|^2$. This completes the proof.\n\n\n\\subsection{Proof of Corollary \\ref{cor:1}}\\label{pf:cor:1}\n\nWe condition on the event \\eqref{event:EE}, which happens with probability $1-\\delta\\pi^2/6$. By Lemma \\ref{lem:4}, we know that, for $t\\geq \\T_1$,\n\\begin{equation*}\nf(\\xb_t) - f(\\ttxb) \\leq (1 - \\phi)^{t-\\T_1}(f(\\xb_{\\T_1}) - f(\\ttxb)) \\leq (1 - \\phi)^{t-\\T_1}(f(\\xb_0) - f(\\ttxb)).\n\\end{equation*}\nBy Lemma \\ref{lem:pre:1}, to have $\\xb_t \\in \\N_\\nu$, it suffices to have $\\xb_t \\in \\bar{\\N}_{\\nu^2/3}$. Thus, we let\n\\begin{equation}\\label{equ:new}\n(1 - \\phi)^{t-\\T_1}(f(\\xb_0) - f(\\ttxb)) \\leq \\frac{\\nu^2\\lambda_{\\min}^3}{3L^2} \\Longleftarrow t-\\T_1 \\geq \\frac{\\log\\rbr{\\frac{3L^2(f(\\xb_0) - f(\\ttxb))}{\\nu^2\\lambda_{\\min}^3}}}{\\log\\rbr{\\frac{1}{1-\\phi}}}\\Longleftarrow t \\geq \\T_1 + \\T_2,\n\\end{equation}\nwhere $\\T_2$ is defined in \\eqref{T2} and the implication uses the fact that $\\log(1/(1-\\phi))\\geq \\phi$. Furthermore, since $f(\\xb_t)$ is always decreasing, we know after $\\T_1+T_2$ iterations $\\xb_t$ stays in $\\bar{\\N}_{\\nu^2/3}$, and hence stays in $\\N_{\\nu}$ (cf. Lemma \\ref{lem:pre:1}). This completes the proof.\n\n\n\\subsection{Proof of Lemma \\ref{lem:5}}\\label{pf:lem:5}\n\nIt suffices to show \n\\begin{equation*}\nf(\\xb_t - (\\tHb_t)^{-1}\\nabla f_t) \\leq f(\\xb_t) - \\beta\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t.\n\\end{equation*}\nBy Assumption \\ref{ass:4}, we have\n\\begin{align}\\label{npequ:3}\n& f(\\xb_t - (\\tHb_t)^{-1}\\nabla f_t) \\nonumber\\\\\n& \\leq f(\\xb_t) - \\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t + \\frac{1}{2}\\nabla f_t^\\top (\\tHb_t)^{-1}\\Hb_t(\\tHb_t)^{-1}\\nabla f_t + \\frac{L}{6}\\|(\\tHb_t)^{-1}\\nabla f_t\\|^3 \\nonumber\\\\\n& \\leq f(\\xb_t) - \\frac{1}{2}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t + \\frac{1}{2}\\nabla f_t^\\top(\\tHb_t)^{-1}(\\Hb_t - \\tHb_t)(\\tHb_t)^{-1}\\nabla f_t + \\frac{L}{6}\\|(\\tHb_t)^{-1}\\nabla f_t\\|^3 \\nonumber\\\\\n& \\leq f(\\xb_t) - \\frac{1}{2}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t + \\frac{1}{2}\\|(\\tHb_t)^{-1/2}(\\Hb_t - \\tHb_t)(\\tHb_t)^{-1/2}\\|\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t \\nonumber\\\\\n& \\quad + \\frac{L}{6}\\|(\\tHb_t)^{-1}\\nabla f_t\\| \\nabla f_t^\\top(\\tHb_t)^{-2}\\nabla f_t \\nonumber\\\\\n& \\leq f(\\xb_t) - \\frac{1}{2}\\nabla f_t^\\top (\\tHb_t)^{-1}\\nabla f_t + \\frac{\\psi}{2(1-\\psi)}\\nabla f_t^\\top (\\tHb_t)^{-1}\\nabla f_t + \\frac{L\\|(\\tHb_t)^{-1}\\nabla f_t\\|}{6(1-\\psi)\\lambda_{\\min}}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t \\nonumber\\\\\n& = f(\\xb_t) - \\rbr{\\frac{1}{2} - \\frac{\\psi}{2(1-\\psi)} - \\frac{L\\|(\\tHb_t)^{-1}\\nabla f_t\\|}{6(1-\\psi)\\lambda_{\\min}}}\\nabla f_t^\\top (\\tHb_t)^{-1}\\nabla f_t.\n\\end{align}\nFor term $\\|(\\tHb_t)^{-1}\\nabla f_t\\|$, we let $\\Hb_t^\\eta = \\Hb(\\xb_t^\\eta)$ with $\\xb_t^\\eta = \\ttxb + \\eta(\\xb_t - \\ttxb)$ for some $\\eta\\in(0, 1)$, and have\n\\begin{align}\\label{npequ:4}\n&\\|(\\tHb_t)^{-1}\\nabla f_t\\|^2  = (\\xb_t - \\ttxb)^\\top (\\ttHb)^{1/2}(\\ttHb)^{-1/2}\\Hb_t^\\eta(\\tHb_t)^{-2}\\Hb_t^\\eta(\\ttHb)^{-1/2}(\\ttHb)^{1/2}(\\xb_t-\\ttxb) \\nonumber\\\\\n& \\leq \\frac{\\|\\xb_t - \\ttxb\\|_{\\ttHb}^2}{(1-\\psi)\\lambda_{\\min}}\\|(\\ttHb)^{-1/2}\\Hb_t^\\eta(\\tHb_t)^{-1}\\Hb_t^\\eta(\\ttHb)^{-1/2}\\| \\nonumber\\\\\n& \\leq \\frac{\\nu^2\\lambda_{\\min}^2}{(1-\\psi)L^2}\\|(\\ttHb)^{-1/2}(\\Hb_t^\\eta)^{1/2}\\|^2\\|(\\Hb_t^\\eta)^{1/2}(\\tHb_t)^{-1}(\\Hb_t^\\eta)^{1/2}\\| \\quad (\\text{since } \\xb_t\\in \\N_{\\nu}) \\nonumber\\\\\n& \\leq \\frac{\\nu^2\\lambda_{\\min}^2}{(1-\\psi)L^2}\\|(\\ttHb)^{-1/2}\\Hb_t^\\eta(\\ttHb)^{-1/2}\\|\\|(\\Hb_t^\\eta)^{1/2}(\\Hb_t)^{-1}(\\Hb_t^\\eta)^{1/2}\\|\\|\\Hb_t^{1/2}(\\tHb_t)^{-1}\\Hb_t^{1/2}\\| \\nonumber\\\\\n& \\leq \\frac{\\nu^2\\lambda_{\\min}^2}{(1-\\psi)^2L^2}\\|(\\ttHb)^{-1/2}\\Hb_t^\\eta(\\ttHb)^{-1/2}\\| \\|(\\Hb_t)^{-1/2}\\Hb_t^\\eta(\\Hb_t)^{-1/2}\\|.\n\\end{align}\nNoting that $\\xb_t^\\eta\\in\\N_{\\nu}$ if $\\xb_t \\in \\N_\\nu$, we apply Lemma \\ref{lem:pre:2} and have\n\\begin{equation}\\label{npequ:5}\n\\|(\\ttHb)^{-1/2}\\Hb_t^\\eta(\\ttHb)^{-1/2}\\| \\vee \\|(\\Hb_t)^{-1/2}\\Hb_t^\\eta(\\Hb_t)^{-1/2}\\| \\leq 1 + \\nu\\leq 2.\n\\end{equation}\nCombining \\eqref{npequ:3}, \\eqref{npequ:4}, and \\eqref{npequ:5}, we finally obtain\n\\begin{equation*}\nf(\\xb_t - (\\tHb_t)^{-1}\\nabla f_t) \\leq f(\\xb_t) - \\rbr{\\frac{1}{2} - \\frac{\\psi}{2(1-\\psi)} - \\frac{\\nu}{3(1-\\psi)^2}}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t.\n\\end{equation*}\nThus, it suffices to let\n\\begin{equation*}\n\\frac{1}{2} - \\frac{\\psi}{2(1-\\psi)} - \\frac{\\nu}{3(1-\\psi)^2} \\geq \\beta \\Longleftarrow \\frac{\\psi}{2(1-\\psi)} \\vee \\frac{\\nu}{3(1-\\psi)^2} \\leq \\frac{1}{2}\\rbr{\\frac{1}{2}-\\beta}\n\\end{equation*}\nas implied by the conditions stated in the lemma. This completes the proof.\n\n\n\n\\subsection{Proof of Lemma \\ref{lem:6}}\\label{pf:lem:6}\n\nSince $\\xb_t\\in \\N_{\\nu}$ with $\\nu \\leq 2/3\\cdot (0.5-\\beta)$, we apply Lemma \\ref{lem:pre:2} and have\n\\begin{equation*}\n\\|(\\ttHb)^{-1/2}(\\Hb_t - \\ttHb)(\\ttHb)^{-1/2}\\| \\leq \\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_t - \\ttxb\\|_{\\ttHb} \\leq \\nu\\leq 0.5-\\beta.\n\\end{equation*}\nThus, we obtain\n\\begin{equation}\\label{pequ:B2}\n\\rbr{0.5+\\beta}\\ttHb\\preceq \\Hb_t\\preceq \\rbr{1.5-\\beta}\\ttHb. \n\\end{equation}\nSince $\\pb_t = -(\\tHb_t)^{-1}\\nabla f_t$ and $\\mu_t=1$, we have\n\\begin{align}\\label{npequ:6}\n& \\nbr{\\xb_{t+1} - \\ttxb}_{\\ttHb} = \\|\\xb_t - (\\tHb_t)^{-1}\\nabla f_t - \\ttxb\\|_{\\ttHb} \\nonumber\\\\\n& \\stackrel{\\mathclap{\\eqref{pequ:B2}}}{\\leq}\\frac{1}{\\sqrt{0.5+\\beta}}\\|\\xb_t - (\\tHb_t)^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} \\nonumber\\\\\n& \\leq \\frac{1}{\\sqrt{0.5+\\beta}}\\cbr{\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} + \\|\\Hb_t^{-1}\\nabla f_t - (\\tHb_t)^{-1}\\nabla f_t\\|_{\\Hb_t}}.\n\\end{align}\nFor the second term on the right hand side, we have\n\\begin{align}\\label{npequ:7}\n& \\|\\Hb_t^{-1}\\nabla f_t - (\\tHb_t)^{-1}\\nabla f_t\\|_{\\Hb_t} \\nonumber\\\\\n& = \\sqrt{\\nabla f_t^\\top(\\Hb_t^{-1} - (\\tHb_t)^{-1})\\Hb_t(\\Hb_t^{-1} - (\\tHb_t)^{-1})\\nabla f_t} \\nonumber\\\\\n& = \\sqrt{\\nabla f_t^\\top\\Hb_t^{-1/2}(\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2})^2\\Hb_t^{-1/2}\\nabla f_t} \\nonumber\\\\\n& \\leq \\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\|\\cdot\\|\\Hb_t^{-1}\\nabla f_t\\|_{\\Hb_t}  \\nonumber\\\\\n& \\leq  \\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\|\\cbr{\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} + \\|\\xb_t - \\ttxb\\|_{\\Hb_t}} \\nonumber\\\\\n& \\stackrel{\\mathclap{\\eqref{pequ:B2}}}{\\leq}\\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\|\\cbr{\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} + \\sqrt{1.5-\\beta}\\|\\xb_t - \\ttxb\\|_{\\ttHb}}.\n\\end{align}\nCombining \\eqref{npequ:6} and \\eqref{npequ:7},\n\\begin{align}\\label{pequ:B3}\n\\|\\xb_{t+1} - \\ttxb\\|_{\\ttHb}& \\leq\\frac{1}{\\sqrt{0.5+\\beta}} \\big\\{(1+\\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\|)\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} \\nonumber\\\\\n& \\quad + \\sqrt{1.5-\\beta}\\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\|\\cdot \\|\\xb_t - \\ttxb\\|_{\\ttHb}\\big\\}.\n\\end{align}\nSince $\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| \\leq \\psi\\leq 1/3$, we have\n\\begin{equation*}\n(1 - \\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| )\\cdot \\Ib \\preceq \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2} \\preceq (1+\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\|)\\cdot\\Ib\n\\end{equation*}\nand further\n\\begin{equation*}\n-\\frac{\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }{1+\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }\\cdot \\Ib\\preceq \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2} - \\Ib \\preceq \\frac{\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }{1-\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }\\cdot\\Ib.\n\\end{equation*}\nThus, we have\n\\begin{equation*}\n\\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\| \\leq \\frac{\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }{1 - \\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }\\leq \\frac{3}{2}\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\|.\n\\end{equation*}\nCombining the above inequality with \\eqref{pequ:B3}, we obtain\n\\begin{align}\\label{pequ:B4}\n&\\|\\xb_{t+1} - \\ttxb\\|_{\\ttHb} \\nonumber\\\\\n&  \\leq \\frac{3}{2\\sqrt{0.5+\\beta}} \\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} + \\frac{3}{2}\\sqrt{\\frac{1.5-\\beta}{0.5+\\beta}}\\cdot\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\|  \\cdot \\|\\xb_t - \\ttxb\\|_{\\ttHb} \\nonumber\\\\\n& \\leq \\frac{3\\sqrt{2}}{2}\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t}  + \\frac{3\\sqrt{3}}{2}\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\|  \\cdot \\|\\xb_t - \\ttxb\\|_{\\ttHb}.\n\\end{align}\nFurthermore,\n\\begin{align*}\n\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} & = \\|\\Hb_t^{-1/2}(\\Hb_t(\\xb_t - \\ttxb) - \\nabla f_t)\\|\\\\\n& \\leq \\frac{1}{\\sqrt{\\lambda_{\\min}}}\\nbr{\\Hb_t(\\xb_t - \\ttxb) - \\rbr{\\int_0^1\\Hb(\\ttxb + \\tau(\\xb_t - \\ttxb))d\\tau }(\\xb_t-\\ttxb)}\\\\\n& \\leq \\frac{\\|\\xb_t -\\ttxb\\|^2}{\\sqrt{\\lambda_{\\min}}}\\cdot \\int_0^1(1-\\tau)L d\\tau \\leq \\frac{L}{2\\lambda_{\\min}^{3/2}}\\|\\xb_t-\\ttxb\\|_{\\ttHb}^2.\n\\end{align*}\nCombining the above inequality with \\eqref{pequ:B4}, we complete the proof.\n\n\n\n\\subsection{Proof of Theorem \\ref{thm:3}}\\label{pf:thm:3}\n\nWe suppose the event \\eqref{event:EE} happens, which has probability $1-\\delta\\pi^2/6$. By Lemma \\ref{lem:4} and Corollary \\ref{cor:1}, we know that $\\xb_t$ converges $R$-linearly for all $t\\geq \\T_1$, and $\\xb_{\\T:\\T+\\J}\\in \\N_{\\nu}$ for any $\\J\\geq 0$.~Thus, it suffices to study the convergence after $\\T+\\J$. For any $t\\geq 0$, to apply Lemmas \\ref{lem:5} and \\ref{lem:6}, we characterize the difference between $\\tHb_{\\T+\\J+t}$ and $\\Hb_{\\T+\\J+t}$. We have\n\\begin{align}\\label{npequ:8}\n& \\|\\Hb_{\\T+\\J + t}^{-1/2}\\tHb_{\\T+\\J + t}\\Hb_{\\T+\\J + t}^{-1/2} -\\Ib\\|  \\nonumber\\\\\n& \\leq \\|\\Hb_{\\T+\\J + t}^{-1/2}(\\tHb_{\\T+\\J + t} - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\| + \\|\\Hb_{\\T+\\J + t}^{-1/2}\\ttHb\\Hb_{\\T+\\J + t}^{-1/2} - \\Ib\\| \\nonumber\\\\\n& \\stackrel{\\mathclap{\\text{Lemma \\ref{lem:pre:2}}}}{\\leq}\\;\\;\\; \\|\\Hb_{\\T+\\J + t}^{-1/2}(\\tHb_{\\T+\\J + t} - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\|  + \\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t} - \\ttxb\\|_{\\ttHb}. \n\\end{align}\nFor the first term $\\|\\Hb_{\\T+\\J + t}^{-1/2}(\\tHb_{\\T+\\J + t} - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\|$, we apply the formula \\eqref{equ:simple} and have\n\\begin{align}\\label{npequ:9}\n&\\|\\Hb_{\\T+\\J + t}^{-1/2}(\\tHb_{\\T+\\J + t} - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\| \\nonumber\\\\\n& \\stackrel{\\mathclap{\\eqref{equ:simple}}}{\\leq} \\frac{1}{\\lambda_{\\min}}\\nbr{\\barEb_{\\T+\\J+t}}  + \\frac{1}{\\T+\\J + t + 1}\\bigg\\{\\sum_{j=0}^{\\T-1}\\|\\Hb_{\\T+\\J + t}^{-1/2}(\\Hb_j - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\| \\nonumber\\\\\n&\\hskip6cm + \\sum_{j=\\T}^{\\T+\\J+t}\\|\\Hb_{\\T+\\J + t}^{-1/2}(\\Hb_j - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\| \\bigg\\} \\nonumber\\\\\n& \\leq \\frac{1}{\\lambda_{\\min}}\\|\\barEb_{\\T+\\J+t}\\| + \\frac{2\\T\\kappa}{\\T+\\J+t+1} + \\frac{L}{\\lambda_{\\min}(\\T+\\J+t+1)}\\sum_{j=\\T}^{\\T+\\J+t}\\|\\xb_j - \\ttxb\\| \\nonumber\\\\\n& \\stackrel{\\mathclap{\\eqref{event:EE}}}{\\leq} 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}} + \\frac{2\\T\\kappa}{\\T+\\J+t+1} \\nonumber\\\\ \n&\\hskip0.5cm +\\frac{L}{\\lambda_{\\min}(\\T+\\J+t+1)}\\sum_{j=\\T}^{\\T+\\J+t}\\sqrt{\\frac{2(f(\\xb_0) - f(\\ttxb))}{\\lambda_{\\min}}}(1 - \\phi)^{(j-\\T_1)/2} \\quad (\\text{also use Lemma \\ref{lem:4}}) \\nonumber\\\\\n& \\leq 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}} + \\frac{2\\T\\kappa}{\\T+\\J+t+1} + \\frac{L\\sqrt{2(f(\\xb_0) - f(\\ttxb))}}{\\lambda_{\\min}^{3/2}(\\T+\\J+t+1)}(1-\\phi)^{\\T_2/2}\\sum_{j=0}^{\\infty}(1-\\phi)^{j/2} \\nonumber\\\\\n& \\stackrel{\\mathclap{\\eqref{equ:new}}}{\\leq}\\; 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}} + \\frac{2\\T\\kappa}{\\T+\\J+t+1} + \\frac{\\sqrt{2}\\nu}{\\sqrt{3}(\\T+\\J+t+1)(1-\\sqrt{1-\\phi})} \\nonumber\\\\\n& \\leq 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}} + \\frac{2\\T\\kappa}{\\T+\\J+t+1} + \\frac{2\\nu}{(\\T+\\J+t+1)\\phi} \\nonumber\\\\\n& \\leq 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}} + \\frac{4\\T\\kappa}{\\T+\\J+t+1} = \\rho_t.\\quad (\\text{since } \\nu/\\phi \\leq1/\\phi\\leq\\T\\kappa)\n\\end{align}\nCombining \\eqref{npequ:8} and \\eqref{npequ:9} together, we obtain\n\\begin{equation}\\label{pequ:B5}\n\\|\\Hb_{\\T+\\J + t}^{-1/2}\\tHb_{\\T+\\J + t}\\Hb_{\\T+\\J + t}^{-1/2} -\\Ib\\|\\leq \\rho_t+\\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t} - \\ttxb\\|_{\\ttHb} \\leq \\rho_t + \\nu\\leq \\rho_0 + \\nu.\n\\end{equation}\nThus, in order to apply Lemma \\ref{lem:6} for all $\\{\\xb_{\\T+\\J+t}\\}_{t\\geq 0}$, we require $\\nu\\leq 2/3\\cdot (0.5-\\beta)$ and\n\\begin{align*}\n\\rho_0 + \\nu \\leq \\frac{0.5-\\beta}{1.5-\\beta} & \\xLeftarrow{\\J = 4\\T\\kappa/\\nu} 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J)/\\delta)}{\\T+\\J}} + 2\\nu \\leq \\frac{0.5-\\beta}{1.5-\\beta} \\\\\n& \\Longleftarrow 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J)/\\delta)}{\\T+\\J}}  \\vee\\nu\\leq \\frac{1}{3}\\frac{0.5-\\beta}{1.5-\\beta}\\\\\n& \\stackrel{\\eqref{cond:epsnu}}{\\Longleftarrow} 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J)/\\delta)}{\\T+\\J}}  \\leq \\epsilon \\; \\text{ and }\\; \\epsilon\\vee \\nu \\leq \\frac{1}{3}\\frac{0.5-\\beta}{1.5-\\beta},\n\\end{align*}\nwhich is implied by \\eqref{cond:epsnu} and \\eqref{nequ:1}. Thus, for any $t\\geq 0$, we apply Lemma \\ref{lem:6} and obtain\n\\begin{align}\\label{pequ:B6}\n& \\|\\xb_{\\T+\\J+t+1}-\\ttxb\\|_{\\ttHb} \\nonumber\\\\\n& \\leq 3\\cbr{\\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t}-\\ttxb\\|_{\\ttHb}^2 + \\|\\Hb_{\\T+\\J + t}^{-1/2}\\tHb_{\\T+\\J + t}\\Hb_{\\T+\\J + t}^{-1/2} -\\Ib\\|\\cdot \\|\\xb_{\\T+\\J+t} - \\ttxb\\|_{\\ttHb}} \\nonumber\\\\\n&\\stackrel{\\mathclap{\\eqref{pequ:B5}}}{\\leq} 3\\cbr{\\frac{2L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t}-\\ttxb\\|_{\\ttHb}^2 +  \\rho_t\\|\\xb_{\\T+\\J+t}-\\ttxb\\|_{\\ttHb}}.\n\\end{align}\nWe then claim that\n\\begin{equation}\\label{pequ:B7}\n\\frac{2L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t}-\\ttxb\\|_{\\ttHb} \\leq 3\\rho_t, \\quad \\forall t\\geq 0.\n\\end{equation}\nWe prove \\eqref{pequ:B7} by induction. For $t = 0$, we note that\n\\begin{equation*}\n\\frac{2L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J}-\\ttxb\\|_{\\ttHb} \\leq 2\\nu = \\frac{12\\T\\kappa}{6\\T\\kappa/\\nu}\\leq \\frac{12\\T\\kappa}{\\T + 4\\T\\kappa/\\nu +1} = \\frac{12\\T\\kappa}{\\T + \\J +1} \\leq 3\\rho_0.\n\\end{equation*}\nSuppose \\eqref{pequ:B7} holds for $t\\geq 0$, then \\eqref{pequ:B6} leads to\n\\begin{equation*}\n\\frac{2L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t+1}-\\ttxb\\|_{\\ttHb} \\leq \\frac{2L}{\\lambda_{\\min}^{3/2}}\\cdot 12\\rho_t\\|\\xb_{\\T+\\J+t}-\\ttxb\\|_{\\ttHb} \\stackrel{\\eqref{pequ:B7}}{\\leq} 36\\rho_t^2.\n\\end{equation*}\nOn the other hand, using $(\\T+\\J+t+2)\\leq 2(\\T+\\J+t+1)$, $\\forall t\\geq 0$, we know $\\rho_t/2 \\leq \\rho_{t+1}$. Thus, it suffices to show $36\\rho_t^2\\leq 3\\rho_t/2\\Longleftrightarrow 24\\rho_t\\leq 1$. Since $\\rho_t$ is decreasing, we require $24\\rho_0\\leq 1$. Note that\n\\begin{equation*}\n\\rho_0 \\leq 8\\Upsilon\\sqrt{\\frac{\\log(d\\T/\\delta)}{\\T}} + \\nu \\stackrel{\\eqref{T2}}{\\leq} \\epsilon + \\nu.\n\\end{equation*}\nThus, $24\\rho_0\\leq 1$ is guaranteed by \\eqref{cond:epsnu}. Combining \\eqref{pequ:B6} and \\eqref{pequ:B7} completes the proof.\n\n\n","units":[{"start":51004,"end":52022,"start_line":398,"end_line":407,"raw_tex":"\\begin{lemma}[Adapted from Theorem 2.3 in \\cite{Tropp2011Freedmans}]\\label{lem:2}\nLet $t\\geq 0$ be a fixed integer. Consider a $d$-dimensional martingale difference $\\{\\Eb_i\\}_{i=0}^{t}$ (i.e., $\\mE_i[\\Eb_i] = \\0$). Suppose there exists a function $g_t:\\Theta_t\\rightarrow [0, \\infty]$ with $\\Theta_t\\subseteq (0, \\infty)$, and~a~sequence of matrices $\\{\\Ub_i\\}_{i=0}^{t}$, such that for any $i = 0,1,\\ldots,t$,\\footnote{The matrix exponential is defined by power series expansion: $\\exp(\\Ab) = \\Ib + \\sum_{i=1}^{\\infty} \\Ab^i/ i!$.}\n\\begin{equation}\\label{equ:exp:cond}\n\\mE_i\\sbr{\\exp\\rbr{\\theta\\Eb_i}}\\preceq \\exp\\rbr{g_t(\\theta)\\Ub_i} \\quad \\text{ almost surely for each } \\theta\\in \\Theta_t.\n\\end{equation}\nThen, we have for any scalars $\\eta \\geq 0$ and $\\sigma^2>0$,\n\\begin{equation*}\n\\mP\\rbr{\\Big\\|\\sum_{i=0}^{t}\\Eb_i\\Big\\| \\geq \\eta\\; \\text{ and } \\; \\Big\\|\\sum_{i=0}^{t}\\Ub_i\\Big\\|\\leq \\sigma^2} \\leq 2d\\cdot \\inf_{\\theta\\in \\Theta_t}\\exp\\rbr{-\\theta \\eta + g_t(\\theta)\\sigma^2}.\n\\end{equation*}\t\t\n\\end{lemma}","sha256_utf8":"dced947d77c1089a664c7c0cb26a1d525e0d57d12daa2b865d1db89358cd592c"},{"start":52237,"end":52706,"start_line":413,"end_line":419,"raw_tex":"\\begin{theorem}[Concentration of sub-exponential martingale difference]\\label{thm:2}\nUnder Assumption \\ref{ass:3}, for any integer $t\\geq 0$ and scalar $\\eta\\geq 0$, $\\barEb_t$ in \\eqref{equ:simple} satisfies\n\\begin{equation}\\label{equ:subE}\n\\mP\\rbr{\\nbr{\\barEb_t} \\geq \\eta} \\leq 2d\\cdot\\exp\\rbr{-\\frac{\\eta^2/2}{\\Upsilon_E^2\\sum_{i=0}^{t}z_{i,t}^2 + z_t^{(\\max)}\\Upsilon_E\\eta } }\n\\end{equation}\nwhere $z_t^{(\\max)} = \\max_{i\\in \\{0,\\ldots,t\\}}z_{i,t}$.\n\\end{theorem}","sha256_utf8":"0cba1780d2247d230aad6b851c7a28793112428bf2bcf4fab05445ec320d244e"},{"start":52709,"end":52763,"start_line":422,"end_line":424,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{pf:thm:2}.\n\\end{proof}","sha256_utf8":"329fcdbbd8721cd74cc501dbb9776bb3374d9544d635a204a029879e5f48b186"},{"start":52766,"end":53355,"start_line":427,"end_line":430,"raw_tex":"The martingale concentration in Theorem \\ref{thm:2} matches the matrix Bernstein results for independent noises $\\{\\Eb_i\\}_{i=0}^t$ \\citep[cf. Theorems 6.1, 6.2 in][]{Tropp2011User}. For any~$\\delta\\in(0, 1)$, if we let the right hand side of \\eqref{equ:subE} be $\\delta/(t+1)^2$, then we obtain that, with probability at least $1 - \\delta/(t+1)^2$,  \n\\begin{equation}\\label{equ:sub:concen}\n\\small \\nbr{\\barEb_t} \\leq 8\\Upsilon_E\\sqrt{\\log\\rbr{\\frac{d(t+1)}{\\delta}}}\\rbr{\\sqrt{\\sum_{i=0}^{t}z_{i,t}^2} \\; \\vee \\; \\sqrt{\\log\\rbr{\\frac{d(t+1)}{\\delta}}} \\cdot z_t^{(\\max)}}.\n\\end{equation}\n","sha256_utf8":"6baa298236c22cc23cbcd2924f3b511c4311b4479f269d0e56e8370893422607"},{"start":53459,"end":54159,"start_line":434,"end_line":441,"raw_tex":"To achieve the fastest concentration rate, we minimize the right~hand side of \\eqref{equ:sub:concen} under the~restriction $\\sum_{i=0}^t z_{i,t} = 1$. Note that the minimum of both $\\sum_{i=0}^t z_{i,t}^2$ and $z_t^{(\\max)}$ is attained with equal weights, that is $z_{i,t} = 1/(t+1)$, for $ i = 0,1,\\ldots, t$. Thus, the fastest concentration rate is attained~with~equal~weights. Furthermore, a union bound over $t$ leads to\n\\begin{equation}\\label{equ:equal:con}\n\\mP\\rbr{\\forall t:\\ \\nbr{\\barEb_t} \\leq\n8\\Upsilon_E\\bigg(\\sqrt{\\frac{\\log(d(t+1)/\\delta)}{t+1}} \\vee\n  \\frac{\\log(d(t+1)/\\delta)}{t+1}\\bigg)} \n\\geq 1 - \\sum_{t=0}^{\\infty}\n\\frac{\\delta}{(t+1)^2}= 1-\\frac{\\pi^2\\delta}{6}.\n\\end{equation}\n","sha256_utf8":"a0af15488ef9030cd8bb15ff2583d47dad72495638fc39d73f7420405393343b"},{"start":55243,"end":55933,"start_line":451,"end_line":461,"raw_tex":"\\begin{lemma}\\label{lem:3}\nConsider Algorithm \\ref{alg:1} with $w_t = t+1$, $\\forall t\\geq 0$. Under Assumption~\\ref{ass:3}, we let $\\delta, \\epsilon\\in(0, 1)$ with $d/\\delta\\geq e$. We also let\n\\begin{equation}\\label{T1}\n\\T_1 =4\\rbr{1\\vee (8\\Upsilon/\\epsilon)}^2 \\log\\rbr{d/\\delta\\cdot\\cbr{1\\vee(8\\Upsilon/\\epsilon)}}.\n\\end{equation}\nThen, with probability $1-\\delta\\pi^2/6$, the event\n\\begin{equation}\\label{event:EE}\n\\EE = \\bigcap_{t=\\T_1}^{\\infty}\\cbr{\\|\\barEb_t\\| \\leq 8\\Upsilon_E\\sqrt{\\frac{\\log(d(t+1)/\\delta)}{t+1}}}\n\\end{equation}\noccurs, which implies $(1-\\epsilon)\\lambda_{\\min}\\cdot\\Ib\\preceq \\tHb_t \\preceq (1+\\epsilon)\\lambda_{\\max}\\cdot\\Ib$, $\\forall t\\geq \\T_1$.\n\\end{lemma}","sha256_utf8":"baf129e6ab80f2e39ceefa32a50c9e750df4928d363cad5ebb5be6d4aa23560c"},{"start":55935,"end":55990,"start_line":463,"end_line":465,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{pf:lem:3}. \n\\end{proof}","sha256_utf8":"96f3531cedcf83940ff361731518256c1371d3bb6ef9a50d86727518ac3c091d"},{"start":56644,"end":57249,"start_line":473,"end_line":483,"raw_tex":"\\begin{lemma}\\label{lem:4}\nConditioning on the event \\eqref{event:EE}, we let\n\\begin{equation*}\n\\phi = \\frac{4\\rho\\beta(1-\\beta)(1-\\epsilon)}{\\kappa^2(1+\\epsilon)}\n\\end{equation*}\nand have $f(\\xb_{t+1})-f(\\ttxb) \\leq (1-\\phi)(f(\\xb_t) - f(\\ttxb))$, $\\forall t\\geq \\T_1$, which implies $R$-linear convergence of the iterate error,\n\\begin{align*}\n\\|\\xb_t - \\ttxb\\| &\\leq \\cbr{\\frac{2}{\\lambda_{\\min}}(f(\\xb_0) - f(\\ttxb))(1-\\phi)^{t-\\T_1}}^{1/2}, \\quad t\\geq \\T_1,\\\\\n\\|\\xb_t - \\ttxb\\|_{\\ttHb} &\\leq \\cbr{2\\kappa(f(\\xb_0) - f(\\ttxb))(1-\\phi)^{t-\\T_1}}^{1/2}, \\quad\\quad\\; t\\geq \\T_1.\n\\end{align*}\n\\end{lemma}","sha256_utf8":"faee9df9084b18bf44d4cfde5737d8d3bdd1a008d83b24d68ec4ff12db34597c"},{"start":57251,"end":57305,"start_line":485,"end_line":487,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{pf:lem:4}.\n\\end{proof}","sha256_utf8":"99ca29b639f675b941b1e7847f81784aee590728ceca1e3363916a84952a32fc"},{"start":57750,"end":58501,"start_line":498,"end_line":508,"raw_tex":"\\begin{corollary}\\label{cor:1}\nConsider Algorithm \\ref{alg:1} with $w_t=t+1$, $\\forall t\\geq 0$. Under Assumptions \\ref{ass:3} and \\ref{ass:4}, we let $\\delta, \\epsilon\\in(0, 1)$ with $d/\\delta \\geq e$, and define the neighborhood $\\N_{\\nu}$ as \n\\begin{equation}\\label{Nnu}\n\\N_\\nu = \\{\\xb: \\|\\xb - \\ttxb\\|_{\\ttHb} \\leq \\nu \\cdot \\lambda_{\\min}^{3/2}/L\\}\\quad \\text{ for } \\nu \\in (0, 1].\n\\end{equation}\nThen, with probability $1 - \\delta\\pi^2/6$, \nwe have $\\xb_t\\in \\N_\\nu$, for all $t\\geq \\T$ where~$\\T = \\T_1 + \\T_2$ with $\\T_1$ defined in \\eqref{T1} and\n\\begin{equation}\\label{T2}\n\\T_2 =  \\frac {\\kappa^2(1+\\epsilon)}{4\\rho\\beta(1-\\beta)(1-\\epsilon)}\\log\\rbr{\\frac{3L^2(f(\\xb_0) - f(\\ttxb))}{\\nu^2\\lambda_{\\min}^3}}.\n\\end{equation}\t\n\\end{corollary}","sha256_utf8":"0bc668941e6a03ab081fffea713bd64a907d5e104806886dd18552e768fb1ba4"},{"start":58504,"end":58558,"start_line":511,"end_line":513,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{pf:cor:1}.\n\\end{proof}","sha256_utf8":"2143b4062f6d8586e93eb11d969bf58e2af70e7c8649db7d5cb315838eaf4dcb"},{"start":58856,"end":59229,"start_line":517,"end_line":522,"raw_tex":"\\begin{lemma}\\label{lem:5}\nUnder Assumption \\ref{ass:4}, suppose $\\pb_t = -(\\tHb_t)^{-1}\\nabla f_t$. Then $\\mu_t = 1$ if $\\xb_t \\in \\N_{\\nu}$ and $(1-\\psi)\\Hb_t \\preceq \\tHb_t \\preceq(1+\\psi)\\Hb_t$ with $\\nu, \\psi$ satisfying\n\\begin{equation}\\label{cond:nupsi}\n0<\\nu \\leq \\frac{2}{3}(1/2-\\beta), \\quad\\quad 0<\\psi\\leq \\frac{1/2-\\beta}{3/2-\\beta}.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"f490a3a5fff9d960d0831a4a52fef9139c8dcc759f9ac95ba9124eead98d28d9"},{"start":59232,"end":59286,"start_line":525,"end_line":527,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{pf:lem:5}.\n\\end{proof}","sha256_utf8":"3bfb6a814131cf8940a8a1b1a239cb942aba73aeb8dee8b9ba3340f55a21e651"},{"start":59362,"end":59829,"start_line":531,"end_line":536,"raw_tex":"\\begin{lemma}\\label{lem:6}\nUnder Assumption \\ref{ass:4} and suppose $\\pb_t = -(\\tHb_t)^{-1}\\nabla f_t$, $\\xb_t \\in \\N_{\\nu}$, and $(1-\\psi)\\Hb_t \\preceq \\tHb_t \\preceq(1+\\psi)\\Hb_t$ with $\\nu, \\psi$ satisfying \\eqref{cond:nupsi}. Then, we have\n\\begin{equation*}\n\\nbr{\\xb_{t+1} - \\ttxb}_{\\ttHb} \\leq 3\\cbr{ \\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_t-\\ttxb\\|_{\\ttHb}^2 + \\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| \\cdot \\|\\xb_t - \\ttxb\\|_{\\ttHb}}.\n\\end{equation*}\n\\end{lemma}","sha256_utf8":"f263aeef75d6c1e0445a03b4c9160241f20485127a5e7baf008b6b3db3b6722f"},{"start":59831,"end":59885,"start_line":538,"end_line":540,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{pf:lem:6}.\n\\end{proof}","sha256_utf8":"951181f80e1be9a65e1a6d7658f6f704faeff4b546f4ab0eea67289be5a03ea0"},{"start":61519,"end":61573,"start_line":574,"end_line":576,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{pf:thm:3}.\n\\end{proof}","sha256_utf8":"3bca2e83c6ee6e6d69a9cb65e3eda40896da4a518a10f237bb44789915faf045"},{"start":104325,"end":104676,"start_line":1050,"end_line":1056,"raw_tex":"\\begin{lemma}\\label{lem:pre}\nLet $d\\geq 1$ and $\\delta\\in(0,1)$. If $d/\\delta\\geq e$, then for any $0<a\\leq 1$,\n\\begin{equation*}\nt\\geq \\frac {2\\log(d/(\\delta a))}{a}\\quad\\Longrightarrow\\quad\\frac{\\log(dt/\\delta)}{t} \\leq a,\n\\end{equation*}\nand moreover, the right hand side inequality fails if $t= \\log(d/(\\delta a))/a$ and $d/\\delta >e$.\n\\end{lemma}","sha256_utf8":"3e4f42897ab36915471e32573b09c57d6dfa84297fc34ffc845ab523d4c0bb17"},{"start":104678,"end":105682,"start_line":1058,"end_line":1072,"raw_tex":"\\begin{proof}\nLet $t = 2\\log(d/(\\delta a))/a$. We will show for this $t$ that $t\\geq \\log(dt/\\delta)/a$. We have\n\\begin{align*}\nt - \\frac{\\log(dt/\\delta)}{a}\n& =  \\frac{2\\log(d/(\\delta a))}{a} - \\frac{\\log(d/(\\delta a)) + \\log(2\\log(d/(\\delta a)))}{a}\n\\\\\n&=\\frac{\\log(d/(\\delta a)) - \\log(2\\log(d/(\\delta a)))}{a}\\\\\n&= \\frac1a\\log\\Big(\\frac{x}{2\\log(x)}\\Big)\\geq 0,\\qquad\\text{for}\\quad x=\\frac{d}{\\delta a},\n\\end{align*}\nwhere the last step is due to the fact that $x\\geq 2\\log(x)$ for all $x>0$. Now, we consider the function $f(t) = \\log(dt/\\delta)/t$. For any $t\\geq1$, its derivative is bounded as\n\\begin{align*}\n\\frac{\\partial f(t)}{\\partial t} = \\frac{1-\\log(dt/\\delta)}{t^2}\\leq 0,\n\\end{align*}\nwhich implies that for $t\\geq2\\log(d/(\\delta a))/a \\geq 1$ we get $\\log(dt/\\delta)/t =f(t)\\leq f(2\\log(d/(\\delta a))/a)\\leq a$. Finally, if $t=\\log(d/(\\delta a))/a$ and $d/\\delta >e$, then $\\log(dt/\\delta)/a=\\log(d/(\\delta a))/a + \\log(\\log(d/(\\delta a)))/a >t$, which completes the proof.\n\\end{proof}","sha256_utf8":"b572fd52b4c77438410f973fa3d154b1c870ffe168567a2d197faab7a580bd9d"},{"start":105685,"end":106093,"start_line":1075,"end_line":1081,"raw_tex":"\\begin{lemma}\\label{lem:pre:1}\nDefine the neighborhood $\\bar{\\N}_{\\nu} = \\{\\xb: f(\\xb) \\leq f(\\ttxb) + \\nu\\lambda_{\\min}^3/L^2\\}$ for $\\nu>0$. Suppose Assumption \\ref{ass:4} holds, then the following relation holds\n\\begin{equation*}\n\\bar{\\N}_{\\nu^2/3} \\subseteq \\N_{\\nu} \\subseteq \\bar{\\N}_{\\nu^2}, \\quad \\text{for any } \\nu \\in (0, 1]\n\\end{equation*}\nwhere $\\N_{\\nu}$ is defined in \\eqref{Nnu}.\t\n\\end{lemma}","sha256_utf8":"331b4ec44168b49c8293b2934261f9576ccfc8e423af42f02d32a57aec16a227"},{"start":106096,"end":108064,"start_line":1084,"end_line":1106,"raw_tex":"\\begin{proof}\nBy Assumption \\ref{ass:4}, we have\n\\begin{equation}\\label{pequ:A3}\n\\abr{f(\\xb) - f(\\ttxb) - \\frac{1}{2}\\nbr{\\xb - \\ttxb}_{\\ttHb}^2} \\leq \\frac{L}{6}\\nbr{\\xb - \\ttxb}^3.\n\\end{equation}\nSuppose $\\xb \\in \\bar{\\N}_{\\nu^2/3}$ for $\\nu\\in(0, 1]$, we have\n\\begin{equation}\\label{pequ:A4}\nf(\\ttxb) + \\frac{\\lambda_{\\min}}{2}\\|\\xb - \\ttxb\\|^2\\leq f(\\xb) \\leq f(\\ttxb) + \\frac{\\nu^2\\lambda_{\\min}^3}{3L^2} \\Longrightarrow \\|\\xb - \\ttxb\\| \\leq \\frac{\\sqrt{2}\\nu\\lambda_{\\min}}{\\sqrt{3}L}.\n\\end{equation}\nCombining \\eqref{pequ:A3} and \\eqref{pequ:A4}, we know that $\\xb \\in \\bar{\\N}_{\\nu^2/3}$ for $\\nu\\in(0, 1]$ leads to\n\\begin{align*}\nf(\\ttxb) + \\frac{\\nu^2\\lambda_{\\min}^3}{3L^2} \\geq f(\\xb) & \\stackrel{\\mathclap{\\eqref{pequ:A3}}}{\\geq} f(\\ttxb) + \\frac{1}{2}\\|\\xb - \\ttxb\\|_{\\ttHb}^2 - \\frac{L}{6}\\|\\xb-\\ttxb\\|^3\t\\\\\n& \\geq f(\\ttxb) + \\frac{1}{2}\\nbr{\\xb - \\ttxb}_{\\ttHb}^2 - \\frac{L}{6}\\nbr{\\xb - \\ttxb}\\cdot \\frac{\\nbr{\\xb - \\ttxb}_{\\ttHb}^2}{\\lambda_{\\min}}\\\\\n& = f(\\ttxb) + \\rbr{\\frac{1}{2}-\\frac{L\\|\\xb - \\ttxb\\|}{6\\lambda_{\\min}}}\\|\\xb - \\ttxb\\|_{\\ttHb}^2\\\\\n& \\stackrel{\\mathclap{\\eqref{pequ:A4}}}{\\geq} f(\\ttxb) + \\rbr{\\frac{1}{2} - \\frac{\\sqrt{2}\\nu}{6\\sqrt{3}}}\\|\\xb - \\ttxb\\|_{\\ttHb}^2 \\geq f(\\ttxb) + \\frac{\\|\\xb - \\ttxb\\|_{\\ttHb}^2}{3},\n\\end{align*}\nwhich further implies $\\|\\xb - \\ttxb\\|_{\\ttHb} \\leq \\nu\\lambda_{\\min}^{3/2}/L$ and $\\xb\\in \\N_{\\nu}$. On the other hand, suppose $\\xb \\in \\N_{\\nu}$, we have\n\\begin{align*}\nf(\\xb)  &\\stackrel{\\mathclap{\\eqref{pequ:A3}}}{\\leq} f(\\ttxb) + \\frac{1}{2}\\nbr{\\xb - \\ttxb}_{\\ttHb}^2 + \\frac{L}{6}\\nbr{\\xb - \\ttxb}^3 \\leq f(\\ttxb) + \\rbr{ \\frac{1}{2} + \\frac{L\\nbr{\\xb - \\ttxb}_{\\ttHb}}{6\\lambda_{\\min}^{3/2}}} \\nbr{\\xb - \\ttxb}_{\\ttHb}^2\\\\\n& \\leq f(\\ttxb) + \\rbr{ \\frac{1}{2} + \\frac{\\nu}{6}} \\nbr{\\xb - \\ttxb}_{\\ttHb}^2\\leq f(\\ttxb)  + \\nbr{\\xb - \\ttxb}_{\\ttHb}^2 \\leq f(\\ttxb) + \\frac{\\nu^2\\lambda_{\\min}^3}{L^2}.\n\\end{align*}\nThus, we have $\\xb \\in \\bar{\\N}_{\\nu^2}$. This completes the proof.\n\\end{proof}","sha256_utf8":"d80cb7a4969871e58db12b44932df94bf043bb9e4fa1a9511d449405d8ce6bbe"},{"start":108067,"end":108353,"start_line":1109,"end_line":1114,"raw_tex":"\\begin{lemma}\\label{lem:pre:2}\n\tSuppose Assumption \\ref{ass:4} holds, then we have\n\t\\begin{equation*}\n\t\t\\|\\Hb(\\xb_1)^{-1/2}\\Hb(\\xb_2)\\Hb(\\xb_1)^{-1/2} - \\Ib\\| \\leq \\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_1 - \\xb_2\\|_{\\ttHb}, \\quad \\forall \\xb_1, \\xb_2\\in \\mR^d.\n\t\\end{equation*}\n\\end{lemma}","sha256_utf8":"26efdd013e2138d8d09bb383cc8ef20fa001203465dc74998c2d06858a78712a"},{"start":108355,"end":108675,"start_line":1116,"end_line":1122,"raw_tex":"\\begin{proof}\nWe note that\n\\begin{equation*}\n\\|\\Hb(\\xb_1)^{-1/2}\\Hb(\\xb_2)\\Hb(\\xb_1)^{-1/2} - \\Ib\\|  \\leq \\frac{1}{\\lambda_{\\min}}\\|\\Hb(\\xb_1) - \\Hb(\\xb_2)\\| \\leq \\frac{L}{\\lambda_{\\min}}\\|\\xb_1 - \\xb_2\\| \\leq \\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_1 - \\xb_2\\|_{\\ttHb}.\n\\end{equation*}\nThis completes the proof.\n\\end{proof}","sha256_utf8":"48a89b04c5ea4b224a0c4708b1f7fdce1b8eb3d8664b6e3a08ad6034366c1047"},{"start":108697,"end":128470,"start_line":1128,"end_line":1374,"raw_tex":"\\subsection{Proof of Theorem \\ref{thm:2}}\\label{pf:thm:2}\n\nFor any $t\\geq 0$ and scalar $\\theta>0$, we have from Assumption \\ref{ass:3} that\n\\begin{align*}\n\\mE_i\\sbr{\\exp\\rbr{\\theta z_{i,t}\\Eb_i}}  & = \\Ib + \\theta z_{i,t}\\mE_i[\\Eb_i] + \\sum_{j=2}^{\\infty} \\frac{(\\theta z_{i,t})^j\\mE_i[\\Eb_i^j]}{j!}\n\\preceq \\Ib + \\sum_{j=2}^{\\infty}(\\theta z_{i,t}\\Upsilon_E)^{j-2}\\cdot \\frac{(\\theta z_{i,t}\\Upsilon_E)^2}{2}\\cdot\\Ib \\\\\n& = \\Ib + \\frac{(\\theta z_{i,t}\\Upsilon_E)^2}{2(1 - \\theta z_{i,t} \\Upsilon_E)}\\cdot \\Ib \\preceq\\exp\\rbr{ \\frac{(\\theta z_{i,t}\\Upsilon_E)^2}{2(1 - \\theta z_{i,t}\\Upsilon_E)} \\cdot \\Ib}, \\quad \\forall i = 0,1,\\ldots,t,\n\\end{align*}\nwhere the second equality holds for $\\cbr{\\theta: \\theta z_{i,t}\\Upsilon_E < 1}$. Therefore, we let $\\Theta_t \\coloneqq\\{\\theta: \\theta < 1/(z_t^{(\\max)}\\Upsilon_E )\\}$, and have for all $i = 0,1,\\ldots, t$,\n\\begin{equation*}\n\\mE_i\\sbr{\\exp\\rbr{\\theta z_{i,t}\\Eb_i}}  \\preceq \\exp\\rbr{\\frac{(\\theta z_{i,t}\\Upsilon_E)^2}{2(1 - \\theta z_t^{(\\max)} \\Upsilon_E) } \\cdot \\Ib}, \\quad \\forall\\theta\\in \\Theta_t.\n\\end{equation*}\nThe above inequality suggests that the condition \\eqref{equ:exp:cond} in Lemma \\ref{lem:2} holds with\n\\begin{equation*}\ng_t(\\theta) = \\frac{\\theta^2}{2(1 - \\theta z_t^{(\\max)} \\Upsilon_E) }\\quad \\text{ and }\\quad \\Ub_i = z_{i,t}^2\\Upsilon_E^2\\cdot \\Ib.\n\\end{equation*}\nLet $\\sigma^2 = \\|\\sum_{i=0}^{t}\\Ub_i\\| = \\Upsilon_E^2\\sum_{i=0}^{t}z_{i,t}^2$ in Lemma \\ref{lem:2}. Then we have for any $\\eta\\geq 0$,\n\\begin{equation*}\n\\mP\\rbr{\\nbr{\\barEb_t} \\geq \\eta} \\leq 2d\\cdot \\inf_{\\theta \\in \\Theta_t}\\exp\\rbr{-\\theta\\eta + g_t(\\theta)\\sigma^2}.\n\\end{equation*}\nPlugging $\\theta = \\eta/(z_t^{(\\max)} \\Upsilon_E \\eta + \\sigma^2) \\in \\Theta_t$ into the above right hand side, we obtain\n\\begin{equation*}\n\\mP\\rbr{\\|\\barEb_t\\| \\geq \\eta} \\leq 2d\\cdot\\exp\\rbr{-\\frac{\\eta^2/2}{ \\sigma^2 + z_t^{(\\max)} \\Upsilon_E \\eta}  }.\n\\end{equation*}\nThis completes the proof.\n\n\n\\subsection{Proof of Lemma \\ref{lem:3}}\\label{pf:lem:3}\n\nLet us define the event\n\\begin{equation*}\n\\bar{\\EE} = \\bigcap_{t=0}^{\\infty}\\cbr{\\nbr{\\barEb_t} \\leq 8\\Upsilon_E\\rbr{\\sqrt{\\frac{\\log(d(t+1)/\\delta)}{t+1}}\\vee \\frac{\\log(d(t+1)/\\delta)}{t+1} } }.\n\\end{equation*}\nBy Theorem \\ref{thm:2} and \\eqref{equ:equal:con}, we know $\\mP(\\bar{\\EE}) \\geq 1-\\delta\\pi^2/6$. We also note that\n\\begin{align}\\label{nequ:1}\n\\|\\barEb_t\\|\\leq \\epsilon\\lambda_{\\min} & \\Longleftarrow 8\\Upsilon_E\\rbr{\\sqrt{\\frac{\\log(d(t+1)/\\delta)}{t+1}}\\vee \\frac{\\log(d(t+1)/\\delta)}{t+1} } \\leq \\epsilon\\lambda_{\\min} \\nonumber\\\\\n& \\Longleftarrow \\frac{\\log(d(t+1)/\\delta)}{t+1} \\leq \\frac{\\epsilon^2}{64\\Upsilon^2} \\wedge1 = \\rbr{\\frac{\\epsilon}{8\\Upsilon}\\wedge 1}^2 \\nonumber\\\\\n&\\Longleftarrow t \\geq 4\\rbr{\\frac{8\\Upsilon}{\\epsilon}\\vee 1}^2 \\log\\cbr{\\frac{d}{\\delta}\\cdot\\rbr{\\frac{8\\Upsilon}{\\epsilon}\\vee 1}}=\\T_1. \\quad (\\text{Lemma \\ref{lem:pre}})\n\\end{align}\nTherefore, with probability at least $1-\\delta\\pi^2/6$, we have for any $t\\geq \\T_1$ that $\\|\\barEb_t\\|\\leq \\epsilon\\lambda_{\\min}$. This shows that the event $\\EE$ defined in \\eqref{event:EE} happens, and\n\\begin{equation*}\n(1-\\epsilon)\\lambda_{\\min}\\cdot\\Ib\\preceq\\tHb_t \\stackrel{\\eqref{equ:simple}}{=} \\frac{1}{t+1}\\sum_{i=0}^{t}\\Hb_i + \\barEb_t \\preceq (\\lambda_{\\max} + \\epsilon\\lambda_{\\min})\\cdot\\Ib\\preceq (1+\\epsilon)\\lambda_{\\max} \\cdot\\Ib.\n\\end{equation*}\nThis completes the proof.\n\n\n\\subsection{Proof of Lemma \\ref{lem:4}}\\label{pf:lem:4}\n\nBy Lemma \\ref{lem:3}, we know that $\\pb_t = -(\\tHb_t)^{-1}\\nabla f_t$ for $t\\geq \\T_1$. We apply Taylor's expansion:\n\\begin{align*}\n\tf(\\xb_t - \\mu_t(\\tHb_t)^{-1}\\nabla f_t ) & \\leq f(\\xb_t) - \\mu_t \\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t + \\frac{\\lambda_{\\max}\\mu_t^2}{2}\\nabla f_t^\\top(\\tHb_t)^{-2}\\nabla f_t \\\\\n\t& \\leq f(\\xb_t) - \\mu_t \\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t +\\frac{\\kappa\\mu_t^2}{2(1-\\epsilon)}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t \\quad (\\text{Lemma \\ref{lem:3}})\\\\\n\t& =  f(\\xb_t) - \\mu_t\\rbr{1 - \\frac{\\kappa\\mu_t}{2(1-\\epsilon)}}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t.\n\\end{align*}\nThus, the Armijo condition is satisfied if\n\\begin{equation*}\n1 - \\frac{\\kappa\\mu_t}{2(1-\\epsilon)} \\geq \\beta \\Longleftrightarrow \\mu_t \\leq \\frac{2(1-\\beta)(1-\\epsilon)}{\\kappa}.\n\\end{equation*}\nTherefore, the backtracking line search on Line 7 leads to a stepsize satisfying\n\\begin{equation}\\label{npequ:1}\n\\mu_t\\geq \\frac{2\\rho(1-\\beta)(1-\\epsilon)}{\\kappa}.\n\\end{equation}\nMoreover, we apply Lemma \\ref{lem:3} and strong convexity of $f$, and have\n\\begin{equation}\\label{npequ:2}\n\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t \\geq \\frac{1}{(1+\\epsilon)\\lambda_{\\max}}\\|\\nabla f_t\\|^2 \\geq \\frac{2}{(1+\\epsilon)\\kappa}(f(\\xb_t) - f(\\ttxb)).\n\\end{equation}\nCombining \\eqref{npequ:1}, \\eqref{npequ:2} with the Armijo condition, we have\n\\begin{align}\\label{pequ:B1}\nf(\\xb_{t+1}) - f(\\ttxb) &\\leq f(\\xb_t) - f(\\ttxb) - \\mu_t \\beta \\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t \\nonumber\\\\\n& \\leq f(\\xb_t) - f(\\ttxb) - \\frac{2\\rho\\beta(1-\\beta)(1-\\epsilon)}{\\kappa}\\cdot \\frac{2}{(1+\\epsilon)\\kappa}(f(\\xb_t) - f(\\ttxb)) \\nonumber\\\\\n& = (1-\\phi)(f(\\xb_t) - f(\\ttxb)),\n\\end{align}\nwhich shows the first part of the statement. Furthermore, we apply \\eqref{pequ:B1} recursively, apply strong convexity, and have\n\\begin{equation*}\n\\|\\xb_t - \\ttxb\\|^2\\leq \\frac{2(f(\\xb_t)-f(\\ttxb))}{\\lambda_{\\min}} \\\\\n\\stackrel{\\eqref{pequ:B1}}{\\leq} \\frac{2(1-\\phi)^{t-\\T_1}}{\\lambda_{\\min}}(f(\\xb_{\\T_1}) - f(\\ttxb)) \\leq \\frac{2(1-\\phi)^{t-\\T_1}}{\\lambda_{\\min}}(f(\\xb_0) - f(\\ttxb)).\n\\end{equation*}\nThe last statement follows from $\\|\\xb_t-\\ttxb\\|_{\\ttHb}^2\\leq \\lambda_{\\max}\\|\\xb_t - \\ttxb\\|^2$. This completes the proof.\n\n\n\\subsection{Proof of Corollary \\ref{cor:1}}\\label{pf:cor:1}\n\nWe condition on the event \\eqref{event:EE}, which happens with probability $1-\\delta\\pi^2/6$. By Lemma \\ref{lem:4}, we know that, for $t\\geq \\T_1$,\n\\begin{equation*}\nf(\\xb_t) - f(\\ttxb) \\leq (1 - \\phi)^{t-\\T_1}(f(\\xb_{\\T_1}) - f(\\ttxb)) \\leq (1 - \\phi)^{t-\\T_1}(f(\\xb_0) - f(\\ttxb)).\n\\end{equation*}\nBy Lemma \\ref{lem:pre:1}, to have $\\xb_t \\in \\N_\\nu$, it suffices to have $\\xb_t \\in \\bar{\\N}_{\\nu^2/3}$. Thus, we let\n\\begin{equation}\\label{equ:new}\n(1 - \\phi)^{t-\\T_1}(f(\\xb_0) - f(\\ttxb)) \\leq \\frac{\\nu^2\\lambda_{\\min}^3}{3L^2} \\Longleftarrow t-\\T_1 \\geq \\frac{\\log\\rbr{\\frac{3L^2(f(\\xb_0) - f(\\ttxb))}{\\nu^2\\lambda_{\\min}^3}}}{\\log\\rbr{\\frac{1}{1-\\phi}}}\\Longleftarrow t \\geq \\T_1 + \\T_2,\n\\end{equation}\nwhere $\\T_2$ is defined in \\eqref{T2} and the implication uses the fact that $\\log(1/(1-\\phi))\\geq \\phi$. Furthermore, since $f(\\xb_t)$ is always decreasing, we know after $\\T_1+T_2$ iterations $\\xb_t$ stays in $\\bar{\\N}_{\\nu^2/3}$, and hence stays in $\\N_{\\nu}$ (cf. Lemma \\ref{lem:pre:1}). This completes the proof.\n\n\n\\subsection{Proof of Lemma \\ref{lem:5}}\\label{pf:lem:5}\n\nIt suffices to show \n\\begin{equation*}\nf(\\xb_t - (\\tHb_t)^{-1}\\nabla f_t) \\leq f(\\xb_t) - \\beta\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t.\n\\end{equation*}\nBy Assumption \\ref{ass:4}, we have\n\\begin{align}\\label{npequ:3}\n& f(\\xb_t - (\\tHb_t)^{-1}\\nabla f_t) \\nonumber\\\\\n& \\leq f(\\xb_t) - \\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t + \\frac{1}{2}\\nabla f_t^\\top (\\tHb_t)^{-1}\\Hb_t(\\tHb_t)^{-1}\\nabla f_t + \\frac{L}{6}\\|(\\tHb_t)^{-1}\\nabla f_t\\|^3 \\nonumber\\\\\n& \\leq f(\\xb_t) - \\frac{1}{2}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t + \\frac{1}{2}\\nabla f_t^\\top(\\tHb_t)^{-1}(\\Hb_t - \\tHb_t)(\\tHb_t)^{-1}\\nabla f_t + \\frac{L}{6}\\|(\\tHb_t)^{-1}\\nabla f_t\\|^3 \\nonumber\\\\\n& \\leq f(\\xb_t) - \\frac{1}{2}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t + \\frac{1}{2}\\|(\\tHb_t)^{-1/2}(\\Hb_t - \\tHb_t)(\\tHb_t)^{-1/2}\\|\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t \\nonumber\\\\\n& \\quad + \\frac{L}{6}\\|(\\tHb_t)^{-1}\\nabla f_t\\| \\nabla f_t^\\top(\\tHb_t)^{-2}\\nabla f_t \\nonumber\\\\\n& \\leq f(\\xb_t) - \\frac{1}{2}\\nabla f_t^\\top (\\tHb_t)^{-1}\\nabla f_t + \\frac{\\psi}{2(1-\\psi)}\\nabla f_t^\\top (\\tHb_t)^{-1}\\nabla f_t + \\frac{L\\|(\\tHb_t)^{-1}\\nabla f_t\\|}{6(1-\\psi)\\lambda_{\\min}}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t \\nonumber\\\\\n& = f(\\xb_t) - \\rbr{\\frac{1}{2} - \\frac{\\psi}{2(1-\\psi)} - \\frac{L\\|(\\tHb_t)^{-1}\\nabla f_t\\|}{6(1-\\psi)\\lambda_{\\min}}}\\nabla f_t^\\top (\\tHb_t)^{-1}\\nabla f_t.\n\\end{align}\nFor term $\\|(\\tHb_t)^{-1}\\nabla f_t\\|$, we let $\\Hb_t^\\eta = \\Hb(\\xb_t^\\eta)$ with $\\xb_t^\\eta = \\ttxb + \\eta(\\xb_t - \\ttxb)$ for some $\\eta\\in(0, 1)$, and have\n\\begin{align}\\label{npequ:4}\n&\\|(\\tHb_t)^{-1}\\nabla f_t\\|^2  = (\\xb_t - \\ttxb)^\\top (\\ttHb)^{1/2}(\\ttHb)^{-1/2}\\Hb_t^\\eta(\\tHb_t)^{-2}\\Hb_t^\\eta(\\ttHb)^{-1/2}(\\ttHb)^{1/2}(\\xb_t-\\ttxb) \\nonumber\\\\\n& \\leq \\frac{\\|\\xb_t - \\ttxb\\|_{\\ttHb}^2}{(1-\\psi)\\lambda_{\\min}}\\|(\\ttHb)^{-1/2}\\Hb_t^\\eta(\\tHb_t)^{-1}\\Hb_t^\\eta(\\ttHb)^{-1/2}\\| \\nonumber\\\\\n& \\leq \\frac{\\nu^2\\lambda_{\\min}^2}{(1-\\psi)L^2}\\|(\\ttHb)^{-1/2}(\\Hb_t^\\eta)^{1/2}\\|^2\\|(\\Hb_t^\\eta)^{1/2}(\\tHb_t)^{-1}(\\Hb_t^\\eta)^{1/2}\\| \\quad (\\text{since } \\xb_t\\in \\N_{\\nu}) \\nonumber\\\\\n& \\leq \\frac{\\nu^2\\lambda_{\\min}^2}{(1-\\psi)L^2}\\|(\\ttHb)^{-1/2}\\Hb_t^\\eta(\\ttHb)^{-1/2}\\|\\|(\\Hb_t^\\eta)^{1/2}(\\Hb_t)^{-1}(\\Hb_t^\\eta)^{1/2}\\|\\|\\Hb_t^{1/2}(\\tHb_t)^{-1}\\Hb_t^{1/2}\\| \\nonumber\\\\\n& \\leq \\frac{\\nu^2\\lambda_{\\min}^2}{(1-\\psi)^2L^2}\\|(\\ttHb)^{-1/2}\\Hb_t^\\eta(\\ttHb)^{-1/2}\\| \\|(\\Hb_t)^{-1/2}\\Hb_t^\\eta(\\Hb_t)^{-1/2}\\|.\n\\end{align}\nNoting that $\\xb_t^\\eta\\in\\N_{\\nu}$ if $\\xb_t \\in \\N_\\nu$, we apply Lemma \\ref{lem:pre:2} and have\n\\begin{equation}\\label{npequ:5}\n\\|(\\ttHb)^{-1/2}\\Hb_t^\\eta(\\ttHb)^{-1/2}\\| \\vee \\|(\\Hb_t)^{-1/2}\\Hb_t^\\eta(\\Hb_t)^{-1/2}\\| \\leq 1 + \\nu\\leq 2.\n\\end{equation}\nCombining \\eqref{npequ:3}, \\eqref{npequ:4}, and \\eqref{npequ:5}, we finally obtain\n\\begin{equation*}\nf(\\xb_t - (\\tHb_t)^{-1}\\nabla f_t) \\leq f(\\xb_t) - \\rbr{\\frac{1}{2} - \\frac{\\psi}{2(1-\\psi)} - \\frac{\\nu}{3(1-\\psi)^2}}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t.\n\\end{equation*}\nThus, it suffices to let\n\\begin{equation*}\n\\frac{1}{2} - \\frac{\\psi}{2(1-\\psi)} - \\frac{\\nu}{3(1-\\psi)^2} \\geq \\beta \\Longleftarrow \\frac{\\psi}{2(1-\\psi)} \\vee \\frac{\\nu}{3(1-\\psi)^2} \\leq \\frac{1}{2}\\rbr{\\frac{1}{2}-\\beta}\n\\end{equation*}\nas implied by the conditions stated in the lemma. This completes the proof.\n\n\n\n\\subsection{Proof of Lemma \\ref{lem:6}}\\label{pf:lem:6}\n\nSince $\\xb_t\\in \\N_{\\nu}$ with $\\nu \\leq 2/3\\cdot (0.5-\\beta)$, we apply Lemma \\ref{lem:pre:2} and have\n\\begin{equation*}\n\\|(\\ttHb)^{-1/2}(\\Hb_t - \\ttHb)(\\ttHb)^{-1/2}\\| \\leq \\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_t - \\ttxb\\|_{\\ttHb} \\leq \\nu\\leq 0.5-\\beta.\n\\end{equation*}\nThus, we obtain\n\\begin{equation}\\label{pequ:B2}\n\\rbr{0.5+\\beta}\\ttHb\\preceq \\Hb_t\\preceq \\rbr{1.5-\\beta}\\ttHb. \n\\end{equation}\nSince $\\pb_t = -(\\tHb_t)^{-1}\\nabla f_t$ and $\\mu_t=1$, we have\n\\begin{align}\\label{npequ:6}\n& \\nbr{\\xb_{t+1} - \\ttxb}_{\\ttHb} = \\|\\xb_t - (\\tHb_t)^{-1}\\nabla f_t - \\ttxb\\|_{\\ttHb} \\nonumber\\\\\n& \\stackrel{\\mathclap{\\eqref{pequ:B2}}}{\\leq}\\frac{1}{\\sqrt{0.5+\\beta}}\\|\\xb_t - (\\tHb_t)^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} \\nonumber\\\\\n& \\leq \\frac{1}{\\sqrt{0.5+\\beta}}\\cbr{\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} + \\|\\Hb_t^{-1}\\nabla f_t - (\\tHb_t)^{-1}\\nabla f_t\\|_{\\Hb_t}}.\n\\end{align}\nFor the second term on the right hand side, we have\n\\begin{align}\\label{npequ:7}\n& \\|\\Hb_t^{-1}\\nabla f_t - (\\tHb_t)^{-1}\\nabla f_t\\|_{\\Hb_t} \\nonumber\\\\\n& = \\sqrt{\\nabla f_t^\\top(\\Hb_t^{-1} - (\\tHb_t)^{-1})\\Hb_t(\\Hb_t^{-1} - (\\tHb_t)^{-1})\\nabla f_t} \\nonumber\\\\\n& = \\sqrt{\\nabla f_t^\\top\\Hb_t^{-1/2}(\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2})^2\\Hb_t^{-1/2}\\nabla f_t} \\nonumber\\\\\n& \\leq \\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\|\\cdot\\|\\Hb_t^{-1}\\nabla f_t\\|_{\\Hb_t}  \\nonumber\\\\\n& \\leq  \\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\|\\cbr{\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} + \\|\\xb_t - \\ttxb\\|_{\\Hb_t}} \\nonumber\\\\\n& \\stackrel{\\mathclap{\\eqref{pequ:B2}}}{\\leq}\\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\|\\cbr{\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} + \\sqrt{1.5-\\beta}\\|\\xb_t - \\ttxb\\|_{\\ttHb}}.\n\\end{align}\nCombining \\eqref{npequ:6} and \\eqref{npequ:7},\n\\begin{align}\\label{pequ:B3}\n\\|\\xb_{t+1} - \\ttxb\\|_{\\ttHb}& \\leq\\frac{1}{\\sqrt{0.5+\\beta}} \\big\\{(1+\\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\|)\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} \\nonumber\\\\\n& \\quad + \\sqrt{1.5-\\beta}\\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\|\\cdot \\|\\xb_t - \\ttxb\\|_{\\ttHb}\\big\\}.\n\\end{align}\nSince $\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| \\leq \\psi\\leq 1/3$, we have\n\\begin{equation*}\n(1 - \\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| )\\cdot \\Ib \\preceq \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2} \\preceq (1+\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\|)\\cdot\\Ib\n\\end{equation*}\nand further\n\\begin{equation*}\n-\\frac{\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }{1+\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }\\cdot \\Ib\\preceq \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2} - \\Ib \\preceq \\frac{\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }{1-\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }\\cdot\\Ib.\n\\end{equation*}\nThus, we have\n\\begin{equation*}\n\\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\| \\leq \\frac{\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }{1 - \\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }\\leq \\frac{3}{2}\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\|.\n\\end{equation*}\nCombining the above inequality with \\eqref{pequ:B3}, we obtain\n\\begin{align}\\label{pequ:B4}\n&\\|\\xb_{t+1} - \\ttxb\\|_{\\ttHb} \\nonumber\\\\\n&  \\leq \\frac{3}{2\\sqrt{0.5+\\beta}} \\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} + \\frac{3}{2}\\sqrt{\\frac{1.5-\\beta}{0.5+\\beta}}\\cdot\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\|  \\cdot \\|\\xb_t - \\ttxb\\|_{\\ttHb} \\nonumber\\\\\n& \\leq \\frac{3\\sqrt{2}}{2}\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t}  + \\frac{3\\sqrt{3}}{2}\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\|  \\cdot \\|\\xb_t - \\ttxb\\|_{\\ttHb}.\n\\end{align}\nFurthermore,\n\\begin{align*}\n\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} & = \\|\\Hb_t^{-1/2}(\\Hb_t(\\xb_t - \\ttxb) - \\nabla f_t)\\|\\\\\n& \\leq \\frac{1}{\\sqrt{\\lambda_{\\min}}}\\nbr{\\Hb_t(\\xb_t - \\ttxb) - \\rbr{\\int_0^1\\Hb(\\ttxb + \\tau(\\xb_t - \\ttxb))d\\tau }(\\xb_t-\\ttxb)}\\\\\n& \\leq \\frac{\\|\\xb_t -\\ttxb\\|^2}{\\sqrt{\\lambda_{\\min}}}\\cdot \\int_0^1(1-\\tau)L d\\tau \\leq \\frac{L}{2\\lambda_{\\min}^{3/2}}\\|\\xb_t-\\ttxb\\|_{\\ttHb}^2.\n\\end{align*}\nCombining the above inequality with \\eqref{pequ:B4}, we complete the proof.\n\n\n\n\\subsection{Proof of Theorem \\ref{thm:3}}\\label{pf:thm:3}\n\nWe suppose the event \\eqref{event:EE} happens, which has probability $1-\\delta\\pi^2/6$. By Lemma \\ref{lem:4} and Corollary \\ref{cor:1}, we know that $\\xb_t$ converges $R$-linearly for all $t\\geq \\T_1$, and $\\xb_{\\T:\\T+\\J}\\in \\N_{\\nu}$ for any $\\J\\geq 0$.~Thus, it suffices to study the convergence after $\\T+\\J$. For any $t\\geq 0$, to apply Lemmas \\ref{lem:5} and \\ref{lem:6}, we characterize the difference between $\\tHb_{\\T+\\J+t}$ and $\\Hb_{\\T+\\J+t}$. We have\n\\begin{align}\\label{npequ:8}\n& \\|\\Hb_{\\T+\\J + t}^{-1/2}\\tHb_{\\T+\\J + t}\\Hb_{\\T+\\J + t}^{-1/2} -\\Ib\\|  \\nonumber\\\\\n& \\leq \\|\\Hb_{\\T+\\J + t}^{-1/2}(\\tHb_{\\T+\\J + t} - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\| + \\|\\Hb_{\\T+\\J + t}^{-1/2}\\ttHb\\Hb_{\\T+\\J + t}^{-1/2} - \\Ib\\| \\nonumber\\\\\n& \\stackrel{\\mathclap{\\text{Lemma \\ref{lem:pre:2}}}}{\\leq}\\;\\;\\; \\|\\Hb_{\\T+\\J + t}^{-1/2}(\\tHb_{\\T+\\J + t} - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\|  + \\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t} - \\ttxb\\|_{\\ttHb}. \n\\end{align}\nFor the first term $\\|\\Hb_{\\T+\\J + t}^{-1/2}(\\tHb_{\\T+\\J + t} - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\|$, we apply the formula \\eqref{equ:simple} and have\n\\begin{align}\\label{npequ:9}\n&\\|\\Hb_{\\T+\\J + t}^{-1/2}(\\tHb_{\\T+\\J + t} - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\| \\nonumber\\\\\n& \\stackrel{\\mathclap{\\eqref{equ:simple}}}{\\leq} \\frac{1}{\\lambda_{\\min}}\\nbr{\\barEb_{\\T+\\J+t}}  + \\frac{1}{\\T+\\J + t + 1}\\bigg\\{\\sum_{j=0}^{\\T-1}\\|\\Hb_{\\T+\\J + t}^{-1/2}(\\Hb_j - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\| \\nonumber\\\\\n&\\hskip6cm + \\sum_{j=\\T}^{\\T+\\J+t}\\|\\Hb_{\\T+\\J + t}^{-1/2}(\\Hb_j - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\| \\bigg\\} \\nonumber\\\\\n& \\leq \\frac{1}{\\lambda_{\\min}}\\|\\barEb_{\\T+\\J+t}\\| + \\frac{2\\T\\kappa}{\\T+\\J+t+1} + \\frac{L}{\\lambda_{\\min}(\\T+\\J+t+1)}\\sum_{j=\\T}^{\\T+\\J+t}\\|\\xb_j - \\ttxb\\| \\nonumber\\\\\n& \\stackrel{\\mathclap{\\eqref{event:EE}}}{\\leq} 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}} + \\frac{2\\T\\kappa}{\\T+\\J+t+1} \\nonumber\\\\ \n&\\hskip0.5cm +\\frac{L}{\\lambda_{\\min}(\\T+\\J+t+1)}\\sum_{j=\\T}^{\\T+\\J+t}\\sqrt{\\frac{2(f(\\xb_0) - f(\\ttxb))}{\\lambda_{\\min}}}(1 - \\phi)^{(j-\\T_1)/2} \\quad (\\text{also use Lemma \\ref{lem:4}}) \\nonumber\\\\\n& \\leq 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}} + \\frac{2\\T\\kappa}{\\T+\\J+t+1} + \\frac{L\\sqrt{2(f(\\xb_0) - f(\\ttxb))}}{\\lambda_{\\min}^{3/2}(\\T+\\J+t+1)}(1-\\phi)^{\\T_2/2}\\sum_{j=0}^{\\infty}(1-\\phi)^{j/2} \\nonumber\\\\\n& \\stackrel{\\mathclap{\\eqref{equ:new}}}{\\leq}\\; 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}} + \\frac{2\\T\\kappa}{\\T+\\J+t+1} + \\frac{\\sqrt{2}\\nu}{\\sqrt{3}(\\T+\\J+t+1)(1-\\sqrt{1-\\phi})} \\nonumber\\\\\n& \\leq 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}} + \\frac{2\\T\\kappa}{\\T+\\J+t+1} + \\frac{2\\nu}{(\\T+\\J+t+1)\\phi} \\nonumber\\\\\n& \\leq 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}} + \\frac{4\\T\\kappa}{\\T+\\J+t+1} = \\rho_t.\\quad (\\text{since } \\nu/\\phi \\leq1/\\phi\\leq\\T\\kappa)\n\\end{align}\nCombining \\eqref{npequ:8} and \\eqref{npequ:9} together, we obtain\n\\begin{equation}\\label{pequ:B5}\n\\|\\Hb_{\\T+\\J + t}^{-1/2}\\tHb_{\\T+\\J + t}\\Hb_{\\T+\\J + t}^{-1/2} -\\Ib\\|\\leq \\rho_t+\\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t} - \\ttxb\\|_{\\ttHb} \\leq \\rho_t + \\nu\\leq \\rho_0 + \\nu.\n\\end{equation}\nThus, in order to apply Lemma \\ref{lem:6} for all $\\{\\xb_{\\T+\\J+t}\\}_{t\\geq 0}$, we require $\\nu\\leq 2/3\\cdot (0.5-\\beta)$ and\n\\begin{align*}\n\\rho_0 + \\nu \\leq \\frac{0.5-\\beta}{1.5-\\beta} & \\xLeftarrow{\\J = 4\\T\\kappa/\\nu} 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J)/\\delta)}{\\T+\\J}} + 2\\nu \\leq \\frac{0.5-\\beta}{1.5-\\beta} \\\\\n& \\Longleftarrow 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J)/\\delta)}{\\T+\\J}}  \\vee\\nu\\leq \\frac{1}{3}\\frac{0.5-\\beta}{1.5-\\beta}\\\\\n& \\stackrel{\\eqref{cond:epsnu}}{\\Longleftarrow} 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J)/\\delta)}{\\T+\\J}}  \\leq \\epsilon \\; \\text{ and }\\; \\epsilon\\vee \\nu \\leq \\frac{1}{3}\\frac{0.5-\\beta}{1.5-\\beta},\n\\end{align*}\nwhich is implied by \\eqref{cond:epsnu} and \\eqref{nequ:1}. Thus, for any $t\\geq 0$, we apply Lemma \\ref{lem:6} and obtain\n\\begin{align}\\label{pequ:B6}\n& \\|\\xb_{\\T+\\J+t+1}-\\ttxb\\|_{\\ttHb} \\nonumber\\\\\n& \\leq 3\\cbr{\\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t}-\\ttxb\\|_{\\ttHb}^2 + \\|\\Hb_{\\T+\\J + t}^{-1/2}\\tHb_{\\T+\\J + t}\\Hb_{\\T+\\J + t}^{-1/2} -\\Ib\\|\\cdot \\|\\xb_{\\T+\\J+t} - \\ttxb\\|_{\\ttHb}} \\nonumber\\\\\n&\\stackrel{\\mathclap{\\eqref{pequ:B5}}}{\\leq} 3\\cbr{\\frac{2L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t}-\\ttxb\\|_{\\ttHb}^2 +  \\rho_t\\|\\xb_{\\T+\\J+t}-\\ttxb\\|_{\\ttHb}}.\n\\end{align}\nWe then claim that\n\\begin{equation}\\label{pequ:B7}\n\\frac{2L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t}-\\ttxb\\|_{\\ttHb} \\leq 3\\rho_t, \\quad \\forall t\\geq 0.\n\\end{equation}\nWe prove \\eqref{pequ:B7} by induction. For $t = 0$, we note that\n\\begin{equation*}\n\\frac{2L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J}-\\ttxb\\|_{\\ttHb} \\leq 2\\nu = \\frac{12\\T\\kappa}{6\\T\\kappa/\\nu}\\leq \\frac{12\\T\\kappa}{\\T + 4\\T\\kappa/\\nu +1} = \\frac{12\\T\\kappa}{\\T + \\J +1} \\leq 3\\rho_0.\n\\end{equation*}\nSuppose \\eqref{pequ:B7} holds for $t\\geq 0$, then \\eqref{pequ:B6} leads to\n\\begin{equation*}\n\\frac{2L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t+1}-\\ttxb\\|_{\\ttHb} \\leq \\frac{2L}{\\lambda_{\\min}^{3/2}}\\cdot 12\\rho_t\\|\\xb_{\\T+\\J+t}-\\ttxb\\|_{\\ttHb} \\stackrel{\\eqref{pequ:B7}}{\\leq} 36\\rho_t^2.\n\\end{equation*}\nOn the other hand, using $(\\T+\\J+t+2)\\leq 2(\\T+\\J+t+1)$, $\\forall t\\geq 0$, we know $\\rho_t/2 \\leq \\rho_{t+1}$. Thus, it suffices to show $36\\rho_t^2\\leq 3\\rho_t/2\\Longleftrightarrow 24\\rho_t\\leq 1$. Since $\\rho_t$ is decreasing, we require $24\\rho_0\\leq 1$. Note that\n\\begin{equation*}\n\\rho_0 \\leq 8\\Upsilon\\sqrt{\\frac{\\log(d\\T/\\delta)}{\\T}} + \\nu \\stackrel{\\eqref{T2}}{\\leq} \\epsilon + \\nu.\n\\end{equation*}\nThus, $24\\rho_0\\leq 1$ is guaranteed by \\eqref{cond:epsnu}. Combining \\eqref{pequ:B6} and \\eqref{pequ:B7} completes the proof.\n\n\n","sha256_utf8":"8415969562e27a14b80711305c883754c03b021db02fc4979acc808d7ac92597"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"均匀平均随机牛顿收敛","source_units":[{"start":60492,"end":61516,"start_line":556,"end_line":571,"raw_tex":"\\begin{theorem}\\label{thm:3}\t\nConsider Algorithm \\ref{alg:1} with $w_t = t+1$, $\\forall t\\geq 0$. Under Assumptions \\ref{ass:3}, \\ref{ass:4}, we let $\\delta \\in(0, 1)$ satisfy $d/\\delta\\geq e$, and let $\\epsilon, \\nu\\in(0, 1)$ satisfy\n\\begin{equation}\\label{cond:epsnu}\n\\epsilon \\vee \\nu \\leq \\frac{1}{3}\\frac{0.5-\\beta}{1.5-\\beta}\\wedge \\frac{1}{48}.\n\\end{equation}\nDefine the neighborhood $\\N_{\\nu}$ as in \\eqref{Nnu}, and define $\\T = \\T_1 +\\T_2$ with $\\T_1$ given~by \\eqref{T1} and $\\T_2$ given by \\eqref{T2}. We also let $\\J = 4\\T\\kappa/\\nu$. Then, with probability $1-\\delta\\pi^2/6$, we have that $\\xb_{\\T_1:\\T+\\J}$ converges $R$-linearly, $\\xb_{\\T:\\T+\\J}\\in \\N_{\\nu}$, and\n\\begin{equation*}\n\\|\\xb_{\\T+\\J +t +1} - \\ttxb\\|_{\\ttHb} \\leq 12 \\rho_t\\|\\xb_{\\T+\\J +t} - \\ttxb\\|_{\\ttHb}, \\quad \\forall t\\geq 0\n\\end{equation*}\nwith\n\\begin{equation*}\n\\rho_t = \\frac{4\\T\\kappa}{\\T+\\J+t+1} + 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}}\n\\end{equation*}\nsatisfying $24\\rho_t\\leq 1$, $\\forall t\\geq 0$.\n\n\\end{theorem}","sha256_utf8":"7a874d6a41c1461869f8c30050f3e62ae7f47b11f4c9fdd19acabf6d8d3fdf1e"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"主定理完整证明","source_units":[{"start":61519,"end":61573,"start_line":574,"end_line":576,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{pf:thm:3}.\n\\end{proof}","sha256_utf8":"3bca2e83c6ee6e6d69a9cb65e3eda40896da4a518a10f237bb44789915faf045"},{"start":122768,"end":128470,"start_line":1319,"end_line":1374,"raw_tex":"\\subsection{Proof of Theorem \\ref{thm:3}}\\label{pf:thm:3}\n\nWe suppose the event \\eqref{event:EE} happens, which has probability $1-\\delta\\pi^2/6$. By Lemma \\ref{lem:4} and Corollary \\ref{cor:1}, we know that $\\xb_t$ converges $R$-linearly for all $t\\geq \\T_1$, and $\\xb_{\\T:\\T+\\J}\\in \\N_{\\nu}$ for any $\\J\\geq 0$.~Thus, it suffices to study the convergence after $\\T+\\J$. For any $t\\geq 0$, to apply Lemmas \\ref{lem:5} and \\ref{lem:6}, we characterize the difference between $\\tHb_{\\T+\\J+t}$ and $\\Hb_{\\T+\\J+t}$. We have\n\\begin{align}\\label{npequ:8}\n& \\|\\Hb_{\\T+\\J + t}^{-1/2}\\tHb_{\\T+\\J + t}\\Hb_{\\T+\\J + t}^{-1/2} -\\Ib\\|  \\nonumber\\\\\n& \\leq \\|\\Hb_{\\T+\\J + t}^{-1/2}(\\tHb_{\\T+\\J + t} - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\| + \\|\\Hb_{\\T+\\J + t}^{-1/2}\\ttHb\\Hb_{\\T+\\J + t}^{-1/2} - \\Ib\\| \\nonumber\\\\\n& \\stackrel{\\mathclap{\\text{Lemma \\ref{lem:pre:2}}}}{\\leq}\\;\\;\\; \\|\\Hb_{\\T+\\J + t}^{-1/2}(\\tHb_{\\T+\\J + t} - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\|  + \\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t} - \\ttxb\\|_{\\ttHb}. \n\\end{align}\nFor the first term $\\|\\Hb_{\\T+\\J + t}^{-1/2}(\\tHb_{\\T+\\J + t} - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\|$, we apply the formula \\eqref{equ:simple} and have\n\\begin{align}\\label{npequ:9}\n&\\|\\Hb_{\\T+\\J + t}^{-1/2}(\\tHb_{\\T+\\J + t} - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\| \\nonumber\\\\\n& \\stackrel{\\mathclap{\\eqref{equ:simple}}}{\\leq} \\frac{1}{\\lambda_{\\min}}\\nbr{\\barEb_{\\T+\\J+t}}  + \\frac{1}{\\T+\\J + t + 1}\\bigg\\{\\sum_{j=0}^{\\T-1}\\|\\Hb_{\\T+\\J + t}^{-1/2}(\\Hb_j - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\| \\nonumber\\\\\n&\\hskip6cm + \\sum_{j=\\T}^{\\T+\\J+t}\\|\\Hb_{\\T+\\J + t}^{-1/2}(\\Hb_j - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\| \\bigg\\} \\nonumber\\\\\n& \\leq \\frac{1}{\\lambda_{\\min}}\\|\\barEb_{\\T+\\J+t}\\| + \\frac{2\\T\\kappa}{\\T+\\J+t+1} + \\frac{L}{\\lambda_{\\min}(\\T+\\J+t+1)}\\sum_{j=\\T}^{\\T+\\J+t}\\|\\xb_j - \\ttxb\\| \\nonumber\\\\\n& \\stackrel{\\mathclap{\\eqref{event:EE}}}{\\leq} 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}} + \\frac{2\\T\\kappa}{\\T+\\J+t+1} \\nonumber\\\\ \n&\\hskip0.5cm +\\frac{L}{\\lambda_{\\min}(\\T+\\J+t+1)}\\sum_{j=\\T}^{\\T+\\J+t}\\sqrt{\\frac{2(f(\\xb_0) - f(\\ttxb))}{\\lambda_{\\min}}}(1 - \\phi)^{(j-\\T_1)/2} \\quad (\\text{also use Lemma \\ref{lem:4}}) \\nonumber\\\\\n& \\leq 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}} + \\frac{2\\T\\kappa}{\\T+\\J+t+1} + \\frac{L\\sqrt{2(f(\\xb_0) - f(\\ttxb))}}{\\lambda_{\\min}^{3/2}(\\T+\\J+t+1)}(1-\\phi)^{\\T_2/2}\\sum_{j=0}^{\\infty}(1-\\phi)^{j/2} \\nonumber\\\\\n& \\stackrel{\\mathclap{\\eqref{equ:new}}}{\\leq}\\; 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}} + \\frac{2\\T\\kappa}{\\T+\\J+t+1} + \\frac{\\sqrt{2}\\nu}{\\sqrt{3}(\\T+\\J+t+1)(1-\\sqrt{1-\\phi})} \\nonumber\\\\\n& \\leq 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}} + \\frac{2\\T\\kappa}{\\T+\\J+t+1} + \\frac{2\\nu}{(\\T+\\J+t+1)\\phi} \\nonumber\\\\\n& \\leq 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}} + \\frac{4\\T\\kappa}{\\T+\\J+t+1} = \\rho_t.\\quad (\\text{since } \\nu/\\phi \\leq1/\\phi\\leq\\T\\kappa)\n\\end{align}\nCombining \\eqref{npequ:8} and \\eqref{npequ:9} together, we obtain\n\\begin{equation}\\label{pequ:B5}\n\\|\\Hb_{\\T+\\J + t}^{-1/2}\\tHb_{\\T+\\J + t}\\Hb_{\\T+\\J + t}^{-1/2} -\\Ib\\|\\leq \\rho_t+\\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t} - \\ttxb\\|_{\\ttHb} \\leq \\rho_t + \\nu\\leq \\rho_0 + \\nu.\n\\end{equation}\nThus, in order to apply Lemma \\ref{lem:6} for all $\\{\\xb_{\\T+\\J+t}\\}_{t\\geq 0}$, we require $\\nu\\leq 2/3\\cdot (0.5-\\beta)$ and\n\\begin{align*}\n\\rho_0 + \\nu \\leq \\frac{0.5-\\beta}{1.5-\\beta} & \\xLeftarrow{\\J = 4\\T\\kappa/\\nu} 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J)/\\delta)}{\\T+\\J}} + 2\\nu \\leq \\frac{0.5-\\beta}{1.5-\\beta} \\\\\n& \\Longleftarrow 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J)/\\delta)}{\\T+\\J}}  \\vee\\nu\\leq \\frac{1}{3}\\frac{0.5-\\beta}{1.5-\\beta}\\\\\n& \\stackrel{\\eqref{cond:epsnu}}{\\Longleftarrow} 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J)/\\delta)}{\\T+\\J}}  \\leq \\epsilon \\; \\text{ and }\\; \\epsilon\\vee \\nu \\leq \\frac{1}{3}\\frac{0.5-\\beta}{1.5-\\beta},\n\\end{align*}\nwhich is implied by \\eqref{cond:epsnu} and \\eqref{nequ:1}. Thus, for any $t\\geq 0$, we apply Lemma \\ref{lem:6} and obtain\n\\begin{align}\\label{pequ:B6}\n& \\|\\xb_{\\T+\\J+t+1}-\\ttxb\\|_{\\ttHb} \\nonumber\\\\\n& \\leq 3\\cbr{\\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t}-\\ttxb\\|_{\\ttHb}^2 + \\|\\Hb_{\\T+\\J + t}^{-1/2}\\tHb_{\\T+\\J + t}\\Hb_{\\T+\\J + t}^{-1/2} -\\Ib\\|\\cdot \\|\\xb_{\\T+\\J+t} - \\ttxb\\|_{\\ttHb}} \\nonumber\\\\\n&\\stackrel{\\mathclap{\\eqref{pequ:B5}}}{\\leq} 3\\cbr{\\frac{2L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t}-\\ttxb\\|_{\\ttHb}^2 +  \\rho_t\\|\\xb_{\\T+\\J+t}-\\ttxb\\|_{\\ttHb}}.\n\\end{align}\nWe then claim that\n\\begin{equation}\\label{pequ:B7}\n\\frac{2L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t}-\\ttxb\\|_{\\ttHb} \\leq 3\\rho_t, \\quad \\forall t\\geq 0.\n\\end{equation}\nWe prove \\eqref{pequ:B7} by induction. For $t = 0$, we note that\n\\begin{equation*}\n\\frac{2L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J}-\\ttxb\\|_{\\ttHb} \\leq 2\\nu = \\frac{12\\T\\kappa}{6\\T\\kappa/\\nu}\\leq \\frac{12\\T\\kappa}{\\T + 4\\T\\kappa/\\nu +1} = \\frac{12\\T\\kappa}{\\T + \\J +1} \\leq 3\\rho_0.\n\\end{equation*}\nSuppose \\eqref{pequ:B7} holds for $t\\geq 0$, then \\eqref{pequ:B6} leads to\n\\begin{equation*}\n\\frac{2L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t+1}-\\ttxb\\|_{\\ttHb} \\leq \\frac{2L}{\\lambda_{\\min}^{3/2}}\\cdot 12\\rho_t\\|\\xb_{\\T+\\J+t}-\\ttxb\\|_{\\ttHb} \\stackrel{\\eqref{pequ:B7}}{\\leq} 36\\rho_t^2.\n\\end{equation*}\nOn the other hand, using $(\\T+\\J+t+2)\\leq 2(\\T+\\J+t+1)$, $\\forall t\\geq 0$, we know $\\rho_t/2 \\leq \\rho_{t+1}$. Thus, it suffices to show $36\\rho_t^2\\leq 3\\rho_t/2\\Longleftrightarrow 24\\rho_t\\leq 1$. Since $\\rho_t$ is decreasing, we require $24\\rho_0\\leq 1$. Note that\n\\begin{equation*}\n\\rho_0 \\leq 8\\Upsilon\\sqrt{\\frac{\\log(d\\T/\\delta)}{\\T}} + \\nu \\stackrel{\\eqref{T2}}{\\leq} \\epsilon + \\nu.\n\\end{equation*}\nThus, $24\\rho_0\\leq 1$ is guaranteed by \\eqref{cond:epsnu}. Combining \\eqref{pequ:B6} and \\eqref{pequ:B7} completes the proof.\n\n\n","sha256_utf8":"a80182a7de80847a71b61f547aad8a371209ac54f7437ea06e894c3e0ce68795"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"主证明附录定位","source_units":[{"start":61519,"end":61573,"start_line":574,"end_line":576,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{pf:thm:3}.\n\\end{proof}","sha256_utf8":"3bca2e83c6ee6e6d69a9cb65e3eda40896da4a518a10f237bb44789915faf045"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"条件化与相对误差分解","source_units":[{"start":122809,"end":123780,"start_line":1319,"end_line":1326,"raw_tex":"\\label{pf:thm:3}\n\nWe suppose the event \\eqref{event:EE} happens, which has probability $1-\\delta\\pi^2/6$. By Lemma \\ref{lem:4} and Corollary \\ref{cor:1}, we know that $\\xb_t$ converges $R$-linearly for all $t\\geq \\T_1$, and $\\xb_{\\T:\\T+\\J}\\in \\N_{\\nu}$ for any $\\J\\geq 0$.~Thus, it suffices to study the convergence after $\\T+\\J$. For any $t\\geq 0$, to apply Lemmas \\ref{lem:5} and \\ref{lem:6}, we characterize the difference between $\\tHb_{\\T+\\J+t}$ and $\\Hb_{\\T+\\J+t}$. We have\n\\begin{align}\\label{npequ:8}\n& \\|\\Hb_{\\T+\\J + t}^{-1/2}\\tHb_{\\T+\\J + t}\\Hb_{\\T+\\J + t}^{-1/2} -\\Ib\\|  \\nonumber\\\\\n& \\leq \\|\\Hb_{\\T+\\J + t}^{-1/2}(\\tHb_{\\T+\\J + t} - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\| + \\|\\Hb_{\\T+\\J + t}^{-1/2}\\ttHb\\Hb_{\\T+\\J + t}^{-1/2} - \\Ib\\| \\nonumber\\\\\n& \\stackrel{\\mathclap{\\text{Lemma \\ref{lem:pre:2}}}}{\\leq}\\;\\;\\; \\|\\Hb_{\\T+\\J + t}^{-1/2}(\\tHb_{\\T+\\J + t} - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\|  + \\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t} - \\ttxb\\|_{\\ttHb}. \n\\end{align}\n","sha256_utf8":"7d066d441286e02939dd630da8f7bc5312a6ec91aa2ed5c646619f35c61aadad"}]},{"id":"S005","semantic_type":"proof_step","kind":"proof_step","title":"控制历史偏差与平均噪声","source_units":[{"start":123780,"end":125656,"start_line":1327,"end_line":1339,"raw_tex":"For the first term $\\|\\Hb_{\\T+\\J + t}^{-1/2}(\\tHb_{\\T+\\J + t} - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\|$, we apply the formula \\eqref{equ:simple} and have\n\\begin{align}\\label{npequ:9}\n&\\|\\Hb_{\\T+\\J + t}^{-1/2}(\\tHb_{\\T+\\J + t} - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\| \\nonumber\\\\\n& \\stackrel{\\mathclap{\\eqref{equ:simple}}}{\\leq} \\frac{1}{\\lambda_{\\min}}\\nbr{\\barEb_{\\T+\\J+t}}  + \\frac{1}{\\T+\\J + t + 1}\\bigg\\{\\sum_{j=0}^{\\T-1}\\|\\Hb_{\\T+\\J + t}^{-1/2}(\\Hb_j - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\| \\nonumber\\\\\n&\\hskip6cm + \\sum_{j=\\T}^{\\T+\\J+t}\\|\\Hb_{\\T+\\J + t}^{-1/2}(\\Hb_j - \\ttHb)\\Hb_{\\T+\\J + t}^{-1/2}\\| \\bigg\\} \\nonumber\\\\\n& \\leq \\frac{1}{\\lambda_{\\min}}\\|\\barEb_{\\T+\\J+t}\\| + \\frac{2\\T\\kappa}{\\T+\\J+t+1} + \\frac{L}{\\lambda_{\\min}(\\T+\\J+t+1)}\\sum_{j=\\T}^{\\T+\\J+t}\\|\\xb_j - \\ttxb\\| \\nonumber\\\\\n& \\stackrel{\\mathclap{\\eqref{event:EE}}}{\\leq} 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}} + \\frac{2\\T\\kappa}{\\T+\\J+t+1} \\nonumber\\\\ \n&\\hskip0.5cm +\\frac{L}{\\lambda_{\\min}(\\T+\\J+t+1)}\\sum_{j=\\T}^{\\T+\\J+t}\\sqrt{\\frac{2(f(\\xb_0) - f(\\ttxb))}{\\lambda_{\\min}}}(1 - \\phi)^{(j-\\T_1)/2} \\quad (\\text{also use Lemma \\ref{lem:4}}) \\nonumber\\\\\n& \\leq 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}} + \\frac{2\\T\\kappa}{\\T+\\J+t+1} + \\frac{L\\sqrt{2(f(\\xb_0) - f(\\ttxb))}}{\\lambda_{\\min}^{3/2}(\\T+\\J+t+1)}(1-\\phi)^{\\T_2/2}\\sum_{j=0}^{\\infty}(1-\\phi)^{j/2} \\nonumber\\\\\n& \\stackrel{\\mathclap{\\eqref{equ:new}}}{\\leq}\\; 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}} + \\frac{2\\T\\kappa}{\\T+\\J+t+1} + \\frac{\\sqrt{2}\\nu}{\\sqrt{3}(\\T+\\J+t+1)(1-\\sqrt{1-\\phi})} \\nonumber\\\\\n& \\leq 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}} + \\frac{2\\T\\kappa}{\\T+\\J+t+1} + \\frac{2\\nu}{(\\T+\\J+t+1)\\phi} \\nonumber\\\\\n& \\leq 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J+t+1)/\\delta)}{\\T+\\J+t+1}} + \\frac{4\\T\\kappa}{\\T+\\J+t+1} = \\rho_t.\\quad (\\text{since } \\nu/\\phi \\leq1/\\phi\\leq\\T\\kappa)\n\\end{align}\n","sha256_utf8":"61929926aed92a5283401c1dd09346d6efa0c8d58d2a3fd476b34273e161acb9"}]},{"id":"S006","semantic_type":"proof_step","kind":"proof_step","title":"合并相对海森误差界","source_units":[{"start":125656,"end":125950,"start_line":1340,"end_line":1343,"raw_tex":"Combining \\eqref{npequ:8} and \\eqref{npequ:9} together, we obtain\n\\begin{equation}\\label{pequ:B5}\n\\|\\Hb_{\\T+\\J + t}^{-1/2}\\tHb_{\\T+\\J + t}\\Hb_{\\T+\\J + t}^{-1/2} -\\Ib\\|\\leq \\rho_t+\\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t} - \\ttxb\\|_{\\ttHb} \\leq \\rho_t + \\nu\\leq \\rho_0 + \\nu.\n\\end{equation}\n","sha256_utf8":"2c8dbc3f0af0cd228a3c76b6e996d907c46ec853cddfa1558e9b9b5322d6007c"}]},{"id":"S007","semantic_type":"proof_step","kind":"proof_step","title":"验证局部递推适用条件","source_units":[{"start":125950,"end":126720,"start_line":1344,"end_line":1350,"raw_tex":"Thus, in order to apply Lemma \\ref{lem:6} for all $\\{\\xb_{\\T+\\J+t}\\}_{t\\geq 0}$, we require $\\nu\\leq 2/3\\cdot (0.5-\\beta)$ and\n\\begin{align*}\n\\rho_0 + \\nu \\leq \\frac{0.5-\\beta}{1.5-\\beta} & \\xLeftarrow{\\J = 4\\T\\kappa/\\nu} 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J)/\\delta)}{\\T+\\J}} + 2\\nu \\leq \\frac{0.5-\\beta}{1.5-\\beta} \\\\\n& \\Longleftarrow 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J)/\\delta)}{\\T+\\J}}  \\vee\\nu\\leq \\frac{1}{3}\\frac{0.5-\\beta}{1.5-\\beta}\\\\\n& \\stackrel{\\eqref{cond:epsnu}}{\\Longleftarrow} 8\\Upsilon\\sqrt{\\frac{\\log(d(\\T+\\J)/\\delta)}{\\T+\\J}}  \\leq \\epsilon \\; \\text{ and }\\; \\epsilon\\vee \\nu \\leq \\frac{1}{3}\\frac{0.5-\\beta}{1.5-\\beta},\n\\end{align*}\nwhich is implied by \\eqref{cond:epsnu} and \\eqref{nequ:1}. Thus, for any $t\\geq 0$, we apply Lemma \\ref{lem:6} and obtain\n","sha256_utf8":"fbde49358d9c4b49840935e933da7fc42bda2edfa32fe88bbda81bd04e0a0329"}]},{"id":"S008","semantic_type":"proof_step","kind":"proof_step","title":"建立局部误差递推","source_units":[{"start":126720,"end":127169,"start_line":1351,"end_line":1355,"raw_tex":"\\begin{align}\\label{pequ:B6}\n& \\|\\xb_{\\T+\\J+t+1}-\\ttxb\\|_{\\ttHb} \\nonumber\\\\\n& \\leq 3\\cbr{\\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t}-\\ttxb\\|_{\\ttHb}^2 + \\|\\Hb_{\\T+\\J + t}^{-1/2}\\tHb_{\\T+\\J + t}\\Hb_{\\T+\\J + t}^{-1/2} -\\Ib\\|\\cdot \\|\\xb_{\\T+\\J+t} - \\ttxb\\|_{\\ttHb}} \\nonumber\\\\\n&\\stackrel{\\mathclap{\\eqref{pequ:B5}}}{\\leq} 3\\cbr{\\frac{2L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t}-\\ttxb\\|_{\\ttHb}^2 +  \\rho_t\\|\\xb_{\\T+\\J+t}-\\ttxb\\|_{\\ttHb}}.\n\\end{align}\n","sha256_utf8":"5ff8e0590726c05aaddd62999a9dc6bbf61c07231c0e7f486b94e1421754d288"}]},{"id":"S009","semantic_type":"proof_step","kind":"proof_step","title":"归纳命题与初始验证","source_units":[{"start":127169,"end":127631,"start_line":1356,"end_line":1363,"raw_tex":"We then claim that\n\\begin{equation}\\label{pequ:B7}\n\\frac{2L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t}-\\ttxb\\|_{\\ttHb} \\leq 3\\rho_t, \\quad \\forall t\\geq 0.\n\\end{equation}\nWe prove \\eqref{pequ:B7} by induction. For $t = 0$, we note that\n\\begin{equation*}\n\\frac{2L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J}-\\ttxb\\|_{\\ttHb} \\leq 2\\nu = \\frac{12\\T\\kappa}{6\\T\\kappa/\\nu}\\leq \\frac{12\\T\\kappa}{\\T + 4\\T\\kappa/\\nu +1} = \\frac{12\\T\\kappa}{\\T + \\J +1} \\leq 3\\rho_0.\n\\end{equation*}\n","sha256_utf8":"1cdb2933b72954cbf125d3cb5c92e2eb00b7ebb58ec2d4be1a55c615c59a7933"}]},{"id":"S010","semantic_type":"proof_step","kind":"proof_step","title":"归纳推进的平方误差界","source_units":[{"start":127631,"end":127932,"start_line":1364,"end_line":1367,"raw_tex":"Suppose \\eqref{pequ:B7} holds for $t\\geq 0$, then \\eqref{pequ:B6} leads to\n\\begin{equation*}\n\\frac{2L}{\\lambda_{\\min}^{3/2}}\\|\\xb_{\\T+\\J+t+1}-\\ttxb\\|_{\\ttHb} \\leq \\frac{2L}{\\lambda_{\\min}^{3/2}}\\cdot 12\\rho_t\\|\\xb_{\\T+\\J+t}-\\ttxb\\|_{\\ttHb} \\stackrel{\\eqref{pequ:B7}}{\\leq} 36\\rho_t^2.\n\\end{equation*}\n","sha256_utf8":"c88203922ebadcb020c8cef565021e32c49d3be006faa8bf2bd850b0dc268e1a"}]},{"id":"S011","semantic_type":"proof_step","kind":"proof_step","title":"速率比较与归纳闭合","source_units":[{"start":127932,"end":128468,"start_line":1368,"end_line":1372,"raw_tex":"On the other hand, using $(\\T+\\J+t+2)\\leq 2(\\T+\\J+t+1)$, $\\forall t\\geq 0$, we know $\\rho_t/2 \\leq \\rho_{t+1}$. Thus, it suffices to show $36\\rho_t^2\\leq 3\\rho_t/2\\Longleftrightarrow 24\\rho_t\\leq 1$. Since $\\rho_t$ is decreasing, we require $24\\rho_0\\leq 1$. Note that\n\\begin{equation*}\n\\rho_0 \\leq 8\\Upsilon\\sqrt{\\frac{\\log(d\\T/\\delta)}{\\T}} + \\nu \\stackrel{\\eqref{T2}}{\\leq} \\epsilon + \\nu.\n\\end{equation*}\nThus, $24\\rho_0\\leq 1$ is guaranteed by \\eqref{cond:epsnu}. Combining \\eqref{pequ:B6} and \\eqref{pequ:B7} completes the proof.\n","sha256_utf8":"020282ee5036355aa3ba4a27128d9dac89049670e568f56be4b2a835d0ebb317"}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","title":"主定理局部参数限制","source_units":[{"start":60727,"end":60858,"start_line":558,"end_line":560,"raw_tex":"\\begin{equation}\\label{cond:epsnu}\n\\epsilon \\vee \\nu \\leq \\frac{1}{3}\\frac{0.5-\\beta}{1.5-\\beta}\\wedge \\frac{1}{48}.\n\\end{equation}","sha256_utf8":"7fc88094b8ebe6ecc20349e186dbddeda2c1ee2a5071573296c896256b338856"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"辅助陈述中的邻域与时间定义","source_units":[{"start":55438,"end":55578,"start_line":453,"end_line":455,"raw_tex":"\\begin{equation}\\label{T1}\n\\T_1 =4\\rbr{1\\vee (8\\Upsilon/\\epsilon)}^2 \\log\\rbr{d/\\delta\\cdot\\cbr{1\\vee(8\\Upsilon/\\epsilon)}}.\n\\end{equation}\n","sha256_utf8":"955afbfc586dadbd09df632b8cf5fb05aa8e1e1b56e8d01b285756563e01e8e6"},{"start":57996,"end":58153,"start_line":500,"end_line":502,"raw_tex":"\\begin{equation}\\label{Nnu}\n\\N_\\nu = \\{\\xb: \\|\\xb - \\ttxb\\|_{\\ttHb} \\leq \\nu \\cdot \\lambda_{\\min}^{3/2}/L\\}\\quad \\text{ for } \\nu \\in (0, 1].\n\\end{equation}\n","sha256_utf8":"d66b9c7ced34d915859cac93ec00d7f8f9d1446ef63515340fae77d9a7cbc7a6"},{"start":58307,"end":58486,"start_line":505,"end_line":507,"raw_tex":"\\begin{equation}\\label{T2}\n\\T_2 =  \\frac {\\kappa^2(1+\\epsilon)}{4\\rho\\beta(1-\\beta)(1-\\epsilon)}\\log\\rbr{\\frac{3L^2(f(\\xb_0) - f(\\ttxb))}{\\nu^2\\lambda_{\\min}^3}}.\n\\end{equation}\t\n","sha256_utf8":"453e10e10bdcf8819efb5976c36de0678c159d2f73e89b8d5f273ef2945741a2"}]},{"id":"S014","semantic_type":"theorem","kind":"theorem","title":"次指数矩阵鞅集中","source_units":[{"start":52237,"end":52706,"start_line":413,"end_line":419,"raw_tex":"\\begin{theorem}[Concentration of sub-exponential martingale difference]\\label{thm:2}\nUnder Assumption \\ref{ass:3}, for any integer $t\\geq 0$ and scalar $\\eta\\geq 0$, $\\barEb_t$ in \\eqref{equ:simple} satisfies\n\\begin{equation}\\label{equ:subE}\n\\mP\\rbr{\\nbr{\\barEb_t} \\geq \\eta} \\leq 2d\\cdot\\exp\\rbr{-\\frac{\\eta^2/2}{\\Upsilon_E^2\\sum_{i=0}^{t}z_{i,t}^2 + z_t^{(\\max)}\\Upsilon_E\\eta } }\n\\end{equation}\nwhere $z_t^{(\\max)} = \\max_{i\\in \\{0,\\ldots,t\\}}z_{i,t}$.\n\\end{theorem}","sha256_utf8":"0cba1780d2247d230aad6b851c7a28793112428bf2bcf4fab05445ec320d244e"}]},{"id":"S015","semantic_type":"proof","kind":"proof","title":"矩阵鞅集中完整证明","source_units":[{"start":52709,"end":52763,"start_line":422,"end_line":424,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{pf:thm:2}.\n\\end{proof}","sha256_utf8":"329fcdbbd8721cd74cc501dbb9776bb3374d9544d635a204a029879e5f48b186"},{"start":108697,"end":110630,"start_line":1128,"end_line":1154,"raw_tex":"\\subsection{Proof of Theorem \\ref{thm:2}}\\label{pf:thm:2}\n\nFor any $t\\geq 0$ and scalar $\\theta>0$, we have from Assumption \\ref{ass:3} that\n\\begin{align*}\n\\mE_i\\sbr{\\exp\\rbr{\\theta z_{i,t}\\Eb_i}}  & = \\Ib + \\theta z_{i,t}\\mE_i[\\Eb_i] + \\sum_{j=2}^{\\infty} \\frac{(\\theta z_{i,t})^j\\mE_i[\\Eb_i^j]}{j!}\n\\preceq \\Ib + \\sum_{j=2}^{\\infty}(\\theta z_{i,t}\\Upsilon_E)^{j-2}\\cdot \\frac{(\\theta z_{i,t}\\Upsilon_E)^2}{2}\\cdot\\Ib \\\\\n& = \\Ib + \\frac{(\\theta z_{i,t}\\Upsilon_E)^2}{2(1 - \\theta z_{i,t} \\Upsilon_E)}\\cdot \\Ib \\preceq\\exp\\rbr{ \\frac{(\\theta z_{i,t}\\Upsilon_E)^2}{2(1 - \\theta z_{i,t}\\Upsilon_E)} \\cdot \\Ib}, \\quad \\forall i = 0,1,\\ldots,t,\n\\end{align*}\nwhere the second equality holds for $\\cbr{\\theta: \\theta z_{i,t}\\Upsilon_E < 1}$. Therefore, we let $\\Theta_t \\coloneqq\\{\\theta: \\theta < 1/(z_t^{(\\max)}\\Upsilon_E )\\}$, and have for all $i = 0,1,\\ldots, t$,\n\\begin{equation*}\n\\mE_i\\sbr{\\exp\\rbr{\\theta z_{i,t}\\Eb_i}}  \\preceq \\exp\\rbr{\\frac{(\\theta z_{i,t}\\Upsilon_E)^2}{2(1 - \\theta z_t^{(\\max)} \\Upsilon_E) } \\cdot \\Ib}, \\quad \\forall\\theta\\in \\Theta_t.\n\\end{equation*}\nThe above inequality suggests that the condition \\eqref{equ:exp:cond} in Lemma \\ref{lem:2} holds with\n\\begin{equation*}\ng_t(\\theta) = \\frac{\\theta^2}{2(1 - \\theta z_t^{(\\max)} \\Upsilon_E) }\\quad \\text{ and }\\quad \\Ub_i = z_{i,t}^2\\Upsilon_E^2\\cdot \\Ib.\n\\end{equation*}\nLet $\\sigma^2 = \\|\\sum_{i=0}^{t}\\Ub_i\\| = \\Upsilon_E^2\\sum_{i=0}^{t}z_{i,t}^2$ in Lemma \\ref{lem:2}. Then we have for any $\\eta\\geq 0$,\n\\begin{equation*}\n\\mP\\rbr{\\nbr{\\barEb_t} \\geq \\eta} \\leq 2d\\cdot \\inf_{\\theta \\in \\Theta_t}\\exp\\rbr{-\\theta\\eta + g_t(\\theta)\\sigma^2}.\n\\end{equation*}\nPlugging $\\theta = \\eta/(z_t^{(\\max)} \\Upsilon_E \\eta + \\sigma^2) \\in \\Theta_t$ into the above right hand side, we obtain\n\\begin{equation*}\n\\mP\\rbr{\\|\\barEb_t\\| \\geq \\eta} \\leq 2d\\cdot\\exp\\rbr{-\\frac{\\eta^2/2}{ \\sigma^2 + z_t^{(\\max)} \\Upsilon_E \\eta}  }.\n\\end{equation*}\nThis completes the proof.\n\n\n","sha256_utf8":"e82bc051991f7d375caaeb0c506d370f7e8b19d24775eac8d0bfd607df9de258"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"集中证明附录定位","source_units":[{"start":52709,"end":52763,"start_line":422,"end_line":424,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{pf:thm:2}.\n\\end{proof}","sha256_utf8":"329fcdbbd8721cd74cc501dbb9776bb3374d9544d635a204a029879e5f48b186"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"矩条件推出指数矩控制","source_units":[{"start":108738,"end":109773,"start_line":1128,"end_line":1139,"raw_tex":"\\label{pf:thm:2}\n\nFor any $t\\geq 0$ and scalar $\\theta>0$, we have from Assumption \\ref{ass:3} that\n\\begin{align*}\n\\mE_i\\sbr{\\exp\\rbr{\\theta z_{i,t}\\Eb_i}}  & = \\Ib + \\theta z_{i,t}\\mE_i[\\Eb_i] + \\sum_{j=2}^{\\infty} \\frac{(\\theta z_{i,t})^j\\mE_i[\\Eb_i^j]}{j!}\n\\preceq \\Ib + \\sum_{j=2}^{\\infty}(\\theta z_{i,t}\\Upsilon_E)^{j-2}\\cdot \\frac{(\\theta z_{i,t}\\Upsilon_E)^2}{2}\\cdot\\Ib \\\\\n& = \\Ib + \\frac{(\\theta z_{i,t}\\Upsilon_E)^2}{2(1 - \\theta z_{i,t} \\Upsilon_E)}\\cdot \\Ib \\preceq\\exp\\rbr{ \\frac{(\\theta z_{i,t}\\Upsilon_E)^2}{2(1 - \\theta z_{i,t}\\Upsilon_E)} \\cdot \\Ib}, \\quad \\forall i = 0,1,\\ldots,t,\n\\end{align*}\nwhere the second equality holds for $\\cbr{\\theta: \\theta z_{i,t}\\Upsilon_E < 1}$. Therefore, we let $\\Theta_t \\coloneqq\\{\\theta: \\theta < 1/(z_t^{(\\max)}\\Upsilon_E )\\}$, and have for all $i = 0,1,\\ldots, t$,\n\\begin{equation*}\n\\mE_i\\sbr{\\exp\\rbr{\\theta z_{i,t}\\Eb_i}}  \\preceq \\exp\\rbr{\\frac{(\\theta z_{i,t}\\Upsilon_E)^2}{2(1 - \\theta z_t^{(\\max)} \\Upsilon_E) } \\cdot \\Ib}, \\quad \\forall\\theta\\in \\Theta_t.\n\\end{equation*}\n","sha256_utf8":"531cf73b5df8201fb0f745a4657c787f0dbc81cd1e0f2ae5769cf3bdc13123a6"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"代入矩阵鞅指数不等式","source_units":[{"start":109773,"end":110330,"start_line":1140,"end_line":1147,"raw_tex":"The above inequality suggests that the condition \\eqref{equ:exp:cond} in Lemma \\ref{lem:2} holds with\n\\begin{equation*}\ng_t(\\theta) = \\frac{\\theta^2}{2(1 - \\theta z_t^{(\\max)} \\Upsilon_E) }\\quad \\text{ and }\\quad \\Ub_i = z_{i,t}^2\\Upsilon_E^2\\cdot \\Ib.\n\\end{equation*}\nLet $\\sigma^2 = \\|\\sum_{i=0}^{t}\\Ub_i\\| = \\Upsilon_E^2\\sum_{i=0}^{t}z_{i,t}^2$ in Lemma \\ref{lem:2}. Then we have for any $\\eta\\geq 0$,\n\\begin{equation*}\n\\mP\\rbr{\\nbr{\\barEb_t} \\geq \\eta} \\leq 2d\\cdot \\inf_{\\theta \\in \\Theta_t}\\exp\\rbr{-\\theta\\eta + g_t(\\theta)\\sigma^2}.\n\\end{equation*}\n","sha256_utf8":"d5bf6e206c2869b56b643797c2815b8b88062bfa0ee8fff5b1bca95f975170f8"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"选择指数参数得到尾界","source_units":[{"start":110330,"end":110628,"start_line":1148,"end_line":1152,"raw_tex":"Plugging $\\theta = \\eta/(z_t^{(\\max)} \\Upsilon_E \\eta + \\sigma^2) \\in \\Theta_t$ into the above right hand side, we obtain\n\\begin{equation*}\n\\mP\\rbr{\\|\\barEb_t\\| \\geq \\eta} \\leq 2d\\cdot\\exp\\rbr{-\\frac{\\eta^2/2}{ \\sigma^2 + z_t^{(\\max)} \\Upsilon_E \\eta}  }.\n\\end{equation*}\nThis completes the proof.\n","sha256_utf8":"c55620066592c675599c518bb335469c868275fe737c79bc894bf268719a1ad3"}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","title":"矩阵弗里德曼指数界","source_units":[{"start":51004,"end":52022,"start_line":398,"end_line":407,"raw_tex":"\\begin{lemma}[Adapted from Theorem 2.3 in \\cite{Tropp2011Freedmans}]\\label{lem:2}\nLet $t\\geq 0$ be a fixed integer. Consider a $d$-dimensional martingale difference $\\{\\Eb_i\\}_{i=0}^{t}$ (i.e., $\\mE_i[\\Eb_i] = \\0$). Suppose there exists a function $g_t:\\Theta_t\\rightarrow [0, \\infty]$ with $\\Theta_t\\subseteq (0, \\infty)$, and~a~sequence of matrices $\\{\\Ub_i\\}_{i=0}^{t}$, such that for any $i = 0,1,\\ldots,t$,\\footnote{The matrix exponential is defined by power series expansion: $\\exp(\\Ab) = \\Ib + \\sum_{i=1}^{\\infty} \\Ab^i/ i!$.}\n\\begin{equation}\\label{equ:exp:cond}\n\\mE_i\\sbr{\\exp\\rbr{\\theta\\Eb_i}}\\preceq \\exp\\rbr{g_t(\\theta)\\Ub_i} \\quad \\text{ almost surely for each } \\theta\\in \\Theta_t.\n\\end{equation}\nThen, we have for any scalars $\\eta \\geq 0$ and $\\sigma^2>0$,\n\\begin{equation*}\n\\mP\\rbr{\\Big\\|\\sum_{i=0}^{t}\\Eb_i\\Big\\| \\geq \\eta\\; \\text{ and } \\; \\Big\\|\\sum_{i=0}^{t}\\Ub_i\\Big\\|\\leq \\sigma^2} \\leq 2d\\cdot \\inf_{\\theta\\in \\Theta_t}\\exp\\rbr{-\\theta \\eta + g_t(\\theta)\\sigma^2}.\n\\end{equation*}\t\t\n\\end{lemma}","sha256_utf8":"dced947d77c1089a664c7c0cb26a1d525e0d57d12daa2b865d1db89358cd592c"}]},{"id":"S021","semantic_type":"external_fact","kind":"external_fact","title":"矩阵鞅指数界外部依据","source_units":[],"declared_external":true,"declaration_text":["Tropp2011Freedmans, Theorem 2.3, supplies the matrix martingale exponential tail principle used in B0033. The stated version allows optimization over a positive parameter subset on which the conditional matrix exponential bound holds, and bounds both spectral tails with prefactor 2d."]},{"id":"S022","semantic_type":"proof","kind":"proof","title":"均匀噪声全程界推导","source_units":[{"start":52766,"end":53355,"start_line":427,"end_line":430,"raw_tex":"The martingale concentration in Theorem \\ref{thm:2} matches the matrix Bernstein results for independent noises $\\{\\Eb_i\\}_{i=0}^t$ \\citep[cf. Theorems 6.1, 6.2 in][]{Tropp2011User}. For any~$\\delta\\in(0, 1)$, if we let the right hand side of \\eqref{equ:subE} be $\\delta/(t+1)^2$, then we obtain that, with probability at least $1 - \\delta/(t+1)^2$,  \n\\begin{equation}\\label{equ:sub:concen}\n\\small \\nbr{\\barEb_t} \\leq 8\\Upsilon_E\\sqrt{\\log\\rbr{\\frac{d(t+1)}{\\delta}}}\\rbr{\\sqrt{\\sum_{i=0}^{t}z_{i,t}^2} \\; \\vee \\; \\sqrt{\\log\\rbr{\\frac{d(t+1)}{\\delta}}} \\cdot z_t^{(\\max)}}.\n\\end{equation}\n","sha256_utf8":"6baa298236c22cc23cbcd2924f3b511c4311b4479f269d0e56e8370893422607"},{"start":53459,"end":54159,"start_line":434,"end_line":441,"raw_tex":"To achieve the fastest concentration rate, we minimize the right~hand side of \\eqref{equ:sub:concen} under the~restriction $\\sum_{i=0}^t z_{i,t} = 1$. Note that the minimum of both $\\sum_{i=0}^t z_{i,t}^2$ and $z_t^{(\\max)}$ is attained with equal weights, that is $z_{i,t} = 1/(t+1)$, for $ i = 0,1,\\ldots, t$. Thus, the fastest concentration rate is attained~with~equal~weights. Furthermore, a union bound over $t$ leads to\n\\begin{equation}\\label{equ:equal:con}\n\\mP\\rbr{\\forall t:\\ \\nbr{\\barEb_t} \\leq\n8\\Upsilon_E\\bigg(\\sqrt{\\frac{\\log(d(t+1)/\\delta)}{t+1}} \\vee\n  \\frac{\\log(d(t+1)/\\delta)}{t+1}\\bigg)} \n\\geq 1 - \\sum_{t=0}^{\\infty}\n\\frac{\\delta}{(t+1)^2}= 1-\\frac{\\pi^2\\delta}{6}.\n\\end{equation}\n","sha256_utf8":"a0af15488ef9030cd8bb15ff2583d47dad72495638fc39d73f7420405393343b"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"尾界转换为置信界","source_units":[{"start":52766,"end":53355,"start_line":427,"end_line":430,"raw_tex":"The martingale concentration in Theorem \\ref{thm:2} matches the matrix Bernstein results for independent noises $\\{\\Eb_i\\}_{i=0}^t$ \\citep[cf. Theorems 6.1, 6.2 in][]{Tropp2011User}. For any~$\\delta\\in(0, 1)$, if we let the right hand side of \\eqref{equ:subE} be $\\delta/(t+1)^2$, then we obtain that, with probability at least $1 - \\delta/(t+1)^2$,  \n\\begin{equation}\\label{equ:sub:concen}\n\\small \\nbr{\\barEb_t} \\leq 8\\Upsilon_E\\sqrt{\\log\\rbr{\\frac{d(t+1)}{\\delta}}}\\rbr{\\sqrt{\\sum_{i=0}^{t}z_{i,t}^2} \\; \\vee \\; \\sqrt{\\log\\rbr{\\frac{d(t+1)}{\\delta}}} \\cdot z_t^{(\\max)}}.\n\\end{equation}\n","sha256_utf8":"6baa298236c22cc23cbcd2924f3b511c4311b4479f269d0e56e8370893422607"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"等权最小化与联合界准备","source_units":[{"start":53459,"end":53885,"start_line":434,"end_line":434,"raw_tex":"To achieve the fastest concentration rate, we minimize the right~hand side of \\eqref{equ:sub:concen} under the~restriction $\\sum_{i=0}^t z_{i,t} = 1$. Note that the minimum of both $\\sum_{i=0}^t z_{i,t}^2$ and $z_t^{(\\max)}$ is attained with equal weights, that is $z_{i,t} = 1/(t+1)$, for $ i = 0,1,\\ldots, t$. Thus, the fastest concentration rate is attained~with~equal~weights. Furthermore, a union bound over $t$ leads to\n","sha256_utf8":"2b37c4649c9a3998556b6b0e5631919d7bf6553b25330b63c7b806e3442abd1a"}]},{"id":"S025","semantic_type":"lemma","kind":"lemma","title":"平均海森最终一致正定","source_units":[{"start":55243,"end":55933,"start_line":451,"end_line":461,"raw_tex":"\\begin{lemma}\\label{lem:3}\nConsider Algorithm \\ref{alg:1} with $w_t = t+1$, $\\forall t\\geq 0$. Under Assumption~\\ref{ass:3}, we let $\\delta, \\epsilon\\in(0, 1)$ with $d/\\delta\\geq e$. We also let\n\\begin{equation}\\label{T1}\n\\T_1 =4\\rbr{1\\vee (8\\Upsilon/\\epsilon)}^2 \\log\\rbr{d/\\delta\\cdot\\cbr{1\\vee(8\\Upsilon/\\epsilon)}}.\n\\end{equation}\nThen, with probability $1-\\delta\\pi^2/6$, the event\n\\begin{equation}\\label{event:EE}\n\\EE = \\bigcap_{t=\\T_1}^{\\infty}\\cbr{\\|\\barEb_t\\| \\leq 8\\Upsilon_E\\sqrt{\\frac{\\log(d(t+1)/\\delta)}{t+1}}}\n\\end{equation}\noccurs, which implies $(1-\\epsilon)\\lambda_{\\min}\\cdot\\Ib\\preceq \\tHb_t \\preceq (1+\\epsilon)\\lambda_{\\max}\\cdot\\Ib$, $\\forall t\\geq \\T_1$.\n\\end{lemma}","sha256_utf8":"baf129e6ab80f2e39ceefa32a50c9e750df4928d363cad5ebb5be6d4aa23560c"}]},{"id":"S026","semantic_type":"proof","kind":"proof","title":"一致正定性完整证明","source_units":[{"start":55935,"end":55990,"start_line":463,"end_line":465,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{pf:lem:3}. \n\\end{proof}","sha256_utf8":"96f3531cedcf83940ff361731518256c1371d3bb6ef9a50d86727518ac3c091d"},{"start":110630,"end":112067,"start_line":1155,"end_line":1173,"raw_tex":"\\subsection{Proof of Lemma \\ref{lem:3}}\\label{pf:lem:3}\n\nLet us define the event\n\\begin{equation*}\n\\bar{\\EE} = \\bigcap_{t=0}^{\\infty}\\cbr{\\nbr{\\barEb_t} \\leq 8\\Upsilon_E\\rbr{\\sqrt{\\frac{\\log(d(t+1)/\\delta)}{t+1}}\\vee \\frac{\\log(d(t+1)/\\delta)}{t+1} } }.\n\\end{equation*}\nBy Theorem \\ref{thm:2} and \\eqref{equ:equal:con}, we know $\\mP(\\bar{\\EE}) \\geq 1-\\delta\\pi^2/6$. We also note that\n\\begin{align}\\label{nequ:1}\n\\|\\barEb_t\\|\\leq \\epsilon\\lambda_{\\min} & \\Longleftarrow 8\\Upsilon_E\\rbr{\\sqrt{\\frac{\\log(d(t+1)/\\delta)}{t+1}}\\vee \\frac{\\log(d(t+1)/\\delta)}{t+1} } \\leq \\epsilon\\lambda_{\\min} \\nonumber\\\\\n& \\Longleftarrow \\frac{\\log(d(t+1)/\\delta)}{t+1} \\leq \\frac{\\epsilon^2}{64\\Upsilon^2} \\wedge1 = \\rbr{\\frac{\\epsilon}{8\\Upsilon}\\wedge 1}^2 \\nonumber\\\\\n&\\Longleftarrow t \\geq 4\\rbr{\\frac{8\\Upsilon}{\\epsilon}\\vee 1}^2 \\log\\cbr{\\frac{d}{\\delta}\\cdot\\rbr{\\frac{8\\Upsilon}{\\epsilon}\\vee 1}}=\\T_1. \\quad (\\text{Lemma \\ref{lem:pre}})\n\\end{align}\nTherefore, with probability at least $1-\\delta\\pi^2/6$, we have for any $t\\geq \\T_1$ that $\\|\\barEb_t\\|\\leq \\epsilon\\lambda_{\\min}$. This shows that the event $\\EE$ defined in \\eqref{event:EE} happens, and\n\\begin{equation*}\n(1-\\epsilon)\\lambda_{\\min}\\cdot\\Ib\\preceq\\tHb_t \\stackrel{\\eqref{equ:simple}}{=} \\frac{1}{t+1}\\sum_{i=0}^{t}\\Hb_i + \\barEb_t \\preceq (\\lambda_{\\max} + \\epsilon\\lambda_{\\min})\\cdot\\Ib\\preceq (1+\\epsilon)\\lambda_{\\max} \\cdot\\Ib.\n\\end{equation*}\nThis completes the proof.\n\n\n","sha256_utf8":"3a87b04cb9b021a03028d6df93571f03b9742d30c97776d1d97b4cb50c858b70"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"正定性证明附录定位","source_units":[{"start":55935,"end":55990,"start_line":463,"end_line":465,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{pf:lem:3}. \n\\end{proof}","sha256_utf8":"96f3531cedcf83940ff361731518256c1371d3bb6ef9a50d86727518ac3c091d"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"构造全程噪声控制事件","source_units":[{"start":110669,"end":111015,"start_line":1155,"end_line":1161,"raw_tex":"\\label{pf:lem:3}\n\nLet us define the event\n\\begin{equation*}\n\\bar{\\EE} = \\bigcap_{t=0}^{\\infty}\\cbr{\\nbr{\\barEb_t} \\leq 8\\Upsilon_E\\rbr{\\sqrt{\\frac{\\log(d(t+1)/\\delta)}{t+1}}\\vee \\frac{\\log(d(t+1)/\\delta)}{t+1} } }.\n\\end{equation*}\nBy Theorem \\ref{thm:2} and \\eqref{equ:equal:con}, we know $\\mP(\\bar{\\EE}) \\geq 1-\\delta\\pi^2/6$. We also note that\n","sha256_utf8":"de206cd7ecc9cee8c6c88a01d6045de6384d5f92661a6119e6012afe2a24f039"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"确定噪声充分集中时间","source_units":[{"start":111015,"end":111572,"start_line":1162,"end_line":1166,"raw_tex":"\\begin{align}\\label{nequ:1}\n\\|\\barEb_t\\|\\leq \\epsilon\\lambda_{\\min} & \\Longleftarrow 8\\Upsilon_E\\rbr{\\sqrt{\\frac{\\log(d(t+1)/\\delta)}{t+1}}\\vee \\frac{\\log(d(t+1)/\\delta)}{t+1} } \\leq \\epsilon\\lambda_{\\min} \\nonumber\\\\\n& \\Longleftarrow \\frac{\\log(d(t+1)/\\delta)}{t+1} \\leq \\frac{\\epsilon^2}{64\\Upsilon^2} \\wedge1 = \\rbr{\\frac{\\epsilon}{8\\Upsilon}\\wedge 1}^2 \\nonumber\\\\\n&\\Longleftarrow t \\geq 4\\rbr{\\frac{8\\Upsilon}{\\epsilon}\\vee 1}^2 \\log\\cbr{\\frac{d}{\\delta}\\cdot\\rbr{\\frac{8\\Upsilon}{\\epsilon}\\vee 1}}=\\T_1. \\quad (\\text{Lemma \\ref{lem:pre}})\n\\end{align}\n","sha256_utf8":"3c156bb026045ef7d53361d0899e9e47d48d136f40bb839680b4f21b40c49c89"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"噪声界推出海森谱界","source_units":[{"start":111572,"end":112065,"start_line":1167,"end_line":1171,"raw_tex":"Therefore, with probability at least $1-\\delta\\pi^2/6$, we have for any $t\\geq \\T_1$ that $\\|\\barEb_t\\|\\leq \\epsilon\\lambda_{\\min}$. This shows that the event $\\EE$ defined in \\eqref{event:EE} happens, and\n\\begin{equation*}\n(1-\\epsilon)\\lambda_{\\min}\\cdot\\Ib\\preceq\\tHb_t \\stackrel{\\eqref{equ:simple}}{=} \\frac{1}{t+1}\\sum_{i=0}^{t}\\Hb_i + \\barEb_t \\preceq (\\lambda_{\\max} + \\epsilon\\lambda_{\\min})\\cdot\\Ib\\preceq (1+\\epsilon)\\lambda_{\\max} \\cdot\\Ib.\n\\end{equation*}\nThis completes the proof.\n","sha256_utf8":"ec967990ab60ad5820d4db42df70f96ef219d6fb9ba56b7d04cd0e452c368b25"}]},{"id":"S031","semantic_type":"lemma","kind":"lemma","title":"对数比值阈值估计","source_units":[{"start":104325,"end":104676,"start_line":1050,"end_line":1056,"raw_tex":"\\begin{lemma}\\label{lem:pre}\nLet $d\\geq 1$ and $\\delta\\in(0,1)$. If $d/\\delta\\geq e$, then for any $0<a\\leq 1$,\n\\begin{equation*}\nt\\geq \\frac {2\\log(d/(\\delta a))}{a}\\quad\\Longrightarrow\\quad\\frac{\\log(dt/\\delta)}{t} \\leq a,\n\\end{equation*}\nand moreover, the right hand side inequality fails if $t= \\log(d/(\\delta a))/a$ and $d/\\delta >e$.\n\\end{lemma}","sha256_utf8":"3e4f42897ab36915471e32573b09c57d6dfa84297fc34ffc845ab523d4c0bb17"}]},{"id":"S032","semantic_type":"proof","kind":"proof","title":"对数阈值完整证明","source_units":[{"start":104678,"end":105682,"start_line":1058,"end_line":1072,"raw_tex":"\\begin{proof}\nLet $t = 2\\log(d/(\\delta a))/a$. We will show for this $t$ that $t\\geq \\log(dt/\\delta)/a$. We have\n\\begin{align*}\nt - \\frac{\\log(dt/\\delta)}{a}\n& =  \\frac{2\\log(d/(\\delta a))}{a} - \\frac{\\log(d/(\\delta a)) + \\log(2\\log(d/(\\delta a)))}{a}\n\\\\\n&=\\frac{\\log(d/(\\delta a)) - \\log(2\\log(d/(\\delta a)))}{a}\\\\\n&= \\frac1a\\log\\Big(\\frac{x}{2\\log(x)}\\Big)\\geq 0,\\qquad\\text{for}\\quad x=\\frac{d}{\\delta a},\n\\end{align*}\nwhere the last step is due to the fact that $x\\geq 2\\log(x)$ for all $x>0$. Now, we consider the function $f(t) = \\log(dt/\\delta)/t$. For any $t\\geq1$, its derivative is bounded as\n\\begin{align*}\n\\frac{\\partial f(t)}{\\partial t} = \\frac{1-\\log(dt/\\delta)}{t^2}\\leq 0,\n\\end{align*}\nwhich implies that for $t\\geq2\\log(d/(\\delta a))/a \\geq 1$ we get $\\log(dt/\\delta)/t =f(t)\\leq f(2\\log(d/(\\delta a))/a)\\leq a$. Finally, if $t=\\log(d/(\\delta a))/a$ and $d/\\delta >e$, then $\\log(dt/\\delta)/a=\\log(d/(\\delta a))/a + \\log(\\log(d/(\\delta a)))/a >t$, which completes the proof.\n\\end{proof}","sha256_utf8":"b572fd52b4c77438410f973fa3d154b1c870ffe168567a2d197faab7a580bd9d"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"验证显式阈值","source_units":[{"start":104692,"end":105281,"start_line":1059,"end_line":1067,"raw_tex":"Let $t = 2\\log(d/(\\delta a))/a$. We will show for this $t$ that $t\\geq \\log(dt/\\delta)/a$. We have\n\\begin{align*}\nt - \\frac{\\log(dt/\\delta)}{a}\n& =  \\frac{2\\log(d/(\\delta a))}{a} - \\frac{\\log(d/(\\delta a)) + \\log(2\\log(d/(\\delta a)))}{a}\n\\\\\n&=\\frac{\\log(d/(\\delta a)) - \\log(2\\log(d/(\\delta a)))}{a}\\\\\n&= \\frac1a\\log\\Big(\\frac{x}{2\\log(x)}\\Big)\\geq 0,\\qquad\\text{for}\\quad x=\\frac{d}{\\delta a},\n\\end{align*}\nwhere the last step is due to the fact that $x\\geq 2\\log(x)$ for all $x>0$. Now, we consider the function $f(t) = \\log(dt/\\delta)/t$. For any $t\\geq1$, its derivative is bounded as\n","sha256_utf8":"9593730511458185555756bf63e2908d0d11b690d59a5627b61a7ddba65c12a9"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"单调延拓与反向验证","source_units":[{"start":105281,"end":105671,"start_line":1068,"end_line":1071,"raw_tex":"\\begin{align*}\n\\frac{\\partial f(t)}{\\partial t} = \\frac{1-\\log(dt/\\delta)}{t^2}\\leq 0,\n\\end{align*}\nwhich implies that for $t\\geq2\\log(d/(\\delta a))/a \\geq 1$ we get $\\log(dt/\\delta)/t =f(t)\\leq f(2\\log(d/(\\delta a))/a)\\leq a$. Finally, if $t=\\log(d/(\\delta a))/a$ and $d/\\delta >e$, then $\\log(dt/\\delta)/a=\\log(d/(\\delta a))/a + \\log(\\log(d/(\\delta a)))/a >t$, which completes the proof.\n","sha256_utf8":"3b11c43a8b820c87081bf46ae5af19c4c524aed05e6c85a46a6ddc3cd776c0f0"}]},{"id":"S035","semantic_type":"lemma","kind":"lemma","title":"全局目标值线性下降","source_units":[{"start":56644,"end":57249,"start_line":473,"end_line":483,"raw_tex":"\\begin{lemma}\\label{lem:4}\nConditioning on the event \\eqref{event:EE}, we let\n\\begin{equation*}\n\\phi = \\frac{4\\rho\\beta(1-\\beta)(1-\\epsilon)}{\\kappa^2(1+\\epsilon)}\n\\end{equation*}\nand have $f(\\xb_{t+1})-f(\\ttxb) \\leq (1-\\phi)(f(\\xb_t) - f(\\ttxb))$, $\\forall t\\geq \\T_1$, which implies $R$-linear convergence of the iterate error,\n\\begin{align*}\n\\|\\xb_t - \\ttxb\\| &\\leq \\cbr{\\frac{2}{\\lambda_{\\min}}(f(\\xb_0) - f(\\ttxb))(1-\\phi)^{t-\\T_1}}^{1/2}, \\quad t\\geq \\T_1,\\\\\n\\|\\xb_t - \\ttxb\\|_{\\ttHb} &\\leq \\cbr{2\\kappa(f(\\xb_0) - f(\\ttxb))(1-\\phi)^{t-\\T_1}}^{1/2}, \\quad\\quad\\; t\\geq \\T_1.\n\\end{align*}\n\\end{lemma}","sha256_utf8":"faee9df9084b18bf44d4cfde5737d8d3bdd1a008d83b24d68ec4ff12db34597c"}]},{"id":"S036","semantic_type":"proof","kind":"proof","title":"全局线性下降完整证明","source_units":[{"start":57251,"end":57305,"start_line":485,"end_line":487,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{pf:lem:4}.\n\\end{proof}","sha256_utf8":"99ca29b639f675b941b1e7847f81784aee590728ceca1e3363916a84952a32fc"},{"start":112067,"end":114323,"start_line":1174,"end_line":1207,"raw_tex":"\\subsection{Proof of Lemma \\ref{lem:4}}\\label{pf:lem:4}\n\nBy Lemma \\ref{lem:3}, we know that $\\pb_t = -(\\tHb_t)^{-1}\\nabla f_t$ for $t\\geq \\T_1$. We apply Taylor's expansion:\n\\begin{align*}\n\tf(\\xb_t - \\mu_t(\\tHb_t)^{-1}\\nabla f_t ) & \\leq f(\\xb_t) - \\mu_t \\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t + \\frac{\\lambda_{\\max}\\mu_t^2}{2}\\nabla f_t^\\top(\\tHb_t)^{-2}\\nabla f_t \\\\\n\t& \\leq f(\\xb_t) - \\mu_t \\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t +\\frac{\\kappa\\mu_t^2}{2(1-\\epsilon)}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t \\quad (\\text{Lemma \\ref{lem:3}})\\\\\n\t& =  f(\\xb_t) - \\mu_t\\rbr{1 - \\frac{\\kappa\\mu_t}{2(1-\\epsilon)}}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t.\n\\end{align*}\nThus, the Armijo condition is satisfied if\n\\begin{equation*}\n1 - \\frac{\\kappa\\mu_t}{2(1-\\epsilon)} \\geq \\beta \\Longleftrightarrow \\mu_t \\leq \\frac{2(1-\\beta)(1-\\epsilon)}{\\kappa}.\n\\end{equation*}\nTherefore, the backtracking line search on Line 7 leads to a stepsize satisfying\n\\begin{equation}\\label{npequ:1}\n\\mu_t\\geq \\frac{2\\rho(1-\\beta)(1-\\epsilon)}{\\kappa}.\n\\end{equation}\nMoreover, we apply Lemma \\ref{lem:3} and strong convexity of $f$, and have\n\\begin{equation}\\label{npequ:2}\n\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t \\geq \\frac{1}{(1+\\epsilon)\\lambda_{\\max}}\\|\\nabla f_t\\|^2 \\geq \\frac{2}{(1+\\epsilon)\\kappa}(f(\\xb_t) - f(\\ttxb)).\n\\end{equation}\nCombining \\eqref{npequ:1}, \\eqref{npequ:2} with the Armijo condition, we have\n\\begin{align}\\label{pequ:B1}\nf(\\xb_{t+1}) - f(\\ttxb) &\\leq f(\\xb_t) - f(\\ttxb) - \\mu_t \\beta \\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t \\nonumber\\\\\n& \\leq f(\\xb_t) - f(\\ttxb) - \\frac{2\\rho\\beta(1-\\beta)(1-\\epsilon)}{\\kappa}\\cdot \\frac{2}{(1+\\epsilon)\\kappa}(f(\\xb_t) - f(\\ttxb)) \\nonumber\\\\\n& = (1-\\phi)(f(\\xb_t) - f(\\ttxb)),\n\\end{align}\nwhich shows the first part of the statement. Furthermore, we apply \\eqref{pequ:B1} recursively, apply strong convexity, and have\n\\begin{equation*}\n\\|\\xb_t - \\ttxb\\|^2\\leq \\frac{2(f(\\xb_t)-f(\\ttxb))}{\\lambda_{\\min}} \\\\\n\\stackrel{\\eqref{pequ:B1}}{\\leq} \\frac{2(1-\\phi)^{t-\\T_1}}{\\lambda_{\\min}}(f(\\xb_{\\T_1}) - f(\\ttxb)) \\leq \\frac{2(1-\\phi)^{t-\\T_1}}{\\lambda_{\\min}}(f(\\xb_0) - f(\\ttxb)).\n\\end{equation*}\nThe last statement follows from $\\|\\xb_t-\\ttxb\\|_{\\ttHb}^2\\leq \\lambda_{\\max}\\|\\xb_t - \\ttxb\\|^2$. This completes the proof.\n\n\n","sha256_utf8":"8cf2fc3f79a9b7493171d9239ba74b0e442252016a7b13a14d5c7d0920d0d284"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"全局下降证明附录定位","source_units":[{"start":57251,"end":57305,"start_line":485,"end_line":487,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{pf:lem:4}.\n\\end{proof}","sha256_utf8":"99ca29b639f675b941b1e7847f81784aee590728ceca1e3363916a84952a32fc"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"泰勒界给出充分步长","source_units":[{"start":112106,"end":112924,"start_line":1174,"end_line":1185,"raw_tex":"\\label{pf:lem:4}\n\nBy Lemma \\ref{lem:3}, we know that $\\pb_t = -(\\tHb_t)^{-1}\\nabla f_t$ for $t\\geq \\T_1$. We apply Taylor's expansion:\n\\begin{align*}\n\tf(\\xb_t - \\mu_t(\\tHb_t)^{-1}\\nabla f_t ) & \\leq f(\\xb_t) - \\mu_t \\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t + \\frac{\\lambda_{\\max}\\mu_t^2}{2}\\nabla f_t^\\top(\\tHb_t)^{-2}\\nabla f_t \\\\\n\t& \\leq f(\\xb_t) - \\mu_t \\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t +\\frac{\\kappa\\mu_t^2}{2(1-\\epsilon)}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t \\quad (\\text{Lemma \\ref{lem:3}})\\\\\n\t& =  f(\\xb_t) - \\mu_t\\rbr{1 - \\frac{\\kappa\\mu_t}{2(1-\\epsilon)}}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t.\n\\end{align*}\nThus, the Armijo condition is satisfied if\n\\begin{equation*}\n1 - \\frac{\\kappa\\mu_t}{2(1-\\epsilon)} \\geq \\beta \\Longleftrightarrow \\mu_t \\leq \\frac{2(1-\\beta)(1-\\epsilon)}{\\kappa}.\n\\end{equation*}\n","sha256_utf8":"fd99735e72748847af9176fb034255e60bf8be393097e8d4972eb88501761802"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"步长与下降方向下界","source_units":[{"start":112924,"end":113380,"start_line":1186,"end_line":1193,"raw_tex":"Therefore, the backtracking line search on Line 7 leads to a stepsize satisfying\n\\begin{equation}\\label{npequ:1}\n\\mu_t\\geq \\frac{2\\rho(1-\\beta)(1-\\epsilon)}{\\kappa}.\n\\end{equation}\nMoreover, we apply Lemma \\ref{lem:3} and strong convexity of $f$, and have\n\\begin{equation}\\label{npequ:2}\n\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t \\geq \\frac{1}{(1+\\epsilon)\\lambda_{\\max}}\\|\\nabla f_t\\|^2 \\geq \\frac{2}{(1+\\epsilon)\\kappa}(f(\\xb_t) - f(\\ttxb)).\n\\end{equation}\n","sha256_utf8":"e9f73cd37731c0b736ada5a746e010455b674baaa3ec422110f9d97c6b5a1bc3"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"阿米霍条件推出目标收缩","source_units":[{"start":113380,"end":113792,"start_line":1194,"end_line":1199,"raw_tex":"Combining \\eqref{npequ:1}, \\eqref{npequ:2} with the Armijo condition, we have\n\\begin{align}\\label{pequ:B1}\nf(\\xb_{t+1}) - f(\\ttxb) &\\leq f(\\xb_t) - f(\\ttxb) - \\mu_t \\beta \\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t \\nonumber\\\\\n& \\leq f(\\xb_t) - f(\\ttxb) - \\frac{2\\rho\\beta(1-\\beta)(1-\\epsilon)}{\\kappa}\\cdot \\frac{2}{(1+\\epsilon)\\kappa}(f(\\xb_t) - f(\\ttxb)) \\nonumber\\\\\n& = (1-\\phi)(f(\\xb_t) - f(\\ttxb)),\n\\end{align}\n","sha256_utf8":"1a840f1712b12a5099c13dcc5f1cc42ed7bd58dc958a2c6234051c26d988cbb2"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"目标收缩转为迭代误差界","source_units":[{"start":113792,"end":114321,"start_line":1200,"end_line":1205,"raw_tex":"which shows the first part of the statement. Furthermore, we apply \\eqref{pequ:B1} recursively, apply strong convexity, and have\n\\begin{equation*}\n\\|\\xb_t - \\ttxb\\|^2\\leq \\frac{2(f(\\xb_t)-f(\\ttxb))}{\\lambda_{\\min}} \\\\\n\\stackrel{\\eqref{pequ:B1}}{\\leq} \\frac{2(1-\\phi)^{t-\\T_1}}{\\lambda_{\\min}}(f(\\xb_{\\T_1}) - f(\\ttxb)) \\leq \\frac{2(1-\\phi)^{t-\\T_1}}{\\lambda_{\\min}}(f(\\xb_0) - f(\\ttxb)).\n\\end{equation*}\nThe last statement follows from $\\|\\xb_t-\\ttxb\\|_{\\ttHb}^2\\leq \\lambda_{\\max}\\|\\xb_t - \\ttxb\\|^2$. This completes the proof.\n","sha256_utf8":"81301bf65f2bd342c59d940e7667a04f202982a458ca7ab944b1913dbb3d4ba0"}]},{"id":"S042","semantic_type":"corollary","kind":"corollary","title":"进入并保持局部邻域","source_units":[{"start":57750,"end":58501,"start_line":498,"end_line":508,"raw_tex":"\\begin{corollary}\\label{cor:1}\nConsider Algorithm \\ref{alg:1} with $w_t=t+1$, $\\forall t\\geq 0$. Under Assumptions \\ref{ass:3} and \\ref{ass:4}, we let $\\delta, \\epsilon\\in(0, 1)$ with $d/\\delta \\geq e$, and define the neighborhood $\\N_{\\nu}$ as \n\\begin{equation}\\label{Nnu}\n\\N_\\nu = \\{\\xb: \\|\\xb - \\ttxb\\|_{\\ttHb} \\leq \\nu \\cdot \\lambda_{\\min}^{3/2}/L\\}\\quad \\text{ for } \\nu \\in (0, 1].\n\\end{equation}\nThen, with probability $1 - \\delta\\pi^2/6$, \nwe have $\\xb_t\\in \\N_\\nu$, for all $t\\geq \\T$ where~$\\T = \\T_1 + \\T_2$ with $\\T_1$ defined in \\eqref{T1} and\n\\begin{equation}\\label{T2}\n\\T_2 =  \\frac {\\kappa^2(1+\\epsilon)}{4\\rho\\beta(1-\\beta)(1-\\epsilon)}\\log\\rbr{\\frac{3L^2(f(\\xb_0) - f(\\ttxb))}{\\nu^2\\lambda_{\\min}^3}}.\n\\end{equation}\t\n\\end{corollary}","sha256_utf8":"0bc668941e6a03ab081fffea713bd64a907d5e104806886dd18552e768fb1ba4"}]},{"id":"S043","semantic_type":"proof","kind":"proof","title":"邻域进入完整证明","source_units":[{"start":58504,"end":58558,"start_line":511,"end_line":513,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{pf:cor:1}.\n\\end{proof}","sha256_utf8":"2143b4062f6d8586e93eb11d969bf58e2af70e7c8649db7d5cb315838eaf4dcb"},{"start":114323,"end":115413,"start_line":1208,"end_line":1220,"raw_tex":"\\subsection{Proof of Corollary \\ref{cor:1}}\\label{pf:cor:1}\n\nWe condition on the event \\eqref{event:EE}, which happens with probability $1-\\delta\\pi^2/6$. By Lemma \\ref{lem:4}, we know that, for $t\\geq \\T_1$,\n\\begin{equation*}\nf(\\xb_t) - f(\\ttxb) \\leq (1 - \\phi)^{t-\\T_1}(f(\\xb_{\\T_1}) - f(\\ttxb)) \\leq (1 - \\phi)^{t-\\T_1}(f(\\xb_0) - f(\\ttxb)).\n\\end{equation*}\nBy Lemma \\ref{lem:pre:1}, to have $\\xb_t \\in \\N_\\nu$, it suffices to have $\\xb_t \\in \\bar{\\N}_{\\nu^2/3}$. Thus, we let\n\\begin{equation}\\label{equ:new}\n(1 - \\phi)^{t-\\T_1}(f(\\xb_0) - f(\\ttxb)) \\leq \\frac{\\nu^2\\lambda_{\\min}^3}{3L^2} \\Longleftarrow t-\\T_1 \\geq \\frac{\\log\\rbr{\\frac{3L^2(f(\\xb_0) - f(\\ttxb))}{\\nu^2\\lambda_{\\min}^3}}}{\\log\\rbr{\\frac{1}{1-\\phi}}}\\Longleftarrow t \\geq \\T_1 + \\T_2,\n\\end{equation}\nwhere $\\T_2$ is defined in \\eqref{T2} and the implication uses the fact that $\\log(1/(1-\\phi))\\geq \\phi$. Furthermore, since $f(\\xb_t)$ is always decreasing, we know after $\\T_1+T_2$ iterations $\\xb_t$ stays in $\\bar{\\N}_{\\nu^2/3}$, and hence stays in $\\N_{\\nu}$ (cf. Lemma \\ref{lem:pre:1}). This completes the proof.\n\n\n","sha256_utf8":"c5ca93825ea67b18cf6b53a873fbc04ca5d0342b487891ba7f77c0b4b8534c2c"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"邻域证明附录定位","source_units":[{"start":58504,"end":58558,"start_line":511,"end_line":513,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{pf:cor:1}.\n\\end{proof}","sha256_utf8":"2143b4062f6d8586e93eb11d969bf58e2af70e7c8649db7d5cb315838eaf4dcb"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"条件化后的目标误差界","source_units":[{"start":114366,"end":114684,"start_line":1208,"end_line":1213,"raw_tex":"\\label{pf:cor:1}\n\nWe condition on the event \\eqref{event:EE}, which happens with probability $1-\\delta\\pi^2/6$. By Lemma \\ref{lem:4}, we know that, for $t\\geq \\T_1$,\n\\begin{equation*}\nf(\\xb_t) - f(\\ttxb) \\leq (1 - \\phi)^{t-\\T_1}(f(\\xb_{\\T_1}) - f(\\ttxb)) \\leq (1 - \\phi)^{t-\\T_1}(f(\\xb_0) - f(\\ttxb)).\n\\end{equation*}\n","sha256_utf8":"1601aea5ff60488eb95eab989507eefadbd19ca4224fb362b56794d1ba944ed9"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"求解进入时间并保持邻域","source_units":[{"start":114684,"end":115411,"start_line":1214,"end_line":1218,"raw_tex":"By Lemma \\ref{lem:pre:1}, to have $\\xb_t \\in \\N_\\nu$, it suffices to have $\\xb_t \\in \\bar{\\N}_{\\nu^2/3}$. Thus, we let\n\\begin{equation}\\label{equ:new}\n(1 - \\phi)^{t-\\T_1}(f(\\xb_0) - f(\\ttxb)) \\leq \\frac{\\nu^2\\lambda_{\\min}^3}{3L^2} \\Longleftarrow t-\\T_1 \\geq \\frac{\\log\\rbr{\\frac{3L^2(f(\\xb_0) - f(\\ttxb))}{\\nu^2\\lambda_{\\min}^3}}}{\\log\\rbr{\\frac{1}{1-\\phi}}}\\Longleftarrow t \\geq \\T_1 + \\T_2,\n\\end{equation}\nwhere $\\T_2$ is defined in \\eqref{T2} and the implication uses the fact that $\\log(1/(1-\\phi))\\geq \\phi$. Furthermore, since $f(\\xb_t)$ is always decreasing, we know after $\\T_1+T_2$ iterations $\\xb_t$ stays in $\\bar{\\N}_{\\nu^2/3}$, and hence stays in $\\N_{\\nu}$ (cf. Lemma \\ref{lem:pre:1}). This completes the proof.\n","sha256_utf8":"81e7e15c4b15afcb493d39c140f66c31106654cac93740b352a53935a9a9a4a7"}]},{"id":"S047","semantic_type":"lemma","kind":"lemma","title":"函数邻域与范数邻域比较","source_units":[{"start":105685,"end":106093,"start_line":1075,"end_line":1081,"raw_tex":"\\begin{lemma}\\label{lem:pre:1}\nDefine the neighborhood $\\bar{\\N}_{\\nu} = \\{\\xb: f(\\xb) \\leq f(\\ttxb) + \\nu\\lambda_{\\min}^3/L^2\\}$ for $\\nu>0$. Suppose Assumption \\ref{ass:4} holds, then the following relation holds\n\\begin{equation*}\n\\bar{\\N}_{\\nu^2/3} \\subseteq \\N_{\\nu} \\subseteq \\bar{\\N}_{\\nu^2}, \\quad \\text{for any } \\nu \\in (0, 1]\n\\end{equation*}\nwhere $\\N_{\\nu}$ is defined in \\eqref{Nnu}.\t\n\\end{lemma}","sha256_utf8":"331b4ec44168b49c8293b2934261f9576ccfc8e423af42f02d32a57aec16a227"}]},{"id":"S048","semantic_type":"proof","kind":"proof","title":"邻域比较完整证明","source_units":[{"start":106096,"end":108064,"start_line":1084,"end_line":1106,"raw_tex":"\\begin{proof}\nBy Assumption \\ref{ass:4}, we have\n\\begin{equation}\\label{pequ:A3}\n\\abr{f(\\xb) - f(\\ttxb) - \\frac{1}{2}\\nbr{\\xb - \\ttxb}_{\\ttHb}^2} \\leq \\frac{L}{6}\\nbr{\\xb - \\ttxb}^3.\n\\end{equation}\nSuppose $\\xb \\in \\bar{\\N}_{\\nu^2/3}$ for $\\nu\\in(0, 1]$, we have\n\\begin{equation}\\label{pequ:A4}\nf(\\ttxb) + \\frac{\\lambda_{\\min}}{2}\\|\\xb - \\ttxb\\|^2\\leq f(\\xb) \\leq f(\\ttxb) + \\frac{\\nu^2\\lambda_{\\min}^3}{3L^2} \\Longrightarrow \\|\\xb - \\ttxb\\| \\leq \\frac{\\sqrt{2}\\nu\\lambda_{\\min}}{\\sqrt{3}L}.\n\\end{equation}\nCombining \\eqref{pequ:A3} and \\eqref{pequ:A4}, we know that $\\xb \\in \\bar{\\N}_{\\nu^2/3}$ for $\\nu\\in(0, 1]$ leads to\n\\begin{align*}\nf(\\ttxb) + \\frac{\\nu^2\\lambda_{\\min}^3}{3L^2} \\geq f(\\xb) & \\stackrel{\\mathclap{\\eqref{pequ:A3}}}{\\geq} f(\\ttxb) + \\frac{1}{2}\\|\\xb - \\ttxb\\|_{\\ttHb}^2 - \\frac{L}{6}\\|\\xb-\\ttxb\\|^3\t\\\\\n& \\geq f(\\ttxb) + \\frac{1}{2}\\nbr{\\xb - \\ttxb}_{\\ttHb}^2 - \\frac{L}{6}\\nbr{\\xb - \\ttxb}\\cdot \\frac{\\nbr{\\xb - \\ttxb}_{\\ttHb}^2}{\\lambda_{\\min}}\\\\\n& = f(\\ttxb) + \\rbr{\\frac{1}{2}-\\frac{L\\|\\xb - \\ttxb\\|}{6\\lambda_{\\min}}}\\|\\xb - \\ttxb\\|_{\\ttHb}^2\\\\\n& \\stackrel{\\mathclap{\\eqref{pequ:A4}}}{\\geq} f(\\ttxb) + \\rbr{\\frac{1}{2} - \\frac{\\sqrt{2}\\nu}{6\\sqrt{3}}}\\|\\xb - \\ttxb\\|_{\\ttHb}^2 \\geq f(\\ttxb) + \\frac{\\|\\xb - \\ttxb\\|_{\\ttHb}^2}{3},\n\\end{align*}\nwhich further implies $\\|\\xb - \\ttxb\\|_{\\ttHb} \\leq \\nu\\lambda_{\\min}^{3/2}/L$ and $\\xb\\in \\N_{\\nu}$. On the other hand, suppose $\\xb \\in \\N_{\\nu}$, we have\n\\begin{align*}\nf(\\xb)  &\\stackrel{\\mathclap{\\eqref{pequ:A3}}}{\\leq} f(\\ttxb) + \\frac{1}{2}\\nbr{\\xb - \\ttxb}_{\\ttHb}^2 + \\frac{L}{6}\\nbr{\\xb - \\ttxb}^3 \\leq f(\\ttxb) + \\rbr{ \\frac{1}{2} + \\frac{L\\nbr{\\xb - \\ttxb}_{\\ttHb}}{6\\lambda_{\\min}^{3/2}}} \\nbr{\\xb - \\ttxb}_{\\ttHb}^2\\\\\n& \\leq f(\\ttxb) + \\rbr{ \\frac{1}{2} + \\frac{\\nu}{6}} \\nbr{\\xb - \\ttxb}_{\\ttHb}^2\\leq f(\\ttxb)  + \\nbr{\\xb - \\ttxb}_{\\ttHb}^2 \\leq f(\\ttxb) + \\frac{\\nu^2\\lambda_{\\min}^3}{L^2}.\n\\end{align*}\nThus, we have $\\xb \\in \\bar{\\N}_{\\nu^2}$. This completes the proof.\n\\end{proof}","sha256_utf8":"d80cb7a4969871e58db12b44932df94bf043bb9e4fa1a9511d449405d8ce6bbe"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"三阶余项与距离控制","source_units":[{"start":106110,"end":106603,"start_line":1085,"end_line":1092,"raw_tex":"By Assumption \\ref{ass:4}, we have\n\\begin{equation}\\label{pequ:A3}\n\\abr{f(\\xb) - f(\\ttxb) - \\frac{1}{2}\\nbr{\\xb - \\ttxb}_{\\ttHb}^2} \\leq \\frac{L}{6}\\nbr{\\xb - \\ttxb}^3.\n\\end{equation}\nSuppose $\\xb \\in \\bar{\\N}_{\\nu^2/3}$ for $\\nu\\in(0, 1]$, we have\n\\begin{equation}\\label{pequ:A4}\nf(\\ttxb) + \\frac{\\lambda_{\\min}}{2}\\|\\xb - \\ttxb\\|^2\\leq f(\\xb) \\leq f(\\ttxb) + \\frac{\\nu^2\\lambda_{\\min}^3}{3L^2} \\Longrightarrow \\|\\xb - \\ttxb\\| \\leq \\frac{\\sqrt{2}\\nu\\lambda_{\\min}}{\\sqrt{3}L}.\n\\end{equation}\n","sha256_utf8":"106b4bd2dc3ae0e2302b76863b23aeee34e69cdf1c5664f996ab3b3e44524ddb"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"函数邻域包含于范数邻域","source_units":[{"start":106603,"end":107521,"start_line":1093,"end_line":1100,"raw_tex":"Combining \\eqref{pequ:A3} and \\eqref{pequ:A4}, we know that $\\xb \\in \\bar{\\N}_{\\nu^2/3}$ for $\\nu\\in(0, 1]$ leads to\n\\begin{align*}\nf(\\ttxb) + \\frac{\\nu^2\\lambda_{\\min}^3}{3L^2} \\geq f(\\xb) & \\stackrel{\\mathclap{\\eqref{pequ:A3}}}{\\geq} f(\\ttxb) + \\frac{1}{2}\\|\\xb - \\ttxb\\|_{\\ttHb}^2 - \\frac{L}{6}\\|\\xb-\\ttxb\\|^3\t\\\\\n& \\geq f(\\ttxb) + \\frac{1}{2}\\nbr{\\xb - \\ttxb}_{\\ttHb}^2 - \\frac{L}{6}\\nbr{\\xb - \\ttxb}\\cdot \\frac{\\nbr{\\xb - \\ttxb}_{\\ttHb}^2}{\\lambda_{\\min}}\\\\\n& = f(\\ttxb) + \\rbr{\\frac{1}{2}-\\frac{L\\|\\xb - \\ttxb\\|}{6\\lambda_{\\min}}}\\|\\xb - \\ttxb\\|_{\\ttHb}^2\\\\\n& \\stackrel{\\mathclap{\\eqref{pequ:A4}}}{\\geq} f(\\ttxb) + \\rbr{\\frac{1}{2} - \\frac{\\sqrt{2}\\nu}{6\\sqrt{3}}}\\|\\xb - \\ttxb\\|_{\\ttHb}^2 \\geq f(\\ttxb) + \\frac{\\|\\xb - \\ttxb\\|_{\\ttHb}^2}{3},\n\\end{align*}\nwhich further implies $\\|\\xb - \\ttxb\\|_{\\ttHb} \\leq \\nu\\lambda_{\\min}^{3/2}/L$ and $\\xb\\in \\N_{\\nu}$. On the other hand, suppose $\\xb \\in \\N_{\\nu}$, we have\n","sha256_utf8":"cd535ea86cfd2dcc6da3b1de046c18943d820781491fda251625eb5cabe97f3a"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"范数邻域包含于函数邻域","source_units":[{"start":107521,"end":108053,"start_line":1101,"end_line":1105,"raw_tex":"\\begin{align*}\nf(\\xb)  &\\stackrel{\\mathclap{\\eqref{pequ:A3}}}{\\leq} f(\\ttxb) + \\frac{1}{2}\\nbr{\\xb - \\ttxb}_{\\ttHb}^2 + \\frac{L}{6}\\nbr{\\xb - \\ttxb}^3 \\leq f(\\ttxb) + \\rbr{ \\frac{1}{2} + \\frac{L\\nbr{\\xb - \\ttxb}_{\\ttHb}}{6\\lambda_{\\min}^{3/2}}} \\nbr{\\xb - \\ttxb}_{\\ttHb}^2\\\\\n& \\leq f(\\ttxb) + \\rbr{ \\frac{1}{2} + \\frac{\\nu}{6}} \\nbr{\\xb - \\ttxb}_{\\ttHb}^2\\leq f(\\ttxb)  + \\nbr{\\xb - \\ttxb}_{\\ttHb}^2 \\leq f(\\ttxb) + \\frac{\\nu^2\\lambda_{\\min}^3}{L^2}.\n\\end{align*}\nThus, we have $\\xb \\in \\bar{\\N}_{\\nu^2}$. This completes the proof.\n","sha256_utf8":"a1667122422d83bc6246112ce63c33c760a7bcfc00636d246a453bb8ca1f9896"}]},{"id":"S052","semantic_type":"lemma","kind":"lemma","title":"局部接受单位步长","source_units":[{"start":58856,"end":59229,"start_line":517,"end_line":522,"raw_tex":"\\begin{lemma}\\label{lem:5}\nUnder Assumption \\ref{ass:4}, suppose $\\pb_t = -(\\tHb_t)^{-1}\\nabla f_t$. Then $\\mu_t = 1$ if $\\xb_t \\in \\N_{\\nu}$ and $(1-\\psi)\\Hb_t \\preceq \\tHb_t \\preceq(1+\\psi)\\Hb_t$ with $\\nu, \\psi$ satisfying\n\\begin{equation}\\label{cond:nupsi}\n0<\\nu \\leq \\frac{2}{3}(1/2-\\beta), \\quad\\quad 0<\\psi\\leq \\frac{1/2-\\beta}{3/2-\\beta}.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"f490a3a5fff9d960d0831a4a52fef9139c8dcc759f9ac95ba9124eead98d28d9"}]},{"id":"S053","semantic_type":"proof","kind":"proof","title":"单位步长完整证明","source_units":[{"start":59232,"end":59286,"start_line":525,"end_line":527,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{pf:lem:5}.\n\\end{proof}","sha256_utf8":"3bfb6a814131cf8940a8a1b1a239cb942aba73aeb8dee8b9ba3340f55a21e651"},{"start":115413,"end":118710,"start_line":1221,"end_line":1260,"raw_tex":"\\subsection{Proof of Lemma \\ref{lem:5}}\\label{pf:lem:5}\n\nIt suffices to show \n\\begin{equation*}\nf(\\xb_t - (\\tHb_t)^{-1}\\nabla f_t) \\leq f(\\xb_t) - \\beta\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t.\n\\end{equation*}\nBy Assumption \\ref{ass:4}, we have\n\\begin{align}\\label{npequ:3}\n& f(\\xb_t - (\\tHb_t)^{-1}\\nabla f_t) \\nonumber\\\\\n& \\leq f(\\xb_t) - \\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t + \\frac{1}{2}\\nabla f_t^\\top (\\tHb_t)^{-1}\\Hb_t(\\tHb_t)^{-1}\\nabla f_t + \\frac{L}{6}\\|(\\tHb_t)^{-1}\\nabla f_t\\|^3 \\nonumber\\\\\n& \\leq f(\\xb_t) - \\frac{1}{2}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t + \\frac{1}{2}\\nabla f_t^\\top(\\tHb_t)^{-1}(\\Hb_t - \\tHb_t)(\\tHb_t)^{-1}\\nabla f_t + \\frac{L}{6}\\|(\\tHb_t)^{-1}\\nabla f_t\\|^3 \\nonumber\\\\\n& \\leq f(\\xb_t) - \\frac{1}{2}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t + \\frac{1}{2}\\|(\\tHb_t)^{-1/2}(\\Hb_t - \\tHb_t)(\\tHb_t)^{-1/2}\\|\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t \\nonumber\\\\\n& \\quad + \\frac{L}{6}\\|(\\tHb_t)^{-1}\\nabla f_t\\| \\nabla f_t^\\top(\\tHb_t)^{-2}\\nabla f_t \\nonumber\\\\\n& \\leq f(\\xb_t) - \\frac{1}{2}\\nabla f_t^\\top (\\tHb_t)^{-1}\\nabla f_t + \\frac{\\psi}{2(1-\\psi)}\\nabla f_t^\\top (\\tHb_t)^{-1}\\nabla f_t + \\frac{L\\|(\\tHb_t)^{-1}\\nabla f_t\\|}{6(1-\\psi)\\lambda_{\\min}}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t \\nonumber\\\\\n& = f(\\xb_t) - \\rbr{\\frac{1}{2} - \\frac{\\psi}{2(1-\\psi)} - \\frac{L\\|(\\tHb_t)^{-1}\\nabla f_t\\|}{6(1-\\psi)\\lambda_{\\min}}}\\nabla f_t^\\top (\\tHb_t)^{-1}\\nabla f_t.\n\\end{align}\nFor term $\\|(\\tHb_t)^{-1}\\nabla f_t\\|$, we let $\\Hb_t^\\eta = \\Hb(\\xb_t^\\eta)$ with $\\xb_t^\\eta = \\ttxb + \\eta(\\xb_t - \\ttxb)$ for some $\\eta\\in(0, 1)$, and have\n\\begin{align}\\label{npequ:4}\n&\\|(\\tHb_t)^{-1}\\nabla f_t\\|^2  = (\\xb_t - \\ttxb)^\\top (\\ttHb)^{1/2}(\\ttHb)^{-1/2}\\Hb_t^\\eta(\\tHb_t)^{-2}\\Hb_t^\\eta(\\ttHb)^{-1/2}(\\ttHb)^{1/2}(\\xb_t-\\ttxb) \\nonumber\\\\\n& \\leq \\frac{\\|\\xb_t - \\ttxb\\|_{\\ttHb}^2}{(1-\\psi)\\lambda_{\\min}}\\|(\\ttHb)^{-1/2}\\Hb_t^\\eta(\\tHb_t)^{-1}\\Hb_t^\\eta(\\ttHb)^{-1/2}\\| \\nonumber\\\\\n& \\leq \\frac{\\nu^2\\lambda_{\\min}^2}{(1-\\psi)L^2}\\|(\\ttHb)^{-1/2}(\\Hb_t^\\eta)^{1/2}\\|^2\\|(\\Hb_t^\\eta)^{1/2}(\\tHb_t)^{-1}(\\Hb_t^\\eta)^{1/2}\\| \\quad (\\text{since } \\xb_t\\in \\N_{\\nu}) \\nonumber\\\\\n& \\leq \\frac{\\nu^2\\lambda_{\\min}^2}{(1-\\psi)L^2}\\|(\\ttHb)^{-1/2}\\Hb_t^\\eta(\\ttHb)^{-1/2}\\|\\|(\\Hb_t^\\eta)^{1/2}(\\Hb_t)^{-1}(\\Hb_t^\\eta)^{1/2}\\|\\|\\Hb_t^{1/2}(\\tHb_t)^{-1}\\Hb_t^{1/2}\\| \\nonumber\\\\\n& \\leq \\frac{\\nu^2\\lambda_{\\min}^2}{(1-\\psi)^2L^2}\\|(\\ttHb)^{-1/2}\\Hb_t^\\eta(\\ttHb)^{-1/2}\\| \\|(\\Hb_t)^{-1/2}\\Hb_t^\\eta(\\Hb_t)^{-1/2}\\|.\n\\end{align}\nNoting that $\\xb_t^\\eta\\in\\N_{\\nu}$ if $\\xb_t \\in \\N_\\nu$, we apply Lemma \\ref{lem:pre:2} and have\n\\begin{equation}\\label{npequ:5}\n\\|(\\ttHb)^{-1/2}\\Hb_t^\\eta(\\ttHb)^{-1/2}\\| \\vee \\|(\\Hb_t)^{-1/2}\\Hb_t^\\eta(\\Hb_t)^{-1/2}\\| \\leq 1 + \\nu\\leq 2.\n\\end{equation}\nCombining \\eqref{npequ:3}, \\eqref{npequ:4}, and \\eqref{npequ:5}, we finally obtain\n\\begin{equation*}\nf(\\xb_t - (\\tHb_t)^{-1}\\nabla f_t) \\leq f(\\xb_t) - \\rbr{\\frac{1}{2} - \\frac{\\psi}{2(1-\\psi)} - \\frac{\\nu}{3(1-\\psi)^2}}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t.\n\\end{equation*}\nThus, it suffices to let\n\\begin{equation*}\n\\frac{1}{2} - \\frac{\\psi}{2(1-\\psi)} - \\frac{\\nu}{3(1-\\psi)^2} \\geq \\beta \\Longleftarrow \\frac{\\psi}{2(1-\\psi)} \\vee \\frac{\\nu}{3(1-\\psi)^2} \\leq \\frac{1}{2}\\rbr{\\frac{1}{2}-\\beta}\n\\end{equation*}\nas implied by the conditions stated in the lemma. This completes the proof.\n\n\n\n","sha256_utf8":"fd357ecb3ae80a7762c12c65514a646a11b470074352da47077412ae2f316077"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"单位步长证明附录定位","source_units":[{"start":59232,"end":59286,"start_line":525,"end_line":527,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{pf:lem:5}.\n\\end{proof}","sha256_utf8":"3bfb6a814131cf8940a8a1b1a239cb942aba73aeb8dee8b9ba3340f55a21e651"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"展开单位步长目标下降","source_units":[{"start":115452,"end":116822,"start_line":1221,"end_line":1236,"raw_tex":"\\label{pf:lem:5}\n\nIt suffices to show \n\\begin{equation*}\nf(\\xb_t - (\\tHb_t)^{-1}\\nabla f_t) \\leq f(\\xb_t) - \\beta\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t.\n\\end{equation*}\nBy Assumption \\ref{ass:4}, we have\n\\begin{align}\\label{npequ:3}\n& f(\\xb_t - (\\tHb_t)^{-1}\\nabla f_t) \\nonumber\\\\\n& \\leq f(\\xb_t) - \\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t + \\frac{1}{2}\\nabla f_t^\\top (\\tHb_t)^{-1}\\Hb_t(\\tHb_t)^{-1}\\nabla f_t + \\frac{L}{6}\\|(\\tHb_t)^{-1}\\nabla f_t\\|^3 \\nonumber\\\\\n& \\leq f(\\xb_t) - \\frac{1}{2}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t + \\frac{1}{2}\\nabla f_t^\\top(\\tHb_t)^{-1}(\\Hb_t - \\tHb_t)(\\tHb_t)^{-1}\\nabla f_t + \\frac{L}{6}\\|(\\tHb_t)^{-1}\\nabla f_t\\|^3 \\nonumber\\\\\n& \\leq f(\\xb_t) - \\frac{1}{2}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t + \\frac{1}{2}\\|(\\tHb_t)^{-1/2}(\\Hb_t - \\tHb_t)(\\tHb_t)^{-1/2}\\|\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t \\nonumber\\\\\n& \\quad + \\frac{L}{6}\\|(\\tHb_t)^{-1}\\nabla f_t\\| \\nabla f_t^\\top(\\tHb_t)^{-2}\\nabla f_t \\nonumber\\\\\n& \\leq f(\\xb_t) - \\frac{1}{2}\\nabla f_t^\\top (\\tHb_t)^{-1}\\nabla f_t + \\frac{\\psi}{2(1-\\psi)}\\nabla f_t^\\top (\\tHb_t)^{-1}\\nabla f_t + \\frac{L\\|(\\tHb_t)^{-1}\\nabla f_t\\|}{6(1-\\psi)\\lambda_{\\min}}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t \\nonumber\\\\\n& = f(\\xb_t) - \\rbr{\\frac{1}{2} - \\frac{\\psi}{2(1-\\psi)} - \\frac{L\\|(\\tHb_t)^{-1}\\nabla f_t\\|}{6(1-\\psi)\\lambda_{\\min}}}\\nabla f_t^\\top (\\tHb_t)^{-1}\\nabla f_t.\n\\end{align}\n","sha256_utf8":"7f91735c851b1fed62f8654fd256a78064d66d38c9d88602280fd76643243f54"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"控制近似牛顿步长度","source_units":[{"start":116822,"end":117858,"start_line":1237,"end_line":1244,"raw_tex":"For term $\\|(\\tHb_t)^{-1}\\nabla f_t\\|$, we let $\\Hb_t^\\eta = \\Hb(\\xb_t^\\eta)$ with $\\xb_t^\\eta = \\ttxb + \\eta(\\xb_t - \\ttxb)$ for some $\\eta\\in(0, 1)$, and have\n\\begin{align}\\label{npequ:4}\n&\\|(\\tHb_t)^{-1}\\nabla f_t\\|^2  = (\\xb_t - \\ttxb)^\\top (\\ttHb)^{1/2}(\\ttHb)^{-1/2}\\Hb_t^\\eta(\\tHb_t)^{-2}\\Hb_t^\\eta(\\ttHb)^{-1/2}(\\ttHb)^{1/2}(\\xb_t-\\ttxb) \\nonumber\\\\\n& \\leq \\frac{\\|\\xb_t - \\ttxb\\|_{\\ttHb}^2}{(1-\\psi)\\lambda_{\\min}}\\|(\\ttHb)^{-1/2}\\Hb_t^\\eta(\\tHb_t)^{-1}\\Hb_t^\\eta(\\ttHb)^{-1/2}\\| \\nonumber\\\\\n& \\leq \\frac{\\nu^2\\lambda_{\\min}^2}{(1-\\psi)L^2}\\|(\\ttHb)^{-1/2}(\\Hb_t^\\eta)^{1/2}\\|^2\\|(\\Hb_t^\\eta)^{1/2}(\\tHb_t)^{-1}(\\Hb_t^\\eta)^{1/2}\\| \\quad (\\text{since } \\xb_t\\in \\N_{\\nu}) \\nonumber\\\\\n& \\leq \\frac{\\nu^2\\lambda_{\\min}^2}{(1-\\psi)L^2}\\|(\\ttHb)^{-1/2}\\Hb_t^\\eta(\\ttHb)^{-1/2}\\|\\|(\\Hb_t^\\eta)^{1/2}(\\Hb_t)^{-1}(\\Hb_t^\\eta)^{1/2}\\|\\|\\Hb_t^{1/2}(\\tHb_t)^{-1}\\Hb_t^{1/2}\\| \\nonumber\\\\\n& \\leq \\frac{\\nu^2\\lambda_{\\min}^2}{(1-\\psi)^2L^2}\\|(\\ttHb)^{-1/2}\\Hb_t^\\eta(\\ttHb)^{-1/2}\\| \\|(\\Hb_t)^{-1/2}\\Hb_t^\\eta(\\Hb_t)^{-1/2}\\|.\n\\end{align}\n","sha256_utf8":"f7dd888a76b2a9d2411a12752e72b2452f1034383a7d53ded091bc37de359692"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"局部海森比较界","source_units":[{"start":117858,"end":118115,"start_line":1245,"end_line":1248,"raw_tex":"Noting that $\\xb_t^\\eta\\in\\N_{\\nu}$ if $\\xb_t \\in \\N_\\nu$, we apply Lemma \\ref{lem:pre:2} and have\n\\begin{equation}\\label{npequ:5}\n\\|(\\ttHb)^{-1/2}\\Hb_t^\\eta(\\ttHb)^{-1/2}\\| \\vee \\|(\\Hb_t)^{-1/2}\\Hb_t^\\eta(\\Hb_t)^{-1/2}\\| \\leq 1 + \\nu\\leq 2.\n\\end{equation}\n","sha256_utf8":"c937f495c1a1fd09354935206fd64c64fcf2755bdcd6a47819b3e2a93dfd50d1"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"验证单位步长阿米霍条件","source_units":[{"start":118115,"end":118707,"start_line":1249,"end_line":1257,"raw_tex":"Combining \\eqref{npequ:3}, \\eqref{npequ:4}, and \\eqref{npequ:5}, we finally obtain\n\\begin{equation*}\nf(\\xb_t - (\\tHb_t)^{-1}\\nabla f_t) \\leq f(\\xb_t) - \\rbr{\\frac{1}{2} - \\frac{\\psi}{2(1-\\psi)} - \\frac{\\nu}{3(1-\\psi)^2}}\\nabla f_t^\\top(\\tHb_t)^{-1}\\nabla f_t.\n\\end{equation*}\nThus, it suffices to let\n\\begin{equation*}\n\\frac{1}{2} - \\frac{\\psi}{2(1-\\psi)} - \\frac{\\nu}{3(1-\\psi)^2} \\geq \\beta \\Longleftarrow \\frac{\\psi}{2(1-\\psi)} \\vee \\frac{\\nu}{3(1-\\psi)^2} \\leq \\frac{1}{2}\\rbr{\\frac{1}{2}-\\beta}\n\\end{equation*}\nas implied by the conditions stated in the lemma. This completes the proof.\n","sha256_utf8":"1582308fe5d85fc694ee54121a795d943e0c91cc6106fb62add7a4c88a0b0b34"}]},{"id":"S059","semantic_type":"lemma","kind":"lemma","title":"相对海森差的距离界","source_units":[{"start":108067,"end":108353,"start_line":1109,"end_line":1114,"raw_tex":"\\begin{lemma}\\label{lem:pre:2}\n\tSuppose Assumption \\ref{ass:4} holds, then we have\n\t\\begin{equation*}\n\t\t\\|\\Hb(\\xb_1)^{-1/2}\\Hb(\\xb_2)\\Hb(\\xb_1)^{-1/2} - \\Ib\\| \\leq \\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_1 - \\xb_2\\|_{\\ttHb}, \\quad \\forall \\xb_1, \\xb_2\\in \\mR^d.\n\t\\end{equation*}\n\\end{lemma}","sha256_utf8":"26efdd013e2138d8d09bb383cc8ef20fa001203465dc74998c2d06858a78712a"}]},{"id":"S060","semantic_type":"proof","kind":"proof","title":"相对海森差完整证明","source_units":[{"start":108355,"end":108675,"start_line":1116,"end_line":1122,"raw_tex":"\\begin{proof}\nWe note that\n\\begin{equation*}\n\\|\\Hb(\\xb_1)^{-1/2}\\Hb(\\xb_2)\\Hb(\\xb_1)^{-1/2} - \\Ib\\|  \\leq \\frac{1}{\\lambda_{\\min}}\\|\\Hb(\\xb_1) - \\Hb(\\xb_2)\\| \\leq \\frac{L}{\\lambda_{\\min}}\\|\\xb_1 - \\xb_2\\| \\leq \\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_1 - \\xb_2\\|_{\\ttHb}.\n\\end{equation*}\nThis completes the proof.\n\\end{proof}","sha256_utf8":"48a89b04c5ea4b224a0c4708b1f7fdce1b8eb3d8664b6e3a08ad6034366c1047"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"谱界与利普希茨界组合","source_units":[{"start":108369,"end":108664,"start_line":1117,"end_line":1121,"raw_tex":"We note that\n\\begin{equation*}\n\\|\\Hb(\\xb_1)^{-1/2}\\Hb(\\xb_2)\\Hb(\\xb_1)^{-1/2} - \\Ib\\|  \\leq \\frac{1}{\\lambda_{\\min}}\\|\\Hb(\\xb_1) - \\Hb(\\xb_2)\\| \\leq \\frac{L}{\\lambda_{\\min}}\\|\\xb_1 - \\xb_2\\| \\leq \\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_1 - \\xb_2\\|_{\\ttHb}.\n\\end{equation*}\nThis completes the proof.\n","sha256_utf8":"f7f2e0225d5c120838d22164aa68949362503ecaa098167a0b065e761ac94f10"}]},{"id":"S062","semantic_type":"lemma","kind":"lemma","title":"局部线性二次误差递推","source_units":[{"start":59362,"end":59829,"start_line":531,"end_line":536,"raw_tex":"\\begin{lemma}\\label{lem:6}\nUnder Assumption \\ref{ass:4} and suppose $\\pb_t = -(\\tHb_t)^{-1}\\nabla f_t$, $\\xb_t \\in \\N_{\\nu}$, and $(1-\\psi)\\Hb_t \\preceq \\tHb_t \\preceq(1+\\psi)\\Hb_t$ with $\\nu, \\psi$ satisfying \\eqref{cond:nupsi}. Then, we have\n\\begin{equation*}\n\\nbr{\\xb_{t+1} - \\ttxb}_{\\ttHb} \\leq 3\\cbr{ \\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_t-\\ttxb\\|_{\\ttHb}^2 + \\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| \\cdot \\|\\xb_t - \\ttxb\\|_{\\ttHb}}.\n\\end{equation*}\n\\end{lemma}","sha256_utf8":"f263aeef75d6c1e0445a03b4c9160241f20485127a5e7baf008b6b3db3b6722f"}]},{"id":"S063","semantic_type":"proof","kind":"proof","title":"局部误差递推完整证明","source_units":[{"start":59831,"end":59885,"start_line":538,"end_line":540,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{pf:lem:6}.\n\\end{proof}","sha256_utf8":"951181f80e1be9a65e1a6d7658f6f704faeff4b546f4ab0eea67289be5a03ea0"},{"start":118710,"end":122768,"start_line":1261,"end_line":1318,"raw_tex":"\\subsection{Proof of Lemma \\ref{lem:6}}\\label{pf:lem:6}\n\nSince $\\xb_t\\in \\N_{\\nu}$ with $\\nu \\leq 2/3\\cdot (0.5-\\beta)$, we apply Lemma \\ref{lem:pre:2} and have\n\\begin{equation*}\n\\|(\\ttHb)^{-1/2}(\\Hb_t - \\ttHb)(\\ttHb)^{-1/2}\\| \\leq \\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_t - \\ttxb\\|_{\\ttHb} \\leq \\nu\\leq 0.5-\\beta.\n\\end{equation*}\nThus, we obtain\n\\begin{equation}\\label{pequ:B2}\n\\rbr{0.5+\\beta}\\ttHb\\preceq \\Hb_t\\preceq \\rbr{1.5-\\beta}\\ttHb. \n\\end{equation}\nSince $\\pb_t = -(\\tHb_t)^{-1}\\nabla f_t$ and $\\mu_t=1$, we have\n\\begin{align}\\label{npequ:6}\n& \\nbr{\\xb_{t+1} - \\ttxb}_{\\ttHb} = \\|\\xb_t - (\\tHb_t)^{-1}\\nabla f_t - \\ttxb\\|_{\\ttHb} \\nonumber\\\\\n& \\stackrel{\\mathclap{\\eqref{pequ:B2}}}{\\leq}\\frac{1}{\\sqrt{0.5+\\beta}}\\|\\xb_t - (\\tHb_t)^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} \\nonumber\\\\\n& \\leq \\frac{1}{\\sqrt{0.5+\\beta}}\\cbr{\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} + \\|\\Hb_t^{-1}\\nabla f_t - (\\tHb_t)^{-1}\\nabla f_t\\|_{\\Hb_t}}.\n\\end{align}\nFor the second term on the right hand side, we have\n\\begin{align}\\label{npequ:7}\n& \\|\\Hb_t^{-1}\\nabla f_t - (\\tHb_t)^{-1}\\nabla f_t\\|_{\\Hb_t} \\nonumber\\\\\n& = \\sqrt{\\nabla f_t^\\top(\\Hb_t^{-1} - (\\tHb_t)^{-1})\\Hb_t(\\Hb_t^{-1} - (\\tHb_t)^{-1})\\nabla f_t} \\nonumber\\\\\n& = \\sqrt{\\nabla f_t^\\top\\Hb_t^{-1/2}(\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2})^2\\Hb_t^{-1/2}\\nabla f_t} \\nonumber\\\\\n& \\leq \\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\|\\cdot\\|\\Hb_t^{-1}\\nabla f_t\\|_{\\Hb_t}  \\nonumber\\\\\n& \\leq  \\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\|\\cbr{\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} + \\|\\xb_t - \\ttxb\\|_{\\Hb_t}} \\nonumber\\\\\n& \\stackrel{\\mathclap{\\eqref{pequ:B2}}}{\\leq}\\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\|\\cbr{\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} + \\sqrt{1.5-\\beta}\\|\\xb_t - \\ttxb\\|_{\\ttHb}}.\n\\end{align}\nCombining \\eqref{npequ:6} and \\eqref{npequ:7},\n\\begin{align}\\label{pequ:B3}\n\\|\\xb_{t+1} - \\ttxb\\|_{\\ttHb}& \\leq\\frac{1}{\\sqrt{0.5+\\beta}} \\big\\{(1+\\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\|)\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} \\nonumber\\\\\n& \\quad + \\sqrt{1.5-\\beta}\\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\|\\cdot \\|\\xb_t - \\ttxb\\|_{\\ttHb}\\big\\}.\n\\end{align}\nSince $\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| \\leq \\psi\\leq 1/3$, we have\n\\begin{equation*}\n(1 - \\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| )\\cdot \\Ib \\preceq \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2} \\preceq (1+\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\|)\\cdot\\Ib\n\\end{equation*}\nand further\n\\begin{equation*}\n-\\frac{\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }{1+\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }\\cdot \\Ib\\preceq \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2} - \\Ib \\preceq \\frac{\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }{1-\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }\\cdot\\Ib.\n\\end{equation*}\nThus, we have\n\\begin{equation*}\n\\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\| \\leq \\frac{\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }{1 - \\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }\\leq \\frac{3}{2}\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\|.\n\\end{equation*}\nCombining the above inequality with \\eqref{pequ:B3}, we obtain\n\\begin{align}\\label{pequ:B4}\n&\\|\\xb_{t+1} - \\ttxb\\|_{\\ttHb} \\nonumber\\\\\n&  \\leq \\frac{3}{2\\sqrt{0.5+\\beta}} \\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} + \\frac{3}{2}\\sqrt{\\frac{1.5-\\beta}{0.5+\\beta}}\\cdot\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\|  \\cdot \\|\\xb_t - \\ttxb\\|_{\\ttHb} \\nonumber\\\\\n& \\leq \\frac{3\\sqrt{2}}{2}\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t}  + \\frac{3\\sqrt{3}}{2}\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\|  \\cdot \\|\\xb_t - \\ttxb\\|_{\\ttHb}.\n\\end{align}\nFurthermore,\n\\begin{align*}\n\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} & = \\|\\Hb_t^{-1/2}(\\Hb_t(\\xb_t - \\ttxb) - \\nabla f_t)\\|\\\\\n& \\leq \\frac{1}{\\sqrt{\\lambda_{\\min}}}\\nbr{\\Hb_t(\\xb_t - \\ttxb) - \\rbr{\\int_0^1\\Hb(\\ttxb + \\tau(\\xb_t - \\ttxb))d\\tau }(\\xb_t-\\ttxb)}\\\\\n& \\leq \\frac{\\|\\xb_t -\\ttxb\\|^2}{\\sqrt{\\lambda_{\\min}}}\\cdot \\int_0^1(1-\\tau)L d\\tau \\leq \\frac{L}{2\\lambda_{\\min}^{3/2}}\\|\\xb_t-\\ttxb\\|_{\\ttHb}^2.\n\\end{align*}\nCombining the above inequality with \\eqref{pequ:B4}, we complete the proof.\n\n\n\n","sha256_utf8":"97a32ddf6277acf5854d4052e66c7efc6fce9fda513b2867b3626de4d32abd53"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"局部递推证明附录定位","source_units":[{"start":59831,"end":59885,"start_line":538,"end_line":540,"raw_tex":"\\begin{proof}\nSee Appendix \\ref{pf:lem:6}.\n\\end{proof}","sha256_utf8":"951181f80e1be9a65e1a6d7658f6f704faeff4b546f4ab0eea67289be5a03ea0"}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","title":"当前与最优海森比较","source_units":[{"start":118749,"end":119165,"start_line":1261,"end_line":1270,"raw_tex":"\\label{pf:lem:6}\n\nSince $\\xb_t\\in \\N_{\\nu}$ with $\\nu \\leq 2/3\\cdot (0.5-\\beta)$, we apply Lemma \\ref{lem:pre:2} and have\n\\begin{equation*}\n\\|(\\ttHb)^{-1/2}(\\Hb_t - \\ttHb)(\\ttHb)^{-1/2}\\| \\leq \\frac{L}{\\lambda_{\\min}^{3/2}}\\|\\xb_t - \\ttxb\\|_{\\ttHb} \\leq \\nu\\leq 0.5-\\beta.\n\\end{equation*}\nThus, we obtain\n\\begin{equation}\\label{pequ:B2}\n\\rbr{0.5+\\beta}\\ttHb\\preceq \\Hb_t\\preceq \\rbr{1.5-\\beta}\\ttHb. \n\\end{equation}\n","sha256_utf8":"712a6466d297a74e0f9a5984ecf20138c5aba68ea823dda7df4114b43774cddd"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"分离精确步与近似步误差","source_units":[{"start":119165,"end":119655,"start_line":1271,"end_line":1276,"raw_tex":"Since $\\pb_t = -(\\tHb_t)^{-1}\\nabla f_t$ and $\\mu_t=1$, we have\n\\begin{align}\\label{npequ:6}\n& \\nbr{\\xb_{t+1} - \\ttxb}_{\\ttHb} = \\|\\xb_t - (\\tHb_t)^{-1}\\nabla f_t - \\ttxb\\|_{\\ttHb} \\nonumber\\\\\n& \\stackrel{\\mathclap{\\eqref{pequ:B2}}}{\\leq}\\frac{1}{\\sqrt{0.5+\\beta}}\\|\\xb_t - (\\tHb_t)^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} \\nonumber\\\\\n& \\leq \\frac{1}{\\sqrt{0.5+\\beta}}\\cbr{\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} + \\|\\Hb_t^{-1}\\nabla f_t - (\\tHb_t)^{-1}\\nabla f_t\\|_{\\Hb_t}}.\n\\end{align}\n","sha256_utf8":"b6e5e01bd243af69d0eb5ad2d3ffb55f026528fc7157fd5e1578ae88ef9ec407"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"控制逆海森扰动项","source_units":[{"start":119655,"end":120872,"start_line":1277,"end_line":1290,"raw_tex":"For the second term on the right hand side, we have\n\\begin{align}\\label{npequ:7}\n& \\|\\Hb_t^{-1}\\nabla f_t - (\\tHb_t)^{-1}\\nabla f_t\\|_{\\Hb_t} \\nonumber\\\\\n& = \\sqrt{\\nabla f_t^\\top(\\Hb_t^{-1} - (\\tHb_t)^{-1})\\Hb_t(\\Hb_t^{-1} - (\\tHb_t)^{-1})\\nabla f_t} \\nonumber\\\\\n& = \\sqrt{\\nabla f_t^\\top\\Hb_t^{-1/2}(\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2})^2\\Hb_t^{-1/2}\\nabla f_t} \\nonumber\\\\\n& \\leq \\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\|\\cdot\\|\\Hb_t^{-1}\\nabla f_t\\|_{\\Hb_t}  \\nonumber\\\\\n& \\leq  \\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\|\\cbr{\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} + \\|\\xb_t - \\ttxb\\|_{\\Hb_t}} \\nonumber\\\\\n& \\stackrel{\\mathclap{\\eqref{pequ:B2}}}{\\leq}\\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\|\\cbr{\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} + \\sqrt{1.5-\\beta}\\|\\xb_t - \\ttxb\\|_{\\ttHb}}.\n\\end{align}\nCombining \\eqref{npequ:6} and \\eqref{npequ:7},\n\\begin{align}\\label{pequ:B3}\n\\|\\xb_{t+1} - \\ttxb\\|_{\\ttHb}& \\leq\\frac{1}{\\sqrt{0.5+\\beta}} \\big\\{(1+\\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\|)\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} \\nonumber\\\\\n& \\quad + \\sqrt{1.5-\\beta}\\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\|\\cdot \\|\\xb_t - \\ttxb\\|_{\\ttHb}\\big\\}.\n\\end{align}\n","sha256_utf8":"ac03849f157177bd836ac3b2f3e3a67b7f7c3168060bfdea79e534a64f1bc2b3"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"逆矩阵误差转为原矩阵误差","source_units":[{"start":120872,"end":121714,"start_line":1291,"end_line":1302,"raw_tex":"Since $\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| \\leq \\psi\\leq 1/3$, we have\n\\begin{equation*}\n(1 - \\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| )\\cdot \\Ib \\preceq \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2} \\preceq (1+\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\|)\\cdot\\Ib\n\\end{equation*}\nand further\n\\begin{equation*}\n-\\frac{\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }{1+\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }\\cdot \\Ib\\preceq \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2} - \\Ib \\preceq \\frac{\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }{1-\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }\\cdot\\Ib.\n\\end{equation*}\nThus, we have\n\\begin{equation*}\n\\|\\Ib - \\Hb_t^{1/2}(\\tHb_t)^{-1} \\Hb_t^{1/2}\\| \\leq \\frac{\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }{1 - \\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\| }\\leq \\frac{3}{2}\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\|.\n\\end{equation*}\n","sha256_utf8":"31bdaf0bbc16106f8026fcef949ba51036982e68a92a16894661c827efe62d8f"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"合并扰动误差常数","source_units":[{"start":121714,"end":122258,"start_line":1303,"end_line":1308,"raw_tex":"Combining the above inequality with \\eqref{pequ:B3}, we obtain\n\\begin{align}\\label{pequ:B4}\n&\\|\\xb_{t+1} - \\ttxb\\|_{\\ttHb} \\nonumber\\\\\n&  \\leq \\frac{3}{2\\sqrt{0.5+\\beta}} \\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} + \\frac{3}{2}\\sqrt{\\frac{1.5-\\beta}{0.5+\\beta}}\\cdot\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\|  \\cdot \\|\\xb_t - \\ttxb\\|_{\\ttHb} \\nonumber\\\\\n& \\leq \\frac{3\\sqrt{2}}{2}\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t}  + \\frac{3\\sqrt{3}}{2}\\|\\Ib - \\Hb_t^{-1/2}\\tHb_t \\Hb_t^{-1/2}\\|  \\cdot \\|\\xb_t - \\ttxb\\|_{\\ttHb}.\n\\end{align}\n","sha256_utf8":"432a5f79b9753c2406b412678c5cb367e43fb5c0efca45c68903af2e6246e3f2"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"精确牛顿余项二次控制","source_units":[{"start":122258,"end":122765,"start_line":1309,"end_line":1315,"raw_tex":"Furthermore,\n\\begin{align*}\n\\|\\xb_t - \\Hb_t^{-1}\\nabla f_t - \\ttxb\\|_{\\Hb_t} & = \\|\\Hb_t^{-1/2}(\\Hb_t(\\xb_t - \\ttxb) - \\nabla f_t)\\|\\\\\n& \\leq \\frac{1}{\\sqrt{\\lambda_{\\min}}}\\nbr{\\Hb_t(\\xb_t - \\ttxb) - \\rbr{\\int_0^1\\Hb(\\ttxb + \\tau(\\xb_t - \\ttxb))d\\tau }(\\xb_t-\\ttxb)}\\\\\n& \\leq \\frac{\\|\\xb_t -\\ttxb\\|^2}{\\sqrt{\\lambda_{\\min}}}\\cdot \\int_0^1(1-\\tau)L d\\tau \\leq \\frac{L}{2\\lambda_{\\min}^{3/2}}\\|\\xb_t-\\ttxb\\|_{\\ttHb}^2.\n\\end{align*}\nCombining the above inequality with \\eqref{pequ:B4}, we complete the proof.\n","sha256_utf8":"25d791f53d7e36f6f7d4a6640aeeef1d88f006855b05b03450c869e144c1e939"}]},{"id":"S071","semantic_type":"problem_setting","kind":"problem_setting","title":"强凸光滑优化问题","source_units":[{"start":4450,"end":5151,"start_line":99,"end_line":107,"raw_tex":"We consider minimizing a smooth and strongly convex objective function:\n\\begin{equation}\\label{pro:1}\n\\min_{\\xb\\in \\mR^d}\\; f(\\xb),\n\\end{equation}\nwhere $f:\\mR^d\\rightarrow \\mR$ is twice continuously differentiable with $\\Hb(\\xb) = \\nabla^2 f(\\xb) \\in \\mR^{d\\times d}$ being its Hessian matrix. We suppose the Hessian $\\Hb(\\xb)$ satisfies\n\\begin{equation*}\n\\lambda_{\\min}\\cdot \\Ib \\preceq \\Hb(\\xb) \\preceq\\lambda_{\\max}\\cdot\\Ib, \\quad\\;  \\forall \\xb\\in\\mR^d,\n\\end{equation*}\nfor some constants $0<\\lambda_{\\min}\\leq\\lambda_{\\max}$, and we let $\\kappa = \\lambda_{\\max}/\\lambda_{\\min}$ denote its~condition number. We also let $\\xb^\\star = \\arg\\min_{\\xb\\in \\mR^d} f(\\xb)$ be the unique global~solution.\n","sha256_utf8":"001cbb4de324145ce1ae439b592bd025806ee66ff0b729c2e426fc8f833aec81"}]},{"id":"S072","semantic_type":"notation","kind":"notation","title":"牛顿更新与梯度记号","source_units":[{"start":5733,"end":5830,"start_line":111,"end_line":113,"raw_tex":"\\begin{equation}\\label{equ:Newton}\n\\xb_{t+1} = \\xb_t - \\mu_t \\Hb_t^{-1}\\nabla f_t,\n\\end{equation}","sha256_utf8":"49caad6e922d7bc26767e1140ae8058948f78d57744381e6d6258ee6b1a3098e"},{"start":5831,"end":6272,"start_line":114,"end_line":114,"raw_tex":"where $\\nabla f_t = \\nabla f(\\xb_t)$, $\\Hb_t = \\Hb(\\xb_t)$, and $\\mu_t$ is selected by passing a line search condition. Classical results indicate that Newton's method in \\eqref{equ:Newton} \\mbox{exploits}~a~global $Q$-linear convergence in the error of function value $f(\\xb_t)-f(\\tx)$; and a local $Q$-quadratic convergence in the iterate error $\\|\\xb_t-\\tx\\|$. More precisely, Newton's method has two phases, separated by a neighborhood \n","sha256_utf8":"1c3cb30ecaf98d7e79c32f85da76943a5221e01692133e17080b8856b4c1c9d0"}]},{"id":"S073","semantic_type":"problem_setting","kind":"problem_setting","title":"无偏随机海森预言机","source_units":[{"start":9446,"end":10120,"start_line":133,"end_line":136,"raw_tex":"A parallel line of research explores stochastic Newton methods, where a stochastic approximation $\\hHb_t$ is used in place of $\\Hb_t$ in \\eqref{equ:Newton}. The stochasticity of $\\hHb_t$ may come from evaluating the Hessian on a random subset of data points~(i.e., subsampling), or from projecting the Hessian onto a random subspace to~achieve~the dimension reduction (i.e., sketching). To unify different approaches, we consider in this paper a general Hessian approximation given by a \\textit{stochastic oracle}. In particular, we express the approximation $\\hHb(\\xb)$ at any point $\\xb$ by\n\\begin{equation}\\label{equ:error}\n\\hHb(\\xb) = \\Hb(\\xb) + \\Eb(\\xb),\n\\end{equation}","sha256_utf8":"844efd468e353c9913c57f3ae5a2f43af9c226f2938d731bcadb801fca718dfb"},{"start":10121,"end":10565,"start_line":137,"end_line":137,"raw_tex":"where $\\Eb(\\xb)\\in \\mS^{d\\times d}$ is a symmetric random noise matrix following a certain distribution (conditional on $\\xb$) with mean zero. At iterate $\\xb_t$, we query the oracle to obtain an approximation $\\hHb_t = \\hHb(\\xb_t)$, which (implicitly) comes from generating a realization of the random matrix $\\Eb_t = \\Eb(\\xb_t)$, and then adding $\\Eb_t$ to the true Hessian $\\Hb_t$. We do not assume $\\Eb_t$ and $\\Hb_t$ are accessible to us.\n","sha256_utf8":"99b061de9d268b9208fdf9fe4deed7dfae3aa7cd9214e78e0f133541f71090d9"}]},{"id":"S074","semantic_type":"definition","kind":"definition","title":"在线海森平均与等权特例","source_units":[{"start":20751,"end":20977,"start_line":206,"end_line":209,"raw_tex":"We consider an online averaging scheme (we let $\\tHb_{-1} = \\0$):\n\\begin{equation}\\label{equ:ave:Hessian}\n\\tHb_t  = \\frac{w_{t-1}}{w_t} \\tHb_{t-1} + \\rbr{1 - \\frac{w_{t-1}}{w_t}}\\hHb_t, \\quad t = 0,1,2,\\ldots,\\\\\n\\end{equation}","sha256_utf8":"8ffa6c349d5d924fa668363fffcf11e62e98c7abcefba574df54d1e1ea4f8771"},{"start":20978,"end":22344,"start_line":210,"end_line":214,"raw_tex":"where $\\{w_t\\}_{t=-1}^{\\infty}$ is a prespecified increasing non-negative weight sequence~starting with $w_{-1} = 0$. By \\emph{online} we mean that we only keep an average Hessian $\\tHb_t$ in each iteration, and update it as \\eqref{equ:ave:Hessian} when we obtain a new Hessian estimate. This scheme is in contrast to keeping all the past Hessian estimates. By the scheme \\eqref{equ:ave:Hessian}, we note that, at iteration $t$, we re-scale the weights of $\\{\\hHb_i\\}_{t=0}^{t-1}$ equally by a factor of $w_{t-1}/w_t$, instead of completely redefining all the weights. We note that such an online averaging scheme is memory and computation efficient: compared to stochastic Newton methods without averaging, we only require an extra $O(d^2)$ space to save $\\tHb_t$ and $O(d^2)$ flops to update it, which is negligible in the setting of stochastic Newton methods where the Hessian vector product requires $O(d^2)$ flops and solving the exact Newton system requires $O(d^3)$ flops. In \\eqref{equ:ave:Hessian}, we use the ratio factors $w_{t-1}/w_t$ instead of direct weights merely to simplify our later presentation. Furthermore, \\eqref{equ:ave:Hessian} can be reformulated as a general weighted average as follows:\\vskip-0.32cm\n\\begin{equation}\\label{equ:z}\n\\tHb_t =\\sum_{i=0}^tz_{i,t}\\hHb_i,\\qquad\\text{with}\\quad\n  z_{i,t}\\coloneqq(w_i-w_{i-1})/w_t.\n\\end{equation}","sha256_utf8":"08f4d79a53db8506df87be0bf78f4ae730cd67bc95d8a35221724b93aa1714bc"},{"start":22357,"end":22688,"start_line":215,"end_line":215,"raw_tex":"\\noindent In particular, by setting $w_t=t+1$, we obtain $z_{i,t} = 1/(t+1)$ and further have $\\tHb_t = \\frac1{t+1}\\sum_{i=0}^{t}\\hHb_i$. Thus, we recover simple uniform averaging. If we let the sequence $w_t$ grow faster-than-linearly,~this results in a weighted average that is skewed towards the more recent Hessian estimates. \n","sha256_utf8":"3438d5b3ec1a4dcf7d148bc0a4c998e1513b8428b4c8afce3158ffd9bc31cc72"}]},{"id":"S075","semantic_type":"algorithm","kind":"algorithm","title":"海森平均随机牛顿算法","source_units":[{"start":22691,"end":23790,"start_line":219,"end_line":236,"raw_tex":"\\begin{algorithm}[!t]\n\\caption{Stochastic Newton Method with Hessian Averaging}\n\\label{alg:1}\n\\begin{algorithmic}[1]\n\\State \\textbf{Input}: iterate $\\xb_0$, weights $\\{w_t\\}_{t=-1}^{\\infty}$, scalars $\\beta\\in(0, 1/2)$, $\\rho\\in(0, 1)$, and $\\tHb_{-1} = \\0$;\n\\For{ $t = 0,1,2,\\ldots$}\n\\State Obtain a stochastic Hessian approximation $\\hHb_t = \\Hb_t + \\Eb_t$;\n\\State Compute the Hessian average via \\eqref{equ:ave:Hessian}: $\\tHb_t  = \\frac{w_{t-1}}{w_t} \\tHb_{t-1} + \\rbr{1 - \\frac{w_{t-1}}{w_t}}\\hHb_t$;\n\\State Compute the Newton direction $\\pb_t$ by solving: $\\tHb_t\\pb_t = -\\nabla f_t$;\\vspace{1mm}\n\\State \\textbf{if} $\\tHb_t\\pb_t = -\\nabla f_t$ is not solvable \\textbf{or} $\\nabla f_t^\\top\\pb_t\\geq 0$, \\ \\textbf{then} \\ skip iteration with~$\\xb_{t+1}=\\xb_t$; \n\\State Compute a stepsize $\\mu_t = \\rho^{j_t}$, where $j_t$ is the smallest nonnegative integer such that\n\\begin{equation*}\n\\text{Armijo condition}:\\hskip 0.5cm f(\\xb_t+\\mu_t\\pb_t) \\leq f(\\xb_t) + \\mu_t\\beta \\nabla f_t^\\top\\pb_t\n\\end{equation*}\n\\State Update $\\xb_{t+1} = \\xb_t +\\mu_t\\pb_t$;\n\\EndFor\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"e5ed1828ddfcb3cdd24c100b105322c8147a257d1f95d3fbab572666120bff72"}]},{"id":"S076","semantic_type":"notation","kind":"notation","title":"最优海森与矩阵范数记号","source_units":[{"start":27003,"end":27942,"start_line":262,"end_line":262,"raw_tex":"\\noindent We mention that the superlinear rate is measured with respect to the error $\\|\\xb_t-\\ttxb\\|_{\\ttHb} \\coloneqq \\sqrt{(\\xb_t-\\ttxb)^\\top\\ttHb(\\xb_t-\\xb_0)}$ where~$\\ttHb = \\Hb(\\ttxb)$. Before introducing the main results in Theorems \\ref{thm:main-1} and \\ref{thm:main-2}, we summarize them in Table \\ref{tab:1}, showing the transition points for~two weight sequences. The transitions for general weights are provided~in Section~\\ref{sec:4}. We use $\\Upsilon$ to denote the \\textit{noise level} of stochastic Hessian oracle (see Assumption \\ref{ass:3} for a formal definition; typically $\\Upsilon = O(\\kappa)$). We also use $O(\\cdot)$ to suppress the logarithmic factors and the dependence on other constants except $\\Upsilon$ and $\\kappa$. We emphasize that all results of the~paper require that the (weighted) average of the errors $\\{\\Eb_i\\}_{i=0}^t$ is sufficiently concentrated; thus, they hold \\textit{with high probability}.\n","sha256_utf8":"5a7ea5ff53b7b8d8854d00aa6ad1fe0e40fe797e1069cd210f11efacadfce0ee"},{"start":43624,"end":44368,"start_line":346,"end_line":346,"raw_tex":"\\noindent{\\bf Notation}: Throughout the paper, we use $\\Ib$ to denote the identity matrix,~and~$\\0$ to denote the zero vector or matrix. Their dimensions are clear from the~context. We use $\\|\\cdot\\|$ to denote the $\\ell_2$ norm for vectors and spectral norm for matrices.  For a positive semidefinite matrix $\\Ab$, we let $\\|\\xb\\|_{\\Ab} = \\sqrt{\\xb^\\top\\Ab\\xb}$. For two scalars $a, b$, $a\\vee b = \\max(a,b)$ and $a\\wedge b = \\min(a, b)$. For two matrices $\\Ab, \\Bb$, $\\Ab \\prec (\\preceq)\\Bb$ if $\\Bb - \\Ab$ is a positive (semi)definite matrix. Recall that we reserve the notation $\\lambda_{\\min}, \\lambda_{\\max}$ to denote the lower and upper bounds of the true~Hessian, and~$\\kappa = \\lambda_{\\max}/\\lambda_{\\min}$ is the condition number. \n","sha256_utf8":"743f083b742b238d7873e6ac8c1c1dff3055bc176e361c49efb28661ee09ea5a"}]},{"id":"S077","semantic_type":"notation","kind":"notation","title":"过滤族与条件期望","source_units":[{"start":45298,"end":45836,"start_line":357,"end_line":357,"raw_tex":"Given a sequence  of stochastic iterates $\\{\\xb_t\\}_{t=0}^{\\infty}$, we let $\\mF_0\\subseteq\\mF_1\\subseteq\\mF_2\\subseteq\\ldots$ be a filtration where $\\mF_t = \\sigma(\\xb_{0:t})$, $\\forall t\\geq 0$, is the $\\sigma$-algebra~generated~by~the randomness from $\\xb_0$ to $\\xb_t$. With such a filtration, we denote the~conditional~expectation by~$\\mE_t[\\cdot] = \\mE[\\cdot \\mid \\mF_t]$ and the conditional probability by $\\mP_t(\\cdot) = \\mP(\\cdot \\mid \\mF_t)$. We suppose $\\xb_0$ is deterministic, so that $\\mF_0$ is a trivial $\\sigma$-algebra. \n","sha256_utf8":"200add066c7ee79697846f94b2e6cf1647cb8ade6b63219da7be40ac4180bb43"}]},{"id":"S078","semantic_type":"definition","kind":"definition","title":"海森平均与噪声平均分解","source_units":[{"start":45838,"end":46345,"start_line":360,"end_line":363,"raw_tex":"For a given weight sequence $\\{w_t\\}_{t=-1}^\\infty$ with $w_{-1}=0$, the scheme \\eqref{equ:ave:Hessian} leads~to \\vskip-0.4cm\n\\begin{equation}\\label{equ:simple}\n\\tHb_t\\;\\; \\stackrel{\\mathclap{\\eqref{equ:z}}}{=}\\; \\sum_{i=0}^{t}z_{i,t}\\hHb_i \\ \\stackrel{\\mathclap{\\eqref{equ:error}}}{=} \\!\\!\\underbrace{\\sum_{i=0}^{t}z_{i,t}\\Hb_i}_{\\text{Hessian averaging}}\\!\\!+\\!\\!\\quad\\barEb_t\\qquad\\quad\\text{for} \\quad \\barEb_t\\coloneqq\\!\\!\\underbrace{\\sum_{i=0}^{t}z_{i,t}\\Eb_i }_{\\text{noise averaging}}\n\\end{equation}","sha256_utf8":"01cf80e05a6386bbbb2824c17fd0520bcd4f1acfc94292f16f58f341179c6062"},{"start":46358,"end":46985,"start_line":364,"end_line":364,"raw_tex":"\\noindent where $z_{i,t} = (w_i - w_{i-1})/w_t$. Note that $\\sum_{i=0}^t z_{i,t}=1$ and $z_{i,t}\\propto z_i\\coloneqq w_i-w_{i-1}$, i.e., $z_{i,t}$ is proportional to an un-normalized weight $z_i$. We see from \\eqref{equ:simple} that~$\\tHb_t$ consists of the Hessian averaging and noise averaging with the same weights. In principle, the Hessian averaging $\\sum_{i=0}^{t}z_{i,t}\\Hb_i\\rightarrow \\ttHb$ as $\\xb_t\\rightarrow \\tx$, while the noise averaging $\\sum_{i=0}^{t}z_{i,t}\\Eb_i\\rightarrow \\0$ due to the central limit~theorem. We will show that the Hessian averaging (eventually) converges faster than the noise averaging.\n","sha256_utf8":"eb6695e078e39d4022213c6f208603c963d567af1b4d9706ab4990600d054fb6"}]},{"id":"S079","semantic_type":"assumption","kind":"assumption","title":"次指数噪声与尺度参数","source_units":[{"start":46987,"end":47529,"start_line":367,"end_line":367,"raw_tex":"To study the concentration of noise averaging $\\barEb_t$, we use the fact that~$\\{\\Eb_t\\}_{t=0}^\\infty$ is a martingale difference sequence, and rely on concentration inequalities for matrix martingales. These concentration inequalities require a sub-exponential tail condition on the noise.  We say that a random variable $X$ is $K$-sub-exponential if $\\mE[|X|^p]\\leq p!\\cdot K^p/2$ for all $p=2,3,...\\,$, which is consistent (up to constants) with all standard notions of sub-exponentiality \\citep[see Section 2.7 in][]{Vershynin2018High}.\n","sha256_utf8":"e42794037fdfa816f2e48be73de3d976ee82b3ef9930924f36c4b09a58e181e2"},{"start":47530,"end":47808,"start_line":369,"end_line":371,"raw_tex":"\\begin{assumption}[Sub-exponential noise]\\label{ass:3}\nWe assume that  $\\Eb(\\xb)$ is mean zero and $\\|\\Eb(\\xb)\\|$ is $\\Upsilon_E$-sub-exponential for all $\\xb$. Also, we define $\\Upsilon\\coloneqq\\Upsilon_E/\\lambda_{\\min}$ to be the scale-invariant noise level. \n\\end{assumption}","sha256_utf8":"29266f1d1ddceb482cc2228625bd470b4681e41f52db4a67d9eca6ad19d19ac8"}]},{"id":"S080","semantic_type":"assumption","kind":"assumption","title":"海森利普希茨连续","source_units":[{"start":57445,"end":57656,"start_line":491,"end_line":493,"raw_tex":"\\begin{assumption}[Lipschitz Hessian]\\label{ass:4}\nWe assume $\\Hb(\\xb)$ is $L$-Lipschitz continuous. That is $\\|\\Hb(\\xb_1) - \\Hb(\\xb_2)\\|\\leq L\\|\\xb_1 - \\xb_2\\|$ for any $\\xb_1, \\xb_2\\in \\mR^d$.\n\\end{assumption}","sha256_utf8":"b4bbf97df1476ff0ac29d6f4b9d87ca8fd935c6cb00936ec5c2d0df1423ade26"}]},{"id":"S081","semantic_type":"proof_step","kind":"proof_step","title":"可数联合界控制无限序列","source_units":[{"start":53885,"end":54159,"start_line":435,"end_line":441,"raw_tex":"\\begin{equation}\\label{equ:equal:con}\n\\mP\\rbr{\\forall t:\\ \\nbr{\\barEb_t} \\leq\n8\\Upsilon_E\\bigg(\\sqrt{\\frac{\\log(d(t+1)/\\delta)}{t+1}} \\vee\n  \\frac{\\log(d(t+1)/\\delta)}{t+1}\\bigg)} \n\\geq 1 - \\sum_{t=0}^{\\infty}\n\\frac{\\delta}{(t+1)^2}= 1-\\frac{\\pi^2\\delta}{6}.\n\\end{equation}\n","sha256_utf8":"942575e1e2dbb5f77eb06e0a21e0d0b364657a232cb33cff93db5a638fe8c899"}]},{"id":"S082","semantic_type":"external_fact","kind":"external_fact","title":"标准泰勒与强凸不等式","source_units":[],"declared_external":true,"declaration_text":["For twice differentiable f with mI <= Hessian f <= MI, Taylor's quadratic bounds hold, f(x)-f(x*) >= m||x-x*||^2/2, and ||gradient f(x)||^2 >= 2m(f(x)-f(x*)). An L-Lipschitz Hessian gives a Taylor remainder bounded by L||p||^3/6 and gradient f(x)-gradient f(y) = integral_0^1 Hessian f(y+s(x-y))(x-y) ds."]},{"id":"S083","semantic_type":"external_fact","kind":"external_fact","title":"标准谱序与范数运算","source_units":[],"declared_external":true,"declaration_text":["Positive-definite matrix order is preserved by congruence and reversed by inversion. Spectral norms are submultiplicative; mI <= A <= MI implies sqrt(m)||v|| <= ||v||_A <= sqrt(M)||v||. For symmetric E, E^p <= ||E||^p I in positive-semidefinite order. The scalar exponential power series and I+A <= exp(A) for symmetric A yield the matrix exponential estimates used here."]},{"id":"S084","semantic_type":"external_fact","kind":"external_fact","title":"联合界与初等级数估计","source_units":[],"declared_external":true,"declaration_text":["For countably many events, the probability of their union is at most the sum of their probabilities; sum_{k>=1} 1/k^2 = pi^2/6. For |q|<1, sum_{j>=0} q^j = 1/(1-q). Elementary logarithmic inequalities include log(1/(1-a)) >= a for 0<a<1 and x >= 2 log x for x>0."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S006","target_id":"S002","kind":"supports"},{"source_id":"S007","target_id":"S002","kind":"supports"},{"source_id":"S008","target_id":"S002","kind":"supports"},{"source_id":"S009","target_id":"S002","kind":"supports"},{"source_id":"S010","target_id":"S002","kind":"supports"},{"source_id":"S011","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S012","target_id":"S007","kind":"supports"},{"source_id":"S012","target_id":"S011","kind":"supports"},{"source_id":"S013","target_id":"S005","kind":"supports"},{"source_id":"S013","target_id":"S009","kind":"supports"},{"source_id":"S014","target_id":"S023","kind":"supports"},{"source_id":"S014","target_id":"S028","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S015","kind":"supports"},{"source_id":"S018","target_id":"S015","kind":"supports"},{"source_id":"S019","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S020","target_id":"S018","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S028","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S081","target_id":"S022","kind":"supports"},{"source_id":"S023","target_id":"S081","kind":"supports"},{"source_id":"S024","target_id":"S081","kind":"supports"},{"source_id":"S025","target_id":"S004","kind":"supports"},{"source_id":"S025","target_id":"S005","kind":"supports"},{"source_id":"S025","target_id":"S038","kind":"supports"},{"source_id":"S025","target_id":"S039","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S029","target_id":"S026","kind":"supports"},{"source_id":"S030","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S007","kind":"supports"},{"source_id":"S029","target_id":"S011","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S032","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S035","target_id":"S004","kind":"supports"},{"source_id":"S035","target_id":"S005","kind":"supports"},{"source_id":"S035","target_id":"S045","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S039","target_id":"S036","kind":"supports"},{"source_id":"S040","target_id":"S036","kind":"supports"},{"source_id":"S041","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S042","target_id":"S004","kind":"supports"},{"source_id":"S042","target_id":"S006","kind":"supports"},{"source_id":"S042","target_id":"S009","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"},{"source_id":"S046","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S005","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S049","target_id":"S048","kind":"supports"},{"source_id":"S050","target_id":"S048","kind":"supports"},{"source_id":"S051","target_id":"S048","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S051","kind":"supports"},{"source_id":"S052","target_id":"S066","kind":"supports"},{"source_id":"S052","target_id":"S007","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S055","target_id":"S053","kind":"supports"},{"source_id":"S056","target_id":"S053","kind":"supports"},{"source_id":"S057","target_id":"S053","kind":"supports"},{"source_id":"S058","target_id":"S053","kind":"supports"},{"source_id":"S055","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S059","target_id":"S057","kind":"supports"},{"source_id":"S059","target_id":"S065","kind":"supports"},{"source_id":"S059","target_id":"S004","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S007","kind":"supports"},{"source_id":"S062","target_id":"S008","kind":"supports"},{"source_id":"S063","target_id":"S062","kind":"supports"},{"source_id":"S064","target_id":"S063","kind":"supports"},{"source_id":"S065","target_id":"S063","kind":"supports"},{"source_id":"S066","target_id":"S063","kind":"supports"},{"source_id":"S067","target_id":"S063","kind":"supports"},{"source_id":"S068","target_id":"S063","kind":"supports"},{"source_id":"S069","target_id":"S063","kind":"supports"},{"source_id":"S070","target_id":"S063","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S067","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S069","target_id":"S070","kind":"supports"},{"source_id":"S071","target_id":"S030","kind":"supports"},{"source_id":"S071","target_id":"S038","kind":"supports"},{"source_id":"S071","target_id":"S039","kind":"supports"},{"source_id":"S071","target_id":"S041","kind":"supports"},{"source_id":"S071","target_id":"S049","kind":"supports"},{"source_id":"S071","target_id":"S056","kind":"supports"},{"source_id":"S071","target_id":"S061","kind":"supports"},{"source_id":"S071","target_id":"S005","kind":"supports"},{"source_id":"S072","target_id":"S075","kind":"supports"},{"source_id":"S073","target_id":"S078","kind":"supports"},{"source_id":"S073","target_id":"S017","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S078","kind":"supports"},{"source_id":"S074","target_id":"S024","kind":"supports"},{"source_id":"S075","target_id":"S038","kind":"supports"},{"source_id":"S075","target_id":"S039","kind":"supports"},{"source_id":"S075","target_id":"S040","kind":"supports"},{"source_id":"S075","target_id":"S041","kind":"supports"},{"source_id":"S075","target_id":"S046","kind":"supports"},{"source_id":"S075","target_id":"S055","kind":"supports"},{"source_id":"S075","target_id":"S066","kind":"supports"},{"source_id":"S076","target_id":"S001","kind":"supports"},{"source_id":"S076","target_id":"S049","kind":"supports"},{"source_id":"S076","target_id":"S061","kind":"supports"},{"source_id":"S077","target_id":"S017","kind":"supports"},{"source_id":"S078","target_id":"S017","kind":"supports"},{"source_id":"S078","target_id":"S023","kind":"supports"},{"source_id":"S078","target_id":"S030","kind":"supports"},{"source_id":"S078","target_id":"S005","kind":"supports"},{"source_id":"S079","target_id":"S017","kind":"supports"},{"source_id":"S079","target_id":"S029","kind":"supports"},{"source_id":"S080","target_id":"S049","kind":"supports"},{"source_id":"S080","target_id":"S055","kind":"supports"},{"source_id":"S080","target_id":"S061","kind":"supports"},{"source_id":"S080","target_id":"S070","kind":"supports"},{"source_id":"S080","target_id":"S005","kind":"supports"},{"source_id":"S082","target_id":"S038","kind":"supports"},{"source_id":"S082","target_id":"S039","kind":"supports"},{"source_id":"S082","target_id":"S041","kind":"supports"},{"source_id":"S082","target_id":"S049","kind":"supports"},{"source_id":"S082","target_id":"S055","kind":"supports"},{"source_id":"S082","target_id":"S056","kind":"supports"},{"source_id":"S082","target_id":"S070","kind":"supports"},{"source_id":"S083","target_id":"S017","kind":"supports"},{"source_id":"S083","target_id":"S030","kind":"supports"},{"source_id":"S083","target_id":"S056","kind":"supports"},{"source_id":"S083","target_id":"S061","kind":"supports"},{"source_id":"S083","target_id":"S067","kind":"supports"},{"source_id":"S083","target_id":"S068","kind":"supports"},{"source_id":"S084","target_id":"S081","kind":"supports"},{"source_id":"S084","target_id":"S033","kind":"supports"},{"source_id":"S084","target_id":"S046","kind":"supports"},{"source_id":"S084","target_id":"S005","kind":"supports"}]}}
{"record_id":"2205.02045v1","paper":{"title":"Duality in convex stochastic optimization","authors":["Teemu Pennanen","Ari-Pekka Perkkiö"],"arxiv_id":"2205.02045","version":"v1","journal":"Mathematical Programming","doi":"10.1007/s10107-025-02216-1","abstract_url":"https://arxiv.org/abs/2205.02045v1","pdf_url":"https://arxiv.org/pdf/2205.02045v1"},"context":{"raw_tex":"Given a probability space $(\\Omega,\\F,P)$ with a filtration $\\FF$ (an increasing sequence of sub-$\\sigma$-algebras of $\\F$), consider the problem\r\n\\begin{equation}\\label{sp}\\tag{$SP$}\r\n  \\begin{aligned}\r\n    &\\minimize\\quad & & Ef(x,\\bar u):=\\int f(x(\\omega),\\bar u(\\omega),\\omega)dP(\\omega)\\quad\\ovr\\text{$x\\in\\N$}%\\\\\r\n%    &\\st & & x\\in\\N,\r\n  \\end{aligned}\r\n\\end{equation}\n\nwhere $\\N$ is a linear space of stochastic processes $x=(x_t)_{t=0}^T$ adapted to $\\FF$ (i.e., $x_t$ is $\\F_t$-measurable) and $\\bar u$ is a $\\reals^m$-valued random variable. % and $Ef$ is an integral functional on the space $L^0(\\Omega,\\F,P;\\reals^n\\times\\reals^m)$ of $\\reals^n\\times\\reals^m$-valued random variables. \n%\\[\n%Ef(x,u) := \\int_\\Omega f(x(\\omega),u(\\omega),\\omega)dP(\\omega).\n%\\]\r\nWe assume that $f$ is a {\\em convex normal integrand} on $\\reals^n\\times\\reals^m\\times\\Omega$, i.e.\\ $f(\\cdot,\\omega)$ is a closed convex function for every $\\omega\\in\\Omega$ and $\\omega\\mapsto\\epi f(\\cdot,\\omega)$ is an $\\F$-measurable set-valued mapping; see \\cite[Chapter~14]{rw98}. Here and in what follows, we define the integral of an extended real-valued random variable as $+\\infty$ unless its positive part is integrable. The integral of any extended real-valued measurable function is then a well defined extended real number so it follows that $Ef$ is a well-defined convex function on $L^0(\\reals^n\\times\\reals^m)$.\r\n\n\nLet $\\U$ and $\\Y$ be linear spaces of $\\reals^m$-valued random variables in separating duality under the bilinear form\n\\[\n\\langle u,y\\rangle := E[u\\cdot y].\n\\]\nThis means that $u\\cdot y\\in L^1$ for all $u\\in\\U$ and $y\\in\\Y$ and that for every nonzero $u\\in\\U$, there exists a $y\\in\\Y$ such that $\\langle u,y\\rangle\\ne 0$ and vice versa. As usual, we identify random variables that coincide almost surely so the elements of $\\U$ and $\\Y$ are actually equivalence classes of random variables that coincide almost surely. We will also assume that the spaces are {\\em decomposable} and {\\em solid}. Decomposability means that\n\\[\n1_Au+1_{\\Omega\\setminus A}u'\\in\\U\n\\]\nfor every $u\\in\\U$ and $u'\\in L^\\infty$ while solidity means that if $\\bar u\\in\\U$ and $u\\in L^0$ are such that $|u^i|\\le|\\bar u^i|$ almost surely for every $i=1,\\ldots,m$, then $u\\in\\U$; similarly for $\\Y$. Solidity implies that\n\\[\n\\U=\\U_1\\times\\cdots\\times\\U_m\\quad\\text{and}\\quad \\Y=\\Y_1\\times\\cdots\\times\\Y_m,\n\\]\nwhere $\\U_i$ and $\\Y_i$ are solid decomposable spaces of real-valued random variables in separating duality under the bilinear form $(u_i,y_i)\\mapsto E[u_iy_i]$. In particular,\n\\begin{equation}\\label{eq:solidci}\nu_iy_i\\in L^1\\quad\\text{and}\\quad \\langle u,y\\rangle = \\sum_{i=1}^mE[u_iy_i]\\quad\\forall u\\in\\U,y\\in\\Y.\n\\end{equation}\n\n\nGiven a topology on $\\U$, the corresponding topological dual of $\\U$ is the linear space of all continuous linear functionals on $\\U$. A topology is {\\em compatible} with the bilinear form on $\\U\\times\\Y$ if every continuous linear functional can be expressed in the form\n\\[\r\nu\\mapsto\\langle u,y\\rangle\r\n\\]\r\nfor some $y\\in\\Y$. Such topologies can be characterized in terms of the ``weak'' and ``Mackey'' topologies associated with the bilinear form. The {\\em weak topology} $\\sigma(\\U,\\Y)$ on $\\U$ is the topology generated by linear functionals $u\\mapsto\\langle u,y\\rangle$ where $y\\in\\Y$. Similarly for $\\Y$. The {\\em Mackey topology} is the topology generated by the sublinear functionals\r\n\\[\r\n\\sigma_D(u):=\\sup_{y\\in D}\\langle u,y\\rangle,\r\n\\]\r\nwhere $D\\subset\\Y$ is $\\sigma(\\Y,\\U)$-compact. Similarly for $\\Y$. Given a topology on $\\U$, the corresponding topological dual can be identified with $\\Y$ if and only if the topology is between $\\sigma(\\U,\\Y)$ and $\\tau(\\U,\\Y)$. If $\\U$ is Fr\\'echet (e.g.\\ Banach) and $\\Y$ is its topological dual, then the $\\sigma(\\Y,\\U)$-compact sets are the bounded sets in $\\Y$, so $\\tau(\\U,\\Y)$ is the strong topology; see \\cite{kn76}.\n\n\nLet $\\X$ and $\\V$ be decomposable solid spaces of $\\reals^n$-valued random variables in separating duality under the bilinear form\r\n\\[\r\n(x,v)\\mapsto E[x\\cdot v].\r\n\\]\r\n\n\nThis section studies convex integral functionals on paired decomposable spaces $\\U$ and $\\Y$ of random variables. More precisely, we take a normal integrand $h$ and study the integral functionals $Eh:\\U\\to\\ereals$ and $Eh^*:\\Y\\to\\ereals$ defined by\r\n\\[\r\nEh(u) := \\int_\\Omega h(u(\\omega),\\omega)dP(\\omega)\r\n\\]\r\nand\r\n\\[\r\nEh^*(y) := \\int_\\Omega h^*(y(\\omega),\\omega)dP(\\omega).\r\n\\]\r\n\n\n\\begin{equation}\\label{spx}\\tag{$SP_\\X$}\r\n  \\begin{aligned}\r\n    &\\minimize\\quad & & Ef(x,\\bar u):=\\int f(x(\\omega),\\bar u(\\omega),\\omega)dP(\\omega)\\quad\\ovr\\text{$x\\in\\X_a$}%\\\\\r\n%    &\\st & & x\\in\\N,\r\n  \\end{aligned}\r\n\\end{equation}\n\nwhere $\\X\\subset L^0(\\Omega,\\F,P;\\reals^n)$ is a solid decomposable space of random paths and\r\n\\[\r\n\\X_a:=\\X\\cap\\N.\r\n\\]\r\nWe will assume that the parameter $\\bar u$ belongs to another solid decomposable space $\\U\\subset L^0(\\Omega,\\F,P;\\reals^m)$ of random variables. The general theory of convex duality will give a dual problem and optimality conditions for \\eqref{spx}. Section~\\ref{sec:rel} will then extend these results to the original problem \\eqref{sp} where we optimize over general adapted strategies in $L^0$.\r\n\n\nWe embed \\eqref{spx} into the conjugate duality framework by introducing an additional parameter $z\\in\\X$ and the extended real-valued convex function $F$ on $\\X\\times\\X\\times\\U$ defined by\r\n\\[\r\nF(x,z,u) := Ef(x,u) + \\delta_\\N(x-z).\r\n\\]\r\nWe denote the associated optimum value function by\r\n\\[\r\n\\varphi(z,u):=\\inf_{x\\in\\X}\\{Ef(x,u)\\,|\\,x-z\\in\\N\\}.\r\n\\]\r\nWe assume that $\\X$ is in separating duality with a solid decomposable space $\\V\\subset L^0(\\Omega,\\F,P;\\reals^n)$ and that $\\U$ is in separating duality with a solid decomposable space $\\Y\\subset L^0(\\Omega,\\F,P;\\reals^m)$. The bilinear forms are the usual ones, i.e.\r\n\\[\r\n\\langle x,v\\rangle:=E[x\\cdot v]\\quad\\text{and}\\quad \\langle u,y\\rangle := E[u\\cdot y].\r\n\\]\r\nSolidity implies that\r\n\\[\r\n\\X = \\X_0\\times\\cdots\\times\\X_T\\quad\\text{and}\\quad \\V = \\V_0\\times\\cdots\\times\\V_T,\r\n\\]\r\nwhere $\\X_t$ and $\\V_t$ are solid decomposable spaces of $\\reals^{n_t}$-valued random variables in separating duality under the bilinear form $(x_t,v_t)\\mapsto E[x_t\\cdot v_t]$. It follows that\r\n\\[\r\n\\langle x,v\\rangle = \\sum_{t=0}^TE[x_t\\cdot v_t]\\quad\\forall x\\in\\X,\\ v\\in\\V\r\n\\]\r\nand\r\n\\[\r\n\\X_a = \\X_0(\\F_0)\\times\\cdots\\times\\X_T(\\F_T).\r\n\\]\r\n\n\nAccording to the general conjugate duality framework of \\cite{roc74}, the {\\em dual problem} is the concave maximization problem\r\n% \\begin{equation}\\tag{$D_u$}\r\n% \\minimize\\quad F^*(0,p,y)-\\langle u,y\\rangle\\quad\\ovr (p,y)\\in\\V\\times\\Y.\r\n% \\end{equation}\r\n\\begin{equation}\\tag{$D$}\\label{d}\r\n\\maximize\\quad \\langle\\bar  u,y\\rangle - F^*(0,p,y)\\quad\\ovr (p,y)\\in\\V\\times\\Y.\r\n\\end{equation}\r\nMore explicit forms will be given below. By definition, $\\varphi^*(p,y)=F^*(0,p,y)$, so the dual problem can be written as\r\n% \\[\r\n% \\minimize\\quad \\varphi^*(p,y)-\\langle u,y\\rangle\\quad\\ovr (p,y)\\in\\V\\times\\Y.\r\n% \\]\r\n\\[\r\n\\maximize\\quad \\langle\\bar  u,y\\rangle - \\varphi^*(p,y)\\quad\\ovr (p,y)\\in\\V\\times\\Y.\r\n\\]\r\n\n\n\\[\r\n\\X_a^\\perp := \\{v\\in\\V\\,|\\,\\langle x,v\\rangle = 0\\quad \\forall x\\in\\X_a\\}.\r\n\\]\r\n\n\n\\[\n\\N^\\perp:=\\{v\\in L^1\\mid\\langle x,v\\rangle=0\\quad \\forall x\\in\\N\\cap L^\\infty\\}.\n\\]\n\n\n{\\em Lagrangian integrand} $l:\\reals^n\\times\\reals^m\\times\\Omega\\to\\ereals$ defined by\r\n\\[\r\nl(x,y,\\omega) := \\inf_{u\\in\\reals^m}\\{f(x,u,\\omega)-u\\cdot y\\}.\r\n\\]\r\n\n\nGiven extended real-valued random variables $\\xi_1$ and $\\xi_2$, their pointwise sum $\\xi_2+\\xi_2$ is well-defined by the usual algebraic operations of the extended real-line $\\ereals$ except when one of them takes the value $+\\infty$ and  the other one $-\\infty$. In this exceptional case, we define the sum as $+\\infty$.\r\n","units":[{"start":5378,"end":5752,"start_line":150,"end_line":156,"raw_tex":"Given a probability space $(\\Omega,\\F,P)$ with a filtration $\\FF$ (an increasing sequence of sub-$\\sigma$-algebras of $\\F$), consider the problem\r\n\\begin{equation}\\label{sp}\\tag{$SP$}\r\n  \\begin{aligned}\r\n    &\\minimize\\quad & & Ef(x,\\bar u):=\\int f(x(\\omega),\\bar u(\\omega),\\omega)dP(\\omega)\\quad\\ovr\\text{$x\\in\\N$}%\\\\\r\n%    &\\st & & x\\in\\N,\r\n  \\end{aligned}\r\n\\end{equation}","sha256_utf8":"8cf6981c11a5d6aff02c6ea17a7283380bc2139c1a943292317e9c9a2bdde71c"},{"start":5754,"end":6779,"start_line":157,"end_line":161,"raw_tex":"where $\\N$ is a linear space of stochastic processes $x=(x_t)_{t=0}^T$ adapted to $\\FF$ (i.e., $x_t$ is $\\F_t$-measurable) and $\\bar u$ is a $\\reals^m$-valued random variable. % and $Ef$ is an integral functional on the space $L^0(\\Omega,\\F,P;\\reals^n\\times\\reals^m)$ of $\\reals^n\\times\\reals^m$-valued random variables. \n%\\[\n%Ef(x,u) := \\int_\\Omega f(x(\\omega),u(\\omega),\\omega)dP(\\omega).\n%\\]\r\nWe assume that $f$ is a {\\em convex normal integrand} on $\\reals^n\\times\\reals^m\\times\\Omega$, i.e.\\ $f(\\cdot,\\omega)$ is a closed convex function for every $\\omega\\in\\Omega$ and $\\omega\\mapsto\\epi f(\\cdot,\\omega)$ is an $\\F$-measurable set-valued mapping; see \\cite[Chapter~14]{rw98}. Here and in what follows, we define the integral of an extended real-valued random variable as $+\\infty$ unless its positive part is integrable. The integral of any extended real-valued measurable function is then a well defined extended real number so it follows that $Ef$ is a well-defined convex function on $L^0(\\reals^n\\times\\reals^m)$.\r\n","sha256_utf8":"03302020fa1692cf2e03a2e7d18723c80e1d041c29f59cfd59d955f3246e39df"},{"start":12460,"end":13770,"start_line":191,"end_line":206,"raw_tex":"Let $\\U$ and $\\Y$ be linear spaces of $\\reals^m$-valued random variables in separating duality under the bilinear form\n\\[\n\\langle u,y\\rangle := E[u\\cdot y].\n\\]\nThis means that $u\\cdot y\\in L^1$ for all $u\\in\\U$ and $y\\in\\Y$ and that for every nonzero $u\\in\\U$, there exists a $y\\in\\Y$ such that $\\langle u,y\\rangle\\ne 0$ and vice versa. As usual, we identify random variables that coincide almost surely so the elements of $\\U$ and $\\Y$ are actually equivalence classes of random variables that coincide almost surely. We will also assume that the spaces are {\\em decomposable} and {\\em solid}. Decomposability means that\n\\[\n1_Au+1_{\\Omega\\setminus A}u'\\in\\U\n\\]\nfor every $u\\in\\U$ and $u'\\in L^\\infty$ while solidity means that if $\\bar u\\in\\U$ and $u\\in L^0$ are such that $|u^i|\\le|\\bar u^i|$ almost surely for every $i=1,\\ldots,m$, then $u\\in\\U$; similarly for $\\Y$. Solidity implies that\n\\[\n\\U=\\U_1\\times\\cdots\\times\\U_m\\quad\\text{and}\\quad \\Y=\\Y_1\\times\\cdots\\times\\Y_m,\n\\]\nwhere $\\U_i$ and $\\Y_i$ are solid decomposable spaces of real-valued random variables in separating duality under the bilinear form $(u_i,y_i)\\mapsto E[u_iy_i]$. In particular,\n\\begin{equation}\\label{eq:solidci}\nu_iy_i\\in L^1\\quad\\text{and}\\quad \\langle u,y\\rangle = \\sum_{i=1}^mE[u_iy_i]\\quad\\forall u\\in\\U,y\\in\\Y.\n\\end{equation}\n","sha256_utf8":"62d7c02c07641b3f26c61e4ca9bd1732e99663402c89377979d3852f42ee30e5"},{"start":15256,"end":16430,"start_line":217,"end_line":225,"raw_tex":"Given a topology on $\\U$, the corresponding topological dual of $\\U$ is the linear space of all continuous linear functionals on $\\U$. A topology is {\\em compatible} with the bilinear form on $\\U\\times\\Y$ if every continuous linear functional can be expressed in the form\n\\[\r\nu\\mapsto\\langle u,y\\rangle\r\n\\]\r\nfor some $y\\in\\Y$. Such topologies can be characterized in terms of the ``weak'' and ``Mackey'' topologies associated with the bilinear form. The {\\em weak topology} $\\sigma(\\U,\\Y)$ on $\\U$ is the topology generated by linear functionals $u\\mapsto\\langle u,y\\rangle$ where $y\\in\\Y$. Similarly for $\\Y$. The {\\em Mackey topology} is the topology generated by the sublinear functionals\r\n\\[\r\n\\sigma_D(u):=\\sup_{y\\in D}\\langle u,y\\rangle,\r\n\\]\r\nwhere $D\\subset\\Y$ is $\\sigma(\\Y,\\U)$-compact. Similarly for $\\Y$. Given a topology on $\\U$, the corresponding topological dual can be identified with $\\Y$ if and only if the topology is between $\\sigma(\\U,\\Y)$ and $\\tau(\\U,\\Y)$. If $\\U$ is Fr\\'echet (e.g.\\ Banach) and $\\Y$ is its topological dual, then the $\\sigma(\\Y,\\U)$-compact sets are the bounded sets in $\\Y$, so $\\tau(\\U,\\Y)$ is the strong topology; see \\cite{kn76}.\n","sha256_utf8":"4312dc95edfda74648bc2b709f980943dba2e11edac8f719eafd48c68bfdd549"},{"start":18928,"end":19095,"start_line":266,"end_line":269,"raw_tex":"Let $\\X$ and $\\V$ be decomposable solid spaces of $\\reals^n$-valued random variables in separating duality under the bilinear form\r\n\\[\r\n(x,v)\\mapsto E[x\\cdot v].\r\n\\]\r\n","sha256_utf8":"f29dccde9d4a3f29a6a743efaaf0ac35c116f2f3f48378389264ff0731db3fdb"},{"start":20320,"end":20700,"start_line":303,"end_line":310,"raw_tex":"This section studies convex integral functionals on paired decomposable spaces $\\U$ and $\\Y$ of random variables. More precisely, we take a normal integrand $h$ and study the integral functionals $Eh:\\U\\to\\ereals$ and $Eh^*:\\Y\\to\\ereals$ defined by\r\n\\[\r\nEh(u) := \\int_\\Omega h(u(\\omega),\\omega)dP(\\omega)\r\n\\]\r\nand\r\n\\[\r\nEh^*(y) := \\int_\\Omega h^*(y(\\omega),\\omega)dP(\\omega).\r\n\\]\r\n","sha256_utf8":"849c95137ee2aeb2d765dffccf8a2a77b011f844d412aa785af82c93521bae5b"},{"start":24830,"end":25063,"start_line":403,"end_line":408,"raw_tex":"\\begin{equation}\\label{spx}\\tag{$SP_\\X$}\r\n  \\begin{aligned}\r\n    &\\minimize\\quad & & Ef(x,\\bar u):=\\int f(x(\\omega),\\bar u(\\omega),\\omega)dP(\\omega)\\quad\\ovr\\text{$x\\in\\X_a$}%\\\\\r\n%    &\\st & & x\\in\\N,\r\n  \\end{aligned}\r\n\\end{equation}","sha256_utf8":"6a09b81011b2ebab5155645da12b5810e40a1d092f2698fcb143d5ed39e483f9"},{"start":25065,"end":25585,"start_line":409,"end_line":413,"raw_tex":"where $\\X\\subset L^0(\\Omega,\\F,P;\\reals^n)$ is a solid decomposable space of random paths and\r\n\\[\r\n\\X_a:=\\X\\cap\\N.\r\n\\]\r\nWe will assume that the parameter $\\bar u$ belongs to another solid decomposable space $\\U\\subset L^0(\\Omega,\\F,P;\\reals^m)$ of random variables. The general theory of convex duality will give a dual problem and optimality conditions for \\eqref{spx}. Section~\\ref{sec:rel} will then extend these results to the original problem \\eqref{sp} where we optimize over general adapted strategies in $L^0$.\r\n","sha256_utf8":"7413bf60253c5965fa63d0266f8f813769352d07e0e537271a2a18ed073ff577"},{"start":25587,"end":26764,"start_line":415,"end_line":438,"raw_tex":"We embed \\eqref{spx} into the conjugate duality framework by introducing an additional parameter $z\\in\\X$ and the extended real-valued convex function $F$ on $\\X\\times\\X\\times\\U$ defined by\r\n\\[\r\nF(x,z,u) := Ef(x,u) + \\delta_\\N(x-z).\r\n\\]\r\nWe denote the associated optimum value function by\r\n\\[\r\n\\varphi(z,u):=\\inf_{x\\in\\X}\\{Ef(x,u)\\,|\\,x-z\\in\\N\\}.\r\n\\]\r\nWe assume that $\\X$ is in separating duality with a solid decomposable space $\\V\\subset L^0(\\Omega,\\F,P;\\reals^n)$ and that $\\U$ is in separating duality with a solid decomposable space $\\Y\\subset L^0(\\Omega,\\F,P;\\reals^m)$. The bilinear forms are the usual ones, i.e.\r\n\\[\r\n\\langle x,v\\rangle:=E[x\\cdot v]\\quad\\text{and}\\quad \\langle u,y\\rangle := E[u\\cdot y].\r\n\\]\r\nSolidity implies that\r\n\\[\r\n\\X = \\X_0\\times\\cdots\\times\\X_T\\quad\\text{and}\\quad \\V = \\V_0\\times\\cdots\\times\\V_T,\r\n\\]\r\nwhere $\\X_t$ and $\\V_t$ are solid decomposable spaces of $\\reals^{n_t}$-valued random variables in separating duality under the bilinear form $(x_t,v_t)\\mapsto E[x_t\\cdot v_t]$. It follows that\r\n\\[\r\n\\langle x,v\\rangle = \\sum_{t=0}^TE[x_t\\cdot v_t]\\quad\\forall x\\in\\X,\\ v\\in\\V\r\n\\]\r\nand\r\n\\[\r\n\\X_a = \\X_0(\\F_0)\\times\\cdots\\times\\X_T(\\F_T).\r\n\\]\r\n","sha256_utf8":"7e0a69316bc3837983a07c08ada6e986c00c725ffa03f0358f23e6aab255694c"},{"start":26766,"end":27467,"start_line":440,"end_line":453,"raw_tex":"According to the general conjugate duality framework of \\cite{roc74}, the {\\em dual problem} is the concave maximization problem\r\n% \\begin{equation}\\tag{$D_u$}\r\n% \\minimize\\quad F^*(0,p,y)-\\langle u,y\\rangle\\quad\\ovr (p,y)\\in\\V\\times\\Y.\r\n% \\end{equation}\r\n\\begin{equation}\\tag{$D$}\\label{d}\r\n\\maximize\\quad \\langle\\bar  u,y\\rangle - F^*(0,p,y)\\quad\\ovr (p,y)\\in\\V\\times\\Y.\r\n\\end{equation}\r\nMore explicit forms will be given below. By definition, $\\varphi^*(p,y)=F^*(0,p,y)$, so the dual problem can be written as\r\n% \\[\r\n% \\minimize\\quad \\varphi^*(p,y)-\\langle u,y\\rangle\\quad\\ovr (p,y)\\in\\V\\times\\Y.\r\n% \\]\r\n\\[\r\n\\maximize\\quad \\langle\\bar  u,y\\rangle - \\varphi^*(p,y)\\quad\\ovr (p,y)\\in\\V\\times\\Y.\r\n\\]\r\n","sha256_utf8":"838c948c4cb5c68fbd665971f6be0c2ec3709d43d18f597da19681d128593f47"},{"start":32176,"end":32260,"start_line":563,"end_line":565,"raw_tex":"\\[\r\n\\X_a^\\perp := \\{v\\in\\V\\,|\\,\\langle x,v\\rangle = 0\\quad \\forall x\\in\\X_a\\}.\r\n\\]\r\n","sha256_utf8":"9b32a8dc2777ab3c519394b59ccd29a37256540c4d928911559f296b7f79ba89"},{"start":33645,"end":33732,"start_line":599,"end_line":601,"raw_tex":"\\[\n\\N^\\perp:=\\{v\\in L^1\\mid\\langle x,v\\rangle=0\\quad \\forall x\\in\\N\\cap L^\\infty\\}.\n\\]\n","sha256_utf8":"f9e015411c266e84cf1955a6839f90c6989565613dea075313e394f49cdb0d6b"},{"start":39813,"end":39974,"start_line":698,"end_line":701,"raw_tex":"{\\em Lagrangian integrand} $l:\\reals^n\\times\\reals^m\\times\\Omega\\to\\ereals$ defined by\r\n\\[\r\nl(x,y,\\omega) := \\inf_{u\\in\\reals^m}\\{f(x,u,\\omega)-u\\cdot y\\}.\r\n\\]\r\n","sha256_utf8":"77ef3ac49379437cc3cf86bc28fe5cfc1653a32a4f7ee6ffd7d4fa84df0a3941"},{"start":108851,"end":109175,"start_line":2154,"end_line":2154,"raw_tex":"Given extended real-valued random variables $\\xi_1$ and $\\xi_2$, their pointwise sum $\\xi_2+\\xi_2$ is well-defined by the usual algebraic operations of the extended real-line $\\ereals$ except when one of them takes the value $+\\infty$ and  the other one $-\\infty$. In this exceptional case, we define the sum as $+\\infty$.\r\n","sha256_utf8":"764a98dd6e1ad2875f573ee52b8a8cc29fecaa66479c6b64716cf3242ff661ef"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\thlabel{thm:kktr}\r\nIf $\\dom Ef\\cap(\\X\\times\\U)\\ne\\emptyset$ and \\eqref{sp} and \\eqref{d} are feasible, then following are equivalent\r\n\\begin{enumerate}\r\n\\item\r\n$x$ solves \\eqref{sp}, $(p,y)$ solves \\eqref{d} and $\\inf\\eqref{sp}=\\sup\\eqref{d}$,\r\n\\item\r\n$x$ is feasible in \\eqref{sp}, $(p,y)$ is feasible in \\eqref{d} and\r\n\\begin{equation}\\label{eq:kkt}\r\n(p,y)\\in\\partial f(x,\\bar u)\\quad P\\text{-a.s.}\r\n\\end{equation}\r\n\\item\r\n$x$ is feasible in \\eqref{sp}, $(p,y)$ is feasible in \\eqref{d}  and\r\n\\begin{align*}\r\np\\in\\partial_x l(x,y),\\quad \\bar u\\in\\partial_y[-l](x,y)\\quad P\\text{-a.s.}\r\n\\end{align*}\r\n\\end{enumerate}\r\n\\end{theorem}","units":[{"start":52516,"end":53164,"start_line":975,"end_line":991,"raw_tex":"\\begin{theorem}\\thlabel{thm:kktr}\r\nIf $\\dom Ef\\cap(\\X\\times\\U)\\ne\\emptyset$ and \\eqref{sp} and \\eqref{d} are feasible, then following are equivalent\r\n\\begin{enumerate}\r\n\\item\r\n$x$ solves \\eqref{sp}, $(p,y)$ solves \\eqref{d} and $\\inf\\eqref{sp}=\\sup\\eqref{d}$,\r\n\\item\r\n$x$ is feasible in \\eqref{sp}, $(p,y)$ is feasible in \\eqref{d} and\r\n\\begin{equation}\\label{eq:kkt}\r\n(p,y)\\in\\partial f(x,\\bar u)\\quad P\\text{-a.s.}\r\n\\end{equation}\r\n\\item\r\n$x$ is feasible in \\eqref{sp}, $(p,y)$ is feasible in \\eqref{d}  and\r\n\\begin{align*}\r\np\\in\\partial_x l(x,y),\\quad \\bar u\\in\\partial_y[-l](x,y)\\quad P\\text{-a.s.}\r\n\\end{align*}\r\n\\end{enumerate}\r\n\\end{theorem}","sha256_utf8":"2c1517ac3716c79dec8960b94e072e444b8207d321f65d4715d6c59dd72f28bd"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\\thlabel{lem:rel}\r\nWe have $L^\\infty\\subseteq \\U \\subseteq L^1$ and $L^\\infty\\subseteq \\Y\\subseteq L^1$ and \r\n\\begin{align*}\r\n\\sigma(L^1,L^\\infty)|_\\U&\\subseteq\\sigma(\\U,\\Y),\\quad\\sigma(\\U,\\Y)|_{L^\\infty}\\subseteq\\sigma(L^\\infty,L^1),\\\\\r\n\\tau(L^1,L^\\infty)|_\\U&\\subseteq\\tau(\\U,\\Y),\\quad\\tau(\\U,\\Y)|_{L^\\infty}\\subseteq\\tau(L^\\infty,L^1).\r\n\\end{align*}\r\nThe $L^0$-topology on $\\U$ is weaker than $\\tau(\\U,\\Y)$.\r\n\\end{lemma}\n\n\\begin{theorem}\\thlabel{thm:cif0}\r\nIf $h$ is a convex normal integrand with $\\dom Eh\\ne\\emptyset$, then  \r\n\\[\r\n(Eh)^*=Eh^*.\r\n\\]\r\nMoreover, $y\\in\\partial Eh(u)$ if and only if $Eh(u)$ is finite and $y\\in\\partial h(u)$ almost surely.\r\n\\end{theorem}\n\n\\begin{proof}\r\nThe first claim follows by applying \\cite[Theorem~14.60]{rw98} to the normal integrand $h_y(u,\\omega):=h(u,\\omega)-u\\cdot y(\\omega)$, where $y\\in\\Y$. As to the second, we have $y\\in\\partial Eh(u)$ if and only if $Eh(u)$ is finite and $Eh(u)+(Eh)^*(y)=\\langle u,y\\rangle$. Since $(Eh)^*=Eh^*$ by the first part,  the equality holds, by Fenchel's inequality, if and only if\r\n\\[\r\nh(u)+h^*(y)=u\\cdot y\r\n\\]\r\nalmost surely. This means that $y\\in\\partial h(u)$ almost surely.\r\n\\end{proof}\n\n\\begin{corollary}\\thlabel{cor:cif}\r\nLet $h$ be a convex normal integrand. The following are equivalent\r\n\\begin{enumerate}\r\n\\item $\\dom Eh \\ne\\emptyset$ and  $\\dom E h^* \\ne\\emptyset$,\r\n\\item $Eh$ is proper and closed,\r\n%\\item $Eh^*$ is proper and lsc,\r\n\\item $\\dom Eh\\ne\\emptyset$ and there exists $y\\in\\Y$ and $\\alpha\\in L^1$ such that\r\n\\begin{align*}\r\nh(u,\\omega)\\ge u\\cdot y(\\omega)-\\alpha(\\omega)\r\n\\end{align*}\r\n\\end{enumerate}\r\nand imply that  $Eh$ and $E h^*$ are conjugates of each other and that $y\\in\\partial Eh(u)$ if and only if $y\\in\\partial h(u)$ almost surely.\r\n\\end{corollary}\n\n\\begin{proof}\r\nBy \\thref{thm:cif0}, 1 implies 2. Assuming 2, there exists $u\\in\\dom Eh$, $y\\in\\Y$ and $a\\in\\reals$ such that\r\n\\[\r\nEh(u)\\ge \\langle u,y\\rangle-a\\quad\\forall u\\in\\U\r\n\\]\r\nThus, by \\thref{thm:cif0},\r\n\\[\r\na \\ge (Eh)^*(y)=Eh^*(y).\r\n\\]\r\nBy Fenchel's inequality\r\n\\[\r\nh(u,\\omega)+h^*(y,\\omega)\\ge u\\cdot y,\r\n\\]\r\nso 3 holds with $\\alpha(\\omega)=h^*(y(\\omega),\\omega)$. If 3 holds, $Eh^*(y)\\le E\\alpha$, so 1 holds. By \\thref{thm:cif0}, $Eh$ and $Eh^*$ are conjugates of each other and $y\\in\\partial Eh(u)$ implies $y\\in\\partial h(u)$ almost surely. If $y\\in\\partial h(u)$, then $h(u)+h^*(y)=u\\cdot y$ almost surely, where each summand is integrable, since $Eh$ and $Eh^*$ are proper, so $Eh(u)+Eh^*(y)=\\langle u,y\\rangle$, which means that $y\\in\\partial Eh(u)$.\r\n\\end{proof}\n\n\\begin{lemma}\\thlabel{lem:dual}\r\nIf $\\dom Ef\\cap(\\X\\times\\U)\\ne\\emptyset$, then\r\n\\begin{align*}\r\nF^*(v,p,y) &= Ef^*(v+p,y)+\\delta_{\\X_a^\\perp}(p),\r\n\\end{align*}\r\nand, in particular, \r\n\\[\r\n\\varphi^*(p,y)= Ef^*(p,y)+\\delta_{\\X_a^\\perp}(p).\r\n\\]\r\nIf, in addition, $\\dom Ef^*\\cap(\\V\\times\\Y)\\ne\\emptyset$, then $F$ is proper and closed.\r\n\\end{lemma}\n\n\\begin{proof}\r\nBy the interchange rule \\cite[Theorem~14.60]{rw98},\r\n\\begin{align*}\r\nF^*(v,p,y) &= \\sup_{x\\in\\X,z\\in\\X,u\\in\\U}\\{\\langle x,v\\rangle + \\langle z,p\\rangle + \\langle u,y\\rangle - Ef(x,u) \\,|\\, x-z\\in\\X_a\\}\\\\\r\n&= \\sup_{x\\in \\X,z'\\in\\X,u\\in\\U}\\{E[x\\cdot(v+p) + u\\cdot y - f(x,u) - z'\\cdot p] \\,|\\, z'\\in\\X_a\\}\\\\\r\n&= Ef^*(v+p,y)+\\delta_{\\X_a^\\perp}(p).\r\n\\end{align*}\r\nWhen $\\dom Ef^*\\ne\\emptyset$, $Ef$ is proper and closed, by \\thref{cor:cif}, so $F$ is closed as a sum of proper and closed functions. Clearly, $F$ is proper.\r\n\\end{proof}\n\nAs an immediate corollary, we get the following.\r\n\n\n\\begin{theorem}\\thlabel{thm:dualproblem}\r\nIf $\\dom Ef\\cap(\\X\\times\\U)\\ne\\emptyset$, the dual problem \\eqref{d} can be written as\r\n\\begin{equation}\\label{d}\\tag{$D$}\r\n\\maximize\\quad \\langle\\bar u,y\\rangle - Ef^*(p,y)\\quad\\ovr\\quad (p,y)\\in\\X_a^\\perp\\times\\Y.\r\n\\end{equation}\r\n\\end{theorem}\n\n\\begin{lemma}\\thlabel{lem:vperp}\r\nThe set $\\X_a$ is $\\sigma(\\X,\\V)$-closed and\r\n%\\[\r\n%\\X_a = \\{x\\in\\X\\mid \\langle x,v\\rangle = 0\\quad\\forall v\\in\\X_a^\\perp\\}.\r\n%\\]\r\n%If $\\X$ and $\\V$ are solid [this has been already made a standing assumption??], then\r\n\\[\r\n\\X_a^\\perp = \\N^\\perp\\cap\\V = \\{v\\in\\V\\,|\\,E_tv_t=0\\quad t=0,\\ldots,T\\}.\r\n\\]\r\n\\end{lemma}\n\n\\begin{proof}\r\nSince $\\N$ $\\N$ is closed in $L^0$, Lemma~\\ref{lem:rel} implies that $\\X_a$ is closed in $\\tau(\\X,\\V)$ and thus, by convexity, also in $\\sigma(\\X,\\V)$. %The first equality thus follows from the bipolar theorem (see \\thref{cor:bpt}).\r\nSince\r\n\\[\r\n\\X_a = \\X_0(\\F_0)\\times\\cdots\\times\\X_T(\\F_T),\r\n\\]\r\nwe have $v\\in\\X_a^\\perp$ if and only if $E[x_t\\cdot v_t]=0$ for every $x_t\\in\\X_t(\\F_t)$. Here, $E[x_t\\cdot v_t]=E[x_t\\cdot (E_t v_t)]$, by \\thref{lem:ce}.\r\n\\end{proof}\n\n\\begin{lemma}\\thlabel{lem:perp}\r\nLet $x\\in\\N$ and $v\\in\\N^\\perp$. If $E[x\\cdot v]^+\\in L^1$, then $E[x\\cdot v]=0$.\r\n\\end{lemma}\n\n\\begin{proof}\r\nThe assumptions imply that $f$ is proper so the equivalence of 2 and 3 follows from \\cite[Theorem~37.5]{roc70a}. Let $x\\in \\N$ and $(p,y)\\in \\V\\times\\Y$ be feasible. By Fenchel's inequality,\r\n\\[\r\nf(x,u)+f^*(p,y)-\\bar u\\cdot y \\ge x\\cdot p\\quad P\\text{-a.s.}\r\n\\]\r\nso \r\n\\[\r\nEf(x,u)+E[f^*(p,y)-\\bar u\\cdot y]\\ge E[x\\cdot p]\r\n\\]\r\nand one holds as an equality if and only if the other one does. Equality in the former means that 2 holds. By Lemma~\\ref{lem:perp}, $E[x\\cdot p]=0$, so equality in the latter means that 1 holds.\r\n\\end{proof}\n\n\\begin{lemma}\\thlabel{lem:Esum}\r\nGiven extended real-valued random variables $\\xi_1$ and $\\xi_2$, we have\r\n\\[\r\nE[\\xi_1+\\xi_2]=E[\\xi_1]+E[\\xi_2]\r\n\\]\r\nunder any of the following:\r\n\\begin{enumerate}\r\n\\item\\label{int1}\r\n  $\\xi_1^+,\\xi_2^+\\in L^1$ or $\\xi_1^-,\\xi_2^-\\in L^1$.\r\n  \\item\\label{int2}\r\n  $\\xi_1\\in L^1$ or $\\xi_2\\in L^1$,\r\n\\item\\label{int4}\r\n  $\\xi_1$ or $\\xi_2$ is $\\{0,+\\infty\\}$-valued.\r\n\\end{enumerate}\r\n\\end{lemma}\n\n\\begin{lemma}\\thlabel{lem:ce}\r\nLet $\\xi_1$ and $\\xi_2$ be extended real-valued random variables.\r\n\\begin{enumerate}\r\n\\item\r\n  If $\\xi_1$ and $\\xi_2$ are quasi-integrable and satisfy any of the conditions in Lemma~\\ref{lem:Esum}, then $\\xi_1+\\xi_2$ is quasi-integrable and\r\n  \\[\r\n  E^\\G[\\xi_1+\\xi_2] = E^\\G [\\xi_1] + E^\\G[\\xi_2].\r\n  \\]\r\n\\item\r\n  If $\\xi_2$ and $(\\xi_1\\xi_2)$ are quasi-integrable, and $\\xi_1$ is $\\G$-measurable, then\r\n  \\[\r\n  E^\\G[\\xi_1\\xi_2] = \\xi_1E^\\G[\\xi_2].\r\n  \\]\r\n\\end{enumerate}\r\n\\end{lemma}","units":[{"start":16470,"end":16906,"start_line":229,"end_line":236,"raw_tex":"\\begin{lemma}\\thlabel{lem:rel}\r\nWe have $L^\\infty\\subseteq \\U \\subseteq L^1$ and $L^\\infty\\subseteq \\Y\\subseteq L^1$ and \r\n\\begin{align*}\r\n\\sigma(L^1,L^\\infty)|_\\U&\\subseteq\\sigma(\\U,\\Y),\\quad\\sigma(\\U,\\Y)|_{L^\\infty}\\subseteq\\sigma(L^\\infty,L^1),\\\\\r\n\\tau(L^1,L^\\infty)|_\\U&\\subseteq\\tau(\\U,\\Y),\\quad\\tau(\\U,\\Y)|_{L^\\infty}\\subseteq\\tau(L^\\infty,L^1).\r\n\\end{align*}\r\nThe $L^0$-topology on $\\U$ is weaker than $\\tau(\\U,\\Y)$.\r\n\\end{lemma}","sha256_utf8":"47adca9c37c62bddc6a36037bbb67711bab80dbb5edadae1dca72abae1f501cd"},{"start":20841,"end":21087,"start_line":314,"end_line":320,"raw_tex":"\\begin{theorem}\\thlabel{thm:cif0}\r\nIf $h$ is a convex normal integrand with $\\dom Eh\\ne\\emptyset$, then  \r\n\\[\r\n(Eh)^*=Eh^*.\r\n\\]\r\nMoreover, $y\\in\\partial Eh(u)$ if and only if $Eh(u)$ is finite and $y\\in\\partial h(u)$ almost surely.\r\n\\end{theorem}","sha256_utf8":"0bcbffa00a77091b273b0e69f6078b9a29bdd8410397676b22a3bd97d6421e76"},{"start":21091,"end":21587,"start_line":322,"end_line":328,"raw_tex":"\\begin{proof}\r\nThe first claim follows by applying \\cite[Theorem~14.60]{rw98} to the normal integrand $h_y(u,\\omega):=h(u,\\omega)-u\\cdot y(\\omega)$, where $y\\in\\Y$. As to the second, we have $y\\in\\partial Eh(u)$ if and only if $Eh(u)$ is finite and $Eh(u)+(Eh)^*(y)=\\langle u,y\\rangle$. Since $(Eh)^*=Eh^*$ by the first part,  the equality holds, by Fenchel's inequality, if and only if\r\n\\[\r\nh(u)+h^*(y)=u\\cdot y\r\n\\]\r\nalmost surely. This means that $y\\in\\partial h(u)$ almost surely.\r\n\\end{proof}","sha256_utf8":"36d9c9e97dfb0c08a714f03b80979b803c308225dfdf817cb3bbe0bc5d0a8a5e"},{"start":21591,"end":22182,"start_line":330,"end_line":342,"raw_tex":"\\begin{corollary}\\thlabel{cor:cif}\r\nLet $h$ be a convex normal integrand. The following are equivalent\r\n\\begin{enumerate}\r\n\\item $\\dom Eh \\ne\\emptyset$ and  $\\dom E h^* \\ne\\emptyset$,\r\n\\item $Eh$ is proper and closed,\r\n%\\item $Eh^*$ is proper and lsc,\r\n\\item $\\dom Eh\\ne\\emptyset$ and there exists $y\\in\\Y$ and $\\alpha\\in L^1$ such that\r\n\\begin{align*}\r\nh(u,\\omega)\\ge u\\cdot y(\\omega)-\\alpha(\\omega)\r\n\\end{align*}\r\n\\end{enumerate}\r\nand imply that  $Eh$ and $E h^*$ are conjugates of each other and that $y\\in\\partial Eh(u)$ if and only if $y\\in\\partial h(u)$ almost surely.\r\n\\end{corollary}","sha256_utf8":"3dcf7aafbba948f0a2c771e4108f9c3eb983fb66ced0eec7eb4e1d95b767b70c"},{"start":22186,"end":22966,"start_line":344,"end_line":358,"raw_tex":"\\begin{proof}\r\nBy \\thref{thm:cif0}, 1 implies 2. Assuming 2, there exists $u\\in\\dom Eh$, $y\\in\\Y$ and $a\\in\\reals$ such that\r\n\\[\r\nEh(u)\\ge \\langle u,y\\rangle-a\\quad\\forall u\\in\\U\r\n\\]\r\nThus, by \\thref{thm:cif0},\r\n\\[\r\na \\ge (Eh)^*(y)=Eh^*(y).\r\n\\]\r\nBy Fenchel's inequality\r\n\\[\r\nh(u,\\omega)+h^*(y,\\omega)\\ge u\\cdot y,\r\n\\]\r\nso 3 holds with $\\alpha(\\omega)=h^*(y(\\omega),\\omega)$. If 3 holds, $Eh^*(y)\\le E\\alpha$, so 1 holds. By \\thref{thm:cif0}, $Eh$ and $Eh^*$ are conjugates of each other and $y\\in\\partial Eh(u)$ implies $y\\in\\partial h(u)$ almost surely. If $y\\in\\partial h(u)$, then $h(u)+h^*(y)=u\\cdot y$ almost surely, where each summand is integrable, since $Eh$ and $Eh^*$ are proper, so $Eh(u)+Eh^*(y)=\\langle u,y\\rangle$, which means that $y\\in\\partial Eh(u)$.\r\n\\end{proof}","sha256_utf8":"b8ac356e5b54dfb03a0f7a7cf8e109d38cc7b8079ff6c78f55ebe01ae72fb6bc"},{"start":32262,"end":32606,"start_line":567,"end_line":577,"raw_tex":"\\begin{lemma}\\thlabel{lem:dual}\r\nIf $\\dom Ef\\cap(\\X\\times\\U)\\ne\\emptyset$, then\r\n\\begin{align*}\r\nF^*(v,p,y) &= Ef^*(v+p,y)+\\delta_{\\X_a^\\perp}(p),\r\n\\end{align*}\r\nand, in particular, \r\n\\[\r\n\\varphi^*(p,y)= Ef^*(p,y)+\\delta_{\\X_a^\\perp}(p).\r\n\\]\r\nIf, in addition, $\\dom Ef^*\\cap(\\V\\times\\Y)\\ne\\emptyset$, then $F$ is proper and closed.\r\n\\end{lemma}","sha256_utf8":"7c2b47d05c5e030a3b5f7e0d4475168bb96af9c41b746a6d1b99fec27aec74d0"},{"start":32610,"end":33157,"start_line":579,"end_line":587,"raw_tex":"\\begin{proof}\r\nBy the interchange rule \\cite[Theorem~14.60]{rw98},\r\n\\begin{align*}\r\nF^*(v,p,y) &= \\sup_{x\\in\\X,z\\in\\X,u\\in\\U}\\{\\langle x,v\\rangle + \\langle z,p\\rangle + \\langle u,y\\rangle - Ef(x,u) \\,|\\, x-z\\in\\X_a\\}\\\\\r\n&= \\sup_{x\\in \\X,z'\\in\\X,u\\in\\U}\\{E[x\\cdot(v+p) + u\\cdot y - f(x,u) - z'\\cdot p] \\,|\\, z'\\in\\X_a\\}\\\\\r\n&= Ef^*(v+p,y)+\\delta_{\\X_a^\\perp}(p).\r\n\\end{align*}\r\nWhen $\\dom Ef^*\\ne\\emptyset$, $Ef$ is proper and closed, by \\thref{cor:cif}, so $F$ is closed as a sum of proper and closed functions. Clearly, $F$ is proper.\r\n\\end{proof}","sha256_utf8":"279b5b65430f7e0832e90469a4036d2ec55d6808493bb9c1b9706a4f7e8c8525"},{"start":33161,"end":33211,"start_line":589,"end_line":589,"raw_tex":"As an immediate corollary, we get the following.\r\n","sha256_utf8":"7667c07cb4d8b48a1b82889fac93e1a665e632feb881fc6040b5291de10be6a1"},{"start":33213,"end":33501,"start_line":591,"end_line":596,"raw_tex":"\\begin{theorem}\\thlabel{thm:dualproblem}\r\nIf $\\dom Ef\\cap(\\X\\times\\U)\\ne\\emptyset$, the dual problem \\eqref{d} can be written as\r\n\\begin{equation}\\label{d}\\tag{$D$}\r\n\\maximize\\quad \\langle\\bar u,y\\rangle - Ef^*(p,y)\\quad\\ovr\\quad (p,y)\\in\\X_a^\\perp\\times\\Y.\r\n\\end{equation}\r\n\\end{theorem}","sha256_utf8":"abbe0ff7a0ba16047910ebb6c21cc4697d0533d13f2e375c84020c092c7f7790"},{"start":33733,"end":34079,"start_line":603,"end_line":612,"raw_tex":"\\begin{lemma}\\thlabel{lem:vperp}\r\nThe set $\\X_a$ is $\\sigma(\\X,\\V)$-closed and\r\n%\\[\r\n%\\X_a = \\{x\\in\\X\\mid \\langle x,v\\rangle = 0\\quad\\forall v\\in\\X_a^\\perp\\}.\r\n%\\]\r\n%If $\\X$ and $\\V$ are solid [this has been already made a standing assumption??], then\r\n\\[\r\n\\X_a^\\perp = \\N^\\perp\\cap\\V = \\{v\\in\\V\\,|\\,E_tv_t=0\\quad t=0,\\ldots,T\\}.\r\n\\]\r\n\\end{lemma}","sha256_utf8":"16d0cf6e0050d50106c701b1155cb45a74981d52d25e720965a7f3746f2b071a"},{"start":34083,"end":34563,"start_line":614,"end_line":621,"raw_tex":"\\begin{proof}\r\nSince $\\N$ $\\N$ is closed in $L^0$, Lemma~\\ref{lem:rel} implies that $\\X_a$ is closed in $\\tau(\\X,\\V)$ and thus, by convexity, also in $\\sigma(\\X,\\V)$. %The first equality thus follows from the bipolar theorem (see \\thref{cor:bpt}).\r\nSince\r\n\\[\r\n\\X_a = \\X_0(\\F_0)\\times\\cdots\\times\\X_T(\\F_T),\r\n\\]\r\nwe have $v\\in\\X_a^\\perp$ if and only if $E[x_t\\cdot v_t]=0$ for every $x_t\\in\\X_t(\\F_t)$. Here, $E[x_t\\cdot v_t]=E[x_t\\cdot (E_t v_t)]$, by \\thref{lem:ce}.\r\n\\end{proof}","sha256_utf8":"fedb1de8989ac872c7dc680c22ff860e74169c2bbc7173b3306b6c8ca0409493"},{"start":49240,"end":49367,"start_line":917,"end_line":919,"raw_tex":"\\begin{lemma}\\thlabel{lem:perp}\r\nLet $x\\in\\N$ and $v\\in\\N^\\perp$. If $E[x\\cdot v]^+\\in L^1$, then $E[x\\cdot v]=0$.\r\n\\end{lemma}","sha256_utf8":"eeff3932a6477b3e44f134fa0b782ecc705cc9bb98198717058e103467b146cb"},{"start":53168,"end":53716,"start_line":993,"end_line":1003,"raw_tex":"\\begin{proof}\r\nThe assumptions imply that $f$ is proper so the equivalence of 2 and 3 follows from \\cite[Theorem~37.5]{roc70a}. Let $x\\in \\N$ and $(p,y)\\in \\V\\times\\Y$ be feasible. By Fenchel's inequality,\r\n\\[\r\nf(x,u)+f^*(p,y)-\\bar u\\cdot y \\ge x\\cdot p\\quad P\\text{-a.s.}\r\n\\]\r\nso \r\n\\[\r\nEf(x,u)+E[f^*(p,y)-\\bar u\\cdot y]\\ge E[x\\cdot p]\r\n\\]\r\nand one holds as an equality if and only if the other one does. Equality in the former means that 2 holds. By Lemma~\\ref{lem:perp}, $E[x\\cdot p]=0$, so equality in the latter means that 1 holds.\r\n\\end{proof}","sha256_utf8":"855f6769a71ddd9bd6c4ab62dcf9e75e211ca7d32f2391a2fe850094f996b8e7"},{"start":109248,"end":109675,"start_line":2158,"end_line":2172,"raw_tex":"\\begin{lemma}\\thlabel{lem:Esum}\r\nGiven extended real-valued random variables $\\xi_1$ and $\\xi_2$, we have\r\n\\[\r\nE[\\xi_1+\\xi_2]=E[\\xi_1]+E[\\xi_2]\r\n\\]\r\nunder any of the following:\r\n\\begin{enumerate}\r\n\\item\\label{int1}\r\n  $\\xi_1^+,\\xi_2^+\\in L^1$ or $\\xi_1^-,\\xi_2^-\\in L^1$.\r\n  \\item\\label{int2}\r\n  $\\xi_1\\in L^1$ or $\\xi_2\\in L^1$,\r\n\\item\\label{int4}\r\n  $\\xi_1$ or $\\xi_2$ is $\\{0,+\\infty\\}$-valued.\r\n\\end{enumerate}\r\n\\end{lemma}","sha256_utf8":"34012ab7d27ead0849a00f0267f76c04b2ed4a5d8ce4b79da93a4c8fc3d0f3e6"},{"start":109679,"end":110194,"start_line":2174,"end_line":2188,"raw_tex":"\\begin{lemma}\\thlabel{lem:ce}\r\nLet $\\xi_1$ and $\\xi_2$ be extended real-valued random variables.\r\n\\begin{enumerate}\r\n\\item\r\n  If $\\xi_1$ and $\\xi_2$ are quasi-integrable and satisfy any of the conditions in Lemma~\\ref{lem:Esum}, then $\\xi_1+\\xi_2$ is quasi-integrable and\r\n  \\[\r\n  E^\\G[\\xi_1+\\xi_2] = E^\\G [\\xi_1] + E^\\G[\\xi_2].\r\n  \\]\r\n\\item\r\n  If $\\xi_2$ and $(\\xi_1\\xi_2)$ are quasi-integrable, and $\\xi_1$ is $\\G$-measurable, then\r\n  \\[\r\n  E^\\G[\\xi_1\\xi_2] = \\xi_1E^\\G[\\xi_2].\r\n  \\]\r\n\\end{enumerate}\r\n\\end{lemma}","sha256_utf8":"4dd7d0f2e056515e64ed1900098e90d7a8d778bad5f14b9218fcf2f0d306e0fb"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"一般适应策略的逐情景最优性","source_units":[{"start":52516,"end":53164,"start_line":975,"end_line":991,"raw_tex":"\\begin{theorem}\\thlabel{thm:kktr}\r\nIf $\\dom Ef\\cap(\\X\\times\\U)\\ne\\emptyset$ and \\eqref{sp} and \\eqref{d} are feasible, then following are equivalent\r\n\\begin{enumerate}\r\n\\item\r\n$x$ solves \\eqref{sp}, $(p,y)$ solves \\eqref{d} and $\\inf\\eqref{sp}=\\sup\\eqref{d}$,\r\n\\item\r\n$x$ is feasible in \\eqref{sp}, $(p,y)$ is feasible in \\eqref{d} and\r\n\\begin{equation}\\label{eq:kkt}\r\n(p,y)\\in\\partial f(x,\\bar u)\\quad P\\text{-a.s.}\r\n\\end{equation}\r\n\\item\r\n$x$ is feasible in \\eqref{sp}, $(p,y)$ is feasible in \\eqref{d}  and\r\n\\begin{align*}\r\np\\in\\partial_x l(x,y),\\quad \\bar u\\in\\partial_y[-l](x,y)\\quad P\\text{-a.s.}\r\n\\end{align*}\r\n\\end{enumerate}\r\n\\end{theorem}","sha256_utf8":"2c1517ac3716c79dec8960b94e072e444b8207d321f65d4715d6c59dd72f28bd"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"逐情景最优性定理证明","source_units":[{"start":53168,"end":53716,"start_line":993,"end_line":1003,"raw_tex":"\\begin{proof}\r\nThe assumptions imply that $f$ is proper so the equivalence of 2 and 3 follows from \\cite[Theorem~37.5]{roc70a}. Let $x\\in \\N$ and $(p,y)\\in \\V\\times\\Y$ be feasible. By Fenchel's inequality,\r\n\\[\r\nf(x,u)+f^*(p,y)-\\bar u\\cdot y \\ge x\\cdot p\\quad P\\text{-a.s.}\r\n\\]\r\nso \r\n\\[\r\nEf(x,u)+E[f^*(p,y)-\\bar u\\cdot y]\\ge E[x\\cdot p]\r\n\\]\r\nand one holds as an equality if and only if the other one does. Equality in the former means that 2 holds. By Lemma~\\ref{lem:perp}, $E[x\\cdot p]=0$, so equality in the latter means that 1 holds.\r\n\\end{proof}","sha256_utf8":"855f6769a71ddd9bd6c4ab62dcf9e75e211ca7d32f2391a2fe850094f996b8e7"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"部分共轭条件与芬切尔不等式","source_units":[{"start":53168,"end":53446,"start_line":993,"end_line":997,"raw_tex":"\\begin{proof}\r\nThe assumptions imply that $f$ is proper so the equivalence of 2 and 3 follows from \\cite[Theorem~37.5]{roc70a}. Let $x\\in \\N$ and $(p,y)\\in \\V\\times\\Y$ be feasible. By Fenchel's inequality,\r\n\\[\r\nf(x,u)+f^*(p,y)-\\bar u\\cdot y \\ge x\\cdot p\\quad P\\text{-a.s.}\r\n\\]\r\n","sha256_utf8":"547ea48bae3cef0384ae3417f8d137f8b6814415675dc28ccc82884f98cf0bdb"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"积分得到原始对偶界","source_units":[{"start":53446,"end":53509,"start_line":998,"end_line":1001,"raw_tex":"so \r\n\\[\r\nEf(x,u)+E[f^*(p,y)-\\bar u\\cdot y]\\ge E[x\\cdot p]\r\n\\]\r\n","sha256_utf8":"6676dc081ae569195ca47f6f7e09121518211bc3790243c3aa94c7a9667f959d"}]},{"id":"S005","semantic_type":"proof_step","kind":"proof_step","title":"消去正交项并判定等号","source_units":[{"start":53509,"end":53716,"start_line":1002,"end_line":1003,"raw_tex":"and one holds as an equality if and only if the other one does. Equality in the former means that 2 holds. By Lemma~\\ref{lem:perp}, $E[x\\cdot p]=0$, so equality in the latter means that 1 holds.\r\n\\end{proof}","sha256_utf8":"e05ce2c8e2669f91c9c660fd6f25baf2bef0c37508227d40f023d8360faa8843"}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","title":"定义域非空与双方可行","source_units":[{"start":52551,"end":52666,"start_line":976,"end_line":976,"raw_tex":"If $\\dom Ef\\cap(\\X\\times\\U)\\ne\\emptyset$ and \\eqref{sp} and \\eqref{d} are feasible, then following are equivalent\r\n","sha256_utf8":"433f0adef583b2416996fed7787222de7d1188db50c2df3d098589c93d305221"}]},{"id":"S007","semantic_type":"problem_setting","kind":"problem_setting","title":"凸随机优化与适应性","source_units":[{"start":5378,"end":5752,"start_line":150,"end_line":156,"raw_tex":"Given a probability space $(\\Omega,\\F,P)$ with a filtration $\\FF$ (an increasing sequence of sub-$\\sigma$-algebras of $\\F$), consider the problem\r\n\\begin{equation}\\label{sp}\\tag{$SP$}\r\n  \\begin{aligned}\r\n    &\\minimize\\quad & & Ef(x,\\bar u):=\\int f(x(\\omega),\\bar u(\\omega),\\omega)dP(\\omega)\\quad\\ovr\\text{$x\\in\\N$}%\\\\\r\n%    &\\st & & x\\in\\N,\r\n  \\end{aligned}\r\n\\end{equation}","sha256_utf8":"8cf6981c11a5d6aff02c6ea17a7283380bc2139c1a943292317e9c9a2bdde71c"},{"start":5754,"end":6779,"start_line":157,"end_line":161,"raw_tex":"where $\\N$ is a linear space of stochastic processes $x=(x_t)_{t=0}^T$ adapted to $\\FF$ (i.e., $x_t$ is $\\F_t$-measurable) and $\\bar u$ is a $\\reals^m$-valued random variable. % and $Ef$ is an integral functional on the space $L^0(\\Omega,\\F,P;\\reals^n\\times\\reals^m)$ of $\\reals^n\\times\\reals^m$-valued random variables. \n%\\[\n%Ef(x,u) := \\int_\\Omega f(x(\\omega),u(\\omega),\\omega)dP(\\omega).\n%\\]\r\nWe assume that $f$ is a {\\em convex normal integrand} on $\\reals^n\\times\\reals^m\\times\\Omega$, i.e.\\ $f(\\cdot,\\omega)$ is a closed convex function for every $\\omega\\in\\Omega$ and $\\omega\\mapsto\\epi f(\\cdot,\\omega)$ is an $\\F$-measurable set-valued mapping; see \\cite[Chapter~14]{rw98}. Here and in what follows, we define the integral of an extended real-valued random variable as $+\\infty$ unless its positive part is integrable. The integral of any extended real-valued measurable function is then a well defined extended real number so it follows that $Ef$ is a well-defined convex function on $L^0(\\reals^n\\times\\reals^m)$.\r\n","sha256_utf8":"03302020fa1692cf2e03a2e7d18723c80e1d041c29f59cfd59d955f3246e39df"}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","title":"随机变量空间的分离对偶","source_units":[{"start":12460,"end":13770,"start_line":191,"end_line":206,"raw_tex":"Let $\\U$ and $\\Y$ be linear spaces of $\\reals^m$-valued random variables in separating duality under the bilinear form\n\\[\n\\langle u,y\\rangle := E[u\\cdot y].\n\\]\nThis means that $u\\cdot y\\in L^1$ for all $u\\in\\U$ and $y\\in\\Y$ and that for every nonzero $u\\in\\U$, there exists a $y\\in\\Y$ such that $\\langle u,y\\rangle\\ne 0$ and vice versa. As usual, we identify random variables that coincide almost surely so the elements of $\\U$ and $\\Y$ are actually equivalence classes of random variables that coincide almost surely. We will also assume that the spaces are {\\em decomposable} and {\\em solid}. Decomposability means that\n\\[\n1_Au+1_{\\Omega\\setminus A}u'\\in\\U\n\\]\nfor every $u\\in\\U$ and $u'\\in L^\\infty$ while solidity means that if $\\bar u\\in\\U$ and $u\\in L^0$ are such that $|u^i|\\le|\\bar u^i|$ almost surely for every $i=1,\\ldots,m$, then $u\\in\\U$; similarly for $\\Y$. Solidity implies that\n\\[\n\\U=\\U_1\\times\\cdots\\times\\U_m\\quad\\text{and}\\quad \\Y=\\Y_1\\times\\cdots\\times\\Y_m,\n\\]\nwhere $\\U_i$ and $\\Y_i$ are solid decomposable spaces of real-valued random variables in separating duality under the bilinear form $(u_i,y_i)\\mapsto E[u_iy_i]$. In particular,\n\\begin{equation}\\label{eq:solidci}\nu_iy_i\\in L^1\\quad\\text{and}\\quad \\langle u,y\\rangle = \\sum_{i=1}^mE[u_iy_i]\\quad\\forall u\\in\\U,y\\in\\Y.\n\\end{equation}\n","sha256_utf8":"62d7c02c07641b3f26c61e4ca9bd1732e99663402c89377979d3852f42ee30e5"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"相容拓扑与弱拓扑","source_units":[{"start":15256,"end":16430,"start_line":217,"end_line":225,"raw_tex":"Given a topology on $\\U$, the corresponding topological dual of $\\U$ is the linear space of all continuous linear functionals on $\\U$. A topology is {\\em compatible} with the bilinear form on $\\U\\times\\Y$ if every continuous linear functional can be expressed in the form\n\\[\r\nu\\mapsto\\langle u,y\\rangle\r\n\\]\r\nfor some $y\\in\\Y$. Such topologies can be characterized in terms of the ``weak'' and ``Mackey'' topologies associated with the bilinear form. The {\\em weak topology} $\\sigma(\\U,\\Y)$ on $\\U$ is the topology generated by linear functionals $u\\mapsto\\langle u,y\\rangle$ where $y\\in\\Y$. Similarly for $\\Y$. The {\\em Mackey topology} is the topology generated by the sublinear functionals\r\n\\[\r\n\\sigma_D(u):=\\sup_{y\\in D}\\langle u,y\\rangle,\r\n\\]\r\nwhere $D\\subset\\Y$ is $\\sigma(\\Y,\\U)$-compact. Similarly for $\\Y$. Given a topology on $\\U$, the corresponding topological dual can be identified with $\\Y$ if and only if the topology is between $\\sigma(\\U,\\Y)$ and $\\tau(\\U,\\Y)$. If $\\U$ is Fr\\'echet (e.g.\\ Banach) and $\\Y$ is its topological dual, then the $\\sigma(\\Y,\\U)$-compact sets are the bounded sets in $\\Y$, so $\\tau(\\U,\\Y)$ is the strong topology; see \\cite{kn76}.\n","sha256_utf8":"4312dc95edfda74648bc2b709f980943dba2e11edac8f719eafd48c68bfdd549"}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","title":"路径空间及其对偶","source_units":[{"start":18928,"end":19095,"start_line":266,"end_line":269,"raw_tex":"Let $\\X$ and $\\V$ be decomposable solid spaces of $\\reals^n$-valued random variables in separating duality under the bilinear form\r\n\\[\r\n(x,v)\\mapsto E[x\\cdot v].\r\n\\]\r\n","sha256_utf8":"f29dccde9d4a3f29a6a743efaaf0ac35c116f2f3f48378389264ff0731db3fdb"}]},{"id":"S011","semantic_type":"problem_setting","kind":"problem_setting","title":"可积策略限制问题","source_units":[{"start":24830,"end":25063,"start_line":403,"end_line":408,"raw_tex":"\\begin{equation}\\label{spx}\\tag{$SP_\\X$}\r\n  \\begin{aligned}\r\n    &\\minimize\\quad & & Ef(x,\\bar u):=\\int f(x(\\omega),\\bar u(\\omega),\\omega)dP(\\omega)\\quad\\ovr\\text{$x\\in\\X_a$}%\\\\\r\n%    &\\st & & x\\in\\N,\r\n  \\end{aligned}\r\n\\end{equation}","sha256_utf8":"6a09b81011b2ebab5155645da12b5810e40a1d092f2698fcb143d5ed39e483f9"},{"start":25065,"end":25585,"start_line":409,"end_line":413,"raw_tex":"where $\\X\\subset L^0(\\Omega,\\F,P;\\reals^n)$ is a solid decomposable space of random paths and\r\n\\[\r\n\\X_a:=\\X\\cap\\N.\r\n\\]\r\nWe will assume that the parameter $\\bar u$ belongs to another solid decomposable space $\\U\\subset L^0(\\Omega,\\F,P;\\reals^m)$ of random variables. The general theory of convex duality will give a dual problem and optimality conditions for \\eqref{spx}. Section~\\ref{sec:rel} will then extend these results to the original problem \\eqref{sp} where we optimize over general adapted strategies in $L^0$.\r\n","sha256_utf8":"7413bf60253c5965fa63d0266f8f813769352d07e0e537271a2a18ed073ff577"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"适应性扰动与值函数","source_units":[{"start":25587,"end":25939,"start_line":415,"end_line":422,"raw_tex":"We embed \\eqref{spx} into the conjugate duality framework by introducing an additional parameter $z\\in\\X$ and the extended real-valued convex function $F$ on $\\X\\times\\X\\times\\U$ defined by\r\n\\[\r\nF(x,z,u) := Ef(x,u) + \\delta_\\N(x-z).\r\n\\]\r\nWe denote the associated optimum value function by\r\n\\[\r\n\\varphi(z,u):=\\inf_{x\\in\\X}\\{Ef(x,u)\\,|\\,x-z\\in\\N\\}.\r\n\\]\r\n","sha256_utf8":"c6943b5fb58ef8b94c0fc45d4d26eebf5662f3b4272687d06db2fcd7316057dd"}]},{"id":"S013","semantic_type":"assumption","kind":"assumption","title":"双对偶配对与时间分解","source_units":[{"start":25939,"end":26764,"start_line":423,"end_line":438,"raw_tex":"We assume that $\\X$ is in separating duality with a solid decomposable space $\\V\\subset L^0(\\Omega,\\F,P;\\reals^n)$ and that $\\U$ is in separating duality with a solid decomposable space $\\Y\\subset L^0(\\Omega,\\F,P;\\reals^m)$. The bilinear forms are the usual ones, i.e.\r\n\\[\r\n\\langle x,v\\rangle:=E[x\\cdot v]\\quad\\text{and}\\quad \\langle u,y\\rangle := E[u\\cdot y].\r\n\\]\r\nSolidity implies that\r\n\\[\r\n\\X = \\X_0\\times\\cdots\\times\\X_T\\quad\\text{and}\\quad \\V = \\V_0\\times\\cdots\\times\\V_T,\r\n\\]\r\nwhere $\\X_t$ and $\\V_t$ are solid decomposable spaces of $\\reals^{n_t}$-valued random variables in separating duality under the bilinear form $(x_t,v_t)\\mapsto E[x_t\\cdot v_t]$. It follows that\r\n\\[\r\n\\langle x,v\\rangle = \\sum_{t=0}^TE[x_t\\cdot v_t]\\quad\\forall x\\in\\X,\\ v\\in\\V\r\n\\]\r\nand\r\n\\[\r\n\\X_a = \\X_0(\\F_0)\\times\\cdots\\times\\X_T(\\F_T).\r\n\\]\r\n","sha256_utf8":"5df03eecf8d139550c0992bbef138813a2d6e726d7b187e956cc9b784083760d"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"共轭形式的对偶问题","source_units":[{"start":26766,"end":27467,"start_line":440,"end_line":453,"raw_tex":"According to the general conjugate duality framework of \\cite{roc74}, the {\\em dual problem} is the concave maximization problem\r\n% \\begin{equation}\\tag{$D_u$}\r\n% \\minimize\\quad F^*(0,p,y)-\\langle u,y\\rangle\\quad\\ovr (p,y)\\in\\V\\times\\Y.\r\n% \\end{equation}\r\n\\begin{equation}\\tag{$D$}\\label{d}\r\n\\maximize\\quad \\langle\\bar  u,y\\rangle - F^*(0,p,y)\\quad\\ovr (p,y)\\in\\V\\times\\Y.\r\n\\end{equation}\r\nMore explicit forms will be given below. By definition, $\\varphi^*(p,y)=F^*(0,p,y)$, so the dual problem can be written as\r\n% \\[\r\n% \\minimize\\quad \\varphi^*(p,y)-\\langle u,y\\rangle\\quad\\ovr (p,y)\\in\\V\\times\\Y.\r\n% \\]\r\n\\[\r\n\\maximize\\quad \\langle\\bar  u,y\\rangle - \\varphi^*(p,y)\\quad\\ovr (p,y)\\in\\V\\times\\Y.\r\n\\]\r\n","sha256_utf8":"838c948c4cb5c68fbd665971f6be0c2ec3709d43d18f597da19681d128593f47"}]},{"id":"S015","semantic_type":"definition","kind":"definition","title":"适应路径的正交补","source_units":[{"start":32176,"end":32260,"start_line":563,"end_line":565,"raw_tex":"\\[\r\n\\X_a^\\perp := \\{v\\in\\V\\,|\\,\\langle x,v\\rangle = 0\\quad \\forall x\\in\\X_a\\}.\r\n\\]\r\n","sha256_utf8":"9b32a8dc2777ab3c519394b59ccd29a37256540c4d928911559f296b7f79ba89"}]},{"id":"S016","semantic_type":"definition","kind":"definition","title":"有界适应策略的消去空间","source_units":[{"start":33645,"end":33732,"start_line":599,"end_line":601,"raw_tex":"\\[\n\\N^\\perp:=\\{v\\in L^1\\mid\\langle x,v\\rangle=0\\quad \\forall x\\in\\N\\cap L^\\infty\\}.\n\\]\n","sha256_utf8":"f9e015411c266e84cf1955a6839f90c6989565613dea075313e394f49cdb0d6b"}]},{"id":"S017","semantic_type":"definition","kind":"definition","title":"拉格朗日被积函数","source_units":[{"start":39813,"end":39974,"start_line":698,"end_line":701,"raw_tex":"{\\em Lagrangian integrand} $l:\\reals^n\\times\\reals^m\\times\\Omega\\to\\ereals$ defined by\r\n\\[\r\nl(x,y,\\omega) := \\inf_{u\\in\\reals^m}\\{f(x,u,\\omega)-u\\cdot y\\}.\r\n\\]\r\n","sha256_utf8":"77ef3ac49379437cc3cf86bc28fe5cfc1653a32a4f7ee6ffd7d4fa84df0a3941"}]},{"id":"S018","semantic_type":"definition","kind":"definition","title":"积分泛函及其逐点共轭","source_units":[{"start":20320,"end":20700,"start_line":303,"end_line":310,"raw_tex":"This section studies convex integral functionals on paired decomposable spaces $\\U$ and $\\Y$ of random variables. More precisely, we take a normal integrand $h$ and study the integral functionals $Eh:\\U\\to\\ereals$ and $Eh^*:\\Y\\to\\ereals$ defined by\r\n\\[\r\nEh(u) := \\int_\\Omega h(u(\\omega),\\omega)dP(\\omega)\r\n\\]\r\nand\r\n\\[\r\nEh^*(y) := \\int_\\Omega h^*(y(\\omega),\\omega)dP(\\omega).\r\n\\]\r\n","sha256_utf8":"849c95137ee2aeb2d765dffccf8a2a77b011f844d412aa785af82c93521bae5b"}]},{"id":"S019","semantic_type":"definition","kind":"definition","title":"扩展实随机变量求和约定","source_units":[{"start":108851,"end":109175,"start_line":2154,"end_line":2154,"raw_tex":"Given extended real-valued random variables $\\xi_1$ and $\\xi_2$, their pointwise sum $\\xi_2+\\xi_2$ is well-defined by the usual algebraic operations of the extended real-line $\\ereals$ except when one of them takes the value $+\\infty$ and  the other one $-\\infty$. In this exceptional case, we define the sum as $+\\infty$.\r\n","sha256_utf8":"764a98dd6e1ad2875f573ee52b8a8cc29fecaa66479c6b64716cf3242ff661ef"}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","title":"空间嵌入与拓扑比较","source_units":[{"start":16470,"end":16906,"start_line":229,"end_line":236,"raw_tex":"\\begin{lemma}\\thlabel{lem:rel}\r\nWe have $L^\\infty\\subseteq \\U \\subseteq L^1$ and $L^\\infty\\subseteq \\Y\\subseteq L^1$ and \r\n\\begin{align*}\r\n\\sigma(L^1,L^\\infty)|_\\U&\\subseteq\\sigma(\\U,\\Y),\\quad\\sigma(\\U,\\Y)|_{L^\\infty}\\subseteq\\sigma(L^\\infty,L^1),\\\\\r\n\\tau(L^1,L^\\infty)|_\\U&\\subseteq\\tau(\\U,\\Y),\\quad\\tau(\\U,\\Y)|_{L^\\infty}\\subseteq\\tau(L^\\infty,L^1).\r\n\\end{align*}\r\nThe $L^0$-topology on $\\U$ is weaker than $\\tau(\\U,\\Y)$.\r\n\\end{lemma}","sha256_utf8":"47adca9c37c62bddc6a36037bbb67711bab80dbb5edadae1dca72abae1f501cd"}]},{"id":"S021","semantic_type":"theorem","kind":"theorem","title":"积分共轭与逐点次梯度","source_units":[{"start":20841,"end":21087,"start_line":314,"end_line":320,"raw_tex":"\\begin{theorem}\\thlabel{thm:cif0}\r\nIf $h$ is a convex normal integrand with $\\dom Eh\\ne\\emptyset$, then  \r\n\\[\r\n(Eh)^*=Eh^*.\r\n\\]\r\nMoreover, $y\\in\\partial Eh(u)$ if and only if $Eh(u)$ is finite and $y\\in\\partial h(u)$ almost surely.\r\n\\end{theorem}","sha256_utf8":"0bcbffa00a77091b273b0e69f6078b9a29bdd8410397676b22a3bd97d6421e76"}]},{"id":"S022","semantic_type":"proof","kind":"proof","title":"积分共轭定理证明","source_units":[{"start":21091,"end":21587,"start_line":322,"end_line":328,"raw_tex":"\\begin{proof}\r\nThe first claim follows by applying \\cite[Theorem~14.60]{rw98} to the normal integrand $h_y(u,\\omega):=h(u,\\omega)-u\\cdot y(\\omega)$, where $y\\in\\Y$. As to the second, we have $y\\in\\partial Eh(u)$ if and only if $Eh(u)$ is finite and $Eh(u)+(Eh)^*(y)=\\langle u,y\\rangle$. Since $(Eh)^*=Eh^*$ by the first part,  the equality holds, by Fenchel's inequality, if and only if\r\n\\[\r\nh(u)+h^*(y)=u\\cdot y\r\n\\]\r\nalmost surely. This means that $y\\in\\partial h(u)$ almost surely.\r\n\\end{proof}","sha256_utf8":"36d9c9e97dfb0c08a714f03b80979b803c308225dfdf817cb3bbe0bc5d0a8a5e"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"交换积分并表达次梯度等式","source_units":[{"start":21091,"end":21479,"start_line":322,"end_line":323,"raw_tex":"\\begin{proof}\r\nThe first claim follows by applying \\cite[Theorem~14.60]{rw98} to the normal integrand $h_y(u,\\omega):=h(u,\\omega)-u\\cdot y(\\omega)$, where $y\\in\\Y$. As to the second, we have $y\\in\\partial Eh(u)$ if and only if $Eh(u)$ is finite and $Eh(u)+(Eh)^*(y)=\\langle u,y\\rangle$. Since $(Eh)^*=Eh^*$ by the first part,  the equality holds, by Fenchel's inequality, if and only if\r\n","sha256_utf8":"2ed5fea6f0cc87daf34470e09fa5a0378d4fb2819e60633aa39cebb047923c08"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"逐点等号刻画次梯度","source_units":[{"start":21479,"end":21587,"start_line":324,"end_line":328,"raw_tex":"\\[\r\nh(u)+h^*(y)=u\\cdot y\r\n\\]\r\nalmost surely. This means that $y\\in\\partial h(u)$ almost surely.\r\n\\end{proof}","sha256_utf8":"debad7d194e23d6df8898584c616765e98d8f9d53253a0a2469affa1333ee6a2"}]},{"id":"S025","semantic_type":"corollary","kind":"corollary","title":"积分泛函闭性与仿射下界","source_units":[{"start":21591,"end":22182,"start_line":330,"end_line":342,"raw_tex":"\\begin{corollary}\\thlabel{cor:cif}\r\nLet $h$ be a convex normal integrand. The following are equivalent\r\n\\begin{enumerate}\r\n\\item $\\dom Eh \\ne\\emptyset$ and  $\\dom E h^* \\ne\\emptyset$,\r\n\\item $Eh$ is proper and closed,\r\n%\\item $Eh^*$ is proper and lsc,\r\n\\item $\\dom Eh\\ne\\emptyset$ and there exists $y\\in\\Y$ and $\\alpha\\in L^1$ such that\r\n\\begin{align*}\r\nh(u,\\omega)\\ge u\\cdot y(\\omega)-\\alpha(\\omega)\r\n\\end{align*}\r\n\\end{enumerate}\r\nand imply that  $Eh$ and $E h^*$ are conjugates of each other and that $y\\in\\partial Eh(u)$ if and only if $y\\in\\partial h(u)$ almost surely.\r\n\\end{corollary}","sha256_utf8":"3dcf7aafbba948f0a2c771e4108f9c3eb983fb66ced0eec7eb4e1d95b767b70c"}]},{"id":"S026","semantic_type":"proof","kind":"proof","title":"积分闭性等价条件证明","source_units":[{"start":22186,"end":22966,"start_line":344,"end_line":358,"raw_tex":"\\begin{proof}\r\nBy \\thref{thm:cif0}, 1 implies 2. Assuming 2, there exists $u\\in\\dom Eh$, $y\\in\\Y$ and $a\\in\\reals$ such that\r\n\\[\r\nEh(u)\\ge \\langle u,y\\rangle-a\\quad\\forall u\\in\\U\r\n\\]\r\nThus, by \\thref{thm:cif0},\r\n\\[\r\na \\ge (Eh)^*(y)=Eh^*(y).\r\n\\]\r\nBy Fenchel's inequality\r\n\\[\r\nh(u,\\omega)+h^*(y,\\omega)\\ge u\\cdot y,\r\n\\]\r\nso 3 holds with $\\alpha(\\omega)=h^*(y(\\omega),\\omega)$. If 3 holds, $Eh^*(y)\\le E\\alpha$, so 1 holds. By \\thref{thm:cif0}, $Eh$ and $Eh^*$ are conjugates of each other and $y\\in\\partial Eh(u)$ implies $y\\in\\partial h(u)$ almost surely. If $y\\in\\partial h(u)$, then $h(u)+h^*(y)=u\\cdot y$ almost surely, where each summand is integrable, since $Eh$ and $Eh^*$ are proper, so $Eh(u)+Eh^*(y)=\\langle u,y\\rangle$, which means that $y\\in\\partial Eh(u)$.\r\n\\end{proof}","sha256_utf8":"b8ac356e5b54dfb03a0f7a7cf8e109d38cc7b8079ff6c78f55ebe01ae72fb6bc"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"闭性导出共轭有限点","source_units":[{"start":22186,"end":22432,"start_line":344,"end_line":352,"raw_tex":"\\begin{proof}\r\nBy \\thref{thm:cif0}, 1 implies 2. Assuming 2, there exists $u\\in\\dom Eh$, $y\\in\\Y$ and $a\\in\\reals$ such that\r\n\\[\r\nEh(u)\\ge \\langle u,y\\rangle-a\\quad\\forall u\\in\\U\r\n\\]\r\nThus, by \\thref{thm:cif0},\r\n\\[\r\na \\ge (Eh)^*(y)=Eh^*(y).\r\n\\]\r\n","sha256_utf8":"edb1bdb472ed53eec2faf8fb59d69fd8ce958a88a4a0329c5e5edaa756eddba9"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"仿射下界与次梯度双向刻画","source_units":[{"start":22432,"end":22966,"start_line":353,"end_line":358,"raw_tex":"By Fenchel's inequality\r\n\\[\r\nh(u,\\omega)+h^*(y,\\omega)\\ge u\\cdot y,\r\n\\]\r\nso 3 holds with $\\alpha(\\omega)=h^*(y(\\omega),\\omega)$. If 3 holds, $Eh^*(y)\\le E\\alpha$, so 1 holds. By \\thref{thm:cif0}, $Eh$ and $Eh^*$ are conjugates of each other and $y\\in\\partial Eh(u)$ implies $y\\in\\partial h(u)$ almost surely. If $y\\in\\partial h(u)$, then $h(u)+h^*(y)=u\\cdot y$ almost surely, where each summand is integrable, since $Eh$ and $Eh^*$ are proper, so $Eh(u)+Eh^*(y)=\\langle u,y\\rangle$, which means that $y\\in\\partial Eh(u)$.\r\n\\end{proof}","sha256_utf8":"7c079830f6150d33d02f75796174bcc337982ee591143064abe53aaf670493e1"}]},{"id":"S029","semantic_type":"lemma","kind":"lemma","title":"扰动泛函的显式共轭","source_units":[{"start":32262,"end":32606,"start_line":567,"end_line":577,"raw_tex":"\\begin{lemma}\\thlabel{lem:dual}\r\nIf $\\dom Ef\\cap(\\X\\times\\U)\\ne\\emptyset$, then\r\n\\begin{align*}\r\nF^*(v,p,y) &= Ef^*(v+p,y)+\\delta_{\\X_a^\\perp}(p),\r\n\\end{align*}\r\nand, in particular, \r\n\\[\r\n\\varphi^*(p,y)= Ef^*(p,y)+\\delta_{\\X_a^\\perp}(p).\r\n\\]\r\nIf, in addition, $\\dom Ef^*\\cap(\\V\\times\\Y)\\ne\\emptyset$, then $F$ is proper and closed.\r\n\\end{lemma}","sha256_utf8":"7c2b47d05c5e030a3b5f7e0d4475168bb96af9c41b746a6d1b99fec27aec74d0"}]},{"id":"S030","semantic_type":"proof","kind":"proof","title":"扰动共轭公式证明","source_units":[{"start":32610,"end":33157,"start_line":579,"end_line":587,"raw_tex":"\\begin{proof}\r\nBy the interchange rule \\cite[Theorem~14.60]{rw98},\r\n\\begin{align*}\r\nF^*(v,p,y) &= \\sup_{x\\in\\X,z\\in\\X,u\\in\\U}\\{\\langle x,v\\rangle + \\langle z,p\\rangle + \\langle u,y\\rangle - Ef(x,u) \\,|\\, x-z\\in\\X_a\\}\\\\\r\n&= \\sup_{x\\in \\X,z'\\in\\X,u\\in\\U}\\{E[x\\cdot(v+p) + u\\cdot y - f(x,u) - z'\\cdot p] \\,|\\, z'\\in\\X_a\\}\\\\\r\n&= Ef^*(v+p,y)+\\delta_{\\X_a^\\perp}(p).\r\n\\end{align*}\r\nWhen $\\dom Ef^*\\ne\\emptyset$, $Ef$ is proper and closed, by \\thref{cor:cif}, so $F$ is closed as a sum of proper and closed functions. Clearly, $F$ is proper.\r\n\\end{proof}","sha256_utf8":"279b5b65430f7e0832e90469a4036d2ec55d6808493bb9c1b9706a4f7e8c8525"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"变量替换分离正交约束","source_units":[{"start":32610,"end":32986,"start_line":579,"end_line":585,"raw_tex":"\\begin{proof}\r\nBy the interchange rule \\cite[Theorem~14.60]{rw98},\r\n\\begin{align*}\r\nF^*(v,p,y) &= \\sup_{x\\in\\X,z\\in\\X,u\\in\\U}\\{\\langle x,v\\rangle + \\langle z,p\\rangle + \\langle u,y\\rangle - Ef(x,u) \\,|\\, x-z\\in\\X_a\\}\\\\\r\n&= \\sup_{x\\in \\X,z'\\in\\X,u\\in\\U}\\{E[x\\cdot(v+p) + u\\cdot y - f(x,u) - z'\\cdot p] \\,|\\, z'\\in\\X_a\\}\\\\\r\n&= Ef^*(v+p,y)+\\delta_{\\X_a^\\perp}(p).\r\n\\end{align*}\r\n","sha256_utf8":"327ddd9f936221fc4d5482814d422ebaf96134fbe1b9049c37796c0389b73482"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"验证扰动泛函正常闭性","source_units":[{"start":32986,"end":33157,"start_line":586,"end_line":587,"raw_tex":"When $\\dom Ef^*\\ne\\emptyset$, $Ef$ is proper and closed, by \\thref{cor:cif}, so $F$ is closed as a sum of proper and closed functions. Clearly, $F$ is proper.\r\n\\end{proof}","sha256_utf8":"85817ca2ce7331e7b01fa3cd5697e4243ff29192e43665234a4346895d62cafa"}]},{"id":"S033","semantic_type":"theorem","kind":"theorem","title":"双变量显式对偶问题","source_units":[{"start":33213,"end":33501,"start_line":591,"end_line":596,"raw_tex":"\\begin{theorem}\\thlabel{thm:dualproblem}\r\nIf $\\dom Ef\\cap(\\X\\times\\U)\\ne\\emptyset$, the dual problem \\eqref{d} can be written as\r\n\\begin{equation}\\label{d}\\tag{$D$}\r\n\\maximize\\quad \\langle\\bar u,y\\rangle - Ef^*(p,y)\\quad\\ovr\\quad (p,y)\\in\\X_a^\\perp\\times\\Y.\r\n\\end{equation}\r\n\\end{theorem}","sha256_utf8":"abbe0ff7a0ba16047910ebb6c21cc4697d0533d13f2e375c84020c092c7f7790"}]},{"id":"S034","semantic_type":"proof","kind":"proof","title":"显式对偶的直接推论","source_units":[{"start":33161,"end":33211,"start_line":589,"end_line":589,"raw_tex":"As an immediate corollary, we get the following.\r\n","sha256_utf8":"7667c07cb4d8b48a1b82889fac93e1a665e632feb881fc6040b5291de10be6a1"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"将共轭公式代入对偶","source_units":[{"start":33161,"end":33211,"start_line":589,"end_line":589,"raw_tex":"As an immediate corollary, we get the following.\r\n","sha256_utf8":"7667c07cb4d8b48a1b82889fac93e1a665e632feb881fc6040b5291de10be6a1"}]},{"id":"S036","semantic_type":"lemma","kind":"lemma","title":"正交补的条件期望刻画","source_units":[{"start":33733,"end":34079,"start_line":603,"end_line":612,"raw_tex":"\\begin{lemma}\\thlabel{lem:vperp}\r\nThe set $\\X_a$ is $\\sigma(\\X,\\V)$-closed and\r\n%\\[\r\n%\\X_a = \\{x\\in\\X\\mid \\langle x,v\\rangle = 0\\quad\\forall v\\in\\X_a^\\perp\\}.\r\n%\\]\r\n%If $\\X$ and $\\V$ are solid [this has been already made a standing assumption??], then\r\n\\[\r\n\\X_a^\\perp = \\N^\\perp\\cap\\V = \\{v\\in\\V\\,|\\,E_tv_t=0\\quad t=0,\\ldots,T\\}.\r\n\\]\r\n\\end{lemma}","sha256_utf8":"16d0cf6e0050d50106c701b1155cb45a74981d52d25e720965a7f3746f2b071a"}]},{"id":"S037","semantic_type":"proof","kind":"proof","title":"适应空间及正交补证明","source_units":[{"start":34083,"end":34563,"start_line":614,"end_line":621,"raw_tex":"\\begin{proof}\r\nSince $\\N$ $\\N$ is closed in $L^0$, Lemma~\\ref{lem:rel} implies that $\\X_a$ is closed in $\\tau(\\X,\\V)$ and thus, by convexity, also in $\\sigma(\\X,\\V)$. %The first equality thus follows from the bipolar theorem (see \\thref{cor:bpt}).\r\nSince\r\n\\[\r\n\\X_a = \\X_0(\\F_0)\\times\\cdots\\times\\X_T(\\F_T),\r\n\\]\r\nwe have $v\\in\\X_a^\\perp$ if and only if $E[x_t\\cdot v_t]=0$ for every $x_t\\in\\X_t(\\F_t)$. Here, $E[x_t\\cdot v_t]=E[x_t\\cdot (E_t v_t)]$, by \\thref{lem:ce}.\r\n\\end{proof}","sha256_utf8":"fedb1de8989ac872c7dc680c22ff860e74169c2bbc7173b3306b6c8ca0409493"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"概率闭性提升为弱闭性","source_units":[{"start":34083,"end":34332,"start_line":614,"end_line":615,"raw_tex":"\\begin{proof}\r\nSince $\\N$ $\\N$ is closed in $L^0$, Lemma~\\ref{lem:rel} implies that $\\X_a$ is closed in $\\tau(\\X,\\V)$ and thus, by convexity, also in $\\sigma(\\X,\\V)$. %The first equality thus follows from the bipolar theorem (see \\thref{cor:bpt}).\r\n","sha256_utf8":"ab27a105f74f9d8292c6b26d5fe414c01ecfa61ac023bf33c172d8004fb428a1"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"按时间分解并取条件期望","source_units":[{"start":34332,"end":34563,"start_line":616,"end_line":621,"raw_tex":"Since\r\n\\[\r\n\\X_a = \\X_0(\\F_0)\\times\\cdots\\times\\X_T(\\F_T),\r\n\\]\r\nwe have $v\\in\\X_a^\\perp$ if and only if $E[x_t\\cdot v_t]=0$ for every $x_t\\in\\X_t(\\F_t)$. Here, $E[x_t\\cdot v_t]=E[x_t\\cdot (E_t v_t)]$, by \\thref{lem:ce}.\r\n\\end{proof}","sha256_utf8":"a41bd491a543f0d1db574c22a7df955a66d2d83d966ca66e3e09545da3d93497"}]},{"id":"S040","semantic_type":"lemma","kind":"lemma","title":"无界适应策略的正交期望","source_units":[{"start":49240,"end":49367,"start_line":917,"end_line":919,"raw_tex":"\\begin{lemma}\\thlabel{lem:perp}\r\nLet $x\\in\\N$ and $v\\in\\N^\\perp$. If $E[x\\cdot v]^+\\in L^1$, then $E[x\\cdot v]=0$.\r\n\\end{lemma}","sha256_utf8":"eeff3932a6477b3e44f134fa0b782ecc705cc9bb98198717058e103467b146cb"}]},{"id":"S041","semantic_type":"lemma","kind":"lemma","title":"扩展期望的可加条件","source_units":[{"start":109248,"end":109675,"start_line":2158,"end_line":2172,"raw_tex":"\\begin{lemma}\\thlabel{lem:Esum}\r\nGiven extended real-valued random variables $\\xi_1$ and $\\xi_2$, we have\r\n\\[\r\nE[\\xi_1+\\xi_2]=E[\\xi_1]+E[\\xi_2]\r\n\\]\r\nunder any of the following:\r\n\\begin{enumerate}\r\n\\item\\label{int1}\r\n  $\\xi_1^+,\\xi_2^+\\in L^1$ or $\\xi_1^-,\\xi_2^-\\in L^1$.\r\n  \\item\\label{int2}\r\n  $\\xi_1\\in L^1$ or $\\xi_2\\in L^1$,\r\n\\item\\label{int4}\r\n  $\\xi_1$ or $\\xi_2$ is $\\{0,+\\infty\\}$-valued.\r\n\\end{enumerate}\r\n\\end{lemma}","sha256_utf8":"34012ab7d27ead0849a00f0267f76c04b2ed4a5d8ce4b79da93a4c8fc3d0f3e6"}]},{"id":"S042","semantic_type":"lemma","kind":"lemma","title":"条件期望的代数规则","source_units":[{"start":109679,"end":110194,"start_line":2174,"end_line":2188,"raw_tex":"\\begin{lemma}\\thlabel{lem:ce}\r\nLet $\\xi_1$ and $\\xi_2$ be extended real-valued random variables.\r\n\\begin{enumerate}\r\n\\item\r\n  If $\\xi_1$ and $\\xi_2$ are quasi-integrable and satisfy any of the conditions in Lemma~\\ref{lem:Esum}, then $\\xi_1+\\xi_2$ is quasi-integrable and\r\n  \\[\r\n  E^\\G[\\xi_1+\\xi_2] = E^\\G [\\xi_1] + E^\\G[\\xi_2].\r\n  \\]\r\n\\item\r\n  If $\\xi_2$ and $(\\xi_1\\xi_2)$ are quasi-integrable, and $\\xi_1$ is $\\G$-measurable, then\r\n  \\[\r\n  E^\\G[\\xi_1\\xi_2] = \\xi_1E^\\G[\\xi_2].\r\n  \\]\r\n\\end{enumerate}\r\n\\end{lemma}","sha256_utf8":"4dd7d0f2e056515e64ed1900098e90d7a8d778bad5f14b9218fcf2f0d306e0fb"}]},{"id":"S043","semantic_type":"external_fact","kind":"external_fact","title":"积分与优化交换定理","source_units":[],"declared_external":true,"declaration_text":["Rockafellar–Wets, Theorem 14.60: for a normal integrand on a decomposable space, the applicable interchange theorem identifies the infimum of the integral with the integral of the pointwise infimum, subject to its domain and integrability conditions."]},{"id":"S044","semantic_type":"external_fact","kind":"external_fact","title":"芬切尔不等式与等号条件","source_units":[],"declared_external":true,"declaration_text":["For a proper convex function g and its conjugate, g(a)+g*(b) is at least the dual pairing of a and b; equality is equivalent to b belonging to the convex subdifferential of g at a. A nonnegative measurable gap with zero integral vanishes almost surely."]},{"id":"S045","semantic_type":"external_fact","kind":"external_fact","title":"闭凸函数的双共轭表示","source_units":[],"declared_external":true,"declaration_text":["Fenchel–Moreau duality on a separating dual pair represents proper closed convex functions by their biconjugates. Such functions admit continuous affine minorants; a conjugate that is finite somewhere provides an affine lower bound and excludes negative infinite values."]},{"id":"S046","semantic_type":"external_fact","kind":"external_fact","title":"部分共轭的次梯度对应","source_units":[],"declared_external":true,"declaration_text":["Rockafellar, Convex Analysis, Theorem 37.5: for a proper closed convex function f and l(x,y)=inf_u{f(x,u)-u·y}, the condition (p,y) in the subdifferential of f at (x,u) is equivalent to p in the x-subdifferential of l at (x,y) and u in the y-subdifferential of -l there."]},{"id":"S047","semantic_type":"external_fact","kind":"external_fact","title":"随机空间的拓扑边界事实","source_units":[],"declared_external":true,"declaration_text":["The cited pp12 result supplies precisely the embeddings and topology comparisons in B0012. Adapted processes form a set closed under convergence in probability, and convex subsets of a separating dual pair have the same closed sets for the weak and Mackey topologies."]},{"id":"S048","semantic_type":"external_fact","kind":"external_fact","title":"适应策略消去引理","source_units":[],"declared_external":true,"declaration_text":["The result attributed to per16, also referenced through pp22, is the annihilator expectation lemma stated in B0126: its adaptedness, annihilator membership and stated positive-part integrability hypothesis imply that the pairing has expectation zero."]},{"id":"S049","semantic_type":"external_fact","kind":"external_fact","title":"扩展期望可加性","source_units":[],"declared_external":true,"declaration_text":["The cited pp22 result gives expectation additivity under each of the alternatives stated in B0322, using the paper's extended-real summation convention."]},{"id":"S050","semantic_type":"external_fact","kind":"external_fact","title":"广义条件期望规则","source_units":[],"declared_external":true,"declaration_text":["The cited pp22 result gives conditional expectation additivity and extraction of a measurable factor under exactly the quasi-integrability and integrability conditions stated in B0324; ordinary conditional expectation also has the tower property."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S006","target_id":"S003","kind":"supports"},{"source_id":"S006","target_id":"S004","kind":"supports"},{"source_id":"S007","target_id":"S003","kind":"supports"},{"source_id":"S017","target_id":"S003","kind":"supports"},{"source_id":"S033","target_id":"S004","kind":"supports"},{"source_id":"S036","target_id":"S005","kind":"supports"},{"source_id":"S040","target_id":"S005","kind":"supports"},{"source_id":"S016","target_id":"S005","kind":"supports"},{"source_id":"S044","target_id":"S003","kind":"supports"},{"source_id":"S044","target_id":"S005","kind":"supports"},{"source_id":"S046","target_id":"S003","kind":"supports"},{"source_id":"S041","target_id":"S004","kind":"supports"},{"source_id":"S013","target_id":"S004","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S043","target_id":"S023","kind":"supports"},{"source_id":"S044","target_id":"S023","kind":"supports"},{"source_id":"S044","target_id":"S024","kind":"supports"},{"source_id":"S008","target_id":"S023","kind":"supports"},{"source_id":"S018","target_id":"S023","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S021","target_id":"S027","kind":"supports"},{"source_id":"S021","target_id":"S028","kind":"supports"},{"source_id":"S045","target_id":"S027","kind":"supports"},{"source_id":"S044","target_id":"S028","kind":"supports"},{"source_id":"S008","target_id":"S027","kind":"supports"},{"source_id":"S018","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S030","kind":"supports"},{"source_id":"S043","target_id":"S031","kind":"supports"},{"source_id":"S010","target_id":"S031","kind":"supports"},{"source_id":"S011","target_id":"S031","kind":"supports"},{"source_id":"S012","target_id":"S031","kind":"supports"},{"source_id":"S013","target_id":"S031","kind":"supports"},{"source_id":"S015","target_id":"S031","kind":"supports"},{"source_id":"S025","target_id":"S032","kind":"supports"},{"source_id":"S036","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S029","target_id":"S035","kind":"supports"},{"source_id":"S014","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S037","kind":"supports"},{"source_id":"S020","target_id":"S038","kind":"supports"},{"source_id":"S047","target_id":"S038","kind":"supports"},{"source_id":"S009","target_id":"S038","kind":"supports"},{"source_id":"S011","target_id":"S038","kind":"supports"},{"source_id":"S020","target_id":"S039","kind":"supports"},{"source_id":"S042","target_id":"S039","kind":"supports"},{"source_id":"S010","target_id":"S039","kind":"supports"},{"source_id":"S013","target_id":"S039","kind":"supports"},{"source_id":"S015","target_id":"S039","kind":"supports"},{"source_id":"S016","target_id":"S039","kind":"supports"},{"source_id":"S047","target_id":"S020","kind":"supports"},{"source_id":"S008","target_id":"S020","kind":"supports"},{"source_id":"S009","target_id":"S020","kind":"supports"},{"source_id":"S048","target_id":"S040","kind":"supports"},{"source_id":"S049","target_id":"S041","kind":"supports"},{"source_id":"S019","target_id":"S041","kind":"supports"},{"source_id":"S050","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"}]}}
{"record_id":"2207.07470v2","paper":{"title":"Role of Subgradients in Variational Analysis of Polyhedral Functions","authors":["N. T. V. Hang","W. Jung","M. E. Sarabi"],"arxiv_id":"2207.07470","version":"v2","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-024-02378-6","abstract_url":"https://arxiv.org/abs/2207.07470v2","pdf_url":"https://arxiv.org/pdf/2207.07470v2"},"context":{"raw_tex":"In what follows,    we \n denote by $\\B$ the closed unit ball in the space in question and by $\\B_r(x):=x+r\\B$ the closed ball centered at $x$ with radius $r>0$. \n  In the  product space $\\R^n\\times \\R^m$, we use the norm $\\|(w,u)\\|=\\sqrt{\\|w\\|^2+\\|u\\|^2}$ for any $(w,u)\\in \\R^n\\times \\R^m$.\n Given a nonempty set $C\\subset\\R^n$, the symbols $\\inte C$,  $\\ri C$, $\\cone C$, and $\\co C$ signify its interior, relative interior, conic hull, and convex hull, respectively. \n For any set $C$ in $\\R^n$, its indicator function is defined by $\\dd_C(x)=0$ for $x\\in C$ and $\\dd_C(x)=\\infty$ otherwise. We denote\n by $P_C$ the projection mapping onto $C$ and  by $\\dist(x,C)$  the distance between $x\\in \\R^n$ and a set $C$.\n For a vector $w\\in \\R^n$, the subspace $\\{tw |\\, t\\in \\R\\}$ is denoted by $[w]$. The domain and range of a set-valued mapping $F:\\R^n\\tto\\R^m$ are defined, respectively, by  \n$ \\dom F:= \\{x\\in\\R^n\\big|\\;F(x)\\ne\\emp \\}$ and $\\rge F=\\{u\\in \\R^m|\\; \\exists \\,  w\\in \\R^n\\;\\,\\mbox{with}\\;\\; u\\in  F(w) \\}$.  \n\n\nIn this paper, the convergence of a family of sets is always understood in the sense of Painlev\\'e-Kuratowski (cf. \\cite[Definition~4.1]{rw}). \nThis means that the inner limit set of a parameterized family of sets $\\{C^t\\}_{t>0}$ in $\\R^d$, denoted $\\liminf_{t\\searrow 0} C^t$, is the set of points $x$ such that for every\nsequence $t_k \\searrow 0$,  $x$ is the limit of a sequence of points $x^{t_k}\\in C^{t_k}$. The outer limit set of this family of sets, \ndenoted $\\limsup_{t\\searrow 0} C^t$, is the set of points $x$ such that there exist  sequences\n $t_k \\searrow 0$ and $x^{t_k}\\in C^{t_k}$ such that  $x^{t_k}\\to x$ as $k\\to \\infty$. A sequence $\\{f^k\\b$ of functions $f^k:\\R^n\\to \\oR$ is said to {\\em epi-converge} to a function $f:\\R^n\\to \\oR$ if we have $\\epi f^k\\to \\epi f$ as $k\\to \\infty$,\n where $\\epi f=\\{(x,\\al)\\in \\R^n\\times \\R|\\, f(x)\\le \\al\\}$ is   the epigraph of $f$;\n see \\cite[Definition~7.1]{rw} for more details on the epi-convergence of \na sequence of extended-real-valued functions.  \n\n\nGiven a nonempty set $\\Omega\\subset\\R^n$ with $\\ox\\in \\Omega$, the {  tangent cone}  to $\\Omega$ at $\\ox$, denoted $T_\\Omega(\\ox)$,  is defined  by\n\\begin{equation}\\label{tan1}\nT_\\Omega(\\ox) = \\limsup_{t\\searrow 0} \\frac{\\Omega - \\ox}{t}.\n\\end{equation}\nThe  regular/Fr\\'{e}chet normal cone $\\rN_\\Omega(\\ox)$ to $\\Omega$ at $\\ox$ is defined by\n $\\rN_\\Omega(\\ox) = T_\\Omega(\\ox)^*$, the polar of the tangent cone \\eqref{tan1}. The  (limiting/Mordukhovich) normal cone $N_\\Omega(\\ox)$ to $\\Omega$ at $\\ox$ is\n the set of all vectors $\\ov\\in \\R^n$ for which there exist sequences  $\\{x^k\\b$ and  $\\{v^k\\b$ with $v^k\\in \\rN_\\Omega( x^k)$ such that \n$(x^k,v^k)\\to (\\ox,\\ov)$. When $\\Omega$ is convex, both normal cones boil down to that of convex analysis.  \nGiven a function $f:\\R^n \\to \\oR$ and a point $\\ox\\in\\R^n$ with $f(\\ox)$ finite, the subderivative function $\\d f(\\ox)\\colon\\R^n\\to\\oR$ is defined by\n\\begin{equation*}\\label{fsud}\n\\d f(\\ox)(w)=\\liminf_{\\substack{\n   t\\searrow 0 \\\\\n  w'\\to w\n  }} {\\frac{f(\\ox+tw')-f(\\ox)}{t}}.\n\\end{equation*}\nWhen $f$ is convex, its  subdifferential    at $\\ox$, denoted by $\\sub f(\\ox)$,  is understood in the sense of convex analysis (cf. \\cite[page~214]{Roc70}), namely $v\\in \\sub f(\\ox)$ if   $f(x)\\ge  f(\\ox)+\\la v,x-\\ox\\ra $ for any $x\\in \\R^n$.\n\n\nAs pointed earlier,  a proper   function $g:\\R^m\\to \\oR$ is called {\\em polyhedral}  \nif $\\epi g$ is a polyhedral convex set.  According to \\cite[Theorem~2.49]{rw}, this class of convex functions enjoys the  representation\n\\begin{equation*}\\label{polyf}\ng(z) = \\begin{cases}\n\\max\\limits_{j\\in J}\\big\\{\\la a^j, z \\ra - \\alpha_j\\big\\} \\quad& \\text{if } z \\in\\dom g,\\\\\n\\infty & \\text{otherwise,}\n\\end{cases}\n\\end{equation*}\nwhere $J= \\{1,\\ldots, l\\}$ for some $l\\in \\N$, $a^j\\in \\R^m$ and $\\alpha_j\\in \\R$  for all $j\\in J$, and where  $\\dom g=\\{z\\in \\R^m|\\, g(z)<\\infty\\}$ is a polyhedral convex set with the representation \n\\begin{equation}\\label{dom}\n\\dom g = \\big\\{z\\in\\R^m\\, \\big|\\, \\la b^i, z \\ra \\leq \\beta_i,  i\\in I=\\{1, \\ldots, s\\} \\big\\},\n\\end{equation}\nwhere $s\\in \\N$, and $b^i \\in \\R^m$ and $\\beta_i\\in \\R$ for all $i\\in I$. Thus, we can equivalently express a polyhedral function  $g$ as\n\\begin{equation}\\label{polyfunc}\ng(z) = \\max_{j\\in J}\\big\\{\\la a^j, z \\ra - \\alpha_j\\big\\} + \\delta_{\\dom g}(z), \\quad z\\in \\R^m.\n\\end{equation}\nIt was observed in \\cite[Proposition~3.2]{MoS16} that $\\dom g$ can be expressed as the finite  union of the polyhedral convex sets $C_j, j\\in J$, defined by\n\\begin{equation*}\\label{Cj}\nC_j =\\big\\{z\\in \\dom g\\, \\big|\\, g(z) = \\la a^j, z\\ra -\\alpha_j\\big\\} =\\big\\{z \\in \\dom g\\, \\big|\\, \\la a^i -a^j, z\\ra  \\leq \\alpha_i -\\alpha_j, i\\in J\\big\\}.\n\\end{equation*}\nPick $\\oz \\in\\dom g$ and  define the sets of active indices  at $\\oz$ corresponding to the representation \\eqref{dom}    and  {to the expression of $\\dom g$ via the finite union of $C_j$'s} by\n\\begin{equation}\\label{indset4}\nI(\\oz)= \\big\\{i\\in I\\, \\big|\\, \\la b^i, \\oz\\ra = \\beta_i\\big\\}\\quad\\textrm{ and }\\quad J(\\oz)= \\big\\{ j\\in J\\, \\big|\\, \\oz \\in C_j\\big\\}.\n\\end{equation}\nThese sets allow us to conclude from  \\cite[Proposition~3.3]{MoS16} that  the subdifferential of $g$ at $\\oz$ can be calculated as\n\\begin{equation}\\label{subg}\n\\partial g (\\oz) = \\co \\big\\{a^j \\, \\big|\\,  j\\in J(\\oz)\\big\\}+\\cone \\big\\{b^i\\, \\big|\\, i\\in I(\\oz)\\big\\},\n\\end{equation}\nwhich tells us that  any $ \\olm \\in   \\partial g(\\oz)$ can be written as  \n {\\begin{equation}\\label{decomp}\n\\olm = \\sum_{j\\in J(\\oz)} \\osigma_j a^j + \\sum_{i\\in I(\\oz)} \\otau_ib^i\\quad\n \\textrm{ with }\\; \\; \\osigma_j, \\otau_i \\geq 0\\; \\textrm{ and }\\;  \\sum_{j\\in J(\\oz)} \\osigma_j =1.\n\\end{equation}\nFor simplicity, we denote by $\\osigma \\in \\R^l$ and $\\otau \\in \\R^s$ the vectors with components $\\osigma_j$ and $\\otau_i$, respectively, where $\\osigma_j$  is taken from \\eqref{decomp} if   $j\\in J(\\oz)$ and  $\\osigma_j = 0$ if $j\\not\\in J(\\oz)$ and likewise, \n$\\otau_i$  is taken from \\eqref{decomp} if  $i\\in I(\\oz)$ and $\\otau_i =  0$ if $i\\notin I(\\oz)$. Pick the given representation of $\\olm$ in \\eqref{decomp} and define the sets of positive coefficients at $\\oz$ for $\\osigma$ and $\\otau$, respectively, by\n\\begin{equation}\\label{indset}\nJ_+(\\oz, \\osigma) = \\big\\{j\\in J(\\oz)\\, \\big|\\, \\osigma_j >0\\big\\}\\; \\; \\textrm{ and }\\; \\; I_+(\\oz, \\otau)=\\big\\{i\\in I(\\oz)\\, \\big|\\, \\otau_i>0\\big\\}.\n\\end{equation}\n\n\nRecall from \\cite[Exercise~6.47]{rw} that for any polyhedral convex set $C\\subset \\R^n$ and $\\ox\\in C$, we can find  a neighborhood $\\O$ of $0\\in \\R^n$ for which we have \n\\begin{equation}\\label{tanrep}\nT_C(\\ox)\\cap \\O=(C-\\ox)\\cap \\O.\n\\end{equation}\n\n\nGiven a function $f:\\R^n \\to \\oR$ and a point $\\ox\\in\\R^n$ with $f(\\ox)$ finite, the critical cone of $f$ at $\\ox$ for $\\bar v$ with $\\bar v\\in   \\sub f(\\ox)$ is defined by \n\\begin{equation*}\\label{cricone}\n{K_f}(\\ox,\\bar v)=\\big\\{w\\in \\R^n\\,\\big|\\,\\la\\bar v,w\\ra=\\d f(\\ox)(w)\\big\\}.\n\\end{equation*}\nIf, in addition, $f$ is convex and $\\partial f(\\ox)\\neq\\emp$, it follows from   \\cite[Theorem~8.30]{rw} that  its subderivative function is  the support function of $\\partial f(\\ox)$, that is\n\\begin{equation*}\\label{fsud2}\n\\d f(\\ox)(w) = \\sup\\big\\{\\la v, w\\ra\\, \\big|\\, v\\in \\partial f(\\ox)\\big\\},\n\\end{equation*}\nwhich in turn allows us to equivalently describe the critical cone ${K_f}(\\ox,\\bar v)$ as \n\\begin{equation}\\label{cricone1}\nK_f(\\ox, \\bar v) = N_{\\partial f(\\ox)}(\\bar v).\n\\end{equation}\n\n\nLet us recall an important result, used in the proof of next corollary, saying that the conjugate function $g^*$ of  a polyhedral function $g$ is again a polyhedral function, cf. \\cite[Theorem~11.14(a)]{rw}. \n\n\nRecall that   a closed face $F$ of a  polyhedral convex cone $C\\subset \\R^d$ is defined by \n$$\nF=C\\cap [v]^\\perp\\quad \\mbox{for some}\\;\\; v\\in C^*.\n$$\n\n\nTo this end, consider a set-valued mapping $F:\\R^n\\tto \\R^m$. According to \\cite[Definition~8.33]{rw}, the {\\em graphical derivative} of $F$ at $\\ox$ for $\\oy$ with $(\\ox,\\oy) \\in \\gph F$ is the set-valued mapping $DF(\\ox, \\oy): \\R^n\\tto \\R^m$ defined via the tangent cone to $\\gph F$ at $(\\ox, \\oy)$ by\n\\begin{equation*}\nu \\in DF(\\ox, \\oy)(w)\\; \\Longleftrightarrow\\; (w,u)\\in T_{\\gph F}(\\ox, \\oy),\n\\end{equation*}\nor, equivalently, $\\gph DF(\\ox, \\oy) = T_{\\gph F}(\\ox, \\oy)$.  Using the definition of the tangent cone, we can present an alternative definition of $DF(\\ox, \\oy)$ in terms of graphical limits as\n\\begin{equation}\\label{proto}\n\\gph DF(\\ox, \\oy) = \\limsup_{t\\searrow 0}\\frac{\\gph F-(\\ox,\\oy)}{t}.\n\\end{equation}\nThe set-valued mapping $F$ is said to be {\\em proto-differentiable} at $\\ox$ for $\\oy$ if the outer graphical limit in \\eqref{proto} is actually a full limit. If $F$ is proto-differentiable at $\\ox$ for $\\oy$,   its graphical derivative $DF(\\ox, \\oy)$ is   called the \n{\\em proto-derivative} of $F$ at $\\ox$ for $\\oy$. When $F(\\ox)$ is a singleton consisting of $\\oy$ only, the notation $DF(\\ox, \\oy)$ is simplified to $DF(\\ox)$. It is easy to see that \nfor a single-valued function $F$, which is differentiable at $\\ox$,  the graphical derivative  $DF(\\ox)$ boils down to the Jacobian matrix of $F$ at $\\ox$, denoted by $\\nabla F(\\ox)$. \nRecall from \\cite[Definition~9.53]{rw} that the strict graphical derivative \nof  a set-valued mapping $F$ at $\\ox$ for $\\oy$ with $(\\ox,\\oy)\\in \\gph F$, is  the set-valued mapping $D_*F(\\ox,\\oy):\\R^n\\tto \\R^m$, defined  by \n\\begin{equation}\\label{sproto}\n\\gph D_*F(\\ox,\\oy)=\\limsup_{\n  \\substack{\n   t\\searrow 0 \\\\\n  (x,y)\\xrightarrow{ \\gph F}(\\ox,\\oy)\n  }} \\frac{\\gph F-(x,y)}{t}.\n\\end{equation}\nThe set-valued mapping $F$ is said to be {\\em strictly} proto-differentiable at $\\ox$ for $\\oy$ if the outer graphical limit in \\eqref{sproto} is   a full limit.\n When $F$ is strictly proto-differentiable at $\\ox$ for $\\oy$, its strict graphical derivative $D_*F(\\ox, \\oy)$ is   called the {\\em strict} proto-derivative of $F$ at $\\ox$ for $\\oy$. \n To elaborate  more,  take a set  $\\Omega\\subset \\R^d$ and  $\\ox\\in \\Omega$. The regular (Clarke) tangent cone and the paratingent cone to   $\\Omega$ at $\\ox$ are defined, respectively,   by\n\\begin{equation}\\label{tancc}\n \\widehat T_{\\Omega}(\\ox) =\\liminf_{x \\xrightarrow{ \\Omega}\\ox,t\\searrow 0} \\frac{\\Omega-x}{t}\\quad \\mbox{and}\\quad  \\widetilde {T}_{\\Omega}(\\ox) =\\limsup_{x \\xrightarrow{ \\Omega}\\ox,t\\searrow 0} \\frac{\\Omega-x}{t}.\n\\end{equation}\nThe strict proto-differentiability of $F$ at $\\ox$ for $\\oy$ amounts to saying that $ \\widehat T_{\\gph F}(\\ox,\\oy)= \\widetilde {T}_{\\gph F}(\\ox,\\oy)$.\nObserve also that since the inclusions $ \\widehat T_{\\gph F}(\\ox,\\oy)\\subset  T_{\\gph F}(\\ox,\\oy)\\subset  \\widetilde {T}_{\\gph F}(\\ox,\\oy)$ always hold, \nstrict proto-differentiability of $F$ at $\\ox$ for $\\oy$ implies that  its proto-derivative and strict proto-derivative coincide, namely we have \n\\begin{equation}\\label{eqspd}\nD_*F(\\ox,\\oy)=DF(\\ox,\\oy).\n\\end{equation}\n","units":[{"start":11298,"end":12321,"start_line":236,"end_line":243,"raw_tex":"In what follows,    we \n denote by $\\B$ the closed unit ball in the space in question and by $\\B_r(x):=x+r\\B$ the closed ball centered at $x$ with radius $r>0$. \n  In the  product space $\\R^n\\times \\R^m$, we use the norm $\\|(w,u)\\|=\\sqrt{\\|w\\|^2+\\|u\\|^2}$ for any $(w,u)\\in \\R^n\\times \\R^m$.\n Given a nonempty set $C\\subset\\R^n$, the symbols $\\inte C$,  $\\ri C$, $\\cone C$, and $\\co C$ signify its interior, relative interior, conic hull, and convex hull, respectively. \n For any set $C$ in $\\R^n$, its indicator function is defined by $\\dd_C(x)=0$ for $x\\in C$ and $\\dd_C(x)=\\infty$ otherwise. We denote\n by $P_C$ the projection mapping onto $C$ and  by $\\dist(x,C)$  the distance between $x\\in \\R^n$ and a set $C$.\n For a vector $w\\in \\R^n$, the subspace $\\{tw |\\, t\\in \\R\\}$ is denoted by $[w]$. The domain and range of a set-valued mapping $F:\\R^n\\tto\\R^m$ are defined, respectively, by  \n$ \\dom F:= \\{x\\in\\R^n\\big|\\;F(x)\\ne\\emp \\}$ and $\\rge F=\\{u\\in \\R^m|\\; \\exists \\,  w\\in \\R^n\\;\\,\\mbox{with}\\;\\; u\\in  F(w) \\}$.  \n","sha256_utf8":"d9598f2682ba0d995b3dde3549505d0d17227b91faf6d5e13f47806980fe12d6"},{"start":12324,"end":13336,"start_line":246,"end_line":253,"raw_tex":"In this paper, the convergence of a family of sets is always understood in the sense of Painlev\\'e-Kuratowski (cf. \\cite[Definition~4.1]{rw}). \nThis means that the inner limit set of a parameterized family of sets $\\{C^t\\}_{t>0}$ in $\\R^d$, denoted $\\liminf_{t\\searrow 0} C^t$, is the set of points $x$ such that for every\nsequence $t_k \\searrow 0$,  $x$ is the limit of a sequence of points $x^{t_k}\\in C^{t_k}$. The outer limit set of this family of sets, \ndenoted $\\limsup_{t\\searrow 0} C^t$, is the set of points $x$ such that there exist  sequences\n $t_k \\searrow 0$ and $x^{t_k}\\in C^{t_k}$ such that  $x^{t_k}\\to x$ as $k\\to \\infty$. A sequence $\\{f^k\\b$ of functions $f^k:\\R^n\\to \\oR$ is said to {\\em epi-converge} to a function $f:\\R^n\\to \\oR$ if we have $\\epi f^k\\to \\epi f$ as $k\\to \\infty$,\n where $\\epi f=\\{(x,\\al)\\in \\R^n\\times \\R|\\, f(x)\\le \\al\\}$ is   the epigraph of $f$;\n see \\cite[Definition~7.1]{rw} for more details on the epi-convergence of \na sequence of extended-real-valued functions.  \n","sha256_utf8":"e130043e610f1dec0dec00700c744d17718b222c3028d0c6034da99ac6febcd1"},{"start":13337,"end":14627,"start_line":255,"end_line":270,"raw_tex":"Given a nonempty set $\\Omega\\subset\\R^n$ with $\\ox\\in \\Omega$, the {  tangent cone}  to $\\Omega$ at $\\ox$, denoted $T_\\Omega(\\ox)$,  is defined  by\n\\begin{equation}\\label{tan1}\nT_\\Omega(\\ox) = \\limsup_{t\\searrow 0} \\frac{\\Omega - \\ox}{t}.\n\\end{equation}\nThe  regular/Fr\\'{e}chet normal cone $\\rN_\\Omega(\\ox)$ to $\\Omega$ at $\\ox$ is defined by\n $\\rN_\\Omega(\\ox) = T_\\Omega(\\ox)^*$, the polar of the tangent cone \\eqref{tan1}. The  (limiting/Mordukhovich) normal cone $N_\\Omega(\\ox)$ to $\\Omega$ at $\\ox$ is\n the set of all vectors $\\ov\\in \\R^n$ for which there exist sequences  $\\{x^k\\b$ and  $\\{v^k\\b$ with $v^k\\in \\rN_\\Omega( x^k)$ such that \n$(x^k,v^k)\\to (\\ox,\\ov)$. When $\\Omega$ is convex, both normal cones boil down to that of convex analysis.  \nGiven a function $f:\\R^n \\to \\oR$ and a point $\\ox\\in\\R^n$ with $f(\\ox)$ finite, the subderivative function $\\d f(\\ox)\\colon\\R^n\\to\\oR$ is defined by\n\\begin{equation*}\\label{fsud}\n\\d f(\\ox)(w)=\\liminf_{\\substack{\n   t\\searrow 0 \\\\\n  w'\\to w\n  }} {\\frac{f(\\ox+tw')-f(\\ox)}{t}}.\n\\end{equation*}\nWhen $f$ is convex, its  subdifferential    at $\\ox$, denoted by $\\sub f(\\ox)$,  is understood in the sense of convex analysis (cf. \\cite[page~214]{Roc70}), namely $v\\in \\sub f(\\ox)$ if   $f(x)\\ge  f(\\ox)+\\la v,x-\\ox\\ra $ for any $x\\in \\R^n$.\n","sha256_utf8":"a5487dcda557f04cc261dee01c8f72ab1caa113190b3db8bd6afe823111e2c97"},{"start":14629,"end":17711,"start_line":273,"end_line":310,"raw_tex":"As pointed earlier,  a proper   function $g:\\R^m\\to \\oR$ is called {\\em polyhedral}  \nif $\\epi g$ is a polyhedral convex set.  According to \\cite[Theorem~2.49]{rw}, this class of convex functions enjoys the  representation\n\\begin{equation*}\\label{polyf}\ng(z) = \\begin{cases}\n\\max\\limits_{j\\in J}\\big\\{\\la a^j, z \\ra - \\alpha_j\\big\\} \\quad& \\text{if } z \\in\\dom g,\\\\\n\\infty & \\text{otherwise,}\n\\end{cases}\n\\end{equation*}\nwhere $J= \\{1,\\ldots, l\\}$ for some $l\\in \\N$, $a^j\\in \\R^m$ and $\\alpha_j\\in \\R$  for all $j\\in J$, and where  $\\dom g=\\{z\\in \\R^m|\\, g(z)<\\infty\\}$ is a polyhedral convex set with the representation \n\\begin{equation}\\label{dom}\n\\dom g = \\big\\{z\\in\\R^m\\, \\big|\\, \\la b^i, z \\ra \\leq \\beta_i,  i\\in I=\\{1, \\ldots, s\\} \\big\\},\n\\end{equation}\nwhere $s\\in \\N$, and $b^i \\in \\R^m$ and $\\beta_i\\in \\R$ for all $i\\in I$. Thus, we can equivalently express a polyhedral function  $g$ as\n\\begin{equation}\\label{polyfunc}\ng(z) = \\max_{j\\in J}\\big\\{\\la a^j, z \\ra - \\alpha_j\\big\\} + \\delta_{\\dom g}(z), \\quad z\\in \\R^m.\n\\end{equation}\nIt was observed in \\cite[Proposition~3.2]{MoS16} that $\\dom g$ can be expressed as the finite  union of the polyhedral convex sets $C_j, j\\in J$, defined by\n\\begin{equation*}\\label{Cj}\nC_j =\\big\\{z\\in \\dom g\\, \\big|\\, g(z) = \\la a^j, z\\ra -\\alpha_j\\big\\} =\\big\\{z \\in \\dom g\\, \\big|\\, \\la a^i -a^j, z\\ra  \\leq \\alpha_i -\\alpha_j, i\\in J\\big\\}.\n\\end{equation*}\nPick $\\oz \\in\\dom g$ and  define the sets of active indices  at $\\oz$ corresponding to the representation \\eqref{dom}    and  {to the expression of $\\dom g$ via the finite union of $C_j$'s} by\n\\begin{equation}\\label{indset4}\nI(\\oz)= \\big\\{i\\in I\\, \\big|\\, \\la b^i, \\oz\\ra = \\beta_i\\big\\}\\quad\\textrm{ and }\\quad J(\\oz)= \\big\\{ j\\in J\\, \\big|\\, \\oz \\in C_j\\big\\}.\n\\end{equation}\nThese sets allow us to conclude from  \\cite[Proposition~3.3]{MoS16} that  the subdifferential of $g$ at $\\oz$ can be calculated as\n\\begin{equation}\\label{subg}\n\\partial g (\\oz) = \\co \\big\\{a^j \\, \\big|\\,  j\\in J(\\oz)\\big\\}+\\cone \\big\\{b^i\\, \\big|\\, i\\in I(\\oz)\\big\\},\n\\end{equation}\nwhich tells us that  any $ \\olm \\in   \\partial g(\\oz)$ can be written as  \n {\\begin{equation}\\label{decomp}\n\\olm = \\sum_{j\\in J(\\oz)} \\osigma_j a^j + \\sum_{i\\in I(\\oz)} \\otau_ib^i\\quad\n \\textrm{ with }\\; \\; \\osigma_j, \\otau_i \\geq 0\\; \\textrm{ and }\\;  \\sum_{j\\in J(\\oz)} \\osigma_j =1.\n\\end{equation}\nFor simplicity, we denote by $\\osigma \\in \\R^l$ and $\\otau \\in \\R^s$ the vectors with components $\\osigma_j$ and $\\otau_i$, respectively, where $\\osigma_j$  is taken from \\eqref{decomp} if   $j\\in J(\\oz)$ and  $\\osigma_j = 0$ if $j\\not\\in J(\\oz)$ and likewise, \n$\\otau_i$  is taken from \\eqref{decomp} if  $i\\in I(\\oz)$ and $\\otau_i =  0$ if $i\\notin I(\\oz)$. Pick the given representation of $\\olm$ in \\eqref{decomp} and define the sets of positive coefficients at $\\oz$ for $\\osigma$ and $\\otau$, respectively, by\n\\begin{equation}\\label{indset}\nJ_+(\\oz, \\osigma) = \\big\\{j\\in J(\\oz)\\, \\big|\\, \\osigma_j >0\\big\\}\\; \\; \\textrm{ and }\\; \\; I_+(\\oz, \\otau)=\\big\\{i\\in I(\\oz)\\, \\big|\\, \\otau_i>0\\big\\}.\n\\end{equation}\n","sha256_utf8":"62a6e03ed814aaf1cfa2794eb681fe365bd05a1630ff110974eaaf132705b68d"},{"start":21977,"end":22226,"start_line":364,"end_line":367,"raw_tex":"Recall from \\cite[Exercise~6.47]{rw} that for any polyhedral convex set $C\\subset \\R^n$ and $\\ox\\in C$, we can find  a neighborhood $\\O$ of $0\\in \\R^n$ for which we have \n\\begin{equation}\\label{tanrep}\nT_C(\\ox)\\cap \\O=(C-\\ox)\\cap \\O.\n\\end{equation}\n","sha256_utf8":"c27583cf59ba62bd7210080c34aed3c4bf90a386619a7151e908855ff3f6eb56"},{"start":24510,"end":25311,"start_line":414,"end_line":425,"raw_tex":"Given a function $f:\\R^n \\to \\oR$ and a point $\\ox\\in\\R^n$ with $f(\\ox)$ finite, the critical cone of $f$ at $\\ox$ for $\\bar v$ with $\\bar v\\in   \\sub f(\\ox)$ is defined by \n\\begin{equation*}\\label{cricone}\n{K_f}(\\ox,\\bar v)=\\big\\{w\\in \\R^n\\,\\big|\\,\\la\\bar v,w\\ra=\\d f(\\ox)(w)\\big\\}.\n\\end{equation*}\nIf, in addition, $f$ is convex and $\\partial f(\\ox)\\neq\\emp$, it follows from   \\cite[Theorem~8.30]{rw} that  its subderivative function is  the support function of $\\partial f(\\ox)$, that is\n\\begin{equation*}\\label{fsud2}\n\\d f(\\ox)(w) = \\sup\\big\\{\\la v, w\\ra\\, \\big|\\, v\\in \\partial f(\\ox)\\big\\},\n\\end{equation*}\nwhich in turn allows us to equivalently describe the critical cone ${K_f}(\\ox,\\bar v)$ as \n\\begin{equation}\\label{cricone1}\nK_f(\\ox, \\bar v) = N_{\\partial f(\\ox)}(\\bar v).\n\\end{equation}\n","sha256_utf8":"eb08b42d7537b18b1c52f34f7eecedd3ec3c70d91a8c05a1367b70ab84ac3416"},{"start":36059,"end":36268,"start_line":553,"end_line":553,"raw_tex":"Let us recall an important result, used in the proof of next corollary, saying that the conjugate function $g^*$ of  a polyhedral function $g$ is again a polyhedral function, cf. \\cite[Theorem~11.14(a)]{rw}. \n","sha256_utf8":"61fab028a57716324214146fe751f606fff58a50c3367b8fb18d74f2f3e6cad9"},{"start":38370,"end":38521,"start_line":584,"end_line":587,"raw_tex":"Recall that   a closed face $F$ of a  polyhedral convex cone $C\\subset \\R^d$ is defined by \n$$\nF=C\\cap [v]^\\perp\\quad \\mbox{for some}\\;\\; v\\in C^*.\n$$\n","sha256_utf8":"6f6711727f6a7670ea230f7203a058d519288e724d9424a80d78863929382711"},{"start":46951,"end":50035,"start_line":715,"end_line":746,"raw_tex":"To this end, consider a set-valued mapping $F:\\R^n\\tto \\R^m$. According to \\cite[Definition~8.33]{rw}, the {\\em graphical derivative} of $F$ at $\\ox$ for $\\oy$ with $(\\ox,\\oy) \\in \\gph F$ is the set-valued mapping $DF(\\ox, \\oy): \\R^n\\tto \\R^m$ defined via the tangent cone to $\\gph F$ at $(\\ox, \\oy)$ by\n\\begin{equation*}\nu \\in DF(\\ox, \\oy)(w)\\; \\Longleftrightarrow\\; (w,u)\\in T_{\\gph F}(\\ox, \\oy),\n\\end{equation*}\nor, equivalently, $\\gph DF(\\ox, \\oy) = T_{\\gph F}(\\ox, \\oy)$.  Using the definition of the tangent cone, we can present an alternative definition of $DF(\\ox, \\oy)$ in terms of graphical limits as\n\\begin{equation}\\label{proto}\n\\gph DF(\\ox, \\oy) = \\limsup_{t\\searrow 0}\\frac{\\gph F-(\\ox,\\oy)}{t}.\n\\end{equation}\nThe set-valued mapping $F$ is said to be {\\em proto-differentiable} at $\\ox$ for $\\oy$ if the outer graphical limit in \\eqref{proto} is actually a full limit. If $F$ is proto-differentiable at $\\ox$ for $\\oy$,   its graphical derivative $DF(\\ox, \\oy)$ is   called the \n{\\em proto-derivative} of $F$ at $\\ox$ for $\\oy$. When $F(\\ox)$ is a singleton consisting of $\\oy$ only, the notation $DF(\\ox, \\oy)$ is simplified to $DF(\\ox)$. It is easy to see that \nfor a single-valued function $F$, which is differentiable at $\\ox$,  the graphical derivative  $DF(\\ox)$ boils down to the Jacobian matrix of $F$ at $\\ox$, denoted by $\\nabla F(\\ox)$. \nRecall from \\cite[Definition~9.53]{rw} that the strict graphical derivative \nof  a set-valued mapping $F$ at $\\ox$ for $\\oy$ with $(\\ox,\\oy)\\in \\gph F$, is  the set-valued mapping $D_*F(\\ox,\\oy):\\R^n\\tto \\R^m$, defined  by \n\\begin{equation}\\label{sproto}\n\\gph D_*F(\\ox,\\oy)=\\limsup_{\n  \\substack{\n   t\\searrow 0 \\\\\n  (x,y)\\xrightarrow{ \\gph F}(\\ox,\\oy)\n  }} \\frac{\\gph F-(x,y)}{t}.\n\\end{equation}\nThe set-valued mapping $F$ is said to be {\\em strictly} proto-differentiable at $\\ox$ for $\\oy$ if the outer graphical limit in \\eqref{sproto} is   a full limit.\n When $F$ is strictly proto-differentiable at $\\ox$ for $\\oy$, its strict graphical derivative $D_*F(\\ox, \\oy)$ is   called the {\\em strict} proto-derivative of $F$ at $\\ox$ for $\\oy$. \n To elaborate  more,  take a set  $\\Omega\\subset \\R^d$ and  $\\ox\\in \\Omega$. The regular (Clarke) tangent cone and the paratingent cone to   $\\Omega$ at $\\ox$ are defined, respectively,   by\n\\begin{equation}\\label{tancc}\n \\widehat T_{\\Omega}(\\ox) =\\liminf_{x \\xrightarrow{ \\Omega}\\ox,t\\searrow 0} \\frac{\\Omega-x}{t}\\quad \\mbox{and}\\quad  \\widetilde {T}_{\\Omega}(\\ox) =\\limsup_{x \\xrightarrow{ \\Omega}\\ox,t\\searrow 0} \\frac{\\Omega-x}{t}.\n\\end{equation}\nThe strict proto-differentiability of $F$ at $\\ox$ for $\\oy$ amounts to saying that $ \\widehat T_{\\gph F}(\\ox,\\oy)= \\widetilde {T}_{\\gph F}(\\ox,\\oy)$.\nObserve also that since the inclusions $ \\widehat T_{\\gph F}(\\ox,\\oy)\\subset  T_{\\gph F}(\\ox,\\oy)\\subset  \\widetilde {T}_{\\gph F}(\\ox,\\oy)$ always hold, \nstrict proto-differentiability of $F$ at $\\ox$ for $\\oy$ implies that  its proto-derivative and strict proto-derivative coincide, namely we have \n\\begin{equation}\\label{eqspd}\nD_*F(\\ox,\\oy)=DF(\\ox,\\oy).\n\\end{equation}\n","sha256_utf8":"024154ffd4800eae97cafb9dfd74cf0438f25ffe95c2b3f8e391c9793f27b8cc"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{Theorem}\\label{lem:proto1}\nFor a polyhedral function $g:\\R^m\\to \\oR$  and $(\\oz, \\olm)\\in \\gph \\partial g$, the following properties hold.\n\\begin{itemize}[noitemsep,topsep=2pt]\n\\item [ \\rm {(a)}]  The graphical derivative of  $  \\sub g$  at $\\oz$ for $\\olm$ can be obtained by  \n\\begin{equation}\\label{proto4}\n\\gph D(\\sub  g)(\\oz,\\olm) = T_{\\gph\\partial g}(\\oz, \\olm)= \\gph N_\\Kg.\n\\end{equation}\n\n\\item [ \\rm {(b)}]  The strict graphical derivative of  $  \\sub g$  at $\\oz$ for $\\olm$ can be obtained by{\\footnote{This formula was brought to our attention by a reviewer.}}  \n\\begin{equation}\\label{proto4n}\n\\gph D_*(\\sub  g)(\\oz,\\olm) = \\gph N_{\\Kg}-\\gph N_{\\Kg}.\n\\end{equation}\n\n\\item [ \\rm {(c)}]   The condition   $\\olm \\in \\ri\\partial g (\\oz)$ amounts to the existence of  a neighborhood $O$ of $(\\oz,\\olm)$ such that for any $(z,\\lm)\\in O\\cap \\gph \\sub g$, $\\sub g$ is strictly proto-differentiable at $z$ for $\\lm$\nand its proto-derivative and strict proto-derivative  coincide, namely \n\\begin{equation}\\label{gdpd}\nD_*(\\sub  g)(z,\\lm)=D(\\sub  g)(z,\\lm).\n\\end{equation}\n\\end{itemize}\n\\end{Theorem}","units":[{"start":51389,"end":52499,"start_line":759,"end_line":778,"raw_tex":"\\begin{Theorem}\\label{lem:proto1}\nFor a polyhedral function $g:\\R^m\\to \\oR$  and $(\\oz, \\olm)\\in \\gph \\partial g$, the following properties hold.\n\\begin{itemize}[noitemsep,topsep=2pt]\n\\item [ \\rm {(a)}]  The graphical derivative of  $  \\sub g$  at $\\oz$ for $\\olm$ can be obtained by  \n\\begin{equation}\\label{proto4}\n\\gph D(\\sub  g)(\\oz,\\olm) = T_{\\gph\\partial g}(\\oz, \\olm)= \\gph N_\\Kg.\n\\end{equation}\n\n\\item [ \\rm {(b)}]  The strict graphical derivative of  $  \\sub g$  at $\\oz$ for $\\olm$ can be obtained by{\\footnote{This formula was brought to our attention by a reviewer.}}  \n\\begin{equation}\\label{proto4n}\n\\gph D_*(\\sub  g)(\\oz,\\olm) = \\gph N_{\\Kg}-\\gph N_{\\Kg}.\n\\end{equation}\n\n\\item [ \\rm {(c)}]   The condition   $\\olm \\in \\ri\\partial g (\\oz)$ amounts to the existence of  a neighborhood $O$ of $(\\oz,\\olm)$ such that for any $(z,\\lm)\\in O\\cap \\gph \\sub g$, $\\sub g$ is strictly proto-differentiable at $z$ for $\\lm$\nand its proto-derivative and strict proto-derivative  coincide, namely \n\\begin{equation}\\label{gdpd}\nD_*(\\sub  g)(z,\\lm)=D(\\sub  g)(z,\\lm).\n\\end{equation}\n\\end{itemize}\n\\end{Theorem}","sha256_utf8":"128d8d8ce8c7472396eaa0f0b415a2bddac2fe9fb0bf3b4250a7d2275ef6734f"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{Lemma}\\label{lem:indset} Assume that $g:\\R^m\\to \\oR$ is a polyhedral function and that  $(\\oz,\\olm)\\in\\gph\\sub g$. Then there exists $r>0$ such that for any decomposition of $\\olm$ as \n\\eqref{decomp} and any $(z, \\lm) \\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$, we have\n\\begin{equation}\\label{indset7}\n J_+(\\oz, \\osigma) \\subset J(z)\\; \\; \\textrm{ and }\\; \\; I_+(\\oz, \\otau) \\subset I(z). \n\\end{equation}\n\\end{Lemma}\n\n\\begin{proof}\nSuppose by contradiction that for each $k\\in \\N$, there are a decomposition of $\\olm  $ as\n\\begin{equation}\\label{indset1}\n\\olm =  \\sum_{j\\in J(\\oz)}\\osigma_j^ka^j+ \\sum_{i\\in I(\\oz)}\\otau_i^kb^i\\; \\; \\text{ with }\\; \\;  \\osigma_j^k,\\otau_i^k\\ge 0\\; \\text{ and }\\; \\sum_{j\\in J(\\oz)}\\osigma_j^k=1,\n\\end{equation}\nand $(\\zk,\\lmk)\\in\\gph\\sub g$ such that $(\\zk,\\lmk)\\to(\\oz,\\olm)$ as $k\\to\\infty$ with $J_+^k\\not\\subset J(\\zk)$ or $I_+^k\\not\\subset I(\\zk)$, where $J_+^k:= J_+(\\oz,\\osigma^k)$ \nand $I_+^k:=I_+(\\oz,\\otau^k)$ are defined via   \\eqref{indset}. Since $\\zk \\to \\oz$ as $k\\to\\infty$, the inclusions  $J(\\zk)\\subset J(\\oz)$\n and $I(\\zk)\\subset I(\\oz)$ hold for all $k$ sufficiently large. Passing to a subsequence if necessary, we can assume that there exist subsets $\\bar J\\subset J(\\oz)$ and $\\bar I \\subset I(\\oz)$ such that\n\\begin{equation*}\nJ(\\zk)=\\bar J \\quad \\text{ and }\\quad I(\\zk)=\\bar I \\quad\\text{ for all }k\\in \\N,\n\\end{equation*}\nwhich, together with \\eqref{subg}, lead us to \n\\begin{equation*}\n\\partial g(\\zk) = \\co\\big\\{a^j|\\, j\\in \\bar J\\big\\}+\\cone\\big\\{b^i|\\, i\\in \\bar I\\big\\}=:\\Omega\\quad\\textrm{ for all } k\\in \\N.\n\\end{equation*}\nSince $\\lmk \\in \\partial g(\\zk) = \\Omega$ and $\\lmk\\to \\olm$ as $k\\to\\infty$, we arrive at  $\\olm\\in\\Omega =\\partial g(\\zk)$ for all  $k$ sufficiently large. Fix such a  $k\\in \\N$ and  deduce from $\\olm \\in \\partial g(\\zk)$ that\n\\begin{equation}\\label{indset3}\n\\la \\olm, \\oz-\\zk\\ra \\leq g(\\oz)-g(\\zk).\n\\end{equation}\nBy the decomposition in \\eqref{indset1}, we obtain  \n\\begin{align}\\label{indset2}\n\\la \\olm, \\zk - \\oz\\ra &= \\sum_{j\\in J(\\oz)} \\osigma_j^k\\la a^j, \\zk-\\oz\\ra +\\sum_{i\\in I(\\oz)}\\otau_i^k\\la b^i, \\zk - \\oz\\ra\\nonumber\\\\\n&= \\sum_{j\\in J(\\oz)} \\osigma_j^k\\big(\\la a^j, \\zk\\ra - \\alpha_j - g(\\oz)\\big) +\\sum_{i\\in  I(\\oz)}\\otau_i^k \\big(\\la b^i, \\zk\\ra - \\beta_i\\big),\\nonumber\\\\\n&\\leq \\sum_{j\\in J(\\oz)} \\osigma_j^k\\big(g(\\zk) - g(\\oz)\\big),\n\\end{align}\nwhere  the second equality results from  \\eqref{indset4} and the last inequality comes from  the fact that $\\zk\\in \\dom g$, combined with  \\eqref{dom}--\\eqref{polyfunc}. If $J_+^k\\not\\subset J(\\zk)$,  there exists   $\\oj\\in  J_+^k$ such that $\\zk\\notin C_\\oj$, meaning that \n\\begin{equation*}\\label{indset5}\n\\osigma_\\oj^k \\big(\\la a^\\oj, \\zk\\ra - \\alpha_\\oj - g(\\oz)\\big) < \\osigma_\\oj^k \\big(g(\\zk) - g(\\oz)\\big).\n\\end{equation*}\nIf $I_+^k\\not\\subset I(\\zk)$, we find  $\\oi\\in I_+^k$ such that $\\la b^\\oi, \\zk\\ra <\\beta_\\oi$, which implies that \n\\begin{equation*}\\label{indset6}\n\\otau_\\oi^k \\big(\\la b^\\oi, \\zk\\ra - \\beta_\\oi\\big) < 0,\n\\end{equation*}\nwhich tells us that  in both cases the last inequality in \\eqref{indset2} is strict. Thus, our assumption that either $J_+^k\\not\\subset J(\\zk)$ or $I_+^k\\not\\subset I(\\zk)$, together  with the last condition in \\eqref{indset1}, yields\n\\begin{equation*}\n\\la \\olm, \\zk - \\oz\\ra < \\sum_{j\\in J(\\oz)} \\osigma_j^k\\big(g(\\zk) - g(\\oz)\\big) = g(\\zk) - g(\\oz),\n\\end{equation*}\nwhich contradicts \\eqref{indset3} and therefore completes the proof.\n\\end{proof}\n\n\\begin{Proposition}[first-order variational properties] \\label{fopg}\nAssume that $g:\\R^m\\to \\oR$ is a polyhedral function and $\\oz\\in \\dom g$. Then the following properties  hold.\n\\begin{itemize}[noitemsep,topsep=2pt]\n\\item [ \\rm {(a)}] The domain of the subderivative function $\\d g(\\oz)$ can be calculated by \n\\begin{equation*}\\label{domdg}\n\\dom \\d g(\\oz) = T_{\\dom g}(\\oz) = \\bigcup_{j\\in J(\\oz)} T_{C_j}(\\oz).\n\\end{equation*}\n\\item [ \\rm {(b)}] If $w\\in T_{C_j}(\\oz)$ for some $j\\in J(\\oz)$, then we have \n$\\d g(\\oz) (w) =\\la a^j, w\\ra$. \nMoreover, there exists $r> 0$ such that  any  $w\\in T_{\\dom  g}(\\oz)\\cap \\B_r(0)$, the representation \n\\begin{equation}\\label{approx}\ng(\\oz + w) = g(\\oz) +\\d g(\\oz)(w)\n\\end{equation}\nholds.\n\\end{itemize}\n \n\n\\end{Proposition}\n\n\\begin{proof} The first equality in (a) was established in \\cite[Proposition~10.21]{rw}. The second equality results immediately from the fact that $\\dom g=\\bigcup_{j\\in J} C_j$.\nThe first claim in  (b) can be found again in \\cite[Proposition~10.21]{rw}. To prove  \\eqref{approx}, recall that for any $j\\in J(\\oz)$,    $C_j$ is  a polyhedral convex set. Employing \\eqref{tanrep} for these sets, we can find $r>0$ such that\n\\begin{equation*}\nT_{C_j}(\\oz)\\cap \\B_r(0)=(C_j-\\oz)\\cap \\B_r(0)\\quad\\textrm{ for all }\\; j\\in J(\\oz).\n\\end{equation*}\nPick any $w \\in T_{\\dom g}(\\oz)\\cap \\B_r(0)$,  and conclude from  (a) that $w\\in T_{C_\\oj}(\\oz)\\cap \\B_r(0)$ for some $\\oj \\in J(\\oz)$ and from the first claim in (b) that $\\d g(\\oz)(w) =\\la a^\\oj, w\\ra$. Thus, we get    $\\oz + w \\in C_\\oj$, which, together with the definition of $C_{j_0}$ and \\eqref{indset4}, \nleads us to \n\\begin{equation*}\ng(\\oz+ w) = \\la a^\\oj, \\oz+w\\ra-\\alpha_\\oj = g(\\oz) +\\la a^\\oj, w\\ra = g(\\oz)+\\d g(\\oz)(w),\n\\end{equation*}\nand hence completes the proof.\n\\end{proof}\n\n\\begin{Proposition} \\label{nonde1} Assume that $g:\\R^m\\to \\oR$ is a polyhedral function and $(\\oz, \\olm)\\in \\gph \\partial g$. Then the following properties are equivalent:\n\\begin{itemize}[noitemsep,topsep=2pt]\n\\item [ \\rm {(a)}]  $\\olm\\in \\ri \\sub g(\\oz)$;\n\\item [ \\rm {(b)}]  the critical cone $K_g(\\oz,\\olm)$ is a linear subspace.\n\\end{itemize}\n\\end{Proposition}\n\n\\begin{proof} By  \\eqref{cricone1}, we have $\\Kg=N_{\\sub g(\\oz)}(\\olm)$. The equivalence of (a) and $N_{\\sub g(\\oz)}(\\olm)$ being a subspace is a well-known fact from convex analysis. \n\\end{proof}\n\n\\begin{Proposition} [critical cone of polyhedral functions] \\label{prop:cricone} \nAssume that $g:\\R^m\\to \\oR$ is a polyhedral function and $(\\oz, \\olm)\\in \\gph \\partial g$. If $\\olm $ has the decomposition in  \\eqref{decomp}, then  $w\\in \\Kg$ if and only if \n$w$ satisfies the conditions \n\\begin{equation*}\\label{cricone3}\n\\begin{cases}\n\\la a^i-a^j, w\\ra = 0,\\quad&\\textrm{for }\\; i,j \\in J_+(\\oz, \\osigma),\\\\\n\\la a^i-a^j, w\\ra \\leq 0 ,&\\textrm{for }\\; i \\in J(\\oz)\\setminus J_+(\\oz, \\osigma)\\; \\textrm{ and }\\; j \\in J_+(\\oz, \\osigma),\\\\\n\\la b^i, w\\ra = 0,&\\textrm{for }\\; i \\in I_+(\\oz, \\otau),\\\\\n\\la b^i, w\\ra\\leq 0,&\\textrm{for }\\; i \\in I(\\oz)\\setminus I_+(\\oz, \\otau).\n\\end{cases}\n\\end{equation*}\n\\end{Proposition}\n\n\\begin{Theorem}[reduction lemma]\\label{thm:reduction}\nAssume that $g:\\R^m\\to \\oR$ is a polyhedral function and $(\\oz, \\olm)\\in \\gph \\partial g$.  Then there exists $r>0$ such that\n\\begin{equation}\\label{reduction}\n\\big( (\\gph \\sub g ) - (\\oz, \\olm)\\big)\\cap \\B_r(0,0) = \\big(\\gph N_\\Kg\\big)\\cap \\B_r(0,0).\n\\end{equation}\n\\end{Theorem}\n\n\\begin{proof}\nSuppose that the polyhedral function $g$  has the representation   \\eqref{polyfunc}. We first show that there is $r>0$ for which\nthe inclusion `$\\supset$' in \\eqref{reduction} holds. To do so, using  \\eqref{tanrep} for the polyhedral convex set $\\partial g(\\oz)$, we find $r_0>0$ such that \n\\begin{equation}\\label{relem1}\nT_{\\partial g(\\oz)}(\\olm)\\cap \\B_{r_0}(0) = \\big(\\partial g(\\oz) -\\olm\\big)\\cap \\B_{r_0}(0).\n\\end{equation}\nPick now a pair $(w, u)$ from the right-hand side of \\eqref{reduction} with $r=r_0$ therein, which implies that  the conditions $w\\in \\Kg\\cap\\B_{r_0}(0)$, $u \\in \\Kg^*\\cap \\B_{r_0}(0)$, and $\\la u, w\\ra =0$\nare satisfied. Moreover, using \\eqref{cricone1} for the polyhedral function $g$, we get $\\Kg^*= T_{\\partial g(\\oz)}(\\olm)$. Therefore $u$ belongs to the left-hand side of \\eqref{relem1}, and thus $\\olm+u \\in\\partial g(\\oz)$. We now show that $(\\oz +w, \\olm +u) \\in\\gph\\partial g$. To this end, take $z\\in \\R^m$   and observe that \n\\begin{equation}\\label{relem2}\n\\la \\olm+u, z- (\\oz+w)\\ra = \\la \\olm+u, z-\\oz\\ra -\\la\\olm, w\\ra - \\la u, w\\ra\\leq g(z)-g(\\oz) - \\d g(\\oz)(w),\n\\end{equation}\nwhere the last inequality is deduced from the facts that $\\olm +u \\in \\partial g(\\oz)$, $w\\in \\Kg$, and $\\la u, w\\ra =0$. Choosing a smaller radius $r_0$ if necessary, we can assume that  \\eqref{approx} is valid on\n $T_{\\dom g}(\\oz)\\cap \\B_{r_0}(0)$. It follows from Proposition~\\ref{fopg}(a) and $w\\in \\Kg\\cap \\B_{r_0}(0)$ that $w\\in T_{\\dom g}(\\oz)\\cap \\B_{r_0}(0)$, which in turn allows us to conclude via \\eqref{approx} that\n  $g(\\oz)+\\d g(\\oz)(w) = g(\\oz+w)$. Combining this and  \\eqref{relem2} leads us to \n\\begin{equation*}\n\\la \\olm+u, z- (\\oz+w)\\ra \\leq g(z)-g(\\oz+w)\n\\end{equation*}\nfor any arbitrary $z\\in \\R^m$, and thus  to $\\olm+u \\in \\partial g(\\oz+w)$. This proves that  $(w, u)$ belongs to the left-hand side of \\eqref{reduction} with $r=r_0$.\n\n \n\nWe now proceed with proving the inclusion `$\\subset$' in \\eqref{reduction} for some $r>0$. Decompose $\\olm$ into $\\olm = \\olm_1+\\olm_2$ with $\\olm_1, \\olm_2$ taken from  \\eqref{decomp}. \nPick $r_0$ from \\eqref{relem1} and choose $r\\in (0, r_0]$  such that the inclusions in \\eqref{indset7} hold for all $(z, v)\\in \\big(\\gph\\partial g\\big)\\cap\\B_{r}(\\oz, \\olm)$. Pick now a pair $(w, v)$ from the left-hand side of \\eqref{reduction}, and conclude that  \n $(w, u) \\in \\B_{r}(0, 0)$ with $\\olm+u \\in \\partial g(\\oz+w)$. We are going to show that  $u\\in N_\\Kg(w)$, which amounts via \\cite[Proposition~2A.3]{DoR14} to the conditions\n\\begin{equation}\\label{relem4}\nw\\in \\Kg, \\quad u\\in \\Kg^*, \\quad\\textrm{ and }\\quad \\la u, w\\ra =0.\n\\end{equation}\nTaking a smaller radius $r$ if necessary, we can assume that $\\partial g(\\oz+w)\\subset \\partial g(\\oz)$ (cf. \\cite[Proposition~3.3(a)]{MoS16}). Thus, $u \\in \\big(\\partial g(\\oz) - \\olm\\big)\\cap \\B_{r}(0)$. Since $r\\leq r_0$, we deduce from  \\eqref{relem1} that \n $u\\in T_{\\partial g(\\oz)}(\\olm)= \\Kg^*$, which justifies the second inclusion  in \\eqref{relem4}. By shrinking $r$ again if necessary, we can derive  from  $(\\oz +w, \\olm+u) \\in  \\big(\\gph\\partial g\\big)\\cap\\B_{r}(\\oz, \\olm)$ and \\eqref{indset7} that \n\\begin{equation*}\nJ_+(\\oz, \\osigma) \\subset J(\\oz+w)\\subset J(\\oz) \\quad\\textrm{ and }\\quad I_+(\\oz, \\otau)\\subset I(\\oz+w)\\subset I(\\oz).\n\\end{equation*}\n To prove that $w\\in \\Kg$, we use  Proposition~\\ref{prop:cricone} in which an equivalent description of $\\Kg$ was given. We break this task into four cases as follows:\n\\begin{itemize}[noitemsep,topsep=2pt]\n\\item [ \\rm {(i)}]   $i, j \\in J_+(\\oz, \\osigma)$. In this case, we obtain from  the inclusions above  that $i, j \\in  J(\\oz+w)\\subset J(\\oz)$. These inclusions bring us via \\eqref{indset4} to  \n\\begin{equation*}\n\\la a^i, \\oz\\ra -\\alpha_i = \\la a^j, \\oz\\ra -\\alpha_j\\quad\\textrm{ and }\\quad\n\\la a^i, \\oz+w\\ra -\\alpha_i = \\la a^j, \\oz+w\\ra -\\alpha_j.\n\\end{equation*}\nCombining  these confirms that $\\la a^i-a^j, w\\ra  =0$ for all $i, j \\in J_+(\\oz, \\osigma)$.\n\n\\item [ (ii)]  $i\\in J(\\oz)\\setminus J_+(\\oz, \\osigma)$ and $j\\in J_+(\\oz, \\osigma)$. In this case, we arrive at $i, j\\in J(\\oz)$ and $j\\in J(\\oz+v)$, which imply via \\eqref{indset4} that  \n\\begin{equation*}\n\\la a^i, \\oz\\ra -\\alpha_i = \\la a^j, \\oz\\ra -\\alpha_j\\quad\\textrm{ and }\\quad\n\\la a^i, \\oz+w\\ra -\\alpha_i \\leq \\la a^j, \\oz+w\\ra -\\alpha_j.\n\\end{equation*}\nCombining  these confirms that $\\la a^i-a^j, w\\ra  \\le 0$ for all $i\\in J(\\oz)\\setminus J_+(\\oz, \\osigma)$ and $j\\in J_+(\\oz, \\osigma)$.\n\n\\item [{(iii)}]  $i\\in I_+(\\oz, \\otau)$. In this case, we have $i\\in I(\\oz)$ and $i \\in I(\\oz+w)$, which result in $\\la b^i, \\oz\\ra =\\la b^i, \\oz+w\\ra=\\beta_i$. Combining  these confirms that $\\la b^i, w\\ra=0$ for all $i\\in I_+(\\oz, \\otau)$.\n\n\\item[(iv)]  $i\\in I(\\oz)\\setminus I_+(\\oz, \\otau)$. In this case,  we deduce from $\\oz+w\\in \\dom g$ and $i\\in I(\\oz)$ that $\\la b^i, \\oz+w\\ra \\leq \\beta_i=\\la b^i, \\oz\\ra$, which in turn yields  \n$\\la b^i, w\\ra\\le 0$ for all $i\\in I(\\oz)\\setminus I_+(\\oz, \\otau)$.\n\\end{itemize}\nIn summary, we showed that $w$ satisfies the equivalent description of $\\Kg$ from Proposition~\\ref{prop:cricone}, and so $w\\in \\Kg$. \nIt remains to demonstrate that $\\la w,u\\ra=0$. Recall that $\\olm+u\\in \\partial g(\\oz+w)$. Using \\eqref{subg}, we can express $\\olm+u$ as\n\\begin{equation*}\\label{relem5}\n\\olm + u = \\sum_{j\\in J(\\oz+w)} \\sigma_j a^j+ \\sum_{i\\in I(\\oz+w)} \\tau_ib^i\\; \\; \\textrm{ with }\\; \\; \\sigma_j, \\tau_i \\geq 0\\; \\textrm{ and }\\;  \\sum_{j\\in J(\\oz+w)} \\sigma_j =1.\n\\end{equation*}\nThis and the decomposition $\\olm = \\olm_1+\\olm_2$ with $\\olm_1,\\olm_2$ taken from \\eqref{decomp} allow us to conclude that  \n\\begin{align*}\n\\la u, w\\ra &= \\la \\olm+u, w\\ra-\\la \\olm, w\\ra\\\\\n&= \\sum_{j\\in J(\\oz+w)} (\\sigma_j-\\osigma_j)\\la a^j, w\\ra+ \\sum_{i\\in I(\\oz+w)} (\\tau_i-\\otau_i)\\la b^i, w\\ra\\\\\n&= \\sum_{j\\in J(\\oz+w)} (\\sigma_j-\\osigma_j)\\big[\\la a^j, \\oz+w\\ra-\\alpha_j -(\\la a^j, \\oz\\ra-\\alpha_j)\\big]+ \\sum_{i\\in I(\\oz+w)} (\\tau_i-\\otau_i)\\big[\\la b^i, \\oz+w\\ra - \\la b^i, \\oz\\ra\\big]\\\\\n&= \\big(g(\\oz+w)-g(\\oz)\\big)\\sum_{j\\in J(\\oz+w)} (\\sigma_j-\\osigma_j)+ \\sum_{i\\in I(\\oz+w)} (\\tau_i-\\otau_i)(\\beta_i-\\beta_i)\\\\\n&=0,\n\\end{align*}\nwhere  the second equality  results from the inclusions $J_+(\\oz, \\osigma)\\subset J(\\oz+w)$ and  $ I_+(\\oz, \\otau)\\subset I(\\oz+w)$, \n where   the fourth equality comes from the  definitions of the active index sets $J(\\oz+w)$ and $I(\\oz+w)$ as well as the inclusions $J(\\oz+w)\\subset J(\\oz)$ and $I(\\oz+w)\\subset I(\\oz)$, \n and where  the last equality follows from the assumption on $\\sigma_j$ and $\\osigma_j$. This proves the inclusion `$\\subset$' in \\eqref{reduction} for some $r>0$ and thus completes the proof. \n\\end{proof}\n\n\\begin{Corollary}[polar relation of critical cones] \\label{polarity}\nAssume that $g:\\R^m\\to \\oR$ is a polyhedral function and $(\\oz, \\olm)\\in \\gph \\partial g$.  Then $\\Kg$ enjoys the duality relationship \n\\begin{equation}\\label{cricone2}\n\\Kg =  K_{g^*}(\\olm, \\oz)^*.\n\\end{equation}\nConsequently, we have \n\\begin{equation}\\label{ricond}\n\\olm \\in \\ri \\partial g(\\oz) \\; \\textrm{ if and only if } \\; \\oz \\in \\ri \\partial g^*(\\olm).\n\\end{equation}\n\\end{Corollary}\n\n\\begin{proof}\nLet $r>0$ be such that  \\eqref{reduction} holds. Then, we conclude from \\eqref{cricone1} that \n\\begin{align*}\nw \\in \\Kg \\; &\\Longleftrightarrow\\; (w, 0)\\in\\gph N_\\Kg\\\\\n&\\Longleftrightarrow\\; \\exists\\, r'>0: t(w, 0)\\in\\big(\\gph N_\\Kg\\big)\\cap \\B_r(0,0)\\;\\; \\textrm{ for all } \\; t\\in [0, r')\\\\\n&\\Longleftrightarrow\\; \\exists\\, r'>0: (\\oz + t w, \\olm)\\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)\\;\\; \\textrm{ for all } \\; t\\in [0, r')\\\\\n&\\Longleftrightarrow\\; \\exists\\, r'>0: \\olm \\in\\partial g(\\oz+tw)\\;\\; \\textrm{ for all } \\; t\\in [0, r')\\\\\n&\\Longleftrightarrow\\; \\exists\\, r'>0: \\oz+tw \\in\\partial g^*(\\olm)\\;\\; \\textrm{ for all } \\; t\\in [0, r')\\\\\n&\\Longleftrightarrow\\; w\\in T_{\\partial g^*(\\olm)}(\\oz)=\\big(N_{\\sub g^*(\\olm)}(\\oz)\\big)^*=K_{g^*}(\\olm, \\oz)^*,\n\\end{align*}\nwhere the third equivalence follows from \\eqref{reduction} and the last  {equivalence relies on the polyhedrality of $\\partial g^*(\\olm)$}.\nTo justify  \\eqref{ricond}, it follows from Proposition~\\ref{nonde1} that $\\Kg$ is a linear subspace if and only if $\\olm\\in \\ri \\sub g(\\oz)$.\nA similar observation can be made for $K_{g^*}(\\olm, \\oz)$, which together with \\eqref{cricone2} completes the proof of  \\eqref{ricond}.\n\\end{proof}\n\n\\begin{Proposition} \\label{lem:faces}\nAssume that $g:\\R^m\\to \\oR$ is a polyhedral function and $(\\oz, \\olm)\\in \\gph \\partial g$.\nThen there exists $r>0$ such that for all $(z, \\lm)\\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$ the corresponding critical cone $\\Kgzv$ enjoys the following representation\n\\begin{equation*}\n\\Kgzv = F_1 -F_2,\n\\end{equation*}\nfor some faces $F_1, F_2$ of $\\Kg$ with $F_2\\subset F_1$. Conversely, for any pair of faces $F_1, F_2$ of $\\Kg$ with $F_2\\subset F_1$ and any sufficiently small real number $r>0$, there exists $(z, \\lm)\\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$ with $\\Kgzv = F_1-F_2$.\n\\end{Proposition}\n\n\\begin{proof} \nLet $r>0$ be such that   \\eqref{reduction} holds. Choosing a smaller radius $r$ if necessary, we can assume \nwithout loss of generality that the inclusions $\\sub g(z)\\subset \\sub g(\\oz)$ and $N_{\\partial g(\\oz)}(\\lm)\\subset N_{\\partial g(\\oz)}(\\olm)$ hold for any $(z, \\lm) \\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$, which can be guaranteed by    $g$ being a polyhedral function. \nWe first show that for any  $(z, \\lm) \\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$, we have \n\\begin{equation}\\label{co6}\n\\Kgzv = \\Kg\\cap [\\lm-\\olm]^\\perp +[z-\\oz ].\n\\end{equation}\nTo this end, pick $(z, \\lm) \\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$ and observe that the inclusion \n\\begin{equation}\\label{in1}\nN_{\\partial g(z)}(\\lm) \\subset N_{\\partial g(z)}(\\olm)\n\\end{equation}\nholds. To justify it, it is not hard to see $\\sub g(z)=\\sub g(\\oz)\\cap D$ with $D:=\\{v\\in \\R^m|\\, \\la v,z-\\oz\\ra=g(z)-g(\\oz)\\}$. \nIt follows from $\\lm\\in \\sub g(z)\\subset D$ and $\\lm-\\olm\\in N_\\Kg(z-\\oz)$ that $\\olm \\in D$. This, together with the definition of $D$, leads us to $N_D(\\lm)=N_D(\\olm)=[z-\\oz]$.\nSince both $\\sub g(\\oz)$ and $D$ are \npolyhedral, we conclude from the intersection rule for normal cones, holding without any constraint qualification for polyhedral convex sets, that \n\\begin{eqnarray*}\nN_{\\partial g(z)}(\\lm)&=& N_{\\sub g(\\oz)\\cap D}(\\lm)= N_{\\partial g(\\oz)}(\\lm)+N_D(\\lm)\\\\\n&\\subset& N_{\\partial g(\\oz)}(\\olm) +N_D(\\olm) =N_{\\sub g(\\oz)\\cap D}(\\olm)=N_{\\partial g(z)}(\\olm),\n\\end{eqnarray*}\nproving our claimed inclusion.  Since $g^*$ is a polyhedral function as well,  using Lemma~\\ref{lem:indset} for $g^*$ at $(\\olm,\\oz)\\in \\gph \\sub g^*$ and shrinking $r$, if necessary, ensure that $\\oz\\in \\sub g^*(\\lm)$.\nMoreover, it results from \\eqref{in1} that $N_{\\partial g^*(\\lm)}(z) \\subset N_{\\partial g^*(\\lm)}(\\oz)$, which  in combination with \\eqref{cricone1}  \nleads us to \n\\begin{equation*}\\label{co5}\nK_{g^*}(\\lm, z)=N_{\\partial g^*(\\lm)}(z) = N_{\\partial g^*(\\lm)}(\\oz)\\cap [z-\\oz]^\\perp =K_{g^*}(\\lm, \\oz)\\cap [z-\\oz]^\\perp.\n\\end{equation*}\nUsing this and  \\eqref{cricone2} brings us to \n\\begin{equation}\\label{co8}\n\\Kgzv =K_{g^*}(\\lm, z)^*= \\big(K_{g^*}(\\lm, \\oz)\\cap [z-\\oz]^\\perp\\big)^*=K_g(\\oz, \\lm) +[z-\\oz ].\n\\end{equation}\nSimilarly, by  \\eqref{cricone1}   and the definition of the normal cone to the polyhedral convex set $\\sub g(\\oz)$, we obtain \n\\begin{equation*}\\label{co7}\nK_g(\\oz, \\lm) = N_{\\partial g(\\oz)}(\\lm) =N_{\\partial g(\\oz)}(\\olm)\\cap [\\lm-\\olm]^\\perp=  \\Kg\\cap [\\lm-\\olm]^\\perp,\n\\end{equation*}\nwhich, together with \\eqref{co8}, proves \\eqref{co6}.    \n\nAfter these preparations, we are in a position to justify the claimed descriptions of critical cones of $g$. Pick $(z, \\lm) \\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$ and\nset $F_1:= \\Kg\\cap [\\lm-\\olm]^\\perp$, which is clearly a face of $\\Kg$. Because $\\lm-\\olm\\in N_\\Kg(z-\\oz)$, resulting from \\eqref{reduction}, we conclude that $z-\\oz\\in F_1$. \nSince the relative interiors of  nonempty  faces of $F_1$ form a partition of this set (cf. \\cite[Theorem~18.2]{Roc70}), we find a face of $F_1$, denoted by $F_2$, that $z-\\oz\\in \\ri F_2$.\nThis tells us that  $F_2\\subset F_1$. Moreover,   $F_2$ is a  face  of $\\Kg$ as well.\nBy \\eqref{co6}, the inclusion $\\Kgzv \\subset F_1 - F_2$ clearly holds. To get the opposite inclusion, pick $x_i\\in F_i$ for $i=1,2$. It follows from $z-\\oz\\in \\ri F_2$ and \\cite[Theorem~6.4]{Roc70} that there is $t>0$ such that $(1+t)(z-\\oz)-tx_2\\in F_2\\subset F_1$,\nwhich yields \n\\begin{equation}\\label{in2}\nt(x_1-x_2)= tx_1+(1+t)(z-\\oz)-tx_2- (1+t)(z-\\oz)\\in F_1+[z-\\oz],\n\\end{equation}\nwhich confirms via \\eqref{co6} that  the inclusion $F_1-F_2\\subset \\Kgzv$ holds. This shows that  $ \\Kgzv=F_1-F_2$.\n\n \n\nAssume now that $F_1$ and $F_2$ are faces of $\\Kg$ with $F_2\\subset F_1$. \nPick $r>0$ from Theorem \\ref{thm:reduction} and choose $u\\in \\Kg^*$ with $\\|u\\|<r/2$ such that $F_1 = \\Kg\\cap [u]^\\perp$. Since $u\\in \\Kg^*=N_\\Kg(0)$, it follows from Theorem \\ref{thm:reduction} that $\\olm+u \\in \\partial g(\\oz)$.\nPick now $w\\in \\ri F_2$ with $\\|w\\|<r/2$ and  observe from $w\\in F_1$ that $(w, u) \\in \\gph N_\\Kg$.  By Theorem \\ref{thm:reduction}, we arrive at $(\\oz+w, \\olm+u) \\in \\gph \\partial g$. Since $w\\in \\ri F_2$, \nas in \\eqref{in2}, we can show that $F_1-F_2=F_1 +[w]$.  Employing this and  \\eqref{co6} leads us to  \n\\begin{equation*}\nF_1 - F_2 = F_1 +[w] =  \\Kg\\cap [u]^\\perp +[w] = K_g(\\oz+w, \\olm+u),\n\\end{equation*}\nwhich completes the proof.\n\\end{proof}\n\n\\begin{Corollary}\\label{incri}\nFor a polyhedral function $g:\\R^m\\to \\oR$  and $(\\oz, \\olm)\\in \\gph \\partial g$, the following properties hold.\n\\begin{itemize}[noitemsep,topsep=2pt]\n\\item [ \\rm {(a)}] There exists $r>0$ such that for all $(z, \\lm) \\in \\big(\\gph\\partial g\\big)\\cap \\B_r(\\oz, \\olm)$ the following inclusions hold:\n\\begin{subequations}\\label{cri2}\n\\begin{align}\n&\\Kgzv \\subset \\Kg -\\Kg,\\label{cri2:a}\\\\\n&\\Kg \\cap -\\Kg \\subset \\Kgzv.\\label{cri2:b}\n\\end{align}\n\\end{subequations}\n\\item [ \\rm {(b)}] If,  in addition,     $\\olm \\in \\ri\\partial g (\\oz)$, then there exists $r>0$ such that for all $(z, \\lm)\\in \\big(\\gph \\partial  g\\big)\\cap\\B_r(\\oz, \\olm)$, we have $\\lm\\in \\ri \\sub g(z)$ and \n\\begin{equation}\\label{cri1}\n\\Kgzv = \\Kg.\n\\end{equation}\n\\end{itemize}\n\\end{Corollary}\n\n\\begin{proof} The inclusion in \n \\eqref{cri2:a} for any $(z, \\lm) \\in \\gph \\partial g$ sufficiently close to $(\\oz, \\olm)$ falls directly from Proposition~\\ref{lem:faces}. \nAlso, according to Proposition~\\ref{lem:faces}, for any such $(z, \\lm) \\in \\gph \\partial g$, one can find  some faces $F_1, F_2$ of $\\Kg$ with $F_2\\subset F_1$\nsuch that $K_g(z,\\lm)=F_1-F_2$. Since $\\Kg \\cap -\\Kg$ is the smallest face of $\\Kg$, we get $\\Kg \\cap -\\Kg\\subset F_i$ for $i=1,2$, which in turn justify \nthe inclusion in  \\eqref{cri2:b} for all $(z, \\lm) \\in \\gph \\partial g$ sufficiently close to $(\\oz, \\olm)$. \n\nTo justify (b), it follows from $\\olm\\in \\ri\\partial g(\\oz)$ and  Proposition~\\ref{nonde1}  that $\\Kg$ is a linear subspace.  This and the inclusions  \\eqref{cri2:a} and \\eqref{cri2:b}   tell us  that\n\\begin{equation*}\n\\Kg=\\Kg-\\Kg\\supset \\Kgzv \\supset \\Kg\\cap-\\Kg=\\Kg\n\\end{equation*}\nfor all $(z, \\lm)\\in \\gph \\partial  g$ sufficiently close to $(\\oz,\\olm)$, which in turn confirms that  $\\Kgzv$ is a linear subspace. Using Proposition~\\ref{nonde1} again, we have $\\lm\\in \\ri \\sub g(z)$ and hence complete the proof of (b).\n\\end{proof}\n\n\\begin{proof} Observe that the first equality in \\eqref{proto4} is the definition of graphical derivative. Using the equality \\eqref{reduction} and the definition of the tangent cone to $\\gph \\sub g$ at $(\\oz, \\olm)$, we immediately  arrive at the second equality in \\eqref{proto4}.\n\n \nTo justify (b),  we conclude from Theorem~\\ref{thm:reduction} and \\cite[Proposition~4H.10]{DoR14}, respectively, that \n\\begin{equation}\\label{para}\n \\widetilde T_{\\gph \\partial g}(\\oz,\\olm)=\\gph D_*(\\sub  g)(\\oz,\\olm)= \\gph D_*N_{\\Kg}(0,0)=\\gph N_{\\Kg} - \\gph N_{\\Kg}.\n\\end{equation}\n\nFinally, we proceed with the proof of (c). Suppose that  $\\olm\\in \\ri \\sub g(\\oz)$.  By Corollary~\\ref{incri}(b), we find $r>0$ such that for any $(z,\\lm)\\in \\B_r(\\oz,\\olm)\\cap \\gph \\sub g$, the property \n $K_g(z,\\lm)=\\Kg$ holds. Take any such a pair $(z,\\lm) $ and conclude from part (a) that  $$T_{\\gph \\partial g}(z,\\lm)=  \\gph N_{K_g(z,\\lm)}=\\gph N_\\Kg.$$\n It follows then from \\cite[Theorem~6.26]{rw} that \n $$\n\\widehat T_{\\gph \\partial g}(\\oz,\\olm)=\\liminf_{\n  (z,\\lm)\\xrightarrow{   \\gph \\sub g}(\\oz,\\olm)}\n  T_{\\gph \\partial g}(z,\\lm)= \\gph N_\\Kg.\n $$\nSince $\\olm\\in \\ri \\sub g(\\oz)$, we conclude that $\\gph N_{\\Kg}=\\Kg\\times\\Kg^\\perp$ is a linear subspace. These, coupled with \\eqref{para}, lead us to \n $\\widehat T_{\\gph \\partial g}(\\oz,\\olm)= \\widetilde T_{\\gph \\partial g}(\\oz,\\olm)$, meaning that $\\sub g$ is strictly proto-differentiable at $\\oz$ for $\\olm$.\n To achieve a similar conclusion for any pair $(z,\\lm)\\in \\gph \\sub g$ sufficiently close to $(\\oz,\\olm)$, observe from Corollary~\\ref{incri}(b)\nthat for any such a pair, we have $\\lm\\in \\ri \\sub g(z)$. A similar argument as the one presented above for $(\\oz,\\olm)$, shows that $\\sub g$ is strictly   proto-differentiable at $z$ for $\\lm$\nwhenever $(z,\\lm)\\in \\gph \\sub g$  is sufficiently close to $(\\oz,\\olm)$. Finally,  \\eqref{gdpd} results from \\eqref{eqspd}. Conversely, if $\\sub g$ is strictly proto-differentiable at $\\oz$ for $\\olm$, \nour discussion prior to \\eqref{eqspd} implies that\n$$\n\\gph N_{\\Kg}=T_{\\gph \\partial g}(\\oz,\\olm)= \\widetilde T_{\\gph \\partial g}(\\oz,\\olm)=\\gph N_{\\Kg}-\\gph N_{\\Kg},\n$$\nwhere the last equality results from \\eqref{para}. This clearly tells us that  $\\Kg$ is a linear subspace and hence it follows from Proposition~\\ref{nonde1} that  $\\olm\\in \\ri \\sub g(\\oz)$, which completes the proof.\n\\end{proof}","units":[{"start":18347,"end":18778,"start_line":316,"end_line":321,"raw_tex":"\\begin{Lemma}\\label{lem:indset} Assume that $g:\\R^m\\to \\oR$ is a polyhedral function and that  $(\\oz,\\olm)\\in\\gph\\sub g$. Then there exists $r>0$ such that for any decomposition of $\\olm$ as \n\\eqref{decomp} and any $(z, \\lm) \\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$, we have\n\\begin{equation}\\label{indset7}\n J_+(\\oz, \\osigma) \\subset J(z)\\; \\; \\textrm{ and }\\; \\; I_+(\\oz, \\otau) \\subset I(z). \n\\end{equation}\n\\end{Lemma}","sha256_utf8":"fdb8033a276361dc254cac988eb81c2c493dff9b72ab4ef152a0cfa0e7bbad63"},{"start":18780,"end":21825,"start_line":323,"end_line":361,"raw_tex":"\\begin{proof}\nSuppose by contradiction that for each $k\\in \\N$, there are a decomposition of $\\olm  $ as\n\\begin{equation}\\label{indset1}\n\\olm =  \\sum_{j\\in J(\\oz)}\\osigma_j^ka^j+ \\sum_{i\\in I(\\oz)}\\otau_i^kb^i\\; \\; \\text{ with }\\; \\;  \\osigma_j^k,\\otau_i^k\\ge 0\\; \\text{ and }\\; \\sum_{j\\in J(\\oz)}\\osigma_j^k=1,\n\\end{equation}\nand $(\\zk,\\lmk)\\in\\gph\\sub g$ such that $(\\zk,\\lmk)\\to(\\oz,\\olm)$ as $k\\to\\infty$ with $J_+^k\\not\\subset J(\\zk)$ or $I_+^k\\not\\subset I(\\zk)$, where $J_+^k:= J_+(\\oz,\\osigma^k)$ \nand $I_+^k:=I_+(\\oz,\\otau^k)$ are defined via   \\eqref{indset}. Since $\\zk \\to \\oz$ as $k\\to\\infty$, the inclusions  $J(\\zk)\\subset J(\\oz)$\n and $I(\\zk)\\subset I(\\oz)$ hold for all $k$ sufficiently large. Passing to a subsequence if necessary, we can assume that there exist subsets $\\bar J\\subset J(\\oz)$ and $\\bar I \\subset I(\\oz)$ such that\n\\begin{equation*}\nJ(\\zk)=\\bar J \\quad \\text{ and }\\quad I(\\zk)=\\bar I \\quad\\text{ for all }k\\in \\N,\n\\end{equation*}\nwhich, together with \\eqref{subg}, lead us to \n\\begin{equation*}\n\\partial g(\\zk) = \\co\\big\\{a^j|\\, j\\in \\bar J\\big\\}+\\cone\\big\\{b^i|\\, i\\in \\bar I\\big\\}=:\\Omega\\quad\\textrm{ for all } k\\in \\N.\n\\end{equation*}\nSince $\\lmk \\in \\partial g(\\zk) = \\Omega$ and $\\lmk\\to \\olm$ as $k\\to\\infty$, we arrive at  $\\olm\\in\\Omega =\\partial g(\\zk)$ for all  $k$ sufficiently large. Fix such a  $k\\in \\N$ and  deduce from $\\olm \\in \\partial g(\\zk)$ that\n\\begin{equation}\\label{indset3}\n\\la \\olm, \\oz-\\zk\\ra \\leq g(\\oz)-g(\\zk).\n\\end{equation}\nBy the decomposition in \\eqref{indset1}, we obtain  \n\\begin{align}\\label{indset2}\n\\la \\olm, \\zk - \\oz\\ra &= \\sum_{j\\in J(\\oz)} \\osigma_j^k\\la a^j, \\zk-\\oz\\ra +\\sum_{i\\in I(\\oz)}\\otau_i^k\\la b^i, \\zk - \\oz\\ra\\nonumber\\\\\n&= \\sum_{j\\in J(\\oz)} \\osigma_j^k\\big(\\la a^j, \\zk\\ra - \\alpha_j - g(\\oz)\\big) +\\sum_{i\\in  I(\\oz)}\\otau_i^k \\big(\\la b^i, \\zk\\ra - \\beta_i\\big),\\nonumber\\\\\n&\\leq \\sum_{j\\in J(\\oz)} \\osigma_j^k\\big(g(\\zk) - g(\\oz)\\big),\n\\end{align}\nwhere  the second equality results from  \\eqref{indset4} and the last inequality comes from  the fact that $\\zk\\in \\dom g$, combined with  \\eqref{dom}--\\eqref{polyfunc}. If $J_+^k\\not\\subset J(\\zk)$,  there exists   $\\oj\\in  J_+^k$ such that $\\zk\\notin C_\\oj$, meaning that \n\\begin{equation*}\\label{indset5}\n\\osigma_\\oj^k \\big(\\la a^\\oj, \\zk\\ra - \\alpha_\\oj - g(\\oz)\\big) < \\osigma_\\oj^k \\big(g(\\zk) - g(\\oz)\\big).\n\\end{equation*}\nIf $I_+^k\\not\\subset I(\\zk)$, we find  $\\oi\\in I_+^k$ such that $\\la b^\\oi, \\zk\\ra <\\beta_\\oi$, which implies that \n\\begin{equation*}\\label{indset6}\n\\otau_\\oi^k \\big(\\la b^\\oi, \\zk\\ra - \\beta_\\oi\\big) < 0,\n\\end{equation*}\nwhich tells us that  in both cases the last inequality in \\eqref{indset2} is strict. Thus, our assumption that either $J_+^k\\not\\subset J(\\zk)$ or $I_+^k\\not\\subset I(\\zk)$, together  with the last condition in \\eqref{indset1}, yields\n\\begin{equation*}\n\\la \\olm, \\zk - \\oz\\ra < \\sum_{j\\in J(\\oz)} \\osigma_j^k\\big(g(\\zk) - g(\\oz)\\big) = g(\\zk) - g(\\oz),\n\\end{equation*}\nwhich contradicts \\eqref{indset3} and therefore completes the proof.\n\\end{proof}","sha256_utf8":"9d0628e43c80574452804e8e6b0c0667d0bf638dbd2275a1403b8263cc6bfb9a"},{"start":22227,"end":22994,"start_line":369,"end_line":386,"raw_tex":"\\begin{Proposition}[first-order variational properties] \\label{fopg}\nAssume that $g:\\R^m\\to \\oR$ is a polyhedral function and $\\oz\\in \\dom g$. Then the following properties  hold.\n\\begin{itemize}[noitemsep,topsep=2pt]\n\\item [ \\rm {(a)}] The domain of the subderivative function $\\d g(\\oz)$ can be calculated by \n\\begin{equation*}\\label{domdg}\n\\dom \\d g(\\oz) = T_{\\dom g}(\\oz) = \\bigcup_{j\\in J(\\oz)} T_{C_j}(\\oz).\n\\end{equation*}\n\\item [ \\rm {(b)}] If $w\\in T_{C_j}(\\oz)$ for some $j\\in J(\\oz)$, then we have \n$\\d g(\\oz) (w) =\\la a^j, w\\ra$. \nMoreover, there exists $r> 0$ such that  any  $w\\in T_{\\dom  g}(\\oz)\\cap \\B_r(0)$, the representation \n\\begin{equation}\\label{approx}\ng(\\oz + w) = g(\\oz) +\\d g(\\oz)(w)\n\\end{equation}\nholds.\n\\end{itemize}\n \n\n\\end{Proposition}","sha256_utf8":"eed3dc3b80bcd0862551a07e5aae633df65129b2197681723a86526edd6dd59f"},{"start":22997,"end":24033,"start_line":389,"end_line":400,"raw_tex":"\\begin{proof} The first equality in (a) was established in \\cite[Proposition~10.21]{rw}. The second equality results immediately from the fact that $\\dom g=\\bigcup_{j\\in J} C_j$.\nThe first claim in  (b) can be found again in \\cite[Proposition~10.21]{rw}. To prove  \\eqref{approx}, recall that for any $j\\in J(\\oz)$,    $C_j$ is  a polyhedral convex set. Employing \\eqref{tanrep} for these sets, we can find $r>0$ such that\n\\begin{equation*}\nT_{C_j}(\\oz)\\cap \\B_r(0)=(C_j-\\oz)\\cap \\B_r(0)\\quad\\textrm{ for all }\\; j\\in J(\\oz).\n\\end{equation*}\nPick any $w \\in T_{\\dom g}(\\oz)\\cap \\B_r(0)$,  and conclude from  (a) that $w\\in T_{C_\\oj}(\\oz)\\cap \\B_r(0)$ for some $\\oj \\in J(\\oz)$ and from the first claim in (b) that $\\d g(\\oz)(w) =\\la a^\\oj, w\\ra$. Thus, we get    $\\oz + w \\in C_\\oj$, which, together with the definition of $C_{j_0}$ and \\eqref{indset4}, \nleads us to \n\\begin{equation*}\ng(\\oz+ w) = \\la a^\\oj, \\oz+w\\ra-\\alpha_\\oj = g(\\oz) +\\la a^\\oj, w\\ra = g(\\oz)+\\d g(\\oz)(w),\n\\end{equation*}\nand hence completes the proof.\n\\end{proof}","sha256_utf8":"8bf2c2c10c019471de77678e359204d2a5f04549f34235aaac738b2214f9fb7e"},{"start":25312,"end":25676,"start_line":427,"end_line":432,"raw_tex":"\\begin{Proposition} \\label{nonde1} Assume that $g:\\R^m\\to \\oR$ is a polyhedral function and $(\\oz, \\olm)\\in \\gph \\partial g$. Then the following properties are equivalent:\n\\begin{itemize}[noitemsep,topsep=2pt]\n\\item [ \\rm {(a)}]  $\\olm\\in \\ri \\sub g(\\oz)$;\n\\item [ \\rm {(b)}]  the critical cone $K_g(\\oz,\\olm)$ is a linear subspace.\n\\end{itemize}\n\\end{Proposition}","sha256_utf8":"96ad059a74e381220ae32272a9f7db0f12325909718184e52480a51e427f5b70"},{"start":25677,"end":25873,"start_line":433,"end_line":434,"raw_tex":"\\begin{proof} By  \\eqref{cricone1}, we have $\\Kg=N_{\\sub g(\\oz)}(\\olm)$. The equivalence of (a) and $N_{\\sub g(\\oz)}(\\olm)$ being a subspace is a well-known fact from convex analysis. \n\\end{proof}","sha256_utf8":"00775d082d3d4a499489b89e824a0e0e23e9a12bbe52e65b9b55c6bd1777dbcc"},{"start":26008,"end":26728,"start_line":437,"end_line":448,"raw_tex":"\\begin{Proposition} [critical cone of polyhedral functions] \\label{prop:cricone} \nAssume that $g:\\R^m\\to \\oR$ is a polyhedral function and $(\\oz, \\olm)\\in \\gph \\partial g$. If $\\olm $ has the decomposition in  \\eqref{decomp}, then  $w\\in \\Kg$ if and only if \n$w$ satisfies the conditions \n\\begin{equation*}\\label{cricone3}\n\\begin{cases}\n\\la a^i-a^j, w\\ra = 0,\\quad&\\textrm{for }\\; i,j \\in J_+(\\oz, \\osigma),\\\\\n\\la a^i-a^j, w\\ra \\leq 0 ,&\\textrm{for }\\; i \\in J(\\oz)\\setminus J_+(\\oz, \\osigma)\\; \\textrm{ and }\\; j \\in J_+(\\oz, \\osigma),\\\\\n\\la b^i, w\\ra = 0,&\\textrm{for }\\; i \\in I_+(\\oz, \\otau),\\\\\n\\la b^i, w\\ra\\leq 0,&\\textrm{for }\\; i \\in I(\\oz)\\setminus I_+(\\oz, \\otau).\n\\end{cases}\n\\end{equation*}\n\\end{Proposition}","sha256_utf8":"14b24fa49da507e37a94d65b6ccaa43ff7c35deb3c491a4470796a6464f7f82a"},{"start":28301,"end":28635,"start_line":466,"end_line":471,"raw_tex":"\\begin{Theorem}[reduction lemma]\\label{thm:reduction}\nAssume that $g:\\R^m\\to \\oR$ is a polyhedral function and $(\\oz, \\olm)\\in \\gph \\partial g$.  Then there exists $r>0$ such that\n\\begin{equation}\\label{reduction}\n\\big( (\\gph \\sub g ) - (\\oz, \\olm)\\big)\\cap \\B_r(0,0) = \\big(\\gph N_\\Kg\\big)\\cap \\B_r(0,0).\n\\end{equation}\n\\end{Theorem}","sha256_utf8":"f96ac7bac350852a302ddc5cdfea984b34bef3bec8d9746db8890333c630d8ca"},{"start":28637,"end":35314,"start_line":473,"end_line":542,"raw_tex":"\\begin{proof}\nSuppose that the polyhedral function $g$  has the representation   \\eqref{polyfunc}. We first show that there is $r>0$ for which\nthe inclusion `$\\supset$' in \\eqref{reduction} holds. To do so, using  \\eqref{tanrep} for the polyhedral convex set $\\partial g(\\oz)$, we find $r_0>0$ such that \n\\begin{equation}\\label{relem1}\nT_{\\partial g(\\oz)}(\\olm)\\cap \\B_{r_0}(0) = \\big(\\partial g(\\oz) -\\olm\\big)\\cap \\B_{r_0}(0).\n\\end{equation}\nPick now a pair $(w, u)$ from the right-hand side of \\eqref{reduction} with $r=r_0$ therein, which implies that  the conditions $w\\in \\Kg\\cap\\B_{r_0}(0)$, $u \\in \\Kg^*\\cap \\B_{r_0}(0)$, and $\\la u, w\\ra =0$\nare satisfied. Moreover, using \\eqref{cricone1} for the polyhedral function $g$, we get $\\Kg^*= T_{\\partial g(\\oz)}(\\olm)$. Therefore $u$ belongs to the left-hand side of \\eqref{relem1}, and thus $\\olm+u \\in\\partial g(\\oz)$. We now show that $(\\oz +w, \\olm +u) \\in\\gph\\partial g$. To this end, take $z\\in \\R^m$   and observe that \n\\begin{equation}\\label{relem2}\n\\la \\olm+u, z- (\\oz+w)\\ra = \\la \\olm+u, z-\\oz\\ra -\\la\\olm, w\\ra - \\la u, w\\ra\\leq g(z)-g(\\oz) - \\d g(\\oz)(w),\n\\end{equation}\nwhere the last inequality is deduced from the facts that $\\olm +u \\in \\partial g(\\oz)$, $w\\in \\Kg$, and $\\la u, w\\ra =0$. Choosing a smaller radius $r_0$ if necessary, we can assume that  \\eqref{approx} is valid on\n $T_{\\dom g}(\\oz)\\cap \\B_{r_0}(0)$. It follows from Proposition~\\ref{fopg}(a) and $w\\in \\Kg\\cap \\B_{r_0}(0)$ that $w\\in T_{\\dom g}(\\oz)\\cap \\B_{r_0}(0)$, which in turn allows us to conclude via \\eqref{approx} that\n  $g(\\oz)+\\d g(\\oz)(w) = g(\\oz+w)$. Combining this and  \\eqref{relem2} leads us to \n\\begin{equation*}\n\\la \\olm+u, z- (\\oz+w)\\ra \\leq g(z)-g(\\oz+w)\n\\end{equation*}\nfor any arbitrary $z\\in \\R^m$, and thus  to $\\olm+u \\in \\partial g(\\oz+w)$. This proves that  $(w, u)$ belongs to the left-hand side of \\eqref{reduction} with $r=r_0$.\n\n \n\nWe now proceed with proving the inclusion `$\\subset$' in \\eqref{reduction} for some $r>0$. Decompose $\\olm$ into $\\olm = \\olm_1+\\olm_2$ with $\\olm_1, \\olm_2$ taken from  \\eqref{decomp}. \nPick $r_0$ from \\eqref{relem1} and choose $r\\in (0, r_0]$  such that the inclusions in \\eqref{indset7} hold for all $(z, v)\\in \\big(\\gph\\partial g\\big)\\cap\\B_{r}(\\oz, \\olm)$. Pick now a pair $(w, v)$ from the left-hand side of \\eqref{reduction}, and conclude that  \n $(w, u) \\in \\B_{r}(0, 0)$ with $\\olm+u \\in \\partial g(\\oz+w)$. We are going to show that  $u\\in N_\\Kg(w)$, which amounts via \\cite[Proposition~2A.3]{DoR14} to the conditions\n\\begin{equation}\\label{relem4}\nw\\in \\Kg, \\quad u\\in \\Kg^*, \\quad\\textrm{ and }\\quad \\la u, w\\ra =0.\n\\end{equation}\nTaking a smaller radius $r$ if necessary, we can assume that $\\partial g(\\oz+w)\\subset \\partial g(\\oz)$ (cf. \\cite[Proposition~3.3(a)]{MoS16}). Thus, $u \\in \\big(\\partial g(\\oz) - \\olm\\big)\\cap \\B_{r}(0)$. Since $r\\leq r_0$, we deduce from  \\eqref{relem1} that \n $u\\in T_{\\partial g(\\oz)}(\\olm)= \\Kg^*$, which justifies the second inclusion  in \\eqref{relem4}. By shrinking $r$ again if necessary, we can derive  from  $(\\oz +w, \\olm+u) \\in  \\big(\\gph\\partial g\\big)\\cap\\B_{r}(\\oz, \\olm)$ and \\eqref{indset7} that \n\\begin{equation*}\nJ_+(\\oz, \\osigma) \\subset J(\\oz+w)\\subset J(\\oz) \\quad\\textrm{ and }\\quad I_+(\\oz, \\otau)\\subset I(\\oz+w)\\subset I(\\oz).\n\\end{equation*}\n To prove that $w\\in \\Kg$, we use  Proposition~\\ref{prop:cricone} in which an equivalent description of $\\Kg$ was given. We break this task into four cases as follows:\n\\begin{itemize}[noitemsep,topsep=2pt]\n\\item [ \\rm {(i)}]   $i, j \\in J_+(\\oz, \\osigma)$. In this case, we obtain from  the inclusions above  that $i, j \\in  J(\\oz+w)\\subset J(\\oz)$. These inclusions bring us via \\eqref{indset4} to  \n\\begin{equation*}\n\\la a^i, \\oz\\ra -\\alpha_i = \\la a^j, \\oz\\ra -\\alpha_j\\quad\\textrm{ and }\\quad\n\\la a^i, \\oz+w\\ra -\\alpha_i = \\la a^j, \\oz+w\\ra -\\alpha_j.\n\\end{equation*}\nCombining  these confirms that $\\la a^i-a^j, w\\ra  =0$ for all $i, j \\in J_+(\\oz, \\osigma)$.\n\n\\item [ (ii)]  $i\\in J(\\oz)\\setminus J_+(\\oz, \\osigma)$ and $j\\in J_+(\\oz, \\osigma)$. In this case, we arrive at $i, j\\in J(\\oz)$ and $j\\in J(\\oz+v)$, which imply via \\eqref{indset4} that  \n\\begin{equation*}\n\\la a^i, \\oz\\ra -\\alpha_i = \\la a^j, \\oz\\ra -\\alpha_j\\quad\\textrm{ and }\\quad\n\\la a^i, \\oz+w\\ra -\\alpha_i \\leq \\la a^j, \\oz+w\\ra -\\alpha_j.\n\\end{equation*}\nCombining  these confirms that $\\la a^i-a^j, w\\ra  \\le 0$ for all $i\\in J(\\oz)\\setminus J_+(\\oz, \\osigma)$ and $j\\in J_+(\\oz, \\osigma)$.\n\n\\item [{(iii)}]  $i\\in I_+(\\oz, \\otau)$. In this case, we have $i\\in I(\\oz)$ and $i \\in I(\\oz+w)$, which result in $\\la b^i, \\oz\\ra =\\la b^i, \\oz+w\\ra=\\beta_i$. Combining  these confirms that $\\la b^i, w\\ra=0$ for all $i\\in I_+(\\oz, \\otau)$.\n\n\\item[(iv)]  $i\\in I(\\oz)\\setminus I_+(\\oz, \\otau)$. In this case,  we deduce from $\\oz+w\\in \\dom g$ and $i\\in I(\\oz)$ that $\\la b^i, \\oz+w\\ra \\leq \\beta_i=\\la b^i, \\oz\\ra$, which in turn yields  \n$\\la b^i, w\\ra\\le 0$ for all $i\\in I(\\oz)\\setminus I_+(\\oz, \\otau)$.\n\\end{itemize}\nIn summary, we showed that $w$ satisfies the equivalent description of $\\Kg$ from Proposition~\\ref{prop:cricone}, and so $w\\in \\Kg$. \nIt remains to demonstrate that $\\la w,u\\ra=0$. Recall that $\\olm+u\\in \\partial g(\\oz+w)$. Using \\eqref{subg}, we can express $\\olm+u$ as\n\\begin{equation*}\\label{relem5}\n\\olm + u = \\sum_{j\\in J(\\oz+w)} \\sigma_j a^j+ \\sum_{i\\in I(\\oz+w)} \\tau_ib^i\\; \\; \\textrm{ with }\\; \\; \\sigma_j, \\tau_i \\geq 0\\; \\textrm{ and }\\;  \\sum_{j\\in J(\\oz+w)} \\sigma_j =1.\n\\end{equation*}\nThis and the decomposition $\\olm = \\olm_1+\\olm_2$ with $\\olm_1,\\olm_2$ taken from \\eqref{decomp} allow us to conclude that  \n\\begin{align*}\n\\la u, w\\ra &= \\la \\olm+u, w\\ra-\\la \\olm, w\\ra\\\\\n&= \\sum_{j\\in J(\\oz+w)} (\\sigma_j-\\osigma_j)\\la a^j, w\\ra+ \\sum_{i\\in I(\\oz+w)} (\\tau_i-\\otau_i)\\la b^i, w\\ra\\\\\n&= \\sum_{j\\in J(\\oz+w)} (\\sigma_j-\\osigma_j)\\big[\\la a^j, \\oz+w\\ra-\\alpha_j -(\\la a^j, \\oz\\ra-\\alpha_j)\\big]+ \\sum_{i\\in I(\\oz+w)} (\\tau_i-\\otau_i)\\big[\\la b^i, \\oz+w\\ra - \\la b^i, \\oz\\ra\\big]\\\\\n&= \\big(g(\\oz+w)-g(\\oz)\\big)\\sum_{j\\in J(\\oz+w)} (\\sigma_j-\\osigma_j)+ \\sum_{i\\in I(\\oz+w)} (\\tau_i-\\otau_i)(\\beta_i-\\beta_i)\\\\\n&=0,\n\\end{align*}\nwhere  the second equality  results from the inclusions $J_+(\\oz, \\osigma)\\subset J(\\oz+w)$ and  $ I_+(\\oz, \\otau)\\subset I(\\oz+w)$, \n where   the fourth equality comes from the  definitions of the active index sets $J(\\oz+w)$ and $I(\\oz+w)$ as well as the inclusions $J(\\oz+w)\\subset J(\\oz)$ and $I(\\oz+w)\\subset I(\\oz)$, \n and where  the last equality follows from the assumption on $\\sigma_j$ and $\\osigma_j$. This proves the inclusion `$\\subset$' in \\eqref{reduction} for some $r>0$ and thus completes the proof. \n\\end{proof}","sha256_utf8":"4eaff9c8a4a00b8ecee395a1513b87abe8905b29ca884b3be7040fbfb0afea14"},{"start":36451,"end":36910,"start_line":556,"end_line":565,"raw_tex":"\\begin{Corollary}[polar relation of critical cones] \\label{polarity}\nAssume that $g:\\R^m\\to \\oR$ is a polyhedral function and $(\\oz, \\olm)\\in \\gph \\partial g$.  Then $\\Kg$ enjoys the duality relationship \n\\begin{equation}\\label{cricone2}\n\\Kg =  K_{g^*}(\\olm, \\oz)^*.\n\\end{equation}\nConsequently, we have \n\\begin{equation}\\label{ricond}\n\\olm \\in \\ri \\partial g(\\oz) \\; \\textrm{ if and only if } \\; \\oz \\in \\ri \\partial g^*(\\olm).\n\\end{equation}\n\\end{Corollary}","sha256_utf8":"d339c78b3f0acbb000402bc8f3e66e65d74b702f5220b89de0a72659c129c67e"},{"start":36912,"end":38140,"start_line":567,"end_line":580,"raw_tex":"\\begin{proof}\nLet $r>0$ be such that  \\eqref{reduction} holds. Then, we conclude from \\eqref{cricone1} that \n\\begin{align*}\nw \\in \\Kg \\; &\\Longleftrightarrow\\; (w, 0)\\in\\gph N_\\Kg\\\\\n&\\Longleftrightarrow\\; \\exists\\, r'>0: t(w, 0)\\in\\big(\\gph N_\\Kg\\big)\\cap \\B_r(0,0)\\;\\; \\textrm{ for all } \\; t\\in [0, r')\\\\\n&\\Longleftrightarrow\\; \\exists\\, r'>0: (\\oz + t w, \\olm)\\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)\\;\\; \\textrm{ for all } \\; t\\in [0, r')\\\\\n&\\Longleftrightarrow\\; \\exists\\, r'>0: \\olm \\in\\partial g(\\oz+tw)\\;\\; \\textrm{ for all } \\; t\\in [0, r')\\\\\n&\\Longleftrightarrow\\; \\exists\\, r'>0: \\oz+tw \\in\\partial g^*(\\olm)\\;\\; \\textrm{ for all } \\; t\\in [0, r')\\\\\n&\\Longleftrightarrow\\; w\\in T_{\\partial g^*(\\olm)}(\\oz)=\\big(N_{\\sub g^*(\\olm)}(\\oz)\\big)^*=K_{g^*}(\\olm, \\oz)^*,\n\\end{align*}\nwhere the third equivalence follows from \\eqref{reduction} and the last  {equivalence relies on the polyhedrality of $\\partial g^*(\\olm)$}.\nTo justify  \\eqref{ricond}, it follows from Proposition~\\ref{nonde1} that $\\Kg$ is a linear subspace if and only if $\\olm\\in \\ri \\sub g(\\oz)$.\nA similar observation can be made for $K_{g^*}(\\olm, \\oz)$, which together with \\eqref{cricone2} completes the proof of  \\eqref{ricond}.\n\\end{proof}","sha256_utf8":"f6cc30d66e0c2eeacc5ebc41b7aec3ce65e6d69b395e2f2017577318ae9694c5"},{"start":38554,"end":39207,"start_line":589,"end_line":596,"raw_tex":"\\begin{Proposition} \\label{lem:faces}\nAssume that $g:\\R^m\\to \\oR$ is a polyhedral function and $(\\oz, \\olm)\\in \\gph \\partial g$.\nThen there exists $r>0$ such that for all $(z, \\lm)\\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$ the corresponding critical cone $\\Kgzv$ enjoys the following representation\n\\begin{equation*}\n\\Kgzv = F_1 -F_2,\n\\end{equation*}\nfor some faces $F_1, F_2$ of $\\Kg$ with $F_2\\subset F_1$. Conversely, for any pair of faces $F_1, F_2$ of $\\Kg$ with $F_2\\subset F_1$ and any sufficiently small real number $r>0$, there exists $(z, \\lm)\\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$ with $\\Kgzv = F_1-F_2$.\n\\end{Proposition}","sha256_utf8":"bffc95660a74f8f5feff860f07f408839e2c87b9d162d0178b0d9c4caa569a0d"},{"start":39210,"end":43719,"start_line":599,"end_line":655,"raw_tex":"\\begin{proof} \nLet $r>0$ be such that   \\eqref{reduction} holds. Choosing a smaller radius $r$ if necessary, we can assume \nwithout loss of generality that the inclusions $\\sub g(z)\\subset \\sub g(\\oz)$ and $N_{\\partial g(\\oz)}(\\lm)\\subset N_{\\partial g(\\oz)}(\\olm)$ hold for any $(z, \\lm) \\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$, which can be guaranteed by    $g$ being a polyhedral function. \nWe first show that for any  $(z, \\lm) \\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$, we have \n\\begin{equation}\\label{co6}\n\\Kgzv = \\Kg\\cap [\\lm-\\olm]^\\perp +[z-\\oz ].\n\\end{equation}\nTo this end, pick $(z, \\lm) \\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$ and observe that the inclusion \n\\begin{equation}\\label{in1}\nN_{\\partial g(z)}(\\lm) \\subset N_{\\partial g(z)}(\\olm)\n\\end{equation}\nholds. To justify it, it is not hard to see $\\sub g(z)=\\sub g(\\oz)\\cap D$ with $D:=\\{v\\in \\R^m|\\, \\la v,z-\\oz\\ra=g(z)-g(\\oz)\\}$. \nIt follows from $\\lm\\in \\sub g(z)\\subset D$ and $\\lm-\\olm\\in N_\\Kg(z-\\oz)$ that $\\olm \\in D$. This, together with the definition of $D$, leads us to $N_D(\\lm)=N_D(\\olm)=[z-\\oz]$.\nSince both $\\sub g(\\oz)$ and $D$ are \npolyhedral, we conclude from the intersection rule for normal cones, holding without any constraint qualification for polyhedral convex sets, that \n\\begin{eqnarray*}\nN_{\\partial g(z)}(\\lm)&=& N_{\\sub g(\\oz)\\cap D}(\\lm)= N_{\\partial g(\\oz)}(\\lm)+N_D(\\lm)\\\\\n&\\subset& N_{\\partial g(\\oz)}(\\olm) +N_D(\\olm) =N_{\\sub g(\\oz)\\cap D}(\\olm)=N_{\\partial g(z)}(\\olm),\n\\end{eqnarray*}\nproving our claimed inclusion.  Since $g^*$ is a polyhedral function as well,  using Lemma~\\ref{lem:indset} for $g^*$ at $(\\olm,\\oz)\\in \\gph \\sub g^*$ and shrinking $r$, if necessary, ensure that $\\oz\\in \\sub g^*(\\lm)$.\nMoreover, it results from \\eqref{in1} that $N_{\\partial g^*(\\lm)}(z) \\subset N_{\\partial g^*(\\lm)}(\\oz)$, which  in combination with \\eqref{cricone1}  \nleads us to \n\\begin{equation*}\\label{co5}\nK_{g^*}(\\lm, z)=N_{\\partial g^*(\\lm)}(z) = N_{\\partial g^*(\\lm)}(\\oz)\\cap [z-\\oz]^\\perp =K_{g^*}(\\lm, \\oz)\\cap [z-\\oz]^\\perp.\n\\end{equation*}\nUsing this and  \\eqref{cricone2} brings us to \n\\begin{equation}\\label{co8}\n\\Kgzv =K_{g^*}(\\lm, z)^*= \\big(K_{g^*}(\\lm, \\oz)\\cap [z-\\oz]^\\perp\\big)^*=K_g(\\oz, \\lm) +[z-\\oz ].\n\\end{equation}\nSimilarly, by  \\eqref{cricone1}   and the definition of the normal cone to the polyhedral convex set $\\sub g(\\oz)$, we obtain \n\\begin{equation*}\\label{co7}\nK_g(\\oz, \\lm) = N_{\\partial g(\\oz)}(\\lm) =N_{\\partial g(\\oz)}(\\olm)\\cap [\\lm-\\olm]^\\perp=  \\Kg\\cap [\\lm-\\olm]^\\perp,\n\\end{equation*}\nwhich, together with \\eqref{co8}, proves \\eqref{co6}.    \n\nAfter these preparations, we are in a position to justify the claimed descriptions of critical cones of $g$. Pick $(z, \\lm) \\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$ and\nset $F_1:= \\Kg\\cap [\\lm-\\olm]^\\perp$, which is clearly a face of $\\Kg$. Because $\\lm-\\olm\\in N_\\Kg(z-\\oz)$, resulting from \\eqref{reduction}, we conclude that $z-\\oz\\in F_1$. \nSince the relative interiors of  nonempty  faces of $F_1$ form a partition of this set (cf. \\cite[Theorem~18.2]{Roc70}), we find a face of $F_1$, denoted by $F_2$, that $z-\\oz\\in \\ri F_2$.\nThis tells us that  $F_2\\subset F_1$. Moreover,   $F_2$ is a  face  of $\\Kg$ as well.\nBy \\eqref{co6}, the inclusion $\\Kgzv \\subset F_1 - F_2$ clearly holds. To get the opposite inclusion, pick $x_i\\in F_i$ for $i=1,2$. It follows from $z-\\oz\\in \\ri F_2$ and \\cite[Theorem~6.4]{Roc70} that there is $t>0$ such that $(1+t)(z-\\oz)-tx_2\\in F_2\\subset F_1$,\nwhich yields \n\\begin{equation}\\label{in2}\nt(x_1-x_2)= tx_1+(1+t)(z-\\oz)-tx_2- (1+t)(z-\\oz)\\in F_1+[z-\\oz],\n\\end{equation}\nwhich confirms via \\eqref{co6} that  the inclusion $F_1-F_2\\subset \\Kgzv$ holds. This shows that  $ \\Kgzv=F_1-F_2$.\n\n \n\nAssume now that $F_1$ and $F_2$ are faces of $\\Kg$ with $F_2\\subset F_1$. \nPick $r>0$ from Theorem \\ref{thm:reduction} and choose $u\\in \\Kg^*$ with $\\|u\\|<r/2$ such that $F_1 = \\Kg\\cap [u]^\\perp$. Since $u\\in \\Kg^*=N_\\Kg(0)$, it follows from Theorem \\ref{thm:reduction} that $\\olm+u \\in \\partial g(\\oz)$.\nPick now $w\\in \\ri F_2$ with $\\|w\\|<r/2$ and  observe from $w\\in F_1$ that $(w, u) \\in \\gph N_\\Kg$.  By Theorem \\ref{thm:reduction}, we arrive at $(\\oz+w, \\olm+u) \\in \\gph \\partial g$. Since $w\\in \\ri F_2$, \nas in \\eqref{in2}, we can show that $F_1-F_2=F_1 +[w]$.  Employing this and  \\eqref{co6} leads us to  \n\\begin{equation*}\nF_1 - F_2 = F_1 +[w] =  \\Kg\\cap [u]^\\perp +[w] = K_g(\\oz+w, \\olm+u),\n\\end{equation*}\nwhich completes the proof.\n\\end{proof}","sha256_utf8":"e19fe2c48fce473d05f108a81b24310f1282d71bdeaffa6d3f6feeeba75ab6df"},{"start":43736,"end":44525,"start_line":672,"end_line":687,"raw_tex":"\\begin{Corollary}\\label{incri}\nFor a polyhedral function $g:\\R^m\\to \\oR$  and $(\\oz, \\olm)\\in \\gph \\partial g$, the following properties hold.\n\\begin{itemize}[noitemsep,topsep=2pt]\n\\item [ \\rm {(a)}] There exists $r>0$ such that for all $(z, \\lm) \\in \\big(\\gph\\partial g\\big)\\cap \\B_r(\\oz, \\olm)$ the following inclusions hold:\n\\begin{subequations}\\label{cri2}\n\\begin{align}\n&\\Kgzv \\subset \\Kg -\\Kg,\\label{cri2:a}\\\\\n&\\Kg \\cap -\\Kg \\subset \\Kgzv.\\label{cri2:b}\n\\end{align}\n\\end{subequations}\n\\item [ \\rm {(b)}] If,  in addition,     $\\olm \\in \\ri\\partial g (\\oz)$, then there exists $r>0$ such that for all $(z, \\lm)\\in \\big(\\gph \\partial  g\\big)\\cap\\B_r(\\oz, \\olm)$, we have $\\lm\\in \\ri \\sub g(z)$ and \n\\begin{equation}\\label{cri1}\n\\Kgzv = \\Kg.\n\\end{equation}\n\\end{itemize}\n\\end{Corollary}","sha256_utf8":"5c9a11e82b946cb40459c79631fd2f74dab9d399a89d1dfd0904dd519de4b0fe"},{"start":44526,"end":45660,"start_line":688,"end_line":699,"raw_tex":"\\begin{proof} The inclusion in \n \\eqref{cri2:a} for any $(z, \\lm) \\in \\gph \\partial g$ sufficiently close to $(\\oz, \\olm)$ falls directly from Proposition~\\ref{lem:faces}. \nAlso, according to Proposition~\\ref{lem:faces}, for any such $(z, \\lm) \\in \\gph \\partial g$, one can find  some faces $F_1, F_2$ of $\\Kg$ with $F_2\\subset F_1$\nsuch that $K_g(z,\\lm)=F_1-F_2$. Since $\\Kg \\cap -\\Kg$ is the smallest face of $\\Kg$, we get $\\Kg \\cap -\\Kg\\subset F_i$ for $i=1,2$, which in turn justify \nthe inclusion in  \\eqref{cri2:b} for all $(z, \\lm) \\in \\gph \\partial g$ sufficiently close to $(\\oz, \\olm)$. \n\nTo justify (b), it follows from $\\olm\\in \\ri\\partial g(\\oz)$ and  Proposition~\\ref{nonde1}  that $\\Kg$ is a linear subspace.  This and the inclusions  \\eqref{cri2:a} and \\eqref{cri2:b}   tell us  that\n\\begin{equation*}\n\\Kg=\\Kg-\\Kg\\supset \\Kgzv \\supset \\Kg\\cap-\\Kg=\\Kg\n\\end{equation*}\nfor all $(z, \\lm)\\in \\gph \\partial  g$ sufficiently close to $(\\oz,\\olm)$, which in turn confirms that  $\\Kgzv$ is a linear subspace. Using Proposition~\\ref{nonde1} again, we have $\\lm\\in \\ri \\sub g(z)$ and hence complete the proof of (b).\n\\end{proof}","sha256_utf8":"76f5ca697ec959e5a58a4e2a8935a157563ff1bc49796f9cb8180dc2ea7bdcea"},{"start":52501,"end":54885,"start_line":780,"end_line":806,"raw_tex":"\\begin{proof} Observe that the first equality in \\eqref{proto4} is the definition of graphical derivative. Using the equality \\eqref{reduction} and the definition of the tangent cone to $\\gph \\sub g$ at $(\\oz, \\olm)$, we immediately  arrive at the second equality in \\eqref{proto4}.\n\n \nTo justify (b),  we conclude from Theorem~\\ref{thm:reduction} and \\cite[Proposition~4H.10]{DoR14}, respectively, that \n\\begin{equation}\\label{para}\n \\widetilde T_{\\gph \\partial g}(\\oz,\\olm)=\\gph D_*(\\sub  g)(\\oz,\\olm)= \\gph D_*N_{\\Kg}(0,0)=\\gph N_{\\Kg} - \\gph N_{\\Kg}.\n\\end{equation}\n\nFinally, we proceed with the proof of (c). Suppose that  $\\olm\\in \\ri \\sub g(\\oz)$.  By Corollary~\\ref{incri}(b), we find $r>0$ such that for any $(z,\\lm)\\in \\B_r(\\oz,\\olm)\\cap \\gph \\sub g$, the property \n $K_g(z,\\lm)=\\Kg$ holds. Take any such a pair $(z,\\lm) $ and conclude from part (a) that  $$T_{\\gph \\partial g}(z,\\lm)=  \\gph N_{K_g(z,\\lm)}=\\gph N_\\Kg.$$\n It follows then from \\cite[Theorem~6.26]{rw} that \n $$\n\\widehat T_{\\gph \\partial g}(\\oz,\\olm)=\\liminf_{\n  (z,\\lm)\\xrightarrow{   \\gph \\sub g}(\\oz,\\olm)}\n  T_{\\gph \\partial g}(z,\\lm)= \\gph N_\\Kg.\n $$\nSince $\\olm\\in \\ri \\sub g(\\oz)$, we conclude that $\\gph N_{\\Kg}=\\Kg\\times\\Kg^\\perp$ is a linear subspace. These, coupled with \\eqref{para}, lead us to \n $\\widehat T_{\\gph \\partial g}(\\oz,\\olm)= \\widetilde T_{\\gph \\partial g}(\\oz,\\olm)$, meaning that $\\sub g$ is strictly proto-differentiable at $\\oz$ for $\\olm$.\n To achieve a similar conclusion for any pair $(z,\\lm)\\in \\gph \\sub g$ sufficiently close to $(\\oz,\\olm)$, observe from Corollary~\\ref{incri}(b)\nthat for any such a pair, we have $\\lm\\in \\ri \\sub g(z)$. A similar argument as the one presented above for $(\\oz,\\olm)$, shows that $\\sub g$ is strictly   proto-differentiable at $z$ for $\\lm$\nwhenever $(z,\\lm)\\in \\gph \\sub g$  is sufficiently close to $(\\oz,\\olm)$. Finally,  \\eqref{gdpd} results from \\eqref{eqspd}. Conversely, if $\\sub g$ is strictly proto-differentiable at $\\oz$ for $\\olm$, \nour discussion prior to \\eqref{eqspd} implies that\n$$\n\\gph N_{\\Kg}=T_{\\gph \\partial g}(\\oz,\\olm)= \\widetilde T_{\\gph \\partial g}(\\oz,\\olm)=\\gph N_{\\Kg}-\\gph N_{\\Kg},\n$$\nwhere the last equality results from \\eqref{para}. This clearly tells us that  $\\Kg$ is a linear subspace and hence it follows from Proposition~\\ref{nonde1} that  $\\olm\\in \\ri \\sub g(\\oz)$, which completes the proof.\n\\end{proof}","sha256_utf8":"eda09479638eaffa7a61ee88eeeb3b186e7d96bffb63ba1e494c43f354506f49"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"严格原可微性的相对内部刻画","source_units":[{"start":51389,"end":52499,"start_line":759,"end_line":778,"raw_tex":"\\begin{Theorem}\\label{lem:proto1}\nFor a polyhedral function $g:\\R^m\\to \\oR$  and $(\\oz, \\olm)\\in \\gph \\partial g$, the following properties hold.\n\\begin{itemize}[noitemsep,topsep=2pt]\n\\item [ \\rm {(a)}]  The graphical derivative of  $  \\sub g$  at $\\oz$ for $\\olm$ can be obtained by  \n\\begin{equation}\\label{proto4}\n\\gph D(\\sub  g)(\\oz,\\olm) = T_{\\gph\\partial g}(\\oz, \\olm)= \\gph N_\\Kg.\n\\end{equation}\n\n\\item [ \\rm {(b)}]  The strict graphical derivative of  $  \\sub g$  at $\\oz$ for $\\olm$ can be obtained by{\\footnote{This formula was brought to our attention by a reviewer.}}  \n\\begin{equation}\\label{proto4n}\n\\gph D_*(\\sub  g)(\\oz,\\olm) = \\gph N_{\\Kg}-\\gph N_{\\Kg}.\n\\end{equation}\n\n\\item [ \\rm {(c)}]   The condition   $\\olm \\in \\ri\\partial g (\\oz)$ amounts to the existence of  a neighborhood $O$ of $(\\oz,\\olm)$ such that for any $(z,\\lm)\\in O\\cap \\gph \\sub g$, $\\sub g$ is strictly proto-differentiable at $z$ for $\\lm$\nand its proto-derivative and strict proto-derivative  coincide, namely \n\\begin{equation}\\label{gdpd}\nD_*(\\sub  g)(z,\\lm)=D(\\sub  g)(z,\\lm).\n\\end{equation}\n\\end{itemize}\n\\end{Theorem}","sha256_utf8":"128d8d8ce8c7472396eaa0f0b415a2bddac2fe9fb0bf3b4250a7d2275ef6734f"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"主定理完整证明","source_units":[{"start":52501,"end":54885,"start_line":780,"end_line":806,"raw_tex":"\\begin{proof} Observe that the first equality in \\eqref{proto4} is the definition of graphical derivative. Using the equality \\eqref{reduction} and the definition of the tangent cone to $\\gph \\sub g$ at $(\\oz, \\olm)$, we immediately  arrive at the second equality in \\eqref{proto4}.\n\n \nTo justify (b),  we conclude from Theorem~\\ref{thm:reduction} and \\cite[Proposition~4H.10]{DoR14}, respectively, that \n\\begin{equation}\\label{para}\n \\widetilde T_{\\gph \\partial g}(\\oz,\\olm)=\\gph D_*(\\sub  g)(\\oz,\\olm)= \\gph D_*N_{\\Kg}(0,0)=\\gph N_{\\Kg} - \\gph N_{\\Kg}.\n\\end{equation}\n\nFinally, we proceed with the proof of (c). Suppose that  $\\olm\\in \\ri \\sub g(\\oz)$.  By Corollary~\\ref{incri}(b), we find $r>0$ such that for any $(z,\\lm)\\in \\B_r(\\oz,\\olm)\\cap \\gph \\sub g$, the property \n $K_g(z,\\lm)=\\Kg$ holds. Take any such a pair $(z,\\lm) $ and conclude from part (a) that  $$T_{\\gph \\partial g}(z,\\lm)=  \\gph N_{K_g(z,\\lm)}=\\gph N_\\Kg.$$\n It follows then from \\cite[Theorem~6.26]{rw} that \n $$\n\\widehat T_{\\gph \\partial g}(\\oz,\\olm)=\\liminf_{\n  (z,\\lm)\\xrightarrow{   \\gph \\sub g}(\\oz,\\olm)}\n  T_{\\gph \\partial g}(z,\\lm)= \\gph N_\\Kg.\n $$\nSince $\\olm\\in \\ri \\sub g(\\oz)$, we conclude that $\\gph N_{\\Kg}=\\Kg\\times\\Kg^\\perp$ is a linear subspace. These, coupled with \\eqref{para}, lead us to \n $\\widehat T_{\\gph \\partial g}(\\oz,\\olm)= \\widetilde T_{\\gph \\partial g}(\\oz,\\olm)$, meaning that $\\sub g$ is strictly proto-differentiable at $\\oz$ for $\\olm$.\n To achieve a similar conclusion for any pair $(z,\\lm)\\in \\gph \\sub g$ sufficiently close to $(\\oz,\\olm)$, observe from Corollary~\\ref{incri}(b)\nthat for any such a pair, we have $\\lm\\in \\ri \\sub g(z)$. A similar argument as the one presented above for $(\\oz,\\olm)$, shows that $\\sub g$ is strictly   proto-differentiable at $z$ for $\\lm$\nwhenever $(z,\\lm)\\in \\gph \\sub g$  is sufficiently close to $(\\oz,\\olm)$. Finally,  \\eqref{gdpd} results from \\eqref{eqspd}. Conversely, if $\\sub g$ is strictly proto-differentiable at $\\oz$ for $\\olm$, \nour discussion prior to \\eqref{eqspd} implies that\n$$\n\\gph N_{\\Kg}=T_{\\gph \\partial g}(\\oz,\\olm)= \\widetilde T_{\\gph \\partial g}(\\oz,\\olm)=\\gph N_{\\Kg}-\\gph N_{\\Kg},\n$$\nwhere the last equality results from \\eqref{para}. This clearly tells us that  $\\Kg$ is a linear subspace and hence it follows from Proposition~\\ref{nonde1} that  $\\olm\\in \\ri \\sub g(\\oz)$, which completes the proof.\n\\end{proof}","sha256_utf8":"eda09479638eaffa7a61ee88eeeb3b186e7d96bffb63ba1e494c43f354506f49"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"约化图形得到图导数","source_units":[{"start":52501,"end":52784,"start_line":780,"end_line":780,"raw_tex":"\\begin{proof} Observe that the first equality in \\eqref{proto4} is the definition of graphical derivative. Using the equality \\eqref{reduction} and the definition of the tangent cone to $\\gph \\sub g$ at $(\\oz, \\olm)$, we immediately  arrive at the second equality in \\eqref{proto4}.\n","sha256_utf8":"df91572c0a3b9525bbda0934535521bc760609de9c321b140dac28a1fabc5785"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"严格图导数的图差公式","source_units":[{"start":52787,"end":53071,"start_line":783,"end_line":786,"raw_tex":"To justify (b),  we conclude from Theorem~\\ref{thm:reduction} and \\cite[Proposition~4H.10]{DoR14}, respectively, that \n\\begin{equation}\\label{para}\n \\widetilde T_{\\gph \\partial g}(\\oz,\\olm)=\\gph D_*(\\sub  g)(\\oz,\\olm)= \\gph D_*N_{\\Kg}(0,0)=\\gph N_{\\Kg} - \\gph N_{\\Kg}.\n\\end{equation}\n","sha256_utf8":"9f2c9b9065c829d6218f31538c8af1cd15da28c6fc27a4e1d9c2231c1aff4975"}]},{"id":"S005","semantic_type":"proof_step","kind":"proof_step","title":"临界锥稳定确定正则切锥","source_units":[{"start":53072,"end":53632,"start_line":788,"end_line":795,"raw_tex":"Finally, we proceed with the proof of (c). Suppose that  $\\olm\\in \\ri \\sub g(\\oz)$.  By Corollary~\\ref{incri}(b), we find $r>0$ such that for any $(z,\\lm)\\in \\B_r(\\oz,\\olm)\\cap \\gph \\sub g$, the property \n $K_g(z,\\lm)=\\Kg$ holds. Take any such a pair $(z,\\lm) $ and conclude from part (a) that  $$T_{\\gph \\partial g}(z,\\lm)=  \\gph N_{K_g(z,\\lm)}=\\gph N_\\Kg.$$\n It follows then from \\cite[Theorem~6.26]{rw} that \n $$\n\\widehat T_{\\gph \\partial g}(\\oz,\\olm)=\\liminf_{\n  (z,\\lm)\\xrightarrow{   \\gph \\sub g}(\\oz,\\olm)}\n  T_{\\gph \\partial g}(z,\\lm)= \\gph N_\\Kg.\n $$\n","sha256_utf8":"6687c30850bab3ca8498cbb98cec98a69997260a5b285b072c544ba492902fcc"}]},{"id":"S006","semantic_type":"proof_step","kind":"proof_step","title":"子空间结构推出邻域严格原可微","source_units":[{"start":53632,"end":54488,"start_line":796,"end_line":800,"raw_tex":"Since $\\olm\\in \\ri \\sub g(\\oz)$, we conclude that $\\gph N_{\\Kg}=\\Kg\\times\\Kg^\\perp$ is a linear subspace. These, coupled with \\eqref{para}, lead us to \n $\\widehat T_{\\gph \\partial g}(\\oz,\\olm)= \\widetilde T_{\\gph \\partial g}(\\oz,\\olm)$, meaning that $\\sub g$ is strictly proto-differentiable at $\\oz$ for $\\olm$.\n To achieve a similar conclusion for any pair $(z,\\lm)\\in \\gph \\sub g$ sufficiently close to $(\\oz,\\olm)$, observe from Corollary~\\ref{incri}(b)\nthat for any such a pair, we have $\\lm\\in \\ri \\sub g(z)$. A similar argument as the one presented above for $(\\oz,\\olm)$, shows that $\\sub g$ is strictly   proto-differentiable at $z$ for $\\lm$\nwhenever $(z,\\lm)\\in \\gph \\sub g$  is sufficiently close to $(\\oz,\\olm)$. Finally,  \\eqref{gdpd} results from \\eqref{eqspd}. Conversely, if $\\sub g$ is strictly proto-differentiable at $\\oz$ for $\\olm$, \n","sha256_utf8":"0baef00f272bd7a082cab9fa757f5ed0d52e6c40f94f3fa052da7a70a68ce01b"}]},{"id":"S007","semantic_type":"proof_step","kind":"proof_step","title":"严格原可微反推相对内部条件","source_units":[{"start":54488,"end":54874,"start_line":801,"end_line":805,"raw_tex":"our discussion prior to \\eqref{eqspd} implies that\n$$\n\\gph N_{\\Kg}=T_{\\gph \\partial g}(\\oz,\\olm)= \\widetilde T_{\\gph \\partial g}(\\oz,\\olm)=\\gph N_{\\Kg}-\\gph N_{\\Kg},\n$$\nwhere the last equality results from \\eqref{para}. This clearly tells us that  $\\Kg$ is a linear subspace and hence it follows from Proposition~\\ref{nonde1} that  $\\olm\\in \\ri \\sub g(\\oz)$, which completes the proof.\n","sha256_utf8":"968ed7ea6ff8fe9e6aa3de24be8b314e555841ee13b37ab1af621b67b634c9c8"}]},{"id":"S008","semantic_type":"lemma","kind":"lemma","title":"分解无关的正系数指标稳定性","source_units":[{"start":18347,"end":18778,"start_line":316,"end_line":321,"raw_tex":"\\begin{Lemma}\\label{lem:indset} Assume that $g:\\R^m\\to \\oR$ is a polyhedral function and that  $(\\oz,\\olm)\\in\\gph\\sub g$. Then there exists $r>0$ such that for any decomposition of $\\olm$ as \n\\eqref{decomp} and any $(z, \\lm) \\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$, we have\n\\begin{equation}\\label{indset7}\n J_+(\\oz, \\osigma) \\subset J(z)\\; \\; \\textrm{ and }\\; \\; I_+(\\oz, \\otau) \\subset I(z). \n\\end{equation}\n\\end{Lemma}","sha256_utf8":"fdb8033a276361dc254cac988eb81c2c493dff9b72ab4ef152a0cfa0e7bbad63"}]},{"id":"S009","semantic_type":"proof","kind":"proof","title":"指标稳定性完整证明","source_units":[{"start":18780,"end":21825,"start_line":323,"end_line":361,"raw_tex":"\\begin{proof}\nSuppose by contradiction that for each $k\\in \\N$, there are a decomposition of $\\olm  $ as\n\\begin{equation}\\label{indset1}\n\\olm =  \\sum_{j\\in J(\\oz)}\\osigma_j^ka^j+ \\sum_{i\\in I(\\oz)}\\otau_i^kb^i\\; \\; \\text{ with }\\; \\;  \\osigma_j^k,\\otau_i^k\\ge 0\\; \\text{ and }\\; \\sum_{j\\in J(\\oz)}\\osigma_j^k=1,\n\\end{equation}\nand $(\\zk,\\lmk)\\in\\gph\\sub g$ such that $(\\zk,\\lmk)\\to(\\oz,\\olm)$ as $k\\to\\infty$ with $J_+^k\\not\\subset J(\\zk)$ or $I_+^k\\not\\subset I(\\zk)$, where $J_+^k:= J_+(\\oz,\\osigma^k)$ \nand $I_+^k:=I_+(\\oz,\\otau^k)$ are defined via   \\eqref{indset}. Since $\\zk \\to \\oz$ as $k\\to\\infty$, the inclusions  $J(\\zk)\\subset J(\\oz)$\n and $I(\\zk)\\subset I(\\oz)$ hold for all $k$ sufficiently large. Passing to a subsequence if necessary, we can assume that there exist subsets $\\bar J\\subset J(\\oz)$ and $\\bar I \\subset I(\\oz)$ such that\n\\begin{equation*}\nJ(\\zk)=\\bar J \\quad \\text{ and }\\quad I(\\zk)=\\bar I \\quad\\text{ for all }k\\in \\N,\n\\end{equation*}\nwhich, together with \\eqref{subg}, lead us to \n\\begin{equation*}\n\\partial g(\\zk) = \\co\\big\\{a^j|\\, j\\in \\bar J\\big\\}+\\cone\\big\\{b^i|\\, i\\in \\bar I\\big\\}=:\\Omega\\quad\\textrm{ for all } k\\in \\N.\n\\end{equation*}\nSince $\\lmk \\in \\partial g(\\zk) = \\Omega$ and $\\lmk\\to \\olm$ as $k\\to\\infty$, we arrive at  $\\olm\\in\\Omega =\\partial g(\\zk)$ for all  $k$ sufficiently large. Fix such a  $k\\in \\N$ and  deduce from $\\olm \\in \\partial g(\\zk)$ that\n\\begin{equation}\\label{indset3}\n\\la \\olm, \\oz-\\zk\\ra \\leq g(\\oz)-g(\\zk).\n\\end{equation}\nBy the decomposition in \\eqref{indset1}, we obtain  \n\\begin{align}\\label{indset2}\n\\la \\olm, \\zk - \\oz\\ra &= \\sum_{j\\in J(\\oz)} \\osigma_j^k\\la a^j, \\zk-\\oz\\ra +\\sum_{i\\in I(\\oz)}\\otau_i^k\\la b^i, \\zk - \\oz\\ra\\nonumber\\\\\n&= \\sum_{j\\in J(\\oz)} \\osigma_j^k\\big(\\la a^j, \\zk\\ra - \\alpha_j - g(\\oz)\\big) +\\sum_{i\\in  I(\\oz)}\\otau_i^k \\big(\\la b^i, \\zk\\ra - \\beta_i\\big),\\nonumber\\\\\n&\\leq \\sum_{j\\in J(\\oz)} \\osigma_j^k\\big(g(\\zk) - g(\\oz)\\big),\n\\end{align}\nwhere  the second equality results from  \\eqref{indset4} and the last inequality comes from  the fact that $\\zk\\in \\dom g$, combined with  \\eqref{dom}--\\eqref{polyfunc}. If $J_+^k\\not\\subset J(\\zk)$,  there exists   $\\oj\\in  J_+^k$ such that $\\zk\\notin C_\\oj$, meaning that \n\\begin{equation*}\\label{indset5}\n\\osigma_\\oj^k \\big(\\la a^\\oj, \\zk\\ra - \\alpha_\\oj - g(\\oz)\\big) < \\osigma_\\oj^k \\big(g(\\zk) - g(\\oz)\\big).\n\\end{equation*}\nIf $I_+^k\\not\\subset I(\\zk)$, we find  $\\oi\\in I_+^k$ such that $\\la b^\\oi, \\zk\\ra <\\beta_\\oi$, which implies that \n\\begin{equation*}\\label{indset6}\n\\otau_\\oi^k \\big(\\la b^\\oi, \\zk\\ra - \\beta_\\oi\\big) < 0,\n\\end{equation*}\nwhich tells us that  in both cases the last inequality in \\eqref{indset2} is strict. Thus, our assumption that either $J_+^k\\not\\subset J(\\zk)$ or $I_+^k\\not\\subset I(\\zk)$, together  with the last condition in \\eqref{indset1}, yields\n\\begin{equation*}\n\\la \\olm, \\zk - \\oz\\ra < \\sum_{j\\in J(\\oz)} \\osigma_j^k\\big(g(\\zk) - g(\\oz)\\big) = g(\\zk) - g(\\oz),\n\\end{equation*}\nwhich contradicts \\eqref{indset3} and therefore completes the proof.\n\\end{proof}","sha256_utf8":"9d0628e43c80574452804e8e6b0c0667d0bf638dbd2275a1403b8263cc6bfb9a"}]},{"id":"S010","semantic_type":"proof_step","kind":"proof_step","title":"反设并固定有限活动指标","source_units":[{"start":18794,"end":19955,"start_line":324,"end_line":337,"raw_tex":"Suppose by contradiction that for each $k\\in \\N$, there are a decomposition of $\\olm  $ as\n\\begin{equation}\\label{indset1}\n\\olm =  \\sum_{j\\in J(\\oz)}\\osigma_j^ka^j+ \\sum_{i\\in I(\\oz)}\\otau_i^kb^i\\; \\; \\text{ with }\\; \\;  \\osigma_j^k,\\otau_i^k\\ge 0\\; \\text{ and }\\; \\sum_{j\\in J(\\oz)}\\osigma_j^k=1,\n\\end{equation}\nand $(\\zk,\\lmk)\\in\\gph\\sub g$ such that $(\\zk,\\lmk)\\to(\\oz,\\olm)$ as $k\\to\\infty$ with $J_+^k\\not\\subset J(\\zk)$ or $I_+^k\\not\\subset I(\\zk)$, where $J_+^k:= J_+(\\oz,\\osigma^k)$ \nand $I_+^k:=I_+(\\oz,\\otau^k)$ are defined via   \\eqref{indset}. Since $\\zk \\to \\oz$ as $k\\to\\infty$, the inclusions  $J(\\zk)\\subset J(\\oz)$\n and $I(\\zk)\\subset I(\\oz)$ hold for all $k$ sufficiently large. Passing to a subsequence if necessary, we can assume that there exist subsets $\\bar J\\subset J(\\oz)$ and $\\bar I \\subset I(\\oz)$ such that\n\\begin{equation*}\nJ(\\zk)=\\bar J \\quad \\text{ and }\\quad I(\\zk)=\\bar I \\quad\\text{ for all }k\\in \\N,\n\\end{equation*}\nwhich, together with \\eqref{subg}, lead us to \n\\begin{equation*}\n\\partial g(\\zk) = \\co\\big\\{a^j|\\, j\\in \\bar J\\big\\}+\\cone\\big\\{b^i|\\, i\\in \\bar I\\big\\}=:\\Omega\\quad\\textrm{ for all } k\\in \\N.\n\\end{equation*}\n","sha256_utf8":"ac08d75839b4eb8ebe4560c472a56c61b7702c69c1d90ab69258a00944204a9a"}]},{"id":"S011","semantic_type":"proof_step","kind":"proof_step","title":"次梯度不等式与分解估计","source_units":[{"start":19955,"end":20723,"start_line":338,"end_line":347,"raw_tex":"Since $\\lmk \\in \\partial g(\\zk) = \\Omega$ and $\\lmk\\to \\olm$ as $k\\to\\infty$, we arrive at  $\\olm\\in\\Omega =\\partial g(\\zk)$ for all  $k$ sufficiently large. Fix such a  $k\\in \\N$ and  deduce from $\\olm \\in \\partial g(\\zk)$ that\n\\begin{equation}\\label{indset3}\n\\la \\olm, \\oz-\\zk\\ra \\leq g(\\oz)-g(\\zk).\n\\end{equation}\nBy the decomposition in \\eqref{indset1}, we obtain  \n\\begin{align}\\label{indset2}\n\\la \\olm, \\zk - \\oz\\ra &= \\sum_{j\\in J(\\oz)} \\osigma_j^k\\la a^j, \\zk-\\oz\\ra +\\sum_{i\\in I(\\oz)}\\otau_i^k\\la b^i, \\zk - \\oz\\ra\\nonumber\\\\\n&= \\sum_{j\\in J(\\oz)} \\osigma_j^k\\big(\\la a^j, \\zk\\ra - \\alpha_j - g(\\oz)\\big) +\\sum_{i\\in  I(\\oz)}\\otau_i^k \\big(\\la b^i, \\zk\\ra - \\beta_i\\big),\\nonumber\\\\\n&\\leq \\sum_{j\\in J(\\oz)} \\osigma_j^k\\big(g(\\zk) - g(\\oz)\\big),\n\\end{align}\n","sha256_utf8":"4a6e1e21df8aca5b367a464edd92159229ee03565d522b1651aee419410e4497"}]},{"id":"S012","semantic_type":"proof_step","kind":"proof_step","title":"遗漏正系数指标导致严格矛盾","source_units":[{"start":20723,"end":21814,"start_line":348,"end_line":360,"raw_tex":"where  the second equality results from  \\eqref{indset4} and the last inequality comes from  the fact that $\\zk\\in \\dom g$, combined with  \\eqref{dom}--\\eqref{polyfunc}. If $J_+^k\\not\\subset J(\\zk)$,  there exists   $\\oj\\in  J_+^k$ such that $\\zk\\notin C_\\oj$, meaning that \n\\begin{equation*}\\label{indset5}\n\\osigma_\\oj^k \\big(\\la a^\\oj, \\zk\\ra - \\alpha_\\oj - g(\\oz)\\big) < \\osigma_\\oj^k \\big(g(\\zk) - g(\\oz)\\big).\n\\end{equation*}\nIf $I_+^k\\not\\subset I(\\zk)$, we find  $\\oi\\in I_+^k$ such that $\\la b^\\oi, \\zk\\ra <\\beta_\\oi$, which implies that \n\\begin{equation*}\\label{indset6}\n\\otau_\\oi^k \\big(\\la b^\\oi, \\zk\\ra - \\beta_\\oi\\big) < 0,\n\\end{equation*}\nwhich tells us that  in both cases the last inequality in \\eqref{indset2} is strict. Thus, our assumption that either $J_+^k\\not\\subset J(\\zk)$ or $I_+^k\\not\\subset I(\\zk)$, together  with the last condition in \\eqref{indset1}, yields\n\\begin{equation*}\n\\la \\olm, \\zk - \\oz\\ra < \\sum_{j\\in J(\\oz)} \\osigma_j^k\\big(g(\\zk) - g(\\oz)\\big) = g(\\zk) - g(\\oz),\n\\end{equation*}\nwhich contradicts \\eqref{indset3} and therefore completes the proof.\n","sha256_utf8":"443b32350f1c9564c13d5893e4428390033dbfed829e1d2c51ebe49b602d8e25"}]},{"id":"S013","semantic_type":"proposition","kind":"proposition","title":"一阶次导数与局部精确展开","source_units":[{"start":22227,"end":22994,"start_line":369,"end_line":386,"raw_tex":"\\begin{Proposition}[first-order variational properties] \\label{fopg}\nAssume that $g:\\R^m\\to \\oR$ is a polyhedral function and $\\oz\\in \\dom g$. Then the following properties  hold.\n\\begin{itemize}[noitemsep,topsep=2pt]\n\\item [ \\rm {(a)}] The domain of the subderivative function $\\d g(\\oz)$ can be calculated by \n\\begin{equation*}\\label{domdg}\n\\dom \\d g(\\oz) = T_{\\dom g}(\\oz) = \\bigcup_{j\\in J(\\oz)} T_{C_j}(\\oz).\n\\end{equation*}\n\\item [ \\rm {(b)}] If $w\\in T_{C_j}(\\oz)$ for some $j\\in J(\\oz)$, then we have \n$\\d g(\\oz) (w) =\\la a^j, w\\ra$. \nMoreover, there exists $r> 0$ such that  any  $w\\in T_{\\dom  g}(\\oz)\\cap \\B_r(0)$, the representation \n\\begin{equation}\\label{approx}\ng(\\oz + w) = g(\\oz) +\\d g(\\oz)(w)\n\\end{equation}\nholds.\n\\end{itemize}\n \n\n\\end{Proposition}","sha256_utf8":"eed3dc3b80bcd0862551a07e5aae633df65129b2197681723a86526edd6dd59f"}]},{"id":"S014","semantic_type":"proof","kind":"proof","title":"一阶变分性质完整证明","source_units":[{"start":22997,"end":24033,"start_line":389,"end_line":400,"raw_tex":"\\begin{proof} The first equality in (a) was established in \\cite[Proposition~10.21]{rw}. The second equality results immediately from the fact that $\\dom g=\\bigcup_{j\\in J} C_j$.\nThe first claim in  (b) can be found again in \\cite[Proposition~10.21]{rw}. To prove  \\eqref{approx}, recall that for any $j\\in J(\\oz)$,    $C_j$ is  a polyhedral convex set. Employing \\eqref{tanrep} for these sets, we can find $r>0$ such that\n\\begin{equation*}\nT_{C_j}(\\oz)\\cap \\B_r(0)=(C_j-\\oz)\\cap \\B_r(0)\\quad\\textrm{ for all }\\; j\\in J(\\oz).\n\\end{equation*}\nPick any $w \\in T_{\\dom g}(\\oz)\\cap \\B_r(0)$,  and conclude from  (a) that $w\\in T_{C_\\oj}(\\oz)\\cap \\B_r(0)$ for some $\\oj \\in J(\\oz)$ and from the first claim in (b) that $\\d g(\\oz)(w) =\\la a^\\oj, w\\ra$. Thus, we get    $\\oz + w \\in C_\\oj$, which, together with the definition of $C_{j_0}$ and \\eqref{indset4}, \nleads us to \n\\begin{equation*}\ng(\\oz+ w) = \\la a^\\oj, \\oz+w\\ra-\\alpha_\\oj = g(\\oz) +\\la a^\\oj, w\\ra = g(\\oz)+\\d g(\\oz)(w),\n\\end{equation*}\nand hence completes the proof.\n\\end{proof}","sha256_utf8":"8bf2c2c10c019471de77678e359204d2a5f04549f34235aaac738b2214f9fb7e"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"次导数公式与共同切锥邻域","source_units":[{"start":22997,"end":23539,"start_line":389,"end_line":393,"raw_tex":"\\begin{proof} The first equality in (a) was established in \\cite[Proposition~10.21]{rw}. The second equality results immediately from the fact that $\\dom g=\\bigcup_{j\\in J} C_j$.\nThe first claim in  (b) can be found again in \\cite[Proposition~10.21]{rw}. To prove  \\eqref{approx}, recall that for any $j\\in J(\\oz)$,    $C_j$ is  a polyhedral convex set. Employing \\eqref{tanrep} for these sets, we can find $r>0$ such that\n\\begin{equation*}\nT_{C_j}(\\oz)\\cap \\B_r(0)=(C_j-\\oz)\\cap \\B_r(0)\\quad\\textrm{ for all }\\; j\\in J(\\oz).\n\\end{equation*}\n","sha256_utf8":"32d893e0cc72cc848d1cb7882599f388decf52de09f4f6b68090ae5b6654755f"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"选取活动胞腔得到精确展开","source_units":[{"start":23539,"end":24022,"start_line":394,"end_line":399,"raw_tex":"Pick any $w \\in T_{\\dom g}(\\oz)\\cap \\B_r(0)$,  and conclude from  (a) that $w\\in T_{C_\\oj}(\\oz)\\cap \\B_r(0)$ for some $\\oj \\in J(\\oz)$ and from the first claim in (b) that $\\d g(\\oz)(w) =\\la a^\\oj, w\\ra$. Thus, we get    $\\oz + w \\in C_\\oj$, which, together with the definition of $C_{j_0}$ and \\eqref{indset4}, \nleads us to \n\\begin{equation*}\ng(\\oz+ w) = \\la a^\\oj, \\oz+w\\ra-\\alpha_\\oj = g(\\oz) +\\la a^\\oj, w\\ra = g(\\oz)+\\d g(\\oz)(w),\n\\end{equation*}\nand hence completes the proof.\n","sha256_utf8":"49232a2ac2949e9d3e7e22808cc4f6858699c6cfd69537b90b9d64b71ece12ac"}]},{"id":"S017","semantic_type":"proposition","kind":"proposition","title":"相对内部等价于临界锥为子空间","source_units":[{"start":25312,"end":25676,"start_line":427,"end_line":432,"raw_tex":"\\begin{Proposition} \\label{nonde1} Assume that $g:\\R^m\\to \\oR$ is a polyhedral function and $(\\oz, \\olm)\\in \\gph \\partial g$. Then the following properties are equivalent:\n\\begin{itemize}[noitemsep,topsep=2pt]\n\\item [ \\rm {(a)}]  $\\olm\\in \\ri \\sub g(\\oz)$;\n\\item [ \\rm {(b)}]  the critical cone $K_g(\\oz,\\olm)$ is a linear subspace.\n\\end{itemize}\n\\end{Proposition}","sha256_utf8":"96ad059a74e381220ae32272a9f7db0f12325909718184e52480a51e427f5b70"}]},{"id":"S018","semantic_type":"proof","kind":"proof","title":"临界锥子空间刻画完整证明","source_units":[{"start":25677,"end":25873,"start_line":433,"end_line":434,"raw_tex":"\\begin{proof} By  \\eqref{cricone1}, we have $\\Kg=N_{\\sub g(\\oz)}(\\olm)$. The equivalence of (a) and $N_{\\sub g(\\oz)}(\\olm)$ being a subspace is a well-known fact from convex analysis. \n\\end{proof}","sha256_utf8":"00775d082d3d4a499489b89e824a0e0e23e9a12bbe52e65b9b55c6bd1777dbcc"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"应用凸集法锥的相对内部准则","source_units":[{"start":25677,"end":25862,"start_line":433,"end_line":433,"raw_tex":"\\begin{proof} By  \\eqref{cricone1}, we have $\\Kg=N_{\\sub g(\\oz)}(\\olm)$. The equivalence of (a) and $N_{\\sub g(\\oz)}(\\olm)$ being a subspace is a well-known fact from convex analysis. \n","sha256_utf8":"f576e5f63a5949df3dbaa30add1a329dc3abee1cefc9769d899200dc5eefe4d8"}]},{"id":"S020","semantic_type":"proposition","kind":"proposition","title":"临界锥的活动指标表达","source_units":[{"start":26008,"end":26728,"start_line":437,"end_line":448,"raw_tex":"\\begin{Proposition} [critical cone of polyhedral functions] \\label{prop:cricone} \nAssume that $g:\\R^m\\to \\oR$ is a polyhedral function and $(\\oz, \\olm)\\in \\gph \\partial g$. If $\\olm $ has the decomposition in  \\eqref{decomp}, then  $w\\in \\Kg$ if and only if \n$w$ satisfies the conditions \n\\begin{equation*}\\label{cricone3}\n\\begin{cases}\n\\la a^i-a^j, w\\ra = 0,\\quad&\\textrm{for }\\; i,j \\in J_+(\\oz, \\osigma),\\\\\n\\la a^i-a^j, w\\ra \\leq 0 ,&\\textrm{for }\\; i \\in J(\\oz)\\setminus J_+(\\oz, \\osigma)\\; \\textrm{ and }\\; j \\in J_+(\\oz, \\osigma),\\\\\n\\la b^i, w\\ra = 0,&\\textrm{for }\\; i \\in I_+(\\oz, \\otau),\\\\\n\\la b^i, w\\ra\\leq 0,&\\textrm{for }\\; i \\in I(\\oz)\\setminus I_+(\\oz, \\otau).\n\\end{cases}\n\\end{equation*}\n\\end{Proposition}","sha256_utf8":"14b24fa49da507e37a94d65b6ccaa43ff7c35deb3c491a4470796a6464f7f82a"}]},{"id":"S021","semantic_type":"theorem","kind":"theorem","title":"次微分图的局部约化","source_units":[{"start":28301,"end":28635,"start_line":466,"end_line":471,"raw_tex":"\\begin{Theorem}[reduction lemma]\\label{thm:reduction}\nAssume that $g:\\R^m\\to \\oR$ is a polyhedral function and $(\\oz, \\olm)\\in \\gph \\partial g$.  Then there exists $r>0$ such that\n\\begin{equation}\\label{reduction}\n\\big( (\\gph \\sub g ) - (\\oz, \\olm)\\big)\\cap \\B_r(0,0) = \\big(\\gph N_\\Kg\\big)\\cap \\B_r(0,0).\n\\end{equation}\n\\end{Theorem}","sha256_utf8":"f96ac7bac350852a302ddc5cdfea984b34bef3bec8d9746db8890333c630d8ca"}]},{"id":"S022","semantic_type":"proof","kind":"proof","title":"约化引理完整证明","source_units":[{"start":28637,"end":35314,"start_line":473,"end_line":542,"raw_tex":"\\begin{proof}\nSuppose that the polyhedral function $g$  has the representation   \\eqref{polyfunc}. We first show that there is $r>0$ for which\nthe inclusion `$\\supset$' in \\eqref{reduction} holds. To do so, using  \\eqref{tanrep} for the polyhedral convex set $\\partial g(\\oz)$, we find $r_0>0$ such that \n\\begin{equation}\\label{relem1}\nT_{\\partial g(\\oz)}(\\olm)\\cap \\B_{r_0}(0) = \\big(\\partial g(\\oz) -\\olm\\big)\\cap \\B_{r_0}(0).\n\\end{equation}\nPick now a pair $(w, u)$ from the right-hand side of \\eqref{reduction} with $r=r_0$ therein, which implies that  the conditions $w\\in \\Kg\\cap\\B_{r_0}(0)$, $u \\in \\Kg^*\\cap \\B_{r_0}(0)$, and $\\la u, w\\ra =0$\nare satisfied. Moreover, using \\eqref{cricone1} for the polyhedral function $g$, we get $\\Kg^*= T_{\\partial g(\\oz)}(\\olm)$. Therefore $u$ belongs to the left-hand side of \\eqref{relem1}, and thus $\\olm+u \\in\\partial g(\\oz)$. We now show that $(\\oz +w, \\olm +u) \\in\\gph\\partial g$. To this end, take $z\\in \\R^m$   and observe that \n\\begin{equation}\\label{relem2}\n\\la \\olm+u, z- (\\oz+w)\\ra = \\la \\olm+u, z-\\oz\\ra -\\la\\olm, w\\ra - \\la u, w\\ra\\leq g(z)-g(\\oz) - \\d g(\\oz)(w),\n\\end{equation}\nwhere the last inequality is deduced from the facts that $\\olm +u \\in \\partial g(\\oz)$, $w\\in \\Kg$, and $\\la u, w\\ra =0$. Choosing a smaller radius $r_0$ if necessary, we can assume that  \\eqref{approx} is valid on\n $T_{\\dom g}(\\oz)\\cap \\B_{r_0}(0)$. It follows from Proposition~\\ref{fopg}(a) and $w\\in \\Kg\\cap \\B_{r_0}(0)$ that $w\\in T_{\\dom g}(\\oz)\\cap \\B_{r_0}(0)$, which in turn allows us to conclude via \\eqref{approx} that\n  $g(\\oz)+\\d g(\\oz)(w) = g(\\oz+w)$. Combining this and  \\eqref{relem2} leads us to \n\\begin{equation*}\n\\la \\olm+u, z- (\\oz+w)\\ra \\leq g(z)-g(\\oz+w)\n\\end{equation*}\nfor any arbitrary $z\\in \\R^m$, and thus  to $\\olm+u \\in \\partial g(\\oz+w)$. This proves that  $(w, u)$ belongs to the left-hand side of \\eqref{reduction} with $r=r_0$.\n\n \n\nWe now proceed with proving the inclusion `$\\subset$' in \\eqref{reduction} for some $r>0$. Decompose $\\olm$ into $\\olm = \\olm_1+\\olm_2$ with $\\olm_1, \\olm_2$ taken from  \\eqref{decomp}. \nPick $r_0$ from \\eqref{relem1} and choose $r\\in (0, r_0]$  such that the inclusions in \\eqref{indset7} hold for all $(z, v)\\in \\big(\\gph\\partial g\\big)\\cap\\B_{r}(\\oz, \\olm)$. Pick now a pair $(w, v)$ from the left-hand side of \\eqref{reduction}, and conclude that  \n $(w, u) \\in \\B_{r}(0, 0)$ with $\\olm+u \\in \\partial g(\\oz+w)$. We are going to show that  $u\\in N_\\Kg(w)$, which amounts via \\cite[Proposition~2A.3]{DoR14} to the conditions\n\\begin{equation}\\label{relem4}\nw\\in \\Kg, \\quad u\\in \\Kg^*, \\quad\\textrm{ and }\\quad \\la u, w\\ra =0.\n\\end{equation}\nTaking a smaller radius $r$ if necessary, we can assume that $\\partial g(\\oz+w)\\subset \\partial g(\\oz)$ (cf. \\cite[Proposition~3.3(a)]{MoS16}). Thus, $u \\in \\big(\\partial g(\\oz) - \\olm\\big)\\cap \\B_{r}(0)$. Since $r\\leq r_0$, we deduce from  \\eqref{relem1} that \n $u\\in T_{\\partial g(\\oz)}(\\olm)= \\Kg^*$, which justifies the second inclusion  in \\eqref{relem4}. By shrinking $r$ again if necessary, we can derive  from  $(\\oz +w, \\olm+u) \\in  \\big(\\gph\\partial g\\big)\\cap\\B_{r}(\\oz, \\olm)$ and \\eqref{indset7} that \n\\begin{equation*}\nJ_+(\\oz, \\osigma) \\subset J(\\oz+w)\\subset J(\\oz) \\quad\\textrm{ and }\\quad I_+(\\oz, \\otau)\\subset I(\\oz+w)\\subset I(\\oz).\n\\end{equation*}\n To prove that $w\\in \\Kg$, we use  Proposition~\\ref{prop:cricone} in which an equivalent description of $\\Kg$ was given. We break this task into four cases as follows:\n\\begin{itemize}[noitemsep,topsep=2pt]\n\\item [ \\rm {(i)}]   $i, j \\in J_+(\\oz, \\osigma)$. In this case, we obtain from  the inclusions above  that $i, j \\in  J(\\oz+w)\\subset J(\\oz)$. These inclusions bring us via \\eqref{indset4} to  \n\\begin{equation*}\n\\la a^i, \\oz\\ra -\\alpha_i = \\la a^j, \\oz\\ra -\\alpha_j\\quad\\textrm{ and }\\quad\n\\la a^i, \\oz+w\\ra -\\alpha_i = \\la a^j, \\oz+w\\ra -\\alpha_j.\n\\end{equation*}\nCombining  these confirms that $\\la a^i-a^j, w\\ra  =0$ for all $i, j \\in J_+(\\oz, \\osigma)$.\n\n\\item [ (ii)]  $i\\in J(\\oz)\\setminus J_+(\\oz, \\osigma)$ and $j\\in J_+(\\oz, \\osigma)$. In this case, we arrive at $i, j\\in J(\\oz)$ and $j\\in J(\\oz+v)$, which imply via \\eqref{indset4} that  \n\\begin{equation*}\n\\la a^i, \\oz\\ra -\\alpha_i = \\la a^j, \\oz\\ra -\\alpha_j\\quad\\textrm{ and }\\quad\n\\la a^i, \\oz+w\\ra -\\alpha_i \\leq \\la a^j, \\oz+w\\ra -\\alpha_j.\n\\end{equation*}\nCombining  these confirms that $\\la a^i-a^j, w\\ra  \\le 0$ for all $i\\in J(\\oz)\\setminus J_+(\\oz, \\osigma)$ and $j\\in J_+(\\oz, \\osigma)$.\n\n\\item [{(iii)}]  $i\\in I_+(\\oz, \\otau)$. In this case, we have $i\\in I(\\oz)$ and $i \\in I(\\oz+w)$, which result in $\\la b^i, \\oz\\ra =\\la b^i, \\oz+w\\ra=\\beta_i$. Combining  these confirms that $\\la b^i, w\\ra=0$ for all $i\\in I_+(\\oz, \\otau)$.\n\n\\item[(iv)]  $i\\in I(\\oz)\\setminus I_+(\\oz, \\otau)$. In this case,  we deduce from $\\oz+w\\in \\dom g$ and $i\\in I(\\oz)$ that $\\la b^i, \\oz+w\\ra \\leq \\beta_i=\\la b^i, \\oz\\ra$, which in turn yields  \n$\\la b^i, w\\ra\\le 0$ for all $i\\in I(\\oz)\\setminus I_+(\\oz, \\otau)$.\n\\end{itemize}\nIn summary, we showed that $w$ satisfies the equivalent description of $\\Kg$ from Proposition~\\ref{prop:cricone}, and so $w\\in \\Kg$. \nIt remains to demonstrate that $\\la w,u\\ra=0$. Recall that $\\olm+u\\in \\partial g(\\oz+w)$. Using \\eqref{subg}, we can express $\\olm+u$ as\n\\begin{equation*}\\label{relem5}\n\\olm + u = \\sum_{j\\in J(\\oz+w)} \\sigma_j a^j+ \\sum_{i\\in I(\\oz+w)} \\tau_ib^i\\; \\; \\textrm{ with }\\; \\; \\sigma_j, \\tau_i \\geq 0\\; \\textrm{ and }\\;  \\sum_{j\\in J(\\oz+w)} \\sigma_j =1.\n\\end{equation*}\nThis and the decomposition $\\olm = \\olm_1+\\olm_2$ with $\\olm_1,\\olm_2$ taken from \\eqref{decomp} allow us to conclude that  \n\\begin{align*}\n\\la u, w\\ra &= \\la \\olm+u, w\\ra-\\la \\olm, w\\ra\\\\\n&= \\sum_{j\\in J(\\oz+w)} (\\sigma_j-\\osigma_j)\\la a^j, w\\ra+ \\sum_{i\\in I(\\oz+w)} (\\tau_i-\\otau_i)\\la b^i, w\\ra\\\\\n&= \\sum_{j\\in J(\\oz+w)} (\\sigma_j-\\osigma_j)\\big[\\la a^j, \\oz+w\\ra-\\alpha_j -(\\la a^j, \\oz\\ra-\\alpha_j)\\big]+ \\sum_{i\\in I(\\oz+w)} (\\tau_i-\\otau_i)\\big[\\la b^i, \\oz+w\\ra - \\la b^i, \\oz\\ra\\big]\\\\\n&= \\big(g(\\oz+w)-g(\\oz)\\big)\\sum_{j\\in J(\\oz+w)} (\\sigma_j-\\osigma_j)+ \\sum_{i\\in I(\\oz+w)} (\\tau_i-\\otau_i)(\\beta_i-\\beta_i)\\\\\n&=0,\n\\end{align*}\nwhere  the second equality  results from the inclusions $J_+(\\oz, \\osigma)\\subset J(\\oz+w)$ and  $ I_+(\\oz, \\otau)\\subset I(\\oz+w)$, \n where   the fourth equality comes from the  definitions of the active index sets $J(\\oz+w)$ and $I(\\oz+w)$ as well as the inclusions $J(\\oz+w)\\subset J(\\oz)$ and $I(\\oz+w)\\subset I(\\oz)$, \n and where  the last equality follows from the assumption on $\\sigma_j$ and $\\osigma_j$. This proves the inclusion `$\\subset$' in \\eqref{reduction} for some $r>0$ and thus completes the proof. \n\\end{proof}","sha256_utf8":"4eaff9c8a4a00b8ecee395a1513b87abe8905b29ca884b3be7040fbfb0afea14"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"由临界锥法向量恢复基点次梯度","source_units":[{"start":28651,"end":29619,"start_line":474,"end_line":480,"raw_tex":"Suppose that the polyhedral function $g$  has the representation   \\eqref{polyfunc}. We first show that there is $r>0$ for which\nthe inclusion `$\\supset$' in \\eqref{reduction} holds. To do so, using  \\eqref{tanrep} for the polyhedral convex set $\\partial g(\\oz)$, we find $r_0>0$ such that \n\\begin{equation}\\label{relem1}\nT_{\\partial g(\\oz)}(\\olm)\\cap \\B_{r_0}(0) = \\big(\\partial g(\\oz) -\\olm\\big)\\cap \\B_{r_0}(0).\n\\end{equation}\nPick now a pair $(w, u)$ from the right-hand side of \\eqref{reduction} with $r=r_0$ therein, which implies that  the conditions $w\\in \\Kg\\cap\\B_{r_0}(0)$, $u \\in \\Kg^*\\cap \\B_{r_0}(0)$, and $\\la u, w\\ra =0$\nare satisfied. Moreover, using \\eqref{cricone1} for the polyhedral function $g$, we get $\\Kg^*= T_{\\partial g(\\oz)}(\\olm)$. Therefore $u$ belongs to the left-hand side of \\eqref{relem1}, and thus $\\olm+u \\in\\partial g(\\oz)$. We now show that $(\\oz +w, \\olm +u) \\in\\gph\\partial g$. To this end, take $z\\in \\R^m$   and observe that \n","sha256_utf8":"6b370a442454460b7d320cab21ddd7b1525f1bdf3054c416d23d7abd73c3a3c7"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"精确展开证明第一方向包含","source_units":[{"start":29619,"end":30535,"start_line":481,"end_line":490,"raw_tex":"\\begin{equation}\\label{relem2}\n\\la \\olm+u, z- (\\oz+w)\\ra = \\la \\olm+u, z-\\oz\\ra -\\la\\olm, w\\ra - \\la u, w\\ra\\leq g(z)-g(\\oz) - \\d g(\\oz)(w),\n\\end{equation}\nwhere the last inequality is deduced from the facts that $\\olm +u \\in \\partial g(\\oz)$, $w\\in \\Kg$, and $\\la u, w\\ra =0$. Choosing a smaller radius $r_0$ if necessary, we can assume that  \\eqref{approx} is valid on\n $T_{\\dom g}(\\oz)\\cap \\B_{r_0}(0)$. It follows from Proposition~\\ref{fopg}(a) and $w\\in \\Kg\\cap \\B_{r_0}(0)$ that $w\\in T_{\\dom g}(\\oz)\\cap \\B_{r_0}(0)$, which in turn allows us to conclude via \\eqref{approx} that\n  $g(\\oz)+\\d g(\\oz)(w) = g(\\oz+w)$. Combining this and  \\eqref{relem2} leads us to \n\\begin{equation*}\n\\la \\olm+u, z- (\\oz+w)\\ra \\leq g(z)-g(\\oz+w)\n\\end{equation*}\nfor any arbitrary $z\\in \\R^m$, and thus  to $\\olm+u \\in \\partial g(\\oz+w)$. This proves that  $(w, u)$ belongs to the left-hand side of \\eqref{reduction} with $r=r_0$.\n","sha256_utf8":"ca50cee9f1d9148a067bf7084e5506d173b06829fd75d9188a81f44caf38f8ef"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"反向包含的极锥与指标条件","source_units":[{"start":30539,"end":31952,"start_line":494,"end_line":504,"raw_tex":"We now proceed with proving the inclusion `$\\subset$' in \\eqref{reduction} for some $r>0$. Decompose $\\olm$ into $\\olm = \\olm_1+\\olm_2$ with $\\olm_1, \\olm_2$ taken from  \\eqref{decomp}. \nPick $r_0$ from \\eqref{relem1} and choose $r\\in (0, r_0]$  such that the inclusions in \\eqref{indset7} hold for all $(z, v)\\in \\big(\\gph\\partial g\\big)\\cap\\B_{r}(\\oz, \\olm)$. Pick now a pair $(w, v)$ from the left-hand side of \\eqref{reduction}, and conclude that  \n $(w, u) \\in \\B_{r}(0, 0)$ with $\\olm+u \\in \\partial g(\\oz+w)$. We are going to show that  $u\\in N_\\Kg(w)$, which amounts via \\cite[Proposition~2A.3]{DoR14} to the conditions\n\\begin{equation}\\label{relem4}\nw\\in \\Kg, \\quad u\\in \\Kg^*, \\quad\\textrm{ and }\\quad \\la u, w\\ra =0.\n\\end{equation}\nTaking a smaller radius $r$ if necessary, we can assume that $\\partial g(\\oz+w)\\subset \\partial g(\\oz)$ (cf. \\cite[Proposition~3.3(a)]{MoS16}). Thus, $u \\in \\big(\\partial g(\\oz) - \\olm\\big)\\cap \\B_{r}(0)$. Since $r\\leq r_0$, we deduce from  \\eqref{relem1} that \n $u\\in T_{\\partial g(\\oz)}(\\olm)= \\Kg^*$, which justifies the second inclusion  in \\eqref{relem4}. By shrinking $r$ again if necessary, we can derive  from  $(\\oz +w, \\olm+u) \\in  \\big(\\gph\\partial g\\big)\\cap\\B_{r}(\\oz, \\olm)$ and \\eqref{indset7} that \n\\begin{equation*}\nJ_+(\\oz, \\osigma) \\subset J(\\oz+w)\\subset J(\\oz) \\quad\\textrm{ and }\\quad I_+(\\oz, \\otau)\\subset I(\\oz+w)\\subset I(\\oz).\n\\end{equation*}\n","sha256_utf8":"692d6593a679edb74d76aba6919e74d11dde22735ae000a4e31a266ac3e06d25"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"正系数仿射指标给出等式","source_units":[{"start":31952,"end":32617,"start_line":505,"end_line":512,"raw_tex":" To prove that $w\\in \\Kg$, we use  Proposition~\\ref{prop:cricone} in which an equivalent description of $\\Kg$ was given. We break this task into four cases as follows:\n\\begin{itemize}[noitemsep,topsep=2pt]\n\\item [ \\rm {(i)}]   $i, j \\in J_+(\\oz, \\osigma)$. In this case, we obtain from  the inclusions above  that $i, j \\in  J(\\oz+w)\\subset J(\\oz)$. These inclusions bring us via \\eqref{indset4} to  \n\\begin{equation*}\n\\la a^i, \\oz\\ra -\\alpha_i = \\la a^j, \\oz\\ra -\\alpha_j\\quad\\textrm{ and }\\quad\n\\la a^i, \\oz+w\\ra -\\alpha_i = \\la a^j, \\oz+w\\ra -\\alpha_j.\n\\end{equation*}\nCombining  these confirms that $\\la a^i-a^j, w\\ra  =0$ for all $i, j \\in J_+(\\oz, \\osigma)$.\n","sha256_utf8":"57baad1adad6a4e3f7982774b9ccecb102e9c46d4258edff2d59bf52801b8ab5"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"其余仿射指标给出不等式","source_units":[{"start":32618,"end":33119,"start_line":514,"end_line":519,"raw_tex":"\\item [ (ii)]  $i\\in J(\\oz)\\setminus J_+(\\oz, \\osigma)$ and $j\\in J_+(\\oz, \\osigma)$. In this case, we arrive at $i, j\\in J(\\oz)$ and $j\\in J(\\oz+v)$, which imply via \\eqref{indset4} that  \n\\begin{equation*}\n\\la a^i, \\oz\\ra -\\alpha_i = \\la a^j, \\oz\\ra -\\alpha_j\\quad\\textrm{ and }\\quad\n\\la a^i, \\oz+w\\ra -\\alpha_i \\leq \\la a^j, \\oz+w\\ra -\\alpha_j.\n\\end{equation*}\nCombining  these confirms that $\\la a^i-a^j, w\\ra  \\le 0$ for all $i\\in J(\\oz)\\setminus J_+(\\oz, \\osigma)$ and $j\\in J_+(\\oz, \\osigma)$.\n","sha256_utf8":"8a828c1237a6c72184888631c65c50b62fbb6717b7194b717ba1f56f9def3319"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"正系数约束指标给出正交性","source_units":[{"start":33120,"end":33362,"start_line":521,"end_line":521,"raw_tex":"\\item [{(iii)}]  $i\\in I_+(\\oz, \\otau)$. In this case, we have $i\\in I(\\oz)$ and $i \\in I(\\oz+w)$, which result in $\\la b^i, \\oz\\ra =\\la b^i, \\oz+w\\ra=\\beta_i$. Combining  these confirms that $\\la b^i, w\\ra=0$ for all $i\\in I_+(\\oz, \\otau)$.\n","sha256_utf8":"6f80a5c54cfd5cb8fe6b751228398ce307d5609ad2bf514328aa9874f2f9c6e6"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"其余约束完成临界锥成员证明","source_units":[{"start":33363,"end":33777,"start_line":523,"end_line":526,"raw_tex":"\\item[(iv)]  $i\\in I(\\oz)\\setminus I_+(\\oz, \\otau)$. In this case,  we deduce from $\\oz+w\\in \\dom g$ and $i\\in I(\\oz)$ that $\\la b^i, \\oz+w\\ra \\leq \\beta_i=\\la b^i, \\oz\\ra$, which in turn yields  \n$\\la b^i, w\\ra\\le 0$ for all $i\\in I(\\oz)\\setminus I_+(\\oz, \\otau)$.\n\\end{itemize}\nIn summary, we showed that $w$ satisfies the equivalent description of $\\Kg$ from Proposition~\\ref{prop:cricone}, and so $w\\in \\Kg$. \n","sha256_utf8":"b57493a740fe34931d8b7fb198bc09e361d793f52620afcf3e32ef2147a924a2"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"比较次梯度分解证明正交性","source_units":[{"start":33777,"end":35303,"start_line":527,"end_line":541,"raw_tex":"It remains to demonstrate that $\\la w,u\\ra=0$. Recall that $\\olm+u\\in \\partial g(\\oz+w)$. Using \\eqref{subg}, we can express $\\olm+u$ as\n\\begin{equation*}\\label{relem5}\n\\olm + u = \\sum_{j\\in J(\\oz+w)} \\sigma_j a^j+ \\sum_{i\\in I(\\oz+w)} \\tau_ib^i\\; \\; \\textrm{ with }\\; \\; \\sigma_j, \\tau_i \\geq 0\\; \\textrm{ and }\\;  \\sum_{j\\in J(\\oz+w)} \\sigma_j =1.\n\\end{equation*}\nThis and the decomposition $\\olm = \\olm_1+\\olm_2$ with $\\olm_1,\\olm_2$ taken from \\eqref{decomp} allow us to conclude that  \n\\begin{align*}\n\\la u, w\\ra &= \\la \\olm+u, w\\ra-\\la \\olm, w\\ra\\\\\n&= \\sum_{j\\in J(\\oz+w)} (\\sigma_j-\\osigma_j)\\la a^j, w\\ra+ \\sum_{i\\in I(\\oz+w)} (\\tau_i-\\otau_i)\\la b^i, w\\ra\\\\\n&= \\sum_{j\\in J(\\oz+w)} (\\sigma_j-\\osigma_j)\\big[\\la a^j, \\oz+w\\ra-\\alpha_j -(\\la a^j, \\oz\\ra-\\alpha_j)\\big]+ \\sum_{i\\in I(\\oz+w)} (\\tau_i-\\otau_i)\\big[\\la b^i, \\oz+w\\ra - \\la b^i, \\oz\\ra\\big]\\\\\n&= \\big(g(\\oz+w)-g(\\oz)\\big)\\sum_{j\\in J(\\oz+w)} (\\sigma_j-\\osigma_j)+ \\sum_{i\\in I(\\oz+w)} (\\tau_i-\\otau_i)(\\beta_i-\\beta_i)\\\\\n&=0,\n\\end{align*}\nwhere  the second equality  results from the inclusions $J_+(\\oz, \\osigma)\\subset J(\\oz+w)$ and  $ I_+(\\oz, \\otau)\\subset I(\\oz+w)$, \n where   the fourth equality comes from the  definitions of the active index sets $J(\\oz+w)$ and $I(\\oz+w)$ as well as the inclusions $J(\\oz+w)\\subset J(\\oz)$ and $I(\\oz+w)\\subset I(\\oz)$, \n and where  the last equality follows from the assumption on $\\sigma_j$ and $\\osigma_j$. This proves the inclusion `$\\subset$' in \\eqref{reduction} for some $r>0$ and thus completes the proof. \n","sha256_utf8":"2b3ba5fbdf1f8f93e42ae55da511fe0cb56382be2989590dca2afd217f84f0a3"}]},{"id":"S031","semantic_type":"corollary","kind":"corollary","title":"共轭临界锥的极性关系","source_units":[{"start":36451,"end":36910,"start_line":556,"end_line":565,"raw_tex":"\\begin{Corollary}[polar relation of critical cones] \\label{polarity}\nAssume that $g:\\R^m\\to \\oR$ is a polyhedral function and $(\\oz, \\olm)\\in \\gph \\partial g$.  Then $\\Kg$ enjoys the duality relationship \n\\begin{equation}\\label{cricone2}\n\\Kg =  K_{g^*}(\\olm, \\oz)^*.\n\\end{equation}\nConsequently, we have \n\\begin{equation}\\label{ricond}\n\\olm \\in \\ri \\partial g(\\oz) \\; \\textrm{ if and only if } \\; \\oz \\in \\ri \\partial g^*(\\olm).\n\\end{equation}\n\\end{Corollary}","sha256_utf8":"d339c78b3f0acbb000402bc8f3e66e65d74b702f5220b89de0a72659c129c67e"}]},{"id":"S032","semantic_type":"proof","kind":"proof","title":"临界锥极性完整证明","source_units":[{"start":36912,"end":38140,"start_line":567,"end_line":580,"raw_tex":"\\begin{proof}\nLet $r>0$ be such that  \\eqref{reduction} holds. Then, we conclude from \\eqref{cricone1} that \n\\begin{align*}\nw \\in \\Kg \\; &\\Longleftrightarrow\\; (w, 0)\\in\\gph N_\\Kg\\\\\n&\\Longleftrightarrow\\; \\exists\\, r'>0: t(w, 0)\\in\\big(\\gph N_\\Kg\\big)\\cap \\B_r(0,0)\\;\\; \\textrm{ for all } \\; t\\in [0, r')\\\\\n&\\Longleftrightarrow\\; \\exists\\, r'>0: (\\oz + t w, \\olm)\\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)\\;\\; \\textrm{ for all } \\; t\\in [0, r')\\\\\n&\\Longleftrightarrow\\; \\exists\\, r'>0: \\olm \\in\\partial g(\\oz+tw)\\;\\; \\textrm{ for all } \\; t\\in [0, r')\\\\\n&\\Longleftrightarrow\\; \\exists\\, r'>0: \\oz+tw \\in\\partial g^*(\\olm)\\;\\; \\textrm{ for all } \\; t\\in [0, r')\\\\\n&\\Longleftrightarrow\\; w\\in T_{\\partial g^*(\\olm)}(\\oz)=\\big(N_{\\sub g^*(\\olm)}(\\oz)\\big)^*=K_{g^*}(\\olm, \\oz)^*,\n\\end{align*}\nwhere the third equivalence follows from \\eqref{reduction} and the last  {equivalence relies on the polyhedrality of $\\partial g^*(\\olm)$}.\nTo justify  \\eqref{ricond}, it follows from Proposition~\\ref{nonde1} that $\\Kg$ is a linear subspace if and only if $\\olm\\in \\ri \\sub g(\\oz)$.\nA similar observation can be made for $K_{g^*}(\\olm, \\oz)$, which together with \\eqref{cricone2} completes the proof of  \\eqref{ricond}.\n\\end{proof}","sha256_utf8":"f6cc30d66e0c2eeacc5ebc41b7aec3ce65e6d69b395e2f2017577318ae9694c5"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"约化与共轭次微分推出极性","source_units":[{"start":36926,"end":37849,"start_line":568,"end_line":577,"raw_tex":"Let $r>0$ be such that  \\eqref{reduction} holds. Then, we conclude from \\eqref{cricone1} that \n\\begin{align*}\nw \\in \\Kg \\; &\\Longleftrightarrow\\; (w, 0)\\in\\gph N_\\Kg\\\\\n&\\Longleftrightarrow\\; \\exists\\, r'>0: t(w, 0)\\in\\big(\\gph N_\\Kg\\big)\\cap \\B_r(0,0)\\;\\; \\textrm{ for all } \\; t\\in [0, r')\\\\\n&\\Longleftrightarrow\\; \\exists\\, r'>0: (\\oz + t w, \\olm)\\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)\\;\\; \\textrm{ for all } \\; t\\in [0, r')\\\\\n&\\Longleftrightarrow\\; \\exists\\, r'>0: \\olm \\in\\partial g(\\oz+tw)\\;\\; \\textrm{ for all } \\; t\\in [0, r')\\\\\n&\\Longleftrightarrow\\; \\exists\\, r'>0: \\oz+tw \\in\\partial g^*(\\olm)\\;\\; \\textrm{ for all } \\; t\\in [0, r')\\\\\n&\\Longleftrightarrow\\; w\\in T_{\\partial g^*(\\olm)}(\\oz)=\\big(N_{\\sub g^*(\\olm)}(\\oz)\\big)^*=K_{g^*}(\\olm, \\oz)^*,\n\\end{align*}\nwhere the third equivalence follows from \\eqref{reduction} and the last  {equivalence relies on the polyhedrality of $\\partial g^*(\\olm)$}.\n","sha256_utf8":"8da1f2fbf13d9ae56960d65db81da7c1a43ad05f803651d0c9d4cecf0a649605"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"极性传递相对内部条件","source_units":[{"start":37849,"end":38129,"start_line":578,"end_line":579,"raw_tex":"To justify  \\eqref{ricond}, it follows from Proposition~\\ref{nonde1} that $\\Kg$ is a linear subspace if and only if $\\olm\\in \\ri \\sub g(\\oz)$.\nA similar observation can be made for $K_{g^*}(\\olm, \\oz)$, which together with \\eqref{cricone2} completes the proof of  \\eqref{ricond}.\n","sha256_utf8":"b5d799bcd0c6c0205a0a2d601a749aab01928ccd15f00601cf7b7b5a4cbdce5f"}]},{"id":"S035","semantic_type":"proposition","kind":"proposition","title":"邻近临界锥的面差表示","source_units":[{"start":38554,"end":39207,"start_line":589,"end_line":596,"raw_tex":"\\begin{Proposition} \\label{lem:faces}\nAssume that $g:\\R^m\\to \\oR$ is a polyhedral function and $(\\oz, \\olm)\\in \\gph \\partial g$.\nThen there exists $r>0$ such that for all $(z, \\lm)\\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$ the corresponding critical cone $\\Kgzv$ enjoys the following representation\n\\begin{equation*}\n\\Kgzv = F_1 -F_2,\n\\end{equation*}\nfor some faces $F_1, F_2$ of $\\Kg$ with $F_2\\subset F_1$. Conversely, for any pair of faces $F_1, F_2$ of $\\Kg$ with $F_2\\subset F_1$ and any sufficiently small real number $r>0$, there exists $(z, \\lm)\\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$ with $\\Kgzv = F_1-F_2$.\n\\end{Proposition}","sha256_utf8":"bffc95660a74f8f5feff860f07f408839e2c87b9d162d0178b0d9c4caa569a0d"}]},{"id":"S036","semantic_type":"proof","kind":"proof","title":"面差表示完整证明","source_units":[{"start":39210,"end":43719,"start_line":599,"end_line":655,"raw_tex":"\\begin{proof} \nLet $r>0$ be such that   \\eqref{reduction} holds. Choosing a smaller radius $r$ if necessary, we can assume \nwithout loss of generality that the inclusions $\\sub g(z)\\subset \\sub g(\\oz)$ and $N_{\\partial g(\\oz)}(\\lm)\\subset N_{\\partial g(\\oz)}(\\olm)$ hold for any $(z, \\lm) \\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$, which can be guaranteed by    $g$ being a polyhedral function. \nWe first show that for any  $(z, \\lm) \\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$, we have \n\\begin{equation}\\label{co6}\n\\Kgzv = \\Kg\\cap [\\lm-\\olm]^\\perp +[z-\\oz ].\n\\end{equation}\nTo this end, pick $(z, \\lm) \\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$ and observe that the inclusion \n\\begin{equation}\\label{in1}\nN_{\\partial g(z)}(\\lm) \\subset N_{\\partial g(z)}(\\olm)\n\\end{equation}\nholds. To justify it, it is not hard to see $\\sub g(z)=\\sub g(\\oz)\\cap D$ with $D:=\\{v\\in \\R^m|\\, \\la v,z-\\oz\\ra=g(z)-g(\\oz)\\}$. \nIt follows from $\\lm\\in \\sub g(z)\\subset D$ and $\\lm-\\olm\\in N_\\Kg(z-\\oz)$ that $\\olm \\in D$. This, together with the definition of $D$, leads us to $N_D(\\lm)=N_D(\\olm)=[z-\\oz]$.\nSince both $\\sub g(\\oz)$ and $D$ are \npolyhedral, we conclude from the intersection rule for normal cones, holding without any constraint qualification for polyhedral convex sets, that \n\\begin{eqnarray*}\nN_{\\partial g(z)}(\\lm)&=& N_{\\sub g(\\oz)\\cap D}(\\lm)= N_{\\partial g(\\oz)}(\\lm)+N_D(\\lm)\\\\\n&\\subset& N_{\\partial g(\\oz)}(\\olm) +N_D(\\olm) =N_{\\sub g(\\oz)\\cap D}(\\olm)=N_{\\partial g(z)}(\\olm),\n\\end{eqnarray*}\nproving our claimed inclusion.  Since $g^*$ is a polyhedral function as well,  using Lemma~\\ref{lem:indset} for $g^*$ at $(\\olm,\\oz)\\in \\gph \\sub g^*$ and shrinking $r$, if necessary, ensure that $\\oz\\in \\sub g^*(\\lm)$.\nMoreover, it results from \\eqref{in1} that $N_{\\partial g^*(\\lm)}(z) \\subset N_{\\partial g^*(\\lm)}(\\oz)$, which  in combination with \\eqref{cricone1}  \nleads us to \n\\begin{equation*}\\label{co5}\nK_{g^*}(\\lm, z)=N_{\\partial g^*(\\lm)}(z) = N_{\\partial g^*(\\lm)}(\\oz)\\cap [z-\\oz]^\\perp =K_{g^*}(\\lm, \\oz)\\cap [z-\\oz]^\\perp.\n\\end{equation*}\nUsing this and  \\eqref{cricone2} brings us to \n\\begin{equation}\\label{co8}\n\\Kgzv =K_{g^*}(\\lm, z)^*= \\big(K_{g^*}(\\lm, \\oz)\\cap [z-\\oz]^\\perp\\big)^*=K_g(\\oz, \\lm) +[z-\\oz ].\n\\end{equation}\nSimilarly, by  \\eqref{cricone1}   and the definition of the normal cone to the polyhedral convex set $\\sub g(\\oz)$, we obtain \n\\begin{equation*}\\label{co7}\nK_g(\\oz, \\lm) = N_{\\partial g(\\oz)}(\\lm) =N_{\\partial g(\\oz)}(\\olm)\\cap [\\lm-\\olm]^\\perp=  \\Kg\\cap [\\lm-\\olm]^\\perp,\n\\end{equation*}\nwhich, together with \\eqref{co8}, proves \\eqref{co6}.    \n\nAfter these preparations, we are in a position to justify the claimed descriptions of critical cones of $g$. Pick $(z, \\lm) \\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$ and\nset $F_1:= \\Kg\\cap [\\lm-\\olm]^\\perp$, which is clearly a face of $\\Kg$. Because $\\lm-\\olm\\in N_\\Kg(z-\\oz)$, resulting from \\eqref{reduction}, we conclude that $z-\\oz\\in F_1$. \nSince the relative interiors of  nonempty  faces of $F_1$ form a partition of this set (cf. \\cite[Theorem~18.2]{Roc70}), we find a face of $F_1$, denoted by $F_2$, that $z-\\oz\\in \\ri F_2$.\nThis tells us that  $F_2\\subset F_1$. Moreover,   $F_2$ is a  face  of $\\Kg$ as well.\nBy \\eqref{co6}, the inclusion $\\Kgzv \\subset F_1 - F_2$ clearly holds. To get the opposite inclusion, pick $x_i\\in F_i$ for $i=1,2$. It follows from $z-\\oz\\in \\ri F_2$ and \\cite[Theorem~6.4]{Roc70} that there is $t>0$ such that $(1+t)(z-\\oz)-tx_2\\in F_2\\subset F_1$,\nwhich yields \n\\begin{equation}\\label{in2}\nt(x_1-x_2)= tx_1+(1+t)(z-\\oz)-tx_2- (1+t)(z-\\oz)\\in F_1+[z-\\oz],\n\\end{equation}\nwhich confirms via \\eqref{co6} that  the inclusion $F_1-F_2\\subset \\Kgzv$ holds. This shows that  $ \\Kgzv=F_1-F_2$.\n\n \n\nAssume now that $F_1$ and $F_2$ are faces of $\\Kg$ with $F_2\\subset F_1$. \nPick $r>0$ from Theorem \\ref{thm:reduction} and choose $u\\in \\Kg^*$ with $\\|u\\|<r/2$ such that $F_1 = \\Kg\\cap [u]^\\perp$. Since $u\\in \\Kg^*=N_\\Kg(0)$, it follows from Theorem \\ref{thm:reduction} that $\\olm+u \\in \\partial g(\\oz)$.\nPick now $w\\in \\ri F_2$ with $\\|w\\|<r/2$ and  observe from $w\\in F_1$ that $(w, u) \\in \\gph N_\\Kg$.  By Theorem \\ref{thm:reduction}, we arrive at $(\\oz+w, \\olm+u) \\in \\gph \\partial g$. Since $w\\in \\ri F_2$, \nas in \\eqref{in2}, we can show that $F_1-F_2=F_1 +[w]$.  Employing this and  \\eqref{co6} leads us to  \n\\begin{equation*}\nF_1 - F_2 = F_1 +[w] =  \\Kg\\cap [u]^\\perp +[w] = K_g(\\oz+w, \\olm+u),\n\\end{equation*}\nwhich completes the proof.\n\\end{proof}","sha256_utf8":"e19fe2c48fce473d05f108a81b24310f1282d71bdeaffa6d3f6feeeba75ab6df"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"选取局部邻域并提出临界锥恒等式","source_units":[{"start":39225,"end":39801,"start_line":600,"end_line":605,"raw_tex":"Let $r>0$ be such that   \\eqref{reduction} holds. Choosing a smaller radius $r$ if necessary, we can assume \nwithout loss of generality that the inclusions $\\sub g(z)\\subset \\sub g(\\oz)$ and $N_{\\partial g(\\oz)}(\\lm)\\subset N_{\\partial g(\\oz)}(\\olm)$ hold for any $(z, \\lm) \\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$, which can be guaranteed by    $g$ being a polyhedral function. \nWe first show that for any  $(z, \\lm) \\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$, we have \n\\begin{equation}\\label{co6}\n\\Kgzv = \\Kg\\cap [\\lm-\\olm]^\\perp +[z-\\oz ].\n\\end{equation}\n","sha256_utf8":"46c96929f59af114d88bc307a251214cbc6748ae03527d54fb016a904cae8e58"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"交集法锥公式与共轭指标稳定性","source_units":[{"start":39801,"end":40950,"start_line":606,"end_line":618,"raw_tex":"To this end, pick $(z, \\lm) \\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$ and observe that the inclusion \n\\begin{equation}\\label{in1}\nN_{\\partial g(z)}(\\lm) \\subset N_{\\partial g(z)}(\\olm)\n\\end{equation}\nholds. To justify it, it is not hard to see $\\sub g(z)=\\sub g(\\oz)\\cap D$ with $D:=\\{v\\in \\R^m|\\, \\la v,z-\\oz\\ra=g(z)-g(\\oz)\\}$. \nIt follows from $\\lm\\in \\sub g(z)\\subset D$ and $\\lm-\\olm\\in N_\\Kg(z-\\oz)$ that $\\olm \\in D$. This, together with the definition of $D$, leads us to $N_D(\\lm)=N_D(\\olm)=[z-\\oz]$.\nSince both $\\sub g(\\oz)$ and $D$ are \npolyhedral, we conclude from the intersection rule for normal cones, holding without any constraint qualification for polyhedral convex sets, that \n\\begin{eqnarray*}\nN_{\\partial g(z)}(\\lm)&=& N_{\\sub g(\\oz)\\cap D}(\\lm)= N_{\\partial g(\\oz)}(\\lm)+N_D(\\lm)\\\\\n&\\subset& N_{\\partial g(\\oz)}(\\olm) +N_D(\\olm) =N_{\\sub g(\\oz)\\cap D}(\\olm)=N_{\\partial g(z)}(\\olm),\n\\end{eqnarray*}\nproving our claimed inclusion.  Since $g^*$ is a polyhedral function as well,  using Lemma~\\ref{lem:indset} for $g^*$ at $(\\olm,\\oz)\\in \\gph \\sub g^*$ and shrinking $r$, if necessary, ensure that $\\oz\\in \\sub g^*(\\lm)$.\n","sha256_utf8":"ca4d81ad5abd403895f20621a7468c49fdaf7a7fae7013fef602d2ed24983b0d"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"对偶临界锥计算完成恒等式","source_units":[{"start":40950,"end":41822,"start_line":619,"end_line":632,"raw_tex":"Moreover, it results from \\eqref{in1} that $N_{\\partial g^*(\\lm)}(z) \\subset N_{\\partial g^*(\\lm)}(\\oz)$, which  in combination with \\eqref{cricone1}  \nleads us to \n\\begin{equation*}\\label{co5}\nK_{g^*}(\\lm, z)=N_{\\partial g^*(\\lm)}(z) = N_{\\partial g^*(\\lm)}(\\oz)\\cap [z-\\oz]^\\perp =K_{g^*}(\\lm, \\oz)\\cap [z-\\oz]^\\perp.\n\\end{equation*}\nUsing this and  \\eqref{cricone2} brings us to \n\\begin{equation}\\label{co8}\n\\Kgzv =K_{g^*}(\\lm, z)^*= \\big(K_{g^*}(\\lm, \\oz)\\cap [z-\\oz]^\\perp\\big)^*=K_g(\\oz, \\lm) +[z-\\oz ].\n\\end{equation}\nSimilarly, by  \\eqref{cricone1}   and the definition of the normal cone to the polyhedral convex set $\\sub g(\\oz)$, we obtain \n\\begin{equation*}\\label{co7}\nK_g(\\oz, \\lm) = N_{\\partial g(\\oz)}(\\lm) =N_{\\partial g(\\oz)}(\\olm)\\cap [\\lm-\\olm]^\\perp=  \\Kg\\cap [\\lm-\\olm]^\\perp,\n\\end{equation*}\nwhich, together with \\eqref{co8}, proves \\eqref{co6}.    \n","sha256_utf8":"80af386862008cb6668a13dfc134a8ab0cf73d49518ab0ce9494418724b57371"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"相对内部所在面给出面差表示","source_units":[{"start":41823,"end":42958,"start_line":634,"end_line":643,"raw_tex":"After these preparations, we are in a position to justify the claimed descriptions of critical cones of $g$. Pick $(z, \\lm) \\in \\big(\\gph \\partial g\\big)\\cap \\B_r(\\oz, \\olm)$ and\nset $F_1:= \\Kg\\cap [\\lm-\\olm]^\\perp$, which is clearly a face of $\\Kg$. Because $\\lm-\\olm\\in N_\\Kg(z-\\oz)$, resulting from \\eqref{reduction}, we conclude that $z-\\oz\\in F_1$. \nSince the relative interiors of  nonempty  faces of $F_1$ form a partition of this set (cf. \\cite[Theorem~18.2]{Roc70}), we find a face of $F_1$, denoted by $F_2$, that $z-\\oz\\in \\ri F_2$.\nThis tells us that  $F_2\\subset F_1$. Moreover,   $F_2$ is a  face  of $\\Kg$ as well.\nBy \\eqref{co6}, the inclusion $\\Kgzv \\subset F_1 - F_2$ clearly holds. To get the opposite inclusion, pick $x_i\\in F_i$ for $i=1,2$. It follows from $z-\\oz\\in \\ri F_2$ and \\cite[Theorem~6.4]{Roc70} that there is $t>0$ such that $(1+t)(z-\\oz)-tx_2\\in F_2\\subset F_1$,\nwhich yields \n\\begin{equation}\\label{in2}\nt(x_1-x_2)= tx_1+(1+t)(z-\\oz)-tx_2- (1+t)(z-\\oz)\\in F_1+[z-\\oz],\n\\end{equation}\nwhich confirms via \\eqref{co6} that  the inclusion $F_1-F_2\\subset \\Kgzv$ holds. This shows that  $ \\Kgzv=F_1-F_2$.\n","sha256_utf8":"90f6e13384147c2abdf0e6b49a2ccf9cac8deb03c3574c1b7b597850c55ffe2b"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"构造图上邻点实现任意嵌套面差","source_units":[{"start":42962,"end":43708,"start_line":647,"end_line":654,"raw_tex":"Assume now that $F_1$ and $F_2$ are faces of $\\Kg$ with $F_2\\subset F_1$. \nPick $r>0$ from Theorem \\ref{thm:reduction} and choose $u\\in \\Kg^*$ with $\\|u\\|<r/2$ such that $F_1 = \\Kg\\cap [u]^\\perp$. Since $u\\in \\Kg^*=N_\\Kg(0)$, it follows from Theorem \\ref{thm:reduction} that $\\olm+u \\in \\partial g(\\oz)$.\nPick now $w\\in \\ri F_2$ with $\\|w\\|<r/2$ and  observe from $w\\in F_1$ that $(w, u) \\in \\gph N_\\Kg$.  By Theorem \\ref{thm:reduction}, we arrive at $(\\oz+w, \\olm+u) \\in \\gph \\partial g$. Since $w\\in \\ri F_2$, \nas in \\eqref{in2}, we can show that $F_1-F_2=F_1 +[w]$.  Employing this and  \\eqref{co6} leads us to  \n\\begin{equation*}\nF_1 - F_2 = F_1 +[w] =  \\Kg\\cap [u]^\\perp +[w] = K_g(\\oz+w, \\olm+u),\n\\end{equation*}\nwhich completes the proof.\n","sha256_utf8":"194746ab9ce888e836ed3875f14095dd4fdf810b34a00c6782089804248395d9"}]},{"id":"S042","semantic_type":"corollary","kind":"corollary","title":"临界锥夹逼与局部恒定性","source_units":[{"start":43736,"end":44525,"start_line":672,"end_line":687,"raw_tex":"\\begin{Corollary}\\label{incri}\nFor a polyhedral function $g:\\R^m\\to \\oR$  and $(\\oz, \\olm)\\in \\gph \\partial g$, the following properties hold.\n\\begin{itemize}[noitemsep,topsep=2pt]\n\\item [ \\rm {(a)}] There exists $r>0$ such that for all $(z, \\lm) \\in \\big(\\gph\\partial g\\big)\\cap \\B_r(\\oz, \\olm)$ the following inclusions hold:\n\\begin{subequations}\\label{cri2}\n\\begin{align}\n&\\Kgzv \\subset \\Kg -\\Kg,\\label{cri2:a}\\\\\n&\\Kg \\cap -\\Kg \\subset \\Kgzv.\\label{cri2:b}\n\\end{align}\n\\end{subequations}\n\\item [ \\rm {(b)}] If,  in addition,     $\\olm \\in \\ri\\partial g (\\oz)$, then there exists $r>0$ such that for all $(z, \\lm)\\in \\big(\\gph \\partial  g\\big)\\cap\\B_r(\\oz, \\olm)$, we have $\\lm\\in \\ri \\sub g(z)$ and \n\\begin{equation}\\label{cri1}\n\\Kgzv = \\Kg.\n\\end{equation}\n\\end{itemize}\n\\end{Corollary}","sha256_utf8":"5c9a11e82b946cb40459c79631fd2f74dab9d399a89d1dfd0904dd519de4b0fe"}]},{"id":"S043","semantic_type":"proof","kind":"proof","title":"临界锥稳定性完整证明","source_units":[{"start":44526,"end":45660,"start_line":688,"end_line":699,"raw_tex":"\\begin{proof} The inclusion in \n \\eqref{cri2:a} for any $(z, \\lm) \\in \\gph \\partial g$ sufficiently close to $(\\oz, \\olm)$ falls directly from Proposition~\\ref{lem:faces}. \nAlso, according to Proposition~\\ref{lem:faces}, for any such $(z, \\lm) \\in \\gph \\partial g$, one can find  some faces $F_1, F_2$ of $\\Kg$ with $F_2\\subset F_1$\nsuch that $K_g(z,\\lm)=F_1-F_2$. Since $\\Kg \\cap -\\Kg$ is the smallest face of $\\Kg$, we get $\\Kg \\cap -\\Kg\\subset F_i$ for $i=1,2$, which in turn justify \nthe inclusion in  \\eqref{cri2:b} for all $(z, \\lm) \\in \\gph \\partial g$ sufficiently close to $(\\oz, \\olm)$. \n\nTo justify (b), it follows from $\\olm\\in \\ri\\partial g(\\oz)$ and  Proposition~\\ref{nonde1}  that $\\Kg$ is a linear subspace.  This and the inclusions  \\eqref{cri2:a} and \\eqref{cri2:b}   tell us  that\n\\begin{equation*}\n\\Kg=\\Kg-\\Kg\\supset \\Kgzv \\supset \\Kg\\cap-\\Kg=\\Kg\n\\end{equation*}\nfor all $(z, \\lm)\\in \\gph \\partial  g$ sufficiently close to $(\\oz,\\olm)$, which in turn confirms that  $\\Kgzv$ is a linear subspace. Using Proposition~\\ref{nonde1} again, we have $\\lm\\in \\ri \\sub g(z)$ and hence complete the proof of (b).\n\\end{proof}","sha256_utf8":"76f5ca697ec959e5a58a4e2a8935a157563ff1bc49796f9cb8180dc2ea7bdcea"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"面差与最小面推出双向夹逼","source_units":[{"start":44526,"end":45124,"start_line":688,"end_line":692,"raw_tex":"\\begin{proof} The inclusion in \n \\eqref{cri2:a} for any $(z, \\lm) \\in \\gph \\partial g$ sufficiently close to $(\\oz, \\olm)$ falls directly from Proposition~\\ref{lem:faces}. \nAlso, according to Proposition~\\ref{lem:faces}, for any such $(z, \\lm) \\in \\gph \\partial g$, one can find  some faces $F_1, F_2$ of $\\Kg$ with $F_2\\subset F_1$\nsuch that $K_g(z,\\lm)=F_1-F_2$. Since $\\Kg \\cap -\\Kg$ is the smallest face of $\\Kg$, we get $\\Kg \\cap -\\Kg\\subset F_i$ for $i=1,2$, which in turn justify \nthe inclusion in  \\eqref{cri2:b} for all $(z, \\lm) \\in \\gph \\partial g$ sufficiently close to $(\\oz, \\olm)$. \n","sha256_utf8":"2c5ab05ca502b2c27b86471514bd9a8c13dcb6d3e05848b4b57702b5fc8bbf81"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"子空间夹逼推出临界锥及条件稳定","source_units":[{"start":45125,"end":45649,"start_line":694,"end_line":698,"raw_tex":"To justify (b), it follows from $\\olm\\in \\ri\\partial g(\\oz)$ and  Proposition~\\ref{nonde1}  that $\\Kg$ is a linear subspace.  This and the inclusions  \\eqref{cri2:a} and \\eqref{cri2:b}   tell us  that\n\\begin{equation*}\n\\Kg=\\Kg-\\Kg\\supset \\Kgzv \\supset \\Kg\\cap-\\Kg=\\Kg\n\\end{equation*}\nfor all $(z, \\lm)\\in \\gph \\partial  g$ sufficiently close to $(\\oz,\\olm)$, which in turn confirms that  $\\Kgzv$ is a linear subspace. Using Proposition~\\ref{nonde1} again, we have $\\lm\\in \\ri \\sub g(z)$ and hence complete the proof of (b).\n","sha256_utf8":"00d44015bafa568c5b1eb1e291d748b7a01a80f3c100a27e1b35cd8c321fcedc"}]},{"id":"S046","semantic_type":"notation","kind":"notation","title":"球、凸集运算与集合值映射记号","source_units":[{"start":11298,"end":12321,"start_line":236,"end_line":243,"raw_tex":"In what follows,    we \n denote by $\\B$ the closed unit ball in the space in question and by $\\B_r(x):=x+r\\B$ the closed ball centered at $x$ with radius $r>0$. \n  In the  product space $\\R^n\\times \\R^m$, we use the norm $\\|(w,u)\\|=\\sqrt{\\|w\\|^2+\\|u\\|^2}$ for any $(w,u)\\in \\R^n\\times \\R^m$.\n Given a nonempty set $C\\subset\\R^n$, the symbols $\\inte C$,  $\\ri C$, $\\cone C$, and $\\co C$ signify its interior, relative interior, conic hull, and convex hull, respectively. \n For any set $C$ in $\\R^n$, its indicator function is defined by $\\dd_C(x)=0$ for $x\\in C$ and $\\dd_C(x)=\\infty$ otherwise. We denote\n by $P_C$ the projection mapping onto $C$ and  by $\\dist(x,C)$  the distance between $x\\in \\R^n$ and a set $C$.\n For a vector $w\\in \\R^n$, the subspace $\\{tw |\\, t\\in \\R\\}$ is denoted by $[w]$. The domain and range of a set-valued mapping $F:\\R^n\\tto\\R^m$ are defined, respectively, by  \n$ \\dom F:= \\{x\\in\\R^n\\big|\\;F(x)\\ne\\emp \\}$ and $\\rge F=\\{u\\in \\R^m|\\; \\exists \\,  w\\in \\R^n\\;\\,\\mbox{with}\\;\\; u\\in  F(w) \\}$.  \n","sha256_utf8":"d9598f2682ba0d995b3dde3549505d0d17227b91faf6d5e13f47806980fe12d6"}]},{"id":"S047","semantic_type":"definition","kind":"definition","title":"集合内外极限与收敛","source_units":[{"start":12324,"end":13336,"start_line":246,"end_line":253,"raw_tex":"In this paper, the convergence of a family of sets is always understood in the sense of Painlev\\'e-Kuratowski (cf. \\cite[Definition~4.1]{rw}). \nThis means that the inner limit set of a parameterized family of sets $\\{C^t\\}_{t>0}$ in $\\R^d$, denoted $\\liminf_{t\\searrow 0} C^t$, is the set of points $x$ such that for every\nsequence $t_k \\searrow 0$,  $x$ is the limit of a sequence of points $x^{t_k}\\in C^{t_k}$. The outer limit set of this family of sets, \ndenoted $\\limsup_{t\\searrow 0} C^t$, is the set of points $x$ such that there exist  sequences\n $t_k \\searrow 0$ and $x^{t_k}\\in C^{t_k}$ such that  $x^{t_k}\\to x$ as $k\\to \\infty$. A sequence $\\{f^k\\b$ of functions $f^k:\\R^n\\to \\oR$ is said to {\\em epi-converge} to a function $f:\\R^n\\to \\oR$ if we have $\\epi f^k\\to \\epi f$ as $k\\to \\infty$,\n where $\\epi f=\\{(x,\\al)\\in \\R^n\\times \\R|\\, f(x)\\le \\al\\}$ is   the epigraph of $f$;\n see \\cite[Definition~7.1]{rw} for more details on the epi-convergence of \na sequence of extended-real-valued functions.  \n","sha256_utf8":"e130043e610f1dec0dec00700c744d17718b222c3028d0c6034da99ac6febcd1"}]},{"id":"S048","semantic_type":"definition","kind":"definition","title":"切锥与正则及极限法锥","source_units":[{"start":13337,"end":14091,"start_line":255,"end_line":262,"raw_tex":"Given a nonempty set $\\Omega\\subset\\R^n$ with $\\ox\\in \\Omega$, the {  tangent cone}  to $\\Omega$ at $\\ox$, denoted $T_\\Omega(\\ox)$,  is defined  by\n\\begin{equation}\\label{tan1}\nT_\\Omega(\\ox) = \\limsup_{t\\searrow 0} \\frac{\\Omega - \\ox}{t}.\n\\end{equation}\nThe  regular/Fr\\'{e}chet normal cone $\\rN_\\Omega(\\ox)$ to $\\Omega$ at $\\ox$ is defined by\n $\\rN_\\Omega(\\ox) = T_\\Omega(\\ox)^*$, the polar of the tangent cone \\eqref{tan1}. The  (limiting/Mordukhovich) normal cone $N_\\Omega(\\ox)$ to $\\Omega$ at $\\ox$ is\n the set of all vectors $\\ov\\in \\R^n$ for which there exist sequences  $\\{x^k\\b$ and  $\\{v^k\\b$ with $v^k\\in \\rN_\\Omega( x^k)$ such that \n$(x^k,v^k)\\to (\\ox,\\ov)$. When $\\Omega$ is convex, both normal cones boil down to that of convex analysis.  \n","sha256_utf8":"a4bf28e0149ea3519b4d3f2bd8877d877359bdf4dc806d0a0b9d7206ee2dfa7e"}]},{"id":"S049","semantic_type":"definition","kind":"definition","title":"次导数与凸次微分","source_units":[{"start":14091,"end":14627,"start_line":263,"end_line":270,"raw_tex":"Given a function $f:\\R^n \\to \\oR$ and a point $\\ox\\in\\R^n$ with $f(\\ox)$ finite, the subderivative function $\\d f(\\ox)\\colon\\R^n\\to\\oR$ is defined by\n\\begin{equation*}\\label{fsud}\n\\d f(\\ox)(w)=\\liminf_{\\substack{\n   t\\searrow 0 \\\\\n  w'\\to w\n  }} {\\frac{f(\\ox+tw')-f(\\ox)}{t}}.\n\\end{equation*}\nWhen $f$ is convex, its  subdifferential    at $\\ox$, denoted by $\\sub f(\\ox)$,  is understood in the sense of convex analysis (cf. \\cite[page~214]{Roc70}), namely $v\\in \\sub f(\\ox)$ if   $f(x)\\ge  f(\\ox)+\\la v,x-\\ox\\ra $ for any $x\\in \\R^n$.\n","sha256_utf8":"049deac9aaa1ce0eacd2f00bc5e7f19270dcd10c5f93874fda407e02d3553ce2"}]},{"id":"S050","semantic_type":"definition","kind":"definition","title":"多面体函数及有限表示","source_units":[{"start":14629,"end":15674,"start_line":273,"end_line":288,"raw_tex":"As pointed earlier,  a proper   function $g:\\R^m\\to \\oR$ is called {\\em polyhedral}  \nif $\\epi g$ is a polyhedral convex set.  According to \\cite[Theorem~2.49]{rw}, this class of convex functions enjoys the  representation\n\\begin{equation*}\\label{polyf}\ng(z) = \\begin{cases}\n\\max\\limits_{j\\in J}\\big\\{\\la a^j, z \\ra - \\alpha_j\\big\\} \\quad& \\text{if } z \\in\\dom g,\\\\\n\\infty & \\text{otherwise,}\n\\end{cases}\n\\end{equation*}\nwhere $J= \\{1,\\ldots, l\\}$ for some $l\\in \\N$, $a^j\\in \\R^m$ and $\\alpha_j\\in \\R$  for all $j\\in J$, and where  $\\dom g=\\{z\\in \\R^m|\\, g(z)<\\infty\\}$ is a polyhedral convex set with the representation \n\\begin{equation}\\label{dom}\n\\dom g = \\big\\{z\\in\\R^m\\, \\big|\\, \\la b^i, z \\ra \\leq \\beta_i,  i\\in I=\\{1, \\ldots, s\\} \\big\\},\n\\end{equation}\nwhere $s\\in \\N$, and $b^i \\in \\R^m$ and $\\beta_i\\in \\R$ for all $i\\in I$. Thus, we can equivalently express a polyhedral function  $g$ as\n\\begin{equation}\\label{polyfunc}\ng(z) = \\max_{j\\in J}\\big\\{\\la a^j, z \\ra - \\alpha_j\\big\\} + \\delta_{\\dom g}(z), \\quad z\\in \\R^m.\n\\end{equation}\n","sha256_utf8":"cd5d35004c8901ae73d83aa2dd4d0367aae5a610bbae1d33a1501b8c2d383afb"}]},{"id":"S051","semantic_type":"notation","kind":"notation","title":"活动胞腔、指标与次微分公式","source_units":[{"start":15674,"end":16695,"start_line":289,"end_line":300,"raw_tex":"It was observed in \\cite[Proposition~3.2]{MoS16} that $\\dom g$ can be expressed as the finite  union of the polyhedral convex sets $C_j, j\\in J$, defined by\n\\begin{equation*}\\label{Cj}\nC_j =\\big\\{z\\in \\dom g\\, \\big|\\, g(z) = \\la a^j, z\\ra -\\alpha_j\\big\\} =\\big\\{z \\in \\dom g\\, \\big|\\, \\la a^i -a^j, z\\ra  \\leq \\alpha_i -\\alpha_j, i\\in J\\big\\}.\n\\end{equation*}\nPick $\\oz \\in\\dom g$ and  define the sets of active indices  at $\\oz$ corresponding to the representation \\eqref{dom}    and  {to the expression of $\\dom g$ via the finite union of $C_j$'s} by\n\\begin{equation}\\label{indset4}\nI(\\oz)= \\big\\{i\\in I\\, \\big|\\, \\la b^i, \\oz\\ra = \\beta_i\\big\\}\\quad\\textrm{ and }\\quad J(\\oz)= \\big\\{ j\\in J\\, \\big|\\, \\oz \\in C_j\\big\\}.\n\\end{equation}\nThese sets allow us to conclude from  \\cite[Proposition~3.3]{MoS16} that  the subdifferential of $g$ at $\\oz$ can be calculated as\n\\begin{equation}\\label{subg}\n\\partial g (\\oz) = \\co \\big\\{a^j \\, \\big|\\,  j\\in J(\\oz)\\big\\}+\\cone \\big\\{b^i\\, \\big|\\, i\\in I(\\oz)\\big\\},\n\\end{equation}\n","sha256_utf8":"33e594f1da6392fa5b0f460820e011b07d9f44f7acbb143b7ec64c3cd81a3fe8"}]},{"id":"S052","semantic_type":"notation","kind":"notation","title":"次梯度系数分解及正系数指标","source_units":[{"start":16695,"end":17711,"start_line":301,"end_line":310,"raw_tex":"which tells us that  any $ \\olm \\in   \\partial g(\\oz)$ can be written as  \n {\\begin{equation}\\label{decomp}\n\\olm = \\sum_{j\\in J(\\oz)} \\osigma_j a^j + \\sum_{i\\in I(\\oz)} \\otau_ib^i\\quad\n \\textrm{ with }\\; \\; \\osigma_j, \\otau_i \\geq 0\\; \\textrm{ and }\\;  \\sum_{j\\in J(\\oz)} \\osigma_j =1.\n\\end{equation}\nFor simplicity, we denote by $\\osigma \\in \\R^l$ and $\\otau \\in \\R^s$ the vectors with components $\\osigma_j$ and $\\otau_i$, respectively, where $\\osigma_j$  is taken from \\eqref{decomp} if   $j\\in J(\\oz)$ and  $\\osigma_j = 0$ if $j\\not\\in J(\\oz)$ and likewise, \n$\\otau_i$  is taken from \\eqref{decomp} if  $i\\in I(\\oz)$ and $\\otau_i =  0$ if $i\\notin I(\\oz)$. Pick the given representation of $\\olm$ in \\eqref{decomp} and define the sets of positive coefficients at $\\oz$ for $\\osigma$ and $\\otau$, respectively, by\n\\begin{equation}\\label{indset}\nJ_+(\\oz, \\osigma) = \\big\\{j\\in J(\\oz)\\, \\big|\\, \\osigma_j >0\\big\\}\\; \\; \\textrm{ and }\\; \\; I_+(\\oz, \\otau)=\\big\\{i\\in I(\\oz)\\, \\big|\\, \\otau_i>0\\big\\}.\n\\end{equation}\n","sha256_utf8":"5d52c00e9fead4e6ef881dedeec6da22cf89cf9bb115281ec4cbe6132a21cb40"}]},{"id":"S053","semantic_type":"derived_fact","kind":"derived_fact","title":"多面体集合的局部切锥表示","source_units":[{"start":21977,"end":22226,"start_line":364,"end_line":367,"raw_tex":"Recall from \\cite[Exercise~6.47]{rw} that for any polyhedral convex set $C\\subset \\R^n$ and $\\ox\\in C$, we can find  a neighborhood $\\O$ of $0\\in \\R^n$ for which we have \n\\begin{equation}\\label{tanrep}\nT_C(\\ox)\\cap \\O=(C-\\ox)\\cap \\O.\n\\end{equation}\n","sha256_utf8":"c27583cf59ba62bd7210080c34aed3c4bf90a386619a7151e908855ff3f6eb56"}]},{"id":"S054","semantic_type":"definition","kind":"definition","title":"临界锥及其次微分法锥表示","source_units":[{"start":24510,"end":25311,"start_line":414,"end_line":425,"raw_tex":"Given a function $f:\\R^n \\to \\oR$ and a point $\\ox\\in\\R^n$ with $f(\\ox)$ finite, the critical cone of $f$ at $\\ox$ for $\\bar v$ with $\\bar v\\in   \\sub f(\\ox)$ is defined by \n\\begin{equation*}\\label{cricone}\n{K_f}(\\ox,\\bar v)=\\big\\{w\\in \\R^n\\,\\big|\\,\\la\\bar v,w\\ra=\\d f(\\ox)(w)\\big\\}.\n\\end{equation*}\nIf, in addition, $f$ is convex and $\\partial f(\\ox)\\neq\\emp$, it follows from   \\cite[Theorem~8.30]{rw} that  its subderivative function is  the support function of $\\partial f(\\ox)$, that is\n\\begin{equation*}\\label{fsud2}\n\\d f(\\ox)(w) = \\sup\\big\\{\\la v, w\\ra\\, \\big|\\, v\\in \\partial f(\\ox)\\big\\},\n\\end{equation*}\nwhich in turn allows us to equivalently describe the critical cone ${K_f}(\\ox,\\bar v)$ as \n\\begin{equation}\\label{cricone1}\nK_f(\\ox, \\bar v) = N_{\\partial f(\\ox)}(\\bar v).\n\\end{equation}\n","sha256_utf8":"eb08b42d7537b18b1c52f34f7eecedd3ec3c70d91a8c05a1367b70ab84ac3416"}]},{"id":"S055","semantic_type":"derived_fact","kind":"derived_fact","title":"共轭函数保持多面体性","source_units":[{"start":36059,"end":36268,"start_line":553,"end_line":553,"raw_tex":"Let us recall an important result, used in the proof of next corollary, saying that the conjugate function $g^*$ of  a polyhedral function $g$ is again a polyhedral function, cf. \\cite[Theorem~11.14(a)]{rw}. \n","sha256_utf8":"61fab028a57716324214146fe751f606fff58a50c3367b8fb18d74f2f3e6cad9"}]},{"id":"S056","semantic_type":"definition","kind":"definition","title":"多面体凸锥的面","source_units":[{"start":38370,"end":38521,"start_line":584,"end_line":587,"raw_tex":"Recall that   a closed face $F$ of a  polyhedral convex cone $C\\subset \\R^d$ is defined by \n$$\nF=C\\cap [v]^\\perp\\quad \\mbox{for some}\\;\\; v\\in C^*.\n$$\n","sha256_utf8":"6f6711727f6a7670ea230f7203a058d519288e724d9424a80d78863929382711"}]},{"id":"S057","semantic_type":"definition","kind":"definition","title":"图导数与原可微性","source_units":[{"start":46951,"end":48315,"start_line":715,"end_line":725,"raw_tex":"To this end, consider a set-valued mapping $F:\\R^n\\tto \\R^m$. According to \\cite[Definition~8.33]{rw}, the {\\em graphical derivative} of $F$ at $\\ox$ for $\\oy$ with $(\\ox,\\oy) \\in \\gph F$ is the set-valued mapping $DF(\\ox, \\oy): \\R^n\\tto \\R^m$ defined via the tangent cone to $\\gph F$ at $(\\ox, \\oy)$ by\n\\begin{equation*}\nu \\in DF(\\ox, \\oy)(w)\\; \\Longleftrightarrow\\; (w,u)\\in T_{\\gph F}(\\ox, \\oy),\n\\end{equation*}\nor, equivalently, $\\gph DF(\\ox, \\oy) = T_{\\gph F}(\\ox, \\oy)$.  Using the definition of the tangent cone, we can present an alternative definition of $DF(\\ox, \\oy)$ in terms of graphical limits as\n\\begin{equation}\\label{proto}\n\\gph DF(\\ox, \\oy) = \\limsup_{t\\searrow 0}\\frac{\\gph F-(\\ox,\\oy)}{t}.\n\\end{equation}\nThe set-valued mapping $F$ is said to be {\\em proto-differentiable} at $\\ox$ for $\\oy$ if the outer graphical limit in \\eqref{proto} is actually a full limit. If $F$ is proto-differentiable at $\\ox$ for $\\oy$,   its graphical derivative $DF(\\ox, \\oy)$ is   called the \n{\\em proto-derivative} of $F$ at $\\ox$ for $\\oy$. When $F(\\ox)$ is a singleton consisting of $\\oy$ only, the notation $DF(\\ox, \\oy)$ is simplified to $DF(\\ox)$. It is easy to see that \nfor a single-valued function $F$, which is differentiable at $\\ox$,  the graphical derivative  $DF(\\ox)$ boils down to the Jacobian matrix of $F$ at $\\ox$, denoted by $\\nabla F(\\ox)$. \n","sha256_utf8":"1935f2ebf6b2aea55652b8abb70d14ff9f029afb48fdbf883f7002d3597722a5"}]},{"id":"S058","semantic_type":"definition","kind":"definition","title":"严格图导数与严格原可微性","source_units":[{"start":48315,"end":49060,"start_line":726,"end_line":736,"raw_tex":"Recall from \\cite[Definition~9.53]{rw} that the strict graphical derivative \nof  a set-valued mapping $F$ at $\\ox$ for $\\oy$ with $(\\ox,\\oy)\\in \\gph F$, is  the set-valued mapping $D_*F(\\ox,\\oy):\\R^n\\tto \\R^m$, defined  by \n\\begin{equation}\\label{sproto}\n\\gph D_*F(\\ox,\\oy)=\\limsup_{\n  \\substack{\n   t\\searrow 0 \\\\\n  (x,y)\\xrightarrow{ \\gph F}(\\ox,\\oy)\n  }} \\frac{\\gph F-(x,y)}{t}.\n\\end{equation}\nThe set-valued mapping $F$ is said to be {\\em strictly} proto-differentiable at $\\ox$ for $\\oy$ if the outer graphical limit in \\eqref{sproto} is   a full limit.\n When $F$ is strictly proto-differentiable at $\\ox$ for $\\oy$, its strict graphical derivative $D_*F(\\ox, \\oy)$ is   called the {\\em strict} proto-derivative of $F$ at $\\ox$ for $\\oy$. \n","sha256_utf8":"a6f7baac7cca59d26d2de50bf22306825bcced78e1f8d5d2ecfa777395dffcc9"}]},{"id":"S059","semantic_type":"derived_fact","kind":"derived_fact","title":"正则切锥、旁切锥与导数一致性","source_units":[{"start":49060,"end":50035,"start_line":737,"end_line":746,"raw_tex":" To elaborate  more,  take a set  $\\Omega\\subset \\R^d$ and  $\\ox\\in \\Omega$. The regular (Clarke) tangent cone and the paratingent cone to   $\\Omega$ at $\\ox$ are defined, respectively,   by\n\\begin{equation}\\label{tancc}\n \\widehat T_{\\Omega}(\\ox) =\\liminf_{x \\xrightarrow{ \\Omega}\\ox,t\\searrow 0} \\frac{\\Omega-x}{t}\\quad \\mbox{and}\\quad  \\widetilde {T}_{\\Omega}(\\ox) =\\limsup_{x \\xrightarrow{ \\Omega}\\ox,t\\searrow 0} \\frac{\\Omega-x}{t}.\n\\end{equation}\nThe strict proto-differentiability of $F$ at $\\ox$ for $\\oy$ amounts to saying that $ \\widehat T_{\\gph F}(\\ox,\\oy)= \\widetilde {T}_{\\gph F}(\\ox,\\oy)$.\nObserve also that since the inclusions $ \\widehat T_{\\gph F}(\\ox,\\oy)\\subset  T_{\\gph F}(\\ox,\\oy)\\subset  \\widetilde {T}_{\\gph F}(\\ox,\\oy)$ always hold, \nstrict proto-differentiability of $F$ at $\\ox$ for $\\oy$ implies that  its proto-derivative and strict proto-derivative coincide, namely we have \n\\begin{equation}\\label{eqspd}\nD_*F(\\ox,\\oy)=DF(\\ox,\\oy).\n\\end{equation}\n","sha256_utf8":"6b68c54df191dc48cac232d1bbe5d5d7c5d1c07d783041f122e288fba116db26"}]},{"id":"S060","semantic_type":"assumption","kind":"assumption","title":"主定理的多面体函数与图上基点","source_units":[{"start":51423,"end":51535,"start_line":760,"end_line":760,"raw_tex":"For a polyhedral function $g:\\R^m\\to \\oR$  and $(\\oz, \\olm)\\in \\gph \\partial g$, the following properties hold.\n","sha256_utf8":"dbc2f710e2b80a80e12831c8470673ba220037558a65761682cf02e92b92efb1"}]},{"id":"S061","semantic_type":"external_fact","kind":"external_fact","title":"多面体表示与次微分公式","source_units":[],"declared_external":true,"declaration_text":["A proper function with polyhedral convex epigraph is a finite maximum of affine functions plus the indicator of a polyhedral domain. Its active affine cells cover that domain, and its subdifferential is the convex hull of active affine slopes plus the cone of active domain normals. Near a fixed domain point, active indices can only disappear, so nearby subdifferentials are contained in the subdifferential at that point. Sources: rw, Theorem 2.49; MoS16, Propositions 3.2 and 3.3."]},{"id":"S062","semantic_type":"external_fact","kind":"external_fact","title":"多面体切锥的局部精确性","source_units":[],"declared_external":true,"declaration_text":["For a polyhedral convex set C and x in C, C-x agrees with T_C(x) in a neighborhood of zero. Source: rw, Exercise 6.47."]},{"id":"S063","semantic_type":"external_fact","kind":"external_fact","title":"多面体函数的一阶次导数公式","source_units":[],"declared_external":true,"declaration_text":["For a polyhedral function g, dom d g(z)=T_dom g(z). On the tangent cone to an active affine cell C_j, d g(z)(w)=<a^j,w>. Source: rw, Proposition 10.21."]},{"id":"S064","semantic_type":"external_fact","kind":"external_fact","title":"凸次导数的支撑函数公式","source_units":[],"declared_external":true,"declaration_text":["For a convex function at a point with nonempty subdifferential, its subderivative is the support function of that subdifferential. Consequently the critical cone at a subgradient v equals the normal cone to the subdifferential at v. Source: rw, Theorem 8.30."]},{"id":"S065","semantic_type":"external_fact","kind":"external_fact","title":"既有临界锥指标刻画","source_units":[],"declared_external":true,"declaration_text":["The critical cone of a polyhedral function, for a fixed nonnegative coefficient decomposition of its subgradient, is exactly the set satisfying the four groups of active-index equalities and inequalities reproduced in B0038. This is an explicitly imported result, with source msa18, Proposition 3.2, identified at line 436."]},{"id":"S066","semantic_type":"external_fact","kind":"external_fact","title":"凸法锥与多面体极性运算","source_units":[],"declared_external":true,"declaration_text":["In finite-dimensional convex analysis, finitely generated polyhedral sets are closed; N_C(x)=T_C(x)^* for closed convex C; N_C(x) is a linear subspace exactly when x is in ri C. For a closed convex cone K, u is in N_K(w) exactly when w is in K, u is in K^*, and <u,w>=0; for a subspace L its normal-mapping graph is L times L^perp. Polyhedral normal cones satisfy the intersection sum rule without a constraint qualification, and polar intersection formulas are exact for polyhedral cones. Normal cones of a polyhedral set locally decrease as the base point varies. Sources include standard convex analysis and DoR14, Proposition 2A.3."]},{"id":"S067","semantic_type":"external_fact","kind":"external_fact","title":"芬歇尔共轭与次微分反演","source_units":[],"declared_external":true,"declaration_text":["The Fenchel conjugate is g^*(v)=sup_z(<v,z>-g(z)). For proper closed convex g, v is in partial g(z) if and only if z is in partial g^*(v). The conjugate of a proper polyhedral convex function is again polyhedral. Sources: standard Fenchel duality; rw, Theorem 11.14(a)."]},{"id":"S068","semantic_type":"external_fact","kind":"external_fact","title":"多面体面的相对内部结构","source_units":[],"declared_external":true,"declaration_text":["The relative interiors of nonempty faces partition a polyhedral convex set; a face of a face is a face. If p is in ri F and x is in F, then (1+t)p-tx is in F for some t>0. Every face of a polyhedral convex cone K is exposed by a vector in K^*, and K intersect (-K) is its smallest face, contained in every face. Sources: Roc70, Theorems 18.2 and 6.4, and standard polyhedral face geometry."]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","title":"多面体锥法映射的严格图导数","source_units":[],"declared_external":true,"declaration_text":["For a polyhedral convex cone K, gph D_*N_K(0,0)=gph N_K-gph N_K. Source: DoR14, Proposition 4H.10."]},{"id":"S070","semantic_type":"external_fact","kind":"external_fact","title":"正则切锥的邻近切锥内极限","source_units":[],"declared_external":true,"declaration_text":["For a locally closed set C, the regular tangent cone at x equals liminf_{z tending to x within C} T_C(z). Source: rw, Theorem 6.26."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S006","target_id":"S002","kind":"supports"},{"source_id":"S007","target_id":"S002","kind":"supports"},{"source_id":"S021","target_id":"S003","kind":"supports"},{"source_id":"S048","target_id":"S003","kind":"supports"},{"source_id":"S057","target_id":"S003","kind":"supports"},{"source_id":"S021","target_id":"S004","kind":"supports"},{"source_id":"S058","target_id":"S004","kind":"supports"},{"source_id":"S059","target_id":"S004","kind":"supports"},{"source_id":"S069","target_id":"S004","kind":"supports"},{"source_id":"S042","target_id":"S005","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S070","target_id":"S005","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S017","target_id":"S006","kind":"supports"},{"source_id":"S042","target_id":"S006","kind":"supports"},{"source_id":"S059","target_id":"S006","kind":"supports"},{"source_id":"S066","target_id":"S006","kind":"supports"},{"source_id":"S003","target_id":"S007","kind":"supports"},{"source_id":"S004","target_id":"S007","kind":"supports"},{"source_id":"S059","target_id":"S007","kind":"supports"},{"source_id":"S017","target_id":"S007","kind":"supports"},{"source_id":"S060","target_id":"S003","kind":"supports"},{"source_id":"S060","target_id":"S004","kind":"supports"},{"source_id":"S060","target_id":"S005","kind":"supports"},{"source_id":"S060","target_id":"S006","kind":"supports"},{"source_id":"S060","target_id":"S007","kind":"supports"},{"source_id":"S009","target_id":"S008","kind":"supports"},{"source_id":"S010","target_id":"S009","kind":"supports"},{"source_id":"S011","target_id":"S009","kind":"supports"},{"source_id":"S012","target_id":"S009","kind":"supports"},{"source_id":"S050","target_id":"S010","kind":"supports"},{"source_id":"S051","target_id":"S010","kind":"supports"},{"source_id":"S052","target_id":"S010","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S049","target_id":"S011","kind":"supports"},{"source_id":"S050","target_id":"S011","kind":"supports"},{"source_id":"S051","target_id":"S011","kind":"supports"},{"source_id":"S052","target_id":"S011","kind":"supports"},{"source_id":"S066","target_id":"S011","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S051","target_id":"S012","kind":"supports"},{"source_id":"S052","target_id":"S012","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S016","target_id":"S014","kind":"supports"},{"source_id":"S063","target_id":"S015","kind":"supports"},{"source_id":"S051","target_id":"S015","kind":"supports"},{"source_id":"S053","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S051","target_id":"S016","kind":"supports"},{"source_id":"S049","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S054","target_id":"S019","kind":"supports"},{"source_id":"S066","target_id":"S019","kind":"supports"},{"source_id":"S065","target_id":"S020","kind":"supports"},{"source_id":"S052","target_id":"S020","kind":"supports"},{"source_id":"S054","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S022","kind":"supports"},{"source_id":"S026","target_id":"S022","kind":"supports"},{"source_id":"S027","target_id":"S022","kind":"supports"},{"source_id":"S028","target_id":"S022","kind":"supports"},{"source_id":"S029","target_id":"S022","kind":"supports"},{"source_id":"S030","target_id":"S022","kind":"supports"},{"source_id":"S050","target_id":"S023","kind":"supports"},{"source_id":"S051","target_id":"S023","kind":"supports"},{"source_id":"S053","target_id":"S023","kind":"supports"},{"source_id":"S054","target_id":"S023","kind":"supports"},{"source_id":"S066","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S013","target_id":"S024","kind":"supports"},{"source_id":"S049","target_id":"S024","kind":"supports"},{"source_id":"S054","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S008","target_id":"S025","kind":"supports"},{"source_id":"S052","target_id":"S025","kind":"supports"},{"source_id":"S061","target_id":"S025","kind":"supports"},{"source_id":"S066","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S020","target_id":"S026","kind":"supports"},{"source_id":"S051","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S027","kind":"supports"},{"source_id":"S051","target_id":"S027","kind":"supports"},{"source_id":"S025","target_id":"S028","kind":"supports"},{"source_id":"S051","target_id":"S028","kind":"supports"},{"source_id":"S025","target_id":"S029","kind":"supports"},{"source_id":"S026","target_id":"S029","kind":"supports"},{"source_id":"S027","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S020","target_id":"S029","kind":"supports"},{"source_id":"S050","target_id":"S029","kind":"supports"},{"source_id":"S025","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S051","target_id":"S030","kind":"supports"},{"source_id":"S052","target_id":"S030","kind":"supports"},{"source_id":"S066","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S032","kind":"supports"},{"source_id":"S021","target_id":"S033","kind":"supports"},{"source_id":"S053","target_id":"S033","kind":"supports"},{"source_id":"S054","target_id":"S033","kind":"supports"},{"source_id":"S055","target_id":"S033","kind":"supports"},{"source_id":"S066","target_id":"S033","kind":"supports"},{"source_id":"S067","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S017","target_id":"S034","kind":"supports"},{"source_id":"S055","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S039","target_id":"S036","kind":"supports"},{"source_id":"S040","target_id":"S036","kind":"supports"},{"source_id":"S041","target_id":"S036","kind":"supports"},{"source_id":"S021","target_id":"S037","kind":"supports"},{"source_id":"S061","target_id":"S037","kind":"supports"},{"source_id":"S066","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S021","target_id":"S038","kind":"supports"},{"source_id":"S008","target_id":"S038","kind":"supports"},{"source_id":"S049","target_id":"S038","kind":"supports"},{"source_id":"S055","target_id":"S038","kind":"supports"},{"source_id":"S066","target_id":"S038","kind":"supports"},{"source_id":"S067","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S031","target_id":"S039","kind":"supports"},{"source_id":"S054","target_id":"S039","kind":"supports"},{"source_id":"S066","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S021","target_id":"S040","kind":"supports"},{"source_id":"S056","target_id":"S040","kind":"supports"},{"source_id":"S068","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S021","target_id":"S041","kind":"supports"},{"source_id":"S056","target_id":"S041","kind":"supports"},{"source_id":"S066","target_id":"S041","kind":"supports"},{"source_id":"S068","target_id":"S041","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"},{"source_id":"S035","target_id":"S044","kind":"supports"},{"source_id":"S068","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S017","target_id":"S045","kind":"supports"},{"source_id":"S046","target_id":"S010","kind":"supports"},{"source_id":"S046","target_id":"S023","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S057","kind":"supports"},{"source_id":"S047","target_id":"S058","kind":"supports"},{"source_id":"S048","target_id":"S053","kind":"supports"},{"source_id":"S048","target_id":"S054","kind":"supports"},{"source_id":"S049","target_id":"S054","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S057","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S061","target_id":"S050","kind":"supports"},{"source_id":"S061","target_id":"S051","kind":"supports"},{"source_id":"S062","target_id":"S053","kind":"supports"},{"source_id":"S064","target_id":"S054","kind":"supports"},{"source_id":"S067","target_id":"S055","kind":"supports"}]}}
{"record_id":"2208.06524v1","paper":{"title":"A Gradient Complexity Analysis for Minimizing the Sum of Strongly Convex Functions with Varying Condition Numbers","authors":["Nuozhou Wang","Shuzhong Zhang"],"arxiv_id":"2208.06524","version":"v1","journal":"SIAM Journal on Optimization","doi":"10.1137/22m1503646","abstract_url":"https://arxiv.org/abs/2208.06524v1","pdf_url":"https://arxiv.org/pdf/2208.06524v1"},"context":{"raw_tex":"\\begin{equation}\n    \\min_{x\\in \\Gamma} F(x)=\\sum_{i=1}^{m} g_i(x),  \\label{pro}\n\\end{equation}\n\nwhere $g_i(x)$'s are smooth and convex functions, and $\\Gamma$ is a closed convex set. When $m$ is large, computing the exact gradient of $\\sum_{i=1}^{m} g_i(x)$ becomes costly, and it gives rise to the so-called stochastic gradient descent (SGD) method, which only computes a single gradient at random or a mini-batch of randomly selected gradients at each iteration. However, SGD converges with a sublinear rate of $1/\\epsilon$ even if all the functions $g_i(\\cdot)$'s are strongly convex and smooth.\n\n\nIn this paper, we shall use the terminology of {\\it gradient complexity}\\/ to refer to the amount of times that the gradients\nof functions $g_i$'s need be computed before reaching an $\\epsilon$ optimal solution.\n\n\na function $g$ is called $\\mu$-strongly convex if\n\\begin{equation}\n    g(y)\\geq g(x)+\\left\\langle\\nabla g(x),y-x\\right\\rangle+\\frac{\\mu}{2}\\|y-x\\|^2, \\quad \\forall x,y\\in \\mathcal{X}\n\\end{equation}\n\nand\n%\\begin{definition}\na function $g$ is called $L$-Lipschitz smooth if\n\\begin{equation}\n    \\|\\nabla g(x)-\\nabla g(y)\\|\\leq L\\|x-y\\|, \\quad \\forall x,y\\in \\mathcal{X} .\n\\end{equation}\n\n\\begin{assumption}\nIn Model~\\eqref{pro}, $g_i(x)$ is assumed to be $L_i$-smooth and $\\mu_i$-strongly convex, $i=1,2,...,m$.\n%{\\color{red} Define $\\kappa_i= L_i /(m\\mu)+1$ as the condition number for the $i$-th component, $i=1,2,...,m$. (This may be hard to sell! Let us change the model to simply assuming $g_i$ to be $L_i$-gradient smooth and $\\mu_i$-strongly convex, and there is no $h$, and there is a constraint $x\\in \\Gamma$.) }\n\\end{assumption}\n\nRecall the problem\n\\[\nF(x)=\\sum_{i=1}^m g_i(x)=\\sum_{i=1}^m \\hat{g}_i(x)+h(x),\n\\]\n\nwhere we let $\\hat{g}_i(x) := g_i(x)-\\frac{\\mu_i}{2}\\|x\\|^2$, $h(x) := \\frac{\\mu}{2}\\|x\\|^2$, where $\\mu=\\sum_{i=1}^m \\mu_i$.\nWe use the following estimation of the gradient of the first term,\n\\[\n\\widetilde{\\nabla}^{k}=\\frac{1}{\\pi_{i_{k}}}\\left(\\nabla \\hat{g}_{i_k}(y_{i_k}^k)-\\nabla \\hat{g}_{i_k}(\\phi_{i_k}^k)\\right)+  \\sum_{i=1}^m\\nabla \\hat{g}_{i}(\\phi_{i}^k){\\color{red},}\n\\]\n\nwhere $i_k$ is sampled from $\\{1,2,\\cdots,m\\}$ with the probability\n\\[\n\\pi_i=\\frac{\\sqrt{L_i}}{2\\sum_{j=1}^m\\sqrt{L_j}}+\\frac{1}{2m},\\,\\, i=1,2,...,m.\n\\]\n\nObserve the gradient estimator $\\widetilde{\\nabla}^{k}$ is biased with expectation $\\sum_{i=1}^m\\nabla \\hat{g}_{i}(y_{i}^k)$, if we extend the definition to all $y_{i}^{k}:=\\tau_{i} x^{k}+(1-\\tau_{i}) \\phi_{i}^{k}$.\n\n\n\\begin{algorithm}[h]\n\\caption{Generalized SSNM }\n\\label{Algorithm1}\n\\begin{algorithmic}\n\\STATE{$\\phi_1=\\phi_2=\\cdots=\\phi_m=x^1$}\n\\STATE{$\\pi_i=\\frac{\\sqrt{L_i}}{2\\sum_{j=1}^m\\sqrt{L_j}}+\\frac{1}{2m}$}\n\\STATE{$\\tau_i=\\lambda/\\pi_i$}\n\\FOR{$k=1,2,..., K$}\n\\STATE{Sample $i_k$ from $\\{1,2,\\cdots,m\\}$ with probability $\\{\\pi_i\\}_{i=1}^{m}$}\n\\STATE{$y_{i_{k}}^{k}=\\tau_{i_{k}} x^{k}+(1-\\tau_{i_{k}}) \\phi_{i_{k}}^{k}$}\n\\STATE{$\\widetilde{\\nabla}^{k}=\\frac{1}{\\pi_{i_{k}}}\\left(\\nabla \\hat{g}_{i_k}(y_{i_k}^k)-\\nabla \\hat{g}_{i_k}(\\phi_{i_k}^k)\\right)+  \\sum_{i=1}^m\\nabla \\hat{g}_{i}(\\phi_{i}^k)$}\n\\STATE{Perform a proximal/projection step\\\\ $x^{k+1}=\\arg \\min _{x\\in \\Gamma}\\left\\{h(x)+\\left\\langle\\widetilde{\\nabla}^{k}, x\\right\\rangle+\\frac{1}{2 \\eta}\\left\\|x^{k}-x\\right\\|^{2}\\right\\}$}\n\\STATE{Sample $j_k$ from $\\{1,2,\\cdots,m\\}$ with probability $\\{\\pi_i\\}_{i=1}^{m}$}\n\\STATE{$\\phi_{j_{k}}^{k+1}=\\tau_{j_{k}} x^{k+1}+(1-\\tau_{j_{k}}) \\phi_{j_{k}}^{k}$}\n\\ENDFOR\n\\STATE \\textbf{return} $x^{k+1}$ \n\\end{algorithmic}\n\\end{algorithm}","units":[{"start":4029,"end":4124,"start_line":91,"end_line":93,"raw_tex":"\\begin{equation}\n    \\min_{x\\in \\Gamma} F(x)=\\sum_{i=1}^{m} g_i(x),  \\label{pro}\n\\end{equation}","sha256_utf8":"c4eae8234f819a60fa480a6d9dded0ab55446d11564035ff4abb20b754f8264d"},{"start":4125,"end":4628,"start_line":94,"end_line":94,"raw_tex":"where $g_i(x)$'s are smooth and convex functions, and $\\Gamma$ is a closed convex set. When $m$ is large, computing the exact gradient of $\\sum_{i=1}^{m} g_i(x)$ becomes costly, and it gives rise to the so-called stochastic gradient descent (SGD) method, which only computes a single gradient at random or a mini-batch of randomly selected gradients at each iteration. However, SGD converges with a sublinear rate of $1/\\epsilon$ even if all the functions $g_i(\\cdot)$'s are strongly convex and smooth.\n","sha256_utf8":"4a6535054fdb57b0a745171694c08ef44ac37842d137c9044ec105927c0c91fd"},{"start":5070,"end":5282,"start_line":97,"end_line":98,"raw_tex":"In this paper, we shall use the terminology of {\\it gradient complexity}\\/ to refer to the amount of times that the gradients\nof functions $g_i$'s need be computed before reaching an $\\epsilon$ optimal solution.\n","sha256_utf8":"e95723317c7e457ae550d43194a1abc475ade55db01cb6ff558fc8d37c3f227f"},{"start":11959,"end":12156,"start_line":161,"end_line":164,"raw_tex":"a function $g$ is called $\\mu$-strongly convex if\n\\begin{equation}\n    g(y)\\geq g(x)+\\left\\langle\\nabla g(x),y-x\\right\\rangle+\\frac{\\mu}{2}\\|y-x\\|^2, \\quad \\forall x,y\\in \\mathcal{X}\n\\end{equation}","sha256_utf8":"2f7030041d30d0ba34b315362b2472163e47752e3d6c41cc1ee54ffc5a7238f4"},{"start":12175,"end":12360,"start_line":166,"end_line":171,"raw_tex":"and\n%\\begin{definition}\na function $g$ is called $L$-Lipschitz smooth if\n\\begin{equation}\n    \\|\\nabla g(x)-\\nabla g(y)\\|\\leq L\\|x-y\\|, \\quad \\forall x,y\\in \\mathcal{X} .\n\\end{equation}","sha256_utf8":"881d6e3dae5f2995711ffe184a8ba5b7217b69a8968360fb7a6020513538a227"},{"start":12425,"end":12875,"start_line":174,"end_line":177,"raw_tex":"\\begin{assumption}\nIn Model~\\eqref{pro}, $g_i(x)$ is assumed to be $L_i$-smooth and $\\mu_i$-strongly convex, $i=1,2,...,m$.\n%{\\color{red} Define $\\kappa_i= L_i /(m\\mu)+1$ as the condition number for the $i$-th component, $i=1,2,...,m$. (This may be hard to sell! Let us change the model to simply assuming $g_i$ to be $L_i$-gradient smooth and $\\mu_i$-strongly convex, and there is no $h$, and there is a constraint $x\\in \\Gamma$.) }\n\\end{assumption}","sha256_utf8":"87987dbb5f9ebf8ff7aa96c4fed239841bb6d75f100c3874fb379c82c966e5cb"},{"start":20646,"end":20727,"start_line":295,"end_line":298,"raw_tex":"Recall the problem\n\\[\nF(x)=\\sum_{i=1}^m g_i(x)=\\sum_{i=1}^m \\hat{g}_i(x)+h(x),\n\\]","sha256_utf8":"91602698566492534fb30c5a197ceb6080bd8a316e7196d22a3668481bd36aea"},{"start":20728,"end":21109,"start_line":299,"end_line":303,"raw_tex":"where we let $\\hat{g}_i(x) := g_i(x)-\\frac{\\mu_i}{2}\\|x\\|^2$, $h(x) := \\frac{\\mu}{2}\\|x\\|^2$, where $\\mu=\\sum_{i=1}^m \\mu_i$.\nWe use the following estimation of the gradient of the first term,\n\\[\n\\widetilde{\\nabla}^{k}=\\frac{1}{\\pi_{i_{k}}}\\left(\\nabla \\hat{g}_{i_k}(y_{i_k}^k)-\\nabla \\hat{g}_{i_k}(\\phi_{i_k}^k)\\right)+  \\sum_{i=1}^m\\nabla \\hat{g}_{i}(\\phi_{i}^k){\\color{red},}\n\\]","sha256_utf8":"fe29d0b7cde1ef65aebfe45f272a4308319eb7772e7228edb4c0e4ba7522a16e"},{"start":21110,"end":21263,"start_line":304,"end_line":307,"raw_tex":"where $i_k$ is sampled from $\\{1,2,\\cdots,m\\}$ with the probability\n\\[\n\\pi_i=\\frac{\\sqrt{L_i}}{2\\sum_{j=1}^m\\sqrt{L_j}}+\\frac{1}{2m},\\,\\, i=1,2,...,m.\n\\]","sha256_utf8":"7671ce1817e0488ea84443df6182995232d3e61f01e49e89c33e99792a178c4b"},{"start":21265,"end":21481,"start_line":309,"end_line":309,"raw_tex":"Observe the gradient estimator $\\widetilde{\\nabla}^{k}$ is biased with expectation $\\sum_{i=1}^m\\nabla \\hat{g}_{i}(y_{i}^k)$, if we extend the definition to all $y_{i}^{k}:=\\tau_{i} x^{k}+(1-\\tau_{i}) \\phi_{i}^{k}$.\n","sha256_utf8":"4c2e258bd4d70cca8eb78da82d76fa35ee6a8f1b54676372f52e80803efc654a"},{"start":22267,"end":23297,"start_line":316,"end_line":333,"raw_tex":"\\begin{algorithm}[h]\n\\caption{Generalized SSNM }\n\\label{Algorithm1}\n\\begin{algorithmic}\n\\STATE{$\\phi_1=\\phi_2=\\cdots=\\phi_m=x^1$}\n\\STATE{$\\pi_i=\\frac{\\sqrt{L_i}}{2\\sum_{j=1}^m\\sqrt{L_j}}+\\frac{1}{2m}$}\n\\STATE{$\\tau_i=\\lambda/\\pi_i$}\n\\FOR{$k=1,2,..., K$}\n\\STATE{Sample $i_k$ from $\\{1,2,\\cdots,m\\}$ with probability $\\{\\pi_i\\}_{i=1}^{m}$}\n\\STATE{$y_{i_{k}}^{k}=\\tau_{i_{k}} x^{k}+(1-\\tau_{i_{k}}) \\phi_{i_{k}}^{k}$}\n\\STATE{$\\widetilde{\\nabla}^{k}=\\frac{1}{\\pi_{i_{k}}}\\left(\\nabla \\hat{g}_{i_k}(y_{i_k}^k)-\\nabla \\hat{g}_{i_k}(\\phi_{i_k}^k)\\right)+  \\sum_{i=1}^m\\nabla \\hat{g}_{i}(\\phi_{i}^k)$}\n\\STATE{Perform a proximal/projection step\\\\ $x^{k+1}=\\arg \\min _{x\\in \\Gamma}\\left\\{h(x)+\\left\\langle\\widetilde{\\nabla}^{k}, x\\right\\rangle+\\frac{1}{2 \\eta}\\left\\|x^{k}-x\\right\\|^{2}\\right\\}$}\n\\STATE{Sample $j_k$ from $\\{1,2,\\cdots,m\\}$ with probability $\\{\\pi_i\\}_{i=1}^{m}$}\n\\STATE{$\\phi_{j_{k}}^{k+1}=\\tau_{j_{k}} x^{k+1}+(1-\\tau_{j_{k}}) \\phi_{j_{k}}^{k}$}\n\\ENDFOR\n\\STATE \\textbf{return} $x^{k+1}$ \n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"231031e3ab0f0661260ccc2ec22c526b701477d81ee3ccb50ea0e5dc6d8e26b7"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem} \\label{thmssnm}\n%By carefully choosing $\\eta$ and $\\lambda$, we have an\nGeneralized SSNM (Algorithm~\\ref{Algorithm1}) requires no more than\n$$O\\left(\\left(m+\\frac{\\sum_{j=1}^m\\sqrt{L_j}}{\\sqrt{\\sum_{i=1}^m \\mu_i}}\\right)\\log(1/\\epsilon)\\right)$$ gradient computations to reach an $\\epsilon$-solution for Model~\\eqref{pro} in expectation.\n\\end{theorem}","units":[{"start":23426,"end":23793,"start_line":341,"end_line":345,"raw_tex":"\\begin{theorem} \\label{thmssnm}\n%By carefully choosing $\\eta$ and $\\lambda$, we have an\nGeneralized SSNM (Algorithm~\\ref{Algorithm1}) requires no more than\n$$O\\left(\\left(m+\\frac{\\sum_{j=1}^m\\sqrt{L_j}}{\\sqrt{\\sum_{i=1}^m \\mu_i}}\\right)\\log(1/\\epsilon)\\right)$$ gradient computations to reach an $\\epsilon$-solution for Model~\\eqref{pro} in expectation.\n\\end{theorem}","sha256_utf8":"83dd72b1241e50b6f40859c321b1ba09eaf528aa95e1e0a12480e1ab457582fb"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}{(variance upper bound)}\\label{lemma1}\n\\[\n\\mathbb{E}_{i_{k}}\\left[\\left\\|\\widetilde{\\nabla}^{k}- \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)\\right\\|^{2}\\right]\n\\leq \\sum_{i=1}^{m} \\frac{2L_{i}}{\\pi_{i}} \\left(\\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)-\\hat{g}_{i}\\left(y_{i}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right),\\phi_{i}^{k}-y_{i}^{k}\\right\\rangle  \\right).\n\\]\n\\end{lemma}\n\n\\begin{proof}\nSince each $\\hat{g}_i(x)$ is convex and $L_i$ smooth, Theorem 2.1.5 in \\cite{nesterov2018lectures} stipulates\n\\begin{equation}\n\\left\\|\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\nabla \\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)\\right\\|^{2}\\leq 2L_{i}\\left(\\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)-\\hat{g}_{i}\\left(y_{i}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right),\\phi_{i}^{k}-y_{i}^{k}\\right\\rangle  \\right). \\label{squineq}\n\\end{equation}\nObserving\n\\begin{equation} \\label{ex-ineq}\n\\mathbb{E}\\|\\zeta-\\mathbb{E} \\zeta\\|^{2}=\\mathbb{E}\\|\\zeta\\|^{2}-\\|\\mathbb{E} \\zeta\\|^{2},\n\\end{equation}\n%and by using the above inequality, we have\nwe have\n\\begin{eqnarray*}\n    & & \\mathbb{E}_{i_{k}}\\left[\\left\\|\\widetilde{\\nabla}^{k}- \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)\\right\\|^{2}\\right] \\\\\n    &=& \\mathbb{E}_{i_{k}}\\left[\\left\\|\\frac{1}{\\pi_{i_{k}}}\\left(\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right)-\\nabla \\hat{g}_{i_{k}}\\left(\\phi_{i_{k}}^{k}\\right)\\right)- \\sum_{i=1}^{m}\\left(\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\nabla \\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)\\right) \\right\\|^{2}\\right] \\\\\n    & \\overset{\\eqref{ex-ineq}}{\\leq} & \\mathbb{E}_{i_{k}}\\left[\\left\\|\\frac{1}{\\pi_{i_{k}}}\\left(\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right)-\\nabla \\hat{g}_{i_{k}}\\left(\\phi_{i_{k}}^{k}\\right)\\right) \\right\\|^{2}\\right] \\\\ %\\label{ineq1}\\\\\n    & \\overset{\\eqref{squineq}}{\\leq} & \\mathbb{E}_{i_{k}}\\left[\\frac{2L_{i_{k}}}{ \\pi_{i_{k}}^2} \\left(\\hat{g}_{i_{k}}\\left(\\phi_{i_{k}}^{k}\\right)-\\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right),\\phi_{i_{k}}^{k}-y_{i_{k}}^{k}\\right\\rangle  \\right)\\right] \\\\ %\\label{ineq2}\\\\\n    & = & \\sum_{i=1}^{m} \\frac{2L_{i}}{\\pi_{i}} \\left(\\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)-\\hat{g}_{i}\\left(y_{i}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right),\\phi_{i}^{k}-y_{i}^{k}\\right\\rangle \\right).\n\\end{eqnarray*}\n%where \\eqref{ineq1} follows from $\\mathbb{E}\\|\\zeta-\\mathbb{E} \\zeta\\|^{2}=\\mathbb{E}\\|\\zeta\\|^{2}-\\|\\mathbb{E} \\zeta\\|^{2}$ and \\eqref{ineq2} uses \\eqref{squineq}.\n\\end{proof}\n\n\\begin{lemma} \\label{lemma-3}\nSuppose $h(x)$ is $\\mu$-strongly convex, and $x^{k+1}$ satisfies\n\\[\nx^{k+1} := \\arg \\min _{x\\in \\Gamma}\\left\\{h(x)+\\left\\langle\\widetilde{\\nabla}^{k}, x\\right\\rangle+\\frac{1}{2 \\eta}\\left\\|x^{k}-x\\right\\|^{2}\\right\\}.\n\\]\nThen, for all $u\\in \\Gamma$, it holds that\n\\begin{eqnarray*}\n& & \\left\\langle\\widetilde{\\nabla}^{k}, x^{k+1}-u\\right\\rangle \\\\\n& \\leq & -\\frac{1}{2 \\eta}\\left\\|x^{k+1}-x^{k}\\right\\|^{2}+\\frac{1}{2 \\eta}\\left\\|x^{k}-u\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta}\\left\\|x^{k+1}-u\\right\\|^{2}+h(u)-h\\left(x^{k+1}\\right).\n\\end{eqnarray*}\n\\end{lemma}\n\n\\begin{proof}\nBy the optimality condition, there exists an $h'\\in \\partial h(x)|_{x=x^{k+1}}$, such that\n\\[\n\\left\\langle\\frac{1}{\\eta}(x^{k+1}-x^{k})+\\widetilde{\\nabla}^{k}+h',u-x^{k+1} \\right\\rangle\\geq 0,\\,\\,\\, \\forall u \\in \\Gamma.\n\\]\nObserve the identity $\\left\\langle x^{k+1}-x^{k},x^{k+1}-u \\right\\rangle=\\frac 1 2\\|x^{k+1}-x^{k}\\|^2+\\frac 1 2\\|x^{k+1}-u\\|^2-\\frac 1 2\\|x^{k}-u\\|^2$ and the inequality $\\left\\langle h',u-x^{k+1} \\right\\rangle\\leq h(u)-h\\left(x^{k+1}\\right)-\\frac{\\mu}{2}\\|x^{k+1}-u\\|^2$, the lemma follows.\n\\end{proof}\n\n\\begin{proof}[Proof of Theorem \\ref{thmssnm}]\nFirst, we assume that we can choose $\\eta>0$ and $\\lambda>0$ such that $0\\leq \\tau_{i} \\leq 1$, $\\frac{1}{\\eta}-\\frac{\\sum_{i=1}^{m}\\tau_i L_i}{m} \\geq \\frac{L_i\\tau_i}{m \\pi_i(1-\\tau_i)}$, $\\forall i=1,2,...,m$ hold. Later in the proof, we will show how to choose $\\eta>0$ and $\\lambda>0$ to satisfy these inequalities.\n\nBy the convexity of $\\hat{g}_{i_{k}}(\\cdot)$,\n\\begin{eqnarray*}\n & &    \\frac{1}{\\pi_{i_{k}}}\\left(\\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right)-\\hat{g}_{i_{k}}\\left(x^{\\star}\\right)\\right)\n    \\leq \\frac{1}{\\pi_{i_{k}}}\\left\\langle\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right), y_{i_{k}}^{k}-x^{\\star}\\right\\rangle \\\\\n    &=& \\frac{1}{\\pi_{i_{k}}}\\frac{1-\\tau_{i_{k}}}{\\tau_{i_{k}}}\\left\\langle\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right), \\phi_{i_{k}}^{k}-y_{i_{k}}^{k}\\right\\rangle+\\left\\langle\\frac{1}{ \\pi_{i_{k}}}\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right)-\\widetilde{\\nabla}^{k}, x^{k}-x^{\\star}\\right\\rangle\\\\\n    & &+\\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle+\\left\\langle\\widetilde{\\nabla}^{k}, x^{k+1}-x^{\\star}\\right\\rangle\n\\end{eqnarray*}\nwhere the last step uses the definition that $y_{i}=\\tau_{i} x^{k}+(1-\\tau_{i}) \\phi_{i}^{k}$, for $i=1,2,...,m$.\n%$y_{i_{k}}^{k}=\\tau_{i_{k}} x^{k}+(1-\\tau_{i_{k}}) \\phi_{i_{k}}^{k}$. {\\color{red} Replace to $y_{i}=\\tau_{i} x^{k}+(1-\\tau_{i}) \\phi_{i}^{k}, \\forall i=1,2,...,m$.}\n\nObserve\n\\[\n\\mathbb{E}_{i_{k}}\\left[\\widetilde{\\nabla}^{k}-\\frac{1}{\\pi_{i_{k}}}\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right)\\right]=\\mathbb{E}_{i_{k}}\\left[\\widetilde{\\nabla}^{k}\\right]-\\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)=0.\n\\]\nBy taking expectation with respect to sample $i_k$, it follows\n\\begin{eqnarray} \\label{opt-gap} \\\\\n& & \\sum_{i=1}^{m} \\left( \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\hat{g}_i\\left(x^{\\star}\\right) \\right)  \\nonumber  \\\\\n&\\leq& \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle+\\mathbb{E}_{i_{k}}\\left[\\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle\\right]+\\mathbb{E}_{i_{k}}\\left[\\left\\langle\\widetilde{\\nabla}^{k}, x^{k+1}-x^{\\star}\\right\\rangle\\right] . \\nonumber\n\\end{eqnarray}\n\n%{\\color{red}By the convexity of $\\hat{g}_{i}(\\cdot)$,\n%\\begin{eqnarray*}\n%    \\frac{1}{\\pi_{i}}\\left(\\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\hat{g}_{i}\\left(x^{\\star}\\right)\\right)\n%    &\\leq& \\frac{1}{\\pi_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), y_{i}^{k}-x^{\\star}\\right\\rangle \\\\\n%    &=& \\frac{1}{\\pi_{i}}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle+\\left\\langle\\frac{1}{ \\pi_{i}}\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\widetilde{\\nabla}^{k}, x^{k}-x^{\\star}\\right\\rangle\\\\\n%    & &+\\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle+\\left\\langle\\widetilde{\\nabla}^{k}, x^{k+1}-x^{\\star}\\right\\rangle\n%\\end{eqnarray*}\n%where the last step uses the definition that $y_{i}^{k}=\\tau_{i} x^{k}+(1-\\tau_{i}) \\phi_{i}^{k}$.\n\n%Observe\n%\\[\n%\\mathbb{E}_{i_{k}}\\left[\\widetilde{\\nabla}^{k}\\right]=\\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right).\n%\\]\n%By taking expectation with respect to sample $i$ with probability $\\{\\pi_i\\}_{i=1}^{m}$, it follows\n%\\begin{eqnarray} \\label{opt-gap}\n%& & \\sum_{i=1}^{m} \\left( \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\hat{g}_i\\left(x^{\\star}\\right) \\right) \\nonumber \\\\\n%&\\leq& \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle+\\mathbb{E}_{i_{k}}\\left[\\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle\\right]+\\mathbb{E}_{i_{k}}\\left[\\left\\langle\\widetilde{\\nabla}^{k}, x^{k+1}-x^{\\star}\\right\\rangle\\right] .\n%\\end{eqnarray}\n%}\n\nTheorem 2.1.5 in \\cite{nesterov2018lectures} tells us that\n\\[\n\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)-\\hat{g}_{j_{k}}\\left(y_{j_{k}}^{k}\\right) \\leq\\left\\langle\\nabla \\hat{g}_{j_{k}}\\left(y_{j_{k}}^{k}\\right), \\phi_{j_{k}}^{k+1}-y_{j_{k}}^{k}\\right\\rangle+\\frac{L_{j_{k}}}{2}\\left\\|\\phi_{j_{k}}^{k+1}-y_{j_{k}}^{k}\\right\\|^{2} .\n\\]\n\nDividing the above inequality by $\\pi_{j_{k}}\\tau_{j_{k}}$ and taking expectation with respect to sample $j_k$, we obtain\n\\begin{eqnarray*}\n& & \\mathbb{E}_{j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]-\\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_i\\left(y_i^{k}\\right) \\\\\n&\\leq& \\left\\langle \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), x^{k+1}-x^{k}\\right\\rangle+\\frac{\\sum_{i=1}^{m}\\tau_i L_i}{2}\\|x^{k+1}-x^{k}\\|^2,\n\\end{eqnarray*}\nwhich is equivalent to\n\\begin{eqnarray*}\n     \\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle\n     &\\leq&\n    \\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)- \\mathbb{E}_{j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right] \\\\\n    & & +\\left\\langle \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\widetilde{\\nabla}^{k},\n    x^{k+1}-x^{k}\\right\\rangle+\\frac{\\sum_{i=1}^{m}\\tau_i L_i}{2}\\|x^{k+1}-x^{k}\\|^2.\n\\end{eqnarray*}\nTaking expectation with respect to sample $i_k$ we get\n\\begin{eqnarray*}\n     \\mathbb{E}_{i_{k}}\\left[\\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle\\right] &\\leq&\n    \\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)\n    - \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]\\\\\n& & +\\mathbb{E}_{i_{k}}\\left[\\left\\langle \\sum_{i=1}^{m}\n\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\widetilde{\\nabla}^{k}, x^{k+1}-x^{k}\\right\\rangle\\right]+\\frac{\\sum_{i=1}^{m}\\tau_i L_i}{2}\\mathbb{E}_{i_{k}}\\left[\\|x^{k+1}-x^{k}\\|^2\\right] .\n\\end{eqnarray*}\n\nBy Lemma~\\ref{lemma-3}, taking expectation we have\n\\begin{eqnarray*}\n& & \\mathbb{E}_{i_{k}}\\left[ \\left\\langle\\widetilde{\\nabla}^{k}, x^{k+1}-x^\\star\\right\\rangle \\right]  \\\\\n&\\leq& - \\mathbb{E}_{i_{k}}\\left[\\frac{1}{2 \\eta}\\left\\|x^{k+1}-x^{k}\\right\\|^{2}\\right]+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^\\star\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[ \\left\\|x^{k+1}-x^\\star\\right\\|^{2}\\right]+h(x^\\star) - \\mathbb{E}_{i_{k}}\\left[ h\\left(x^{k+1}\\right)\\right] .\n\\end{eqnarray*}\n\nApplying the above inequalities on \\eqref{opt-gap}, we have % \\eqref{eq1}, we obtain\n\\begin{eqnarray*}\n    & & \\sum_{i=1}^{m} \\left( \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\hat{g}_i \\left(x^{\\star}\\right) \\right) \\\\\n    &\\leq&\n     \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle\n    +\\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)\n    - \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right) \\right] \\\\\n   & & +\\mathbb{E}_{i_{k}}\\left[\\left\\langle \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\widetilde{\\nabla}^{k}, x^{k+1}-x^{k}\\right\\rangle\\right]+\\frac{\\sum_{i=1}^{m}\\tau_i L_i}{2}\\mathbb{E}_{i_{k}}\\left[\\|x^{k+1}-x^{k}\\|^2\\right]\\\\\n   & & -\\frac{1}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{k}\\right\\|^{2}\\right]+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right]\n    +h\\left(x^{\\star}\\right)-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right]\\\\\n   & \\leq&  \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle\n    +\\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)\n    - \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]\\\\\n   & & + \\frac{1}{2\\left(\\frac{1}{\\eta}-\\sum_{i=1}^{m}\\tau_i L_i \\right)}\\mathbb{E}_{i_{k}}\\left[\\left\\| \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\widetilde{\\nabla}^{k}\\right\\|^2\\right]+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}\\\\\n   & & -\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right]+h\\left(x^{\\star}\\right)-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right]\\\\\n   &\\leq&  \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle\n    +\\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)\n    - \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]\\\\\n   & &+ \\frac{1}{2\\left(\\frac{1}{\\eta}-\\sum_{i=1}^{m}\\tau_i L_i \\right)}\\sum_{i=1}^{m} \\frac{2L_{i}}{\\pi_{i}} \\left(\\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)-\\hat{g}_{i}\\left(y_{i}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right),\\phi_{i}^{k}-y_{i}^{k}\\right\\rangle  \\right)\\\\\n   & &+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right]+h\\left(x^{\\star}\\right)-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right]\\\\\n   &\\leq&  \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle\n    +\\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)\n    - \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]\\\\\n  &  &+ \\sum_{i=1}^{m} \\frac{1-\\tau_{i}}{\\tau_{i}} \\left(\\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)-\\hat{g}_{i}\\left(y_{i}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right),\\phi_{i}^{k}-y_{i}^{k}\\right\\rangle\\right)\\\\\n  &  &+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right]+h\\left(x^{\\star}\\right)-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right]\n\\end{eqnarray*}\nwhere the second inequality follows from the inequality $\\langle a, b\\rangle \\leq \\frac{1}{2 \\beta}\\|a\\|^{2}+\\frac{\\beta}{2}\\|b\\|^{2}$ and $\\frac{1}{\\eta}-\\sum_{i=1}^{m}\\tau_i L_i >0$, and the third inequality is due to Lemma \\ref{lemma1}, and the last inequality holds by the assumption that $\\frac{1}{\\eta}-\\sum_{i=1}^{m}\\tau_i L_i \\geq \\frac{L_i\\tau_i}{\\pi_i(1-\\tau_i)}$.\n\nCanceling out and rearranging the terms, we have\n\\begin{eqnarray*}\n    &     &  \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}} \\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]-F(x^{\\star})\\\\\n    &\\leq & \\sum_{i=1}^{m} \\frac{1-\\tau_{i}}{\\tau_{i}} \\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right]-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right].\n\\end{eqnarray*}\n\nBy the convexity of $h(\\cdot)$ and $\\phi_{j_{k}}^{k+1}=\\tau_{j_{k}} x^{k+1}+(1-\\tau_{j_{k}}) \\phi_{j_{k}}^{k}$, as long as $0\\leq \\tau_{j_{k}} \\leq 1$, we have\n\\[\nh(\\phi_{j_{k}}^{k+1})\\leq \\tau_{j_{k}} h(x^{k+1})+(1-\\tau_{j_{k}}) h(\\phi_{j_{k}}^{k}) .\n\\]\nDividing the above inequality by $m \\pi_{j_{k}} \\tau_{j_{k}}$ and taking expectation with respect to sample $j_k$ and sample $i_k$, we obtain\n\\[\n-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right] \\leq  \\sum_{i=1}^{m} \\frac{1-\\tau_{j_{k}}}{m \\tau_{j_{k}}} h\\left(\\phi_{i}^{k}\\right)- \\mathbb{E}_{i_{k}, j_{k}}\\left[\\frac{1}{m \\pi_{j_{k}} \\tau_{j_{k}}}     h\\left(\\phi_{j_{k}}^{k+1}\\right)\\right] .\n\\]\nApplying the above inequality and using $F_{i}(\\cdot):=\\hat{g}_{i}(\\cdot)+\\frac 1 m h(\\cdot)$, we further derive\n\\begin{eqnarray*}\n    &      &  \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}F_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]-F(x^{\\star})\\\\\n    &\\leq  & \\sum_{i=1}^{m} \\frac{1-\\tau_{i}}{\\tau_{i}} F_{i}\\left(\\phi_{i}^{k}\\right)+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right] .\n\\end{eqnarray*}\n\nAdding $\\mathbb{E}_{i_{k},j_{k}}\\left[\\sum_{i\\neq j_k}\\frac{1}{\\pi_i\\tau_i}F_i\\left(\\phi_i^{k}\\right)\\right]$ to both sides, we have\n\\begin{eqnarray*}\n    &     & \\mathbb{E}_{i_{k},j_{k}}\\left[\\sum_{i=1}^m\\frac{1}{\\pi_i\\tau_i}\\left(F_i\\left(\\phi_i^{k+1}\\right)-F_i(x^{\\star})\\right)\\right]\\\\\n    &\\leq & \\sum_{i=1}^{m} \\frac{1-\\pi_i\\tau_{i}}{\\pi_i\\tau_{i}} \\left(F_{i}\\left(\\phi_{i}^{k}\\right)-F_i(x^{\\star})\\right)+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}} \\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2} \\right] .\n\\end{eqnarray*}\nNoticing $\\sum_{i=1}^{m} \\left(F_{i}\\left(\\phi_{i}^{k}\\right)-F_i(x^{\\star})\\right)$ may not be positive, we need to add the following term in our Lyapunov function\n\\begin{eqnarray*}\n  & &    -\\sum_{i=1}^{m}\\frac{1}{\\pi_i\\tau_i}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k+1}-x^{\\star}\\right\\rangle \\\\\n  &=&  -\\frac{1}{\\pi_{j_{k}} \\tau_{j_{k}}}\\left\\langle\\nabla F_{j_{k}}\\left(x^{\\star}\\right), \\phi_{j_{k}}^{k+1}-x^{\\star}\\right\\rangle-\n  \\sum_{i \\neq j_k } %I \\atop \\mathfrak{l}}^{m}\n  \\frac{1}{\\pi_i\\tau_i}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k}-x^{\\star}\\right\\rangle\\\\\n  &=& -\\frac{1}{\\pi_{j_{k}}}\\left\\langle\\nabla F_{j_{k}}\\left(x^{\\star}\\right), x^{k+1}-x^{\\star}\\right\\rangle\n    +\\frac{1}{\\pi_{j_{k}}}\\left\\langle\\nabla F_{j_{k}}\\left(x^{\\star}\\right), \\phi_{j_{k}}^{k}-x^{\\star}\\right\\rangle\\\\\n  &  & - \\sum_{i =1}^{m}\\frac{1}{\\pi_i\\tau_i}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k}-x^{\\star}\\right\\rangle .\n\\end{eqnarray*}\n\nTaking expectation with respect to samples $i_k$ and $j_k$ and using $\\pi_i\\tau_i=\\lambda$, we get\n\\[\n\\mathbb{E}_{i_{k},j_{k}}\\left[-\\frac{1}{\\lambda} \\sum_{i=1}^{m}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k+1}-x^{\\star}\\right\\rangle\\right]=\n-\\frac{1-\\lambda}{\\lambda} \\sum_{i =1}^{m}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k}-x^{\\star}\\right\\rangle .\n\\]\n\nDenoting $D^k:=\\sum_{i =1}^{m}F_{i}\\left(\\phi_{i}^{k}\\right)-F\\left(x^{\\star}\\right)-\\sum_{i =1}^{m}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k}-x^{\\star}\\right\\rangle$ and $P^{k} :=\\left\\|x^{k}-x^{\\star}\\right\\|^{2}$, we have\n\\[\n\\frac{1}{\\lambda}\\mathbb{E}_{i_{k},j_{k}}\\left[D^{k+1}\\right]+\\frac{1+\\eta \\mu}{2 \\eta}\\mathbb{E}_{i_{k}}\\left[P^{k+1}\\right]\\leq \\frac{1-\\lambda}{\\lambda} D^k+\\frac{1}{2 \\eta} P^k.\n\\]\n\nFinally, it remains to choose $\\eta$ and $\\lambda$ so as to satisfy\n \\[\n 0\\leq \\tau_{i} \\leq 1, \\mbox{ and }  \\frac{1}{\\eta}-\\sum_{i=1}^{m}\\tau_i L_i \\geq \\frac{L_i\\tau_i}{\\pi_i(1-\\tau_i)},\\,\\,\\, i=1,2,...,m.\n \\]\nWe consider two cases separately.\n\n\\textbf{Case I.} If $\\sqrt{\\mu}\\leq \\frac{\\sum_{j=1}^m\\sqrt{L_j}}{m}$, then we choose\n\\[\n\\lambda := \\frac{\\sqrt{\\mu}}{4\\sum_{j=1}^m\\sqrt{L_j}}, \\quad \\eta := \\frac{1}{4\\sqrt{\\mu}\\sum_{j=1}^{m}\\sqrt{L_j} }.\n\\]\n\nRecall\n$\\pi_i=\\frac{\\sqrt{L_i}}{2\\sum_{j=1}^m\\sqrt{L_j}}+\\frac{1}{2m}$ for all $1\\le i\\le m$. Therefore,\n\\[\n\\tau_i=\\frac{\\lambda}{\\pi_i}=\\frac{\\sqrt{\\mu}}{2\\sqrt{L_i}+\\frac 2 m \\sum_{j=1}^m\\sqrt{L_j}}\\leq \\frac{\\sqrt{\\mu}}{\\frac 2 m \\sum_{j=1}^m\\sqrt{L_j}}\\leq \\frac 1 2\n\\]\nand\n\\begin{eqnarray*}\n     \\sum_{j=1}^{m}\\tau_j L_j +\\frac{L_i\\tau_i}{\\pi_i(1-\\tau_i)} &\\leq& \\sum_{j=1}^{m}L_j\\frac{\\sqrt{\\mu}}{2\\sqrt{L_j}}+\\frac{2\\lambda L_i}{ \\pi_i^2}\\\\\n    &\\leq& \\frac{\\sqrt{\\mu}\\sum_{j=1}^{m}\\sqrt{L_j} }{2}+\\frac{8(\\sum_{j=1}^{m}\\sqrt{L_j})^2\\lambda L_i}{L_i}\\\\\n    &\\leq& 3\\sqrt{\\mu}\\sum_{j=1}^{m}\\sqrt{L_j} .\n\\end{eqnarray*}\n\nSince $\\eta=\\frac{1}{4\\sqrt{\\mu}\\sum_{j=1}^{m}\\sqrt{L_j} }$, we know $\\frac{1}{\\eta}-\\sum_{j=1}^{m}\\tau_j L_j \\geq \\frac{L_i\\tau_i}{ \\pi_i(1-\\tau_i)}$ and\n\\[\n\\frac{1}{\\lambda}\\geq (1+\\eta \\mu)\\frac{1-\\lambda}{\\lambda} .\n\\]\nThus, in this case we obtain\n\\[\n\\frac{1}{\\lambda}\\mathbb{E}_{i_{k},j_{k}}\\left[D^{k+1}\\right]+\\frac{1+\\eta \\mu}{2 \\eta}\\mathbb{E}_{i_{k}}\\left[P^{k+1}\\right]\\leq (1+\\eta \\mu)^{-1}\\left( \\frac{1}{\\lambda} D^k+\\frac{1+\\eta \\mu}{2 \\eta} P^k\\right).\n\\]\n\nTelescoping the above inequalities, we have\n%\\begin{equation}\n\\begin{eqnarray*}\n& & \\mathbb{E}\\left[\\left\\|x^{K+1}-x^{\\star}\\right\\|^{2}\\right] \\\\\n&\\leq& \\left(1+\\frac{\\sqrt{\\mu}}{4\\sum_{j=1}^m\\sqrt{L_j}}\\right)^{-K} \\cdot\\left(\\frac{2\\eta}{\\lambda(1+\\eta \\mu)}\\left(F\\left(x^{1}\\right)-F\\left(x^{\\star}\\right)\\right)+\\left\\|x^{1}-x^{\\star}\\right\\|^{2}\\right) .\n\\end{eqnarray*}\n%\\end{equation}\n\n\\textbf{Case II.} If $\\sqrt{\\mu}> \\frac{\\sum_{j=1}^m\\sqrt{L_j}}{m}$, then we choose\n\\[\n\\lambda := \\frac{1}{4m}, \\quad \\eta := \\frac{1}{4\\mu m}.\n\\]\n\nIn this case, we have\n\\[\n\\tau_i=\\frac{\\lambda}{\\pi_i}=\\frac{\\frac{1}{2m}\\sum_{j=1}^m\\sqrt{L_j}}{\\sqrt{L_i}+\\frac 1 m \\sum_{j=1}^m\\sqrt{L_j}}\\leq \\frac 1 2\n\\]\nand\n\\begin{eqnarray*}\n    \\sum_{j=1}^{m}\\tau_j L_j +\\frac{L_i\\tau_i}{\\pi_i(1-\\tau_i)}&\\leq& \\frac{1}{2m}\\left(\\sum_{j=1}^m\\sqrt{L_j}\\right)^2+\\frac{2\\lambda L_i}{ \\pi_i^2}\\\\\n    &\\leq& \\frac{\\left(\\sum_{j=1}^m\\sqrt{L_j}\\right)^2}{2m}+\\frac{8\\left(\\sum_{j=1}^{m}\\sqrt{L_j}\\right)^2\\lambda L_i}{L_i}\\\\\n    &\\leq& \\frac{3\\left(\\sum_{j=1}^{m}\\sqrt{L_j}\\right)^2 }{m}\\\\\n    &\\leq& 3 \\mu m.\n\\end{eqnarray*}\n\nSince $\\eta=\\frac{1}{4\\mu m}$, we have $\\frac{1}{\\eta}-\\sum_{j=1}^{m}\\tau_j L_j \\geq \\frac{L_i\\tau_i}{\\pi_i(1-\\tau_i)}$, and\n\\[\n\\frac{1}{\\lambda}\\geq (1+\\eta \\mu)\\frac{1-\\lambda}{\\lambda} .\n\\]\n\nAs a consequence,\n\\[\n\\frac{1}{\\lambda}\\mathbb{E}_{i_{k},j_{k}}\\left[D^{k+1}\\right]+\\frac{1+\\eta \\mu}{2 \\eta}\\mathbb{E}_{i_{k}}\\left[P^{k+1}\\right]\\leq (1+\\eta \\mu)^{-1}\\left( \\frac{1}{\\lambda} D^k+\\frac{1+\\eta \\mu}{2 \\eta} P^k\\right).\n\\]\nTaking expectations and telescoping the above inequalities, we have\n\\begin{equation}\n\\mathbb{E}\\left[\\left\\|x^{K+1}-x^{\\star}\\right\\|^{2}\\right] \\leq\\left(1+\\frac{1}{4m}\\right)^{-K} \\cdot\\left(\\frac{2\\eta}{\\lambda(1+\\eta \\mu)}\\left(F\\left(x^{1}\\right)-F\\left(x^{\\star}\\right)\\right)+\\left\\|x^{1}-x^{\\star}\\right\\|^{2}\\right) .\n\\end{equation}\nSummarizing Case I and Case II, the theorem is proven.\n\\end{proof}","units":[{"start":25143,"end":25562,"start_line":365,"end_line":370,"raw_tex":"\\begin{lemma}{(variance upper bound)}\\label{lemma1}\n\\[\n\\mathbb{E}_{i_{k}}\\left[\\left\\|\\widetilde{\\nabla}^{k}- \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)\\right\\|^{2}\\right]\n\\leq \\sum_{i=1}^{m} \\frac{2L_{i}}{\\pi_{i}} \\left(\\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)-\\hat{g}_{i}\\left(y_{i}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right),\\phi_{i}^{k}-y_{i}^{k}\\right\\rangle  \\right).\n\\]\n\\end{lemma}","sha256_utf8":"25a0eeb012de56460f9aabd473042b7121e7ae1fa399b44c85d2b0e1f7361322"},{"start":25564,"end":27693,"start_line":372,"end_line":391,"raw_tex":"\\begin{proof}\nSince each $\\hat{g}_i(x)$ is convex and $L_i$ smooth, Theorem 2.1.5 in \\cite{nesterov2018lectures} stipulates\n\\begin{equation}\n\\left\\|\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\nabla \\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)\\right\\|^{2}\\leq 2L_{i}\\left(\\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)-\\hat{g}_{i}\\left(y_{i}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right),\\phi_{i}^{k}-y_{i}^{k}\\right\\rangle  \\right). \\label{squineq}\n\\end{equation}\nObserving\n\\begin{equation} \\label{ex-ineq}\n\\mathbb{E}\\|\\zeta-\\mathbb{E} \\zeta\\|^{2}=\\mathbb{E}\\|\\zeta\\|^{2}-\\|\\mathbb{E} \\zeta\\|^{2},\n\\end{equation}\n%and by using the above inequality, we have\nwe have\n\\begin{eqnarray*}\n    & & \\mathbb{E}_{i_{k}}\\left[\\left\\|\\widetilde{\\nabla}^{k}- \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)\\right\\|^{2}\\right] \\\\\n    &=& \\mathbb{E}_{i_{k}}\\left[\\left\\|\\frac{1}{\\pi_{i_{k}}}\\left(\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right)-\\nabla \\hat{g}_{i_{k}}\\left(\\phi_{i_{k}}^{k}\\right)\\right)- \\sum_{i=1}^{m}\\left(\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\nabla \\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)\\right) \\right\\|^{2}\\right] \\\\\n    & \\overset{\\eqref{ex-ineq}}{\\leq} & \\mathbb{E}_{i_{k}}\\left[\\left\\|\\frac{1}{\\pi_{i_{k}}}\\left(\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right)-\\nabla \\hat{g}_{i_{k}}\\left(\\phi_{i_{k}}^{k}\\right)\\right) \\right\\|^{2}\\right] \\\\ %\\label{ineq1}\\\\\n    & \\overset{\\eqref{squineq}}{\\leq} & \\mathbb{E}_{i_{k}}\\left[\\frac{2L_{i_{k}}}{ \\pi_{i_{k}}^2} \\left(\\hat{g}_{i_{k}}\\left(\\phi_{i_{k}}^{k}\\right)-\\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right),\\phi_{i_{k}}^{k}-y_{i_{k}}^{k}\\right\\rangle  \\right)\\right] \\\\ %\\label{ineq2}\\\\\n    & = & \\sum_{i=1}^{m} \\frac{2L_{i}}{\\pi_{i}} \\left(\\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)-\\hat{g}_{i}\\left(y_{i}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right),\\phi_{i}^{k}-y_{i}^{k}\\right\\rangle \\right).\n\\end{eqnarray*}\n%where \\eqref{ineq1} follows from $\\mathbb{E}\\|\\zeta-\\mathbb{E} \\zeta\\|^{2}=\\mathbb{E}\\|\\zeta\\|^{2}-\\|\\mathbb{E} \\zeta\\|^{2}$ and \\eqref{ineq2} uses \\eqref{squineq}.\n\\end{proof}","sha256_utf8":"aa073a0022ccd6855836d0075365be193b198ab788290fbe17245f11214b7bfe"},{"start":27771,"end":28360,"start_line":395,"end_line":405,"raw_tex":"\\begin{lemma} \\label{lemma-3}\nSuppose $h(x)$ is $\\mu$-strongly convex, and $x^{k+1}$ satisfies\n\\[\nx^{k+1} := \\arg \\min _{x\\in \\Gamma}\\left\\{h(x)+\\left\\langle\\widetilde{\\nabla}^{k}, x\\right\\rangle+\\frac{1}{2 \\eta}\\left\\|x^{k}-x\\right\\|^{2}\\right\\}.\n\\]\nThen, for all $u\\in \\Gamma$, it holds that\n\\begin{eqnarray*}\n& & \\left\\langle\\widetilde{\\nabla}^{k}, x^{k+1}-u\\right\\rangle \\\\\n& \\leq & -\\frac{1}{2 \\eta}\\left\\|x^{k+1}-x^{k}\\right\\|^{2}+\\frac{1}{2 \\eta}\\left\\|x^{k}-u\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta}\\left\\|x^{k+1}-u\\right\\|^{2}+h(u)-h\\left(x^{k+1}\\right).\n\\end{eqnarray*}\n\\end{lemma}","sha256_utf8":"f8fc9191fbc5a4d9f927b3f42e8cc8e94e00c450d444305c3ece4bb17e20b4dd"},{"start":28362,"end":28903,"start_line":407,"end_line":413,"raw_tex":"\\begin{proof}\nBy the optimality condition, there exists an $h'\\in \\partial h(x)|_{x=x^{k+1}}$, such that\n\\[\n\\left\\langle\\frac{1}{\\eta}(x^{k+1}-x^{k})+\\widetilde{\\nabla}^{k}+h',u-x^{k+1} \\right\\rangle\\geq 0,\\,\\,\\, \\forall u \\in \\Gamma.\n\\]\nObserve the identity $\\left\\langle x^{k+1}-x^{k},x^{k+1}-u \\right\\rangle=\\frac 1 2\\|x^{k+1}-x^{k}\\|^2+\\frac 1 2\\|x^{k+1}-u\\|^2-\\frac 1 2\\|x^{k}-u\\|^2$ and the inequality $\\left\\langle h',u-x^{k+1} \\right\\rangle\\leq h(u)-h\\left(x^{k+1}\\right)-\\frac{\\mu}{2}\\|x^{k+1}-u\\|^2$, the lemma follows.\n\\end{proof}","sha256_utf8":"d1fce1abc62962fab9f688e300c2fdb6586368a467486f6299ec82f2a54ec259"},{"start":28952,"end":46761,"start_line":417,"end_line":639,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{thmssnm}]\nFirst, we assume that we can choose $\\eta>0$ and $\\lambda>0$ such that $0\\leq \\tau_{i} \\leq 1$, $\\frac{1}{\\eta}-\\frac{\\sum_{i=1}^{m}\\tau_i L_i}{m} \\geq \\frac{L_i\\tau_i}{m \\pi_i(1-\\tau_i)}$, $\\forall i=1,2,...,m$ hold. Later in the proof, we will show how to choose $\\eta>0$ and $\\lambda>0$ to satisfy these inequalities.\n\nBy the convexity of $\\hat{g}_{i_{k}}(\\cdot)$,\n\\begin{eqnarray*}\n & &    \\frac{1}{\\pi_{i_{k}}}\\left(\\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right)-\\hat{g}_{i_{k}}\\left(x^{\\star}\\right)\\right)\n    \\leq \\frac{1}{\\pi_{i_{k}}}\\left\\langle\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right), y_{i_{k}}^{k}-x^{\\star}\\right\\rangle \\\\\n    &=& \\frac{1}{\\pi_{i_{k}}}\\frac{1-\\tau_{i_{k}}}{\\tau_{i_{k}}}\\left\\langle\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right), \\phi_{i_{k}}^{k}-y_{i_{k}}^{k}\\right\\rangle+\\left\\langle\\frac{1}{ \\pi_{i_{k}}}\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right)-\\widetilde{\\nabla}^{k}, x^{k}-x^{\\star}\\right\\rangle\\\\\n    & &+\\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle+\\left\\langle\\widetilde{\\nabla}^{k}, x^{k+1}-x^{\\star}\\right\\rangle\n\\end{eqnarray*}\nwhere the last step uses the definition that $y_{i}=\\tau_{i} x^{k}+(1-\\tau_{i}) \\phi_{i}^{k}$, for $i=1,2,...,m$.\n%$y_{i_{k}}^{k}=\\tau_{i_{k}} x^{k}+(1-\\tau_{i_{k}}) \\phi_{i_{k}}^{k}$. {\\color{red} Replace to $y_{i}=\\tau_{i} x^{k}+(1-\\tau_{i}) \\phi_{i}^{k}, \\forall i=1,2,...,m$.}\n\nObserve\n\\[\n\\mathbb{E}_{i_{k}}\\left[\\widetilde{\\nabla}^{k}-\\frac{1}{\\pi_{i_{k}}}\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right)\\right]=\\mathbb{E}_{i_{k}}\\left[\\widetilde{\\nabla}^{k}\\right]-\\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)=0.\n\\]\nBy taking expectation with respect to sample $i_k$, it follows\n\\begin{eqnarray} \\label{opt-gap} \\\\\n& & \\sum_{i=1}^{m} \\left( \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\hat{g}_i\\left(x^{\\star}\\right) \\right)  \\nonumber  \\\\\n&\\leq& \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle+\\mathbb{E}_{i_{k}}\\left[\\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle\\right]+\\mathbb{E}_{i_{k}}\\left[\\left\\langle\\widetilde{\\nabla}^{k}, x^{k+1}-x^{\\star}\\right\\rangle\\right] . \\nonumber\n\\end{eqnarray}\n\n%{\\color{red}By the convexity of $\\hat{g}_{i}(\\cdot)$,\n%\\begin{eqnarray*}\n%    \\frac{1}{\\pi_{i}}\\left(\\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\hat{g}_{i}\\left(x^{\\star}\\right)\\right)\n%    &\\leq& \\frac{1}{\\pi_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), y_{i}^{k}-x^{\\star}\\right\\rangle \\\\\n%    &=& \\frac{1}{\\pi_{i}}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle+\\left\\langle\\frac{1}{ \\pi_{i}}\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\widetilde{\\nabla}^{k}, x^{k}-x^{\\star}\\right\\rangle\\\\\n%    & &+\\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle+\\left\\langle\\widetilde{\\nabla}^{k}, x^{k+1}-x^{\\star}\\right\\rangle\n%\\end{eqnarray*}\n%where the last step uses the definition that $y_{i}^{k}=\\tau_{i} x^{k}+(1-\\tau_{i}) \\phi_{i}^{k}$.\n\n%Observe\n%\\[\n%\\mathbb{E}_{i_{k}}\\left[\\widetilde{\\nabla}^{k}\\right]=\\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right).\n%\\]\n%By taking expectation with respect to sample $i$ with probability $\\{\\pi_i\\}_{i=1}^{m}$, it follows\n%\\begin{eqnarray} \\label{opt-gap}\n%& & \\sum_{i=1}^{m} \\left( \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\hat{g}_i\\left(x^{\\star}\\right) \\right) \\nonumber \\\\\n%&\\leq& \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle+\\mathbb{E}_{i_{k}}\\left[\\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle\\right]+\\mathbb{E}_{i_{k}}\\left[\\left\\langle\\widetilde{\\nabla}^{k}, x^{k+1}-x^{\\star}\\right\\rangle\\right] .\n%\\end{eqnarray}\n%}\n\nTheorem 2.1.5 in \\cite{nesterov2018lectures} tells us that\n\\[\n\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)-\\hat{g}_{j_{k}}\\left(y_{j_{k}}^{k}\\right) \\leq\\left\\langle\\nabla \\hat{g}_{j_{k}}\\left(y_{j_{k}}^{k}\\right), \\phi_{j_{k}}^{k+1}-y_{j_{k}}^{k}\\right\\rangle+\\frac{L_{j_{k}}}{2}\\left\\|\\phi_{j_{k}}^{k+1}-y_{j_{k}}^{k}\\right\\|^{2} .\n\\]\n\nDividing the above inequality by $\\pi_{j_{k}}\\tau_{j_{k}}$ and taking expectation with respect to sample $j_k$, we obtain\n\\begin{eqnarray*}\n& & \\mathbb{E}_{j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]-\\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_i\\left(y_i^{k}\\right) \\\\\n&\\leq& \\left\\langle \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), x^{k+1}-x^{k}\\right\\rangle+\\frac{\\sum_{i=1}^{m}\\tau_i L_i}{2}\\|x^{k+1}-x^{k}\\|^2,\n\\end{eqnarray*}\nwhich is equivalent to\n\\begin{eqnarray*}\n     \\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle\n     &\\leq&\n    \\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)- \\mathbb{E}_{j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right] \\\\\n    & & +\\left\\langle \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\widetilde{\\nabla}^{k},\n    x^{k+1}-x^{k}\\right\\rangle+\\frac{\\sum_{i=1}^{m}\\tau_i L_i}{2}\\|x^{k+1}-x^{k}\\|^2.\n\\end{eqnarray*}\nTaking expectation with respect to sample $i_k$ we get\n\\begin{eqnarray*}\n     \\mathbb{E}_{i_{k}}\\left[\\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle\\right] &\\leq&\n    \\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)\n    - \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]\\\\\n& & +\\mathbb{E}_{i_{k}}\\left[\\left\\langle \\sum_{i=1}^{m}\n\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\widetilde{\\nabla}^{k}, x^{k+1}-x^{k}\\right\\rangle\\right]+\\frac{\\sum_{i=1}^{m}\\tau_i L_i}{2}\\mathbb{E}_{i_{k}}\\left[\\|x^{k+1}-x^{k}\\|^2\\right] .\n\\end{eqnarray*}\n\nBy Lemma~\\ref{lemma-3}, taking expectation we have\n\\begin{eqnarray*}\n& & \\mathbb{E}_{i_{k}}\\left[ \\left\\langle\\widetilde{\\nabla}^{k}, x^{k+1}-x^\\star\\right\\rangle \\right]  \\\\\n&\\leq& - \\mathbb{E}_{i_{k}}\\left[\\frac{1}{2 \\eta}\\left\\|x^{k+1}-x^{k}\\right\\|^{2}\\right]+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^\\star\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[ \\left\\|x^{k+1}-x^\\star\\right\\|^{2}\\right]+h(x^\\star) - \\mathbb{E}_{i_{k}}\\left[ h\\left(x^{k+1}\\right)\\right] .\n\\end{eqnarray*}\n\nApplying the above inequalities on \\eqref{opt-gap}, we have % \\eqref{eq1}, we obtain\n\\begin{eqnarray*}\n    & & \\sum_{i=1}^{m} \\left( \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\hat{g}_i \\left(x^{\\star}\\right) \\right) \\\\\n    &\\leq&\n     \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle\n    +\\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)\n    - \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right) \\right] \\\\\n   & & +\\mathbb{E}_{i_{k}}\\left[\\left\\langle \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\widetilde{\\nabla}^{k}, x^{k+1}-x^{k}\\right\\rangle\\right]+\\frac{\\sum_{i=1}^{m}\\tau_i L_i}{2}\\mathbb{E}_{i_{k}}\\left[\\|x^{k+1}-x^{k}\\|^2\\right]\\\\\n   & & -\\frac{1}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{k}\\right\\|^{2}\\right]+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right]\n    +h\\left(x^{\\star}\\right)-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right]\\\\\n   & \\leq&  \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle\n    +\\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)\n    - \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]\\\\\n   & & + \\frac{1}{2\\left(\\frac{1}{\\eta}-\\sum_{i=1}^{m}\\tau_i L_i \\right)}\\mathbb{E}_{i_{k}}\\left[\\left\\| \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\widetilde{\\nabla}^{k}\\right\\|^2\\right]+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}\\\\\n   & & -\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right]+h\\left(x^{\\star}\\right)-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right]\\\\\n   &\\leq&  \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle\n    +\\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)\n    - \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]\\\\\n   & &+ \\frac{1}{2\\left(\\frac{1}{\\eta}-\\sum_{i=1}^{m}\\tau_i L_i \\right)}\\sum_{i=1}^{m} \\frac{2L_{i}}{\\pi_{i}} \\left(\\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)-\\hat{g}_{i}\\left(y_{i}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right),\\phi_{i}^{k}-y_{i}^{k}\\right\\rangle  \\right)\\\\\n   & &+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right]+h\\left(x^{\\star}\\right)-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right]\\\\\n   &\\leq&  \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle\n    +\\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)\n    - \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]\\\\\n  &  &+ \\sum_{i=1}^{m} \\frac{1-\\tau_{i}}{\\tau_{i}} \\left(\\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)-\\hat{g}_{i}\\left(y_{i}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right),\\phi_{i}^{k}-y_{i}^{k}\\right\\rangle\\right)\\\\\n  &  &+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right]+h\\left(x^{\\star}\\right)-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right]\n\\end{eqnarray*}\nwhere the second inequality follows from the inequality $\\langle a, b\\rangle \\leq \\frac{1}{2 \\beta}\\|a\\|^{2}+\\frac{\\beta}{2}\\|b\\|^{2}$ and $\\frac{1}{\\eta}-\\sum_{i=1}^{m}\\tau_i L_i >0$, and the third inequality is due to Lemma \\ref{lemma1}, and the last inequality holds by the assumption that $\\frac{1}{\\eta}-\\sum_{i=1}^{m}\\tau_i L_i \\geq \\frac{L_i\\tau_i}{\\pi_i(1-\\tau_i)}$.\n\nCanceling out and rearranging the terms, we have\n\\begin{eqnarray*}\n    &     &  \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}} \\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]-F(x^{\\star})\\\\\n    &\\leq & \\sum_{i=1}^{m} \\frac{1-\\tau_{i}}{\\tau_{i}} \\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right]-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right].\n\\end{eqnarray*}\n\nBy the convexity of $h(\\cdot)$ and $\\phi_{j_{k}}^{k+1}=\\tau_{j_{k}} x^{k+1}+(1-\\tau_{j_{k}}) \\phi_{j_{k}}^{k}$, as long as $0\\leq \\tau_{j_{k}} \\leq 1$, we have\n\\[\nh(\\phi_{j_{k}}^{k+1})\\leq \\tau_{j_{k}} h(x^{k+1})+(1-\\tau_{j_{k}}) h(\\phi_{j_{k}}^{k}) .\n\\]\nDividing the above inequality by $m \\pi_{j_{k}} \\tau_{j_{k}}$ and taking expectation with respect to sample $j_k$ and sample $i_k$, we obtain\n\\[\n-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right] \\leq  \\sum_{i=1}^{m} \\frac{1-\\tau_{j_{k}}}{m \\tau_{j_{k}}} h\\left(\\phi_{i}^{k}\\right)- \\mathbb{E}_{i_{k}, j_{k}}\\left[\\frac{1}{m \\pi_{j_{k}} \\tau_{j_{k}}}     h\\left(\\phi_{j_{k}}^{k+1}\\right)\\right] .\n\\]\nApplying the above inequality and using $F_{i}(\\cdot):=\\hat{g}_{i}(\\cdot)+\\frac 1 m h(\\cdot)$, we further derive\n\\begin{eqnarray*}\n    &      &  \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}F_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]-F(x^{\\star})\\\\\n    &\\leq  & \\sum_{i=1}^{m} \\frac{1-\\tau_{i}}{\\tau_{i}} F_{i}\\left(\\phi_{i}^{k}\\right)+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right] .\n\\end{eqnarray*}\n\nAdding $\\mathbb{E}_{i_{k},j_{k}}\\left[\\sum_{i\\neq j_k}\\frac{1}{\\pi_i\\tau_i}F_i\\left(\\phi_i^{k}\\right)\\right]$ to both sides, we have\n\\begin{eqnarray*}\n    &     & \\mathbb{E}_{i_{k},j_{k}}\\left[\\sum_{i=1}^m\\frac{1}{\\pi_i\\tau_i}\\left(F_i\\left(\\phi_i^{k+1}\\right)-F_i(x^{\\star})\\right)\\right]\\\\\n    &\\leq & \\sum_{i=1}^{m} \\frac{1-\\pi_i\\tau_{i}}{\\pi_i\\tau_{i}} \\left(F_{i}\\left(\\phi_{i}^{k}\\right)-F_i(x^{\\star})\\right)+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}} \\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2} \\right] .\n\\end{eqnarray*}\nNoticing $\\sum_{i=1}^{m} \\left(F_{i}\\left(\\phi_{i}^{k}\\right)-F_i(x^{\\star})\\right)$ may not be positive, we need to add the following term in our Lyapunov function\n\\begin{eqnarray*}\n  & &    -\\sum_{i=1}^{m}\\frac{1}{\\pi_i\\tau_i}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k+1}-x^{\\star}\\right\\rangle \\\\\n  &=&  -\\frac{1}{\\pi_{j_{k}} \\tau_{j_{k}}}\\left\\langle\\nabla F_{j_{k}}\\left(x^{\\star}\\right), \\phi_{j_{k}}^{k+1}-x^{\\star}\\right\\rangle-\n  \\sum_{i \\neq j_k } %I \\atop \\mathfrak{l}}^{m}\n  \\frac{1}{\\pi_i\\tau_i}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k}-x^{\\star}\\right\\rangle\\\\\n  &=& -\\frac{1}{\\pi_{j_{k}}}\\left\\langle\\nabla F_{j_{k}}\\left(x^{\\star}\\right), x^{k+1}-x^{\\star}\\right\\rangle\n    +\\frac{1}{\\pi_{j_{k}}}\\left\\langle\\nabla F_{j_{k}}\\left(x^{\\star}\\right), \\phi_{j_{k}}^{k}-x^{\\star}\\right\\rangle\\\\\n  &  & - \\sum_{i =1}^{m}\\frac{1}{\\pi_i\\tau_i}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k}-x^{\\star}\\right\\rangle .\n\\end{eqnarray*}\n\nTaking expectation with respect to samples $i_k$ and $j_k$ and using $\\pi_i\\tau_i=\\lambda$, we get\n\\[\n\\mathbb{E}_{i_{k},j_{k}}\\left[-\\frac{1}{\\lambda} \\sum_{i=1}^{m}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k+1}-x^{\\star}\\right\\rangle\\right]=\n-\\frac{1-\\lambda}{\\lambda} \\sum_{i =1}^{m}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k}-x^{\\star}\\right\\rangle .\n\\]\n\nDenoting $D^k:=\\sum_{i =1}^{m}F_{i}\\left(\\phi_{i}^{k}\\right)-F\\left(x^{\\star}\\right)-\\sum_{i =1}^{m}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k}-x^{\\star}\\right\\rangle$ and $P^{k} :=\\left\\|x^{k}-x^{\\star}\\right\\|^{2}$, we have\n\\[\n\\frac{1}{\\lambda}\\mathbb{E}_{i_{k},j_{k}}\\left[D^{k+1}\\right]+\\frac{1+\\eta \\mu}{2 \\eta}\\mathbb{E}_{i_{k}}\\left[P^{k+1}\\right]\\leq \\frac{1-\\lambda}{\\lambda} D^k+\\frac{1}{2 \\eta} P^k.\n\\]\n\nFinally, it remains to choose $\\eta$ and $\\lambda$ so as to satisfy\n \\[\n 0\\leq \\tau_{i} \\leq 1, \\mbox{ and }  \\frac{1}{\\eta}-\\sum_{i=1}^{m}\\tau_i L_i \\geq \\frac{L_i\\tau_i}{\\pi_i(1-\\tau_i)},\\,\\,\\, i=1,2,...,m.\n \\]\nWe consider two cases separately.\n\n\\textbf{Case I.} If $\\sqrt{\\mu}\\leq \\frac{\\sum_{j=1}^m\\sqrt{L_j}}{m}$, then we choose\n\\[\n\\lambda := \\frac{\\sqrt{\\mu}}{4\\sum_{j=1}^m\\sqrt{L_j}}, \\quad \\eta := \\frac{1}{4\\sqrt{\\mu}\\sum_{j=1}^{m}\\sqrt{L_j} }.\n\\]\n\nRecall\n$\\pi_i=\\frac{\\sqrt{L_i}}{2\\sum_{j=1}^m\\sqrt{L_j}}+\\frac{1}{2m}$ for all $1\\le i\\le m$. Therefore,\n\\[\n\\tau_i=\\frac{\\lambda}{\\pi_i}=\\frac{\\sqrt{\\mu}}{2\\sqrt{L_i}+\\frac 2 m \\sum_{j=1}^m\\sqrt{L_j}}\\leq \\frac{\\sqrt{\\mu}}{\\frac 2 m \\sum_{j=1}^m\\sqrt{L_j}}\\leq \\frac 1 2\n\\]\nand\n\\begin{eqnarray*}\n     \\sum_{j=1}^{m}\\tau_j L_j +\\frac{L_i\\tau_i}{\\pi_i(1-\\tau_i)} &\\leq& \\sum_{j=1}^{m}L_j\\frac{\\sqrt{\\mu}}{2\\sqrt{L_j}}+\\frac{2\\lambda L_i}{ \\pi_i^2}\\\\\n    &\\leq& \\frac{\\sqrt{\\mu}\\sum_{j=1}^{m}\\sqrt{L_j} }{2}+\\frac{8(\\sum_{j=1}^{m}\\sqrt{L_j})^2\\lambda L_i}{L_i}\\\\\n    &\\leq& 3\\sqrt{\\mu}\\sum_{j=1}^{m}\\sqrt{L_j} .\n\\end{eqnarray*}\n\nSince $\\eta=\\frac{1}{4\\sqrt{\\mu}\\sum_{j=1}^{m}\\sqrt{L_j} }$, we know $\\frac{1}{\\eta}-\\sum_{j=1}^{m}\\tau_j L_j \\geq \\frac{L_i\\tau_i}{ \\pi_i(1-\\tau_i)}$ and\n\\[\n\\frac{1}{\\lambda}\\geq (1+\\eta \\mu)\\frac{1-\\lambda}{\\lambda} .\n\\]\nThus, in this case we obtain\n\\[\n\\frac{1}{\\lambda}\\mathbb{E}_{i_{k},j_{k}}\\left[D^{k+1}\\right]+\\frac{1+\\eta \\mu}{2 \\eta}\\mathbb{E}_{i_{k}}\\left[P^{k+1}\\right]\\leq (1+\\eta \\mu)^{-1}\\left( \\frac{1}{\\lambda} D^k+\\frac{1+\\eta \\mu}{2 \\eta} P^k\\right).\n\\]\n\nTelescoping the above inequalities, we have\n%\\begin{equation}\n\\begin{eqnarray*}\n& & \\mathbb{E}\\left[\\left\\|x^{K+1}-x^{\\star}\\right\\|^{2}\\right] \\\\\n&\\leq& \\left(1+\\frac{\\sqrt{\\mu}}{4\\sum_{j=1}^m\\sqrt{L_j}}\\right)^{-K} \\cdot\\left(\\frac{2\\eta}{\\lambda(1+\\eta \\mu)}\\left(F\\left(x^{1}\\right)-F\\left(x^{\\star}\\right)\\right)+\\left\\|x^{1}-x^{\\star}\\right\\|^{2}\\right) .\n\\end{eqnarray*}\n%\\end{equation}\n\n\\textbf{Case II.} If $\\sqrt{\\mu}> \\frac{\\sum_{j=1}^m\\sqrt{L_j}}{m}$, then we choose\n\\[\n\\lambda := \\frac{1}{4m}, \\quad \\eta := \\frac{1}{4\\mu m}.\n\\]\n\nIn this case, we have\n\\[\n\\tau_i=\\frac{\\lambda}{\\pi_i}=\\frac{\\frac{1}{2m}\\sum_{j=1}^m\\sqrt{L_j}}{\\sqrt{L_i}+\\frac 1 m \\sum_{j=1}^m\\sqrt{L_j}}\\leq \\frac 1 2\n\\]\nand\n\\begin{eqnarray*}\n    \\sum_{j=1}^{m}\\tau_j L_j +\\frac{L_i\\tau_i}{\\pi_i(1-\\tau_i)}&\\leq& \\frac{1}{2m}\\left(\\sum_{j=1}^m\\sqrt{L_j}\\right)^2+\\frac{2\\lambda L_i}{ \\pi_i^2}\\\\\n    &\\leq& \\frac{\\left(\\sum_{j=1}^m\\sqrt{L_j}\\right)^2}{2m}+\\frac{8\\left(\\sum_{j=1}^{m}\\sqrt{L_j}\\right)^2\\lambda L_i}{L_i}\\\\\n    &\\leq& \\frac{3\\left(\\sum_{j=1}^{m}\\sqrt{L_j}\\right)^2 }{m}\\\\\n    &\\leq& 3 \\mu m.\n\\end{eqnarray*}\n\nSince $\\eta=\\frac{1}{4\\mu m}$, we have $\\frac{1}{\\eta}-\\sum_{j=1}^{m}\\tau_j L_j \\geq \\frac{L_i\\tau_i}{\\pi_i(1-\\tau_i)}$, and\n\\[\n\\frac{1}{\\lambda}\\geq (1+\\eta \\mu)\\frac{1-\\lambda}{\\lambda} .\n\\]\n\nAs a consequence,\n\\[\n\\frac{1}{\\lambda}\\mathbb{E}_{i_{k},j_{k}}\\left[D^{k+1}\\right]+\\frac{1+\\eta \\mu}{2 \\eta}\\mathbb{E}_{i_{k}}\\left[P^{k+1}\\right]\\leq (1+\\eta \\mu)^{-1}\\left( \\frac{1}{\\lambda} D^k+\\frac{1+\\eta \\mu}{2 \\eta} P^k\\right).\n\\]\nTaking expectations and telescoping the above inequalities, we have\n\\begin{equation}\n\\mathbb{E}\\left[\\left\\|x^{K+1}-x^{\\star}\\right\\|^{2}\\right] \\leq\\left(1+\\frac{1}{4m}\\right)^{-K} \\cdot\\left(\\frac{2\\eta}{\\lambda(1+\\eta \\mu)}\\left(F\\left(x^{1}\\right)-F\\left(x^{\\star}\\right)\\right)+\\left\\|x^{1}-x^{\\star}\\right\\|^{2}\\right) .\n\\end{equation}\nSummarizing Case I and Case II, the theorem is proven.\n\\end{proof}","sha256_utf8":"dadc8058ab57d41116bd37086e631bfd5a01d74b12c8d6c7f8034ea8fb30868d"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"有限和约束优化模型","source_units":[{"start":4029,"end":4124,"start_line":91,"end_line":93,"raw_tex":"\\begin{equation}\n    \\min_{x\\in \\Gamma} F(x)=\\sum_{i=1}^{m} g_i(x),  \\label{pro}\n\\end{equation}","sha256_utf8":"c4eae8234f819a60fa480a6d9dded0ab55446d11564035ff4abb20b754f8264d"},{"start":4125,"end":4628,"start_line":94,"end_line":94,"raw_tex":"where $g_i(x)$'s are smooth and convex functions, and $\\Gamma$ is a closed convex set. When $m$ is large, computing the exact gradient of $\\sum_{i=1}^{m} g_i(x)$ becomes costly, and it gives rise to the so-called stochastic gradient descent (SGD) method, which only computes a single gradient at random or a mini-batch of randomly selected gradients at each iteration. However, SGD converges with a sublinear rate of $1/\\epsilon$ even if all the functions $g_i(\\cdot)$'s are strongly convex and smooth.\n","sha256_utf8":"4a6535054fdb57b0a745171694c08ef44ac37842d137c9044ec105927c0c91fd"}]},{"id":"S002","semantic_type":"definition","kind":"definition","title":"梯度复杂度定义","source_units":[{"start":5070,"end":5282,"start_line":97,"end_line":98,"raw_tex":"In this paper, we shall use the terminology of {\\it gradient complexity}\\/ to refer to the amount of times that the gradients\nof functions $g_i$'s need be computed before reaching an $\\epsilon$ optimal solution.\n","sha256_utf8":"e95723317c7e457ae550d43194a1abc475ade55db01cb6ff558fc8d37c3f227f"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"强凸性定义","source_units":[{"start":11959,"end":12156,"start_line":161,"end_line":164,"raw_tex":"a function $g$ is called $\\mu$-strongly convex if\n\\begin{equation}\n    g(y)\\geq g(x)+\\left\\langle\\nabla g(x),y-x\\right\\rangle+\\frac{\\mu}{2}\\|y-x\\|^2, \\quad \\forall x,y\\in \\mathcal{X}\n\\end{equation}","sha256_utf8":"2f7030041d30d0ba34b315362b2472163e47752e3d6c41cc1ee54ffc5a7238f4"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"梯度光滑性定义","source_units":[{"start":12175,"end":12360,"start_line":166,"end_line":171,"raw_tex":"and\n%\\begin{definition}\na function $g$ is called $L$-Lipschitz smooth if\n\\begin{equation}\n    \\|\\nabla g(x)-\\nabla g(y)\\|\\leq L\\|x-y\\|, \\quad \\forall x,y\\in \\mathcal{X} .\n\\end{equation}","sha256_utf8":"881d6e3dae5f2995711ffe184a8ba5b7217b69a8968360fb7a6020513538a227"}]},{"id":"S005","semantic_type":"assumption","kind":"assumption","title":"分量强凸与光滑假设","source_units":[{"start":12425,"end":12875,"start_line":174,"end_line":177,"raw_tex":"\\begin{assumption}\nIn Model~\\eqref{pro}, $g_i(x)$ is assumed to be $L_i$-smooth and $\\mu_i$-strongly convex, $i=1,2,...,m$.\n%{\\color{red} Define $\\kappa_i= L_i /(m\\mu)+1$ as the condition number for the $i$-th component, $i=1,2,...,m$. (This may be hard to sell! Let us change the model to simply assuming $g_i$ to be $L_i$-gradient smooth and $\\mu_i$-strongly convex, and there is no $h$, and there is a constraint $x\\in \\Gamma$.) }\n\\end{assumption}","sha256_utf8":"87987dbb5f9ebf8ff7aa96c4fed239841bb6d75f100c3874fb379c82c966e5cb"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"分离强凸二次项","source_units":[{"start":20646,"end":20727,"start_line":295,"end_line":298,"raw_tex":"Recall the problem\n\\[\nF(x)=\\sum_{i=1}^m g_i(x)=\\sum_{i=1}^m \\hat{g}_i(x)+h(x),\n\\]","sha256_utf8":"91602698566492534fb30c5a197ceb6080bd8a316e7196d22a3668481bd36aea"},{"start":20728,"end":20854,"start_line":299,"end_line":299,"raw_tex":"where we let $\\hat{g}_i(x) := g_i(x)-\\frac{\\mu_i}{2}\\|x\\|^2$, $h(x) := \\frac{\\mu}{2}\\|x\\|^2$, where $\\mu=\\sum_{i=1}^m \\mu_i$.\n","sha256_utf8":"f1548d65e9f7725162bba145d5d07174574f7005296a7de69c33566f4e0f6592"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"梯度估计与抽样概率","source_units":[{"start":20854,"end":21109,"start_line":300,"end_line":303,"raw_tex":"We use the following estimation of the gradient of the first term,\n\\[\n\\widetilde{\\nabla}^{k}=\\frac{1}{\\pi_{i_{k}}}\\left(\\nabla \\hat{g}_{i_k}(y_{i_k}^k)-\\nabla \\hat{g}_{i_k}(\\phi_{i_k}^k)\\right)+  \\sum_{i=1}^m\\nabla \\hat{g}_{i}(\\phi_{i}^k){\\color{red},}\n\\]","sha256_utf8":"c346789883642b0d215d7966e3393ce86b3a048639f937739705979511c4e628"},{"start":21110,"end":21263,"start_line":304,"end_line":307,"raw_tex":"where $i_k$ is sampled from $\\{1,2,\\cdots,m\\}$ with the probability\n\\[\n\\pi_i=\\frac{\\sqrt{L_i}}{2\\sum_{j=1}^m\\sqrt{L_j}}+\\frac{1}{2m},\\,\\, i=1,2,...,m.\n\\]","sha256_utf8":"7671ce1817e0488ea84443df6182995232d3e61f01e49e89c33e99792a178c4b"},{"start":21265,"end":21481,"start_line":309,"end_line":309,"raw_tex":"Observe the gradient estimator $\\widetilde{\\nabla}^{k}$ is biased with expectation $\\sum_{i=1}^m\\nabla \\hat{g}_{i}(y_{i}^k)$, if we extend the definition to all $y_{i}^{k}:=\\tau_{i} x^{k}+(1-\\tau_{i}) \\phi_{i}^{k}$.\n","sha256_utf8":"4c2e258bd4d70cca8eb78da82d76fa35ee6a8f1b54676372f52e80803efc654a"}]},{"id":"S008","semantic_type":"algorithm","kind":"algorithm","title":"广义SSNM算法","source_units":[{"start":22267,"end":23297,"start_line":316,"end_line":333,"raw_tex":"\\begin{algorithm}[h]\n\\caption{Generalized SSNM }\n\\label{Algorithm1}\n\\begin{algorithmic}\n\\STATE{$\\phi_1=\\phi_2=\\cdots=\\phi_m=x^1$}\n\\STATE{$\\pi_i=\\frac{\\sqrt{L_i}}{2\\sum_{j=1}^m\\sqrt{L_j}}+\\frac{1}{2m}$}\n\\STATE{$\\tau_i=\\lambda/\\pi_i$}\n\\FOR{$k=1,2,..., K$}\n\\STATE{Sample $i_k$ from $\\{1,2,\\cdots,m\\}$ with probability $\\{\\pi_i\\}_{i=1}^{m}$}\n\\STATE{$y_{i_{k}}^{k}=\\tau_{i_{k}} x^{k}+(1-\\tau_{i_{k}}) \\phi_{i_{k}}^{k}$}\n\\STATE{$\\widetilde{\\nabla}^{k}=\\frac{1}{\\pi_{i_{k}}}\\left(\\nabla \\hat{g}_{i_k}(y_{i_k}^k)-\\nabla \\hat{g}_{i_k}(\\phi_{i_k}^k)\\right)+  \\sum_{i=1}^m\\nabla \\hat{g}_{i}(\\phi_{i}^k)$}\n\\STATE{Perform a proximal/projection step\\\\ $x^{k+1}=\\arg \\min _{x\\in \\Gamma}\\left\\{h(x)+\\left\\langle\\widetilde{\\nabla}^{k}, x\\right\\rangle+\\frac{1}{2 \\eta}\\left\\|x^{k}-x\\right\\|^{2}\\right\\}$}\n\\STATE{Sample $j_k$ from $\\{1,2,\\cdots,m\\}$ with probability $\\{\\pi_i\\}_{i=1}^{m}$}\n\\STATE{$\\phi_{j_{k}}^{k+1}=\\tau_{j_{k}} x^{k+1}+(1-\\tau_{j_{k}}) \\phi_{j_{k}}^{k}$}\n\\ENDFOR\n\\STATE \\textbf{return} $x^{k+1}$ \n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"231031e3ab0f0661260ccc2ec22c526b701477d81ee3ccb50ea0e5dc6d8e26b7"}]},{"id":"S009","semantic_type":"theorem","kind":"theorem","title":"广义SSNM梯度复杂度","source_units":[{"start":23426,"end":23793,"start_line":341,"end_line":345,"raw_tex":"\\begin{theorem} \\label{thmssnm}\n%By carefully choosing $\\eta$ and $\\lambda$, we have an\nGeneralized SSNM (Algorithm~\\ref{Algorithm1}) requires no more than\n$$O\\left(\\left(m+\\frac{\\sum_{j=1}^m\\sqrt{L_j}}{\\sqrt{\\sum_{i=1}^m \\mu_i}}\\right)\\log(1/\\epsilon)\\right)$$ gradient computations to reach an $\\epsilon$-solution for Model~\\eqref{pro} in expectation.\n\\end{theorem}","sha256_utf8":"83dd72b1241e50b6f40859c321b1ba09eaf528aa95e1e0a12480e1ab457582fb"}]},{"id":"S010","semantic_type":"proof","kind":"proof","title":"梯度复杂度完整证明","source_units":[{"start":28952,"end":46761,"start_line":417,"end_line":639,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{thmssnm}]\nFirst, we assume that we can choose $\\eta>0$ and $\\lambda>0$ such that $0\\leq \\tau_{i} \\leq 1$, $\\frac{1}{\\eta}-\\frac{\\sum_{i=1}^{m}\\tau_i L_i}{m} \\geq \\frac{L_i\\tau_i}{m \\pi_i(1-\\tau_i)}$, $\\forall i=1,2,...,m$ hold. Later in the proof, we will show how to choose $\\eta>0$ and $\\lambda>0$ to satisfy these inequalities.\n\nBy the convexity of $\\hat{g}_{i_{k}}(\\cdot)$,\n\\begin{eqnarray*}\n & &    \\frac{1}{\\pi_{i_{k}}}\\left(\\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right)-\\hat{g}_{i_{k}}\\left(x^{\\star}\\right)\\right)\n    \\leq \\frac{1}{\\pi_{i_{k}}}\\left\\langle\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right), y_{i_{k}}^{k}-x^{\\star}\\right\\rangle \\\\\n    &=& \\frac{1}{\\pi_{i_{k}}}\\frac{1-\\tau_{i_{k}}}{\\tau_{i_{k}}}\\left\\langle\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right), \\phi_{i_{k}}^{k}-y_{i_{k}}^{k}\\right\\rangle+\\left\\langle\\frac{1}{ \\pi_{i_{k}}}\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right)-\\widetilde{\\nabla}^{k}, x^{k}-x^{\\star}\\right\\rangle\\\\\n    & &+\\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle+\\left\\langle\\widetilde{\\nabla}^{k}, x^{k+1}-x^{\\star}\\right\\rangle\n\\end{eqnarray*}\nwhere the last step uses the definition that $y_{i}=\\tau_{i} x^{k}+(1-\\tau_{i}) \\phi_{i}^{k}$, for $i=1,2,...,m$.\n%$y_{i_{k}}^{k}=\\tau_{i_{k}} x^{k}+(1-\\tau_{i_{k}}) \\phi_{i_{k}}^{k}$. {\\color{red} Replace to $y_{i}=\\tau_{i} x^{k}+(1-\\tau_{i}) \\phi_{i}^{k}, \\forall i=1,2,...,m$.}\n\nObserve\n\\[\n\\mathbb{E}_{i_{k}}\\left[\\widetilde{\\nabla}^{k}-\\frac{1}{\\pi_{i_{k}}}\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right)\\right]=\\mathbb{E}_{i_{k}}\\left[\\widetilde{\\nabla}^{k}\\right]-\\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)=0.\n\\]\nBy taking expectation with respect to sample $i_k$, it follows\n\\begin{eqnarray} \\label{opt-gap} \\\\\n& & \\sum_{i=1}^{m} \\left( \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\hat{g}_i\\left(x^{\\star}\\right) \\right)  \\nonumber  \\\\\n&\\leq& \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle+\\mathbb{E}_{i_{k}}\\left[\\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle\\right]+\\mathbb{E}_{i_{k}}\\left[\\left\\langle\\widetilde{\\nabla}^{k}, x^{k+1}-x^{\\star}\\right\\rangle\\right] . \\nonumber\n\\end{eqnarray}\n\n%{\\color{red}By the convexity of $\\hat{g}_{i}(\\cdot)$,\n%\\begin{eqnarray*}\n%    \\frac{1}{\\pi_{i}}\\left(\\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\hat{g}_{i}\\left(x^{\\star}\\right)\\right)\n%    &\\leq& \\frac{1}{\\pi_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), y_{i}^{k}-x^{\\star}\\right\\rangle \\\\\n%    &=& \\frac{1}{\\pi_{i}}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle+\\left\\langle\\frac{1}{ \\pi_{i}}\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\widetilde{\\nabla}^{k}, x^{k}-x^{\\star}\\right\\rangle\\\\\n%    & &+\\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle+\\left\\langle\\widetilde{\\nabla}^{k}, x^{k+1}-x^{\\star}\\right\\rangle\n%\\end{eqnarray*}\n%where the last step uses the definition that $y_{i}^{k}=\\tau_{i} x^{k}+(1-\\tau_{i}) \\phi_{i}^{k}$.\n\n%Observe\n%\\[\n%\\mathbb{E}_{i_{k}}\\left[\\widetilde{\\nabla}^{k}\\right]=\\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right).\n%\\]\n%By taking expectation with respect to sample $i$ with probability $\\{\\pi_i\\}_{i=1}^{m}$, it follows\n%\\begin{eqnarray} \\label{opt-gap}\n%& & \\sum_{i=1}^{m} \\left( \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\hat{g}_i\\left(x^{\\star}\\right) \\right) \\nonumber \\\\\n%&\\leq& \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle+\\mathbb{E}_{i_{k}}\\left[\\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle\\right]+\\mathbb{E}_{i_{k}}\\left[\\left\\langle\\widetilde{\\nabla}^{k}, x^{k+1}-x^{\\star}\\right\\rangle\\right] .\n%\\end{eqnarray}\n%}\n\nTheorem 2.1.5 in \\cite{nesterov2018lectures} tells us that\n\\[\n\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)-\\hat{g}_{j_{k}}\\left(y_{j_{k}}^{k}\\right) \\leq\\left\\langle\\nabla \\hat{g}_{j_{k}}\\left(y_{j_{k}}^{k}\\right), \\phi_{j_{k}}^{k+1}-y_{j_{k}}^{k}\\right\\rangle+\\frac{L_{j_{k}}}{2}\\left\\|\\phi_{j_{k}}^{k+1}-y_{j_{k}}^{k}\\right\\|^{2} .\n\\]\n\nDividing the above inequality by $\\pi_{j_{k}}\\tau_{j_{k}}$ and taking expectation with respect to sample $j_k$, we obtain\n\\begin{eqnarray*}\n& & \\mathbb{E}_{j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]-\\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_i\\left(y_i^{k}\\right) \\\\\n&\\leq& \\left\\langle \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), x^{k+1}-x^{k}\\right\\rangle+\\frac{\\sum_{i=1}^{m}\\tau_i L_i}{2}\\|x^{k+1}-x^{k}\\|^2,\n\\end{eqnarray*}\nwhich is equivalent to\n\\begin{eqnarray*}\n     \\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle\n     &\\leq&\n    \\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)- \\mathbb{E}_{j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right] \\\\\n    & & +\\left\\langle \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\widetilde{\\nabla}^{k},\n    x^{k+1}-x^{k}\\right\\rangle+\\frac{\\sum_{i=1}^{m}\\tau_i L_i}{2}\\|x^{k+1}-x^{k}\\|^2.\n\\end{eqnarray*}\nTaking expectation with respect to sample $i_k$ we get\n\\begin{eqnarray*}\n     \\mathbb{E}_{i_{k}}\\left[\\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle\\right] &\\leq&\n    \\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)\n    - \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]\\\\\n& & +\\mathbb{E}_{i_{k}}\\left[\\left\\langle \\sum_{i=1}^{m}\n\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\widetilde{\\nabla}^{k}, x^{k+1}-x^{k}\\right\\rangle\\right]+\\frac{\\sum_{i=1}^{m}\\tau_i L_i}{2}\\mathbb{E}_{i_{k}}\\left[\\|x^{k+1}-x^{k}\\|^2\\right] .\n\\end{eqnarray*}\n\nBy Lemma~\\ref{lemma-3}, taking expectation we have\n\\begin{eqnarray*}\n& & \\mathbb{E}_{i_{k}}\\left[ \\left\\langle\\widetilde{\\nabla}^{k}, x^{k+1}-x^\\star\\right\\rangle \\right]  \\\\\n&\\leq& - \\mathbb{E}_{i_{k}}\\left[\\frac{1}{2 \\eta}\\left\\|x^{k+1}-x^{k}\\right\\|^{2}\\right]+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^\\star\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[ \\left\\|x^{k+1}-x^\\star\\right\\|^{2}\\right]+h(x^\\star) - \\mathbb{E}_{i_{k}}\\left[ h\\left(x^{k+1}\\right)\\right] .\n\\end{eqnarray*}\n\nApplying the above inequalities on \\eqref{opt-gap}, we have % \\eqref{eq1}, we obtain\n\\begin{eqnarray*}\n    & & \\sum_{i=1}^{m} \\left( \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\hat{g}_i \\left(x^{\\star}\\right) \\right) \\\\\n    &\\leq&\n     \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle\n    +\\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)\n    - \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right) \\right] \\\\\n   & & +\\mathbb{E}_{i_{k}}\\left[\\left\\langle \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\widetilde{\\nabla}^{k}, x^{k+1}-x^{k}\\right\\rangle\\right]+\\frac{\\sum_{i=1}^{m}\\tau_i L_i}{2}\\mathbb{E}_{i_{k}}\\left[\\|x^{k+1}-x^{k}\\|^2\\right]\\\\\n   & & -\\frac{1}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{k}\\right\\|^{2}\\right]+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right]\n    +h\\left(x^{\\star}\\right)-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right]\\\\\n   & \\leq&  \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle\n    +\\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)\n    - \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]\\\\\n   & & + \\frac{1}{2\\left(\\frac{1}{\\eta}-\\sum_{i=1}^{m}\\tau_i L_i \\right)}\\mathbb{E}_{i_{k}}\\left[\\left\\| \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\widetilde{\\nabla}^{k}\\right\\|^2\\right]+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}\\\\\n   & & -\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right]+h\\left(x^{\\star}\\right)-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right]\\\\\n   &\\leq&  \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle\n    +\\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)\n    - \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]\\\\\n   & &+ \\frac{1}{2\\left(\\frac{1}{\\eta}-\\sum_{i=1}^{m}\\tau_i L_i \\right)}\\sum_{i=1}^{m} \\frac{2L_{i}}{\\pi_{i}} \\left(\\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)-\\hat{g}_{i}\\left(y_{i}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right),\\phi_{i}^{k}-y_{i}^{k}\\right\\rangle  \\right)\\\\\n   & &+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right]+h\\left(x^{\\star}\\right)-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right]\\\\\n   &\\leq&  \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle\n    +\\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)\n    - \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]\\\\\n  &  &+ \\sum_{i=1}^{m} \\frac{1-\\tau_{i}}{\\tau_{i}} \\left(\\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)-\\hat{g}_{i}\\left(y_{i}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right),\\phi_{i}^{k}-y_{i}^{k}\\right\\rangle\\right)\\\\\n  &  &+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right]+h\\left(x^{\\star}\\right)-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right]\n\\end{eqnarray*}\nwhere the second inequality follows from the inequality $\\langle a, b\\rangle \\leq \\frac{1}{2 \\beta}\\|a\\|^{2}+\\frac{\\beta}{2}\\|b\\|^{2}$ and $\\frac{1}{\\eta}-\\sum_{i=1}^{m}\\tau_i L_i >0$, and the third inequality is due to Lemma \\ref{lemma1}, and the last inequality holds by the assumption that $\\frac{1}{\\eta}-\\sum_{i=1}^{m}\\tau_i L_i \\geq \\frac{L_i\\tau_i}{\\pi_i(1-\\tau_i)}$.\n\nCanceling out and rearranging the terms, we have\n\\begin{eqnarray*}\n    &     &  \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}} \\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]-F(x^{\\star})\\\\\n    &\\leq & \\sum_{i=1}^{m} \\frac{1-\\tau_{i}}{\\tau_{i}} \\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right]-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right].\n\\end{eqnarray*}\n\nBy the convexity of $h(\\cdot)$ and $\\phi_{j_{k}}^{k+1}=\\tau_{j_{k}} x^{k+1}+(1-\\tau_{j_{k}}) \\phi_{j_{k}}^{k}$, as long as $0\\leq \\tau_{j_{k}} \\leq 1$, we have\n\\[\nh(\\phi_{j_{k}}^{k+1})\\leq \\tau_{j_{k}} h(x^{k+1})+(1-\\tau_{j_{k}}) h(\\phi_{j_{k}}^{k}) .\n\\]\nDividing the above inequality by $m \\pi_{j_{k}} \\tau_{j_{k}}$ and taking expectation with respect to sample $j_k$ and sample $i_k$, we obtain\n\\[\n-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right] \\leq  \\sum_{i=1}^{m} \\frac{1-\\tau_{j_{k}}}{m \\tau_{j_{k}}} h\\left(\\phi_{i}^{k}\\right)- \\mathbb{E}_{i_{k}, j_{k}}\\left[\\frac{1}{m \\pi_{j_{k}} \\tau_{j_{k}}}     h\\left(\\phi_{j_{k}}^{k+1}\\right)\\right] .\n\\]\nApplying the above inequality and using $F_{i}(\\cdot):=\\hat{g}_{i}(\\cdot)+\\frac 1 m h(\\cdot)$, we further derive\n\\begin{eqnarray*}\n    &      &  \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}F_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]-F(x^{\\star})\\\\\n    &\\leq  & \\sum_{i=1}^{m} \\frac{1-\\tau_{i}}{\\tau_{i}} F_{i}\\left(\\phi_{i}^{k}\\right)+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right] .\n\\end{eqnarray*}\n\nAdding $\\mathbb{E}_{i_{k},j_{k}}\\left[\\sum_{i\\neq j_k}\\frac{1}{\\pi_i\\tau_i}F_i\\left(\\phi_i^{k}\\right)\\right]$ to both sides, we have\n\\begin{eqnarray*}\n    &     & \\mathbb{E}_{i_{k},j_{k}}\\left[\\sum_{i=1}^m\\frac{1}{\\pi_i\\tau_i}\\left(F_i\\left(\\phi_i^{k+1}\\right)-F_i(x^{\\star})\\right)\\right]\\\\\n    &\\leq & \\sum_{i=1}^{m} \\frac{1-\\pi_i\\tau_{i}}{\\pi_i\\tau_{i}} \\left(F_{i}\\left(\\phi_{i}^{k}\\right)-F_i(x^{\\star})\\right)+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}} \\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2} \\right] .\n\\end{eqnarray*}\nNoticing $\\sum_{i=1}^{m} \\left(F_{i}\\left(\\phi_{i}^{k}\\right)-F_i(x^{\\star})\\right)$ may not be positive, we need to add the following term in our Lyapunov function\n\\begin{eqnarray*}\n  & &    -\\sum_{i=1}^{m}\\frac{1}{\\pi_i\\tau_i}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k+1}-x^{\\star}\\right\\rangle \\\\\n  &=&  -\\frac{1}{\\pi_{j_{k}} \\tau_{j_{k}}}\\left\\langle\\nabla F_{j_{k}}\\left(x^{\\star}\\right), \\phi_{j_{k}}^{k+1}-x^{\\star}\\right\\rangle-\n  \\sum_{i \\neq j_k } %I \\atop \\mathfrak{l}}^{m}\n  \\frac{1}{\\pi_i\\tau_i}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k}-x^{\\star}\\right\\rangle\\\\\n  &=& -\\frac{1}{\\pi_{j_{k}}}\\left\\langle\\nabla F_{j_{k}}\\left(x^{\\star}\\right), x^{k+1}-x^{\\star}\\right\\rangle\n    +\\frac{1}{\\pi_{j_{k}}}\\left\\langle\\nabla F_{j_{k}}\\left(x^{\\star}\\right), \\phi_{j_{k}}^{k}-x^{\\star}\\right\\rangle\\\\\n  &  & - \\sum_{i =1}^{m}\\frac{1}{\\pi_i\\tau_i}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k}-x^{\\star}\\right\\rangle .\n\\end{eqnarray*}\n\nTaking expectation with respect to samples $i_k$ and $j_k$ and using $\\pi_i\\tau_i=\\lambda$, we get\n\\[\n\\mathbb{E}_{i_{k},j_{k}}\\left[-\\frac{1}{\\lambda} \\sum_{i=1}^{m}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k+1}-x^{\\star}\\right\\rangle\\right]=\n-\\frac{1-\\lambda}{\\lambda} \\sum_{i =1}^{m}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k}-x^{\\star}\\right\\rangle .\n\\]\n\nDenoting $D^k:=\\sum_{i =1}^{m}F_{i}\\left(\\phi_{i}^{k}\\right)-F\\left(x^{\\star}\\right)-\\sum_{i =1}^{m}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k}-x^{\\star}\\right\\rangle$ and $P^{k} :=\\left\\|x^{k}-x^{\\star}\\right\\|^{2}$, we have\n\\[\n\\frac{1}{\\lambda}\\mathbb{E}_{i_{k},j_{k}}\\left[D^{k+1}\\right]+\\frac{1+\\eta \\mu}{2 \\eta}\\mathbb{E}_{i_{k}}\\left[P^{k+1}\\right]\\leq \\frac{1-\\lambda}{\\lambda} D^k+\\frac{1}{2 \\eta} P^k.\n\\]\n\nFinally, it remains to choose $\\eta$ and $\\lambda$ so as to satisfy\n \\[\n 0\\leq \\tau_{i} \\leq 1, \\mbox{ and }  \\frac{1}{\\eta}-\\sum_{i=1}^{m}\\tau_i L_i \\geq \\frac{L_i\\tau_i}{\\pi_i(1-\\tau_i)},\\,\\,\\, i=1,2,...,m.\n \\]\nWe consider two cases separately.\n\n\\textbf{Case I.} If $\\sqrt{\\mu}\\leq \\frac{\\sum_{j=1}^m\\sqrt{L_j}}{m}$, then we choose\n\\[\n\\lambda := \\frac{\\sqrt{\\mu}}{4\\sum_{j=1}^m\\sqrt{L_j}}, \\quad \\eta := \\frac{1}{4\\sqrt{\\mu}\\sum_{j=1}^{m}\\sqrt{L_j} }.\n\\]\n\nRecall\n$\\pi_i=\\frac{\\sqrt{L_i}}{2\\sum_{j=1}^m\\sqrt{L_j}}+\\frac{1}{2m}$ for all $1\\le i\\le m$. Therefore,\n\\[\n\\tau_i=\\frac{\\lambda}{\\pi_i}=\\frac{\\sqrt{\\mu}}{2\\sqrt{L_i}+\\frac 2 m \\sum_{j=1}^m\\sqrt{L_j}}\\leq \\frac{\\sqrt{\\mu}}{\\frac 2 m \\sum_{j=1}^m\\sqrt{L_j}}\\leq \\frac 1 2\n\\]\nand\n\\begin{eqnarray*}\n     \\sum_{j=1}^{m}\\tau_j L_j +\\frac{L_i\\tau_i}{\\pi_i(1-\\tau_i)} &\\leq& \\sum_{j=1}^{m}L_j\\frac{\\sqrt{\\mu}}{2\\sqrt{L_j}}+\\frac{2\\lambda L_i}{ \\pi_i^2}\\\\\n    &\\leq& \\frac{\\sqrt{\\mu}\\sum_{j=1}^{m}\\sqrt{L_j} }{2}+\\frac{8(\\sum_{j=1}^{m}\\sqrt{L_j})^2\\lambda L_i}{L_i}\\\\\n    &\\leq& 3\\sqrt{\\mu}\\sum_{j=1}^{m}\\sqrt{L_j} .\n\\end{eqnarray*}\n\nSince $\\eta=\\frac{1}{4\\sqrt{\\mu}\\sum_{j=1}^{m}\\sqrt{L_j} }$, we know $\\frac{1}{\\eta}-\\sum_{j=1}^{m}\\tau_j L_j \\geq \\frac{L_i\\tau_i}{ \\pi_i(1-\\tau_i)}$ and\n\\[\n\\frac{1}{\\lambda}\\geq (1+\\eta \\mu)\\frac{1-\\lambda}{\\lambda} .\n\\]\nThus, in this case we obtain\n\\[\n\\frac{1}{\\lambda}\\mathbb{E}_{i_{k},j_{k}}\\left[D^{k+1}\\right]+\\frac{1+\\eta \\mu}{2 \\eta}\\mathbb{E}_{i_{k}}\\left[P^{k+1}\\right]\\leq (1+\\eta \\mu)^{-1}\\left( \\frac{1}{\\lambda} D^k+\\frac{1+\\eta \\mu}{2 \\eta} P^k\\right).\n\\]\n\nTelescoping the above inequalities, we have\n%\\begin{equation}\n\\begin{eqnarray*}\n& & \\mathbb{E}\\left[\\left\\|x^{K+1}-x^{\\star}\\right\\|^{2}\\right] \\\\\n&\\leq& \\left(1+\\frac{\\sqrt{\\mu}}{4\\sum_{j=1}^m\\sqrt{L_j}}\\right)^{-K} \\cdot\\left(\\frac{2\\eta}{\\lambda(1+\\eta \\mu)}\\left(F\\left(x^{1}\\right)-F\\left(x^{\\star}\\right)\\right)+\\left\\|x^{1}-x^{\\star}\\right\\|^{2}\\right) .\n\\end{eqnarray*}\n%\\end{equation}\n\n\\textbf{Case II.} If $\\sqrt{\\mu}> \\frac{\\sum_{j=1}^m\\sqrt{L_j}}{m}$, then we choose\n\\[\n\\lambda := \\frac{1}{4m}, \\quad \\eta := \\frac{1}{4\\mu m}.\n\\]\n\nIn this case, we have\n\\[\n\\tau_i=\\frac{\\lambda}{\\pi_i}=\\frac{\\frac{1}{2m}\\sum_{j=1}^m\\sqrt{L_j}}{\\sqrt{L_i}+\\frac 1 m \\sum_{j=1}^m\\sqrt{L_j}}\\leq \\frac 1 2\n\\]\nand\n\\begin{eqnarray*}\n    \\sum_{j=1}^{m}\\tau_j L_j +\\frac{L_i\\tau_i}{\\pi_i(1-\\tau_i)}&\\leq& \\frac{1}{2m}\\left(\\sum_{j=1}^m\\sqrt{L_j}\\right)^2+\\frac{2\\lambda L_i}{ \\pi_i^2}\\\\\n    &\\leq& \\frac{\\left(\\sum_{j=1}^m\\sqrt{L_j}\\right)^2}{2m}+\\frac{8\\left(\\sum_{j=1}^{m}\\sqrt{L_j}\\right)^2\\lambda L_i}{L_i}\\\\\n    &\\leq& \\frac{3\\left(\\sum_{j=1}^{m}\\sqrt{L_j}\\right)^2 }{m}\\\\\n    &\\leq& 3 \\mu m.\n\\end{eqnarray*}\n\nSince $\\eta=\\frac{1}{4\\mu m}$, we have $\\frac{1}{\\eta}-\\sum_{j=1}^{m}\\tau_j L_j \\geq \\frac{L_i\\tau_i}{\\pi_i(1-\\tau_i)}$, and\n\\[\n\\frac{1}{\\lambda}\\geq (1+\\eta \\mu)\\frac{1-\\lambda}{\\lambda} .\n\\]\n\nAs a consequence,\n\\[\n\\frac{1}{\\lambda}\\mathbb{E}_{i_{k},j_{k}}\\left[D^{k+1}\\right]+\\frac{1+\\eta \\mu}{2 \\eta}\\mathbb{E}_{i_{k}}\\left[P^{k+1}\\right]\\leq (1+\\eta \\mu)^{-1}\\left( \\frac{1}{\\lambda} D^k+\\frac{1+\\eta \\mu}{2 \\eta} P^k\\right).\n\\]\nTaking expectations and telescoping the above inequalities, we have\n\\begin{equation}\n\\mathbb{E}\\left[\\left\\|x^{K+1}-x^{\\star}\\right\\|^{2}\\right] \\leq\\left(1+\\frac{1}{4m}\\right)^{-K} \\cdot\\left(\\frac{2\\eta}{\\lambda(1+\\eta \\mu)}\\left(F\\left(x^{1}\\right)-F\\left(x^{\\star}\\right)\\right)+\\left\\|x^{1}-x^{\\star}\\right\\|^{2}\\right) .\n\\end{equation}\nSummarizing Case I and Case II, the theorem is proven.\n\\end{proof}","sha256_utf8":"dadc8058ab57d41116bd37086e631bfd5a01d74b12c8d6c7f8034ea8fb30868d"}]},{"id":"S011","semantic_type":"lemma","kind":"lemma","title":"梯度估计方差上界","source_units":[{"start":25143,"end":25562,"start_line":365,"end_line":370,"raw_tex":"\\begin{lemma}{(variance upper bound)}\\label{lemma1}\n\\[\n\\mathbb{E}_{i_{k}}\\left[\\left\\|\\widetilde{\\nabla}^{k}- \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)\\right\\|^{2}\\right]\n\\leq \\sum_{i=1}^{m} \\frac{2L_{i}}{\\pi_{i}} \\left(\\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)-\\hat{g}_{i}\\left(y_{i}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right),\\phi_{i}^{k}-y_{i}^{k}\\right\\rangle  \\right).\n\\]\n\\end{lemma}","sha256_utf8":"25a0eeb012de56460f9aabd473042b7121e7ae1fa399b44c85d2b0e1f7361322"}]},{"id":"S012","semantic_type":"proof","kind":"proof","title":"方差上界完整证明","source_units":[{"start":25564,"end":27693,"start_line":372,"end_line":391,"raw_tex":"\\begin{proof}\nSince each $\\hat{g}_i(x)$ is convex and $L_i$ smooth, Theorem 2.1.5 in \\cite{nesterov2018lectures} stipulates\n\\begin{equation}\n\\left\\|\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\nabla \\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)\\right\\|^{2}\\leq 2L_{i}\\left(\\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)-\\hat{g}_{i}\\left(y_{i}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right),\\phi_{i}^{k}-y_{i}^{k}\\right\\rangle  \\right). \\label{squineq}\n\\end{equation}\nObserving\n\\begin{equation} \\label{ex-ineq}\n\\mathbb{E}\\|\\zeta-\\mathbb{E} \\zeta\\|^{2}=\\mathbb{E}\\|\\zeta\\|^{2}-\\|\\mathbb{E} \\zeta\\|^{2},\n\\end{equation}\n%and by using the above inequality, we have\nwe have\n\\begin{eqnarray*}\n    & & \\mathbb{E}_{i_{k}}\\left[\\left\\|\\widetilde{\\nabla}^{k}- \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)\\right\\|^{2}\\right] \\\\\n    &=& \\mathbb{E}_{i_{k}}\\left[\\left\\|\\frac{1}{\\pi_{i_{k}}}\\left(\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right)-\\nabla \\hat{g}_{i_{k}}\\left(\\phi_{i_{k}}^{k}\\right)\\right)- \\sum_{i=1}^{m}\\left(\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\nabla \\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)\\right) \\right\\|^{2}\\right] \\\\\n    & \\overset{\\eqref{ex-ineq}}{\\leq} & \\mathbb{E}_{i_{k}}\\left[\\left\\|\\frac{1}{\\pi_{i_{k}}}\\left(\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right)-\\nabla \\hat{g}_{i_{k}}\\left(\\phi_{i_{k}}^{k}\\right)\\right) \\right\\|^{2}\\right] \\\\ %\\label{ineq1}\\\\\n    & \\overset{\\eqref{squineq}}{\\leq} & \\mathbb{E}_{i_{k}}\\left[\\frac{2L_{i_{k}}}{ \\pi_{i_{k}}^2} \\left(\\hat{g}_{i_{k}}\\left(\\phi_{i_{k}}^{k}\\right)-\\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right),\\phi_{i_{k}}^{k}-y_{i_{k}}^{k}\\right\\rangle  \\right)\\right] \\\\ %\\label{ineq2}\\\\\n    & = & \\sum_{i=1}^{m} \\frac{2L_{i}}{\\pi_{i}} \\left(\\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)-\\hat{g}_{i}\\left(y_{i}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right),\\phi_{i}^{k}-y_{i}^{k}\\right\\rangle \\right).\n\\end{eqnarray*}\n%where \\eqref{ineq1} follows from $\\mathbb{E}\\|\\zeta-\\mathbb{E} \\zeta\\|^{2}=\\mathbb{E}\\|\\zeta\\|^{2}-\\|\\mathbb{E} \\zeta\\|^{2}$ and \\eqref{ineq2} uses \\eqref{squineq}.\n\\end{proof}","sha256_utf8":"aa073a0022ccd6855836d0075365be193b198ab788290fbe17245f11214b7bfe"}]},{"id":"S013","semantic_type":"proof_step","kind":"proof_step","title":"以函数余量控制梯度差","source_units":[{"start":25578,"end":26027,"start_line":373,"end_line":376,"raw_tex":"Since each $\\hat{g}_i(x)$ is convex and $L_i$ smooth, Theorem 2.1.5 in \\cite{nesterov2018lectures} stipulates\n\\begin{equation}\n\\left\\|\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\nabla \\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)\\right\\|^{2}\\leq 2L_{i}\\left(\\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)-\\hat{g}_{i}\\left(y_{i}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right),\\phi_{i}^{k}-y_{i}^{k}\\right\\rangle  \\right). \\label{squineq}\n\\end{equation}\n","sha256_utf8":"2c7d30b4bfa09a8c3ffd9ea1e24297c1853ac9cf13a535f606eaa6ba3a53b18f"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"随机向量方差恒等式","source_units":[{"start":26027,"end":26176,"start_line":377,"end_line":380,"raw_tex":"Observing\n\\begin{equation} \\label{ex-ineq}\n\\mathbb{E}\\|\\zeta-\\mathbb{E} \\zeta\\|^{2}=\\mathbb{E}\\|\\zeta\\|^{2}-\\|\\mathbb{E} \\zeta\\|^{2},\n\\end{equation}\n","sha256_utf8":"f38ee8f979423114da54acdddc41b2fb0f4f6c5e1c111ad7d19fc16bbefd504c"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"展开抽样方差并求和","source_units":[{"start":26220,"end":27682,"start_line":382,"end_line":390,"raw_tex":"we have\n\\begin{eqnarray*}\n    & & \\mathbb{E}_{i_{k}}\\left[\\left\\|\\widetilde{\\nabla}^{k}- \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)\\right\\|^{2}\\right] \\\\\n    &=& \\mathbb{E}_{i_{k}}\\left[\\left\\|\\frac{1}{\\pi_{i_{k}}}\\left(\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right)-\\nabla \\hat{g}_{i_{k}}\\left(\\phi_{i_{k}}^{k}\\right)\\right)- \\sum_{i=1}^{m}\\left(\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\nabla \\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)\\right) \\right\\|^{2}\\right] \\\\\n    & \\overset{\\eqref{ex-ineq}}{\\leq} & \\mathbb{E}_{i_{k}}\\left[\\left\\|\\frac{1}{\\pi_{i_{k}}}\\left(\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right)-\\nabla \\hat{g}_{i_{k}}\\left(\\phi_{i_{k}}^{k}\\right)\\right) \\right\\|^{2}\\right] \\\\ %\\label{ineq1}\\\\\n    & \\overset{\\eqref{squineq}}{\\leq} & \\mathbb{E}_{i_{k}}\\left[\\frac{2L_{i_{k}}}{ \\pi_{i_{k}}^2} \\left(\\hat{g}_{i_{k}}\\left(\\phi_{i_{k}}^{k}\\right)-\\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right),\\phi_{i_{k}}^{k}-y_{i_{k}}^{k}\\right\\rangle  \\right)\\right] \\\\ %\\label{ineq2}\\\\\n    & = & \\sum_{i=1}^{m} \\frac{2L_{i}}{\\pi_{i}} \\left(\\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)-\\hat{g}_{i}\\left(y_{i}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right),\\phi_{i}^{k}-y_{i}^{k}\\right\\rangle \\right).\n\\end{eqnarray*}\n%where \\eqref{ineq1} follows from $\\mathbb{E}\\|\\zeta-\\mathbb{E} \\zeta\\|^{2}=\\mathbb{E}\\|\\zeta\\|^{2}-\\|\\mathbb{E} \\zeta\\|^{2}$ and \\eqref{ineq2} uses \\eqref{squineq}.\n","sha256_utf8":"409c8335e135d7ce94c99c101907a9dffa42eb2bd1a026ad3f390d8ee8f6ae82"}]},{"id":"S016","semantic_type":"lemma","kind":"lemma","title":"强凸近端三点不等式","source_units":[{"start":27771,"end":28360,"start_line":395,"end_line":405,"raw_tex":"\\begin{lemma} \\label{lemma-3}\nSuppose $h(x)$ is $\\mu$-strongly convex, and $x^{k+1}$ satisfies\n\\[\nx^{k+1} := \\arg \\min _{x\\in \\Gamma}\\left\\{h(x)+\\left\\langle\\widetilde{\\nabla}^{k}, x\\right\\rangle+\\frac{1}{2 \\eta}\\left\\|x^{k}-x\\right\\|^{2}\\right\\}.\n\\]\nThen, for all $u\\in \\Gamma$, it holds that\n\\begin{eqnarray*}\n& & \\left\\langle\\widetilde{\\nabla}^{k}, x^{k+1}-u\\right\\rangle \\\\\n& \\leq & -\\frac{1}{2 \\eta}\\left\\|x^{k+1}-x^{k}\\right\\|^{2}+\\frac{1}{2 \\eta}\\left\\|x^{k}-u\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta}\\left\\|x^{k+1}-u\\right\\|^{2}+h(u)-h\\left(x^{k+1}\\right).\n\\end{eqnarray*}\n\\end{lemma}","sha256_utf8":"f8fc9191fbc5a4d9f927b3f42e8cc8e94e00c450d444305c3ece4bb17e20b4dd"}]},{"id":"S017","semantic_type":"proof","kind":"proof","title":"近端不等式完整证明","source_units":[{"start":28362,"end":28903,"start_line":407,"end_line":413,"raw_tex":"\\begin{proof}\nBy the optimality condition, there exists an $h'\\in \\partial h(x)|_{x=x^{k+1}}$, such that\n\\[\n\\left\\langle\\frac{1}{\\eta}(x^{k+1}-x^{k})+\\widetilde{\\nabla}^{k}+h',u-x^{k+1} \\right\\rangle\\geq 0,\\,\\,\\, \\forall u \\in \\Gamma.\n\\]\nObserve the identity $\\left\\langle x^{k+1}-x^{k},x^{k+1}-u \\right\\rangle=\\frac 1 2\\|x^{k+1}-x^{k}\\|^2+\\frac 1 2\\|x^{k+1}-u\\|^2-\\frac 1 2\\|x^{k}-u\\|^2$ and the inequality $\\left\\langle h',u-x^{k+1} \\right\\rangle\\leq h(u)-h\\left(x^{k+1}\\right)-\\frac{\\mu}{2}\\|x^{k+1}-u\\|^2$, the lemma follows.\n\\end{proof}","sha256_utf8":"d1fce1abc62962fab9f688e300c2fdb6586368a467486f6299ec82f2a54ec259"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"近端更新最优性条件","source_units":[{"start":28376,"end":28600,"start_line":408,"end_line":411,"raw_tex":"By the optimality condition, there exists an $h'\\in \\partial h(x)|_{x=x^{k+1}}$, such that\n\\[\n\\left\\langle\\frac{1}{\\eta}(x^{k+1}-x^{k})+\\widetilde{\\nabla}^{k}+h',u-x^{k+1} \\right\\rangle\\geq 0,\\,\\,\\, \\forall u \\in \\Gamma.\n\\]\n","sha256_utf8":"f5a7b6006649768ed992cb480867c529d44397ea360982fd601d52be99e1c390"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"结合平方恒等式与强凸性","source_units":[{"start":28600,"end":28892,"start_line":412,"end_line":412,"raw_tex":"Observe the identity $\\left\\langle x^{k+1}-x^{k},x^{k+1}-u \\right\\rangle=\\frac 1 2\\|x^{k+1}-x^{k}\\|^2+\\frac 1 2\\|x^{k+1}-u\\|^2-\\frac 1 2\\|x^{k}-u\\|^2$ and the inequality $\\left\\langle h',u-x^{k+1} \\right\\rangle\\leq h(u)-h\\left(x^{k+1}\\right)-\\frac{\\mu}{2}\\|x^{k+1}-u\\|^2$, the lemma follows.\n","sha256_utf8":"4c4286679f77f6a1576a50a3e7e0a4fae5deb27fc703dc9fb80ee8b751143f93"}]},{"id":"S020","semantic_type":"assumption","kind":"assumption","title":"暂定参数可行条件","source_units":[{"start":28998,"end":29319,"start_line":418,"end_line":418,"raw_tex":"First, we assume that we can choose $\\eta>0$ and $\\lambda>0$ such that $0\\leq \\tau_{i} \\leq 1$, $\\frac{1}{\\eta}-\\frac{\\sum_{i=1}^{m}\\tau_i L_i}{m} \\geq \\frac{L_i\\tau_i}{m \\pi_i(1-\\tau_i)}$, $\\forall i=1,2,...,m$ hold. Later in the proof, we will show how to choose $\\eta>0$ and $\\lambda>0$ to satisfy these inequalities.\n","sha256_utf8":"ef6269965f421c21ac0caf500f6fb299bd848cbe5ac6469938e3cb51d21b93db"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"分解分量最优性差距","source_units":[{"start":29320,"end":30214,"start_line":420,"end_line":427,"raw_tex":"By the convexity of $\\hat{g}_{i_{k}}(\\cdot)$,\n\\begin{eqnarray*}\n & &    \\frac{1}{\\pi_{i_{k}}}\\left(\\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right)-\\hat{g}_{i_{k}}\\left(x^{\\star}\\right)\\right)\n    \\leq \\frac{1}{\\pi_{i_{k}}}\\left\\langle\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right), y_{i_{k}}^{k}-x^{\\star}\\right\\rangle \\\\\n    &=& \\frac{1}{\\pi_{i_{k}}}\\frac{1-\\tau_{i_{k}}}{\\tau_{i_{k}}}\\left\\langle\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right), \\phi_{i_{k}}^{k}-y_{i_{k}}^{k}\\right\\rangle+\\left\\langle\\frac{1}{ \\pi_{i_{k}}}\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right)-\\widetilde{\\nabla}^{k}, x^{k}-x^{\\star}\\right\\rangle\\\\\n    & &+\\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle+\\left\\langle\\widetilde{\\nabla}^{k}, x^{k+1}-x^{\\star}\\right\\rangle\n\\end{eqnarray*}\nwhere the last step uses the definition that $y_{i}=\\tau_{i} x^{k}+(1-\\tau_{i}) \\phi_{i}^{k}$, for $i=1,2,...,m$.\n","sha256_utf8":"4c266b13ff0b08b8f76d2cfbdd410f2e3bc66df73a0b35b329c9fdfb493b40ff"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"取期望消除估计偏差项","source_units":[{"start":30382,"end":31204,"start_line":430,"end_line":438,"raw_tex":"Observe\n\\[\n\\mathbb{E}_{i_{k}}\\left[\\widetilde{\\nabla}^{k}-\\frac{1}{\\pi_{i_{k}}}\\nabla \\hat{g}_{i_{k}}\\left(y_{i_{k}}^{k}\\right)\\right]=\\mathbb{E}_{i_{k}}\\left[\\widetilde{\\nabla}^{k}\\right]-\\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)=0.\n\\]\nBy taking expectation with respect to sample $i_k$, it follows\n\\begin{eqnarray} \\label{opt-gap} \\\\\n& & \\sum_{i=1}^{m} \\left( \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\hat{g}_i\\left(x^{\\star}\\right) \\right)  \\nonumber  \\\\\n&\\leq& \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle+\\mathbb{E}_{i_{k}}\\left[\\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle\\right]+\\mathbb{E}_{i_{k}}\\left[\\left\\langle\\widetilde{\\nabla}^{k}, x^{k+1}-x^{\\star}\\right\\rangle\\right] . \\nonumber\n\\end{eqnarray}\n","sha256_utf8":"253918515bba72c30bd55dea64a1a5922bc4fdbc1f5b2c2441eb3e65a6614f9d"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"记忆点更新的光滑上界","source_units":[{"start":32758,"end":33097,"start_line":460,"end_line":463,"raw_tex":"Theorem 2.1.5 in \\cite{nesterov2018lectures} tells us that\n\\[\n\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)-\\hat{g}_{j_{k}}\\left(y_{j_{k}}^{k}\\right) \\leq\\left\\langle\\nabla \\hat{g}_{j_{k}}\\left(y_{j_{k}}^{k}\\right), \\phi_{j_{k}}^{k+1}-y_{j_{k}}^{k}\\right\\rangle+\\frac{L_{j_{k}}}{2}\\left\\|\\phi_{j_{k}}^{k+1}-y_{j_{k}}^{k}\\right\\|^{2} .\n\\]\n","sha256_utf8":"8dade8728150f0cd8e47db4d69718c22dff9fe96c825d8c293a97dc20f8c773a"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"对记忆点抽样取期望","source_units":[{"start":33098,"end":34103,"start_line":465,"end_line":477,"raw_tex":"Dividing the above inequality by $\\pi_{j_{k}}\\tau_{j_{k}}$ and taking expectation with respect to sample $j_k$, we obtain\n\\begin{eqnarray*}\n& & \\mathbb{E}_{j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]-\\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_i\\left(y_i^{k}\\right) \\\\\n&\\leq& \\left\\langle \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), x^{k+1}-x^{k}\\right\\rangle+\\frac{\\sum_{i=1}^{m}\\tau_i L_i}{2}\\|x^{k+1}-x^{k}\\|^2,\n\\end{eqnarray*}\nwhich is equivalent to\n\\begin{eqnarray*}\n     \\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle\n     &\\leq&\n    \\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)- \\mathbb{E}_{j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right] \\\\\n    & & +\\left\\langle \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\widetilde{\\nabla}^{k},\n    x^{k+1}-x^{k}\\right\\rangle+\\frac{\\sum_{i=1}^{m}\\tau_i L_i}{2}\\|x^{k+1}-x^{k}\\|^2.\n\\end{eqnarray*}\n","sha256_utf8":"3a75794254624967e88d8be437e8828f5a7291fa8ec0595e4e6d99c92ad2dc28"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"进一步平均梯度抽样","source_units":[{"start":34103,"end":34736,"start_line":478,"end_line":485,"raw_tex":"Taking expectation with respect to sample $i_k$ we get\n\\begin{eqnarray*}\n     \\mathbb{E}_{i_{k}}\\left[\\left\\langle\\widetilde{\\nabla}^{k}, x^{k}-x^{k+1}\\right\\rangle\\right] &\\leq&\n    \\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)\n    - \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]\\\\\n& & +\\mathbb{E}_{i_{k}}\\left[\\left\\langle \\sum_{i=1}^{m}\n\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\widetilde{\\nabla}^{k}, x^{k+1}-x^{k}\\right\\rangle\\right]+\\frac{\\sum_{i=1}^{m}\\tau_i L_i}{2}\\mathbb{E}_{i_{k}}\\left[\\|x^{k+1}-x^{k}\\|^2\\right] .\n\\end{eqnarray*}\n","sha256_utf8":"9ea5e5db40e626fa838e57699fcebe66d2db365b68c51eab7bef90b38c9ffda3"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"应用近端三点界","source_units":[{"start":34737,"end":35228,"start_line":487,"end_line":491,"raw_tex":"By Lemma~\\ref{lemma-3}, taking expectation we have\n\\begin{eqnarray*}\n& & \\mathbb{E}_{i_{k}}\\left[ \\left\\langle\\widetilde{\\nabla}^{k}, x^{k+1}-x^\\star\\right\\rangle \\right]  \\\\\n&\\leq& - \\mathbb{E}_{i_{k}}\\left[\\frac{1}{2 \\eta}\\left\\|x^{k+1}-x^{k}\\right\\|^{2}\\right]+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^\\star\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[ \\left\\|x^{k+1}-x^\\star\\right\\|^{2}\\right]+h(x^\\star) - \\mathbb{E}_{i_{k}}\\left[ h\\left(x^{k+1}\\right)\\right] .\n\\end{eqnarray*}\n","sha256_utf8":"d83ef3e6753279cba101c2d5e48a75dadba51423e88abd58871166f6b92164df"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"合并差距界并吸收方差","source_units":[{"start":35229,"end":39165,"start_line":493,"end_line":519,"raw_tex":"Applying the above inequalities on \\eqref{opt-gap}, we have % \\eqref{eq1}, we obtain\n\\begin{eqnarray*}\n    & & \\sum_{i=1}^{m} \\left( \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\hat{g}_i \\left(x^{\\star}\\right) \\right) \\\\\n    &\\leq&\n     \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle\n    +\\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)\n    - \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right) \\right] \\\\\n   & & +\\mathbb{E}_{i_{k}}\\left[\\left\\langle \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\widetilde{\\nabla}^{k}, x^{k+1}-x^{k}\\right\\rangle\\right]+\\frac{\\sum_{i=1}^{m}\\tau_i L_i}{2}\\mathbb{E}_{i_{k}}\\left[\\|x^{k+1}-x^{k}\\|^2\\right]\\\\\n   & & -\\frac{1}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{k}\\right\\|^{2}\\right]+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right]\n    +h\\left(x^{\\star}\\right)-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right]\\\\\n   & \\leq&  \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle\n    +\\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)\n    - \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]\\\\\n   & & + \\frac{1}{2\\left(\\frac{1}{\\eta}-\\sum_{i=1}^{m}\\tau_i L_i \\right)}\\mathbb{E}_{i_{k}}\\left[\\left\\| \\sum_{i=1}^{m} \\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right)-\\widetilde{\\nabla}^{k}\\right\\|^2\\right]+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}\\\\\n   & & -\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right]+h\\left(x^{\\star}\\right)-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right]\\\\\n   &\\leq&  \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle\n    +\\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)\n    - \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]\\\\\n   & &+ \\frac{1}{2\\left(\\frac{1}{\\eta}-\\sum_{i=1}^{m}\\tau_i L_i \\right)}\\sum_{i=1}^{m} \\frac{2L_{i}}{\\pi_{i}} \\left(\\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)-\\hat{g}_{i}\\left(y_{i}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right),\\phi_{i}^{k}-y_{i}^{k}\\right\\rangle  \\right)\\\\\n   & &+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right]+h\\left(x^{\\star}\\right)-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right]\\\\\n   &\\leq&  \\sum_{i=1}^{m}\\frac{1-\\tau_{i}}{\\tau_{i}}\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right), \\phi_{i}^{k}-y_{i}^{k}\\right\\rangle\n    +\\sum_{i=1}^{m} \\frac{1}{\\tau_i} \\hat{g}_{i}\\left(y_{i}^{k}\\right)\n    - \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}\\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]\\\\\n  &  &+ \\sum_{i=1}^{m} \\frac{1-\\tau_{i}}{\\tau_{i}} \\left(\\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)-\\hat{g}_{i}\\left(y_{i}^{k}\\right) -\\left\\langle\\nabla \\hat{g}_{i}\\left(y_{i}^{k}\\right),\\phi_{i}^{k}-y_{i}^{k}\\right\\rangle\\right)\\\\\n  &  &+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right]+h\\left(x^{\\star}\\right)-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right]\n\\end{eqnarray*}\nwhere the second inequality follows from the inequality $\\langle a, b\\rangle \\leq \\frac{1}{2 \\beta}\\|a\\|^{2}+\\frac{\\beta}{2}\\|b\\|^{2}$ and $\\frac{1}{\\eta}-\\sum_{i=1}^{m}\\tau_i L_i >0$, and the third inequality is due to Lemma \\ref{lemma1}, and the last inequality holds by the assumption that $\\frac{1}{\\eta}-\\sum_{i=1}^{m}\\tau_i L_i \\geq \\frac{L_i\\tau_i}{\\pi_i(1-\\tau_i)}$.\n","sha256_utf8":"beadc466c9a9d7174430f0959070a5bfa571d05693a6a73922932072d9072fe5"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"消项整理单步递推","source_units":[{"start":39166,"end":39686,"start_line":521,"end_line":525,"raw_tex":"Canceling out and rearranging the terms, we have\n\\begin{eqnarray*}\n    &     &  \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}} \\hat{g}_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]-F(x^{\\star})\\\\\n    &\\leq & \\sum_{i=1}^{m} \\frac{1-\\tau_{i}}{\\tau_{i}} \\hat{g}_{i}\\left(\\phi_{i}^{k}\\right)+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right]-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right].\n\\end{eqnarray*}\n","sha256_utf8":"f00c7afa7967cf2c7138be5c01afe531bb2aede05419518bcf9a20eb8a680138"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"利用凸性转移二次项","source_units":[{"start":39687,"end":40345,"start_line":527,"end_line":534,"raw_tex":"By the convexity of $h(\\cdot)$ and $\\phi_{j_{k}}^{k+1}=\\tau_{j_{k}} x^{k+1}+(1-\\tau_{j_{k}}) \\phi_{j_{k}}^{k}$, as long as $0\\leq \\tau_{j_{k}} \\leq 1$, we have\n\\[\nh(\\phi_{j_{k}}^{k+1})\\leq \\tau_{j_{k}} h(x^{k+1})+(1-\\tau_{j_{k}}) h(\\phi_{j_{k}}^{k}) .\n\\]\nDividing the above inequality by $m \\pi_{j_{k}} \\tau_{j_{k}}$ and taking expectation with respect to sample $j_k$ and sample $i_k$, we obtain\n\\[\n-\\mathbb{E}_{i_{k}}\\left[h\\left(x^{k+1}\\right)\\right] \\leq  \\sum_{i=1}^{m} \\frac{1-\\tau_{j_{k}}}{m \\tau_{j_{k}}} h\\left(\\phi_{i}^{k}\\right)- \\mathbb{E}_{i_{k}, j_{k}}\\left[\\frac{1}{m \\pi_{j_{k}} \\tau_{j_{k}}}     h\\left(\\phi_{j_{k}}^{k+1}\\right)\\right] .\n\\]\n","sha256_utf8":"3e2afd95d39217f14ac54b372e5aee72f1ea3cac74c2386ea3eaf6fef191a2d1"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"定义重分配后的分量函数","source_units":[{"start":40345,"end":40866,"start_line":535,"end_line":539,"raw_tex":"Applying the above inequality and using $F_{i}(\\cdot):=\\hat{g}_{i}(\\cdot)+\\frac 1 m h(\\cdot)$, we further derive\n\\begin{eqnarray*}\n    &      &  \\mathbb{E}_{i_{k},j_{k}}\\left[\\frac{1}{\\pi_{j_{k}}\\tau_{j_{k}}}F_{j_{k}}\\left(\\phi_{j_{k}}^{k+1}\\right)\\right]-F(x^{\\star})\\\\\n    &\\leq  & \\sum_{i=1}^{m} \\frac{1-\\tau_{i}}{\\tau_{i}} F_{i}\\left(\\phi_{i}^{k}\\right)+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}}\\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2}\\right] .\n\\end{eqnarray*}\n","sha256_utf8":"c836c9a6e54b71224316702b18ad7fcadb9c551aa3415781ac5502083a9187e7"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"补齐全部记忆点函数项","source_units":[{"start":40867,"end":41448,"start_line":541,"end_line":545,"raw_tex":"Adding $\\mathbb{E}_{i_{k},j_{k}}\\left[\\sum_{i\\neq j_k}\\frac{1}{\\pi_i\\tau_i}F_i\\left(\\phi_i^{k}\\right)\\right]$ to both sides, we have\n\\begin{eqnarray*}\n    &     & \\mathbb{E}_{i_{k},j_{k}}\\left[\\sum_{i=1}^m\\frac{1}{\\pi_i\\tau_i}\\left(F_i\\left(\\phi_i^{k+1}\\right)-F_i(x^{\\star})\\right)\\right]\\\\\n    &\\leq & \\sum_{i=1}^{m} \\frac{1-\\pi_i\\tau_{i}}{\\pi_i\\tau_{i}} \\left(F_{i}\\left(\\phi_{i}^{k}\\right)-F_i(x^{\\star})\\right)+\\frac{1}{2 \\eta}\\left\\|x^{k}-x^{\\star}\\right\\|^{2}-\\frac{1+\\eta \\mu}{2 \\eta} \\mathbb{E}_{i_{k}} \\left[\\left\\|x^{k+1}-x^{\\star}\\right\\|^{2} \\right] .\n\\end{eqnarray*}\n","sha256_utf8":"851e99674d298d91ba3e4941404dea17ef0d2ff053e8b035eba57af7432a2d9c"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"加入最优点梯度线性修正","source_units":[{"start":41448,"end":42437,"start_line":546,"end_line":555,"raw_tex":"Noticing $\\sum_{i=1}^{m} \\left(F_{i}\\left(\\phi_{i}^{k}\\right)-F_i(x^{\\star})\\right)$ may not be positive, we need to add the following term in our Lyapunov function\n\\begin{eqnarray*}\n  & &    -\\sum_{i=1}^{m}\\frac{1}{\\pi_i\\tau_i}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k+1}-x^{\\star}\\right\\rangle \\\\\n  &=&  -\\frac{1}{\\pi_{j_{k}} \\tau_{j_{k}}}\\left\\langle\\nabla F_{j_{k}}\\left(x^{\\star}\\right), \\phi_{j_{k}}^{k+1}-x^{\\star}\\right\\rangle-\n  \\sum_{i \\neq j_k } %I \\atop \\mathfrak{l}}^{m}\n  \\frac{1}{\\pi_i\\tau_i}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k}-x^{\\star}\\right\\rangle\\\\\n  &=& -\\frac{1}{\\pi_{j_{k}}}\\left\\langle\\nabla F_{j_{k}}\\left(x^{\\star}\\right), x^{k+1}-x^{\\star}\\right\\rangle\n    +\\frac{1}{\\pi_{j_{k}}}\\left\\langle\\nabla F_{j_{k}}\\left(x^{\\star}\\right), \\phi_{j_{k}}^{k}-x^{\\star}\\right\\rangle\\\\\n  &  & - \\sum_{i =1}^{m}\\frac{1}{\\pi_i\\tau_i}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k}-x^{\\star}\\right\\rangle .\n\\end{eqnarray*}\n","sha256_utf8":"1fef25b1195fac4724d0e16090a4da115984f4206c9e0e2315fd5dc15d4f5656"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"线性修正项的期望递推","source_units":[{"start":42438,"end":42828,"start_line":557,"end_line":561,"raw_tex":"Taking expectation with respect to samples $i_k$ and $j_k$ and using $\\pi_i\\tau_i=\\lambda$, we get\n\\[\n\\mathbb{E}_{i_{k},j_{k}}\\left[-\\frac{1}{\\lambda} \\sum_{i=1}^{m}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k+1}-x^{\\star}\\right\\rangle\\right]=\n-\\frac{1-\\lambda}{\\lambda} \\sum_{i =1}^{m}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k}-x^{\\star}\\right\\rangle .\n\\]\n","sha256_utf8":"5708a063c2617aab221aee3763a75e72ad01983748e71c69729662bd2657f59b"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"构造李雅普诺夫递推","source_units":[{"start":42829,"end":43260,"start_line":563,"end_line":566,"raw_tex":"Denoting $D^k:=\\sum_{i =1}^{m}F_{i}\\left(\\phi_{i}^{k}\\right)-F\\left(x^{\\star}\\right)-\\sum_{i =1}^{m}\\left\\langle\\nabla F_{i}\\left(x^{\\star}\\right), \\phi_{i}^{k}-x^{\\star}\\right\\rangle$ and $P^{k} :=\\left\\|x^{k}-x^{\\star}\\right\\|^{2}$, we have\n\\[\n\\frac{1}{\\lambda}\\mathbb{E}_{i_{k},j_{k}}\\left[D^{k+1}\\right]+\\frac{1+\\eta \\mu}{2 \\eta}\\mathbb{E}_{i_{k}}\\left[P^{k+1}\\right]\\leq \\frac{1-\\lambda}{\\lambda} D^k+\\frac{1}{2 \\eta} P^k.\n\\]\n","sha256_utf8":"800ebfbf4074ad045fd7638e8311eb00f966cb799e2b854fa313b66e018785e4"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"列出待验证参数条件","source_units":[{"start":43261,"end":43508,"start_line":568,"end_line":572,"raw_tex":"Finally, it remains to choose $\\eta$ and $\\lambda$ so as to satisfy\n \\[\n 0\\leq \\tau_{i} \\leq 1, \\mbox{ and }  \\frac{1}{\\eta}-\\sum_{i=1}^{m}\\tau_i L_i \\geq \\frac{L_i\\tau_i}{\\pi_i(1-\\tau_i)},\\,\\,\\, i=1,2,...,m.\n \\]\nWe consider two cases separately.\n","sha256_utf8":"a2060a3ffaccde9fd8322423044de35008f1e24ab8122138a07d6135c904c654"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"第一种情形的参数选择","source_units":[{"start":43509,"end":43718,"start_line":574,"end_line":577,"raw_tex":"\\textbf{Case I.} If $\\sqrt{\\mu}\\leq \\frac{\\sum_{j=1}^m\\sqrt{L_j}}{m}$, then we choose\n\\[\n\\lambda := \\frac{\\sqrt{\\mu}}{4\\sum_{j=1}^m\\sqrt{L_j}}, \\quad \\eta := \\frac{1}{4\\sqrt{\\mu}\\sum_{j=1}^{m}\\sqrt{L_j} }.\n\\]\n","sha256_utf8":"42ff019f88e530365816511695ebca4477b41c432bd9de3256ce808fd83aaee7"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"第一种情形的可行性估计","source_units":[{"start":43719,"end":44344,"start_line":579,"end_line":589,"raw_tex":"Recall\n$\\pi_i=\\frac{\\sqrt{L_i}}{2\\sum_{j=1}^m\\sqrt{L_j}}+\\frac{1}{2m}$ for all $1\\le i\\le m$. Therefore,\n\\[\n\\tau_i=\\frac{\\lambda}{\\pi_i}=\\frac{\\sqrt{\\mu}}{2\\sqrt{L_i}+\\frac 2 m \\sum_{j=1}^m\\sqrt{L_j}}\\leq \\frac{\\sqrt{\\mu}}{\\frac 2 m \\sum_{j=1}^m\\sqrt{L_j}}\\leq \\frac 1 2\n\\]\nand\n\\begin{eqnarray*}\n     \\sum_{j=1}^{m}\\tau_j L_j +\\frac{L_i\\tau_i}{\\pi_i(1-\\tau_i)} &\\leq& \\sum_{j=1}^{m}L_j\\frac{\\sqrt{\\mu}}{2\\sqrt{L_j}}+\\frac{2\\lambda L_i}{ \\pi_i^2}\\\\\n    &\\leq& \\frac{\\sqrt{\\mu}\\sum_{j=1}^{m}\\sqrt{L_j} }{2}+\\frac{8(\\sum_{j=1}^{m}\\sqrt{L_j})^2\\lambda L_i}{L_i}\\\\\n    &\\leq& 3\\sqrt{\\mu}\\sum_{j=1}^{m}\\sqrt{L_j} .\n\\end{eqnarray*}\n","sha256_utf8":"a41d432f1c234c68c8c2d74b60c9d1aed84919c3c8174ae0e322f07d0a001a7d"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"验证第一种情形收缩条件","source_units":[{"start":44345,"end":44568,"start_line":591,"end_line":594,"raw_tex":"Since $\\eta=\\frac{1}{4\\sqrt{\\mu}\\sum_{j=1}^{m}\\sqrt{L_j} }$, we know $\\frac{1}{\\eta}-\\sum_{j=1}^{m}\\tau_j L_j \\geq \\frac{L_i\\tau_i}{ \\pi_i(1-\\tau_i)}$ and\n\\[\n\\frac{1}{\\lambda}\\geq (1+\\eta \\mu)\\frac{1-\\lambda}{\\lambda} .\n\\]\n","sha256_utf8":"20b25b7a752b82cf59e9b6d810851947490cf5b4508baa1e1a2771f2431657fd"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"第一种情形的势函数收缩","source_units":[{"start":44568,"end":44817,"start_line":595,"end_line":598,"raw_tex":"Thus, in this case we obtain\n\\[\n\\frac{1}{\\lambda}\\mathbb{E}_{i_{k},j_{k}}\\left[D^{k+1}\\right]+\\frac{1+\\eta \\mu}{2 \\eta}\\mathbb{E}_{i_{k}}\\left[P^{k+1}\\right]\\leq (1+\\eta \\mu)^{-1}\\left( \\frac{1}{\\lambda} D^k+\\frac{1+\\eta \\mu}{2 \\eta} P^k\\right).\n\\]\n","sha256_utf8":"60b29148cf56fa3786c4e001441d653ea1d456c94f4baad03b84d54f46c1b5ea"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"第一种情形的距离收敛界","source_units":[{"start":44818,"end":45196,"start_line":600,"end_line":605,"raw_tex":"Telescoping the above inequalities, we have\n%\\begin{equation}\n\\begin{eqnarray*}\n& & \\mathbb{E}\\left[\\left\\|x^{K+1}-x^{\\star}\\right\\|^{2}\\right] \\\\\n&\\leq& \\left(1+\\frac{\\sqrt{\\mu}}{4\\sum_{j=1}^m\\sqrt{L_j}}\\right)^{-K} \\cdot\\left(\\frac{2\\eta}{\\lambda(1+\\eta \\mu)}\\left(F\\left(x^{1}\\right)-F\\left(x^{\\star}\\right)\\right)+\\left\\|x^{1}-x^{\\star}\\right\\|^{2}\\right) .\n\\end{eqnarray*}\n","sha256_utf8":"286d10eb4b95bdde6c0798a10a1e79fc38b350b8e196c25ea851e8a238421bc7"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"第二种情形的参数选择","source_units":[{"start":45213,"end":45360,"start_line":608,"end_line":611,"raw_tex":"\\textbf{Case II.} If $\\sqrt{\\mu}> \\frac{\\sum_{j=1}^m\\sqrt{L_j}}{m}$, then we choose\n\\[\n\\lambda := \\frac{1}{4m}, \\quad \\eta := \\frac{1}{4\\mu m}.\n\\]\n","sha256_utf8":"d26e72f33f7aaf08f9870909f1679155de1810b52efa7d3ab72653a4e175eaa7"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"第二种情形的可行性估计","source_units":[{"start":45361,"end":45920,"start_line":613,"end_line":623,"raw_tex":"In this case, we have\n\\[\n\\tau_i=\\frac{\\lambda}{\\pi_i}=\\frac{\\frac{1}{2m}\\sum_{j=1}^m\\sqrt{L_j}}{\\sqrt{L_i}+\\frac 1 m \\sum_{j=1}^m\\sqrt{L_j}}\\leq \\frac 1 2\n\\]\nand\n\\begin{eqnarray*}\n    \\sum_{j=1}^{m}\\tau_j L_j +\\frac{L_i\\tau_i}{\\pi_i(1-\\tau_i)}&\\leq& \\frac{1}{2m}\\left(\\sum_{j=1}^m\\sqrt{L_j}\\right)^2+\\frac{2\\lambda L_i}{ \\pi_i^2}\\\\\n    &\\leq& \\frac{\\left(\\sum_{j=1}^m\\sqrt{L_j}\\right)^2}{2m}+\\frac{8\\left(\\sum_{j=1}^{m}\\sqrt{L_j}\\right)^2\\lambda L_i}{L_i}\\\\\n    &\\leq& \\frac{3\\left(\\sum_{j=1}^{m}\\sqrt{L_j}\\right)^2 }{m}\\\\\n    &\\leq& 3 \\mu m.\n\\end{eqnarray*}\n","sha256_utf8":"11d7af923e4b9c4ec9028943f54ff71d787f1257d2e5d08da8fb611d6a86aca3"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"验证第二种情形收缩条件","source_units":[{"start":45921,"end":46114,"start_line":625,"end_line":628,"raw_tex":"Since $\\eta=\\frac{1}{4\\mu m}$, we have $\\frac{1}{\\eta}-\\sum_{j=1}^{m}\\tau_j L_j \\geq \\frac{L_i\\tau_i}{\\pi_i(1-\\tau_i)}$, and\n\\[\n\\frac{1}{\\lambda}\\geq (1+\\eta \\mu)\\frac{1-\\lambda}{\\lambda} .\n\\]\n","sha256_utf8":"768bfca55c967af3c6e2c5e330d7e43dd534e32240b22f78b2d572de98eed9de"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"第二种情形的势函数收缩","source_units":[{"start":46115,"end":46353,"start_line":630,"end_line":633,"raw_tex":"As a consequence,\n\\[\n\\frac{1}{\\lambda}\\mathbb{E}_{i_{k},j_{k}}\\left[D^{k+1}\\right]+\\frac{1+\\eta \\mu}{2 \\eta}\\mathbb{E}_{i_{k}}\\left[P^{k+1}\\right]\\leq (1+\\eta \\mu)^{-1}\\left( \\frac{1}{\\lambda} D^k+\\frac{1+\\eta \\mu}{2 \\eta} P^k\\right).\n\\]\n","sha256_utf8":"9f2b77be79de259bd9fd4f7833b347e04b8b56d4b8c33d9566be50afcc4026db"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"第二种情形收敛与两情形合并","source_units":[{"start":46353,"end":46750,"start_line":634,"end_line":638,"raw_tex":"Taking expectations and telescoping the above inequalities, we have\n\\begin{equation}\n\\mathbb{E}\\left[\\left\\|x^{K+1}-x^{\\star}\\right\\|^{2}\\right] \\leq\\left(1+\\frac{1}{4m}\\right)^{-K} \\cdot\\left(\\frac{2\\eta}{\\lambda(1+\\eta \\mu)}\\left(F\\left(x^{1}\\right)-F\\left(x^{\\star}\\right)\\right)+\\left\\|x^{1}-x^{\\star}\\right\\|^{2}\\right) .\n\\end{equation}\nSummarizing Case I and Case II, the theorem is proven.\n","sha256_utf8":"5867fd640656f7663b7473bc098a14dc8ac12db3164f026858d8ba84da9f9667"}]},{"id":"S046","semantic_type":"external_fact","kind":"external_fact","title":"光滑凸函数基本不等式","source_units":[],"declared_external":true,"declaration_text":["For a differentiable convex L-smooth function f, f(v) <= f(u) + <grad f(u), v-u> + (L/2)||v-u||^2, and ||grad f(v)-grad f(u)||^2 <= 2L(f(v)-f(u)-<grad f(u),v-u>). These are the standard facts invoked through Nesterov (2018), Theorem 2.1.5."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S009","edges":[{"source_id":"S001","target_id":"S008","kind":"supports"},{"source_id":"S002","target_id":"S009","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S006","target_id":"S013","kind":"supports"},{"source_id":"S046","target_id":"S013","kind":"supports"},{"source_id":"S007","target_id":"S015","kind":"supports"},{"source_id":"S013","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S014","target_id":"S012","kind":"supports"},{"source_id":"S015","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S011","kind":"supports"},{"source_id":"S001","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S003","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S006","target_id":"S021","kind":"supports"},{"source_id":"S008","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S007","target_id":"S022","kind":"supports"},{"source_id":"S046","target_id":"S023","kind":"supports"},{"source_id":"S006","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S008","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S016","target_id":"S026","kind":"supports"},{"source_id":"S006","target_id":"S026","kind":"supports"},{"source_id":"S008","target_id":"S026","kind":"supports"},{"source_id":"S020","target_id":"S027","kind":"supports"},{"source_id":"S022","target_id":"S027","kind":"supports"},{"source_id":"S025","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S011","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S006","target_id":"S029","kind":"supports"},{"source_id":"S008","target_id":"S029","kind":"supports"},{"source_id":"S020","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S008","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S008","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S008","target_id":"S033","kind":"supports"},{"source_id":"S031","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S007","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S034","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S008","target_id":"S040","kind":"supports"},{"source_id":"S035","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S007","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S041","target_id":"S043","kind":"supports"},{"source_id":"S034","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S040","target_id":"S045","kind":"supports"},{"source_id":"S008","target_id":"S045","kind":"supports"},{"source_id":"S021","target_id":"S010","kind":"supports"},{"source_id":"S022","target_id":"S010","kind":"supports"},{"source_id":"S023","target_id":"S010","kind":"supports"},{"source_id":"S024","target_id":"S010","kind":"supports"},{"source_id":"S025","target_id":"S010","kind":"supports"},{"source_id":"S026","target_id":"S010","kind":"supports"},{"source_id":"S027","target_id":"S010","kind":"supports"},{"source_id":"S028","target_id":"S010","kind":"supports"},{"source_id":"S029","target_id":"S010","kind":"supports"},{"source_id":"S030","target_id":"S010","kind":"supports"},{"source_id":"S031","target_id":"S010","kind":"supports"},{"source_id":"S032","target_id":"S010","kind":"supports"},{"source_id":"S033","target_id":"S010","kind":"supports"},{"source_id":"S034","target_id":"S010","kind":"supports"},{"source_id":"S035","target_id":"S010","kind":"supports"},{"source_id":"S036","target_id":"S010","kind":"supports"},{"source_id":"S037","target_id":"S010","kind":"supports"},{"source_id":"S038","target_id":"S010","kind":"supports"},{"source_id":"S039","target_id":"S010","kind":"supports"},{"source_id":"S040","target_id":"S010","kind":"supports"},{"source_id":"S041","target_id":"S010","kind":"supports"},{"source_id":"S042","target_id":"S010","kind":"supports"},{"source_id":"S043","target_id":"S010","kind":"supports"},{"source_id":"S044","target_id":"S010","kind":"supports"},{"source_id":"S045","target_id":"S010","kind":"supports"},{"source_id":"S010","target_id":"S009","kind":"supports"}]}}
{"record_id":"2209.10825v4","paper":{"title":"Nonsmooth Nonconvex-Nonconcave Minimax Optimization: Primal-Dual Balancing and Iteration Complexity Analysis","authors":["Jiajin Li","Linglingzhi Zhu","Anthony Man-Cho So"],"arxiv_id":"2209.10825","version":"v4","journal":"Mathematical Programming","doi":"10.1007/s10107-025-02197-1","abstract_url":"https://arxiv.org/abs/2209.10825v4","pdf_url":"https://arxiv.org/pdf/2209.10825v4"},"context":{"raw_tex":"\\begin{equation}\n\\label{eq:problem}\n\\min_{x\\in \\mathcal{X}} \\max_{y\\in\\mathcal{Y}} F(x,y).\n\\tag{P}\n\\end{equation}\n\nHere, $F:\\R^n \\times \\R^d \\rightarrow \\R$ can be both nonconvex with respect to $x$ and nonconcave with respect to $y$, $\\mathcal{X}\\subseteq \\R^n$ is a nonempty closed convex set, and $\\mathcal{Y}\\subseteq \\R^d$ is a nonempty compact convex set. This problem has a wide range of applications in machine learning and operations research. Examples include learning with non-decomposable loss~\\citep{shalev2016minimizing,zhang2018ell_1}, training generative adversarial networks~\\citep{arjovsky2017wasserstein,goodfellow2020generative}, adversarial training~\\citep{madry2017towards,sinha2017certifying}, and (distributionally) robust optimization (DRO)~\\citep{bertsimas2011theory,rahimian2019distributionally}. \n\n\n\\begin{assumption}[Problem setup] \\label{ass:basic}\nThe following assumptions on the objective function $F$ of problem \\eqref{eq:problem} hold throughout the paper. \n\\begin{enumerate}[label=\\normalfont(\\alph*)]\n\\item   $(\\textrm{\\bf Primal function})$ For all $y\\in \\mathcal{Y}$, the function $F(\\cdot,y): \\mathbb{R}^n \\rightarrow \\mathbb{R}$ takes the form $F(\\cdot, y):=h_y\\circ c_y(\\cdot)$, where $c_y:\\R^n\\rightarrow \\R^m$ is \ncontinuously differentiable\nwith $L_c$-Lipschitz continuous Jacobian mapping, i.e.,\n\\[\n\\|\\nabla c_y(x)-\\nabla c_y(x')\\| \\leq L_c\\|x-x'\\| \\quad \\text { for all } x, x' \\in \\mathcal{X};\n\\]\nand $h_y:\\R^m\\rightarrow \\R$  is  convex and $L_h$-Lipschitz continuous, i.e.,\n\\[\n|h_y(z)-h_y(z')| \\leq L_h\\|z-z'\\| \\quad \\text { for all } z, z' \\in \\R^{m}.\n\\]\n\\item  $(\\textrm{\\bf Dual function})$ For all $x\\in \\mathcal{X}$, the function $F(x, \\cdot):\\R^d \\rightarrow \\R$ is continuously differentiable  with $\\nabla_y F(\\cdot,\\cdot)$ being $L$-Lipschitz continuous on $\\mathcal{X}\\times \\mathcal{Y}$, i.e., \n\\[\n\\|\\nabla_y F(x,y)-\\nabla_y F(x',y')\\| \\leq L\\|(x,y)-(x',y')\\| \\,\n\\text{ for all }  (x,y), (x',y') \\in \\mathcal{X}\\times\\mathcal{Y}.\n\\]\nWithout loss of generality, we may take $L=L_h L_c$.\n\\end{enumerate}\n\\end{assumption}\n\n\\begin{assumption}[K\\L{} property with exponent $\\theta$ for dual problem]\n\\label{ass:kl}\nFor all  $x\\in\\mathcal{X}$, the problem $\\max_{y\\in \\mathcal{Y}} F(x,y)$ has a nonempty solution set and a finite optimal value. Moreover, there exist $\\mu>0$ and $\\theta \\in [0,1)$ such that  \n\\[\n\\dist(0, -\\nabla_y F(x,y)+\\partial\\iota_{\\mathcal{Y}}(y)) \\ge \\mu  \\left(\\max\\limits_{y'\\in \\mathcal{Y}} F(x,y')-F(x,y)\\right)^{\\theta}\n\\]\nfor all $x \\in \\mathcal{X}$ and  $y \\in \\mathcal{Y}\\setminus \\mathcal{Y}^\\star(x)$, where $\\mathcal{Y}^\\star(x):= \\argmax_{y' \\in \\mathcal{Y}} F(x,y')$ \n% satisfying $F(x,y)<\\max_{y'\\in\\mathcal{Y}} F(x,y')$\n.\n\\end{assumption}\n\nLet us now examine the stationarity measures discussed in this paper. We define the value function $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$, the potential function $F_r:\\R^n\\times \\R^d\\times \\R^n\\rightarrow\\R$, and the dual potential value function $d_r:\\R^d\\times \\R^n\\rightarrow \\R$ by\n% \\[\n% F_r(x,y,z) := F(x,y)+\\frac{r}{2}\\|x-z\\|^2\\quad \\text{and}\\quad d_r(y,z):=\\min_{x\\in\\mathcal{X}} F_r(x,y,z),\n% \\]\n\\begin{align*}\nf(x) := \\max_{y \\in \\mathcal{Y}} F(x,y),\\quad F_r(x,y,z) := F(x,y) + \\frac{r}{2} \\| x-z \\|^2,\\quad d_r(y,z) := \\min_{x \\in \\mathcal{X}} F_r(x,y,z),\n\\end{align*}\nrespectively, where we assume $r> L$ in the rest of this paper.\n\n\n\\begin{definition}[Stationarity measures]\n\\label{defi:primal-dual}\nLet $\\epsilon \\ge 0$ be given.\n\\begin{enumerate}[label=\\normalfont(\\alph*)]\n\\item The point $x\\in\\mathcal{X}$ is an $\\epsilon$-optimization-stationary point {\\rm($\\epsilon$-OS)} of problem \\eqref{eq:problem} if \n\\[\\|\\prox_{\\frac{1}{r}f+\\iota_{\\mathcal{X}}}(x)-x\\|\\leq \\epsilon.\\]\n\\item The pair $(x, y)\\in\\mathcal{X}\\times \\mathcal{Y}$ is an $\\epsilon$-game-stationary point {\\rm ($\\epsilon$-GS)} of problem \\eqref{eq:problem} if \n% there exists a pair $(u, v)$ such that\n\\[\n\\|\\nabla_{z} d_r(y,x)\\|\\leq\\epsilon \\quad \\text{and} \\quad {\\rm dist}(0,-\\nabla_{y} F(x, y)+\\partial\\iota_{\\mathcal{Y}}(y))\\leq\\epsilon. \\footnote{For any $x \\in \\mathbb{R}^{\\ell}$ and $\\mathcal{S} \\subseteq \\mathbb{R}^{\\ell}$, we use $\\dist(x, \\mathcal{S}):=\\inf\\limits_{z\\in\\mathcal{S}}\\|x-z\\|$ to denote the distance from $x$ to $\\mathcal{S}$.} \n% \\quad \\text { and } \\quad\\|u\\|,\\|v\\| \\leq \\epsilon.\n\\]\n\\end{enumerate}\n% where $1_{\\mathcal{Y}}$ is the indicator function of the set $\\mathcal{Y}$. \n% A pair $(x, y)$ is a stationary solution if $\\mu = 0, \\nu = 0$. \n\\end{definition}\n\n\\begin{equation}\n    \\label{eq:primal_update}\n    x^{k+1} \\in \\mathop{\\argmin}_{x\\in\\mathcal{X}} \\left\\{ F_{x^k,\\lambda}(x,y^k)\n  +\\frac{r}{2}\\|x-z^{k}\\|^2\n    \\right\\},\n\\end{equation}\n\n\\[\nF_{x^k,\\lambda}(x,y^k):=h_{y^k}\\left(c_{y^k}(x^{k})+\\nabla c_{y^k}(x^{k})^\\top(x-x^{k})\\right)+\\frac{1}{2\\lambda}\\|x-x^k\\|^2.\n\\]\n\n\\begin{algorithm}[H]\\label{alg}\n    \\caption{\\textbf{Smoothed PLDA}}\n    \\SetKwInOut{Input}{Input}\n    \\Input{Initial point $(x^0,y^0,z^0)$ and parameters $r>L$, $\\lambda> 0$, $\\alpha>0$, $\\beta\\in(0,1)$}\n    \\For{$k=0,1,2,\\ldots$}{\n    $x^{k+1}:= \\mathop{\\argmin}\\limits_{x \\in \\mathcal{X}} \\left\\{F_{x^k,\\lambda}(x,y^k) + \\frac{r}{2}\\|x-z^k\\|^2\\right\\}$\\\\\n    $y^{k+1}:=\\proj_{\\mathcal{Y}}\\left(y^k+\\alpha \\nabla_{y} F(x^{k+1}, y^k)\\right)$ \\\\\n    $z^{k+1}:=z^k+\\beta(x^{k+1}-z^k)$}\n\\end{algorithm}\n\nWe emphasize that the optimal approach to solving \\eqref{eq:primal_update} depends on the problem's specific structure. For simplicity and to focus on our main contributions, we assume the exactness of the primal update and concentrate on the iteration complexity of the outer loop in subsequent analyses.\n\n\n\\begin{table*}[ht]\n  \\caption{Notation}\n \\centering\n\\begin{tikzpicture}\n \\node[drop shadow,fill=white,inner sep=0pt] \n {\n \\resizebox{0.98\\textwidth}{!}{ \n\\begin{tabular}{|c|c|c|}\n \\hline\n \\textbf{Notation}  & \\textbf{Definition} & \\textbf{Notes} \\\\ \\hline\n $F_r(x,y,z)$   & $F(x,y)+\\frac{r}{2}\\|x-z\\|^2$ & potential function               \\\\ \\hline\n$d_r(y,z)$           & $\\min\\limits_{x \\in \\mathcal{X}}F_r(x,y,z)$ &    dual potential value function      \\\\ \\hline\n$p_r(z)$           & $\\max\\limits_{y \\in \\mathcal{Y}}\\min\\limits_{x \\in \\mathcal{X}}F_r(x,y,z)$ &    proximal function        \\\\ \\hline\n%  \\rowcolor[HTML]{ECF4FF} \n$x_r(y,z)$  & $\\mathop{\\argmin}\\limits_{x \\in \\mathcal{X}}F_r(x,y,z)$  & ---\\\\ \\hline\n$x^\\star_r(z)$   & $\\mathop{\\argmin}\\limits_{x \\in \\mathcal{X}}\\max\\limits_{y \\in \\mathcal{Y}}F_r(x,y,z)$ & ---               \\\\ \\hline\n$Y(z)$     & $\\mathop{\\argmax}\\limits_{y\\in \\mathcal{Y}}d_r(y,z)$ &   $y(z) \\in Y(z)$  \\\\ \\hline\n%  \\rowcolor[HTML]{ECF4FF} \n$y_{+}(z)$    & $\\proj_{\\mathcal{Y}}\\left(y+\\alpha \\nabla_{y} F(x_r(y, z), y)\\right)$  & one-step projected gradient ascent on the dual function\\\\ \\hline\n\\end{tabular}}\n};\n\\end{tikzpicture}\n\\label{tab:notation}\n\\end{table*}\n\n\\[\n\\Phi_r(x,y,z):= \\underbrace{F_r(x,y,z)-d_r(y,z)}_{\\textnormal{Primal Descent}} + \\underbrace{p_r(z)-d_r(y,z)}_{\\textnormal{Dual Ascent}} + \\underbrace{p_r(z)}_{\\textnormal{Proximal Descent}}. \n\\]\n\n\n\\begin{equation}\n\\label{eq:step_dual}\n\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\| \\leq \\omega \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^\\upsilon\n\\end{equation}\n\nfor some constants $\\omega>0$ and $\\upsilon>0$. \n\n\n\\begin{definition}[Weak convexity]\n\\label{defi:weak-convex}\n A function $\\ell:\\R^n \\rightarrow \\R$ is said to be $\\rho$-weakly convex on a set $\\mathcal{X}\\subseteq \\R^n$ for some constant $\\rho \\ge 0$ if for any $x, y \\in \\mathcal{X}$ and $\\tau \\in [0,1]$, we have\n$$\n\\ell(\\tau x+(1-\\tau) y) \\leq \\tau \\ell(x)+(1-\\tau) \\ell(y)+\\frac{\\rho \\tau(1-\\tau)}{2}\\|x-y\\|^{2}.\n$$\n% When $\\ell$ is locally Lipschitz, \nThe above definition is equivalent to the convexity of the function $\\ell(\\cdot) + \\tfrac{\\rho}{2} \\| \\cdot \\|^2$ on $\\mathcal{X}$.\n\\end{definition}\n\n\\begin{definition}[Proximal mapping]\n% Consider a $\\rho$-weakly convex function \nLet $\\ell:\\R^n\\rightarrow \\R \\cup \\{+\\infty\\}$ \n% and fix a parameter $\\mu < \\rho^{-1}$\nbe a proper lower-semicontinuous function.  The proximal mapping of $\\ell$ with parameter $\\mu>0$ at the point $x \\in \\mathbb{R}^n$ is defined by\n\\[\n\\prox_{\\mu \\ell}(x) = \\mathop{\\argmin}_{y\\in \\R^n} \\left\\{ \\ell(y) + \\frac{1}{2\\mu} \\|y-x\\|^2\\right\\}. \n\\]\n\\label{defi:prox}\n\\end{definition}\n\nUsing Definition \\ref{defi:weak-convex} and Fact \\ref{fact-1}, we can deduce that $F_r(\\cdot,y,z)$ is $(r-L)$-strongly convex for any $y \\in \\mathcal{Y}$ and $z \\in \\mathbb{R}^n$.\n\n\nThe following lemma provides a dual error bound for the case where $F(x,\\cdot)$ is concave for any $x \\in \\mathbb{R}^n$. Its proof is based on  Lemma B.10 in \\citep{zhang2020single}.\n","units":[{"start":10315,"end":10428,"start_line":304,"end_line":308,"raw_tex":"\\begin{equation}\n\\label{eq:problem}\n\\min_{x\\in \\mathcal{X}} \\max_{y\\in\\mathcal{Y}} F(x,y).\n\\tag{P}\n\\end{equation}","sha256_utf8":"41a53c209aedb3c19ae918fda10b12b793a7abdea30cf8e8cdd48fce95641c76"},{"start":10429,"end":11139,"start_line":309,"end_line":309,"raw_tex":"Here, $F:\\R^n \\times \\R^d \\rightarrow \\R$ can be both nonconvex with respect to $x$ and nonconcave with respect to $y$, $\\mathcal{X}\\subseteq \\R^n$ is a nonempty closed convex set, and $\\mathcal{Y}\\subseteq \\R^d$ is a nonempty compact convex set. This problem has a wide range of applications in machine learning and operations research. Examples include learning with non-decomposable loss~\\citep{shalev2016minimizing,zhang2018ell_1}, training generative adversarial networks~\\citep{arjovsky2017wasserstein,goodfellow2020generative}, adversarial training~\\citep{madry2017towards,sinha2017certifying}, and (distributionally) robust optimization (DRO)~\\citep{bertsimas2011theory,rahimian2019distributionally}. \n","sha256_utf8":"1e4e7dedcf3236e5d2380907b69a9c1927c638de103a4f5ab55eddb445c6af53"},{"start":32054,"end":33306,"start_line":481,"end_line":501,"raw_tex":"\\begin{assumption}[Problem setup] \\label{ass:basic}\nThe following assumptions on the objective function $F$ of problem \\eqref{eq:problem} hold throughout the paper. \n\\begin{enumerate}[label=\\normalfont(\\alph*)]\n\\item   $(\\textrm{\\bf Primal function})$ For all $y\\in \\mathcal{Y}$, the function $F(\\cdot,y): \\mathbb{R}^n \\rightarrow \\mathbb{R}$ takes the form $F(\\cdot, y):=h_y\\circ c_y(\\cdot)$, where $c_y:\\R^n\\rightarrow \\R^m$ is \ncontinuously differentiable\nwith $L_c$-Lipschitz continuous Jacobian mapping, i.e.,\n\\[\n\\|\\nabla c_y(x)-\\nabla c_y(x')\\| \\leq L_c\\|x-x'\\| \\quad \\text { for all } x, x' \\in \\mathcal{X};\n\\]\nand $h_y:\\R^m\\rightarrow \\R$  is  convex and $L_h$-Lipschitz continuous, i.e.,\n\\[\n|h_y(z)-h_y(z')| \\leq L_h\\|z-z'\\| \\quad \\text { for all } z, z' \\in \\R^{m}.\n\\]\n\\item  $(\\textrm{\\bf Dual function})$ For all $x\\in \\mathcal{X}$, the function $F(x, \\cdot):\\R^d \\rightarrow \\R$ is continuously differentiable  with $\\nabla_y F(\\cdot,\\cdot)$ being $L$-Lipschitz continuous on $\\mathcal{X}\\times \\mathcal{Y}$, i.e., \n\\[\n\\|\\nabla_y F(x,y)-\\nabla_y F(x',y')\\| \\leq L\\|(x,y)-(x',y')\\| \\,\n\\text{ for all }  (x,y), (x',y') \\in \\mathcal{X}\\times\\mathcal{Y}.\n\\]\nWithout loss of generality, we may take $L=L_h L_c$.\n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"431e6386f7b15b64d25ba45b08423b0976018f935a3e14c24aeb9d4e6c45accb"},{"start":33326,"end":33977,"start_line":503,"end_line":512,"raw_tex":"\\begin{assumption}[K\\L{} property with exponent $\\theta$ for dual problem]\n\\label{ass:kl}\nFor all  $x\\in\\mathcal{X}$, the problem $\\max_{y\\in \\mathcal{Y}} F(x,y)$ has a nonempty solution set and a finite optimal value. Moreover, there exist $\\mu>0$ and $\\theta \\in [0,1)$ such that  \n\\[\n\\dist(0, -\\nabla_y F(x,y)+\\partial\\iota_{\\mathcal{Y}}(y)) \\ge \\mu  \\left(\\max\\limits_{y'\\in \\mathcal{Y}} F(x,y')-F(x,y)\\right)^{\\theta}\n\\]\nfor all $x \\in \\mathcal{X}$ and  $y \\in \\mathcal{Y}\\setminus \\mathcal{Y}^\\star(x)$, where $\\mathcal{Y}^\\star(x):= \\argmax_{y' \\in \\mathcal{Y}} F(x,y')$ \n% satisfying $F(x,y)<\\max_{y'\\in\\mathcal{Y}} F(x,y')$\n.\n\\end{assumption}","sha256_utf8":"32db9ebc090f408ba374e09d5d78e03361990fd8bf847391c9086570d3662925"},{"start":34648,"end":35295,"start_line":517,"end_line":524,"raw_tex":"Let us now examine the stationarity measures discussed in this paper. We define the value function $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$, the potential function $F_r:\\R^n\\times \\R^d\\times \\R^n\\rightarrow\\R$, and the dual potential value function $d_r:\\R^d\\times \\R^n\\rightarrow \\R$ by\n% \\[\n% F_r(x,y,z) := F(x,y)+\\frac{r}{2}\\|x-z\\|^2\\quad \\text{and}\\quad d_r(y,z):=\\min_{x\\in\\mathcal{X}} F_r(x,y,z),\n% \\]\n\\begin{align*}\nf(x) := \\max_{y \\in \\mathcal{Y}} F(x,y),\\quad F_r(x,y,z) := F(x,y) + \\frac{r}{2} \\| x-z \\|^2,\\quad d_r(y,z) := \\min_{x \\in \\mathcal{X}} F_r(x,y,z),\n\\end{align*}\nrespectively, where we assume $r> L$ in the rest of this paper.\n","sha256_utf8":"5b8a00e496ef108c89659f0b4d70e5bb3bf3e71db938544692e5094dc50016ce"},{"start":36155,"end":37281,"start_line":531,"end_line":546,"raw_tex":"\\begin{definition}[Stationarity measures]\n\\label{defi:primal-dual}\nLet $\\epsilon \\ge 0$ be given.\n\\begin{enumerate}[label=\\normalfont(\\alph*)]\n\\item The point $x\\in\\mathcal{X}$ is an $\\epsilon$-optimization-stationary point {\\rm($\\epsilon$-OS)} of problem \\eqref{eq:problem} if \n\\[\\|\\prox_{\\frac{1}{r}f+\\iota_{\\mathcal{X}}}(x)-x\\|\\leq \\epsilon.\\]\n\\item The pair $(x, y)\\in\\mathcal{X}\\times \\mathcal{Y}$ is an $\\epsilon$-game-stationary point {\\rm ($\\epsilon$-GS)} of problem \\eqref{eq:problem} if \n% there exists a pair $(u, v)$ such that\n\\[\n\\|\\nabla_{z} d_r(y,x)\\|\\leq\\epsilon \\quad \\text{and} \\quad {\\rm dist}(0,-\\nabla_{y} F(x, y)+\\partial\\iota_{\\mathcal{Y}}(y))\\leq\\epsilon. \\footnote{For any $x \\in \\mathbb{R}^{\\ell}$ and $\\mathcal{S} \\subseteq \\mathbb{R}^{\\ell}$, we use $\\dist(x, \\mathcal{S}):=\\inf\\limits_{z\\in\\mathcal{S}}\\|x-z\\|$ to denote the distance from $x$ to $\\mathcal{S}$.} \n% \\quad \\text { and } \\quad\\|u\\|,\\|v\\| \\leq \\epsilon.\n\\]\n\\end{enumerate}\n% where $1_{\\mathcal{Y}}$ is the indicator function of the set $\\mathcal{Y}$. \n% A pair $(x, y)$ is a stationary solution if $\\mu = 0, \\nu = 0$. \n\\end{definition}","sha256_utf8":"c81daf33b7ca14ae00947d607130e7a98055c91267f5f346f316e8fc269f67f3"},{"start":38861,"end":39045,"start_line":558,"end_line":563,"raw_tex":"\\begin{equation}\n    \\label{eq:primal_update}\n    x^{k+1} \\in \\mathop{\\argmin}_{x\\in\\mathcal{X}} \\left\\{ F_{x^k,\\lambda}(x,y^k)\n  +\\frac{r}{2}\\|x-z^{k}\\|^2\n    \\right\\},\n\\end{equation}","sha256_utf8":"4b468df5f339c31684935559aa35cef678d1c26cc09a6abd9b36646a68f518d4"},{"start":39113,"end":39244,"start_line":566,"end_line":568,"raw_tex":"\\[\nF_{x^k,\\lambda}(x,y^k):=h_{y^k}\\left(c_{y^k}(x^{k})+\\nabla c_{y^k}(x^{k})^\\top(x-x^{k})\\right)+\\frac{1}{2\\lambda}\\|x-x^k\\|^2.\n\\]","sha256_utf8":"8f956515a758592912d779a050d64678ad711e6f9107b934d20c8a1331a2d494"},{"start":39652,"end":40152,"start_line":575,"end_line":583,"raw_tex":"\\begin{algorithm}[H]\\label{alg}\n    \\caption{\\textbf{Smoothed PLDA}}\n    \\SetKwInOut{Input}{Input}\n    \\Input{Initial point $(x^0,y^0,z^0)$ and parameters $r>L$, $\\lambda> 0$, $\\alpha>0$, $\\beta\\in(0,1)$}\n    \\For{$k=0,1,2,\\ldots$}{\n    $x^{k+1}:= \\mathop{\\argmin}\\limits_{x \\in \\mathcal{X}} \\left\\{F_{x^k,\\lambda}(x,y^k) + \\frac{r}{2}\\|x-z^k\\|^2\\right\\}$\\\\\n    $y^{k+1}:=\\proj_{\\mathcal{Y}}\\left(y^k+\\alpha \\nabla_{y} F(x^{k+1}, y^k)\\right)$ \\\\\n    $z^{k+1}:=z^k+\\beta(x^{k+1}-z^k)$}\n\\end{algorithm}","sha256_utf8":"3f8d4cd468c2539d16af3842b770925ff3f55896350e1a8031d28361f0371dcf"},{"start":40890,"end":41196,"start_line":588,"end_line":588,"raw_tex":"We emphasize that the optimal approach to solving \\eqref{eq:primal_update} depends on the problem's specific structure. For simplicity and to focus on our main contributions, we assume the exactness of the primal update and concentrate on the iteration complexity of the outer loop in subsequent analyses.\n","sha256_utf8":"299de7e0ec9f20164fc88daabe94a46579e0d40a641c348a1a729e2bc9a17ada"},{"start":42895,"end":44095,"start_line":614,"end_line":637,"raw_tex":"\\begin{table*}[ht]\n  \\caption{Notation}\n \\centering\n\\begin{tikzpicture}\n \\node[drop shadow,fill=white,inner sep=0pt] \n {\n \\resizebox{0.98\\textwidth}{!}{ \n\\begin{tabular}{|c|c|c|}\n \\hline\n \\textbf{Notation}  & \\textbf{Definition} & \\textbf{Notes} \\\\ \\hline\n $F_r(x,y,z)$   & $F(x,y)+\\frac{r}{2}\\|x-z\\|^2$ & potential function               \\\\ \\hline\n$d_r(y,z)$           & $\\min\\limits_{x \\in \\mathcal{X}}F_r(x,y,z)$ &    dual potential value function      \\\\ \\hline\n$p_r(z)$           & $\\max\\limits_{y \\in \\mathcal{Y}}\\min\\limits_{x \\in \\mathcal{X}}F_r(x,y,z)$ &    proximal function        \\\\ \\hline\n%  \\rowcolor[HTML]{ECF4FF} \n$x_r(y,z)$  & $\\mathop{\\argmin}\\limits_{x \\in \\mathcal{X}}F_r(x,y,z)$  & ---\\\\ \\hline\n$x^\\star_r(z)$   & $\\mathop{\\argmin}\\limits_{x \\in \\mathcal{X}}\\max\\limits_{y \\in \\mathcal{Y}}F_r(x,y,z)$ & ---               \\\\ \\hline\n$Y(z)$     & $\\mathop{\\argmax}\\limits_{y\\in \\mathcal{Y}}d_r(y,z)$ &   $y(z) \\in Y(z)$  \\\\ \\hline\n%  \\rowcolor[HTML]{ECF4FF} \n$y_{+}(z)$    & $\\proj_{\\mathcal{Y}}\\left(y+\\alpha \\nabla_{y} F(x_r(y, z), y)\\right)$  & one-step projected gradient ascent on the dual function\\\\ \\hline\n\\end{tabular}}\n};\n\\end{tikzpicture}\n\\label{tab:notation}\n\\end{table*}","sha256_utf8":"16d7b502bb3550827bb97e0ff518f3b12f2cf92fcc2da9430d0172478f415ce6"},{"start":44475,"end":44674,"start_line":646,"end_line":648,"raw_tex":"\\[\n\\Phi_r(x,y,z):= \\underbrace{F_r(x,y,z)-d_r(y,z)}_{\\textnormal{Primal Descent}} + \\underbrace{p_r(z)-d_r(y,z)}_{\\textnormal{Dual Ascent}} + \\underbrace{p_r(z)}_{\\textnormal{Proximal Descent}}. \n\\]\n","sha256_utf8":"9628e85a77c62ae5dc2d224d50815d44a5ec406e91622d71b3c1086fa85167b4"},{"start":48694,"end":48858,"start_line":693,"end_line":696,"raw_tex":"\\begin{equation}\n\\label{eq:step_dual}\n\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\| \\leq \\omega \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^\\upsilon\n\\end{equation}","sha256_utf8":"455ccc81ada05f6a918584d0401d99e8489a5316bff55c7f68c303efc415ad22"},{"start":48859,"end":48908,"start_line":697,"end_line":697,"raw_tex":"for some constants $\\omega>0$ and $\\upsilon>0$. \n","sha256_utf8":"6c2e3961090c52189d87e0229cb4b9100f3ea726c2ffcd48fbfa6e3a41e6fa41"},{"start":134121,"end":134677,"start_line":1725,"end_line":1733,"raw_tex":"\\begin{definition}[Weak convexity]\n\\label{defi:weak-convex}\n A function $\\ell:\\R^n \\rightarrow \\R$ is said to be $\\rho$-weakly convex on a set $\\mathcal{X}\\subseteq \\R^n$ for some constant $\\rho \\ge 0$ if for any $x, y \\in \\mathcal{X}$ and $\\tau \\in [0,1]$, we have\n$$\n\\ell(\\tau x+(1-\\tau) y) \\leq \\tau \\ell(x)+(1-\\tau) \\ell(y)+\\frac{\\rho \\tau(1-\\tau)}{2}\\|x-y\\|^{2}.\n$$\n% When $\\ell$ is locally Lipschitz, \nThe above definition is equivalent to the convexity of the function $\\ell(\\cdot) + \\tfrac{\\rho}{2} \\| \\cdot \\|^2$ on $\\mathcal{X}$.\n\\end{definition}","sha256_utf8":"956648c4626d0065f1d71763ecbcf583023121c866b4888f4b844139ee9c1f32"},{"start":134678,"end":135137,"start_line":1734,"end_line":1743,"raw_tex":"\\begin{definition}[Proximal mapping]\n% Consider a $\\rho$-weakly convex function \nLet $\\ell:\\R^n\\rightarrow \\R \\cup \\{+\\infty\\}$ \n% and fix a parameter $\\mu < \\rho^{-1}$\nbe a proper lower-semicontinuous function.  The proximal mapping of $\\ell$ with parameter $\\mu>0$ at the point $x \\in \\mathbb{R}^n$ is defined by\n\\[\n\\prox_{\\mu \\ell}(x) = \\mathop{\\argmin}_{y\\in \\R^n} \\left\\{ \\ell(y) + \\frac{1}{2\\mu} \\|y-x\\|^2\\right\\}. \n\\]\n\\label{defi:prox}\n\\end{definition}","sha256_utf8":"aaa7e46475b4866439c6580347dce9315e15bee47a1ec175542c3d52d2f498b0"},{"start":135706,"end":135886,"start_line":1754,"end_line":1754,"raw_tex":"Using Definition \\ref{defi:weak-convex} and Fact \\ref{fact-1}, we can deduce that $F_r(\\cdot,y,z)$ is $(r-L)$-strongly convex for any $y \\in \\mathcal{Y}$ and $z \\in \\mathbb{R}^n$.\n","sha256_utf8":"f6b90b9bb8b505a28f614ee750e9c942faffc3cc3109b17b39c64a568c37f583"},{"start":163231,"end":163414,"start_line":2202,"end_line":2202,"raw_tex":"The following lemma provides a dual error bound for the case where $F(x,\\cdot)$ is concave for any $x \\in \\mathbb{R}^n$. Its proof is based on  Lemma B.10 in \\citep{zhang2020single}.\n","sha256_utf8":"6ba7de2b8652f710587cca8db48accb1704676af90c3b5eb0e39517f2ebd555a"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}[Main theorem]\n\\label{thm:general}\n % Suppose that \n % \\[r\\ge3L,\\ \\lambda\\le \\frac{1}{L},\\ \\alpha \\leq \\min\\left\\{\\frac{1}{10L},\\ \\frac{1}{4L\\zeta^2} \\right\\},\\ \\beta  \\leq \\min \\left\\{\\frac{1}{28},\\ \\frac{(r-L)^2}{14\\alpha r(2r-L)^2}\\right\\}.\\] \n Under the setting of Proposition \\ref{prop:decrease}, for any integer $K>0$, there exists an index $k \\in\\{1,2, \\ldots, K\\}$ such that the following hold:\n \\begin{enumerate}[label=\\normalfont(\\alph*)]\n   \\item \\rm{(Concave):} Suppose that $F(x,\\cdot)$ is concave for all $x \\in \\mathcal{X}$. If $\\max_{y\\in \\mathcal{Y}} F(\\cdot,y)$ is bounded below on $\\mathcal{X}$ and $\\beta\\leq \\mathcal{O}(K^{-\\frac{1}{2}})$,  then $\\left(x^{k+1}, y^{k+1}\\right)$ is an $\\mathcal{O}(K^{-\\frac{1}{4}})$-GS and $z^{k+1}$ is an $\\mathcal{O}(K^{-\\frac{1}{4}})$-OS of problem \\eqref{eq:problem}.\n  \\item \\rm{(K\\L\\ exponent $\\theta\\in(\\frac{1}{2},1)$):} Suppose that Assumption \\ref{ass:kl} holds with $\\theta \\in (\\tfrac{1}{2},1)$. If $\\mathcal{X}$ is compact and  $ \\beta\\leq \\mathcal{O}(K^{-\\frac{2\\theta-1}{2\\theta}})$,\n   then $\\left(x^{k+1}, y^{k+1}\\right)$ is an $\\mathcal{O}( K^{-\\frac{1}{4\\theta}})$-GS and $z^{k+1}$ is an $\\mathcal{O}( K^{-\\frac{1}{4\\theta}})$-OS  of problem \\eqref{eq:problem}.\n  \\item \\rm{(K\\L\\ exponent $\\theta\\in[0,\\frac{1}{2}]$):} Suppose that Assumption \\ref{ass:kl} holds with $\\theta \\in [0, \\tfrac{1}{2}]$.\n If $\\mathcal{X}$ is compact and  $\\beta  \\leq \\frac{ {\\rm diam}(\\mathcal{Y})^{\\frac{2\\theta-1}{\\theta}}}{224 \\alpha r\\omega_2^2}$ when $\\theta \\in (0,\\tfrac{1}{2}]$ {\\rm (}resp., $\\beta  \\leq \\tfrac{ 1}{224 \\alpha r\\omega_1^2}$ when $\\theta=0${\\rm )}, then $\\left(x^{k+1}, y^{k+1}\\right)$ is an $\\mathcal{O}(K^{-\\frac{1}{2}})$-GS and $z^{k+1}$ is an $\\mathcal{O}(K^{-\\frac{1}{2}})$-OS of problem \\eqref{eq:problem}.\n  \\end{enumerate}\n  \n\\end{theorem}","units":[{"start":73033,"end":74871,"start_line":1020,"end_line":1033,"raw_tex":"\\begin{theorem}[Main theorem]\n\\label{thm:general}\n % Suppose that \n % \\[r\\ge3L,\\ \\lambda\\le \\frac{1}{L},\\ \\alpha \\leq \\min\\left\\{\\frac{1}{10L},\\ \\frac{1}{4L\\zeta^2} \\right\\},\\ \\beta  \\leq \\min \\left\\{\\frac{1}{28},\\ \\frac{(r-L)^2}{14\\alpha r(2r-L)^2}\\right\\}.\\] \n Under the setting of Proposition \\ref{prop:decrease}, for any integer $K>0$, there exists an index $k \\in\\{1,2, \\ldots, K\\}$ such that the following hold:\n \\begin{enumerate}[label=\\normalfont(\\alph*)]\n   \\item \\rm{(Concave):} Suppose that $F(x,\\cdot)$ is concave for all $x \\in \\mathcal{X}$. If $\\max_{y\\in \\mathcal{Y}} F(\\cdot,y)$ is bounded below on $\\mathcal{X}$ and $\\beta\\leq \\mathcal{O}(K^{-\\frac{1}{2}})$,  then $\\left(x^{k+1}, y^{k+1}\\right)$ is an $\\mathcal{O}(K^{-\\frac{1}{4}})$-GS and $z^{k+1}$ is an $\\mathcal{O}(K^{-\\frac{1}{4}})$-OS of problem \\eqref{eq:problem}.\n  \\item \\rm{(K\\L\\ exponent $\\theta\\in(\\frac{1}{2},1)$):} Suppose that Assumption \\ref{ass:kl} holds with $\\theta \\in (\\tfrac{1}{2},1)$. If $\\mathcal{X}$ is compact and  $ \\beta\\leq \\mathcal{O}(K^{-\\frac{2\\theta-1}{2\\theta}})$,\n   then $\\left(x^{k+1}, y^{k+1}\\right)$ is an $\\mathcal{O}( K^{-\\frac{1}{4\\theta}})$-GS and $z^{k+1}$ is an $\\mathcal{O}( K^{-\\frac{1}{4\\theta}})$-OS  of problem \\eqref{eq:problem}.\n  \\item \\rm{(K\\L\\ exponent $\\theta\\in[0,\\frac{1}{2}]$):} Suppose that Assumption \\ref{ass:kl} holds with $\\theta \\in [0, \\tfrac{1}{2}]$.\n If $\\mathcal{X}$ is compact and  $\\beta  \\leq \\frac{ {\\rm diam}(\\mathcal{Y})^{\\frac{2\\theta-1}{\\theta}}}{224 \\alpha r\\omega_2^2}$ when $\\theta \\in (0,\\tfrac{1}{2}]$ {\\rm (}resp., $\\beta  \\leq \\tfrac{ 1}{224 \\alpha r\\omega_1^2}$ when $\\theta=0${\\rm )}, then $\\left(x^{k+1}, y^{k+1}\\right)$ is an $\\mathcal{O}(K^{-\\frac{1}{2}})$-GS and $z^{k+1}$ is an $\\mathcal{O}(K^{-\\frac{1}{2}})$-OS of problem \\eqref{eq:problem}.\n  \\end{enumerate}\n  \n\\end{theorem}","sha256_utf8":"34ab601eebc9204c5cf3fa9e38d8e96cb1429710a4ebbc9be17efc41c3f131a7"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{proposition}[Basic descent estimate of $\\Phi_r$]\n\\label{prop:decrease}\n    Let \n    % $r\\ge3L$, $\\lambda\\le \\frac{1}{L}$, $\\alpha \\leq \\min\\left\\{\\frac{1}{10L}, \\frac{1}{4L\\zeta^2} \\right\\}$, and $\\beta  \\leq \\min \\left\\{\\frac{1}{28}, \\frac{(r-L)^2}{14\\alpha r(2r-L)^2}\\right\\}$,     \n    \\[r\\ge3L,\\ \\lambda\\le \\frac{1}{L},\\ \\alpha \\leq \\min\\left\\{\\frac{1}{10L},\\ \\frac{1}{4L\\zeta^2} \\right\\},\\ \\beta  \\leq \\min \\left\\{\\frac{1}{28},\\ \\frac{(r-L)^2}{14\\alpha r(2r-L)^2}\\right\\},\\] \n    where $\\zeta>0$ is the constant defined in Proposition \\ref{prop:lip}.\n    Then, for any $k\\ge0$, we have\n\\begin{equation*}\n\\begin{aligned}\n\\Phi_r^k-\\Phi_r^{k+1}\\ge\\ &  \\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{8\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 +\\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2} \\\\\n& - 14r\\beta\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2},\n\\end{aligned}\n\\end{equation*}\nwhere $\\Phi_r^k:= \\Phi_r(x^k,y^k,z^k)$. \n% \\footnote{The proof of Proposition \\ref{prop:decrease}, which is based on the primal descent, dual ascent, and proximal descent properties, is given in Appendix \\ref{sec:suff_decrease}. }\n\\end{proposition}\n\n\\begin{proposition} [Lipschitz-type primal error bound]\n\\label{prop:lip}\n%  Suppose that $r-L > 0$. Then \n For any $k\\ge0$, we have\n    \\begin{equation}\\label{iter-result1}\n        \\|x^{k+1}-x_r(y^{k}, z^{k})\\| \\leq \\zeta\\|x^{k}-x^{k+1}\\|,\n        \\end{equation}\nwhere   $\\zeta: =\\frac{2(r-L)^{-1}+(\\lambda^{-1}+L) ^{-1}}{(\\lambda^{-1}+L) ^{-1}} \\left(\\sqrt{\\frac{2L}{\\lambda^{-1}+L}}+1\\right)$.\n% and $\\eta:=\\alpha L \\zeta$. \n\\end{proposition}\n\n\\begin{proof}\nFor ease of notation, we define $\\hat{F}_{y,z}(\\cdot): = F_r(\\cdot,y,z) + \\iota_{\\mathcal{X}}(\\cdot)$ for any $y\\in\\mathcal{Y}$ and $z\\in\\R^n$.  As $x_r(y,z)$ is the optimal solution of $\\min_{x \\in \\mathbb{R}^n} \\hat{F}_{y,z} (x)$ and $F_r(\\cdot,y,z)$ is $(r-L)$-strongly convex (see the remark after Fact \\ref{fact-2} in Appendix \\ref{sec:lemmas}), we have\n\\begin{equation}\n    \\label{eq:lip_1} \n    \\hat{F}_{y^k,z^k}(x)-\\hat{F}_{y^k,z^k}(x_r(y^k,z^k)) \\ge \\frac{r-L}{2} \\|x-x_r(y^k,z^k)\\|^2 \\quad \\text{for all}\\ x \\in \\mathcal{X}.\n\\end{equation}\nIn addition, by the convexity of $\\hat{F}_{y^k,z^k}$, we obtain \n\\begin{equation}\n    \\label{eq:lip_2}\n    \\begin{aligned}\n    \\hat{F}_{y^k,z^k}(x)-\\hat{F}_{y^k,z^k}(x_r(y^k,z^k)) & \\leq \\dist(0, \\partial \\hat{F}_{y^k,z^k}(x))\\cdot \\|x-x_r(y^k,z^k)\\| \\\\ \n    \\end{aligned}\n\\end{equation}\n$\\text{for all}\\ x\\in\\mathcal{X}$.\nCombining \\eqref{eq:lip_1} and \\eqref{eq:lip_2} yields\n\\[\n\\|x-x_r(y^k,z^k)\\| \\leq \\frac{2}{r-L} \\dist(0, \\partial \\hat{F}_{y^k,z^k}(x)) \\quad \\text{for all}\\ x\\in\\mathcal{X}. \n\\]\nBy utilizing the equivalence between proximal and subdifferential error bounds \\citep[Theorem 3.4]{drusvyatskiy2018error}, we  conclude that\n\\begin{equation}\n\\label{eq:;lip_3}\n    \\|x-x_r(y^k,z^k)\\|  \\leq \\frac{2(r-L)^{-1}+t}{t} \\left\\|x - \\prox_{t  \\hat{F}_{y^k,z^k}}(x)\\right\\| \\text{ for all } t>0, \\, x\\in\\mathcal{X}.\n\\end{equation}\nNow, let us examine the relationship between  $\\|x^{k+1}-x^k\\|$ and $\\|x^{k+1} - \\prox_{t \\hat{F}_{y^k,z^k}}(x^{k+1})\\|$. Let $\\varphi_k:\\R^n\\rightarrow \\R \\cup \\{+\\infty\\}$ be the function defined by\n\\[\n\\varphi_k(x):=\\hat{F}_{y^k,z^k}(x)+\\frac{\\lambda^{-1}+L}{2}\\|x-x^k\\|^{2}-\\frac{\\lambda^{-1} + L}{2}\\|x-x^{k+1}\\|^{2}.\n\\]\n% By combining Fact \\ref{fact-2} and the strong convexity of $F_{x^k,\\lambda}(\\cdot,y^k)$\nIt is clear that for any $x\\in\\mathcal{X}$,\n\\begin{align*}\\label{safeguard-key1}\n            & \\hat{F}_{y^k,z^k}(x)\n             = F(x,y^k) + \\frac{r}{2}\\|x-z^k\\|^2 \\notag\\\\\n             \\ge\\ & F_{x^k,\\lambda}(x,y^k) -\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k}\\|^2 + \\frac{r}{2}\\|x-z^k\\|^2\\notag \\\\ \n              \\ge\\ &  F_{x^k,\\lambda}(x^{k+1},y^k) +\\frac{\\lambda^{-1}+r}{2}\\|x-x^{k+1}\\|^2 -\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k}\\|^2 +\\frac{r}{2}\\|x^{k+1}-z^k\\|^2\\notag\\\\\n             \\ge\\ & F(x^{k+1},y^k) + \\frac{\\lambda^{-1}-L}{2}\\|x^{k}-x^{k+1}\\|^2 +\\frac{\\lambda^{-1}+r}{2}\\|x-x^{k+1}\\|^2\\ \\\\\n             & \\ -\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k}\\|^2 + \\frac{r}{2}\\|x^{k+1}-z^k\\|^2 \\notag \\\\ \n             =\\ & \\hat{F}_{y^k,z^k}(x^{k+1})+\\frac{\\lambda^{-1}-L}{2}\\|x^{k+1}-x^k\\|^2 +\\frac{\\lambda^{-1}+r}{2}\\|x-x^{k+1}\\|^2 -\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k}\\|^2,\n\\end{align*}\nwhere the first and third inequalities follow from Fact \\ref{fact-2}, and the second one is from the $(\\lambda^{-1} + r)$-strong convexity of $F_{x^k,\\lambda}(\\cdot,y^k) +\\frac{r}{2}\\|\\cdot-z^k\\|^2$ on $\\mathcal{X}$ and the definition of $x^{k+1}$.\nSince $r > L$, we see that for any $x\\in\\mathcal{X}$,\n\\begin{align*}\n\\varphi_k(x)\n&\\ge \\hat{F}_{y^k,z^k}(x^{k+1})+\\frac{\\lambda^{-1}-L}{2}\\|x^{k+1}-x^{k}\\|^2+\\frac{r-L}{2}\\|x-x^{k+1}\\|^2\\notag\\\\\n&\\ge\\hat{F}_{y^k,z^k}(x^{k+1})+\\frac{\\lambda^{-1}-L}{2}\\|x^{k+1}-x^{k}\\|^2.\n\\end{align*}\nIt then follows from the definition of $\\varphi_k$ that \n\\[\\varphi_k(x^{k+1})-\\inf_{x\\in\\R^n} \\varphi_k(x)\\leq L \\|x^{k+1}-x^k\\|^2.\\]\nThis, together with the definition of the proximal mapping, implies that for any $\\rho>0$,\n\\begin{align}\n&\\varphi_k\\left(\\operatorname{prox}_{\\frac{1}{\\rho} \\varphi_k}(x^{k+1})\\right)+\\frac{\\rho}{2}\\|\\operatorname{prox}_{\\frac{1}{\\rho} \\varphi_k}(x^{k+1})-x^{k+1}\\|^{2}\\leq  \\varphi_k(x^{k+1})\\notag\\\\\n\\leq\\ & \\inf_{x\\in\\R^n} \\varphi_k(x)+L \\|x^{k+1}-x^k\\|^2\n\\leq \\varphi_k\\left(\\operatorname{prox}_{\\frac{1}{\\rho} \\varphi_k}(x^{k+1})\\right)+L \\|x^{k+1}-x^k\\|^2,\\notag\n\\end{align}\nwhich in turn implies that\n\\begin{equation}\\label{safeguard-key111}\n\\|\\operatorname{prox}_{\\frac{1}{\\rho} \\varphi_k}(x^{k+1})-x^{k+1}\\|\\leq \\sqrt{\\frac{2L}{\\rho}} \\|x^{k+1}-x^k\\|.    \n\\end{equation}\nRecall that our goal is to find a relationship between  $\\|x^{k+1}-x^k\\|$ and $\\|x^{k+1} - \\prox_{t \\hat{F}_{y^k,z^k}}(x^{k+1})\\|$. To do this, it suffices to relate $\\prox_{t \\hat{F}_{y^k,z^k}}(x^{k+1})$ to $\\operatorname{prox}_{\\frac{1}{\\rho} \\varphi_k}(x^{k+1})$.  By definition, we have\n\\[\\hat{F}_{y^k,z^k}(x)+\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k}\\|^{2}=\\varphi_k(x)+\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k+1}\\|^{2},\\]\nwhich implies that $\\operatorname{prox}_{\\frac{1}{\\lambda^{-1}+L} \\varphi_k}(x^{k+1})=\\operatorname{prox}_{\\frac{1 }{\\lambda^{-1}+L}\\hat{F}_{y^k,z^k}}(x^k)$. \nBy setting $t = (\\lambda^{-1} + L)^{-1}$ in \\eqref{eq:;lip_3} and $\\rho = \\lambda^{-1} + L$ in \\eqref{safeguard-key111}, we conclude that\n\\begin{align*}\n    &\\|x^{k+1}-x_r(y^{k}, z^{k})\\| \\notag\\\\\n    \\leq\\ & \\frac{2(r-L)^{-1}+(\\lambda^{-1}+L) ^{-1}}{(\\lambda^{-1}+L) ^{-1}}\\left\\|x^{k+1}- \\prox_{\\frac{1}{\\lambda^{-1}+L} \\hat{F}_{y^k,z^k}}(x^{k+1})\\right\\| \\\\\n    \\leq\\ &   \\frac{2(r-L)^{-1}+(\\lambda^{-1}+L) ^{-1}}{(\\lambda^{-1}+L) ^{-1}}\\left(\\left\\|x^{k+1}- \\prox_{\\frac{1}{\\lambda^{-1}+L} \\hat{F}_{y^k,z^k}}(x^{k})\\right\\|+ \\|x^{k+1}-x^k\\|\\right) \\\\\n     =\\ &   \\frac{2(r-L)^{-1}+(\\lambda^{-1}+L) ^{-1}}{(\\lambda^{-1}+L) ^{-1}}\\left(\\left\\|x^{k+1}- \\prox_{\\frac{1}{\\lambda^{-1}+L} \\varphi_k}(x^{k+1})\\right\\|+ \\|x^{k+1}-x^k\\|\\right) \\\\\n    \\leq\\ & \\frac{2(r-L)^{-1}+(\\lambda^{-1}+L) ^{-1}}{(\\lambda^{-1}+L) ^{-1}} \\left(\\sqrt{\\frac{2L}{\\lambda^{-1}+L}}+1\\right) \\|x^{k+1}-x^k\\|,\n\\end{align*}\nwhere the second inequality is due to the nonexpansiveness of the proximal mapping. The proof is complete.\n\\end{proof}\n\n\\begin{proposition}[Dual error bound with K\\L{} exponent] \n\\label{prop:dual_eb_KL}\n% Let $\\delta>0$. \nSuppose that Assumption \\ref{ass:kl} holds. Then, for any $y\\in\\mathcal{Y}$ and $z \\in \\mathbb{R}^n$, we have\n% \\[\n% \\|x_r^\\star(z)-x_r(y_+(z),z)\\| \\leq \\omega \\|y-y_+(z)\\|^{\\frac{1}{2\\theta}},\n% \\]\n% where $\\omega:=\\frac{\\sqrt{2}}{\\sqrt{r-L}}\\left(\\frac{1+\\alpha L(1+\\sigma_2)}{\\alpha\\mu}\\right)^{\\frac{1}{2\\theta}}$.\n\\begin{enumerate}[label=\\normalfont(\\alph*)]\n \\item \\rm{(K\\L\\ exponent $\\theta=0$):} $\\|x_r(y(z),z)-x_r(y_+(z),z)\\| \\leq \\omega_1 \\|y-y_+(z)\\|$, \n  \\item \\rm{(K\\L\\ exponent $\\theta\\in(0,1)$):} $\\|x_r(y(z),z)-x_r(y_+(z),z)\\| \\leq \\omega_2\\|y-y_+(z)\\|^{\\frac{1}{2\\theta}}$, \n  \\end{enumerate}\n    where $\\omega_1:= \\frac{\\sqrt{2L\\cdot {\\rm diam}(\\mathcal{Y})}\\cdot (1+\\alpha L)}{\\alpha\\mu\\sqrt{r-L}}$ and $\\omega_2:=\\frac{\\sqrt{2}}{\\sqrt{r-L}}\\left(\\frac{1+\\alpha L(1+\\sigma_2)}{\\alpha\\mu}\\right)^{\\frac{1}{2\\theta}}$ (recall that $y_+(z) = \\proj_{\\mathcal{Y}} ( y + \\alpha \\nabla_y F(x_r(y,z), y)$).  \n\\end{proposition}\n\n\\begin{proof}\nLet $\\psi:\\R^n\\times \\R^n\\rightarrow\\R$ be the function defined by\n\\[\n\\psi(x,z) =  F_r(x,y(z),z). \n\\]\nConsider arbitrary $x \\in \\mathcal{X}$, $y \\in \\mathcal{Y}$, and $z \\in \\mathbb{R}^n$. Note that the function $\\psi(\\cdot,z)$ is $(r-L)$-strongly convex. Since\n\\begin{align*}\n\\mathop{\\argmin}_{x'\\in\\mathcal{X}}\\psi(x',z) & =\\mathop{\\argmin}_{x'\\in\\mathcal{X}} F_r(x',y(z),z) =  x_r(y(z),z),\n\\end{align*}\nwe see that \n% for any $x\\in \\mathcal{X}$, \n\\begin{equation}\n\\label{eq:dual1}\n\\psi(x,z) - \\psi(x_r(y(z),z),z) \\ge \\frac{r-L}{2}\\|x-x_r(y(z),z)\\|^2. \n\\end{equation}\nIn addition, we have\n\\begin{align} \n\\label{eq:dual2}\n&\\psi(x,z) - \\psi(x_r(y(z),z),z)\\notag\\\\   \n\\leq \\  &\\psi(x,z)-F_r(x_r(y_+(z),z),y_+(z),z)\\notag \\\\\n\\leq \\ & \\max_{y^\\prime\\in \\mathcal{Y}} F(x,y^\\prime) + \\frac{r}{2}\\|x-z\\|^2-F_r(x_r(y_+(z),z),y_+(z),z)\\notag\\\\\n=\\,& \\max_{y^\\prime\\in \\mathcal{Y}}F(x,y^\\prime)-F(x_r(y_+(z),z),y_+(z)) + \\frac{r}{2}\\|x-z\\|^2-\\frac{r}{2}\\|x_r(y_+(z),z)-z\\|^2,\n\\end{align}\nwhere the first inequality follows from\n\\begin{align*}\nF_r(x_r(y_+(z),z),y_+(z),z) \n=\\ & \\min_{x'\\in\\mathcal{X}}\\left\\{ F(x',y_+(z)) +\\frac{r}{2}\\|x'-z\\|^2\\right\\} \\nonumber \\\\\n\\leq\\ & \\max_{y' \\in \\mathcal{Y}}\\min_{x'\\in\\mathcal{X}}\\left\\{F(x',y') +\\frac{r}{2}\\|x'-z\\|^2\\right\\} \\nonumber \\\\\n=\\ & \\min_{x'\\in\\mathcal{X}}\\left\\{F(x',y(z)) +\\frac{r}{2}\\|x'-z\\|^2\\right\\} \\nonumber \\\\\n=\\ & \\min_{x'\\in\\mathcal{X}} \\psi(x',z)\n= \\psi(x_r(y(z),z),z). % \\label{eq:Fr-one}\n\\end{align*}\n% \\Jiajin{We may have to double-check the above derivation. --- confirmed. With the uniform KL property, we can confirm  $x(y(z),z)$ is unique over $Y(z)$.}\nAs \\eqref{eq:dual1} and \\eqref{eq:dual2} hold for any $x\\in\\mathcal{X}$, we obtain the intermediate relation\n\\begin{equation}\\label{OS-GS-better-key3-main} \n% \\begin{aligned}\n     \\frac{r-L}{2}\\|x_r(y(z),z)-x_r(y_+(z),z)\\|^2\\leq  \\max\\limits_{y^\\prime\\in \\mathcal{Y}} F(x_r(y_+(z),z),y^\\prime) - F(x_r(y_+(z),z),y_+(z)) \n% \\end{aligned}\n\\end{equation}\nby taking $x= x_r(y_+(z),z)$.\n\nNow, we utilize the K{\\L} exponent $\\theta$ of the dual problem to bound the right-hand side of \n\\eqref{OS-GS-better-key3-main}  in terms of $\\|y-y_+(z)\\|$. Consider the following two cases:\n \\begin{enumerate}[label=\\normalfont(\\alph*)]\n \\item Suppose that $\\theta=0$. If $F(x_r(y_+(z),z),y_+(z))=\\max_{y'\\in\\mathcal{Y}} F(x_r(y_+(z),z),y')$, then the desired inequality follows trivially from~\\eqref{OS-GS-better-key3-main}. Otherwise, we have $y_+(z) \\in \\mathcal{Y} \\setminus \\mathcal{Y}^\\star(x_r(y_+(z),z))$. By Assumption~\\ref{ass:kl}, we have\n% we deduce using~\\eqref{eq:Fr-one} that\n% \\begin{align*}\n% F_r(x_r(y_+(z),z),y_+(z),z)\n% &=\\max_{y'\\in\\mathcal{Y}} F_r(x_r(y_+(z),z),y',z)\\\\\n% &\\ge F_r(x_r(y_+(z),z),y_+(z),z)\\\\\n% &\\ge F_r(x_r(y(z),z),y(z),z).\n% \\end{align*}\n%By definition of $y(z)$, it follows that $y_+(z)=y(z)$, and consequently the conclusion trivially holds. \n\\[\n\\dist(0, -\\nabla_y F(x_r(y_+(z),z),y_+(z))+\\partial\\iota_{\\mathcal{Y}}(y_+(z)))\\ge \\mu.\n\\]\nSince  $F(x,\\cdot)$ is continuously differentiable on the compact set $\\mathcal{Y}$, $\\|\\nabla_y F(x,\\cdot)\\|$ is bounded on $\\mathcal{Y}$. Without loss of generality, we can assume that  $\\|\\nabla_y F(x,\\cdot)\\|\\leq L$ on $\\mathcal{Y}$. According to  the mean value theorem, we have \n\\[\n\\max\\limits_{y^\\prime\\in \\mathcal{Y}} F(x_r(y_+(z),z),y^\\prime) - F(x_r(y_+(z),z),y_+(z))\n\\leq L \\cdot {\\rm diam}(\\mathcal{Y}).\n\\]\nUsing the fact that $y_+(z) = \\proj_{\\mathcal{Y}} (y + \\alpha \\nabla_y F(x_r(y,z), y))$, we bound\n\\begin{align*}\n &\\max\\limits_{y^\\prime\\in \\mathcal{Y}} F(x_r(y_+(z),z),y^\\prime) - F(x_r(y_+(z),z),y_+(z))\\notag\\\\\n\\leq\\ & \\frac{L \\cdot {\\rm diam}(\\mathcal{Y})}{\\mu^2}\\cdot\\dist^2(0, -\\nabla_y F(x_r(y_+(z),z),y_+(z))+\\partial\\iota_{\\mathcal{Y}}(y_+(z)))\\notag\\\\ \n% \\stackrel{\\text{\\ding{172}}}{\\leq} \\ &   \\frac{L \\cdot {\\rm diam}(\\mathcal{Y})}{\\mu^2} b^2 \\|\\proj_{\\mathcal{Y}}(y+\\alpha\\nabla_y F(x_r(y_+(z),z),y))-y\\|^2\n% \\notag \\\\\n% \\leq \\ &  \\frac{ L b^2 \\cdot {\\rm diam}(\\mathcal{Y})}{\\mu^2}(\n% \\|\\proj_{\\mathcal{Y}}(y+\\alpha\\nabla_y F(x_r(y_+(z),z),y))-y_+(z)\\|\\ +\\|y_+(z)-y\\|)^2 \\\\ \n% \\leq \\ & \\frac{L b^2 \\cdot {\\rm diam}(\\mathcal{Y})}{\\mu^2}(\\alpha L \\|x_r(y_+(z),z)-x_r(y,z)\\| + \\|y-y_+(z)\\|)^2\\\\\n\\leq \\ & \\frac{L b^2 \\cdot {\\rm diam}(\\mathcal{Y})}{\\mu^2}\\|y_+(z)-y\\|^2,\n\\end{align*}\nwhere $b:=\\frac{1}{\\alpha} + L > 0$ and the last inequality is due to the fact that the projected gradient ascent method satisfies the so-called \\textit{relative error condition}\n\\citep[Section 5]{attouch2013convergence}. \nIt follows that\n\\[\n\\|x_r(y(z),z)-x_r(y_+(z),z)\\| \\leq\\frac{\\sqrt{2L\\cdot {\\rm diam}(\\mathcal{Y})}\\cdot (1+\\alpha L)}{\\alpha\\mu\\sqrt{r-L}}\\|y-y_+(z)\\|. \n\\]\n\\item If $\\theta \\in (0,1)$, then we have\n\\begin{align*}\n% \\label{KL-key-01}\n& \\mu\\left(\\max\\limits_{y^\\prime\\in \\mathcal{Y}} F(x_r(y_+(z),z),y^\\prime) - F(x_r(y_+(z),z),y_+(z))\\right)^{\\theta}\\notag\\\\\n\\leq \\ & \\dist(0, -\\nabla_y F(x_r(y_+(z),z),y_+(z))+\\partial\\iota_{\\mathcal{Y}}(y_+(z))) \\notag\\\\ \n\\leq \\ & \\dist(0, -\\nabla_y F(x_r(y,z),y_+(z)) +\\partial\\iota_{\\mathcal{Y}}(y_+(z)))   \\notag\\\\\n\\ & + \\|\\nabla_y F(x_r(y,z),y_+(z))-\\nabla_y F(x_r(y_+(z),z),y_+(z))\\|\\notag\\\\ \n{\\leq} \\ & \\dist(0, -\\nabla_y F(x_r(y,z),y_+(z))+\\partial\\iota_{\\mathcal{Y}}(y_+(z)))+ L\\sigma_2\\|y-y_+(z)\\|\\notag\\\\\n{\\leq} \\ & \\left(\\frac{1}{\\alpha}+L+L\\sigma_2\\right)\\|y_+(z)-y\\|,\n\\end{align*}\nwhere the third inequality follows from the $L$-Lipschitz continuity  of $\\nabla_y F(\\cdot,\\cdot)$ and  \\eqref{lip-y};  the last inequality follows from the relative error condition of the projected gradient ascent method. This, together with \\eqref{OS-GS-better-key3-main}, yields\n\\[\n\\|x_r(y(z),z)-x_r(y_+(z),z)\\|\\leq \\frac{\\sqrt{2}}{\\sqrt{r-L}}\\left(\\frac{1+\\alpha L(1+\\sigma_2)}{\\alpha\\mu}\\right)^{\\frac{1}{2\\theta}} \n\\|y-y_+(z)\\|^{\\frac{1}{2\\theta}}.\n\\]\n \\end{enumerate}\nThe proof is complete.\n\\end{proof}\n\n\\begin{corollary}[Alternative dual error bound with K\\L{} exponent]\n% Let $\\delta>0$. \nSuppose that  Assumption \\ref{ass:kl} holds. Then, for any $y \\in \\mathcal{Y}$ and $z \\in \\mathbb{R}^n$,  we have\n% \\[\n% \\|x_r^\\star(z)-x_r(y_+(z),z)\\| \\leq \\omega \\|y-y_+(z)\\|^{\\frac{1}{2\\theta}},\n% \\]\n% where $\\omega:=\\frac{\\sqrt{2}}{\\sqrt{r-L}}\\left(\\frac{1+\\alpha L(1+\\sigma_2)}{\\alpha\\mu}\\right)^{\\frac{1}{2\\theta}}$.\n\\begin{enumerate}[label=\\normalfont(\\alph*)]\n \\item \\rm{(K\\L\\ exponent $\\theta=0$):} $\\|x_r^\\star(z)-x_r(y_+(z),z)\\| \\leq \\omega_1 \\|y-y_+(z)\\|$,\n  \\item \\rm{(K\\L\\ exponent $\\theta\\in(0,1)$):} $\\|x_r^\\star(z)-x_r(y_+(z),z)\\| \\leq \\omega_2\\|y-y_+(z)\\|^{\\frac{1}{2\\theta}}$. \n  \\end{enumerate}\n \\label{prop:dual_eb_KL2}   \n\\end{corollary}\n\n\\begin{lemma} \\label{lemma-episolcol}\n        Let $\\epsilon\\ge 0$ be given. Suppose that\n        \\[\n        \\max\\left\\{\\|x^{k+1}-x^{k}\\|,\\|y^{k}_+(z^{k+1}) - y^k\\|,\\|x^{k+1}-z^k\\|\\right\\} \\leq  \\epsilon.\n        \\]\n        Then,  $\\left(x^{k+1}, y^{k+1}\\right)$ is a $\\rho \\epsilon$-GS of problem \\eqref{eq:problem}, where\n\\[\\rho = \\max\\left\\{(\\eta+1+\\sigma_1\\alpha\\beta L)\\left(\\frac{1}{\\alpha}+L\\right), r(\\zeta+\\sigma_2(\\eta+1+\\sigma_1\\alpha \\beta L)+\\sigma_1)\\right\\}\\] \n with $\\sigma_{1}:=\\frac{r}{r-L}$ and $\\sigma_{2}:=\\frac{2r-L}{r-L}$.\n        \\end{lemma}\n\n\\begin{proof}\nBy Definition \\ref{defi:primal-dual}, we only have to quantify $\\|\\nabla_{z} d_r(y^{k+1},x^{k+1})\\|$ and ${\\rm dist}(0,-\\nabla_{y} F(x^{k+1}, y^{k+1})+\\partial\\iota_{\\mathcal{Y}}(y^{k+1}))$.\nTo begin, observe that\n\\begin{equation*}\n\\begin{aligned}\n& \\| \\nabla_{z} d_r(y^{k+1},x^{k+1})\\| \n\\\\\n=\\ & r \\|x^{k+1}-x_r (y^{k+1},x^{k+1})\\|\\\\  \n\\leq\\ & r(\\|x^{k+1}-x_r(y^{k},z^{k})\\| +\\|x_r(y^{k},z^{k})-x_r(y^{k+1},z^{k})\\|)  +r \\|x_r(y^{k+1},z^{k})-x_r(y^{k+1},x^{k+1})\\|  \\\\\n\\leq\\  &  r\\left(\\zeta\\|x^{k+1}-x^k\\| +\\sigma_2\\|y^{k}-y^{k+1}\\| + \\sigma_1 \\|x^{k+1}-z^{k}\\| \\right) \\\\\n% \\leq & r\\left(\\|x^{k+1}-x_r(y^{k},z^{k})\\| +\\|x_r(y^{k},z^{k})-x_r(y^{k+1},z^{k})\\| + \\|x_r(y^{k+1},z^{k})-x_r(y^{k+1},x^{k+1})\\| \\right) \\\\\n\\leq\\  &  r\\left(\\zeta\\|x^{k+1}-x^k\\| + \\sigma_1 \\|x^{k+1}-z^{k}\\| \\right) +r\\sigma_2 (\\eta\\| x^{k+1}-x^k\\|+\\sigma_1\\alpha L\\|z^{k+1}-z^k\\|+\\| y_+^k(z^{k+1})-y^k\\|)\\\\\n\\leq\\ & r(\\zeta+\\sigma_2(\\eta+1+\\sigma_1\\alpha \\beta L)+\\sigma_1)\\epsilon,\n\\end{aligned}\n\\end{equation*}\nwhere the second inequality is due to Proposition \\ref{prop:lip} and Lemma \\ref{lemma-sollip}, the third is due to Lemma \\ref{lemma-dualxy}, and the fourth follows from the update $z^{k+1} = z^k + \\beta (x^{k+1} - x^k)$ and the assumption of the lemma. Next, recall that\n\\begin{equation*}\ny^{k+1} = \\proj_{\\mathcal{Y}}\\left(y^k + \\alpha \\nabla_y F(x^{k+1}, y^k)\\right)=\\argmin_{y \\in \\mathcal{Y}}\\left\\{\\|y-y^k - \\alpha \\nabla_y F(x^{k+1}, y^k)\\|^2\\right\\}.\n\\end{equation*}\nThe necessary optimality condition yields\n\\begin{align*}\n0\n&\\in y^{k+1}-y^k - \\alpha \\nabla_y F(x^{k+1}, y^k)+\\partial \\iota_{\\mathcal{Y}}(y^{k+1})\\notag\\\\\n&=y^{k+1}-y^k - \\alpha \\nabla_y F(x^{k+1}, y^{k+1})+\\partial \\iota_{\\mathcal{Y}}(y^{k+1}) +\\alpha (\\nabla_y F(x^{k+1}, y^{k+1})- \\nabla_y F(x^{k+1}, y^{k})).\n\\end{align*}\nSince $\\partial \\iota_{\\mathcal{Y}} (y^{k+1})$ is the normal cone to $\\mathcal{Y}$ at $y^{k+1}$, we have\n\\begin{align*}\nv\n&:=-\\frac{1}{\\alpha}(y^{k+1}-y^{k})+\\nabla_y F(x^{k+1}, y^{k})- \\nabla_y F(x^{k+1}, y^{k+1})\\in -\\nabla_{y} F(x^{k+1}, y^{k+1})+\\partial \\iota_{\\mathcal{Y}}(y^{k+1}).\n\\end{align*}\nIt follows that\n\\begin{align*}\n\\|v\\| \\ & \\leq \\left(\\frac{1}{\\alpha}+L\\right)\\|y^{k+1}-y^{k}\\|\\\\\n\\ &  \\leq  \\left(\\frac{1}{\\alpha}+L\\right)\\left(\\eta\\|x^{k}-x^{k+1}\\|+\\sigma_1\\alpha L\\|z^{k+1}-z^k\\|+\\|y_+^k(z^{k+1})-y^k\\|\\right) \\\\\n\\ & \\leq (\\eta+1+\\sigma_1\\alpha\\beta L)\\left(\\frac{1}{\\alpha}+L\\right)\\epsilon,\n\\end{align*}\nwhere the first inequality is due to the $L$-Lipschitz continuity of $\\nabla_y F(\\cdot,\\cdot)$ and the second is due to Lemma \\ref{lemma-dualxy}. Putting everything together yields the desired result.\n\\end{proof}\n\n\\begin{proof}\n\nAs mentioned in Section \\ref{sec:framework}, the global convergence rate of smoothed PLDA depends on the specific regime of the growth power $v$ in \\eqref{eq:step_dual}. Let us begin by considering cases (a) and (b), i.e., either the dual function is concave or the dual problem possesses the K\\L{} property with exponent $\\theta\\in(\\frac{1}{2},1)$.\n\n\nIn view of the basic descent estimate in Proposition \\ref{prop:decrease}, we consider the following two scenarios separately:\\\\\n(1) There exists a $ k \\in\\{0,1, \\ldots, K-1\\}$ such that\n\\begin{align*}\n\\ &\\frac{1}{2}\\max \\left\\{\\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}, \\frac{1}{8\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 , \\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}\\right\\}\\\\\n\\leq \\ & 14r\\beta\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2}.\n\\end{align*}\n(2) For any $ k \\in\\{0,1, \\ldots, K-1\\}$, we have\n\\begin{align*}\n\\ &\\frac{1}{2}\\max \\left\\{\\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}, \\frac{1}{8\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 , \\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}\\right\\}\\\\\n\\ge\\ & 14r\\beta\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2}.\n\\end{align*}\n\n\\textbf{Scenario (1)}. For the case where the K{\\L} exponent $\\theta \\in (\\tfrac{1}{2},1)$, we deduce from Proposition \\ref{prop:dual_eb_KL}(b) that \n\\begin{align*}\n    \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2  & \\leq 224r\\alpha\\beta\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2}  \\\\ &\\leq  224r\\alpha\\beta \\omega_2^2 \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{\\theta}},  \n\\end{align*}\nwhich gives $\\|y^{k}-y_{+}^{k}(z^{k+1})\\|\\leq \\rho_1\\beta^{\\frac{\\theta}{2\\theta-1}}$ with $\\rho_1 := (224r\\alpha\\omega_2^2)^{\\frac{\\theta}{2\\theta-1}}$. Additionally, using the update $z^{k+1} = z^k + \\beta( x^{k+1} - z^k )$ and Proposition \\ref{prop:dual_eb_KL}(b), we have\n\\begin{align*}\n\\|x^{k+1}-z^{k}\\|^{2}  \\ & =\\frac{1}{\\beta^2}\\|z^{k+1}-z^{k}\\|^{2}\\\\\n\\ &  \\leq 49\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2} \\\\\n\\ & \\leq  49 \\omega_2^2 \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{\\theta}}\\leq \\rho_2^2\\beta^{\\frac{1}{2\\theta-1}},\n\\end{align*}\nwhere $\\rho_2 := \\sqrt{49\\omega_2^2\\rho_1^{\\frac{1}{\\theta}}}$. \nLastly, we have\n\\begin{align*}\n    \\|x^{k+1}-x^k\\|^2 \\ & \\leq {\\frac{224}{3}r\\beta\\lambda}\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2} \\\\\n   \\  &\\leq {\\frac{224}{3}r\\omega^2_2\\beta\\lambda}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{\\theta}} \\leq \\rho_3^2 \\beta^{\\frac{2\\theta}{2\\theta-1}},  \n\\end{align*}\nwhere $\\rho_3 := \\sqrt{{\\frac{224}{3}r\\omega_2^2\\lambda}\\rho_1^{\\frac{1}{\\theta}}}$. \nCombining the above inequalities, we have\n\\begin{align*}\n& \\max \\left\\{\\|x^{k}-x^{k+1}\\|,\\|y^{k}-y_{+}^{k}(z^{k+1})\\|,\\|z^{k}-x^{k+1}\\|\\right\\} \\leq  \\max\\left\\{ \\rho_1\\beta^{\\frac{\\theta}{2\\theta-1}}, \\rho_2\\beta^{\\frac{1}{4\\theta-2}}, \\rho_3 \\beta^{\\frac{\\theta}{2\\theta-1}}\\right\\}. \n\\end{align*}\nIt then follows from Lemma \\ref{lemma-episolcol}\n that $\\left(x^{k+1}, y^{k+1}\\right)$ is a $\\rho\\cdot\\max\\{ \\rho_1\\beta^{{\\theta}/{(2\\theta-1)}},$ $ \\rho_2\\beta^{1/{(4\\theta-2)}}, \\rho_3\\beta^{{\\theta}/{(2\\theta-1)}}\\}$-GS of problem \\eqref{eq:problem}. \n \n For the concave case, we can replace the primal-dual quantity $x_r( y(z^{k+1}), z^{k+1} )$ by the primal quantity $x_r^\\star( z^{k+1} )$ on the right-hand side of the inequality for Scenario (1) and apply Lemma \\ref{lemma-dual-bd} in Appendix \\ref{sec:dual} (so that $\\theta$ and $\\omega_2$ are replaced by $1$ and $\\kappa = \\tfrac{1 + \\alpha L \\sigma_2 + \\alpha L}{\\alpha (r-L)} \\cdot \\textrm{diam}(\\mathcal{Y})$, respectively) to conclude that $\\left(x^{k+1}, y^{k+1}\\right)$ is a $\\max\\{ \\rho_1\\beta, \\rho_2\\beta^{1/2}, \\rho_3 \\beta\\}$-GS of problem \\eqref{eq:problem}.  \n \n\n\nFurthermore, observe that for the case where $\\theta \\in (\\tfrac{1}{2}, 1)$, the optimization-stationarity measure of $z^{k+1}$ satisfies\n\\begin{equation}\n\\label{eq:os-gs}\n\\begin{aligned}\n    & \\|z^{k+1}-x_r^\\star(z^{k+1})\\| \\\\\n    \\leq\\ &  \\|z^{k+1}-x^{k+1}\\|+\\| x^{k+1} - x_r(y^k,z^k) \\| + \\| x_r(y^k,z^k) - x_r(y^k,z^{k+1}) \\|\\\\\n    \\ & + \\|x_r(y^k,z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|+ \\|x_r(y_{+}^{k}(z^{k+1}), z^{k+1})-x_r^\\star(z^{k+1})\\|\\\\\n     \\leq\\ &  (1+\\sigma_1)\\|z^{k}-x^{k+1}\\| + \\zeta\\|x^{k}-x^{k+1}\\| + \\sigma_2 \\|y^k-y_{+}^{k}(z^{k+1})\\|+ \\omega_2 \\|y^k-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{2\\theta}} \\\\\n    \\leq\\ & \\max\\left\\{\\left(\\rho_2(1+\\sigma_1)+ \\omega_2 \\rho_1^\\frac{1}{2\\theta}\\right)\\beta^{\\frac{1}{4\\theta-2}},(\\zeta\\rho_3+\\sigma_2\\rho_1) \\beta^{\\frac{\\theta}{2\\theta-1}} \\right\\}, \n\\end{aligned}\n\\end{equation}\nwhere the second inequality follows from the update $z^{k+1} = z^k + \\beta (x^{k+1} - z^k)$, Proposition \\ref{prop:lip}, Corollary \\ref{prop:dual_eb_KL2}, Lemma \\ref{lemma-sollip}, and the fact that $0 < \\beta \\le \\tfrac{1}{28}$.\nIt is evident that the dependence on $\\beta$ is identical to that of the game-stationarity measure of $(x^{k+1}, y^{k+1})$.\nFor the concave case, the optimization-stationarity measure of $z^{k+1}$ can be bounded similarly.\n\n\\textbf{Scenario (2)}. Proposition \\ref{prop:decrease} implies that for any $k \\in\\{0,1, \\ldots, K-1\\}$, we have \n\\begin{align}\n\\label{eq:suff-decrease}\n\\Phi_r^k-\\Phi_r^{k+1} \n\\ge \\frac{3}{16\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{16\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 +\\frac{2r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}.\n\\end{align}\nThe assumptions in case (a)  imply that \n\\[ \\min_{x\\in\\mathcal{X}}\\max_{y\\in\\mathcal{Y}} F_r(x,y,z) = \\max_{y\\in\\mathcal{Y}} \\min_{x\\in\\mathcal{X}} F_r(x,y,z) \\]\nfor all $z \\in \\mathbb{R}^n$ (cf.~the proof of~\\citep[Lemma B.7]{zhang2020single}) and lead to the lower boundedness of $p_r$, while the assumptions in case (b) directly imply that $p_r$ is lower bounded.\nIn both cases, there exists a $\\underline{\\Phi} > -\\infty$ such that for all $x \\in \\mathcal{X}$, $y \\in \\mathcal{Y}$, and $z \\in \\mathbb{R}^n$,\n\\begin{align*}\n& \\Phi_r(x, y, z)  = p_r(z)+(F_r(x,y,z)-d_r(y, z))+(p_r(z)-d_r(y, z))\\geq p_r(z)\\geq \\underline{\\Phi} >-\\infty.\n\\end{align*}\nIt follows that\n    \\begin{align*}\n    &\\Phi_r^0-\\underline{\\Phi} \\geq \\sum_{k=0}^{K-1}\\Phi_r(x^k,y^k,z^k)-\\Phi_r(x^{k+1},y^{k+1},z^{k+1}) \\\\\n    % \\geq\\ & \\sum_{k=0}^{K-1} \\frac{1}{32\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{16\\alpha}\\|y^{k}-y_{+}^{k}(z^{k})\\|^2 +\\frac{2r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2} \\\\\n    \\geq \\ & \\sum_{k=0}^{K-1} \\min\\left\\{\\frac{3}{16\\lambda},\\frac{1}{16\\alpha},\\frac{2\\beta r}{7}\\right\\} \\left(\\|x^{k}-x^{k+1}\\|^{2}+\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 + \\|z^{k}-x^{k+1}\\|^{2} \\right),\n    \\end{align*}\n    where the last inequality is due to \\eqref{eq:suff-decrease} and the update $z^{k+1}=z^{k}+\\beta(x^{k+1}-z^k)$.\n    In particular, there exists a $k \\in\\{0,1, \\ldots, K-1\\}$ such that\n    \\[\n    \\max \\left\\{\\|x^{k}-x^{k+1}\\|^{2},\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2}, \\|x^{k+1}-z^{k}\\|^{2}\\right\\} \\leq \\frac{\\Phi_r^0-\\underline{\\Phi}}{\\min\\left\\{\\frac{3}{16\\lambda},\\frac{1}{16\\alpha},\\frac{2\\beta  r}{7}\\right\\}  K}.\n    \\]\n% \\Jiajin{continue here!}\nBased on Lemma \\ref{lemma-episolcol} and the fact that $0 < \\beta \\le \\tfrac{1}{28}$,  we conclude that $\\left(x^{k+1}, y^{k+1}\\right)$ is an $\\mathcal{O}(\\sqrt{1/{K\\beta}})$-GS of problem \\eqref{eq:problem}. Moreover, by using the same argument as in \\eqref{eq:os-gs}, we can show that $z^{k+1}$ is an $\\mathcal{O}(\\sqrt{1/{K\\beta}})$-OS of problem \\eqref{eq:problem}. Upon setting $\\beta = C K^{-\\frac{1}{2}}$ for the concave case and  $\\beta = C K^{-\\frac{2\\theta-1}{2\\theta}}$ for the case where the K{\\L} exponent $\\theta \\in (\\tfrac{1}{2}, 1)$, with $C$ being a suitably chosen constant, we see that the \n% where $C$ is a constant.\n% Similarly, for the general concave case, we can set $\\beta = C K^{-\\frac{1}{2}}$. \nbounds obtained for the optimization-stationarity and game-stationarity measures are of the same order for both Scenarios (1) and (2). This establishes cases (a) and (b).\n\nNext, let us consider case (c). First, suppose that the K{\\L} exponent $\\theta \\in (0, \\tfrac{1}{2}]$. \nOnce again, we can use Proposition \\ref{prop:decrease} to obtain \n\\begin{align*}\n\\Phi_r^k-\\Phi_r^{k+1} \n\\ge\\ &  \\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{8\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 +\\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}\\\\\n\\ & -14r\\beta\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2}.\n\\end{align*}\nUsing Proposition \\ref{prop:dual_eb_KL}(b) and the boundedness of $\\mathcal{Y}$, we have\n% there exists  for each $\\theta \\in(0,\\frac{1}{2})$ that \n\\begin{equation}\\label{dual-eb-bd}\n\\begin{aligned}\n  \\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|\n&\\leq \\omega_2 \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{2\\theta}}\\\\\n&=  \\omega_2 \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{2\\theta}-1} \\|y^{k}-y_{+}^{k}(z^{k+1})\\|\\\\\n&\\leq   \\omega_2\\cdot {\\rm diam}(\\mathcal{Y})^{\\frac{1}{2\\theta}-1}\\cdot \\|y^{k}-y_{+}^{k}(z^{k+1})\\|.\n\\end{aligned}\n\\end{equation}\nIt follows that\n\\begin{align*}\n\\Phi_r^k-\\Phi_r^{k+1} \n\\ge \\ &  \\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\left(\\frac{1}{8\\alpha}-14r\\beta\\omega_2^2\\cdot{\\rm diam}(\\mathcal{Y})^{\\frac{1-2\\theta}{\\theta}}\\right) \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2  +\\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}.\n\\end{align*}\nSince \n$\\beta  \\leq \\frac{ {\\rm diam}(\\mathcal{Y})^{\\frac{2\\theta-1}{\\theta}}}{224 \\alpha r\\omega_2^2}$, we have \n\\begin{align*}\n&\\Phi_r^k-\\Phi_r^{k+1} \\ge  \\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{16\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 +\\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}.\n\\end{align*}\nThe desired result can then be obtained by following a similar argument as that in the paragraph proceeding \\eqref{eq:suff-decrease}.\n\nWhen the K{\\L} exponent $\\theta=0$, we can apply Proposition \\ref{prop:dual_eb_KL}(a) to derive the inequality \\eqref{dual-eb-bd} \n% by letting $\\theta=\\frac{1}{2}$ and replacing $\\omega_2$ with $\\omega_1$. \nwith $\\theta$ and $\\omega_2$ being replaced by $\\tfrac{1}{2}$ and $\\omega_1$, respectively.\nThe desired result then follows by adapting the argument in the preceding paragraph.\n\\end{proof}\n\n\\begin{fact}\\label{fact-1}\nThe functions $F(\\cdot, y)$ for any $y\\in\\mathcal{Y}$ and $f$ are $L$-weakly convex on $\\mathcal{X}$.\n\\end{fact}\n\n\\begin{fact}\\label{fact-2}\nLet $y\\in\\R^d$ and $r>0$ be given. For all $x, \\bar{x} \\in\\mathcal{X}$, we have\n\\[\n-\\frac{r^{-1}+L}{2}\\|x-\\bar{x}\\|^{2} \\leq F(x,y)-F_{\\bar{x},r}(x, y) \\leq \\frac{-r^{-1} + L}{2}\\|x-\\bar{x}\\|^{2} .\n\\]\n\\end{fact}\n\n\\begin{lemma}\\label{lemma-sollip}\nFor any $y, y^{\\prime} \\in \\mathcal{Y}$ and $z, z^{\\prime} \\in \\R^n$, we have \n    \\begin{align}\n        &\\|x_r(y, z)-x_r(y, z^{\\prime})\\| \\leq \\sigma_{1}\\|z-z^{\\prime}\\|, \\label{lip-z}\\\\\n        &\\|x_r^\\star(z)-x_r^\\star(z^{\\prime})\\| \\leq \\sigma_{1}\\|z-z^{\\prime}\\|, \\label{lip-starz}\\\\\n        &\\|x_r(y, z)-x_r(y^{\\prime}, z)\\| \\leq \\sigma_{2}\\|y-y^{\\prime}\\|, \\label{lip-y}\n        \\end{align}\n     where $\\sigma_{1}:=\\frac{r}{r-L}$ and $\\sigma_{2}:=\\frac{2r-L}{r-L}$.\n    \\end{lemma}\n\n\\begin{proof}\nFrom the definition of $F_r$, we know that for any $x\\in\\mathcal{X}$, $y\\in\\mathcal{Y}$, and $z,z'\\in\\R^n$, \n\\begin{equation}\\label{func-cal}\nF_r(x,y,z^{\\prime})-F_r(x,y,z)=\\frac{r}{2}(\\|x-z'\\|^2-\\|x-z\\|^2)=\\frac{r}{2}(\\|z^{\\prime}\\|^{2}-2(z^{\\prime}-z)^\\top x-\\|z\\|^{2}).\n\\end{equation}\nFact \\ref{fact-1} implies that $F(\\cdot, y)$ is $L$-weakly convex for any $y \\in \\mathcal{Y}$. Therefore, for any $x\\in\\mathcal{X}$, $y\\in\\mathcal{Y}$, and $z\\in\\R^n$, we have\n\\begin{equation}\\label{psi-strongcvx}\nF_r(x,y, z)- F_r(x_r(y,z),y,z)\n\\geq \\frac{r-L}{2}\\|x-x_r(y,z)\\|^{2}.\n\\end{equation}\nCombining \\eqref{func-cal} and \\eqref{psi-strongcvx} yields\n\\begin{align}\\label{lip-key1}\n    &F_r(x_r(y,z),y,z^{\\prime})-F_r(x_r(y,z^{\\prime}),y,z^{\\prime}) \\notag\\\\\n    =\\ &F_r(x_r(y,z),y,z^{\\prime})-F_r(x_r(y,z),y,z)+F_r(x_r(y,z),y,z)-F_r(x_r(y,z^{\\prime}),y,z)\\, \\notag\\\\\n    &-(F_r(x_r(y,z^{\\prime}),y, z^{\\prime})-F_r(x_r(y,z^{\\prime}),y, z)) \\notag\\\\\n    \\leq\\ &\\frac{r}{2}(\\|z^{\\prime}\\|^{2}-2(z^{\\prime}-z)^\\top x_r(y,z)-\\|z\\|^{2})-\\frac{r-L}{2}\\|x_r(y,z)-x_r(y,z^{\\prime})\\|^{2}\\,  \\notag\\\\\n    &-\\frac{r}{2}(\\|z^{\\prime}\\|^{2}-2(z^{\\prime}-z)^\\top x_r(y,z^{\\prime})-\\|z\\|^{2}) \\notag\\\\\n    \\leq\\ &r(z^{\\prime}-z)^\\top(x_r(y,z^{\\prime})-x_r(y,z))-\\frac{r-L}{2}\\|x_r(y,z)-x_r(y,z^{\\prime})\\|^{2}.\n    \\end{align}\n On the other hand, \\eqref{psi-strongcvx} implies that\n    \\[\n    F_r(x_r(y,z),y, z^{\\prime})-F_r(x_r(y,z^{\\prime}),y,z^{\\prime}) \\geq \\frac{r-L}{2}\\|x_r(y,z)-x_r(y,z^{\\prime})\\|^{2}.\n    \\]\nCombining this inequality with \\eqref{lip-key1}, we get\n    \\[\n    (r-L)\\|x_r(y,z)-x_r(y,z^{\\prime})\\|^{2}\\leq r(z^{\\prime}-z)^\\top (x_r(y,z^{\\prime})-x_r(y,z)).\n    \\]\nUsing the Cauchy-Schwarz inequality, we obtain\n    \\[\n    \\|x_r(y,z)-x_r(y,z^{\\prime})\\| \\leq \\frac{r}{r-L}\\|z-z^{\\prime}\\|.\n    \\]\nTherefore, we conclude that \\eqref{lip-z} holds with $\\sigma_1=\\frac{r}{r-L}$. Since $f=\\max_{y \\in \\mathcal{Y}} F(\\cdot,y)$ is also $L$-weakly convex by  Fact \\ref{fact-1}, we can prove that \\eqref{lip-starz} holds by using a similar argument as above.\n    \n\nWe  now proceed to prove \\eqref{lip-y}. Using \\eqref{psi-strongcvx}, we have\n\\begin{align}\n&F_r(x_r(y^{\\prime}, z),y, z)-F_r(x_r(y, z), y,z) \\geq \\frac{r-L}{2}\\|x_r(y, z)-x_r(y^{\\prime}, z)\\|^{2},\\label{psi-strongcvxineq1} \\\\\n&F_r(x_r(y, z), y^{\\prime},z)-F_r(x_r(y^{\\prime}, z), y^{\\prime},z) \\geq \\frac{r-L}{2}\\|x_r(y, z)-x_r(y^{\\prime}, z)\\|^{2}\\label{psi-strongcvxineq2}.\n\\end{align}\nMoreover, by the $L$-Lipschitz continuity of $\\nabla_{y} F_r(x, \\cdot, z)$ for any $x \\in \\mathcal{X}$ and $z \\in \\mathbb{R}^n$, we have\n\\begin{equation}\n\\begin{aligned}\\label{psi-yconcave}\n F_r(x_r(y, z), y^{\\prime},z)-F_r(x_r(y, z), y,z)\\leq &\\langle\\nabla_{y} F_r(x_r(y, z),y,z), y^{\\prime}-y\\rangle +\\frac{L}{2}\\|y-y^{\\prime}\\|^{2}\n\\end{aligned}\n\\end{equation}\nand \n\\begin{equation}\n\\begin{aligned}\\label{psi-ylip}\n F_r(x_r(y^{\\prime}, z), y, z)-F_r(x_r(y^{\\prime}, z), y^{\\prime},z) \\leq & \\langle\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime}, z), y-y^{\\prime}\\rangle + \\frac{L}{2}\\|y-y^{\\prime}\\|^{2}.\n\\end{aligned}\n\\end{equation}\nIncorporating \\eqref{psi-strongcvxineq1}--\\eqref{psi-ylip}, we obtain\n\\begin{equation*}\n% \\label{lip-key2}\n\\begin{aligned}\n& (r-L)\\|x_r(y, z)-x_r(y^{\\prime}, z)\\|^{2}  \\\\ \\leq \\ &\\langle\\nabla_{y} F_r(x_r(y, z), y, z)-\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime}, z), y^{\\prime}-y\\rangle +L\\|y-y^{\\prime}\\|^{2}. \n\\end{aligned}\n\\end{equation*}\nSince $\\nabla_y F_r(\\cdot,\\cdot,z) = \\nabla_y F(\\cdot,\\cdot)$ is $L$-Lipschitz for any $z \\in \\mathbb{R}^n$, we have\n\\begin{align*}\n&\\|\\nabla_{y} F_r(x_r(y, z), y, z)-\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime}, z)\\| \\notag\\\\\n\\leq\\ &\\|\\nabla_{y} F_r(x_r(y, z), y, z)-\\nabla_{y}F_r(x_r(y, z), y^{\\prime}, z)\\|  +\\|\\nabla_{y}F_r(x_r(y, z), y^{\\prime}, z)-\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime}, z)\\|\\notag\\\\\n\\leq\\ &L(\\|y-y^{\\prime}\\|+\\|x_r(y^{\\prime}, z)-x_r(y, z)\\|).\\notag\n\\end{align*}\nIt follows that\n\\begin{align*}\n(r-L)\\|x_r(y, z)-x_r(y^{\\prime}, z)\\|^{2} \n\\leq L\\|x_r(y^{\\prime}, z)-x_r(y, z)\\|\\cdot\\|y-y^{\\prime}\\|+2L\\|y-y^{\\prime}\\|^{2}.\n\\end{align*}\nLet $\\zeta:=\\frac{\\|x_r(y, z)-x_r(y^{\\prime}, z)\\|}{\\|y-y^{\\prime}\\|}$. \nThen, the above inequality gives\n\\begin{align*}\n\\zeta^{2}\n& \\ \\leq \\frac{L}{r-L} \\zeta+\\frac{2L}{(r-L)}\\leq \\frac{1}{2}\\zeta^2+\\frac{L^2}{2(r-L)^2} +\\frac{2L}{(r-L)}\\\\\n& \\ \\leq \\frac{1}{2} \\zeta^{2}+\\frac{L^2+4L(r-L)}{2(r-L)^2}  \\leq \\frac{1}{2} \\zeta^{2}+\\frac{(L+2(r-L))^2}{2(r-L)^2},\n\\end{align*}\nwhere the second inequality holds because $ab\\leq \\frac{1}{2}(a^2+b^2)$ for any $a,b\\in\\R$. Thus, we get\n\\[\n\\|x_r(y, z)-x_r(y^{\\prime}, z)\\| \\leq  \\frac{2r-L}{r-L}\\|y-y^{\\prime}\\|,\n\\]\nwhich shows that \\eqref{lip-y} holds with $\\sigma_{2}=\\frac{2r-L}{r-L}$. The proof is complete.\n\\end{proof}\n\n\\begin{lemma}\\label{lemma-dualdiff} \n    The dual potential value function $d_r$ is differentiable on $\\mathcal{Y}\\times \\R^n$ with\n    \\begin{align*}\n    &\\nabla_{y} d_r(y,z)=\\nabla_{y} F(x_r(y,z),y),\\\\\n    &\\nabla_{z} d_r(y,z)=\\nabla_{z} F_r(x_r(y, z),y, z)=r(z-x_r(y,z))\n    \\end{align*}\n    for any $y \\in \\mathcal{Y}$ and $z \\in \\mathbb{R}^n$. Moreover, for any $y \\in \\mathcal{Y}$ and $z \\in \\mathbb{R}^n$,  the gradients $\\nabla_y d_r(\\cdot,z)$ and $\\nabla_z d_r(y,\\cdot)$ are Lipschitz continuous, i.e.,\n    \\begin{align}\n    &\\|\\nabla_{y} d_r(y^{\\prime},z)-\\nabla_{y} d_r(y^{\\prime \\prime},z)\\| \\leq L_{d_r}\\|y^{\\prime}-y^{\\prime \\prime}\\| \\quad \\text{for all}\\ y^{\\prime}, y^{\\prime \\prime} \\in \\mathcal{Y},\\notag\\\\\n    &\\|\\nabla_{z} d_r(y,z')-\\nabla_{z} d_r(y,z'')\\| \\leq L_{d_r}'\\|z^{\\prime}-z^{\\prime \\prime}\\| \\quad \\text{for all}\\ z^{\\prime}, z^{\\prime \\prime} \\in \\R^n,\\notag\n    \\end{align}\n    where $L_{d_r}:=(\\sigma_{2}+1)L$ and $L_{d_r}':=(\\sigma_1 + 1)r$.\n    \\end{lemma}\n\n\\begin{proof}\nSince $F_r(\\cdot,y,z)$ is strongly convex, $F_r(x,\\cdot,z)$ is weakly concave, and $F_r(x,y,\\cdot)$ is strongly convex for any $x \\in \\mathcal{X}$, $y \\in \\mathcal{Y}$, and $z \\in \\mathbb{R}^n$, the function\n    \\[\n    d_r(\\cdot,\\cdot)=\\min_{x\\in\\mathcal{X}} F_r(x,\\cdot,\\cdot)\n    \\] \n    is differentiable on $\\mathcal{Y}\\times \\R^n$ due to \\citep[Theorem 10.31]{rockafellar2009variational}. In particular, for any $y\\in\\mathcal{Y}$ and $z\\in\\R^n$, one has\n    \\[\n    \\nabla_{y} d_r(y, z)=\\nabla_{y} F_r(x_r(y, z),y, z)=\\nabla_{y} F(x_r(y, z), y).\n    \\]\nUsing  \\citep[Lemma 4.3]{drusvyatskiy2019efficiency}, we have \n    \\[\n    \\nabla_{z} d_r(y,z)=\\nabla_{z} F_r(x_r(y, z),y, z)=r(z-\\prox_{\\frac{1}{r}F(\\cdot,y)+\\iota_{\\mathcal{X}}}(z))=r(z-x_r(y,z)).\n    \\]\n    It follows that for  any $y^{\\prime}, y^{\\prime \\prime} \\in \\mathcal{Y}$, \n    \\begin{align*}\n    &\\|\\nabla_{y} d_r(y^{\\prime}, z)-\\nabla_{y} d_r(y^{\\prime \\prime}, z)\\|\\notag\\\\\n    =\\ &\\|\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime},z)-\\nabla_{y} F_r(x_r(y^{\\prime \\prime}, z), y^{\\prime \\prime},z)\\| \\\\\n    \\leq\\ &\\|\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime},z)-\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime \\prime}, z)\\|+\\|\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime \\prime},z)-\\nabla_{y} F_r(x_r(y^{\\prime \\prime}, z), y^{\\prime \\prime}, z)\\| \\\\\n    \\leq\\ & L\\|y^{\\prime}-y^{\\prime \\prime}\\|+L\\|x_r(y^{\\prime}, z)-x_r(y^{\\prime \\prime}, z)\\|\\notag\\\\\n    \\leq\\ & L\\|y^{\\prime}-y^{\\prime \\prime}\\|+L \\sigma_{2}\\|y^{\\prime}-y^{\\prime \\prime}\\|\\leq L_{d_r}\\|y^{\\prime}-y^{\\prime \\prime}\\|,\n    \\end{align*}\n    where the third inequality is due to \\eqref{lip-y}. Moreover, we have\n    \\begin{align*}\n    \\|\\nabla_{z} d_r(y,z')-\\nabla_{z} d_r(y,z'')\\|=\\ &r\\|z'-x_r(y,z')-(z''-x_r(y,z''))\\| \\\\\n    \\leq\\ & r\\|z^{\\prime}-z^{\\prime \\prime}\\|+r\\|x_r(y, z')-x_r(y, z'')\\| \\\\\n    \\leq\\ & r\\|z^{\\prime}-z^{\\prime \\prime}\\|+ r\\sigma_{1}\\|z^{\\prime}-z^{\\prime \\prime}\\|\\leq L_{d_r}'\\|z^{\\prime}-z^{\\prime \\prime}\\|,\n    \\end{align*}\nwhere the second inequality is due to \\eqref{lip-z}. The proof is complete.\n\\end{proof}\n\n\\begin{lemma}\\label{lemma-dualxy}\nFor any $k\\ge0$, we have\n    \\begin{equation*}\n        \\|y^{k+1}-y_{+}^{k}(z^{k+1})\\| \\leq \\eta \\|x^{k}-x^{k+1}\\|+\\sigma_1\\alpha L\\|z^{k+1}-z^k\\|,\n        \\end{equation*}\nwhere $\\eta:=\\alpha L \\zeta$.\n\\end{lemma}\n\n\\begin{proof}\nWe compute\n\\begin{equation*}\n% \\label{iter-result2}\n\\begin{aligned}\n\\ &\\|y^{k+1}-y_{+}^{k}(z^{k+1})\\|\\\\ =\\ &\\|\\proj_{\\mathcal{Y}}(y^{k}+\\alpha \\nabla_{y} F(x^{k+1},y^{k}))-\\proj_{\\mathcal{Y}}(y^{k}+\\alpha \\nabla_{y} F(x_r(y^{k}, z^{k+1}),y^{k}))\\|\\notag \\\\\n\\leq\\ &\\|y^{k}+\\alpha \\nabla_{y} F(x^{k+1}, y^{k})-(y^{k}+\\alpha \\nabla_{y} F(x_r(y^{k}, z^{k+1}),  y^{k}))\\|\\notag \\\\\n\\leq\\ & \\alpha L(\\|x^{k+1}-x_r(y^{k}, z^{k})\\|+\\|x_r(y^{k}, z^{k})-x_r(y^{k}, z^{k+1})\\|) \\notag \\\\\n\\leq\\ & \\eta\\|x^{k}-x^{k+1}\\|+\\sigma_1\\alpha L\\|z^{k+1}-z^k\\|,\n\\end{aligned}\n\\end{equation*}\nwhere the first inequality is due to the nonexpansiveness of the projection operator, the second is due to the $L$-Lipschitz continuity of $\\nabla_y F(\\cdot,\\cdot)$, and the third follows from Proposition \\ref{prop:lip} and \\eqref{lip-z}. The proof is complete.\n\\end{proof}\n\n\\begin{lemma}[Primal descent]\n\\label{lemma-F-des} \nFor any $k\\ge0$, we have\n\\begin{align*}\n& F_r(x^{k}, y^{k},z^k)-F_r(x^{k+1}, y^{k+1},z^{k+1})\\\\\n\\ge\\ &\n\\frac{2\\lambda^{-1}+r-L}{2}\\|x^k-x^{k+1}\\|^2 + \\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k}-y^{k+1}\\rangle-\\frac{L}{2}\\|y^k-y^{k+1}\\|^2 \\\\  &  +\\frac{(2-\\beta)r}{2 \\beta}\\|z^{k}-z^{k+1}\\|^{2}. \n\\end{align*}\n\\end{lemma}\n\n\\begin{proof}\nOne can infer from the definition that $F_{x^k,\\lambda}(\\cdot,y^k)$ is $\\lambda^{-1}$-strongly convex. Therefore, we have\n\\begin{equation*}\n\\begin{aligned}\nF_r(x^{k}, y^{k},z^{k}) & =F(x^{k}, y^{k})+ \\frac{r}{2}\\|x^k-z^{k}\\|^2  = F_{x^k,\\lambda}(x^k,y^k) + \\frac{r}{2}\\|x^k-z^{k}\\|^2 \\\\\n& \\ge F_{x^k,\\lambda}(x^{k+1},y^k)+\\frac{r}{2}\\|x^{k+1}-z^{k}\\|^2+\\frac{\\lambda^{-1}+r}{2}\\|x^k-x^{k+1}\\|^2.\n\\end{aligned}\n\\end{equation*}\nMoreover, Fact \\ref{fact-2} implies that\n\\begin{equation*} \nF_{x^k,\\lambda}(x^{k+1},y^k)\\ge F(x^{k+1},y^k) + \\frac{\\lambda^{-1}-L}{2}\\|x^{k+1}-x^k\\|^2.\n\\end{equation*}\nIt follows that\n\\begin{align}\\label{primaldec-key1}\nF_r(x^{k},y^{k}, z^k) &\\ge F(x^{k+1},y^k)+\\frac{r}{2}\\|x^{k+1}-z^k\\|^2+\\frac{2\\lambda^{-1}+r-L}{2}\\|x^k-x^{k+1}\\|^2\\notag\\\\\n&= F_r(x^{k+1},y^{k}, z^{k})+\\frac{2\\lambda^{-1}+r-L}{2}\\|x^k-x^{k+1}\\|^2.\n\\end{align}\nNext,  as $\\nabla_y F_r(x,\\cdot,z)$ is $L$-Lipschitz continuous for any $x\\in\\mathcal{X}$ and $z\\in\\R^n$, we have \n    \\begin{align}\\label{primaldec-key2}\n        F_r(x^{k+1}, y^{k},z^k)-F_r(x^{k+1}, y^{k+1},z^k) \\ge \\langle\\nabla_{y} F_r(x^{k+1},y^{k}, z^k), y^{k}-y^{k+1}\\rangle-\\frac{L}{2}\\|y^k-y^{k+1}\\|^2.\n    \\end{align}\nAt last, based on the update $z^{k+1}=z^{k}+\\beta(x^{k+1}-z^{k})$, we get\n\\begin{equation}\\label{primaldec-key3}\nF_r(x^{k+1}, y^{k+1},z^{k})-F_r(x^{k+1},y^{k+1}, z^{k+1})  = \\frac{(2-\\beta)r}{2 \\beta}\\|z^{k}-z^{k+1}\\|^{2}.\n\\end{equation}\nBy summing up \\eqref{primaldec-key1}, \\eqref{primaldec-key2}, and \\eqref{primaldec-key3}, we obtain the desired result.\n\\end{proof}\n\n\\begin{lemma}[Dual ascent]\\label{lemma-d-aes} \nFor any $k\\ge0$, we have\n\\begin{equation*}\n% \\label{dual-ascent}\n \\begin{aligned}\n    \\ & d_r(y^{k+1},z^{k+1})-d_r(y^{k},z^k)\\\\\n    \\ge\\ &\\langle\\nabla_{y} F_r(x_r(y^{k}, z^{k}),  y^{k}, z^{k}), y^{k+1}-y^{k}\\rangle-\\frac{L_{d_r}}{2}\\|y^{k}-y^{k+1}\\|^{2}\\,\\\\\n    &+\\frac{r}{2}\\left\\langle z^{k+1}-z^{k},z^{k+1}+z^{k}-2 x_r(y^{k+1}, z^{k+1})\\right\\rangle. \n    \\end{aligned}\n\\end{equation*}\n\\end{lemma}\n\n\\begin{proof}\nBased on Lemma \\ref{lemma-dualdiff}, $\\nabla_{y} d_r(\\cdot,z)$ is $L_{d_r}$-Lipschitz continuous for any $z \\in \\mathbb{R}^n$. Thus, we have \n        \\begin{align*}\n        \\ & d_r(y^{k+1},z^k)-d_r(y^{k},z^k)  \n        \\geq \\langle\\nabla_{y} d_r(y^{k},z^k), y^{k+1}-y^{k}\\rangle-\\frac{L_{d_r}}{2}\\|y^{k}-y^{k+1}\\|^{2} \\\\\n        =\\ & \\langle \\nabla_{y} F_r(x_r(y^{k}, z^{k}), y^{k}, z^{k}), y^{k+1}-y^{k}\\rangle-\\frac{L_{d_r}}{2}\\|y^{k}-y^{k+1}\\|^{2}.\n        \\end{align*}\nIn addition, one has \n        \\begin{align*}\n        &d_r(y^{k+1}, z^{k+1})-d_r(y^{k+1}, z^{k})\\notag\\\\ \n        =\\ & F_r(x_r(y^{k+1}, z^{k+1}), y^{k+1}, z^{k+1})-F_r(x_r(y^{k+1}, z^{k}), y^{k+1},z^{k}) \\\\\n        \\geq\\ &  F_r(x_r(y^{k+1}, z^{k+1}), y^{k+1},z^{k+1})-F_r(x_r(y^{k+1}, z^{k+1}), y^{k+1},z^{k}) \\\\\n        =\\ & \\frac{r}{2}\\|x_r(y^{k+1}, z^{k+1})-z^{k+1}\\|^{2}-\\frac{r}{2}\\|x_r(y^{k+1}, z^{k+1})-z^{k}\\|^{2} \\\\\n        =\\ & \\frac{r}{2} \\left\\langle z^{k+1}-z^{k},z^{k+1}+z^{k}-2 x_r(y^{k+1}, z^{k+1})\\right\\rangle.\n        \\end{align*}\nFinally, by combining the above inequalities, the proof is complete.\n\\end{proof}\n\n\\begin{lemma}[Proximal descent (smoothness)]\n\\label{lemma-p-des} \n    For any $k\\ge0$, we have\n   \\begin{equation*}\n   % \\label{proximal-descent}\n    p_r(z^k)-p_r(z^{k+1}) \\ge \\frac{r}{2}\\left\\langle z^{k+1}-z^{k},2 x_r(y(z^{k+1}), z^{k})-z^{k}-z^{k+1}\\right\\rangle, \n    \\end{equation*}\n    where $y(z^{k+1})\\in Y(z^{k+1})$.\n    \\end{lemma}\n\n\\begin{proof}\n% From Sion's minimax theorem \\citep{sion1958general}, we know that\n%     \\[\n%     \\min _{x \\in \\mathcal{X}} \\max _{y \\in \\mathcal{Y}} F_r(x, y, z)=\\max_{y \\in \\mathcal{Y}} \\min_{x \\in \\mathcal{X}} F_r(x,y,z)\n%     \\]\n%     which implies\nRecall that \n% $p_r(z)$:\n    % \\[\n    $p_r(z)=\\max _{y \\in \\mathcal{Y}} d_r(y, z)$.\n    % \\]\nDue to the definition of $y(z^{k+1})$, we have\n    \\begin{align*}\n     &p_r(z^{k+1})-p_r(z^{k})\\notag\\\\\n    \\leq\\ & d_r(y(z^{k+1}), z^{k+1})-d_r(y(z^{k+1}), z^{k}) \\\\\n    \\leq\\ & F_r(x_r(y(z^{k+1}), z^{k}), y(z^{k+1}), z^{k+1})-F_r(x_r(y(z^{k+1}), z^{k}), y(z^{k+1}),z^{k}) \\\\\n    =\\ & \\frac{r}{2}\\left\\langle z^{k+1}-z^{k},z^{k+1}+z^{k}-2 x_r(y(z^{k+1}), z^{k})\\right\\rangle,\n    \\end{align*}\n    where the second inequality follows from the fact that \\[F_r(x', y, z)\\geq \\min_{x\\in\\mathcal{X}} F_r(x,y,z) = d_r(y, z)\\] for any $x'\\in\\mathcal{X}$. The proof is complete.\n    \\end{proof}\n\n\\paragraph{{\\bf Proof of Proposition \\ref{prop:decrease}}}\n% \\begin{proof}\nFrom Lemmas \\ref{lemma-F-des}, \\ref{lemma-d-aes}, and \\ref{lemma-p-des}, we have\n        \\begin{align*}\n        % \\label{desascent-key1}\n        & \\Phi_r(x^k,y^k,z^k)-\\Phi_r(x^{k+1},y^{k+1},z^{k+1}) \\notag\\\\\n         =\\ & F_r(x^k,y^k,z^k)-F_r(x^{k+1},y^{k+1},z^{k+1}) +2(d_r(y^{k+1},z^{k+1})-d_r(y^k,z^k))+2(p_r(z^k)-p_r(z^{k+1})) \\notag\\\\\n         \\ge\\ & \\frac{2\\lambda^{-1}+r-L}{2}\\|x^k-x^{k+1}\\|^2 +\\frac{(2-\\beta)r}{2 \\beta}\\|z^{k}-z^{k+1}\\|^{2}-\\left(L_{d_r}+\\frac{L}{2}\\right)\\|y^{k}-y^{k+1}\\|^{2}\\,  \\notag\\\\\n        &  + \\underbrace{\\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k}-y^{k+1}\\rangle+2\\langle\\nabla_{y} F_r(x_r(y^{k}, z^{k}),  y^{k}, z^{k}), y^{k+1}-y^{k}\\rangle}_{\\text{\\ding{172}}}\\, \n      \\notag \\\\ \n      & + \\underbrace{2 r\\left\\langle z^{k+1}-z^{k},x_r(y(z^{k+1}), z^{k})-x_r(y^{k+1}, z^{k+1})\\right\\rangle}_{\\text{\\ding{173}}}.\n        \\end{align*}\n% Subsequently, we simplify the terms \\text{\\ding{172}} and  \\text{\\ding{173}}. \nFirst, we simplify the term \\text{\\ding{172}}. We know that\n        \\begin{align*}\n        \\text{\\ding{172}} \n         =\\ & \\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k}-y^{k+1}\\rangle+2\\langle\\nabla_{y} F_r(x_r(y^{k}, z^{k}), y^{k}, z^{k}), y^{k+1}-y^{k}\\rangle\\\\\n        =\\ & \\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k+1}-y^{k}\\rangle + 2\\langle\\nabla_{y} F_r(x_r(y^{k}, z^{k}), y^{k}, z^{k})-\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k+1}-y^{k}\\rangle.\n        \\end{align*}\n        For the first term, we have\n        \\begin{align*}\n    &\\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k+1}-y^{k}\\rangle \\notag\\\\\n     =\\ &  \\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k)+\\frac{1}{\\alpha}(y^{k}-y^{k+1}), y^{k+1}-y^{k}\\rangle + \\frac{1}{\\alpha}\\|y^{k}-y^{k+1}\\|^2\\\\\n     =\\ & \\frac{1}{\\alpha}\\langle y^{k} + \\alpha \\nabla_{y} F_r(x^{k+1}, y^{k},z^k)-y^{k+1}, y^{k+1}-y^{k}\\rangle +\\frac{1}{\\alpha}\\|y^{k}-y^{k+1}\\|^2\\\\\n     \\ge\\ &  \\frac{1}{\\alpha}\\|y^{k}-y^{k+1}\\|^2, \n\\end{align*}\nwhere the last inequality follows from the property of the projection operator and the dual update $\ny^{k+1} = \\proj_{\\mathcal{Y}}( y^{k} + \\alpha \\nabla_{y} F_r(x^{k+1}, y^{k},z^k))\n$ (recall that $\\nabla_y F_r(x,y,z) = \\nabla_y F(x,y)$ for any $x \\in \\mathcal{X}$, $y \\in \\mathcal{Y}$, and $z \\in \\mathbb{R}^n$). For the remaining terms, we have\n                \\begin{align*}\n        %\\label{desascent-key3}\n         &2\\langle\\nabla_{y} F_r(x_r(y^{k}, z^{k}), y^{k}, z^{k})-\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k+1}-y^{k}\\rangle \\notag\\\\\n         \\ge & -2\\|\\nabla_{y} F_r(x_r(y^{k}, z^{k}),y^{k},z^{k})-\\nabla_{y} F_r(x^{k+1}, y^{k},z^{k})\\| \\cdot\\|y^{k+1}-y^{k}\\| \\notag\\\\\n         \\ge & -2 L\\|x^{k+1}-x_r(y^{k}, z^{k})\\| \\cdot\\|y^{k}-y^{k+1}\\| \\notag\\\\\n        \\ge  & -L \\zeta^2\\|y^{k}-y^{k+1}\\|^{2}-L\\zeta^{-2} \\|x^{k+1}-x_r(y^{k}, z^{k})\\|^{2} \\notag\\\\\n         \\ge & -L \\zeta^2\\|y^{k}-y^{k+1}\\|^{2}-L\\|x^{k+1}-x^{k}\\|^{2},\n        \\end{align*}\n        where the third inequality \n        % follows from $2|x||y| \\leq \\tau x^2 + \\frac{1}{\\tau}y^2$ \n        holds because $2 |a| |b| \\le \\tau a^2 + \\tfrac{1}{\\tau} b^2$\n        for any $a,b \\in \\mathbb{R}$ and $\\tau>0$, and the last inequality follows from Proposition \\ref{prop:lip}. \n        % where the first inequality is because of the Cauchy-Schwarz in equality, the second inequality is because $\\nabla_{y} K=\\nabla_{y} f$ is $L$-Lipschitz-continuous, the third inequality is due to the AM-GM inequality and the last is because of (B.5).\n% Subsequently, we focus on the left term in $\\text{\\ding{172}}$ that \nPutting the above together, we obtain\n\\begin{equation}\n\\label{desascent-key3}\n    \\text{\\ding{172}}  \\ge \\left(\\frac{1}{\\alpha}-L\\zeta^2\\right) \\|y^{k}-y^{k+1}\\|^{2}-L\\|x^{k+1}-x^{k}\\|^{2}. \n\\end{equation}\nNext, we bound the term $\\text{\\ding{173}}$ by \n\\begin{align}\\label{desascent-key2}\n  \\text{\\ding{173}} \n =\\ & 2 r \\left\\langle z^{k+1}-z^{k}, x_r(y(z^{k+1}), z^{k})-x_r(y^{k+1}, z^{k+1})\\right\\rangle \\notag\\\\\n =\\ & 2 r \\left\\langle z^{k+1}-z^{k}, x_r(y(z^{k+1}), z^{k})-x_r(y(z^{k+1}), z^{k+1})\\right\\rangle\\, \\notag\\\\\n & + 2 r\\left\\langle z^{k+1}-z^{k}, x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\right\\rangle\\notag \\\\\n\\ge &  -2r \\sigma_1\\|z^{k+1}-z^k\\|^2+2 r\\left\\langle z^{k+1}-z^{k}, x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\right\\rangle \\notag\\\\\n\\ge & -2r \\sigma_1\\|z^{k+1}-z^k\\|^2-\\frac{r}{7\\beta}\\|z^{k+1}-z^k\\|^2  -7r\\beta \\|x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\|^2, \n\\end{align}\nwhere the first inequality is due to \\eqref{lip-z} and the Cauchy-Schwarz inequality, and the second inequality again follows from the fact that $2 |a| |b| \\le \\tau a^2 + \\tfrac{1}{\\tau} b^2$ for any $a,b \\in \\mathbb{R}$ and $\\tau > 0$. Now, the inequalities \\eqref{desascent-key3} and \\eqref{desascent-key2} imply that\n\\begin{align}\\label{desas-keymid}\n        & \\Phi_r(x^k,y^k,z^k)-\\Phi_r(x^{k+1},y^{k+1},z^{k+1}) \\notag\\\\\n        \\geq\\ & \\frac{2\\lambda^{-1}+r-3L}{2}\\|x^{k}-x^{k+1}\\|^{2}+\\left(\\frac{1}{ \\alpha}-L_{d_r}-\\frac{L}{2}-L\\zeta^2\\right)\\|y^{k}-y^{k+1}\\|^{2}\\, \\notag\\\\\n        &+\\left(\\frac{(2-\\beta)r}{2 \\beta}-2r\\sigma_1-\\frac{r}{7\\beta}\\right)\\|z^{k}-z^{k+1}\\|^{2}-7r\\beta \\|x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\|^2. \n\\end{align}\nBy Lemma \\ref{lemma-dualxy} and the fact that $\\| u+v \\|^2 \\le 2(\\|u\\|^2+\\|v\\|^2)$ for any $u,v \\in \\mathbb{R}^d$, we have\n\\begin{align}\\label{desas-final-key1}\n\\|y^{k+1}-y^{k}\\|^{2} \n&=\\|y^{k+1}-y_{+}^{k}(z^{k+1})+y_{+}^{k}(z^{k+1})-y^{k}\\|^{2}\\notag\\\\\n&\\geq\\frac{1}{2}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2} -\\|y^{k+1}-y_{+}^{k}(z^{k+1})\\|^{2}\\notag\\\\\n% & \\geq \\frac{1}{2}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2} -\\alpha^2L^2\\|x^{k+1}-x_r(y^k,z^{k+1})\\|^2\\notag\\\\\n& \\geq \\frac{1}{2}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2} -2\\eta^2\\|x^{k+1}-x^k\\|^2-2\\sigma_1^2\\alpha^2L^2\\|z^{k+1}-z^k\\|^2.\n\\end{align}\n% where we recall that \n% \\[\n% y_{+}^{k}(z^{k+1}) = \\proj_{\\mathcal{Y}}(y^k+\\alpha \\nabla_y F_r (x_r(y^k,z^{k+1}),y^k,z^{k+1})).\n% \\]\n% \\leq\\ & \\alpha L\\|x^{k+1}-x_r(y^{k}, z^{k})\\| \\notag \\\\\n% \\leq\\ & \\eta\\|x^{k}-x^{k+1}\\|,\nSimilarly, by Lemma \\ref{lemma-sollip} and Lemma \\ref{lemma-dualxy}, we have\n\\begin{align}\\label{desas-final-key2}\n&\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\|^{2}\\notag\\\\\n\\leq \\ & 2\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_+^{k}(z^{k+1}),z^{k+1})\\|^2   + 2\\|x_r(y_+^{k}(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\|^2 \\notag\\\\\n\\leq \\ & 2\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_+^{k}(z^{k+1}),z^{k+1})\\|^2+2\\sigma_2^2 \\|y^{k+1}-y_{+}^{k}(z^{k+1})\\|^{2}\\notag\\\\\n\\leq \\ & 2\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_+^{k}(z^{k+1}),z^{k+1})\\|^2 + 4\\sigma_2^2\\eta^2 \\|x^{k+1}-x^k\\|^2+4\\sigma_2^2\\sigma_1^2\\alpha^2L^2\\|z^{k+1}-z^k\\|^2.\n\\end{align}\n\n% \\begin{align}\\label{desas-final-key2}\n% &\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\|^{2}\\notag\\\\\n% % =\\ & \\|x_r^\\star(z^{k+1})-x_r^\\star(z^{k})+x_r^\\star(z^{k})-x_r(y_{+}^{k}(z^{k}), z^{k})+x_r(y_{+}^{k}(z^{k}), z^{k})-x_r(y^{k+1}, z^{k})+x_r(y^{k+1}, z^{k})-x_r(y^{k+1}, z^{k+1})\\|^2 \\notag\\\\\n% \\leq\\ & 4\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y(z^{k}), z^{k})\\|^{2}+4\\|x_r(y(z^{k}), z^{k})-x_r(y_{+}^{k}(z^{k}), z^{k})\\|^{2}\\ +\\notag \\\\\n% &4\\|x_r(y_{+}^{k}(z^{k}), z^{k})-x_r(y^{k+1}, z^{k})\\|^{2}+4\\|x_r(y^{k+1}, z^{k})-x_r(y^{k+1}, z^{k+1})\\|^{2} \\notag\\\\\n% \\leq\\ & 8 \\sigma_{1}^{2}\\|z^{k}-z^{k+1}\\|^{2}+4\\|x_r(y(z^{k}), z^{k})-x_r(y_{+}^{k}(z^{k}), z^{k})\\|^{2} +4 \\sigma_{2}^{2} \\eta^2\\|x^{k}-x^{k+1}\\|^2.\n% \\end{align}\n% Besides, we recall that \n% \\begin{align}\\label{desas-final-key0}\n% x^{k+1}-z^{k} =\\frac{1}{\\beta}(z^{k+1}-z^k).\n% \\end{align} \nSubstituting \\eqref{desas-final-key1} and \\eqref{desas-final-key2} into \\eqref{desas-keymid} yields\n\\begin{align*}\n&\\Phi_r(x^k,y^k,z^k)-\\Phi_r(x^{k+1},y^{k+1},z^{k+1})\\notag\\\\\n \\geq\\ &\\left( \\frac{2\\lambda^{-1}+r-3L}{2} -28 r\\beta \\sigma_2^2\\eta^2 \\right)\\|x^{k}-x^{k+1}\\|^{2}\\ +\\left(\\frac{1}{ \\alpha}-L_{d_r}-\\frac{L}{2}-L\\zeta^2\\right) \\cdot \\\\\n& \\left(\\frac{1}{2}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2} -2\\eta^2\\|x^{k+1}-x^k\\|^2-2\\sigma_1^2\\alpha^2L^2\\|z^{k+1}-z^k\\|^2 \\right)\\ \\notag\\\\\n&+ \\left(\\frac{(2-\\beta)r}{2 \\beta}-2r\\sigma_1-\\frac{r}{7\\beta}-28r\\beta\\sigma_1^2\\sigma_2^2 \\alpha^2 L^2 \\right)\\|z^{k}-z^{k+1}\\|^{2} \\\\  & -  14r\\beta\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_+^{k}(z^{k+1}),z^{k+1})\\|^2.\n\\end{align*}\n\nSuppose that $r\\ge 3L$,\n% (which implies $L_{d_{r}}+\\frac{L}{2} \\leq 5L$)\nwhich implies that $L_{d_{r}}+\\frac{L}{2} \\leq 5L$. We observe the following:\n\\begin{itemize}\n\\item As $\\alpha \\leq \\min\\left\\{\\frac{1}{10L}, \\frac{1}{4L\\zeta^2} \\right\\}$, we have $\\frac{1}{ \\alpha}-L_{d_r}-\\frac{L}{2} \\ge \\frac{1}{2\\alpha}$ and \n$\n\\frac{1}{2\\alpha} - L\\zeta^2 \\ge \\frac{1}{4\\alpha}. \n$\n\\item As $\\beta\\leq\\frac{1}{28}$ and $\\sigma_1 \\leq \\frac{3}{2}$, $\\sigma_2\\leq \\frac{5}{2}$, we have \n\\begin{align*}\n& \\frac{(2-\\beta)r}{2 \\beta}-2r\\sigma_1-\\frac{r}{7\\beta}-28r\\beta\\sigma_1^2 \\sigma_2^2 \\alpha^2 L^2  -\\frac{1}{2\\alpha} \\sigma_1^2\\alpha^2L^2 \\\\ \n\\ge\\ & \\frac{6r}{7\\beta} -\\frac{7r}{2}- 28r\\beta\\left(\\frac{3}{2}\\right)^2\\left(\\frac{5}{2}\\right)^2\\left(\\frac{1}{10L}\\right)^2L^2-\\frac{r}{24}\\\\ \n\\ge\\ &  \\frac{r}{\\beta}\\left(\\frac{6}{7}-4\\beta- 14\\beta^2\\right) \\ge \\frac{4r}{7\\beta}.\n\\end{align*}\n\\item As $\\lambda^{-1} \\ge L$ and recalling that $\\eta = \\alpha L \\zeta$, we have \n\\[\\alpha\\leq\\frac{1}{4L \\zeta^2} = \\frac{\\lambda^{-1}}{4L^2\\zeta^2} \\frac{L}{\\lambda^{-1}} \\leq \\frac{\\lambda^{-1}}{4L^2\\zeta^2}\\quad \\text{and}\\quad \n\\frac{\\eta^2}{2\\alpha} = \\frac{\\alpha L^2\\zeta^2}{2} \\leq \\frac{1 }{8\\lambda}.\n\\]\nMoreover, since $\\beta  \\leq \\tfrac{(r-L)^2}{14 \\alpha r (2r-L)^2} = \\tfrac{1}{14 \\alpha r \\sigma_2^2}$,\n we have \n \\[\n 28 r\\beta \\sigma_2^2\\eta^2 \\leq \\frac{ 2\\eta^2}{\\alpha} \\leq \\frac{1}{2\\lambda}\\quad\\text{and}\\quad \\frac{2\\lambda^{-1}+r-3L}{2} -28 r\\beta \\sigma_2^2\\eta^2 - \\frac{\\eta^2}{2\\alpha} \\ge \\frac{3}{8\\lambda}.\n\\]\n\\end{itemize}\n% Together all pieces, we get \nPutting everything together yields\n\\begin{align*}\n&\\Phi_r(x^k,y^k,z^k)-\\Phi_r(x^{k+1},y^{k+1},z^{k+1}) \\\\\n\\ge\\ & \\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{8\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2} +\\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2} \\\\  & -14r\\beta\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2}.\n\\end{align*}\nThe proof is complete.\n% \\end{proof}\n\n\n\\begin{proof}\nThe proof follows closely that of Proposition \\ref{prop:dual_eb_KL}. Let $\\psi:\\R^n\\times \\R^n\\rightarrow\\R$ be the function defined by\n\\[\n\\psi(x,z) = \\max\\limits_{y\\in \\mathcal{Y}} F_r(x,y,z)\n\\]\nand consider arbitrary $x \\in \\mathcal{X}$, $y \\in \\mathcal{Y}$, and $z \\in \\mathbb{R}^n$. Again, note that  $\\psi(\\cdot,z)$ is $(r-L)$-strongly convex. This implies that\n% for any $x\\in \\mathcal{X}$, we have\n\\begin{equation}\n\\label{OS-GS-better-key1}\n\\psi(x,z) - \\psi(x^\\star_r(z),z) \\ge \\frac{r-L}{2}\\|x-x^\\star_r(z)\\|^2. \n\\end{equation}\nMoreover, \n% for any $x\\in \\mathcal{X}$, \nwe  have\n\\begin{align} \n\\label{OS-GS-better-key2}\n&\\psi(x,z)-\\psi(x_r^\\star(z),z) \\notag\\\\   \n\\leq \\  &\\psi(x,z)-F_r(x_r(y_+(z),z),y_+(z),z)\\notag \\\\\n= \\ & \\max_{y^\\prime\\in \\mathcal{Y}} F(x,y^\\prime) + \\frac{r}{2}\\|x-z\\|^2-F_r(x_r(y_+(z),z),y_+(z),z)\\notag\\\\\n=\\,& \\max_{y^\\prime\\in \\mathcal{Y}}F(x,y^\\prime)-F(x_r(y_+(z),z),y_+(z)) + \\frac{r}{2}\\|x-z\\|^2-\\frac{r}{2}\\|x_r(y_+(z),z)-z\\|^2,\n\\end{align}\nwhere the  inequality follows from\n\\begin{align*}\nF_r(x_r(y_+(z),z),y_+(z),z) \n=\\ & \\min_{x'\\in\\mathcal{X}}\\left\\{ F(x',y_+(z)) +\\frac{r}{2}\\|x'-z\\|^2\\right\\} \\\\\n\\leq\\ & \\max_{y' \\in \\mathcal{Y}}\\min_{x'\\in\\mathcal{X}}\\left\\{F(x',y') +\\frac{r}{2}\\|x'-z\\|^2\\right\\} \\\\\n\\leq  \\ & \\min_{x'\\in\\mathcal{X}}\\max_{y' \\in \\mathcal{Y}}\\left\\{F(x',y') +\\frac{r}{2}\\|x'-z\\|^2\\right\\}\n= \\psi(x_r^\\star(z),z).\n\\end{align*}\nAs \\eqref{OS-GS-better-key1} and \\eqref{OS-GS-better-key2} hold for any $x\\in\\mathcal{X}$, we obtain the intermediate relation\n\\begin{equation*}\n% \\label{OS-GS-better-key3} \n    \\frac{r-L}{2}\\|x_r^\\star(z)-x_r(y_+(z),z)\\|^2 \\leq \\max\\limits_{y^\\prime\\in \\mathcal{Y}} F(x_r(y_+(z),z),y^\\prime) - F(x_r(y_+(z),z),y_+(z)) \n\\end{equation*}\nby taking $x= x_r(y_+(z),z)$.\n\nThe remaining steps of the proof are the same as those of Proposition \\ref{prop:dual_eb_KL}. For brevity, we omit them here.\n\\end{proof}\n\n\\begin{lemma}\\label{lemma-dual-bd}\nFor any $y \\in \\mathcal{Y}$ and $z\\in\\R^n$, we have\n\\begin{equation*}\n% \\label{dual-bd}\n\\|x^\\star_r(z)-x_r(y_{+}(z), z)\\|^{2} \n \\leq \\kappa \\|y-y_{+}(z)\\| ,\n\\end{equation*}\nwhere $\\kappa:=\\frac{1+ \\alpha L\\sigma_2+\\alpha L}{\\alpha(r-L)}\\cdot{\\rm diam}(\\mathcal{Y})$.\n\\end{lemma}\n\n\\begin{proof}\nLet $y \\in \\mathcal{Y}$ and $z \\in \\mathbb{R}^n$ be arbitrary. Recall that $y(z)$ is an arbitrary vector in $Y(z)$. By the $(r-L)$-strong convexity of $F_r(\\cdot, y, z)$, we have\n\\begin{align}\n&F_r(x_r^\\star(z), y_{+}(z), z)-F_r(x_r(y_{+}(z), z), y_{+}(z), z)  \\geq \\frac{r-L}{2}\\|x_r(y_{+}(z), z)-x^\\star_r(z)\\|^{2}, \\notag\\\\\n&F_r(x_r(y_{+}(z), z), y(z), z)-F_r(x_r^\\star(z), y(z), z)  \\geq \\frac{r-L}{2}\\|x_r(y_{+}(z), z)-x_r^\\star(z)\\|^{2}.\\notag\n\\end{align}\n Moreover, since $(x_r^\\star(z),y(z))$ is a saddle point of the convex-concave function $F_r(\\cdot,\\cdot,z)$ \\citep{sion1958general}, we have $F_r(x_r^\\star(z), y(z), z) \\geq F_r(x_r^\\star(z), y_{+}(z), z)$, which implies that\n\\begin{equation}\\label{dualbd-strcvx}\nF_r(x_r(y_{+}(z), z), y(z), z)-F_r(x_r(y_{+}(z), z), y_{+}(z), z) \\geq  (r-L)\\|x_r(y_{+}(z), z)-x_r^\\star(z)\\|^{2}.\n\\end{equation}\nNote that $y_{+}(z)$ is the maximizer of \n\\[\n\\max_{y' \\in \\mathcal{Y}}\\big\\langle y+\\alpha \\nabla_{y} F_r(x_r(y, z), y, z)-y_{+}(z) , y'\\big\\rangle.\n\\]\nFor simplicity, we define the function $\\xi: \\mathbb{R}^d \\rightarrow \\mathbb{R}$ by\n\\begin{align*}\n\\xi(\\cdot)\n:=\\ &\\alpha F_r(x_r(y_{+}(z), z),\\cdot, z)-\\big\\langle \\alpha\\nabla_{y} F_r(x_r(y_{+}(z), z), y_{+}(z), z),\\cdot\\big\\rangle \\\\ \\ &-\\big\\langle y_{+}(z)-y-\\alpha \\nabla_{y} F_r(x_r(y, z), y, z), \\cdot\\big\\rangle.\n\\end{align*}\nThen, we have\n\\begin{align*}\n\\max_{y' \\in \\mathcal{Y}}\\ \\xi(y')\n&\\leq \\alpha F_r(x_r(y_{+}(z), z),y_+(z), z)-\\big\\langle\\alpha\\nabla_y F_r(x_r(y_{+}(z), z),y_+(z), z), y_+(z)\\big\\rangle\\, \\notag\\\\\n&\\quad + \\max_{y' \\in \\mathcal{Y}} \\big\\langle y+\\alpha \\nabla_{y} F_r(x_r(y, z), y, z)-y_{+}(z), y'\\big\\rangle \\notag\\\\\n&\\leq \\xi(y_{+}(z)),\n\\end{align*}\nwhere the first inequality holds because $F_r(x_r(y_{+}(z), z),\\cdot,z)$ is concave. Therefore, we have $\\xi(y(z))\\leq \\xi(y_{+}(z))$, which implies that\n\\begin{align}\\label{dualbd-key}\n&F_r(x_r(y_{+}(z), z), y(z), z)-F_r(x_r(y_{+}(z), z), y_{+}(z), z) \\notag\\\\\n\\leq &\\ \\frac{1}{\\alpha}\\Big\\langle y(z)-y_{+}(z), \\alpha\\nabla_{y} F_r(x_r(y_{+}(z), z), y_{+}(z), z)-\\alpha \\nabla_{y} F_r(x_r(y, z), y, z)\\Big\\rangle \\notag \\\\ \\ & \\ + \\frac{1}{\\alpha}\\Big\\langle y(z)-y_{+}(z), y_{+}(z)-y\\Big\\rangle \\notag\\\\\n\\leq &\\ L \\|y_{+}(z)-y(z)\\|(\\|x_r(y_{+}(z), z)-x_r(y, z)\\|+ \\|y_{+}(z)-y\\|)  + \\frac{1}{\\alpha}\\|y_{+}(z)-y(z)\\|\\cdot \\|y-y_{+}(z)\\|\\notag\\\\\n\\leq &\\ \\left(\\frac{1}{\\alpha}+L\\sigma_2+L\\right)\\|y_{+}(z)-y(z)\\| \\cdot\\|y-y_{+}(z)\\|.\n\\end{align}\nHere, the second inequality is due to the $L$-Lipschitz continuity of $\\nabla_y F_r(\\cdot,\\cdot,z)$ and the last inequality is from \\eqref{lip-y}.\nHence, by combining \\eqref{dualbd-strcvx} and \\eqref{dualbd-key}, we obtain\n\\[\n(r-L)\\|x_r^\\star(z)-x_r(y_{+}(z), z)\\|^{2}\\leq \\left(\\frac{1}{\\alpha}+ L\\sigma_2+L\\right){\\rm diam}(\\mathcal{Y})\\cdot\\|y-y_{+}(z)\\|,\n\\]\nwhich proves the desired result.\n\\end{proof}","units":[{"start":46206,"end":47355,"start_line":661,"end_line":676,"raw_tex":"\\begin{proposition}[Basic descent estimate of $\\Phi_r$]\n\\label{prop:decrease}\n    Let \n    % $r\\ge3L$, $\\lambda\\le \\frac{1}{L}$, $\\alpha \\leq \\min\\left\\{\\frac{1}{10L}, \\frac{1}{4L\\zeta^2} \\right\\}$, and $\\beta  \\leq \\min \\left\\{\\frac{1}{28}, \\frac{(r-L)^2}{14\\alpha r(2r-L)^2}\\right\\}$,     \n    \\[r\\ge3L,\\ \\lambda\\le \\frac{1}{L},\\ \\alpha \\leq \\min\\left\\{\\frac{1}{10L},\\ \\frac{1}{4L\\zeta^2} \\right\\},\\ \\beta  \\leq \\min \\left\\{\\frac{1}{28},\\ \\frac{(r-L)^2}{14\\alpha r(2r-L)^2}\\right\\},\\] \n    where $\\zeta>0$ is the constant defined in Proposition \\ref{prop:lip}.\n    Then, for any $k\\ge0$, we have\n\\begin{equation*}\n\\begin{aligned}\n\\Phi_r^k-\\Phi_r^{k+1}\\ge\\ &  \\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{8\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 +\\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2} \\\\\n& - 14r\\beta\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2},\n\\end{aligned}\n\\end{equation*}\nwhere $\\Phi_r^k:= \\Phi_r(x^k,y^k,z^k)$. \n% \\footnote{The proof of Proposition \\ref{prop:decrease}, which is based on the primal descent, dual ascent, and proximal descent properties, is given in Appendix \\ref{sec:suff_decrease}. }\n\\end{proposition}","sha256_utf8":"86abbffd20190bd6193654e966d5e253dd7f7f1e02c236ccfa73dfcd87017a32"},{"start":51418,"end":51862,"start_line":731,"end_line":740,"raw_tex":"\\begin{proposition} [Lipschitz-type primal error bound]\n\\label{prop:lip}\n%  Suppose that $r-L > 0$. Then \n For any $k\\ge0$, we have\n    \\begin{equation}\\label{iter-result1}\n        \\|x^{k+1}-x_r(y^{k}, z^{k})\\| \\leq \\zeta\\|x^{k}-x^{k+1}\\|,\n        \\end{equation}\nwhere   $\\zeta: =\\frac{2(r-L)^{-1}+(\\lambda^{-1}+L) ^{-1}}{(\\lambda^{-1}+L) ^{-1}} \\left(\\sqrt{\\frac{2L}{\\lambda^{-1}+L}}+1\\right)$.\n% and $\\eta:=\\alpha L \\zeta$. \n\\end{proposition}","sha256_utf8":"9af2994d7f47588252dd8a5fa56e2740c866e3a1a2d6da2caca70e95959059f2"},{"start":51863,"end":57479,"start_line":741,"end_line":810,"raw_tex":"\\begin{proof}\nFor ease of notation, we define $\\hat{F}_{y,z}(\\cdot): = F_r(\\cdot,y,z) + \\iota_{\\mathcal{X}}(\\cdot)$ for any $y\\in\\mathcal{Y}$ and $z\\in\\R^n$.  As $x_r(y,z)$ is the optimal solution of $\\min_{x \\in \\mathbb{R}^n} \\hat{F}_{y,z} (x)$ and $F_r(\\cdot,y,z)$ is $(r-L)$-strongly convex (see the remark after Fact \\ref{fact-2} in Appendix \\ref{sec:lemmas}), we have\n\\begin{equation}\n    \\label{eq:lip_1} \n    \\hat{F}_{y^k,z^k}(x)-\\hat{F}_{y^k,z^k}(x_r(y^k,z^k)) \\ge \\frac{r-L}{2} \\|x-x_r(y^k,z^k)\\|^2 \\quad \\text{for all}\\ x \\in \\mathcal{X}.\n\\end{equation}\nIn addition, by the convexity of $\\hat{F}_{y^k,z^k}$, we obtain \n\\begin{equation}\n    \\label{eq:lip_2}\n    \\begin{aligned}\n    \\hat{F}_{y^k,z^k}(x)-\\hat{F}_{y^k,z^k}(x_r(y^k,z^k)) & \\leq \\dist(0, \\partial \\hat{F}_{y^k,z^k}(x))\\cdot \\|x-x_r(y^k,z^k)\\| \\\\ \n    \\end{aligned}\n\\end{equation}\n$\\text{for all}\\ x\\in\\mathcal{X}$.\nCombining \\eqref{eq:lip_1} and \\eqref{eq:lip_2} yields\n\\[\n\\|x-x_r(y^k,z^k)\\| \\leq \\frac{2}{r-L} \\dist(0, \\partial \\hat{F}_{y^k,z^k}(x)) \\quad \\text{for all}\\ x\\in\\mathcal{X}. \n\\]\nBy utilizing the equivalence between proximal and subdifferential error bounds \\citep[Theorem 3.4]{drusvyatskiy2018error}, we  conclude that\n\\begin{equation}\n\\label{eq:;lip_3}\n    \\|x-x_r(y^k,z^k)\\|  \\leq \\frac{2(r-L)^{-1}+t}{t} \\left\\|x - \\prox_{t  \\hat{F}_{y^k,z^k}}(x)\\right\\| \\text{ for all } t>0, \\, x\\in\\mathcal{X}.\n\\end{equation}\nNow, let us examine the relationship between  $\\|x^{k+1}-x^k\\|$ and $\\|x^{k+1} - \\prox_{t \\hat{F}_{y^k,z^k}}(x^{k+1})\\|$. Let $\\varphi_k:\\R^n\\rightarrow \\R \\cup \\{+\\infty\\}$ be the function defined by\n\\[\n\\varphi_k(x):=\\hat{F}_{y^k,z^k}(x)+\\frac{\\lambda^{-1}+L}{2}\\|x-x^k\\|^{2}-\\frac{\\lambda^{-1} + L}{2}\\|x-x^{k+1}\\|^{2}.\n\\]\n% By combining Fact \\ref{fact-2} and the strong convexity of $F_{x^k,\\lambda}(\\cdot,y^k)$\nIt is clear that for any $x\\in\\mathcal{X}$,\n\\begin{align*}\\label{safeguard-key1}\n            & \\hat{F}_{y^k,z^k}(x)\n             = F(x,y^k) + \\frac{r}{2}\\|x-z^k\\|^2 \\notag\\\\\n             \\ge\\ & F_{x^k,\\lambda}(x,y^k) -\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k}\\|^2 + \\frac{r}{2}\\|x-z^k\\|^2\\notag \\\\ \n              \\ge\\ &  F_{x^k,\\lambda}(x^{k+1},y^k) +\\frac{\\lambda^{-1}+r}{2}\\|x-x^{k+1}\\|^2 -\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k}\\|^2 +\\frac{r}{2}\\|x^{k+1}-z^k\\|^2\\notag\\\\\n             \\ge\\ & F(x^{k+1},y^k) + \\frac{\\lambda^{-1}-L}{2}\\|x^{k}-x^{k+1}\\|^2 +\\frac{\\lambda^{-1}+r}{2}\\|x-x^{k+1}\\|^2\\ \\\\\n             & \\ -\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k}\\|^2 + \\frac{r}{2}\\|x^{k+1}-z^k\\|^2 \\notag \\\\ \n             =\\ & \\hat{F}_{y^k,z^k}(x^{k+1})+\\frac{\\lambda^{-1}-L}{2}\\|x^{k+1}-x^k\\|^2 +\\frac{\\lambda^{-1}+r}{2}\\|x-x^{k+1}\\|^2 -\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k}\\|^2,\n\\end{align*}\nwhere the first and third inequalities follow from Fact \\ref{fact-2}, and the second one is from the $(\\lambda^{-1} + r)$-strong convexity of $F_{x^k,\\lambda}(\\cdot,y^k) +\\frac{r}{2}\\|\\cdot-z^k\\|^2$ on $\\mathcal{X}$ and the definition of $x^{k+1}$.\nSince $r > L$, we see that for any $x\\in\\mathcal{X}$,\n\\begin{align*}\n\\varphi_k(x)\n&\\ge \\hat{F}_{y^k,z^k}(x^{k+1})+\\frac{\\lambda^{-1}-L}{2}\\|x^{k+1}-x^{k}\\|^2+\\frac{r-L}{2}\\|x-x^{k+1}\\|^2\\notag\\\\\n&\\ge\\hat{F}_{y^k,z^k}(x^{k+1})+\\frac{\\lambda^{-1}-L}{2}\\|x^{k+1}-x^{k}\\|^2.\n\\end{align*}\nIt then follows from the definition of $\\varphi_k$ that \n\\[\\varphi_k(x^{k+1})-\\inf_{x\\in\\R^n} \\varphi_k(x)\\leq L \\|x^{k+1}-x^k\\|^2.\\]\nThis, together with the definition of the proximal mapping, implies that for any $\\rho>0$,\n\\begin{align}\n&\\varphi_k\\left(\\operatorname{prox}_{\\frac{1}{\\rho} \\varphi_k}(x^{k+1})\\right)+\\frac{\\rho}{2}\\|\\operatorname{prox}_{\\frac{1}{\\rho} \\varphi_k}(x^{k+1})-x^{k+1}\\|^{2}\\leq  \\varphi_k(x^{k+1})\\notag\\\\\n\\leq\\ & \\inf_{x\\in\\R^n} \\varphi_k(x)+L \\|x^{k+1}-x^k\\|^2\n\\leq \\varphi_k\\left(\\operatorname{prox}_{\\frac{1}{\\rho} \\varphi_k}(x^{k+1})\\right)+L \\|x^{k+1}-x^k\\|^2,\\notag\n\\end{align}\nwhich in turn implies that\n\\begin{equation}\\label{safeguard-key111}\n\\|\\operatorname{prox}_{\\frac{1}{\\rho} \\varphi_k}(x^{k+1})-x^{k+1}\\|\\leq \\sqrt{\\frac{2L}{\\rho}} \\|x^{k+1}-x^k\\|.    \n\\end{equation}\nRecall that our goal is to find a relationship between  $\\|x^{k+1}-x^k\\|$ and $\\|x^{k+1} - \\prox_{t \\hat{F}_{y^k,z^k}}(x^{k+1})\\|$. To do this, it suffices to relate $\\prox_{t \\hat{F}_{y^k,z^k}}(x^{k+1})$ to $\\operatorname{prox}_{\\frac{1}{\\rho} \\varphi_k}(x^{k+1})$.  By definition, we have\n\\[\\hat{F}_{y^k,z^k}(x)+\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k}\\|^{2}=\\varphi_k(x)+\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k+1}\\|^{2},\\]\nwhich implies that $\\operatorname{prox}_{\\frac{1}{\\lambda^{-1}+L} \\varphi_k}(x^{k+1})=\\operatorname{prox}_{\\frac{1 }{\\lambda^{-1}+L}\\hat{F}_{y^k,z^k}}(x^k)$. \nBy setting $t = (\\lambda^{-1} + L)^{-1}$ in \\eqref{eq:;lip_3} and $\\rho = \\lambda^{-1} + L$ in \\eqref{safeguard-key111}, we conclude that\n\\begin{align*}\n    &\\|x^{k+1}-x_r(y^{k}, z^{k})\\| \\notag\\\\\n    \\leq\\ & \\frac{2(r-L)^{-1}+(\\lambda^{-1}+L) ^{-1}}{(\\lambda^{-1}+L) ^{-1}}\\left\\|x^{k+1}- \\prox_{\\frac{1}{\\lambda^{-1}+L} \\hat{F}_{y^k,z^k}}(x^{k+1})\\right\\| \\\\\n    \\leq\\ &   \\frac{2(r-L)^{-1}+(\\lambda^{-1}+L) ^{-1}}{(\\lambda^{-1}+L) ^{-1}}\\left(\\left\\|x^{k+1}- \\prox_{\\frac{1}{\\lambda^{-1}+L} \\hat{F}_{y^k,z^k}}(x^{k})\\right\\|+ \\|x^{k+1}-x^k\\|\\right) \\\\\n     =\\ &   \\frac{2(r-L)^{-1}+(\\lambda^{-1}+L) ^{-1}}{(\\lambda^{-1}+L) ^{-1}}\\left(\\left\\|x^{k+1}- \\prox_{\\frac{1}{\\lambda^{-1}+L} \\varphi_k}(x^{k+1})\\right\\|+ \\|x^{k+1}-x^k\\|\\right) \\\\\n    \\leq\\ & \\frac{2(r-L)^{-1}+(\\lambda^{-1}+L) ^{-1}}{(\\lambda^{-1}+L) ^{-1}} \\left(\\sqrt{\\frac{2L}{\\lambda^{-1}+L}}+1\\right) \\|x^{k+1}-x^k\\|,\n\\end{align*}\nwhere the second inequality is due to the nonexpansiveness of the proximal mapping. The proof is complete.\n\\end{proof}","sha256_utf8":"a6423bde18fe6e8ef63f4e8bbc2615ca03fa20925b9ba370949d8808e76c22e5"},{"start":59051,"end":60090,"start_line":823,"end_line":836,"raw_tex":"\\begin{proposition}[Dual error bound with K\\L{} exponent] \n\\label{prop:dual_eb_KL}\n% Let $\\delta>0$. \nSuppose that Assumption \\ref{ass:kl} holds. Then, for any $y\\in\\mathcal{Y}$ and $z \\in \\mathbb{R}^n$, we have\n% \\[\n% \\|x_r^\\star(z)-x_r(y_+(z),z)\\| \\leq \\omega \\|y-y_+(z)\\|^{\\frac{1}{2\\theta}},\n% \\]\n% where $\\omega:=\\frac{\\sqrt{2}}{\\sqrt{r-L}}\\left(\\frac{1+\\alpha L(1+\\sigma_2)}{\\alpha\\mu}\\right)^{\\frac{1}{2\\theta}}$.\n\\begin{enumerate}[label=\\normalfont(\\alph*)]\n \\item \\rm{(K\\L\\ exponent $\\theta=0$):} $\\|x_r(y(z),z)-x_r(y_+(z),z)\\| \\leq \\omega_1 \\|y-y_+(z)\\|$, \n  \\item \\rm{(K\\L\\ exponent $\\theta\\in(0,1)$):} $\\|x_r(y(z),z)-x_r(y_+(z),z)\\| \\leq \\omega_2\\|y-y_+(z)\\|^{\\frac{1}{2\\theta}}$, \n  \\end{enumerate}\n    where $\\omega_1:= \\frac{\\sqrt{2L\\cdot {\\rm diam}(\\mathcal{Y})}\\cdot (1+\\alpha L)}{\\alpha\\mu\\sqrt{r-L}}$ and $\\omega_2:=\\frac{\\sqrt{2}}{\\sqrt{r-L}}\\left(\\frac{1+\\alpha L(1+\\sigma_2)}{\\alpha\\mu}\\right)^{\\frac{1}{2\\theta}}$ (recall that $y_+(z) = \\proj_{\\mathcal{Y}} ( y + \\alpha \\nabla_y F(x_r(y,z), y)$).  \n\\end{proposition}","sha256_utf8":"0fd1bfcc25854866cf9ef3e5a269afeeb493832130cbd82c99fe6db3cfd5dca1"},{"start":60091,"end":65950,"start_line":837,"end_line":932,"raw_tex":"\\begin{proof}\nLet $\\psi:\\R^n\\times \\R^n\\rightarrow\\R$ be the function defined by\n\\[\n\\psi(x,z) =  F_r(x,y(z),z). \n\\]\nConsider arbitrary $x \\in \\mathcal{X}$, $y \\in \\mathcal{Y}$, and $z \\in \\mathbb{R}^n$. Note that the function $\\psi(\\cdot,z)$ is $(r-L)$-strongly convex. Since\n\\begin{align*}\n\\mathop{\\argmin}_{x'\\in\\mathcal{X}}\\psi(x',z) & =\\mathop{\\argmin}_{x'\\in\\mathcal{X}} F_r(x',y(z),z) =  x_r(y(z),z),\n\\end{align*}\nwe see that \n% for any $x\\in \\mathcal{X}$, \n\\begin{equation}\n\\label{eq:dual1}\n\\psi(x,z) - \\psi(x_r(y(z),z),z) \\ge \\frac{r-L}{2}\\|x-x_r(y(z),z)\\|^2. \n\\end{equation}\nIn addition, we have\n\\begin{align} \n\\label{eq:dual2}\n&\\psi(x,z) - \\psi(x_r(y(z),z),z)\\notag\\\\   \n\\leq \\  &\\psi(x,z)-F_r(x_r(y_+(z),z),y_+(z),z)\\notag \\\\\n\\leq \\ & \\max_{y^\\prime\\in \\mathcal{Y}} F(x,y^\\prime) + \\frac{r}{2}\\|x-z\\|^2-F_r(x_r(y_+(z),z),y_+(z),z)\\notag\\\\\n=\\,& \\max_{y^\\prime\\in \\mathcal{Y}}F(x,y^\\prime)-F(x_r(y_+(z),z),y_+(z)) + \\frac{r}{2}\\|x-z\\|^2-\\frac{r}{2}\\|x_r(y_+(z),z)-z\\|^2,\n\\end{align}\nwhere the first inequality follows from\n\\begin{align*}\nF_r(x_r(y_+(z),z),y_+(z),z) \n=\\ & \\min_{x'\\in\\mathcal{X}}\\left\\{ F(x',y_+(z)) +\\frac{r}{2}\\|x'-z\\|^2\\right\\} \\nonumber \\\\\n\\leq\\ & \\max_{y' \\in \\mathcal{Y}}\\min_{x'\\in\\mathcal{X}}\\left\\{F(x',y') +\\frac{r}{2}\\|x'-z\\|^2\\right\\} \\nonumber \\\\\n=\\ & \\min_{x'\\in\\mathcal{X}}\\left\\{F(x',y(z)) +\\frac{r}{2}\\|x'-z\\|^2\\right\\} \\nonumber \\\\\n=\\ & \\min_{x'\\in\\mathcal{X}} \\psi(x',z)\n= \\psi(x_r(y(z),z),z). % \\label{eq:Fr-one}\n\\end{align*}\n% \\Jiajin{We may have to double-check the above derivation. --- confirmed. With the uniform KL property, we can confirm  $x(y(z),z)$ is unique over $Y(z)$.}\nAs \\eqref{eq:dual1} and \\eqref{eq:dual2} hold for any $x\\in\\mathcal{X}$, we obtain the intermediate relation\n\\begin{equation}\\label{OS-GS-better-key3-main} \n% \\begin{aligned}\n     \\frac{r-L}{2}\\|x_r(y(z),z)-x_r(y_+(z),z)\\|^2\\leq  \\max\\limits_{y^\\prime\\in \\mathcal{Y}} F(x_r(y_+(z),z),y^\\prime) - F(x_r(y_+(z),z),y_+(z)) \n% \\end{aligned}\n\\end{equation}\nby taking $x= x_r(y_+(z),z)$.\n\nNow, we utilize the K{\\L} exponent $\\theta$ of the dual problem to bound the right-hand side of \n\\eqref{OS-GS-better-key3-main}  in terms of $\\|y-y_+(z)\\|$. Consider the following two cases:\n \\begin{enumerate}[label=\\normalfont(\\alph*)]\n \\item Suppose that $\\theta=0$. If $F(x_r(y_+(z),z),y_+(z))=\\max_{y'\\in\\mathcal{Y}} F(x_r(y_+(z),z),y')$, then the desired inequality follows trivially from~\\eqref{OS-GS-better-key3-main}. Otherwise, we have $y_+(z) \\in \\mathcal{Y} \\setminus \\mathcal{Y}^\\star(x_r(y_+(z),z))$. By Assumption~\\ref{ass:kl}, we have\n% we deduce using~\\eqref{eq:Fr-one} that\n% \\begin{align*}\n% F_r(x_r(y_+(z),z),y_+(z),z)\n% &=\\max_{y'\\in\\mathcal{Y}} F_r(x_r(y_+(z),z),y',z)\\\\\n% &\\ge F_r(x_r(y_+(z),z),y_+(z),z)\\\\\n% &\\ge F_r(x_r(y(z),z),y(z),z).\n% \\end{align*}\n%By definition of $y(z)$, it follows that $y_+(z)=y(z)$, and consequently the conclusion trivially holds. \n\\[\n\\dist(0, -\\nabla_y F(x_r(y_+(z),z),y_+(z))+\\partial\\iota_{\\mathcal{Y}}(y_+(z)))\\ge \\mu.\n\\]\nSince  $F(x,\\cdot)$ is continuously differentiable on the compact set $\\mathcal{Y}$, $\\|\\nabla_y F(x,\\cdot)\\|$ is bounded on $\\mathcal{Y}$. Without loss of generality, we can assume that  $\\|\\nabla_y F(x,\\cdot)\\|\\leq L$ on $\\mathcal{Y}$. According to  the mean value theorem, we have \n\\[\n\\max\\limits_{y^\\prime\\in \\mathcal{Y}} F(x_r(y_+(z),z),y^\\prime) - F(x_r(y_+(z),z),y_+(z))\n\\leq L \\cdot {\\rm diam}(\\mathcal{Y}).\n\\]\nUsing the fact that $y_+(z) = \\proj_{\\mathcal{Y}} (y + \\alpha \\nabla_y F(x_r(y,z), y))$, we bound\n\\begin{align*}\n &\\max\\limits_{y^\\prime\\in \\mathcal{Y}} F(x_r(y_+(z),z),y^\\prime) - F(x_r(y_+(z),z),y_+(z))\\notag\\\\\n\\leq\\ & \\frac{L \\cdot {\\rm diam}(\\mathcal{Y})}{\\mu^2}\\cdot\\dist^2(0, -\\nabla_y F(x_r(y_+(z),z),y_+(z))+\\partial\\iota_{\\mathcal{Y}}(y_+(z)))\\notag\\\\ \n% \\stackrel{\\text{\\ding{172}}}{\\leq} \\ &   \\frac{L \\cdot {\\rm diam}(\\mathcal{Y})}{\\mu^2} b^2 \\|\\proj_{\\mathcal{Y}}(y+\\alpha\\nabla_y F(x_r(y_+(z),z),y))-y\\|^2\n% \\notag \\\\\n% \\leq \\ &  \\frac{ L b^2 \\cdot {\\rm diam}(\\mathcal{Y})}{\\mu^2}(\n% \\|\\proj_{\\mathcal{Y}}(y+\\alpha\\nabla_y F(x_r(y_+(z),z),y))-y_+(z)\\|\\ +\\|y_+(z)-y\\|)^2 \\\\ \n% \\leq \\ & \\frac{L b^2 \\cdot {\\rm diam}(\\mathcal{Y})}{\\mu^2}(\\alpha L \\|x_r(y_+(z),z)-x_r(y,z)\\| + \\|y-y_+(z)\\|)^2\\\\\n\\leq \\ & \\frac{L b^2 \\cdot {\\rm diam}(\\mathcal{Y})}{\\mu^2}\\|y_+(z)-y\\|^2,\n\\end{align*}\nwhere $b:=\\frac{1}{\\alpha} + L > 0$ and the last inequality is due to the fact that the projected gradient ascent method satisfies the so-called \\textit{relative error condition}\n\\citep[Section 5]{attouch2013convergence}. \nIt follows that\n\\[\n\\|x_r(y(z),z)-x_r(y_+(z),z)\\| \\leq\\frac{\\sqrt{2L\\cdot {\\rm diam}(\\mathcal{Y})}\\cdot (1+\\alpha L)}{\\alpha\\mu\\sqrt{r-L}}\\|y-y_+(z)\\|. \n\\]\n\\item If $\\theta \\in (0,1)$, then we have\n\\begin{align*}\n% \\label{KL-key-01}\n& \\mu\\left(\\max\\limits_{y^\\prime\\in \\mathcal{Y}} F(x_r(y_+(z),z),y^\\prime) - F(x_r(y_+(z),z),y_+(z))\\right)^{\\theta}\\notag\\\\\n\\leq \\ & \\dist(0, -\\nabla_y F(x_r(y_+(z),z),y_+(z))+\\partial\\iota_{\\mathcal{Y}}(y_+(z))) \\notag\\\\ \n\\leq \\ & \\dist(0, -\\nabla_y F(x_r(y,z),y_+(z)) +\\partial\\iota_{\\mathcal{Y}}(y_+(z)))   \\notag\\\\\n\\ & + \\|\\nabla_y F(x_r(y,z),y_+(z))-\\nabla_y F(x_r(y_+(z),z),y_+(z))\\|\\notag\\\\ \n{\\leq} \\ & \\dist(0, -\\nabla_y F(x_r(y,z),y_+(z))+\\partial\\iota_{\\mathcal{Y}}(y_+(z)))+ L\\sigma_2\\|y-y_+(z)\\|\\notag\\\\\n{\\leq} \\ & \\left(\\frac{1}{\\alpha}+L+L\\sigma_2\\right)\\|y_+(z)-y\\|,\n\\end{align*}\nwhere the third inequality follows from the $L$-Lipschitz continuity  of $\\nabla_y F(\\cdot,\\cdot)$ and  \\eqref{lip-y};  the last inequality follows from the relative error condition of the projected gradient ascent method. This, together with \\eqref{OS-GS-better-key3-main}, yields\n\\[\n\\|x_r(y(z),z)-x_r(y_+(z),z)\\|\\leq \\frac{\\sqrt{2}}{\\sqrt{r-L}}\\left(\\frac{1+\\alpha L(1+\\sigma_2)}{\\alpha\\mu}\\right)^{\\frac{1}{2\\theta}} \n\\|y-y_+(z)\\|^{\\frac{1}{2\\theta}}.\n\\]\n \\end{enumerate}\nThe proof is complete.\n\\end{proof}","sha256_utf8":"5671404bb10c8c6f2789f876a5b7ff434c665172422a86e000fd5314c8d2e90a"},{"start":67130,"end":67876,"start_line":938,"end_line":950,"raw_tex":"\\begin{corollary}[Alternative dual error bound with K\\L{} exponent]\n% Let $\\delta>0$. \nSuppose that  Assumption \\ref{ass:kl} holds. Then, for any $y \\in \\mathcal{Y}$ and $z \\in \\mathbb{R}^n$,  we have\n% \\[\n% \\|x_r^\\star(z)-x_r(y_+(z),z)\\| \\leq \\omega \\|y-y_+(z)\\|^{\\frac{1}{2\\theta}},\n% \\]\n% where $\\omega:=\\frac{\\sqrt{2}}{\\sqrt{r-L}}\\left(\\frac{1+\\alpha L(1+\\sigma_2)}{\\alpha\\mu}\\right)^{\\frac{1}{2\\theta}}$.\n\\begin{enumerate}[label=\\normalfont(\\alph*)]\n \\item \\rm{(K\\L\\ exponent $\\theta=0$):} $\\|x_r^\\star(z)-x_r(y_+(z),z)\\| \\leq \\omega_1 \\|y-y_+(z)\\|$,\n  \\item \\rm{(K\\L\\ exponent $\\theta\\in(0,1)$):} $\\|x_r^\\star(z)-x_r(y_+(z),z)\\| \\leq \\omega_2\\|y-y_+(z)\\|^{\\frac{1}{2\\theta}}$. \n  \\end{enumerate}\n \\label{prop:dual_eb_KL2}   \n\\end{corollary}","sha256_utf8":"b9552f35e65917abfd8aeb78107bad2e1f1e77cca97b73fb222b5f23e07c63b8"},{"start":69575,"end":70139,"start_line":970,"end_line":978,"raw_tex":"\\begin{lemma} \\label{lemma-episolcol}\n        Let $\\epsilon\\ge 0$ be given. Suppose that\n        \\[\n        \\max\\left\\{\\|x^{k+1}-x^{k}\\|,\\|y^{k}_+(z^{k+1}) - y^k\\|,\\|x^{k+1}-z^k\\|\\right\\} \\leq  \\epsilon.\n        \\]\n        Then,  $\\left(x^{k+1}, y^{k+1}\\right)$ is a $\\rho \\epsilon$-GS of problem \\eqref{eq:problem}, where\n\\[\\rho = \\max\\left\\{(\\eta+1+\\sigma_1\\alpha\\beta L)\\left(\\frac{1}{\\alpha}+L\\right), r(\\zeta+\\sigma_2(\\eta+1+\\sigma_1\\alpha \\beta L)+\\sigma_1)\\right\\}\\] \n with $\\sigma_{1}:=\\frac{r}{r-L}$ and $\\sigma_{2}:=\\frac{2r-L}{r-L}$.\n        \\end{lemma}","sha256_utf8":"6accc28c7cde6c83b8b6e58c31fbd1d3bd1cbefc637c885ebb860c3d3e27c361"},{"start":70140,"end":72783,"start_line":979,"end_line":1016,"raw_tex":"\\begin{proof}\nBy Definition \\ref{defi:primal-dual}, we only have to quantify $\\|\\nabla_{z} d_r(y^{k+1},x^{k+1})\\|$ and ${\\rm dist}(0,-\\nabla_{y} F(x^{k+1}, y^{k+1})+\\partial\\iota_{\\mathcal{Y}}(y^{k+1}))$.\nTo begin, observe that\n\\begin{equation*}\n\\begin{aligned}\n& \\| \\nabla_{z} d_r(y^{k+1},x^{k+1})\\| \n\\\\\n=\\ & r \\|x^{k+1}-x_r (y^{k+1},x^{k+1})\\|\\\\  \n\\leq\\ & r(\\|x^{k+1}-x_r(y^{k},z^{k})\\| +\\|x_r(y^{k},z^{k})-x_r(y^{k+1},z^{k})\\|)  +r \\|x_r(y^{k+1},z^{k})-x_r(y^{k+1},x^{k+1})\\|  \\\\\n\\leq\\  &  r\\left(\\zeta\\|x^{k+1}-x^k\\| +\\sigma_2\\|y^{k}-y^{k+1}\\| + \\sigma_1 \\|x^{k+1}-z^{k}\\| \\right) \\\\\n% \\leq & r\\left(\\|x^{k+1}-x_r(y^{k},z^{k})\\| +\\|x_r(y^{k},z^{k})-x_r(y^{k+1},z^{k})\\| + \\|x_r(y^{k+1},z^{k})-x_r(y^{k+1},x^{k+1})\\| \\right) \\\\\n\\leq\\  &  r\\left(\\zeta\\|x^{k+1}-x^k\\| + \\sigma_1 \\|x^{k+1}-z^{k}\\| \\right) +r\\sigma_2 (\\eta\\| x^{k+1}-x^k\\|+\\sigma_1\\alpha L\\|z^{k+1}-z^k\\|+\\| y_+^k(z^{k+1})-y^k\\|)\\\\\n\\leq\\ & r(\\zeta+\\sigma_2(\\eta+1+\\sigma_1\\alpha \\beta L)+\\sigma_1)\\epsilon,\n\\end{aligned}\n\\end{equation*}\nwhere the second inequality is due to Proposition \\ref{prop:lip} and Lemma \\ref{lemma-sollip}, the third is due to Lemma \\ref{lemma-dualxy}, and the fourth follows from the update $z^{k+1} = z^k + \\beta (x^{k+1} - x^k)$ and the assumption of the lemma. Next, recall that\n\\begin{equation*}\ny^{k+1} = \\proj_{\\mathcal{Y}}\\left(y^k + \\alpha \\nabla_y F(x^{k+1}, y^k)\\right)=\\argmin_{y \\in \\mathcal{Y}}\\left\\{\\|y-y^k - \\alpha \\nabla_y F(x^{k+1}, y^k)\\|^2\\right\\}.\n\\end{equation*}\nThe necessary optimality condition yields\n\\begin{align*}\n0\n&\\in y^{k+1}-y^k - \\alpha \\nabla_y F(x^{k+1}, y^k)+\\partial \\iota_{\\mathcal{Y}}(y^{k+1})\\notag\\\\\n&=y^{k+1}-y^k - \\alpha \\nabla_y F(x^{k+1}, y^{k+1})+\\partial \\iota_{\\mathcal{Y}}(y^{k+1}) +\\alpha (\\nabla_y F(x^{k+1}, y^{k+1})- \\nabla_y F(x^{k+1}, y^{k})).\n\\end{align*}\nSince $\\partial \\iota_{\\mathcal{Y}} (y^{k+1})$ is the normal cone to $\\mathcal{Y}$ at $y^{k+1}$, we have\n\\begin{align*}\nv\n&:=-\\frac{1}{\\alpha}(y^{k+1}-y^{k})+\\nabla_y F(x^{k+1}, y^{k})- \\nabla_y F(x^{k+1}, y^{k+1})\\in -\\nabla_{y} F(x^{k+1}, y^{k+1})+\\partial \\iota_{\\mathcal{Y}}(y^{k+1}).\n\\end{align*}\nIt follows that\n\\begin{align*}\n\\|v\\| \\ & \\leq \\left(\\frac{1}{\\alpha}+L\\right)\\|y^{k+1}-y^{k}\\|\\\\\n\\ &  \\leq  \\left(\\frac{1}{\\alpha}+L\\right)\\left(\\eta\\|x^{k}-x^{k+1}\\|+\\sigma_1\\alpha L\\|z^{k+1}-z^k\\|+\\|y_+^k(z^{k+1})-y^k\\|\\right) \\\\\n\\ & \\leq (\\eta+1+\\sigma_1\\alpha\\beta L)\\left(\\frac{1}{\\alpha}+L\\right)\\epsilon,\n\\end{align*}\nwhere the first inequality is due to the $L$-Lipschitz continuity of $\\nabla_y F(\\cdot,\\cdot)$ and the second is due to Lemma \\ref{lemma-dualxy}. Putting everything together yields the desired result.\n\\end{proof}","sha256_utf8":"611ece6f700405f9a53cee16df384bf2f522e45cfb68650f9d8d74a8d8947d77"},{"start":77680,"end":87655,"start_line":1059,"end_line":1182,"raw_tex":"\\begin{proof}\n\nAs mentioned in Section \\ref{sec:framework}, the global convergence rate of smoothed PLDA depends on the specific regime of the growth power $v$ in \\eqref{eq:step_dual}. Let us begin by considering cases (a) and (b), i.e., either the dual function is concave or the dual problem possesses the K\\L{} property with exponent $\\theta\\in(\\frac{1}{2},1)$.\n\n\nIn view of the basic descent estimate in Proposition \\ref{prop:decrease}, we consider the following two scenarios separately:\\\\\n(1) There exists a $ k \\in\\{0,1, \\ldots, K-1\\}$ such that\n\\begin{align*}\n\\ &\\frac{1}{2}\\max \\left\\{\\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}, \\frac{1}{8\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 , \\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}\\right\\}\\\\\n\\leq \\ & 14r\\beta\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2}.\n\\end{align*}\n(2) For any $ k \\in\\{0,1, \\ldots, K-1\\}$, we have\n\\begin{align*}\n\\ &\\frac{1}{2}\\max \\left\\{\\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}, \\frac{1}{8\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 , \\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}\\right\\}\\\\\n\\ge\\ & 14r\\beta\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2}.\n\\end{align*}\n\n\\textbf{Scenario (1)}. For the case where the K{\\L} exponent $\\theta \\in (\\tfrac{1}{2},1)$, we deduce from Proposition \\ref{prop:dual_eb_KL}(b) that \n\\begin{align*}\n    \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2  & \\leq 224r\\alpha\\beta\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2}  \\\\ &\\leq  224r\\alpha\\beta \\omega_2^2 \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{\\theta}},  \n\\end{align*}\nwhich gives $\\|y^{k}-y_{+}^{k}(z^{k+1})\\|\\leq \\rho_1\\beta^{\\frac{\\theta}{2\\theta-1}}$ with $\\rho_1 := (224r\\alpha\\omega_2^2)^{\\frac{\\theta}{2\\theta-1}}$. Additionally, using the update $z^{k+1} = z^k + \\beta( x^{k+1} - z^k )$ and Proposition \\ref{prop:dual_eb_KL}(b), we have\n\\begin{align*}\n\\|x^{k+1}-z^{k}\\|^{2}  \\ & =\\frac{1}{\\beta^2}\\|z^{k+1}-z^{k}\\|^{2}\\\\\n\\ &  \\leq 49\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2} \\\\\n\\ & \\leq  49 \\omega_2^2 \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{\\theta}}\\leq \\rho_2^2\\beta^{\\frac{1}{2\\theta-1}},\n\\end{align*}\nwhere $\\rho_2 := \\sqrt{49\\omega_2^2\\rho_1^{\\frac{1}{\\theta}}}$. \nLastly, we have\n\\begin{align*}\n    \\|x^{k+1}-x^k\\|^2 \\ & \\leq {\\frac{224}{3}r\\beta\\lambda}\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2} \\\\\n   \\  &\\leq {\\frac{224}{3}r\\omega^2_2\\beta\\lambda}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{\\theta}} \\leq \\rho_3^2 \\beta^{\\frac{2\\theta}{2\\theta-1}},  \n\\end{align*}\nwhere $\\rho_3 := \\sqrt{{\\frac{224}{3}r\\omega_2^2\\lambda}\\rho_1^{\\frac{1}{\\theta}}}$. \nCombining the above inequalities, we have\n\\begin{align*}\n& \\max \\left\\{\\|x^{k}-x^{k+1}\\|,\\|y^{k}-y_{+}^{k}(z^{k+1})\\|,\\|z^{k}-x^{k+1}\\|\\right\\} \\leq  \\max\\left\\{ \\rho_1\\beta^{\\frac{\\theta}{2\\theta-1}}, \\rho_2\\beta^{\\frac{1}{4\\theta-2}}, \\rho_3 \\beta^{\\frac{\\theta}{2\\theta-1}}\\right\\}. \n\\end{align*}\nIt then follows from Lemma \\ref{lemma-episolcol}\n that $\\left(x^{k+1}, y^{k+1}\\right)$ is a $\\rho\\cdot\\max\\{ \\rho_1\\beta^{{\\theta}/{(2\\theta-1)}},$ $ \\rho_2\\beta^{1/{(4\\theta-2)}}, \\rho_3\\beta^{{\\theta}/{(2\\theta-1)}}\\}$-GS of problem \\eqref{eq:problem}. \n \n For the concave case, we can replace the primal-dual quantity $x_r( y(z^{k+1}), z^{k+1} )$ by the primal quantity $x_r^\\star( z^{k+1} )$ on the right-hand side of the inequality for Scenario (1) and apply Lemma \\ref{lemma-dual-bd} in Appendix \\ref{sec:dual} (so that $\\theta$ and $\\omega_2$ are replaced by $1$ and $\\kappa = \\tfrac{1 + \\alpha L \\sigma_2 + \\alpha L}{\\alpha (r-L)} \\cdot \\textrm{diam}(\\mathcal{Y})$, respectively) to conclude that $\\left(x^{k+1}, y^{k+1}\\right)$ is a $\\max\\{ \\rho_1\\beta, \\rho_2\\beta^{1/2}, \\rho_3 \\beta\\}$-GS of problem \\eqref{eq:problem}.  \n \n\n\nFurthermore, observe that for the case where $\\theta \\in (\\tfrac{1}{2}, 1)$, the optimization-stationarity measure of $z^{k+1}$ satisfies\n\\begin{equation}\n\\label{eq:os-gs}\n\\begin{aligned}\n    & \\|z^{k+1}-x_r^\\star(z^{k+1})\\| \\\\\n    \\leq\\ &  \\|z^{k+1}-x^{k+1}\\|+\\| x^{k+1} - x_r(y^k,z^k) \\| + \\| x_r(y^k,z^k) - x_r(y^k,z^{k+1}) \\|\\\\\n    \\ & + \\|x_r(y^k,z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|+ \\|x_r(y_{+}^{k}(z^{k+1}), z^{k+1})-x_r^\\star(z^{k+1})\\|\\\\\n     \\leq\\ &  (1+\\sigma_1)\\|z^{k}-x^{k+1}\\| + \\zeta\\|x^{k}-x^{k+1}\\| + \\sigma_2 \\|y^k-y_{+}^{k}(z^{k+1})\\|+ \\omega_2 \\|y^k-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{2\\theta}} \\\\\n    \\leq\\ & \\max\\left\\{\\left(\\rho_2(1+\\sigma_1)+ \\omega_2 \\rho_1^\\frac{1}{2\\theta}\\right)\\beta^{\\frac{1}{4\\theta-2}},(\\zeta\\rho_3+\\sigma_2\\rho_1) \\beta^{\\frac{\\theta}{2\\theta-1}} \\right\\}, \n\\end{aligned}\n\\end{equation}\nwhere the second inequality follows from the update $z^{k+1} = z^k + \\beta (x^{k+1} - z^k)$, Proposition \\ref{prop:lip}, Corollary \\ref{prop:dual_eb_KL2}, Lemma \\ref{lemma-sollip}, and the fact that $0 < \\beta \\le \\tfrac{1}{28}$.\nIt is evident that the dependence on $\\beta$ is identical to that of the game-stationarity measure of $(x^{k+1}, y^{k+1})$.\nFor the concave case, the optimization-stationarity measure of $z^{k+1}$ can be bounded similarly.\n\n\\textbf{Scenario (2)}. Proposition \\ref{prop:decrease} implies that for any $k \\in\\{0,1, \\ldots, K-1\\}$, we have \n\\begin{align}\n\\label{eq:suff-decrease}\n\\Phi_r^k-\\Phi_r^{k+1} \n\\ge \\frac{3}{16\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{16\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 +\\frac{2r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}.\n\\end{align}\nThe assumptions in case (a)  imply that \n\\[ \\min_{x\\in\\mathcal{X}}\\max_{y\\in\\mathcal{Y}} F_r(x,y,z) = \\max_{y\\in\\mathcal{Y}} \\min_{x\\in\\mathcal{X}} F_r(x,y,z) \\]\nfor all $z \\in \\mathbb{R}^n$ (cf.~the proof of~\\citep[Lemma B.7]{zhang2020single}) and lead to the lower boundedness of $p_r$, while the assumptions in case (b) directly imply that $p_r$ is lower bounded.\nIn both cases, there exists a $\\underline{\\Phi} > -\\infty$ such that for all $x \\in \\mathcal{X}$, $y \\in \\mathcal{Y}$, and $z \\in \\mathbb{R}^n$,\n\\begin{align*}\n& \\Phi_r(x, y, z)  = p_r(z)+(F_r(x,y,z)-d_r(y, z))+(p_r(z)-d_r(y, z))\\geq p_r(z)\\geq \\underline{\\Phi} >-\\infty.\n\\end{align*}\nIt follows that\n    \\begin{align*}\n    &\\Phi_r^0-\\underline{\\Phi} \\geq \\sum_{k=0}^{K-1}\\Phi_r(x^k,y^k,z^k)-\\Phi_r(x^{k+1},y^{k+1},z^{k+1}) \\\\\n    % \\geq\\ & \\sum_{k=0}^{K-1} \\frac{1}{32\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{16\\alpha}\\|y^{k}-y_{+}^{k}(z^{k})\\|^2 +\\frac{2r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2} \\\\\n    \\geq \\ & \\sum_{k=0}^{K-1} \\min\\left\\{\\frac{3}{16\\lambda},\\frac{1}{16\\alpha},\\frac{2\\beta r}{7}\\right\\} \\left(\\|x^{k}-x^{k+1}\\|^{2}+\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 + \\|z^{k}-x^{k+1}\\|^{2} \\right),\n    \\end{align*}\n    where the last inequality is due to \\eqref{eq:suff-decrease} and the update $z^{k+1}=z^{k}+\\beta(x^{k+1}-z^k)$.\n    In particular, there exists a $k \\in\\{0,1, \\ldots, K-1\\}$ such that\n    \\[\n    \\max \\left\\{\\|x^{k}-x^{k+1}\\|^{2},\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2}, \\|x^{k+1}-z^{k}\\|^{2}\\right\\} \\leq \\frac{\\Phi_r^0-\\underline{\\Phi}}{\\min\\left\\{\\frac{3}{16\\lambda},\\frac{1}{16\\alpha},\\frac{2\\beta  r}{7}\\right\\}  K}.\n    \\]\n% \\Jiajin{continue here!}\nBased on Lemma \\ref{lemma-episolcol} and the fact that $0 < \\beta \\le \\tfrac{1}{28}$,  we conclude that $\\left(x^{k+1}, y^{k+1}\\right)$ is an $\\mathcal{O}(\\sqrt{1/{K\\beta}})$-GS of problem \\eqref{eq:problem}. Moreover, by using the same argument as in \\eqref{eq:os-gs}, we can show that $z^{k+1}$ is an $\\mathcal{O}(\\sqrt{1/{K\\beta}})$-OS of problem \\eqref{eq:problem}. Upon setting $\\beta = C K^{-\\frac{1}{2}}$ for the concave case and  $\\beta = C K^{-\\frac{2\\theta-1}{2\\theta}}$ for the case where the K{\\L} exponent $\\theta \\in (\\tfrac{1}{2}, 1)$, with $C$ being a suitably chosen constant, we see that the \n% where $C$ is a constant.\n% Similarly, for the general concave case, we can set $\\beta = C K^{-\\frac{1}{2}}$. \nbounds obtained for the optimization-stationarity and game-stationarity measures are of the same order for both Scenarios (1) and (2). This establishes cases (a) and (b).\n\nNext, let us consider case (c). First, suppose that the K{\\L} exponent $\\theta \\in (0, \\tfrac{1}{2}]$. \nOnce again, we can use Proposition \\ref{prop:decrease} to obtain \n\\begin{align*}\n\\Phi_r^k-\\Phi_r^{k+1} \n\\ge\\ &  \\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{8\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 +\\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}\\\\\n\\ & -14r\\beta\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2}.\n\\end{align*}\nUsing Proposition \\ref{prop:dual_eb_KL}(b) and the boundedness of $\\mathcal{Y}$, we have\n% there exists  for each $\\theta \\in(0,\\frac{1}{2})$ that \n\\begin{equation}\\label{dual-eb-bd}\n\\begin{aligned}\n  \\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|\n&\\leq \\omega_2 \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{2\\theta}}\\\\\n&=  \\omega_2 \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{2\\theta}-1} \\|y^{k}-y_{+}^{k}(z^{k+1})\\|\\\\\n&\\leq   \\omega_2\\cdot {\\rm diam}(\\mathcal{Y})^{\\frac{1}{2\\theta}-1}\\cdot \\|y^{k}-y_{+}^{k}(z^{k+1})\\|.\n\\end{aligned}\n\\end{equation}\nIt follows that\n\\begin{align*}\n\\Phi_r^k-\\Phi_r^{k+1} \n\\ge \\ &  \\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\left(\\frac{1}{8\\alpha}-14r\\beta\\omega_2^2\\cdot{\\rm diam}(\\mathcal{Y})^{\\frac{1-2\\theta}{\\theta}}\\right) \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2  +\\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}.\n\\end{align*}\nSince \n$\\beta  \\leq \\frac{ {\\rm diam}(\\mathcal{Y})^{\\frac{2\\theta-1}{\\theta}}}{224 \\alpha r\\omega_2^2}$, we have \n\\begin{align*}\n&\\Phi_r^k-\\Phi_r^{k+1} \\ge  \\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{16\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 +\\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}.\n\\end{align*}\nThe desired result can then be obtained by following a similar argument as that in the paragraph proceeding \\eqref{eq:suff-decrease}.\n\nWhen the K{\\L} exponent $\\theta=0$, we can apply Proposition \\ref{prop:dual_eb_KL}(a) to derive the inequality \\eqref{dual-eb-bd} \n% by letting $\\theta=\\frac{1}{2}$ and replacing $\\omega_2$ with $\\omega_1$. \nwith $\\theta$ and $\\omega_2$ being replaced by $\\tfrac{1}{2}$ and $\\omega_1$, respectively.\nThe desired result then follows by adapting the argument in the preceding paragraph.\n\\end{proof}","sha256_utf8":"5e26e8ec1486aa58d52f3d5f049005bfb0905f56ca12e76ea1b096b21890486a"},{"start":135327,"end":135466,"start_line":1745,"end_line":1747,"raw_tex":"\\begin{fact}\\label{fact-1}\nThe functions $F(\\cdot, y)$ for any $y\\in\\mathcal{Y}$ and $f$ are $L$-weakly convex on $\\mathcal{X}$.\n\\end{fact}","sha256_utf8":"407b2e0012dbf88272415edada15a73c8fa7d4d0695e21e1e942e4b7fec9c48e"},{"start":135467,"end":135705,"start_line":1748,"end_line":1753,"raw_tex":"\\begin{fact}\\label{fact-2}\nLet $y\\in\\R^d$ and $r>0$ be given. For all $x, \\bar{x} \\in\\mathcal{X}$, we have\n\\[\n-\\frac{r^{-1}+L}{2}\\|x-\\bar{x}\\|^{2} \\leq F(x,y)-F_{\\bar{x},r}(x, y) \\leq \\frac{-r^{-1} + L}{2}\\|x-\\bar{x}\\|^{2} .\n\\]\n\\end{fact}","sha256_utf8":"2b89c576f272a98ea11cbc84f525b1053b0f4fe7f41e69774a876c6ae4ef9827"},{"start":136469,"end":136991,"start_line":1763,"end_line":1771,"raw_tex":"\\begin{lemma}\\label{lemma-sollip}\nFor any $y, y^{\\prime} \\in \\mathcal{Y}$ and $z, z^{\\prime} \\in \\R^n$, we have \n    \\begin{align}\n        &\\|x_r(y, z)-x_r(y, z^{\\prime})\\| \\leq \\sigma_{1}\\|z-z^{\\prime}\\|, \\label{lip-z}\\\\\n        &\\|x_r^\\star(z)-x_r^\\star(z^{\\prime})\\| \\leq \\sigma_{1}\\|z-z^{\\prime}\\|, \\label{lip-starz}\\\\\n        &\\|x_r(y, z)-x_r(y^{\\prime}, z)\\| \\leq \\sigma_{2}\\|y-y^{\\prime}\\|, \\label{lip-y}\n        \\end{align}\n     where $\\sigma_{1}:=\\frac{r}{r-L}$ and $\\sigma_{2}:=\\frac{2r-L}{r-L}$.\n    \\end{lemma}","sha256_utf8":"63d9a21b3c8aa3c0176617e5351915d9e845af79e5e07cb6f5f8e21a9a4dd8dd"},{"start":136993,"end":141766,"start_line":1773,"end_line":1854,"raw_tex":"\\begin{proof}\nFrom the definition of $F_r$, we know that for any $x\\in\\mathcal{X}$, $y\\in\\mathcal{Y}$, and $z,z'\\in\\R^n$, \n\\begin{equation}\\label{func-cal}\nF_r(x,y,z^{\\prime})-F_r(x,y,z)=\\frac{r}{2}(\\|x-z'\\|^2-\\|x-z\\|^2)=\\frac{r}{2}(\\|z^{\\prime}\\|^{2}-2(z^{\\prime}-z)^\\top x-\\|z\\|^{2}).\n\\end{equation}\nFact \\ref{fact-1} implies that $F(\\cdot, y)$ is $L$-weakly convex for any $y \\in \\mathcal{Y}$. Therefore, for any $x\\in\\mathcal{X}$, $y\\in\\mathcal{Y}$, and $z\\in\\R^n$, we have\n\\begin{equation}\\label{psi-strongcvx}\nF_r(x,y, z)- F_r(x_r(y,z),y,z)\n\\geq \\frac{r-L}{2}\\|x-x_r(y,z)\\|^{2}.\n\\end{equation}\nCombining \\eqref{func-cal} and \\eqref{psi-strongcvx} yields\n\\begin{align}\\label{lip-key1}\n    &F_r(x_r(y,z),y,z^{\\prime})-F_r(x_r(y,z^{\\prime}),y,z^{\\prime}) \\notag\\\\\n    =\\ &F_r(x_r(y,z),y,z^{\\prime})-F_r(x_r(y,z),y,z)+F_r(x_r(y,z),y,z)-F_r(x_r(y,z^{\\prime}),y,z)\\, \\notag\\\\\n    &-(F_r(x_r(y,z^{\\prime}),y, z^{\\prime})-F_r(x_r(y,z^{\\prime}),y, z)) \\notag\\\\\n    \\leq\\ &\\frac{r}{2}(\\|z^{\\prime}\\|^{2}-2(z^{\\prime}-z)^\\top x_r(y,z)-\\|z\\|^{2})-\\frac{r-L}{2}\\|x_r(y,z)-x_r(y,z^{\\prime})\\|^{2}\\,  \\notag\\\\\n    &-\\frac{r}{2}(\\|z^{\\prime}\\|^{2}-2(z^{\\prime}-z)^\\top x_r(y,z^{\\prime})-\\|z\\|^{2}) \\notag\\\\\n    \\leq\\ &r(z^{\\prime}-z)^\\top(x_r(y,z^{\\prime})-x_r(y,z))-\\frac{r-L}{2}\\|x_r(y,z)-x_r(y,z^{\\prime})\\|^{2}.\n    \\end{align}\n On the other hand, \\eqref{psi-strongcvx} implies that\n    \\[\n    F_r(x_r(y,z),y, z^{\\prime})-F_r(x_r(y,z^{\\prime}),y,z^{\\prime}) \\geq \\frac{r-L}{2}\\|x_r(y,z)-x_r(y,z^{\\prime})\\|^{2}.\n    \\]\nCombining this inequality with \\eqref{lip-key1}, we get\n    \\[\n    (r-L)\\|x_r(y,z)-x_r(y,z^{\\prime})\\|^{2}\\leq r(z^{\\prime}-z)^\\top (x_r(y,z^{\\prime})-x_r(y,z)).\n    \\]\nUsing the Cauchy-Schwarz inequality, we obtain\n    \\[\n    \\|x_r(y,z)-x_r(y,z^{\\prime})\\| \\leq \\frac{r}{r-L}\\|z-z^{\\prime}\\|.\n    \\]\nTherefore, we conclude that \\eqref{lip-z} holds with $\\sigma_1=\\frac{r}{r-L}$. Since $f=\\max_{y \\in \\mathcal{Y}} F(\\cdot,y)$ is also $L$-weakly convex by  Fact \\ref{fact-1}, we can prove that \\eqref{lip-starz} holds by using a similar argument as above.\n    \n\nWe  now proceed to prove \\eqref{lip-y}. Using \\eqref{psi-strongcvx}, we have\n\\begin{align}\n&F_r(x_r(y^{\\prime}, z),y, z)-F_r(x_r(y, z), y,z) \\geq \\frac{r-L}{2}\\|x_r(y, z)-x_r(y^{\\prime}, z)\\|^{2},\\label{psi-strongcvxineq1} \\\\\n&F_r(x_r(y, z), y^{\\prime},z)-F_r(x_r(y^{\\prime}, z), y^{\\prime},z) \\geq \\frac{r-L}{2}\\|x_r(y, z)-x_r(y^{\\prime}, z)\\|^{2}\\label{psi-strongcvxineq2}.\n\\end{align}\nMoreover, by the $L$-Lipschitz continuity of $\\nabla_{y} F_r(x, \\cdot, z)$ for any $x \\in \\mathcal{X}$ and $z \\in \\mathbb{R}^n$, we have\n\\begin{equation}\n\\begin{aligned}\\label{psi-yconcave}\n F_r(x_r(y, z), y^{\\prime},z)-F_r(x_r(y, z), y,z)\\leq &\\langle\\nabla_{y} F_r(x_r(y, z),y,z), y^{\\prime}-y\\rangle +\\frac{L}{2}\\|y-y^{\\prime}\\|^{2}\n\\end{aligned}\n\\end{equation}\nand \n\\begin{equation}\n\\begin{aligned}\\label{psi-ylip}\n F_r(x_r(y^{\\prime}, z), y, z)-F_r(x_r(y^{\\prime}, z), y^{\\prime},z) \\leq & \\langle\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime}, z), y-y^{\\prime}\\rangle + \\frac{L}{2}\\|y-y^{\\prime}\\|^{2}.\n\\end{aligned}\n\\end{equation}\nIncorporating \\eqref{psi-strongcvxineq1}--\\eqref{psi-ylip}, we obtain\n\\begin{equation*}\n% \\label{lip-key2}\n\\begin{aligned}\n& (r-L)\\|x_r(y, z)-x_r(y^{\\prime}, z)\\|^{2}  \\\\ \\leq \\ &\\langle\\nabla_{y} F_r(x_r(y, z), y, z)-\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime}, z), y^{\\prime}-y\\rangle +L\\|y-y^{\\prime}\\|^{2}. \n\\end{aligned}\n\\end{equation*}\nSince $\\nabla_y F_r(\\cdot,\\cdot,z) = \\nabla_y F(\\cdot,\\cdot)$ is $L$-Lipschitz for any $z \\in \\mathbb{R}^n$, we have\n\\begin{align*}\n&\\|\\nabla_{y} F_r(x_r(y, z), y, z)-\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime}, z)\\| \\notag\\\\\n\\leq\\ &\\|\\nabla_{y} F_r(x_r(y, z), y, z)-\\nabla_{y}F_r(x_r(y, z), y^{\\prime}, z)\\|  +\\|\\nabla_{y}F_r(x_r(y, z), y^{\\prime}, z)-\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime}, z)\\|\\notag\\\\\n\\leq\\ &L(\\|y-y^{\\prime}\\|+\\|x_r(y^{\\prime}, z)-x_r(y, z)\\|).\\notag\n\\end{align*}\nIt follows that\n\\begin{align*}\n(r-L)\\|x_r(y, z)-x_r(y^{\\prime}, z)\\|^{2} \n\\leq L\\|x_r(y^{\\prime}, z)-x_r(y, z)\\|\\cdot\\|y-y^{\\prime}\\|+2L\\|y-y^{\\prime}\\|^{2}.\n\\end{align*}\nLet $\\zeta:=\\frac{\\|x_r(y, z)-x_r(y^{\\prime}, z)\\|}{\\|y-y^{\\prime}\\|}$. \nThen, the above inequality gives\n\\begin{align*}\n\\zeta^{2}\n& \\ \\leq \\frac{L}{r-L} \\zeta+\\frac{2L}{(r-L)}\\leq \\frac{1}{2}\\zeta^2+\\frac{L^2}{2(r-L)^2} +\\frac{2L}{(r-L)}\\\\\n& \\ \\leq \\frac{1}{2} \\zeta^{2}+\\frac{L^2+4L(r-L)}{2(r-L)^2}  \\leq \\frac{1}{2} \\zeta^{2}+\\frac{(L+2(r-L))^2}{2(r-L)^2},\n\\end{align*}\nwhere the second inequality holds because $ab\\leq \\frac{1}{2}(a^2+b^2)$ for any $a,b\\in\\R$. Thus, we get\n\\[\n\\|x_r(y, z)-x_r(y^{\\prime}, z)\\| \\leq  \\frac{2r-L}{r-L}\\|y-y^{\\prime}\\|,\n\\]\nwhich shows that \\eqref{lip-y} holds with $\\sigma_{2}=\\frac{2r-L}{r-L}$. The proof is complete.\n\\end{proof}","sha256_utf8":"975f9b5be0505a2866a2c738bb6c480dd48a8707f0beb360f2806bbea16ede80"},{"start":141767,"end":142760,"start_line":1855,"end_line":1867,"raw_tex":"\\begin{lemma}\\label{lemma-dualdiff} \n    The dual potential value function $d_r$ is differentiable on $\\mathcal{Y}\\times \\R^n$ with\n    \\begin{align*}\n    &\\nabla_{y} d_r(y,z)=\\nabla_{y} F(x_r(y,z),y),\\\\\n    &\\nabla_{z} d_r(y,z)=\\nabla_{z} F_r(x_r(y, z),y, z)=r(z-x_r(y,z))\n    \\end{align*}\n    for any $y \\in \\mathcal{Y}$ and $z \\in \\mathbb{R}^n$. Moreover, for any $y \\in \\mathcal{Y}$ and $z \\in \\mathbb{R}^n$,  the gradients $\\nabla_y d_r(\\cdot,z)$ and $\\nabla_z d_r(y,\\cdot)$ are Lipschitz continuous, i.e.,\n    \\begin{align}\n    &\\|\\nabla_{y} d_r(y^{\\prime},z)-\\nabla_{y} d_r(y^{\\prime \\prime},z)\\| \\leq L_{d_r}\\|y^{\\prime}-y^{\\prime \\prime}\\| \\quad \\text{for all}\\ y^{\\prime}, y^{\\prime \\prime} \\in \\mathcal{Y},\\notag\\\\\n    &\\|\\nabla_{z} d_r(y,z')-\\nabla_{z} d_r(y,z'')\\| \\leq L_{d_r}'\\|z^{\\prime}-z^{\\prime \\prime}\\| \\quad \\text{for all}\\ z^{\\prime}, z^{\\prime \\prime} \\in \\R^n,\\notag\n    \\end{align}\n    where $L_{d_r}:=(\\sigma_{2}+1)L$ and $L_{d_r}':=(\\sigma_1 + 1)r$.\n    \\end{lemma}","sha256_utf8":"8c9815666665db65e874f439da9e93422a15bb71279a19db729b47504015ee6f"},{"start":142761,"end":144854,"start_line":1868,"end_line":1896,"raw_tex":"\\begin{proof}\nSince $F_r(\\cdot,y,z)$ is strongly convex, $F_r(x,\\cdot,z)$ is weakly concave, and $F_r(x,y,\\cdot)$ is strongly convex for any $x \\in \\mathcal{X}$, $y \\in \\mathcal{Y}$, and $z \\in \\mathbb{R}^n$, the function\n    \\[\n    d_r(\\cdot,\\cdot)=\\min_{x\\in\\mathcal{X}} F_r(x,\\cdot,\\cdot)\n    \\] \n    is differentiable on $\\mathcal{Y}\\times \\R^n$ due to \\citep[Theorem 10.31]{rockafellar2009variational}. In particular, for any $y\\in\\mathcal{Y}$ and $z\\in\\R^n$, one has\n    \\[\n    \\nabla_{y} d_r(y, z)=\\nabla_{y} F_r(x_r(y, z),y, z)=\\nabla_{y} F(x_r(y, z), y).\n    \\]\nUsing  \\citep[Lemma 4.3]{drusvyatskiy2019efficiency}, we have \n    \\[\n    \\nabla_{z} d_r(y,z)=\\nabla_{z} F_r(x_r(y, z),y, z)=r(z-\\prox_{\\frac{1}{r}F(\\cdot,y)+\\iota_{\\mathcal{X}}}(z))=r(z-x_r(y,z)).\n    \\]\n    It follows that for  any $y^{\\prime}, y^{\\prime \\prime} \\in \\mathcal{Y}$, \n    \\begin{align*}\n    &\\|\\nabla_{y} d_r(y^{\\prime}, z)-\\nabla_{y} d_r(y^{\\prime \\prime}, z)\\|\\notag\\\\\n    =\\ &\\|\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime},z)-\\nabla_{y} F_r(x_r(y^{\\prime \\prime}, z), y^{\\prime \\prime},z)\\| \\\\\n    \\leq\\ &\\|\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime},z)-\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime \\prime}, z)\\|+\\|\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime \\prime},z)-\\nabla_{y} F_r(x_r(y^{\\prime \\prime}, z), y^{\\prime \\prime}, z)\\| \\\\\n    \\leq\\ & L\\|y^{\\prime}-y^{\\prime \\prime}\\|+L\\|x_r(y^{\\prime}, z)-x_r(y^{\\prime \\prime}, z)\\|\\notag\\\\\n    \\leq\\ & L\\|y^{\\prime}-y^{\\prime \\prime}\\|+L \\sigma_{2}\\|y^{\\prime}-y^{\\prime \\prime}\\|\\leq L_{d_r}\\|y^{\\prime}-y^{\\prime \\prime}\\|,\n    \\end{align*}\n    where the third inequality is due to \\eqref{lip-y}. Moreover, we have\n    \\begin{align*}\n    \\|\\nabla_{z} d_r(y,z')-\\nabla_{z} d_r(y,z'')\\|=\\ &r\\|z'-x_r(y,z')-(z''-x_r(y,z''))\\| \\\\\n    \\leq\\ & r\\|z^{\\prime}-z^{\\prime \\prime}\\|+r\\|x_r(y, z')-x_r(y, z'')\\| \\\\\n    \\leq\\ & r\\|z^{\\prime}-z^{\\prime \\prime}\\|+ r\\sigma_{1}\\|z^{\\prime}-z^{\\prime \\prime}\\|\\leq L_{d_r}'\\|z^{\\prime}-z^{\\prime \\prime}\\|,\n    \\end{align*}\nwhere the second inequality is due to \\eqref{lip-z}. The proof is complete.\n\\end{proof}","sha256_utf8":"126d677cd571469d2a1a6edb7281def76af3e85e92e50d8ead8ad3c810e5bebc"},{"start":144856,"end":145102,"start_line":1898,"end_line":1904,"raw_tex":"\\begin{lemma}\\label{lemma-dualxy}\nFor any $k\\ge0$, we have\n    \\begin{equation*}\n        \\|y^{k+1}-y_{+}^{k}(z^{k+1})\\| \\leq \\eta \\|x^{k}-x^{k+1}\\|+\\sigma_1\\alpha L\\|z^{k+1}-z^k\\|,\n        \\end{equation*}\nwhere $\\eta:=\\alpha L \\zeta$.\n\\end{lemma}","sha256_utf8":"f034b5faa4b85e8150fd4ff2f646e940331dae5e1f46ab6d84b8f1472d84eb3e"},{"start":145234,"end":146090,"start_line":1909,"end_line":1921,"raw_tex":"\\begin{proof}\nWe compute\n\\begin{equation*}\n% \\label{iter-result2}\n\\begin{aligned}\n\\ &\\|y^{k+1}-y_{+}^{k}(z^{k+1})\\|\\\\ =\\ &\\|\\proj_{\\mathcal{Y}}(y^{k}+\\alpha \\nabla_{y} F(x^{k+1},y^{k}))-\\proj_{\\mathcal{Y}}(y^{k}+\\alpha \\nabla_{y} F(x_r(y^{k}, z^{k+1}),y^{k}))\\|\\notag \\\\\n\\leq\\ &\\|y^{k}+\\alpha \\nabla_{y} F(x^{k+1}, y^{k})-(y^{k}+\\alpha \\nabla_{y} F(x_r(y^{k}, z^{k+1}),  y^{k}))\\|\\notag \\\\\n\\leq\\ & \\alpha L(\\|x^{k+1}-x_r(y^{k}, z^{k})\\|+\\|x_r(y^{k}, z^{k})-x_r(y^{k}, z^{k+1})\\|) \\notag \\\\\n\\leq\\ & \\eta\\|x^{k}-x^{k+1}\\|+\\sigma_1\\alpha L\\|z^{k+1}-z^k\\|,\n\\end{aligned}\n\\end{equation*}\nwhere the first inequality is due to the nonexpansiveness of the projection operator, the second is due to the $L$-Lipschitz continuity of $\\nabla_y F(\\cdot,\\cdot)$, and the third follows from Proposition \\ref{prop:lip} and \\eqref{lip-z}. The proof is complete.\n\\end{proof}","sha256_utf8":"3f8f9a6f994b1bb27efb63c5e2df29f3d22107674fe4603ce472235c53f9ee22"},{"start":146177,"end":146553,"start_line":1924,"end_line":1932,"raw_tex":"\\begin{lemma}[Primal descent]\n\\label{lemma-F-des} \nFor any $k\\ge0$, we have\n\\begin{align*}\n& F_r(x^{k}, y^{k},z^k)-F_r(x^{k+1}, y^{k+1},z^{k+1})\\\\\n\\ge\\ &\n\\frac{2\\lambda^{-1}+r-L}{2}\\|x^k-x^{k+1}\\|^2 + \\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k}-y^{k+1}\\rangle-\\frac{L}{2}\\|y^k-y^{k+1}\\|^2 \\\\  &  +\\frac{(2-\\beta)r}{2 \\beta}\\|z^{k}-z^{k+1}\\|^{2}. \n\\end{align*}\n\\end{lemma}","sha256_utf8":"bb2b0ea13a14297b2ef77720110f89681fa273a2fbb408d9f6141d99e1ed706a"},{"start":146555,"end":148122,"start_line":1934,"end_line":1960,"raw_tex":"\\begin{proof}\nOne can infer from the definition that $F_{x^k,\\lambda}(\\cdot,y^k)$ is $\\lambda^{-1}$-strongly convex. Therefore, we have\n\\begin{equation*}\n\\begin{aligned}\nF_r(x^{k}, y^{k},z^{k}) & =F(x^{k}, y^{k})+ \\frac{r}{2}\\|x^k-z^{k}\\|^2  = F_{x^k,\\lambda}(x^k,y^k) + \\frac{r}{2}\\|x^k-z^{k}\\|^2 \\\\\n& \\ge F_{x^k,\\lambda}(x^{k+1},y^k)+\\frac{r}{2}\\|x^{k+1}-z^{k}\\|^2+\\frac{\\lambda^{-1}+r}{2}\\|x^k-x^{k+1}\\|^2.\n\\end{aligned}\n\\end{equation*}\nMoreover, Fact \\ref{fact-2} implies that\n\\begin{equation*} \nF_{x^k,\\lambda}(x^{k+1},y^k)\\ge F(x^{k+1},y^k) + \\frac{\\lambda^{-1}-L}{2}\\|x^{k+1}-x^k\\|^2.\n\\end{equation*}\nIt follows that\n\\begin{align}\\label{primaldec-key1}\nF_r(x^{k},y^{k}, z^k) &\\ge F(x^{k+1},y^k)+\\frac{r}{2}\\|x^{k+1}-z^k\\|^2+\\frac{2\\lambda^{-1}+r-L}{2}\\|x^k-x^{k+1}\\|^2\\notag\\\\\n&= F_r(x^{k+1},y^{k}, z^{k})+\\frac{2\\lambda^{-1}+r-L}{2}\\|x^k-x^{k+1}\\|^2.\n\\end{align}\nNext,  as $\\nabla_y F_r(x,\\cdot,z)$ is $L$-Lipschitz continuous for any $x\\in\\mathcal{X}$ and $z\\in\\R^n$, we have \n    \\begin{align}\\label{primaldec-key2}\n        F_r(x^{k+1}, y^{k},z^k)-F_r(x^{k+1}, y^{k+1},z^k) \\ge \\langle\\nabla_{y} F_r(x^{k+1},y^{k}, z^k), y^{k}-y^{k+1}\\rangle-\\frac{L}{2}\\|y^k-y^{k+1}\\|^2.\n    \\end{align}\nAt last, based on the update $z^{k+1}=z^{k}+\\beta(x^{k+1}-z^{k})$, we get\n\\begin{equation}\\label{primaldec-key3}\nF_r(x^{k+1}, y^{k+1},z^{k})-F_r(x^{k+1},y^{k+1}, z^{k+1})  = \\frac{(2-\\beta)r}{2 \\beta}\\|z^{k}-z^{k+1}\\|^{2}.\n\\end{equation}\nBy summing up \\eqref{primaldec-key1}, \\eqref{primaldec-key2}, and \\eqref{primaldec-key3}, we obtain the desired result.\n\\end{proof}","sha256_utf8":"7cdf51f6f2028d1f2e0065955eef43aa7c22d81a989c663163a619836980fa02"},{"start":148124,"end":148572,"start_line":1962,"end_line":1972,"raw_tex":"\\begin{lemma}[Dual ascent]\\label{lemma-d-aes} \nFor any $k\\ge0$, we have\n\\begin{equation*}\n% \\label{dual-ascent}\n \\begin{aligned}\n    \\ & d_r(y^{k+1},z^{k+1})-d_r(y^{k},z^k)\\\\\n    \\ge\\ &\\langle\\nabla_{y} F_r(x_r(y^{k}, z^{k}),  y^{k}, z^{k}), y^{k+1}-y^{k}\\rangle-\\frac{L_{d_r}}{2}\\|y^{k}-y^{k+1}\\|^{2}\\,\\\\\n    &+\\frac{r}{2}\\left\\langle z^{k+1}-z^{k},z^{k+1}+z^{k}-2 x_r(y^{k+1}, z^{k+1})\\right\\rangle. \n    \\end{aligned}\n\\end{equation*}\n\\end{lemma}","sha256_utf8":"4810d4fb308dd628cdca84176c306e2341c98226a404391e5ad015b3466f88c1"},{"start":148573,"end":149689,"start_line":1973,"end_line":1989,"raw_tex":"\\begin{proof}\nBased on Lemma \\ref{lemma-dualdiff}, $\\nabla_{y} d_r(\\cdot,z)$ is $L_{d_r}$-Lipschitz continuous for any $z \\in \\mathbb{R}^n$. Thus, we have \n        \\begin{align*}\n        \\ & d_r(y^{k+1},z^k)-d_r(y^{k},z^k)  \n        \\geq \\langle\\nabla_{y} d_r(y^{k},z^k), y^{k+1}-y^{k}\\rangle-\\frac{L_{d_r}}{2}\\|y^{k}-y^{k+1}\\|^{2} \\\\\n        =\\ & \\langle \\nabla_{y} F_r(x_r(y^{k}, z^{k}), y^{k}, z^{k}), y^{k+1}-y^{k}\\rangle-\\frac{L_{d_r}}{2}\\|y^{k}-y^{k+1}\\|^{2}.\n        \\end{align*}\nIn addition, one has \n        \\begin{align*}\n        &d_r(y^{k+1}, z^{k+1})-d_r(y^{k+1}, z^{k})\\notag\\\\ \n        =\\ & F_r(x_r(y^{k+1}, z^{k+1}), y^{k+1}, z^{k+1})-F_r(x_r(y^{k+1}, z^{k}), y^{k+1},z^{k}) \\\\\n        \\geq\\ &  F_r(x_r(y^{k+1}, z^{k+1}), y^{k+1},z^{k+1})-F_r(x_r(y^{k+1}, z^{k+1}), y^{k+1},z^{k}) \\\\\n        =\\ & \\frac{r}{2}\\|x_r(y^{k+1}, z^{k+1})-z^{k+1}\\|^{2}-\\frac{r}{2}\\|x_r(y^{k+1}, z^{k+1})-z^{k}\\|^{2} \\\\\n        =\\ & \\frac{r}{2} \\left\\langle z^{k+1}-z^{k},z^{k+1}+z^{k}-2 x_r(y^{k+1}, z^{k+1})\\right\\rangle.\n        \\end{align*}\nFinally, by combining the above inequalities, the proof is complete.\n\\end{proof}","sha256_utf8":"1fca2c302b6635dfb08a57fe9e91583ef2cdd4ffd23d9b7ed5e3a1487861df53"},{"start":149692,"end":150033,"start_line":1991,"end_line":1999,"raw_tex":"\\begin{lemma}[Proximal descent (smoothness)]\n\\label{lemma-p-des} \n    For any $k\\ge0$, we have\n   \\begin{equation*}\n   % \\label{proximal-descent}\n    p_r(z^k)-p_r(z^{k+1}) \\ge \\frac{r}{2}\\left\\langle z^{k+1}-z^{k},2 x_r(y(z^{k+1}), z^{k})-z^{k}-z^{k+1}\\right\\rangle, \n    \\end{equation*}\n    where $y(z^{k+1})\\in Y(z^{k+1})$.\n    \\end{lemma}","sha256_utf8":"9c6a3944d717fde029af8be7da5ca2165489e0edff20206d5bdf932bee919fde"},{"start":150034,"end":150966,"start_line":2000,"end_line":2019,"raw_tex":"\\begin{proof}\n% From Sion's minimax theorem \\citep{sion1958general}, we know that\n%     \\[\n%     \\min _{x \\in \\mathcal{X}} \\max _{y \\in \\mathcal{Y}} F_r(x, y, z)=\\max_{y \\in \\mathcal{Y}} \\min_{x \\in \\mathcal{X}} F_r(x,y,z)\n%     \\]\n%     which implies\nRecall that \n% $p_r(z)$:\n    % \\[\n    $p_r(z)=\\max _{y \\in \\mathcal{Y}} d_r(y, z)$.\n    % \\]\nDue to the definition of $y(z^{k+1})$, we have\n    \\begin{align*}\n     &p_r(z^{k+1})-p_r(z^{k})\\notag\\\\\n    \\leq\\ & d_r(y(z^{k+1}), z^{k+1})-d_r(y(z^{k+1}), z^{k}) \\\\\n    \\leq\\ & F_r(x_r(y(z^{k+1}), z^{k}), y(z^{k+1}), z^{k+1})-F_r(x_r(y(z^{k+1}), z^{k}), y(z^{k+1}),z^{k}) \\\\\n    =\\ & \\frac{r}{2}\\left\\langle z^{k+1}-z^{k},z^{k+1}+z^{k}-2 x_r(y(z^{k+1}), z^{k})\\right\\rangle,\n    \\end{align*}\n    where the second inequality follows from the fact that \\[F_r(x', y, z)\\geq \\min_{x\\in\\mathcal{X}} F_r(x,y,z) = d_r(y, z)\\] for any $x'\\in\\mathcal{X}$. The proof is complete.\n    \\end{proof}","sha256_utf8":"70489bb26fa0f3126d0a04bc185df6656040df690cbc783423cd775558ea9059"},{"start":150967,"end":161183,"start_line":2020,"end_line":2155,"raw_tex":"\\paragraph{{\\bf Proof of Proposition \\ref{prop:decrease}}}\n% \\begin{proof}\nFrom Lemmas \\ref{lemma-F-des}, \\ref{lemma-d-aes}, and \\ref{lemma-p-des}, we have\n        \\begin{align*}\n        % \\label{desascent-key1}\n        & \\Phi_r(x^k,y^k,z^k)-\\Phi_r(x^{k+1},y^{k+1},z^{k+1}) \\notag\\\\\n         =\\ & F_r(x^k,y^k,z^k)-F_r(x^{k+1},y^{k+1},z^{k+1}) +2(d_r(y^{k+1},z^{k+1})-d_r(y^k,z^k))+2(p_r(z^k)-p_r(z^{k+1})) \\notag\\\\\n         \\ge\\ & \\frac{2\\lambda^{-1}+r-L}{2}\\|x^k-x^{k+1}\\|^2 +\\frac{(2-\\beta)r}{2 \\beta}\\|z^{k}-z^{k+1}\\|^{2}-\\left(L_{d_r}+\\frac{L}{2}\\right)\\|y^{k}-y^{k+1}\\|^{2}\\,  \\notag\\\\\n        &  + \\underbrace{\\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k}-y^{k+1}\\rangle+2\\langle\\nabla_{y} F_r(x_r(y^{k}, z^{k}),  y^{k}, z^{k}), y^{k+1}-y^{k}\\rangle}_{\\text{\\ding{172}}}\\, \n      \\notag \\\\ \n      & + \\underbrace{2 r\\left\\langle z^{k+1}-z^{k},x_r(y(z^{k+1}), z^{k})-x_r(y^{k+1}, z^{k+1})\\right\\rangle}_{\\text{\\ding{173}}}.\n        \\end{align*}\n% Subsequently, we simplify the terms \\text{\\ding{172}} and  \\text{\\ding{173}}. \nFirst, we simplify the term \\text{\\ding{172}}. We know that\n        \\begin{align*}\n        \\text{\\ding{172}} \n         =\\ & \\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k}-y^{k+1}\\rangle+2\\langle\\nabla_{y} F_r(x_r(y^{k}, z^{k}), y^{k}, z^{k}), y^{k+1}-y^{k}\\rangle\\\\\n        =\\ & \\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k+1}-y^{k}\\rangle + 2\\langle\\nabla_{y} F_r(x_r(y^{k}, z^{k}), y^{k}, z^{k})-\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k+1}-y^{k}\\rangle.\n        \\end{align*}\n        For the first term, we have\n        \\begin{align*}\n    &\\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k+1}-y^{k}\\rangle \\notag\\\\\n     =\\ &  \\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k)+\\frac{1}{\\alpha}(y^{k}-y^{k+1}), y^{k+1}-y^{k}\\rangle + \\frac{1}{\\alpha}\\|y^{k}-y^{k+1}\\|^2\\\\\n     =\\ & \\frac{1}{\\alpha}\\langle y^{k} + \\alpha \\nabla_{y} F_r(x^{k+1}, y^{k},z^k)-y^{k+1}, y^{k+1}-y^{k}\\rangle +\\frac{1}{\\alpha}\\|y^{k}-y^{k+1}\\|^2\\\\\n     \\ge\\ &  \\frac{1}{\\alpha}\\|y^{k}-y^{k+1}\\|^2, \n\\end{align*}\nwhere the last inequality follows from the property of the projection operator and the dual update $\ny^{k+1} = \\proj_{\\mathcal{Y}}( y^{k} + \\alpha \\nabla_{y} F_r(x^{k+1}, y^{k},z^k))\n$ (recall that $\\nabla_y F_r(x,y,z) = \\nabla_y F(x,y)$ for any $x \\in \\mathcal{X}$, $y \\in \\mathcal{Y}$, and $z \\in \\mathbb{R}^n$). For the remaining terms, we have\n                \\begin{align*}\n        %\\label{desascent-key3}\n         &2\\langle\\nabla_{y} F_r(x_r(y^{k}, z^{k}), y^{k}, z^{k})-\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k+1}-y^{k}\\rangle \\notag\\\\\n         \\ge & -2\\|\\nabla_{y} F_r(x_r(y^{k}, z^{k}),y^{k},z^{k})-\\nabla_{y} F_r(x^{k+1}, y^{k},z^{k})\\| \\cdot\\|y^{k+1}-y^{k}\\| \\notag\\\\\n         \\ge & -2 L\\|x^{k+1}-x_r(y^{k}, z^{k})\\| \\cdot\\|y^{k}-y^{k+1}\\| \\notag\\\\\n        \\ge  & -L \\zeta^2\\|y^{k}-y^{k+1}\\|^{2}-L\\zeta^{-2} \\|x^{k+1}-x_r(y^{k}, z^{k})\\|^{2} \\notag\\\\\n         \\ge & -L \\zeta^2\\|y^{k}-y^{k+1}\\|^{2}-L\\|x^{k+1}-x^{k}\\|^{2},\n        \\end{align*}\n        where the third inequality \n        % follows from $2|x||y| \\leq \\tau x^2 + \\frac{1}{\\tau}y^2$ \n        holds because $2 |a| |b| \\le \\tau a^2 + \\tfrac{1}{\\tau} b^2$\n        for any $a,b \\in \\mathbb{R}$ and $\\tau>0$, and the last inequality follows from Proposition \\ref{prop:lip}. \n        % where the first inequality is because of the Cauchy-Schwarz in equality, the second inequality is because $\\nabla_{y} K=\\nabla_{y} f$ is $L$-Lipschitz-continuous, the third inequality is due to the AM-GM inequality and the last is because of (B.5).\n% Subsequently, we focus on the left term in $\\text{\\ding{172}}$ that \nPutting the above together, we obtain\n\\begin{equation}\n\\label{desascent-key3}\n    \\text{\\ding{172}}  \\ge \\left(\\frac{1}{\\alpha}-L\\zeta^2\\right) \\|y^{k}-y^{k+1}\\|^{2}-L\\|x^{k+1}-x^{k}\\|^{2}. \n\\end{equation}\nNext, we bound the term $\\text{\\ding{173}}$ by \n\\begin{align}\\label{desascent-key2}\n  \\text{\\ding{173}} \n =\\ & 2 r \\left\\langle z^{k+1}-z^{k}, x_r(y(z^{k+1}), z^{k})-x_r(y^{k+1}, z^{k+1})\\right\\rangle \\notag\\\\\n =\\ & 2 r \\left\\langle z^{k+1}-z^{k}, x_r(y(z^{k+1}), z^{k})-x_r(y(z^{k+1}), z^{k+1})\\right\\rangle\\, \\notag\\\\\n & + 2 r\\left\\langle z^{k+1}-z^{k}, x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\right\\rangle\\notag \\\\\n\\ge &  -2r \\sigma_1\\|z^{k+1}-z^k\\|^2+2 r\\left\\langle z^{k+1}-z^{k}, x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\right\\rangle \\notag\\\\\n\\ge & -2r \\sigma_1\\|z^{k+1}-z^k\\|^2-\\frac{r}{7\\beta}\\|z^{k+1}-z^k\\|^2  -7r\\beta \\|x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\|^2, \n\\end{align}\nwhere the first inequality is due to \\eqref{lip-z} and the Cauchy-Schwarz inequality, and the second inequality again follows from the fact that $2 |a| |b| \\le \\tau a^2 + \\tfrac{1}{\\tau} b^2$ for any $a,b \\in \\mathbb{R}$ and $\\tau > 0$. Now, the inequalities \\eqref{desascent-key3} and \\eqref{desascent-key2} imply that\n\\begin{align}\\label{desas-keymid}\n        & \\Phi_r(x^k,y^k,z^k)-\\Phi_r(x^{k+1},y^{k+1},z^{k+1}) \\notag\\\\\n        \\geq\\ & \\frac{2\\lambda^{-1}+r-3L}{2}\\|x^{k}-x^{k+1}\\|^{2}+\\left(\\frac{1}{ \\alpha}-L_{d_r}-\\frac{L}{2}-L\\zeta^2\\right)\\|y^{k}-y^{k+1}\\|^{2}\\, \\notag\\\\\n        &+\\left(\\frac{(2-\\beta)r}{2 \\beta}-2r\\sigma_1-\\frac{r}{7\\beta}\\right)\\|z^{k}-z^{k+1}\\|^{2}-7r\\beta \\|x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\|^2. \n\\end{align}\nBy Lemma \\ref{lemma-dualxy} and the fact that $\\| u+v \\|^2 \\le 2(\\|u\\|^2+\\|v\\|^2)$ for any $u,v \\in \\mathbb{R}^d$, we have\n\\begin{align}\\label{desas-final-key1}\n\\|y^{k+1}-y^{k}\\|^{2} \n&=\\|y^{k+1}-y_{+}^{k}(z^{k+1})+y_{+}^{k}(z^{k+1})-y^{k}\\|^{2}\\notag\\\\\n&\\geq\\frac{1}{2}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2} -\\|y^{k+1}-y_{+}^{k}(z^{k+1})\\|^{2}\\notag\\\\\n% & \\geq \\frac{1}{2}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2} -\\alpha^2L^2\\|x^{k+1}-x_r(y^k,z^{k+1})\\|^2\\notag\\\\\n& \\geq \\frac{1}{2}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2} -2\\eta^2\\|x^{k+1}-x^k\\|^2-2\\sigma_1^2\\alpha^2L^2\\|z^{k+1}-z^k\\|^2.\n\\end{align}\n% where we recall that \n% \\[\n% y_{+}^{k}(z^{k+1}) = \\proj_{\\mathcal{Y}}(y^k+\\alpha \\nabla_y F_r (x_r(y^k,z^{k+1}),y^k,z^{k+1})).\n% \\]\n% \\leq\\ & \\alpha L\\|x^{k+1}-x_r(y^{k}, z^{k})\\| \\notag \\\\\n% \\leq\\ & \\eta\\|x^{k}-x^{k+1}\\|,\nSimilarly, by Lemma \\ref{lemma-sollip} and Lemma \\ref{lemma-dualxy}, we have\n\\begin{align}\\label{desas-final-key2}\n&\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\|^{2}\\notag\\\\\n\\leq \\ & 2\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_+^{k}(z^{k+1}),z^{k+1})\\|^2   + 2\\|x_r(y_+^{k}(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\|^2 \\notag\\\\\n\\leq \\ & 2\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_+^{k}(z^{k+1}),z^{k+1})\\|^2+2\\sigma_2^2 \\|y^{k+1}-y_{+}^{k}(z^{k+1})\\|^{2}\\notag\\\\\n\\leq \\ & 2\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_+^{k}(z^{k+1}),z^{k+1})\\|^2 + 4\\sigma_2^2\\eta^2 \\|x^{k+1}-x^k\\|^2+4\\sigma_2^2\\sigma_1^2\\alpha^2L^2\\|z^{k+1}-z^k\\|^2.\n\\end{align}\n\n% \\begin{align}\\label{desas-final-key2}\n% &\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\|^{2}\\notag\\\\\n% % =\\ & \\|x_r^\\star(z^{k+1})-x_r^\\star(z^{k})+x_r^\\star(z^{k})-x_r(y_{+}^{k}(z^{k}), z^{k})+x_r(y_{+}^{k}(z^{k}), z^{k})-x_r(y^{k+1}, z^{k})+x_r(y^{k+1}, z^{k})-x_r(y^{k+1}, z^{k+1})\\|^2 \\notag\\\\\n% \\leq\\ & 4\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y(z^{k}), z^{k})\\|^{2}+4\\|x_r(y(z^{k}), z^{k})-x_r(y_{+}^{k}(z^{k}), z^{k})\\|^{2}\\ +\\notag \\\\\n% &4\\|x_r(y_{+}^{k}(z^{k}), z^{k})-x_r(y^{k+1}, z^{k})\\|^{2}+4\\|x_r(y^{k+1}, z^{k})-x_r(y^{k+1}, z^{k+1})\\|^{2} \\notag\\\\\n% \\leq\\ & 8 \\sigma_{1}^{2}\\|z^{k}-z^{k+1}\\|^{2}+4\\|x_r(y(z^{k}), z^{k})-x_r(y_{+}^{k}(z^{k}), z^{k})\\|^{2} +4 \\sigma_{2}^{2} \\eta^2\\|x^{k}-x^{k+1}\\|^2.\n% \\end{align}\n% Besides, we recall that \n% \\begin{align}\\label{desas-final-key0}\n% x^{k+1}-z^{k} =\\frac{1}{\\beta}(z^{k+1}-z^k).\n% \\end{align} \nSubstituting \\eqref{desas-final-key1} and \\eqref{desas-final-key2} into \\eqref{desas-keymid} yields\n\\begin{align*}\n&\\Phi_r(x^k,y^k,z^k)-\\Phi_r(x^{k+1},y^{k+1},z^{k+1})\\notag\\\\\n \\geq\\ &\\left( \\frac{2\\lambda^{-1}+r-3L}{2} -28 r\\beta \\sigma_2^2\\eta^2 \\right)\\|x^{k}-x^{k+1}\\|^{2}\\ +\\left(\\frac{1}{ \\alpha}-L_{d_r}-\\frac{L}{2}-L\\zeta^2\\right) \\cdot \\\\\n& \\left(\\frac{1}{2}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2} -2\\eta^2\\|x^{k+1}-x^k\\|^2-2\\sigma_1^2\\alpha^2L^2\\|z^{k+1}-z^k\\|^2 \\right)\\ \\notag\\\\\n&+ \\left(\\frac{(2-\\beta)r}{2 \\beta}-2r\\sigma_1-\\frac{r}{7\\beta}-28r\\beta\\sigma_1^2\\sigma_2^2 \\alpha^2 L^2 \\right)\\|z^{k}-z^{k+1}\\|^{2} \\\\  & -  14r\\beta\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_+^{k}(z^{k+1}),z^{k+1})\\|^2.\n\\end{align*}\n\nSuppose that $r\\ge 3L$,\n% (which implies $L_{d_{r}}+\\frac{L}{2} \\leq 5L$)\nwhich implies that $L_{d_{r}}+\\frac{L}{2} \\leq 5L$. We observe the following:\n\\begin{itemize}\n\\item As $\\alpha \\leq \\min\\left\\{\\frac{1}{10L}, \\frac{1}{4L\\zeta^2} \\right\\}$, we have $\\frac{1}{ \\alpha}-L_{d_r}-\\frac{L}{2} \\ge \\frac{1}{2\\alpha}$ and \n$\n\\frac{1}{2\\alpha} - L\\zeta^2 \\ge \\frac{1}{4\\alpha}. \n$\n\\item As $\\beta\\leq\\frac{1}{28}$ and $\\sigma_1 \\leq \\frac{3}{2}$, $\\sigma_2\\leq \\frac{5}{2}$, we have \n\\begin{align*}\n& \\frac{(2-\\beta)r}{2 \\beta}-2r\\sigma_1-\\frac{r}{7\\beta}-28r\\beta\\sigma_1^2 \\sigma_2^2 \\alpha^2 L^2  -\\frac{1}{2\\alpha} \\sigma_1^2\\alpha^2L^2 \\\\ \n\\ge\\ & \\frac{6r}{7\\beta} -\\frac{7r}{2}- 28r\\beta\\left(\\frac{3}{2}\\right)^2\\left(\\frac{5}{2}\\right)^2\\left(\\frac{1}{10L}\\right)^2L^2-\\frac{r}{24}\\\\ \n\\ge\\ &  \\frac{r}{\\beta}\\left(\\frac{6}{7}-4\\beta- 14\\beta^2\\right) \\ge \\frac{4r}{7\\beta}.\n\\end{align*}\n\\item As $\\lambda^{-1} \\ge L$ and recalling that $\\eta = \\alpha L \\zeta$, we have \n\\[\\alpha\\leq\\frac{1}{4L \\zeta^2} = \\frac{\\lambda^{-1}}{4L^2\\zeta^2} \\frac{L}{\\lambda^{-1}} \\leq \\frac{\\lambda^{-1}}{4L^2\\zeta^2}\\quad \\text{and}\\quad \n\\frac{\\eta^2}{2\\alpha} = \\frac{\\alpha L^2\\zeta^2}{2} \\leq \\frac{1 }{8\\lambda}.\n\\]\nMoreover, since $\\beta  \\leq \\tfrac{(r-L)^2}{14 \\alpha r (2r-L)^2} = \\tfrac{1}{14 \\alpha r \\sigma_2^2}$,\n we have \n \\[\n 28 r\\beta \\sigma_2^2\\eta^2 \\leq \\frac{ 2\\eta^2}{\\alpha} \\leq \\frac{1}{2\\lambda}\\quad\\text{and}\\quad \\frac{2\\lambda^{-1}+r-3L}{2} -28 r\\beta \\sigma_2^2\\eta^2 - \\frac{\\eta^2}{2\\alpha} \\ge \\frac{3}{8\\lambda}.\n\\]\n\\end{itemize}\n% Together all pieces, we get \nPutting everything together yields\n\\begin{align*}\n&\\Phi_r(x^k,y^k,z^k)-\\Phi_r(x^{k+1},y^{k+1},z^{k+1}) \\\\\n\\ge\\ & \\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{8\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2} +\\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2} \\\\  & -14r\\beta\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2}.\n\\end{align*}\nThe proof is complete.\n% \\end{proof}\n","sha256_utf8":"f5aafd7236ce5634367e5fc252462af53717fddaf3b70fb8b1119eebdadd3aa7"},{"start":161253,"end":163160,"start_line":2159,"end_line":2196,"raw_tex":"\\begin{proof}\nThe proof follows closely that of Proposition \\ref{prop:dual_eb_KL}. Let $\\psi:\\R^n\\times \\R^n\\rightarrow\\R$ be the function defined by\n\\[\n\\psi(x,z) = \\max\\limits_{y\\in \\mathcal{Y}} F_r(x,y,z)\n\\]\nand consider arbitrary $x \\in \\mathcal{X}$, $y \\in \\mathcal{Y}$, and $z \\in \\mathbb{R}^n$. Again, note that  $\\psi(\\cdot,z)$ is $(r-L)$-strongly convex. This implies that\n% for any $x\\in \\mathcal{X}$, we have\n\\begin{equation}\n\\label{OS-GS-better-key1}\n\\psi(x,z) - \\psi(x^\\star_r(z),z) \\ge \\frac{r-L}{2}\\|x-x^\\star_r(z)\\|^2. \n\\end{equation}\nMoreover, \n% for any $x\\in \\mathcal{X}$, \nwe  have\n\\begin{align} \n\\label{OS-GS-better-key2}\n&\\psi(x,z)-\\psi(x_r^\\star(z),z) \\notag\\\\   \n\\leq \\  &\\psi(x,z)-F_r(x_r(y_+(z),z),y_+(z),z)\\notag \\\\\n= \\ & \\max_{y^\\prime\\in \\mathcal{Y}} F(x,y^\\prime) + \\frac{r}{2}\\|x-z\\|^2-F_r(x_r(y_+(z),z),y_+(z),z)\\notag\\\\\n=\\,& \\max_{y^\\prime\\in \\mathcal{Y}}F(x,y^\\prime)-F(x_r(y_+(z),z),y_+(z)) + \\frac{r}{2}\\|x-z\\|^2-\\frac{r}{2}\\|x_r(y_+(z),z)-z\\|^2,\n\\end{align}\nwhere the  inequality follows from\n\\begin{align*}\nF_r(x_r(y_+(z),z),y_+(z),z) \n=\\ & \\min_{x'\\in\\mathcal{X}}\\left\\{ F(x',y_+(z)) +\\frac{r}{2}\\|x'-z\\|^2\\right\\} \\\\\n\\leq\\ & \\max_{y' \\in \\mathcal{Y}}\\min_{x'\\in\\mathcal{X}}\\left\\{F(x',y') +\\frac{r}{2}\\|x'-z\\|^2\\right\\} \\\\\n\\leq  \\ & \\min_{x'\\in\\mathcal{X}}\\max_{y' \\in \\mathcal{Y}}\\left\\{F(x',y') +\\frac{r}{2}\\|x'-z\\|^2\\right\\}\n= \\psi(x_r^\\star(z),z).\n\\end{align*}\nAs \\eqref{OS-GS-better-key1} and \\eqref{OS-GS-better-key2} hold for any $x\\in\\mathcal{X}$, we obtain the intermediate relation\n\\begin{equation*}\n% \\label{OS-GS-better-key3} \n    \\frac{r-L}{2}\\|x_r^\\star(z)-x_r(y_+(z),z)\\|^2 \\leq \\max\\limits_{y^\\prime\\in \\mathcal{Y}} F(x_r(y_+(z),z),y^\\prime) - F(x_r(y_+(z),z),y_+(z)) \n\\end{equation*}\nby taking $x= x_r(y_+(z),z)$.\n\nThe remaining steps of the proof are the same as those of Proposition \\ref{prop:dual_eb_KL}. For brevity, we omit them here.\n\\end{proof}","sha256_utf8":"f0e754c01317418b0878168a5a6ce6ca1187717e613229e65b45af4cbed08729"},{"start":163432,"end":163745,"start_line":2205,"end_line":2213,"raw_tex":"\\begin{lemma}\\label{lemma-dual-bd}\nFor any $y \\in \\mathcal{Y}$ and $z\\in\\R^n$, we have\n\\begin{equation*}\n% \\label{dual-bd}\n\\|x^\\star_r(z)-x_r(y_{+}(z), z)\\|^{2} \n \\leq \\kappa \\|y-y_{+}(z)\\| ,\n\\end{equation*}\nwhere $\\kappa:=\\frac{1+ \\alpha L\\sigma_2+\\alpha L}{\\alpha(r-L)}\\cdot{\\rm diam}(\\mathcal{Y})$.\n\\end{lemma}","sha256_utf8":"6f833961db4b3446d9467678bb3f0c9af107a4490c9171bc01ba2614caa82ef4"},{"start":163746,"end":166612,"start_line":2214,"end_line":2253,"raw_tex":"\\begin{proof}\nLet $y \\in \\mathcal{Y}$ and $z \\in \\mathbb{R}^n$ be arbitrary. Recall that $y(z)$ is an arbitrary vector in $Y(z)$. By the $(r-L)$-strong convexity of $F_r(\\cdot, y, z)$, we have\n\\begin{align}\n&F_r(x_r^\\star(z), y_{+}(z), z)-F_r(x_r(y_{+}(z), z), y_{+}(z), z)  \\geq \\frac{r-L}{2}\\|x_r(y_{+}(z), z)-x^\\star_r(z)\\|^{2}, \\notag\\\\\n&F_r(x_r(y_{+}(z), z), y(z), z)-F_r(x_r^\\star(z), y(z), z)  \\geq \\frac{r-L}{2}\\|x_r(y_{+}(z), z)-x_r^\\star(z)\\|^{2}.\\notag\n\\end{align}\n Moreover, since $(x_r^\\star(z),y(z))$ is a saddle point of the convex-concave function $F_r(\\cdot,\\cdot,z)$ \\citep{sion1958general}, we have $F_r(x_r^\\star(z), y(z), z) \\geq F_r(x_r^\\star(z), y_{+}(z), z)$, which implies that\n\\begin{equation}\\label{dualbd-strcvx}\nF_r(x_r(y_{+}(z), z), y(z), z)-F_r(x_r(y_{+}(z), z), y_{+}(z), z) \\geq  (r-L)\\|x_r(y_{+}(z), z)-x_r^\\star(z)\\|^{2}.\n\\end{equation}\nNote that $y_{+}(z)$ is the maximizer of \n\\[\n\\max_{y' \\in \\mathcal{Y}}\\big\\langle y+\\alpha \\nabla_{y} F_r(x_r(y, z), y, z)-y_{+}(z) , y'\\big\\rangle.\n\\]\nFor simplicity, we define the function $\\xi: \\mathbb{R}^d \\rightarrow \\mathbb{R}$ by\n\\begin{align*}\n\\xi(\\cdot)\n:=\\ &\\alpha F_r(x_r(y_{+}(z), z),\\cdot, z)-\\big\\langle \\alpha\\nabla_{y} F_r(x_r(y_{+}(z), z), y_{+}(z), z),\\cdot\\big\\rangle \\\\ \\ &-\\big\\langle y_{+}(z)-y-\\alpha \\nabla_{y} F_r(x_r(y, z), y, z), \\cdot\\big\\rangle.\n\\end{align*}\nThen, we have\n\\begin{align*}\n\\max_{y' \\in \\mathcal{Y}}\\ \\xi(y')\n&\\leq \\alpha F_r(x_r(y_{+}(z), z),y_+(z), z)-\\big\\langle\\alpha\\nabla_y F_r(x_r(y_{+}(z), z),y_+(z), z), y_+(z)\\big\\rangle\\, \\notag\\\\\n&\\quad + \\max_{y' \\in \\mathcal{Y}} \\big\\langle y+\\alpha \\nabla_{y} F_r(x_r(y, z), y, z)-y_{+}(z), y'\\big\\rangle \\notag\\\\\n&\\leq \\xi(y_{+}(z)),\n\\end{align*}\nwhere the first inequality holds because $F_r(x_r(y_{+}(z), z),\\cdot,z)$ is concave. Therefore, we have $\\xi(y(z))\\leq \\xi(y_{+}(z))$, which implies that\n\\begin{align}\\label{dualbd-key}\n&F_r(x_r(y_{+}(z), z), y(z), z)-F_r(x_r(y_{+}(z), z), y_{+}(z), z) \\notag\\\\\n\\leq &\\ \\frac{1}{\\alpha}\\Big\\langle y(z)-y_{+}(z), \\alpha\\nabla_{y} F_r(x_r(y_{+}(z), z), y_{+}(z), z)-\\alpha \\nabla_{y} F_r(x_r(y, z), y, z)\\Big\\rangle \\notag \\\\ \\ & \\ + \\frac{1}{\\alpha}\\Big\\langle y(z)-y_{+}(z), y_{+}(z)-y\\Big\\rangle \\notag\\\\\n\\leq &\\ L \\|y_{+}(z)-y(z)\\|(\\|x_r(y_{+}(z), z)-x_r(y, z)\\|+ \\|y_{+}(z)-y\\|)  + \\frac{1}{\\alpha}\\|y_{+}(z)-y(z)\\|\\cdot \\|y-y_{+}(z)\\|\\notag\\\\\n\\leq &\\ \\left(\\frac{1}{\\alpha}+L\\sigma_2+L\\right)\\|y_{+}(z)-y(z)\\| \\cdot\\|y-y_{+}(z)\\|.\n\\end{align}\nHere, the second inequality is due to the $L$-Lipschitz continuity of $\\nabla_y F_r(\\cdot,\\cdot,z)$ and the last inequality is from \\eqref{lip-y}.\nHence, by combining \\eqref{dualbd-strcvx} and \\eqref{dualbd-key}, we obtain\n\\[\n(r-L)\\|x_r^\\star(z)-x_r(y_{+}(z), z)\\|^{2}\\leq \\left(\\frac{1}{\\alpha}+ L\\sigma_2+L\\right){\\rm diam}(\\mathcal{Y})\\cdot\\|y-y_{+}(z)\\|,\n\\]\nwhich proves the desired result.\n\\end{proof}","sha256_utf8":"6c7718cd244789a2247de5b2385d88b7131ff0cb917f535c271d1b8971b947cd"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"平滑PLDA迭代复杂度","source_units":[{"start":73033,"end":74871,"start_line":1020,"end_line":1033,"raw_tex":"\\begin{theorem}[Main theorem]\n\\label{thm:general}\n % Suppose that \n % \\[r\\ge3L,\\ \\lambda\\le \\frac{1}{L},\\ \\alpha \\leq \\min\\left\\{\\frac{1}{10L},\\ \\frac{1}{4L\\zeta^2} \\right\\},\\ \\beta  \\leq \\min \\left\\{\\frac{1}{28},\\ \\frac{(r-L)^2}{14\\alpha r(2r-L)^2}\\right\\}.\\] \n Under the setting of Proposition \\ref{prop:decrease}, for any integer $K>0$, there exists an index $k \\in\\{1,2, \\ldots, K\\}$ such that the following hold:\n \\begin{enumerate}[label=\\normalfont(\\alph*)]\n   \\item \\rm{(Concave):} Suppose that $F(x,\\cdot)$ is concave for all $x \\in \\mathcal{X}$. If $\\max_{y\\in \\mathcal{Y}} F(\\cdot,y)$ is bounded below on $\\mathcal{X}$ and $\\beta\\leq \\mathcal{O}(K^{-\\frac{1}{2}})$,  then $\\left(x^{k+1}, y^{k+1}\\right)$ is an $\\mathcal{O}(K^{-\\frac{1}{4}})$-GS and $z^{k+1}$ is an $\\mathcal{O}(K^{-\\frac{1}{4}})$-OS of problem \\eqref{eq:problem}.\n  \\item \\rm{(K\\L\\ exponent $\\theta\\in(\\frac{1}{2},1)$):} Suppose that Assumption \\ref{ass:kl} holds with $\\theta \\in (\\tfrac{1}{2},1)$. If $\\mathcal{X}$ is compact and  $ \\beta\\leq \\mathcal{O}(K^{-\\frac{2\\theta-1}{2\\theta}})$,\n   then $\\left(x^{k+1}, y^{k+1}\\right)$ is an $\\mathcal{O}( K^{-\\frac{1}{4\\theta}})$-GS and $z^{k+1}$ is an $\\mathcal{O}( K^{-\\frac{1}{4\\theta}})$-OS  of problem \\eqref{eq:problem}.\n  \\item \\rm{(K\\L\\ exponent $\\theta\\in[0,\\frac{1}{2}]$):} Suppose that Assumption \\ref{ass:kl} holds with $\\theta \\in [0, \\tfrac{1}{2}]$.\n If $\\mathcal{X}$ is compact and  $\\beta  \\leq \\frac{ {\\rm diam}(\\mathcal{Y})^{\\frac{2\\theta-1}{\\theta}}}{224 \\alpha r\\omega_2^2}$ when $\\theta \\in (0,\\tfrac{1}{2}]$ {\\rm (}resp., $\\beta  \\leq \\tfrac{ 1}{224 \\alpha r\\omega_1^2}$ when $\\theta=0${\\rm )}, then $\\left(x^{k+1}, y^{k+1}\\right)$ is an $\\mathcal{O}(K^{-\\frac{1}{2}})$-GS and $z^{k+1}$ is an $\\mathcal{O}(K^{-\\frac{1}{2}})$-OS of problem \\eqref{eq:problem}.\n  \\end{enumerate}\n  \n\\end{theorem}","sha256_utf8":"34ab601eebc9204c5cf3fa9e38d8e96cb1429710a4ebbc9be17efc41c3f131a7"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"主定理完整证明","source_units":[{"start":77680,"end":87655,"start_line":1059,"end_line":1182,"raw_tex":"\\begin{proof}\n\nAs mentioned in Section \\ref{sec:framework}, the global convergence rate of smoothed PLDA depends on the specific regime of the growth power $v$ in \\eqref{eq:step_dual}. Let us begin by considering cases (a) and (b), i.e., either the dual function is concave or the dual problem possesses the K\\L{} property with exponent $\\theta\\in(\\frac{1}{2},1)$.\n\n\nIn view of the basic descent estimate in Proposition \\ref{prop:decrease}, we consider the following two scenarios separately:\\\\\n(1) There exists a $ k \\in\\{0,1, \\ldots, K-1\\}$ such that\n\\begin{align*}\n\\ &\\frac{1}{2}\\max \\left\\{\\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}, \\frac{1}{8\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 , \\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}\\right\\}\\\\\n\\leq \\ & 14r\\beta\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2}.\n\\end{align*}\n(2) For any $ k \\in\\{0,1, \\ldots, K-1\\}$, we have\n\\begin{align*}\n\\ &\\frac{1}{2}\\max \\left\\{\\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}, \\frac{1}{8\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 , \\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}\\right\\}\\\\\n\\ge\\ & 14r\\beta\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2}.\n\\end{align*}\n\n\\textbf{Scenario (1)}. For the case where the K{\\L} exponent $\\theta \\in (\\tfrac{1}{2},1)$, we deduce from Proposition \\ref{prop:dual_eb_KL}(b) that \n\\begin{align*}\n    \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2  & \\leq 224r\\alpha\\beta\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2}  \\\\ &\\leq  224r\\alpha\\beta \\omega_2^2 \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{\\theta}},  \n\\end{align*}\nwhich gives $\\|y^{k}-y_{+}^{k}(z^{k+1})\\|\\leq \\rho_1\\beta^{\\frac{\\theta}{2\\theta-1}}$ with $\\rho_1 := (224r\\alpha\\omega_2^2)^{\\frac{\\theta}{2\\theta-1}}$. Additionally, using the update $z^{k+1} = z^k + \\beta( x^{k+1} - z^k )$ and Proposition \\ref{prop:dual_eb_KL}(b), we have\n\\begin{align*}\n\\|x^{k+1}-z^{k}\\|^{2}  \\ & =\\frac{1}{\\beta^2}\\|z^{k+1}-z^{k}\\|^{2}\\\\\n\\ &  \\leq 49\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2} \\\\\n\\ & \\leq  49 \\omega_2^2 \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{\\theta}}\\leq \\rho_2^2\\beta^{\\frac{1}{2\\theta-1}},\n\\end{align*}\nwhere $\\rho_2 := \\sqrt{49\\omega_2^2\\rho_1^{\\frac{1}{\\theta}}}$. \nLastly, we have\n\\begin{align*}\n    \\|x^{k+1}-x^k\\|^2 \\ & \\leq {\\frac{224}{3}r\\beta\\lambda}\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2} \\\\\n   \\  &\\leq {\\frac{224}{3}r\\omega^2_2\\beta\\lambda}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{\\theta}} \\leq \\rho_3^2 \\beta^{\\frac{2\\theta}{2\\theta-1}},  \n\\end{align*}\nwhere $\\rho_3 := \\sqrt{{\\frac{224}{3}r\\omega_2^2\\lambda}\\rho_1^{\\frac{1}{\\theta}}}$. \nCombining the above inequalities, we have\n\\begin{align*}\n& \\max \\left\\{\\|x^{k}-x^{k+1}\\|,\\|y^{k}-y_{+}^{k}(z^{k+1})\\|,\\|z^{k}-x^{k+1}\\|\\right\\} \\leq  \\max\\left\\{ \\rho_1\\beta^{\\frac{\\theta}{2\\theta-1}}, \\rho_2\\beta^{\\frac{1}{4\\theta-2}}, \\rho_3 \\beta^{\\frac{\\theta}{2\\theta-1}}\\right\\}. \n\\end{align*}\nIt then follows from Lemma \\ref{lemma-episolcol}\n that $\\left(x^{k+1}, y^{k+1}\\right)$ is a $\\rho\\cdot\\max\\{ \\rho_1\\beta^{{\\theta}/{(2\\theta-1)}},$ $ \\rho_2\\beta^{1/{(4\\theta-2)}}, \\rho_3\\beta^{{\\theta}/{(2\\theta-1)}}\\}$-GS of problem \\eqref{eq:problem}. \n \n For the concave case, we can replace the primal-dual quantity $x_r( y(z^{k+1}), z^{k+1} )$ by the primal quantity $x_r^\\star( z^{k+1} )$ on the right-hand side of the inequality for Scenario (1) and apply Lemma \\ref{lemma-dual-bd} in Appendix \\ref{sec:dual} (so that $\\theta$ and $\\omega_2$ are replaced by $1$ and $\\kappa = \\tfrac{1 + \\alpha L \\sigma_2 + \\alpha L}{\\alpha (r-L)} \\cdot \\textrm{diam}(\\mathcal{Y})$, respectively) to conclude that $\\left(x^{k+1}, y^{k+1}\\right)$ is a $\\max\\{ \\rho_1\\beta, \\rho_2\\beta^{1/2}, \\rho_3 \\beta\\}$-GS of problem \\eqref{eq:problem}.  \n \n\n\nFurthermore, observe that for the case where $\\theta \\in (\\tfrac{1}{2}, 1)$, the optimization-stationarity measure of $z^{k+1}$ satisfies\n\\begin{equation}\n\\label{eq:os-gs}\n\\begin{aligned}\n    & \\|z^{k+1}-x_r^\\star(z^{k+1})\\| \\\\\n    \\leq\\ &  \\|z^{k+1}-x^{k+1}\\|+\\| x^{k+1} - x_r(y^k,z^k) \\| + \\| x_r(y^k,z^k) - x_r(y^k,z^{k+1}) \\|\\\\\n    \\ & + \\|x_r(y^k,z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|+ \\|x_r(y_{+}^{k}(z^{k+1}), z^{k+1})-x_r^\\star(z^{k+1})\\|\\\\\n     \\leq\\ &  (1+\\sigma_1)\\|z^{k}-x^{k+1}\\| + \\zeta\\|x^{k}-x^{k+1}\\| + \\sigma_2 \\|y^k-y_{+}^{k}(z^{k+1})\\|+ \\omega_2 \\|y^k-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{2\\theta}} \\\\\n    \\leq\\ & \\max\\left\\{\\left(\\rho_2(1+\\sigma_1)+ \\omega_2 \\rho_1^\\frac{1}{2\\theta}\\right)\\beta^{\\frac{1}{4\\theta-2}},(\\zeta\\rho_3+\\sigma_2\\rho_1) \\beta^{\\frac{\\theta}{2\\theta-1}} \\right\\}, \n\\end{aligned}\n\\end{equation}\nwhere the second inequality follows from the update $z^{k+1} = z^k + \\beta (x^{k+1} - z^k)$, Proposition \\ref{prop:lip}, Corollary \\ref{prop:dual_eb_KL2}, Lemma \\ref{lemma-sollip}, and the fact that $0 < \\beta \\le \\tfrac{1}{28}$.\nIt is evident that the dependence on $\\beta$ is identical to that of the game-stationarity measure of $(x^{k+1}, y^{k+1})$.\nFor the concave case, the optimization-stationarity measure of $z^{k+1}$ can be bounded similarly.\n\n\\textbf{Scenario (2)}. Proposition \\ref{prop:decrease} implies that for any $k \\in\\{0,1, \\ldots, K-1\\}$, we have \n\\begin{align}\n\\label{eq:suff-decrease}\n\\Phi_r^k-\\Phi_r^{k+1} \n\\ge \\frac{3}{16\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{16\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 +\\frac{2r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}.\n\\end{align}\nThe assumptions in case (a)  imply that \n\\[ \\min_{x\\in\\mathcal{X}}\\max_{y\\in\\mathcal{Y}} F_r(x,y,z) = \\max_{y\\in\\mathcal{Y}} \\min_{x\\in\\mathcal{X}} F_r(x,y,z) \\]\nfor all $z \\in \\mathbb{R}^n$ (cf.~the proof of~\\citep[Lemma B.7]{zhang2020single}) and lead to the lower boundedness of $p_r$, while the assumptions in case (b) directly imply that $p_r$ is lower bounded.\nIn both cases, there exists a $\\underline{\\Phi} > -\\infty$ such that for all $x \\in \\mathcal{X}$, $y \\in \\mathcal{Y}$, and $z \\in \\mathbb{R}^n$,\n\\begin{align*}\n& \\Phi_r(x, y, z)  = p_r(z)+(F_r(x,y,z)-d_r(y, z))+(p_r(z)-d_r(y, z))\\geq p_r(z)\\geq \\underline{\\Phi} >-\\infty.\n\\end{align*}\nIt follows that\n    \\begin{align*}\n    &\\Phi_r^0-\\underline{\\Phi} \\geq \\sum_{k=0}^{K-1}\\Phi_r(x^k,y^k,z^k)-\\Phi_r(x^{k+1},y^{k+1},z^{k+1}) \\\\\n    % \\geq\\ & \\sum_{k=0}^{K-1} \\frac{1}{32\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{16\\alpha}\\|y^{k}-y_{+}^{k}(z^{k})\\|^2 +\\frac{2r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2} \\\\\n    \\geq \\ & \\sum_{k=0}^{K-1} \\min\\left\\{\\frac{3}{16\\lambda},\\frac{1}{16\\alpha},\\frac{2\\beta r}{7}\\right\\} \\left(\\|x^{k}-x^{k+1}\\|^{2}+\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 + \\|z^{k}-x^{k+1}\\|^{2} \\right),\n    \\end{align*}\n    where the last inequality is due to \\eqref{eq:suff-decrease} and the update $z^{k+1}=z^{k}+\\beta(x^{k+1}-z^k)$.\n    In particular, there exists a $k \\in\\{0,1, \\ldots, K-1\\}$ such that\n    \\[\n    \\max \\left\\{\\|x^{k}-x^{k+1}\\|^{2},\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2}, \\|x^{k+1}-z^{k}\\|^{2}\\right\\} \\leq \\frac{\\Phi_r^0-\\underline{\\Phi}}{\\min\\left\\{\\frac{3}{16\\lambda},\\frac{1}{16\\alpha},\\frac{2\\beta  r}{7}\\right\\}  K}.\n    \\]\n% \\Jiajin{continue here!}\nBased on Lemma \\ref{lemma-episolcol} and the fact that $0 < \\beta \\le \\tfrac{1}{28}$,  we conclude that $\\left(x^{k+1}, y^{k+1}\\right)$ is an $\\mathcal{O}(\\sqrt{1/{K\\beta}})$-GS of problem \\eqref{eq:problem}. Moreover, by using the same argument as in \\eqref{eq:os-gs}, we can show that $z^{k+1}$ is an $\\mathcal{O}(\\sqrt{1/{K\\beta}})$-OS of problem \\eqref{eq:problem}. Upon setting $\\beta = C K^{-\\frac{1}{2}}$ for the concave case and  $\\beta = C K^{-\\frac{2\\theta-1}{2\\theta}}$ for the case where the K{\\L} exponent $\\theta \\in (\\tfrac{1}{2}, 1)$, with $C$ being a suitably chosen constant, we see that the \n% where $C$ is a constant.\n% Similarly, for the general concave case, we can set $\\beta = C K^{-\\frac{1}{2}}$. \nbounds obtained for the optimization-stationarity and game-stationarity measures are of the same order for both Scenarios (1) and (2). This establishes cases (a) and (b).\n\nNext, let us consider case (c). First, suppose that the K{\\L} exponent $\\theta \\in (0, \\tfrac{1}{2}]$. \nOnce again, we can use Proposition \\ref{prop:decrease} to obtain \n\\begin{align*}\n\\Phi_r^k-\\Phi_r^{k+1} \n\\ge\\ &  \\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{8\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 +\\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}\\\\\n\\ & -14r\\beta\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2}.\n\\end{align*}\nUsing Proposition \\ref{prop:dual_eb_KL}(b) and the boundedness of $\\mathcal{Y}$, we have\n% there exists  for each $\\theta \\in(0,\\frac{1}{2})$ that \n\\begin{equation}\\label{dual-eb-bd}\n\\begin{aligned}\n  \\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|\n&\\leq \\omega_2 \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{2\\theta}}\\\\\n&=  \\omega_2 \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{2\\theta}-1} \\|y^{k}-y_{+}^{k}(z^{k+1})\\|\\\\\n&\\leq   \\omega_2\\cdot {\\rm diam}(\\mathcal{Y})^{\\frac{1}{2\\theta}-1}\\cdot \\|y^{k}-y_{+}^{k}(z^{k+1})\\|.\n\\end{aligned}\n\\end{equation}\nIt follows that\n\\begin{align*}\n\\Phi_r^k-\\Phi_r^{k+1} \n\\ge \\ &  \\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\left(\\frac{1}{8\\alpha}-14r\\beta\\omega_2^2\\cdot{\\rm diam}(\\mathcal{Y})^{\\frac{1-2\\theta}{\\theta}}\\right) \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2  +\\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}.\n\\end{align*}\nSince \n$\\beta  \\leq \\frac{ {\\rm diam}(\\mathcal{Y})^{\\frac{2\\theta-1}{\\theta}}}{224 \\alpha r\\omega_2^2}$, we have \n\\begin{align*}\n&\\Phi_r^k-\\Phi_r^{k+1} \\ge  \\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{16\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 +\\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}.\n\\end{align*}\nThe desired result can then be obtained by following a similar argument as that in the paragraph proceeding \\eqref{eq:suff-decrease}.\n\nWhen the K{\\L} exponent $\\theta=0$, we can apply Proposition \\ref{prop:dual_eb_KL}(a) to derive the inequality \\eqref{dual-eb-bd} \n% by letting $\\theta=\\frac{1}{2}$ and replacing $\\omega_2$ with $\\omega_1$. \nwith $\\theta$ and $\\omega_2$ being replaced by $\\tfrac{1}{2}$ and $\\omega_1$, respectively.\nThe desired result then follows by adapting the argument in the preceding paragraph.\n\\end{proof}","sha256_utf8":"5e26e8ec1486aa58d52f3d5f049005bfb0905f56ca12e76ea1b096b21890486a"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"极小极大问题与可行域","source_units":[{"start":10315,"end":10428,"start_line":304,"end_line":308,"raw_tex":"\\begin{equation}\n\\label{eq:problem}\n\\min_{x\\in \\mathcal{X}} \\max_{y\\in\\mathcal{Y}} F(x,y).\n\\tag{P}\n\\end{equation}","sha256_utf8":"41a53c209aedb3c19ae918fda10b12b793a7abdea30cf8e8cdd48fce95641c76"},{"start":10429,"end":11139,"start_line":309,"end_line":309,"raw_tex":"Here, $F:\\R^n \\times \\R^d \\rightarrow \\R$ can be both nonconvex with respect to $x$ and nonconcave with respect to $y$, $\\mathcal{X}\\subseteq \\R^n$ is a nonempty closed convex set, and $\\mathcal{Y}\\subseteq \\R^d$ is a nonempty compact convex set. This problem has a wide range of applications in machine learning and operations research. Examples include learning with non-decomposable loss~\\citep{shalev2016minimizing,zhang2018ell_1}, training generative adversarial networks~\\citep{arjovsky2017wasserstein,goodfellow2020generative}, adversarial training~\\citep{madry2017towards,sinha2017certifying}, and (distributionally) robust optimization (DRO)~\\citep{bertsimas2011theory,rahimian2019distributionally}. \n","sha256_utf8":"1e4e7dedcf3236e5d2380907b69a9c1927c638de103a4f5ab55eddb445c6af53"}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","title":"复合原始结构与对偶光滑性","source_units":[{"start":32054,"end":33306,"start_line":481,"end_line":501,"raw_tex":"\\begin{assumption}[Problem setup] \\label{ass:basic}\nThe following assumptions on the objective function $F$ of problem \\eqref{eq:problem} hold throughout the paper. \n\\begin{enumerate}[label=\\normalfont(\\alph*)]\n\\item   $(\\textrm{\\bf Primal function})$ For all $y\\in \\mathcal{Y}$, the function $F(\\cdot,y): \\mathbb{R}^n \\rightarrow \\mathbb{R}$ takes the form $F(\\cdot, y):=h_y\\circ c_y(\\cdot)$, where $c_y:\\R^n\\rightarrow \\R^m$ is \ncontinuously differentiable\nwith $L_c$-Lipschitz continuous Jacobian mapping, i.e.,\n\\[\n\\|\\nabla c_y(x)-\\nabla c_y(x')\\| \\leq L_c\\|x-x'\\| \\quad \\text { for all } x, x' \\in \\mathcal{X};\n\\]\nand $h_y:\\R^m\\rightarrow \\R$  is  convex and $L_h$-Lipschitz continuous, i.e.,\n\\[\n|h_y(z)-h_y(z')| \\leq L_h\\|z-z'\\| \\quad \\text { for all } z, z' \\in \\R^{m}.\n\\]\n\\item  $(\\textrm{\\bf Dual function})$ For all $x\\in \\mathcal{X}$, the function $F(x, \\cdot):\\R^d \\rightarrow \\R$ is continuously differentiable  with $\\nabla_y F(\\cdot,\\cdot)$ being $L$-Lipschitz continuous on $\\mathcal{X}\\times \\mathcal{Y}$, i.e., \n\\[\n\\|\\nabla_y F(x,y)-\\nabla_y F(x',y')\\| \\leq L\\|(x,y)-(x',y')\\| \\,\n\\text{ for all }  (x,y), (x',y') \\in \\mathcal{X}\\times\\mathcal{Y}.\n\\]\nWithout loss of generality, we may take $L=L_h L_c$.\n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"431e6386f7b15b64d25ba45b08423b0976018f935a3e14c24aeb9d4e6c45accb"}]},{"id":"S005","semantic_type":"assumption","kind":"assumption","title":"统一对偶KL条件","source_units":[{"start":33326,"end":33977,"start_line":503,"end_line":512,"raw_tex":"\\begin{assumption}[K\\L{} property with exponent $\\theta$ for dual problem]\n\\label{ass:kl}\nFor all  $x\\in\\mathcal{X}$, the problem $\\max_{y\\in \\mathcal{Y}} F(x,y)$ has a nonempty solution set and a finite optimal value. Moreover, there exist $\\mu>0$ and $\\theta \\in [0,1)$ such that  \n\\[\n\\dist(0, -\\nabla_y F(x,y)+\\partial\\iota_{\\mathcal{Y}}(y)) \\ge \\mu  \\left(\\max\\limits_{y'\\in \\mathcal{Y}} F(x,y')-F(x,y)\\right)^{\\theta}\n\\]\nfor all $x \\in \\mathcal{X}$ and  $y \\in \\mathcal{Y}\\setminus \\mathcal{Y}^\\star(x)$, where $\\mathcal{Y}^\\star(x):= \\argmax_{y' \\in \\mathcal{Y}} F(x,y')$ \n% satisfying $F(x,y)<\\max_{y'\\in\\mathcal{Y}} F(x,y')$\n.\n\\end{assumption}","sha256_utf8":"32db9ebc090f408ba374e09d5d78e03361990fd8bf847391c9086570d3662925"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"值函数与正则势函数","source_units":[{"start":34648,"end":35295,"start_line":517,"end_line":524,"raw_tex":"Let us now examine the stationarity measures discussed in this paper. We define the value function $f:\\mathbb{R}^n \\rightarrow \\mathbb{R}$, the potential function $F_r:\\R^n\\times \\R^d\\times \\R^n\\rightarrow\\R$, and the dual potential value function $d_r:\\R^d\\times \\R^n\\rightarrow \\R$ by\n% \\[\n% F_r(x,y,z) := F(x,y)+\\frac{r}{2}\\|x-z\\|^2\\quad \\text{and}\\quad d_r(y,z):=\\min_{x\\in\\mathcal{X}} F_r(x,y,z),\n% \\]\n\\begin{align*}\nf(x) := \\max_{y \\in \\mathcal{Y}} F(x,y),\\quad F_r(x,y,z) := F(x,y) + \\frac{r}{2} \\| x-z \\|^2,\\quad d_r(y,z) := \\min_{x \\in \\mathcal{X}} F_r(x,y,z),\n\\end{align*}\nrespectively, where we assume $r> L$ in the rest of this paper.\n","sha256_utf8":"5b8a00e496ef108c89659f0b4d70e5bb3bf3e71db938544692e5094dc50016ce"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"优化驻点与博弈驻点","source_units":[{"start":36155,"end":37281,"start_line":531,"end_line":546,"raw_tex":"\\begin{definition}[Stationarity measures]\n\\label{defi:primal-dual}\nLet $\\epsilon \\ge 0$ be given.\n\\begin{enumerate}[label=\\normalfont(\\alph*)]\n\\item The point $x\\in\\mathcal{X}$ is an $\\epsilon$-optimization-stationary point {\\rm($\\epsilon$-OS)} of problem \\eqref{eq:problem} if \n\\[\\|\\prox_{\\frac{1}{r}f+\\iota_{\\mathcal{X}}}(x)-x\\|\\leq \\epsilon.\\]\n\\item The pair $(x, y)\\in\\mathcal{X}\\times \\mathcal{Y}$ is an $\\epsilon$-game-stationary point {\\rm ($\\epsilon$-GS)} of problem \\eqref{eq:problem} if \n% there exists a pair $(u, v)$ such that\n\\[\n\\|\\nabla_{z} d_r(y,x)\\|\\leq\\epsilon \\quad \\text{and} \\quad {\\rm dist}(0,-\\nabla_{y} F(x, y)+\\partial\\iota_{\\mathcal{Y}}(y))\\leq\\epsilon. \\footnote{For any $x \\in \\mathbb{R}^{\\ell}$ and $\\mathcal{S} \\subseteq \\mathbb{R}^{\\ell}$, we use $\\dist(x, \\mathcal{S}):=\\inf\\limits_{z\\in\\mathcal{S}}\\|x-z\\|$ to denote the distance from $x$ to $\\mathcal{S}$.} \n% \\quad \\text { and } \\quad\\|u\\|,\\|v\\| \\leq \\epsilon.\n\\]\n\\end{enumerate}\n% where $1_{\\mathcal{Y}}$ is the indicator function of the set $\\mathcal{Y}$. \n% A pair $(x, y)$ is a stationary solution if $\\mu = 0, \\nu = 0$. \n\\end{definition}","sha256_utf8":"c81daf33b7ca14ae00947d607130e7a98055c91267f5f346f316e8fc269f67f3"}]},{"id":"S008","semantic_type":"algorithm","kind":"algorithm","title":"近端线性原始更新","source_units":[{"start":38861,"end":39045,"start_line":558,"end_line":563,"raw_tex":"\\begin{equation}\n    \\label{eq:primal_update}\n    x^{k+1} \\in \\mathop{\\argmin}_{x\\in\\mathcal{X}} \\left\\{ F_{x^k,\\lambda}(x,y^k)\n  +\\frac{r}{2}\\|x-z^{k}\\|^2\n    \\right\\},\n\\end{equation}","sha256_utf8":"4b468df5f339c31684935559aa35cef678d1c26cc09a6abd9b36646a68f518d4"},{"start":39113,"end":39244,"start_line":566,"end_line":568,"raw_tex":"\\[\nF_{x^k,\\lambda}(x,y^k):=h_{y^k}\\left(c_{y^k}(x^{k})+\\nabla c_{y^k}(x^{k})^\\top(x-x^{k})\\right)+\\frac{1}{2\\lambda}\\|x-x^k\\|^2.\n\\]","sha256_utf8":"8f956515a758592912d779a050d64678ad711e6f9107b934d20c8a1331a2d494"}]},{"id":"S009","semantic_type":"algorithm","kind":"algorithm","title":"平滑PLDA算法","source_units":[{"start":39652,"end":40152,"start_line":575,"end_line":583,"raw_tex":"\\begin{algorithm}[H]\\label{alg}\n    \\caption{\\textbf{Smoothed PLDA}}\n    \\SetKwInOut{Input}{Input}\n    \\Input{Initial point $(x^0,y^0,z^0)$ and parameters $r>L$, $\\lambda> 0$, $\\alpha>0$, $\\beta\\in(0,1)$}\n    \\For{$k=0,1,2,\\ldots$}{\n    $x^{k+1}:= \\mathop{\\argmin}\\limits_{x \\in \\mathcal{X}} \\left\\{F_{x^k,\\lambda}(x,y^k) + \\frac{r}{2}\\|x-z^k\\|^2\\right\\}$\\\\\n    $y^{k+1}:=\\proj_{\\mathcal{Y}}\\left(y^k+\\alpha \\nabla_{y} F(x^{k+1}, y^k)\\right)$ \\\\\n    $z^{k+1}:=z^k+\\beta(x^{k+1}-z^k)$}\n\\end{algorithm}","sha256_utf8":"3f8d4cd468c2539d16af3842b770925ff3f55896350e1a8031d28361f0371dcf"}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","title":"原始子问题精确求解","source_units":[{"start":40890,"end":41196,"start_line":588,"end_line":588,"raw_tex":"We emphasize that the optimal approach to solving \\eqref{eq:primal_update} depends on the problem's specific structure. For simplicity and to focus on our main contributions, we assume the exactness of the primal update and concentrate on the iteration complexity of the outer loop in subsequent analyses.\n","sha256_utf8":"299de7e0ec9f20164fc88daabe94a46579e0d40a641c348a1a729e2bc9a17ada"}]},{"id":"S011","semantic_type":"notation","kind":"notation","title":"正则解映射与对偶投影步","source_units":[{"start":42895,"end":44095,"start_line":614,"end_line":637,"raw_tex":"\\begin{table*}[ht]\n  \\caption{Notation}\n \\centering\n\\begin{tikzpicture}\n \\node[drop shadow,fill=white,inner sep=0pt] \n {\n \\resizebox{0.98\\textwidth}{!}{ \n\\begin{tabular}{|c|c|c|}\n \\hline\n \\textbf{Notation}  & \\textbf{Definition} & \\textbf{Notes} \\\\ \\hline\n $F_r(x,y,z)$   & $F(x,y)+\\frac{r}{2}\\|x-z\\|^2$ & potential function               \\\\ \\hline\n$d_r(y,z)$           & $\\min\\limits_{x \\in \\mathcal{X}}F_r(x,y,z)$ &    dual potential value function      \\\\ \\hline\n$p_r(z)$           & $\\max\\limits_{y \\in \\mathcal{Y}}\\min\\limits_{x \\in \\mathcal{X}}F_r(x,y,z)$ &    proximal function        \\\\ \\hline\n%  \\rowcolor[HTML]{ECF4FF} \n$x_r(y,z)$  & $\\mathop{\\argmin}\\limits_{x \\in \\mathcal{X}}F_r(x,y,z)$  & ---\\\\ \\hline\n$x^\\star_r(z)$   & $\\mathop{\\argmin}\\limits_{x \\in \\mathcal{X}}\\max\\limits_{y \\in \\mathcal{Y}}F_r(x,y,z)$ & ---               \\\\ \\hline\n$Y(z)$     & $\\mathop{\\argmax}\\limits_{y\\in \\mathcal{Y}}d_r(y,z)$ &   $y(z) \\in Y(z)$  \\\\ \\hline\n%  \\rowcolor[HTML]{ECF4FF} \n$y_{+}(z)$    & $\\proj_{\\mathcal{Y}}\\left(y+\\alpha \\nabla_{y} F(x_r(y, z), y)\\right)$  & one-step projected gradient ascent on the dual function\\\\ \\hline\n\\end{tabular}}\n};\n\\end{tikzpicture}\n\\label{tab:notation}\n\\end{table*}","sha256_utf8":"16d7b502bb3550827bb97e0ff518f3b12f2cf92fcc2da9430d0172478f415ce6"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"李雅普诺夫函数","source_units":[{"start":44475,"end":44674,"start_line":646,"end_line":648,"raw_tex":"\\[\n\\Phi_r(x,y,z):= \\underbrace{F_r(x,y,z)-d_r(y,z)}_{\\textnormal{Primal Descent}} + \\underbrace{p_r(z)-d_r(y,z)}_{\\textnormal{Dual Ascent}} + \\underbrace{p_r(z)}_{\\textnormal{Proximal Descent}}. \n\\]\n","sha256_utf8":"9628e85a77c62ae5dc2d224d50815d44a5ec406e91622d71b3c1086fa85167b4"}]},{"id":"S013","semantic_type":"notation","kind":"notation","title":"对偶误差增长幂","source_units":[{"start":48694,"end":48858,"start_line":693,"end_line":696,"raw_tex":"\\begin{equation}\n\\label{eq:step_dual}\n\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\| \\leq \\omega \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^\\upsilon\n\\end{equation}","sha256_utf8":"455ccc81ada05f6a918584d0401d99e8489a5316bff55c7f68c303efc415ad22"},{"start":48859,"end":48908,"start_line":697,"end_line":697,"raw_tex":"for some constants $\\omega>0$ and $\\upsilon>0$. \n","sha256_utf8":"6c2e3961090c52189d87e0229cb4b9100f3ea726c2ffcd48fbfa6e3a41e6fa41"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"弱凸性","source_units":[{"start":134121,"end":134677,"start_line":1725,"end_line":1733,"raw_tex":"\\begin{definition}[Weak convexity]\n\\label{defi:weak-convex}\n A function $\\ell:\\R^n \\rightarrow \\R$ is said to be $\\rho$-weakly convex on a set $\\mathcal{X}\\subseteq \\R^n$ for some constant $\\rho \\ge 0$ if for any $x, y \\in \\mathcal{X}$ and $\\tau \\in [0,1]$, we have\n$$\n\\ell(\\tau x+(1-\\tau) y) \\leq \\tau \\ell(x)+(1-\\tau) \\ell(y)+\\frac{\\rho \\tau(1-\\tau)}{2}\\|x-y\\|^{2}.\n$$\n% When $\\ell$ is locally Lipschitz, \nThe above definition is equivalent to the convexity of the function $\\ell(\\cdot) + \\tfrac{\\rho}{2} \\| \\cdot \\|^2$ on $\\mathcal{X}$.\n\\end{definition}","sha256_utf8":"956648c4626d0065f1d71763ecbcf583023121c866b4888f4b844139ee9c1f32"}]},{"id":"S015","semantic_type":"definition","kind":"definition","title":"近端映射","source_units":[{"start":134678,"end":135137,"start_line":1734,"end_line":1743,"raw_tex":"\\begin{definition}[Proximal mapping]\n% Consider a $\\rho$-weakly convex function \nLet $\\ell:\\R^n\\rightarrow \\R \\cup \\{+\\infty\\}$ \n% and fix a parameter $\\mu < \\rho^{-1}$\nbe a proper lower-semicontinuous function.  The proximal mapping of $\\ell$ with parameter $\\mu>0$ at the point $x \\in \\mathbb{R}^n$ is defined by\n\\[\n\\prox_{\\mu \\ell}(x) = \\mathop{\\argmin}_{y\\in \\R^n} \\left\\{ \\ell(y) + \\frac{1}{2\\mu} \\|y-x\\|^2\\right\\}. \n\\]\n\\label{defi:prox}\n\\end{definition}","sha256_utf8":"aaa7e46475b4866439c6580347dce9315e15bee47a1ec175542c3d52d2f498b0"}]},{"id":"S016","semantic_type":"derived_fact","kind":"derived_fact","title":"正则势函数强凸性","source_units":[{"start":135706,"end":135886,"start_line":1754,"end_line":1754,"raw_tex":"Using Definition \\ref{defi:weak-convex} and Fact \\ref{fact-1}, we can deduce that $F_r(\\cdot,y,z)$ is $(r-L)$-strongly convex for any $y \\in \\mathcal{Y}$ and $z \\in \\mathbb{R}^n$.\n","sha256_utf8":"f6b90b9bb8b505a28f614ee750e9c942faffc3cc3109b17b39c64a568c37f583"}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","title":"凹对偶误差界适用条件","source_units":[{"start":163231,"end":163414,"start_line":2202,"end_line":2202,"raw_tex":"The following lemma provides a dual error bound for the case where $F(x,\\cdot)$ is concave for any $x \\in \\mathbb{R}^n$. Its proof is based on  Lemma B.10 in \\citep{zhang2020single}.\n","sha256_utf8":"6ba7de2b8652f710587cca8db48accb1704676af90c3b5eb0e39517f2ebd555a"}]},{"id":"S018","semantic_type":"derived_fact","kind":"derived_fact","title":"原始函数和值函数弱凸性","source_units":[{"start":135327,"end":135466,"start_line":1745,"end_line":1747,"raw_tex":"\\begin{fact}\\label{fact-1}\nThe functions $F(\\cdot, y)$ for any $y\\in\\mathcal{Y}$ and $f$ are $L$-weakly convex on $\\mathcal{X}$.\n\\end{fact}","sha256_utf8":"407b2e0012dbf88272415edada15a73c8fa7d4d0695e21e1e942e4b7fec9c48e"}]},{"id":"S019","semantic_type":"derived_fact","kind":"derived_fact","title":"近端线性模型误差","source_units":[{"start":135467,"end":135705,"start_line":1748,"end_line":1753,"raw_tex":"\\begin{fact}\\label{fact-2}\nLet $y\\in\\R^d$ and $r>0$ be given. For all $x, \\bar{x} \\in\\mathcal{X}$, we have\n\\[\n-\\frac{r^{-1}+L}{2}\\|x-\\bar{x}\\|^{2} \\leq F(x,y)-F_{\\bar{x},r}(x, y) \\leq \\frac{-r^{-1} + L}{2}\\|x-\\bar{x}\\|^{2} .\n\\]\n\\end{fact}","sha256_utf8":"2b89c576f272a98ea11cbc84f525b1053b0f4fe7f41e69774a876c6ae4ef9827"}]},{"id":"S020","semantic_type":"proposition","kind":"proposition","title":"原始变量误差界","source_units":[{"start":51418,"end":51862,"start_line":731,"end_line":740,"raw_tex":"\\begin{proposition} [Lipschitz-type primal error bound]\n\\label{prop:lip}\n%  Suppose that $r-L > 0$. Then \n For any $k\\ge0$, we have\n    \\begin{equation}\\label{iter-result1}\n        \\|x^{k+1}-x_r(y^{k}, z^{k})\\| \\leq \\zeta\\|x^{k}-x^{k+1}\\|,\n        \\end{equation}\nwhere   $\\zeta: =\\frac{2(r-L)^{-1}+(\\lambda^{-1}+L) ^{-1}}{(\\lambda^{-1}+L) ^{-1}} \\left(\\sqrt{\\frac{2L}{\\lambda^{-1}+L}}+1\\right)$.\n% and $\\eta:=\\alpha L \\zeta$. \n\\end{proposition}","sha256_utf8":"9af2994d7f47588252dd8a5fa56e2740c866e3a1a2d6da2caca70e95959059f2"}]},{"id":"S021","semantic_type":"proof","kind":"proof","title":"原始误差界完整证明","source_units":[{"start":51863,"end":57479,"start_line":741,"end_line":810,"raw_tex":"\\begin{proof}\nFor ease of notation, we define $\\hat{F}_{y,z}(\\cdot): = F_r(\\cdot,y,z) + \\iota_{\\mathcal{X}}(\\cdot)$ for any $y\\in\\mathcal{Y}$ and $z\\in\\R^n$.  As $x_r(y,z)$ is the optimal solution of $\\min_{x \\in \\mathbb{R}^n} \\hat{F}_{y,z} (x)$ and $F_r(\\cdot,y,z)$ is $(r-L)$-strongly convex (see the remark after Fact \\ref{fact-2} in Appendix \\ref{sec:lemmas}), we have\n\\begin{equation}\n    \\label{eq:lip_1} \n    \\hat{F}_{y^k,z^k}(x)-\\hat{F}_{y^k,z^k}(x_r(y^k,z^k)) \\ge \\frac{r-L}{2} \\|x-x_r(y^k,z^k)\\|^2 \\quad \\text{for all}\\ x \\in \\mathcal{X}.\n\\end{equation}\nIn addition, by the convexity of $\\hat{F}_{y^k,z^k}$, we obtain \n\\begin{equation}\n    \\label{eq:lip_2}\n    \\begin{aligned}\n    \\hat{F}_{y^k,z^k}(x)-\\hat{F}_{y^k,z^k}(x_r(y^k,z^k)) & \\leq \\dist(0, \\partial \\hat{F}_{y^k,z^k}(x))\\cdot \\|x-x_r(y^k,z^k)\\| \\\\ \n    \\end{aligned}\n\\end{equation}\n$\\text{for all}\\ x\\in\\mathcal{X}$.\nCombining \\eqref{eq:lip_1} and \\eqref{eq:lip_2} yields\n\\[\n\\|x-x_r(y^k,z^k)\\| \\leq \\frac{2}{r-L} \\dist(0, \\partial \\hat{F}_{y^k,z^k}(x)) \\quad \\text{for all}\\ x\\in\\mathcal{X}. \n\\]\nBy utilizing the equivalence between proximal and subdifferential error bounds \\citep[Theorem 3.4]{drusvyatskiy2018error}, we  conclude that\n\\begin{equation}\n\\label{eq:;lip_3}\n    \\|x-x_r(y^k,z^k)\\|  \\leq \\frac{2(r-L)^{-1}+t}{t} \\left\\|x - \\prox_{t  \\hat{F}_{y^k,z^k}}(x)\\right\\| \\text{ for all } t>0, \\, x\\in\\mathcal{X}.\n\\end{equation}\nNow, let us examine the relationship between  $\\|x^{k+1}-x^k\\|$ and $\\|x^{k+1} - \\prox_{t \\hat{F}_{y^k,z^k}}(x^{k+1})\\|$. Let $\\varphi_k:\\R^n\\rightarrow \\R \\cup \\{+\\infty\\}$ be the function defined by\n\\[\n\\varphi_k(x):=\\hat{F}_{y^k,z^k}(x)+\\frac{\\lambda^{-1}+L}{2}\\|x-x^k\\|^{2}-\\frac{\\lambda^{-1} + L}{2}\\|x-x^{k+1}\\|^{2}.\n\\]\n% By combining Fact \\ref{fact-2} and the strong convexity of $F_{x^k,\\lambda}(\\cdot,y^k)$\nIt is clear that for any $x\\in\\mathcal{X}$,\n\\begin{align*}\\label{safeguard-key1}\n            & \\hat{F}_{y^k,z^k}(x)\n             = F(x,y^k) + \\frac{r}{2}\\|x-z^k\\|^2 \\notag\\\\\n             \\ge\\ & F_{x^k,\\lambda}(x,y^k) -\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k}\\|^2 + \\frac{r}{2}\\|x-z^k\\|^2\\notag \\\\ \n              \\ge\\ &  F_{x^k,\\lambda}(x^{k+1},y^k) +\\frac{\\lambda^{-1}+r}{2}\\|x-x^{k+1}\\|^2 -\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k}\\|^2 +\\frac{r}{2}\\|x^{k+1}-z^k\\|^2\\notag\\\\\n             \\ge\\ & F(x^{k+1},y^k) + \\frac{\\lambda^{-1}-L}{2}\\|x^{k}-x^{k+1}\\|^2 +\\frac{\\lambda^{-1}+r}{2}\\|x-x^{k+1}\\|^2\\ \\\\\n             & \\ -\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k}\\|^2 + \\frac{r}{2}\\|x^{k+1}-z^k\\|^2 \\notag \\\\ \n             =\\ & \\hat{F}_{y^k,z^k}(x^{k+1})+\\frac{\\lambda^{-1}-L}{2}\\|x^{k+1}-x^k\\|^2 +\\frac{\\lambda^{-1}+r}{2}\\|x-x^{k+1}\\|^2 -\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k}\\|^2,\n\\end{align*}\nwhere the first and third inequalities follow from Fact \\ref{fact-2}, and the second one is from the $(\\lambda^{-1} + r)$-strong convexity of $F_{x^k,\\lambda}(\\cdot,y^k) +\\frac{r}{2}\\|\\cdot-z^k\\|^2$ on $\\mathcal{X}$ and the definition of $x^{k+1}$.\nSince $r > L$, we see that for any $x\\in\\mathcal{X}$,\n\\begin{align*}\n\\varphi_k(x)\n&\\ge \\hat{F}_{y^k,z^k}(x^{k+1})+\\frac{\\lambda^{-1}-L}{2}\\|x^{k+1}-x^{k}\\|^2+\\frac{r-L}{2}\\|x-x^{k+1}\\|^2\\notag\\\\\n&\\ge\\hat{F}_{y^k,z^k}(x^{k+1})+\\frac{\\lambda^{-1}-L}{2}\\|x^{k+1}-x^{k}\\|^2.\n\\end{align*}\nIt then follows from the definition of $\\varphi_k$ that \n\\[\\varphi_k(x^{k+1})-\\inf_{x\\in\\R^n} \\varphi_k(x)\\leq L \\|x^{k+1}-x^k\\|^2.\\]\nThis, together with the definition of the proximal mapping, implies that for any $\\rho>0$,\n\\begin{align}\n&\\varphi_k\\left(\\operatorname{prox}_{\\frac{1}{\\rho} \\varphi_k}(x^{k+1})\\right)+\\frac{\\rho}{2}\\|\\operatorname{prox}_{\\frac{1}{\\rho} \\varphi_k}(x^{k+1})-x^{k+1}\\|^{2}\\leq  \\varphi_k(x^{k+1})\\notag\\\\\n\\leq\\ & \\inf_{x\\in\\R^n} \\varphi_k(x)+L \\|x^{k+1}-x^k\\|^2\n\\leq \\varphi_k\\left(\\operatorname{prox}_{\\frac{1}{\\rho} \\varphi_k}(x^{k+1})\\right)+L \\|x^{k+1}-x^k\\|^2,\\notag\n\\end{align}\nwhich in turn implies that\n\\begin{equation}\\label{safeguard-key111}\n\\|\\operatorname{prox}_{\\frac{1}{\\rho} \\varphi_k}(x^{k+1})-x^{k+1}\\|\\leq \\sqrt{\\frac{2L}{\\rho}} \\|x^{k+1}-x^k\\|.    \n\\end{equation}\nRecall that our goal is to find a relationship between  $\\|x^{k+1}-x^k\\|$ and $\\|x^{k+1} - \\prox_{t \\hat{F}_{y^k,z^k}}(x^{k+1})\\|$. To do this, it suffices to relate $\\prox_{t \\hat{F}_{y^k,z^k}}(x^{k+1})$ to $\\operatorname{prox}_{\\frac{1}{\\rho} \\varphi_k}(x^{k+1})$.  By definition, we have\n\\[\\hat{F}_{y^k,z^k}(x)+\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k}\\|^{2}=\\varphi_k(x)+\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k+1}\\|^{2},\\]\nwhich implies that $\\operatorname{prox}_{\\frac{1}{\\lambda^{-1}+L} \\varphi_k}(x^{k+1})=\\operatorname{prox}_{\\frac{1 }{\\lambda^{-1}+L}\\hat{F}_{y^k,z^k}}(x^k)$. \nBy setting $t = (\\lambda^{-1} + L)^{-1}$ in \\eqref{eq:;lip_3} and $\\rho = \\lambda^{-1} + L$ in \\eqref{safeguard-key111}, we conclude that\n\\begin{align*}\n    &\\|x^{k+1}-x_r(y^{k}, z^{k})\\| \\notag\\\\\n    \\leq\\ & \\frac{2(r-L)^{-1}+(\\lambda^{-1}+L) ^{-1}}{(\\lambda^{-1}+L) ^{-1}}\\left\\|x^{k+1}- \\prox_{\\frac{1}{\\lambda^{-1}+L} \\hat{F}_{y^k,z^k}}(x^{k+1})\\right\\| \\\\\n    \\leq\\ &   \\frac{2(r-L)^{-1}+(\\lambda^{-1}+L) ^{-1}}{(\\lambda^{-1}+L) ^{-1}}\\left(\\left\\|x^{k+1}- \\prox_{\\frac{1}{\\lambda^{-1}+L} \\hat{F}_{y^k,z^k}}(x^{k})\\right\\|+ \\|x^{k+1}-x^k\\|\\right) \\\\\n     =\\ &   \\frac{2(r-L)^{-1}+(\\lambda^{-1}+L) ^{-1}}{(\\lambda^{-1}+L) ^{-1}}\\left(\\left\\|x^{k+1}- \\prox_{\\frac{1}{\\lambda^{-1}+L} \\varphi_k}(x^{k+1})\\right\\|+ \\|x^{k+1}-x^k\\|\\right) \\\\\n    \\leq\\ & \\frac{2(r-L)^{-1}+(\\lambda^{-1}+L) ^{-1}}{(\\lambda^{-1}+L) ^{-1}} \\left(\\sqrt{\\frac{2L}{\\lambda^{-1}+L}}+1\\right) \\|x^{k+1}-x^k\\|,\n\\end{align*}\nwhere the second inequality is due to the nonexpansiveness of the proximal mapping. The proof is complete.\n\\end{proof}","sha256_utf8":"a6423bde18fe6e8ef63f4e8bbc2615ca03fa20925b9ba370949d8808e76c22e5"}]},{"id":"S022","semantic_type":"lemma","kind":"lemma","title":"正则解映射的利普希茨性","source_units":[{"start":136469,"end":136991,"start_line":1763,"end_line":1771,"raw_tex":"\\begin{lemma}\\label{lemma-sollip}\nFor any $y, y^{\\prime} \\in \\mathcal{Y}$ and $z, z^{\\prime} \\in \\R^n$, we have \n    \\begin{align}\n        &\\|x_r(y, z)-x_r(y, z^{\\prime})\\| \\leq \\sigma_{1}\\|z-z^{\\prime}\\|, \\label{lip-z}\\\\\n        &\\|x_r^\\star(z)-x_r^\\star(z^{\\prime})\\| \\leq \\sigma_{1}\\|z-z^{\\prime}\\|, \\label{lip-starz}\\\\\n        &\\|x_r(y, z)-x_r(y^{\\prime}, z)\\| \\leq \\sigma_{2}\\|y-y^{\\prime}\\|, \\label{lip-y}\n        \\end{align}\n     where $\\sigma_{1}:=\\frac{r}{r-L}$ and $\\sigma_{2}:=\\frac{2r-L}{r-L}$.\n    \\end{lemma}","sha256_utf8":"63d9a21b3c8aa3c0176617e5351915d9e845af79e5e07cb6f5f8e21a9a4dd8dd"}]},{"id":"S023","semantic_type":"proof","kind":"proof","title":"解映射连续性完整证明","source_units":[{"start":136993,"end":141766,"start_line":1773,"end_line":1854,"raw_tex":"\\begin{proof}\nFrom the definition of $F_r$, we know that for any $x\\in\\mathcal{X}$, $y\\in\\mathcal{Y}$, and $z,z'\\in\\R^n$, \n\\begin{equation}\\label{func-cal}\nF_r(x,y,z^{\\prime})-F_r(x,y,z)=\\frac{r}{2}(\\|x-z'\\|^2-\\|x-z\\|^2)=\\frac{r}{2}(\\|z^{\\prime}\\|^{2}-2(z^{\\prime}-z)^\\top x-\\|z\\|^{2}).\n\\end{equation}\nFact \\ref{fact-1} implies that $F(\\cdot, y)$ is $L$-weakly convex for any $y \\in \\mathcal{Y}$. Therefore, for any $x\\in\\mathcal{X}$, $y\\in\\mathcal{Y}$, and $z\\in\\R^n$, we have\n\\begin{equation}\\label{psi-strongcvx}\nF_r(x,y, z)- F_r(x_r(y,z),y,z)\n\\geq \\frac{r-L}{2}\\|x-x_r(y,z)\\|^{2}.\n\\end{equation}\nCombining \\eqref{func-cal} and \\eqref{psi-strongcvx} yields\n\\begin{align}\\label{lip-key1}\n    &F_r(x_r(y,z),y,z^{\\prime})-F_r(x_r(y,z^{\\prime}),y,z^{\\prime}) \\notag\\\\\n    =\\ &F_r(x_r(y,z),y,z^{\\prime})-F_r(x_r(y,z),y,z)+F_r(x_r(y,z),y,z)-F_r(x_r(y,z^{\\prime}),y,z)\\, \\notag\\\\\n    &-(F_r(x_r(y,z^{\\prime}),y, z^{\\prime})-F_r(x_r(y,z^{\\prime}),y, z)) \\notag\\\\\n    \\leq\\ &\\frac{r}{2}(\\|z^{\\prime}\\|^{2}-2(z^{\\prime}-z)^\\top x_r(y,z)-\\|z\\|^{2})-\\frac{r-L}{2}\\|x_r(y,z)-x_r(y,z^{\\prime})\\|^{2}\\,  \\notag\\\\\n    &-\\frac{r}{2}(\\|z^{\\prime}\\|^{2}-2(z^{\\prime}-z)^\\top x_r(y,z^{\\prime})-\\|z\\|^{2}) \\notag\\\\\n    \\leq\\ &r(z^{\\prime}-z)^\\top(x_r(y,z^{\\prime})-x_r(y,z))-\\frac{r-L}{2}\\|x_r(y,z)-x_r(y,z^{\\prime})\\|^{2}.\n    \\end{align}\n On the other hand, \\eqref{psi-strongcvx} implies that\n    \\[\n    F_r(x_r(y,z),y, z^{\\prime})-F_r(x_r(y,z^{\\prime}),y,z^{\\prime}) \\geq \\frac{r-L}{2}\\|x_r(y,z)-x_r(y,z^{\\prime})\\|^{2}.\n    \\]\nCombining this inequality with \\eqref{lip-key1}, we get\n    \\[\n    (r-L)\\|x_r(y,z)-x_r(y,z^{\\prime})\\|^{2}\\leq r(z^{\\prime}-z)^\\top (x_r(y,z^{\\prime})-x_r(y,z)).\n    \\]\nUsing the Cauchy-Schwarz inequality, we obtain\n    \\[\n    \\|x_r(y,z)-x_r(y,z^{\\prime})\\| \\leq \\frac{r}{r-L}\\|z-z^{\\prime}\\|.\n    \\]\nTherefore, we conclude that \\eqref{lip-z} holds with $\\sigma_1=\\frac{r}{r-L}$. Since $f=\\max_{y \\in \\mathcal{Y}} F(\\cdot,y)$ is also $L$-weakly convex by  Fact \\ref{fact-1}, we can prove that \\eqref{lip-starz} holds by using a similar argument as above.\n    \n\nWe  now proceed to prove \\eqref{lip-y}. Using \\eqref{psi-strongcvx}, we have\n\\begin{align}\n&F_r(x_r(y^{\\prime}, z),y, z)-F_r(x_r(y, z), y,z) \\geq \\frac{r-L}{2}\\|x_r(y, z)-x_r(y^{\\prime}, z)\\|^{2},\\label{psi-strongcvxineq1} \\\\\n&F_r(x_r(y, z), y^{\\prime},z)-F_r(x_r(y^{\\prime}, z), y^{\\prime},z) \\geq \\frac{r-L}{2}\\|x_r(y, z)-x_r(y^{\\prime}, z)\\|^{2}\\label{psi-strongcvxineq2}.\n\\end{align}\nMoreover, by the $L$-Lipschitz continuity of $\\nabla_{y} F_r(x, \\cdot, z)$ for any $x \\in \\mathcal{X}$ and $z \\in \\mathbb{R}^n$, we have\n\\begin{equation}\n\\begin{aligned}\\label{psi-yconcave}\n F_r(x_r(y, z), y^{\\prime},z)-F_r(x_r(y, z), y,z)\\leq &\\langle\\nabla_{y} F_r(x_r(y, z),y,z), y^{\\prime}-y\\rangle +\\frac{L}{2}\\|y-y^{\\prime}\\|^{2}\n\\end{aligned}\n\\end{equation}\nand \n\\begin{equation}\n\\begin{aligned}\\label{psi-ylip}\n F_r(x_r(y^{\\prime}, z), y, z)-F_r(x_r(y^{\\prime}, z), y^{\\prime},z) \\leq & \\langle\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime}, z), y-y^{\\prime}\\rangle + \\frac{L}{2}\\|y-y^{\\prime}\\|^{2}.\n\\end{aligned}\n\\end{equation}\nIncorporating \\eqref{psi-strongcvxineq1}--\\eqref{psi-ylip}, we obtain\n\\begin{equation*}\n% \\label{lip-key2}\n\\begin{aligned}\n& (r-L)\\|x_r(y, z)-x_r(y^{\\prime}, z)\\|^{2}  \\\\ \\leq \\ &\\langle\\nabla_{y} F_r(x_r(y, z), y, z)-\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime}, z), y^{\\prime}-y\\rangle +L\\|y-y^{\\prime}\\|^{2}. \n\\end{aligned}\n\\end{equation*}\nSince $\\nabla_y F_r(\\cdot,\\cdot,z) = \\nabla_y F(\\cdot,\\cdot)$ is $L$-Lipschitz for any $z \\in \\mathbb{R}^n$, we have\n\\begin{align*}\n&\\|\\nabla_{y} F_r(x_r(y, z), y, z)-\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime}, z)\\| \\notag\\\\\n\\leq\\ &\\|\\nabla_{y} F_r(x_r(y, z), y, z)-\\nabla_{y}F_r(x_r(y, z), y^{\\prime}, z)\\|  +\\|\\nabla_{y}F_r(x_r(y, z), y^{\\prime}, z)-\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime}, z)\\|\\notag\\\\\n\\leq\\ &L(\\|y-y^{\\prime}\\|+\\|x_r(y^{\\prime}, z)-x_r(y, z)\\|).\\notag\n\\end{align*}\nIt follows that\n\\begin{align*}\n(r-L)\\|x_r(y, z)-x_r(y^{\\prime}, z)\\|^{2} \n\\leq L\\|x_r(y^{\\prime}, z)-x_r(y, z)\\|\\cdot\\|y-y^{\\prime}\\|+2L\\|y-y^{\\prime}\\|^{2}.\n\\end{align*}\nLet $\\zeta:=\\frac{\\|x_r(y, z)-x_r(y^{\\prime}, z)\\|}{\\|y-y^{\\prime}\\|}$. \nThen, the above inequality gives\n\\begin{align*}\n\\zeta^{2}\n& \\ \\leq \\frac{L}{r-L} \\zeta+\\frac{2L}{(r-L)}\\leq \\frac{1}{2}\\zeta^2+\\frac{L^2}{2(r-L)^2} +\\frac{2L}{(r-L)}\\\\\n& \\ \\leq \\frac{1}{2} \\zeta^{2}+\\frac{L^2+4L(r-L)}{2(r-L)^2}  \\leq \\frac{1}{2} \\zeta^{2}+\\frac{(L+2(r-L))^2}{2(r-L)^2},\n\\end{align*}\nwhere the second inequality holds because $ab\\leq \\frac{1}{2}(a^2+b^2)$ for any $a,b\\in\\R$. Thus, we get\n\\[\n\\|x_r(y, z)-x_r(y^{\\prime}, z)\\| \\leq  \\frac{2r-L}{r-L}\\|y-y^{\\prime}\\|,\n\\]\nwhich shows that \\eqref{lip-y} holds with $\\sigma_{2}=\\frac{2r-L}{r-L}$. The proof is complete.\n\\end{proof}","sha256_utf8":"975f9b5be0505a2866a2c738bb6c480dd48a8707f0beb360f2806bbea16ede80"}]},{"id":"S024","semantic_type":"lemma","kind":"lemma","title":"对偶势值函数可微性","source_units":[{"start":141767,"end":142760,"start_line":1855,"end_line":1867,"raw_tex":"\\begin{lemma}\\label{lemma-dualdiff} \n    The dual potential value function $d_r$ is differentiable on $\\mathcal{Y}\\times \\R^n$ with\n    \\begin{align*}\n    &\\nabla_{y} d_r(y,z)=\\nabla_{y} F(x_r(y,z),y),\\\\\n    &\\nabla_{z} d_r(y,z)=\\nabla_{z} F_r(x_r(y, z),y, z)=r(z-x_r(y,z))\n    \\end{align*}\n    for any $y \\in \\mathcal{Y}$ and $z \\in \\mathbb{R}^n$. Moreover, for any $y \\in \\mathcal{Y}$ and $z \\in \\mathbb{R}^n$,  the gradients $\\nabla_y d_r(\\cdot,z)$ and $\\nabla_z d_r(y,\\cdot)$ are Lipschitz continuous, i.e.,\n    \\begin{align}\n    &\\|\\nabla_{y} d_r(y^{\\prime},z)-\\nabla_{y} d_r(y^{\\prime \\prime},z)\\| \\leq L_{d_r}\\|y^{\\prime}-y^{\\prime \\prime}\\| \\quad \\text{for all}\\ y^{\\prime}, y^{\\prime \\prime} \\in \\mathcal{Y},\\notag\\\\\n    &\\|\\nabla_{z} d_r(y,z')-\\nabla_{z} d_r(y,z'')\\| \\leq L_{d_r}'\\|z^{\\prime}-z^{\\prime \\prime}\\| \\quad \\text{for all}\\ z^{\\prime}, z^{\\prime \\prime} \\in \\R^n,\\notag\n    \\end{align}\n    where $L_{d_r}:=(\\sigma_{2}+1)L$ and $L_{d_r}':=(\\sigma_1 + 1)r$.\n    \\end{lemma}","sha256_utf8":"8c9815666665db65e874f439da9e93422a15bb71279a19db729b47504015ee6f"}]},{"id":"S025","semantic_type":"proof","kind":"proof","title":"势值函数可微性完整证明","source_units":[{"start":142761,"end":144854,"start_line":1868,"end_line":1896,"raw_tex":"\\begin{proof}\nSince $F_r(\\cdot,y,z)$ is strongly convex, $F_r(x,\\cdot,z)$ is weakly concave, and $F_r(x,y,\\cdot)$ is strongly convex for any $x \\in \\mathcal{X}$, $y \\in \\mathcal{Y}$, and $z \\in \\mathbb{R}^n$, the function\n    \\[\n    d_r(\\cdot,\\cdot)=\\min_{x\\in\\mathcal{X}} F_r(x,\\cdot,\\cdot)\n    \\] \n    is differentiable on $\\mathcal{Y}\\times \\R^n$ due to \\citep[Theorem 10.31]{rockafellar2009variational}. In particular, for any $y\\in\\mathcal{Y}$ and $z\\in\\R^n$, one has\n    \\[\n    \\nabla_{y} d_r(y, z)=\\nabla_{y} F_r(x_r(y, z),y, z)=\\nabla_{y} F(x_r(y, z), y).\n    \\]\nUsing  \\citep[Lemma 4.3]{drusvyatskiy2019efficiency}, we have \n    \\[\n    \\nabla_{z} d_r(y,z)=\\nabla_{z} F_r(x_r(y, z),y, z)=r(z-\\prox_{\\frac{1}{r}F(\\cdot,y)+\\iota_{\\mathcal{X}}}(z))=r(z-x_r(y,z)).\n    \\]\n    It follows that for  any $y^{\\prime}, y^{\\prime \\prime} \\in \\mathcal{Y}$, \n    \\begin{align*}\n    &\\|\\nabla_{y} d_r(y^{\\prime}, z)-\\nabla_{y} d_r(y^{\\prime \\prime}, z)\\|\\notag\\\\\n    =\\ &\\|\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime},z)-\\nabla_{y} F_r(x_r(y^{\\prime \\prime}, z), y^{\\prime \\prime},z)\\| \\\\\n    \\leq\\ &\\|\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime},z)-\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime \\prime}, z)\\|+\\|\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime \\prime},z)-\\nabla_{y} F_r(x_r(y^{\\prime \\prime}, z), y^{\\prime \\prime}, z)\\| \\\\\n    \\leq\\ & L\\|y^{\\prime}-y^{\\prime \\prime}\\|+L\\|x_r(y^{\\prime}, z)-x_r(y^{\\prime \\prime}, z)\\|\\notag\\\\\n    \\leq\\ & L\\|y^{\\prime}-y^{\\prime \\prime}\\|+L \\sigma_{2}\\|y^{\\prime}-y^{\\prime \\prime}\\|\\leq L_{d_r}\\|y^{\\prime}-y^{\\prime \\prime}\\|,\n    \\end{align*}\n    where the third inequality is due to \\eqref{lip-y}. Moreover, we have\n    \\begin{align*}\n    \\|\\nabla_{z} d_r(y,z')-\\nabla_{z} d_r(y,z'')\\|=\\ &r\\|z'-x_r(y,z')-(z''-x_r(y,z''))\\| \\\\\n    \\leq\\ & r\\|z^{\\prime}-z^{\\prime \\prime}\\|+r\\|x_r(y, z')-x_r(y, z'')\\| \\\\\n    \\leq\\ & r\\|z^{\\prime}-z^{\\prime \\prime}\\|+ r\\sigma_{1}\\|z^{\\prime}-z^{\\prime \\prime}\\|\\leq L_{d_r}'\\|z^{\\prime}-z^{\\prime \\prime}\\|,\n    \\end{align*}\nwhere the second inequality is due to \\eqref{lip-z}. The proof is complete.\n\\end{proof}","sha256_utf8":"126d677cd571469d2a1a6edb7281def76af3e85e92e50d8ead8ad3c810e5bebc"}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","title":"实际与理想对偶步之差","source_units":[{"start":144856,"end":145102,"start_line":1898,"end_line":1904,"raw_tex":"\\begin{lemma}\\label{lemma-dualxy}\nFor any $k\\ge0$, we have\n    \\begin{equation*}\n        \\|y^{k+1}-y_{+}^{k}(z^{k+1})\\| \\leq \\eta \\|x^{k}-x^{k+1}\\|+\\sigma_1\\alpha L\\|z^{k+1}-z^k\\|,\n        \\end{equation*}\nwhere $\\eta:=\\alpha L \\zeta$.\n\\end{lemma}","sha256_utf8":"f034b5faa4b85e8150fd4ff2f646e940331dae5e1f46ab6d84b8f1472d84eb3e"}]},{"id":"S027","semantic_type":"proof","kind":"proof","title":"对偶步差完整证明","source_units":[{"start":145234,"end":146090,"start_line":1909,"end_line":1921,"raw_tex":"\\begin{proof}\nWe compute\n\\begin{equation*}\n% \\label{iter-result2}\n\\begin{aligned}\n\\ &\\|y^{k+1}-y_{+}^{k}(z^{k+1})\\|\\\\ =\\ &\\|\\proj_{\\mathcal{Y}}(y^{k}+\\alpha \\nabla_{y} F(x^{k+1},y^{k}))-\\proj_{\\mathcal{Y}}(y^{k}+\\alpha \\nabla_{y} F(x_r(y^{k}, z^{k+1}),y^{k}))\\|\\notag \\\\\n\\leq\\ &\\|y^{k}+\\alpha \\nabla_{y} F(x^{k+1}, y^{k})-(y^{k}+\\alpha \\nabla_{y} F(x_r(y^{k}, z^{k+1}),  y^{k}))\\|\\notag \\\\\n\\leq\\ & \\alpha L(\\|x^{k+1}-x_r(y^{k}, z^{k})\\|+\\|x_r(y^{k}, z^{k})-x_r(y^{k}, z^{k+1})\\|) \\notag \\\\\n\\leq\\ & \\eta\\|x^{k}-x^{k+1}\\|+\\sigma_1\\alpha L\\|z^{k+1}-z^k\\|,\n\\end{aligned}\n\\end{equation*}\nwhere the first inequality is due to the nonexpansiveness of the projection operator, the second is due to the $L$-Lipschitz continuity of $\\nabla_y F(\\cdot,\\cdot)$, and the third follows from Proposition \\ref{prop:lip} and \\eqref{lip-z}. The proof is complete.\n\\end{proof}","sha256_utf8":"3f8f9a6f994b1bb27efb63c5e2df29f3d22107674fe4603ce472235c53f9ee22"}]},{"id":"S028","semantic_type":"lemma","kind":"lemma","title":"原始势函数下降","source_units":[{"start":146177,"end":146553,"start_line":1924,"end_line":1932,"raw_tex":"\\begin{lemma}[Primal descent]\n\\label{lemma-F-des} \nFor any $k\\ge0$, we have\n\\begin{align*}\n& F_r(x^{k}, y^{k},z^k)-F_r(x^{k+1}, y^{k+1},z^{k+1})\\\\\n\\ge\\ &\n\\frac{2\\lambda^{-1}+r-L}{2}\\|x^k-x^{k+1}\\|^2 + \\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k}-y^{k+1}\\rangle-\\frac{L}{2}\\|y^k-y^{k+1}\\|^2 \\\\  &  +\\frac{(2-\\beta)r}{2 \\beta}\\|z^{k}-z^{k+1}\\|^{2}. \n\\end{align*}\n\\end{lemma}","sha256_utf8":"bb2b0ea13a14297b2ef77720110f89681fa273a2fbb408d9f6141d99e1ed706a"}]},{"id":"S029","semantic_type":"proof","kind":"proof","title":"原始下降完整证明","source_units":[{"start":146555,"end":148122,"start_line":1934,"end_line":1960,"raw_tex":"\\begin{proof}\nOne can infer from the definition that $F_{x^k,\\lambda}(\\cdot,y^k)$ is $\\lambda^{-1}$-strongly convex. Therefore, we have\n\\begin{equation*}\n\\begin{aligned}\nF_r(x^{k}, y^{k},z^{k}) & =F(x^{k}, y^{k})+ \\frac{r}{2}\\|x^k-z^{k}\\|^2  = F_{x^k,\\lambda}(x^k,y^k) + \\frac{r}{2}\\|x^k-z^{k}\\|^2 \\\\\n& \\ge F_{x^k,\\lambda}(x^{k+1},y^k)+\\frac{r}{2}\\|x^{k+1}-z^{k}\\|^2+\\frac{\\lambda^{-1}+r}{2}\\|x^k-x^{k+1}\\|^2.\n\\end{aligned}\n\\end{equation*}\nMoreover, Fact \\ref{fact-2} implies that\n\\begin{equation*} \nF_{x^k,\\lambda}(x^{k+1},y^k)\\ge F(x^{k+1},y^k) + \\frac{\\lambda^{-1}-L}{2}\\|x^{k+1}-x^k\\|^2.\n\\end{equation*}\nIt follows that\n\\begin{align}\\label{primaldec-key1}\nF_r(x^{k},y^{k}, z^k) &\\ge F(x^{k+1},y^k)+\\frac{r}{2}\\|x^{k+1}-z^k\\|^2+\\frac{2\\lambda^{-1}+r-L}{2}\\|x^k-x^{k+1}\\|^2\\notag\\\\\n&= F_r(x^{k+1},y^{k}, z^{k})+\\frac{2\\lambda^{-1}+r-L}{2}\\|x^k-x^{k+1}\\|^2.\n\\end{align}\nNext,  as $\\nabla_y F_r(x,\\cdot,z)$ is $L$-Lipschitz continuous for any $x\\in\\mathcal{X}$ and $z\\in\\R^n$, we have \n    \\begin{align}\\label{primaldec-key2}\n        F_r(x^{k+1}, y^{k},z^k)-F_r(x^{k+1}, y^{k+1},z^k) \\ge \\langle\\nabla_{y} F_r(x^{k+1},y^{k}, z^k), y^{k}-y^{k+1}\\rangle-\\frac{L}{2}\\|y^k-y^{k+1}\\|^2.\n    \\end{align}\nAt last, based on the update $z^{k+1}=z^{k}+\\beta(x^{k+1}-z^{k})$, we get\n\\begin{equation}\\label{primaldec-key3}\nF_r(x^{k+1}, y^{k+1},z^{k})-F_r(x^{k+1},y^{k+1}, z^{k+1})  = \\frac{(2-\\beta)r}{2 \\beta}\\|z^{k}-z^{k+1}\\|^{2}.\n\\end{equation}\nBy summing up \\eqref{primaldec-key1}, \\eqref{primaldec-key2}, and \\eqref{primaldec-key3}, we obtain the desired result.\n\\end{proof}","sha256_utf8":"7cdf51f6f2028d1f2e0065955eef43aa7c22d81a989c663163a619836980fa02"}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","title":"对偶势值函数上升","source_units":[{"start":148124,"end":148572,"start_line":1962,"end_line":1972,"raw_tex":"\\begin{lemma}[Dual ascent]\\label{lemma-d-aes} \nFor any $k\\ge0$, we have\n\\begin{equation*}\n% \\label{dual-ascent}\n \\begin{aligned}\n    \\ & d_r(y^{k+1},z^{k+1})-d_r(y^{k},z^k)\\\\\n    \\ge\\ &\\langle\\nabla_{y} F_r(x_r(y^{k}, z^{k}),  y^{k}, z^{k}), y^{k+1}-y^{k}\\rangle-\\frac{L_{d_r}}{2}\\|y^{k}-y^{k+1}\\|^{2}\\,\\\\\n    &+\\frac{r}{2}\\left\\langle z^{k+1}-z^{k},z^{k+1}+z^{k}-2 x_r(y^{k+1}, z^{k+1})\\right\\rangle. \n    \\end{aligned}\n\\end{equation*}\n\\end{lemma}","sha256_utf8":"4810d4fb308dd628cdca84176c306e2341c98226a404391e5ad015b3466f88c1"}]},{"id":"S031","semantic_type":"proof","kind":"proof","title":"对偶上升完整证明","source_units":[{"start":148573,"end":149689,"start_line":1973,"end_line":1989,"raw_tex":"\\begin{proof}\nBased on Lemma \\ref{lemma-dualdiff}, $\\nabla_{y} d_r(\\cdot,z)$ is $L_{d_r}$-Lipschitz continuous for any $z \\in \\mathbb{R}^n$. Thus, we have \n        \\begin{align*}\n        \\ & d_r(y^{k+1},z^k)-d_r(y^{k},z^k)  \n        \\geq \\langle\\nabla_{y} d_r(y^{k},z^k), y^{k+1}-y^{k}\\rangle-\\frac{L_{d_r}}{2}\\|y^{k}-y^{k+1}\\|^{2} \\\\\n        =\\ & \\langle \\nabla_{y} F_r(x_r(y^{k}, z^{k}), y^{k}, z^{k}), y^{k+1}-y^{k}\\rangle-\\frac{L_{d_r}}{2}\\|y^{k}-y^{k+1}\\|^{2}.\n        \\end{align*}\nIn addition, one has \n        \\begin{align*}\n        &d_r(y^{k+1}, z^{k+1})-d_r(y^{k+1}, z^{k})\\notag\\\\ \n        =\\ & F_r(x_r(y^{k+1}, z^{k+1}), y^{k+1}, z^{k+1})-F_r(x_r(y^{k+1}, z^{k}), y^{k+1},z^{k}) \\\\\n        \\geq\\ &  F_r(x_r(y^{k+1}, z^{k+1}), y^{k+1},z^{k+1})-F_r(x_r(y^{k+1}, z^{k+1}), y^{k+1},z^{k}) \\\\\n        =\\ & \\frac{r}{2}\\|x_r(y^{k+1}, z^{k+1})-z^{k+1}\\|^{2}-\\frac{r}{2}\\|x_r(y^{k+1}, z^{k+1})-z^{k}\\|^{2} \\\\\n        =\\ & \\frac{r}{2} \\left\\langle z^{k+1}-z^{k},z^{k+1}+z^{k}-2 x_r(y^{k+1}, z^{k+1})\\right\\rangle.\n        \\end{align*}\nFinally, by combining the above inequalities, the proof is complete.\n\\end{proof}","sha256_utf8":"1fca2c302b6635dfb08a57fe9e91583ef2cdd4ffd23d9b7ed5e3a1487861df53"}]},{"id":"S032","semantic_type":"lemma","kind":"lemma","title":"近端值函数下降","source_units":[{"start":149692,"end":150033,"start_line":1991,"end_line":1999,"raw_tex":"\\begin{lemma}[Proximal descent (smoothness)]\n\\label{lemma-p-des} \n    For any $k\\ge0$, we have\n   \\begin{equation*}\n   % \\label{proximal-descent}\n    p_r(z^k)-p_r(z^{k+1}) \\ge \\frac{r}{2}\\left\\langle z^{k+1}-z^{k},2 x_r(y(z^{k+1}), z^{k})-z^{k}-z^{k+1}\\right\\rangle, \n    \\end{equation*}\n    where $y(z^{k+1})\\in Y(z^{k+1})$.\n    \\end{lemma}","sha256_utf8":"9c6a3944d717fde029af8be7da5ca2165489e0edff20206d5bdf932bee919fde"}]},{"id":"S033","semantic_type":"proof","kind":"proof","title":"近端下降完整证明","source_units":[{"start":150034,"end":150966,"start_line":2000,"end_line":2019,"raw_tex":"\\begin{proof}\n% From Sion's minimax theorem \\citep{sion1958general}, we know that\n%     \\[\n%     \\min _{x \\in \\mathcal{X}} \\max _{y \\in \\mathcal{Y}} F_r(x, y, z)=\\max_{y \\in \\mathcal{Y}} \\min_{x \\in \\mathcal{X}} F_r(x,y,z)\n%     \\]\n%     which implies\nRecall that \n% $p_r(z)$:\n    % \\[\n    $p_r(z)=\\max _{y \\in \\mathcal{Y}} d_r(y, z)$.\n    % \\]\nDue to the definition of $y(z^{k+1})$, we have\n    \\begin{align*}\n     &p_r(z^{k+1})-p_r(z^{k})\\notag\\\\\n    \\leq\\ & d_r(y(z^{k+1}), z^{k+1})-d_r(y(z^{k+1}), z^{k}) \\\\\n    \\leq\\ & F_r(x_r(y(z^{k+1}), z^{k}), y(z^{k+1}), z^{k+1})-F_r(x_r(y(z^{k+1}), z^{k}), y(z^{k+1}),z^{k}) \\\\\n    =\\ & \\frac{r}{2}\\left\\langle z^{k+1}-z^{k},z^{k+1}+z^{k}-2 x_r(y(z^{k+1}), z^{k})\\right\\rangle,\n    \\end{align*}\n    where the second inequality follows from the fact that \\[F_r(x', y, z)\\geq \\min_{x\\in\\mathcal{X}} F_r(x,y,z) = d_r(y, z)\\] for any $x'\\in\\mathcal{X}$. The proof is complete.\n    \\end{proof}","sha256_utf8":"70489bb26fa0f3126d0a04bc185df6656040df690cbc783423cd775558ea9059"}]},{"id":"S034","semantic_type":"proposition","kind":"proposition","title":"李雅普诺夫基本下降估计","source_units":[{"start":46206,"end":47355,"start_line":661,"end_line":676,"raw_tex":"\\begin{proposition}[Basic descent estimate of $\\Phi_r$]\n\\label{prop:decrease}\n    Let \n    % $r\\ge3L$, $\\lambda\\le \\frac{1}{L}$, $\\alpha \\leq \\min\\left\\{\\frac{1}{10L}, \\frac{1}{4L\\zeta^2} \\right\\}$, and $\\beta  \\leq \\min \\left\\{\\frac{1}{28}, \\frac{(r-L)^2}{14\\alpha r(2r-L)^2}\\right\\}$,     \n    \\[r\\ge3L,\\ \\lambda\\le \\frac{1}{L},\\ \\alpha \\leq \\min\\left\\{\\frac{1}{10L},\\ \\frac{1}{4L\\zeta^2} \\right\\},\\ \\beta  \\leq \\min \\left\\{\\frac{1}{28},\\ \\frac{(r-L)^2}{14\\alpha r(2r-L)^2}\\right\\},\\] \n    where $\\zeta>0$ is the constant defined in Proposition \\ref{prop:lip}.\n    Then, for any $k\\ge0$, we have\n\\begin{equation*}\n\\begin{aligned}\n\\Phi_r^k-\\Phi_r^{k+1}\\ge\\ &  \\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{8\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 +\\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2} \\\\\n& - 14r\\beta\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2},\n\\end{aligned}\n\\end{equation*}\nwhere $\\Phi_r^k:= \\Phi_r(x^k,y^k,z^k)$. \n% \\footnote{The proof of Proposition \\ref{prop:decrease}, which is based on the primal descent, dual ascent, and proximal descent properties, is given in Appendix \\ref{sec:suff_decrease}. }\n\\end{proposition}","sha256_utf8":"86abbffd20190bd6193654e966d5e253dd7f7f1e02c236ccfa73dfcd87017a32"}]},{"id":"S035","semantic_type":"proof","kind":"proof","title":"基本下降估计完整证明","source_units":[{"start":150967,"end":161183,"start_line":2020,"end_line":2155,"raw_tex":"\\paragraph{{\\bf Proof of Proposition \\ref{prop:decrease}}}\n% \\begin{proof}\nFrom Lemmas \\ref{lemma-F-des}, \\ref{lemma-d-aes}, and \\ref{lemma-p-des}, we have\n        \\begin{align*}\n        % \\label{desascent-key1}\n        & \\Phi_r(x^k,y^k,z^k)-\\Phi_r(x^{k+1},y^{k+1},z^{k+1}) \\notag\\\\\n         =\\ & F_r(x^k,y^k,z^k)-F_r(x^{k+1},y^{k+1},z^{k+1}) +2(d_r(y^{k+1},z^{k+1})-d_r(y^k,z^k))+2(p_r(z^k)-p_r(z^{k+1})) \\notag\\\\\n         \\ge\\ & \\frac{2\\lambda^{-1}+r-L}{2}\\|x^k-x^{k+1}\\|^2 +\\frac{(2-\\beta)r}{2 \\beta}\\|z^{k}-z^{k+1}\\|^{2}-\\left(L_{d_r}+\\frac{L}{2}\\right)\\|y^{k}-y^{k+1}\\|^{2}\\,  \\notag\\\\\n        &  + \\underbrace{\\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k}-y^{k+1}\\rangle+2\\langle\\nabla_{y} F_r(x_r(y^{k}, z^{k}),  y^{k}, z^{k}), y^{k+1}-y^{k}\\rangle}_{\\text{\\ding{172}}}\\, \n      \\notag \\\\ \n      & + \\underbrace{2 r\\left\\langle z^{k+1}-z^{k},x_r(y(z^{k+1}), z^{k})-x_r(y^{k+1}, z^{k+1})\\right\\rangle}_{\\text{\\ding{173}}}.\n        \\end{align*}\n% Subsequently, we simplify the terms \\text{\\ding{172}} and  \\text{\\ding{173}}. \nFirst, we simplify the term \\text{\\ding{172}}. We know that\n        \\begin{align*}\n        \\text{\\ding{172}} \n         =\\ & \\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k}-y^{k+1}\\rangle+2\\langle\\nabla_{y} F_r(x_r(y^{k}, z^{k}), y^{k}, z^{k}), y^{k+1}-y^{k}\\rangle\\\\\n        =\\ & \\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k+1}-y^{k}\\rangle + 2\\langle\\nabla_{y} F_r(x_r(y^{k}, z^{k}), y^{k}, z^{k})-\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k+1}-y^{k}\\rangle.\n        \\end{align*}\n        For the first term, we have\n        \\begin{align*}\n    &\\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k+1}-y^{k}\\rangle \\notag\\\\\n     =\\ &  \\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k)+\\frac{1}{\\alpha}(y^{k}-y^{k+1}), y^{k+1}-y^{k}\\rangle + \\frac{1}{\\alpha}\\|y^{k}-y^{k+1}\\|^2\\\\\n     =\\ & \\frac{1}{\\alpha}\\langle y^{k} + \\alpha \\nabla_{y} F_r(x^{k+1}, y^{k},z^k)-y^{k+1}, y^{k+1}-y^{k}\\rangle +\\frac{1}{\\alpha}\\|y^{k}-y^{k+1}\\|^2\\\\\n     \\ge\\ &  \\frac{1}{\\alpha}\\|y^{k}-y^{k+1}\\|^2, \n\\end{align*}\nwhere the last inequality follows from the property of the projection operator and the dual update $\ny^{k+1} = \\proj_{\\mathcal{Y}}( y^{k} + \\alpha \\nabla_{y} F_r(x^{k+1}, y^{k},z^k))\n$ (recall that $\\nabla_y F_r(x,y,z) = \\nabla_y F(x,y)$ for any $x \\in \\mathcal{X}$, $y \\in \\mathcal{Y}$, and $z \\in \\mathbb{R}^n$). For the remaining terms, we have\n                \\begin{align*}\n        %\\label{desascent-key3}\n         &2\\langle\\nabla_{y} F_r(x_r(y^{k}, z^{k}), y^{k}, z^{k})-\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k+1}-y^{k}\\rangle \\notag\\\\\n         \\ge & -2\\|\\nabla_{y} F_r(x_r(y^{k}, z^{k}),y^{k},z^{k})-\\nabla_{y} F_r(x^{k+1}, y^{k},z^{k})\\| \\cdot\\|y^{k+1}-y^{k}\\| \\notag\\\\\n         \\ge & -2 L\\|x^{k+1}-x_r(y^{k}, z^{k})\\| \\cdot\\|y^{k}-y^{k+1}\\| \\notag\\\\\n        \\ge  & -L \\zeta^2\\|y^{k}-y^{k+1}\\|^{2}-L\\zeta^{-2} \\|x^{k+1}-x_r(y^{k}, z^{k})\\|^{2} \\notag\\\\\n         \\ge & -L \\zeta^2\\|y^{k}-y^{k+1}\\|^{2}-L\\|x^{k+1}-x^{k}\\|^{2},\n        \\end{align*}\n        where the third inequality \n        % follows from $2|x||y| \\leq \\tau x^2 + \\frac{1}{\\tau}y^2$ \n        holds because $2 |a| |b| \\le \\tau a^2 + \\tfrac{1}{\\tau} b^2$\n        for any $a,b \\in \\mathbb{R}$ and $\\tau>0$, and the last inequality follows from Proposition \\ref{prop:lip}. \n        % where the first inequality is because of the Cauchy-Schwarz in equality, the second inequality is because $\\nabla_{y} K=\\nabla_{y} f$ is $L$-Lipschitz-continuous, the third inequality is due to the AM-GM inequality and the last is because of (B.5).\n% Subsequently, we focus on the left term in $\\text{\\ding{172}}$ that \nPutting the above together, we obtain\n\\begin{equation}\n\\label{desascent-key3}\n    \\text{\\ding{172}}  \\ge \\left(\\frac{1}{\\alpha}-L\\zeta^2\\right) \\|y^{k}-y^{k+1}\\|^{2}-L\\|x^{k+1}-x^{k}\\|^{2}. \n\\end{equation}\nNext, we bound the term $\\text{\\ding{173}}$ by \n\\begin{align}\\label{desascent-key2}\n  \\text{\\ding{173}} \n =\\ & 2 r \\left\\langle z^{k+1}-z^{k}, x_r(y(z^{k+1}), z^{k})-x_r(y^{k+1}, z^{k+1})\\right\\rangle \\notag\\\\\n =\\ & 2 r \\left\\langle z^{k+1}-z^{k}, x_r(y(z^{k+1}), z^{k})-x_r(y(z^{k+1}), z^{k+1})\\right\\rangle\\, \\notag\\\\\n & + 2 r\\left\\langle z^{k+1}-z^{k}, x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\right\\rangle\\notag \\\\\n\\ge &  -2r \\sigma_1\\|z^{k+1}-z^k\\|^2+2 r\\left\\langle z^{k+1}-z^{k}, x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\right\\rangle \\notag\\\\\n\\ge & -2r \\sigma_1\\|z^{k+1}-z^k\\|^2-\\frac{r}{7\\beta}\\|z^{k+1}-z^k\\|^2  -7r\\beta \\|x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\|^2, \n\\end{align}\nwhere the first inequality is due to \\eqref{lip-z} and the Cauchy-Schwarz inequality, and the second inequality again follows from the fact that $2 |a| |b| \\le \\tau a^2 + \\tfrac{1}{\\tau} b^2$ for any $a,b \\in \\mathbb{R}$ and $\\tau > 0$. Now, the inequalities \\eqref{desascent-key3} and \\eqref{desascent-key2} imply that\n\\begin{align}\\label{desas-keymid}\n        & \\Phi_r(x^k,y^k,z^k)-\\Phi_r(x^{k+1},y^{k+1},z^{k+1}) \\notag\\\\\n        \\geq\\ & \\frac{2\\lambda^{-1}+r-3L}{2}\\|x^{k}-x^{k+1}\\|^{2}+\\left(\\frac{1}{ \\alpha}-L_{d_r}-\\frac{L}{2}-L\\zeta^2\\right)\\|y^{k}-y^{k+1}\\|^{2}\\, \\notag\\\\\n        &+\\left(\\frac{(2-\\beta)r}{2 \\beta}-2r\\sigma_1-\\frac{r}{7\\beta}\\right)\\|z^{k}-z^{k+1}\\|^{2}-7r\\beta \\|x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\|^2. \n\\end{align}\nBy Lemma \\ref{lemma-dualxy} and the fact that $\\| u+v \\|^2 \\le 2(\\|u\\|^2+\\|v\\|^2)$ for any $u,v \\in \\mathbb{R}^d$, we have\n\\begin{align}\\label{desas-final-key1}\n\\|y^{k+1}-y^{k}\\|^{2} \n&=\\|y^{k+1}-y_{+}^{k}(z^{k+1})+y_{+}^{k}(z^{k+1})-y^{k}\\|^{2}\\notag\\\\\n&\\geq\\frac{1}{2}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2} -\\|y^{k+1}-y_{+}^{k}(z^{k+1})\\|^{2}\\notag\\\\\n% & \\geq \\frac{1}{2}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2} -\\alpha^2L^2\\|x^{k+1}-x_r(y^k,z^{k+1})\\|^2\\notag\\\\\n& \\geq \\frac{1}{2}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2} -2\\eta^2\\|x^{k+1}-x^k\\|^2-2\\sigma_1^2\\alpha^2L^2\\|z^{k+1}-z^k\\|^2.\n\\end{align}\n% where we recall that \n% \\[\n% y_{+}^{k}(z^{k+1}) = \\proj_{\\mathcal{Y}}(y^k+\\alpha \\nabla_y F_r (x_r(y^k,z^{k+1}),y^k,z^{k+1})).\n% \\]\n% \\leq\\ & \\alpha L\\|x^{k+1}-x_r(y^{k}, z^{k})\\| \\notag \\\\\n% \\leq\\ & \\eta\\|x^{k}-x^{k+1}\\|,\nSimilarly, by Lemma \\ref{lemma-sollip} and Lemma \\ref{lemma-dualxy}, we have\n\\begin{align}\\label{desas-final-key2}\n&\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\|^{2}\\notag\\\\\n\\leq \\ & 2\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_+^{k}(z^{k+1}),z^{k+1})\\|^2   + 2\\|x_r(y_+^{k}(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\|^2 \\notag\\\\\n\\leq \\ & 2\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_+^{k}(z^{k+1}),z^{k+1})\\|^2+2\\sigma_2^2 \\|y^{k+1}-y_{+}^{k}(z^{k+1})\\|^{2}\\notag\\\\\n\\leq \\ & 2\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_+^{k}(z^{k+1}),z^{k+1})\\|^2 + 4\\sigma_2^2\\eta^2 \\|x^{k+1}-x^k\\|^2+4\\sigma_2^2\\sigma_1^2\\alpha^2L^2\\|z^{k+1}-z^k\\|^2.\n\\end{align}\n\n% \\begin{align}\\label{desas-final-key2}\n% &\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\|^{2}\\notag\\\\\n% % =\\ & \\|x_r^\\star(z^{k+1})-x_r^\\star(z^{k})+x_r^\\star(z^{k})-x_r(y_{+}^{k}(z^{k}), z^{k})+x_r(y_{+}^{k}(z^{k}), z^{k})-x_r(y^{k+1}, z^{k})+x_r(y^{k+1}, z^{k})-x_r(y^{k+1}, z^{k+1})\\|^2 \\notag\\\\\n% \\leq\\ & 4\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y(z^{k}), z^{k})\\|^{2}+4\\|x_r(y(z^{k}), z^{k})-x_r(y_{+}^{k}(z^{k}), z^{k})\\|^{2}\\ +\\notag \\\\\n% &4\\|x_r(y_{+}^{k}(z^{k}), z^{k})-x_r(y^{k+1}, z^{k})\\|^{2}+4\\|x_r(y^{k+1}, z^{k})-x_r(y^{k+1}, z^{k+1})\\|^{2} \\notag\\\\\n% \\leq\\ & 8 \\sigma_{1}^{2}\\|z^{k}-z^{k+1}\\|^{2}+4\\|x_r(y(z^{k}), z^{k})-x_r(y_{+}^{k}(z^{k}), z^{k})\\|^{2} +4 \\sigma_{2}^{2} \\eta^2\\|x^{k}-x^{k+1}\\|^2.\n% \\end{align}\n% Besides, we recall that \n% \\begin{align}\\label{desas-final-key0}\n% x^{k+1}-z^{k} =\\frac{1}{\\beta}(z^{k+1}-z^k).\n% \\end{align} \nSubstituting \\eqref{desas-final-key1} and \\eqref{desas-final-key2} into \\eqref{desas-keymid} yields\n\\begin{align*}\n&\\Phi_r(x^k,y^k,z^k)-\\Phi_r(x^{k+1},y^{k+1},z^{k+1})\\notag\\\\\n \\geq\\ &\\left( \\frac{2\\lambda^{-1}+r-3L}{2} -28 r\\beta \\sigma_2^2\\eta^2 \\right)\\|x^{k}-x^{k+1}\\|^{2}\\ +\\left(\\frac{1}{ \\alpha}-L_{d_r}-\\frac{L}{2}-L\\zeta^2\\right) \\cdot \\\\\n& \\left(\\frac{1}{2}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2} -2\\eta^2\\|x^{k+1}-x^k\\|^2-2\\sigma_1^2\\alpha^2L^2\\|z^{k+1}-z^k\\|^2 \\right)\\ \\notag\\\\\n&+ \\left(\\frac{(2-\\beta)r}{2 \\beta}-2r\\sigma_1-\\frac{r}{7\\beta}-28r\\beta\\sigma_1^2\\sigma_2^2 \\alpha^2 L^2 \\right)\\|z^{k}-z^{k+1}\\|^{2} \\\\  & -  14r\\beta\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_+^{k}(z^{k+1}),z^{k+1})\\|^2.\n\\end{align*}\n\nSuppose that $r\\ge 3L$,\n% (which implies $L_{d_{r}}+\\frac{L}{2} \\leq 5L$)\nwhich implies that $L_{d_{r}}+\\frac{L}{2} \\leq 5L$. We observe the following:\n\\begin{itemize}\n\\item As $\\alpha \\leq \\min\\left\\{\\frac{1}{10L}, \\frac{1}{4L\\zeta^2} \\right\\}$, we have $\\frac{1}{ \\alpha}-L_{d_r}-\\frac{L}{2} \\ge \\frac{1}{2\\alpha}$ and \n$\n\\frac{1}{2\\alpha} - L\\zeta^2 \\ge \\frac{1}{4\\alpha}. \n$\n\\item As $\\beta\\leq\\frac{1}{28}$ and $\\sigma_1 \\leq \\frac{3}{2}$, $\\sigma_2\\leq \\frac{5}{2}$, we have \n\\begin{align*}\n& \\frac{(2-\\beta)r}{2 \\beta}-2r\\sigma_1-\\frac{r}{7\\beta}-28r\\beta\\sigma_1^2 \\sigma_2^2 \\alpha^2 L^2  -\\frac{1}{2\\alpha} \\sigma_1^2\\alpha^2L^2 \\\\ \n\\ge\\ & \\frac{6r}{7\\beta} -\\frac{7r}{2}- 28r\\beta\\left(\\frac{3}{2}\\right)^2\\left(\\frac{5}{2}\\right)^2\\left(\\frac{1}{10L}\\right)^2L^2-\\frac{r}{24}\\\\ \n\\ge\\ &  \\frac{r}{\\beta}\\left(\\frac{6}{7}-4\\beta- 14\\beta^2\\right) \\ge \\frac{4r}{7\\beta}.\n\\end{align*}\n\\item As $\\lambda^{-1} \\ge L$ and recalling that $\\eta = \\alpha L \\zeta$, we have \n\\[\\alpha\\leq\\frac{1}{4L \\zeta^2} = \\frac{\\lambda^{-1}}{4L^2\\zeta^2} \\frac{L}{\\lambda^{-1}} \\leq \\frac{\\lambda^{-1}}{4L^2\\zeta^2}\\quad \\text{and}\\quad \n\\frac{\\eta^2}{2\\alpha} = \\frac{\\alpha L^2\\zeta^2}{2} \\leq \\frac{1 }{8\\lambda}.\n\\]\nMoreover, since $\\beta  \\leq \\tfrac{(r-L)^2}{14 \\alpha r (2r-L)^2} = \\tfrac{1}{14 \\alpha r \\sigma_2^2}$,\n we have \n \\[\n 28 r\\beta \\sigma_2^2\\eta^2 \\leq \\frac{ 2\\eta^2}{\\alpha} \\leq \\frac{1}{2\\lambda}\\quad\\text{and}\\quad \\frac{2\\lambda^{-1}+r-3L}{2} -28 r\\beta \\sigma_2^2\\eta^2 - \\frac{\\eta^2}{2\\alpha} \\ge \\frac{3}{8\\lambda}.\n\\]\n\\end{itemize}\n% Together all pieces, we get \nPutting everything together yields\n\\begin{align*}\n&\\Phi_r(x^k,y^k,z^k)-\\Phi_r(x^{k+1},y^{k+1},z^{k+1}) \\\\\n\\ge\\ & \\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{8\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2} +\\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2} \\\\  & -14r\\beta\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2}.\n\\end{align*}\nThe proof is complete.\n% \\end{proof}\n","sha256_utf8":"f5aafd7236ce5634367e5fc252462af53717fddaf3b70fb8b1119eebdadd3aa7"}]},{"id":"S036","semantic_type":"proposition","kind":"proposition","title":"KL对偶误差界","source_units":[{"start":59051,"end":60090,"start_line":823,"end_line":836,"raw_tex":"\\begin{proposition}[Dual error bound with K\\L{} exponent] \n\\label{prop:dual_eb_KL}\n% Let $\\delta>0$. \nSuppose that Assumption \\ref{ass:kl} holds. Then, for any $y\\in\\mathcal{Y}$ and $z \\in \\mathbb{R}^n$, we have\n% \\[\n% \\|x_r^\\star(z)-x_r(y_+(z),z)\\| \\leq \\omega \\|y-y_+(z)\\|^{\\frac{1}{2\\theta}},\n% \\]\n% where $\\omega:=\\frac{\\sqrt{2}}{\\sqrt{r-L}}\\left(\\frac{1+\\alpha L(1+\\sigma_2)}{\\alpha\\mu}\\right)^{\\frac{1}{2\\theta}}$.\n\\begin{enumerate}[label=\\normalfont(\\alph*)]\n \\item \\rm{(K\\L\\ exponent $\\theta=0$):} $\\|x_r(y(z),z)-x_r(y_+(z),z)\\| \\leq \\omega_1 \\|y-y_+(z)\\|$, \n  \\item \\rm{(K\\L\\ exponent $\\theta\\in(0,1)$):} $\\|x_r(y(z),z)-x_r(y_+(z),z)\\| \\leq \\omega_2\\|y-y_+(z)\\|^{\\frac{1}{2\\theta}}$, \n  \\end{enumerate}\n    where $\\omega_1:= \\frac{\\sqrt{2L\\cdot {\\rm diam}(\\mathcal{Y})}\\cdot (1+\\alpha L)}{\\alpha\\mu\\sqrt{r-L}}$ and $\\omega_2:=\\frac{\\sqrt{2}}{\\sqrt{r-L}}\\left(\\frac{1+\\alpha L(1+\\sigma_2)}{\\alpha\\mu}\\right)^{\\frac{1}{2\\theta}}$ (recall that $y_+(z) = \\proj_{\\mathcal{Y}} ( y + \\alpha \\nabla_y F(x_r(y,z), y)$).  \n\\end{proposition}","sha256_utf8":"0fd1bfcc25854866cf9ef3e5a269afeeb493832130cbd82c99fe6db3cfd5dca1"}]},{"id":"S037","semantic_type":"proof","kind":"proof","title":"KL对偶误差界完整证明","source_units":[{"start":60091,"end":65950,"start_line":837,"end_line":932,"raw_tex":"\\begin{proof}\nLet $\\psi:\\R^n\\times \\R^n\\rightarrow\\R$ be the function defined by\n\\[\n\\psi(x,z) =  F_r(x,y(z),z). \n\\]\nConsider arbitrary $x \\in \\mathcal{X}$, $y \\in \\mathcal{Y}$, and $z \\in \\mathbb{R}^n$. Note that the function $\\psi(\\cdot,z)$ is $(r-L)$-strongly convex. Since\n\\begin{align*}\n\\mathop{\\argmin}_{x'\\in\\mathcal{X}}\\psi(x',z) & =\\mathop{\\argmin}_{x'\\in\\mathcal{X}} F_r(x',y(z),z) =  x_r(y(z),z),\n\\end{align*}\nwe see that \n% for any $x\\in \\mathcal{X}$, \n\\begin{equation}\n\\label{eq:dual1}\n\\psi(x,z) - \\psi(x_r(y(z),z),z) \\ge \\frac{r-L}{2}\\|x-x_r(y(z),z)\\|^2. \n\\end{equation}\nIn addition, we have\n\\begin{align} \n\\label{eq:dual2}\n&\\psi(x,z) - \\psi(x_r(y(z),z),z)\\notag\\\\   \n\\leq \\  &\\psi(x,z)-F_r(x_r(y_+(z),z),y_+(z),z)\\notag \\\\\n\\leq \\ & \\max_{y^\\prime\\in \\mathcal{Y}} F(x,y^\\prime) + \\frac{r}{2}\\|x-z\\|^2-F_r(x_r(y_+(z),z),y_+(z),z)\\notag\\\\\n=\\,& \\max_{y^\\prime\\in \\mathcal{Y}}F(x,y^\\prime)-F(x_r(y_+(z),z),y_+(z)) + \\frac{r}{2}\\|x-z\\|^2-\\frac{r}{2}\\|x_r(y_+(z),z)-z\\|^2,\n\\end{align}\nwhere the first inequality follows from\n\\begin{align*}\nF_r(x_r(y_+(z),z),y_+(z),z) \n=\\ & \\min_{x'\\in\\mathcal{X}}\\left\\{ F(x',y_+(z)) +\\frac{r}{2}\\|x'-z\\|^2\\right\\} \\nonumber \\\\\n\\leq\\ & \\max_{y' \\in \\mathcal{Y}}\\min_{x'\\in\\mathcal{X}}\\left\\{F(x',y') +\\frac{r}{2}\\|x'-z\\|^2\\right\\} \\nonumber \\\\\n=\\ & \\min_{x'\\in\\mathcal{X}}\\left\\{F(x',y(z)) +\\frac{r}{2}\\|x'-z\\|^2\\right\\} \\nonumber \\\\\n=\\ & \\min_{x'\\in\\mathcal{X}} \\psi(x',z)\n= \\psi(x_r(y(z),z),z). % \\label{eq:Fr-one}\n\\end{align*}\n% \\Jiajin{We may have to double-check the above derivation. --- confirmed. With the uniform KL property, we can confirm  $x(y(z),z)$ is unique over $Y(z)$.}\nAs \\eqref{eq:dual1} and \\eqref{eq:dual2} hold for any $x\\in\\mathcal{X}$, we obtain the intermediate relation\n\\begin{equation}\\label{OS-GS-better-key3-main} \n% \\begin{aligned}\n     \\frac{r-L}{2}\\|x_r(y(z),z)-x_r(y_+(z),z)\\|^2\\leq  \\max\\limits_{y^\\prime\\in \\mathcal{Y}} F(x_r(y_+(z),z),y^\\prime) - F(x_r(y_+(z),z),y_+(z)) \n% \\end{aligned}\n\\end{equation}\nby taking $x= x_r(y_+(z),z)$.\n\nNow, we utilize the K{\\L} exponent $\\theta$ of the dual problem to bound the right-hand side of \n\\eqref{OS-GS-better-key3-main}  in terms of $\\|y-y_+(z)\\|$. Consider the following two cases:\n \\begin{enumerate}[label=\\normalfont(\\alph*)]\n \\item Suppose that $\\theta=0$. If $F(x_r(y_+(z),z),y_+(z))=\\max_{y'\\in\\mathcal{Y}} F(x_r(y_+(z),z),y')$, then the desired inequality follows trivially from~\\eqref{OS-GS-better-key3-main}. Otherwise, we have $y_+(z) \\in \\mathcal{Y} \\setminus \\mathcal{Y}^\\star(x_r(y_+(z),z))$. By Assumption~\\ref{ass:kl}, we have\n% we deduce using~\\eqref{eq:Fr-one} that\n% \\begin{align*}\n% F_r(x_r(y_+(z),z),y_+(z),z)\n% &=\\max_{y'\\in\\mathcal{Y}} F_r(x_r(y_+(z),z),y',z)\\\\\n% &\\ge F_r(x_r(y_+(z),z),y_+(z),z)\\\\\n% &\\ge F_r(x_r(y(z),z),y(z),z).\n% \\end{align*}\n%By definition of $y(z)$, it follows that $y_+(z)=y(z)$, and consequently the conclusion trivially holds. \n\\[\n\\dist(0, -\\nabla_y F(x_r(y_+(z),z),y_+(z))+\\partial\\iota_{\\mathcal{Y}}(y_+(z)))\\ge \\mu.\n\\]\nSince  $F(x,\\cdot)$ is continuously differentiable on the compact set $\\mathcal{Y}$, $\\|\\nabla_y F(x,\\cdot)\\|$ is bounded on $\\mathcal{Y}$. Without loss of generality, we can assume that  $\\|\\nabla_y F(x,\\cdot)\\|\\leq L$ on $\\mathcal{Y}$. According to  the mean value theorem, we have \n\\[\n\\max\\limits_{y^\\prime\\in \\mathcal{Y}} F(x_r(y_+(z),z),y^\\prime) - F(x_r(y_+(z),z),y_+(z))\n\\leq L \\cdot {\\rm diam}(\\mathcal{Y}).\n\\]\nUsing the fact that $y_+(z) = \\proj_{\\mathcal{Y}} (y + \\alpha \\nabla_y F(x_r(y,z), y))$, we bound\n\\begin{align*}\n &\\max\\limits_{y^\\prime\\in \\mathcal{Y}} F(x_r(y_+(z),z),y^\\prime) - F(x_r(y_+(z),z),y_+(z))\\notag\\\\\n\\leq\\ & \\frac{L \\cdot {\\rm diam}(\\mathcal{Y})}{\\mu^2}\\cdot\\dist^2(0, -\\nabla_y F(x_r(y_+(z),z),y_+(z))+\\partial\\iota_{\\mathcal{Y}}(y_+(z)))\\notag\\\\ \n% \\stackrel{\\text{\\ding{172}}}{\\leq} \\ &   \\frac{L \\cdot {\\rm diam}(\\mathcal{Y})}{\\mu^2} b^2 \\|\\proj_{\\mathcal{Y}}(y+\\alpha\\nabla_y F(x_r(y_+(z),z),y))-y\\|^2\n% \\notag \\\\\n% \\leq \\ &  \\frac{ L b^2 \\cdot {\\rm diam}(\\mathcal{Y})}{\\mu^2}(\n% \\|\\proj_{\\mathcal{Y}}(y+\\alpha\\nabla_y F(x_r(y_+(z),z),y))-y_+(z)\\|\\ +\\|y_+(z)-y\\|)^2 \\\\ \n% \\leq \\ & \\frac{L b^2 \\cdot {\\rm diam}(\\mathcal{Y})}{\\mu^2}(\\alpha L \\|x_r(y_+(z),z)-x_r(y,z)\\| + \\|y-y_+(z)\\|)^2\\\\\n\\leq \\ & \\frac{L b^2 \\cdot {\\rm diam}(\\mathcal{Y})}{\\mu^2}\\|y_+(z)-y\\|^2,\n\\end{align*}\nwhere $b:=\\frac{1}{\\alpha} + L > 0$ and the last inequality is due to the fact that the projected gradient ascent method satisfies the so-called \\textit{relative error condition}\n\\citep[Section 5]{attouch2013convergence}. \nIt follows that\n\\[\n\\|x_r(y(z),z)-x_r(y_+(z),z)\\| \\leq\\frac{\\sqrt{2L\\cdot {\\rm diam}(\\mathcal{Y})}\\cdot (1+\\alpha L)}{\\alpha\\mu\\sqrt{r-L}}\\|y-y_+(z)\\|. \n\\]\n\\item If $\\theta \\in (0,1)$, then we have\n\\begin{align*}\n% \\label{KL-key-01}\n& \\mu\\left(\\max\\limits_{y^\\prime\\in \\mathcal{Y}} F(x_r(y_+(z),z),y^\\prime) - F(x_r(y_+(z),z),y_+(z))\\right)^{\\theta}\\notag\\\\\n\\leq \\ & \\dist(0, -\\nabla_y F(x_r(y_+(z),z),y_+(z))+\\partial\\iota_{\\mathcal{Y}}(y_+(z))) \\notag\\\\ \n\\leq \\ & \\dist(0, -\\nabla_y F(x_r(y,z),y_+(z)) +\\partial\\iota_{\\mathcal{Y}}(y_+(z)))   \\notag\\\\\n\\ & + \\|\\nabla_y F(x_r(y,z),y_+(z))-\\nabla_y F(x_r(y_+(z),z),y_+(z))\\|\\notag\\\\ \n{\\leq} \\ & \\dist(0, -\\nabla_y F(x_r(y,z),y_+(z))+\\partial\\iota_{\\mathcal{Y}}(y_+(z)))+ L\\sigma_2\\|y-y_+(z)\\|\\notag\\\\\n{\\leq} \\ & \\left(\\frac{1}{\\alpha}+L+L\\sigma_2\\right)\\|y_+(z)-y\\|,\n\\end{align*}\nwhere the third inequality follows from the $L$-Lipschitz continuity  of $\\nabla_y F(\\cdot,\\cdot)$ and  \\eqref{lip-y};  the last inequality follows from the relative error condition of the projected gradient ascent method. This, together with \\eqref{OS-GS-better-key3-main}, yields\n\\[\n\\|x_r(y(z),z)-x_r(y_+(z),z)\\|\\leq \\frac{\\sqrt{2}}{\\sqrt{r-L}}\\left(\\frac{1+\\alpha L(1+\\sigma_2)}{\\alpha\\mu}\\right)^{\\frac{1}{2\\theta}} \n\\|y-y_+(z)\\|^{\\frac{1}{2\\theta}}.\n\\]\n \\end{enumerate}\nThe proof is complete.\n\\end{proof}","sha256_utf8":"5671404bb10c8c6f2789f876a5b7ff434c665172422a86e000fd5314c8d2e90a"}]},{"id":"S038","semantic_type":"corollary","kind":"corollary","title":"优化驻点所需对偶误差界","source_units":[{"start":67130,"end":67876,"start_line":938,"end_line":950,"raw_tex":"\\begin{corollary}[Alternative dual error bound with K\\L{} exponent]\n% Let $\\delta>0$. \nSuppose that  Assumption \\ref{ass:kl} holds. Then, for any $y \\in \\mathcal{Y}$ and $z \\in \\mathbb{R}^n$,  we have\n% \\[\n% \\|x_r^\\star(z)-x_r(y_+(z),z)\\| \\leq \\omega \\|y-y_+(z)\\|^{\\frac{1}{2\\theta}},\n% \\]\n% where $\\omega:=\\frac{\\sqrt{2}}{\\sqrt{r-L}}\\left(\\frac{1+\\alpha L(1+\\sigma_2)}{\\alpha\\mu}\\right)^{\\frac{1}{2\\theta}}$.\n\\begin{enumerate}[label=\\normalfont(\\alph*)]\n \\item \\rm{(K\\L\\ exponent $\\theta=0$):} $\\|x_r^\\star(z)-x_r(y_+(z),z)\\| \\leq \\omega_1 \\|y-y_+(z)\\|$,\n  \\item \\rm{(K\\L\\ exponent $\\theta\\in(0,1)$):} $\\|x_r^\\star(z)-x_r(y_+(z),z)\\| \\leq \\omega_2\\|y-y_+(z)\\|^{\\frac{1}{2\\theta}}$. \n  \\end{enumerate}\n \\label{prop:dual_eb_KL2}   \n\\end{corollary}","sha256_utf8":"b9552f35e65917abfd8aeb78107bad2e1f1e77cca97b73fb222b5f23e07c63b8"}]},{"id":"S039","semantic_type":"proof","kind":"proof","title":"替代对偶误差界完整证明","source_units":[{"start":161253,"end":163160,"start_line":2159,"end_line":2196,"raw_tex":"\\begin{proof}\nThe proof follows closely that of Proposition \\ref{prop:dual_eb_KL}. Let $\\psi:\\R^n\\times \\R^n\\rightarrow\\R$ be the function defined by\n\\[\n\\psi(x,z) = \\max\\limits_{y\\in \\mathcal{Y}} F_r(x,y,z)\n\\]\nand consider arbitrary $x \\in \\mathcal{X}$, $y \\in \\mathcal{Y}$, and $z \\in \\mathbb{R}^n$. Again, note that  $\\psi(\\cdot,z)$ is $(r-L)$-strongly convex. This implies that\n% for any $x\\in \\mathcal{X}$, we have\n\\begin{equation}\n\\label{OS-GS-better-key1}\n\\psi(x,z) - \\psi(x^\\star_r(z),z) \\ge \\frac{r-L}{2}\\|x-x^\\star_r(z)\\|^2. \n\\end{equation}\nMoreover, \n% for any $x\\in \\mathcal{X}$, \nwe  have\n\\begin{align} \n\\label{OS-GS-better-key2}\n&\\psi(x,z)-\\psi(x_r^\\star(z),z) \\notag\\\\   \n\\leq \\  &\\psi(x,z)-F_r(x_r(y_+(z),z),y_+(z),z)\\notag \\\\\n= \\ & \\max_{y^\\prime\\in \\mathcal{Y}} F(x,y^\\prime) + \\frac{r}{2}\\|x-z\\|^2-F_r(x_r(y_+(z),z),y_+(z),z)\\notag\\\\\n=\\,& \\max_{y^\\prime\\in \\mathcal{Y}}F(x,y^\\prime)-F(x_r(y_+(z),z),y_+(z)) + \\frac{r}{2}\\|x-z\\|^2-\\frac{r}{2}\\|x_r(y_+(z),z)-z\\|^2,\n\\end{align}\nwhere the  inequality follows from\n\\begin{align*}\nF_r(x_r(y_+(z),z),y_+(z),z) \n=\\ & \\min_{x'\\in\\mathcal{X}}\\left\\{ F(x',y_+(z)) +\\frac{r}{2}\\|x'-z\\|^2\\right\\} \\\\\n\\leq\\ & \\max_{y' \\in \\mathcal{Y}}\\min_{x'\\in\\mathcal{X}}\\left\\{F(x',y') +\\frac{r}{2}\\|x'-z\\|^2\\right\\} \\\\\n\\leq  \\ & \\min_{x'\\in\\mathcal{X}}\\max_{y' \\in \\mathcal{Y}}\\left\\{F(x',y') +\\frac{r}{2}\\|x'-z\\|^2\\right\\}\n= \\psi(x_r^\\star(z),z).\n\\end{align*}\nAs \\eqref{OS-GS-better-key1} and \\eqref{OS-GS-better-key2} hold for any $x\\in\\mathcal{X}$, we obtain the intermediate relation\n\\begin{equation*}\n% \\label{OS-GS-better-key3} \n    \\frac{r-L}{2}\\|x_r^\\star(z)-x_r(y_+(z),z)\\|^2 \\leq \\max\\limits_{y^\\prime\\in \\mathcal{Y}} F(x_r(y_+(z),z),y^\\prime) - F(x_r(y_+(z),z),y_+(z)) \n\\end{equation*}\nby taking $x= x_r(y_+(z),z)$.\n\nThe remaining steps of the proof are the same as those of Proposition \\ref{prop:dual_eb_KL}. For brevity, we omit them here.\n\\end{proof}","sha256_utf8":"f0e754c01317418b0878168a5a6ce6ca1187717e613229e65b45af4cbed08729"}]},{"id":"S040","semantic_type":"lemma","kind":"lemma","title":"凹情形对偶误差界","source_units":[{"start":163432,"end":163745,"start_line":2205,"end_line":2213,"raw_tex":"\\begin{lemma}\\label{lemma-dual-bd}\nFor any $y \\in \\mathcal{Y}$ and $z\\in\\R^n$, we have\n\\begin{equation*}\n% \\label{dual-bd}\n\\|x^\\star_r(z)-x_r(y_{+}(z), z)\\|^{2} \n \\leq \\kappa \\|y-y_{+}(z)\\| ,\n\\end{equation*}\nwhere $\\kappa:=\\frac{1+ \\alpha L\\sigma_2+\\alpha L}{\\alpha(r-L)}\\cdot{\\rm diam}(\\mathcal{Y})$.\n\\end{lemma}","sha256_utf8":"6f833961db4b3446d9467678bb3f0c9af107a4490c9171bc01ba2614caa82ef4"}]},{"id":"S041","semantic_type":"proof","kind":"proof","title":"凹情形误差界完整证明","source_units":[{"start":163746,"end":166612,"start_line":2214,"end_line":2253,"raw_tex":"\\begin{proof}\nLet $y \\in \\mathcal{Y}$ and $z \\in \\mathbb{R}^n$ be arbitrary. Recall that $y(z)$ is an arbitrary vector in $Y(z)$. By the $(r-L)$-strong convexity of $F_r(\\cdot, y, z)$, we have\n\\begin{align}\n&F_r(x_r^\\star(z), y_{+}(z), z)-F_r(x_r(y_{+}(z), z), y_{+}(z), z)  \\geq \\frac{r-L}{2}\\|x_r(y_{+}(z), z)-x^\\star_r(z)\\|^{2}, \\notag\\\\\n&F_r(x_r(y_{+}(z), z), y(z), z)-F_r(x_r^\\star(z), y(z), z)  \\geq \\frac{r-L}{2}\\|x_r(y_{+}(z), z)-x_r^\\star(z)\\|^{2}.\\notag\n\\end{align}\n Moreover, since $(x_r^\\star(z),y(z))$ is a saddle point of the convex-concave function $F_r(\\cdot,\\cdot,z)$ \\citep{sion1958general}, we have $F_r(x_r^\\star(z), y(z), z) \\geq F_r(x_r^\\star(z), y_{+}(z), z)$, which implies that\n\\begin{equation}\\label{dualbd-strcvx}\nF_r(x_r(y_{+}(z), z), y(z), z)-F_r(x_r(y_{+}(z), z), y_{+}(z), z) \\geq  (r-L)\\|x_r(y_{+}(z), z)-x_r^\\star(z)\\|^{2}.\n\\end{equation}\nNote that $y_{+}(z)$ is the maximizer of \n\\[\n\\max_{y' \\in \\mathcal{Y}}\\big\\langle y+\\alpha \\nabla_{y} F_r(x_r(y, z), y, z)-y_{+}(z) , y'\\big\\rangle.\n\\]\nFor simplicity, we define the function $\\xi: \\mathbb{R}^d \\rightarrow \\mathbb{R}$ by\n\\begin{align*}\n\\xi(\\cdot)\n:=\\ &\\alpha F_r(x_r(y_{+}(z), z),\\cdot, z)-\\big\\langle \\alpha\\nabla_{y} F_r(x_r(y_{+}(z), z), y_{+}(z), z),\\cdot\\big\\rangle \\\\ \\ &-\\big\\langle y_{+}(z)-y-\\alpha \\nabla_{y} F_r(x_r(y, z), y, z), \\cdot\\big\\rangle.\n\\end{align*}\nThen, we have\n\\begin{align*}\n\\max_{y' \\in \\mathcal{Y}}\\ \\xi(y')\n&\\leq \\alpha F_r(x_r(y_{+}(z), z),y_+(z), z)-\\big\\langle\\alpha\\nabla_y F_r(x_r(y_{+}(z), z),y_+(z), z), y_+(z)\\big\\rangle\\, \\notag\\\\\n&\\quad + \\max_{y' \\in \\mathcal{Y}} \\big\\langle y+\\alpha \\nabla_{y} F_r(x_r(y, z), y, z)-y_{+}(z), y'\\big\\rangle \\notag\\\\\n&\\leq \\xi(y_{+}(z)),\n\\end{align*}\nwhere the first inequality holds because $F_r(x_r(y_{+}(z), z),\\cdot,z)$ is concave. Therefore, we have $\\xi(y(z))\\leq \\xi(y_{+}(z))$, which implies that\n\\begin{align}\\label{dualbd-key}\n&F_r(x_r(y_{+}(z), z), y(z), z)-F_r(x_r(y_{+}(z), z), y_{+}(z), z) \\notag\\\\\n\\leq &\\ \\frac{1}{\\alpha}\\Big\\langle y(z)-y_{+}(z), \\alpha\\nabla_{y} F_r(x_r(y_{+}(z), z), y_{+}(z), z)-\\alpha \\nabla_{y} F_r(x_r(y, z), y, z)\\Big\\rangle \\notag \\\\ \\ & \\ + \\frac{1}{\\alpha}\\Big\\langle y(z)-y_{+}(z), y_{+}(z)-y\\Big\\rangle \\notag\\\\\n\\leq &\\ L \\|y_{+}(z)-y(z)\\|(\\|x_r(y_{+}(z), z)-x_r(y, z)\\|+ \\|y_{+}(z)-y\\|)  + \\frac{1}{\\alpha}\\|y_{+}(z)-y(z)\\|\\cdot \\|y-y_{+}(z)\\|\\notag\\\\\n\\leq &\\ \\left(\\frac{1}{\\alpha}+L\\sigma_2+L\\right)\\|y_{+}(z)-y(z)\\| \\cdot\\|y-y_{+}(z)\\|.\n\\end{align}\nHere, the second inequality is due to the $L$-Lipschitz continuity of $\\nabla_y F_r(\\cdot,\\cdot,z)$ and the last inequality is from \\eqref{lip-y}.\nHence, by combining \\eqref{dualbd-strcvx} and \\eqref{dualbd-key}, we obtain\n\\[\n(r-L)\\|x_r^\\star(z)-x_r(y_{+}(z), z)\\|^{2}\\leq \\left(\\frac{1}{\\alpha}+ L\\sigma_2+L\\right){\\rm diam}(\\mathcal{Y})\\cdot\\|y-y_{+}(z)\\|,\n\\]\nwhich proves the desired result.\n\\end{proof}","sha256_utf8":"6c7718cd244789a2247de5b2385d88b7131ff0cb917f535c271d1b8971b947cd"}]},{"id":"S042","semantic_type":"lemma","kind":"lemma","title":"迭代间隙控制博弈驻点","source_units":[{"start":69575,"end":70139,"start_line":970,"end_line":978,"raw_tex":"\\begin{lemma} \\label{lemma-episolcol}\n        Let $\\epsilon\\ge 0$ be given. Suppose that\n        \\[\n        \\max\\left\\{\\|x^{k+1}-x^{k}\\|,\\|y^{k}_+(z^{k+1}) - y^k\\|,\\|x^{k+1}-z^k\\|\\right\\} \\leq  \\epsilon.\n        \\]\n        Then,  $\\left(x^{k+1}, y^{k+1}\\right)$ is a $\\rho \\epsilon$-GS of problem \\eqref{eq:problem}, where\n\\[\\rho = \\max\\left\\{(\\eta+1+\\sigma_1\\alpha\\beta L)\\left(\\frac{1}{\\alpha}+L\\right), r(\\zeta+\\sigma_2(\\eta+1+\\sigma_1\\alpha \\beta L)+\\sigma_1)\\right\\}\\] \n with $\\sigma_{1}:=\\frac{r}{r-L}$ and $\\sigma_{2}:=\\frac{2r-L}{r-L}$.\n        \\end{lemma}","sha256_utf8":"6accc28c7cde6c83b8b6e58c31fbd1d3bd1cbefc637c885ebb860c3d3e27c361"}]},{"id":"S043","semantic_type":"proof","kind":"proof","title":"驻点间隙引理完整证明","source_units":[{"start":70140,"end":72783,"start_line":979,"end_line":1016,"raw_tex":"\\begin{proof}\nBy Definition \\ref{defi:primal-dual}, we only have to quantify $\\|\\nabla_{z} d_r(y^{k+1},x^{k+1})\\|$ and ${\\rm dist}(0,-\\nabla_{y} F(x^{k+1}, y^{k+1})+\\partial\\iota_{\\mathcal{Y}}(y^{k+1}))$.\nTo begin, observe that\n\\begin{equation*}\n\\begin{aligned}\n& \\| \\nabla_{z} d_r(y^{k+1},x^{k+1})\\| \n\\\\\n=\\ & r \\|x^{k+1}-x_r (y^{k+1},x^{k+1})\\|\\\\  \n\\leq\\ & r(\\|x^{k+1}-x_r(y^{k},z^{k})\\| +\\|x_r(y^{k},z^{k})-x_r(y^{k+1},z^{k})\\|)  +r \\|x_r(y^{k+1},z^{k})-x_r(y^{k+1},x^{k+1})\\|  \\\\\n\\leq\\  &  r\\left(\\zeta\\|x^{k+1}-x^k\\| +\\sigma_2\\|y^{k}-y^{k+1}\\| + \\sigma_1 \\|x^{k+1}-z^{k}\\| \\right) \\\\\n% \\leq & r\\left(\\|x^{k+1}-x_r(y^{k},z^{k})\\| +\\|x_r(y^{k},z^{k})-x_r(y^{k+1},z^{k})\\| + \\|x_r(y^{k+1},z^{k})-x_r(y^{k+1},x^{k+1})\\| \\right) \\\\\n\\leq\\  &  r\\left(\\zeta\\|x^{k+1}-x^k\\| + \\sigma_1 \\|x^{k+1}-z^{k}\\| \\right) +r\\sigma_2 (\\eta\\| x^{k+1}-x^k\\|+\\sigma_1\\alpha L\\|z^{k+1}-z^k\\|+\\| y_+^k(z^{k+1})-y^k\\|)\\\\\n\\leq\\ & r(\\zeta+\\sigma_2(\\eta+1+\\sigma_1\\alpha \\beta L)+\\sigma_1)\\epsilon,\n\\end{aligned}\n\\end{equation*}\nwhere the second inequality is due to Proposition \\ref{prop:lip} and Lemma \\ref{lemma-sollip}, the third is due to Lemma \\ref{lemma-dualxy}, and the fourth follows from the update $z^{k+1} = z^k + \\beta (x^{k+1} - x^k)$ and the assumption of the lemma. Next, recall that\n\\begin{equation*}\ny^{k+1} = \\proj_{\\mathcal{Y}}\\left(y^k + \\alpha \\nabla_y F(x^{k+1}, y^k)\\right)=\\argmin_{y \\in \\mathcal{Y}}\\left\\{\\|y-y^k - \\alpha \\nabla_y F(x^{k+1}, y^k)\\|^2\\right\\}.\n\\end{equation*}\nThe necessary optimality condition yields\n\\begin{align*}\n0\n&\\in y^{k+1}-y^k - \\alpha \\nabla_y F(x^{k+1}, y^k)+\\partial \\iota_{\\mathcal{Y}}(y^{k+1})\\notag\\\\\n&=y^{k+1}-y^k - \\alpha \\nabla_y F(x^{k+1}, y^{k+1})+\\partial \\iota_{\\mathcal{Y}}(y^{k+1}) +\\alpha (\\nabla_y F(x^{k+1}, y^{k+1})- \\nabla_y F(x^{k+1}, y^{k})).\n\\end{align*}\nSince $\\partial \\iota_{\\mathcal{Y}} (y^{k+1})$ is the normal cone to $\\mathcal{Y}$ at $y^{k+1}$, we have\n\\begin{align*}\nv\n&:=-\\frac{1}{\\alpha}(y^{k+1}-y^{k})+\\nabla_y F(x^{k+1}, y^{k})- \\nabla_y F(x^{k+1}, y^{k+1})\\in -\\nabla_{y} F(x^{k+1}, y^{k+1})+\\partial \\iota_{\\mathcal{Y}}(y^{k+1}).\n\\end{align*}\nIt follows that\n\\begin{align*}\n\\|v\\| \\ & \\leq \\left(\\frac{1}{\\alpha}+L\\right)\\|y^{k+1}-y^{k}\\|\\\\\n\\ &  \\leq  \\left(\\frac{1}{\\alpha}+L\\right)\\left(\\eta\\|x^{k}-x^{k+1}\\|+\\sigma_1\\alpha L\\|z^{k+1}-z^k\\|+\\|y_+^k(z^{k+1})-y^k\\|\\right) \\\\\n\\ & \\leq (\\eta+1+\\sigma_1\\alpha\\beta L)\\left(\\frac{1}{\\alpha}+L\\right)\\epsilon,\n\\end{align*}\nwhere the first inequality is due to the $L$-Lipschitz continuity of $\\nabla_y F(\\cdot,\\cdot)$ and the second is due to Lemma \\ref{lemma-dualxy}. Putting everything together yields the desired result.\n\\end{proof}","sha256_utf8":"611ece6f700405f9a53cee16df384bf2f522e45cfb68650f9d8d74a8d8947d77"}]},{"id":"S044","semantic_type":"external_fact","kind":"external_fact","title":"复合模型的标准估计","source_units":[],"declared_external":true,"declaration_text":["若 h 凸且 L_h-Lipschitz，c 的雅可比为 L_c-Lipschitz，则 h∘c 为 L_h L_c-弱凸，且其一阶复合线性模型与函数值之差的绝对值不超过 (L_h L_c/2)||x−x̄||²；共同弱凸参数的函数族的有限上确界仍具有该弱凸参数。对应 drusvyatskiy2019efficiency，Lemmas 3.2、4.2。"]},{"id":"S045","semantic_type":"external_fact","kind":"external_fact","title":"次梯度与近端误差界转换","source_units":[],"declared_external":true,"declaration_text":["对适用的闭凸函数 g，若 dist(x,argmin g)≤κ dist(0,∂g(x))，则对 t>0 有 dist(x,argmin g)≤((κ+t)/t)||x−prox_{tg}(x)||。对应 drusvyatskiy2018error，Theorem 3.4。"]},{"id":"S046","semantic_type":"external_fact","kind":"external_fact","title":"凸分析与光滑估计","source_units":[],"declared_external":true,"declaration_text":["ι_C 为集合 C 的示性函数，闭凸集上 ∂ι_C 为法锥。凸函数满足次梯度不等式与次梯度最优性条件；闭凸集投影满足变分不等式，投影及闭凸函数近端映射非扩张；L-Lipschitz 梯度给出误差至多 (L/2)||u−v||² 的一阶函数值上下界。另使用柯西–施瓦茨、三角不等式和 Young 不等式。"]},{"id":"S047","semantic_type":"external_fact","kind":"external_fact","title":"投影梯度相对误差条件","source_units":[],"declared_external":true,"declaration_text":["若 g 的梯度为 L-Lipschitz，C 闭凸且 y⁺=proj_C(y+α∇g(y))，则 dist(0,−∇g(y⁺)+N_C(y⁺))≤(α⁻¹+L)||y⁺−y||。对应 attouch2013convergence，Section 5。"]},{"id":"S048","semantic_type":"external_fact","kind":"external_fact","title":"唯一极小解的值函数微分","source_units":[],"declared_external":true,"declaration_text":["在文中参数化极小化满足的正则性条件下，强凸性保证唯一极小解，值函数对光滑参数可微，其梯度等于目标函数在该极小解处的参数梯度。对应 rockafellar2009variational，Theorem 10.31。"]},{"id":"S049","semantic_type":"external_fact","kind":"external_fact","title":"弱凸Moreau包络梯度","source_units":[],"declared_external":true,"declaration_text":["若 g 为适当下半连续 L-弱凸函数且 r>L，则 e(z)=min_x{g(x)+(r/2)||x−z||²} 可微，并有 ∇e(z)=r(z−prox_{g/r}(z))。对应 drusvyatskiy2019efficiency，Lemma 4.3。"]},{"id":"S050","semantic_type":"external_fact","kind":"external_fact","title":"凸凹极小极大交换","source_units":[],"declared_external":true,"declaration_text":["Sion 极小极大定理：在适当半连续性、关于两变量的凸凹性及一侧可行域紧致的条件下，min_x max_y H(x,y)=max_y min_x H(x,y)；若两侧最优解存在，其组成鞍点。文中将其用于正则后的 F_r，并用于由原值函数下界推出 p_r 的下界；主证明另引 zhang2020single，Lemma B.7。"]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"强凸性给出次梯度误差界","source_units":[{"start":51863,"end":52929,"start_line":741,"end_line":758,"raw_tex":"\\begin{proof}\nFor ease of notation, we define $\\hat{F}_{y,z}(\\cdot): = F_r(\\cdot,y,z) + \\iota_{\\mathcal{X}}(\\cdot)$ for any $y\\in\\mathcal{Y}$ and $z\\in\\R^n$.  As $x_r(y,z)$ is the optimal solution of $\\min_{x \\in \\mathbb{R}^n} \\hat{F}_{y,z} (x)$ and $F_r(\\cdot,y,z)$ is $(r-L)$-strongly convex (see the remark after Fact \\ref{fact-2} in Appendix \\ref{sec:lemmas}), we have\n\\begin{equation}\n    \\label{eq:lip_1} \n    \\hat{F}_{y^k,z^k}(x)-\\hat{F}_{y^k,z^k}(x_r(y^k,z^k)) \\ge \\frac{r-L}{2} \\|x-x_r(y^k,z^k)\\|^2 \\quad \\text{for all}\\ x \\in \\mathcal{X}.\n\\end{equation}\nIn addition, by the convexity of $\\hat{F}_{y^k,z^k}$, we obtain \n\\begin{equation}\n    \\label{eq:lip_2}\n    \\begin{aligned}\n    \\hat{F}_{y^k,z^k}(x)-\\hat{F}_{y^k,z^k}(x_r(y^k,z^k)) & \\leq \\dist(0, \\partial \\hat{F}_{y^k,z^k}(x))\\cdot \\|x-x_r(y^k,z^k)\\| \\\\ \n    \\end{aligned}\n\\end{equation}\n$\\text{for all}\\ x\\in\\mathcal{X}$.\nCombining \\eqref{eq:lip_1} and \\eqref{eq:lip_2} yields\n\\[\n\\|x-x_r(y^k,z^k)\\| \\leq \\frac{2}{r-L} \\dist(0, \\partial \\hat{F}_{y^k,z^k}(x)) \\quad \\text{for all}\\ x\\in\\mathcal{X}. \n\\]\n","sha256_utf8":"0fe7fde2da475136a32b7bf8d52df66868fb7f27f671c0f3029d2afd381fdc1f"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"转换为近端残差误差界","source_units":[{"start":52929,"end":53266,"start_line":759,"end_line":763,"raw_tex":"By utilizing the equivalence between proximal and subdifferential error bounds \\citep[Theorem 3.4]{drusvyatskiy2018error}, we  conclude that\n\\begin{equation}\n\\label{eq:;lip_3}\n    \\|x-x_r(y^k,z^k)\\|  \\leq \\frac{2(r-L)^{-1}+t}{t} \\left\\|x - \\prox_{t  \\hat{F}_{y^k,z^k}}(x)\\right\\| \\text{ for all } t>0, \\, x\\in\\mathcal{X}.\n\\end{equation}\n","sha256_utf8":"84ce3756db3b0a1a3ee4f483a9d760109fa328173ced2ea74b84a86ebdaf0e63"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"构造辅助函数并比较模型","source_units":[{"start":53266,"end":54795,"start_line":764,"end_line":779,"raw_tex":"Now, let us examine the relationship between  $\\|x^{k+1}-x^k\\|$ and $\\|x^{k+1} - \\prox_{t \\hat{F}_{y^k,z^k}}(x^{k+1})\\|$. Let $\\varphi_k:\\R^n\\rightarrow \\R \\cup \\{+\\infty\\}$ be the function defined by\n\\[\n\\varphi_k(x):=\\hat{F}_{y^k,z^k}(x)+\\frac{\\lambda^{-1}+L}{2}\\|x-x^k\\|^{2}-\\frac{\\lambda^{-1} + L}{2}\\|x-x^{k+1}\\|^{2}.\n\\]\n% By combining Fact \\ref{fact-2} and the strong convexity of $F_{x^k,\\lambda}(\\cdot,y^k)$\nIt is clear that for any $x\\in\\mathcal{X}$,\n\\begin{align*}\\label{safeguard-key1}\n            & \\hat{F}_{y^k,z^k}(x)\n             = F(x,y^k) + \\frac{r}{2}\\|x-z^k\\|^2 \\notag\\\\\n             \\ge\\ & F_{x^k,\\lambda}(x,y^k) -\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k}\\|^2 + \\frac{r}{2}\\|x-z^k\\|^2\\notag \\\\ \n              \\ge\\ &  F_{x^k,\\lambda}(x^{k+1},y^k) +\\frac{\\lambda^{-1}+r}{2}\\|x-x^{k+1}\\|^2 -\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k}\\|^2 +\\frac{r}{2}\\|x^{k+1}-z^k\\|^2\\notag\\\\\n             \\ge\\ & F(x^{k+1},y^k) + \\frac{\\lambda^{-1}-L}{2}\\|x^{k}-x^{k+1}\\|^2 +\\frac{\\lambda^{-1}+r}{2}\\|x-x^{k+1}\\|^2\\ \\\\\n             & \\ -\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k}\\|^2 + \\frac{r}{2}\\|x^{k+1}-z^k\\|^2 \\notag \\\\ \n             =\\ & \\hat{F}_{y^k,z^k}(x^{k+1})+\\frac{\\lambda^{-1}-L}{2}\\|x^{k+1}-x^k\\|^2 +\\frac{\\lambda^{-1}+r}{2}\\|x-x^{k+1}\\|^2 -\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k}\\|^2,\n\\end{align*}\nwhere the first and third inequalities follow from Fact \\ref{fact-2}, and the second one is from the $(\\lambda^{-1} + r)$-strong convexity of $F_{x^k,\\lambda}(\\cdot,y^k) +\\frac{r}{2}\\|\\cdot-z^k\\|^2$ on $\\mathcal{X}$ and the definition of $x^{k+1}$.\n","sha256_utf8":"448c19c6289b3000f6913647b48e2766455f8ee7441441f2ed70185af1ac5f89"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"辅助函数次优性估计","source_units":[{"start":54795,"end":55213,"start_line":780,"end_line":787,"raw_tex":"Since $r > L$, we see that for any $x\\in\\mathcal{X}$,\n\\begin{align*}\n\\varphi_k(x)\n&\\ge \\hat{F}_{y^k,z^k}(x^{k+1})+\\frac{\\lambda^{-1}-L}{2}\\|x^{k+1}-x^{k}\\|^2+\\frac{r-L}{2}\\|x-x^{k+1}\\|^2\\notag\\\\\n&\\ge\\hat{F}_{y^k,z^k}(x^{k+1})+\\frac{\\lambda^{-1}-L}{2}\\|x^{k+1}-x^{k}\\|^2.\n\\end{align*}\nIt then follows from the definition of $\\varphi_k$ that \n\\[\\varphi_k(x^{k+1})-\\inf_{x\\in\\R^n} \\varphi_k(x)\\leq L \\|x^{k+1}-x^k\\|^2.\\]\n","sha256_utf8":"431875adc6650014bae2571a83d25b3d5d3deaa521107dc6382a74da462fd93d"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"辅助近端残差受间隙控制","source_units":[{"start":55213,"end":55893,"start_line":788,"end_line":797,"raw_tex":"This, together with the definition of the proximal mapping, implies that for any $\\rho>0$,\n\\begin{align}\n&\\varphi_k\\left(\\operatorname{prox}_{\\frac{1}{\\rho} \\varphi_k}(x^{k+1})\\right)+\\frac{\\rho}{2}\\|\\operatorname{prox}_{\\frac{1}{\\rho} \\varphi_k}(x^{k+1})-x^{k+1}\\|^{2}\\leq  \\varphi_k(x^{k+1})\\notag\\\\\n\\leq\\ & \\inf_{x\\in\\R^n} \\varphi_k(x)+L \\|x^{k+1}-x^k\\|^2\n\\leq \\varphi_k\\left(\\operatorname{prox}_{\\frac{1}{\\rho} \\varphi_k}(x^{k+1})\\right)+L \\|x^{k+1}-x^k\\|^2,\\notag\n\\end{align}\nwhich in turn implies that\n\\begin{equation}\\label{safeguard-key111}\n\\|\\operatorname{prox}_{\\frac{1}{\\rho} \\varphi_k}(x^{k+1})-x^{k+1}\\|\\leq \\sqrt{\\frac{2L}{\\rho}} \\|x^{k+1}-x^k\\|.    \n\\end{equation}\n","sha256_utf8":"b307d40823e88f0c1dd0f78f1c81598ed995da664779ea305234e9c1f2db93a3"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"关联近端映射并得到原始界","source_units":[{"start":55893,"end":57479,"start_line":798,"end_line":810,"raw_tex":"Recall that our goal is to find a relationship between  $\\|x^{k+1}-x^k\\|$ and $\\|x^{k+1} - \\prox_{t \\hat{F}_{y^k,z^k}}(x^{k+1})\\|$. To do this, it suffices to relate $\\prox_{t \\hat{F}_{y^k,z^k}}(x^{k+1})$ to $\\operatorname{prox}_{\\frac{1}{\\rho} \\varphi_k}(x^{k+1})$.  By definition, we have\n\\[\\hat{F}_{y^k,z^k}(x)+\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k}\\|^{2}=\\varphi_k(x)+\\frac{\\lambda^{-1}+L}{2}\\|x-x^{k+1}\\|^{2},\\]\nwhich implies that $\\operatorname{prox}_{\\frac{1}{\\lambda^{-1}+L} \\varphi_k}(x^{k+1})=\\operatorname{prox}_{\\frac{1 }{\\lambda^{-1}+L}\\hat{F}_{y^k,z^k}}(x^k)$. \nBy setting $t = (\\lambda^{-1} + L)^{-1}$ in \\eqref{eq:;lip_3} and $\\rho = \\lambda^{-1} + L$ in \\eqref{safeguard-key111}, we conclude that\n\\begin{align*}\n    &\\|x^{k+1}-x_r(y^{k}, z^{k})\\| \\notag\\\\\n    \\leq\\ & \\frac{2(r-L)^{-1}+(\\lambda^{-1}+L) ^{-1}}{(\\lambda^{-1}+L) ^{-1}}\\left\\|x^{k+1}- \\prox_{\\frac{1}{\\lambda^{-1}+L} \\hat{F}_{y^k,z^k}}(x^{k+1})\\right\\| \\\\\n    \\leq\\ &   \\frac{2(r-L)^{-1}+(\\lambda^{-1}+L) ^{-1}}{(\\lambda^{-1}+L) ^{-1}}\\left(\\left\\|x^{k+1}- \\prox_{\\frac{1}{\\lambda^{-1}+L} \\hat{F}_{y^k,z^k}}(x^{k})\\right\\|+ \\|x^{k+1}-x^k\\|\\right) \\\\\n     =\\ &   \\frac{2(r-L)^{-1}+(\\lambda^{-1}+L) ^{-1}}{(\\lambda^{-1}+L) ^{-1}}\\left(\\left\\|x^{k+1}- \\prox_{\\frac{1}{\\lambda^{-1}+L} \\varphi_k}(x^{k+1})\\right\\|+ \\|x^{k+1}-x^k\\|\\right) \\\\\n    \\leq\\ & \\frac{2(r-L)^{-1}+(\\lambda^{-1}+L) ^{-1}}{(\\lambda^{-1}+L) ^{-1}} \\left(\\sqrt{\\frac{2L}{\\lambda^{-1}+L}}+1\\right) \\|x^{k+1}-x^k\\|,\n\\end{align*}\nwhere the second inequality is due to the nonexpansiveness of the proximal mapping. The proof is complete.\n\\end{proof}","sha256_utf8":"0d2ad55d8e5288877c0dd28a4c0964cc1283ad52212e1669ede9dfd6368ecf22"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"势函数差与强凸增长","source_units":[{"start":136993,"end":137593,"start_line":1773,"end_line":1782,"raw_tex":"\\begin{proof}\nFrom the definition of $F_r$, we know that for any $x\\in\\mathcal{X}$, $y\\in\\mathcal{Y}$, and $z,z'\\in\\R^n$, \n\\begin{equation}\\label{func-cal}\nF_r(x,y,z^{\\prime})-F_r(x,y,z)=\\frac{r}{2}(\\|x-z'\\|^2-\\|x-z\\|^2)=\\frac{r}{2}(\\|z^{\\prime}\\|^{2}-2(z^{\\prime}-z)^\\top x-\\|z\\|^{2}).\n\\end{equation}\nFact \\ref{fact-1} implies that $F(\\cdot, y)$ is $L$-weakly convex for any $y \\in \\mathcal{Y}$. Therefore, for any $x\\in\\mathcal{X}$, $y\\in\\mathcal{Y}$, and $z\\in\\R^n$, we have\n\\begin{equation}\\label{psi-strongcvx}\nF_r(x,y, z)- F_r(x_r(y,z),y,z)\n\\geq \\frac{r-L}{2}\\|x-x_r(y,z)\\|^{2}.\n\\end{equation}\n","sha256_utf8":"0c59e350295cc936757230cc52b27f2d02332e197ea7e47cf57691dada1bc653"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"比较不同平滑中心","source_units":[{"start":137593,"end":138315,"start_line":1783,"end_line":1791,"raw_tex":"Combining \\eqref{func-cal} and \\eqref{psi-strongcvx} yields\n\\begin{align}\\label{lip-key1}\n    &F_r(x_r(y,z),y,z^{\\prime})-F_r(x_r(y,z^{\\prime}),y,z^{\\prime}) \\notag\\\\\n    =\\ &F_r(x_r(y,z),y,z^{\\prime})-F_r(x_r(y,z),y,z)+F_r(x_r(y,z),y,z)-F_r(x_r(y,z^{\\prime}),y,z)\\, \\notag\\\\\n    &-(F_r(x_r(y,z^{\\prime}),y, z^{\\prime})-F_r(x_r(y,z^{\\prime}),y, z)) \\notag\\\\\n    \\leq\\ &\\frac{r}{2}(\\|z^{\\prime}\\|^{2}-2(z^{\\prime}-z)^\\top x_r(y,z)-\\|z\\|^{2})-\\frac{r-L}{2}\\|x_r(y,z)-x_r(y,z^{\\prime})\\|^{2}\\,  \\notag\\\\\n    &-\\frac{r}{2}(\\|z^{\\prime}\\|^{2}-2(z^{\\prime}-z)^\\top x_r(y,z^{\\prime})-\\|z\\|^{2}) \\notag\\\\\n    \\leq\\ &r(z^{\\prime}-z)^\\top(x_r(y,z^{\\prime})-x_r(y,z))-\\frac{r-L}{2}\\|x_r(y,z)-x_r(y,z^{\\prime})\\|^{2}.\n    \\end{align}\n","sha256_utf8":"5ebc524c6d7c14c2c0aff68eb799e93f717d120a8312ebffaff5e0a8d494ec00"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"两类中心解映射连续性","source_units":[{"start":138315,"end":139061,"start_line":1792,"end_line":1804,"raw_tex":" On the other hand, \\eqref{psi-strongcvx} implies that\n    \\[\n    F_r(x_r(y,z),y, z^{\\prime})-F_r(x_r(y,z^{\\prime}),y,z^{\\prime}) \\geq \\frac{r-L}{2}\\|x_r(y,z)-x_r(y,z^{\\prime})\\|^{2}.\n    \\]\nCombining this inequality with \\eqref{lip-key1}, we get\n    \\[\n    (r-L)\\|x_r(y,z)-x_r(y,z^{\\prime})\\|^{2}\\leq r(z^{\\prime}-z)^\\top (x_r(y,z^{\\prime})-x_r(y,z)).\n    \\]\nUsing the Cauchy-Schwarz inequality, we obtain\n    \\[\n    \\|x_r(y,z)-x_r(y,z^{\\prime})\\| \\leq \\frac{r}{r-L}\\|z-z^{\\prime}\\|.\n    \\]\nTherefore, we conclude that \\eqref{lip-z} holds with $\\sigma_1=\\frac{r}{r-L}$. Since $f=\\max_{y \\in \\mathcal{Y}} F(\\cdot,y)$ is also $L$-weakly convex by  Fact \\ref{fact-1}, we can prove that \\eqref{lip-starz} holds by using a similar argument as above.\n","sha256_utf8":"733457415cc1d094f8bfcbe7009cda3528c77ed61c745bb2ff563fb96648fbab"}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","title":"对偶参数变化的函数值界","source_units":[{"start":139061,"end":140092,"start_line":1805,"end_line":1823,"raw_tex":"    \n\nWe  now proceed to prove \\eqref{lip-y}. Using \\eqref{psi-strongcvx}, we have\n\\begin{align}\n&F_r(x_r(y^{\\prime}, z),y, z)-F_r(x_r(y, z), y,z) \\geq \\frac{r-L}{2}\\|x_r(y, z)-x_r(y^{\\prime}, z)\\|^{2},\\label{psi-strongcvxineq1} \\\\\n&F_r(x_r(y, z), y^{\\prime},z)-F_r(x_r(y^{\\prime}, z), y^{\\prime},z) \\geq \\frac{r-L}{2}\\|x_r(y, z)-x_r(y^{\\prime}, z)\\|^{2}\\label{psi-strongcvxineq2}.\n\\end{align}\nMoreover, by the $L$-Lipschitz continuity of $\\nabla_{y} F_r(x, \\cdot, z)$ for any $x \\in \\mathcal{X}$ and $z \\in \\mathbb{R}^n$, we have\n\\begin{equation}\n\\begin{aligned}\\label{psi-yconcave}\n F_r(x_r(y, z), y^{\\prime},z)-F_r(x_r(y, z), y,z)\\leq &\\langle\\nabla_{y} F_r(x_r(y, z),y,z), y^{\\prime}-y\\rangle +\\frac{L}{2}\\|y-y^{\\prime}\\|^{2}\n\\end{aligned}\n\\end{equation}\nand \n\\begin{equation}\n\\begin{aligned}\\label{psi-ylip}\n F_r(x_r(y^{\\prime}, z), y, z)-F_r(x_r(y^{\\prime}, z), y^{\\prime},z) \\leq & \\langle\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime}, z), y-y^{\\prime}\\rangle + \\frac{L}{2}\\|y-y^{\\prime}\\|^{2}.\n\\end{aligned}\n\\end{equation}\n","sha256_utf8":"987db33011f618c004770c36aea23b52d4691f51b2aedb04ecfa73be2bb1dd8d"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"合并强凸与梯度连续估计","source_units":[{"start":140092,"end":141102,"start_line":1824,"end_line":1841,"raw_tex":"Incorporating \\eqref{psi-strongcvxineq1}--\\eqref{psi-ylip}, we obtain\n\\begin{equation*}\n% \\label{lip-key2}\n\\begin{aligned}\n& (r-L)\\|x_r(y, z)-x_r(y^{\\prime}, z)\\|^{2}  \\\\ \\leq \\ &\\langle\\nabla_{y} F_r(x_r(y, z), y, z)-\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime}, z), y^{\\prime}-y\\rangle +L\\|y-y^{\\prime}\\|^{2}. \n\\end{aligned}\n\\end{equation*}\nSince $\\nabla_y F_r(\\cdot,\\cdot,z) = \\nabla_y F(\\cdot,\\cdot)$ is $L$-Lipschitz for any $z \\in \\mathbb{R}^n$, we have\n\\begin{align*}\n&\\|\\nabla_{y} F_r(x_r(y, z), y, z)-\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime}, z)\\| \\notag\\\\\n\\leq\\ &\\|\\nabla_{y} F_r(x_r(y, z), y, z)-\\nabla_{y}F_r(x_r(y, z), y^{\\prime}, z)\\|  +\\|\\nabla_{y}F_r(x_r(y, z), y^{\\prime}, z)-\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime}, z)\\|\\notag\\\\\n\\leq\\ &L(\\|y-y^{\\prime}\\|+\\|x_r(y^{\\prime}, z)-x_r(y, z)\\|).\\notag\n\\end{align*}\nIt follows that\n\\begin{align*}\n(r-L)\\|x_r(y, z)-x_r(y^{\\prime}, z)\\|^{2} \n\\leq L\\|x_r(y^{\\prime}, z)-x_r(y, z)\\|\\cdot\\|y-y^{\\prime}\\|+2L\\|y-y^{\\prime}\\|^{2}.\n\\end{align*}\n","sha256_utf8":"1f63a5ec4f1f29b9825de4a1d42893cca7d5b8f092e0ef2055fd782b49f14afb"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"求解比值界得到对偶连续性","source_units":[{"start":141102,"end":141766,"start_line":1842,"end_line":1854,"raw_tex":"Let $\\zeta:=\\frac{\\|x_r(y, z)-x_r(y^{\\prime}, z)\\|}{\\|y-y^{\\prime}\\|}$. \nThen, the above inequality gives\n\\begin{align*}\n\\zeta^{2}\n& \\ \\leq \\frac{L}{r-L} \\zeta+\\frac{2L}{(r-L)}\\leq \\frac{1}{2}\\zeta^2+\\frac{L^2}{2(r-L)^2} +\\frac{2L}{(r-L)}\\\\\n& \\ \\leq \\frac{1}{2} \\zeta^{2}+\\frac{L^2+4L(r-L)}{2(r-L)^2}  \\leq \\frac{1}{2} \\zeta^{2}+\\frac{(L+2(r-L))^2}{2(r-L)^2},\n\\end{align*}\nwhere the second inequality holds because $ab\\leq \\frac{1}{2}(a^2+b^2)$ for any $a,b\\in\\R$. Thus, we get\n\\[\n\\|x_r(y, z)-x_r(y^{\\prime}, z)\\| \\leq  \\frac{2r-L}{r-L}\\|y-y^{\\prime}\\|,\n\\]\nwhich shows that \\eqref{lip-y} holds with $\\sigma_{2}=\\frac{2r-L}{r-L}$. The proof is complete.\n\\end{proof}","sha256_utf8":"36d87af7280e124d55aa59dab61730bde8ba4a1df375c47d52843d48dcca577e"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"计算势值函数两类梯度","source_units":[{"start":142761,"end":143537,"start_line":1868,"end_line":1880,"raw_tex":"\\begin{proof}\nSince $F_r(\\cdot,y,z)$ is strongly convex, $F_r(x,\\cdot,z)$ is weakly concave, and $F_r(x,y,\\cdot)$ is strongly convex for any $x \\in \\mathcal{X}$, $y \\in \\mathcal{Y}$, and $z \\in \\mathbb{R}^n$, the function\n    \\[\n    d_r(\\cdot,\\cdot)=\\min_{x\\in\\mathcal{X}} F_r(x,\\cdot,\\cdot)\n    \\] \n    is differentiable on $\\mathcal{Y}\\times \\R^n$ due to \\citep[Theorem 10.31]{rockafellar2009variational}. In particular, for any $y\\in\\mathcal{Y}$ and $z\\in\\R^n$, one has\n    \\[\n    \\nabla_{y} d_r(y, z)=\\nabla_{y} F_r(x_r(y, z),y, z)=\\nabla_{y} F(x_r(y, z), y).\n    \\]\nUsing  \\citep[Lemma 4.3]{drusvyatskiy2019efficiency}, we have \n    \\[\n    \\nabla_{z} d_r(y,z)=\\nabla_{z} F_r(x_r(y, z),y, z)=r(z-\\prox_{\\frac{1}{r}F(\\cdot,y)+\\iota_{\\mathcal{X}}}(z))=r(z-x_r(y,z)).\n    \\]\n","sha256_utf8":"b959b7943bdd5337190399d860bc3f3a8439ea10f7862fbe87b3d7e91f4ae921"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"对偶梯度利普希茨界","source_units":[{"start":143537,"end":144425,"start_line":1881,"end_line":1889,"raw_tex":"    It follows that for  any $y^{\\prime}, y^{\\prime \\prime} \\in \\mathcal{Y}$, \n    \\begin{align*}\n    &\\|\\nabla_{y} d_r(y^{\\prime}, z)-\\nabla_{y} d_r(y^{\\prime \\prime}, z)\\|\\notag\\\\\n    =\\ &\\|\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime},z)-\\nabla_{y} F_r(x_r(y^{\\prime \\prime}, z), y^{\\prime \\prime},z)\\| \\\\\n    \\leq\\ &\\|\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime},z)-\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime \\prime}, z)\\|+\\|\\nabla_{y} F_r(x_r(y^{\\prime}, z), y^{\\prime \\prime},z)-\\nabla_{y} F_r(x_r(y^{\\prime \\prime}, z), y^{\\prime \\prime}, z)\\| \\\\\n    \\leq\\ & L\\|y^{\\prime}-y^{\\prime \\prime}\\|+L\\|x_r(y^{\\prime}, z)-x_r(y^{\\prime \\prime}, z)\\|\\notag\\\\\n    \\leq\\ & L\\|y^{\\prime}-y^{\\prime \\prime}\\|+L \\sigma_{2}\\|y^{\\prime}-y^{\\prime \\prime}\\|\\leq L_{d_r}\\|y^{\\prime}-y^{\\prime \\prime}\\|,\n    \\end{align*}\n    where the third inequality is due to \\eqref{lip-y}. Moreover, we have\n","sha256_utf8":"5f5435164420d732ec9c489213d0a59aadb1bec745264372452528a78b601121"}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","title":"中心梯度利普希茨界","source_units":[{"start":144425,"end":144854,"start_line":1890,"end_line":1896,"raw_tex":"    \\begin{align*}\n    \\|\\nabla_{z} d_r(y,z')-\\nabla_{z} d_r(y,z'')\\|=\\ &r\\|z'-x_r(y,z')-(z''-x_r(y,z''))\\| \\\\\n    \\leq\\ & r\\|z^{\\prime}-z^{\\prime \\prime}\\|+r\\|x_r(y, z')-x_r(y, z'')\\| \\\\\n    \\leq\\ & r\\|z^{\\prime}-z^{\\prime \\prime}\\|+ r\\sigma_{1}\\|z^{\\prime}-z^{\\prime \\prime}\\|\\leq L_{d_r}'\\|z^{\\prime}-z^{\\prime \\prime}\\|,\n    \\end{align*}\nwhere the second inequality is due to \\eqref{lip-z}. The proof is complete.\n\\end{proof}","sha256_utf8":"59c44a7a979056230e18182443b3a3265b000869a43c1aff59609a2d2f357f1c"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"投影非扩张控制对偶步差","source_units":[{"start":145234,"end":146090,"start_line":1909,"end_line":1921,"raw_tex":"\\begin{proof}\nWe compute\n\\begin{equation*}\n% \\label{iter-result2}\n\\begin{aligned}\n\\ &\\|y^{k+1}-y_{+}^{k}(z^{k+1})\\|\\\\ =\\ &\\|\\proj_{\\mathcal{Y}}(y^{k}+\\alpha \\nabla_{y} F(x^{k+1},y^{k}))-\\proj_{\\mathcal{Y}}(y^{k}+\\alpha \\nabla_{y} F(x_r(y^{k}, z^{k+1}),y^{k}))\\|\\notag \\\\\n\\leq\\ &\\|y^{k}+\\alpha \\nabla_{y} F(x^{k+1}, y^{k})-(y^{k}+\\alpha \\nabla_{y} F(x_r(y^{k}, z^{k+1}),  y^{k}))\\|\\notag \\\\\n\\leq\\ & \\alpha L(\\|x^{k+1}-x_r(y^{k}, z^{k})\\|+\\|x_r(y^{k}, z^{k})-x_r(y^{k}, z^{k+1})\\|) \\notag \\\\\n\\leq\\ & \\eta\\|x^{k}-x^{k+1}\\|+\\sigma_1\\alpha L\\|z^{k+1}-z^k\\|,\n\\end{aligned}\n\\end{equation*}\nwhere the first inequality is due to the nonexpansiveness of the projection operator, the second is due to the $L$-Lipschitz continuity of $\\nabla_y F(\\cdot,\\cdot)$, and the third follows from Proposition \\ref{prop:lip} and \\eqref{lip-z}. The proof is complete.\n\\end{proof}","sha256_utf8":"3f8f9a6f994b1bb27efb63c5e2df29f3d22107674fe4603ce472235c53f9ee22"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"原始更新与模型误差下降","source_units":[{"start":146555,"end":147426,"start_line":1934,"end_line":1950,"raw_tex":"\\begin{proof}\nOne can infer from the definition that $F_{x^k,\\lambda}(\\cdot,y^k)$ is $\\lambda^{-1}$-strongly convex. Therefore, we have\n\\begin{equation*}\n\\begin{aligned}\nF_r(x^{k}, y^{k},z^{k}) & =F(x^{k}, y^{k})+ \\frac{r}{2}\\|x^k-z^{k}\\|^2  = F_{x^k,\\lambda}(x^k,y^k) + \\frac{r}{2}\\|x^k-z^{k}\\|^2 \\\\\n& \\ge F_{x^k,\\lambda}(x^{k+1},y^k)+\\frac{r}{2}\\|x^{k+1}-z^{k}\\|^2+\\frac{\\lambda^{-1}+r}{2}\\|x^k-x^{k+1}\\|^2.\n\\end{aligned}\n\\end{equation*}\nMoreover, Fact \\ref{fact-2} implies that\n\\begin{equation*} \nF_{x^k,\\lambda}(x^{k+1},y^k)\\ge F(x^{k+1},y^k) + \\frac{\\lambda^{-1}-L}{2}\\|x^{k+1}-x^k\\|^2.\n\\end{equation*}\nIt follows that\n\\begin{align}\\label{primaldec-key1}\nF_r(x^{k},y^{k}, z^k) &\\ge F(x^{k+1},y^k)+\\frac{r}{2}\\|x^{k+1}-z^k\\|^2+\\frac{2\\lambda^{-1}+r-L}{2}\\|x^k-x^{k+1}\\|^2\\notag\\\\\n&= F_r(x^{k+1},y^{k}, z^{k})+\\frac{2\\lambda^{-1}+r-L}{2}\\|x^k-x^{k+1}\\|^2.\n\\end{align}\n","sha256_utf8":"b929cb3d04981d53f2f19c53f4ad83ba14a6c0a60101735e3d2b5f6ccff190d0"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"对偶变化的光滑函数估计","source_units":[{"start":147426,"end":147753,"start_line":1951,"end_line":1954,"raw_tex":"Next,  as $\\nabla_y F_r(x,\\cdot,z)$ is $L$-Lipschitz continuous for any $x\\in\\mathcal{X}$ and $z\\in\\R^n$, we have \n    \\begin{align}\\label{primaldec-key2}\n        F_r(x^{k+1}, y^{k},z^k)-F_r(x^{k+1}, y^{k+1},z^k) \\ge \\langle\\nabla_{y} F_r(x^{k+1},y^{k}, z^k), y^{k}-y^{k+1}\\rangle-\\frac{L}{2}\\|y^k-y^{k+1}\\|^2.\n    \\end{align}\n","sha256_utf8":"e8ae57f05957daa9e8a327be3f51a232d84ae777d19653c426861e6d24100a08"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"中心更新下降并合并","source_units":[{"start":147753,"end":148122,"start_line":1955,"end_line":1960,"raw_tex":"At last, based on the update $z^{k+1}=z^{k}+\\beta(x^{k+1}-z^{k})$, we get\n\\begin{equation}\\label{primaldec-key3}\nF_r(x^{k+1}, y^{k+1},z^{k})-F_r(x^{k+1},y^{k+1}, z^{k+1})  = \\frac{(2-\\beta)r}{2 \\beta}\\|z^{k}-z^{k+1}\\|^{2}.\n\\end{equation}\nBy summing up \\eqref{primaldec-key1}, \\eqref{primaldec-key2}, and \\eqref{primaldec-key3}, we obtain the desired result.\n\\end{proof}","sha256_utf8":"6688b557b733e8233d03b22e809f7ab0b7e0b4a0ba1446670e67e925f9b0770c"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"对偶梯度给出上升下界","source_units":[{"start":148573,"end":149060,"start_line":1973,"end_line":1979,"raw_tex":"\\begin{proof}\nBased on Lemma \\ref{lemma-dualdiff}, $\\nabla_{y} d_r(\\cdot,z)$ is $L_{d_r}$-Lipschitz continuous for any $z \\in \\mathbb{R}^n$. Thus, we have \n        \\begin{align*}\n        \\ & d_r(y^{k+1},z^k)-d_r(y^{k},z^k)  \n        \\geq \\langle\\nabla_{y} d_r(y^{k},z^k), y^{k+1}-y^{k}\\rangle-\\frac{L_{d_r}}{2}\\|y^{k}-y^{k+1}\\|^{2} \\\\\n        =\\ & \\langle \\nabla_{y} F_r(x_r(y^{k}, z^{k}), y^{k}, z^{k}), y^{k+1}-y^{k}\\rangle-\\frac{L_{d_r}}{2}\\|y^{k}-y^{k+1}\\|^{2}.\n        \\end{align*}\n","sha256_utf8":"fa49d796e39af632ac39a4a31fdc1c48932efa1343c191d2bb751dc4dd12e7de"}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","title":"中心变化贡献并合并","source_units":[{"start":149060,"end":149689,"start_line":1980,"end_line":1989,"raw_tex":"In addition, one has \n        \\begin{align*}\n        &d_r(y^{k+1}, z^{k+1})-d_r(y^{k+1}, z^{k})\\notag\\\\ \n        =\\ & F_r(x_r(y^{k+1}, z^{k+1}), y^{k+1}, z^{k+1})-F_r(x_r(y^{k+1}, z^{k}), y^{k+1},z^{k}) \\\\\n        \\geq\\ &  F_r(x_r(y^{k+1}, z^{k+1}), y^{k+1},z^{k+1})-F_r(x_r(y^{k+1}, z^{k+1}), y^{k+1},z^{k}) \\\\\n        =\\ & \\frac{r}{2}\\|x_r(y^{k+1}, z^{k+1})-z^{k+1}\\|^{2}-\\frac{r}{2}\\|x_r(y^{k+1}, z^{k+1})-z^{k}\\|^{2} \\\\\n        =\\ & \\frac{r}{2} \\left\\langle z^{k+1}-z^{k},z^{k+1}+z^{k}-2 x_r(y^{k+1}, z^{k+1})\\right\\rangle.\n        \\end{align*}\nFinally, by combining the above inequalities, the proof is complete.\n\\end{proof}","sha256_utf8":"087591606191f88c4d5396ea075aa8335b28b5d61411a2261a39b2cdb80495bb"}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","title":"选择近端值函数对偶最优解","source_units":[{"start":150034,"end":150426,"start_line":2000,"end_line":2011,"raw_tex":"\\begin{proof}\n% From Sion's minimax theorem \\citep{sion1958general}, we know that\n%     \\[\n%     \\min _{x \\in \\mathcal{X}} \\max _{y \\in \\mathcal{Y}} F_r(x, y, z)=\\max_{y \\in \\mathcal{Y}} \\min_{x \\in \\mathcal{X}} F_r(x,y,z)\n%     \\]\n%     which implies\nRecall that \n% $p_r(z)$:\n    % \\[\n    $p_r(z)=\\max _{y \\in \\mathcal{Y}} d_r(y, z)$.\n    % \\]\nDue to the definition of $y(z^{k+1})$, we have\n","sha256_utf8":"5cfbb3389a2ab14ab06eacd6c9db324ff207eb961ab10c806bf82865d3c9e191"}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","title":"比较固定对偶下的最小值","source_units":[{"start":150426,"end":150966,"start_line":2012,"end_line":2019,"raw_tex":"    \\begin{align*}\n     &p_r(z^{k+1})-p_r(z^{k})\\notag\\\\\n    \\leq\\ & d_r(y(z^{k+1}), z^{k+1})-d_r(y(z^{k+1}), z^{k}) \\\\\n    \\leq\\ & F_r(x_r(y(z^{k+1}), z^{k}), y(z^{k+1}), z^{k+1})-F_r(x_r(y(z^{k+1}), z^{k}), y(z^{k+1}),z^{k}) \\\\\n    =\\ & \\frac{r}{2}\\left\\langle z^{k+1}-z^{k},z^{k+1}+z^{k}-2 x_r(y(z^{k+1}), z^{k})\\right\\rangle,\n    \\end{align*}\n    where the second inequality follows from the fact that \\[F_r(x', y, z)\\geq \\min_{x\\in\\mathcal{X}} F_r(x,y,z) = d_r(y, z)\\] for any $x'\\in\\mathcal{X}$. The proof is complete.\n    \\end{proof}","sha256_utf8":"dee85cee889e8e4445049c8136bc3bae65bb2f249387e065c9610b42aa3a039f"}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","title":"合并三类势函数变化","source_units":[{"start":150967,"end":151920,"start_line":2020,"end_line":2031,"raw_tex":"\\paragraph{{\\bf Proof of Proposition \\ref{prop:decrease}}}\n% \\begin{proof}\nFrom Lemmas \\ref{lemma-F-des}, \\ref{lemma-d-aes}, and \\ref{lemma-p-des}, we have\n        \\begin{align*}\n        % \\label{desascent-key1}\n        & \\Phi_r(x^k,y^k,z^k)-\\Phi_r(x^{k+1},y^{k+1},z^{k+1}) \\notag\\\\\n         =\\ & F_r(x^k,y^k,z^k)-F_r(x^{k+1},y^{k+1},z^{k+1}) +2(d_r(y^{k+1},z^{k+1})-d_r(y^k,z^k))+2(p_r(z^k)-p_r(z^{k+1})) \\notag\\\\\n         \\ge\\ & \\frac{2\\lambda^{-1}+r-L}{2}\\|x^k-x^{k+1}\\|^2 +\\frac{(2-\\beta)r}{2 \\beta}\\|z^{k}-z^{k+1}\\|^{2}-\\left(L_{d_r}+\\frac{L}{2}\\right)\\|y^{k}-y^{k+1}\\|^{2}\\,  \\notag\\\\\n        &  + \\underbrace{\\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k}-y^{k+1}\\rangle+2\\langle\\nabla_{y} F_r(x_r(y^{k}, z^{k}),  y^{k}, z^{k}), y^{k+1}-y^{k}\\rangle}_{\\text{\\ding{172}}}\\, \n      \\notag \\\\ \n      & + \\underbrace{2 r\\left\\langle z^{k+1}-z^{k},x_r(y(z^{k+1}), z^{k})-x_r(y^{k+1}, z^{k+1})\\right\\rangle}_{\\text{\\ding{173}}}.\n        \\end{align*}\n","sha256_utf8":"079e24e122b535d27ed2d2cfeb1d9f3c0c4966294cc9942f85c8e7a7532d2c0f"}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","title":"展开对偶内积项","source_units":[{"start":151920,"end":152483,"start_line":2032,"end_line":2038,"raw_tex":"% Subsequently, we simplify the terms \\text{\\ding{172}} and  \\text{\\ding{173}}. \nFirst, we simplify the term \\text{\\ding{172}}. We know that\n        \\begin{align*}\n        \\text{\\ding{172}} \n         =\\ & \\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k}-y^{k+1}\\rangle+2\\langle\\nabla_{y} F_r(x_r(y^{k}, z^{k}), y^{k}, z^{k}), y^{k+1}-y^{k}\\rangle\\\\\n        =\\ & \\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k+1}-y^{k}\\rangle + 2\\langle\\nabla_{y} F_r(x_r(y^{k}, z^{k}), y^{k}, z^{k})-\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k+1}-y^{k}\\rangle.\n        \\end{align*}\n","sha256_utf8":"9243eb50cfaeb5b66bb7615e60dd856749bd15c1d78bf965c52ab3d6ec91104f"}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","title":"投影最优性给出正贡献","source_units":[{"start":152483,"end":153332,"start_line":2039,"end_line":2048,"raw_tex":"        For the first term, we have\n        \\begin{align*}\n    &\\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k+1}-y^{k}\\rangle \\notag\\\\\n     =\\ &  \\langle\\nabla_{y} F_r(x^{k+1}, y^{k},z^k)+\\frac{1}{\\alpha}(y^{k}-y^{k+1}), y^{k+1}-y^{k}\\rangle + \\frac{1}{\\alpha}\\|y^{k}-y^{k+1}\\|^2\\\\\n     =\\ & \\frac{1}{\\alpha}\\langle y^{k} + \\alpha \\nabla_{y} F_r(x^{k+1}, y^{k},z^k)-y^{k+1}, y^{k+1}-y^{k}\\rangle +\\frac{1}{\\alpha}\\|y^{k}-y^{k+1}\\|^2\\\\\n     \\ge\\ &  \\frac{1}{\\alpha}\\|y^{k}-y^{k+1}\\|^2, \n\\end{align*}\nwhere the last inequality follows from the property of the projection operator and the dual update $\ny^{k+1} = \\proj_{\\mathcal{Y}}( y^{k} + \\alpha \\nabla_{y} F_r(x^{k+1}, y^{k},z^k))\n$ (recall that $\\nabla_y F_r(x,y,z) = \\nabla_y F(x,y)$ for any $x \\in \\mathcal{X}$, $y \\in \\mathcal{Y}$, and $z \\in \\mathbb{R}^n$). For the remaining terms, we have\n","sha256_utf8":"e83e44d4cdf30d27cb4e0ae7b123a85894ebfd88dd77a24b367eadd1159721eb"}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","title":"原始误差界控制梯度扰动","source_units":[{"start":153332,"end":154764,"start_line":2049,"end_line":2067,"raw_tex":"                \\begin{align*}\n        %\\label{desascent-key3}\n         &2\\langle\\nabla_{y} F_r(x_r(y^{k}, z^{k}), y^{k}, z^{k})-\\nabla_{y} F_r(x^{k+1}, y^{k},z^k), y^{k+1}-y^{k}\\rangle \\notag\\\\\n         \\ge & -2\\|\\nabla_{y} F_r(x_r(y^{k}, z^{k}),y^{k},z^{k})-\\nabla_{y} F_r(x^{k+1}, y^{k},z^{k})\\| \\cdot\\|y^{k+1}-y^{k}\\| \\notag\\\\\n         \\ge & -2 L\\|x^{k+1}-x_r(y^{k}, z^{k})\\| \\cdot\\|y^{k}-y^{k+1}\\| \\notag\\\\\n        \\ge  & -L \\zeta^2\\|y^{k}-y^{k+1}\\|^{2}-L\\zeta^{-2} \\|x^{k+1}-x_r(y^{k}, z^{k})\\|^{2} \\notag\\\\\n         \\ge & -L \\zeta^2\\|y^{k}-y^{k+1}\\|^{2}-L\\|x^{k+1}-x^{k}\\|^{2},\n        \\end{align*}\n        where the third inequality \n        % follows from $2|x||y| \\leq \\tau x^2 + \\frac{1}{\\tau}y^2$ \n        holds because $2 |a| |b| \\le \\tau a^2 + \\tfrac{1}{\\tau} b^2$\n        for any $a,b \\in \\mathbb{R}$ and $\\tau>0$, and the last inequality follows from Proposition \\ref{prop:lip}. \n        % where the first inequality is because of the Cauchy-Schwarz in equality, the second inequality is because $\\nabla_{y} K=\\nabla_{y} f$ is $L$-Lipschitz-continuous, the third inequality is due to the AM-GM inequality and the last is because of (B.5).\n% Subsequently, we focus on the left term in $\\text{\\ding{172}}$ that \nPutting the above together, we obtain\n\\begin{equation}\n\\label{desascent-key3}\n    \\text{\\ding{172}}  \\ge \\left(\\frac{1}{\\alpha}-L\\zeta^2\\right) \\|y^{k}-y^{k+1}\\|^{2}-L\\|x^{k+1}-x^{k}\\|^{2}. \n\\end{equation}\n","sha256_utf8":"3a6bc96a89464b7c937eb041b82de817200bb36c78073e74635870d50b68aaea"}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","title":"解映射连续性控制中心内积","source_units":[{"start":154764,"end":155793,"start_line":2068,"end_line":2077,"raw_tex":"Next, we bound the term $\\text{\\ding{173}}$ by \n\\begin{align}\\label{desascent-key2}\n  \\text{\\ding{173}} \n =\\ & 2 r \\left\\langle z^{k+1}-z^{k}, x_r(y(z^{k+1}), z^{k})-x_r(y^{k+1}, z^{k+1})\\right\\rangle \\notag\\\\\n =\\ & 2 r \\left\\langle z^{k+1}-z^{k}, x_r(y(z^{k+1}), z^{k})-x_r(y(z^{k+1}), z^{k+1})\\right\\rangle\\, \\notag\\\\\n & + 2 r\\left\\langle z^{k+1}-z^{k}, x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\right\\rangle\\notag \\\\\n\\ge &  -2r \\sigma_1\\|z^{k+1}-z^k\\|^2+2 r\\left\\langle z^{k+1}-z^{k}, x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\right\\rangle \\notag\\\\\n\\ge & -2r \\sigma_1\\|z^{k+1}-z^k\\|^2-\\frac{r}{7\\beta}\\|z^{k+1}-z^k\\|^2  -7r\\beta \\|x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\|^2, \n\\end{align}\nwhere the first inequality is due to \\eqref{lip-z} and the Cauchy-Schwarz inequality, and the second inequality again follows from the fact that $2 |a| |b| \\le \\tau a^2 + \\tfrac{1}{\\tau} b^2$ for any $a,b \\in \\mathbb{R}$ and $\\tau > 0$. Now, the inequalities \\eqref{desascent-key3} and \\eqref{desascent-key2} imply that\n","sha256_utf8":"4597b112ff3b359cbb7feb1a3f69ac1eb4c0b651e96d93414622daaf87878577"}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","title":"得到中间下降估计","source_units":[{"start":155793,"end":156230,"start_line":2078,"end_line":2082,"raw_tex":"\\begin{align}\\label{desas-keymid}\n        & \\Phi_r(x^k,y^k,z^k)-\\Phi_r(x^{k+1},y^{k+1},z^{k+1}) \\notag\\\\\n        \\geq\\ & \\frac{2\\lambda^{-1}+r-3L}{2}\\|x^{k}-x^{k+1}\\|^{2}+\\left(\\frac{1}{ \\alpha}-L_{d_r}-\\frac{L}{2}-L\\zeta^2\\right)\\|y^{k}-y^{k+1}\\|^{2}\\, \\notag\\\\\n        &+\\left(\\frac{(2-\\beta)r}{2 \\beta}-2r\\sigma_1-\\frac{r}{7\\beta}\\right)\\|z^{k}-z^{k+1}\\|^{2}-7r\\beta \\|x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\|^2. \n\\end{align}\n","sha256_utf8":"9c15c72e8d165cec2362565cdbc02db244dd06663ad94230bcf15ed48c35ca58"}]},{"id":"S080","semantic_type":"proof_step","kind":"proof_step","title":"用理想对偶间隙替换实际间隙","source_units":[{"start":156230,"end":157036,"start_line":2083,"end_line":2096,"raw_tex":"By Lemma \\ref{lemma-dualxy} and the fact that $\\| u+v \\|^2 \\le 2(\\|u\\|^2+\\|v\\|^2)$ for any $u,v \\in \\mathbb{R}^d$, we have\n\\begin{align}\\label{desas-final-key1}\n\\|y^{k+1}-y^{k}\\|^{2} \n&=\\|y^{k+1}-y_{+}^{k}(z^{k+1})+y_{+}^{k}(z^{k+1})-y^{k}\\|^{2}\\notag\\\\\n&\\geq\\frac{1}{2}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2} -\\|y^{k+1}-y_{+}^{k}(z^{k+1})\\|^{2}\\notag\\\\\n% & \\geq \\frac{1}{2}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2} -\\alpha^2L^2\\|x^{k+1}-x_r(y^k,z^{k+1})\\|^2\\notag\\\\\n& \\geq \\frac{1}{2}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2} -2\\eta^2\\|x^{k+1}-x^k\\|^2-2\\sigma_1^2\\alpha^2L^2\\|z^{k+1}-z^k\\|^2.\n\\end{align}\n% where we recall that \n% \\[\n% y_{+}^{k}(z^{k+1}) = \\proj_{\\mathcal{Y}}(y^k+\\alpha \\nabla_y F_r (x_r(y^k,z^{k+1}),y^k,z^{k+1})).\n% \\]\n% \\leq\\ & \\alpha L\\|x^{k+1}-x_r(y^{k}, z^{k})\\| \\notag \\\\\n% \\leq\\ & \\eta\\|x^{k}-x^{k+1}\\|,\n","sha256_utf8":"a89a87b86f90c6222415657899de3f507db0075fbea89fdbe83bdd0957122714"}]},{"id":"S081","semantic_type":"proof_step","kind":"proof_step","title":"分解对偶解映射误差","source_units":[{"start":157036,"end":158513,"start_line":2097,"end_line":2115,"raw_tex":"Similarly, by Lemma \\ref{lemma-sollip} and Lemma \\ref{lemma-dualxy}, we have\n\\begin{align}\\label{desas-final-key2}\n&\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\|^{2}\\notag\\\\\n\\leq \\ & 2\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_+^{k}(z^{k+1}),z^{k+1})\\|^2   + 2\\|x_r(y_+^{k}(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\|^2 \\notag\\\\\n\\leq \\ & 2\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_+^{k}(z^{k+1}),z^{k+1})\\|^2+2\\sigma_2^2 \\|y^{k+1}-y_{+}^{k}(z^{k+1})\\|^{2}\\notag\\\\\n\\leq \\ & 2\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_+^{k}(z^{k+1}),z^{k+1})\\|^2 + 4\\sigma_2^2\\eta^2 \\|x^{k+1}-x^k\\|^2+4\\sigma_2^2\\sigma_1^2\\alpha^2L^2\\|z^{k+1}-z^k\\|^2.\n\\end{align}\n\n% \\begin{align}\\label{desas-final-key2}\n% &\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y^{k+1}, z^{k+1})\\|^{2}\\notag\\\\\n% % =\\ & \\|x_r^\\star(z^{k+1})-x_r^\\star(z^{k})+x_r^\\star(z^{k})-x_r(y_{+}^{k}(z^{k}), z^{k})+x_r(y_{+}^{k}(z^{k}), z^{k})-x_r(y^{k+1}, z^{k})+x_r(y^{k+1}, z^{k})-x_r(y^{k+1}, z^{k+1})\\|^2 \\notag\\\\\n% \\leq\\ & 4\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y(z^{k}), z^{k})\\|^{2}+4\\|x_r(y(z^{k}), z^{k})-x_r(y_{+}^{k}(z^{k}), z^{k})\\|^{2}\\ +\\notag \\\\\n% &4\\|x_r(y_{+}^{k}(z^{k}), z^{k})-x_r(y^{k+1}, z^{k})\\|^{2}+4\\|x_r(y^{k+1}, z^{k})-x_r(y^{k+1}, z^{k+1})\\|^{2} \\notag\\\\\n% \\leq\\ & 8 \\sigma_{1}^{2}\\|z^{k}-z^{k+1}\\|^{2}+4\\|x_r(y(z^{k}), z^{k})-x_r(y_{+}^{k}(z^{k}), z^{k})\\|^{2} +4 \\sigma_{2}^{2} \\eta^2\\|x^{k}-x^{k+1}\\|^2.\n% \\end{align}\n% Besides, we recall that \n% \\begin{align}\\label{desas-final-key0}\n% x^{k+1}-z^{k} =\\frac{1}{\\beta}(z^{k+1}-z^k).\n% \\end{align} \n","sha256_utf8":"4f46accb4c81be5283bb2afdddec4709ac6b86eec631ed055ceaa532953896ba"}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","title":"代入两类间隙替换界","source_units":[{"start":158513,"end":159224,"start_line":2116,"end_line":2122,"raw_tex":"Substituting \\eqref{desas-final-key1} and \\eqref{desas-final-key2} into \\eqref{desas-keymid} yields\n\\begin{align*}\n&\\Phi_r(x^k,y^k,z^k)-\\Phi_r(x^{k+1},y^{k+1},z^{k+1})\\notag\\\\\n \\geq\\ &\\left( \\frac{2\\lambda^{-1}+r-3L}{2} -28 r\\beta \\sigma_2^2\\eta^2 \\right)\\|x^{k}-x^{k+1}\\|^{2}\\ +\\left(\\frac{1}{ \\alpha}-L_{d_r}-\\frac{L}{2}-L\\zeta^2\\right) \\cdot \\\\\n& \\left(\\frac{1}{2}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2} -2\\eta^2\\|x^{k+1}-x^k\\|^2-2\\sigma_1^2\\alpha^2L^2\\|z^{k+1}-z^k\\|^2 \\right)\\ \\notag\\\\\n&+ \\left(\\frac{(2-\\beta)r}{2 \\beta}-2r\\sigma_1-\\frac{r}{7\\beta}-28r\\beta\\sigma_1^2\\sigma_2^2 \\alpha^2 L^2 \\right)\\|z^{k}-z^{k+1}\\|^{2} \\\\  & -  14r\\beta\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_+^{k}(z^{k+1}),z^{k+1})\\|^2.\n\\end{align*}\n","sha256_utf8":"cccd0d4a95a1cb59385f0174900815280cc2546090e506257b7e562784665005"}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","title":"步长保证对偶系数为正","source_units":[{"start":159224,"end":159604,"start_line":2123,"end_line":2131,"raw_tex":"\nSuppose that $r\\ge 3L$,\n% (which implies $L_{d_{r}}+\\frac{L}{2} \\leq 5L$)\nwhich implies that $L_{d_{r}}+\\frac{L}{2} \\leq 5L$. We observe the following:\n\\begin{itemize}\n\\item As $\\alpha \\leq \\min\\left\\{\\frac{1}{10L}, \\frac{1}{4L\\zeta^2} \\right\\}$, we have $\\frac{1}{ \\alpha}-L_{d_r}-\\frac{L}{2} \\ge \\frac{1}{2\\alpha}$ and \n$\n\\frac{1}{2\\alpha} - L\\zeta^2 \\ge \\frac{1}{4\\alpha}. \n$\n","sha256_utf8":"08cceffe9b6b2b1711cdadad91627c06dc3d7b089759a2cc2f5a94f02a5613c8"}]},{"id":"S084","semantic_type":"proof_step","kind":"proof_step","title":"估计中心间隙系数","source_units":[{"start":159604,"end":160118,"start_line":2132,"end_line":2137,"raw_tex":"\\item As $\\beta\\leq\\frac{1}{28}$ and $\\sigma_1 \\leq \\frac{3}{2}$, $\\sigma_2\\leq \\frac{5}{2}$, we have \n\\begin{align*}\n& \\frac{(2-\\beta)r}{2 \\beta}-2r\\sigma_1-\\frac{r}{7\\beta}-28r\\beta\\sigma_1^2 \\sigma_2^2 \\alpha^2 L^2  -\\frac{1}{2\\alpha} \\sigma_1^2\\alpha^2L^2 \\\\ \n\\ge\\ & \\frac{6r}{7\\beta} -\\frac{7r}{2}- 28r\\beta\\left(\\frac{3}{2}\\right)^2\\left(\\frac{5}{2}\\right)^2\\left(\\frac{1}{10L}\\right)^2L^2-\\frac{r}{24}\\\\ \n\\ge\\ &  \\frac{r}{\\beta}\\left(\\frac{6}{7}-4\\beta- 14\\beta^2\\right) \\ge \\frac{4r}{7\\beta}.\n\\end{align*}\n","sha256_utf8":"645a8f7fce53bf84dd56cb9d3343e32b7bd658487021c02294282201940d33ae"}]},{"id":"S085","semantic_type":"proof_step","kind":"proof_step","title":"估计原始间隙系数","source_units":[{"start":160118,"end":160777,"start_line":2138,"end_line":2147,"raw_tex":"\\item As $\\lambda^{-1} \\ge L$ and recalling that $\\eta = \\alpha L \\zeta$, we have \n\\[\\alpha\\leq\\frac{1}{4L \\zeta^2} = \\frac{\\lambda^{-1}}{4L^2\\zeta^2} \\frac{L}{\\lambda^{-1}} \\leq \\frac{\\lambda^{-1}}{4L^2\\zeta^2}\\quad \\text{and}\\quad \n\\frac{\\eta^2}{2\\alpha} = \\frac{\\alpha L^2\\zeta^2}{2} \\leq \\frac{1 }{8\\lambda}.\n\\]\nMoreover, since $\\beta  \\leq \\tfrac{(r-L)^2}{14 \\alpha r (2r-L)^2} = \\tfrac{1}{14 \\alpha r \\sigma_2^2}$,\n we have \n \\[\n 28 r\\beta \\sigma_2^2\\eta^2 \\leq \\frac{ 2\\eta^2}{\\alpha} \\leq \\frac{1}{2\\lambda}\\quad\\text{and}\\quad \\frac{2\\lambda^{-1}+r-3L}{2} -28 r\\beta \\sigma_2^2\\eta^2 - \\frac{\\eta^2}{2\\alpha} \\ge \\frac{3}{8\\lambda}.\n\\]\n\\end{itemize}\n","sha256_utf8":"bfbed0f52645f91d902af5c19309b091052265db8a32c92e3bc44ca51139bd89"}]},{"id":"S086","semantic_type":"proof_step","kind":"proof_step","title":"得到基本下降不等式","source_units":[{"start":160777,"end":161183,"start_line":2148,"end_line":2155,"raw_tex":"% Together all pieces, we get \nPutting everything together yields\n\\begin{align*}\n&\\Phi_r(x^k,y^k,z^k)-\\Phi_r(x^{k+1},y^{k+1},z^{k+1}) \\\\\n\\ge\\ & \\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{8\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2} +\\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2} \\\\  & -14r\\beta\\|x_r(y(z^{k+1}), z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2}.\n\\end{align*}\nThe proof is complete.\n% \\end{proof}\n","sha256_utf8":"de0b06ba93dc7c4838b06c7f6993ea39bf10b1e65a750431b3b7de88fb674328"}]},{"id":"S087","semantic_type":"proof_step","kind":"proof_step","title":"固定对偶最优解并利用强凸性","source_units":[{"start":60091,"end":60675,"start_line":837,"end_line":851,"raw_tex":"\\begin{proof}\nLet $\\psi:\\R^n\\times \\R^n\\rightarrow\\R$ be the function defined by\n\\[\n\\psi(x,z) =  F_r(x,y(z),z). \n\\]\nConsider arbitrary $x \\in \\mathcal{X}$, $y \\in \\mathcal{Y}$, and $z \\in \\mathbb{R}^n$. Note that the function $\\psi(\\cdot,z)$ is $(r-L)$-strongly convex. Since\n\\begin{align*}\n\\mathop{\\argmin}_{x'\\in\\mathcal{X}}\\psi(x',z) & =\\mathop{\\argmin}_{x'\\in\\mathcal{X}} F_r(x',y(z),z) =  x_r(y(z),z),\n\\end{align*}\nwe see that \n% for any $x\\in \\mathcal{X}$, \n\\begin{equation}\n\\label{eq:dual1}\n\\psi(x,z) - \\psi(x_r(y(z),z),z) \\ge \\frac{r-L}{2}\\|x-x_r(y(z),z)\\|^2. \n\\end{equation}\n","sha256_utf8":"6ac6ee9dd4cb47a0d128f507337f946cc09385d41bb92f3f509de0089dde4a5d"}]},{"id":"S088","semantic_type":"proof_step","kind":"proof_step","title":"比较对偶势值与最大函数值","source_units":[{"start":60675,"end":61562,"start_line":852,"end_line":868,"raw_tex":"In addition, we have\n\\begin{align} \n\\label{eq:dual2}\n&\\psi(x,z) - \\psi(x_r(y(z),z),z)\\notag\\\\   \n\\leq \\  &\\psi(x,z)-F_r(x_r(y_+(z),z),y_+(z),z)\\notag \\\\\n\\leq \\ & \\max_{y^\\prime\\in \\mathcal{Y}} F(x,y^\\prime) + \\frac{r}{2}\\|x-z\\|^2-F_r(x_r(y_+(z),z),y_+(z),z)\\notag\\\\\n=\\,& \\max_{y^\\prime\\in \\mathcal{Y}}F(x,y^\\prime)-F(x_r(y_+(z),z),y_+(z)) + \\frac{r}{2}\\|x-z\\|^2-\\frac{r}{2}\\|x_r(y_+(z),z)-z\\|^2,\n\\end{align}\nwhere the first inequality follows from\n\\begin{align*}\nF_r(x_r(y_+(z),z),y_+(z),z) \n=\\ & \\min_{x'\\in\\mathcal{X}}\\left\\{ F(x',y_+(z)) +\\frac{r}{2}\\|x'-z\\|^2\\right\\} \\nonumber \\\\\n\\leq\\ & \\max_{y' \\in \\mathcal{Y}}\\min_{x'\\in\\mathcal{X}}\\left\\{F(x',y') +\\frac{r}{2}\\|x'-z\\|^2\\right\\} \\nonumber \\\\\n=\\ & \\min_{x'\\in\\mathcal{X}}\\left\\{F(x',y(z)) +\\frac{r}{2}\\|x'-z\\|^2\\right\\} \\nonumber \\\\\n=\\ & \\min_{x'\\in\\mathcal{X}} \\psi(x',z)\n= \\psi(x_r(y(z),z),z). % \\label{eq:Fr-one}\n\\end{align*}\n","sha256_utf8":"f2f8d44f6aeb7cf3af3ace07756d28f7a91b5a7f806732dbc787eec9122b118c"}]},{"id":"S089","semantic_type":"proof_step","kind":"proof_step","title":"以对偶次优性控制解距离","source_units":[{"start":61562,"end":62293,"start_line":869,"end_line":879,"raw_tex":"% \\Jiajin{We may have to double-check the above derivation. --- confirmed. With the uniform KL property, we can confirm  $x(y(z),z)$ is unique over $Y(z)$.}\nAs \\eqref{eq:dual1} and \\eqref{eq:dual2} hold for any $x\\in\\mathcal{X}$, we obtain the intermediate relation\n\\begin{equation}\\label{OS-GS-better-key3-main} \n% \\begin{aligned}\n     \\frac{r-L}{2}\\|x_r(y(z),z)-x_r(y_+(z),z)\\|^2\\leq  \\max\\limits_{y^\\prime\\in \\mathcal{Y}} F(x_r(y_+(z),z),y^\\prime) - F(x_r(y_+(z),z),y_+(z)) \n% \\end{aligned}\n\\end{equation}\nby taking $x= x_r(y_+(z),z)$.\n\nNow, we utilize the K{\\L} exponent $\\theta$ of the dual problem to bound the right-hand side of \n\\eqref{OS-GS-better-key3-main}  in terms of $\\|y-y_+(z)\\|$. Consider the following two cases:\n","sha256_utf8":"790bfc8418ab1e555913156f2b46247e0732109408e4f4e04700d00e8de58770"}]},{"id":"S090","semantic_type":"proof_step","kind":"proof_step","title":"零指数下的残差与值差界","source_units":[{"start":62293,"end":63498,"start_line":880,"end_line":897,"raw_tex":" \\begin{enumerate}[label=\\normalfont(\\alph*)]\n \\item Suppose that $\\theta=0$. If $F(x_r(y_+(z),z),y_+(z))=\\max_{y'\\in\\mathcal{Y}} F(x_r(y_+(z),z),y')$, then the desired inequality follows trivially from~\\eqref{OS-GS-better-key3-main}. Otherwise, we have $y_+(z) \\in \\mathcal{Y} \\setminus \\mathcal{Y}^\\star(x_r(y_+(z),z))$. By Assumption~\\ref{ass:kl}, we have\n% we deduce using~\\eqref{eq:Fr-one} that\n% \\begin{align*}\n% F_r(x_r(y_+(z),z),y_+(z),z)\n% &=\\max_{y'\\in\\mathcal{Y}} F_r(x_r(y_+(z),z),y',z)\\\\\n% &\\ge F_r(x_r(y_+(z),z),y_+(z),z)\\\\\n% &\\ge F_r(x_r(y(z),z),y(z),z).\n% \\end{align*}\n%By definition of $y(z)$, it follows that $y_+(z)=y(z)$, and consequently the conclusion trivially holds. \n\\[\n\\dist(0, -\\nabla_y F(x_r(y_+(z),z),y_+(z))+\\partial\\iota_{\\mathcal{Y}}(y_+(z)))\\ge \\mu.\n\\]\nSince  $F(x,\\cdot)$ is continuously differentiable on the compact set $\\mathcal{Y}$, $\\|\\nabla_y F(x,\\cdot)\\|$ is bounded on $\\mathcal{Y}$. Without loss of generality, we can assume that  $\\|\\nabla_y F(x,\\cdot)\\|\\leq L$ on $\\mathcal{Y}$. According to  the mean value theorem, we have \n\\[\n\\max\\limits_{y^\\prime\\in \\mathcal{Y}} F(x_r(y_+(z),z),y^\\prime) - F(x_r(y_+(z),z),y_+(z))\n\\leq L \\cdot {\\rm diam}(\\mathcal{Y}).\n\\]\n","sha256_utf8":"b75581d13b831031e9dfccff56cdcd859ca97e02e6ec9802714d162cc4436e20"}]},{"id":"S091","semantic_type":"proof_step","kind":"proof_step","title":"相对误差控制零指数值差","source_units":[{"start":63498,"end":64613,"start_line":898,"end_line":910,"raw_tex":"Using the fact that $y_+(z) = \\proj_{\\mathcal{Y}} (y + \\alpha \\nabla_y F(x_r(y,z), y))$, we bound\n\\begin{align*}\n &\\max\\limits_{y^\\prime\\in \\mathcal{Y}} F(x_r(y_+(z),z),y^\\prime) - F(x_r(y_+(z),z),y_+(z))\\notag\\\\\n\\leq\\ & \\frac{L \\cdot {\\rm diam}(\\mathcal{Y})}{\\mu^2}\\cdot\\dist^2(0, -\\nabla_y F(x_r(y_+(z),z),y_+(z))+\\partial\\iota_{\\mathcal{Y}}(y_+(z)))\\notag\\\\ \n% \\stackrel{\\text{\\ding{172}}}{\\leq} \\ &   \\frac{L \\cdot {\\rm diam}(\\mathcal{Y})}{\\mu^2} b^2 \\|\\proj_{\\mathcal{Y}}(y+\\alpha\\nabla_y F(x_r(y_+(z),z),y))-y\\|^2\n% \\notag \\\\\n% \\leq \\ &  \\frac{ L b^2 \\cdot {\\rm diam}(\\mathcal{Y})}{\\mu^2}(\n% \\|\\proj_{\\mathcal{Y}}(y+\\alpha\\nabla_y F(x_r(y_+(z),z),y))-y_+(z)\\|\\ +\\|y_+(z)-y\\|)^2 \\\\ \n% \\leq \\ & \\frac{L b^2 \\cdot {\\rm diam}(\\mathcal{Y})}{\\mu^2}(\\alpha L \\|x_r(y_+(z),z)-x_r(y,z)\\| + \\|y-y_+(z)\\|)^2\\\\\n\\leq \\ & \\frac{L b^2 \\cdot {\\rm diam}(\\mathcal{Y})}{\\mu^2}\\|y_+(z)-y\\|^2,\n\\end{align*}\nwhere $b:=\\frac{1}{\\alpha} + L > 0$ and the last inequality is due to the fact that the projected gradient ascent method satisfies the so-called \\textit{relative error condition}\n\\citep[Section 5]{attouch2013convergence}. \n","sha256_utf8":"06905a560ef939d7a289836bf9d307561fed464e5a4c4601698c5e04ba99265b"}]},{"id":"S092","semantic_type":"proof_step","kind":"proof_step","title":"得到零指数对偶误差界","source_units":[{"start":64613,"end":64768,"start_line":911,"end_line":914,"raw_tex":"It follows that\n\\[\n\\|x_r(y(z),z)-x_r(y_+(z),z)\\| \\leq\\frac{\\sqrt{2L\\cdot {\\rm diam}(\\mathcal{Y})}\\cdot (1+\\alpha L)}{\\alpha\\mu\\sqrt{r-L}}\\|y-y_+(z)\\|. \n\\]\n","sha256_utf8":"a9b46d57e8b44e804f29e4e6ab66459fa4f20c79b9c8646e6f2de941583c8400"}]},{"id":"S093","semantic_type":"proof_step","kind":"proof_step","title":"正指数KL条件给出幂次误差界","source_units":[{"start":64768,"end":65950,"start_line":915,"end_line":932,"raw_tex":"\\item If $\\theta \\in (0,1)$, then we have\n\\begin{align*}\n% \\label{KL-key-01}\n& \\mu\\left(\\max\\limits_{y^\\prime\\in \\mathcal{Y}} F(x_r(y_+(z),z),y^\\prime) - F(x_r(y_+(z),z),y_+(z))\\right)^{\\theta}\\notag\\\\\n\\leq \\ & \\dist(0, -\\nabla_y F(x_r(y_+(z),z),y_+(z))+\\partial\\iota_{\\mathcal{Y}}(y_+(z))) \\notag\\\\ \n\\leq \\ & \\dist(0, -\\nabla_y F(x_r(y,z),y_+(z)) +\\partial\\iota_{\\mathcal{Y}}(y_+(z)))   \\notag\\\\\n\\ & + \\|\\nabla_y F(x_r(y,z),y_+(z))-\\nabla_y F(x_r(y_+(z),z),y_+(z))\\|\\notag\\\\ \n{\\leq} \\ & \\dist(0, -\\nabla_y F(x_r(y,z),y_+(z))+\\partial\\iota_{\\mathcal{Y}}(y_+(z)))+ L\\sigma_2\\|y-y_+(z)\\|\\notag\\\\\n{\\leq} \\ & \\left(\\frac{1}{\\alpha}+L+L\\sigma_2\\right)\\|y_+(z)-y\\|,\n\\end{align*}\nwhere the third inequality follows from the $L$-Lipschitz continuity  of $\\nabla_y F(\\cdot,\\cdot)$ and  \\eqref{lip-y};  the last inequality follows from the relative error condition of the projected gradient ascent method. This, together with \\eqref{OS-GS-better-key3-main}, yields\n\\[\n\\|x_r(y(z),z)-x_r(y_+(z),z)\\|\\leq \\frac{\\sqrt{2}}{\\sqrt{r-L}}\\left(\\frac{1+\\alpha L(1+\\sigma_2)}{\\alpha\\mu}\\right)^{\\frac{1}{2\\theta}} \n\\|y-y_+(z)\\|^{\\frac{1}{2\\theta}}.\n\\]\n \\end{enumerate}\nThe proof is complete.\n\\end{proof}","sha256_utf8":"0daa0bdf83daa11a0ccd3333f04c5c2f62e2f38d45f510e452b220601c60e31e"}]},{"id":"S094","semantic_type":"proof_step","kind":"proof_step","title":"最大势函数的强凸增长","source_units":[{"start":161253,"end":161803,"start_line":2159,"end_line":2169,"raw_tex":"\\begin{proof}\nThe proof follows closely that of Proposition \\ref{prop:dual_eb_KL}. Let $\\psi:\\R^n\\times \\R^n\\rightarrow\\R$ be the function defined by\n\\[\n\\psi(x,z) = \\max\\limits_{y\\in \\mathcal{Y}} F_r(x,y,z)\n\\]\nand consider arbitrary $x \\in \\mathcal{X}$, $y \\in \\mathcal{Y}$, and $z \\in \\mathbb{R}^n$. Again, note that  $\\psi(\\cdot,z)$ is $(r-L)$-strongly convex. This implies that\n% for any $x\\in \\mathcal{X}$, we have\n\\begin{equation}\n\\label{OS-GS-better-key1}\n\\psi(x,z) - \\psi(x^\\star_r(z),z) \\ge \\frac{r-L}{2}\\|x-x^\\star_r(z)\\|^2. \n\\end{equation}\n","sha256_utf8":"71cc5bf12ab82850c40cff2ef93174e4d1e604c7899fa27ee152e2500b618d6f"}]},{"id":"S095","semantic_type":"proof_step","kind":"proof_step","title":"弱对偶性比较两类正则最优值","source_units":[{"start":161803,"end":162657,"start_line":2170,"end_line":2187,"raw_tex":"Moreover, \n% for any $x\\in \\mathcal{X}$, \nwe  have\n\\begin{align} \n\\label{OS-GS-better-key2}\n&\\psi(x,z)-\\psi(x_r^\\star(z),z) \\notag\\\\   \n\\leq \\  &\\psi(x,z)-F_r(x_r(y_+(z),z),y_+(z),z)\\notag \\\\\n= \\ & \\max_{y^\\prime\\in \\mathcal{Y}} F(x,y^\\prime) + \\frac{r}{2}\\|x-z\\|^2-F_r(x_r(y_+(z),z),y_+(z),z)\\notag\\\\\n=\\,& \\max_{y^\\prime\\in \\mathcal{Y}}F(x,y^\\prime)-F(x_r(y_+(z),z),y_+(z)) + \\frac{r}{2}\\|x-z\\|^2-\\frac{r}{2}\\|x_r(y_+(z),z)-z\\|^2,\n\\end{align}\nwhere the  inequality follows from\n\\begin{align*}\nF_r(x_r(y_+(z),z),y_+(z),z) \n=\\ & \\min_{x'\\in\\mathcal{X}}\\left\\{ F(x',y_+(z)) +\\frac{r}{2}\\|x'-z\\|^2\\right\\} \\\\\n\\leq\\ & \\max_{y' \\in \\mathcal{Y}}\\min_{x'\\in\\mathcal{X}}\\left\\{F(x',y') +\\frac{r}{2}\\|x'-z\\|^2\\right\\} \\\\\n\\leq  \\ & \\min_{x'\\in\\mathcal{X}}\\max_{y' \\in \\mathcal{Y}}\\left\\{F(x',y') +\\frac{r}{2}\\|x'-z\\|^2\\right\\}\n= \\psi(x_r^\\star(z),z).\n\\end{align*}\n","sha256_utf8":"c1e4f93ff15cd678c8631ad04ee94e848707478099a923a10a6ef897aaa6063d"}]},{"id":"S096","semantic_type":"proof_step","kind":"proof_step","title":"代入解映射并复用KL论证","source_units":[{"start":162657,"end":163160,"start_line":2188,"end_line":2196,"raw_tex":"As \\eqref{OS-GS-better-key1} and \\eqref{OS-GS-better-key2} hold for any $x\\in\\mathcal{X}$, we obtain the intermediate relation\n\\begin{equation*}\n% \\label{OS-GS-better-key3} \n    \\frac{r-L}{2}\\|x_r^\\star(z)-x_r(y_+(z),z)\\|^2 \\leq \\max\\limits_{y^\\prime\\in \\mathcal{Y}} F(x_r(y_+(z),z),y^\\prime) - F(x_r(y_+(z),z),y_+(z)) \n\\end{equation*}\nby taking $x= x_r(y_+(z),z)$.\n\nThe remaining steps of the proof are the same as those of Proposition \\ref{prop:dual_eb_KL}. For brevity, we omit them here.\n\\end{proof}","sha256_utf8":"2ba6637548d9e069847c40c0273f21908aec9c1fc23816a8f4888a5017068466"}]},{"id":"S097","semantic_type":"proof_step","kind":"proof_step","title":"强凸性与鞍点性质控制距离","source_units":[{"start":163746,"end":164618,"start_line":2214,"end_line":2223,"raw_tex":"\\begin{proof}\nLet $y \\in \\mathcal{Y}$ and $z \\in \\mathbb{R}^n$ be arbitrary. Recall that $y(z)$ is an arbitrary vector in $Y(z)$. By the $(r-L)$-strong convexity of $F_r(\\cdot, y, z)$, we have\n\\begin{align}\n&F_r(x_r^\\star(z), y_{+}(z), z)-F_r(x_r(y_{+}(z), z), y_{+}(z), z)  \\geq \\frac{r-L}{2}\\|x_r(y_{+}(z), z)-x^\\star_r(z)\\|^{2}, \\notag\\\\\n&F_r(x_r(y_{+}(z), z), y(z), z)-F_r(x_r^\\star(z), y(z), z)  \\geq \\frac{r-L}{2}\\|x_r(y_{+}(z), z)-x_r^\\star(z)\\|^{2}.\\notag\n\\end{align}\n Moreover, since $(x_r^\\star(z),y(z))$ is a saddle point of the convex-concave function $F_r(\\cdot,\\cdot,z)$ \\citep{sion1958general}, we have $F_r(x_r^\\star(z), y(z), z) \\geq F_r(x_r^\\star(z), y_{+}(z), z)$, which implies that\n\\begin{equation}\\label{dualbd-strcvx}\nF_r(x_r(y_{+}(z), z), y(z), z)-F_r(x_r(y_{+}(z), z), y_{+}(z), z) \\geq  (r-L)\\|x_r(y_{+}(z), z)-x_r^\\star(z)\\|^{2}.\n\\end{equation}\n","sha256_utf8":"2749edd5c76670d06c0c057a5dd11c4f9100c5ef4e3cbd5b0ec208688cb8a856"}]},{"id":"S098","semantic_type":"proof_step","kind":"proof_step","title":"投影最优性与辅助凹函数","source_units":[{"start":164618,"end":165106,"start_line":2224,"end_line":2232,"raw_tex":"Note that $y_{+}(z)$ is the maximizer of \n\\[\n\\max_{y' \\in \\mathcal{Y}}\\big\\langle y+\\alpha \\nabla_{y} F_r(x_r(y, z), y, z)-y_{+}(z) , y'\\big\\rangle.\n\\]\nFor simplicity, we define the function $\\xi: \\mathbb{R}^d \\rightarrow \\mathbb{R}$ by\n\\begin{align*}\n\\xi(\\cdot)\n:=\\ &\\alpha F_r(x_r(y_{+}(z), z),\\cdot, z)-\\big\\langle \\alpha\\nabla_{y} F_r(x_r(y_{+}(z), z), y_{+}(z), z),\\cdot\\big\\rangle \\\\ \\ &-\\big\\langle y_{+}(z)-y-\\alpha \\nabla_{y} F_r(x_r(y, z), y, z), \\cdot\\big\\rangle.\n\\end{align*}\n","sha256_utf8":"62775bc1c678897b80c4dd6ce826c8b92199a7840e747d10365fb448fd981c09"}]},{"id":"S099","semantic_type":"proof_step","kind":"proof_step","title":"辅助函数在投影点达到最大","source_units":[{"start":165106,"end":165612,"start_line":2233,"end_line":2240,"raw_tex":"Then, we have\n\\begin{align*}\n\\max_{y' \\in \\mathcal{Y}}\\ \\xi(y')\n&\\leq \\alpha F_r(x_r(y_{+}(z), z),y_+(z), z)-\\big\\langle\\alpha\\nabla_y F_r(x_r(y_{+}(z), z),y_+(z), z), y_+(z)\\big\\rangle\\, \\notag\\\\\n&\\quad + \\max_{y' \\in \\mathcal{Y}} \\big\\langle y+\\alpha \\nabla_{y} F_r(x_r(y, z), y, z)-y_{+}(z), y'\\big\\rangle \\notag\\\\\n&\\leq \\xi(y_{+}(z)),\n\\end{align*}\nwhere the first inequality holds because $F_r(x_r(y_{+}(z), z),\\cdot,z)$ is concave. Therefore, we have $\\xi(y(z))\\leq \\xi(y_{+}(z))$, which implies that\n","sha256_utf8":"3078b98e38333db4a166ff42d8e0d44747a36401c00a14b215c0ebddd18287bb"}]},{"id":"S100","semantic_type":"proof_step","kind":"proof_step","title":"梯度连续性控制对偶值差","source_units":[{"start":165612,"end":166353,"start_line":2241,"end_line":2247,"raw_tex":"\\begin{align}\\label{dualbd-key}\n&F_r(x_r(y_{+}(z), z), y(z), z)-F_r(x_r(y_{+}(z), z), y_{+}(z), z) \\notag\\\\\n\\leq &\\ \\frac{1}{\\alpha}\\Big\\langle y(z)-y_{+}(z), \\alpha\\nabla_{y} F_r(x_r(y_{+}(z), z), y_{+}(z), z)-\\alpha \\nabla_{y} F_r(x_r(y, z), y, z)\\Big\\rangle \\notag \\\\ \\ & \\ + \\frac{1}{\\alpha}\\Big\\langle y(z)-y_{+}(z), y_{+}(z)-y\\Big\\rangle \\notag\\\\\n\\leq &\\ L \\|y_{+}(z)-y(z)\\|(\\|x_r(y_{+}(z), z)-x_r(y, z)\\|+ \\|y_{+}(z)-y\\|)  + \\frac{1}{\\alpha}\\|y_{+}(z)-y(z)\\|\\cdot \\|y-y_{+}(z)\\|\\notag\\\\\n\\leq &\\ \\left(\\frac{1}{\\alpha}+L\\sigma_2+L\\right)\\|y_{+}(z)-y(z)\\| \\cdot\\|y-y_{+}(z)\\|.\n\\end{align}\nHere, the second inequality is due to the $L$-Lipschitz continuity of $\\nabla_y F_r(\\cdot,\\cdot,z)$ and the last inequality is from \\eqref{lip-y}.\n","sha256_utf8":"5a77f53dc1b3fb82621c0d8d2517035b78a78c6582ad854cbcaf071c823f1894"}]},{"id":"S101","semantic_type":"proof_step","kind":"proof_step","title":"紧致对偶域给出凹情形界","source_units":[{"start":166353,"end":166612,"start_line":2248,"end_line":2253,"raw_tex":"Hence, by combining \\eqref{dualbd-strcvx} and \\eqref{dualbd-key}, we obtain\n\\[\n(r-L)\\|x_r^\\star(z)-x_r(y_{+}(z), z)\\|^{2}\\leq \\left(\\frac{1}{\\alpha}+ L\\sigma_2+L\\right){\\rm diam}(\\mathcal{Y})\\cdot\\|y-y_{+}(z)\\|,\n\\]\nwhich proves the desired result.\n\\end{proof}","sha256_utf8":"a3d116752dae811b37570fbfd4db819ecf1fc9d1b801a217d82aa4e59897605f"}]},{"id":"S102","semantic_type":"proof_step","kind":"proof_step","title":"三个迭代间隙控制原始驻点量","source_units":[{"start":70140,"end":71414,"start_line":979,"end_line":994,"raw_tex":"\\begin{proof}\nBy Definition \\ref{defi:primal-dual}, we only have to quantify $\\|\\nabla_{z} d_r(y^{k+1},x^{k+1})\\|$ and ${\\rm dist}(0,-\\nabla_{y} F(x^{k+1}, y^{k+1})+\\partial\\iota_{\\mathcal{Y}}(y^{k+1}))$.\nTo begin, observe that\n\\begin{equation*}\n\\begin{aligned}\n& \\| \\nabla_{z} d_r(y^{k+1},x^{k+1})\\| \n\\\\\n=\\ & r \\|x^{k+1}-x_r (y^{k+1},x^{k+1})\\|\\\\  \n\\leq\\ & r(\\|x^{k+1}-x_r(y^{k},z^{k})\\| +\\|x_r(y^{k},z^{k})-x_r(y^{k+1},z^{k})\\|)  +r \\|x_r(y^{k+1},z^{k})-x_r(y^{k+1},x^{k+1})\\|  \\\\\n\\leq\\  &  r\\left(\\zeta\\|x^{k+1}-x^k\\| +\\sigma_2\\|y^{k}-y^{k+1}\\| + \\sigma_1 \\|x^{k+1}-z^{k}\\| \\right) \\\\\n% \\leq & r\\left(\\|x^{k+1}-x_r(y^{k},z^{k})\\| +\\|x_r(y^{k},z^{k})-x_r(y^{k+1},z^{k})\\| + \\|x_r(y^{k+1},z^{k})-x_r(y^{k+1},x^{k+1})\\| \\right) \\\\\n\\leq\\  &  r\\left(\\zeta\\|x^{k+1}-x^k\\| + \\sigma_1 \\|x^{k+1}-z^{k}\\| \\right) +r\\sigma_2 (\\eta\\| x^{k+1}-x^k\\|+\\sigma_1\\alpha L\\|z^{k+1}-z^k\\|+\\| y_+^k(z^{k+1})-y^k\\|)\\\\\n\\leq\\ & r(\\zeta+\\sigma_2(\\eta+1+\\sigma_1\\alpha \\beta L)+\\sigma_1)\\epsilon,\n\\end{aligned}\n\\end{equation*}\nwhere the second inequality is due to Proposition \\ref{prop:lip} and Lemma \\ref{lemma-sollip}, the third is due to Lemma \\ref{lemma-dualxy}, and the fourth follows from the update $z^{k+1} = z^k + \\beta (x^{k+1} - x^k)$ and the assumption of the lemma. Next, recall that\n","sha256_utf8":"97c88c8d19141e9ad57953f8f5e1d96ea080500d06a4c902dc245efc9dfcd4cd"}]},{"id":"S103","semantic_type":"proof_step","kind":"proof_step","title":"对偶投影步的最优性条件","source_units":[{"start":71414,"end":71944,"start_line":995,"end_line":1003,"raw_tex":"\\begin{equation*}\ny^{k+1} = \\proj_{\\mathcal{Y}}\\left(y^k + \\alpha \\nabla_y F(x^{k+1}, y^k)\\right)=\\argmin_{y \\in \\mathcal{Y}}\\left\\{\\|y-y^k - \\alpha \\nabla_y F(x^{k+1}, y^k)\\|^2\\right\\}.\n\\end{equation*}\nThe necessary optimality condition yields\n\\begin{align*}\n0\n&\\in y^{k+1}-y^k - \\alpha \\nabla_y F(x^{k+1}, y^k)+\\partial \\iota_{\\mathcal{Y}}(y^{k+1})\\notag\\\\\n&=y^{k+1}-y^k - \\alpha \\nabla_y F(x^{k+1}, y^{k+1})+\\partial \\iota_{\\mathcal{Y}}(y^{k+1}) +\\alpha (\\nabla_y F(x^{k+1}, y^{k+1})- \\nabla_y F(x^{k+1}, y^{k})).\n\\end{align*}\n","sha256_utf8":"a4598823d0a47e5ed76dc535ebcb1330c5469e8464287d7c8c95392a824d1cc7"}]},{"id":"S104","semantic_type":"proof_step","kind":"proof_step","title":"构造对偶驻点残差向量","source_units":[{"start":71944,"end":72246,"start_line":1004,"end_line":1008,"raw_tex":"Since $\\partial \\iota_{\\mathcal{Y}} (y^{k+1})$ is the normal cone to $\\mathcal{Y}$ at $y^{k+1}$, we have\n\\begin{align*}\nv\n&:=-\\frac{1}{\\alpha}(y^{k+1}-y^{k})+\\nabla_y F(x^{k+1}, y^{k})- \\nabla_y F(x^{k+1}, y^{k+1})\\in -\\nabla_{y} F(x^{k+1}, y^{k+1})+\\partial \\iota_{\\mathcal{Y}}(y^{k+1}).\n\\end{align*}\n","sha256_utf8":"6a795843eba7fdd757902e63afd6117ee2314e5c62936b2f25cb7357f80ff058"}]},{"id":"S105","semantic_type":"proof_step","kind":"proof_step","title":"界定对偶残差并得到博弈驻点","source_units":[{"start":72246,"end":72783,"start_line":1009,"end_line":1016,"raw_tex":"It follows that\n\\begin{align*}\n\\|v\\| \\ & \\leq \\left(\\frac{1}{\\alpha}+L\\right)\\|y^{k+1}-y^{k}\\|\\\\\n\\ &  \\leq  \\left(\\frac{1}{\\alpha}+L\\right)\\left(\\eta\\|x^{k}-x^{k+1}\\|+\\sigma_1\\alpha L\\|z^{k+1}-z^k\\|+\\|y_+^k(z^{k+1})-y^k\\|\\right) \\\\\n\\ & \\leq (\\eta+1+\\sigma_1\\alpha\\beta L)\\left(\\frac{1}{\\alpha}+L\\right)\\epsilon,\n\\end{align*}\nwhere the first inequality is due to the $L$-Lipschitz continuity of $\\nabla_y F(\\cdot,\\cdot)$ and the second is due to Lemma \\ref{lemma-dualxy}. Putting everything together yields the desired result.\n\\end{proof}","sha256_utf8":"dfb5a82cc57b712abaa8e3bcc83de2ec764945c87f155f9cb5a3f7ebe5f42262"}]},{"id":"S106","semantic_type":"proof_step","kind":"proof_step","title":"按对偶增长幂组织证明","source_units":[{"start":77680,"end":78175,"start_line":1059,"end_line":1064,"raw_tex":"\\begin{proof}\n\nAs mentioned in Section \\ref{sec:framework}, the global convergence rate of smoothed PLDA depends on the specific regime of the growth power $v$ in \\eqref{eq:step_dual}. Let us begin by considering cases (a) and (b), i.e., either the dual function is concave or the dual problem possesses the K\\L{} property with exponent $\\theta\\in(\\frac{1}{2},1)$.\n\n\nIn view of the basic descent estimate in Proposition \\ref{prop:decrease}, we consider the following two scenarios separately:\\\\\n","sha256_utf8":"e8c7774f3aead686ee3d0ee1f844531ffadd154d5c8cafddaf26dd94c9c4a45d"}]},{"id":"S107","semantic_type":"proof_step","kind":"proof_step","title":"划分误差主导与下降主导情形","source_units":[{"start":78175,"end":78835,"start_line":1065,"end_line":1074,"raw_tex":"(1) There exists a $ k \\in\\{0,1, \\ldots, K-1\\}$ such that\n\\begin{align*}\n\\ &\\frac{1}{2}\\max \\left\\{\\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}, \\frac{1}{8\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 , \\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}\\right\\}\\\\\n\\leq \\ & 14r\\beta\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2}.\n\\end{align*}\n(2) For any $ k \\in\\{0,1, \\ldots, K-1\\}$, we have\n\\begin{align*}\n\\ &\\frac{1}{2}\\max \\left\\{\\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}, \\frac{1}{8\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 , \\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}\\right\\}\\\\\n\\ge\\ & 14r\\beta\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2}.\n\\end{align*}\n","sha256_utf8":"ec2361dbe527c50f9bfd6e7d82739d9acdcd1a42ee372821eb17560c0009f4e3"}]},{"id":"S108","semantic_type":"proof_step","kind":"proof_step","title":"误差主导时界定对偶与中心间隙","source_units":[{"start":78835,"end":79857,"start_line":1075,"end_line":1086,"raw_tex":"\n\\textbf{Scenario (1)}. For the case where the K{\\L} exponent $\\theta \\in (\\tfrac{1}{2},1)$, we deduce from Proposition \\ref{prop:dual_eb_KL}(b) that \n\\begin{align*}\n    \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2  & \\leq 224r\\alpha\\beta\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2}  \\\\ &\\leq  224r\\alpha\\beta \\omega_2^2 \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{\\theta}},  \n\\end{align*}\nwhich gives $\\|y^{k}-y_{+}^{k}(z^{k+1})\\|\\leq \\rho_1\\beta^{\\frac{\\theta}{2\\theta-1}}$ with $\\rho_1 := (224r\\alpha\\omega_2^2)^{\\frac{\\theta}{2\\theta-1}}$. Additionally, using the update $z^{k+1} = z^k + \\beta( x^{k+1} - z^k )$ and Proposition \\ref{prop:dual_eb_KL}(b), we have\n\\begin{align*}\n\\|x^{k+1}-z^{k}\\|^{2}  \\ & =\\frac{1}{\\beta^2}\\|z^{k+1}-z^{k}\\|^{2}\\\\\n\\ &  \\leq 49\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2} \\\\\n\\ & \\leq  49 \\omega_2^2 \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{\\theta}}\\leq \\rho_2^2\\beta^{\\frac{1}{2\\theta-1}},\n\\end{align*}\nwhere $\\rho_2 := \\sqrt{49\\omega_2^2\\rho_1^{\\frac{1}{\\theta}}}$. \n","sha256_utf8":"369bc094de0eee2ab655621d6b1f18d4b9c52ebec3c18bff1a0a11b8eb72b754"}]},{"id":"S109","semantic_type":"proof_step","kind":"proof_step","title":"误差主导时界定原始间隙","source_units":[{"start":79857,"end":80263,"start_line":1087,"end_line":1092,"raw_tex":"Lastly, we have\n\\begin{align*}\n    \\|x^{k+1}-x^k\\|^2 \\ & \\leq {\\frac{224}{3}r\\beta\\lambda}\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2} \\\\\n   \\  &\\leq {\\frac{224}{3}r\\omega^2_2\\beta\\lambda}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{\\theta}} \\leq \\rho_3^2 \\beta^{\\frac{2\\theta}{2\\theta-1}},  \n\\end{align*}\nwhere $\\rho_3 := \\sqrt{{\\frac{224}{3}r\\omega_2^2\\lambda}\\rho_1^{\\frac{1}{\\theta}}}$. \n","sha256_utf8":"9d51ccaee3993370edab78021dc9f8df4652d6d7aedb41d30ac73e3d87f2b932"}]},{"id":"S110","semantic_type":"proof_step","kind":"proof_step","title":"得到博弈驻点并处理凹情形","source_units":[{"start":80263,"end":81397,"start_line":1093,"end_line":1100,"raw_tex":"Combining the above inequalities, we have\n\\begin{align*}\n& \\max \\left\\{\\|x^{k}-x^{k+1}\\|,\\|y^{k}-y_{+}^{k}(z^{k+1})\\|,\\|z^{k}-x^{k+1}\\|\\right\\} \\leq  \\max\\left\\{ \\rho_1\\beta^{\\frac{\\theta}{2\\theta-1}}, \\rho_2\\beta^{\\frac{1}{4\\theta-2}}, \\rho_3 \\beta^{\\frac{\\theta}{2\\theta-1}}\\right\\}. \n\\end{align*}\nIt then follows from Lemma \\ref{lemma-episolcol}\n that $\\left(x^{k+1}, y^{k+1}\\right)$ is a $\\rho\\cdot\\max\\{ \\rho_1\\beta^{{\\theta}/{(2\\theta-1)}},$ $ \\rho_2\\beta^{1/{(4\\theta-2)}}, \\rho_3\\beta^{{\\theta}/{(2\\theta-1)}}\\}$-GS of problem \\eqref{eq:problem}. \n \n For the concave case, we can replace the primal-dual quantity $x_r( y(z^{k+1}), z^{k+1} )$ by the primal quantity $x_r^\\star( z^{k+1} )$ on the right-hand side of the inequality for Scenario (1) and apply Lemma \\ref{lemma-dual-bd} in Appendix \\ref{sec:dual} (so that $\\theta$ and $\\omega_2$ are replaced by $1$ and $\\kappa = \\tfrac{1 + \\alpha L \\sigma_2 + \\alpha L}{\\alpha (r-L)} \\cdot \\textrm{diam}(\\mathcal{Y})$, respectively) to conclude that $\\left(x^{k+1}, y^{k+1}\\right)$ is a $\\max\\{ \\rho_1\\beta, \\rho_2\\beta^{1/2}, \\rho_3 \\beta\\}$-GS of problem \\eqref{eq:problem}.  \n","sha256_utf8":"09c6519c078326717615edc34482cbb6b0d9dd045fd887c30e28fc32a0607747"}]},{"id":"S111","semantic_type":"proof_step","kind":"proof_step","title":"误差主导时控制优化驻点量","source_units":[{"start":81397,"end":82695,"start_line":1101,"end_line":1117,"raw_tex":" \n\n\nFurthermore, observe that for the case where $\\theta \\in (\\tfrac{1}{2}, 1)$, the optimization-stationarity measure of $z^{k+1}$ satisfies\n\\begin{equation}\n\\label{eq:os-gs}\n\\begin{aligned}\n    & \\|z^{k+1}-x_r^\\star(z^{k+1})\\| \\\\\n    \\leq\\ &  \\|z^{k+1}-x^{k+1}\\|+\\| x^{k+1} - x_r(y^k,z^k) \\| + \\| x_r(y^k,z^k) - x_r(y^k,z^{k+1}) \\|\\\\\n    \\ & + \\|x_r(y^k,z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|+ \\|x_r(y_{+}^{k}(z^{k+1}), z^{k+1})-x_r^\\star(z^{k+1})\\|\\\\\n     \\leq\\ &  (1+\\sigma_1)\\|z^{k}-x^{k+1}\\| + \\zeta\\|x^{k}-x^{k+1}\\| + \\sigma_2 \\|y^k-y_{+}^{k}(z^{k+1})\\|+ \\omega_2 \\|y^k-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{2\\theta}} \\\\\n    \\leq\\ & \\max\\left\\{\\left(\\rho_2(1+\\sigma_1)+ \\omega_2 \\rho_1^\\frac{1}{2\\theta}\\right)\\beta^{\\frac{1}{4\\theta-2}},(\\zeta\\rho_3+\\sigma_2\\rho_1) \\beta^{\\frac{\\theta}{2\\theta-1}} \\right\\}, \n\\end{aligned}\n\\end{equation}\nwhere the second inequality follows from the update $z^{k+1} = z^k + \\beta (x^{k+1} - z^k)$, Proposition \\ref{prop:lip}, Corollary \\ref{prop:dual_eb_KL2}, Lemma \\ref{lemma-sollip}, and the fact that $0 < \\beta \\le \\tfrac{1}{28}$.\nIt is evident that the dependence on $\\beta$ is identical to that of the game-stationarity measure of $(x^{k+1}, y^{k+1})$.\nFor the concave case, the optimization-stationarity measure of $z^{k+1}$ can be bounded similarly.\n","sha256_utf8":"4a15b2ac087cee56a8762330cc965a69eed1d21ec96bc8f2c31344b638a95cf9"}]},{"id":"S112","semantic_type":"proof_step","kind":"proof_step","title":"下降主导时得到充分下降","source_units":[{"start":82695,"end":83019,"start_line":1118,"end_line":1124,"raw_tex":"\n\\textbf{Scenario (2)}. Proposition \\ref{prop:decrease} implies that for any $k \\in\\{0,1, \\ldots, K-1\\}$, we have \n\\begin{align}\n\\label{eq:suff-decrease}\n\\Phi_r^k-\\Phi_r^{k+1} \n\\ge \\frac{3}{16\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{16\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 +\\frac{2r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}.\n\\end{align}\n","sha256_utf8":"38fa73a77cb49947b249b86e8e3e4ccf7b2bb3fec487d7dbf08aaa5ae511989d"}]},{"id":"S113","semantic_type":"proof_step","kind":"proof_step","title":"建立李雅普诺夫函数统一下界","source_units":[{"start":83019,"end":83671,"start_line":1125,"end_line":1131,"raw_tex":"The assumptions in case (a)  imply that \n\\[ \\min_{x\\in\\mathcal{X}}\\max_{y\\in\\mathcal{Y}} F_r(x,y,z) = \\max_{y\\in\\mathcal{Y}} \\min_{x\\in\\mathcal{X}} F_r(x,y,z) \\]\nfor all $z \\in \\mathbb{R}^n$ (cf.~the proof of~\\citep[Lemma B.7]{zhang2020single}) and lead to the lower boundedness of $p_r$, while the assumptions in case (b) directly imply that $p_r$ is lower bounded.\nIn both cases, there exists a $\\underline{\\Phi} > -\\infty$ such that for all $x \\in \\mathcal{X}$, $y \\in \\mathcal{Y}$, and $z \\in \\mathbb{R}^n$,\n\\begin{align*}\n& \\Phi_r(x, y, z)  = p_r(z)+(F_r(x,y,z)-d_r(y, z))+(p_r(z)-d_r(y, z))\\geq p_r(z)\\geq \\underline{\\Phi} >-\\infty.\n\\end{align*}\n","sha256_utf8":"52a36565097a33c39cb6e4814575fd5cb805c86eec6af12621a4fa9ddb60045d"}]},{"id":"S114","semantic_type":"proof_step","kind":"proof_step","title":"求和选取迭代点并平衡步长","source_units":[{"start":83671,"end":85536,"start_line":1132,"end_line":1147,"raw_tex":"It follows that\n    \\begin{align*}\n    &\\Phi_r^0-\\underline{\\Phi} \\geq \\sum_{k=0}^{K-1}\\Phi_r(x^k,y^k,z^k)-\\Phi_r(x^{k+1},y^{k+1},z^{k+1}) \\\\\n    % \\geq\\ & \\sum_{k=0}^{K-1} \\frac{1}{32\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{16\\alpha}\\|y^{k}-y_{+}^{k}(z^{k})\\|^2 +\\frac{2r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2} \\\\\n    \\geq \\ & \\sum_{k=0}^{K-1} \\min\\left\\{\\frac{3}{16\\lambda},\\frac{1}{16\\alpha},\\frac{2\\beta r}{7}\\right\\} \\left(\\|x^{k}-x^{k+1}\\|^{2}+\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 + \\|z^{k}-x^{k+1}\\|^{2} \\right),\n    \\end{align*}\n    where the last inequality is due to \\eqref{eq:suff-decrease} and the update $z^{k+1}=z^{k}+\\beta(x^{k+1}-z^k)$.\n    In particular, there exists a $k \\in\\{0,1, \\ldots, K-1\\}$ such that\n    \\[\n    \\max \\left\\{\\|x^{k}-x^{k+1}\\|^{2},\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{2}, \\|x^{k+1}-z^{k}\\|^{2}\\right\\} \\leq \\frac{\\Phi_r^0-\\underline{\\Phi}}{\\min\\left\\{\\frac{3}{16\\lambda},\\frac{1}{16\\alpha},\\frac{2\\beta  r}{7}\\right\\}  K}.\n    \\]\n% \\Jiajin{continue here!}\nBased on Lemma \\ref{lemma-episolcol} and the fact that $0 < \\beta \\le \\tfrac{1}{28}$,  we conclude that $\\left(x^{k+1}, y^{k+1}\\right)$ is an $\\mathcal{O}(\\sqrt{1/{K\\beta}})$-GS of problem \\eqref{eq:problem}. Moreover, by using the same argument as in \\eqref{eq:os-gs}, we can show that $z^{k+1}$ is an $\\mathcal{O}(\\sqrt{1/{K\\beta}})$-OS of problem \\eqref{eq:problem}. Upon setting $\\beta = C K^{-\\frac{1}{2}}$ for the concave case and  $\\beta = C K^{-\\frac{2\\theta-1}{2\\theta}}$ for the case where the K{\\L} exponent $\\theta \\in (\\tfrac{1}{2}, 1)$, with $C$ being a suitably chosen constant, we see that the \n% where $C$ is a constant.\n% Similarly, for the general concave case, we can set $\\beta = C K^{-\\frac{1}{2}}$. \nbounds obtained for the optimization-stationarity and game-stationarity measures are of the same order for both Scenarios (1) and (2). This establishes cases (a) and (b).\n","sha256_utf8":"867a3b6f475ec1760b4b984481420783ab69effbee837299261d70d6b2a807a8"}]},{"id":"S115","semantic_type":"proof_step","kind":"proof_step","title":"小KL指数情形的基本下降式","source_units":[{"start":85536,"end":85975,"start_line":1148,"end_line":1155,"raw_tex":"\nNext, let us consider case (c). First, suppose that the K{\\L} exponent $\\theta \\in (0, \\tfrac{1}{2}]$. \nOnce again, we can use Proposition \\ref{prop:decrease} to obtain \n\\begin{align*}\n\\Phi_r^k-\\Phi_r^{k+1} \n\\ge\\ &  \\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{8\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 +\\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}\\\\\n\\ & -14r\\beta\\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|^{2}.\n\\end{align*}\n","sha256_utf8":"3cb16ec184466b411bfd385a013cf245c75111a6bb2959b63d00578673b6b77e"}]},{"id":"S116","semantic_type":"proof_step","kind":"proof_step","title":"紧致性将幂次误差界齐次化","source_units":[{"start":85975,"end":86530,"start_line":1156,"end_line":1165,"raw_tex":"Using Proposition \\ref{prop:dual_eb_KL}(b) and the boundedness of $\\mathcal{Y}$, we have\n% there exists  for each $\\theta \\in(0,\\frac{1}{2})$ that \n\\begin{equation}\\label{dual-eb-bd}\n\\begin{aligned}\n  \\|x_r(y(z^{k+1}),z^{k+1})-x_r(y_{+}^{k}(z^{k+1}), z^{k+1})\\|\n&\\leq \\omega_2 \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{2\\theta}}\\\\\n&=  \\omega_2 \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^{\\frac{1}{2\\theta}-1} \\|y^{k}-y_{+}^{k}(z^{k+1})\\|\\\\\n&\\leq   \\omega_2\\cdot {\\rm diam}(\\mathcal{Y})^{\\frac{1}{2\\theta}-1}\\cdot \\|y^{k}-y_{+}^{k}(z^{k+1})\\|.\n\\end{aligned}\n\\end{equation}\n","sha256_utf8":"213e4a29ba2af81f3c7f058e6e12b6067dda2d8c22b5dffbab9404e5afd42080"}]},{"id":"S117","semantic_type":"proof_step","kind":"proof_step","title":"吸收负项得到平方根收敛率","source_units":[{"start":86530,"end":87258,"start_line":1166,"end_line":1176,"raw_tex":"It follows that\n\\begin{align*}\n\\Phi_r^k-\\Phi_r^{k+1} \n\\ge \\ &  \\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\left(\\frac{1}{8\\alpha}-14r\\beta\\omega_2^2\\cdot{\\rm diam}(\\mathcal{Y})^{\\frac{1-2\\theta}{\\theta}}\\right) \\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2  +\\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}.\n\\end{align*}\nSince \n$\\beta  \\leq \\frac{ {\\rm diam}(\\mathcal{Y})^{\\frac{2\\theta-1}{\\theta}}}{224 \\alpha r\\omega_2^2}$, we have \n\\begin{align*}\n&\\Phi_r^k-\\Phi_r^{k+1} \\ge  \\frac{3}{8\\lambda}\\|x^{k}-x^{k+1}\\|^{2}+\\frac{1}{16\\alpha}\\|y^{k}-y_{+}^{k}(z^{k+1})\\|^2 +\\frac{4r}{7\\beta}\\|z^{k}-z^{k+1}\\|^{2}.\n\\end{align*}\nThe desired result can then be obtained by following a similar argument as that in the paragraph proceeding \\eqref{eq:suff-decrease}.\n","sha256_utf8":"b73de71c2e6c5a0024f7d95f1091c94ed6cb1bb9940ffa6f7d7e3c0d1ce835db"}]},{"id":"S118","semantic_type":"proof_step","kind":"proof_step","title":"零KL指数复用充分下降论证","source_units":[{"start":87258,"end":87655,"start_line":1177,"end_line":1182,"raw_tex":"\nWhen the K{\\L} exponent $\\theta=0$, we can apply Proposition \\ref{prop:dual_eb_KL}(a) to derive the inequality \\eqref{dual-eb-bd} \n% by letting $\\theta=\\frac{1}{2}$ and replacing $\\omega_2$ with $\\omega_1$. \nwith $\\theta$ and $\\omega_2$ being replaced by $\\tfrac{1}{2}$ and $\\omega_1$, respectively.\nThe desired result then follows by adapting the argument in the preceding paragraph.\n\\end{proof}","sha256_utf8":"e96ad3aa2732c1b260791c3bac5a9789bc68f8035d7ca4b5d26dd35f426a329c"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S003","target_id":"S006","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S015","target_id":"S007","kind":"supports"},{"source_id":"S004","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S010","target_id":"S053","kind":"supports"},{"source_id":"S010","target_id":"S067","kind":"supports"},{"source_id":"S006","target_id":"S011","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S004","target_id":"S018","kind":"supports"},{"source_id":"S004","target_id":"S019","kind":"supports"},{"source_id":"S044","target_id":"S018","kind":"supports"},{"source_id":"S044","target_id":"S019","kind":"supports"},{"source_id":"S014","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S016","kind":"supports"},{"source_id":"S006","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S051","kind":"supports"},{"source_id":"S011","target_id":"S051","kind":"supports"},{"source_id":"S046","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S045","target_id":"S052","kind":"supports"},{"source_id":"S015","target_id":"S052","kind":"supports"},{"source_id":"S008","target_id":"S053","kind":"supports"},{"source_id":"S019","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S015","target_id":"S055","kind":"supports"},{"source_id":"S052","target_id":"S056","kind":"supports"},{"source_id":"S053","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S046","target_id":"S056","kind":"supports"},{"source_id":"S051","target_id":"S021","kind":"supports"},{"source_id":"S052","target_id":"S021","kind":"supports"},{"source_id":"S053","target_id":"S021","kind":"supports"},{"source_id":"S054","target_id":"S021","kind":"supports"},{"source_id":"S055","target_id":"S021","kind":"supports"},{"source_id":"S056","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S006","target_id":"S057","kind":"supports"},{"source_id":"S011","target_id":"S057","kind":"supports"},{"source_id":"S016","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S018","target_id":"S059","kind":"supports"},{"source_id":"S046","target_id":"S059","kind":"supports"},{"source_id":"S016","target_id":"S060","kind":"supports"},{"source_id":"S004","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S004","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S046","target_id":"S062","kind":"supports"},{"source_id":"S057","target_id":"S023","kind":"supports"},{"source_id":"S058","target_id":"S023","kind":"supports"},{"source_id":"S059","target_id":"S023","kind":"supports"},{"source_id":"S060","target_id":"S023","kind":"supports"},{"source_id":"S061","target_id":"S023","kind":"supports"},{"source_id":"S062","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S016","target_id":"S063","kind":"supports"},{"source_id":"S004","target_id":"S063","kind":"supports"},{"source_id":"S006","target_id":"S063","kind":"supports"},{"source_id":"S048","target_id":"S063","kind":"supports"},{"source_id":"S049","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S022","target_id":"S064","kind":"supports"},{"source_id":"S004","target_id":"S064","kind":"supports"},{"source_id":"S063","target_id":"S065","kind":"supports"},{"source_id":"S022","target_id":"S065","kind":"supports"},{"source_id":"S063","target_id":"S025","kind":"supports"},{"source_id":"S064","target_id":"S025","kind":"supports"},{"source_id":"S065","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S009","target_id":"S066","kind":"supports"},{"source_id":"S011","target_id":"S066","kind":"supports"},{"source_id":"S004","target_id":"S066","kind":"supports"},{"source_id":"S020","target_id":"S066","kind":"supports"},{"source_id":"S022","target_id":"S066","kind":"supports"},{"source_id":"S046","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S008","target_id":"S067","kind":"supports"},{"source_id":"S009","target_id":"S067","kind":"supports"},{"source_id":"S019","target_id":"S067","kind":"supports"},{"source_id":"S004","target_id":"S068","kind":"supports"},{"source_id":"S046","target_id":"S068","kind":"supports"},{"source_id":"S009","target_id":"S069","kind":"supports"},{"source_id":"S067","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S067","target_id":"S029","kind":"supports"},{"source_id":"S068","target_id":"S029","kind":"supports"},{"source_id":"S069","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S024","target_id":"S070","kind":"supports"},{"source_id":"S046","target_id":"S070","kind":"supports"},{"source_id":"S006","target_id":"S071","kind":"supports"},{"source_id":"S011","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S031","kind":"supports"},{"source_id":"S071","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S011","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S006","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S033","kind":"supports"},{"source_id":"S073","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S012","target_id":"S074","kind":"supports"},{"source_id":"S028","target_id":"S074","kind":"supports"},{"source_id":"S030","target_id":"S074","kind":"supports"},{"source_id":"S032","target_id":"S074","kind":"supports"},{"source_id":"S075","target_id":"S077","kind":"supports"},{"source_id":"S009","target_id":"S076","kind":"supports"},{"source_id":"S046","target_id":"S076","kind":"supports"},{"source_id":"S076","target_id":"S077","kind":"supports"},{"source_id":"S004","target_id":"S077","kind":"supports"},{"source_id":"S020","target_id":"S077","kind":"supports"},{"source_id":"S046","target_id":"S077","kind":"supports"},{"source_id":"S022","target_id":"S078","kind":"supports"},{"source_id":"S046","target_id":"S078","kind":"supports"},{"source_id":"S074","target_id":"S079","kind":"supports"},{"source_id":"S077","target_id":"S079","kind":"supports"},{"source_id":"S078","target_id":"S079","kind":"supports"},{"source_id":"S026","target_id":"S080","kind":"supports"},{"source_id":"S046","target_id":"S080","kind":"supports"},{"source_id":"S022","target_id":"S081","kind":"supports"},{"source_id":"S026","target_id":"S081","kind":"supports"},{"source_id":"S079","target_id":"S082","kind":"supports"},{"source_id":"S080","target_id":"S082","kind":"supports"},{"source_id":"S081","target_id":"S082","kind":"supports"},{"source_id":"S024","target_id":"S083","kind":"supports"},{"source_id":"S022","target_id":"S084","kind":"supports"},{"source_id":"S026","target_id":"S085","kind":"supports"},{"source_id":"S020","target_id":"S085","kind":"supports"},{"source_id":"S082","target_id":"S086","kind":"supports"},{"source_id":"S083","target_id":"S086","kind":"supports"},{"source_id":"S084","target_id":"S086","kind":"supports"},{"source_id":"S085","target_id":"S086","kind":"supports"},{"source_id":"S074","target_id":"S035","kind":"supports"},{"source_id":"S075","target_id":"S035","kind":"supports"},{"source_id":"S076","target_id":"S035","kind":"supports"},{"source_id":"S077","target_id":"S035","kind":"supports"},{"source_id":"S078","target_id":"S035","kind":"supports"},{"source_id":"S079","target_id":"S035","kind":"supports"},{"source_id":"S080","target_id":"S035","kind":"supports"},{"source_id":"S081","target_id":"S035","kind":"supports"},{"source_id":"S082","target_id":"S035","kind":"supports"},{"source_id":"S083","target_id":"S035","kind":"supports"},{"source_id":"S084","target_id":"S035","kind":"supports"},{"source_id":"S085","target_id":"S035","kind":"supports"},{"source_id":"S086","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S011","target_id":"S087","kind":"supports"},{"source_id":"S016","target_id":"S087","kind":"supports"},{"source_id":"S011","target_id":"S088","kind":"supports"},{"source_id":"S087","target_id":"S089","kind":"supports"},{"source_id":"S088","target_id":"S089","kind":"supports"},{"source_id":"S005","target_id":"S090","kind":"supports"},{"source_id":"S089","target_id":"S090","kind":"supports"},{"source_id":"S003","target_id":"S090","kind":"supports"},{"source_id":"S090","target_id":"S091","kind":"supports"},{"source_id":"S047","target_id":"S091","kind":"supports"},{"source_id":"S091","target_id":"S092","kind":"supports"},{"source_id":"S089","target_id":"S092","kind":"supports"},{"source_id":"S005","target_id":"S093","kind":"supports"},{"source_id":"S004","target_id":"S093","kind":"supports"},{"source_id":"S022","target_id":"S093","kind":"supports"},{"source_id":"S047","target_id":"S093","kind":"supports"},{"source_id":"S089","target_id":"S093","kind":"supports"},{"source_id":"S087","target_id":"S037","kind":"supports"},{"source_id":"S088","target_id":"S037","kind":"supports"},{"source_id":"S089","target_id":"S037","kind":"supports"},{"source_id":"S090","target_id":"S037","kind":"supports"},{"source_id":"S091","target_id":"S037","kind":"supports"},{"source_id":"S092","target_id":"S037","kind":"supports"},{"source_id":"S093","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S018","target_id":"S094","kind":"supports"},{"source_id":"S006","target_id":"S094","kind":"supports"},{"source_id":"S011","target_id":"S094","kind":"supports"},{"source_id":"S006","target_id":"S095","kind":"supports"},{"source_id":"S011","target_id":"S095","kind":"supports"},{"source_id":"S094","target_id":"S096","kind":"supports"},{"source_id":"S095","target_id":"S096","kind":"supports"},{"source_id":"S090","target_id":"S096","kind":"supports"},{"source_id":"S091","target_id":"S096","kind":"supports"},{"source_id":"S092","target_id":"S096","kind":"supports"},{"source_id":"S093","target_id":"S096","kind":"supports"},{"source_id":"S094","target_id":"S039","kind":"supports"},{"source_id":"S095","target_id":"S039","kind":"supports"},{"source_id":"S096","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S017","target_id":"S097","kind":"supports"},{"source_id":"S016","target_id":"S097","kind":"supports"},{"source_id":"S011","target_id":"S097","kind":"supports"},{"source_id":"S050","target_id":"S097","kind":"supports"},{"source_id":"S011","target_id":"S098","kind":"supports"},{"source_id":"S046","target_id":"S098","kind":"supports"},{"source_id":"S098","target_id":"S099","kind":"supports"},{"source_id":"S017","target_id":"S099","kind":"supports"},{"source_id":"S099","target_id":"S100","kind":"supports"},{"source_id":"S004","target_id":"S100","kind":"supports"},{"source_id":"S022","target_id":"S100","kind":"supports"},{"source_id":"S097","target_id":"S101","kind":"supports"},{"source_id":"S100","target_id":"S101","kind":"supports"},{"source_id":"S003","target_id":"S101","kind":"supports"},{"source_id":"S097","target_id":"S041","kind":"supports"},{"source_id":"S098","target_id":"S041","kind":"supports"},{"source_id":"S099","target_id":"S041","kind":"supports"},{"source_id":"S100","target_id":"S041","kind":"supports"},{"source_id":"S101","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S007","target_id":"S102","kind":"supports"},{"source_id":"S024","target_id":"S102","kind":"supports"},{"source_id":"S020","target_id":"S102","kind":"supports"},{"source_id":"S022","target_id":"S102","kind":"supports"},{"source_id":"S026","target_id":"S102","kind":"supports"},{"source_id":"S009","target_id":"S102","kind":"supports"},{"source_id":"S009","target_id":"S103","kind":"supports"},{"source_id":"S046","target_id":"S103","kind":"supports"},{"source_id":"S103","target_id":"S104","kind":"supports"},{"source_id":"S046","target_id":"S104","kind":"supports"},{"source_id":"S104","target_id":"S105","kind":"supports"},{"source_id":"S004","target_id":"S105","kind":"supports"},{"source_id":"S026","target_id":"S105","kind":"supports"},{"source_id":"S102","target_id":"S105","kind":"supports"},{"source_id":"S007","target_id":"S105","kind":"supports"},{"source_id":"S102","target_id":"S043","kind":"supports"},{"source_id":"S103","target_id":"S043","kind":"supports"},{"source_id":"S104","target_id":"S043","kind":"supports"},{"source_id":"S105","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S013","target_id":"S106","kind":"supports"},{"source_id":"S034","target_id":"S106","kind":"supports"},{"source_id":"S034","target_id":"S107","kind":"supports"},{"source_id":"S107","target_id":"S108","kind":"supports"},{"source_id":"S036","target_id":"S108","kind":"supports"},{"source_id":"S009","target_id":"S108","kind":"supports"},{"source_id":"S107","target_id":"S109","kind":"supports"},{"source_id":"S108","target_id":"S109","kind":"supports"},{"source_id":"S036","target_id":"S109","kind":"supports"},{"source_id":"S108","target_id":"S110","kind":"supports"},{"source_id":"S109","target_id":"S110","kind":"supports"},{"source_id":"S042","target_id":"S110","kind":"supports"},{"source_id":"S040","target_id":"S110","kind":"supports"},{"source_id":"S050","target_id":"S110","kind":"supports"},{"source_id":"S108","target_id":"S111","kind":"supports"},{"source_id":"S109","target_id":"S111","kind":"supports"},{"source_id":"S009","target_id":"S111","kind":"supports"},{"source_id":"S020","target_id":"S111","kind":"supports"},{"source_id":"S022","target_id":"S111","kind":"supports"},{"source_id":"S038","target_id":"S111","kind":"supports"},{"source_id":"S040","target_id":"S111","kind":"supports"},{"source_id":"S007","target_id":"S111","kind":"supports"},{"source_id":"S107","target_id":"S112","kind":"supports"},{"source_id":"S034","target_id":"S112","kind":"supports"},{"source_id":"S050","target_id":"S113","kind":"supports"},{"source_id":"S006","target_id":"S113","kind":"supports"},{"source_id":"S011","target_id":"S113","kind":"supports"},{"source_id":"S012","target_id":"S113","kind":"supports"},{"source_id":"S112","target_id":"S114","kind":"supports"},{"source_id":"S113","target_id":"S114","kind":"supports"},{"source_id":"S009","target_id":"S114","kind":"supports"},{"source_id":"S042","target_id":"S114","kind":"supports"},{"source_id":"S110","target_id":"S114","kind":"supports"},{"source_id":"S111","target_id":"S114","kind":"supports"},{"source_id":"S034","target_id":"S115","kind":"supports"},{"source_id":"S036","target_id":"S116","kind":"supports"},{"source_id":"S003","target_id":"S116","kind":"supports"},{"source_id":"S115","target_id":"S117","kind":"supports"},{"source_id":"S116","target_id":"S117","kind":"supports"},{"source_id":"S113","target_id":"S117","kind":"supports"},{"source_id":"S114","target_id":"S117","kind":"supports"},{"source_id":"S036","target_id":"S118","kind":"supports"},{"source_id":"S117","target_id":"S118","kind":"supports"},{"source_id":"S106","target_id":"S002","kind":"supports"},{"source_id":"S107","target_id":"S002","kind":"supports"},{"source_id":"S108","target_id":"S002","kind":"supports"},{"source_id":"S109","target_id":"S002","kind":"supports"},{"source_id":"S110","target_id":"S002","kind":"supports"},{"source_id":"S111","target_id":"S002","kind":"supports"},{"source_id":"S112","target_id":"S002","kind":"supports"},{"source_id":"S113","target_id":"S002","kind":"supports"},{"source_id":"S114","target_id":"S002","kind":"supports"},{"source_id":"S115","target_id":"S002","kind":"supports"},{"source_id":"S116","target_id":"S002","kind":"supports"},{"source_id":"S117","target_id":"S002","kind":"supports"},{"source_id":"S118","target_id":"S002","kind":"supports"}]}}
{"record_id":"2210.02531v2","paper":{"title":"Nonconvex quasi-variational inequalities: stability analysis and application to numerical optimization","authors":["Joydeep Dutta","Lahoussine Lafhim","Alain Zemkoho","Shenglong Zhou"],"arxiv_id":"2210.02531","version":"v2","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-024-02582-4","abstract_url":"https://arxiv.org/abs/2210.02531v2","pdf_url":"https://arxiv.org/pdf/2210.02531v2"},"context":{"raw_tex":"Our basic problem of interest in this paper is the quasi-variational inequality (QVI) problem to find $y\\in K(x, y)$, for a given parameter $x\\in \\mathbb{R}^n$, such that we have \n\\begin{equation}\\tag{QVI}\\label{VI}\n\\left\\langle f_0(x,y), \\;\\, \\varsigma - y \\right\\rangle \\geq 0 \\;\\;\\;\n \\forall \\varsigma\\in  K(x, y):= \\left\\{\\varsigma \\in \\mathbb{R}^m\\left|\\; g_0(x, y, \\varsigma)\\leq 0\\right.\\right\\},\n\\end{equation}\n\nwhere $f_0 : \\mathbb{R}^n\\times \\mathbb{R}^m \\rightarrow \\mathbb{R}^m$ is a continuously differentiable function\nand $K :\\mathbb{R}^n\\times \\mathbb{R}^m \\rightrightarrows \\mathbb{R}^m$ denotes the feasibility\nset-valued map defined by a function $g_0\\, : \\mathbb{R}^n\\times \\mathbb{R}^m\\times \\mathbb{R}^m \\rightarrow \\mathbb{R}^q$,\n which is also assumed to be continuously differentiable.\n\n\n\\begin{equation}\\label{S-Soultion-Map}\n\\begin{array}{c}\n  S(x):=\\left\\{y\\in K(x,y)\\left|\\; \\left\\langle f_0(x,y), \\;\\, \\varsigma - y \\right\\rangle \\geq 0  \\;\\;\\; \\forall \\varsigma\\in K(x, y)\\right.\\right\\}\n\\end{array}\n\\end{equation}\n\n\\begin{equation}\\label{VI-1}\n S(x):=\\left\\{y\\in K(x,y)\\left|\\;\\, y^\\top f_0(x,y)-\\varphi(x,y)\\leq 0\\right.\\right\\}\n%    y\\in K(x, y) \\mbox{ such that } f(x,y,y)-\\varphi(x,y)\\leq 0.\n\\end{equation}\n\nfor all $x\\in \\mathbb{R}^n$. Here, $\\varphi$ denotes the optimal value function\n\\begin{equation}\\label{varphi}\n   \\varphi(x,y):= \\underset{\\varsigma}\\min \\{\\varsigma^\\top f_0(x, y)|\\;\\; \\varsigma\\in K(x, y)\\}.\n\\end{equation}\n\nFor the points $(\\bar x, \\bar y)$, $(\\bar x, \\bar y, \\bar y)$, and $(\\bar x, \\bar y, \\bar\\varsigma)$, the following notation will be used to collect { {active index sets}} of constraints for the feasible sets associated to problems \\eqref{VI} and \\eqref{OPQVIC}, which are defined by the functions $(x, y) \\mapsto G(x, y)$, $(x, y) \\mapsto g_0(x, y, y)$, and $(x, y, \\varsigma) \\mapsto g_0(x, y, \\varsigma)$, respectively:\n\\begin{equation}\\label{I1}\n\\begin{array}{lclll}\n     I^1&:=& I^G(\\bar x, \\bar y)                   &:=&  \\left\\{i\\in \\{1, \\ldots, p\\}\\left|~G_i(\\bar x, \\bar y)=0\\right.\\right\\},\\\\[1ex]\n    I^2 &:=& I^{g_0}(\\bar x, \\bar y, \\bar y)       &:=& \\left\\{j\\in \\{1, \\ldots, q\\}|~g_{0_j}(\\bar x, \\bar y, \\bar y)=0\\right\\},\\\\[1ex]\n    I^3 &:=& I^{g_0}(\\bar x, \\bar y, \\bar\\varsigma)&:=& \\left\\{j\\in \\{1, \\ldots, q\\}|~g_{0_j}(\\bar x, \\bar y, \\bar\\varsigma)=0\\right\\}.\n\\end{array}\n\\end{equation}\n\n{ {Although symbols that we use here are very standard, it might be useful to make it clear how we use the symbol $\\nabla$ interchangeably between the gradient of a scalar function and the Jacobian of a vector-valued function. For a continuously differentiable function $\\psi : \\mathbb{R}^n\\times \\mathbb{R}^m \\times \\mathbb{R}^p \\rightarrow \\mathbb{R}^q$, in case there is ambiguity in the use of the usual partial derivatives, we will utilize $\\nabla_1 \\psi(x, y, \\varsigma)$, (resp. $\\nabla_2 \\psi(x, y, \\varsigma)$, $\\nabla_3 \\psi(x, y, \\varsigma)$) to represent the Jacobian of the function $x \\mapsto \\psi(x, y, \\varsigma)$ (resp. $y \\mapsto \\psi(x, y, \\varsigma)$, $\\varsigma \\mapsto \\psi(x, y, \\varsigma)$) when the vector $(y, \\varsigma)$ (resp. $(x, \\varsigma)$, $(y, \\varsigma)$) is fixed. A similar notation will be used for the gradient of $\\psi$ if it is a real-valued function. Common scenarios for $\\psi$ in this paper will be $\\psi(x, y, \\varsigma):= g_0(x, y, \\varsigma)$ and $\\psi(x, y, \\varsigma):= \\varsigma^\\top f_0(x, y)$ when studying our \\eqref{VI}.\n\n\nGiven that the optimal value function $\\varphi$ \\eqref{varphi} is nonsmooth in general, we need generalized concepts of\n differentiability to deal with it. We start the discussion on this by recalling the \\emph{generalized directional derivative} in the sense of\n Clarke \\cite{ClarkeBook1983}, which, for a function $\\psi : \\mathbb{R}^n \\rightarrow \\mathbb{R}$, is defined at a point $\\bar{x}\\in \\mathbb{R}^n$ in direction $d\\in \\mathbb{R}^n$ by\n \\begin{equation*}\\label{Clarke Directional Derivative}\n    \\psi^0(\\bar x; d):=\\underset{t\\downarrow 0}{\\underset{x \\rightarrow \\bar x}\\limsup} \\frac{1}{t}\\left[\\psi(x + td)-\\psi(x)\\right].\n\\end{equation*}\n This quantity exists if $\\psi$ is a Lipschitz continuous function around $\\bar x$ \\cite[Proposition 2.1.1]{ClarkeBook1983}.\n  Utilizing this notion, the \\textit{Clarke subdifferential} of $\\psi$ at the point $\\bar{x}$ can be defined by \n\\begin{equation}\\label{Clarke Subdifferential}\n    \\bar\\partial \\psi(\\bar x):= \\left\\{\\zeta\\in \\mathbb{R}^n|\\; \\psi^o(\\bar x; d)\\geq \\langle\\zeta, d\\rangle, \\; \\forall d\\in \\mathbb{R}^n  \\right\\}.\n\\end{equation}\nNote that {{$\\bar\\partial \\psi(\\bar x) = \\left\\{\\nabla \\psi(\\bar x)\\right\\}$ }} if $\\psi$ is continuously differentiable at $\\bar x$, and for a convex function,\nthis concept coincides with the subdifferential in the sense of convex analysis.\nFurthermore, this concept can be extended to a vector-valued function $\\psi : \\mathbb{R}^n \\rightarrow \\mathbb{R}^m$ that\nis Lipschitz continuous around a point $\\bar x$, as such a function is differentiable almost everywhere around this point. Hence,\n the \\emph{Clarke generalized Jacobian} of the function $\\psi$ at the point $\\bar{x}$ can be written as \n\\begin{equation}\\label{Clarke Subdifferential-Derivative}\n    \\bar\\partial \\psi(\\bar x):= \\mbox{co}\\left\\{\\lim\\nabla\\psi(x^n)\\left|~x^n \\rightarrow \\bar x, \\;\\, x^n\\in D_\\psi \\right.\\right\\},\n\\end{equation}\nwhere ``co'' stands for the convex hull and  $D_\\psi$ represents the set of points where $\\psi : \\mathbb{R}^n \\rightarrow \\mathbb{R}^m$ is\ndifferentiable. { {Note that the quantity defined in  \\eqref{Clarke Subdifferential-Derivative} coincides with the one in \\eqref{Clarke Subdifferential}\n for real-valued functions}}. Following  the expression in \\eqref{Clarke Subdifferential-Derivative},\n\n\n{ {In the sequel, the concept of basic normal cone will also be necessary. \n For a closed subset $C$ of $\\mathbb{R}^n$, the \\textit{basic} (also known as \\emph{Mordukhovich} or \\textit{limiting}) \\textit{normal cone}\n to $C$ at one of its points $\\bar x$ is the set\n\\begin{equation}\\label{basic normal cone}\n N_C(\\bar x):= \\left\\{v\\in \\mathbb{R}^n\\left|\\; \\exists v_k \\rightarrow v, \\, x_k \\rightarrow \\bar x \\,(x_k\\in C): \\, v_k\\in \\widehat{N}_C(x_k)\\right.\\right\\},\n\\end{equation}\nwhere $\\widehat{N}_C$ denotes the polar of the contingent/Boulingand tangent cone to $C$}}:\n\\begin{equation}\\label{polar normal cone}\n \\widehat{N}_C(\\bar x):=\\left\\{v\\in \\mathbb{R}^n \\left |\\; \\langle v, u-\\bar x\\rangle \\leq o(\\|u-\\bar x\\|)\\;\\; \\forall u\\in C\\right. \\right\\}.\n\\end{equation}\n$\\widehat{N}_C(\\bar{x})$ is also known as the \\textit{regular} or \\textit{Fr\\'{e}chet} normal cone. A set $C \\subset \\mathbb{R}^n$ is said to be \\textit{normally regular} at $\\bar{x} \\in C$ if $ N_C(\\bar{x}) = \\widehat{N}_C(\\bar{x})$. If $C$ is a convex set, it is normally regular at all points of $C$, given that both of the above cones coincide with the normal cone in the sense of convex analysis.\n\nThe regular normal cone allows us to introduce the notion of a \\textit{regular subdifferential} also known as the Fr\\'{e}chet subdifferential. Let the function $\\psi : \\mathbb{R}^n \\rightarrow \\mathbb{R}$ be Lipschitz continuous around $\\bar x$. Then its regular subdifferential at $\\bar{x} \\in \\mathbb{R}^n$ is given as \n\\[\n    \\hat{\\partial}\\psi(\\bar{x}) = \\left\\{v\\in \\mathbb{R}^n\\left|~(v, -1) \\in \\widehat{N}_{\\mbox{epi}\\psi} (\\bar{x}, \\psi(\\bar{x}))\\right.\\right\\},\n\\]\nwhere epi$\\psi$ denotes the epigraph of the function $\\psi$.\nLet us now introduce the notion of a \\textit{basic subdifferential} which is also known as the \\textit{limiting} subdifferential or the \\textit{Mordukhovich} subdifferential. For $\\psi : \\mathbb{R}^n \\rightarrow \\mathbb{R}$, which Lipschitz continous around $\\bar x$, its basic subdifferential at this point is given by \n\\[\n    \\partial\\psi(\\bar{x}) = \\left\\{v\\in \\mathbb{R}^n\\left|~(v, -1) \\in N_{\\mbox{epi}\\psi} (\\bar{x}, \\psi(\\bar{x}))\\right.\\right\\}.\n\\]\nConsidering the interplay between \\eqref{basic normal cone} and \\eqref{polar normal cone}, $v \\in \\partial \\psi(\\bar{x})$ if and only if there exist sequences $\\{x^k \\}$ with $x^k \\rightarrow \\bar{x}$ and $\\{v^k\\}$ with $ v^k \\rightarrow v$ such that $v^k \\in \\hat{\\partial} \\psi(x^k)$. The basic subdifferential for a locally Lipschitz function is always non-empty and compact. If $\\psi$ is a convex function, then both its regular and basic subdifferentials coincides with the subdifferential in the sense of convex analysis. Furthermore, if a function $\\psi : \\mathbb{R}^n \\rightarrow \\mathbb{R}$ is Lipschitz continous around a point $\\bar x$, then we have the nice relationship\n\\begin{equation}\\label{LimitingClarke}\n    \\bar{\\partial}\\psi = \\mbox{cl} \\mbox{co}\\partial \\psi (\\bar{x})\n\\end{equation}\nbetween the Clarke and basic subdifferentials; cf. \\cite[Theorem 3.57 of Volume I]{MordukhovichBook2006}, where $\\mbox{cl} \\mbox{co}$ stands for the closure of the convex hull of the corresponding set. Hence, the Clarke subdifferential is often referred to in some literature as the \\textit{convexified subdifferential}.\n\n\nSet-valued mappings and some related concepts will play a key role in the derivation of key results in this paper. To proceed, we consider a set-valued mapping $\\Psi :\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^m$. \n{ {The domain of $\\Psi$ is denoted as $\\text{dom}\\Psi$ and defined by\n$\\text{dom}\\Psi := \\{ x \\in \\mathbb{R}^n  \\left| \\ \\Psi(x) \\neq \\emptyset\\right\\}$,\nwhile its graph, denoted as $\\text{gph} \\Psi$, is given by\n\\begin{equation*}\n    \\text{gph} \\Psi  :=  \\{ (x, y) \\in \\mathbb{R}^m \\times \\mathbb{R}^m \\left | \\ y \\in \\Psi(x) \\right \\} = \\{ (x, y) \\in \\text{dom} \\Psi \\times \\mathbb{R}^m \\left|\\ y \\in \\Psi(x)\\right\\}.\n\\end{equation*}\n}} \n\n\n { {The set-valued mapping $\\Psi :\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^m$ is said to be \n { {inner semicontinuous at $(\\bar{x},\\bar{y})\\in \\mbox{gph}\\Psi$ w.r.t. $\\Omega\\in\\mathbb{R}^{n}$ whenever for each neighborhood $V\\subset \\mathbb{R}^{m}$ of $\\bar{y}$, there exists a neighborhood $U\\subset \\mathbb{R}^{n}$ of $\\bar{x}$ such that $\\Psi (x)\\cap V\\neq \\emptyset$ for all $x \\in \\Omega \\cap U$. If $\\Omega = \\mathbb{R}^{n}$ can be chosen, $\\Psi$ is called inner semicontinuous at $(\\bar{x},\\bar{y})$ for brevity; i.e., if for every sequence $x_k\\rightarrow\\bar{x}$, there is a sequence of $y_k\\in\\Psi(x_k)$ that converges to $\\bar{y}$ as $k\\rightarrow\\infty$. For single-valued mappings $\\Psi :\\mathbb{R}^n\\to\\mathbb{R}^m$, this property obviously reduces to the continuity of $\\Psi$ at $\\bar x$.}} %Furthermore, if $\\Psi :\\mathbb{R}^n \\rightrightarrows\\mathbb{R}^m$ is inner semicompact at $\\bar{x}$ with $\\Psi(\\bar{x})=\\{\\bar{y}\\}$, then $\\Psi$ is inner semicontinuous at $(\\bar x, \\bar y)$.\n Obviously, the inner semicontinuity property is more restrictive than the inner semicompactness while bringing us to more precise results of the coderivative calculus, as it will be clear in the next section.}} \n\n\nAs it will become clear in the subsequent sections, some Lipschitz-type properties will also play a central role in the main results. Hence, we introduce some useful concepts.  A set-valued mapping $\\Psi : \\mathbb{R}^{n}\\rightrightarrows \\mathbb{R}^{m}$\nis said to be \\emph{Lipschitz-like} (or satisfies the \\textit{Aubin property} \\cite{Aubin1984}) around $\\left(\\bar{x},\\bar{y}\\right)\\in \\mbox{gph}\\Psi$ if\n there exist neighborhoods $U$ of $\\bar{x}$, $V$ of $\\bar{y}$, and a constant $l > 0$ such that\n\\begin{equation}\\label{exact-estim}\n\t\\Psi\\left( x\\right) \\cap V \\subseteq \\Psi\\left( u\\right) +l\\|u-x\\|\\mathbb{B} \\ \\ \\text{for all} \\ x,u \\in U\n\\end{equation}\nwith $\\mathbb{B}$ denoting the unit ball in $\\mathbb{R}^{m}$ (see; e.g., \\cite{HenrionJouraniOutrataCalmness2002}). This property can be equivalently expressed as \n\\begin{equation}\\label{Aubin-Property}\n    d(y, \\Psi(u)) \\le l \\| x - u \\| \\,\\mbox{ for all }\\, x, u \\in U \\;\\, \\mbox{and}\\;\\, y \\in V \\cap\\Psi(x).\n\\end{equation}\nThe weaker calmness property can be obtained from \\eqref{Aubin-Property} by setting $ u = \\bar{x}$. Thus, the set-valued mapping $\\Psi :\\mathbb{R}^n \\rightrightarrows \\mathbb{R}^m$ will be said to be \\textit{calm} (or satisfy the {\\em calmness} property) at the point $(\\bar x, \\bar y)$ with $\\bar y\\in \\Psi(\\bar x)$, if there exist neighborhoods $U$ and $V$ of $\\bar x$ and $\\bar y$, respectively, and a constant $\\ell >0$ such that\n\\begin{equation}\\label{Definition Calmness}\nd(y, \\Psi(\\bar x))\\leq \\ell \\|x-\\bar x\\| \\,\\mbox{ for all }\\, y\\in V\\cap \\Psi(x), \\;\\, x\\in U.\n\\end{equation}\n\n\n  As an important application of calmness, note that based on \\cite[Theorem 4.1]{HenrionJouraniOutrataCalmness2002}, inclusion \\eqref{normal cone to operator constraint} also holds if the following set-valued mapping is calm at the point $(0, \\bar x)$ with $\\bar x\\in \\Omega \\cap \\psi^{-1}(\\Xi)$: \n\\begin{equation}\\label{CalmnesMapAgain}\n\\Psi(\\varsigma):=\\left\\{x\\in \\Omega\\left|\\;\\, \\psi(x) + \\varsigma \\in \\Xi\\right.\\right\\}.\n\\end{equation}\n\n  Next, we use the concept of basic normal cone introduced in the previous subsection to define the notion of {\\em coderivative}, which will allow us to present the most powerful characterization of the Lipschitz-likeness property.  For a given set-valued mapping $\\Psi :\\mathbb{R}^n \\rightrightarrows \\mathbb{R}^m$,  its coderivative at some point $(\\bar{x}, \\bar{y})\\in \\text{gph}\\, \\Psi$ corresponds to another set-valued mapping \n$D^*\\Psi(\\bar{x},\\bar{y}): \\mathbb{R}^m \\rightrightarrows \\mathbb{R}^n$ that is defined at any $y^*\\in \\mathbb{R}^m$ by\n\\begin{equation*}\\label{cod-definition}\n   D^*\\Psi(\\bar{x},\\bar{y})(y^*):= \\left\\{x^*\\in \\mathbb{R}^n|\\; (x^*, -y^*)\\in N_{\\text{gph}\\, \\Psi}(\\bar{x}, \\bar{y})\\right\\}.\n\\end{equation*}\nFor a single-valued locally Lipschitz continuous function $\\psi : \\mathbb{R}^n \\rightarrow \\mathbb{R}^m$, the coderivative reduces to  \n\\begin{equation*}\\label{coderivative-sub}\n    D^* \\psi (\\bar{x}) (y^*) = \\partial \\langle y^*,  \\psi \\rangle (\\bar{x}),\n\\end{equation*}\nwhere similarly to \\eqref{normal cone to operator constraint}, the term $\\partial \\langle y^*, \\psi \\rangle (\\bar x)$ denotes the basic subdifferential of the function $x \\mapsto \\sum^m_{i=1} y^*_i \\psi_i(x)$ at $\\bar x$.\nThe coderivative plays a key role in identifying whether a set-valued mapping is Lipschitz-like; i.e., the well-known \\textit{Murdokhovich} criterion (see, e.g., \\cite{RockafellarWetsBook1998,MordukhovichGeneralizedDifferential1994}), which says that for a set-valued mapping $\\Psi : \\mathbb{R}^{n}\\rightrightarrows \\mathbb{R}^{m}$, which has a closed graph around $(\\bar x, \\bar y)\\in \\text{gph}\\Psi$, it is Lipschitz-like around this point if and only if \n\\begin{equation*} \\label{Aubin}\n    D^* \\Psi ( \\bar{x}, \\bar{y})(0) = \\{0\\}.\n\\end{equation*}\nThanks to the concept of coderivative, we can also characterize the \\textit{Lipschitz modulus} of a set-valued mapping when it is Lipschitz-like \\eqref{exact-estim}. To present this characterization, we first introduce the notion a Lipschitz modulus of a set-valued mapping $\\Psi : \\mathbb{R}^{n}\\rightrightarrows \\mathbb{R}^{m}$. For this, we need  the notion of \\textit{outer norm}  %and \\textit{inner norm}  \nof a positively homogeneous set-valued mapping. { {Note that $\\Psi$ is said to be positively homogeneous if $0\\in \\Psi (0)$ and $\\Psi (\\lambda x) = \\lambda \\Psi (x) $ for all $\\lambda > 0$ and $ x \\in \\mathbb{R}^n$.}}\nIf $\\Psi$ is a positively homogeneous\nmapping, its \\emph{outer norm} is defined by\n\\begin{equation*}\n\t\\|\\Psi\\|^{+} :={\\displaystyle \\sup_{x\\in\\mathbb{B}}} \\\n{\\displaystyle \\sup_{u\\in\\Psi\\left( x\\right) }}\\|u\\|.  %\\ \\ \\ \\left(\\text{resp.} \\\n%\\|\\Psi\\|^{-} :={\\displaystyle \\sup_{x\\in\\mathbb{B}}} \\ {\\displaystyle \\inf_{u\\in\\Psi\\left( x\\right) }} \\|u\\|\\right).\n\\end{equation*}\nFurthermore, the infinimum of all $l > 0$ for which \\eqref{exact-estim} holds, also known as the \\emph{Lipschitz modulus} of $\\Psi$ at $\\left(\\bar{x},\\bar{y}\\right)$, is given as follows (cf. \\cite[Theorem 9.40]{RockafellarWetsBook1998}),  via the outer norm of the coderivative of the set-valued mapping $\\Psi$:\n\\begin{equation}\\label{LipModulus}\n\t\\text{lip} \\Psi \\left( \\bar{x},\\bar{y}\\right) := \\inf \\left\\{l\\in ]0, +\\infty[ \\ |\n\\eqref{exact-estim} \\ \\text{holds for some} \\  U \\ \\text{and} \\ V  \\right\\}=\\left\\|D^{\\ast}\\Psi\\left( \\bar{x},\\bar{y}\\right)\\right\\|^{+}.\n\\end{equation}\n\n\nThe plan here is to use the value function reformulation \\eqref{VI-1} of  $S$ \\eqref{S-Soultion-Map} to construct an estimate for its coderivative  in terms of the derivatives of $f_0$ and $g_0$, and subsequently, to leverage on this result to provide new rules for the Lipschitz-like property of this set-valued mapping. To proceed,  some further definitions and notations are in order. First, the Mangasarian-Fromowitz constraint qualification (MFCQ) for the constraint describing $K$ \\eqref{VI},  which will be said to hold at $(\\bar x, \\bar y, \\bar\\varsigma)$ if there exists a vector $d\\in \\mathbb{R}^m$ such that\n\\begin{equation}\\label{LL regularity}\n    \\nabla_3 g_{0j}(\\bar x, \\bar y, \\bar\\varsigma)^\\top d < 0\\;  \n    \\mbox{ for all }\\; j\\in I^3.  %:= I^3(\\bar x, \\bar y, \\bar\\varsigma):= \\left\\{i|\\; g_{0_{i}}(\\bar x, \\bar y, \\bar\\varsigma)=0\\right\\}\n\\end{equation}\n\nRecall that $I^3$ is defined in \\eqref{I1} and $\\nabla_3 g_{0j}(x, y, \\varsigma)$ represents the Jacobian of the function $\\varsigma \\mapsto g_{0j}(x, y, \\varsigma)$ for a fixed vector $(x, y)\\in \\mathbb{R}^n\\times \\mathbb{R}^m$.\nNext, we associate to a vector $(\\bar x, \\bar y, \\bar\\varsigma, \\varsigma^*)$ a set of  Lagrange multipliers\n\\begin{equation*}\n  \\Lambda^{0}(\\bar x, \\bar y, \\bar\\varsigma, \\varsigma^*)\n   :=  \\left\\{\\left(\\begin{array}{c}\n        v\\\\ w\\\\ \\lambda\n\\end{array}\\right)\\in \\mathbb{R}^{2q + 1} \\left|\n\\left.\\begin{array}{l}\n   \\lambda \\geq 0,\\; v\\geq 0, \\; g_0(\\bar x, \\bar y, \\bar{y})\\leq 0, \\; v^\\top g_0(\\bar x, \\bar y, \\bar{y})=0,\\;  w\\in \\Lambda(\\bar x, \\bar y, \\bar \\varsigma)\\\\[1ex]\n     \\varsigma^*  + \\lambda\\left.\\left(f_0(\\bar x, \\bar y) + \\nabla_2f_0(\\bar x, \\bar y)^\\top (\\bar y - \\bar\\varsigma) - \\nabla_2 g_{0}(\\bar x, \\bar y, \\bar\\varsigma)^{\\top}w  \\right)\\right.\\\\[1ex]\n      \\qquad \\qquad \\qquad \\qquad    \\qquad \\qquad \\qquad \\qquad + \\nabla_2 g_{0}(\\bar x, \\bar y, \\bar y)^{\\top}v=0\n      \\end{array}\\right.\\right.\\right\\} \n\\end{equation*}\n\nrelated to the  problem  \\eqref{VI}, with $\\nabla_2 f_{0}(x, y)$ denoting the Jacobian of the function $f_{0}(x, .)$ for a fixed vector $x\\in \\mathbb{R}^n$, while $\\nabla_2 g_{0}(x, y, \\varsigma)$ represents the Jacobian of the function $g_{0}(x, ., \\varsigma)$ with $(x, \\varsigma)\\in \\mathbb{R}^n\\times \\mathbb{R}^m$ fixed. Furthermore, recall that the other involved set of Lagrange multipliers %$\\Lambda(\\bar x, \\bar y, \\hat{y})$ is defined by\n\\begin{equation}\\label{Lambda(x,y)}\n\\begin{array}{rll}\n\\Lambda(x, y, \\varsigma) &:=& \\left\\{w\\in \\mathbb{R}^q\\left|\\;\\nabla_3\\ell(x, y, \\varsigma, w)=0,\\right.\\right.\\left. \\left. w\\geq 0, \\; g_0(x, y, \\varsigma)\\leq 0,\\; w^{\\top}g_0(x, y, \\varsigma)= 0 \\right.\\right\\}\n\\end{array}\n\\end{equation}\n\nis defined with $x\\in \\mathbb{R}^n$ and $(y,\\varsigma)\\in \\mathbb{R}^{2m}$. In \\eqref{Lambda(x,y)}, the function $\\ell$ represents the Lagrangian function associated with the underlying parametric optimization problem \\eqref{varphi}:\n\n\n\\begin{equation}\\label{ell}\n    \\ell(x, y, \\varsigma, w):=\\varsigma^\\top f_0(x, y)+ w^\\top g_0(x, y, \\varsigma).\n \\end{equation}\n\nFurthermore, in \\eqref{Lambda(x,y)},  $\\nabla_3 \\ell(x, y, \\varsigma, w) := f_0(x, y) + \\nabla_3 g_{0}(x, y, \\varsigma)^\\top w$ with $\\nabla_3 g_{0}(x, y, \\varsigma)$ denoting the Jacobian of the function $\\varsigma \\mapsto g_{0j}(x, y, \\varsigma)$\nwhen the vectors $x\\in \\mathbb{R}^n$ and $y\\in \\mathbb{R}^m$ are fixed.\n\n\n$S(x)=\\left\\{y\\in \\mathbb{R}^m|\\, y\\in S_0(x, y)\\right\\}$,\nwhere  $S_0 : \\mathbb{R}^n\\times \\mathbb{R}^m \\rightrightarrows \\mathbb{R}^m$ is given as\n\\begin{equation}\\label{S-v}\n S_0(x,y)  := \\arg\\underset{\\varsigma}\\min~\\left\\{\\varsigma^\\top f_0(x,y)\\left|\\; \\varsigma\\in K(x, y)\\right.\\right\\}.\n\\end{equation}\n\nwith the notation $z:=(x, y)$, $g(x,y):= g_0(x, y, y)$, and $f(x,y) := y^\\top f_0  (x,y)$, %introduced in Section \\ref{Basic concepts and background material}\nas well as the optimal value function $\\varphi$ defined in \\eqref{varphi}.\n","units":[{"start":9389,"end":9807,"start_line":105,"end_line":109,"raw_tex":"Our basic problem of interest in this paper is the quasi-variational inequality (QVI) problem to find $y\\in K(x, y)$, for a given parameter $x\\in \\mathbb{R}^n$, such that we have \n\\begin{equation}\\tag{QVI}\\label{VI}\n\\left\\langle f_0(x,y), \\;\\, \\varsigma - y \\right\\rangle \\geq 0 \\;\\;\\;\n \\forall \\varsigma\\in  K(x, y):= \\left\\{\\varsigma \\in \\mathbb{R}^m\\left|\\; g_0(x, y, \\varsigma)\\leq 0\\right.\\right\\},\n\\end{equation}","sha256_utf8":"3c250861b52dcb137b9bd54ea3cfa8a544d594bd1e212d5c8667bed0af4931bb"},{"start":9808,"end":10199,"start_line":110,"end_line":113,"raw_tex":"where $f_0 : \\mathbb{R}^n\\times \\mathbb{R}^m \\rightarrow \\mathbb{R}^m$ is a continuously differentiable function\nand $K :\\mathbb{R}^n\\times \\mathbb{R}^m \\rightrightarrows \\mathbb{R}^m$ denotes the feasibility\nset-valued map defined by a function $g_0\\, : \\mathbb{R}^n\\times \\mathbb{R}^m\\times \\mathbb{R}^m \\rightarrow \\mathbb{R}^q$,\n which is also assumed to be continuously differentiable.\n","sha256_utf8":"0f24621595d683f6a21b2bc84714822f6b8939e59bad8c455d20cc058ffe3bdd"},{"start":10624,"end":10856,"start_line":119,"end_line":123,"raw_tex":"\\begin{equation}\\label{S-Soultion-Map}\n\\begin{array}{c}\n  S(x):=\\left\\{y\\in K(x,y)\\left|\\; \\left\\langle f_0(x,y), \\;\\, \\varsigma - y \\right\\rangle \\geq 0  \\;\\;\\; \\forall \\varsigma\\in K(x, y)\\right.\\right\\}\n\\end{array}\n\\end{equation}","sha256_utf8":"10ed60a80754e45e1a1c38bfb87b20cc7d6dbe45510d0c89ecab48e0d89273b7"},{"start":12741,"end":12936,"start_line":135,"end_line":138,"raw_tex":"\\begin{equation}\\label{VI-1}\n S(x):=\\left\\{y\\in K(x,y)\\left|\\;\\, y^\\top f_0(x,y)-\\varphi(x,y)\\leq 0\\right.\\right\\}\n%    y\\in K(x, y) \\mbox{ such that } f(x,y,y)-\\varphi(x,y)\\leq 0.\n\\end{equation}","sha256_utf8":"808e7c93da3c3cfa663d17e5cfa8b22e1eecb44b19286dd2b6b3b23847074002"},{"start":12937,"end":13161,"start_line":139,"end_line":142,"raw_tex":"for all $x\\in \\mathbb{R}^n$. Here, $\\varphi$ denotes the optimal value function\n\\begin{equation}\\label{varphi}\n   \\varphi(x,y):= \\underset{\\varsigma}\\min \\{\\varsigma^\\top f_0(x, y)|\\;\\; \\varsigma\\in K(x, y)\\}.\n\\end{equation}","sha256_utf8":"f50d1696f2d473f5fad9214584c25ef47e753f26ec64d7821a0f73fa8780adde"},{"start":16295,"end":17201,"start_line":164,"end_line":171,"raw_tex":"For the points $(\\bar x, \\bar y)$, $(\\bar x, \\bar y, \\bar y)$, and $(\\bar x, \\bar y, \\bar\\varsigma)$, the following notation will be used to collect { {active index sets}} of constraints for the feasible sets associated to problems \\eqref{VI} and \\eqref{OPQVIC}, which are defined by the functions $(x, y) \\mapsto G(x, y)$, $(x, y) \\mapsto g_0(x, y, y)$, and $(x, y, \\varsigma) \\mapsto g_0(x, y, \\varsigma)$, respectively:\n\\begin{equation}\\label{I1}\n\\begin{array}{lclll}\n     I^1&:=& I^G(\\bar x, \\bar y)                   &:=&  \\left\\{i\\in \\{1, \\ldots, p\\}\\left|~G_i(\\bar x, \\bar y)=0\\right.\\right\\},\\\\[1ex]\n    I^2 &:=& I^{g_0}(\\bar x, \\bar y, \\bar y)       &:=& \\left\\{j\\in \\{1, \\ldots, q\\}|~g_{0_j}(\\bar x, \\bar y, \\bar y)=0\\right\\},\\\\[1ex]\n    I^3 &:=& I^{g_0}(\\bar x, \\bar y, \\bar\\varsigma)&:=& \\left\\{j\\in \\{1, \\ldots, q\\}|~g_{0_j}(\\bar x, \\bar y, \\bar\\varsigma)=0\\right\\}.\n\\end{array}\n\\end{equation}","sha256_utf8":"f9d937cccda16b05bf3847b336da96a8d312bfe4b182ec6453c8e666d3eb1d65"},{"start":18729,"end":19804,"start_line":189,"end_line":189,"raw_tex":"{ {Although symbols that we use here are very standard, it might be useful to make it clear how we use the symbol $\\nabla$ interchangeably between the gradient of a scalar function and the Jacobian of a vector-valued function. For a continuously differentiable function $\\psi : \\mathbb{R}^n\\times \\mathbb{R}^m \\times \\mathbb{R}^p \\rightarrow \\mathbb{R}^q$, in case there is ambiguity in the use of the usual partial derivatives, we will utilize $\\nabla_1 \\psi(x, y, \\varsigma)$, (resp. $\\nabla_2 \\psi(x, y, \\varsigma)$, $\\nabla_3 \\psi(x, y, \\varsigma)$) to represent the Jacobian of the function $x \\mapsto \\psi(x, y, \\varsigma)$ (resp. $y \\mapsto \\psi(x, y, \\varsigma)$, $\\varsigma \\mapsto \\psi(x, y, \\varsigma)$) when the vector $(y, \\varsigma)$ (resp. $(x, \\varsigma)$, $(y, \\varsigma)$) is fixed. A similar notation will be used for the gradient of $\\psi$ if it is a real-valued function. Common scenarios for $\\psi$ in this paper will be $\\psi(x, y, \\varsigma):= g_0(x, y, \\varsigma)$ and $\\psi(x, y, \\varsigma):= \\varsigma^\\top f_0(x, y)$ when studying our \\eqref{VI}.\n","sha256_utf8":"1f9cb743627846be8c6c278fa42f65752ef175fa5e07577c34d927156e525c59"},{"start":20790,"end":23092,"start_line":197,"end_line":218,"raw_tex":"Given that the optimal value function $\\varphi$ \\eqref{varphi} is nonsmooth in general, we need generalized concepts of\n differentiability to deal with it. We start the discussion on this by recalling the \\emph{generalized directional derivative} in the sense of\n Clarke \\cite{ClarkeBook1983}, which, for a function $\\psi : \\mathbb{R}^n \\rightarrow \\mathbb{R}$, is defined at a point $\\bar{x}\\in \\mathbb{R}^n$ in direction $d\\in \\mathbb{R}^n$ by\n \\begin{equation*}\\label{Clarke Directional Derivative}\n    \\psi^0(\\bar x; d):=\\underset{t\\downarrow 0}{\\underset{x \\rightarrow \\bar x}\\limsup} \\frac{1}{t}\\left[\\psi(x + td)-\\psi(x)\\right].\n\\end{equation*}\n This quantity exists if $\\psi$ is a Lipschitz continuous function around $\\bar x$ \\cite[Proposition 2.1.1]{ClarkeBook1983}.\n  Utilizing this notion, the \\textit{Clarke subdifferential} of $\\psi$ at the point $\\bar{x}$ can be defined by \n\\begin{equation}\\label{Clarke Subdifferential}\n    \\bar\\partial \\psi(\\bar x):= \\left\\{\\zeta\\in \\mathbb{R}^n|\\; \\psi^o(\\bar x; d)\\geq \\langle\\zeta, d\\rangle, \\; \\forall d\\in \\mathbb{R}^n  \\right\\}.\n\\end{equation}\nNote that {{$\\bar\\partial \\psi(\\bar x) = \\left\\{\\nabla \\psi(\\bar x)\\right\\}$ }} if $\\psi$ is continuously differentiable at $\\bar x$, and for a convex function,\nthis concept coincides with the subdifferential in the sense of convex analysis.\nFurthermore, this concept can be extended to a vector-valued function $\\psi : \\mathbb{R}^n \\rightarrow \\mathbb{R}^m$ that\nis Lipschitz continuous around a point $\\bar x$, as such a function is differentiable almost everywhere around this point. Hence,\n the \\emph{Clarke generalized Jacobian} of the function $\\psi$ at the point $\\bar{x}$ can be written as \n\\begin{equation}\\label{Clarke Subdifferential-Derivative}\n    \\bar\\partial \\psi(\\bar x):= \\mbox{co}\\left\\{\\lim\\nabla\\psi(x^n)\\left|~x^n \\rightarrow \\bar x, \\;\\, x^n\\in D_\\psi \\right.\\right\\},\n\\end{equation}\nwhere ``co'' stands for the convex hull and  $D_\\psi$ represents the set of points where $\\psi : \\mathbb{R}^n \\rightarrow \\mathbb{R}^m$ is\ndifferentiable. { {Note that the quantity defined in  \\eqref{Clarke Subdifferential-Derivative} coincides with the one in \\eqref{Clarke Subdifferential}\n for real-valued functions}}. Following  the expression in \\eqref{Clarke Subdifferential-Derivative},\n","sha256_utf8":"aafd8694e913b1eee3ce37ac09a910b165fb7a49f793a523b2c7e7134ab6086f"},{"start":24721,"end":28019,"start_line":238,"end_line":263,"raw_tex":"{ {In the sequel, the concept of basic normal cone will also be necessary. \n For a closed subset $C$ of $\\mathbb{R}^n$, the \\textit{basic} (also known as \\emph{Mordukhovich} or \\textit{limiting}) \\textit{normal cone}\n to $C$ at one of its points $\\bar x$ is the set\n\\begin{equation}\\label{basic normal cone}\n N_C(\\bar x):= \\left\\{v\\in \\mathbb{R}^n\\left|\\; \\exists v_k \\rightarrow v, \\, x_k \\rightarrow \\bar x \\,(x_k\\in C): \\, v_k\\in \\widehat{N}_C(x_k)\\right.\\right\\},\n\\end{equation}\nwhere $\\widehat{N}_C$ denotes the polar of the contingent/Boulingand tangent cone to $C$}}:\n\\begin{equation}\\label{polar normal cone}\n \\widehat{N}_C(\\bar x):=\\left\\{v\\in \\mathbb{R}^n \\left |\\; \\langle v, u-\\bar x\\rangle \\leq o(\\|u-\\bar x\\|)\\;\\; \\forall u\\in C\\right. \\right\\}.\n\\end{equation}\n$\\widehat{N}_C(\\bar{x})$ is also known as the \\textit{regular} or \\textit{Fr\\'{e}chet} normal cone. A set $C \\subset \\mathbb{R}^n$ is said to be \\textit{normally regular} at $\\bar{x} \\in C$ if $ N_C(\\bar{x}) = \\widehat{N}_C(\\bar{x})$. If $C$ is a convex set, it is normally regular at all points of $C$, given that both of the above cones coincide with the normal cone in the sense of convex analysis.\n\nThe regular normal cone allows us to introduce the notion of a \\textit{regular subdifferential} also known as the Fr\\'{e}chet subdifferential. Let the function $\\psi : \\mathbb{R}^n \\rightarrow \\mathbb{R}$ be Lipschitz continuous around $\\bar x$. Then its regular subdifferential at $\\bar{x} \\in \\mathbb{R}^n$ is given as \n\\[\n    \\hat{\\partial}\\psi(\\bar{x}) = \\left\\{v\\in \\mathbb{R}^n\\left|~(v, -1) \\in \\widehat{N}_{\\mbox{epi}\\psi} (\\bar{x}, \\psi(\\bar{x}))\\right.\\right\\},\n\\]\nwhere epi$\\psi$ denotes the epigraph of the function $\\psi$.\nLet us now introduce the notion of a \\textit{basic subdifferential} which is also known as the \\textit{limiting} subdifferential or the \\textit{Mordukhovich} subdifferential. For $\\psi : \\mathbb{R}^n \\rightarrow \\mathbb{R}$, which Lipschitz continous around $\\bar x$, its basic subdifferential at this point is given by \n\\[\n    \\partial\\psi(\\bar{x}) = \\left\\{v\\in \\mathbb{R}^n\\left|~(v, -1) \\in N_{\\mbox{epi}\\psi} (\\bar{x}, \\psi(\\bar{x}))\\right.\\right\\}.\n\\]\nConsidering the interplay between \\eqref{basic normal cone} and \\eqref{polar normal cone}, $v \\in \\partial \\psi(\\bar{x})$ if and only if there exist sequences $\\{x^k \\}$ with $x^k \\rightarrow \\bar{x}$ and $\\{v^k\\}$ with $ v^k \\rightarrow v$ such that $v^k \\in \\hat{\\partial} \\psi(x^k)$. The basic subdifferential for a locally Lipschitz function is always non-empty and compact. If $\\psi$ is a convex function, then both its regular and basic subdifferentials coincides with the subdifferential in the sense of convex analysis. Furthermore, if a function $\\psi : \\mathbb{R}^n \\rightarrow \\mathbb{R}$ is Lipschitz continous around a point $\\bar x$, then we have the nice relationship\n\\begin{equation}\\label{LimitingClarke}\n    \\bar{\\partial}\\psi = \\mbox{cl} \\mbox{co}\\partial \\psi (\\bar{x})\n\\end{equation}\nbetween the Clarke and basic subdifferentials; cf. \\cite[Theorem 3.57 of Volume I]{MordukhovichBook2006}, where $\\mbox{cl} \\mbox{co}$ stands for the closure of the convex hull of the corresponding set. Hence, the Clarke subdifferential is often referred to in some literature as the \\textit{convexified subdifferential}.\n","sha256_utf8":"bb3a14d1b0a89618446477a45be3f875d0dd4b8768c7c3bcdae1796c2a2cd940"},{"start":30397,"end":31049,"start_line":288,"end_line":295,"raw_tex":"Set-valued mappings and some related concepts will play a key role in the derivation of key results in this paper. To proceed, we consider a set-valued mapping $\\Psi :\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^m$. \n{ {The domain of $\\Psi$ is denoted as $\\text{dom}\\Psi$ and defined by\n$\\text{dom}\\Psi := \\{ x \\in \\mathbb{R}^n  \\left| \\ \\Psi(x) \\neq \\emptyset\\right\\}$,\nwhile its graph, denoted as $\\text{gph} \\Psi$, is given by\n\\begin{equation*}\n    \\text{gph} \\Psi  :=  \\{ (x, y) \\in \\mathbb{R}^m \\times \\mathbb{R}^m \\left | \\ y \\in \\Psi(x) \\right \\} = \\{ (x, y) \\in \\text{dom} \\Psi \\times \\mathbb{R}^m \\left|\\ y \\in \\Psi(x)\\right\\}.\n\\end{equation*}\n}} \n","sha256_utf8":"91b1b2cad1964a80e762999ae4235260bee076367b92216dbea35014b63a9aac"},{"start":31618,"end":32831,"start_line":298,"end_line":300,"raw_tex":" { {The set-valued mapping $\\Psi :\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^m$ is said to be \n { {inner semicontinuous at $(\\bar{x},\\bar{y})\\in \\mbox{gph}\\Psi$ w.r.t. $\\Omega\\in\\mathbb{R}^{n}$ whenever for each neighborhood $V\\subset \\mathbb{R}^{m}$ of $\\bar{y}$, there exists a neighborhood $U\\subset \\mathbb{R}^{n}$ of $\\bar{x}$ such that $\\Psi (x)\\cap V\\neq \\emptyset$ for all $x \\in \\Omega \\cap U$. If $\\Omega = \\mathbb{R}^{n}$ can be chosen, $\\Psi$ is called inner semicontinuous at $(\\bar{x},\\bar{y})$ for brevity; i.e., if for every sequence $x_k\\rightarrow\\bar{x}$, there is a sequence of $y_k\\in\\Psi(x_k)$ that converges to $\\bar{y}$ as $k\\rightarrow\\infty$. For single-valued mappings $\\Psi :\\mathbb{R}^n\\to\\mathbb{R}^m$, this property obviously reduces to the continuity of $\\Psi$ at $\\bar x$.}} %Furthermore, if $\\Psi :\\mathbb{R}^n \\rightrightarrows\\mathbb{R}^m$ is inner semicompact at $\\bar{x}$ with $\\Psi(\\bar{x})=\\{\\bar{y}\\}$, then $\\Psi$ is inner semicontinuous at $(\\bar x, \\bar y)$.\n Obviously, the inner semicontinuity property is more restrictive than the inner semicompactness while bringing us to more precise results of the coderivative calculus, as it will be clear in the next section.}} \n","sha256_utf8":"62b91b26426fa593bd481fdeaf705b6485546f74b096684b5398fdbea7a6de95"},{"start":32832,"end":34412,"start_line":302,"end_line":315,"raw_tex":"As it will become clear in the subsequent sections, some Lipschitz-type properties will also play a central role in the main results. Hence, we introduce some useful concepts.  A set-valued mapping $\\Psi : \\mathbb{R}^{n}\\rightrightarrows \\mathbb{R}^{m}$\nis said to be \\emph{Lipschitz-like} (or satisfies the \\textit{Aubin property} \\cite{Aubin1984}) around $\\left(\\bar{x},\\bar{y}\\right)\\in \\mbox{gph}\\Psi$ if\n there exist neighborhoods $U$ of $\\bar{x}$, $V$ of $\\bar{y}$, and a constant $l > 0$ such that\n\\begin{equation}\\label{exact-estim}\n\t\\Psi\\left( x\\right) \\cap V \\subseteq \\Psi\\left( u\\right) +l\\|u-x\\|\\mathbb{B} \\ \\ \\text{for all} \\ x,u \\in U\n\\end{equation}\nwith $\\mathbb{B}$ denoting the unit ball in $\\mathbb{R}^{m}$ (see; e.g., \\cite{HenrionJouraniOutrataCalmness2002}). This property can be equivalently expressed as \n\\begin{equation}\\label{Aubin-Property}\n    d(y, \\Psi(u)) \\le l \\| x - u \\| \\,\\mbox{ for all }\\, x, u \\in U \\;\\, \\mbox{and}\\;\\, y \\in V \\cap\\Psi(x).\n\\end{equation}\nThe weaker calmness property can be obtained from \\eqref{Aubin-Property} by setting $ u = \\bar{x}$. Thus, the set-valued mapping $\\Psi :\\mathbb{R}^n \\rightrightarrows \\mathbb{R}^m$ will be said to be \\textit{calm} (or satisfy the {\\em calmness} property) at the point $(\\bar x, \\bar y)$ with $\\bar y\\in \\Psi(\\bar x)$, if there exist neighborhoods $U$ and $V$ of $\\bar x$ and $\\bar y$, respectively, and a constant $\\ell >0$ such that\n\\begin{equation}\\label{Definition Calmness}\nd(y, \\Psi(\\bar x))\\leq \\ell \\|x-\\bar x\\| \\,\\mbox{ for all }\\, y\\in V\\cap \\Psi(x), \\;\\, x\\in U.\n\\end{equation}\n","sha256_utf8":"953a8abc867b58fe3b602cb15b9e9f842f5055b9f9d978344b96dfff4c31ca80"},{"start":35043,"end":38860,"start_line":320,"end_line":352,"raw_tex":"  As an important application of calmness, note that based on \\cite[Theorem 4.1]{HenrionJouraniOutrataCalmness2002}, inclusion \\eqref{normal cone to operator constraint} also holds if the following set-valued mapping is calm at the point $(0, \\bar x)$ with $\\bar x\\in \\Omega \\cap \\psi^{-1}(\\Xi)$: \n\\begin{equation}\\label{CalmnesMapAgain}\n\\Psi(\\varsigma):=\\left\\{x\\in \\Omega\\left|\\;\\, \\psi(x) + \\varsigma \\in \\Xi\\right.\\right\\}.\n\\end{equation}\n\n  Next, we use the concept of basic normal cone introduced in the previous subsection to define the notion of {\\em coderivative}, which will allow us to present the most powerful characterization of the Lipschitz-likeness property.  For a given set-valued mapping $\\Psi :\\mathbb{R}^n \\rightrightarrows \\mathbb{R}^m$,  its coderivative at some point $(\\bar{x}, \\bar{y})\\in \\text{gph}\\, \\Psi$ corresponds to another set-valued mapping \n$D^*\\Psi(\\bar{x},\\bar{y}): \\mathbb{R}^m \\rightrightarrows \\mathbb{R}^n$ that is defined at any $y^*\\in \\mathbb{R}^m$ by\n\\begin{equation*}\\label{cod-definition}\n   D^*\\Psi(\\bar{x},\\bar{y})(y^*):= \\left\\{x^*\\in \\mathbb{R}^n|\\; (x^*, -y^*)\\in N_{\\text{gph}\\, \\Psi}(\\bar{x}, \\bar{y})\\right\\}.\n\\end{equation*}\nFor a single-valued locally Lipschitz continuous function $\\psi : \\mathbb{R}^n \\rightarrow \\mathbb{R}^m$, the coderivative reduces to  \n\\begin{equation*}\\label{coderivative-sub}\n    D^* \\psi (\\bar{x}) (y^*) = \\partial \\langle y^*,  \\psi \\rangle (\\bar{x}),\n\\end{equation*}\nwhere similarly to \\eqref{normal cone to operator constraint}, the term $\\partial \\langle y^*, \\psi \\rangle (\\bar x)$ denotes the basic subdifferential of the function $x \\mapsto \\sum^m_{i=1} y^*_i \\psi_i(x)$ at $\\bar x$.\nThe coderivative plays a key role in identifying whether a set-valued mapping is Lipschitz-like; i.e., the well-known \\textit{Murdokhovich} criterion (see, e.g., \\cite{RockafellarWetsBook1998,MordukhovichGeneralizedDifferential1994}), which says that for a set-valued mapping $\\Psi : \\mathbb{R}^{n}\\rightrightarrows \\mathbb{R}^{m}$, which has a closed graph around $(\\bar x, \\bar y)\\in \\text{gph}\\Psi$, it is Lipschitz-like around this point if and only if \n\\begin{equation*} \\label{Aubin}\n    D^* \\Psi ( \\bar{x}, \\bar{y})(0) = \\{0\\}.\n\\end{equation*}\nThanks to the concept of coderivative, we can also characterize the \\textit{Lipschitz modulus} of a set-valued mapping when it is Lipschitz-like \\eqref{exact-estim}. To present this characterization, we first introduce the notion a Lipschitz modulus of a set-valued mapping $\\Psi : \\mathbb{R}^{n}\\rightrightarrows \\mathbb{R}^{m}$. For this, we need  the notion of \\textit{outer norm}  %and \\textit{inner norm}  \nof a positively homogeneous set-valued mapping. { {Note that $\\Psi$ is said to be positively homogeneous if $0\\in \\Psi (0)$ and $\\Psi (\\lambda x) = \\lambda \\Psi (x) $ for all $\\lambda > 0$ and $ x \\in \\mathbb{R}^n$.}}\nIf $\\Psi$ is a positively homogeneous\nmapping, its \\emph{outer norm} is defined by\n\\begin{equation*}\n\t\\|\\Psi\\|^{+} :={\\displaystyle \\sup_{x\\in\\mathbb{B}}} \\\n{\\displaystyle \\sup_{u\\in\\Psi\\left( x\\right) }}\\|u\\|.  %\\ \\ \\ \\left(\\text{resp.} \\\n%\\|\\Psi\\|^{-} :={\\displaystyle \\sup_{x\\in\\mathbb{B}}} \\ {\\displaystyle \\inf_{u\\in\\Psi\\left( x\\right) }} \\|u\\|\\right).\n\\end{equation*}\nFurthermore, the infinimum of all $l > 0$ for which \\eqref{exact-estim} holds, also known as the \\emph{Lipschitz modulus} of $\\Psi$ at $\\left(\\bar{x},\\bar{y}\\right)$, is given as follows (cf. \\cite[Theorem 9.40]{RockafellarWetsBook1998}),  via the outer norm of the coderivative of the set-valued mapping $\\Psi$:\n\\begin{equation}\\label{LipModulus}\n\t\\text{lip} \\Psi \\left( \\bar{x},\\bar{y}\\right) := \\inf \\left\\{l\\in ]0, +\\infty[ \\ |\n\\eqref{exact-estim} \\ \\text{holds for some} \\  U \\ \\text{and} \\ V  \\right\\}=\\left\\|D^{\\ast}\\Psi\\left( \\bar{x},\\bar{y}\\right)\\right\\|^{+}.\n\\end{equation}\n","sha256_utf8":"553dfa68ae6629cd1f5c3a71d201c3a207e6090e25b6f91bc46f2d4dab41cd43"},{"start":40294,"end":41169,"start_line":377,"end_line":381,"raw_tex":"The plan here is to use the value function reformulation \\eqref{VI-1} of  $S$ \\eqref{S-Soultion-Map} to construct an estimate for its coderivative  in terms of the derivatives of $f_0$ and $g_0$, and subsequently, to leverage on this result to provide new rules for the Lipschitz-like property of this set-valued mapping. To proceed,  some further definitions and notations are in order. First, the Mangasarian-Fromowitz constraint qualification (MFCQ) for the constraint describing $K$ \\eqref{VI},  which will be said to hold at $(\\bar x, \\bar y, \\bar\\varsigma)$ if there exists a vector $d\\in \\mathbb{R}^m$ such that\n\\begin{equation}\\label{LL regularity}\n    \\nabla_3 g_{0j}(\\bar x, \\bar y, \\bar\\varsigma)^\\top d < 0\\;  \n    \\mbox{ for all }\\; j\\in I^3.  %:= I^3(\\bar x, \\bar y, \\bar\\varsigma):= \\left\\{i|\\; g_{0_{i}}(\\bar x, \\bar y, \\bar\\varsigma)=0\\right\\}\n\\end{equation}","sha256_utf8":"f608d5cc0345a277d66f1ac61588703ad1de4abf3ce268c3bd91af6e7bb81b25"},{"start":41170,"end":42254,"start_line":382,"end_line":394,"raw_tex":"Recall that $I^3$ is defined in \\eqref{I1} and $\\nabla_3 g_{0j}(x, y, \\varsigma)$ represents the Jacobian of the function $\\varsigma \\mapsto g_{0j}(x, y, \\varsigma)$ for a fixed vector $(x, y)\\in \\mathbb{R}^n\\times \\mathbb{R}^m$.\nNext, we associate to a vector $(\\bar x, \\bar y, \\bar\\varsigma, \\varsigma^*)$ a set of  Lagrange multipliers\n\\begin{equation*}\n  \\Lambda^{0}(\\bar x, \\bar y, \\bar\\varsigma, \\varsigma^*)\n   :=  \\left\\{\\left(\\begin{array}{c}\n        v\\\\ w\\\\ \\lambda\n\\end{array}\\right)\\in \\mathbb{R}^{2q + 1} \\left|\n\\left.\\begin{array}{l}\n   \\lambda \\geq 0,\\; v\\geq 0, \\; g_0(\\bar x, \\bar y, \\bar{y})\\leq 0, \\; v^\\top g_0(\\bar x, \\bar y, \\bar{y})=0,\\;  w\\in \\Lambda(\\bar x, \\bar y, \\bar \\varsigma)\\\\[1ex]\n     \\varsigma^*  + \\lambda\\left.\\left(f_0(\\bar x, \\bar y) + \\nabla_2f_0(\\bar x, \\bar y)^\\top (\\bar y - \\bar\\varsigma) - \\nabla_2 g_{0}(\\bar x, \\bar y, \\bar\\varsigma)^{\\top}w  \\right)\\right.\\\\[1ex]\n      \\qquad \\qquad \\qquad \\qquad    \\qquad \\qquad \\qquad \\qquad + \\nabla_2 g_{0}(\\bar x, \\bar y, \\bar y)^{\\top}v=0\n      \\end{array}\\right.\\right.\\right\\} \n\\end{equation*}","sha256_utf8":"c5f85d109602ddcebe3de3587422945713ad65e64eab87b7892260b1760e0277"},{"start":43351,"end":44095,"start_line":412,"end_line":417,"raw_tex":"related to the  problem  \\eqref{VI}, with $\\nabla_2 f_{0}(x, y)$ denoting the Jacobian of the function $f_{0}(x, .)$ for a fixed vector $x\\in \\mathbb{R}^n$, while $\\nabla_2 g_{0}(x, y, \\varsigma)$ represents the Jacobian of the function $g_{0}(x, ., \\varsigma)$ with $(x, \\varsigma)\\in \\mathbb{R}^n\\times \\mathbb{R}^m$ fixed. Furthermore, recall that the other involved set of Lagrange multipliers %$\\Lambda(\\bar x, \\bar y, \\hat{y})$ is defined by\n\\begin{equation}\\label{Lambda(x,y)}\n\\begin{array}{rll}\n\\Lambda(x, y, \\varsigma) &:=& \\left\\{w\\in \\mathbb{R}^q\\left|\\;\\nabla_3\\ell(x, y, \\varsigma, w)=0,\\right.\\right.\\left. \\left. w\\geq 0, \\; g_0(x, y, \\varsigma)\\leq 0,\\; w^{\\top}g_0(x, y, \\varsigma)= 0 \\right.\\right\\}\n\\end{array}\n\\end{equation}","sha256_utf8":"16b226971b5ad85c6fc01d6e709dbb855af19a366ae92d68b17e8c98f75eb73c"},{"start":44096,"end":44330,"start_line":418,"end_line":418,"raw_tex":"is defined with $x\\in \\mathbb{R}^n$ and $(y,\\varsigma)\\in \\mathbb{R}^{2m}$. In \\eqref{Lambda(x,y)}, the function $\\ell$ represents the Lagrangian function associated with the underlying parametric optimization problem \\eqref{varphi}:\n","sha256_utf8":"bd1dbc2cca76c40fd3dcd16bd11eebdc517bea07af19945c1f9e39429bf3979a"},{"start":44331,"end":44459,"start_line":419,"end_line":421,"raw_tex":"\\begin{equation}\\label{ell}\n    \\ell(x, y, \\varsigma, w):=\\varsigma^\\top f_0(x, y)+ w^\\top g_0(x, y, \\varsigma).\n \\end{equation}","sha256_utf8":"f63ab3307752012fd615a3acc657b54931a3acf94f238551e09f92abac7222fd"},{"start":44460,"end":44781,"start_line":422,"end_line":423,"raw_tex":"Furthermore, in \\eqref{Lambda(x,y)},  $\\nabla_3 \\ell(x, y, \\varsigma, w) := f_0(x, y) + \\nabla_3 g_{0}(x, y, \\varsigma)^\\top w$ with $\\nabla_3 g_{0}(x, y, \\varsigma)$ denoting the Jacobian of the function $\\varsigma \\mapsto g_{0j}(x, y, \\varsigma)$\nwhen the vectors $x\\in \\mathbb{R}^n$ and $y\\in \\mathbb{R}^m$ are fixed.\n","sha256_utf8":"5e7127163710bd162c47e0d76458fccc943eabd91579b7448f7e38ec963654bd"},{"start":45452,"end":45762,"start_line":431,"end_line":435,"raw_tex":"$S(x)=\\left\\{y\\in \\mathbb{R}^m|\\, y\\in S_0(x, y)\\right\\}$,\nwhere  $S_0 : \\mathbb{R}^n\\times \\mathbb{R}^m \\rightrightarrows \\mathbb{R}^m$ is given as\n\\begin{equation}\\label{S-v}\n S_0(x,y)  := \\arg\\underset{\\varsigma}\\min~\\left\\{\\varsigma^\\top f_0(x,y)\\left|\\; \\varsigma\\in K(x, y)\\right.\\right\\}.\n\\end{equation}","sha256_utf8":"92d6425ee52184203a537cfc0c8e03d16de41e72d8aa490d39572db8b70a4747"},{"start":84019,"end":84253,"start_line":912,"end_line":913,"raw_tex":"with the notation $z:=(x, y)$, $g(x,y):= g_0(x, y, y)$, and $f(x,y) := y^\\top f_0  (x,y)$, %introduced in Section \\ref{Basic concepts and background material}\nas well as the optimal value function $\\varphi$ defined in \\eqref{varphi}.\n","sha256_utf8":"5fc8e0ee93f3b96b3f55c9ab70ae6b4fa65bcf26e01859191a05e7bb7faf3ea7"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{I}\nConsider a point $(\\bar x, \\bar y)$ such that $\\bar y\\in S(\\bar x)$ and let $\\bar{\\varsigma} \\in S_0(\\bar x, \\bar y)$.\n\\begin{itemize}\n  \\item[\\emph{(i)}]Suppose that the MFCQ \\eqref{LL regularity} holds at $(\\bar x, \\bar y, \\bar y)$ and $(\\bar x, \\bar y, \\bar{\\varsigma})$.\n  Furthermore, assume that $S_0$ \\eqref{S-v} is inner semicontinuous at $(\\bar x, \\bar y, \\bar{\\varsigma})$ and let the set-valued mapping\n\\begin{equation}\\label{PsiCalm}\n\\Psi(\\theta):=\\left\\{(x,y)\\in \\mathbb{R}^n \\times \\mathbb{R}^m\\left|\\; g_0(x, y, y)\\leq 0, \\;\\; y^\\top f_0(x, y) - \\varphi(x, y)\\leq \\theta \\right.\\right\\}\n\\end{equation}\n be calm at $(0, \\bar x, \\bar y)$. Then for all $y^*\\in \\mathbb{R}^m$, it holds that \n{\\small{\n\\begin{equation}\\label{CoD S}\n\\begin{array}{l}\n  D^*S(\\bar x, \\bar y)(y^*) \\subseteq\n   \\underset{(v, w, \\lambda) \\in \\Lambda^0 (\\bar x, \\bar y, \\bar{\\varsigma}, y^*)} \\bigcup\n  % \\left\\{\\nabla_1 g_0(\\bar x, \\bar y, \\bar y)^\\top v  \\underset{(v, w, \\lambda) \\in \\Lambda^0 (\\bar x, \\bar y, \\bar{\\varsigma}, y^*)}  \\bigcup\n \\left\\{\\nabla_1 g_0(\\bar x, \\bar y, \\bar y)^\\top v + \\lambda \\left(\\displaystyle \n  \\nabla_1 f_{0}(\\bar x, \\bar y)^\\top  (\\bar y- \\bar{\\varsigma})  - \\nabla_1 g_0(\\bar x, \\bar y, \\bar{\\varsigma})^\\top w\\right)\\right\\}.\n\\end{array}\n\\end{equation}}}\n\\item[\\emph{(ii)}] Suppose that in addition to the assumptions in \\emph{(i)}, the following qualification condition holds:\n{\\small{\n\\begin{equation}\\label{Stab CQ-1}\n\\begin{array}{l}\n(v, w, \\lambda) \\in \\Lambda^0(\\bar x, \\bar y, \\bar{\\varsigma}, 0) \\;\\; \\Longrightarrow \\;\\;  \n  \\left[\\nabla_1 g_0(\\bar x, \\bar y, \\bar y)^\\top v\n + \\lambda\\left(\\displaystyle \\nabla_1 f_{0}(\\bar x, \\bar y)^\\top  (\\bar y- \\bar{\\varsigma})\n    - \\nabla_1 g_0(\\bar x, \\bar y, \\bar{\\varsigma})^\\top w\\right) = 0\\right].\n\\end{array}\n\\end{equation}}}\n${}$\\\\\nThen the set-valued mapping $S$ \\eqref{S-Soultion-Map}  is Lipschitz-like around $(\\bar x, \\bar y)$ and its Lipschitz modulus at this point has the following upper estimate:\n\\[\n\\text{\\emph{lip}}\\, S(\\bar x, \\bar y) \\leq \\underset{\\underset{(v, w, \\lambda)\\in \\Lambda^0(\\bar x, \\bar y, \\bar{\\varsigma}, y^*)}{\\|y^*\\|\\leq 1}}{\\sup}\n\\left\\{\\left\\|\\nabla_1 g_0(\\bar x, \\bar y, \\bar y)^\\top v + \\lambda\\left(\\nabla_1 f_{0}(\\bar x, \\bar y)^\\top  (\\bar y- \\bar{\\varsigma})\n-\\nabla_1 g_0(\\bar x,\\bar y,\\bar{\\varsigma})^\\top w\\right)\\right\\|\\right\\}.\n\\]\n\\end{itemize}\n\n\\end{theorem}","units":[{"start":45763,"end":48179,"start_line":436,"end_line":474,"raw_tex":"\\begin{theorem}\\label{I}\nConsider a point $(\\bar x, \\bar y)$ such that $\\bar y\\in S(\\bar x)$ and let $\\bar{\\varsigma} \\in S_0(\\bar x, \\bar y)$.\n\\begin{itemize}\n  \\item[\\emph{(i)}]Suppose that the MFCQ \\eqref{LL regularity} holds at $(\\bar x, \\bar y, \\bar y)$ and $(\\bar x, \\bar y, \\bar{\\varsigma})$.\n  Furthermore, assume that $S_0$ \\eqref{S-v} is inner semicontinuous at $(\\bar x, \\bar y, \\bar{\\varsigma})$ and let the set-valued mapping\n\\begin{equation}\\label{PsiCalm}\n\\Psi(\\theta):=\\left\\{(x,y)\\in \\mathbb{R}^n \\times \\mathbb{R}^m\\left|\\; g_0(x, y, y)\\leq 0, \\;\\; y^\\top f_0(x, y) - \\varphi(x, y)\\leq \\theta \\right.\\right\\}\n\\end{equation}\n be calm at $(0, \\bar x, \\bar y)$. Then for all $y^*\\in \\mathbb{R}^m$, it holds that \n{\\small{\n\\begin{equation}\\label{CoD S}\n\\begin{array}{l}\n  D^*S(\\bar x, \\bar y)(y^*) \\subseteq\n   \\underset{(v, w, \\lambda) \\in \\Lambda^0 (\\bar x, \\bar y, \\bar{\\varsigma}, y^*)} \\bigcup\n  % \\left\\{\\nabla_1 g_0(\\bar x, \\bar y, \\bar y)^\\top v  \\underset{(v, w, \\lambda) \\in \\Lambda^0 (\\bar x, \\bar y, \\bar{\\varsigma}, y^*)}  \\bigcup\n \\left\\{\\nabla_1 g_0(\\bar x, \\bar y, \\bar y)^\\top v + \\lambda \\left(\\displaystyle \n  \\nabla_1 f_{0}(\\bar x, \\bar y)^\\top  (\\bar y- \\bar{\\varsigma})  - \\nabla_1 g_0(\\bar x, \\bar y, \\bar{\\varsigma})^\\top w\\right)\\right\\}.\n\\end{array}\n\\end{equation}}}\n\\item[\\emph{(ii)}] Suppose that in addition to the assumptions in \\emph{(i)}, the following qualification condition holds:\n{\\small{\n\\begin{equation}\\label{Stab CQ-1}\n\\begin{array}{l}\n(v, w, \\lambda) \\in \\Lambda^0(\\bar x, \\bar y, \\bar{\\varsigma}, 0) \\;\\; \\Longrightarrow \\;\\;  \n  \\left[\\nabla_1 g_0(\\bar x, \\bar y, \\bar y)^\\top v\n + \\lambda\\left(\\displaystyle \\nabla_1 f_{0}(\\bar x, \\bar y)^\\top  (\\bar y- \\bar{\\varsigma})\n    - \\nabla_1 g_0(\\bar x, \\bar y, \\bar{\\varsigma})^\\top w\\right) = 0\\right].\n\\end{array}\n\\end{equation}}}\n${}$\\\\\nThen the set-valued mapping $S$ \\eqref{S-Soultion-Map}  is Lipschitz-like around $(\\bar x, \\bar y)$ and its Lipschitz modulus at this point has the following upper estimate:\n\\[\n\\text{\\emph{lip}}\\, S(\\bar x, \\bar y) \\leq \\underset{\\underset{(v, w, \\lambda)\\in \\Lambda^0(\\bar x, \\bar y, \\bar{\\varsigma}, y^*)}{\\|y^*\\|\\leq 1}}{\\sup}\n\\left\\{\\left\\|\\nabla_1 g_0(\\bar x, \\bar y, \\bar y)^\\top v + \\lambda\\left(\\nabla_1 f_{0}(\\bar x, \\bar y)^\\top  (\\bar y- \\bar{\\varsigma})\n-\\nabla_1 g_0(\\bar x,\\bar y,\\bar{\\varsigma})^\\top w\\right)\\right\\|\\right\\}.\n\\]\n\\end{itemize}\n\n\\end{theorem}","sha256_utf8":"73887d1b9da36d9dc3ddefdcc5a97bb0c2533f23baa9ced96caa77ac9262ff86"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}[see Theorem 6.10 in \\cite{MordukhovichGeneralizedDifferential1994} or Theorem 6.14 in  \\cite{RockafellarWetsBook1998}]\\label{normal cone estimate}\nLet $C:=\\Omega \\cap \\psi^{-1}(\\Xi)$, where $\\Omega \\subseteq \\mathbb{R}^n$ and $\\Xi\\subseteq \\mathbb{R}^m$\nare closed sets and the function $\\psi :\\mathbb{R}^n \\rightarrow \\mathbb{R}^m$ is  Lipschitz  around $\\bar{x}$. Furthermore, assume that %Assume that the following qualification condition holds at $\\bar{x} \\in C $: whenever $v\\in   N_{\\varsigma}(\\psi(\\bar{x}))$, be such that\n\\begin{equation}\\label{BCQ}\n\\left.\n\\begin{array}{r}\n   0\\in  \\partial \\langle v, \\psi \\rangle (\\bar{x})+N_{\\Omega}(\\bar x)\\\\\n   v\\in   N_{\\Xi}(\\psi(\\bar{x}))\n\\end{array}\n\\right\\} \\Longrightarrow v=0.    \n\\end{equation}\n%{ {(The limiting subdiffential does not seem to have been introduced yet; this should be done and link to Clarke subdifferential discussed. This is needed in the proof of Theorem 3.1.)}}\nThen the following estimate holds:\n\\begin{equation}\\label{normal cone to operator constraint}\n   N_C(\\bar x) \\subseteq     \\bigcup\\left\\{\\partial \\langle v, \\psi \\rangle (\\bar x) + N_{\\Omega}(\\bar x)\\left|\\; v\\in N_\\Xi(\\psi(\\bar x)) \\right.\\right\\}.\n\\end{equation}\nEquality holds in \\eqref{normal cone to operator constraint}, provided that $\\Xi$ is normally regular at $\\psi(\\bar x) $ and $ \\Omega$ is normally regular at $\\bar{x}$.\n\\end{lemma}\n\n\\begin{proof}\nTo begin, note that with the expression of the set-valued mapping $S$ \\eqref{VI-1}, we have  \n\\[\n\\begin{array}{l}\n\\mbox{gph}S = \\Omega \\cap \\psi^{-1}(\\mathbb{R}_-)\\,\n\\mbox{ with } \\, \\Omega:=\\left\\{(x, y)\\in \\mathbb{R}^n\\times \\mathbb{R}^m\\left|\\, \\tilde{g}(x, y)\\leq 0 \\right.\\right\\},\\, \\psi(x, y):= \\tilde{f}(x, y) - \\varphi(x, y),\n\\end{array}\n\\]\nwhere $\\tilde{g}(x,y):= g_0(x, y, y)$ and $\\tilde{f}(x, y):=y^\\top f_0(x, y)$. Then observe that the function $\\varphi$ is Lipshitz continuous around  $(\\bar x, \\bar y)$ given that the set-valued mapping $S_0$ \\eqref{S-v} is inner semicontinuous at $(\\bar x, \\bar y, \\bar{\\varsigma})$ and the MFCQ is satisfied at this same point. Hence, from Lemma \\ref{normal cone estimate}, with the calmness of the set-valued mapping \\eqref{PsiCalm}, as corresponding expression of \\eqref{CalmnesMapAgain}, it follows that \n\\begin{equation} \\label{estimate-eq-1}\n    N_{\\text{gph}S} ( \\bar{x}, \\bar{y}) \\subseteq \\bigcup \\left \\{\\partial \\langle\\lambda, \\psi\\rangle (\\bar{x}, \\bar{y}) + N_\\Omega (\\bar{x}, \\bar{y}) \\left |\\; \\lambda \\ge 0\\right.\\right\\},\n\\end{equation}\nwhile considering the fact that $\\psi(\\bar x, \\bar y)=0$, as $\\bar y\\in S(\\bar x)$, and the functions $\\tilde{f}$ and $\\tilde{g}$ continuously differentiable. Now, considering the definition of the coderivative and inclusion \\eqref{estimate-eq-1}, it follows that for all $y^*\\in \\mathbb{R}^m$, \n\\begin{equation} \\label{estimate-eq-2}\n  D^*S(\\bar{x}, \\bar{y}) (y^*) \\subseteq \\{ x^* \\in \\mathbb{R}^n \\left |\\; \\exists \\lambda \\ge 0:\\quad  \\left(x^*, -y^*\\right)\\in \\partial \\langle\\lambda, \\psi\\rangle (\\bar{x}, \\bar{y}) + N_\\Omega(\\bar{x}, \\bar{y}) \\right \\}.\n\\end{equation}\nSince the MFCQ also holds at $(\\bar{x}, \\bar{y}, \\bar{y})$, using Lemma \\ref{normal cone estimate} one more time, \n\\begin{equation*}\\label{NOmega}\n    N_\\Omega(\\bar{x}, \\bar{y}) \\subseteq \\left\\{\\nabla g(\\bar{x}, \\bar{y})^\\top v \\left|\\; v \\ge 0,\\; g(\\bar{x}, \\bar{y}) \\le 0, \\; v^T g(\\bar{x}, \\bar{y})=0\\right.\\right\\}.\n\\end{equation*}\n\nOn the other hand,  observe that \n\\begin{equation*}\\label{partialLP}\n \\partial \\langle\\lambda, \\psi\\rangle (\\bar{x}, \\bar{y}) \\subseteq \\lambda \\nabla f(\\bar{x}, \\bar{y}) + \\partial \\langle \\lambda, \\, -\\varphi\\rangle (\\bar{x}, \\bar{y}) \\subseteq \\lambda \\nabla f(\\bar{x}, \\bar{y}) -\\lambda \\bar{\\partial} \\varphi (\\bar{x}, \\bar{y})  \n\\end{equation*}\ngiven that as $\\varphi$ is Lipschitz continuous around $(\\bar x, \\bar y)$, its limiting subdifferential is included in the Clarke subdifferential, according to \\eqref{LimitingClarke}. Furthermore, it follows from \\cite[Corollary 5.3]{MordukhovichNamPhanVarAnalMargBlP} that the combination of the inner semicontinuity of $S_0$ and MFCQ \\eqref{LL regularity} both holding at the point $(\\bar x, \\bar y, \\bar{\\varsigma})$ leads to\n\\begin{equation}\\label{SubPhi}\n\\begin{array}{c}\n\\bar\\partial \\varphi(\\bar x, \\bar y) \\;\\, \\subseteq \\;\\, \\left\\{\\left.\\nabla f_0(\\bar x, \\bar y)^\\top\\bar{\\varsigma}  +  \\nabla_{1, 2} g_0(\\bar x, \\bar y, \\bar{\\varsigma})^\\top w\\right|\\;\\, w\\in \\Lambda (\\bar x, \\bar y, \\bar{\\varsigma})\\right\\}.\n\\end{array}\n\\end{equation}\n\nAlso note that as $\\tilde{f}(x, y) = y^T f_0(x, y)$, it holds that\n\\begin{equation}\\label{NablaF}\n  \\nabla \\tilde{f}(x,y)= f_0(x,y) + \\nabla f_0(x, y)^\\top y.\n\\end{equation}\nNow, let $x^* \\in D^*S(\\bar{x}, \\bar{y})(y^*)$. Then it follows from \\eqref{estimate-eq-2}--\\eqref{NablaF} that we can find $\\lambda$, $v$, $w$ such that \n\\[\n\\begin{array}{r}\n\\lambda \\geq 0,\\; v\\geq 0, \\; g_0(\\bar x, \\bar y, \\bar{y})\\leq 0, \\; v^\\top g_0(\\bar x, \\bar y, \\bar{y})=0,\\;  w\\in \\Lambda(\\bar x, \\bar y, \\bar\\varsigma),\\\\[1.5ex]\n\\left[\\begin{array}{r}\n     x^*\\\\\n     -y^*\n\\end{array}\\right]=\\left[\\begin{array}{r}\n \\lambda \\left(\n  \\nabla_1 f_{0}(\\bar x, \\bar y)^\\top  (\\bar y- \\bar{\\varsigma})  - \\nabla_1 g_0(\\bar x, \\bar y, \\bar{\\varsigma})^\\top w\\right) + \\nabla_1 g_0(\\bar x, \\bar y, \\bar y)^\\top v\\\\[1.5ex]\n  \\lambda\\left.\\left(f_0(\\bar x, \\bar y) + \\nabla_2f_0(\\bar x, \\bar y)^\\top (\\bar y - \\bar\\varsigma) - \\nabla_2 g_{0}(\\bar x, \\bar y, \\bar\\varsigma)^{\\top}w  \\right)\\right. + \\nabla_2 g_{0}(\\bar x, \\bar y, \\bar y)^{\\top}v\n\\end{array}\\right].\n\\end{array}\n\\]\nConsidering this and the definition of $\\Lambda^0 (\\bar x, \\bar y, \\bar{\\varsigma}, y^*)$, we have inclusion \\eqref{CoD S}.\n\n\n\nFor assertion (ii), observe that from \\eqref{CoD S}, we have the following inclusion:\n\\[\n\\begin{array}{ll}\n  D^*S(\\bar x, \\bar y)(0) \\subseteq\n  & \\underset{(v, w, \\lambda) \\in \\Lambda^0 (\\bar x, \\bar y, \\bar\\varsigma, 0)}  \\bigcup\n  \\left\\{\\lambda\\left(\\nabla_1 f_{0}(\\bar x, \\bar y)^\\top (\\bar y - \\bar\\varsigma) - \\nabla_1 g_0(\\bar x, \\bar y, \\bar\\varsigma)^\\top w\\right) + \\nabla_1 g_0(\\bar x, \\bar y, \\bar y)^\\top v\\right\\}.\n\\end{array}\n\\]\nHence, it is clear that condition \\eqref{Stab CQ-1} is sufficient for $D^*S(\\bar x, \\bar y)(0) \\subseteq \\{0\\}$. Thus, since the coderivative mapping is positively homogeneous (i.e., in particular, we have $0\\in D^*S(\\bar x, \\bar y)(0)$),  the coderivative/Mordukhovich criterion $D^*S(\\bar x, \\bar y)(0) = \\{0\\}$ is satisfied. This implies that $S$ is Lipschitz-like around $(\\bar x, \\bar y)$, and it follows from \\eqref{LipModulus} that the exact Lipschitzian bound of $S$ around $(\\bar x, \\bar y)$ can be obtained as\n$$\n\\text{{lip}}\\, S(\\bar x, \\bar y) = \\|D^*S(\\bar x, \\bar y)\\|^+:=\\sup\\left\\{\\|u\\|\\left|\\, u\\in D^*S(\\bar x, \\bar y)(y^*), \\, \\|y^*\\|\\leq 1\\right.\\right\\}.\n$$\nConsidering inclusion \\eqref{CoD S} one more time, we have the stated upper bound on the quantity  $\\text{{lip}}\\, S(\\bar x, \\bar y)$. \\qed\n\\end{proof}","units":[{"start":28361,"end":29757,"start_line":268,"end_line":285,"raw_tex":"\\begin{lemma}[see Theorem 6.10 in \\cite{MordukhovichGeneralizedDifferential1994} or Theorem 6.14 in  \\cite{RockafellarWetsBook1998}]\\label{normal cone estimate}\nLet $C:=\\Omega \\cap \\psi^{-1}(\\Xi)$, where $\\Omega \\subseteq \\mathbb{R}^n$ and $\\Xi\\subseteq \\mathbb{R}^m$\nare closed sets and the function $\\psi :\\mathbb{R}^n \\rightarrow \\mathbb{R}^m$ is  Lipschitz  around $\\bar{x}$. Furthermore, assume that %Assume that the following qualification condition holds at $\\bar{x} \\in C $: whenever $v\\in   N_{\\varsigma}(\\psi(\\bar{x}))$, be such that\n\\begin{equation}\\label{BCQ}\n\\left.\n\\begin{array}{r}\n   0\\in  \\partial \\langle v, \\psi \\rangle (\\bar{x})+N_{\\Omega}(\\bar x)\\\\\n   v\\in   N_{\\Xi}(\\psi(\\bar{x}))\n\\end{array}\n\\right\\} \\Longrightarrow v=0.    \n\\end{equation}\n%{ {(The limiting subdiffential does not seem to have been introduced yet; this should be done and link to Clarke subdifferential discussed. This is needed in the proof of Theorem 3.1.)}}\nThen the following estimate holds:\n\\begin{equation}\\label{normal cone to operator constraint}\n   N_C(\\bar x) \\subseteq     \\bigcup\\left\\{\\partial \\langle v, \\psi \\rangle (\\bar x) + N_{\\Omega}(\\bar x)\\left|\\; v\\in N_\\Xi(\\psi(\\bar x)) \\right.\\right\\}.\n\\end{equation}\nEquality holds in \\eqref{normal cone to operator constraint}, provided that $\\Xi$ is normally regular at $\\psi(\\bar x) $ and $ \\Omega$ is normally regular at $\\bar{x}$.\n\\end{lemma}","sha256_utf8":"286bb30ae2e24919bc96abc9de030975f49d88d492d30ea906a50cfa87c4d560"},{"start":48180,"end":53775,"start_line":475,"end_line":542,"raw_tex":"\\begin{proof}\nTo begin, note that with the expression of the set-valued mapping $S$ \\eqref{VI-1}, we have  \n\\[\n\\begin{array}{l}\n\\mbox{gph}S = \\Omega \\cap \\psi^{-1}(\\mathbb{R}_-)\\,\n\\mbox{ with } \\, \\Omega:=\\left\\{(x, y)\\in \\mathbb{R}^n\\times \\mathbb{R}^m\\left|\\, \\tilde{g}(x, y)\\leq 0 \\right.\\right\\},\\, \\psi(x, y):= \\tilde{f}(x, y) - \\varphi(x, y),\n\\end{array}\n\\]\nwhere $\\tilde{g}(x,y):= g_0(x, y, y)$ and $\\tilde{f}(x, y):=y^\\top f_0(x, y)$. Then observe that the function $\\varphi$ is Lipshitz continuous around  $(\\bar x, \\bar y)$ given that the set-valued mapping $S_0$ \\eqref{S-v} is inner semicontinuous at $(\\bar x, \\bar y, \\bar{\\varsigma})$ and the MFCQ is satisfied at this same point. Hence, from Lemma \\ref{normal cone estimate}, with the calmness of the set-valued mapping \\eqref{PsiCalm}, as corresponding expression of \\eqref{CalmnesMapAgain}, it follows that \n\\begin{equation} \\label{estimate-eq-1}\n    N_{\\text{gph}S} ( \\bar{x}, \\bar{y}) \\subseteq \\bigcup \\left \\{\\partial \\langle\\lambda, \\psi\\rangle (\\bar{x}, \\bar{y}) + N_\\Omega (\\bar{x}, \\bar{y}) \\left |\\; \\lambda \\ge 0\\right.\\right\\},\n\\end{equation}\nwhile considering the fact that $\\psi(\\bar x, \\bar y)=0$, as $\\bar y\\in S(\\bar x)$, and the functions $\\tilde{f}$ and $\\tilde{g}$ continuously differentiable. Now, considering the definition of the coderivative and inclusion \\eqref{estimate-eq-1}, it follows that for all $y^*\\in \\mathbb{R}^m$, \n\\begin{equation} \\label{estimate-eq-2}\n  D^*S(\\bar{x}, \\bar{y}) (y^*) \\subseteq \\{ x^* \\in \\mathbb{R}^n \\left |\\; \\exists \\lambda \\ge 0:\\quad  \\left(x^*, -y^*\\right)\\in \\partial \\langle\\lambda, \\psi\\rangle (\\bar{x}, \\bar{y}) + N_\\Omega(\\bar{x}, \\bar{y}) \\right \\}.\n\\end{equation}\nSince the MFCQ also holds at $(\\bar{x}, \\bar{y}, \\bar{y})$, using Lemma \\ref{normal cone estimate} one more time, \n\\begin{equation*}\\label{NOmega}\n    N_\\Omega(\\bar{x}, \\bar{y}) \\subseteq \\left\\{\\nabla g(\\bar{x}, \\bar{y})^\\top v \\left|\\; v \\ge 0,\\; g(\\bar{x}, \\bar{y}) \\le 0, \\; v^T g(\\bar{x}, \\bar{y})=0\\right.\\right\\}.\n\\end{equation*}\n\nOn the other hand,  observe that \n\\begin{equation*}\\label{partialLP}\n \\partial \\langle\\lambda, \\psi\\rangle (\\bar{x}, \\bar{y}) \\subseteq \\lambda \\nabla f(\\bar{x}, \\bar{y}) + \\partial \\langle \\lambda, \\, -\\varphi\\rangle (\\bar{x}, \\bar{y}) \\subseteq \\lambda \\nabla f(\\bar{x}, \\bar{y}) -\\lambda \\bar{\\partial} \\varphi (\\bar{x}, \\bar{y})  \n\\end{equation*}\ngiven that as $\\varphi$ is Lipschitz continuous around $(\\bar x, \\bar y)$, its limiting subdifferential is included in the Clarke subdifferential, according to \\eqref{LimitingClarke}. Furthermore, it follows from \\cite[Corollary 5.3]{MordukhovichNamPhanVarAnalMargBlP} that the combination of the inner semicontinuity of $S_0$ and MFCQ \\eqref{LL regularity} both holding at the point $(\\bar x, \\bar y, \\bar{\\varsigma})$ leads to\n\\begin{equation}\\label{SubPhi}\n\\begin{array}{c}\n\\bar\\partial \\varphi(\\bar x, \\bar y) \\;\\, \\subseteq \\;\\, \\left\\{\\left.\\nabla f_0(\\bar x, \\bar y)^\\top\\bar{\\varsigma}  +  \\nabla_{1, 2} g_0(\\bar x, \\bar y, \\bar{\\varsigma})^\\top w\\right|\\;\\, w\\in \\Lambda (\\bar x, \\bar y, \\bar{\\varsigma})\\right\\}.\n\\end{array}\n\\end{equation}\n\nAlso note that as $\\tilde{f}(x, y) = y^T f_0(x, y)$, it holds that\n\\begin{equation}\\label{NablaF}\n  \\nabla \\tilde{f}(x,y)= f_0(x,y) + \\nabla f_0(x, y)^\\top y.\n\\end{equation}\nNow, let $x^* \\in D^*S(\\bar{x}, \\bar{y})(y^*)$. Then it follows from \\eqref{estimate-eq-2}--\\eqref{NablaF} that we can find $\\lambda$, $v$, $w$ such that \n\\[\n\\begin{array}{r}\n\\lambda \\geq 0,\\; v\\geq 0, \\; g_0(\\bar x, \\bar y, \\bar{y})\\leq 0, \\; v^\\top g_0(\\bar x, \\bar y, \\bar{y})=0,\\;  w\\in \\Lambda(\\bar x, \\bar y, \\bar\\varsigma),\\\\[1.5ex]\n\\left[\\begin{array}{r}\n     x^*\\\\\n     -y^*\n\\end{array}\\right]=\\left[\\begin{array}{r}\n \\lambda \\left(\n  \\nabla_1 f_{0}(\\bar x, \\bar y)^\\top  (\\bar y- \\bar{\\varsigma})  - \\nabla_1 g_0(\\bar x, \\bar y, \\bar{\\varsigma})^\\top w\\right) + \\nabla_1 g_0(\\bar x, \\bar y, \\bar y)^\\top v\\\\[1.5ex]\n  \\lambda\\left.\\left(f_0(\\bar x, \\bar y) + \\nabla_2f_0(\\bar x, \\bar y)^\\top (\\bar y - \\bar\\varsigma) - \\nabla_2 g_{0}(\\bar x, \\bar y, \\bar\\varsigma)^{\\top}w  \\right)\\right. + \\nabla_2 g_{0}(\\bar x, \\bar y, \\bar y)^{\\top}v\n\\end{array}\\right].\n\\end{array}\n\\]\nConsidering this and the definition of $\\Lambda^0 (\\bar x, \\bar y, \\bar{\\varsigma}, y^*)$, we have inclusion \\eqref{CoD S}.\n\n\n\nFor assertion (ii), observe that from \\eqref{CoD S}, we have the following inclusion:\n\\[\n\\begin{array}{ll}\n  D^*S(\\bar x, \\bar y)(0) \\subseteq\n  & \\underset{(v, w, \\lambda) \\in \\Lambda^0 (\\bar x, \\bar y, \\bar\\varsigma, 0)}  \\bigcup\n  \\left\\{\\lambda\\left(\\nabla_1 f_{0}(\\bar x, \\bar y)^\\top (\\bar y - \\bar\\varsigma) - \\nabla_1 g_0(\\bar x, \\bar y, \\bar\\varsigma)^\\top w\\right) + \\nabla_1 g_0(\\bar x, \\bar y, \\bar y)^\\top v\\right\\}.\n\\end{array}\n\\]\nHence, it is clear that condition \\eqref{Stab CQ-1} is sufficient for $D^*S(\\bar x, \\bar y)(0) \\subseteq \\{0\\}$. Thus, since the coderivative mapping is positively homogeneous (i.e., in particular, we have $0\\in D^*S(\\bar x, \\bar y)(0)$),  the coderivative/Mordukhovich criterion $D^*S(\\bar x, \\bar y)(0) = \\{0\\}$ is satisfied. This implies that $S$ is Lipschitz-like around $(\\bar x, \\bar y)$, and it follows from \\eqref{LipModulus} that the exact Lipschitzian bound of $S$ around $(\\bar x, \\bar y)$ can be obtained as\n$$\n\\text{{lip}}\\, S(\\bar x, \\bar y) = \\|D^*S(\\bar x, \\bar y)\\|^+:=\\sup\\left\\{\\|u\\|\\left|\\, u\\in D^*S(\\bar x, \\bar y)(y^*), \\, \\|y^*\\|\\leq 1\\right.\\right\\}.\n$$\nConsidering inclusion \\eqref{CoD S} one more time, we have the stated upper bound on the quantity  $\\text{{lip}}\\, S(\\bar x, \\bar y)$. \\qed\n\\end{proof}","sha256_utf8":"404780dc4880c32f1a7ae100186415304138e4572a9325d7796c5d25c000d84f"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"参数拟变分不等式","source_units":[{"start":9389,"end":9807,"start_line":105,"end_line":109,"raw_tex":"Our basic problem of interest in this paper is the quasi-variational inequality (QVI) problem to find $y\\in K(x, y)$, for a given parameter $x\\in \\mathbb{R}^n$, such that we have \n\\begin{equation}\\tag{QVI}\\label{VI}\n\\left\\langle f_0(x,y), \\;\\, \\varsigma - y \\right\\rangle \\geq 0 \\;\\;\\;\n \\forall \\varsigma\\in  K(x, y):= \\left\\{\\varsigma \\in \\mathbb{R}^m\\left|\\; g_0(x, y, \\varsigma)\\leq 0\\right.\\right\\},\n\\end{equation}","sha256_utf8":"3c250861b52dcb137b9bd54ea3cfa8a544d594bd1e212d5c8667bed0af4931bb"},{"start":9808,"end":10199,"start_line":110,"end_line":113,"raw_tex":"where $f_0 : \\mathbb{R}^n\\times \\mathbb{R}^m \\rightarrow \\mathbb{R}^m$ is a continuously differentiable function\nand $K :\\mathbb{R}^n\\times \\mathbb{R}^m \\rightrightarrows \\mathbb{R}^m$ denotes the feasibility\nset-valued map defined by a function $g_0\\, : \\mathbb{R}^n\\times \\mathbb{R}^m\\times \\mathbb{R}^m \\rightarrow \\mathbb{R}^q$,\n which is also assumed to be continuously differentiable.\n","sha256_utf8":"0f24621595d683f6a21b2bc84714822f6b8939e59bad8c455d20cc058ffe3bdd"}]},{"id":"S002","semantic_type":"definition","kind":"definition","title":"解映射与值函数重构","source_units":[{"start":10624,"end":10856,"start_line":119,"end_line":123,"raw_tex":"\\begin{equation}\\label{S-Soultion-Map}\n\\begin{array}{c}\n  S(x):=\\left\\{y\\in K(x,y)\\left|\\; \\left\\langle f_0(x,y), \\;\\, \\varsigma - y \\right\\rangle \\geq 0  \\;\\;\\; \\forall \\varsigma\\in K(x, y)\\right.\\right\\}\n\\end{array}\n\\end{equation}","sha256_utf8":"10ed60a80754e45e1a1c38bfb87b20cc7d6dbe45510d0c89ecab48e0d89273b7"},{"start":12741,"end":12936,"start_line":135,"end_line":138,"raw_tex":"\\begin{equation}\\label{VI-1}\n S(x):=\\left\\{y\\in K(x,y)\\left|\\;\\, y^\\top f_0(x,y)-\\varphi(x,y)\\leq 0\\right.\\right\\}\n%    y\\in K(x, y) \\mbox{ such that } f(x,y,y)-\\varphi(x,y)\\leq 0.\n\\end{equation}","sha256_utf8":"808e7c93da3c3cfa663d17e5cfa8b22e1eecb44b19286dd2b6b3b23847074002"},{"start":12937,"end":13161,"start_line":139,"end_line":142,"raw_tex":"for all $x\\in \\mathbb{R}^n$. Here, $\\varphi$ denotes the optimal value function\n\\begin{equation}\\label{varphi}\n   \\varphi(x,y):= \\underset{\\varsigma}\\min \\{\\varsigma^\\top f_0(x, y)|\\;\\; \\varsigma\\in K(x, y)\\}.\n\\end{equation}","sha256_utf8":"f50d1696f2d473f5fad9214584c25ef47e753f26ec64d7821a0f73fa8780adde"}]},{"id":"S003","semantic_type":"notation","kind":"notation","title":"活跃集与偏导记号","source_units":[{"start":16295,"end":17201,"start_line":164,"end_line":171,"raw_tex":"For the points $(\\bar x, \\bar y)$, $(\\bar x, \\bar y, \\bar y)$, and $(\\bar x, \\bar y, \\bar\\varsigma)$, the following notation will be used to collect { {active index sets}} of constraints for the feasible sets associated to problems \\eqref{VI} and \\eqref{OPQVIC}, which are defined by the functions $(x, y) \\mapsto G(x, y)$, $(x, y) \\mapsto g_0(x, y, y)$, and $(x, y, \\varsigma) \\mapsto g_0(x, y, \\varsigma)$, respectively:\n\\begin{equation}\\label{I1}\n\\begin{array}{lclll}\n     I^1&:=& I^G(\\bar x, \\bar y)                   &:=&  \\left\\{i\\in \\{1, \\ldots, p\\}\\left|~G_i(\\bar x, \\bar y)=0\\right.\\right\\},\\\\[1ex]\n    I^2 &:=& I^{g_0}(\\bar x, \\bar y, \\bar y)       &:=& \\left\\{j\\in \\{1, \\ldots, q\\}|~g_{0_j}(\\bar x, \\bar y, \\bar y)=0\\right\\},\\\\[1ex]\n    I^3 &:=& I^{g_0}(\\bar x, \\bar y, \\bar\\varsigma)&:=& \\left\\{j\\in \\{1, \\ldots, q\\}|~g_{0_j}(\\bar x, \\bar y, \\bar\\varsigma)=0\\right\\}.\n\\end{array}\n\\end{equation}","sha256_utf8":"f9d937cccda16b05bf3847b336da96a8d312bfe4b182ec6453c8e666d3eb1d65"},{"start":18729,"end":19804,"start_line":189,"end_line":189,"raw_tex":"{ {Although symbols that we use here are very standard, it might be useful to make it clear how we use the symbol $\\nabla$ interchangeably between the gradient of a scalar function and the Jacobian of a vector-valued function. For a continuously differentiable function $\\psi : \\mathbb{R}^n\\times \\mathbb{R}^m \\times \\mathbb{R}^p \\rightarrow \\mathbb{R}^q$, in case there is ambiguity in the use of the usual partial derivatives, we will utilize $\\nabla_1 \\psi(x, y, \\varsigma)$, (resp. $\\nabla_2 \\psi(x, y, \\varsigma)$, $\\nabla_3 \\psi(x, y, \\varsigma)$) to represent the Jacobian of the function $x \\mapsto \\psi(x, y, \\varsigma)$ (resp. $y \\mapsto \\psi(x, y, \\varsigma)$, $\\varsigma \\mapsto \\psi(x, y, \\varsigma)$) when the vector $(y, \\varsigma)$ (resp. $(x, \\varsigma)$, $(y, \\varsigma)$) is fixed. A similar notation will be used for the gradient of $\\psi$ if it is a real-valued function. Common scenarios for $\\psi$ in this paper will be $\\psi(x, y, \\varsigma):= g_0(x, y, \\varsigma)$ and $\\psi(x, y, \\varsigma):= \\varsigma^\\top f_0(x, y)$ when studying our \\eqref{VI}.\n","sha256_utf8":"1f9cb743627846be8c6c278fa42f65752ef175fa5e07577c34d927156e525c59"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"Clarke广义微分","source_units":[{"start":20790,"end":23092,"start_line":197,"end_line":218,"raw_tex":"Given that the optimal value function $\\varphi$ \\eqref{varphi} is nonsmooth in general, we need generalized concepts of\n differentiability to deal with it. We start the discussion on this by recalling the \\emph{generalized directional derivative} in the sense of\n Clarke \\cite{ClarkeBook1983}, which, for a function $\\psi : \\mathbb{R}^n \\rightarrow \\mathbb{R}$, is defined at a point $\\bar{x}\\in \\mathbb{R}^n$ in direction $d\\in \\mathbb{R}^n$ by\n \\begin{equation*}\\label{Clarke Directional Derivative}\n    \\psi^0(\\bar x; d):=\\underset{t\\downarrow 0}{\\underset{x \\rightarrow \\bar x}\\limsup} \\frac{1}{t}\\left[\\psi(x + td)-\\psi(x)\\right].\n\\end{equation*}\n This quantity exists if $\\psi$ is a Lipschitz continuous function around $\\bar x$ \\cite[Proposition 2.1.1]{ClarkeBook1983}.\n  Utilizing this notion, the \\textit{Clarke subdifferential} of $\\psi$ at the point $\\bar{x}$ can be defined by \n\\begin{equation}\\label{Clarke Subdifferential}\n    \\bar\\partial \\psi(\\bar x):= \\left\\{\\zeta\\in \\mathbb{R}^n|\\; \\psi^o(\\bar x; d)\\geq \\langle\\zeta, d\\rangle, \\; \\forall d\\in \\mathbb{R}^n  \\right\\}.\n\\end{equation}\nNote that {{$\\bar\\partial \\psi(\\bar x) = \\left\\{\\nabla \\psi(\\bar x)\\right\\}$ }} if $\\psi$ is continuously differentiable at $\\bar x$, and for a convex function,\nthis concept coincides with the subdifferential in the sense of convex analysis.\nFurthermore, this concept can be extended to a vector-valued function $\\psi : \\mathbb{R}^n \\rightarrow \\mathbb{R}^m$ that\nis Lipschitz continuous around a point $\\bar x$, as such a function is differentiable almost everywhere around this point. Hence,\n the \\emph{Clarke generalized Jacobian} of the function $\\psi$ at the point $\\bar{x}$ can be written as \n\\begin{equation}\\label{Clarke Subdifferential-Derivative}\n    \\bar\\partial \\psi(\\bar x):= \\mbox{co}\\left\\{\\lim\\nabla\\psi(x^n)\\left|~x^n \\rightarrow \\bar x, \\;\\, x^n\\in D_\\psi \\right.\\right\\},\n\\end{equation}\nwhere ``co'' stands for the convex hull and  $D_\\psi$ represents the set of points where $\\psi : \\mathbb{R}^n \\rightarrow \\mathbb{R}^m$ is\ndifferentiable. { {Note that the quantity defined in  \\eqref{Clarke Subdifferential-Derivative} coincides with the one in \\eqref{Clarke Subdifferential}\n for real-valued functions}}. Following  the expression in \\eqref{Clarke Subdifferential-Derivative},\n","sha256_utf8":"aafd8694e913b1eee3ce37ac09a910b165fb7a49f793a523b2c7e7134ab6086f"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"极限与正则法锥","source_units":[{"start":24721,"end":25898,"start_line":238,"end_line":248,"raw_tex":"{ {In the sequel, the concept of basic normal cone will also be necessary. \n For a closed subset $C$ of $\\mathbb{R}^n$, the \\textit{basic} (also known as \\emph{Mordukhovich} or \\textit{limiting}) \\textit{normal cone}\n to $C$ at one of its points $\\bar x$ is the set\n\\begin{equation}\\label{basic normal cone}\n N_C(\\bar x):= \\left\\{v\\in \\mathbb{R}^n\\left|\\; \\exists v_k \\rightarrow v, \\, x_k \\rightarrow \\bar x \\,(x_k\\in C): \\, v_k\\in \\widehat{N}_C(x_k)\\right.\\right\\},\n\\end{equation}\nwhere $\\widehat{N}_C$ denotes the polar of the contingent/Boulingand tangent cone to $C$}}:\n\\begin{equation}\\label{polar normal cone}\n \\widehat{N}_C(\\bar x):=\\left\\{v\\in \\mathbb{R}^n \\left |\\; \\langle v, u-\\bar x\\rangle \\leq o(\\|u-\\bar x\\|)\\;\\; \\forall u\\in C\\right. \\right\\}.\n\\end{equation}\n$\\widehat{N}_C(\\bar{x})$ is also known as the \\textit{regular} or \\textit{Fr\\'{e}chet} normal cone. A set $C \\subset \\mathbb{R}^n$ is said to be \\textit{normally regular} at $\\bar{x} \\in C$ if $ N_C(\\bar{x}) = \\widehat{N}_C(\\bar{x})$. If $C$ is a convex set, it is normally regular at all points of $C$, given that both of the above cones coincide with the normal cone in the sense of convex analysis.\n","sha256_utf8":"00f13aaef0fc00da6c02325c28a34b13f6eaf51744ead63858164eca4600200f"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"次微分及凸化关系","source_units":[{"start":25899,"end":28019,"start_line":250,"end_line":263,"raw_tex":"The regular normal cone allows us to introduce the notion of a \\textit{regular subdifferential} also known as the Fr\\'{e}chet subdifferential. Let the function $\\psi : \\mathbb{R}^n \\rightarrow \\mathbb{R}$ be Lipschitz continuous around $\\bar x$. Then its regular subdifferential at $\\bar{x} \\in \\mathbb{R}^n$ is given as \n\\[\n    \\hat{\\partial}\\psi(\\bar{x}) = \\left\\{v\\in \\mathbb{R}^n\\left|~(v, -1) \\in \\widehat{N}_{\\mbox{epi}\\psi} (\\bar{x}, \\psi(\\bar{x}))\\right.\\right\\},\n\\]\nwhere epi$\\psi$ denotes the epigraph of the function $\\psi$.\nLet us now introduce the notion of a \\textit{basic subdifferential} which is also known as the \\textit{limiting} subdifferential or the \\textit{Mordukhovich} subdifferential. For $\\psi : \\mathbb{R}^n \\rightarrow \\mathbb{R}$, which Lipschitz continous around $\\bar x$, its basic subdifferential at this point is given by \n\\[\n    \\partial\\psi(\\bar{x}) = \\left\\{v\\in \\mathbb{R}^n\\left|~(v, -1) \\in N_{\\mbox{epi}\\psi} (\\bar{x}, \\psi(\\bar{x}))\\right.\\right\\}.\n\\]\nConsidering the interplay between \\eqref{basic normal cone} and \\eqref{polar normal cone}, $v \\in \\partial \\psi(\\bar{x})$ if and only if there exist sequences $\\{x^k \\}$ with $x^k \\rightarrow \\bar{x}$ and $\\{v^k\\}$ with $ v^k \\rightarrow v$ such that $v^k \\in \\hat{\\partial} \\psi(x^k)$. The basic subdifferential for a locally Lipschitz function is always non-empty and compact. If $\\psi$ is a convex function, then both its regular and basic subdifferentials coincides with the subdifferential in the sense of convex analysis. Furthermore, if a function $\\psi : \\mathbb{R}^n \\rightarrow \\mathbb{R}$ is Lipschitz continous around a point $\\bar x$, then we have the nice relationship\n\\begin{equation}\\label{LimitingClarke}\n    \\bar{\\partial}\\psi = \\mbox{cl} \\mbox{co}\\partial \\psi (\\bar{x})\n\\end{equation}\nbetween the Clarke and basic subdifferentials; cf. \\cite[Theorem 3.57 of Volume I]{MordukhovichBook2006}, where $\\mbox{cl} \\mbox{co}$ stands for the closure of the convex hull of the corresponding set. Hence, the Clarke subdifferential is often referred to in some literature as the \\textit{convexified subdifferential}.\n","sha256_utf8":"655406116c80e54ad51a49e8e4d22709ab4ba52bdde23a14d5e5f12a3ec26e72"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"图与内半连续性","source_units":[{"start":30397,"end":31049,"start_line":288,"end_line":295,"raw_tex":"Set-valued mappings and some related concepts will play a key role in the derivation of key results in this paper. To proceed, we consider a set-valued mapping $\\Psi :\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^m$. \n{ {The domain of $\\Psi$ is denoted as $\\text{dom}\\Psi$ and defined by\n$\\text{dom}\\Psi := \\{ x \\in \\mathbb{R}^n  \\left| \\ \\Psi(x) \\neq \\emptyset\\right\\}$,\nwhile its graph, denoted as $\\text{gph} \\Psi$, is given by\n\\begin{equation*}\n    \\text{gph} \\Psi  :=  \\{ (x, y) \\in \\mathbb{R}^m \\times \\mathbb{R}^m \\left | \\ y \\in \\Psi(x) \\right \\} = \\{ (x, y) \\in \\text{dom} \\Psi \\times \\mathbb{R}^m \\left|\\ y \\in \\Psi(x)\\right\\}.\n\\end{equation*}\n}} \n","sha256_utf8":"91b1b2cad1964a80e762999ae4235260bee076367b92216dbea35014b63a9aac"},{"start":31618,"end":32831,"start_line":298,"end_line":300,"raw_tex":" { {The set-valued mapping $\\Psi :\\mathbb{R}^n\\rightrightarrows\\mathbb{R}^m$ is said to be \n { {inner semicontinuous at $(\\bar{x},\\bar{y})\\in \\mbox{gph}\\Psi$ w.r.t. $\\Omega\\in\\mathbb{R}^{n}$ whenever for each neighborhood $V\\subset \\mathbb{R}^{m}$ of $\\bar{y}$, there exists a neighborhood $U\\subset \\mathbb{R}^{n}$ of $\\bar{x}$ such that $\\Psi (x)\\cap V\\neq \\emptyset$ for all $x \\in \\Omega \\cap U$. If $\\Omega = \\mathbb{R}^{n}$ can be chosen, $\\Psi$ is called inner semicontinuous at $(\\bar{x},\\bar{y})$ for brevity; i.e., if for every sequence $x_k\\rightarrow\\bar{x}$, there is a sequence of $y_k\\in\\Psi(x_k)$ that converges to $\\bar{y}$ as $k\\rightarrow\\infty$. For single-valued mappings $\\Psi :\\mathbb{R}^n\\to\\mathbb{R}^m$, this property obviously reduces to the continuity of $\\Psi$ at $\\bar x$.}} %Furthermore, if $\\Psi :\\mathbb{R}^n \\rightrightarrows\\mathbb{R}^m$ is inner semicompact at $\\bar{x}$ with $\\Psi(\\bar{x})=\\{\\bar{y}\\}$, then $\\Psi$ is inner semicontinuous at $(\\bar x, \\bar y)$.\n Obviously, the inner semicontinuity property is more restrictive than the inner semicompactness while bringing us to more precise results of the coderivative calculus, as it will be clear in the next section.}} \n","sha256_utf8":"62b91b26426fa593bd481fdeaf705b6485546f74b096684b5398fdbea7a6de95"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"Aubin性质与平静性","source_units":[{"start":32832,"end":34412,"start_line":302,"end_line":315,"raw_tex":"As it will become clear in the subsequent sections, some Lipschitz-type properties will also play a central role in the main results. Hence, we introduce some useful concepts.  A set-valued mapping $\\Psi : \\mathbb{R}^{n}\\rightrightarrows \\mathbb{R}^{m}$\nis said to be \\emph{Lipschitz-like} (or satisfies the \\textit{Aubin property} \\cite{Aubin1984}) around $\\left(\\bar{x},\\bar{y}\\right)\\in \\mbox{gph}\\Psi$ if\n there exist neighborhoods $U$ of $\\bar{x}$, $V$ of $\\bar{y}$, and a constant $l > 0$ such that\n\\begin{equation}\\label{exact-estim}\n\t\\Psi\\left( x\\right) \\cap V \\subseteq \\Psi\\left( u\\right) +l\\|u-x\\|\\mathbb{B} \\ \\ \\text{for all} \\ x,u \\in U\n\\end{equation}\nwith $\\mathbb{B}$ denoting the unit ball in $\\mathbb{R}^{m}$ (see; e.g., \\cite{HenrionJouraniOutrataCalmness2002}). This property can be equivalently expressed as \n\\begin{equation}\\label{Aubin-Property}\n    d(y, \\Psi(u)) \\le l \\| x - u \\| \\,\\mbox{ for all }\\, x, u \\in U \\;\\, \\mbox{and}\\;\\, y \\in V \\cap\\Psi(x).\n\\end{equation}\nThe weaker calmness property can be obtained from \\eqref{Aubin-Property} by setting $ u = \\bar{x}$. Thus, the set-valued mapping $\\Psi :\\mathbb{R}^n \\rightrightarrows \\mathbb{R}^m$ will be said to be \\textit{calm} (or satisfy the {\\em calmness} property) at the point $(\\bar x, \\bar y)$ with $\\bar y\\in \\Psi(\\bar x)$, if there exist neighborhoods $U$ and $V$ of $\\bar x$ and $\\bar y$, respectively, and a constant $\\ell >0$ such that\n\\begin{equation}\\label{Definition Calmness}\nd(y, \\Psi(\\bar x))\\leq \\ell \\|x-\\bar x\\| \\,\\mbox{ for all }\\, y\\in V\\cap \\Psi(x), \\;\\, x\\in U.\n\\end{equation}\n","sha256_utf8":"953a8abc867b58fe3b602cb15b9e9f842f5055b9f9d978344b96dfff4c31ca80"}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","title":"平静性下的法锥估计","source_units":[{"start":35043,"end":35486,"start_line":320,"end_line":323,"raw_tex":"  As an important application of calmness, note that based on \\cite[Theorem 4.1]{HenrionJouraniOutrataCalmness2002}, inclusion \\eqref{normal cone to operator constraint} also holds if the following set-valued mapping is calm at the point $(0, \\bar x)$ with $\\bar x\\in \\Omega \\cap \\psi^{-1}(\\Xi)$: \n\\begin{equation}\\label{CalmnesMapAgain}\n\\Psi(\\varsigma):=\\left\\{x\\in \\Omega\\left|\\;\\, \\psi(x) + \\varsigma \\in \\Xi\\right.\\right\\}.\n\\end{equation}\n","sha256_utf8":"c8b003be5f2f4aa730e430d3965e9def184cfcb28f53e8a4aebbffeef56ae96f"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"余导数与Aubin判据","source_units":[{"start":35487,"end":37271,"start_line":325,"end_line":338,"raw_tex":"  Next, we use the concept of basic normal cone introduced in the previous subsection to define the notion of {\\em coderivative}, which will allow us to present the most powerful characterization of the Lipschitz-likeness property.  For a given set-valued mapping $\\Psi :\\mathbb{R}^n \\rightrightarrows \\mathbb{R}^m$,  its coderivative at some point $(\\bar{x}, \\bar{y})\\in \\text{gph}\\, \\Psi$ corresponds to another set-valued mapping \n$D^*\\Psi(\\bar{x},\\bar{y}): \\mathbb{R}^m \\rightrightarrows \\mathbb{R}^n$ that is defined at any $y^*\\in \\mathbb{R}^m$ by\n\\begin{equation*}\\label{cod-definition}\n   D^*\\Psi(\\bar{x},\\bar{y})(y^*):= \\left\\{x^*\\in \\mathbb{R}^n|\\; (x^*, -y^*)\\in N_{\\text{gph}\\, \\Psi}(\\bar{x}, \\bar{y})\\right\\}.\n\\end{equation*}\nFor a single-valued locally Lipschitz continuous function $\\psi : \\mathbb{R}^n \\rightarrow \\mathbb{R}^m$, the coderivative reduces to  \n\\begin{equation*}\\label{coderivative-sub}\n    D^* \\psi (\\bar{x}) (y^*) = \\partial \\langle y^*,  \\psi \\rangle (\\bar{x}),\n\\end{equation*}\nwhere similarly to \\eqref{normal cone to operator constraint}, the term $\\partial \\langle y^*, \\psi \\rangle (\\bar x)$ denotes the basic subdifferential of the function $x \\mapsto \\sum^m_{i=1} y^*_i \\psi_i(x)$ at $\\bar x$.\nThe coderivative plays a key role in identifying whether a set-valued mapping is Lipschitz-like; i.e., the well-known \\textit{Murdokhovich} criterion (see, e.g., \\cite{RockafellarWetsBook1998,MordukhovichGeneralizedDifferential1994}), which says that for a set-valued mapping $\\Psi : \\mathbb{R}^{n}\\rightrightarrows \\mathbb{R}^{m}$, which has a closed graph around $(\\bar x, \\bar y)\\in \\text{gph}\\Psi$, it is Lipschitz-like around this point if and only if \n\\begin{equation*} \\label{Aubin}\n    D^* \\Psi ( \\bar{x}, \\bar{y})(0) = \\{0\\}.\n\\end{equation*}\n","sha256_utf8":"0c16f5c2423591320ac9f39373267e2392527cebaa8c7b2cd03ca2a9aed43c2d"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"外范数与Lipschitz模","source_units":[{"start":37271,"end":38860,"start_line":339,"end_line":352,"raw_tex":"Thanks to the concept of coderivative, we can also characterize the \\textit{Lipschitz modulus} of a set-valued mapping when it is Lipschitz-like \\eqref{exact-estim}. To present this characterization, we first introduce the notion a Lipschitz modulus of a set-valued mapping $\\Psi : \\mathbb{R}^{n}\\rightrightarrows \\mathbb{R}^{m}$. For this, we need  the notion of \\textit{outer norm}  %and \\textit{inner norm}  \nof a positively homogeneous set-valued mapping. { {Note that $\\Psi$ is said to be positively homogeneous if $0\\in \\Psi (0)$ and $\\Psi (\\lambda x) = \\lambda \\Psi (x) $ for all $\\lambda > 0$ and $ x \\in \\mathbb{R}^n$.}}\nIf $\\Psi$ is a positively homogeneous\nmapping, its \\emph{outer norm} is defined by\n\\begin{equation*}\n\t\\|\\Psi\\|^{+} :={\\displaystyle \\sup_{x\\in\\mathbb{B}}} \\\n{\\displaystyle \\sup_{u\\in\\Psi\\left( x\\right) }}\\|u\\|.  %\\ \\ \\ \\left(\\text{resp.} \\\n%\\|\\Psi\\|^{-} :={\\displaystyle \\sup_{x\\in\\mathbb{B}}} \\ {\\displaystyle \\inf_{u\\in\\Psi\\left( x\\right) }} \\|u\\|\\right).\n\\end{equation*}\nFurthermore, the infinimum of all $l > 0$ for which \\eqref{exact-estim} holds, also known as the \\emph{Lipschitz modulus} of $\\Psi$ at $\\left(\\bar{x},\\bar{y}\\right)$, is given as follows (cf. \\cite[Theorem 9.40]{RockafellarWetsBook1998}),  via the outer norm of the coderivative of the set-valued mapping $\\Psi$:\n\\begin{equation}\\label{LipModulus}\n\t\\text{lip} \\Psi \\left( \\bar{x},\\bar{y}\\right) := \\inf \\left\\{l\\in ]0, +\\infty[ \\ |\n\\eqref{exact-estim} \\ \\text{holds for some} \\  U \\ \\text{and} \\ V  \\right\\}=\\left\\|D^{\\ast}\\Psi\\left( \\bar{x},\\bar{y}\\right)\\right\\|^{+}.\n\\end{equation}\n","sha256_utf8":"2ecafa6f8e5ac42c29279f8c64d0c2935241b95bbc141887f7c312a29daaee7a"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"下层MFCQ","source_units":[{"start":40294,"end":41169,"start_line":377,"end_line":381,"raw_tex":"The plan here is to use the value function reformulation \\eqref{VI-1} of  $S$ \\eqref{S-Soultion-Map} to construct an estimate for its coderivative  in terms of the derivatives of $f_0$ and $g_0$, and subsequently, to leverage on this result to provide new rules for the Lipschitz-like property of this set-valued mapping. To proceed,  some further definitions and notations are in order. First, the Mangasarian-Fromowitz constraint qualification (MFCQ) for the constraint describing $K$ \\eqref{VI},  which will be said to hold at $(\\bar x, \\bar y, \\bar\\varsigma)$ if there exists a vector $d\\in \\mathbb{R}^m$ such that\n\\begin{equation}\\label{LL regularity}\n    \\nabla_3 g_{0j}(\\bar x, \\bar y, \\bar\\varsigma)^\\top d < 0\\;  \n    \\mbox{ for all }\\; j\\in I^3.  %:= I^3(\\bar x, \\bar y, \\bar\\varsigma):= \\left\\{i|\\; g_{0_{i}}(\\bar x, \\bar y, \\bar\\varsigma)=0\\right\\}\n\\end{equation}","sha256_utf8":"f608d5cc0345a277d66f1ac61588703ad1de4abf3ce268c3bd91af6e7bb81b25"},{"start":41170,"end":41400,"start_line":382,"end_line":382,"raw_tex":"Recall that $I^3$ is defined in \\eqref{I1} and $\\nabla_3 g_{0j}(x, y, \\varsigma)$ represents the Jacobian of the function $\\varsigma \\mapsto g_{0j}(x, y, \\varsigma)$ for a fixed vector $(x, y)\\in \\mathbb{R}^n\\times \\mathbb{R}^m$.\n","sha256_utf8":"f9cabe5fab81009db58ba57c548770c5b07b410a1306ea6a9d0f5dd9ad73d287"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"余导数乘子集合","source_units":[{"start":41400,"end":42254,"start_line":383,"end_line":394,"raw_tex":"Next, we associate to a vector $(\\bar x, \\bar y, \\bar\\varsigma, \\varsigma^*)$ a set of  Lagrange multipliers\n\\begin{equation*}\n  \\Lambda^{0}(\\bar x, \\bar y, \\bar\\varsigma, \\varsigma^*)\n   :=  \\left\\{\\left(\\begin{array}{c}\n        v\\\\ w\\\\ \\lambda\n\\end{array}\\right)\\in \\mathbb{R}^{2q + 1} \\left|\n\\left.\\begin{array}{l}\n   \\lambda \\geq 0,\\; v\\geq 0, \\; g_0(\\bar x, \\bar y, \\bar{y})\\leq 0, \\; v^\\top g_0(\\bar x, \\bar y, \\bar{y})=0,\\;  w\\in \\Lambda(\\bar x, \\bar y, \\bar \\varsigma)\\\\[1ex]\n     \\varsigma^*  + \\lambda\\left.\\left(f_0(\\bar x, \\bar y) + \\nabla_2f_0(\\bar x, \\bar y)^\\top (\\bar y - \\bar\\varsigma) - \\nabla_2 g_{0}(\\bar x, \\bar y, \\bar\\varsigma)^{\\top}w  \\right)\\right.\\\\[1ex]\n      \\qquad \\qquad \\qquad \\qquad    \\qquad \\qquad \\qquad \\qquad + \\nabla_2 g_{0}(\\bar x, \\bar y, \\bar y)^{\\top}v=0\n      \\end{array}\\right.\\right.\\right\\} \n\\end{equation*}","sha256_utf8":"2bfb2e7e01609092777616325a9867b735c06a80f446a0fbb2ac71f0b2d0f316"},{"start":43351,"end":43799,"start_line":412,"end_line":412,"raw_tex":"related to the  problem  \\eqref{VI}, with $\\nabla_2 f_{0}(x, y)$ denoting the Jacobian of the function $f_{0}(x, .)$ for a fixed vector $x\\in \\mathbb{R}^n$, while $\\nabla_2 g_{0}(x, y, \\varsigma)$ represents the Jacobian of the function $g_{0}(x, ., \\varsigma)$ with $(x, \\varsigma)\\in \\mathbb{R}^n\\times \\mathbb{R}^m$ fixed. Furthermore, recall that the other involved set of Lagrange multipliers %$\\Lambda(\\bar x, \\bar y, \\hat{y})$ is defined by\n","sha256_utf8":"40a62d277925dc886161cf2b11d5af20429f55bf0ae4d9a207a701164972c4c1"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"下层乘子与拉格朗日函数","source_units":[{"start":43799,"end":44095,"start_line":413,"end_line":417,"raw_tex":"\\begin{equation}\\label{Lambda(x,y)}\n\\begin{array}{rll}\n\\Lambda(x, y, \\varsigma) &:=& \\left\\{w\\in \\mathbb{R}^q\\left|\\;\\nabla_3\\ell(x, y, \\varsigma, w)=0,\\right.\\right.\\left. \\left. w\\geq 0, \\; g_0(x, y, \\varsigma)\\leq 0,\\; w^{\\top}g_0(x, y, \\varsigma)= 0 \\right.\\right\\}\n\\end{array}\n\\end{equation}","sha256_utf8":"7feccab2a4f71788f6f2dd248eb0454b6cb6796bcf8f96f267cbd23cedea1011"},{"start":44096,"end":44330,"start_line":418,"end_line":418,"raw_tex":"is defined with $x\\in \\mathbb{R}^n$ and $(y,\\varsigma)\\in \\mathbb{R}^{2m}$. In \\eqref{Lambda(x,y)}, the function $\\ell$ represents the Lagrangian function associated with the underlying parametric optimization problem \\eqref{varphi}:\n","sha256_utf8":"bd1dbc2cca76c40fd3dcd16bd11eebdc517bea07af19945c1f9e39429bf3979a"},{"start":44331,"end":44459,"start_line":419,"end_line":421,"raw_tex":"\\begin{equation}\\label{ell}\n    \\ell(x, y, \\varsigma, w):=\\varsigma^\\top f_0(x, y)+ w^\\top g_0(x, y, \\varsigma).\n \\end{equation}","sha256_utf8":"f63ab3307752012fd615a3acc657b54931a3acf94f238551e09f92abac7222fd"},{"start":44460,"end":44781,"start_line":422,"end_line":423,"raw_tex":"Furthermore, in \\eqref{Lambda(x,y)},  $\\nabla_3 \\ell(x, y, \\varsigma, w) := f_0(x, y) + \\nabla_3 g_{0}(x, y, \\varsigma)^\\top w$ with $\\nabla_3 g_{0}(x, y, \\varsigma)$ denoting the Jacobian of the function $\\varsigma \\mapsto g_{0j}(x, y, \\varsigma)$\nwhen the vectors $x\\in \\mathbb{R}^n$ and $y\\in \\mathbb{R}^m$ are fixed.\n","sha256_utf8":"5e7127163710bd162c47e0d76458fccc943eabd91579b7448f7e38ec963654bd"}]},{"id":"S015","semantic_type":"definition","kind":"definition","title":"下层最优解映射","source_units":[{"start":45452,"end":45762,"start_line":431,"end_line":435,"raw_tex":"$S(x)=\\left\\{y\\in \\mathbb{R}^m|\\, y\\in S_0(x, y)\\right\\}$,\nwhere  $S_0 : \\mathbb{R}^n\\times \\mathbb{R}^m \\rightrightarrows \\mathbb{R}^m$ is given as\n\\begin{equation}\\label{S-v}\n S_0(x,y)  := \\arg\\underset{\\varsigma}\\min~\\left\\{\\varsigma^\\top f_0(x,y)\\left|\\; \\varsigma\\in K(x, y)\\right.\\right\\}.\n\\end{equation}","sha256_utf8":"92d6425ee52184203a537cfc0c8e03d16de41e72d8aa490d39572db8b70a4747"}]},{"id":"S016","semantic_type":"notation","kind":"notation","title":"复合函数简记","source_units":[{"start":84019,"end":84253,"start_line":912,"end_line":913,"raw_tex":"with the notation $z:=(x, y)$, $g(x,y):= g_0(x, y, y)$, and $f(x,y) := y^\\top f_0  (x,y)$, %introduced in Section \\ref{Basic concepts and background material}\nas well as the optimal value function $\\varphi$ defined in \\eqref{varphi}.\n","sha256_utf8":"5fc8e0ee93f3b96b3f55c9ab70ae6b4fa65bcf26e01859191a05e7bb7faf3ea7"}]},{"id":"S017","semantic_type":"lemma","kind":"lemma","title":"约束逆像的法锥估计","source_units":[{"start":28361,"end":29757,"start_line":268,"end_line":285,"raw_tex":"\\begin{lemma}[see Theorem 6.10 in \\cite{MordukhovichGeneralizedDifferential1994} or Theorem 6.14 in  \\cite{RockafellarWetsBook1998}]\\label{normal cone estimate}\nLet $C:=\\Omega \\cap \\psi^{-1}(\\Xi)$, where $\\Omega \\subseteq \\mathbb{R}^n$ and $\\Xi\\subseteq \\mathbb{R}^m$\nare closed sets and the function $\\psi :\\mathbb{R}^n \\rightarrow \\mathbb{R}^m$ is  Lipschitz  around $\\bar{x}$. Furthermore, assume that %Assume that the following qualification condition holds at $\\bar{x} \\in C $: whenever $v\\in   N_{\\varsigma}(\\psi(\\bar{x}))$, be such that\n\\begin{equation}\\label{BCQ}\n\\left.\n\\begin{array}{r}\n   0\\in  \\partial \\langle v, \\psi \\rangle (\\bar{x})+N_{\\Omega}(\\bar x)\\\\\n   v\\in   N_{\\Xi}(\\psi(\\bar{x}))\n\\end{array}\n\\right\\} \\Longrightarrow v=0.    \n\\end{equation}\n%{ {(The limiting subdiffential does not seem to have been introduced yet; this should be done and link to Clarke subdifferential discussed. This is needed in the proof of Theorem 3.1.)}}\nThen the following estimate holds:\n\\begin{equation}\\label{normal cone to operator constraint}\n   N_C(\\bar x) \\subseteq     \\bigcup\\left\\{\\partial \\langle v, \\psi \\rangle (\\bar x) + N_{\\Omega}(\\bar x)\\left|\\; v\\in N_\\Xi(\\psi(\\bar x)) \\right.\\right\\}.\n\\end{equation}\nEquality holds in \\eqref{normal cone to operator constraint}, provided that $\\Xi$ is normally regular at $\\psi(\\bar x) $ and $ \\Omega$ is normally regular at $\\bar{x}$.\n\\end{lemma}","sha256_utf8":"286bb30ae2e24919bc96abc9de030975f49d88d492d30ea906a50cfa87c4d560"}]},{"id":"S018","semantic_type":"theorem","kind":"theorem","title":"非凸QVI余导数与稳定性","source_units":[{"start":45763,"end":48179,"start_line":436,"end_line":474,"raw_tex":"\\begin{theorem}\\label{I}\nConsider a point $(\\bar x, \\bar y)$ such that $\\bar y\\in S(\\bar x)$ and let $\\bar{\\varsigma} \\in S_0(\\bar x, \\bar y)$.\n\\begin{itemize}\n  \\item[\\emph{(i)}]Suppose that the MFCQ \\eqref{LL regularity} holds at $(\\bar x, \\bar y, \\bar y)$ and $(\\bar x, \\bar y, \\bar{\\varsigma})$.\n  Furthermore, assume that $S_0$ \\eqref{S-v} is inner semicontinuous at $(\\bar x, \\bar y, \\bar{\\varsigma})$ and let the set-valued mapping\n\\begin{equation}\\label{PsiCalm}\n\\Psi(\\theta):=\\left\\{(x,y)\\in \\mathbb{R}^n \\times \\mathbb{R}^m\\left|\\; g_0(x, y, y)\\leq 0, \\;\\; y^\\top f_0(x, y) - \\varphi(x, y)\\leq \\theta \\right.\\right\\}\n\\end{equation}\n be calm at $(0, \\bar x, \\bar y)$. Then for all $y^*\\in \\mathbb{R}^m$, it holds that \n{\\small{\n\\begin{equation}\\label{CoD S}\n\\begin{array}{l}\n  D^*S(\\bar x, \\bar y)(y^*) \\subseteq\n   \\underset{(v, w, \\lambda) \\in \\Lambda^0 (\\bar x, \\bar y, \\bar{\\varsigma}, y^*)} \\bigcup\n  % \\left\\{\\nabla_1 g_0(\\bar x, \\bar y, \\bar y)^\\top v  \\underset{(v, w, \\lambda) \\in \\Lambda^0 (\\bar x, \\bar y, \\bar{\\varsigma}, y^*)}  \\bigcup\n \\left\\{\\nabla_1 g_0(\\bar x, \\bar y, \\bar y)^\\top v + \\lambda \\left(\\displaystyle \n  \\nabla_1 f_{0}(\\bar x, \\bar y)^\\top  (\\bar y- \\bar{\\varsigma})  - \\nabla_1 g_0(\\bar x, \\bar y, \\bar{\\varsigma})^\\top w\\right)\\right\\}.\n\\end{array}\n\\end{equation}}}\n\\item[\\emph{(ii)}] Suppose that in addition to the assumptions in \\emph{(i)}, the following qualification condition holds:\n{\\small{\n\\begin{equation}\\label{Stab CQ-1}\n\\begin{array}{l}\n(v, w, \\lambda) \\in \\Lambda^0(\\bar x, \\bar y, \\bar{\\varsigma}, 0) \\;\\; \\Longrightarrow \\;\\;  \n  \\left[\\nabla_1 g_0(\\bar x, \\bar y, \\bar y)^\\top v\n + \\lambda\\left(\\displaystyle \\nabla_1 f_{0}(\\bar x, \\bar y)^\\top  (\\bar y- \\bar{\\varsigma})\n    - \\nabla_1 g_0(\\bar x, \\bar y, \\bar{\\varsigma})^\\top w\\right) = 0\\right].\n\\end{array}\n\\end{equation}}}\n${}$\\\\\nThen the set-valued mapping $S$ \\eqref{S-Soultion-Map}  is Lipschitz-like around $(\\bar x, \\bar y)$ and its Lipschitz modulus at this point has the following upper estimate:\n\\[\n\\text{\\emph{lip}}\\, S(\\bar x, \\bar y) \\leq \\underset{\\underset{(v, w, \\lambda)\\in \\Lambda^0(\\bar x, \\bar y, \\bar{\\varsigma}, y^*)}{\\|y^*\\|\\leq 1}}{\\sup}\n\\left\\{\\left\\|\\nabla_1 g_0(\\bar x, \\bar y, \\bar y)^\\top v + \\lambda\\left(\\nabla_1 f_{0}(\\bar x, \\bar y)^\\top  (\\bar y- \\bar{\\varsigma})\n-\\nabla_1 g_0(\\bar x,\\bar y,\\bar{\\varsigma})^\\top w\\right)\\right\\|\\right\\}.\n\\]\n\\end{itemize}\n\n\\end{theorem}","sha256_utf8":"73887d1b9da36d9dc3ddefdcc5a97bb0c2533f23baa9ced96caa77ac9262ff86"}]},{"id":"S019","semantic_type":"proof","kind":"proof","title":"余导数与稳定性完整证明","source_units":[{"start":48180,"end":53775,"start_line":475,"end_line":542,"raw_tex":"\\begin{proof}\nTo begin, note that with the expression of the set-valued mapping $S$ \\eqref{VI-1}, we have  \n\\[\n\\begin{array}{l}\n\\mbox{gph}S = \\Omega \\cap \\psi^{-1}(\\mathbb{R}_-)\\,\n\\mbox{ with } \\, \\Omega:=\\left\\{(x, y)\\in \\mathbb{R}^n\\times \\mathbb{R}^m\\left|\\, \\tilde{g}(x, y)\\leq 0 \\right.\\right\\},\\, \\psi(x, y):= \\tilde{f}(x, y) - \\varphi(x, y),\n\\end{array}\n\\]\nwhere $\\tilde{g}(x,y):= g_0(x, y, y)$ and $\\tilde{f}(x, y):=y^\\top f_0(x, y)$. Then observe that the function $\\varphi$ is Lipshitz continuous around  $(\\bar x, \\bar y)$ given that the set-valued mapping $S_0$ \\eqref{S-v} is inner semicontinuous at $(\\bar x, \\bar y, \\bar{\\varsigma})$ and the MFCQ is satisfied at this same point. Hence, from Lemma \\ref{normal cone estimate}, with the calmness of the set-valued mapping \\eqref{PsiCalm}, as corresponding expression of \\eqref{CalmnesMapAgain}, it follows that \n\\begin{equation} \\label{estimate-eq-1}\n    N_{\\text{gph}S} ( \\bar{x}, \\bar{y}) \\subseteq \\bigcup \\left \\{\\partial \\langle\\lambda, \\psi\\rangle (\\bar{x}, \\bar{y}) + N_\\Omega (\\bar{x}, \\bar{y}) \\left |\\; \\lambda \\ge 0\\right.\\right\\},\n\\end{equation}\nwhile considering the fact that $\\psi(\\bar x, \\bar y)=0$, as $\\bar y\\in S(\\bar x)$, and the functions $\\tilde{f}$ and $\\tilde{g}$ continuously differentiable. Now, considering the definition of the coderivative and inclusion \\eqref{estimate-eq-1}, it follows that for all $y^*\\in \\mathbb{R}^m$, \n\\begin{equation} \\label{estimate-eq-2}\n  D^*S(\\bar{x}, \\bar{y}) (y^*) \\subseteq \\{ x^* \\in \\mathbb{R}^n \\left |\\; \\exists \\lambda \\ge 0:\\quad  \\left(x^*, -y^*\\right)\\in \\partial \\langle\\lambda, \\psi\\rangle (\\bar{x}, \\bar{y}) + N_\\Omega(\\bar{x}, \\bar{y}) \\right \\}.\n\\end{equation}\nSince the MFCQ also holds at $(\\bar{x}, \\bar{y}, \\bar{y})$, using Lemma \\ref{normal cone estimate} one more time, \n\\begin{equation*}\\label{NOmega}\n    N_\\Omega(\\bar{x}, \\bar{y}) \\subseteq \\left\\{\\nabla g(\\bar{x}, \\bar{y})^\\top v \\left|\\; v \\ge 0,\\; g(\\bar{x}, \\bar{y}) \\le 0, \\; v^T g(\\bar{x}, \\bar{y})=0\\right.\\right\\}.\n\\end{equation*}\n\nOn the other hand,  observe that \n\\begin{equation*}\\label{partialLP}\n \\partial \\langle\\lambda, \\psi\\rangle (\\bar{x}, \\bar{y}) \\subseteq \\lambda \\nabla f(\\bar{x}, \\bar{y}) + \\partial \\langle \\lambda, \\, -\\varphi\\rangle (\\bar{x}, \\bar{y}) \\subseteq \\lambda \\nabla f(\\bar{x}, \\bar{y}) -\\lambda \\bar{\\partial} \\varphi (\\bar{x}, \\bar{y})  \n\\end{equation*}\ngiven that as $\\varphi$ is Lipschitz continuous around $(\\bar x, \\bar y)$, its limiting subdifferential is included in the Clarke subdifferential, according to \\eqref{LimitingClarke}. Furthermore, it follows from \\cite[Corollary 5.3]{MordukhovichNamPhanVarAnalMargBlP} that the combination of the inner semicontinuity of $S_0$ and MFCQ \\eqref{LL regularity} both holding at the point $(\\bar x, \\bar y, \\bar{\\varsigma})$ leads to\n\\begin{equation}\\label{SubPhi}\n\\begin{array}{c}\n\\bar\\partial \\varphi(\\bar x, \\bar y) \\;\\, \\subseteq \\;\\, \\left\\{\\left.\\nabla f_0(\\bar x, \\bar y)^\\top\\bar{\\varsigma}  +  \\nabla_{1, 2} g_0(\\bar x, \\bar y, \\bar{\\varsigma})^\\top w\\right|\\;\\, w\\in \\Lambda (\\bar x, \\bar y, \\bar{\\varsigma})\\right\\}.\n\\end{array}\n\\end{equation}\n\nAlso note that as $\\tilde{f}(x, y) = y^T f_0(x, y)$, it holds that\n\\begin{equation}\\label{NablaF}\n  \\nabla \\tilde{f}(x,y)= f_0(x,y) + \\nabla f_0(x, y)^\\top y.\n\\end{equation}\nNow, let $x^* \\in D^*S(\\bar{x}, \\bar{y})(y^*)$. Then it follows from \\eqref{estimate-eq-2}--\\eqref{NablaF} that we can find $\\lambda$, $v$, $w$ such that \n\\[\n\\begin{array}{r}\n\\lambda \\geq 0,\\; v\\geq 0, \\; g_0(\\bar x, \\bar y, \\bar{y})\\leq 0, \\; v^\\top g_0(\\bar x, \\bar y, \\bar{y})=0,\\;  w\\in \\Lambda(\\bar x, \\bar y, \\bar\\varsigma),\\\\[1.5ex]\n\\left[\\begin{array}{r}\n     x^*\\\\\n     -y^*\n\\end{array}\\right]=\\left[\\begin{array}{r}\n \\lambda \\left(\n  \\nabla_1 f_{0}(\\bar x, \\bar y)^\\top  (\\bar y- \\bar{\\varsigma})  - \\nabla_1 g_0(\\bar x, \\bar y, \\bar{\\varsigma})^\\top w\\right) + \\nabla_1 g_0(\\bar x, \\bar y, \\bar y)^\\top v\\\\[1.5ex]\n  \\lambda\\left.\\left(f_0(\\bar x, \\bar y) + \\nabla_2f_0(\\bar x, \\bar y)^\\top (\\bar y - \\bar\\varsigma) - \\nabla_2 g_{0}(\\bar x, \\bar y, \\bar\\varsigma)^{\\top}w  \\right)\\right. + \\nabla_2 g_{0}(\\bar x, \\bar y, \\bar y)^{\\top}v\n\\end{array}\\right].\n\\end{array}\n\\]\nConsidering this and the definition of $\\Lambda^0 (\\bar x, \\bar y, \\bar{\\varsigma}, y^*)$, we have inclusion \\eqref{CoD S}.\n\n\n\nFor assertion (ii), observe that from \\eqref{CoD S}, we have the following inclusion:\n\\[\n\\begin{array}{ll}\n  D^*S(\\bar x, \\bar y)(0) \\subseteq\n  & \\underset{(v, w, \\lambda) \\in \\Lambda^0 (\\bar x, \\bar y, \\bar\\varsigma, 0)}  \\bigcup\n  \\left\\{\\lambda\\left(\\nabla_1 f_{0}(\\bar x, \\bar y)^\\top (\\bar y - \\bar\\varsigma) - \\nabla_1 g_0(\\bar x, \\bar y, \\bar\\varsigma)^\\top w\\right) + \\nabla_1 g_0(\\bar x, \\bar y, \\bar y)^\\top v\\right\\}.\n\\end{array}\n\\]\nHence, it is clear that condition \\eqref{Stab CQ-1} is sufficient for $D^*S(\\bar x, \\bar y)(0) \\subseteq \\{0\\}$. Thus, since the coderivative mapping is positively homogeneous (i.e., in particular, we have $0\\in D^*S(\\bar x, \\bar y)(0)$),  the coderivative/Mordukhovich criterion $D^*S(\\bar x, \\bar y)(0) = \\{0\\}$ is satisfied. This implies that $S$ is Lipschitz-like around $(\\bar x, \\bar y)$, and it follows from \\eqref{LipModulus} that the exact Lipschitzian bound of $S$ around $(\\bar x, \\bar y)$ can be obtained as\n$$\n\\text{{lip}}\\, S(\\bar x, \\bar y) = \\|D^*S(\\bar x, \\bar y)\\|^+:=\\sup\\left\\{\\|u\\|\\left|\\, u\\in D^*S(\\bar x, \\bar y)(y^*), \\, \\|y^*\\|\\leq 1\\right.\\right\\}.\n$$\nConsidering inclusion \\eqref{CoD S} one more time, we have the stated upper bound on the quantity  $\\text{{lip}}\\, S(\\bar x, \\bar y)$. \\qed\n\\end{proof}","sha256_utf8":"404780dc4880c32f1a7ae100186415304138e4572a9325d7796c5d25c000d84f"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"重构解图并估计法锥","source_units":[{"start":48180,"end":49301,"start_line":475,"end_line":486,"raw_tex":"\\begin{proof}\nTo begin, note that with the expression of the set-valued mapping $S$ \\eqref{VI-1}, we have  \n\\[\n\\begin{array}{l}\n\\mbox{gph}S = \\Omega \\cap \\psi^{-1}(\\mathbb{R}_-)\\,\n\\mbox{ with } \\, \\Omega:=\\left\\{(x, y)\\in \\mathbb{R}^n\\times \\mathbb{R}^m\\left|\\, \\tilde{g}(x, y)\\leq 0 \\right.\\right\\},\\, \\psi(x, y):= \\tilde{f}(x, y) - \\varphi(x, y),\n\\end{array}\n\\]\nwhere $\\tilde{g}(x,y):= g_0(x, y, y)$ and $\\tilde{f}(x, y):=y^\\top f_0(x, y)$. Then observe that the function $\\varphi$ is Lipshitz continuous around  $(\\bar x, \\bar y)$ given that the set-valued mapping $S_0$ \\eqref{S-v} is inner semicontinuous at $(\\bar x, \\bar y, \\bar{\\varsigma})$ and the MFCQ is satisfied at this same point. Hence, from Lemma \\ref{normal cone estimate}, with the calmness of the set-valued mapping \\eqref{PsiCalm}, as corresponding expression of \\eqref{CalmnesMapAgain}, it follows that \n\\begin{equation} \\label{estimate-eq-1}\n    N_{\\text{gph}S} ( \\bar{x}, \\bar{y}) \\subseteq \\bigcup \\left \\{\\partial \\langle\\lambda, \\psi\\rangle (\\bar{x}, \\bar{y}) + N_\\Omega (\\bar{x}, \\bar{y}) \\left |\\; \\lambda \\ge 0\\right.\\right\\},\n\\end{equation}\n","sha256_utf8":"ede4fc7a30b2202cdb8ccb303f56cd58172f1f48276f85b92f8b8edfd1dc2a37"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"转化为余导数包含式","source_units":[{"start":49301,"end":49877,"start_line":487,"end_line":490,"raw_tex":"while considering the fact that $\\psi(\\bar x, \\bar y)=0$, as $\\bar y\\in S(\\bar x)$, and the functions $\\tilde{f}$ and $\\tilde{g}$ continuously differentiable. Now, considering the definition of the coderivative and inclusion \\eqref{estimate-eq-1}, it follows that for all $y^*\\in \\mathbb{R}^m$, \n\\begin{equation} \\label{estimate-eq-2}\n  D^*S(\\bar{x}, \\bar{y}) (y^*) \\subseteq \\{ x^* \\in \\mathbb{R}^n \\left |\\; \\exists \\lambda \\ge 0:\\quad  \\left(x^*, -y^*\\right)\\in \\partial \\langle\\lambda, \\psi\\rangle (\\bar{x}, \\bar{y}) + N_\\Omega(\\bar{x}, \\bar{y}) \\right \\}.\n\\end{equation}\n","sha256_utf8":"2066ca0689c3c48529ea971e0f14e693b6562dcc78cfb7419c02014db90c1f5f"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"估计可行集法锥","source_units":[{"start":49877,"end":50214,"start_line":491,"end_line":494,"raw_tex":"Since the MFCQ also holds at $(\\bar{x}, \\bar{y}, \\bar{y})$, using Lemma \\ref{normal cone estimate} one more time, \n\\begin{equation*}\\label{NOmega}\n    N_\\Omega(\\bar{x}, \\bar{y}) \\subseteq \\left\\{\\nabla g(\\bar{x}, \\bar{y})^\\top v \\left|\\; v \\ge 0,\\; g(\\bar{x}, \\bar{y}) \\le 0, \\; v^T g(\\bar{x}, \\bar{y})=0\\right.\\right\\}.\n\\end{equation*}\n","sha256_utf8":"080615ae6c66799405600eb5bfd4b7e0275d4a64eda67e6b507ede1505ff2078"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"估计值函数次微分","source_units":[{"start":50215,"end":51316,"start_line":496,"end_line":505,"raw_tex":"On the other hand,  observe that \n\\begin{equation*}\\label{partialLP}\n \\partial \\langle\\lambda, \\psi\\rangle (\\bar{x}, \\bar{y}) \\subseteq \\lambda \\nabla f(\\bar{x}, \\bar{y}) + \\partial \\langle \\lambda, \\, -\\varphi\\rangle (\\bar{x}, \\bar{y}) \\subseteq \\lambda \\nabla f(\\bar{x}, \\bar{y}) -\\lambda \\bar{\\partial} \\varphi (\\bar{x}, \\bar{y})  \n\\end{equation*}\ngiven that as $\\varphi$ is Lipschitz continuous around $(\\bar x, \\bar y)$, its limiting subdifferential is included in the Clarke subdifferential, according to \\eqref{LimitingClarke}. Furthermore, it follows from \\cite[Corollary 5.3]{MordukhovichNamPhanVarAnalMargBlP} that the combination of the inner semicontinuity of $S_0$ and MFCQ \\eqref{LL regularity} both holding at the point $(\\bar x, \\bar y, \\bar{\\varsigma})$ leads to\n\\begin{equation}\\label{SubPhi}\n\\begin{array}{c}\n\\bar\\partial \\varphi(\\bar x, \\bar y) \\;\\, \\subseteq \\;\\, \\left\\{\\left.\\nabla f_0(\\bar x, \\bar y)^\\top\\bar{\\varsigma}  +  \\nabla_{1, 2} g_0(\\bar x, \\bar y, \\bar{\\varsigma})^\\top w\\right|\\;\\, w\\in \\Lambda (\\bar x, \\bar y, \\bar{\\varsigma})\\right\\}.\n\\end{array}\n\\end{equation}\n","sha256_utf8":"4fa27891fd9c65f6c92f4cde0cdcf31921c7eca121656efa3aecd6e3b5cdcb1f"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"计算复合目标梯度","source_units":[{"start":51317,"end":51491,"start_line":507,"end_line":510,"raw_tex":"Also note that as $\\tilde{f}(x, y) = y^T f_0(x, y)$, it holds that\n\\begin{equation}\\label{NablaF}\n  \\nabla \\tilde{f}(x,y)= f_0(x,y) + \\nabla f_0(x, y)^\\top y.\n\\end{equation}\n","sha256_utf8":"52af96d7b522bd70fdd4cb9fa438f3c2a0519facca71c31e7b4663c9c7a9f170"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"合并乘子得到余导数估计","source_units":[{"start":51491,"end":52497,"start_line":511,"end_line":525,"raw_tex":"Now, let $x^* \\in D^*S(\\bar{x}, \\bar{y})(y^*)$. Then it follows from \\eqref{estimate-eq-2}--\\eqref{NablaF} that we can find $\\lambda$, $v$, $w$ such that \n\\[\n\\begin{array}{r}\n\\lambda \\geq 0,\\; v\\geq 0, \\; g_0(\\bar x, \\bar y, \\bar{y})\\leq 0, \\; v^\\top g_0(\\bar x, \\bar y, \\bar{y})=0,\\;  w\\in \\Lambda(\\bar x, \\bar y, \\bar\\varsigma),\\\\[1.5ex]\n\\left[\\begin{array}{r}\n     x^*\\\\\n     -y^*\n\\end{array}\\right]=\\left[\\begin{array}{r}\n \\lambda \\left(\n  \\nabla_1 f_{0}(\\bar x, \\bar y)^\\top  (\\bar y- \\bar{\\varsigma})  - \\nabla_1 g_0(\\bar x, \\bar y, \\bar{\\varsigma})^\\top w\\right) + \\nabla_1 g_0(\\bar x, \\bar y, \\bar y)^\\top v\\\\[1.5ex]\n  \\lambda\\left.\\left(f_0(\\bar x, \\bar y) + \\nabla_2f_0(\\bar x, \\bar y)^\\top (\\bar y - \\bar\\varsigma) - \\nabla_2 g_{0}(\\bar x, \\bar y, \\bar\\varsigma)^{\\top}w  \\right)\\right. + \\nabla_2 g_{0}(\\bar x, \\bar y, \\bar y)^{\\top}v\n\\end{array}\\right].\n\\end{array}\n\\]\nConsidering this and the definition of $\\Lambda^0 (\\bar x, \\bar y, \\bar{\\varsigma}, y^*)$, we have inclusion \\eqref{CoD S}.\n","sha256_utf8":"de35ffb33807937e0b7964c8f5271fc3bd8d4af7e1c6c6cffb5428f26fc1f94f"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"专门化零方向余导数","source_units":[{"start":52500,"end":52945,"start_line":529,"end_line":536,"raw_tex":"For assertion (ii), observe that from \\eqref{CoD S}, we have the following inclusion:\n\\[\n\\begin{array}{ll}\n  D^*S(\\bar x, \\bar y)(0) \\subseteq\n  & \\underset{(v, w, \\lambda) \\in \\Lambda^0 (\\bar x, \\bar y, \\bar\\varsigma, 0)}  \\bigcup\n  \\left\\{\\lambda\\left(\\nabla_1 f_{0}(\\bar x, \\bar y)^\\top (\\bar y - \\bar\\varsigma) - \\nabla_1 g_0(\\bar x, \\bar y, \\bar\\varsigma)^\\top w\\right) + \\nabla_1 g_0(\\bar x, \\bar y, \\bar y)^\\top v\\right\\}.\n\\end{array}\n\\]\n","sha256_utf8":"ce7e0d505352a18f84f77a8d3313affc8512ea63a574be9641c734d7f3bef921"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"应用稳定性判据与模公式","source_units":[{"start":52945,"end":53624,"start_line":537,"end_line":540,"raw_tex":"Hence, it is clear that condition \\eqref{Stab CQ-1} is sufficient for $D^*S(\\bar x, \\bar y)(0) \\subseteq \\{0\\}$. Thus, since the coderivative mapping is positively homogeneous (i.e., in particular, we have $0\\in D^*S(\\bar x, \\bar y)(0)$),  the coderivative/Mordukhovich criterion $D^*S(\\bar x, \\bar y)(0) = \\{0\\}$ is satisfied. This implies that $S$ is Lipschitz-like around $(\\bar x, \\bar y)$, and it follows from \\eqref{LipModulus} that the exact Lipschitzian bound of $S$ around $(\\bar x, \\bar y)$ can be obtained as\n$$\n\\text{{lip}}\\, S(\\bar x, \\bar y) = \\|D^*S(\\bar x, \\bar y)\\|^+:=\\sup\\left\\{\\|u\\|\\left|\\, u\\in D^*S(\\bar x, \\bar y)(y^*), \\, \\|y^*\\|\\leq 1\\right.\\right\\}.\n$$\n","sha256_utf8":"e6803c8a1c9a62748d2d590a0f490f8908971f6bca2ddba707600a2bbac4405e"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"推出Lipschitz模上界","source_units":[{"start":53624,"end":53775,"start_line":541,"end_line":542,"raw_tex":"Considering inclusion \\eqref{CoD S} one more time, we have the stated upper bound on the quantity  $\\text{{lip}}\\, S(\\bar x, \\bar y)$. \\qed\n\\end{proof}","sha256_utf8":"25acd00571ef094b81503227e5c3c2c7e1c37d751053b8f71df3a9adbe2f2f33"}]},{"id":"S029","semantic_type":"assumption","kind":"assumption","title":"解点与下层正则性假设","source_units":[{"start":45788,"end":46202,"start_line":437,"end_line":440,"raw_tex":"Consider a point $(\\bar x, \\bar y)$ such that $\\bar y\\in S(\\bar x)$ and let $\\bar{\\varsigma} \\in S_0(\\bar x, \\bar y)$.\n\\begin{itemize}\n  \\item[\\emph{(i)}]Suppose that the MFCQ \\eqref{LL regularity} holds at $(\\bar x, \\bar y, \\bar y)$ and $(\\bar x, \\bar y, \\bar{\\varsigma})$.\n  Furthermore, assume that $S_0$ \\eqref{S-v} is inner semicontinuous at $(\\bar x, \\bar y, \\bar{\\varsigma})$ and let the set-valued mapping\n","sha256_utf8":"507c836f3f6c51d805944b389d6ad6793a3911ae56d41a752ff8d6cdd98b0e5d"}]},{"id":"S030","semantic_type":"assumption","kind":"assumption","title":"扰动映射平静性假设","source_units":[{"start":46202,"end":46491,"start_line":441,"end_line":444,"raw_tex":"\\begin{equation}\\label{PsiCalm}\n\\Psi(\\theta):=\\left\\{(x,y)\\in \\mathbb{R}^n \\times \\mathbb{R}^m\\left|\\; g_0(x, y, y)\\leq 0, \\;\\; y^\\top f_0(x, y) - \\varphi(x, y)\\leq \\theta \\right.\\right\\}\n\\end{equation}\n be calm at $(0, \\bar x, \\bar y)$. Then for all $y^*\\in \\mathbb{R}^m$, it holds that \n","sha256_utf8":"39d7da68959755a958350068244472ea4901628bc15969b154a2b5178c6fb2c8"}]},{"id":"S031","semantic_type":"assumption","kind":"assumption","title":"零方向乘子资格条件","source_units":[{"start":47070,"end":47599,"start_line":455,"end_line":464,"raw_tex":"\\item[\\emph{(ii)}] Suppose that in addition to the assumptions in \\emph{(i)}, the following qualification condition holds:\n{\\small{\n\\begin{equation}\\label{Stab CQ-1}\n\\begin{array}{l}\n(v, w, \\lambda) \\in \\Lambda^0(\\bar x, \\bar y, \\bar{\\varsigma}, 0) \\;\\; \\Longrightarrow \\;\\;  \n  \\left[\\nabla_1 g_0(\\bar x, \\bar y, \\bar y)^\\top v\n + \\lambda\\left(\\displaystyle \\nabla_1 f_{0}(\\bar x, \\bar y)^\\top  (\\bar y- \\bar{\\varsigma})\n    - \\nabla_1 g_0(\\bar x, \\bar y, \\bar{\\varsigma})^\\top w\\right) = 0\\right].\n\\end{array}\n\\end{equation}}}\n","sha256_utf8":"4ff03839a62d7557a2194652c1a017aa65e80397ea5694944c314742f889d1d4"}]},{"id":"S032","semantic_type":"external_fact","kind":"external_fact","title":"标准法锥逆像演算","source_units":[],"declared_external":true,"declaration_text":["The normal-cone inverse-image estimate under the basic constraint qualification is the cited result stated in B0026. Its inclusion remains valid under calmness of the perturbation mapping defined at lines 320–323, by Henrion, Jourani and Outrata, Theorem 4.1."]},{"id":"S033","semantic_type":"external_fact","kind":"external_fact","title":"标准次微分演算","source_units":[],"declared_external":true,"declaration_text":["For a locally Lipschitz function, the limiting subdifferential is contained in the Clarke subdifferential, whose closed convexification representation is stated at lines 259–263. Smooth addition, nonnegative scalar multiplication, and Clarke negation calculus give the inclusions used at lines 497–500."]},{"id":"S034","semantic_type":"external_fact","kind":"external_fact","title":"边际函数敏感性定理","source_units":[],"declared_external":true,"declaration_text":["For the lower-level problem defining the value function, inner semicontinuity of its solution mapping at the chosen optimizer together with lower-level MFCQ yields local Lipschitz continuity of the value function and the Clarke subdifferential estimate displayed at lines 501–505; the author invokes Mordukhovich, Nam and Phan, Corollary 5.3."]},{"id":"S035","semantic_type":"external_fact","kind":"external_fact","title":"余导数稳定性判据","source_units":[],"declared_external":true,"declaration_text":["For a mapping with locally closed graph, the Aubin property at a graph point is equivalent to its limiting coderivative at zero being the singleton zero, as stated at lines 335–338."]},{"id":"S036","semantic_type":"external_fact","kind":"external_fact","title":"精确Lipschitz模公式","source_units":[],"declared_external":true,"declaration_text":["The exact Lipschitz modulus of a mapping with the Aubin property equals the outer norm of its limiting coderivative, as stated at lines 339–352 and attributed to Rockafellar and Wets, Theorem 9.40."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S018","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S001","target_id":"S014","kind":"supports"},{"source_id":"S002","target_id":"S015","kind":"supports"},{"source_id":"S003","target_id":"S012","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S005","target_id":"S017","kind":"supports"},{"source_id":"S006","target_id":"S017","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S032","target_id":"S017","kind":"supports"},{"source_id":"S032","target_id":"S009","kind":"supports"},{"source_id":"S002","target_id":"S020","kind":"supports"},{"source_id":"S007","target_id":"S020","kind":"supports"},{"source_id":"S012","target_id":"S020","kind":"supports"},{"source_id":"S015","target_id":"S020","kind":"supports"},{"source_id":"S017","target_id":"S020","kind":"supports"},{"source_id":"S009","target_id":"S020","kind":"supports"},{"source_id":"S029","target_id":"S020","kind":"supports"},{"source_id":"S030","target_id":"S020","kind":"supports"},{"source_id":"S034","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S010","target_id":"S021","kind":"supports"},{"source_id":"S012","target_id":"S022","kind":"supports"},{"source_id":"S016","target_id":"S022","kind":"supports"},{"source_id":"S017","target_id":"S022","kind":"supports"},{"source_id":"S029","target_id":"S022","kind":"supports"},{"source_id":"S020","target_id":"S022","kind":"supports"},{"source_id":"S004","target_id":"S023","kind":"supports"},{"source_id":"S006","target_id":"S023","kind":"supports"},{"source_id":"S014","target_id":"S023","kind":"supports"},{"source_id":"S016","target_id":"S023","kind":"supports"},{"source_id":"S020","target_id":"S023","kind":"supports"},{"source_id":"S029","target_id":"S023","kind":"supports"},{"source_id":"S033","target_id":"S023","kind":"supports"},{"source_id":"S034","target_id":"S023","kind":"supports"},{"source_id":"S001","target_id":"S024","kind":"supports"},{"source_id":"S003","target_id":"S024","kind":"supports"},{"source_id":"S020","target_id":"S024","kind":"supports"},{"source_id":"S013","target_id":"S025","kind":"supports"},{"source_id":"S021","target_id":"S025","kind":"supports"},{"source_id":"S022","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S031","target_id":"S027","kind":"supports"},{"source_id":"S008","target_id":"S027","kind":"supports"},{"source_id":"S010","target_id":"S027","kind":"supports"},{"source_id":"S011","target_id":"S027","kind":"supports"},{"source_id":"S035","target_id":"S027","kind":"supports"},{"source_id":"S036","target_id":"S027","kind":"supports"},{"source_id":"S025","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S019","kind":"supports"},{"source_id":"S022","target_id":"S019","kind":"supports"},{"source_id":"S023","target_id":"S019","kind":"supports"},{"source_id":"S024","target_id":"S019","kind":"supports"},{"source_id":"S025","target_id":"S019","kind":"supports"},{"source_id":"S026","target_id":"S019","kind":"supports"},{"source_id":"S027","target_id":"S019","kind":"supports"},{"source_id":"S028","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"}]}}
{"record_id":"2212.01504v1","paper":{"title":"A mirror inertial forward-reflected-backward splitting: Global convergence and linesearch extension beyond convexity and Lipschitz smoothness","authors":["Ziyuan Wang","Andreas Themelis","Hongjia Ou","Xianfu Wang"],"arxiv_id":"2212.01504","version":"v1","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-024-02383-9","abstract_url":"https://arxiv.org/abs/2212.01504v1","pdf_url":"https://arxiv.org/pdf/2212.01504v1"},"context":{"raw_tex":"\\begin{equation*}\n\t\\tag{P}\\label{eq:P}\n\t\\minimize_{x\\in\\R^n}\\varphi(x)\\coloneqq f(x)+g(x)\n\\quad\n\t\\stt x\\in\\closure C,\n\\end{equation*}\n\nwhere \\(C\\) is a nonempty open and convex set with closure \\(\\closure C\\),  \\(\\func{f}{\\R^n}{\\Rinf\\coloneqq\\R\\cup\\set{\\pm\\infty}}\\) is differentiable on $\\inte\\dom f\\neq\\emptyset$, and \\(\\func{g}{\\R^n}{\\Rinf}\\) is proper and lower semicontinuous (lsc).\nFor notational brevity, we define \\(\\cost\\coloneqq\\varphi+\\indicator_{\\closure C}\\) with \\(\\indicator_X\\) denoting the indicator function of set \\(X\\subseteq\\R^n\\), namely such that \\(\\indicator_X(x)=0\\) if \\(x\\in X\\) and \\(\\infty\\) otherwise.\n\n\n\t\t\tThroughout, we fix a Legendre kernel \\(\\func{h}{\\R^n}{\\Rinf}\\) with \\(\\dom\\nabla h=\\interior\\dom h=C\\),\nnamely, a proper, lsc, and strictly convex function  that is 1-coercive and \\DEF{essentially smooth}, \\ie such that \\(\\|\\nabla h(x_k)\\|\\to\\infty\\) for every sequence \\(\\seq{x_k}\\subset C\\) converging to a boundary point of \\(C\\).\nWe will consider the following iterative scheme for addressing problem \\eqref{eq:P},\nwhere \\(\\func{\\D}{\\R^n\\times\\R^n}{\\Rinf}\\) denotes the \\DEF{Bregman distance} induced by \\(h\\), defined as\n\\begin{equation}\\label{eq:D}\n\t\\D(x,y)\n{}\\coloneqq{}\n\t\\begin{ifcases}\n\t\th(x)-h(y)-\\innprod{\\nabla h(y)}{x-y} & y\\in C,\n\t\\\\\n\t\t\\infty\\otherwise.\n\t\\end{ifcases}\n\\end{equation}\n\n\\begin{algorithm}\n\t\\caption{Mirror inertial forward-reflected-backward (\\protect\\BiFRB)}%\n\t\\label{alg:BiFRB}%\n\t\\begin{algorithmic}[1]%\n\\setlength{\\itemsep}{1ex}%\n\\item[%\n\tChoose \\(x^{-1},x^0\\in C\\),~\n\tinertial parameter \\(\\beta\\in\\R\\),~ and\n\tstepsize \\(\\gamma>0\\)%\n]%\n\\item[%\n\tIterate for \\(k=0,1,\\ldots\\) until a termination criterion is met (cf. \\cref{thm:termination})\n]%\n\\State\\label{state:BiFRB:y}%\n\tSet \\(y^k\\) such that\n\t\\(\n\t\t\\nabla h(y^k)\n\t{}={}\n\t\t\\nabla h(x^k)-\\gamma\\bigl(\\nabla f(x^k)-\\nabla f(x^{k-1})\\bigr)\n\t\\)%\n\\State\\label{state:BiFRB:x}%\n\tChoose\n\t\\(\n\t\\ifsiam\n\t\t\\mathtight\n\t\\fi\n\t\tx^{k+1}\n\t{}\\in{}\n\t\t\\argmin\\limits_{w\\in\\R^n}\\set{\n\t\t\tg(w)\n\t\t\t{}+{}\n\t\t\t\\innprod*{w}{\\nabla f(x^k)-\\tfrac\\beta\\gamma\\bigl(\\nabla h(x^k)-\\nabla h(x^{k-1})\\bigr)}\n\t\t\t{}+{}\n\t\t\t\\tfrac1\\gamma\\D(w,y^k)\n\t\t}\n\t\\)%\n\\end{algorithmic}\n\n\\end{algorithm}\n\n\\begin{assumption}\\label{ass:basic}%\n\tThe following hold in problem \\eqref{eq:P}:\n\t\\begin{enumeratass}\n\t\\item\\label{ass:f}%\n\t\t\\(\\func{f}{\\R^n}{\\Rinf}\\) is smooth relative to \\(h\\) (see \\cref{sec:relsmooth}).\n\t\\item\\label{ass:g}%\n\t\t\\(\\func{g}{\\R^n}{\\Rinf}\\) is proper and lsc.\n\t\\item\\label{ass:phi}%\n\t\t\\(\\inf\\cost>-\\infty\\).\n\t\\item\\label{ass:T}%\n\t\tFor any \\(v\\in\\R^n\\) and \\(\\gamma>0\\), \\(\\argmin\\set{\\gamma g+h-\\innprod{v}{\\cdot}}\\subseteq C\\).\n\t\\end{enumeratass}\n\\end{assumption}\n\n\t\t\tThe extended-real line is denoted by \\(\\Rinf\\coloneqq\\R\\cup\\set{\\pm\\infty}\\).\nThe positive and negative part of \\(r\\in\\R\\) are respectively defined as \\([r]_+\\coloneqq\\max\\set{0,r}\\) and \\([r]_-\\coloneqq\\max\\set{0,-r}\\), so that \\(r=[r]_+-[r]_-\\).\n\nThe distance of a point \\(x\\in\\R^n\\) to a nonempty set \\(S\\subseteq\\R^n\\) is given by \\(\\dist(x,S)=\\inf_{z\\in S} \\|z-x\\|\\).\nThe interior, closure, and boundary of \\(S\\) are respectively denoted as \\(\\interior S\\), \\(\\overline S\\), and \\(\\boundary S=\\overline S\\setminus\\interior S\\).\nThe \\DEF{indicator function} of \\(S\\) is \\(\\func{\\indicator_S}{\\R^n}{\\Rinf}\\) defined as \\(\\indicator_S(x)=0\\) if \\(x\\in S\\) and \\(\\infty\\) otherwise.\n\nA function \\(\\func{f}{\\R^n}{\\Rinf}\\) is \\DEF{proper} if \\(f\\not\\equiv\\infty\\) and \\(f>-\\infty\\), in which case its \\DEF{domain} is defined as the set \\(\\dom f\\coloneqq\\set{x\\in\\R^n}[f(x)<\\infty]\\).\nFor \\(\\alpha\\in\\R\\), \\([f\\leq\\alpha]\\coloneqq\\set{x\\in\\R^n}[f(x)\\leq\\alpha]\\) denotes the \\(\\alpha\\)-sublevel set of \\(f\\);\n\t\\([\\alpha\\leq f\\leq\\beta]\\) with \\(\\alpha,\\beta\\in\\R\\) is defined accordingly. We say that \\(f\\) is \\DEF{level bounded} (or \\DEF{coercive}) if \\(\\liminf_{\\|x\\|\\to\\infty}f(x)=\\infty\\), and \\(1\\)-coercive if\n\\(\\lim_{\\|x\\|\\to\\infty}f(x)/\\|x\\|=\\infty\\).\n\n\nGiven \\(x\\in\\dom f\\), \\(\\partial f(x)\\) denotes the \\DEF{Mordukhovich (limiting) subdifferential} of \\(f\\) at \\(x\\), given by\n\\[\n\t\\partial f(x)\n{}\\coloneqq{}\n\t\\set{v\\in\\R^n}[\n\t\t\\exists\\seq{x^k,v^k}~\\text{s.t.}~ x^k\\to x,~f(x^k)\\to f(x),~\n\t\t\\hat\\partial f(x^k)\\ni v^k\\to v\n\t],\n\\]\nand \\(\\hat\\partial f(x)\\) is the set of \\DEF{regular subgradients} of \\(f\\) at \\(x\\), namely vectors \\(v\\in\\R^n\\) such that\n\\(\n\t\\liminf_{\\limsubstack{z&\\to&x\\\\z&\\neq&x}}{\n\t\t\\frac{\n\t\t\tf(z)-f(x)-\\innprod{v}{z-x}\n\t\t}{\n\t\t\t\\|z-x\\|\n\t\t}\n\t}\n{}\\geq{}\n\t0\n\\);\nsee, e.g.,~\\cite{rockafellar2011variational,mordukhovich2018variational}.\n\\(\\C^k(\\mathcal U)\\) is the set of functions \\(\\mathcal U\\to\\R\\) which are \\(k\\) times continuously differentiable.\nWe write \\(\\C^k\\) if \\(\\mathcal U\\) is clear from context.\nThe notation \\(\\ffunc{T}{\\R^n}{\\R^n}\\) indicates a set-valued mapping, whose \\DEF{domain} and \\DEF{graph} are respectively defined as\n\\(\n\t\\dom T=\\set{x\\in\\R^n}[T(x)\\neq\\emptyset]\n\\)\nand\n\\(\n\t\\graph T=\\set{(x,y)\\in\\R^n\\times\\R^n}[y\\in T(x)]\n\\).\n\n\n\t\t\tIn the following definition, as well as throughout the entire paper, we follow the ex\\-tend\\-ed-real convention \\(\\infty-\\infty=\\infty\\) to resolve possible ill definitions of difference of extended-real--valued functions.\nSimilarly, we will adopt the convention that \\(\\nicefrac10=\\infty\\).\n\n\n\\begin{definition}[relative smoothness]\\label{def:relsmooth}%\n\tWe say that an lsc function \\(\\func{f}{\\R^n}{\\Rinf}\\) is smooth relative to \\(h\\) if \\(\\dom f\\supseteq\\dom h\\) and there exists a constant \\(L_{f,h}\\geq0\\) such that \\(L_{f,h}h\\pm f\\) are convex functions  on \\(\\inte\\dom h\\).\n\tWe may alternatively say that \\(f\\) is \\(L_{f,h}\\)-smooth relative to \\(h\\) to make the smoothness modulus \\(L_{f,h}\\) explicit.\n\\end{definition}\n\n\\begin{definition}[relative weak convexity]\\label{def:relweak}%\n\tWe say that an lsc function \\(\\func{f}{\\R^n}{\\Rinf}\\) is weakly convex relative to \\(h\\) if there exists a (possibly negative) constant \\(\\sigma_{f,h}\\in\\R\\) such that \\(f-\\sigma_{f,h}h\\) is a convex function.\n\tWe may alternatively say that \\(f\\) is \\(\\sigma_{f,h}\\)-weakly convex relative to \\(h\\) to make the weak convexity modulus \\(\\sigma_{f,h}\\) explicit.\n\\end{definition}\n\nNote that relative smoothness of \\(f\\) is equivalent to the relative weak convexity of \\(\\pm f\\).\nIndeed, if \\(f\\) is \\(L_{f,h}\\)-smooth relative to \\(h\\), then both \\(f\\) and \\(-f\\) are \\((-L_{f,h})\\)-weakly convex.\nConversely, if \\(f\\) and \\(-f\\) are \\(\\sigma_{f,h}\\)- and \\(\\sigma_{-f,h}\\)-weakly convex relative to \\(h\\), respectively, then \\(f\\) (as well as \\(-f\\)) is \\(L_{f,h}\\)-smooth relative to \\(h\\) with\n\\begin{equation}\\label{eq:Lf}\n\tL_{f,h}=\\max\\set{|\\sigma_{f,h}|,|\\sigma_{-f,h}|}.\n\\end{equation}\nThe relative weak convexity moduli \\(\\sigma_{\\pm f,h}\\) will be henceforth adopted when referring to \\cref{ass:f}.\nIt will be convenient to \\emph{normalize} these quantities into pure numbers\n\\begin{equation}\\label{eq:pf}\n\tp_{\\pm f,h}\n{}\\coloneqq{}\n\t\\tfrac{\\sigma_{\\pm f,h}}{L_{f,h}}\n{}\\in{}\n\t[-1,1].\n\\end{equation}\nTo make all definitions and implications well posed, we will ignore the uninsteresting case in which \\(f\\) is affine, in which case \\(L_{f,h}=0\\) for any \\(h\\).\n\n\n\\begin{equation}\n\t\\h\\coloneqq\\tfrac1\\gamma h-f\n\\quad\\text{and}\\quad\n\t\\f\\coloneqq f-\\tfrac\\beta\\gamma h,\n\\end{equation}\n\n\tWe will also (ab)use the notation \\(\\D_\\psi\\) of the Bregman distance for functions \\(\\func{\\psi}{\\R^n}{\\Rinf}\\) differentiable on \\(C\\) that are not necessarily convex, thereby possibly having \\(\\D_\\psi\\not\\geq0\\).\n\tThis notational abuse is justified by the fact that all algebraic identities of the Bregman distance used in the manuscript (e.g., the three-point identity \\cite[Lem. 3.1]{chen1993convergence}) are valid regardless of whether \\(\\psi\\) is convex or not, and will overall yield a major simplification of the math.\n\n\n\\begin{align}\n\t\\label{eq:M-innprod}\n\t\t\\M(w;x,x^-)\n\t{}={} &\n\t\t\\varphi(w)\n\t\t{}+{}\n\t\t\\D_{\\h}(w,x)\n\t\t{}+{}\n\t\t\\innprod*{w-x}{\\nabla\\f(x)-\\nabla\\f(x^-)}\n\t\\\\\n\t\\label{eq:M-D}\n\t{}={} &\n\t\t\\varphi(w)\n\t\t{}+{}\n\t\t\\D_{\\h-\\f}(w,x)\n\t\t{}+{}\n\t\t\\D_{\\f}(w,x^-)\n\t\t{}-{}\n\t\t\\D_{\\f}(x,x^-),\n\t\\end{align}\n\n\\begin{align}\n\t\tx^{k+1}\n\t{}\\in{} &\n\t\t\\T(x^k,x^{k-1}),\n\t\\shortintertext{%\n\t\twhere \\(\\ffunc{\\T}{C\\times C}{C}\\) defined by\n\t}\n\t\\label{eq:T}\n\t\t\\T(x,x^-)\n\t{}\\coloneqq{} &\n\t\t\\argmin_{w\\in\\R^n}\\M(w;x,x^-)\n\t\\end{align}\n\n\\begin{definition}[\\BiFRB-envelope]\\label{def: envelope}%\n\tThe envelope function associated to \\BiFRB\\ with stepsize \\(\\gamma<\\nicefrac{1}{[\\sigma_{f,h}]_-}\\)  and inertia \\(\\beta\\in\\R\\) is \\(\\func{\\env}{C\\times C}{\\R}\\) defined as\n\t\\begin{equation}\\label{eq:env}\n\t\t\\env(x,x^-)\n\t{}\\coloneqq{}\n\t\t\\inf_{w\\in\\R^n}\\M(w;x,x^-).\n\t\\end{equation}\n\\end{definition}\n\n\tThe convergence analysis in both cases revolves around the identification of a constant \\(c>0\\) determining a lower bounded merit function\n\n\n\\begin{equation}\\label{eq:LL}\n\t\t\\LL=\\env+\\tfrac{c}{2\\gamma}\\D+\\D_\\xi.\n\t\\end{equation}\n\n\tThe difference between the two cases is determined by function \\(\\xi\\) appearing in the last Bregman operator \\(\\D_\\xi\\), having \\(\\xi=\\f\\) in the former case and \\(\\xi=L_{\\f}\\j\\) in the latter, where \\(L_{\\f}\\) is a Lipschitz constant for \\(\\nabla\\f\\) and\n\n\n\\begin{equation}\\label{eq:j}\n\t\t\\j\\coloneqq\\tfrac12\\|{}\\cdot{}\\|^2\n\t\\end{equation}\n\n\tis the squared Euclidean norm.\n\n\nIn what follows, patterning the normalization of \\(\\sigma_{\\pm f,h}\\) into \\(p_{\\pm f,h}\\) detailed in \\cref{sec:relsmooth} we also introduce the scaled stepsize\n\\begin{equation}\n\t\\alpha\\coloneqq\\gamma L_{f,h},\n\\end{equation}\n\n\\begin{assumption}\\label{ass:h}\n\tFunction \\(h\\) has full domain, that is, \\(C=\\R^n\\).\n\\end{assumption}\n\nFor $\\eta\\in(0,\\infty]$, denote by $\\Psi_\\eta$ the class of functions $\\psi:[0,\\eta)\\rightarrow\\R_+$ satisfying the following: (i) $\\psi(t)$ is right-continuous at $t=0$ with $\\psi(0)=0$; (ii) $\\psi$ is strictly increasing on $[0,\\eta)$. \n\n\n\\begin{definition}[{\\cite[Def. 5]{wang2022exact}}]\\label{def: GCKL}%\n\tLet $\\func{f}{\\R^n}{\\Rinf}$ be proper and lsc.\n\tLet $\\bar x\\in\\dom\\partial f$ and $\\mu\\in\\R$, and let $V\\subseteq\\dom\\partial f$ be a nonempty subset.\n\t\\begin{enumerate}\n\t\\item\n\t\tWe say that $f$ has the pointwise generalized concave Kurdyka-\\L ojasiewicz~(KL) property at $\\bar x\\in\\dom\\partial f$, if there exist a neighborhood $U\\ni\\bar x$, $\\eta\\in(0,\\infty]$ and a concave $\\psi\\in\\Psi_\\eta$, such that for all $x\\in U\\cap[0<f-f(\\bar x)<\\eta]$,\n\t\t\\[\n\t\t\t\\psi'_-\\bigl(f(x)-f(\\bar x)\\bigr)\\cdot\\dist\\bigl(0,\\partial f(x)\\bigr)\\geq1,\n\t\t\\]\n\t\twhere $\\psi_-'$ denotes the left derivative.\n\t\tMoreover, $f$ is a generalized concave KL function if it has the generalized concave KL property at every $x\\in\\dom\\partial f$.\n\t\\item\n\t\tSuppose that $f(x)=\\mu$ on $V$.\n\t\tWe say $f$ has the setwise\\footnote{%\n\t\t\tWe shall omit adjectives ``pointwise'' and ``setwise'' whenever there is no ambiguity.%\n\t\t}\n\t\tgeneralized concave KL property on $V$ if there exist $U\\supset V$, $\\eta\\in(0,\\infty]$ and a concave $\\psi\\in\\Psi_\\eta$ such that for every $x\\in U\\cap[0<f-\\mu<\\eta]$,\n\t\t\\[\n\t\t\t\\psi'_-\\bigl(f(x)-\\mu\\bigr)\\cdot\\dist\\bigl(0,\\partial f(x)\\bigr)\\geq1.\n\t\t\\]\n\t\\end{enumerate}\n\\end{definition}\n\nFor a subset $\\Omega\\subseteq\\R^n$, define $(\\forall \\varepsilon>0)$ $\\Omega_\\varepsilon=\\set{x\\in\\R^n}[\\dist(x,\\Omega)<\\varepsilon]$. \n\n\nIn the remainder of the section, we will make use of the norm \\(\\norm*{{}\\cdot{}}\\) on the product space \\(\\R^n\\times\\R^n\\) defined as \\(\\norm*{(x,y)}=\\|x\\|+\\|y\\|\\).\n","units":[{"start":9372,"end":9505,"start_line":296,"end_line":301,"raw_tex":"\\begin{equation*}\n\t\\tag{P}\\label{eq:P}\n\t\\minimize_{x\\in\\R^n}\\varphi(x)\\coloneqq f(x)+g(x)\n\\quad\n\t\\stt x\\in\\closure C,\n\\end{equation*}","sha256_utf8":"cf1da1dc084695e8506f730792dfd28d6ad6f62dabe8b2cf17f78d41384566a3"},{"start":9506,"end":10003,"start_line":302,"end_line":303,"raw_tex":"where \\(C\\) is a nonempty open and convex set with closure \\(\\closure C\\),  \\(\\func{f}{\\R^n}{\\Rinf\\coloneqq\\R\\cup\\set{\\pm\\infty}}\\) is differentiable on $\\inte\\dom f\\neq\\emptyset$, and \\(\\func{g}{\\R^n}{\\Rinf}\\) is proper and lower semicontinuous (lsc).\nFor notational brevity, we define \\(\\cost\\coloneqq\\varphi+\\indicator_{\\closure C}\\) with \\(\\indicator_X\\) denoting the indicator function of set \\(X\\subseteq\\R^n\\), namely such that \\(\\indicator_X(x)=0\\) if \\(x\\in X\\) and \\(\\infty\\) otherwise.\n","sha256_utf8":"24dd8170f84239d7bcc4f5b1a9381052e553d6bc2673ac0dcfe31be9f785bc3d"},{"start":15224,"end":15924,"start_line":348,"end_line":360,"raw_tex":"\t\t\tThroughout, we fix a Legendre kernel \\(\\func{h}{\\R^n}{\\Rinf}\\) with \\(\\dom\\nabla h=\\interior\\dom h=C\\),\nnamely, a proper, lsc, and strictly convex function  that is 1-coercive and \\DEF{essentially smooth}, \\ie such that \\(\\|\\nabla h(x_k)\\|\\to\\infty\\) for every sequence \\(\\seq{x_k}\\subset C\\) converging to a boundary point of \\(C\\).\nWe will consider the following iterative scheme for addressing problem \\eqref{eq:P},\nwhere \\(\\func{\\D}{\\R^n\\times\\R^n}{\\Rinf}\\) denotes the \\DEF{Bregman distance} induced by \\(h\\), defined as\n\\begin{equation}\\label{eq:D}\n\t\\D(x,y)\n{}\\coloneqq{}\n\t\\begin{ifcases}\n\t\th(x)-h(y)-\\innprod{\\nabla h(y)}{x-y} & y\\in C,\n\t\\\\\n\t\t\\infty\\otherwise.\n\t\\end{ifcases}\n\\end{equation}","sha256_utf8":"ee78a8aa51f82edf14fccd5c866a8b9cd89c0339f6fd916943ed9b05a585ed17"},{"start":15926,"end":16758,"start_line":362,"end_line":400,"raw_tex":"\\begin{algorithm}\n\t\\caption{Mirror inertial forward-reflected-backward (\\protect\\BiFRB)}%\n\t\\label{alg:BiFRB}%\n\t\\begin{algorithmic}[1]%\n\\setlength{\\itemsep}{1ex}%\n\\item[%\n\tChoose \\(x^{-1},x^0\\in C\\),~\n\tinertial parameter \\(\\beta\\in\\R\\),~ and\n\tstepsize \\(\\gamma>0\\)%\n]%\n\\item[%\n\tIterate for \\(k=0,1,\\ldots\\) until a termination criterion is met (cf. \\cref{thm:termination})\n]%\n\\State\\label{state:BiFRB:y}%\n\tSet \\(y^k\\) such that\n\t\\(\n\t\t\\nabla h(y^k)\n\t{}={}\n\t\t\\nabla h(x^k)-\\gamma\\bigl(\\nabla f(x^k)-\\nabla f(x^{k-1})\\bigr)\n\t\\)%\n\\State\\label{state:BiFRB:x}%\n\tChoose\n\t\\(\n\t\\ifsiam\n\t\t\\mathtight\n\t\\fi\n\t\tx^{k+1}\n\t{}\\in{}\n\t\t\\argmin\\limits_{w\\in\\R^n}\\set{\n\t\t\tg(w)\n\t\t\t{}+{}\n\t\t\t\\innprod*{w}{\\nabla f(x^k)-\\tfrac\\beta\\gamma\\bigl(\\nabla h(x^k)-\\nabla h(x^{k-1})\\bigr)}\n\t\t\t{}+{}\n\t\t\t\\tfrac1\\gamma\\D(w,y^k)\n\t\t}\n\t\\)%\n\\end{algorithmic}\n\n\\end{algorithm}","sha256_utf8":"037f93ccc5846059cd02f0ca2948b78d4187e2f58746ac4ade631c7bf411fe81"},{"start":16904,"end":17384,"start_line":405,"end_line":417,"raw_tex":"\\begin{assumption}\\label{ass:basic}%\n\tThe following hold in problem \\eqref{eq:P}:\n\t\\begin{enumeratass}\n\t\\item\\label{ass:f}%\n\t\t\\(\\func{f}{\\R^n}{\\Rinf}\\) is smooth relative to \\(h\\) (see \\cref{sec:relsmooth}).\n\t\\item\\label{ass:g}%\n\t\t\\(\\func{g}{\\R^n}{\\Rinf}\\) is proper and lsc.\n\t\\item\\label{ass:phi}%\n\t\t\\(\\inf\\cost>-\\infty\\).\n\t\\item\\label{ass:T}%\n\t\tFor any \\(v\\in\\R^n\\) and \\(\\gamma>0\\), \\(\\argmin\\set{\\gamma g+h-\\innprod{v}{\\cdot}}\\subseteq C\\).\n\t\\end{enumeratass}\n\\end{assumption}","sha256_utf8":"25bdcab7c6c4e8833a336d4ffdf7ba4ca7569bddcdb71ed01268c75147b9702d"},{"start":19007,"end":20268,"start_line":439,"end_line":449,"raw_tex":"\t\t\tThe extended-real line is denoted by \\(\\Rinf\\coloneqq\\R\\cup\\set{\\pm\\infty}\\).\nThe positive and negative part of \\(r\\in\\R\\) are respectively defined as \\([r]_+\\coloneqq\\max\\set{0,r}\\) and \\([r]_-\\coloneqq\\max\\set{0,-r}\\), so that \\(r=[r]_+-[r]_-\\).\n\nThe distance of a point \\(x\\in\\R^n\\) to a nonempty set \\(S\\subseteq\\R^n\\) is given by \\(\\dist(x,S)=\\inf_{z\\in S} \\|z-x\\|\\).\nThe interior, closure, and boundary of \\(S\\) are respectively denoted as \\(\\interior S\\), \\(\\overline S\\), and \\(\\boundary S=\\overline S\\setminus\\interior S\\).\nThe \\DEF{indicator function} of \\(S\\) is \\(\\func{\\indicator_S}{\\R^n}{\\Rinf}\\) defined as \\(\\indicator_S(x)=0\\) if \\(x\\in S\\) and \\(\\infty\\) otherwise.\n\nA function \\(\\func{f}{\\R^n}{\\Rinf}\\) is \\DEF{proper} if \\(f\\not\\equiv\\infty\\) and \\(f>-\\infty\\), in which case its \\DEF{domain} is defined as the set \\(\\dom f\\coloneqq\\set{x\\in\\R^n}[f(x)<\\infty]\\).\nFor \\(\\alpha\\in\\R\\), \\([f\\leq\\alpha]\\coloneqq\\set{x\\in\\R^n}[f(x)\\leq\\alpha]\\) denotes the \\(\\alpha\\)-sublevel set of \\(f\\);\n\t\\([\\alpha\\leq f\\leq\\beta]\\) with \\(\\alpha,\\beta\\in\\R\\) is defined accordingly. We say that \\(f\\) is \\DEF{level bounded} (or \\DEF{coercive}) if \\(\\liminf_{\\|x\\|\\to\\infty}f(x)=\\infty\\), and \\(1\\)-coercive if\n\\(\\lim_{\\|x\\|\\to\\infty}f(x)/\\|x\\|=\\infty\\).\n","sha256_utf8":"4031c4e000fb8c005fb3e52d3dfe8df1e874a683d3467bf9cca7289a49c0f2ad"},{"start":20731,"end":21751,"start_line":454,"end_line":485,"raw_tex":"Given \\(x\\in\\dom f\\), \\(\\partial f(x)\\) denotes the \\DEF{Mordukhovich (limiting) subdifferential} of \\(f\\) at \\(x\\), given by\n\\[\n\t\\partial f(x)\n{}\\coloneqq{}\n\t\\set{v\\in\\R^n}[\n\t\t\\exists\\seq{x^k,v^k}~\\text{s.t.}~ x^k\\to x,~f(x^k)\\to f(x),~\n\t\t\\hat\\partial f(x^k)\\ni v^k\\to v\n\t],\n\\]\nand \\(\\hat\\partial f(x)\\) is the set of \\DEF{regular subgradients} of \\(f\\) at \\(x\\), namely vectors \\(v\\in\\R^n\\) such that\n\\(\n\t\\liminf_{\\limsubstack{z&\\to&x\\\\z&\\neq&x}}{\n\t\t\\frac{\n\t\t\tf(z)-f(x)-\\innprod{v}{z-x}\n\t\t}{\n\t\t\t\\|z-x\\|\n\t\t}\n\t}\n{}\\geq{}\n\t0\n\\);\nsee, e.g.,~\\cite{rockafellar2011variational,mordukhovich2018variational}.\n\\(\\C^k(\\mathcal U)\\) is the set of functions \\(\\mathcal U\\to\\R\\) which are \\(k\\) times continuously differentiable.\nWe write \\(\\C^k\\) if \\(\\mathcal U\\) is clear from context.\nThe notation \\(\\ffunc{T}{\\R^n}{\\R^n}\\) indicates a set-valued mapping, whose \\DEF{domain} and \\DEF{graph} are respectively defined as\n\\(\n\t\\dom T=\\set{x\\in\\R^n}[T(x)\\neq\\emptyset]\n\\)\nand\n\\(\n\t\\graph T=\\set{(x,y)\\in\\R^n\\times\\R^n}[y\\in T(x)]\n\\).\n","sha256_utf8":"65b0b231e8816c5c70d00c588b98c73d128683bbd0ef01122a501a1264f8eef5"},{"start":21830,"end":22125,"start_line":489,"end_line":490,"raw_tex":"\t\t\tIn the following definition, as well as throughout the entire paper, we follow the ex\\-tend\\-ed-real convention \\(\\infty-\\infty=\\infty\\) to resolve possible ill definitions of difference of extended-real--valued functions.\nSimilarly, we will adopt the convention that \\(\\nicefrac10=\\infty\\).\n","sha256_utf8":"2d073ee11ff207ac4a1b5b821efad13d5596a66e129c441150571c07863b006f"},{"start":22126,"end":22561,"start_line":492,"end_line":495,"raw_tex":"\\begin{definition}[relative smoothness]\\label{def:relsmooth}%\n\tWe say that an lsc function \\(\\func{f}{\\R^n}{\\Rinf}\\) is smooth relative to \\(h\\) if \\(\\dom f\\supseteq\\dom h\\) and there exists a constant \\(L_{f,h}\\geq0\\) such that \\(L_{f,h}h\\pm f\\) are convex functions  on \\(\\inte\\dom h\\).\n\tWe may alternatively say that \\(f\\) is \\(L_{f,h}\\)-smooth relative to \\(h\\) to make the smoothness modulus \\(L_{f,h}\\) explicit.\n\\end{definition}","sha256_utf8":"06a675bc422b7340c80f233bc789dc9a897a5a6ebc254ea45aa06052efcc46b0"},{"start":23818,"end":24260,"start_line":508,"end_line":511,"raw_tex":"\\begin{definition}[relative weak convexity]\\label{def:relweak}%\n\tWe say that an lsc function \\(\\func{f}{\\R^n}{\\Rinf}\\) is weakly convex relative to \\(h\\) if there exists a (possibly negative) constant \\(\\sigma_{f,h}\\in\\R\\) such that \\(f-\\sigma_{f,h}h\\) is a convex function.\n\tWe may alternatively say that \\(f\\) is \\(\\sigma_{f,h}\\)-weakly convex relative to \\(h\\) to make the weak convexity modulus \\(\\sigma_{f,h}\\) explicit.\n\\end{definition}","sha256_utf8":"6fbf38b6f3dd85872864aef85bb3e65fd2e3e99ba39393ae1b6e26d1c62d21f5"},{"start":24263,"end":25252,"start_line":513,"end_line":528,"raw_tex":"Note that relative smoothness of \\(f\\) is equivalent to the relative weak convexity of \\(\\pm f\\).\nIndeed, if \\(f\\) is \\(L_{f,h}\\)-smooth relative to \\(h\\), then both \\(f\\) and \\(-f\\) are \\((-L_{f,h})\\)-weakly convex.\nConversely, if \\(f\\) and \\(-f\\) are \\(\\sigma_{f,h}\\)- and \\(\\sigma_{-f,h}\\)-weakly convex relative to \\(h\\), respectively, then \\(f\\) (as well as \\(-f\\)) is \\(L_{f,h}\\)-smooth relative to \\(h\\) with\n\\begin{equation}\\label{eq:Lf}\n\tL_{f,h}=\\max\\set{|\\sigma_{f,h}|,|\\sigma_{-f,h}|}.\n\\end{equation}\nThe relative weak convexity moduli \\(\\sigma_{\\pm f,h}\\) will be henceforth adopted when referring to \\cref{ass:f}.\nIt will be convenient to \\emph{normalize} these quantities into pure numbers\n\\begin{equation}\\label{eq:pf}\n\tp_{\\pm f,h}\n{}\\coloneqq{}\n\t\\tfrac{\\sigma_{\\pm f,h}}{L_{f,h}}\n{}\\in{}\n\t[-1,1].\n\\end{equation}\nTo make all definitions and implications well posed, we will ignore the uninsteresting case in which \\(f\\) is affine, in which case \\(L_{f,h}=0\\) for any \\(h\\).\n","sha256_utf8":"4c706813a9aadf596bad592e9a84099fe584af7077dc862272d4880740a3c4f6"},{"start":32640,"end":32758,"start_line":668,"end_line":672,"raw_tex":"\\begin{equation}\n\t\\h\\coloneqq\\tfrac1\\gamma h-f\n\\quad\\text{and}\\quad\n\t\\f\\coloneqq f-\\tfrac\\beta\\gamma h,\n\\end{equation}","sha256_utf8":"560dc6c67e69e58cb31570d1f9dbac1489079262e2677d20b597ed629a0baecc"},{"start":33090,"end":33620,"start_line":681,"end_line":682,"raw_tex":"\tWe will also (ab)use the notation \\(\\D_\\psi\\) of the Bregman distance for functions \\(\\func{\\psi}{\\R^n}{\\Rinf}\\) differentiable on \\(C\\) that are not necessarily convex, thereby possibly having \\(\\D_\\psi\\not\\geq0\\).\n\tThis notational abuse is justified by the fact that all algebraic identities of the Bregman distance used in the manuscript (e.g., the three-point identity \\cite[Lem. 3.1]{chen1993convergence}) are valid regardless of whether \\(\\psi\\) is convex or not, and will overall yield a major simplification of the math.\n","sha256_utf8":"a9937ff51bc945216d380da7f1ab7dcce8625f90bed755a94135bedfd32479e8"},{"start":33931,"end":34209,"start_line":690,"end_line":709,"raw_tex":"\\begin{align}\n\t\\label{eq:M-innprod}\n\t\t\\M(w;x,x^-)\n\t{}={} &\n\t\t\\varphi(w)\n\t\t{}+{}\n\t\t\\D_{\\h}(w,x)\n\t\t{}+{}\n\t\t\\innprod*{w-x}{\\nabla\\f(x)-\\nabla\\f(x^-)}\n\t\\\\\n\t\\label{eq:M-D}\n\t{}={} &\n\t\t\\varphi(w)\n\t\t{}+{}\n\t\t\\D_{\\h-\\f}(w,x)\n\t\t{}+{}\n\t\t\\D_{\\f}(w,x^-)\n\t\t{}-{}\n\t\t\\D_{\\f}(x,x^-),\n\t\\end{align}","sha256_utf8":"c7fb4d52cbf09fea913bbd7da0d3c4bf9bf985c80795338aa1b6e974fc948e37"},{"start":34338,"end":34549,"start_line":713,"end_line":724,"raw_tex":"\\begin{align}\n\t\tx^{k+1}\n\t{}\\in{} &\n\t\t\\T(x^k,x^{k-1}),\n\t\\shortintertext{%\n\t\twhere \\(\\ffunc{\\T}{C\\times C}{C}\\) defined by\n\t}\n\t\\label{eq:T}\n\t\t\\T(x,x^-)\n\t{}\\coloneqq{} &\n\t\t\\argmin_{w\\in\\R^n}\\M(w;x,x^-)\n\t\\end{align}","sha256_utf8":"6b2287cadd92f4bfc4a0682f6497d398c35d5a71be0e867352d8b8533d565515"},{"start":40340,"end":40695,"start_line":824,"end_line":831,"raw_tex":"\\begin{definition}[\\BiFRB-envelope]\\label{def: envelope}%\n\tThe envelope function associated to \\BiFRB\\ with stepsize \\(\\gamma<\\nicefrac{1}{[\\sigma_{f,h}]_-}\\)  and inertia \\(\\beta\\in\\R\\) is \\(\\func{\\env}{C\\times C}{\\R}\\) defined as\n\t\\begin{equation}\\label{eq:env}\n\t\t\\env(x,x^-)\n\t{}\\coloneqq{}\n\t\t\\inf_{w\\in\\R^n}\\M(w;x,x^-).\n\t\\end{equation}\n\\end{definition}","sha256_utf8":"8e9e8b4837c204e394c15ab9eb4d1f8f1c999a8710745d20b864fae1bd765a3e"},{"start":45429,"end":45569,"start_line":912,"end_line":912,"raw_tex":"\tThe convergence analysis in both cases revolves around the identification of a constant \\(c>0\\) determining a lower bounded merit function\n","sha256_utf8":"c26de6798824a4ed5e64205a031ccd45b8534f4fd735e2b8c13e806c0ac57831"},{"start":45570,"end":45655,"start_line":913,"end_line":915,"raw_tex":"\\begin{equation}\\label{eq:LL}\n\t\t\\LL=\\env+\\tfrac{c}{2\\gamma}\\D+\\D_\\xi.\n\t\\end{equation}","sha256_utf8":"355e59227a9de410966d77cc18aff6e678e8f78f74c9cfd5fe8fc0402fb361f4"},{"start":45711,"end":45969,"start_line":917,"end_line":917,"raw_tex":"\tThe difference between the two cases is determined by function \\(\\xi\\) appearing in the last Bregman operator \\(\\D_\\xi\\), having \\(\\xi=\\f\\) in the former case and \\(\\xi=L_{\\f}\\j\\) in the latter, where \\(L_{\\f}\\) is a Lipschitz constant for \\(\\nabla\\f\\) and\n","sha256_utf8":"2e7aa940a0c939b83ce9b6ab9a7a1271a0de67a73995db5eaa49f9627ca5fd8f"},{"start":45970,"end":46051,"start_line":918,"end_line":920,"raw_tex":"\\begin{equation}\\label{eq:j}\n\t\t\\j\\coloneqq\\tfrac12\\|{}\\cdot{}\\|^2\n\t\\end{equation}","sha256_utf8":"6ab166c891e878618dbeb333787b16df273a6005b87bafa59bb98414f065a497"},{"start":46052,"end":46084,"start_line":921,"end_line":921,"raw_tex":"\tis the squared Euclidean norm.\n","sha256_utf8":"4114f82700c4c5ce8af531516adcc395067812a35f35c4a1315433f26d72872e"},{"start":46287,"end":46512,"start_line":925,"end_line":928,"raw_tex":"In what follows, patterning the normalization of \\(\\sigma_{\\pm f,h}\\) into \\(p_{\\pm f,h}\\) detailed in \\cref{sec:relsmooth} we also introduce the scaled stepsize\n\\begin{equation}\n\t\\alpha\\coloneqq\\gamma L_{f,h},\n\\end{equation}","sha256_utf8":"1cb00383c6298e5fb8ea86f0b9e99eb01a9c5b238053f296d745a94f70d01147"},{"start":62220,"end":62322,"start_line":1371,"end_line":1373,"raw_tex":"\\begin{assumption}\\label{ass:h}\n\tFunction \\(h\\) has full domain, that is, \\(C=\\R^n\\).\n\\end{assumption}","sha256_utf8":"01a4bc484033a90b325eca074cc1b07c97ea0dea52295120f47538f2b3a13d33"},{"start":69024,"end":69263,"start_line":1483,"end_line":1483,"raw_tex":"For $\\eta\\in(0,\\infty]$, denote by $\\Psi_\\eta$ the class of functions $\\psi:[0,\\eta)\\rightarrow\\R_+$ satisfying the following: (i) $\\psi(t)$ is right-continuous at $t=0$ with $\\psi(0)=0$; (ii) $\\psi$ is strictly increasing on $[0,\\eta)$. \n","sha256_utf8":"d67776e8cc2e428fb33ff037a310217614ff93dc1b06d02e8d1b120abdaa8279"},{"start":69264,"end":70514,"start_line":1485,"end_line":1506,"raw_tex":"\\begin{definition}[{\\cite[Def. 5]{wang2022exact}}]\\label{def: GCKL}%\n\tLet $\\func{f}{\\R^n}{\\Rinf}$ be proper and lsc.\n\tLet $\\bar x\\in\\dom\\partial f$ and $\\mu\\in\\R$, and let $V\\subseteq\\dom\\partial f$ be a nonempty subset.\n\t\\begin{enumerate}\n\t\\item\n\t\tWe say that $f$ has the pointwise generalized concave Kurdyka-\\L ojasiewicz~(KL) property at $\\bar x\\in\\dom\\partial f$, if there exist a neighborhood $U\\ni\\bar x$, $\\eta\\in(0,\\infty]$ and a concave $\\psi\\in\\Psi_\\eta$, such that for all $x\\in U\\cap[0<f-f(\\bar x)<\\eta]$,\n\t\t\\[\n\t\t\t\\psi'_-\\bigl(f(x)-f(\\bar x)\\bigr)\\cdot\\dist\\bigl(0,\\partial f(x)\\bigr)\\geq1,\n\t\t\\]\n\t\twhere $\\psi_-'$ denotes the left derivative.\n\t\tMoreover, $f$ is a generalized concave KL function if it has the generalized concave KL property at every $x\\in\\dom\\partial f$.\n\t\\item\n\t\tSuppose that $f(x)=\\mu$ on $V$.\n\t\tWe say $f$ has the setwise\\footnote{%\n\t\t\tWe shall omit adjectives ``pointwise'' and ``setwise'' whenever there is no ambiguity.%\n\t\t}\n\t\tgeneralized concave KL property on $V$ if there exist $U\\supset V$, $\\eta\\in(0,\\infty]$ and a concave $\\psi\\in\\Psi_\\eta$ such that for every $x\\in U\\cap[0<f-\\mu<\\eta]$,\n\t\t\\[\n\t\t\t\\psi'_-\\bigl(f(x)-\\mu\\bigr)\\cdot\\dist\\bigl(0,\\partial f(x)\\bigr)\\geq1.\n\t\t\\]\n\t\\end{enumerate}\n\\end{definition}","sha256_utf8":"ea727b979f2b10106e4251060a4479efb8f832a67881de06c284b3b5a1f1fa9f"},{"start":70516,"end":70652,"start_line":1508,"end_line":1508,"raw_tex":"For a subset $\\Omega\\subseteq\\R^n$, define $(\\forall \\varepsilon>0)$ $\\Omega_\\varepsilon=\\set{x\\in\\R^n}[\\dist(x,\\Omega)<\\varepsilon]$. \n","sha256_utf8":"baa71d6c2911e5a25c7ebc1705e13a22d108a59d00a8c1dd07f1e36ce3710298"},{"start":72243,"end":72409,"start_line":1542,"end_line":1542,"raw_tex":"In the remainder of the section, we will make use of the norm \\(\\norm*{{}\\cdot{}}\\) on the product space \\(\\R^n\\times\\R^n\\) defined as \\(\\norm*{(x,y)}=\\|x\\|+\\|y\\|\\).\n","sha256_utf8":"4acc8f87f587935972aba6683d715d7f44644c5caee9506a09131fe47d57ae3e"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}[sequential convergence of \\BiFRB]\\label{thm:sequential convergence}%\n\tSuppose that \\cref{ass:h} holds, and let $\\seq{x^k}$ be a sequence generated by \\BiFRB* in the setting of \\cref{thm:SD}.\n\tDefine $(\\forall k\\in\\N)$~$z^k=(x^{k+1},x^k,x^{k-1})$ and let $\\omega(z^0)$ be the set of limit points of \\((z_k)_{k\\in\\N}\\).\n\tDefine\n\t\\[\n\t\t(\\forall \\omega,x,x^-\\in\\R^n )~\\F(\\omega,x,x^-)=\\M(\\omega;x,x^-)+\\tfrac{c}{2\\gamma}\\D(x,x^-)+D_{\\xi}(x,x^-).\n\t\\]\n\tAssume in addition the following:\n\t\\begin{enumeratass}\n\t\\item\\label{bounded}%\n\t\t\\(\\varphi\\) is level bounded.\n\t\\item\\label{c2 and pd}%\n\t\t$f,h$ are twice continuously differentiable and $\\nabla^2h$ is positive definite.\n\t\\item\\label{GCKL}%\n\t\t$(\\exists r>0)~(\\exists\\eta\\in[0,\\infty))$ $\\F$ satisfies the generalized concave KL property on $\\Omega=\\omega(z^0)$ with respect to $U=\\Omega_r$ and $\\eta$.\n\t\\end{enumeratass}\n\tThen \\(\\sum_{k=0}^\\infty\\norm{x^{k+1}-x^k}<\\infty\\) and there exists $x^\\star$ with $0\\in\\hat\\partial\\varphi(x^{\\star})$ such that \\(x^k\\to x^\\star\\) as \\(k\\to\\infty\\).\n\tTo be specific, there exists $k_0\\in\\N$ such that for all $l\\geq k_0+1$\n\t\\begin{equation}\\label{finite length}\n\t\t\\sum_{k=0}^\\infty\\norm{x^{k+1}-x^k}\n\t{}\\leq{}\n\t\t\\sum_{k=0}^l\\norm{x^{k+1}-x^k}\n\t\t{}+{}\n\t\t\\tfrac{4\\gamma M}{c\\sigma}\n\t\t\\tilde\\psi\\left(\\LL(x^{l},x^{l-1})-\\varphi^\\star\\right),\n\t\\end{equation}\n\twhere $M>0$ is some constant, $\\sigma>0$ is the strong convexity modulus of $h$ on $\\mathcal{B}$, the closed ball in which $\\seq{x^k}$ lies,\n\tand $\\tilde{\\psi}$ is the exact modulus of the generalized concave KL property associated with $\\F$ produced by \\cref{uniformize exact modulus}.\n\\end{theorem}","units":[{"start":72410,"end":74067,"start_line":1544,"end_line":1572,"raw_tex":"\\begin{theorem}[sequential convergence of \\BiFRB]\\label{thm:sequential convergence}%\n\tSuppose that \\cref{ass:h} holds, and let $\\seq{x^k}$ be a sequence generated by \\BiFRB* in the setting of \\cref{thm:SD}.\n\tDefine $(\\forall k\\in\\N)$~$z^k=(x^{k+1},x^k,x^{k-1})$ and let $\\omega(z^0)$ be the set of limit points of \\((z_k)_{k\\in\\N}\\).\n\tDefine\n\t\\[\n\t\t(\\forall \\omega,x,x^-\\in\\R^n )~\\F(\\omega,x,x^-)=\\M(\\omega;x,x^-)+\\tfrac{c}{2\\gamma}\\D(x,x^-)+D_{\\xi}(x,x^-).\n\t\\]\n\tAssume in addition the following:\n\t\\begin{enumeratass}\n\t\\item\\label{bounded}%\n\t\t\\(\\varphi\\) is level bounded.\n\t\\item\\label{c2 and pd}%\n\t\t$f,h$ are twice continuously differentiable and $\\nabla^2h$ is positive definite.\n\t\\item\\label{GCKL}%\n\t\t$(\\exists r>0)~(\\exists\\eta\\in[0,\\infty))$ $\\F$ satisfies the generalized concave KL property on $\\Omega=\\omega(z^0)$ with respect to $U=\\Omega_r$ and $\\eta$.\n\t\\end{enumeratass}\n\tThen \\(\\sum_{k=0}^\\infty\\norm{x^{k+1}-x^k}<\\infty\\) and there exists $x^\\star$ with $0\\in\\hat\\partial\\varphi(x^{\\star})$ such that \\(x^k\\to x^\\star\\) as \\(k\\to\\infty\\).\n\tTo be specific, there exists $k_0\\in\\N$ such that for all $l\\geq k_0+1$\n\t\\begin{equation}\\label{finite length}\n\t\t\\sum_{k=0}^\\infty\\norm{x^{k+1}-x^k}\n\t{}\\leq{}\n\t\t\\sum_{k=0}^l\\norm{x^{k+1}-x^k}\n\t\t{}+{}\n\t\t\\tfrac{4\\gamma M}{c\\sigma}\n\t\t\\tilde\\psi\\left(\\LL(x^{l},x^{l-1})-\\varphi^\\star\\right),\n\t\\end{equation}\n\twhere $M>0$ is some constant, $\\sigma>0$ is the strong convexity modulus of $h$ on $\\mathcal{B}$, the closed ball in which $\\seq{x^k}$ lies,\n\tand $\\tilde{\\psi}$ is the exact modulus of the generalized concave KL property associated with $\\F$ produced by \\cref{uniformize exact modulus}.\n\\end{theorem}","sha256_utf8":"696f875361a6e897347d7125b770e4152035b32bb003248cb980f4cfab585d4b"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}[relative prox-boundedness]\\label{thm:proxbounded}%\n\tSuppose that \\cref{ass:basic} holds.\n\tThen, the set \\(\\argmin\\set{\\gamma g+h+\\innprod{v}{\\cdot}}\\) as in \\cref{ass:T} is nonempty for any \\(v\\in\\R^n\\) and \\(0<\\gamma<\\nicefrac{1}{[\\sigma_{-f,h}]_-}\\).\n\tIn other words, \\(g\\) is prox bounded relative to \\(h\\) with threshold \\(\\gamma_{g,h}\\geq\\nicefrac{1}{[\\sigma_{-f,h}]_-}\\) \\cite[Def. 2.3]{kan2012moreau}.\n\\end{lemma}\n\n\\begin{proof}\n\tRecall that a proper convex function admits affine minorant; see, e.g,~\\cite[Cor. 16.18]{bauschke2017convex}.\n\tIt then follows that \\(\\gamma g+h\\) is 1-coercive by observing that\n\t\\[\n\t\t\\gamma g+h\n\t{}={}\n\t\t\\gamma\\cost-\\gamma f+h\n\t{}={}\n\t\t\\gamma\\underbracket*{\n\t\t\t\\vphantom{\\left([\\sigma_{-f,h}]_-\\right)}\n\t\t\t\\cost\n\t\t}_{\\mathclap{\\geq\\inf\\cost}}\n\t\t{}+{}\n\t\t\\gamma\\underbracket*{\n\t\t\t\\left(-f+[\\sigma_{-f,h}]_-h\\right)\n\t\t}_{\\text{convex}}\n\t\t{}+{}\n\t\t\\underbracket*{\n\t\t\t\\left(1-\\gamma[\\sigma_{-f,h}]_-\\right)\n\t\t}_{>0}\n\t\t\\overbracket*{\n\t\t\t\\,h\\,\n\t\t}^{\\text{\\clap{1-coercive}}}\n\t\\]\n\ton \\(\\closure C\\) and \\(\\infty\\) on \\(\\R^n\\setminus\\closure C\\).\n\\end{proof}\n\n\\begin{lemma}[{{\\cite[Thm. 4.1]{ahookhosh2021bregman}}}]\\label{thm:Legendre}%\n\tSuppose that \\cref{ass:f} holds.\n\tThen, for every \\(\\gamma<\\nicefrac{1}{[\\sigma_{-f,h}]_-}\\) the function \\(\\h\\) is a Legendre kernel with \\(\\dom\\h=\\dom h\\).\n\\end{lemma}\n\n\\begin{lemma}[basic properties of the model \\(\\M\\) and the operator \\(\\T\\)]\\label{thm:basic}%\n\tSuppose that \\cref{ass:basic} holds, and let \\(\\gamma<\\nicefrac{1}{[\\sigma_{-f,h}]_-}\\) and \\(\\beta\\in\\R\\) be fixed.\n\tThe following hold:\n\t\\begin{enumerate}\n\t\\item \\label{thm:M:eq}%\n\t\t\\(\n\t\t\t\\M(x;x,x^-)\n\t\t{}={}\n\t\t\t\\cost(x)\n\t\t\\)\n\t\tfor all \\(x,x^-\\in C\\).\n\t\\item \\label{thm:M:lb}%\n\t\t\\(\\M(w;x,x^-)\\) is level bounded in \\(w\\) locally uniformly in \\((x,x^-)\\) on \\(C\\times C\\).\\footnote{%\n\t\t\tNamely, \\(\\bigcup_{(x,x^-)\\in\\mathcal V}\\set{w\\in\\R^n}[\\M(w;x,x^-)\\leq\\alpha]\\) is bounded for any \\(\\mathcal V\\subset C\\times C\\) compact and \\(\\alpha\\in\\R\\).%\n\t\t}\n\t\\item \\label{thm:T:osc}%\n\t\t\\(\\T\\) is locally bounded and osc,\\footnote{%\n\t\t\tBeing \\(\\T\\) defined on \\(C\\times C\\), osc and local boundedness are meant relative to \\(C\\times C\\).\n\t\t\tNamely, \\(\\graph\\T\\) is closed relative to \\(C\\times C\\times\\R^n\\), and \\(\\bigcup_{(x,x^-)\\in\\mathcal V}\\T(x,x^-)\\) is bounded for any \\(\\mathcal V\\subset C\\times C\\) compact.%\n\t\t}\n\t\tand \\(T(x,x^-)\\) is a nonempty and compact subset of \\(C\\) for any \\(x,x^-\\in C\\).\n\t\\item \\label{thm:T:partial}%\n\t\t\\(\\nabla\\h(x)-\\nabla\\h(\\bar x)-\\nabla\\f(x)+\\nabla\\f(x^-)\\in\\hat\\partial\\varphi(\\bar x)\\) for any \\(x,x^-\\in C\\) and \\(\\bar x\\in\\T(x,x^-)\\).\n\t\\item \\label{thm:T:fix}%\n\t\tIf \\(x\\in\\T(x,x)\\), then \\(0\\in\\hat\\partial\\varphi(x)\\) and \\(\\def\\stepsize{\\gamma'}\\T(x,x)=\\set{x}\\) for every \\(\\gamma'\\in(0,\\gamma)\\).\n\t\\end{enumerate}\n\t\\begin{proof}\n\t\tWe start by observing that \\cref{thm:proxbounded} ensures that the set \\(\\T(x,x^-)\\) is non\\-empty for any \\(x,x^-\\in C\\); this follows by considering the expression \\eqref{eq:M-innprod} of the model, by observing that, for any \\(x\\in C\\),\n\t\t\\(\n\t\t\t\\varphi+\\D_{\\h}({}\\cdot{},x)\n\t\t{}={}\n\t\t\tg+\\tfrac1\\gamma h-\\h(x)-\\innprod*{\\nabla\\h(x)}{{}\\cdot{}-x}\n\t\t\\).\n\t\tFor the same reason, it then follows from \\cref{ass:T} that \\(T(x,x^-)\\subset C\\).\n\t\t\\begin{proofitemize}\n\t\t\\item \\ref{thm:M:eq}~\n\t\t\tApparent, by considering \\(w=x\\) in \\eqref{eq:M-D}.\n\t\t\\item \\ref{thm:M:lb} \\& \\ref{thm:T:osc}~\n\t\t\tThe first assertion owes to the fact that \\(\\h\\) is 1-coercive by \\cref{thm:Legendre} and that both \\(\\h\\) and \\(\\nabla\\f\\) are continuous on \\(C\\), so that for any compact \\(\\mathcal V\\subset\\R^n\\times\\R^n\\) one has that\n\t\t\t\\(\n\t\t\t\t\\lim_{\\|w\\|\\to\\infty}\\inf_{(x,x^-)\\in\\mathcal V}\\M(w;x,x^-)=\\infty\n\t\t\t\\),\n\t\t\tas is apparent from \\eqref{eq:M-innprod}.\n\t\t\tIn turn, the second assertion follows from \\cite[Thm. 1.17]{rockafellar2011variational}.\n\t\t\\item \\ref{thm:T:partial}~\n\t\t\tFollows from the optimality conditions of \\(\\bar x\\in\\argmin\\M({}\\cdot{};x,x^-)\\), for \\(\\bar x\\in C\\) by assumption and thus the calculus rule of \\cite[Ex. 8.8(c)]{rockafellar2011variational} applies (having \\(\\h\\) smooth around \\(\\bar x\\in C\\)).\n\t\t\\item \\ref{thm:T:fix}~\n\t\t\tThat \\(0\\in\\hat\\partial\\varphi(x)\\) follows from assertion \\ref{thm:T:partial}, and the other claim from \\cite[Lem. 3.6]{ahookhosh2021bregman} by observing that \\(\\T(x,x)=\\argmin\\set{\\varphi+\\D_{\\h}({}\\cdot{},x)}\\) for any \\(\\gamma>0\\) and \\(\\beta\\in\\R\\).\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}\n\\end{lemma}\n\n\\begin{lemma}[basic properties of \\(\\env\\)]%\n\tSuppose that \\cref{ass:basic} holds.\n\tThen, for any \\(\\gamma<\\nicefrac{1}{[\\sigma_{f,h}]_-}\\) and \\(\\beta\\in\\R\\) the following hold:\n\t\\begin{enumerate}\n\t\\item\\label{thm:env:C0}%\n\t\t\\(\\env\\) is (real-valued and) continuous on \\(C\\times C\\); in fact, it is locally Lipschitz provided that \\(f,h\\in\\C^2(C)\\).\n\t\\item\\label{thm:env:eq}%\n\t\tFor any \\(x,x^-\\in C\\) and \\(\\bar x\\in\\T(x,x^-)\\)\n\t\t\\[\n\t\t\\ifsiam\n\t\t\t\\mathtight[0.45]\n\t\t\\fi\n\t\t\t\\env(x,x^-)\n\t\t{}={}\n\t\t\t\\M(\\bar x;x,x^-)\n\t\t{}={}\n\t\t\t\\varphi(\\bar x)\n\t\t\t{}+{}\n\t\t\t\\D_{\\h-\\f}(\\bar x,x)\n\t\t\t{}+{}\n\t\t\t\\D_{\\f}(\\bar x,x^-)\n\t\t\t{}-{}\n\t\t\t\\D_{\\f}(x,x^-).\n\t\t\\]\n\t\\item\\label{thm:env:leq}%\n\t\t\\(\n\t\t\t\\env(x,x^-)\n\t\t{}\\leq{}\n\t\t\t\\varphi(x)\n\t\t\\)\n\t\tfor any \\(x,x^-\\in C\\).\n\t\\end{enumerate}\n\t\\begin{proof}~\n\t\t\\begin{proofitemize}\n\t\t\\item \\ref{thm:env:C0}~\n\t\t\tIn light of the uniform level boundedness asserted in \\cref{thm:M:lb}, continuity of \\(\\env\\) follows from $\\text{\\cite[Thm. 1.17(c)]{rockafellar2011variational}}$ by observing that the mapping \\((x,x^-)\\mapsto\\M(w;x,x^-)\\) is continuous for every \\(w\\);\n\t\t\tin fact, when \\(f\\) and \\(h\\) are both \\(\\C^2\\) on \\(C\\), its gradient exists and is continuous with respect to all its arguments, which together with local boundedness of \\(\\T\\), cf. \\cref{thm:T:osc}, gives that \\(-\\env\\) is a lower-\\(\\C^1\\) function in the sense of \\cite[Def. 10.29]{rockafellar2011variational}, and in particular locally Lipschitz continuous by virtue of \\cite[Thm.s 10.31 and 9.2]{rockafellar2011variational}.\n\t\t\\item \\ref{thm:env:eq} \\& \\ref{thm:env:leq}~\n\t\t\tThe identity follows by definition, cf. \\eqref{eq:env} and \\eqref{eq:T}.\n\t\t\tThe inequality follows by considering \\(w=x\\) in \\eqref{eq:env} and \\eqref{eq:M-D}.\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}\n\\end{lemma}\n\n\t\t\tWe now work towards establishing a merit function for \\BiFRB, starting from comparing the values of \\(\\env(\\bar x,x)\\) and \\(\\env(x,x^-)\\), with \\(\\bar x\\in\\T(x,x^-)\\).\nOwing to \\cref{thm:env:leq}, we have\n\\begin{align}\n\\nonumber\n\t\\env(\\bar x,x)\n{}\\leq{}\n\t\\varphi(\\bar x)\n{}={} &\n\t\\cost(\\bar x)\n\\\\\n\\label{eq:descent}\n{}={} &\n\t\\env(x,x^-)\n\t{}-{}\n\t\\D_{\\h-\\f}(\\bar x,x)\n\t{}-{}\n\t\\overbracket*{\n\t\t\\D_{\\f}(\\bar x,x^-)\n\t}\n\t{}+{}\n\t\\D_{\\f}(x,x^-).\n\\end{align}\n\n\n\\begin{theorem}\\label{thm:SD}%\n\tSuppose that \\cref{ass:basic} holds and consider one of the following scenarios:\n\t\\begin{enumerator}\n\t\\item\\label{thm:SD:fconv}%\n\t\teither \\(\\f\\) is convex (\\ie, \\(\\alpha p_{f,h}-\\beta\\geq0\\)) and\n\t\t\\(\n\t\t\t\\beta\n\t\t{}>{}\n\t\t\t-\\nicefrac{(1+3\\alpha p_{-f,h})}{2}\n\t\t\\),\n\t\tin which case\n\t\t\\[\n\t\t\t\\xi\\coloneqq\\f\n\t\t\\quad\\text{and}\\quad\n\t\t\t\tc\n\t\t\t{}\\coloneqq{}\n\t\t\t\t1+2\\beta+3\\alpha p_{-f,h}>0,\n\t\t\\]\n\t\\item\\label{thm:SD:fsmooth}%\n\t\tor \\(\\f\\) is \\(L_{\\f}\\)-Lipschitz differentiable, \\(h\\) is \\(\\sigma_h\\)-strongly convex, and\n\t\t\\[\n\t\t\tc\n\t\t{}\\coloneqq{}\n\t\t\t(1+\\alpha p_{-f,h})\\sigma_h\n\t\t\t{}-{}\n\t\t\t2\\gamma L_{\\f}\n\t\t{}>{}\n\t\t\t0,\n\t\t\\]\n\t\tin which case \\(\\xi\\coloneqq L_{\\f}\\j\\).\n\t\\end{enumerator}\n\tThen, for \\(\\LL\\) as in \\eqref{eq:LL} the following assertions hold:\n\t\\begin{enumerate}\n\t\\item\\label{thm:SD:leq}%\n\t\t\\begin{subequations}\n\t\t\tFor every \\(x,x^-\\in C\\) and \\(\\bar x\\in\\T(x,x^-)\\),\n\t\t\t\\begin{align}\n\t\t\t\\label{eq:SD}\n\t\t\t\t\\LL(\\bar x,x)\n\t\t\t{}\\leq{} &\n\t\t\t\t\\LL(x,x^-)\n\t\t\t\t{}-{}\n\t\t\t\t\\tfrac{c}{2\\gamma}\\D(\\bar x,x)\n\t\t\t\t{}-{}\n\t\t\t\t\\tfrac{c}{2\\gamma}\\D(x,x^-)\n\t\t\t\\shortintertext{and}\n\t\t\t\\label{eq:L:geq}\n\t\t\t\t\\cost(\\bar x)\n\t\t\t{}\\leq{} &\n\t\t\t\t\\LL(x,x^-)\n\t\t\t\t{}-{}\n\t\t\t\t\\tfrac{c}{\\gamma}\\D(\\bar x,x)\n\t\t\t\t{}-{}\n\t\t\t\t\\tfrac{c}{2\\gamma}\\D(x,x^-).\n\t\t\t\\end{align}\n\t\t\\end{subequations}\n\t\\item\\label{thm:SD:inf}%\n\t\t\\(\\inf\\LL=\\inf\\cost\\), and \\(\\LL\\) is level bounded iff so is \\(\\cost\\).\n\t\\end{enumerate}\n\\end{theorem}\n\n\\begin{theorem}[function value convergence of \\BiFRB]\\label{thm:subseq}%\n\tLet $\\seq{x^k}$ be a sequence generated by \\BiFRB* in the setting of \\cref{thm:SD}.\n\tThen,\n\t\\begin{enumerate}\n\t\\item\\label{thm:subseq:SD}%\n\t\tIt holds that\n\t\t\\begin{equation}\\label{SD inequality}\n\t\t\t\\LL\\left(x^{k+1},x^k\\right)\\leq\\LL\\left(x^k,x^{k-1}\\right)-\\tfrac{c}{2\\gamma}\\D\\left(x^{k+1},x^k\\right)-\\tfrac{c}{2\\gamma}\\D\\left(x^k,x^{k-1}\\right).\n\t\t\\end{equation}\n\t\tThen, $\\sum_{k=0}^\\infty \\D\\left(x^k,x^{k-1}\\right)<\\infty$ and $\\LL(x^{k},x^{k-1})\\to\\varphi^\\star$ as $k\\to\\infty$ for some $\\varphi^\\star\\geq\\inf\\cost$.\n\t\\item\\label{thm:subseq:bounded}%\n\t\tIf \\(\\cost\\) is level bounded, then \\(\\seq{x^k}\\) is bounded.\n\t\\item\\label{thm:subseq:omega}%\n\t\tSuppose that \\cref{ass:h} holds, and let \\(\\omega\\) be the set of limit points of \\(\\seq{x^k}\\).\n\t\tThen, \\(\\varphi\\) is constant on \\(\\omega\\) with value \\(\\varphi^\\star\\), and for every \\(x^\\star\\in\\omega\\) it holds that \\(x^\\star\\in\\T(x^\\star,x^\\star)\\) and \\(0\\in\\hat\\partial\\varphi(x^\\star)\\).\n\t\\end{enumerate}\n\t\\begin{proof}~\n\t\t\\begin{proofitemize}\n\t\t\\item\\ref{thm:subseq:SD}~\n\t\t\tRecall from \\cref{thm:SD} that \\eqref{SD inequality} holds and that \\(\\inf\\LL=\\inf\\cost>-\\infty\\), from which the convergence of \\(\\seq{\\LL(x^{k},x^{k-1})}\\) readily follows.\n\t\t\tIn turn, a telescoping argument on \\eqref{SD inequality} shows that \\(\\sum_{k\\in\\N}\\D(x^k,x^{k-1})<\\infty\\).\n\t\t\\item\\ref{thm:subseq:bounded}~\n\t\t\tIt follows from \\cref{thm:subseq:SD} that \\(\\LL\\left(x^{k+1},x^k\\right)\\leq\\LL(x^0,x^{-1})\\) holds for every \\(k\\).\n\t\t\tThen boundedness of \\(\\seq{x^k}\\) is implied by level boundedness of \\(\\LL\\); see \\cref{thm:SD:inf}.\n\t\t\\item\\ref{thm:subseq:omega}~\n\t\t\tSuppose that a subsequence \\(\\seq{x^{k_j}}[j\\in\\N]\\) converges to a point \\(x^\\star\\), then so do \\(\\seq{x^{k_j\\pm1}}\\) by \\cref{thm:subseq:SD} and \\cite[Prop. 2.2(iii)]{bauschke2003iterating}.\n\t\t\tSince \\(x^{k_j+1}\\in\\T(x^{k_j},x^{k_j-1})\\), by passing to the limit osc of \\(\\T\\) (\\cref{thm:T:osc}) implies that \\(x^\\star\\in\\T(x^\\star,x^\\star)\\).\n\t\t\tInvoking \\cref{thm:T:fix} yields the stationarity condition \\(0\\in\\hat\\partial\\varphi(x^\\star)\\).\n\t\t\tMoreover, by continuity of \\(\\LL\\) one has\n\t\t\t\\[\n\t\t\t\t\\varphi^\\star\n\t\t\t{}\\defeq{}\n\t\t\t\t\\lim_{k\\to\\infty}\\LL\\left(x^k,x^{k-1}\\right)\n\t\t\t{}={}\n\t\t\t\t\\LL(x^\\star,x^\\star)\n\t\t\t{}={}\n\t\t\t\t\\varphi(x^\\star),\n\t\t\t\\]\n\t\t\twhere the last equality follows from \\cref{thm:env:eq}, owing to the inclusion \\(x^\\star\\in\\T(x^\\star,x^\\star)\\) (and the fact that \\(\\D_\\psi(x,x)=0\\) for any differentiable function \\(\\psi\\)).\n\t\t\tFrom the arbitrarity of \\(x^\\star\\in\\omega\\) we conclude that \\(\\varphi\\equiv\\varphi^\\star\\) on \\(\\omega\\).\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}\n\\end{theorem}\n\n\\begin{lemma}[{{\\cite[Lem. 4.4]{wang2022exact}}}]\\label{lemma: Uniformize GCKL}%\n\tLet $\\func{f}{\\R^n}{\\Rinf}$ be proper lsc and let $\\mu\\in\\R$.\n\tLet $\\Omega\\subseteq\\dom\\partial f$ be a nonempty compact set on which $f(x)=\\mu$ for all $x\\in\\Omega$.\n\tThen the following hold:\n\t\\begin{enumerate}\n\t\\item\n\t\tSuppose that $f$ satisfies the pointwise generalized concave KL property at each $x\\in\\Omega$.\n\t\tThen there exist $\\varepsilon>0,\\eta\\in(0,\\infty]$ and $\\psi\\in\\Psi_\\eta$ such that $f$ has the setwise generalized concave KL property on $\\Omega$ with respect to $U=\\Omega_\\varepsilon$, $\\eta$ and $\\psi$.\n\t\\item \\label{uniformize exact modulus}%\n\t\tSet $U=\\Omega_\\varepsilon$ and define $h:(0,\\eta)\\rightarrow\\R_+$ by\n\t\t\\[\n\t\t\th(s)=\\sup\\set{\\dist^{-1}\\left(0,\\partial f(x)\\right)}[{x\\in U\\cap[s\\leq f-\\mu<\\eta]}].\n\t\t\\]\n\t\tThen the function $\\func{\\tilde\\psi}{[0,\\eta)}{\\R_+}$ defined by \\((\\forall 0<t<\\eta)~\\tilde\\psi(t)=\\int_0^th(s)ds\\) with \\(\\tilde{\\psi}(0)=0\\) is well defined and belongs to $\\Psi_\\eta$.\n\t\tThe function $f$ has the setwise generalized concave KL property on $\\Omega$ with respect to $U$, $\\eta$ and $\\tilde\\psi$.\n\t\tMoreover,\n\t\t\\[\n\t\t\t\\tilde{\\psi}\n\t\t{}={}\n\t\t\t\\inf\\left\\{\n\t\t\t\t\\psi\\in\\Psi_\\eta\n\t\t\t\t\\,\\Big|\\,\n\t\t\t\t\\text{\\begin{tabular}{@{}c@{}}%\n\t\t\t\t\t\t\\(\\psi\\) is a concave desingularizing function\\\\\n\t\t\t\t\t\tof $f$ on $\\Omega$ with respect to $U$ and $\\eta$\n\t\t\t\t\\end{tabular}}\n\t\t\t\\right\\}.\n\t\t\\]\n\t\\end{enumerate}\n\tWe say that $\\tilde{\\psi}$ is the exact modulus of the setwise generalized concave KL property of \\(f\\) on \\(\\Omega\\) with respect to $U$ and $\\eta$.\n\\end{lemma}\n\n\\begin{proof}\n\tSet \\((\\forall k\\in\\N)\\) $\\delta_{k}=\\F(z^k)$ for simplicity.\n\tThen $\\delta_k=\\LL(x^k,x^{k-1})$, $\\delta_{k}\\to\\varphi^\\star$ decreasingly and $\\dist(x^k,\\omega)\\to0$ as $k\\to\\infty$ by invoking \\cref{thm:subseq}.\n\tAssume without loss of generality that \\((\\forall k\\in\\N)\\) $\\delta_{k}>\\varphi^\\star$, otherwise we would have $(\\exists k_0\\in\\N)$ $x^{k_0}=x^{k_0+1}$ due to \\cref{thm:subseq:SD}, from which the desired result readily follows by simple induction.\n\tThus $(\\exists k_0\\in\\N)$ $(\\forall k\\geq k_0)$ $z^k\\in\\Omega_r\\cap[0<\\F-\\varphi^\\star<\\eta]$.\n\tAppealing to \\cref{thm:subseq:omega} and \\cref{thm:M:eq} yields that $\\F$ is constantly equal to $\\varphi^\\star$ on the compact set $\\omega(z^0)$.\n\tIn turn, all conditions in \\cref{uniformize exact modulus} are satisfied, which implies that for $k\\geq k_0$\n\t\\begin{equation}\\label{formula: KL inequality of F}\n\t\t1\\leq(\\tilde\\psi)'_-\\left(\\delta_{k}-\\varphi^\\star\\right)\\dist\\left(0,\\partial\\F(z^k)\\right).\n\t\\end{equation}\n\tDefine $(\\forall k\\in\\N)$\n\t\\begin{align*}\n\t\tu^k={}\n\t&\n\t\t\\nabla^2\\h(x^k)(x^k-x^{k+1})+\\nabla^2\\f(x^k)(x^{k+1}-x^k)+\\tfrac{c}{2\\gamma}\\bigl(\\nabla h(x^k)-\\nabla h(x^{k-1})\\bigr)\n\t\\\\\n\t&\n\t\t+\\nabla\\f(x^{k-1})-\\nabla\\f(x^k)+\\nabla\\xi(x^k)-\\nabla\\xi(x^{k-1}),\n\t\\\\\n\t\tv^k={}\n\t&\n\t\t\\nabla^2\\f(x^{k-1})(x^k-x^{k+1})+\\tfrac{c}{2\\gamma}\\nabla^2h(x^{k-1})(x^{k-1}-x^k)+\\nabla^2\\xi(x^{k-1})(x^{k-1}-x^k).\n\t\\end{align*}\n\tThen $\\norm{u^k}\\leq(M_1+M_2)\\norm{x^{k+1}-x^k}+(M_1+(\\nicefrac{c}{2\\gamma})M_3+M_4)\\norm{x^k-x^{k-1}}$ and $\\norm{v^k}\\leq M_1\\norm{x^{k+1}-x^k}+((\\nicefrac{c}{2\\gamma})M_3+M_4)\\norm{x^k-x^{k-1}}$, where $M_1=\\sup\\|\\nabla^2\\f(\\mathcal{B})\\|$, $M_2=\\sup\\|\\nabla^2\\h(\\mathcal{B})\\|$, $M_3=\\sup\\norm{\\nabla^2h(\\mathcal{B})}$, and $M_4=\\sup\\norm{\\nabla^2\\xi(\\mathcal{B})}$.\n\tApplying subdifferential calculus to $\\partial\\F(z_k)$ yields that\n\t\\[\n\t\t\\partial\\F(z^k)\n\t{}={}\n\t\t\\bigl(\\partial\\varphi(x^{k+1})+\\nabla\\h(x^{k+1})-\\nabla\\h(x^k)+\\nabla\\f(x^{k})-\\nabla \\f(x^{k-1})\\bigr)\\times\\{u^k\\}\\times\\{v^k\\},\n\t\\]\n\twhich together with \\cref{thm:T:partial} entails that $(0,u^k,v^k)\\in\\partial\\F(z^k)$.\n\tIn turn, summing the aforementioned bounds on $\\norm{u^k}$ and $\\norm{v^k}$ gives\n\t\\begin{equation}\\label{subd F upper bound}\n\t\t\\dist\\left(0,\\partial\\F(z^k)\\right)\\leq M\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})},\n\t\\end{equation}\n\twhere $M=\\max\\set{2M_1+M_2,M_1+(\\nicefrac{c}{\\gamma})M_3+2M_4}$.\n\n\tFinally, we show that $\\seq{x^k}$ is convergent.\n\tFor simplicity, define $(\\forall k,l\\in\\N)$ $\\Delta_{k,l}=\\tilde\\psi\\left(\\delta_{k}-\\varphi^\\star\\right)-\\tilde\\psi\\left(\\delta_{l}-\\varphi^\\star\\right)$.\n\tThen, combining~\\eqref{formula: KL inequality of F} and~\\eqref{subd F upper bound} yields\n\t\\begin{align*}\n\t\t1\n\t&{}\\leq{}\n\t\tM(\\tilde{\\psi})'_-(\\delta_{k}-\\varphi^\\star)\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}\n\t\t{}\\leq{}\n\t\t\\tfrac{M\\Delta_{k,k+1}}{\\delta_k-\\delta_{k+1}}\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}\n\t\\\\\n\t&{}\\leq{}\n\t\t\\frac{2\\gamma M\\Delta_{k,k+1}}{c\\left(\\D(x^{k+1},x^k)+\\D(x^k,x^{k-1})\\right)}\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}\n\t\\\\\n\t&{}\\leq{}\n\t\t\\frac{2\\gamma M\\Delta_{k,k+1}\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}}{c \\sigma\\norm{x^{k+1}-x^k}^2+c \\sigma\\norm{x^k-x^{k-1}}^2}\\leq \\frac{4\\gamma M\\Delta_{k,k+1}}{c \\sigma\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}},\n\t\\end{align*}\n\twhere the second inequality is implied by concavity of $\\tilde\\psi$, the third one follows from \\eqref{SD inequality}, and the fourth one holds because $\\sigma>0$ is the strong convexity modulus of $h$ on $\\mathcal{B}$.\n\tHence,\n\t\\begin{equation}\\label{consecutive gap}\n\t\t\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}\\leq\\tfrac{4\\gamma M}{c\\sigma}\\Delta_{k,k+1}.\n\t\\end{equation} \n\tSumming~\\eqref{consecutive gap} from $k=k_0$ to an arbitrary $l\\geq k_0+1$ and passing $l$ to infinity justifies~\\eqref{finite length}.\n\tA similar procedure shows that $\\seq{x^k}$ is Cauchy, which together with \\cref{thm:subseq:omega} entails the rest of the statement. \n\\end{proof}\n\n\\begin{lemma}\\label{thm:fconv:general}%\n\tSuppose that \\cref{ass:basic} holds and let \\(\\gamma>0\\) and \\(\\beta\\in\\R\\) be such that \\(\\f\\coloneqq f-\\frac\\beta\\gamma h\\) is a convex function.\n\tThen, for every \\(x,x^-\\in C\\) and \\(\\bar x\\in\\T(x,x^-)\\)\n\t\\begin{align}\\label{eq:decrease:fconv}\n\t\t\\left(\\env+\\D_{\\f}\\right)(\\bar x,x)\n\t{}\\leq{} &\n\t\t\\left(\\env+\\D_{\\f}\\right)(x,x^-)\n\t\t{}-{}\n\t\t\\D_{\\h-2\\f}(\\bar x,x).\n% \t\\shortintertext{%\n% \t\tIf \\(\\h-\\f\\) too is convex, then it also holds that\n% \t}\n% \t\t\\varphi(\\bar x)\n% \t{}\\leq{} &\n% \t\t\\left(\\env+\\D_{\\f}\\right)(x,x^-)\n% \t\t{}-{}\n% \t\t\\D_{\\h-2\\f}(\\bar x,x),\n\t\\end{align}\n\tIf \\(\\h-\\f\\) too is convex, then \\(\\env+\\D_{\\f}\\) has the same infimum of \\(\\cost\\), and is level bounded iff so is \\(\\cost\\).%\n\t\\begin{proof}\n\t\tAll the claimed inequalities follows from \\eqref{eq:descent} together with the fact that \\(\\D_{\\f}\\geq0\\), and that \\(\\D_{\\h-\\f}\\geq0\\) too when \\(\\h-\\f\\) is convex.\n\t\tWhen both \\(\\f\\) and \\(\\h-\\f\\) are convex, \\cref{thm:env:eq} implies that\n\t\t\\begin{equation}\\label{eq:fconv:geq}\n\t\t\t\\env(y,y^-)+\\D_{\\f}(y,y^-)\\geq\\varphi(\\bar y)\\geq\\inf\\cost\n\t\t\\quad\n\t\t\t\\forall y,y^-\\in C,\\ \\bar y\\in\\T(y,y^-),\n\t\t\\end{equation}\n\t\tproving that \\(\\inf(\\env+\\D_{\\f})\\geq\\inf\\cost\\).\n\t\tThe converse inequality follows from \\cref{thm:env:leq} by observing that \\((\\env+\\D_{\\f})(x,x)=\\env(x,x)\\leq\\varphi(x)\\).\n\n\t\tTo conclude, suppose that \\(\\env+\\D_{\\f}\\) is not level bounded, and consider an unbounded sequence \\(\\seq{x_k,x_k^-}\\) such that \\(\\env(x_k,x_k^-)+\\D_{\\f}(x_k,x_k^-)\\leq\\ell\\), for some \\(\\ell\\in\\R\\).\n\t\tThen, it follows from \\eqref{eq:fconv:geq} that \\(\\varphi(\\bar x_k)\\leq\\ell\\), where \\(\\bar x_k\\in\\T(x_k,x_k^-)\\).\n\t\tFrom local boundedness of \\(\\T\\) (\\cref{thm:T:osc}) it follows that \\(\\seq{\\bar x_k}\\) too is unbounded, showing that \\(\\varphi\\) is not level bounded either.\n\t\tThe converse holds by observing that \\((\\env+\\D_{\\f})(x,x)=\\env(x,x)\\leq\\cost(x)\\), cf. \\cref{thm:env:leq}.\n\t\\end{proof}\n\\end{lemma}\n\nIn the setting of \\cref{thm:SD:fconv}, inequality \\eqref{eq:decrease:fconv} can equivalently be written in terms of \\(\\LL\\) as\n\\begin{align*}\n\t\\LL(\\bar x,x)\n{}\\leq{} &\n\t\\LL(x,x^-)\n\t{}-{}\n\t\\D_{\\h-2\\f-\\frac c\\gamma h}(\\bar x,x)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(\\bar x,x)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(x,x^-)\n\\\\\n{}\\leq{} &\n\t\\LL(x,x^-)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(\\bar x,x)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(x,x^-),\n\\end{align*}\nwhere the second inequality owes to the fact that \\(\\D_{\\h-2\\f-\\frac c\\gamma h}\\geq0\\), since \\(\\h-2\\f-\\frac c\\gamma h\\) is convex, having\n\\[\n\t\\h-2\\f-\\tfrac c\\gamma h\n{}={}\n\t\\tfrac{1+2\\beta-c}{\\gamma}h-3f\n{}={}\n\t\\overbracket*{\\tfrac{1+2\\beta+3\\alpha p_{-f,h}-c}{\\gamma}}^{=0}h+2(-f-\\sigma_{-f,h}h),\n\\]\nthe coefficient of \\(h\\) being null by definition of \\(c\\), and \\(-f-\\sigma_{-f,h}h\\) being convex by definition of the relative weak convexity modulus \\(\\sigma_{-f,h}\\), cf. \\cref{def:relweak}.\nThis proves \\eqref{eq:SD}; inequality \\eqref{eq:L:geq} follows similarly by observing that\n\\[\n\t0\n{}\\leq{}\n\t\\D_{\\h-2\\f-\\frac c\\gamma h}\n{}={}\n\t\\D_{\\h-\\f}-\\D_{\\f}-\\tfrac c\\gamma\\D\n{}\\leq{}\n\t\\D_{\\h-\\f}-\\tfrac c\\gamma\\D,\n\\]\nso that\n\\[\n\t\\LL(x,x^-)\n{}\\overrel{\\ref{thm:env:eq}}{}\n\t\\varphi(\\bar x)\n\t{}+{}\n\t\\overbracket*{\n\t\t\\D_{\\h-\\f}\n\t\t\\vphantom{x^-}\n\t}^{\\geq\\frac c\\gamma\\D}(\\bar x,x)\n\t{}+{}\n\t\\overbracket*{\n\t\t\\D_{\\f}(\\bar x,x^-)\n\t}^{\\geq0}\n\t{}+{}\n\t\\tfrac{c}{2\\gamma}\\D(x,x^-).\n\\]\nIn turn, \\cref{thm:SD:inf} follows from the same arguments as in the proof of \\cref{thm:fconv:general}.\n\n\n\\begin{lemma}\\label{thm:fsmooth:general}%\n\tAdditionally to \\cref{ass:basic}, suppose that \\(\\f\\) is \\(L_{\\f}\\)-Lipschitz differentiable for some \\(L_{\\f}\\geq0\\).\n\tThen, for every \\(x,x^-\\in C\\) and \\(\\bar x\\in\\T(x,x^-)\\)\n\t\\begin{equation}\\label{eq:decrease:fsmooth}\n\t\t\\left(\\env+\\D_{L_{\\f}\\j}\\right)(\\bar x,x)\n\t{}\\leq{}\n\t\t\\left(\\env+\\D_{L_{\\f}\\j}\\right)(x,x^-)\n\t\t{}-{}\n\t\t\\D_{\\h-2L_{\\f}\\j}(\\bar x,x).\n\t\\end{equation}\n\tIf \\(\\h-L_{\\f}\\j\\) is a convex function, then \\(\\env+\\D_{L_{\\f}\\j}\\) has the same infimum of \\(\\cost\\), and is level bounded iff so is \\(\\cost\\).\n\t\\begin{proof}\n\t\tBy means of the three-point identity, that is, by using \\eqref{eq:M-innprod} in place of \\eqref{eq:M-D}, inequality \\eqref{eq:descent} can equivalently be written as\n\t\t\\begin{align*}\n\t\t\t\\env(\\bar x,x)\n\t\t{}\\leq{}\n\t\t\t\\cost(\\bar x)\n\t\t{}={} &\n\t\t\t\\env(x,x^-)\n\t\t\t{}-{}\n\t\t\t\\D_{\\h}(\\bar x,x)\n\t\t\t{}-{}\n\t\t\t\\innprod*{\\bar x-x}{\\nabla\\f(x)-\\nabla\\f(x^-)}\n\t\t\\shortintertext{%\n\t\t\twhich by using Young's inequality on the inner product and \\(L_{\\f}\\)-Lipschitz differentiability yields\n\t\t}\n\t\t{}\\leq{} &\n\t\t\t\\env(x,x^-)\n\t\t\t{}-{}\n\t\t\t\\D_{\\h}(\\bar x,x)\n\t\t\t{}+{}\n\t\t\t\\tfrac{L_{\\f}}{2}\\|\\bar x-x\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{L_{\\f}}{2}\\|x-x^-\\|^2.\n\t\t\\end{align*}\n\t\tRearranging and using the fact that \\(\\D_{\\j}(x,y)=\\frac12\\|x-y\\|^2\\) yields the claimed inequality.\n\t\t\n\t\tUnder convexity of \\(\\h-L_{\\f}\\j\\), we may draw the same conclusions as in the proof of \\cref{thm:fconv:general} by observing that the same application of Young's inequality above, combined with the expression \\eqref{eq:M-innprod} of \\(\\M\\), gives\n\t\t\\[\n\t\t\t\\env(y,y^-)\n\t\t{}={}\n\t\t\t\\M(\\bar y;y,y^-)\n\t\t{}\\geq{}\n\t\t\t\\varphi(\\bar y)\n\t\t\t{}+{}\n\t\t\t\\D_{\\h}(\\bar y,y)\n\t\t\t{}-{}\n\t\t\t\\tfrac{L_{\\f}}{2}\\|\\bar y-y\\|^2\n\t\t\t{}-{}\n\t\t\t\\tfrac{L_{\\f}}{2}\\|y-y^-\\|^2,\n\t\t\\]\n\t\tholding for any \\(y,y^-\\in C\\) and \\(\\bar y\\in\\T(y,y^-)\\).\n\t\\end{proof}\n\\end{lemma}\n\nWe will pattern the arguments in the previous section, and observe that inequality \\eqref{eq:decrease:fsmooth} can equivalently be written in terms of \\(\\LL\\) as\n\\[\n\t\\LL(\\bar x,x)\n{}\\leq{}\n\t\\LL(x,x^-)\n\t{}-{}\n\t\\overbracket*{\n\t\t\\D_{\\h-2L_{\\f}\\j-\\frac c\\gamma h}\n\t}^{\\geq0}(\\bar x,x)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(\\bar x,x)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(x,x^-).\n\\]\nOnce again, the fact that \\(\\D_{\\h-2L_{\\f}\\j-\\frac c\\gamma h}\\geq0\\) owes to convexity of \\(\\h-2L{\\f}\\j-\\frac c\\gamma h\\), having\n{%\n\\ifsiam\n\t\\mathtight[0.8]%\n\\fi\n\t\\begin{align*}\n\t\t\\h-2L_{\\f}\\j-\\tfrac c\\gamma h\n\t{}={} &\n\t\t\\tfrac{1-c}{\\gamma}h\n\t\t{}-{}\n\t\tf\n\t\t{}-{}\n\t\t2L_{\\f}\\j\n\t\\\\\n\t{}={} &\n\t\t\\def\\myvphantom{\\vphantom{\\tfrac{1+\\gamma\\sigma_{-f,h}-c}{\\gamma}}}\n\t\t\\overbracket[0.5pt]{\n\t\t\t\\myvphantom\n\t\t\t\\tfrac{1+\\gamma\\sigma_{-f,h}-c}{\\gamma}\n\t\t}^{=\\nicefrac{2L_{\\f}}{\\sigma_h}\\geq0}\n\t\t(\\overbracket*{\n\t\t\t\\myvphantom\n\t\t\th-\\sigma_h\\j\n\t\t}^{\\text{convex}})\n\t\t{}+{}\n\t\t(\\overbracket*{\n\t\t\t\\myvphantom\n\t\t\t-f-\\sigma_{-f,h}h\n\t\t}^{\\text{convex}})\n\t\t{}+{}\n\t\t\\bigl(\\overbracket*{\n\t\t\t\\myvphantom\n\t\t\t\\tfrac{1+\\gamma\\sigma_{-f,h}-c}{\\gamma}\\sigma_h\n\t\t\t{}-{}\n\t\t\t2L_{\\f}\n\t\t}^{=0}\n\t\t\\bigr)\\j\n\t\\end{align*}\n}%\naltogether proving \\eqref{eq:SD}.\nSimilarly, inequality \\eqref{eq:L:geq} follows by observing that\n\\(\n\t0\n{}\\leq{}\n\t\\D_{\\h-2\\f-\\frac c\\gamma h}\n{}={}\n\t\\D_{\\h-\\f}-\\D_{\\f}-\\frac c\\gamma\\D\n{}\\leq{}\n\t\\D_{\\h-\\f}-\\frac c\\gamma\\D\n\\),\nso that\n\\[\n\t\\LL(x,x^-)\n{}\\overrel{\\ref{thm:env:eq}}{}\n\t\\varphi(\\bar x)\n\t{}+{}\n\t\\overbracket*{\n\t\t\\D_{\\h-\\f}\n\t\t\\vphantom{x^-}\n\t}^{\\geq\\frac c\\gamma\\D}(\\bar x,x)\n\t{}+{}\n\t\\overbracket*{\n\t\t\\D_{\\f}(\\bar x,x^-)\n\t}^{\\geq0}\n\t{}+{}\n\t\\tfrac{c}{2\\gamma}\\D(x,x^-).\n\\]\nThe assertion of \\cref{thm:SD:inf} once again follows from the same arguments as in the proof of \\cref{thm:fsmooth:general}.\n","units":[{"start":27070,"end":27504,"start_line":550,"end_line":554,"raw_tex":"\\begin{lemma}[relative prox-boundedness]\\label{thm:proxbounded}%\n\tSuppose that \\cref{ass:basic} holds.\n\tThen, the set \\(\\argmin\\set{\\gamma g+h+\\innprod{v}{\\cdot}}\\) as in \\cref{ass:T} is nonempty for any \\(v\\in\\R^n\\) and \\(0<\\gamma<\\nicefrac{1}{[\\sigma_{-f,h}]_-}\\).\n\tIn other words, \\(g\\) is prox bounded relative to \\(h\\) with threshold \\(\\gamma_{g,h}\\geq\\nicefrac{1}{[\\sigma_{-f,h}]_-}\\) \\cite[Def. 2.3]{kan2012moreau}.\n\\end{lemma}","sha256_utf8":"614714c10968a347c99627685b9ba744572ce44b88386e01513be0bb0e055c56"},{"start":27505,"end":28169,"start_line":555,"end_line":580,"raw_tex":"\\begin{proof}\n\tRecall that a proper convex function admits affine minorant; see, e.g,~\\cite[Cor. 16.18]{bauschke2017convex}.\n\tIt then follows that \\(\\gamma g+h\\) is 1-coercive by observing that\n\t\\[\n\t\t\\gamma g+h\n\t{}={}\n\t\t\\gamma\\cost-\\gamma f+h\n\t{}={}\n\t\t\\gamma\\underbracket*{\n\t\t\t\\vphantom{\\left([\\sigma_{-f,h}]_-\\right)}\n\t\t\t\\cost\n\t\t}_{\\mathclap{\\geq\\inf\\cost}}\n\t\t{}+{}\n\t\t\\gamma\\underbracket*{\n\t\t\t\\left(-f+[\\sigma_{-f,h}]_-h\\right)\n\t\t}_{\\text{convex}}\n\t\t{}+{}\n\t\t\\underbracket*{\n\t\t\t\\left(1-\\gamma[\\sigma_{-f,h}]_-\\right)\n\t\t}_{>0}\n\t\t\\overbracket*{\n\t\t\t\\,h\\,\n\t\t}^{\\text{\\clap{1-coercive}}}\n\t\\]\n\ton \\(\\closure C\\) and \\(\\infty\\) on \\(\\R^n\\setminus\\closure C\\).\n\\end{proof}","sha256_utf8":"8783a14ea33006b37834fbe69b1a2e2a9507b4f79885fcb61608d2584c7fe090"},{"start":32839,"end":33087,"start_line":675,"end_line":678,"raw_tex":"\\begin{lemma}[{{\\cite[Thm. 4.1]{ahookhosh2021bregman}}}]\\label{thm:Legendre}%\n\tSuppose that \\cref{ass:f} holds.\n\tThen, for every \\(\\gamma<\\nicefrac{1}{[\\sigma_{-f,h}]_-}\\) the function \\(\\h\\) is a Legendre kernel with \\(\\dom\\h=\\dom h\\).\n\\end{lemma}","sha256_utf8":"0aace98996aff81c09a35bcb2bf6200352c4ac1124dbbe5195e28f0d28302580"},{"start":35169,"end":38286,"start_line":732,"end_line":783,"raw_tex":"\\begin{lemma}[basic properties of the model \\(\\M\\) and the operator \\(\\T\\)]\\label{thm:basic}%\n\tSuppose that \\cref{ass:basic} holds, and let \\(\\gamma<\\nicefrac{1}{[\\sigma_{-f,h}]_-}\\) and \\(\\beta\\in\\R\\) be fixed.\n\tThe following hold:\n\t\\begin{enumerate}\n\t\\item \\label{thm:M:eq}%\n\t\t\\(\n\t\t\t\\M(x;x,x^-)\n\t\t{}={}\n\t\t\t\\cost(x)\n\t\t\\)\n\t\tfor all \\(x,x^-\\in C\\).\n\t\\item \\label{thm:M:lb}%\n\t\t\\(\\M(w;x,x^-)\\) is level bounded in \\(w\\) locally uniformly in \\((x,x^-)\\) on \\(C\\times C\\).\\footnote{%\n\t\t\tNamely, \\(\\bigcup_{(x,x^-)\\in\\mathcal V}\\set{w\\in\\R^n}[\\M(w;x,x^-)\\leq\\alpha]\\) is bounded for any \\(\\mathcal V\\subset C\\times C\\) compact and \\(\\alpha\\in\\R\\).%\n\t\t}\n\t\\item \\label{thm:T:osc}%\n\t\t\\(\\T\\) is locally bounded and osc,\\footnote{%\n\t\t\tBeing \\(\\T\\) defined on \\(C\\times C\\), osc and local boundedness are meant relative to \\(C\\times C\\).\n\t\t\tNamely, \\(\\graph\\T\\) is closed relative to \\(C\\times C\\times\\R^n\\), and \\(\\bigcup_{(x,x^-)\\in\\mathcal V}\\T(x,x^-)\\) is bounded for any \\(\\mathcal V\\subset C\\times C\\) compact.%\n\t\t}\n\t\tand \\(T(x,x^-)\\) is a nonempty and compact subset of \\(C\\) for any \\(x,x^-\\in C\\).\n\t\\item \\label{thm:T:partial}%\n\t\t\\(\\nabla\\h(x)-\\nabla\\h(\\bar x)-\\nabla\\f(x)+\\nabla\\f(x^-)\\in\\hat\\partial\\varphi(\\bar x)\\) for any \\(x,x^-\\in C\\) and \\(\\bar x\\in\\T(x,x^-)\\).\n\t\\item \\label{thm:T:fix}%\n\t\tIf \\(x\\in\\T(x,x)\\), then \\(0\\in\\hat\\partial\\varphi(x)\\) and \\(\\def\\stepsize{\\gamma'}\\T(x,x)=\\set{x}\\) for every \\(\\gamma'\\in(0,\\gamma)\\).\n\t\\end{enumerate}\n\t\\begin{proof}\n\t\tWe start by observing that \\cref{thm:proxbounded} ensures that the set \\(\\T(x,x^-)\\) is non\\-empty for any \\(x,x^-\\in C\\); this follows by considering the expression \\eqref{eq:M-innprod} of the model, by observing that, for any \\(x\\in C\\),\n\t\t\\(\n\t\t\t\\varphi+\\D_{\\h}({}\\cdot{},x)\n\t\t{}={}\n\t\t\tg+\\tfrac1\\gamma h-\\h(x)-\\innprod*{\\nabla\\h(x)}{{}\\cdot{}-x}\n\t\t\\).\n\t\tFor the same reason, it then follows from \\cref{ass:T} that \\(T(x,x^-)\\subset C\\).\n\t\t\\begin{proofitemize}\n\t\t\\item \\ref{thm:M:eq}~\n\t\t\tApparent, by considering \\(w=x\\) in \\eqref{eq:M-D}.\n\t\t\\item \\ref{thm:M:lb} \\& \\ref{thm:T:osc}~\n\t\t\tThe first assertion owes to the fact that \\(\\h\\) is 1-coercive by \\cref{thm:Legendre} and that both \\(\\h\\) and \\(\\nabla\\f\\) are continuous on \\(C\\), so that for any compact \\(\\mathcal V\\subset\\R^n\\times\\R^n\\) one has that\n\t\t\t\\(\n\t\t\t\t\\lim_{\\|w\\|\\to\\infty}\\inf_{(x,x^-)\\in\\mathcal V}\\M(w;x,x^-)=\\infty\n\t\t\t\\),\n\t\t\tas is apparent from \\eqref{eq:M-innprod}.\n\t\t\tIn turn, the second assertion follows from \\cite[Thm. 1.17]{rockafellar2011variational}.\n\t\t\\item \\ref{thm:T:partial}~\n\t\t\tFollows from the optimality conditions of \\(\\bar x\\in\\argmin\\M({}\\cdot{};x,x^-)\\), for \\(\\bar x\\in C\\) by assumption and thus the calculus rule of \\cite[Ex. 8.8(c)]{rockafellar2011variational} applies (having \\(\\h\\) smooth around \\(\\bar x\\in C\\)).\n\t\t\\item \\ref{thm:T:fix}~\n\t\t\tThat \\(0\\in\\hat\\partial\\varphi(x)\\) follows from assertion \\ref{thm:T:partial}, and the other claim from \\cite[Lem. 3.6]{ahookhosh2021bregman} by observing that \\(\\T(x,x)=\\argmin\\set{\\varphi+\\D_{\\h}({}\\cdot{},x)}\\) for any \\(\\gamma>0\\) and \\(\\beta\\in\\R\\).\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}\n\\end{lemma}","sha256_utf8":"88074a8193650d09601b557fc75f6573c8bb3226b3c16475962b49416a53099b"},{"start":42169,"end":43950,"start_line":835,"end_line":878,"raw_tex":"\\begin{lemma}[basic properties of \\(\\env\\)]%\n\tSuppose that \\cref{ass:basic} holds.\n\tThen, for any \\(\\gamma<\\nicefrac{1}{[\\sigma_{f,h}]_-}\\) and \\(\\beta\\in\\R\\) the following hold:\n\t\\begin{enumerate}\n\t\\item\\label{thm:env:C0}%\n\t\t\\(\\env\\) is (real-valued and) continuous on \\(C\\times C\\); in fact, it is locally Lipschitz provided that \\(f,h\\in\\C^2(C)\\).\n\t\\item\\label{thm:env:eq}%\n\t\tFor any \\(x,x^-\\in C\\) and \\(\\bar x\\in\\T(x,x^-)\\)\n\t\t\\[\n\t\t\\ifsiam\n\t\t\t\\mathtight[0.45]\n\t\t\\fi\n\t\t\t\\env(x,x^-)\n\t\t{}={}\n\t\t\t\\M(\\bar x;x,x^-)\n\t\t{}={}\n\t\t\t\\varphi(\\bar x)\n\t\t\t{}+{}\n\t\t\t\\D_{\\h-\\f}(\\bar x,x)\n\t\t\t{}+{}\n\t\t\t\\D_{\\f}(\\bar x,x^-)\n\t\t\t{}-{}\n\t\t\t\\D_{\\f}(x,x^-).\n\t\t\\]\n\t\\item\\label{thm:env:leq}%\n\t\t\\(\n\t\t\t\\env(x,x^-)\n\t\t{}\\leq{}\n\t\t\t\\varphi(x)\n\t\t\\)\n\t\tfor any \\(x,x^-\\in C\\).\n\t\\end{enumerate}\n\t\\begin{proof}~\n\t\t\\begin{proofitemize}\n\t\t\\item \\ref{thm:env:C0}~\n\t\t\tIn light of the uniform level boundedness asserted in \\cref{thm:M:lb}, continuity of \\(\\env\\) follows from $\\text{\\cite[Thm. 1.17(c)]{rockafellar2011variational}}$ by observing that the mapping \\((x,x^-)\\mapsto\\M(w;x,x^-)\\) is continuous for every \\(w\\);\n\t\t\tin fact, when \\(f\\) and \\(h\\) are both \\(\\C^2\\) on \\(C\\), its gradient exists and is continuous with respect to all its arguments, which together with local boundedness of \\(\\T\\), cf. \\cref{thm:T:osc}, gives that \\(-\\env\\) is a lower-\\(\\C^1\\) function in the sense of \\cite[Def. 10.29]{rockafellar2011variational}, and in particular locally Lipschitz continuous by virtue of \\cite[Thm.s 10.31 and 9.2]{rockafellar2011variational}.\n\t\t\\item \\ref{thm:env:eq} \\& \\ref{thm:env:leq}~\n\t\t\tThe identity follows by definition, cf. \\eqref{eq:env} and \\eqref{eq:T}.\n\t\t\tThe inequality follows by considering \\(w=x\\) in \\eqref{eq:env} and \\eqref{eq:M-D}.\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}\n\\end{lemma}","sha256_utf8":"d34287a8ad94fc70c52a01b959c530058757765c14425710cbba83f171da6e80"},{"start":43999,"end":44453,"start_line":882,"end_line":903,"raw_tex":"\t\t\tWe now work towards establishing a merit function for \\BiFRB, starting from comparing the values of \\(\\env(\\bar x,x)\\) and \\(\\env(x,x^-)\\), with \\(\\bar x\\in\\T(x,x^-)\\).\nOwing to \\cref{thm:env:leq}, we have\n\\begin{align}\n\\nonumber\n\t\\env(\\bar x,x)\n{}\\leq{}\n\t\\varphi(\\bar x)\n{}={} &\n\t\\cost(\\bar x)\n\\\\\n\\label{eq:descent}\n{}={} &\n\t\\env(x,x^-)\n\t{}-{}\n\t\\D_{\\h-\\f}(\\bar x,x)\n\t{}-{}\n\t\\overbracket*{\n\t\t\\D_{\\f}(\\bar x,x^-)\n\t}\n\t{}+{}\n\t\\D_{\\f}(x,x^-).\n\\end{align}\n","sha256_utf8":"1154f621838c7306ab5e5072f163edc59d987efd841e9afb53486473557fe604"},{"start":46604,"end":48018,"start_line":931,"end_line":990,"raw_tex":"\\begin{theorem}\\label{thm:SD}%\n\tSuppose that \\cref{ass:basic} holds and consider one of the following scenarios:\n\t\\begin{enumerator}\n\t\\item\\label{thm:SD:fconv}%\n\t\teither \\(\\f\\) is convex (\\ie, \\(\\alpha p_{f,h}-\\beta\\geq0\\)) and\n\t\t\\(\n\t\t\t\\beta\n\t\t{}>{}\n\t\t\t-\\nicefrac{(1+3\\alpha p_{-f,h})}{2}\n\t\t\\),\n\t\tin which case\n\t\t\\[\n\t\t\t\\xi\\coloneqq\\f\n\t\t\\quad\\text{and}\\quad\n\t\t\t\tc\n\t\t\t{}\\coloneqq{}\n\t\t\t\t1+2\\beta+3\\alpha p_{-f,h}>0,\n\t\t\\]\n\t\\item\\label{thm:SD:fsmooth}%\n\t\tor \\(\\f\\) is \\(L_{\\f}\\)-Lipschitz differentiable, \\(h\\) is \\(\\sigma_h\\)-strongly convex, and\n\t\t\\[\n\t\t\tc\n\t\t{}\\coloneqq{}\n\t\t\t(1+\\alpha p_{-f,h})\\sigma_h\n\t\t\t{}-{}\n\t\t\t2\\gamma L_{\\f}\n\t\t{}>{}\n\t\t\t0,\n\t\t\\]\n\t\tin which case \\(\\xi\\coloneqq L_{\\f}\\j\\).\n\t\\end{enumerator}\n\tThen, for \\(\\LL\\) as in \\eqref{eq:LL} the following assertions hold:\n\t\\begin{enumerate}\n\t\\item\\label{thm:SD:leq}%\n\t\t\\begin{subequations}\n\t\t\tFor every \\(x,x^-\\in C\\) and \\(\\bar x\\in\\T(x,x^-)\\),\n\t\t\t\\begin{align}\n\t\t\t\\label{eq:SD}\n\t\t\t\t\\LL(\\bar x,x)\n\t\t\t{}\\leq{} &\n\t\t\t\t\\LL(x,x^-)\n\t\t\t\t{}-{}\n\t\t\t\t\\tfrac{c}{2\\gamma}\\D(\\bar x,x)\n\t\t\t\t{}-{}\n\t\t\t\t\\tfrac{c}{2\\gamma}\\D(x,x^-)\n\t\t\t\\shortintertext{and}\n\t\t\t\\label{eq:L:geq}\n\t\t\t\t\\cost(\\bar x)\n\t\t\t{}\\leq{} &\n\t\t\t\t\\LL(x,x^-)\n\t\t\t\t{}-{}\n\t\t\t\t\\tfrac{c}{\\gamma}\\D(\\bar x,x)\n\t\t\t\t{}-{}\n\t\t\t\t\\tfrac{c}{2\\gamma}\\D(x,x^-).\n\t\t\t\\end{align}\n\t\t\\end{subequations}\n\t\\item\\label{thm:SD:inf}%\n\t\t\\(\\inf\\LL=\\inf\\cost\\), and \\(\\LL\\) is level bounded iff so is \\(\\cost\\).\n\t\\end{enumerate}\n\\end{theorem}","sha256_utf8":"5b26b57b6d5f5ec973201d005f94854e606961b5a82063fc9e3d05920f960682"},{"start":62695,"end":65404,"start_line":1382,"end_line":1425,"raw_tex":"\\begin{theorem}[function value convergence of \\BiFRB]\\label{thm:subseq}%\n\tLet $\\seq{x^k}$ be a sequence generated by \\BiFRB* in the setting of \\cref{thm:SD}.\n\tThen,\n\t\\begin{enumerate}\n\t\\item\\label{thm:subseq:SD}%\n\t\tIt holds that\n\t\t\\begin{equation}\\label{SD inequality}\n\t\t\t\\LL\\left(x^{k+1},x^k\\right)\\leq\\LL\\left(x^k,x^{k-1}\\right)-\\tfrac{c}{2\\gamma}\\D\\left(x^{k+1},x^k\\right)-\\tfrac{c}{2\\gamma}\\D\\left(x^k,x^{k-1}\\right).\n\t\t\\end{equation}\n\t\tThen, $\\sum_{k=0}^\\infty \\D\\left(x^k,x^{k-1}\\right)<\\infty$ and $\\LL(x^{k},x^{k-1})\\to\\varphi^\\star$ as $k\\to\\infty$ for some $\\varphi^\\star\\geq\\inf\\cost$.\n\t\\item\\label{thm:subseq:bounded}%\n\t\tIf \\(\\cost\\) is level bounded, then \\(\\seq{x^k}\\) is bounded.\n\t\\item\\label{thm:subseq:omega}%\n\t\tSuppose that \\cref{ass:h} holds, and let \\(\\omega\\) be the set of limit points of \\(\\seq{x^k}\\).\n\t\tThen, \\(\\varphi\\) is constant on \\(\\omega\\) with value \\(\\varphi^\\star\\), and for every \\(x^\\star\\in\\omega\\) it holds that \\(x^\\star\\in\\T(x^\\star,x^\\star)\\) and \\(0\\in\\hat\\partial\\varphi(x^\\star)\\).\n\t\\end{enumerate}\n\t\\begin{proof}~\n\t\t\\begin{proofitemize}\n\t\t\\item\\ref{thm:subseq:SD}~\n\t\t\tRecall from \\cref{thm:SD} that \\eqref{SD inequality} holds and that \\(\\inf\\LL=\\inf\\cost>-\\infty\\), from which the convergence of \\(\\seq{\\LL(x^{k},x^{k-1})}\\) readily follows.\n\t\t\tIn turn, a telescoping argument on \\eqref{SD inequality} shows that \\(\\sum_{k\\in\\N}\\D(x^k,x^{k-1})<\\infty\\).\n\t\t\\item\\ref{thm:subseq:bounded}~\n\t\t\tIt follows from \\cref{thm:subseq:SD} that \\(\\LL\\left(x^{k+1},x^k\\right)\\leq\\LL(x^0,x^{-1})\\) holds for every \\(k\\).\n\t\t\tThen boundedness of \\(\\seq{x^k}\\) is implied by level boundedness of \\(\\LL\\); see \\cref{thm:SD:inf}.\n\t\t\\item\\ref{thm:subseq:omega}~\n\t\t\tSuppose that a subsequence \\(\\seq{x^{k_j}}[j\\in\\N]\\) converges to a point \\(x^\\star\\), then so do \\(\\seq{x^{k_j\\pm1}}\\) by \\cref{thm:subseq:SD} and \\cite[Prop. 2.2(iii)]{bauschke2003iterating}.\n\t\t\tSince \\(x^{k_j+1}\\in\\T(x^{k_j},x^{k_j-1})\\), by passing to the limit osc of \\(\\T\\) (\\cref{thm:T:osc}) implies that \\(x^\\star\\in\\T(x^\\star,x^\\star)\\).\n\t\t\tInvoking \\cref{thm:T:fix} yields the stationarity condition \\(0\\in\\hat\\partial\\varphi(x^\\star)\\).\n\t\t\tMoreover, by continuity of \\(\\LL\\) one has\n\t\t\t\\[\n\t\t\t\t\\varphi^\\star\n\t\t\t{}\\defeq{}\n\t\t\t\t\\lim_{k\\to\\infty}\\LL\\left(x^k,x^{k-1}\\right)\n\t\t\t{}={}\n\t\t\t\t\\LL(x^\\star,x^\\star)\n\t\t\t{}={}\n\t\t\t\t\\varphi(x^\\star),\n\t\t\t\\]\n\t\t\twhere the last equality follows from \\cref{thm:env:eq}, owing to the inclusion \\(x^\\star\\in\\T(x^\\star,x^\\star)\\) (and the fact that \\(\\D_\\psi(x,x)=0\\) for any differentiable function \\(\\psi\\)).\n\t\t\tFrom the arbitrarity of \\(x^\\star\\in\\omega\\) we conclude that \\(\\varphi\\equiv\\varphi^\\star\\) on \\(\\omega\\).\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}\n\\end{theorem}","sha256_utf8":"97dc4f1e26d3959dfb79aa47538266f7a81d906fd56908e06ffa0eae2657a169"},{"start":70653,"end":72241,"start_line":1510,"end_line":1540,"raw_tex":"\\begin{lemma}[{{\\cite[Lem. 4.4]{wang2022exact}}}]\\label{lemma: Uniformize GCKL}%\n\tLet $\\func{f}{\\R^n}{\\Rinf}$ be proper lsc and let $\\mu\\in\\R$.\n\tLet $\\Omega\\subseteq\\dom\\partial f$ be a nonempty compact set on which $f(x)=\\mu$ for all $x\\in\\Omega$.\n\tThen the following hold:\n\t\\begin{enumerate}\n\t\\item\n\t\tSuppose that $f$ satisfies the pointwise generalized concave KL property at each $x\\in\\Omega$.\n\t\tThen there exist $\\varepsilon>0,\\eta\\in(0,\\infty]$ and $\\psi\\in\\Psi_\\eta$ such that $f$ has the setwise generalized concave KL property on $\\Omega$ with respect to $U=\\Omega_\\varepsilon$, $\\eta$ and $\\psi$.\n\t\\item \\label{uniformize exact modulus}%\n\t\tSet $U=\\Omega_\\varepsilon$ and define $h:(0,\\eta)\\rightarrow\\R_+$ by\n\t\t\\[\n\t\t\th(s)=\\sup\\set{\\dist^{-1}\\left(0,\\partial f(x)\\right)}[{x\\in U\\cap[s\\leq f-\\mu<\\eta]}].\n\t\t\\]\n\t\tThen the function $\\func{\\tilde\\psi}{[0,\\eta)}{\\R_+}$ defined by \\((\\forall 0<t<\\eta)~\\tilde\\psi(t)=\\int_0^th(s)ds\\) with \\(\\tilde{\\psi}(0)=0\\) is well defined and belongs to $\\Psi_\\eta$.\n\t\tThe function $f$ has the setwise generalized concave KL property on $\\Omega$ with respect to $U$, $\\eta$ and $\\tilde\\psi$.\n\t\tMoreover,\n\t\t\\[\n\t\t\t\\tilde{\\psi}\n\t\t{}={}\n\t\t\t\\inf\\left\\{\n\t\t\t\t\\psi\\in\\Psi_\\eta\n\t\t\t\t\\,\\Big|\\,\n\t\t\t\t\\text{\\begin{tabular}{@{}c@{}}%\n\t\t\t\t\t\t\\(\\psi\\) is a concave desingularizing function\\\\\n\t\t\t\t\t\tof $f$ on $\\Omega$ with respect to $U$ and $\\eta$\n\t\t\t\t\\end{tabular}}\n\t\t\t\\right\\}.\n\t\t\\]\n\t\\end{enumerate}\n\tWe say that $\\tilde{\\psi}$ is the exact modulus of the setwise generalized concave KL property of \\(f\\) on \\(\\Omega\\) with respect to $U$ and $\\eta$.\n\\end{lemma}","sha256_utf8":"f9974ac62794acdeb94495d8a13230cc61f7b2bd910998e10cfbc2201918bd14"},{"start":74068,"end":77966,"start_line":1573,"end_line":1633,"raw_tex":"\\begin{proof}\n\tSet \\((\\forall k\\in\\N)\\) $\\delta_{k}=\\F(z^k)$ for simplicity.\n\tThen $\\delta_k=\\LL(x^k,x^{k-1})$, $\\delta_{k}\\to\\varphi^\\star$ decreasingly and $\\dist(x^k,\\omega)\\to0$ as $k\\to\\infty$ by invoking \\cref{thm:subseq}.\n\tAssume without loss of generality that \\((\\forall k\\in\\N)\\) $\\delta_{k}>\\varphi^\\star$, otherwise we would have $(\\exists k_0\\in\\N)$ $x^{k_0}=x^{k_0+1}$ due to \\cref{thm:subseq:SD}, from which the desired result readily follows by simple induction.\n\tThus $(\\exists k_0\\in\\N)$ $(\\forall k\\geq k_0)$ $z^k\\in\\Omega_r\\cap[0<\\F-\\varphi^\\star<\\eta]$.\n\tAppealing to \\cref{thm:subseq:omega} and \\cref{thm:M:eq} yields that $\\F$ is constantly equal to $\\varphi^\\star$ on the compact set $\\omega(z^0)$.\n\tIn turn, all conditions in \\cref{uniformize exact modulus} are satisfied, which implies that for $k\\geq k_0$\n\t\\begin{equation}\\label{formula: KL inequality of F}\n\t\t1\\leq(\\tilde\\psi)'_-\\left(\\delta_{k}-\\varphi^\\star\\right)\\dist\\left(0,\\partial\\F(z^k)\\right).\n\t\\end{equation}\n\tDefine $(\\forall k\\in\\N)$\n\t\\begin{align*}\n\t\tu^k={}\n\t&\n\t\t\\nabla^2\\h(x^k)(x^k-x^{k+1})+\\nabla^2\\f(x^k)(x^{k+1}-x^k)+\\tfrac{c}{2\\gamma}\\bigl(\\nabla h(x^k)-\\nabla h(x^{k-1})\\bigr)\n\t\\\\\n\t&\n\t\t+\\nabla\\f(x^{k-1})-\\nabla\\f(x^k)+\\nabla\\xi(x^k)-\\nabla\\xi(x^{k-1}),\n\t\\\\\n\t\tv^k={}\n\t&\n\t\t\\nabla^2\\f(x^{k-1})(x^k-x^{k+1})+\\tfrac{c}{2\\gamma}\\nabla^2h(x^{k-1})(x^{k-1}-x^k)+\\nabla^2\\xi(x^{k-1})(x^{k-1}-x^k).\n\t\\end{align*}\n\tThen $\\norm{u^k}\\leq(M_1+M_2)\\norm{x^{k+1}-x^k}+(M_1+(\\nicefrac{c}{2\\gamma})M_3+M_4)\\norm{x^k-x^{k-1}}$ and $\\norm{v^k}\\leq M_1\\norm{x^{k+1}-x^k}+((\\nicefrac{c}{2\\gamma})M_3+M_4)\\norm{x^k-x^{k-1}}$, where $M_1=\\sup\\|\\nabla^2\\f(\\mathcal{B})\\|$, $M_2=\\sup\\|\\nabla^2\\h(\\mathcal{B})\\|$, $M_3=\\sup\\norm{\\nabla^2h(\\mathcal{B})}$, and $M_4=\\sup\\norm{\\nabla^2\\xi(\\mathcal{B})}$.\n\tApplying subdifferential calculus to $\\partial\\F(z_k)$ yields that\n\t\\[\n\t\t\\partial\\F(z^k)\n\t{}={}\n\t\t\\bigl(\\partial\\varphi(x^{k+1})+\\nabla\\h(x^{k+1})-\\nabla\\h(x^k)+\\nabla\\f(x^{k})-\\nabla \\f(x^{k-1})\\bigr)\\times\\{u^k\\}\\times\\{v^k\\},\n\t\\]\n\twhich together with \\cref{thm:T:partial} entails that $(0,u^k,v^k)\\in\\partial\\F(z^k)$.\n\tIn turn, summing the aforementioned bounds on $\\norm{u^k}$ and $\\norm{v^k}$ gives\n\t\\begin{equation}\\label{subd F upper bound}\n\t\t\\dist\\left(0,\\partial\\F(z^k)\\right)\\leq M\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})},\n\t\\end{equation}\n\twhere $M=\\max\\set{2M_1+M_2,M_1+(\\nicefrac{c}{\\gamma})M_3+2M_4}$.\n\n\tFinally, we show that $\\seq{x^k}$ is convergent.\n\tFor simplicity, define $(\\forall k,l\\in\\N)$ $\\Delta_{k,l}=\\tilde\\psi\\left(\\delta_{k}-\\varphi^\\star\\right)-\\tilde\\psi\\left(\\delta_{l}-\\varphi^\\star\\right)$.\n\tThen, combining~\\eqref{formula: KL inequality of F} and~\\eqref{subd F upper bound} yields\n\t\\begin{align*}\n\t\t1\n\t&{}\\leq{}\n\t\tM(\\tilde{\\psi})'_-(\\delta_{k}-\\varphi^\\star)\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}\n\t\t{}\\leq{}\n\t\t\\tfrac{M\\Delta_{k,k+1}}{\\delta_k-\\delta_{k+1}}\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}\n\t\\\\\n\t&{}\\leq{}\n\t\t\\frac{2\\gamma M\\Delta_{k,k+1}}{c\\left(\\D(x^{k+1},x^k)+\\D(x^k,x^{k-1})\\right)}\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}\n\t\\\\\n\t&{}\\leq{}\n\t\t\\frac{2\\gamma M\\Delta_{k,k+1}\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}}{c \\sigma\\norm{x^{k+1}-x^k}^2+c \\sigma\\norm{x^k-x^{k-1}}^2}\\leq \\frac{4\\gamma M\\Delta_{k,k+1}}{c \\sigma\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}},\n\t\\end{align*}\n\twhere the second inequality is implied by concavity of $\\tilde\\psi$, the third one follows from \\eqref{SD inequality}, and the fourth one holds because $\\sigma>0$ is the strong convexity modulus of $h$ on $\\mathcal{B}$.\n\tHence,\n\t\\begin{equation}\\label{consecutive gap}\n\t\t\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}\\leq\\tfrac{4\\gamma M}{c\\sigma}\\Delta_{k,k+1}.\n\t\\end{equation} \n\tSumming~\\eqref{consecutive gap} from $k=k_0$ to an arbitrary $l\\geq k_0+1$ and passing $l$ to infinity justifies~\\eqref{finite length}.\n\tA similar procedure shows that $\\seq{x^k}$ is Cauchy, which together with \\cref{thm:subseq:omega} entails the rest of the statement. \n\\end{proof}","sha256_utf8":"26406169647f52eb3d869ad835e01459865a325ce246f4c6420d556399423fce"},{"start":99471,"end":101432,"start_line":2095,"end_line":2130,"raw_tex":"\\begin{lemma}\\label{thm:fconv:general}%\n\tSuppose that \\cref{ass:basic} holds and let \\(\\gamma>0\\) and \\(\\beta\\in\\R\\) be such that \\(\\f\\coloneqq f-\\frac\\beta\\gamma h\\) is a convex function.\n\tThen, for every \\(x,x^-\\in C\\) and \\(\\bar x\\in\\T(x,x^-)\\)\n\t\\begin{align}\\label{eq:decrease:fconv}\n\t\t\\left(\\env+\\D_{\\f}\\right)(\\bar x,x)\n\t{}\\leq{} &\n\t\t\\left(\\env+\\D_{\\f}\\right)(x,x^-)\n\t\t{}-{}\n\t\t\\D_{\\h-2\\f}(\\bar x,x).\n% \t\\shortintertext{%\n% \t\tIf \\(\\h-\\f\\) too is convex, then it also holds that\n% \t}\n% \t\t\\varphi(\\bar x)\n% \t{}\\leq{} &\n% \t\t\\left(\\env+\\D_{\\f}\\right)(x,x^-)\n% \t\t{}-{}\n% \t\t\\D_{\\h-2\\f}(\\bar x,x),\n\t\\end{align}\n\tIf \\(\\h-\\f\\) too is convex, then \\(\\env+\\D_{\\f}\\) has the same infimum of \\(\\cost\\), and is level bounded iff so is \\(\\cost\\).%\n\t\\begin{proof}\n\t\tAll the claimed inequalities follows from \\eqref{eq:descent} together with the fact that \\(\\D_{\\f}\\geq0\\), and that \\(\\D_{\\h-\\f}\\geq0\\) too when \\(\\h-\\f\\) is convex.\n\t\tWhen both \\(\\f\\) and \\(\\h-\\f\\) are convex, \\cref{thm:env:eq} implies that\n\t\t\\begin{equation}\\label{eq:fconv:geq}\n\t\t\t\\env(y,y^-)+\\D_{\\f}(y,y^-)\\geq\\varphi(\\bar y)\\geq\\inf\\cost\n\t\t\\quad\n\t\t\t\\forall y,y^-\\in C,\\ \\bar y\\in\\T(y,y^-),\n\t\t\\end{equation}\n\t\tproving that \\(\\inf(\\env+\\D_{\\f})\\geq\\inf\\cost\\).\n\t\tThe converse inequality follows from \\cref{thm:env:leq} by observing that \\((\\env+\\D_{\\f})(x,x)=\\env(x,x)\\leq\\varphi(x)\\).\n\n\t\tTo conclude, suppose that \\(\\env+\\D_{\\f}\\) is not level bounded, and consider an unbounded sequence \\(\\seq{x_k,x_k^-}\\) such that \\(\\env(x_k,x_k^-)+\\D_{\\f}(x_k,x_k^-)\\leq\\ell\\), for some \\(\\ell\\in\\R\\).\n\t\tThen, it follows from \\eqref{eq:fconv:geq} that \\(\\varphi(\\bar x_k)\\leq\\ell\\), where \\(\\bar x_k\\in\\T(x_k,x_k^-)\\).\n\t\tFrom local boundedness of \\(\\T\\) (\\cref{thm:T:osc}) it follows that \\(\\seq{\\bar x_k}\\) too is unbounded, showing that \\(\\varphi\\) is not level bounded either.\n\t\tThe converse holds by observing that \\((\\env+\\D_{\\f})(x,x)=\\env(x,x)\\leq\\cost(x)\\), cf. \\cref{thm:env:leq}.\n\t\\end{proof}\n\\end{lemma}","sha256_utf8":"309df4d0ba7c2bc3ede6d44f1c972f06cca864d21d7c0d4b85ee4781f9933d3b"},{"start":101434,"end":102926,"start_line":2132,"end_line":2187,"raw_tex":"In the setting of \\cref{thm:SD:fconv}, inequality \\eqref{eq:decrease:fconv} can equivalently be written in terms of \\(\\LL\\) as\n\\begin{align*}\n\t\\LL(\\bar x,x)\n{}\\leq{} &\n\t\\LL(x,x^-)\n\t{}-{}\n\t\\D_{\\h-2\\f-\\frac c\\gamma h}(\\bar x,x)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(\\bar x,x)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(x,x^-)\n\\\\\n{}\\leq{} &\n\t\\LL(x,x^-)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(\\bar x,x)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(x,x^-),\n\\end{align*}\nwhere the second inequality owes to the fact that \\(\\D_{\\h-2\\f-\\frac c\\gamma h}\\geq0\\), since \\(\\h-2\\f-\\frac c\\gamma h\\) is convex, having\n\\[\n\t\\h-2\\f-\\tfrac c\\gamma h\n{}={}\n\t\\tfrac{1+2\\beta-c}{\\gamma}h-3f\n{}={}\n\t\\overbracket*{\\tfrac{1+2\\beta+3\\alpha p_{-f,h}-c}{\\gamma}}^{=0}h+2(-f-\\sigma_{-f,h}h),\n\\]\nthe coefficient of \\(h\\) being null by definition of \\(c\\), and \\(-f-\\sigma_{-f,h}h\\) being convex by definition of the relative weak convexity modulus \\(\\sigma_{-f,h}\\), cf. \\cref{def:relweak}.\nThis proves \\eqref{eq:SD}; inequality \\eqref{eq:L:geq} follows similarly by observing that\n\\[\n\t0\n{}\\leq{}\n\t\\D_{\\h-2\\f-\\frac c\\gamma h}\n{}={}\n\t\\D_{\\h-\\f}-\\D_{\\f}-\\tfrac c\\gamma\\D\n{}\\leq{}\n\t\\D_{\\h-\\f}-\\tfrac c\\gamma\\D,\n\\]\nso that\n\\[\n\t\\LL(x,x^-)\n{}\\overrel{\\ref{thm:env:eq}}{}\n\t\\varphi(\\bar x)\n\t{}+{}\n\t\\overbracket*{\n\t\t\\D_{\\h-\\f}\n\t\t\\vphantom{x^-}\n\t}^{\\geq\\frac c\\gamma\\D}(\\bar x,x)\n\t{}+{}\n\t\\overbracket*{\n\t\t\\D_{\\f}(\\bar x,x^-)\n\t}^{\\geq0}\n\t{}+{}\n\t\\tfrac{c}{2\\gamma}\\D(x,x^-).\n\\]\nIn turn, \\cref{thm:SD:inf} follows from the same arguments as in the proof of \\cref{thm:fconv:general}.\n","sha256_utf8":"95f55c5c502d51ccf9312f87675b8cc417267c3ccd740252d787d9c7ca6e92e2"},{"start":105893,"end":107747,"start_line":2270,"end_line":2323,"raw_tex":"\\begin{lemma}\\label{thm:fsmooth:general}%\n\tAdditionally to \\cref{ass:basic}, suppose that \\(\\f\\) is \\(L_{\\f}\\)-Lipschitz differentiable for some \\(L_{\\f}\\geq0\\).\n\tThen, for every \\(x,x^-\\in C\\) and \\(\\bar x\\in\\T(x,x^-)\\)\n\t\\begin{equation}\\label{eq:decrease:fsmooth}\n\t\t\\left(\\env+\\D_{L_{\\f}\\j}\\right)(\\bar x,x)\n\t{}\\leq{}\n\t\t\\left(\\env+\\D_{L_{\\f}\\j}\\right)(x,x^-)\n\t\t{}-{}\n\t\t\\D_{\\h-2L_{\\f}\\j}(\\bar x,x).\n\t\\end{equation}\n\tIf \\(\\h-L_{\\f}\\j\\) is a convex function, then \\(\\env+\\D_{L_{\\f}\\j}\\) has the same infimum of \\(\\cost\\), and is level bounded iff so is \\(\\cost\\).\n\t\\begin{proof}\n\t\tBy means of the three-point identity, that is, by using \\eqref{eq:M-innprod} in place of \\eqref{eq:M-D}, inequality \\eqref{eq:descent} can equivalently be written as\n\t\t\\begin{align*}\n\t\t\t\\env(\\bar x,x)\n\t\t{}\\leq{}\n\t\t\t\\cost(\\bar x)\n\t\t{}={} &\n\t\t\t\\env(x,x^-)\n\t\t\t{}-{}\n\t\t\t\\D_{\\h}(\\bar x,x)\n\t\t\t{}-{}\n\t\t\t\\innprod*{\\bar x-x}{\\nabla\\f(x)-\\nabla\\f(x^-)}\n\t\t\\shortintertext{%\n\t\t\twhich by using Young's inequality on the inner product and \\(L_{\\f}\\)-Lipschitz differentiability yields\n\t\t}\n\t\t{}\\leq{} &\n\t\t\t\\env(x,x^-)\n\t\t\t{}-{}\n\t\t\t\\D_{\\h}(\\bar x,x)\n\t\t\t{}+{}\n\t\t\t\\tfrac{L_{\\f}}{2}\\|\\bar x-x\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{L_{\\f}}{2}\\|x-x^-\\|^2.\n\t\t\\end{align*}\n\t\tRearranging and using the fact that \\(\\D_{\\j}(x,y)=\\frac12\\|x-y\\|^2\\) yields the claimed inequality.\n\t\t\n\t\tUnder convexity of \\(\\h-L_{\\f}\\j\\), we may draw the same conclusions as in the proof of \\cref{thm:fconv:general} by observing that the same application of Young's inequality above, combined with the expression \\eqref{eq:M-innprod} of \\(\\M\\), gives\n\t\t\\[\n\t\t\t\\env(y,y^-)\n\t\t{}={}\n\t\t\t\\M(\\bar y;y,y^-)\n\t\t{}\\geq{}\n\t\t\t\\varphi(\\bar y)\n\t\t\t{}+{}\n\t\t\t\\D_{\\h}(\\bar y,y)\n\t\t\t{}-{}\n\t\t\t\\tfrac{L_{\\f}}{2}\\|\\bar y-y\\|^2\n\t\t\t{}-{}\n\t\t\t\\tfrac{L_{\\f}}{2}\\|y-y^-\\|^2,\n\t\t\\]\n\t\tholding for any \\(y,y^-\\in C\\) and \\(\\bar y\\in\\T(y,y^-)\\).\n\t\\end{proof}\n\\end{lemma}","sha256_utf8":"e82e5f5d3bb203ad6b4b5eedb97b43e9d369438ee29f95920d56e33216ac7c7d"},{"start":107749,"end":109502,"start_line":2325,"end_line":2406,"raw_tex":"We will pattern the arguments in the previous section, and observe that inequality \\eqref{eq:decrease:fsmooth} can equivalently be written in terms of \\(\\LL\\) as\n\\[\n\t\\LL(\\bar x,x)\n{}\\leq{}\n\t\\LL(x,x^-)\n\t{}-{}\n\t\\overbracket*{\n\t\t\\D_{\\h-2L_{\\f}\\j-\\frac c\\gamma h}\n\t}^{\\geq0}(\\bar x,x)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(\\bar x,x)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(x,x^-).\n\\]\nOnce again, the fact that \\(\\D_{\\h-2L_{\\f}\\j-\\frac c\\gamma h}\\geq0\\) owes to convexity of \\(\\h-2L{\\f}\\j-\\frac c\\gamma h\\), having\n{%\n\\ifsiam\n\t\\mathtight[0.8]%\n\\fi\n\t\\begin{align*}\n\t\t\\h-2L_{\\f}\\j-\\tfrac c\\gamma h\n\t{}={} &\n\t\t\\tfrac{1-c}{\\gamma}h\n\t\t{}-{}\n\t\tf\n\t\t{}-{}\n\t\t2L_{\\f}\\j\n\t\\\\\n\t{}={} &\n\t\t\\def\\myvphantom{\\vphantom{\\tfrac{1+\\gamma\\sigma_{-f,h}-c}{\\gamma}}}\n\t\t\\overbracket[0.5pt]{\n\t\t\t\\myvphantom\n\t\t\t\\tfrac{1+\\gamma\\sigma_{-f,h}-c}{\\gamma}\n\t\t}^{=\\nicefrac{2L_{\\f}}{\\sigma_h}\\geq0}\n\t\t(\\overbracket*{\n\t\t\t\\myvphantom\n\t\t\th-\\sigma_h\\j\n\t\t}^{\\text{convex}})\n\t\t{}+{}\n\t\t(\\overbracket*{\n\t\t\t\\myvphantom\n\t\t\t-f-\\sigma_{-f,h}h\n\t\t}^{\\text{convex}})\n\t\t{}+{}\n\t\t\\bigl(\\overbracket*{\n\t\t\t\\myvphantom\n\t\t\t\\tfrac{1+\\gamma\\sigma_{-f,h}-c}{\\gamma}\\sigma_h\n\t\t\t{}-{}\n\t\t\t2L_{\\f}\n\t\t}^{=0}\n\t\t\\bigr)\\j\n\t\\end{align*}\n}%\naltogether proving \\eqref{eq:SD}.\nSimilarly, inequality \\eqref{eq:L:geq} follows by observing that\n\\(\n\t0\n{}\\leq{}\n\t\\D_{\\h-2\\f-\\frac c\\gamma h}\n{}={}\n\t\\D_{\\h-\\f}-\\D_{\\f}-\\frac c\\gamma\\D\n{}\\leq{}\n\t\\D_{\\h-\\f}-\\frac c\\gamma\\D\n\\),\nso that\n\\[\n\t\\LL(x,x^-)\n{}\\overrel{\\ref{thm:env:eq}}{}\n\t\\varphi(\\bar x)\n\t{}+{}\n\t\\overbracket*{\n\t\t\\D_{\\h-\\f}\n\t\t\\vphantom{x^-}\n\t}^{\\geq\\frac c\\gamma\\D}(\\bar x,x)\n\t{}+{}\n\t\\overbracket*{\n\t\t\\D_{\\f}(\\bar x,x^-)\n\t}^{\\geq0}\n\t{}+{}\n\t\\tfrac{c}{2\\gamma}\\D(x,x^-).\n\\]\nThe assertion of \\cref{thm:SD:inf} once again follows from the same arguments as in the proof of \\cref{thm:fsmooth:general}.\n","sha256_utf8":"874ffe08190c917e1a17b1bbc2beca0c0b8dfe1113ef48e2937519cbc49d7f30"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"镜像惯性算法的全序列收敛","source_units":[{"start":72410,"end":74067,"start_line":1544,"end_line":1572,"raw_tex":"\\begin{theorem}[sequential convergence of \\BiFRB]\\label{thm:sequential convergence}%\n\tSuppose that \\cref{ass:h} holds, and let $\\seq{x^k}$ be a sequence generated by \\BiFRB* in the setting of \\cref{thm:SD}.\n\tDefine $(\\forall k\\in\\N)$~$z^k=(x^{k+1},x^k,x^{k-1})$ and let $\\omega(z^0)$ be the set of limit points of \\((z_k)_{k\\in\\N}\\).\n\tDefine\n\t\\[\n\t\t(\\forall \\omega,x,x^-\\in\\R^n )~\\F(\\omega,x,x^-)=\\M(\\omega;x,x^-)+\\tfrac{c}{2\\gamma}\\D(x,x^-)+D_{\\xi}(x,x^-).\n\t\\]\n\tAssume in addition the following:\n\t\\begin{enumeratass}\n\t\\item\\label{bounded}%\n\t\t\\(\\varphi\\) is level bounded.\n\t\\item\\label{c2 and pd}%\n\t\t$f,h$ are twice continuously differentiable and $\\nabla^2h$ is positive definite.\n\t\\item\\label{GCKL}%\n\t\t$(\\exists r>0)~(\\exists\\eta\\in[0,\\infty))$ $\\F$ satisfies the generalized concave KL property on $\\Omega=\\omega(z^0)$ with respect to $U=\\Omega_r$ and $\\eta$.\n\t\\end{enumeratass}\n\tThen \\(\\sum_{k=0}^\\infty\\norm{x^{k+1}-x^k}<\\infty\\) and there exists $x^\\star$ with $0\\in\\hat\\partial\\varphi(x^{\\star})$ such that \\(x^k\\to x^\\star\\) as \\(k\\to\\infty\\).\n\tTo be specific, there exists $k_0\\in\\N$ such that for all $l\\geq k_0+1$\n\t\\begin{equation}\\label{finite length}\n\t\t\\sum_{k=0}^\\infty\\norm{x^{k+1}-x^k}\n\t{}\\leq{}\n\t\t\\sum_{k=0}^l\\norm{x^{k+1}-x^k}\n\t\t{}+{}\n\t\t\\tfrac{4\\gamma M}{c\\sigma}\n\t\t\\tilde\\psi\\left(\\LL(x^{l},x^{l-1})-\\varphi^\\star\\right),\n\t\\end{equation}\n\twhere $M>0$ is some constant, $\\sigma>0$ is the strong convexity modulus of $h$ on $\\mathcal{B}$, the closed ball in which $\\seq{x^k}$ lies,\n\tand $\\tilde{\\psi}$ is the exact modulus of the generalized concave KL property associated with $\\F$ produced by \\cref{uniformize exact modulus}.\n\\end{theorem}","sha256_utf8":"696f875361a6e897347d7125b770e4152035b32bb003248cb980f4cfab585d4b"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"全序列收敛的完整证明","source_units":[{"start":74068,"end":77966,"start_line":1573,"end_line":1633,"raw_tex":"\\begin{proof}\n\tSet \\((\\forall k\\in\\N)\\) $\\delta_{k}=\\F(z^k)$ for simplicity.\n\tThen $\\delta_k=\\LL(x^k,x^{k-1})$, $\\delta_{k}\\to\\varphi^\\star$ decreasingly and $\\dist(x^k,\\omega)\\to0$ as $k\\to\\infty$ by invoking \\cref{thm:subseq}.\n\tAssume without loss of generality that \\((\\forall k\\in\\N)\\) $\\delta_{k}>\\varphi^\\star$, otherwise we would have $(\\exists k_0\\in\\N)$ $x^{k_0}=x^{k_0+1}$ due to \\cref{thm:subseq:SD}, from which the desired result readily follows by simple induction.\n\tThus $(\\exists k_0\\in\\N)$ $(\\forall k\\geq k_0)$ $z^k\\in\\Omega_r\\cap[0<\\F-\\varphi^\\star<\\eta]$.\n\tAppealing to \\cref{thm:subseq:omega} and \\cref{thm:M:eq} yields that $\\F$ is constantly equal to $\\varphi^\\star$ on the compact set $\\omega(z^0)$.\n\tIn turn, all conditions in \\cref{uniformize exact modulus} are satisfied, which implies that for $k\\geq k_0$\n\t\\begin{equation}\\label{formula: KL inequality of F}\n\t\t1\\leq(\\tilde\\psi)'_-\\left(\\delta_{k}-\\varphi^\\star\\right)\\dist\\left(0,\\partial\\F(z^k)\\right).\n\t\\end{equation}\n\tDefine $(\\forall k\\in\\N)$\n\t\\begin{align*}\n\t\tu^k={}\n\t&\n\t\t\\nabla^2\\h(x^k)(x^k-x^{k+1})+\\nabla^2\\f(x^k)(x^{k+1}-x^k)+\\tfrac{c}{2\\gamma}\\bigl(\\nabla h(x^k)-\\nabla h(x^{k-1})\\bigr)\n\t\\\\\n\t&\n\t\t+\\nabla\\f(x^{k-1})-\\nabla\\f(x^k)+\\nabla\\xi(x^k)-\\nabla\\xi(x^{k-1}),\n\t\\\\\n\t\tv^k={}\n\t&\n\t\t\\nabla^2\\f(x^{k-1})(x^k-x^{k+1})+\\tfrac{c}{2\\gamma}\\nabla^2h(x^{k-1})(x^{k-1}-x^k)+\\nabla^2\\xi(x^{k-1})(x^{k-1}-x^k).\n\t\\end{align*}\n\tThen $\\norm{u^k}\\leq(M_1+M_2)\\norm{x^{k+1}-x^k}+(M_1+(\\nicefrac{c}{2\\gamma})M_3+M_4)\\norm{x^k-x^{k-1}}$ and $\\norm{v^k}\\leq M_1\\norm{x^{k+1}-x^k}+((\\nicefrac{c}{2\\gamma})M_3+M_4)\\norm{x^k-x^{k-1}}$, where $M_1=\\sup\\|\\nabla^2\\f(\\mathcal{B})\\|$, $M_2=\\sup\\|\\nabla^2\\h(\\mathcal{B})\\|$, $M_3=\\sup\\norm{\\nabla^2h(\\mathcal{B})}$, and $M_4=\\sup\\norm{\\nabla^2\\xi(\\mathcal{B})}$.\n\tApplying subdifferential calculus to $\\partial\\F(z_k)$ yields that\n\t\\[\n\t\t\\partial\\F(z^k)\n\t{}={}\n\t\t\\bigl(\\partial\\varphi(x^{k+1})+\\nabla\\h(x^{k+1})-\\nabla\\h(x^k)+\\nabla\\f(x^{k})-\\nabla \\f(x^{k-1})\\bigr)\\times\\{u^k\\}\\times\\{v^k\\},\n\t\\]\n\twhich together with \\cref{thm:T:partial} entails that $(0,u^k,v^k)\\in\\partial\\F(z^k)$.\n\tIn turn, summing the aforementioned bounds on $\\norm{u^k}$ and $\\norm{v^k}$ gives\n\t\\begin{equation}\\label{subd F upper bound}\n\t\t\\dist\\left(0,\\partial\\F(z^k)\\right)\\leq M\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})},\n\t\\end{equation}\n\twhere $M=\\max\\set{2M_1+M_2,M_1+(\\nicefrac{c}{\\gamma})M_3+2M_4}$.\n\n\tFinally, we show that $\\seq{x^k}$ is convergent.\n\tFor simplicity, define $(\\forall k,l\\in\\N)$ $\\Delta_{k,l}=\\tilde\\psi\\left(\\delta_{k}-\\varphi^\\star\\right)-\\tilde\\psi\\left(\\delta_{l}-\\varphi^\\star\\right)$.\n\tThen, combining~\\eqref{formula: KL inequality of F} and~\\eqref{subd F upper bound} yields\n\t\\begin{align*}\n\t\t1\n\t&{}\\leq{}\n\t\tM(\\tilde{\\psi})'_-(\\delta_{k}-\\varphi^\\star)\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}\n\t\t{}\\leq{}\n\t\t\\tfrac{M\\Delta_{k,k+1}}{\\delta_k-\\delta_{k+1}}\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}\n\t\\\\\n\t&{}\\leq{}\n\t\t\\frac{2\\gamma M\\Delta_{k,k+1}}{c\\left(\\D(x^{k+1},x^k)+\\D(x^k,x^{k-1})\\right)}\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}\n\t\\\\\n\t&{}\\leq{}\n\t\t\\frac{2\\gamma M\\Delta_{k,k+1}\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}}{c \\sigma\\norm{x^{k+1}-x^k}^2+c \\sigma\\norm{x^k-x^{k-1}}^2}\\leq \\frac{4\\gamma M\\Delta_{k,k+1}}{c \\sigma\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}},\n\t\\end{align*}\n\twhere the second inequality is implied by concavity of $\\tilde\\psi$, the third one follows from \\eqref{SD inequality}, and the fourth one holds because $\\sigma>0$ is the strong convexity modulus of $h$ on $\\mathcal{B}$.\n\tHence,\n\t\\begin{equation}\\label{consecutive gap}\n\t\t\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}\\leq\\tfrac{4\\gamma M}{c\\sigma}\\Delta_{k,k+1}.\n\t\\end{equation} \n\tSumming~\\eqref{consecutive gap} from $k=k_0$ to an arbitrary $l\\geq k_0+1$ and passing $l$ to infinity justifies~\\eqref{finite length}.\n\tA similar procedure shows that $\\seq{x^k}$ is Cauchy, which together with \\cref{thm:subseq:omega} entails the rest of the statement. \n\\end{proof}","sha256_utf8":"26406169647f52eb3d869ad835e01459865a325ce246f4c6420d556399423fce"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"势函数极限与有限终止情形","source_units":[{"start":74068,"end":74547,"start_line":1573,"end_line":1576,"raw_tex":"\\begin{proof}\n\tSet \\((\\forall k\\in\\N)\\) $\\delta_{k}=\\F(z^k)$ for simplicity.\n\tThen $\\delta_k=\\LL(x^k,x^{k-1})$, $\\delta_{k}\\to\\varphi^\\star$ decreasingly and $\\dist(x^k,\\omega)\\to0$ as $k\\to\\infty$ by invoking \\cref{thm:subseq}.\n\tAssume without loss of generality that \\((\\forall k\\in\\N)\\) $\\delta_{k}>\\varphi^\\star$, otherwise we would have $(\\exists k_0\\in\\N)$ $x^{k_0}=x^{k_0+1}$ due to \\cref{thm:subseq:SD}, from which the desired result readily follows by simple induction.\n","sha256_utf8":"2e06c959873d9796369303d20ea560746bb90acda802eb2bf933f3cd82bfa31e"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"聚点集上的精确KL不等式","source_units":[{"start":74547,"end":75066,"start_line":1577,"end_line":1582,"raw_tex":"\tThus $(\\exists k_0\\in\\N)$ $(\\forall k\\geq k_0)$ $z^k\\in\\Omega_r\\cap[0<\\F-\\varphi^\\star<\\eta]$.\n\tAppealing to \\cref{thm:subseq:omega} and \\cref{thm:M:eq} yields that $\\F$ is constantly equal to $\\varphi^\\star$ on the compact set $\\omega(z^0)$.\n\tIn turn, all conditions in \\cref{uniformize exact modulus} are satisfied, which implies that for $k\\geq k_0$\n\t\\begin{equation}\\label{formula: KL inequality of F}\n\t\t1\\leq(\\tilde\\psi)'_-\\left(\\delta_{k}-\\varphi^\\star\\right)\\dist\\left(0,\\partial\\F(z^k)\\right).\n\t\\end{equation}\n","sha256_utf8":"d7883e8a1d3cd59443165067e998de96ea8ee407dec90c799a7dfaa566624ca5"}]},{"id":"S005","semantic_type":"proof_step","kind":"proof_step","title":"构造偏导分量并估计范数","source_units":[{"start":75066,"end":75842,"start_line":1583,"end_line":1596,"raw_tex":"\tDefine $(\\forall k\\in\\N)$\n\t\\begin{align*}\n\t\tu^k={}\n\t&\n\t\t\\nabla^2\\h(x^k)(x^k-x^{k+1})+\\nabla^2\\f(x^k)(x^{k+1}-x^k)+\\tfrac{c}{2\\gamma}\\bigl(\\nabla h(x^k)-\\nabla h(x^{k-1})\\bigr)\n\t\\\\\n\t&\n\t\t+\\nabla\\f(x^{k-1})-\\nabla\\f(x^k)+\\nabla\\xi(x^k)-\\nabla\\xi(x^{k-1}),\n\t\\\\\n\t\tv^k={}\n\t&\n\t\t\\nabla^2\\f(x^{k-1})(x^k-x^{k+1})+\\tfrac{c}{2\\gamma}\\nabla^2h(x^{k-1})(x^{k-1}-x^k)+\\nabla^2\\xi(x^{k-1})(x^{k-1}-x^k).\n\t\\end{align*}\n\tThen $\\norm{u^k}\\leq(M_1+M_2)\\norm{x^{k+1}-x^k}+(M_1+(\\nicefrac{c}{2\\gamma})M_3+M_4)\\norm{x^k-x^{k-1}}$ and $\\norm{v^k}\\leq M_1\\norm{x^{k+1}-x^k}+((\\nicefrac{c}{2\\gamma})M_3+M_4)\\norm{x^k-x^{k-1}}$, where $M_1=\\sup\\|\\nabla^2\\f(\\mathcal{B})\\|$, $M_2=\\sup\\|\\nabla^2\\h(\\mathcal{B})\\|$, $M_3=\\sup\\norm{\\nabla^2h(\\mathcal{B})}$, and $M_4=\\sup\\norm{\\nabla^2\\xi(\\mathcal{B})}$.\n","sha256_utf8":"5498c328e825b725478af7dd14deb16fb9b6f267f3e02397a31fd7d549087f14"}]},{"id":"S006","semantic_type":"proof_step","kind":"proof_step","title":"最优性给出乘积空间次梯度","source_units":[{"start":75842,"end":76164,"start_line":1597,"end_line":1603,"raw_tex":"\tApplying subdifferential calculus to $\\partial\\F(z_k)$ yields that\n\t\\[\n\t\t\\partial\\F(z^k)\n\t{}={}\n\t\t\\bigl(\\partial\\varphi(x^{k+1})+\\nabla\\h(x^{k+1})-\\nabla\\h(x^k)+\\nabla\\f(x^{k})-\\nabla \\f(x^{k-1})\\bigr)\\times\\{u^k\\}\\times\\{v^k\\},\n\t\\]\n\twhich together with \\cref{thm:T:partial} entails that $(0,u^k,v^k)\\in\\partial\\F(z^k)$.\n","sha256_utf8":"56494b0ba5dfc7a2c1e60d2647559665ce1fc8c0b31d02a716cebf879c0616cb"}]},{"id":"S007","semantic_type":"proof_step","kind":"proof_step","title":"建立相对误差上界","source_units":[{"start":76164,"end":76451,"start_line":1604,"end_line":1608,"raw_tex":"\tIn turn, summing the aforementioned bounds on $\\norm{u^k}$ and $\\norm{v^k}$ gives\n\t\\begin{equation}\\label{subd F upper bound}\n\t\t\\dist\\left(0,\\partial\\F(z^k)\\right)\\leq M\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})},\n\t\\end{equation}\n\twhere $M=\\max\\set{2M_1+M_2,M_1+(\\nicefrac{c}{\\gamma})M_3+2M_4}$.\n","sha256_utf8":"4f840bcfb254d5ce5e48a063713891313c23b6e86b60b70d58fb236767043e6e"}]},{"id":"S008","semantic_type":"proof_step","kind":"proof_step","title":"结合KL凹性与充分下降","source_units":[{"start":76452,"end":77536,"start_line":1610,"end_line":1626,"raw_tex":"\tFinally, we show that $\\seq{x^k}$ is convergent.\n\tFor simplicity, define $(\\forall k,l\\in\\N)$ $\\Delta_{k,l}=\\tilde\\psi\\left(\\delta_{k}-\\varphi^\\star\\right)-\\tilde\\psi\\left(\\delta_{l}-\\varphi^\\star\\right)$.\n\tThen, combining~\\eqref{formula: KL inequality of F} and~\\eqref{subd F upper bound} yields\n\t\\begin{align*}\n\t\t1\n\t&{}\\leq{}\n\t\tM(\\tilde{\\psi})'_-(\\delta_{k}-\\varphi^\\star)\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}\n\t\t{}\\leq{}\n\t\t\\tfrac{M\\Delta_{k,k+1}}{\\delta_k-\\delta_{k+1}}\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}\n\t\\\\\n\t&{}\\leq{}\n\t\t\\frac{2\\gamma M\\Delta_{k,k+1}}{c\\left(\\D(x^{k+1},x^k)+\\D(x^k,x^{k-1})\\right)}\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}\n\t\\\\\n\t&{}\\leq{}\n\t\t\\frac{2\\gamma M\\Delta_{k,k+1}\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}}{c \\sigma\\norm{x^{k+1}-x^k}^2+c \\sigma\\norm{x^k-x^{k-1}}^2}\\leq \\frac{4\\gamma M\\Delta_{k,k+1}}{c \\sigma\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}},\n\t\\end{align*}\n\twhere the second inequality is implied by concavity of $\\tilde\\psi$, the third one follows from \\eqref{SD inequality}, and the fourth one holds because $\\sigma>0$ is the strong convexity modulus of $h$ on $\\mathcal{B}$.\n","sha256_utf8":"e2d82d97eb6f759f51091eaf23cb80167d4977620e43ae4189fe8b7f7f666e65"}]},{"id":"S009","semantic_type":"proof_step","kind":"proof_step","title":"获得相邻步长的可求和控制","source_units":[{"start":77536,"end":77683,"start_line":1627,"end_line":1630,"raw_tex":"\tHence,\n\t\\begin{equation}\\label{consecutive gap}\n\t\t\\norm*{(x^{k+1}-x^k,x^k-x^{k-1})}\\leq\\tfrac{4\\gamma M}{c\\sigma}\\Delta_{k,k+1}.\n\t\\end{equation} \n","sha256_utf8":"1a8e19890c60a316f7110655dc2c48fc56ccba4267687b47bc89cbcfb4b21e44"}]},{"id":"S010","semantic_type":"proof_step","kind":"proof_step","title":"有限长度与驻点收敛","source_units":[{"start":77683,"end":77966,"start_line":1631,"end_line":1633,"raw_tex":"\tSumming~\\eqref{consecutive gap} from $k=k_0$ to an arbitrary $l\\geq k_0+1$ and passing $l$ to infinity justifies~\\eqref{finite length}.\n\tA similar procedure shows that $\\seq{x^k}$ is Cauchy, which together with \\cref{thm:subseq:omega} entails the rest of the statement. \n\\end{proof}","sha256_utf8":"ba67d49c49c18f013a719a5dfd5771ad2098ea4d1555b754017de925c636d065"}]},{"id":"S011","semantic_type":"problem_setting","kind":"problem_setting","title":"复合约束优化问题","source_units":[{"start":9372,"end":9505,"start_line":296,"end_line":301,"raw_tex":"\\begin{equation*}\n\t\\tag{P}\\label{eq:P}\n\t\\minimize_{x\\in\\R^n}\\varphi(x)\\coloneqq f(x)+g(x)\n\\quad\n\t\\stt x\\in\\closure C,\n\\end{equation*}","sha256_utf8":"cf1da1dc084695e8506f730792dfd28d6ad6f62dabe8b2cf17f78d41384566a3"},{"start":9506,"end":10003,"start_line":302,"end_line":303,"raw_tex":"where \\(C\\) is a nonempty open and convex set with closure \\(\\closure C\\),  \\(\\func{f}{\\R^n}{\\Rinf\\coloneqq\\R\\cup\\set{\\pm\\infty}}\\) is differentiable on $\\inte\\dom f\\neq\\emptyset$, and \\(\\func{g}{\\R^n}{\\Rinf}\\) is proper and lower semicontinuous (lsc).\nFor notational brevity, we define \\(\\cost\\coloneqq\\varphi+\\indicator_{\\closure C}\\) with \\(\\indicator_X\\) denoting the indicator function of set \\(X\\subseteq\\R^n\\), namely such that \\(\\indicator_X(x)=0\\) if \\(x\\in X\\) and \\(\\infty\\) otherwise.\n","sha256_utf8":"24dd8170f84239d7bcc4f5b1a9381052e553d6bc2673ac0dcfe31be9f785bc3d"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"勒让德核与Bregman距离","source_units":[{"start":15224,"end":15924,"start_line":348,"end_line":360,"raw_tex":"\t\t\tThroughout, we fix a Legendre kernel \\(\\func{h}{\\R^n}{\\Rinf}\\) with \\(\\dom\\nabla h=\\interior\\dom h=C\\),\nnamely, a proper, lsc, and strictly convex function  that is 1-coercive and \\DEF{essentially smooth}, \\ie such that \\(\\|\\nabla h(x_k)\\|\\to\\infty\\) for every sequence \\(\\seq{x_k}\\subset C\\) converging to a boundary point of \\(C\\).\nWe will consider the following iterative scheme for addressing problem \\eqref{eq:P},\nwhere \\(\\func{\\D}{\\R^n\\times\\R^n}{\\Rinf}\\) denotes the \\DEF{Bregman distance} induced by \\(h\\), defined as\n\\begin{equation}\\label{eq:D}\n\t\\D(x,y)\n{}\\coloneqq{}\n\t\\begin{ifcases}\n\t\th(x)-h(y)-\\innprod{\\nabla h(y)}{x-y} & y\\in C,\n\t\\\\\n\t\t\\infty\\otherwise.\n\t\\end{ifcases}\n\\end{equation}","sha256_utf8":"ee78a8aa51f82edf14fccd5c866a8b9cd89c0339f6fd916943ed9b05a585ed17"}]},{"id":"S013","semantic_type":"algorithm","kind":"algorithm","title":"镜像惯性前向反射后向迭代","source_units":[{"start":15926,"end":16758,"start_line":362,"end_line":400,"raw_tex":"\\begin{algorithm}\n\t\\caption{Mirror inertial forward-reflected-backward (\\protect\\BiFRB)}%\n\t\\label{alg:BiFRB}%\n\t\\begin{algorithmic}[1]%\n\\setlength{\\itemsep}{1ex}%\n\\item[%\n\tChoose \\(x^{-1},x^0\\in C\\),~\n\tinertial parameter \\(\\beta\\in\\R\\),~ and\n\tstepsize \\(\\gamma>0\\)%\n]%\n\\item[%\n\tIterate for \\(k=0,1,\\ldots\\) until a termination criterion is met (cf. \\cref{thm:termination})\n]%\n\\State\\label{state:BiFRB:y}%\n\tSet \\(y^k\\) such that\n\t\\(\n\t\t\\nabla h(y^k)\n\t{}={}\n\t\t\\nabla h(x^k)-\\gamma\\bigl(\\nabla f(x^k)-\\nabla f(x^{k-1})\\bigr)\n\t\\)%\n\\State\\label{state:BiFRB:x}%\n\tChoose\n\t\\(\n\t\\ifsiam\n\t\t\\mathtight\n\t\\fi\n\t\tx^{k+1}\n\t{}\\in{}\n\t\t\\argmin\\limits_{w\\in\\R^n}\\set{\n\t\t\tg(w)\n\t\t\t{}+{}\n\t\t\t\\innprod*{w}{\\nabla f(x^k)-\\tfrac\\beta\\gamma\\bigl(\\nabla h(x^k)-\\nabla h(x^{k-1})\\bigr)}\n\t\t\t{}+{}\n\t\t\t\\tfrac1\\gamma\\D(w,y^k)\n\t\t}\n\t\\)%\n\\end{algorithmic}\n\n\\end{algorithm}","sha256_utf8":"037f93ccc5846059cd02f0ca2948b78d4187e2f58746ac4ade631c7bf411fe81"}]},{"id":"S014","semantic_type":"assumption","kind":"assumption","title":"目标函数与子问题基本假设","source_units":[{"start":16904,"end":17384,"start_line":405,"end_line":417,"raw_tex":"\\begin{assumption}\\label{ass:basic}%\n\tThe following hold in problem \\eqref{eq:P}:\n\t\\begin{enumeratass}\n\t\\item\\label{ass:f}%\n\t\t\\(\\func{f}{\\R^n}{\\Rinf}\\) is smooth relative to \\(h\\) (see \\cref{sec:relsmooth}).\n\t\\item\\label{ass:g}%\n\t\t\\(\\func{g}{\\R^n}{\\Rinf}\\) is proper and lsc.\n\t\\item\\label{ass:phi}%\n\t\t\\(\\inf\\cost>-\\infty\\).\n\t\\item\\label{ass:T}%\n\t\tFor any \\(v\\in\\R^n\\) and \\(\\gamma>0\\), \\(\\argmin\\set{\\gamma g+h-\\innprod{v}{\\cdot}}\\subseteq C\\).\n\t\\end{enumeratass}\n\\end{assumption}","sha256_utf8":"25bdcab7c6c4e8833a336d4ffdf7ba4ca7569bddcdb71ed01268c75147b9702d"}]},{"id":"S015","semantic_type":"notation","kind":"notation","title":"距离与水平有界等记号","source_units":[{"start":19007,"end":20268,"start_line":439,"end_line":449,"raw_tex":"\t\t\tThe extended-real line is denoted by \\(\\Rinf\\coloneqq\\R\\cup\\set{\\pm\\infty}\\).\nThe positive and negative part of \\(r\\in\\R\\) are respectively defined as \\([r]_+\\coloneqq\\max\\set{0,r}\\) and \\([r]_-\\coloneqq\\max\\set{0,-r}\\), so that \\(r=[r]_+-[r]_-\\).\n\nThe distance of a point \\(x\\in\\R^n\\) to a nonempty set \\(S\\subseteq\\R^n\\) is given by \\(\\dist(x,S)=\\inf_{z\\in S} \\|z-x\\|\\).\nThe interior, closure, and boundary of \\(S\\) are respectively denoted as \\(\\interior S\\), \\(\\overline S\\), and \\(\\boundary S=\\overline S\\setminus\\interior S\\).\nThe \\DEF{indicator function} of \\(S\\) is \\(\\func{\\indicator_S}{\\R^n}{\\Rinf}\\) defined as \\(\\indicator_S(x)=0\\) if \\(x\\in S\\) and \\(\\infty\\) otherwise.\n\nA function \\(\\func{f}{\\R^n}{\\Rinf}\\) is \\DEF{proper} if \\(f\\not\\equiv\\infty\\) and \\(f>-\\infty\\), in which case its \\DEF{domain} is defined as the set \\(\\dom f\\coloneqq\\set{x\\in\\R^n}[f(x)<\\infty]\\).\nFor \\(\\alpha\\in\\R\\), \\([f\\leq\\alpha]\\coloneqq\\set{x\\in\\R^n}[f(x)\\leq\\alpha]\\) denotes the \\(\\alpha\\)-sublevel set of \\(f\\);\n\t\\([\\alpha\\leq f\\leq\\beta]\\) with \\(\\alpha,\\beta\\in\\R\\) is defined accordingly. We say that \\(f\\) is \\DEF{level bounded} (or \\DEF{coercive}) if \\(\\liminf_{\\|x\\|\\to\\infty}f(x)=\\infty\\), and \\(1\\)-coercive if\n\\(\\lim_{\\|x\\|\\to\\infty}f(x)/\\|x\\|=\\infty\\).\n","sha256_utf8":"4031c4e000fb8c005fb3e52d3dfe8df1e874a683d3467bf9cca7289a49c0f2ad"}]},{"id":"S016","semantic_type":"definition","kind":"definition","title":"次微分与光滑映射记号","source_units":[{"start":20731,"end":21751,"start_line":454,"end_line":485,"raw_tex":"Given \\(x\\in\\dom f\\), \\(\\partial f(x)\\) denotes the \\DEF{Mordukhovich (limiting) subdifferential} of \\(f\\) at \\(x\\), given by\n\\[\n\t\\partial f(x)\n{}\\coloneqq{}\n\t\\set{v\\in\\R^n}[\n\t\t\\exists\\seq{x^k,v^k}~\\text{s.t.}~ x^k\\to x,~f(x^k)\\to f(x),~\n\t\t\\hat\\partial f(x^k)\\ni v^k\\to v\n\t],\n\\]\nand \\(\\hat\\partial f(x)\\) is the set of \\DEF{regular subgradients} of \\(f\\) at \\(x\\), namely vectors \\(v\\in\\R^n\\) such that\n\\(\n\t\\liminf_{\\limsubstack{z&\\to&x\\\\z&\\neq&x}}{\n\t\t\\frac{\n\t\t\tf(z)-f(x)-\\innprod{v}{z-x}\n\t\t}{\n\t\t\t\\|z-x\\|\n\t\t}\n\t}\n{}\\geq{}\n\t0\n\\);\nsee, e.g.,~\\cite{rockafellar2011variational,mordukhovich2018variational}.\n\\(\\C^k(\\mathcal U)\\) is the set of functions \\(\\mathcal U\\to\\R\\) which are \\(k\\) times continuously differentiable.\nWe write \\(\\C^k\\) if \\(\\mathcal U\\) is clear from context.\nThe notation \\(\\ffunc{T}{\\R^n}{\\R^n}\\) indicates a set-valued mapping, whose \\DEF{domain} and \\DEF{graph} are respectively defined as\n\\(\n\t\\dom T=\\set{x\\in\\R^n}[T(x)\\neq\\emptyset]\n\\)\nand\n\\(\n\t\\graph T=\\set{(x,y)\\in\\R^n\\times\\R^n}[y\\in T(x)]\n\\).\n","sha256_utf8":"65b0b231e8816c5c70d00c588b98c73d128683bbd0ef01122a501a1264f8eef5"}]},{"id":"S017","semantic_type":"notation","kind":"notation","title":"扩展实数运算约定","source_units":[{"start":21830,"end":22125,"start_line":489,"end_line":490,"raw_tex":"\t\t\tIn the following definition, as well as throughout the entire paper, we follow the ex\\-tend\\-ed-real convention \\(\\infty-\\infty=\\infty\\) to resolve possible ill definitions of difference of extended-real--valued functions.\nSimilarly, we will adopt the convention that \\(\\nicefrac10=\\infty\\).\n","sha256_utf8":"2d073ee11ff207ac4a1b5b821efad13d5596a66e129c441150571c07863b006f"}]},{"id":"S018","semantic_type":"definition","kind":"definition","title":"相对光滑性","source_units":[{"start":22126,"end":22561,"start_line":492,"end_line":495,"raw_tex":"\\begin{definition}[relative smoothness]\\label{def:relsmooth}%\n\tWe say that an lsc function \\(\\func{f}{\\R^n}{\\Rinf}\\) is smooth relative to \\(h\\) if \\(\\dom f\\supseteq\\dom h\\) and there exists a constant \\(L_{f,h}\\geq0\\) such that \\(L_{f,h}h\\pm f\\) are convex functions  on \\(\\inte\\dom h\\).\n\tWe may alternatively say that \\(f\\) is \\(L_{f,h}\\)-smooth relative to \\(h\\) to make the smoothness modulus \\(L_{f,h}\\) explicit.\n\\end{definition}","sha256_utf8":"06a675bc422b7340c80f233bc789dc9a897a5a6ebc254ea45aa06052efcc46b0"}]},{"id":"S019","semantic_type":"definition","kind":"definition","title":"相对弱凸性","source_units":[{"start":23818,"end":24260,"start_line":508,"end_line":511,"raw_tex":"\\begin{definition}[relative weak convexity]\\label{def:relweak}%\n\tWe say that an lsc function \\(\\func{f}{\\R^n}{\\Rinf}\\) is weakly convex relative to \\(h\\) if there exists a (possibly negative) constant \\(\\sigma_{f,h}\\in\\R\\) such that \\(f-\\sigma_{f,h}h\\) is a convex function.\n\tWe may alternatively say that \\(f\\) is \\(\\sigma_{f,h}\\)-weakly convex relative to \\(h\\) to make the weak convexity modulus \\(\\sigma_{f,h}\\) explicit.\n\\end{definition}","sha256_utf8":"6fbf38b6f3dd85872864aef85bb3e65fd2e3e99ba39393ae1b6e26d1c62d21f5"}]},{"id":"S020","semantic_type":"notation","kind":"notation","title":"单双侧模量及其归一化","source_units":[{"start":24263,"end":25252,"start_line":513,"end_line":528,"raw_tex":"Note that relative smoothness of \\(f\\) is equivalent to the relative weak convexity of \\(\\pm f\\).\nIndeed, if \\(f\\) is \\(L_{f,h}\\)-smooth relative to \\(h\\), then both \\(f\\) and \\(-f\\) are \\((-L_{f,h})\\)-weakly convex.\nConversely, if \\(f\\) and \\(-f\\) are \\(\\sigma_{f,h}\\)- and \\(\\sigma_{-f,h}\\)-weakly convex relative to \\(h\\), respectively, then \\(f\\) (as well as \\(-f\\)) is \\(L_{f,h}\\)-smooth relative to \\(h\\) with\n\\begin{equation}\\label{eq:Lf}\n\tL_{f,h}=\\max\\set{|\\sigma_{f,h}|,|\\sigma_{-f,h}|}.\n\\end{equation}\nThe relative weak convexity moduli \\(\\sigma_{\\pm f,h}\\) will be henceforth adopted when referring to \\cref{ass:f}.\nIt will be convenient to \\emph{normalize} these quantities into pure numbers\n\\begin{equation}\\label{eq:pf}\n\tp_{\\pm f,h}\n{}\\coloneqq{}\n\t\\tfrac{\\sigma_{\\pm f,h}}{L_{f,h}}\n{}\\in{}\n\t[-1,1].\n\\end{equation}\nTo make all definitions and implications well posed, we will ignore the uninsteresting case in which \\(f\\) is affine, in which case \\(L_{f,h}=0\\) for any \\(h\\).\n","sha256_utf8":"4c706813a9aadf596bad592e9a84099fe584af7077dc862272d4880740a3c4f6"}]},{"id":"S021","semantic_type":"definition","kind":"definition","title":"辅助核与广义Bregman记号","source_units":[{"start":32640,"end":32758,"start_line":668,"end_line":672,"raw_tex":"\\begin{equation}\n\t\\h\\coloneqq\\tfrac1\\gamma h-f\n\\quad\\text{and}\\quad\n\t\\f\\coloneqq f-\\tfrac\\beta\\gamma h,\n\\end{equation}","sha256_utf8":"560dc6c67e69e58cb31570d1f9dbac1489079262e2677d20b597ed629a0baecc"},{"start":33090,"end":33620,"start_line":681,"end_line":682,"raw_tex":"\tWe will also (ab)use the notation \\(\\D_\\psi\\) of the Bregman distance for functions \\(\\func{\\psi}{\\R^n}{\\Rinf}\\) differentiable on \\(C\\) that are not necessarily convex, thereby possibly having \\(\\D_\\psi\\not\\geq0\\).\n\tThis notational abuse is justified by the fact that all algebraic identities of the Bregman distance used in the manuscript (e.g., the three-point identity \\cite[Lem. 3.1]{chen1993convergence}) are valid regardless of whether \\(\\psi\\) is convex or not, and will overall yield a major simplification of the math.\n","sha256_utf8":"a9937ff51bc945216d380da7f1ab7dcce8625f90bed755a94135bedfd32479e8"}]},{"id":"S022","semantic_type":"definition","kind":"definition","title":"参数模型与迭代算子","source_units":[{"start":33931,"end":34209,"start_line":690,"end_line":709,"raw_tex":"\\begin{align}\n\t\\label{eq:M-innprod}\n\t\t\\M(w;x,x^-)\n\t{}={} &\n\t\t\\varphi(w)\n\t\t{}+{}\n\t\t\\D_{\\h}(w,x)\n\t\t{}+{}\n\t\t\\innprod*{w-x}{\\nabla\\f(x)-\\nabla\\f(x^-)}\n\t\\\\\n\t\\label{eq:M-D}\n\t{}={} &\n\t\t\\varphi(w)\n\t\t{}+{}\n\t\t\\D_{\\h-\\f}(w,x)\n\t\t{}+{}\n\t\t\\D_{\\f}(w,x^-)\n\t\t{}-{}\n\t\t\\D_{\\f}(x,x^-),\n\t\\end{align}","sha256_utf8":"c7fb4d52cbf09fea913bbd7da0d3c4bf9bf985c80795338aa1b6e974fc948e37"},{"start":34338,"end":34549,"start_line":713,"end_line":724,"raw_tex":"\\begin{align}\n\t\tx^{k+1}\n\t{}\\in{} &\n\t\t\\T(x^k,x^{k-1}),\n\t\\shortintertext{%\n\t\twhere \\(\\ffunc{\\T}{C\\times C}{C}\\) defined by\n\t}\n\t\\label{eq:T}\n\t\t\\T(x,x^-)\n\t{}\\coloneqq{} &\n\t\t\\argmin_{w\\in\\R^n}\\M(w;x,x^-)\n\t\\end{align}","sha256_utf8":"6b2287cadd92f4bfc4a0682f6497d398c35d5a71be0e867352d8b8533d565515"}]},{"id":"S023","semantic_type":"definition","kind":"definition","title":"惯性反射包络","source_units":[{"start":40340,"end":40695,"start_line":824,"end_line":831,"raw_tex":"\\begin{definition}[\\BiFRB-envelope]\\label{def: envelope}%\n\tThe envelope function associated to \\BiFRB\\ with stepsize \\(\\gamma<\\nicefrac{1}{[\\sigma_{f,h}]_-}\\)  and inertia \\(\\beta\\in\\R\\) is \\(\\func{\\env}{C\\times C}{\\R}\\) defined as\n\t\\begin{equation}\\label{eq:env}\n\t\t\\env(x,x^-)\n\t{}\\coloneqq{}\n\t\t\\inf_{w\\in\\R^n}\\M(w;x,x^-).\n\t\\end{equation}\n\\end{definition}","sha256_utf8":"8e9e8b4837c204e394c15ab9eb4d1f8f1c999a8710745d20b864fae1bd765a3e"}]},{"id":"S024","semantic_type":"definition","kind":"definition","title":"势函数与缩放步长","source_units":[{"start":45429,"end":45569,"start_line":912,"end_line":912,"raw_tex":"\tThe convergence analysis in both cases revolves around the identification of a constant \\(c>0\\) determining a lower bounded merit function\n","sha256_utf8":"c26de6798824a4ed5e64205a031ccd45b8534f4fd735e2b8c13e806c0ac57831"},{"start":45570,"end":45655,"start_line":913,"end_line":915,"raw_tex":"\\begin{equation}\\label{eq:LL}\n\t\t\\LL=\\env+\\tfrac{c}{2\\gamma}\\D+\\D_\\xi.\n\t\\end{equation}","sha256_utf8":"355e59227a9de410966d77cc18aff6e678e8f78f74c9cfd5fe8fc0402fb361f4"},{"start":45711,"end":45969,"start_line":917,"end_line":917,"raw_tex":"\tThe difference between the two cases is determined by function \\(\\xi\\) appearing in the last Bregman operator \\(\\D_\\xi\\), having \\(\\xi=\\f\\) in the former case and \\(\\xi=L_{\\f}\\j\\) in the latter, where \\(L_{\\f}\\) is a Lipschitz constant for \\(\\nabla\\f\\) and\n","sha256_utf8":"2e7aa940a0c939b83ce9b6ab9a7a1271a0de67a73995db5eaa49f9627ca5fd8f"},{"start":45970,"end":46051,"start_line":918,"end_line":920,"raw_tex":"\\begin{equation}\\label{eq:j}\n\t\t\\j\\coloneqq\\tfrac12\\|{}\\cdot{}\\|^2\n\t\\end{equation}","sha256_utf8":"6ab166c891e878618dbeb333787b16df273a6005b87bafa59bb98414f065a497"},{"start":46052,"end":46084,"start_line":921,"end_line":921,"raw_tex":"\tis the squared Euclidean norm.\n","sha256_utf8":"4114f82700c4c5ce8af531516adcc395067812a35f35c4a1315433f26d72872e"},{"start":46287,"end":46512,"start_line":925,"end_line":928,"raw_tex":"In what follows, patterning the normalization of \\(\\sigma_{\\pm f,h}\\) into \\(p_{\\pm f,h}\\) detailed in \\cref{sec:relsmooth} we also introduce the scaled stepsize\n\\begin{equation}\n\t\\alpha\\coloneqq\\gamma L_{f,h},\n\\end{equation}","sha256_utf8":"1cb00383c6298e5fb8ea86f0b9e99eb01a9c5b238053f296d745a94f70d01147"}]},{"id":"S025","semantic_type":"assumption","kind":"assumption","title":"核函数全定义域","source_units":[{"start":62220,"end":62322,"start_line":1371,"end_line":1373,"raw_tex":"\\begin{assumption}\\label{ass:h}\n\tFunction \\(h\\) has full domain, that is, \\(C=\\R^n\\).\n\\end{assumption}","sha256_utf8":"01a4bc484033a90b325eca074cc1b07c97ea0dea52295120f47538f2b3a13d33"}]},{"id":"S026","semantic_type":"definition","kind":"definition","title":"广义凹KL性质","source_units":[{"start":69024,"end":69263,"start_line":1483,"end_line":1483,"raw_tex":"For $\\eta\\in(0,\\infty]$, denote by $\\Psi_\\eta$ the class of functions $\\psi:[0,\\eta)\\rightarrow\\R_+$ satisfying the following: (i) $\\psi(t)$ is right-continuous at $t=0$ with $\\psi(0)=0$; (ii) $\\psi$ is strictly increasing on $[0,\\eta)$. \n","sha256_utf8":"d67776e8cc2e428fb33ff037a310217614ff93dc1b06d02e8d1b120abdaa8279"},{"start":69264,"end":70514,"start_line":1485,"end_line":1506,"raw_tex":"\\begin{definition}[{\\cite[Def. 5]{wang2022exact}}]\\label{def: GCKL}%\n\tLet $\\func{f}{\\R^n}{\\Rinf}$ be proper and lsc.\n\tLet $\\bar x\\in\\dom\\partial f$ and $\\mu\\in\\R$, and let $V\\subseteq\\dom\\partial f$ be a nonempty subset.\n\t\\begin{enumerate}\n\t\\item\n\t\tWe say that $f$ has the pointwise generalized concave Kurdyka-\\L ojasiewicz~(KL) property at $\\bar x\\in\\dom\\partial f$, if there exist a neighborhood $U\\ni\\bar x$, $\\eta\\in(0,\\infty]$ and a concave $\\psi\\in\\Psi_\\eta$, such that for all $x\\in U\\cap[0<f-f(\\bar x)<\\eta]$,\n\t\t\\[\n\t\t\t\\psi'_-\\bigl(f(x)-f(\\bar x)\\bigr)\\cdot\\dist\\bigl(0,\\partial f(x)\\bigr)\\geq1,\n\t\t\\]\n\t\twhere $\\psi_-'$ denotes the left derivative.\n\t\tMoreover, $f$ is a generalized concave KL function if it has the generalized concave KL property at every $x\\in\\dom\\partial f$.\n\t\\item\n\t\tSuppose that $f(x)=\\mu$ on $V$.\n\t\tWe say $f$ has the setwise\\footnote{%\n\t\t\tWe shall omit adjectives ``pointwise'' and ``setwise'' whenever there is no ambiguity.%\n\t\t}\n\t\tgeneralized concave KL property on $V$ if there exist $U\\supset V$, $\\eta\\in(0,\\infty]$ and a concave $\\psi\\in\\Psi_\\eta$ such that for every $x\\in U\\cap[0<f-\\mu<\\eta]$,\n\t\t\\[\n\t\t\t\\psi'_-\\bigl(f(x)-\\mu\\bigr)\\cdot\\dist\\bigl(0,\\partial f(x)\\bigr)\\geq1.\n\t\t\\]\n\t\\end{enumerate}\n\\end{definition}","sha256_utf8":"ea727b979f2b10106e4251060a4479efb8f832a67881de06c284b3b5a1f1fa9f"}]},{"id":"S027","semantic_type":"notation","kind":"notation","title":"集合邻域与乘积范数","source_units":[{"start":70516,"end":70652,"start_line":1508,"end_line":1508,"raw_tex":"For a subset $\\Omega\\subseteq\\R^n$, define $(\\forall \\varepsilon>0)$ $\\Omega_\\varepsilon=\\set{x\\in\\R^n}[\\dist(x,\\Omega)<\\varepsilon]$. \n","sha256_utf8":"baa71d6c2911e5a25c7ebc1705e13a22d108a59d00a8c1dd07f1e36ce3710298"},{"start":72243,"end":72409,"start_line":1542,"end_line":1542,"raw_tex":"In the remainder of the section, we will make use of the norm \\(\\norm*{{}\\cdot{}}\\) on the product space \\(\\R^n\\times\\R^n\\) defined as \\(\\norm*{(x,y)}=\\|x\\|+\\|y\\|\\).\n","sha256_utf8":"4acc8f87f587935972aba6683d715d7f44644c5caee9506a09131fe47d57ae3e"}]},{"id":"S028","semantic_type":"theorem","kind":"theorem","title":"函数值收敛及聚点驻性","source_units":[{"start":62695,"end":63739,"start_line":1382,"end_line":1397,"raw_tex":"\\begin{theorem}[function value convergence of \\BiFRB]\\label{thm:subseq}%\n\tLet $\\seq{x^k}$ be a sequence generated by \\BiFRB* in the setting of \\cref{thm:SD}.\n\tThen,\n\t\\begin{enumerate}\n\t\\item\\label{thm:subseq:SD}%\n\t\tIt holds that\n\t\t\\begin{equation}\\label{SD inequality}\n\t\t\t\\LL\\left(x^{k+1},x^k\\right)\\leq\\LL\\left(x^k,x^{k-1}\\right)-\\tfrac{c}{2\\gamma}\\D\\left(x^{k+1},x^k\\right)-\\tfrac{c}{2\\gamma}\\D\\left(x^k,x^{k-1}\\right).\n\t\t\\end{equation}\n\t\tThen, $\\sum_{k=0}^\\infty \\D\\left(x^k,x^{k-1}\\right)<\\infty$ and $\\LL(x^{k},x^{k-1})\\to\\varphi^\\star$ as $k\\to\\infty$ for some $\\varphi^\\star\\geq\\inf\\cost$.\n\t\\item\\label{thm:subseq:bounded}%\n\t\tIf \\(\\cost\\) is level bounded, then \\(\\seq{x^k}\\) is bounded.\n\t\\item\\label{thm:subseq:omega}%\n\t\tSuppose that \\cref{ass:h} holds, and let \\(\\omega\\) be the set of limit points of \\(\\seq{x^k}\\).\n\t\tThen, \\(\\varphi\\) is constant on \\(\\omega\\) with value \\(\\varphi^\\star\\), and for every \\(x^\\star\\in\\omega\\) it holds that \\(x^\\star\\in\\T(x^\\star,x^\\star)\\) and \\(0\\in\\hat\\partial\\varphi(x^\\star)\\).\n\t\\end{enumerate}\n","sha256_utf8":"2d6f12fcf40e1d206b01f7b7968ed9e0ca196232b95f177a8cd9f87ac3be4966"},{"start":65390,"end":65404,"start_line":1424,"end_line":1425,"raw_tex":"\n\\end{theorem}","sha256_utf8":"fec3781d35a0cf747033913fb320e2aeb3f98bb83cba1742a86c56488a75a969"}]},{"id":"S029","semantic_type":"proof","kind":"proof","title":"函数值与聚点性质的完整证明","source_units":[{"start":63740,"end":65390,"start_line":1398,"end_line":1424,"raw_tex":"\\begin{proof}~\n\t\t\\begin{proofitemize}\n\t\t\\item\\ref{thm:subseq:SD}~\n\t\t\tRecall from \\cref{thm:SD} that \\eqref{SD inequality} holds and that \\(\\inf\\LL=\\inf\\cost>-\\infty\\), from which the convergence of \\(\\seq{\\LL(x^{k},x^{k-1})}\\) readily follows.\n\t\t\tIn turn, a telescoping argument on \\eqref{SD inequality} shows that \\(\\sum_{k\\in\\N}\\D(x^k,x^{k-1})<\\infty\\).\n\t\t\\item\\ref{thm:subseq:bounded}~\n\t\t\tIt follows from \\cref{thm:subseq:SD} that \\(\\LL\\left(x^{k+1},x^k\\right)\\leq\\LL(x^0,x^{-1})\\) holds for every \\(k\\).\n\t\t\tThen boundedness of \\(\\seq{x^k}\\) is implied by level boundedness of \\(\\LL\\); see \\cref{thm:SD:inf}.\n\t\t\\item\\ref{thm:subseq:omega}~\n\t\t\tSuppose that a subsequence \\(\\seq{x^{k_j}}[j\\in\\N]\\) converges to a point \\(x^\\star\\), then so do \\(\\seq{x^{k_j\\pm1}}\\) by \\cref{thm:subseq:SD} and \\cite[Prop. 2.2(iii)]{bauschke2003iterating}.\n\t\t\tSince \\(x^{k_j+1}\\in\\T(x^{k_j},x^{k_j-1})\\), by passing to the limit osc of \\(\\T\\) (\\cref{thm:T:osc}) implies that \\(x^\\star\\in\\T(x^\\star,x^\\star)\\).\n\t\t\tInvoking \\cref{thm:T:fix} yields the stationarity condition \\(0\\in\\hat\\partial\\varphi(x^\\star)\\).\n\t\t\tMoreover, by continuity of \\(\\LL\\) one has\n\t\t\t\\[\n\t\t\t\t\\varphi^\\star\n\t\t\t{}\\defeq{}\n\t\t\t\t\\lim_{k\\to\\infty}\\LL\\left(x^k,x^{k-1}\\right)\n\t\t\t{}={}\n\t\t\t\t\\LL(x^\\star,x^\\star)\n\t\t\t{}={}\n\t\t\t\t\\varphi(x^\\star),\n\t\t\t\\]\n\t\t\twhere the last equality follows from \\cref{thm:env:eq}, owing to the inclusion \\(x^\\star\\in\\T(x^\\star,x^\\star)\\) (and the fact that \\(\\D_\\psi(x,x)=0\\) for any differentiable function \\(\\psi\\)).\n\t\t\tFrom the arbitrarity of \\(x^\\star\\in\\omega\\) we conclude that \\(\\varphi\\equiv\\varphi^\\star\\) on \\(\\omega\\).\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}","sha256_utf8":"a69f98b89e24b9a2b506b927899a238b1ad5ae4c8bbadbf33b9ef34a26595b5e"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"下降与距离级数收敛","source_units":[{"start":63740,"end":64096,"start_line":1398,"end_line":1402,"raw_tex":"\\begin{proof}~\n\t\t\\begin{proofitemize}\n\t\t\\item\\ref{thm:subseq:SD}~\n\t\t\tRecall from \\cref{thm:SD} that \\eqref{SD inequality} holds and that \\(\\inf\\LL=\\inf\\cost>-\\infty\\), from which the convergence of \\(\\seq{\\LL(x^{k},x^{k-1})}\\) readily follows.\n\t\t\tIn turn, a telescoping argument on \\eqref{SD inequality} shows that \\(\\sum_{k\\in\\N}\\D(x^k,x^{k-1})<\\infty\\).\n","sha256_utf8":"f77a3ff8aa28b3a9697f3aa923308aec43eafd6ed65ca6763bd72b474ad88527"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"水平有界性保证迭代有界","source_units":[{"start":64096,"end":64352,"start_line":1403,"end_line":1405,"raw_tex":"\t\t\\item\\ref{thm:subseq:bounded}~\n\t\t\tIt follows from \\cref{thm:subseq:SD} that \\(\\LL\\left(x^{k+1},x^k\\right)\\leq\\LL(x^0,x^{-1})\\) holds for every \\(k\\).\n\t\t\tThen boundedness of \\(\\seq{x^k}\\) is implied by level boundedness of \\(\\LL\\); see \\cref{thm:SD:inf}.\n","sha256_utf8":"1a6490581d65bc1e64c719d9e547a7a3982f8ea85acb3fc22fb30d7d71c4bdbc"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"相邻子列同极限与驻性","source_units":[{"start":64352,"end":64834,"start_line":1406,"end_line":1409,"raw_tex":"\t\t\\item\\ref{thm:subseq:omega}~\n\t\t\tSuppose that a subsequence \\(\\seq{x^{k_j}}[j\\in\\N]\\) converges to a point \\(x^\\star\\), then so do \\(\\seq{x^{k_j\\pm1}}\\) by \\cref{thm:subseq:SD} and \\cite[Prop. 2.2(iii)]{bauschke2003iterating}.\n\t\t\tSince \\(x^{k_j+1}\\in\\T(x^{k_j},x^{k_j-1})\\), by passing to the limit osc of \\(\\T\\) (\\cref{thm:T:osc}) implies that \\(x^\\star\\in\\T(x^\\star,x^\\star)\\).\n\t\t\tInvoking \\cref{thm:T:fix} yields the stationarity condition \\(0\\in\\hat\\partial\\varphi(x^\\star)\\).\n","sha256_utf8":"4bb2c08ab630de72e193495ed7b73407d7ba1117383432395c86be78da7d8100"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"聚点集上的共同函数值","source_units":[{"start":64834,"end":65390,"start_line":1410,"end_line":1424,"raw_tex":"\t\t\tMoreover, by continuity of \\(\\LL\\) one has\n\t\t\t\\[\n\t\t\t\t\\varphi^\\star\n\t\t\t{}\\defeq{}\n\t\t\t\t\\lim_{k\\to\\infty}\\LL\\left(x^k,x^{k-1}\\right)\n\t\t\t{}={}\n\t\t\t\t\\LL(x^\\star,x^\\star)\n\t\t\t{}={}\n\t\t\t\t\\varphi(x^\\star),\n\t\t\t\\]\n\t\t\twhere the last equality follows from \\cref{thm:env:eq}, owing to the inclusion \\(x^\\star\\in\\T(x^\\star,x^\\star)\\) (and the fact that \\(\\D_\\psi(x,x)=0\\) for any differentiable function \\(\\psi\\)).\n\t\t\tFrom the arbitrarity of \\(x^\\star\\in\\omega\\) we conclude that \\(\\varphi\\equiv\\varphi^\\star\\) on \\(\\omega\\).\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}","sha256_utf8":"e2b59c3b809dd6c42e889e63af0078aed4b6bcca55b45b17d7003105cf38aa96"}]},{"id":"S034","semantic_type":"theorem","kind":"theorem","title":"势函数充分下降与下界","source_units":[{"start":46604,"end":48018,"start_line":931,"end_line":990,"raw_tex":"\\begin{theorem}\\label{thm:SD}%\n\tSuppose that \\cref{ass:basic} holds and consider one of the following scenarios:\n\t\\begin{enumerator}\n\t\\item\\label{thm:SD:fconv}%\n\t\teither \\(\\f\\) is convex (\\ie, \\(\\alpha p_{f,h}-\\beta\\geq0\\)) and\n\t\t\\(\n\t\t\t\\beta\n\t\t{}>{}\n\t\t\t-\\nicefrac{(1+3\\alpha p_{-f,h})}{2}\n\t\t\\),\n\t\tin which case\n\t\t\\[\n\t\t\t\\xi\\coloneqq\\f\n\t\t\\quad\\text{and}\\quad\n\t\t\t\tc\n\t\t\t{}\\coloneqq{}\n\t\t\t\t1+2\\beta+3\\alpha p_{-f,h}>0,\n\t\t\\]\n\t\\item\\label{thm:SD:fsmooth}%\n\t\tor \\(\\f\\) is \\(L_{\\f}\\)-Lipschitz differentiable, \\(h\\) is \\(\\sigma_h\\)-strongly convex, and\n\t\t\\[\n\t\t\tc\n\t\t{}\\coloneqq{}\n\t\t\t(1+\\alpha p_{-f,h})\\sigma_h\n\t\t\t{}-{}\n\t\t\t2\\gamma L_{\\f}\n\t\t{}>{}\n\t\t\t0,\n\t\t\\]\n\t\tin which case \\(\\xi\\coloneqq L_{\\f}\\j\\).\n\t\\end{enumerator}\n\tThen, for \\(\\LL\\) as in \\eqref{eq:LL} the following assertions hold:\n\t\\begin{enumerate}\n\t\\item\\label{thm:SD:leq}%\n\t\t\\begin{subequations}\n\t\t\tFor every \\(x,x^-\\in C\\) and \\(\\bar x\\in\\T(x,x^-)\\),\n\t\t\t\\begin{align}\n\t\t\t\\label{eq:SD}\n\t\t\t\t\\LL(\\bar x,x)\n\t\t\t{}\\leq{} &\n\t\t\t\t\\LL(x,x^-)\n\t\t\t\t{}-{}\n\t\t\t\t\\tfrac{c}{2\\gamma}\\D(\\bar x,x)\n\t\t\t\t{}-{}\n\t\t\t\t\\tfrac{c}{2\\gamma}\\D(x,x^-)\n\t\t\t\\shortintertext{and}\n\t\t\t\\label{eq:L:geq}\n\t\t\t\t\\cost(\\bar x)\n\t\t\t{}\\leq{} &\n\t\t\t\t\\LL(x,x^-)\n\t\t\t\t{}-{}\n\t\t\t\t\\tfrac{c}{\\gamma}\\D(\\bar x,x)\n\t\t\t\t{}-{}\n\t\t\t\t\\tfrac{c}{2\\gamma}\\D(x,x^-).\n\t\t\t\\end{align}\n\t\t\\end{subequations}\n\t\\item\\label{thm:SD:inf}%\n\t\t\\(\\inf\\LL=\\inf\\cost\\), and \\(\\LL\\) is level bounded iff so is \\(\\cost\\).\n\t\\end{enumerate}\n\\end{theorem}","sha256_utf8":"5b26b57b6d5f5ec973201d005f94854e606961b5a82063fc9e3d05920f960682"}]},{"id":"S035","semantic_type":"proof","kind":"proof","title":"两种情形下势函数定理的完整证明","source_units":[{"start":101434,"end":102926,"start_line":2132,"end_line":2187,"raw_tex":"In the setting of \\cref{thm:SD:fconv}, inequality \\eqref{eq:decrease:fconv} can equivalently be written in terms of \\(\\LL\\) as\n\\begin{align*}\n\t\\LL(\\bar x,x)\n{}\\leq{} &\n\t\\LL(x,x^-)\n\t{}-{}\n\t\\D_{\\h-2\\f-\\frac c\\gamma h}(\\bar x,x)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(\\bar x,x)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(x,x^-)\n\\\\\n{}\\leq{} &\n\t\\LL(x,x^-)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(\\bar x,x)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(x,x^-),\n\\end{align*}\nwhere the second inequality owes to the fact that \\(\\D_{\\h-2\\f-\\frac c\\gamma h}\\geq0\\), since \\(\\h-2\\f-\\frac c\\gamma h\\) is convex, having\n\\[\n\t\\h-2\\f-\\tfrac c\\gamma h\n{}={}\n\t\\tfrac{1+2\\beta-c}{\\gamma}h-3f\n{}={}\n\t\\overbracket*{\\tfrac{1+2\\beta+3\\alpha p_{-f,h}-c}{\\gamma}}^{=0}h+2(-f-\\sigma_{-f,h}h),\n\\]\nthe coefficient of \\(h\\) being null by definition of \\(c\\), and \\(-f-\\sigma_{-f,h}h\\) being convex by definition of the relative weak convexity modulus \\(\\sigma_{-f,h}\\), cf. \\cref{def:relweak}.\nThis proves \\eqref{eq:SD}; inequality \\eqref{eq:L:geq} follows similarly by observing that\n\\[\n\t0\n{}\\leq{}\n\t\\D_{\\h-2\\f-\\frac c\\gamma h}\n{}={}\n\t\\D_{\\h-\\f}-\\D_{\\f}-\\tfrac c\\gamma\\D\n{}\\leq{}\n\t\\D_{\\h-\\f}-\\tfrac c\\gamma\\D,\n\\]\nso that\n\\[\n\t\\LL(x,x^-)\n{}\\overrel{\\ref{thm:env:eq}}{}\n\t\\varphi(\\bar x)\n\t{}+{}\n\t\\overbracket*{\n\t\t\\D_{\\h-\\f}\n\t\t\\vphantom{x^-}\n\t}^{\\geq\\frac c\\gamma\\D}(\\bar x,x)\n\t{}+{}\n\t\\overbracket*{\n\t\t\\D_{\\f}(\\bar x,x^-)\n\t}^{\\geq0}\n\t{}+{}\n\t\\tfrac{c}{2\\gamma}\\D(x,x^-).\n\\]\nIn turn, \\cref{thm:SD:inf} follows from the same arguments as in the proof of \\cref{thm:fconv:general}.\n","sha256_utf8":"95f55c5c502d51ccf9312f87675b8cc417267c3ccd740252d787d9c7ca6e92e2"},{"start":107749,"end":109502,"start_line":2325,"end_line":2406,"raw_tex":"We will pattern the arguments in the previous section, and observe that inequality \\eqref{eq:decrease:fsmooth} can equivalently be written in terms of \\(\\LL\\) as\n\\[\n\t\\LL(\\bar x,x)\n{}\\leq{}\n\t\\LL(x,x^-)\n\t{}-{}\n\t\\overbracket*{\n\t\t\\D_{\\h-2L_{\\f}\\j-\\frac c\\gamma h}\n\t}^{\\geq0}(\\bar x,x)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(\\bar x,x)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(x,x^-).\n\\]\nOnce again, the fact that \\(\\D_{\\h-2L_{\\f}\\j-\\frac c\\gamma h}\\geq0\\) owes to convexity of \\(\\h-2L{\\f}\\j-\\frac c\\gamma h\\), having\n{%\n\\ifsiam\n\t\\mathtight[0.8]%\n\\fi\n\t\\begin{align*}\n\t\t\\h-2L_{\\f}\\j-\\tfrac c\\gamma h\n\t{}={} &\n\t\t\\tfrac{1-c}{\\gamma}h\n\t\t{}-{}\n\t\tf\n\t\t{}-{}\n\t\t2L_{\\f}\\j\n\t\\\\\n\t{}={} &\n\t\t\\def\\myvphantom{\\vphantom{\\tfrac{1+\\gamma\\sigma_{-f,h}-c}{\\gamma}}}\n\t\t\\overbracket[0.5pt]{\n\t\t\t\\myvphantom\n\t\t\t\\tfrac{1+\\gamma\\sigma_{-f,h}-c}{\\gamma}\n\t\t}^{=\\nicefrac{2L_{\\f}}{\\sigma_h}\\geq0}\n\t\t(\\overbracket*{\n\t\t\t\\myvphantom\n\t\t\th-\\sigma_h\\j\n\t\t}^{\\text{convex}})\n\t\t{}+{}\n\t\t(\\overbracket*{\n\t\t\t\\myvphantom\n\t\t\t-f-\\sigma_{-f,h}h\n\t\t}^{\\text{convex}})\n\t\t{}+{}\n\t\t\\bigl(\\overbracket*{\n\t\t\t\\myvphantom\n\t\t\t\\tfrac{1+\\gamma\\sigma_{-f,h}-c}{\\gamma}\\sigma_h\n\t\t\t{}-{}\n\t\t\t2L_{\\f}\n\t\t}^{=0}\n\t\t\\bigr)\\j\n\t\\end{align*}\n}%\naltogether proving \\eqref{eq:SD}.\nSimilarly, inequality \\eqref{eq:L:geq} follows by observing that\n\\(\n\t0\n{}\\leq{}\n\t\\D_{\\h-2\\f-\\frac c\\gamma h}\n{}={}\n\t\\D_{\\h-\\f}-\\D_{\\f}-\\frac c\\gamma\\D\n{}\\leq{}\n\t\\D_{\\h-\\f}-\\frac c\\gamma\\D\n\\),\nso that\n\\[\n\t\\LL(x,x^-)\n{}\\overrel{\\ref{thm:env:eq}}{}\n\t\\varphi(\\bar x)\n\t{}+{}\n\t\\overbracket*{\n\t\t\\D_{\\h-\\f}\n\t\t\\vphantom{x^-}\n\t}^{\\geq\\frac c\\gamma\\D}(\\bar x,x)\n\t{}+{}\n\t\\overbracket*{\n\t\t\\D_{\\f}(\\bar x,x^-)\n\t}^{\\geq0}\n\t{}+{}\n\t\\tfrac{c}{2\\gamma}\\D(x,x^-).\n\\]\nThe assertion of \\cref{thm:SD:inf} once again follows from the same arguments as in the proof of \\cref{thm:fsmooth:general}.\n","sha256_utf8":"874ffe08190c917e1a17b1bbc2beca0c0b8dfe1113ef48e2937519cbc49d7f30"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"凸情形重写势函数下降","source_units":[{"start":101434,"end":101850,"start_line":2132,"end_line":2150,"raw_tex":"In the setting of \\cref{thm:SD:fconv}, inequality \\eqref{eq:decrease:fconv} can equivalently be written in terms of \\(\\LL\\) as\n\\begin{align*}\n\t\\LL(\\bar x,x)\n{}\\leq{} &\n\t\\LL(x,x^-)\n\t{}-{}\n\t\\D_{\\h-2\\f-\\frac c\\gamma h}(\\bar x,x)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(\\bar x,x)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(x,x^-)\n\\\\\n{}\\leq{} &\n\t\\LL(x,x^-)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(\\bar x,x)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(x,x^-),\n\\end{align*}\n","sha256_utf8":"4d8f4ad1e462228ebcdb02c73e449fdf8679ce6b4131ad0e450573fd688467e6"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"弱凸模量保证余项非负","source_units":[{"start":101850,"end":102347,"start_line":2151,"end_line":2159,"raw_tex":"where the second inequality owes to the fact that \\(\\D_{\\h-2\\f-\\frac c\\gamma h}\\geq0\\), since \\(\\h-2\\f-\\frac c\\gamma h\\) is convex, having\n\\[\n\t\\h-2\\f-\\tfrac c\\gamma h\n{}={}\n\t\\tfrac{1+2\\beta-c}{\\gamma}h-3f\n{}={}\n\t\\overbracket*{\\tfrac{1+2\\beta+3\\alpha p_{-f,h}-c}{\\gamma}}^{=0}h+2(-f-\\sigma_{-f,h}h),\n\\]\nthe coefficient of \\(h\\) being null by definition of \\(c\\), and \\(-f-\\sigma_{-f,h}h\\) being convex by definition of the relative weak convexity modulus \\(\\sigma_{-f,h}\\), cf. \\cref{def:relweak}.\n","sha256_utf8":"07acd76067582ef67c8dd4ec2e61b37f67d669c227469b44b84b6b6df18e648d"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"凸情形的距离下界","source_units":[{"start":102347,"end":102567,"start_line":2160,"end_line":2169,"raw_tex":"This proves \\eqref{eq:SD}; inequality \\eqref{eq:L:geq} follows similarly by observing that\n\\[\n\t0\n{}\\leq{}\n\t\\D_{\\h-2\\f-\\frac c\\gamma h}\n{}={}\n\t\\D_{\\h-\\f}-\\D_{\\f}-\\tfrac c\\gamma\\D\n{}\\leq{}\n\t\\D_{\\h-\\f}-\\tfrac c\\gamma\\D,\n\\]\n","sha256_utf8":"1dedde9794261fb828c9dbc37c2c827cf1564098e156f48bed0316d3f1fc4ff3"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"凸情形的目标值控制","source_units":[{"start":102567,"end":102822,"start_line":2170,"end_line":2186,"raw_tex":"so that\n\\[\n\t\\LL(x,x^-)\n{}\\overrel{\\ref{thm:env:eq}}{}\n\t\\varphi(\\bar x)\n\t{}+{}\n\t\\overbracket*{\n\t\t\\D_{\\h-\\f}\n\t\t\\vphantom{x^-}\n\t}^{\\geq\\frac c\\gamma\\D}(\\bar x,x)\n\t{}+{}\n\t\\overbracket*{\n\t\t\\D_{\\f}(\\bar x,x^-)\n\t}^{\\geq0}\n\t{}+{}\n\t\\tfrac{c}{2\\gamma}\\D(x,x^-).\n\\]\n","sha256_utf8":"d89546a19db4f8f2616846af59f7eb84bda6c997d4b3ddbb3f89a10ca77a615e"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"凸情形的下确界与水平有界性","source_units":[{"start":102822,"end":102926,"start_line":2187,"end_line":2187,"raw_tex":"In turn, \\cref{thm:SD:inf} follows from the same arguments as in the proof of \\cref{thm:fconv:general}.\n","sha256_utf8":"0f33a189d3f5c84f90ca19db4f55670b5c7be154fce70dcce181ca0082107f37"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"光滑情形重写势函数下降","source_units":[{"start":107749,"end":108109,"start_line":2325,"end_line":2338,"raw_tex":"We will pattern the arguments in the previous section, and observe that inequality \\eqref{eq:decrease:fsmooth} can equivalently be written in terms of \\(\\LL\\) as\n\\[\n\t\\LL(\\bar x,x)\n{}\\leq{}\n\t\\LL(x,x^-)\n\t{}-{}\n\t\\overbracket*{\n\t\t\\D_{\\h-2L_{\\f}\\j-\\frac c\\gamma h}\n\t}^{\\geq0}(\\bar x,x)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(\\bar x,x)\n\t{}-{}\n\t\\tfrac{c}{2\\gamma}\\D(x,x^-).\n\\]\n","sha256_utf8":"b09b4c1a5eef11068ef16b7818920fbc031ea1fb2581d89ae0d9a1827d642bbc"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"强凸性与模量保证余项非负","source_units":[{"start":108109,"end":108930,"start_line":2339,"end_line":2378,"raw_tex":"Once again, the fact that \\(\\D_{\\h-2L_{\\f}\\j-\\frac c\\gamma h}\\geq0\\) owes to convexity of \\(\\h-2L{\\f}\\j-\\frac c\\gamma h\\), having\n{%\n\\ifsiam\n\t\\mathtight[0.8]%\n\\fi\n\t\\begin{align*}\n\t\t\\h-2L_{\\f}\\j-\\tfrac c\\gamma h\n\t{}={} &\n\t\t\\tfrac{1-c}{\\gamma}h\n\t\t{}-{}\n\t\tf\n\t\t{}-{}\n\t\t2L_{\\f}\\j\n\t\\\\\n\t{}={} &\n\t\t\\def\\myvphantom{\\vphantom{\\tfrac{1+\\gamma\\sigma_{-f,h}-c}{\\gamma}}}\n\t\t\\overbracket[0.5pt]{\n\t\t\t\\myvphantom\n\t\t\t\\tfrac{1+\\gamma\\sigma_{-f,h}-c}{\\gamma}\n\t\t}^{=\\nicefrac{2L_{\\f}}{\\sigma_h}\\geq0}\n\t\t(\\overbracket*{\n\t\t\t\\myvphantom\n\t\t\th-\\sigma_h\\j\n\t\t}^{\\text{convex}})\n\t\t{}+{}\n\t\t(\\overbracket*{\n\t\t\t\\myvphantom\n\t\t\t-f-\\sigma_{-f,h}h\n\t\t}^{\\text{convex}})\n\t\t{}+{}\n\t\t\\bigl(\\overbracket*{\n\t\t\t\\myvphantom\n\t\t\t\\tfrac{1+\\gamma\\sigma_{-f,h}-c}{\\gamma}\\sigma_h\n\t\t\t{}-{}\n\t\t\t2L_{\\f}\n\t\t}^{=0}\n\t\t\\bigr)\\j\n\t\\end{align*}\n}%\naltogether proving \\eqref{eq:SD}.\n","sha256_utf8":"4d4065c4f7627b2218c99eaab9a7841408f410bc866f00aa3f62524e23829226"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"光滑情形的目标值控制","source_units":[{"start":108930,"end":109377,"start_line":2379,"end_line":2405,"raw_tex":"Similarly, inequality \\eqref{eq:L:geq} follows by observing that\n\\(\n\t0\n{}\\leq{}\n\t\\D_{\\h-2\\f-\\frac c\\gamma h}\n{}={}\n\t\\D_{\\h-\\f}-\\D_{\\f}-\\frac c\\gamma\\D\n{}\\leq{}\n\t\\D_{\\h-\\f}-\\frac c\\gamma\\D\n\\),\nso that\n\\[\n\t\\LL(x,x^-)\n{}\\overrel{\\ref{thm:env:eq}}{}\n\t\\varphi(\\bar x)\n\t{}+{}\n\t\\overbracket*{\n\t\t\\D_{\\h-\\f}\n\t\t\\vphantom{x^-}\n\t}^{\\geq\\frac c\\gamma\\D}(\\bar x,x)\n\t{}+{}\n\t\\overbracket*{\n\t\t\\D_{\\f}(\\bar x,x^-)\n\t}^{\\geq0}\n\t{}+{}\n\t\\tfrac{c}{2\\gamma}\\D(x,x^-).\n\\]\n","sha256_utf8":"e95b7527102a16c2004ffa8cd801dbe44b20e0415a0517699f4545bb13cd7f83"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"光滑情形的下确界与水平有界性","source_units":[{"start":109377,"end":109502,"start_line":2406,"end_line":2406,"raw_tex":"The assertion of \\cref{thm:SD:inf} once again follows from the same arguments as in the proof of \\cref{thm:fsmooth:general}.\n","sha256_utf8":"7739195ade10bffcfbe9b02fb833a1f6e6aa15dfed9647b4aee7ee42f4b0fc94"}]},{"id":"S045","semantic_type":"lemma","kind":"lemma","title":"凸辅助函数的一般下降引理","source_units":[{"start":99471,"end":100209,"start_line":2095,"end_line":2113,"raw_tex":"\\begin{lemma}\\label{thm:fconv:general}%\n\tSuppose that \\cref{ass:basic} holds and let \\(\\gamma>0\\) and \\(\\beta\\in\\R\\) be such that \\(\\f\\coloneqq f-\\frac\\beta\\gamma h\\) is a convex function.\n\tThen, for every \\(x,x^-\\in C\\) and \\(\\bar x\\in\\T(x,x^-)\\)\n\t\\begin{align}\\label{eq:decrease:fconv}\n\t\t\\left(\\env+\\D_{\\f}\\right)(\\bar x,x)\n\t{}\\leq{} &\n\t\t\\left(\\env+\\D_{\\f}\\right)(x,x^-)\n\t\t{}-{}\n\t\t\\D_{\\h-2\\f}(\\bar x,x).\n% \t\\shortintertext{%\n% \t\tIf \\(\\h-\\f\\) too is convex, then it also holds that\n% \t}\n% \t\t\\varphi(\\bar x)\n% \t{}\\leq{} &\n% \t\t\\left(\\env+\\D_{\\f}\\right)(x,x^-)\n% \t\t{}-{}\n% \t\t\\D_{\\h-2\\f}(\\bar x,x),\n\t\\end{align}\n\tIf \\(\\h-\\f\\) too is convex, then \\(\\env+\\D_{\\f}\\) has the same infimum of \\(\\cost\\), and is level bounded iff so is \\(\\cost\\).%\n","sha256_utf8":"6578fb99568ca81c859b01a6929b03ecffdbe5d7452a0a2a22f0185d5d863ea2"},{"start":101420,"end":101432,"start_line":2129,"end_line":2130,"raw_tex":"\n\\end{lemma}","sha256_utf8":"896e76217f21eff3f136c06bfdb40107e2e8a48d744a916c99f19effd7da4ea8"}]},{"id":"S046","semantic_type":"proof","kind":"proof","title":"凸情形一般引理的完整证明","source_units":[{"start":100210,"end":101420,"start_line":2114,"end_line":2129,"raw_tex":"\\begin{proof}\n\t\tAll the claimed inequalities follows from \\eqref{eq:descent} together with the fact that \\(\\D_{\\f}\\geq0\\), and that \\(\\D_{\\h-\\f}\\geq0\\) too when \\(\\h-\\f\\) is convex.\n\t\tWhen both \\(\\f\\) and \\(\\h-\\f\\) are convex, \\cref{thm:env:eq} implies that\n\t\t\\begin{equation}\\label{eq:fconv:geq}\n\t\t\t\\env(y,y^-)+\\D_{\\f}(y,y^-)\\geq\\varphi(\\bar y)\\geq\\inf\\cost\n\t\t\\quad\n\t\t\t\\forall y,y^-\\in C,\\ \\bar y\\in\\T(y,y^-),\n\t\t\\end{equation}\n\t\tproving that \\(\\inf(\\env+\\D_{\\f})\\geq\\inf\\cost\\).\n\t\tThe converse inequality follows from \\cref{thm:env:leq} by observing that \\((\\env+\\D_{\\f})(x,x)=\\env(x,x)\\leq\\varphi(x)\\).\n\n\t\tTo conclude, suppose that \\(\\env+\\D_{\\f}\\) is not level bounded, and consider an unbounded sequence \\(\\seq{x_k,x_k^-}\\) such that \\(\\env(x_k,x_k^-)+\\D_{\\f}(x_k,x_k^-)\\leq\\ell\\), for some \\(\\ell\\in\\R\\).\n\t\tThen, it follows from \\eqref{eq:fconv:geq} that \\(\\varphi(\\bar x_k)\\leq\\ell\\), where \\(\\bar x_k\\in\\T(x_k,x_k^-)\\).\n\t\tFrom local boundedness of \\(\\T\\) (\\cref{thm:T:osc}) it follows that \\(\\seq{\\bar x_k}\\) too is unbounded, showing that \\(\\varphi\\) is not level bounded either.\n\t\tThe converse holds by observing that \\((\\env+\\D_{\\f})(x,x)=\\env(x,x)\\leq\\cost(x)\\), cf. \\cref{thm:env:leq}.\n\t\\end{proof}","sha256_utf8":"d14b3d4c633efb3c960b938c3fc273247e689c9a914d30f09a2c98f1d01ee861"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"非负Bregman项导出下降","source_units":[{"start":100210,"end":100392,"start_line":2114,"end_line":2115,"raw_tex":"\\begin{proof}\n\t\tAll the claimed inequalities follows from \\eqref{eq:descent} together with the fact that \\(\\D_{\\f}\\geq0\\), and that \\(\\D_{\\h-\\f}\\geq0\\) too when \\(\\h-\\f\\) is convex.\n","sha256_utf8":"f0d20cad9e2b89253ebb0c4c7faeaab83d1bb11254d24fefddf40175530c7757"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"上下比较给出相同下确界","source_units":[{"start":100392,"end":100815,"start_line":2116,"end_line":2123,"raw_tex":"\t\tWhen both \\(\\f\\) and \\(\\h-\\f\\) are convex, \\cref{thm:env:eq} implies that\n\t\t\\begin{equation}\\label{eq:fconv:geq}\n\t\t\t\\env(y,y^-)+\\D_{\\f}(y,y^-)\\geq\\varphi(\\bar y)\\geq\\inf\\cost\n\t\t\\quad\n\t\t\t\\forall y,y^-\\in C,\\ \\bar y\\in\\T(y,y^-),\n\t\t\\end{equation}\n\t\tproving that \\(\\inf(\\env+\\D_{\\f})\\geq\\inf\\cost\\).\n\t\tThe converse inequality follows from \\cref{thm:env:leq} by observing that \\((\\env+\\D_{\\f})(x,x)=\\env(x,x)\\leq\\varphi(x)\\).\n","sha256_utf8":"6bc520d9b28817646210c6671726e290650dd73ca89813e4c517de9820500cc3"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"证明水平有界性的等价","source_units":[{"start":100815,"end":101420,"start_line":2124,"end_line":2129,"raw_tex":"\n\t\tTo conclude, suppose that \\(\\env+\\D_{\\f}\\) is not level bounded, and consider an unbounded sequence \\(\\seq{x_k,x_k^-}\\) such that \\(\\env(x_k,x_k^-)+\\D_{\\f}(x_k,x_k^-)\\leq\\ell\\), for some \\(\\ell\\in\\R\\).\n\t\tThen, it follows from \\eqref{eq:fconv:geq} that \\(\\varphi(\\bar x_k)\\leq\\ell\\), where \\(\\bar x_k\\in\\T(x_k,x_k^-)\\).\n\t\tFrom local boundedness of \\(\\T\\) (\\cref{thm:T:osc}) it follows that \\(\\seq{\\bar x_k}\\) too is unbounded, showing that \\(\\varphi\\) is not level bounded either.\n\t\tThe converse holds by observing that \\((\\env+\\D_{\\f})(x,x)=\\env(x,x)\\leq\\cost(x)\\), cf. \\cref{thm:env:leq}.\n\t\\end{proof}","sha256_utf8":"d885e5a655fffc30d56925f96b7f4290b32d54964417f874a5a47866e4dae8c2"}]},{"id":"S050","semantic_type":"lemma","kind":"lemma","title":"光滑辅助函数的一般下降引理","source_units":[{"start":105893,"end":106456,"start_line":2270,"end_line":2280,"raw_tex":"\\begin{lemma}\\label{thm:fsmooth:general}%\n\tAdditionally to \\cref{ass:basic}, suppose that \\(\\f\\) is \\(L_{\\f}\\)-Lipschitz differentiable for some \\(L_{\\f}\\geq0\\).\n\tThen, for every \\(x,x^-\\in C\\) and \\(\\bar x\\in\\T(x,x^-)\\)\n\t\\begin{equation}\\label{eq:decrease:fsmooth}\n\t\t\\left(\\env+\\D_{L_{\\f}\\j}\\right)(\\bar x,x)\n\t{}\\leq{}\n\t\t\\left(\\env+\\D_{L_{\\f}\\j}\\right)(x,x^-)\n\t\t{}-{}\n\t\t\\D_{\\h-2L_{\\f}\\j}(\\bar x,x).\n\t\\end{equation}\n\tIf \\(\\h-L_{\\f}\\j\\) is a convex function, then \\(\\env+\\D_{L_{\\f}\\j}\\) has the same infimum of \\(\\cost\\), and is level bounded iff so is \\(\\cost\\).\n","sha256_utf8":"12eb3992e74882917c25b2952599b1da3ed3bf532aeb6a30084ca120dfa1efa3"},{"start":107735,"end":107747,"start_line":2322,"end_line":2323,"raw_tex":"\n\\end{lemma}","sha256_utf8":"896e76217f21eff3f136c06bfdb40107e2e8a48d744a916c99f19effd7da4ea8"}]},{"id":"S051","semantic_type":"proof","kind":"proof","title":"光滑情形一般引理的完整证明","source_units":[{"start":106457,"end":107735,"start_line":2281,"end_line":2322,"raw_tex":"\\begin{proof}\n\t\tBy means of the three-point identity, that is, by using \\eqref{eq:M-innprod} in place of \\eqref{eq:M-D}, inequality \\eqref{eq:descent} can equivalently be written as\n\t\t\\begin{align*}\n\t\t\t\\env(\\bar x,x)\n\t\t{}\\leq{}\n\t\t\t\\cost(\\bar x)\n\t\t{}={} &\n\t\t\t\\env(x,x^-)\n\t\t\t{}-{}\n\t\t\t\\D_{\\h}(\\bar x,x)\n\t\t\t{}-{}\n\t\t\t\\innprod*{\\bar x-x}{\\nabla\\f(x)-\\nabla\\f(x^-)}\n\t\t\\shortintertext{%\n\t\t\twhich by using Young's inequality on the inner product and \\(L_{\\f}\\)-Lipschitz differentiability yields\n\t\t}\n\t\t{}\\leq{} &\n\t\t\t\\env(x,x^-)\n\t\t\t{}-{}\n\t\t\t\\D_{\\h}(\\bar x,x)\n\t\t\t{}+{}\n\t\t\t\\tfrac{L_{\\f}}{2}\\|\\bar x-x\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{L_{\\f}}{2}\\|x-x^-\\|^2.\n\t\t\\end{align*}\n\t\tRearranging and using the fact that \\(\\D_{\\j}(x,y)=\\frac12\\|x-y\\|^2\\) yields the claimed inequality.\n\t\t\n\t\tUnder convexity of \\(\\h-L_{\\f}\\j\\), we may draw the same conclusions as in the proof of \\cref{thm:fconv:general} by observing that the same application of Young's inequality above, combined with the expression \\eqref{eq:M-innprod} of \\(\\M\\), gives\n\t\t\\[\n\t\t\t\\env(y,y^-)\n\t\t{}={}\n\t\t\t\\M(\\bar y;y,y^-)\n\t\t{}\\geq{}\n\t\t\t\\varphi(\\bar y)\n\t\t\t{}+{}\n\t\t\t\\D_{\\h}(\\bar y,y)\n\t\t\t{}-{}\n\t\t\t\\tfrac{L_{\\f}}{2}\\|\\bar y-y\\|^2\n\t\t\t{}-{}\n\t\t\t\\tfrac{L_{\\f}}{2}\\|y-y^-\\|^2,\n\t\t\\]\n\t\tholding for any \\(y,y^-\\in C\\) and \\(\\bar y\\in\\T(y,y^-)\\).\n\t\\end{proof}","sha256_utf8":"398232e35fb79a20bb610cab3996492dd108aa3eb0f0b897a7d771532248bf4b"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"三点恒等式与Young估计","source_units":[{"start":106457,"end":107210,"start_line":2281,"end_line":2305,"raw_tex":"\\begin{proof}\n\t\tBy means of the three-point identity, that is, by using \\eqref{eq:M-innprod} in place of \\eqref{eq:M-D}, inequality \\eqref{eq:descent} can equivalently be written as\n\t\t\\begin{align*}\n\t\t\t\\env(\\bar x,x)\n\t\t{}\\leq{}\n\t\t\t\\cost(\\bar x)\n\t\t{}={} &\n\t\t\t\\env(x,x^-)\n\t\t\t{}-{}\n\t\t\t\\D_{\\h}(\\bar x,x)\n\t\t\t{}-{}\n\t\t\t\\innprod*{\\bar x-x}{\\nabla\\f(x)-\\nabla\\f(x^-)}\n\t\t\\shortintertext{%\n\t\t\twhich by using Young's inequality on the inner product and \\(L_{\\f}\\)-Lipschitz differentiability yields\n\t\t}\n\t\t{}\\leq{} &\n\t\t\t\\env(x,x^-)\n\t\t\t{}-{}\n\t\t\t\\D_{\\h}(\\bar x,x)\n\t\t\t{}+{}\n\t\t\t\\tfrac{L_{\\f}}{2}\\|\\bar x-x\\|^2\n\t\t\t{}+{}\n\t\t\t\\tfrac{L_{\\f}}{2}\\|x-x^-\\|^2.\n\t\t\\end{align*}\n\t\tRearranging and using the fact that \\(\\D_{\\j}(x,y)=\\frac12\\|x-y\\|^2\\) yields the claimed inequality.\n","sha256_utf8":"b927fa2b193cc9a8991a4abca854e9c0ac2c1cb8dc5bff42213acd3f0165d5ff"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"光滑情形的下界及有界性论证","source_units":[{"start":107210,"end":107735,"start_line":2306,"end_line":2322,"raw_tex":"\t\t\n\t\tUnder convexity of \\(\\h-L_{\\f}\\j\\), we may draw the same conclusions as in the proof of \\cref{thm:fconv:general} by observing that the same application of Young's inequality above, combined with the expression \\eqref{eq:M-innprod} of \\(\\M\\), gives\n\t\t\\[\n\t\t\t\\env(y,y^-)\n\t\t{}={}\n\t\t\t\\M(\\bar y;y,y^-)\n\t\t{}\\geq{}\n\t\t\t\\varphi(\\bar y)\n\t\t\t{}+{}\n\t\t\t\\D_{\\h}(\\bar y,y)\n\t\t\t{}-{}\n\t\t\t\\tfrac{L_{\\f}}{2}\\|\\bar y-y\\|^2\n\t\t\t{}-{}\n\t\t\t\\tfrac{L_{\\f}}{2}\\|y-y^-\\|^2,\n\t\t\\]\n\t\tholding for any \\(y,y^-\\in C\\) and \\(\\bar y\\in\\T(y,y^-)\\).\n\t\\end{proof}","sha256_utf8":"9f38ce8c994aa1b8280ecf63804d245fd1de681f1a7e4085d69df44b10142adb"}]},{"id":"S054","semantic_type":"proof","kind":"proof","title":"包络值比较的完整推导","source_units":[{"start":43999,"end":44453,"start_line":882,"end_line":903,"raw_tex":"\t\t\tWe now work towards establishing a merit function for \\BiFRB, starting from comparing the values of \\(\\env(\\bar x,x)\\) and \\(\\env(x,x^-)\\), with \\(\\bar x\\in\\T(x,x^-)\\).\nOwing to \\cref{thm:env:leq}, we have\n\\begin{align}\n\\nonumber\n\t\\env(\\bar x,x)\n{}\\leq{}\n\t\\varphi(\\bar x)\n{}={} &\n\t\\cost(\\bar x)\n\\\\\n\\label{eq:descent}\n{}={} &\n\t\\env(x,x^-)\n\t{}-{}\n\t\\D_{\\h-\\f}(\\bar x,x)\n\t{}-{}\n\t\\overbracket*{\n\t\t\\D_{\\f}(\\bar x,x^-)\n\t}\n\t{}+{}\n\t\\D_{\\f}(x,x^-).\n\\end{align}\n","sha256_utf8":"1154f621838c7306ab5e5072f163edc59d987efd841e9afb53486473557fe604"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"以目标值连接相邻包络","source_units":[{"start":43999,"end":44453,"start_line":882,"end_line":903,"raw_tex":"\t\t\tWe now work towards establishing a merit function for \\BiFRB, starting from comparing the values of \\(\\env(\\bar x,x)\\) and \\(\\env(x,x^-)\\), with \\(\\bar x\\in\\T(x,x^-)\\).\nOwing to \\cref{thm:env:leq}, we have\n\\begin{align}\n\\nonumber\n\t\\env(\\bar x,x)\n{}\\leq{}\n\t\\varphi(\\bar x)\n{}={} &\n\t\\cost(\\bar x)\n\\\\\n\\label{eq:descent}\n{}={} &\n\t\\env(x,x^-)\n\t{}-{}\n\t\\D_{\\h-\\f}(\\bar x,x)\n\t{}-{}\n\t\\overbracket*{\n\t\t\\D_{\\f}(\\bar x,x^-)\n\t}\n\t{}+{}\n\t\\D_{\\f}(x,x^-).\n\\end{align}\n","sha256_utf8":"1154f621838c7306ab5e5072f163edc59d987efd841e9afb53486473557fe604"}]},{"id":"S056","semantic_type":"lemma","kind":"lemma","title":"包络连续性与模型恒等式","source_units":[{"start":42169,"end":42927,"start_line":835,"end_line":866,"raw_tex":"\\begin{lemma}[basic properties of \\(\\env\\)]%\n\tSuppose that \\cref{ass:basic} holds.\n\tThen, for any \\(\\gamma<\\nicefrac{1}{[\\sigma_{f,h}]_-}\\) and \\(\\beta\\in\\R\\) the following hold:\n\t\\begin{enumerate}\n\t\\item\\label{thm:env:C0}%\n\t\t\\(\\env\\) is (real-valued and) continuous on \\(C\\times C\\); in fact, it is locally Lipschitz provided that \\(f,h\\in\\C^2(C)\\).\n\t\\item\\label{thm:env:eq}%\n\t\tFor any \\(x,x^-\\in C\\) and \\(\\bar x\\in\\T(x,x^-)\\)\n\t\t\\[\n\t\t\\ifsiam\n\t\t\t\\mathtight[0.45]\n\t\t\\fi\n\t\t\t\\env(x,x^-)\n\t\t{}={}\n\t\t\t\\M(\\bar x;x,x^-)\n\t\t{}={}\n\t\t\t\\varphi(\\bar x)\n\t\t\t{}+{}\n\t\t\t\\D_{\\h-\\f}(\\bar x,x)\n\t\t\t{}+{}\n\t\t\t\\D_{\\f}(\\bar x,x^-)\n\t\t\t{}-{}\n\t\t\t\\D_{\\f}(x,x^-).\n\t\t\\]\n\t\\item\\label{thm:env:leq}%\n\t\t\\(\n\t\t\t\\env(x,x^-)\n\t\t{}\\leq{}\n\t\t\t\\varphi(x)\n\t\t\\)\n\t\tfor any \\(x,x^-\\in C\\).\n\t\\end{enumerate}\n","sha256_utf8":"ddddc350e9f494b1e78c13544b86f0488e3038cb37be736ac7403d4bdbaa3aca"},{"start":43938,"end":43950,"start_line":877,"end_line":878,"raw_tex":"\n\\end{lemma}","sha256_utf8":"896e76217f21eff3f136c06bfdb40107e2e8a48d744a916c99f19effd7da4ea8"}]},{"id":"S057","semantic_type":"proof","kind":"proof","title":"包络基本性质的完整证明","source_units":[{"start":42928,"end":43938,"start_line":867,"end_line":877,"raw_tex":"\\begin{proof}~\n\t\t\\begin{proofitemize}\n\t\t\\item \\ref{thm:env:C0}~\n\t\t\tIn light of the uniform level boundedness asserted in \\cref{thm:M:lb}, continuity of \\(\\env\\) follows from $\\text{\\cite[Thm. 1.17(c)]{rockafellar2011variational}}$ by observing that the mapping \\((x,x^-)\\mapsto\\M(w;x,x^-)\\) is continuous for every \\(w\\);\n\t\t\tin fact, when \\(f\\) and \\(h\\) are both \\(\\C^2\\) on \\(C\\), its gradient exists and is continuous with respect to all its arguments, which together with local boundedness of \\(\\T\\), cf. \\cref{thm:T:osc}, gives that \\(-\\env\\) is a lower-\\(\\C^1\\) function in the sense of \\cite[Def. 10.29]{rockafellar2011variational}, and in particular locally Lipschitz continuous by virtue of \\cite[Thm.s 10.31 and 9.2]{rockafellar2011variational}.\n\t\t\\item \\ref{thm:env:eq} \\& \\ref{thm:env:leq}~\n\t\t\tThe identity follows by definition, cf. \\eqref{eq:env} and \\eqref{eq:T}.\n\t\t\tThe inequality follows by considering \\(w=x\\) in \\eqref{eq:env} and \\eqref{eq:M-D}.\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}","sha256_utf8":"3d512946e7575b0737e048215fcd4e437ca20f285dfed39c5a816f00c19f3793"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"参数极小化保证包络连续","source_units":[{"start":42928,"end":43684,"start_line":867,"end_line":871,"raw_tex":"\\begin{proof}~\n\t\t\\begin{proofitemize}\n\t\t\\item \\ref{thm:env:C0}~\n\t\t\tIn light of the uniform level boundedness asserted in \\cref{thm:M:lb}, continuity of \\(\\env\\) follows from $\\text{\\cite[Thm. 1.17(c)]{rockafellar2011variational}}$ by observing that the mapping \\((x,x^-)\\mapsto\\M(w;x,x^-)\\) is continuous for every \\(w\\);\n\t\t\tin fact, when \\(f\\) and \\(h\\) are both \\(\\C^2\\) on \\(C\\), its gradient exists and is continuous with respect to all its arguments, which together with local boundedness of \\(\\T\\), cf. \\cref{thm:T:osc}, gives that \\(-\\env\\) is a lower-\\(\\C^1\\) function in the sense of \\cite[Def. 10.29]{rockafellar2011variational}, and in particular locally Lipschitz continuous by virtue of \\cite[Thm.s 10.31 and 9.2]{rockafellar2011variational}.\n","sha256_utf8":"8373aa374e1c78e713a99d7a8e49df2e5b921863855a9b63a81b88728325ce5d"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"由定义得到恒等式与上界","source_units":[{"start":43684,"end":43938,"start_line":872,"end_line":877,"raw_tex":"\t\t\\item \\ref{thm:env:eq} \\& \\ref{thm:env:leq}~\n\t\t\tThe identity follows by definition, cf. \\eqref{eq:env} and \\eqref{eq:T}.\n\t\t\tThe inequality follows by considering \\(w=x\\) in \\eqref{eq:env} and \\eqref{eq:M-D}.\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}","sha256_utf8":"99c4b9fa9b25e7fc8543de9aaae642a8b2e7c4fc17f6f25afeb9729bd5e3d394"}]},{"id":"S060","semantic_type":"lemma","kind":"lemma","title":"模型与算子的基本性质","source_units":[{"start":35169,"end":36619,"start_line":732,"end_line":757,"raw_tex":"\\begin{lemma}[basic properties of the model \\(\\M\\) and the operator \\(\\T\\)]\\label{thm:basic}%\n\tSuppose that \\cref{ass:basic} holds, and let \\(\\gamma<\\nicefrac{1}{[\\sigma_{-f,h}]_-}\\) and \\(\\beta\\in\\R\\) be fixed.\n\tThe following hold:\n\t\\begin{enumerate}\n\t\\item \\label{thm:M:eq}%\n\t\t\\(\n\t\t\t\\M(x;x,x^-)\n\t\t{}={}\n\t\t\t\\cost(x)\n\t\t\\)\n\t\tfor all \\(x,x^-\\in C\\).\n\t\\item \\label{thm:M:lb}%\n\t\t\\(\\M(w;x,x^-)\\) is level bounded in \\(w\\) locally uniformly in \\((x,x^-)\\) on \\(C\\times C\\).\\footnote{%\n\t\t\tNamely, \\(\\bigcup_{(x,x^-)\\in\\mathcal V}\\set{w\\in\\R^n}[\\M(w;x,x^-)\\leq\\alpha]\\) is bounded for any \\(\\mathcal V\\subset C\\times C\\) compact and \\(\\alpha\\in\\R\\).%\n\t\t}\n\t\\item \\label{thm:T:osc}%\n\t\t\\(\\T\\) is locally bounded and osc,\\footnote{%\n\t\t\tBeing \\(\\T\\) defined on \\(C\\times C\\), osc and local boundedness are meant relative to \\(C\\times C\\).\n\t\t\tNamely, \\(\\graph\\T\\) is closed relative to \\(C\\times C\\times\\R^n\\), and \\(\\bigcup_{(x,x^-)\\in\\mathcal V}\\T(x,x^-)\\) is bounded for any \\(\\mathcal V\\subset C\\times C\\) compact.%\n\t\t}\n\t\tand \\(T(x,x^-)\\) is a nonempty and compact subset of \\(C\\) for any \\(x,x^-\\in C\\).\n\t\\item \\label{thm:T:partial}%\n\t\t\\(\\nabla\\h(x)-\\nabla\\h(\\bar x)-\\nabla\\f(x)+\\nabla\\f(x^-)\\in\\hat\\partial\\varphi(\\bar x)\\) for any \\(x,x^-\\in C\\) and \\(\\bar x\\in\\T(x,x^-)\\).\n\t\\item \\label{thm:T:fix}%\n\t\tIf \\(x\\in\\T(x,x)\\), then \\(0\\in\\hat\\partial\\varphi(x)\\) and \\(\\def\\stepsize{\\gamma'}\\T(x,x)=\\set{x}\\) for every \\(\\gamma'\\in(0,\\gamma)\\).\n\t\\end{enumerate}\n","sha256_utf8":"3f18ccb9d5dba71b807f22af6300adcbe9f230d52143564391d7e1db903f8065"},{"start":38274,"end":38286,"start_line":782,"end_line":783,"raw_tex":"\n\\end{lemma}","sha256_utf8":"896e76217f21eff3f136c06bfdb40107e2e8a48d744a916c99f19effd7da4ea8"}]},{"id":"S061","semantic_type":"proof","kind":"proof","title":"模型及算子性质的完整证明","source_units":[{"start":36620,"end":38274,"start_line":758,"end_line":782,"raw_tex":"\\begin{proof}\n\t\tWe start by observing that \\cref{thm:proxbounded} ensures that the set \\(\\T(x,x^-)\\) is non\\-empty for any \\(x,x^-\\in C\\); this follows by considering the expression \\eqref{eq:M-innprod} of the model, by observing that, for any \\(x\\in C\\),\n\t\t\\(\n\t\t\t\\varphi+\\D_{\\h}({}\\cdot{},x)\n\t\t{}={}\n\t\t\tg+\\tfrac1\\gamma h-\\h(x)-\\innprod*{\\nabla\\h(x)}{{}\\cdot{}-x}\n\t\t\\).\n\t\tFor the same reason, it then follows from \\cref{ass:T} that \\(T(x,x^-)\\subset C\\).\n\t\t\\begin{proofitemize}\n\t\t\\item \\ref{thm:M:eq}~\n\t\t\tApparent, by considering \\(w=x\\) in \\eqref{eq:M-D}.\n\t\t\\item \\ref{thm:M:lb} \\& \\ref{thm:T:osc}~\n\t\t\tThe first assertion owes to the fact that \\(\\h\\) is 1-coercive by \\cref{thm:Legendre} and that both \\(\\h\\) and \\(\\nabla\\f\\) are continuous on \\(C\\), so that for any compact \\(\\mathcal V\\subset\\R^n\\times\\R^n\\) one has that\n\t\t\t\\(\n\t\t\t\t\\lim_{\\|w\\|\\to\\infty}\\inf_{(x,x^-)\\in\\mathcal V}\\M(w;x,x^-)=\\infty\n\t\t\t\\),\n\t\t\tas is apparent from \\eqref{eq:M-innprod}.\n\t\t\tIn turn, the second assertion follows from \\cite[Thm. 1.17]{rockafellar2011variational}.\n\t\t\\item \\ref{thm:T:partial}~\n\t\t\tFollows from the optimality conditions of \\(\\bar x\\in\\argmin\\M({}\\cdot{};x,x^-)\\), for \\(\\bar x\\in C\\) by assumption and thus the calculus rule of \\cite[Ex. 8.8(c)]{rockafellar2011variational} applies (having \\(\\h\\) smooth around \\(\\bar x\\in C\\)).\n\t\t\\item \\ref{thm:T:fix}~\n\t\t\tThat \\(0\\in\\hat\\partial\\varphi(x)\\) follows from assertion \\ref{thm:T:partial}, and the other claim from \\cite[Lem. 3.6]{ahookhosh2021bregman} by observing that \\(\\T(x,x)=\\argmin\\set{\\varphi+\\D_{\\h}({}\\cdot{},x)}\\) for any \\(\\gamma>0\\) and \\(\\beta\\in\\R\\).\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}","sha256_utf8":"663f8d3bb363839e842b5478b7f21194bf12d2e3aafc47d185c1a32df454203c"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"子问题解存在且位于核内部","source_units":[{"start":36620,"end":37075,"start_line":758,"end_line":765,"raw_tex":"\\begin{proof}\n\t\tWe start by observing that \\cref{thm:proxbounded} ensures that the set \\(\\T(x,x^-)\\) is non\\-empty for any \\(x,x^-\\in C\\); this follows by considering the expression \\eqref{eq:M-innprod} of the model, by observing that, for any \\(x\\in C\\),\n\t\t\\(\n\t\t\t\\varphi+\\D_{\\h}({}\\cdot{},x)\n\t\t{}={}\n\t\t\tg+\\tfrac1\\gamma h-\\h(x)-\\innprod*{\\nabla\\h(x)}{{}\\cdot{}-x}\n\t\t\\).\n\t\tFor the same reason, it then follows from \\cref{ass:T} that \\(T(x,x^-)\\subset C\\).\n","sha256_utf8":"77cc0f794262935febde83daa3eae79677d765181171cb42ce3df16666430c96"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"模型在基点处与目标相切","source_units":[{"start":37075,"end":37177,"start_line":766,"end_line":768,"raw_tex":"\t\t\\begin{proofitemize}\n\t\t\\item \\ref{thm:M:eq}~\n\t\t\tApparent, by considering \\(w=x\\) in \\eqref{eq:M-D}.\n","sha256_utf8":"fb519c1feaac4bf209e7048e5fc3c872506756bb7c2782a1460f5714a76d4319"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"一致水平有界与外半连续","source_units":[{"start":37177,"end":37666,"start_line":769,"end_line":775,"raw_tex":"\t\t\\item \\ref{thm:M:lb} \\& \\ref{thm:T:osc}~\n\t\t\tThe first assertion owes to the fact that \\(\\h\\) is 1-coercive by \\cref{thm:Legendre} and that both \\(\\h\\) and \\(\\nabla\\f\\) are continuous on \\(C\\), so that for any compact \\(\\mathcal V\\subset\\R^n\\times\\R^n\\) one has that\n\t\t\t\\(\n\t\t\t\t\\lim_{\\|w\\|\\to\\infty}\\inf_{(x,x^-)\\in\\mathcal V}\\M(w;x,x^-)=\\infty\n\t\t\t\\),\n\t\t\tas is apparent from \\eqref{eq:M-innprod}.\n\t\t\tIn turn, the second assertion follows from \\cite[Thm. 1.17]{rockafellar2011variational}.\n","sha256_utf8":"2aadb05ea6da6a1f74571eca17670b51533c1bb4a1753dd16287aa29c8ebd88d"}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","title":"子问题最优性给出正则次梯度","source_units":[{"start":37666,"end":37946,"start_line":776,"end_line":777,"raw_tex":"\t\t\\item \\ref{thm:T:partial}~\n\t\t\tFollows from the optimality conditions of \\(\\bar x\\in\\argmin\\M({}\\cdot{};x,x^-)\\), for \\(\\bar x\\in C\\) by assumption and thus the calculus rule of \\cite[Ex. 8.8(c)]{rockafellar2011variational} applies (having \\(\\h\\) smooth around \\(\\bar x\\in C\\)).\n","sha256_utf8":"ea9b7cf44763632fd81f646c05a7f0807544a1db76c28fb53c69ec7db891d8bb"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"不动点驻性与步长缩减","source_units":[{"start":37946,"end":38274,"start_line":778,"end_line":782,"raw_tex":"\t\t\\item \\ref{thm:T:fix}~\n\t\t\tThat \\(0\\in\\hat\\partial\\varphi(x)\\) follows from assertion \\ref{thm:T:partial}, and the other claim from \\cite[Lem. 3.6]{ahookhosh2021bregman} by observing that \\(\\T(x,x)=\\argmin\\set{\\varphi+\\D_{\\h}({}\\cdot{},x)}\\) for any \\(\\gamma>0\\) and \\(\\beta\\in\\R\\).\n\t\t\\qedhere\n\t\t\\end{proofitemize}\n\t\\end{proof}","sha256_utf8":"eb613186710d78c6dd5666e958f4f9e380e9f68627e0d81c9e1de91eb7a224df"}]},{"id":"S067","semantic_type":"lemma","kind":"lemma","title":"相对近端有界与解存在","source_units":[{"start":27070,"end":27504,"start_line":550,"end_line":554,"raw_tex":"\\begin{lemma}[relative prox-boundedness]\\label{thm:proxbounded}%\n\tSuppose that \\cref{ass:basic} holds.\n\tThen, the set \\(\\argmin\\set{\\gamma g+h+\\innprod{v}{\\cdot}}\\) as in \\cref{ass:T} is nonempty for any \\(v\\in\\R^n\\) and \\(0<\\gamma<\\nicefrac{1}{[\\sigma_{-f,h}]_-}\\).\n\tIn other words, \\(g\\) is prox bounded relative to \\(h\\) with threshold \\(\\gamma_{g,h}\\geq\\nicefrac{1}{[\\sigma_{-f,h}]_-}\\) \\cite[Def. 2.3]{kan2012moreau}.\n\\end{lemma}","sha256_utf8":"614714c10968a347c99627685b9ba744572ce44b88386e01513be0bb0e055c56"}]},{"id":"S068","semantic_type":"proof","kind":"proof","title":"相对近端有界性的完整证明","source_units":[{"start":27505,"end":28169,"start_line":555,"end_line":580,"raw_tex":"\\begin{proof}\n\tRecall that a proper convex function admits affine minorant; see, e.g,~\\cite[Cor. 16.18]{bauschke2017convex}.\n\tIt then follows that \\(\\gamma g+h\\) is 1-coercive by observing that\n\t\\[\n\t\t\\gamma g+h\n\t{}={}\n\t\t\\gamma\\cost-\\gamma f+h\n\t{}={}\n\t\t\\gamma\\underbracket*{\n\t\t\t\\vphantom{\\left([\\sigma_{-f,h}]_-\\right)}\n\t\t\t\\cost\n\t\t}_{\\mathclap{\\geq\\inf\\cost}}\n\t\t{}+{}\n\t\t\\gamma\\underbracket*{\n\t\t\t\\left(-f+[\\sigma_{-f,h}]_-h\\right)\n\t\t}_{\\text{convex}}\n\t\t{}+{}\n\t\t\\underbracket*{\n\t\t\t\\left(1-\\gamma[\\sigma_{-f,h}]_-\\right)\n\t\t}_{>0}\n\t\t\\overbracket*{\n\t\t\t\\,h\\,\n\t\t}^{\\text{\\clap{1-coercive}}}\n\t\\]\n\ton \\(\\closure C\\) and \\(\\infty\\) on \\(\\R^n\\setminus\\closure C\\).\n\\end{proof}","sha256_utf8":"8783a14ea33006b37834fbe69b1a2e2a9507b4f79885fcb61608d2584c7fe090"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"调用凸函数的仿射下界","source_units":[{"start":27505,"end":27630,"start_line":555,"end_line":556,"raw_tex":"\\begin{proof}\n\tRecall that a proper convex function admits affine minorant; see, e.g,~\\cite[Cor. 16.18]{bauschke2017convex}.\n","sha256_utf8":"7269e3822550d30714934bb905257a09e9f2c1fc5646247b1c441c320d929dd0"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"分解目标证明一阶强制性","source_units":[{"start":27630,"end":28169,"start_line":557,"end_line":580,"raw_tex":"\tIt then follows that \\(\\gamma g+h\\) is 1-coercive by observing that\n\t\\[\n\t\t\\gamma g+h\n\t{}={}\n\t\t\\gamma\\cost-\\gamma f+h\n\t{}={}\n\t\t\\gamma\\underbracket*{\n\t\t\t\\vphantom{\\left([\\sigma_{-f,h}]_-\\right)}\n\t\t\t\\cost\n\t\t}_{\\mathclap{\\geq\\inf\\cost}}\n\t\t{}+{}\n\t\t\\gamma\\underbracket*{\n\t\t\t\\left(-f+[\\sigma_{-f,h}]_-h\\right)\n\t\t}_{\\text{convex}}\n\t\t{}+{}\n\t\t\\underbracket*{\n\t\t\t\\left(1-\\gamma[\\sigma_{-f,h}]_-\\right)\n\t\t}_{>0}\n\t\t\\overbracket*{\n\t\t\t\\,h\\,\n\t\t}^{\\text{\\clap{1-coercive}}}\n\t\\]\n\ton \\(\\closure C\\) and \\(\\infty\\) on \\(\\R^n\\setminus\\closure C\\).\n\\end{proof}","sha256_utf8":"9be746d6e17d60b6e87d9906f472d6c10a5cd64a4e69501daf60e3567d5f3538"}]},{"id":"S071","semantic_type":"lemma","kind":"lemma","title":"修改核保持勒让德性质","source_units":[{"start":32839,"end":33087,"start_line":675,"end_line":678,"raw_tex":"\\begin{lemma}[{{\\cite[Thm. 4.1]{ahookhosh2021bregman}}}]\\label{thm:Legendre}%\n\tSuppose that \\cref{ass:f} holds.\n\tThen, for every \\(\\gamma<\\nicefrac{1}{[\\sigma_{-f,h}]_-}\\) the function \\(\\h\\) is a Legendre kernel with \\(\\dom\\h=\\dom h\\).\n\\end{lemma}","sha256_utf8":"0aace98996aff81c09a35bcb2bf6200352c4ac1124dbbe5195e28f0d28302580"}]},{"id":"S072","semantic_type":"lemma","kind":"lemma","title":"一致KL性质与精确模量","source_units":[{"start":70653,"end":72241,"start_line":1510,"end_line":1540,"raw_tex":"\\begin{lemma}[{{\\cite[Lem. 4.4]{wang2022exact}}}]\\label{lemma: Uniformize GCKL}%\n\tLet $\\func{f}{\\R^n}{\\Rinf}$ be proper lsc and let $\\mu\\in\\R$.\n\tLet $\\Omega\\subseteq\\dom\\partial f$ be a nonempty compact set on which $f(x)=\\mu$ for all $x\\in\\Omega$.\n\tThen the following hold:\n\t\\begin{enumerate}\n\t\\item\n\t\tSuppose that $f$ satisfies the pointwise generalized concave KL property at each $x\\in\\Omega$.\n\t\tThen there exist $\\varepsilon>0,\\eta\\in(0,\\infty]$ and $\\psi\\in\\Psi_\\eta$ such that $f$ has the setwise generalized concave KL property on $\\Omega$ with respect to $U=\\Omega_\\varepsilon$, $\\eta$ and $\\psi$.\n\t\\item \\label{uniformize exact modulus}%\n\t\tSet $U=\\Omega_\\varepsilon$ and define $h:(0,\\eta)\\rightarrow\\R_+$ by\n\t\t\\[\n\t\t\th(s)=\\sup\\set{\\dist^{-1}\\left(0,\\partial f(x)\\right)}[{x\\in U\\cap[s\\leq f-\\mu<\\eta]}].\n\t\t\\]\n\t\tThen the function $\\func{\\tilde\\psi}{[0,\\eta)}{\\R_+}$ defined by \\((\\forall 0<t<\\eta)~\\tilde\\psi(t)=\\int_0^th(s)ds\\) with \\(\\tilde{\\psi}(0)=0\\) is well defined and belongs to $\\Psi_\\eta$.\n\t\tThe function $f$ has the setwise generalized concave KL property on $\\Omega$ with respect to $U$, $\\eta$ and $\\tilde\\psi$.\n\t\tMoreover,\n\t\t\\[\n\t\t\t\\tilde{\\psi}\n\t\t{}={}\n\t\t\t\\inf\\left\\{\n\t\t\t\t\\psi\\in\\Psi_\\eta\n\t\t\t\t\\,\\Big|\\,\n\t\t\t\t\\text{\\begin{tabular}{@{}c@{}}%\n\t\t\t\t\t\t\\(\\psi\\) is a concave desingularizing function\\\\\n\t\t\t\t\t\tof $f$ on $\\Omega$ with respect to $U$ and $\\eta$\n\t\t\t\t\\end{tabular}}\n\t\t\t\\right\\}.\n\t\t\\]\n\t\\end{enumerate}\n\tWe say that $\\tilde{\\psi}$ is the exact modulus of the setwise generalized concave KL property of \\(f\\) on \\(\\Omega\\) with respect to $U$ and $\\eta$.\n\\end{lemma}","sha256_utf8":"f9974ac62794acdeb94495d8a13230cc61f7b2bd910998e10cfbc2201918bd14"}]},{"id":"S073","semantic_type":"assumption","kind":"assumption","title":"目标函数水平有界","source_units":[{"start":72927,"end":72982,"start_line":1553,"end_line":1554,"raw_tex":"\t\\item\\label{bounded}%\n\t\t\\(\\varphi\\) is level bounded.\n","sha256_utf8":"9784b623619d6bc24e0bab753b6b824931740a228e4be8198f54691c45c281fb"}]},{"id":"S074","semantic_type":"assumption","kind":"assumption","title":"二阶连续性与正定核Hessian","source_units":[{"start":72982,"end":73091,"start_line":1555,"end_line":1556,"raw_tex":"\t\\item\\label{c2 and pd}%\n\t\t$f,h$ are twice continuously differentiable and $\\nabla^2h$ is positive definite.\n","sha256_utf8":"623049424c65a2d643547ca4ab1d153b5cc9d4f982bfa5d9b98e1f1496266986"}]},{"id":"S075","semantic_type":"assumption","kind":"assumption","title":"提升函数的集合KL假设","source_units":[{"start":73091,"end":73272,"start_line":1557,"end_line":1558,"raw_tex":"\t\\item\\label{GCKL}%\n\t\t$(\\exists r>0)~(\\exists\\eta\\in[0,\\infty))$ $\\F$ satisfies the generalized concave KL property on $\\Omega=\\omega(z^0)$ with respect to $U=\\Omega_r$ and $\\eta$.\n","sha256_utf8":"d091b4161b33c53067396fc56ae84258bdf7b672c5cd3e43293985e9228f7fae"}]},{"id":"S076","semantic_type":"assumption","kind":"assumption","title":"全域算法序列及参数条件","source_units":[{"start":72495,"end":72617,"start_line":1545,"end_line":1545,"raw_tex":"\tSuppose that \\cref{ass:h} holds, and let $\\seq{x^k}$ be a sequence generated by \\BiFRB* in the setting of \\cref{thm:SD}.\n","sha256_utf8":"2f93f0bdf1225d2b2479c234bf9c5b1c21e94361b2afc95ede21af9e697d5b2e"}]},{"id":"S077","semantic_type":"assumption","kind":"assumption","title":"凸辅助函数情形的参数假设","source_units":[{"start":46635,"end":47022,"start_line":932,"end_line":948,"raw_tex":"\tSuppose that \\cref{ass:basic} holds and consider one of the following scenarios:\n\t\\begin{enumerator}\n\t\\item\\label{thm:SD:fconv}%\n\t\teither \\(\\f\\) is convex (\\ie, \\(\\alpha p_{f,h}-\\beta\\geq0\\)) and\n\t\t\\(\n\t\t\t\\beta\n\t\t{}>{}\n\t\t\t-\\nicefrac{(1+3\\alpha p_{-f,h})}{2}\n\t\t\\),\n\t\tin which case\n\t\t\\[\n\t\t\t\\xi\\coloneqq\\f\n\t\t\\quad\\text{and}\\quad\n\t\t\t\tc\n\t\t\t{}\\coloneqq{}\n\t\t\t\t1+2\\beta+3\\alpha p_{-f,h}>0,\n\t\t\\]\n","sha256_utf8":"a436c2735a4e2129108175875bf363ee8a6ae9640df4832de691e373a162c553"}]},{"id":"S078","semantic_type":"assumption","kind":"assumption","title":"光滑辅助函数情形的参数假设","source_units":[{"start":47022,"end":47293,"start_line":949,"end_line":960,"raw_tex":"\t\\item\\label{thm:SD:fsmooth}%\n\t\tor \\(\\f\\) is \\(L_{\\f}\\)-Lipschitz differentiable, \\(h\\) is \\(\\sigma_h\\)-strongly convex, and\n\t\t\\[\n\t\t\tc\n\t\t{}\\coloneqq{}\n\t\t\t(1+\\alpha p_{-f,h})\\sigma_h\n\t\t\t{}-{}\n\t\t\t2\\gamma L_{\\f}\n\t\t{}>{}\n\t\t\t0,\n\t\t\\]\n\t\tin which case \\(\\xi\\coloneqq L_{\\f}\\j\\).\n","sha256_utf8":"4bc92475056e4ded994a4df402dce217c47a245204be52dfcfbc2daaed78e109"}]},{"id":"S079","semantic_type":"assumption","kind":"assumption","title":"一般凸情形的辅助函数假设","source_units":[{"start":99511,"end":99660,"start_line":2096,"end_line":2096,"raw_tex":"\tSuppose that \\cref{ass:basic} holds and let \\(\\gamma>0\\) and \\(\\beta\\in\\R\\) be such that \\(\\f\\coloneqq f-\\frac\\beta\\gamma h\\) is a convex function.\n","sha256_utf8":"a15a967315b9dabfa20c509b1a8f2aeca6ab75de1344b7842d5585b57dedcaa4"}]},{"id":"S080","semantic_type":"assumption","kind":"assumption","title":"一般光滑情形的Lipschitz假设","source_units":[{"start":105935,"end":106055,"start_line":2271,"end_line":2271,"raw_tex":"\tAdditionally to \\cref{ass:basic}, suppose that \\(\\f\\) is \\(L_{\\f}\\)-Lipschitz differentiable for some \\(L_{\\f}\\geq0\\).\n","sha256_utf8":"41df68f327461ef7290fc2b9f97e11953a30742dbebb501b4f01abebbbfff340"}]},{"id":"S081","semantic_type":"assumption","kind":"assumption","title":"算子性质所需步长条件","source_units":[{"start":35263,"end":35381,"start_line":733,"end_line":733,"raw_tex":"\tSuppose that \\cref{ass:basic} holds, and let \\(\\gamma<\\nicefrac{1}{[\\sigma_{-f,h}]_-}\\) and \\(\\beta\\in\\R\\) be fixed.\n","sha256_utf8":"3193c713b4947f386e21d1ebf9e409b467bfede8acc3205cbeed9173402ab393"}]},{"id":"S082","semantic_type":"assumption","kind":"assumption","title":"函数值定理的算法序列假设","source_units":[{"start":62768,"end":62853,"start_line":1383,"end_line":1383,"raw_tex":"\tLet $\\seq{x^k}$ be a sequence generated by \\BiFRB* in the setting of \\cref{thm:SD}.\n","sha256_utf8":"ca3dda1a1476d0911b78c3f8f5ab9b5358474b1df349284aabbf31ab106bc1bc"}]},{"id":"S083","semantic_type":"external_fact","kind":"external_fact","title":"仿射下界与强制函数取极小值","source_units":[],"declared_external":true,"declaration_text":["A proper lower semicontinuous convex function has an affine minorant. A proper lower semicontinuous coercive function on finite-dimensional Euclidean space attains its minimum; adding an affine function to a 1-coercive function preserves coercivity."]},{"id":"S084","semantic_type":"external_fact","kind":"external_fact","title":"勒让德核的相对光滑扰动","source_units":[],"declared_external":true,"declaration_text":["The cited ahookhosh2021bregman, Theorem 4.1, supplies the Legendre-kernel preservation result stated in B0033 under its relative-smoothness and stepsize hypotheses."]},{"id":"S085","semantic_type":"external_fact","kind":"external_fact","title":"参数极小化的稳定性定理","source_units":[],"declared_external":true,"declaration_text":["Under the lower semicontinuity, continuity and local uniform level-boundedness conditions invoked by the authors, parametric minimization has nonempty compact minimizers, a locally bounded outer-semicontinuous solution mapping, and a continuous value function; see rockafellar2011variational, Theorem 1.17."]},{"id":"S086","semantic_type":"external_fact","kind":"external_fact","title":"费马条件与光滑扰动次微分法则","source_units":[],"declared_external":true,"declaration_text":["At a local minimizer of a proper function, zero is a regular subgradient. For a continuously differentiable perturbation q, the regular and limiting subdifferentials of p+q equal the corresponding subdifferentials of p translated by the gradient of q; the analogous separable product-space rule applies."]},{"id":"S087","semantic_type":"external_fact","kind":"external_fact","title":"缩减步长后的唯一近端不动点","source_units":[],"declared_external":true,"declaration_text":["For the diagonal Bregman proximal minimization used in the paper, if x is a minimizer at stepsize gamma, strict reduction to any gamma' in (0,gamma) makes x the unique minimizer; see ahookhosh2021bregman, Lemma 3.6."]},{"id":"S088","semantic_type":"external_fact","kind":"external_fact","title":"下C一表示的局部Lipschitz性","source_units":[],"declared_external":true,"declaration_text":["The locally bounded minimizer representation with continuously varying parameter gradients used by the authors makes the negative envelope lower-C1, hence locally Lipschitz; see rockafellar2011variational, Definition 10.29 and Theorems 10.31 and 9.2."]},{"id":"S089","semantic_type":"external_fact","kind":"external_fact","title":"Bregman距离的序列一致性","source_units":[],"declared_external":true,"declaration_text":["For the full-domain Legendre kernel in the selected argument, vanishing adjacent Bregman distances together with convergence of a subsequence imply convergence of its preceding and succeeding iterates to the same point, as invoked from bauschke2003iterating, Proposition 2.2(iii)."]},{"id":"S090","semantic_type":"external_fact","kind":"external_fact","title":"广义凹KL的精确一致化定理","source_units":[],"declared_external":true,"declaration_text":["The cited wang2022exact, Lemma 4.4, supplies the compact-set uniformization of the generalized concave KL property and the integral exact modulus, including its desingularizing inequality and pointwise minimality, as stated in B0091."]},{"id":"S091","semantic_type":"external_fact","kind":"external_fact","title":"Bregman三点恒等式","source_units":[],"declared_external":true,"declaration_text":["For a differentiable generator psi, D_psi(a,c)-D_psi(a,b)-D_psi(b,c) equals the inner product of a-b with grad psi(b)-grad psi(c). This algebraic identity does not require convexity of psi; convexity additionally gives nonnegativity of its Bregman distance."]},{"id":"S092","semantic_type":"external_fact","kind":"external_fact","title":"有限维分析与求和基本事实","source_units":[],"declared_external":true,"declaration_text":["Young and Cauchy-Schwarz inequalities control the displayed inner products and squared norms. A concave increasing function satisfies the left-derivative secant bound used in the KL argument. Continuous Hessians are bounded on a compact ball, and a continuous positive-definite Hessian has a uniform positive lower eigenvalue there, giving a quadratic Bregman lower bound. Bounded sequences have compact nonempty cluster sets and approach those sets. Telescoping descent and finite total Euclidean length yield a Cauchy sequence, which converges in Euclidean space."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S006","target_id":"S002","kind":"supports"},{"source_id":"S007","target_id":"S002","kind":"supports"},{"source_id":"S008","target_id":"S002","kind":"supports"},{"source_id":"S009","target_id":"S002","kind":"supports"},{"source_id":"S010","target_id":"S002","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S004","target_id":"S008","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S028","target_id":"S003","kind":"supports"},{"source_id":"S028","target_id":"S004","kind":"supports"},{"source_id":"S028","target_id":"S008","kind":"supports"},{"source_id":"S028","target_id":"S010","kind":"supports"},{"source_id":"S060","target_id":"S004","kind":"supports"},{"source_id":"S060","target_id":"S006","kind":"supports"},{"source_id":"S072","target_id":"S004","kind":"supports"},{"source_id":"S073","target_id":"S003","kind":"supports"},{"source_id":"S073","target_id":"S004","kind":"supports"},{"source_id":"S073","target_id":"S005","kind":"supports"},{"source_id":"S074","target_id":"S005","kind":"supports"},{"source_id":"S074","target_id":"S006","kind":"supports"},{"source_id":"S074","target_id":"S008","kind":"supports"},{"source_id":"S075","target_id":"S004","kind":"supports"},{"source_id":"S076","target_id":"S003","kind":"supports"},{"source_id":"S076","target_id":"S006","kind":"supports"},{"source_id":"S024","target_id":"S003","kind":"supports"},{"source_id":"S021","target_id":"S005","kind":"supports"},{"source_id":"S021","target_id":"S006","kind":"supports"},{"source_id":"S022","target_id":"S006","kind":"supports"},{"source_id":"S016","target_id":"S006","kind":"supports"},{"source_id":"S027","target_id":"S004","kind":"supports"},{"source_id":"S027","target_id":"S007","kind":"supports"},{"source_id":"S027","target_id":"S008","kind":"supports"},{"source_id":"S086","target_id":"S006","kind":"supports"},{"source_id":"S092","target_id":"S003","kind":"supports"},{"source_id":"S092","target_id":"S004","kind":"supports"},{"source_id":"S092","target_id":"S005","kind":"supports"},{"source_id":"S092","target_id":"S008","kind":"supports"},{"source_id":"S092","target_id":"S010","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S032","target_id":"S029","kind":"supports"},{"source_id":"S033","target_id":"S029","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S030","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S034","target_id":"S030","kind":"supports"},{"source_id":"S034","target_id":"S031","kind":"supports"},{"source_id":"S082","target_id":"S030","kind":"supports"},{"source_id":"S013","target_id":"S030","kind":"supports"},{"source_id":"S025","target_id":"S032","kind":"supports"},{"source_id":"S060","target_id":"S032","kind":"supports"},{"source_id":"S056","target_id":"S033","kind":"supports"},{"source_id":"S024","target_id":"S033","kind":"supports"},{"source_id":"S089","target_id":"S032","kind":"supports"},{"source_id":"S092","target_id":"S030","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S035","kind":"supports"},{"source_id":"S039","target_id":"S035","kind":"supports"},{"source_id":"S040","target_id":"S035","kind":"supports"},{"source_id":"S041","target_id":"S035","kind":"supports"},{"source_id":"S042","target_id":"S035","kind":"supports"},{"source_id":"S043","target_id":"S035","kind":"supports"},{"source_id":"S044","target_id":"S035","kind":"supports"},{"source_id":"S045","target_id":"S036","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S045","target_id":"S040","kind":"supports"},{"source_id":"S056","target_id":"S039","kind":"supports"},{"source_id":"S077","target_id":"S036","kind":"supports"},{"source_id":"S077","target_id":"S037","kind":"supports"},{"source_id":"S077","target_id":"S038","kind":"supports"},{"source_id":"S019","target_id":"S037","kind":"supports"},{"source_id":"S020","target_id":"S037","kind":"supports"},{"source_id":"S021","target_id":"S037","kind":"supports"},{"source_id":"S024","target_id":"S036","kind":"supports"},{"source_id":"S050","target_id":"S041","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S050","target_id":"S044","kind":"supports"},{"source_id":"S056","target_id":"S043","kind":"supports"},{"source_id":"S078","target_id":"S041","kind":"supports"},{"source_id":"S078","target_id":"S042","kind":"supports"},{"source_id":"S078","target_id":"S043","kind":"supports"},{"source_id":"S019","target_id":"S042","kind":"supports"},{"source_id":"S020","target_id":"S042","kind":"supports"},{"source_id":"S021","target_id":"S042","kind":"supports"},{"source_id":"S024","target_id":"S041","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S049","target_id":"S046","kind":"supports"},{"source_id":"S054","target_id":"S047","kind":"supports"},{"source_id":"S079","target_id":"S047","kind":"supports"},{"source_id":"S079","target_id":"S048","kind":"supports"},{"source_id":"S056","target_id":"S048","kind":"supports"},{"source_id":"S056","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S060","target_id":"S049","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S022","target_id":"S052","kind":"supports"},{"source_id":"S022","target_id":"S053","kind":"supports"},{"source_id":"S080","target_id":"S052","kind":"supports"},{"source_id":"S080","target_id":"S053","kind":"supports"},{"source_id":"S024","target_id":"S052","kind":"supports"},{"source_id":"S045","target_id":"S053","kind":"supports"},{"source_id":"S091","target_id":"S052","kind":"supports"},{"source_id":"S092","target_id":"S052","kind":"supports"},{"source_id":"S092","target_id":"S053","kind":"supports"},{"source_id":"S055","target_id":"S054","kind":"supports"},{"source_id":"S056","target_id":"S055","kind":"supports"},{"source_id":"S011","target_id":"S055","kind":"supports"},{"source_id":"S021","target_id":"S055","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S059","target_id":"S057","kind":"supports"},{"source_id":"S060","target_id":"S058","kind":"supports"},{"source_id":"S085","target_id":"S058","kind":"supports"},{"source_id":"S088","target_id":"S058","kind":"supports"},{"source_id":"S023","target_id":"S059","kind":"supports"},{"source_id":"S022","target_id":"S059","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S063","target_id":"S061","kind":"supports"},{"source_id":"S064","target_id":"S061","kind":"supports"},{"source_id":"S065","target_id":"S061","kind":"supports"},{"source_id":"S066","target_id":"S061","kind":"supports"},{"source_id":"S067","target_id":"S062","kind":"supports"},{"source_id":"S014","target_id":"S062","kind":"supports"},{"source_id":"S022","target_id":"S062","kind":"supports"},{"source_id":"S022","target_id":"S063","kind":"supports"},{"source_id":"S022","target_id":"S064","kind":"supports"},{"source_id":"S022","target_id":"S065","kind":"supports"},{"source_id":"S022","target_id":"S066","kind":"supports"},{"source_id":"S071","target_id":"S064","kind":"supports"},{"source_id":"S085","target_id":"S064","kind":"supports"},{"source_id":"S062","target_id":"S065","kind":"supports"},{"source_id":"S086","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S087","target_id":"S066","kind":"supports"},{"source_id":"S081","target_id":"S062","kind":"supports"},{"source_id":"S081","target_id":"S064","kind":"supports"},{"source_id":"S016","target_id":"S065","kind":"supports"},{"source_id":"S068","target_id":"S067","kind":"supports"},{"source_id":"S069","target_id":"S068","kind":"supports"},{"source_id":"S070","target_id":"S068","kind":"supports"},{"source_id":"S069","target_id":"S070","kind":"supports"},{"source_id":"S083","target_id":"S069","kind":"supports"},{"source_id":"S083","target_id":"S070","kind":"supports"},{"source_id":"S014","target_id":"S070","kind":"supports"},{"source_id":"S012","target_id":"S070","kind":"supports"},{"source_id":"S019","target_id":"S070","kind":"supports"},{"source_id":"S015","target_id":"S070","kind":"supports"},{"source_id":"S011","target_id":"S070","kind":"supports"},{"source_id":"S084","target_id":"S071","kind":"supports"},{"source_id":"S018","target_id":"S071","kind":"supports"},{"source_id":"S021","target_id":"S071","kind":"supports"},{"source_id":"S090","target_id":"S072","kind":"supports"},{"source_id":"S026","target_id":"S072","kind":"supports"},{"source_id":"S027","target_id":"S072","kind":"supports"},{"source_id":"S018","target_id":"S014","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S017","target_id":"S020","kind":"supports"},{"source_id":"S012","target_id":"S021","kind":"supports"},{"source_id":"S091","target_id":"S022","kind":"supports"},{"source_id":"S013","target_id":"S076","kind":"supports"},{"source_id":"S025","target_id":"S076","kind":"supports"},{"source_id":"S034","target_id":"S076","kind":"supports"},{"source_id":"S013","target_id":"S082","kind":"supports"},{"source_id":"S034","target_id":"S082","kind":"supports"}]}}
{"record_id":"2212.14577v1","paper":{"title":"Extended convergence analysis of the Scholtes-type regularization for cardinality-constrained optimization problems","authors":["Sebastian Lammel","Vladimir Shikhman"],"arxiv_id":"2212.14577","version":"v1","journal":"Mathematical Programming","doi":"10.1007/s10107-024-02082-3","abstract_url":"https://arxiv.org/abs/2212.14577v1","pdf_url":"https://arxiv.org/pdf/2212.14577v1"},"context":{"raw_tex":"In this paper, we study the Scholtes-type regularization method for the class of cardinality-constrained optimization problems:\n\\[\n\\mbox{CCOP}: \\quad\n\\min_{x} \\,\\, f(x)\\quad \\mbox{s.\\,t.} \\quad h(x)=0, \\quad g(x)\\ge 0, \\quad \\left\\|x\\right\\|_0\\le s\n\\]\nwith the feasible set given by equality, inequality, and cardinality constraints, where the so-called zero ”norm” $\\left\\|x\\right\\|_0 = \\left\\vert \\left\\{i \\in \\{1,\\ldots, n\\}\\; \\vert\\; x_i\\ne 0\\right\\}\\right\\vert$ is counting non-zero entries of $x$.\nHere, we assume that the objective function $f$, as well as the equality and inequality constraints \n$h=\\left(h_p, p \\in P\\right)$,\n$g=\\left(g_q, q \\in Q\\right)$ are twice continuously differentiable, and $s \\in \\{0,1,\\ldots, n-1\\}$ is an integer. \n\n\nOur notation is standard. The cardinality of a finite set $A$ is denoted by $\\vert A \\vert$. The $n$-dimensional Euclidean space is denoted by $\\mathbb{R}^n$ with the coordinate vectors $e_i,i= 1,\\ldots, n$. The vector consisting of ones is denoted by $e$. Given a twice continuously differentiable function $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$, $\\nabla f$ denotes its gradient, and $D^2f$ stands for its Hessian.\n\n\nWe associate with CCOP the regularized continuous reformulation from \\cite{laemmel:reform}:\n\\[\n     \\begin{array}{rl}\n    \\displaystyle \\mathcal{R}(c,\\varepsilon): \\quad \\min_{x,y} \\,\\, f(x) +c^Ty\\quad \\mbox{s.\\,t.} \n   &h(x)=0, \\quad g(x)\\ge 0, \\quad \\displaystyle \\sum_{i=1}^{n} y_i \\geq  n - s, \\quad \\\\\n   &x_i y_i =0, \\quad 0 \\leq y_i \\leq 1+\\varepsilon,  \\quad i=1, \\ldots, n,\n   \\end{array}\n\\]\nwhere the components of $c \\in \\R^n$ $c$ are positive and pairwise different, and $0 < \\varepsilon \\leq \\frac{1}{n-s}$.\nGiven a feasible point  $(\\bar x,\\bar y)$ of $\\mathcal{R}$, we define the index sets which correspond to the orthogonality type constraints $x_i y_i=0$, $y_i \\geq 0$:\n    \\[\n     \\begin{array}{c}\n        a_{01}\\left(\\bar x,\\bar y\\right)=\\left\\{i\\,\\left\\vert\\, \\bar x_i=0,\\bar y_i>0\\right.\\right\\}, \\quad \n    a_{10}\\left(\\bar x,\\bar y\\right)=\\left\\{i\\,\\left\\vert\\, \\bar x_i\\ne 0,\\bar y_i=0\\right.\\right\\}, \\\\\n    a_{00}\\left(\\bar x,\\bar y\\right)=\\left\\{i\\,\\left\\vert\\, \\bar x_i=0,\\bar y_i=0\\right.\\right\\}.       \n     \\end{array}\n    \\]\n%The $y$-components which attain the upper bound are stored in  \n%     \\[\\mathcal{E}(\\bar y)=\\left\\{i \\in\\{1,\\ldots,n\\}\\left\\vert\\,\\bar y_i=1+\\varepsilon\\right.\\right\\}.\\]\nThe index sets of the active inequality constraints will be denoted by\n\\[\n Q_0(\\bar x)=\\left\\{q \\in Q \\, \\left\\vert\\, g_q(\\bar x)=0\\right.\\right\\}, \\quad\\mathcal{E}(\\bar y)=\\left\\{i \\,\\left\\vert\\,\\bar y_i=1+\\varepsilon\\right.\\right\\}.\n\\]\n\n\n\\begin{definition}[MPOC-LICQ, \\cite{laemmel:reform}]\nWe say that a feasible point $(\\bar x,\\bar y)$ of $\\mathcal{R}$ satisfies the MPOC-tailored linear independence constraint qualification (MPOC-LICQ) if the following vectors are linearly independent:\n\\[\n\\begin{array}{c}\n\\begin{pmatrix}\n\\nabla h_p(\\bar x)\\\\0\n\\end{pmatrix}, p \\in P, \\quad \n\\begin{pmatrix}\n\\nabla g_q(\\bar x)\\\\0\n\\end{pmatrix}, q \\in Q_0(\\bar x), \\quad  \n\\begin{pmatrix}\n0\\\\\ne_i\n\\end{pmatrix},i\\in \\mathcal{E}(\\bar y), \\quad\n\\begin{pmatrix}\n0\\\\\ne\n\\end{pmatrix}\n\\mbox{ if } \\sum\\limits_{i=1}^{n} \\bar y_i = n - s,\n\\\\\n\\begin{pmatrix}\ne_i\\\\\n0\n\\end{pmatrix}, i \\in a_{01}\\left(\\bar x,\\bar y\\right)\\cup a_{00}\\left(\\bar x,\\bar y\\right), \\quad \n\\begin{pmatrix}\n0\\\\\ne_i\n\\end{pmatrix}, i \\in a_{10}\\left(\\bar x, \\bar y\\right)\\cup a_{00}\\left(\\bar x, \\bar y\\right).\n\\end{array}\n\\]\n\\end{definition}\n\n\\begin{definition}[T-stationary point, \\cite{laemmel:reform}]\n\\label{def:t-stat}\nA feasible point $(\\bar x,\\bar y)$ of $\\mathcal{R}$ is called T-stationary if there exists multipliers \n\\[\n\\begin{array}{l}\n\\bar \\lambda_p, p \\in P,\n\\bar \\mu_{1,q}, q\\in Q_0(\\bar x),\\bar \\mu_{2,i},i \\in \\mathcal{E}(\\bar y),\\bar \\mu_3,\\\\ \\bar \\sigma_{1,i}, i \\in a_{01}\\left(\\bar x,\\bar y\\right),\\bar \\sigma_{2,i}, i \\in a_{10}\\left(\\bar x,\\bar y\\right),\\bar \\varrho_{1,i},\\bar \\varrho_{2,i}, i \\in a_{00}\\left(\\bar x, \\bar y\\right),\n\\end{array}\n\\]\nsuch that the following conditions hold:\n\\begin{equation}\n   \\label{eq:tstat-1} \n   \\begin{array}{rcl}\n  \\begin{pmatrix}\n   \\nabla f(\\bar x)\\\\\n   c\n  \\end{pmatrix}&=& \\displaystyle\\sum\\limits_{p \\in P}\\bar \\lambda_p \n  \\begin{pmatrix}\n  \\nabla h_p(\\bar x)\\\\\n  0\n  \\end{pmatrix}+\n    \\sum\\limits_{q \\in Q_0(\\bar x)}\\bar \\mu_{1,q} \n    \\begin{pmatrix}\n    \\nabla g_q(\\bar x)\\\\\n    0\n    \\end{pmatrix}-\n    \\sum\\limits_{i\\in\\mathcal{E}(\\bar y)} \\bar \\mu_{2,i} \\begin{pmatrix}\n    0\\\\\n    e_i\n\\end{pmatrix}\\\\ \n    && +\n\\bar \\mu_3 \\begin{pmatrix}\n0\\\\\ne\n\\end{pmatrix}\\displaystyle +\\sum\\limits_{i \\in a_{01}\\left(\\bar x,\\bar y\\right)} \\bar \\sigma_{1,i}\n    \\begin{pmatrix}\n    e_{ i}\\\\\n    0\n    \\end{pmatrix}\n    +\\sum\\limits_{i \\in a_{10}\\left(\\bar x,\\bar y \\right)} \\bar \\sigma_{2,i}    \\begin{pmatrix}\n    0\\\\\n    e_{ i}\n    \\end{pmatrix}\\\\ \n    &&\\displaystyle+\\sum\\limits_{i \\in a_{00}\\left(\\bar x, \\bar y\\right)} \\left(\\bar \\varrho_{1,i}\n    \\begin{pmatrix}\n   e_{ i}\\\\\n    0\n    \\end{pmatrix}\n   +\\bar \\varrho_{2,i}\n    \\begin{pmatrix}\n    0\\\\\n    e_{ i}\n    \\end{pmatrix}\\right), \\end{array}\n\\end{equation}\n\\begin{equation}\n   \\label{eq:tstat-2} \\bar \\mu_{1,q} \\ge 0,q\\in Q_0\\left(\\bar x\\right),\n   \\bar \\mu_{2,i} \\ge 0, i\\in \\mathcal{E}(\\bar y),\\bar \\mu_3\\ge 0, \\bar\\mu_3\\left(\n   \\sum\\limits_{i=1}^{n} \\bar y_i -(n-s)\\right)=0,\n\\end{equation}\n\\begin{equation}\n   \\label{eq:tstat-3} \\bar \\varrho_{1,i}=0 \\mbox{ or }\\bar \\varrho_{2,i}\\le 0, i \\in a_{00}\\left(\\bar x,\\bar y\\right).\n\\end{equation}\n\\end{definition}\n\nWe define the appropriate Lagrange function:\n\\[\n        \\begin{array}{rcl}\n        L^\\mathcal{R}(x,y)&=& \\displaystyle\n   f(x)+\n   c^T y -\n        \\sum\\limits_{p \\in P}\\bar \\lambda_p\n  h_p( x)-\n    \\sum\\limits_{q \\in Q_0(\\bar x)}\\bar \\mu_{1,q} \n    g_q(x) \\\\  & &\\displaystyle+\\sum\\limits_{i\\in\\mathcal{E}(\\bar y)} \\bar \\mu_{2,i} \n   \\left( y_i- (1+\\varepsilon)\\right)\n-\n\\bar \\mu_3 \n \\left(\\sum\\limits_{i=1}^{n} y_i - (n-s)\\right)\n   \\\\ \n    &&\\displaystyle -\\sum\\limits_{i \\in a_{01}\\left(\\bar x, \\bar y\\right)} \\bar \\sigma_{1,i}\n     x_{ i}\n    -\\sum\\limits_{i \\in a_{10}\\left(\\bar x, \\bar y\\right)} \\bar \\sigma_{2,i}   \n     y_{ i} -\\sum\\limits_{i \\in a_{00}\\left(\\bar x, \\bar y \\right)} \\left(\\bar \\varrho_{1,i}\n    x_{ i}\n   +\\bar \\varrho_{2,i}\n     y_{ i}\n    \\right).\n    \\end{array}\n\\]\nMoreover, we set for the corresponding tangent space:\n\n\\[\n    \\mathcal{T}^{\\mathcal{R}}_{(\\bar x,\\bar y)} =\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\nDh_p(\\bar x), 0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(\\bar x), 0\\end{pmatrix}\\xi=0,q \\in Q_0(\\bar x),\\\\\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(\\bar y), \n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0 \\mbox{ if } \\displaystyle \\sum_{i=1}^{n} \\bar y_i = n - s,\n\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y) \\cup a_{01}(\\bar x,\\bar y),\\\\\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y) \\cup a_{10}(\\bar x,\\bar y)\n\\end{array}\n\\right.\\right\\}.\n\\]\n\n\n\\begin{definition}[Nondegenerate T-stationary point, \\cite{laemmel:reform}]\nA T-stationary point $(\\bar x,\\bar y)$ of $\\mathcal{R}$ with multipliers $(\\bar \\lambda, \\bar \\mu, \\bar \\sigma, \\bar \\varrho)$ is called nondegenerate if\n\nNDT1: MPOC-LICQ holds at $(\\bar x,\\bar y)$,\n\nNDT2: $\\bar \\mu_{1,q}>0$, $q \\in Q_0\\left(\\bar x\\right)$, $\\bar \\mu_{2,i}>0$, $i \\in \\mathcal{E}\\left(\\bar y\\right)$, and $\\bar \\mu_3>0$ if  $\\sum\\limits_{i=1}^{n} \\bar  y_i =n-s$,\n\nNDT3: $\\bar\\varrho_{1,i}\\ne 0$ and $\\bar\\varrho_{2,i}< 0$, $i\\in a_{00}\\left(\\bar x,\\bar y\\right)$,\n    \nNDT4: the matrix $D^2 L^{\\mathcal{R}}(\\bar x,\\bar y)\\restriction_{\\mathcal{T}^\\mathcal{R}_{(\\bar x,\\bar y)}}$ is nonsingular.\n\n\\noindent\nFor a nondegenerate T-stationary point we eventually use an additional condition:\n%\\item [] NDT5: if $a_{00}\\left(\\bar x,\\bar y\\right)\\not = \\emptyset$, then $\\bar \\sigma_{1,i_{01}}\\ne 0$ for all $i_{01}\\in a_{01}(\\bar x, \\bar y)$.\n\nNDT6: $\\bar \\sigma_{1,i}\\ne 0$, $i\\in a_{01}(\\bar x, \\bar y)$.\n\\end{definition}\n\n\\begin{definition}[T-index, \\cite{laemmel:reform}]\nLet $(\\bar x,\\bar y)$ be a nondegenerate T-stationary point of $\\mathcal{R}$ with unique multipliers $\\left(\\bar \\lambda,\\bar \\mu, \\bar \\sigma,\\bar \\varrho\\right)$. The number of negative eigenvalues of the matrix $D^2 L^{\\mathcal{R}}(\\bar x,\\bar y)\\restriction_{\\mathcal{T}^\\mathcal{R}_{(\\bar x,\\bar y)}}$ is called its quadratic index ($QI$). The cardinality of $a_{00}\\left(\\bar x,\\bar y\\right)$ is called the biactive index ($BI$) of $(\\bar x,\\bar y)$. We define the T-index ($TI$) as the sum of both, i.\\,e. $TI=QI+BI$.\n\\end{definition}\n\nLet us now regularize the orthogonality type constraints in $\\mathcal{R}$ by using the Scholte's idea, cf. \\cite{scholtes:2001}: \n\\[\n     \\begin{array}{rl}\n    \\displaystyle \\mathcal{S}(t): \\quad \\min_{x,y} \\,\\, f(x) +c^Ty\\quad \\mbox{s.\\,t.} \n   &h(x)=0, \\quad g(x)\\ge 0, \\quad\\displaystyle \\sum_{i=1}^{n} y_i \\geq  n - s, \\\\\n   & -t\\le x_i y_i \\le t, \\quad 0 \\leq y_i \\leq 1+\\varepsilon,  \\quad \n         i=1, \\ldots, n,\n   \\end{array}\n\\]\nwhere $t>0$. \n%Sometimes, we write $S(t)$ if the dependence on $t$ is highlighted. \nNote that $\\mathcal{S}$ from above falls into the scope of nonlinear programming, i.e. just smooth equality and inequality constraints are present. The notation for the sets %$a_{00}\\left(x, y\\right)$, $a_{01}\\left(x, y\\right)$, $a_{10}\\left(x, y\\right)$, \n$Q_0(x)$ and $\\mathcal{E}(y)$, which were used for $\\mathcal{R}$, will be used here again.\nFurthermore, we define for a feasible point $\\left( x, y\\right)$ of $\\mathcal{S}$ the index set of vanishing $y$-components as well as the index sets of active relaxed orthogonality type constraints:\n\\[\n\\mathcal{N}( y)=\\left\\{i\\, \\left\\vert\\,  y_i=0\\right.\\right\\},\n\\quad\n\\mathcal{H}^{\\ge}\\left(x, y\\right)=\\left\\{i\\,\\left\\vert \\,x_i  y_i=-t\\right.\\right\\}, \\mathcal{H}^{\\le}\\left( x, y\\right)=\\left\\{i\\,\\left\\vert \\, x_i  y_i=t\\right.\\right\\}.\n\\]\nWe also eventually use the following index sets:\n\\[\n\\mathcal{H}\\left(x, y\\right)=\\mathcal{H}^\\geq\\left(x, y\\right) \\cup \\mathcal{H}^\\leq\\left( x, y\\right), \\quad \n\\mathcal{O}\\left( x, y\\right)=\\left(\\mathcal{E}\\left( y\\right)\\cup \\mathcal{N}\\left(y\\right)\\cup \\mathcal{H}\\left( x, y\\right)\\right)^c.\n\\]\n\n\n\\begin{definition}[LICQ]\nWe say that a feasible point $(x, y)$ of $\\mathcal{S}$ satisfies the linear independence constraint qualification (LICQ) if the following vectors are linearly independent:\n\\[\n\\begin{array}{c}\n      \\begin{pmatrix}\n\\nabla h_p(x)\\\\0\n\\end{pmatrix}, p \\in P, \\quad \n\\begin{pmatrix}\n\\nabla g_q(x)\\\\0\n\\end{pmatrix}, q \\in Q_0(x), \\quad  \n\\begin{pmatrix}\n0\\\\\ne_i\n\\end{pmatrix},i\\in \\mathcal{E}(y),\n\\\\ \\begin{pmatrix}\n0\\\\\ne\n\\end{pmatrix}\n\\mbox{ if } \\sum\\limits_{i=1}^{n} y_i = n - s, \\quad\n\\begin{pmatrix}\ny_i e_i\\\\\nx_i e_i\n\\end{pmatrix}, i \\in \\mathcal{H}\\left(x,y\\right), \\quad \n\\begin{pmatrix}\n0\\\\\ne_i\n\\end{pmatrix}, i \\in \\mathcal{N}(y).\n\\end{array}\n\\]\n\\end{definition}\n\n\\begin{definition}[Karush-Kuhn-Tucker point]\n    A feasible point $(x, y)$ of $\\mathcal{S}$ is called Kurush-Kuhn-Tucker if there exist multipliers \n\\[\n\\lambda_p, p \\in P,\n \\mu_{1,q}, q\\in Q_0( x), \\mu_{2,i},i \\in \\mathcal{E}( y), \\mu_3,\n\\eta^\\ge_i, \\eta^\\le_i, \\nu_i, i\\in\\left\\{1,\\ldots,n\\right\\},\n\\]\nsuch that the following conditions hold:\n\\begin{equation}\n   \\label{eq:kkt-1} \n   \\begin{array}{rcl}\n  \\begin{pmatrix}\n   \\nabla f( x)\\\\\n   c\n  \\end{pmatrix}&=& \\displaystyle \\sum\\limits_{p \\in P} \\lambda_p \n  \\begin{pmatrix}\n  \\nabla h_p( x)\\\\\n  0\n  \\end{pmatrix}+\n    \\sum\\limits_{q \\in Q_0( x)} \\mu_{1,q} \n    \\begin{pmatrix}\n    \\nabla g_q( x)\\\\\n    0\n    \\end{pmatrix}\n   - \\sum\\limits_{i\\in\\mathcal{E}( y)}  \\mu_{2,i} \\begin{pmatrix}\n    0\\\\\n     e_i\n\\end{pmatrix}+\n \\mu_3 \\begin{pmatrix}\n0\\\\\ne\n\\end{pmatrix} \\\\ \n    && \\displaystyle\n+ \\sum\\limits_{i\\in\\mathcal{H}^\\ge( x,  y)}  \\eta^\\ge_{i} \\begin{pmatrix}\n    y_ie_i\\\\\n   x_ie_i\n\\end{pmatrix}\n-\\sum\\limits_{i\\in\\mathcal{H}^\\le( x,  y)}  \\eta^\\le_{i} \\begin{pmatrix}\n    y_ie_i\\\\\n    x_ie_i\n\\end{pmatrix}\n+\n    \\sum\\limits_{i\\in\\mathcal{N}( y)}  \\nu_{i} \\begin{pmatrix}\n    0\\\\\n    e_i\n\\end{pmatrix},\n\\end{array}\n\\end{equation}\n\\begin{equation}\n   \\label{eq:kkt-2}  \\mu_{1,q} \\ge 0,q\\in Q_0\\left( x\\right),\n    \\mu_{2,i} \\ge 0, i\\in \\mathcal{E}(y),\n    \\mu_3\\ge 0, \\mu_3\\left(\n   \\sum\\limits_{i=1}^{n}  y_i -(n-s)\\right)=0,\n\\end{equation}\n\\begin{equation}\n   \\label{eq:kkt-3} \n  \\eta^\\ge_i\\ge 0,i \\in \\mathcal{H}^\\ge( x,  y), \\eta^\\le_i\\ge 0,i \\in \\mathcal{H}^\\le( x,  y), \\nu_i\\ge 0, i \\in \\mathcal{N}( y).\n\\end{equation}\n\\end{definition}\n\nWe again define the Lagrange function as\n\n\\[\n        \\begin{array}{rcl}\n        L^\\mathcal{S}(x,y)&=& \\displaystyle\n   f(x)+c^Ty\n       - \\sum\\limits_{p \\in P} \\lambda_p \n  h_p(x)-\n    \\sum\\limits_{q \\in Q_0(x)} \\mu_{1,q} \n    g_q(x)\n      \\\\&& \\displaystyle +\\sum\\limits_{i\\in\\mathcal{E}(y)}\n   \\mu_{2,i}\n    \\left(y_i - (1+\\varepsilon)\\right)\n -\\mu_3\\left(\\sum\\limits_{i=1}^n y_i - (n-s)\\right)  \\\\ \n    && \\displaystyle\n- \\sum\\limits_{i\\in\\mathcal{H}^\\ge( x,  y)}  \\eta^\\ge_{i}\n    \\left(x_iy_i +t \\right)\n+\\sum\\limits_{i\\in\\mathcal{H}^\\le( x,  y)}  \\eta^\\le_{i}\n    \\left(x_iy_i-t\\right)\n-\n    \\sum\\limits_{i\\in\\mathcal{N}( y)}  \\nu_{i}\n    y_i.\n\\end{array}\n\\]\nThe tangent space is given by\n\n\\[\n    \\mathcal{T}^\\mathcal{S}_{(x,y)} =\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\nDh_p( x),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q( x),0\\end{pmatrix}\\xi=0,q \\in Q_0( x),\\\\ \\displaystyle\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(y),\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0 \\mbox{ if } \\sum_{i=1}^{n} y_i = n - s,\\\\\n\\begin{pmatrix}\ny_ie_i,x_ie_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{H}\\left(x,y\\right), \n%\\begin{pmatrix} -y_ie_i&-x_ie_i \\end{pmatrix}\\xi=0, i \\in \\mathcal{H}^\\le\\left(x,y\\right),\\\\\n\\begin{pmatrix}\n    0,e_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{N}(y)\n\\end{array}\n\\right.\\right\\}.\n\\]\n\n\n\\begin{definition}[Nondegenerate Karush-Kuhn-Tucker point]\nA Karush-Kuhn-Tucker point $(x, y)$ of $\\mathcal{S}$ with multipliers $(\\lambda, \\mu, \\eta, \\nu)$\nis called nondegenerate if \n\nND1: LICQ holds at $( x, y)$,\n\nND2: $ \\mu_{1,q}>0$, $q \\in Q_0\\left(x\\right)$, $ \\mu_{2,i}>0$, $i \\in \\mathcal{E}\\left(y\\right)$,\n    $\\eta^\\ge_i>0$, $i \\in \\mathcal{H}^\\ge(x,y)$,\n    $\\eta^\\le_i>0$, $i \\in \\mathcal{H}^\\le(x,y)$,\n    $\\nu_i >0$, $i \\in \\mathcal{N}(y)$, and $ \\mu_3>0$ if  $\\sum\\limits_{i=1}^{n} y_i =n-s$,\n\nND3: the matrix $D^2 L^\\mathcal{S}(x,y) \\restriction_{\\mathcal{T}^\\mathcal{S}_{(x,y)}}$ is nonsingular.\n\\end{definition}\n\n\\begin{definition}[Quadratic index]\nLet $(x,y)$ be a Karush-Kuhn-Tucker point of $\\mathcal{S}$ with unique multipliers $(\\lambda, \\mu, \\eta, \\nu)$.\nThe  number of negative eigenvalues of the matrix $D^2 L^\\mathcal{S}(x,y)\\restriction_{\\mathcal{T}^\\mathcal{S}_{(x,y)}}$ is called its quadratic index ($QI$).\n\\end{definition}\n\nFrom the proof of Theorem \\ref{thm:regul} in \\cite{laemmel:mpoc} also the convergence of the corresponding multipliers can be deduced.\n\n\n\\begin{remark}[Convergence of multipliers]\n\\label{rem:multipliers}\n Let $\\left(\\lambda^t,\\mu^t,\\eta^t,\\nu^t\\right)$ be the multipliers of the Karush-Kuhn-Tucker points $\\left(x^{t},y^{t}\\right)$ of $\\mathcal{S}$ and $\\left(\\bar \\lambda,\\bar \\mu,\\bar \\sigma, \\bar \\varrho\\right)$ of the T-stationary point $\\left(\\bar x, \\bar y\\right)$ of $\\mathcal{R}$ as in Theorem \\ref{thm:regul}. Due to MPOC-LICQ at $\\left(\\bar x, \\bar y\\right)$, we have: \n \na) $\\lim\\limits_{t \\to 0} \\lambda^t=\\bar \\lambda$,\n    $\\lim\\limits_{t \\to 0} \\mu^t=\\bar \\mu$,\n\nb) $\\lim\\limits_{t \\to 0} \\left(\\eta_{i}^{\\ge,t}-\\eta_{i}^{\\le,t}\\right)y_{i}^t=\\bar \\sigma_{1,i}$, $i \\in a_{01}\\left(\\bar x, \\bar y\\right)$,\n \n c) $\\lim\\limits_{t \\to 0} \\nu_{i}^{t}+ \\left(\\eta_{i}^{\\ge,t}-\\eta_{i}^{\\le,t}\\right)x_{i}^t=\\bar \\sigma_{2,i}$, $i \\in a_{10}\\left(\\bar x, \\bar y\\right)$,\n\nd) $\\lim\\limits_{t \\to 0} \\left(\\eta_{i}^{\\ge,t}-\\eta_{i}^{\\le,t}\\right)y_{i}^t=\\bar \\varrho_{1,i}$, $\\lim\\limits_{t \\to 0} \\nu_{i}^{t}+ \\left(\\eta_{i}^{\\ge,t}-\\eta_{i}^{\\le,t}\\right)x_{i}^t=\\bar \\varrho_{2,i}$, $i \\in a_{00}\\left(\\bar x, \\bar y\\right)$.\n\\end{remark}","units":[{"start":3268,"end":4021,"start_line":48,"end_line":56,"raw_tex":"In this paper, we study the Scholtes-type regularization method for the class of cardinality-constrained optimization problems:\n\\[\n\\mbox{CCOP}: \\quad\n\\min_{x} \\,\\, f(x)\\quad \\mbox{s.\\,t.} \\quad h(x)=0, \\quad g(x)\\ge 0, \\quad \\left\\|x\\right\\|_0\\le s\n\\]\nwith the feasible set given by equality, inequality, and cardinality constraints, where the so-called zero ”norm” $\\left\\|x\\right\\|_0 = \\left\\vert \\left\\{i \\in \\{1,\\ldots, n\\}\\; \\vert\\; x_i\\ne 0\\right\\}\\right\\vert$ is counting non-zero entries of $x$.\nHere, we assume that the objective function $f$, as well as the equality and inequality constraints \n$h=\\left(h_p, p \\in P\\right)$,\n$g=\\left(g_q, q \\in Q\\right)$ are twice continuously differentiable, and $s \\in \\{0,1,\\ldots, n-1\\}$ is an integer. \n","sha256_utf8":"348b1d38b9a7de96363fc9ca7c63ecfc90ce60163b1c7bcefbc741b8bec77a04"},{"start":8502,"end":8918,"start_line":92,"end_line":92,"raw_tex":"Our notation is standard. The cardinality of a finite set $A$ is denoted by $\\vert A \\vert$. The $n$-dimensional Euclidean space is denoted by $\\mathbb{R}^n$ with the coordinate vectors $e_i,i= 1,\\ldots, n$. The vector consisting of ones is denoted by $e$. Given a twice continuously differentiable function $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$, $\\nabla f$ denotes its gradient, and $D^2f$ stands for its Hessian.\n","sha256_utf8":"2529c2b903c6fb8bffac0f5ffb04035a2322d7c17a765d4470e9567379e00803"},{"start":8966,"end":10435,"start_line":95,"end_line":117,"raw_tex":"We associate with CCOP the regularized continuous reformulation from \\cite{laemmel:reform}:\n\\[\n     \\begin{array}{rl}\n    \\displaystyle \\mathcal{R}(c,\\varepsilon): \\quad \\min_{x,y} \\,\\, f(x) +c^Ty\\quad \\mbox{s.\\,t.} \n   &h(x)=0, \\quad g(x)\\ge 0, \\quad \\displaystyle \\sum_{i=1}^{n} y_i \\geq  n - s, \\quad \\\\\n   &x_i y_i =0, \\quad 0 \\leq y_i \\leq 1+\\varepsilon,  \\quad i=1, \\ldots, n,\n   \\end{array}\n\\]\nwhere the components of $c \\in \\R^n$ $c$ are positive and pairwise different, and $0 < \\varepsilon \\leq \\frac{1}{n-s}$.\nGiven a feasible point  $(\\bar x,\\bar y)$ of $\\mathcal{R}$, we define the index sets which correspond to the orthogonality type constraints $x_i y_i=0$, $y_i \\geq 0$:\n    \\[\n     \\begin{array}{c}\n        a_{01}\\left(\\bar x,\\bar y\\right)=\\left\\{i\\,\\left\\vert\\, \\bar x_i=0,\\bar y_i>0\\right.\\right\\}, \\quad \n    a_{10}\\left(\\bar x,\\bar y\\right)=\\left\\{i\\,\\left\\vert\\, \\bar x_i\\ne 0,\\bar y_i=0\\right.\\right\\}, \\\\\n    a_{00}\\left(\\bar x,\\bar y\\right)=\\left\\{i\\,\\left\\vert\\, \\bar x_i=0,\\bar y_i=0\\right.\\right\\}.       \n     \\end{array}\n    \\]\n%The $y$-components which attain the upper bound are stored in  \n%     \\[\\mathcal{E}(\\bar y)=\\left\\{i \\in\\{1,\\ldots,n\\}\\left\\vert\\,\\bar y_i=1+\\varepsilon\\right.\\right\\}.\\]\nThe index sets of the active inequality constraints will be denoted by\n\\[\n Q_0(\\bar x)=\\left\\{q \\in Q \\, \\left\\vert\\, g_q(\\bar x)=0\\right.\\right\\}, \\quad\\mathcal{E}(\\bar y)=\\left\\{i \\,\\left\\vert\\,\\bar y_i=1+\\varepsilon\\right.\\right\\}.\n\\]\n","sha256_utf8":"68cc0f1e18768210097ba4ca01104a30db26619dfb5d4e7be0487fbc94612b16"},{"start":11495,"end":12350,"start_line":127,"end_line":157,"raw_tex":"\\begin{definition}[MPOC-LICQ, \\cite{laemmel:reform}]\nWe say that a feasible point $(\\bar x,\\bar y)$ of $\\mathcal{R}$ satisfies the MPOC-tailored linear independence constraint qualification (MPOC-LICQ) if the following vectors are linearly independent:\n\\[\n\\begin{array}{c}\n\\begin{pmatrix}\n\\nabla h_p(\\bar x)\\\\0\n\\end{pmatrix}, p \\in P, \\quad \n\\begin{pmatrix}\n\\nabla g_q(\\bar x)\\\\0\n\\end{pmatrix}, q \\in Q_0(\\bar x), \\quad  \n\\begin{pmatrix}\n0\\\\\ne_i\n\\end{pmatrix},i\\in \\mathcal{E}(\\bar y), \\quad\n\\begin{pmatrix}\n0\\\\\ne\n\\end{pmatrix}\n\\mbox{ if } \\sum\\limits_{i=1}^{n} \\bar y_i = n - s,\n\\\\\n\\begin{pmatrix}\ne_i\\\\\n0\n\\end{pmatrix}, i \\in a_{01}\\left(\\bar x,\\bar y\\right)\\cup a_{00}\\left(\\bar x,\\bar y\\right), \\quad \n\\begin{pmatrix}\n0\\\\\ne_i\n\\end{pmatrix}, i \\in a_{10}\\left(\\bar x, \\bar y\\right)\\cup a_{00}\\left(\\bar x, \\bar y\\right).\n\\end{array}\n\\]\n\\end{definition}","sha256_utf8":"b52ecba7ac654c2c5d20767453f0fd664013a7d1ee2428a35b341c363193ff30"},{"start":12352,"end":14396,"start_line":159,"end_line":221,"raw_tex":"\\begin{definition}[T-stationary point, \\cite{laemmel:reform}]\n\\label{def:t-stat}\nA feasible point $(\\bar x,\\bar y)$ of $\\mathcal{R}$ is called T-stationary if there exists multipliers \n\\[\n\\begin{array}{l}\n\\bar \\lambda_p, p \\in P,\n\\bar \\mu_{1,q}, q\\in Q_0(\\bar x),\\bar \\mu_{2,i},i \\in \\mathcal{E}(\\bar y),\\bar \\mu_3,\\\\ \\bar \\sigma_{1,i}, i \\in a_{01}\\left(\\bar x,\\bar y\\right),\\bar \\sigma_{2,i}, i \\in a_{10}\\left(\\bar x,\\bar y\\right),\\bar \\varrho_{1,i},\\bar \\varrho_{2,i}, i \\in a_{00}\\left(\\bar x, \\bar y\\right),\n\\end{array}\n\\]\nsuch that the following conditions hold:\n\\begin{equation}\n   \\label{eq:tstat-1} \n   \\begin{array}{rcl}\n  \\begin{pmatrix}\n   \\nabla f(\\bar x)\\\\\n   c\n  \\end{pmatrix}&=& \\displaystyle\\sum\\limits_{p \\in P}\\bar \\lambda_p \n  \\begin{pmatrix}\n  \\nabla h_p(\\bar x)\\\\\n  0\n  \\end{pmatrix}+\n    \\sum\\limits_{q \\in Q_0(\\bar x)}\\bar \\mu_{1,q} \n    \\begin{pmatrix}\n    \\nabla g_q(\\bar x)\\\\\n    0\n    \\end{pmatrix}-\n    \\sum\\limits_{i\\in\\mathcal{E}(\\bar y)} \\bar \\mu_{2,i} \\begin{pmatrix}\n    0\\\\\n    e_i\n\\end{pmatrix}\\\\ \n    && +\n\\bar \\mu_3 \\begin{pmatrix}\n0\\\\\ne\n\\end{pmatrix}\\displaystyle +\\sum\\limits_{i \\in a_{01}\\left(\\bar x,\\bar y\\right)} \\bar \\sigma_{1,i}\n    \\begin{pmatrix}\n    e_{ i}\\\\\n    0\n    \\end{pmatrix}\n    +\\sum\\limits_{i \\in a_{10}\\left(\\bar x,\\bar y \\right)} \\bar \\sigma_{2,i}    \\begin{pmatrix}\n    0\\\\\n    e_{ i}\n    \\end{pmatrix}\\\\ \n    &&\\displaystyle+\\sum\\limits_{i \\in a_{00}\\left(\\bar x, \\bar y\\right)} \\left(\\bar \\varrho_{1,i}\n    \\begin{pmatrix}\n   e_{ i}\\\\\n    0\n    \\end{pmatrix}\n   +\\bar \\varrho_{2,i}\n    \\begin{pmatrix}\n    0\\\\\n    e_{ i}\n    \\end{pmatrix}\\right), \\end{array}\n\\end{equation}\n\\begin{equation}\n   \\label{eq:tstat-2} \\bar \\mu_{1,q} \\ge 0,q\\in Q_0\\left(\\bar x\\right),\n   \\bar \\mu_{2,i} \\ge 0, i\\in \\mathcal{E}(\\bar y),\\bar \\mu_3\\ge 0, \\bar\\mu_3\\left(\n   \\sum\\limits_{i=1}^{n} \\bar y_i -(n-s)\\right)=0,\n\\end{equation}\n\\begin{equation}\n   \\label{eq:tstat-3} \\bar \\varrho_{1,i}=0 \\mbox{ or }\\bar \\varrho_{2,i}\\le 0, i \\in a_{00}\\left(\\bar x,\\bar y\\right).\n\\end{equation}\n\\end{definition}","sha256_utf8":"45d0493eef254723b50d1eda72bfab429bad0ae48140682d537d9036879543ea"},{"start":14397,"end":15885,"start_line":222,"end_line":270,"raw_tex":"We define the appropriate Lagrange function:\n\\[\n        \\begin{array}{rcl}\n        L^\\mathcal{R}(x,y)&=& \\displaystyle\n   f(x)+\n   c^T y -\n        \\sum\\limits_{p \\in P}\\bar \\lambda_p\n  h_p( x)-\n    \\sum\\limits_{q \\in Q_0(\\bar x)}\\bar \\mu_{1,q} \n    g_q(x) \\\\  & &\\displaystyle+\\sum\\limits_{i\\in\\mathcal{E}(\\bar y)} \\bar \\mu_{2,i} \n   \\left( y_i- (1+\\varepsilon)\\right)\n-\n\\bar \\mu_3 \n \\left(\\sum\\limits_{i=1}^{n} y_i - (n-s)\\right)\n   \\\\ \n    &&\\displaystyle -\\sum\\limits_{i \\in a_{01}\\left(\\bar x, \\bar y\\right)} \\bar \\sigma_{1,i}\n     x_{ i}\n    -\\sum\\limits_{i \\in a_{10}\\left(\\bar x, \\bar y\\right)} \\bar \\sigma_{2,i}   \n     y_{ i} -\\sum\\limits_{i \\in a_{00}\\left(\\bar x, \\bar y \\right)} \\left(\\bar \\varrho_{1,i}\n    x_{ i}\n   +\\bar \\varrho_{2,i}\n     y_{ i}\n    \\right).\n    \\end{array}\n\\]\nMoreover, we set for the corresponding tangent space:\n\n\\[\n    \\mathcal{T}^{\\mathcal{R}}_{(\\bar x,\\bar y)} =\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\nDh_p(\\bar x), 0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(\\bar x), 0\\end{pmatrix}\\xi=0,q \\in Q_0(\\bar x),\\\\\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(\\bar y), \n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0 \\mbox{ if } \\displaystyle \\sum_{i=1}^{n} \\bar y_i = n - s,\n\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y) \\cup a_{01}(\\bar x,\\bar y),\\\\\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y) \\cup a_{10}(\\bar x,\\bar y)\n\\end{array}\n\\right.\\right\\}.\n\\]\n","sha256_utf8":"a957e3f8dd163dc04f5b5a447c73347b15d5dd1082ac484a858ee927cd79828d"},{"start":15886,"end":16898,"start_line":272,"end_line":288,"raw_tex":"\\begin{definition}[Nondegenerate T-stationary point, \\cite{laemmel:reform}]\nA T-stationary point $(\\bar x,\\bar y)$ of $\\mathcal{R}$ with multipliers $(\\bar \\lambda, \\bar \\mu, \\bar \\sigma, \\bar \\varrho)$ is called nondegenerate if\n\nNDT1: MPOC-LICQ holds at $(\\bar x,\\bar y)$,\n\nNDT2: $\\bar \\mu_{1,q}>0$, $q \\in Q_0\\left(\\bar x\\right)$, $\\bar \\mu_{2,i}>0$, $i \\in \\mathcal{E}\\left(\\bar y\\right)$, and $\\bar \\mu_3>0$ if  $\\sum\\limits_{i=1}^{n} \\bar  y_i =n-s$,\n\nNDT3: $\\bar\\varrho_{1,i}\\ne 0$ and $\\bar\\varrho_{2,i}< 0$, $i\\in a_{00}\\left(\\bar x,\\bar y\\right)$,\n    \nNDT4: the matrix $D^2 L^{\\mathcal{R}}(\\bar x,\\bar y)\\restriction_{\\mathcal{T}^\\mathcal{R}_{(\\bar x,\\bar y)}}$ is nonsingular.\n\n\\noindent\nFor a nondegenerate T-stationary point we eventually use an additional condition:\n%\\item [] NDT5: if $a_{00}\\left(\\bar x,\\bar y\\right)\\not = \\emptyset$, then $\\bar \\sigma_{1,i_{01}}\\ne 0$ for all $i_{01}\\in a_{01}(\\bar x, \\bar y)$.\n\nNDT6: $\\bar \\sigma_{1,i}\\ne 0$, $i\\in a_{01}(\\bar x, \\bar y)$.\n\\end{definition}","sha256_utf8":"c90cd99097bec75cfa77e4dbd09d44ad130c1ada5ebd01ab770c76a504238867"},{"start":16900,"end":17492,"start_line":290,"end_line":292,"raw_tex":"\\begin{definition}[T-index, \\cite{laemmel:reform}]\nLet $(\\bar x,\\bar y)$ be a nondegenerate T-stationary point of $\\mathcal{R}$ with unique multipliers $\\left(\\bar \\lambda,\\bar \\mu, \\bar \\sigma,\\bar \\varrho\\right)$. The number of negative eigenvalues of the matrix $D^2 L^{\\mathcal{R}}(\\bar x,\\bar y)\\restriction_{\\mathcal{T}^\\mathcal{R}_{(\\bar x,\\bar y)}}$ is called its quadratic index ($QI$). The cardinality of $a_{00}\\left(\\bar x,\\bar y\\right)$ is called the biactive index ($BI$) of $(\\bar x,\\bar y)$. We define the T-index ($TI$) as the sum of both, i.\\,e. $TI=QI+BI$.\n\\end{definition}","sha256_utf8":"17778710c27216cc1e3080168a41bb09472670aee023c5e062d8245da72c1dc7"},{"start":22596,"end":24219,"start_line":362,"end_line":385,"raw_tex":"Let us now regularize the orthogonality type constraints in $\\mathcal{R}$ by using the Scholte's idea, cf. \\cite{scholtes:2001}: \n\\[\n     \\begin{array}{rl}\n    \\displaystyle \\mathcal{S}(t): \\quad \\min_{x,y} \\,\\, f(x) +c^Ty\\quad \\mbox{s.\\,t.} \n   &h(x)=0, \\quad g(x)\\ge 0, \\quad\\displaystyle \\sum_{i=1}^{n} y_i \\geq  n - s, \\\\\n   & -t\\le x_i y_i \\le t, \\quad 0 \\leq y_i \\leq 1+\\varepsilon,  \\quad \n         i=1, \\ldots, n,\n   \\end{array}\n\\]\nwhere $t>0$. \n%Sometimes, we write $S(t)$ if the dependence on $t$ is highlighted. \nNote that $\\mathcal{S}$ from above falls into the scope of nonlinear programming, i.e. just smooth equality and inequality constraints are present. The notation for the sets %$a_{00}\\left(x, y\\right)$, $a_{01}\\left(x, y\\right)$, $a_{10}\\left(x, y\\right)$, \n$Q_0(x)$ and $\\mathcal{E}(y)$, which were used for $\\mathcal{R}$, will be used here again.\nFurthermore, we define for a feasible point $\\left( x, y\\right)$ of $\\mathcal{S}$ the index set of vanishing $y$-components as well as the index sets of active relaxed orthogonality type constraints:\n\\[\n\\mathcal{N}( y)=\\left\\{i\\, \\left\\vert\\,  y_i=0\\right.\\right\\},\n\\quad\n\\mathcal{H}^{\\ge}\\left(x, y\\right)=\\left\\{i\\,\\left\\vert \\,x_i  y_i=-t\\right.\\right\\}, \\mathcal{H}^{\\le}\\left( x, y\\right)=\\left\\{i\\,\\left\\vert \\, x_i  y_i=t\\right.\\right\\}.\n\\]\nWe also eventually use the following index sets:\n\\[\n\\mathcal{H}\\left(x, y\\right)=\\mathcal{H}^\\geq\\left(x, y\\right) \\cup \\mathcal{H}^\\leq\\left( x, y\\right), \\quad \n\\mathcal{O}\\left( x, y\\right)=\\left(\\mathcal{E}\\left( y\\right)\\cup \\mathcal{N}\\left(y\\right)\\cup \\mathcal{H}\\left( x, y\\right)\\right)^c.\n\\]\n","sha256_utf8":"e2ce68583dd0784228ec2d4978c6cff09adbc75c123bc7563ede2326d27eb11a"},{"start":24362,"end":25053,"start_line":389,"end_line":418,"raw_tex":"\\begin{definition}[LICQ]\nWe say that a feasible point $(x, y)$ of $\\mathcal{S}$ satisfies the linear independence constraint qualification (LICQ) if the following vectors are linearly independent:\n\\[\n\\begin{array}{c}\n      \\begin{pmatrix}\n\\nabla h_p(x)\\\\0\n\\end{pmatrix}, p \\in P, \\quad \n\\begin{pmatrix}\n\\nabla g_q(x)\\\\0\n\\end{pmatrix}, q \\in Q_0(x), \\quad  \n\\begin{pmatrix}\n0\\\\\ne_i\n\\end{pmatrix},i\\in \\mathcal{E}(y),\n\\\\ \\begin{pmatrix}\n0\\\\\ne\n\\end{pmatrix}\n\\mbox{ if } \\sum\\limits_{i=1}^{n} y_i = n - s, \\quad\n\\begin{pmatrix}\ny_i e_i\\\\\nx_i e_i\n\\end{pmatrix}, i \\in \\mathcal{H}\\left(x,y\\right), \\quad \n\\begin{pmatrix}\n0\\\\\ne_i\n\\end{pmatrix}, i \\in \\mathcal{N}(y).\n\\end{array}\n\\]\n\\end{definition}","sha256_utf8":"52b085f26c648f2aea8cf7732f25498de5741679cebaf928932002f584e73a0b"},{"start":32014,"end":33613,"start_line":623,"end_line":681,"raw_tex":"\\begin{definition}[Karush-Kuhn-Tucker point]\n    A feasible point $(x, y)$ of $\\mathcal{S}$ is called Kurush-Kuhn-Tucker if there exist multipliers \n\\[\n\\lambda_p, p \\in P,\n \\mu_{1,q}, q\\in Q_0( x), \\mu_{2,i},i \\in \\mathcal{E}( y), \\mu_3,\n\\eta^\\ge_i, \\eta^\\le_i, \\nu_i, i\\in\\left\\{1,\\ldots,n\\right\\},\n\\]\nsuch that the following conditions hold:\n\\begin{equation}\n   \\label{eq:kkt-1} \n   \\begin{array}{rcl}\n  \\begin{pmatrix}\n   \\nabla f( x)\\\\\n   c\n  \\end{pmatrix}&=& \\displaystyle \\sum\\limits_{p \\in P} \\lambda_p \n  \\begin{pmatrix}\n  \\nabla h_p( x)\\\\\n  0\n  \\end{pmatrix}+\n    \\sum\\limits_{q \\in Q_0( x)} \\mu_{1,q} \n    \\begin{pmatrix}\n    \\nabla g_q( x)\\\\\n    0\n    \\end{pmatrix}\n   - \\sum\\limits_{i\\in\\mathcal{E}( y)}  \\mu_{2,i} \\begin{pmatrix}\n    0\\\\\n     e_i\n\\end{pmatrix}+\n \\mu_3 \\begin{pmatrix}\n0\\\\\ne\n\\end{pmatrix} \\\\ \n    && \\displaystyle\n+ \\sum\\limits_{i\\in\\mathcal{H}^\\ge( x,  y)}  \\eta^\\ge_{i} \\begin{pmatrix}\n    y_ie_i\\\\\n   x_ie_i\n\\end{pmatrix}\n-\\sum\\limits_{i\\in\\mathcal{H}^\\le( x,  y)}  \\eta^\\le_{i} \\begin{pmatrix}\n    y_ie_i\\\\\n    x_ie_i\n\\end{pmatrix}\n+\n    \\sum\\limits_{i\\in\\mathcal{N}( y)}  \\nu_{i} \\begin{pmatrix}\n    0\\\\\n    e_i\n\\end{pmatrix},\n\\end{array}\n\\end{equation}\n\\begin{equation}\n   \\label{eq:kkt-2}  \\mu_{1,q} \\ge 0,q\\in Q_0\\left( x\\right),\n    \\mu_{2,i} \\ge 0, i\\in \\mathcal{E}(y),\n    \\mu_3\\ge 0, \\mu_3\\left(\n   \\sum\\limits_{i=1}^{n}  y_i -(n-s)\\right)=0,\n\\end{equation}\n\\begin{equation}\n   \\label{eq:kkt-3} \n  \\eta^\\ge_i\\ge 0,i \\in \\mathcal{H}^\\ge( x,  y), \\eta^\\le_i\\ge 0,i \\in \\mathcal{H}^\\le( x,  y), \\nu_i\\ge 0, i \\in \\mathcal{N}( y).\n\\end{equation}\n\\end{definition}","sha256_utf8":"eab33f17a0c5df5616c4c8f6b492afdf2b013d0317289cea29b66a55490e01fd"},{"start":33614,"end":34961,"start_line":682,"end_line":729,"raw_tex":"We again define the Lagrange function as\n\n\\[\n        \\begin{array}{rcl}\n        L^\\mathcal{S}(x,y)&=& \\displaystyle\n   f(x)+c^Ty\n       - \\sum\\limits_{p \\in P} \\lambda_p \n  h_p(x)-\n    \\sum\\limits_{q \\in Q_0(x)} \\mu_{1,q} \n    g_q(x)\n      \\\\&& \\displaystyle +\\sum\\limits_{i\\in\\mathcal{E}(y)}\n   \\mu_{2,i}\n    \\left(y_i - (1+\\varepsilon)\\right)\n -\\mu_3\\left(\\sum\\limits_{i=1}^n y_i - (n-s)\\right)  \\\\ \n    && \\displaystyle\n- \\sum\\limits_{i\\in\\mathcal{H}^\\ge( x,  y)}  \\eta^\\ge_{i}\n    \\left(x_iy_i +t \\right)\n+\\sum\\limits_{i\\in\\mathcal{H}^\\le( x,  y)}  \\eta^\\le_{i}\n    \\left(x_iy_i-t\\right)\n-\n    \\sum\\limits_{i\\in\\mathcal{N}( y)}  \\nu_{i}\n    y_i.\n\\end{array}\n\\]\nThe tangent space is given by\n\n\\[\n    \\mathcal{T}^\\mathcal{S}_{(x,y)} =\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\nDh_p( x),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q( x),0\\end{pmatrix}\\xi=0,q \\in Q_0( x),\\\\ \\displaystyle\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(y),\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0 \\mbox{ if } \\sum_{i=1}^{n} y_i = n - s,\\\\\n\\begin{pmatrix}\ny_ie_i,x_ie_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{H}\\left(x,y\\right), \n%\\begin{pmatrix} -y_ie_i&-x_ie_i \\end{pmatrix}\\xi=0, i \\in \\mathcal{H}^\\le\\left(x,y\\right),\\\\\n\\begin{pmatrix}\n    0,e_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{N}(y)\n\\end{array}\n\\right.\\right\\}.\n\\]\n","sha256_utf8":"0959f7eab6e0630a60e423efb8a94e976cc9c54cf3d5ad4d9812646242968bcd"},{"start":34962,"end":35592,"start_line":731,"end_line":743,"raw_tex":"\\begin{definition}[Nondegenerate Karush-Kuhn-Tucker point]\nA Karush-Kuhn-Tucker point $(x, y)$ of $\\mathcal{S}$ with multipliers $(\\lambda, \\mu, \\eta, \\nu)$\nis called nondegenerate if \n\nND1: LICQ holds at $( x, y)$,\n\nND2: $ \\mu_{1,q}>0$, $q \\in Q_0\\left(x\\right)$, $ \\mu_{2,i}>0$, $i \\in \\mathcal{E}\\left(y\\right)$,\n    $\\eta^\\ge_i>0$, $i \\in \\mathcal{H}^\\ge(x,y)$,\n    $\\eta^\\le_i>0$, $i \\in \\mathcal{H}^\\le(x,y)$,\n    $\\nu_i >0$, $i \\in \\mathcal{N}(y)$, and $ \\mu_3>0$ if  $\\sum\\limits_{i=1}^{n} y_i =n-s$,\n\nND3: the matrix $D^2 L^\\mathcal{S}(x,y) \\restriction_{\\mathcal{T}^\\mathcal{S}_{(x,y)}}$ is nonsingular.\n\\end{definition}","sha256_utf8":"082fd403dc6dbdc7f685b1ad9efdd2841f14150d1b70cca35d6f3988937d6057"},{"start":35594,"end":35917,"start_line":745,"end_line":748,"raw_tex":"\\begin{definition}[Quadratic index]\nLet $(x,y)$ be a Karush-Kuhn-Tucker point of $\\mathcal{S}$ with unique multipliers $(\\lambda, \\mu, \\eta, \\nu)$.\nThe  number of negative eigenvalues of the matrix $D^2 L^\\mathcal{S}(x,y)\\restriction_{\\mathcal{T}^\\mathcal{S}_{(x,y)}}$ is called its quadratic index ($QI$).\n\\end{definition}","sha256_utf8":"183908b18c3317474bedda15a2fc2e9a337556f1cc937a81c963099ed67d3337"},{"start":40426,"end":40561,"start_line":819,"end_line":819,"raw_tex":"From the proof of Theorem \\ref{thm:regul} in \\cite{laemmel:mpoc} also the convergence of the corresponding multipliers can be deduced.\n","sha256_utf8":"7f6bd18e0bf9305b9f7c3e36c297c3eb5578cabb369fb1d710a6b7951bc3e488"},{"start":40562,"end":41674,"start_line":821,"end_line":833,"raw_tex":"\\begin{remark}[Convergence of multipliers]\n\\label{rem:multipliers}\n Let $\\left(\\lambda^t,\\mu^t,\\eta^t,\\nu^t\\right)$ be the multipliers of the Karush-Kuhn-Tucker points $\\left(x^{t},y^{t}\\right)$ of $\\mathcal{S}$ and $\\left(\\bar \\lambda,\\bar \\mu,\\bar \\sigma, \\bar \\varrho\\right)$ of the T-stationary point $\\left(\\bar x, \\bar y\\right)$ of $\\mathcal{R}$ as in Theorem \\ref{thm:regul}. Due to MPOC-LICQ at $\\left(\\bar x, \\bar y\\right)$, we have: \n \na) $\\lim\\limits_{t \\to 0} \\lambda^t=\\bar \\lambda$,\n    $\\lim\\limits_{t \\to 0} \\mu^t=\\bar \\mu$,\n\nb) $\\lim\\limits_{t \\to 0} \\left(\\eta_{i}^{\\ge,t}-\\eta_{i}^{\\le,t}\\right)y_{i}^t=\\bar \\sigma_{1,i}$, $i \\in a_{01}\\left(\\bar x, \\bar y\\right)$,\n \n c) $\\lim\\limits_{t \\to 0} \\nu_{i}^{t}+ \\left(\\eta_{i}^{\\ge,t}-\\eta_{i}^{\\le,t}\\right)x_{i}^t=\\bar \\sigma_{2,i}$, $i \\in a_{10}\\left(\\bar x, \\bar y\\right)$,\n\nd) $\\lim\\limits_{t \\to 0} \\left(\\eta_{i}^{\\ge,t}-\\eta_{i}^{\\le,t}\\right)y_{i}^t=\\bar \\varrho_{1,i}$, $\\lim\\limits_{t \\to 0} \\nu_{i}^{t}+ \\left(\\eta_{i}^{\\ge,t}-\\eta_{i}^{\\le,t}\\right)x_{i}^t=\\bar \\varrho_{2,i}$, $i \\in a_{00}\\left(\\bar x, \\bar y\\right)$.\n\\end{remark}","sha256_utf8":"061ed992b5e4a45ae7e086e729bf33e6f2da805134e800853d8d82aecab8e3d2"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}[Convergence from $\\mathcal{S}$ to $\\mathcal{R}$ again]\n\\label{thm:kktsequence}\n%Let Assumption \\ref{ass:c} be fulfilled and $\\varepsilon \\le \\frac{1}{n-s}$. \nSuppose that a sequence of nondegenerate Karush-Kuhn-Tucker points $(x^{t},y^{t})$ of \n$\\mathcal{S}$ with quadratic index $m$ converges to $\\left(\\bar x,\\bar y\\right)$ for $t \\to 0$. If $(\\bar x,\\bar y)$ is a nondegenerate T-stationary point of $\\mathcal{R}$, then we have for its T-Index: \n\\[\n\\max\\left\\{m - \\left\\vert\\left\\{i\\in a_{01}\\left(\\bar x,\\bar y\\right)\\,\\left\\vert\\,\\bar \\sigma_{1,i}=0\\right.\\right\\} \\right\\vert, 0\\right\\}\\le TI \\leq m.\n\\]\nIf additionally NDT6 holds at $(\\bar x,\\bar y)$, then the indices coincide, i.e. $TI= m$.\n\\end{theorem}","units":[{"start":50309,"end":51038,"start_line":995,"end_line":1004,"raw_tex":"\\begin{theorem}[Convergence from $\\mathcal{S}$ to $\\mathcal{R}$ again]\n\\label{thm:kktsequence}\n%Let Assumption \\ref{ass:c} be fulfilled and $\\varepsilon \\le \\frac{1}{n-s}$. \nSuppose that a sequence of nondegenerate Karush-Kuhn-Tucker points $(x^{t},y^{t})$ of \n$\\mathcal{S}$ with quadratic index $m$ converges to $\\left(\\bar x,\\bar y\\right)$ for $t \\to 0$. If $(\\bar x,\\bar y)$ is a nondegenerate T-stationary point of $\\mathcal{R}$, then we have for its T-Index: \n\\[\n\\max\\left\\{m - \\left\\vert\\left\\{i\\in a_{01}\\left(\\bar x,\\bar y\\right)\\,\\left\\vert\\,\\bar \\sigma_{1,i}=0\\right.\\right\\} \\right\\vert, 0\\right\\}\\le TI \\leq m.\n\\]\nIf additionally NDT6 holds at $(\\bar x,\\bar y)$, then the indices coincide, i.e. $TI= m$.\n\\end{theorem}","sha256_utf8":"376fd63e5d763a395a796f325f7a17246030d2b5c7ce7e65334ee2be1ffd26a4"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}[Auxiliary $y$-variables in $\\mathcal{R}$, \\cite{laemmel:reform}]\n\\label{lem:a01} \n%Assume that the components of $c$ are pairwise different and $\\varepsilon\\le\\frac{1}{n-s}$. \nLet $(\\bar x,\\bar y)$ be a T-stationary point of $\\mathcal{R}$, then it holds:\n\na) the summation inequality constraint is active, i.\\,e. $ \\sum\\limits_{i=1}^{n} \\bar y_i =n - s$,\n\nb) the index set $a_{01}(\\bar x,\\bar y)$ consists of exactly $n-s$ elements,\n   \nc) $n-s-1$ components of $\\bar y$ are equal to $1+\\varepsilon$, one component is equal to $1-(n-s-1)\\varepsilon$, and $s$ remaining components vanish.\n\\end{lemma}\n\n\\begin{proof}\n a)\nLet $(\\bar x,\\bar y)$ be a T-stationary point of $\\mathcal{R}$\nand $\\displaystyle \\sum_{i=1}^{n} \\bar y_i >n - s$. Then, there exist multipliers\n$(\\bar \\lambda, \\bar \\mu, \\bar \\sigma, \\bar \\varrho)$\nsuch that (\\ref{eq:tstat-1})--(\\ref{eq:tstat-3}) are fulfilled.\nSince $\\bar \\mu_3=0$, we have that the $(n+i)$-th row of (\\ref{eq:tstat-1}) reads as\n\\[\nc_i=\\left\\{\n\\begin{array}{ll}\n     -\\bar \\mu_{2,i},&\\mbox{for }i \\in \\mathcal{E}(\\bar y),  \\\\\n     \\bar \\sigma_{2,i}, &\\mbox{for }i \\in a_{10}\\left(\\bar x,\\bar y\\right),\\\\\n     \\bar \\varrho_{2,i}, &\\mbox{for }i\\in a_{00}\\left(\\bar x,\\bar y\\right),\\\\\n     0,&\\mbox{else}.\n\\end{array}\\right.\n\\]\nDue to $c>0$, the sets $a_{01}\\left(\\bar x,\\bar y\\right)$ and $\\mathcal{E}(\\bar y)$ have to be equal and, moreover,\ndue to $\\bar \\mu_{2,i}\\ge 0$, they have to be empty. But then, clearly, $ \\sum_{i=1}^{n} \\bar  y_i =0<n - s$, a contradiction.\n\nb)\nSince $(\\bar x,\\bar y)$ is a T-stationary point, there exist\n$(\\bar \\lambda, \\bar \\mu, \\bar \\sigma, \\bar \\varrho)$\nsuch that (\\ref{eq:tstat-1})--(\\ref{eq:tstat-3}). By the proof of statement a) we can conclude that\n$\\bar \\mu_3 >0$. Hence,\nthe $(n+i)$-th row reads as\n\\begin{equation}\n    \\label{eq:k-throw}\nc_i=\\left\\{\n\\begin{array}{ll}\n     -\\bar \\mu_{2,i}+\\bar \\mu_3,&\\mbox{for }i \\in \\mathcal{E}(\\bar y),  \\\\\n     \\bar \\sigma_{2,i}+\\bar \\mu_3, &\\mbox{for }i \\in a_{10}\\left(\\bar x,\\bar y\\right),\\\\\n     \\bar \\varrho_{2,i}+\\bar \\mu_3, &\\mbox{for }i\\in a_{00}\\left(\\bar x,\\bar y\\right),\\\\\n     \\bar \\mu_3,&\\mbox{else}.\n\\end{array}\\right.\n\\end{equation}\nLet us assume that the index set $a_{01}(\\bar x,\\bar y)$ consists of fewer than $n-s$ elements. Then, we have by using the assumption on $\\varepsilon$:\n\\[ \\sum_{i=1}^{n} \\bar y_i \\le (n-s-1)\\cdot (1+\\varepsilon)\\le n-s-1+\\frac{n-s-1}{n-s} <n - s,\n\\]\na contradiction to feasibility. \nLet us assume that the index set $a_{01}(\\bar x,\\bar y)$ consists of more than $n-s$ elements instead.\nSince the components of $c$ are assumed to be pairwise different, we see from (\\ref{eq:k-throw}) that there exists at most one element in $a_{01}\\left(\\bar x,\\bar y\\right)\\backslash \\mathcal{E}(\\bar y)$ and, consequently, there are at least $n-s$ elements in $\\mathcal{E}(\\bar y)$. Therefore, we have:\n\\[\n\\sum_{i=1}^{n} \\bar y_i \\geq (n-s)\\cdot (1+\\varepsilon) > n-s,\n\\]\nwhich contradicts a). \n\nc) Due to b), $a_{01}(\\bar x,\\bar y)$ consists of exactly $n-s$ elements. We conclude as in b) that there is at most one element in $a_{01}\\left(\\bar x,\\bar y\\right)\\backslash \\mathcal{E}(\\bar y)$. In view of statement a), $\\mathcal{E}(\\bar y)$ cannot consist of $n-s$ elements and, thus, must consist of $n-s-1$ elements. Hence, the statement follows immediately.\n\\end{proof}\n\n\\begin{lemma}[Auxiliary $y$-variables in $\\mathcal{S}$]\n\\label{lem:EandH}\n%Suppose Assumption \\ref{ass:c} to be fulfilled.\nLet $(x, y)$ be a Karush-Kuhn-Tucker point of $\\mathcal{S}$. Then, it holds:\n\na) the summation inequality constraint is active, i.\\,e. $\\sum\\limits_{i=1}^n  y_i=n-s$,\n\nb) the index set $\\mathcal{E}( y) \\cup \\mathcal{H}(x, y)$ consists of at least $n-s-1$ elements,\nand the index set $\\mathcal{N}( y)$ consists of at most $s$ elements. Additionally, there is at most one index, that does not belong to any of these sets, i.\\,e. $\\left\\vert \\mathcal{O}\\left( x,  y\\right)\\right\\vert \\le 1$.\n\\end{lemma}\n\n\\begin{proof}\na) Let $(x, y)$ be a Karush-Kuhn-Tucker point of $\\mathcal{S}$ and $\\sum\\limits_{i=1}^n  y_i>n-s$. Then, there exist multipliers $(\\lambda,  \\mu,  \\eta,  \\nu)$, such that (\\ref{eq:kkt-1})--(\\ref{eq:kkt-3}) are fulfilled. Since $ \\mu_3=0$, we have that the $(n+i)$-th row of (\\ref{eq:kkt-1}) reads as\n\\[\nc_i=\\left\\{\n\\begin{array}{ll}\n     - \\mu_{2,i},&\\mbox{for }i \\in \\mathcal{E}( y)\\backslash\\mathcal{H}( x, y),  \\\\\n    - \\mu_{2,i}+ \\eta^\\ge_i  x_i, &\\mbox{for }i \\in \\mathcal{E}( y) \\cap \\mathcal{H}^\\ge( x, y),\\\\\n    - \\mu_{2,i}- \\eta^\\le_i  x_i, &\\mbox{for }i \\in \\mathcal{E}( y) \\cap  \\mathcal{H}^\\le( x, y),\\\\\n \\eta^\\ge_i  x_i, &\\mbox{for }i \\in \\mathcal{H}^\\ge( x, y)\\backslash \\mathcal{E}( y),\\\\\n- \\eta^\\le_i  x_i, &\\mbox{for }i \\in \\mathcal{H}^\\le( x, y)\\backslash \\mathcal{E}( y),\\\\\n \\nu_i,&\\mbox{for }i \\in \\mathcal{N}( y),\\\\\n     0,&\\mbox{else}.\n\\end{array}\\right.\n\\]\nDue to (\\ref{eq:kkt-2}), (\\ref{eq:kkt-3}), and $c>0$, it must hold that $i\\in \\mathcal{N}( y)$ for all $i \\in \\left\\{1,\\ldots,n\\right\\}$. This, however, contradicts $\\sum\\limits_{i=1}^n  y_i>n-s$.\n\nb) \n%Since $( x, y)$ is a Karush-Kuhn-Tucker point of RCCOP$(c,\\varepsilon,t)$ there exist multipliers $( \\lambda,  \\mu,  \\eta^\\ge,  \\eta^\\le,  \\nu)$ such that (\\ref{eq:kkt-1})--(\\ref{eq:kkt-3}). \nAs in the proof of statement a), we conclude that $ \\mu_3>0$ for a Karush-Kuhn-Tucker point $( x, y)$ of $\\mathcal{S}$. Hence, the $(n+i)$-th row now reads as\n\\begin{equation}\n\\label{eq:kth-rowkkt}    \nc_i=\\left\\{\n\\begin{array}{ll}\n     - \\mu_{2,i}+ \\mu_3,&\\mbox{for }i \\in \\mathcal{E}( y)\\backslash \\mathcal{H}( x, y),  \\\\\n    - \\mu_{2,i}+ \\mu_3+ \\eta^\\ge_i  x_i, &\\mbox{for }i \\in \\mathcal{E}( y) \\cap \\mathcal{H}^\\ge( x, y),\\\\\n    - \\mu_{2,i}+ \\mu_3- \\eta^\\le_i  x_i, &\\mbox{for }i \\in \\mathcal{E}( y) \\cap  \\mathcal{H}^\\le( x, y),\\\\\n \\mu_3+ \\eta^\\ge_i  x_i, &\\mbox{for }i \\in \\mathcal{H}^\\ge( x, y)\\backslash \\mathcal{E}( y),\\\\\n \\mu_3- \\eta^\\le_i  x_i, &\\mbox{for }i \\in \\mathcal{H}^\\le( x, y)\\backslash \\mathcal{E}( y),\\\\\n \\mu_3+ \\nu_i,&\\mbox{for }i \\in \\mathcal{N}( y),\\\\\n      \\mu_3,&\\mbox{else}.\n\\end{array}\\right.\n\\end{equation}\nIt follows from (\\ref{eq:kth-rowkkt}) and  the components of $c$ being pairwise different that there can be at most one element ${\\bar i} \\in \\mathcal{O}\\left( x,  y\\right)$. \nIf $\\mathcal{E}( y) \\cup \\mathcal{H}( x,  y)$ consists of fewer than $n-s-1$ elements, we get:\n\\[\n\\sum\\limits_{i=1}^n  y_i\\le (n-s-2)\\cdot (1+\\varepsilon)+y_{\\bar i}<(n-s-1)\\cdot(1+\\varepsilon)<n-s,\n\\]\na contradiction.\nFinally, we assume that $\\mathcal{N}( y)$ consists of more than $s$ elements.\nIn this case, there are at most $n-s-1$ nonvanishing components of $y$. Consequently, \n\\[\n\\sum\\limits_{i=1}^n  y_i\\le (n-s-1)\\cdot (1+\\varepsilon)<n-s\n\\]\nprovides a contradiction.\n\\end{proof}\n\n\\begin{theorem}[Convergence from $\\mathcal{S}$ to $\\mathcal{R}$, cf. \\cite{laemmel:mpoc}]\n\\label{thm:regul}\n%Let $\\left\\{^t\\right\\}_{{t}\\in \\N}$  be a sequence of positive regularization parameters converging to zero. \nSuppose that a sequence of Karush-Kuhn-Tucker points $(x^{t},y^{t})$ of \n$\\mathcal{S}$ converges to $\\left(\\bar x,\\bar y\\right)$ for $t \\rightarrow 0$. If MPOC-LICQ holds at $\\left(\\bar x,\\bar y\\right)$, then it is a T-stationary point of $\\mathcal{R}$.\n\\end{theorem}\n\n\\begin{lemma}[Active index sets]\n\\label{lem:QandE}\nSuppose a sequence of Karush-Kuhn-Tucker points $\\left(x^{t},y^{t}\\right)$ of $\\mathcal{S}(t)$ converges to $\\left(\\bar x,\\bar y\\right)$ for $t \\to 0$. Moreover, let $(\\bar x,\\bar y)$ be a nondegenerate T-stationary point of $\\mathcal{R}$. Then, for all sufficiently small $t$ it holds:\n\na) $Q_0\\left(\\bar x\\right)=Q_0\\left(x^{t}\\right)$,\n\nb) $\\mathcal{E}\\left(\\bar y\\right)=\\mathcal{E}\\left(y^{t}\\right) $,\n\nc) $a_{00}\\left(\\bar x, \\bar y\\right) \\subset \\mathcal{H}\\left(x^t,y^t\\right)$,\n\nd) $\\mathcal{N}\\left(y^t\\right)\\subset\na_{10}\\left(\\bar x,\\bar y\\right)\\subset \n\\mathcal{N}\\left(y^t\\right) \\cup \\mathcal{H}\\left(x^t,y^t\\right)$.    \n\n\\end{lemma}\n\n\\begin{proof}\na) We start by proving $Q_0\\left(\\bar x\\right)=Q_0\\left(x^{t}\\right)$.\nDue to continuity arguments, we have $Q_0\\left(x^{t}\\right)\\subset Q_0\\left(\\bar x\\right)$ for all sufficiently small $t$.\nLet us now assume that there exists\n$\\bar i \\in  Q_0\\left(\\bar x\\right)\\backslash Q_0\\left(x^{t}\\right)$ along a subsequence.\nHence, for the corresponding multipliers it holds $\\mu_{\\bar i}^t=0$.\nNDT1 allows us to apply Remark \\ref{rem:multipliers}, and we thus have $\\bar \\mu_{\\bar i}=\\lim\\limits_{t \\rightarrow \\infty} \\mu_{\\bar i}^t=0$, a contradiction to NDT2. Consequently, $Q_0\\left(\\bar x\\right)=Q_0\\left(x^{t}\\right)$ holds for all sufficiently small $t$.\n\nb) Next, we prove $\\mathcal{E}\\left(\\bar y\\right)=\\mathcal{E}\\left(y^{t}\\right)$.\nAgain, continuity arguments provide $\\mathcal{E}\\left(y^{t}\\right)\\subset \\mathcal{E}\\left(\\bar y\\right)$ for all sufficiently small $t$.\nSimilar to the first part of the proof, we now assume there exists $\\bar i \\in \\mathcal{E}\\left(\\bar y\\right)\\backslash \\mathcal{E}\\left(y^{t}\\right)$ along a subsequence. \nAs we have seen in Lemma \\ref{lem:a01}, T-stationarity of $(\\bar x, \\bar y)$ implies in particular $c_{\\bar i}=-\\bar \\mu_{2,\\bar i}+\\bar \\mu_{3,\\bar i}$. \nMoreover, NDT1 and Remark \\ref{rem:multipliers} provide \n$\\lim\\limits_{t \\to 0}\\mu_{3}^{t}=\\bar \\mu_{3}$.\nSince $\\bar i \\notin\\mathcal{N}\\left(y^{t}\\right)$, we distinguish the following cases:\n\n$(i)$ $\\bar i \\in\\mathcal{H}^{\\ge}\\left(x^{t},y^{t}\\right)\\backslash \\mathcal{E}\\left(y^{t}\\right)$. Karush-Kuhn-Tucker conditions for $\\left(x^t, y^t\\right)$ imply $c_{\\bar i}= \\mu^t_{3,\\bar i}+ \\eta^{\\ge,t}_{\\bar i}x_{\\bar i}$, cf. (\\ref{eq:kth-rowkkt}).\nIt follows $\n-\\bar \\mu_{2,\\bar i}+\\bar \\mu_{3,\\bar i}=\\mu^t_{3,\\bar i}+ \\eta^{\\ge,t}_{\\bar i}x_{\\bar i}$.\nBy taking the limit, we can cancel out $\\bar \\mu_{3,\\bar i}$ and $\\mu^t_{3,\\bar i}$.\nThis leads to a contradiction because the left-hand side of the equation is strictly negative due to NDT2 and the right-hand side is nonnegative since  $\\eta^{\\ge,t}_{\\bar i}$ is nonnegative and $x_{\\bar i}$ is positive.\n\n$(ii)$ $\\bar i \\in\\mathcal{H}^{\\le}\\left(x^{t},y^{t}\\right)\\backslash \\mathcal{E}\\left(y^{t}\\right)$.\nBy using (\\ref{eq:kth-rowkkt}), we get\n$c_{\\bar i}= \\mu^t_{3,\\bar i}- \\eta^{\\le,t}_{\\bar i}x_{\\bar i}$.\nThis leads to a contradiction just as in the previous case.\n\n$(iii)$ $\\bar i \\in\\mathcal{O}\\left(x^{t},y^{t}\\right)$.\nAnalogously, we obtain $c_{\\bar i}= \\mu^t_{3,\\bar i}$ from (\\ref{eq:kth-rowkkt}).\nIt follows\n$-\\bar \\mu_{2,\\bar i}+\\bar \\mu_{3,\\bar i}=\\mu^t_{3,\\bar i}$. \nTaking the limits leads to $\\bar \\mu_{2,\\bar i}=0$, a contradiction with NDT2.\n\n\\noindent\n Altogether, $\\mathcal{E}\\left(\\bar y\\right)\\backslash \\mathcal{E}\\left(y^{t}\\right)=\\emptyset$ for all sufficiently small $t$, and the assertion follows. \n\n c) Clearly, $a_{00}\\left(\\bar x, \\bar y\\right) \\cap \\mathcal{E}(y^t)=\\emptyset$ for sufficiently small $t$.\n %, hence, by definition $a_{00}\\left(\\bar x, \\bar y\\right) \\subset \\mathcal{H}\\left(x^t,y^t\\right)\\cup \\mathcal{N}\\left(y^t\\right)\\cup \\mathcal{O}\\left(x^t,y^t\\right)$.\nLet us assume  there exists an $\\bar i \\in a_{00}(\\bar x,\\bar y)\\cap\\mathcal{N}\\left(y^t\\right)$. Due to (\\ref{eq:kth-rowkkt}), we then have $c_{\\bar i}= \\mu_3^t + \\nu_{\\bar i}^t$,\nwhereas (\\ref{eq:k-throw}) provides\n$ c_{\\bar i}=\\bar \\mu_3 + \\bar \\varrho_{2,\\bar i}$.\nAccording to Remark \\ref{rem:multipliers}, we have \n$\\lim\\limits_{t \\to 0}\\mu_{3}^{t}=\\bar \\mu_3$. Consequently, it must hold\n$\n\\lim\\limits_{t \\to 0} \\nu_{\\bar i}^t=\\bar \\varrho_{2,\\bar i}$.\nThis, however, cannot be true since $\\nu_{\\bar i}^t\\geq 0$, while\n$\\bar \\varrho_{2,\\bar i}<0$ due to NDT3 from the nondegeneracy of $(\\bar x, \\bar y)$, a contradiction. \nLet us assume now that there exists an $\\bar i \\in a_{00}(\\bar x,\\bar y)\\cap \\mathcal{O}\\left(x^t,y^t\\right)$. Analogously, we get $\\bar \\varrho_{2,\\bar i}=0$, again a contradiction to NDT3. Overall, we get the assertion.\n\n d) Clearly, $a_{01}\\left(\\bar x,\\bar y\\right)\\cap \\mathcal{N}(y^t)  =\\emptyset$ for sufficiently small $t$. From c) we also know that $a_{00}\\left(\\bar x,\\bar y\\right)\\cap \\mathcal{N}(y^t)  =\\emptyset$. Altogether, the first inclusion of the assertion follows immediately. Further, it also holds\n$a_{10}\\left(\\bar x, \\bar y\\right) \\cap \\mathcal{E}(y^t)= \\emptyset$ for sufficiently small $t$.\nLet us assume there exists an  $\\bar i \\in a_{10}(\\bar x,\\bar y)\\cap\\mathcal{O}\\left(x^t,y^t\\right)$. Due to (\\ref{eq:kth-rowkkt}), we have $c_{\\bar i}=\\mu_3^t$.\nIn view of Lemma \\ref{lem:a01}c), there exists an index $\\tilde i \\in a_{01}(\\bar x,\\bar y)\\backslash \\mathcal{E}\\left(\\bar y\\right)$. Thus, T-stationarity of $(\\bar x, \\bar y)$ implies via (\\ref{eq:k-throw}) that $c_{\\tilde i}=\\bar \\mu_3$.\nBy taking the limit and Remark \\ref{rem:multipliers}, we obtain $c_{\\bar i}=c_{\\tilde i}$, but $\\bar i\\not=\\tilde i$, a contradiction to the choice of $c$.\n\\end{proof}\n\n\\begin{proof}\nThe proof will be divided into 4 major steps.\n\n{\\bf Step 1a.}\nWe rewrite the tangent space corresponding to the Karush-Kuhn-Tucker point $\\left(x^t,y^t\\right)$. For that, we use Lemma \\ref{lem:EandH}a) which provides that the summation constraint is active: \n\\[\n    \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}=\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\nDh_p( x^t),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q( x^t),0\\end{pmatrix}\\xi=0,q \\in Q_0( x^t),\\\\\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(y^t),\\\\\n\\begin{pmatrix}\ny_i^te_i,x_i^te_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{00}\\left(\\bar x, \\bar y\\right),\\\\\n\\begin{pmatrix}\ny_i^te_i,x_i^te_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x, \\bar y\\right),\\\\\n\\begin{pmatrix}\ny_i^te_i,x_i^te_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{10}\\left(\\bar x, \\bar y\\right),\\\\\n\\begin{pmatrix}\n    0,e_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{N}\\left(y^t\\right)\n\\end{array}\n\\right.\\right\\}.\n\\]\nIn total there are, due to LICQ,\n\\[\n\\begin{array}{rcl}\n\\alpha^\\mathcal{S}_t&=&\n\\left\\vert P\\right\\vert +\\left\\vert Q_0\\left(x^t\\right)\\right\\vert +1+\n\\left\\vert \\mathcal{E}\\left(y^t\\right)\\right\\vert +\n\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{00}\\left(\\bar x,\\bar y\\right)\\right\\vert \n\\\\\n&&\n+\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert \n+\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{10}\\left(\\bar x,\\bar y\\right)\\right\\vert \n+\\left\\vert \\mathcal{N}\\left(y^t\\right)\\right\\vert \n\\end{array}\n\\]\nlinearly independent vectors involved. We use Lemma \\ref{lem:QandE}a) and \\ref{lem:QandE}b) to\nsubstitute $\\left\\vert Q_0\\left(x^t\\right)\\right\\vert $ with $\\left\\vert Q_0\\left(\\bar x\\right)\\right\\vert $ and $\\left\\vert \\mathcal{E}\\left(y^t\\right)\\right\\vert $ with\n$\\left\\vert \\mathcal{E}\\left(\\bar y\\right)\\right\\vert $, respectively. The latter set has cardinality of  $n-s-1$ due to Lemma \\ref{lem:a01}c). Additionally, we use Lemma \\ref{lem:QandE}c) and \\ref{lem:QandE}d) to conclude:\n\\[\n\\begin{array}{rcl}\n\\alpha^\\mathcal{S}_{t}&=&\n\\left\\vert P\\right\\vert +\\left\\vert Q_0\\left(\\bar x\\right)\\right\\vert +1+\nn-s-1\n+\\left\\vert a_{00}\\left(\\bar x,\\bar y\\right)\\right\\vert \n\\\\\n&&\n+\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert \n+\\left\\vert a_{10}\\left(\\bar x,\\bar y\\right)\\right\\vert .\n\\end{array}\n\\]\nFinally, $\\left\\vert a_{00}\\left(\\bar x,\\bar y\\right)\\right\\vert \n+\\left\\vert a_{10}\\left(\\bar x,\\bar y\\right)\\right\\vert =s$, cf. Lemma \\ref{lem:a01}b).\nThus, we have:\n\\[\n\\begin{array}{rcl}\n\\alpha^\\mathcal{S}_{t}&=&\n\\left\\vert P\\right\\vert +\\left\\vert Q_0\\left(\\bar x\\right)\\right\\vert \n+\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert \n+n.\n\\end{array}\n\\]\n{\\bf Step 1b.} We examine the tangent space corresponding to the T-stationary point $(\\bar x,\\bar y)$. For this purpose, we consider the following vectors from its definition:\n\\[\n\\begin{pmatrix}\n0\\\\\ne_i\n\\end{pmatrix},i\\in \\mathcal{E}(\\bar y),\n\\begin{pmatrix}\n0\\\\\ne_i\n\\end{pmatrix}, i \\in a_{10}\\left(\\bar x, \\bar y\\right)\\cup a_{00}\\left(\\bar x, \\bar y\\right),\n\\begin{pmatrix}\n0\\\\\ne\n\\end{pmatrix}.\\]\nThe latter vector is involved due to Lemma \\ref{lem:a01}a). The number of these vectors is due to Lemma \\ref{lem:a01}c) equal to $(n-s-1)+\ns+1=n$. Moreover, they are linearly independent due to MPOC-LICQ.\nHence, we can write the respective tangent space as follows:\n\\[\n    \\mathcal{T}^\\mathcal{R}_{(\\bar x,\\bar y)}=\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\n\nDh_p(\\bar x), 0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(\\bar x), 0\\end{pmatrix}\\xi=0,q \\in Q_0(\\bar x),\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y) \\cup a_{01}(\\bar x,\\bar y),\n\\xi_{n+1}=\\ldots=\\xi_{2n}=0.\n\\end{array}\n\\right.\\right\\}\n\\]\nIn total there are, due to MPOC-LICQ,\n\\[\n\\alpha^\\mathcal{R}=\\left\\vert P\\right\\vert +\\left\\vert Q_0\\left(\\bar x\\right)\\right\\vert \n+\\left\\vert a_{00}(\\bar x,\\bar y)\\right\\vert +\\left\\vert a_{01}(\\bar x,\\bar y)\\right\\vert +n\n\\]\nlinearly independent vectors involved.\n\n{\\bf Step 2.}\nLet $\\mathcal{T}\\subset \\R^{2n}$ be a linear subspace. We denote the number of negative\neigenvalues of $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}}$ by $QI^\\mathcal{S}_{t,\\mathcal{T}}$. Analogously, $QI^\\mathcal{R}_{t,\\mathcal{T}}$ stands for the number of negative eigenvalues of $D^2 L^\\mathcal{R}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}}$ and $\\overline{QI}^\\mathcal{R}_{\\mathcal{T}}$ stands for the number of negative eigenvalues of $D^2 L^\\mathcal{R}\\left(\\bar x , \\bar y \\right)\\restriction_{\\mathcal{T}}$. \nWe have the following relation between the involved Hessians of the Lagrange functions by denoting $E(i)=e_ie_{n+i}^T+e_{n+i}e_i^T$, $i =1, \\ldots, n$:\n\\begin{equation}\n\\label{eq:Hessetandbar}\n\\begin{array}{rcl}\n\\displaystyle\nD^2 L^\\mathcal{S}\\left( x^t, y^t\\right) &=& \\displaystyle\nD^2 L^\\mathcal{R}\\left( x^t, y^t\\right) - \\sum\\limits_{i\\in \\mathcal{H}^\\ge\\left(x^t,y^t\\right)}  \\eta_i^{\\ge,t} E(i)\\displaystyle \n+\n\\sum\\limits_{i\\in \\mathcal{H}^\\le\\left(x^t,y^t\\right)} \\eta_i^{\\le,t} E(i).\n\\end{array}\n\\end{equation}\n\n{\\bf Step 2a.}\nIt holds for $t$ sufficiently small:\n\\[\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}=\\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}.\n\\]\n%Assume first $QI^\\mathcal{S}_{\\left(t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}>QI^\\mathcal{R}_{\\left(t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}$.\n%Then, there exists $\\bar \\xi \\in \\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}$, such that\n%\\[\n%\\bar \\xi^T D^2 L^\\mathcal{S}\\left( x^t, y^t\\right) \\bar \\xi < 0,\\quad\n%\\bar \\xi^T D^2 L^\\mathcal{R}\\left( x^t, y^t\\right)\\bar \\xi\\ge0.\n%\\]\nIndeed, by using (\\ref{eq:Hessetandbar}), we derive for any $\\xi \\in  \\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}$:\n\\begin{equation}\n    \\label{eq:s-r-h1}\n\\begin{array}{rcl}\n\\xi^T D^2 L^\\mathcal{S}\\left( x^t, y^t\\right) \\xi&=&\n%\\bar \\xi^T D^2 L^\\mathcal{R}\\left( x^t, y^t\\right)\\bar \\xi - \\bar \\xi^T \\left(\\sum\\limits_{i\\in \\mathcal{H}^\\ge\\left(x^t,y^t\\right)} \\bar \\eta_i^{t,\\ge} E(i)\\right)\\bar \\xi +\n%\\bar \\xi^T \\left(\\sum\\limits_{i\\in \\mathcal{H}^\\le\\left(x^t,y^t\\right)} \\bar \\eta_i^{t,\\le} E(i)\\right)\\bar \\xi\\\\\n%&=&\n \\xi^T D^2 L^\\mathcal{R}\\left(x^t, y^t\\right) \\xi - \\displaystyle \\sum\\limits_{i\\in \\mathcal{H}^\\ge\\left(x^t,y^t\\right)} 2\\eta_i^{\\ge,t} \\xi_i  \\xi_{n+i} \\\\\n&&\\displaystyle +\n\\sum\\limits_{i\\in \\mathcal{H}^\\le\\left(x^t,y^t\\right)} 2\\eta_i^{\\le,t}  \\xi_i \\xi_{n+i}=\n \\xi^T D^2 L^\\mathcal{R}\\left( x^t, y^t\\right)  \\xi,\n\\end{array}\n\\end{equation}\nsince $\\xi_{n+1}=\\ldots=\\xi_{2n}=0$ as seen in Step 1b.\n%Assuming  $QI^\\mathcal{S}_{\\left(t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}<QI^\\mathcal{R}_{\\left(t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}$ instead will analogously lead to a contradiction.\nHence, we get\n $\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}=QI^\\mathcal{R}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$.\nDue to NDT4, continuity arguments provide\n$\nQI^\\mathcal{R}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}=\\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$.\n\n{\\bf Step 2b.}\nWe claim that the numbers of positive and negative eigenvalues of $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$ and of $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\prime}$, respectively, coincide, where\n\\[\n  \\mathcal{T}^{\\prime} = \\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\nDh_p(x^t),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(x^t),0\\end{pmatrix}\\xi=0,q \\in Q_0(\\bar x),\\\\\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(\\bar y),\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y),\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y),\\\\\n\\begin{pmatrix}\ny^t_ie_i,x^t_ie_i\n\\end{pmatrix}\\xi=0, i \\in a_{01}(\\bar x,\\bar y)\\cup a_{10}(\\bar x,\\bar y)\n\\end{array}\n\\right.\\right\\}.\n\\]\nLet $\\left\\{\\lambda^+_1,\\ldots,\\lambda^+_{k^+}\\right\\}$ be the positive \neigenvalues of \n$D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$ with corresponding eigenvectors \n$\\left\\{\\xi^+_1,\\ldots,\\xi^+_{k^+}\\right\\}$.\nHence, for all $k=1,\\ldots, k^+$:\n\\[\n{\\xi^+_k}^T D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\xi^+_k>0.\n\\]\nWe rewrite the tangent space as follows:\n\\[\n    \\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}=\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\n\nDh_p(\\bar x),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(\\bar x),0\\end{pmatrix}\\xi=0,q \\in Q_0(\\bar x),\\\\\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(\\bar y),\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y),\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y),\\\\\n\\begin{pmatrix}\n\\bar y_ie_i,\\bar x_ie_i\n\\end{pmatrix}\\xi=0, i \\in a_{01}(\\bar x,\\bar y)\\cup a_{10}(\\bar x,\\bar y)\n\\end{array}\n\\right.\\right\\}.\n\\]\nDue to MPOC-LICQ, the application of the implicit function theorem provides the existence of\n$\\delta_2,\\delta_3>0$ such that for all $k=1,\\ldots,k^+$ and $t<\\delta_2$ there exists $\\xi_{k,t}$ with $\\left\\| \\xi_{k,t}-\\xi^+_k\\right\\|<\\delta_3$ and $\\xi_{k,t}\\in \\mathcal{T}^\\prime$.\n We can choose $t$ even smaller, such that $\\xi_{1,t},\\ldots,\\xi_{k^+,t}$ remain linearly independent and for all $k=1,\\ldots,k^+$ it holds:\n\\[\n{ \\xi_{k,t}}^T D^2 L^\\mathcal{S}\\left( x^t, y^t\\right) \\xi_{k,t}>0.\n\\]\n%whenever $\\NNorm{\\xi_k-\\xi^+_k}{}$ $<\\delta_1$, $k=1, \\ldots, k^+$.\n%\n%Altogether, for $t$ small enough there exist a set of linearly independent vectors \n%$\\left\\{\\tilde \\xi^+_1,\\ldots,\\tilde \\xi^+_{k^+}\\right\\}$ with\n%$\\tilde \\xi^+_k \\in \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$, for all $k=1,\\ldots,k^+$,\n%and\n%\\[\n%{\\tilde \\xi_k^+}{^T} D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\tilde \\xi^+_k>0.\n%\\]\nHence, $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\prime}$ has at least $k^+$ positive eigenvalues. If we repeat the above reasoning for negative eigenvalues, the matrix $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\prime}$ has at least as many negative eigenvalues as $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$. Additionally, we show that the dimensions of $\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}$ and $\\mathcal{T}^\\prime$ coincide. By Step 1b, we have $2n-\\alpha^\\mathcal{R}$ for the dimension of $\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}$. Since MPOC-LICQ remains valid in the neighborhood of $(\\bar x, \\bar y)$, we get again \n$2n-\\alpha^{\\mathcal{R}}$ for the dimension of $\\mathcal{T}^\\prime$. By continuity arguments, NDT4 and (\\ref{eq:s-r-h1}) provide that $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$ is nonsingular. Altogether, the assertion follows.\n\n{\\bf Step 2c.} \nWe claim that \n\\[\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}}\\le QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}+\\alpha^\\mathcal{R}-\\alpha^\\mathcal{S}_t.\n\\]\nFor that, we focus on the dimension of $\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$. As a consequence of Step 1a it is $2n-\\alpha^\\mathcal{S}_t$. \nDue to continuity arguments, we can choose $t$ small enough to ensure \n$x^t_{i}\\ne 0$, $i\\in a_{10}(\\bar x,\\bar y)$ and\n$y^t_{i}\\ne 0$, $i\\in a_{01}(\\bar x,\\bar y)$.\nUsing this and Lemma \\ref{lem:QandE}a), \\ref{lem:QandE}b), and \\ref{lem:QandE}d), it follows\nthat $\\mathcal{T}^\\prime \\subset \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$.\nTherefore, using Step 2b,  \n$\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}}\\le 2n-\\alpha^\\mathcal{S}_t-k^+$.\nWe observe in view of NDT4 and Step 1b that \n$\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}%\\overset{Step\\,\\,2a}{=}QI^\\mathcal{R}_{\\left(t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}%=\\overline{QI}^\\mathcal{R}_{\\left(\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}\n=2n-\\alpha^\\mathcal{R}-k^+$.\nThe assertion follows immediately.\n\n{\\bf Step 3.} Let us show that\n\\[\n\\max\\left\\{m - \\left\\vert \\left\\{i\\in a_{01}\\left(\\bar x,\\bar y\\right)\\,\\left\\vert\\,\\bar \\sigma_{1,i}=0\\right.\\right\\} \\right\\vert , 0\\right\\}\\le TI.\n\\]\nIn view of Step 2a, Step 2c and due to continuity, we have for $t$ sufficiently small:\n\n\\[\n\\begin{array}{rcl}\nm=QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}}&\\overset{Step\\,2c}{\\le}& QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}+\\alpha^\\mathcal{R}-\\alpha^\\mathcal{S}_t \\overset{Step\\,2a}{=} \n\\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}\n+\\alpha^\\mathcal{R}-\\alpha^\\mathcal{S}_t\\\\\n&\\overset{Step\\,1}{=}& \\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}\n+\\left\\vert a_{00}(\\bar x,\\bar y)\\right\\vert +\\left\\vert a_{01}(\\bar x,\\bar y)\\right\\vert -\n\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert \\\\\n&=&TI+\\left\\vert a_{01}(\\bar x,\\bar y)\\right\\vert -\n\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert .\n\\end{array}\n\\]\nWe show for $t$ sufficiently small:\n\\[\\left\\vert a_{01}(\\bar x,\\bar y)\\right\\vert -\n\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert \\le\n\\left\\vert \\left\\{i\\in a_{01}\\left(\\bar x,\\bar y\\right)\\,\\left\\vert\\,\\bar \\sigma_{1,i}=0\\right.\\right\\} \\right\\vert ,\n\\]\nand the assertion will follow immediately since $TI\\ge 0$.\nClearly,\n\\[\\left\\vert a_{01}(\\bar x,\\bar y)\\right\\vert -\n\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert =\n\\left\\vert a_{01}(\\bar x,\\bar y)\\backslash \\mathcal{H}\\left(x^t,y^t\\right) \\right\\vert .\n\\]\nSuppose $\\bar i \\in a_{01}\\left(\\bar x,\\bar y\\right)$ with $\\bar \\sigma_{1,\\bar i}\\ne 0$. In view of Remark \\ref{rem:multipliers}, the difference $\\eta^{\\ge,t}_{\\bar i}-\\eta^{\\le,t}_{\\bar i}$ cannot vanish for all $t$ sufficiently small. In particular, one of the multipliers $\\eta^{\\ge,t}_{\\bar i}$ or $\\eta^{\\le,t}_{\\bar i}$ has to be not vanishing for all $t$ sufficiently small. Hence, $\\bar i\\in \\mathcal{H}\\left(x^t,y^t\\right)$. We therefore have: \n\\[\na_{01}(\\bar x,\\bar y)\\backslash \\mathcal{H}\\left(x^t,y^t\\right) \\subset  \\left\\{i\\in a_{01}\\left(\\bar x,\\bar y\\right)\\,\\left\\vert\\,\\bar \\sigma_{1,i}= 0\\right.\\right\\}.\n\\]\n\n\n{\\bf Step 4.}\nWithout loss of generality -- considering subsequences if needed -- we can assume that for any $i\\in a_{00}(\\bar x,\\bar y)$ at least one of the sequences $\\frac{x_i^t}{y_i^t}$ or $\\frac{y_i^t}{x_i^t}$ is convergent. First, we note that the quotients are well defined due to Lemma \\ref{lem:QandE}c). Moreover, if the former sequence does not contain a convergent subsequence, we find a subsequence that tends to plus or minus infinity. Consequently, the corresponding subsequence of the latter reciprocal sequence has to converge to zero.\nWe define the following auxiliary sets:\n\\[\na_{00}^x(\\bar x,\\bar y)=\\left\\{i \\in a_{00}(\\bar x,\\bar y)\\,\\left\\vert\\, \\frac{x^t_i}{y^t_i}\\mbox{ converges for } t \\to 0 \\right. \\right\\},\na_{00}^y(\\bar x,\\bar y)=a_{00}(\\bar x,\\bar y)\\backslash a_{00}^x(\\bar x,\\bar y).\n\\]\n\nFor $\\bar i \\in a_{00}^x\\left(\\bar x,\\bar y\\right)$ we consider $\\mathcal{T}^{\\mathcal{R}}_{\\left(\\bar x,\\bar y\\right)}$ and replace two of the involved equations, namely \n$\\begin{pmatrix}\ne_{\\bar i},0\n\\end{pmatrix}\\xi=0$ and \n$\\begin{pmatrix}\n0,e_{\\bar i}\n\\end{pmatrix}\\xi=0$ by one equation\n$\\begin{pmatrix}\ne_{\\bar i},\\lim\\limits_{t \\to 0}\\frac{x_{\\bar i}^t}{y_{\\bar i}^t}e_{\\bar i}\n\\end{pmatrix}\\xi=0$.\nClearly, the vectors involved in the definition of the newly generated linear space, i.\\,e.\n\\[\n    \\mathcal{T}^{{\\bar i}}=\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\n\nDh_p(\\bar x),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(\\bar x),0\\end{pmatrix}\\xi=0,q \\in Q_0(\\bar x),\\\\\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(\\bar y),\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y)\\backslash\\{\\bar i\\},\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y)\\backslash\\{\\bar i\\},\\\\\n\\begin{pmatrix}\ne_{\\bar i},\\lim\\limits_{t \\to 0}\\frac{x_{\\bar i}^t}{y_{\\bar i}^t}e_{\\bar i}\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n\\bar y_ie_i,\\bar x_ie_i\n\\end{pmatrix}\\xi=0, i \\in a_{01}(\\bar x,\\bar y)\\cup a_{10}(\\bar x,\\bar y)\\}\n\\end{array}\n\\right.\\right\\},\n\\]\nremain linearly independent.\nThe dimension of $ \\mathcal{T}^{{\\bar i}}$ is greater than the dimension of\n$\\mathcal{T}^{\\mathcal{R}}_{\\left(\\bar x,\\bar y\\right)}$ by one. Moreover, there exists\n$\\xi^{\\bar i} \\in \\mathcal{T}^{{\\bar i}}$ with \n$\n%\\xi^{\\bar i}_{n+i} =0, i \\in a_{00}(\\bar x, \\bar y)\\backslash\\{\\bar i\\}, \\quad \n\\xi^{\\bar i}_{n+\\bar i} \\not =0$.\nIndeed, assume that no such $\\xi^{\\bar i}$ exists, then we can add the equation \n$\\begin{pmatrix}\n0,e_{\\bar i}\n\\end{pmatrix}\\xi=0$ to the defining equations of $ \\mathcal{T}^{ {\\bar i}}$ wihout changing it. The resulting space, however, is identical to\n$\\mathcal{T}^{\\mathcal{R}}_{\\left(\\bar x,\\bar y\\right)}$, a contradiction.\nWithout loss of generality, we assume \n$\\xi^{\\bar i}_{n+\\bar i}=1$.\nFurther, by straightforward application of the implicit function theorem and due to Lemma \\ref{lem:QandE}a) and \\ref{lem:QandE}b), we find a sequence of vectors $\\xi^{\\bar i}_t \\in \\mathcal{T}^{ \\bar i}_{t}$ that converges to $\\xi^{\\bar i}$ for $t \\to 0$. \nFor this, we define\n\\[\n\\mathcal{T}^{\\bar i}_t=\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\nDh_p(x^t),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(x^t),0\\end{pmatrix}\\xi=0,q \\in Q_0(x^t),\\\\\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(y^t),\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y)\\backslash\\{\\bar i\\},\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y)\\backslash\\{\\bar i\\},\\\\\n\\begin{pmatrix}\ne_{\\bar i},\\frac{x_{\\bar i}^t}{y_{\\bar i}^t} e_{\\bar i}\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\ny^t_ie_i,x^t_ie_i\n\\end{pmatrix}\\xi=0, i \\in a_{01}(\\bar x,\\bar y)\\cup a_{10}(\\bar x,\\bar y)\n\\end{array}\n\\right.\\right\\}.\n\\]\nWe again have, due to continuity arguments, that \n$\\xi^{\\bar i}_{t,n+\\bar i} \\not =0$.\n%\\left\\{\n%\\begin{array}{ll}\n%=0& \\mbox{for }i\\ne \\bar i,\\\\\n%\\ne 0& \\mbox{for } i=\\bar i.\n%\\end{array}\\right.\n%\\]\nFor $\\bar i \\in a_{00}^y\\left(\\bar x,\\bar y\\right)$ we proceed\n%we construct $\\xi^{\\bar i}$ and a sequence of vectors $\\xi^{\\bar i}_t \\in \\mathcal{T}^{ \\bar i}_{t}$ that converges to $\\xi^{\\bar i}$ for $t \\to 0$, \nanalogously by\nconsidering $\\mathcal{T}^{\\mathcal{R}}_{\\left(\\bar x,\\bar y\\right)}$ again and replace two of the  involved equations \n$\\begin{pmatrix}\ne_{\\bar i},0\n\\end{pmatrix}\\xi=0$ and \n$\\begin{pmatrix}\n0,e_{\\bar i}\n\\end{pmatrix}\\xi=0$ by the equation\n$\\begin{pmatrix}\n\\lim\\limits_{t \\to 0}\\frac{y_{\\bar i}^t}{x_{\\bar i}^t} e_{\\bar i},e_{\\bar i}\n\\end{pmatrix}\\xi=0$. \nBy the same arguments as before, we find\n$\\xi^{\\bar i} \\in \\mathcal{T}^{\\bar i}$ with\n$\\xi^{\\bar i}_{\\bar i} \\ne 0$. Again we will assume $\\xi^{\\bar i}_{\\bar i} = 1$ and find a sequence of vectors $\\xi^{\\bar i}_t \\in \\mathcal{T}^{ \\bar i}_{t}$ that converges to $\\xi^{\\bar i}$ for $t \\to 0$. Due to continuity, it holds then $\\xi^{\\bar i}_{t,\\bar i} \\ne 0$.\n\n\nIt is straightforward to verify the following observations for $t$ sufficiently small:\n\n    a) Let $\\left\\{\\xi^{\\prime,1},\\ldots,\\xi^{\\prime,\\ell}\\right\\}$ be a base of $\\mathcal{T}^{\\prime}$, cf. Step 2b, then\n    $\\left\\{\\xi^{\\prime,1},\\ldots,\\xi^{\\prime,\\ell}\\right\\}\\cup \\left\\{\\xi^{\\bar i}_{t}\\,\\left\\vert\\,\\bar i \\in a_{00}(\\bar x,\\bar y)\\right.\\right\\}$\nis a set of linear independent vectors.\nIn fact, suppose for some coefficients $b_{i}, \\in \\R$, $i \\in a_{00}\\left(\\bar x,\\bar y\\right),\\beta_i \\in \\R, i=1,\\ldots,\\ell$ it holds:\n\\[\n\\sum\\limits_{i \\in a_{00}\\left(\\bar x,\\bar y\\right)} b_{i}\\xi^{i}_{t}\n+\\sum\\limits_{i=1}^{\\ell}\\beta_i\\xi^{\\prime,i}=0.\n\\]\nFor $\\bar i \\in a_{00}^x\\left(\\bar x,\\bar y\\right)$ we consider the $(n+\\bar i)$-th row of this sum\n\\[\nb_{\\bar i}\\underbrace{\\xi^{\\bar i}_{t,n+\\bar i}}_{\\ne 0}+\\sum\\limits_{i \\in a_{00}\\left(\\bar x,\\bar y\\right)\\backslash\\left\\{\\bar i\\right\\}} b_{i}\\underbrace{\\xi^{i}_{t,n+\\bar i}}_{=0}+\\sum\\limits_{i=1}^{\\ell}\\beta_i\\underbrace{\\xi^{\\prime,i}_{n+\\bar i}}_{=0}=0.\n\\]\nIf instead $\\bar i \\in a_{00}^y\\left(\\bar x,\\bar y\\right)$ we consider the $\\bar i$-th row of the sum\n\\[\nb_{\\bar i}\\underbrace{\\xi^{\\bar i}_{t,\\bar i}}_{\\ne 0}+\\sum\\limits_{i \\in a_{00}\\left(\\bar x,\\bar y\\right)\\backslash\\left\\{\\bar i\\right\\}} b_{i}\\underbrace{\\xi^{i}_{t,\\bar i}}_{=0}+\\sum\\limits_{i=1}^{\\ell}\\beta_i\\underbrace{\\xi^{\\prime,i}_{\\bar i}}_{=0}=0.\n\\]\nAltogether, it must hold $b_{i} =0$, $i \\in a_{00}\\left(\\bar x,\\bar y\\right)$. However, this implies\n\\[\n\\sum\\limits_{i=1}^{\\ell}\\beta_i\\xi^{\\prime,i}=0.\n\\]\nHence, $\\beta_i=0$ for $i=1,\\ldots,\\ell$.\n\n    b) It holds $\\xi^{\\bar i}_{t} \\in \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$, cf. Step 1a, for any $\\bar i \\in a_{00}(\\bar x,\\bar y)$. \n    \n    c) It holds $\\left(\\eta_i^{\\le,t}-\\eta_i^{\\ge,t}\\right)\\xi^{\\bar i}_{t,i}\\xi^{\\bar i}_{t,n+i} \\leq 0, i \\in \\mathcal{H}\\left(x^t,y^t\\right)$. Since $\\xi^{\\bar i}_{t} \\in \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$, we obtain:\n    \\[\n\\begin{array}{rcl}\n\\left(\\eta_i^{\\le,t}-\\eta_i^{\\ge,t}\\right)\\xi^{\\bar i}_{t,i}\\xi^{\\bar i}_{t,n+i}\n&=&\\left(\\eta_i^{\\ge,t}-\\eta_i^{\\le,t}\\right)\\left(\\xi^{\\bar i}_{t,n+i}\\right)^2\\frac{x^t_i}{y^t_i}.\n\\end{array}\n\\]\nIf $i\\in \\mathcal{H}^{\\ge}\\left(x^t,y^t\\right)$,\nwe have $x_i^t<0,y_i^t>0$ and $\\eta_i^{\\le,t}=0$. Moreover, due to ND2, we have\n$\\eta_i^{\\ge,t}>0$. The assertion follows immediately. The other case $i\\in \\mathcal{H}^{\\le}\\left(x^t,y^t\\right)$ is completely analogous.\n\n    d) It holds $\\lim\\limits_{t\\to 0}\\left(\\eta_{\\bar i}^{\\le,t}-\\eta_{\\bar i}^{\\ge,t}\\right)\\xi^{\\bar i}_{t,\\bar i}\\xi^{\\bar i}_{t,n+\\bar i}=-\\infty$. \n    We calculate:\n\\[\n\\begin{array}{rcl}\n\\left(\\eta_{\\bar i}^{\\le,t}-\\eta_{\\bar i}^{\\ge,t}\\right)\\xi^{\\bar i}_{t,{\\bar i}}\\xi^{\\bar i}_{t,n+{\\bar i}}\n&=&\\left\\{\n\\begin{array}{ll}\n   \\left(\\eta_{\\bar i}^{\\ge,t}-\\eta_{\\bar i}^{\\le,t}\\right)\\left(\\xi^{\\bar i}_{t,n+\\bar i}\\right)^2\\frac{x^t_{\\bar i}}{y^t_{\\bar i}}& \\mbox{for } \\bar i \\in a_{00}^x(\\bar x,\\bar y),\\\\\n   \\left(\\eta_{\\bar i}^{\\ge,t}-\\eta_{\\bar i}^{\\le,t}\\right)\\left(\\xi^{\\bar i}_{t,\\bar i}\\right)^2\\frac{y^t_{\\bar i}}{x^t_{\\bar i}}& \\mbox{for } \\bar i \\in a_{00}^y(\\bar x,\\bar y).\n\\end{array}\\right.\n\\end{array}\n\\]\nLet us suppose $\\bar i \\in a_{00}^x(\\bar x,\\bar y)$. We have $y^t_{\\bar i}\\ne 0$ and, thus, $\\nu^t_{\\bar i}=0$. We use Remark \\ref{rem:multipliers} and NDT3 to conclude that the sequence\n$ \\left(\\eta_{\\bar i}^{\\ge,t}-\\eta_{\\bar i}^{\\le,t}\\right)x^t_{\\bar i}$ converges to $\\varrho_{2,\\bar i}<0$ for $t \\to 0$. Further,\n$\\frac{1}{y^t_{\\bar i}}>0$ tends to infinity for $t \\to 0$. Finally,\n$\\left(\\xi^{\\bar i}_{t,n+\\bar i}\\right)^2$ converges to $1$ for $t \\to 0$, due to the construction of\n$\\xi^{\\bar i}_t$.\nThus, the assertion follows.\nInstead, let us suppose $\\bar i \\in a_{00}^y(\\bar x,\\bar y)$. \nThis time, we have that\n$\\left(\\xi^{\\bar i}_{t,\\bar i}\\right)^2$ converges to $1$ for $t \\to 0$.\nDue to Remark \\ref{rem:multipliers} and NDT3, $\\left(\\eta_{\\bar i}^{\\ge,t}-\\eta_{\\bar i}^{\\le,t}\\right)y^t_{\\bar i}$ converges to $\\varrho_{1,\\bar i}\\ne 0$ for $t \\to 0$.\nIf $\\bar i\\in \\mathcal{H}^{\\ge}\\left(x^t,y^t\\right)$, then $\\varrho_{1,\\bar i}$ is positive from here. Also, $\\frac{1}{x^t_{\\bar i}}<0$ tends to minus infinity for $t \\to 0$. The other case $\\bar i\\in \\mathcal{H}^{\\le}\\left(x^t,y^t\\right)$ is completely analogous. \n\ne) We notice that $\\displaystyle \\xi^{\\bar i ^T}_{t} D^2 L^\\mathcal{R}\\left( x^t, y^t\\right)\\xi^{\\bar i}_{t}$ converges for $t \\to 0$ due to the construction above.\n\nFinally, for $\\bar i\\in  a_{00}(\\bar x,\\bar y)$ we estimate:\n\\[\n\\begin{array}{rcl}\n\\xi_t^{\\bar i ^T} D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\xi_t^{\\bar i}&\\overset{(\\ref{eq:Hessetandbar})}{=}&\n\\displaystyle \\xi^{\\bar i ^T}_{t} D^2 L^\\mathcal{R}\\left(x^t, y^t\\right)\\xi^{\\bar i}_{t}\\\\ && \\displaystyle\n- \\sum\\limits_{i\\in \\mathcal{H}^\\ge\\left(x^t,y^t\\right)} 2 \\eta_i^{\\ge,t} \\xi_{t,i}^{\\bar i}  \\xi_{t,n+i}^{\\bar i}+\n\\sum\\limits_{i\\in \\mathcal{H}^\\le\\left(x^t,y^t\\right)} 2 \\eta_i^{\\le,t}\\xi_{t,i}^{\\bar i}\\xi_{t,n+i}^{\\bar i} \\\\\n&= &\n\\displaystyle \\xi^{\\bar i ^T}_{t} D^2 L^\\mathcal{R}\\left(x^t, y^t\\right)\\xi^{\\bar i}_{t}\n +2\\sum\\limits_{i\\in \\mathcal{H}\\left(x^t,y^t\\right)} \\left( \\eta_i^{\\le,t}-\\eta_i^{\\ge,t} \\right)\\xi_{t,i}^{\\bar i}  \\xi_{t,n+i}^{\\bar i}\\\\ \n&\\overset{c)}{\\le}&\\displaystyle \\xi^{\\bar i ^T}_{t} D^2 L^\\mathcal{R}\\left(x^t, y^t\\right)\\xi^{\\bar i}_{t}+2\\left(\\eta_{\\bar i}^{\\le,t}-\\eta_{\\bar i}^{\\ge,t}\\right)\\xi^{\\bar i}_{t,\\bar i}\\xi^{\\bar i}_{t,n+\\bar i}.\n\\end{array}\n\\]\nThus, due to d) and e), $\\xi^{\\bar i ^T}_{t} D^2 L^\\mathcal{S}\\left(x^t, y^t\\right) \\xi^{\\bar i}_{t}$ has to be negative for $t$ small enough. \nAs we have seen in Step 2c, it holds $\\mathcal{T}^\\prime \\subset \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$. Then, due a) and b), we have therefore:\n\\[\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}}\n\\geq QI^\\mathcal{S}_{t,\\mathcal{T}^\\prime} + \\left\\vert a_{00}(\\bar x,\\bar y)\\right\\vert.\n\\]\nBy Step 2a, we have $QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}=\\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$, and by Step 2b, $QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}=QI^\\mathcal{S}_{t,\\mathcal{T}^\\prime}$. Overall, we obtain:\n\\[\nm=QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}}\n\\ge\n\\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}+\\left\\vert a_{00}(\\bar x,\\bar y)\\right\\vert=TI.\n\\]\n\\end{proof}","units":[{"start":19220,"end":19833,"start_line":300,"end_line":310,"raw_tex":"\\begin{lemma}[Auxiliary $y$-variables in $\\mathcal{R}$, \\cite{laemmel:reform}]\n\\label{lem:a01} \n%Assume that the components of $c$ are pairwise different and $\\varepsilon\\le\\frac{1}{n-s}$. \nLet $(\\bar x,\\bar y)$ be a T-stationary point of $\\mathcal{R}$, then it holds:\n\na) the summation inequality constraint is active, i.\\,e. $ \\sum\\limits_{i=1}^{n} \\bar y_i =n - s$,\n\nb) the index set $a_{01}(\\bar x,\\bar y)$ consists of exactly $n-s$ elements,\n   \nc) $n-s-1$ components of $\\bar y$ are equal to $1+\\varepsilon$, one component is equal to $1-(n-s-1)\\varepsilon$, and $s$ remaining components vanish.\n\\end{lemma}","sha256_utf8":"26efbfef9b8d941acca784362e19bd7f33da86055a7d3d29d3cf459eabd77e3e"},{"start":19834,"end":22554,"start_line":311,"end_line":358,"raw_tex":"\\begin{proof}\n a)\nLet $(\\bar x,\\bar y)$ be a T-stationary point of $\\mathcal{R}$\nand $\\displaystyle \\sum_{i=1}^{n} \\bar y_i >n - s$. Then, there exist multipliers\n$(\\bar \\lambda, \\bar \\mu, \\bar \\sigma, \\bar \\varrho)$\nsuch that (\\ref{eq:tstat-1})--(\\ref{eq:tstat-3}) are fulfilled.\nSince $\\bar \\mu_3=0$, we have that the $(n+i)$-th row of (\\ref{eq:tstat-1}) reads as\n\\[\nc_i=\\left\\{\n\\begin{array}{ll}\n     -\\bar \\mu_{2,i},&\\mbox{for }i \\in \\mathcal{E}(\\bar y),  \\\\\n     \\bar \\sigma_{2,i}, &\\mbox{for }i \\in a_{10}\\left(\\bar x,\\bar y\\right),\\\\\n     \\bar \\varrho_{2,i}, &\\mbox{for }i\\in a_{00}\\left(\\bar x,\\bar y\\right),\\\\\n     0,&\\mbox{else}.\n\\end{array}\\right.\n\\]\nDue to $c>0$, the sets $a_{01}\\left(\\bar x,\\bar y\\right)$ and $\\mathcal{E}(\\bar y)$ have to be equal and, moreover,\ndue to $\\bar \\mu_{2,i}\\ge 0$, they have to be empty. But then, clearly, $ \\sum_{i=1}^{n} \\bar  y_i =0<n - s$, a contradiction.\n\nb)\nSince $(\\bar x,\\bar y)$ is a T-stationary point, there exist\n$(\\bar \\lambda, \\bar \\mu, \\bar \\sigma, \\bar \\varrho)$\nsuch that (\\ref{eq:tstat-1})--(\\ref{eq:tstat-3}). By the proof of statement a) we can conclude that\n$\\bar \\mu_3 >0$. Hence,\nthe $(n+i)$-th row reads as\n\\begin{equation}\n    \\label{eq:k-throw}\nc_i=\\left\\{\n\\begin{array}{ll}\n     -\\bar \\mu_{2,i}+\\bar \\mu_3,&\\mbox{for }i \\in \\mathcal{E}(\\bar y),  \\\\\n     \\bar \\sigma_{2,i}+\\bar \\mu_3, &\\mbox{for }i \\in a_{10}\\left(\\bar x,\\bar y\\right),\\\\\n     \\bar \\varrho_{2,i}+\\bar \\mu_3, &\\mbox{for }i\\in a_{00}\\left(\\bar x,\\bar y\\right),\\\\\n     \\bar \\mu_3,&\\mbox{else}.\n\\end{array}\\right.\n\\end{equation}\nLet us assume that the index set $a_{01}(\\bar x,\\bar y)$ consists of fewer than $n-s$ elements. Then, we have by using the assumption on $\\varepsilon$:\n\\[ \\sum_{i=1}^{n} \\bar y_i \\le (n-s-1)\\cdot (1+\\varepsilon)\\le n-s-1+\\frac{n-s-1}{n-s} <n - s,\n\\]\na contradiction to feasibility. \nLet us assume that the index set $a_{01}(\\bar x,\\bar y)$ consists of more than $n-s$ elements instead.\nSince the components of $c$ are assumed to be pairwise different, we see from (\\ref{eq:k-throw}) that there exists at most one element in $a_{01}\\left(\\bar x,\\bar y\\right)\\backslash \\mathcal{E}(\\bar y)$ and, consequently, there are at least $n-s$ elements in $\\mathcal{E}(\\bar y)$. Therefore, we have:\n\\[\n\\sum_{i=1}^{n} \\bar y_i \\geq (n-s)\\cdot (1+\\varepsilon) > n-s,\n\\]\nwhich contradicts a). \n\nc) Due to b), $a_{01}(\\bar x,\\bar y)$ consists of exactly $n-s$ elements. We conclude as in b) that there is at most one element in $a_{01}\\left(\\bar x,\\bar y\\right)\\backslash \\mathcal{E}(\\bar y)$. In view of statement a), $\\mathcal{E}(\\bar y)$ cannot consist of $n-s$ elements and, thus, must consist of $n-s-1$ elements. Hence, the statement follows immediately.\n\\end{proof}","sha256_utf8":"b09faccc332acbedd4f890ef4a11d8d190657d2e13886174fbb4e5010b705e88"},{"start":36348,"end":36971,"start_line":754,"end_line":763,"raw_tex":"\\begin{lemma}[Auxiliary $y$-variables in $\\mathcal{S}$]\n\\label{lem:EandH}\n%Suppose Assumption \\ref{ass:c} to be fulfilled.\nLet $(x, y)$ be a Karush-Kuhn-Tucker point of $\\mathcal{S}$. Then, it holds:\n\na) the summation inequality constraint is active, i.\\,e. $\\sum\\limits_{i=1}^n  y_i=n-s$,\n\nb) the index set $\\mathcal{E}( y) \\cup \\mathcal{H}(x, y)$ consists of at least $n-s-1$ elements,\nand the index set $\\mathcal{N}( y)$ consists of at most $s$ elements. Additionally, there is at most one index, that does not belong to any of these sets, i.\\,e. $\\left\\vert \\mathcal{O}\\left( x,  y\\right)\\right\\vert \\le 1$.\n\\end{lemma}","sha256_utf8":"ced5e7b0038957b7bbf316c10a1d1cadb3c0b83e998ef25ddf1ea8cc23707776"},{"start":36972,"end":39763,"start_line":764,"end_line":808,"raw_tex":"\\begin{proof}\na) Let $(x, y)$ be a Karush-Kuhn-Tucker point of $\\mathcal{S}$ and $\\sum\\limits_{i=1}^n  y_i>n-s$. Then, there exist multipliers $(\\lambda,  \\mu,  \\eta,  \\nu)$, such that (\\ref{eq:kkt-1})--(\\ref{eq:kkt-3}) are fulfilled. Since $ \\mu_3=0$, we have that the $(n+i)$-th row of (\\ref{eq:kkt-1}) reads as\n\\[\nc_i=\\left\\{\n\\begin{array}{ll}\n     - \\mu_{2,i},&\\mbox{for }i \\in \\mathcal{E}( y)\\backslash\\mathcal{H}( x, y),  \\\\\n    - \\mu_{2,i}+ \\eta^\\ge_i  x_i, &\\mbox{for }i \\in \\mathcal{E}( y) \\cap \\mathcal{H}^\\ge( x, y),\\\\\n    - \\mu_{2,i}- \\eta^\\le_i  x_i, &\\mbox{for }i \\in \\mathcal{E}( y) \\cap  \\mathcal{H}^\\le( x, y),\\\\\n \\eta^\\ge_i  x_i, &\\mbox{for }i \\in \\mathcal{H}^\\ge( x, y)\\backslash \\mathcal{E}( y),\\\\\n- \\eta^\\le_i  x_i, &\\mbox{for }i \\in \\mathcal{H}^\\le( x, y)\\backslash \\mathcal{E}( y),\\\\\n \\nu_i,&\\mbox{for }i \\in \\mathcal{N}( y),\\\\\n     0,&\\mbox{else}.\n\\end{array}\\right.\n\\]\nDue to (\\ref{eq:kkt-2}), (\\ref{eq:kkt-3}), and $c>0$, it must hold that $i\\in \\mathcal{N}( y)$ for all $i \\in \\left\\{1,\\ldots,n\\right\\}$. This, however, contradicts $\\sum\\limits_{i=1}^n  y_i>n-s$.\n\nb) \n%Since $( x, y)$ is a Karush-Kuhn-Tucker point of RCCOP$(c,\\varepsilon,t)$ there exist multipliers $( \\lambda,  \\mu,  \\eta^\\ge,  \\eta^\\le,  \\nu)$ such that (\\ref{eq:kkt-1})--(\\ref{eq:kkt-3}). \nAs in the proof of statement a), we conclude that $ \\mu_3>0$ for a Karush-Kuhn-Tucker point $( x, y)$ of $\\mathcal{S}$. Hence, the $(n+i)$-th row now reads as\n\\begin{equation}\n\\label{eq:kth-rowkkt}    \nc_i=\\left\\{\n\\begin{array}{ll}\n     - \\mu_{2,i}+ \\mu_3,&\\mbox{for }i \\in \\mathcal{E}( y)\\backslash \\mathcal{H}( x, y),  \\\\\n    - \\mu_{2,i}+ \\mu_3+ \\eta^\\ge_i  x_i, &\\mbox{for }i \\in \\mathcal{E}( y) \\cap \\mathcal{H}^\\ge( x, y),\\\\\n    - \\mu_{2,i}+ \\mu_3- \\eta^\\le_i  x_i, &\\mbox{for }i \\in \\mathcal{E}( y) \\cap  \\mathcal{H}^\\le( x, y),\\\\\n \\mu_3+ \\eta^\\ge_i  x_i, &\\mbox{for }i \\in \\mathcal{H}^\\ge( x, y)\\backslash \\mathcal{E}( y),\\\\\n \\mu_3- \\eta^\\le_i  x_i, &\\mbox{for }i \\in \\mathcal{H}^\\le( x, y)\\backslash \\mathcal{E}( y),\\\\\n \\mu_3+ \\nu_i,&\\mbox{for }i \\in \\mathcal{N}( y),\\\\\n      \\mu_3,&\\mbox{else}.\n\\end{array}\\right.\n\\end{equation}\nIt follows from (\\ref{eq:kth-rowkkt}) and  the components of $c$ being pairwise different that there can be at most one element ${\\bar i} \\in \\mathcal{O}\\left( x,  y\\right)$. \nIf $\\mathcal{E}( y) \\cup \\mathcal{H}( x,  y)$ consists of fewer than $n-s-1$ elements, we get:\n\\[\n\\sum\\limits_{i=1}^n  y_i\\le (n-s-2)\\cdot (1+\\varepsilon)+y_{\\bar i}<(n-s-1)\\cdot(1+\\varepsilon)<n-s,\n\\]\na contradiction.\nFinally, we assume that $\\mathcal{N}( y)$ consists of more than $s$ elements.\nIn this case, there are at most $n-s-1$ nonvanishing components of $y$. Consequently, \n\\[\n\\sum\\limits_{i=1}^n  y_i\\le (n-s-1)\\cdot (1+\\varepsilon)<n-s\n\\]\nprovides a contradiction.\n\\end{proof}","sha256_utf8":"00e66523fa6573e02cc116429b00d031e5cedf1dfc772fd4187f56f5f90fc73e"},{"start":39938,"end":40424,"start_line":812,"end_line":817,"raw_tex":"\\begin{theorem}[Convergence from $\\mathcal{S}$ to $\\mathcal{R}$, cf. \\cite{laemmel:mpoc}]\n\\label{thm:regul}\n%Let $\\left\\{^t\\right\\}_{{t}\\in \\N}$  be a sequence of positive regularization parameters converging to zero. \nSuppose that a sequence of Karush-Kuhn-Tucker points $(x^{t},y^{t})$ of \n$\\mathcal{S}$ converges to $\\left(\\bar x,\\bar y\\right)$ for $t \\rightarrow 0$. If MPOC-LICQ holds at $\\left(\\bar x,\\bar y\\right)$, then it is a T-stationary point of $\\mathcal{R}$.\n\\end{theorem}","sha256_utf8":"574fce9dddb5db6dc773a028f1ba1fd8e433902c2ab26e19f8eecfeb35e075d1"},{"start":44576,"end":45280,"start_line":922,"end_line":936,"raw_tex":"\\begin{lemma}[Active index sets]\n\\label{lem:QandE}\nSuppose a sequence of Karush-Kuhn-Tucker points $\\left(x^{t},y^{t}\\right)$ of $\\mathcal{S}(t)$ converges to $\\left(\\bar x,\\bar y\\right)$ for $t \\to 0$. Moreover, let $(\\bar x,\\bar y)$ be a nondegenerate T-stationary point of $\\mathcal{R}$. Then, for all sufficiently small $t$ it holds:\n\na) $Q_0\\left(\\bar x\\right)=Q_0\\left(x^{t}\\right)$,\n\nb) $\\mathcal{E}\\left(\\bar y\\right)=\\mathcal{E}\\left(y^{t}\\right) $,\n\nc) $a_{00}\\left(\\bar x, \\bar y\\right) \\subset \\mathcal{H}\\left(x^t,y^t\\right)$,\n\nd) $\\mathcal{N}\\left(y^t\\right)\\subset\na_{10}\\left(\\bar x,\\bar y\\right)\\subset \n\\mathcal{N}\\left(y^t\\right) \\cup \\mathcal{H}\\left(x^t,y^t\\right)$.    \n\n\\end{lemma}","sha256_utf8":"ec51ba6bf3b3cd696c2d6213ab537f012647f9499edcbb2c66cc299014e3abd0"},{"start":45281,"end":50189,"start_line":937,"end_line":991,"raw_tex":"\\begin{proof}\na) We start by proving $Q_0\\left(\\bar x\\right)=Q_0\\left(x^{t}\\right)$.\nDue to continuity arguments, we have $Q_0\\left(x^{t}\\right)\\subset Q_0\\left(\\bar x\\right)$ for all sufficiently small $t$.\nLet us now assume that there exists\n$\\bar i \\in  Q_0\\left(\\bar x\\right)\\backslash Q_0\\left(x^{t}\\right)$ along a subsequence.\nHence, for the corresponding multipliers it holds $\\mu_{\\bar i}^t=0$.\nNDT1 allows us to apply Remark \\ref{rem:multipliers}, and we thus have $\\bar \\mu_{\\bar i}=\\lim\\limits_{t \\rightarrow \\infty} \\mu_{\\bar i}^t=0$, a contradiction to NDT2. Consequently, $Q_0\\left(\\bar x\\right)=Q_0\\left(x^{t}\\right)$ holds for all sufficiently small $t$.\n\nb) Next, we prove $\\mathcal{E}\\left(\\bar y\\right)=\\mathcal{E}\\left(y^{t}\\right)$.\nAgain, continuity arguments provide $\\mathcal{E}\\left(y^{t}\\right)\\subset \\mathcal{E}\\left(\\bar y\\right)$ for all sufficiently small $t$.\nSimilar to the first part of the proof, we now assume there exists $\\bar i \\in \\mathcal{E}\\left(\\bar y\\right)\\backslash \\mathcal{E}\\left(y^{t}\\right)$ along a subsequence. \nAs we have seen in Lemma \\ref{lem:a01}, T-stationarity of $(\\bar x, \\bar y)$ implies in particular $c_{\\bar i}=-\\bar \\mu_{2,\\bar i}+\\bar \\mu_{3,\\bar i}$. \nMoreover, NDT1 and Remark \\ref{rem:multipliers} provide \n$\\lim\\limits_{t \\to 0}\\mu_{3}^{t}=\\bar \\mu_{3}$.\nSince $\\bar i \\notin\\mathcal{N}\\left(y^{t}\\right)$, we distinguish the following cases:\n\n$(i)$ $\\bar i \\in\\mathcal{H}^{\\ge}\\left(x^{t},y^{t}\\right)\\backslash \\mathcal{E}\\left(y^{t}\\right)$. Karush-Kuhn-Tucker conditions for $\\left(x^t, y^t\\right)$ imply $c_{\\bar i}= \\mu^t_{3,\\bar i}+ \\eta^{\\ge,t}_{\\bar i}x_{\\bar i}$, cf. (\\ref{eq:kth-rowkkt}).\nIt follows $\n-\\bar \\mu_{2,\\bar i}+\\bar \\mu_{3,\\bar i}=\\mu^t_{3,\\bar i}+ \\eta^{\\ge,t}_{\\bar i}x_{\\bar i}$.\nBy taking the limit, we can cancel out $\\bar \\mu_{3,\\bar i}$ and $\\mu^t_{3,\\bar i}$.\nThis leads to a contradiction because the left-hand side of the equation is strictly negative due to NDT2 and the right-hand side is nonnegative since  $\\eta^{\\ge,t}_{\\bar i}$ is nonnegative and $x_{\\bar i}$ is positive.\n\n$(ii)$ $\\bar i \\in\\mathcal{H}^{\\le}\\left(x^{t},y^{t}\\right)\\backslash \\mathcal{E}\\left(y^{t}\\right)$.\nBy using (\\ref{eq:kth-rowkkt}), we get\n$c_{\\bar i}= \\mu^t_{3,\\bar i}- \\eta^{\\le,t}_{\\bar i}x_{\\bar i}$.\nThis leads to a contradiction just as in the previous case.\n\n$(iii)$ $\\bar i \\in\\mathcal{O}\\left(x^{t},y^{t}\\right)$.\nAnalogously, we obtain $c_{\\bar i}= \\mu^t_{3,\\bar i}$ from (\\ref{eq:kth-rowkkt}).\nIt follows\n$-\\bar \\mu_{2,\\bar i}+\\bar \\mu_{3,\\bar i}=\\mu^t_{3,\\bar i}$. \nTaking the limits leads to $\\bar \\mu_{2,\\bar i}=0$, a contradiction with NDT2.\n\n\\noindent\n Altogether, $\\mathcal{E}\\left(\\bar y\\right)\\backslash \\mathcal{E}\\left(y^{t}\\right)=\\emptyset$ for all sufficiently small $t$, and the assertion follows. \n\n c) Clearly, $a_{00}\\left(\\bar x, \\bar y\\right) \\cap \\mathcal{E}(y^t)=\\emptyset$ for sufficiently small $t$.\n %, hence, by definition $a_{00}\\left(\\bar x, \\bar y\\right) \\subset \\mathcal{H}\\left(x^t,y^t\\right)\\cup \\mathcal{N}\\left(y^t\\right)\\cup \\mathcal{O}\\left(x^t,y^t\\right)$.\nLet us assume  there exists an $\\bar i \\in a_{00}(\\bar x,\\bar y)\\cap\\mathcal{N}\\left(y^t\\right)$. Due to (\\ref{eq:kth-rowkkt}), we then have $c_{\\bar i}= \\mu_3^t + \\nu_{\\bar i}^t$,\nwhereas (\\ref{eq:k-throw}) provides\n$ c_{\\bar i}=\\bar \\mu_3 + \\bar \\varrho_{2,\\bar i}$.\nAccording to Remark \\ref{rem:multipliers}, we have \n$\\lim\\limits_{t \\to 0}\\mu_{3}^{t}=\\bar \\mu_3$. Consequently, it must hold\n$\n\\lim\\limits_{t \\to 0} \\nu_{\\bar i}^t=\\bar \\varrho_{2,\\bar i}$.\nThis, however, cannot be true since $\\nu_{\\bar i}^t\\geq 0$, while\n$\\bar \\varrho_{2,\\bar i}<0$ due to NDT3 from the nondegeneracy of $(\\bar x, \\bar y)$, a contradiction. \nLet us assume now that there exists an $\\bar i \\in a_{00}(\\bar x,\\bar y)\\cap \\mathcal{O}\\left(x^t,y^t\\right)$. Analogously, we get $\\bar \\varrho_{2,\\bar i}=0$, again a contradiction to NDT3. Overall, we get the assertion.\n\n d) Clearly, $a_{01}\\left(\\bar x,\\bar y\\right)\\cap \\mathcal{N}(y^t)  =\\emptyset$ for sufficiently small $t$. From c) we also know that $a_{00}\\left(\\bar x,\\bar y\\right)\\cap \\mathcal{N}(y^t)  =\\emptyset$. Altogether, the first inclusion of the assertion follows immediately. Further, it also holds\n$a_{10}\\left(\\bar x, \\bar y\\right) \\cap \\mathcal{E}(y^t)= \\emptyset$ for sufficiently small $t$.\nLet us assume there exists an  $\\bar i \\in a_{10}(\\bar x,\\bar y)\\cap\\mathcal{O}\\left(x^t,y^t\\right)$. Due to (\\ref{eq:kth-rowkkt}), we have $c_{\\bar i}=\\mu_3^t$.\nIn view of Lemma \\ref{lem:a01}c), there exists an index $\\tilde i \\in a_{01}(\\bar x,\\bar y)\\backslash \\mathcal{E}\\left(\\bar y\\right)$. Thus, T-stationarity of $(\\bar x, \\bar y)$ implies via (\\ref{eq:k-throw}) that $c_{\\tilde i}=\\bar \\mu_3$.\nBy taking the limit and Remark \\ref{rem:multipliers}, we obtain $c_{\\bar i}=c_{\\tilde i}$, but $\\bar i\\not=\\tilde i$, a contradiction to the choice of $c$.\n\\end{proof}","sha256_utf8":"9768288f3496f9451de22fbbe387d433dd88b31b92dc2a7a5957ba9d3a261942"},{"start":51039,"end":77534,"start_line":1005,"end_line":1501,"raw_tex":"\\begin{proof}\nThe proof will be divided into 4 major steps.\n\n{\\bf Step 1a.}\nWe rewrite the tangent space corresponding to the Karush-Kuhn-Tucker point $\\left(x^t,y^t\\right)$. For that, we use Lemma \\ref{lem:EandH}a) which provides that the summation constraint is active: \n\\[\n    \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}=\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\nDh_p( x^t),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q( x^t),0\\end{pmatrix}\\xi=0,q \\in Q_0( x^t),\\\\\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(y^t),\\\\\n\\begin{pmatrix}\ny_i^te_i,x_i^te_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{00}\\left(\\bar x, \\bar y\\right),\\\\\n\\begin{pmatrix}\ny_i^te_i,x_i^te_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x, \\bar y\\right),\\\\\n\\begin{pmatrix}\ny_i^te_i,x_i^te_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{10}\\left(\\bar x, \\bar y\\right),\\\\\n\\begin{pmatrix}\n    0,e_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{N}\\left(y^t\\right)\n\\end{array}\n\\right.\\right\\}.\n\\]\nIn total there are, due to LICQ,\n\\[\n\\begin{array}{rcl}\n\\alpha^\\mathcal{S}_t&=&\n\\left\\vert P\\right\\vert +\\left\\vert Q_0\\left(x^t\\right)\\right\\vert +1+\n\\left\\vert \\mathcal{E}\\left(y^t\\right)\\right\\vert +\n\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{00}\\left(\\bar x,\\bar y\\right)\\right\\vert \n\\\\\n&&\n+\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert \n+\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{10}\\left(\\bar x,\\bar y\\right)\\right\\vert \n+\\left\\vert \\mathcal{N}\\left(y^t\\right)\\right\\vert \n\\end{array}\n\\]\nlinearly independent vectors involved. We use Lemma \\ref{lem:QandE}a) and \\ref{lem:QandE}b) to\nsubstitute $\\left\\vert Q_0\\left(x^t\\right)\\right\\vert $ with $\\left\\vert Q_0\\left(\\bar x\\right)\\right\\vert $ and $\\left\\vert \\mathcal{E}\\left(y^t\\right)\\right\\vert $ with\n$\\left\\vert \\mathcal{E}\\left(\\bar y\\right)\\right\\vert $, respectively. The latter set has cardinality of  $n-s-1$ due to Lemma \\ref{lem:a01}c). Additionally, we use Lemma \\ref{lem:QandE}c) and \\ref{lem:QandE}d) to conclude:\n\\[\n\\begin{array}{rcl}\n\\alpha^\\mathcal{S}_{t}&=&\n\\left\\vert P\\right\\vert +\\left\\vert Q_0\\left(\\bar x\\right)\\right\\vert +1+\nn-s-1\n+\\left\\vert a_{00}\\left(\\bar x,\\bar y\\right)\\right\\vert \n\\\\\n&&\n+\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert \n+\\left\\vert a_{10}\\left(\\bar x,\\bar y\\right)\\right\\vert .\n\\end{array}\n\\]\nFinally, $\\left\\vert a_{00}\\left(\\bar x,\\bar y\\right)\\right\\vert \n+\\left\\vert a_{10}\\left(\\bar x,\\bar y\\right)\\right\\vert =s$, cf. Lemma \\ref{lem:a01}b).\nThus, we have:\n\\[\n\\begin{array}{rcl}\n\\alpha^\\mathcal{S}_{t}&=&\n\\left\\vert P\\right\\vert +\\left\\vert Q_0\\left(\\bar x\\right)\\right\\vert \n+\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert \n+n.\n\\end{array}\n\\]\n{\\bf Step 1b.} We examine the tangent space corresponding to the T-stationary point $(\\bar x,\\bar y)$. For this purpose, we consider the following vectors from its definition:\n\\[\n\\begin{pmatrix}\n0\\\\\ne_i\n\\end{pmatrix},i\\in \\mathcal{E}(\\bar y),\n\\begin{pmatrix}\n0\\\\\ne_i\n\\end{pmatrix}, i \\in a_{10}\\left(\\bar x, \\bar y\\right)\\cup a_{00}\\left(\\bar x, \\bar y\\right),\n\\begin{pmatrix}\n0\\\\\ne\n\\end{pmatrix}.\\]\nThe latter vector is involved due to Lemma \\ref{lem:a01}a). The number of these vectors is due to Lemma \\ref{lem:a01}c) equal to $(n-s-1)+\ns+1=n$. Moreover, they are linearly independent due to MPOC-LICQ.\nHence, we can write the respective tangent space as follows:\n\\[\n    \\mathcal{T}^\\mathcal{R}_{(\\bar x,\\bar y)}=\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\n\nDh_p(\\bar x), 0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(\\bar x), 0\\end{pmatrix}\\xi=0,q \\in Q_0(\\bar x),\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y) \\cup a_{01}(\\bar x,\\bar y),\n\\xi_{n+1}=\\ldots=\\xi_{2n}=0.\n\\end{array}\n\\right.\\right\\}\n\\]\nIn total there are, due to MPOC-LICQ,\n\\[\n\\alpha^\\mathcal{R}=\\left\\vert P\\right\\vert +\\left\\vert Q_0\\left(\\bar x\\right)\\right\\vert \n+\\left\\vert a_{00}(\\bar x,\\bar y)\\right\\vert +\\left\\vert a_{01}(\\bar x,\\bar y)\\right\\vert +n\n\\]\nlinearly independent vectors involved.\n\n{\\bf Step 2.}\nLet $\\mathcal{T}\\subset \\R^{2n}$ be a linear subspace. We denote the number of negative\neigenvalues of $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}}$ by $QI^\\mathcal{S}_{t,\\mathcal{T}}$. Analogously, $QI^\\mathcal{R}_{t,\\mathcal{T}}$ stands for the number of negative eigenvalues of $D^2 L^\\mathcal{R}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}}$ and $\\overline{QI}^\\mathcal{R}_{\\mathcal{T}}$ stands for the number of negative eigenvalues of $D^2 L^\\mathcal{R}\\left(\\bar x , \\bar y \\right)\\restriction_{\\mathcal{T}}$. \nWe have the following relation between the involved Hessians of the Lagrange functions by denoting $E(i)=e_ie_{n+i}^T+e_{n+i}e_i^T$, $i =1, \\ldots, n$:\n\\begin{equation}\n\\label{eq:Hessetandbar}\n\\begin{array}{rcl}\n\\displaystyle\nD^2 L^\\mathcal{S}\\left( x^t, y^t\\right) &=& \\displaystyle\nD^2 L^\\mathcal{R}\\left( x^t, y^t\\right) - \\sum\\limits_{i\\in \\mathcal{H}^\\ge\\left(x^t,y^t\\right)}  \\eta_i^{\\ge,t} E(i)\\displaystyle \n+\n\\sum\\limits_{i\\in \\mathcal{H}^\\le\\left(x^t,y^t\\right)} \\eta_i^{\\le,t} E(i).\n\\end{array}\n\\end{equation}\n\n{\\bf Step 2a.}\nIt holds for $t$ sufficiently small:\n\\[\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}=\\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}.\n\\]\n%Assume first $QI^\\mathcal{S}_{\\left(t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}>QI^\\mathcal{R}_{\\left(t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}$.\n%Then, there exists $\\bar \\xi \\in \\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}$, such that\n%\\[\n%\\bar \\xi^T D^2 L^\\mathcal{S}\\left( x^t, y^t\\right) \\bar \\xi < 0,\\quad\n%\\bar \\xi^T D^2 L^\\mathcal{R}\\left( x^t, y^t\\right)\\bar \\xi\\ge0.\n%\\]\nIndeed, by using (\\ref{eq:Hessetandbar}), we derive for any $\\xi \\in  \\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}$:\n\\begin{equation}\n    \\label{eq:s-r-h1}\n\\begin{array}{rcl}\n\\xi^T D^2 L^\\mathcal{S}\\left( x^t, y^t\\right) \\xi&=&\n%\\bar \\xi^T D^2 L^\\mathcal{R}\\left( x^t, y^t\\right)\\bar \\xi - \\bar \\xi^T \\left(\\sum\\limits_{i\\in \\mathcal{H}^\\ge\\left(x^t,y^t\\right)} \\bar \\eta_i^{t,\\ge} E(i)\\right)\\bar \\xi +\n%\\bar \\xi^T \\left(\\sum\\limits_{i\\in \\mathcal{H}^\\le\\left(x^t,y^t\\right)} \\bar \\eta_i^{t,\\le} E(i)\\right)\\bar \\xi\\\\\n%&=&\n \\xi^T D^2 L^\\mathcal{R}\\left(x^t, y^t\\right) \\xi - \\displaystyle \\sum\\limits_{i\\in \\mathcal{H}^\\ge\\left(x^t,y^t\\right)} 2\\eta_i^{\\ge,t} \\xi_i  \\xi_{n+i} \\\\\n&&\\displaystyle +\n\\sum\\limits_{i\\in \\mathcal{H}^\\le\\left(x^t,y^t\\right)} 2\\eta_i^{\\le,t}  \\xi_i \\xi_{n+i}=\n \\xi^T D^2 L^\\mathcal{R}\\left( x^t, y^t\\right)  \\xi,\n\\end{array}\n\\end{equation}\nsince $\\xi_{n+1}=\\ldots=\\xi_{2n}=0$ as seen in Step 1b.\n%Assuming  $QI^\\mathcal{S}_{\\left(t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}<QI^\\mathcal{R}_{\\left(t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}$ instead will analogously lead to a contradiction.\nHence, we get\n $\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}=QI^\\mathcal{R}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$.\nDue to NDT4, continuity arguments provide\n$\nQI^\\mathcal{R}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}=\\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$.\n\n{\\bf Step 2b.}\nWe claim that the numbers of positive and negative eigenvalues of $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$ and of $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\prime}$, respectively, coincide, where\n\\[\n  \\mathcal{T}^{\\prime} = \\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\nDh_p(x^t),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(x^t),0\\end{pmatrix}\\xi=0,q \\in Q_0(\\bar x),\\\\\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(\\bar y),\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y),\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y),\\\\\n\\begin{pmatrix}\ny^t_ie_i,x^t_ie_i\n\\end{pmatrix}\\xi=0, i \\in a_{01}(\\bar x,\\bar y)\\cup a_{10}(\\bar x,\\bar y)\n\\end{array}\n\\right.\\right\\}.\n\\]\nLet $\\left\\{\\lambda^+_1,\\ldots,\\lambda^+_{k^+}\\right\\}$ be the positive \neigenvalues of \n$D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$ with corresponding eigenvectors \n$\\left\\{\\xi^+_1,\\ldots,\\xi^+_{k^+}\\right\\}$.\nHence, for all $k=1,\\ldots, k^+$:\n\\[\n{\\xi^+_k}^T D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\xi^+_k>0.\n\\]\nWe rewrite the tangent space as follows:\n\\[\n    \\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}=\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\n\nDh_p(\\bar x),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(\\bar x),0\\end{pmatrix}\\xi=0,q \\in Q_0(\\bar x),\\\\\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(\\bar y),\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y),\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y),\\\\\n\\begin{pmatrix}\n\\bar y_ie_i,\\bar x_ie_i\n\\end{pmatrix}\\xi=0, i \\in a_{01}(\\bar x,\\bar y)\\cup a_{10}(\\bar x,\\bar y)\n\\end{array}\n\\right.\\right\\}.\n\\]\nDue to MPOC-LICQ, the application of the implicit function theorem provides the existence of\n$\\delta_2,\\delta_3>0$ such that for all $k=1,\\ldots,k^+$ and $t<\\delta_2$ there exists $\\xi_{k,t}$ with $\\left\\| \\xi_{k,t}-\\xi^+_k\\right\\|<\\delta_3$ and $\\xi_{k,t}\\in \\mathcal{T}^\\prime$.\n We can choose $t$ even smaller, such that $\\xi_{1,t},\\ldots,\\xi_{k^+,t}$ remain linearly independent and for all $k=1,\\ldots,k^+$ it holds:\n\\[\n{ \\xi_{k,t}}^T D^2 L^\\mathcal{S}\\left( x^t, y^t\\right) \\xi_{k,t}>0.\n\\]\n%whenever $\\NNorm{\\xi_k-\\xi^+_k}{}$ $<\\delta_1$, $k=1, \\ldots, k^+$.\n%\n%Altogether, for $t$ small enough there exist a set of linearly independent vectors \n%$\\left\\{\\tilde \\xi^+_1,\\ldots,\\tilde \\xi^+_{k^+}\\right\\}$ with\n%$\\tilde \\xi^+_k \\in \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$, for all $k=1,\\ldots,k^+$,\n%and\n%\\[\n%{\\tilde \\xi_k^+}{^T} D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\tilde \\xi^+_k>0.\n%\\]\nHence, $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\prime}$ has at least $k^+$ positive eigenvalues. If we repeat the above reasoning for negative eigenvalues, the matrix $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\prime}$ has at least as many negative eigenvalues as $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$. Additionally, we show that the dimensions of $\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}$ and $\\mathcal{T}^\\prime$ coincide. By Step 1b, we have $2n-\\alpha^\\mathcal{R}$ for the dimension of $\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}$. Since MPOC-LICQ remains valid in the neighborhood of $(\\bar x, \\bar y)$, we get again \n$2n-\\alpha^{\\mathcal{R}}$ for the dimension of $\\mathcal{T}^\\prime$. By continuity arguments, NDT4 and (\\ref{eq:s-r-h1}) provide that $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$ is nonsingular. Altogether, the assertion follows.\n\n{\\bf Step 2c.} \nWe claim that \n\\[\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}}\\le QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}+\\alpha^\\mathcal{R}-\\alpha^\\mathcal{S}_t.\n\\]\nFor that, we focus on the dimension of $\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$. As a consequence of Step 1a it is $2n-\\alpha^\\mathcal{S}_t$. \nDue to continuity arguments, we can choose $t$ small enough to ensure \n$x^t_{i}\\ne 0$, $i\\in a_{10}(\\bar x,\\bar y)$ and\n$y^t_{i}\\ne 0$, $i\\in a_{01}(\\bar x,\\bar y)$.\nUsing this and Lemma \\ref{lem:QandE}a), \\ref{lem:QandE}b), and \\ref{lem:QandE}d), it follows\nthat $\\mathcal{T}^\\prime \\subset \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$.\nTherefore, using Step 2b,  \n$\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}}\\le 2n-\\alpha^\\mathcal{S}_t-k^+$.\nWe observe in view of NDT4 and Step 1b that \n$\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}%\\overset{Step\\,\\,2a}{=}QI^\\mathcal{R}_{\\left(t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}%=\\overline{QI}^\\mathcal{R}_{\\left(\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}\n=2n-\\alpha^\\mathcal{R}-k^+$.\nThe assertion follows immediately.\n\n{\\bf Step 3.} Let us show that\n\\[\n\\max\\left\\{m - \\left\\vert \\left\\{i\\in a_{01}\\left(\\bar x,\\bar y\\right)\\,\\left\\vert\\,\\bar \\sigma_{1,i}=0\\right.\\right\\} \\right\\vert , 0\\right\\}\\le TI.\n\\]\nIn view of Step 2a, Step 2c and due to continuity, we have for $t$ sufficiently small:\n\n\\[\n\\begin{array}{rcl}\nm=QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}}&\\overset{Step\\,2c}{\\le}& QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}+\\alpha^\\mathcal{R}-\\alpha^\\mathcal{S}_t \\overset{Step\\,2a}{=} \n\\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}\n+\\alpha^\\mathcal{R}-\\alpha^\\mathcal{S}_t\\\\\n&\\overset{Step\\,1}{=}& \\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}\n+\\left\\vert a_{00}(\\bar x,\\bar y)\\right\\vert +\\left\\vert a_{01}(\\bar x,\\bar y)\\right\\vert -\n\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert \\\\\n&=&TI+\\left\\vert a_{01}(\\bar x,\\bar y)\\right\\vert -\n\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert .\n\\end{array}\n\\]\nWe show for $t$ sufficiently small:\n\\[\\left\\vert a_{01}(\\bar x,\\bar y)\\right\\vert -\n\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert \\le\n\\left\\vert \\left\\{i\\in a_{01}\\left(\\bar x,\\bar y\\right)\\,\\left\\vert\\,\\bar \\sigma_{1,i}=0\\right.\\right\\} \\right\\vert ,\n\\]\nand the assertion will follow immediately since $TI\\ge 0$.\nClearly,\n\\[\\left\\vert a_{01}(\\bar x,\\bar y)\\right\\vert -\n\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert =\n\\left\\vert a_{01}(\\bar x,\\bar y)\\backslash \\mathcal{H}\\left(x^t,y^t\\right) \\right\\vert .\n\\]\nSuppose $\\bar i \\in a_{01}\\left(\\bar x,\\bar y\\right)$ with $\\bar \\sigma_{1,\\bar i}\\ne 0$. In view of Remark \\ref{rem:multipliers}, the difference $\\eta^{\\ge,t}_{\\bar i}-\\eta^{\\le,t}_{\\bar i}$ cannot vanish for all $t$ sufficiently small. In particular, one of the multipliers $\\eta^{\\ge,t}_{\\bar i}$ or $\\eta^{\\le,t}_{\\bar i}$ has to be not vanishing for all $t$ sufficiently small. Hence, $\\bar i\\in \\mathcal{H}\\left(x^t,y^t\\right)$. We therefore have: \n\\[\na_{01}(\\bar x,\\bar y)\\backslash \\mathcal{H}\\left(x^t,y^t\\right) \\subset  \\left\\{i\\in a_{01}\\left(\\bar x,\\bar y\\right)\\,\\left\\vert\\,\\bar \\sigma_{1,i}= 0\\right.\\right\\}.\n\\]\n\n\n{\\bf Step 4.}\nWithout loss of generality -- considering subsequences if needed -- we can assume that for any $i\\in a_{00}(\\bar x,\\bar y)$ at least one of the sequences $\\frac{x_i^t}{y_i^t}$ or $\\frac{y_i^t}{x_i^t}$ is convergent. First, we note that the quotients are well defined due to Lemma \\ref{lem:QandE}c). Moreover, if the former sequence does not contain a convergent subsequence, we find a subsequence that tends to plus or minus infinity. Consequently, the corresponding subsequence of the latter reciprocal sequence has to converge to zero.\nWe define the following auxiliary sets:\n\\[\na_{00}^x(\\bar x,\\bar y)=\\left\\{i \\in a_{00}(\\bar x,\\bar y)\\,\\left\\vert\\, \\frac{x^t_i}{y^t_i}\\mbox{ converges for } t \\to 0 \\right. \\right\\},\na_{00}^y(\\bar x,\\bar y)=a_{00}(\\bar x,\\bar y)\\backslash a_{00}^x(\\bar x,\\bar y).\n\\]\n\nFor $\\bar i \\in a_{00}^x\\left(\\bar x,\\bar y\\right)$ we consider $\\mathcal{T}^{\\mathcal{R}}_{\\left(\\bar x,\\bar y\\right)}$ and replace two of the involved equations, namely \n$\\begin{pmatrix}\ne_{\\bar i},0\n\\end{pmatrix}\\xi=0$ and \n$\\begin{pmatrix}\n0,e_{\\bar i}\n\\end{pmatrix}\\xi=0$ by one equation\n$\\begin{pmatrix}\ne_{\\bar i},\\lim\\limits_{t \\to 0}\\frac{x_{\\bar i}^t}{y_{\\bar i}^t}e_{\\bar i}\n\\end{pmatrix}\\xi=0$.\nClearly, the vectors involved in the definition of the newly generated linear space, i.\\,e.\n\\[\n    \\mathcal{T}^{{\\bar i}}=\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\n\nDh_p(\\bar x),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(\\bar x),0\\end{pmatrix}\\xi=0,q \\in Q_0(\\bar x),\\\\\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(\\bar y),\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y)\\backslash\\{\\bar i\\},\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y)\\backslash\\{\\bar i\\},\\\\\n\\begin{pmatrix}\ne_{\\bar i},\\lim\\limits_{t \\to 0}\\frac{x_{\\bar i}^t}{y_{\\bar i}^t}e_{\\bar i}\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n\\bar y_ie_i,\\bar x_ie_i\n\\end{pmatrix}\\xi=0, i \\in a_{01}(\\bar x,\\bar y)\\cup a_{10}(\\bar x,\\bar y)\\}\n\\end{array}\n\\right.\\right\\},\n\\]\nremain linearly independent.\nThe dimension of $ \\mathcal{T}^{{\\bar i}}$ is greater than the dimension of\n$\\mathcal{T}^{\\mathcal{R}}_{\\left(\\bar x,\\bar y\\right)}$ by one. Moreover, there exists\n$\\xi^{\\bar i} \\in \\mathcal{T}^{{\\bar i}}$ with \n$\n%\\xi^{\\bar i}_{n+i} =0, i \\in a_{00}(\\bar x, \\bar y)\\backslash\\{\\bar i\\}, \\quad \n\\xi^{\\bar i}_{n+\\bar i} \\not =0$.\nIndeed, assume that no such $\\xi^{\\bar i}$ exists, then we can add the equation \n$\\begin{pmatrix}\n0,e_{\\bar i}\n\\end{pmatrix}\\xi=0$ to the defining equations of $ \\mathcal{T}^{ {\\bar i}}$ wihout changing it. The resulting space, however, is identical to\n$\\mathcal{T}^{\\mathcal{R}}_{\\left(\\bar x,\\bar y\\right)}$, a contradiction.\nWithout loss of generality, we assume \n$\\xi^{\\bar i}_{n+\\bar i}=1$.\nFurther, by straightforward application of the implicit function theorem and due to Lemma \\ref{lem:QandE}a) and \\ref{lem:QandE}b), we find a sequence of vectors $\\xi^{\\bar i}_t \\in \\mathcal{T}^{ \\bar i}_{t}$ that converges to $\\xi^{\\bar i}$ for $t \\to 0$. \nFor this, we define\n\\[\n\\mathcal{T}^{\\bar i}_t=\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\nDh_p(x^t),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(x^t),0\\end{pmatrix}\\xi=0,q \\in Q_0(x^t),\\\\\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(y^t),\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y)\\backslash\\{\\bar i\\},\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y)\\backslash\\{\\bar i\\},\\\\\n\\begin{pmatrix}\ne_{\\bar i},\\frac{x_{\\bar i}^t}{y_{\\bar i}^t} e_{\\bar i}\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\ny^t_ie_i,x^t_ie_i\n\\end{pmatrix}\\xi=0, i \\in a_{01}(\\bar x,\\bar y)\\cup a_{10}(\\bar x,\\bar y)\n\\end{array}\n\\right.\\right\\}.\n\\]\nWe again have, due to continuity arguments, that \n$\\xi^{\\bar i}_{t,n+\\bar i} \\not =0$.\n%\\left\\{\n%\\begin{array}{ll}\n%=0& \\mbox{for }i\\ne \\bar i,\\\\\n%\\ne 0& \\mbox{for } i=\\bar i.\n%\\end{array}\\right.\n%\\]\nFor $\\bar i \\in a_{00}^y\\left(\\bar x,\\bar y\\right)$ we proceed\n%we construct $\\xi^{\\bar i}$ and a sequence of vectors $\\xi^{\\bar i}_t \\in \\mathcal{T}^{ \\bar i}_{t}$ that converges to $\\xi^{\\bar i}$ for $t \\to 0$, \nanalogously by\nconsidering $\\mathcal{T}^{\\mathcal{R}}_{\\left(\\bar x,\\bar y\\right)}$ again and replace two of the  involved equations \n$\\begin{pmatrix}\ne_{\\bar i},0\n\\end{pmatrix}\\xi=0$ and \n$\\begin{pmatrix}\n0,e_{\\bar i}\n\\end{pmatrix}\\xi=0$ by the equation\n$\\begin{pmatrix}\n\\lim\\limits_{t \\to 0}\\frac{y_{\\bar i}^t}{x_{\\bar i}^t} e_{\\bar i},e_{\\bar i}\n\\end{pmatrix}\\xi=0$. \nBy the same arguments as before, we find\n$\\xi^{\\bar i} \\in \\mathcal{T}^{\\bar i}$ with\n$\\xi^{\\bar i}_{\\bar i} \\ne 0$. Again we will assume $\\xi^{\\bar i}_{\\bar i} = 1$ and find a sequence of vectors $\\xi^{\\bar i}_t \\in \\mathcal{T}^{ \\bar i}_{t}$ that converges to $\\xi^{\\bar i}$ for $t \\to 0$. Due to continuity, it holds then $\\xi^{\\bar i}_{t,\\bar i} \\ne 0$.\n\n\nIt is straightforward to verify the following observations for $t$ sufficiently small:\n\n    a) Let $\\left\\{\\xi^{\\prime,1},\\ldots,\\xi^{\\prime,\\ell}\\right\\}$ be a base of $\\mathcal{T}^{\\prime}$, cf. Step 2b, then\n    $\\left\\{\\xi^{\\prime,1},\\ldots,\\xi^{\\prime,\\ell}\\right\\}\\cup \\left\\{\\xi^{\\bar i}_{t}\\,\\left\\vert\\,\\bar i \\in a_{00}(\\bar x,\\bar y)\\right.\\right\\}$\nis a set of linear independent vectors.\nIn fact, suppose for some coefficients $b_{i}, \\in \\R$, $i \\in a_{00}\\left(\\bar x,\\bar y\\right),\\beta_i \\in \\R, i=1,\\ldots,\\ell$ it holds:\n\\[\n\\sum\\limits_{i \\in a_{00}\\left(\\bar x,\\bar y\\right)} b_{i}\\xi^{i}_{t}\n+\\sum\\limits_{i=1}^{\\ell}\\beta_i\\xi^{\\prime,i}=0.\n\\]\nFor $\\bar i \\in a_{00}^x\\left(\\bar x,\\bar y\\right)$ we consider the $(n+\\bar i)$-th row of this sum\n\\[\nb_{\\bar i}\\underbrace{\\xi^{\\bar i}_{t,n+\\bar i}}_{\\ne 0}+\\sum\\limits_{i \\in a_{00}\\left(\\bar x,\\bar y\\right)\\backslash\\left\\{\\bar i\\right\\}} b_{i}\\underbrace{\\xi^{i}_{t,n+\\bar i}}_{=0}+\\sum\\limits_{i=1}^{\\ell}\\beta_i\\underbrace{\\xi^{\\prime,i}_{n+\\bar i}}_{=0}=0.\n\\]\nIf instead $\\bar i \\in a_{00}^y\\left(\\bar x,\\bar y\\right)$ we consider the $\\bar i$-th row of the sum\n\\[\nb_{\\bar i}\\underbrace{\\xi^{\\bar i}_{t,\\bar i}}_{\\ne 0}+\\sum\\limits_{i \\in a_{00}\\left(\\bar x,\\bar y\\right)\\backslash\\left\\{\\bar i\\right\\}} b_{i}\\underbrace{\\xi^{i}_{t,\\bar i}}_{=0}+\\sum\\limits_{i=1}^{\\ell}\\beta_i\\underbrace{\\xi^{\\prime,i}_{\\bar i}}_{=0}=0.\n\\]\nAltogether, it must hold $b_{i} =0$, $i \\in a_{00}\\left(\\bar x,\\bar y\\right)$. However, this implies\n\\[\n\\sum\\limits_{i=1}^{\\ell}\\beta_i\\xi^{\\prime,i}=0.\n\\]\nHence, $\\beta_i=0$ for $i=1,\\ldots,\\ell$.\n\n    b) It holds $\\xi^{\\bar i}_{t} \\in \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$, cf. Step 1a, for any $\\bar i \\in a_{00}(\\bar x,\\bar y)$. \n    \n    c) It holds $\\left(\\eta_i^{\\le,t}-\\eta_i^{\\ge,t}\\right)\\xi^{\\bar i}_{t,i}\\xi^{\\bar i}_{t,n+i} \\leq 0, i \\in \\mathcal{H}\\left(x^t,y^t\\right)$. Since $\\xi^{\\bar i}_{t} \\in \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$, we obtain:\n    \\[\n\\begin{array}{rcl}\n\\left(\\eta_i^{\\le,t}-\\eta_i^{\\ge,t}\\right)\\xi^{\\bar i}_{t,i}\\xi^{\\bar i}_{t,n+i}\n&=&\\left(\\eta_i^{\\ge,t}-\\eta_i^{\\le,t}\\right)\\left(\\xi^{\\bar i}_{t,n+i}\\right)^2\\frac{x^t_i}{y^t_i}.\n\\end{array}\n\\]\nIf $i\\in \\mathcal{H}^{\\ge}\\left(x^t,y^t\\right)$,\nwe have $x_i^t<0,y_i^t>0$ and $\\eta_i^{\\le,t}=0$. Moreover, due to ND2, we have\n$\\eta_i^{\\ge,t}>0$. The assertion follows immediately. The other case $i\\in \\mathcal{H}^{\\le}\\left(x^t,y^t\\right)$ is completely analogous.\n\n    d) It holds $\\lim\\limits_{t\\to 0}\\left(\\eta_{\\bar i}^{\\le,t}-\\eta_{\\bar i}^{\\ge,t}\\right)\\xi^{\\bar i}_{t,\\bar i}\\xi^{\\bar i}_{t,n+\\bar i}=-\\infty$. \n    We calculate:\n\\[\n\\begin{array}{rcl}\n\\left(\\eta_{\\bar i}^{\\le,t}-\\eta_{\\bar i}^{\\ge,t}\\right)\\xi^{\\bar i}_{t,{\\bar i}}\\xi^{\\bar i}_{t,n+{\\bar i}}\n&=&\\left\\{\n\\begin{array}{ll}\n   \\left(\\eta_{\\bar i}^{\\ge,t}-\\eta_{\\bar i}^{\\le,t}\\right)\\left(\\xi^{\\bar i}_{t,n+\\bar i}\\right)^2\\frac{x^t_{\\bar i}}{y^t_{\\bar i}}& \\mbox{for } \\bar i \\in a_{00}^x(\\bar x,\\bar y),\\\\\n   \\left(\\eta_{\\bar i}^{\\ge,t}-\\eta_{\\bar i}^{\\le,t}\\right)\\left(\\xi^{\\bar i}_{t,\\bar i}\\right)^2\\frac{y^t_{\\bar i}}{x^t_{\\bar i}}& \\mbox{for } \\bar i \\in a_{00}^y(\\bar x,\\bar y).\n\\end{array}\\right.\n\\end{array}\n\\]\nLet us suppose $\\bar i \\in a_{00}^x(\\bar x,\\bar y)$. We have $y^t_{\\bar i}\\ne 0$ and, thus, $\\nu^t_{\\bar i}=0$. We use Remark \\ref{rem:multipliers} and NDT3 to conclude that the sequence\n$ \\left(\\eta_{\\bar i}^{\\ge,t}-\\eta_{\\bar i}^{\\le,t}\\right)x^t_{\\bar i}$ converges to $\\varrho_{2,\\bar i}<0$ for $t \\to 0$. Further,\n$\\frac{1}{y^t_{\\bar i}}>0$ tends to infinity for $t \\to 0$. Finally,\n$\\left(\\xi^{\\bar i}_{t,n+\\bar i}\\right)^2$ converges to $1$ for $t \\to 0$, due to the construction of\n$\\xi^{\\bar i}_t$.\nThus, the assertion follows.\nInstead, let us suppose $\\bar i \\in a_{00}^y(\\bar x,\\bar y)$. \nThis time, we have that\n$\\left(\\xi^{\\bar i}_{t,\\bar i}\\right)^2$ converges to $1$ for $t \\to 0$.\nDue to Remark \\ref{rem:multipliers} and NDT3, $\\left(\\eta_{\\bar i}^{\\ge,t}-\\eta_{\\bar i}^{\\le,t}\\right)y^t_{\\bar i}$ converges to $\\varrho_{1,\\bar i}\\ne 0$ for $t \\to 0$.\nIf $\\bar i\\in \\mathcal{H}^{\\ge}\\left(x^t,y^t\\right)$, then $\\varrho_{1,\\bar i}$ is positive from here. Also, $\\frac{1}{x^t_{\\bar i}}<0$ tends to minus infinity for $t \\to 0$. The other case $\\bar i\\in \\mathcal{H}^{\\le}\\left(x^t,y^t\\right)$ is completely analogous. \n\ne) We notice that $\\displaystyle \\xi^{\\bar i ^T}_{t} D^2 L^\\mathcal{R}\\left( x^t, y^t\\right)\\xi^{\\bar i}_{t}$ converges for $t \\to 0$ due to the construction above.\n\nFinally, for $\\bar i\\in  a_{00}(\\bar x,\\bar y)$ we estimate:\n\\[\n\\begin{array}{rcl}\n\\xi_t^{\\bar i ^T} D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\xi_t^{\\bar i}&\\overset{(\\ref{eq:Hessetandbar})}{=}&\n\\displaystyle \\xi^{\\bar i ^T}_{t} D^2 L^\\mathcal{R}\\left(x^t, y^t\\right)\\xi^{\\bar i}_{t}\\\\ && \\displaystyle\n- \\sum\\limits_{i\\in \\mathcal{H}^\\ge\\left(x^t,y^t\\right)} 2 \\eta_i^{\\ge,t} \\xi_{t,i}^{\\bar i}  \\xi_{t,n+i}^{\\bar i}+\n\\sum\\limits_{i\\in \\mathcal{H}^\\le\\left(x^t,y^t\\right)} 2 \\eta_i^{\\le,t}\\xi_{t,i}^{\\bar i}\\xi_{t,n+i}^{\\bar i} \\\\\n&= &\n\\displaystyle \\xi^{\\bar i ^T}_{t} D^2 L^\\mathcal{R}\\left(x^t, y^t\\right)\\xi^{\\bar i}_{t}\n +2\\sum\\limits_{i\\in \\mathcal{H}\\left(x^t,y^t\\right)} \\left( \\eta_i^{\\le,t}-\\eta_i^{\\ge,t} \\right)\\xi_{t,i}^{\\bar i}  \\xi_{t,n+i}^{\\bar i}\\\\ \n&\\overset{c)}{\\le}&\\displaystyle \\xi^{\\bar i ^T}_{t} D^2 L^\\mathcal{R}\\left(x^t, y^t\\right)\\xi^{\\bar i}_{t}+2\\left(\\eta_{\\bar i}^{\\le,t}-\\eta_{\\bar i}^{\\ge,t}\\right)\\xi^{\\bar i}_{t,\\bar i}\\xi^{\\bar i}_{t,n+\\bar i}.\n\\end{array}\n\\]\nThus, due to d) and e), $\\xi^{\\bar i ^T}_{t} D^2 L^\\mathcal{S}\\left(x^t, y^t\\right) \\xi^{\\bar i}_{t}$ has to be negative for $t$ small enough. \nAs we have seen in Step 2c, it holds $\\mathcal{T}^\\prime \\subset \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$. Then, due a) and b), we have therefore:\n\\[\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}}\n\\geq QI^\\mathcal{S}_{t,\\mathcal{T}^\\prime} + \\left\\vert a_{00}(\\bar x,\\bar y)\\right\\vert.\n\\]\nBy Step 2a, we have $QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}=\\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$, and by Step 2b, $QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}=QI^\\mathcal{S}_{t,\\mathcal{T}^\\prime}$. Overall, we obtain:\n\\[\nm=QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}}\n\\ge\n\\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}+\\left\\vert a_{00}(\\bar x,\\bar y)\\right\\vert=TI.\n\\]\n\\end{proof}","sha256_utf8":"b2e74bacf727dd3e44c229f85e40663937c640a334e64802caad8b8230d74877"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"基数约束问题与光滑性","source_units":[{"start":3268,"end":4021,"start_line":48,"end_line":56,"raw_tex":"In this paper, we study the Scholtes-type regularization method for the class of cardinality-constrained optimization problems:\n\\[\n\\mbox{CCOP}: \\quad\n\\min_{x} \\,\\, f(x)\\quad \\mbox{s.\\,t.} \\quad h(x)=0, \\quad g(x)\\ge 0, \\quad \\left\\|x\\right\\|_0\\le s\n\\]\nwith the feasible set given by equality, inequality, and cardinality constraints, where the so-called zero ”norm” $\\left\\|x\\right\\|_0 = \\left\\vert \\left\\{i \\in \\{1,\\ldots, n\\}\\; \\vert\\; x_i\\ne 0\\right\\}\\right\\vert$ is counting non-zero entries of $x$.\nHere, we assume that the objective function $f$, as well as the equality and inequality constraints \n$h=\\left(h_p, p \\in P\\right)$,\n$g=\\left(g_q, q \\in Q\\right)$ are twice continuously differentiable, and $s \\in \\{0,1,\\ldots, n-1\\}$ is an integer. \n","sha256_utf8":"348b1d38b9a7de96363fc9ca7c63ecfc90ce60163b1c7bcefbc741b8bec77a04"}]},{"id":"S002","semantic_type":"notation","kind":"notation","title":"向量与微分记号","source_units":[{"start":8502,"end":8918,"start_line":92,"end_line":92,"raw_tex":"Our notation is standard. The cardinality of a finite set $A$ is denoted by $\\vert A \\vert$. The $n$-dimensional Euclidean space is denoted by $\\mathbb{R}^n$ with the coordinate vectors $e_i,i= 1,\\ldots, n$. The vector consisting of ones is denoted by $e$. Given a twice continuously differentiable function $f:\\mathbb{R}^n\\rightarrow \\mathbb{R}$, $\\nabla f$ denotes its gradient, and $D^2f$ stands for its Hessian.\n","sha256_utf8":"2529c2b903c6fb8bffac0f5ffb04035a2322d7c17a765d4470e9567379e00803"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"正则重构与活动集","source_units":[{"start":8966,"end":10435,"start_line":95,"end_line":117,"raw_tex":"We associate with CCOP the regularized continuous reformulation from \\cite{laemmel:reform}:\n\\[\n     \\begin{array}{rl}\n    \\displaystyle \\mathcal{R}(c,\\varepsilon): \\quad \\min_{x,y} \\,\\, f(x) +c^Ty\\quad \\mbox{s.\\,t.} \n   &h(x)=0, \\quad g(x)\\ge 0, \\quad \\displaystyle \\sum_{i=1}^{n} y_i \\geq  n - s, \\quad \\\\\n   &x_i y_i =0, \\quad 0 \\leq y_i \\leq 1+\\varepsilon,  \\quad i=1, \\ldots, n,\n   \\end{array}\n\\]\nwhere the components of $c \\in \\R^n$ $c$ are positive and pairwise different, and $0 < \\varepsilon \\leq \\frac{1}{n-s}$.\nGiven a feasible point  $(\\bar x,\\bar y)$ of $\\mathcal{R}$, we define the index sets which correspond to the orthogonality type constraints $x_i y_i=0$, $y_i \\geq 0$:\n    \\[\n     \\begin{array}{c}\n        a_{01}\\left(\\bar x,\\bar y\\right)=\\left\\{i\\,\\left\\vert\\, \\bar x_i=0,\\bar y_i>0\\right.\\right\\}, \\quad \n    a_{10}\\left(\\bar x,\\bar y\\right)=\\left\\{i\\,\\left\\vert\\, \\bar x_i\\ne 0,\\bar y_i=0\\right.\\right\\}, \\\\\n    a_{00}\\left(\\bar x,\\bar y\\right)=\\left\\{i\\,\\left\\vert\\, \\bar x_i=0,\\bar y_i=0\\right.\\right\\}.       \n     \\end{array}\n    \\]\n%The $y$-components which attain the upper bound are stored in  \n%     \\[\\mathcal{E}(\\bar y)=\\left\\{i \\in\\{1,\\ldots,n\\}\\left\\vert\\,\\bar y_i=1+\\varepsilon\\right.\\right\\}.\\]\nThe index sets of the active inequality constraints will be denoted by\n\\[\n Q_0(\\bar x)=\\left\\{q \\in Q \\, \\left\\vert\\, g_q(\\bar x)=0\\right.\\right\\}, \\quad\\mathcal{E}(\\bar y)=\\left\\{i \\,\\left\\vert\\,\\bar y_i=1+\\varepsilon\\right.\\right\\}.\n\\]\n","sha256_utf8":"68cc0f1e18768210097ba4ca01104a30db26619dfb5d4e7be0487fbc94612b16"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"正交约束线性独立性","source_units":[{"start":11495,"end":12350,"start_line":127,"end_line":157,"raw_tex":"\\begin{definition}[MPOC-LICQ, \\cite{laemmel:reform}]\nWe say that a feasible point $(\\bar x,\\bar y)$ of $\\mathcal{R}$ satisfies the MPOC-tailored linear independence constraint qualification (MPOC-LICQ) if the following vectors are linearly independent:\n\\[\n\\begin{array}{c}\n\\begin{pmatrix}\n\\nabla h_p(\\bar x)\\\\0\n\\end{pmatrix}, p \\in P, \\quad \n\\begin{pmatrix}\n\\nabla g_q(\\bar x)\\\\0\n\\end{pmatrix}, q \\in Q_0(\\bar x), \\quad  \n\\begin{pmatrix}\n0\\\\\ne_i\n\\end{pmatrix},i\\in \\mathcal{E}(\\bar y), \\quad\n\\begin{pmatrix}\n0\\\\\ne\n\\end{pmatrix}\n\\mbox{ if } \\sum\\limits_{i=1}^{n} \\bar y_i = n - s,\n\\\\\n\\begin{pmatrix}\ne_i\\\\\n0\n\\end{pmatrix}, i \\in a_{01}\\left(\\bar x,\\bar y\\right)\\cup a_{00}\\left(\\bar x,\\bar y\\right), \\quad \n\\begin{pmatrix}\n0\\\\\ne_i\n\\end{pmatrix}, i \\in a_{10}\\left(\\bar x, \\bar y\\right)\\cup a_{00}\\left(\\bar x, \\bar y\\right).\n\\end{array}\n\\]\n\\end{definition}","sha256_utf8":"b52ecba7ac654c2c5d20767453f0fd664013a7d1ee2428a35b341c363193ff30"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"T驻点及乘子条件","source_units":[{"start":12352,"end":14396,"start_line":159,"end_line":221,"raw_tex":"\\begin{definition}[T-stationary point, \\cite{laemmel:reform}]\n\\label{def:t-stat}\nA feasible point $(\\bar x,\\bar y)$ of $\\mathcal{R}$ is called T-stationary if there exists multipliers \n\\[\n\\begin{array}{l}\n\\bar \\lambda_p, p \\in P,\n\\bar \\mu_{1,q}, q\\in Q_0(\\bar x),\\bar \\mu_{2,i},i \\in \\mathcal{E}(\\bar y),\\bar \\mu_3,\\\\ \\bar \\sigma_{1,i}, i \\in a_{01}\\left(\\bar x,\\bar y\\right),\\bar \\sigma_{2,i}, i \\in a_{10}\\left(\\bar x,\\bar y\\right),\\bar \\varrho_{1,i},\\bar \\varrho_{2,i}, i \\in a_{00}\\left(\\bar x, \\bar y\\right),\n\\end{array}\n\\]\nsuch that the following conditions hold:\n\\begin{equation}\n   \\label{eq:tstat-1} \n   \\begin{array}{rcl}\n  \\begin{pmatrix}\n   \\nabla f(\\bar x)\\\\\n   c\n  \\end{pmatrix}&=& \\displaystyle\\sum\\limits_{p \\in P}\\bar \\lambda_p \n  \\begin{pmatrix}\n  \\nabla h_p(\\bar x)\\\\\n  0\n  \\end{pmatrix}+\n    \\sum\\limits_{q \\in Q_0(\\bar x)}\\bar \\mu_{1,q} \n    \\begin{pmatrix}\n    \\nabla g_q(\\bar x)\\\\\n    0\n    \\end{pmatrix}-\n    \\sum\\limits_{i\\in\\mathcal{E}(\\bar y)} \\bar \\mu_{2,i} \\begin{pmatrix}\n    0\\\\\n    e_i\n\\end{pmatrix}\\\\ \n    && +\n\\bar \\mu_3 \\begin{pmatrix}\n0\\\\\ne\n\\end{pmatrix}\\displaystyle +\\sum\\limits_{i \\in a_{01}\\left(\\bar x,\\bar y\\right)} \\bar \\sigma_{1,i}\n    \\begin{pmatrix}\n    e_{ i}\\\\\n    0\n    \\end{pmatrix}\n    +\\sum\\limits_{i \\in a_{10}\\left(\\bar x,\\bar y \\right)} \\bar \\sigma_{2,i}    \\begin{pmatrix}\n    0\\\\\n    e_{ i}\n    \\end{pmatrix}\\\\ \n    &&\\displaystyle+\\sum\\limits_{i \\in a_{00}\\left(\\bar x, \\bar y\\right)} \\left(\\bar \\varrho_{1,i}\n    \\begin{pmatrix}\n   e_{ i}\\\\\n    0\n    \\end{pmatrix}\n   +\\bar \\varrho_{2,i}\n    \\begin{pmatrix}\n    0\\\\\n    e_{ i}\n    \\end{pmatrix}\\right), \\end{array}\n\\end{equation}\n\\begin{equation}\n   \\label{eq:tstat-2} \\bar \\mu_{1,q} \\ge 0,q\\in Q_0\\left(\\bar x\\right),\n   \\bar \\mu_{2,i} \\ge 0, i\\in \\mathcal{E}(\\bar y),\\bar \\mu_3\\ge 0, \\bar\\mu_3\\left(\n   \\sum\\limits_{i=1}^{n} \\bar y_i -(n-s)\\right)=0,\n\\end{equation}\n\\begin{equation}\n   \\label{eq:tstat-3} \\bar \\varrho_{1,i}=0 \\mbox{ or }\\bar \\varrho_{2,i}\\le 0, i \\in a_{00}\\left(\\bar x,\\bar y\\right).\n\\end{equation}\n\\end{definition}","sha256_utf8":"45d0493eef254723b50d1eda72bfab429bad0ae48140682d537d9036879543ea"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"重构拉格朗日函数与切空间","source_units":[{"start":14397,"end":15885,"start_line":222,"end_line":270,"raw_tex":"We define the appropriate Lagrange function:\n\\[\n        \\begin{array}{rcl}\n        L^\\mathcal{R}(x,y)&=& \\displaystyle\n   f(x)+\n   c^T y -\n        \\sum\\limits_{p \\in P}\\bar \\lambda_p\n  h_p( x)-\n    \\sum\\limits_{q \\in Q_0(\\bar x)}\\bar \\mu_{1,q} \n    g_q(x) \\\\  & &\\displaystyle+\\sum\\limits_{i\\in\\mathcal{E}(\\bar y)} \\bar \\mu_{2,i} \n   \\left( y_i- (1+\\varepsilon)\\right)\n-\n\\bar \\mu_3 \n \\left(\\sum\\limits_{i=1}^{n} y_i - (n-s)\\right)\n   \\\\ \n    &&\\displaystyle -\\sum\\limits_{i \\in a_{01}\\left(\\bar x, \\bar y\\right)} \\bar \\sigma_{1,i}\n     x_{ i}\n    -\\sum\\limits_{i \\in a_{10}\\left(\\bar x, \\bar y\\right)} \\bar \\sigma_{2,i}   \n     y_{ i} -\\sum\\limits_{i \\in a_{00}\\left(\\bar x, \\bar y \\right)} \\left(\\bar \\varrho_{1,i}\n    x_{ i}\n   +\\bar \\varrho_{2,i}\n     y_{ i}\n    \\right).\n    \\end{array}\n\\]\nMoreover, we set for the corresponding tangent space:\n\n\\[\n    \\mathcal{T}^{\\mathcal{R}}_{(\\bar x,\\bar y)} =\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\nDh_p(\\bar x), 0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(\\bar x), 0\\end{pmatrix}\\xi=0,q \\in Q_0(\\bar x),\\\\\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(\\bar y), \n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0 \\mbox{ if } \\displaystyle \\sum_{i=1}^{n} \\bar y_i = n - s,\n\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y) \\cup a_{01}(\\bar x,\\bar y),\\\\\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y) \\cup a_{10}(\\bar x,\\bar y)\n\\end{array}\n\\right.\\right\\}.\n\\]\n","sha256_utf8":"a957e3f8dd163dc04f5b5a447c73347b15d5dd1082ac484a858ee927cd79828d"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"T驻点非退化条件","source_units":[{"start":15886,"end":16898,"start_line":272,"end_line":288,"raw_tex":"\\begin{definition}[Nondegenerate T-stationary point, \\cite{laemmel:reform}]\nA T-stationary point $(\\bar x,\\bar y)$ of $\\mathcal{R}$ with multipliers $(\\bar \\lambda, \\bar \\mu, \\bar \\sigma, \\bar \\varrho)$ is called nondegenerate if\n\nNDT1: MPOC-LICQ holds at $(\\bar x,\\bar y)$,\n\nNDT2: $\\bar \\mu_{1,q}>0$, $q \\in Q_0\\left(\\bar x\\right)$, $\\bar \\mu_{2,i}>0$, $i \\in \\mathcal{E}\\left(\\bar y\\right)$, and $\\bar \\mu_3>0$ if  $\\sum\\limits_{i=1}^{n} \\bar  y_i =n-s$,\n\nNDT3: $\\bar\\varrho_{1,i}\\ne 0$ and $\\bar\\varrho_{2,i}< 0$, $i\\in a_{00}\\left(\\bar x,\\bar y\\right)$,\n    \nNDT4: the matrix $D^2 L^{\\mathcal{R}}(\\bar x,\\bar y)\\restriction_{\\mathcal{T}^\\mathcal{R}_{(\\bar x,\\bar y)}}$ is nonsingular.\n\n\\noindent\nFor a nondegenerate T-stationary point we eventually use an additional condition:\n%\\item [] NDT5: if $a_{00}\\left(\\bar x,\\bar y\\right)\\not = \\emptyset$, then $\\bar \\sigma_{1,i_{01}}\\ne 0$ for all $i_{01}\\in a_{01}(\\bar x, \\bar y)$.\n\nNDT6: $\\bar \\sigma_{1,i}\\ne 0$, $i\\in a_{01}(\\bar x, \\bar y)$.\n\\end{definition}","sha256_utf8":"c90cd99097bec75cfa77e4dbd09d44ad130c1ada5ebd01ab770c76a504238867"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"二次指标与T指标","source_units":[{"start":16900,"end":17492,"start_line":290,"end_line":292,"raw_tex":"\\begin{definition}[T-index, \\cite{laemmel:reform}]\nLet $(\\bar x,\\bar y)$ be a nondegenerate T-stationary point of $\\mathcal{R}$ with unique multipliers $\\left(\\bar \\lambda,\\bar \\mu, \\bar \\sigma,\\bar \\varrho\\right)$. The number of negative eigenvalues of the matrix $D^2 L^{\\mathcal{R}}(\\bar x,\\bar y)\\restriction_{\\mathcal{T}^\\mathcal{R}_{(\\bar x,\\bar y)}}$ is called its quadratic index ($QI$). The cardinality of $a_{00}\\left(\\bar x,\\bar y\\right)$ is called the biactive index ($BI$) of $(\\bar x,\\bar y)$. We define the T-index ($TI$) as the sum of both, i.\\,e. $TI=QI+BI$.\n\\end{definition}","sha256_utf8":"17778710c27216cc1e3080168a41bb09472670aee023c5e062d8245da72c1dc7"}]},{"id":"S009","semantic_type":"problem_setting","kind":"problem_setting","title":"Scholtes正则化与活动集","source_units":[{"start":22596,"end":24219,"start_line":362,"end_line":385,"raw_tex":"Let us now regularize the orthogonality type constraints in $\\mathcal{R}$ by using the Scholte's idea, cf. \\cite{scholtes:2001}: \n\\[\n     \\begin{array}{rl}\n    \\displaystyle \\mathcal{S}(t): \\quad \\min_{x,y} \\,\\, f(x) +c^Ty\\quad \\mbox{s.\\,t.} \n   &h(x)=0, \\quad g(x)\\ge 0, \\quad\\displaystyle \\sum_{i=1}^{n} y_i \\geq  n - s, \\\\\n   & -t\\le x_i y_i \\le t, \\quad 0 \\leq y_i \\leq 1+\\varepsilon,  \\quad \n         i=1, \\ldots, n,\n   \\end{array}\n\\]\nwhere $t>0$. \n%Sometimes, we write $S(t)$ if the dependence on $t$ is highlighted. \nNote that $\\mathcal{S}$ from above falls into the scope of nonlinear programming, i.e. just smooth equality and inequality constraints are present. The notation for the sets %$a_{00}\\left(x, y\\right)$, $a_{01}\\left(x, y\\right)$, $a_{10}\\left(x, y\\right)$, \n$Q_0(x)$ and $\\mathcal{E}(y)$, which were used for $\\mathcal{R}$, will be used here again.\nFurthermore, we define for a feasible point $\\left( x, y\\right)$ of $\\mathcal{S}$ the index set of vanishing $y$-components as well as the index sets of active relaxed orthogonality type constraints:\n\\[\n\\mathcal{N}( y)=\\left\\{i\\, \\left\\vert\\,  y_i=0\\right.\\right\\},\n\\quad\n\\mathcal{H}^{\\ge}\\left(x, y\\right)=\\left\\{i\\,\\left\\vert \\,x_i  y_i=-t\\right.\\right\\}, \\mathcal{H}^{\\le}\\left( x, y\\right)=\\left\\{i\\,\\left\\vert \\, x_i  y_i=t\\right.\\right\\}.\n\\]\nWe also eventually use the following index sets:\n\\[\n\\mathcal{H}\\left(x, y\\right)=\\mathcal{H}^\\geq\\left(x, y\\right) \\cup \\mathcal{H}^\\leq\\left( x, y\\right), \\quad \n\\mathcal{O}\\left( x, y\\right)=\\left(\\mathcal{E}\\left( y\\right)\\cup \\mathcal{N}\\left(y\\right)\\cup \\mathcal{H}\\left( x, y\\right)\\right)^c.\n\\]\n","sha256_utf8":"e2ce68583dd0784228ec2d4978c6cff09adbc75c123bc7563ede2326d27eb11a"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"正则化线性独立性","source_units":[{"start":24362,"end":25053,"start_line":389,"end_line":418,"raw_tex":"\\begin{definition}[LICQ]\nWe say that a feasible point $(x, y)$ of $\\mathcal{S}$ satisfies the linear independence constraint qualification (LICQ) if the following vectors are linearly independent:\n\\[\n\\begin{array}{c}\n      \\begin{pmatrix}\n\\nabla h_p(x)\\\\0\n\\end{pmatrix}, p \\in P, \\quad \n\\begin{pmatrix}\n\\nabla g_q(x)\\\\0\n\\end{pmatrix}, q \\in Q_0(x), \\quad  \n\\begin{pmatrix}\n0\\\\\ne_i\n\\end{pmatrix},i\\in \\mathcal{E}(y),\n\\\\ \\begin{pmatrix}\n0\\\\\ne\n\\end{pmatrix}\n\\mbox{ if } \\sum\\limits_{i=1}^{n} y_i = n - s, \\quad\n\\begin{pmatrix}\ny_i e_i\\\\\nx_i e_i\n\\end{pmatrix}, i \\in \\mathcal{H}\\left(x,y\\right), \\quad \n\\begin{pmatrix}\n0\\\\\ne_i\n\\end{pmatrix}, i \\in \\mathcal{N}(y).\n\\end{array}\n\\]\n\\end{definition}","sha256_utf8":"52b085f26c648f2aea8cf7732f25498de5741679cebaf928932002f584e73a0b"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"正则化KKT条件","source_units":[{"start":32014,"end":33613,"start_line":623,"end_line":681,"raw_tex":"\\begin{definition}[Karush-Kuhn-Tucker point]\n    A feasible point $(x, y)$ of $\\mathcal{S}$ is called Kurush-Kuhn-Tucker if there exist multipliers \n\\[\n\\lambda_p, p \\in P,\n \\mu_{1,q}, q\\in Q_0( x), \\mu_{2,i},i \\in \\mathcal{E}( y), \\mu_3,\n\\eta^\\ge_i, \\eta^\\le_i, \\nu_i, i\\in\\left\\{1,\\ldots,n\\right\\},\n\\]\nsuch that the following conditions hold:\n\\begin{equation}\n   \\label{eq:kkt-1} \n   \\begin{array}{rcl}\n  \\begin{pmatrix}\n   \\nabla f( x)\\\\\n   c\n  \\end{pmatrix}&=& \\displaystyle \\sum\\limits_{p \\in P} \\lambda_p \n  \\begin{pmatrix}\n  \\nabla h_p( x)\\\\\n  0\n  \\end{pmatrix}+\n    \\sum\\limits_{q \\in Q_0( x)} \\mu_{1,q} \n    \\begin{pmatrix}\n    \\nabla g_q( x)\\\\\n    0\n    \\end{pmatrix}\n   - \\sum\\limits_{i\\in\\mathcal{E}( y)}  \\mu_{2,i} \\begin{pmatrix}\n    0\\\\\n     e_i\n\\end{pmatrix}+\n \\mu_3 \\begin{pmatrix}\n0\\\\\ne\n\\end{pmatrix} \\\\ \n    && \\displaystyle\n+ \\sum\\limits_{i\\in\\mathcal{H}^\\ge( x,  y)}  \\eta^\\ge_{i} \\begin{pmatrix}\n    y_ie_i\\\\\n   x_ie_i\n\\end{pmatrix}\n-\\sum\\limits_{i\\in\\mathcal{H}^\\le( x,  y)}  \\eta^\\le_{i} \\begin{pmatrix}\n    y_ie_i\\\\\n    x_ie_i\n\\end{pmatrix}\n+\n    \\sum\\limits_{i\\in\\mathcal{N}( y)}  \\nu_{i} \\begin{pmatrix}\n    0\\\\\n    e_i\n\\end{pmatrix},\n\\end{array}\n\\end{equation}\n\\begin{equation}\n   \\label{eq:kkt-2}  \\mu_{1,q} \\ge 0,q\\in Q_0\\left( x\\right),\n    \\mu_{2,i} \\ge 0, i\\in \\mathcal{E}(y),\n    \\mu_3\\ge 0, \\mu_3\\left(\n   \\sum\\limits_{i=1}^{n}  y_i -(n-s)\\right)=0,\n\\end{equation}\n\\begin{equation}\n   \\label{eq:kkt-3} \n  \\eta^\\ge_i\\ge 0,i \\in \\mathcal{H}^\\ge( x,  y), \\eta^\\le_i\\ge 0,i \\in \\mathcal{H}^\\le( x,  y), \\nu_i\\ge 0, i \\in \\mathcal{N}( y).\n\\end{equation}\n\\end{definition}","sha256_utf8":"eab33f17a0c5df5616c4c8f6b492afdf2b013d0317289cea29b66a55490e01fd"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"正则化拉格朗日函数与切空间","source_units":[{"start":33614,"end":34961,"start_line":682,"end_line":729,"raw_tex":"We again define the Lagrange function as\n\n\\[\n        \\begin{array}{rcl}\n        L^\\mathcal{S}(x,y)&=& \\displaystyle\n   f(x)+c^Ty\n       - \\sum\\limits_{p \\in P} \\lambda_p \n  h_p(x)-\n    \\sum\\limits_{q \\in Q_0(x)} \\mu_{1,q} \n    g_q(x)\n      \\\\&& \\displaystyle +\\sum\\limits_{i\\in\\mathcal{E}(y)}\n   \\mu_{2,i}\n    \\left(y_i - (1+\\varepsilon)\\right)\n -\\mu_3\\left(\\sum\\limits_{i=1}^n y_i - (n-s)\\right)  \\\\ \n    && \\displaystyle\n- \\sum\\limits_{i\\in\\mathcal{H}^\\ge( x,  y)}  \\eta^\\ge_{i}\n    \\left(x_iy_i +t \\right)\n+\\sum\\limits_{i\\in\\mathcal{H}^\\le( x,  y)}  \\eta^\\le_{i}\n    \\left(x_iy_i-t\\right)\n-\n    \\sum\\limits_{i\\in\\mathcal{N}( y)}  \\nu_{i}\n    y_i.\n\\end{array}\n\\]\nThe tangent space is given by\n\n\\[\n    \\mathcal{T}^\\mathcal{S}_{(x,y)} =\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\nDh_p( x),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q( x),0\\end{pmatrix}\\xi=0,q \\in Q_0( x),\\\\ \\displaystyle\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(y),\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0 \\mbox{ if } \\sum_{i=1}^{n} y_i = n - s,\\\\\n\\begin{pmatrix}\ny_ie_i,x_ie_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{H}\\left(x,y\\right), \n%\\begin{pmatrix} -y_ie_i&-x_ie_i \\end{pmatrix}\\xi=0, i \\in \\mathcal{H}^\\le\\left(x,y\\right),\\\\\n\\begin{pmatrix}\n    0,e_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{N}(y)\n\\end{array}\n\\right.\\right\\}.\n\\]\n","sha256_utf8":"0959f7eab6e0630a60e423efb8a94e976cc9c54cf3d5ad4d9812646242968bcd"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"KKT点非退化条件","source_units":[{"start":34962,"end":35592,"start_line":731,"end_line":743,"raw_tex":"\\begin{definition}[Nondegenerate Karush-Kuhn-Tucker point]\nA Karush-Kuhn-Tucker point $(x, y)$ of $\\mathcal{S}$ with multipliers $(\\lambda, \\mu, \\eta, \\nu)$\nis called nondegenerate if \n\nND1: LICQ holds at $( x, y)$,\n\nND2: $ \\mu_{1,q}>0$, $q \\in Q_0\\left(x\\right)$, $ \\mu_{2,i}>0$, $i \\in \\mathcal{E}\\left(y\\right)$,\n    $\\eta^\\ge_i>0$, $i \\in \\mathcal{H}^\\ge(x,y)$,\n    $\\eta^\\le_i>0$, $i \\in \\mathcal{H}^\\le(x,y)$,\n    $\\nu_i >0$, $i \\in \\mathcal{N}(y)$, and $ \\mu_3>0$ if  $\\sum\\limits_{i=1}^{n} y_i =n-s$,\n\nND3: the matrix $D^2 L^\\mathcal{S}(x,y) \\restriction_{\\mathcal{T}^\\mathcal{S}_{(x,y)}}$ is nonsingular.\n\\end{definition}","sha256_utf8":"082fd403dc6dbdc7f685b1ad9efdd2841f14150d1b70cca35d6f3988937d6057"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"KKT点二次指标","source_units":[{"start":35594,"end":35917,"start_line":745,"end_line":748,"raw_tex":"\\begin{definition}[Quadratic index]\nLet $(x,y)$ be a Karush-Kuhn-Tucker point of $\\mathcal{S}$ with unique multipliers $(\\lambda, \\mu, \\eta, \\nu)$.\nThe  number of negative eigenvalues of the matrix $D^2 L^\\mathcal{S}(x,y)\\restriction_{\\mathcal{T}^\\mathcal{S}_{(x,y)}}$ is called its quadratic index ($QI$).\n\\end{definition}","sha256_utf8":"183908b18c3317474bedda15a2fc2e9a337556f1cc937a81c963099ed67d3337"}]},{"id":"S015","semantic_type":"external_fact","kind":"external_fact","title":"已有MPOC收敛定理","source_units":[],"declared_external":true,"declaration_text":["For Scholtes-type regularization of an MPOC, a convergent sequence of KKT points whose limit satisfies MPOC-LICQ has a T-stationary limit. Under that qualification the equality and ordinary inequality multipliers converge, while the product-constraint multipliers converge after multiplication by the corresponding coordinates: the limits are exactly those stated in B0054(a)-(d). These are the cited boundary facts imported from laemmel:mpoc."]},{"id":"S016","semantic_type":"theorem","kind":"theorem","title":"已有T驻点收敛结论","source_units":[{"start":39938,"end":40424,"start_line":812,"end_line":817,"raw_tex":"\\begin{theorem}[Convergence from $\\mathcal{S}$ to $\\mathcal{R}$, cf. \\cite{laemmel:mpoc}]\n\\label{thm:regul}\n%Let $\\left\\{^t\\right\\}_{{t}\\in \\N}$  be a sequence of positive regularization parameters converging to zero. \nSuppose that a sequence of Karush-Kuhn-Tucker points $(x^{t},y^{t})$ of \n$\\mathcal{S}$ converges to $\\left(\\bar x,\\bar y\\right)$ for $t \\rightarrow 0$. If MPOC-LICQ holds at $\\left(\\bar x,\\bar y\\right)$, then it is a T-stationary point of $\\mathcal{R}$.\n\\end{theorem}","sha256_utf8":"574fce9dddb5db6dc773a028f1ba1fd8e433902c2ab26e19f8eecfeb35e075d1"}]},{"id":"S017","semantic_type":"derived_fact","kind":"derived_fact","title":"乘子及其缩放量收敛","source_units":[{"start":40426,"end":40561,"start_line":819,"end_line":819,"raw_tex":"From the proof of Theorem \\ref{thm:regul} in \\cite{laemmel:mpoc} also the convergence of the corresponding multipliers can be deduced.\n","sha256_utf8":"7f6bd18e0bf9305b9f7c3e36c297c3eb5578cabb369fb1d710a6b7951bc3e488"},{"start":40562,"end":41674,"start_line":821,"end_line":833,"raw_tex":"\\begin{remark}[Convergence of multipliers]\n\\label{rem:multipliers}\n Let $\\left(\\lambda^t,\\mu^t,\\eta^t,\\nu^t\\right)$ be the multipliers of the Karush-Kuhn-Tucker points $\\left(x^{t},y^{t}\\right)$ of $\\mathcal{S}$ and $\\left(\\bar \\lambda,\\bar \\mu,\\bar \\sigma, \\bar \\varrho\\right)$ of the T-stationary point $\\left(\\bar x, \\bar y\\right)$ of $\\mathcal{R}$ as in Theorem \\ref{thm:regul}. Due to MPOC-LICQ at $\\left(\\bar x, \\bar y\\right)$, we have: \n \na) $\\lim\\limits_{t \\to 0} \\lambda^t=\\bar \\lambda$,\n    $\\lim\\limits_{t \\to 0} \\mu^t=\\bar \\mu$,\n\nb) $\\lim\\limits_{t \\to 0} \\left(\\eta_{i}^{\\ge,t}-\\eta_{i}^{\\le,t}\\right)y_{i}^t=\\bar \\sigma_{1,i}$, $i \\in a_{01}\\left(\\bar x, \\bar y\\right)$,\n \n c) $\\lim\\limits_{t \\to 0} \\nu_{i}^{t}+ \\left(\\eta_{i}^{\\ge,t}-\\eta_{i}^{\\le,t}\\right)x_{i}^t=\\bar \\sigma_{2,i}$, $i \\in a_{10}\\left(\\bar x, \\bar y\\right)$,\n\nd) $\\lim\\limits_{t \\to 0} \\left(\\eta_{i}^{\\ge,t}-\\eta_{i}^{\\le,t}\\right)y_{i}^t=\\bar \\varrho_{1,i}$, $\\lim\\limits_{t \\to 0} \\nu_{i}^{t}+ \\left(\\eta_{i}^{\\ge,t}-\\eta_{i}^{\\le,t}\\right)x_{i}^t=\\bar \\varrho_{2,i}$, $i \\in a_{00}\\left(\\bar x, \\bar y\\right)$.\n\\end{remark}","sha256_utf8":"061ed992b5e4a45ae7e086e729bf33e6f2da805134e800853d8d82aecab8e3d2"}]},{"id":"S018","semantic_type":"external_fact","kind":"external_fact","title":"隐函数定理","source_units":[],"declared_external":true,"declaration_text":["The finite-dimensional implicit function theorem gives nearby solutions of a continuously differentiable system when its derivative in the solved variables is invertible. For a continuously varying full-row-rank linear constraint matrix, vectors in its kernel can be continued to nearby kernel vectors."]},{"id":"S019","semantic_type":"external_fact","kind":"external_fact","title":"秩稳定性与惯性维数","source_units":[],"declared_external":true,"declaration_text":["Full rank persists under sufficiently small perturbations. The inertia of a continuously varying nonsingular real symmetric matrix is locally constant. A symmetric quadratic form's negative index is the maximal dimension of a subspace on which it is negative definite; restriction to a subspace cannot increase that index, and enlargement by d dimensions can increase it by at most d."]},{"id":"S020","semantic_type":"lemma","kind":"lemma","title":"重构辅助变量结构","source_units":[{"start":19220,"end":19833,"start_line":300,"end_line":310,"raw_tex":"\\begin{lemma}[Auxiliary $y$-variables in $\\mathcal{R}$, \\cite{laemmel:reform}]\n\\label{lem:a01} \n%Assume that the components of $c$ are pairwise different and $\\varepsilon\\le\\frac{1}{n-s}$. \nLet $(\\bar x,\\bar y)$ be a T-stationary point of $\\mathcal{R}$, then it holds:\n\na) the summation inequality constraint is active, i.\\,e. $ \\sum\\limits_{i=1}^{n} \\bar y_i =n - s$,\n\nb) the index set $a_{01}(\\bar x,\\bar y)$ consists of exactly $n-s$ elements,\n   \nc) $n-s-1$ components of $\\bar y$ are equal to $1+\\varepsilon$, one component is equal to $1-(n-s-1)\\varepsilon$, and $s$ remaining components vanish.\n\\end{lemma}","sha256_utf8":"26efbfef9b8d941acca784362e19bd7f33da86055a7d3d29d3cf459eabd77e3e"}]},{"id":"S021","semantic_type":"proof","kind":"proof","title":"重构辅助变量引理证明","source_units":[{"start":19834,"end":22554,"start_line":311,"end_line":358,"raw_tex":"\\begin{proof}\n a)\nLet $(\\bar x,\\bar y)$ be a T-stationary point of $\\mathcal{R}$\nand $\\displaystyle \\sum_{i=1}^{n} \\bar y_i >n - s$. Then, there exist multipliers\n$(\\bar \\lambda, \\bar \\mu, \\bar \\sigma, \\bar \\varrho)$\nsuch that (\\ref{eq:tstat-1})--(\\ref{eq:tstat-3}) are fulfilled.\nSince $\\bar \\mu_3=0$, we have that the $(n+i)$-th row of (\\ref{eq:tstat-1}) reads as\n\\[\nc_i=\\left\\{\n\\begin{array}{ll}\n     -\\bar \\mu_{2,i},&\\mbox{for }i \\in \\mathcal{E}(\\bar y),  \\\\\n     \\bar \\sigma_{2,i}, &\\mbox{for }i \\in a_{10}\\left(\\bar x,\\bar y\\right),\\\\\n     \\bar \\varrho_{2,i}, &\\mbox{for }i\\in a_{00}\\left(\\bar x,\\bar y\\right),\\\\\n     0,&\\mbox{else}.\n\\end{array}\\right.\n\\]\nDue to $c>0$, the sets $a_{01}\\left(\\bar x,\\bar y\\right)$ and $\\mathcal{E}(\\bar y)$ have to be equal and, moreover,\ndue to $\\bar \\mu_{2,i}\\ge 0$, they have to be empty. But then, clearly, $ \\sum_{i=1}^{n} \\bar  y_i =0<n - s$, a contradiction.\n\nb)\nSince $(\\bar x,\\bar y)$ is a T-stationary point, there exist\n$(\\bar \\lambda, \\bar \\mu, \\bar \\sigma, \\bar \\varrho)$\nsuch that (\\ref{eq:tstat-1})--(\\ref{eq:tstat-3}). By the proof of statement a) we can conclude that\n$\\bar \\mu_3 >0$. Hence,\nthe $(n+i)$-th row reads as\n\\begin{equation}\n    \\label{eq:k-throw}\nc_i=\\left\\{\n\\begin{array}{ll}\n     -\\bar \\mu_{2,i}+\\bar \\mu_3,&\\mbox{for }i \\in \\mathcal{E}(\\bar y),  \\\\\n     \\bar \\sigma_{2,i}+\\bar \\mu_3, &\\mbox{for }i \\in a_{10}\\left(\\bar x,\\bar y\\right),\\\\\n     \\bar \\varrho_{2,i}+\\bar \\mu_3, &\\mbox{for }i\\in a_{00}\\left(\\bar x,\\bar y\\right),\\\\\n     \\bar \\mu_3,&\\mbox{else}.\n\\end{array}\\right.\n\\end{equation}\nLet us assume that the index set $a_{01}(\\bar x,\\bar y)$ consists of fewer than $n-s$ elements. Then, we have by using the assumption on $\\varepsilon$:\n\\[ \\sum_{i=1}^{n} \\bar y_i \\le (n-s-1)\\cdot (1+\\varepsilon)\\le n-s-1+\\frac{n-s-1}{n-s} <n - s,\n\\]\na contradiction to feasibility. \nLet us assume that the index set $a_{01}(\\bar x,\\bar y)$ consists of more than $n-s$ elements instead.\nSince the components of $c$ are assumed to be pairwise different, we see from (\\ref{eq:k-throw}) that there exists at most one element in $a_{01}\\left(\\bar x,\\bar y\\right)\\backslash \\mathcal{E}(\\bar y)$ and, consequently, there are at least $n-s$ elements in $\\mathcal{E}(\\bar y)$. Therefore, we have:\n\\[\n\\sum_{i=1}^{n} \\bar y_i \\geq (n-s)\\cdot (1+\\varepsilon) > n-s,\n\\]\nwhich contradicts a). \n\nc) Due to b), $a_{01}(\\bar x,\\bar y)$ consists of exactly $n-s$ elements. We conclude as in b) that there is at most one element in $a_{01}\\left(\\bar x,\\bar y\\right)\\backslash \\mathcal{E}(\\bar y)$. In view of statement a), $\\mathcal{E}(\\bar y)$ cannot consist of $n-s$ elements and, thus, must consist of $n-s-1$ elements. Hence, the statement follows immediately.\n\\end{proof}","sha256_utf8":"b09faccc332acbedd4f890ef4a11d8d190657d2e13886174fbb4e5010b705e88"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"求和约束必须活动","source_units":[{"start":19834,"end":20739,"start_line":311,"end_line":328,"raw_tex":"\\begin{proof}\n a)\nLet $(\\bar x,\\bar y)$ be a T-stationary point of $\\mathcal{R}$\nand $\\displaystyle \\sum_{i=1}^{n} \\bar y_i >n - s$. Then, there exist multipliers\n$(\\bar \\lambda, \\bar \\mu, \\bar \\sigma, \\bar \\varrho)$\nsuch that (\\ref{eq:tstat-1})--(\\ref{eq:tstat-3}) are fulfilled.\nSince $\\bar \\mu_3=0$, we have that the $(n+i)$-th row of (\\ref{eq:tstat-1}) reads as\n\\[\nc_i=\\left\\{\n\\begin{array}{ll}\n     -\\bar \\mu_{2,i},&\\mbox{for }i \\in \\mathcal{E}(\\bar y),  \\\\\n     \\bar \\sigma_{2,i}, &\\mbox{for }i \\in a_{10}\\left(\\bar x,\\bar y\\right),\\\\\n     \\bar \\varrho_{2,i}, &\\mbox{for }i\\in a_{00}\\left(\\bar x,\\bar y\\right),\\\\\n     0,&\\mbox{else}.\n\\end{array}\\right.\n\\]\nDue to $c>0$, the sets $a_{01}\\left(\\bar x,\\bar y\\right)$ and $\\mathcal{E}(\\bar y)$ have to be equal and, moreover,\ndue to $\\bar \\mu_{2,i}\\ge 0$, they have to be empty. But then, clearly, $ \\sum_{i=1}^{n} \\bar  y_i =0<n - s$, a contradiction.\n","sha256_utf8":"207d8c56476da099ce25d368750218e1b5a44643c4c968c36106a16f02b968ac"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"正分量数量的下界","source_units":[{"start":20740,"end":21680,"start_line":330,"end_line":349,"raw_tex":"b)\nSince $(\\bar x,\\bar y)$ is a T-stationary point, there exist\n$(\\bar \\lambda, \\bar \\mu, \\bar \\sigma, \\bar \\varrho)$\nsuch that (\\ref{eq:tstat-1})--(\\ref{eq:tstat-3}). By the proof of statement a) we can conclude that\n$\\bar \\mu_3 >0$. Hence,\nthe $(n+i)$-th row reads as\n\\begin{equation}\n    \\label{eq:k-throw}\nc_i=\\left\\{\n\\begin{array}{ll}\n     -\\bar \\mu_{2,i}+\\bar \\mu_3,&\\mbox{for }i \\in \\mathcal{E}(\\bar y),  \\\\\n     \\bar \\sigma_{2,i}+\\bar \\mu_3, &\\mbox{for }i \\in a_{10}\\left(\\bar x,\\bar y\\right),\\\\\n     \\bar \\varrho_{2,i}+\\bar \\mu_3, &\\mbox{for }i\\in a_{00}\\left(\\bar x,\\bar y\\right),\\\\\n     \\bar \\mu_3,&\\mbox{else}.\n\\end{array}\\right.\n\\end{equation}\nLet us assume that the index set $a_{01}(\\bar x,\\bar y)$ consists of fewer than $n-s$ elements. Then, we have by using the assumption on $\\varepsilon$:\n\\[ \\sum_{i=1}^{n} \\bar y_i \\le (n-s-1)\\cdot (1+\\varepsilon)\\le n-s-1+\\frac{n-s-1}{n-s} <n - s,\n\\]\na contradiction to feasibility. \n","sha256_utf8":"6f059b7296f06a39b5231f34a8536e5cac1dd6ba5a3f075c76ca0814fad6d471"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"正分量数量的上界","source_units":[{"start":21680,"end":22177,"start_line":350,"end_line":355,"raw_tex":"Let us assume that the index set $a_{01}(\\bar x,\\bar y)$ consists of more than $n-s$ elements instead.\nSince the components of $c$ are assumed to be pairwise different, we see from (\\ref{eq:k-throw}) that there exists at most one element in $a_{01}\\left(\\bar x,\\bar y\\right)\\backslash \\mathcal{E}(\\bar y)$ and, consequently, there are at least $n-s$ elements in $\\mathcal{E}(\\bar y)$. Therefore, we have:\n\\[\n\\sum_{i=1}^{n} \\bar y_i \\geq (n-s)\\cdot (1+\\varepsilon) > n-s,\n\\]\nwhich contradicts a). \n","sha256_utf8":"a0e8e3db3dc680eb3218ed65730f7071c9f1ae82e33cac3cef1c13d1744e70da"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"确定辅助变量取值","source_units":[{"start":22178,"end":22554,"start_line":357,"end_line":358,"raw_tex":"c) Due to b), $a_{01}(\\bar x,\\bar y)$ consists of exactly $n-s$ elements. We conclude as in b) that there is at most one element in $a_{01}\\left(\\bar x,\\bar y\\right)\\backslash \\mathcal{E}(\\bar y)$. In view of statement a), $\\mathcal{E}(\\bar y)$ cannot consist of $n-s$ elements and, thus, must consist of $n-s-1$ elements. Hence, the statement follows immediately.\n\\end{proof}","sha256_utf8":"90eaa4a94fcf6fd80920f88e58e2c251b375962c58c60d46e4409fec9daf75b1"}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","title":"正则化辅助变量结构","source_units":[{"start":36348,"end":36971,"start_line":754,"end_line":763,"raw_tex":"\\begin{lemma}[Auxiliary $y$-variables in $\\mathcal{S}$]\n\\label{lem:EandH}\n%Suppose Assumption \\ref{ass:c} to be fulfilled.\nLet $(x, y)$ be a Karush-Kuhn-Tucker point of $\\mathcal{S}$. Then, it holds:\n\na) the summation inequality constraint is active, i.\\,e. $\\sum\\limits_{i=1}^n  y_i=n-s$,\n\nb) the index set $\\mathcal{E}( y) \\cup \\mathcal{H}(x, y)$ consists of at least $n-s-1$ elements,\nand the index set $\\mathcal{N}( y)$ consists of at most $s$ elements. Additionally, there is at most one index, that does not belong to any of these sets, i.\\,e. $\\left\\vert \\mathcal{O}\\left( x,  y\\right)\\right\\vert \\le 1$.\n\\end{lemma}","sha256_utf8":"ced5e7b0038957b7bbf316c10a1d1cadb3c0b83e998ef25ddf1ea8cc23707776"}]},{"id":"S027","semantic_type":"proof","kind":"proof","title":"正则化辅助变量引理证明","source_units":[{"start":36972,"end":39763,"start_line":764,"end_line":808,"raw_tex":"\\begin{proof}\na) Let $(x, y)$ be a Karush-Kuhn-Tucker point of $\\mathcal{S}$ and $\\sum\\limits_{i=1}^n  y_i>n-s$. Then, there exist multipliers $(\\lambda,  \\mu,  \\eta,  \\nu)$, such that (\\ref{eq:kkt-1})--(\\ref{eq:kkt-3}) are fulfilled. Since $ \\mu_3=0$, we have that the $(n+i)$-th row of (\\ref{eq:kkt-1}) reads as\n\\[\nc_i=\\left\\{\n\\begin{array}{ll}\n     - \\mu_{2,i},&\\mbox{for }i \\in \\mathcal{E}( y)\\backslash\\mathcal{H}( x, y),  \\\\\n    - \\mu_{2,i}+ \\eta^\\ge_i  x_i, &\\mbox{for }i \\in \\mathcal{E}( y) \\cap \\mathcal{H}^\\ge( x, y),\\\\\n    - \\mu_{2,i}- \\eta^\\le_i  x_i, &\\mbox{for }i \\in \\mathcal{E}( y) \\cap  \\mathcal{H}^\\le( x, y),\\\\\n \\eta^\\ge_i  x_i, &\\mbox{for }i \\in \\mathcal{H}^\\ge( x, y)\\backslash \\mathcal{E}( y),\\\\\n- \\eta^\\le_i  x_i, &\\mbox{for }i \\in \\mathcal{H}^\\le( x, y)\\backslash \\mathcal{E}( y),\\\\\n \\nu_i,&\\mbox{for }i \\in \\mathcal{N}( y),\\\\\n     0,&\\mbox{else}.\n\\end{array}\\right.\n\\]\nDue to (\\ref{eq:kkt-2}), (\\ref{eq:kkt-3}), and $c>0$, it must hold that $i\\in \\mathcal{N}( y)$ for all $i \\in \\left\\{1,\\ldots,n\\right\\}$. This, however, contradicts $\\sum\\limits_{i=1}^n  y_i>n-s$.\n\nb) \n%Since $( x, y)$ is a Karush-Kuhn-Tucker point of RCCOP$(c,\\varepsilon,t)$ there exist multipliers $( \\lambda,  \\mu,  \\eta^\\ge,  \\eta^\\le,  \\nu)$ such that (\\ref{eq:kkt-1})--(\\ref{eq:kkt-3}). \nAs in the proof of statement a), we conclude that $ \\mu_3>0$ for a Karush-Kuhn-Tucker point $( x, y)$ of $\\mathcal{S}$. Hence, the $(n+i)$-th row now reads as\n\\begin{equation}\n\\label{eq:kth-rowkkt}    \nc_i=\\left\\{\n\\begin{array}{ll}\n     - \\mu_{2,i}+ \\mu_3,&\\mbox{for }i \\in \\mathcal{E}( y)\\backslash \\mathcal{H}( x, y),  \\\\\n    - \\mu_{2,i}+ \\mu_3+ \\eta^\\ge_i  x_i, &\\mbox{for }i \\in \\mathcal{E}( y) \\cap \\mathcal{H}^\\ge( x, y),\\\\\n    - \\mu_{2,i}+ \\mu_3- \\eta^\\le_i  x_i, &\\mbox{for }i \\in \\mathcal{E}( y) \\cap  \\mathcal{H}^\\le( x, y),\\\\\n \\mu_3+ \\eta^\\ge_i  x_i, &\\mbox{for }i \\in \\mathcal{H}^\\ge( x, y)\\backslash \\mathcal{E}( y),\\\\\n \\mu_3- \\eta^\\le_i  x_i, &\\mbox{for }i \\in \\mathcal{H}^\\le( x, y)\\backslash \\mathcal{E}( y),\\\\\n \\mu_3+ \\nu_i,&\\mbox{for }i \\in \\mathcal{N}( y),\\\\\n      \\mu_3,&\\mbox{else}.\n\\end{array}\\right.\n\\end{equation}\nIt follows from (\\ref{eq:kth-rowkkt}) and  the components of $c$ being pairwise different that there can be at most one element ${\\bar i} \\in \\mathcal{O}\\left( x,  y\\right)$. \nIf $\\mathcal{E}( y) \\cup \\mathcal{H}( x,  y)$ consists of fewer than $n-s-1$ elements, we get:\n\\[\n\\sum\\limits_{i=1}^n  y_i\\le (n-s-2)\\cdot (1+\\varepsilon)+y_{\\bar i}<(n-s-1)\\cdot(1+\\varepsilon)<n-s,\n\\]\na contradiction.\nFinally, we assume that $\\mathcal{N}( y)$ consists of more than $s$ elements.\nIn this case, there are at most $n-s-1$ nonvanishing components of $y$. Consequently, \n\\[\n\\sum\\limits_{i=1}^n  y_i\\le (n-s-1)\\cdot (1+\\varepsilon)<n-s\n\\]\nprovides a contradiction.\n\\end{proof}","sha256_utf8":"00e66523fa6573e02cc116429b00d031e5cedf1dfc772fd4187f56f5f90fc73e"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"正则化求和约束活动","source_units":[{"start":36972,"end":38063,"start_line":764,"end_line":778,"raw_tex":"\\begin{proof}\na) Let $(x, y)$ be a Karush-Kuhn-Tucker point of $\\mathcal{S}$ and $\\sum\\limits_{i=1}^n  y_i>n-s$. Then, there exist multipliers $(\\lambda,  \\mu,  \\eta,  \\nu)$, such that (\\ref{eq:kkt-1})--(\\ref{eq:kkt-3}) are fulfilled. Since $ \\mu_3=0$, we have that the $(n+i)$-th row of (\\ref{eq:kkt-1}) reads as\n\\[\nc_i=\\left\\{\n\\begin{array}{ll}\n     - \\mu_{2,i},&\\mbox{for }i \\in \\mathcal{E}( y)\\backslash\\mathcal{H}( x, y),  \\\\\n    - \\mu_{2,i}+ \\eta^\\ge_i  x_i, &\\mbox{for }i \\in \\mathcal{E}( y) \\cap \\mathcal{H}^\\ge( x, y),\\\\\n    - \\mu_{2,i}- \\eta^\\le_i  x_i, &\\mbox{for }i \\in \\mathcal{E}( y) \\cap  \\mathcal{H}^\\le( x, y),\\\\\n \\eta^\\ge_i  x_i, &\\mbox{for }i \\in \\mathcal{H}^\\ge( x, y)\\backslash \\mathcal{E}( y),\\\\\n- \\eta^\\le_i  x_i, &\\mbox{for }i \\in \\mathcal{H}^\\le( x, y)\\backslash \\mathcal{E}( y),\\\\\n \\nu_i,&\\mbox{for }i \\in \\mathcal{N}( y),\\\\\n     0,&\\mbox{else}.\n\\end{array}\\right.\n\\]\nDue to (\\ref{eq:kkt-2}), (\\ref{eq:kkt-3}), and $c>0$, it must hold that $i\\in \\mathcal{N}( y)$ for all $i \\in \\left\\{1,\\ldots,n\\right\\}$. This, however, contradicts $\\sum\\limits_{i=1}^n  y_i>n-s$.\n","sha256_utf8":"031bc4d6b0c054d4d99ee9255b29a08659e9c85a6f74abd6095e90111d970167"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"逐行条件限制自由指标","source_units":[{"start":38064,"end":39275,"start_line":780,"end_line":796,"raw_tex":"b) \n%Since $( x, y)$ is a Karush-Kuhn-Tucker point of RCCOP$(c,\\varepsilon,t)$ there exist multipliers $( \\lambda,  \\mu,  \\eta^\\ge,  \\eta^\\le,  \\nu)$ such that (\\ref{eq:kkt-1})--(\\ref{eq:kkt-3}). \nAs in the proof of statement a), we conclude that $ \\mu_3>0$ for a Karush-Kuhn-Tucker point $( x, y)$ of $\\mathcal{S}$. Hence, the $(n+i)$-th row now reads as\n\\begin{equation}\n\\label{eq:kth-rowkkt}    \nc_i=\\left\\{\n\\begin{array}{ll}\n     - \\mu_{2,i}+ \\mu_3,&\\mbox{for }i \\in \\mathcal{E}( y)\\backslash \\mathcal{H}( x, y),  \\\\\n    - \\mu_{2,i}+ \\mu_3+ \\eta^\\ge_i  x_i, &\\mbox{for }i \\in \\mathcal{E}( y) \\cap \\mathcal{H}^\\ge( x, y),\\\\\n    - \\mu_{2,i}+ \\mu_3- \\eta^\\le_i  x_i, &\\mbox{for }i \\in \\mathcal{E}( y) \\cap  \\mathcal{H}^\\le( x, y),\\\\\n \\mu_3+ \\eta^\\ge_i  x_i, &\\mbox{for }i \\in \\mathcal{H}^\\ge( x, y)\\backslash \\mathcal{E}( y),\\\\\n \\mu_3- \\eta^\\le_i  x_i, &\\mbox{for }i \\in \\mathcal{H}^\\le( x, y)\\backslash \\mathcal{E}( y),\\\\\n \\mu_3+ \\nu_i,&\\mbox{for }i \\in \\mathcal{N}( y),\\\\\n      \\mu_3,&\\mbox{else}.\n\\end{array}\\right.\n\\end{equation}\nIt follows from (\\ref{eq:kth-rowkkt}) and  the components of $c$ being pairwise different that there can be at most one element ${\\bar i} \\in \\mathcal{O}\\left( x,  y\\right)$. \n","sha256_utf8":"5e35fce87e8f6c8c7190cafbbd144ff0516d34a7714663337ac1feab127f1366"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"活动集与零分量计数","source_units":[{"start":39275,"end":39763,"start_line":797,"end_line":808,"raw_tex":"If $\\mathcal{E}( y) \\cup \\mathcal{H}( x,  y)$ consists of fewer than $n-s-1$ elements, we get:\n\\[\n\\sum\\limits_{i=1}^n  y_i\\le (n-s-2)\\cdot (1+\\varepsilon)+y_{\\bar i}<(n-s-1)\\cdot(1+\\varepsilon)<n-s,\n\\]\na contradiction.\nFinally, we assume that $\\mathcal{N}( y)$ consists of more than $s$ elements.\nIn this case, there are at most $n-s-1$ nonvanishing components of $y$. Consequently, \n\\[\n\\sum\\limits_{i=1}^n  y_i\\le (n-s-1)\\cdot (1+\\varepsilon)<n-s\n\\]\nprovides a contradiction.\n\\end{proof}","sha256_utf8":"1db413052df511d9a949f2072a2804bf4e8cfae0d0f4cfa882c2bed258927f1f"}]},{"id":"S031","semantic_type":"lemma","kind":"lemma","title":"极限前后活动集关系","source_units":[{"start":44576,"end":45280,"start_line":922,"end_line":936,"raw_tex":"\\begin{lemma}[Active index sets]\n\\label{lem:QandE}\nSuppose a sequence of Karush-Kuhn-Tucker points $\\left(x^{t},y^{t}\\right)$ of $\\mathcal{S}(t)$ converges to $\\left(\\bar x,\\bar y\\right)$ for $t \\to 0$. Moreover, let $(\\bar x,\\bar y)$ be a nondegenerate T-stationary point of $\\mathcal{R}$. Then, for all sufficiently small $t$ it holds:\n\na) $Q_0\\left(\\bar x\\right)=Q_0\\left(x^{t}\\right)$,\n\nb) $\\mathcal{E}\\left(\\bar y\\right)=\\mathcal{E}\\left(y^{t}\\right) $,\n\nc) $a_{00}\\left(\\bar x, \\bar y\\right) \\subset \\mathcal{H}\\left(x^t,y^t\\right)$,\n\nd) $\\mathcal{N}\\left(y^t\\right)\\subset\na_{10}\\left(\\bar x,\\bar y\\right)\\subset \n\\mathcal{N}\\left(y^t\\right) \\cup \\mathcal{H}\\left(x^t,y^t\\right)$.    \n\n\\end{lemma}","sha256_utf8":"ec51ba6bf3b3cd696c2d6213ab537f012647f9499edcbb2c66cc299014e3abd0"}]},{"id":"S032","semantic_type":"proof","kind":"proof","title":"活动集关系引理证明","source_units":[{"start":45281,"end":50189,"start_line":937,"end_line":991,"raw_tex":"\\begin{proof}\na) We start by proving $Q_0\\left(\\bar x\\right)=Q_0\\left(x^{t}\\right)$.\nDue to continuity arguments, we have $Q_0\\left(x^{t}\\right)\\subset Q_0\\left(\\bar x\\right)$ for all sufficiently small $t$.\nLet us now assume that there exists\n$\\bar i \\in  Q_0\\left(\\bar x\\right)\\backslash Q_0\\left(x^{t}\\right)$ along a subsequence.\nHence, for the corresponding multipliers it holds $\\mu_{\\bar i}^t=0$.\nNDT1 allows us to apply Remark \\ref{rem:multipliers}, and we thus have $\\bar \\mu_{\\bar i}=\\lim\\limits_{t \\rightarrow \\infty} \\mu_{\\bar i}^t=0$, a contradiction to NDT2. Consequently, $Q_0\\left(\\bar x\\right)=Q_0\\left(x^{t}\\right)$ holds for all sufficiently small $t$.\n\nb) Next, we prove $\\mathcal{E}\\left(\\bar y\\right)=\\mathcal{E}\\left(y^{t}\\right)$.\nAgain, continuity arguments provide $\\mathcal{E}\\left(y^{t}\\right)\\subset \\mathcal{E}\\left(\\bar y\\right)$ for all sufficiently small $t$.\nSimilar to the first part of the proof, we now assume there exists $\\bar i \\in \\mathcal{E}\\left(\\bar y\\right)\\backslash \\mathcal{E}\\left(y^{t}\\right)$ along a subsequence. \nAs we have seen in Lemma \\ref{lem:a01}, T-stationarity of $(\\bar x, \\bar y)$ implies in particular $c_{\\bar i}=-\\bar \\mu_{2,\\bar i}+\\bar \\mu_{3,\\bar i}$. \nMoreover, NDT1 and Remark \\ref{rem:multipliers} provide \n$\\lim\\limits_{t \\to 0}\\mu_{3}^{t}=\\bar \\mu_{3}$.\nSince $\\bar i \\notin\\mathcal{N}\\left(y^{t}\\right)$, we distinguish the following cases:\n\n$(i)$ $\\bar i \\in\\mathcal{H}^{\\ge}\\left(x^{t},y^{t}\\right)\\backslash \\mathcal{E}\\left(y^{t}\\right)$. Karush-Kuhn-Tucker conditions for $\\left(x^t, y^t\\right)$ imply $c_{\\bar i}= \\mu^t_{3,\\bar i}+ \\eta^{\\ge,t}_{\\bar i}x_{\\bar i}$, cf. (\\ref{eq:kth-rowkkt}).\nIt follows $\n-\\bar \\mu_{2,\\bar i}+\\bar \\mu_{3,\\bar i}=\\mu^t_{3,\\bar i}+ \\eta^{\\ge,t}_{\\bar i}x_{\\bar i}$.\nBy taking the limit, we can cancel out $\\bar \\mu_{3,\\bar i}$ and $\\mu^t_{3,\\bar i}$.\nThis leads to a contradiction because the left-hand side of the equation is strictly negative due to NDT2 and the right-hand side is nonnegative since  $\\eta^{\\ge,t}_{\\bar i}$ is nonnegative and $x_{\\bar i}$ is positive.\n\n$(ii)$ $\\bar i \\in\\mathcal{H}^{\\le}\\left(x^{t},y^{t}\\right)\\backslash \\mathcal{E}\\left(y^{t}\\right)$.\nBy using (\\ref{eq:kth-rowkkt}), we get\n$c_{\\bar i}= \\mu^t_{3,\\bar i}- \\eta^{\\le,t}_{\\bar i}x_{\\bar i}$.\nThis leads to a contradiction just as in the previous case.\n\n$(iii)$ $\\bar i \\in\\mathcal{O}\\left(x^{t},y^{t}\\right)$.\nAnalogously, we obtain $c_{\\bar i}= \\mu^t_{3,\\bar i}$ from (\\ref{eq:kth-rowkkt}).\nIt follows\n$-\\bar \\mu_{2,\\bar i}+\\bar \\mu_{3,\\bar i}=\\mu^t_{3,\\bar i}$. \nTaking the limits leads to $\\bar \\mu_{2,\\bar i}=0$, a contradiction with NDT2.\n\n\\noindent\n Altogether, $\\mathcal{E}\\left(\\bar y\\right)\\backslash \\mathcal{E}\\left(y^{t}\\right)=\\emptyset$ for all sufficiently small $t$, and the assertion follows. \n\n c) Clearly, $a_{00}\\left(\\bar x, \\bar y\\right) \\cap \\mathcal{E}(y^t)=\\emptyset$ for sufficiently small $t$.\n %, hence, by definition $a_{00}\\left(\\bar x, \\bar y\\right) \\subset \\mathcal{H}\\left(x^t,y^t\\right)\\cup \\mathcal{N}\\left(y^t\\right)\\cup \\mathcal{O}\\left(x^t,y^t\\right)$.\nLet us assume  there exists an $\\bar i \\in a_{00}(\\bar x,\\bar y)\\cap\\mathcal{N}\\left(y^t\\right)$. Due to (\\ref{eq:kth-rowkkt}), we then have $c_{\\bar i}= \\mu_3^t + \\nu_{\\bar i}^t$,\nwhereas (\\ref{eq:k-throw}) provides\n$ c_{\\bar i}=\\bar \\mu_3 + \\bar \\varrho_{2,\\bar i}$.\nAccording to Remark \\ref{rem:multipliers}, we have \n$\\lim\\limits_{t \\to 0}\\mu_{3}^{t}=\\bar \\mu_3$. Consequently, it must hold\n$\n\\lim\\limits_{t \\to 0} \\nu_{\\bar i}^t=\\bar \\varrho_{2,\\bar i}$.\nThis, however, cannot be true since $\\nu_{\\bar i}^t\\geq 0$, while\n$\\bar \\varrho_{2,\\bar i}<0$ due to NDT3 from the nondegeneracy of $(\\bar x, \\bar y)$, a contradiction. \nLet us assume now that there exists an $\\bar i \\in a_{00}(\\bar x,\\bar y)\\cap \\mathcal{O}\\left(x^t,y^t\\right)$. Analogously, we get $\\bar \\varrho_{2,\\bar i}=0$, again a contradiction to NDT3. Overall, we get the assertion.\n\n d) Clearly, $a_{01}\\left(\\bar x,\\bar y\\right)\\cap \\mathcal{N}(y^t)  =\\emptyset$ for sufficiently small $t$. From c) we also know that $a_{00}\\left(\\bar x,\\bar y\\right)\\cap \\mathcal{N}(y^t)  =\\emptyset$. Altogether, the first inclusion of the assertion follows immediately. Further, it also holds\n$a_{10}\\left(\\bar x, \\bar y\\right) \\cap \\mathcal{E}(y^t)= \\emptyset$ for sufficiently small $t$.\nLet us assume there exists an  $\\bar i \\in a_{10}(\\bar x,\\bar y)\\cap\\mathcal{O}\\left(x^t,y^t\\right)$. Due to (\\ref{eq:kth-rowkkt}), we have $c_{\\bar i}=\\mu_3^t$.\nIn view of Lemma \\ref{lem:a01}c), there exists an index $\\tilde i \\in a_{01}(\\bar x,\\bar y)\\backslash \\mathcal{E}\\left(\\bar y\\right)$. Thus, T-stationarity of $(\\bar x, \\bar y)$ implies via (\\ref{eq:k-throw}) that $c_{\\tilde i}=\\bar \\mu_3$.\nBy taking the limit and Remark \\ref{rem:multipliers}, we obtain $c_{\\bar i}=c_{\\tilde i}$, but $\\bar i\\not=\\tilde i$, a contradiction to the choice of $c$.\n\\end{proof}","sha256_utf8":"9768288f3496f9451de22fbbe387d433dd88b31b92dc2a7a5957ba9d3a261942"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"普通不等式活动集稳定","source_units":[{"start":45281,"end":45953,"start_line":937,"end_line":943,"raw_tex":"\\begin{proof}\na) We start by proving $Q_0\\left(\\bar x\\right)=Q_0\\left(x^{t}\\right)$.\nDue to continuity arguments, we have $Q_0\\left(x^{t}\\right)\\subset Q_0\\left(\\bar x\\right)$ for all sufficiently small $t$.\nLet us now assume that there exists\n$\\bar i \\in  Q_0\\left(\\bar x\\right)\\backslash Q_0\\left(x^{t}\\right)$ along a subsequence.\nHence, for the corresponding multipliers it holds $\\mu_{\\bar i}^t=0$.\nNDT1 allows us to apply Remark \\ref{rem:multipliers}, and we thus have $\\bar \\mu_{\\bar i}=\\lim\\limits_{t \\rightarrow \\infty} \\mu_{\\bar i}^t=0$, a contradiction to NDT2. Consequently, $Q_0\\left(\\bar x\\right)=Q_0\\left(x^{t}\\right)$ holds for all sufficiently small $t$.\n","sha256_utf8":"49be6571711f44610acba96d3229e33667e8c9eef05573a53841da9d2612a409"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"上界活动集反证准备","source_units":[{"start":45954,"end":46696,"start_line":945,"end_line":951,"raw_tex":"b) Next, we prove $\\mathcal{E}\\left(\\bar y\\right)=\\mathcal{E}\\left(y^{t}\\right)$.\nAgain, continuity arguments provide $\\mathcal{E}\\left(y^{t}\\right)\\subset \\mathcal{E}\\left(\\bar y\\right)$ for all sufficiently small $t$.\nSimilar to the first part of the proof, we now assume there exists $\\bar i \\in \\mathcal{E}\\left(\\bar y\\right)\\backslash \\mathcal{E}\\left(y^{t}\\right)$ along a subsequence. \nAs we have seen in Lemma \\ref{lem:a01}, T-stationarity of $(\\bar x, \\bar y)$ implies in particular $c_{\\bar i}=-\\bar \\mu_{2,\\bar i}+\\bar \\mu_{3,\\bar i}$. \nMoreover, NDT1 and Remark \\ref{rem:multipliers} provide \n$\\lim\\limits_{t \\to 0}\\mu_{3}^{t}=\\bar \\mu_{3}$.\nSince $\\bar i \\notin\\mathcal{N}\\left(y^{t}\\right)$, we distinguish the following cases:\n","sha256_utf8":"75648d291170c0ac45f75f172132616fc16ab81a83cd5958372f08149c57e451"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"排除下侧乘积活动情形","source_units":[{"start":46697,"end":47366,"start_line":953,"end_line":957,"raw_tex":"$(i)$ $\\bar i \\in\\mathcal{H}^{\\ge}\\left(x^{t},y^{t}\\right)\\backslash \\mathcal{E}\\left(y^{t}\\right)$. Karush-Kuhn-Tucker conditions for $\\left(x^t, y^t\\right)$ imply $c_{\\bar i}= \\mu^t_{3,\\bar i}+ \\eta^{\\ge,t}_{\\bar i}x_{\\bar i}$, cf. (\\ref{eq:kth-rowkkt}).\nIt follows $\n-\\bar \\mu_{2,\\bar i}+\\bar \\mu_{3,\\bar i}=\\mu^t_{3,\\bar i}+ \\eta^{\\ge,t}_{\\bar i}x_{\\bar i}$.\nBy taking the limit, we can cancel out $\\bar \\mu_{3,\\bar i}$ and $\\mu^t_{3,\\bar i}$.\nThis leads to a contradiction because the left-hand side of the equation is strictly negative due to NDT2 and the right-hand side is nonnegative since  $\\eta^{\\ge,t}_{\\bar i}$ is nonnegative and $x_{\\bar i}$ is positive.\n","sha256_utf8":"f75a5c7edd039dd7360d5373c3ee2275dea124176da97f234f800a76b649e966"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"排除上侧乘积活动情形","source_units":[{"start":47367,"end":47633,"start_line":959,"end_line":962,"raw_tex":"$(ii)$ $\\bar i \\in\\mathcal{H}^{\\le}\\left(x^{t},y^{t}\\right)\\backslash \\mathcal{E}\\left(y^{t}\\right)$.\nBy using (\\ref{eq:kth-rowkkt}), we get\n$c_{\\bar i}= \\mu^t_{3,\\bar i}- \\eta^{\\le,t}_{\\bar i}x_{\\bar i}$.\nThis leads to a contradiction just as in the previous case.\n","sha256_utf8":"0321b2f5864cd373b924de5d7f317aae1dd31bd2120244f830ae2bcae48f443a"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"排除自由情形并确定上界集","source_units":[{"start":47634,"end":48092,"start_line":964,"end_line":971,"raw_tex":"$(iii)$ $\\bar i \\in\\mathcal{O}\\left(x^{t},y^{t}\\right)$.\nAnalogously, we obtain $c_{\\bar i}= \\mu^t_{3,\\bar i}$ from (\\ref{eq:kth-rowkkt}).\nIt follows\n$-\\bar \\mu_{2,\\bar i}+\\bar \\mu_{3,\\bar i}=\\mu^t_{3,\\bar i}$. \nTaking the limits leads to $\\bar \\mu_{2,\\bar i}=0$, a contradiction with NDT2.\n\n\\noindent\n Altogether, $\\mathcal{E}\\left(\\bar y\\right)\\backslash \\mathcal{E}\\left(y^{t}\\right)=\\emptyset$ for all sufficiently small $t$, and the assertion follows. \n","sha256_utf8":"1ec8cb6bff518198df3f0c688ad12b3dbd8d22eef59d9dee393ccf7a5f8026af"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"双活动指标保持乘积活动","source_units":[{"start":48093,"end":49224,"start_line":973,"end_line":984,"raw_tex":" c) Clearly, $a_{00}\\left(\\bar x, \\bar y\\right) \\cap \\mathcal{E}(y^t)=\\emptyset$ for sufficiently small $t$.\n %, hence, by definition $a_{00}\\left(\\bar x, \\bar y\\right) \\subset \\mathcal{H}\\left(x^t,y^t\\right)\\cup \\mathcal{N}\\left(y^t\\right)\\cup \\mathcal{O}\\left(x^t,y^t\\right)$.\nLet us assume  there exists an $\\bar i \\in a_{00}(\\bar x,\\bar y)\\cap\\mathcal{N}\\left(y^t\\right)$. Due to (\\ref{eq:kth-rowkkt}), we then have $c_{\\bar i}= \\mu_3^t + \\nu_{\\bar i}^t$,\nwhereas (\\ref{eq:k-throw}) provides\n$ c_{\\bar i}=\\bar \\mu_3 + \\bar \\varrho_{2,\\bar i}$.\nAccording to Remark \\ref{rem:multipliers}, we have \n$\\lim\\limits_{t \\to 0}\\mu_{3}^{t}=\\bar \\mu_3$. Consequently, it must hold\n$\n\\lim\\limits_{t \\to 0} \\nu_{\\bar i}^t=\\bar \\varrho_{2,\\bar i}$.\nThis, however, cannot be true since $\\nu_{\\bar i}^t\\geq 0$, while\n$\\bar \\varrho_{2,\\bar i}<0$ due to NDT3 from the nondegeneracy of $(\\bar x, \\bar y)$, a contradiction. \nLet us assume now that there exists an $\\bar i \\in a_{00}(\\bar x,\\bar y)\\cap \\mathcal{O}\\left(x^t,y^t\\right)$. Analogously, we get $\\bar \\varrho_{2,\\bar i}=0$, again a contradiction to NDT3. Overall, we get the assertion.\n","sha256_utf8":"e049760d8a7a4064b46b2cad7d942d1e0a2c43c1af3e6e20ab332681729b44d4"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"单活动指标与零分量关系","source_units":[{"start":49225,"end":50189,"start_line":986,"end_line":991,"raw_tex":" d) Clearly, $a_{01}\\left(\\bar x,\\bar y\\right)\\cap \\mathcal{N}(y^t)  =\\emptyset$ for sufficiently small $t$. From c) we also know that $a_{00}\\left(\\bar x,\\bar y\\right)\\cap \\mathcal{N}(y^t)  =\\emptyset$. Altogether, the first inclusion of the assertion follows immediately. Further, it also holds\n$a_{10}\\left(\\bar x, \\bar y\\right) \\cap \\mathcal{E}(y^t)= \\emptyset$ for sufficiently small $t$.\nLet us assume there exists an  $\\bar i \\in a_{10}(\\bar x,\\bar y)\\cap\\mathcal{O}\\left(x^t,y^t\\right)$. Due to (\\ref{eq:kth-rowkkt}), we have $c_{\\bar i}=\\mu_3^t$.\nIn view of Lemma \\ref{lem:a01}c), there exists an index $\\tilde i \\in a_{01}(\\bar x,\\bar y)\\backslash \\mathcal{E}\\left(\\bar y\\right)$. Thus, T-stationarity of $(\\bar x, \\bar y)$ implies via (\\ref{eq:k-throw}) that $c_{\\tilde i}=\\bar \\mu_3$.\nBy taking the limit and Remark \\ref{rem:multipliers}, we obtain $c_{\\bar i}=c_{\\tilde i}$, but $\\bar i\\not=\\tilde i$, a contradiction to the choice of $c$.\n\\end{proof}","sha256_utf8":"61a0f5a726e29ef41a004c09621280dd8b20d90e7f97bda192ce8726eeb5bd06"}]},{"id":"S040","semantic_type":"theorem","kind":"theorem","title":"正则化收敛的指标界与保持","source_units":[{"start":50309,"end":51038,"start_line":995,"end_line":1004,"raw_tex":"\\begin{theorem}[Convergence from $\\mathcal{S}$ to $\\mathcal{R}$ again]\n\\label{thm:kktsequence}\n%Let Assumption \\ref{ass:c} be fulfilled and $\\varepsilon \\le \\frac{1}{n-s}$. \nSuppose that a sequence of nondegenerate Karush-Kuhn-Tucker points $(x^{t},y^{t})$ of \n$\\mathcal{S}$ with quadratic index $m$ converges to $\\left(\\bar x,\\bar y\\right)$ for $t \\to 0$. If $(\\bar x,\\bar y)$ is a nondegenerate T-stationary point of $\\mathcal{R}$, then we have for its T-Index: \n\\[\n\\max\\left\\{m - \\left\\vert\\left\\{i\\in a_{01}\\left(\\bar x,\\bar y\\right)\\,\\left\\vert\\,\\bar \\sigma_{1,i}=0\\right.\\right\\} \\right\\vert, 0\\right\\}\\le TI \\leq m.\n\\]\nIf additionally NDT6 holds at $(\\bar x,\\bar y)$, then the indices coincide, i.e. $TI= m$.\n\\end{theorem}","sha256_utf8":"376fd63e5d763a395a796f325f7a17246030d2b5c7ce7e65334ee2be1ffd26a4"}]},{"id":"S041","semantic_type":"proof","kind":"proof","title":"指标收敛定理完整证明","source_units":[{"start":51039,"end":77534,"start_line":1005,"end_line":1501,"raw_tex":"\\begin{proof}\nThe proof will be divided into 4 major steps.\n\n{\\bf Step 1a.}\nWe rewrite the tangent space corresponding to the Karush-Kuhn-Tucker point $\\left(x^t,y^t\\right)$. For that, we use Lemma \\ref{lem:EandH}a) which provides that the summation constraint is active: \n\\[\n    \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}=\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\nDh_p( x^t),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q( x^t),0\\end{pmatrix}\\xi=0,q \\in Q_0( x^t),\\\\\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(y^t),\\\\\n\\begin{pmatrix}\ny_i^te_i,x_i^te_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{00}\\left(\\bar x, \\bar y\\right),\\\\\n\\begin{pmatrix}\ny_i^te_i,x_i^te_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x, \\bar y\\right),\\\\\n\\begin{pmatrix}\ny_i^te_i,x_i^te_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{10}\\left(\\bar x, \\bar y\\right),\\\\\n\\begin{pmatrix}\n    0,e_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{N}\\left(y^t\\right)\n\\end{array}\n\\right.\\right\\}.\n\\]\nIn total there are, due to LICQ,\n\\[\n\\begin{array}{rcl}\n\\alpha^\\mathcal{S}_t&=&\n\\left\\vert P\\right\\vert +\\left\\vert Q_0\\left(x^t\\right)\\right\\vert +1+\n\\left\\vert \\mathcal{E}\\left(y^t\\right)\\right\\vert +\n\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{00}\\left(\\bar x,\\bar y\\right)\\right\\vert \n\\\\\n&&\n+\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert \n+\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{10}\\left(\\bar x,\\bar y\\right)\\right\\vert \n+\\left\\vert \\mathcal{N}\\left(y^t\\right)\\right\\vert \n\\end{array}\n\\]\nlinearly independent vectors involved. We use Lemma \\ref{lem:QandE}a) and \\ref{lem:QandE}b) to\nsubstitute $\\left\\vert Q_0\\left(x^t\\right)\\right\\vert $ with $\\left\\vert Q_0\\left(\\bar x\\right)\\right\\vert $ and $\\left\\vert \\mathcal{E}\\left(y^t\\right)\\right\\vert $ with\n$\\left\\vert \\mathcal{E}\\left(\\bar y\\right)\\right\\vert $, respectively. The latter set has cardinality of  $n-s-1$ due to Lemma \\ref{lem:a01}c). Additionally, we use Lemma \\ref{lem:QandE}c) and \\ref{lem:QandE}d) to conclude:\n\\[\n\\begin{array}{rcl}\n\\alpha^\\mathcal{S}_{t}&=&\n\\left\\vert P\\right\\vert +\\left\\vert Q_0\\left(\\bar x\\right)\\right\\vert +1+\nn-s-1\n+\\left\\vert a_{00}\\left(\\bar x,\\bar y\\right)\\right\\vert \n\\\\\n&&\n+\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert \n+\\left\\vert a_{10}\\left(\\bar x,\\bar y\\right)\\right\\vert .\n\\end{array}\n\\]\nFinally, $\\left\\vert a_{00}\\left(\\bar x,\\bar y\\right)\\right\\vert \n+\\left\\vert a_{10}\\left(\\bar x,\\bar y\\right)\\right\\vert =s$, cf. Lemma \\ref{lem:a01}b).\nThus, we have:\n\\[\n\\begin{array}{rcl}\n\\alpha^\\mathcal{S}_{t}&=&\n\\left\\vert P\\right\\vert +\\left\\vert Q_0\\left(\\bar x\\right)\\right\\vert \n+\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert \n+n.\n\\end{array}\n\\]\n{\\bf Step 1b.} We examine the tangent space corresponding to the T-stationary point $(\\bar x,\\bar y)$. For this purpose, we consider the following vectors from its definition:\n\\[\n\\begin{pmatrix}\n0\\\\\ne_i\n\\end{pmatrix},i\\in \\mathcal{E}(\\bar y),\n\\begin{pmatrix}\n0\\\\\ne_i\n\\end{pmatrix}, i \\in a_{10}\\left(\\bar x, \\bar y\\right)\\cup a_{00}\\left(\\bar x, \\bar y\\right),\n\\begin{pmatrix}\n0\\\\\ne\n\\end{pmatrix}.\\]\nThe latter vector is involved due to Lemma \\ref{lem:a01}a). The number of these vectors is due to Lemma \\ref{lem:a01}c) equal to $(n-s-1)+\ns+1=n$. Moreover, they are linearly independent due to MPOC-LICQ.\nHence, we can write the respective tangent space as follows:\n\\[\n    \\mathcal{T}^\\mathcal{R}_{(\\bar x,\\bar y)}=\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\n\nDh_p(\\bar x), 0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(\\bar x), 0\\end{pmatrix}\\xi=0,q \\in Q_0(\\bar x),\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y) \\cup a_{01}(\\bar x,\\bar y),\n\\xi_{n+1}=\\ldots=\\xi_{2n}=0.\n\\end{array}\n\\right.\\right\\}\n\\]\nIn total there are, due to MPOC-LICQ,\n\\[\n\\alpha^\\mathcal{R}=\\left\\vert P\\right\\vert +\\left\\vert Q_0\\left(\\bar x\\right)\\right\\vert \n+\\left\\vert a_{00}(\\bar x,\\bar y)\\right\\vert +\\left\\vert a_{01}(\\bar x,\\bar y)\\right\\vert +n\n\\]\nlinearly independent vectors involved.\n\n{\\bf Step 2.}\nLet $\\mathcal{T}\\subset \\R^{2n}$ be a linear subspace. We denote the number of negative\neigenvalues of $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}}$ by $QI^\\mathcal{S}_{t,\\mathcal{T}}$. Analogously, $QI^\\mathcal{R}_{t,\\mathcal{T}}$ stands for the number of negative eigenvalues of $D^2 L^\\mathcal{R}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}}$ and $\\overline{QI}^\\mathcal{R}_{\\mathcal{T}}$ stands for the number of negative eigenvalues of $D^2 L^\\mathcal{R}\\left(\\bar x , \\bar y \\right)\\restriction_{\\mathcal{T}}$. \nWe have the following relation between the involved Hessians of the Lagrange functions by denoting $E(i)=e_ie_{n+i}^T+e_{n+i}e_i^T$, $i =1, \\ldots, n$:\n\\begin{equation}\n\\label{eq:Hessetandbar}\n\\begin{array}{rcl}\n\\displaystyle\nD^2 L^\\mathcal{S}\\left( x^t, y^t\\right) &=& \\displaystyle\nD^2 L^\\mathcal{R}\\left( x^t, y^t\\right) - \\sum\\limits_{i\\in \\mathcal{H}^\\ge\\left(x^t,y^t\\right)}  \\eta_i^{\\ge,t} E(i)\\displaystyle \n+\n\\sum\\limits_{i\\in \\mathcal{H}^\\le\\left(x^t,y^t\\right)} \\eta_i^{\\le,t} E(i).\n\\end{array}\n\\end{equation}\n\n{\\bf Step 2a.}\nIt holds for $t$ sufficiently small:\n\\[\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}=\\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}.\n\\]\n%Assume first $QI^\\mathcal{S}_{\\left(t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}>QI^\\mathcal{R}_{\\left(t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}$.\n%Then, there exists $\\bar \\xi \\in \\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}$, such that\n%\\[\n%\\bar \\xi^T D^2 L^\\mathcal{S}\\left( x^t, y^t\\right) \\bar \\xi < 0,\\quad\n%\\bar \\xi^T D^2 L^\\mathcal{R}\\left( x^t, y^t\\right)\\bar \\xi\\ge0.\n%\\]\nIndeed, by using (\\ref{eq:Hessetandbar}), we derive for any $\\xi \\in  \\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}$:\n\\begin{equation}\n    \\label{eq:s-r-h1}\n\\begin{array}{rcl}\n\\xi^T D^2 L^\\mathcal{S}\\left( x^t, y^t\\right) \\xi&=&\n%\\bar \\xi^T D^2 L^\\mathcal{R}\\left( x^t, y^t\\right)\\bar \\xi - \\bar \\xi^T \\left(\\sum\\limits_{i\\in \\mathcal{H}^\\ge\\left(x^t,y^t\\right)} \\bar \\eta_i^{t,\\ge} E(i)\\right)\\bar \\xi +\n%\\bar \\xi^T \\left(\\sum\\limits_{i\\in \\mathcal{H}^\\le\\left(x^t,y^t\\right)} \\bar \\eta_i^{t,\\le} E(i)\\right)\\bar \\xi\\\\\n%&=&\n \\xi^T D^2 L^\\mathcal{R}\\left(x^t, y^t\\right) \\xi - \\displaystyle \\sum\\limits_{i\\in \\mathcal{H}^\\ge\\left(x^t,y^t\\right)} 2\\eta_i^{\\ge,t} \\xi_i  \\xi_{n+i} \\\\\n&&\\displaystyle +\n\\sum\\limits_{i\\in \\mathcal{H}^\\le\\left(x^t,y^t\\right)} 2\\eta_i^{\\le,t}  \\xi_i \\xi_{n+i}=\n \\xi^T D^2 L^\\mathcal{R}\\left( x^t, y^t\\right)  \\xi,\n\\end{array}\n\\end{equation}\nsince $\\xi_{n+1}=\\ldots=\\xi_{2n}=0$ as seen in Step 1b.\n%Assuming  $QI^\\mathcal{S}_{\\left(t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}<QI^\\mathcal{R}_{\\left(t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}$ instead will analogously lead to a contradiction.\nHence, we get\n $\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}=QI^\\mathcal{R}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$.\nDue to NDT4, continuity arguments provide\n$\nQI^\\mathcal{R}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}=\\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$.\n\n{\\bf Step 2b.}\nWe claim that the numbers of positive and negative eigenvalues of $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$ and of $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\prime}$, respectively, coincide, where\n\\[\n  \\mathcal{T}^{\\prime} = \\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\nDh_p(x^t),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(x^t),0\\end{pmatrix}\\xi=0,q \\in Q_0(\\bar x),\\\\\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(\\bar y),\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y),\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y),\\\\\n\\begin{pmatrix}\ny^t_ie_i,x^t_ie_i\n\\end{pmatrix}\\xi=0, i \\in a_{01}(\\bar x,\\bar y)\\cup a_{10}(\\bar x,\\bar y)\n\\end{array}\n\\right.\\right\\}.\n\\]\nLet $\\left\\{\\lambda^+_1,\\ldots,\\lambda^+_{k^+}\\right\\}$ be the positive \neigenvalues of \n$D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$ with corresponding eigenvectors \n$\\left\\{\\xi^+_1,\\ldots,\\xi^+_{k^+}\\right\\}$.\nHence, for all $k=1,\\ldots, k^+$:\n\\[\n{\\xi^+_k}^T D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\xi^+_k>0.\n\\]\nWe rewrite the tangent space as follows:\n\\[\n    \\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}=\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\n\nDh_p(\\bar x),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(\\bar x),0\\end{pmatrix}\\xi=0,q \\in Q_0(\\bar x),\\\\\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(\\bar y),\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y),\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y),\\\\\n\\begin{pmatrix}\n\\bar y_ie_i,\\bar x_ie_i\n\\end{pmatrix}\\xi=0, i \\in a_{01}(\\bar x,\\bar y)\\cup a_{10}(\\bar x,\\bar y)\n\\end{array}\n\\right.\\right\\}.\n\\]\nDue to MPOC-LICQ, the application of the implicit function theorem provides the existence of\n$\\delta_2,\\delta_3>0$ such that for all $k=1,\\ldots,k^+$ and $t<\\delta_2$ there exists $\\xi_{k,t}$ with $\\left\\| \\xi_{k,t}-\\xi^+_k\\right\\|<\\delta_3$ and $\\xi_{k,t}\\in \\mathcal{T}^\\prime$.\n We can choose $t$ even smaller, such that $\\xi_{1,t},\\ldots,\\xi_{k^+,t}$ remain linearly independent and for all $k=1,\\ldots,k^+$ it holds:\n\\[\n{ \\xi_{k,t}}^T D^2 L^\\mathcal{S}\\left( x^t, y^t\\right) \\xi_{k,t}>0.\n\\]\n%whenever $\\NNorm{\\xi_k-\\xi^+_k}{}$ $<\\delta_1$, $k=1, \\ldots, k^+$.\n%\n%Altogether, for $t$ small enough there exist a set of linearly independent vectors \n%$\\left\\{\\tilde \\xi^+_1,\\ldots,\\tilde \\xi^+_{k^+}\\right\\}$ with\n%$\\tilde \\xi^+_k \\in \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$, for all $k=1,\\ldots,k^+$,\n%and\n%\\[\n%{\\tilde \\xi_k^+}{^T} D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\tilde \\xi^+_k>0.\n%\\]\nHence, $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\prime}$ has at least $k^+$ positive eigenvalues. If we repeat the above reasoning for negative eigenvalues, the matrix $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\prime}$ has at least as many negative eigenvalues as $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$. Additionally, we show that the dimensions of $\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}$ and $\\mathcal{T}^\\prime$ coincide. By Step 1b, we have $2n-\\alpha^\\mathcal{R}$ for the dimension of $\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}$. Since MPOC-LICQ remains valid in the neighborhood of $(\\bar x, \\bar y)$, we get again \n$2n-\\alpha^{\\mathcal{R}}$ for the dimension of $\\mathcal{T}^\\prime$. By continuity arguments, NDT4 and (\\ref{eq:s-r-h1}) provide that $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$ is nonsingular. Altogether, the assertion follows.\n\n{\\bf Step 2c.} \nWe claim that \n\\[\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}}\\le QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}+\\alpha^\\mathcal{R}-\\alpha^\\mathcal{S}_t.\n\\]\nFor that, we focus on the dimension of $\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$. As a consequence of Step 1a it is $2n-\\alpha^\\mathcal{S}_t$. \nDue to continuity arguments, we can choose $t$ small enough to ensure \n$x^t_{i}\\ne 0$, $i\\in a_{10}(\\bar x,\\bar y)$ and\n$y^t_{i}\\ne 0$, $i\\in a_{01}(\\bar x,\\bar y)$.\nUsing this and Lemma \\ref{lem:QandE}a), \\ref{lem:QandE}b), and \\ref{lem:QandE}d), it follows\nthat $\\mathcal{T}^\\prime \\subset \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$.\nTherefore, using Step 2b,  \n$\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}}\\le 2n-\\alpha^\\mathcal{S}_t-k^+$.\nWe observe in view of NDT4 and Step 1b that \n$\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}%\\overset{Step\\,\\,2a}{=}QI^\\mathcal{R}_{\\left(t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}%=\\overline{QI}^\\mathcal{R}_{\\left(\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}\n=2n-\\alpha^\\mathcal{R}-k^+$.\nThe assertion follows immediately.\n\n{\\bf Step 3.} Let us show that\n\\[\n\\max\\left\\{m - \\left\\vert \\left\\{i\\in a_{01}\\left(\\bar x,\\bar y\\right)\\,\\left\\vert\\,\\bar \\sigma_{1,i}=0\\right.\\right\\} \\right\\vert , 0\\right\\}\\le TI.\n\\]\nIn view of Step 2a, Step 2c and due to continuity, we have for $t$ sufficiently small:\n\n\\[\n\\begin{array}{rcl}\nm=QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}}&\\overset{Step\\,2c}{\\le}& QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}+\\alpha^\\mathcal{R}-\\alpha^\\mathcal{S}_t \\overset{Step\\,2a}{=} \n\\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}\n+\\alpha^\\mathcal{R}-\\alpha^\\mathcal{S}_t\\\\\n&\\overset{Step\\,1}{=}& \\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}\n+\\left\\vert a_{00}(\\bar x,\\bar y)\\right\\vert +\\left\\vert a_{01}(\\bar x,\\bar y)\\right\\vert -\n\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert \\\\\n&=&TI+\\left\\vert a_{01}(\\bar x,\\bar y)\\right\\vert -\n\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert .\n\\end{array}\n\\]\nWe show for $t$ sufficiently small:\n\\[\\left\\vert a_{01}(\\bar x,\\bar y)\\right\\vert -\n\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert \\le\n\\left\\vert \\left\\{i\\in a_{01}\\left(\\bar x,\\bar y\\right)\\,\\left\\vert\\,\\bar \\sigma_{1,i}=0\\right.\\right\\} \\right\\vert ,\n\\]\nand the assertion will follow immediately since $TI\\ge 0$.\nClearly,\n\\[\\left\\vert a_{01}(\\bar x,\\bar y)\\right\\vert -\n\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert =\n\\left\\vert a_{01}(\\bar x,\\bar y)\\backslash \\mathcal{H}\\left(x^t,y^t\\right) \\right\\vert .\n\\]\nSuppose $\\bar i \\in a_{01}\\left(\\bar x,\\bar y\\right)$ with $\\bar \\sigma_{1,\\bar i}\\ne 0$. In view of Remark \\ref{rem:multipliers}, the difference $\\eta^{\\ge,t}_{\\bar i}-\\eta^{\\le,t}_{\\bar i}$ cannot vanish for all $t$ sufficiently small. In particular, one of the multipliers $\\eta^{\\ge,t}_{\\bar i}$ or $\\eta^{\\le,t}_{\\bar i}$ has to be not vanishing for all $t$ sufficiently small. Hence, $\\bar i\\in \\mathcal{H}\\left(x^t,y^t\\right)$. We therefore have: \n\\[\na_{01}(\\bar x,\\bar y)\\backslash \\mathcal{H}\\left(x^t,y^t\\right) \\subset  \\left\\{i\\in a_{01}\\left(\\bar x,\\bar y\\right)\\,\\left\\vert\\,\\bar \\sigma_{1,i}= 0\\right.\\right\\}.\n\\]\n\n\n{\\bf Step 4.}\nWithout loss of generality -- considering subsequences if needed -- we can assume that for any $i\\in a_{00}(\\bar x,\\bar y)$ at least one of the sequences $\\frac{x_i^t}{y_i^t}$ or $\\frac{y_i^t}{x_i^t}$ is convergent. First, we note that the quotients are well defined due to Lemma \\ref{lem:QandE}c). Moreover, if the former sequence does not contain a convergent subsequence, we find a subsequence that tends to plus or minus infinity. Consequently, the corresponding subsequence of the latter reciprocal sequence has to converge to zero.\nWe define the following auxiliary sets:\n\\[\na_{00}^x(\\bar x,\\bar y)=\\left\\{i \\in a_{00}(\\bar x,\\bar y)\\,\\left\\vert\\, \\frac{x^t_i}{y^t_i}\\mbox{ converges for } t \\to 0 \\right. \\right\\},\na_{00}^y(\\bar x,\\bar y)=a_{00}(\\bar x,\\bar y)\\backslash a_{00}^x(\\bar x,\\bar y).\n\\]\n\nFor $\\bar i \\in a_{00}^x\\left(\\bar x,\\bar y\\right)$ we consider $\\mathcal{T}^{\\mathcal{R}}_{\\left(\\bar x,\\bar y\\right)}$ and replace two of the involved equations, namely \n$\\begin{pmatrix}\ne_{\\bar i},0\n\\end{pmatrix}\\xi=0$ and \n$\\begin{pmatrix}\n0,e_{\\bar i}\n\\end{pmatrix}\\xi=0$ by one equation\n$\\begin{pmatrix}\ne_{\\bar i},\\lim\\limits_{t \\to 0}\\frac{x_{\\bar i}^t}{y_{\\bar i}^t}e_{\\bar i}\n\\end{pmatrix}\\xi=0$.\nClearly, the vectors involved in the definition of the newly generated linear space, i.\\,e.\n\\[\n    \\mathcal{T}^{{\\bar i}}=\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\n\nDh_p(\\bar x),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(\\bar x),0\\end{pmatrix}\\xi=0,q \\in Q_0(\\bar x),\\\\\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(\\bar y),\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y)\\backslash\\{\\bar i\\},\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y)\\backslash\\{\\bar i\\},\\\\\n\\begin{pmatrix}\ne_{\\bar i},\\lim\\limits_{t \\to 0}\\frac{x_{\\bar i}^t}{y_{\\bar i}^t}e_{\\bar i}\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n\\bar y_ie_i,\\bar x_ie_i\n\\end{pmatrix}\\xi=0, i \\in a_{01}(\\bar x,\\bar y)\\cup a_{10}(\\bar x,\\bar y)\\}\n\\end{array}\n\\right.\\right\\},\n\\]\nremain linearly independent.\nThe dimension of $ \\mathcal{T}^{{\\bar i}}$ is greater than the dimension of\n$\\mathcal{T}^{\\mathcal{R}}_{\\left(\\bar x,\\bar y\\right)}$ by one. Moreover, there exists\n$\\xi^{\\bar i} \\in \\mathcal{T}^{{\\bar i}}$ with \n$\n%\\xi^{\\bar i}_{n+i} =0, i \\in a_{00}(\\bar x, \\bar y)\\backslash\\{\\bar i\\}, \\quad \n\\xi^{\\bar i}_{n+\\bar i} \\not =0$.\nIndeed, assume that no such $\\xi^{\\bar i}$ exists, then we can add the equation \n$\\begin{pmatrix}\n0,e_{\\bar i}\n\\end{pmatrix}\\xi=0$ to the defining equations of $ \\mathcal{T}^{ {\\bar i}}$ wihout changing it. The resulting space, however, is identical to\n$\\mathcal{T}^{\\mathcal{R}}_{\\left(\\bar x,\\bar y\\right)}$, a contradiction.\nWithout loss of generality, we assume \n$\\xi^{\\bar i}_{n+\\bar i}=1$.\nFurther, by straightforward application of the implicit function theorem and due to Lemma \\ref{lem:QandE}a) and \\ref{lem:QandE}b), we find a sequence of vectors $\\xi^{\\bar i}_t \\in \\mathcal{T}^{ \\bar i}_{t}$ that converges to $\\xi^{\\bar i}$ for $t \\to 0$. \nFor this, we define\n\\[\n\\mathcal{T}^{\\bar i}_t=\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\nDh_p(x^t),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(x^t),0\\end{pmatrix}\\xi=0,q \\in Q_0(x^t),\\\\\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(y^t),\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y)\\backslash\\{\\bar i\\},\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y)\\backslash\\{\\bar i\\},\\\\\n\\begin{pmatrix}\ne_{\\bar i},\\frac{x_{\\bar i}^t}{y_{\\bar i}^t} e_{\\bar i}\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\ny^t_ie_i,x^t_ie_i\n\\end{pmatrix}\\xi=0, i \\in a_{01}(\\bar x,\\bar y)\\cup a_{10}(\\bar x,\\bar y)\n\\end{array}\n\\right.\\right\\}.\n\\]\nWe again have, due to continuity arguments, that \n$\\xi^{\\bar i}_{t,n+\\bar i} \\not =0$.\n%\\left\\{\n%\\begin{array}{ll}\n%=0& \\mbox{for }i\\ne \\bar i,\\\\\n%\\ne 0& \\mbox{for } i=\\bar i.\n%\\end{array}\\right.\n%\\]\nFor $\\bar i \\in a_{00}^y\\left(\\bar x,\\bar y\\right)$ we proceed\n%we construct $\\xi^{\\bar i}$ and a sequence of vectors $\\xi^{\\bar i}_t \\in \\mathcal{T}^{ \\bar i}_{t}$ that converges to $\\xi^{\\bar i}$ for $t \\to 0$, \nanalogously by\nconsidering $\\mathcal{T}^{\\mathcal{R}}_{\\left(\\bar x,\\bar y\\right)}$ again and replace two of the  involved equations \n$\\begin{pmatrix}\ne_{\\bar i},0\n\\end{pmatrix}\\xi=0$ and \n$\\begin{pmatrix}\n0,e_{\\bar i}\n\\end{pmatrix}\\xi=0$ by the equation\n$\\begin{pmatrix}\n\\lim\\limits_{t \\to 0}\\frac{y_{\\bar i}^t}{x_{\\bar i}^t} e_{\\bar i},e_{\\bar i}\n\\end{pmatrix}\\xi=0$. \nBy the same arguments as before, we find\n$\\xi^{\\bar i} \\in \\mathcal{T}^{\\bar i}$ with\n$\\xi^{\\bar i}_{\\bar i} \\ne 0$. Again we will assume $\\xi^{\\bar i}_{\\bar i} = 1$ and find a sequence of vectors $\\xi^{\\bar i}_t \\in \\mathcal{T}^{ \\bar i}_{t}$ that converges to $\\xi^{\\bar i}$ for $t \\to 0$. Due to continuity, it holds then $\\xi^{\\bar i}_{t,\\bar i} \\ne 0$.\n\n\nIt is straightforward to verify the following observations for $t$ sufficiently small:\n\n    a) Let $\\left\\{\\xi^{\\prime,1},\\ldots,\\xi^{\\prime,\\ell}\\right\\}$ be a base of $\\mathcal{T}^{\\prime}$, cf. Step 2b, then\n    $\\left\\{\\xi^{\\prime,1},\\ldots,\\xi^{\\prime,\\ell}\\right\\}\\cup \\left\\{\\xi^{\\bar i}_{t}\\,\\left\\vert\\,\\bar i \\in a_{00}(\\bar x,\\bar y)\\right.\\right\\}$\nis a set of linear independent vectors.\nIn fact, suppose for some coefficients $b_{i}, \\in \\R$, $i \\in a_{00}\\left(\\bar x,\\bar y\\right),\\beta_i \\in \\R, i=1,\\ldots,\\ell$ it holds:\n\\[\n\\sum\\limits_{i \\in a_{00}\\left(\\bar x,\\bar y\\right)} b_{i}\\xi^{i}_{t}\n+\\sum\\limits_{i=1}^{\\ell}\\beta_i\\xi^{\\prime,i}=0.\n\\]\nFor $\\bar i \\in a_{00}^x\\left(\\bar x,\\bar y\\right)$ we consider the $(n+\\bar i)$-th row of this sum\n\\[\nb_{\\bar i}\\underbrace{\\xi^{\\bar i}_{t,n+\\bar i}}_{\\ne 0}+\\sum\\limits_{i \\in a_{00}\\left(\\bar x,\\bar y\\right)\\backslash\\left\\{\\bar i\\right\\}} b_{i}\\underbrace{\\xi^{i}_{t,n+\\bar i}}_{=0}+\\sum\\limits_{i=1}^{\\ell}\\beta_i\\underbrace{\\xi^{\\prime,i}_{n+\\bar i}}_{=0}=0.\n\\]\nIf instead $\\bar i \\in a_{00}^y\\left(\\bar x,\\bar y\\right)$ we consider the $\\bar i$-th row of the sum\n\\[\nb_{\\bar i}\\underbrace{\\xi^{\\bar i}_{t,\\bar i}}_{\\ne 0}+\\sum\\limits_{i \\in a_{00}\\left(\\bar x,\\bar y\\right)\\backslash\\left\\{\\bar i\\right\\}} b_{i}\\underbrace{\\xi^{i}_{t,\\bar i}}_{=0}+\\sum\\limits_{i=1}^{\\ell}\\beta_i\\underbrace{\\xi^{\\prime,i}_{\\bar i}}_{=0}=0.\n\\]\nAltogether, it must hold $b_{i} =0$, $i \\in a_{00}\\left(\\bar x,\\bar y\\right)$. However, this implies\n\\[\n\\sum\\limits_{i=1}^{\\ell}\\beta_i\\xi^{\\prime,i}=0.\n\\]\nHence, $\\beta_i=0$ for $i=1,\\ldots,\\ell$.\n\n    b) It holds $\\xi^{\\bar i}_{t} \\in \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$, cf. Step 1a, for any $\\bar i \\in a_{00}(\\bar x,\\bar y)$. \n    \n    c) It holds $\\left(\\eta_i^{\\le,t}-\\eta_i^{\\ge,t}\\right)\\xi^{\\bar i}_{t,i}\\xi^{\\bar i}_{t,n+i} \\leq 0, i \\in \\mathcal{H}\\left(x^t,y^t\\right)$. Since $\\xi^{\\bar i}_{t} \\in \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$, we obtain:\n    \\[\n\\begin{array}{rcl}\n\\left(\\eta_i^{\\le,t}-\\eta_i^{\\ge,t}\\right)\\xi^{\\bar i}_{t,i}\\xi^{\\bar i}_{t,n+i}\n&=&\\left(\\eta_i^{\\ge,t}-\\eta_i^{\\le,t}\\right)\\left(\\xi^{\\bar i}_{t,n+i}\\right)^2\\frac{x^t_i}{y^t_i}.\n\\end{array}\n\\]\nIf $i\\in \\mathcal{H}^{\\ge}\\left(x^t,y^t\\right)$,\nwe have $x_i^t<0,y_i^t>0$ and $\\eta_i^{\\le,t}=0$. Moreover, due to ND2, we have\n$\\eta_i^{\\ge,t}>0$. The assertion follows immediately. The other case $i\\in \\mathcal{H}^{\\le}\\left(x^t,y^t\\right)$ is completely analogous.\n\n    d) It holds $\\lim\\limits_{t\\to 0}\\left(\\eta_{\\bar i}^{\\le,t}-\\eta_{\\bar i}^{\\ge,t}\\right)\\xi^{\\bar i}_{t,\\bar i}\\xi^{\\bar i}_{t,n+\\bar i}=-\\infty$. \n    We calculate:\n\\[\n\\begin{array}{rcl}\n\\left(\\eta_{\\bar i}^{\\le,t}-\\eta_{\\bar i}^{\\ge,t}\\right)\\xi^{\\bar i}_{t,{\\bar i}}\\xi^{\\bar i}_{t,n+{\\bar i}}\n&=&\\left\\{\n\\begin{array}{ll}\n   \\left(\\eta_{\\bar i}^{\\ge,t}-\\eta_{\\bar i}^{\\le,t}\\right)\\left(\\xi^{\\bar i}_{t,n+\\bar i}\\right)^2\\frac{x^t_{\\bar i}}{y^t_{\\bar i}}& \\mbox{for } \\bar i \\in a_{00}^x(\\bar x,\\bar y),\\\\\n   \\left(\\eta_{\\bar i}^{\\ge,t}-\\eta_{\\bar i}^{\\le,t}\\right)\\left(\\xi^{\\bar i}_{t,\\bar i}\\right)^2\\frac{y^t_{\\bar i}}{x^t_{\\bar i}}& \\mbox{for } \\bar i \\in a_{00}^y(\\bar x,\\bar y).\n\\end{array}\\right.\n\\end{array}\n\\]\nLet us suppose $\\bar i \\in a_{00}^x(\\bar x,\\bar y)$. We have $y^t_{\\bar i}\\ne 0$ and, thus, $\\nu^t_{\\bar i}=0$. We use Remark \\ref{rem:multipliers} and NDT3 to conclude that the sequence\n$ \\left(\\eta_{\\bar i}^{\\ge,t}-\\eta_{\\bar i}^{\\le,t}\\right)x^t_{\\bar i}$ converges to $\\varrho_{2,\\bar i}<0$ for $t \\to 0$. Further,\n$\\frac{1}{y^t_{\\bar i}}>0$ tends to infinity for $t \\to 0$. Finally,\n$\\left(\\xi^{\\bar i}_{t,n+\\bar i}\\right)^2$ converges to $1$ for $t \\to 0$, due to the construction of\n$\\xi^{\\bar i}_t$.\nThus, the assertion follows.\nInstead, let us suppose $\\bar i \\in a_{00}^y(\\bar x,\\bar y)$. \nThis time, we have that\n$\\left(\\xi^{\\bar i}_{t,\\bar i}\\right)^2$ converges to $1$ for $t \\to 0$.\nDue to Remark \\ref{rem:multipliers} and NDT3, $\\left(\\eta_{\\bar i}^{\\ge,t}-\\eta_{\\bar i}^{\\le,t}\\right)y^t_{\\bar i}$ converges to $\\varrho_{1,\\bar i}\\ne 0$ for $t \\to 0$.\nIf $\\bar i\\in \\mathcal{H}^{\\ge}\\left(x^t,y^t\\right)$, then $\\varrho_{1,\\bar i}$ is positive from here. Also, $\\frac{1}{x^t_{\\bar i}}<0$ tends to minus infinity for $t \\to 0$. The other case $\\bar i\\in \\mathcal{H}^{\\le}\\left(x^t,y^t\\right)$ is completely analogous. \n\ne) We notice that $\\displaystyle \\xi^{\\bar i ^T}_{t} D^2 L^\\mathcal{R}\\left( x^t, y^t\\right)\\xi^{\\bar i}_{t}$ converges for $t \\to 0$ due to the construction above.\n\nFinally, for $\\bar i\\in  a_{00}(\\bar x,\\bar y)$ we estimate:\n\\[\n\\begin{array}{rcl}\n\\xi_t^{\\bar i ^T} D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\xi_t^{\\bar i}&\\overset{(\\ref{eq:Hessetandbar})}{=}&\n\\displaystyle \\xi^{\\bar i ^T}_{t} D^2 L^\\mathcal{R}\\left(x^t, y^t\\right)\\xi^{\\bar i}_{t}\\\\ && \\displaystyle\n- \\sum\\limits_{i\\in \\mathcal{H}^\\ge\\left(x^t,y^t\\right)} 2 \\eta_i^{\\ge,t} \\xi_{t,i}^{\\bar i}  \\xi_{t,n+i}^{\\bar i}+\n\\sum\\limits_{i\\in \\mathcal{H}^\\le\\left(x^t,y^t\\right)} 2 \\eta_i^{\\le,t}\\xi_{t,i}^{\\bar i}\\xi_{t,n+i}^{\\bar i} \\\\\n&= &\n\\displaystyle \\xi^{\\bar i ^T}_{t} D^2 L^\\mathcal{R}\\left(x^t, y^t\\right)\\xi^{\\bar i}_{t}\n +2\\sum\\limits_{i\\in \\mathcal{H}\\left(x^t,y^t\\right)} \\left( \\eta_i^{\\le,t}-\\eta_i^{\\ge,t} \\right)\\xi_{t,i}^{\\bar i}  \\xi_{t,n+i}^{\\bar i}\\\\ \n&\\overset{c)}{\\le}&\\displaystyle \\xi^{\\bar i ^T}_{t} D^2 L^\\mathcal{R}\\left(x^t, y^t\\right)\\xi^{\\bar i}_{t}+2\\left(\\eta_{\\bar i}^{\\le,t}-\\eta_{\\bar i}^{\\ge,t}\\right)\\xi^{\\bar i}_{t,\\bar i}\\xi^{\\bar i}_{t,n+\\bar i}.\n\\end{array}\n\\]\nThus, due to d) and e), $\\xi^{\\bar i ^T}_{t} D^2 L^\\mathcal{S}\\left(x^t, y^t\\right) \\xi^{\\bar i}_{t}$ has to be negative for $t$ small enough. \nAs we have seen in Step 2c, it holds $\\mathcal{T}^\\prime \\subset \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$. Then, due a) and b), we have therefore:\n\\[\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}}\n\\geq QI^\\mathcal{S}_{t,\\mathcal{T}^\\prime} + \\left\\vert a_{00}(\\bar x,\\bar y)\\right\\vert.\n\\]\nBy Step 2a, we have $QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}=\\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$, and by Step 2b, $QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}=QI^\\mathcal{S}_{t,\\mathcal{T}^\\prime}$. Overall, we obtain:\n\\[\nm=QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}}\n\\ge\n\\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}+\\left\\vert a_{00}(\\bar x,\\bar y)\\right\\vert=TI.\n\\]\n\\end{proof}","sha256_utf8":"b2e74bacf727dd3e44c229f85e40663937c640a334e64802caad8b8230d74877"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"安排四步证明","source_units":[{"start":51039,"end":51099,"start_line":1005,"end_line":1006,"raw_tex":"\\begin{proof}\nThe proof will be divided into 4 major steps.\n","sha256_utf8":"d6faf75d2826f25ffcefc429f6962274f1c3de269726957085b3268c1c56484f"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"正则化切空间与约束计数","source_units":[{"start":51100,"end":52717,"start_line":1008,"end_line":1050,"raw_tex":"{\\bf Step 1a.}\nWe rewrite the tangent space corresponding to the Karush-Kuhn-Tucker point $\\left(x^t,y^t\\right)$. For that, we use Lemma \\ref{lem:EandH}a) which provides that the summation constraint is active: \n\\[\n    \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}=\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\nDh_p( x^t),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q( x^t),0\\end{pmatrix}\\xi=0,q \\in Q_0( x^t),\\\\\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(y^t),\\\\\n\\begin{pmatrix}\ny_i^te_i,x_i^te_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{00}\\left(\\bar x, \\bar y\\right),\\\\\n\\begin{pmatrix}\ny_i^te_i,x_i^te_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x, \\bar y\\right),\\\\\n\\begin{pmatrix}\ny_i^te_i,x_i^te_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{10}\\left(\\bar x, \\bar y\\right),\\\\\n\\begin{pmatrix}\n    0,e_i\n\\end{pmatrix}\\xi=0, i \\in \\mathcal{N}\\left(y^t\\right)\n\\end{array}\n\\right.\\right\\}.\n\\]\nIn total there are, due to LICQ,\n\\[\n\\begin{array}{rcl}\n\\alpha^\\mathcal{S}_t&=&\n\\left\\vert P\\right\\vert +\\left\\vert Q_0\\left(x^t\\right)\\right\\vert +1+\n\\left\\vert \\mathcal{E}\\left(y^t\\right)\\right\\vert +\n\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{00}\\left(\\bar x,\\bar y\\right)\\right\\vert \n\\\\\n&&\n+\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert \n+\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{10}\\left(\\bar x,\\bar y\\right)\\right\\vert \n+\\left\\vert \\mathcal{N}\\left(y^t\\right)\\right\\vert \n\\end{array}\n\\]\n","sha256_utf8":"3ac8d089ee2d3e9141e4c42acc269b194c29a0e1eb7e9cef113fb9b6e20e5a55"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"利用活动集简化余维","source_units":[{"start":52717,"end":53964,"start_line":1051,"end_line":1076,"raw_tex":"linearly independent vectors involved. We use Lemma \\ref{lem:QandE}a) and \\ref{lem:QandE}b) to\nsubstitute $\\left\\vert Q_0\\left(x^t\\right)\\right\\vert $ with $\\left\\vert Q_0\\left(\\bar x\\right)\\right\\vert $ and $\\left\\vert \\mathcal{E}\\left(y^t\\right)\\right\\vert $ with\n$\\left\\vert \\mathcal{E}\\left(\\bar y\\right)\\right\\vert $, respectively. The latter set has cardinality of  $n-s-1$ due to Lemma \\ref{lem:a01}c). Additionally, we use Lemma \\ref{lem:QandE}c) and \\ref{lem:QandE}d) to conclude:\n\\[\n\\begin{array}{rcl}\n\\alpha^\\mathcal{S}_{t}&=&\n\\left\\vert P\\right\\vert +\\left\\vert Q_0\\left(\\bar x\\right)\\right\\vert +1+\nn-s-1\n+\\left\\vert a_{00}\\left(\\bar x,\\bar y\\right)\\right\\vert \n\\\\\n&&\n+\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert \n+\\left\\vert a_{10}\\left(\\bar x,\\bar y\\right)\\right\\vert .\n\\end{array}\n\\]\nFinally, $\\left\\vert a_{00}\\left(\\bar x,\\bar y\\right)\\right\\vert \n+\\left\\vert a_{10}\\left(\\bar x,\\bar y\\right)\\right\\vert =s$, cf. Lemma \\ref{lem:a01}b).\nThus, we have:\n\\[\n\\begin{array}{rcl}\n\\alpha^\\mathcal{S}_{t}&=&\n\\left\\vert P\\right\\vert +\\left\\vert Q_0\\left(\\bar x\\right)\\right\\vert \n+\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert \n+n.\n\\end{array}\n\\]\n","sha256_utf8":"fa62a0c49b37bfc195faa3fba3c39f6204eab06423f809b0d0b5ac1c48cbfe05"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"重构切空间与约束计数","source_units":[{"start":53964,"end":55292,"start_line":1077,"end_line":1113,"raw_tex":"{\\bf Step 1b.} We examine the tangent space corresponding to the T-stationary point $(\\bar x,\\bar y)$. For this purpose, we consider the following vectors from its definition:\n\\[\n\\begin{pmatrix}\n0\\\\\ne_i\n\\end{pmatrix},i\\in \\mathcal{E}(\\bar y),\n\\begin{pmatrix}\n0\\\\\ne_i\n\\end{pmatrix}, i \\in a_{10}\\left(\\bar x, \\bar y\\right)\\cup a_{00}\\left(\\bar x, \\bar y\\right),\n\\begin{pmatrix}\n0\\\\\ne\n\\end{pmatrix}.\\]\nThe latter vector is involved due to Lemma \\ref{lem:a01}a). The number of these vectors is due to Lemma \\ref{lem:a01}c) equal to $(n-s-1)+\ns+1=n$. Moreover, they are linearly independent due to MPOC-LICQ.\nHence, we can write the respective tangent space as follows:\n\\[\n    \\mathcal{T}^\\mathcal{R}_{(\\bar x,\\bar y)}=\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\n\nDh_p(\\bar x), 0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(\\bar x), 0\\end{pmatrix}\\xi=0,q \\in Q_0(\\bar x),\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y) \\cup a_{01}(\\bar x,\\bar y),\n\\xi_{n+1}=\\ldots=\\xi_{2n}=0.\n\\end{array}\n\\right.\\right\\}\n\\]\nIn total there are, due to MPOC-LICQ,\n\\[\n\\alpha^\\mathcal{R}=\\left\\vert P\\right\\vert +\\left\\vert Q_0\\left(\\bar x\\right)\\right\\vert \n+\\left\\vert a_{00}(\\bar x,\\bar y)\\right\\vert +\\left\\vert a_{01}(\\bar x,\\bar y)\\right\\vert +n\n\\]\nlinearly independent vectors involved.\n","sha256_utf8":"7987be589542a50063f22459e8b87eb1d3dc5314144fdc081b11059296187fe5"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"定义局部指标并比较Hessian","source_units":[{"start":55293,"end":56371,"start_line":1115,"end_line":1128,"raw_tex":"{\\bf Step 2.}\nLet $\\mathcal{T}\\subset \\R^{2n}$ be a linear subspace. We denote the number of negative\neigenvalues of $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}}$ by $QI^\\mathcal{S}_{t,\\mathcal{T}}$. Analogously, $QI^\\mathcal{R}_{t,\\mathcal{T}}$ stands for the number of negative eigenvalues of $D^2 L^\\mathcal{R}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}}$ and $\\overline{QI}^\\mathcal{R}_{\\mathcal{T}}$ stands for the number of negative eigenvalues of $D^2 L^\\mathcal{R}\\left(\\bar x , \\bar y \\right)\\restriction_{\\mathcal{T}}$. \nWe have the following relation between the involved Hessians of the Lagrange functions by denoting $E(i)=e_ie_{n+i}^T+e_{n+i}e_i^T$, $i =1, \\ldots, n$:\n\\begin{equation}\n\\label{eq:Hessetandbar}\n\\begin{array}{rcl}\n\\displaystyle\nD^2 L^\\mathcal{S}\\left( x^t, y^t\\right) &=& \\displaystyle\nD^2 L^\\mathcal{R}\\left( x^t, y^t\\right) - \\sum\\limits_{i\\in \\mathcal{H}^\\ge\\left(x^t,y^t\\right)}  \\eta_i^{\\ge,t} E(i)\\displaystyle \n+\n\\sum\\limits_{i\\in \\mathcal{H}^\\le\\left(x^t,y^t\\right)} \\eta_i^{\\le,t} E(i).\n\\end{array}\n\\end{equation}\n","sha256_utf8":"63aa26939266b61ec10f0b1a539849d7e21610f99e6290eb6d38822cc769cd0c"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"重构切空间上的指标稳定","source_units":[{"start":56372,"end":58541,"start_line":1130,"end_line":1162,"raw_tex":"{\\bf Step 2a.}\nIt holds for $t$ sufficiently small:\n\\[\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}=\\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}.\n\\]\n%Assume first $QI^\\mathcal{S}_{\\left(t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}>QI^\\mathcal{R}_{\\left(t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}$.\n%Then, there exists $\\bar \\xi \\in \\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}$, such that\n%\\[\n%\\bar \\xi^T D^2 L^\\mathcal{S}\\left( x^t, y^t\\right) \\bar \\xi < 0,\\quad\n%\\bar \\xi^T D^2 L^\\mathcal{R}\\left( x^t, y^t\\right)\\bar \\xi\\ge0.\n%\\]\nIndeed, by using (\\ref{eq:Hessetandbar}), we derive for any $\\xi \\in  \\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}$:\n\\begin{equation}\n    \\label{eq:s-r-h1}\n\\begin{array}{rcl}\n\\xi^T D^2 L^\\mathcal{S}\\left( x^t, y^t\\right) \\xi&=&\n%\\bar \\xi^T D^2 L^\\mathcal{R}\\left( x^t, y^t\\right)\\bar \\xi - \\bar \\xi^T \\left(\\sum\\limits_{i\\in \\mathcal{H}^\\ge\\left(x^t,y^t\\right)} \\bar \\eta_i^{t,\\ge} E(i)\\right)\\bar \\xi +\n%\\bar \\xi^T \\left(\\sum\\limits_{i\\in \\mathcal{H}^\\le\\left(x^t,y^t\\right)} \\bar \\eta_i^{t,\\le} E(i)\\right)\\bar \\xi\\\\\n%&=&\n \\xi^T D^2 L^\\mathcal{R}\\left(x^t, y^t\\right) \\xi - \\displaystyle \\sum\\limits_{i\\in \\mathcal{H}^\\ge\\left(x^t,y^t\\right)} 2\\eta_i^{\\ge,t} \\xi_i  \\xi_{n+i} \\\\\n&&\\displaystyle +\n\\sum\\limits_{i\\in \\mathcal{H}^\\le\\left(x^t,y^t\\right)} 2\\eta_i^{\\le,t}  \\xi_i \\xi_{n+i}=\n \\xi^T D^2 L^\\mathcal{R}\\left( x^t, y^t\\right)  \\xi,\n\\end{array}\n\\end{equation}\nsince $\\xi_{n+1}=\\ldots=\\xi_{2n}=0$ as seen in Step 1b.\n%Assuming  $QI^\\mathcal{S}_{\\left(t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}<QI^\\mathcal{R}_{\\left(t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}$ instead will analogously lead to a contradiction.\nHence, we get\n $\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}=QI^\\mathcal{R}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$.\nDue to NDT4, continuity arguments provide\n$\nQI^\\mathcal{R}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}=\\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$.\n","sha256_utf8":"55e7f6018385352faf6cb1aa48a14e2b510b04ba502539dedaa379ab920e50f4"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"构造比较切空间与特征向量","source_units":[{"start":58542,"end":60513,"start_line":1164,"end_line":1223,"raw_tex":"{\\bf Step 2b.}\nWe claim that the numbers of positive and negative eigenvalues of $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$ and of $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\prime}$, respectively, coincide, where\n\\[\n  \\mathcal{T}^{\\prime} = \\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\nDh_p(x^t),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(x^t),0\\end{pmatrix}\\xi=0,q \\in Q_0(\\bar x),\\\\\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(\\bar y),\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y),\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y),\\\\\n\\begin{pmatrix}\ny^t_ie_i,x^t_ie_i\n\\end{pmatrix}\\xi=0, i \\in a_{01}(\\bar x,\\bar y)\\cup a_{10}(\\bar x,\\bar y)\n\\end{array}\n\\right.\\right\\}.\n\\]\nLet $\\left\\{\\lambda^+_1,\\ldots,\\lambda^+_{k^+}\\right\\}$ be the positive \neigenvalues of \n$D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$ with corresponding eigenvectors \n$\\left\\{\\xi^+_1,\\ldots,\\xi^+_{k^+}\\right\\}$.\nHence, for all $k=1,\\ldots, k^+$:\n\\[\n{\\xi^+_k}^T D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\xi^+_k>0.\n\\]\nWe rewrite the tangent space as follows:\n\\[\n    \\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}=\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\n\nDh_p(\\bar x),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(\\bar x),0\\end{pmatrix}\\xi=0,q \\in Q_0(\\bar x),\\\\\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(\\bar y),\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y),\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y),\\\\\n\\begin{pmatrix}\n\\bar y_ie_i,\\bar x_ie_i\n\\end{pmatrix}\\xi=0, i \\in a_{01}(\\bar x,\\bar y)\\cup a_{10}(\\bar x,\\bar y)\n\\end{array}\n\\right.\\right\\}.\n\\]\n","sha256_utf8":"06de60dcb13787c536c9db3a435edeb8f5e4345e866e2071030b84ed72944da7"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"延拓向量并比较正负惯性","source_units":[{"start":60513,"end":62477,"start_line":1224,"end_line":1240,"raw_tex":"Due to MPOC-LICQ, the application of the implicit function theorem provides the existence of\n$\\delta_2,\\delta_3>0$ such that for all $k=1,\\ldots,k^+$ and $t<\\delta_2$ there exists $\\xi_{k,t}$ with $\\left\\| \\xi_{k,t}-\\xi^+_k\\right\\|<\\delta_3$ and $\\xi_{k,t}\\in \\mathcal{T}^\\prime$.\n We can choose $t$ even smaller, such that $\\xi_{1,t},\\ldots,\\xi_{k^+,t}$ remain linearly independent and for all $k=1,\\ldots,k^+$ it holds:\n\\[\n{ \\xi_{k,t}}^T D^2 L^\\mathcal{S}\\left( x^t, y^t\\right) \\xi_{k,t}>0.\n\\]\n%whenever $\\NNorm{\\xi_k-\\xi^+_k}{}$ $<\\delta_1$, $k=1, \\ldots, k^+$.\n%\n%Altogether, for $t$ small enough there exist a set of linearly independent vectors \n%$\\left\\{\\tilde \\xi^+_1,\\ldots,\\tilde \\xi^+_{k^+}\\right\\}$ with\n%$\\tilde \\xi^+_k \\in \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$, for all $k=1,\\ldots,k^+$,\n%and\n%\\[\n%{\\tilde \\xi_k^+}{^T} D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\tilde \\xi^+_k>0.\n%\\]\nHence, $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\prime}$ has at least $k^+$ positive eigenvalues. If we repeat the above reasoning for negative eigenvalues, the matrix $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\prime}$ has at least as many negative eigenvalues as $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$. Additionally, we show that the dimensions of $\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}$ and $\\mathcal{T}^\\prime$ coincide. By Step 1b, we have $2n-\\alpha^\\mathcal{R}$ for the dimension of $\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}$. Since MPOC-LICQ remains valid in the neighborhood of $(\\bar x, \\bar y)$, we get again \n$2n-\\alpha^{\\mathcal{R}}$ for the dimension of $\\mathcal{T}^\\prime$. By continuity arguments, NDT4 and (\\ref{eq:s-r-h1}) provide that $D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\restriction_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$ is nonsingular. Altogether, the assertion follows.\n","sha256_utf8":"9c01b941d8d357c56652da0b8baecf2fa874f949dfa259bdc90013a766258219"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"由空间包含得到指标差界","source_units":[{"start":62478,"end":63704,"start_line":1242,"end_line":1260,"raw_tex":"{\\bf Step 2c.} \nWe claim that \n\\[\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}}\\le QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}+\\alpha^\\mathcal{R}-\\alpha^\\mathcal{S}_t.\n\\]\nFor that, we focus on the dimension of $\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$. As a consequence of Step 1a it is $2n-\\alpha^\\mathcal{S}_t$. \nDue to continuity arguments, we can choose $t$ small enough to ensure \n$x^t_{i}\\ne 0$, $i\\in a_{10}(\\bar x,\\bar y)$ and\n$y^t_{i}\\ne 0$, $i\\in a_{01}(\\bar x,\\bar y)$.\nUsing this and Lemma \\ref{lem:QandE}a), \\ref{lem:QandE}b), and \\ref{lem:QandE}d), it follows\nthat $\\mathcal{T}^\\prime \\subset \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$.\nTherefore, using Step 2b,  \n$\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}}\\le 2n-\\alpha^\\mathcal{S}_t-k^+$.\nWe observe in view of NDT4 and Step 1b that \n$\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}%\\overset{Step\\,\\,2a}{=}QI^\\mathcal{R}_{\\left(t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}%=\\overline{QI}^\\mathcal{R}_{\\left(\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}\\right)}\n=2n-\\alpha^\\mathcal{R}-k^+$.\nThe assertion follows immediately.\n","sha256_utf8":"535b3b40bc67fd7ebf1adfc3efac6efce76f2036877c2d5a44ed3e1e6d112c7b"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"将指标差转化为未活动数量","source_units":[{"start":63705,"end":64804,"start_line":1262,"end_line":1279,"raw_tex":"{\\bf Step 3.} Let us show that\n\\[\n\\max\\left\\{m - \\left\\vert \\left\\{i\\in a_{01}\\left(\\bar x,\\bar y\\right)\\,\\left\\vert\\,\\bar \\sigma_{1,i}=0\\right.\\right\\} \\right\\vert , 0\\right\\}\\le TI.\n\\]\nIn view of Step 2a, Step 2c and due to continuity, we have for $t$ sufficiently small:\n\n\\[\n\\begin{array}{rcl}\nm=QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}}&\\overset{Step\\,2c}{\\le}& QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}+\\alpha^\\mathcal{R}-\\alpha^\\mathcal{S}_t \\overset{Step\\,2a}{=} \n\\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}\n+\\alpha^\\mathcal{R}-\\alpha^\\mathcal{S}_t\\\\\n&\\overset{Step\\,1}{=}& \\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}\n+\\left\\vert a_{00}(\\bar x,\\bar y)\\right\\vert +\\left\\vert a_{01}(\\bar x,\\bar y)\\right\\vert -\n\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert \\\\\n&=&TI+\\left\\vert a_{01}(\\bar x,\\bar y)\\right\\vert -\n\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert .\n\\end{array}\n\\]\n","sha256_utf8":"af7c2ae0033faed48b0b06e0db3096481756dff19b3ce5b04dedbe098eac271b"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"零乘子数量控制指标损失","source_units":[{"start":64804,"end":66034,"start_line":1280,"end_line":1294,"raw_tex":"We show for $t$ sufficiently small:\n\\[\\left\\vert a_{01}(\\bar x,\\bar y)\\right\\vert -\n\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert \\le\n\\left\\vert \\left\\{i\\in a_{01}\\left(\\bar x,\\bar y\\right)\\,\\left\\vert\\,\\bar \\sigma_{1,i}=0\\right.\\right\\} \\right\\vert ,\n\\]\nand the assertion will follow immediately since $TI\\ge 0$.\nClearly,\n\\[\\left\\vert a_{01}(\\bar x,\\bar y)\\right\\vert -\n\\left\\vert \\mathcal{H}\\left(x^t,y^t\\right)\\cap a_{01}\\left(\\bar x,\\bar y\\right)\\right\\vert =\n\\left\\vert a_{01}(\\bar x,\\bar y)\\backslash \\mathcal{H}\\left(x^t,y^t\\right) \\right\\vert .\n\\]\nSuppose $\\bar i \\in a_{01}\\left(\\bar x,\\bar y\\right)$ with $\\bar \\sigma_{1,\\bar i}\\ne 0$. In view of Remark \\ref{rem:multipliers}, the difference $\\eta^{\\ge,t}_{\\bar i}-\\eta^{\\le,t}_{\\bar i}$ cannot vanish for all $t$ sufficiently small. In particular, one of the multipliers $\\eta^{\\ge,t}_{\\bar i}$ or $\\eta^{\\le,t}_{\\bar i}$ has to be not vanishing for all $t$ sufficiently small. Hence, $\\bar i\\in \\mathcal{H}\\left(x^t,y^t\\right)$. We therefore have: \n\\[\na_{01}(\\bar x,\\bar y)\\backslash \\mathcal{H}\\left(x^t,y^t\\right) \\subset  \\left\\{i\\in a_{01}\\left(\\bar x,\\bar y\\right)\\,\\left\\vert\\,\\bar \\sigma_{1,i}= 0\\right.\\right\\}.\n\\]\n","sha256_utf8":"d7e65a18653142124e0550a4de5871478f382273189f6bb08a35637d63e4dd30"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"按双活动坐标比值划分子列","source_units":[{"start":66036,"end":66856,"start_line":1297,"end_line":1303,"raw_tex":"{\\bf Step 4.}\nWithout loss of generality -- considering subsequences if needed -- we can assume that for any $i\\in a_{00}(\\bar x,\\bar y)$ at least one of the sequences $\\frac{x_i^t}{y_i^t}$ or $\\frac{y_i^t}{x_i^t}$ is convergent. First, we note that the quotients are well defined due to Lemma \\ref{lem:QandE}c). Moreover, if the former sequence does not contain a convergent subsequence, we find a subsequence that tends to plus or minus infinity. Consequently, the corresponding subsequence of the latter reciprocal sequence has to converge to zero.\nWe define the following auxiliary sets:\n\\[\na_{00}^x(\\bar x,\\bar y)=\\left\\{i \\in a_{00}(\\bar x,\\bar y)\\,\\left\\vert\\, \\frac{x^t_i}{y^t_i}\\mbox{ converges for } t \\to 0 \\right. \\right\\},\na_{00}^y(\\bar x,\\bar y)=a_{00}(\\bar x,\\bar y)\\backslash a_{00}^x(\\bar x,\\bar y).\n\\]\n","sha256_utf8":"76efcdd22ecc2b5512efd6e665adb9982b9b9205d9eb3fb77be34c582de0dfee"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"放松一个双活动方向","source_units":[{"start":66857,"end":68881,"start_line":1305,"end_line":1357,"raw_tex":"For $\\bar i \\in a_{00}^x\\left(\\bar x,\\bar y\\right)$ we consider $\\mathcal{T}^{\\mathcal{R}}_{\\left(\\bar x,\\bar y\\right)}$ and replace two of the involved equations, namely \n$\\begin{pmatrix}\ne_{\\bar i},0\n\\end{pmatrix}\\xi=0$ and \n$\\begin{pmatrix}\n0,e_{\\bar i}\n\\end{pmatrix}\\xi=0$ by one equation\n$\\begin{pmatrix}\ne_{\\bar i},\\lim\\limits_{t \\to 0}\\frac{x_{\\bar i}^t}{y_{\\bar i}^t}e_{\\bar i}\n\\end{pmatrix}\\xi=0$.\nClearly, the vectors involved in the definition of the newly generated linear space, i.\\,e.\n\\[\n    \\mathcal{T}^{{\\bar i}}=\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\n\nDh_p(\\bar x),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(\\bar x),0\\end{pmatrix}\\xi=0,q \\in Q_0(\\bar x),\\\\\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(\\bar y),\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y)\\backslash\\{\\bar i\\},\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y)\\backslash\\{\\bar i\\},\\\\\n\\begin{pmatrix}\ne_{\\bar i},\\lim\\limits_{t \\to 0}\\frac{x_{\\bar i}^t}{y_{\\bar i}^t}e_{\\bar i}\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n\\bar y_ie_i,\\bar x_ie_i\n\\end{pmatrix}\\xi=0, i \\in a_{01}(\\bar x,\\bar y)\\cup a_{10}(\\bar x,\\bar y)\\}\n\\end{array}\n\\right.\\right\\},\n\\]\nremain linearly independent.\nThe dimension of $ \\mathcal{T}^{{\\bar i}}$ is greater than the dimension of\n$\\mathcal{T}^{\\mathcal{R}}_{\\left(\\bar x,\\bar y\\right)}$ by one. Moreover, there exists\n$\\xi^{\\bar i} \\in \\mathcal{T}^{{\\bar i}}$ with \n$\n%\\xi^{\\bar i}_{n+i} =0, i \\in a_{00}(\\bar x, \\bar y)\\backslash\\{\\bar i\\}, \\quad \n\\xi^{\\bar i}_{n+\\bar i} \\not =0$.\nIndeed, assume that no such $\\xi^{\\bar i}$ exists, then we can add the equation \n$\\begin{pmatrix}\n0,e_{\\bar i}\n\\end{pmatrix}\\xi=0$ to the defining equations of $ \\mathcal{T}^{ {\\bar i}}$ wihout changing it. The resulting space, however, is identical to\n$\\mathcal{T}^{\\mathcal{R}}_{\\left(\\bar x,\\bar y\\right)}$, a contradiction.\nWithout loss of generality, we assume \n$\\xi^{\\bar i}_{n+\\bar i}=1$.\n","sha256_utf8":"fa4393f87b3ecd64d689fb5e5e0701eb6b288ea88601b24051d3ddd66b29d5b0"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"延拓新增双活动方向","source_units":[{"start":68881,"end":70084,"start_line":1358,"end_line":1394,"raw_tex":"Further, by straightforward application of the implicit function theorem and due to Lemma \\ref{lem:QandE}a) and \\ref{lem:QandE}b), we find a sequence of vectors $\\xi^{\\bar i}_t \\in \\mathcal{T}^{ \\bar i}_{t}$ that converges to $\\xi^{\\bar i}$ for $t \\to 0$. \nFor this, we define\n\\[\n\\mathcal{T}^{\\bar i}_t=\\left\\{\n\\xi \\in \\R^{2n}\\,\\left\\vert\\, \\begin{array}{l} \\begin{pmatrix}\nDh_p(x^t),0\\end{pmatrix} \\xi=0, p \\in P, \n\\begin{pmatrix}\nDg_q(x^t),0\\end{pmatrix}\\xi=0,q \\in Q_0(x^t),\\\\\n\\begin{pmatrix}\n0,e\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i\\in \\mathcal{E}(y^t),\\\\\n\\begin{pmatrix}\ne_i,0\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y)\\backslash\\{\\bar i\\},\n\\begin{pmatrix}\n0,e_i\n\\end{pmatrix}\\xi=0, i \\in a_{00}(\\bar x,\\bar y)\\backslash\\{\\bar i\\},\\\\\n\\begin{pmatrix}\ne_{\\bar i},\\frac{x_{\\bar i}^t}{y_{\\bar i}^t} e_{\\bar i}\n\\end{pmatrix}\\xi=0,\n\\begin{pmatrix}\ny^t_ie_i,x^t_ie_i\n\\end{pmatrix}\\xi=0, i \\in a_{01}(\\bar x,\\bar y)\\cup a_{10}(\\bar x,\\bar y)\n\\end{array}\n\\right.\\right\\}.\n\\]\nWe again have, due to continuity arguments, that \n$\\xi^{\\bar i}_{t,n+\\bar i} \\not =0$.\n%\\left\\{\n%\\begin{array}{ll}\n%=0& \\mbox{for }i\\ne \\bar i,\\\\\n%\\ne 0& \\mbox{for } i=\\bar i.\n%\\end{array}\\right.\n%\\]\n","sha256_utf8":"c34f954981933297087a0657baf14651b8998df8fa3ce1f86671dcab75e0aa3c"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"构造倒数比值对应方向","source_units":[{"start":70084,"end":71027,"start_line":1395,"end_line":1410,"raw_tex":"For $\\bar i \\in a_{00}^y\\left(\\bar x,\\bar y\\right)$ we proceed\n%we construct $\\xi^{\\bar i}$ and a sequence of vectors $\\xi^{\\bar i}_t \\in \\mathcal{T}^{ \\bar i}_{t}$ that converges to $\\xi^{\\bar i}$ for $t \\to 0$, \nanalogously by\nconsidering $\\mathcal{T}^{\\mathcal{R}}_{\\left(\\bar x,\\bar y\\right)}$ again and replace two of the  involved equations \n$\\begin{pmatrix}\ne_{\\bar i},0\n\\end{pmatrix}\\xi=0$ and \n$\\begin{pmatrix}\n0,e_{\\bar i}\n\\end{pmatrix}\\xi=0$ by the equation\n$\\begin{pmatrix}\n\\lim\\limits_{t \\to 0}\\frac{y_{\\bar i}^t}{x_{\\bar i}^t} e_{\\bar i},e_{\\bar i}\n\\end{pmatrix}\\xi=0$. \nBy the same arguments as before, we find\n$\\xi^{\\bar i} \\in \\mathcal{T}^{\\bar i}$ with\n$\\xi^{\\bar i}_{\\bar i} \\ne 0$. Again we will assume $\\xi^{\\bar i}_{\\bar i} = 1$ and find a sequence of vectors $\\xi^{\\bar i}_t \\in \\mathcal{T}^{ \\bar i}_{t}$ that converges to $\\xi^{\\bar i}$ for $t \\to 0$. Due to continuity, it holds then $\\xi^{\\bar i}_{t,\\bar i} \\ne 0$.\n","sha256_utf8":"5648f258bf3775d3126be1ff0ecf56c1f8c5a2c14a5fc662a82a49b4a8e94f7b"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"新增方向与原空间线性独立","source_units":[{"start":71029,"end":72627,"start_line":1413,"end_line":1435,"raw_tex":"It is straightforward to verify the following observations for $t$ sufficiently small:\n\n    a) Let $\\left\\{\\xi^{\\prime,1},\\ldots,\\xi^{\\prime,\\ell}\\right\\}$ be a base of $\\mathcal{T}^{\\prime}$, cf. Step 2b, then\n    $\\left\\{\\xi^{\\prime,1},\\ldots,\\xi^{\\prime,\\ell}\\right\\}\\cup \\left\\{\\xi^{\\bar i}_{t}\\,\\left\\vert\\,\\bar i \\in a_{00}(\\bar x,\\bar y)\\right.\\right\\}$\nis a set of linear independent vectors.\nIn fact, suppose for some coefficients $b_{i}, \\in \\R$, $i \\in a_{00}\\left(\\bar x,\\bar y\\right),\\beta_i \\in \\R, i=1,\\ldots,\\ell$ it holds:\n\\[\n\\sum\\limits_{i \\in a_{00}\\left(\\bar x,\\bar y\\right)} b_{i}\\xi^{i}_{t}\n+\\sum\\limits_{i=1}^{\\ell}\\beta_i\\xi^{\\prime,i}=0.\n\\]\nFor $\\bar i \\in a_{00}^x\\left(\\bar x,\\bar y\\right)$ we consider the $(n+\\bar i)$-th row of this sum\n\\[\nb_{\\bar i}\\underbrace{\\xi^{\\bar i}_{t,n+\\bar i}}_{\\ne 0}+\\sum\\limits_{i \\in a_{00}\\left(\\bar x,\\bar y\\right)\\backslash\\left\\{\\bar i\\right\\}} b_{i}\\underbrace{\\xi^{i}_{t,n+\\bar i}}_{=0}+\\sum\\limits_{i=1}^{\\ell}\\beta_i\\underbrace{\\xi^{\\prime,i}_{n+\\bar i}}_{=0}=0.\n\\]\nIf instead $\\bar i \\in a_{00}^y\\left(\\bar x,\\bar y\\right)$ we consider the $\\bar i$-th row of the sum\n\\[\nb_{\\bar i}\\underbrace{\\xi^{\\bar i}_{t,\\bar i}}_{\\ne 0}+\\sum\\limits_{i \\in a_{00}\\left(\\bar x,\\bar y\\right)\\backslash\\left\\{\\bar i\\right\\}} b_{i}\\underbrace{\\xi^{i}_{t,\\bar i}}_{=0}+\\sum\\limits_{i=1}^{\\ell}\\beta_i\\underbrace{\\xi^{\\prime,i}_{\\bar i}}_{=0}=0.\n\\]\nAltogether, it must hold $b_{i} =0$, $i \\in a_{00}\\left(\\bar x,\\bar y\\right)$. However, this implies\n\\[\n\\sum\\limits_{i=1}^{\\ell}\\beta_i\\xi^{\\prime,i}=0.\n\\]\nHence, $\\beta_i=0$ for $i=1,\\ldots,\\ell$.\n","sha256_utf8":"7786796bc22f93a380f3ffb7355139e405ad024d65c9763bd0b2202acbf8eea3"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"新增方向属于正则化切空间","source_units":[{"start":72628,"end":72773,"start_line":1437,"end_line":1437,"raw_tex":"    b) It holds $\\xi^{\\bar i}_{t} \\in \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$, cf. Step 1a, for any $\\bar i \\in a_{00}(\\bar x,\\bar y)$. \n","sha256_utf8":"09f9923517fc5d1a7aa6e087c32fa5d609dc99a42eeccdb590d091321791be72"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"乘积Hessian贡献非正","source_units":[{"start":72778,"end":73504,"start_line":1439,"end_line":1448,"raw_tex":"    c) It holds $\\left(\\eta_i^{\\le,t}-\\eta_i^{\\ge,t}\\right)\\xi^{\\bar i}_{t,i}\\xi^{\\bar i}_{t,n+i} \\leq 0, i \\in \\mathcal{H}\\left(x^t,y^t\\right)$. Since $\\xi^{\\bar i}_{t} \\in \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$, we obtain:\n    \\[\n\\begin{array}{rcl}\n\\left(\\eta_i^{\\le,t}-\\eta_i^{\\ge,t}\\right)\\xi^{\\bar i}_{t,i}\\xi^{\\bar i}_{t,n+i}\n&=&\\left(\\eta_i^{\\ge,t}-\\eta_i^{\\le,t}\\right)\\left(\\xi^{\\bar i}_{t,n+i}\\right)^2\\frac{x^t_i}{y^t_i}.\n\\end{array}\n\\]\nIf $i\\in \\mathcal{H}^{\\ge}\\left(x^t,y^t\\right)$,\nwe have $x_i^t<0,y_i^t>0$ and $\\eta_i^{\\le,t}=0$. Moreover, due to ND2, we have\n$\\eta_i^{\\ge,t}>0$. The assertion follows immediately. The other case $i\\in \\mathcal{H}^{\\le}\\left(x^t,y^t\\right)$ is completely analogous.\n","sha256_utf8":"e476115e984d8dfb0b0886972cbab9aaa9a5be36d75cd88a63e12e5bb6161ecb"}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","title":"双活动贡献趋于负无穷","source_units":[{"start":73505,"end":75368,"start_line":1450,"end_line":1472,"raw_tex":"    d) It holds $\\lim\\limits_{t\\to 0}\\left(\\eta_{\\bar i}^{\\le,t}-\\eta_{\\bar i}^{\\ge,t}\\right)\\xi^{\\bar i}_{t,\\bar i}\\xi^{\\bar i}_{t,n+\\bar i}=-\\infty$. \n    We calculate:\n\\[\n\\begin{array}{rcl}\n\\left(\\eta_{\\bar i}^{\\le,t}-\\eta_{\\bar i}^{\\ge,t}\\right)\\xi^{\\bar i}_{t,{\\bar i}}\\xi^{\\bar i}_{t,n+{\\bar i}}\n&=&\\left\\{\n\\begin{array}{ll}\n   \\left(\\eta_{\\bar i}^{\\ge,t}-\\eta_{\\bar i}^{\\le,t}\\right)\\left(\\xi^{\\bar i}_{t,n+\\bar i}\\right)^2\\frac{x^t_{\\bar i}}{y^t_{\\bar i}}& \\mbox{for } \\bar i \\in a_{00}^x(\\bar x,\\bar y),\\\\\n   \\left(\\eta_{\\bar i}^{\\ge,t}-\\eta_{\\bar i}^{\\le,t}\\right)\\left(\\xi^{\\bar i}_{t,\\bar i}\\right)^2\\frac{y^t_{\\bar i}}{x^t_{\\bar i}}& \\mbox{for } \\bar i \\in a_{00}^y(\\bar x,\\bar y).\n\\end{array}\\right.\n\\end{array}\n\\]\nLet us suppose $\\bar i \\in a_{00}^x(\\bar x,\\bar y)$. We have $y^t_{\\bar i}\\ne 0$ and, thus, $\\nu^t_{\\bar i}=0$. We use Remark \\ref{rem:multipliers} and NDT3 to conclude that the sequence\n$ \\left(\\eta_{\\bar i}^{\\ge,t}-\\eta_{\\bar i}^{\\le,t}\\right)x^t_{\\bar i}$ converges to $\\varrho_{2,\\bar i}<0$ for $t \\to 0$. Further,\n$\\frac{1}{y^t_{\\bar i}}>0$ tends to infinity for $t \\to 0$. Finally,\n$\\left(\\xi^{\\bar i}_{t,n+\\bar i}\\right)^2$ converges to $1$ for $t \\to 0$, due to the construction of\n$\\xi^{\\bar i}_t$.\nThus, the assertion follows.\nInstead, let us suppose $\\bar i \\in a_{00}^y(\\bar x,\\bar y)$. \nThis time, we have that\n$\\left(\\xi^{\\bar i}_{t,\\bar i}\\right)^2$ converges to $1$ for $t \\to 0$.\nDue to Remark \\ref{rem:multipliers} and NDT3, $\\left(\\eta_{\\bar i}^{\\ge,t}-\\eta_{\\bar i}^{\\le,t}\\right)y^t_{\\bar i}$ converges to $\\varrho_{1,\\bar i}\\ne 0$ for $t \\to 0$.\nIf $\\bar i\\in \\mathcal{H}^{\\ge}\\left(x^t,y^t\\right)$, then $\\varrho_{1,\\bar i}$ is positive from here. Also, $\\frac{1}{x^t_{\\bar i}}<0$ tends to minus infinity for $t \\to 0$. The other case $\\bar i\\in \\mathcal{H}^{\\le}\\left(x^t,y^t\\right)$ is completely analogous. \n","sha256_utf8":"482f47e7855539c98b2229a563b98dd43a5e2f75fac92c034a2833627917b82d"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"重构二次项保持有界","source_units":[{"start":75369,"end":75534,"start_line":1474,"end_line":1474,"raw_tex":"e) We notice that $\\displaystyle \\xi^{\\bar i ^T}_{t} D^2 L^\\mathcal{R}\\left( x^t, y^t\\right)\\xi^{\\bar i}_{t}$ converges for $t \\to 0$ due to the construction above.\n","sha256_utf8":"283aa2bd6f79c0877fff9a6587f70b3c63887a79fdeb4b869f8196d23f0fde5f"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"新增方向给出负二次值","source_units":[{"start":75535,"end":76675,"start_line":1476,"end_line":1489,"raw_tex":"Finally, for $\\bar i\\in  a_{00}(\\bar x,\\bar y)$ we estimate:\n\\[\n\\begin{array}{rcl}\n\\xi_t^{\\bar i ^T} D^2 L^\\mathcal{S}\\left( x^t, y^t\\right)\\xi_t^{\\bar i}&\\overset{(\\ref{eq:Hessetandbar})}{=}&\n\\displaystyle \\xi^{\\bar i ^T}_{t} D^2 L^\\mathcal{R}\\left(x^t, y^t\\right)\\xi^{\\bar i}_{t}\\\\ && \\displaystyle\n- \\sum\\limits_{i\\in \\mathcal{H}^\\ge\\left(x^t,y^t\\right)} 2 \\eta_i^{\\ge,t} \\xi_{t,i}^{\\bar i}  \\xi_{t,n+i}^{\\bar i}+\n\\sum\\limits_{i\\in \\mathcal{H}^\\le\\left(x^t,y^t\\right)} 2 \\eta_i^{\\le,t}\\xi_{t,i}^{\\bar i}\\xi_{t,n+i}^{\\bar i} \\\\\n&= &\n\\displaystyle \\xi^{\\bar i ^T}_{t} D^2 L^\\mathcal{R}\\left(x^t, y^t\\right)\\xi^{\\bar i}_{t}\n +2\\sum\\limits_{i\\in \\mathcal{H}\\left(x^t,y^t\\right)} \\left( \\eta_i^{\\le,t}-\\eta_i^{\\ge,t} \\right)\\xi_{t,i}^{\\bar i}  \\xi_{t,n+i}^{\\bar i}\\\\ \n&\\overset{c)}{\\le}&\\displaystyle \\xi^{\\bar i ^T}_{t} D^2 L^\\mathcal{R}\\left(x^t, y^t\\right)\\xi^{\\bar i}_{t}+2\\left(\\eta_{\\bar i}^{\\le,t}-\\eta_{\\bar i}^{\\ge,t}\\right)\\xi^{\\bar i}_{t,\\bar i}\\xi^{\\bar i}_{t,n+\\bar i}.\n\\end{array}\n\\]\nThus, due to d) and e), $\\xi^{\\bar i ^T}_{t} D^2 L^\\mathcal{S}\\left(x^t, y^t\\right) \\xi^{\\bar i}_{t}$ has to be negative for $t$ small enough. \n","sha256_utf8":"949ae199faee8eadd2b4ce8de87897cfa926e431f387271801858dd69eba3a8d"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"加入双活动指标完成上界","source_units":[{"start":76675,"end":77534,"start_line":1490,"end_line":1501,"raw_tex":"As we have seen in Step 2c, it holds $\\mathcal{T}^\\prime \\subset \\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}$. Then, due a) and b), we have therefore:\n\\[\nQI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}}\n\\geq QI^\\mathcal{S}_{t,\\mathcal{T}^\\prime} + \\left\\vert a_{00}(\\bar x,\\bar y)\\right\\vert.\n\\]\nBy Step 2a, we have $QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}=\\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}$, and by Step 2b, $QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}=QI^\\mathcal{S}_{t,\\mathcal{T}^\\prime}$. Overall, we obtain:\n\\[\nm=QI^\\mathcal{S}_{t,\\mathcal{T}^\\mathcal{S}_{\\left(x^t,y^t\\right)}}\n\\ge\n\\overline{QI}^\\mathcal{R}_{\\mathcal{T}^\\mathcal{R}_{\\left(\\bar x,\\bar y\\right)}}+\\left\\vert a_{00}(\\bar x,\\bar y)\\right\\vert=TI.\n\\]\n\\end{proof}","sha256_utf8":"4e92ed7a4693e3e4c92468c7f12f0ec56f1a45c56ac0f5a27f5abbbb69fc4d40"}]},{"id":"S064","semantic_type":"external_fact","kind":"external_fact","title":"坐标比值子列抽取","source_units":[],"declared_external":true,"declaration_text":["For a sequence of nonzero real numbers, either it has a finite convergent subsequence or it has a subsequence tending to positive or negative infinity, along which its reciprocal tends to zero. A common subsequence can be chosen for finitely many such coordinate sequences."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S040","edges":[{"source_id":"S001","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S006","kind":"supports"},{"source_id":"S002","target_id":"S012","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S003","target_id":"S006","kind":"supports"},{"source_id":"S003","target_id":"S009","kind":"supports"},{"source_id":"S004","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S011","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S010","target_id":"S013","kind":"supports"},{"source_id":"S011","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S014","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S003","target_id":"S022","kind":"supports"},{"source_id":"S005","target_id":"S022","kind":"supports"},{"source_id":"S003","target_id":"S023","kind":"supports"},{"source_id":"S005","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S003","target_id":"S024","kind":"supports"},{"source_id":"S022","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S022","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S023","target_id":"S021","kind":"supports"},{"source_id":"S024","target_id":"S021","kind":"supports"},{"source_id":"S025","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S003","target_id":"S028","kind":"supports"},{"source_id":"S009","target_id":"S028","kind":"supports"},{"source_id":"S011","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S003","target_id":"S029","kind":"supports"},{"source_id":"S011","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S003","target_id":"S030","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S007","target_id":"S033","kind":"supports"},{"source_id":"S017","target_id":"S033","kind":"supports"},{"source_id":"S003","target_id":"S034","kind":"supports"},{"source_id":"S007","target_id":"S034","kind":"supports"},{"source_id":"S017","target_id":"S034","kind":"supports"},{"source_id":"S020","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S029","target_id":"S035","kind":"supports"},{"source_id":"S011","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S036","kind":"supports"},{"source_id":"S029","target_id":"S036","kind":"supports"},{"source_id":"S011","target_id":"S036","kind":"supports"},{"source_id":"S034","target_id":"S037","kind":"supports"},{"source_id":"S029","target_id":"S037","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S007","target_id":"S038","kind":"supports"},{"source_id":"S017","target_id":"S038","kind":"supports"},{"source_id":"S023","target_id":"S038","kind":"supports"},{"source_id":"S029","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S020","target_id":"S039","kind":"supports"},{"source_id":"S023","target_id":"S039","kind":"supports"},{"source_id":"S029","target_id":"S039","kind":"supports"},{"source_id":"S017","target_id":"S039","kind":"supports"},{"source_id":"S003","target_id":"S039","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S032","kind":"supports"},{"source_id":"S035","target_id":"S032","kind":"supports"},{"source_id":"S036","target_id":"S032","kind":"supports"},{"source_id":"S037","target_id":"S032","kind":"supports"},{"source_id":"S038","target_id":"S032","kind":"supports"},{"source_id":"S039","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S008","target_id":"S040","kind":"supports"},{"source_id":"S014","target_id":"S040","kind":"supports"},{"source_id":"S026","target_id":"S043","kind":"supports"},{"source_id":"S012","target_id":"S043","kind":"supports"},{"source_id":"S013","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S020","target_id":"S044","kind":"supports"},{"source_id":"S031","target_id":"S044","kind":"supports"},{"source_id":"S004","target_id":"S045","kind":"supports"},{"source_id":"S006","target_id":"S045","kind":"supports"},{"source_id":"S007","target_id":"S045","kind":"supports"},{"source_id":"S020","target_id":"S045","kind":"supports"},{"source_id":"S006","target_id":"S046","kind":"supports"},{"source_id":"S012","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S007","target_id":"S047","kind":"supports"},{"source_id":"S019","target_id":"S047","kind":"supports"},{"source_id":"S045","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S004","target_id":"S049","kind":"supports"},{"source_id":"S007","target_id":"S049","kind":"supports"},{"source_id":"S018","target_id":"S049","kind":"supports"},{"source_id":"S019","target_id":"S049","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S043","target_id":"S050","kind":"supports"},{"source_id":"S045","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S031","target_id":"S050","kind":"supports"},{"source_id":"S019","target_id":"S050","kind":"supports"},{"source_id":"S044","target_id":"S051","kind":"supports"},{"source_id":"S045","target_id":"S051","kind":"supports"},{"source_id":"S047","target_id":"S051","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S008","target_id":"S051","kind":"supports"},{"source_id":"S014","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S017","target_id":"S052","kind":"supports"},{"source_id":"S011","target_id":"S052","kind":"supports"},{"source_id":"S031","target_id":"S053","kind":"supports"},{"source_id":"S064","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S045","target_id":"S054","kind":"supports"},{"source_id":"S004","target_id":"S054","kind":"supports"},{"source_id":"S019","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S018","target_id":"S055","kind":"supports"},{"source_id":"S031","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S056","kind":"supports"},{"source_id":"S054","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S018","target_id":"S056","kind":"supports"},{"source_id":"S048","target_id":"S057","kind":"supports"},{"source_id":"S055","target_id":"S057","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S043","target_id":"S058","kind":"supports"},{"source_id":"S055","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S058","kind":"supports"},{"source_id":"S031","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S011","target_id":"S059","kind":"supports"},{"source_id":"S013","target_id":"S059","kind":"supports"},{"source_id":"S055","target_id":"S060","kind":"supports"},{"source_id":"S056","target_id":"S060","kind":"supports"},{"source_id":"S058","target_id":"S060","kind":"supports"},{"source_id":"S017","target_id":"S060","kind":"supports"},{"source_id":"S007","target_id":"S060","kind":"supports"},{"source_id":"S055","target_id":"S061","kind":"supports"},{"source_id":"S056","target_id":"S061","kind":"supports"},{"source_id":"S006","target_id":"S061","kind":"supports"},{"source_id":"S046","target_id":"S062","kind":"supports"},{"source_id":"S059","target_id":"S062","kind":"supports"},{"source_id":"S060","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S047","target_id":"S063","kind":"supports"},{"source_id":"S049","target_id":"S063","kind":"supports"},{"source_id":"S050","target_id":"S063","kind":"supports"},{"source_id":"S057","target_id":"S063","kind":"supports"},{"source_id":"S058","target_id":"S063","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S008","target_id":"S063","kind":"supports"},{"source_id":"S014","target_id":"S063","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S043","target_id":"S041","kind":"supports"},{"source_id":"S044","target_id":"S041","kind":"supports"},{"source_id":"S045","target_id":"S041","kind":"supports"},{"source_id":"S046","target_id":"S041","kind":"supports"},{"source_id":"S047","target_id":"S041","kind":"supports"},{"source_id":"S048","target_id":"S041","kind":"supports"},{"source_id":"S049","target_id":"S041","kind":"supports"},{"source_id":"S050","target_id":"S041","kind":"supports"},{"source_id":"S051","target_id":"S041","kind":"supports"},{"source_id":"S052","target_id":"S041","kind":"supports"},{"source_id":"S053","target_id":"S041","kind":"supports"},{"source_id":"S054","target_id":"S041","kind":"supports"},{"source_id":"S055","target_id":"S041","kind":"supports"},{"source_id":"S056","target_id":"S041","kind":"supports"},{"source_id":"S057","target_id":"S041","kind":"supports"},{"source_id":"S058","target_id":"S041","kind":"supports"},{"source_id":"S059","target_id":"S041","kind":"supports"},{"source_id":"S060","target_id":"S041","kind":"supports"},{"source_id":"S061","target_id":"S041","kind":"supports"},{"source_id":"S062","target_id":"S041","kind":"supports"},{"source_id":"S063","target_id":"S041","kind":"supports"},{"source_id":"S007","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"}]}}
{"record_id":"2301.03139v1","paper":{"title":"A Newton-CG based augmented Lagrangian method for finding a second-order stationary point of nonconvex equality constrained optimization with complexity guarantees","authors":["Chuan He","Zhaosong Lu","Ting Kei Pong"],"arxiv_id":"2301.03139","version":"v1","journal":"SIAM Journal on Optimization","doi":"10.1137/22m1489824","abstract_url":"https://arxiv.org/abs/2301.03139v1","pdf_url":"https://arxiv.org/pdf/2301.03139v1"},"context":{"raw_tex":"In this paper we consider nonconvex equality constrained optimization problem\r\n\\beq\\label{model:equa-cnstr}\r\n\\min_{x\\in \\bR^n}\\ f(x)\\quad \\st\\ c(x)=0,\r\n\\eeq\n\nwhere $f:\\bR^n\\to\\bR$ and $c:\\bR^n\\to\\bR^m$ are twice continuously differentiable, and we assume that problem~\\eqref{model:equa-cnstr} has at least one optimal solution. Since \\eqref{model:equa-cnstr} is a nonconvex optimization problem, it may have many local but non-global minimizers and finding its global minimizer is generally NP-hard. A first-order stationary point (FOSP) of it is usually found in practice instead. Nevertheless,  a mere FOSP may sometimes not suit our needs and a  \\emph{second-order stationary point} (SOSP) needs to be sought. For example, in the context of linear semidefinite programming (SDP), a powerful approach to solving it is by solving an equivalent nonconvex equality constrained optimization problem \\cite{BuMo03-1,BuMo05-1}. It was shown in \\cite{BuMo05-1,BVB16} that under some mild conditions an SOSP of the latter problem can yield an optimal solution of  the linear SDP, while a mere FOSP generally cannot.  It is therefore important to find an SOSP of problem~\\eqref{model:equa-cnstr}.\r\n\n\nThroughout this paper, we let $\\bR^n$ denote the $n$-dimensional Euclidean space. We use $\\|\\cdot\\|$ to denote the Euclidean norm of a vector or the spectral norm of a matrix. For a real symmetric matrix $H$, we use $\\lambda_{\\min}(H)$ to denote its minimum eigenvalue.\r\nThe Euclidean ball centered at the origin with radius $R\\ge0$ is denoted by $\\mathcal{B}_R:=\\{x:\\|x\\|\\le R\\}$, and we use $\\Pi_{\\mathcal{B}_R}(v)$ to denote the Euclidean projection of a vector $v$ onto $\\mathcal{B}_R$. For a given finite set $\\cA$, we let $|\\cA|$ denote its cardinality. For any $s\\in \\mathbb{R}$, we let ${\\rm sgn}(s)$ be $1$ if $s \\ge 0$ and let it be $-1$ otherwise. In addition, $\\widetilde{\\cO}(\\cdot)$ represents $\\cO(\\cdot)$ with logarithmic terms omitted. %In addition, for any $s\\in \\mathbb{R}$, we let ${\\rm sgn}(s)$ be $1$ if $s \\ge 0$ and let it be $-1$ otherwise.\r\n\n\n$\\nabla c(x^*):=[\\nabla c_1(x^*) \\ \\nabla c_2(x^*) \\ \\cdots \\ \\nabla c_m(x^*)]$\r\n\n\n\\begin{equation}\\label{def:critical-cone} \r\n\\cC(x):=\\{d\\in\\bR^n:\\nabla c(x)^T d=0\\}.\r\n\\end{equation}\n\n\\begin{definition}[{{\\bf $\\epsilon_1$-first-order stationary point}}]\\label{def:FOSP}\r\nLet $\\epsilon_1>0$. We say that $x\\in\\bR^n$ is an $\\epsilon_1$-first-order stationary point {\\rm ($\\epsilon_1$-FOSP)} of problem \\eqref{model:equa-cnstr} if it, together with some $\\lambda\\in\\bR^m$, satisfies \r\n\\begin{equation}\\label{optcond:1st-equa-cnstr} \r\n\\|\\nabla f(x)+\\nabla c(x)\\lambda\\|\\le \\epsilon_1,\\quad \\|c(x)\\|\\le \\epsilon_1.\r\n\\end{equation}\r\n\\end{definition}\n\n\\begin{definition}[{{\\bf $(\\epsilon_1,\\epsilon_2)$-second-order stationary point}}]\\label{def:SOSP}\r\nLet $\\epsilon_1,\\epsilon_2>0$. We say that $x\\in\\bR^n$ is an $(\\epsilon_1,\r\n\\epsilon_2)$-second-order stationary point {\\rm (($\\epsilon_1,\r\n\\epsilon_2$)-SOSP)} of problem \\eqref{model:equa-cnstr} if it, together with some $\\lambda\\in\\bR^m$, satisfies \\eqref{optcond:1st-equa-cnstr} and additionally \r\n\\begin{equation}\\label{optcond:2nd-equa-cnstr} \r\nd^T\\left(\\nabla^2 f(x)+\\sum_{i=1}^m\\lambda_i\\nabla^2 c_i(x)\\right)d\\ge-\\epsilon_2\\|d\\|^2,\\quad \\forall d\\in\\cC(x),\r\n\\end{equation}\r\nwhere $\\cC(\\cdot)$ is defined as in \\eqref{def:critical-cone}.\r\n\\end{definition}\n\n\\begin{equation}\\label{unconstrained-prob} \r\n\\min_{x\\in\\bR^n}\\ F(x),\r\n\\end{equation}\n\n\\begin{assumption}\\label{asp:NCG-cmplxity}\r\n\\begin{enumerate}[{\\rm (a)}]\r\n\\item The level set $\\scL_F(u^0):=\\{x: F(x)\\le F(u^0)\\}$ is compact for some $u^0\\in \\bR^n$.%, where $z^0\\in\\bR^n$ is the initial iterate of the Newton-CG method in \\cite{RNW18}.\r\n\\item The function $F$ is twice Lipschitz continuously differentiable in\r\n%{\\color{black}some} convex open neighborhood $\\Omega$ of $\\{x|\\  F(x)\\le F(u^0)\\}$,\r\n%the convex hull of $\\Omega$, denoted by $\\conv(\\Omega)$,\r\na convex open neighborhood, denoted by $\\Omega$, of $\\scL_F(u^0)$,\r\nthat is, there exists $L_{H}^F>0$ such that\r\n\\begin{equation}\\label{F-Hess-Lip}\r\n\\|\\nabla^2 F(x)-\\nabla^2 F(y)\\|\\le L_{H}^F\\|x-y\\|,\\quad \\forall x,y\\in \\Omega.\r\n\\end{equation}\r\n\\end{enumerate}\r\n\\end{assumption}\n\nBy Assumption \\ref{asp:NCG-cmplxity}, there exist $\\Fl\\in \\bR$, $U_g^F>0$ and $U_H^F>0$ such that\r\n\\begin{equation}\\label{lwbd-Hgupbd}\r\nF(x)\\ge \\Fl,\\quad \\|\\nabla F(x)\\|\\le U_g^F,\\quad \\|\\nabla^2 F(x)\\|\\le U_H^F, \\quad \\forall x\\in\\scL_F(u^0).\r\n\\end{equation}\r\n\n\nThe first main component of the Newton-CG method in \\cite{RNW18} is a {\\it capped CG method} \\cite[Algorithm~1]{RNW18}, which is a modified CG method, for solving a possibly indefinite linear system \r\n\\begin{equation}\\label{indef-sys} \r\n(H+2\\varepsilon I)d=-g,\r\n\\end{equation}\n\nwhere $0\\neq g\\in\\bR^n$, $\\varepsilon>0$, and $H\\in\\bR^{n\\times n}$ is a symmetric matrix. This capped CG method terminates within a finite number of iterations. It outputs either an approximate solution $d$ to \\eqref{indef-sys} such that $\\|(H+2\\varepsilon I)d+g\\|\\le\\widehat{\\zeta}\\|g\\|$ and $d^THd\\ge-\\varepsilon\\|d\\|^2$ for some $\\widehat{\\zeta}\\in(0,1)$ or a sufficiently negative curvature direction $d$ of $H$ with $d^THd<-\\varepsilon\\|d\\|^2$. The second main component of the Newton-CG method in \\cite{RNW18} is a minimum eigenvalue oracle that either produces a sufficiently negative curvature direction $v$ of $H$ with $\\|v\\|=1$ and  $v^THv\\le-\\varepsilon/2$ or certifies that $\\lambda_{\\min}(H)\\ge-\\varepsilon$ holds with high probability. For ease of reference, we present these two components in Algorithms~\\ref{alg:capped-CG}  and \\ref{pro:meo} in Appendices~\\ref{appendix:capped-CG} and \\ref{appendix:meo}, respectively.\r\n\n\n\\begin{algorithm}[h]\r\n\\caption{A Newton-CG method for problem \\eqref{unconstrained-prob}}\r\n\\label{alg:NCG}\r\n{\\footnotesize\r\n\\begin{algorithmic}\r\n\\State \\noindent\\textit{Input}: Tolerances $\\epsilon_g,\\epsilon_H\\in(0,1)$, backtracking ratio $\\theta\\in(0,1)$, starting point $u^0$, CG-accuracy parameter $\\zeta\\in(0,1)$, line-search parameter $\\eta\\in(0,1)$, probability parameter $\\delta\\in(0,1)$.\\\\\r\nSet $x^0=u^0$;\r\n\\For{$t=0,1,2,\\ldots$}\r\n\\If{$\\|\\nabla F(x^t)\\|>\\epsilon_g$}\r\n\\State Call Algorithm~\\ref{alg:capped-CG} with $H=\\nabla^2 F(x^t)$, $\\varepsilon=\\epsilon_H$, $g=\\nabla F(x^t)$, accuracy parameter $\\zeta$, and $U=0$ to obtain outputs $d$,\r\n\\State d$\\_$type;\r\n\\If{d$\\_$type=NC}\r\n%\\State\r\n\\begin{equation}\\label{dk-nc}\r\nd^t\\leftarrow -\\sgn(d^T\\nabla F(x^t))\\frac{|d^T\\nabla^2 F(x^t) d|}{\\|d\\|^3}d;\r\n\\end{equation}\r\n\\Else\\ \\{d$\\_$type=SOL\\}\r\n%\\State\r\n\\begin{equation}\\label{dk-sol}\r\nd^t\\leftarrow d;\r\n\\end{equation}\r\n\\EndIf\r\n\\State Go to {\\bf Line Search};\r\n\\Else\r\n\\State Call Algorithm~\\ref{pro:meo} with $H=\\nabla^2 F(x^t)$, $\\varepsilon=\\epsilon_H$, and probability parameter $\\delta$;\r\n\\If{Algorithm~\\ref{pro:meo} certifies that $\\lambda_{\\min}(\\nabla^2 F(x^t))\\ge-\\epsilon_H$}\r\n\\State Output $x^t$ and terminate;\r\n\\Else\\ \\{Sufficiently negative curvature direction $v$ returned by Algorithm~\\ref{pro:meo}\\}\r\n\\State Set d$\\_$type=NC and\r\n\\begin{equation}\\label{dk-2nd-nc}\r\nd^t\\leftarrow -\\sgn(v^T\\nabla F(x^t))|v^T\\nabla^2F(x^t)v|v;\r\n\\end{equation}\r\n\\State Go to {\\bf Line Search};\r\n\\EndIf\r\n\\EndIf\r\n\\State{\\bf Line Search:}\r\n\\If{d$\\_$type=SOL}\r\n\\State Find $\\alpha_t=\\theta^{j_t}$, where $j_t$ is the smallest nonnegative integer $j$ such that\r\n\\begin{equation}\\label{ls-sol}\r\nF(x^t+\\theta^jd^t)<F(x^t)-\\eta\\epsilon_H\\theta^{2j}\\|d^t\\|^2;\r\n\\end{equation}\r\n\\Else\\ \\{d$\\_$type=NC\\}\r\n\\State Find $\\alpha_t=\\theta^{j_t}$, where $j_t$ is the smallest nonnegative integer $j$ such that\r\n\\begin{equation}\\label{ls-nc}\r\nF(x^t+\\theta^jd^t)<F(x^t)-\\eta\\theta^{2j}\\|d^t\\|^3/2;\r\n\\end{equation}\r\n\\EndIf\r\n\\State $x^{t+1}=x^t+\\alpha_td^t$;\r\n\\EndFor\r\n\\end{algorithmic}\r\n}\r\n\\end{algorithm}\n\n\\begin{assumption}\\label{asp:lowbd-knownfeas}\t\t\r\n\\begin{enumerate}[{\\rm (a)}]\r\n\\item An $\\epsilon_1/2$-approximately feasible point $z_{\\epsilon_1}$ of problem~\\eqref{model:equa-cnstr}, namely satisfying $\\|c(z_{\\epsilon_1})\\|\\le\\epsilon_1/2$, is known.\t\r\n\\item There exist constants $\\fh$, $\\fl$ and $\\gamma>0$, independent of $\\epsilon_1$ and $\\epsilon_2$,  such that\r\n\\begin{eqnarray}\r\n&& f(z_{\\epsilon_1})\\le\\fh, \\label{hbd}\\\\\r\n&& f(x)+\\gamma\\|c(x)\\|^2/2\\ge \\fl,\\quad \\forall x\\in\\bR^n, \\label{lbd}\r\n\\end{eqnarray}\r\n where $z_{\\epsilon_1}$ is given in (a).\r\n\\item %{  For any $\\epsilon \\in (0,1)$ and the corresponding $z^0$ from (b),}\r\nThere exist some $\\bdelta_f,\\bdelta_c>0$ such that the set\r\n\\begin{equation}\\label{nearly-feas-level-set}\r\n\\cS(\\bdelta_f,\\bdelta_c):=\\{x:f(x)\\le \\fh+\\bdelta_f,\\ \\|c(x)\\|\\le1+\\bdelta_c\\}\r\n\\end{equation}\r\n%is compact, { with $\\fh$ as in \\eqref{hbd}}. \r\nis compact with $\\fh$ given above. Also, $\\nabla^2 f$ and $\\nabla^2 c_i,\\ i=1,2,\\ldots,m$, are Lipschitz continuous in a convex open neighborhood, denoted by $\\Omega(\\bdelta_f,\\bdelta_c)$, of $\\cS(\\bdelta_f,\\bdelta_c)$.\r\n\\end{enumerate}\r\n\\end{assumption}\n\n\\begin{equation}\\label{model:equa-cnstr-pert}\r\n\\min_{x\\in \\bR^n}\\ f(x)\\quad \\st\\ \\tilde{c}(x):=c(x)-c(z_{\\epsilon_1})=0,\r\n\\end{equation}\n\n\\begin{equation}\\label{tL-tc}\r\n\\min_{x\\in\\bR^n} \\big\\{\\widetilde{\\cL}(x,\\lambda^k,\\rho_k):= f(x)+(\\lambda^k)^T\\tilde{c}(x)+\\rho_k\\|\\tilde{c}(x)\\|^2/2\\big\\}\r\n\\end{equation}\n\n\\begin{algorithm}[h]\r\n\\caption{A Newton-CG based AL method for problem\r\n\\eqref{model:equa-cnstr}}\r\n\\label{alg:2nd-order-AL-nonconvex}\r\n{\\small\r\nLet $\\gamma$ be given in Assumption \\ref{asp:lowbd-knownfeas}.\\\\\r\n\\noindent\\textbf{Input}: $\\epsilon_1,\\epsilon_2\\in(0,1)$, $\\Lambda>0$, $x^0\\in\\bR^n$, $\\lambda^0 \\in \\mathcal{B}_{\\Lambda}$, $\\rho_0>2\\gamma$, $\\alpha\\in(0,1)$, ${r}>1$, $\\delta\\in(0,1)$, and  $z_{\\epsilon_1}$ given in Assumption~\\ref{asp:lowbd-knownfeas}.\r\n\\begin{algorithmic}[1]\r\n\\State Set $k=0$.\r\n\\State Set $\\tau_k^g=\\max\\{\\epsilon_1, {r}^{k\\log\\epsilon_1/\\log 2}\\}$ and $\\tau_k^H=\\max\\{\\epsilon_2, {r}^{k\\log\\epsilon_2/\\log 2}\\}$.\r\n\\State  Call Algorithm \\ref{alg:NCG} with $\\epsilon_g=\\tau_k^g$, $\\epsilon_H=\\tau_k^H$ and $u^0=x^{k}_{\\init}$ to find an approximate solution $x^{k+1}$ to $\\min_{x\\in\\bR^n}\\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$ such that \r\n\\begin{eqnarray}\r\n&&\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k)\\le f(z_{\\epsilon_1}),\\ \\|\\nabla_x \\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k)\\|\\le\\tau_k^g,\\label{algstop:1st-order}\\\\\r\n&&\\lambda_{\\min}(\\nabla^2_{xx}\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k))\\ge-\\tau_k^H\\  \\text{with probability at least } 1-\\delta, \\label{algstop:2nd-order}\r\n\\end{eqnarray}\r\nwhere\r\n\\begin{equation}\\label{def:initial-iterate-subprob}\r\nx^k_{\\init}=\\left\\{\\begin{array}{ll}\r\nz_{\\epsilon_1}&\\text{if }\\ \\widetilde{\\cL}(x^k,\\lambda^k;\\rho_k)> f(z_{\\epsilon_1}),\\\\\r\nx^{k}&\\text{otherwise},\r\n\\end{array}\\right.\\quad\\text{for } k\\ge 0.\r\n\\end{equation}\\label{algstp:ALsubpb}\r\n\\State Set $\\tilde{\\lambda}^{k+1}=\\lambda^k+\\rho_k\\tilde{c}(x^{k+1})$.\r\n\\State If $\\tau_k^g\\le\\epsilon_1$, $\\tau_k^H\\le\\epsilon_2$ and $\\|c(x^{k+1})\\|\\le\\epsilon_1$, then output $(x^{k+1},\\tilde{\\lambda}^{k+1})$ and terminate.\\label{algstep:stop}\r\n\\State Set $\\lambda^{k+1}=\\Pi_{\\mathcal{B}_\\Lambda}(\\tilde{\\lambda}^{k+1})$.\\label{algstep:proj-multiplier}\r\n\\State If $k=0$ or $\\|\\tilde{c}(x^{k+1})\\|>\\alpha\\|\\tilde{c}(x^k)\\|$, set $\\rho_{k+1}={r}\\rho_k$. Otherwise, set $\\rho_{k+1}=\\rho_k$.\\label{altstep:penalty}\r\n\\State Set $k\\leftarrow k+1$, and go to step 2.\r\n\\end{algorithmic}\r\n}\r\n\\end{algorithm}\n\nIn the rest of this section, we study the worst-case complexity of Algorithm \\ref{alg:2nd-order-AL-nonconvex}. Since our method has two nested loops, particularly,  outer loops executed by the AL method and inner loops executed by the Newton-CG method for solving the AL subproblems, we consider the following measures of complexity for Algorithm \\ref{alg:2nd-order-AL-nonconvex}.\r\n\\begin{itemize}\r\n\\item \\emph{Outer iteration complexity}, which measures the number of outer iterations of Algorithm \\ref{alg:2nd-order-AL-nonconvex};\r\n\\item  \\emph{Total inner iteration complexity}, which measures the total number of iterations of the Newton-CG method that are performed in Algorithm \\ref{alg:2nd-order-AL-nonconvex};\r\n\\item \\emph{Operation complexity}, which measures the total number of matrix-vector products involving the Hessian of the augmented Lagrangian function that are evaluated in Algorithm \\ref{alg:2nd-order-AL-nonconvex}.\r\n\\end{itemize}\r\n\n\nFor notational convenience, we rewrite $(\\tau_k^g,\\tau_k^H)$ arising in Algorithm \\ref{alg:2nd-order-AL-nonconvex} as\r\n\\begin{equation}\\label{omega-tolerance}\r\n(\\tau_k^g,\\tau_k^H)=(\\max\\{\\epsilon_1,\\omega_1^k\\},\\max\\{\\epsilon_2,\\omega_2^k\\})\\ \\mbox{ with }\\ (\\omega_1,\\omega_2):=({r}^{\\log\\epsilon_1/\\log 2},{r}^{\\log\\epsilon_2/\\log 2}),\r\n\\end{equation}\r\nwhere $\\epsilon_1$, $\\epsilon_2$ and $r$ are the input parameters of Algorithm \\ref{alg:2nd-order-AL-nonconvex}. Since $r>1$ and $\\epsilon_1,\\epsilon_2\\in(0,1)$, it is not hard to verify that $\\omega_1,\\omega_2\\in(0,1)$.\r\nAlso, we introduce the following quantity that will be used frequently later:\r\n\\begin{equation}\r\nK_{\\epsilon_1}:= \\left\\lceil \\min\\{k\\ge 0:\\omega_1^{k}\\le\\epsilon_1\\} \\right\\rceil=\\left\\lceil\\log\\epsilon_1/\\log\\omega_1\\right\\rceil. \\label{T-epsilon-g}\r\n\\end{equation}\r\n\n\n\\begin{remark}[{{\\bf Upper bounds for $\\overline{K}_{\\epsilon_1}$ and $\\{\\rho_k\\}$}}]\\label{order-out-itera-penalty}\r\nAs observed from Theorem \\ref{thm:out-itr-cmplxity-1}, the number of outer iterations of Algorithm \\ref{alg:2nd-order-AL-nonconvex} for finding a stochastic  $(\\epsilon_1,\r\n\\epsilon_2)$-SOSP of problem \\eqref{model:equa-cnstr} is $\\overline{K}_{\\epsilon_1}+1$, which is at most of $\\cO(|\\log\\epsilon_1|^2)$. In addition, the penalty parameters $\\{\\rho_k\\}$ generated in this algorithm are at most of $\\cO(\\epsilon_1^{-2})$.\r\n\\end{remark}\n\nOne can observe from Assumption~\\ref{asp:NCG-cmplxity}(b) that for all $x$ and $y\\in \\Omega$,\r\n\\begin{eqnarray}\r\n&&\\ \\ \\ \\|\\nabla F(y)-\\nabla F(x)-\\nabla^2 F(x)(y-x)\\|\\le L_H^F\\|y-x\\|^2/2, \\label{apx-nxt-grad}\\\\\r\n&&\\ \\ \\ F(y)\\le F(x) + \\nabla F(x)^T(y-x) + (y-x)^T\\nabla^2 F(x) (y-x)/2 +  L_H^F\\|y-x\\|^3/6. \\label{desc-ineq}\r\n\\end{eqnarray}\r\n\n\n\\begin{algorithm}[h]\r\n\\caption{A capped conjugate gradient method}\r\n\\label{alg:capped-CG}\r\n{\\footnotesize\r\n\\begin{algorithmic}\r\n\\State \\noindent\\textit{Inputs}: symmetric matrix $H\\in\\bR^{n\\times n}$, vector $g\\neq0$, damping parameter $\\varepsilon\\in(0,1)$, desired relative accuracy $\\zeta\\in(0,1)$.\r\n\\State \\textit{Optional input:} scalar $U\\ge0$ (set to $0$ if not provided).\r\n\\State \\textit{Outputs:} d$\\_$type, $d$.\r\n\\State \\textit{Secondary outputs:} final values of $U,\\kappa,\\widehat{\\zeta},\\tau,$ and $T$.\r\n\\State Set\r\n\\begin{equation*}\r\n\\textstyle\r\n\\bar{H}:=H+2\\varepsilon I,\\quad \\kappa:=\\frac{U+2\\varepsilon}{\\varepsilon},\\quad\\widehat{\\zeta}:=\\frac{\\zeta}{3\\kappa},\\quad\\tau:=\\frac{\\sqrt{\\kappa}}{\\sqrt{\\kappa}+1},\\quad T:=\\frac{4\\kappa^4}{(1-\\sqrt{\\tau})^2},\r\n\\end{equation*}\r\n$y^0\\leftarrow 0,r^0\\leftarrow g,p^0\\leftarrow -g, j\\leftarrow 0$.\r\n\\If {$(p^0)^T \\bar{H}p^0<\\varepsilon\\|p^0\\|^2$}\r\n\\State Set $d\\leftarrow p^0$ and terminate with d$\\_$type = NC;\r\n\\ElsIf {\\ $\\|Hp^0\\|>U\\|p^0\\|\\ $}\r\n\\State Set $U\\leftarrow\\|Hp^0\\|/\\|p^0\\|$ and update $\\kappa,\\widehat{\\zeta},\\tau, T$ accordingly;\r\n\\EndIf\r\n\\While{TRUE}\r\n\\State $\\alpha_j\\leftarrow (r^j)^T r^j/(p^j)^T\\bar{H}p^j$; \\{Begin Standard CG Operations\\}\r\n\\State $y^{j+1}\\leftarrow y^j+\\alpha_jp^j$;\r\n\\State $r^{j+1}\\leftarrow r^j+\\alpha_j\\bar{H}p^j$;\r\n\\State $\\beta_{j+1}\\leftarrow\\|r^{j+1}\\|^2/\\|r^j\\|^2$;\r\n\\State $p^{j+1}\\leftarrow-r^{j+1}+\\beta_{j+1}p^j$; \\{End Standard CG Operations\\}\r\n\\State $j\\leftarrow j+1$;\r\n\\If {$\\|Hp^j\\|>U\\|p^j\\|$}\r\n\\State Set $U\\leftarrow\\|Hp^j\\|/\\|p^j\\|$ and update $\\kappa,\\widehat{\\zeta},\\tau,T$ accordingly;\r\n\\EndIf\r\n\\If {\\ $\\|Hy^j\\|>U\\|y^j\\|\\ $}\r\n\\State Set $U\\leftarrow\\|Hy^j\\|/\\|y^j\\|$ and update $\\kappa,\\widehat{\\zeta},\\tau,T$ accordingly;\r\n\\EndIf\r\n\\If {\\ $\\|Hr^j\\|>U\\|r^j\\|\\ $}\r\n\\State Set $U\\leftarrow\\|Hr^j\\|/\\|r^j\\|$ and update $\\kappa,\\widehat{\\zeta},\\tau,T$ accordingly;\r\n\\EndIf\r\n\\If {$(y^j)^T\\bar{H}y^j<\\varepsilon\\|y^j\\|^2$}\r\n\\State Set $d\\leftarrow y^j$ and terminate with d$\\_$type = NC;\r\n\\ElsIf {\\ $\\|r^j\\|\\le\\widehat{\\zeta}\\|r^0\\|$}\r\n\\State Set $d\\leftarrow y^j$ and terminate with d$\\_$type = SOL;\r\n\\ElsIf{\\ $(p^j)^T\\bar{H}p^j<\\varepsilon\\|p^j\\|^2$}\r\n\\State Set $d\\leftarrow p^j$ and terminate with d$\\_$type = NC;\r\n\\ElsIf {\\ $\\|r^j\\|>\\sqrt{T}\\tau^{j/2}\\|r^0\\| $}\r\n\\State Compute $\\alpha_j, y^{j+1}$ as in the main loop above;\r\n\\State Find $i\\in\\{0,\\ldots,j-1\\}$ such that\r\n\\[\r\n(y^{j+1}-y^i)^T\\bar{H}(y^{j+1}-y^i)<\\varepsilon\\|y^{j+1}-y^i\\|^2;\r\n\\]\r\n\\State Set $d\\leftarrow y^{j+1}-y^i$ and terminate with d$\\_$type = NC;\r\n\\EndIf\r\n\\EndWhile\r\n\\end{algorithmic}\r\n}\r\n\\end{algorithm}\n\n\\begin{algorithm}[h]\r\n\\caption{A randomized Lanczos based minimum eigenvalue oracle}\r\n\\label{pro:meo}\r\n\\noindent\\textit{Input}: symmetric matrix $H\\in\\bR^{n\\times n}$, tolerance $\\varepsilon>0$, and probability parameter $\\delta\\in(0,1)$.\\\\\r\n\\noindent\\textit{Output}: a sufficiently negative curvature direction $v$ satisfying $v^THv\\le-\\varepsilon/2$ and $\\|v\\|=1$; or a certificate that $\\lambda_{\\min}(H)\\ge-\\varepsilon$ with probability at least  $1-\\delta$.\\\\\r\nApply the Lanczos method \\cite{KW92LR} to estimate $\\lambda_{\\min}(H)$ starting with a random vector uniformly generated on the unit sphere, and run it for at most \r\n\\begin{equation}\\label{N-iter} \r\nN(\\varepsilon,\\delta):=\\min\\left\\{n,1+\\left\\lceil\\frac{\\ln(2.75n/\\delta^2)}{2}\\sqrt{\\frac{\\|H\\|}{\\varepsilon}}\\right\\rceil\\right\\}\r\n\\end{equation}\r\niterations. If a unit vector $v$ with $v^THv \\le -\\varepsilon/2$ is found at some iteration, terminate immediately and return $v$.\r\n\\end{algorithm}","units":[{"start":10189,"end":10345,"start_line":187,"end_line":190,"raw_tex":"In this paper we consider nonconvex equality constrained optimization problem\r\n\\beq\\label{model:equa-cnstr}\r\n\\min_{x\\in \\bR^n}\\ f(x)\\quad \\st\\ c(x)=0,\r\n\\eeq","sha256_utf8":"0d0f6c0525ca53820a91acaa5ea9b152ed4f80114628b9bca25cbef3ad0c463e"},{"start":10347,"end":11379,"start_line":191,"end_line":191,"raw_tex":"where $f:\\bR^n\\to\\bR$ and $c:\\bR^n\\to\\bR^m$ are twice continuously differentiable, and we assume that problem~\\eqref{model:equa-cnstr} has at least one optimal solution. Since \\eqref{model:equa-cnstr} is a nonconvex optimization problem, it may have many local but non-global minimizers and finding its global minimizer is generally NP-hard. A first-order stationary point (FOSP) of it is usually found in practice instead. Nevertheless,  a mere FOSP may sometimes not suit our needs and a  \\emph{second-order stationary point} (SOSP) needs to be sought. For example, in the context of linear semidefinite programming (SDP), a powerful approach to solving it is by solving an equivalent nonconvex equality constrained optimization problem \\cite{BuMo03-1,BuMo05-1}. It was shown in \\cite{BuMo05-1,BVB16} that under some mild conditions an SOSP of the latter problem can yield an optimal solution of  the linear SDP, while a mere FOSP generally cannot.  It is therefore important to find an SOSP of problem~\\eqref{model:equa-cnstr}.\r\n","sha256_utf8":"ec25db1aa2183305549d74c93d7a2f47a09cb54ee5eda53910304f7d5fba2f98"},{"start":29839,"end":30706,"start_line":282,"end_line":283,"raw_tex":"Throughout this paper, we let $\\bR^n$ denote the $n$-dimensional Euclidean space. We use $\\|\\cdot\\|$ to denote the Euclidean norm of a vector or the spectral norm of a matrix. For a real symmetric matrix $H$, we use $\\lambda_{\\min}(H)$ to denote its minimum eigenvalue.\r\nThe Euclidean ball centered at the origin with radius $R\\ge0$ is denoted by $\\mathcal{B}_R:=\\{x:\\|x\\|\\le R\\}$, and we use $\\Pi_{\\mathcal{B}_R}(v)$ to denote the Euclidean projection of a vector $v$ onto $\\mathcal{B}_R$. For a given finite set $\\cA$, we let $|\\cA|$ denote its cardinality. For any $s\\in \\mathbb{R}$, we let ${\\rm sgn}(s)$ be $1$ if $s \\ge 0$ and let it be $-1$ otherwise. In addition, $\\widetilde{\\cO}(\\cdot)$ represents $\\cO(\\cdot)$ with logarithmic terms omitted. %In addition, for any $s\\in \\mathbb{R}$, we let ${\\rm sgn}(s)$ be $1$ if $s \\ge 0$ and let it be $-1$ otherwise.\r\n","sha256_utf8":"c73ded9ca3704f2854921d298aea325f83da5d58140eee85e6ed54e9353455da"},{"start":30861,"end":30942,"start_line":286,"end_line":286,"raw_tex":"$\\nabla c(x^*):=[\\nabla c_1(x^*) \\ \\nabla c_2(x^*) \\ \\cdots \\ \\nabla c_m(x^*)]$\r\n","sha256_utf8":"fb7b8bb32f90b7d614e244d75b0244ac29a397a6bbbbde21a3583031423bb27b"},{"start":31305,"end":31405,"start_line":293,"end_line":295,"raw_tex":"\\begin{equation}\\label{def:critical-cone} \r\n\\cC(x):=\\{d\\in\\bR^n:\\nabla c(x)^T d=0\\}.\r\n\\end{equation}","sha256_utf8":"3c4082685c81665869d1685b07deef9f286fed55d9eb968c7dbbbd5bcafe5182"},{"start":32420,"end":32879,"start_line":301,"end_line":306,"raw_tex":"\\begin{definition}[{{\\bf $\\epsilon_1$-first-order stationary point}}]\\label{def:FOSP}\r\nLet $\\epsilon_1>0$. We say that $x\\in\\bR^n$ is an $\\epsilon_1$-first-order stationary point {\\rm ($\\epsilon_1$-FOSP)} of problem \\eqref{model:equa-cnstr} if it, together with some $\\lambda\\in\\bR^m$, satisfies \r\n\\begin{equation}\\label{optcond:1st-equa-cnstr} \r\n\\|\\nabla f(x)+\\nabla c(x)\\lambda\\|\\le \\epsilon_1,\\quad \\|c(x)\\|\\le \\epsilon_1.\r\n\\end{equation}\r\n\\end{definition}","sha256_utf8":"3c4f8291bd638421f5f6354ac0ea0d37bbbefdbcc781b7c6dfafbace32a3eaf9"},{"start":32884,"end":33547,"start_line":308,"end_line":316,"raw_tex":"\\begin{definition}[{{\\bf $(\\epsilon_1,\\epsilon_2)$-second-order stationary point}}]\\label{def:SOSP}\r\nLet $\\epsilon_1,\\epsilon_2>0$. We say that $x\\in\\bR^n$ is an $(\\epsilon_1,\r\n\\epsilon_2)$-second-order stationary point {\\rm (($\\epsilon_1,\r\n\\epsilon_2$)-SOSP)} of problem \\eqref{model:equa-cnstr} if it, together with some $\\lambda\\in\\bR^m$, satisfies \\eqref{optcond:1st-equa-cnstr} and additionally \r\n\\begin{equation}\\label{optcond:2nd-equa-cnstr} \r\nd^T\\left(\\nabla^2 f(x)+\\sum_{i=1}^m\\lambda_i\\nabla^2 c_i(x)\\right)d\\ge-\\epsilon_2\\|d\\|^2,\\quad \\forall d\\in\\cC(x),\r\n\\end{equation}\r\nwhere $\\cC(\\cdot)$ is defined as in \\eqref{def:critical-cone}.\r\n\\end{definition}","sha256_utf8":"d7c04b559ea43b9f1872cabf9c5be36cab3b7ec1c7361d867a968972309e1ba2"},{"start":33963,"end":34047,"start_line":322,"end_line":324,"raw_tex":"\\begin{equation}\\label{unconstrained-prob} \r\n\\min_{x\\in\\bR^n}\\ F(x),\r\n\\end{equation}","sha256_utf8":"6984bb855072f3ff943fefd64437b787e2274123ce3b03f935cbce0450beade1"},{"start":34112,"end":34863,"start_line":327,"end_line":339,"raw_tex":"\\begin{assumption}\\label{asp:NCG-cmplxity}\r\n\\begin{enumerate}[{\\rm (a)}]\r\n\\item The level set $\\scL_F(u^0):=\\{x: F(x)\\le F(u^0)\\}$ is compact for some $u^0\\in \\bR^n$.%, where $z^0\\in\\bR^n$ is the initial iterate of the Newton-CG method in \\cite{RNW18}.\r\n\\item The function $F$ is twice Lipschitz continuously differentiable in\r\n%{\\color{black}some} convex open neighborhood $\\Omega$ of $\\{x|\\  F(x)\\le F(u^0)\\}$,\r\n%the convex hull of $\\Omega$, denoted by $\\conv(\\Omega)$,\r\na convex open neighborhood, denoted by $\\Omega$, of $\\scL_F(u^0)$,\r\nthat is, there exists $L_{H}^F>0$ such that\r\n\\begin{equation}\\label{F-Hess-Lip}\r\n\\|\\nabla^2 F(x)-\\nabla^2 F(y)\\|\\le L_{H}^F\\|x-y\\|,\\quad \\forall x,y\\in \\Omega.\r\n\\end{equation}\r\n\\end{enumerate}\r\n\\end{assumption}","sha256_utf8":"c6204f4910089cd9e9856894698f3c03766bee3a3ff1a1a8208cfef57f5c7c9f"},{"start":34867,"end":35128,"start_line":341,"end_line":344,"raw_tex":"By Assumption \\ref{asp:NCG-cmplxity}, there exist $\\Fl\\in \\bR$, $U_g^F>0$ and $U_H^F>0$ such that\r\n\\begin{equation}\\label{lwbd-Hgupbd}\r\nF(x)\\ge \\Fl,\\quad \\|\\nabla F(x)\\|\\le U_g^F,\\quad \\|\\nabla^2 F(x)\\|\\le U_H^F, \\quad \\forall x\\in\\scL_F(u^0).\r\n\\end{equation}\r\n","sha256_utf8":"9b0d0c24fd40c80e521968f37bbf9e95b45840aa58a745aeb3320c034b850319"},{"start":39469,"end":39745,"start_line":368,"end_line":371,"raw_tex":"The first main component of the Newton-CG method in \\cite{RNW18} is a {\\it capped CG method} \\cite[Algorithm~1]{RNW18}, which is a modified CG method, for solving a possibly indefinite linear system \r\n\\begin{equation}\\label{indef-sys} \r\n(H+2\\varepsilon I)d=-g,\r\n\\end{equation}","sha256_utf8":"6d88d1db14a9d2becedf0a88781065f1ea27b6d390c43581957f03723739171a"},{"start":39747,"end":40684,"start_line":372,"end_line":372,"raw_tex":"where $0\\neq g\\in\\bR^n$, $\\varepsilon>0$, and $H\\in\\bR^{n\\times n}$ is a symmetric matrix. This capped CG method terminates within a finite number of iterations. It outputs either an approximate solution $d$ to \\eqref{indef-sys} such that $\\|(H+2\\varepsilon I)d+g\\|\\le\\widehat{\\zeta}\\|g\\|$ and $d^THd\\ge-\\varepsilon\\|d\\|^2$ for some $\\widehat{\\zeta}\\in(0,1)$ or a sufficiently negative curvature direction $d$ of $H$ with $d^THd<-\\varepsilon\\|d\\|^2$. The second main component of the Newton-CG method in \\cite{RNW18} is a minimum eigenvalue oracle that either produces a sufficiently negative curvature direction $v$ of $H$ with $\\|v\\|=1$ and  $v^THv\\le-\\varepsilon/2$ or certifies that $\\lambda_{\\min}(H)\\ge-\\varepsilon$ holds with high probability. For ease of reference, we present these two components in Algorithms~\\ref{alg:capped-CG}  and \\ref{pro:meo} in Appendices~\\ref{appendix:capped-CG} and \\ref{appendix:meo}, respectively.\r\n","sha256_utf8":"386ca237be8a6cbac5406fcf007ffe9001c74190b7f71267e986eef35982e64d"},{"start":40686,"end":42771,"start_line":374,"end_line":425,"raw_tex":"\\begin{algorithm}[h]\r\n\\caption{A Newton-CG method for problem \\eqref{unconstrained-prob}}\r\n\\label{alg:NCG}\r\n{\\footnotesize\r\n\\begin{algorithmic}\r\n\\State \\noindent\\textit{Input}: Tolerances $\\epsilon_g,\\epsilon_H\\in(0,1)$, backtracking ratio $\\theta\\in(0,1)$, starting point $u^0$, CG-accuracy parameter $\\zeta\\in(0,1)$, line-search parameter $\\eta\\in(0,1)$, probability parameter $\\delta\\in(0,1)$.\\\\\r\nSet $x^0=u^0$;\r\n\\For{$t=0,1,2,\\ldots$}\r\n\\If{$\\|\\nabla F(x^t)\\|>\\epsilon_g$}\r\n\\State Call Algorithm~\\ref{alg:capped-CG} with $H=\\nabla^2 F(x^t)$, $\\varepsilon=\\epsilon_H$, $g=\\nabla F(x^t)$, accuracy parameter $\\zeta$, and $U=0$ to obtain outputs $d$,\r\n\\State d$\\_$type;\r\n\\If{d$\\_$type=NC}\r\n%\\State\r\n\\begin{equation}\\label{dk-nc}\r\nd^t\\leftarrow -\\sgn(d^T\\nabla F(x^t))\\frac{|d^T\\nabla^2 F(x^t) d|}{\\|d\\|^3}d;\r\n\\end{equation}\r\n\\Else\\ \\{d$\\_$type=SOL\\}\r\n%\\State\r\n\\begin{equation}\\label{dk-sol}\r\nd^t\\leftarrow d;\r\n\\end{equation}\r\n\\EndIf\r\n\\State Go to {\\bf Line Search};\r\n\\Else\r\n\\State Call Algorithm~\\ref{pro:meo} with $H=\\nabla^2 F(x^t)$, $\\varepsilon=\\epsilon_H$, and probability parameter $\\delta$;\r\n\\If{Algorithm~\\ref{pro:meo} certifies that $\\lambda_{\\min}(\\nabla^2 F(x^t))\\ge-\\epsilon_H$}\r\n\\State Output $x^t$ and terminate;\r\n\\Else\\ \\{Sufficiently negative curvature direction $v$ returned by Algorithm~\\ref{pro:meo}\\}\r\n\\State Set d$\\_$type=NC and\r\n\\begin{equation}\\label{dk-2nd-nc}\r\nd^t\\leftarrow -\\sgn(v^T\\nabla F(x^t))|v^T\\nabla^2F(x^t)v|v;\r\n\\end{equation}\r\n\\State Go to {\\bf Line Search};\r\n\\EndIf\r\n\\EndIf\r\n\\State{\\bf Line Search:}\r\n\\If{d$\\_$type=SOL}\r\n\\State Find $\\alpha_t=\\theta^{j_t}$, where $j_t$ is the smallest nonnegative integer $j$ such that\r\n\\begin{equation}\\label{ls-sol}\r\nF(x^t+\\theta^jd^t)<F(x^t)-\\eta\\epsilon_H\\theta^{2j}\\|d^t\\|^2;\r\n\\end{equation}\r\n\\Else\\ \\{d$\\_$type=NC\\}\r\n\\State Find $\\alpha_t=\\theta^{j_t}$, where $j_t$ is the smallest nonnegative integer $j$ such that\r\n\\begin{equation}\\label{ls-nc}\r\nF(x^t+\\theta^jd^t)<F(x^t)-\\eta\\theta^{2j}\\|d^t\\|^3/2;\r\n\\end{equation}\r\n\\EndIf\r\n\\State $x^{t+1}=x^t+\\alpha_td^t$;\r\n\\EndFor\r\n\\end{algorithmic}\r\n}\r\n\\end{algorithm}","sha256_utf8":"f1b083d5e3850c5bf6ea60ca44648301e635f88a5f99557a86c09cfc21beea82"},{"start":48220,"end":49366,"start_line":481,"end_line":498,"raw_tex":"\\begin{assumption}\\label{asp:lowbd-knownfeas}\t\t\r\n\\begin{enumerate}[{\\rm (a)}]\r\n\\item An $\\epsilon_1/2$-approximately feasible point $z_{\\epsilon_1}$ of problem~\\eqref{model:equa-cnstr}, namely satisfying $\\|c(z_{\\epsilon_1})\\|\\le\\epsilon_1/2$, is known.\t\r\n\\item There exist constants $\\fh$, $\\fl$ and $\\gamma>0$, independent of $\\epsilon_1$ and $\\epsilon_2$,  such that\r\n\\begin{eqnarray}\r\n&& f(z_{\\epsilon_1})\\le\\fh, \\label{hbd}\\\\\r\n&& f(x)+\\gamma\\|c(x)\\|^2/2\\ge \\fl,\\quad \\forall x\\in\\bR^n, \\label{lbd}\r\n\\end{eqnarray}\r\n where $z_{\\epsilon_1}$ is given in (a).\r\n\\item %{  For any $\\epsilon \\in (0,1)$ and the corresponding $z^0$ from (b),}\r\nThere exist some $\\bdelta_f,\\bdelta_c>0$ such that the set\r\n\\begin{equation}\\label{nearly-feas-level-set}\r\n\\cS(\\bdelta_f,\\bdelta_c):=\\{x:f(x)\\le \\fh+\\bdelta_f,\\ \\|c(x)\\|\\le1+\\bdelta_c\\}\r\n\\end{equation}\r\n%is compact, { with $\\fh$ as in \\eqref{hbd}}. \r\nis compact with $\\fh$ given above. Also, $\\nabla^2 f$ and $\\nabla^2 c_i,\\ i=1,2,\\ldots,m$, are Lipschitz continuous in a convex open neighborhood, denoted by $\\Omega(\\bdelta_f,\\bdelta_c)$, of $\\cS(\\bdelta_f,\\bdelta_c)$.\r\n\\end{enumerate}\r\n\\end{assumption}","sha256_utf8":"2090a19b451da2b6ad7b1df1fdee680b8857f7e5b1ab93a8f15eeec0d4887aad"},{"start":53806,"end":53942,"start_line":519,"end_line":521,"raw_tex":"\\begin{equation}\\label{model:equa-cnstr-pert}\r\n\\min_{x\\in \\bR^n}\\ f(x)\\quad \\st\\ \\tilde{c}(x):=c(x)-c(z_{\\epsilon_1})=0,\r\n\\end{equation}","sha256_utf8":"eb2fae3cfb2deaadc996cd9a5c74c76c5b8349d38ff0aeb1239a353fdb3de556"},{"start":54244,"end":54415,"start_line":523,"end_line":525,"raw_tex":"\\begin{equation}\\label{tL-tc}\r\n\\min_{x\\in\\bR^n} \\big\\{\\widetilde{\\cL}(x,\\lambda^k,\\rho_k):= f(x)+(\\lambda^k)^T\\tilde{c}(x)+\\rho_k\\|\\tilde{c}(x)\\|^2/2\\big\\}\r\n\\end{equation}","sha256_utf8":"78330de4c947d372f5df356a78ce625dfb448b058d04b2c913ca519cb83f11f7"},{"start":55931,"end":58049,"start_line":531,"end_line":560,"raw_tex":"\\begin{algorithm}[h]\r\n\\caption{A Newton-CG based AL method for problem\r\n\\eqref{model:equa-cnstr}}\r\n\\label{alg:2nd-order-AL-nonconvex}\r\n{\\small\r\nLet $\\gamma$ be given in Assumption \\ref{asp:lowbd-knownfeas}.\\\\\r\n\\noindent\\textbf{Input}: $\\epsilon_1,\\epsilon_2\\in(0,1)$, $\\Lambda>0$, $x^0\\in\\bR^n$, $\\lambda^0 \\in \\mathcal{B}_{\\Lambda}$, $\\rho_0>2\\gamma$, $\\alpha\\in(0,1)$, ${r}>1$, $\\delta\\in(0,1)$, and  $z_{\\epsilon_1}$ given in Assumption~\\ref{asp:lowbd-knownfeas}.\r\n\\begin{algorithmic}[1]\r\n\\State Set $k=0$.\r\n\\State Set $\\tau_k^g=\\max\\{\\epsilon_1, {r}^{k\\log\\epsilon_1/\\log 2}\\}$ and $\\tau_k^H=\\max\\{\\epsilon_2, {r}^{k\\log\\epsilon_2/\\log 2}\\}$.\r\n\\State  Call Algorithm \\ref{alg:NCG} with $\\epsilon_g=\\tau_k^g$, $\\epsilon_H=\\tau_k^H$ and $u^0=x^{k}_{\\init}$ to find an approximate solution $x^{k+1}$ to $\\min_{x\\in\\bR^n}\\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$ such that \r\n\\begin{eqnarray}\r\n&&\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k)\\le f(z_{\\epsilon_1}),\\ \\|\\nabla_x \\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k)\\|\\le\\tau_k^g,\\label{algstop:1st-order}\\\\\r\n&&\\lambda_{\\min}(\\nabla^2_{xx}\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k))\\ge-\\tau_k^H\\  \\text{with probability at least } 1-\\delta, \\label{algstop:2nd-order}\r\n\\end{eqnarray}\r\nwhere\r\n\\begin{equation}\\label{def:initial-iterate-subprob}\r\nx^k_{\\init}=\\left\\{\\begin{array}{ll}\r\nz_{\\epsilon_1}&\\text{if }\\ \\widetilde{\\cL}(x^k,\\lambda^k;\\rho_k)> f(z_{\\epsilon_1}),\\\\\r\nx^{k}&\\text{otherwise},\r\n\\end{array}\\right.\\quad\\text{for } k\\ge 0.\r\n\\end{equation}\\label{algstp:ALsubpb}\r\n\\State Set $\\tilde{\\lambda}^{k+1}=\\lambda^k+\\rho_k\\tilde{c}(x^{k+1})$.\r\n\\State If $\\tau_k^g\\le\\epsilon_1$, $\\tau_k^H\\le\\epsilon_2$ and $\\|c(x^{k+1})\\|\\le\\epsilon_1$, then output $(x^{k+1},\\tilde{\\lambda}^{k+1})$ and terminate.\\label{algstep:stop}\r\n\\State Set $\\lambda^{k+1}=\\Pi_{\\mathcal{B}_\\Lambda}(\\tilde{\\lambda}^{k+1})$.\\label{algstep:proj-multiplier}\r\n\\State If $k=0$ or $\\|\\tilde{c}(x^{k+1})\\|>\\alpha\\|\\tilde{c}(x^k)\\|$, set $\\rho_{k+1}={r}\\rho_k$. Otherwise, set $\\rho_{k+1}=\\rho_k$.\\label{altstep:penalty}\r\n\\State Set $k\\leftarrow k+1$, and go to step 2.\r\n\\end{algorithmic}\r\n}\r\n\\end{algorithm}","sha256_utf8":"71b60c2a2e209a4983f0997bc9b97ea8791132daa930ede6535dedb4cf1112d5"},{"start":67364,"end":68317,"start_line":624,"end_line":629,"raw_tex":"In the rest of this section, we study the worst-case complexity of Algorithm \\ref{alg:2nd-order-AL-nonconvex}. Since our method has two nested loops, particularly,  outer loops executed by the AL method and inner loops executed by the Newton-CG method for solving the AL subproblems, we consider the following measures of complexity for Algorithm \\ref{alg:2nd-order-AL-nonconvex}.\r\n\\begin{itemize}\r\n\\item \\emph{Outer iteration complexity}, which measures the number of outer iterations of Algorithm \\ref{alg:2nd-order-AL-nonconvex};\r\n\\item  \\emph{Total inner iteration complexity}, which measures the total number of iterations of the Newton-CG method that are performed in Algorithm \\ref{alg:2nd-order-AL-nonconvex};\r\n\\item \\emph{Operation complexity}, which measures the total number of matrix-vector products involving the Hessian of the augmented Lagrangian function that are evaluated in Algorithm \\ref{alg:2nd-order-AL-nonconvex}.\r\n\\end{itemize}\r\n","sha256_utf8":"d8d63658f7a4c26ece661626521fb55c8eef9952c39cad938ed1a1d35592c9d9"},{"start":68602,"end":69448,"start_line":636,"end_line":644,"raw_tex":"For notational convenience, we rewrite $(\\tau_k^g,\\tau_k^H)$ arising in Algorithm \\ref{alg:2nd-order-AL-nonconvex} as\r\n\\begin{equation}\\label{omega-tolerance}\r\n(\\tau_k^g,\\tau_k^H)=(\\max\\{\\epsilon_1,\\omega_1^k\\},\\max\\{\\epsilon_2,\\omega_2^k\\})\\ \\mbox{ with }\\ (\\omega_1,\\omega_2):=({r}^{\\log\\epsilon_1/\\log 2},{r}^{\\log\\epsilon_2/\\log 2}),\r\n\\end{equation}\r\nwhere $\\epsilon_1$, $\\epsilon_2$ and $r$ are the input parameters of Algorithm \\ref{alg:2nd-order-AL-nonconvex}. Since $r>1$ and $\\epsilon_1,\\epsilon_2\\in(0,1)$, it is not hard to verify that $\\omega_1,\\omega_2\\in(0,1)$.\r\nAlso, we introduce the following quantity that will be used frequently later:\r\n\\begin{equation}\r\nK_{\\epsilon_1}:= \\left\\lceil \\min\\{k\\ge 0:\\omega_1^{k}\\le\\epsilon_1\\} \\right\\rceil=\\left\\lceil\\log\\epsilon_1/\\log\\omega_1\\right\\rceil. \\label{T-epsilon-g}\r\n\\end{equation}\r\n","sha256_utf8":"7ea8b2257d245867ded47d90698cc6e4dab480ac307e7c3fe4723bcb7393613e"},{"start":71680,"end":72235,"start_line":668,"end_line":671,"raw_tex":"\\begin{remark}[{{\\bf Upper bounds for $\\overline{K}_{\\epsilon_1}$ and $\\{\\rho_k\\}$}}]\\label{order-out-itera-penalty}\r\nAs observed from Theorem \\ref{thm:out-itr-cmplxity-1}, the number of outer iterations of Algorithm \\ref{alg:2nd-order-AL-nonconvex} for finding a stochastic  $(\\epsilon_1,\r\n\\epsilon_2)$-SOSP of problem \\eqref{model:equa-cnstr} is $\\overline{K}_{\\epsilon_1}+1$, which is at most of $\\cO(|\\log\\epsilon_1|^2)$. In addition, the penalty parameters $\\{\\rho_k\\}$ generated in this algorithm are at most of $\\cO(\\epsilon_1^{-2})$.\r\n\\end{remark}","sha256_utf8":"2769ad9afd8130ec9adac9f4d7ee9b7ba406ed8740aee60ea92a70eb45e3583b"},{"start":89743,"end":90085,"start_line":819,"end_line":823,"raw_tex":"One can observe from Assumption~\\ref{asp:NCG-cmplxity}(b) that for all $x$ and $y\\in \\Omega$,\r\n\\begin{eqnarray}\r\n&&\\ \\ \\ \\|\\nabla F(y)-\\nabla F(x)-\\nabla^2 F(x)(y-x)\\|\\le L_H^F\\|y-x\\|^2/2, \\label{apx-nxt-grad}\\\\\r\n&&\\ \\ \\ F(y)\\le F(x) + \\nabla F(x)^T(y-x) + (y-x)^T\\nabla^2 F(x) (y-x)/2 +  L_H^F\\|y-x\\|^3/6. \\label{desc-ineq}\r\n\\end{eqnarray}\r\n","sha256_utf8":"8a4213c6cb5e6f385f173f59a4d74867b94cad89696aa8c5867a3d49908ed366"},{"start":165040,"end":167637,"start_line":1733,"end_line":1786,"raw_tex":"\\begin{algorithm}[h]\r\n\\caption{A capped conjugate gradient method}\r\n\\label{alg:capped-CG}\r\n{\\footnotesize\r\n\\begin{algorithmic}\r\n\\State \\noindent\\textit{Inputs}: symmetric matrix $H\\in\\bR^{n\\times n}$, vector $g\\neq0$, damping parameter $\\varepsilon\\in(0,1)$, desired relative accuracy $\\zeta\\in(0,1)$.\r\n\\State \\textit{Optional input:} scalar $U\\ge0$ (set to $0$ if not provided).\r\n\\State \\textit{Outputs:} d$\\_$type, $d$.\r\n\\State \\textit{Secondary outputs:} final values of $U,\\kappa,\\widehat{\\zeta},\\tau,$ and $T$.\r\n\\State Set\r\n\\begin{equation*}\r\n\\textstyle\r\n\\bar{H}:=H+2\\varepsilon I,\\quad \\kappa:=\\frac{U+2\\varepsilon}{\\varepsilon},\\quad\\widehat{\\zeta}:=\\frac{\\zeta}{3\\kappa},\\quad\\tau:=\\frac{\\sqrt{\\kappa}}{\\sqrt{\\kappa}+1},\\quad T:=\\frac{4\\kappa^4}{(1-\\sqrt{\\tau})^2},\r\n\\end{equation*}\r\n$y^0\\leftarrow 0,r^0\\leftarrow g,p^0\\leftarrow -g, j\\leftarrow 0$.\r\n\\If {$(p^0)^T \\bar{H}p^0<\\varepsilon\\|p^0\\|^2$}\r\n\\State Set $d\\leftarrow p^0$ and terminate with d$\\_$type = NC;\r\n\\ElsIf {\\ $\\|Hp^0\\|>U\\|p^0\\|\\ $}\r\n\\State Set $U\\leftarrow\\|Hp^0\\|/\\|p^0\\|$ and update $\\kappa,\\widehat{\\zeta},\\tau, T$ accordingly;\r\n\\EndIf\r\n\\While{TRUE}\r\n\\State $\\alpha_j\\leftarrow (r^j)^T r^j/(p^j)^T\\bar{H}p^j$; \\{Begin Standard CG Operations\\}\r\n\\State $y^{j+1}\\leftarrow y^j+\\alpha_jp^j$;\r\n\\State $r^{j+1}\\leftarrow r^j+\\alpha_j\\bar{H}p^j$;\r\n\\State $\\beta_{j+1}\\leftarrow\\|r^{j+1}\\|^2/\\|r^j\\|^2$;\r\n\\State $p^{j+1}\\leftarrow-r^{j+1}+\\beta_{j+1}p^j$; \\{End Standard CG Operations\\}\r\n\\State $j\\leftarrow j+1$;\r\n\\If {$\\|Hp^j\\|>U\\|p^j\\|$}\r\n\\State Set $U\\leftarrow\\|Hp^j\\|/\\|p^j\\|$ and update $\\kappa,\\widehat{\\zeta},\\tau,T$ accordingly;\r\n\\EndIf\r\n\\If {\\ $\\|Hy^j\\|>U\\|y^j\\|\\ $}\r\n\\State Set $U\\leftarrow\\|Hy^j\\|/\\|y^j\\|$ and update $\\kappa,\\widehat{\\zeta},\\tau,T$ accordingly;\r\n\\EndIf\r\n\\If {\\ $\\|Hr^j\\|>U\\|r^j\\|\\ $}\r\n\\State Set $U\\leftarrow\\|Hr^j\\|/\\|r^j\\|$ and update $\\kappa,\\widehat{\\zeta},\\tau,T$ accordingly;\r\n\\EndIf\r\n\\If {$(y^j)^T\\bar{H}y^j<\\varepsilon\\|y^j\\|^2$}\r\n\\State Set $d\\leftarrow y^j$ and terminate with d$\\_$type = NC;\r\n\\ElsIf {\\ $\\|r^j\\|\\le\\widehat{\\zeta}\\|r^0\\|$}\r\n\\State Set $d\\leftarrow y^j$ and terminate with d$\\_$type = SOL;\r\n\\ElsIf{\\ $(p^j)^T\\bar{H}p^j<\\varepsilon\\|p^j\\|^2$}\r\n\\State Set $d\\leftarrow p^j$ and terminate with d$\\_$type = NC;\r\n\\ElsIf {\\ $\\|r^j\\|>\\sqrt{T}\\tau^{j/2}\\|r^0\\| $}\r\n\\State Compute $\\alpha_j, y^{j+1}$ as in the main loop above;\r\n\\State Find $i\\in\\{0,\\ldots,j-1\\}$ such that\r\n\\[\r\n(y^{j+1}-y^i)^T\\bar{H}(y^{j+1}-y^i)<\\varepsilon\\|y^{j+1}-y^i\\|^2;\r\n\\]\r\n\\State Set $d\\leftarrow y^{j+1}-y^i$ and terminate with d$\\_$type = NC;\r\n\\EndIf\r\n\\EndWhile\r\n\\end{algorithmic}\r\n}\r\n\\end{algorithm}","sha256_utf8":"440b17ac5f36cdc0d2184335c209838c531ab42bcdba231d1deb0ff7069142bc"},{"start":169623,"end":170583,"start_line":1806,"end_line":1816,"raw_tex":"\\begin{algorithm}[h]\r\n\\caption{A randomized Lanczos based minimum eigenvalue oracle}\r\n\\label{pro:meo}\r\n\\noindent\\textit{Input}: symmetric matrix $H\\in\\bR^{n\\times n}$, tolerance $\\varepsilon>0$, and probability parameter $\\delta\\in(0,1)$.\\\\\r\n\\noindent\\textit{Output}: a sufficiently negative curvature direction $v$ satisfying $v^THv\\le-\\varepsilon/2$ and $\\|v\\|=1$; or a certificate that $\\lambda_{\\min}(H)\\ge-\\varepsilon$ with probability at least  $1-\\delta$.\\\\\r\nApply the Lanczos method \\cite{KW92LR} to estimate $\\lambda_{\\min}(H)$ starting with a random vector uniformly generated on the unit sphere, and run it for at most \r\n\\begin{equation}\\label{N-iter} \r\nN(\\varepsilon,\\delta):=\\min\\left\\{n,1+\\left\\lceil\\frac{\\ln(2.75n/\\delta^2)}{2}\\sqrt{\\frac{\\|H\\|}{\\varepsilon}}\\right\\rceil\\right\\}\r\n\\end{equation}\r\niterations. If a unit vector $v$ with $v^THv \\le -\\varepsilon/2$ is found at some iteration, terminate immediately and return $v$.\r\n\\end{algorithm}","sha256_utf8":"5e4b2120d167492b986ef3a45cb6ef9e754e3197c00b847c4632f0c175f64cff"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}%[{\\bf Total inner iteration and operation complexity of Algorithm \\ref{alg:2nd-order-AL-nonconvex}}]\r\n\\label{thm:total-iter-cmplxity}\r\nSuppose that Assumption \\ref{asp:lowbd-knownfeas} holds and that $\\rho_0$ is sufficiently large such that $\\delta_{f,1} \\le \\bdelta_f$ and  $\\delta_{c,1} \\le \\bdelta_c$, where $\\delta_{f,1}$ and $\\delta_{c,1}$ are defined in \\eqref{def:delta0c-rhobar1xxx}. Then the following statements hold.\r\n\\begin{enumerate}[{\\rm (i)}]\r\n\\item %The total number of inner iterations of Algorithm \\ref{alg:2nd-order-AL-nonconvex}, namely,\r\nThe total number of iterations of Algorithm~\\ref{alg:NCG} performed in Algorithm \\ref{alg:2nd-order-AL-nonconvex} is at most $\\widetilde{\\cO}(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\})$. If $c$ is further assumed to be affine, then it is at most $\\widetilde{\\cO}(\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\})$.\r\n\\item The total number of matrix-vector products performed  by Algorithm~\\ref{alg:NCG} in  Algorithm \\ref{alg:2nd-order-AL-nonconvex} is at most $\\widetilde{\\cO}(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}\\min\\{n,\\epsilon_1^{-1}\\epsilon_2^{-1/2}\\})$. If $c$ is further assumed to be affine, then it is at most $\\widetilde{\\cO}(\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}\\min\\{n,\\epsilon_1^{-1}\\epsilon_2^{-1/2}\\})$.\r\n\\end{enumerate}\r\n\\end{theorem}","units":[{"start":72661,"end":74050,"start_line":680,"end_line":688,"raw_tex":"\\begin{theorem}%[{\\bf Total inner iteration and operation complexity of Algorithm \\ref{alg:2nd-order-AL-nonconvex}}]\r\n\\label{thm:total-iter-cmplxity}\r\nSuppose that Assumption \\ref{asp:lowbd-knownfeas} holds and that $\\rho_0$ is sufficiently large such that $\\delta_{f,1} \\le \\bdelta_f$ and  $\\delta_{c,1} \\le \\bdelta_c$, where $\\delta_{f,1}$ and $\\delta_{c,1}$ are defined in \\eqref{def:delta0c-rhobar1xxx}. Then the following statements hold.\r\n\\begin{enumerate}[{\\rm (i)}]\r\n\\item %The total number of inner iterations of Algorithm \\ref{alg:2nd-order-AL-nonconvex}, namely,\r\nThe total number of iterations of Algorithm~\\ref{alg:NCG} performed in Algorithm \\ref{alg:2nd-order-AL-nonconvex} is at most $\\widetilde{\\cO}(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\})$. If $c$ is further assumed to be affine, then it is at most $\\widetilde{\\cO}(\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\})$.\r\n\\item The total number of matrix-vector products performed  by Algorithm~\\ref{alg:NCG} in  Algorithm \\ref{alg:2nd-order-AL-nonconvex} is at most $\\widetilde{\\cO}(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}\\min\\{n,\\epsilon_1^{-1}\\epsilon_2^{-1/2}\\})$. If $c$ is further assumed to be affine, then it is at most $\\widetilde{\\cO}(\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}\\min\\{n,\\epsilon_1^{-1}\\epsilon_2^{-1/2}\\})$.\r\n\\end{enumerate}\r\n\\end{theorem}","sha256_utf8":"d0b4a59201b4475c29d9b1c00700720f0ae625c47561ad021ebd07c730e0db75"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{theorem}%[{{\\bf Complexity of Algorithm~\\ref{alg:NCG}}}]\r\n\\label{thm:NCG-iter-oper-cmplxity}\r\nSuppose that Assumption~\\ref{asp:NCG-cmplxity} holds. Let \r\n\\begin{equation}\r\nT_1:=\\left\\lceil\\frac{\\Fh-\\Fl}{\\min\\{c_{\\rm sol},c_{\\rm nc}\\}}\\max\\{\\epsilon_g^{-2}\\epsilon_H,\\epsilon_H^{-3}\\}\\right\\rceil+\\left\\lceil\\frac{\\Fh-\\Fl}{c_{\\rm nc}}\\epsilon_H^{-3}\\right\\rceil+1, \\\r\nT_2:=\\left\\lceil\\frac{\\Fh-\\Fl}{c_{\\rm nc}}\\epsilon_H^{-3}\\right\\rceil+1,\\label{T1}\r\n\\end{equation}\r\n%where $\\Fh=F(u^0)$, $\\Fl$ is given in \\eqref{lwbd-Hgupbd}, and $c_{\\rm sol}$ and $c_{\\rm nc}$ are defined in \\eqref{csol} and \\eqref{cnc}, respectively.\r\nwhere $\\Fh=F(u^0)$, $\\Fl$ is given in \\eqref{lwbd-Hgupbd}, and \r\n\\beqa\r\n&&c_{\\rm sol} :=\\eta\\min\\left\\{\\left[\\frac{4}{4+\\zeta+\\sqrt{(4+\\zeta)^2+8L^F_H}}\\right]^2,\\left[\\frac{\\min\\{6(1-\\eta),2\\}\\theta}{L^F_H}\\right]^2\\right\\}, \\label{csol} \\\\\r\n&& c_{\\rm nc} := \\frac{\\eta}{16} \\min\\left\\{1,\\left[\\frac{\\min\\{3(1-\\eta),1\\}\\theta}{L_H^F}\\right]^2\\right\\}.   \\label{cnc}\r\n\\eeqa\r\n Then the following statements hold.\r\n\\begin{enumerate}[{\\rm (i)}]\r\n\\item The total number of calls of Algorithm~\\ref{pro:meo} in Algorithm~\\ref{alg:NCG} is at most $T_2$.\t\r\n\\item The total number of calls of Algorithm~\\ref{alg:capped-CG} in Algorithm~\\ref{alg:NCG} is at most $T_1$.\r\n\\item {\\rm ({\\bf iteration complexity})} Algorithm~\\ref{alg:NCG} terminates in at most $T_1+T_2$ iterations with \r\n\\begin{equation}\\label{NCG-iter} \r\nT_1+T_2=\\cO((\\Fh-\\Fl)(L_H^F)^2\\max\\{\\epsilon_g^{-2}\\epsilon_H,\\epsilon_H^{-3}\\}).\r\n\\end{equation}\r\nAlso, its output $x^t$ satisfies $\\|\\nabla F(x^t)\\|\\le \\epsilon_g$ deterministically and\r\n$\\lambda_{\\min}(\\nabla^2F(x^t))$ $\\ge-\\epsilon_H$ with probability at least $1-\\delta$ for\r\nsome $0 \\le t \\le T_1+T_2$.\r\n\\item {\\rm ({\\bf operation complexity})} Algorithm~\\ref{alg:NCG} requires at most \r\n\\[\r\n\\widetilde{\\cO}((\\Fh-\\Fl)(L_H^F)^2\\max\\{\\epsilon_g^{-2}\\epsilon_H,\\epsilon_H^{-3}\\}\\min\\{n,(U_H^F/\\epsilon_H)^{1/2}\\})\r\n\\]\r\nmatrix-vector products, where $U_H^F$ is given in \\eqref{lwbd-Hgupbd}.\r\n\\end{enumerate}\r\n\\end{theorem}\n\nTo this end, we start by noting from \\eqref{hbd}, \\eqref{model:equa-cnstr-pert}, \\eqref{tL-tc} and \\eqref{def:initial-iterate-subprob} that\r\n\\beq \\label{L-xinit}\r\n\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k) \\le \\max\\{\\widetilde{\\cL}(z_{\\epsilon_1},\\lambda^k;\\rho_k),f(z_{\\epsilon_1})\\}=f(z_{\\epsilon_1})\\le \\fh.\r\n\\eeq\r\n\n\n\\begin{lemma}%[{{\\bf level- and lower-boundedness of $\\widetilde{\\cL}(\\cdot,\\lambda^k;\\rho_k)$}}]\r\n\\label{lem:level-set-augmented-lagrangian-func}\r\nSuppose that Assumption~\\ref{asp:lowbd-knownfeas} holds. Let $(\\lambda^k, \\rho_k)$ be generated at the $k$th iteration of Algorithm~\\ref{alg:2nd-order-AL-nonconvex} for some $k \\ge 0$, and $\\cS(\\bdelta_f,\\bdelta_c)$ and $x_{\\init}^k$ be defined in \\eqref{nearly-feas-level-set} and \\eqref{def:initial-iterate-subprob}, respectively, and let $\\fh$, $\\fl$, $\\bdelta_f$ and $\\bdelta_c$ be given in Assumption~\\ref{asp:lowbd-knownfeas}. Suppose that $\\rho_0$ is sufficiently large such that $\\delta_{f,1} \\le \\bdelta_f$ and  $\\delta_{c,1} \\le \\bdelta_c$, where \r\n\\begin{equation} \r\n\\delta_{f,1}:=\\Lambda^2/(2\\rho_0)\\quad and\\quad \\delta_{c,1}:=\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_0-2\\gamma}+\\frac{\\Lambda^2}{(\\rho_0-2\\gamma)^2}}+\\frac{\\Lambda}{\\rho_0-2\\gamma}\\label{def:delta0c-rhobar1xxx}.\r\n\\end{equation}\r\nThen the following statements hold.\r\n\\begin{enumerate}[{\\rm (i)}]\r\n\\item $\\{x:\\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\}\\subseteq \\cS(\\bdelta_f,\\bdelta_c)$.\r\n\\item $\\inf_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k) \\ge\\fl - \\gamma-\\Lambda\\bdelta_c$.\r\n\\end{enumerate}\r\n\\end{lemma}\n\nUsing Lemma \\ref{lem:level-set-augmented-lagrangian-func}, we can verify that the Newton-CG method (Algorithm \\ref{alg:NCG}), starting with $u^0=x^{k}_{\\init}$, is capable of finding an approximate solution $x^{k+1}$ of the AL subproblem $\\min_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$ satisfying \\eqref{algstop:1st-order} and \\eqref{algstop:2nd-order}. Indeed,\r\nlet $F(\\cdot)=\\widetilde{\\cL}(\\cdot,\\lambda^k;\\rho_k)$ and $u^0=x^{k}_{\\init}$. By these and Lemma \\ref{lem:level-set-augmented-lagrangian-func}, one can see that $\\{x:F(x) \\le F(u^0)\\} \\subseteq \\cS(\\bdelta_f,\\bdelta_c)$. It then follows from this and Assumption \\ref{asp:lowbd-knownfeas}(c) that the level set $\\{x : F(x) \\le F(u^0)\\}$ is compact  and $\\nabla^2 F$ is Lipschitz continuous on a convex open neighborhood of $\\{x:F(x) \\le F(u^0)\\}$. Thus, such $F$ and $u^0$ satisfy Assumption \\ref{asp:NCG-cmplxity}. Based on this and the discussion in Section \\ref{sec:sbpb-solver}, one can conclude that Algorithm~\\ref{alg:NCG}, starting with $u^0=x^{k}_{\\init}$, is applicable to the AL subproblem $\\min_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$. Moreover, it follows from Theorem \\ref{thm:NCG-iter-oper-cmplxity} that this algorithm with $(\\epsilon_g,\\epsilon_H)=(\\tau_k^g,\\tau_k^H)$ can produce a point $x^{k+1}$ satisfying \\eqref{algstop:2nd-order} and also the second relation in \\eqref{algstop:1st-order}. In addition, since this algorithm\r\nis descent and its starting point is $x^{k}_{\\init}$, its output $x^{k+1}$ must satisfy $\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k) \\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)$, which along with \\eqref{L-xinit} implies that $\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k) \\le f(z_{\\epsilon_1})$ and thus $x^{k+1}$ also satisfies the first relation in \\eqref{algstop:1st-order}.\r\n\n\n\\begin{theorem}%[{{\\bf Well-definedness of Algorithm \\ref{alg:2nd-order-AL-nonconvex}}}]\r\n\\label{subprob-solver}\r\nUnder the same settings as in Lemma \\ref{lem:level-set-augmented-lagrangian-func}, the Newton-CG method (Algorithm \\ref{alg:NCG}) applied to the AL subproblem $\\min_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$ with $u^0=x^{k}_{\\init}$ finds a point $x^{k+1}$ satisfying \\eqref{algstop:1st-order} and \\eqref{algstop:2nd-order}.\r\n\\end{theorem}\n\n\\begin{theorem}%[{{\\bf Output of Algorithm \\ref{alg:2nd-order-AL-nonconvex}}}]\r\n\\label{thm:output-alg1}\r\nSuppose that Assumption~\\ref{asp:lowbd-knownfeas} holds and that $\\rho_0$ is sufficiently large such that $\\delta_{f,1} \\le \\bdelta_f$ and  $\\delta_{c,1} \\le \\bdelta_c$, where\r\n$\\delta_{f,1}$ and $\\delta_{c,1}$ are defined in \\eqref{def:delta0c-rhobar1xxx}.\r\nIf Algorithm \\ref{alg:2nd-order-AL-nonconvex} terminates at some iteration $k$, then $x^{k+1}$ is a deterministic $\\epsilon_1$-FOSP of problem \\eqref{model:equa-cnstr}, and moreover, it is an $(\\epsilon_1,\\epsilon_2)$-SOSP of \\eqref{model:equa-cnstr} with probability at least $1-\\delta$.\r\n\\end{theorem}\n\nIn view of \\eqref{omega-tolerance}, \\eqref{T-epsilon-g} and the fact that\r\n\\begin{equation}\\label{epsw1-epsw2}\r\n%\\log\\epsilon_1/\\log\\omega_1=\\log2/\\log r=\\log\\epsilon_2/\\log\\omega_2,\r\n\\log\\epsilon_1/\\log\\omega_1=\\log\\epsilon_2/\\log\\omega_2=\\log2/\\log r,\r\n\\end{equation}\r\nwe see that $(\\tau_k^g,\\tau_k^H)=(\\epsilon_1,\\epsilon_2)$ for all $k\\ge K_{\\epsilon_1}$. This along with the termination criterion of Algorithm \\ref{alg:2nd-order-AL-nonconvex} implies that it runs for at least $K_{\\epsilon_1}$ iterations and  terminates once $\\|c(x^{k+1})\\|\\le\\epsilon_1$ for some $k\\ge K_{\\epsilon_1}$. As a result, to establish outer iteration complexity of Algorithm \\ref{alg:2nd-order-AL-nonconvex}, it suffices to bound such  $k$. The resulting outer iteration complexity of Algorithm \\ref{alg:2nd-order-AL-nonconvex} is presented below, whose proof is deferred to Section \\ref{sec:pf-AL}.\r\n\n\n\\begin{theorem}%[{{\\bf Outer iteration complexity of Algorithm \\ref{alg:2nd-order-AL-nonconvex}}}]\r\n\\label{thm:out-itr-cmplxity-1}\r\nSuppose that Assumption~\\ref{asp:lowbd-knownfeas} holds and that $\\rho_0$ is sufficiently large such that $\\delta_{f,1} \\le \\bdelta_f$ and  $\\delta_{c,1} \\le \\bdelta_c$, where\r\n$\\delta_{f,1}$ and $\\delta_{c,1}$ are defined in \\eqref{def:delta0c-rhobar1xxx}. Let\r\n\\begin{eqnarray}\r\n&&{\\rho}_{\\epsilon_1}:=\\max\\left\\{8(\\fh-\\fl+\\gamma)\\epsilon_1^{-2}+4\\Lambda\\epsilon_1^{-1}+2\\gamma,2\\rho_0\\right\\},\\label{def:delta0c-rhobar1}\\\\\r\n&&\\overline{K}_{\\epsilon_1}:=\\inf\\{k\\ge K_{\\epsilon_1}: \\|c(x^{k+1})\\|\\le\\epsilon_1\\},\\label{number-outer-iteration}\r\n\\end{eqnarray}\r\nwhere $K_{\\epsilon_1}$ is defined in \\eqref{T-epsilon-g},  and $\\gamma$, $\\fh$ and $\\fl$ are given in Assumption \\ref{asp:lowbd-knownfeas}. Then $\\overline{K}_{\\epsilon_1}$ is finite, and Algorithm \\ref{alg:2nd-order-AL-nonconvex} terminates at iteration $\\overline{K}_{\\epsilon_1}$ with\r\n\\begin{equation}\\label{outer-iteration-cmplxty}\r\n\\overline{K}_{\\epsilon_1}\\le\\left(\\frac{\\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})}{\\log{r}}+1\\right)\\left(\\left|\\frac{\\log(\\epsilon_1(2\\delta_{c,1})^{-1})}{\\log \\alpha}\\right|+2\\right)+1.\r\n\\end{equation}\r\nMoreover, $\\rho_k\\le{r}{\\rho}_{\\epsilon_1}$ holds for $0 \\le k \\le \\overline{K}_{\\epsilon_1}$\r\n\\end{theorem}\n\n\\begin{lemma}\\label{lem:SOL-NC-ppty}\r\nSuppose that Assumption~\\ref{asp:NCG-cmplxity} holds and the direction $d^t$ results from  the output $d$ of Algorithm \\ref{alg:capped-CG} with a type specified in d$\\_$type  at some iteration $t$ of Algorithm \\ref{alg:NCG}. Then the following statements hold.\r\n\\begin{enumerate}[{\\rm (i)}]\r\n\\item If d$\\_$type=SOL, then  $d^t$ satisfies\r\n\\beqa\r\n&\\epsilon_H\\|d^t\\|^2\\le (d^t)^T\\left(\\nabla^2 F(x^t)+2\\epsilon_H I\\right)d^t,\\label{SOL-ppty-1}\\\\\r\n&\\|d^t\\|\\le1.1 \\epsilon_H^{-1}\\|\\nabla F(x^t)\\|,\\label{SOL-ppty-2}\\\\\r\n&(d^t)^T\\nabla F(x^t)=-(d^t)^T\\left(\\nabla^2F(x^t)+2\\epsilon_HI\\right)d^t,\\label{SOL-ppty-3}\\\\\r\n&\\|(\\nabla^2F(x^t)+2\\epsilon_HI)d^t+\\nabla F(x^t)\\|\\le\\epsilon_H\\zeta\\|d^t\\|/2.\\label{SOL-ppty-4}\r\n\\eeqa\r\n\\item If d$\\_$type=NC, then  $d^t$ satisfies $(d^t)^T\\nabla F(x^t)\\le0$ and\r\n\\begin{equation}\\label{NC-ppty}\r\n(d^t)^T\\nabla^2F(x^t)d^t/\\|d^t\\|^2=-\\|d^t\\|\\le-\\epsilon_H.\r\n\\end{equation}\r\n\\end{enumerate}\r\n\\end{lemma}\n\n\\begin{lemma}\\label{lem:sol}\r\nSuppose that Assumption~\\ref{asp:NCG-cmplxity} holds and the direction $d^t$ results from  the output $d$ of Algorithm~\\ref{alg:capped-CG} with d$\\_$type=SOL at some iteration $t$ of Algorithm~\\ref{alg:NCG}. Let $U_g^F$ and $c_{\\rm sol}$ be given in \\eqref{lwbd-Hgupbd} and \\eqref{csol}, respectively.  Then the following statements hold.\r\n\\begin{enumerate}[{\\rm (i)}]\r\n\\item The step length $\\alpha_t$ is well-defined, and moreover,\r\n\\begin{equation}\\label{lwbd-ak-sol}\r\n\\alpha_t\\ge\\min\\left\\{1,\\sqrt{\\frac{\\min\\{6(1-\\eta),2\\}}{1.1 L_H^F U_g^F}}\\theta\\epsilon_H\\right\\}.\r\n\\end{equation}\r\n\\item The next iterate $x^{t+1}=x^t+\\alpha_td^t$ satisfies %$F(x^t)-F(x^{t+1})\\ge c_{\\rm sol}\\min\\{\\|\\nabla F(x^{t+1})\\|^2\\epsilon_H^{-1},\\epsilon_H^3\\}$.\r\n\\begin{equation}\\label{suff-desc-sol}\r\nF(x^t)-F(x^{t+1})\\ge c_{\\rm sol}\\min\\{\\|\\nabla F(x^{t+1})\\|^2\\epsilon_H^{-1},\\epsilon_H^3\\}.\r\n\\end{equation}\r\n\\end{enumerate}\r\n\\end{lemma}\n\n\\begin{proof}\r\nOne can observe that $F$ is descent along the iterates (whenever well-defined) generated by Algorithm~\\ref{alg:NCG}, which together with $x^0=u^0$ implies that $F(x^t)\\le F(u^0)$ and hence\r\n$\\|\\nabla F(x^t)\\|\\le U^F_g$ due to \\eqref{lwbd-Hgupbd}.\r\n%Since $d^t$ { results from} the output $d$ of Algorithm~\\ref{alg:capped-CG} with d$\\_$type=SOL, it follows from Lemma~\\ref{lem:SOL-NC-ppty}(i) that \\eqref{SOL-ppty-1}-\\eqref{SOL-ppty-4} hold for $d^t$.\r\n%In particular, we see from \\eqref{SOL-ppty-2} and \\eqref{SOL-ppty-4} that $\\|(\\nabla^2F(x^t)+2\\epsilon_HI)d^t+\\nabla F(x^t)\\|\\le0.55\\zeta\\|\\nabla F(x^t)\\|$. Since we also have $\\|\\nabla F(x^t)\\| > \\epsilon_g$ in this case and recall that $ \\zeta \\in (0,1)$, we conclude that $d^t\\neq 0$.\r\nIn addition, since $d^t$ results from the output $d$ of Algorithm~\\ref{alg:capped-CG} with d$\\_$type=SOL, one can see that $\\|\\nabla F(x^t)\\| > \\epsilon_g$ and \\eqref{SOL-ppty-1}-\\eqref{SOL-ppty-4} hold for $d^t$. Moreover, by $\\|\\nabla F(x^t)\\| > \\epsilon_g$ and \\eqref{SOL-ppty-4}, one can conclude that $d^t\\neq 0$.\r\n\r\nWe first prove statement (i). If \\eqref{ls-sol} holds for $j=0$, then $\\alpha_t=1$, which clearly implies that \\eqref{lwbd-ak-sol} holds. We now suppose that \\eqref{ls-sol} fails for $j=0$. Claim that for all $j\\ge0$ that violate \\eqref{ls-sol}, it holds that\r\n\\begin{equation}\\label{lwbd-sol-ak}\r\n\\theta^{2j}\\ge\\min\\{6(1-\\eta),2\\}\\epsilon_H(L^F_H)^{-1}\\|d^t\\|^{-1}.\r\n\\end{equation}\r\nIndeed, suppose that \\eqref{ls-sol} is violated by some $j \\ge 0$. We now show that \\eqref{lwbd-sol-ak} holds for such $j$ by considering two separate cases below.\r\n\r\nCase 1) $F(x^t+\\theta^jd^t)>F(x^t)$. Let $\\phi(\\alpha)=F(x^t+\\alpha d^t)$. Then $\\phi(\\theta^j)>\\phi(0)$. Also, since $d^t\\neq0$, by \\eqref{SOL-ppty-1} and \\eqref{SOL-ppty-3}, one has \r\n\\[\r\n\\phi^\\prime(0)  =\\nabla F(x^t)^Td^t = -(d^t)^T(\\nabla^2 F(x^t)+2\\epsilon_HI)d^t \r\n \\le-\\epsilon_H\\|d^t\\|^2< 0.\r\n\\]\r\nUsing these, we can observe that there exists a local minimizer $\\alpha^*\\in(0,\\theta^j)$ of $\\phi$ such that $\\phi^\\prime(\\alpha^*)=\\nabla F(x^t+\\alpha^*d^t)^Td^t=0$ and $\\phi(\\alpha^*)<\\phi(0)$, which implies that $F(x^t+\\alpha^*d^t)< F(x^t)\\le F(u^0)$. Hence, \\eqref{apx-nxt-grad} holds for $x=x^t$ and $y=x^t+\\alpha^*d^t$. Using this, $0<\\alpha^*<\\theta^j\\le1$ and $\\nabla F(x^t+\\alpha^*d^t)^Td^t=0$, we obtain\r\n\\[\r\n\\begin{array}{l}\r\n\\frac{(\\alpha^*)^2L_H^F}{2}\\|d^t\\|^3\\overset{\\eqref{apx-nxt-grad}}{\\ge}\\|d^t\\|\\|\\nabla F(x^t+\\alpha^*d^t)-\\nabla F(x^t)-\\alpha^*\\nabla^2 F(x^t)d^t\\|\\\\[6pt]\r\n\\ge(d^t)^T(\\nabla F(x^t+\\alpha^*d^t)-\\nabla F(x^t)-\\alpha^*\\nabla^2 F(x^t)d^t)=-(d^t)^T\\nabla F(x^t)-\\alpha^*(d^t)^T\\nabla^2 F(x^t)d^t\\\\[6pt]\r\n\\overset{\\eqref{SOL-ppty-3}}{=}(1-\\alpha^*)(d^t)^T(\\nabla^2 F(x^t)+2\\epsilon_HI)d^t +2 \\alpha^*\\epsilon_H\\|d^t\\|^2\\overset{\\eqref{SOL-ppty-1}}{\\ge} (1+\\alpha^*)\\epsilon_H\\|d^t\\|^2\\ge \\epsilon_H \\|d^t\\|^2,\r\n\\end{array}\r\n\\]\r\nwhich along with $d^t\\neq 0$ implies that $(\\alpha^*)^2\\ge 2\\epsilon_H(L_H^F)^{-1}\\|d^t\\|^{-1}$. Using this and $\\theta^{j}>\\alpha^*$, we conclude that \\eqref{lwbd-sol-ak} holds in this case.\r\n\r\nCase 2) $F(x^t+\\theta^jd^t)\\le F(x^t)$. This together with $F(x^t)\\le F(u^0)$ implies that\r\n\\eqref{desc-ineq} holds for $x=x^t$ and $y=x^t+\\theta^jd^t$. Then, because $j$ violates \\eqref{ls-sol}, we obtain\r\n%By this and the fact that $j$ violates \\eqref{ls-sol}, we obtain that\r\n\\beqa\r\n&&-\\eta\\epsilon_H\\theta^{2j}\\|d^t\\|^2\\le F(x^t+\\theta^jd^t)-F(x^t)\\overset{\\eqref{desc-ineq}}{\\le} \\theta^j \\nabla F(x^t)^Td^t + \\frac{\\theta^{2j}}{2}(d^t)^T\\nabla^2 F(x^t) d^t + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3 \\nn \\\\\r\n&&\\overset{\\eqref{SOL-ppty-3}}{=}-\\theta^j (d^t)^T(\\nabla^2 F(x^t) +2\\epsilon_H I)d^t + \\frac{\\theta^{2j}}{2}(d^t)^T\\nabla^2 F(x^t) d^t + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3 \\nn \\\\\r\n&&=-\\theta^j\\left(1-\\frac{\\theta^j}{2}\\right)(d^t)^T(\\nabla^2 F(x^t) +2\\epsilon_H I)d^t - \\theta^{2j} \\epsilon_H\\|d^t\\|^2 + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3 \\nn \\\\\r\n&&\\overset{\\eqref{SOL-ppty-1}}{\\le} -\\theta^j\\left(1-\\frac{\\theta^j}{2}\\right) \\epsilon_H\\|d^t\\|^2 - \\theta^{2j}\\epsilon_H\\|d^t\\|^2 + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3 \\le -\\theta^j \\epsilon_H\\|d^t\\|^2 + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3. \\label{ineq:desc-ls-sol}\r\n\\eeqa\r\nRecall that $d^t \\neq 0$. Dividing both sides of \\eqref{ineq:desc-ls-sol} by $L_H^F\\theta^j\\|d^t\\|^3/6$ and using $\\eta, \\theta\\in(0,1)$, we obtain that \r\n\\[\r\n\\theta^{2j}\\ge 6(1-\\theta^j\\eta)\\epsilon_H(L_H^F)^{-1}\\|d^t\\|^{-1}\\ge6(1-\\eta)\\epsilon_H(L_H^F)^{-1}\\|d^t\\|^{-1}.\r\n\\] \r\n Hence, \\eqref{lwbd-sol-ak} also holds in this case.\r\n\r\nCombining the above two cases, we conclude that  \\eqref{lwbd-sol-ak} holds for any $j\\ge0$ that violates \\eqref{ls-sol}.\r\nBy this and $\\theta\\in(0,1)$, one can see that all $j\\ge0$ that violate \\eqref{ls-sol} must be bounded above. It then follows that the step length $\\alpha_t$ associated with \\eqref{ls-sol} is well-defined. We next prove \\eqref{lwbd-ak-sol}. Observe from the definition of $j_t$ in Algorithm~\\ref{alg:NCG} that $j=j_t-1$ violates \\eqref{ls-sol} and hence \\eqref{lwbd-sol-ak} holds for $j=j_t-1$. Then, by \\eqref{lwbd-sol-ak} with $j=j_t-1$ and $\\alpha_t = \\theta^{j_t}$, one has\r\n\\begin{equation}\\label{lwbd-sol-ak-1}\r\n\\alpha_t = \\theta^{j_t} \\ge\\sqrt{\\min\\{6(1-\\eta),2\\}\\epsilon_H(L^F_H)^{-1}} \\ \\theta\\|d^t\\|^{-1/2},\r\n\\end{equation}\r\nwhich,  along with \\eqref{SOL-ppty-2} and $\\|\\nabla F(x^t)\\|\\le U^F_g$, implies \\eqref{lwbd-ak-sol}. This proves statement (i).\r\n\r\nWe next prove statement (ii) %, particularly,  \\eqref{suff-desc-sol}\r\nby considering two separate cases below.\r\n\r\nCase 1) $\\alpha_t=1$. By this, one knows that \\eqref{ls-sol} holds for $j=0$. It then follows that $F(x^t+d^t)\\le F(x^t)\\le F(u^0)$, which implies that \\eqref{apx-nxt-grad} holds for $x=x^t$ and $y=x^t+d^t$. By this and \\eqref{SOL-ppty-4}, one has\r\n\\[\r\n\\begin{array}{lcl}\r\n%\\|\\nabla F(x^{t+1})\\|=\\|\\nabla F(x^{t}+d^t)-\\nabla F(x^t)+\\nabla F(x^t)\\|\\\\[4pt]\r\n\\|\\nabla F(x^{t+1})\\|=\\|\\nabla F(x^{t}+d^t)\\|\r\n&\\le&\\|\\nabla F(x^t+d^t)-\\nabla F(x^t)-\\nabla^2 F(x^t)d^t\\| \\\\[4pt]\r\n&&+\\|(\\nabla^2 F(x^t)+2\\epsilon_HI)d^t+\\nabla F(x^t)\\|+2\\epsilon_H\\|d^t\\|\\\\[4pt]\r\n&\\le& \\frac{L_H^F}{2}\\|d^t\\|^2 + \\frac{4+\\zeta}{2}\\epsilon_H\\|d^t\\|,\r\n\\end{array}\r\n\\]\r\nwhere the last inequality follows from \\eqref{apx-nxt-grad} and \\eqref{SOL-ppty-4}. Solving the above inequality for $\\|d^t\\|$ and using the fact that $\\|d^t\\|>0$, we obtain that\r\n\\[\r\n\\begin{array}{rcll}\r\n\\|d^t\\|&\\ge&\\frac{-(4+\\zeta)\\epsilon_H + \\sqrt{(4+\\zeta)^2\\epsilon_H^2+8L_H^F\\|\\nabla F(x^{t+1})\\|}}{2L_H^F}\\ \\ge\\\r\n\\frac{-(4+\\zeta)\\epsilon_H + \\sqrt{(4+\\zeta)^2\\epsilon_H^2+8L_H^F\\epsilon_H^2}}{2L_H^F}\\min\\{\\|\\nabla F(x^{t+1})\\|/\\epsilon_H^2,1\\}\\\\[6pt]\r\n&=&\\frac{4}{4+\\zeta+\\sqrt{(4+\\zeta)^2+8L_H^F}}\\min\\{\\|\\nabla F(x^{t+1})\\|/\\epsilon_H,\\epsilon_H\\},\r\n\\end{array}\r\n\\]\r\nwhere the second inequality follows from the inequality $-a+\\sqrt{a^2+bs}\\ge(-a+\\sqrt{a^2+b})\\min\\{s,1\\}$ for all $a,b,s\\ge0$,\r\n%which can be seen to hold by performing rationalization to the terms $-a+\\sqrt{a^2+b}$ and $-a+\\sqrt{a^2+bs}$.\r\nwhich can be verified by performing a rationalization to the terms $-a+\\sqrt{a^2+b}$ and $-a+\\sqrt{a^2+bs}$, respectively.\r\nBy this, $\\alpha_t=1$, \\eqref{ls-sol} and \\eqref{csol}, one can see that \\eqref{suff-desc-sol} holds.\r\n\r\nCase 2) $\\alpha_t<1$.\r\nIt then follows that $j=0$ violates \\eqref{ls-sol} and hence \\eqref{lwbd-sol-ak} holds for $j=0$.\r\nNow, letting $j=0$ in \\eqref{lwbd-sol-ak}, we obtain that $\\|d^t\\|\\ge \\min\\{6(1-\\eta),2\\}\\epsilon_H/L_H^F$,\r\n which together with \\eqref{ls-sol} and \\eqref{lwbd-sol-ak-1} implies that\r\n\\[\r\nF(x^t)-F(x^{t+1}) \\ge \\eta \\epsilon_H\\theta^{2j_t}\\|d^t\\|^2 \\ge \\eta\\frac{\\min\\{6(1-\\eta),2\\}\\epsilon_H^2}{L_H^F}\\theta^2\\|d^t\\|\\ge \\eta\\left[\\frac{\\min\\{6(1-\\eta),2\\}\\theta}{L_H^F}\\right]^2\\epsilon_H^3.\r\n\\]\r\nBy this and \\eqref{csol}, one can see that \\eqref{suff-desc-sol} also holds in this case.\r\n\\end{proof}\n\n\\begin{lemma}\\label{lem:nc}\r\nSuppose that Assumption \\ref{asp:NCG-cmplxity} holds and the direction $d^t$ results from  either the output $d$ of Algorithm~\\ref{alg:capped-CG} with d$\\_$type=NC or the output $v$ of Algorithm~\\ref{pro:meo} at some iteration $t$ of Algorithm~\\ref{alg:NCG}.\r\n%\\begin{equation}\\label{cnc}\r\n%c_{\\rm nc} = \\frac{\\eta}{2} \\min\\left\\{1,\\left[\\frac{3(1-\\eta)\\theta}{L_H^F}\\right]^2\\right\\}.\r\n%\\end{equation}\r\nLet  $c_{\\rm nc}$ be defined in \\eqref{cnc}.\r\n%\\begin{equation}\\label{cnc}\r\n%c_{\\rm nc} := \\frac{\\eta}{2} \\min\\left\\{1,\\left[\\frac{\\min\\{3(1-\\eta),1\\}\\theta}{L_H^F}\\right]^2\\right\\}.\r\n%\\end{equation}\r\nThen the following statements hold.\r\n\\begin{enumerate}[{\\rm (i)}]\r\n\\item The step length $\\alpha_t$ is well-defined, and $\\alpha_t\\ge \\min\\{1,\\theta/L_H^F,3(1-\\eta)\\theta/L_H^F\\}$.\r\n%\\item If $d^t$ results from the output $d$ of Algorithm~\\ref{alg:capped-CG} with d$\\_$type=NC, then the next iterate $x^{t+1}=x^t+\\alpha_td^t$ satisfies $F(x^t)-F(x^{t+1})\\ge c_{\\rm nc}\\epsilon_H^3$.\r\n%\\item If $d^t$ { results from} the output $v$ of Algorithm~\\ref{pro:meo}, then the next iterate $x^{t+1}=x^t+\\alpha_td^t$ satisfies $F(x^t)-F(x^{t+1})\\ge c_{\\rm nc}\\epsilon_H^3/8$.\r\n\\item The next iterate $x^{t+1}=x^t+\\alpha_td^t$ satisfies $F(x^t)-F(x^{t+1})\\ge c_{\\rm nc}\\epsilon_H^3$.\r\n\\end{enumerate}\r\n\\end{lemma}\n\n\\begin{proof}\r\nObserve that $F$ is descent along the iterates (whenever well-defined) generated by Algorithm~\\ref{alg:NCG}. Using this and $x^0=u^0$, we have $F(x^t)\\le F(u^0)$.\r\nBy the assumption on $d^t$, one can see from Algorithm \\ref{alg:NCG} that\r\n%$d^t$ is given in \\eqref{dk-nc} or \\eqref{dk-2nd-nc}. Now, notice that the (negative curvature) direction $v$ returned from Algorithm~\\ref{pro:meo} satisfies $\\|v\\|=1$. This observation (for $d^t$ in \\eqref{dk-2nd-nc}) together with  Lemma~\\ref{lem:SOL-NC-ppty}(ii) (for $d^t$ in \\eqref{dk-nc}) shows that $d^t$ must satisfy\r\n$d^t$ is a negative curvature direction given in \\eqref{dk-nc} or \\eqref{dk-2nd-nc}. Also, notice that the vector $v$ returned from Algorithm~\\ref{pro:meo} satisfies $\\|v\\|=1$. By these, Lemma~\\ref{lem:SOL-NC-ppty}(ii), \\eqref{dk-nc} and \\eqref{dk-2nd-nc}, one can observe that\r\n\\begin{equation}\\label{dk-Hk-dk3}\r\n\\nabla F(x^t)^Td^t\\le 0, \\quad (d^t)^T\\nabla^2 F(x^t)d^t=-\\|d^t\\|^3<0.\r\n\\end{equation}\r\n\r\nWe first prove statement (i). If \\eqref{ls-nc} holds for $j=0$, then $\\alpha_t=1$, which clearly implies that $\\alpha_t\\ge\\min\\{1,\\theta/L_H^F,3(1-\\eta)\\theta/L_H^F\\}$. We now suppose that \\eqref{ls-nc} fails for $j=0$. Claim that for all $j\\ge0$ that violate \\eqref{ls-nc}, it holds that\r\n\\begin{equation}\\label{lwbd-at-nc}\r\n\\theta^j\\ge \\min\\{1/L_H^F,3(1-\\eta)/L_H^F\\}.\r\n\\end{equation}\r\nIndeed, suppose that \\eqref{ls-nc} is violated by some $j \\ge 0$. We now show that \\eqref{lwbd-at-nc} holds for such $j$ by considering two separate cases below.\r\n\r\nCase 1) $F(x^t+\\theta^jd^t)>F(x^t)$. Let $\\phi(\\alpha)=F(x^t+\\alpha d^t)$. Then $\\phi(\\theta^j)>\\phi(0)$. Also, by  \\eqref{dk-Hk-dk3}, one has \r\n\\[\r\n\\phi^\\prime(0)=\\nabla F(x^t)^Td^t\\le 0, \\quad \\phi^{\\prime\\prime}(0)=(d^t)^T\\nabla^2 F(x^t)d^t<0.\r\n\\]\r\n Using these, we can observe that there exists a local minimizer $\\alpha^*\\in(0,\\theta^j)$ of $\\phi$ such that $\\phi(\\alpha^*)<\\phi(0)$, namely,  $F(x^t+\\alpha^*d^t)< F(x^t)$. By the second-order optimality condition of $\\phi$ at $\\alpha^*$, one has $\\phi^{\\prime\\prime}(\\alpha^*)=(d^t)^T\\nabla^2 F(x^t+\\alpha^*d^t)d^t\\ge 0$. Since $F(x^t+\\alpha^*d^t)<F(x^t)\\le F(u^0)$, it follows that \\eqref{F-Hess-Lip} holds for $x=x^t$ and $y=x^t+\\alpha^*d^t$. Using this, the second relation in \\eqref{dk-Hk-dk3} and $(d^t)^T\\nabla^2 F(x^t+\\alpha^*d^t)d^t\\ge0$, we obtain that\r\nin \\eqref{dk-Hk-dk3} and $(d^t)^T\\nabla^2 F(x^t+\\alpha^*d^t)d^t\\ge0$, we obtain that\r\n\\beqa\r\nL_H^F\\alpha^*\\|d^t\\|^3 &\\overset{\\eqref{F-Hess-Lip}}{\\ge}&\\|d^t\\|^2\\|\\nabla^2 F(x^t+\\alpha^*d^t)-\\nabla^2 F(x^t)\\| \\ge (d^t)^T(\\nabla^2 F(x^t+\\alpha^*d^t)-\\nabla^2 F(x^t))d^t \\nn \\\\\r\n&\\ge& -(d^t)^T\\nabla^2 F(x^t)d^t=\\|d^t\\|^3. \\label{lwbd-dt-nc-j1}\r\n\\eeqa\r\nRecall from \\eqref{dk-Hk-dk3} that $d^t\\neq 0$. It then follows from \\eqref{lwbd-dt-nc-j1} that $\\alpha^*\\ge 1/L_H^F$, which along with $\\theta^j>\\alpha^*$ implies that $ \\theta^j> 1/L_H^F$. Hence, \\eqref{lwbd-at-nc} holds in this case.\r\n\r\nCase 2) $F(x^t+\\theta^jd^t)\\le F(x^t)$. It follows from this and $F(x^t)\\le F(u^0)$ that \\eqref{desc-ineq} holds for $x=x^t$ and $y=x^t+\\theta^jd^t$. By this and the fact that $j$ violates \\eqref{ls-nc}, one has\r\n\\begin{equation*}\\label{lwbd-dt-nc}\r\n\\begin{array}{lcl}\r\n-\\frac{\\eta}{2}\\theta^{2j}\\|d^t\\|^3 &\\le& F(x^t+\\theta^j d^t) - F(x^t) \\overset{\\eqref{desc-ineq}}{\\le} \\theta^j \\nabla F(x^t)^Td^t + \\frac{\\theta^{2j}}{2}(d^t)^T\\nabla^2 F(x^t) d^t + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3\\\\[6pt]\r\n&\\overset{\\eqref{dk-Hk-dk3}}{\\le}& -\\frac{\\theta^{2j}}{2}\\|d^t\\|^3 + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3,\r\n\\end{array}\r\n\\end{equation*}\r\nwhich together with $d^t\\neq 0$ implies that $\\theta^j\\ge 3(1-\\eta)/L_H^F$. Hence, \\eqref{lwbd-at-nc} also holds  in this case.\r\n\r\nCombining the above two cases, we conclude that \\eqref{lwbd-at-nc} holds for any $j\\ge0$ that violates \\eqref{ls-nc}. By this and $\\theta\\in(0,1)$, one can see that all $j\\ge0$ that violate \\eqref{ls-nc} must be bounded above. It then follows that the step length $\\alpha_t$ associated with \\eqref{ls-nc} is well-defined. We next derive a lower bound for $\\alpha_t$. Notice from the definition of $j_t$ in Algorithm~\\ref{alg:NCG} that $j=j_t-1$ violates \\eqref{ls-nc} and hence \\eqref{lwbd-at-nc} holds for $j=j_t-1$. Then, by \\eqref{lwbd-at-nc} with $j=j_t-1$ and $\\alpha_t = \\theta^{j_t}$, one has $\\alpha_t=\\theta^{j_t} \\ge \\min\\{\\theta/L_H^F,3(1-\\eta)\\theta/L_H^F\\}$, which immediately yields $\\alpha_t\\ge \\min\\{1,\\theta/L_H^F,3(1-\\eta)\\theta/L_H^F\\}$ as desired.\r\n\r\nWe next prove statement (ii) by considering two separate cases below.\r\n\r\nCase 1) $d^t$ results from the output $d$ of Algorithm~\\ref{alg:capped-CG} with d$\\_$type=NC. It then follows from  \\eqref{NC-ppty} that $\\|d^t\\|\\ge\\epsilon_H$. \r\n%{ This together with item (i) and \\eqref{ls-nc} implies (ii).}\r\nThis together with  \\eqref{ls-nc} and statement (i) implies that statement (ii) holds.\r\n\r\nCase 2) $d^t$ results from the output $v$ of Algorithm~\\ref{pro:meo}. Notice from\r\nAlgorithm~\\ref{pro:meo} that $\\|v\\|=1$ and $v^T\\nabla^2F(x^t)v\\le-\\epsilon_H/2$, which along with \\eqref{dk-2nd-nc} yields $\\|d^t\\|\\ge\\epsilon_H/2$. By this, \\eqref{ls-nc} and statement (i), one can see that statement (ii) again holds.\r\n\\end{proof}\n\n\\begin{proof}[Proof of Theorem \\ref{thm:NCG-iter-oper-cmplxity}]\r\nFor notational convenience, we let $\\{x^t\\}_{t\\in\\bT}$ denote all the iterates generated by Algorithm~\\ref{alg:NCG}, where $\\bT$ is a set of consecutive nonnegative integers starting from $0$. Notice that $F$ is descent along the iterates generated by Algorithm~\\ref{alg:NCG}, which together with $x^0=u^0$ implies that $x^t\\in\\{x:F(x)\\le F(u^0)\\}$. It then follows from\r\n\\eqref{lwbd-Hgupbd} that $\\|\\nabla^2 F(x^t)\\|\\le U^F_H$ holds for all $t\\in\\bT$.\r\n\r\n(i) Suppose for contradiction that the total number of calls of Algorithm~\\ref{pro:meo} in Algorithm~\\ref{alg:NCG} is more than $T_2$. Notice from Algorithm~\\ref{alg:NCG} and Lemma~\\ref{lem:nc}(ii) that each of these calls, except the last one, returns a sufficiently negative curvature direction, and each of them results in a reduction on $F$ of at least $c_{\\rm nc} \\epsilon_H^3$. Hence, \r\n\\[\r\nT_2 c_{\\rm nc} \\epsilon_H^3 \\le \\sum_{t\\in\\bT} [F(x^t)-F(x^{t+1})] \\le F(x^0)-\\Fl = \\Fh-\\Fl,\r\n\\]\r\nwhich contradicts the definition of $T_2$ given in \\eqref{T1}. Hence, statement (i) of Theorem \\ref{thm:NCG-iter-oper-cmplxity} holds.\r\n\r\n(ii) Suppose for contradiction that the total number of calls of Algorithm~\\ref{alg:capped-CG} in Algorithm~\\ref{alg:NCG} is more than $T_1$. Observe that if Algorithm~\\ref{alg:capped-CG} is called at some iteration $t$ and generates the next iterate $x^{t+1}$ satisfying $\\|\\nabla F(x^{t+1})\\|\\le\\epsilon_g$, then Algorithm~\\ref{pro:meo} must be called at the next iteration $t+1$. In view of this and statement (i) of Theorem \\ref{thm:NCG-iter-oper-cmplxity}, we see that the total number of such iterations $t$ is at most $T_2$. Hence, the total number of iterations $t$ of Algorithm~\\ref{alg:NCG} at which Algorithm~\\ref{alg:capped-CG} is called and generates the next iterate $x^{t+1}$ satisfying $\\|\\nabla F(x^{t+1})\\|>\\epsilon_g$ is at least $T_1-T_2+1$. Moreover, for each of such iterations $t$, we observe from Lemmas~\\ref{lem:sol}(ii) and \\ref{lem:nc}(ii) that $F(x^t)-F(x^{t+1})\\ge \\min\\{c_{\\rm sol},c_{\\rm nc}\\}\\min\\{\\epsilon_g^2\\epsilon_H^{-1},\\epsilon_H^3\\}$. It then follows that \r\n\\[\r\n(T_1-T_2+1) \\min\\{c_{\\rm sol},c_{\\rm nc}\\}\\min\\{\\epsilon_g^2\\epsilon_H^{-1},\\epsilon_H^3\\} \\le \\sum_{t\\in\\bT}[F(x^t)-F(x^{t+1})]\\le \\Fh-\\Fl,\r\n\\]\r\nwhich contradicts the definition of $T_1$ and $T_2$ given in \\eqref{T1}. Hence, statement (ii) of Theorem \\ref{thm:NCG-iter-oper-cmplxity} holds.\r\n\r\n(iii) Notice that either Algorithm~\\ref{alg:capped-CG} or \\ref{pro:meo} is called at each iteration of Algorithm~\\ref{alg:NCG}. It follows from this and statements (i) and (ii) of Theorem \\ref{thm:NCG-iter-oper-cmplxity} that the total number of iterations of Algorithm~\\ref{alg:NCG} is at most $T_1+T_2$. In addition, the relation \\eqref{NCG-iter} follows from \\eqref{csol}, \\eqref{cnc} and \\eqref{T1}. One can also observe that the output $x^t$ of Algorithm~\\ref{alg:NCG} satisfies $\\|\\nabla F(x^t)\\|\\le \\epsilon_g$ deterministically and $\\lambda_{\\min}(\\nabla^2F(x^t))\\ge-\\epsilon_H$ with probability at least $1-\\delta$ for some $0 \\le t \\le T_1+T_2$, where the latter part is due to\r\nAlgorithm~\\ref{pro:meo}. This completes the proof of statement (ii) of Theorem \\ref{thm:NCG-iter-oper-cmplxity}.\r\n\r\n(iv) By Theorem~\\ref{lem:capped-CG} with $(H,\\varepsilon)=(\\nabla^2 F(x^t),\\epsilon_H)$ and the fact that $\\|\\nabla^2 F(x^t)\\|\\le U^F_H$, one can observe that  the number of Hessian-vector products required by each call of Algorithm~\\ref{alg:capped-CG} with input $U = 0$ is at most $\\widetilde{\\cO}(\\min\\{n,(U_H^F/\\epsilon_H)^{1/2}\\})$. In addition, by Theorem~\\ref{rand-Lanczos} with $(H,\\varepsilon)=(\\nabla^2 F(x^t),\\epsilon_H)$, $\\|\\nabla^2 F(x^t)\\|\\le U^F_H$, and the fact that each iteration of the Lanczos method requires only one matrix-vector product, one can observe that the number of Hessian-vector products required by each call of Algorithm~\\ref{pro:meo} is also at most $\\widetilde{\\cO}(\\min\\{n,(U_H^F/\\epsilon_H)^{1/2}\\})$. Based on these observations and statement (iii) of Theorem \\ref{thm:NCG-iter-oper-cmplxity}, we see that  statement (iv) of this theorem holds.\r\n\\end{proof}\n\nRecall from Assumption~\\ref{asp:lowbd-knownfeas}(a) that $\\|c(z_{\\epsilon_1})\\|\\le\\epsilon_1/2<1$. By virtue of this, \\eqref{lbd} and the definition of $\\tilde{c}$ in\r\n\\eqref{model:equa-cnstr-pert},\r\nwe obtain that\r\n\\begin{equation}\\label{ineq:p-lowbd}\r\nf(x)+\\gamma\\|\\tilde{c}(x)\\|^2 \\ge f(x)+ \\gamma\\|c(x)\\|^2/2 - \\gamma \\|c(z_{\\epsilon_1})\\|^2 \\ge \\fl - \\gamma,\\quad \\forall x\\in\\bR^n.\r\n\\end{equation}\r\n\n\n\\begin{lemma}\\label{tech-1}\r\nSuppose that Assumption \\ref{asp:lowbd-knownfeas} holds. Let $\\gamma$, $\\fh$ and $\\fl$ be given in Assumption \\ref{asp:lowbd-knownfeas}.  Assume that $\\rho>2\\gamma$, $\\lambda\\in\\bR^m$, and $x\\in\\bR^n$ satisfy\r\n\\begin{equation}\\label{AL-upperbound}\r\n\\widetilde{\\cL}(x,\\lambda;\\rho)\\le \\fh,\r\n\\end{equation}\r\n%where $\\fh=f(x^{\\feas})$.\r\n%where $\\gamma$ and $\\fh$ are given in Assumption \\ref{asp:lowbd-knownfeas}.\r\nwhere $\\widetilde{\\cL}$ is defined in \\eqref{tL-tc}.\r\nThen the following statements hold.\r\n\\begin{enumerate}[{\\rm (i)}]\r\n\\item $f(x)\\le \\fh+\\|\\lambda\\|^2/(2\\rho)$.\r\n\\item $\\|\\tilde{c}(x)\\|\\le\\sqrt{2(\\fh-\\fl+\\gamma)/(\\rho-2\\gamma)+\\|\\lambda\\|^2/(\\rho-2\\gamma)^2}+\\|\\lambda\\|/(\\rho-2\\gamma)$.\r\n\\item If $\\rho\\ge\\|\\lambda\\|^2/(2\\tdf)$ for some $\\tdf>0$, then $f(x)\\le \\fh+\\tdf$.\r\n\\item If\r\n%\\begin{equation}\\label{bound-rho-deltac}\r\n%\\rho\\ge\\frac{2(\\fh-\\fl)}{\\delta_c^2}+\\frac{2\\|\\lambda\\|}{\\delta_c}+\\gamma\r\n%\\end{equation}\r\n\\begin{equation}\\label{bound-rho-deltac}\r\n\\rho\\ge 2(\\fh-\\fl+\\gamma)\\tdc^{-2}+ 2\\|\\lambda\\|\\tdc^{-1}+2\\gamma\r\n\\end{equation}\r\nfor some $\\tdc>0$, then $\\|\\tilde{c}(x)\\|\\le\\tdc$.\r\n\\end{enumerate}\r\n\\end{lemma}\n\n\\begin{proof}\r\n(i) It follows from\r\n%{ the definition of { $\\widetilde{\\cL}$} in} \\eqref{tL-tc} and \\eqref{AL-upperbound}\r\n \\eqref{AL-upperbound} and the definition of $\\widetilde{\\cL}$ in \\eqref{tL-tc}\r\nthat\r\n\\[\r\n\\textstyle\r\n\\fh\\ge f(x)+\\lambda^T \\tilde{c}(x)+\\frac{\\rho}{2}\\|\\tilde{c}(x)\\|^2 = f(x)+\\frac{\\rho}{2}\\left\\|\\tilde{c}(x)+\\frac{\\lambda}{\\rho}\\right\\|^2-\\frac{\\|\\lambda\\|^2}{2\\rho} \\ge f(x)-\\frac{\\|\\lambda\\|^2}{2\\rho}.\r\n\\]\r\nHence, statement (i) holds.\r\n\r\n(ii) In view of \\eqref{ineq:p-lowbd} and \\eqref{AL-upperbound}, one has\r\n\\begin{equation*}\\label{ineq:bound-cx-step1}\r\n\\begin{array}{l}\r\n\\fh\\overset{\\eqref{AL-upperbound}}{\\ge}f(x)+\\lambda^T\\tilde{c}(x)+\\frac{\\rho}{2}\\|\\tilde{c}(x)\\|^2=f(x)+\\gamma\\|\\tilde{c}(x)\\|^2\r\n+\\frac{\\rho-2\\gamma}{2}\\left\\|\\tilde{c}(x)+\\frac{\\lambda}{\\rho-2\\gamma}\\right\\|^2-\\frac{\\|\\lambda\\|^2}{2(\\rho-2\\gamma)}\\\\[2pt]\r\n\\ \\ \\ \\ \\overset{\\eqref{ineq:p-lowbd}}{\\ge}\\fl- \\gamma+\\frac{\\rho-2\\gamma}{2}\\left\\|\\tilde{c}(x)+\\frac{\\lambda}{\\rho-2\\gamma}\\right\\|^2-\\frac{\\|\\lambda\\|^2}{2(\\rho-2\\gamma)}.\r\n\\end{array}\r\n\\end{equation*}\r\nIt then follows that $\\left\\|\\tilde{c}(x)+\\frac{\\lambda}{\\rho-2\\gamma}\\right\\|\\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho-2\\gamma}+\\frac{\\|\\lambda\\|^2}{(\\rho-2\\gamma)^2}}$,  which implies that statement (ii) holds.\r\n\r\n(iii) Statement (iii) immediately follows from statement (i) and $\\rho\\ge\\|\\lambda\\|^2/(2\\tdf)$.\r\n\r\n(iv) Suppose that \\eqref{bound-rho-deltac} holds. Multiplying both sides of \\eqref{bound-rho-deltac} by $\\tdc^{2}$ and rearranging the terms, we have\r\n\\[\r\n(\\rho-2\\gamma)\\tdc^2-2\\|\\lambda\\|\\tdc-2(\\fh-\\fl+\\gamma)\\ge0.\r\n\\]\r\nRecall that $\\rho>2\\gamma$ and $\\tdc>0$. Solving this inequality for $\\tdc$ yields \r\n\\[\r\n\\tdc\\ge\\sqrt{2(\\fh-\\fl+\\gamma)/(\\rho-2\\gamma)+\\|\\lambda\\|^2/(\\rho-2\\gamma)^2}+\\|\\lambda\\|/(\\rho-2\\gamma),\r\n\\]\r\nwhich along with statement (ii) implies that $\\|\\tilde{c}(x)\\|\\le\\tdc$. Hence, statement (iv) holds.\r\n\\end{proof}\n\n\\begin{proof}[Proof of Lemma \\ref{lem:level-set-augmented-lagrangian-func}]\r\n(i) Let $x$ be any point such that $\\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)$. It then follows from \\eqref{L-xinit} that $\\widetilde{\\cL}(x,\\lambda^k;\\rho_k) \\le \\fh$. By this, $\\|\\lambda^k\\|\\le\\Lambda$, $\\rho_k \\ge \\rho_0>2\\gamma$, $\\delta_{f,1} \\le \\bdelta_f$, $\\delta_{c,1} \\le \\bdelta_c$, and Lemma \\ref{tech-1} with $(\\lambda,\\rho)=(\\lambda^k,\\rho_k)$,  one has \r\n\\begin{align}\r\nf(x) & \\le \\fh+ \\|\\lambda^k\\|^2/(2\\rho_k)\\le \\fh+ \\Lambda^2/(2\\rho_0)=\\fh+\\delta_{f,1} \\le \\fh+\\bdelta_f, \\nonumber \\\\\r\n\\textstyle\\|\\tilde{c}(x)\\|&\\textstyle\\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_k-2\\gamma}+\\frac{\\|\\lambda^k\\|^2}{(\\rho_k-2\\gamma)^2}}+\\frac{\\|\\lambda^k\\|}{\\rho_k-2\\gamma}\\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_0-2\\gamma}+\\frac{\\Lambda^2}{(\\rho_0-2\\gamma)^2}}+\\frac{\\Lambda}{\\rho_0-2\\gamma}=\\delta_{c,1} \\le \\bdelta_c. \\label{ineq:feasible-bd}\r\n\\end{align}\r\n%\\begin{eqnarray}\r\n%&f(x)\\le \\fh+\\frac{\\|\\lambda^k\\|^2}{2\\rho_k}\\le \\fh+\\frac{\\Lambda^2}{2\\rho_0}=\\fh+\\delta_{f,1} \\le \\fh+\\bdelta_f,\\nn \\\\\r\n%&\\|\\tilde{c}(x)\\|\\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_k-2\\gamma}+\\frac{\\|\\lambda^k\\|^2}{(\\rho_k-2\\gamma)^2}}+\\frac{\\|\\lambda^k\\|}{\\rho_k-2\\gamma}\\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_0-2\\gamma}+\\frac{\\Lambda^2}{(\\rho_0-2\\gamma)^2}}+\\frac{\\Lambda}{\\rho_0-2\\gamma}=\\delta_{c,1} \\le \\bdelta_c.\\label{ineq:feasible-bd}\r\n%\\end{eqnarray}\r\nAlso, recall from the definition of $\\tilde{c}$ in \\eqref{model:equa-cnstr-pert}\r\nand $\\|c(z_{\\epsilon_1})\\|\\le1$ that $\\|c(x)\\|\\le1+\\|\\tilde{c}(x)\\|$. This together with the above inequalities and \\eqref{nearly-feas-level-set} implies $x\\in\\cS(\\delta_{f},\\delta_{c})$. Hence, statement (i) of Lemma \\ref{lem:level-set-augmented-lagrangian-func} holds.\r\n\r\n(ii) Note that $\\inf\\limits_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)=\\inf\\limits_{x\\in\\bR^n}\\{\\widetilde{\\cL}(x,\\lambda^k;\\rho_k):\\widetilde{\\cL}(x,\\lambda^k;\\rho_k) \\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\}$. Consequently, to prove statement (ii) of Lemma \\ref{lem:level-set-augmented-lagrangian-func}, it suffices to show that\r\n\\begin{equation} \\label{L-lwbd}\r\n\\inf\\limits_{x\\in\\bR^n}\\{\\widetilde{\\cL}(x,\\lambda^k;\\rho_k): \\widetilde{\\cL}(x,\\lambda^k;\\rho_k) \\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\} \\ge\\fl - \\gamma-\\Lambda\\bdelta_c.\r\n\\end{equation}\r\nTo this end, let $x$ be any point satisfying $\\widetilde{\\cL}(x,\\lambda^k;\\rho_k) \\le\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)$. We then know from \\eqref{ineq:feasible-bd} that $\\|\\tilde{c}(x)\\|\\le\\bdelta_c$. By this, $\\|\\lambda^k\\|\\le\\Lambda$, $\\rho_k>2\\gamma$, and \\eqref{ineq:p-lowbd}, one has\r\n\\begin{equation*}\\label{ineq:lowerbound-ALfunc} \r\n\\begin{array}{l}\r\n\\widetilde{\\cL}(x,\\lambda^k;\\rho_k) %= f(x)+(\\lambda^k)^T\\tilde{c}(x)+\\frac{\\rho_k}{2}\\|\\tilde{c}(x)\\|^2\\\\[4pt]\r\n=f(x)+\\gamma\\|\\tilde{c}(x)\\|^2+(\\lambda^k)^T\\tilde{c}(x)+\\frac{\\rho_k-2\\gamma}{2}\\|\\tilde{c}(x)\\|^2\\\\[4pt]\r\n\\ge f(x)+\\gamma\\|\\tilde{c}(x)\\|^2 - \\Lambda\\|\\tilde{c}(x)\\| \\ge\\fl - \\gamma-\\Lambda\\bdelta_c,\r\n\\end{array}\r\n\\end{equation*}\r\nand hence \\eqref{L-lwbd} holds as desired.\r\n\\end{proof}\n\n\\begin{proof}[Proof of Theorem \\ref{thm:output-alg1}]\r\nSuppose that Algorithm \\ref{alg:2nd-order-AL-nonconvex} terminates at some iteration $k$, that is, $\\tau_k^g\\le\\epsilon_1$, $\\tau_k^H\\le\\epsilon_2$, and $\\|c(x^{k+1})\\|\\le\\epsilon_1$ hold. Then, by $\\tau_k^g\\le\\epsilon_1$, $\\tilde{\\lambda}^{k+1}=\\lambda^k+\\rho_k\\tilde{c}(x^{k+1})$, $\\nabla \\tilde{c}=\\nabla c$ and the second relation in \\eqref{algstop:1st-order}, one has \r\n \\begin{align*}\r\n\\|\\nabla f(x^{k+1})+\\nabla c(x^{k+1})\\tilde{\\lambda}^{k+1}\\| &=\\|\\nabla f(x^{k+1})+\\nabla \\tilde{c}(x^{k+1})(\\lambda^k+\\rho_k \\tilde{c}(x^{k+1}))\\| \\\\ \r\n&=\\|\\nabla_x\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k)\\|\\le \\tau_k^g \\le \\epsilon_1.\r\n\\end{align*} \r\n Hence, $(x^{k+1},\\tilde{\\lambda}^{k+1})$ satisfies the first relation in \\eqref{optcond:1st-equa-cnstr}. In addition, by \\eqref{algstop:2nd-order} and $\\tau_k^H\\le\\epsilon_2$, one can show that $\\lambda_{\\min}(\\nabla^2_{xx}\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k))\\ge-\\epsilon_2$ with probability at least $1-\\delta$, which leads to $d^T\\nabla^2_{xx}\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k)d \\ge -\\epsilon_2\\|d\\|^2$ for all $d\\in\\bR^n$ with probability at least $1-\\delta$.  Using this, $\\tilde{\\lambda}^{k+1}=\\lambda^k+\\rho_k\\tilde{c}(x^{k+1})$, $\\nabla \\tilde{c} = \\nabla c$, and $\\nabla^2 \\tilde{c}_i=\\nabla^2 c_i$ for $1\\le i\\le m$, we see that with probability at least $1-\\delta$, it holds that \r\n%$d^T (\\nabla^2 f(x^{k+1})+\\sum_{i=1}^m\\tilde{\\lambda}_i^{k+1}\\nabla^2c_i(x^{k+1})+\\rho_k\\nabla c(x^{k+1})\\nabla c(x^{k+1})^T )d\\ge -\\epsilon_2\\|d\\|^2$ for all $d\\in\\bR^n$},\r\n\\[\r\nd^T\\left(\\nabla^2 f(x^{k+1})+\\sum_{i=1}^m\\tilde{\\lambda}_i^{k+1}\\nabla^2c_i(x^{k+1})+\\rho_k\\nabla c(x^{k+1})\\nabla c(x^{k+1})^T\\right)d\\ge -\\epsilon_2\\|d\\|^2 \\ \\  \\forall d\\in\\bR^n,\r\n\\]\r\nwhich implies $d^T(\\nabla^2 f(x^{k+1})+\\sum_{i=1}^m\\tilde{\\lambda}_i^{k+1}\\nabla^2 c_i(x^{k+1}))d\\ge -\\epsilon_2\\|d\\|^2$ for all $d\\in \\cC(x^{k+1})$, where $\\cC(\\cdot)$ is defined in \\eqref{def:critical-cone}. Hence, $(x^{k+1},\\tilde{\\lambda}^{k+1})$ satisfies \\eqref{optcond:2nd-equa-cnstr} with probability at least $1-\\delta$. Combining these with $\\|c(x^{k+1})\\|\\le\\epsilon_1$, we conclude that $x^{k+1}$ is a deterministic $\\epsilon_1$-FOSP of \\eqref{model:equa-cnstr} and an $(\\epsilon_1,\\epsilon_2)$-SOSP of \\eqref{model:equa-cnstr} with probability at least $1-\\delta$. Hence, Theorem \\ref{thm:output-alg1} holds.\r\n\\end{proof}\n\n\\begin{proof}[Proof of Theorem \\ref{thm:out-itr-cmplxity-1}]\r\nIt follows from \\eqref{def:delta0c-rhobar1} that ${\\rho}_{\\epsilon_1}\\ge 2\\rho_0$.\r\nBy this, one has\r\n\\begin{equation}\\label{T-epsilong}\r\nK_{\\epsilon_1} \\ \\overset{\\eqref{T-epsilon-g}}{=} \\ \\left\\lceil\\log\\epsilon_1/\\log \\omega_1 \\right\\rceil \\ \\overset{\\eqref{omega-tolerance}}{=} \\ \\left\\lceil\\log{2}/\\log{r}\\right\\rceil  \\\r\n\\le \\ \\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})/\\log{r}+1.\r\n\\end{equation}\r\nNotice that $\\{\\rho_k\\}$ is either unchanged or increased by a ratio $r$ as $k$ increases.\r\nBy this fact and \\eqref{T-epsilong}, we see that\r\n\\begin{equation}\\label{rho:upper-bound-Tepsilong}\r\n\\max_{0 \\le k \\le K_{\\epsilon_1}} \\rho_k \\le{r}^{K_{\\epsilon_1}}\\rho_0\\overset{\\eqref{T-epsilong}}{\\le}{r}^{\\frac{\\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})}{\\log{r}}+1}\\rho_0={r}{\\rho}_{\\epsilon_1}.\r\n\\end{equation}\r\nIn addition, notice that $\\rho_k>2\\gamma$ and $\\|\\lambda^k\\|\\le \\Lambda$. Using these, \\eqref{hbd}, the first relation in \\eqref{algstop:1st-order}, and Lemma \\ref{tech-1}(ii) with $(x,\\lambda,\\rho)=(x^{k+1},\\lambda^k,\\rho_k)$,  we obtain that\r\n\\begin{equation}\\label{bd-ck}\\textstyle\r\n\\|\\tilde{c}(x^{k+1})\\| \\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_k-2\\gamma}+\\frac{\\|\\lambda^k\\|^2}{(\\rho_k-2\\gamma)^2}}+\\frac{\\|\\lambda^k\\|}{\\rho_k-2\\gamma}\\le  \\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_k-2\\gamma}+\\frac{\\Lambda^2}{(\\rho_k-2\\gamma)^2}}+\\frac{\\Lambda}{\\rho_k-2\\gamma}.\r\n\\end{equation}\r\nAlso, we observe from $\\|c(z_{\\epsilon_1})\\|\\le\\epsilon_1/2$ and the definition of $\\tilde{c}$ in\r\n\\eqref{model:equa-cnstr-pert} that\r\n\\begin{equation}\\label{bd-cxk}\r\n\\|c(x^{k+1})\\|\\le\\|\\tilde{c}(x^{k+1})\\|+\\|c(z_{\\epsilon_1})\\|\\le \\|\\tilde{c}(x^{k+1})\\|+\\epsilon_1/2.\r\n\\end{equation}\r\n\r\nWe now prove that $\\overline{K}_{\\epsilon_1}$ is finite. Suppose for contradiction that $\\overline{K}_{\\epsilon_1}$ is infinite. It then follows from this and \\eqref{number-outer-iteration} that $\\|c(x^{k+1})\\|>\\epsilon_1$ for all $k\\ge K_{\\epsilon_1}$, which along with \\eqref{bd-cxk} implies that $\\|\\tilde{c}(x^{k+1})\\|>\\epsilon_1/2$ for all $k\\ge K_{\\epsilon_1}$. It then follows that $\\|\\tilde{c}(x^{k+1})\\|>\\alpha\\|\\tilde{c}(x^k)\\|$ must hold for infinitely many $k$'s. Using this and the update scheme on $\\{\\rho_k\\}$, we deduce that $\\rho_{k+1}={r}\\rho_k$ holds for infinitely many $k$'s, which together with the monotonicity of $\\{\\rho_k\\}$ implies that\r\n$\\rho_k\\to\\infty$ as $k\\to\\infty$. By this and \\eqref{bd-ck}, one can see that  $\\|\\tilde{c}(x^{k+1})\\|\\to0$ as $k\\to\\infty$, which contradicts the fact that $\\|\\tilde{c}(x^{k+1})\\|>\\epsilon_1/2$ holds for all $k\\ge K_{\\epsilon_1}$. Hence, $\\overline{K}_{\\epsilon_1}$ is finite.\r\nIn addition, notice from \\eqref{omega-tolerance}, \\eqref{T-epsilon-g} and \\eqref{epsw1-epsw2} that $(\\tau_k^g,\\tau_k^H)=(\\epsilon_1,\\epsilon_2)$ for all $k\\ge K_{\\epsilon_1}$. This along with the termination criterion of Algorithm \\ref{alg:2nd-order-AL-nonconvex}  and the definition of $\\overline{K}_{\\epsilon_1}$ implies that Algorithm \\ref{alg:2nd-order-AL-nonconvex} must terminate at iteration $\\overline{K}_{\\epsilon_1}$.\r\n\r\nWe next show that \\eqref{outer-iteration-cmplxty} and $\\rho_k\\le{r}{\\rho}_{\\epsilon_1}$ hold for $0 \\le k \\le \\overline{K}_{\\epsilon_1}$ by considering two separate cases below.\r\n\r\nCase 1) $\\|c(x^{K_{\\epsilon_1}+1})\\|\\le\\epsilon_1$.  By this and \\eqref{number-outer-iteration}, one can see that  $\\overline{K}_{\\epsilon_1}=K_{\\epsilon_1}$, which together with \\eqref{T-epsilong} and \\eqref{rho:upper-bound-Tepsilong} implies that \\eqref{outer-iteration-cmplxty} and $\\rho_k\\le{r}{\\rho}_{\\epsilon_1}$ hold for $0 \\le k \\le \\overline{K}_{\\epsilon_1}$.\r\n\r\nCase 2) $\\|c(x^{K_{\\epsilon_1}+1})\\|>\\epsilon_1$. By this and \\eqref{number-outer-iteration}, one can observe that $\\overline{K}_{\\epsilon_1}>K_{\\epsilon_1}$  and also $\\|c(x^{k+1})\\|>\\epsilon_1$ for all $K_{\\epsilon_1} \\le k \\le \\overline{K}_{\\epsilon_1}-1$, which together with \\eqref{bd-cxk} implies\r\n\\begin{equation}\\label{lbd-tc}\r\n\\|\\tilde{c}(x^{k+1})\\|>\\epsilon_1/2,\\quad \\forall K_{\\epsilon_1} \\le k \\le \\overline{K}_{\\epsilon_1}-1.\r\n\\end{equation}\r\nIt then follows from $\\|\\lambda^k\\|\\le\\Lambda$, \\eqref{hbd}, the first relation in \\eqref{algstop:1st-order}, and Lemma \\ref{tech-1}(iv) with $(x,\\lambda,\\rho,\\tdc)=(x^{k+1},\\lambda^k,\\rho_k,\\epsilon_1/2)$ that\r\n\\begin{equation}\\label{bound-rhok}\r\n\\begin{aligned}\r\n\\rho_k &< 8(\\fh-\\fl+\\gamma)\\epsilon_1^{-2}+4\\|\\lambda^k\\|\\epsilon_1^{-1}+2\\gamma\\\\\r\n&\\le 8(\\fh-\\fl+\\gamma)\\epsilon_1^{-2}+4\\Lambda\\epsilon_1^{-1}+2\\gamma\\overset{\\eqref{def:delta0c-rhobar1}}{\\le}{\\rho}_{\\epsilon_1},\\quad\r\n\\forall K_{\\epsilon_1} \\le k \\le \\overline{K}_{\\epsilon_1}-1.\r\n\\end{aligned}\r\n\\end{equation}\r\nCombining this relation, \\eqref{rho:upper-bound-Tepsilong}, and the fact $\\rho_{\\overline{K}_{\\epsilon_1}}\\le {r}\\rho_{\\overline{K}_{\\epsilon_1}-1}$, we conclude that  $\\rho_k\\le{r}{\\rho}_{\\epsilon_1}$ holds for $0 \\le k\\le \\overline{K}_{\\epsilon_1}$. It remains to\r\nshow that \\eqref{outer-iteration-cmplxty} holds.  To this end, let \r\n\\[\r\n\\bK =\\{k:\\rho_{k+1}={r}\\rho_{k}, K_{\\epsilon_1}\\le k\\le \\overline{K}_{\\epsilon_1}-2\\}.\r\n\\]\r\nIt follows from \\eqref{bound-rhok} and the update scheme of $\\rho_k$ that \r\n\\[\r\n{r}^{|\\bK|}\\rho_{K_{\\epsilon_1}}=\\max_{K_{\\epsilon_1}\\le k \\le \\overline{K}_{\\epsilon_1}-1}\\{\\rho_k\\}\\le {\\rho}_{\\epsilon_1},\r\n\\]\r\nwhich together with $\\rho_{K_{\\epsilon_1}}\\ge\\rho_0$ implies that\r\n\\begin{equation}\\label{bound-cU}\r\n|\\bK|\\le \\log({\\rho}_{\\epsilon_1}\\rho_{K_{\\epsilon_1}}^{-1})/ \\log{r} \\le \\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})/ \\log{r}.\r\n\\end{equation}\r\nLet $\\{k_1,k_2,\\ldots,k_{|\\bK|}\\}$ denote all the elements of $\\mathcal{\\bK}$ arranged in ascending order, and let $k_0=K_{\\epsilon_1}$ and $k_{|\\bK|+1}=\\overline{K}_{\\epsilon_1}-1$. We next derive an upper bound for $k_{j+1}-k_{j}$ for $j=0,1,\\ldots,|\\bK|$. By the definition of $\\bK$, one can observe that $\\rho_{k}=\\rho_{k'}$ for $k_j< k,k'\\le k_{j+1}$. Using this and the update scheme of $\\rho_k$, we deduce that\r\n\\begin{equation}\\label{ineq:sequence-cxk}\r\n\\|\\tilde{c}(x^{k+1})\\|\\le\\alpha\\|\\tilde{c}(x^{k})\\|,\\quad \\forall k_j<k< k_{j+1}.\r\n\\end{equation}\r\nOn the other hand, by \\eqref{def:delta0c-rhobar1xxx}, \\eqref{bd-ck} and $\\rho_k\\ge\\rho_0$, one has $\\|\\tilde{c}(x^{k+1})\\|\\le \\delta_{c,1}$ for $0 \\le k \\le \\overline{K}_{\\epsilon_1}$. By this and \\eqref{lbd-tc}, one can see that\r\n\\begin{equation}\\label{ineq:sequence-cxk2222}\r\n\\epsilon_1/2<\\|\\tilde{c}(x^{k+1})\\|\\le\\delta_{c,1},\\quad \\forall K_{\\epsilon_1} \\le k \\le \\overline{K}_{\\epsilon_1}-1.\r\n\\end{equation}\r\nNow, note that either $k_{j+1} - k_j = 1$ or $k_{j+1} - k_j > 1$. In the latter case, we can apply \\eqref{ineq:sequence-cxk} with $k = k_{j+1}-1,\\ldots,k_j+1$ together with \\eqref{ineq:sequence-cxk2222} to deduce that\r\n\\[\r\n\\epsilon_1/2<\\|\\tilde{c}(x^{k_{j+1}})\\|\\le\\alpha \\|\\tilde{c}(x^{k_{j+1}-1})\\|\\le\\cdots\\le \\alpha^{k_{j+1}-k_j-1}\\|\\tilde{c}(x^{k_{j}+1})\\|\\le \\alpha^{k_{j+1}-k_j-1}\\delta_{c,1},\\quad \\forall j=0,1,\\ldots,|\\bK|.\r\n\\]\r\nCombining these two cases, we have \r\n\\begin{equation}\\label{ineq:subsequence-rho-same}\r\nk_{j+1}-k_{j}\\le |\\log(\\epsilon_1(2\\delta_{c,1})^{-1}))/\\log \\alpha|+1,\\quad \\forall j=0,1,\\ldots,|\\bK|.\r\n\\end{equation}\r\nSumming up these inequalities, and using \\eqref{T-epsilong}, \\eqref{bound-cU},  $k_0=K_{\\epsilon_1}$ and $k_{|\\bK|+1}=\\overline{K}_{\\epsilon_1}-1$, we have\r\n\\begin{eqnarray}\r\n&\\hspace{-.85in}\\overline{K}_{\\epsilon_1} = 1+k_{|\\bK|+1}=1+k_0+\\sum_{j=0}^{|\\bK|}(k_{j+1}-k_{j}) \\overset{\\eqref{ineq:subsequence-rho-same}}{\\le}1+K_{\\epsilon_1}+(|\\bK|+1)\\left(\\left|\\frac{\\log(\\epsilon_1(2\\delta_{c,1})^{-1})}{\\log \\alpha}\\right|+1\\right) \\nonumber \\\\[6pt]\r\n&\\le2+\\frac{\\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})}{\\log{r}}+\\left(\\frac{\\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})}{\\log{r}}+1\\right)\\left(\\left|\\frac{\\log(\\epsilon_1(2\\delta_{c,1})^{-1})}{\\log \\alpha}\\right|+1\\right) =1+ \\left(\\frac{\\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})}{\\log{r}}+1\\right)\\left(\\left|\\frac{\\log(\\epsilon_1(2\\delta_{c,1})^{-1})}{\\log \\alpha}\\right|+2\\right), \\nn\r\n\\end{eqnarray}\r\nwhere the second inequality is due to \\eqref{T-epsilong} and \\eqref{bound-cU}. Hence, \\eqref{outer-iteration-cmplxty} also holds in this case.\r\n\\end{proof}\n\nWe next prove Theorem \\ref{thm:total-iter-cmplxity}. Before proceeding, we introduce some notation that will be used shortly. Let $L_{k,H}$ denote the Lipschitz constant of $\\nabla^2_{xx} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$ on the convex open neighborhood $\\Omega(\\bdelta_{f},\\bdelta_{c})$ of\r\n$\\cS(\\bdelta_f,\\bdelta_c)$, where $\\cS(\\bdelta_f,\\bdelta_c)$ is defined in \\eqref{nearly-feas-level-set}, and let $U_{k,H}=\\sup_{x\\in\\cS(\\bdelta_f,\\bdelta_c)} \\|\\nabla^2_{xx} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\|$. Notice from \\eqref{model:equa-cnstr-pert} and \\eqref{tL-tc} that\r\n\\begin{equation}\\label{eq:hessian-augmented-Lagrangian}\r\n\\nabla^2_{xx}\\widetilde{\\cL}(x,\\lambda^k;\\rho_k) = \\nabla^2 f(x) +\\sum_{i=1}^m\\lambda^k_{i}\\nabla^2 c_i(x) + \\rho_k \\bigg(\\nabla c(x)\\nabla c(x)^T + \\sum_{i=1}^m \\tilde{c}_i(x)\\nabla^2 c_i(x)\\bigg).\r\n\\end{equation}\r\nBy this, $\\|\\lambda^k\\| \\le \\Lambda$, the definition of $\\tilde{c}$, and the Lipschitz continuity of $\\nabla^2 f$ and $\\nabla^2 c_i$'s (see Assumption~\\ref{asp:lowbd-knownfeas}(c)), one can observe that there exist some constants ${L}_{1}$, ${L}_{2}$, ${U}_{1}$ and ${U}_{2}$, depending only on $f$, $c$, $\\Lambda$, $\\bdelta_{f}$ and $\\bdelta_{c}$,  such that\r\n\\begin{equation}\\label{ineq:bound-norm-hessian-Liphessian}\r\nL_{k,H}\\le {L}_{1}+\\rho_k{L}_{2},\\quad U_{k,H}\\le{U}_{1}+\\rho_k{U}_{2}.\r\n\\end{equation}\r\n\n\n\\begin{proof}[Proof of Theorem \\ref{thm:total-iter-cmplxity}]\r\nLet $T_k$ and $N_k$ denote the number of iterations and matrix-vector products performed by Algorithm~\\ref{alg:NCG} at the outer iteration $k$ of Algorithm \\ref{alg:2nd-order-AL-nonconvex}, respectively. It then follows from Theorem \\ref{thm:out-itr-cmplxity-1} that the total number of iterations and matrix-vector products performed by Algorithm~\\ref{alg:NCG} in Algorithm \\ref{alg:2nd-order-AL-nonconvex} are $\\sum_{k=0}^{\\overline{K}_{\\epsilon_1}}T_k$ and $\\sum_{k=0}^{\\overline{K}_{\\epsilon_1}}N_k$, respectively. In addition, notice from \\eqref{def:delta0c-rhobar1} and Theorem \\ref{thm:out-itr-cmplxity-1} that ${\\rho}_{\\epsilon_1}=\\cO(\\epsilon_1^{-2})$ and  $\\rho_k\\le{r}{\\rho}_{\\epsilon_1}$, which yield $\\rho_k=\\cO(\\epsilon_1^{-2})$.\r\n\r\n%(i) {  We first prove that $(\\tau_k^g)^2/\\tau_k^H\\ge\\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$ holds for any $k\\ge0$. If $\\epsilon_1^2{ \\ge}\\epsilon_2$, it follows from $\\tau_k^g\\ge\\epsilon_1$ and $\\tau_k^H\\le 1$ that $(\\tau_k^g)^2/\\tau_k^H\\ge(\\tau_k^g)^2\\ge \\epsilon_1^2{ \\ge}\\epsilon_2$, which along with $\\epsilon_2\\in(0,1)$ implies $(\\tau_k^g)^2/\\tau_k^H{ \\ge}\\epsilon_2^3$. On the other hand, if $\\epsilon_1^2{ <}\\epsilon_2$, it follows from \\eqref{omega-tolerance} that $\\omega_1^2/\\omega_2{ <} 1$. Also, recall from \\eqref{omega-tolerance}, \\eqref{T-epsilon-g} and \\eqref{epsw1-epsw2} that\r\n%\\[\r\n%K_{\\epsilon_1}=\\left\\lceil \\log\\epsilon_1/\\log\\omega_1\\right\\rceil=\\left\\lceil \\log\\epsilon_2/\\log\\omega_2\\right\\rceil,\\quad\r\n%(\\tau_k^g,\\tau_k^H)=\\left\\{\\begin{array}{ll}\r\n%(\\omega_1^k,\\omega_2^k)&\\text{if }\\ 0\\le k< K_{\\epsilon_1},\\\\\r\n%(\\epsilon_1,\\epsilon_2)&\\text{otherwise}.\r\n%\\end{array}\\right.\r\n%\\]\r\n%In view of these, we see that the sequence $\\{(\\tau_k^g)^2/\\tau_k^H\\}$ is geometrically decreasing with ratio $\\omega_1^2/\\omega_2{ <} 1$ when $0\\le k< K_{\\epsilon_1}$ and equals $\\epsilon_1^2/\\epsilon_2$ for all $k\\ge K_{\\epsilon_1}$. In addition, notice from \\eqref{T-epsilon-g} and \\eqref{epsw1-epsw2} that\r\n%\\[\r\n%\\left\\lceil\\log(\\epsilon_1^2/\\epsilon_2)/\\log(\\omega_1^2/\\omega_2)\\right\\rceil =\\left\\lceil \\log 2/\\log r\\right\\rceil = K_{\\epsilon_1}.\r\n%\\]\r\n%Thus, we have $(\\tau_k^g)^{2}/\\tau_k^H=\\max\\{\\epsilon_1^2/\\epsilon_2,(\\omega_1^2/\\omega_2)^k\\}\\ge\\epsilon_1^2/\\epsilon_2$ when $\\epsilon_1^2{ <}\\epsilon_2$. Combining this with the fact that $(\\tau_k^g)^2/\\tau_k^H{ \\ge}\\epsilon_2^3$ when $\\epsilon_1^2{ \\ge}\\epsilon_2$, we obtain that $(\\tau_k^g)^2/\\tau_k^H\\ge\\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$ holds for any $k\\ge0$.}\r\n\r\nWe first claim that $(\\tau_k^g)^2/\\tau_k^H\\ge\\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$ holds for any $k\\ge0$. Indeed, let $\\bar t=\\log\\epsilon_1/\\log\\omega_1$ and $\\psi(t)=\\max\\{\\epsilon_1,\\omega_1^t\\}^2/\\max\\{\\epsilon_2,\\omega_2^t\\}$ for all $t\\in\\bR$. It then follows from \\eqref{epsw1-epsw2} that $\\omega_1^{\\bar t}=\\epsilon_1$ and $\\omega_2^{\\bar t}=\\epsilon_2$. By this and $\\omega_1,\\omega_2\\in(0,1)$, one can observe that  $\\psi(t)=(\\omega_1^2/\\omega_2)^t$ if $t \\leq \\bar t$ and $\\psi(t)=\\epsilon_1^2/\\epsilon_2$ otherwise. This along with $\\epsilon_2\\in(0,1)$ implies that\r\n\\[\r\n\\min_{t\\in[0,\\infty)}\\psi(t) = \\min\\{\\psi(0),\\psi(\\bar t)\\}=\\min\\{1,\\epsilon_1^2/\\epsilon_2\\} \\geq \\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\},\r\n\\]\r\nwhich together with \\eqref{omega-tolerance} yields\r\n$(\\tau_k^g)^2/\\tau_k^H=\\psi(k) \\geq \\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$ for all $k \\geq 0$.\r\n\r\n(i) From Lemma \\ref{lem:level-set-augmented-lagrangian-func}(i) and the definitions of $\\Omega(\\bdelta_{f},\\bdelta_{c})$ and $L_{k,H}$, we see that $L_{k,H}$ is a Lipschitz constant of $\\nabla^2_{xx} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$ on a convex open neighborhood  of $\\{x:\\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\}$.\r\n% In addition, by Lemma \\ref{lem:level-set-augmented-lagrangian-func}(ii) and the assumption $\\delta_{c,1} \\le \\bdelta_c$, one has $\\inf_x \\cL(x,\\lambda^k;\\rho_k) \\ge\\fl-\\Lambda\\bdelta_c$.\r\nAlso, recall from Lemma \\ref{lem:level-set-augmented-lagrangian-func}(ii) that $\\inf_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k) $ $\\ge\\fl-\\gamma-\\Lambda\\bdelta_c$. By these, $\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\le \\fh$ (see \\eqref{L-xinit}) and Theorem \\ref{thm:NCG-iter-oper-cmplxity}(iii) with $(\\Fh,\\Fl,L_{H}^F,\\epsilon_g,\\epsilon_H)=(\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k),\\fl-\\gamma-\\Lambda \\bdelta_c,L_{k,H},\\tau_k^g,\\tau_k^H)$, one has\r\n\\begin{equation}\\label{eq:alg1-inner-iter-kthAL}\r\n\\begin{array}{rcl}\r\nT_k&=&\\cO((\\fh-\\fl+\\gamma+\\Lambda\\bdelta_{c})L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\})\\\\[2pt]\r\n&\\overset{\\eqref{ineq:bound-norm-hessian-Liphessian}}{=}&\\cO(\\rho_k^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\})=\\cO(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}),\r\n\\end{array}\r\n\\end{equation}\r\n%\\begin{equation}\\label{eq:alg1-inner-iter-kthAL}\r\n%\\begin{array}{rcl}\r\n%T_k&=&\\cO((\\cL(x^{k}_{\\init},\\lambda^k;\\rho_k)-\\fl+\\gamma+\\Lambda\\bdelta_{c})L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\})\\\\[4pt]\r\n%&=&\\cO((\\fh-\\fl+\\gamma+\\Lambda\\bdelta_{c})L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\})\\\\[4pt]\r\n%&\\overset{\\eqref{ineq:bound-norm-hessian-Liphessian}}{=}&\\cO(\\rho_k^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\})=\\cO(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}),\r\n%\\end{array}\r\n%\\end{equation}\r\nwhere the last equality is from $(\\tau_k^g)^2/\\tau_k^H\\ge\\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$, $\\tau_k^H\\ge\\epsilon_2$, and $\\rho_k=\\cO(\\epsilon_1^{-2})$.\r\n\r\nNext, if %$c$ is assumed to be affine, namely,\r\n$c(x)=Ax-b$ for some $A\\in\\bR^{m\\times n}$ and $b\\in\\bR^m$, then $\\nabla c(x)=A^T$ and $\\nabla^2 c_i(x)=0$ for $1 \\le i \\le m$. By these and \\eqref{eq:hessian-augmented-Lagrangian}, one has $L_{k,H}=\\cO(1)$. Using this and similar arguments as for \\eqref{eq:alg1-inner-iter-kthAL}, we obtain that $T_k=\\cO(\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\})$. By this, \\eqref{eq:alg1-inner-iter-kthAL} and $\\overline{K}_{\\epsilon_1}=\\cO(|\\log\\epsilon_1|^2)$ (see Remark \\ref{order-out-itera-penalty}), we conclude that statement (i) of Theorem \\ref{thm:total-iter-cmplxity} holds.\r\n\r\n (ii) In view of Lemma \\ref{lem:level-set-augmented-lagrangian-func}(i) and the definition of\r\n $U_{k,H}$, one can see that \r\n\\[\r\nU_{k,H} \\ge \\sup_{x\\in\\bR^n}\\{\\|\\nabla^2_{xx} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\|: \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\}.\r\n\\] \r\n Using this, $\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)$ $\\le \\fh$ and Theorem \\ref{thm:NCG-iter-oper-cmplxity}(iv) with $(\\Fh,\\Fl,L_{H}^F,U_{H}^F,\\epsilon_g,\\epsilon_H)=(\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k),\r\n\\fl-\\gamma-\\Lambda\\bdelta_c,L_{k,H},U_{k,H},\\tau_k^g,\\tau_k^H)$, we obtain that\r\n\\begin{equation}\\label{eq:alg1-oper-kthAL}\r\n\\begin{array}{rcl}\r\nN_k&=&\\widetilde{\\cO}((\\fh-\\fl+\\gamma+\\Lambda\\bdelta_c)L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\}\\min\\{n,(U_{k,H}/\\tau_k^H)^{1/2}\\})\\\\[4pt]\r\n&\\overset{\\eqref{ineq:bound-norm-hessian-Liphessian}}{=}&\\widetilde{\\cO}(\\rho_k^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\}\\min\\{n,(\\rho_k/\\tau_k^H)^{1/2}\\})\\\\[4pt]\r\n&=&\\widetilde{\\cO}(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}\\min\\{n,\\epsilon_1^{-1}\\epsilon_2^{-1/2}\\}),\t\r\n\\end{array}\r\n\\end{equation}\r\n%\\begin{equation}\\label{eq:alg1-oper-kthAL}\r\n%\\begin{array}{rcl}\r\n%\\hspace{-2mm}N_k&=&\\widetilde{\\cO}((\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)-\\fl+\\gamma+\\Lambda\\bdelta_c)L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\}\\min\\{n,(U_{k,H}/\\tau_k^H)^{1/2}\\})\\\\[4pt]\r\n%&=&\\widetilde{\\cO}((\\fh-\\fl+\\gamma+\\Lambda\\bdelta_c)L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\}\\min\\{n,(U_{k,H}/\\tau_k^H)^{1/2}\\})\\\\[4pt]\r\n%&\\overset{\\eqref{ineq:bound-norm-hessian-Liphessian}}{=}&\\widetilde{\\cO}(\\rho_k^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\}\\min\\{n,(\\rho_k/\\tau_k^H)^{1/2}\\})\\\\[4pt]\r\n%&=&\\widetilde{\\cO}(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}\\min\\{n,\\epsilon_1^{-1}\\epsilon_2^{-1/2}\\}),\t\r\n%\\end{array}\r\n%\\end{equation}\r\nwhere the last equality  is from $(\\tau_k^g)^2/\\tau_k^H\\ge\\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$, $\\tau_k^H\\ge\\epsilon_2$, and $\\rho_k=\\cO(\\epsilon_1^{-2})$.\r\n\r\nOn the other hand, if $c$ is assumed to be affine, it follows from the above discussion that $L_{k,H}=\\cO(1)$. Using this, $U_{k,H}\\le U_1+\\rho_k U_2$, and similar arguments as for \\eqref{eq:alg1-oper-kthAL}, we obtain that $N_k=\\widetilde{\\cO}(\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}\\min\\{n,\\epsilon_1^{-1}\\epsilon_2^{-1/2}\\})$. By this, \\eqref{eq:alg1-oper-kthAL} and $\\overline{K}_{\\epsilon_1}=\\cO(|\\log\\epsilon_1|^2)$ (see Remark \\ref{order-out-itera-penalty}), we conclude that statement (ii) of Theorem \\ref{thm:total-iter-cmplxity} holds.\r\n\\end{proof}\n\n\\begin{theorem}[{\\bf iteration complexity of Algorithm \\ref{alg:capped-CG}}]\\label{lem:capped-CG}\r\nConsider applying Algorithm~\\ref{alg:capped-CG} with input $U = 0$ to the linear system~\\eqref{indef-sys} with $g\\neq 0$, $\\varepsilon>0$, and $H$ being an $n\\times n$ symmetric matrix. Then the number of iterations of Algorithm~\\ref{alg:capped-CG} is $\\widetilde{\\cO}(\\min\\{n,\\sqrt{\\|H\\|/\\varepsilon}\\})$.\r\n\\end{theorem}\n\n\\begin{proof}\r\nFrom \\cite[Lemma~1]{RNW18}, we know that the number of iterations of Algorithm \\ref{alg:capped-CG} is bounded by $\\min\\{n,J(U,\\varepsilon,\\zeta)\\}$, where $J(U,\\varepsilon,\\zeta)$ is the smallest integer $J$ such that $\\sqrt{T}\\tau^{J/2}\\le\\widehat{\\zeta}$, with $U,\\widehat{\\zeta},T$ and $\\tau$ being the values returned by Algorithm \\ref{alg:capped-CG}. In addition, it was shown in \\cite[Section 3.1]{RNW18} that\r\n$J(U,\\varepsilon,\\zeta)\\le\\left\\lceil \\left(\\sqrt{\\kappa}+\\frac{1}{2}\\right)\\ln\\left(\\frac{144(\\sqrt{\\kappa}+1)^2\\kappa^6}{\\zeta^2}\\right)\\right\\rceil$,\r\nwhere $\\kappa={\\cO}(U/\\varepsilon)$ is an output by Algorithm \\ref{alg:capped-CG}. Then one can see that $J(U,\\varepsilon,\\zeta)=\\widetilde{\\cO}(\\sqrt{U/\\varepsilon})$. Notice from\r\nAlgorithm~\\ref{alg:capped-CG} that the output $U\\le\\|H\\|$. Combining these, we obtain the conclusion as desired.\r\n\\end{proof}\n\n\\begin{theorem}[{\\bf iteration complexity of Algorithm~\\ref{pro:meo}}]\\label{rand-Lanczos}\r\nConsider Algorithm~\\ref{pro:meo} with tolerance $\\varepsilon>0$, probability parameter $\\delta\\in(0,1)$, and symmetric matrix $H\\in\\bR^{n\\times n}$ as its input. Then it either finds a sufficiently negative curvature direction $v$ satisfying $v^THv\\le-\\varepsilon/2$ and $\\|v\\|=1$ or certifies that $\\lambda_{\\min}(H)\\ge-\\varepsilon$ holds with probability at least  $1-\\delta$  in at most $N(\\varepsilon,\\delta)$ iterations, where $N(\\varepsilon,\\delta)$ is defined in \\eqref{N-iter}.\r\n\\end{theorem}","units":[{"start":45742,"end":47805,"start_line":442,"end_line":472,"raw_tex":"\\begin{theorem}%[{{\\bf Complexity of Algorithm~\\ref{alg:NCG}}}]\r\n\\label{thm:NCG-iter-oper-cmplxity}\r\nSuppose that Assumption~\\ref{asp:NCG-cmplxity} holds. Let \r\n\\begin{equation}\r\nT_1:=\\left\\lceil\\frac{\\Fh-\\Fl}{\\min\\{c_{\\rm sol},c_{\\rm nc}\\}}\\max\\{\\epsilon_g^{-2}\\epsilon_H,\\epsilon_H^{-3}\\}\\right\\rceil+\\left\\lceil\\frac{\\Fh-\\Fl}{c_{\\rm nc}}\\epsilon_H^{-3}\\right\\rceil+1, \\\r\nT_2:=\\left\\lceil\\frac{\\Fh-\\Fl}{c_{\\rm nc}}\\epsilon_H^{-3}\\right\\rceil+1,\\label{T1}\r\n\\end{equation}\r\n%where $\\Fh=F(u^0)$, $\\Fl$ is given in \\eqref{lwbd-Hgupbd}, and $c_{\\rm sol}$ and $c_{\\rm nc}$ are defined in \\eqref{csol} and \\eqref{cnc}, respectively.\r\nwhere $\\Fh=F(u^0)$, $\\Fl$ is given in \\eqref{lwbd-Hgupbd}, and \r\n\\beqa\r\n&&c_{\\rm sol} :=\\eta\\min\\left\\{\\left[\\frac{4}{4+\\zeta+\\sqrt{(4+\\zeta)^2+8L^F_H}}\\right]^2,\\left[\\frac{\\min\\{6(1-\\eta),2\\}\\theta}{L^F_H}\\right]^2\\right\\}, \\label{csol} \\\\\r\n&& c_{\\rm nc} := \\frac{\\eta}{16} \\min\\left\\{1,\\left[\\frac{\\min\\{3(1-\\eta),1\\}\\theta}{L_H^F}\\right]^2\\right\\}.   \\label{cnc}\r\n\\eeqa\r\n Then the following statements hold.\r\n\\begin{enumerate}[{\\rm (i)}]\r\n\\item The total number of calls of Algorithm~\\ref{pro:meo} in Algorithm~\\ref{alg:NCG} is at most $T_2$.\t\r\n\\item The total number of calls of Algorithm~\\ref{alg:capped-CG} in Algorithm~\\ref{alg:NCG} is at most $T_1$.\r\n\\item {\\rm ({\\bf iteration complexity})} Algorithm~\\ref{alg:NCG} terminates in at most $T_1+T_2$ iterations with \r\n\\begin{equation}\\label{NCG-iter} \r\nT_1+T_2=\\cO((\\Fh-\\Fl)(L_H^F)^2\\max\\{\\epsilon_g^{-2}\\epsilon_H,\\epsilon_H^{-3}\\}).\r\n\\end{equation}\r\nAlso, its output $x^t$ satisfies $\\|\\nabla F(x^t)\\|\\le \\epsilon_g$ deterministically and\r\n$\\lambda_{\\min}(\\nabla^2F(x^t))$ $\\ge-\\epsilon_H$ with probability at least $1-\\delta$ for\r\nsome $0 \\le t \\le T_1+T_2$.\r\n\\item {\\rm ({\\bf operation complexity})} Algorithm~\\ref{alg:NCG} requires at most \r\n\\[\r\n\\widetilde{\\cO}((\\Fh-\\Fl)(L_H^F)^2\\max\\{\\epsilon_g^{-2}\\epsilon_H,\\epsilon_H^{-3}\\}\\min\\{n,(U_H^F/\\epsilon_H)^{1/2}\\})\r\n\\]\r\nmatrix-vector products, where $U_H^F$ is given in \\eqref{lwbd-Hgupbd}.\r\n\\end{enumerate}\r\n\\end{theorem}","sha256_utf8":"3ac7e03cb2167ac9459371655e8a7ce438fc71908c3747d210422e3226ad34fa"},{"start":60569,"end":60892,"start_line":575,"end_line":578,"raw_tex":"To this end, we start by noting from \\eqref{hbd}, \\eqref{model:equa-cnstr-pert}, \\eqref{tL-tc} and \\eqref{def:initial-iterate-subprob} that\r\n\\beq \\label{L-xinit}\r\n\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k) \\le \\max\\{\\widetilde{\\cL}(z_{\\epsilon_1},\\lambda^k;\\rho_k),f(z_{\\epsilon_1})\\}=f(z_{\\epsilon_1})\\le \\fh.\r\n\\eeq\r\n","sha256_utf8":"151b0a168f6900da7bd7afc4c52bbb9a50533f101392e158dbf7717977b6670e"},{"start":61144,"end":62424,"start_line":581,"end_line":592,"raw_tex":"\\begin{lemma}%[{{\\bf level- and lower-boundedness of $\\widetilde{\\cL}(\\cdot,\\lambda^k;\\rho_k)$}}]\r\n\\label{lem:level-set-augmented-lagrangian-func}\r\nSuppose that Assumption~\\ref{asp:lowbd-knownfeas} holds. Let $(\\lambda^k, \\rho_k)$ be generated at the $k$th iteration of Algorithm~\\ref{alg:2nd-order-AL-nonconvex} for some $k \\ge 0$, and $\\cS(\\bdelta_f,\\bdelta_c)$ and $x_{\\init}^k$ be defined in \\eqref{nearly-feas-level-set} and \\eqref{def:initial-iterate-subprob}, respectively, and let $\\fh$, $\\fl$, $\\bdelta_f$ and $\\bdelta_c$ be given in Assumption~\\ref{asp:lowbd-knownfeas}. Suppose that $\\rho_0$ is sufficiently large such that $\\delta_{f,1} \\le \\bdelta_f$ and  $\\delta_{c,1} \\le \\bdelta_c$, where \r\n\\begin{equation} \r\n\\delta_{f,1}:=\\Lambda^2/(2\\rho_0)\\quad and\\quad \\delta_{c,1}:=\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_0-2\\gamma}+\\frac{\\Lambda^2}{(\\rho_0-2\\gamma)^2}}+\\frac{\\Lambda}{\\rho_0-2\\gamma}\\label{def:delta0c-rhobar1xxx}.\r\n\\end{equation}\r\nThen the following statements hold.\r\n\\begin{enumerate}[{\\rm (i)}]\r\n\\item $\\{x:\\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\}\\subseteq \\cS(\\bdelta_f,\\bdelta_c)$.\r\n\\item $\\inf_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k) \\ge\\fl - \\gamma-\\Lambda\\bdelta_c$.\r\n\\end{enumerate}\r\n\\end{lemma}","sha256_utf8":"2e08f3f548c1c40bacea5166473a286bca3fb20064023e573095fa50a2e5eebf"},{"start":62429,"end":64236,"start_line":594,"end_line":596,"raw_tex":"Using Lemma \\ref{lem:level-set-augmented-lagrangian-func}, we can verify that the Newton-CG method (Algorithm \\ref{alg:NCG}), starting with $u^0=x^{k}_{\\init}$, is capable of finding an approximate solution $x^{k+1}$ of the AL subproblem $\\min_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$ satisfying \\eqref{algstop:1st-order} and \\eqref{algstop:2nd-order}. Indeed,\r\nlet $F(\\cdot)=\\widetilde{\\cL}(\\cdot,\\lambda^k;\\rho_k)$ and $u^0=x^{k}_{\\init}$. By these and Lemma \\ref{lem:level-set-augmented-lagrangian-func}, one can see that $\\{x:F(x) \\le F(u^0)\\} \\subseteq \\cS(\\bdelta_f,\\bdelta_c)$. It then follows from this and Assumption \\ref{asp:lowbd-knownfeas}(c) that the level set $\\{x : F(x) \\le F(u^0)\\}$ is compact  and $\\nabla^2 F$ is Lipschitz continuous on a convex open neighborhood of $\\{x:F(x) \\le F(u^0)\\}$. Thus, such $F$ and $u^0$ satisfy Assumption \\ref{asp:NCG-cmplxity}. Based on this and the discussion in Section \\ref{sec:sbpb-solver}, one can conclude that Algorithm~\\ref{alg:NCG}, starting with $u^0=x^{k}_{\\init}$, is applicable to the AL subproblem $\\min_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$. Moreover, it follows from Theorem \\ref{thm:NCG-iter-oper-cmplxity} that this algorithm with $(\\epsilon_g,\\epsilon_H)=(\\tau_k^g,\\tau_k^H)$ can produce a point $x^{k+1}$ satisfying \\eqref{algstop:2nd-order} and also the second relation in \\eqref{algstop:1st-order}. In addition, since this algorithm\r\nis descent and its starting point is $x^{k}_{\\init}$, its output $x^{k+1}$ must satisfy $\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k) \\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)$, which along with \\eqref{L-xinit} implies that $\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k) \\le f(z_{\\epsilon_1})$ and thus $x^{k+1}$ also satisfies the first relation in \\eqref{algstop:1st-order}.\r\n","sha256_utf8":"6771a3fd93555cf7600608162da89662a961cbd3a131ba71be61b2a45f7e65f6"},{"start":64691,"end":65150,"start_line":601,"end_line":604,"raw_tex":"\\begin{theorem}%[{{\\bf Well-definedness of Algorithm \\ref{alg:2nd-order-AL-nonconvex}}}]\r\n\\label{subprob-solver}\r\nUnder the same settings as in Lemma \\ref{lem:level-set-augmented-lagrangian-func}, the Newton-CG method (Algorithm \\ref{alg:NCG}) applied to the AL subproblem $\\min_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$ with $u^0=x^{k}_{\\init}$ finds a point $x^{k+1}$ satisfying \\eqref{algstop:1st-order} and \\eqref{algstop:2nd-order}.\r\n\\end{theorem}","sha256_utf8":"0cc2737c50333c3be13b26f59f0dbca02ef86a8bac1702faf0ed2317d1c4dbdf"},{"start":65552,"end":66219,"start_line":610,"end_line":615,"raw_tex":"\\begin{theorem}%[{{\\bf Output of Algorithm \\ref{alg:2nd-order-AL-nonconvex}}}]\r\n\\label{thm:output-alg1}\r\nSuppose that Assumption~\\ref{asp:lowbd-knownfeas} holds and that $\\rho_0$ is sufficiently large such that $\\delta_{f,1} \\le \\bdelta_f$ and  $\\delta_{c,1} \\le \\bdelta_c$, where\r\n$\\delta_{f,1}$ and $\\delta_{c,1}$ are defined in \\eqref{def:delta0c-rhobar1xxx}.\r\nIf Algorithm \\ref{alg:2nd-order-AL-nonconvex} terminates at some iteration $k$, then $x^{k+1}$ is a deterministic $\\epsilon_1$-FOSP of problem \\eqref{model:equa-cnstr}, and moreover, it is an $(\\epsilon_1,\\epsilon_2)$-SOSP of \\eqref{model:equa-cnstr} with probability at least $1-\\delta$.\r\n\\end{theorem}","sha256_utf8":"7f3ec12c6b91f9a3374d6692d0224a4ecd3e6c937c57ca823666f8457a070216"},{"start":69448,"end":70333,"start_line":645,"end_line":650,"raw_tex":"In view of \\eqref{omega-tolerance}, \\eqref{T-epsilon-g} and the fact that\r\n\\begin{equation}\\label{epsw1-epsw2}\r\n%\\log\\epsilon_1/\\log\\omega_1=\\log2/\\log r=\\log\\epsilon_2/\\log\\omega_2,\r\n\\log\\epsilon_1/\\log\\omega_1=\\log\\epsilon_2/\\log\\omega_2=\\log2/\\log r,\r\n\\end{equation}\r\nwe see that $(\\tau_k^g,\\tau_k^H)=(\\epsilon_1,\\epsilon_2)$ for all $k\\ge K_{\\epsilon_1}$. This along with the termination criterion of Algorithm \\ref{alg:2nd-order-AL-nonconvex} implies that it runs for at least $K_{\\epsilon_1}$ iterations and  terminates once $\\|c(x^{k+1})\\|\\le\\epsilon_1$ for some $k\\ge K_{\\epsilon_1}$. As a result, to establish outer iteration complexity of Algorithm \\ref{alg:2nd-order-AL-nonconvex}, it suffices to bound such  $k$. The resulting outer iteration complexity of Algorithm \\ref{alg:2nd-order-AL-nonconvex} is presented below, whose proof is deferred to Section \\ref{sec:pf-AL}.\r\n","sha256_utf8":"a48fb043b2885e9e88961d18be9150c7fa29b62b368b998cb181596b79a69d69"},{"start":70335,"end":71674,"start_line":652,"end_line":665,"raw_tex":"\\begin{theorem}%[{{\\bf Outer iteration complexity of Algorithm \\ref{alg:2nd-order-AL-nonconvex}}}]\r\n\\label{thm:out-itr-cmplxity-1}\r\nSuppose that Assumption~\\ref{asp:lowbd-knownfeas} holds and that $\\rho_0$ is sufficiently large such that $\\delta_{f,1} \\le \\bdelta_f$ and  $\\delta_{c,1} \\le \\bdelta_c$, where\r\n$\\delta_{f,1}$ and $\\delta_{c,1}$ are defined in \\eqref{def:delta0c-rhobar1xxx}. Let\r\n\\begin{eqnarray}\r\n&&{\\rho}_{\\epsilon_1}:=\\max\\left\\{8(\\fh-\\fl+\\gamma)\\epsilon_1^{-2}+4\\Lambda\\epsilon_1^{-1}+2\\gamma,2\\rho_0\\right\\},\\label{def:delta0c-rhobar1}\\\\\r\n&&\\overline{K}_{\\epsilon_1}:=\\inf\\{k\\ge K_{\\epsilon_1}: \\|c(x^{k+1})\\|\\le\\epsilon_1\\},\\label{number-outer-iteration}\r\n\\end{eqnarray}\r\nwhere $K_{\\epsilon_1}$ is defined in \\eqref{T-epsilon-g},  and $\\gamma$, $\\fh$ and $\\fl$ are given in Assumption \\ref{asp:lowbd-knownfeas}. Then $\\overline{K}_{\\epsilon_1}$ is finite, and Algorithm \\ref{alg:2nd-order-AL-nonconvex} terminates at iteration $\\overline{K}_{\\epsilon_1}$ with\r\n\\begin{equation}\\label{outer-iteration-cmplxty}\r\n\\overline{K}_{\\epsilon_1}\\le\\left(\\frac{\\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})}{\\log{r}}+1\\right)\\left(\\left|\\frac{\\log(\\epsilon_1(2\\delta_{c,1})^{-1})}{\\log \\alpha}\\right|+2\\right)+1.\r\n\\end{equation}\r\nMoreover, $\\rho_k\\le{r}{\\rho}_{\\epsilon_1}$ holds for $0 \\le k \\le \\overline{K}_{\\epsilon_1}$\r\n\\end{theorem}","sha256_utf8":"bff7cab74fae8a2400d41fb0c8c88c69f3fb0758a2e7717546a7e474aea613dd"},{"start":90352,"end":91321,"start_line":828,"end_line":843,"raw_tex":"\\begin{lemma}\\label{lem:SOL-NC-ppty}\r\nSuppose that Assumption~\\ref{asp:NCG-cmplxity} holds and the direction $d^t$ results from  the output $d$ of Algorithm \\ref{alg:capped-CG} with a type specified in d$\\_$type  at some iteration $t$ of Algorithm \\ref{alg:NCG}. Then the following statements hold.\r\n\\begin{enumerate}[{\\rm (i)}]\r\n\\item If d$\\_$type=SOL, then  $d^t$ satisfies\r\n\\beqa\r\n&\\epsilon_H\\|d^t\\|^2\\le (d^t)^T\\left(\\nabla^2 F(x^t)+2\\epsilon_H I\\right)d^t,\\label{SOL-ppty-1}\\\\\r\n&\\|d^t\\|\\le1.1 \\epsilon_H^{-1}\\|\\nabla F(x^t)\\|,\\label{SOL-ppty-2}\\\\\r\n&(d^t)^T\\nabla F(x^t)=-(d^t)^T\\left(\\nabla^2F(x^t)+2\\epsilon_HI\\right)d^t,\\label{SOL-ppty-3}\\\\\r\n&\\|(\\nabla^2F(x^t)+2\\epsilon_HI)d^t+\\nabla F(x^t)\\|\\le\\epsilon_H\\zeta\\|d^t\\|/2.\\label{SOL-ppty-4}\r\n\\eeqa\r\n\\item If d$\\_$type=NC, then  $d^t$ satisfies $(d^t)^T\\nabla F(x^t)\\le0$ and\r\n\\begin{equation}\\label{NC-ppty}\r\n(d^t)^T\\nabla^2F(x^t)d^t/\\|d^t\\|^2=-\\|d^t\\|\\le-\\epsilon_H.\r\n\\end{equation}\r\n\\end{enumerate}\r\n\\end{lemma}","sha256_utf8":"836b543cd346b2b0722fffb46acd32a866b1acf9c964c19d3d7c2d952bf48afa"},{"start":91494,"end":92446,"start_line":847,"end_line":859,"raw_tex":"\\begin{lemma}\\label{lem:sol}\r\nSuppose that Assumption~\\ref{asp:NCG-cmplxity} holds and the direction $d^t$ results from  the output $d$ of Algorithm~\\ref{alg:capped-CG} with d$\\_$type=SOL at some iteration $t$ of Algorithm~\\ref{alg:NCG}. Let $U_g^F$ and $c_{\\rm sol}$ be given in \\eqref{lwbd-Hgupbd} and \\eqref{csol}, respectively.  Then the following statements hold.\r\n\\begin{enumerate}[{\\rm (i)}]\r\n\\item The step length $\\alpha_t$ is well-defined, and moreover,\r\n\\begin{equation}\\label{lwbd-ak-sol}\r\n\\alpha_t\\ge\\min\\left\\{1,\\sqrt{\\frac{\\min\\{6(1-\\eta),2\\}}{1.1 L_H^F U_g^F}}\\theta\\epsilon_H\\right\\}.\r\n\\end{equation}\r\n\\item The next iterate $x^{t+1}=x^t+\\alpha_td^t$ satisfies %$F(x^t)-F(x^{t+1})\\ge c_{\\rm sol}\\min\\{\\|\\nabla F(x^{t+1})\\|^2\\epsilon_H^{-1},\\epsilon_H^3\\}$.\r\n\\begin{equation}\\label{suff-desc-sol}\r\nF(x^t)-F(x^{t+1})\\ge c_{\\rm sol}\\min\\{\\|\\nabla F(x^{t+1})\\|^2\\epsilon_H^{-1},\\epsilon_H^3\\}.\r\n\\end{equation}\r\n\\end{enumerate}\r\n\\end{lemma}","sha256_utf8":"b3bcda6d6344a8b5f0d021f2ccb97298124b433ae6e67225fdc98f62bc26fbab"},{"start":92450,"end":100325,"start_line":861,"end_line":945,"raw_tex":"\\begin{proof}\r\nOne can observe that $F$ is descent along the iterates (whenever well-defined) generated by Algorithm~\\ref{alg:NCG}, which together with $x^0=u^0$ implies that $F(x^t)\\le F(u^0)$ and hence\r\n$\\|\\nabla F(x^t)\\|\\le U^F_g$ due to \\eqref{lwbd-Hgupbd}.\r\n%Since $d^t$ { results from} the output $d$ of Algorithm~\\ref{alg:capped-CG} with d$\\_$type=SOL, it follows from Lemma~\\ref{lem:SOL-NC-ppty}(i) that \\eqref{SOL-ppty-1}-\\eqref{SOL-ppty-4} hold for $d^t$.\r\n%In particular, we see from \\eqref{SOL-ppty-2} and \\eqref{SOL-ppty-4} that $\\|(\\nabla^2F(x^t)+2\\epsilon_HI)d^t+\\nabla F(x^t)\\|\\le0.55\\zeta\\|\\nabla F(x^t)\\|$. Since we also have $\\|\\nabla F(x^t)\\| > \\epsilon_g$ in this case and recall that $ \\zeta \\in (0,1)$, we conclude that $d^t\\neq 0$.\r\nIn addition, since $d^t$ results from the output $d$ of Algorithm~\\ref{alg:capped-CG} with d$\\_$type=SOL, one can see that $\\|\\nabla F(x^t)\\| > \\epsilon_g$ and \\eqref{SOL-ppty-1}-\\eqref{SOL-ppty-4} hold for $d^t$. Moreover, by $\\|\\nabla F(x^t)\\| > \\epsilon_g$ and \\eqref{SOL-ppty-4}, one can conclude that $d^t\\neq 0$.\r\n\r\nWe first prove statement (i). If \\eqref{ls-sol} holds for $j=0$, then $\\alpha_t=1$, which clearly implies that \\eqref{lwbd-ak-sol} holds. We now suppose that \\eqref{ls-sol} fails for $j=0$. Claim that for all $j\\ge0$ that violate \\eqref{ls-sol}, it holds that\r\n\\begin{equation}\\label{lwbd-sol-ak}\r\n\\theta^{2j}\\ge\\min\\{6(1-\\eta),2\\}\\epsilon_H(L^F_H)^{-1}\\|d^t\\|^{-1}.\r\n\\end{equation}\r\nIndeed, suppose that \\eqref{ls-sol} is violated by some $j \\ge 0$. We now show that \\eqref{lwbd-sol-ak} holds for such $j$ by considering two separate cases below.\r\n\r\nCase 1) $F(x^t+\\theta^jd^t)>F(x^t)$. Let $\\phi(\\alpha)=F(x^t+\\alpha d^t)$. Then $\\phi(\\theta^j)>\\phi(0)$. Also, since $d^t\\neq0$, by \\eqref{SOL-ppty-1} and \\eqref{SOL-ppty-3}, one has \r\n\\[\r\n\\phi^\\prime(0)  =\\nabla F(x^t)^Td^t = -(d^t)^T(\\nabla^2 F(x^t)+2\\epsilon_HI)d^t \r\n \\le-\\epsilon_H\\|d^t\\|^2< 0.\r\n\\]\r\nUsing these, we can observe that there exists a local minimizer $\\alpha^*\\in(0,\\theta^j)$ of $\\phi$ such that $\\phi^\\prime(\\alpha^*)=\\nabla F(x^t+\\alpha^*d^t)^Td^t=0$ and $\\phi(\\alpha^*)<\\phi(0)$, which implies that $F(x^t+\\alpha^*d^t)< F(x^t)\\le F(u^0)$. Hence, \\eqref{apx-nxt-grad} holds for $x=x^t$ and $y=x^t+\\alpha^*d^t$. Using this, $0<\\alpha^*<\\theta^j\\le1$ and $\\nabla F(x^t+\\alpha^*d^t)^Td^t=0$, we obtain\r\n\\[\r\n\\begin{array}{l}\r\n\\frac{(\\alpha^*)^2L_H^F}{2}\\|d^t\\|^3\\overset{\\eqref{apx-nxt-grad}}{\\ge}\\|d^t\\|\\|\\nabla F(x^t+\\alpha^*d^t)-\\nabla F(x^t)-\\alpha^*\\nabla^2 F(x^t)d^t\\|\\\\[6pt]\r\n\\ge(d^t)^T(\\nabla F(x^t+\\alpha^*d^t)-\\nabla F(x^t)-\\alpha^*\\nabla^2 F(x^t)d^t)=-(d^t)^T\\nabla F(x^t)-\\alpha^*(d^t)^T\\nabla^2 F(x^t)d^t\\\\[6pt]\r\n\\overset{\\eqref{SOL-ppty-3}}{=}(1-\\alpha^*)(d^t)^T(\\nabla^2 F(x^t)+2\\epsilon_HI)d^t +2 \\alpha^*\\epsilon_H\\|d^t\\|^2\\overset{\\eqref{SOL-ppty-1}}{\\ge} (1+\\alpha^*)\\epsilon_H\\|d^t\\|^2\\ge \\epsilon_H \\|d^t\\|^2,\r\n\\end{array}\r\n\\]\r\nwhich along with $d^t\\neq 0$ implies that $(\\alpha^*)^2\\ge 2\\epsilon_H(L_H^F)^{-1}\\|d^t\\|^{-1}$. Using this and $\\theta^{j}>\\alpha^*$, we conclude that \\eqref{lwbd-sol-ak} holds in this case.\r\n\r\nCase 2) $F(x^t+\\theta^jd^t)\\le F(x^t)$. This together with $F(x^t)\\le F(u^0)$ implies that\r\n\\eqref{desc-ineq} holds for $x=x^t$ and $y=x^t+\\theta^jd^t$. Then, because $j$ violates \\eqref{ls-sol}, we obtain\r\n%By this and the fact that $j$ violates \\eqref{ls-sol}, we obtain that\r\n\\beqa\r\n&&-\\eta\\epsilon_H\\theta^{2j}\\|d^t\\|^2\\le F(x^t+\\theta^jd^t)-F(x^t)\\overset{\\eqref{desc-ineq}}{\\le} \\theta^j \\nabla F(x^t)^Td^t + \\frac{\\theta^{2j}}{2}(d^t)^T\\nabla^2 F(x^t) d^t + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3 \\nn \\\\\r\n&&\\overset{\\eqref{SOL-ppty-3}}{=}-\\theta^j (d^t)^T(\\nabla^2 F(x^t) +2\\epsilon_H I)d^t + \\frac{\\theta^{2j}}{2}(d^t)^T\\nabla^2 F(x^t) d^t + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3 \\nn \\\\\r\n&&=-\\theta^j\\left(1-\\frac{\\theta^j}{2}\\right)(d^t)^T(\\nabla^2 F(x^t) +2\\epsilon_H I)d^t - \\theta^{2j} \\epsilon_H\\|d^t\\|^2 + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3 \\nn \\\\\r\n&&\\overset{\\eqref{SOL-ppty-1}}{\\le} -\\theta^j\\left(1-\\frac{\\theta^j}{2}\\right) \\epsilon_H\\|d^t\\|^2 - \\theta^{2j}\\epsilon_H\\|d^t\\|^2 + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3 \\le -\\theta^j \\epsilon_H\\|d^t\\|^2 + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3. \\label{ineq:desc-ls-sol}\r\n\\eeqa\r\nRecall that $d^t \\neq 0$. Dividing both sides of \\eqref{ineq:desc-ls-sol} by $L_H^F\\theta^j\\|d^t\\|^3/6$ and using $\\eta, \\theta\\in(0,1)$, we obtain that \r\n\\[\r\n\\theta^{2j}\\ge 6(1-\\theta^j\\eta)\\epsilon_H(L_H^F)^{-1}\\|d^t\\|^{-1}\\ge6(1-\\eta)\\epsilon_H(L_H^F)^{-1}\\|d^t\\|^{-1}.\r\n\\] \r\n Hence, \\eqref{lwbd-sol-ak} also holds in this case.\r\n\r\nCombining the above two cases, we conclude that  \\eqref{lwbd-sol-ak} holds for any $j\\ge0$ that violates \\eqref{ls-sol}.\r\nBy this and $\\theta\\in(0,1)$, one can see that all $j\\ge0$ that violate \\eqref{ls-sol} must be bounded above. It then follows that the step length $\\alpha_t$ associated with \\eqref{ls-sol} is well-defined. We next prove \\eqref{lwbd-ak-sol}. Observe from the definition of $j_t$ in Algorithm~\\ref{alg:NCG} that $j=j_t-1$ violates \\eqref{ls-sol} and hence \\eqref{lwbd-sol-ak} holds for $j=j_t-1$. Then, by \\eqref{lwbd-sol-ak} with $j=j_t-1$ and $\\alpha_t = \\theta^{j_t}$, one has\r\n\\begin{equation}\\label{lwbd-sol-ak-1}\r\n\\alpha_t = \\theta^{j_t} \\ge\\sqrt{\\min\\{6(1-\\eta),2\\}\\epsilon_H(L^F_H)^{-1}} \\ \\theta\\|d^t\\|^{-1/2},\r\n\\end{equation}\r\nwhich,  along with \\eqref{SOL-ppty-2} and $\\|\\nabla F(x^t)\\|\\le U^F_g$, implies \\eqref{lwbd-ak-sol}. This proves statement (i).\r\n\r\nWe next prove statement (ii) %, particularly,  \\eqref{suff-desc-sol}\r\nby considering two separate cases below.\r\n\r\nCase 1) $\\alpha_t=1$. By this, one knows that \\eqref{ls-sol} holds for $j=0$. It then follows that $F(x^t+d^t)\\le F(x^t)\\le F(u^0)$, which implies that \\eqref{apx-nxt-grad} holds for $x=x^t$ and $y=x^t+d^t$. By this and \\eqref{SOL-ppty-4}, one has\r\n\\[\r\n\\begin{array}{lcl}\r\n%\\|\\nabla F(x^{t+1})\\|=\\|\\nabla F(x^{t}+d^t)-\\nabla F(x^t)+\\nabla F(x^t)\\|\\\\[4pt]\r\n\\|\\nabla F(x^{t+1})\\|=\\|\\nabla F(x^{t}+d^t)\\|\r\n&\\le&\\|\\nabla F(x^t+d^t)-\\nabla F(x^t)-\\nabla^2 F(x^t)d^t\\| \\\\[4pt]\r\n&&+\\|(\\nabla^2 F(x^t)+2\\epsilon_HI)d^t+\\nabla F(x^t)\\|+2\\epsilon_H\\|d^t\\|\\\\[4pt]\r\n&\\le& \\frac{L_H^F}{2}\\|d^t\\|^2 + \\frac{4+\\zeta}{2}\\epsilon_H\\|d^t\\|,\r\n\\end{array}\r\n\\]\r\nwhere the last inequality follows from \\eqref{apx-nxt-grad} and \\eqref{SOL-ppty-4}. Solving the above inequality for $\\|d^t\\|$ and using the fact that $\\|d^t\\|>0$, we obtain that\r\n\\[\r\n\\begin{array}{rcll}\r\n\\|d^t\\|&\\ge&\\frac{-(4+\\zeta)\\epsilon_H + \\sqrt{(4+\\zeta)^2\\epsilon_H^2+8L_H^F\\|\\nabla F(x^{t+1})\\|}}{2L_H^F}\\ \\ge\\\r\n\\frac{-(4+\\zeta)\\epsilon_H + \\sqrt{(4+\\zeta)^2\\epsilon_H^2+8L_H^F\\epsilon_H^2}}{2L_H^F}\\min\\{\\|\\nabla F(x^{t+1})\\|/\\epsilon_H^2,1\\}\\\\[6pt]\r\n&=&\\frac{4}{4+\\zeta+\\sqrt{(4+\\zeta)^2+8L_H^F}}\\min\\{\\|\\nabla F(x^{t+1})\\|/\\epsilon_H,\\epsilon_H\\},\r\n\\end{array}\r\n\\]\r\nwhere the second inequality follows from the inequality $-a+\\sqrt{a^2+bs}\\ge(-a+\\sqrt{a^2+b})\\min\\{s,1\\}$ for all $a,b,s\\ge0$,\r\n%which can be seen to hold by performing rationalization to the terms $-a+\\sqrt{a^2+b}$ and $-a+\\sqrt{a^2+bs}$.\r\nwhich can be verified by performing a rationalization to the terms $-a+\\sqrt{a^2+b}$ and $-a+\\sqrt{a^2+bs}$, respectively.\r\nBy this, $\\alpha_t=1$, \\eqref{ls-sol} and \\eqref{csol}, one can see that \\eqref{suff-desc-sol} holds.\r\n\r\nCase 2) $\\alpha_t<1$.\r\nIt then follows that $j=0$ violates \\eqref{ls-sol} and hence \\eqref{lwbd-sol-ak} holds for $j=0$.\r\nNow, letting $j=0$ in \\eqref{lwbd-sol-ak}, we obtain that $\\|d^t\\|\\ge \\min\\{6(1-\\eta),2\\}\\epsilon_H/L_H^F$,\r\n which together with \\eqref{ls-sol} and \\eqref{lwbd-sol-ak-1} implies that\r\n\\[\r\nF(x^t)-F(x^{t+1}) \\ge \\eta \\epsilon_H\\theta^{2j_t}\\|d^t\\|^2 \\ge \\eta\\frac{\\min\\{6(1-\\eta),2\\}\\epsilon_H^2}{L_H^F}\\theta^2\\|d^t\\|\\ge \\eta\\left[\\frac{\\min\\{6(1-\\eta),2\\}\\theta}{L_H^F}\\right]^2\\epsilon_H^3.\r\n\\]\r\nBy this and \\eqref{csol}, one can see that \\eqref{suff-desc-sol} also holds in this case.\r\n\\end{proof}","sha256_utf8":"b5aa356e6de839c0b8e9f7d4010b548d411ec9ee649ae87ad9b35e148bdabd75"},{"start":100598,"end":101934,"start_line":950,"end_line":966,"raw_tex":"\\begin{lemma}\\label{lem:nc}\r\nSuppose that Assumption \\ref{asp:NCG-cmplxity} holds and the direction $d^t$ results from  either the output $d$ of Algorithm~\\ref{alg:capped-CG} with d$\\_$type=NC or the output $v$ of Algorithm~\\ref{pro:meo} at some iteration $t$ of Algorithm~\\ref{alg:NCG}.\r\n%\\begin{equation}\\label{cnc}\r\n%c_{\\rm nc} = \\frac{\\eta}{2} \\min\\left\\{1,\\left[\\frac{3(1-\\eta)\\theta}{L_H^F}\\right]^2\\right\\}.\r\n%\\end{equation}\r\nLet  $c_{\\rm nc}$ be defined in \\eqref{cnc}.\r\n%\\begin{equation}\\label{cnc}\r\n%c_{\\rm nc} := \\frac{\\eta}{2} \\min\\left\\{1,\\left[\\frac{\\min\\{3(1-\\eta),1\\}\\theta}{L_H^F}\\right]^2\\right\\}.\r\n%\\end{equation}\r\nThen the following statements hold.\r\n\\begin{enumerate}[{\\rm (i)}]\r\n\\item The step length $\\alpha_t$ is well-defined, and $\\alpha_t\\ge \\min\\{1,\\theta/L_H^F,3(1-\\eta)\\theta/L_H^F\\}$.\r\n%\\item If $d^t$ results from the output $d$ of Algorithm~\\ref{alg:capped-CG} with d$\\_$type=NC, then the next iterate $x^{t+1}=x^t+\\alpha_td^t$ satisfies $F(x^t)-F(x^{t+1})\\ge c_{\\rm nc}\\epsilon_H^3$.\r\n%\\item If $d^t$ { results from} the output $v$ of Algorithm~\\ref{pro:meo}, then the next iterate $x^{t+1}=x^t+\\alpha_td^t$ satisfies $F(x^t)-F(x^{t+1})\\ge c_{\\rm nc}\\epsilon_H^3/8$.\r\n\\item The next iterate $x^{t+1}=x^t+\\alpha_td^t$ satisfies $F(x^t)-F(x^{t+1})\\ge c_{\\rm nc}\\epsilon_H^3$.\r\n\\end{enumerate}\r\n\\end{lemma}","sha256_utf8":"05b954ac8c0eb148c391d61a3fc6fdf39619a3ecf6d8c40fc736e2c70a33aa69"},{"start":101938,"end":107144,"start_line":968,"end_line":1014,"raw_tex":"\\begin{proof}\r\nObserve that $F$ is descent along the iterates (whenever well-defined) generated by Algorithm~\\ref{alg:NCG}. Using this and $x^0=u^0$, we have $F(x^t)\\le F(u^0)$.\r\nBy the assumption on $d^t$, one can see from Algorithm \\ref{alg:NCG} that\r\n%$d^t$ is given in \\eqref{dk-nc} or \\eqref{dk-2nd-nc}. Now, notice that the (negative curvature) direction $v$ returned from Algorithm~\\ref{pro:meo} satisfies $\\|v\\|=1$. This observation (for $d^t$ in \\eqref{dk-2nd-nc}) together with  Lemma~\\ref{lem:SOL-NC-ppty}(ii) (for $d^t$ in \\eqref{dk-nc}) shows that $d^t$ must satisfy\r\n$d^t$ is a negative curvature direction given in \\eqref{dk-nc} or \\eqref{dk-2nd-nc}. Also, notice that the vector $v$ returned from Algorithm~\\ref{pro:meo} satisfies $\\|v\\|=1$. By these, Lemma~\\ref{lem:SOL-NC-ppty}(ii), \\eqref{dk-nc} and \\eqref{dk-2nd-nc}, one can observe that\r\n\\begin{equation}\\label{dk-Hk-dk3}\r\n\\nabla F(x^t)^Td^t\\le 0, \\quad (d^t)^T\\nabla^2 F(x^t)d^t=-\\|d^t\\|^3<0.\r\n\\end{equation}\r\n\r\nWe first prove statement (i). If \\eqref{ls-nc} holds for $j=0$, then $\\alpha_t=1$, which clearly implies that $\\alpha_t\\ge\\min\\{1,\\theta/L_H^F,3(1-\\eta)\\theta/L_H^F\\}$. We now suppose that \\eqref{ls-nc} fails for $j=0$. Claim that for all $j\\ge0$ that violate \\eqref{ls-nc}, it holds that\r\n\\begin{equation}\\label{lwbd-at-nc}\r\n\\theta^j\\ge \\min\\{1/L_H^F,3(1-\\eta)/L_H^F\\}.\r\n\\end{equation}\r\nIndeed, suppose that \\eqref{ls-nc} is violated by some $j \\ge 0$. We now show that \\eqref{lwbd-at-nc} holds for such $j$ by considering two separate cases below.\r\n\r\nCase 1) $F(x^t+\\theta^jd^t)>F(x^t)$. Let $\\phi(\\alpha)=F(x^t+\\alpha d^t)$. Then $\\phi(\\theta^j)>\\phi(0)$. Also, by  \\eqref{dk-Hk-dk3}, one has \r\n\\[\r\n\\phi^\\prime(0)=\\nabla F(x^t)^Td^t\\le 0, \\quad \\phi^{\\prime\\prime}(0)=(d^t)^T\\nabla^2 F(x^t)d^t<0.\r\n\\]\r\n Using these, we can observe that there exists a local minimizer $\\alpha^*\\in(0,\\theta^j)$ of $\\phi$ such that $\\phi(\\alpha^*)<\\phi(0)$, namely,  $F(x^t+\\alpha^*d^t)< F(x^t)$. By the second-order optimality condition of $\\phi$ at $\\alpha^*$, one has $\\phi^{\\prime\\prime}(\\alpha^*)=(d^t)^T\\nabla^2 F(x^t+\\alpha^*d^t)d^t\\ge 0$. Since $F(x^t+\\alpha^*d^t)<F(x^t)\\le F(u^0)$, it follows that \\eqref{F-Hess-Lip} holds for $x=x^t$ and $y=x^t+\\alpha^*d^t$. Using this, the second relation in \\eqref{dk-Hk-dk3} and $(d^t)^T\\nabla^2 F(x^t+\\alpha^*d^t)d^t\\ge0$, we obtain that\r\nin \\eqref{dk-Hk-dk3} and $(d^t)^T\\nabla^2 F(x^t+\\alpha^*d^t)d^t\\ge0$, we obtain that\r\n\\beqa\r\nL_H^F\\alpha^*\\|d^t\\|^3 &\\overset{\\eqref{F-Hess-Lip}}{\\ge}&\\|d^t\\|^2\\|\\nabla^2 F(x^t+\\alpha^*d^t)-\\nabla^2 F(x^t)\\| \\ge (d^t)^T(\\nabla^2 F(x^t+\\alpha^*d^t)-\\nabla^2 F(x^t))d^t \\nn \\\\\r\n&\\ge& -(d^t)^T\\nabla^2 F(x^t)d^t=\\|d^t\\|^3. \\label{lwbd-dt-nc-j1}\r\n\\eeqa\r\nRecall from \\eqref{dk-Hk-dk3} that $d^t\\neq 0$. It then follows from \\eqref{lwbd-dt-nc-j1} that $\\alpha^*\\ge 1/L_H^F$, which along with $\\theta^j>\\alpha^*$ implies that $ \\theta^j> 1/L_H^F$. Hence, \\eqref{lwbd-at-nc} holds in this case.\r\n\r\nCase 2) $F(x^t+\\theta^jd^t)\\le F(x^t)$. It follows from this and $F(x^t)\\le F(u^0)$ that \\eqref{desc-ineq} holds for $x=x^t$ and $y=x^t+\\theta^jd^t$. By this and the fact that $j$ violates \\eqref{ls-nc}, one has\r\n\\begin{equation*}\\label{lwbd-dt-nc}\r\n\\begin{array}{lcl}\r\n-\\frac{\\eta}{2}\\theta^{2j}\\|d^t\\|^3 &\\le& F(x^t+\\theta^j d^t) - F(x^t) \\overset{\\eqref{desc-ineq}}{\\le} \\theta^j \\nabla F(x^t)^Td^t + \\frac{\\theta^{2j}}{2}(d^t)^T\\nabla^2 F(x^t) d^t + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3\\\\[6pt]\r\n&\\overset{\\eqref{dk-Hk-dk3}}{\\le}& -\\frac{\\theta^{2j}}{2}\\|d^t\\|^3 + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3,\r\n\\end{array}\r\n\\end{equation*}\r\nwhich together with $d^t\\neq 0$ implies that $\\theta^j\\ge 3(1-\\eta)/L_H^F$. Hence, \\eqref{lwbd-at-nc} also holds  in this case.\r\n\r\nCombining the above two cases, we conclude that \\eqref{lwbd-at-nc} holds for any $j\\ge0$ that violates \\eqref{ls-nc}. By this and $\\theta\\in(0,1)$, one can see that all $j\\ge0$ that violate \\eqref{ls-nc} must be bounded above. It then follows that the step length $\\alpha_t$ associated with \\eqref{ls-nc} is well-defined. We next derive a lower bound for $\\alpha_t$. Notice from the definition of $j_t$ in Algorithm~\\ref{alg:NCG} that $j=j_t-1$ violates \\eqref{ls-nc} and hence \\eqref{lwbd-at-nc} holds for $j=j_t-1$. Then, by \\eqref{lwbd-at-nc} with $j=j_t-1$ and $\\alpha_t = \\theta^{j_t}$, one has $\\alpha_t=\\theta^{j_t} \\ge \\min\\{\\theta/L_H^F,3(1-\\eta)\\theta/L_H^F\\}$, which immediately yields $\\alpha_t\\ge \\min\\{1,\\theta/L_H^F,3(1-\\eta)\\theta/L_H^F\\}$ as desired.\r\n\r\nWe next prove statement (ii) by considering two separate cases below.\r\n\r\nCase 1) $d^t$ results from the output $d$ of Algorithm~\\ref{alg:capped-CG} with d$\\_$type=NC. It then follows from  \\eqref{NC-ppty} that $\\|d^t\\|\\ge\\epsilon_H$. \r\n%{ This together with item (i) and \\eqref{ls-nc} implies (ii).}\r\nThis together with  \\eqref{ls-nc} and statement (i) implies that statement (ii) holds.\r\n\r\nCase 2) $d^t$ results from the output $v$ of Algorithm~\\ref{pro:meo}. Notice from\r\nAlgorithm~\\ref{pro:meo} that $\\|v\\|=1$ and $v^T\\nabla^2F(x^t)v\\le-\\epsilon_H/2$, which along with \\eqref{dk-2nd-nc} yields $\\|d^t\\|\\ge\\epsilon_H/2$. By this, \\eqref{ls-nc} and statement (i), one can see that statement (ii) again holds.\r\n\\end{proof}","sha256_utf8":"e7cee1969989e8053db0b77d15fa1dc3bd293f7b10c0f6e39315550d78afb293"},{"start":107220,"end":111374,"start_line":1018,"end_line":1038,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{thm:NCG-iter-oper-cmplxity}]\r\nFor notational convenience, we let $\\{x^t\\}_{t\\in\\bT}$ denote all the iterates generated by Algorithm~\\ref{alg:NCG}, where $\\bT$ is a set of consecutive nonnegative integers starting from $0$. Notice that $F$ is descent along the iterates generated by Algorithm~\\ref{alg:NCG}, which together with $x^0=u^0$ implies that $x^t\\in\\{x:F(x)\\le F(u^0)\\}$. It then follows from\r\n\\eqref{lwbd-Hgupbd} that $\\|\\nabla^2 F(x^t)\\|\\le U^F_H$ holds for all $t\\in\\bT$.\r\n\r\n(i) Suppose for contradiction that the total number of calls of Algorithm~\\ref{pro:meo} in Algorithm~\\ref{alg:NCG} is more than $T_2$. Notice from Algorithm~\\ref{alg:NCG} and Lemma~\\ref{lem:nc}(ii) that each of these calls, except the last one, returns a sufficiently negative curvature direction, and each of them results in a reduction on $F$ of at least $c_{\\rm nc} \\epsilon_H^3$. Hence, \r\n\\[\r\nT_2 c_{\\rm nc} \\epsilon_H^3 \\le \\sum_{t\\in\\bT} [F(x^t)-F(x^{t+1})] \\le F(x^0)-\\Fl = \\Fh-\\Fl,\r\n\\]\r\nwhich contradicts the definition of $T_2$ given in \\eqref{T1}. Hence, statement (i) of Theorem \\ref{thm:NCG-iter-oper-cmplxity} holds.\r\n\r\n(ii) Suppose for contradiction that the total number of calls of Algorithm~\\ref{alg:capped-CG} in Algorithm~\\ref{alg:NCG} is more than $T_1$. Observe that if Algorithm~\\ref{alg:capped-CG} is called at some iteration $t$ and generates the next iterate $x^{t+1}$ satisfying $\\|\\nabla F(x^{t+1})\\|\\le\\epsilon_g$, then Algorithm~\\ref{pro:meo} must be called at the next iteration $t+1$. In view of this and statement (i) of Theorem \\ref{thm:NCG-iter-oper-cmplxity}, we see that the total number of such iterations $t$ is at most $T_2$. Hence, the total number of iterations $t$ of Algorithm~\\ref{alg:NCG} at which Algorithm~\\ref{alg:capped-CG} is called and generates the next iterate $x^{t+1}$ satisfying $\\|\\nabla F(x^{t+1})\\|>\\epsilon_g$ is at least $T_1-T_2+1$. Moreover, for each of such iterations $t$, we observe from Lemmas~\\ref{lem:sol}(ii) and \\ref{lem:nc}(ii) that $F(x^t)-F(x^{t+1})\\ge \\min\\{c_{\\rm sol},c_{\\rm nc}\\}\\min\\{\\epsilon_g^2\\epsilon_H^{-1},\\epsilon_H^3\\}$. It then follows that \r\n\\[\r\n(T_1-T_2+1) \\min\\{c_{\\rm sol},c_{\\rm nc}\\}\\min\\{\\epsilon_g^2\\epsilon_H^{-1},\\epsilon_H^3\\} \\le \\sum_{t\\in\\bT}[F(x^t)-F(x^{t+1})]\\le \\Fh-\\Fl,\r\n\\]\r\nwhich contradicts the definition of $T_1$ and $T_2$ given in \\eqref{T1}. Hence, statement (ii) of Theorem \\ref{thm:NCG-iter-oper-cmplxity} holds.\r\n\r\n(iii) Notice that either Algorithm~\\ref{alg:capped-CG} or \\ref{pro:meo} is called at each iteration of Algorithm~\\ref{alg:NCG}. It follows from this and statements (i) and (ii) of Theorem \\ref{thm:NCG-iter-oper-cmplxity} that the total number of iterations of Algorithm~\\ref{alg:NCG} is at most $T_1+T_2$. In addition, the relation \\eqref{NCG-iter} follows from \\eqref{csol}, \\eqref{cnc} and \\eqref{T1}. One can also observe that the output $x^t$ of Algorithm~\\ref{alg:NCG} satisfies $\\|\\nabla F(x^t)\\|\\le \\epsilon_g$ deterministically and $\\lambda_{\\min}(\\nabla^2F(x^t))\\ge-\\epsilon_H$ with probability at least $1-\\delta$ for some $0 \\le t \\le T_1+T_2$, where the latter part is due to\r\nAlgorithm~\\ref{pro:meo}. This completes the proof of statement (ii) of Theorem \\ref{thm:NCG-iter-oper-cmplxity}.\r\n\r\n(iv) By Theorem~\\ref{lem:capped-CG} with $(H,\\varepsilon)=(\\nabla^2 F(x^t),\\epsilon_H)$ and the fact that $\\|\\nabla^2 F(x^t)\\|\\le U^F_H$, one can observe that  the number of Hessian-vector products required by each call of Algorithm~\\ref{alg:capped-CG} with input $U = 0$ is at most $\\widetilde{\\cO}(\\min\\{n,(U_H^F/\\epsilon_H)^{1/2}\\})$. In addition, by Theorem~\\ref{rand-Lanczos} with $(H,\\varepsilon)=(\\nabla^2 F(x^t),\\epsilon_H)$, $\\|\\nabla^2 F(x^t)\\|\\le U^F_H$, and the fact that each iteration of the Lanczos method requires only one matrix-vector product, one can observe that the number of Hessian-vector products required by each call of Algorithm~\\ref{pro:meo} is also at most $\\widetilde{\\cO}(\\min\\{n,(U_H^F/\\epsilon_H)^{1/2}\\})$. Based on these observations and statement (iii) of Theorem \\ref{thm:NCG-iter-oper-cmplxity}, we see that  statement (iv) of this theorem holds.\r\n\\end{proof}","sha256_utf8":"f0b0db3ffa1c46596185333b0a964b61b0e8d729b68d7f1b1038ff735fbd5a0f"},{"start":111695,"end":112100,"start_line":1044,"end_line":1049,"raw_tex":"Recall from Assumption~\\ref{asp:lowbd-knownfeas}(a) that $\\|c(z_{\\epsilon_1})\\|\\le\\epsilon_1/2<1$. By virtue of this, \\eqref{lbd} and the definition of $\\tilde{c}$ in\r\n\\eqref{model:equa-cnstr-pert},\r\nwe obtain that\r\n\\begin{equation}\\label{ineq:p-lowbd}\r\nf(x)+\\gamma\\|\\tilde{c}(x)\\|^2 \\ge f(x)+ \\gamma\\|c(x)\\|^2/2 - \\gamma \\|c(z_{\\epsilon_1})\\|^2 \\ge \\fl - \\gamma,\\quad \\forall x\\in\\bR^n.\r\n\\end{equation}\r\n","sha256_utf8":"34eee3a647b57760922cb9dc9513639ce5da8537bec7bc590664de4a470118d7"},{"start":112184,"end":113353,"start_line":1053,"end_line":1075,"raw_tex":"\\begin{lemma}\\label{tech-1}\r\nSuppose that Assumption \\ref{asp:lowbd-knownfeas} holds. Let $\\gamma$, $\\fh$ and $\\fl$ be given in Assumption \\ref{asp:lowbd-knownfeas}.  Assume that $\\rho>2\\gamma$, $\\lambda\\in\\bR^m$, and $x\\in\\bR^n$ satisfy\r\n\\begin{equation}\\label{AL-upperbound}\r\n\\widetilde{\\cL}(x,\\lambda;\\rho)\\le \\fh,\r\n\\end{equation}\r\n%where $\\fh=f(x^{\\feas})$.\r\n%where $\\gamma$ and $\\fh$ are given in Assumption \\ref{asp:lowbd-knownfeas}.\r\nwhere $\\widetilde{\\cL}$ is defined in \\eqref{tL-tc}.\r\nThen the following statements hold.\r\n\\begin{enumerate}[{\\rm (i)}]\r\n\\item $f(x)\\le \\fh+\\|\\lambda\\|^2/(2\\rho)$.\r\n\\item $\\|\\tilde{c}(x)\\|\\le\\sqrt{2(\\fh-\\fl+\\gamma)/(\\rho-2\\gamma)+\\|\\lambda\\|^2/(\\rho-2\\gamma)^2}+\\|\\lambda\\|/(\\rho-2\\gamma)$.\r\n\\item If $\\rho\\ge\\|\\lambda\\|^2/(2\\tdf)$ for some $\\tdf>0$, then $f(x)\\le \\fh+\\tdf$.\r\n\\item If\r\n%\\begin{equation}\\label{bound-rho-deltac}\r\n%\\rho\\ge\\frac{2(\\fh-\\fl)}{\\delta_c^2}+\\frac{2\\|\\lambda\\|}{\\delta_c}+\\gamma\r\n%\\end{equation}\r\n\\begin{equation}\\label{bound-rho-deltac}\r\n\\rho\\ge 2(\\fh-\\fl+\\gamma)\\tdc^{-2}+ 2\\|\\lambda\\|\\tdc^{-1}+2\\gamma\r\n\\end{equation}\r\nfor some $\\tdc>0$, then $\\|\\tilde{c}(x)\\|\\le\\tdc$.\r\n\\end{enumerate}\r\n\\end{lemma}","sha256_utf8":"3b3a1dc4358f5989bbcae5820bb9800459a59ab2267aeac5a0640b71f8d272dd"},{"start":113358,"end":115274,"start_line":1077,"end_line":1109,"raw_tex":"\\begin{proof}\r\n(i) It follows from\r\n%{ the definition of { $\\widetilde{\\cL}$} in} \\eqref{tL-tc} and \\eqref{AL-upperbound}\r\n \\eqref{AL-upperbound} and the definition of $\\widetilde{\\cL}$ in \\eqref{tL-tc}\r\nthat\r\n\\[\r\n\\textstyle\r\n\\fh\\ge f(x)+\\lambda^T \\tilde{c}(x)+\\frac{\\rho}{2}\\|\\tilde{c}(x)\\|^2 = f(x)+\\frac{\\rho}{2}\\left\\|\\tilde{c}(x)+\\frac{\\lambda}{\\rho}\\right\\|^2-\\frac{\\|\\lambda\\|^2}{2\\rho} \\ge f(x)-\\frac{\\|\\lambda\\|^2}{2\\rho}.\r\n\\]\r\nHence, statement (i) holds.\r\n\r\n(ii) In view of \\eqref{ineq:p-lowbd} and \\eqref{AL-upperbound}, one has\r\n\\begin{equation*}\\label{ineq:bound-cx-step1}\r\n\\begin{array}{l}\r\n\\fh\\overset{\\eqref{AL-upperbound}}{\\ge}f(x)+\\lambda^T\\tilde{c}(x)+\\frac{\\rho}{2}\\|\\tilde{c}(x)\\|^2=f(x)+\\gamma\\|\\tilde{c}(x)\\|^2\r\n+\\frac{\\rho-2\\gamma}{2}\\left\\|\\tilde{c}(x)+\\frac{\\lambda}{\\rho-2\\gamma}\\right\\|^2-\\frac{\\|\\lambda\\|^2}{2(\\rho-2\\gamma)}\\\\[2pt]\r\n\\ \\ \\ \\ \\overset{\\eqref{ineq:p-lowbd}}{\\ge}\\fl- \\gamma+\\frac{\\rho-2\\gamma}{2}\\left\\|\\tilde{c}(x)+\\frac{\\lambda}{\\rho-2\\gamma}\\right\\|^2-\\frac{\\|\\lambda\\|^2}{2(\\rho-2\\gamma)}.\r\n\\end{array}\r\n\\end{equation*}\r\nIt then follows that $\\left\\|\\tilde{c}(x)+\\frac{\\lambda}{\\rho-2\\gamma}\\right\\|\\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho-2\\gamma}+\\frac{\\|\\lambda\\|^2}{(\\rho-2\\gamma)^2}}$,  which implies that statement (ii) holds.\r\n\r\n(iii) Statement (iii) immediately follows from statement (i) and $\\rho\\ge\\|\\lambda\\|^2/(2\\tdf)$.\r\n\r\n(iv) Suppose that \\eqref{bound-rho-deltac} holds. Multiplying both sides of \\eqref{bound-rho-deltac} by $\\tdc^{2}$ and rearranging the terms, we have\r\n\\[\r\n(\\rho-2\\gamma)\\tdc^2-2\\|\\lambda\\|\\tdc-2(\\fh-\\fl+\\gamma)\\ge0.\r\n\\]\r\nRecall that $\\rho>2\\gamma$ and $\\tdc>0$. Solving this inequality for $\\tdc$ yields \r\n\\[\r\n\\tdc\\ge\\sqrt{2(\\fh-\\fl+\\gamma)/(\\rho-2\\gamma)+\\|\\lambda\\|^2/(\\rho-2\\gamma)^2}+\\|\\lambda\\|/(\\rho-2\\gamma),\r\n\\]\r\nwhich along with statement (ii) implies that $\\|\\tilde{c}(x)\\|\\le\\tdc$. Hence, statement (iv) holds.\r\n\\end{proof}","sha256_utf8":"0e2b45498deacd339f364c14bc529ad2421cafeb8eadfd6248b7f691d7c4f7c3"},{"start":115361,"end":118533,"start_line":1113,"end_line":1139,"raw_tex":"\\begin{proof}[Proof of Lemma \\ref{lem:level-set-augmented-lagrangian-func}]\r\n(i) Let $x$ be any point such that $\\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)$. It then follows from \\eqref{L-xinit} that $\\widetilde{\\cL}(x,\\lambda^k;\\rho_k) \\le \\fh$. By this, $\\|\\lambda^k\\|\\le\\Lambda$, $\\rho_k \\ge \\rho_0>2\\gamma$, $\\delta_{f,1} \\le \\bdelta_f$, $\\delta_{c,1} \\le \\bdelta_c$, and Lemma \\ref{tech-1} with $(\\lambda,\\rho)=(\\lambda^k,\\rho_k)$,  one has \r\n\\begin{align}\r\nf(x) & \\le \\fh+ \\|\\lambda^k\\|^2/(2\\rho_k)\\le \\fh+ \\Lambda^2/(2\\rho_0)=\\fh+\\delta_{f,1} \\le \\fh+\\bdelta_f, \\nonumber \\\\\r\n\\textstyle\\|\\tilde{c}(x)\\|&\\textstyle\\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_k-2\\gamma}+\\frac{\\|\\lambda^k\\|^2}{(\\rho_k-2\\gamma)^2}}+\\frac{\\|\\lambda^k\\|}{\\rho_k-2\\gamma}\\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_0-2\\gamma}+\\frac{\\Lambda^2}{(\\rho_0-2\\gamma)^2}}+\\frac{\\Lambda}{\\rho_0-2\\gamma}=\\delta_{c,1} \\le \\bdelta_c. \\label{ineq:feasible-bd}\r\n\\end{align}\r\n%\\begin{eqnarray}\r\n%&f(x)\\le \\fh+\\frac{\\|\\lambda^k\\|^2}{2\\rho_k}\\le \\fh+\\frac{\\Lambda^2}{2\\rho_0}=\\fh+\\delta_{f,1} \\le \\fh+\\bdelta_f,\\nn \\\\\r\n%&\\|\\tilde{c}(x)\\|\\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_k-2\\gamma}+\\frac{\\|\\lambda^k\\|^2}{(\\rho_k-2\\gamma)^2}}+\\frac{\\|\\lambda^k\\|}{\\rho_k-2\\gamma}\\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_0-2\\gamma}+\\frac{\\Lambda^2}{(\\rho_0-2\\gamma)^2}}+\\frac{\\Lambda}{\\rho_0-2\\gamma}=\\delta_{c,1} \\le \\bdelta_c.\\label{ineq:feasible-bd}\r\n%\\end{eqnarray}\r\nAlso, recall from the definition of $\\tilde{c}$ in \\eqref{model:equa-cnstr-pert}\r\nand $\\|c(z_{\\epsilon_1})\\|\\le1$ that $\\|c(x)\\|\\le1+\\|\\tilde{c}(x)\\|$. This together with the above inequalities and \\eqref{nearly-feas-level-set} implies $x\\in\\cS(\\delta_{f},\\delta_{c})$. Hence, statement (i) of Lemma \\ref{lem:level-set-augmented-lagrangian-func} holds.\r\n\r\n(ii) Note that $\\inf\\limits_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)=\\inf\\limits_{x\\in\\bR^n}\\{\\widetilde{\\cL}(x,\\lambda^k;\\rho_k):\\widetilde{\\cL}(x,\\lambda^k;\\rho_k) \\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\}$. Consequently, to prove statement (ii) of Lemma \\ref{lem:level-set-augmented-lagrangian-func}, it suffices to show that\r\n\\begin{equation} \\label{L-lwbd}\r\n\\inf\\limits_{x\\in\\bR^n}\\{\\widetilde{\\cL}(x,\\lambda^k;\\rho_k): \\widetilde{\\cL}(x,\\lambda^k;\\rho_k) \\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\} \\ge\\fl - \\gamma-\\Lambda\\bdelta_c.\r\n\\end{equation}\r\nTo this end, let $x$ be any point satisfying $\\widetilde{\\cL}(x,\\lambda^k;\\rho_k) \\le\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)$. We then know from \\eqref{ineq:feasible-bd} that $\\|\\tilde{c}(x)\\|\\le\\bdelta_c$. By this, $\\|\\lambda^k\\|\\le\\Lambda$, $\\rho_k>2\\gamma$, and \\eqref{ineq:p-lowbd}, one has\r\n\\begin{equation*}\\label{ineq:lowerbound-ALfunc} \r\n\\begin{array}{l}\r\n\\widetilde{\\cL}(x,\\lambda^k;\\rho_k) %= f(x)+(\\lambda^k)^T\\tilde{c}(x)+\\frac{\\rho_k}{2}\\|\\tilde{c}(x)\\|^2\\\\[4pt]\r\n=f(x)+\\gamma\\|\\tilde{c}(x)\\|^2+(\\lambda^k)^T\\tilde{c}(x)+\\frac{\\rho_k-2\\gamma}{2}\\|\\tilde{c}(x)\\|^2\\\\[4pt]\r\n\\ge f(x)+\\gamma\\|\\tilde{c}(x)\\|^2 - \\Lambda\\|\\tilde{c}(x)\\| \\ge\\fl - \\gamma-\\Lambda\\bdelta_c,\r\n\\end{array}\r\n\\end{equation*}\r\nand hence \\eqref{L-lwbd} holds as desired.\r\n\\end{proof}","sha256_utf8":"c7ae0aac3e79e04df28f352bf6d9554a9ea40e058b0c3e5a3260a53c69263d9a"},{"start":118585,"end":120991,"start_line":1143,"end_line":1155,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{thm:output-alg1}]\r\nSuppose that Algorithm \\ref{alg:2nd-order-AL-nonconvex} terminates at some iteration $k$, that is, $\\tau_k^g\\le\\epsilon_1$, $\\tau_k^H\\le\\epsilon_2$, and $\\|c(x^{k+1})\\|\\le\\epsilon_1$ hold. Then, by $\\tau_k^g\\le\\epsilon_1$, $\\tilde{\\lambda}^{k+1}=\\lambda^k+\\rho_k\\tilde{c}(x^{k+1})$, $\\nabla \\tilde{c}=\\nabla c$ and the second relation in \\eqref{algstop:1st-order}, one has \r\n \\begin{align*}\r\n\\|\\nabla f(x^{k+1})+\\nabla c(x^{k+1})\\tilde{\\lambda}^{k+1}\\| &=\\|\\nabla f(x^{k+1})+\\nabla \\tilde{c}(x^{k+1})(\\lambda^k+\\rho_k \\tilde{c}(x^{k+1}))\\| \\\\ \r\n&=\\|\\nabla_x\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k)\\|\\le \\tau_k^g \\le \\epsilon_1.\r\n\\end{align*} \r\n Hence, $(x^{k+1},\\tilde{\\lambda}^{k+1})$ satisfies the first relation in \\eqref{optcond:1st-equa-cnstr}. In addition, by \\eqref{algstop:2nd-order} and $\\tau_k^H\\le\\epsilon_2$, one can show that $\\lambda_{\\min}(\\nabla^2_{xx}\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k))\\ge-\\epsilon_2$ with probability at least $1-\\delta$, which leads to $d^T\\nabla^2_{xx}\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k)d \\ge -\\epsilon_2\\|d\\|^2$ for all $d\\in\\bR^n$ with probability at least $1-\\delta$.  Using this, $\\tilde{\\lambda}^{k+1}=\\lambda^k+\\rho_k\\tilde{c}(x^{k+1})$, $\\nabla \\tilde{c} = \\nabla c$, and $\\nabla^2 \\tilde{c}_i=\\nabla^2 c_i$ for $1\\le i\\le m$, we see that with probability at least $1-\\delta$, it holds that \r\n%$d^T (\\nabla^2 f(x^{k+1})+\\sum_{i=1}^m\\tilde{\\lambda}_i^{k+1}\\nabla^2c_i(x^{k+1})+\\rho_k\\nabla c(x^{k+1})\\nabla c(x^{k+1})^T )d\\ge -\\epsilon_2\\|d\\|^2$ for all $d\\in\\bR^n$},\r\n\\[\r\nd^T\\left(\\nabla^2 f(x^{k+1})+\\sum_{i=1}^m\\tilde{\\lambda}_i^{k+1}\\nabla^2c_i(x^{k+1})+\\rho_k\\nabla c(x^{k+1})\\nabla c(x^{k+1})^T\\right)d\\ge -\\epsilon_2\\|d\\|^2 \\ \\  \\forall d\\in\\bR^n,\r\n\\]\r\nwhich implies $d^T(\\nabla^2 f(x^{k+1})+\\sum_{i=1}^m\\tilde{\\lambda}_i^{k+1}\\nabla^2 c_i(x^{k+1}))d\\ge -\\epsilon_2\\|d\\|^2$ for all $d\\in \\cC(x^{k+1})$, where $\\cC(\\cdot)$ is defined in \\eqref{def:critical-cone}. Hence, $(x^{k+1},\\tilde{\\lambda}^{k+1})$ satisfies \\eqref{optcond:2nd-equa-cnstr} with probability at least $1-\\delta$. Combining these with $\\|c(x^{k+1})\\|\\le\\epsilon_1$, we conclude that $x^{k+1}$ is a deterministic $\\epsilon_1$-FOSP of \\eqref{model:equa-cnstr} and an $(\\epsilon_1,\\epsilon_2)$-SOSP of \\eqref{model:equa-cnstr} with probability at least $1-\\delta$. Hence, Theorem \\ref{thm:output-alg1} holds.\r\n\\end{proof}","sha256_utf8":"30769953bf70b9aa890de207664a9ed5711907874d71a30504f347cc0f60e5f9"},{"start":121051,"end":129248,"start_line":1159,"end_line":1236,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{thm:out-itr-cmplxity-1}]\r\nIt follows from \\eqref{def:delta0c-rhobar1} that ${\\rho}_{\\epsilon_1}\\ge 2\\rho_0$.\r\nBy this, one has\r\n\\begin{equation}\\label{T-epsilong}\r\nK_{\\epsilon_1} \\ \\overset{\\eqref{T-epsilon-g}}{=} \\ \\left\\lceil\\log\\epsilon_1/\\log \\omega_1 \\right\\rceil \\ \\overset{\\eqref{omega-tolerance}}{=} \\ \\left\\lceil\\log{2}/\\log{r}\\right\\rceil  \\\r\n\\le \\ \\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})/\\log{r}+1.\r\n\\end{equation}\r\nNotice that $\\{\\rho_k\\}$ is either unchanged or increased by a ratio $r$ as $k$ increases.\r\nBy this fact and \\eqref{T-epsilong}, we see that\r\n\\begin{equation}\\label{rho:upper-bound-Tepsilong}\r\n\\max_{0 \\le k \\le K_{\\epsilon_1}} \\rho_k \\le{r}^{K_{\\epsilon_1}}\\rho_0\\overset{\\eqref{T-epsilong}}{\\le}{r}^{\\frac{\\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})}{\\log{r}}+1}\\rho_0={r}{\\rho}_{\\epsilon_1}.\r\n\\end{equation}\r\nIn addition, notice that $\\rho_k>2\\gamma$ and $\\|\\lambda^k\\|\\le \\Lambda$. Using these, \\eqref{hbd}, the first relation in \\eqref{algstop:1st-order}, and Lemma \\ref{tech-1}(ii) with $(x,\\lambda,\\rho)=(x^{k+1},\\lambda^k,\\rho_k)$,  we obtain that\r\n\\begin{equation}\\label{bd-ck}\\textstyle\r\n\\|\\tilde{c}(x^{k+1})\\| \\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_k-2\\gamma}+\\frac{\\|\\lambda^k\\|^2}{(\\rho_k-2\\gamma)^2}}+\\frac{\\|\\lambda^k\\|}{\\rho_k-2\\gamma}\\le  \\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_k-2\\gamma}+\\frac{\\Lambda^2}{(\\rho_k-2\\gamma)^2}}+\\frac{\\Lambda}{\\rho_k-2\\gamma}.\r\n\\end{equation}\r\nAlso, we observe from $\\|c(z_{\\epsilon_1})\\|\\le\\epsilon_1/2$ and the definition of $\\tilde{c}$ in\r\n\\eqref{model:equa-cnstr-pert} that\r\n\\begin{equation}\\label{bd-cxk}\r\n\\|c(x^{k+1})\\|\\le\\|\\tilde{c}(x^{k+1})\\|+\\|c(z_{\\epsilon_1})\\|\\le \\|\\tilde{c}(x^{k+1})\\|+\\epsilon_1/2.\r\n\\end{equation}\r\n\r\nWe now prove that $\\overline{K}_{\\epsilon_1}$ is finite. Suppose for contradiction that $\\overline{K}_{\\epsilon_1}$ is infinite. It then follows from this and \\eqref{number-outer-iteration} that $\\|c(x^{k+1})\\|>\\epsilon_1$ for all $k\\ge K_{\\epsilon_1}$, which along with \\eqref{bd-cxk} implies that $\\|\\tilde{c}(x^{k+1})\\|>\\epsilon_1/2$ for all $k\\ge K_{\\epsilon_1}$. It then follows that $\\|\\tilde{c}(x^{k+1})\\|>\\alpha\\|\\tilde{c}(x^k)\\|$ must hold for infinitely many $k$'s. Using this and the update scheme on $\\{\\rho_k\\}$, we deduce that $\\rho_{k+1}={r}\\rho_k$ holds for infinitely many $k$'s, which together with the monotonicity of $\\{\\rho_k\\}$ implies that\r\n$\\rho_k\\to\\infty$ as $k\\to\\infty$. By this and \\eqref{bd-ck}, one can see that  $\\|\\tilde{c}(x^{k+1})\\|\\to0$ as $k\\to\\infty$, which contradicts the fact that $\\|\\tilde{c}(x^{k+1})\\|>\\epsilon_1/2$ holds for all $k\\ge K_{\\epsilon_1}$. Hence, $\\overline{K}_{\\epsilon_1}$ is finite.\r\nIn addition, notice from \\eqref{omega-tolerance}, \\eqref{T-epsilon-g} and \\eqref{epsw1-epsw2} that $(\\tau_k^g,\\tau_k^H)=(\\epsilon_1,\\epsilon_2)$ for all $k\\ge K_{\\epsilon_1}$. This along with the termination criterion of Algorithm \\ref{alg:2nd-order-AL-nonconvex}  and the definition of $\\overline{K}_{\\epsilon_1}$ implies that Algorithm \\ref{alg:2nd-order-AL-nonconvex} must terminate at iteration $\\overline{K}_{\\epsilon_1}$.\r\n\r\nWe next show that \\eqref{outer-iteration-cmplxty} and $\\rho_k\\le{r}{\\rho}_{\\epsilon_1}$ hold for $0 \\le k \\le \\overline{K}_{\\epsilon_1}$ by considering two separate cases below.\r\n\r\nCase 1) $\\|c(x^{K_{\\epsilon_1}+1})\\|\\le\\epsilon_1$.  By this and \\eqref{number-outer-iteration}, one can see that  $\\overline{K}_{\\epsilon_1}=K_{\\epsilon_1}$, which together with \\eqref{T-epsilong} and \\eqref{rho:upper-bound-Tepsilong} implies that \\eqref{outer-iteration-cmplxty} and $\\rho_k\\le{r}{\\rho}_{\\epsilon_1}$ hold for $0 \\le k \\le \\overline{K}_{\\epsilon_1}$.\r\n\r\nCase 2) $\\|c(x^{K_{\\epsilon_1}+1})\\|>\\epsilon_1$. By this and \\eqref{number-outer-iteration}, one can observe that $\\overline{K}_{\\epsilon_1}>K_{\\epsilon_1}$  and also $\\|c(x^{k+1})\\|>\\epsilon_1$ for all $K_{\\epsilon_1} \\le k \\le \\overline{K}_{\\epsilon_1}-1$, which together with \\eqref{bd-cxk} implies\r\n\\begin{equation}\\label{lbd-tc}\r\n\\|\\tilde{c}(x^{k+1})\\|>\\epsilon_1/2,\\quad \\forall K_{\\epsilon_1} \\le k \\le \\overline{K}_{\\epsilon_1}-1.\r\n\\end{equation}\r\nIt then follows from $\\|\\lambda^k\\|\\le\\Lambda$, \\eqref{hbd}, the first relation in \\eqref{algstop:1st-order}, and Lemma \\ref{tech-1}(iv) with $(x,\\lambda,\\rho,\\tdc)=(x^{k+1},\\lambda^k,\\rho_k,\\epsilon_1/2)$ that\r\n\\begin{equation}\\label{bound-rhok}\r\n\\begin{aligned}\r\n\\rho_k &< 8(\\fh-\\fl+\\gamma)\\epsilon_1^{-2}+4\\|\\lambda^k\\|\\epsilon_1^{-1}+2\\gamma\\\\\r\n&\\le 8(\\fh-\\fl+\\gamma)\\epsilon_1^{-2}+4\\Lambda\\epsilon_1^{-1}+2\\gamma\\overset{\\eqref{def:delta0c-rhobar1}}{\\le}{\\rho}_{\\epsilon_1},\\quad\r\n\\forall K_{\\epsilon_1} \\le k \\le \\overline{K}_{\\epsilon_1}-1.\r\n\\end{aligned}\r\n\\end{equation}\r\nCombining this relation, \\eqref{rho:upper-bound-Tepsilong}, and the fact $\\rho_{\\overline{K}_{\\epsilon_1}}\\le {r}\\rho_{\\overline{K}_{\\epsilon_1}-1}$, we conclude that  $\\rho_k\\le{r}{\\rho}_{\\epsilon_1}$ holds for $0 \\le k\\le \\overline{K}_{\\epsilon_1}$. It remains to\r\nshow that \\eqref{outer-iteration-cmplxty} holds.  To this end, let \r\n\\[\r\n\\bK =\\{k:\\rho_{k+1}={r}\\rho_{k}, K_{\\epsilon_1}\\le k\\le \\overline{K}_{\\epsilon_1}-2\\}.\r\n\\]\r\nIt follows from \\eqref{bound-rhok} and the update scheme of $\\rho_k$ that \r\n\\[\r\n{r}^{|\\bK|}\\rho_{K_{\\epsilon_1}}=\\max_{K_{\\epsilon_1}\\le k \\le \\overline{K}_{\\epsilon_1}-1}\\{\\rho_k\\}\\le {\\rho}_{\\epsilon_1},\r\n\\]\r\nwhich together with $\\rho_{K_{\\epsilon_1}}\\ge\\rho_0$ implies that\r\n\\begin{equation}\\label{bound-cU}\r\n|\\bK|\\le \\log({\\rho}_{\\epsilon_1}\\rho_{K_{\\epsilon_1}}^{-1})/ \\log{r} \\le \\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})/ \\log{r}.\r\n\\end{equation}\r\nLet $\\{k_1,k_2,\\ldots,k_{|\\bK|}\\}$ denote all the elements of $\\mathcal{\\bK}$ arranged in ascending order, and let $k_0=K_{\\epsilon_1}$ and $k_{|\\bK|+1}=\\overline{K}_{\\epsilon_1}-1$. We next derive an upper bound for $k_{j+1}-k_{j}$ for $j=0,1,\\ldots,|\\bK|$. By the definition of $\\bK$, one can observe that $\\rho_{k}=\\rho_{k'}$ for $k_j< k,k'\\le k_{j+1}$. Using this and the update scheme of $\\rho_k$, we deduce that\r\n\\begin{equation}\\label{ineq:sequence-cxk}\r\n\\|\\tilde{c}(x^{k+1})\\|\\le\\alpha\\|\\tilde{c}(x^{k})\\|,\\quad \\forall k_j<k< k_{j+1}.\r\n\\end{equation}\r\nOn the other hand, by \\eqref{def:delta0c-rhobar1xxx}, \\eqref{bd-ck} and $\\rho_k\\ge\\rho_0$, one has $\\|\\tilde{c}(x^{k+1})\\|\\le \\delta_{c,1}$ for $0 \\le k \\le \\overline{K}_{\\epsilon_1}$. By this and \\eqref{lbd-tc}, one can see that\r\n\\begin{equation}\\label{ineq:sequence-cxk2222}\r\n\\epsilon_1/2<\\|\\tilde{c}(x^{k+1})\\|\\le\\delta_{c,1},\\quad \\forall K_{\\epsilon_1} \\le k \\le \\overline{K}_{\\epsilon_1}-1.\r\n\\end{equation}\r\nNow, note that either $k_{j+1} - k_j = 1$ or $k_{j+1} - k_j > 1$. In the latter case, we can apply \\eqref{ineq:sequence-cxk} with $k = k_{j+1}-1,\\ldots,k_j+1$ together with \\eqref{ineq:sequence-cxk2222} to deduce that\r\n\\[\r\n\\epsilon_1/2<\\|\\tilde{c}(x^{k_{j+1}})\\|\\le\\alpha \\|\\tilde{c}(x^{k_{j+1}-1})\\|\\le\\cdots\\le \\alpha^{k_{j+1}-k_j-1}\\|\\tilde{c}(x^{k_{j}+1})\\|\\le \\alpha^{k_{j+1}-k_j-1}\\delta_{c,1},\\quad \\forall j=0,1,\\ldots,|\\bK|.\r\n\\]\r\nCombining these two cases, we have \r\n\\begin{equation}\\label{ineq:subsequence-rho-same}\r\nk_{j+1}-k_{j}\\le |\\log(\\epsilon_1(2\\delta_{c,1})^{-1}))/\\log \\alpha|+1,\\quad \\forall j=0,1,\\ldots,|\\bK|.\r\n\\end{equation}\r\nSumming up these inequalities, and using \\eqref{T-epsilong}, \\eqref{bound-cU},  $k_0=K_{\\epsilon_1}$ and $k_{|\\bK|+1}=\\overline{K}_{\\epsilon_1}-1$, we have\r\n\\begin{eqnarray}\r\n&\\hspace{-.85in}\\overline{K}_{\\epsilon_1} = 1+k_{|\\bK|+1}=1+k_0+\\sum_{j=0}^{|\\bK|}(k_{j+1}-k_{j}) \\overset{\\eqref{ineq:subsequence-rho-same}}{\\le}1+K_{\\epsilon_1}+(|\\bK|+1)\\left(\\left|\\frac{\\log(\\epsilon_1(2\\delta_{c,1})^{-1})}{\\log \\alpha}\\right|+1\\right) \\nonumber \\\\[6pt]\r\n&\\le2+\\frac{\\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})}{\\log{r}}+\\left(\\frac{\\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})}{\\log{r}}+1\\right)\\left(\\left|\\frac{\\log(\\epsilon_1(2\\delta_{c,1})^{-1})}{\\log \\alpha}\\right|+1\\right) =1+ \\left(\\frac{\\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})}{\\log{r}}+1\\right)\\left(\\left|\\frac{\\log(\\epsilon_1(2\\delta_{c,1})^{-1})}{\\log \\alpha}\\right|+2\\right), \\nn\r\n\\end{eqnarray}\r\nwhere the second inequality is due to \\eqref{T-epsilong} and \\eqref{bound-cU}. Hence, \\eqref{outer-iteration-cmplxty} also holds in this case.\r\n\\end{proof}","sha256_utf8":"7fe19accb1087686205a121a8cd7f2527034b7128119089ce42992a8a1df6dba"},{"start":129252,"end":130611,"start_line":1238,"end_line":1246,"raw_tex":"We next prove Theorem \\ref{thm:total-iter-cmplxity}. Before proceeding, we introduce some notation that will be used shortly. Let $L_{k,H}$ denote the Lipschitz constant of $\\nabla^2_{xx} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$ on the convex open neighborhood $\\Omega(\\bdelta_{f},\\bdelta_{c})$ of\r\n$\\cS(\\bdelta_f,\\bdelta_c)$, where $\\cS(\\bdelta_f,\\bdelta_c)$ is defined in \\eqref{nearly-feas-level-set}, and let $U_{k,H}=\\sup_{x\\in\\cS(\\bdelta_f,\\bdelta_c)} \\|\\nabla^2_{xx} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\|$. Notice from \\eqref{model:equa-cnstr-pert} and \\eqref{tL-tc} that\r\n\\begin{equation}\\label{eq:hessian-augmented-Lagrangian}\r\n\\nabla^2_{xx}\\widetilde{\\cL}(x,\\lambda^k;\\rho_k) = \\nabla^2 f(x) +\\sum_{i=1}^m\\lambda^k_{i}\\nabla^2 c_i(x) + \\rho_k \\bigg(\\nabla c(x)\\nabla c(x)^T + \\sum_{i=1}^m \\tilde{c}_i(x)\\nabla^2 c_i(x)\\bigg).\r\n\\end{equation}\r\nBy this, $\\|\\lambda^k\\| \\le \\Lambda$, the definition of $\\tilde{c}$, and the Lipschitz continuity of $\\nabla^2 f$ and $\\nabla^2 c_i$'s (see Assumption~\\ref{asp:lowbd-knownfeas}(c)), one can observe that there exist some constants ${L}_{1}$, ${L}_{2}$, ${U}_{1}$ and ${U}_{2}$, depending only on $f$, $c$, $\\Lambda$, $\\bdelta_{f}$ and $\\bdelta_{c}$,  such that\r\n\\begin{equation}\\label{ineq:bound-norm-hessian-Liphessian}\r\nL_{k,H}\\le {L}_{1}+\\rho_k{L}_{2},\\quad U_{k,H}\\le{U}_{1}+\\rho_k{U}_{2}.\r\n\\end{equation}\r\n","sha256_utf8":"7c7a3c233d960f95c961c6a9f73d6d7fc48f521e31d1683ff59adf951956b64b"},{"start":130682,"end":139592,"start_line":1250,"end_line":1320,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{thm:total-iter-cmplxity}]\r\nLet $T_k$ and $N_k$ denote the number of iterations and matrix-vector products performed by Algorithm~\\ref{alg:NCG} at the outer iteration $k$ of Algorithm \\ref{alg:2nd-order-AL-nonconvex}, respectively. It then follows from Theorem \\ref{thm:out-itr-cmplxity-1} that the total number of iterations and matrix-vector products performed by Algorithm~\\ref{alg:NCG} in Algorithm \\ref{alg:2nd-order-AL-nonconvex} are $\\sum_{k=0}^{\\overline{K}_{\\epsilon_1}}T_k$ and $\\sum_{k=0}^{\\overline{K}_{\\epsilon_1}}N_k$, respectively. In addition, notice from \\eqref{def:delta0c-rhobar1} and Theorem \\ref{thm:out-itr-cmplxity-1} that ${\\rho}_{\\epsilon_1}=\\cO(\\epsilon_1^{-2})$ and  $\\rho_k\\le{r}{\\rho}_{\\epsilon_1}$, which yield $\\rho_k=\\cO(\\epsilon_1^{-2})$.\r\n\r\n%(i) {  We first prove that $(\\tau_k^g)^2/\\tau_k^H\\ge\\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$ holds for any $k\\ge0$. If $\\epsilon_1^2{ \\ge}\\epsilon_2$, it follows from $\\tau_k^g\\ge\\epsilon_1$ and $\\tau_k^H\\le 1$ that $(\\tau_k^g)^2/\\tau_k^H\\ge(\\tau_k^g)^2\\ge \\epsilon_1^2{ \\ge}\\epsilon_2$, which along with $\\epsilon_2\\in(0,1)$ implies $(\\tau_k^g)^2/\\tau_k^H{ \\ge}\\epsilon_2^3$. On the other hand, if $\\epsilon_1^2{ <}\\epsilon_2$, it follows from \\eqref{omega-tolerance} that $\\omega_1^2/\\omega_2{ <} 1$. Also, recall from \\eqref{omega-tolerance}, \\eqref{T-epsilon-g} and \\eqref{epsw1-epsw2} that\r\n%\\[\r\n%K_{\\epsilon_1}=\\left\\lceil \\log\\epsilon_1/\\log\\omega_1\\right\\rceil=\\left\\lceil \\log\\epsilon_2/\\log\\omega_2\\right\\rceil,\\quad\r\n%(\\tau_k^g,\\tau_k^H)=\\left\\{\\begin{array}{ll}\r\n%(\\omega_1^k,\\omega_2^k)&\\text{if }\\ 0\\le k< K_{\\epsilon_1},\\\\\r\n%(\\epsilon_1,\\epsilon_2)&\\text{otherwise}.\r\n%\\end{array}\\right.\r\n%\\]\r\n%In view of these, we see that the sequence $\\{(\\tau_k^g)^2/\\tau_k^H\\}$ is geometrically decreasing with ratio $\\omega_1^2/\\omega_2{ <} 1$ when $0\\le k< K_{\\epsilon_1}$ and equals $\\epsilon_1^2/\\epsilon_2$ for all $k\\ge K_{\\epsilon_1}$. In addition, notice from \\eqref{T-epsilon-g} and \\eqref{epsw1-epsw2} that\r\n%\\[\r\n%\\left\\lceil\\log(\\epsilon_1^2/\\epsilon_2)/\\log(\\omega_1^2/\\omega_2)\\right\\rceil =\\left\\lceil \\log 2/\\log r\\right\\rceil = K_{\\epsilon_1}.\r\n%\\]\r\n%Thus, we have $(\\tau_k^g)^{2}/\\tau_k^H=\\max\\{\\epsilon_1^2/\\epsilon_2,(\\omega_1^2/\\omega_2)^k\\}\\ge\\epsilon_1^2/\\epsilon_2$ when $\\epsilon_1^2{ <}\\epsilon_2$. Combining this with the fact that $(\\tau_k^g)^2/\\tau_k^H{ \\ge}\\epsilon_2^3$ when $\\epsilon_1^2{ \\ge}\\epsilon_2$, we obtain that $(\\tau_k^g)^2/\\tau_k^H\\ge\\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$ holds for any $k\\ge0$.}\r\n\r\nWe first claim that $(\\tau_k^g)^2/\\tau_k^H\\ge\\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$ holds for any $k\\ge0$. Indeed, let $\\bar t=\\log\\epsilon_1/\\log\\omega_1$ and $\\psi(t)=\\max\\{\\epsilon_1,\\omega_1^t\\}^2/\\max\\{\\epsilon_2,\\omega_2^t\\}$ for all $t\\in\\bR$. It then follows from \\eqref{epsw1-epsw2} that $\\omega_1^{\\bar t}=\\epsilon_1$ and $\\omega_2^{\\bar t}=\\epsilon_2$. By this and $\\omega_1,\\omega_2\\in(0,1)$, one can observe that  $\\psi(t)=(\\omega_1^2/\\omega_2)^t$ if $t \\leq \\bar t$ and $\\psi(t)=\\epsilon_1^2/\\epsilon_2$ otherwise. This along with $\\epsilon_2\\in(0,1)$ implies that\r\n\\[\r\n\\min_{t\\in[0,\\infty)}\\psi(t) = \\min\\{\\psi(0),\\psi(\\bar t)\\}=\\min\\{1,\\epsilon_1^2/\\epsilon_2\\} \\geq \\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\},\r\n\\]\r\nwhich together with \\eqref{omega-tolerance} yields\r\n$(\\tau_k^g)^2/\\tau_k^H=\\psi(k) \\geq \\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$ for all $k \\geq 0$.\r\n\r\n(i) From Lemma \\ref{lem:level-set-augmented-lagrangian-func}(i) and the definitions of $\\Omega(\\bdelta_{f},\\bdelta_{c})$ and $L_{k,H}$, we see that $L_{k,H}$ is a Lipschitz constant of $\\nabla^2_{xx} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$ on a convex open neighborhood  of $\\{x:\\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\}$.\r\n% In addition, by Lemma \\ref{lem:level-set-augmented-lagrangian-func}(ii) and the assumption $\\delta_{c,1} \\le \\bdelta_c$, one has $\\inf_x \\cL(x,\\lambda^k;\\rho_k) \\ge\\fl-\\Lambda\\bdelta_c$.\r\nAlso, recall from Lemma \\ref{lem:level-set-augmented-lagrangian-func}(ii) that $\\inf_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k) $ $\\ge\\fl-\\gamma-\\Lambda\\bdelta_c$. By these, $\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\le \\fh$ (see \\eqref{L-xinit}) and Theorem \\ref{thm:NCG-iter-oper-cmplxity}(iii) with $(\\Fh,\\Fl,L_{H}^F,\\epsilon_g,\\epsilon_H)=(\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k),\\fl-\\gamma-\\Lambda \\bdelta_c,L_{k,H},\\tau_k^g,\\tau_k^H)$, one has\r\n\\begin{equation}\\label{eq:alg1-inner-iter-kthAL}\r\n\\begin{array}{rcl}\r\nT_k&=&\\cO((\\fh-\\fl+\\gamma+\\Lambda\\bdelta_{c})L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\})\\\\[2pt]\r\n&\\overset{\\eqref{ineq:bound-norm-hessian-Liphessian}}{=}&\\cO(\\rho_k^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\})=\\cO(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}),\r\n\\end{array}\r\n\\end{equation}\r\n%\\begin{equation}\\label{eq:alg1-inner-iter-kthAL}\r\n%\\begin{array}{rcl}\r\n%T_k&=&\\cO((\\cL(x^{k}_{\\init},\\lambda^k;\\rho_k)-\\fl+\\gamma+\\Lambda\\bdelta_{c})L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\})\\\\[4pt]\r\n%&=&\\cO((\\fh-\\fl+\\gamma+\\Lambda\\bdelta_{c})L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\})\\\\[4pt]\r\n%&\\overset{\\eqref{ineq:bound-norm-hessian-Liphessian}}{=}&\\cO(\\rho_k^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\})=\\cO(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}),\r\n%\\end{array}\r\n%\\end{equation}\r\nwhere the last equality is from $(\\tau_k^g)^2/\\tau_k^H\\ge\\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$, $\\tau_k^H\\ge\\epsilon_2$, and $\\rho_k=\\cO(\\epsilon_1^{-2})$.\r\n\r\nNext, if %$c$ is assumed to be affine, namely,\r\n$c(x)=Ax-b$ for some $A\\in\\bR^{m\\times n}$ and $b\\in\\bR^m$, then $\\nabla c(x)=A^T$ and $\\nabla^2 c_i(x)=0$ for $1 \\le i \\le m$. By these and \\eqref{eq:hessian-augmented-Lagrangian}, one has $L_{k,H}=\\cO(1)$. Using this and similar arguments as for \\eqref{eq:alg1-inner-iter-kthAL}, we obtain that $T_k=\\cO(\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\})$. By this, \\eqref{eq:alg1-inner-iter-kthAL} and $\\overline{K}_{\\epsilon_1}=\\cO(|\\log\\epsilon_1|^2)$ (see Remark \\ref{order-out-itera-penalty}), we conclude that statement (i) of Theorem \\ref{thm:total-iter-cmplxity} holds.\r\n\r\n (ii) In view of Lemma \\ref{lem:level-set-augmented-lagrangian-func}(i) and the definition of\r\n $U_{k,H}$, one can see that \r\n\\[\r\nU_{k,H} \\ge \\sup_{x\\in\\bR^n}\\{\\|\\nabla^2_{xx} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\|: \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\}.\r\n\\] \r\n Using this, $\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)$ $\\le \\fh$ and Theorem \\ref{thm:NCG-iter-oper-cmplxity}(iv) with $(\\Fh,\\Fl,L_{H}^F,U_{H}^F,\\epsilon_g,\\epsilon_H)=(\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k),\r\n\\fl-\\gamma-\\Lambda\\bdelta_c,L_{k,H},U_{k,H},\\tau_k^g,\\tau_k^H)$, we obtain that\r\n\\begin{equation}\\label{eq:alg1-oper-kthAL}\r\n\\begin{array}{rcl}\r\nN_k&=&\\widetilde{\\cO}((\\fh-\\fl+\\gamma+\\Lambda\\bdelta_c)L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\}\\min\\{n,(U_{k,H}/\\tau_k^H)^{1/2}\\})\\\\[4pt]\r\n&\\overset{\\eqref{ineq:bound-norm-hessian-Liphessian}}{=}&\\widetilde{\\cO}(\\rho_k^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\}\\min\\{n,(\\rho_k/\\tau_k^H)^{1/2}\\})\\\\[4pt]\r\n&=&\\widetilde{\\cO}(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}\\min\\{n,\\epsilon_1^{-1}\\epsilon_2^{-1/2}\\}),\t\r\n\\end{array}\r\n\\end{equation}\r\n%\\begin{equation}\\label{eq:alg1-oper-kthAL}\r\n%\\begin{array}{rcl}\r\n%\\hspace{-2mm}N_k&=&\\widetilde{\\cO}((\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)-\\fl+\\gamma+\\Lambda\\bdelta_c)L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\}\\min\\{n,(U_{k,H}/\\tau_k^H)^{1/2}\\})\\\\[4pt]\r\n%&=&\\widetilde{\\cO}((\\fh-\\fl+\\gamma+\\Lambda\\bdelta_c)L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\}\\min\\{n,(U_{k,H}/\\tau_k^H)^{1/2}\\})\\\\[4pt]\r\n%&\\overset{\\eqref{ineq:bound-norm-hessian-Liphessian}}{=}&\\widetilde{\\cO}(\\rho_k^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\}\\min\\{n,(\\rho_k/\\tau_k^H)^{1/2}\\})\\\\[4pt]\r\n%&=&\\widetilde{\\cO}(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}\\min\\{n,\\epsilon_1^{-1}\\epsilon_2^{-1/2}\\}),\t\r\n%\\end{array}\r\n%\\end{equation}\r\nwhere the last equality  is from $(\\tau_k^g)^2/\\tau_k^H\\ge\\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$, $\\tau_k^H\\ge\\epsilon_2$, and $\\rho_k=\\cO(\\epsilon_1^{-2})$.\r\n\r\nOn the other hand, if $c$ is assumed to be affine, it follows from the above discussion that $L_{k,H}=\\cO(1)$. Using this, $U_{k,H}\\le U_1+\\rho_k U_2$, and similar arguments as for \\eqref{eq:alg1-oper-kthAL}, we obtain that $N_k=\\widetilde{\\cO}(\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}\\min\\{n,\\epsilon_1^{-1}\\epsilon_2^{-1/2}\\})$. By this, \\eqref{eq:alg1-oper-kthAL} and $\\overline{K}_{\\epsilon_1}=\\cO(|\\log\\epsilon_1|^2)$ (see Remark \\ref{order-out-itera-penalty}), we conclude that statement (ii) of Theorem \\ref{thm:total-iter-cmplxity} holds.\r\n\\end{proof}","sha256_utf8":"8e959f05c7eca16be7f15318bcfb94c14be130df16f17fb539d5ea353dcc857f"},{"start":167734,"end":168154,"start_line":1790,"end_line":1792,"raw_tex":"\\begin{theorem}[{\\bf iteration complexity of Algorithm \\ref{alg:capped-CG}}]\\label{lem:capped-CG}\r\nConsider applying Algorithm~\\ref{alg:capped-CG} with input $U = 0$ to the linear system~\\eqref{indef-sys} with $g\\neq 0$, $\\varepsilon>0$, and $H$ being an $n\\times n$ symmetric matrix. Then the number of iterations of Algorithm~\\ref{alg:capped-CG} is $\\widetilde{\\cO}(\\min\\{n,\\sqrt{\\|H\\|/\\varepsilon}\\})$.\r\n\\end{theorem}","sha256_utf8":"f7c4493ef27767b411a4741873fa7da7275ea5e2d75d698a3982a3b28b772460"},{"start":168158,"end":169051,"start_line":1794,"end_line":1799,"raw_tex":"\\begin{proof}\r\nFrom \\cite[Lemma~1]{RNW18}, we know that the number of iterations of Algorithm \\ref{alg:capped-CG} is bounded by $\\min\\{n,J(U,\\varepsilon,\\zeta)\\}$, where $J(U,\\varepsilon,\\zeta)$ is the smallest integer $J$ such that $\\sqrt{T}\\tau^{J/2}\\le\\widehat{\\zeta}$, with $U,\\widehat{\\zeta},T$ and $\\tau$ being the values returned by Algorithm \\ref{alg:capped-CG}. In addition, it was shown in \\cite[Section 3.1]{RNW18} that\r\n$J(U,\\varepsilon,\\zeta)\\le\\left\\lceil \\left(\\sqrt{\\kappa}+\\frac{1}{2}\\right)\\ln\\left(\\frac{144(\\sqrt{\\kappa}+1)^2\\kappa^6}{\\zeta^2}\\right)\\right\\rceil$,\r\nwhere $\\kappa={\\cO}(U/\\varepsilon)$ is an output by Algorithm \\ref{alg:capped-CG}. Then one can see that $J(U,\\varepsilon,\\zeta)=\\widetilde{\\cO}(\\sqrt{U/\\varepsilon})$. Notice from\r\nAlgorithm~\\ref{alg:capped-CG} that the output $U\\le\\|H\\|$. Combining these, we obtain the conclusion as desired.\r\n\\end{proof}","sha256_utf8":"9d043e6fddb51e9414a9e7b4b12912227dadf4c6bf875078f60f24efb43814f0"},{"start":170794,"end":171386,"start_line":1820,"end_line":1822,"raw_tex":"\\begin{theorem}[{\\bf iteration complexity of Algorithm~\\ref{pro:meo}}]\\label{rand-Lanczos}\r\nConsider Algorithm~\\ref{pro:meo} with tolerance $\\varepsilon>0$, probability parameter $\\delta\\in(0,1)$, and symmetric matrix $H\\in\\bR^{n\\times n}$ as its input. Then it either finds a sufficiently negative curvature direction $v$ satisfying $v^THv\\le-\\varepsilon/2$ and $\\|v\\|=1$ or certifies that $\\lambda_{\\min}(H)\\ge-\\varepsilon$ holds with probability at least  $1-\\delta$  in at most $N(\\varepsilon,\\delta)$ iterations, where $N(\\varepsilon,\\delta)$ is defined in \\eqref{N-iter}.\r\n\\end{theorem}","sha256_utf8":"b124349ac65adf475f45b63764024f4e30b0757660f7fb14dee3dea4637286cd"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"无需约束资格的总复杂度","source_units":[{"start":72661,"end":74050,"start_line":680,"end_line":688,"raw_tex":"\\begin{theorem}%[{\\bf Total inner iteration and operation complexity of Algorithm \\ref{alg:2nd-order-AL-nonconvex}}]\r\n\\label{thm:total-iter-cmplxity}\r\nSuppose that Assumption \\ref{asp:lowbd-knownfeas} holds and that $\\rho_0$ is sufficiently large such that $\\delta_{f,1} \\le \\bdelta_f$ and  $\\delta_{c,1} \\le \\bdelta_c$, where $\\delta_{f,1}$ and $\\delta_{c,1}$ are defined in \\eqref{def:delta0c-rhobar1xxx}. Then the following statements hold.\r\n\\begin{enumerate}[{\\rm (i)}]\r\n\\item %The total number of inner iterations of Algorithm \\ref{alg:2nd-order-AL-nonconvex}, namely,\r\nThe total number of iterations of Algorithm~\\ref{alg:NCG} performed in Algorithm \\ref{alg:2nd-order-AL-nonconvex} is at most $\\widetilde{\\cO}(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\})$. If $c$ is further assumed to be affine, then it is at most $\\widetilde{\\cO}(\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\})$.\r\n\\item The total number of matrix-vector products performed  by Algorithm~\\ref{alg:NCG} in  Algorithm \\ref{alg:2nd-order-AL-nonconvex} is at most $\\widetilde{\\cO}(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}\\min\\{n,\\epsilon_1^{-1}\\epsilon_2^{-1/2}\\})$. If $c$ is further assumed to be affine, then it is at most $\\widetilde{\\cO}(\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}\\min\\{n,\\epsilon_1^{-1}\\epsilon_2^{-1/2}\\})$.\r\n\\end{enumerate}\r\n\\end{theorem}","sha256_utf8":"d0b4a59201b4475c29d9b1c00700720f0ae625c47561ad021ebd07c730e0db75"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"总复杂度完整证明","source_units":[{"start":130682,"end":139592,"start_line":1250,"end_line":1320,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{thm:total-iter-cmplxity}]\r\nLet $T_k$ and $N_k$ denote the number of iterations and matrix-vector products performed by Algorithm~\\ref{alg:NCG} at the outer iteration $k$ of Algorithm \\ref{alg:2nd-order-AL-nonconvex}, respectively. It then follows from Theorem \\ref{thm:out-itr-cmplxity-1} that the total number of iterations and matrix-vector products performed by Algorithm~\\ref{alg:NCG} in Algorithm \\ref{alg:2nd-order-AL-nonconvex} are $\\sum_{k=0}^{\\overline{K}_{\\epsilon_1}}T_k$ and $\\sum_{k=0}^{\\overline{K}_{\\epsilon_1}}N_k$, respectively. In addition, notice from \\eqref{def:delta0c-rhobar1} and Theorem \\ref{thm:out-itr-cmplxity-1} that ${\\rho}_{\\epsilon_1}=\\cO(\\epsilon_1^{-2})$ and  $\\rho_k\\le{r}{\\rho}_{\\epsilon_1}$, which yield $\\rho_k=\\cO(\\epsilon_1^{-2})$.\r\n\r\n%(i) {  We first prove that $(\\tau_k^g)^2/\\tau_k^H\\ge\\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$ holds for any $k\\ge0$. If $\\epsilon_1^2{ \\ge}\\epsilon_2$, it follows from $\\tau_k^g\\ge\\epsilon_1$ and $\\tau_k^H\\le 1$ that $(\\tau_k^g)^2/\\tau_k^H\\ge(\\tau_k^g)^2\\ge \\epsilon_1^2{ \\ge}\\epsilon_2$, which along with $\\epsilon_2\\in(0,1)$ implies $(\\tau_k^g)^2/\\tau_k^H{ \\ge}\\epsilon_2^3$. On the other hand, if $\\epsilon_1^2{ <}\\epsilon_2$, it follows from \\eqref{omega-tolerance} that $\\omega_1^2/\\omega_2{ <} 1$. Also, recall from \\eqref{omega-tolerance}, \\eqref{T-epsilon-g} and \\eqref{epsw1-epsw2} that\r\n%\\[\r\n%K_{\\epsilon_1}=\\left\\lceil \\log\\epsilon_1/\\log\\omega_1\\right\\rceil=\\left\\lceil \\log\\epsilon_2/\\log\\omega_2\\right\\rceil,\\quad\r\n%(\\tau_k^g,\\tau_k^H)=\\left\\{\\begin{array}{ll}\r\n%(\\omega_1^k,\\omega_2^k)&\\text{if }\\ 0\\le k< K_{\\epsilon_1},\\\\\r\n%(\\epsilon_1,\\epsilon_2)&\\text{otherwise}.\r\n%\\end{array}\\right.\r\n%\\]\r\n%In view of these, we see that the sequence $\\{(\\tau_k^g)^2/\\tau_k^H\\}$ is geometrically decreasing with ratio $\\omega_1^2/\\omega_2{ <} 1$ when $0\\le k< K_{\\epsilon_1}$ and equals $\\epsilon_1^2/\\epsilon_2$ for all $k\\ge K_{\\epsilon_1}$. In addition, notice from \\eqref{T-epsilon-g} and \\eqref{epsw1-epsw2} that\r\n%\\[\r\n%\\left\\lceil\\log(\\epsilon_1^2/\\epsilon_2)/\\log(\\omega_1^2/\\omega_2)\\right\\rceil =\\left\\lceil \\log 2/\\log r\\right\\rceil = K_{\\epsilon_1}.\r\n%\\]\r\n%Thus, we have $(\\tau_k^g)^{2}/\\tau_k^H=\\max\\{\\epsilon_1^2/\\epsilon_2,(\\omega_1^2/\\omega_2)^k\\}\\ge\\epsilon_1^2/\\epsilon_2$ when $\\epsilon_1^2{ <}\\epsilon_2$. Combining this with the fact that $(\\tau_k^g)^2/\\tau_k^H{ \\ge}\\epsilon_2^3$ when $\\epsilon_1^2{ \\ge}\\epsilon_2$, we obtain that $(\\tau_k^g)^2/\\tau_k^H\\ge\\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$ holds for any $k\\ge0$.}\r\n\r\nWe first claim that $(\\tau_k^g)^2/\\tau_k^H\\ge\\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$ holds for any $k\\ge0$. Indeed, let $\\bar t=\\log\\epsilon_1/\\log\\omega_1$ and $\\psi(t)=\\max\\{\\epsilon_1,\\omega_1^t\\}^2/\\max\\{\\epsilon_2,\\omega_2^t\\}$ for all $t\\in\\bR$. It then follows from \\eqref{epsw1-epsw2} that $\\omega_1^{\\bar t}=\\epsilon_1$ and $\\omega_2^{\\bar t}=\\epsilon_2$. By this and $\\omega_1,\\omega_2\\in(0,1)$, one can observe that  $\\psi(t)=(\\omega_1^2/\\omega_2)^t$ if $t \\leq \\bar t$ and $\\psi(t)=\\epsilon_1^2/\\epsilon_2$ otherwise. This along with $\\epsilon_2\\in(0,1)$ implies that\r\n\\[\r\n\\min_{t\\in[0,\\infty)}\\psi(t) = \\min\\{\\psi(0),\\psi(\\bar t)\\}=\\min\\{1,\\epsilon_1^2/\\epsilon_2\\} \\geq \\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\},\r\n\\]\r\nwhich together with \\eqref{omega-tolerance} yields\r\n$(\\tau_k^g)^2/\\tau_k^H=\\psi(k) \\geq \\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$ for all $k \\geq 0$.\r\n\r\n(i) From Lemma \\ref{lem:level-set-augmented-lagrangian-func}(i) and the definitions of $\\Omega(\\bdelta_{f},\\bdelta_{c})$ and $L_{k,H}$, we see that $L_{k,H}$ is a Lipschitz constant of $\\nabla^2_{xx} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$ on a convex open neighborhood  of $\\{x:\\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\}$.\r\n% In addition, by Lemma \\ref{lem:level-set-augmented-lagrangian-func}(ii) and the assumption $\\delta_{c,1} \\le \\bdelta_c$, one has $\\inf_x \\cL(x,\\lambda^k;\\rho_k) \\ge\\fl-\\Lambda\\bdelta_c$.\r\nAlso, recall from Lemma \\ref{lem:level-set-augmented-lagrangian-func}(ii) that $\\inf_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k) $ $\\ge\\fl-\\gamma-\\Lambda\\bdelta_c$. By these, $\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\le \\fh$ (see \\eqref{L-xinit}) and Theorem \\ref{thm:NCG-iter-oper-cmplxity}(iii) with $(\\Fh,\\Fl,L_{H}^F,\\epsilon_g,\\epsilon_H)=(\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k),\\fl-\\gamma-\\Lambda \\bdelta_c,L_{k,H},\\tau_k^g,\\tau_k^H)$, one has\r\n\\begin{equation}\\label{eq:alg1-inner-iter-kthAL}\r\n\\begin{array}{rcl}\r\nT_k&=&\\cO((\\fh-\\fl+\\gamma+\\Lambda\\bdelta_{c})L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\})\\\\[2pt]\r\n&\\overset{\\eqref{ineq:bound-norm-hessian-Liphessian}}{=}&\\cO(\\rho_k^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\})=\\cO(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}),\r\n\\end{array}\r\n\\end{equation}\r\n%\\begin{equation}\\label{eq:alg1-inner-iter-kthAL}\r\n%\\begin{array}{rcl}\r\n%T_k&=&\\cO((\\cL(x^{k}_{\\init},\\lambda^k;\\rho_k)-\\fl+\\gamma+\\Lambda\\bdelta_{c})L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\})\\\\[4pt]\r\n%&=&\\cO((\\fh-\\fl+\\gamma+\\Lambda\\bdelta_{c})L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\})\\\\[4pt]\r\n%&\\overset{\\eqref{ineq:bound-norm-hessian-Liphessian}}{=}&\\cO(\\rho_k^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\})=\\cO(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}),\r\n%\\end{array}\r\n%\\end{equation}\r\nwhere the last equality is from $(\\tau_k^g)^2/\\tau_k^H\\ge\\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$, $\\tau_k^H\\ge\\epsilon_2$, and $\\rho_k=\\cO(\\epsilon_1^{-2})$.\r\n\r\nNext, if %$c$ is assumed to be affine, namely,\r\n$c(x)=Ax-b$ for some $A\\in\\bR^{m\\times n}$ and $b\\in\\bR^m$, then $\\nabla c(x)=A^T$ and $\\nabla^2 c_i(x)=0$ for $1 \\le i \\le m$. By these and \\eqref{eq:hessian-augmented-Lagrangian}, one has $L_{k,H}=\\cO(1)$. Using this and similar arguments as for \\eqref{eq:alg1-inner-iter-kthAL}, we obtain that $T_k=\\cO(\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\})$. By this, \\eqref{eq:alg1-inner-iter-kthAL} and $\\overline{K}_{\\epsilon_1}=\\cO(|\\log\\epsilon_1|^2)$ (see Remark \\ref{order-out-itera-penalty}), we conclude that statement (i) of Theorem \\ref{thm:total-iter-cmplxity} holds.\r\n\r\n (ii) In view of Lemma \\ref{lem:level-set-augmented-lagrangian-func}(i) and the definition of\r\n $U_{k,H}$, one can see that \r\n\\[\r\nU_{k,H} \\ge \\sup_{x\\in\\bR^n}\\{\\|\\nabla^2_{xx} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\|: \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\}.\r\n\\] \r\n Using this, $\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)$ $\\le \\fh$ and Theorem \\ref{thm:NCG-iter-oper-cmplxity}(iv) with $(\\Fh,\\Fl,L_{H}^F,U_{H}^F,\\epsilon_g,\\epsilon_H)=(\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k),\r\n\\fl-\\gamma-\\Lambda\\bdelta_c,L_{k,H},U_{k,H},\\tau_k^g,\\tau_k^H)$, we obtain that\r\n\\begin{equation}\\label{eq:alg1-oper-kthAL}\r\n\\begin{array}{rcl}\r\nN_k&=&\\widetilde{\\cO}((\\fh-\\fl+\\gamma+\\Lambda\\bdelta_c)L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\}\\min\\{n,(U_{k,H}/\\tau_k^H)^{1/2}\\})\\\\[4pt]\r\n&\\overset{\\eqref{ineq:bound-norm-hessian-Liphessian}}{=}&\\widetilde{\\cO}(\\rho_k^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\}\\min\\{n,(\\rho_k/\\tau_k^H)^{1/2}\\})\\\\[4pt]\r\n&=&\\widetilde{\\cO}(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}\\min\\{n,\\epsilon_1^{-1}\\epsilon_2^{-1/2}\\}),\t\r\n\\end{array}\r\n\\end{equation}\r\n%\\begin{equation}\\label{eq:alg1-oper-kthAL}\r\n%\\begin{array}{rcl}\r\n%\\hspace{-2mm}N_k&=&\\widetilde{\\cO}((\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)-\\fl+\\gamma+\\Lambda\\bdelta_c)L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\}\\min\\{n,(U_{k,H}/\\tau_k^H)^{1/2}\\})\\\\[4pt]\r\n%&=&\\widetilde{\\cO}((\\fh-\\fl+\\gamma+\\Lambda\\bdelta_c)L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\}\\min\\{n,(U_{k,H}/\\tau_k^H)^{1/2}\\})\\\\[4pt]\r\n%&\\overset{\\eqref{ineq:bound-norm-hessian-Liphessian}}{=}&\\widetilde{\\cO}(\\rho_k^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\}\\min\\{n,(\\rho_k/\\tau_k^H)^{1/2}\\})\\\\[4pt]\r\n%&=&\\widetilde{\\cO}(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}\\min\\{n,\\epsilon_1^{-1}\\epsilon_2^{-1/2}\\}),\t\r\n%\\end{array}\r\n%\\end{equation}\r\nwhere the last equality  is from $(\\tau_k^g)^2/\\tau_k^H\\ge\\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$, $\\tau_k^H\\ge\\epsilon_2$, and $\\rho_k=\\cO(\\epsilon_1^{-2})$.\r\n\r\nOn the other hand, if $c$ is assumed to be affine, it follows from the above discussion that $L_{k,H}=\\cO(1)$. Using this, $U_{k,H}\\le U_1+\\rho_k U_2$, and similar arguments as for \\eqref{eq:alg1-oper-kthAL}, we obtain that $N_k=\\widetilde{\\cO}(\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}\\min\\{n,\\epsilon_1^{-1}\\epsilon_2^{-1/2}\\})$. By this, \\eqref{eq:alg1-oper-kthAL} and $\\overline{K}_{\\epsilon_1}=\\cO(|\\log\\epsilon_1|^2)$ (see Remark \\ref{order-out-itera-penalty}), we conclude that statement (ii) of Theorem \\ref{thm:total-iter-cmplxity} holds.\r\n\\end{proof}","sha256_utf8":"8e959f05c7eca16be7f15318bcfb94c14be130df16f17fb539d5ea353dcc857f"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"非凸等式约束问题","source_units":[{"start":10189,"end":10345,"start_line":187,"end_line":190,"raw_tex":"In this paper we consider nonconvex equality constrained optimization problem\r\n\\beq\\label{model:equa-cnstr}\r\n\\min_{x\\in \\bR^n}\\ f(x)\\quad \\st\\ c(x)=0,\r\n\\eeq","sha256_utf8":"0d0f6c0525ca53820a91acaa5ea9b152ed4f80114628b9bca25cbef3ad0c463e"},{"start":10347,"end":11379,"start_line":191,"end_line":191,"raw_tex":"where $f:\\bR^n\\to\\bR$ and $c:\\bR^n\\to\\bR^m$ are twice continuously differentiable, and we assume that problem~\\eqref{model:equa-cnstr} has at least one optimal solution. Since \\eqref{model:equa-cnstr} is a nonconvex optimization problem, it may have many local but non-global minimizers and finding its global minimizer is generally NP-hard. A first-order stationary point (FOSP) of it is usually found in practice instead. Nevertheless,  a mere FOSP may sometimes not suit our needs and a  \\emph{second-order stationary point} (SOSP) needs to be sought. For example, in the context of linear semidefinite programming (SDP), a powerful approach to solving it is by solving an equivalent nonconvex equality constrained optimization problem \\cite{BuMo03-1,BuMo05-1}. It was shown in \\cite{BuMo05-1,BVB16} that under some mild conditions an SOSP of the latter problem can yield an optimal solution of  the linear SDP, while a mere FOSP generally cannot.  It is therefore important to find an SOSP of problem~\\eqref{model:equa-cnstr}.\r\n","sha256_utf8":"ec25db1aa2183305549d74c93d7a2f47a09cb54ee5eda53910304f7d5fba2f98"}]},{"id":"S004","semantic_type":"notation","kind":"notation","title":"范数与雅可比记号","source_units":[{"start":29839,"end":30706,"start_line":282,"end_line":283,"raw_tex":"Throughout this paper, we let $\\bR^n$ denote the $n$-dimensional Euclidean space. We use $\\|\\cdot\\|$ to denote the Euclidean norm of a vector or the spectral norm of a matrix. For a real symmetric matrix $H$, we use $\\lambda_{\\min}(H)$ to denote its minimum eigenvalue.\r\nThe Euclidean ball centered at the origin with radius $R\\ge0$ is denoted by $\\mathcal{B}_R:=\\{x:\\|x\\|\\le R\\}$, and we use $\\Pi_{\\mathcal{B}_R}(v)$ to denote the Euclidean projection of a vector $v$ onto $\\mathcal{B}_R$. For a given finite set $\\cA$, we let $|\\cA|$ denote its cardinality. For any $s\\in \\mathbb{R}$, we let ${\\rm sgn}(s)$ be $1$ if $s \\ge 0$ and let it be $-1$ otherwise. In addition, $\\widetilde{\\cO}(\\cdot)$ represents $\\cO(\\cdot)$ with logarithmic terms omitted. %In addition, for any $s\\in \\mathbb{R}$, we let ${\\rm sgn}(s)$ be $1$ if $s \\ge 0$ and let it be $-1$ otherwise.\r\n","sha256_utf8":"c73ded9ca3704f2854921d298aea325f83da5d58140eee85e6ed54e9353455da"},{"start":30861,"end":30942,"start_line":286,"end_line":286,"raw_tex":"$\\nabla c(x^*):=[\\nabla c_1(x^*) \\ \\nabla c_2(x^*) \\ \\cdots \\ \\nabla c_m(x^*)]$\r\n","sha256_utf8":"fb7b8bb32f90b7d614e244d75b0244ac29a397a6bbbbde21a3583031423bb27b"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"约束切空间","source_units":[{"start":31305,"end":31405,"start_line":293,"end_line":295,"raw_tex":"\\begin{equation}\\label{def:critical-cone} \r\n\\cC(x):=\\{d\\in\\bR^n:\\nabla c(x)^T d=0\\}.\r\n\\end{equation}","sha256_utf8":"3c4082685c81665869d1685b07deef9f286fed55d9eb968c7dbbbd5bcafe5182"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"近似一阶驻点","source_units":[{"start":32420,"end":32879,"start_line":301,"end_line":306,"raw_tex":"\\begin{definition}[{{\\bf $\\epsilon_1$-first-order stationary point}}]\\label{def:FOSP}\r\nLet $\\epsilon_1>0$. We say that $x\\in\\bR^n$ is an $\\epsilon_1$-first-order stationary point {\\rm ($\\epsilon_1$-FOSP)} of problem \\eqref{model:equa-cnstr} if it, together with some $\\lambda\\in\\bR^m$, satisfies \r\n\\begin{equation}\\label{optcond:1st-equa-cnstr} \r\n\\|\\nabla f(x)+\\nabla c(x)\\lambda\\|\\le \\epsilon_1,\\quad \\|c(x)\\|\\le \\epsilon_1.\r\n\\end{equation}\r\n\\end{definition}","sha256_utf8":"3c4f8291bd638421f5f6354ac0ea0d37bbbefdbcc781b7c6dfafbace32a3eaf9"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"近似二阶驻点","source_units":[{"start":32884,"end":33547,"start_line":308,"end_line":316,"raw_tex":"\\begin{definition}[{{\\bf $(\\epsilon_1,\\epsilon_2)$-second-order stationary point}}]\\label{def:SOSP}\r\nLet $\\epsilon_1,\\epsilon_2>0$. We say that $x\\in\\bR^n$ is an $(\\epsilon_1,\r\n\\epsilon_2)$-second-order stationary point {\\rm (($\\epsilon_1,\r\n\\epsilon_2$)-SOSP)} of problem \\eqref{model:equa-cnstr} if it, together with some $\\lambda\\in\\bR^m$, satisfies \\eqref{optcond:1st-equa-cnstr} and additionally \r\n\\begin{equation}\\label{optcond:2nd-equa-cnstr} \r\nd^T\\left(\\nabla^2 f(x)+\\sum_{i=1}^m\\lambda_i\\nabla^2 c_i(x)\\right)d\\ge-\\epsilon_2\\|d\\|^2,\\quad \\forall d\\in\\cC(x),\r\n\\end{equation}\r\nwhere $\\cC(\\cdot)$ is defined as in \\eqref{def:critical-cone}.\r\n\\end{definition}","sha256_utf8":"d7c04b559ea43b9f1872cabf9c5be36cab3b7ec1c7361d867a968972309e1ba2"}]},{"id":"S008","semantic_type":"problem_setting","kind":"problem_setting","title":"无约束子问题","source_units":[{"start":33963,"end":34047,"start_line":322,"end_line":324,"raw_tex":"\\begin{equation}\\label{unconstrained-prob} \r\n\\min_{x\\in\\bR^n}\\ F(x),\r\n\\end{equation}","sha256_utf8":"6984bb855072f3ff943fefd64437b787e2274123ce3b03f935cbce0450beade1"}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","title":"子问题紧性与光滑性","source_units":[{"start":34112,"end":34863,"start_line":327,"end_line":339,"raw_tex":"\\begin{assumption}\\label{asp:NCG-cmplxity}\r\n\\begin{enumerate}[{\\rm (a)}]\r\n\\item The level set $\\scL_F(u^0):=\\{x: F(x)\\le F(u^0)\\}$ is compact for some $u^0\\in \\bR^n$.%, where $z^0\\in\\bR^n$ is the initial iterate of the Newton-CG method in \\cite{RNW18}.\r\n\\item The function $F$ is twice Lipschitz continuously differentiable in\r\n%{\\color{black}some} convex open neighborhood $\\Omega$ of $\\{x|\\  F(x)\\le F(u^0)\\}$,\r\n%the convex hull of $\\Omega$, denoted by $\\conv(\\Omega)$,\r\na convex open neighborhood, denoted by $\\Omega$, of $\\scL_F(u^0)$,\r\nthat is, there exists $L_{H}^F>0$ such that\r\n\\begin{equation}\\label{F-Hess-Lip}\r\n\\|\\nabla^2 F(x)-\\nabla^2 F(y)\\|\\le L_{H}^F\\|x-y\\|,\\quad \\forall x,y\\in \\Omega.\r\n\\end{equation}\r\n\\end{enumerate}\r\n\\end{assumption}","sha256_utf8":"c6204f4910089cd9e9856894698f3c03766bee3a3ff1a1a8208cfef57f5c7c9f"}]},{"id":"S010","semantic_type":"derived_fact","kind":"derived_fact","title":"水平集上的统一界","source_units":[{"start":34867,"end":35128,"start_line":341,"end_line":344,"raw_tex":"By Assumption \\ref{asp:NCG-cmplxity}, there exist $\\Fl\\in \\bR$, $U_g^F>0$ and $U_H^F>0$ such that\r\n\\begin{equation}\\label{lwbd-Hgupbd}\r\nF(x)\\ge \\Fl,\\quad \\|\\nabla F(x)\\|\\le U_g^F,\\quad \\|\\nabla^2 F(x)\\|\\le U_H^F, \\quad \\forall x\\in\\scL_F(u^0).\r\n\\end{equation}\r\n","sha256_utf8":"9b0d0c24fd40c80e521968f37bbf9e95b45840aa58a745aeb3320c034b850319"}]},{"id":"S011","semantic_type":"problem_setting","kind":"problem_setting","title":"阻尼系统与两类预言机","source_units":[{"start":39469,"end":39745,"start_line":368,"end_line":371,"raw_tex":"The first main component of the Newton-CG method in \\cite{RNW18} is a {\\it capped CG method} \\cite[Algorithm~1]{RNW18}, which is a modified CG method, for solving a possibly indefinite linear system \r\n\\begin{equation}\\label{indef-sys} \r\n(H+2\\varepsilon I)d=-g,\r\n\\end{equation}","sha256_utf8":"6d88d1db14a9d2becedf0a88781065f1ea27b6d390c43581957f03723739171a"},{"start":39747,"end":40684,"start_line":372,"end_line":372,"raw_tex":"where $0\\neq g\\in\\bR^n$, $\\varepsilon>0$, and $H\\in\\bR^{n\\times n}$ is a symmetric matrix. This capped CG method terminates within a finite number of iterations. It outputs either an approximate solution $d$ to \\eqref{indef-sys} such that $\\|(H+2\\varepsilon I)d+g\\|\\le\\widehat{\\zeta}\\|g\\|$ and $d^THd\\ge-\\varepsilon\\|d\\|^2$ for some $\\widehat{\\zeta}\\in(0,1)$ or a sufficiently negative curvature direction $d$ of $H$ with $d^THd<-\\varepsilon\\|d\\|^2$. The second main component of the Newton-CG method in \\cite{RNW18} is a minimum eigenvalue oracle that either produces a sufficiently negative curvature direction $v$ of $H$ with $\\|v\\|=1$ and  $v^THv\\le-\\varepsilon/2$ or certifies that $\\lambda_{\\min}(H)\\ge-\\varepsilon$ holds with high probability. For ease of reference, we present these two components in Algorithms~\\ref{alg:capped-CG}  and \\ref{pro:meo} in Appendices~\\ref{appendix:capped-CG} and \\ref{appendix:meo}, respectively.\r\n","sha256_utf8":"386ca237be8a6cbac5406fcf007ffe9001c74190b7f71267e986eef35982e64d"}]},{"id":"S012","semantic_type":"algorithm","kind":"algorithm","title":"混合线搜索牛顿共轭梯度法","source_units":[{"start":40686,"end":42771,"start_line":374,"end_line":425,"raw_tex":"\\begin{algorithm}[h]\r\n\\caption{A Newton-CG method for problem \\eqref{unconstrained-prob}}\r\n\\label{alg:NCG}\r\n{\\footnotesize\r\n\\begin{algorithmic}\r\n\\State \\noindent\\textit{Input}: Tolerances $\\epsilon_g,\\epsilon_H\\in(0,1)$, backtracking ratio $\\theta\\in(0,1)$, starting point $u^0$, CG-accuracy parameter $\\zeta\\in(0,1)$, line-search parameter $\\eta\\in(0,1)$, probability parameter $\\delta\\in(0,1)$.\\\\\r\nSet $x^0=u^0$;\r\n\\For{$t=0,1,2,\\ldots$}\r\n\\If{$\\|\\nabla F(x^t)\\|>\\epsilon_g$}\r\n\\State Call Algorithm~\\ref{alg:capped-CG} with $H=\\nabla^2 F(x^t)$, $\\varepsilon=\\epsilon_H$, $g=\\nabla F(x^t)$, accuracy parameter $\\zeta$, and $U=0$ to obtain outputs $d$,\r\n\\State d$\\_$type;\r\n\\If{d$\\_$type=NC}\r\n%\\State\r\n\\begin{equation}\\label{dk-nc}\r\nd^t\\leftarrow -\\sgn(d^T\\nabla F(x^t))\\frac{|d^T\\nabla^2 F(x^t) d|}{\\|d\\|^3}d;\r\n\\end{equation}\r\n\\Else\\ \\{d$\\_$type=SOL\\}\r\n%\\State\r\n\\begin{equation}\\label{dk-sol}\r\nd^t\\leftarrow d;\r\n\\end{equation}\r\n\\EndIf\r\n\\State Go to {\\bf Line Search};\r\n\\Else\r\n\\State Call Algorithm~\\ref{pro:meo} with $H=\\nabla^2 F(x^t)$, $\\varepsilon=\\epsilon_H$, and probability parameter $\\delta$;\r\n\\If{Algorithm~\\ref{pro:meo} certifies that $\\lambda_{\\min}(\\nabla^2 F(x^t))\\ge-\\epsilon_H$}\r\n\\State Output $x^t$ and terminate;\r\n\\Else\\ \\{Sufficiently negative curvature direction $v$ returned by Algorithm~\\ref{pro:meo}\\}\r\n\\State Set d$\\_$type=NC and\r\n\\begin{equation}\\label{dk-2nd-nc}\r\nd^t\\leftarrow -\\sgn(v^T\\nabla F(x^t))|v^T\\nabla^2F(x^t)v|v;\r\n\\end{equation}\r\n\\State Go to {\\bf Line Search};\r\n\\EndIf\r\n\\EndIf\r\n\\State{\\bf Line Search:}\r\n\\If{d$\\_$type=SOL}\r\n\\State Find $\\alpha_t=\\theta^{j_t}$, where $j_t$ is the smallest nonnegative integer $j$ such that\r\n\\begin{equation}\\label{ls-sol}\r\nF(x^t+\\theta^jd^t)<F(x^t)-\\eta\\epsilon_H\\theta^{2j}\\|d^t\\|^2;\r\n\\end{equation}\r\n\\Else\\ \\{d$\\_$type=NC\\}\r\n\\State Find $\\alpha_t=\\theta^{j_t}$, where $j_t$ is the smallest nonnegative integer $j$ such that\r\n\\begin{equation}\\label{ls-nc}\r\nF(x^t+\\theta^jd^t)<F(x^t)-\\eta\\theta^{2j}\\|d^t\\|^3/2;\r\n\\end{equation}\r\n\\EndIf\r\n\\State $x^{t+1}=x^t+\\alpha_td^t$;\r\n\\EndFor\r\n\\end{algorithmic}\r\n}\r\n\\end{algorithm}","sha256_utf8":"f1b083d5e3850c5bf6ea60ca44648301e635f88a5f99557a86c09cfc21beea82"}]},{"id":"S013","semantic_type":"assumption","kind":"assumption","title":"增广拉格朗日基本假设","source_units":[{"start":48220,"end":49366,"start_line":481,"end_line":498,"raw_tex":"\\begin{assumption}\\label{asp:lowbd-knownfeas}\t\t\r\n\\begin{enumerate}[{\\rm (a)}]\r\n\\item An $\\epsilon_1/2$-approximately feasible point $z_{\\epsilon_1}$ of problem~\\eqref{model:equa-cnstr}, namely satisfying $\\|c(z_{\\epsilon_1})\\|\\le\\epsilon_1/2$, is known.\t\r\n\\item There exist constants $\\fh$, $\\fl$ and $\\gamma>0$, independent of $\\epsilon_1$ and $\\epsilon_2$,  such that\r\n\\begin{eqnarray}\r\n&& f(z_{\\epsilon_1})\\le\\fh, \\label{hbd}\\\\\r\n&& f(x)+\\gamma\\|c(x)\\|^2/2\\ge \\fl,\\quad \\forall x\\in\\bR^n, \\label{lbd}\r\n\\end{eqnarray}\r\n where $z_{\\epsilon_1}$ is given in (a).\r\n\\item %{  For any $\\epsilon \\in (0,1)$ and the corresponding $z^0$ from (b),}\r\nThere exist some $\\bdelta_f,\\bdelta_c>0$ such that the set\r\n\\begin{equation}\\label{nearly-feas-level-set}\r\n\\cS(\\bdelta_f,\\bdelta_c):=\\{x:f(x)\\le \\fh+\\bdelta_f,\\ \\|c(x)\\|\\le1+\\bdelta_c\\}\r\n\\end{equation}\r\n%is compact, { with $\\fh$ as in \\eqref{hbd}}. \r\nis compact with $\\fh$ given above. Also, $\\nabla^2 f$ and $\\nabla^2 c_i,\\ i=1,2,\\ldots,m$, are Lipschitz continuous in a convex open neighborhood, denoted by $\\Omega(\\bdelta_f,\\bdelta_c)$, of $\\cS(\\bdelta_f,\\bdelta_c)$.\r\n\\end{enumerate}\r\n\\end{assumption}","sha256_utf8":"2090a19b451da2b6ad7b1df1fdee680b8857f7e5b1ab93a8f15eeec0d4887aad"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"扰动约束与增广函数","source_units":[{"start":53806,"end":53942,"start_line":519,"end_line":521,"raw_tex":"\\begin{equation}\\label{model:equa-cnstr-pert}\r\n\\min_{x\\in \\bR^n}\\ f(x)\\quad \\st\\ \\tilde{c}(x):=c(x)-c(z_{\\epsilon_1})=0,\r\n\\end{equation}","sha256_utf8":"eb2fae3cfb2deaadc996cd9a5c74c76c5b8349d38ff0aeb1239a353fdb3de556"},{"start":54244,"end":54415,"start_line":523,"end_line":525,"raw_tex":"\\begin{equation}\\label{tL-tc}\r\n\\min_{x\\in\\bR^n} \\big\\{\\widetilde{\\cL}(x,\\lambda^k,\\rho_k):= f(x)+(\\lambda^k)^T\\tilde{c}(x)+\\rho_k\\|\\tilde{c}(x)\\|^2/2\\big\\}\r\n\\end{equation}","sha256_utf8":"78330de4c947d372f5df356a78ce625dfb448b058d04b2c913ca519cb83f11f7"}]},{"id":"S015","semantic_type":"algorithm","kind":"algorithm","title":"保障乘子的增广拉格朗日法","source_units":[{"start":55931,"end":58049,"start_line":531,"end_line":560,"raw_tex":"\\begin{algorithm}[h]\r\n\\caption{A Newton-CG based AL method for problem\r\n\\eqref{model:equa-cnstr}}\r\n\\label{alg:2nd-order-AL-nonconvex}\r\n{\\small\r\nLet $\\gamma$ be given in Assumption \\ref{asp:lowbd-knownfeas}.\\\\\r\n\\noindent\\textbf{Input}: $\\epsilon_1,\\epsilon_2\\in(0,1)$, $\\Lambda>0$, $x^0\\in\\bR^n$, $\\lambda^0 \\in \\mathcal{B}_{\\Lambda}$, $\\rho_0>2\\gamma$, $\\alpha\\in(0,1)$, ${r}>1$, $\\delta\\in(0,1)$, and  $z_{\\epsilon_1}$ given in Assumption~\\ref{asp:lowbd-knownfeas}.\r\n\\begin{algorithmic}[1]\r\n\\State Set $k=0$.\r\n\\State Set $\\tau_k^g=\\max\\{\\epsilon_1, {r}^{k\\log\\epsilon_1/\\log 2}\\}$ and $\\tau_k^H=\\max\\{\\epsilon_2, {r}^{k\\log\\epsilon_2/\\log 2}\\}$.\r\n\\State  Call Algorithm \\ref{alg:NCG} with $\\epsilon_g=\\tau_k^g$, $\\epsilon_H=\\tau_k^H$ and $u^0=x^{k}_{\\init}$ to find an approximate solution $x^{k+1}$ to $\\min_{x\\in\\bR^n}\\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$ such that \r\n\\begin{eqnarray}\r\n&&\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k)\\le f(z_{\\epsilon_1}),\\ \\|\\nabla_x \\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k)\\|\\le\\tau_k^g,\\label{algstop:1st-order}\\\\\r\n&&\\lambda_{\\min}(\\nabla^2_{xx}\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k))\\ge-\\tau_k^H\\  \\text{with probability at least } 1-\\delta, \\label{algstop:2nd-order}\r\n\\end{eqnarray}\r\nwhere\r\n\\begin{equation}\\label{def:initial-iterate-subprob}\r\nx^k_{\\init}=\\left\\{\\begin{array}{ll}\r\nz_{\\epsilon_1}&\\text{if }\\ \\widetilde{\\cL}(x^k,\\lambda^k;\\rho_k)> f(z_{\\epsilon_1}),\\\\\r\nx^{k}&\\text{otherwise},\r\n\\end{array}\\right.\\quad\\text{for } k\\ge 0.\r\n\\end{equation}\\label{algstp:ALsubpb}\r\n\\State Set $\\tilde{\\lambda}^{k+1}=\\lambda^k+\\rho_k\\tilde{c}(x^{k+1})$.\r\n\\State If $\\tau_k^g\\le\\epsilon_1$, $\\tau_k^H\\le\\epsilon_2$ and $\\|c(x^{k+1})\\|\\le\\epsilon_1$, then output $(x^{k+1},\\tilde{\\lambda}^{k+1})$ and terminate.\\label{algstep:stop}\r\n\\State Set $\\lambda^{k+1}=\\Pi_{\\mathcal{B}_\\Lambda}(\\tilde{\\lambda}^{k+1})$.\\label{algstep:proj-multiplier}\r\n\\State If $k=0$ or $\\|\\tilde{c}(x^{k+1})\\|>\\alpha\\|\\tilde{c}(x^k)\\|$, set $\\rho_{k+1}={r}\\rho_k$. Otherwise, set $\\rho_{k+1}=\\rho_k$.\\label{altstep:penalty}\r\n\\State Set $k\\leftarrow k+1$, and go to step 2.\r\n\\end{algorithmic}\r\n}\r\n\\end{algorithm}","sha256_utf8":"71b60c2a2e209a4983f0997bc9b97ea8791132daa930ede6535dedb4cf1112d5"}]},{"id":"S016","semantic_type":"definition","kind":"definition","title":"三种复杂度度量","source_units":[{"start":67364,"end":68317,"start_line":624,"end_line":629,"raw_tex":"In the rest of this section, we study the worst-case complexity of Algorithm \\ref{alg:2nd-order-AL-nonconvex}. Since our method has two nested loops, particularly,  outer loops executed by the AL method and inner loops executed by the Newton-CG method for solving the AL subproblems, we consider the following measures of complexity for Algorithm \\ref{alg:2nd-order-AL-nonconvex}.\r\n\\begin{itemize}\r\n\\item \\emph{Outer iteration complexity}, which measures the number of outer iterations of Algorithm \\ref{alg:2nd-order-AL-nonconvex};\r\n\\item  \\emph{Total inner iteration complexity}, which measures the total number of iterations of the Newton-CG method that are performed in Algorithm \\ref{alg:2nd-order-AL-nonconvex};\r\n\\item \\emph{Operation complexity}, which measures the total number of matrix-vector products involving the Hessian of the augmented Lagrangian function that are evaluated in Algorithm \\ref{alg:2nd-order-AL-nonconvex}.\r\n\\end{itemize}\r\n","sha256_utf8":"d8d63658f7a4c26ece661626521fb55c8eef9952c39cad938ed1a1d35592c9d9"}]},{"id":"S017","semantic_type":"notation","kind":"notation","title":"递减容差与同步阈值","source_units":[{"start":68602,"end":69448,"start_line":636,"end_line":644,"raw_tex":"For notational convenience, we rewrite $(\\tau_k^g,\\tau_k^H)$ arising in Algorithm \\ref{alg:2nd-order-AL-nonconvex} as\r\n\\begin{equation}\\label{omega-tolerance}\r\n(\\tau_k^g,\\tau_k^H)=(\\max\\{\\epsilon_1,\\omega_1^k\\},\\max\\{\\epsilon_2,\\omega_2^k\\})\\ \\mbox{ with }\\ (\\omega_1,\\omega_2):=({r}^{\\log\\epsilon_1/\\log 2},{r}^{\\log\\epsilon_2/\\log 2}),\r\n\\end{equation}\r\nwhere $\\epsilon_1$, $\\epsilon_2$ and $r$ are the input parameters of Algorithm \\ref{alg:2nd-order-AL-nonconvex}. Since $r>1$ and $\\epsilon_1,\\epsilon_2\\in(0,1)$, it is not hard to verify that $\\omega_1,\\omega_2\\in(0,1)$.\r\nAlso, we introduce the following quantity that will be used frequently later:\r\n\\begin{equation}\r\nK_{\\epsilon_1}:= \\left\\lceil \\min\\{k\\ge 0:\\omega_1^{k}\\le\\epsilon_1\\} \\right\\rceil=\\left\\lceil\\log\\epsilon_1/\\log\\omega_1\\right\\rceil. \\label{T-epsilon-g}\r\n\\end{equation}\r\n","sha256_utf8":"7ea8b2257d245867ded47d90698cc6e4dab480ac307e7c3fe4723bcb7393613e"}]},{"id":"S018","semantic_type":"derived_fact","kind":"derived_fact","title":"梯度与函数泰勒余项界","source_units":[{"start":89743,"end":90085,"start_line":819,"end_line":823,"raw_tex":"One can observe from Assumption~\\ref{asp:NCG-cmplxity}(b) that for all $x$ and $y\\in \\Omega$,\r\n\\begin{eqnarray}\r\n&&\\ \\ \\ \\|\\nabla F(y)-\\nabla F(x)-\\nabla^2 F(x)(y-x)\\|\\le L_H^F\\|y-x\\|^2/2, \\label{apx-nxt-grad}\\\\\r\n&&\\ \\ \\ F(y)\\le F(x) + \\nabla F(x)^T(y-x) + (y-x)^T\\nabla^2 F(x) (y-x)/2 +  L_H^F\\|y-x\\|^3/6. \\label{desc-ineq}\r\n\\end{eqnarray}\r\n","sha256_utf8":"8a4213c6cb5e6f385f173f59a4d74867b94cad89696aa8c5867a3d49908ed366"}]},{"id":"S019","semantic_type":"algorithm","kind":"algorithm","title":"截断共轭梯度算法","source_units":[{"start":165040,"end":167637,"start_line":1733,"end_line":1786,"raw_tex":"\\begin{algorithm}[h]\r\n\\caption{A capped conjugate gradient method}\r\n\\label{alg:capped-CG}\r\n{\\footnotesize\r\n\\begin{algorithmic}\r\n\\State \\noindent\\textit{Inputs}: symmetric matrix $H\\in\\bR^{n\\times n}$, vector $g\\neq0$, damping parameter $\\varepsilon\\in(0,1)$, desired relative accuracy $\\zeta\\in(0,1)$.\r\n\\State \\textit{Optional input:} scalar $U\\ge0$ (set to $0$ if not provided).\r\n\\State \\textit{Outputs:} d$\\_$type, $d$.\r\n\\State \\textit{Secondary outputs:} final values of $U,\\kappa,\\widehat{\\zeta},\\tau,$ and $T$.\r\n\\State Set\r\n\\begin{equation*}\r\n\\textstyle\r\n\\bar{H}:=H+2\\varepsilon I,\\quad \\kappa:=\\frac{U+2\\varepsilon}{\\varepsilon},\\quad\\widehat{\\zeta}:=\\frac{\\zeta}{3\\kappa},\\quad\\tau:=\\frac{\\sqrt{\\kappa}}{\\sqrt{\\kappa}+1},\\quad T:=\\frac{4\\kappa^4}{(1-\\sqrt{\\tau})^2},\r\n\\end{equation*}\r\n$y^0\\leftarrow 0,r^0\\leftarrow g,p^0\\leftarrow -g, j\\leftarrow 0$.\r\n\\If {$(p^0)^T \\bar{H}p^0<\\varepsilon\\|p^0\\|^2$}\r\n\\State Set $d\\leftarrow p^0$ and terminate with d$\\_$type = NC;\r\n\\ElsIf {\\ $\\|Hp^0\\|>U\\|p^0\\|\\ $}\r\n\\State Set $U\\leftarrow\\|Hp^0\\|/\\|p^0\\|$ and update $\\kappa,\\widehat{\\zeta},\\tau, T$ accordingly;\r\n\\EndIf\r\n\\While{TRUE}\r\n\\State $\\alpha_j\\leftarrow (r^j)^T r^j/(p^j)^T\\bar{H}p^j$; \\{Begin Standard CG Operations\\}\r\n\\State $y^{j+1}\\leftarrow y^j+\\alpha_jp^j$;\r\n\\State $r^{j+1}\\leftarrow r^j+\\alpha_j\\bar{H}p^j$;\r\n\\State $\\beta_{j+1}\\leftarrow\\|r^{j+1}\\|^2/\\|r^j\\|^2$;\r\n\\State $p^{j+1}\\leftarrow-r^{j+1}+\\beta_{j+1}p^j$; \\{End Standard CG Operations\\}\r\n\\State $j\\leftarrow j+1$;\r\n\\If {$\\|Hp^j\\|>U\\|p^j\\|$}\r\n\\State Set $U\\leftarrow\\|Hp^j\\|/\\|p^j\\|$ and update $\\kappa,\\widehat{\\zeta},\\tau,T$ accordingly;\r\n\\EndIf\r\n\\If {\\ $\\|Hy^j\\|>U\\|y^j\\|\\ $}\r\n\\State Set $U\\leftarrow\\|Hy^j\\|/\\|y^j\\|$ and update $\\kappa,\\widehat{\\zeta},\\tau,T$ accordingly;\r\n\\EndIf\r\n\\If {\\ $\\|Hr^j\\|>U\\|r^j\\|\\ $}\r\n\\State Set $U\\leftarrow\\|Hr^j\\|/\\|r^j\\|$ and update $\\kappa,\\widehat{\\zeta},\\tau,T$ accordingly;\r\n\\EndIf\r\n\\If {$(y^j)^T\\bar{H}y^j<\\varepsilon\\|y^j\\|^2$}\r\n\\State Set $d\\leftarrow y^j$ and terminate with d$\\_$type = NC;\r\n\\ElsIf {\\ $\\|r^j\\|\\le\\widehat{\\zeta}\\|r^0\\|$}\r\n\\State Set $d\\leftarrow y^j$ and terminate with d$\\_$type = SOL;\r\n\\ElsIf{\\ $(p^j)^T\\bar{H}p^j<\\varepsilon\\|p^j\\|^2$}\r\n\\State Set $d\\leftarrow p^j$ and terminate with d$\\_$type = NC;\r\n\\ElsIf {\\ $\\|r^j\\|>\\sqrt{T}\\tau^{j/2}\\|r^0\\| $}\r\n\\State Compute $\\alpha_j, y^{j+1}$ as in the main loop above;\r\n\\State Find $i\\in\\{0,\\ldots,j-1\\}$ such that\r\n\\[\r\n(y^{j+1}-y^i)^T\\bar{H}(y^{j+1}-y^i)<\\varepsilon\\|y^{j+1}-y^i\\|^2;\r\n\\]\r\n\\State Set $d\\leftarrow y^{j+1}-y^i$ and terminate with d$\\_$type = NC;\r\n\\EndIf\r\n\\EndWhile\r\n\\end{algorithmic}\r\n}\r\n\\end{algorithm}","sha256_utf8":"440b17ac5f36cdc0d2184335c209838c531ab42bcdba231d1deb0ff7069142bc"}]},{"id":"S020","semantic_type":"algorithm","kind":"algorithm","title":"随机朗佐斯特征值预言机","source_units":[{"start":169623,"end":170583,"start_line":1806,"end_line":1816,"raw_tex":"\\begin{algorithm}[h]\r\n\\caption{A randomized Lanczos based minimum eigenvalue oracle}\r\n\\label{pro:meo}\r\n\\noindent\\textit{Input}: symmetric matrix $H\\in\\bR^{n\\times n}$, tolerance $\\varepsilon>0$, and probability parameter $\\delta\\in(0,1)$.\\\\\r\n\\noindent\\textit{Output}: a sufficiently negative curvature direction $v$ satisfying $v^THv\\le-\\varepsilon/2$ and $\\|v\\|=1$; or a certificate that $\\lambda_{\\min}(H)\\ge-\\varepsilon$ with probability at least  $1-\\delta$.\\\\\r\nApply the Lanczos method \\cite{KW92LR} to estimate $\\lambda_{\\min}(H)$ starting with a random vector uniformly generated on the unit sphere, and run it for at most \r\n\\begin{equation}\\label{N-iter} \r\nN(\\varepsilon,\\delta):=\\min\\left\\{n,1+\\left\\lceil\\frac{\\ln(2.75n/\\delta^2)}{2}\\sqrt{\\frac{\\|H\\|}{\\varepsilon}}\\right\\rceil\\right\\}\r\n\\end{equation}\r\niterations. If a unit vector $v$ with $v^THv \\le -\\varepsilon/2$ is found at some iteration, terminate immediately and return $v$.\r\n\\end{algorithm}","sha256_utf8":"5e4b2120d167492b986ef3a45cb6ef9e754e3197c00b847c4632f0c175f64cff"}]},{"id":"S021","semantic_type":"theorem","kind":"theorem","title":"牛顿共轭梯度复杂度","source_units":[{"start":45742,"end":47805,"start_line":442,"end_line":472,"raw_tex":"\\begin{theorem}%[{{\\bf Complexity of Algorithm~\\ref{alg:NCG}}}]\r\n\\label{thm:NCG-iter-oper-cmplxity}\r\nSuppose that Assumption~\\ref{asp:NCG-cmplxity} holds. Let \r\n\\begin{equation}\r\nT_1:=\\left\\lceil\\frac{\\Fh-\\Fl}{\\min\\{c_{\\rm sol},c_{\\rm nc}\\}}\\max\\{\\epsilon_g^{-2}\\epsilon_H,\\epsilon_H^{-3}\\}\\right\\rceil+\\left\\lceil\\frac{\\Fh-\\Fl}{c_{\\rm nc}}\\epsilon_H^{-3}\\right\\rceil+1, \\\r\nT_2:=\\left\\lceil\\frac{\\Fh-\\Fl}{c_{\\rm nc}}\\epsilon_H^{-3}\\right\\rceil+1,\\label{T1}\r\n\\end{equation}\r\n%where $\\Fh=F(u^0)$, $\\Fl$ is given in \\eqref{lwbd-Hgupbd}, and $c_{\\rm sol}$ and $c_{\\rm nc}$ are defined in \\eqref{csol} and \\eqref{cnc}, respectively.\r\nwhere $\\Fh=F(u^0)$, $\\Fl$ is given in \\eqref{lwbd-Hgupbd}, and \r\n\\beqa\r\n&&c_{\\rm sol} :=\\eta\\min\\left\\{\\left[\\frac{4}{4+\\zeta+\\sqrt{(4+\\zeta)^2+8L^F_H}}\\right]^2,\\left[\\frac{\\min\\{6(1-\\eta),2\\}\\theta}{L^F_H}\\right]^2\\right\\}, \\label{csol} \\\\\r\n&& c_{\\rm nc} := \\frac{\\eta}{16} \\min\\left\\{1,\\left[\\frac{\\min\\{3(1-\\eta),1\\}\\theta}{L_H^F}\\right]^2\\right\\}.   \\label{cnc}\r\n\\eeqa\r\n Then the following statements hold.\r\n\\begin{enumerate}[{\\rm (i)}]\r\n\\item The total number of calls of Algorithm~\\ref{pro:meo} in Algorithm~\\ref{alg:NCG} is at most $T_2$.\t\r\n\\item The total number of calls of Algorithm~\\ref{alg:capped-CG} in Algorithm~\\ref{alg:NCG} is at most $T_1$.\r\n\\item {\\rm ({\\bf iteration complexity})} Algorithm~\\ref{alg:NCG} terminates in at most $T_1+T_2$ iterations with \r\n\\begin{equation}\\label{NCG-iter} \r\nT_1+T_2=\\cO((\\Fh-\\Fl)(L_H^F)^2\\max\\{\\epsilon_g^{-2}\\epsilon_H,\\epsilon_H^{-3}\\}).\r\n\\end{equation}\r\nAlso, its output $x^t$ satisfies $\\|\\nabla F(x^t)\\|\\le \\epsilon_g$ deterministically and\r\n$\\lambda_{\\min}(\\nabla^2F(x^t))$ $\\ge-\\epsilon_H$ with probability at least $1-\\delta$ for\r\nsome $0 \\le t \\le T_1+T_2$.\r\n\\item {\\rm ({\\bf operation complexity})} Algorithm~\\ref{alg:NCG} requires at most \r\n\\[\r\n\\widetilde{\\cO}((\\Fh-\\Fl)(L_H^F)^2\\max\\{\\epsilon_g^{-2}\\epsilon_H,\\epsilon_H^{-3}\\}\\min\\{n,(U_H^F/\\epsilon_H)^{1/2}\\})\r\n\\]\r\nmatrix-vector products, where $U_H^F$ is given in \\eqref{lwbd-Hgupbd}.\r\n\\end{enumerate}\r\n\\end{theorem}","sha256_utf8":"3ac7e03cb2167ac9459371655e8a7ce438fc71908c3747d210422e3226ad34fa"}]},{"id":"S022","semantic_type":"proof","kind":"proof","title":"牛顿共轭梯度复杂度证明","source_units":[{"start":107220,"end":111374,"start_line":1018,"end_line":1038,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{thm:NCG-iter-oper-cmplxity}]\r\nFor notational convenience, we let $\\{x^t\\}_{t\\in\\bT}$ denote all the iterates generated by Algorithm~\\ref{alg:NCG}, where $\\bT$ is a set of consecutive nonnegative integers starting from $0$. Notice that $F$ is descent along the iterates generated by Algorithm~\\ref{alg:NCG}, which together with $x^0=u^0$ implies that $x^t\\in\\{x:F(x)\\le F(u^0)\\}$. It then follows from\r\n\\eqref{lwbd-Hgupbd} that $\\|\\nabla^2 F(x^t)\\|\\le U^F_H$ holds for all $t\\in\\bT$.\r\n\r\n(i) Suppose for contradiction that the total number of calls of Algorithm~\\ref{pro:meo} in Algorithm~\\ref{alg:NCG} is more than $T_2$. Notice from Algorithm~\\ref{alg:NCG} and Lemma~\\ref{lem:nc}(ii) that each of these calls, except the last one, returns a sufficiently negative curvature direction, and each of them results in a reduction on $F$ of at least $c_{\\rm nc} \\epsilon_H^3$. Hence, \r\n\\[\r\nT_2 c_{\\rm nc} \\epsilon_H^3 \\le \\sum_{t\\in\\bT} [F(x^t)-F(x^{t+1})] \\le F(x^0)-\\Fl = \\Fh-\\Fl,\r\n\\]\r\nwhich contradicts the definition of $T_2$ given in \\eqref{T1}. Hence, statement (i) of Theorem \\ref{thm:NCG-iter-oper-cmplxity} holds.\r\n\r\n(ii) Suppose for contradiction that the total number of calls of Algorithm~\\ref{alg:capped-CG} in Algorithm~\\ref{alg:NCG} is more than $T_1$. Observe that if Algorithm~\\ref{alg:capped-CG} is called at some iteration $t$ and generates the next iterate $x^{t+1}$ satisfying $\\|\\nabla F(x^{t+1})\\|\\le\\epsilon_g$, then Algorithm~\\ref{pro:meo} must be called at the next iteration $t+1$. In view of this and statement (i) of Theorem \\ref{thm:NCG-iter-oper-cmplxity}, we see that the total number of such iterations $t$ is at most $T_2$. Hence, the total number of iterations $t$ of Algorithm~\\ref{alg:NCG} at which Algorithm~\\ref{alg:capped-CG} is called and generates the next iterate $x^{t+1}$ satisfying $\\|\\nabla F(x^{t+1})\\|>\\epsilon_g$ is at least $T_1-T_2+1$. Moreover, for each of such iterations $t$, we observe from Lemmas~\\ref{lem:sol}(ii) and \\ref{lem:nc}(ii) that $F(x^t)-F(x^{t+1})\\ge \\min\\{c_{\\rm sol},c_{\\rm nc}\\}\\min\\{\\epsilon_g^2\\epsilon_H^{-1},\\epsilon_H^3\\}$. It then follows that \r\n\\[\r\n(T_1-T_2+1) \\min\\{c_{\\rm sol},c_{\\rm nc}\\}\\min\\{\\epsilon_g^2\\epsilon_H^{-1},\\epsilon_H^3\\} \\le \\sum_{t\\in\\bT}[F(x^t)-F(x^{t+1})]\\le \\Fh-\\Fl,\r\n\\]\r\nwhich contradicts the definition of $T_1$ and $T_2$ given in \\eqref{T1}. Hence, statement (ii) of Theorem \\ref{thm:NCG-iter-oper-cmplxity} holds.\r\n\r\n(iii) Notice that either Algorithm~\\ref{alg:capped-CG} or \\ref{pro:meo} is called at each iteration of Algorithm~\\ref{alg:NCG}. It follows from this and statements (i) and (ii) of Theorem \\ref{thm:NCG-iter-oper-cmplxity} that the total number of iterations of Algorithm~\\ref{alg:NCG} is at most $T_1+T_2$. In addition, the relation \\eqref{NCG-iter} follows from \\eqref{csol}, \\eqref{cnc} and \\eqref{T1}. One can also observe that the output $x^t$ of Algorithm~\\ref{alg:NCG} satisfies $\\|\\nabla F(x^t)\\|\\le \\epsilon_g$ deterministically and $\\lambda_{\\min}(\\nabla^2F(x^t))\\ge-\\epsilon_H$ with probability at least $1-\\delta$ for some $0 \\le t \\le T_1+T_2$, where the latter part is due to\r\nAlgorithm~\\ref{pro:meo}. This completes the proof of statement (ii) of Theorem \\ref{thm:NCG-iter-oper-cmplxity}.\r\n\r\n(iv) By Theorem~\\ref{lem:capped-CG} with $(H,\\varepsilon)=(\\nabla^2 F(x^t),\\epsilon_H)$ and the fact that $\\|\\nabla^2 F(x^t)\\|\\le U^F_H$, one can observe that  the number of Hessian-vector products required by each call of Algorithm~\\ref{alg:capped-CG} with input $U = 0$ is at most $\\widetilde{\\cO}(\\min\\{n,(U_H^F/\\epsilon_H)^{1/2}\\})$. In addition, by Theorem~\\ref{rand-Lanczos} with $(H,\\varepsilon)=(\\nabla^2 F(x^t),\\epsilon_H)$, $\\|\\nabla^2 F(x^t)\\|\\le U^F_H$, and the fact that each iteration of the Lanczos method requires only one matrix-vector product, one can observe that the number of Hessian-vector products required by each call of Algorithm~\\ref{pro:meo} is also at most $\\widetilde{\\cO}(\\min\\{n,(U_H^F/\\epsilon_H)^{1/2}\\})$. Based on these observations and statement (iii) of Theorem \\ref{thm:NCG-iter-oper-cmplxity}, we see that  statement (iv) of this theorem holds.\r\n\\end{proof}","sha256_utf8":"f0b0db3ffa1c46596185333b0a964b61b0e8d729b68d7f1b1038ff735fbd5a0f"}]},{"id":"S023","semantic_type":"lemma","kind":"lemma","title":"共轭梯度方向性质","source_units":[{"start":90352,"end":91321,"start_line":828,"end_line":843,"raw_tex":"\\begin{lemma}\\label{lem:SOL-NC-ppty}\r\nSuppose that Assumption~\\ref{asp:NCG-cmplxity} holds and the direction $d^t$ results from  the output $d$ of Algorithm \\ref{alg:capped-CG} with a type specified in d$\\_$type  at some iteration $t$ of Algorithm \\ref{alg:NCG}. Then the following statements hold.\r\n\\begin{enumerate}[{\\rm (i)}]\r\n\\item If d$\\_$type=SOL, then  $d^t$ satisfies\r\n\\beqa\r\n&\\epsilon_H\\|d^t\\|^2\\le (d^t)^T\\left(\\nabla^2 F(x^t)+2\\epsilon_H I\\right)d^t,\\label{SOL-ppty-1}\\\\\r\n&\\|d^t\\|\\le1.1 \\epsilon_H^{-1}\\|\\nabla F(x^t)\\|,\\label{SOL-ppty-2}\\\\\r\n&(d^t)^T\\nabla F(x^t)=-(d^t)^T\\left(\\nabla^2F(x^t)+2\\epsilon_HI\\right)d^t,\\label{SOL-ppty-3}\\\\\r\n&\\|(\\nabla^2F(x^t)+2\\epsilon_HI)d^t+\\nabla F(x^t)\\|\\le\\epsilon_H\\zeta\\|d^t\\|/2.\\label{SOL-ppty-4}\r\n\\eeqa\r\n\\item If d$\\_$type=NC, then  $d^t$ satisfies $(d^t)^T\\nabla F(x^t)\\le0$ and\r\n\\begin{equation}\\label{NC-ppty}\r\n(d^t)^T\\nabla^2F(x^t)d^t/\\|d^t\\|^2=-\\|d^t\\|\\le-\\epsilon_H.\r\n\\end{equation}\r\n\\end{enumerate}\r\n\\end{lemma}","sha256_utf8":"836b543cd346b2b0722fffb46acd32a866b1acf9c964c19d3d7c2d952bf48afa"}]},{"id":"S024","semantic_type":"lemma","kind":"lemma","title":"近似牛顿步的充分下降","source_units":[{"start":91494,"end":92446,"start_line":847,"end_line":859,"raw_tex":"\\begin{lemma}\\label{lem:sol}\r\nSuppose that Assumption~\\ref{asp:NCG-cmplxity} holds and the direction $d^t$ results from  the output $d$ of Algorithm~\\ref{alg:capped-CG} with d$\\_$type=SOL at some iteration $t$ of Algorithm~\\ref{alg:NCG}. Let $U_g^F$ and $c_{\\rm sol}$ be given in \\eqref{lwbd-Hgupbd} and \\eqref{csol}, respectively.  Then the following statements hold.\r\n\\begin{enumerate}[{\\rm (i)}]\r\n\\item The step length $\\alpha_t$ is well-defined, and moreover,\r\n\\begin{equation}\\label{lwbd-ak-sol}\r\n\\alpha_t\\ge\\min\\left\\{1,\\sqrt{\\frac{\\min\\{6(1-\\eta),2\\}}{1.1 L_H^F U_g^F}}\\theta\\epsilon_H\\right\\}.\r\n\\end{equation}\r\n\\item The next iterate $x^{t+1}=x^t+\\alpha_td^t$ satisfies %$F(x^t)-F(x^{t+1})\\ge c_{\\rm sol}\\min\\{\\|\\nabla F(x^{t+1})\\|^2\\epsilon_H^{-1},\\epsilon_H^3\\}$.\r\n\\begin{equation}\\label{suff-desc-sol}\r\nF(x^t)-F(x^{t+1})\\ge c_{\\rm sol}\\min\\{\\|\\nabla F(x^{t+1})\\|^2\\epsilon_H^{-1},\\epsilon_H^3\\}.\r\n\\end{equation}\r\n\\end{enumerate}\r\n\\end{lemma}","sha256_utf8":"b3bcda6d6344a8b5f0d021f2ccb97298124b433ae6e67225fdc98f62bc26fbab"}]},{"id":"S025","semantic_type":"proof","kind":"proof","title":"近似牛顿步下降证明","source_units":[{"start":92450,"end":100325,"start_line":861,"end_line":945,"raw_tex":"\\begin{proof}\r\nOne can observe that $F$ is descent along the iterates (whenever well-defined) generated by Algorithm~\\ref{alg:NCG}, which together with $x^0=u^0$ implies that $F(x^t)\\le F(u^0)$ and hence\r\n$\\|\\nabla F(x^t)\\|\\le U^F_g$ due to \\eqref{lwbd-Hgupbd}.\r\n%Since $d^t$ { results from} the output $d$ of Algorithm~\\ref{alg:capped-CG} with d$\\_$type=SOL, it follows from Lemma~\\ref{lem:SOL-NC-ppty}(i) that \\eqref{SOL-ppty-1}-\\eqref{SOL-ppty-4} hold for $d^t$.\r\n%In particular, we see from \\eqref{SOL-ppty-2} and \\eqref{SOL-ppty-4} that $\\|(\\nabla^2F(x^t)+2\\epsilon_HI)d^t+\\nabla F(x^t)\\|\\le0.55\\zeta\\|\\nabla F(x^t)\\|$. Since we also have $\\|\\nabla F(x^t)\\| > \\epsilon_g$ in this case and recall that $ \\zeta \\in (0,1)$, we conclude that $d^t\\neq 0$.\r\nIn addition, since $d^t$ results from the output $d$ of Algorithm~\\ref{alg:capped-CG} with d$\\_$type=SOL, one can see that $\\|\\nabla F(x^t)\\| > \\epsilon_g$ and \\eqref{SOL-ppty-1}-\\eqref{SOL-ppty-4} hold for $d^t$. Moreover, by $\\|\\nabla F(x^t)\\| > \\epsilon_g$ and \\eqref{SOL-ppty-4}, one can conclude that $d^t\\neq 0$.\r\n\r\nWe first prove statement (i). If \\eqref{ls-sol} holds for $j=0$, then $\\alpha_t=1$, which clearly implies that \\eqref{lwbd-ak-sol} holds. We now suppose that \\eqref{ls-sol} fails for $j=0$. Claim that for all $j\\ge0$ that violate \\eqref{ls-sol}, it holds that\r\n\\begin{equation}\\label{lwbd-sol-ak}\r\n\\theta^{2j}\\ge\\min\\{6(1-\\eta),2\\}\\epsilon_H(L^F_H)^{-1}\\|d^t\\|^{-1}.\r\n\\end{equation}\r\nIndeed, suppose that \\eqref{ls-sol} is violated by some $j \\ge 0$. We now show that \\eqref{lwbd-sol-ak} holds for such $j$ by considering two separate cases below.\r\n\r\nCase 1) $F(x^t+\\theta^jd^t)>F(x^t)$. Let $\\phi(\\alpha)=F(x^t+\\alpha d^t)$. Then $\\phi(\\theta^j)>\\phi(0)$. Also, since $d^t\\neq0$, by \\eqref{SOL-ppty-1} and \\eqref{SOL-ppty-3}, one has \r\n\\[\r\n\\phi^\\prime(0)  =\\nabla F(x^t)^Td^t = -(d^t)^T(\\nabla^2 F(x^t)+2\\epsilon_HI)d^t \r\n \\le-\\epsilon_H\\|d^t\\|^2< 0.\r\n\\]\r\nUsing these, we can observe that there exists a local minimizer $\\alpha^*\\in(0,\\theta^j)$ of $\\phi$ such that $\\phi^\\prime(\\alpha^*)=\\nabla F(x^t+\\alpha^*d^t)^Td^t=0$ and $\\phi(\\alpha^*)<\\phi(0)$, which implies that $F(x^t+\\alpha^*d^t)< F(x^t)\\le F(u^0)$. Hence, \\eqref{apx-nxt-grad} holds for $x=x^t$ and $y=x^t+\\alpha^*d^t$. Using this, $0<\\alpha^*<\\theta^j\\le1$ and $\\nabla F(x^t+\\alpha^*d^t)^Td^t=0$, we obtain\r\n\\[\r\n\\begin{array}{l}\r\n\\frac{(\\alpha^*)^2L_H^F}{2}\\|d^t\\|^3\\overset{\\eqref{apx-nxt-grad}}{\\ge}\\|d^t\\|\\|\\nabla F(x^t+\\alpha^*d^t)-\\nabla F(x^t)-\\alpha^*\\nabla^2 F(x^t)d^t\\|\\\\[6pt]\r\n\\ge(d^t)^T(\\nabla F(x^t+\\alpha^*d^t)-\\nabla F(x^t)-\\alpha^*\\nabla^2 F(x^t)d^t)=-(d^t)^T\\nabla F(x^t)-\\alpha^*(d^t)^T\\nabla^2 F(x^t)d^t\\\\[6pt]\r\n\\overset{\\eqref{SOL-ppty-3}}{=}(1-\\alpha^*)(d^t)^T(\\nabla^2 F(x^t)+2\\epsilon_HI)d^t +2 \\alpha^*\\epsilon_H\\|d^t\\|^2\\overset{\\eqref{SOL-ppty-1}}{\\ge} (1+\\alpha^*)\\epsilon_H\\|d^t\\|^2\\ge \\epsilon_H \\|d^t\\|^2,\r\n\\end{array}\r\n\\]\r\nwhich along with $d^t\\neq 0$ implies that $(\\alpha^*)^2\\ge 2\\epsilon_H(L_H^F)^{-1}\\|d^t\\|^{-1}$. Using this and $\\theta^{j}>\\alpha^*$, we conclude that \\eqref{lwbd-sol-ak} holds in this case.\r\n\r\nCase 2) $F(x^t+\\theta^jd^t)\\le F(x^t)$. This together with $F(x^t)\\le F(u^0)$ implies that\r\n\\eqref{desc-ineq} holds for $x=x^t$ and $y=x^t+\\theta^jd^t$. Then, because $j$ violates \\eqref{ls-sol}, we obtain\r\n%By this and the fact that $j$ violates \\eqref{ls-sol}, we obtain that\r\n\\beqa\r\n&&-\\eta\\epsilon_H\\theta^{2j}\\|d^t\\|^2\\le F(x^t+\\theta^jd^t)-F(x^t)\\overset{\\eqref{desc-ineq}}{\\le} \\theta^j \\nabla F(x^t)^Td^t + \\frac{\\theta^{2j}}{2}(d^t)^T\\nabla^2 F(x^t) d^t + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3 \\nn \\\\\r\n&&\\overset{\\eqref{SOL-ppty-3}}{=}-\\theta^j (d^t)^T(\\nabla^2 F(x^t) +2\\epsilon_H I)d^t + \\frac{\\theta^{2j}}{2}(d^t)^T\\nabla^2 F(x^t) d^t + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3 \\nn \\\\\r\n&&=-\\theta^j\\left(1-\\frac{\\theta^j}{2}\\right)(d^t)^T(\\nabla^2 F(x^t) +2\\epsilon_H I)d^t - \\theta^{2j} \\epsilon_H\\|d^t\\|^2 + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3 \\nn \\\\\r\n&&\\overset{\\eqref{SOL-ppty-1}}{\\le} -\\theta^j\\left(1-\\frac{\\theta^j}{2}\\right) \\epsilon_H\\|d^t\\|^2 - \\theta^{2j}\\epsilon_H\\|d^t\\|^2 + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3 \\le -\\theta^j \\epsilon_H\\|d^t\\|^2 + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3. \\label{ineq:desc-ls-sol}\r\n\\eeqa\r\nRecall that $d^t \\neq 0$. Dividing both sides of \\eqref{ineq:desc-ls-sol} by $L_H^F\\theta^j\\|d^t\\|^3/6$ and using $\\eta, \\theta\\in(0,1)$, we obtain that \r\n\\[\r\n\\theta^{2j}\\ge 6(1-\\theta^j\\eta)\\epsilon_H(L_H^F)^{-1}\\|d^t\\|^{-1}\\ge6(1-\\eta)\\epsilon_H(L_H^F)^{-1}\\|d^t\\|^{-1}.\r\n\\] \r\n Hence, \\eqref{lwbd-sol-ak} also holds in this case.\r\n\r\nCombining the above two cases, we conclude that  \\eqref{lwbd-sol-ak} holds for any $j\\ge0$ that violates \\eqref{ls-sol}.\r\nBy this and $\\theta\\in(0,1)$, one can see that all $j\\ge0$ that violate \\eqref{ls-sol} must be bounded above. It then follows that the step length $\\alpha_t$ associated with \\eqref{ls-sol} is well-defined. We next prove \\eqref{lwbd-ak-sol}. Observe from the definition of $j_t$ in Algorithm~\\ref{alg:NCG} that $j=j_t-1$ violates \\eqref{ls-sol} and hence \\eqref{lwbd-sol-ak} holds for $j=j_t-1$. Then, by \\eqref{lwbd-sol-ak} with $j=j_t-1$ and $\\alpha_t = \\theta^{j_t}$, one has\r\n\\begin{equation}\\label{lwbd-sol-ak-1}\r\n\\alpha_t = \\theta^{j_t} \\ge\\sqrt{\\min\\{6(1-\\eta),2\\}\\epsilon_H(L^F_H)^{-1}} \\ \\theta\\|d^t\\|^{-1/2},\r\n\\end{equation}\r\nwhich,  along with \\eqref{SOL-ppty-2} and $\\|\\nabla F(x^t)\\|\\le U^F_g$, implies \\eqref{lwbd-ak-sol}. This proves statement (i).\r\n\r\nWe next prove statement (ii) %, particularly,  \\eqref{suff-desc-sol}\r\nby considering two separate cases below.\r\n\r\nCase 1) $\\alpha_t=1$. By this, one knows that \\eqref{ls-sol} holds for $j=0$. It then follows that $F(x^t+d^t)\\le F(x^t)\\le F(u^0)$, which implies that \\eqref{apx-nxt-grad} holds for $x=x^t$ and $y=x^t+d^t$. By this and \\eqref{SOL-ppty-4}, one has\r\n\\[\r\n\\begin{array}{lcl}\r\n%\\|\\nabla F(x^{t+1})\\|=\\|\\nabla F(x^{t}+d^t)-\\nabla F(x^t)+\\nabla F(x^t)\\|\\\\[4pt]\r\n\\|\\nabla F(x^{t+1})\\|=\\|\\nabla F(x^{t}+d^t)\\|\r\n&\\le&\\|\\nabla F(x^t+d^t)-\\nabla F(x^t)-\\nabla^2 F(x^t)d^t\\| \\\\[4pt]\r\n&&+\\|(\\nabla^2 F(x^t)+2\\epsilon_HI)d^t+\\nabla F(x^t)\\|+2\\epsilon_H\\|d^t\\|\\\\[4pt]\r\n&\\le& \\frac{L_H^F}{2}\\|d^t\\|^2 + \\frac{4+\\zeta}{2}\\epsilon_H\\|d^t\\|,\r\n\\end{array}\r\n\\]\r\nwhere the last inequality follows from \\eqref{apx-nxt-grad} and \\eqref{SOL-ppty-4}. Solving the above inequality for $\\|d^t\\|$ and using the fact that $\\|d^t\\|>0$, we obtain that\r\n\\[\r\n\\begin{array}{rcll}\r\n\\|d^t\\|&\\ge&\\frac{-(4+\\zeta)\\epsilon_H + \\sqrt{(4+\\zeta)^2\\epsilon_H^2+8L_H^F\\|\\nabla F(x^{t+1})\\|}}{2L_H^F}\\ \\ge\\\r\n\\frac{-(4+\\zeta)\\epsilon_H + \\sqrt{(4+\\zeta)^2\\epsilon_H^2+8L_H^F\\epsilon_H^2}}{2L_H^F}\\min\\{\\|\\nabla F(x^{t+1})\\|/\\epsilon_H^2,1\\}\\\\[6pt]\r\n&=&\\frac{4}{4+\\zeta+\\sqrt{(4+\\zeta)^2+8L_H^F}}\\min\\{\\|\\nabla F(x^{t+1})\\|/\\epsilon_H,\\epsilon_H\\},\r\n\\end{array}\r\n\\]\r\nwhere the second inequality follows from the inequality $-a+\\sqrt{a^2+bs}\\ge(-a+\\sqrt{a^2+b})\\min\\{s,1\\}$ for all $a,b,s\\ge0$,\r\n%which can be seen to hold by performing rationalization to the terms $-a+\\sqrt{a^2+b}$ and $-a+\\sqrt{a^2+bs}$.\r\nwhich can be verified by performing a rationalization to the terms $-a+\\sqrt{a^2+b}$ and $-a+\\sqrt{a^2+bs}$, respectively.\r\nBy this, $\\alpha_t=1$, \\eqref{ls-sol} and \\eqref{csol}, one can see that \\eqref{suff-desc-sol} holds.\r\n\r\nCase 2) $\\alpha_t<1$.\r\nIt then follows that $j=0$ violates \\eqref{ls-sol} and hence \\eqref{lwbd-sol-ak} holds for $j=0$.\r\nNow, letting $j=0$ in \\eqref{lwbd-sol-ak}, we obtain that $\\|d^t\\|\\ge \\min\\{6(1-\\eta),2\\}\\epsilon_H/L_H^F$,\r\n which together with \\eqref{ls-sol} and \\eqref{lwbd-sol-ak-1} implies that\r\n\\[\r\nF(x^t)-F(x^{t+1}) \\ge \\eta \\epsilon_H\\theta^{2j_t}\\|d^t\\|^2 \\ge \\eta\\frac{\\min\\{6(1-\\eta),2\\}\\epsilon_H^2}{L_H^F}\\theta^2\\|d^t\\|\\ge \\eta\\left[\\frac{\\min\\{6(1-\\eta),2\\}\\theta}{L_H^F}\\right]^2\\epsilon_H^3.\r\n\\]\r\nBy this and \\eqref{csol}, one can see that \\eqref{suff-desc-sol} also holds in this case.\r\n\\end{proof}","sha256_utf8":"b5aa356e6de839c0b8e9f7d4010b548d411ec9ee649ae87ad9b35e148bdabd75"}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","title":"负曲率步的充分下降","source_units":[{"start":100598,"end":101934,"start_line":950,"end_line":966,"raw_tex":"\\begin{lemma}\\label{lem:nc}\r\nSuppose that Assumption \\ref{asp:NCG-cmplxity} holds and the direction $d^t$ results from  either the output $d$ of Algorithm~\\ref{alg:capped-CG} with d$\\_$type=NC or the output $v$ of Algorithm~\\ref{pro:meo} at some iteration $t$ of Algorithm~\\ref{alg:NCG}.\r\n%\\begin{equation}\\label{cnc}\r\n%c_{\\rm nc} = \\frac{\\eta}{2} \\min\\left\\{1,\\left[\\frac{3(1-\\eta)\\theta}{L_H^F}\\right]^2\\right\\}.\r\n%\\end{equation}\r\nLet  $c_{\\rm nc}$ be defined in \\eqref{cnc}.\r\n%\\begin{equation}\\label{cnc}\r\n%c_{\\rm nc} := \\frac{\\eta}{2} \\min\\left\\{1,\\left[\\frac{\\min\\{3(1-\\eta),1\\}\\theta}{L_H^F}\\right]^2\\right\\}.\r\n%\\end{equation}\r\nThen the following statements hold.\r\n\\begin{enumerate}[{\\rm (i)}]\r\n\\item The step length $\\alpha_t$ is well-defined, and $\\alpha_t\\ge \\min\\{1,\\theta/L_H^F,3(1-\\eta)\\theta/L_H^F\\}$.\r\n%\\item If $d^t$ results from the output $d$ of Algorithm~\\ref{alg:capped-CG} with d$\\_$type=NC, then the next iterate $x^{t+1}=x^t+\\alpha_td^t$ satisfies $F(x^t)-F(x^{t+1})\\ge c_{\\rm nc}\\epsilon_H^3$.\r\n%\\item If $d^t$ { results from} the output $v$ of Algorithm~\\ref{pro:meo}, then the next iterate $x^{t+1}=x^t+\\alpha_td^t$ satisfies $F(x^t)-F(x^{t+1})\\ge c_{\\rm nc}\\epsilon_H^3/8$.\r\n\\item The next iterate $x^{t+1}=x^t+\\alpha_td^t$ satisfies $F(x^t)-F(x^{t+1})\\ge c_{\\rm nc}\\epsilon_H^3$.\r\n\\end{enumerate}\r\n\\end{lemma}","sha256_utf8":"05b954ac8c0eb148c391d61a3fc6fdf39619a3ecf6d8c40fc736e2c70a33aa69"}]},{"id":"S027","semantic_type":"proof","kind":"proof","title":"负曲率步下降证明","source_units":[{"start":101938,"end":107144,"start_line":968,"end_line":1014,"raw_tex":"\\begin{proof}\r\nObserve that $F$ is descent along the iterates (whenever well-defined) generated by Algorithm~\\ref{alg:NCG}. Using this and $x^0=u^0$, we have $F(x^t)\\le F(u^0)$.\r\nBy the assumption on $d^t$, one can see from Algorithm \\ref{alg:NCG} that\r\n%$d^t$ is given in \\eqref{dk-nc} or \\eqref{dk-2nd-nc}. Now, notice that the (negative curvature) direction $v$ returned from Algorithm~\\ref{pro:meo} satisfies $\\|v\\|=1$. This observation (for $d^t$ in \\eqref{dk-2nd-nc}) together with  Lemma~\\ref{lem:SOL-NC-ppty}(ii) (for $d^t$ in \\eqref{dk-nc}) shows that $d^t$ must satisfy\r\n$d^t$ is a negative curvature direction given in \\eqref{dk-nc} or \\eqref{dk-2nd-nc}. Also, notice that the vector $v$ returned from Algorithm~\\ref{pro:meo} satisfies $\\|v\\|=1$. By these, Lemma~\\ref{lem:SOL-NC-ppty}(ii), \\eqref{dk-nc} and \\eqref{dk-2nd-nc}, one can observe that\r\n\\begin{equation}\\label{dk-Hk-dk3}\r\n\\nabla F(x^t)^Td^t\\le 0, \\quad (d^t)^T\\nabla^2 F(x^t)d^t=-\\|d^t\\|^3<0.\r\n\\end{equation}\r\n\r\nWe first prove statement (i). If \\eqref{ls-nc} holds for $j=0$, then $\\alpha_t=1$, which clearly implies that $\\alpha_t\\ge\\min\\{1,\\theta/L_H^F,3(1-\\eta)\\theta/L_H^F\\}$. We now suppose that \\eqref{ls-nc} fails for $j=0$. Claim that for all $j\\ge0$ that violate \\eqref{ls-nc}, it holds that\r\n\\begin{equation}\\label{lwbd-at-nc}\r\n\\theta^j\\ge \\min\\{1/L_H^F,3(1-\\eta)/L_H^F\\}.\r\n\\end{equation}\r\nIndeed, suppose that \\eqref{ls-nc} is violated by some $j \\ge 0$. We now show that \\eqref{lwbd-at-nc} holds for such $j$ by considering two separate cases below.\r\n\r\nCase 1) $F(x^t+\\theta^jd^t)>F(x^t)$. Let $\\phi(\\alpha)=F(x^t+\\alpha d^t)$. Then $\\phi(\\theta^j)>\\phi(0)$. Also, by  \\eqref{dk-Hk-dk3}, one has \r\n\\[\r\n\\phi^\\prime(0)=\\nabla F(x^t)^Td^t\\le 0, \\quad \\phi^{\\prime\\prime}(0)=(d^t)^T\\nabla^2 F(x^t)d^t<0.\r\n\\]\r\n Using these, we can observe that there exists a local minimizer $\\alpha^*\\in(0,\\theta^j)$ of $\\phi$ such that $\\phi(\\alpha^*)<\\phi(0)$, namely,  $F(x^t+\\alpha^*d^t)< F(x^t)$. By the second-order optimality condition of $\\phi$ at $\\alpha^*$, one has $\\phi^{\\prime\\prime}(\\alpha^*)=(d^t)^T\\nabla^2 F(x^t+\\alpha^*d^t)d^t\\ge 0$. Since $F(x^t+\\alpha^*d^t)<F(x^t)\\le F(u^0)$, it follows that \\eqref{F-Hess-Lip} holds for $x=x^t$ and $y=x^t+\\alpha^*d^t$. Using this, the second relation in \\eqref{dk-Hk-dk3} and $(d^t)^T\\nabla^2 F(x^t+\\alpha^*d^t)d^t\\ge0$, we obtain that\r\nin \\eqref{dk-Hk-dk3} and $(d^t)^T\\nabla^2 F(x^t+\\alpha^*d^t)d^t\\ge0$, we obtain that\r\n\\beqa\r\nL_H^F\\alpha^*\\|d^t\\|^3 &\\overset{\\eqref{F-Hess-Lip}}{\\ge}&\\|d^t\\|^2\\|\\nabla^2 F(x^t+\\alpha^*d^t)-\\nabla^2 F(x^t)\\| \\ge (d^t)^T(\\nabla^2 F(x^t+\\alpha^*d^t)-\\nabla^2 F(x^t))d^t \\nn \\\\\r\n&\\ge& -(d^t)^T\\nabla^2 F(x^t)d^t=\\|d^t\\|^3. \\label{lwbd-dt-nc-j1}\r\n\\eeqa\r\nRecall from \\eqref{dk-Hk-dk3} that $d^t\\neq 0$. It then follows from \\eqref{lwbd-dt-nc-j1} that $\\alpha^*\\ge 1/L_H^F$, which along with $\\theta^j>\\alpha^*$ implies that $ \\theta^j> 1/L_H^F$. Hence, \\eqref{lwbd-at-nc} holds in this case.\r\n\r\nCase 2) $F(x^t+\\theta^jd^t)\\le F(x^t)$. It follows from this and $F(x^t)\\le F(u^0)$ that \\eqref{desc-ineq} holds for $x=x^t$ and $y=x^t+\\theta^jd^t$. By this and the fact that $j$ violates \\eqref{ls-nc}, one has\r\n\\begin{equation*}\\label{lwbd-dt-nc}\r\n\\begin{array}{lcl}\r\n-\\frac{\\eta}{2}\\theta^{2j}\\|d^t\\|^3 &\\le& F(x^t+\\theta^j d^t) - F(x^t) \\overset{\\eqref{desc-ineq}}{\\le} \\theta^j \\nabla F(x^t)^Td^t + \\frac{\\theta^{2j}}{2}(d^t)^T\\nabla^2 F(x^t) d^t + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3\\\\[6pt]\r\n&\\overset{\\eqref{dk-Hk-dk3}}{\\le}& -\\frac{\\theta^{2j}}{2}\\|d^t\\|^3 + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3,\r\n\\end{array}\r\n\\end{equation*}\r\nwhich together with $d^t\\neq 0$ implies that $\\theta^j\\ge 3(1-\\eta)/L_H^F$. Hence, \\eqref{lwbd-at-nc} also holds  in this case.\r\n\r\nCombining the above two cases, we conclude that \\eqref{lwbd-at-nc} holds for any $j\\ge0$ that violates \\eqref{ls-nc}. By this and $\\theta\\in(0,1)$, one can see that all $j\\ge0$ that violate \\eqref{ls-nc} must be bounded above. It then follows that the step length $\\alpha_t$ associated with \\eqref{ls-nc} is well-defined. We next derive a lower bound for $\\alpha_t$. Notice from the definition of $j_t$ in Algorithm~\\ref{alg:NCG} that $j=j_t-1$ violates \\eqref{ls-nc} and hence \\eqref{lwbd-at-nc} holds for $j=j_t-1$. Then, by \\eqref{lwbd-at-nc} with $j=j_t-1$ and $\\alpha_t = \\theta^{j_t}$, one has $\\alpha_t=\\theta^{j_t} \\ge \\min\\{\\theta/L_H^F,3(1-\\eta)\\theta/L_H^F\\}$, which immediately yields $\\alpha_t\\ge \\min\\{1,\\theta/L_H^F,3(1-\\eta)\\theta/L_H^F\\}$ as desired.\r\n\r\nWe next prove statement (ii) by considering two separate cases below.\r\n\r\nCase 1) $d^t$ results from the output $d$ of Algorithm~\\ref{alg:capped-CG} with d$\\_$type=NC. It then follows from  \\eqref{NC-ppty} that $\\|d^t\\|\\ge\\epsilon_H$. \r\n%{ This together with item (i) and \\eqref{ls-nc} implies (ii).}\r\nThis together with  \\eqref{ls-nc} and statement (i) implies that statement (ii) holds.\r\n\r\nCase 2) $d^t$ results from the output $v$ of Algorithm~\\ref{pro:meo}. Notice from\r\nAlgorithm~\\ref{pro:meo} that $\\|v\\|=1$ and $v^T\\nabla^2F(x^t)v\\le-\\epsilon_H/2$, which along with \\eqref{dk-2nd-nc} yields $\\|d^t\\|\\ge\\epsilon_H/2$. By this, \\eqref{ls-nc} and statement (i), one can see that statement (ii) again holds.\r\n\\end{proof}","sha256_utf8":"e7cee1969989e8053db0b77d15fa1dc3bd293f7b10c0f6e39315550d78afb293"}]},{"id":"S028","semantic_type":"lemma","kind":"lemma","title":"增广函数水平集与下界","source_units":[{"start":61144,"end":62424,"start_line":581,"end_line":592,"raw_tex":"\\begin{lemma}%[{{\\bf level- and lower-boundedness of $\\widetilde{\\cL}(\\cdot,\\lambda^k;\\rho_k)$}}]\r\n\\label{lem:level-set-augmented-lagrangian-func}\r\nSuppose that Assumption~\\ref{asp:lowbd-knownfeas} holds. Let $(\\lambda^k, \\rho_k)$ be generated at the $k$th iteration of Algorithm~\\ref{alg:2nd-order-AL-nonconvex} for some $k \\ge 0$, and $\\cS(\\bdelta_f,\\bdelta_c)$ and $x_{\\init}^k$ be defined in \\eqref{nearly-feas-level-set} and \\eqref{def:initial-iterate-subprob}, respectively, and let $\\fh$, $\\fl$, $\\bdelta_f$ and $\\bdelta_c$ be given in Assumption~\\ref{asp:lowbd-knownfeas}. Suppose that $\\rho_0$ is sufficiently large such that $\\delta_{f,1} \\le \\bdelta_f$ and  $\\delta_{c,1} \\le \\bdelta_c$, where \r\n\\begin{equation} \r\n\\delta_{f,1}:=\\Lambda^2/(2\\rho_0)\\quad and\\quad \\delta_{c,1}:=\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_0-2\\gamma}+\\frac{\\Lambda^2}{(\\rho_0-2\\gamma)^2}}+\\frac{\\Lambda}{\\rho_0-2\\gamma}\\label{def:delta0c-rhobar1xxx}.\r\n\\end{equation}\r\nThen the following statements hold.\r\n\\begin{enumerate}[{\\rm (i)}]\r\n\\item $\\{x:\\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\}\\subseteq \\cS(\\bdelta_f,\\bdelta_c)$.\r\n\\item $\\inf_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k) \\ge\\fl - \\gamma-\\Lambda\\bdelta_c$.\r\n\\end{enumerate}\r\n\\end{lemma}","sha256_utf8":"2e08f3f548c1c40bacea5166473a286bca3fb20064023e573095fa50a2e5eebf"}]},{"id":"S029","semantic_type":"proof","kind":"proof","title":"水平集与下界完整证明","source_units":[{"start":115361,"end":118533,"start_line":1113,"end_line":1139,"raw_tex":"\\begin{proof}[Proof of Lemma \\ref{lem:level-set-augmented-lagrangian-func}]\r\n(i) Let $x$ be any point such that $\\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)$. It then follows from \\eqref{L-xinit} that $\\widetilde{\\cL}(x,\\lambda^k;\\rho_k) \\le \\fh$. By this, $\\|\\lambda^k\\|\\le\\Lambda$, $\\rho_k \\ge \\rho_0>2\\gamma$, $\\delta_{f,1} \\le \\bdelta_f$, $\\delta_{c,1} \\le \\bdelta_c$, and Lemma \\ref{tech-1} with $(\\lambda,\\rho)=(\\lambda^k,\\rho_k)$,  one has \r\n\\begin{align}\r\nf(x) & \\le \\fh+ \\|\\lambda^k\\|^2/(2\\rho_k)\\le \\fh+ \\Lambda^2/(2\\rho_0)=\\fh+\\delta_{f,1} \\le \\fh+\\bdelta_f, \\nonumber \\\\\r\n\\textstyle\\|\\tilde{c}(x)\\|&\\textstyle\\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_k-2\\gamma}+\\frac{\\|\\lambda^k\\|^2}{(\\rho_k-2\\gamma)^2}}+\\frac{\\|\\lambda^k\\|}{\\rho_k-2\\gamma}\\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_0-2\\gamma}+\\frac{\\Lambda^2}{(\\rho_0-2\\gamma)^2}}+\\frac{\\Lambda}{\\rho_0-2\\gamma}=\\delta_{c,1} \\le \\bdelta_c. \\label{ineq:feasible-bd}\r\n\\end{align}\r\n%\\begin{eqnarray}\r\n%&f(x)\\le \\fh+\\frac{\\|\\lambda^k\\|^2}{2\\rho_k}\\le \\fh+\\frac{\\Lambda^2}{2\\rho_0}=\\fh+\\delta_{f,1} \\le \\fh+\\bdelta_f,\\nn \\\\\r\n%&\\|\\tilde{c}(x)\\|\\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_k-2\\gamma}+\\frac{\\|\\lambda^k\\|^2}{(\\rho_k-2\\gamma)^2}}+\\frac{\\|\\lambda^k\\|}{\\rho_k-2\\gamma}\\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_0-2\\gamma}+\\frac{\\Lambda^2}{(\\rho_0-2\\gamma)^2}}+\\frac{\\Lambda}{\\rho_0-2\\gamma}=\\delta_{c,1} \\le \\bdelta_c.\\label{ineq:feasible-bd}\r\n%\\end{eqnarray}\r\nAlso, recall from the definition of $\\tilde{c}$ in \\eqref{model:equa-cnstr-pert}\r\nand $\\|c(z_{\\epsilon_1})\\|\\le1$ that $\\|c(x)\\|\\le1+\\|\\tilde{c}(x)\\|$. This together with the above inequalities and \\eqref{nearly-feas-level-set} implies $x\\in\\cS(\\delta_{f},\\delta_{c})$. Hence, statement (i) of Lemma \\ref{lem:level-set-augmented-lagrangian-func} holds.\r\n\r\n(ii) Note that $\\inf\\limits_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)=\\inf\\limits_{x\\in\\bR^n}\\{\\widetilde{\\cL}(x,\\lambda^k;\\rho_k):\\widetilde{\\cL}(x,\\lambda^k;\\rho_k) \\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\}$. Consequently, to prove statement (ii) of Lemma \\ref{lem:level-set-augmented-lagrangian-func}, it suffices to show that\r\n\\begin{equation} \\label{L-lwbd}\r\n\\inf\\limits_{x\\in\\bR^n}\\{\\widetilde{\\cL}(x,\\lambda^k;\\rho_k): \\widetilde{\\cL}(x,\\lambda^k;\\rho_k) \\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\} \\ge\\fl - \\gamma-\\Lambda\\bdelta_c.\r\n\\end{equation}\r\nTo this end, let $x$ be any point satisfying $\\widetilde{\\cL}(x,\\lambda^k;\\rho_k) \\le\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)$. We then know from \\eqref{ineq:feasible-bd} that $\\|\\tilde{c}(x)\\|\\le\\bdelta_c$. By this, $\\|\\lambda^k\\|\\le\\Lambda$, $\\rho_k>2\\gamma$, and \\eqref{ineq:p-lowbd}, one has\r\n\\begin{equation*}\\label{ineq:lowerbound-ALfunc} \r\n\\begin{array}{l}\r\n\\widetilde{\\cL}(x,\\lambda^k;\\rho_k) %= f(x)+(\\lambda^k)^T\\tilde{c}(x)+\\frac{\\rho_k}{2}\\|\\tilde{c}(x)\\|^2\\\\[4pt]\r\n=f(x)+\\gamma\\|\\tilde{c}(x)\\|^2+(\\lambda^k)^T\\tilde{c}(x)+\\frac{\\rho_k-2\\gamma}{2}\\|\\tilde{c}(x)\\|^2\\\\[4pt]\r\n\\ge f(x)+\\gamma\\|\\tilde{c}(x)\\|^2 - \\Lambda\\|\\tilde{c}(x)\\| \\ge\\fl - \\gamma-\\Lambda\\bdelta_c,\r\n\\end{array}\r\n\\end{equation*}\r\nand hence \\eqref{L-lwbd} holds as desired.\r\n\\end{proof}","sha256_utf8":"c7ae0aac3e79e04df28f352bf6d9554a9ea40e058b0c3e5a3260a53c69263d9a"}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","title":"函数值与约束违反界","source_units":[{"start":112184,"end":113353,"start_line":1053,"end_line":1075,"raw_tex":"\\begin{lemma}\\label{tech-1}\r\nSuppose that Assumption \\ref{asp:lowbd-knownfeas} holds. Let $\\gamma$, $\\fh$ and $\\fl$ be given in Assumption \\ref{asp:lowbd-knownfeas}.  Assume that $\\rho>2\\gamma$, $\\lambda\\in\\bR^m$, and $x\\in\\bR^n$ satisfy\r\n\\begin{equation}\\label{AL-upperbound}\r\n\\widetilde{\\cL}(x,\\lambda;\\rho)\\le \\fh,\r\n\\end{equation}\r\n%where $\\fh=f(x^{\\feas})$.\r\n%where $\\gamma$ and $\\fh$ are given in Assumption \\ref{asp:lowbd-knownfeas}.\r\nwhere $\\widetilde{\\cL}$ is defined in \\eqref{tL-tc}.\r\nThen the following statements hold.\r\n\\begin{enumerate}[{\\rm (i)}]\r\n\\item $f(x)\\le \\fh+\\|\\lambda\\|^2/(2\\rho)$.\r\n\\item $\\|\\tilde{c}(x)\\|\\le\\sqrt{2(\\fh-\\fl+\\gamma)/(\\rho-2\\gamma)+\\|\\lambda\\|^2/(\\rho-2\\gamma)^2}+\\|\\lambda\\|/(\\rho-2\\gamma)$.\r\n\\item If $\\rho\\ge\\|\\lambda\\|^2/(2\\tdf)$ for some $\\tdf>0$, then $f(x)\\le \\fh+\\tdf$.\r\n\\item If\r\n%\\begin{equation}\\label{bound-rho-deltac}\r\n%\\rho\\ge\\frac{2(\\fh-\\fl)}{\\delta_c^2}+\\frac{2\\|\\lambda\\|}{\\delta_c}+\\gamma\r\n%\\end{equation}\r\n\\begin{equation}\\label{bound-rho-deltac}\r\n\\rho\\ge 2(\\fh-\\fl+\\gamma)\\tdc^{-2}+ 2\\|\\lambda\\|\\tdc^{-1}+2\\gamma\r\n\\end{equation}\r\nfor some $\\tdc>0$, then $\\|\\tilde{c}(x)\\|\\le\\tdc$.\r\n\\end{enumerate}\r\n\\end{lemma}","sha256_utf8":"3b3a1dc4358f5989bbcae5820bb9800459a59ab2267aeac5a0640b71f8d272dd"}]},{"id":"S031","semantic_type":"proof","kind":"proof","title":"增广函数技术界证明","source_units":[{"start":113358,"end":115274,"start_line":1077,"end_line":1109,"raw_tex":"\\begin{proof}\r\n(i) It follows from\r\n%{ the definition of { $\\widetilde{\\cL}$} in} \\eqref{tL-tc} and \\eqref{AL-upperbound}\r\n \\eqref{AL-upperbound} and the definition of $\\widetilde{\\cL}$ in \\eqref{tL-tc}\r\nthat\r\n\\[\r\n\\textstyle\r\n\\fh\\ge f(x)+\\lambda^T \\tilde{c}(x)+\\frac{\\rho}{2}\\|\\tilde{c}(x)\\|^2 = f(x)+\\frac{\\rho}{2}\\left\\|\\tilde{c}(x)+\\frac{\\lambda}{\\rho}\\right\\|^2-\\frac{\\|\\lambda\\|^2}{2\\rho} \\ge f(x)-\\frac{\\|\\lambda\\|^2}{2\\rho}.\r\n\\]\r\nHence, statement (i) holds.\r\n\r\n(ii) In view of \\eqref{ineq:p-lowbd} and \\eqref{AL-upperbound}, one has\r\n\\begin{equation*}\\label{ineq:bound-cx-step1}\r\n\\begin{array}{l}\r\n\\fh\\overset{\\eqref{AL-upperbound}}{\\ge}f(x)+\\lambda^T\\tilde{c}(x)+\\frac{\\rho}{2}\\|\\tilde{c}(x)\\|^2=f(x)+\\gamma\\|\\tilde{c}(x)\\|^2\r\n+\\frac{\\rho-2\\gamma}{2}\\left\\|\\tilde{c}(x)+\\frac{\\lambda}{\\rho-2\\gamma}\\right\\|^2-\\frac{\\|\\lambda\\|^2}{2(\\rho-2\\gamma)}\\\\[2pt]\r\n\\ \\ \\ \\ \\overset{\\eqref{ineq:p-lowbd}}{\\ge}\\fl- \\gamma+\\frac{\\rho-2\\gamma}{2}\\left\\|\\tilde{c}(x)+\\frac{\\lambda}{\\rho-2\\gamma}\\right\\|^2-\\frac{\\|\\lambda\\|^2}{2(\\rho-2\\gamma)}.\r\n\\end{array}\r\n\\end{equation*}\r\nIt then follows that $\\left\\|\\tilde{c}(x)+\\frac{\\lambda}{\\rho-2\\gamma}\\right\\|\\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho-2\\gamma}+\\frac{\\|\\lambda\\|^2}{(\\rho-2\\gamma)^2}}$,  which implies that statement (ii) holds.\r\n\r\n(iii) Statement (iii) immediately follows from statement (i) and $\\rho\\ge\\|\\lambda\\|^2/(2\\tdf)$.\r\n\r\n(iv) Suppose that \\eqref{bound-rho-deltac} holds. Multiplying both sides of \\eqref{bound-rho-deltac} by $\\tdc^{2}$ and rearranging the terms, we have\r\n\\[\r\n(\\rho-2\\gamma)\\tdc^2-2\\|\\lambda\\|\\tdc-2(\\fh-\\fl+\\gamma)\\ge0.\r\n\\]\r\nRecall that $\\rho>2\\gamma$ and $\\tdc>0$. Solving this inequality for $\\tdc$ yields \r\n\\[\r\n\\tdc\\ge\\sqrt{2(\\fh-\\fl+\\gamma)/(\\rho-2\\gamma)+\\|\\lambda\\|^2/(\\rho-2\\gamma)^2}+\\|\\lambda\\|/(\\rho-2\\gamma),\r\n\\]\r\nwhich along with statement (ii) implies that $\\|\\tilde{c}(x)\\|\\le\\tdc$. Hence, statement (iv) holds.\r\n\\end{proof}","sha256_utf8":"0e2b45498deacd339f364c14bc529ad2421cafeb8eadfd6248b7f691d7c4f7c3"}]},{"id":"S032","semantic_type":"theorem","kind":"theorem","title":"外层终止与罚参数界","source_units":[{"start":70335,"end":71674,"start_line":652,"end_line":665,"raw_tex":"\\begin{theorem}%[{{\\bf Outer iteration complexity of Algorithm \\ref{alg:2nd-order-AL-nonconvex}}}]\r\n\\label{thm:out-itr-cmplxity-1}\r\nSuppose that Assumption~\\ref{asp:lowbd-knownfeas} holds and that $\\rho_0$ is sufficiently large such that $\\delta_{f,1} \\le \\bdelta_f$ and  $\\delta_{c,1} \\le \\bdelta_c$, where\r\n$\\delta_{f,1}$ and $\\delta_{c,1}$ are defined in \\eqref{def:delta0c-rhobar1xxx}. Let\r\n\\begin{eqnarray}\r\n&&{\\rho}_{\\epsilon_1}:=\\max\\left\\{8(\\fh-\\fl+\\gamma)\\epsilon_1^{-2}+4\\Lambda\\epsilon_1^{-1}+2\\gamma,2\\rho_0\\right\\},\\label{def:delta0c-rhobar1}\\\\\r\n&&\\overline{K}_{\\epsilon_1}:=\\inf\\{k\\ge K_{\\epsilon_1}: \\|c(x^{k+1})\\|\\le\\epsilon_1\\},\\label{number-outer-iteration}\r\n\\end{eqnarray}\r\nwhere $K_{\\epsilon_1}$ is defined in \\eqref{T-epsilon-g},  and $\\gamma$, $\\fh$ and $\\fl$ are given in Assumption \\ref{asp:lowbd-knownfeas}. Then $\\overline{K}_{\\epsilon_1}$ is finite, and Algorithm \\ref{alg:2nd-order-AL-nonconvex} terminates at iteration $\\overline{K}_{\\epsilon_1}$ with\r\n\\begin{equation}\\label{outer-iteration-cmplxty}\r\n\\overline{K}_{\\epsilon_1}\\le\\left(\\frac{\\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})}{\\log{r}}+1\\right)\\left(\\left|\\frac{\\log(\\epsilon_1(2\\delta_{c,1})^{-1})}{\\log \\alpha}\\right|+2\\right)+1.\r\n\\end{equation}\r\nMoreover, $\\rho_k\\le{r}{\\rho}_{\\epsilon_1}$ holds for $0 \\le k \\le \\overline{K}_{\\epsilon_1}$\r\n\\end{theorem}","sha256_utf8":"bff7cab74fae8a2400d41fb0c8c88c69f3fb0758a2e7717546a7e474aea613dd"}]},{"id":"S033","semantic_type":"proof","kind":"proof","title":"外层复杂度完整证明","source_units":[{"start":121051,"end":129248,"start_line":1159,"end_line":1236,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{thm:out-itr-cmplxity-1}]\r\nIt follows from \\eqref{def:delta0c-rhobar1} that ${\\rho}_{\\epsilon_1}\\ge 2\\rho_0$.\r\nBy this, one has\r\n\\begin{equation}\\label{T-epsilong}\r\nK_{\\epsilon_1} \\ \\overset{\\eqref{T-epsilon-g}}{=} \\ \\left\\lceil\\log\\epsilon_1/\\log \\omega_1 \\right\\rceil \\ \\overset{\\eqref{omega-tolerance}}{=} \\ \\left\\lceil\\log{2}/\\log{r}\\right\\rceil  \\\r\n\\le \\ \\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})/\\log{r}+1.\r\n\\end{equation}\r\nNotice that $\\{\\rho_k\\}$ is either unchanged or increased by a ratio $r$ as $k$ increases.\r\nBy this fact and \\eqref{T-epsilong}, we see that\r\n\\begin{equation}\\label{rho:upper-bound-Tepsilong}\r\n\\max_{0 \\le k \\le K_{\\epsilon_1}} \\rho_k \\le{r}^{K_{\\epsilon_1}}\\rho_0\\overset{\\eqref{T-epsilong}}{\\le}{r}^{\\frac{\\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})}{\\log{r}}+1}\\rho_0={r}{\\rho}_{\\epsilon_1}.\r\n\\end{equation}\r\nIn addition, notice that $\\rho_k>2\\gamma$ and $\\|\\lambda^k\\|\\le \\Lambda$. Using these, \\eqref{hbd}, the first relation in \\eqref{algstop:1st-order}, and Lemma \\ref{tech-1}(ii) with $(x,\\lambda,\\rho)=(x^{k+1},\\lambda^k,\\rho_k)$,  we obtain that\r\n\\begin{equation}\\label{bd-ck}\\textstyle\r\n\\|\\tilde{c}(x^{k+1})\\| \\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_k-2\\gamma}+\\frac{\\|\\lambda^k\\|^2}{(\\rho_k-2\\gamma)^2}}+\\frac{\\|\\lambda^k\\|}{\\rho_k-2\\gamma}\\le  \\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_k-2\\gamma}+\\frac{\\Lambda^2}{(\\rho_k-2\\gamma)^2}}+\\frac{\\Lambda}{\\rho_k-2\\gamma}.\r\n\\end{equation}\r\nAlso, we observe from $\\|c(z_{\\epsilon_1})\\|\\le\\epsilon_1/2$ and the definition of $\\tilde{c}$ in\r\n\\eqref{model:equa-cnstr-pert} that\r\n\\begin{equation}\\label{bd-cxk}\r\n\\|c(x^{k+1})\\|\\le\\|\\tilde{c}(x^{k+1})\\|+\\|c(z_{\\epsilon_1})\\|\\le \\|\\tilde{c}(x^{k+1})\\|+\\epsilon_1/2.\r\n\\end{equation}\r\n\r\nWe now prove that $\\overline{K}_{\\epsilon_1}$ is finite. Suppose for contradiction that $\\overline{K}_{\\epsilon_1}$ is infinite. It then follows from this and \\eqref{number-outer-iteration} that $\\|c(x^{k+1})\\|>\\epsilon_1$ for all $k\\ge K_{\\epsilon_1}$, which along with \\eqref{bd-cxk} implies that $\\|\\tilde{c}(x^{k+1})\\|>\\epsilon_1/2$ for all $k\\ge K_{\\epsilon_1}$. It then follows that $\\|\\tilde{c}(x^{k+1})\\|>\\alpha\\|\\tilde{c}(x^k)\\|$ must hold for infinitely many $k$'s. Using this and the update scheme on $\\{\\rho_k\\}$, we deduce that $\\rho_{k+1}={r}\\rho_k$ holds for infinitely many $k$'s, which together with the monotonicity of $\\{\\rho_k\\}$ implies that\r\n$\\rho_k\\to\\infty$ as $k\\to\\infty$. By this and \\eqref{bd-ck}, one can see that  $\\|\\tilde{c}(x^{k+1})\\|\\to0$ as $k\\to\\infty$, which contradicts the fact that $\\|\\tilde{c}(x^{k+1})\\|>\\epsilon_1/2$ holds for all $k\\ge K_{\\epsilon_1}$. Hence, $\\overline{K}_{\\epsilon_1}$ is finite.\r\nIn addition, notice from \\eqref{omega-tolerance}, \\eqref{T-epsilon-g} and \\eqref{epsw1-epsw2} that $(\\tau_k^g,\\tau_k^H)=(\\epsilon_1,\\epsilon_2)$ for all $k\\ge K_{\\epsilon_1}$. This along with the termination criterion of Algorithm \\ref{alg:2nd-order-AL-nonconvex}  and the definition of $\\overline{K}_{\\epsilon_1}$ implies that Algorithm \\ref{alg:2nd-order-AL-nonconvex} must terminate at iteration $\\overline{K}_{\\epsilon_1}$.\r\n\r\nWe next show that \\eqref{outer-iteration-cmplxty} and $\\rho_k\\le{r}{\\rho}_{\\epsilon_1}$ hold for $0 \\le k \\le \\overline{K}_{\\epsilon_1}$ by considering two separate cases below.\r\n\r\nCase 1) $\\|c(x^{K_{\\epsilon_1}+1})\\|\\le\\epsilon_1$.  By this and \\eqref{number-outer-iteration}, one can see that  $\\overline{K}_{\\epsilon_1}=K_{\\epsilon_1}$, which together with \\eqref{T-epsilong} and \\eqref{rho:upper-bound-Tepsilong} implies that \\eqref{outer-iteration-cmplxty} and $\\rho_k\\le{r}{\\rho}_{\\epsilon_1}$ hold for $0 \\le k \\le \\overline{K}_{\\epsilon_1}$.\r\n\r\nCase 2) $\\|c(x^{K_{\\epsilon_1}+1})\\|>\\epsilon_1$. By this and \\eqref{number-outer-iteration}, one can observe that $\\overline{K}_{\\epsilon_1}>K_{\\epsilon_1}$  and also $\\|c(x^{k+1})\\|>\\epsilon_1$ for all $K_{\\epsilon_1} \\le k \\le \\overline{K}_{\\epsilon_1}-1$, which together with \\eqref{bd-cxk} implies\r\n\\begin{equation}\\label{lbd-tc}\r\n\\|\\tilde{c}(x^{k+1})\\|>\\epsilon_1/2,\\quad \\forall K_{\\epsilon_1} \\le k \\le \\overline{K}_{\\epsilon_1}-1.\r\n\\end{equation}\r\nIt then follows from $\\|\\lambda^k\\|\\le\\Lambda$, \\eqref{hbd}, the first relation in \\eqref{algstop:1st-order}, and Lemma \\ref{tech-1}(iv) with $(x,\\lambda,\\rho,\\tdc)=(x^{k+1},\\lambda^k,\\rho_k,\\epsilon_1/2)$ that\r\n\\begin{equation}\\label{bound-rhok}\r\n\\begin{aligned}\r\n\\rho_k &< 8(\\fh-\\fl+\\gamma)\\epsilon_1^{-2}+4\\|\\lambda^k\\|\\epsilon_1^{-1}+2\\gamma\\\\\r\n&\\le 8(\\fh-\\fl+\\gamma)\\epsilon_1^{-2}+4\\Lambda\\epsilon_1^{-1}+2\\gamma\\overset{\\eqref{def:delta0c-rhobar1}}{\\le}{\\rho}_{\\epsilon_1},\\quad\r\n\\forall K_{\\epsilon_1} \\le k \\le \\overline{K}_{\\epsilon_1}-1.\r\n\\end{aligned}\r\n\\end{equation}\r\nCombining this relation, \\eqref{rho:upper-bound-Tepsilong}, and the fact $\\rho_{\\overline{K}_{\\epsilon_1}}\\le {r}\\rho_{\\overline{K}_{\\epsilon_1}-1}$, we conclude that  $\\rho_k\\le{r}{\\rho}_{\\epsilon_1}$ holds for $0 \\le k\\le \\overline{K}_{\\epsilon_1}$. It remains to\r\nshow that \\eqref{outer-iteration-cmplxty} holds.  To this end, let \r\n\\[\r\n\\bK =\\{k:\\rho_{k+1}={r}\\rho_{k}, K_{\\epsilon_1}\\le k\\le \\overline{K}_{\\epsilon_1}-2\\}.\r\n\\]\r\nIt follows from \\eqref{bound-rhok} and the update scheme of $\\rho_k$ that \r\n\\[\r\n{r}^{|\\bK|}\\rho_{K_{\\epsilon_1}}=\\max_{K_{\\epsilon_1}\\le k \\le \\overline{K}_{\\epsilon_1}-1}\\{\\rho_k\\}\\le {\\rho}_{\\epsilon_1},\r\n\\]\r\nwhich together with $\\rho_{K_{\\epsilon_1}}\\ge\\rho_0$ implies that\r\n\\begin{equation}\\label{bound-cU}\r\n|\\bK|\\le \\log({\\rho}_{\\epsilon_1}\\rho_{K_{\\epsilon_1}}^{-1})/ \\log{r} \\le \\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})/ \\log{r}.\r\n\\end{equation}\r\nLet $\\{k_1,k_2,\\ldots,k_{|\\bK|}\\}$ denote all the elements of $\\mathcal{\\bK}$ arranged in ascending order, and let $k_0=K_{\\epsilon_1}$ and $k_{|\\bK|+1}=\\overline{K}_{\\epsilon_1}-1$. We next derive an upper bound for $k_{j+1}-k_{j}$ for $j=0,1,\\ldots,|\\bK|$. By the definition of $\\bK$, one can observe that $\\rho_{k}=\\rho_{k'}$ for $k_j< k,k'\\le k_{j+1}$. Using this and the update scheme of $\\rho_k$, we deduce that\r\n\\begin{equation}\\label{ineq:sequence-cxk}\r\n\\|\\tilde{c}(x^{k+1})\\|\\le\\alpha\\|\\tilde{c}(x^{k})\\|,\\quad \\forall k_j<k< k_{j+1}.\r\n\\end{equation}\r\nOn the other hand, by \\eqref{def:delta0c-rhobar1xxx}, \\eqref{bd-ck} and $\\rho_k\\ge\\rho_0$, one has $\\|\\tilde{c}(x^{k+1})\\|\\le \\delta_{c,1}$ for $0 \\le k \\le \\overline{K}_{\\epsilon_1}$. By this and \\eqref{lbd-tc}, one can see that\r\n\\begin{equation}\\label{ineq:sequence-cxk2222}\r\n\\epsilon_1/2<\\|\\tilde{c}(x^{k+1})\\|\\le\\delta_{c,1},\\quad \\forall K_{\\epsilon_1} \\le k \\le \\overline{K}_{\\epsilon_1}-1.\r\n\\end{equation}\r\nNow, note that either $k_{j+1} - k_j = 1$ or $k_{j+1} - k_j > 1$. In the latter case, we can apply \\eqref{ineq:sequence-cxk} with $k = k_{j+1}-1,\\ldots,k_j+1$ together with \\eqref{ineq:sequence-cxk2222} to deduce that\r\n\\[\r\n\\epsilon_1/2<\\|\\tilde{c}(x^{k_{j+1}})\\|\\le\\alpha \\|\\tilde{c}(x^{k_{j+1}-1})\\|\\le\\cdots\\le \\alpha^{k_{j+1}-k_j-1}\\|\\tilde{c}(x^{k_{j}+1})\\|\\le \\alpha^{k_{j+1}-k_j-1}\\delta_{c,1},\\quad \\forall j=0,1,\\ldots,|\\bK|.\r\n\\]\r\nCombining these two cases, we have \r\n\\begin{equation}\\label{ineq:subsequence-rho-same}\r\nk_{j+1}-k_{j}\\le |\\log(\\epsilon_1(2\\delta_{c,1})^{-1}))/\\log \\alpha|+1,\\quad \\forall j=0,1,\\ldots,|\\bK|.\r\n\\end{equation}\r\nSumming up these inequalities, and using \\eqref{T-epsilong}, \\eqref{bound-cU},  $k_0=K_{\\epsilon_1}$ and $k_{|\\bK|+1}=\\overline{K}_{\\epsilon_1}-1$, we have\r\n\\begin{eqnarray}\r\n&\\hspace{-.85in}\\overline{K}_{\\epsilon_1} = 1+k_{|\\bK|+1}=1+k_0+\\sum_{j=0}^{|\\bK|}(k_{j+1}-k_{j}) \\overset{\\eqref{ineq:subsequence-rho-same}}{\\le}1+K_{\\epsilon_1}+(|\\bK|+1)\\left(\\left|\\frac{\\log(\\epsilon_1(2\\delta_{c,1})^{-1})}{\\log \\alpha}\\right|+1\\right) \\nonumber \\\\[6pt]\r\n&\\le2+\\frac{\\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})}{\\log{r}}+\\left(\\frac{\\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})}{\\log{r}}+1\\right)\\left(\\left|\\frac{\\log(\\epsilon_1(2\\delta_{c,1})^{-1})}{\\log \\alpha}\\right|+1\\right) =1+ \\left(\\frac{\\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})}{\\log{r}}+1\\right)\\left(\\left|\\frac{\\log(\\epsilon_1(2\\delta_{c,1})^{-1})}{\\log \\alpha}\\right|+2\\right), \\nn\r\n\\end{eqnarray}\r\nwhere the second inequality is due to \\eqref{T-epsilong} and \\eqref{bound-cU}. Hence, \\eqref{outer-iteration-cmplxty} also holds in this case.\r\n\\end{proof}","sha256_utf8":"7fe19accb1087686205a121a8cd7f2527034b7128119089ce42992a8a1df6dba"}]},{"id":"S034","semantic_type":"theorem","kind":"theorem","title":"输出的随机二阶驻点保证","source_units":[{"start":65552,"end":66219,"start_line":610,"end_line":615,"raw_tex":"\\begin{theorem}%[{{\\bf Output of Algorithm \\ref{alg:2nd-order-AL-nonconvex}}}]\r\n\\label{thm:output-alg1}\r\nSuppose that Assumption~\\ref{asp:lowbd-knownfeas} holds and that $\\rho_0$ is sufficiently large such that $\\delta_{f,1} \\le \\bdelta_f$ and  $\\delta_{c,1} \\le \\bdelta_c$, where\r\n$\\delta_{f,1}$ and $\\delta_{c,1}$ are defined in \\eqref{def:delta0c-rhobar1xxx}.\r\nIf Algorithm \\ref{alg:2nd-order-AL-nonconvex} terminates at some iteration $k$, then $x^{k+1}$ is a deterministic $\\epsilon_1$-FOSP of problem \\eqref{model:equa-cnstr}, and moreover, it is an $(\\epsilon_1,\\epsilon_2)$-SOSP of \\eqref{model:equa-cnstr} with probability at least $1-\\delta$.\r\n\\end{theorem}","sha256_utf8":"7f3ec12c6b91f9a3374d6692d0224a4ecd3e6c937c57ca823666f8457a070216"}]},{"id":"S035","semantic_type":"proof","kind":"proof","title":"输出驻点保证证明","source_units":[{"start":118585,"end":120991,"start_line":1143,"end_line":1155,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{thm:output-alg1}]\r\nSuppose that Algorithm \\ref{alg:2nd-order-AL-nonconvex} terminates at some iteration $k$, that is, $\\tau_k^g\\le\\epsilon_1$, $\\tau_k^H\\le\\epsilon_2$, and $\\|c(x^{k+1})\\|\\le\\epsilon_1$ hold. Then, by $\\tau_k^g\\le\\epsilon_1$, $\\tilde{\\lambda}^{k+1}=\\lambda^k+\\rho_k\\tilde{c}(x^{k+1})$, $\\nabla \\tilde{c}=\\nabla c$ and the second relation in \\eqref{algstop:1st-order}, one has \r\n \\begin{align*}\r\n\\|\\nabla f(x^{k+1})+\\nabla c(x^{k+1})\\tilde{\\lambda}^{k+1}\\| &=\\|\\nabla f(x^{k+1})+\\nabla \\tilde{c}(x^{k+1})(\\lambda^k+\\rho_k \\tilde{c}(x^{k+1}))\\| \\\\ \r\n&=\\|\\nabla_x\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k)\\|\\le \\tau_k^g \\le \\epsilon_1.\r\n\\end{align*} \r\n Hence, $(x^{k+1},\\tilde{\\lambda}^{k+1})$ satisfies the first relation in \\eqref{optcond:1st-equa-cnstr}. In addition, by \\eqref{algstop:2nd-order} and $\\tau_k^H\\le\\epsilon_2$, one can show that $\\lambda_{\\min}(\\nabla^2_{xx}\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k))\\ge-\\epsilon_2$ with probability at least $1-\\delta$, which leads to $d^T\\nabla^2_{xx}\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k)d \\ge -\\epsilon_2\\|d\\|^2$ for all $d\\in\\bR^n$ with probability at least $1-\\delta$.  Using this, $\\tilde{\\lambda}^{k+1}=\\lambda^k+\\rho_k\\tilde{c}(x^{k+1})$, $\\nabla \\tilde{c} = \\nabla c$, and $\\nabla^2 \\tilde{c}_i=\\nabla^2 c_i$ for $1\\le i\\le m$, we see that with probability at least $1-\\delta$, it holds that \r\n%$d^T (\\nabla^2 f(x^{k+1})+\\sum_{i=1}^m\\tilde{\\lambda}_i^{k+1}\\nabla^2c_i(x^{k+1})+\\rho_k\\nabla c(x^{k+1})\\nabla c(x^{k+1})^T )d\\ge -\\epsilon_2\\|d\\|^2$ for all $d\\in\\bR^n$},\r\n\\[\r\nd^T\\left(\\nabla^2 f(x^{k+1})+\\sum_{i=1}^m\\tilde{\\lambda}_i^{k+1}\\nabla^2c_i(x^{k+1})+\\rho_k\\nabla c(x^{k+1})\\nabla c(x^{k+1})^T\\right)d\\ge -\\epsilon_2\\|d\\|^2 \\ \\  \\forall d\\in\\bR^n,\r\n\\]\r\nwhich implies $d^T(\\nabla^2 f(x^{k+1})+\\sum_{i=1}^m\\tilde{\\lambda}_i^{k+1}\\nabla^2 c_i(x^{k+1}))d\\ge -\\epsilon_2\\|d\\|^2$ for all $d\\in \\cC(x^{k+1})$, where $\\cC(\\cdot)$ is defined in \\eqref{def:critical-cone}. Hence, $(x^{k+1},\\tilde{\\lambda}^{k+1})$ satisfies \\eqref{optcond:2nd-equa-cnstr} with probability at least $1-\\delta$. Combining these with $\\|c(x^{k+1})\\|\\le\\epsilon_1$, we conclude that $x^{k+1}$ is a deterministic $\\epsilon_1$-FOSP of \\eqref{model:equa-cnstr} and an $(\\epsilon_1,\\epsilon_2)$-SOSP of \\eqref{model:equa-cnstr} with probability at least $1-\\delta$. Hence, Theorem \\ref{thm:output-alg1} holds.\r\n\\end{proof}","sha256_utf8":"30769953bf70b9aa890de207664a9ed5711907874d71a30504f347cc0f60e5f9"}]},{"id":"S036","semantic_type":"derived_fact","kind":"derived_fact","title":"外层次数与罚参数渐近阶","source_units":[{"start":71680,"end":72235,"start_line":668,"end_line":671,"raw_tex":"\\begin{remark}[{{\\bf Upper bounds for $\\overline{K}_{\\epsilon_1}$ and $\\{\\rho_k\\}$}}]\\label{order-out-itera-penalty}\r\nAs observed from Theorem \\ref{thm:out-itr-cmplxity-1}, the number of outer iterations of Algorithm \\ref{alg:2nd-order-AL-nonconvex} for finding a stochastic  $(\\epsilon_1,\r\n\\epsilon_2)$-SOSP of problem \\eqref{model:equa-cnstr} is $\\overline{K}_{\\epsilon_1}+1$, which is at most of $\\cO(|\\log\\epsilon_1|^2)$. In addition, the penalty parameters $\\{\\rho_k\\}$ generated in this algorithm are at most of $\\cO(\\epsilon_1^{-2})$.\r\n\\end{remark}","sha256_utf8":"2769ad9afd8130ec9adac9f4d7ee9b7ba406ed8740aee60ea92a70eb45e3583b"}]},{"id":"S037","semantic_type":"theorem","kind":"theorem","title":"截断共轭梯度迭代界","source_units":[{"start":167734,"end":168154,"start_line":1790,"end_line":1792,"raw_tex":"\\begin{theorem}[{\\bf iteration complexity of Algorithm \\ref{alg:capped-CG}}]\\label{lem:capped-CG}\r\nConsider applying Algorithm~\\ref{alg:capped-CG} with input $U = 0$ to the linear system~\\eqref{indef-sys} with $g\\neq 0$, $\\varepsilon>0$, and $H$ being an $n\\times n$ symmetric matrix. Then the number of iterations of Algorithm~\\ref{alg:capped-CG} is $\\widetilde{\\cO}(\\min\\{n,\\sqrt{\\|H\\|/\\varepsilon}\\})$.\r\n\\end{theorem}","sha256_utf8":"f7c4493ef27767b411a4741873fa7da7275ea5e2d75d698a3982a3b28b772460"}]},{"id":"S038","semantic_type":"proof","kind":"proof","title":"截断共轭梯度迭代界证明","source_units":[{"start":168158,"end":169051,"start_line":1794,"end_line":1799,"raw_tex":"\\begin{proof}\r\nFrom \\cite[Lemma~1]{RNW18}, we know that the number of iterations of Algorithm \\ref{alg:capped-CG} is bounded by $\\min\\{n,J(U,\\varepsilon,\\zeta)\\}$, where $J(U,\\varepsilon,\\zeta)$ is the smallest integer $J$ such that $\\sqrt{T}\\tau^{J/2}\\le\\widehat{\\zeta}$, with $U,\\widehat{\\zeta},T$ and $\\tau$ being the values returned by Algorithm \\ref{alg:capped-CG}. In addition, it was shown in \\cite[Section 3.1]{RNW18} that\r\n$J(U,\\varepsilon,\\zeta)\\le\\left\\lceil \\left(\\sqrt{\\kappa}+\\frac{1}{2}\\right)\\ln\\left(\\frac{144(\\sqrt{\\kappa}+1)^2\\kappa^6}{\\zeta^2}\\right)\\right\\rceil$,\r\nwhere $\\kappa={\\cO}(U/\\varepsilon)$ is an output by Algorithm \\ref{alg:capped-CG}. Then one can see that $J(U,\\varepsilon,\\zeta)=\\widetilde{\\cO}(\\sqrt{U/\\varepsilon})$. Notice from\r\nAlgorithm~\\ref{alg:capped-CG} that the output $U\\le\\|H\\|$. Combining these, we obtain the conclusion as desired.\r\n\\end{proof}","sha256_utf8":"9d043e6fddb51e9414a9e7b4b12912227dadf4c6bf875078f60f24efb43814f0"}]},{"id":"S039","semantic_type":"theorem","kind":"theorem","title":"朗佐斯预言机保证","source_units":[{"start":170794,"end":171386,"start_line":1820,"end_line":1822,"raw_tex":"\\begin{theorem}[{\\bf iteration complexity of Algorithm~\\ref{pro:meo}}]\\label{rand-Lanczos}\r\nConsider Algorithm~\\ref{pro:meo} with tolerance $\\varepsilon>0$, probability parameter $\\delta\\in(0,1)$, and symmetric matrix $H\\in\\bR^{n\\times n}$ as its input. Then it either finds a sufficiently negative curvature direction $v$ satisfying $v^THv\\le-\\varepsilon/2$ and $\\|v\\|=1$ or certifies that $\\lambda_{\\min}(H)\\ge-\\varepsilon$ holds with probability at least  $1-\\delta$  in at most $N(\\varepsilon,\\delta)$ iterations, where $N(\\varepsilon,\\delta)$ is defined in \\eqref{N-iter}.\r\n\\end{theorem}","sha256_utf8":"b124349ac65adf475f45b63764024f4e30b0757660f7fb14dee3dea4637286cd"}]},{"id":"S040","semantic_type":"proof","kind":"proof","title":"子问题初始函数值界推导","source_units":[{"start":60569,"end":60892,"start_line":575,"end_line":578,"raw_tex":"To this end, we start by noting from \\eqref{hbd}, \\eqref{model:equa-cnstr-pert}, \\eqref{tL-tc} and \\eqref{def:initial-iterate-subprob} that\r\n\\beq \\label{L-xinit}\r\n\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k) \\le \\max\\{\\widetilde{\\cL}(z_{\\epsilon_1},\\lambda^k;\\rho_k),f(z_{\\epsilon_1})\\}=f(z_{\\epsilon_1})\\le \\fh.\r\n\\eeq\r\n","sha256_utf8":"151b0a168f6900da7bd7afc4c52bbb9a50533f101392e158dbf7717977b6670e"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"重启规则给出统一初值界","source_units":[{"start":60569,"end":60892,"start_line":575,"end_line":578,"raw_tex":"To this end, we start by noting from \\eqref{hbd}, \\eqref{model:equa-cnstr-pert}, \\eqref{tL-tc} and \\eqref{def:initial-iterate-subprob} that\r\n\\beq \\label{L-xinit}\r\n\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k) \\le \\max\\{\\widetilde{\\cL}(z_{\\epsilon_1},\\lambda^k;\\rho_k),f(z_{\\epsilon_1})\\}=f(z_{\\epsilon_1})\\le \\fh.\r\n\\eeq\r\n","sha256_utf8":"151b0a168f6900da7bd7afc4c52bbb9a50533f101392e158dbf7717977b6670e"}]},{"id":"S042","semantic_type":"proof","kind":"proof","title":"容差同步与终止条件推导","source_units":[{"start":69448,"end":70333,"start_line":645,"end_line":650,"raw_tex":"In view of \\eqref{omega-tolerance}, \\eqref{T-epsilon-g} and the fact that\r\n\\begin{equation}\\label{epsw1-epsw2}\r\n%\\log\\epsilon_1/\\log\\omega_1=\\log2/\\log r=\\log\\epsilon_2/\\log\\omega_2,\r\n\\log\\epsilon_1/\\log\\omega_1=\\log\\epsilon_2/\\log\\omega_2=\\log2/\\log r,\r\n\\end{equation}\r\nwe see that $(\\tau_k^g,\\tau_k^H)=(\\epsilon_1,\\epsilon_2)$ for all $k\\ge K_{\\epsilon_1}$. This along with the termination criterion of Algorithm \\ref{alg:2nd-order-AL-nonconvex} implies that it runs for at least $K_{\\epsilon_1}$ iterations and  terminates once $\\|c(x^{k+1})\\|\\le\\epsilon_1$ for some $k\\ge K_{\\epsilon_1}$. As a result, to establish outer iteration complexity of Algorithm \\ref{alg:2nd-order-AL-nonconvex}, it suffices to bound such  $k$. The resulting outer iteration complexity of Algorithm \\ref{alg:2nd-order-AL-nonconvex} is presented below, whose proof is deferred to Section \\ref{sec:pf-AL}.\r\n","sha256_utf8":"a48fb043b2885e9e88961d18be9150c7fa29b62b368b998cb181596b79a69d69"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"两种容差同时达到目标","source_units":[{"start":69448,"end":70333,"start_line":645,"end_line":650,"raw_tex":"In view of \\eqref{omega-tolerance}, \\eqref{T-epsilon-g} and the fact that\r\n\\begin{equation}\\label{epsw1-epsw2}\r\n%\\log\\epsilon_1/\\log\\omega_1=\\log2/\\log r=\\log\\epsilon_2/\\log\\omega_2,\r\n\\log\\epsilon_1/\\log\\omega_1=\\log\\epsilon_2/\\log\\omega_2=\\log2/\\log r,\r\n\\end{equation}\r\nwe see that $(\\tau_k^g,\\tau_k^H)=(\\epsilon_1,\\epsilon_2)$ for all $k\\ge K_{\\epsilon_1}$. This along with the termination criterion of Algorithm \\ref{alg:2nd-order-AL-nonconvex} implies that it runs for at least $K_{\\epsilon_1}$ iterations and  terminates once $\\|c(x^{k+1})\\|\\le\\epsilon_1$ for some $k\\ge K_{\\epsilon_1}$. As a result, to establish outer iteration complexity of Algorithm \\ref{alg:2nd-order-AL-nonconvex}, it suffices to bound such  $k$. The resulting outer iteration complexity of Algorithm \\ref{alg:2nd-order-AL-nonconvex} is presented below, whose proof is deferred to Section \\ref{sec:pf-AL}.\r\n","sha256_utf8":"a48fb043b2885e9e88961d18be9150c7fa29b62b368b998cb181596b79a69d69"}]},{"id":"S044","semantic_type":"proof","kind":"proof","title":"扰动罚函数下界推导","source_units":[{"start":111695,"end":112100,"start_line":1044,"end_line":1049,"raw_tex":"Recall from Assumption~\\ref{asp:lowbd-knownfeas}(a) that $\\|c(z_{\\epsilon_1})\\|\\le\\epsilon_1/2<1$. By virtue of this, \\eqref{lbd} and the definition of $\\tilde{c}$ in\r\n\\eqref{model:equa-cnstr-pert},\r\nwe obtain that\r\n\\begin{equation}\\label{ineq:p-lowbd}\r\nf(x)+\\gamma\\|\\tilde{c}(x)\\|^2 \\ge f(x)+ \\gamma\\|c(x)\\|^2/2 - \\gamma \\|c(z_{\\epsilon_1})\\|^2 \\ge \\fl - \\gamma,\\quad \\forall x\\in\\bR^n.\r\n\\end{equation}\r\n","sha256_utf8":"34eee3a647b57760922cb9dc9513639ce5da8537bec7bc590664de4a470118d7"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"平移约束保留罚函数下界","source_units":[{"start":111695,"end":112100,"start_line":1044,"end_line":1049,"raw_tex":"Recall from Assumption~\\ref{asp:lowbd-knownfeas}(a) that $\\|c(z_{\\epsilon_1})\\|\\le\\epsilon_1/2<1$. By virtue of this, \\eqref{lbd} and the definition of $\\tilde{c}$ in\r\n\\eqref{model:equa-cnstr-pert},\r\nwe obtain that\r\n\\begin{equation}\\label{ineq:p-lowbd}\r\nf(x)+\\gamma\\|\\tilde{c}(x)\\|^2 \\ge f(x)+ \\gamma\\|c(x)\\|^2/2 - \\gamma \\|c(z_{\\epsilon_1})\\|^2 \\ge \\fl - \\gamma,\\quad \\forall x\\in\\bR^n.\r\n\\end{equation}\r\n","sha256_utf8":"34eee3a647b57760922cb9dc9513639ce5da8537bec7bc590664de4a470118d7"}]},{"id":"S046","semantic_type":"proof","kind":"proof","title":"增广海森矩阵尺度推导","source_units":[{"start":129252,"end":130611,"start_line":1238,"end_line":1246,"raw_tex":"We next prove Theorem \\ref{thm:total-iter-cmplxity}. Before proceeding, we introduce some notation that will be used shortly. Let $L_{k,H}$ denote the Lipschitz constant of $\\nabla^2_{xx} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$ on the convex open neighborhood $\\Omega(\\bdelta_{f},\\bdelta_{c})$ of\r\n$\\cS(\\bdelta_f,\\bdelta_c)$, where $\\cS(\\bdelta_f,\\bdelta_c)$ is defined in \\eqref{nearly-feas-level-set}, and let $U_{k,H}=\\sup_{x\\in\\cS(\\bdelta_f,\\bdelta_c)} \\|\\nabla^2_{xx} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\|$. Notice from \\eqref{model:equa-cnstr-pert} and \\eqref{tL-tc} that\r\n\\begin{equation}\\label{eq:hessian-augmented-Lagrangian}\r\n\\nabla^2_{xx}\\widetilde{\\cL}(x,\\lambda^k;\\rho_k) = \\nabla^2 f(x) +\\sum_{i=1}^m\\lambda^k_{i}\\nabla^2 c_i(x) + \\rho_k \\bigg(\\nabla c(x)\\nabla c(x)^T + \\sum_{i=1}^m \\tilde{c}_i(x)\\nabla^2 c_i(x)\\bigg).\r\n\\end{equation}\r\nBy this, $\\|\\lambda^k\\| \\le \\Lambda$, the definition of $\\tilde{c}$, and the Lipschitz continuity of $\\nabla^2 f$ and $\\nabla^2 c_i$'s (see Assumption~\\ref{asp:lowbd-knownfeas}(c)), one can observe that there exist some constants ${L}_{1}$, ${L}_{2}$, ${U}_{1}$ and ${U}_{2}$, depending only on $f$, $c$, $\\Lambda$, $\\bdelta_{f}$ and $\\bdelta_{c}$,  such that\r\n\\begin{equation}\\label{ineq:bound-norm-hessian-Liphessian}\r\nL_{k,H}\\le {L}_{1}+\\rho_k{L}_{2},\\quad U_{k,H}\\le{U}_{1}+\\rho_k{U}_{2}.\r\n\\end{equation}\r\n","sha256_utf8":"7c7a3c233d960f95c961c6a9f73d6d7fc48f521e31d1683ff59adf951956b64b"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"定义海森界并展开增广海森矩阵","source_units":[{"start":129252,"end":130101,"start_line":1238,"end_line":1242,"raw_tex":"We next prove Theorem \\ref{thm:total-iter-cmplxity}. Before proceeding, we introduce some notation that will be used shortly. Let $L_{k,H}$ denote the Lipschitz constant of $\\nabla^2_{xx} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$ on the convex open neighborhood $\\Omega(\\bdelta_{f},\\bdelta_{c})$ of\r\n$\\cS(\\bdelta_f,\\bdelta_c)$, where $\\cS(\\bdelta_f,\\bdelta_c)$ is defined in \\eqref{nearly-feas-level-set}, and let $U_{k,H}=\\sup_{x\\in\\cS(\\bdelta_f,\\bdelta_c)} \\|\\nabla^2_{xx} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\|$. Notice from \\eqref{model:equa-cnstr-pert} and \\eqref{tL-tc} that\r\n\\begin{equation}\\label{eq:hessian-augmented-Lagrangian}\r\n\\nabla^2_{xx}\\widetilde{\\cL}(x,\\lambda^k;\\rho_k) = \\nabla^2 f(x) +\\sum_{i=1}^m\\lambda^k_{i}\\nabla^2 c_i(x) + \\rho_k \\bigg(\\nabla c(x)\\nabla c(x)^T + \\sum_{i=1}^m \\tilde{c}_i(x)\\nabla^2 c_i(x)\\bigg).\r\n\\end{equation}\r\n","sha256_utf8":"1f8fe45c871665d8c0d2ed6dbd6d437893de709b91cff8a13c936642a4596a97"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"海森范数与光滑常数线性增长","source_units":[{"start":130101,"end":130611,"start_line":1243,"end_line":1246,"raw_tex":"By this, $\\|\\lambda^k\\| \\le \\Lambda$, the definition of $\\tilde{c}$, and the Lipschitz continuity of $\\nabla^2 f$ and $\\nabla^2 c_i$'s (see Assumption~\\ref{asp:lowbd-knownfeas}(c)), one can observe that there exist some constants ${L}_{1}$, ${L}_{2}$, ${U}_{1}$ and ${U}_{2}$, depending only on $f$, $c$, $\\Lambda$, $\\bdelta_{f}$ and $\\bdelta_{c}$,  such that\r\n\\begin{equation}\\label{ineq:bound-norm-hessian-Liphessian}\r\nL_{k,H}\\le {L}_{1}+\\rho_k{L}_{2},\\quad U_{k,H}\\le{U}_{1}+\\rho_k{U}_{2}.\r\n\\end{equation}\r\n","sha256_utf8":"ab76acd68cdede7289c5c5268c12d010227162f8e331b2bda654e998e50ba3c8"}]},{"id":"S049","semantic_type":"external_fact","kind":"external_fact","title":"标准共轭梯度输出性质","source_units":[],"declared_external":true,"declaration_text":["RNW18, Lemma 3, and NW19, Lemma 7, supply the standard capped-CG properties recalled in B0080. For a SOL direction, with A=∇²F(xᵗ)+2ε_H I, ε_H‖dᵗ‖²≤(dᵗ)ᵀAdᵗ, ‖dᵗ‖≤1.1ε_H⁻¹‖∇F(xᵗ)‖, (dᵗ)ᵀ∇F(xᵗ)=−(dᵗ)ᵀAdᵗ, and ‖Adᵗ+∇F(xᵗ)‖≤ε_Hζ‖dᵗ‖/2. After the stated NC scaling, (dᵗ)ᵀ∇F(xᵗ)≤0 and (dᵗ)ᵀ∇²F(xᵗ)dᵗ/‖dᵗ‖²=−‖dᵗ‖≤−ε_H."]},{"id":"S050","semantic_type":"external_fact","kind":"external_fact","title":"已有共轭梯度终止估计","source_units":[],"declared_external":true,"declaration_text":["RNW18, Lemma 1 and Section 3.1: capped CG terminates within min{n,J(U,ε,ζ)} iterations, where J is the smallest integer satisfying √T τ^(J/2)≤ζ_hat and J≤ceil[(√κ+1/2) ln(144(√κ+1)²κ⁶/ζ²)], with the returned parameters defined by the capped-CG algorithm."]},{"id":"S051","semantic_type":"external_fact","kind":"external_fact","title":"标准随机朗佐斯保证","source_units":[],"declared_external":true,"declaration_text":["The randomized Lanczos guarantee cited as RNW18, Lemma 2, for the method of KW92LR: for symmetric H, ε>0 and δ∈(0,1), within N=min{n,1+ceil[(ln(2.75n/δ²)/2)√(‖H‖/ε)]} iterations, the oracle returns a unit vector v with vᵀHv≤−ε/2 or certifies λ_min(H)≥−ε with probability at least 1−δ. Each Lanczos iteration uses one matrix-vector product."]},{"id":"S052","semantic_type":"external_fact","kind":"external_fact","title":"紧性与光滑分析事实","source_units":[],"declared_external":true,"declaration_text":["Continuous real-valued and matrix-valued functions are bounded on compact sets. A Hessian with Lipschitz constant L on a convex neighborhood gives gradient Taylor error at most L‖y−x‖²/2 and function Taylor error at most L‖y−x‖³/6. A twice differentiable scalar function at an interior local minimum has zero first derivative and nonnegative second derivative."]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"分解总成本并控制罚参数","source_units":[{"start":130682,"end":131490,"start_line":1250,"end_line":1251,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{thm:total-iter-cmplxity}]\r\nLet $T_k$ and $N_k$ denote the number of iterations and matrix-vector products performed by Algorithm~\\ref{alg:NCG} at the outer iteration $k$ of Algorithm \\ref{alg:2nd-order-AL-nonconvex}, respectively. It then follows from Theorem \\ref{thm:out-itr-cmplxity-1} that the total number of iterations and matrix-vector products performed by Algorithm~\\ref{alg:NCG} in Algorithm \\ref{alg:2nd-order-AL-nonconvex} are $\\sum_{k=0}^{\\overline{K}_{\\epsilon_1}}T_k$ and $\\sum_{k=0}^{\\overline{K}_{\\epsilon_1}}N_k$, respectively. In addition, notice from \\eqref{def:delta0c-rhobar1} and Theorem \\ref{thm:out-itr-cmplxity-1} that ${\\rho}_{\\epsilon_1}=\\cO(\\epsilon_1^{-2})$ and  $\\rho_k\\le{r}{\\rho}_{\\epsilon_1}$, which yield $\\rho_k=\\cO(\\epsilon_1^{-2})$.\r\n","sha256_utf8":"2558ead7a38510df5356deae43fe2f1b6dcb753791856bd1fe19bce4ebaf61df"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"统一控制内层容差比","source_units":[{"start":131492,"end":134147,"start_line":1253,"end_line":1272,"raw_tex":"%(i) {  We first prove that $(\\tau_k^g)^2/\\tau_k^H\\ge\\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$ holds for any $k\\ge0$. If $\\epsilon_1^2{ \\ge}\\epsilon_2$, it follows from $\\tau_k^g\\ge\\epsilon_1$ and $\\tau_k^H\\le 1$ that $(\\tau_k^g)^2/\\tau_k^H\\ge(\\tau_k^g)^2\\ge \\epsilon_1^2{ \\ge}\\epsilon_2$, which along with $\\epsilon_2\\in(0,1)$ implies $(\\tau_k^g)^2/\\tau_k^H{ \\ge}\\epsilon_2^3$. On the other hand, if $\\epsilon_1^2{ <}\\epsilon_2$, it follows from \\eqref{omega-tolerance} that $\\omega_1^2/\\omega_2{ <} 1$. Also, recall from \\eqref{omega-tolerance}, \\eqref{T-epsilon-g} and \\eqref{epsw1-epsw2} that\r\n%\\[\r\n%K_{\\epsilon_1}=\\left\\lceil \\log\\epsilon_1/\\log\\omega_1\\right\\rceil=\\left\\lceil \\log\\epsilon_2/\\log\\omega_2\\right\\rceil,\\quad\r\n%(\\tau_k^g,\\tau_k^H)=\\left\\{\\begin{array}{ll}\r\n%(\\omega_1^k,\\omega_2^k)&\\text{if }\\ 0\\le k< K_{\\epsilon_1},\\\\\r\n%(\\epsilon_1,\\epsilon_2)&\\text{otherwise}.\r\n%\\end{array}\\right.\r\n%\\]\r\n%In view of these, we see that the sequence $\\{(\\tau_k^g)^2/\\tau_k^H\\}$ is geometrically decreasing with ratio $\\omega_1^2/\\omega_2{ <} 1$ when $0\\le k< K_{\\epsilon_1}$ and equals $\\epsilon_1^2/\\epsilon_2$ for all $k\\ge K_{\\epsilon_1}$. In addition, notice from \\eqref{T-epsilon-g} and \\eqref{epsw1-epsw2} that\r\n%\\[\r\n%\\left\\lceil\\log(\\epsilon_1^2/\\epsilon_2)/\\log(\\omega_1^2/\\omega_2)\\right\\rceil =\\left\\lceil \\log 2/\\log r\\right\\rceil = K_{\\epsilon_1}.\r\n%\\]\r\n%Thus, we have $(\\tau_k^g)^{2}/\\tau_k^H=\\max\\{\\epsilon_1^2/\\epsilon_2,(\\omega_1^2/\\omega_2)^k\\}\\ge\\epsilon_1^2/\\epsilon_2$ when $\\epsilon_1^2{ <}\\epsilon_2$. Combining this with the fact that $(\\tau_k^g)^2/\\tau_k^H{ \\ge}\\epsilon_2^3$ when $\\epsilon_1^2{ \\ge}\\epsilon_2$, we obtain that $(\\tau_k^g)^2/\\tau_k^H\\ge\\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$ holds for any $k\\ge0$.}\r\n\r\nWe first claim that $(\\tau_k^g)^2/\\tau_k^H\\ge\\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$ holds for any $k\\ge0$. Indeed, let $\\bar t=\\log\\epsilon_1/\\log\\omega_1$ and $\\psi(t)=\\max\\{\\epsilon_1,\\omega_1^t\\}^2/\\max\\{\\epsilon_2,\\omega_2^t\\}$ for all $t\\in\\bR$. It then follows from \\eqref{epsw1-epsw2} that $\\omega_1^{\\bar t}=\\epsilon_1$ and $\\omega_2^{\\bar t}=\\epsilon_2$. By this and $\\omega_1,\\omega_2\\in(0,1)$, one can observe that  $\\psi(t)=(\\omega_1^2/\\omega_2)^t$ if $t \\leq \\bar t$ and $\\psi(t)=\\epsilon_1^2/\\epsilon_2$ otherwise. This along with $\\epsilon_2\\in(0,1)$ implies that\r\n\\[\r\n\\min_{t\\in[0,\\infty)}\\psi(t) = \\min\\{\\psi(0),\\psi(\\bar t)\\}=\\min\\{1,\\epsilon_1^2/\\epsilon_2\\} \\geq \\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\},\r\n\\]\r\nwhich together with \\eqref{omega-tolerance} yields\r\n$(\\tau_k^g)^2/\\tau_k^H=\\psi(k) \\geq \\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$ for all $k \\geq 0$.\r\n","sha256_utf8":"175d7af2214abe16e8482b11d202636335800147602da1d3a1ca731fdb52ea4e"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"逐子问题估计内层迭代次数","source_units":[{"start":134149,"end":136289,"start_line":1274,"end_line":1290,"raw_tex":"(i) From Lemma \\ref{lem:level-set-augmented-lagrangian-func}(i) and the definitions of $\\Omega(\\bdelta_{f},\\bdelta_{c})$ and $L_{k,H}$, we see that $L_{k,H}$ is a Lipschitz constant of $\\nabla^2_{xx} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$ on a convex open neighborhood  of $\\{x:\\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\}$.\r\n% In addition, by Lemma \\ref{lem:level-set-augmented-lagrangian-func}(ii) and the assumption $\\delta_{c,1} \\le \\bdelta_c$, one has $\\inf_x \\cL(x,\\lambda^k;\\rho_k) \\ge\\fl-\\Lambda\\bdelta_c$.\r\nAlso, recall from Lemma \\ref{lem:level-set-augmented-lagrangian-func}(ii) that $\\inf_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k) $ $\\ge\\fl-\\gamma-\\Lambda\\bdelta_c$. By these, $\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\le \\fh$ (see \\eqref{L-xinit}) and Theorem \\ref{thm:NCG-iter-oper-cmplxity}(iii) with $(\\Fh,\\Fl,L_{H}^F,\\epsilon_g,\\epsilon_H)=(\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k),\\fl-\\gamma-\\Lambda \\bdelta_c,L_{k,H},\\tau_k^g,\\tau_k^H)$, one has\r\n\\begin{equation}\\label{eq:alg1-inner-iter-kthAL}\r\n\\begin{array}{rcl}\r\nT_k&=&\\cO((\\fh-\\fl+\\gamma+\\Lambda\\bdelta_{c})L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\})\\\\[2pt]\r\n&\\overset{\\eqref{ineq:bound-norm-hessian-Liphessian}}{=}&\\cO(\\rho_k^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\})=\\cO(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}),\r\n\\end{array}\r\n\\end{equation}\r\n%\\begin{equation}\\label{eq:alg1-inner-iter-kthAL}\r\n%\\begin{array}{rcl}\r\n%T_k&=&\\cO((\\cL(x^{k}_{\\init},\\lambda^k;\\rho_k)-\\fl+\\gamma+\\Lambda\\bdelta_{c})L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\})\\\\[4pt]\r\n%&=&\\cO((\\fh-\\fl+\\gamma+\\Lambda\\bdelta_{c})L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\})\\\\[4pt]\r\n%&\\overset{\\eqref{ineq:bound-norm-hessian-Liphessian}}{=}&\\cO(\\rho_k^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\})=\\cO(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}),\r\n%\\end{array}\r\n%\\end{equation}\r\nwhere the last equality is from $(\\tau_k^g)^2/\\tau_k^H\\ge\\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$, $\\tau_k^H\\ge\\epsilon_2$, and $\\rho_k=\\cO(\\epsilon_1^{-2})$.\r\n","sha256_utf8":"8f0bb45add94bf2f3a71762b68d41c06d4cb6c30b6d89a47db887192236b5c98"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"仿射约束改进并累加迭代次数","source_units":[{"start":136291,"end":136920,"start_line":1292,"end_line":1293,"raw_tex":"Next, if %$c$ is assumed to be affine, namely,\r\n$c(x)=Ax-b$ for some $A\\in\\bR^{m\\times n}$ and $b\\in\\bR^m$, then $\\nabla c(x)=A^T$ and $\\nabla^2 c_i(x)=0$ for $1 \\le i \\le m$. By these and \\eqref{eq:hessian-augmented-Lagrangian}, one has $L_{k,H}=\\cO(1)$. Using this and similar arguments as for \\eqref{eq:alg1-inner-iter-kthAL}, we obtain that $T_k=\\cO(\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\})$. By this, \\eqref{eq:alg1-inner-iter-kthAL} and $\\overline{K}_{\\epsilon_1}=\\cO(|\\log\\epsilon_1|^2)$ (see Remark \\ref{order-out-itera-penalty}), we conclude that statement (i) of Theorem \\ref{thm:total-iter-cmplxity} holds.\r\n","sha256_utf8":"ce4f9fced4d3b89d850b6f42664185e49d767d9a36a18cf127e4483c34485ece"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"逐子问题估计海森向量乘积","source_units":[{"start":136922,"end":139022,"start_line":1295,"end_line":1317,"raw_tex":" (ii) In view of Lemma \\ref{lem:level-set-augmented-lagrangian-func}(i) and the definition of\r\n $U_{k,H}$, one can see that \r\n\\[\r\nU_{k,H} \\ge \\sup_{x\\in\\bR^n}\\{\\|\\nabla^2_{xx} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\|: \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\}.\r\n\\] \r\n Using this, $\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)$ $\\le \\fh$ and Theorem \\ref{thm:NCG-iter-oper-cmplxity}(iv) with $(\\Fh,\\Fl,L_{H}^F,U_{H}^F,\\epsilon_g,\\epsilon_H)=(\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k),\r\n\\fl-\\gamma-\\Lambda\\bdelta_c,L_{k,H},U_{k,H},\\tau_k^g,\\tau_k^H)$, we obtain that\r\n\\begin{equation}\\label{eq:alg1-oper-kthAL}\r\n\\begin{array}{rcl}\r\nN_k&=&\\widetilde{\\cO}((\\fh-\\fl+\\gamma+\\Lambda\\bdelta_c)L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\}\\min\\{n,(U_{k,H}/\\tau_k^H)^{1/2}\\})\\\\[4pt]\r\n&\\overset{\\eqref{ineq:bound-norm-hessian-Liphessian}}{=}&\\widetilde{\\cO}(\\rho_k^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\}\\min\\{n,(\\rho_k/\\tau_k^H)^{1/2}\\})\\\\[4pt]\r\n&=&\\widetilde{\\cO}(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}\\min\\{n,\\epsilon_1^{-1}\\epsilon_2^{-1/2}\\}),\t\r\n\\end{array}\r\n\\end{equation}\r\n%\\begin{equation}\\label{eq:alg1-oper-kthAL}\r\n%\\begin{array}{rcl}\r\n%\\hspace{-2mm}N_k&=&\\widetilde{\\cO}((\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)-\\fl+\\gamma+\\Lambda\\bdelta_c)L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\}\\min\\{n,(U_{k,H}/\\tau_k^H)^{1/2}\\})\\\\[4pt]\r\n%&=&\\widetilde{\\cO}((\\fh-\\fl+\\gamma+\\Lambda\\bdelta_c)L_{k,H}^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\}\\min\\{n,(U_{k,H}/\\tau_k^H)^{1/2}\\})\\\\[4pt]\r\n%&\\overset{\\eqref{ineq:bound-norm-hessian-Liphessian}}{=}&\\widetilde{\\cO}(\\rho_k^2\\max\\{(\\tau^g_k)^{-2}\\tau^H_k,(\\tau_k^H)^{-3}\\}\\min\\{n,(\\rho_k/\\tau_k^H)^{1/2}\\})\\\\[4pt]\r\n%&=&\\widetilde{\\cO}(\\epsilon_1^{-4}\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}\\min\\{n,\\epsilon_1^{-1}\\epsilon_2^{-1/2}\\}),\t\r\n%\\end{array}\r\n%\\end{equation}\r\nwhere the last equality  is from $(\\tau_k^g)^2/\\tau_k^H\\ge\\min\\{\\epsilon_1^2/\\epsilon_2,\\epsilon_2^3\\}$, $\\tau_k^H\\ge\\epsilon_2$, and $\\rho_k=\\cO(\\epsilon_1^{-2})$.\r\n","sha256_utf8":"384b8e5ea19cbd37734b91853676a4227644d2634281641e2651081582972658"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"仿射约束改进并累加运算成本","source_units":[{"start":139024,"end":139592,"start_line":1319,"end_line":1320,"raw_tex":"On the other hand, if $c$ is assumed to be affine, it follows from the above discussion that $L_{k,H}=\\cO(1)$. Using this, $U_{k,H}\\le U_1+\\rho_k U_2$, and similar arguments as for \\eqref{eq:alg1-oper-kthAL}, we obtain that $N_k=\\widetilde{\\cO}(\\max\\{\\epsilon_1^{-2}\\epsilon_2,\\epsilon_2^{-3}\\}\\min\\{n,\\epsilon_1^{-1}\\epsilon_2^{-1/2}\\})$. By this, \\eqref{eq:alg1-oper-kthAL} and $\\overline{K}_{\\epsilon_1}=\\cO(|\\log\\epsilon_1|^2)$ (see Remark \\ref{order-out-itera-penalty}), we conclude that statement (ii) of Theorem \\ref{thm:total-iter-cmplxity} holds.\r\n\\end{proof}","sha256_utf8":"d617469284c36e9cf2d8097b15605739ad19ee2677834359f400adf71554ba62"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"下降迭代保持海森范数界","source_units":[{"start":107220,"end":107740,"start_line":1018,"end_line":1020,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{thm:NCG-iter-oper-cmplxity}]\r\nFor notational convenience, we let $\\{x^t\\}_{t\\in\\bT}$ denote all the iterates generated by Algorithm~\\ref{alg:NCG}, where $\\bT$ is a set of consecutive nonnegative integers starting from $0$. Notice that $F$ is descent along the iterates generated by Algorithm~\\ref{alg:NCG}, which together with $x^0=u^0$ implies that $x^t\\in\\{x:F(x)\\le F(u^0)\\}$. It then follows from\r\n\\eqref{lwbd-Hgupbd} that $\\|\\nabla^2 F(x^t)\\|\\le U^F_H$ holds for all $t\\in\\bT$.\r\n","sha256_utf8":"c28d45bc870fcec57761c8a61f7dd5b9b529837ce406678ca90f784b3d8b9f4b"}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","title":"望远镜求和限制特征值调用","source_units":[{"start":107742,"end":108373,"start_line":1022,"end_line":1026,"raw_tex":"(i) Suppose for contradiction that the total number of calls of Algorithm~\\ref{pro:meo} in Algorithm~\\ref{alg:NCG} is more than $T_2$. Notice from Algorithm~\\ref{alg:NCG} and Lemma~\\ref{lem:nc}(ii) that each of these calls, except the last one, returns a sufficiently negative curvature direction, and each of them results in a reduction on $F$ of at least $c_{\\rm nc} \\epsilon_H^3$. Hence, \r\n\\[\r\nT_2 c_{\\rm nc} \\epsilon_H^3 \\le \\sum_{t\\in\\bT} [F(x^t)-F(x^{t+1})] \\le F(x^0)-\\Fl = \\Fh-\\Fl,\r\n\\]\r\nwhich contradicts the definition of $T_2$ given in \\eqref{T1}. Hence, statement (i) of Theorem \\ref{thm:NCG-iter-oper-cmplxity} holds.\r\n","sha256_utf8":"7ec72ccdcb8fa9575d65aacc260f55f27dadf4d35e005e9db378002e367a86d0"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"分类计数限制共轭梯度调用","source_units":[{"start":108375,"end":109670,"start_line":1028,"end_line":1032,"raw_tex":"(ii) Suppose for contradiction that the total number of calls of Algorithm~\\ref{alg:capped-CG} in Algorithm~\\ref{alg:NCG} is more than $T_1$. Observe that if Algorithm~\\ref{alg:capped-CG} is called at some iteration $t$ and generates the next iterate $x^{t+1}$ satisfying $\\|\\nabla F(x^{t+1})\\|\\le\\epsilon_g$, then Algorithm~\\ref{pro:meo} must be called at the next iteration $t+1$. In view of this and statement (i) of Theorem \\ref{thm:NCG-iter-oper-cmplxity}, we see that the total number of such iterations $t$ is at most $T_2$. Hence, the total number of iterations $t$ of Algorithm~\\ref{alg:NCG} at which Algorithm~\\ref{alg:capped-CG} is called and generates the next iterate $x^{t+1}$ satisfying $\\|\\nabla F(x^{t+1})\\|>\\epsilon_g$ is at least $T_1-T_2+1$. Moreover, for each of such iterations $t$, we observe from Lemmas~\\ref{lem:sol}(ii) and \\ref{lem:nc}(ii) that $F(x^t)-F(x^{t+1})\\ge \\min\\{c_{\\rm sol},c_{\\rm nc}\\}\\min\\{\\epsilon_g^2\\epsilon_H^{-1},\\epsilon_H^3\\}$. It then follows that \r\n\\[\r\n(T_1-T_2+1) \\min\\{c_{\\rm sol},c_{\\rm nc}\\}\\min\\{\\epsilon_g^2\\epsilon_H^{-1},\\epsilon_H^3\\} \\le \\sum_{t\\in\\bT}[F(x^t)-F(x^{t+1})]\\le \\Fh-\\Fl,\r\n\\]\r\nwhich contradicts the definition of $T_1$ and $T_2$ given in \\eqref{T1}. Hence, statement (ii) of Theorem \\ref{thm:NCG-iter-oper-cmplxity} holds.\r\n","sha256_utf8":"fd618eb5984482692631c33e832f26a828c5d09553d019293b4abf683d664a59"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"合并调用次数与驻点保证","source_units":[{"start":109672,"end":110475,"start_line":1034,"end_line":1035,"raw_tex":"(iii) Notice that either Algorithm~\\ref{alg:capped-CG} or \\ref{pro:meo} is called at each iteration of Algorithm~\\ref{alg:NCG}. It follows from this and statements (i) and (ii) of Theorem \\ref{thm:NCG-iter-oper-cmplxity} that the total number of iterations of Algorithm~\\ref{alg:NCG} is at most $T_1+T_2$. In addition, the relation \\eqref{NCG-iter} follows from \\eqref{csol}, \\eqref{cnc} and \\eqref{T1}. One can also observe that the output $x^t$ of Algorithm~\\ref{alg:NCG} satisfies $\\|\\nabla F(x^t)\\|\\le \\epsilon_g$ deterministically and $\\lambda_{\\min}(\\nabla^2F(x^t))\\ge-\\epsilon_H$ with probability at least $1-\\delta$ for some $0 \\le t \\le T_1+T_2$, where the latter part is due to\r\nAlgorithm~\\ref{pro:meo}. This completes the proof of statement (ii) of Theorem \\ref{thm:NCG-iter-oper-cmplxity}.\r\n","sha256_utf8":"57679cb5fa05ded0c2a103df01369b4ffafeb6810de5afaa07e3bd24e3858660"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"调用成本转化为总乘积次数","source_units":[{"start":110477,"end":111374,"start_line":1037,"end_line":1038,"raw_tex":"(iv) By Theorem~\\ref{lem:capped-CG} with $(H,\\varepsilon)=(\\nabla^2 F(x^t),\\epsilon_H)$ and the fact that $\\|\\nabla^2 F(x^t)\\|\\le U^F_H$, one can observe that  the number of Hessian-vector products required by each call of Algorithm~\\ref{alg:capped-CG} with input $U = 0$ is at most $\\widetilde{\\cO}(\\min\\{n,(U_H^F/\\epsilon_H)^{1/2}\\})$. In addition, by Theorem~\\ref{rand-Lanczos} with $(H,\\varepsilon)=(\\nabla^2 F(x^t),\\epsilon_H)$, $\\|\\nabla^2 F(x^t)\\|\\le U^F_H$, and the fact that each iteration of the Lanczos method requires only one matrix-vector product, one can observe that the number of Hessian-vector products required by each call of Algorithm~\\ref{pro:meo} is also at most $\\widetilde{\\cO}(\\min\\{n,(U_H^F/\\epsilon_H)^{1/2}\\})$. Based on these observations and statement (iii) of Theorem \\ref{thm:NCG-iter-oper-cmplxity}, we see that  statement (iv) of this theorem holds.\r\n\\end{proof}","sha256_utf8":"4368314d3b9b398fd9baf6aa1c75b7c7ddd3474a6c2b50b6988d5d8dc9f34b5e"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"近似牛顿方向非零且梯度有界","source_units":[{"start":92450,"end":93527,"start_line":861,"end_line":866,"raw_tex":"\\begin{proof}\r\nOne can observe that $F$ is descent along the iterates (whenever well-defined) generated by Algorithm~\\ref{alg:NCG}, which together with $x^0=u^0$ implies that $F(x^t)\\le F(u^0)$ and hence\r\n$\\|\\nabla F(x^t)\\|\\le U^F_g$ due to \\eqref{lwbd-Hgupbd}.\r\n%Since $d^t$ { results from} the output $d$ of Algorithm~\\ref{alg:capped-CG} with d$\\_$type=SOL, it follows from Lemma~\\ref{lem:SOL-NC-ppty}(i) that \\eqref{SOL-ppty-1}-\\eqref{SOL-ppty-4} hold for $d^t$.\r\n%In particular, we see from \\eqref{SOL-ppty-2} and \\eqref{SOL-ppty-4} that $\\|(\\nabla^2F(x^t)+2\\epsilon_HI)d^t+\\nabla F(x^t)\\|\\le0.55\\zeta\\|\\nabla F(x^t)\\|$. Since we also have $\\|\\nabla F(x^t)\\| > \\epsilon_g$ in this case and recall that $ \\zeta \\in (0,1)$, we conclude that $d^t\\neq 0$.\r\nIn addition, since $d^t$ results from the output $d$ of Algorithm~\\ref{alg:capped-CG} with d$\\_$type=SOL, one can see that $\\|\\nabla F(x^t)\\| > \\epsilon_g$ and \\eqref{SOL-ppty-1}-\\eqref{SOL-ppty-4} hold for $d^t$. Moreover, by $\\|\\nabla F(x^t)\\| > \\epsilon_g$ and \\eqref{SOL-ppty-4}, one can conclude that $d^t\\neq 0$.\r\n","sha256_utf8":"72d468c7a205025670d360a4424227a459e40040a9b6b9dc567f3ed1993ea092"}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","title":"设置失败试探步的下界目标","source_units":[{"start":93529,"end":94078,"start_line":868,"end_line":872,"raw_tex":"We first prove statement (i). If \\eqref{ls-sol} holds for $j=0$, then $\\alpha_t=1$, which clearly implies that \\eqref{lwbd-ak-sol} holds. We now suppose that \\eqref{ls-sol} fails for $j=0$. Claim that for all $j\\ge0$ that violate \\eqref{ls-sol}, it holds that\r\n\\begin{equation}\\label{lwbd-sol-ak}\r\n\\theta^{2j}\\ge\\min\\{6(1-\\eta),2\\}\\epsilon_H(L^F_H)^{-1}\\|d^t\\|^{-1}.\r\n\\end{equation}\r\nIndeed, suppose that \\eqref{ls-sol} is violated by some $j \\ge 0$. We now show that \\eqref{lwbd-sol-ak} holds for such $j$ by considering two separate cases below.\r\n","sha256_utf8":"3395369c4aa07f40bbf5e8622caa7a305a2ccba076ada0a7574f0c3f86885375"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"上升试探点通过线内极小值估计","source_units":[{"start":94080,"end":95540,"start_line":874,"end_line":887,"raw_tex":"Case 1) $F(x^t+\\theta^jd^t)>F(x^t)$. Let $\\phi(\\alpha)=F(x^t+\\alpha d^t)$. Then $\\phi(\\theta^j)>\\phi(0)$. Also, since $d^t\\neq0$, by \\eqref{SOL-ppty-1} and \\eqref{SOL-ppty-3}, one has \r\n\\[\r\n\\phi^\\prime(0)  =\\nabla F(x^t)^Td^t = -(d^t)^T(\\nabla^2 F(x^t)+2\\epsilon_HI)d^t \r\n \\le-\\epsilon_H\\|d^t\\|^2< 0.\r\n\\]\r\nUsing these, we can observe that there exists a local minimizer $\\alpha^*\\in(0,\\theta^j)$ of $\\phi$ such that $\\phi^\\prime(\\alpha^*)=\\nabla F(x^t+\\alpha^*d^t)^Td^t=0$ and $\\phi(\\alpha^*)<\\phi(0)$, which implies that $F(x^t+\\alpha^*d^t)< F(x^t)\\le F(u^0)$. Hence, \\eqref{apx-nxt-grad} holds for $x=x^t$ and $y=x^t+\\alpha^*d^t$. Using this, $0<\\alpha^*<\\theta^j\\le1$ and $\\nabla F(x^t+\\alpha^*d^t)^Td^t=0$, we obtain\r\n\\[\r\n\\begin{array}{l}\r\n\\frac{(\\alpha^*)^2L_H^F}{2}\\|d^t\\|^3\\overset{\\eqref{apx-nxt-grad}}{\\ge}\\|d^t\\|\\|\\nabla F(x^t+\\alpha^*d^t)-\\nabla F(x^t)-\\alpha^*\\nabla^2 F(x^t)d^t\\|\\\\[6pt]\r\n\\ge(d^t)^T(\\nabla F(x^t+\\alpha^*d^t)-\\nabla F(x^t)-\\alpha^*\\nabla^2 F(x^t)d^t)=-(d^t)^T\\nabla F(x^t)-\\alpha^*(d^t)^T\\nabla^2 F(x^t)d^t\\\\[6pt]\r\n\\overset{\\eqref{SOL-ppty-3}}{=}(1-\\alpha^*)(d^t)^T(\\nabla^2 F(x^t)+2\\epsilon_HI)d^t +2 \\alpha^*\\epsilon_H\\|d^t\\|^2\\overset{\\eqref{SOL-ppty-1}}{\\ge} (1+\\alpha^*)\\epsilon_H\\|d^t\\|^2\\ge \\epsilon_H \\|d^t\\|^2,\r\n\\end{array}\r\n\\]\r\nwhich along with $d^t\\neq 0$ implies that $(\\alpha^*)^2\\ge 2\\epsilon_H(L_H^F)^{-1}\\|d^t\\|^{-1}$. Using this and $\\theta^{j}>\\alpha^*$, we conclude that \\eqref{lwbd-sol-ak} holds in this case.\r\n","sha256_utf8":"02310d06736b71f5e97854e49740e39239b023a76842efef0b78323e858cbcbe"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"下降试探点通过泰勒展开估计","source_units":[{"start":95542,"end":97010,"start_line":889,"end_line":902,"raw_tex":"Case 2) $F(x^t+\\theta^jd^t)\\le F(x^t)$. This together with $F(x^t)\\le F(u^0)$ implies that\r\n\\eqref{desc-ineq} holds for $x=x^t$ and $y=x^t+\\theta^jd^t$. Then, because $j$ violates \\eqref{ls-sol}, we obtain\r\n%By this and the fact that $j$ violates \\eqref{ls-sol}, we obtain that\r\n\\beqa\r\n&&-\\eta\\epsilon_H\\theta^{2j}\\|d^t\\|^2\\le F(x^t+\\theta^jd^t)-F(x^t)\\overset{\\eqref{desc-ineq}}{\\le} \\theta^j \\nabla F(x^t)^Td^t + \\frac{\\theta^{2j}}{2}(d^t)^T\\nabla^2 F(x^t) d^t + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3 \\nn \\\\\r\n&&\\overset{\\eqref{SOL-ppty-3}}{=}-\\theta^j (d^t)^T(\\nabla^2 F(x^t) +2\\epsilon_H I)d^t + \\frac{\\theta^{2j}}{2}(d^t)^T\\nabla^2 F(x^t) d^t + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3 \\nn \\\\\r\n&&=-\\theta^j\\left(1-\\frac{\\theta^j}{2}\\right)(d^t)^T(\\nabla^2 F(x^t) +2\\epsilon_H I)d^t - \\theta^{2j} \\epsilon_H\\|d^t\\|^2 + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3 \\nn \\\\\r\n&&\\overset{\\eqref{SOL-ppty-1}}{\\le} -\\theta^j\\left(1-\\frac{\\theta^j}{2}\\right) \\epsilon_H\\|d^t\\|^2 - \\theta^{2j}\\epsilon_H\\|d^t\\|^2 + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3 \\le -\\theta^j \\epsilon_H\\|d^t\\|^2 + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3. \\label{ineq:desc-ls-sol}\r\n\\eeqa\r\nRecall that $d^t \\neq 0$. Dividing both sides of \\eqref{ineq:desc-ls-sol} by $L_H^F\\theta^j\\|d^t\\|^3/6$ and using $\\eta, \\theta\\in(0,1)$, we obtain that \r\n\\[\r\n\\theta^{2j}\\ge 6(1-\\theta^j\\eta)\\epsilon_H(L_H^F)^{-1}\\|d^t\\|^{-1}\\ge6(1-\\eta)\\epsilon_H(L_H^F)^{-1}\\|d^t\\|^{-1}.\r\n\\] \r\n Hence, \\eqref{lwbd-sol-ak} also holds in this case.\r\n","sha256_utf8":"146b64525a077158041bede50a10eacdb1fe3b673cc3b2824b906dc36cf35acc"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"证明回溯终止并给出步长界","source_units":[{"start":97012,"end":97898,"start_line":904,"end_line":909,"raw_tex":"Combining the above two cases, we conclude that  \\eqref{lwbd-sol-ak} holds for any $j\\ge0$ that violates \\eqref{ls-sol}.\r\nBy this and $\\theta\\in(0,1)$, one can see that all $j\\ge0$ that violate \\eqref{ls-sol} must be bounded above. It then follows that the step length $\\alpha_t$ associated with \\eqref{ls-sol} is well-defined. We next prove \\eqref{lwbd-ak-sol}. Observe from the definition of $j_t$ in Algorithm~\\ref{alg:NCG} that $j=j_t-1$ violates \\eqref{ls-sol} and hence \\eqref{lwbd-sol-ak} holds for $j=j_t-1$. Then, by \\eqref{lwbd-sol-ak} with $j=j_t-1$ and $\\alpha_t = \\theta^{j_t}$, one has\r\n\\begin{equation}\\label{lwbd-sol-ak-1}\r\n\\alpha_t = \\theta^{j_t} \\ge\\sqrt{\\min\\{6(1-\\eta),2\\}\\epsilon_H(L^F_H)^{-1}} \\ \\theta\\|d^t\\|^{-1/2},\r\n\\end{equation}\r\nwhich,  along with \\eqref{SOL-ppty-2} and $\\|\\nabla F(x^t)\\|\\le U^F_g$, implies \\eqref{lwbd-ak-sol}. This proves statement (i).\r\n","sha256_utf8":"76610eb5762bd19819fe2b16441318b095693409799b9ad1f3f71525c19ad0fc"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"全步长时由新梯度推出下降量","source_units":[{"start":97900,"end":99701,"start_line":911,"end_line":935,"raw_tex":"We next prove statement (ii) %, particularly,  \\eqref{suff-desc-sol}\r\nby considering two separate cases below.\r\n\r\nCase 1) $\\alpha_t=1$. By this, one knows that \\eqref{ls-sol} holds for $j=0$. It then follows that $F(x^t+d^t)\\le F(x^t)\\le F(u^0)$, which implies that \\eqref{apx-nxt-grad} holds for $x=x^t$ and $y=x^t+d^t$. By this and \\eqref{SOL-ppty-4}, one has\r\n\\[\r\n\\begin{array}{lcl}\r\n%\\|\\nabla F(x^{t+1})\\|=\\|\\nabla F(x^{t}+d^t)-\\nabla F(x^t)+\\nabla F(x^t)\\|\\\\[4pt]\r\n\\|\\nabla F(x^{t+1})\\|=\\|\\nabla F(x^{t}+d^t)\\|\r\n&\\le&\\|\\nabla F(x^t+d^t)-\\nabla F(x^t)-\\nabla^2 F(x^t)d^t\\| \\\\[4pt]\r\n&&+\\|(\\nabla^2 F(x^t)+2\\epsilon_HI)d^t+\\nabla F(x^t)\\|+2\\epsilon_H\\|d^t\\|\\\\[4pt]\r\n&\\le& \\frac{L_H^F}{2}\\|d^t\\|^2 + \\frac{4+\\zeta}{2}\\epsilon_H\\|d^t\\|,\r\n\\end{array}\r\n\\]\r\nwhere the last inequality follows from \\eqref{apx-nxt-grad} and \\eqref{SOL-ppty-4}. Solving the above inequality for $\\|d^t\\|$ and using the fact that $\\|d^t\\|>0$, we obtain that\r\n\\[\r\n\\begin{array}{rcll}\r\n\\|d^t\\|&\\ge&\\frac{-(4+\\zeta)\\epsilon_H + \\sqrt{(4+\\zeta)^2\\epsilon_H^2+8L_H^F\\|\\nabla F(x^{t+1})\\|}}{2L_H^F}\\ \\ge\\\r\n\\frac{-(4+\\zeta)\\epsilon_H + \\sqrt{(4+\\zeta)^2\\epsilon_H^2+8L_H^F\\epsilon_H^2}}{2L_H^F}\\min\\{\\|\\nabla F(x^{t+1})\\|/\\epsilon_H^2,1\\}\\\\[6pt]\r\n&=&\\frac{4}{4+\\zeta+\\sqrt{(4+\\zeta)^2+8L_H^F}}\\min\\{\\|\\nabla F(x^{t+1})\\|/\\epsilon_H,\\epsilon_H\\},\r\n\\end{array}\r\n\\]\r\nwhere the second inequality follows from the inequality $-a+\\sqrt{a^2+bs}\\ge(-a+\\sqrt{a^2+b})\\min\\{s,1\\}$ for all $a,b,s\\ge0$,\r\n%which can be seen to hold by performing rationalization to the terms $-a+\\sqrt{a^2+b}$ and $-a+\\sqrt{a^2+bs}$.\r\nwhich can be verified by performing a rationalization to the terms $-a+\\sqrt{a^2+b}$ and $-a+\\sqrt{a^2+bs}$, respectively.\r\nBy this, $\\alpha_t=1$, \\eqref{ls-sol} and \\eqref{csol}, one can see that \\eqref{suff-desc-sol} holds.\r\n","sha256_utf8":"eb9c24c8f53073cbc733ffa6ad5589fc9d85a3cdd96f5f872b5a73dc3376b5d6"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"缩短步长时推出立方下降量","source_units":[{"start":99703,"end":100325,"start_line":937,"end_line":945,"raw_tex":"Case 2) $\\alpha_t<1$.\r\nIt then follows that $j=0$ violates \\eqref{ls-sol} and hence \\eqref{lwbd-sol-ak} holds for $j=0$.\r\nNow, letting $j=0$ in \\eqref{lwbd-sol-ak}, we obtain that $\\|d^t\\|\\ge \\min\\{6(1-\\eta),2\\}\\epsilon_H/L_H^F$,\r\n which together with \\eqref{ls-sol} and \\eqref{lwbd-sol-ak-1} implies that\r\n\\[\r\nF(x^t)-F(x^{t+1}) \\ge \\eta \\epsilon_H\\theta^{2j_t}\\|d^t\\|^2 \\ge \\eta\\frac{\\min\\{6(1-\\eta),2\\}\\epsilon_H^2}{L_H^F}\\theta^2\\|d^t\\|\\ge \\eta\\left[\\frac{\\min\\{6(1-\\eta),2\\}\\theta}{L_H^F}\\right]^2\\epsilon_H^3.\r\n\\]\r\nBy this and \\eqref{csol}, one can see that \\eqref{suff-desc-sol} also holds in this case.\r\n\\end{proof}","sha256_utf8":"ab26257acdffc53d80326a9daa2a05377b140e91410c1097d1e181e54677074b"}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","title":"两类负曲率方向的统一恒等式","source_units":[{"start":101938,"end":102921,"start_line":968,"end_line":975,"raw_tex":"\\begin{proof}\r\nObserve that $F$ is descent along the iterates (whenever well-defined) generated by Algorithm~\\ref{alg:NCG}. Using this and $x^0=u^0$, we have $F(x^t)\\le F(u^0)$.\r\nBy the assumption on $d^t$, one can see from Algorithm \\ref{alg:NCG} that\r\n%$d^t$ is given in \\eqref{dk-nc} or \\eqref{dk-2nd-nc}. Now, notice that the (negative curvature) direction $v$ returned from Algorithm~\\ref{pro:meo} satisfies $\\|v\\|=1$. This observation (for $d^t$ in \\eqref{dk-2nd-nc}) together with  Lemma~\\ref{lem:SOL-NC-ppty}(ii) (for $d^t$ in \\eqref{dk-nc}) shows that $d^t$ must satisfy\r\n$d^t$ is a negative curvature direction given in \\eqref{dk-nc} or \\eqref{dk-2nd-nc}. Also, notice that the vector $v$ returned from Algorithm~\\ref{pro:meo} satisfies $\\|v\\|=1$. By these, Lemma~\\ref{lem:SOL-NC-ppty}(ii), \\eqref{dk-nc} and \\eqref{dk-2nd-nc}, one can observe that\r\n\\begin{equation}\\label{dk-Hk-dk3}\r\n\\nabla F(x^t)^Td^t\\le 0, \\quad (d^t)^T\\nabla^2 F(x^t)d^t=-\\|d^t\\|^3<0.\r\n\\end{equation}\r\n","sha256_utf8":"2829c0da66f7fa8fa2d626114a33a32e993f6aaa29e0068af0de0871c9cd9a92"}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","title":"设置负曲率回溯下界目标","source_units":[{"start":102923,"end":103474,"start_line":977,"end_line":981,"raw_tex":"We first prove statement (i). If \\eqref{ls-nc} holds for $j=0$, then $\\alpha_t=1$, which clearly implies that $\\alpha_t\\ge\\min\\{1,\\theta/L_H^F,3(1-\\eta)\\theta/L_H^F\\}$. We now suppose that \\eqref{ls-nc} fails for $j=0$. Claim that for all $j\\ge0$ that violate \\eqref{ls-nc}, it holds that\r\n\\begin{equation}\\label{lwbd-at-nc}\r\n\\theta^j\\ge \\min\\{1/L_H^F,3(1-\\eta)/L_H^F\\}.\r\n\\end{equation}\r\nIndeed, suppose that \\eqref{ls-nc} is violated by some $j \\ge 0$. We now show that \\eqref{lwbd-at-nc} holds for such $j$ by considering two separate cases below.\r\n","sha256_utf8":"e28212789cdbf1d8aad664685fb96e05065963caaa223c973eecd123f413513b"}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","title":"上升试探点利用二阶极小条件","source_units":[{"start":103476,"end":104883,"start_line":983,"end_line":993,"raw_tex":"Case 1) $F(x^t+\\theta^jd^t)>F(x^t)$. Let $\\phi(\\alpha)=F(x^t+\\alpha d^t)$. Then $\\phi(\\theta^j)>\\phi(0)$. Also, by  \\eqref{dk-Hk-dk3}, one has \r\n\\[\r\n\\phi^\\prime(0)=\\nabla F(x^t)^Td^t\\le 0, \\quad \\phi^{\\prime\\prime}(0)=(d^t)^T\\nabla^2 F(x^t)d^t<0.\r\n\\]\r\n Using these, we can observe that there exists a local minimizer $\\alpha^*\\in(0,\\theta^j)$ of $\\phi$ such that $\\phi(\\alpha^*)<\\phi(0)$, namely,  $F(x^t+\\alpha^*d^t)< F(x^t)$. By the second-order optimality condition of $\\phi$ at $\\alpha^*$, one has $\\phi^{\\prime\\prime}(\\alpha^*)=(d^t)^T\\nabla^2 F(x^t+\\alpha^*d^t)d^t\\ge 0$. Since $F(x^t+\\alpha^*d^t)<F(x^t)\\le F(u^0)$, it follows that \\eqref{F-Hess-Lip} holds for $x=x^t$ and $y=x^t+\\alpha^*d^t$. Using this, the second relation in \\eqref{dk-Hk-dk3} and $(d^t)^T\\nabla^2 F(x^t+\\alpha^*d^t)d^t\\ge0$, we obtain that\r\nin \\eqref{dk-Hk-dk3} and $(d^t)^T\\nabla^2 F(x^t+\\alpha^*d^t)d^t\\ge0$, we obtain that\r\n\\beqa\r\nL_H^F\\alpha^*\\|d^t\\|^3 &\\overset{\\eqref{F-Hess-Lip}}{\\ge}&\\|d^t\\|^2\\|\\nabla^2 F(x^t+\\alpha^*d^t)-\\nabla^2 F(x^t)\\| \\ge (d^t)^T(\\nabla^2 F(x^t+\\alpha^*d^t)-\\nabla^2 F(x^t))d^t \\nn \\\\\r\n&\\ge& -(d^t)^T\\nabla^2 F(x^t)d^t=\\|d^t\\|^3. \\label{lwbd-dt-nc-j1}\r\n\\eeqa\r\nRecall from \\eqref{dk-Hk-dk3} that $d^t\\neq 0$. It then follows from \\eqref{lwbd-dt-nc-j1} that $\\alpha^*\\ge 1/L_H^F$, which along with $\\theta^j>\\alpha^*$ implies that $ \\theta^j> 1/L_H^F$. Hence, \\eqref{lwbd-at-nc} holds in this case.\r\n","sha256_utf8":"e13635e141f637ea6c9f7a6ec6076a3e91dc61031142909786722fe0f8710966"}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","title":"下降试探点利用三阶余项","source_units":[{"start":104885,"end":105649,"start_line":995,"end_line":1002,"raw_tex":"Case 2) $F(x^t+\\theta^jd^t)\\le F(x^t)$. It follows from this and $F(x^t)\\le F(u^0)$ that \\eqref{desc-ineq} holds for $x=x^t$ and $y=x^t+\\theta^jd^t$. By this and the fact that $j$ violates \\eqref{ls-nc}, one has\r\n\\begin{equation*}\\label{lwbd-dt-nc}\r\n\\begin{array}{lcl}\r\n-\\frac{\\eta}{2}\\theta^{2j}\\|d^t\\|^3 &\\le& F(x^t+\\theta^j d^t) - F(x^t) \\overset{\\eqref{desc-ineq}}{\\le} \\theta^j \\nabla F(x^t)^Td^t + \\frac{\\theta^{2j}}{2}(d^t)^T\\nabla^2 F(x^t) d^t + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3\\\\[6pt]\r\n&\\overset{\\eqref{dk-Hk-dk3}}{\\le}& -\\frac{\\theta^{2j}}{2}\\|d^t\\|^3 + \\frac{L_H^F}{6}\\theta^{3j}\\|d^t\\|^3,\r\n\\end{array}\r\n\\end{equation*}\r\nwhich together with $d^t\\neq 0$ implies that $\\theta^j\\ge 3(1-\\eta)/L_H^F$. Hence, \\eqref{lwbd-at-nc} also holds  in this case.\r\n","sha256_utf8":"64654dab754a26bf46a48e90f39fa935d68cb9096a317251eb1311bf35dc2b0e"}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","title":"负曲率回溯终止与步长界","source_units":[{"start":105651,"end":106420,"start_line":1004,"end_line":1004,"raw_tex":"Combining the above two cases, we conclude that \\eqref{lwbd-at-nc} holds for any $j\\ge0$ that violates \\eqref{ls-nc}. By this and $\\theta\\in(0,1)$, one can see that all $j\\ge0$ that violate \\eqref{ls-nc} must be bounded above. It then follows that the step length $\\alpha_t$ associated with \\eqref{ls-nc} is well-defined. We next derive a lower bound for $\\alpha_t$. Notice from the definition of $j_t$ in Algorithm~\\ref{alg:NCG} that $j=j_t-1$ violates \\eqref{ls-nc} and hence \\eqref{lwbd-at-nc} holds for $j=j_t-1$. Then, by \\eqref{lwbd-at-nc} with $j=j_t-1$ and $\\alpha_t = \\theta^{j_t}$, one has $\\alpha_t=\\theta^{j_t} \\ge \\min\\{\\theta/L_H^F,3(1-\\eta)\\theta/L_H^F\\}$, which immediately yields $\\alpha_t\\ge \\min\\{1,\\theta/L_H^F,3(1-\\eta)\\theta/L_H^F\\}$ as desired.\r\n","sha256_utf8":"c2e5658a99a3a4a23ef81ccbcef8a083e281f684abec859292156f8787b1bd3c"}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","title":"共轭梯度负曲率步的下降量","source_units":[{"start":106422,"end":106811,"start_line":1006,"end_line":1010,"raw_tex":"We next prove statement (ii) by considering two separate cases below.\r\n\r\nCase 1) $d^t$ results from the output $d$ of Algorithm~\\ref{alg:capped-CG} with d$\\_$type=NC. It then follows from  \\eqref{NC-ppty} that $\\|d^t\\|\\ge\\epsilon_H$. \r\n%{ This together with item (i) and \\eqref{ls-nc} implies (ii).}\r\nThis together with  \\eqref{ls-nc} and statement (i) implies that statement (ii) holds.\r\n","sha256_utf8":"eda190b69bc1053f86202afbeebc4b2be1880458315120296587f4b8c7b1882c"}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","title":"朗佐斯负曲率步的下降量","source_units":[{"start":106813,"end":107144,"start_line":1012,"end_line":1014,"raw_tex":"Case 2) $d^t$ results from the output $v$ of Algorithm~\\ref{pro:meo}. Notice from\r\nAlgorithm~\\ref{pro:meo} that $\\|v\\|=1$ and $v^T\\nabla^2F(x^t)v\\le-\\epsilon_H/2$, which along with \\eqref{dk-2nd-nc} yields $\\|d^t\\|\\ge\\epsilon_H/2$. By this, \\eqref{ls-nc} and statement (i), one can see that statement (ii) again holds.\r\n\\end{proof}","sha256_utf8":"23c05ac91a69008ce6537a74eaa4994bb98374eb0e6e020a798ba5e33c947b21"}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","title":"配方得到目标函数上界","source_units":[{"start":113358,"end":113824,"start_line":1077,"end_line":1086,"raw_tex":"\\begin{proof}\r\n(i) It follows from\r\n%{ the definition of { $\\widetilde{\\cL}$} in} \\eqref{tL-tc} and \\eqref{AL-upperbound}\r\n \\eqref{AL-upperbound} and the definition of $\\widetilde{\\cL}$ in \\eqref{tL-tc}\r\nthat\r\n\\[\r\n\\textstyle\r\n\\fh\\ge f(x)+\\lambda^T \\tilde{c}(x)+\\frac{\\rho}{2}\\|\\tilde{c}(x)\\|^2 = f(x)+\\frac{\\rho}{2}\\left\\|\\tilde{c}(x)+\\frac{\\lambda}{\\rho}\\right\\|^2-\\frac{\\|\\lambda\\|^2}{2\\rho} \\ge f(x)-\\frac{\\|\\lambda\\|^2}{2\\rho}.\r\n\\]\r\nHence, statement (i) holds.\r\n","sha256_utf8":"006bbfe1c6bdf4e44da2157a00f87bd42a7e35896ad074f84b4a2e49f1c3766e"}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","title":"配方得到扰动约束范数界","source_units":[{"start":113826,"end":114638,"start_line":1088,"end_line":1096,"raw_tex":"(ii) In view of \\eqref{ineq:p-lowbd} and \\eqref{AL-upperbound}, one has\r\n\\begin{equation*}\\label{ineq:bound-cx-step1}\r\n\\begin{array}{l}\r\n\\fh\\overset{\\eqref{AL-upperbound}}{\\ge}f(x)+\\lambda^T\\tilde{c}(x)+\\frac{\\rho}{2}\\|\\tilde{c}(x)\\|^2=f(x)+\\gamma\\|\\tilde{c}(x)\\|^2\r\n+\\frac{\\rho-2\\gamma}{2}\\left\\|\\tilde{c}(x)+\\frac{\\lambda}{\\rho-2\\gamma}\\right\\|^2-\\frac{\\|\\lambda\\|^2}{2(\\rho-2\\gamma)}\\\\[2pt]\r\n\\ \\ \\ \\ \\overset{\\eqref{ineq:p-lowbd}}{\\ge}\\fl- \\gamma+\\frac{\\rho-2\\gamma}{2}\\left\\|\\tilde{c}(x)+\\frac{\\lambda}{\\rho-2\\gamma}\\right\\|^2-\\frac{\\|\\lambda\\|^2}{2(\\rho-2\\gamma)}.\r\n\\end{array}\r\n\\end{equation*}\r\nIt then follows that $\\left\\|\\tilde{c}(x)+\\frac{\\lambda}{\\rho-2\\gamma}\\right\\|\\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho-2\\gamma}+\\frac{\\|\\lambda\\|^2}{(\\rho-2\\gamma)^2}}$,  which implies that statement (ii) holds.\r\n","sha256_utf8":"4198d2bd1d29a54caf1ff2f09933718526d3145e47c3508ae2d46660b6890ff5"}]},{"id":"S080","semantic_type":"proof_step","kind":"proof_step","title":"罚参数阈值控制目标函数","source_units":[{"start":114640,"end":114738,"start_line":1098,"end_line":1098,"raw_tex":"(iii) Statement (iii) immediately follows from statement (i) and $\\rho\\ge\\|\\lambda\\|^2/(2\\tdf)$.\r\n","sha256_utf8":"d30ad353dfab4cc48c2e8a2402c4ef10de839e1525213db1b0aff16b0fd5641a"}]},{"id":"S081","semantic_type":"proof_step","kind":"proof_step","title":"解二次不等式控制约束违反","source_units":[{"start":114740,"end":115274,"start_line":1100,"end_line":1109,"raw_tex":"(iv) Suppose that \\eqref{bound-rho-deltac} holds. Multiplying both sides of \\eqref{bound-rho-deltac} by $\\tdc^{2}$ and rearranging the terms, we have\r\n\\[\r\n(\\rho-2\\gamma)\\tdc^2-2\\|\\lambda\\|\\tdc-2(\\fh-\\fl+\\gamma)\\ge0.\r\n\\]\r\nRecall that $\\rho>2\\gamma$ and $\\tdc>0$. Solving this inequality for $\\tdc$ yields \r\n\\[\r\n\\tdc\\ge\\sqrt{2(\\fh-\\fl+\\gamma)/(\\rho-2\\gamma)+\\|\\lambda\\|^2/(\\rho-2\\gamma)^2}+\\|\\lambda\\|/(\\rho-2\\gamma),\r\n\\]\r\nwhich along with statement (ii) implies that $\\|\\tilde{c}(x)\\|\\le\\tdc$. Hence, statement (iv) holds.\r\n\\end{proof}","sha256_utf8":"b383cd44bbaf4d8a80d947dfa7159a116d433339c6ccf88368216d70d89e0eba"}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","title":"低水平集包含于统一紧集","source_units":[{"start":115361,"end":117173,"start_line":1113,"end_line":1124,"raw_tex":"\\begin{proof}[Proof of Lemma \\ref{lem:level-set-augmented-lagrangian-func}]\r\n(i) Let $x$ be any point such that $\\widetilde{\\cL}(x,\\lambda^k;\\rho_k)\\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)$. It then follows from \\eqref{L-xinit} that $\\widetilde{\\cL}(x,\\lambda^k;\\rho_k) \\le \\fh$. By this, $\\|\\lambda^k\\|\\le\\Lambda$, $\\rho_k \\ge \\rho_0>2\\gamma$, $\\delta_{f,1} \\le \\bdelta_f$, $\\delta_{c,1} \\le \\bdelta_c$, and Lemma \\ref{tech-1} with $(\\lambda,\\rho)=(\\lambda^k,\\rho_k)$,  one has \r\n\\begin{align}\r\nf(x) & \\le \\fh+ \\|\\lambda^k\\|^2/(2\\rho_k)\\le \\fh+ \\Lambda^2/(2\\rho_0)=\\fh+\\delta_{f,1} \\le \\fh+\\bdelta_f, \\nonumber \\\\\r\n\\textstyle\\|\\tilde{c}(x)\\|&\\textstyle\\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_k-2\\gamma}+\\frac{\\|\\lambda^k\\|^2}{(\\rho_k-2\\gamma)^2}}+\\frac{\\|\\lambda^k\\|}{\\rho_k-2\\gamma}\\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_0-2\\gamma}+\\frac{\\Lambda^2}{(\\rho_0-2\\gamma)^2}}+\\frac{\\Lambda}{\\rho_0-2\\gamma}=\\delta_{c,1} \\le \\bdelta_c. \\label{ineq:feasible-bd}\r\n\\end{align}\r\n%\\begin{eqnarray}\r\n%&f(x)\\le \\fh+\\frac{\\|\\lambda^k\\|^2}{2\\rho_k}\\le \\fh+\\frac{\\Lambda^2}{2\\rho_0}=\\fh+\\delta_{f,1} \\le \\fh+\\bdelta_f,\\nn \\\\\r\n%&\\|\\tilde{c}(x)\\|\\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_k-2\\gamma}+\\frac{\\|\\lambda^k\\|^2}{(\\rho_k-2\\gamma)^2}}+\\frac{\\|\\lambda^k\\|}{\\rho_k-2\\gamma}\\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_0-2\\gamma}+\\frac{\\Lambda^2}{(\\rho_0-2\\gamma)^2}}+\\frac{\\Lambda}{\\rho_0-2\\gamma}=\\delta_{c,1} \\le \\bdelta_c.\\label{ineq:feasible-bd}\r\n%\\end{eqnarray}\r\nAlso, recall from the definition of $\\tilde{c}$ in \\eqref{model:equa-cnstr-pert}\r\nand $\\|c(z_{\\epsilon_1})\\|\\le1$ that $\\|c(x)\\|\\le1+\\|\\tilde{c}(x)\\|$. This together with the above inequalities and \\eqref{nearly-feas-level-set} implies $x\\in\\cS(\\delta_{f},\\delta_{c})$. Hence, statement (i) of Lemma \\ref{lem:level-set-augmented-lagrangian-func} holds.\r\n","sha256_utf8":"025c781275110703ab23b1c7238934ed529ac2ce3e9a70fa2a7d38337d170bfc"}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","title":"在低水平集上建立全局下界","source_units":[{"start":117175,"end":118533,"start_line":1126,"end_line":1139,"raw_tex":"(ii) Note that $\\inf\\limits_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)=\\inf\\limits_{x\\in\\bR^n}\\{\\widetilde{\\cL}(x,\\lambda^k;\\rho_k):\\widetilde{\\cL}(x,\\lambda^k;\\rho_k) \\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\}$. Consequently, to prove statement (ii) of Lemma \\ref{lem:level-set-augmented-lagrangian-func}, it suffices to show that\r\n\\begin{equation} \\label{L-lwbd}\r\n\\inf\\limits_{x\\in\\bR^n}\\{\\widetilde{\\cL}(x,\\lambda^k;\\rho_k): \\widetilde{\\cL}(x,\\lambda^k;\\rho_k) \\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)\\} \\ge\\fl - \\gamma-\\Lambda\\bdelta_c.\r\n\\end{equation}\r\nTo this end, let $x$ be any point satisfying $\\widetilde{\\cL}(x,\\lambda^k;\\rho_k) \\le\\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)$. We then know from \\eqref{ineq:feasible-bd} that $\\|\\tilde{c}(x)\\|\\le\\bdelta_c$. By this, $\\|\\lambda^k\\|\\le\\Lambda$, $\\rho_k>2\\gamma$, and \\eqref{ineq:p-lowbd}, one has\r\n\\begin{equation*}\\label{ineq:lowerbound-ALfunc} \r\n\\begin{array}{l}\r\n\\widetilde{\\cL}(x,\\lambda^k;\\rho_k) %= f(x)+(\\lambda^k)^T\\tilde{c}(x)+\\frac{\\rho_k}{2}\\|\\tilde{c}(x)\\|^2\\\\[4pt]\r\n=f(x)+\\gamma\\|\\tilde{c}(x)\\|^2+(\\lambda^k)^T\\tilde{c}(x)+\\frac{\\rho_k-2\\gamma}{2}\\|\\tilde{c}(x)\\|^2\\\\[4pt]\r\n\\ge f(x)+\\gamma\\|\\tilde{c}(x)\\|^2 - \\Lambda\\|\\tilde{c}(x)\\| \\ge\\fl - \\gamma-\\Lambda\\bdelta_c,\r\n\\end{array}\r\n\\end{equation*}\r\nand hence \\eqref{L-lwbd} holds as desired.\r\n\\end{proof}","sha256_utf8":"54edc0ea2062d7d123025eeb08608b1aa1e0d888a89e2f6e85ef8146402bb733"}]},{"id":"S084","semantic_type":"proof_step","kind":"proof_step","title":"乘子更新转移一阶残差","source_units":[{"start":118585,"end":119285,"start_line":1143,"end_line":1148,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{thm:output-alg1}]\r\nSuppose that Algorithm \\ref{alg:2nd-order-AL-nonconvex} terminates at some iteration $k$, that is, $\\tau_k^g\\le\\epsilon_1$, $\\tau_k^H\\le\\epsilon_2$, and $\\|c(x^{k+1})\\|\\le\\epsilon_1$ hold. Then, by $\\tau_k^g\\le\\epsilon_1$, $\\tilde{\\lambda}^{k+1}=\\lambda^k+\\rho_k\\tilde{c}(x^{k+1})$, $\\nabla \\tilde{c}=\\nabla c$ and the second relation in \\eqref{algstop:1st-order}, one has \r\n \\begin{align*}\r\n\\|\\nabla f(x^{k+1})+\\nabla c(x^{k+1})\\tilde{\\lambda}^{k+1}\\| &=\\|\\nabla f(x^{k+1})+\\nabla \\tilde{c}(x^{k+1})(\\lambda^k+\\rho_k \\tilde{c}(x^{k+1}))\\| \\\\ \r\n&=\\|\\nabla_x\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k)\\|\\le \\tau_k^g \\le \\epsilon_1.\r\n\\end{align*} \r\n","sha256_utf8":"84dc4f7347079a2ef3b5c59e72532143a65f2c5d4588b9a024f0680f8f78c7b0"}]},{"id":"S085","semantic_type":"proof_step","kind":"proof_step","title":"展开输出处的二阶概率界","source_units":[{"start":119285,"end":120357,"start_line":1149,"end_line":1153,"raw_tex":" Hence, $(x^{k+1},\\tilde{\\lambda}^{k+1})$ satisfies the first relation in \\eqref{optcond:1st-equa-cnstr}. In addition, by \\eqref{algstop:2nd-order} and $\\tau_k^H\\le\\epsilon_2$, one can show that $\\lambda_{\\min}(\\nabla^2_{xx}\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k))\\ge-\\epsilon_2$ with probability at least $1-\\delta$, which leads to $d^T\\nabla^2_{xx}\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k)d \\ge -\\epsilon_2\\|d\\|^2$ for all $d\\in\\bR^n$ with probability at least $1-\\delta$.  Using this, $\\tilde{\\lambda}^{k+1}=\\lambda^k+\\rho_k\\tilde{c}(x^{k+1})$, $\\nabla \\tilde{c} = \\nabla c$, and $\\nabla^2 \\tilde{c}_i=\\nabla^2 c_i$ for $1\\le i\\le m$, we see that with probability at least $1-\\delta$, it holds that \r\n%$d^T (\\nabla^2 f(x^{k+1})+\\sum_{i=1}^m\\tilde{\\lambda}_i^{k+1}\\nabla^2c_i(x^{k+1})+\\rho_k\\nabla c(x^{k+1})\\nabla c(x^{k+1})^T )d\\ge -\\epsilon_2\\|d\\|^2$ for all $d\\in\\bR^n$},\r\n\\[\r\nd^T\\left(\\nabla^2 f(x^{k+1})+\\sum_{i=1}^m\\tilde{\\lambda}_i^{k+1}\\nabla^2c_i(x^{k+1})+\\rho_k\\nabla c(x^{k+1})\\nabla c(x^{k+1})^T\\right)d\\ge -\\epsilon_2\\|d\\|^2 \\ \\  \\forall d\\in\\bR^n,\r\n\\]\r\n","sha256_utf8":"7ea436be63fd02f1dc07a5c4443637fd3e44a5cc942d70f3cd7024d01cc60bdd"}]},{"id":"S086","semantic_type":"proof_step","kind":"proof_step","title":"限制到切空间得到二阶驻点","source_units":[{"start":120357,"end":120991,"start_line":1154,"end_line":1155,"raw_tex":"which implies $d^T(\\nabla^2 f(x^{k+1})+\\sum_{i=1}^m\\tilde{\\lambda}_i^{k+1}\\nabla^2 c_i(x^{k+1}))d\\ge -\\epsilon_2\\|d\\|^2$ for all $d\\in \\cC(x^{k+1})$, where $\\cC(\\cdot)$ is defined in \\eqref{def:critical-cone}. Hence, $(x^{k+1},\\tilde{\\lambda}^{k+1})$ satisfies \\eqref{optcond:2nd-equa-cnstr} with probability at least $1-\\delta$. Combining these with $\\|c(x^{k+1})\\|\\le\\epsilon_1$, we conclude that $x^{k+1}$ is a deterministic $\\epsilon_1$-FOSP of \\eqref{model:equa-cnstr} and an $(\\epsilon_1,\\epsilon_2)$-SOSP of \\eqref{model:equa-cnstr} with probability at least $1-\\delta$. Hence, Theorem \\ref{thm:output-alg1} holds.\r\n\\end{proof}","sha256_utf8":"998176705f40d4b9298b4abb7fcad236ae2b9333212d5f60bb93d20b3ae39998"}]},{"id":"S087","semantic_type":"proof_step","kind":"proof_step","title":"同步阈值之前的罚参数界","source_units":[{"start":121051,"end":121915,"start_line":1159,"end_line":1170,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{thm:out-itr-cmplxity-1}]\r\nIt follows from \\eqref{def:delta0c-rhobar1} that ${\\rho}_{\\epsilon_1}\\ge 2\\rho_0$.\r\nBy this, one has\r\n\\begin{equation}\\label{T-epsilong}\r\nK_{\\epsilon_1} \\ \\overset{\\eqref{T-epsilon-g}}{=} \\ \\left\\lceil\\log\\epsilon_1/\\log \\omega_1 \\right\\rceil \\ \\overset{\\eqref{omega-tolerance}}{=} \\ \\left\\lceil\\log{2}/\\log{r}\\right\\rceil  \\\r\n\\le \\ \\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})/\\log{r}+1.\r\n\\end{equation}\r\nNotice that $\\{\\rho_k\\}$ is either unchanged or increased by a ratio $r$ as $k$ increases.\r\nBy this fact and \\eqref{T-epsilong}, we see that\r\n\\begin{equation}\\label{rho:upper-bound-Tepsilong}\r\n\\max_{0 \\le k \\le K_{\\epsilon_1}} \\rho_k \\le{r}^{K_{\\epsilon_1}}\\rho_0\\overset{\\eqref{T-epsilong}}{\\le}{r}^{\\frac{\\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})}{\\log{r}}+1}\\rho_0={r}{\\rho}_{\\epsilon_1}.\r\n\\end{equation}\r\n","sha256_utf8":"76e17d7dd58da9a94815c0df778a0a15b4baffdd05bbfc43d563453d1c647f45"}]},{"id":"S088","semantic_type":"proof_step","kind":"proof_step","title":"约束违反量的统一估计","source_units":[{"start":121915,"end":122779,"start_line":1171,"end_line":1179,"raw_tex":"In addition, notice that $\\rho_k>2\\gamma$ and $\\|\\lambda^k\\|\\le \\Lambda$. Using these, \\eqref{hbd}, the first relation in \\eqref{algstop:1st-order}, and Lemma \\ref{tech-1}(ii) with $(x,\\lambda,\\rho)=(x^{k+1},\\lambda^k,\\rho_k)$,  we obtain that\r\n\\begin{equation}\\label{bd-ck}\\textstyle\r\n\\|\\tilde{c}(x^{k+1})\\| \\le\\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_k-2\\gamma}+\\frac{\\|\\lambda^k\\|^2}{(\\rho_k-2\\gamma)^2}}+\\frac{\\|\\lambda^k\\|}{\\rho_k-2\\gamma}\\le  \\sqrt{\\frac{2(\\fh-\\fl+\\gamma)}{\\rho_k-2\\gamma}+\\frac{\\Lambda^2}{(\\rho_k-2\\gamma)^2}}+\\frac{\\Lambda}{\\rho_k-2\\gamma}.\r\n\\end{equation}\r\nAlso, we observe from $\\|c(z_{\\epsilon_1})\\|\\le\\epsilon_1/2$ and the definition of $\\tilde{c}$ in\r\n\\eqref{model:equa-cnstr-pert} that\r\n\\begin{equation}\\label{bd-cxk}\r\n\\|c(x^{k+1})\\|\\le\\|\\tilde{c}(x^{k+1})\\|+\\|c(z_{\\epsilon_1})\\|\\le \\|\\tilde{c}(x^{k+1})\\|+\\epsilon_1/2.\r\n\\end{equation}\r\n","sha256_utf8":"488c3c4614551d3ef7dace0264ada6df33430c2f46fbdac15a912ef4486a0fa0"}]},{"id":"S089","semantic_type":"proof_step","kind":"proof_step","title":"反证有限终止并识别终止指标","source_units":[{"start":122781,"end":124154,"start_line":1181,"end_line":1183,"raw_tex":"We now prove that $\\overline{K}_{\\epsilon_1}$ is finite. Suppose for contradiction that $\\overline{K}_{\\epsilon_1}$ is infinite. It then follows from this and \\eqref{number-outer-iteration} that $\\|c(x^{k+1})\\|>\\epsilon_1$ for all $k\\ge K_{\\epsilon_1}$, which along with \\eqref{bd-cxk} implies that $\\|\\tilde{c}(x^{k+1})\\|>\\epsilon_1/2$ for all $k\\ge K_{\\epsilon_1}$. It then follows that $\\|\\tilde{c}(x^{k+1})\\|>\\alpha\\|\\tilde{c}(x^k)\\|$ must hold for infinitely many $k$'s. Using this and the update scheme on $\\{\\rho_k\\}$, we deduce that $\\rho_{k+1}={r}\\rho_k$ holds for infinitely many $k$'s, which together with the monotonicity of $\\{\\rho_k\\}$ implies that\r\n$\\rho_k\\to\\infty$ as $k\\to\\infty$. By this and \\eqref{bd-ck}, one can see that  $\\|\\tilde{c}(x^{k+1})\\|\\to0$ as $k\\to\\infty$, which contradicts the fact that $\\|\\tilde{c}(x^{k+1})\\|>\\epsilon_1/2$ holds for all $k\\ge K_{\\epsilon_1}$. Hence, $\\overline{K}_{\\epsilon_1}$ is finite.\r\nIn addition, notice from \\eqref{omega-tolerance}, \\eqref{T-epsilon-g} and \\eqref{epsw1-epsw2} that $(\\tau_k^g,\\tau_k^H)=(\\epsilon_1,\\epsilon_2)$ for all $k\\ge K_{\\epsilon_1}$. This along with the termination criterion of Algorithm \\ref{alg:2nd-order-AL-nonconvex}  and the definition of $\\overline{K}_{\\epsilon_1}$ implies that Algorithm \\ref{alg:2nd-order-AL-nonconvex} must terminate at iteration $\\overline{K}_{\\epsilon_1}$.\r\n","sha256_utf8":"650f07ac5b545a3f764503a95677b19a67acb72ebcf87accd59c3a41963dcc7f"}]},{"id":"S090","semantic_type":"proof_step","kind":"proof_step","title":"同步阈值立即可行情形","source_units":[{"start":124156,"end":124707,"start_line":1185,"end_line":1187,"raw_tex":"We next show that \\eqref{outer-iteration-cmplxty} and $\\rho_k\\le{r}{\\rho}_{\\epsilon_1}$ hold for $0 \\le k \\le \\overline{K}_{\\epsilon_1}$ by considering two separate cases below.\r\n\r\nCase 1) $\\|c(x^{K_{\\epsilon_1}+1})\\|\\le\\epsilon_1$.  By this and \\eqref{number-outer-iteration}, one can see that  $\\overline{K}_{\\epsilon_1}=K_{\\epsilon_1}$, which together with \\eqref{T-epsilong} and \\eqref{rho:upper-bound-Tepsilong} implies that \\eqref{outer-iteration-cmplxty} and $\\rho_k\\le{r}{\\rho}_{\\epsilon_1}$ hold for $0 \\le k \\le \\overline{K}_{\\epsilon_1}$.\r\n","sha256_utf8":"01eefd9f30c5647cc368c95aee43c650b06c2ad58a9e21238917614bf683521a"}]},{"id":"S091","semantic_type":"proof_step","kind":"proof_step","title":"尚不可行时限制罚参数增长","source_units":[{"start":124709,"end":126014,"start_line":1189,"end_line":1201,"raw_tex":"Case 2) $\\|c(x^{K_{\\epsilon_1}+1})\\|>\\epsilon_1$. By this and \\eqref{number-outer-iteration}, one can observe that $\\overline{K}_{\\epsilon_1}>K_{\\epsilon_1}$  and also $\\|c(x^{k+1})\\|>\\epsilon_1$ for all $K_{\\epsilon_1} \\le k \\le \\overline{K}_{\\epsilon_1}-1$, which together with \\eqref{bd-cxk} implies\r\n\\begin{equation}\\label{lbd-tc}\r\n\\|\\tilde{c}(x^{k+1})\\|>\\epsilon_1/2,\\quad \\forall K_{\\epsilon_1} \\le k \\le \\overline{K}_{\\epsilon_1}-1.\r\n\\end{equation}\r\nIt then follows from $\\|\\lambda^k\\|\\le\\Lambda$, \\eqref{hbd}, the first relation in \\eqref{algstop:1st-order}, and Lemma \\ref{tech-1}(iv) with $(x,\\lambda,\\rho,\\tdc)=(x^{k+1},\\lambda^k,\\rho_k,\\epsilon_1/2)$ that\r\n\\begin{equation}\\label{bound-rhok}\r\n\\begin{aligned}\r\n\\rho_k &< 8(\\fh-\\fl+\\gamma)\\epsilon_1^{-2}+4\\|\\lambda^k\\|\\epsilon_1^{-1}+2\\gamma\\\\\r\n&\\le 8(\\fh-\\fl+\\gamma)\\epsilon_1^{-2}+4\\Lambda\\epsilon_1^{-1}+2\\gamma\\overset{\\eqref{def:delta0c-rhobar1}}{\\le}{\\rho}_{\\epsilon_1},\\quad\r\n\\forall K_{\\epsilon_1} \\le k \\le \\overline{K}_{\\epsilon_1}-1.\r\n\\end{aligned}\r\n\\end{equation}\r\nCombining this relation, \\eqref{rho:upper-bound-Tepsilong}, and the fact $\\rho_{\\overline{K}_{\\epsilon_1}}\\le {r}\\rho_{\\overline{K}_{\\epsilon_1}-1}$, we conclude that  $\\rho_k\\le{r}{\\rho}_{\\epsilon_1}$ holds for $0 \\le k\\le \\overline{K}_{\\epsilon_1}$. It remains to\r\n","sha256_utf8":"59b8cfed604ecb030cd0dc3f0651929b6e0b5aa9a6ac0487c113fc623bab5d66"}]},{"id":"S092","semantic_type":"proof_step","kind":"proof_step","title":"罚参数增长事件的次数界","source_units":[{"start":126014,"end":126629,"start_line":1202,"end_line":1213,"raw_tex":"show that \\eqref{outer-iteration-cmplxty} holds.  To this end, let \r\n\\[\r\n\\bK =\\{k:\\rho_{k+1}={r}\\rho_{k}, K_{\\epsilon_1}\\le k\\le \\overline{K}_{\\epsilon_1}-2\\}.\r\n\\]\r\nIt follows from \\eqref{bound-rhok} and the update scheme of $\\rho_k$ that \r\n\\[\r\n{r}^{|\\bK|}\\rho_{K_{\\epsilon_1}}=\\max_{K_{\\epsilon_1}\\le k \\le \\overline{K}_{\\epsilon_1}-1}\\{\\rho_k\\}\\le {\\rho}_{\\epsilon_1},\r\n\\]\r\nwhich together with $\\rho_{K_{\\epsilon_1}}\\ge\\rho_0$ implies that\r\n\\begin{equation}\\label{bound-cU}\r\n|\\bK|\\le \\log({\\rho}_{\\epsilon_1}\\rho_{K_{\\epsilon_1}}^{-1})/ \\log{r} \\le \\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})/ \\log{r}.\r\n\\end{equation}\r\n","sha256_utf8":"25faa94f5d9d0a97d00f0c4ed9098638517b7985e8edc5dd8ec93a461d4c4a73"}]},{"id":"S093","semantic_type":"proof_step","kind":"proof_step","title":"固定罚参数区间的几何收缩","source_units":[{"start":126629,"end":127604,"start_line":1214,"end_line":1221,"raw_tex":"Let $\\{k_1,k_2,\\ldots,k_{|\\bK|}\\}$ denote all the elements of $\\mathcal{\\bK}$ arranged in ascending order, and let $k_0=K_{\\epsilon_1}$ and $k_{|\\bK|+1}=\\overline{K}_{\\epsilon_1}-1$. We next derive an upper bound for $k_{j+1}-k_{j}$ for $j=0,1,\\ldots,|\\bK|$. By the definition of $\\bK$, one can observe that $\\rho_{k}=\\rho_{k'}$ for $k_j< k,k'\\le k_{j+1}$. Using this and the update scheme of $\\rho_k$, we deduce that\r\n\\begin{equation}\\label{ineq:sequence-cxk}\r\n\\|\\tilde{c}(x^{k+1})\\|\\le\\alpha\\|\\tilde{c}(x^{k})\\|,\\quad \\forall k_j<k< k_{j+1}.\r\n\\end{equation}\r\nOn the other hand, by \\eqref{def:delta0c-rhobar1xxx}, \\eqref{bd-ck} and $\\rho_k\\ge\\rho_0$, one has $\\|\\tilde{c}(x^{k+1})\\|\\le \\delta_{c,1}$ for $0 \\le k \\le \\overline{K}_{\\epsilon_1}$. By this and \\eqref{lbd-tc}, one can see that\r\n\\begin{equation}\\label{ineq:sequence-cxk2222}\r\n\\epsilon_1/2<\\|\\tilde{c}(x^{k+1})\\|\\le\\delta_{c,1},\\quad \\forall K_{\\epsilon_1} \\le k \\le \\overline{K}_{\\epsilon_1}-1.\r\n\\end{equation}\r\n","sha256_utf8":"9b83c01024cc1327463c9c82ad5fe8c27ab6fa54c2e8a2ff6132d38d1e390ae1"}]},{"id":"S094","semantic_type":"proof_step","kind":"proof_step","title":"固定罚参数区间的长度界","source_units":[{"start":127604,"end":128253,"start_line":1222,"end_line":1229,"raw_tex":"Now, note that either $k_{j+1} - k_j = 1$ or $k_{j+1} - k_j > 1$. In the latter case, we can apply \\eqref{ineq:sequence-cxk} with $k = k_{j+1}-1,\\ldots,k_j+1$ together with \\eqref{ineq:sequence-cxk2222} to deduce that\r\n\\[\r\n\\epsilon_1/2<\\|\\tilde{c}(x^{k_{j+1}})\\|\\le\\alpha \\|\\tilde{c}(x^{k_{j+1}-1})\\|\\le\\cdots\\le \\alpha^{k_{j+1}-k_j-1}\\|\\tilde{c}(x^{k_{j}+1})\\|\\le \\alpha^{k_{j+1}-k_j-1}\\delta_{c,1},\\quad \\forall j=0,1,\\ldots,|\\bK|.\r\n\\]\r\nCombining these two cases, we have \r\n\\begin{equation}\\label{ineq:subsequence-rho-same}\r\nk_{j+1}-k_{j}\\le |\\log(\\epsilon_1(2\\delta_{c,1})^{-1}))/\\log \\alpha|+1,\\quad \\forall j=0,1,\\ldots,|\\bK|.\r\n\\end{equation}\r\n","sha256_utf8":"78dcaa6232ad3981ada92e73c87ef9978a765bf4831ac3d715d7df62990cf82a"}]},{"id":"S095","semantic_type":"proof_step","kind":"proof_step","title":"累加区间长度得到外层界","source_units":[{"start":128253,"end":129248,"start_line":1230,"end_line":1236,"raw_tex":"Summing up these inequalities, and using \\eqref{T-epsilong}, \\eqref{bound-cU},  $k_0=K_{\\epsilon_1}$ and $k_{|\\bK|+1}=\\overline{K}_{\\epsilon_1}-1$, we have\r\n\\begin{eqnarray}\r\n&\\hspace{-.85in}\\overline{K}_{\\epsilon_1} = 1+k_{|\\bK|+1}=1+k_0+\\sum_{j=0}^{|\\bK|}(k_{j+1}-k_{j}) \\overset{\\eqref{ineq:subsequence-rho-same}}{\\le}1+K_{\\epsilon_1}+(|\\bK|+1)\\left(\\left|\\frac{\\log(\\epsilon_1(2\\delta_{c,1})^{-1})}{\\log \\alpha}\\right|+1\\right) \\nonumber \\\\[6pt]\r\n&\\le2+\\frac{\\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})}{\\log{r}}+\\left(\\frac{\\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})}{\\log{r}}+1\\right)\\left(\\left|\\frac{\\log(\\epsilon_1(2\\delta_{c,1})^{-1})}{\\log \\alpha}\\right|+1\\right) =1+ \\left(\\frac{\\log({\\rho}_{\\epsilon_1}\\rho_0^{-1})}{\\log{r}}+1\\right)\\left(\\left|\\frac{\\log(\\epsilon_1(2\\delta_{c,1})^{-1})}{\\log \\alpha}\\right|+2\\right), \\nn\r\n\\end{eqnarray}\r\nwhere the second inequality is due to \\eqref{T-epsilong} and \\eqref{bound-cU}. Hence, \\eqref{outer-iteration-cmplxty} also holds in this case.\r\n\\end{proof}","sha256_utf8":"ff6cdad0ddb7eeb289095f94524cd9fd79b90e97f1609e78ad19456393d440d3"}]},{"id":"S096","semantic_type":"proof_step","kind":"proof_step","title":"调用已有共轭梯度终止估计","source_units":[{"start":168158,"end":168926,"start_line":1794,"end_line":1797,"raw_tex":"\\begin{proof}\r\nFrom \\cite[Lemma~1]{RNW18}, we know that the number of iterations of Algorithm \\ref{alg:capped-CG} is bounded by $\\min\\{n,J(U,\\varepsilon,\\zeta)\\}$, where $J(U,\\varepsilon,\\zeta)$ is the smallest integer $J$ such that $\\sqrt{T}\\tau^{J/2}\\le\\widehat{\\zeta}$, with $U,\\widehat{\\zeta},T$ and $\\tau$ being the values returned by Algorithm \\ref{alg:capped-CG}. In addition, it was shown in \\cite[Section 3.1]{RNW18} that\r\n$J(U,\\varepsilon,\\zeta)\\le\\left\\lceil \\left(\\sqrt{\\kappa}+\\frac{1}{2}\\right)\\ln\\left(\\frac{144(\\sqrt{\\kappa}+1)^2\\kappa^6}{\\zeta^2}\\right)\\right\\rceil$,\r\nwhere $\\kappa={\\cO}(U/\\varepsilon)$ is an output by Algorithm \\ref{alg:capped-CG}. Then one can see that $J(U,\\varepsilon,\\zeta)=\\widetilde{\\cO}(\\sqrt{U/\\varepsilon})$. Notice from\r\n","sha256_utf8":"b2f3454e8ba6913b99c9d38c7fa86a5acd0136b1e34c069facaf57633e7dcae0"}]},{"id":"S097","semantic_type":"proof_step","kind":"proof_step","title":"输出范数估计转为海森范数界","source_units":[{"start":168926,"end":169051,"start_line":1798,"end_line":1799,"raw_tex":"Algorithm~\\ref{alg:capped-CG} that the output $U\\le\\|H\\|$. Combining these, we obtain the conclusion as desired.\r\n\\end{proof}","sha256_utf8":"6de28f957cfd919ae419962500c12b9480b7e26860856bc973866532f5b9bd35"}]},{"id":"S098","semantic_type":"theorem","kind":"theorem","title":"内层求解与算法良定义性","source_units":[{"start":64691,"end":65150,"start_line":601,"end_line":604,"raw_tex":"\\begin{theorem}%[{{\\bf Well-definedness of Algorithm \\ref{alg:2nd-order-AL-nonconvex}}}]\r\n\\label{subprob-solver}\r\nUnder the same settings as in Lemma \\ref{lem:level-set-augmented-lagrangian-func}, the Newton-CG method (Algorithm \\ref{alg:NCG}) applied to the AL subproblem $\\min_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$ with $u^0=x^{k}_{\\init}$ finds a point $x^{k+1}$ satisfying \\eqref{algstop:1st-order} and \\eqref{algstop:2nd-order}.\r\n\\end{theorem}","sha256_utf8":"0cc2737c50333c3be13b26f59f0dbca02ef86a8bac1702faf0ed2317d1c4dbdf"}]},{"id":"S099","semantic_type":"proof","kind":"proof","title":"内层求解良定义性完整论证","source_units":[{"start":62429,"end":64236,"start_line":594,"end_line":596,"raw_tex":"Using Lemma \\ref{lem:level-set-augmented-lagrangian-func}, we can verify that the Newton-CG method (Algorithm \\ref{alg:NCG}), starting with $u^0=x^{k}_{\\init}$, is capable of finding an approximate solution $x^{k+1}$ of the AL subproblem $\\min_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$ satisfying \\eqref{algstop:1st-order} and \\eqref{algstop:2nd-order}. Indeed,\r\nlet $F(\\cdot)=\\widetilde{\\cL}(\\cdot,\\lambda^k;\\rho_k)$ and $u^0=x^{k}_{\\init}$. By these and Lemma \\ref{lem:level-set-augmented-lagrangian-func}, one can see that $\\{x:F(x) \\le F(u^0)\\} \\subseteq \\cS(\\bdelta_f,\\bdelta_c)$. It then follows from this and Assumption \\ref{asp:lowbd-knownfeas}(c) that the level set $\\{x : F(x) \\le F(u^0)\\}$ is compact  and $\\nabla^2 F$ is Lipschitz continuous on a convex open neighborhood of $\\{x:F(x) \\le F(u^0)\\}$. Thus, such $F$ and $u^0$ satisfy Assumption \\ref{asp:NCG-cmplxity}. Based on this and the discussion in Section \\ref{sec:sbpb-solver}, one can conclude that Algorithm~\\ref{alg:NCG}, starting with $u^0=x^{k}_{\\init}$, is applicable to the AL subproblem $\\min_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$. Moreover, it follows from Theorem \\ref{thm:NCG-iter-oper-cmplxity} that this algorithm with $(\\epsilon_g,\\epsilon_H)=(\\tau_k^g,\\tau_k^H)$ can produce a point $x^{k+1}$ satisfying \\eqref{algstop:2nd-order} and also the second relation in \\eqref{algstop:1st-order}. In addition, since this algorithm\r\nis descent and its starting point is $x^{k}_{\\init}$, its output $x^{k+1}$ must satisfy $\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k) \\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)$, which along with \\eqref{L-xinit} implies that $\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k) \\le f(z_{\\epsilon_1})$ and thus $x^{k+1}$ also satisfies the first relation in \\eqref{algstop:1st-order}.\r\n","sha256_utf8":"6771a3fd93555cf7600608162da89662a961cbd3a131ba71be61b2a45f7e65f6"}]},{"id":"S100","semantic_type":"proof_step","kind":"proof_step","title":"验证子问题假设并取得驻点残差","source_units":[{"start":62429,"end":63855,"start_line":594,"end_line":595,"raw_tex":"Using Lemma \\ref{lem:level-set-augmented-lagrangian-func}, we can verify that the Newton-CG method (Algorithm \\ref{alg:NCG}), starting with $u^0=x^{k}_{\\init}$, is capable of finding an approximate solution $x^{k+1}$ of the AL subproblem $\\min_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$ satisfying \\eqref{algstop:1st-order} and \\eqref{algstop:2nd-order}. Indeed,\r\nlet $F(\\cdot)=\\widetilde{\\cL}(\\cdot,\\lambda^k;\\rho_k)$ and $u^0=x^{k}_{\\init}$. By these and Lemma \\ref{lem:level-set-augmented-lagrangian-func}, one can see that $\\{x:F(x) \\le F(u^0)\\} \\subseteq \\cS(\\bdelta_f,\\bdelta_c)$. It then follows from this and Assumption \\ref{asp:lowbd-knownfeas}(c) that the level set $\\{x : F(x) \\le F(u^0)\\}$ is compact  and $\\nabla^2 F$ is Lipschitz continuous on a convex open neighborhood of $\\{x:F(x) \\le F(u^0)\\}$. Thus, such $F$ and $u^0$ satisfy Assumption \\ref{asp:NCG-cmplxity}. Based on this and the discussion in Section \\ref{sec:sbpb-solver}, one can conclude that Algorithm~\\ref{alg:NCG}, starting with $u^0=x^{k}_{\\init}$, is applicable to the AL subproblem $\\min_{x\\in\\bR^n} \\widetilde{\\cL}(x,\\lambda^k;\\rho_k)$. Moreover, it follows from Theorem \\ref{thm:NCG-iter-oper-cmplxity} that this algorithm with $(\\epsilon_g,\\epsilon_H)=(\\tau_k^g,\\tau_k^H)$ can produce a point $x^{k+1}$ satisfying \\eqref{algstop:2nd-order} and also the second relation in \\eqref{algstop:1st-order}. In addition, since this algorithm\r\n","sha256_utf8":"1279f0f105f5788d4645a8376d8dbea152f9e48f072d7714eaa7b45abd504b2b"}]},{"id":"S101","semantic_type":"proof_step","kind":"proof_step","title":"下降性质保证子问题函数值门槛","source_units":[{"start":63855,"end":64236,"start_line":596,"end_line":596,"raw_tex":"is descent and its starting point is $x^{k}_{\\init}$, its output $x^{k+1}$ must satisfy $\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k) \\le \\widetilde{\\cL}(x^{k}_{\\init},\\lambda^k;\\rho_k)$, which along with \\eqref{L-xinit} implies that $\\widetilde{\\cL}(x^{k+1},\\lambda^k;\\rho_k) \\le f(z_{\\epsilon_1})$ and thus $x^{k+1}$ also satisfies the first relation in \\eqref{algstop:1st-order}.\r\n","sha256_utf8":"3cfcfddb28dd610caf630b3c8a46b6760e021745d54f2fce8fd881eaeabb6795"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S014","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S052","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S018","kind":"supports"},{"source_id":"S052","target_id":"S018","kind":"supports"},{"source_id":"S008","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S019","target_id":"S012","kind":"supports"},{"source_id":"S020","target_id":"S012","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S012","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S053","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S032","target_id":"S036","kind":"supports"},{"source_id":"S034","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S049","target_id":"S023","kind":"supports"},{"source_id":"S051","target_id":"S039","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S013","target_id":"S041","kind":"supports"},{"source_id":"S014","target_id":"S041","kind":"supports"},{"source_id":"S015","target_id":"S041","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S017","target_id":"S043","kind":"supports"},{"source_id":"S015","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S013","target_id":"S045","kind":"supports"},{"source_id":"S014","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S014","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S013","target_id":"S048","kind":"supports"},{"source_id":"S015","target_id":"S048","kind":"supports"},{"source_id":"S032","target_id":"S053","kind":"supports"},{"source_id":"S015","target_id":"S053","kind":"supports"},{"source_id":"S098","target_id":"S053","kind":"supports"},{"source_id":"S017","target_id":"S054","kind":"supports"},{"source_id":"S042","target_id":"S054","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S028","target_id":"S055","kind":"supports"},{"source_id":"S040","target_id":"S055","kind":"supports"},{"source_id":"S021","target_id":"S055","kind":"supports"},{"source_id":"S046","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S047","target_id":"S056","kind":"supports"},{"source_id":"S036","target_id":"S056","kind":"supports"},{"source_id":"S053","target_id":"S057","kind":"supports"},{"source_id":"S054","target_id":"S057","kind":"supports"},{"source_id":"S028","target_id":"S057","kind":"supports"},{"source_id":"S040","target_id":"S057","kind":"supports"},{"source_id":"S021","target_id":"S057","kind":"supports"},{"source_id":"S046","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S047","target_id":"S058","kind":"supports"},{"source_id":"S036","target_id":"S058","kind":"supports"},{"source_id":"S053","target_id":"S002","kind":"supports"},{"source_id":"S054","target_id":"S002","kind":"supports"},{"source_id":"S055","target_id":"S002","kind":"supports"},{"source_id":"S056","target_id":"S002","kind":"supports"},{"source_id":"S057","target_id":"S002","kind":"supports"},{"source_id":"S058","target_id":"S002","kind":"supports"},{"source_id":"S012","target_id":"S059","kind":"supports"},{"source_id":"S010","target_id":"S059","kind":"supports"},{"source_id":"S012","target_id":"S060","kind":"supports"},{"source_id":"S026","target_id":"S060","kind":"supports"},{"source_id":"S010","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S024","target_id":"S061","kind":"supports"},{"source_id":"S026","target_id":"S061","kind":"supports"},{"source_id":"S012","target_id":"S061","kind":"supports"},{"source_id":"S010","target_id":"S061","kind":"supports"},{"source_id":"S060","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S012","target_id":"S062","kind":"supports"},{"source_id":"S039","target_id":"S062","kind":"supports"},{"source_id":"S059","target_id":"S063","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S037","target_id":"S063","kind":"supports"},{"source_id":"S039","target_id":"S063","kind":"supports"},{"source_id":"S051","target_id":"S063","kind":"supports"},{"source_id":"S059","target_id":"S022","kind":"supports"},{"source_id":"S060","target_id":"S022","kind":"supports"},{"source_id":"S061","target_id":"S022","kind":"supports"},{"source_id":"S062","target_id":"S022","kind":"supports"},{"source_id":"S063","target_id":"S022","kind":"supports"},{"source_id":"S012","target_id":"S064","kind":"supports"},{"source_id":"S010","target_id":"S064","kind":"supports"},{"source_id":"S023","target_id":"S064","kind":"supports"},{"source_id":"S012","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S066","kind":"supports"},{"source_id":"S023","target_id":"S066","kind":"supports"},{"source_id":"S018","target_id":"S066","kind":"supports"},{"source_id":"S052","target_id":"S066","kind":"supports"},{"source_id":"S064","target_id":"S067","kind":"supports"},{"source_id":"S023","target_id":"S067","kind":"supports"},{"source_id":"S018","target_id":"S067","kind":"supports"},{"source_id":"S012","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S023","target_id":"S068","kind":"supports"},{"source_id":"S010","target_id":"S068","kind":"supports"},{"source_id":"S012","target_id":"S068","kind":"supports"},{"source_id":"S064","target_id":"S069","kind":"supports"},{"source_id":"S018","target_id":"S069","kind":"supports"},{"source_id":"S023","target_id":"S069","kind":"supports"},{"source_id":"S012","target_id":"S069","kind":"supports"},{"source_id":"S066","target_id":"S070","kind":"supports"},{"source_id":"S067","target_id":"S070","kind":"supports"},{"source_id":"S068","target_id":"S070","kind":"supports"},{"source_id":"S012","target_id":"S070","kind":"supports"},{"source_id":"S064","target_id":"S025","kind":"supports"},{"source_id":"S065","target_id":"S025","kind":"supports"},{"source_id":"S066","target_id":"S025","kind":"supports"},{"source_id":"S067","target_id":"S025","kind":"supports"},{"source_id":"S068","target_id":"S025","kind":"supports"},{"source_id":"S069","target_id":"S025","kind":"supports"},{"source_id":"S070","target_id":"S025","kind":"supports"},{"source_id":"S012","target_id":"S071","kind":"supports"},{"source_id":"S023","target_id":"S071","kind":"supports"},{"source_id":"S020","target_id":"S071","kind":"supports"},{"source_id":"S012","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S073","kind":"supports"},{"source_id":"S009","target_id":"S073","kind":"supports"},{"source_id":"S052","target_id":"S073","kind":"supports"},{"source_id":"S071","target_id":"S074","kind":"supports"},{"source_id":"S018","target_id":"S074","kind":"supports"},{"source_id":"S012","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S012","target_id":"S075","kind":"supports"},{"source_id":"S023","target_id":"S076","kind":"supports"},{"source_id":"S075","target_id":"S076","kind":"supports"},{"source_id":"S012","target_id":"S076","kind":"supports"},{"source_id":"S020","target_id":"S077","kind":"supports"},{"source_id":"S075","target_id":"S077","kind":"supports"},{"source_id":"S012","target_id":"S077","kind":"supports"},{"source_id":"S071","target_id":"S027","kind":"supports"},{"source_id":"S072","target_id":"S027","kind":"supports"},{"source_id":"S073","target_id":"S027","kind":"supports"},{"source_id":"S074","target_id":"S027","kind":"supports"},{"source_id":"S075","target_id":"S027","kind":"supports"},{"source_id":"S076","target_id":"S027","kind":"supports"},{"source_id":"S077","target_id":"S027","kind":"supports"},{"source_id":"S014","target_id":"S078","kind":"supports"},{"source_id":"S014","target_id":"S079","kind":"supports"},{"source_id":"S044","target_id":"S079","kind":"supports"},{"source_id":"S078","target_id":"S080","kind":"supports"},{"source_id":"S079","target_id":"S081","kind":"supports"},{"source_id":"S078","target_id":"S031","kind":"supports"},{"source_id":"S079","target_id":"S031","kind":"supports"},{"source_id":"S080","target_id":"S031","kind":"supports"},{"source_id":"S081","target_id":"S031","kind":"supports"},{"source_id":"S040","target_id":"S082","kind":"supports"},{"source_id":"S030","target_id":"S082","kind":"supports"},{"source_id":"S013","target_id":"S082","kind":"supports"},{"source_id":"S014","target_id":"S082","kind":"supports"},{"source_id":"S015","target_id":"S082","kind":"supports"},{"source_id":"S082","target_id":"S083","kind":"supports"},{"source_id":"S044","target_id":"S083","kind":"supports"},{"source_id":"S014","target_id":"S083","kind":"supports"},{"source_id":"S015","target_id":"S083","kind":"supports"},{"source_id":"S082","target_id":"S029","kind":"supports"},{"source_id":"S083","target_id":"S029","kind":"supports"},{"source_id":"S015","target_id":"S084","kind":"supports"},{"source_id":"S014","target_id":"S084","kind":"supports"},{"source_id":"S015","target_id":"S085","kind":"supports"},{"source_id":"S014","target_id":"S085","kind":"supports"},{"source_id":"S084","target_id":"S086","kind":"supports"},{"source_id":"S085","target_id":"S086","kind":"supports"},{"source_id":"S005","target_id":"S086","kind":"supports"},{"source_id":"S006","target_id":"S086","kind":"supports"},{"source_id":"S007","target_id":"S086","kind":"supports"},{"source_id":"S084","target_id":"S035","kind":"supports"},{"source_id":"S085","target_id":"S035","kind":"supports"},{"source_id":"S086","target_id":"S035","kind":"supports"},{"source_id":"S017","target_id":"S087","kind":"supports"},{"source_id":"S015","target_id":"S087","kind":"supports"},{"source_id":"S030","target_id":"S088","kind":"supports"},{"source_id":"S013","target_id":"S088","kind":"supports"},{"source_id":"S014","target_id":"S088","kind":"supports"},{"source_id":"S015","target_id":"S088","kind":"supports"},{"source_id":"S088","target_id":"S089","kind":"supports"},{"source_id":"S042","target_id":"S089","kind":"supports"},{"source_id":"S015","target_id":"S089","kind":"supports"},{"source_id":"S098","target_id":"S089","kind":"supports"},{"source_id":"S087","target_id":"S090","kind":"supports"},{"source_id":"S089","target_id":"S090","kind":"supports"},{"source_id":"S088","target_id":"S091","kind":"supports"},{"source_id":"S089","target_id":"S091","kind":"supports"},{"source_id":"S030","target_id":"S091","kind":"supports"},{"source_id":"S013","target_id":"S091","kind":"supports"},{"source_id":"S015","target_id":"S091","kind":"supports"},{"source_id":"S087","target_id":"S091","kind":"supports"},{"source_id":"S091","target_id":"S092","kind":"supports"},{"source_id":"S015","target_id":"S092","kind":"supports"},{"source_id":"S015","target_id":"S093","kind":"supports"},{"source_id":"S088","target_id":"S093","kind":"supports"},{"source_id":"S091","target_id":"S093","kind":"supports"},{"source_id":"S093","target_id":"S094","kind":"supports"},{"source_id":"S087","target_id":"S095","kind":"supports"},{"source_id":"S092","target_id":"S095","kind":"supports"},{"source_id":"S094","target_id":"S095","kind":"supports"},{"source_id":"S087","target_id":"S033","kind":"supports"},{"source_id":"S088","target_id":"S033","kind":"supports"},{"source_id":"S089","target_id":"S033","kind":"supports"},{"source_id":"S090","target_id":"S033","kind":"supports"},{"source_id":"S091","target_id":"S033","kind":"supports"},{"source_id":"S092","target_id":"S033","kind":"supports"},{"source_id":"S093","target_id":"S033","kind":"supports"},{"source_id":"S094","target_id":"S033","kind":"supports"},{"source_id":"S095","target_id":"S033","kind":"supports"},{"source_id":"S050","target_id":"S096","kind":"supports"},{"source_id":"S019","target_id":"S096","kind":"supports"},{"source_id":"S096","target_id":"S097","kind":"supports"},{"source_id":"S019","target_id":"S097","kind":"supports"},{"source_id":"S096","target_id":"S038","kind":"supports"},{"source_id":"S097","target_id":"S038","kind":"supports"},{"source_id":"S099","target_id":"S098","kind":"supports"},{"source_id":"S028","target_id":"S100","kind":"supports"},{"source_id":"S013","target_id":"S100","kind":"supports"},{"source_id":"S009","target_id":"S100","kind":"supports"},{"source_id":"S021","target_id":"S100","kind":"supports"},{"source_id":"S014","target_id":"S100","kind":"supports"},{"source_id":"S012","target_id":"S100","kind":"supports"},{"source_id":"S100","target_id":"S101","kind":"supports"},{"source_id":"S040","target_id":"S101","kind":"supports"},{"source_id":"S012","target_id":"S101","kind":"supports"},{"source_id":"S100","target_id":"S099","kind":"supports"},{"source_id":"S101","target_id":"S099","kind":"supports"}]}}
{"record_id":"2301.04431v4","paper":{"title":"Adaptive proximal algorithms for convex optimization under local Lipschitz continuity of the gradient","authors":["Puya Latafat","Andreas Themelis","Lorenzo Stella","Panagiotis Patrinos"],"arxiv_id":"2301.04431","version":"v4","journal":"Mathematical Programming","doi":"10.1007/s10107-024-02143-7","abstract_url":"https://arxiv.org/abs/2301.04431v4","pdf_url":"https://arxiv.org/pdf/2301.04431v4"},"context":{"raw_tex":"\\begin{equation}\\label{eq:MML_k}\n\t\t\tL_k\n\t\t\\coloneqq\n\t\t\t\\frac{\n\t\t\t\t\\|\\nabla f(x^k) - \\nabla f(x^{k-1})\\|\n\t\t\t}{\n\t\t\t\t\\|x^k - x^{k-1}\\|\n\t\t\t}\n\t\t\\end{equation}\n\n\\begin{align}\n\t\t\t\t\\Lk\n\t\t\t\\coloneqq{} &\n\t\t\t\t\\frac{\n\t\t\t\t\t\\innprod{\\nabla f(x^{k-1})-\\nabla f(x^k)}{x^{k-1}-x^k}\n\t\t\t\t}{\n\t\t\t\t\t\\|x^{k-1}-x^k\\|^2\n\t\t\t\t}\n\t\t\t\\shortintertext{and}\n\t\t\t\t\\ck\n\t\t\t\\coloneqq{} &\n\t\t\t\t\\frac{\n\t\t\t\t\t\\|\\nabla f(x^{k-1})-\\nabla f(x^k)\\|^2\n\t\t\t\t}{\n\t\t\t\t\t\\innprod{\\nabla f(x^{k-1})-\\nabla f(x^k)}{x^{k-1}-x^k}\n\t\t\t\t}\n\t\t\t\\end{align}\n\n\t\tfor a pair of points \\(x^{k-1},x^k\\in\\R^n\\).\n\t\tNoting that the denominator of \\(\\Lk\\) or \\(\\ck\\) is zero iff \\(\\nabla f(x^k)-\\nabla f(x^{k-1})=0\\) (the latter owing to the Baillon-Haddad theorem \\cite[Cor. 10]{baillon1977quelques}; see \\cref{lem:enlarged:ck} for the local version needed here), we stick to the convention \\(\\frac00=0\\) so that both \\(\\Lk\\) and \\(\\ck\\) are (well-defined, positive) real numbers.\n\t\tThroughout, we shall also adhere to \\(\\frac10=\\infty\\).\n\n\n\\begin{equation}\\label{eq:PG:gamk}\n\t\t\t\\gamk*\n\t\t=\n\t\t\t\\min\\set{\n\t\t\t\t\\gamk\\sqrt{1+\\tfrac{\\gamk}{\\gamma_{k-1}}},\\,\n\t\t\t\t\\frac{\n\t\t\t\t\t\\gamk\n\t\t\t\t}{\n\t\t\t\t\t2\\sqrt{\\left[\\gamk\\Lk(\\gamk\\ck-1)\\right]_+}\n\t\t\t\t}\n\t\t\t}\n\t\t\\end{equation}\n\n\t\t\tThe set of real and extended-real numbers are \\(\\R\\coloneqq(-\\infty,\\infty)\\) and \\(\\Rinf\\coloneqq\\R\\cup\\set\\infty\\), while the positive and strictly positive reals are \\(\\R_+\\coloneqq[0,\\infty)\\) and \\(\\R_{++}\\coloneqq(0,\\infty)\\).\n\t\t\tWe use the notation \\([x]_+ = \\max\\set{0, x}\\).\n\t\t\tWith \\(\\id\\) we indicate the identity function defined on a suitable space.\n\t\t\tWe denote by \\(\\innprod{{}\\cdot{}}{{}\\cdot{}}\\) and \\(\\|{}\\cdot{}\\|\\) the standard Euclidean inner product and the induced norm.\n\n\n\t\t\tThe diameter of \\(\\mathcal D\\) is \\(\\diam\\mathcal D\\coloneqq\\sup\\set{\\|x-y\\|}[x,y\\in\\mathcal D]\\), while its indicator function is \\(\\indicator_{\\mathcal D}\\), namely \\(\\indicator_{\\mathcal D}(x)=0\\) if \\(x\\in \\mathcal D\\) and \\(\\infty\\) otherwise.\n\t\t\tGiven two sets \\(\\mathcal D,\\mathcal D'\\subseteq\\R^n\\), the notation \\(\\mathcal D+\\mathcal D'\\coloneqq\\set{x+x'}[x\\in\\mathcal D,\\ x'\\in\\mathcal D']\\) indicates their Minkowski sum.\n\n\n\t\t\tThe notation \\(\\cball{\\bar x}{r}\\coloneqq\\set{x}[\\|x-\\bar x\\|\\leq r]\\) indicates the closed ball centered at \\(\\bar x\\) and with radius \\(r\\).\n\t\t\tThe \\DEF{domain} and \\DEF{epigraph} of an extended-real-valued function \\(\\func{h}{\\R^n}{\\Rinf}\\) are the sets\n\t\t\t\\(\n\t\t\t\t\\dom h\n\t\t\t\\coloneqq\n\t\t\t\t\\set{x\\in\\R^n}[\n\t\t\t\t\th(x)<\\infty\n\t\t\t\t]\n\t\t\t\\)\n\t\t\tand\n\t\t\t\\(\n\t\t\t\t\\epi h\n\t\t\t\\coloneqq\n\t\t\t\t\\set{(x, c )\\in\\R^n\\times\\R}[\n\t\t\t\t\th(x)\\leq c\n\t\t\t\t]\n\t\t\t\\).\n\t\t\tFunction \\(h\\) is said to be \\DEF{proper} if \\(\\dom h\\neq\\emptyset\\), and \\DEF{lower semicontinuous (lsc)} if \\(\\epi h\\) is a closed subset of \\(\\R^{n+1}\\).\n\n\n\t\t\tWe say that a differentiable function \\(\\func{h}{\\R^n}{\\R}\\) has \\DEF{locally Lipschitz continuous gradient} if for every convex and compact set \\(\\V\\subset\\R^n\\) there exists \\(L_{h,\\V}>0\\) such that\n\t\t\t\\(\n\t\t\t\t\\|\\nabla h(x)-\\nabla h(x')\\|\\leq L_{h,\\V}\\|x-x'\\|\n\t\t\t\\)\n\t\t\tholds for all \\(x,x'\\in\\mathcal\\V\\).\\footnote{%\n\t\t\t\tBy virtue of \\cite[Thm. 9.2]{rockafellar2009variational}, this condition is equivalent to \\DEF{strict continuity} of \\(\\nabla h\\) at every point, in the sense of \\cite[Def. 9.1]{rockafellar2009variational}.\n\t\t\t}\n\n\n\\begin{equation}\\label{eq:PG}\n\t\t\t\\minimize_{x\\in\\R^n}\\; \\varphi(x) \\coloneqq f(x)+g(x).\n\t\t\\end{equation}\n\n\\begin{assumption}[Requirements for the proximal gradient setting]\\label{ass:PG}\n\t\t\tThe following hold in problem \\eqref{eq:PG}:\n\t\t\t\\begin{enumeratass}\n\t\t\t\\item \\label{ass:f}%\n\t\t\t\t\\(\\func{f}{\\R^n}{\\R}\\) is convex and has locally Lipschitz continuous gradient.\n\t\t\t\\item\n\t\t\t\t\\(\\func{g}{\\R^n}{\\Rinf}\\) is proper, lsc, and convex.\n\t\t\t\\item\n\t\t\t\tA solution exists: \\(\\argmin\\varphi\\neq\\emptyset\\).\n\t\t\t\\end{enumeratass}\n\t\t\\end{assumption}\n\n\t\tIn addition to the gradient of the differentiable term, the fundamental oracle of {\\algnamefont PGM} is the \\emph{proximal mapping} \\cite[Def. 6.1]{beck2017first}\n\t\tdefined as\n\t\t\\[\n\t\t\t\\prox_{\\tau g}(x)\n\t\t\\coloneqq\n\t\t\t\\argmin_{w\\in\\R^n}\n\t\t\t\\set{g(w) + \\tfrac{1}{2\\tau}\\|w-x\\|^2},\n\t\t\\]\n\t\twhere \\(\\tau>0\\) is a given stepsize.\n\t\tIn the convex setting the proximal map is single valued and in fact Lipschitz continuous.\n\n\n\\begin{algorithm}[t]\n\t\t\t\\caption{Adaptive proximal gradient method (\\protect\\adaPG)}\n\t\t\t\\label{alg:PG}\n\t\t\t\\begin{algorithmic}[1]\n\t\t\t\\itemsep=3pt\n\t\t\t\\Require\n\t\t\t\tstarting point \\(x^{-1}\\in\\R^n\\)\n\t\t\t\t~and~\n\t\t\t\tstepsizes \\(\\gamma_0\\geq\\gamma_{-1}>0\\)\n\t\t\t\\Initialize\n\t\t\t\t\\(x^0=\\FBk[0]{x^{-1}}\\)\n\t\t\t\\item[\\algfont{Repeat for} \\(k=0,1,\\ldots\\) until convergence]\n\t\t\t\\State \\label{state:PG:gamk}%\n\t\t\t\tWith \\(\\Lk\\) and \\(\\ck\\) as in \\eqref{eq:CL}, define the stepsize as\n\t\t\t\\hfill{\\gray\\footnotesize\\sf // \\([r]_+\\coloneqq\\max\\set{r,0}\\)}\n\t\t\t\t\\[\n\t\t\t\t\t\\gamk*\n\t\t\t\t=\n\t\t\t\t\t\\gamk\\min\\set{\n\t\t\t\t\t\t\\sqrt{1+\\tfrac{\\gamk}{\\gamma_{k-1}}},\\,\n\t\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\t1\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t2\\sqrt{\\left[\\gamk\\Lk(\\gamk\\ck-1)\\right]_+}\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\\]\n\t\t\t\\State\n\t\t\t\t\\(x^{k+1}=\\FBk*{x^k}\\)\n\t\t\t\\end{algorithmic}\n\t\t\\end{algorithm}\n\n\\[\n\t\t\t\t\\Hk\\coloneqq\\Fwk{}.\n\t\t\t\\]\n\n\\begin{equation}\\label{eq:rhoP}\n\t\t\t\t\\rhok\\coloneqq\\tfrac{\\gamk}{\\gamma_{k-1}}\n\t\t\t\\quad\\text{and}\\quad\n\t\t\t\tP_k\\coloneqq\\varphi(x^k)-\\min\\varphi.\n\t\t\t\\end{equation}","units":[{"start":9103,"end":9256,"start_line":245,"end_line":253,"raw_tex":"\\begin{equation}\\label{eq:MML_k}\n\t\t\tL_k\n\t\t\\coloneqq\n\t\t\t\\frac{\n\t\t\t\t\\|\\nabla f(x^k) - \\nabla f(x^{k-1})\\|\n\t\t\t}{\n\t\t\t\t\\|x^k - x^{k-1}\\|\n\t\t\t}\n\t\t\\end{equation}","sha256_utf8":"9016350beed93d8e038f77c3c8a71bd859edf305f3b4df025bddd573d111c539"},{"start":11631,"end":11967,"start_line":267,"end_line":283,"raw_tex":"\\begin{align}\n\t\t\t\t\\Lk\n\t\t\t\\coloneqq{} &\n\t\t\t\t\\frac{\n\t\t\t\t\t\\innprod{\\nabla f(x^{k-1})-\\nabla f(x^k)}{x^{k-1}-x^k}\n\t\t\t\t}{\n\t\t\t\t\t\\|x^{k-1}-x^k\\|^2\n\t\t\t\t}\n\t\t\t\\shortintertext{and}\n\t\t\t\t\\ck\n\t\t\t\\coloneqq{} &\n\t\t\t\t\\frac{\n\t\t\t\t\t\\|\\nabla f(x^{k-1})-\\nabla f(x^k)\\|^2\n\t\t\t\t}{\n\t\t\t\t\t\\innprod{\\nabla f(x^{k-1})-\\nabla f(x^k)}{x^{k-1}-x^k}\n\t\t\t\t}\n\t\t\t\\end{align}","sha256_utf8":"999b4daac8e2b5c6aae676739e81806de62df0ec1c727911789eeb64ee74d18c"},{"start":11989,"end":12461,"start_line":285,"end_line":287,"raw_tex":"\t\tfor a pair of points \\(x^{k-1},x^k\\in\\R^n\\).\n\t\tNoting that the denominator of \\(\\Lk\\) or \\(\\ck\\) is zero iff \\(\\nabla f(x^k)-\\nabla f(x^{k-1})=0\\) (the latter owing to the Baillon-Haddad theorem \\cite[Cor. 10]{baillon1977quelques}; see \\cref{lem:enlarged:ck} for the local version needed here), we stick to the convention \\(\\frac00=0\\) so that both \\(\\Lk\\) and \\(\\ck\\) are (well-defined, positive) real numbers.\n\t\tThroughout, we shall also adhere to \\(\\frac10=\\infty\\).\n","sha256_utf8":"960eaa2fb26df45293ddc078a9b18af533ec480f06a3a316bb76ce14eec2328a"},{"start":12938,"end":13154,"start_line":290,"end_line":301,"raw_tex":"\\begin{equation}\\label{eq:PG:gamk}\n\t\t\t\\gamk*\n\t\t=\n\t\t\t\\min\\set{\n\t\t\t\t\\gamk\\sqrt{1+\\tfrac{\\gamk}{\\gamma_{k-1}}},\\,\n\t\t\t\t\\frac{\n\t\t\t\t\t\\gamk\n\t\t\t\t}{\n\t\t\t\t\t2\\sqrt{\\left[\\gamk\\Lk(\\gamk\\ck-1)\\right]_+}\n\t\t\t\t}\n\t\t\t}\n\t\t\\end{equation}","sha256_utf8":"83377508269fde753f530934060cd6aecc8788607e7fe51aebb1e999e08c4560"},{"start":21635,"end":22133,"start_line":383,"end_line":386,"raw_tex":"\t\t\tThe set of real and extended-real numbers are \\(\\R\\coloneqq(-\\infty,\\infty)\\) and \\(\\Rinf\\coloneqq\\R\\cup\\set\\infty\\), while the positive and strictly positive reals are \\(\\R_+\\coloneqq[0,\\infty)\\) and \\(\\R_{++}\\coloneqq(0,\\infty)\\).\n\t\t\tWe use the notation \\([x]_+ = \\max\\set{0, x}\\).\n\t\t\tWith \\(\\id\\) we indicate the identity function defined on a suitable space.\n\t\t\tWe denote by \\(\\innprod{{}\\cdot{}}{{}\\cdot{}}\\) and \\(\\|{}\\cdot{}\\|\\) the standard Euclidean inner product and the induced norm.\n","sha256_utf8":"47f326f89f92ce6c6ca2168671f0ac191ecd2f23c98649ecc55619156892a8b4"},{"start":22465,"end":22901,"start_line":388,"end_line":389,"raw_tex":"\t\t\tThe diameter of \\(\\mathcal D\\) is \\(\\diam\\mathcal D\\coloneqq\\sup\\set{\\|x-y\\|}[x,y\\in\\mathcal D]\\), while its indicator function is \\(\\indicator_{\\mathcal D}\\), namely \\(\\indicator_{\\mathcal D}(x)=0\\) if \\(x\\in \\mathcal D\\) and \\(\\infty\\) otherwise.\n\t\t\tGiven two sets \\(\\mathcal D,\\mathcal D'\\subseteq\\R^n\\), the notation \\(\\mathcal D+\\mathcal D'\\coloneqq\\set{x+x'}[x\\in\\mathcal D,\\ x'\\in\\mathcal D']\\) indicates their Minkowski sum.\n","sha256_utf8":"c90368feff6880a0b61286189593328912b5165fa0ddc0fb2b1a61dffa93946a"},{"start":22902,"end":23501,"start_line":391,"end_line":408,"raw_tex":"\t\t\tThe notation \\(\\cball{\\bar x}{r}\\coloneqq\\set{x}[\\|x-\\bar x\\|\\leq r]\\) indicates the closed ball centered at \\(\\bar x\\) and with radius \\(r\\).\n\t\t\tThe \\DEF{domain} and \\DEF{epigraph} of an extended-real-valued function \\(\\func{h}{\\R^n}{\\Rinf}\\) are the sets\n\t\t\t\\(\n\t\t\t\t\\dom h\n\t\t\t\\coloneqq\n\t\t\t\t\\set{x\\in\\R^n}[\n\t\t\t\t\th(x)<\\infty\n\t\t\t\t]\n\t\t\t\\)\n\t\t\tand\n\t\t\t\\(\n\t\t\t\t\\epi h\n\t\t\t\\coloneqq\n\t\t\t\t\\set{(x, c )\\in\\R^n\\times\\R}[\n\t\t\t\t\th(x)\\leq c\n\t\t\t\t]\n\t\t\t\\).\n\t\t\tFunction \\(h\\) is said to be \\DEF{proper} if \\(\\dom h\\neq\\emptyset\\), and \\DEF{lower semicontinuous (lsc)} if \\(\\epi h\\) is a closed subset of \\(\\R^{n+1}\\).\n","sha256_utf8":"f9a52010f53129d358a8988ddf6513c4ea2f94eb52a0195beb90c20e17deb7fb"},{"start":23825,"end":24362,"start_line":425,"end_line":431,"raw_tex":"\t\t\tWe say that a differentiable function \\(\\func{h}{\\R^n}{\\R}\\) has \\DEF{locally Lipschitz continuous gradient} if for every convex and compact set \\(\\V\\subset\\R^n\\) there exists \\(L_{h,\\V}>0\\) such that\n\t\t\t\\(\n\t\t\t\t\\|\\nabla h(x)-\\nabla h(x')\\|\\leq L_{h,\\V}\\|x-x'\\|\n\t\t\t\\)\n\t\t\tholds for all \\(x,x'\\in\\mathcal\\V\\).\\footnote{%\n\t\t\t\tBy virtue of \\cite[Thm. 9.2]{rockafellar2009variational}, this condition is equivalent to \\DEF{strict continuity} of \\(\\nabla h\\) at every point, in the sense of \\cite[Def. 9.1]{rockafellar2009variational}.\n\t\t\t}\n","sha256_utf8":"ad7909ccef8a22aa994f70400d09245d1a530d8b73927b1b1a250460f6c369f3"},{"start":24667,"end":24771,"start_line":439,"end_line":441,"raw_tex":"\\begin{equation}\\label{eq:PG}\n\t\t\t\\minimize_{x\\in\\R^n}\\; \\varphi(x) \\coloneqq f(x)+g(x).\n\t\t\\end{equation}","sha256_utf8":"623169c5934295751ebdc92d57663af1e5ca66453ce6ee57feab3d07fb79b52c"},{"start":24876,"end":25307,"start_line":444,"end_line":454,"raw_tex":"\\begin{assumption}[Requirements for the proximal gradient setting]\\label{ass:PG}\n\t\t\tThe following hold in problem \\eqref{eq:PG}:\n\t\t\t\\begin{enumeratass}\n\t\t\t\\item \\label{ass:f}%\n\t\t\t\t\\(\\func{f}{\\R^n}{\\R}\\) is convex and has locally Lipschitz continuous gradient.\n\t\t\t\\item\n\t\t\t\t\\(\\func{g}{\\R^n}{\\Rinf}\\) is proper, lsc, and convex.\n\t\t\t\\item\n\t\t\t\tA solution exists: \\(\\argmin\\varphi\\neq\\emptyset\\).\n\t\t\t\\end{enumeratass}\n\t\t\\end{assumption}","sha256_utf8":"910852942f97654f72df3f04e8ea50f2877d7c1958547dae05026331f32aad29"},{"start":25309,"end":25727,"start_line":456,"end_line":465,"raw_tex":"\t\tIn addition to the gradient of the differentiable term, the fundamental oracle of {\\algnamefont PGM} is the \\emph{proximal mapping} \\cite[Def. 6.1]{beck2017first}\n\t\tdefined as\n\t\t\\[\n\t\t\t\\prox_{\\tau g}(x)\n\t\t\\coloneqq\n\t\t\t\\argmin_{w\\in\\R^n}\n\t\t\t\\set{g(w) + \\tfrac{1}{2\\tau}\\|w-x\\|^2},\n\t\t\\]\n\t\twhere \\(\\tau>0\\) is a given stepsize.\n\t\tIn the convex setting the proximal map is single valued and in fact Lipschitz continuous.\n","sha256_utf8":"28851be81c4162f48b98241d56c5a5e122ada5e2886227bd3590629d8d2388ba"},{"start":26220,"end":27021,"start_line":470,"end_line":500,"raw_tex":"\\begin{algorithm}[t]\n\t\t\t\\caption{Adaptive proximal gradient method (\\protect\\adaPG)}\n\t\t\t\\label{alg:PG}\n\t\t\t\\begin{algorithmic}[1]\n\t\t\t\\itemsep=3pt\n\t\t\t\\Require\n\t\t\t\tstarting point \\(x^{-1}\\in\\R^n\\)\n\t\t\t\t~and~\n\t\t\t\tstepsizes \\(\\gamma_0\\geq\\gamma_{-1}>0\\)\n\t\t\t\\Initialize\n\t\t\t\t\\(x^0=\\FBk[0]{x^{-1}}\\)\n\t\t\t\\item[\\algfont{Repeat for} \\(k=0,1,\\ldots\\) until convergence]\n\t\t\t\\State \\label{state:PG:gamk}%\n\t\t\t\tWith \\(\\Lk\\) and \\(\\ck\\) as in \\eqref{eq:CL}, define the stepsize as\n\t\t\t\\hfill{\\gray\\footnotesize\\sf // \\([r]_+\\coloneqq\\max\\set{r,0}\\)}\n\t\t\t\t\\[\n\t\t\t\t\t\\gamk*\n\t\t\t\t=\n\t\t\t\t\t\\gamk\\min\\set{\n\t\t\t\t\t\t\\sqrt{1+\\tfrac{\\gamk}{\\gamma_{k-1}}},\\,\n\t\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\t1\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t2\\sqrt{\\left[\\gamk\\Lk(\\gamk\\ck-1)\\right]_+}\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\\]\n\t\t\t\\State\n\t\t\t\t\\(x^{k+1}=\\FBk*{x^k}\\)\n\t\t\t\\end{algorithmic}\n\t\t\\end{algorithm}","sha256_utf8":"f153ee0ce3a37543ed05f617694791992d8337fd8ee8f1cf2fa7b5f3dd50bd32"},{"start":30276,"end":30308,"start_line":535,"end_line":537,"raw_tex":"\\[\n\t\t\t\t\\Hk\\coloneqq\\Fwk{}.\n\t\t\t\\]","sha256_utf8":"de94aba0f3369e92f0efe0956634bfd94d985d1f603e247bb44055a3b9dd29ef"},{"start":33055,"end":33216,"start_line":597,"end_line":601,"raw_tex":"\\begin{equation}\\label{eq:rhoP}\n\t\t\t\t\\rhok\\coloneqq\\tfrac{\\gamk}{\\gamma_{k-1}}\n\t\t\t\\quad\\text{and}\\quad\n\t\t\t\tP_k\\coloneqq\\varphi(x^k)-\\min\\varphi.\n\t\t\t\\end{equation}","sha256_utf8":"e35c4cbc39abfdc07fab9bc144057aec05609890af743d2bb9ffa8c5b996697d"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{thm:PG:convergence}%\n\t\t\t\tSuppose that \\cref{ass:PG} holds, and consider the iterates generated by \\refadaPG*.\n\t\t\t\tThen, for any \\(x^\\star\\in\\argmin\\varphi\\), with \\(\\rhok\\) and \\(P_k\\) as in \\eqref{eq:rhoP} and with \\(\\Uk(x^\\star)\\) defined as\n\t\t\t\t\\[\n\t\t\t\t\t\\Uk (x^\\star) \t\t\t\t\\coloneqq\n\t\t\t\t\t\\tfrac12\\|x^k-x^\\star\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac14\\|x^k-x^{k-1}\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\gamk\\left(1+\\rhok\\right)P_{k-1},\n\t\t\t\t\\]\n\t\t\t\tthe following hold:\n\t\t\t\t\\begin{enumerate}[widest*=4]\n\t\t\t\t\\item \\label{thm:PG:SD}%\n\t\t\t\t\tFor all \\(k\\geq1\\),\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\Uk*(x^\\star)\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\Uk (x^\\star)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{1}{4}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\rhok*^2\\gamk\\Lk(\\gamk\\ck-1)\n\t\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\t\\|x^{k-1}-x^k\\|^2\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\gamk\\bigl(\n\t\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\\vphantom{\\tfrac{1}{4}}\n\t\t\t\t\t\t\t\t{1+\\rhok-\\rhok*^2}\n\t\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\\bigr)P_{k-1}\n\t\t\t\t\t\\).\n\t\t\t\t\\item \\label{thm:PG:gamk}%\n\t\t\t\t\tThe sequence \\(\\seq{x^k}\\) is bounded and \\(\\gamk>\\frac{1}{2L_{f,\\V}}\\) holds for every \\(k\\geq k_0\\coloneqq\\left\\lfloor2\\log_2\\frac{1}{2\\gamma_0L_{f,\\V}}\\right\\rfloor_+\\), where \\(L_{f,\\V}\\) is a Lipschitz modulus for \\(\\nabla f\\) on a compact convex set \\(\\V\\) containing \\(\\seq{x^k}\\).\n\n\t\t\t\t\\item \\label{thm:PG:x*}%\n\t\t\t\t\tThe sequence \\(\\seq{x^k}\\) converges to a solution.\n\t\t\t\t\tThe claim remains true if the stepsizes are chosen in such a way that \\eqref{eq:PG:gamk} holds with ``\\(\\leq\\)'', as long as \\(\\seq{\\gamk}\\) is bounded away from zero.\n\t\t\t\t\t\\item \\label{thm:PG:sublin}%\n\t\t\t\t\tThe following rate holds\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\min_{k=0,1,\\ldots, K} (\\varphi(x^k) - \\min \\varphi)\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\frac{\\U_1(x^\\star)}{\\sum_{k=1}^{K+1}\\gamk}, \t\t\t\t\t\\]\n\t\t\t\t\twhich by assertion \\itemref{thm:PG:gamk} implies a sublinear \\(\\mathcal O(\\nicefrac1K)\\) convergence rate on the best-so-far cost.\n\t\t\t\t\t\\end{enumerate}\n\t\t\t\t\n\n\n\t\t\t\\end{theorem}","units":[{"start":39578,"end":41451,"start_line":797,"end_line":847,"raw_tex":"\\begin{theorem}\\label{thm:PG:convergence}%\n\t\t\t\tSuppose that \\cref{ass:PG} holds, and consider the iterates generated by \\refadaPG*.\n\t\t\t\tThen, for any \\(x^\\star\\in\\argmin\\varphi\\), with \\(\\rhok\\) and \\(P_k\\) as in \\eqref{eq:rhoP} and with \\(\\Uk(x^\\star)\\) defined as\n\t\t\t\t\\[\n\t\t\t\t\t\\Uk (x^\\star) \t\t\t\t\\coloneqq\n\t\t\t\t\t\\tfrac12\\|x^k-x^\\star\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac14\\|x^k-x^{k-1}\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\gamk\\left(1+\\rhok\\right)P_{k-1},\n\t\t\t\t\\]\n\t\t\t\tthe following hold:\n\t\t\t\t\\begin{enumerate}[widest*=4]\n\t\t\t\t\\item \\label{thm:PG:SD}%\n\t\t\t\t\tFor all \\(k\\geq1\\),\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\Uk*(x^\\star)\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\Uk (x^\\star)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{1}{4}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\rhok*^2\\gamk\\Lk(\\gamk\\ck-1)\n\t\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\t\\|x^{k-1}-x^k\\|^2\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\gamk\\bigl(\n\t\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\\vphantom{\\tfrac{1}{4}}\n\t\t\t\t\t\t\t\t{1+\\rhok-\\rhok*^2}\n\t\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\\bigr)P_{k-1}\n\t\t\t\t\t\\).\n\t\t\t\t\\item \\label{thm:PG:gamk}%\n\t\t\t\t\tThe sequence \\(\\seq{x^k}\\) is bounded and \\(\\gamk>\\frac{1}{2L_{f,\\V}}\\) holds for every \\(k\\geq k_0\\coloneqq\\left\\lfloor2\\log_2\\frac{1}{2\\gamma_0L_{f,\\V}}\\right\\rfloor_+\\), where \\(L_{f,\\V}\\) is a Lipschitz modulus for \\(\\nabla f\\) on a compact convex set \\(\\V\\) containing \\(\\seq{x^k}\\).\n\n\t\t\t\t\\item \\label{thm:PG:x*}%\n\t\t\t\t\tThe sequence \\(\\seq{x^k}\\) converges to a solution.\n\t\t\t\t\tThe claim remains true if the stepsizes are chosen in such a way that \\eqref{eq:PG:gamk} holds with ``\\(\\leq\\)'', as long as \\(\\seq{\\gamk}\\) is bounded away from zero.\n\t\t\t\t\t\\item \\label{thm:PG:sublin}%\n\t\t\t\t\tThe following rate holds\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\min_{k=0,1,\\ldots, K} (\\varphi(x^k) - \\min \\varphi)\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\frac{\\U_1(x^\\star)}{\\sum_{k=1}^{K+1}\\gamk}, \t\t\t\t\t\\]\n\t\t\t\t\twhich by assertion \\itemref{thm:PG:gamk} implies a sublinear \\(\\mathcal O(\\nicefrac1K)\\) convergence rate on the best-so-far cost.\n\t\t\t\t\t\\end{enumerate}\n\t\t\t\t","sha256_utf8":"248e29c2c371925f9febb932e3c43160fc898567a8fce71934328c98e6ccac99"},{"start":54121,"end":54138,"start_line":1155,"end_line":1156,"raw_tex":"\n\t\t\t\\end{theorem}","sha256_utf8":"6727a931b7123809dbadf28e49e5715f1d3264affc29f43f789dd7f49c419214"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\\label{thm:PG:cLl}%\n\t\t\t\tSuppose that \\cref{ass:PG} holds, and let \\(x^{k-1},x^k\\in\\R^n\\).\n\t\t\t\tThen, with \\(L_k\\), \\(\\Lk\\), and \\(\\ck\\) as in \\eqref{eq:MML_k} and \\eqref{eq:CL} the following hold:\n\t\t\t\t\\begin{enumerate}\n\t\t\t\t\\item \\label{thm:PG:CL}%\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\|\\nabla f(x^{k-1})-\\nabla f(x^k)\\|^2\n\t\t\t\t\t=\n\t\t\t\t\t\t\\ck\\Lk\\|x^{k-1}-x^k\\|^2\n\t\t\t\t\t\\), that is, \\(\\ck\\Lk=L_k^2\\).\n\t\t\t\t\\item \\label{thm:PG:H}%\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\|\\Hk(x^{k-1})-\\Hk(x^k)\\|^2\n\t\t\t\t\t=\n\t\t\t\t\t\t\\bigl(1-\\gamk\\Lk(2-\\gamk\\ck)\\bigr)\n\t\t\t\t\t\t\\|x^{k-1}-x^k\\|^2\n\t\t\t\t\t\\).\n\t\t\t\t\\item \\label{thm:Lk<=Ck}%\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\Lk\n\t\t\t\t\t\\leq\n\t\t\t\t\t\tL_k\n\t\t\t\t\t\\leq\n\t\t\t\t\t\tL_{f,\\V}\n\t\t\t\t\t\\)\n\t\t\t\t\tand \\(L_k\\leq\\ck\\), where \\(L_{f,\\V}\\) is a Lipschitz modulus for \\(\\nabla f\\) on a compact convex set \\(\\V\\) containing \\(x^{k-1}\\) and \\(x^k\\).\n\t\t\t\t\\end{enumerate}\n\t\t\t\t\\begin{proof}\n\t\t\t\t\tThe first assertion is of trivial verification, and similarly the third one follows from the Cauchy-Schwarz inequality and the Baillon-Haddad theorem \\cite[Cor. 10]{baillon1977quelques}, see also \\cite[Cor. 18.17]{bauschke2017convex}.\n\t\t\t\t\tTo conclude, observe that\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\|\\Hk(x^{k-1})-\\Hk(x^k)\\|^2\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\|x^{k-1}-x^k\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\gamk^2\n\t\t\t\t\t\t\\|\\nabla f(x^{k-1})-\\nabla f(x^k)\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t-\n\t\t\t\t\t\t2\\gamk\\innprod{\\nabla f(x^{k-1})-\\nabla f(x^k)}{x^{k-1}-x^k},\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tfrom which assertion \\itemref{thm:PG:H} follows.\n\t\t\t\t\\end{proof}\n\t\t\t\\end{lemma}\n\n\\begin{lemma}\\label{thm:PG:descent}%\n\t\t\t\tSuppose \\cref{ass:PG} holds, and consider a sequence \\(\\seq{x^k}\\) generated by {\\algnamefont PGM} iterations \\(x^{k+1}=\\FBk*{x^k}\\).\n\t\t\t\tThen%\n\t\t\t\t\\begin{align*}\n\t\t\t\t&\n\t\t\t\t\t\\tfrac{1}{2}\\|x^{k+1}-x^\\star\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\gamk*(1+\\rhok*)P_k\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{1}{4}\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{1}{2}\\|x^k-x^\\star\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\rhok*\\gamk*P_{k-1}\n\t\t\t\t\t-\n\t\t\t\t\t\\rhok*^2\\gamk\\Lk \\left(\n\t\t\t \t\t\t1-\\gamk \\ck  \t\t\\right)\n\t\t\t\t\t\\|x^{k-1}-x^k\\|^2\n\t\t\t\t\\end{align*}\n\t\t\t\tholds for any \\(k\\geq1\\) and \\(x^\\star\\in\\argmin\\varphi\\), where \\(\\Lk\\) and \\(\\ck\\) are as in \\eqref{eq:CL}, and \\(\\rhok\\) and \\(P_k\\) as in \\eqref{eq:rhoP}.%\n\t\t\t\t\\begin{proof}\n\t\t\t\t\tThe subgradient characterization\n\t\t\t\t\t\\begin{equation}\\label{eq:PG:subgrad}\n\t\t\t\t\t\t\\tfrac{\\Hk*(x^k)-x^{k+1}}{\\gamk*}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\tfrac{x^k-x^{k+1}}{\\gamk*}-\\nabla f(x^k)\n\t\t\t\t\t\\in\n\t\t\t\t\t\t\\partial g(x^{k+1})\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\tof \\(x^{k+1}=\\FBk*{x^k}\\) implies that\n\t\t\t\t\t{\\mathtight[0.95]%\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\tg(x^\\star)-g(x^{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x^\\star- x^{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{\\gamk*}\\innprod{x^k-x^{k+1}}{x^\\star-x^{k+1}}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\tg(x^\\star)-g(x^{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x^\\star- x^{k+1}}\n\t\t\t\t\t\t\t}_{\\text{(A)}}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\gamk*}\\|x^k-x^\\star\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\gamk*}\\|x^{k+1}-x^\\star\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\gamk*}\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t}%\n\t\t\t\t\tholds for any solution \\(x^\\star\\).\n\t\t\t\t\tWe next proceed to upper bound the term (A) as\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\text{(A)}\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x^\\star-x^k}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x^k-x^{k+1}}\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x^\\star-x^k}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{\\gamk}\\innprod{\\Hk(x^{k-1})-x^k}{x^{k+1}-x^k}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{\\gamk}\\innprod{\\Hk(x^{k-1})-\\Hk(x^k)}{x^k-x^{k+1}}\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:PG:subgrad}}{} &\n\t\t\t\t\t\t\\fillwidthof[c]{\\innprod{\\nabla f(x^k)}{x^\\star-x^k}}{\n\t\t\t\t\t\t\tf(x^\\star) - f(x^k)\n\t\t\t\t\t\t}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\fillwidthof[c]{\\tfrac{1}{\\gamk}\\innprod{\\Hk(x^{k-1})-x^k}{x^{k+1}-x^k}}{\n\t\t\t\t\t\t\tg(x^{k+1})-g(x^k)\n\t\t\t\t\t\t}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\\tfrac{1}{\\gamk}\\innprod{\\Hk(x^{k-1})-\\Hk(x^k)}{x^k-x^{k+1}}\n\t\t\t\t\t\t}_{\\text{(B)}}.\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tWe bound the term (B) by Young's inequality with parameter \\(\\varepsilon_{k+1}\\) as\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\text{(B)}\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\tfrac{\\varepsilon_{k+1}}{2\\gamk}\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{2\\varepsilon_{k+1}\\gamk}\\|\\Hk(x^{k-1})-\\Hk(x^k)\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\overrel{\\ref{thm:PG:H}}{} &\n\t\t\t\t\t\t\\tfrac{\\varepsilon_{k+1}}{2\\gamk}\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1-\\gamk\\Lk(2-\\gamk\\ck)}{2\\varepsilon_{k+1}\\gamk}\\|x^{k-1}-x^k\\|^2.\n\t\t\t\t\t\t\\numberthis\\label{eq:B:CL}\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tLet \\(\\varphi_\\star \\coloneqq \\min \\varphi\\).\n\t\t\t\t\tThe three inequalities combined give\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t0\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\varphi_\\star-\\varphi(x^k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{2\\gamk*}\\|x^k-x^\\star\\|^2\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{1}{2\\gamk*}\\|x^{k+1}-x^\\star\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\left(\\tfrac{\\varepsilon_{k+1}}{2\\gamk}-\\tfrac{1}{2\\gamk*}\\right)\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1-\\gamk\\Lk(2-\\gamk\\ck)}{2\\varepsilon_{k+1}\\gamk}\\|x^{k-1}-x^k\\|^2.\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tUsing again the subgradient \\eqref{eq:PG:subgrad} (since \\(\\partial\\varphi=\\nabla f+\\partial g\\)) one has\n\t\t\t\t\t\\begin{equation}\\label{eq:PG:vk}\n\t\t\t\t\t\tv^k\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\tfrac{x^{k-1}-x^k}{\\gamk}-(\\nabla f(x^{k-1})-\\nabla f(x^k))\n\t\t\t\t\t\\in\n\t\t\t\t\t\t\\partial\\varphi(x^k),\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\thence, for any \\(\\mytheta_{k+1}\\geq0\\),\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t0\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\mytheta_{k+1}\\left(\n\t\t\t\t\t\t\t\\varphi(x^{k-1})-\\varphi(x^k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\innprod{v^k}{x^{k-1}-x^k}\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\mytheta_{k+1}\\left(\n\t\t\t\t\t\t\t\\varphi(x^{k-1})-\\varphi(x^k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{\\gamk}\\|x^k-x^{k-1}\\|^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla f(x^{k-1})-\\nabla f(x^k)}{x^{k-1}-x^k}\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\mytheta_{k+1}\\left(\n\t\t\t\t\t\t\t\\varphi(x^{k-1})-\\varphi(x^k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1-\\gamk\\Lk}{\\gamk}\\|x^k-x^{k-1}\\|^2\n\t\t\t\t\t\t\\right).\n\t\t\t\t\t\t\\numberthis\\label{eq:PG:subgradineq}\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tBy summing the last two inequalities, multiplying by \\(\\gamk*\\), and observing that\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\varphi_\\star-\\varphi(x^k)+\\mytheta_{k+1}(\\varphi(x^{k-1})-\\varphi(x^k))\n\t\t\t\t\t=\n\t\t\t\t\t\t\\mytheta_{k+1}P_{k-1}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t(1+\\mytheta_{k+1})P_k,\n\t\t\t\t\t\\]\n\t\t\t\t\twe obtain\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t&\n\t\t\t\t\t\t\\tfrac{1}{2}\\|x^{k+1}-x^\\star\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\gamk*(1+\\mytheta_{k+1})P_k\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1-\\varepsilon_{k+1}\\rhok*}{2}\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\tfrac{1}{2}\\|x^k-x^\\star\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\mytheta_{k+1}\\gamk*P_{k-1}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\rhok*\\left(\n\t\t\t\t\t\t\t\\tfrac{1-\\gamk\\Lk(2-\\gamk\\ck)}{2\\varepsilon_{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\mytheta_{k+1}(1-\\gamk\\Lk)\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\\|x^{k-1}-x^k\\|^2.\n\t\t\t\t\t\\numberthis\\label{eq:PG:descentdelta}\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tSelecting \\(\\mytheta_{k+1}=\\rhok*\\) and \\(\\varepsilon_{k+1}=\\nicefrac{1}{2\\rhok*}\\) results in the claimed inequality.\n\t\t\t\t\t\\qedhere\n\t\t\t\t\\end{proof}\n\t\t\t\\end{lemma}\n\n\\begin{proof}~\n\t\t\t\t\t\\begin{proofitemize}\n\t\t\t\t\t\\item \\itemref{thm:PG:SD}~\n\t\t\t\t\t\tExpressed in terms of \\(\\U_k(x^\\star)\\), the inequality in \\cref{thm:PG:descent} simplifies to the one in the statement. The update rule for \\(\\gamk\\) implies that the coefficients indicated in the statement are greater than or equal to zero, and the claim follows.\n\n\t\t\t\t\t\\item \\itemref{thm:PG:gamk}~\n\t\t\t\t\t\tThe proven inequality implies that \\(\\seq{\\U_k(x_{}^{\\star })}\\) converges for any \\(x^\\star\\in\\argmin\\varphi\\).\n\t\t\t\t\t\tTaking into account the nonnegativity of the last two terms in the definition of \\(\\U_k(x^\\star)\\), it follows that \\(\\seq{x^k}\\) is bounded.\n\t\t\t\t\t\tAs such, \\(\\V\\) and \\(L_{f,\\V}\\) as in the statement exist.\n\t\t\t\t\t\t\tObserve that\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:relate:MM}\n\t\t\t\t\t\t\t\t\\tfrac{\\gamk}{2\\sqrt{[\\gamk\\Lk(\\gamk\\ck-1)]_+}}\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\tfrac12\n\t\t\t\t\t\t\t\t\\sqrt{\\tfrac{1}{\\Lk\\ck}\\tfrac{\\gamk\\ck}{[\\gamk\\ck-1]_+}}\n\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t\\tfrac{1}{2\\sqrt{\\Lk\\ck}}\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\tfrac{1}{2L_k},\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\twhere the last identity owes to \\cref{thm:PG:CL}, and therefore\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:MM}\n\t\t\t\t\t\t\t\t\\gamk*\n\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t\\min\\set{\\gamk\\sqrt{1+\\rhok},\\tfrac{1}{2L_k}}\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tholds for every \\(k\\).\n\t\t\t\t\t\t\t\t\t\t\t\tSuppose that \\(\\gamk\\leq\\nicefrac{1}{2L_{f,\\V}}\\) for \\(k=0,\\dots,K\\).\n\t\t\t\t\t\t\tBecause of \\eqref{eq:MM} and \\cref{thm:Lk<=Ck}, this implies that\n\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\\gamk*\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\gamk\\sqrt{1+\\rhok}\n\t\t\t\t\t\t\t\\)\n\t\t\t\t\t\t\tfor \\(k=0,\\dots,K-1\\).\n\t\t\t\t\t\t\t\t\t\t\t\tThen, for these \\(k\\) one has that\n\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\\rhok*^2\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t1+\\rhok\n\t\t\t\t\t\t\t>\n\t\t\t\t\t\t\t\t1\n\t\t\t\t\t\t\t\\),\n\t\t\t\t\t\t\twhich inductively gives\n\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\\rhok^2\n\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t2\n\t\t\t\t\t\t\t\\)\n\t\t\t\t\t\t\tfor all \\(k=1,\\dots,K\\) (since \\(\\rho_0 \\geq 1\\)).\n\t\t\t\t\t\t\tWe then have\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:PG:gam2k}\n\t\t\t\t\t\t\t\t\\gamma_K^2\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\rho_K^2\\gamma_{K-1}^2\n\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t2\\gamma_{K-1}^2\n\t\t\t\t\t\t\t\\geq\\dots\\geq\n\t\t\t\t\t\t\t\t2^K\\gamma_0^2, \t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tshowing that in at most \\(k_0\\) iterations stepsize exceeds \\(\\nicefrac{1}{2L_{f,\\V}}\\).\n\t\t\t\t\t\t\tFrom this point, a trivial induction argument using \\eqref{eq:MM} reveals that \\(\\gamk\\geq\\nicefrac{1}{2L_{f,\\V}}\\) for all \\(k\\geq k_0\\).\n\t\t\t\t\t\\item \\itemref{thm:PG:x*}~\n\t\t\t\t\t\tWe begin by observing that, as is apparent from its proof, assertion \\itemref{thm:PG:SD} remains valid if the identity in \\eqref{eq:PG:gamk} is replaced by an inequality ``\\(\\leq\\)''.\n\t\t\t\t\t\tEither way, a telescoping argument yields that\n\t\t\t\t\t\t\\begin{equation}\\label{eq:gPto0}\n\t\t\t\t\t\t\t\\gamk(1+\\rhok-\\rhok*^2)P_{k-1}\\to0\n\t\t\t\t\t\t\\quad\\text{as}\\quad\n\t\t\t\t\t\t\tk\\to\\infty.\n\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\tWe proceed by intermediate claims.\n\t\t\t\t\t\t\\begin{claims}[widest*=3]%\n\t\t\t\t\t\t\\item \\label{claim:infP}\n\t\t\t\t\t\t\t{\\it \\(\\liminf_{k\\to\\infty}P_k=0\\), and in particular \\(\\seq{x^k}\\) admits a limit point \\(x_\\infty \\in \\argmin \\varphi \\).}\n\n\t\t\t\t\t\t\tSince \\(\\seq{x^k}\\) is bounded and \\(\\varphi\\) is lsc, it suffices to show that \\(\\liminf_{k\\to\\infty}P_k=0\\).\n\t\t\t\t\t\t\tIf \\(\\limsup_{k\\to\\infty}(1+\\rhok-\\rhok*^2)>0\\), then \\eqref{eq:gPto0} yields the claim, since \\(\\gamk\\) is bounded away from zero.\n\t\t\t\t\t\t\tAlternatively, since \\(1+\\rhok-\\rhok*^2\\geq0\\), necessarily \\(1+\\rhok-\\rhok*^2\\to0\\), from which it easily follows that \\(\\liminf_{k\\to\\infty}\\rhok>1\\), and that therefore \\(\\gamk\\to\\infty\\);\n\t\t\t\t\t\t\tin this case, because of the inequalities \\(\\gamk\\left(1+\\rhok\\right)P_{k-1}\\leq\\U_k\\leq\\U_1\\), this directly proves that \\(P_k\\to0\\).\n\n\t\t\t\t\t\t\\item\\label{claim:thm:PG:optunique}%\n\t\t\t\t\t\t\t{\\it \\(x_\\infty\\) is the only limit point that belongs to \\(\\argmin\\varphi\\).}\n\n\t\t\t\t\t\t\tSuppose that \\(x_\\infty'\\in\\argmin\\varphi\\) is a limit point.\n\t\t\t\t\t\t\tObserve that\n\t\t\t\t\t\t\t\\[\n\t\t\t\t\t\t\t\t\\innprod{x^k}{x_\\infty - x_\\infty'}\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\U_k(x_\\infty') - \\U_k(x_\\infty) + \\tfrac12\\|x_\\infty\\|^2 - \\tfrac12\\|x_\\infty'\\|^2,\n\t\t\t\t\t\t\t\\]\n\t\t\t\t\t\t\tand since \\(\\seq{\\U_k(x^\\star)}\\) is convergent for all \\(x^\\star\\in \\argmin \\varphi\\), then so is \\(\\seq{\\innprod{x^k}{x_\\infty - x_\\infty'}}\\).\n\t\t\t\t\t\t\tPassing to the limit along the two converging subsequences thus yields\n\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\\innprod{x_\\infty}{x_\\infty - x_\\infty'}\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\innprod{x_\\infty'}{x_\\infty - x_\\infty'}\n\t\t\t\t\t\t\t\\),\n\t\t\t\t\t\t\twhich after rearranging results in \\(\\|x_\\infty - x_\\infty'\\|^2=0\\), establishing the claim.\n\n\t\t\t\t\t\t\\item\\label{claim:thm:PG:allopt}%\n\t\t\t\t\t\t\t{\\it \\(\\seq{x^k}\\) converges to a point in \\(\\argmin\\varphi\\).}\n\n\t\t\t\t\t\t\tLet \\(x_\\infty\\) be the (unique) optimal limit point of \\(\\seq{x^k}\\) as in \\cref{claim:thm:PG:optunique}, and let\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:U}\n\t\t\t\t\t\t\t\tU\\coloneqq\\lim_{k\\to\\infty}\\U_k(x_\\infty),\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\twhich exists since the sequence is monotonically decreasing by assertion \\itemref{thm:PG:SD}.\n\t\t\t\t\t\t\tBecause of the inequality \\(\\frac{1}{2}\\|x^k-x_{\\infty}\\|^2\\leq\\U_k(x_{\\infty})\\), it suffices to show that \\(U=0\\).\n\t\t\t\t\t\t\tObserve that, for every \\(k\\),\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:rhomax}\n\t\t\t\t\t\t\t\t\\rhok\\leq\\rho_{\\rm max}\n\t\t\t\t\t\t\t\\quad\\text{where}\\quad\n\t\t\t\t\t\t\t \t\\rho_{\\rm max}\n\t\t\t\t\t\t\t \\coloneqq\n\t\t\t\t\t\t\t\t\\max\\set{\\tfrac{\\gamma_0}{\\gamma_{-1}}, \\tfrac12(1+\\sqrt{5})}\n\t\t\t\t\t\t\t>\n\t\t\t\t\t\t\t\t1,\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tas it can be verified by induction.\n\t\t\t\t\t\t\tWe consider two mutually exclusive scenarios.\n\t\t\t\t\t\t\t\\begin{proofitemize}[leftmargin=*]\n\t\t\t\t\t\t\t\\item\n\t\t\t\t\t\t\t\tSuppose that \\(\\seq{\\gamk}\\) is bounded, and consider a subsequence \\(\\seq{x^k}[k\\in K]\\) such that \\(\\lim_{K\\ni k\\to\\infty}P_k=0\\), which exists and converges to \\(x_{\\infty}\\) by virtue of the previous claims.\n\t\t\t\t\t\t\t\tExtract \\(K'\\subseteq K\\) such that \\(\\seq{\\gamk*}[k\\in K']\\to\\gamma\\) for some \\(\\gamma>0\\).\n\t\t\t\t\t\t\t\tThen, \\(x^{k+1}=\\prox_{\\gamk*g}(x^k-\\gamk*\\nabla f(x^k))\\to\\prox_{\\gamma g}(x_\\infty-\\gamma\\nabla f(x_\\infty))=x_\\infty\\) as \\(K'\\ni k\\to\\infty\\) as well.\n\t\t\t\t\t\t\t\tThis yields the sought identity\n\t\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\tU\n\t\t\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\t\t\\lim_{k\\to\\infty}\\U_k(x_{\\infty})\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\lim_{K'\\ni k\\to\\infty}\\U_{k+1}(x_{\\infty})\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\t\t\\lim_{K'\\ni k\\to\\infty}\\left(\n\t\t\t\t\t\t\t\t\t\t\\tfrac{1}2\\|x^{k+1}-x_{\\infty}\\|^2\n\t\t\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\t\t\\tfrac{1}2\\|x^{k+1}-x^k\\|^2\n\t\t\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\t\t\\gamk*\\left(1+\\rhok*\\right)P_k\n\t\t\t\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\t\t0,\n\t\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\t\twhere the vanishing of the last term owes to boundedness of \\(\\seq{\\gamk}\\) and \\(\\seq{\\rhok}\\).\n\n\t\t\t\t\t\t\t\\item\n\t\t\t\t\t\t\t\tSuppose instead that \\(\\seq{\\gamk}\\) is unbounded, and let \\(\\seq{\\gamk}[k\\in K']\\) be an arbitrary subsequence such that \\(\\gamk\\to\\infty\\) as \\(K'\\ni k\\to\\infty\\).\n\t\t\t\t\t\t\t\tFrom the inequality\n\t\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\t\\gamk P_{k-1} \\leq \\U_k(x_{\\infty}) \\leq \\U_1(x_\\infty) < \\infty\n\t\t\t\t\t\t\t\t\\),\n\t\t\t\t\t\t\t\twe conclude that\n\t\t\t\t\t\t\t\t\\begin{equation}\\label{eq:Pk:xk-}\n\t\t\t\t\t\t\t\t\tP_{k-1}\\to0\n\t\t\t\t\t\t\t\t~\\text{ and }~\n\t\t\t\t\t\t\t\t\tx^{k-1}\\to x_\\infty\n\t\t\t\t\t\t\t\t\\text{ as }\n\t\t\t\t\t\t\t\t\tK'\\ni k\\to\\infty\n\t\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\t\t(recall that \\(\\varphi\\) is lsc, \\(\\seq{x^k}\\) is bounded, and \\(x_\\infty\\) is the \\emph{unique} optimal limit point).\n\t\t\t\t\t\t\t\tThen, since by \\eqref{eq:rhomax} \\(\\gamma_{k-1}\\geq\\nicefrac{\\gamk}{\\rho_{\\rm max}}\\to\\infty\\) as \\(K'\\ni k\\to\\infty\\), by the same argument also \\(x^{k-2}\\to x_\\infty\\) as \\(K'\\ni k\\to\\infty\\).\n\t\t\t\t\t\t\t\tWith \\(U\\) as in \\eqref{eq:U}, we thus have\n\t\t\t\t\t\t\t\t\\begin{equation}\\label{eq:U:gam:unbounded}\n\t\t\t\t\t\t\t\t\t\tU\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\lim_{k\\to\\infty}\\U_k(x_\\infty)\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\lim_{K'\\ni k\\to\\infty}\\U_{k-1}(x_\\infty)\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\lim_{K'\\ni k\\to\\infty}\\gamma_{k-1}(1+\\rho_{k-1})P_{k-2}\n\t\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\t\tholding for \\emph{any} set of indices \\(K'\\subseteq\\N\\) along which the stepsizes are divergent.\n\n\t\t\t\t\t\t\t\tWe now construct a specific subsequence \\(K\\coloneqq\\set{k_0,k_1,\\dots}\\) as follows: start with \\(k_0=1\\), and for \\(i\\geq0\\) let\n\t\t\t\t\t\t\t\t\\begin{equation}\\label{eq:ki+}\n\t\t\t\t\t\t\t\t\tk_{i+1}=\\min\\set{k\\geq k_i}[\\gamk \\geq \\rho_{\\rm max}\\gamma_{k_i}].\n\t\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\t\tThen, \\(\\seq{\\gamma_{k_i}}[i\\in\\N]\\to\\infty\\), which as argued after \\eqref{eq:Pk:xk-} implies that both \\(\\seq{x^{k_i-1}}[i\\in\\N]\\) and \\(\\seq{x^{k_i-2}}[i\\in\\N]\\) converge to \\(x_\\infty\\).\n\t\t\t\t\t\t\t\tObserve that \\(\\rho_{k_i}>1\\) holds for all \\(i\\in\\N\\) by minimality of \\(k_i\\).\n\t\t\t\t\t\t\t\tNotice further that\n\t\t\t\t\t\t\t\t\\begin{equation}\\label{eq:k-1}\n\t\t\t\t\t\t\t\t\t\\rho_{k_i-1}\\geq\\rho_{\\rm max}^{-1}\n\t\t\t\t\t\t\t\t\\quad\n\t\t\t\t\t\t\t\t\t\\forall i\\in\\N.\n\t\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\t\tTo see why, suppose on the contrary that\n\t\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\t\\rho_{k_i-1}<\\rho_{\\rm max}^{-1}\n\t\t\t\t\t\t\t\t\\);\n\t\t\t\t\t\t\t\tthen in particular \\(\\rho_{k_i-1}<1\\), thereby ensuring by the previous observation that \\(k_i-1\\notin K\\), and thus \\(k_{i-1}\\leq k_i-2\\).\n\t\t\t\t\t\t\t\tThis leads to the contradiction\n\t\t\t\t\t\t\t\t\\[\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_{i-1}}\n\t\t\t\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:ki+}}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i}\n\t\t\t\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:rhomax}}\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-1}\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\rho_{k_i-1}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-2}\n\t\t\t\t\t\t\t\t<\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-2}\n\t\t\t\t\t\t\t\t<\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_{i-1}},\n\t\t\t\t\t\t\t\t\\]\n\t\t\t\t\t\t\t\twhere the last inequality follows either in case \\(k_i-2=k_{i-1}\\) (since \\(\\rho_{\\rm max}>1\\)) or when \\(k_{i-1}<k_i-2<k_i\\) (from minimality in the definition of \\(k_i\\)).\n\t\t\t\t\t\t\t\tThis shows \\eqref{eq:k-1}.\n\n\t\t\t\t\t\t\t\tNext, by the stepsize update at \\cref{state:PG:gamk},\n\t\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\t\\tfrac{1}{\\rho_{\\rm max}}\\gamma_{k_i-2}L_{k_i-2}\n\t\t\t\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:k-1}}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-1}L_{k_i-2}\n\t\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\t\\frac{\\gamma_{k_i-2}L_{k_i-2}}{2\\sqrt{\\left[\\gamma_{k_i-2}^2\\ell_{k_i-2}c_{k_i-2}-\\gamma_{k_i-2}\\ell_{k_i-2}\\right]_+}}\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\t\\frac{\\gamma_{k_i-2}L_{k_i-2}}{2\\sqrt{\\left[\\gamma_{k_i-2}^2L_{k_i-2}^2-\\gamma_{k_i-2}L_{k_i-2}\\right]_+}}\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\t\t\\frac{1}{2\\sqrt{\\left[1-\\frac{1}{\\gamma_{k_i-2}L_{k_i-2}}\\right]_+}},\n\t\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\t\twhere the third inequality uses the fact that \\(L_{k_i-2}^2 = \\ell_{k_i-2}c_{k_i-2}\\), and \\(\\ell_{k_i-2}\\leq L_{k_i-2}\\) by \\cref{thm:PG:cLl}.\n\t\t\t\t\t\t\t\tThis implies that \\(\\seq{\\gamma_{k_i-2}L_{k_i-2}}[i\\in\\N]\\) must be bounded.\n\t\t\t\t\t\t\t\tThe subgradient characterization in \\eqref{eq:PG:vk} and the Cauchy-Schwarz inequality then yield\n\t\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-1} P_{k_i-2}\n\t\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\t\\rho_{k_i-1}\n\t\t\t\t\t\t\t\t\t\\innprod{x^{k_i-2}-x_{\\infty}}{\n\t\t\t\t\t\t\t\t\t\tx^{k_i-2}-x^{k_i-3}\n\t\t\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\t\t\\gamma_{k_i-2}(\\nabla f(x^{k_i-2})-\\nabla f(x^{k_i-3}))\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\t(1+\\gamma_{k_i-2}L_{k_i-2})\n\t\t\t\t\t\t\t\t\t}_{\\text{bounded}}\n\t\t\t\t\t\t\t\t\t\\,\n\t\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\t\\|x^{k_i-2}-x_{\\infty}\\|\n\t\t\t\t\t\t\t\t\t\t\\|x^{k_i-2}-x^{k_i-3}\\|\n\t\t\t\t\t\t\t\t\t}_{\\to0}\n\t\t\t\t\t\t\t\t\\to\n\t\t\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\t\t\\quad\n\t\t\t\t\t\t\t\t\t\\text{as }i\\to\\infty.\n\t\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\t\tCombined with the fact that \\(\\rho_{k_i-1} \\leq \\rho_{\\rm max}\\), it follows from \\eqref{eq:U:gam:unbounded} that \\(U=0\\).\n\t\t\t\t\t\t\t\\end{proofitemize}\n\t\t\t\t\t\t\t\t\t\t\t\t\\end{claims}%\n\n\t\t\t\t\t\\item \\itemref{thm:PG:sublin}~\n\t\t\t\t\t\tA telescoping argument in the descent inequality of assertion \\itemref{thm:PG:SD} yields\n\t\t\t\t\t\t\\[\n\t\t\t\t\t\t\t\\mathcal U_{K+1} + \\sum_{k=1}^K\\gamk\\left(1+\\rhok-\\rhok*^2\\right)P_{k-1} \\leq \\mathcal U_1\n\t\t\t\t\t\t\\]\n\t\t\t\t\t\tfor \\(K\\geq1\\).\n\t\t\t\t\t\tBy further lower bounding \\(\\mathcal U_{K+1}\\) we obtain\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\textstyle\n\t\t\t\t\t\t\t\\gamma_{K+1}(1+\\rho_{K+1})P_K\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\sum_{k=1}^K\\gamk\\left(1+\\rhok-\\rhok*^2\\right)P_{k-1}\n\t\t\t\t\t\t\t\\leq \\mathcal U_1.\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\tConsidering the best-so-far quantity\n\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\\min_{k=0,1,\\ldots, K} P_k\n\t\t\t\t\t\t\\), it follows that\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\min_{k=0,1,\\dots, K} P_k\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\frac{\\mathcal U_1}{\n\t\t\t\t\t\t\t\t\\gamma_{K+1}(1+\\rho_{K+1})\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\sum_{k=1}^K\\gamk(1+\\rhok-\\rhok*^2)\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\frac{\\mathcal U_1}{\n\t\t\t\t\t\t\t\t\\gamma_{K+1}(1+\\rho_{K+1})\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\sum_{k=1}^K\\gamk\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\sum_{k=1}^K(\\gamk\\rhok-\\gamk*\\rhok*)\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\numberthis\\label{eq:O(1k)}\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\frac{\\mathcal U_1}{\n\t\t\t\t\t\t\t\t\\gamma_1\\rho_1+\\sum_{k=1}^{K+1}\\gamk\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\leq\n\t\t\t\t\t\t\t\\frac{\\mathcal U_1}{\n\t\t\t\t\t\t\t\t(K+1)\\gamma_{\\rm min}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\twhere \\(\\gamma_{\\rm min}\\coloneqq\\min\\set{\\gamma_0,\\frac{1}{2L_{f,\\mathcal V}}}\\) is a lower bound on the stepsize, as demonstrated in the proof of assertion \\itemref{thm:PG:gamk}.\n\t\t\t\t\t\t\\qedhere\n\t\t\t\t\t\\end{proofitemize}\n\t\t\t\t\\end{proof}\n\n\\begin{lemma}\\label{lem:enlarged:ck}\n\t\t\t\tSuppose that \\cref{ass:f} holds.\n\t\t\t\tThen, for every \\(x,y\\in\\R^n\\) it holds that\n\t\t\t\t\\[\n\t\t\t\t\tf(y)\n\t\t\t\t\\leq\n\t\t\t\t\tf(x)\n\t\t\t\t\t+\n\t\t\t\t\t\\innprod{\\nabla f(x)}{y-x}\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{1}{2L_{f,\\overline\\V}}\n\t\t\t\t\t\\|\\nabla f(x) - \\nabla f(y)\\|^{2},\n\t\t\t\t\\]\n\t\t\t\twhere \\(L_{f,\\overline{\\V}}\\) is a Lipschitz modulus for \\(\\nabla f\\) on \\(\\overline{\\V}\\coloneqq\\V+\\cball{0}{\\diam(\\V)}\\), and \\(\\V\\subseteq\\R^n\\) is a bounded and convex set that contains \\(x\\) and \\(y\\).\n\t\t\t\tIn particular,\n\t\t\t\t\\[\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\|\\nabla f(x) - \\nabla f(y)\\|^2\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\innprod{\\nabla f(x) - \\nabla f(y)}{x-y}\n\t\t\t\t\t}\n\t\t\t\t\\leq\n\t\t\t\t\tL_{f,\\overline\\V}.\n\t\t\t\t\\]\n\t\t\t\t\\begin{proof}\n\t\t\t\t\tWe henceforth fix \\(x,y\\in \\V\\subseteq\\overline\\V\\).\n\t\t\t\t\tSince $\\nabla f$ is Lipschitz continuous on $\\overline\\V$ with modulus \\(L_{f,\\overline\\V}\\), it follows from the descent inequality \\cite[Prop. A.24]{bertsekas2016nonlinear} that\n\t\t\t\t\t\\begin{equation}\\label{eq:holder_ineq}\n\t\t\t\t\t\tf(z)\n\t\t\t\t\t\\leq\n\t\t\t\t\t\tf(y)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla f(y)}{z-y}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{L_{f,\\overline\\V}}{2}\\|z-y\\|^{2}\n\t\t\t\t\t\\quad\n\t\t\t\t\t\t\\forall z \\in \\overline\\V.\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\tLet \\(l_x(y) \\coloneqq f(y) - f(x) - \\innprod{\\nabla f(x)}{y-x}\\), and note that \\(l_x\\) is a convex function with \\(\\nabla l_x(y)=\\nabla f(y)-\\nabla f(x)\\).\n\t\t\t\t\tFor any \\(z \\in \\overline\\V\\), we have\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\tl_x(z)\n\t\t\t\t\t={} &\n\t\t\t\t\t\tf(z) - f(x) - \\innprod{\\nabla f(x)}{z-x}\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:holder_ineq}}{} &\n\t\t\t\t\t\tf(y)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla f(y)}{z-y}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{L_{f,\\overline\\V}}{2}\\|z-y\\|^{2}\n\t\t\t\t\t\t-\n\t\t\t\t\t\tf(x)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\innprod{\\nabla f(x)}{z-x}\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\tf(y) - f(x) - \\innprod{\\nabla f(x)}{y-x}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla f(y) - \\nabla f(x)}{z-y}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{L_{f,\\overline\\V}}{2}\\|z-y\\|^{2}\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\tl_x(y)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla l_x(y)}{z-y}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{L_{f,\\overline\\V}}{2}\\|z-y\\|^{2}.\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tNoticing that $\\nabla l_x(x) = 0$, it follows from convexity of \\(l_x\\) that $x$ is its global minimizer. Hence that \\(\\min l_x=l_x(x)=0\\).\n\t\t\t\t\tLet us denote \\(v\\coloneqq\\frac{1}{\\|\\nabla l_x(y)\\|}\\nabla l_x(y)\\) and set\n\t\t\t\t\t\\(\n\t\t\t\t\t\tz = y - \\frac{\\|\\nabla l_x(y)\\|}{L_{f,\\overline\\V}}v\n\t\t\t\t\t\\).\n\t\t\t\t\tNote that\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\|z-y\\|\n\t\t\t\t\t=\n\t\t\t\t\t\t\\tfrac{\\|\\nabla l_x(y)\\|}{L_{f,\\overline\\V}}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\tfrac{\\|\\nabla f(y)-\\nabla f(x)\\|}{L_{f,\\overline\\V}}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\|y-x\\|,\n\t\t\t\t\t\\]\n\t\t\t\t\tand in particular \\(z\\in \\V+\\cball{y}{\\|y-x\\|}\\subseteq\\overline\\V\\).\n\t\t\t\t\tFrom the previous inequality we get\n\t\t\t\t\t\\begin{qedalign*}\n\t\t\t\t\t\t0\n\t\t\t\t\t=\n\t\t\t\t\t\t\\min l_x\n\t\t\t\t\t=\n\t\t\t\t\t\tl_x(x)\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\tl_x\\Bigl(\n\t\t\t\t\t\t\ty - \\tfrac{\\|\\nabla l_x(y)\\|}{L_{f,\\overline\\V}}v\n\t\t\t\t\t\t\\Bigr)\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:holder_ineq}}{} &\n\t\t\t\t\t\tl_x(y)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{\\|\\nabla l_x(y)\\|}{L_{f,\\overline\\V}}\n\t\t\t\t\t\t\\innprod{\\nabla l_x(y)}{v}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{L_{f,\\overline\\V}}{2}\n\t\t\t\t\t\t\\tfrac{1}{L_{f,\\overline\\V}^{2}}\n\t\t\t\t\t\t\\|\\nabla l_x(y)\\|^{2}\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\tf(y) - f(x) - \\innprod{\\nabla f(x)}{y-x}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{1}{2L_{f,\\overline\\V}}\n\t\t\t\t\t\t\\|\\nabla f(x) - \\nabla f(y)\\|^{2}.\n\t\t\t\t\t\\end{qedalign*}\n\t\t\t\t\\end{proof}\n\t\t\t\\end{lemma}","units":[{"start":30817,"end":32262,"start_line":547,"end_line":591,"raw_tex":"\\begin{lemma}\\label{thm:PG:cLl}%\n\t\t\t\tSuppose that \\cref{ass:PG} holds, and let \\(x^{k-1},x^k\\in\\R^n\\).\n\t\t\t\tThen, with \\(L_k\\), \\(\\Lk\\), and \\(\\ck\\) as in \\eqref{eq:MML_k} and \\eqref{eq:CL} the following hold:\n\t\t\t\t\\begin{enumerate}\n\t\t\t\t\\item \\label{thm:PG:CL}%\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\|\\nabla f(x^{k-1})-\\nabla f(x^k)\\|^2\n\t\t\t\t\t=\n\t\t\t\t\t\t\\ck\\Lk\\|x^{k-1}-x^k\\|^2\n\t\t\t\t\t\\), that is, \\(\\ck\\Lk=L_k^2\\).\n\t\t\t\t\\item \\label{thm:PG:H}%\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\|\\Hk(x^{k-1})-\\Hk(x^k)\\|^2\n\t\t\t\t\t=\n\t\t\t\t\t\t\\bigl(1-\\gamk\\Lk(2-\\gamk\\ck)\\bigr)\n\t\t\t\t\t\t\\|x^{k-1}-x^k\\|^2\n\t\t\t\t\t\\).\n\t\t\t\t\\item \\label{thm:Lk<=Ck}%\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\Lk\n\t\t\t\t\t\\leq\n\t\t\t\t\t\tL_k\n\t\t\t\t\t\\leq\n\t\t\t\t\t\tL_{f,\\V}\n\t\t\t\t\t\\)\n\t\t\t\t\tand \\(L_k\\leq\\ck\\), where \\(L_{f,\\V}\\) is a Lipschitz modulus for \\(\\nabla f\\) on a compact convex set \\(\\V\\) containing \\(x^{k-1}\\) and \\(x^k\\).\n\t\t\t\t\\end{enumerate}\n\t\t\t\t\\begin{proof}\n\t\t\t\t\tThe first assertion is of trivial verification, and similarly the third one follows from the Cauchy-Schwarz inequality and the Baillon-Haddad theorem \\cite[Cor. 10]{baillon1977quelques}, see also \\cite[Cor. 18.17]{bauschke2017convex}.\n\t\t\t\t\tTo conclude, observe that\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\|\\Hk(x^{k-1})-\\Hk(x^k)\\|^2\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\|x^{k-1}-x^k\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\gamk^2\n\t\t\t\t\t\t\\|\\nabla f(x^{k-1})-\\nabla f(x^k)\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t-\n\t\t\t\t\t\t2\\gamk\\innprod{\\nabla f(x^{k-1})-\\nabla f(x^k)}{x^{k-1}-x^k},\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tfrom which assertion \\itemref{thm:PG:H} follows.\n\t\t\t\t\\end{proof}\n\t\t\t\\end{lemma}","sha256_utf8":"35cc9497f232e7a84a36468c48a55ae8d7ff4bfd826fb7e39f53ecc8332d2189"},{"start":33220,"end":38514,"start_line":602,"end_line":785,"raw_tex":"\\begin{lemma}\\label{thm:PG:descent}%\n\t\t\t\tSuppose \\cref{ass:PG} holds, and consider a sequence \\(\\seq{x^k}\\) generated by {\\algnamefont PGM} iterations \\(x^{k+1}=\\FBk*{x^k}\\).\n\t\t\t\tThen%\n\t\t\t\t\\begin{align*}\n\t\t\t\t&\n\t\t\t\t\t\\tfrac{1}{2}\\|x^{k+1}-x^\\star\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\gamk*(1+\\rhok*)P_k\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{1}{4}\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{1}{2}\\|x^k-x^\\star\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\rhok*\\gamk*P_{k-1}\n\t\t\t\t\t-\n\t\t\t\t\t\\rhok*^2\\gamk\\Lk \\left(\n\t\t\t \t\t\t1-\\gamk \\ck  \t\t\\right)\n\t\t\t\t\t\\|x^{k-1}-x^k\\|^2\n\t\t\t\t\\end{align*}\n\t\t\t\tholds for any \\(k\\geq1\\) and \\(x^\\star\\in\\argmin\\varphi\\), where \\(\\Lk\\) and \\(\\ck\\) are as in \\eqref{eq:CL}, and \\(\\rhok\\) and \\(P_k\\) as in \\eqref{eq:rhoP}.%\n\t\t\t\t\\begin{proof}\n\t\t\t\t\tThe subgradient characterization\n\t\t\t\t\t\\begin{equation}\\label{eq:PG:subgrad}\n\t\t\t\t\t\t\\tfrac{\\Hk*(x^k)-x^{k+1}}{\\gamk*}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\tfrac{x^k-x^{k+1}}{\\gamk*}-\\nabla f(x^k)\n\t\t\t\t\t\\in\n\t\t\t\t\t\t\\partial g(x^{k+1})\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\tof \\(x^{k+1}=\\FBk*{x^k}\\) implies that\n\t\t\t\t\t{\\mathtight[0.95]%\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\tg(x^\\star)-g(x^{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x^\\star- x^{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{\\gamk*}\\innprod{x^k-x^{k+1}}{x^\\star-x^{k+1}}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\tg(x^\\star)-g(x^{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x^\\star- x^{k+1}}\n\t\t\t\t\t\t\t}_{\\text{(A)}}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\gamk*}\\|x^k-x^\\star\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\gamk*}\\|x^{k+1}-x^\\star\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\gamk*}\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t}%\n\t\t\t\t\tholds for any solution \\(x^\\star\\).\n\t\t\t\t\tWe next proceed to upper bound the term (A) as\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\text{(A)}\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x^\\star-x^k}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x^k-x^{k+1}}\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x^\\star-x^k}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{\\gamk}\\innprod{\\Hk(x^{k-1})-x^k}{x^{k+1}-x^k}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{\\gamk}\\innprod{\\Hk(x^{k-1})-\\Hk(x^k)}{x^k-x^{k+1}}\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:PG:subgrad}}{} &\n\t\t\t\t\t\t\\fillwidthof[c]{\\innprod{\\nabla f(x^k)}{x^\\star-x^k}}{\n\t\t\t\t\t\t\tf(x^\\star) - f(x^k)\n\t\t\t\t\t\t}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\fillwidthof[c]{\\tfrac{1}{\\gamk}\\innprod{\\Hk(x^{k-1})-x^k}{x^{k+1}-x^k}}{\n\t\t\t\t\t\t\tg(x^{k+1})-g(x^k)\n\t\t\t\t\t\t}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\\tfrac{1}{\\gamk}\\innprod{\\Hk(x^{k-1})-\\Hk(x^k)}{x^k-x^{k+1}}\n\t\t\t\t\t\t}_{\\text{(B)}}.\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tWe bound the term (B) by Young's inequality with parameter \\(\\varepsilon_{k+1}\\) as\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\text{(B)}\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\tfrac{\\varepsilon_{k+1}}{2\\gamk}\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{2\\varepsilon_{k+1}\\gamk}\\|\\Hk(x^{k-1})-\\Hk(x^k)\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\overrel{\\ref{thm:PG:H}}{} &\n\t\t\t\t\t\t\\tfrac{\\varepsilon_{k+1}}{2\\gamk}\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1-\\gamk\\Lk(2-\\gamk\\ck)}{2\\varepsilon_{k+1}\\gamk}\\|x^{k-1}-x^k\\|^2.\n\t\t\t\t\t\t\\numberthis\\label{eq:B:CL}\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tLet \\(\\varphi_\\star \\coloneqq \\min \\varphi\\).\n\t\t\t\t\tThe three inequalities combined give\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t0\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\varphi_\\star-\\varphi(x^k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{2\\gamk*}\\|x^k-x^\\star\\|^2\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{1}{2\\gamk*}\\|x^{k+1}-x^\\star\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\left(\\tfrac{\\varepsilon_{k+1}}{2\\gamk}-\\tfrac{1}{2\\gamk*}\\right)\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1-\\gamk\\Lk(2-\\gamk\\ck)}{2\\varepsilon_{k+1}\\gamk}\\|x^{k-1}-x^k\\|^2.\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tUsing again the subgradient \\eqref{eq:PG:subgrad} (since \\(\\partial\\varphi=\\nabla f+\\partial g\\)) one has\n\t\t\t\t\t\\begin{equation}\\label{eq:PG:vk}\n\t\t\t\t\t\tv^k\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\tfrac{x^{k-1}-x^k}{\\gamk}-(\\nabla f(x^{k-1})-\\nabla f(x^k))\n\t\t\t\t\t\\in\n\t\t\t\t\t\t\\partial\\varphi(x^k),\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\thence, for any \\(\\mytheta_{k+1}\\geq0\\),\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t0\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\mytheta_{k+1}\\left(\n\t\t\t\t\t\t\t\\varphi(x^{k-1})-\\varphi(x^k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\innprod{v^k}{x^{k-1}-x^k}\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\mytheta_{k+1}\\left(\n\t\t\t\t\t\t\t\\varphi(x^{k-1})-\\varphi(x^k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{\\gamk}\\|x^k-x^{k-1}\\|^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla f(x^{k-1})-\\nabla f(x^k)}{x^{k-1}-x^k}\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\mytheta_{k+1}\\left(\n\t\t\t\t\t\t\t\\varphi(x^{k-1})-\\varphi(x^k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1-\\gamk\\Lk}{\\gamk}\\|x^k-x^{k-1}\\|^2\n\t\t\t\t\t\t\\right).\n\t\t\t\t\t\t\\numberthis\\label{eq:PG:subgradineq}\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tBy summing the last two inequalities, multiplying by \\(\\gamk*\\), and observing that\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\varphi_\\star-\\varphi(x^k)+\\mytheta_{k+1}(\\varphi(x^{k-1})-\\varphi(x^k))\n\t\t\t\t\t=\n\t\t\t\t\t\t\\mytheta_{k+1}P_{k-1}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t(1+\\mytheta_{k+1})P_k,\n\t\t\t\t\t\\]\n\t\t\t\t\twe obtain\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t&\n\t\t\t\t\t\t\\tfrac{1}{2}\\|x^{k+1}-x^\\star\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\gamk*(1+\\mytheta_{k+1})P_k\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1-\\varepsilon_{k+1}\\rhok*}{2}\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\tfrac{1}{2}\\|x^k-x^\\star\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\mytheta_{k+1}\\gamk*P_{k-1}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\rhok*\\left(\n\t\t\t\t\t\t\t\\tfrac{1-\\gamk\\Lk(2-\\gamk\\ck)}{2\\varepsilon_{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\mytheta_{k+1}(1-\\gamk\\Lk)\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\\|x^{k-1}-x^k\\|^2.\n\t\t\t\t\t\\numberthis\\label{eq:PG:descentdelta}\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tSelecting \\(\\mytheta_{k+1}=\\rhok*\\) and \\(\\varepsilon_{k+1}=\\nicefrac{1}{2\\rhok*}\\) results in the claimed inequality.\n\t\t\t\t\t\\qedhere\n\t\t\t\t\\end{proof}\n\t\t\t\\end{lemma}","sha256_utf8":"15292d5bf1560a6a0a1b9e3eeda5d121c896eb21e9b288146a67e4d0ae18fb9d"},{"start":41451,"end":54121,"start_line":847,"end_line":1155,"raw_tex":"\\begin{proof}~\n\t\t\t\t\t\\begin{proofitemize}\n\t\t\t\t\t\\item \\itemref{thm:PG:SD}~\n\t\t\t\t\t\tExpressed in terms of \\(\\U_k(x^\\star)\\), the inequality in \\cref{thm:PG:descent} simplifies to the one in the statement. The update rule for \\(\\gamk\\) implies that the coefficients indicated in the statement are greater than or equal to zero, and the claim follows.\n\n\t\t\t\t\t\\item \\itemref{thm:PG:gamk}~\n\t\t\t\t\t\tThe proven inequality implies that \\(\\seq{\\U_k(x_{}^{\\star })}\\) converges for any \\(x^\\star\\in\\argmin\\varphi\\).\n\t\t\t\t\t\tTaking into account the nonnegativity of the last two terms in the definition of \\(\\U_k(x^\\star)\\), it follows that \\(\\seq{x^k}\\) is bounded.\n\t\t\t\t\t\tAs such, \\(\\V\\) and \\(L_{f,\\V}\\) as in the statement exist.\n\t\t\t\t\t\t\tObserve that\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:relate:MM}\n\t\t\t\t\t\t\t\t\\tfrac{\\gamk}{2\\sqrt{[\\gamk\\Lk(\\gamk\\ck-1)]_+}}\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\tfrac12\n\t\t\t\t\t\t\t\t\\sqrt{\\tfrac{1}{\\Lk\\ck}\\tfrac{\\gamk\\ck}{[\\gamk\\ck-1]_+}}\n\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t\\tfrac{1}{2\\sqrt{\\Lk\\ck}}\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\tfrac{1}{2L_k},\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\twhere the last identity owes to \\cref{thm:PG:CL}, and therefore\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:MM}\n\t\t\t\t\t\t\t\t\\gamk*\n\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t\\min\\set{\\gamk\\sqrt{1+\\rhok},\\tfrac{1}{2L_k}}\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tholds for every \\(k\\).\n\t\t\t\t\t\t\t\t\t\t\t\tSuppose that \\(\\gamk\\leq\\nicefrac{1}{2L_{f,\\V}}\\) for \\(k=0,\\dots,K\\).\n\t\t\t\t\t\t\tBecause of \\eqref{eq:MM} and \\cref{thm:Lk<=Ck}, this implies that\n\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\\gamk*\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\gamk\\sqrt{1+\\rhok}\n\t\t\t\t\t\t\t\\)\n\t\t\t\t\t\t\tfor \\(k=0,\\dots,K-1\\).\n\t\t\t\t\t\t\t\t\t\t\t\tThen, for these \\(k\\) one has that\n\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\\rhok*^2\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t1+\\rhok\n\t\t\t\t\t\t\t>\n\t\t\t\t\t\t\t\t1\n\t\t\t\t\t\t\t\\),\n\t\t\t\t\t\t\twhich inductively gives\n\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\\rhok^2\n\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t2\n\t\t\t\t\t\t\t\\)\n\t\t\t\t\t\t\tfor all \\(k=1,\\dots,K\\) (since \\(\\rho_0 \\geq 1\\)).\n\t\t\t\t\t\t\tWe then have\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:PG:gam2k}\n\t\t\t\t\t\t\t\t\\gamma_K^2\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\rho_K^2\\gamma_{K-1}^2\n\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t2\\gamma_{K-1}^2\n\t\t\t\t\t\t\t\\geq\\dots\\geq\n\t\t\t\t\t\t\t\t2^K\\gamma_0^2, \t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tshowing that in at most \\(k_0\\) iterations stepsize exceeds \\(\\nicefrac{1}{2L_{f,\\V}}\\).\n\t\t\t\t\t\t\tFrom this point, a trivial induction argument using \\eqref{eq:MM} reveals that \\(\\gamk\\geq\\nicefrac{1}{2L_{f,\\V}}\\) for all \\(k\\geq k_0\\).\n\t\t\t\t\t\\item \\itemref{thm:PG:x*}~\n\t\t\t\t\t\tWe begin by observing that, as is apparent from its proof, assertion \\itemref{thm:PG:SD} remains valid if the identity in \\eqref{eq:PG:gamk} is replaced by an inequality ``\\(\\leq\\)''.\n\t\t\t\t\t\tEither way, a telescoping argument yields that\n\t\t\t\t\t\t\\begin{equation}\\label{eq:gPto0}\n\t\t\t\t\t\t\t\\gamk(1+\\rhok-\\rhok*^2)P_{k-1}\\to0\n\t\t\t\t\t\t\\quad\\text{as}\\quad\n\t\t\t\t\t\t\tk\\to\\infty.\n\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\tWe proceed by intermediate claims.\n\t\t\t\t\t\t\\begin{claims}[widest*=3]%\n\t\t\t\t\t\t\\item \\label{claim:infP}\n\t\t\t\t\t\t\t{\\it \\(\\liminf_{k\\to\\infty}P_k=0\\), and in particular \\(\\seq{x^k}\\) admits a limit point \\(x_\\infty \\in \\argmin \\varphi \\).}\n\n\t\t\t\t\t\t\tSince \\(\\seq{x^k}\\) is bounded and \\(\\varphi\\) is lsc, it suffices to show that \\(\\liminf_{k\\to\\infty}P_k=0\\).\n\t\t\t\t\t\t\tIf \\(\\limsup_{k\\to\\infty}(1+\\rhok-\\rhok*^2)>0\\), then \\eqref{eq:gPto0} yields the claim, since \\(\\gamk\\) is bounded away from zero.\n\t\t\t\t\t\t\tAlternatively, since \\(1+\\rhok-\\rhok*^2\\geq0\\), necessarily \\(1+\\rhok-\\rhok*^2\\to0\\), from which it easily follows that \\(\\liminf_{k\\to\\infty}\\rhok>1\\), and that therefore \\(\\gamk\\to\\infty\\);\n\t\t\t\t\t\t\tin this case, because of the inequalities \\(\\gamk\\left(1+\\rhok\\right)P_{k-1}\\leq\\U_k\\leq\\U_1\\), this directly proves that \\(P_k\\to0\\).\n\n\t\t\t\t\t\t\\item\\label{claim:thm:PG:optunique}%\n\t\t\t\t\t\t\t{\\it \\(x_\\infty\\) is the only limit point that belongs to \\(\\argmin\\varphi\\).}\n\n\t\t\t\t\t\t\tSuppose that \\(x_\\infty'\\in\\argmin\\varphi\\) is a limit point.\n\t\t\t\t\t\t\tObserve that\n\t\t\t\t\t\t\t\\[\n\t\t\t\t\t\t\t\t\\innprod{x^k}{x_\\infty - x_\\infty'}\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\U_k(x_\\infty') - \\U_k(x_\\infty) + \\tfrac12\\|x_\\infty\\|^2 - \\tfrac12\\|x_\\infty'\\|^2,\n\t\t\t\t\t\t\t\\]\n\t\t\t\t\t\t\tand since \\(\\seq{\\U_k(x^\\star)}\\) is convergent for all \\(x^\\star\\in \\argmin \\varphi\\), then so is \\(\\seq{\\innprod{x^k}{x_\\infty - x_\\infty'}}\\).\n\t\t\t\t\t\t\tPassing to the limit along the two converging subsequences thus yields\n\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\\innprod{x_\\infty}{x_\\infty - x_\\infty'}\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\innprod{x_\\infty'}{x_\\infty - x_\\infty'}\n\t\t\t\t\t\t\t\\),\n\t\t\t\t\t\t\twhich after rearranging results in \\(\\|x_\\infty - x_\\infty'\\|^2=0\\), establishing the claim.\n\n\t\t\t\t\t\t\\item\\label{claim:thm:PG:allopt}%\n\t\t\t\t\t\t\t{\\it \\(\\seq{x^k}\\) converges to a point in \\(\\argmin\\varphi\\).}\n\n\t\t\t\t\t\t\tLet \\(x_\\infty\\) be the (unique) optimal limit point of \\(\\seq{x^k}\\) as in \\cref{claim:thm:PG:optunique}, and let\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:U}\n\t\t\t\t\t\t\t\tU\\coloneqq\\lim_{k\\to\\infty}\\U_k(x_\\infty),\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\twhich exists since the sequence is monotonically decreasing by assertion \\itemref{thm:PG:SD}.\n\t\t\t\t\t\t\tBecause of the inequality \\(\\frac{1}{2}\\|x^k-x_{\\infty}\\|^2\\leq\\U_k(x_{\\infty})\\), it suffices to show that \\(U=0\\).\n\t\t\t\t\t\t\tObserve that, for every \\(k\\),\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:rhomax}\n\t\t\t\t\t\t\t\t\\rhok\\leq\\rho_{\\rm max}\n\t\t\t\t\t\t\t\\quad\\text{where}\\quad\n\t\t\t\t\t\t\t \t\\rho_{\\rm max}\n\t\t\t\t\t\t\t \\coloneqq\n\t\t\t\t\t\t\t\t\\max\\set{\\tfrac{\\gamma_0}{\\gamma_{-1}}, \\tfrac12(1+\\sqrt{5})}\n\t\t\t\t\t\t\t>\n\t\t\t\t\t\t\t\t1,\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tas it can be verified by induction.\n\t\t\t\t\t\t\tWe consider two mutually exclusive scenarios.\n\t\t\t\t\t\t\t\\begin{proofitemize}[leftmargin=*]\n\t\t\t\t\t\t\t\\item\n\t\t\t\t\t\t\t\tSuppose that \\(\\seq{\\gamk}\\) is bounded, and consider a subsequence \\(\\seq{x^k}[k\\in K]\\) such that \\(\\lim_{K\\ni k\\to\\infty}P_k=0\\), which exists and converges to \\(x_{\\infty}\\) by virtue of the previous claims.\n\t\t\t\t\t\t\t\tExtract \\(K'\\subseteq K\\) such that \\(\\seq{\\gamk*}[k\\in K']\\to\\gamma\\) for some \\(\\gamma>0\\).\n\t\t\t\t\t\t\t\tThen, \\(x^{k+1}=\\prox_{\\gamk*g}(x^k-\\gamk*\\nabla f(x^k))\\to\\prox_{\\gamma g}(x_\\infty-\\gamma\\nabla f(x_\\infty))=x_\\infty\\) as \\(K'\\ni k\\to\\infty\\) as well.\n\t\t\t\t\t\t\t\tThis yields the sought identity\n\t\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\tU\n\t\t\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\t\t\\lim_{k\\to\\infty}\\U_k(x_{\\infty})\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\lim_{K'\\ni k\\to\\infty}\\U_{k+1}(x_{\\infty})\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\t\t\\lim_{K'\\ni k\\to\\infty}\\left(\n\t\t\t\t\t\t\t\t\t\t\\tfrac{1}2\\|x^{k+1}-x_{\\infty}\\|^2\n\t\t\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\t\t\\tfrac{1}2\\|x^{k+1}-x^k\\|^2\n\t\t\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\t\t\\gamk*\\left(1+\\rhok*\\right)P_k\n\t\t\t\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\t\t0,\n\t\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\t\twhere the vanishing of the last term owes to boundedness of \\(\\seq{\\gamk}\\) and \\(\\seq{\\rhok}\\).\n\n\t\t\t\t\t\t\t\\item\n\t\t\t\t\t\t\t\tSuppose instead that \\(\\seq{\\gamk}\\) is unbounded, and let \\(\\seq{\\gamk}[k\\in K']\\) be an arbitrary subsequence such that \\(\\gamk\\to\\infty\\) as \\(K'\\ni k\\to\\infty\\).\n\t\t\t\t\t\t\t\tFrom the inequality\n\t\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\t\\gamk P_{k-1} \\leq \\U_k(x_{\\infty}) \\leq \\U_1(x_\\infty) < \\infty\n\t\t\t\t\t\t\t\t\\),\n\t\t\t\t\t\t\t\twe conclude that\n\t\t\t\t\t\t\t\t\\begin{equation}\\label{eq:Pk:xk-}\n\t\t\t\t\t\t\t\t\tP_{k-1}\\to0\n\t\t\t\t\t\t\t\t~\\text{ and }~\n\t\t\t\t\t\t\t\t\tx^{k-1}\\to x_\\infty\n\t\t\t\t\t\t\t\t\\text{ as }\n\t\t\t\t\t\t\t\t\tK'\\ni k\\to\\infty\n\t\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\t\t(recall that \\(\\varphi\\) is lsc, \\(\\seq{x^k}\\) is bounded, and \\(x_\\infty\\) is the \\emph{unique} optimal limit point).\n\t\t\t\t\t\t\t\tThen, since by \\eqref{eq:rhomax} \\(\\gamma_{k-1}\\geq\\nicefrac{\\gamk}{\\rho_{\\rm max}}\\to\\infty\\) as \\(K'\\ni k\\to\\infty\\), by the same argument also \\(x^{k-2}\\to x_\\infty\\) as \\(K'\\ni k\\to\\infty\\).\n\t\t\t\t\t\t\t\tWith \\(U\\) as in \\eqref{eq:U}, we thus have\n\t\t\t\t\t\t\t\t\\begin{equation}\\label{eq:U:gam:unbounded}\n\t\t\t\t\t\t\t\t\t\tU\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\lim_{k\\to\\infty}\\U_k(x_\\infty)\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\lim_{K'\\ni k\\to\\infty}\\U_{k-1}(x_\\infty)\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\lim_{K'\\ni k\\to\\infty}\\gamma_{k-1}(1+\\rho_{k-1})P_{k-2}\n\t\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\t\tholding for \\emph{any} set of indices \\(K'\\subseteq\\N\\) along which the stepsizes are divergent.\n\n\t\t\t\t\t\t\t\tWe now construct a specific subsequence \\(K\\coloneqq\\set{k_0,k_1,\\dots}\\) as follows: start with \\(k_0=1\\), and for \\(i\\geq0\\) let\n\t\t\t\t\t\t\t\t\\begin{equation}\\label{eq:ki+}\n\t\t\t\t\t\t\t\t\tk_{i+1}=\\min\\set{k\\geq k_i}[\\gamk \\geq \\rho_{\\rm max}\\gamma_{k_i}].\n\t\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\t\tThen, \\(\\seq{\\gamma_{k_i}}[i\\in\\N]\\to\\infty\\), which as argued after \\eqref{eq:Pk:xk-} implies that both \\(\\seq{x^{k_i-1}}[i\\in\\N]\\) and \\(\\seq{x^{k_i-2}}[i\\in\\N]\\) converge to \\(x_\\infty\\).\n\t\t\t\t\t\t\t\tObserve that \\(\\rho_{k_i}>1\\) holds for all \\(i\\in\\N\\) by minimality of \\(k_i\\).\n\t\t\t\t\t\t\t\tNotice further that\n\t\t\t\t\t\t\t\t\\begin{equation}\\label{eq:k-1}\n\t\t\t\t\t\t\t\t\t\\rho_{k_i-1}\\geq\\rho_{\\rm max}^{-1}\n\t\t\t\t\t\t\t\t\\quad\n\t\t\t\t\t\t\t\t\t\\forall i\\in\\N.\n\t\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\t\tTo see why, suppose on the contrary that\n\t\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\t\\rho_{k_i-1}<\\rho_{\\rm max}^{-1}\n\t\t\t\t\t\t\t\t\\);\n\t\t\t\t\t\t\t\tthen in particular \\(\\rho_{k_i-1}<1\\), thereby ensuring by the previous observation that \\(k_i-1\\notin K\\), and thus \\(k_{i-1}\\leq k_i-2\\).\n\t\t\t\t\t\t\t\tThis leads to the contradiction\n\t\t\t\t\t\t\t\t\\[\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_{i-1}}\n\t\t\t\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:ki+}}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i}\n\t\t\t\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:rhomax}}\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-1}\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\rho_{k_i-1}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-2}\n\t\t\t\t\t\t\t\t<\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-2}\n\t\t\t\t\t\t\t\t<\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_{i-1}},\n\t\t\t\t\t\t\t\t\\]\n\t\t\t\t\t\t\t\twhere the last inequality follows either in case \\(k_i-2=k_{i-1}\\) (since \\(\\rho_{\\rm max}>1\\)) or when \\(k_{i-1}<k_i-2<k_i\\) (from minimality in the definition of \\(k_i\\)).\n\t\t\t\t\t\t\t\tThis shows \\eqref{eq:k-1}.\n\n\t\t\t\t\t\t\t\tNext, by the stepsize update at \\cref{state:PG:gamk},\n\t\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\t\\tfrac{1}{\\rho_{\\rm max}}\\gamma_{k_i-2}L_{k_i-2}\n\t\t\t\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:k-1}}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-1}L_{k_i-2}\n\t\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\t\\frac{\\gamma_{k_i-2}L_{k_i-2}}{2\\sqrt{\\left[\\gamma_{k_i-2}^2\\ell_{k_i-2}c_{k_i-2}-\\gamma_{k_i-2}\\ell_{k_i-2}\\right]_+}}\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\t\\frac{\\gamma_{k_i-2}L_{k_i-2}}{2\\sqrt{\\left[\\gamma_{k_i-2}^2L_{k_i-2}^2-\\gamma_{k_i-2}L_{k_i-2}\\right]_+}}\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\t\t\\frac{1}{2\\sqrt{\\left[1-\\frac{1}{\\gamma_{k_i-2}L_{k_i-2}}\\right]_+}},\n\t\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\t\twhere the third inequality uses the fact that \\(L_{k_i-2}^2 = \\ell_{k_i-2}c_{k_i-2}\\), and \\(\\ell_{k_i-2}\\leq L_{k_i-2}\\) by \\cref{thm:PG:cLl}.\n\t\t\t\t\t\t\t\tThis implies that \\(\\seq{\\gamma_{k_i-2}L_{k_i-2}}[i\\in\\N]\\) must be bounded.\n\t\t\t\t\t\t\t\tThe subgradient characterization in \\eqref{eq:PG:vk} and the Cauchy-Schwarz inequality then yield\n\t\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-1} P_{k_i-2}\n\t\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\t\\rho_{k_i-1}\n\t\t\t\t\t\t\t\t\t\\innprod{x^{k_i-2}-x_{\\infty}}{\n\t\t\t\t\t\t\t\t\t\tx^{k_i-2}-x^{k_i-3}\n\t\t\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\t\t\\gamma_{k_i-2}(\\nabla f(x^{k_i-2})-\\nabla f(x^{k_i-3}))\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\t(1+\\gamma_{k_i-2}L_{k_i-2})\n\t\t\t\t\t\t\t\t\t}_{\\text{bounded}}\n\t\t\t\t\t\t\t\t\t\\,\n\t\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\t\\|x^{k_i-2}-x_{\\infty}\\|\n\t\t\t\t\t\t\t\t\t\t\\|x^{k_i-2}-x^{k_i-3}\\|\n\t\t\t\t\t\t\t\t\t}_{\\to0}\n\t\t\t\t\t\t\t\t\\to\n\t\t\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\t\t\\quad\n\t\t\t\t\t\t\t\t\t\\text{as }i\\to\\infty.\n\t\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\t\tCombined with the fact that \\(\\rho_{k_i-1} \\leq \\rho_{\\rm max}\\), it follows from \\eqref{eq:U:gam:unbounded} that \\(U=0\\).\n\t\t\t\t\t\t\t\\end{proofitemize}\n\t\t\t\t\t\t\t\t\t\t\t\t\\end{claims}%\n\n\t\t\t\t\t\\item \\itemref{thm:PG:sublin}~\n\t\t\t\t\t\tA telescoping argument in the descent inequality of assertion \\itemref{thm:PG:SD} yields\n\t\t\t\t\t\t\\[\n\t\t\t\t\t\t\t\\mathcal U_{K+1} + \\sum_{k=1}^K\\gamk\\left(1+\\rhok-\\rhok*^2\\right)P_{k-1} \\leq \\mathcal U_1\n\t\t\t\t\t\t\\]\n\t\t\t\t\t\tfor \\(K\\geq1\\).\n\t\t\t\t\t\tBy further lower bounding \\(\\mathcal U_{K+1}\\) we obtain\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\textstyle\n\t\t\t\t\t\t\t\\gamma_{K+1}(1+\\rho_{K+1})P_K\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\sum_{k=1}^K\\gamk\\left(1+\\rhok-\\rhok*^2\\right)P_{k-1}\n\t\t\t\t\t\t\t\\leq \\mathcal U_1.\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\tConsidering the best-so-far quantity\n\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\\min_{k=0,1,\\ldots, K} P_k\n\t\t\t\t\t\t\\), it follows that\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\min_{k=0,1,\\dots, K} P_k\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\frac{\\mathcal U_1}{\n\t\t\t\t\t\t\t\t\\gamma_{K+1}(1+\\rho_{K+1})\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\sum_{k=1}^K\\gamk(1+\\rhok-\\rhok*^2)\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\frac{\\mathcal U_1}{\n\t\t\t\t\t\t\t\t\\gamma_{K+1}(1+\\rho_{K+1})\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\sum_{k=1}^K\\gamk\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\sum_{k=1}^K(\\gamk\\rhok-\\gamk*\\rhok*)\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\numberthis\\label{eq:O(1k)}\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\frac{\\mathcal U_1}{\n\t\t\t\t\t\t\t\t\\gamma_1\\rho_1+\\sum_{k=1}^{K+1}\\gamk\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\leq\n\t\t\t\t\t\t\t\\frac{\\mathcal U_1}{\n\t\t\t\t\t\t\t\t(K+1)\\gamma_{\\rm min}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\twhere \\(\\gamma_{\\rm min}\\coloneqq\\min\\set{\\gamma_0,\\frac{1}{2L_{f,\\mathcal V}}}\\) is a lower bound on the stepsize, as demonstrated in the proof of assertion \\itemref{thm:PG:gamk}.\n\t\t\t\t\t\t\\qedhere\n\t\t\t\t\t\\end{proofitemize}\n\t\t\t\t\\end{proof}","sha256_utf8":"6c07b5b733ff1d4819214f3ef59e240e700b5ba39e0c986ea1c9330226bdef49"},{"start":111163,"end":114439,"start_line":2434,"end_line":2549,"raw_tex":"\\begin{lemma}\\label{lem:enlarged:ck}\n\t\t\t\tSuppose that \\cref{ass:f} holds.\n\t\t\t\tThen, for every \\(x,y\\in\\R^n\\) it holds that\n\t\t\t\t\\[\n\t\t\t\t\tf(y)\n\t\t\t\t\\leq\n\t\t\t\t\tf(x)\n\t\t\t\t\t+\n\t\t\t\t\t\\innprod{\\nabla f(x)}{y-x}\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{1}{2L_{f,\\overline\\V}}\n\t\t\t\t\t\\|\\nabla f(x) - \\nabla f(y)\\|^{2},\n\t\t\t\t\\]\n\t\t\t\twhere \\(L_{f,\\overline{\\V}}\\) is a Lipschitz modulus for \\(\\nabla f\\) on \\(\\overline{\\V}\\coloneqq\\V+\\cball{0}{\\diam(\\V)}\\), and \\(\\V\\subseteq\\R^n\\) is a bounded and convex set that contains \\(x\\) and \\(y\\).\n\t\t\t\tIn particular,\n\t\t\t\t\\[\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\|\\nabla f(x) - \\nabla f(y)\\|^2\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\innprod{\\nabla f(x) - \\nabla f(y)}{x-y}\n\t\t\t\t\t}\n\t\t\t\t\\leq\n\t\t\t\t\tL_{f,\\overline\\V}.\n\t\t\t\t\\]\n\t\t\t\t\\begin{proof}\n\t\t\t\t\tWe henceforth fix \\(x,y\\in \\V\\subseteq\\overline\\V\\).\n\t\t\t\t\tSince $\\nabla f$ is Lipschitz continuous on $\\overline\\V$ with modulus \\(L_{f,\\overline\\V}\\), it follows from the descent inequality \\cite[Prop. A.24]{bertsekas2016nonlinear} that\n\t\t\t\t\t\\begin{equation}\\label{eq:holder_ineq}\n\t\t\t\t\t\tf(z)\n\t\t\t\t\t\\leq\n\t\t\t\t\t\tf(y)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla f(y)}{z-y}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{L_{f,\\overline\\V}}{2}\\|z-y\\|^{2}\n\t\t\t\t\t\\quad\n\t\t\t\t\t\t\\forall z \\in \\overline\\V.\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\tLet \\(l_x(y) \\coloneqq f(y) - f(x) - \\innprod{\\nabla f(x)}{y-x}\\), and note that \\(l_x\\) is a convex function with \\(\\nabla l_x(y)=\\nabla f(y)-\\nabla f(x)\\).\n\t\t\t\t\tFor any \\(z \\in \\overline\\V\\), we have\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\tl_x(z)\n\t\t\t\t\t={} &\n\t\t\t\t\t\tf(z) - f(x) - \\innprod{\\nabla f(x)}{z-x}\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:holder_ineq}}{} &\n\t\t\t\t\t\tf(y)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla f(y)}{z-y}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{L_{f,\\overline\\V}}{2}\\|z-y\\|^{2}\n\t\t\t\t\t\t-\n\t\t\t\t\t\tf(x)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\innprod{\\nabla f(x)}{z-x}\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\tf(y) - f(x) - \\innprod{\\nabla f(x)}{y-x}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla f(y) - \\nabla f(x)}{z-y}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{L_{f,\\overline\\V}}{2}\\|z-y\\|^{2}\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\tl_x(y)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla l_x(y)}{z-y}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{L_{f,\\overline\\V}}{2}\\|z-y\\|^{2}.\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tNoticing that $\\nabla l_x(x) = 0$, it follows from convexity of \\(l_x\\) that $x$ is its global minimizer. Hence that \\(\\min l_x=l_x(x)=0\\).\n\t\t\t\t\tLet us denote \\(v\\coloneqq\\frac{1}{\\|\\nabla l_x(y)\\|}\\nabla l_x(y)\\) and set\n\t\t\t\t\t\\(\n\t\t\t\t\t\tz = y - \\frac{\\|\\nabla l_x(y)\\|}{L_{f,\\overline\\V}}v\n\t\t\t\t\t\\).\n\t\t\t\t\tNote that\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\|z-y\\|\n\t\t\t\t\t=\n\t\t\t\t\t\t\\tfrac{\\|\\nabla l_x(y)\\|}{L_{f,\\overline\\V}}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\tfrac{\\|\\nabla f(y)-\\nabla f(x)\\|}{L_{f,\\overline\\V}}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\|y-x\\|,\n\t\t\t\t\t\\]\n\t\t\t\t\tand in particular \\(z\\in \\V+\\cball{y}{\\|y-x\\|}\\subseteq\\overline\\V\\).\n\t\t\t\t\tFrom the previous inequality we get\n\t\t\t\t\t\\begin{qedalign*}\n\t\t\t\t\t\t0\n\t\t\t\t\t=\n\t\t\t\t\t\t\\min l_x\n\t\t\t\t\t=\n\t\t\t\t\t\tl_x(x)\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\tl_x\\Bigl(\n\t\t\t\t\t\t\ty - \\tfrac{\\|\\nabla l_x(y)\\|}{L_{f,\\overline\\V}}v\n\t\t\t\t\t\t\\Bigr)\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:holder_ineq}}{} &\n\t\t\t\t\t\tl_x(y)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{\\|\\nabla l_x(y)\\|}{L_{f,\\overline\\V}}\n\t\t\t\t\t\t\\innprod{\\nabla l_x(y)}{v}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{L_{f,\\overline\\V}}{2}\n\t\t\t\t\t\t\\tfrac{1}{L_{f,\\overline\\V}^{2}}\n\t\t\t\t\t\t\\|\\nabla l_x(y)\\|^{2}\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\tf(y) - f(x) - \\innprod{\\nabla f(x)}{y-x}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{1}{2L_{f,\\overline\\V}}\n\t\t\t\t\t\t\\|\\nabla f(x) - \\nabla f(y)\\|^{2}.\n\t\t\t\t\t\\end{qedalign*}\n\t\t\t\t\\end{proof}\n\t\t\t\\end{lemma}","sha256_utf8":"7468a94812a8fa392eb38cef1457f564bbd0ccdfb4d4d1b2f435131d721d9ed0"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"自适应近端梯度收敛与速率","source_units":[{"start":39578,"end":41451,"start_line":797,"end_line":847,"raw_tex":"\\begin{theorem}\\label{thm:PG:convergence}%\n\t\t\t\tSuppose that \\cref{ass:PG} holds, and consider the iterates generated by \\refadaPG*.\n\t\t\t\tThen, for any \\(x^\\star\\in\\argmin\\varphi\\), with \\(\\rhok\\) and \\(P_k\\) as in \\eqref{eq:rhoP} and with \\(\\Uk(x^\\star)\\) defined as\n\t\t\t\t\\[\n\t\t\t\t\t\\Uk (x^\\star) \t\t\t\t\\coloneqq\n\t\t\t\t\t\\tfrac12\\|x^k-x^\\star\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac14\\|x^k-x^{k-1}\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\gamk\\left(1+\\rhok\\right)P_{k-1},\n\t\t\t\t\\]\n\t\t\t\tthe following hold:\n\t\t\t\t\\begin{enumerate}[widest*=4]\n\t\t\t\t\\item \\label{thm:PG:SD}%\n\t\t\t\t\tFor all \\(k\\geq1\\),\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\Uk*(x^\\star)\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\Uk (x^\\star)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{1}{4}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\rhok*^2\\gamk\\Lk(\\gamk\\ck-1)\n\t\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\t\\|x^{k-1}-x^k\\|^2\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\gamk\\bigl(\n\t\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\\vphantom{\\tfrac{1}{4}}\n\t\t\t\t\t\t\t\t{1+\\rhok-\\rhok*^2}\n\t\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\\bigr)P_{k-1}\n\t\t\t\t\t\\).\n\t\t\t\t\\item \\label{thm:PG:gamk}%\n\t\t\t\t\tThe sequence \\(\\seq{x^k}\\) is bounded and \\(\\gamk>\\frac{1}{2L_{f,\\V}}\\) holds for every \\(k\\geq k_0\\coloneqq\\left\\lfloor2\\log_2\\frac{1}{2\\gamma_0L_{f,\\V}}\\right\\rfloor_+\\), where \\(L_{f,\\V}\\) is a Lipschitz modulus for \\(\\nabla f\\) on a compact convex set \\(\\V\\) containing \\(\\seq{x^k}\\).\n\n\t\t\t\t\\item \\label{thm:PG:x*}%\n\t\t\t\t\tThe sequence \\(\\seq{x^k}\\) converges to a solution.\n\t\t\t\t\tThe claim remains true if the stepsizes are chosen in such a way that \\eqref{eq:PG:gamk} holds with ``\\(\\leq\\)'', as long as \\(\\seq{\\gamk}\\) is bounded away from zero.\n\t\t\t\t\t\\item \\label{thm:PG:sublin}%\n\t\t\t\t\tThe following rate holds\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\min_{k=0,1,\\ldots, K} (\\varphi(x^k) - \\min \\varphi)\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\frac{\\U_1(x^\\star)}{\\sum_{k=1}^{K+1}\\gamk}, \t\t\t\t\t\\]\n\t\t\t\t\twhich by assertion \\itemref{thm:PG:gamk} implies a sublinear \\(\\mathcal O(\\nicefrac1K)\\) convergence rate on the best-so-far cost.\n\t\t\t\t\t\\end{enumerate}\n\t\t\t\t","sha256_utf8":"248e29c2c371925f9febb932e3c43160fc898567a8fce71934328c98e6ccac99"},{"start":54121,"end":54138,"start_line":1155,"end_line":1156,"raw_tex":"\n\t\t\t\\end{theorem}","sha256_utf8":"6727a931b7123809dbadf28e49e5715f1d3264affc29f43f789dd7f49c419214"}]},{"id":"S002","semantic_type":"problem_setting","kind":"problem_setting","title":"复合凸最小化问题","source_units":[{"start":24667,"end":24771,"start_line":439,"end_line":441,"raw_tex":"\\begin{equation}\\label{eq:PG}\n\t\t\t\\minimize_{x\\in\\R^n}\\; \\varphi(x) \\coloneqq f(x)+g(x).\n\t\t\\end{equation}","sha256_utf8":"623169c5934295751ebdc92d57663af1e5ca66453ce6ee57feab3d07fb79b52c"}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","title":"光滑性与解存在假设","source_units":[{"start":24876,"end":25307,"start_line":444,"end_line":454,"raw_tex":"\\begin{assumption}[Requirements for the proximal gradient setting]\\label{ass:PG}\n\t\t\tThe following hold in problem \\eqref{eq:PG}:\n\t\t\t\\begin{enumeratass}\n\t\t\t\\item \\label{ass:f}%\n\t\t\t\t\\(\\func{f}{\\R^n}{\\R}\\) is convex and has locally Lipschitz continuous gradient.\n\t\t\t\\item\n\t\t\t\t\\(\\func{g}{\\R^n}{\\Rinf}\\) is proper, lsc, and convex.\n\t\t\t\\item\n\t\t\t\tA solution exists: \\(\\argmin\\varphi\\neq\\emptyset\\).\n\t\t\t\\end{enumeratass}\n\t\t\\end{assumption}","sha256_utf8":"910852942f97654f72df3f04e8ea50f2877d7c1958547dae05026331f32aad29"}]},{"id":"S004","semantic_type":"algorithm","kind":"algorithm","title":"自适应近端梯度算法","source_units":[{"start":26220,"end":27021,"start_line":470,"end_line":500,"raw_tex":"\\begin{algorithm}[t]\n\t\t\t\\caption{Adaptive proximal gradient method (\\protect\\adaPG)}\n\t\t\t\\label{alg:PG}\n\t\t\t\\begin{algorithmic}[1]\n\t\t\t\\itemsep=3pt\n\t\t\t\\Require\n\t\t\t\tstarting point \\(x^{-1}\\in\\R^n\\)\n\t\t\t\t~and~\n\t\t\t\tstepsizes \\(\\gamma_0\\geq\\gamma_{-1}>0\\)\n\t\t\t\\Initialize\n\t\t\t\t\\(x^0=\\FBk[0]{x^{-1}}\\)\n\t\t\t\\item[\\algfont{Repeat for} \\(k=0,1,\\ldots\\) until convergence]\n\t\t\t\\State \\label{state:PG:gamk}%\n\t\t\t\tWith \\(\\Lk\\) and \\(\\ck\\) as in \\eqref{eq:CL}, define the stepsize as\n\t\t\t\\hfill{\\gray\\footnotesize\\sf // \\([r]_+\\coloneqq\\max\\set{r,0}\\)}\n\t\t\t\t\\[\n\t\t\t\t\t\\gamk*\n\t\t\t\t=\n\t\t\t\t\t\\gamk\\min\\set{\n\t\t\t\t\t\t\\sqrt{1+\\tfrac{\\gamk}{\\gamma_{k-1}}},\\,\n\t\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\t1\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t2\\sqrt{\\left[\\gamk\\Lk(\\gamk\\ck-1)\\right]_+}\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\\]\n\t\t\t\\State\n\t\t\t\t\\(x^{k+1}=\\FBk*{x^k}\\)\n\t\t\t\\end{algorithmic}\n\t\t\\end{algorithm}","sha256_utf8":"f153ee0ce3a37543ed05f617694791992d8337fd8ee8f1cf2fa7b5f3dd50bd32"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"局部梯度变化率","source_units":[{"start":9103,"end":9256,"start_line":245,"end_line":253,"raw_tex":"\\begin{equation}\\label{eq:MML_k}\n\t\t\tL_k\n\t\t\\coloneqq\n\t\t\t\\frac{\n\t\t\t\t\\|\\nabla f(x^k) - \\nabla f(x^{k-1})\\|\n\t\t\t}{\n\t\t\t\t\\|x^k - x^{k-1}\\|\n\t\t\t}\n\t\t\\end{equation}","sha256_utf8":"9016350beed93d8e038f77c3c8a71bd859edf305f3b4df025bddd573d111c539"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"局部几何估计与除零约定","source_units":[{"start":11631,"end":11967,"start_line":267,"end_line":283,"raw_tex":"\\begin{align}\n\t\t\t\t\\Lk\n\t\t\t\\coloneqq{} &\n\t\t\t\t\\frac{\n\t\t\t\t\t\\innprod{\\nabla f(x^{k-1})-\\nabla f(x^k)}{x^{k-1}-x^k}\n\t\t\t\t}{\n\t\t\t\t\t\\|x^{k-1}-x^k\\|^2\n\t\t\t\t}\n\t\t\t\\shortintertext{and}\n\t\t\t\t\\ck\n\t\t\t\\coloneqq{} &\n\t\t\t\t\\frac{\n\t\t\t\t\t\\|\\nabla f(x^{k-1})-\\nabla f(x^k)\\|^2\n\t\t\t\t}{\n\t\t\t\t\t\\innprod{\\nabla f(x^{k-1})-\\nabla f(x^k)}{x^{k-1}-x^k}\n\t\t\t\t}\n\t\t\t\\end{align}","sha256_utf8":"999b4daac8e2b5c6aae676739e81806de62df0ec1c727911789eeb64ee74d18c"},{"start":11989,"end":12461,"start_line":285,"end_line":287,"raw_tex":"\t\tfor a pair of points \\(x^{k-1},x^k\\in\\R^n\\).\n\t\tNoting that the denominator of \\(\\Lk\\) or \\(\\ck\\) is zero iff \\(\\nabla f(x^k)-\\nabla f(x^{k-1})=0\\) (the latter owing to the Baillon-Haddad theorem \\cite[Cor. 10]{baillon1977quelques}; see \\cref{lem:enlarged:ck} for the local version needed here), we stick to the convention \\(\\frac00=0\\) so that both \\(\\Lk\\) and \\(\\ck\\) are (well-defined, positive) real numbers.\n\t\tThroughout, we shall also adhere to \\(\\frac10=\\infty\\).\n","sha256_utf8":"960eaa2fb26df45293ddc078a9b18af533ec480f06a3a316bb76ce14eec2328a"}]},{"id":"S007","semantic_type":"algorithm","kind":"algorithm","title":"自适应步长更新","source_units":[{"start":12938,"end":13154,"start_line":290,"end_line":301,"raw_tex":"\\begin{equation}\\label{eq:PG:gamk}\n\t\t\t\\gamk*\n\t\t=\n\t\t\t\\min\\set{\n\t\t\t\t\\gamk\\sqrt{1+\\tfrac{\\gamk}{\\gamma_{k-1}}},\\,\n\t\t\t\t\\frac{\n\t\t\t\t\t\\gamk\n\t\t\t\t}{\n\t\t\t\t\t2\\sqrt{\\left[\\gamk\\Lk(\\gamk\\ck-1)\\right]_+}\n\t\t\t\t}\n\t\t\t}\n\t\t\\end{equation}","sha256_utf8":"83377508269fde753f530934060cd6aecc8788607e7fe51aebb1e999e08c4560"}]},{"id":"S008","semantic_type":"notation","kind":"notation","title":"数域与欧氏记号","source_units":[{"start":21635,"end":22133,"start_line":383,"end_line":386,"raw_tex":"\t\t\tThe set of real and extended-real numbers are \\(\\R\\coloneqq(-\\infty,\\infty)\\) and \\(\\Rinf\\coloneqq\\R\\cup\\set\\infty\\), while the positive and strictly positive reals are \\(\\R_+\\coloneqq[0,\\infty)\\) and \\(\\R_{++}\\coloneqq(0,\\infty)\\).\n\t\t\tWe use the notation \\([x]_+ = \\max\\set{0, x}\\).\n\t\t\tWith \\(\\id\\) we indicate the identity function defined on a suitable space.\n\t\t\tWe denote by \\(\\innprod{{}\\cdot{}}{{}\\cdot{}}\\) and \\(\\|{}\\cdot{}\\|\\) the standard Euclidean inner product and the induced norm.\n","sha256_utf8":"47f326f89f92ce6c6ca2168671f0ac191ecd2f23c98649ecc55619156892a8b4"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"集合与闭凸函数记号","source_units":[{"start":22465,"end":22901,"start_line":388,"end_line":389,"raw_tex":"\t\t\tThe diameter of \\(\\mathcal D\\) is \\(\\diam\\mathcal D\\coloneqq\\sup\\set{\\|x-y\\|}[x,y\\in\\mathcal D]\\), while its indicator function is \\(\\indicator_{\\mathcal D}\\), namely \\(\\indicator_{\\mathcal D}(x)=0\\) if \\(x\\in \\mathcal D\\) and \\(\\infty\\) otherwise.\n\t\t\tGiven two sets \\(\\mathcal D,\\mathcal D'\\subseteq\\R^n\\), the notation \\(\\mathcal D+\\mathcal D'\\coloneqq\\set{x+x'}[x\\in\\mathcal D,\\ x'\\in\\mathcal D']\\) indicates their Minkowski sum.\n","sha256_utf8":"c90368feff6880a0b61286189593328912b5165fa0ddc0fb2b1a61dffa93946a"},{"start":22902,"end":23501,"start_line":391,"end_line":408,"raw_tex":"\t\t\tThe notation \\(\\cball{\\bar x}{r}\\coloneqq\\set{x}[\\|x-\\bar x\\|\\leq r]\\) indicates the closed ball centered at \\(\\bar x\\) and with radius \\(r\\).\n\t\t\tThe \\DEF{domain} and \\DEF{epigraph} of an extended-real-valued function \\(\\func{h}{\\R^n}{\\Rinf}\\) are the sets\n\t\t\t\\(\n\t\t\t\t\\dom h\n\t\t\t\\coloneqq\n\t\t\t\t\\set{x\\in\\R^n}[\n\t\t\t\t\th(x)<\\infty\n\t\t\t\t]\n\t\t\t\\)\n\t\t\tand\n\t\t\t\\(\n\t\t\t\t\\epi h\n\t\t\t\\coloneqq\n\t\t\t\t\\set{(x, c )\\in\\R^n\\times\\R}[\n\t\t\t\t\th(x)\\leq c\n\t\t\t\t]\n\t\t\t\\).\n\t\t\tFunction \\(h\\) is said to be \\DEF{proper} if \\(\\dom h\\neq\\emptyset\\), and \\DEF{lower semicontinuous (lsc)} if \\(\\epi h\\) is a closed subset of \\(\\R^{n+1}\\).\n","sha256_utf8":"f9a52010f53129d358a8988ddf6513c4ea2f94eb52a0195beb90c20e17deb7fb"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"梯度局部利普希茨连续","source_units":[{"start":23825,"end":24362,"start_line":425,"end_line":431,"raw_tex":"\t\t\tWe say that a differentiable function \\(\\func{h}{\\R^n}{\\R}\\) has \\DEF{locally Lipschitz continuous gradient} if for every convex and compact set \\(\\V\\subset\\R^n\\) there exists \\(L_{h,\\V}>0\\) such that\n\t\t\t\\(\n\t\t\t\t\\|\\nabla h(x)-\\nabla h(x')\\|\\leq L_{h,\\V}\\|x-x'\\|\n\t\t\t\\)\n\t\t\tholds for all \\(x,x'\\in\\mathcal\\V\\).\\footnote{%\n\t\t\t\tBy virtue of \\cite[Thm. 9.2]{rockafellar2009variational}, this condition is equivalent to \\DEF{strict continuity} of \\(\\nabla h\\) at every point, in the sense of \\cite[Def. 9.1]{rockafellar2009variational}.\n\t\t\t}\n","sha256_utf8":"ad7909ccef8a22aa994f70400d09245d1a530d8b73927b1b1a250460f6c369f3"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"近端映射及连续性","source_units":[{"start":25309,"end":25727,"start_line":456,"end_line":465,"raw_tex":"\t\tIn addition to the gradient of the differentiable term, the fundamental oracle of {\\algnamefont PGM} is the \\emph{proximal mapping} \\cite[Def. 6.1]{beck2017first}\n\t\tdefined as\n\t\t\\[\n\t\t\t\\prox_{\\tau g}(x)\n\t\t\\coloneqq\n\t\t\t\\argmin_{w\\in\\R^n}\n\t\t\t\\set{g(w) + \\tfrac{1}{2\\tau}\\|w-x\\|^2},\n\t\t\\]\n\t\twhere \\(\\tau>0\\) is a given stepsize.\n\t\tIn the convex setting the proximal map is single valued and in fact Lipschitz continuous.\n","sha256_utf8":"28851be81c4162f48b98241d56c5a5e122ada5e2886227bd3590629d8d2388ba"}]},{"id":"S012","semantic_type":"notation","kind":"notation","title":"前向算子记号","source_units":[{"start":30276,"end":30308,"start_line":535,"end_line":537,"raw_tex":"\\[\n\t\t\t\t\\Hk\\coloneqq\\Fwk{}.\n\t\t\t\\]","sha256_utf8":"de94aba0f3369e92f0efe0956634bfd94d985d1f603e247bb44055a3b9dd29ef"}]},{"id":"S013","semantic_type":"notation","kind":"notation","title":"步长比与目标差","source_units":[{"start":33055,"end":33216,"start_line":597,"end_line":601,"raw_tex":"\\begin{equation}\\label{eq:rhoP}\n\t\t\t\t\\rhok\\coloneqq\\tfrac{\\gamk}{\\gamma_{k-1}}\n\t\t\t\\quad\\text{and}\\quad\n\t\t\t\tP_k\\coloneqq\\varphi(x^k)-\\min\\varphi.\n\t\t\t\\end{equation}","sha256_utf8":"e35c4cbc39abfdc07fab9bc144057aec05609890af743d2bb9ffa8c5b996697d"}]},{"id":"S014","semantic_type":"lemma","kind":"lemma","title":"局部估计与前向算子恒等式","source_units":[{"start":30817,"end":31629,"start_line":547,"end_line":574,"raw_tex":"\\begin{lemma}\\label{thm:PG:cLl}%\n\t\t\t\tSuppose that \\cref{ass:PG} holds, and let \\(x^{k-1},x^k\\in\\R^n\\).\n\t\t\t\tThen, with \\(L_k\\), \\(\\Lk\\), and \\(\\ck\\) as in \\eqref{eq:MML_k} and \\eqref{eq:CL} the following hold:\n\t\t\t\t\\begin{enumerate}\n\t\t\t\t\\item \\label{thm:PG:CL}%\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\|\\nabla f(x^{k-1})-\\nabla f(x^k)\\|^2\n\t\t\t\t\t=\n\t\t\t\t\t\t\\ck\\Lk\\|x^{k-1}-x^k\\|^2\n\t\t\t\t\t\\), that is, \\(\\ck\\Lk=L_k^2\\).\n\t\t\t\t\\item \\label{thm:PG:H}%\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\|\\Hk(x^{k-1})-\\Hk(x^k)\\|^2\n\t\t\t\t\t=\n\t\t\t\t\t\t\\bigl(1-\\gamk\\Lk(2-\\gamk\\ck)\\bigr)\n\t\t\t\t\t\t\\|x^{k-1}-x^k\\|^2\n\t\t\t\t\t\\).\n\t\t\t\t\\item \\label{thm:Lk<=Ck}%\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\Lk\n\t\t\t\t\t\\leq\n\t\t\t\t\t\tL_k\n\t\t\t\t\t\\leq\n\t\t\t\t\t\tL_{f,\\V}\n\t\t\t\t\t\\)\n\t\t\t\t\tand \\(L_k\\leq\\ck\\), where \\(L_{f,\\V}\\) is a Lipschitz modulus for \\(\\nabla f\\) on a compact convex set \\(\\V\\) containing \\(x^{k-1}\\) and \\(x^k\\).\n\t\t\t\t\\end{enumerate}\n\t\t\t\t","sha256_utf8":"876cbe1296e21f53e333e224204f3273102e3e8b152e4e10e5a54933ff5dd9f3"},{"start":32247,"end":32262,"start_line":590,"end_line":591,"raw_tex":"\n\t\t\t\\end{lemma}","sha256_utf8":"89722f535bb9a4a7116c0f15610164ec8ca7063e5e49cf48287165d918fc35e7"}]},{"id":"S015","semantic_type":"proof","kind":"proof","title":"局部估计引理证明","source_units":[{"start":31629,"end":32247,"start_line":574,"end_line":590,"raw_tex":"\\begin{proof}\n\t\t\t\t\tThe first assertion is of trivial verification, and similarly the third one follows from the Cauchy-Schwarz inequality and the Baillon-Haddad theorem \\cite[Cor. 10]{baillon1977quelques}, see also \\cite[Cor. 18.17]{bauschke2017convex}.\n\t\t\t\t\tTo conclude, observe that\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\|\\Hk(x^{k-1})-\\Hk(x^k)\\|^2\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\|x^{k-1}-x^k\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\gamk^2\n\t\t\t\t\t\t\\|\\nabla f(x^{k-1})-\\nabla f(x^k)\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t-\n\t\t\t\t\t\t2\\gamk\\innprod{\\nabla f(x^{k-1})-\\nabla f(x^k)}{x^{k-1}-x^k},\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tfrom which assertion \\itemref{thm:PG:H} follows.\n\t\t\t\t\\end{proof}","sha256_utf8":"6af562b91adc55e7ddaf317dfb38e09bae91c3ce9f244354980d910c0c6b2b99"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"验证乘积关系与模量界","source_units":[{"start":31643,"end":31883,"start_line":575,"end_line":575,"raw_tex":"\t\t\t\t\tThe first assertion is of trivial verification, and similarly the third one follows from the Cauchy-Schwarz inequality and the Baillon-Haddad theorem \\cite[Cor. 10]{baillon1977quelques}, see also \\cite[Cor. 18.17]{bauschke2017convex}.\n","sha256_utf8":"12dd48ab7b30a42be80c3c1104fcfb03a6528732235db15a971778d05fe24d3d"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"展开前向算子差的平方","source_units":[{"start":31883,"end":32232,"start_line":576,"end_line":589,"raw_tex":"\t\t\t\t\tTo conclude, observe that\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\|\\Hk(x^{k-1})-\\Hk(x^k)\\|^2\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\|x^{k-1}-x^k\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\gamk^2\n\t\t\t\t\t\t\\|\\nabla f(x^{k-1})-\\nabla f(x^k)\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t-\n\t\t\t\t\t\t2\\gamk\\innprod{\\nabla f(x^{k-1})-\\nabla f(x^k)}{x^{k-1}-x^k},\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tfrom which assertion \\itemref{thm:PG:H} follows.\n","sha256_utf8":"2b84089fa40afc17028a1e41cf1fea71dfdef4d65443037b654e0f35b311cc42"}]},{"id":"S018","semantic_type":"lemma","kind":"lemma","title":"任意步长的进展不等式","source_units":[{"start":33220,"end":33904,"start_line":602,"end_line":623,"raw_tex":"\\begin{lemma}\\label{thm:PG:descent}%\n\t\t\t\tSuppose \\cref{ass:PG} holds, and consider a sequence \\(\\seq{x^k}\\) generated by {\\algnamefont PGM} iterations \\(x^{k+1}=\\FBk*{x^k}\\).\n\t\t\t\tThen%\n\t\t\t\t\\begin{align*}\n\t\t\t\t&\n\t\t\t\t\t\\tfrac{1}{2}\\|x^{k+1}-x^\\star\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\gamk*(1+\\rhok*)P_k\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{1}{4}\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{1}{2}\\|x^k-x^\\star\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\rhok*\\gamk*P_{k-1}\n\t\t\t\t\t-\n\t\t\t\t\t\\rhok*^2\\gamk\\Lk \\left(\n\t\t\t \t\t\t1-\\gamk \\ck  \t\t\\right)\n\t\t\t\t\t\\|x^{k-1}-x^k\\|^2\n\t\t\t\t\\end{align*}\n\t\t\t\tholds for any \\(k\\geq1\\) and \\(x^\\star\\in\\argmin\\varphi\\), where \\(\\Lk\\) and \\(\\ck\\) are as in \\eqref{eq:CL}, and \\(\\rhok\\) and \\(P_k\\) as in \\eqref{eq:rhoP}.%\n\t\t\t\t","sha256_utf8":"e93e89d7f4cbe843c9e54bdfd5fe2efdb0ce67a1a9d2a628a930d9560935b410"},{"start":38499,"end":38514,"start_line":784,"end_line":785,"raw_tex":"\n\t\t\t\\end{lemma}","sha256_utf8":"89722f535bb9a4a7116c0f15610164ec8ca7063e5e49cf48287165d918fc35e7"}]},{"id":"S019","semantic_type":"proof","kind":"proof","title":"进展不等式完整证明","source_units":[{"start":33904,"end":38499,"start_line":623,"end_line":784,"raw_tex":"\\begin{proof}\n\t\t\t\t\tThe subgradient characterization\n\t\t\t\t\t\\begin{equation}\\label{eq:PG:subgrad}\n\t\t\t\t\t\t\\tfrac{\\Hk*(x^k)-x^{k+1}}{\\gamk*}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\tfrac{x^k-x^{k+1}}{\\gamk*}-\\nabla f(x^k)\n\t\t\t\t\t\\in\n\t\t\t\t\t\t\\partial g(x^{k+1})\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\tof \\(x^{k+1}=\\FBk*{x^k}\\) implies that\n\t\t\t\t\t{\\mathtight[0.95]%\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\tg(x^\\star)-g(x^{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x^\\star- x^{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{\\gamk*}\\innprod{x^k-x^{k+1}}{x^\\star-x^{k+1}}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\tg(x^\\star)-g(x^{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x^\\star- x^{k+1}}\n\t\t\t\t\t\t\t}_{\\text{(A)}}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\gamk*}\\|x^k-x^\\star\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\gamk*}\\|x^{k+1}-x^\\star\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\gamk*}\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t}%\n\t\t\t\t\tholds for any solution \\(x^\\star\\).\n\t\t\t\t\tWe next proceed to upper bound the term (A) as\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\text{(A)}\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x^\\star-x^k}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x^k-x^{k+1}}\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x^\\star-x^k}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{\\gamk}\\innprod{\\Hk(x^{k-1})-x^k}{x^{k+1}-x^k}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{\\gamk}\\innprod{\\Hk(x^{k-1})-\\Hk(x^k)}{x^k-x^{k+1}}\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:PG:subgrad}}{} &\n\t\t\t\t\t\t\\fillwidthof[c]{\\innprod{\\nabla f(x^k)}{x^\\star-x^k}}{\n\t\t\t\t\t\t\tf(x^\\star) - f(x^k)\n\t\t\t\t\t\t}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\fillwidthof[c]{\\tfrac{1}{\\gamk}\\innprod{\\Hk(x^{k-1})-x^k}{x^{k+1}-x^k}}{\n\t\t\t\t\t\t\tg(x^{k+1})-g(x^k)\n\t\t\t\t\t\t}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\\tfrac{1}{\\gamk}\\innprod{\\Hk(x^{k-1})-\\Hk(x^k)}{x^k-x^{k+1}}\n\t\t\t\t\t\t}_{\\text{(B)}}.\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tWe bound the term (B) by Young's inequality with parameter \\(\\varepsilon_{k+1}\\) as\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\text{(B)}\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\tfrac{\\varepsilon_{k+1}}{2\\gamk}\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{2\\varepsilon_{k+1}\\gamk}\\|\\Hk(x^{k-1})-\\Hk(x^k)\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\overrel{\\ref{thm:PG:H}}{} &\n\t\t\t\t\t\t\\tfrac{\\varepsilon_{k+1}}{2\\gamk}\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1-\\gamk\\Lk(2-\\gamk\\ck)}{2\\varepsilon_{k+1}\\gamk}\\|x^{k-1}-x^k\\|^2.\n\t\t\t\t\t\t\\numberthis\\label{eq:B:CL}\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tLet \\(\\varphi_\\star \\coloneqq \\min \\varphi\\).\n\t\t\t\t\tThe three inequalities combined give\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t0\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\varphi_\\star-\\varphi(x^k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{2\\gamk*}\\|x^k-x^\\star\\|^2\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{1}{2\\gamk*}\\|x^{k+1}-x^\\star\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\left(\\tfrac{\\varepsilon_{k+1}}{2\\gamk}-\\tfrac{1}{2\\gamk*}\\right)\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1-\\gamk\\Lk(2-\\gamk\\ck)}{2\\varepsilon_{k+1}\\gamk}\\|x^{k-1}-x^k\\|^2.\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tUsing again the subgradient \\eqref{eq:PG:subgrad} (since \\(\\partial\\varphi=\\nabla f+\\partial g\\)) one has\n\t\t\t\t\t\\begin{equation}\\label{eq:PG:vk}\n\t\t\t\t\t\tv^k\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\tfrac{x^{k-1}-x^k}{\\gamk}-(\\nabla f(x^{k-1})-\\nabla f(x^k))\n\t\t\t\t\t\\in\n\t\t\t\t\t\t\\partial\\varphi(x^k),\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\thence, for any \\(\\mytheta_{k+1}\\geq0\\),\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t0\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\mytheta_{k+1}\\left(\n\t\t\t\t\t\t\t\\varphi(x^{k-1})-\\varphi(x^k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\innprod{v^k}{x^{k-1}-x^k}\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\mytheta_{k+1}\\left(\n\t\t\t\t\t\t\t\\varphi(x^{k-1})-\\varphi(x^k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{\\gamk}\\|x^k-x^{k-1}\\|^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla f(x^{k-1})-\\nabla f(x^k)}{x^{k-1}-x^k}\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\mytheta_{k+1}\\left(\n\t\t\t\t\t\t\t\\varphi(x^{k-1})-\\varphi(x^k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1-\\gamk\\Lk}{\\gamk}\\|x^k-x^{k-1}\\|^2\n\t\t\t\t\t\t\\right).\n\t\t\t\t\t\t\\numberthis\\label{eq:PG:subgradineq}\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tBy summing the last two inequalities, multiplying by \\(\\gamk*\\), and observing that\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\varphi_\\star-\\varphi(x^k)+\\mytheta_{k+1}(\\varphi(x^{k-1})-\\varphi(x^k))\n\t\t\t\t\t=\n\t\t\t\t\t\t\\mytheta_{k+1}P_{k-1}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t(1+\\mytheta_{k+1})P_k,\n\t\t\t\t\t\\]\n\t\t\t\t\twe obtain\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t&\n\t\t\t\t\t\t\\tfrac{1}{2}\\|x^{k+1}-x^\\star\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\gamk*(1+\\mytheta_{k+1})P_k\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1-\\varepsilon_{k+1}\\rhok*}{2}\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\tfrac{1}{2}\\|x^k-x^\\star\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\mytheta_{k+1}\\gamk*P_{k-1}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\rhok*\\left(\n\t\t\t\t\t\t\t\\tfrac{1-\\gamk\\Lk(2-\\gamk\\ck)}{2\\varepsilon_{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\mytheta_{k+1}(1-\\gamk\\Lk)\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\\|x^{k-1}-x^k\\|^2.\n\t\t\t\t\t\\numberthis\\label{eq:PG:descentdelta}\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tSelecting \\(\\mytheta_{k+1}=\\rhok*\\) and \\(\\varepsilon_{k+1}=\\nicefrac{1}{2\\rhok*}\\) results in the claimed inequality.\n\t\t\t\t\t\\qedhere\n\t\t\t\t\\end{proof}","sha256_utf8":"4136d3d697ca9670aab38930e4ccb8a50b4d57785003d9934a2a22403b772ad6"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"近端最优性给出距离不等式","source_units":[{"start":33918,"end":34809,"start_line":624,"end_line":657,"raw_tex":"\t\t\t\t\tThe subgradient characterization\n\t\t\t\t\t\\begin{equation}\\label{eq:PG:subgrad}\n\t\t\t\t\t\t\\tfrac{\\Hk*(x^k)-x^{k+1}}{\\gamk*}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\tfrac{x^k-x^{k+1}}{\\gamk*}-\\nabla f(x^k)\n\t\t\t\t\t\\in\n\t\t\t\t\t\t\\partial g(x^{k+1})\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\tof \\(x^{k+1}=\\FBk*{x^k}\\) implies that\n\t\t\t\t\t{\\mathtight[0.95]%\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\tg(x^\\star)-g(x^{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x^\\star- x^{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{\\gamk*}\\innprod{x^k-x^{k+1}}{x^\\star-x^{k+1}}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\tg(x^\\star)-g(x^{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x^\\star- x^{k+1}}\n\t\t\t\t\t\t\t}_{\\text{(A)}}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\gamk*}\\|x^k-x^\\star\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\gamk*}\\|x^{k+1}-x^\\star\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\gamk*}\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t}%\n\t\t\t\t\tholds for any solution \\(x^\\star\\).\n","sha256_utf8":"9eb42fecd82aa781062a7091dc0f74e56d71d30347fba80fb84c1782d9b03f27"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"凸性分解梯度内积","source_units":[{"start":34809,"end":35626,"start_line":658,"end_line":685,"raw_tex":"\t\t\t\t\tWe next proceed to upper bound the term (A) as\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\text{(A)}\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x^\\star-x^k}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x^k-x^{k+1}}\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x^\\star-x^k}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{\\gamk}\\innprod{\\Hk(x^{k-1})-x^k}{x^{k+1}-x^k}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{\\gamk}\\innprod{\\Hk(x^{k-1})-\\Hk(x^k)}{x^k-x^{k+1}}\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:PG:subgrad}}{} &\n\t\t\t\t\t\t\\fillwidthof[c]{\\innprod{\\nabla f(x^k)}{x^\\star-x^k}}{\n\t\t\t\t\t\t\tf(x^\\star) - f(x^k)\n\t\t\t\t\t\t}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\fillwidthof[c]{\\tfrac{1}{\\gamk}\\innprod{\\Hk(x^{k-1})-x^k}{x^{k+1}-x^k}}{\n\t\t\t\t\t\t\tg(x^{k+1})-g(x^k)\n\t\t\t\t\t\t}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\\tfrac{1}{\\gamk}\\innprod{\\Hk(x^{k-1})-\\Hk(x^k)}{x^k-x^{k+1}}\n\t\t\t\t\t\t}_{\\text{(B)}}.\n\t\t\t\t\t\\end{align*}\n","sha256_utf8":"2b74ee4079846700487d22482edad154030acebb2f14521de78f0baee1c00522"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"杨氏不等式控制前向项","source_units":[{"start":35626,"end":36137,"start_line":686,"end_line":699,"raw_tex":"\t\t\t\t\tWe bound the term (B) by Young's inequality with parameter \\(\\varepsilon_{k+1}\\) as\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\text{(B)}\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\tfrac{\\varepsilon_{k+1}}{2\\gamk}\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{2\\varepsilon_{k+1}\\gamk}\\|\\Hk(x^{k-1})-\\Hk(x^k)\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\overrel{\\ref{thm:PG:H}}{} &\n\t\t\t\t\t\t\\tfrac{\\varepsilon_{k+1}}{2\\gamk}\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1-\\gamk\\Lk(2-\\gamk\\ck)}{2\\varepsilon_{k+1}\\gamk}\\|x^{k-1}-x^k\\|^2.\n\t\t\t\t\t\t\\numberthis\\label{eq:B:CL}\n\t\t\t\t\t\\end{align*}\n","sha256_utf8":"a09c8355a42721f63a23705cfea0f5959cee6c9292614e3f8ebb4d58c94bab53"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"合并三项基本估计","source_units":[{"start":36137,"end":36627,"start_line":700,"end_line":716,"raw_tex":"\t\t\t\t\tLet \\(\\varphi_\\star \\coloneqq \\min \\varphi\\).\n\t\t\t\t\tThe three inequalities combined give\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t0\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\varphi_\\star-\\varphi(x^k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{2\\gamk*}\\|x^k-x^\\star\\|^2\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{1}{2\\gamk*}\\|x^{k+1}-x^\\star\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\left(\\tfrac{\\varepsilon_{k+1}}{2\\gamk}-\\tfrac{1}{2\\gamk*}\\right)\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1-\\gamk\\Lk(2-\\gamk\\ck)}{2\\varepsilon_{k+1}\\gamk}\\|x^{k-1}-x^k\\|^2.\n\t\t\t\t\t\\end{align*}\n","sha256_utf8":"0e807c8d8f3e6870df1ec1d2642b46c83e278fb48c2669445b135d27ad687da6"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"目标次梯度补充相邻目标差","source_units":[{"start":36627,"end":37569,"start_line":717,"end_line":751,"raw_tex":"\t\t\t\t\tUsing again the subgradient \\eqref{eq:PG:subgrad} (since \\(\\partial\\varphi=\\nabla f+\\partial g\\)) one has\n\t\t\t\t\t\\begin{equation}\\label{eq:PG:vk}\n\t\t\t\t\t\tv^k\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\tfrac{x^{k-1}-x^k}{\\gamk}-(\\nabla f(x^{k-1})-\\nabla f(x^k))\n\t\t\t\t\t\\in\n\t\t\t\t\t\t\\partial\\varphi(x^k),\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\thence, for any \\(\\mytheta_{k+1}\\geq0\\),\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t0\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\mytheta_{k+1}\\left(\n\t\t\t\t\t\t\t\\varphi(x^{k-1})-\\varphi(x^k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\innprod{v^k}{x^{k-1}-x^k}\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\mytheta_{k+1}\\left(\n\t\t\t\t\t\t\t\\varphi(x^{k-1})-\\varphi(x^k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{\\gamk}\\|x^k-x^{k-1}\\|^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla f(x^{k-1})-\\nabla f(x^k)}{x^{k-1}-x^k}\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\mytheta_{k+1}\\left(\n\t\t\t\t\t\t\t\\varphi(x^{k-1})-\\varphi(x^k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1-\\gamk\\Lk}{\\gamk}\\|x^k-x^{k-1}\\|^2\n\t\t\t\t\t\t\\right).\n\t\t\t\t\t\t\\numberthis\\label{eq:PG:subgradineq}\n\t\t\t\t\t\\end{align*}\n","sha256_utf8":"3fe2409d99d21e2cb6399ccb124fd56294c1bbba4882cebcde560e367e358694"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"合并并选择参数得到引理","source_units":[{"start":37569,"end":38484,"start_line":752,"end_line":783,"raw_tex":"\t\t\t\t\tBy summing the last two inequalities, multiplying by \\(\\gamk*\\), and observing that\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\varphi_\\star-\\varphi(x^k)+\\mytheta_{k+1}(\\varphi(x^{k-1})-\\varphi(x^k))\n\t\t\t\t\t=\n\t\t\t\t\t\t\\mytheta_{k+1}P_{k-1}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t(1+\\mytheta_{k+1})P_k,\n\t\t\t\t\t\\]\n\t\t\t\t\twe obtain\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t&\n\t\t\t\t\t\t\\tfrac{1}{2}\\|x^{k+1}-x^\\star\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\gamk*(1+\\mytheta_{k+1})P_k\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1-\\varepsilon_{k+1}\\rhok*}{2}\\|x^k-x^{k+1}\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\tfrac{1}{2}\\|x^k-x^\\star\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\mytheta_{k+1}\\gamk*P_{k-1}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\rhok*\\left(\n\t\t\t\t\t\t\t\\tfrac{1-\\gamk\\Lk(2-\\gamk\\ck)}{2\\varepsilon_{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\mytheta_{k+1}(1-\\gamk\\Lk)\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\\|x^{k-1}-x^k\\|^2.\n\t\t\t\t\t\\numberthis\\label{eq:PG:descentdelta}\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tSelecting \\(\\mytheta_{k+1}=\\rhok*\\) and \\(\\varepsilon_{k+1}=\\nicefrac{1}{2\\rhok*}\\) results in the claimed inequality.\n\t\t\t\t\t\\qedhere\n","sha256_utf8":"a0188ca08f3dd63496b645a42c8065ed075c11e5dc719b140876d31f4a5e6030"}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","title":"扩张集合上的局部余强制界","source_units":[{"start":111163,"end":111843,"start_line":2434,"end_line":2458,"raw_tex":"\\begin{lemma}\\label{lem:enlarged:ck}\n\t\t\t\tSuppose that \\cref{ass:f} holds.\n\t\t\t\tThen, for every \\(x,y\\in\\R^n\\) it holds that\n\t\t\t\t\\[\n\t\t\t\t\tf(y)\n\t\t\t\t\\leq\n\t\t\t\t\tf(x)\n\t\t\t\t\t+\n\t\t\t\t\t\\innprod{\\nabla f(x)}{y-x}\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{1}{2L_{f,\\overline\\V}}\n\t\t\t\t\t\\|\\nabla f(x) - \\nabla f(y)\\|^{2},\n\t\t\t\t\\]\n\t\t\t\twhere \\(L_{f,\\overline{\\V}}\\) is a Lipschitz modulus for \\(\\nabla f\\) on \\(\\overline{\\V}\\coloneqq\\V+\\cball{0}{\\diam(\\V)}\\), and \\(\\V\\subseteq\\R^n\\) is a bounded and convex set that contains \\(x\\) and \\(y\\).\n\t\t\t\tIn particular,\n\t\t\t\t\\[\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\|\\nabla f(x) - \\nabla f(y)\\|^2\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\innprod{\\nabla f(x) - \\nabla f(y)}{x-y}\n\t\t\t\t\t}\n\t\t\t\t\\leq\n\t\t\t\t\tL_{f,\\overline\\V}.\n\t\t\t\t\\]\n\t\t\t\t","sha256_utf8":"25f958f295a10b016a5955cd791d70a9cdd38764e113052c67aa7ffc2db4e678"},{"start":114424,"end":114439,"start_line":2548,"end_line":2549,"raw_tex":"\n\t\t\t\\end{lemma}","sha256_utf8":"89722f535bb9a4a7116c0f15610164ec8ca7063e5e49cf48287165d918fc35e7"}]},{"id":"S027","semantic_type":"proof","kind":"proof","title":"局部余强制界完整证明","source_units":[{"start":111843,"end":114424,"start_line":2458,"end_line":2548,"raw_tex":"\\begin{proof}\n\t\t\t\t\tWe henceforth fix \\(x,y\\in \\V\\subseteq\\overline\\V\\).\n\t\t\t\t\tSince $\\nabla f$ is Lipschitz continuous on $\\overline\\V$ with modulus \\(L_{f,\\overline\\V}\\), it follows from the descent inequality \\cite[Prop. A.24]{bertsekas2016nonlinear} that\n\t\t\t\t\t\\begin{equation}\\label{eq:holder_ineq}\n\t\t\t\t\t\tf(z)\n\t\t\t\t\t\\leq\n\t\t\t\t\t\tf(y)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla f(y)}{z-y}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{L_{f,\\overline\\V}}{2}\\|z-y\\|^{2}\n\t\t\t\t\t\\quad\n\t\t\t\t\t\t\\forall z \\in \\overline\\V.\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\tLet \\(l_x(y) \\coloneqq f(y) - f(x) - \\innprod{\\nabla f(x)}{y-x}\\), and note that \\(l_x\\) is a convex function with \\(\\nabla l_x(y)=\\nabla f(y)-\\nabla f(x)\\).\n\t\t\t\t\tFor any \\(z \\in \\overline\\V\\), we have\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\tl_x(z)\n\t\t\t\t\t={} &\n\t\t\t\t\t\tf(z) - f(x) - \\innprod{\\nabla f(x)}{z-x}\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:holder_ineq}}{} &\n\t\t\t\t\t\tf(y)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla f(y)}{z-y}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{L_{f,\\overline\\V}}{2}\\|z-y\\|^{2}\n\t\t\t\t\t\t-\n\t\t\t\t\t\tf(x)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\innprod{\\nabla f(x)}{z-x}\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\tf(y) - f(x) - \\innprod{\\nabla f(x)}{y-x}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla f(y) - \\nabla f(x)}{z-y}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{L_{f,\\overline\\V}}{2}\\|z-y\\|^{2}\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\tl_x(y)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla l_x(y)}{z-y}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{L_{f,\\overline\\V}}{2}\\|z-y\\|^{2}.\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tNoticing that $\\nabla l_x(x) = 0$, it follows from convexity of \\(l_x\\) that $x$ is its global minimizer. Hence that \\(\\min l_x=l_x(x)=0\\).\n\t\t\t\t\tLet us denote \\(v\\coloneqq\\frac{1}{\\|\\nabla l_x(y)\\|}\\nabla l_x(y)\\) and set\n\t\t\t\t\t\\(\n\t\t\t\t\t\tz = y - \\frac{\\|\\nabla l_x(y)\\|}{L_{f,\\overline\\V}}v\n\t\t\t\t\t\\).\n\t\t\t\t\tNote that\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\|z-y\\|\n\t\t\t\t\t=\n\t\t\t\t\t\t\\tfrac{\\|\\nabla l_x(y)\\|}{L_{f,\\overline\\V}}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\tfrac{\\|\\nabla f(y)-\\nabla f(x)\\|}{L_{f,\\overline\\V}}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\|y-x\\|,\n\t\t\t\t\t\\]\n\t\t\t\t\tand in particular \\(z\\in \\V+\\cball{y}{\\|y-x\\|}\\subseteq\\overline\\V\\).\n\t\t\t\t\tFrom the previous inequality we get\n\t\t\t\t\t\\begin{qedalign*}\n\t\t\t\t\t\t0\n\t\t\t\t\t=\n\t\t\t\t\t\t\\min l_x\n\t\t\t\t\t=\n\t\t\t\t\t\tl_x(x)\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\tl_x\\Bigl(\n\t\t\t\t\t\t\ty - \\tfrac{\\|\\nabla l_x(y)\\|}{L_{f,\\overline\\V}}v\n\t\t\t\t\t\t\\Bigr)\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:holder_ineq}}{} &\n\t\t\t\t\t\tl_x(y)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{\\|\\nabla l_x(y)\\|}{L_{f,\\overline\\V}}\n\t\t\t\t\t\t\\innprod{\\nabla l_x(y)}{v}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{L_{f,\\overline\\V}}{2}\n\t\t\t\t\t\t\\tfrac{1}{L_{f,\\overline\\V}^{2}}\n\t\t\t\t\t\t\\|\\nabla l_x(y)\\|^{2}\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\tf(y) - f(x) - \\innprod{\\nabla f(x)}{y-x}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{1}{2L_{f,\\overline\\V}}\n\t\t\t\t\t\t\\|\\nabla f(x) - \\nabla f(y)\\|^{2}.\n\t\t\t\t\t\\end{qedalign*}\n\t\t\t\t\\end{proof}","sha256_utf8":"3cbb6783b34958be1b786fe44d479ea1fb8167d915f18f2a630de03effdee017"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"扩张集合上应用下降引理","source_units":[{"start":111857,"end":112335,"start_line":2459,"end_line":2471,"raw_tex":"\t\t\t\t\tWe henceforth fix \\(x,y\\in \\V\\subseteq\\overline\\V\\).\n\t\t\t\t\tSince $\\nabla f$ is Lipschitz continuous on $\\overline\\V$ with modulus \\(L_{f,\\overline\\V}\\), it follows from the descent inequality \\cite[Prop. A.24]{bertsekas2016nonlinear} that\n\t\t\t\t\t\\begin{equation}\\label{eq:holder_ineq}\n\t\t\t\t\t\tf(z)\n\t\t\t\t\t\\leq\n\t\t\t\t\t\tf(y)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla f(y)}{z-y}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{L_{f,\\overline\\V}}{2}\\|z-y\\|^{2}\n\t\t\t\t\t\\quad\n\t\t\t\t\t\t\\forall z \\in \\overline\\V.\n\t\t\t\t\t\\end{equation}\n","sha256_utf8":"b44edc0745ff5713ebcbf8b56dd4ec2fad6458d8748a74b6092f239cf9c8f15e"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"构造非负凸函数并估计","source_units":[{"start":112335,"end":113323,"start_line":2472,"end_line":2504,"raw_tex":"\t\t\t\t\tLet \\(l_x(y) \\coloneqq f(y) - f(x) - \\innprod{\\nabla f(x)}{y-x}\\), and note that \\(l_x\\) is a convex function with \\(\\nabla l_x(y)=\\nabla f(y)-\\nabla f(x)\\).\n\t\t\t\t\tFor any \\(z \\in \\overline\\V\\), we have\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\tl_x(z)\n\t\t\t\t\t={} &\n\t\t\t\t\t\tf(z) - f(x) - \\innprod{\\nabla f(x)}{z-x}\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:holder_ineq}}{} &\n\t\t\t\t\t\tf(y)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla f(y)}{z-y}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{L_{f,\\overline\\V}}{2}\\|z-y\\|^{2}\n\t\t\t\t\t\t-\n\t\t\t\t\t\tf(x)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\innprod{\\nabla f(x)}{z-x}\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\tf(y) - f(x) - \\innprod{\\nabla f(x)}{y-x}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla f(y) - \\nabla f(x)}{z-y}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{L_{f,\\overline\\V}}{2}\\|z-y\\|^{2}\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\tl_x(y)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla l_x(y)}{z-y}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{L_{f,\\overline\\V}}{2}\\|z-y\\|^{2}.\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tNoticing that $\\nabla l_x(x) = 0$, it follows from convexity of \\(l_x\\) that $x$ is its global minimizer. Hence that \\(\\min l_x=l_x(x)=0\\).\n","sha256_utf8":"fd8665525c708fb46d2a9edabdb843750cc69476b9236d702834034bc8a6141d"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"验证试探点属于扩张集合","source_units":[{"start":113323,"end":113752,"start_line":2505,"end_line":2519,"raw_tex":"\t\t\t\t\tLet us denote \\(v\\coloneqq\\frac{1}{\\|\\nabla l_x(y)\\|}\\nabla l_x(y)\\) and set\n\t\t\t\t\t\\(\n\t\t\t\t\t\tz = y - \\frac{\\|\\nabla l_x(y)\\|}{L_{f,\\overline\\V}}v\n\t\t\t\t\t\\).\n\t\t\t\t\tNote that\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\|z-y\\|\n\t\t\t\t\t=\n\t\t\t\t\t\t\\tfrac{\\|\\nabla l_x(y)\\|}{L_{f,\\overline\\V}}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\tfrac{\\|\\nabla f(y)-\\nabla f(x)\\|}{L_{f,\\overline\\V}}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\|y-x\\|,\n\t\t\t\t\t\\]\n\t\t\t\t\tand in particular \\(z\\in \\V+\\cball{y}{\\|y-x\\|}\\subseteq\\overline\\V\\).\n","sha256_utf8":"ed2696207813d536bcffb441e2f48499315d1348b564e850f961f33ff4fe2d17"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"代入试探点完成局部估计","source_units":[{"start":113752,"end":114409,"start_line":2520,"end_line":2547,"raw_tex":"\t\t\t\t\tFrom the previous inequality we get\n\t\t\t\t\t\\begin{qedalign*}\n\t\t\t\t\t\t0\n\t\t\t\t\t=\n\t\t\t\t\t\t\\min l_x\n\t\t\t\t\t=\n\t\t\t\t\t\tl_x(x)\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\tl_x\\Bigl(\n\t\t\t\t\t\t\ty - \\tfrac{\\|\\nabla l_x(y)\\|}{L_{f,\\overline\\V}}v\n\t\t\t\t\t\t\\Bigr)\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:holder_ineq}}{} &\n\t\t\t\t\t\tl_x(y)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{\\|\\nabla l_x(y)\\|}{L_{f,\\overline\\V}}\n\t\t\t\t\t\t\\innprod{\\nabla l_x(y)}{v}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{L_{f,\\overline\\V}}{2}\n\t\t\t\t\t\t\\tfrac{1}{L_{f,\\overline\\V}^{2}}\n\t\t\t\t\t\t\\|\\nabla l_x(y)\\|^{2}\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\tf(y) - f(x) - \\innprod{\\nabla f(x)}{y-x}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{1}{2L_{f,\\overline\\V}}\n\t\t\t\t\t\t\\|\\nabla f(x) - \\nabla f(y)\\|^{2}.\n\t\t\t\t\t\\end{qedalign*}\n","sha256_utf8":"46c0ba56009e73e8e2e68cb04ce0996beda28d8a82bd925f52f9aa95fdeea905"}]},{"id":"S032","semantic_type":"proof","kind":"proof","title":"主收敛定理完整证明","source_units":[{"start":41451,"end":54121,"start_line":847,"end_line":1155,"raw_tex":"\\begin{proof}~\n\t\t\t\t\t\\begin{proofitemize}\n\t\t\t\t\t\\item \\itemref{thm:PG:SD}~\n\t\t\t\t\t\tExpressed in terms of \\(\\U_k(x^\\star)\\), the inequality in \\cref{thm:PG:descent} simplifies to the one in the statement. The update rule for \\(\\gamk\\) implies that the coefficients indicated in the statement are greater than or equal to zero, and the claim follows.\n\n\t\t\t\t\t\\item \\itemref{thm:PG:gamk}~\n\t\t\t\t\t\tThe proven inequality implies that \\(\\seq{\\U_k(x_{}^{\\star })}\\) converges for any \\(x^\\star\\in\\argmin\\varphi\\).\n\t\t\t\t\t\tTaking into account the nonnegativity of the last two terms in the definition of \\(\\U_k(x^\\star)\\), it follows that \\(\\seq{x^k}\\) is bounded.\n\t\t\t\t\t\tAs such, \\(\\V\\) and \\(L_{f,\\V}\\) as in the statement exist.\n\t\t\t\t\t\t\tObserve that\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:relate:MM}\n\t\t\t\t\t\t\t\t\\tfrac{\\gamk}{2\\sqrt{[\\gamk\\Lk(\\gamk\\ck-1)]_+}}\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\tfrac12\n\t\t\t\t\t\t\t\t\\sqrt{\\tfrac{1}{\\Lk\\ck}\\tfrac{\\gamk\\ck}{[\\gamk\\ck-1]_+}}\n\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t\\tfrac{1}{2\\sqrt{\\Lk\\ck}}\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\tfrac{1}{2L_k},\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\twhere the last identity owes to \\cref{thm:PG:CL}, and therefore\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:MM}\n\t\t\t\t\t\t\t\t\\gamk*\n\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t\\min\\set{\\gamk\\sqrt{1+\\rhok},\\tfrac{1}{2L_k}}\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tholds for every \\(k\\).\n\t\t\t\t\t\t\t\t\t\t\t\tSuppose that \\(\\gamk\\leq\\nicefrac{1}{2L_{f,\\V}}\\) for \\(k=0,\\dots,K\\).\n\t\t\t\t\t\t\tBecause of \\eqref{eq:MM} and \\cref{thm:Lk<=Ck}, this implies that\n\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\\gamk*\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\gamk\\sqrt{1+\\rhok}\n\t\t\t\t\t\t\t\\)\n\t\t\t\t\t\t\tfor \\(k=0,\\dots,K-1\\).\n\t\t\t\t\t\t\t\t\t\t\t\tThen, for these \\(k\\) one has that\n\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\\rhok*^2\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t1+\\rhok\n\t\t\t\t\t\t\t>\n\t\t\t\t\t\t\t\t1\n\t\t\t\t\t\t\t\\),\n\t\t\t\t\t\t\twhich inductively gives\n\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\\rhok^2\n\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t2\n\t\t\t\t\t\t\t\\)\n\t\t\t\t\t\t\tfor all \\(k=1,\\dots,K\\) (since \\(\\rho_0 \\geq 1\\)).\n\t\t\t\t\t\t\tWe then have\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:PG:gam2k}\n\t\t\t\t\t\t\t\t\\gamma_K^2\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\rho_K^2\\gamma_{K-1}^2\n\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t2\\gamma_{K-1}^2\n\t\t\t\t\t\t\t\\geq\\dots\\geq\n\t\t\t\t\t\t\t\t2^K\\gamma_0^2, \t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tshowing that in at most \\(k_0\\) iterations stepsize exceeds \\(\\nicefrac{1}{2L_{f,\\V}}\\).\n\t\t\t\t\t\t\tFrom this point, a trivial induction argument using \\eqref{eq:MM} reveals that \\(\\gamk\\geq\\nicefrac{1}{2L_{f,\\V}}\\) for all \\(k\\geq k_0\\).\n\t\t\t\t\t\\item \\itemref{thm:PG:x*}~\n\t\t\t\t\t\tWe begin by observing that, as is apparent from its proof, assertion \\itemref{thm:PG:SD} remains valid if the identity in \\eqref{eq:PG:gamk} is replaced by an inequality ``\\(\\leq\\)''.\n\t\t\t\t\t\tEither way, a telescoping argument yields that\n\t\t\t\t\t\t\\begin{equation}\\label{eq:gPto0}\n\t\t\t\t\t\t\t\\gamk(1+\\rhok-\\rhok*^2)P_{k-1}\\to0\n\t\t\t\t\t\t\\quad\\text{as}\\quad\n\t\t\t\t\t\t\tk\\to\\infty.\n\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\tWe proceed by intermediate claims.\n\t\t\t\t\t\t\\begin{claims}[widest*=3]%\n\t\t\t\t\t\t\\item \\label{claim:infP}\n\t\t\t\t\t\t\t{\\it \\(\\liminf_{k\\to\\infty}P_k=0\\), and in particular \\(\\seq{x^k}\\) admits a limit point \\(x_\\infty \\in \\argmin \\varphi \\).}\n\n\t\t\t\t\t\t\tSince \\(\\seq{x^k}\\) is bounded and \\(\\varphi\\) is lsc, it suffices to show that \\(\\liminf_{k\\to\\infty}P_k=0\\).\n\t\t\t\t\t\t\tIf \\(\\limsup_{k\\to\\infty}(1+\\rhok-\\rhok*^2)>0\\), then \\eqref{eq:gPto0} yields the claim, since \\(\\gamk\\) is bounded away from zero.\n\t\t\t\t\t\t\tAlternatively, since \\(1+\\rhok-\\rhok*^2\\geq0\\), necessarily \\(1+\\rhok-\\rhok*^2\\to0\\), from which it easily follows that \\(\\liminf_{k\\to\\infty}\\rhok>1\\), and that therefore \\(\\gamk\\to\\infty\\);\n\t\t\t\t\t\t\tin this case, because of the inequalities \\(\\gamk\\left(1+\\rhok\\right)P_{k-1}\\leq\\U_k\\leq\\U_1\\), this directly proves that \\(P_k\\to0\\).\n\n\t\t\t\t\t\t\\item\\label{claim:thm:PG:optunique}%\n\t\t\t\t\t\t\t{\\it \\(x_\\infty\\) is the only limit point that belongs to \\(\\argmin\\varphi\\).}\n\n\t\t\t\t\t\t\tSuppose that \\(x_\\infty'\\in\\argmin\\varphi\\) is a limit point.\n\t\t\t\t\t\t\tObserve that\n\t\t\t\t\t\t\t\\[\n\t\t\t\t\t\t\t\t\\innprod{x^k}{x_\\infty - x_\\infty'}\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\U_k(x_\\infty') - \\U_k(x_\\infty) + \\tfrac12\\|x_\\infty\\|^2 - \\tfrac12\\|x_\\infty'\\|^2,\n\t\t\t\t\t\t\t\\]\n\t\t\t\t\t\t\tand since \\(\\seq{\\U_k(x^\\star)}\\) is convergent for all \\(x^\\star\\in \\argmin \\varphi\\), then so is \\(\\seq{\\innprod{x^k}{x_\\infty - x_\\infty'}}\\).\n\t\t\t\t\t\t\tPassing to the limit along the two converging subsequences thus yields\n\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\\innprod{x_\\infty}{x_\\infty - x_\\infty'}\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\innprod{x_\\infty'}{x_\\infty - x_\\infty'}\n\t\t\t\t\t\t\t\\),\n\t\t\t\t\t\t\twhich after rearranging results in \\(\\|x_\\infty - x_\\infty'\\|^2=0\\), establishing the claim.\n\n\t\t\t\t\t\t\\item\\label{claim:thm:PG:allopt}%\n\t\t\t\t\t\t\t{\\it \\(\\seq{x^k}\\) converges to a point in \\(\\argmin\\varphi\\).}\n\n\t\t\t\t\t\t\tLet \\(x_\\infty\\) be the (unique) optimal limit point of \\(\\seq{x^k}\\) as in \\cref{claim:thm:PG:optunique}, and let\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:U}\n\t\t\t\t\t\t\t\tU\\coloneqq\\lim_{k\\to\\infty}\\U_k(x_\\infty),\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\twhich exists since the sequence is monotonically decreasing by assertion \\itemref{thm:PG:SD}.\n\t\t\t\t\t\t\tBecause of the inequality \\(\\frac{1}{2}\\|x^k-x_{\\infty}\\|^2\\leq\\U_k(x_{\\infty})\\), it suffices to show that \\(U=0\\).\n\t\t\t\t\t\t\tObserve that, for every \\(k\\),\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:rhomax}\n\t\t\t\t\t\t\t\t\\rhok\\leq\\rho_{\\rm max}\n\t\t\t\t\t\t\t\\quad\\text{where}\\quad\n\t\t\t\t\t\t\t \t\\rho_{\\rm max}\n\t\t\t\t\t\t\t \\coloneqq\n\t\t\t\t\t\t\t\t\\max\\set{\\tfrac{\\gamma_0}{\\gamma_{-1}}, \\tfrac12(1+\\sqrt{5})}\n\t\t\t\t\t\t\t>\n\t\t\t\t\t\t\t\t1,\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tas it can be verified by induction.\n\t\t\t\t\t\t\tWe consider two mutually exclusive scenarios.\n\t\t\t\t\t\t\t\\begin{proofitemize}[leftmargin=*]\n\t\t\t\t\t\t\t\\item\n\t\t\t\t\t\t\t\tSuppose that \\(\\seq{\\gamk}\\) is bounded, and consider a subsequence \\(\\seq{x^k}[k\\in K]\\) such that \\(\\lim_{K\\ni k\\to\\infty}P_k=0\\), which exists and converges to \\(x_{\\infty}\\) by virtue of the previous claims.\n\t\t\t\t\t\t\t\tExtract \\(K'\\subseteq K\\) such that \\(\\seq{\\gamk*}[k\\in K']\\to\\gamma\\) for some \\(\\gamma>0\\).\n\t\t\t\t\t\t\t\tThen, \\(x^{k+1}=\\prox_{\\gamk*g}(x^k-\\gamk*\\nabla f(x^k))\\to\\prox_{\\gamma g}(x_\\infty-\\gamma\\nabla f(x_\\infty))=x_\\infty\\) as \\(K'\\ni k\\to\\infty\\) as well.\n\t\t\t\t\t\t\t\tThis yields the sought identity\n\t\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\tU\n\t\t\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\t\t\\lim_{k\\to\\infty}\\U_k(x_{\\infty})\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\lim_{K'\\ni k\\to\\infty}\\U_{k+1}(x_{\\infty})\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\t\t\\lim_{K'\\ni k\\to\\infty}\\left(\n\t\t\t\t\t\t\t\t\t\t\\tfrac{1}2\\|x^{k+1}-x_{\\infty}\\|^2\n\t\t\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\t\t\\tfrac{1}2\\|x^{k+1}-x^k\\|^2\n\t\t\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\t\t\\gamk*\\left(1+\\rhok*\\right)P_k\n\t\t\t\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\t\t0,\n\t\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\t\twhere the vanishing of the last term owes to boundedness of \\(\\seq{\\gamk}\\) and \\(\\seq{\\rhok}\\).\n\n\t\t\t\t\t\t\t\\item\n\t\t\t\t\t\t\t\tSuppose instead that \\(\\seq{\\gamk}\\) is unbounded, and let \\(\\seq{\\gamk}[k\\in K']\\) be an arbitrary subsequence such that \\(\\gamk\\to\\infty\\) as \\(K'\\ni k\\to\\infty\\).\n\t\t\t\t\t\t\t\tFrom the inequality\n\t\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\t\\gamk P_{k-1} \\leq \\U_k(x_{\\infty}) \\leq \\U_1(x_\\infty) < \\infty\n\t\t\t\t\t\t\t\t\\),\n\t\t\t\t\t\t\t\twe conclude that\n\t\t\t\t\t\t\t\t\\begin{equation}\\label{eq:Pk:xk-}\n\t\t\t\t\t\t\t\t\tP_{k-1}\\to0\n\t\t\t\t\t\t\t\t~\\text{ and }~\n\t\t\t\t\t\t\t\t\tx^{k-1}\\to x_\\infty\n\t\t\t\t\t\t\t\t\\text{ as }\n\t\t\t\t\t\t\t\t\tK'\\ni k\\to\\infty\n\t\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\t\t(recall that \\(\\varphi\\) is lsc, \\(\\seq{x^k}\\) is bounded, and \\(x_\\infty\\) is the \\emph{unique} optimal limit point).\n\t\t\t\t\t\t\t\tThen, since by \\eqref{eq:rhomax} \\(\\gamma_{k-1}\\geq\\nicefrac{\\gamk}{\\rho_{\\rm max}}\\to\\infty\\) as \\(K'\\ni k\\to\\infty\\), by the same argument also \\(x^{k-2}\\to x_\\infty\\) as \\(K'\\ni k\\to\\infty\\).\n\t\t\t\t\t\t\t\tWith \\(U\\) as in \\eqref{eq:U}, we thus have\n\t\t\t\t\t\t\t\t\\begin{equation}\\label{eq:U:gam:unbounded}\n\t\t\t\t\t\t\t\t\t\tU\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\lim_{k\\to\\infty}\\U_k(x_\\infty)\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\lim_{K'\\ni k\\to\\infty}\\U_{k-1}(x_\\infty)\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\lim_{K'\\ni k\\to\\infty}\\gamma_{k-1}(1+\\rho_{k-1})P_{k-2}\n\t\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\t\tholding for \\emph{any} set of indices \\(K'\\subseteq\\N\\) along which the stepsizes are divergent.\n\n\t\t\t\t\t\t\t\tWe now construct a specific subsequence \\(K\\coloneqq\\set{k_0,k_1,\\dots}\\) as follows: start with \\(k_0=1\\), and for \\(i\\geq0\\) let\n\t\t\t\t\t\t\t\t\\begin{equation}\\label{eq:ki+}\n\t\t\t\t\t\t\t\t\tk_{i+1}=\\min\\set{k\\geq k_i}[\\gamk \\geq \\rho_{\\rm max}\\gamma_{k_i}].\n\t\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\t\tThen, \\(\\seq{\\gamma_{k_i}}[i\\in\\N]\\to\\infty\\), which as argued after \\eqref{eq:Pk:xk-} implies that both \\(\\seq{x^{k_i-1}}[i\\in\\N]\\) and \\(\\seq{x^{k_i-2}}[i\\in\\N]\\) converge to \\(x_\\infty\\).\n\t\t\t\t\t\t\t\tObserve that \\(\\rho_{k_i}>1\\) holds for all \\(i\\in\\N\\) by minimality of \\(k_i\\).\n\t\t\t\t\t\t\t\tNotice further that\n\t\t\t\t\t\t\t\t\\begin{equation}\\label{eq:k-1}\n\t\t\t\t\t\t\t\t\t\\rho_{k_i-1}\\geq\\rho_{\\rm max}^{-1}\n\t\t\t\t\t\t\t\t\\quad\n\t\t\t\t\t\t\t\t\t\\forall i\\in\\N.\n\t\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\t\tTo see why, suppose on the contrary that\n\t\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\t\\rho_{k_i-1}<\\rho_{\\rm max}^{-1}\n\t\t\t\t\t\t\t\t\\);\n\t\t\t\t\t\t\t\tthen in particular \\(\\rho_{k_i-1}<1\\), thereby ensuring by the previous observation that \\(k_i-1\\notin K\\), and thus \\(k_{i-1}\\leq k_i-2\\).\n\t\t\t\t\t\t\t\tThis leads to the contradiction\n\t\t\t\t\t\t\t\t\\[\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_{i-1}}\n\t\t\t\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:ki+}}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i}\n\t\t\t\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:rhomax}}\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-1}\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\rho_{k_i-1}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-2}\n\t\t\t\t\t\t\t\t<\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-2}\n\t\t\t\t\t\t\t\t<\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_{i-1}},\n\t\t\t\t\t\t\t\t\\]\n\t\t\t\t\t\t\t\twhere the last inequality follows either in case \\(k_i-2=k_{i-1}\\) (since \\(\\rho_{\\rm max}>1\\)) or when \\(k_{i-1}<k_i-2<k_i\\) (from minimality in the definition of \\(k_i\\)).\n\t\t\t\t\t\t\t\tThis shows \\eqref{eq:k-1}.\n\n\t\t\t\t\t\t\t\tNext, by the stepsize update at \\cref{state:PG:gamk},\n\t\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\t\\tfrac{1}{\\rho_{\\rm max}}\\gamma_{k_i-2}L_{k_i-2}\n\t\t\t\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:k-1}}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-1}L_{k_i-2}\n\t\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\t\\frac{\\gamma_{k_i-2}L_{k_i-2}}{2\\sqrt{\\left[\\gamma_{k_i-2}^2\\ell_{k_i-2}c_{k_i-2}-\\gamma_{k_i-2}\\ell_{k_i-2}\\right]_+}}\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\t\\frac{\\gamma_{k_i-2}L_{k_i-2}}{2\\sqrt{\\left[\\gamma_{k_i-2}^2L_{k_i-2}^2-\\gamma_{k_i-2}L_{k_i-2}\\right]_+}}\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\t\t\\frac{1}{2\\sqrt{\\left[1-\\frac{1}{\\gamma_{k_i-2}L_{k_i-2}}\\right]_+}},\n\t\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\t\twhere the third inequality uses the fact that \\(L_{k_i-2}^2 = \\ell_{k_i-2}c_{k_i-2}\\), and \\(\\ell_{k_i-2}\\leq L_{k_i-2}\\) by \\cref{thm:PG:cLl}.\n\t\t\t\t\t\t\t\tThis implies that \\(\\seq{\\gamma_{k_i-2}L_{k_i-2}}[i\\in\\N]\\) must be bounded.\n\t\t\t\t\t\t\t\tThe subgradient characterization in \\eqref{eq:PG:vk} and the Cauchy-Schwarz inequality then yield\n\t\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-1} P_{k_i-2}\n\t\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\t\\rho_{k_i-1}\n\t\t\t\t\t\t\t\t\t\\innprod{x^{k_i-2}-x_{\\infty}}{\n\t\t\t\t\t\t\t\t\t\tx^{k_i-2}-x^{k_i-3}\n\t\t\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\t\t\\gamma_{k_i-2}(\\nabla f(x^{k_i-2})-\\nabla f(x^{k_i-3}))\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\t(1+\\gamma_{k_i-2}L_{k_i-2})\n\t\t\t\t\t\t\t\t\t}_{\\text{bounded}}\n\t\t\t\t\t\t\t\t\t\\,\n\t\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\t\\|x^{k_i-2}-x_{\\infty}\\|\n\t\t\t\t\t\t\t\t\t\t\\|x^{k_i-2}-x^{k_i-3}\\|\n\t\t\t\t\t\t\t\t\t}_{\\to0}\n\t\t\t\t\t\t\t\t\\to\n\t\t\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\t\t\\quad\n\t\t\t\t\t\t\t\t\t\\text{as }i\\to\\infty.\n\t\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\t\tCombined with the fact that \\(\\rho_{k_i-1} \\leq \\rho_{\\rm max}\\), it follows from \\eqref{eq:U:gam:unbounded} that \\(U=0\\).\n\t\t\t\t\t\t\t\\end{proofitemize}\n\t\t\t\t\t\t\t\t\t\t\t\t\\end{claims}%\n\n\t\t\t\t\t\\item \\itemref{thm:PG:sublin}~\n\t\t\t\t\t\tA telescoping argument in the descent inequality of assertion \\itemref{thm:PG:SD} yields\n\t\t\t\t\t\t\\[\n\t\t\t\t\t\t\t\\mathcal U_{K+1} + \\sum_{k=1}^K\\gamk\\left(1+\\rhok-\\rhok*^2\\right)P_{k-1} \\leq \\mathcal U_1\n\t\t\t\t\t\t\\]\n\t\t\t\t\t\tfor \\(K\\geq1\\).\n\t\t\t\t\t\tBy further lower bounding \\(\\mathcal U_{K+1}\\) we obtain\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\textstyle\n\t\t\t\t\t\t\t\\gamma_{K+1}(1+\\rho_{K+1})P_K\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\sum_{k=1}^K\\gamk\\left(1+\\rhok-\\rhok*^2\\right)P_{k-1}\n\t\t\t\t\t\t\t\\leq \\mathcal U_1.\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\tConsidering the best-so-far quantity\n\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\\min_{k=0,1,\\ldots, K} P_k\n\t\t\t\t\t\t\\), it follows that\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\min_{k=0,1,\\dots, K} P_k\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\frac{\\mathcal U_1}{\n\t\t\t\t\t\t\t\t\\gamma_{K+1}(1+\\rho_{K+1})\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\sum_{k=1}^K\\gamk(1+\\rhok-\\rhok*^2)\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\frac{\\mathcal U_1}{\n\t\t\t\t\t\t\t\t\\gamma_{K+1}(1+\\rho_{K+1})\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\sum_{k=1}^K\\gamk\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\sum_{k=1}^K(\\gamk\\rhok-\\gamk*\\rhok*)\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\numberthis\\label{eq:O(1k)}\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\frac{\\mathcal U_1}{\n\t\t\t\t\t\t\t\t\\gamma_1\\rho_1+\\sum_{k=1}^{K+1}\\gamk\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\leq\n\t\t\t\t\t\t\t\\frac{\\mathcal U_1}{\n\t\t\t\t\t\t\t\t(K+1)\\gamma_{\\rm min}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\twhere \\(\\gamma_{\\rm min}\\coloneqq\\min\\set{\\gamma_0,\\frac{1}{2L_{f,\\mathcal V}}}\\) is a lower bound on the stepsize, as demonstrated in the proof of assertion \\itemref{thm:PG:gamk}.\n\t\t\t\t\t\t\\qedhere\n\t\t\t\t\t\\end{proofitemize}\n\t\t\t\t\\end{proof}","sha256_utf8":"6c07b5b733ff1d4819214f3ef59e240e700b5ba39e0c986ea1c9330226bdef49"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"步长更新确保能量下降","source_units":[{"start":41464,"end":41796,"start_line":847,"end_line":850,"raw_tex":"~\n\t\t\t\t\t\\begin{proofitemize}\n\t\t\t\t\t\\item \\itemref{thm:PG:SD}~\n\t\t\t\t\t\tExpressed in terms of \\(\\U_k(x^\\star)\\), the inequality in \\cref{thm:PG:descent} simplifies to the one in the statement. The update rule for \\(\\gamk\\) implies that the coefficients indicated in the statement are greater than or equal to zero, and the claim follows.\n","sha256_utf8":"c729b424dea745a68040dbc05163488d7d8f485828c9811b40a50ebb8cf5a518"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"能量下降推出迭代有界","source_units":[{"start":41797,"end":42164,"start_line":852,"end_line":855,"raw_tex":"\t\t\t\t\t\\item \\itemref{thm:PG:gamk}~\n\t\t\t\t\t\tThe proven inequality implies that \\(\\seq{\\U_k(x_{}^{\\star })}\\) converges for any \\(x^\\star\\in\\argmin\\varphi\\).\n\t\t\t\t\t\tTaking into account the nonnegativity of the last two terms in the definition of \\(\\U_k(x^\\star)\\), it follows that \\(\\seq{x^k}\\) is bounded.\n\t\t\t\t\t\tAs such, \\(\\V\\) and \\(L_{f,\\V}\\) as in the statement exist.\n","sha256_utf8":"1adad4ce066d7095b4a4ed2c223f03cfd5f0d038775fe806b7bf3e7da376b010"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"建立步长的比较下界","source_units":[{"start":42164,"end":42718,"start_line":856,"end_line":873,"raw_tex":"\t\t\t\t\t\t\tObserve that\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:relate:MM}\n\t\t\t\t\t\t\t\t\\tfrac{\\gamk}{2\\sqrt{[\\gamk\\Lk(\\gamk\\ck-1)]_+}}\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\tfrac12\n\t\t\t\t\t\t\t\t\\sqrt{\\tfrac{1}{\\Lk\\ck}\\tfrac{\\gamk\\ck}{[\\gamk\\ck-1]_+}}\n\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t\\tfrac{1}{2\\sqrt{\\Lk\\ck}}\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\tfrac{1}{2L_k},\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\twhere the last identity owes to \\cref{thm:PG:CL}, and therefore\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:MM}\n\t\t\t\t\t\t\t\t\\gamk*\n\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t\\min\\set{\\gamk\\sqrt{1+\\rhok},\\tfrac{1}{2L_k}}\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tholds for every \\(k\\).\n","sha256_utf8":"03137093eee2497b6965f867f87ccfeb86ca362c05d69cbcd21698373a791cfc"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"小步长几何增长并保持下界","source_units":[{"start":42718,"end":43718,"start_line":874,"end_line":907,"raw_tex":"\t\t\t\t\t\t\t\t\t\t\t\tSuppose that \\(\\gamk\\leq\\nicefrac{1}{2L_{f,\\V}}\\) for \\(k=0,\\dots,K\\).\n\t\t\t\t\t\t\tBecause of \\eqref{eq:MM} and \\cref{thm:Lk<=Ck}, this implies that\n\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\\gamk*\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\gamk\\sqrt{1+\\rhok}\n\t\t\t\t\t\t\t\\)\n\t\t\t\t\t\t\tfor \\(k=0,\\dots,K-1\\).\n\t\t\t\t\t\t\t\t\t\t\t\tThen, for these \\(k\\) one has that\n\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\\rhok*^2\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t1+\\rhok\n\t\t\t\t\t\t\t>\n\t\t\t\t\t\t\t\t1\n\t\t\t\t\t\t\t\\),\n\t\t\t\t\t\t\twhich inductively gives\n\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\\rhok^2\n\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t2\n\t\t\t\t\t\t\t\\)\n\t\t\t\t\t\t\tfor all \\(k=1,\\dots,K\\) (since \\(\\rho_0 \\geq 1\\)).\n\t\t\t\t\t\t\tWe then have\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:PG:gam2k}\n\t\t\t\t\t\t\t\t\\gamma_K^2\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\rho_K^2\\gamma_{K-1}^2\n\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t2\\gamma_{K-1}^2\n\t\t\t\t\t\t\t\\geq\\dots\\geq\n\t\t\t\t\t\t\t\t2^K\\gamma_0^2, \t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tshowing that in at most \\(k_0\\) iterations stepsize exceeds \\(\\nicefrac{1}{2L_{f,\\V}}\\).\n\t\t\t\t\t\t\tFrom this point, a trivial induction argument using \\eqref{eq:MM} reveals that \\(\\gamk\\geq\\nicefrac{1}{2L_{f,\\V}}\\) for all \\(k\\geq k_0\\).\n","sha256_utf8":"90ea891849486b4ae7c3c304051db10d9c27278d5a235fbf36f8d5deb1ce6ee3"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"推广步长条件并望远镜求和","source_units":[{"start":43718,"end":44214,"start_line":908,"end_line":917,"raw_tex":"\t\t\t\t\t\\item \\itemref{thm:PG:x*}~\n\t\t\t\t\t\tWe begin by observing that, as is apparent from its proof, assertion \\itemref{thm:PG:SD} remains valid if the identity in \\eqref{eq:PG:gamk} is replaced by an inequality ``\\(\\leq\\)''.\n\t\t\t\t\t\tEither way, a telescoping argument yields that\n\t\t\t\t\t\t\\begin{equation}\\label{eq:gPto0}\n\t\t\t\t\t\t\t\\gamk(1+\\rhok-\\rhok*^2)P_{k-1}\\to0\n\t\t\t\t\t\t\\quad\\text{as}\\quad\n\t\t\t\t\t\t\tk\\to\\infty.\n\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\tWe proceed by intermediate claims.\n\t\t\t\t\t\t\\begin{claims}[widest*=3]%\n","sha256_utf8":"ed7aa90b83ea802c2e55bce2fb56f6075c389bc829dbbeb0fc67fdf351ac11cc"}]},{"id":"S038","semantic_type":"derived_fact","kind":"derived_fact","title":"存在最优聚点","source_units":[{"start":44214,"end":44377,"start_line":918,"end_line":919,"raw_tex":"\t\t\t\t\t\t\\item \\label{claim:infP}\n\t\t\t\t\t\t\t{\\it \\(\\liminf_{k\\to\\infty}P_k=0\\), and in particular \\(\\seq{x^k}\\) admits a limit point \\(x_\\infty \\in \\argmin \\varphi \\).}\n","sha256_utf8":"b7d07780611f94719c9bbb98721df65c3736240a313e534c0ee6e7809e0038e0"}]},{"id":"S039","semantic_type":"proof","kind":"proof","title":"最优聚点存在性证明","source_units":[{"start":44378,"end":44976,"start_line":921,"end_line":924,"raw_tex":"\t\t\t\t\t\t\tSince \\(\\seq{x^k}\\) is bounded and \\(\\varphi\\) is lsc, it suffices to show that \\(\\liminf_{k\\to\\infty}P_k=0\\).\n\t\t\t\t\t\t\tIf \\(\\limsup_{k\\to\\infty}(1+\\rhok-\\rhok*^2)>0\\), then \\eqref{eq:gPto0} yields the claim, since \\(\\gamk\\) is bounded away from zero.\n\t\t\t\t\t\t\tAlternatively, since \\(1+\\rhok-\\rhok*^2\\geq0\\), necessarily \\(1+\\rhok-\\rhok*^2\\to0\\), from which it easily follows that \\(\\liminf_{k\\to\\infty}\\rhok>1\\), and that therefore \\(\\gamk\\to\\infty\\);\n\t\t\t\t\t\t\tin this case, because of the inequalities \\(\\gamk\\left(1+\\rhok\\right)P_{k-1}\\leq\\U_k\\leq\\U_1\\), this directly proves that \\(P_k\\to0\\).\n","sha256_utf8":"e32a2aa5da81e43cabe91c78c850b21392089dbd9cc7c3a9c8baa64cf7a42220"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"正系数子列推出目标差趋零","source_units":[{"start":44378,"end":44635,"start_line":921,"end_line":922,"raw_tex":"\t\t\t\t\t\t\tSince \\(\\seq{x^k}\\) is bounded and \\(\\varphi\\) is lsc, it suffices to show that \\(\\liminf_{k\\to\\infty}P_k=0\\).\n\t\t\t\t\t\t\tIf \\(\\limsup_{k\\to\\infty}(1+\\rhok-\\rhok*^2)>0\\), then \\eqref{eq:gPto0} yields the claim, since \\(\\gamk\\) is bounded away from zero.\n","sha256_utf8":"e0f49aa5c74b5c42a6f59076efd0e2aa99662e1695a0f6c37cb3c8af73a59292"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"系数消失时步长发散","source_units":[{"start":44635,"end":44976,"start_line":923,"end_line":924,"raw_tex":"\t\t\t\t\t\t\tAlternatively, since \\(1+\\rhok-\\rhok*^2\\geq0\\), necessarily \\(1+\\rhok-\\rhok*^2\\to0\\), from which it easily follows that \\(\\liminf_{k\\to\\infty}\\rhok>1\\), and that therefore \\(\\gamk\\to\\infty\\);\n\t\t\t\t\t\t\tin this case, because of the inequalities \\(\\gamk\\left(1+\\rhok\\right)P_{k-1}\\leq\\U_k\\leq\\U_1\\), this directly proves that \\(P_k\\to0\\).\n","sha256_utf8":"3199ba9af05421681f18b0f303bb462762f8faee893b040c34f79d5eb6113198"}]},{"id":"S042","semantic_type":"derived_fact","kind":"derived_fact","title":"最优聚点唯一","source_units":[{"start":44977,"end":45106,"start_line":926,"end_line":927,"raw_tex":"\t\t\t\t\t\t\\item\\label{claim:thm:PG:optunique}%\n\t\t\t\t\t\t\t{\\it \\(x_\\infty\\) is the only limit point that belongs to \\(\\argmin\\varphi\\).}\n","sha256_utf8":"16a018c9b4e39861bcdc5364ea499328ef256ec0aa6d476d0fdef16412ad0006"}]},{"id":"S043","semantic_type":"proof","kind":"proof","title":"最优聚点唯一性证明","source_units":[{"start":45107,"end":45822,"start_line":929,"end_line":943,"raw_tex":"\t\t\t\t\t\t\tSuppose that \\(x_\\infty'\\in\\argmin\\varphi\\) is a limit point.\n\t\t\t\t\t\t\tObserve that\n\t\t\t\t\t\t\t\\[\n\t\t\t\t\t\t\t\t\\innprod{x^k}{x_\\infty - x_\\infty'}\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\U_k(x_\\infty') - \\U_k(x_\\infty) + \\tfrac12\\|x_\\infty\\|^2 - \\tfrac12\\|x_\\infty'\\|^2,\n\t\t\t\t\t\t\t\\]\n\t\t\t\t\t\t\tand since \\(\\seq{\\U_k(x^\\star)}\\) is convergent for all \\(x^\\star\\in \\argmin \\varphi\\), then so is \\(\\seq{\\innprod{x^k}{x_\\infty - x_\\infty'}}\\).\n\t\t\t\t\t\t\tPassing to the limit along the two converging subsequences thus yields\n\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\\innprod{x_\\infty}{x_\\infty - x_\\infty'}\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\innprod{x_\\infty'}{x_\\infty - x_\\infty'}\n\t\t\t\t\t\t\t\\),\n\t\t\t\t\t\t\twhich after rearranging results in \\(\\|x_\\infty - x_\\infty'\\|^2=0\\), establishing the claim.\n","sha256_utf8":"1510b46af4ed514365ec2afd4ac15155d10a7eef74f65dd06e0576c32bdfbe56"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"能量差推出内积收敛","source_units":[{"start":45107,"end":45515,"start_line":929,"end_line":936,"raw_tex":"\t\t\t\t\t\t\tSuppose that \\(x_\\infty'\\in\\argmin\\varphi\\) is a limit point.\n\t\t\t\t\t\t\tObserve that\n\t\t\t\t\t\t\t\\[\n\t\t\t\t\t\t\t\t\\innprod{x^k}{x_\\infty - x_\\infty'}\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\U_k(x_\\infty') - \\U_k(x_\\infty) + \\tfrac12\\|x_\\infty\\|^2 - \\tfrac12\\|x_\\infty'\\|^2,\n\t\t\t\t\t\t\t\\]\n\t\t\t\t\t\t\tand since \\(\\seq{\\U_k(x^\\star)}\\) is convergent for all \\(x^\\star\\in \\argmin \\varphi\\), then so is \\(\\seq{\\innprod{x^k}{x_\\infty - x_\\infty'}}\\).\n","sha256_utf8":"bf27a656601354648def80bb363ea350e17abac1e4c5d8b176f37a47db9f0217"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"沿两子列取极限得到唯一性","source_units":[{"start":45515,"end":45822,"start_line":937,"end_line":943,"raw_tex":"\t\t\t\t\t\t\tPassing to the limit along the two converging subsequences thus yields\n\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\\innprod{x_\\infty}{x_\\infty - x_\\infty'}\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\innprod{x_\\infty'}{x_\\infty - x_\\infty'}\n\t\t\t\t\t\t\t\\),\n\t\t\t\t\t\t\twhich after rearranging results in \\(\\|x_\\infty - x_\\infty'\\|^2=0\\), establishing the claim.\n","sha256_utf8":"cc48f1148019e304399f5ac511e4cb4a81307e97001ac937a4ae8b8c4033fa06"}]},{"id":"S046","semantic_type":"derived_fact","kind":"derived_fact","title":"全序列收敛到最优解","source_units":[{"start":45823,"end":45934,"start_line":945,"end_line":946,"raw_tex":"\t\t\t\t\t\t\\item\\label{claim:thm:PG:allopt}%\n\t\t\t\t\t\t\t{\\it \\(\\seq{x^k}\\) converges to a point in \\(\\argmin\\varphi\\).}\n","sha256_utf8":"dced3765a95bf7bbfa756cb5a540028791f3ca4f8a049176d764990dbb429fb5"}]},{"id":"S047","semantic_type":"proof","kind":"proof","title":"全序列收敛的分类证明","source_units":[{"start":45935,"end":46782,"start_line":948,"end_line":965,"raw_tex":"\t\t\t\t\t\t\tLet \\(x_\\infty\\) be the (unique) optimal limit point of \\(\\seq{x^k}\\) as in \\cref{claim:thm:PG:optunique}, and let\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:U}\n\t\t\t\t\t\t\t\tU\\coloneqq\\lim_{k\\to\\infty}\\U_k(x_\\infty),\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\twhich exists since the sequence is monotonically decreasing by assertion \\itemref{thm:PG:SD}.\n\t\t\t\t\t\t\tBecause of the inequality \\(\\frac{1}{2}\\|x^k-x_{\\infty}\\|^2\\leq\\U_k(x_{\\infty})\\), it suffices to show that \\(U=0\\).\n\t\t\t\t\t\t\tObserve that, for every \\(k\\),\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:rhomax}\n\t\t\t\t\t\t\t\t\\rhok\\leq\\rho_{\\rm max}\n\t\t\t\t\t\t\t\\quad\\text{where}\\quad\n\t\t\t\t\t\t\t \t\\rho_{\\rm max}\n\t\t\t\t\t\t\t \\coloneqq\n\t\t\t\t\t\t\t\t\\max\\set{\\tfrac{\\gamma_0}{\\gamma_{-1}}, \\tfrac12(1+\\sqrt{5})}\n\t\t\t\t\t\t\t>\n\t\t\t\t\t\t\t\t1,\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tas it can be verified by induction.\n\t\t\t\t\t\t\tWe consider two mutually exclusive scenarios.\n","sha256_utf8":"5e508f4378f2551fec287c8662c7cb718d088a991a4ec8a78a227a35e0474223"},{"start":46789,"end":52603,"start_line":966,"end_line":1103,"raw_tex":"\\begin{proofitemize}[leftmargin=*]\n\t\t\t\t\t\t\t\\item\n\t\t\t\t\t\t\t\tSuppose that \\(\\seq{\\gamk}\\) is bounded, and consider a subsequence \\(\\seq{x^k}[k\\in K]\\) such that \\(\\lim_{K\\ni k\\to\\infty}P_k=0\\), which exists and converges to \\(x_{\\infty}\\) by virtue of the previous claims.\n\t\t\t\t\t\t\t\tExtract \\(K'\\subseteq K\\) such that \\(\\seq{\\gamk*}[k\\in K']\\to\\gamma\\) for some \\(\\gamma>0\\).\n\t\t\t\t\t\t\t\tThen, \\(x^{k+1}=\\prox_{\\gamk*g}(x^k-\\gamk*\\nabla f(x^k))\\to\\prox_{\\gamma g}(x_\\infty-\\gamma\\nabla f(x_\\infty))=x_\\infty\\) as \\(K'\\ni k\\to\\infty\\) as well.\n\t\t\t\t\t\t\t\tThis yields the sought identity\n\t\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\tU\n\t\t\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\t\t\\lim_{k\\to\\infty}\\U_k(x_{\\infty})\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\lim_{K'\\ni k\\to\\infty}\\U_{k+1}(x_{\\infty})\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\t\t\\lim_{K'\\ni k\\to\\infty}\\left(\n\t\t\t\t\t\t\t\t\t\t\\tfrac{1}2\\|x^{k+1}-x_{\\infty}\\|^2\n\t\t\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\t\t\\tfrac{1}2\\|x^{k+1}-x^k\\|^2\n\t\t\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\t\t\\gamk*\\left(1+\\rhok*\\right)P_k\n\t\t\t\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\t\t0,\n\t\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\t\twhere the vanishing of the last term owes to boundedness of \\(\\seq{\\gamk}\\) and \\(\\seq{\\rhok}\\).\n\n\t\t\t\t\t\t\t\\item\n\t\t\t\t\t\t\t\tSuppose instead that \\(\\seq{\\gamk}\\) is unbounded, and let \\(\\seq{\\gamk}[k\\in K']\\) be an arbitrary subsequence such that \\(\\gamk\\to\\infty\\) as \\(K'\\ni k\\to\\infty\\).\n\t\t\t\t\t\t\t\tFrom the inequality\n\t\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\t\\gamk P_{k-1} \\leq \\U_k(x_{\\infty}) \\leq \\U_1(x_\\infty) < \\infty\n\t\t\t\t\t\t\t\t\\),\n\t\t\t\t\t\t\t\twe conclude that\n\t\t\t\t\t\t\t\t\\begin{equation}\\label{eq:Pk:xk-}\n\t\t\t\t\t\t\t\t\tP_{k-1}\\to0\n\t\t\t\t\t\t\t\t~\\text{ and }~\n\t\t\t\t\t\t\t\t\tx^{k-1}\\to x_\\infty\n\t\t\t\t\t\t\t\t\\text{ as }\n\t\t\t\t\t\t\t\t\tK'\\ni k\\to\\infty\n\t\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\t\t(recall that \\(\\varphi\\) is lsc, \\(\\seq{x^k}\\) is bounded, and \\(x_\\infty\\) is the \\emph{unique} optimal limit point).\n\t\t\t\t\t\t\t\tThen, since by \\eqref{eq:rhomax} \\(\\gamma_{k-1}\\geq\\nicefrac{\\gamk}{\\rho_{\\rm max}}\\to\\infty\\) as \\(K'\\ni k\\to\\infty\\), by the same argument also \\(x^{k-2}\\to x_\\infty\\) as \\(K'\\ni k\\to\\infty\\).\n\t\t\t\t\t\t\t\tWith \\(U\\) as in \\eqref{eq:U}, we thus have\n\t\t\t\t\t\t\t\t\\begin{equation}\\label{eq:U:gam:unbounded}\n\t\t\t\t\t\t\t\t\t\tU\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\lim_{k\\to\\infty}\\U_k(x_\\infty)\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\lim_{K'\\ni k\\to\\infty}\\U_{k-1}(x_\\infty)\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\lim_{K'\\ni k\\to\\infty}\\gamma_{k-1}(1+\\rho_{k-1})P_{k-2}\n\t\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\t\tholding for \\emph{any} set of indices \\(K'\\subseteq\\N\\) along which the stepsizes are divergent.\n\n\t\t\t\t\t\t\t\tWe now construct a specific subsequence \\(K\\coloneqq\\set{k_0,k_1,\\dots}\\) as follows: start with \\(k_0=1\\), and for \\(i\\geq0\\) let\n\t\t\t\t\t\t\t\t\\begin{equation}\\label{eq:ki+}\n\t\t\t\t\t\t\t\t\tk_{i+1}=\\min\\set{k\\geq k_i}[\\gamk \\geq \\rho_{\\rm max}\\gamma_{k_i}].\n\t\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\t\tThen, \\(\\seq{\\gamma_{k_i}}[i\\in\\N]\\to\\infty\\), which as argued after \\eqref{eq:Pk:xk-} implies that both \\(\\seq{x^{k_i-1}}[i\\in\\N]\\) and \\(\\seq{x^{k_i-2}}[i\\in\\N]\\) converge to \\(x_\\infty\\).\n\t\t\t\t\t\t\t\tObserve that \\(\\rho_{k_i}>1\\) holds for all \\(i\\in\\N\\) by minimality of \\(k_i\\).\n\t\t\t\t\t\t\t\tNotice further that\n\t\t\t\t\t\t\t\t\\begin{equation}\\label{eq:k-1}\n\t\t\t\t\t\t\t\t\t\\rho_{k_i-1}\\geq\\rho_{\\rm max}^{-1}\n\t\t\t\t\t\t\t\t\\quad\n\t\t\t\t\t\t\t\t\t\\forall i\\in\\N.\n\t\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\t\tTo see why, suppose on the contrary that\n\t\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\t\\rho_{k_i-1}<\\rho_{\\rm max}^{-1}\n\t\t\t\t\t\t\t\t\\);\n\t\t\t\t\t\t\t\tthen in particular \\(\\rho_{k_i-1}<1\\), thereby ensuring by the previous observation that \\(k_i-1\\notin K\\), and thus \\(k_{i-1}\\leq k_i-2\\).\n\t\t\t\t\t\t\t\tThis leads to the contradiction\n\t\t\t\t\t\t\t\t\\[\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_{i-1}}\n\t\t\t\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:ki+}}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i}\n\t\t\t\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:rhomax}}\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-1}\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\rho_{k_i-1}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-2}\n\t\t\t\t\t\t\t\t<\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-2}\n\t\t\t\t\t\t\t\t<\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_{i-1}},\n\t\t\t\t\t\t\t\t\\]\n\t\t\t\t\t\t\t\twhere the last inequality follows either in case \\(k_i-2=k_{i-1}\\) (since \\(\\rho_{\\rm max}>1\\)) or when \\(k_{i-1}<k_i-2<k_i\\) (from minimality in the definition of \\(k_i\\)).\n\t\t\t\t\t\t\t\tThis shows \\eqref{eq:k-1}.\n\n\t\t\t\t\t\t\t\tNext, by the stepsize update at \\cref{state:PG:gamk},\n\t\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\t\\tfrac{1}{\\rho_{\\rm max}}\\gamma_{k_i-2}L_{k_i-2}\n\t\t\t\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:k-1}}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-1}L_{k_i-2}\n\t\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\t\\frac{\\gamma_{k_i-2}L_{k_i-2}}{2\\sqrt{\\left[\\gamma_{k_i-2}^2\\ell_{k_i-2}c_{k_i-2}-\\gamma_{k_i-2}\\ell_{k_i-2}\\right]_+}}\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\t\\frac{\\gamma_{k_i-2}L_{k_i-2}}{2\\sqrt{\\left[\\gamma_{k_i-2}^2L_{k_i-2}^2-\\gamma_{k_i-2}L_{k_i-2}\\right]_+}}\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\t\t\\frac{1}{2\\sqrt{\\left[1-\\frac{1}{\\gamma_{k_i-2}L_{k_i-2}}\\right]_+}},\n\t\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\t\twhere the third inequality uses the fact that \\(L_{k_i-2}^2 = \\ell_{k_i-2}c_{k_i-2}\\), and \\(\\ell_{k_i-2}\\leq L_{k_i-2}\\) by \\cref{thm:PG:cLl}.\n\t\t\t\t\t\t\t\tThis implies that \\(\\seq{\\gamma_{k_i-2}L_{k_i-2}}[i\\in\\N]\\) must be bounded.\n\t\t\t\t\t\t\t\tThe subgradient characterization in \\eqref{eq:PG:vk} and the Cauchy-Schwarz inequality then yield\n\t\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-1} P_{k_i-2}\n\t\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\t\\rho_{k_i-1}\n\t\t\t\t\t\t\t\t\t\\innprod{x^{k_i-2}-x_{\\infty}}{\n\t\t\t\t\t\t\t\t\t\tx^{k_i-2}-x^{k_i-3}\n\t\t\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\t\t\\gamma_{k_i-2}(\\nabla f(x^{k_i-2})-\\nabla f(x^{k_i-3}))\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\t(1+\\gamma_{k_i-2}L_{k_i-2})\n\t\t\t\t\t\t\t\t\t}_{\\text{bounded}}\n\t\t\t\t\t\t\t\t\t\\,\n\t\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\t\\|x^{k_i-2}-x_{\\infty}\\|\n\t\t\t\t\t\t\t\t\t\t\\|x^{k_i-2}-x^{k_i-3}\\|\n\t\t\t\t\t\t\t\t\t}_{\\to0}\n\t\t\t\t\t\t\t\t\\to\n\t\t\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\t\t\\quad\n\t\t\t\t\t\t\t\t\t\\text{as }i\\to\\infty.\n\t\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\t\tCombined with the fact that \\(\\rho_{k_i-1} \\leq \\rho_{\\rm max}\\), it follows from \\eqref{eq:U:gam:unbounded} that \\(U=0\\).\n\t\t\t\t\t\t\t\\end{proofitemize}","sha256_utf8":"e5d0505085c02270ace78460bd32030598bcccecf7c4a60584f84ef3a28b0785"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"归约为能量极限为零","source_units":[{"start":45935,"end":46824,"start_line":948,"end_line":966,"raw_tex":"\t\t\t\t\t\t\tLet \\(x_\\infty\\) be the (unique) optimal limit point of \\(\\seq{x^k}\\) as in \\cref{claim:thm:PG:optunique}, and let\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:U}\n\t\t\t\t\t\t\t\tU\\coloneqq\\lim_{k\\to\\infty}\\U_k(x_\\infty),\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\twhich exists since the sequence is monotonically decreasing by assertion \\itemref{thm:PG:SD}.\n\t\t\t\t\t\t\tBecause of the inequality \\(\\frac{1}{2}\\|x^k-x_{\\infty}\\|^2\\leq\\U_k(x_{\\infty})\\), it suffices to show that \\(U=0\\).\n\t\t\t\t\t\t\tObserve that, for every \\(k\\),\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:rhomax}\n\t\t\t\t\t\t\t\t\\rhok\\leq\\rho_{\\rm max}\n\t\t\t\t\t\t\t\\quad\\text{where}\\quad\n\t\t\t\t\t\t\t \t\\rho_{\\rm max}\n\t\t\t\t\t\t\t \\coloneqq\n\t\t\t\t\t\t\t\t\\max\\set{\\tfrac{\\gamma_0}{\\gamma_{-1}}, \\tfrac12(1+\\sqrt{5})}\n\t\t\t\t\t\t\t>\n\t\t\t\t\t\t\t\t1,\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tas it can be verified by induction.\n\t\t\t\t\t\t\tWe consider two mutually exclusive scenarios.\n\t\t\t\t\t\t\t\\begin{proofitemize}[leftmargin=*]\n","sha256_utf8":"b2ca0b9d464e03bcf55c4db1a8153b436e531ff8aaf9bd492cc716ca4179463d"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"有界步长情形的能量极限","source_units":[{"start":46824,"end":47908,"start_line":967,"end_line":991,"raw_tex":"\t\t\t\t\t\t\t\\item\n\t\t\t\t\t\t\t\tSuppose that \\(\\seq{\\gamk}\\) is bounded, and consider a subsequence \\(\\seq{x^k}[k\\in K]\\) such that \\(\\lim_{K\\ni k\\to\\infty}P_k=0\\), which exists and converges to \\(x_{\\infty}\\) by virtue of the previous claims.\n\t\t\t\t\t\t\t\tExtract \\(K'\\subseteq K\\) such that \\(\\seq{\\gamk*}[k\\in K']\\to\\gamma\\) for some \\(\\gamma>0\\).\n\t\t\t\t\t\t\t\tThen, \\(x^{k+1}=\\prox_{\\gamk*g}(x^k-\\gamk*\\nabla f(x^k))\\to\\prox_{\\gamma g}(x_\\infty-\\gamma\\nabla f(x_\\infty))=x_\\infty\\) as \\(K'\\ni k\\to\\infty\\) as well.\n\t\t\t\t\t\t\t\tThis yields the sought identity\n\t\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\tU\n\t\t\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\t\t\\lim_{k\\to\\infty}\\U_k(x_{\\infty})\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\lim_{K'\\ni k\\to\\infty}\\U_{k+1}(x_{\\infty})\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\t\t\\lim_{K'\\ni k\\to\\infty}\\left(\n\t\t\t\t\t\t\t\t\t\t\\tfrac{1}2\\|x^{k+1}-x_{\\infty}\\|^2\n\t\t\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\t\t\\tfrac{1}2\\|x^{k+1}-x^k\\|^2\n\t\t\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\t\t\\gamk*\\left(1+\\rhok*\\right)P_k\n\t\t\t\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\t\t0,\n\t\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\t\twhere the vanishing of the last term owes to boundedness of \\(\\seq{\\gamk}\\) and \\(\\seq{\\rhok}\\).\n","sha256_utf8":"cab93205aefc4e0c33a9445ad74fdde67338a5497b1a3086ac17fc576d583482"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"无界步长子列上的能量表达","source_units":[{"start":47909,"end":49191,"start_line":993,"end_line":1019,"raw_tex":"\t\t\t\t\t\t\t\\item\n\t\t\t\t\t\t\t\tSuppose instead that \\(\\seq{\\gamk}\\) is unbounded, and let \\(\\seq{\\gamk}[k\\in K']\\) be an arbitrary subsequence such that \\(\\gamk\\to\\infty\\) as \\(K'\\ni k\\to\\infty\\).\n\t\t\t\t\t\t\t\tFrom the inequality\n\t\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\t\\gamk P_{k-1} \\leq \\U_k(x_{\\infty}) \\leq \\U_1(x_\\infty) < \\infty\n\t\t\t\t\t\t\t\t\\),\n\t\t\t\t\t\t\t\twe conclude that\n\t\t\t\t\t\t\t\t\\begin{equation}\\label{eq:Pk:xk-}\n\t\t\t\t\t\t\t\t\tP_{k-1}\\to0\n\t\t\t\t\t\t\t\t~\\text{ and }~\n\t\t\t\t\t\t\t\t\tx^{k-1}\\to x_\\infty\n\t\t\t\t\t\t\t\t\\text{ as }\n\t\t\t\t\t\t\t\t\tK'\\ni k\\to\\infty\n\t\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\t\t(recall that \\(\\varphi\\) is lsc, \\(\\seq{x^k}\\) is bounded, and \\(x_\\infty\\) is the \\emph{unique} optimal limit point).\n\t\t\t\t\t\t\t\tThen, since by \\eqref{eq:rhomax} \\(\\gamma_{k-1}\\geq\\nicefrac{\\gamk}{\\rho_{\\rm max}}\\to\\infty\\) as \\(K'\\ni k\\to\\infty\\), by the same argument also \\(x^{k-2}\\to x_\\infty\\) as \\(K'\\ni k\\to\\infty\\).\n\t\t\t\t\t\t\t\tWith \\(U\\) as in \\eqref{eq:U}, we thus have\n\t\t\t\t\t\t\t\t\\begin{equation}\\label{eq:U:gam:unbounded}\n\t\t\t\t\t\t\t\t\t\tU\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\lim_{k\\to\\infty}\\U_k(x_\\infty)\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\lim_{K'\\ni k\\to\\infty}\\U_{k-1}(x_\\infty)\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\lim_{K'\\ni k\\to\\infty}\\gamma_{k-1}(1+\\rho_{k-1})P_{k-2}\n\t\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\t\tholding for \\emph{any} set of indices \\(K'\\subseteq\\N\\) along which the stepsizes are divergent.\n","sha256_utf8":"865279f7ce21fe19fb467c0d997a7d38e999beb0bb8f7d65645093026266161d"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"构造增长子列并控制相邻比值","source_units":[{"start":49192,"end":50849,"start_line":1021,"end_line":1058,"raw_tex":"\t\t\t\t\t\t\t\tWe now construct a specific subsequence \\(K\\coloneqq\\set{k_0,k_1,\\dots}\\) as follows: start with \\(k_0=1\\), and for \\(i\\geq0\\) let\n\t\t\t\t\t\t\t\t\\begin{equation}\\label{eq:ki+}\n\t\t\t\t\t\t\t\t\tk_{i+1}=\\min\\set{k\\geq k_i}[\\gamk \\geq \\rho_{\\rm max}\\gamma_{k_i}].\n\t\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\t\tThen, \\(\\seq{\\gamma_{k_i}}[i\\in\\N]\\to\\infty\\), which as argued after \\eqref{eq:Pk:xk-} implies that both \\(\\seq{x^{k_i-1}}[i\\in\\N]\\) and \\(\\seq{x^{k_i-2}}[i\\in\\N]\\) converge to \\(x_\\infty\\).\n\t\t\t\t\t\t\t\tObserve that \\(\\rho_{k_i}>1\\) holds for all \\(i\\in\\N\\) by minimality of \\(k_i\\).\n\t\t\t\t\t\t\t\tNotice further that\n\t\t\t\t\t\t\t\t\\begin{equation}\\label{eq:k-1}\n\t\t\t\t\t\t\t\t\t\\rho_{k_i-1}\\geq\\rho_{\\rm max}^{-1}\n\t\t\t\t\t\t\t\t\\quad\n\t\t\t\t\t\t\t\t\t\\forall i\\in\\N.\n\t\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\t\tTo see why, suppose on the contrary that\n\t\t\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\t\t\\rho_{k_i-1}<\\rho_{\\rm max}^{-1}\n\t\t\t\t\t\t\t\t\\);\n\t\t\t\t\t\t\t\tthen in particular \\(\\rho_{k_i-1}<1\\), thereby ensuring by the previous observation that \\(k_i-1\\notin K\\), and thus \\(k_{i-1}\\leq k_i-2\\).\n\t\t\t\t\t\t\t\tThis leads to the contradiction\n\t\t\t\t\t\t\t\t\\[\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_{i-1}}\n\t\t\t\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:ki+}}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i}\n\t\t\t\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:rhomax}}\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-1}\n\t\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\rho_{k_i-1}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-2}\n\t\t\t\t\t\t\t\t<\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-2}\n\t\t\t\t\t\t\t\t<\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_{i-1}},\n\t\t\t\t\t\t\t\t\\]\n\t\t\t\t\t\t\t\twhere the last inequality follows either in case \\(k_i-2=k_{i-1}\\) (since \\(\\rho_{\\rm max}>1\\)) or when \\(k_{i-1}<k_i-2<k_i\\) (from minimality in the definition of \\(k_i\\)).\n\t\t\t\t\t\t\t\tThis shows \\eqref{eq:k-1}.\n","sha256_utf8":"f21a9f56e1781e3e33651f8da72f3d4225f0c859dd7b12406dfe69e26542bea1"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"步长规则控制步长模量乘积","source_units":[{"start":50850,"end":51717,"start_line":1060,"end_line":1075,"raw_tex":"\t\t\t\t\t\t\t\tNext, by the stepsize update at \\cref{state:PG:gamk},\n\t\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\t\\tfrac{1}{\\rho_{\\rm max}}\\gamma_{k_i-2}L_{k_i-2}\n\t\t\t\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:k-1}}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-1}L_{k_i-2}\n\t\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\t\\frac{\\gamma_{k_i-2}L_{k_i-2}}{2\\sqrt{\\left[\\gamma_{k_i-2}^2\\ell_{k_i-2}c_{k_i-2}-\\gamma_{k_i-2}\\ell_{k_i-2}\\right]_+}}\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\t\\frac{\\gamma_{k_i-2}L_{k_i-2}}{2\\sqrt{\\left[\\gamma_{k_i-2}^2L_{k_i-2}^2-\\gamma_{k_i-2}L_{k_i-2}\\right]_+}}\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\t\t\\frac{1}{2\\sqrt{\\left[1-\\frac{1}{\\gamma_{k_i-2}L_{k_i-2}}\\right]_+}},\n\t\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\t\twhere the third inequality uses the fact that \\(L_{k_i-2}^2 = \\ell_{k_i-2}c_{k_i-2}\\), and \\(\\ell_{k_i-2}\\leq L_{k_i-2}\\) by \\cref{thm:PG:cLl}.\n\t\t\t\t\t\t\t\tThis implies that \\(\\seq{\\gamma_{k_i-2}L_{k_i-2}}[i\\in\\N]\\) must be bounded.\n","sha256_utf8":"e70af111c3596cdc51388b685a95aaa601fb7bf840c58651ec6980fc0037fef5"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"次梯度估计消去剩余能量","source_units":[{"start":51717,"end":52631,"start_line":1076,"end_line":1105,"raw_tex":"\t\t\t\t\t\t\t\tThe subgradient characterization in \\eqref{eq:PG:vk} and the Cauchy-Schwarz inequality then yield\n\t\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\t\\gamma_{k_i-1} P_{k_i-2}\n\t\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\t\\rho_{k_i-1}\n\t\t\t\t\t\t\t\t\t\\innprod{x^{k_i-2}-x_{\\infty}}{\n\t\t\t\t\t\t\t\t\t\tx^{k_i-2}-x^{k_i-3}\n\t\t\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\t\t\\gamma_{k_i-2}(\\nabla f(x^{k_i-2})-\\nabla f(x^{k_i-3}))\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\t(1+\\gamma_{k_i-2}L_{k_i-2})\n\t\t\t\t\t\t\t\t\t}_{\\text{bounded}}\n\t\t\t\t\t\t\t\t\t\\,\n\t\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\t\\|x^{k_i-2}-x_{\\infty}\\|\n\t\t\t\t\t\t\t\t\t\t\\|x^{k_i-2}-x^{k_i-3}\\|\n\t\t\t\t\t\t\t\t\t}_{\\to0}\n\t\t\t\t\t\t\t\t\\to\n\t\t\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\t\t\\quad\n\t\t\t\t\t\t\t\t\t\\text{as }i\\to\\infty.\n\t\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\t\tCombined with the fact that \\(\\rho_{k_i-1} \\leq \\rho_{\\rm max}\\), it follows from \\eqref{eq:U:gam:unbounded} that \\(U=0\\).\n\t\t\t\t\t\t\t\\end{proofitemize}\n\t\t\t\t\t\t\t\t\t\t\t\t\\end{claims}%\n\n","sha256_utf8":"6cab99f31ef3e3265e79e3ff97adae200e157cf8fffdca5196881292e9e3ba4e"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"下降求和控制加权目标差","source_units":[{"start":52631,"end":53153,"start_line":1106,"end_line":1119,"raw_tex":"\t\t\t\t\t\\item \\itemref{thm:PG:sublin}~\n\t\t\t\t\t\tA telescoping argument in the descent inequality of assertion \\itemref{thm:PG:SD} yields\n\t\t\t\t\t\t\\[\n\t\t\t\t\t\t\t\\mathcal U_{K+1} + \\sum_{k=1}^K\\gamk\\left(1+\\rhok-\\rhok*^2\\right)P_{k-1} \\leq \\mathcal U_1\n\t\t\t\t\t\t\\]\n\t\t\t\t\t\tfor \\(K\\geq1\\).\n\t\t\t\t\t\tBy further lower bounding \\(\\mathcal U_{K+1}\\) we obtain\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\textstyle\n\t\t\t\t\t\t\t\\gamma_{K+1}(1+\\rho_{K+1})P_K\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\sum_{k=1}^K\\gamk\\left(1+\\rhok-\\rhok*^2\\right)P_{k-1}\n\t\t\t\t\t\t\t\\leq \\mathcal U_1.\n\t\t\t\t\t\t\\end{align*}\n","sha256_utf8":"3dbfa6d4014a8108d1faf80eadd5f7b238d7272333f1793d4905238ee2006fe3"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"权重消项得到次线性速率","source_units":[{"start":53153,"end":54110,"start_line":1120,"end_line":1155,"raw_tex":"\t\t\t\t\t\tConsidering the best-so-far quantity\n\t\t\t\t\t\t\\(\n\t\t\t\t\t\t\t\\min_{k=0,1,\\ldots, K} P_k\n\t\t\t\t\t\t\\), it follows that\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\min_{k=0,1,\\dots, K} P_k\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\frac{\\mathcal U_1}{\n\t\t\t\t\t\t\t\t\\gamma_{K+1}(1+\\rho_{K+1})\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\sum_{k=1}^K\\gamk(1+\\rhok-\\rhok*^2)\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\frac{\\mathcal U_1}{\n\t\t\t\t\t\t\t\t\\gamma_{K+1}(1+\\rho_{K+1})\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\sum_{k=1}^K\\gamk\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\sum_{k=1}^K(\\gamk\\rhok-\\gamk*\\rhok*)\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\numberthis\\label{eq:O(1k)}\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\frac{\\mathcal U_1}{\n\t\t\t\t\t\t\t\t\\gamma_1\\rho_1+\\sum_{k=1}^{K+1}\\gamk\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\leq\n\t\t\t\t\t\t\t\\frac{\\mathcal U_1}{\n\t\t\t\t\t\t\t\t(K+1)\\gamma_{\\rm min}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\twhere \\(\\gamma_{\\rm min}\\coloneqq\\min\\set{\\gamma_0,\\frac{1}{2L_{f,\\mathcal V}}}\\) is a lower bound on the stepsize, as demonstrated in the proof of assertion \\itemref{thm:PG:gamk}.\n\t\t\t\t\t\t\\qedhere\n\t\t\t\t\t\\end{proofitemize}\n\t\t\t\t","sha256_utf8":"74120604593c1da0858981be7b7630fc650042d109ce647e53c51f8edfad9c73"}]},{"id":"S056","semantic_type":"external_fact","kind":"external_fact","title":"内积与杨氏不等式","source_units":[],"declared_external":true,"declaration_text":["For Euclidean vectors a,b, |⟨a,b⟩|≤‖a‖‖b‖, and ⟨a,b⟩≤(ε/2)‖a‖²+(1/(2ε))‖b‖² for ε>0."]},{"id":"S057","semantic_type":"external_fact","kind":"external_fact","title":"凸次梯度与近端最优性","source_units":[],"declared_external":true,"declaration_text":["For proper lower semicontinuous convex g and τ>0, p=prox_{τg}(u) iff (u-p)/τ∈∂g(p). Convex subgradients satisfy their supporting inequalities; Fermat's rule and ∂(f+g)=∇f+∂g hold for finite differentiable convex f. The proximal mapping is single valued and continuous jointly in u and positive τ."]},{"id":"S058","semantic_type":"external_fact","kind":"external_fact","title":"光滑函数下降引理","source_units":[],"declared_external":true,"declaration_text":["If ∇f is L-Lipschitz on a convex set containing y and z, then f(z)≤f(y)+⟨∇f(y),z-y⟩+(L/2)‖z-y‖²."]},{"id":"S059","semantic_type":"external_fact","kind":"external_fact","title":"有限维紧性与极限事实","source_units":[],"declared_external":true,"declaration_text":["Bounded Euclidean sequences admit convergent subsequences; lower semicontinuity gives f(x)≤liminf f(x_j) when x_j→x; nonnegative decreasing real sequences converge. A bounded sequence with a unique cluster point converges to that point."]},{"id":"S060","semantic_type":"external_fact","kind":"external_fact","title":"局部连续性定义的等价性","source_units":[],"declared_external":true,"declaration_text":["For a mapping on finite-dimensional Euclidean space, Lipschitz continuity on each compact convex set is equivalent to strict continuity at every point, in the sense cited from Rockafellar–Wets, Theorem 9.2 and Definition 9.1."]},{"id":"S061","semantic_type":"external_fact","kind":"external_fact","title":"全局余强制性定理","source_units":[],"declared_external":true,"declaration_text":["The Baillon–Haddad theorem states that the gradient of a convex differentiable function with globally L-Lipschitz gradient is 1/L-cocoercive: ‖∇f(x)-∇f(y)‖²≤L⟨∇f(x)-∇f(y),x-y⟩. Local applicability is supplied separately by the selected author proof of B0139."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S013","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S011","target_id":"S004","kind":"supports"},{"source_id":"S005","target_id":"S016","kind":"supports"},{"source_id":"S006","target_id":"S016","kind":"supports"},{"source_id":"S008","target_id":"S006","kind":"supports"},{"source_id":"S009","target_id":"S003","kind":"supports"},{"source_id":"S009","target_id":"S030","kind":"supports"},{"source_id":"S010","target_id":"S028","kind":"supports"},{"source_id":"S003","target_id":"S028","kind":"supports"},{"source_id":"S058","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S057","target_id":"S029","kind":"supports"},{"source_id":"S010","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S027","kind":"supports"},{"source_id":"S031","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S006","kind":"supports"},{"source_id":"S026","target_id":"S016","kind":"supports"},{"source_id":"S056","target_id":"S016","kind":"supports"},{"source_id":"S061","target_id":"S016","kind":"supports"},{"source_id":"S010","target_id":"S016","kind":"supports"},{"source_id":"S006","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S003","target_id":"S020","kind":"supports"},{"source_id":"S004","target_id":"S020","kind":"supports"},{"source_id":"S011","target_id":"S020","kind":"supports"},{"source_id":"S012","target_id":"S020","kind":"supports"},{"source_id":"S057","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S057","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S014","target_id":"S022","kind":"supports"},{"source_id":"S056","target_id":"S022","kind":"supports"},{"source_id":"S020","target_id":"S023","kind":"supports"},{"source_id":"S021","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S020","target_id":"S024","kind":"supports"},{"source_id":"S006","target_id":"S024","kind":"supports"},{"source_id":"S057","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S013","target_id":"S025","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S019","kind":"supports"},{"source_id":"S022","target_id":"S019","kind":"supports"},{"source_id":"S023","target_id":"S019","kind":"supports"},{"source_id":"S024","target_id":"S019","kind":"supports"},{"source_id":"S025","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S033","kind":"supports"},{"source_id":"S004","target_id":"S033","kind":"supports"},{"source_id":"S007","target_id":"S033","kind":"supports"},{"source_id":"S013","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S003","target_id":"S034","kind":"supports"},{"source_id":"S010","target_id":"S034","kind":"supports"},{"source_id":"S059","target_id":"S034","kind":"supports"},{"source_id":"S014","target_id":"S035","kind":"supports"},{"source_id":"S006","target_id":"S035","kind":"supports"},{"source_id":"S007","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S014","target_id":"S036","kind":"supports"},{"source_id":"S004","target_id":"S036","kind":"supports"},{"source_id":"S033","target_id":"S037","kind":"supports"},{"source_id":"S007","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S034","target_id":"S040","kind":"supports"},{"source_id":"S036","target_id":"S040","kind":"supports"},{"source_id":"S059","target_id":"S040","kind":"supports"},{"source_id":"S037","target_id":"S041","kind":"supports"},{"source_id":"S033","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S041","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S044","kind":"supports"},{"source_id":"S034","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S038","target_id":"S048","kind":"supports"},{"source_id":"S042","target_id":"S048","kind":"supports"},{"source_id":"S033","target_id":"S048","kind":"supports"},{"source_id":"S007","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S038","target_id":"S049","kind":"supports"},{"source_id":"S042","target_id":"S049","kind":"supports"},{"source_id":"S004","target_id":"S049","kind":"supports"},{"source_id":"S011","target_id":"S049","kind":"supports"},{"source_id":"S057","target_id":"S049","kind":"supports"},{"source_id":"S059","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S042","target_id":"S050","kind":"supports"},{"source_id":"S034","target_id":"S050","kind":"supports"},{"source_id":"S059","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S048","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S004","target_id":"S052","kind":"supports"},{"source_id":"S014","target_id":"S052","kind":"supports"},{"source_id":"S005","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S050","target_id":"S053","kind":"supports"},{"source_id":"S024","target_id":"S053","kind":"supports"},{"source_id":"S056","target_id":"S053","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S049","target_id":"S047","kind":"supports"},{"source_id":"S050","target_id":"S047","kind":"supports"},{"source_id":"S051","target_id":"S047","kind":"supports"},{"source_id":"S052","target_id":"S047","kind":"supports"},{"source_id":"S053","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S033","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S013","target_id":"S055","kind":"supports"},{"source_id":"S036","target_id":"S055","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S032","kind":"supports"},{"source_id":"S035","target_id":"S032","kind":"supports"},{"source_id":"S036","target_id":"S032","kind":"supports"},{"source_id":"S037","target_id":"S032","kind":"supports"},{"source_id":"S046","target_id":"S032","kind":"supports"},{"source_id":"S054","target_id":"S032","kind":"supports"},{"source_id":"S055","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S001","kind":"supports"},{"source_id":"S060","target_id":"S010","kind":"supports"},{"source_id":"S057","target_id":"S011","kind":"supports"}]}}
{"record_id":"2302.03550v2","paper":{"title":"Exponential convergence rates for momentum stochastic gradient descent in the overparametrized setting","authors":["Benjamin Gess","Sebastian Kassing"],"arxiv_id":"2302.03550","version":"v2","journal":"Mathematical Programming","doi":"10.1007/s10107-025-02308-y","abstract_url":"https://arxiv.org/abs/2302.03550v2","pdf_url":"https://arxiv.org/pdf/2302.03550v2"},"context":{"raw_tex":"We assume throughout that $f: \\R^d \\to \\R$ is a differentiable function with $C_L$-Lipschitz continuous gradient,\\footnote{We comment on the necessity of this global Lipschitz continuity in Remark~\\ref{rem:Lip1} and Remark~\\ref{rem:Lip2} below.} for some constant $C_L \\ge 0$, such that $\\inf_{x \\in \\R^d} f(x)=0$. \r\nLet $(\\Omega, (\\mathcal F_{n})_{n \\in \\N_0}, \\mathcal F, \\P)$ be a filtered probability space and let $(X_n)_{n \\in \\N_0}$, $(V_n)_{n \\in \\N_0}$ be $(\\mathcal F_n)_{n \\in \\N_0}$-adapted processes satisfying for all $n \\in \\N_0$\r\n\\begin{align}\\begin{split} \\label{eq:MSGDintro}\r\n\t\tX_{n+1} &= X_n + \\gamma_{n+1}V_{n+1}, \\\\\r\n\t\tV_{n+1}&= V_n - \\gamma_{n+1}\\mu V_n -\\gamma_{n+1}\\nabla f(X_n) + \\gamma_{n+1} D_{n+1},\r\n\t\\end{split}\r\n\\end{align}\n\nwhere $X_0, V_0 \\in L^2(\\Omega, \\cF_0)$, $(\\gamma_{n})_{n \\in \\N}$ is a sequence of strictly positive reals, $\\mu>0$ and $(D_n)_{n \\in \\N}$ is a sequence of $L^2$-martingale differences with respect to the filtration $(\\cF_n)_{n \\in \\N_0}$. In the following, we also call $(X_n)_{n \\in \\N_0}$ given by \\eqref{eq:MSGDintro} the MSGD scheme with step-sizes $(\\gamma_{n})_{n \\in \\N}$ and friction parameter $\\mu$. The choice $(D_n)_{n \\in \\N}=(\\nabla  f(X_{n-1})-\\nabla g(X_{n-1},\\Gamma_{n}))_{n \\in \\N}$ recovers the algorithm \\eqref{eq:MSGDapp}.\r\n\n\n\\textbf{Notation:} We denote by $v^\\dagger$ the transpose of a vector $v \\in \\R^d$, by $A^\\dagger$ the transpose of a matrix $A \\in \\R^{n \\times k}$ and by $\\|A\\|_F$, respectively $\\|A\\|$, the Frobenius norm, respectively operator norm of $A$. Moreover,  $|\\cdot|$ denotes the standard Euclidean norm and $\\langle \\cdot, \\cdot \\rangle$ the standard scalar product on the Euclidean space.\r\n\n\n\tLet $L>0, \\sigma \\ge 0$.\r\n\tLet $\\mathcal D\\subset \\R^d$ be an open set and assume that \r\n\t$$\r\n\t|\\grad f(x)|^2 \\ge 2L f(x)\r\n\t$$\r\n\tfor all $x \\in \\mathcal D$.\r\n\tMoreover, for $n \\in \\N_0$, let\r\n\t$\r\n\t\\IA_n = \\{X_i(\\omega) \\in \\mathcal D \\text{ for all } i =0, \\dots, n\\}\r\n\t$\r\n\tand assume that\r\n\t$$ \r\n\t\\E[|D_{n+1}|^2 | \\cF_n] \\le \\sigma f(X_n), \\quad \\text{ on } \\IA_{n}.\r\n\t$$\r\n\n\nLet $a,b >0$ and let $(E_n)_{n \\in \\N_0}$ be the $(\\cF_n)_{n \\in \\N_0}$-adapted stochastic process given by\r\n\\begin{align} \\label{eq:Lyapunov}\r\n\tE_n = a f(X_{n}) + \\langle \\grad f (X_{n}), V_n \\rangle +\\frac{b}{2} |V_n|^2.\r\n\\end{align}\n\nIn our setting, $(E_n)_{n \\in \\N_0}$ plays the role of a random Lyapunov function. Although, in general, $(E_n)_{n \\in \\N_0}$ might take negative values, assuming the inverse PL-condition there exist choices for $a$ and $b$ such that $(E_n)_{n \\in \\N_0}$ is a non-negative process.\r\n","units":[{"start":12814,"end":13568,"start_line":302,"end_line":308,"raw_tex":"We assume throughout that $f: \\R^d \\to \\R$ is a differentiable function with $C_L$-Lipschitz continuous gradient,\\footnote{We comment on the necessity of this global Lipschitz continuity in Remark~\\ref{rem:Lip1} and Remark~\\ref{rem:Lip2} below.} for some constant $C_L \\ge 0$, such that $\\inf_{x \\in \\R^d} f(x)=0$. \r\nLet $(\\Omega, (\\mathcal F_{n})_{n \\in \\N_0}, \\mathcal F, \\P)$ be a filtered probability space and let $(X_n)_{n \\in \\N_0}$, $(V_n)_{n \\in \\N_0}$ be $(\\mathcal F_n)_{n \\in \\N_0}$-adapted processes satisfying for all $n \\in \\N_0$\r\n\\begin{align}\\begin{split} \\label{eq:MSGDintro}\r\n\t\tX_{n+1} &= X_n + \\gamma_{n+1}V_{n+1}, \\\\\r\n\t\tV_{n+1}&= V_n - \\gamma_{n+1}\\mu V_n -\\gamma_{n+1}\\nabla f(X_n) + \\gamma_{n+1} D_{n+1},\r\n\t\\end{split}\r\n\\end{align}","sha256_utf8":"ba7e59bd1aae8721f2a55e5e7bd37e9277070484b9d951ee543ea828b5955694"},{"start":13570,"end":14116,"start_line":309,"end_line":309,"raw_tex":"where $X_0, V_0 \\in L^2(\\Omega, \\cF_0)$, $(\\gamma_{n})_{n \\in \\N}$ is a sequence of strictly positive reals, $\\mu>0$ and $(D_n)_{n \\in \\N}$ is a sequence of $L^2$-martingale differences with respect to the filtration $(\\cF_n)_{n \\in \\N_0}$. In the following, we also call $(X_n)_{n \\in \\N_0}$ given by \\eqref{eq:MSGDintro} the MSGD scheme with step-sizes $(\\gamma_{n})_{n \\in \\N}$ and friction parameter $\\mu$. The choice $(D_n)_{n \\in \\N}=(\\nabla  f(X_{n-1})-\\nabla g(X_{n-1},\\Gamma_{n}))_{n \\in \\N}$ recovers the algorithm \\eqref{eq:MSGDapp}.\r\n","sha256_utf8":"559507fb5235b739f5f766ab7b11fd63a37e501e4a09de846cca4b887a523cec"},{"start":29291,"end":29680,"start_line":423,"end_line":423,"raw_tex":"\\textbf{Notation:} We denote by $v^\\dagger$ the transpose of a vector $v \\in \\R^d$, by $A^\\dagger$ the transpose of a matrix $A \\in \\R^{n \\times k}$ and by $\\|A\\|_F$, respectively $\\|A\\|$, the Frobenius norm, respectively operator norm of $A$. Moreover,  $|\\cdot|$ denotes the standard Euclidean norm and $\\langle \\cdot, \\cdot \\rangle$ the standard scalar product on the Euclidean space.\r\n","sha256_utf8":"7894398d665a5ced10ae80caf41ad24d96b984b0010af9ee3eff1582febd4b5e"},{"start":35987,"end":36362,"start_line":498,"end_line":511,"raw_tex":"\tLet $L>0, \\sigma \\ge 0$.\r\n\tLet $\\mathcal D\\subset \\R^d$ be an open set and assume that \r\n\t$$\r\n\t|\\grad f(x)|^2 \\ge 2L f(x)\r\n\t$$\r\n\tfor all $x \\in \\mathcal D$.\r\n\tMoreover, for $n \\in \\N_0$, let\r\n\t$\r\n\t\\IA_n = \\{X_i(\\omega) \\in \\mathcal D \\text{ for all } i =0, \\dots, n\\}\r\n\t$\r\n\tand assume that\r\n\t$$ \r\n\t\\E[|D_{n+1}|^2 | \\cF_n] \\le \\sigma f(X_n), \\quad \\text{ on } \\IA_{n}.\r\n\t$$\r\n","sha256_utf8":"b90df0d8a20d7538461393acb3c88792757e00502662bc8d58c965d224df324f"},{"start":50011,"end":50246,"start_line":710,"end_line":713,"raw_tex":"Let $a,b >0$ and let $(E_n)_{n \\in \\N_0}$ be the $(\\cF_n)_{n \\in \\N_0}$-adapted stochastic process given by\r\n\\begin{align} \\label{eq:Lyapunov}\r\n\tE_n = a f(X_{n}) + \\langle \\grad f (X_{n}), V_n \\rangle +\\frac{b}{2} |V_n|^2.\r\n\\end{align}","sha256_utf8":"c6df097fb0297cbd139a5e89fa291f6b77e73f325cc07f96dbffbc3bba100a1e"},{"start":50248,"end":50531,"start_line":714,"end_line":714,"raw_tex":"In our setting, $(E_n)_{n \\in \\N_0}$ plays the role of a random Lyapunov function. Although, in general, $(E_n)_{n \\in \\N_0}$ might take negative values, assuming the inverse PL-condition there exist choices for $a$ and $b$ such that $(E_n)_{n \\in \\N_0}$ is a non-negative process.\r\n","sha256_utf8":"aac402c1f3824c0a68de981de39e1265dba4ed39230e979f95adcc47ade3f34a"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem} \\label{theo2}\r\n\tSet $\\kappa = \\frac{C_L}{L}$. Let \r\n\t\\begin{align*}\r\n\t\t\\mu \\in \r\n\t\t\\begin{cases}\r\n\t\t\t\\bigl[\\frac{1}{\\sqrt 8}\\bigl(5-\\sqrt{ 9 - 8\\kappa}\\bigr)\\sqrt{L},\\frac{1}{\\sqrt 8}\\bigl(5+\\sqrt{ 9 - 8\\kappa}\\bigr)\\sqrt{L} \\, \\bigr], & \\text{ if } \\kappa < \\frac 98, \\\\\r\n\t\t\t\\bigl\\{(2\\sqrt \\kappa-\\sqrt{\\kappa-1})\\sqrt L \\bigr\\}, & \\text{ if } \\kappa \\ge \\frac 98.\r\n\t\t\\end{cases}\r\n\t\\end{align*}\r\n\tThen, under the assumptions of Theorem~\\ref{theo1}, for every $\\eps>0$ there exist $C, \\bar \\gamma\\ge 0$ such that if $\\sup_{n \\in \\N}\\gamma_n \\le \\bar \\gamma$ it holds that\r\n\t\\begin{align*}\r\n\t\t\\E[\\1_{\\IA_{n-1}} f(X_n)] \\le C \\exp(-(m-\\eps)t_n), \\quad \\text{ for all } n \\in \\N,\r\n\t\\end{align*}\r\n\twhere  \r\n\t$$\r\n\tm = \r\n\t\\begin{cases} \\sqrt{2L},& \\text{ if } \\kappa<\\frac{9}{8},\\\\\r\n\t\t2(\\sqrt{\\kappa}-\\sqrt{\\kappa-1})\\sqrt{L},& \\text{ if } \\kappa \\ge \\frac 98.\r\n\t\\end{cases}\r\n\t$$\r\n\\end{theorem}","units":[{"start":38721,"end":39625,"start_line":532,"end_line":552,"raw_tex":"\\begin{theorem} \\label{theo2}\r\n\tSet $\\kappa = \\frac{C_L}{L}$. Let \r\n\t\\begin{align*}\r\n\t\t\\mu \\in \r\n\t\t\\begin{cases}\r\n\t\t\t\\bigl[\\frac{1}{\\sqrt 8}\\bigl(5-\\sqrt{ 9 - 8\\kappa}\\bigr)\\sqrt{L},\\frac{1}{\\sqrt 8}\\bigl(5+\\sqrt{ 9 - 8\\kappa}\\bigr)\\sqrt{L} \\, \\bigr], & \\text{ if } \\kappa < \\frac 98, \\\\\r\n\t\t\t\\bigl\\{(2\\sqrt \\kappa-\\sqrt{\\kappa-1})\\sqrt L \\bigr\\}, & \\text{ if } \\kappa \\ge \\frac 98.\r\n\t\t\\end{cases}\r\n\t\\end{align*}\r\n\tThen, under the assumptions of Theorem~\\ref{theo1}, for every $\\eps>0$ there exist $C, \\bar \\gamma\\ge 0$ such that if $\\sup_{n \\in \\N}\\gamma_n \\le \\bar \\gamma$ it holds that\r\n\t\\begin{align*}\r\n\t\t\\E[\\1_{\\IA_{n-1}} f(X_n)] \\le C \\exp(-(m-\\eps)t_n), \\quad \\text{ for all } n \\in \\N,\r\n\t\\end{align*}\r\n\twhere  \r\n\t$$\r\n\tm = \r\n\t\\begin{cases} \\sqrt{2L},& \\text{ if } \\kappa<\\frac{9}{8},\\\\\r\n\t\t2(\\sqrt{\\kappa}-\\sqrt{\\kappa-1})\\sqrt{L},& \\text{ if } \\kappa \\ge \\frac 98.\r\n\t\\end{cases}\r\n\t$$\r\n\\end{theorem}","sha256_utf8":"f9af5cca8652ed293ae4a721763f6a7f49269a6dd20986199267e0297dbeadda"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma} \\label{rem:Lipschitz}\r\n\tLet $r>0$, $y \\in \\R^d$ and assume that $\\grad f$ is $C_L$-Lipschitz continuous on $B_r(y)$ and $\\inf_{x \\in B_r(y)}f(x)=f(y)$. Then, for all $x \\in B_{r/2}(y)$ it holds that\r\n\t\\begin{align} \\label{eq:inversePL}\r\n\t\t|\\grad f(x)|^2 \\le 2 C_L (f(x)-f(y)).\r\n\t\\end{align}\r\n\\end{lemma}\n\n\\begin{proof}\r\n\tSince $y$ is a critical point of $f$ we have for all $x \\in B_{r/2}(y)$\r\n\t$$\r\n\t|\\grad f(x)| = |\\grad f(x)-\\grad f(y)| \\le \\frac{C_L r}{2}.\r\n\t$$\r\n\tIf $\\grad f(x)=0$ the statement is obviously true. \r\n\tIf $\\grad f(x) \\neq 0$ consider the function \r\n\t$$\r\n\tg(t) = f\\Bigl(x-t \\frac{\\grad f(x)}{|\\grad f(x)|}\\Bigr).\r\n\t$$\r\n\tNote that for $x \\in B_{r/2}(y)$ and all $t \\in [0, \\frac{|\\grad f(x)|}{C_L}]$ we have $x-t \\frac{\\grad f(x)}{|\\grad f(x)|} \\in B_r(y)$ so that with the Lipschitz continuity of $\\grad f$ and since $y$ is a local minimum\r\n\t\\begin{align*}\r\n\t\tf(y)-f(x) &\\le g\\Bigl(\\frac{|\\grad f(x)|}{C_L}\\Bigr)-g(0) = \\int_0^{\\frac{|\\grad f(x)|}{C_L}} g'(s)  \\, ds \\\\\r\n\t\t& \\le \\frac{|\\grad f(x)|}{C_L} g'(0) +\\frac{|\\grad f(x)|^2}{2C_L}= -\\frac{|\\grad f(x)|^2}{2C_L}.\r\n\t\\end{align*}\r\n\\end{proof}\n\n\\begin{lemma} \\label{lem:Epositive}\r\n\tLet $a, b, C_L>0$ and\r\n\t\\begin{align*}\r\n\t\tE(x,y) = a f(x) + \\langle \\grad f(x), y \\rangle + \\frac{b}{2} |y|^2.\r\n\t\\end{align*}\r\n\tIf $ab\\ge C_L$ then for all $x \\in \\R^d$ satisfying $2C_L f(x)\\ge |\\grad f(x)|^2$ we have $E(x,y)\\ge 0$, for all $y \\in \\R^d$.\r\n\\end{lemma}\n\n\\begin{proof}\r\n\tIf $\\grad f(x)=0$ the statement is trivial. If $\\grad f(x) \\neq 0$, we denote $\\rho=\\frac{|y|}{|\\grad f(x)|}$ and get\r\n\t$$\r\n\tE(x,y) \\ge \\Bigl(\\frac{a}{2C_L}-\\rho+\\frac{b}{2}\\rho^2 \\Bigr) |\\grad f(x)|^2.\r\n\t$$\r\n\tThe quadratic function $\\varphi(\\rho)=\\frac{a}{2C_L}-\\rho+\\frac{b}{2}\\rho^2$ attains its global minimum at $\\rho=\\frac{1}{b}$ and using $ab \\ge C_L$ we deduce that\r\n\t\\begin{align*}\r\n\t\t\\varphi\\Bigl(\\frac 1b\\Bigr)=\\frac{a}{2C_L}-\\frac{1}{2b}\\ge 0.\r\n\t\\end{align*}\r\n\\end{proof}\n\n\\begin{proposition} \\label{prop:SHB}\r\n\tLet $L,a,b>0, \\sigma \\ge 0$. Let $(\\IA_n)_{n \\in \\N_0}$ be a decreasing sequence of events such that, for all $n \\in \\N_0$, $\\IA_n \\in \\cF_n$ and on $\\IA_n$ it holds that\r\n\t\\begin{align} \\label{eq:1293182}\r\n\t\t|\\grad f(X_n)|^2 \\ge 2L f(X_n) \\quad \\text{and} \\quad \\E[|D_{n+1}|^2 | \\cF_n] \\le \\sigma f(X_n).\r\n\t\\end{align}\r\n\tLet $(\\alpha_n)_{n \\in \\N}, (\\beta_n)_{n \\in \\N}, (\\delta_n)_{n \\in \\N}$ and $(\\epsilon_n)_{n \\in \\N}$ be given by $\\eqref{eq:alpha}$. Assume that $(\\beta_n)_{n \\in \\N}$, $(\\alpha_{n+1}-a\\delta_{n+1}+2\\beta_{n+1}L)_{n \\in \\N_0}$, $(\\epsilon_{n+1}-\\frac{b}{2} \\delta_{n+1})_{n \\in \\N_0}$ and $(\\frac{C_L}{2} \\gamma_{n}^2- \\frac{b}{2}\\gamma_{n})_{n \\in \\N_0}$ are sequences of non-positive reals and $(\\delta_n)_{n \\in \\N}$ is a sequence of non-negative reals.\r\n\tMoreover, if $\\P(\\bigcap_{n \\in \\N_0} \\IA_n)<\\P(\\IA_0)$ additionally assume that $ab \\ge C_L$.\r\n\tThen, for all $n \\in \\N$ it holds that\r\n\t\\begin{align}\\begin{split} \\label{eq:238477}\r\n\t\t\t& \\E[\\1_{\\IA_{n-1}} f(X_{n})]\\le \\Bigl(\\prod_{i=1}^{n} (1+a\\gamma_{i})^{-1}\\Bigr) \\\\\r\n\t\t\t& \\qquad \\qquad \\qquad \\Bigl( \\E[\\1_{\\IA_0} f(X_0)] + \\sum_{i=1}^{n}  \\frac{\\gamma_{i}}{1+a\\gamma_{i}} \\Bigl(\\prod_{j=1}^{i} (1+a\\gamma_{j})\\Bigr) \\Bigl(\\prod_{j=1}^{i}\\delta_{j} \\Bigr)\\E[\\1_{\\IA_0}E_0] \\Bigr).\r\n\t\t\\end{split} \r\n\t\\end{align}\t\r\n\\end{proposition}\n\n\\begin{proof}\r\n\tIn a first step, we derive a convergence rate for the expectation of the Lyapunov process $(E_n)_{n \\in \\N_0}$. For this, we consider the time evolution of the three summands in \\eqref{eq:Lyapunov}, separately.\r\n\t\r\n\tFirst, we look at the evolution of $(f(X_{n}))_{n \\in \\N_0}$.\r\n\tLet $x,y \\in \\R^d$ and note that with the Lipschitz-continuity of $\\grad f$ we get\r\n\t\\begin{align}\r\n\t\t\\begin{split}\r\n\t\t\t\\label{eq:Taylorp1}\r\n\t\t\tf(y)&\\le f(x) + \\langle \\grad f(x),y-x\\rangle  + \\frac{C_L}{2} |y-x|^2.\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n\t\r\n\tNow, for $n \\in \\N_0$ we use \\eqref{eq:Taylorp1} with $x=X_{n}$ and $y=X_{n+1}$ and \\eqref{eq:MSGDintro} to get \r\n\t\\begin{align} \\begin{split} \r\n\t\t\t\\label{eq:273856230}\r\n\t\t\t&\\E[\\1_{\\IA_n}f(X_{n+1})]\\le \\E\\Big[\\1_{\\IA_n}\\bigl(f(X_{n})-\\gamma_{n+1}^2 |\\grad f(X_n)|^2 \\\\\r\n\t\t\t&\\qquad \\qquad \\qquad \\qquad \\quad + (\\gamma_{n+1}-\\gamma_{n+1}^2\\mu) \\langle \\grad f(X_{n}),V_n\\rangle + \\frac{C_L}{2} \\gamma_{n+1}^2 \\bigl|V_{n+1}\\bigr|^2 \\bigr) \\Big].\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n\t\r\n\tNext, we control the evolution of $(|V_n|^2)_{n \\in \\N_0}$. Using \\eqref{eq:1293182}, we get\r\n\t\\begin{align} \\begin{split}\r\n\t\t\t\\E[\\1_{\\IA_n}|V_{n+1}|^2]\r\n\t\t\t\\le  \\E[\\1_{\\IA_n}(&(1-2\\gamma_{n+1}\\mu+\\gamma_{n+1}^2\\mu^2)|V_n|^2+ \\gamma_{n+1}^2 |\\grad f(X_n)|^2  \\\\\r\n\t\t\t&  - 2 (\\gamma_{n+1}-\\gamma_{n+1}^2\\mu) \\langle V_n, \\grad f(X_n)\\rangle  + \\gamma_{n+1}^2 \\sigma f(X_n))].\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n\tLastly,\r\n\t\\begin{align} \\begin{split} \\label{eq:1764443}\r\n\t\t\t\\E[\\1_{\\IA_n} &\\langle \\grad f(X_{n+1}), V_{n+1} \\rangle] = \\E[ \\1_{\\IA_n} (\\langle \\grad f(X_{n}), V_{n+1} \\rangle + \\langle \\grad f(X_{n+1})- \\grad f(X_{n}), V_{n+1} \\rangle )] \\\\\r\n\t\t\t&\\le \\E[\\1_{\\IA_n} ((1-\\gamma_{n+1}\\mu) \\langle \\grad f(X_n), V_n \\rangle - \\gamma_{n+1}  |\\grad f(X_n)|^2 + C_L \\gamma_{n+1} |V_{n+1}|^2)].\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n\t\r\n\tCombining the estimates \\eqref{eq:273856230}-\\eqref{eq:1764443}, we obtain\r\n\t\\begin{align*}\r\n\t\t&\\E[\\1_{\\IA_{n}} E_{n+1}] \\\\\r\n\t\t&\\le \\E[\\1_{\\IA_{n}}(\\alpha_{n+1} f(X_n) + \\beta_{n+1} |\\grad f(X_n)|^2 +\\delta_{n+1} \\langle \\grad f(X_n),V_n \\rangle + \\epsilon_{n+1} |V_n|^2)],\r\n\t\\end{align*}\r\n\twhere\r\n\t\\begin{align} \\label{eq:alpha}\r\n\t\t\\begin{split}\r\n\t\t\t\\alpha_{n+1} = &a + \\Bigl(\\frac{b}{2}+C_L \\gamma_{n+1}\\Bigl(1+\\frac{\\gamma_{n+1}a}{2}\\Bigr)\\Bigr)\\gamma_{n+1}^2 \\sigma, \\\\\r\n\t\t\t\\beta_{n+1} = &- \\gamma_{n+1} - a\\gamma_{n+1}^2 + \\Bigl(\\frac{b}{2}+C_L \\gamma_{n+1}\\Bigl(1+\\frac{\\gamma_{n+1}a}{2}\\Bigr)\\Bigr)\\gamma_{n+1}^2,\\\\\r\n\t\t\t\\delta_{n+1} = &1-\\mu\\gamma_{n+1}+ a(\\gamma_{n+1}-\\gamma_{n+1}^2\\mu) \\\\\r\n\t\t\t&- 2\\Bigl(\\frac{b}{2}+C_L \\gamma_{n+1}\\Bigl(1+\\frac{\\gamma_{n+1}a}{2}\\Bigr)\\Bigr) (\\gamma_{n+1}-\\gamma_{n+1}^2\\mu), \\\\\r\n\t\t\t\\epsilon_{n+1} = &\\Bigl(\\frac{b}{2}+C_L \\gamma_{n+1}\\Bigl(1+\\frac{\\gamma_{n+1}a}{2}\\Bigr)\\Bigr) (1-2\\gamma_{n+1}\\mu+\\gamma_{n+1}^2\\mu^2).\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n\tBy definition\r\n\t$\r\n\t\\E[\\1_{\\IA_n} \\langle \\grad f(X_n), V_n \\rangle ] =  \\E[\\1_{\\IA_n}(E_n -a f(X_n)-\\sfrac{b}{2} |V_n|^2)],\r\n\t$\r\n\tso that, using the PL-inequality \\eqref{eq:1293182} and the fact that $(\\beta_n)_{n \\in \\N}$ is non-positive, we get \r\n\t\\begin{align} \\begin{split} \\label{eq:579999663}\r\n\t\t\t&\\E[\\1_{\\IA_{n}}E_{n+1}] \\\\\r\n\t\t\t&\\le \\E[\\1_{\\IA_n}(\\delta_{n+1}E_n + (\\alpha_{n+1}-a\\delta_{n+1}+2\\beta_{n+1}L) f(X_n)  + (\\epsilon_{n+1}-\\frac{b}{2} \\delta_{n+1}) |V_n|^2)].\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n\tWith the assumptions on $(\\alpha_{n+1}-a\\delta_{n+1}+2\\beta_{n+1}L)_{n \\in \\N_0}$ and $(\\epsilon_{n+1}-\\frac{b}{2} \\delta_{n+1})_{n \\in \\N_0}$ we have\r\n\t$\r\n\t\\E[\\1_{\\IA_{n}} E_{n+1}] \\le \\delta_{n+1} \\E[\\1_{\\IA_{n}} E_n].\r\n\t$\r\n\tFor $n\\in \\N$, we use Lemma~\\ref{lem:Epositive} and the monotonicity of $(\\IA_n)_{n \\in \\N_0}$ in order to show that $\\E[\\1_{\\IA_{n}} E_n]\\le \\E[\\1_{\\IA_{n-1}} E_n]$ so that, iteratively, \r\n\t\\begin{align} \\label{eq:2776362562}\r\n\t\t\\E[\\1_{\\IA_{n-1}} E_{n}] \\le \\Bigl( \\prod_{i=1}^{n}\\delta_{i} \\Bigr)\\E[\\1_{\\IA_{0}} E_0].\r\n\t\\end{align}\r\n\tNext, we bound the expectation of $(f(X_n))_{n \\in \\N_0}$ using \\eqref{eq:2776362562}. Analogously to \\eqref{eq:273856230}, we have for all $n \\in \\N_0$ that\r\n\t\\begin{align*}\r\n\t\t\\E[\\1_{\\IA_{n}}f(X_{n+1})]\\le \\E\\Big[\\1_{\\IA_{n}} \\bigl( f(X_{n})+\\gamma_{n+1}\\langle \\grad f(X_{n+1}),V_{n+1}\\rangle + \\frac{C_L}{2} \\gamma_{n+1}^2 \\bigl|V_{n+1}\\bigr|^2\\bigr) \\Big]\r\n\t\\end{align*}\r\n\tso that, by definition of $E_{n+1}$,\r\n\t\\begin{align*} \r\n\t\t&(1+a\\gamma_{n+1})\\E[\\1_{\\IA_{n}}f(X_{n+1})]\\\\\r\n\t\t&\\le \\E\\Big[ \\1_{\\IA_{n}} \\bigl(f(X_{n})+\\gamma_{n+1} E_{n+1} + (\\frac{C_L}{2} \\gamma_{n+1}^2- \\frac{b}{2}\\gamma_{n+1}) \\bigl|V_{n+1}\\bigr|^2 \\bigr)\\Big].\r\n\t\\end{align*}\r\n\tBy assumption, $(\\frac{C_L}{2} \\gamma_{n}^2- \\frac{b}{2}\\gamma_{n})_{n \\in \\N}$ is a sequence of non-positive reals. Therefore, we can neglect the last term in the upper bound above and get\r\n\t\\begin{align} \\label{eq:21378861}\r\n\t\t\\E[\\1_{\\IA_{n}}f(X_{n+1})] \\le (1+a\\gamma_{n+1})^{-1} \\E[\\1_{\\IA_n} f(X_{n})]+\\frac{\\gamma_{n+1}}{1+a\\gamma_{n+1}} \\Bigl( \\prod_{i=1}^{n+1}\\delta_{i} \\Bigr)\\E[\\1_{\\IA_0}E_0].\r\n\t\\end{align}\r\n\tUsing the non-negativity of $f(X_n)$, the monotonicity of $(\\IA_n)_{n \\in \\N_0}$ and \\eqref{eq:21378861}, one can inductively show that, for all $n \\in \\N$,\r\n\t\\begin{align*}\r\n\t\t&\\E[\\1_{\\IA_{n-1}} f(X_{n})] \\\\\r\n\t\t&\\le \\Bigl(\\prod_{i=1}^{n} (1+a\\gamma_{i})^{-1}\\Bigr) \\Bigl( \\E[\\1_{\\IA_0} f(X_0)] + \\sum_{i=1}^{n}  \\frac{\\gamma_{i}}{1+a\\gamma_{i}} \\Bigl(\\prod_{j=1}^{i} (1+a\\gamma_{j})\\Bigr) \\Bigl(\\prod_{j=1}^{i}\\delta_{j} \\Bigr)\\E[\\1_{\\IA_0}E_0] \\Bigr).\r\n\t\\end{align*}\r\n\\end{proof}\n\n\\begin{proposition} \\label{prop:convrate}\r\n\tLet $L, a,b,\\mu > 0$ and $\\sigma \\ge 0$. Let $(\\IA_n)_{n \\in \\N_0}$ be a decreasing sequence of events such that, for all $n \\in \\N_0$, $\\IA_n \\in \\cF_n$ and on $\\IA_n$ it holds that\r\n\t\\begin{align*} \r\n\t\t|\\grad f(X_n)|^2 \\ge 2L f(X_n)  \\quad \\text{and} \\quad \\E[|D_{n+1}|^2 | \\cF_n] \\le \\sigma f(X_n).\r\n\t\\end{align*} \r\n\tAssume that\r\n\t\\begin{align} \\label{eq:238877}\r\n\t\t\\mu-a+b>0 \\quad , \\quad a\\mu-a^2+ab-2L <0 \\quad \\text{and} \\quad C_L -\\frac{b}{2}(\\mu+a-b)<0.\r\n\t\\end{align}\r\n\tIf $\\P(\\bigcap_{n \\in \\N_0} \\IA_n)< \\P(\\IA_0)$ we additionally assume that $ab \\ge C_L$.\r\n\tThen, for every $0<\\eps<m:= \\min (a, \\mu-a+b)$ there exist constants $C, \\bar \\gamma \\ge 0$ such that if $\\sup_{n \\in \\N}\\gamma_n \\le \\bar \\gamma$ it holds that \r\n\t\\begin{enumerate}\r\n\t\t\\item[(i)] \r\n\t\t$$\r\n\t\t\\max (\\E[\\1_{\\IA_{n-1}} f(X_n)], \t\\E[\\1_{\\IA_{n-1}}|V_n|^2] ) \\le C \\exp(- (m-\\eps) t_n)\r\n\t\t$$\r\n\t\tfor all $n \\in \\N$, where $t_n = \\sum_{i=1}^n \\gamma_i$.\r\n\t\t\\item[(ii)] Let $m'<m-\\eps$ and assume that $ \\sum_{i=0}^\\infty \\exp((m'-(m-\\eps)) t_i)<\\infty$. Then $\\exp(m' t_n) f(X_n) \\to 0$ almost surely on the event $\\IA_\\infty= \\bigcap_{n \\in \\N_0} \\IA_n$.\r\n\t\t\\item[(iii)] The process $(X_n)_{n \\in \\N_0}$ converges almost surely on $\\IA_\\infty$.\r\n\t\\end{enumerate}\r\n\\end{proposition}\n\n\\begin{proof}\r\n\t(i): \r\n\tFirst, note that, for all $x\\in \\R$, $1+x\\le \\exp(x)$ and $(1+x)^{-1}=e^{-x+o(x)}$. Thus, for every $\\eps'>0$ there exists a $\\bar \\gamma'>0$ such that if $\\max_{i=1, \\dots, n} \\gamma_i \\le \\bar \\gamma'$ we have\r\n\t$$\r\n\t\\prod_{i=1}^{n} (1+a\\gamma_{i})^{-1} \\le \\Bigl(\\prod_{i=1}^{n} \\exp((-a+\\eps')\\gamma_i)\\Bigr) = \\exp((-a+\\eps')t_n ).\r\n\t$$\r\n\tMoreover, for $(\\delta_n)_{n \\in \\N}$ given in \\eqref{eq:alpha} we have $\\delta_n = 1-\\gamma_n(\\mu-a+b)+o(\\gamma_n)$ so that, for all $\\eps''>0$, there exists a $\\bar \\gamma ''>0$ such that if $\\max_{i=1, \\dots, n} \\gamma_i \\le \\bar \\gamma''$ we have $\\delta_i\\ge 0$ for all $i=1, \\dots, n$ and\r\n\t$$\r\n\t\\prod_{i=1}^{n}\\delta_{i} \\le \\exp(-(\\mu-a+b-\\eps'')t_n).\r\n\t$$\r\n\tNote that $\\beta_n = -\\gamma_n + o(\\gamma_n)$, $\\alpha_n-a\\delta_n+2\\beta_n L =  \\gamma_n (a\\mu-a^2+ab-2 L)+o(\\gamma_n)$, \r\n\t$$\r\n\t\\epsilon_n-\\frac b2 \\delta_n =  \\gamma_n \\Bigl(C_L-\\frac{b}{2}(\\mu+a-b)\\Bigr) +o(\\gamma_n) \\;  \\text{ and } \\; \\sfrac {C_L}{2}\\gamma^2-\\sfrac{b}{2}\\gamma=-\\sfrac{b}{2}\\gamma+o(\\gamma),\r\n\t$$\r\n\tand using assumption \\eqref{eq:238877} we can choose $\\bar \\gamma \\le \\min(\\bar \\gamma',\\bar \\gamma'')$ sufficiently small such that if $\\sup_{n \\in \\N}\\gamma_n \\le \\bar \\gamma$ all of the above terms are strictly negative. Then, using Proposition~\\ref{prop:SHB} we get that for all $n \\in \\N$\r\n\t\\begin{align*}\r\n\t\t\\E[\\1_{\\IA_{n-1}} f(X_{n})] \r\n\t\t\\le \\exp((&-a+\\eps'+\\eps'')t_n )  \\\\\r\n\t\t& \\Bigl( \\E[ \\1_{\\IA_0}f(X_0)]+ \\sum_{i=1}^{n}  \\gamma_{i}  \\exp(-(\\mu-2a+b)t_i) \\E[\\1_{\\IA_0}E_0] \\Bigr).\r\n\t\\end{align*}\r\n\tIf $a< \\mu-a+b$, the function $t \\mapsto \\exp(-(\\mu-2a+b)t)$ is monotonously decreasing and we get\r\n\t\\begin{align*}\r\n\t\t\\sum_{i=1}^{n}  \\gamma_{i}  \\exp(-(\\mu-2a+b)t_i) &\\le \\int_{0}^{t_n} \\exp(-(\\mu-2a+b)t) \\, dt.\r\n\t\\end{align*}\r\n\tThus,\r\n\t$$\r\n\t\\E[\\1_{\\IA_{n-1}} f(X_{n})] \\le\\exp((-a+\\eps'+\\eps'')t_n ) \\Bigl( \\E[\\1_{\\IA_0} f(X_0)] + \\frac{\\E[\\1_{\\IA_0}E_0]}{\\mu-2a+b}  \\Bigr).\r\n\t$$\r\n\tFor $a>\\mu-a+b$,  the function $t \\mapsto \\exp(-(\\mu-2a+b)t)$ is monotonously increasing and we get\r\n\t\\begin{align*}\r\n\t\t\\sum_{i=1}^{n}  &\\gamma_{i}  \\exp(-(\\mu-2a+b)t_i) \\le \\int_{0}^{t_n} \\exp(-(\\mu-2a+b)(t+\\bar \\gamma)) \\, dt.\r\n\t\\end{align*}\r\n\tThus,\r\n\t\\begin{align*}\r\n\t\t\\E[\\1_{\\IA_{n-1}} f(X_{n})] \\le&\\exp((-(\\mu-a+b)+\\eps'+\\eps'')t_n ) \\Bigl( \\E[\\1_{\\IA_0} f(X_0)] \\\\\r\n\t\t& \\qquad \\qquad \\qquad  +\\frac{\\E[\\1_{\\IA_0}E_0]}{2a-\\mu-b} \\bigl(\\exp(-(\\mu-2a+b)\\bar \\gamma)  \\Bigr).\r\n\t\\end{align*}\r\n\tLastly, for $a=\\mu-a+b$ we get \r\n\t$\r\n\t\\sum_{i=1}^{n}  \\gamma_{i}  \\exp(-(\\mu-2a+b)t_i)=t_n\r\n\t$\r\n\tand, thus,\r\n\t$$\r\n\t\\E[\\1_{\\IA_{n-1}}f(X_n)] \\le \\exp((-a+\\eps'+\\eps'')t_n ) \\Bigl( \\E[ \\1_{\\IA_0}f(X_0)] + t_n\\E[\\1_{\\IA_0}E_0] \\Bigr).\r\n\t$$\r\n\tNote that $\\exp(-\\eps''' t)t \\to 0$ for all $\\eps'''>0$. Therefore, there exists a constant $C'>0$ such that\r\n\t$$\r\n\t\\E[\\1_{\\IA_{n-1}}f(X_{n})] \\le \\exp((-a+\\eps'+\\eps''+\\eps''')t_n ) \\Bigl( \\E[\\1_{\\IA_0} f(X_0)] + C'\\E[\\1_{\\IA_0}E_0] \\Bigr).\r\n\t$$\r\n\tThe proof of the first assertion follows by choosing $\\eps'=\\eps''= \\eps''' =\\frac 13 \\eps$.\r\n\t\r\n\tFor the second assertion, note that, by Lemma~\\ref{rem:Lipschitz}, we have $|\\grad f(x)|^2 \\le 2 C_L f(x)$ for all $x \\in \\R^d$\r\n\tand, using that $f(X_n)\\ge 0$, we get by the Cauchy-Schwarz inequality\r\n\t\\begin{align*}\r\n\t\t\\frac b2 |V_n|^2 &\\le E_n - \\langle \\grad f(X_n), V_n \\rangle \\le E_n + |\\grad f(X_n)| \\, |V_n|,\r\n\t\\end{align*}\r\n\twhere $E_n$ is defined by \\eqref{eq:Lyapunov}.\r\n\tThus, using Young's inequality, \r\n\t\\begin{align} \\label{eq:Yconv2}\r\n\t\t\\frac b2 \\E\\bigl[\\1_{\\IA_{n-1}}  |V_n|^2\\bigr]\r\n\t\t&\\le \\E[\\1_{\\IA_{n-1}} E_n] + \\frac 1b \\E[\\1_{\\IA_{n-1}} |\\grad f(X_n)|^2] + \\frac b4\\E[ \\1_{\\IA_{n-1}} |V_n|^2] \r\n\t\\end{align}\r\n\tand, with the bound for $\\E[\\1_{\\IA_{n-1}} f(X_n)]$ and \\eqref{eq:2776362562}, we get a constant $C\\ge 0$ such that \r\n\t\\begin{align} \\label{eq:Yconv} \r\n\t\t\\E[\\1_{\\IA_{n-1}}|V_n|^2]^{1/2} \\le C \\exp\\Bigl(-\\frac 12 (m -\\eps)t_n\\Bigr).\r\n\t\\end{align}\r\n\t\r\n\t(ii): Let $m'<m-\\eps$. For $\\eps' >0$ and $n \\in \\N$ consider the set $\\IB_n = \\IA_\\infty \\cap \\{\\sup_{i \\ge n} \\exp(m' t_i) f(X_i) \\ge \\eps' \\}$.\r\n\tWith the Markov inequality and (i) there exists a $C>0$ such that\r\n\t\\begin{align*}\r\n\t\t\\P(\\IB_n) &\\le \\sum_{i=n}^\\infty \\P\\bigl(\\IA_{i-1} \\cap \\bigl\\{\\exp(m' t_i) f(X_i) \\ge \\frac{\\eps'}{2}\\bigr\\}\\bigr) \\\\\r\n\t\t& \\le \\sum_{i=n}^\\infty \\frac{2\\exp(m' t_i)}{\\eps'} \\E[\\1_{\\IA_{i-1}} f(X_i)] \\le \\frac{C}{\\eps'} \\sum_{i=n}^\\infty \\exp((m'-(m-\\eps)) t_i) \\overset{n \\to \\infty}{\\longrightarrow} 0.\r\n\t\\end{align*}\r\n\tWith\r\n\t$$\r\n\t\\P\\Bigl(\\IA_\\infty \\cap \\big\\{\\limsup_{n \\to \\infty} \\exp(m' t_n)f(X_n) \\ge \\eps'\\big\\}\\Bigr) \\le \\P\\Bigl(\\bigcap_{n \\in \\N} B_n\\Bigr) =0\r\n\t$$\r\n\twe get $\\exp(m' t_n)f(X_n)\\to 0$ almost surely on $\\IA_\\infty$.\r\n\t\r\n\t(iii):\r\n\tWe consider the event $\\IA_\\infty$ and bound the distance that the process $(X_n)_{n \\in \\N_0}$ travels. Since $\\eps < m$ the mapping $t  \\mapsto \\exp(-\\frac 12 (m -\\eps)t)$ is monotonously decreasing. Thus, using \\eqref{eq:Yconv} we get\r\n\t\\begin{align*}\r\n\t\t\\E\\Bigl[\\1_{\\IA_\\infty} \\sum_{i=1}^\\infty |X_i-X_{i-1}|\\Bigr] &\\le   \\sum_{i=1}^\\infty \\gamma_i \\E[\\1_{\\IA_{i-1}}  |V_i|^2]^{1/2} \\le C \\sum_{i=1}^\\infty \\gamma_i  \\exp\\Bigl(-\\frac 12 (m -\\eps)t_i\\Bigr)\\\\\r\n\t\t&\\le C \\int_0^\\infty \\exp\\Bigl(-\\frac 12 (m -\\eps)t\\Bigr) \\, dt < \\infty,\r\n\t\\end{align*}\r\n\twhich implies that $\\sum_{i=1}^\\infty |X_i-X_{i-1}|$ is almost surely finite, on $\\IA_\\infty$. Thus, $(X_n)_{n \\in \\N_0}$ almost surely converges on $\\IA_{\\infty}$.\r\n\\end{proof}\n\n\\begin{lemma} \\label{lem:feasible1}\r\n\tFor all $\\mu >0$ there exist $a,b >0$ such that $ab \\ge C_L$ and \\eqref{eq:238877} is satisfied.\r\n\\end{lemma}\n\n\\begin{proof}\r\n\tLet $\\eps,b>0$ and choose $a=b+2\\frac{C_L}{b}-\\mu+\\eps$. Note that $a>0$ iff $\\mu < b+2\\frac {C_L}{b}+\\eps$. Moreover, $\\mu-a+b=2(\\mu-\\frac{C_L}{b})-\\eps$ is positive iff $\\mu > \\frac{C_L}{b}+\\frac{\\eps}{2}$. Now, \r\n\t$\r\n\tC_L-\\frac b2(\\mu+a-b)=-\\frac{b}{2}\\eps < 0\r\n\t$\r\n\tand we have \r\n\t$$\r\n\ta\\mu-a^2+ab-2L = -2\\mu^2+2\\mu\\Bigl(b+\\frac{3C_L}{b}+\\frac{3\\eps}{2}\\Bigr)-2C_L-2L-4\\frac{C_L^2}{b^2}-2\\eps\\Bigr(\\frac b2 + 2\\frac{C_L}{b}+\\frac{\\eps}{2}\\Bigl).\r\n\t$$\r\n\tThe latter term is a quadratic function in $\\mu$ that is negative outside of the two roots. Therefore, $a\\mu-a^2+ab-2L <0$ iff $\\mu \\notin [\\mu_-^{\\eps,b},\\mu_+^{\\eps,b}]$, where\r\n\t\\begin{align} \\label{eq:mu}\r\n\t\t\\mu_{\\pm}^{\\eps,b} = \\frac 12 \\Bigl( b+\\frac{3C_L}{b}+\\frac{3\\eps}{2} \\pm \\sqrt{\\Bigl(b+\\frac{C_L}{b}+\\frac{\\eps}{2}\\Bigr)^2-4L} \\,  \\Bigr).\r\n\t\\end{align}\r\n\tNote that $(b+\\frac{C_L}{b})^2\\ge 4C_L$, for all $b >0$, and $C_L\\ge L$ so that $\\mu_{\\pm}^{\\eps,b}$ is well-defined.\r\n\tMoreover, $\\mu_-^{\\eps,b}>\\frac{C_L}{b}+\\frac{\\eps}{2}$ and $\\mu_+^{\\eps,b}<b+2\\frac{C_L}{b}+\\eps$. The additional assumption $ab\\ge C_L$ is satisfied iff $\\mu\\le b+ \\frac{C_L}{b}+\\eps$. Therefore, the set of friction parameters $\\mu$ that satisfy \\eqref{eq:238877} for the given pair $(a,b)$ is equal to\r\n\t$\r\n\t(\\frac{C_L}{b}+\\frac{\\eps}{2},\\mu_-^{\\eps,b}) \\cup (\\mu_+^{\\eps,b}, b+2\\frac{C_L}{b}+\\eps)\r\n\t$\r\n\tand the set of friction parameters $\\mu$ that satisfy both \\eqref{eq:238877} and $ab \\ge C_L$ for the given pair $(a,b)$ contains the interval\r\n\t$\r\n\t(\\sfrac{C_L}{b}+\\sfrac{\\eps}{2}, \\mu_-^{\\eps,b} \\wedge (b+\\sfrac{C_L}{b}+\\eps)).\r\n\t$\r\n\tFor all $b>0$, the latter interval is non-empty, the upper and lower limits are continuous in $b$ and the lower limit satisfies \r\n\t$$\r\n\t\\frac{C_L}{b}+\\frac{\\eps}{2} \\overset{b \\to \\infty}{\\longrightarrow} \\frac{\\eps}{2} \\quad \\text{ and } \\quad \\frac{C_L}{b}+\\frac{\\eps}{2} \\overset{b \\to 0}{\\longrightarrow} \\infty.\r\n\t$$\r\n\tBy letting $\\eps \\to 0$ we showed that for every $\\mu>0$ there exists a pair $(a,b)$ such that \\eqref{eq:238877} is satisfied and $ab \\ge C_L$.\r\n\\end{proof}\n\n\\begin{lemma} \\label{lem:49378}\r\n\tLet $L, b, \\eps > 0$ and $\\sigma \\ge 0$. Let $(\\IA_n)_{n \\in \\N_0}$ be a decreasing sequence of events such that, for all $n \\in \\N_0$, $\\IA_n \\in \\cF_n$ and on $\\IA_n$ it holds that\r\n\t\\begin{align*} \r\n\t\t|\\grad f(X_n)|^2 \\ge 2L f(X_n) \\quad \\text{and} \\quad \\E[|D_{n+1}|^2 | \\cF_n] \\le \\sigma f(X_n).\r\n\t\\end{align*} \r\n\tLet \r\n\t$\r\n\t\\mu \\in (\\frac{C_L}{b}+\\frac{\\eps}{2}, \\mu_-^{\\eps,b}) \\cup (\\mu_+^{\\eps,b}, b+2\\frac{C_L}{b}+\\eps),\r\n\t$\r\n\twhere $\\mu_{\\pm}^{\\eps,b}$ is defined by \\eqref{eq:mu}.\r\n\tIf $\\P(\\bigcap_{n \\in \\N_0}\\IA_n)<\\P(\\IA_0)$ additionally assume that $\\mu \\le b+\\frac{C_L}{b}+\\eps$. Then, there exist $C, \\bar \\gamma>0$ such that if $\\sup_{n \\in \\N}\\gamma_n \\le \\bar \\gamma$ we have\r\n\t$$\r\n\t\\E[\\1_{\\IA_{n-1}} f(X_n)] \\le C \\exp(-m(\\eps,b,\\mu) t_n),\r\n\t$$\r\n\twhere\r\n\t\\begin{align*} \r\n\t\tm(\\eps,b,\\mu) = \\begin{cases} \r\n\t\t\t2(\\mu-\\frac{C_L}{b}-\\eps), & \\text{ if } \\mu < \\frac{1}{3} (b+4\\frac{C_L}{b}+2\\eps) \\\\\r\n\t\t\tb+2\\frac{C_L}{b}-\\mu, & \\text{ if } \\mu \\ge \\frac{1}{3} (b+4\\frac{C_L}{b}+2\\eps).\r\n\t\t\\end{cases}\r\n\t\\end{align*}\r\n\\end{lemma}\n\n\\begin{proof}\r\n\tLet $\\eps, b>0$ and $\\mu$ as in the assumptions of the lemma and set $a=b+2\\frac{C_L}{b}-\\mu+\\eps$. Recall that in the proof of Lemma~\\ref{lem:feasible1} we showed that this choice of parameters satisfies \\eqref{eq:238877} and if $\\P(\\bigcap_{n \\in \\N_0}\\IA_n)<\\P(\\IA_0)$, additionally, $ab \\ge C_L$. Hence, we can apply Proposition~\\ref{prop:convrate} and deduce that there exist constants $C, \\bar \\gamma >0$ such that, if $\\sup_{n \\in \\N}\\gamma_n \\le \\bar \\gamma$, we have\r\n\t$$\r\n\t\\E[\\1_{\\IA_{n-1}}f(X_n)] \\le C \\exp(-m(\\eps,b,r) t_n),\r\n\t$$\r\n\twhere\r\n\t$$\r\n\tm(\\eps,b,\\mu)= \\min(a,\\mu-a+b)-\\eps= \\min\\Bigl(b+2\\frac{C_L}{b}-\\mu,2\\Bigl(\\mu-\\frac{C_L}{b}-\\eps\\Bigr)\\Bigr).\r\n\t$$\r\n\tThe evaluation of the minimum is straight-forward.\r\n\\end{proof}\n\n\\begin{proof} [Proof of Theorem~\\ref{theo2}]\r\n\tWe maximize the convergence rate derived in Lemma~\\ref{lem:49378} over all admissible parameters $\\mu$ and $b$.\r\n\tFirst, assume that $\\kappa=\\frac{C_L}{L}<\\frac 98$. Then, we have for all sufficiently small $\\eps>0$ that $\\kappa^\\eps := \\bigl(\\frac{\\sqrt {C_L}}{\\sqrt{L}}+\\frac{\\eps}{4\\sqrt{L}}\\bigr)^2<\\frac{9}{8}$ so that Set \\begin{align*}\r\n\t\tb_{\\pm}^\\eps := \\frac{3}{\\sqrt 8} \\sqrt L -\\frac{\\eps}{4} \\pm \\sqrt{\\Bigl( \\frac{3}{\\sqrt 8} \\sqrt L -\\frac{\\eps}{4} \\Bigr)^2-C_L}\r\n\t\\end{align*}\r\n\tis well-defined.\r\n\tFor $b \\in (b_-^\\eps, b_+^\\eps)$ we have $\\frac{1}{3} (b+4\\frac{C_L}{b}+2\\eps)<\\mu_-^{\\eps, b}$, such that with Lemma~\\ref{lem:49378} we get for $\\mu \\in (\\frac{C_L}{b}+\\frac{\\eps}{2},\\frac 13(b+4\\frac{C_L}{b}+2\\eps))$ the convergence rate $m(\\eps, b ,\\mu)=2\\bigl( \\mu-\\frac{C_L}{b} -\\eps\\bigr)$. Note that\r\n\t$$\r\n\t\\frac{d}{d\\eps}\\big|_{\\eps=0} \\frac 13 \\Bigl(b_+^\\eps-\\eps^2+4\\frac{C_L}{b_+^\\eps-\\eps^2}+2\\eps\\Bigr) =\\frac 14 + \\frac{3 \\sqrt L}{4 \\sqrt{9L-8C_L}}>0.\r\n\t$$ \r\n\tTherefore, for sufficiently small $\\eps$, we have \r\n\t$$\r\n\t\\mu^*_1 := \\frac{1}{3} \\Bigl(b_+^0+4\\frac{C_L}{b_+^0}\\Bigr) = \\frac{1}{\\sqrt 8}(5-\\sqrt{ 9 - 8\\kappa})\\sqrt{L}<\\frac 13 \\Bigl(b_+^\\eps-\\eps^2+4\\frac{C_L}{b_+^\\eps-\\eps^2}+2\\eps\\Bigr)\r\n\t$$\r\n\tand we\r\n\tget by continuity that\r\n\t\\begin{align*}\r\n\t\tm(\\eps, b_+^\\eps-\\eps^2,\\mu^*_1) &\\overset{\\eps \\to 0}{\\longrightarrow} \\frac 23 \\Bigl(b_+^0+\\frac{C_L}{b_+^0}\\Bigr) = \\frac 23 \\frac{(b_+^0)^2+C_L}{b_+^0} = \\sqrt {2L}.\r\n\t\\end{align*}\r\n\tMoreover, note that $b_+^0$ satisfies $\\frac 13 (b_+^0+4\\frac{C_L}{b_+^0})<b_+^0+\\frac{C_L}{b_+^0}$ since \r\n\t$$\r\n\t\\frac{C_L}{b_+^0} = \\frac {1}{\\sqrt 8} (3 \\sqrt L -\\sqrt{9L-8C_L})<\\frac{1}{\\sqrt 2}(3\\sqrt L +\\sqrt{9L-8 C_L})=2b_+^0.\r\n\t$$\r\n\tThus, $b_+^0 (b_+^0+2\\frac{C_L}{b_+^0}-\\mu^*_1) > C_L$ and, for sufficiently small $\\eps>0$, the parameters $b= b_+^\\eps-\\eps^2$ and $a=b_+^\\eps-\\eps^2+2\\frac{C_L}{b_+^\\eps-\\eps^2}-\\mu^*_1+\\eps$ satisfy $ab\\ge C_L$.\r\n\t\r\n\tAnalogously, we get the convergence rate $m(\\eps,b,\\mu)=b+2\\frac{C_L}{b}-\\mu$ if $b \\in (b_-^\\eps,b_+^\\eps)$ and $\\mu \\in (\\frac 13(b+4\\frac{C_L}{b}+2\\eps),\\mu_-^{\\eps,b})$. Note that, since $\\kappa < \\frac 98$ we have\r\n\t$$\r\n\t\\frac{d}{d\\eps}\\big|_{\\eps=0} \\frac 13 \\Bigl(b_-^\\eps+\\eps^2+4\\frac{C_L}{b_-^\\eps+\\eps^2}+2\\eps\\Bigr) = \\frac 14 - \\frac{3 \\sqrt{L}}{4\\sqrt{9L-8C_L}}<0.\r\n\t$$\r\n\tTherefore, for sufficiently small $\\eps$, we have\r\n\t$$\r\n\t\\mu^*_2 := \\frac{1}{3} \\Bigl(b_-^0+4\\frac{C_L}{b_-^0}\\Bigr) = \\frac{1}{\\sqrt 8}\\bigl(5+\\sqrt{ 9 - 8\\kappa}\\bigr)\\sqrt{L} > \\frac 13\\Bigl(b_-^\\eps+\\eps^2+4\\frac{C_L}{b_-^\\eps+\\eps^2}+2\\eps\\Bigr)\r\n\t$$\r\n\tand we get by continuity that\r\n\t\\begin{align*}\r\n\t\tm(\\eps, b_-^\\eps+\\eps^2,\\mu^*_2) &\\overset{\\eps \\to 0}{\\longrightarrow} b_-^0+2\\frac{C_L}{b_-^0}-\\mu^* = \\frac 23 \\frac{(b_-^0)^2+C_L}{b_-^0}  = \\sqrt {2L}.\r\n\t\\end{align*}\r\n\tIf $L<C_L$, $b_-^0$ satisfies $\\frac 13 (b_-^0+4\\frac{C_L}{b_-^0})<b_-^0+\\frac{C_L}{b_-^0}$ since \r\n\t$$\r\n\t\\frac{C_L}{b_-^0} = \\frac {1}{\\sqrt 8} (3 \\sqrt L +\\sqrt{9L-8C_L})<\\frac{1}{\\sqrt 2}(3\\sqrt L -\\sqrt{9L-8 C_L})=2b_-^0.\r\n\t$$\r\n\tThus, for sufficiently small $\\eps>0$, $b=b_-^\\eps+\\eps^2$ and $a=b_-^\\eps+\\eps^2+2\\frac{C_L}{b_-^\\eps+\\eps^2}-\\mu^*_2+\\eps$ satisfy $ab\\ge C_L$.\r\n\tIf $L=C_L$, we have $\\mu^*_2=b_-^0+\\frac{C_L}{b_-^0}$. Using\r\n\t$\r\n\t\\frac{d}{d\\eps}\\big|_{\\eps=0} \\Bigl( b_-^\\eps+\\frac{C_L}{b_-^\\eps}+\\eps\\Bigr) = \\frac 12,\r\n\t$\r\n\twe get $\\mu^*_2 \\le b_-^\\eps +\\eps^2 + \\frac{C_L}{b_-^\\eps+\\eps^2}+\\eps$ for sufficiently small $\\eps>0$ and, thus, $ab\\ge C_L$ for the parameters $b=b_-^\\eps+\\eps^2$ and $a=b_-^\\eps+\\eps^2+2\\frac{C_L}{b_-^\\eps+\\eps^2}-\\mu^*_2+\\eps$.\r\n\tLastly, note that without loss of generality we can increase the Lipschitz constant $C_L$ as long as $\\kappa<\\frac 98$. We thus get that all friction parameters\r\n\t$$\r\n\t\\mu \\in \\Bigl[\\frac{1}{\\sqrt 8}\\bigl(5-\\sqrt{ 9 - 8\\kappa}\\bigr)\\sqrt{L}, \\frac{1}{\\sqrt 8}\\bigl(5+\\sqrt{ 9 - 8\\kappa}\\bigr)\\sqrt{L}  \\Bigr] \\Big\\backslash\\Bigl\\{\\frac{5}{\\sqrt 8}\\sqrt{L}\\Bigr\\}\r\n\t$$\r\n\tgive an optimal convergence rate of $\\sqrt{2L}-\\eps$. The case $\\mu=\\frac{5}{\\sqrt 8}\\sqrt{L}$ corresponds to $\\kappa=\\frac 98$ and will be treated below.\r\n\t\r\n\tNext, assume that $\\kappa \\ge \\frac 98$ which implies that, for all $\\eps > 0$, we have $\\kappa^\\eps \\ge \\frac{9}{8}$. \r\n\tUsing Lemma~\\ref{lem:49378}, we get the convergence rate $m(\\eps,b,\\mu)=2(\\mu-\\frac{C_L}{b}-\\eps)$ if $\\mu \\in (\\frac{C_L}{b}+\\frac{\\eps}{2}, \\mu_-^{\\eps,b})$.\t\r\n\tFirst, note that if $\\mu_-^{0,b} \\in (\\frac{C_L}{b}+\\frac{\\eps}{2}, \\mu_-^{\\eps,b})$ we have\r\n\t$$\r\n\tm(\\eps, b, \\mu_-^{0,b})=  b+\\frac{C_L}{b} - \\sqrt{\\Bigl(b+\\frac{C_L}{b}\\Bigr)^2-4L}-2\\eps.\r\n\t$$\r\n\tMoreover, $m(\\eps, b,\\mu_-^{0,b}) \\overset{b \\to 0}{\\longrightarrow}-2\\eps$ and $m(\\eps, b,\\mu_-^0) \\overset{b \\to \\infty}{\\longrightarrow}-2\\eps$ as well as\r\n\t$$\r\n\t\\frac{d}{db} m(\\eps,b,\\mu_-^{0,b}) = \\Bigl( 1-\\frac{C_L}{b^2} \\Bigr) \\Bigl( 1-\\frac{b+C_L/b}{\\sqrt{(b+C_L/b)^2-4L}} \\Bigr),\r\n\t$$\r\n\tso that\t\r\n\t$\\frac{d}{db}m(\\eps,b,\\mu_-^{0,b})>0$ for all $b< \\sqrt{C_L}$ and $\\frac{d}{db}m(\\eps,b,\\mu_-^{0,b})<0$ for all $b> \\sqrt{C_L}$. Therefore, the maximal value for $m(\\eps,b,\\mu_-^{0,b})$ is attained at $b^*=\\sqrt{C_L}$, where \r\n\t$$\r\n\t\\mu^*_3= \\mu_-^{0,b^*} = 2\\sqrt{C_L}-\\sqrt{C_L-L} = (2\\sqrt \\kappa-\\sqrt{\\kappa-1})\\sqrt L\r\n\t$$\r\n\tand \r\n\t$$\r\n\tm(\\eps, b^*,\\mu^*_3)=2(\\sqrt{C_L}-\\sqrt{C_L-L}-\\eps) \\overset{\\eps \\to 0}{\\longrightarrow} 2(\\sqrt{\\kappa}-\\sqrt{\\kappa-1})\\sqrt L.\r\n\t$$\r\n\tNow, consider \r\n\t\\begin{align*}\r\n\t\t\\mu_{-}^{\\eps,b^*} &= \\frac 12 \\Bigl( 4 \\sqrt{C_L}+\\frac{3\\eps}{2} - \\sqrt{\\bigl(2 \\sqrt{C_L}+\\frac{\\eps}{2}\\bigr)^2-4L} \\,  \\Bigr)\r\n\t\\end{align*}\r\n\tas a function of $\\eps$. Note that for all $\\eps\\ge 0$ we have $\r\n\t\\frac{L}{C_L} \\le \\frac 29 \\Bigl( 2+\\frac{\\eps}{2\\sqrt{C_L}} \\Bigr)^2,\r\n\t$ and, thus,\r\n\t\\begin{align*}\r\n\t\t\\frac{d}{d\\eps} \\mu_{-}^{\\eps,b^*} &= \\frac 12 \\Bigl( \\frac 32 - \\frac 12 \\Bigl(\\bigl(2 \\sqrt{C_L}+\\frac{\\eps}{2}\\bigr)^2-4L\\Bigr)^{-1/2}  \\bigl(2 \\sqrt{C_L}+\\frac{\\eps}{2}\\bigr) \\Bigr) \\ge 0\r\n\t\\end{align*}\r\n\tTherefore, for sufficiently small $\\eps >0$ we have $\\mu^*_3 \\in (\\frac{C_L}{b^*}+\\frac{\\eps}{2}, \\mu_-^{\\eps,b^*})$.\r\n\tMoreover, note that\r\n\t$\r\n\t\\mu^* _3 < 2 \\sqrt{C_L} = b^* + \\frac{C_L}{b^*}\r\n\t$\r\n\tsuch that the parameters $b^*$ and $a=b^*+2\\frac{C_L}{b^*}-\\mu+\\eps$ satisfy $ab^*\\ge C_L$. Finally, for $\\kappa = \\frac 98$ we get $2(\\sqrt{\\kappa}-\\sqrt{\\kappa-1})\\sqrt L= \\sqrt{2L}$ and $\\mu^*_3=\\frac{5}{\\sqrt 8}\\sqrt L$.\r\n\\end{proof}","units":[{"start":33667,"end":33984,"start_line":457,"end_line":462,"raw_tex":"\\begin{lemma} \\label{rem:Lipschitz}\r\n\tLet $r>0$, $y \\in \\R^d$ and assume that $\\grad f$ is $C_L$-Lipschitz continuous on $B_r(y)$ and $\\inf_{x \\in B_r(y)}f(x)=f(y)$. Then, for all $x \\in B_{r/2}(y)$ it holds that\r\n\t\\begin{align} \\label{eq:inversePL}\r\n\t\t|\\grad f(x)|^2 \\le 2 C_L (f(x)-f(y)).\r\n\t\\end{align}\r\n\\end{lemma}","sha256_utf8":"fd1f3e7394a69ba188e3d9bbe9724741a79213317219580f24e27fad8d53fa7a"},{"start":33988,"end":34798,"start_line":464,"end_line":479,"raw_tex":"\\begin{proof}\r\n\tSince $y$ is a critical point of $f$ we have for all $x \\in B_{r/2}(y)$\r\n\t$$\r\n\t|\\grad f(x)| = |\\grad f(x)-\\grad f(y)| \\le \\frac{C_L r}{2}.\r\n\t$$\r\n\tIf $\\grad f(x)=0$ the statement is obviously true. \r\n\tIf $\\grad f(x) \\neq 0$ consider the function \r\n\t$$\r\n\tg(t) = f\\Bigl(x-t \\frac{\\grad f(x)}{|\\grad f(x)|}\\Bigr).\r\n\t$$\r\n\tNote that for $x \\in B_{r/2}(y)$ and all $t \\in [0, \\frac{|\\grad f(x)|}{C_L}]$ we have $x-t \\frac{\\grad f(x)}{|\\grad f(x)|} \\in B_r(y)$ so that with the Lipschitz continuity of $\\grad f$ and since $y$ is a local minimum\r\n\t\\begin{align*}\r\n\t\tf(y)-f(x) &\\le g\\Bigl(\\frac{|\\grad f(x)|}{C_L}\\Bigr)-g(0) = \\int_0^{\\frac{|\\grad f(x)|}{C_L}} g'(s)  \\, ds \\\\\r\n\t\t& \\le \\frac{|\\grad f(x)|}{C_L} g'(0) +\\frac{|\\grad f(x)|^2}{2C_L}= -\\frac{|\\grad f(x)|^2}{2C_L}.\r\n\t\\end{align*}\r\n\\end{proof}","sha256_utf8":"ca6cec0bf84bdfb2ec95b5c08d6592d1c718b6cbe86f9dba1252d79938e72355"},{"start":50533,"end":50838,"start_line":716,"end_line":722,"raw_tex":"\\begin{lemma} \\label{lem:Epositive}\r\n\tLet $a, b, C_L>0$ and\r\n\t\\begin{align*}\r\n\t\tE(x,y) = a f(x) + \\langle \\grad f(x), y \\rangle + \\frac{b}{2} |y|^2.\r\n\t\\end{align*}\r\n\tIf $ab\\ge C_L$ then for all $x \\in \\R^d$ satisfying $2C_L f(x)\\ge |\\grad f(x)|^2$ we have $E(x,y)\\ge 0$, for all $y \\in \\R^d$.\r\n\\end{lemma}","sha256_utf8":"ea59f082829b35d294ac1566438591d1f2f958d2e6a190bb98b48fe1aa02fdd4"},{"start":50842,"end":51341,"start_line":724,"end_line":733,"raw_tex":"\\begin{proof}\r\n\tIf $\\grad f(x)=0$ the statement is trivial. If $\\grad f(x) \\neq 0$, we denote $\\rho=\\frac{|y|}{|\\grad f(x)|}$ and get\r\n\t$$\r\n\tE(x,y) \\ge \\Bigl(\\frac{a}{2C_L}-\\rho+\\frac{b}{2}\\rho^2 \\Bigr) |\\grad f(x)|^2.\r\n\t$$\r\n\tThe quadratic function $\\varphi(\\rho)=\\frac{a}{2C_L}-\\rho+\\frac{b}{2}\\rho^2$ attains its global minimum at $\\rho=\\frac{1}{b}$ and using $ab \\ge C_L$ we deduce that\r\n\t\\begin{align*}\r\n\t\t\\varphi\\Bigl(\\frac 1b\\Bigr)=\\frac{a}{2C_L}-\\frac{1}{2b}\\ge 0.\r\n\t\\end{align*}\r\n\\end{proof}","sha256_utf8":"38bc2643eb6cc437d01430f9995845a656dc39feffd8b802dfccfe15d7edde2a"},{"start":51476,"end":52834,"start_line":737,"end_line":750,"raw_tex":"\\begin{proposition} \\label{prop:SHB}\r\n\tLet $L,a,b>0, \\sigma \\ge 0$. Let $(\\IA_n)_{n \\in \\N_0}$ be a decreasing sequence of events such that, for all $n \\in \\N_0$, $\\IA_n \\in \\cF_n$ and on $\\IA_n$ it holds that\r\n\t\\begin{align} \\label{eq:1293182}\r\n\t\t|\\grad f(X_n)|^2 \\ge 2L f(X_n) \\quad \\text{and} \\quad \\E[|D_{n+1}|^2 | \\cF_n] \\le \\sigma f(X_n).\r\n\t\\end{align}\r\n\tLet $(\\alpha_n)_{n \\in \\N}, (\\beta_n)_{n \\in \\N}, (\\delta_n)_{n \\in \\N}$ and $(\\epsilon_n)_{n \\in \\N}$ be given by $\\eqref{eq:alpha}$. Assume that $(\\beta_n)_{n \\in \\N}$, $(\\alpha_{n+1}-a\\delta_{n+1}+2\\beta_{n+1}L)_{n \\in \\N_0}$, $(\\epsilon_{n+1}-\\frac{b}{2} \\delta_{n+1})_{n \\in \\N_0}$ and $(\\frac{C_L}{2} \\gamma_{n}^2- \\frac{b}{2}\\gamma_{n})_{n \\in \\N_0}$ are sequences of non-positive reals and $(\\delta_n)_{n \\in \\N}$ is a sequence of non-negative reals.\r\n\tMoreover, if $\\P(\\bigcap_{n \\in \\N_0} \\IA_n)<\\P(\\IA_0)$ additionally assume that $ab \\ge C_L$.\r\n\tThen, for all $n \\in \\N$ it holds that\r\n\t\\begin{align}\\begin{split} \\label{eq:238477}\r\n\t\t\t& \\E[\\1_{\\IA_{n-1}} f(X_{n})]\\le \\Bigl(\\prod_{i=1}^{n} (1+a\\gamma_{i})^{-1}\\Bigr) \\\\\r\n\t\t\t& \\qquad \\qquad \\qquad \\Bigl( \\E[\\1_{\\IA_0} f(X_0)] + \\sum_{i=1}^{n}  \\frac{\\gamma_{i}}{1+a\\gamma_{i}} \\Bigl(\\prod_{j=1}^{i} (1+a\\gamma_{j})\\Bigr) \\Bigl(\\prod_{j=1}^{i}\\delta_{j} \\Bigr)\\E[\\1_{\\IA_0}E_0] \\Bigr).\r\n\t\t\\end{split} \r\n\t\\end{align}\t\r\n\\end{proposition}","sha256_utf8":"f970d882ffe7e71481b24745627a10b1bdb41cd88050ba94bc551c47332a336e"},{"start":52838,"end":58316,"start_line":752,"end_line":837,"raw_tex":"\\begin{proof}\r\n\tIn a first step, we derive a convergence rate for the expectation of the Lyapunov process $(E_n)_{n \\in \\N_0}$. For this, we consider the time evolution of the three summands in \\eqref{eq:Lyapunov}, separately.\r\n\t\r\n\tFirst, we look at the evolution of $(f(X_{n}))_{n \\in \\N_0}$.\r\n\tLet $x,y \\in \\R^d$ and note that with the Lipschitz-continuity of $\\grad f$ we get\r\n\t\\begin{align}\r\n\t\t\\begin{split}\r\n\t\t\t\\label{eq:Taylorp1}\r\n\t\t\tf(y)&\\le f(x) + \\langle \\grad f(x),y-x\\rangle  + \\frac{C_L}{2} |y-x|^2.\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n\t\r\n\tNow, for $n \\in \\N_0$ we use \\eqref{eq:Taylorp1} with $x=X_{n}$ and $y=X_{n+1}$ and \\eqref{eq:MSGDintro} to get \r\n\t\\begin{align} \\begin{split} \r\n\t\t\t\\label{eq:273856230}\r\n\t\t\t&\\E[\\1_{\\IA_n}f(X_{n+1})]\\le \\E\\Big[\\1_{\\IA_n}\\bigl(f(X_{n})-\\gamma_{n+1}^2 |\\grad f(X_n)|^2 \\\\\r\n\t\t\t&\\qquad \\qquad \\qquad \\qquad \\quad + (\\gamma_{n+1}-\\gamma_{n+1}^2\\mu) \\langle \\grad f(X_{n}),V_n\\rangle + \\frac{C_L}{2} \\gamma_{n+1}^2 \\bigl|V_{n+1}\\bigr|^2 \\bigr) \\Big].\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n\t\r\n\tNext, we control the evolution of $(|V_n|^2)_{n \\in \\N_0}$. Using \\eqref{eq:1293182}, we get\r\n\t\\begin{align} \\begin{split}\r\n\t\t\t\\E[\\1_{\\IA_n}|V_{n+1}|^2]\r\n\t\t\t\\le  \\E[\\1_{\\IA_n}(&(1-2\\gamma_{n+1}\\mu+\\gamma_{n+1}^2\\mu^2)|V_n|^2+ \\gamma_{n+1}^2 |\\grad f(X_n)|^2  \\\\\r\n\t\t\t&  - 2 (\\gamma_{n+1}-\\gamma_{n+1}^2\\mu) \\langle V_n, \\grad f(X_n)\\rangle  + \\gamma_{n+1}^2 \\sigma f(X_n))].\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n\tLastly,\r\n\t\\begin{align} \\begin{split} \\label{eq:1764443}\r\n\t\t\t\\E[\\1_{\\IA_n} &\\langle \\grad f(X_{n+1}), V_{n+1} \\rangle] = \\E[ \\1_{\\IA_n} (\\langle \\grad f(X_{n}), V_{n+1} \\rangle + \\langle \\grad f(X_{n+1})- \\grad f(X_{n}), V_{n+1} \\rangle )] \\\\\r\n\t\t\t&\\le \\E[\\1_{\\IA_n} ((1-\\gamma_{n+1}\\mu) \\langle \\grad f(X_n), V_n \\rangle - \\gamma_{n+1}  |\\grad f(X_n)|^2 + C_L \\gamma_{n+1} |V_{n+1}|^2)].\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n\t\r\n\tCombining the estimates \\eqref{eq:273856230}-\\eqref{eq:1764443}, we obtain\r\n\t\\begin{align*}\r\n\t\t&\\E[\\1_{\\IA_{n}} E_{n+1}] \\\\\r\n\t\t&\\le \\E[\\1_{\\IA_{n}}(\\alpha_{n+1} f(X_n) + \\beta_{n+1} |\\grad f(X_n)|^2 +\\delta_{n+1} \\langle \\grad f(X_n),V_n \\rangle + \\epsilon_{n+1} |V_n|^2)],\r\n\t\\end{align*}\r\n\twhere\r\n\t\\begin{align} \\label{eq:alpha}\r\n\t\t\\begin{split}\r\n\t\t\t\\alpha_{n+1} = &a + \\Bigl(\\frac{b}{2}+C_L \\gamma_{n+1}\\Bigl(1+\\frac{\\gamma_{n+1}a}{2}\\Bigr)\\Bigr)\\gamma_{n+1}^2 \\sigma, \\\\\r\n\t\t\t\\beta_{n+1} = &- \\gamma_{n+1} - a\\gamma_{n+1}^2 + \\Bigl(\\frac{b}{2}+C_L \\gamma_{n+1}\\Bigl(1+\\frac{\\gamma_{n+1}a}{2}\\Bigr)\\Bigr)\\gamma_{n+1}^2,\\\\\r\n\t\t\t\\delta_{n+1} = &1-\\mu\\gamma_{n+1}+ a(\\gamma_{n+1}-\\gamma_{n+1}^2\\mu) \\\\\r\n\t\t\t&- 2\\Bigl(\\frac{b}{2}+C_L \\gamma_{n+1}\\Bigl(1+\\frac{\\gamma_{n+1}a}{2}\\Bigr)\\Bigr) (\\gamma_{n+1}-\\gamma_{n+1}^2\\mu), \\\\\r\n\t\t\t\\epsilon_{n+1} = &\\Bigl(\\frac{b}{2}+C_L \\gamma_{n+1}\\Bigl(1+\\frac{\\gamma_{n+1}a}{2}\\Bigr)\\Bigr) (1-2\\gamma_{n+1}\\mu+\\gamma_{n+1}^2\\mu^2).\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n\tBy definition\r\n\t$\r\n\t\\E[\\1_{\\IA_n} \\langle \\grad f(X_n), V_n \\rangle ] =  \\E[\\1_{\\IA_n}(E_n -a f(X_n)-\\sfrac{b}{2} |V_n|^2)],\r\n\t$\r\n\tso that, using the PL-inequality \\eqref{eq:1293182} and the fact that $(\\beta_n)_{n \\in \\N}$ is non-positive, we get \r\n\t\\begin{align} \\begin{split} \\label{eq:579999663}\r\n\t\t\t&\\E[\\1_{\\IA_{n}}E_{n+1}] \\\\\r\n\t\t\t&\\le \\E[\\1_{\\IA_n}(\\delta_{n+1}E_n + (\\alpha_{n+1}-a\\delta_{n+1}+2\\beta_{n+1}L) f(X_n)  + (\\epsilon_{n+1}-\\frac{b}{2} \\delta_{n+1}) |V_n|^2)].\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n\tWith the assumptions on $(\\alpha_{n+1}-a\\delta_{n+1}+2\\beta_{n+1}L)_{n \\in \\N_0}$ and $(\\epsilon_{n+1}-\\frac{b}{2} \\delta_{n+1})_{n \\in \\N_0}$ we have\r\n\t$\r\n\t\\E[\\1_{\\IA_{n}} E_{n+1}] \\le \\delta_{n+1} \\E[\\1_{\\IA_{n}} E_n].\r\n\t$\r\n\tFor $n\\in \\N$, we use Lemma~\\ref{lem:Epositive} and the monotonicity of $(\\IA_n)_{n \\in \\N_0}$ in order to show that $\\E[\\1_{\\IA_{n}} E_n]\\le \\E[\\1_{\\IA_{n-1}} E_n]$ so that, iteratively, \r\n\t\\begin{align} \\label{eq:2776362562}\r\n\t\t\\E[\\1_{\\IA_{n-1}} E_{n}] \\le \\Bigl( \\prod_{i=1}^{n}\\delta_{i} \\Bigr)\\E[\\1_{\\IA_{0}} E_0].\r\n\t\\end{align}\r\n\tNext, we bound the expectation of $(f(X_n))_{n \\in \\N_0}$ using \\eqref{eq:2776362562}. Analogously to \\eqref{eq:273856230}, we have for all $n \\in \\N_0$ that\r\n\t\\begin{align*}\r\n\t\t\\E[\\1_{\\IA_{n}}f(X_{n+1})]\\le \\E\\Big[\\1_{\\IA_{n}} \\bigl( f(X_{n})+\\gamma_{n+1}\\langle \\grad f(X_{n+1}),V_{n+1}\\rangle + \\frac{C_L}{2} \\gamma_{n+1}^2 \\bigl|V_{n+1}\\bigr|^2\\bigr) \\Big]\r\n\t\\end{align*}\r\n\tso that, by definition of $E_{n+1}$,\r\n\t\\begin{align*} \r\n\t\t&(1+a\\gamma_{n+1})\\E[\\1_{\\IA_{n}}f(X_{n+1})]\\\\\r\n\t\t&\\le \\E\\Big[ \\1_{\\IA_{n}} \\bigl(f(X_{n})+\\gamma_{n+1} E_{n+1} + (\\frac{C_L}{2} \\gamma_{n+1}^2- \\frac{b}{2}\\gamma_{n+1}) \\bigl|V_{n+1}\\bigr|^2 \\bigr)\\Big].\r\n\t\\end{align*}\r\n\tBy assumption, $(\\frac{C_L}{2} \\gamma_{n}^2- \\frac{b}{2}\\gamma_{n})_{n \\in \\N}$ is a sequence of non-positive reals. Therefore, we can neglect the last term in the upper bound above and get\r\n\t\\begin{align} \\label{eq:21378861}\r\n\t\t\\E[\\1_{\\IA_{n}}f(X_{n+1})] \\le (1+a\\gamma_{n+1})^{-1} \\E[\\1_{\\IA_n} f(X_{n})]+\\frac{\\gamma_{n+1}}{1+a\\gamma_{n+1}} \\Bigl( \\prod_{i=1}^{n+1}\\delta_{i} \\Bigr)\\E[\\1_{\\IA_0}E_0].\r\n\t\\end{align}\r\n\tUsing the non-negativity of $f(X_n)$, the monotonicity of $(\\IA_n)_{n \\in \\N_0}$ and \\eqref{eq:21378861}, one can inductively show that, for all $n \\in \\N$,\r\n\t\\begin{align*}\r\n\t\t&\\E[\\1_{\\IA_{n-1}} f(X_{n})] \\\\\r\n\t\t&\\le \\Bigl(\\prod_{i=1}^{n} (1+a\\gamma_{i})^{-1}\\Bigr) \\Bigl( \\E[\\1_{\\IA_0} f(X_0)] + \\sum_{i=1}^{n}  \\frac{\\gamma_{i}}{1+a\\gamma_{i}} \\Bigl(\\prod_{j=1}^{i} (1+a\\gamma_{j})\\Bigr) \\Bigl(\\prod_{j=1}^{i}\\delta_{j} \\Bigr)\\E[\\1_{\\IA_0}E_0] \\Bigr).\r\n\t\\end{align*}\r\n\\end{proof}","sha256_utf8":"524f99a8c8d01407bbc7f04e570ca835bba0d7b7420ff11e6dee39773705a1b4"},{"start":59545,"end":60848,"start_line":858,"end_line":878,"raw_tex":"\\begin{proposition} \\label{prop:convrate}\r\n\tLet $L, a,b,\\mu > 0$ and $\\sigma \\ge 0$. Let $(\\IA_n)_{n \\in \\N_0}$ be a decreasing sequence of events such that, for all $n \\in \\N_0$, $\\IA_n \\in \\cF_n$ and on $\\IA_n$ it holds that\r\n\t\\begin{align*} \r\n\t\t|\\grad f(X_n)|^2 \\ge 2L f(X_n)  \\quad \\text{and} \\quad \\E[|D_{n+1}|^2 | \\cF_n] \\le \\sigma f(X_n).\r\n\t\\end{align*} \r\n\tAssume that\r\n\t\\begin{align} \\label{eq:238877}\r\n\t\t\\mu-a+b>0 \\quad , \\quad a\\mu-a^2+ab-2L <0 \\quad \\text{and} \\quad C_L -\\frac{b}{2}(\\mu+a-b)<0.\r\n\t\\end{align}\r\n\tIf $\\P(\\bigcap_{n \\in \\N_0} \\IA_n)< \\P(\\IA_0)$ we additionally assume that $ab \\ge C_L$.\r\n\tThen, for every $0<\\eps<m:= \\min (a, \\mu-a+b)$ there exist constants $C, \\bar \\gamma \\ge 0$ such that if $\\sup_{n \\in \\N}\\gamma_n \\le \\bar \\gamma$ it holds that \r\n\t\\begin{enumerate}\r\n\t\t\\item[(i)] \r\n\t\t$$\r\n\t\t\\max (\\E[\\1_{\\IA_{n-1}} f(X_n)], \t\\E[\\1_{\\IA_{n-1}}|V_n|^2] ) \\le C \\exp(- (m-\\eps) t_n)\r\n\t\t$$\r\n\t\tfor all $n \\in \\N$, where $t_n = \\sum_{i=1}^n \\gamma_i$.\r\n\t\t\\item[(ii)] Let $m'<m-\\eps$ and assume that $ \\sum_{i=0}^\\infty \\exp((m'-(m-\\eps)) t_i)<\\infty$. Then $\\exp(m' t_n) f(X_n) \\to 0$ almost surely on the event $\\IA_\\infty= \\bigcap_{n \\in \\N_0} \\IA_n$.\r\n\t\t\\item[(iii)] The process $(X_n)_{n \\in \\N_0}$ converges almost surely on $\\IA_\\infty$.\r\n\t\\end{enumerate}\r\n\\end{proposition}","sha256_utf8":"82be553abd499efdd29a619c443e93b91d2bda515822549ed8a332a3719d808b"},{"start":60852,"end":66297,"start_line":880,"end_line":966,"raw_tex":"\\begin{proof}\r\n\t(i): \r\n\tFirst, note that, for all $x\\in \\R$, $1+x\\le \\exp(x)$ and $(1+x)^{-1}=e^{-x+o(x)}$. Thus, for every $\\eps'>0$ there exists a $\\bar \\gamma'>0$ such that if $\\max_{i=1, \\dots, n} \\gamma_i \\le \\bar \\gamma'$ we have\r\n\t$$\r\n\t\\prod_{i=1}^{n} (1+a\\gamma_{i})^{-1} \\le \\Bigl(\\prod_{i=1}^{n} \\exp((-a+\\eps')\\gamma_i)\\Bigr) = \\exp((-a+\\eps')t_n ).\r\n\t$$\r\n\tMoreover, for $(\\delta_n)_{n \\in \\N}$ given in \\eqref{eq:alpha} we have $\\delta_n = 1-\\gamma_n(\\mu-a+b)+o(\\gamma_n)$ so that, for all $\\eps''>0$, there exists a $\\bar \\gamma ''>0$ such that if $\\max_{i=1, \\dots, n} \\gamma_i \\le \\bar \\gamma''$ we have $\\delta_i\\ge 0$ for all $i=1, \\dots, n$ and\r\n\t$$\r\n\t\\prod_{i=1}^{n}\\delta_{i} \\le \\exp(-(\\mu-a+b-\\eps'')t_n).\r\n\t$$\r\n\tNote that $\\beta_n = -\\gamma_n + o(\\gamma_n)$, $\\alpha_n-a\\delta_n+2\\beta_n L =  \\gamma_n (a\\mu-a^2+ab-2 L)+o(\\gamma_n)$, \r\n\t$$\r\n\t\\epsilon_n-\\frac b2 \\delta_n =  \\gamma_n \\Bigl(C_L-\\frac{b}{2}(\\mu+a-b)\\Bigr) +o(\\gamma_n) \\;  \\text{ and } \\; \\sfrac {C_L}{2}\\gamma^2-\\sfrac{b}{2}\\gamma=-\\sfrac{b}{2}\\gamma+o(\\gamma),\r\n\t$$\r\n\tand using assumption \\eqref{eq:238877} we can choose $\\bar \\gamma \\le \\min(\\bar \\gamma',\\bar \\gamma'')$ sufficiently small such that if $\\sup_{n \\in \\N}\\gamma_n \\le \\bar \\gamma$ all of the above terms are strictly negative. Then, using Proposition~\\ref{prop:SHB} we get that for all $n \\in \\N$\r\n\t\\begin{align*}\r\n\t\t\\E[\\1_{\\IA_{n-1}} f(X_{n})] \r\n\t\t\\le \\exp((&-a+\\eps'+\\eps'')t_n )  \\\\\r\n\t\t& \\Bigl( \\E[ \\1_{\\IA_0}f(X_0)]+ \\sum_{i=1}^{n}  \\gamma_{i}  \\exp(-(\\mu-2a+b)t_i) \\E[\\1_{\\IA_0}E_0] \\Bigr).\r\n\t\\end{align*}\r\n\tIf $a< \\mu-a+b$, the function $t \\mapsto \\exp(-(\\mu-2a+b)t)$ is monotonously decreasing and we get\r\n\t\\begin{align*}\r\n\t\t\\sum_{i=1}^{n}  \\gamma_{i}  \\exp(-(\\mu-2a+b)t_i) &\\le \\int_{0}^{t_n} \\exp(-(\\mu-2a+b)t) \\, dt.\r\n\t\\end{align*}\r\n\tThus,\r\n\t$$\r\n\t\\E[\\1_{\\IA_{n-1}} f(X_{n})] \\le\\exp((-a+\\eps'+\\eps'')t_n ) \\Bigl( \\E[\\1_{\\IA_0} f(X_0)] + \\frac{\\E[\\1_{\\IA_0}E_0]}{\\mu-2a+b}  \\Bigr).\r\n\t$$\r\n\tFor $a>\\mu-a+b$,  the function $t \\mapsto \\exp(-(\\mu-2a+b)t)$ is monotonously increasing and we get\r\n\t\\begin{align*}\r\n\t\t\\sum_{i=1}^{n}  &\\gamma_{i}  \\exp(-(\\mu-2a+b)t_i) \\le \\int_{0}^{t_n} \\exp(-(\\mu-2a+b)(t+\\bar \\gamma)) \\, dt.\r\n\t\\end{align*}\r\n\tThus,\r\n\t\\begin{align*}\r\n\t\t\\E[\\1_{\\IA_{n-1}} f(X_{n})] \\le&\\exp((-(\\mu-a+b)+\\eps'+\\eps'')t_n ) \\Bigl( \\E[\\1_{\\IA_0} f(X_0)] \\\\\r\n\t\t& \\qquad \\qquad \\qquad  +\\frac{\\E[\\1_{\\IA_0}E_0]}{2a-\\mu-b} \\bigl(\\exp(-(\\mu-2a+b)\\bar \\gamma)  \\Bigr).\r\n\t\\end{align*}\r\n\tLastly, for $a=\\mu-a+b$ we get \r\n\t$\r\n\t\\sum_{i=1}^{n}  \\gamma_{i}  \\exp(-(\\mu-2a+b)t_i)=t_n\r\n\t$\r\n\tand, thus,\r\n\t$$\r\n\t\\E[\\1_{\\IA_{n-1}}f(X_n)] \\le \\exp((-a+\\eps'+\\eps'')t_n ) \\Bigl( \\E[ \\1_{\\IA_0}f(X_0)] + t_n\\E[\\1_{\\IA_0}E_0] \\Bigr).\r\n\t$$\r\n\tNote that $\\exp(-\\eps''' t)t \\to 0$ for all $\\eps'''>0$. Therefore, there exists a constant $C'>0$ such that\r\n\t$$\r\n\t\\E[\\1_{\\IA_{n-1}}f(X_{n})] \\le \\exp((-a+\\eps'+\\eps''+\\eps''')t_n ) \\Bigl( \\E[\\1_{\\IA_0} f(X_0)] + C'\\E[\\1_{\\IA_0}E_0] \\Bigr).\r\n\t$$\r\n\tThe proof of the first assertion follows by choosing $\\eps'=\\eps''= \\eps''' =\\frac 13 \\eps$.\r\n\t\r\n\tFor the second assertion, note that, by Lemma~\\ref{rem:Lipschitz}, we have $|\\grad f(x)|^2 \\le 2 C_L f(x)$ for all $x \\in \\R^d$\r\n\tand, using that $f(X_n)\\ge 0$, we get by the Cauchy-Schwarz inequality\r\n\t\\begin{align*}\r\n\t\t\\frac b2 |V_n|^2 &\\le E_n - \\langle \\grad f(X_n), V_n \\rangle \\le E_n + |\\grad f(X_n)| \\, |V_n|,\r\n\t\\end{align*}\r\n\twhere $E_n$ is defined by \\eqref{eq:Lyapunov}.\r\n\tThus, using Young's inequality, \r\n\t\\begin{align} \\label{eq:Yconv2}\r\n\t\t\\frac b2 \\E\\bigl[\\1_{\\IA_{n-1}}  |V_n|^2\\bigr]\r\n\t\t&\\le \\E[\\1_{\\IA_{n-1}} E_n] + \\frac 1b \\E[\\1_{\\IA_{n-1}} |\\grad f(X_n)|^2] + \\frac b4\\E[ \\1_{\\IA_{n-1}} |V_n|^2] \r\n\t\\end{align}\r\n\tand, with the bound for $\\E[\\1_{\\IA_{n-1}} f(X_n)]$ and \\eqref{eq:2776362562}, we get a constant $C\\ge 0$ such that \r\n\t\\begin{align} \\label{eq:Yconv} \r\n\t\t\\E[\\1_{\\IA_{n-1}}|V_n|^2]^{1/2} \\le C \\exp\\Bigl(-\\frac 12 (m -\\eps)t_n\\Bigr).\r\n\t\\end{align}\r\n\t\r\n\t(ii): Let $m'<m-\\eps$. For $\\eps' >0$ and $n \\in \\N$ consider the set $\\IB_n = \\IA_\\infty \\cap \\{\\sup_{i \\ge n} \\exp(m' t_i) f(X_i) \\ge \\eps' \\}$.\r\n\tWith the Markov inequality and (i) there exists a $C>0$ such that\r\n\t\\begin{align*}\r\n\t\t\\P(\\IB_n) &\\le \\sum_{i=n}^\\infty \\P\\bigl(\\IA_{i-1} \\cap \\bigl\\{\\exp(m' t_i) f(X_i) \\ge \\frac{\\eps'}{2}\\bigr\\}\\bigr) \\\\\r\n\t\t& \\le \\sum_{i=n}^\\infty \\frac{2\\exp(m' t_i)}{\\eps'} \\E[\\1_{\\IA_{i-1}} f(X_i)] \\le \\frac{C}{\\eps'} \\sum_{i=n}^\\infty \\exp((m'-(m-\\eps)) t_i) \\overset{n \\to \\infty}{\\longrightarrow} 0.\r\n\t\\end{align*}\r\n\tWith\r\n\t$$\r\n\t\\P\\Bigl(\\IA_\\infty \\cap \\big\\{\\limsup_{n \\to \\infty} \\exp(m' t_n)f(X_n) \\ge \\eps'\\big\\}\\Bigr) \\le \\P\\Bigl(\\bigcap_{n \\in \\N} B_n\\Bigr) =0\r\n\t$$\r\n\twe get $\\exp(m' t_n)f(X_n)\\to 0$ almost surely on $\\IA_\\infty$.\r\n\t\r\n\t(iii):\r\n\tWe consider the event $\\IA_\\infty$ and bound the distance that the process $(X_n)_{n \\in \\N_0}$ travels. Since $\\eps < m$ the mapping $t  \\mapsto \\exp(-\\frac 12 (m -\\eps)t)$ is monotonously decreasing. Thus, using \\eqref{eq:Yconv} we get\r\n\t\\begin{align*}\r\n\t\t\\E\\Bigl[\\1_{\\IA_\\infty} \\sum_{i=1}^\\infty |X_i-X_{i-1}|\\Bigr] &\\le   \\sum_{i=1}^\\infty \\gamma_i \\E[\\1_{\\IA_{i-1}}  |V_i|^2]^{1/2} \\le C \\sum_{i=1}^\\infty \\gamma_i  \\exp\\Bigl(-\\frac 12 (m -\\eps)t_i\\Bigr)\\\\\r\n\t\t&\\le C \\int_0^\\infty \\exp\\Bigl(-\\frac 12 (m -\\eps)t\\Bigr) \\, dt < \\infty,\r\n\t\\end{align*}\r\n\twhich implies that $\\sum_{i=1}^\\infty |X_i-X_{i-1}|$ is almost surely finite, on $\\IA_\\infty$. Thus, $(X_n)_{n \\in \\N_0}$ almost surely converges on $\\IA_{\\infty}$.\r\n\\end{proof}","sha256_utf8":"1a7d915f4bf039d1016cc97c8101793bd97a5edfe0fc68f2c4ae928f0caa3e4f"},{"start":66301,"end":66448,"start_line":968,"end_line":970,"raw_tex":"\\begin{lemma} \\label{lem:feasible1}\r\n\tFor all $\\mu >0$ there exist $a,b >0$ such that $ab \\ge C_L$ and \\eqref{eq:238877} is satisfied.\r\n\\end{lemma}","sha256_utf8":"73f30cf75c4a9684abcd1afbb1e3f35cc6868a820e2244337488c8e147a27568"},{"start":66452,"end":68538,"start_line":972,"end_line":999,"raw_tex":"\\begin{proof}\r\n\tLet $\\eps,b>0$ and choose $a=b+2\\frac{C_L}{b}-\\mu+\\eps$. Note that $a>0$ iff $\\mu < b+2\\frac {C_L}{b}+\\eps$. Moreover, $\\mu-a+b=2(\\mu-\\frac{C_L}{b})-\\eps$ is positive iff $\\mu > \\frac{C_L}{b}+\\frac{\\eps}{2}$. Now, \r\n\t$\r\n\tC_L-\\frac b2(\\mu+a-b)=-\\frac{b}{2}\\eps < 0\r\n\t$\r\n\tand we have \r\n\t$$\r\n\ta\\mu-a^2+ab-2L = -2\\mu^2+2\\mu\\Bigl(b+\\frac{3C_L}{b}+\\frac{3\\eps}{2}\\Bigr)-2C_L-2L-4\\frac{C_L^2}{b^2}-2\\eps\\Bigr(\\frac b2 + 2\\frac{C_L}{b}+\\frac{\\eps}{2}\\Bigl).\r\n\t$$\r\n\tThe latter term is a quadratic function in $\\mu$ that is negative outside of the two roots. Therefore, $a\\mu-a^2+ab-2L <0$ iff $\\mu \\notin [\\mu_-^{\\eps,b},\\mu_+^{\\eps,b}]$, where\r\n\t\\begin{align} \\label{eq:mu}\r\n\t\t\\mu_{\\pm}^{\\eps,b} = \\frac 12 \\Bigl( b+\\frac{3C_L}{b}+\\frac{3\\eps}{2} \\pm \\sqrt{\\Bigl(b+\\frac{C_L}{b}+\\frac{\\eps}{2}\\Bigr)^2-4L} \\,  \\Bigr).\r\n\t\\end{align}\r\n\tNote that $(b+\\frac{C_L}{b})^2\\ge 4C_L$, for all $b >0$, and $C_L\\ge L$ so that $\\mu_{\\pm}^{\\eps,b}$ is well-defined.\r\n\tMoreover, $\\mu_-^{\\eps,b}>\\frac{C_L}{b}+\\frac{\\eps}{2}$ and $\\mu_+^{\\eps,b}<b+2\\frac{C_L}{b}+\\eps$. The additional assumption $ab\\ge C_L$ is satisfied iff $\\mu\\le b+ \\frac{C_L}{b}+\\eps$. Therefore, the set of friction parameters $\\mu$ that satisfy \\eqref{eq:238877} for the given pair $(a,b)$ is equal to\r\n\t$\r\n\t(\\frac{C_L}{b}+\\frac{\\eps}{2},\\mu_-^{\\eps,b}) \\cup (\\mu_+^{\\eps,b}, b+2\\frac{C_L}{b}+\\eps)\r\n\t$\r\n\tand the set of friction parameters $\\mu$ that satisfy both \\eqref{eq:238877} and $ab \\ge C_L$ for the given pair $(a,b)$ contains the interval\r\n\t$\r\n\t(\\sfrac{C_L}{b}+\\sfrac{\\eps}{2}, \\mu_-^{\\eps,b} \\wedge (b+\\sfrac{C_L}{b}+\\eps)).\r\n\t$\r\n\tFor all $b>0$, the latter interval is non-empty, the upper and lower limits are continuous in $b$ and the lower limit satisfies \r\n\t$$\r\n\t\\frac{C_L}{b}+\\frac{\\eps}{2} \\overset{b \\to \\infty}{\\longrightarrow} \\frac{\\eps}{2} \\quad \\text{ and } \\quad \\frac{C_L}{b}+\\frac{\\eps}{2} \\overset{b \\to 0}{\\longrightarrow} \\infty.\r\n\t$$\r\n\tBy letting $\\eps \\to 0$ we showed that for every $\\mu>0$ there exists a pair $(a,b)$ such that \\eqref{eq:238877} is satisfied and $ab \\ge C_L$.\r\n\\end{proof}","sha256_utf8":"2028199afbe4821ae29b2c02723b4140336e27c09f1b55ac052ada1b3d301092"},{"start":69101,"end":70181,"start_line":1009,"end_line":1030,"raw_tex":"\\begin{lemma} \\label{lem:49378}\r\n\tLet $L, b, \\eps > 0$ and $\\sigma \\ge 0$. Let $(\\IA_n)_{n \\in \\N_0}$ be a decreasing sequence of events such that, for all $n \\in \\N_0$, $\\IA_n \\in \\cF_n$ and on $\\IA_n$ it holds that\r\n\t\\begin{align*} \r\n\t\t|\\grad f(X_n)|^2 \\ge 2L f(X_n) \\quad \\text{and} \\quad \\E[|D_{n+1}|^2 | \\cF_n] \\le \\sigma f(X_n).\r\n\t\\end{align*} \r\n\tLet \r\n\t$\r\n\t\\mu \\in (\\frac{C_L}{b}+\\frac{\\eps}{2}, \\mu_-^{\\eps,b}) \\cup (\\mu_+^{\\eps,b}, b+2\\frac{C_L}{b}+\\eps),\r\n\t$\r\n\twhere $\\mu_{\\pm}^{\\eps,b}$ is defined by \\eqref{eq:mu}.\r\n\tIf $\\P(\\bigcap_{n \\in \\N_0}\\IA_n)<\\P(\\IA_0)$ additionally assume that $\\mu \\le b+\\frac{C_L}{b}+\\eps$. Then, there exist $C, \\bar \\gamma>0$ such that if $\\sup_{n \\in \\N}\\gamma_n \\le \\bar \\gamma$ we have\r\n\t$$\r\n\t\\E[\\1_{\\IA_{n-1}} f(X_n)] \\le C \\exp(-m(\\eps,b,\\mu) t_n),\r\n\t$$\r\n\twhere\r\n\t\\begin{align*} \r\n\t\tm(\\eps,b,\\mu) = \\begin{cases} \r\n\t\t\t2(\\mu-\\frac{C_L}{b}-\\eps), & \\text{ if } \\mu < \\frac{1}{3} (b+4\\frac{C_L}{b}+2\\eps) \\\\\r\n\t\t\tb+2\\frac{C_L}{b}-\\mu, & \\text{ if } \\mu \\ge \\frac{1}{3} (b+4\\frac{C_L}{b}+2\\eps).\r\n\t\t\\end{cases}\r\n\t\\end{align*}\r\n\\end{lemma}","sha256_utf8":"850cb3ef0ab85758ca1831002a483f32406d32c3be3f74fc53dc88d0d393b307"},{"start":70185,"end":70940,"start_line":1032,"end_line":1042,"raw_tex":"\\begin{proof}\r\n\tLet $\\eps, b>0$ and $\\mu$ as in the assumptions of the lemma and set $a=b+2\\frac{C_L}{b}-\\mu+\\eps$. Recall that in the proof of Lemma~\\ref{lem:feasible1} we showed that this choice of parameters satisfies \\eqref{eq:238877} and if $\\P(\\bigcap_{n \\in \\N_0}\\IA_n)<\\P(\\IA_0)$, additionally, $ab \\ge C_L$. Hence, we can apply Proposition~\\ref{prop:convrate} and deduce that there exist constants $C, \\bar \\gamma >0$ such that, if $\\sup_{n \\in \\N}\\gamma_n \\le \\bar \\gamma$, we have\r\n\t$$\r\n\t\\E[\\1_{\\IA_{n-1}}f(X_n)] \\le C \\exp(-m(\\eps,b,r) t_n),\r\n\t$$\r\n\twhere\r\n\t$$\r\n\tm(\\eps,b,\\mu)= \\min(a,\\mu-a+b)-\\eps= \\min\\Bigl(b+2\\frac{C_L}{b}-\\mu,2\\Bigl(\\mu-\\frac{C_L}{b}-\\eps\\Bigr)\\Bigr).\r\n\t$$\r\n\tThe evaluation of the minimum is straight-forward.\r\n\\end{proof}","sha256_utf8":"4efc76d4faf9aae088797dd5cb21003733635f1366f15bc805e4c1f76b9fd92d"},{"start":70947,"end":77383,"start_line":1045,"end_line":1133,"raw_tex":"\\begin{proof} [Proof of Theorem~\\ref{theo2}]\r\n\tWe maximize the convergence rate derived in Lemma~\\ref{lem:49378} over all admissible parameters $\\mu$ and $b$.\r\n\tFirst, assume that $\\kappa=\\frac{C_L}{L}<\\frac 98$. Then, we have for all sufficiently small $\\eps>0$ that $\\kappa^\\eps := \\bigl(\\frac{\\sqrt {C_L}}{\\sqrt{L}}+\\frac{\\eps}{4\\sqrt{L}}\\bigr)^2<\\frac{9}{8}$ so that Set \\begin{align*}\r\n\t\tb_{\\pm}^\\eps := \\frac{3}{\\sqrt 8} \\sqrt L -\\frac{\\eps}{4} \\pm \\sqrt{\\Bigl( \\frac{3}{\\sqrt 8} \\sqrt L -\\frac{\\eps}{4} \\Bigr)^2-C_L}\r\n\t\\end{align*}\r\n\tis well-defined.\r\n\tFor $b \\in (b_-^\\eps, b_+^\\eps)$ we have $\\frac{1}{3} (b+4\\frac{C_L}{b}+2\\eps)<\\mu_-^{\\eps, b}$, such that with Lemma~\\ref{lem:49378} we get for $\\mu \\in (\\frac{C_L}{b}+\\frac{\\eps}{2},\\frac 13(b+4\\frac{C_L}{b}+2\\eps))$ the convergence rate $m(\\eps, b ,\\mu)=2\\bigl( \\mu-\\frac{C_L}{b} -\\eps\\bigr)$. Note that\r\n\t$$\r\n\t\\frac{d}{d\\eps}\\big|_{\\eps=0} \\frac 13 \\Bigl(b_+^\\eps-\\eps^2+4\\frac{C_L}{b_+^\\eps-\\eps^2}+2\\eps\\Bigr) =\\frac 14 + \\frac{3 \\sqrt L}{4 \\sqrt{9L-8C_L}}>0.\r\n\t$$ \r\n\tTherefore, for sufficiently small $\\eps$, we have \r\n\t$$\r\n\t\\mu^*_1 := \\frac{1}{3} \\Bigl(b_+^0+4\\frac{C_L}{b_+^0}\\Bigr) = \\frac{1}{\\sqrt 8}(5-\\sqrt{ 9 - 8\\kappa})\\sqrt{L}<\\frac 13 \\Bigl(b_+^\\eps-\\eps^2+4\\frac{C_L}{b_+^\\eps-\\eps^2}+2\\eps\\Bigr)\r\n\t$$\r\n\tand we\r\n\tget by continuity that\r\n\t\\begin{align*}\r\n\t\tm(\\eps, b_+^\\eps-\\eps^2,\\mu^*_1) &\\overset{\\eps \\to 0}{\\longrightarrow} \\frac 23 \\Bigl(b_+^0+\\frac{C_L}{b_+^0}\\Bigr) = \\frac 23 \\frac{(b_+^0)^2+C_L}{b_+^0} = \\sqrt {2L}.\r\n\t\\end{align*}\r\n\tMoreover, note that $b_+^0$ satisfies $\\frac 13 (b_+^0+4\\frac{C_L}{b_+^0})<b_+^0+\\frac{C_L}{b_+^0}$ since \r\n\t$$\r\n\t\\frac{C_L}{b_+^0} = \\frac {1}{\\sqrt 8} (3 \\sqrt L -\\sqrt{9L-8C_L})<\\frac{1}{\\sqrt 2}(3\\sqrt L +\\sqrt{9L-8 C_L})=2b_+^0.\r\n\t$$\r\n\tThus, $b_+^0 (b_+^0+2\\frac{C_L}{b_+^0}-\\mu^*_1) > C_L$ and, for sufficiently small $\\eps>0$, the parameters $b= b_+^\\eps-\\eps^2$ and $a=b_+^\\eps-\\eps^2+2\\frac{C_L}{b_+^\\eps-\\eps^2}-\\mu^*_1+\\eps$ satisfy $ab\\ge C_L$.\r\n\t\r\n\tAnalogously, we get the convergence rate $m(\\eps,b,\\mu)=b+2\\frac{C_L}{b}-\\mu$ if $b \\in (b_-^\\eps,b_+^\\eps)$ and $\\mu \\in (\\frac 13(b+4\\frac{C_L}{b}+2\\eps),\\mu_-^{\\eps,b})$. Note that, since $\\kappa < \\frac 98$ we have\r\n\t$$\r\n\t\\frac{d}{d\\eps}\\big|_{\\eps=0} \\frac 13 \\Bigl(b_-^\\eps+\\eps^2+4\\frac{C_L}{b_-^\\eps+\\eps^2}+2\\eps\\Bigr) = \\frac 14 - \\frac{3 \\sqrt{L}}{4\\sqrt{9L-8C_L}}<0.\r\n\t$$\r\n\tTherefore, for sufficiently small $\\eps$, we have\r\n\t$$\r\n\t\\mu^*_2 := \\frac{1}{3} \\Bigl(b_-^0+4\\frac{C_L}{b_-^0}\\Bigr) = \\frac{1}{\\sqrt 8}\\bigl(5+\\sqrt{ 9 - 8\\kappa}\\bigr)\\sqrt{L} > \\frac 13\\Bigl(b_-^\\eps+\\eps^2+4\\frac{C_L}{b_-^\\eps+\\eps^2}+2\\eps\\Bigr)\r\n\t$$\r\n\tand we get by continuity that\r\n\t\\begin{align*}\r\n\t\tm(\\eps, b_-^\\eps+\\eps^2,\\mu^*_2) &\\overset{\\eps \\to 0}{\\longrightarrow} b_-^0+2\\frac{C_L}{b_-^0}-\\mu^* = \\frac 23 \\frac{(b_-^0)^2+C_L}{b_-^0}  = \\sqrt {2L}.\r\n\t\\end{align*}\r\n\tIf $L<C_L$, $b_-^0$ satisfies $\\frac 13 (b_-^0+4\\frac{C_L}{b_-^0})<b_-^0+\\frac{C_L}{b_-^0}$ since \r\n\t$$\r\n\t\\frac{C_L}{b_-^0} = \\frac {1}{\\sqrt 8} (3 \\sqrt L +\\sqrt{9L-8C_L})<\\frac{1}{\\sqrt 2}(3\\sqrt L -\\sqrt{9L-8 C_L})=2b_-^0.\r\n\t$$\r\n\tThus, for sufficiently small $\\eps>0$, $b=b_-^\\eps+\\eps^2$ and $a=b_-^\\eps+\\eps^2+2\\frac{C_L}{b_-^\\eps+\\eps^2}-\\mu^*_2+\\eps$ satisfy $ab\\ge C_L$.\r\n\tIf $L=C_L$, we have $\\mu^*_2=b_-^0+\\frac{C_L}{b_-^0}$. Using\r\n\t$\r\n\t\\frac{d}{d\\eps}\\big|_{\\eps=0} \\Bigl( b_-^\\eps+\\frac{C_L}{b_-^\\eps}+\\eps\\Bigr) = \\frac 12,\r\n\t$\r\n\twe get $\\mu^*_2 \\le b_-^\\eps +\\eps^2 + \\frac{C_L}{b_-^\\eps+\\eps^2}+\\eps$ for sufficiently small $\\eps>0$ and, thus, $ab\\ge C_L$ for the parameters $b=b_-^\\eps+\\eps^2$ and $a=b_-^\\eps+\\eps^2+2\\frac{C_L}{b_-^\\eps+\\eps^2}-\\mu^*_2+\\eps$.\r\n\tLastly, note that without loss of generality we can increase the Lipschitz constant $C_L$ as long as $\\kappa<\\frac 98$. We thus get that all friction parameters\r\n\t$$\r\n\t\\mu \\in \\Bigl[\\frac{1}{\\sqrt 8}\\bigl(5-\\sqrt{ 9 - 8\\kappa}\\bigr)\\sqrt{L}, \\frac{1}{\\sqrt 8}\\bigl(5+\\sqrt{ 9 - 8\\kappa}\\bigr)\\sqrt{L}  \\Bigr] \\Big\\backslash\\Bigl\\{\\frac{5}{\\sqrt 8}\\sqrt{L}\\Bigr\\}\r\n\t$$\r\n\tgive an optimal convergence rate of $\\sqrt{2L}-\\eps$. The case $\\mu=\\frac{5}{\\sqrt 8}\\sqrt{L}$ corresponds to $\\kappa=\\frac 98$ and will be treated below.\r\n\t\r\n\tNext, assume that $\\kappa \\ge \\frac 98$ which implies that, for all $\\eps > 0$, we have $\\kappa^\\eps \\ge \\frac{9}{8}$. \r\n\tUsing Lemma~\\ref{lem:49378}, we get the convergence rate $m(\\eps,b,\\mu)=2(\\mu-\\frac{C_L}{b}-\\eps)$ if $\\mu \\in (\\frac{C_L}{b}+\\frac{\\eps}{2}, \\mu_-^{\\eps,b})$.\t\r\n\tFirst, note that if $\\mu_-^{0,b} \\in (\\frac{C_L}{b}+\\frac{\\eps}{2}, \\mu_-^{\\eps,b})$ we have\r\n\t$$\r\n\tm(\\eps, b, \\mu_-^{0,b})=  b+\\frac{C_L}{b} - \\sqrt{\\Bigl(b+\\frac{C_L}{b}\\Bigr)^2-4L}-2\\eps.\r\n\t$$\r\n\tMoreover, $m(\\eps, b,\\mu_-^{0,b}) \\overset{b \\to 0}{\\longrightarrow}-2\\eps$ and $m(\\eps, b,\\mu_-^0) \\overset{b \\to \\infty}{\\longrightarrow}-2\\eps$ as well as\r\n\t$$\r\n\t\\frac{d}{db} m(\\eps,b,\\mu_-^{0,b}) = \\Bigl( 1-\\frac{C_L}{b^2} \\Bigr) \\Bigl( 1-\\frac{b+C_L/b}{\\sqrt{(b+C_L/b)^2-4L}} \\Bigr),\r\n\t$$\r\n\tso that\t\r\n\t$\\frac{d}{db}m(\\eps,b,\\mu_-^{0,b})>0$ for all $b< \\sqrt{C_L}$ and $\\frac{d}{db}m(\\eps,b,\\mu_-^{0,b})<0$ for all $b> \\sqrt{C_L}$. Therefore, the maximal value for $m(\\eps,b,\\mu_-^{0,b})$ is attained at $b^*=\\sqrt{C_L}$, where \r\n\t$$\r\n\t\\mu^*_3= \\mu_-^{0,b^*} = 2\\sqrt{C_L}-\\sqrt{C_L-L} = (2\\sqrt \\kappa-\\sqrt{\\kappa-1})\\sqrt L\r\n\t$$\r\n\tand \r\n\t$$\r\n\tm(\\eps, b^*,\\mu^*_3)=2(\\sqrt{C_L}-\\sqrt{C_L-L}-\\eps) \\overset{\\eps \\to 0}{\\longrightarrow} 2(\\sqrt{\\kappa}-\\sqrt{\\kappa-1})\\sqrt L.\r\n\t$$\r\n\tNow, consider \r\n\t\\begin{align*}\r\n\t\t\\mu_{-}^{\\eps,b^*} &= \\frac 12 \\Bigl( 4 \\sqrt{C_L}+\\frac{3\\eps}{2} - \\sqrt{\\bigl(2 \\sqrt{C_L}+\\frac{\\eps}{2}\\bigr)^2-4L} \\,  \\Bigr)\r\n\t\\end{align*}\r\n\tas a function of $\\eps$. Note that for all $\\eps\\ge 0$ we have $\r\n\t\\frac{L}{C_L} \\le \\frac 29 \\Bigl( 2+\\frac{\\eps}{2\\sqrt{C_L}} \\Bigr)^2,\r\n\t$ and, thus,\r\n\t\\begin{align*}\r\n\t\t\\frac{d}{d\\eps} \\mu_{-}^{\\eps,b^*} &= \\frac 12 \\Bigl( \\frac 32 - \\frac 12 \\Bigl(\\bigl(2 \\sqrt{C_L}+\\frac{\\eps}{2}\\bigr)^2-4L\\Bigr)^{-1/2}  \\bigl(2 \\sqrt{C_L}+\\frac{\\eps}{2}\\bigr) \\Bigr) \\ge 0\r\n\t\\end{align*}\r\n\tTherefore, for sufficiently small $\\eps >0$ we have $\\mu^*_3 \\in (\\frac{C_L}{b^*}+\\frac{\\eps}{2}, \\mu_-^{\\eps,b^*})$.\r\n\tMoreover, note that\r\n\t$\r\n\t\\mu^* _3 < 2 \\sqrt{C_L} = b^* + \\frac{C_L}{b^*}\r\n\t$\r\n\tsuch that the parameters $b^*$ and $a=b^*+2\\frac{C_L}{b^*}-\\mu+\\eps$ satisfy $ab^*\\ge C_L$. Finally, for $\\kappa = \\frac 98$ we get $2(\\sqrt{\\kappa}-\\sqrt{\\kappa-1})\\sqrt L= \\sqrt{2L}$ and $\\mu^*_3=\\frac{5}{\\sqrt 8}\\sqrt L$.\r\n\\end{proof}","sha256_utf8":"4d7e30cbab28f587fa95dd1817730f24b91fdca539beacc9b7aeb6441fafdb3b"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"目标函数与概率空间","source_units":[{"start":12814,"end":13360,"start_line":302,"end_line":303,"raw_tex":"We assume throughout that $f: \\R^d \\to \\R$ is a differentiable function with $C_L$-Lipschitz continuous gradient,\\footnote{We comment on the necessity of this global Lipschitz continuity in Remark~\\ref{rem:Lip1} and Remark~\\ref{rem:Lip2} below.} for some constant $C_L \\ge 0$, such that $\\inf_{x \\in \\R^d} f(x)=0$. \r\nLet $(\\Omega, (\\mathcal F_{n})_{n \\in \\N_0}, \\mathcal F, \\P)$ be a filtered probability space and let $(X_n)_{n \\in \\N_0}$, $(V_n)_{n \\in \\N_0}$ be $(\\mathcal F_n)_{n \\in \\N_0}$-adapted processes satisfying for all $n \\in \\N_0$\r\n","sha256_utf8":"241fd8e65c535efb16c2d9bcd20fc12ea4171e09367fa806d62c4adae7c675ea"}]},{"id":"S002","semantic_type":"algorithm","kind":"algorithm","title":"离散随机动量迭代","source_units":[{"start":13360,"end":13568,"start_line":304,"end_line":308,"raw_tex":"\\begin{align}\\begin{split} \\label{eq:MSGDintro}\r\n\t\tX_{n+1} &= X_n + \\gamma_{n+1}V_{n+1}, \\\\\r\n\t\tV_{n+1}&= V_n - \\gamma_{n+1}\\mu V_n -\\gamma_{n+1}\\nabla f(X_n) + \\gamma_{n+1} D_{n+1},\r\n\t\\end{split}\r\n\\end{align}","sha256_utf8":"74454d0558128caf01499d9f51bab3ece68375e290847d383996bf286c2c1796"},{"start":13570,"end":14116,"start_line":309,"end_line":309,"raw_tex":"where $X_0, V_0 \\in L^2(\\Omega, \\cF_0)$, $(\\gamma_{n})_{n \\in \\N}$ is a sequence of strictly positive reals, $\\mu>0$ and $(D_n)_{n \\in \\N}$ is a sequence of $L^2$-martingale differences with respect to the filtration $(\\cF_n)_{n \\in \\N_0}$. In the following, we also call $(X_n)_{n \\in \\N_0}$ given by \\eqref{eq:MSGDintro} the MSGD scheme with step-sizes $(\\gamma_{n})_{n \\in \\N}$ and friction parameter $\\mu$. The choice $(D_n)_{n \\in \\N}=(\\nabla  f(X_{n-1})-\\nabla g(X_{n-1},\\Gamma_{n}))_{n \\in \\N}$ recovers the algorithm \\eqref{eq:MSGDapp}.\r\n","sha256_utf8":"559507fb5235b739f5f766ab7b11fd63a37e501e4a09de846cca4b887a523cec"}]},{"id":"S003","semantic_type":"notation","kind":"notation","title":"范数与内积记号","source_units":[{"start":29291,"end":29680,"start_line":423,"end_line":423,"raw_tex":"\\textbf{Notation:} We denote by $v^\\dagger$ the transpose of a vector $v \\in \\R^d$, by $A^\\dagger$ the transpose of a matrix $A \\in \\R^{n \\times k}$ and by $\\|A\\|_F$, respectively $\\|A\\|$, the Frobenius norm, respectively operator norm of $A$. Moreover,  $|\\cdot|$ denotes the standard Euclidean norm and $\\langle \\cdot, \\cdot \\rangle$ the standard scalar product on the Euclidean space.\r\n","sha256_utf8":"7894398d665a5ced10ae80caf41ad24d96b984b0010af9ee3eff1582febd4b5e"}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","title":"局部梯度与噪声条件","source_units":[{"start":35987,"end":36362,"start_line":498,"end_line":511,"raw_tex":"\tLet $L>0, \\sigma \\ge 0$.\r\n\tLet $\\mathcal D\\subset \\R^d$ be an open set and assume that \r\n\t$$\r\n\t|\\grad f(x)|^2 \\ge 2L f(x)\r\n\t$$\r\n\tfor all $x \\in \\mathcal D$.\r\n\tMoreover, for $n \\in \\N_0$, let\r\n\t$\r\n\t\\IA_n = \\{X_i(\\omega) \\in \\mathcal D \\text{ for all } i =0, \\dots, n\\}\r\n\t$\r\n\tand assume that\r\n\t$$ \r\n\t\\E[|D_{n+1}|^2 | \\cF_n] \\le \\sigma f(X_n), \\quad \\text{ on } \\IA_{n}.\r\n\t$$\r\n","sha256_utf8":"b90df0d8a20d7538461393acb3c88792757e00502662bc8d58c965d224df324f"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"随机李雅普诺夫函数","source_units":[{"start":50011,"end":50246,"start_line":710,"end_line":713,"raw_tex":"Let $a,b >0$ and let $(E_n)_{n \\in \\N_0}$ be the $(\\cF_n)_{n \\in \\N_0}$-adapted stochastic process given by\r\n\\begin{align} \\label{eq:Lyapunov}\r\n\tE_n = a f(X_{n}) + \\langle \\grad f (X_{n}), V_n \\rangle +\\frac{b}{2} |V_n|^2.\r\n\\end{align}","sha256_utf8":"c6df097fb0297cbd139a5e89fa291f6b77e73f325cc07f96dbffbc3bba100a1e"},{"start":50248,"end":50531,"start_line":714,"end_line":714,"raw_tex":"In our setting, $(E_n)_{n \\in \\N_0}$ plays the role of a random Lyapunov function. Although, in general, $(E_n)_{n \\in \\N_0}$ might take negative values, assuming the inverse PL-condition there exist choices for $a$ and $b$ such that $(E_n)_{n \\in \\N_0}$ is a non-negative process.\r\n","sha256_utf8":"aac402c1f3824c0a68de981de39e1265dba4ed39230e979f95adcc47ade3f34a"}]},{"id":"S006","semantic_type":"theorem","kind":"theorem","title":"小步长下的显式指数速率","source_units":[{"start":38721,"end":39625,"start_line":532,"end_line":552,"raw_tex":"\\begin{theorem} \\label{theo2}\r\n\tSet $\\kappa = \\frac{C_L}{L}$. Let \r\n\t\\begin{align*}\r\n\t\t\\mu \\in \r\n\t\t\\begin{cases}\r\n\t\t\t\\bigl[\\frac{1}{\\sqrt 8}\\bigl(5-\\sqrt{ 9 - 8\\kappa}\\bigr)\\sqrt{L},\\frac{1}{\\sqrt 8}\\bigl(5+\\sqrt{ 9 - 8\\kappa}\\bigr)\\sqrt{L} \\, \\bigr], & \\text{ if } \\kappa < \\frac 98, \\\\\r\n\t\t\t\\bigl\\{(2\\sqrt \\kappa-\\sqrt{\\kappa-1})\\sqrt L \\bigr\\}, & \\text{ if } \\kappa \\ge \\frac 98.\r\n\t\t\\end{cases}\r\n\t\\end{align*}\r\n\tThen, under the assumptions of Theorem~\\ref{theo1}, for every $\\eps>0$ there exist $C, \\bar \\gamma\\ge 0$ such that if $\\sup_{n \\in \\N}\\gamma_n \\le \\bar \\gamma$ it holds that\r\n\t\\begin{align*}\r\n\t\t\\E[\\1_{\\IA_{n-1}} f(X_n)] \\le C \\exp(-(m-\\eps)t_n), \\quad \\text{ for all } n \\in \\N,\r\n\t\\end{align*}\r\n\twhere  \r\n\t$$\r\n\tm = \r\n\t\\begin{cases} \\sqrt{2L},& \\text{ if } \\kappa<\\frac{9}{8},\\\\\r\n\t\t2(\\sqrt{\\kappa}-\\sqrt{\\kappa-1})\\sqrt{L},& \\text{ if } \\kappa \\ge \\frac 98.\r\n\t\\end{cases}\r\n\t$$\r\n\\end{theorem}","sha256_utf8":"f9af5cca8652ed293ae4a721763f6a7f49269a6dd20986199267e0297dbeadda"}]},{"id":"S007","semantic_type":"assumption","kind":"assumption","title":"条件数与摩擦参数选择","source_units":[{"start":38752,"end":39134,"start_line":533,"end_line":540,"raw_tex":"\tSet $\\kappa = \\frac{C_L}{L}$. Let \r\n\t\\begin{align*}\r\n\t\t\\mu \\in \r\n\t\t\\begin{cases}\r\n\t\t\t\\bigl[\\frac{1}{\\sqrt 8}\\bigl(5-\\sqrt{ 9 - 8\\kappa}\\bigr)\\sqrt{L},\\frac{1}{\\sqrt 8}\\bigl(5+\\sqrt{ 9 - 8\\kappa}\\bigr)\\sqrt{L} \\, \\bigr], & \\text{ if } \\kappa < \\frac 98, \\\\\r\n\t\t\t\\bigl\\{(2\\sqrt \\kappa-\\sqrt{\\kappa-1})\\sqrt L \\bigr\\}, & \\text{ if } \\kappa \\ge \\frac 98.\r\n\t\t\\end{cases}\r\n\t\\end{align*}\r\n","sha256_utf8":"3a19b1dbc60a9214d798ed35441d65cceef24c351f1053873aa7432de74e3f98"}]},{"id":"S008","semantic_type":"proof","kind":"proof","title":"显式速率定理完整证明","source_units":[{"start":70947,"end":77383,"start_line":1045,"end_line":1133,"raw_tex":"\\begin{proof} [Proof of Theorem~\\ref{theo2}]\r\n\tWe maximize the convergence rate derived in Lemma~\\ref{lem:49378} over all admissible parameters $\\mu$ and $b$.\r\n\tFirst, assume that $\\kappa=\\frac{C_L}{L}<\\frac 98$. Then, we have for all sufficiently small $\\eps>0$ that $\\kappa^\\eps := \\bigl(\\frac{\\sqrt {C_L}}{\\sqrt{L}}+\\frac{\\eps}{4\\sqrt{L}}\\bigr)^2<\\frac{9}{8}$ so that Set \\begin{align*}\r\n\t\tb_{\\pm}^\\eps := \\frac{3}{\\sqrt 8} \\sqrt L -\\frac{\\eps}{4} \\pm \\sqrt{\\Bigl( \\frac{3}{\\sqrt 8} \\sqrt L -\\frac{\\eps}{4} \\Bigr)^2-C_L}\r\n\t\\end{align*}\r\n\tis well-defined.\r\n\tFor $b \\in (b_-^\\eps, b_+^\\eps)$ we have $\\frac{1}{3} (b+4\\frac{C_L}{b}+2\\eps)<\\mu_-^{\\eps, b}$, such that with Lemma~\\ref{lem:49378} we get for $\\mu \\in (\\frac{C_L}{b}+\\frac{\\eps}{2},\\frac 13(b+4\\frac{C_L}{b}+2\\eps))$ the convergence rate $m(\\eps, b ,\\mu)=2\\bigl( \\mu-\\frac{C_L}{b} -\\eps\\bigr)$. Note that\r\n\t$$\r\n\t\\frac{d}{d\\eps}\\big|_{\\eps=0} \\frac 13 \\Bigl(b_+^\\eps-\\eps^2+4\\frac{C_L}{b_+^\\eps-\\eps^2}+2\\eps\\Bigr) =\\frac 14 + \\frac{3 \\sqrt L}{4 \\sqrt{9L-8C_L}}>0.\r\n\t$$ \r\n\tTherefore, for sufficiently small $\\eps$, we have \r\n\t$$\r\n\t\\mu^*_1 := \\frac{1}{3} \\Bigl(b_+^0+4\\frac{C_L}{b_+^0}\\Bigr) = \\frac{1}{\\sqrt 8}(5-\\sqrt{ 9 - 8\\kappa})\\sqrt{L}<\\frac 13 \\Bigl(b_+^\\eps-\\eps^2+4\\frac{C_L}{b_+^\\eps-\\eps^2}+2\\eps\\Bigr)\r\n\t$$\r\n\tand we\r\n\tget by continuity that\r\n\t\\begin{align*}\r\n\t\tm(\\eps, b_+^\\eps-\\eps^2,\\mu^*_1) &\\overset{\\eps \\to 0}{\\longrightarrow} \\frac 23 \\Bigl(b_+^0+\\frac{C_L}{b_+^0}\\Bigr) = \\frac 23 \\frac{(b_+^0)^2+C_L}{b_+^0} = \\sqrt {2L}.\r\n\t\\end{align*}\r\n\tMoreover, note that $b_+^0$ satisfies $\\frac 13 (b_+^0+4\\frac{C_L}{b_+^0})<b_+^0+\\frac{C_L}{b_+^0}$ since \r\n\t$$\r\n\t\\frac{C_L}{b_+^0} = \\frac {1}{\\sqrt 8} (3 \\sqrt L -\\sqrt{9L-8C_L})<\\frac{1}{\\sqrt 2}(3\\sqrt L +\\sqrt{9L-8 C_L})=2b_+^0.\r\n\t$$\r\n\tThus, $b_+^0 (b_+^0+2\\frac{C_L}{b_+^0}-\\mu^*_1) > C_L$ and, for sufficiently small $\\eps>0$, the parameters $b= b_+^\\eps-\\eps^2$ and $a=b_+^\\eps-\\eps^2+2\\frac{C_L}{b_+^\\eps-\\eps^2}-\\mu^*_1+\\eps$ satisfy $ab\\ge C_L$.\r\n\t\r\n\tAnalogously, we get the convergence rate $m(\\eps,b,\\mu)=b+2\\frac{C_L}{b}-\\mu$ if $b \\in (b_-^\\eps,b_+^\\eps)$ and $\\mu \\in (\\frac 13(b+4\\frac{C_L}{b}+2\\eps),\\mu_-^{\\eps,b})$. Note that, since $\\kappa < \\frac 98$ we have\r\n\t$$\r\n\t\\frac{d}{d\\eps}\\big|_{\\eps=0} \\frac 13 \\Bigl(b_-^\\eps+\\eps^2+4\\frac{C_L}{b_-^\\eps+\\eps^2}+2\\eps\\Bigr) = \\frac 14 - \\frac{3 \\sqrt{L}}{4\\sqrt{9L-8C_L}}<0.\r\n\t$$\r\n\tTherefore, for sufficiently small $\\eps$, we have\r\n\t$$\r\n\t\\mu^*_2 := \\frac{1}{3} \\Bigl(b_-^0+4\\frac{C_L}{b_-^0}\\Bigr) = \\frac{1}{\\sqrt 8}\\bigl(5+\\sqrt{ 9 - 8\\kappa}\\bigr)\\sqrt{L} > \\frac 13\\Bigl(b_-^\\eps+\\eps^2+4\\frac{C_L}{b_-^\\eps+\\eps^2}+2\\eps\\Bigr)\r\n\t$$\r\n\tand we get by continuity that\r\n\t\\begin{align*}\r\n\t\tm(\\eps, b_-^\\eps+\\eps^2,\\mu^*_2) &\\overset{\\eps \\to 0}{\\longrightarrow} b_-^0+2\\frac{C_L}{b_-^0}-\\mu^* = \\frac 23 \\frac{(b_-^0)^2+C_L}{b_-^0}  = \\sqrt {2L}.\r\n\t\\end{align*}\r\n\tIf $L<C_L$, $b_-^0$ satisfies $\\frac 13 (b_-^0+4\\frac{C_L}{b_-^0})<b_-^0+\\frac{C_L}{b_-^0}$ since \r\n\t$$\r\n\t\\frac{C_L}{b_-^0} = \\frac {1}{\\sqrt 8} (3 \\sqrt L +\\sqrt{9L-8C_L})<\\frac{1}{\\sqrt 2}(3\\sqrt L -\\sqrt{9L-8 C_L})=2b_-^0.\r\n\t$$\r\n\tThus, for sufficiently small $\\eps>0$, $b=b_-^\\eps+\\eps^2$ and $a=b_-^\\eps+\\eps^2+2\\frac{C_L}{b_-^\\eps+\\eps^2}-\\mu^*_2+\\eps$ satisfy $ab\\ge C_L$.\r\n\tIf $L=C_L$, we have $\\mu^*_2=b_-^0+\\frac{C_L}{b_-^0}$. Using\r\n\t$\r\n\t\\frac{d}{d\\eps}\\big|_{\\eps=0} \\Bigl( b_-^\\eps+\\frac{C_L}{b_-^\\eps}+\\eps\\Bigr) = \\frac 12,\r\n\t$\r\n\twe get $\\mu^*_2 \\le b_-^\\eps +\\eps^2 + \\frac{C_L}{b_-^\\eps+\\eps^2}+\\eps$ for sufficiently small $\\eps>0$ and, thus, $ab\\ge C_L$ for the parameters $b=b_-^\\eps+\\eps^2$ and $a=b_-^\\eps+\\eps^2+2\\frac{C_L}{b_-^\\eps+\\eps^2}-\\mu^*_2+\\eps$.\r\n\tLastly, note that without loss of generality we can increase the Lipschitz constant $C_L$ as long as $\\kappa<\\frac 98$. We thus get that all friction parameters\r\n\t$$\r\n\t\\mu \\in \\Bigl[\\frac{1}{\\sqrt 8}\\bigl(5-\\sqrt{ 9 - 8\\kappa}\\bigr)\\sqrt{L}, \\frac{1}{\\sqrt 8}\\bigl(5+\\sqrt{ 9 - 8\\kappa}\\bigr)\\sqrt{L}  \\Bigr] \\Big\\backslash\\Bigl\\{\\frac{5}{\\sqrt 8}\\sqrt{L}\\Bigr\\}\r\n\t$$\r\n\tgive an optimal convergence rate of $\\sqrt{2L}-\\eps$. The case $\\mu=\\frac{5}{\\sqrt 8}\\sqrt{L}$ corresponds to $\\kappa=\\frac 98$ and will be treated below.\r\n\t\r\n\tNext, assume that $\\kappa \\ge \\frac 98$ which implies that, for all $\\eps > 0$, we have $\\kappa^\\eps \\ge \\frac{9}{8}$. \r\n\tUsing Lemma~\\ref{lem:49378}, we get the convergence rate $m(\\eps,b,\\mu)=2(\\mu-\\frac{C_L}{b}-\\eps)$ if $\\mu \\in (\\frac{C_L}{b}+\\frac{\\eps}{2}, \\mu_-^{\\eps,b})$.\t\r\n\tFirst, note that if $\\mu_-^{0,b} \\in (\\frac{C_L}{b}+\\frac{\\eps}{2}, \\mu_-^{\\eps,b})$ we have\r\n\t$$\r\n\tm(\\eps, b, \\mu_-^{0,b})=  b+\\frac{C_L}{b} - \\sqrt{\\Bigl(b+\\frac{C_L}{b}\\Bigr)^2-4L}-2\\eps.\r\n\t$$\r\n\tMoreover, $m(\\eps, b,\\mu_-^{0,b}) \\overset{b \\to 0}{\\longrightarrow}-2\\eps$ and $m(\\eps, b,\\mu_-^0) \\overset{b \\to \\infty}{\\longrightarrow}-2\\eps$ as well as\r\n\t$$\r\n\t\\frac{d}{db} m(\\eps,b,\\mu_-^{0,b}) = \\Bigl( 1-\\frac{C_L}{b^2} \\Bigr) \\Bigl( 1-\\frac{b+C_L/b}{\\sqrt{(b+C_L/b)^2-4L}} \\Bigr),\r\n\t$$\r\n\tso that\t\r\n\t$\\frac{d}{db}m(\\eps,b,\\mu_-^{0,b})>0$ for all $b< \\sqrt{C_L}$ and $\\frac{d}{db}m(\\eps,b,\\mu_-^{0,b})<0$ for all $b> \\sqrt{C_L}$. Therefore, the maximal value for $m(\\eps,b,\\mu_-^{0,b})$ is attained at $b^*=\\sqrt{C_L}$, where \r\n\t$$\r\n\t\\mu^*_3= \\mu_-^{0,b^*} = 2\\sqrt{C_L}-\\sqrt{C_L-L} = (2\\sqrt \\kappa-\\sqrt{\\kappa-1})\\sqrt L\r\n\t$$\r\n\tand \r\n\t$$\r\n\tm(\\eps, b^*,\\mu^*_3)=2(\\sqrt{C_L}-\\sqrt{C_L-L}-\\eps) \\overset{\\eps \\to 0}{\\longrightarrow} 2(\\sqrt{\\kappa}-\\sqrt{\\kappa-1})\\sqrt L.\r\n\t$$\r\n\tNow, consider \r\n\t\\begin{align*}\r\n\t\t\\mu_{-}^{\\eps,b^*} &= \\frac 12 \\Bigl( 4 \\sqrt{C_L}+\\frac{3\\eps}{2} - \\sqrt{\\bigl(2 \\sqrt{C_L}+\\frac{\\eps}{2}\\bigr)^2-4L} \\,  \\Bigr)\r\n\t\\end{align*}\r\n\tas a function of $\\eps$. Note that for all $\\eps\\ge 0$ we have $\r\n\t\\frac{L}{C_L} \\le \\frac 29 \\Bigl( 2+\\frac{\\eps}{2\\sqrt{C_L}} \\Bigr)^2,\r\n\t$ and, thus,\r\n\t\\begin{align*}\r\n\t\t\\frac{d}{d\\eps} \\mu_{-}^{\\eps,b^*} &= \\frac 12 \\Bigl( \\frac 32 - \\frac 12 \\Bigl(\\bigl(2 \\sqrt{C_L}+\\frac{\\eps}{2}\\bigr)^2-4L\\Bigr)^{-1/2}  \\bigl(2 \\sqrt{C_L}+\\frac{\\eps}{2}\\bigr) \\Bigr) \\ge 0\r\n\t\\end{align*}\r\n\tTherefore, for sufficiently small $\\eps >0$ we have $\\mu^*_3 \\in (\\frac{C_L}{b^*}+\\frac{\\eps}{2}, \\mu_-^{\\eps,b^*})$.\r\n\tMoreover, note that\r\n\t$\r\n\t\\mu^* _3 < 2 \\sqrt{C_L} = b^* + \\frac{C_L}{b^*}\r\n\t$\r\n\tsuch that the parameters $b^*$ and $a=b^*+2\\frac{C_L}{b^*}-\\mu+\\eps$ satisfy $ab^*\\ge C_L$. Finally, for $\\kappa = \\frac 98$ we get $2(\\sqrt{\\kappa}-\\sqrt{\\kappa-1})\\sqrt L= \\sqrt{2L}$ and $\\mu^*_3=\\frac{5}{\\sqrt 8}\\sqrt L$.\r\n\\end{proof}","sha256_utf8":"4d7e30cbab28f587fa95dd1817730f24b91fdca539beacc9b7aeb6441fafdb3b"}]},{"id":"S009","semantic_type":"proof_step","kind":"proof_step","title":"低条件数分支的参数区间","source_units":[{"start":70993,"end":71506,"start_line":1046,"end_line":1050,"raw_tex":"\tWe maximize the convergence rate derived in Lemma~\\ref{lem:49378} over all admissible parameters $\\mu$ and $b$.\r\n\tFirst, assume that $\\kappa=\\frac{C_L}{L}<\\frac 98$. Then, we have for all sufficiently small $\\eps>0$ that $\\kappa^\\eps := \\bigl(\\frac{\\sqrt {C_L}}{\\sqrt{L}}+\\frac{\\eps}{4\\sqrt{L}}\\bigr)^2<\\frac{9}{8}$ so that Set \\begin{align*}\r\n\t\tb_{\\pm}^\\eps := \\frac{3}{\\sqrt 8} \\sqrt L -\\frac{\\eps}{4} \\pm \\sqrt{\\Bigl( \\frac{3}{\\sqrt 8} \\sqrt L -\\frac{\\eps}{4} \\Bigr)^2-C_L}\r\n\t\\end{align*}\r\n\tis well-defined.\r\n","sha256_utf8":"6b455765d5ead3c3d095bc688db5a76887b1e646f1975514c2e9ab7b3ab541ed"}]},{"id":"S010","semantic_type":"proof_step","kind":"proof_step","title":"下端摩擦参数的可行性","source_units":[{"start":71506,"end":71980,"start_line":1051,"end_line":1054,"raw_tex":"\tFor $b \\in (b_-^\\eps, b_+^\\eps)$ we have $\\frac{1}{3} (b+4\\frac{C_L}{b}+2\\eps)<\\mu_-^{\\eps, b}$, such that with Lemma~\\ref{lem:49378} we get for $\\mu \\in (\\frac{C_L}{b}+\\frac{\\eps}{2},\\frac 13(b+4\\frac{C_L}{b}+2\\eps))$ the convergence rate $m(\\eps, b ,\\mu)=2\\bigl( \\mu-\\frac{C_L}{b} -\\eps\\bigr)$. Note that\r\n\t$$\r\n\t\\frac{d}{d\\eps}\\big|_{\\eps=0} \\frac 13 \\Bigl(b_+^\\eps-\\eps^2+4\\frac{C_L}{b_+^\\eps-\\eps^2}+2\\eps\\Bigr) =\\frac 14 + \\frac{3 \\sqrt L}{4 \\sqrt{9L-8C_L}}>0.\r\n\t$$ \r\n","sha256_utf8":"ae0ab566e873aecea19593b7946b88baa0e51bbbcc98247907e551b3ffaba11d"}]},{"id":"S011","semantic_type":"proof_step","kind":"proof_step","title":"下端参数的极限速率","source_units":[{"start":71980,"end":72467,"start_line":1055,"end_line":1063,"raw_tex":"\tTherefore, for sufficiently small $\\eps$, we have \r\n\t$$\r\n\t\\mu^*_1 := \\frac{1}{3} \\Bigl(b_+^0+4\\frac{C_L}{b_+^0}\\Bigr) = \\frac{1}{\\sqrt 8}(5-\\sqrt{ 9 - 8\\kappa})\\sqrt{L}<\\frac 13 \\Bigl(b_+^\\eps-\\eps^2+4\\frac{C_L}{b_+^\\eps-\\eps^2}+2\\eps\\Bigr)\r\n\t$$\r\n\tand we\r\n\tget by continuity that\r\n\t\\begin{align*}\r\n\t\tm(\\eps, b_+^\\eps-\\eps^2,\\mu^*_1) &\\overset{\\eps \\to 0}{\\longrightarrow} \\frac 23 \\Bigl(b_+^0+\\frac{C_L}{b_+^0}\\Bigr) = \\frac 23 \\frac{(b_+^0)^2+C_L}{b_+^0} = \\sqrt {2L}.\r\n\t\\end{align*}\r\n","sha256_utf8":"b395c17b3024c2e12c9fd484b6463b78a57439841ddf64d949afa0d4160e6ae2"}]},{"id":"S012","semantic_type":"proof_step","kind":"proof_step","title":"下端参数的非负性能量条件","source_units":[{"start":72467,"end":72926,"start_line":1064,"end_line":1068,"raw_tex":"\tMoreover, note that $b_+^0$ satisfies $\\frac 13 (b_+^0+4\\frac{C_L}{b_+^0})<b_+^0+\\frac{C_L}{b_+^0}$ since \r\n\t$$\r\n\t\\frac{C_L}{b_+^0} = \\frac {1}{\\sqrt 8} (3 \\sqrt L -\\sqrt{9L-8C_L})<\\frac{1}{\\sqrt 2}(3\\sqrt L +\\sqrt{9L-8 C_L})=2b_+^0.\r\n\t$$\r\n\tThus, $b_+^0 (b_+^0+2\\frac{C_L}{b_+^0}-\\mu^*_1) > C_L$ and, for sufficiently small $\\eps>0$, the parameters $b= b_+^\\eps-\\eps^2$ and $a=b_+^\\eps-\\eps^2+2\\frac{C_L}{b_+^\\eps-\\eps^2}-\\mu^*_1+\\eps$ satisfy $ab\\ge C_L$.\r\n","sha256_utf8":"4eb79ee64a781d18461a07cde43fb2bc6496f35cafbea170e26691609d1e4b5b"}]},{"id":"S013","semantic_type":"proof_step","kind":"proof_step","title":"上端摩擦参数的可行性","source_units":[{"start":72929,"end":73315,"start_line":1070,"end_line":1073,"raw_tex":"\tAnalogously, we get the convergence rate $m(\\eps,b,\\mu)=b+2\\frac{C_L}{b}-\\mu$ if $b \\in (b_-^\\eps,b_+^\\eps)$ and $\\mu \\in (\\frac 13(b+4\\frac{C_L}{b}+2\\eps),\\mu_-^{\\eps,b})$. Note that, since $\\kappa < \\frac 98$ we have\r\n\t$$\r\n\t\\frac{d}{d\\eps}\\big|_{\\eps=0} \\frac 13 \\Bigl(b_-^\\eps+\\eps^2+4\\frac{C_L}{b_-^\\eps+\\eps^2}+2\\eps\\Bigr) = \\frac 14 - \\frac{3 \\sqrt{L}}{4\\sqrt{9L-8C_L}}<0.\r\n\t$$\r\n","sha256_utf8":"d763120a5f931d47d1671615b56a5fb6303cde973ca7890f0e8eae58e7b5b804"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"上端参数的极限速率","source_units":[{"start":73315,"end":73797,"start_line":1074,"end_line":1081,"raw_tex":"\tTherefore, for sufficiently small $\\eps$, we have\r\n\t$$\r\n\t\\mu^*_2 := \\frac{1}{3} \\Bigl(b_-^0+4\\frac{C_L}{b_-^0}\\Bigr) = \\frac{1}{\\sqrt 8}\\bigl(5+\\sqrt{ 9 - 8\\kappa}\\bigr)\\sqrt{L} > \\frac 13\\Bigl(b_-^\\eps+\\eps^2+4\\frac{C_L}{b_-^\\eps+\\eps^2}+2\\eps\\Bigr)\r\n\t$$\r\n\tand we get by continuity that\r\n\t\\begin{align*}\r\n\t\tm(\\eps, b_-^\\eps+\\eps^2,\\mu^*_2) &\\overset{\\eps \\to 0}{\\longrightarrow} b_-^0+2\\frac{C_L}{b_-^0}-\\mu^* = \\frac 23 \\frac{(b_-^0)^2+C_L}{b_-^0}  = \\sqrt {2L}.\r\n\t\\end{align*}\r\n","sha256_utf8":"4d3ccd687300e42266dbb2fbd3a2627f1e3e4817d08fa252f8272850704993f2"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"严格条件数情形的能量条件","source_units":[{"start":73797,"end":74178,"start_line":1082,"end_line":1086,"raw_tex":"\tIf $L<C_L$, $b_-^0$ satisfies $\\frac 13 (b_-^0+4\\frac{C_L}{b_-^0})<b_-^0+\\frac{C_L}{b_-^0}$ since \r\n\t$$\r\n\t\\frac{C_L}{b_-^0} = \\frac {1}{\\sqrt 8} (3 \\sqrt L +\\sqrt{9L-8C_L})<\\frac{1}{\\sqrt 2}(3\\sqrt L -\\sqrt{9L-8 C_L})=2b_-^0.\r\n\t$$\r\n\tThus, for sufficiently small $\\eps>0$, $b=b_-^\\eps+\\eps^2$ and $a=b_-^\\eps+\\eps^2+2\\frac{C_L}{b_-^\\eps+\\eps^2}-\\mu^*_2+\\eps$ satisfy $ab\\ge C_L$.\r\n","sha256_utf8":"b70be6ecffb9557e683a11022daadaa18896837605014b6dc8e204865a393eb4"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"单位条件数情形的能量条件","source_units":[{"start":74178,"end":74577,"start_line":1087,"end_line":1091,"raw_tex":"\tIf $L=C_L$, we have $\\mu^*_2=b_-^0+\\frac{C_L}{b_-^0}$. Using\r\n\t$\r\n\t\\frac{d}{d\\eps}\\big|_{\\eps=0} \\Bigl( b_-^\\eps+\\frac{C_L}{b_-^\\eps}+\\eps\\Bigr) = \\frac 12,\r\n\t$\r\n\twe get $\\mu^*_2 \\le b_-^\\eps +\\eps^2 + \\frac{C_L}{b_-^\\eps+\\eps^2}+\\eps$ for sufficiently small $\\eps>0$ and, thus, $ab\\ge C_L$ for the parameters $b=b_-^\\eps+\\eps^2$ and $a=b_-^\\eps+\\eps^2+2\\frac{C_L}{b_-^\\eps+\\eps^2}-\\mu^*_2+\\eps$.\r\n","sha256_utf8":"0cdcb509336b720adb7c91e33f013560b03ed5c1145985ce037182f7076489ff"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"扩展至整个摩擦参数区间","source_units":[{"start":74577,"end":75104,"start_line":1092,"end_line":1096,"raw_tex":"\tLastly, note that without loss of generality we can increase the Lipschitz constant $C_L$ as long as $\\kappa<\\frac 98$. We thus get that all friction parameters\r\n\t$$\r\n\t\\mu \\in \\Bigl[\\frac{1}{\\sqrt 8}\\bigl(5-\\sqrt{ 9 - 8\\kappa}\\bigr)\\sqrt{L}, \\frac{1}{\\sqrt 8}\\bigl(5+\\sqrt{ 9 - 8\\kappa}\\bigr)\\sqrt{L}  \\Bigr] \\Big\\backslash\\Bigl\\{\\frac{5}{\\sqrt 8}\\sqrt{L}\\Bigr\\}\r\n\t$$\r\n\tgive an optimal convergence rate of $\\sqrt{2L}-\\eps$. The case $\\mu=\\frac{5}{\\sqrt 8}\\sqrt{L}$ corresponds to $\\kappa=\\frac 98$ and will be treated below.\r\n","sha256_utf8":"5dffe89159b70cf12063e3df6078580dd9b566fd4df90b8290ebb531c3442957"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"高条件数分支的候选速率","source_units":[{"start":75107,"end":75590,"start_line":1098,"end_line":1103,"raw_tex":"\tNext, assume that $\\kappa \\ge \\frac 98$ which implies that, for all $\\eps > 0$, we have $\\kappa^\\eps \\ge \\frac{9}{8}$. \r\n\tUsing Lemma~\\ref{lem:49378}, we get the convergence rate $m(\\eps,b,\\mu)=2(\\mu-\\frac{C_L}{b}-\\eps)$ if $\\mu \\in (\\frac{C_L}{b}+\\frac{\\eps}{2}, \\mu_-^{\\eps,b})$.\t\r\n\tFirst, note that if $\\mu_-^{0,b} \\in (\\frac{C_L}{b}+\\frac{\\eps}{2}, \\mu_-^{\\eps,b})$ we have\r\n\t$$\r\n\tm(\\eps, b, \\mu_-^{0,b})=  b+\\frac{C_L}{b} - \\sqrt{\\Bigl(b+\\frac{C_L}{b}\\Bigr)^2-4L}-2\\eps.\r\n\t$$\r\n","sha256_utf8":"eec0a96ade2860d4c608e2f9a64a73b891ec3aee7372a636ed5b059370b66f18"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"优化辅助参数与摩擦参数","source_units":[{"start":75590,"end":76379,"start_line":1104,"end_line":1116,"raw_tex":"\tMoreover, $m(\\eps, b,\\mu_-^{0,b}) \\overset{b \\to 0}{\\longrightarrow}-2\\eps$ and $m(\\eps, b,\\mu_-^0) \\overset{b \\to \\infty}{\\longrightarrow}-2\\eps$ as well as\r\n\t$$\r\n\t\\frac{d}{db} m(\\eps,b,\\mu_-^{0,b}) = \\Bigl( 1-\\frac{C_L}{b^2} \\Bigr) \\Bigl( 1-\\frac{b+C_L/b}{\\sqrt{(b+C_L/b)^2-4L}} \\Bigr),\r\n\t$$\r\n\tso that\t\r\n\t$\\frac{d}{db}m(\\eps,b,\\mu_-^{0,b})>0$ for all $b< \\sqrt{C_L}$ and $\\frac{d}{db}m(\\eps,b,\\mu_-^{0,b})<0$ for all $b> \\sqrt{C_L}$. Therefore, the maximal value for $m(\\eps,b,\\mu_-^{0,b})$ is attained at $b^*=\\sqrt{C_L}$, where \r\n\t$$\r\n\t\\mu^*_3= \\mu_-^{0,b^*} = 2\\sqrt{C_L}-\\sqrt{C_L-L} = (2\\sqrt \\kappa-\\sqrt{\\kappa-1})\\sqrt L\r\n\t$$\r\n\tand \r\n\t$$\r\n\tm(\\eps, b^*,\\mu^*_3)=2(\\sqrt{C_L}-\\sqrt{C_L-L}-\\eps) \\overset{\\eps \\to 0}{\\longrightarrow} 2(\\sqrt{\\kappa}-\\sqrt{\\kappa-1})\\sqrt L.\r\n\t$$\r\n","sha256_utf8":"f98e4a7b556b72ad4f8ca6a3f0956870f476c7d58abc551d05d2521cfcde351b"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"验证最优摩擦参数的可行性","source_units":[{"start":76379,"end":77065,"start_line":1117,"end_line":1127,"raw_tex":"\tNow, consider \r\n\t\\begin{align*}\r\n\t\t\\mu_{-}^{\\eps,b^*} &= \\frac 12 \\Bigl( 4 \\sqrt{C_L}+\\frac{3\\eps}{2} - \\sqrt{\\bigl(2 \\sqrt{C_L}+\\frac{\\eps}{2}\\bigr)^2-4L} \\,  \\Bigr)\r\n\t\\end{align*}\r\n\tas a function of $\\eps$. Note that for all $\\eps\\ge 0$ we have $\r\n\t\\frac{L}{C_L} \\le \\frac 29 \\Bigl( 2+\\frac{\\eps}{2\\sqrt{C_L}} \\Bigr)^2,\r\n\t$ and, thus,\r\n\t\\begin{align*}\r\n\t\t\\frac{d}{d\\eps} \\mu_{-}^{\\eps,b^*} &= \\frac 12 \\Bigl( \\frac 32 - \\frac 12 \\Bigl(\\bigl(2 \\sqrt{C_L}+\\frac{\\eps}{2}\\bigr)^2-4L\\Bigr)^{-1/2}  \\bigl(2 \\sqrt{C_L}+\\frac{\\eps}{2}\\bigr) \\Bigr) \\ge 0\r\n\t\\end{align*}\r\n\tTherefore, for sufficiently small $\\eps >0$ we have $\\mu^*_3 \\in (\\frac{C_L}{b^*}+\\frac{\\eps}{2}, \\mu_-^{\\eps,b^*})$.\r\n","sha256_utf8":"127a3cd90f982c339510dac8d70cdb167cf89587751f507b3e1d9a5581201801"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"验证能量条件并衔接临界情形","source_units":[{"start":77065,"end":77372,"start_line":1128,"end_line":1132,"raw_tex":"\tMoreover, note that\r\n\t$\r\n\t\\mu^* _3 < 2 \\sqrt{C_L} = b^* + \\frac{C_L}{b^*}\r\n\t$\r\n\tsuch that the parameters $b^*$ and $a=b^*+2\\frac{C_L}{b^*}-\\mu+\\eps$ satisfy $ab^*\\ge C_L$. Finally, for $\\kappa = \\frac 98$ we get $2(\\sqrt{\\kappa}-\\sqrt{\\kappa-1})\\sqrt L= \\sqrt{2L}$ and $\\mu^*_3=\\frac{5}{\\sqrt 8}\\sqrt L$.\r\n","sha256_utf8":"1c459a59b78c39e4e1ea03d27c0c5ff9d74207164a0f86be18d5ebff360bdf5e"}]},{"id":"S022","semantic_type":"lemma","kind":"lemma","title":"依赖辅助参数的分段速率","source_units":[{"start":69101,"end":70181,"start_line":1009,"end_line":1030,"raw_tex":"\\begin{lemma} \\label{lem:49378}\r\n\tLet $L, b, \\eps > 0$ and $\\sigma \\ge 0$. Let $(\\IA_n)_{n \\in \\N_0}$ be a decreasing sequence of events such that, for all $n \\in \\N_0$, $\\IA_n \\in \\cF_n$ and on $\\IA_n$ it holds that\r\n\t\\begin{align*} \r\n\t\t|\\grad f(X_n)|^2 \\ge 2L f(X_n) \\quad \\text{and} \\quad \\E[|D_{n+1}|^2 | \\cF_n] \\le \\sigma f(X_n).\r\n\t\\end{align*} \r\n\tLet \r\n\t$\r\n\t\\mu \\in (\\frac{C_L}{b}+\\frac{\\eps}{2}, \\mu_-^{\\eps,b}) \\cup (\\mu_+^{\\eps,b}, b+2\\frac{C_L}{b}+\\eps),\r\n\t$\r\n\twhere $\\mu_{\\pm}^{\\eps,b}$ is defined by \\eqref{eq:mu}.\r\n\tIf $\\P(\\bigcap_{n \\in \\N_0}\\IA_n)<\\P(\\IA_0)$ additionally assume that $\\mu \\le b+\\frac{C_L}{b}+\\eps$. Then, there exist $C, \\bar \\gamma>0$ such that if $\\sup_{n \\in \\N}\\gamma_n \\le \\bar \\gamma$ we have\r\n\t$$\r\n\t\\E[\\1_{\\IA_{n-1}} f(X_n)] \\le C \\exp(-m(\\eps,b,\\mu) t_n),\r\n\t$$\r\n\twhere\r\n\t\\begin{align*} \r\n\t\tm(\\eps,b,\\mu) = \\begin{cases} \r\n\t\t\t2(\\mu-\\frac{C_L}{b}-\\eps), & \\text{ if } \\mu < \\frac{1}{3} (b+4\\frac{C_L}{b}+2\\eps) \\\\\r\n\t\t\tb+2\\frac{C_L}{b}-\\mu, & \\text{ if } \\mu \\ge \\frac{1}{3} (b+4\\frac{C_L}{b}+2\\eps).\r\n\t\t\\end{cases}\r\n\t\\end{align*}\r\n\\end{lemma}","sha256_utf8":"850cb3ef0ab85758ca1831002a483f32406d32c3be3f74fc53dc88d0d393b307"}]},{"id":"S023","semantic_type":"proof","kind":"proof","title":"分段速率引理完整证明","source_units":[{"start":70185,"end":70940,"start_line":1032,"end_line":1042,"raw_tex":"\\begin{proof}\r\n\tLet $\\eps, b>0$ and $\\mu$ as in the assumptions of the lemma and set $a=b+2\\frac{C_L}{b}-\\mu+\\eps$. Recall that in the proof of Lemma~\\ref{lem:feasible1} we showed that this choice of parameters satisfies \\eqref{eq:238877} and if $\\P(\\bigcap_{n \\in \\N_0}\\IA_n)<\\P(\\IA_0)$, additionally, $ab \\ge C_L$. Hence, we can apply Proposition~\\ref{prop:convrate} and deduce that there exist constants $C, \\bar \\gamma >0$ such that, if $\\sup_{n \\in \\N}\\gamma_n \\le \\bar \\gamma$, we have\r\n\t$$\r\n\t\\E[\\1_{\\IA_{n-1}}f(X_n)] \\le C \\exp(-m(\\eps,b,r) t_n),\r\n\t$$\r\n\twhere\r\n\t$$\r\n\tm(\\eps,b,\\mu)= \\min(a,\\mu-a+b)-\\eps= \\min\\Bigl(b+2\\frac{C_L}{b}-\\mu,2\\Bigl(\\mu-\\frac{C_L}{b}-\\eps\\Bigr)\\Bigr).\r\n\t$$\r\n\tThe evaluation of the minimum is straight-forward.\r\n\\end{proof}","sha256_utf8":"4efc76d4faf9aae088797dd5cb21003733635f1366f15bc805e4c1f76b9fd92d"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"构造可行参数并应用收敛命题","source_units":[{"start":70200,"end":70745,"start_line":1033,"end_line":1036,"raw_tex":"\tLet $\\eps, b>0$ and $\\mu$ as in the assumptions of the lemma and set $a=b+2\\frac{C_L}{b}-\\mu+\\eps$. Recall that in the proof of Lemma~\\ref{lem:feasible1} we showed that this choice of parameters satisfies \\eqref{eq:238877} and if $\\P(\\bigcap_{n \\in \\N_0}\\IA_n)<\\P(\\IA_0)$, additionally, $ab \\ge C_L$. Hence, we can apply Proposition~\\ref{prop:convrate} and deduce that there exist constants $C, \\bar \\gamma >0$ such that, if $\\sup_{n \\in \\N}\\gamma_n \\le \\bar \\gamma$, we have\r\n\t$$\r\n\t\\E[\\1_{\\IA_{n-1}}f(X_n)] \\le C \\exp(-m(\\eps,b,r) t_n),\r\n\t$$\r\n","sha256_utf8":"67521b8c653d4d879f845a38325ab278fe74524521e9ab85a06df8111badf9cc"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"计算最小值的分段表达式","source_units":[{"start":70745,"end":70929,"start_line":1037,"end_line":1041,"raw_tex":"\twhere\r\n\t$$\r\n\tm(\\eps,b,\\mu)= \\min(a,\\mu-a+b)-\\eps= \\min\\Bigl(b+2\\frac{C_L}{b}-\\mu,2\\Bigl(\\mu-\\frac{C_L}{b}-\\eps\\Bigr)\\Bigr).\r\n\t$$\r\n\tThe evaluation of the minimum is straight-forward.\r\n","sha256_utf8":"739c5268f5b48366c9411b9982347f71f3d638a28a822cb656f2802dceff6bb3"}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","title":"任意摩擦参数的可行性","source_units":[{"start":66301,"end":66448,"start_line":968,"end_line":970,"raw_tex":"\\begin{lemma} \\label{lem:feasible1}\r\n\tFor all $\\mu >0$ there exist $a,b >0$ such that $ab \\ge C_L$ and \\eqref{eq:238877} is satisfied.\r\n\\end{lemma}","sha256_utf8":"73f30cf75c4a9684abcd1afbb1e3f35cc6868a820e2244337488c8e147a27568"}]},{"id":"S027","semantic_type":"proof","kind":"proof","title":"参数可行性引理完整证明","source_units":[{"start":66452,"end":68538,"start_line":972,"end_line":999,"raw_tex":"\\begin{proof}\r\n\tLet $\\eps,b>0$ and choose $a=b+2\\frac{C_L}{b}-\\mu+\\eps$. Note that $a>0$ iff $\\mu < b+2\\frac {C_L}{b}+\\eps$. Moreover, $\\mu-a+b=2(\\mu-\\frac{C_L}{b})-\\eps$ is positive iff $\\mu > \\frac{C_L}{b}+\\frac{\\eps}{2}$. Now, \r\n\t$\r\n\tC_L-\\frac b2(\\mu+a-b)=-\\frac{b}{2}\\eps < 0\r\n\t$\r\n\tand we have \r\n\t$$\r\n\ta\\mu-a^2+ab-2L = -2\\mu^2+2\\mu\\Bigl(b+\\frac{3C_L}{b}+\\frac{3\\eps}{2}\\Bigr)-2C_L-2L-4\\frac{C_L^2}{b^2}-2\\eps\\Bigr(\\frac b2 + 2\\frac{C_L}{b}+\\frac{\\eps}{2}\\Bigl).\r\n\t$$\r\n\tThe latter term is a quadratic function in $\\mu$ that is negative outside of the two roots. Therefore, $a\\mu-a^2+ab-2L <0$ iff $\\mu \\notin [\\mu_-^{\\eps,b},\\mu_+^{\\eps,b}]$, where\r\n\t\\begin{align} \\label{eq:mu}\r\n\t\t\\mu_{\\pm}^{\\eps,b} = \\frac 12 \\Bigl( b+\\frac{3C_L}{b}+\\frac{3\\eps}{2} \\pm \\sqrt{\\Bigl(b+\\frac{C_L}{b}+\\frac{\\eps}{2}\\Bigr)^2-4L} \\,  \\Bigr).\r\n\t\\end{align}\r\n\tNote that $(b+\\frac{C_L}{b})^2\\ge 4C_L$, for all $b >0$, and $C_L\\ge L$ so that $\\mu_{\\pm}^{\\eps,b}$ is well-defined.\r\n\tMoreover, $\\mu_-^{\\eps,b}>\\frac{C_L}{b}+\\frac{\\eps}{2}$ and $\\mu_+^{\\eps,b}<b+2\\frac{C_L}{b}+\\eps$. The additional assumption $ab\\ge C_L$ is satisfied iff $\\mu\\le b+ \\frac{C_L}{b}+\\eps$. Therefore, the set of friction parameters $\\mu$ that satisfy \\eqref{eq:238877} for the given pair $(a,b)$ is equal to\r\n\t$\r\n\t(\\frac{C_L}{b}+\\frac{\\eps}{2},\\mu_-^{\\eps,b}) \\cup (\\mu_+^{\\eps,b}, b+2\\frac{C_L}{b}+\\eps)\r\n\t$\r\n\tand the set of friction parameters $\\mu$ that satisfy both \\eqref{eq:238877} and $ab \\ge C_L$ for the given pair $(a,b)$ contains the interval\r\n\t$\r\n\t(\\sfrac{C_L}{b}+\\sfrac{\\eps}{2}, \\mu_-^{\\eps,b} \\wedge (b+\\sfrac{C_L}{b}+\\eps)).\r\n\t$\r\n\tFor all $b>0$, the latter interval is non-empty, the upper and lower limits are continuous in $b$ and the lower limit satisfies \r\n\t$$\r\n\t\\frac{C_L}{b}+\\frac{\\eps}{2} \\overset{b \\to \\infty}{\\longrightarrow} \\frac{\\eps}{2} \\quad \\text{ and } \\quad \\frac{C_L}{b}+\\frac{\\eps}{2} \\overset{b \\to 0}{\\longrightarrow} \\infty.\r\n\t$$\r\n\tBy letting $\\eps \\to 0$ we showed that for every $\\mu>0$ there exists a pair $(a,b)$ such that \\eqref{eq:238877} is satisfied and $ab \\ge C_L$.\r\n\\end{proof}","sha256_utf8":"2028199afbe4821ae29b2c02723b4140336e27c09f1b55ac052ada1b3d301092"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"构造参数并展开约束","source_units":[{"start":66467,"end":66924,"start_line":973,"end_line":980,"raw_tex":"\tLet $\\eps,b>0$ and choose $a=b+2\\frac{C_L}{b}-\\mu+\\eps$. Note that $a>0$ iff $\\mu < b+2\\frac {C_L}{b}+\\eps$. Moreover, $\\mu-a+b=2(\\mu-\\frac{C_L}{b})-\\eps$ is positive iff $\\mu > \\frac{C_L}{b}+\\frac{\\eps}{2}$. Now, \r\n\t$\r\n\tC_L-\\frac b2(\\mu+a-b)=-\\frac{b}{2}\\eps < 0\r\n\t$\r\n\tand we have \r\n\t$$\r\n\ta\\mu-a^2+ab-2L = -2\\mu^2+2\\mu\\Bigl(b+\\frac{3C_L}{b}+\\frac{3\\eps}{2}\\Bigr)-2C_L-2L-4\\frac{C_L^2}{b^2}-2\\eps\\Bigr(\\frac b2 + 2\\frac{C_L}{b}+\\frac{\\eps}{2}\\Bigl).\r\n\t$$\r\n","sha256_utf8":"d520d97ec185613bf8d4d0c048dd45ef2b7014570b4b377174e61c07c5c7f25e"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"计算约束二次式的根","source_units":[{"start":66924,"end":67413,"start_line":981,"end_line":985,"raw_tex":"\tThe latter term is a quadratic function in $\\mu$ that is negative outside of the two roots. Therefore, $a\\mu-a^2+ab-2L <0$ iff $\\mu \\notin [\\mu_-^{\\eps,b},\\mu_+^{\\eps,b}]$, where\r\n\t\\begin{align} \\label{eq:mu}\r\n\t\t\\mu_{\\pm}^{\\eps,b} = \\frac 12 \\Bigl( b+\\frac{3C_L}{b}+\\frac{3\\eps}{2} \\pm \\sqrt{\\Bigl(b+\\frac{C_L}{b}+\\frac{\\eps}{2}\\Bigr)^2-4L} \\,  \\Bigr).\r\n\t\\end{align}\r\n\tNote that $(b+\\frac{C_L}{b})^2\\ge 4C_L$, for all $b >0$, and $C_L\\ge L$ so that $\\mu_{\\pm}^{\\eps,b}$ is well-defined.\r\n","sha256_utf8":"5cee242fb082ea8ba6db0e0ff30763f8d663ca9c838ca89b58a74f8b81752b15"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"确定满足能量条件的可行区间","source_units":[{"start":67413,"end":68057,"start_line":986,"end_line":993,"raw_tex":"\tMoreover, $\\mu_-^{\\eps,b}>\\frac{C_L}{b}+\\frac{\\eps}{2}$ and $\\mu_+^{\\eps,b}<b+2\\frac{C_L}{b}+\\eps$. The additional assumption $ab\\ge C_L$ is satisfied iff $\\mu\\le b+ \\frac{C_L}{b}+\\eps$. Therefore, the set of friction parameters $\\mu$ that satisfy \\eqref{eq:238877} for the given pair $(a,b)$ is equal to\r\n\t$\r\n\t(\\frac{C_L}{b}+\\frac{\\eps}{2},\\mu_-^{\\eps,b}) \\cup (\\mu_+^{\\eps,b}, b+2\\frac{C_L}{b}+\\eps)\r\n\t$\r\n\tand the set of friction parameters $\\mu$ that satisfy both \\eqref{eq:238877} and $ab \\ge C_L$ for the given pair $(a,b)$ contains the interval\r\n\t$\r\n\t(\\sfrac{C_L}{b}+\\sfrac{\\eps}{2}, \\mu_-^{\\eps,b} \\wedge (b+\\sfrac{C_L}{b}+\\eps)).\r\n\t$\r\n","sha256_utf8":"cc8991d0ab98cbcec7e65c409cffe5f0d5d3c3cf8bcfd9cf11e763e97fee0d50"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"连续性覆盖全部正摩擦参数","source_units":[{"start":68057,"end":68527,"start_line":994,"end_line":998,"raw_tex":"\tFor all $b>0$, the latter interval is non-empty, the upper and lower limits are continuous in $b$ and the lower limit satisfies \r\n\t$$\r\n\t\\frac{C_L}{b}+\\frac{\\eps}{2} \\overset{b \\to \\infty}{\\longrightarrow} \\frac{\\eps}{2} \\quad \\text{ and } \\quad \\frac{C_L}{b}+\\frac{\\eps}{2} \\overset{b \\to 0}{\\longrightarrow} \\infty.\r\n\t$$\r\n\tBy letting $\\eps \\to 0$ we showed that for every $\\mu>0$ there exists a pair $(a,b)$ such that \\eqref{eq:238877} is satisfied and $ab \\ge C_L$.\r\n","sha256_utf8":"3e5017c0d350c23da413869d5fc24c259f01d195aa1f2ff47cbdd3d925f9ad01"}]},{"id":"S032","semantic_type":"proposition","kind":"proposition","title":"小步长指数衰减与路径收敛","source_units":[{"start":59545,"end":60848,"start_line":858,"end_line":878,"raw_tex":"\\begin{proposition} \\label{prop:convrate}\r\n\tLet $L, a,b,\\mu > 0$ and $\\sigma \\ge 0$. Let $(\\IA_n)_{n \\in \\N_0}$ be a decreasing sequence of events such that, for all $n \\in \\N_0$, $\\IA_n \\in \\cF_n$ and on $\\IA_n$ it holds that\r\n\t\\begin{align*} \r\n\t\t|\\grad f(X_n)|^2 \\ge 2L f(X_n)  \\quad \\text{and} \\quad \\E[|D_{n+1}|^2 | \\cF_n] \\le \\sigma f(X_n).\r\n\t\\end{align*} \r\n\tAssume that\r\n\t\\begin{align} \\label{eq:238877}\r\n\t\t\\mu-a+b>0 \\quad , \\quad a\\mu-a^2+ab-2L <0 \\quad \\text{and} \\quad C_L -\\frac{b}{2}(\\mu+a-b)<0.\r\n\t\\end{align}\r\n\tIf $\\P(\\bigcap_{n \\in \\N_0} \\IA_n)< \\P(\\IA_0)$ we additionally assume that $ab \\ge C_L$.\r\n\tThen, for every $0<\\eps<m:= \\min (a, \\mu-a+b)$ there exist constants $C, \\bar \\gamma \\ge 0$ such that if $\\sup_{n \\in \\N}\\gamma_n \\le \\bar \\gamma$ it holds that \r\n\t\\begin{enumerate}\r\n\t\t\\item[(i)] \r\n\t\t$$\r\n\t\t\\max (\\E[\\1_{\\IA_{n-1}} f(X_n)], \t\\E[\\1_{\\IA_{n-1}}|V_n|^2] ) \\le C \\exp(- (m-\\eps) t_n)\r\n\t\t$$\r\n\t\tfor all $n \\in \\N$, where $t_n = \\sum_{i=1}^n \\gamma_i$.\r\n\t\t\\item[(ii)] Let $m'<m-\\eps$ and assume that $ \\sum_{i=0}^\\infty \\exp((m'-(m-\\eps)) t_i)<\\infty$. Then $\\exp(m' t_n) f(X_n) \\to 0$ almost surely on the event $\\IA_\\infty= \\bigcap_{n \\in \\N_0} \\IA_n$.\r\n\t\t\\item[(iii)] The process $(X_n)_{n \\in \\N_0}$ converges almost surely on $\\IA_\\infty$.\r\n\t\\end{enumerate}\r\n\\end{proposition}","sha256_utf8":"82be553abd499efdd29a619c443e93b91d2bda515822549ed8a332a3719d808b"}]},{"id":"S033","semantic_type":"proof","kind":"proof","title":"小步长收敛命题完整证明","source_units":[{"start":60852,"end":66297,"start_line":880,"end_line":966,"raw_tex":"\\begin{proof}\r\n\t(i): \r\n\tFirst, note that, for all $x\\in \\R$, $1+x\\le \\exp(x)$ and $(1+x)^{-1}=e^{-x+o(x)}$. Thus, for every $\\eps'>0$ there exists a $\\bar \\gamma'>0$ such that if $\\max_{i=1, \\dots, n} \\gamma_i \\le \\bar \\gamma'$ we have\r\n\t$$\r\n\t\\prod_{i=1}^{n} (1+a\\gamma_{i})^{-1} \\le \\Bigl(\\prod_{i=1}^{n} \\exp((-a+\\eps')\\gamma_i)\\Bigr) = \\exp((-a+\\eps')t_n ).\r\n\t$$\r\n\tMoreover, for $(\\delta_n)_{n \\in \\N}$ given in \\eqref{eq:alpha} we have $\\delta_n = 1-\\gamma_n(\\mu-a+b)+o(\\gamma_n)$ so that, for all $\\eps''>0$, there exists a $\\bar \\gamma ''>0$ such that if $\\max_{i=1, \\dots, n} \\gamma_i \\le \\bar \\gamma''$ we have $\\delta_i\\ge 0$ for all $i=1, \\dots, n$ and\r\n\t$$\r\n\t\\prod_{i=1}^{n}\\delta_{i} \\le \\exp(-(\\mu-a+b-\\eps'')t_n).\r\n\t$$\r\n\tNote that $\\beta_n = -\\gamma_n + o(\\gamma_n)$, $\\alpha_n-a\\delta_n+2\\beta_n L =  \\gamma_n (a\\mu-a^2+ab-2 L)+o(\\gamma_n)$, \r\n\t$$\r\n\t\\epsilon_n-\\frac b2 \\delta_n =  \\gamma_n \\Bigl(C_L-\\frac{b}{2}(\\mu+a-b)\\Bigr) +o(\\gamma_n) \\;  \\text{ and } \\; \\sfrac {C_L}{2}\\gamma^2-\\sfrac{b}{2}\\gamma=-\\sfrac{b}{2}\\gamma+o(\\gamma),\r\n\t$$\r\n\tand using assumption \\eqref{eq:238877} we can choose $\\bar \\gamma \\le \\min(\\bar \\gamma',\\bar \\gamma'')$ sufficiently small such that if $\\sup_{n \\in \\N}\\gamma_n \\le \\bar \\gamma$ all of the above terms are strictly negative. Then, using Proposition~\\ref{prop:SHB} we get that for all $n \\in \\N$\r\n\t\\begin{align*}\r\n\t\t\\E[\\1_{\\IA_{n-1}} f(X_{n})] \r\n\t\t\\le \\exp((&-a+\\eps'+\\eps'')t_n )  \\\\\r\n\t\t& \\Bigl( \\E[ \\1_{\\IA_0}f(X_0)]+ \\sum_{i=1}^{n}  \\gamma_{i}  \\exp(-(\\mu-2a+b)t_i) \\E[\\1_{\\IA_0}E_0] \\Bigr).\r\n\t\\end{align*}\r\n\tIf $a< \\mu-a+b$, the function $t \\mapsto \\exp(-(\\mu-2a+b)t)$ is monotonously decreasing and we get\r\n\t\\begin{align*}\r\n\t\t\\sum_{i=1}^{n}  \\gamma_{i}  \\exp(-(\\mu-2a+b)t_i) &\\le \\int_{0}^{t_n} \\exp(-(\\mu-2a+b)t) \\, dt.\r\n\t\\end{align*}\r\n\tThus,\r\n\t$$\r\n\t\\E[\\1_{\\IA_{n-1}} f(X_{n})] \\le\\exp((-a+\\eps'+\\eps'')t_n ) \\Bigl( \\E[\\1_{\\IA_0} f(X_0)] + \\frac{\\E[\\1_{\\IA_0}E_0]}{\\mu-2a+b}  \\Bigr).\r\n\t$$\r\n\tFor $a>\\mu-a+b$,  the function $t \\mapsto \\exp(-(\\mu-2a+b)t)$ is monotonously increasing and we get\r\n\t\\begin{align*}\r\n\t\t\\sum_{i=1}^{n}  &\\gamma_{i}  \\exp(-(\\mu-2a+b)t_i) \\le \\int_{0}^{t_n} \\exp(-(\\mu-2a+b)(t+\\bar \\gamma)) \\, dt.\r\n\t\\end{align*}\r\n\tThus,\r\n\t\\begin{align*}\r\n\t\t\\E[\\1_{\\IA_{n-1}} f(X_{n})] \\le&\\exp((-(\\mu-a+b)+\\eps'+\\eps'')t_n ) \\Bigl( \\E[\\1_{\\IA_0} f(X_0)] \\\\\r\n\t\t& \\qquad \\qquad \\qquad  +\\frac{\\E[\\1_{\\IA_0}E_0]}{2a-\\mu-b} \\bigl(\\exp(-(\\mu-2a+b)\\bar \\gamma)  \\Bigr).\r\n\t\\end{align*}\r\n\tLastly, for $a=\\mu-a+b$ we get \r\n\t$\r\n\t\\sum_{i=1}^{n}  \\gamma_{i}  \\exp(-(\\mu-2a+b)t_i)=t_n\r\n\t$\r\n\tand, thus,\r\n\t$$\r\n\t\\E[\\1_{\\IA_{n-1}}f(X_n)] \\le \\exp((-a+\\eps'+\\eps'')t_n ) \\Bigl( \\E[ \\1_{\\IA_0}f(X_0)] + t_n\\E[\\1_{\\IA_0}E_0] \\Bigr).\r\n\t$$\r\n\tNote that $\\exp(-\\eps''' t)t \\to 0$ for all $\\eps'''>0$. Therefore, there exists a constant $C'>0$ such that\r\n\t$$\r\n\t\\E[\\1_{\\IA_{n-1}}f(X_{n})] \\le \\exp((-a+\\eps'+\\eps''+\\eps''')t_n ) \\Bigl( \\E[\\1_{\\IA_0} f(X_0)] + C'\\E[\\1_{\\IA_0}E_0] \\Bigr).\r\n\t$$\r\n\tThe proof of the first assertion follows by choosing $\\eps'=\\eps''= \\eps''' =\\frac 13 \\eps$.\r\n\t\r\n\tFor the second assertion, note that, by Lemma~\\ref{rem:Lipschitz}, we have $|\\grad f(x)|^2 \\le 2 C_L f(x)$ for all $x \\in \\R^d$\r\n\tand, using that $f(X_n)\\ge 0$, we get by the Cauchy-Schwarz inequality\r\n\t\\begin{align*}\r\n\t\t\\frac b2 |V_n|^2 &\\le E_n - \\langle \\grad f(X_n), V_n \\rangle \\le E_n + |\\grad f(X_n)| \\, |V_n|,\r\n\t\\end{align*}\r\n\twhere $E_n$ is defined by \\eqref{eq:Lyapunov}.\r\n\tThus, using Young's inequality, \r\n\t\\begin{align} \\label{eq:Yconv2}\r\n\t\t\\frac b2 \\E\\bigl[\\1_{\\IA_{n-1}}  |V_n|^2\\bigr]\r\n\t\t&\\le \\E[\\1_{\\IA_{n-1}} E_n] + \\frac 1b \\E[\\1_{\\IA_{n-1}} |\\grad f(X_n)|^2] + \\frac b4\\E[ \\1_{\\IA_{n-1}} |V_n|^2] \r\n\t\\end{align}\r\n\tand, with the bound for $\\E[\\1_{\\IA_{n-1}} f(X_n)]$ and \\eqref{eq:2776362562}, we get a constant $C\\ge 0$ such that \r\n\t\\begin{align} \\label{eq:Yconv} \r\n\t\t\\E[\\1_{\\IA_{n-1}}|V_n|^2]^{1/2} \\le C \\exp\\Bigl(-\\frac 12 (m -\\eps)t_n\\Bigr).\r\n\t\\end{align}\r\n\t\r\n\t(ii): Let $m'<m-\\eps$. For $\\eps' >0$ and $n \\in \\N$ consider the set $\\IB_n = \\IA_\\infty \\cap \\{\\sup_{i \\ge n} \\exp(m' t_i) f(X_i) \\ge \\eps' \\}$.\r\n\tWith the Markov inequality and (i) there exists a $C>0$ such that\r\n\t\\begin{align*}\r\n\t\t\\P(\\IB_n) &\\le \\sum_{i=n}^\\infty \\P\\bigl(\\IA_{i-1} \\cap \\bigl\\{\\exp(m' t_i) f(X_i) \\ge \\frac{\\eps'}{2}\\bigr\\}\\bigr) \\\\\r\n\t\t& \\le \\sum_{i=n}^\\infty \\frac{2\\exp(m' t_i)}{\\eps'} \\E[\\1_{\\IA_{i-1}} f(X_i)] \\le \\frac{C}{\\eps'} \\sum_{i=n}^\\infty \\exp((m'-(m-\\eps)) t_i) \\overset{n \\to \\infty}{\\longrightarrow} 0.\r\n\t\\end{align*}\r\n\tWith\r\n\t$$\r\n\t\\P\\Bigl(\\IA_\\infty \\cap \\big\\{\\limsup_{n \\to \\infty} \\exp(m' t_n)f(X_n) \\ge \\eps'\\big\\}\\Bigr) \\le \\P\\Bigl(\\bigcap_{n \\in \\N} B_n\\Bigr) =0\r\n\t$$\r\n\twe get $\\exp(m' t_n)f(X_n)\\to 0$ almost surely on $\\IA_\\infty$.\r\n\t\r\n\t(iii):\r\n\tWe consider the event $\\IA_\\infty$ and bound the distance that the process $(X_n)_{n \\in \\N_0}$ travels. Since $\\eps < m$ the mapping $t  \\mapsto \\exp(-\\frac 12 (m -\\eps)t)$ is monotonously decreasing. Thus, using \\eqref{eq:Yconv} we get\r\n\t\\begin{align*}\r\n\t\t\\E\\Bigl[\\1_{\\IA_\\infty} \\sum_{i=1}^\\infty |X_i-X_{i-1}|\\Bigr] &\\le   \\sum_{i=1}^\\infty \\gamma_i \\E[\\1_{\\IA_{i-1}}  |V_i|^2]^{1/2} \\le C \\sum_{i=1}^\\infty \\gamma_i  \\exp\\Bigl(-\\frac 12 (m -\\eps)t_i\\Bigr)\\\\\r\n\t\t&\\le C \\int_0^\\infty \\exp\\Bigl(-\\frac 12 (m -\\eps)t\\Bigr) \\, dt < \\infty,\r\n\t\\end{align*}\r\n\twhich implies that $\\sum_{i=1}^\\infty |X_i-X_{i-1}|$ is almost surely finite, on $\\IA_\\infty$. Thus, $(X_n)_{n \\in \\N_0}$ almost surely converges on $\\IA_{\\infty}$.\r\n\\end{proof}","sha256_utf8":"1a7d915f4bf039d1016cc97c8101793bd97a5edfe0fc68f2c4ae928f0caa3e4f"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"控制目标函数递推的乘积因子","source_units":[{"start":60867,"end":61219,"start_line":881,"end_line":885,"raw_tex":"\t(i): \r\n\tFirst, note that, for all $x\\in \\R$, $1+x\\le \\exp(x)$ and $(1+x)^{-1}=e^{-x+o(x)}$. Thus, for every $\\eps'>0$ there exists a $\\bar \\gamma'>0$ such that if $\\max_{i=1, \\dots, n} \\gamma_i \\le \\bar \\gamma'$ we have\r\n\t$$\r\n\t\\prod_{i=1}^{n} (1+a\\gamma_{i})^{-1} \\le \\Bigl(\\prod_{i=1}^{n} \\exp((-a+\\eps')\\gamma_i)\\Bigr) = \\exp((-a+\\eps')t_n ).\r\n\t$$\r\n","sha256_utf8":"4ae6e78051caa3cfe3d98a3b8b1306e556b32eba98794416640f25709a7eff91"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"控制能量递推的乘积因子","source_units":[{"start":61219,"end":61586,"start_line":886,"end_line":889,"raw_tex":"\tMoreover, for $(\\delta_n)_{n \\in \\N}$ given in \\eqref{eq:alpha} we have $\\delta_n = 1-\\gamma_n(\\mu-a+b)+o(\\gamma_n)$ so that, for all $\\eps''>0$, there exists a $\\bar \\gamma ''>0$ such that if $\\max_{i=1, \\dots, n} \\gamma_i \\le \\bar \\gamma''$ we have $\\delta_i\\ge 0$ for all $i=1, \\dots, n$ and\r\n\t$$\r\n\t\\prod_{i=1}^{n}\\delta_{i} \\le \\exp(-(\\mu-a+b-\\eps'')t_n).\r\n\t$$\r\n","sha256_utf8":"4c7157e166c34110c12e13617d674baa9f334d2ad78cc7ff1ab7a28c6df18edb"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"验证递推条件并得到指数和","source_units":[{"start":61586,"end":62418,"start_line":890,"end_line":899,"raw_tex":"\tNote that $\\beta_n = -\\gamma_n + o(\\gamma_n)$, $\\alpha_n-a\\delta_n+2\\beta_n L =  \\gamma_n (a\\mu-a^2+ab-2 L)+o(\\gamma_n)$, \r\n\t$$\r\n\t\\epsilon_n-\\frac b2 \\delta_n =  \\gamma_n \\Bigl(C_L-\\frac{b}{2}(\\mu+a-b)\\Bigr) +o(\\gamma_n) \\;  \\text{ and } \\; \\sfrac {C_L}{2}\\gamma^2-\\sfrac{b}{2}\\gamma=-\\sfrac{b}{2}\\gamma+o(\\gamma),\r\n\t$$\r\n\tand using assumption \\eqref{eq:238877} we can choose $\\bar \\gamma \\le \\min(\\bar \\gamma',\\bar \\gamma'')$ sufficiently small such that if $\\sup_{n \\in \\N}\\gamma_n \\le \\bar \\gamma$ all of the above terms are strictly negative. Then, using Proposition~\\ref{prop:SHB} we get that for all $n \\in \\N$\r\n\t\\begin{align*}\r\n\t\t\\E[\\1_{\\IA_{n-1}} f(X_{n})] \r\n\t\t\\le \\exp((&-a+\\eps'+\\eps'')t_n )  \\\\\r\n\t\t& \\Bigl( \\E[ \\1_{\\IA_0}f(X_0)]+ \\sum_{i=1}^{n}  \\gamma_{i}  \\exp(-(\\mu-2a+b)t_i) \\E[\\1_{\\IA_0}E_0] \\Bigr).\r\n\t\\end{align*}\r\n","sha256_utf8":"6d4b5c1ac214cce93488023bf74cd968d52ef63d8c8b857da6b1aacb27dea8c1"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"目标函数主导衰减的情形","source_units":[{"start":62418,"end":62803,"start_line":900,"end_line":907,"raw_tex":"\tIf $a< \\mu-a+b$, the function $t \\mapsto \\exp(-(\\mu-2a+b)t)$ is monotonously decreasing and we get\r\n\t\\begin{align*}\r\n\t\t\\sum_{i=1}^{n}  \\gamma_{i}  \\exp(-(\\mu-2a+b)t_i) &\\le \\int_{0}^{t_n} \\exp(-(\\mu-2a+b)t) \\, dt.\r\n\t\\end{align*}\r\n\tThus,\r\n\t$$\r\n\t\\E[\\1_{\\IA_{n-1}} f(X_{n})] \\le\\exp((-a+\\eps'+\\eps'')t_n ) \\Bigl( \\E[\\1_{\\IA_0} f(X_0)] + \\frac{\\E[\\1_{\\IA_0}E_0]}{\\mu-2a+b}  \\Bigr).\r\n\t$$\r\n","sha256_utf8":"a35dc2ce312872f0b432faf809a08dc6ffc3e3d09dbb694d3ef18f5c0c0e51f9"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"能量主导衰减的情形","source_units":[{"start":62803,"end":63299,"start_line":908,"end_line":916,"raw_tex":"\tFor $a>\\mu-a+b$,  the function $t \\mapsto \\exp(-(\\mu-2a+b)t)$ is monotonously increasing and we get\r\n\t\\begin{align*}\r\n\t\t\\sum_{i=1}^{n}  &\\gamma_{i}  \\exp(-(\\mu-2a+b)t_i) \\le \\int_{0}^{t_n} \\exp(-(\\mu-2a+b)(t+\\bar \\gamma)) \\, dt.\r\n\t\\end{align*}\r\n\tThus,\r\n\t\\begin{align*}\r\n\t\t\\E[\\1_{\\IA_{n-1}} f(X_{n})] \\le&\\exp((-(\\mu-a+b)+\\eps'+\\eps'')t_n ) \\Bigl( \\E[\\1_{\\IA_0} f(X_0)] \\\\\r\n\t\t& \\qquad \\qquad \\qquad  +\\frac{\\E[\\1_{\\IA_0}E_0]}{2a-\\mu-b} \\bigl(\\exp(-(\\mu-2a+b)\\bar \\gamma)  \\Bigr).\r\n\t\\end{align*}\r\n","sha256_utf8":"8377d1358675b0ab96fff789a2b120ad2b5284912f8dc1519d9efd10407e86c1"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"重合速率情形与误差分配","source_units":[{"start":63299,"end":63882,"start_line":917,"end_line":929,"raw_tex":"\tLastly, for $a=\\mu-a+b$ we get \r\n\t$\r\n\t\\sum_{i=1}^{n}  \\gamma_{i}  \\exp(-(\\mu-2a+b)t_i)=t_n\r\n\t$\r\n\tand, thus,\r\n\t$$\r\n\t\\E[\\1_{\\IA_{n-1}}f(X_n)] \\le \\exp((-a+\\eps'+\\eps'')t_n ) \\Bigl( \\E[ \\1_{\\IA_0}f(X_0)] + t_n\\E[\\1_{\\IA_0}E_0] \\Bigr).\r\n\t$$\r\n\tNote that $\\exp(-\\eps''' t)t \\to 0$ for all $\\eps'''>0$. Therefore, there exists a constant $C'>0$ such that\r\n\t$$\r\n\t\\E[\\1_{\\IA_{n-1}}f(X_{n})] \\le \\exp((-a+\\eps'+\\eps''+\\eps''')t_n ) \\Bigl( \\E[\\1_{\\IA_0} f(X_0)] + C'\\E[\\1_{\\IA_0}E_0] \\Bigr).\r\n\t$$\r\n\tThe proof of the first assertion follows by choosing $\\eps'=\\eps''= \\eps''' =\\frac 13 \\eps$.\r\n","sha256_utf8":"565ab2160b4b5bad8af0b5c0e4a6d1ff7006bd0c2429cc59f81765de6c990e2a"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"由能量控制速度二阶矩","source_units":[{"start":63885,"end":64519,"start_line":931,"end_line":941,"raw_tex":"\tFor the second assertion, note that, by Lemma~\\ref{rem:Lipschitz}, we have $|\\grad f(x)|^2 \\le 2 C_L f(x)$ for all $x \\in \\R^d$\r\n\tand, using that $f(X_n)\\ge 0$, we get by the Cauchy-Schwarz inequality\r\n\t\\begin{align*}\r\n\t\t\\frac b2 |V_n|^2 &\\le E_n - \\langle \\grad f(X_n), V_n \\rangle \\le E_n + |\\grad f(X_n)| \\, |V_n|,\r\n\t\\end{align*}\r\n\twhere $E_n$ is defined by \\eqref{eq:Lyapunov}.\r\n\tThus, using Young's inequality, \r\n\t\\begin{align} \\label{eq:Yconv2}\r\n\t\t\\frac b2 \\E\\bigl[\\1_{\\IA_{n-1}}  |V_n|^2\\bigr]\r\n\t\t&\\le \\E[\\1_{\\IA_{n-1}} E_n] + \\frac 1b \\E[\\1_{\\IA_{n-1}} |\\grad f(X_n)|^2] + \\frac b4\\E[ \\1_{\\IA_{n-1}} |V_n|^2] \r\n\t\\end{align}\r\n","sha256_utf8":"ba81636a69aaceffe0d9177152a9a845ee512a1ca2beb45876aaa500fe38df9a"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"得到速度的指数衰减","source_units":[{"start":64519,"end":64767,"start_line":942,"end_line":945,"raw_tex":"\tand, with the bound for $\\E[\\1_{\\IA_{n-1}} f(X_n)]$ and \\eqref{eq:2776362562}, we get a constant $C\\ge 0$ such that \r\n\t\\begin{align} \\label{eq:Yconv} \r\n\t\t\\E[\\1_{\\IA_{n-1}}|V_n|^2]^{1/2} \\le C \\exp\\Bigl(-\\frac 12 (m -\\eps)t_n\\Bigr).\r\n\t\\end{align}\r\n","sha256_utf8":"37f5337d5e169bc98d9544628931a786d8cea2bd4f0e815410e2ee8a73946e18"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"尾事件估计推出几乎处处衰减","source_units":[{"start":64770,"end":65550,"start_line":947,"end_line":957,"raw_tex":"\t(ii): Let $m'<m-\\eps$. For $\\eps' >0$ and $n \\in \\N$ consider the set $\\IB_n = \\IA_\\infty \\cap \\{\\sup_{i \\ge n} \\exp(m' t_i) f(X_i) \\ge \\eps' \\}$.\r\n\tWith the Markov inequality and (i) there exists a $C>0$ such that\r\n\t\\begin{align*}\r\n\t\t\\P(\\IB_n) &\\le \\sum_{i=n}^\\infty \\P\\bigl(\\IA_{i-1} \\cap \\bigl\\{\\exp(m' t_i) f(X_i) \\ge \\frac{\\eps'}{2}\\bigr\\}\\bigr) \\\\\r\n\t\t& \\le \\sum_{i=n}^\\infty \\frac{2\\exp(m' t_i)}{\\eps'} \\E[\\1_{\\IA_{i-1}} f(X_i)] \\le \\frac{C}{\\eps'} \\sum_{i=n}^\\infty \\exp((m'-(m-\\eps)) t_i) \\overset{n \\to \\infty}{\\longrightarrow} 0.\r\n\t\\end{align*}\r\n\tWith\r\n\t$$\r\n\t\\P\\Bigl(\\IA_\\infty \\cap \\big\\{\\limsup_{n \\to \\infty} \\exp(m' t_n)f(X_n) \\ge \\eps'\\big\\}\\Bigr) \\le \\P\\Bigl(\\bigcap_{n \\in \\N} B_n\\Bigr) =0\r\n\t$$\r\n\twe get $\\exp(m' t_n)f(X_n)\\to 0$ almost surely on $\\IA_\\infty$.\r\n","sha256_utf8":"8ef9f09be20690dc1639847cee80a4d0cf1b2a94ef589c006ca24e3606800aec"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"有限路径长度推出迭代收敛","source_units":[{"start":65553,"end":66286,"start_line":959,"end_line":965,"raw_tex":"\t(iii):\r\n\tWe consider the event $\\IA_\\infty$ and bound the distance that the process $(X_n)_{n \\in \\N_0}$ travels. Since $\\eps < m$ the mapping $t  \\mapsto \\exp(-\\frac 12 (m -\\eps)t)$ is monotonously decreasing. Thus, using \\eqref{eq:Yconv} we get\r\n\t\\begin{align*}\r\n\t\t\\E\\Bigl[\\1_{\\IA_\\infty} \\sum_{i=1}^\\infty |X_i-X_{i-1}|\\Bigr] &\\le   \\sum_{i=1}^\\infty \\gamma_i \\E[\\1_{\\IA_{i-1}}  |V_i|^2]^{1/2} \\le C \\sum_{i=1}^\\infty \\gamma_i  \\exp\\Bigl(-\\frac 12 (m -\\eps)t_i\\Bigr)\\\\\r\n\t\t&\\le C \\int_0^\\infty \\exp\\Bigl(-\\frac 12 (m -\\eps)t\\Bigr) \\, dt < \\infty,\r\n\t\\end{align*}\r\n\twhich implies that $\\sum_{i=1}^\\infty |X_i-X_{i-1}|$ is almost surely finite, on $\\IA_\\infty$. Thus, $(X_n)_{n \\in \\N_0}$ almost surely converges on $\\IA_{\\infty}$.\r\n","sha256_utf8":"87666f39aad0ac51658ee2ae3bb901582a37e8565bf37a015ece43efc1a34912"}]},{"id":"S044","semantic_type":"proposition","kind":"proposition","title":"局部能量递推与目标函数界","source_units":[{"start":51476,"end":52834,"start_line":737,"end_line":750,"raw_tex":"\\begin{proposition} \\label{prop:SHB}\r\n\tLet $L,a,b>0, \\sigma \\ge 0$. Let $(\\IA_n)_{n \\in \\N_0}$ be a decreasing sequence of events such that, for all $n \\in \\N_0$, $\\IA_n \\in \\cF_n$ and on $\\IA_n$ it holds that\r\n\t\\begin{align} \\label{eq:1293182}\r\n\t\t|\\grad f(X_n)|^2 \\ge 2L f(X_n) \\quad \\text{and} \\quad \\E[|D_{n+1}|^2 | \\cF_n] \\le \\sigma f(X_n).\r\n\t\\end{align}\r\n\tLet $(\\alpha_n)_{n \\in \\N}, (\\beta_n)_{n \\in \\N}, (\\delta_n)_{n \\in \\N}$ and $(\\epsilon_n)_{n \\in \\N}$ be given by $\\eqref{eq:alpha}$. Assume that $(\\beta_n)_{n \\in \\N}$, $(\\alpha_{n+1}-a\\delta_{n+1}+2\\beta_{n+1}L)_{n \\in \\N_0}$, $(\\epsilon_{n+1}-\\frac{b}{2} \\delta_{n+1})_{n \\in \\N_0}$ and $(\\frac{C_L}{2} \\gamma_{n}^2- \\frac{b}{2}\\gamma_{n})_{n \\in \\N_0}$ are sequences of non-positive reals and $(\\delta_n)_{n \\in \\N}$ is a sequence of non-negative reals.\r\n\tMoreover, if $\\P(\\bigcap_{n \\in \\N_0} \\IA_n)<\\P(\\IA_0)$ additionally assume that $ab \\ge C_L$.\r\n\tThen, for all $n \\in \\N$ it holds that\r\n\t\\begin{align}\\begin{split} \\label{eq:238477}\r\n\t\t\t& \\E[\\1_{\\IA_{n-1}} f(X_{n})]\\le \\Bigl(\\prod_{i=1}^{n} (1+a\\gamma_{i})^{-1}\\Bigr) \\\\\r\n\t\t\t& \\qquad \\qquad \\qquad \\Bigl( \\E[\\1_{\\IA_0} f(X_0)] + \\sum_{i=1}^{n}  \\frac{\\gamma_{i}}{1+a\\gamma_{i}} \\Bigl(\\prod_{j=1}^{i} (1+a\\gamma_{j})\\Bigr) \\Bigl(\\prod_{j=1}^{i}\\delta_{j} \\Bigr)\\E[\\1_{\\IA_0}E_0] \\Bigr).\r\n\t\t\\end{split} \r\n\t\\end{align}\t\r\n\\end{proposition}","sha256_utf8":"f970d882ffe7e71481b24745627a10b1bdb41cd88050ba94bc551c47332a336e"}]},{"id":"S045","semantic_type":"proof","kind":"proof","title":"能量递推命题完整证明","source_units":[{"start":52838,"end":58316,"start_line":752,"end_line":837,"raw_tex":"\\begin{proof}\r\n\tIn a first step, we derive a convergence rate for the expectation of the Lyapunov process $(E_n)_{n \\in \\N_0}$. For this, we consider the time evolution of the three summands in \\eqref{eq:Lyapunov}, separately.\r\n\t\r\n\tFirst, we look at the evolution of $(f(X_{n}))_{n \\in \\N_0}$.\r\n\tLet $x,y \\in \\R^d$ and note that with the Lipschitz-continuity of $\\grad f$ we get\r\n\t\\begin{align}\r\n\t\t\\begin{split}\r\n\t\t\t\\label{eq:Taylorp1}\r\n\t\t\tf(y)&\\le f(x) + \\langle \\grad f(x),y-x\\rangle  + \\frac{C_L}{2} |y-x|^2.\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n\t\r\n\tNow, for $n \\in \\N_0$ we use \\eqref{eq:Taylorp1} with $x=X_{n}$ and $y=X_{n+1}$ and \\eqref{eq:MSGDintro} to get \r\n\t\\begin{align} \\begin{split} \r\n\t\t\t\\label{eq:273856230}\r\n\t\t\t&\\E[\\1_{\\IA_n}f(X_{n+1})]\\le \\E\\Big[\\1_{\\IA_n}\\bigl(f(X_{n})-\\gamma_{n+1}^2 |\\grad f(X_n)|^2 \\\\\r\n\t\t\t&\\qquad \\qquad \\qquad \\qquad \\quad + (\\gamma_{n+1}-\\gamma_{n+1}^2\\mu) \\langle \\grad f(X_{n}),V_n\\rangle + \\frac{C_L}{2} \\gamma_{n+1}^2 \\bigl|V_{n+1}\\bigr|^2 \\bigr) \\Big].\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n\t\r\n\tNext, we control the evolution of $(|V_n|^2)_{n \\in \\N_0}$. Using \\eqref{eq:1293182}, we get\r\n\t\\begin{align} \\begin{split}\r\n\t\t\t\\E[\\1_{\\IA_n}|V_{n+1}|^2]\r\n\t\t\t\\le  \\E[\\1_{\\IA_n}(&(1-2\\gamma_{n+1}\\mu+\\gamma_{n+1}^2\\mu^2)|V_n|^2+ \\gamma_{n+1}^2 |\\grad f(X_n)|^2  \\\\\r\n\t\t\t&  - 2 (\\gamma_{n+1}-\\gamma_{n+1}^2\\mu) \\langle V_n, \\grad f(X_n)\\rangle  + \\gamma_{n+1}^2 \\sigma f(X_n))].\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n\tLastly,\r\n\t\\begin{align} \\begin{split} \\label{eq:1764443}\r\n\t\t\t\\E[\\1_{\\IA_n} &\\langle \\grad f(X_{n+1}), V_{n+1} \\rangle] = \\E[ \\1_{\\IA_n} (\\langle \\grad f(X_{n}), V_{n+1} \\rangle + \\langle \\grad f(X_{n+1})- \\grad f(X_{n}), V_{n+1} \\rangle )] \\\\\r\n\t\t\t&\\le \\E[\\1_{\\IA_n} ((1-\\gamma_{n+1}\\mu) \\langle \\grad f(X_n), V_n \\rangle - \\gamma_{n+1}  |\\grad f(X_n)|^2 + C_L \\gamma_{n+1} |V_{n+1}|^2)].\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n\t\r\n\tCombining the estimates \\eqref{eq:273856230}-\\eqref{eq:1764443}, we obtain\r\n\t\\begin{align*}\r\n\t\t&\\E[\\1_{\\IA_{n}} E_{n+1}] \\\\\r\n\t\t&\\le \\E[\\1_{\\IA_{n}}(\\alpha_{n+1} f(X_n) + \\beta_{n+1} |\\grad f(X_n)|^2 +\\delta_{n+1} \\langle \\grad f(X_n),V_n \\rangle + \\epsilon_{n+1} |V_n|^2)],\r\n\t\\end{align*}\r\n\twhere\r\n\t\\begin{align} \\label{eq:alpha}\r\n\t\t\\begin{split}\r\n\t\t\t\\alpha_{n+1} = &a + \\Bigl(\\frac{b}{2}+C_L \\gamma_{n+1}\\Bigl(1+\\frac{\\gamma_{n+1}a}{2}\\Bigr)\\Bigr)\\gamma_{n+1}^2 \\sigma, \\\\\r\n\t\t\t\\beta_{n+1} = &- \\gamma_{n+1} - a\\gamma_{n+1}^2 + \\Bigl(\\frac{b}{2}+C_L \\gamma_{n+1}\\Bigl(1+\\frac{\\gamma_{n+1}a}{2}\\Bigr)\\Bigr)\\gamma_{n+1}^2,\\\\\r\n\t\t\t\\delta_{n+1} = &1-\\mu\\gamma_{n+1}+ a(\\gamma_{n+1}-\\gamma_{n+1}^2\\mu) \\\\\r\n\t\t\t&- 2\\Bigl(\\frac{b}{2}+C_L \\gamma_{n+1}\\Bigl(1+\\frac{\\gamma_{n+1}a}{2}\\Bigr)\\Bigr) (\\gamma_{n+1}-\\gamma_{n+1}^2\\mu), \\\\\r\n\t\t\t\\epsilon_{n+1} = &\\Bigl(\\frac{b}{2}+C_L \\gamma_{n+1}\\Bigl(1+\\frac{\\gamma_{n+1}a}{2}\\Bigr)\\Bigr) (1-2\\gamma_{n+1}\\mu+\\gamma_{n+1}^2\\mu^2).\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n\tBy definition\r\n\t$\r\n\t\\E[\\1_{\\IA_n} \\langle \\grad f(X_n), V_n \\rangle ] =  \\E[\\1_{\\IA_n}(E_n -a f(X_n)-\\sfrac{b}{2} |V_n|^2)],\r\n\t$\r\n\tso that, using the PL-inequality \\eqref{eq:1293182} and the fact that $(\\beta_n)_{n \\in \\N}$ is non-positive, we get \r\n\t\\begin{align} \\begin{split} \\label{eq:579999663}\r\n\t\t\t&\\E[\\1_{\\IA_{n}}E_{n+1}] \\\\\r\n\t\t\t&\\le \\E[\\1_{\\IA_n}(\\delta_{n+1}E_n + (\\alpha_{n+1}-a\\delta_{n+1}+2\\beta_{n+1}L) f(X_n)  + (\\epsilon_{n+1}-\\frac{b}{2} \\delta_{n+1}) |V_n|^2)].\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n\tWith the assumptions on $(\\alpha_{n+1}-a\\delta_{n+1}+2\\beta_{n+1}L)_{n \\in \\N_0}$ and $(\\epsilon_{n+1}-\\frac{b}{2} \\delta_{n+1})_{n \\in \\N_0}$ we have\r\n\t$\r\n\t\\E[\\1_{\\IA_{n}} E_{n+1}] \\le \\delta_{n+1} \\E[\\1_{\\IA_{n}} E_n].\r\n\t$\r\n\tFor $n\\in \\N$, we use Lemma~\\ref{lem:Epositive} and the monotonicity of $(\\IA_n)_{n \\in \\N_0}$ in order to show that $\\E[\\1_{\\IA_{n}} E_n]\\le \\E[\\1_{\\IA_{n-1}} E_n]$ so that, iteratively, \r\n\t\\begin{align} \\label{eq:2776362562}\r\n\t\t\\E[\\1_{\\IA_{n-1}} E_{n}] \\le \\Bigl( \\prod_{i=1}^{n}\\delta_{i} \\Bigr)\\E[\\1_{\\IA_{0}} E_0].\r\n\t\\end{align}\r\n\tNext, we bound the expectation of $(f(X_n))_{n \\in \\N_0}$ using \\eqref{eq:2776362562}. Analogously to \\eqref{eq:273856230}, we have for all $n \\in \\N_0$ that\r\n\t\\begin{align*}\r\n\t\t\\E[\\1_{\\IA_{n}}f(X_{n+1})]\\le \\E\\Big[\\1_{\\IA_{n}} \\bigl( f(X_{n})+\\gamma_{n+1}\\langle \\grad f(X_{n+1}),V_{n+1}\\rangle + \\frac{C_L}{2} \\gamma_{n+1}^2 \\bigl|V_{n+1}\\bigr|^2\\bigr) \\Big]\r\n\t\\end{align*}\r\n\tso that, by definition of $E_{n+1}$,\r\n\t\\begin{align*} \r\n\t\t&(1+a\\gamma_{n+1})\\E[\\1_{\\IA_{n}}f(X_{n+1})]\\\\\r\n\t\t&\\le \\E\\Big[ \\1_{\\IA_{n}} \\bigl(f(X_{n})+\\gamma_{n+1} E_{n+1} + (\\frac{C_L}{2} \\gamma_{n+1}^2- \\frac{b}{2}\\gamma_{n+1}) \\bigl|V_{n+1}\\bigr|^2 \\bigr)\\Big].\r\n\t\\end{align*}\r\n\tBy assumption, $(\\frac{C_L}{2} \\gamma_{n}^2- \\frac{b}{2}\\gamma_{n})_{n \\in \\N}$ is a sequence of non-positive reals. Therefore, we can neglect the last term in the upper bound above and get\r\n\t\\begin{align} \\label{eq:21378861}\r\n\t\t\\E[\\1_{\\IA_{n}}f(X_{n+1})] \\le (1+a\\gamma_{n+1})^{-1} \\E[\\1_{\\IA_n} f(X_{n})]+\\frac{\\gamma_{n+1}}{1+a\\gamma_{n+1}} \\Bigl( \\prod_{i=1}^{n+1}\\delta_{i} \\Bigr)\\E[\\1_{\\IA_0}E_0].\r\n\t\\end{align}\r\n\tUsing the non-negativity of $f(X_n)$, the monotonicity of $(\\IA_n)_{n \\in \\N_0}$ and \\eqref{eq:21378861}, one can inductively show that, for all $n \\in \\N$,\r\n\t\\begin{align*}\r\n\t\t&\\E[\\1_{\\IA_{n-1}} f(X_{n})] \\\\\r\n\t\t&\\le \\Bigl(\\prod_{i=1}^{n} (1+a\\gamma_{i})^{-1}\\Bigr) \\Bigl( \\E[\\1_{\\IA_0} f(X_0)] + \\sum_{i=1}^{n}  \\frac{\\gamma_{i}}{1+a\\gamma_{i}} \\Bigl(\\prod_{j=1}^{i} (1+a\\gamma_{j})\\Bigr) \\Bigl(\\prod_{j=1}^{i}\\delta_{j} \\Bigr)\\E[\\1_{\\IA_0}E_0] \\Bigr).\r\n\t\\end{align*}\r\n\\end{proof}","sha256_utf8":"524f99a8c8d01407bbc7f04e570ca835bba0d7b7420ff11e6dee39773705a1b4"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"梯度光滑性给出泰勒上界","source_units":[{"start":52853,"end":53380,"start_line":753,"end_line":762,"raw_tex":"\tIn a first step, we derive a convergence rate for the expectation of the Lyapunov process $(E_n)_{n \\in \\N_0}$. For this, we consider the time evolution of the three summands in \\eqref{eq:Lyapunov}, separately.\r\n\t\r\n\tFirst, we look at the evolution of $(f(X_{n}))_{n \\in \\N_0}$.\r\n\tLet $x,y \\in \\R^d$ and note that with the Lipschitz-continuity of $\\grad f$ we get\r\n\t\\begin{align}\r\n\t\t\\begin{split}\r\n\t\t\t\\label{eq:Taylorp1}\r\n\t\t\tf(y)&\\le f(x) + \\langle \\grad f(x),y-x\\rangle  + \\frac{C_L}{2} |y-x|^2.\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n","sha256_utf8":"26f85ef9cabf733916fb66f89a6ee5d86f3501fc172900a14816ab9e41832655"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"估计一步目标函数演化","source_units":[{"start":53383,"end":53858,"start_line":764,"end_line":770,"raw_tex":"\tNow, for $n \\in \\N_0$ we use \\eqref{eq:Taylorp1} with $x=X_{n}$ and $y=X_{n+1}$ and \\eqref{eq:MSGDintro} to get \r\n\t\\begin{align} \\begin{split} \r\n\t\t\t\\label{eq:273856230}\r\n\t\t\t&\\E[\\1_{\\IA_n}f(X_{n+1})]\\le \\E\\Big[\\1_{\\IA_n}\\bigl(f(X_{n})-\\gamma_{n+1}^2 |\\grad f(X_n)|^2 \\\\\r\n\t\t\t&\\qquad \\qquad \\qquad \\qquad \\quad + (\\gamma_{n+1}-\\gamma_{n+1}^2\\mu) \\langle \\grad f(X_{n}),V_n\\rangle + \\frac{C_L}{2} \\gamma_{n+1}^2 \\bigl|V_{n+1}\\bigr|^2 \\bigr) \\Big].\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n","sha256_utf8":"f6bfc43376a6f27c28d1af5c0e5acff98bb03adde7317ab26a10a31931a01bf8"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"估计一步速度二阶矩","source_units":[{"start":53861,"end":54266,"start_line":772,"end_line":778,"raw_tex":"\tNext, we control the evolution of $(|V_n|^2)_{n \\in \\N_0}$. Using \\eqref{eq:1293182}, we get\r\n\t\\begin{align} \\begin{split}\r\n\t\t\t\\E[\\1_{\\IA_n}|V_{n+1}|^2]\r\n\t\t\t\\le  \\E[\\1_{\\IA_n}(&(1-2\\gamma_{n+1}\\mu+\\gamma_{n+1}^2\\mu^2)|V_n|^2+ \\gamma_{n+1}^2 |\\grad f(X_n)|^2  \\\\\r\n\t\t\t&  - 2 (\\gamma_{n+1}-\\gamma_{n+1}^2\\mu) \\langle V_n, \\grad f(X_n)\\rangle  + \\gamma_{n+1}^2 \\sigma f(X_n))].\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n","sha256_utf8":"7af04074b088e39a4a5a70dda07b5796dcf1c7c439b8340709f71b19d608872f"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"估计梯度速度交叉项","source_units":[{"start":54266,"end":54685,"start_line":779,"end_line":784,"raw_tex":"\tLastly,\r\n\t\\begin{align} \\begin{split} \\label{eq:1764443}\r\n\t\t\t\\E[\\1_{\\IA_n} &\\langle \\grad f(X_{n+1}), V_{n+1} \\rangle] = \\E[ \\1_{\\IA_n} (\\langle \\grad f(X_{n}), V_{n+1} \\rangle + \\langle \\grad f(X_{n+1})- \\grad f(X_{n}), V_{n+1} \\rangle )] \\\\\r\n\t\t\t&\\le \\E[\\1_{\\IA_n} ((1-\\gamma_{n+1}\\mu) \\langle \\grad f(X_n), V_n \\rangle - \\gamma_{n+1}  |\\grad f(X_n)|^2 + C_L \\gamma_{n+1} |V_{n+1}|^2)].\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n","sha256_utf8":"a37b09ecc766fa03dffb6b0c50dc323385971ba17eff0935df2e57350fa34bb7"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"合并三项并定义递推系数","source_units":[{"start":54688,"end":55683,"start_line":786,"end_line":800,"raw_tex":"\tCombining the estimates \\eqref{eq:273856230}-\\eqref{eq:1764443}, we obtain\r\n\t\\begin{align*}\r\n\t\t&\\E[\\1_{\\IA_{n}} E_{n+1}] \\\\\r\n\t\t&\\le \\E[\\1_{\\IA_{n}}(\\alpha_{n+1} f(X_n) + \\beta_{n+1} |\\grad f(X_n)|^2 +\\delta_{n+1} \\langle \\grad f(X_n),V_n \\rangle + \\epsilon_{n+1} |V_n|^2)],\r\n\t\\end{align*}\r\n\twhere\r\n\t\\begin{align} \\label{eq:alpha}\r\n\t\t\\begin{split}\r\n\t\t\t\\alpha_{n+1} = &a + \\Bigl(\\frac{b}{2}+C_L \\gamma_{n+1}\\Bigl(1+\\frac{\\gamma_{n+1}a}{2}\\Bigr)\\Bigr)\\gamma_{n+1}^2 \\sigma, \\\\\r\n\t\t\t\\beta_{n+1} = &- \\gamma_{n+1} - a\\gamma_{n+1}^2 + \\Bigl(\\frac{b}{2}+C_L \\gamma_{n+1}\\Bigl(1+\\frac{\\gamma_{n+1}a}{2}\\Bigr)\\Bigr)\\gamma_{n+1}^2,\\\\\r\n\t\t\t\\delta_{n+1} = &1-\\mu\\gamma_{n+1}+ a(\\gamma_{n+1}-\\gamma_{n+1}^2\\mu) \\\\\r\n\t\t\t&- 2\\Bigl(\\frac{b}{2}+C_L \\gamma_{n+1}\\Bigl(1+\\frac{\\gamma_{n+1}a}{2}\\Bigr)\\Bigr) (\\gamma_{n+1}-\\gamma_{n+1}^2\\mu), \\\\\r\n\t\t\t\\epsilon_{n+1} = &\\Bigl(\\frac{b}{2}+C_L \\gamma_{n+1}\\Bigl(1+\\frac{\\gamma_{n+1}a}{2}\\Bigr)\\Bigr) (1-2\\gamma_{n+1}\\mu+\\gamma_{n+1}^2\\mu^2).\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n","sha256_utf8":"c6fae27c38a6fb74a34a85058ca239acf65001739dc65bed492dd137235f4f86"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"代入能量定义和梯度下界","source_units":[{"start":55683,"end":56193,"start_line":801,"end_line":810,"raw_tex":"\tBy definition\r\n\t$\r\n\t\\E[\\1_{\\IA_n} \\langle \\grad f(X_n), V_n \\rangle ] =  \\E[\\1_{\\IA_n}(E_n -a f(X_n)-\\sfrac{b}{2} |V_n|^2)],\r\n\t$\r\n\tso that, using the PL-inequality \\eqref{eq:1293182} and the fact that $(\\beta_n)_{n \\in \\N}$ is non-positive, we get \r\n\t\\begin{align} \\begin{split} \\label{eq:579999663}\r\n\t\t\t&\\E[\\1_{\\IA_{n}}E_{n+1}] \\\\\r\n\t\t\t&\\le \\E[\\1_{\\IA_n}(\\delta_{n+1}E_n + (\\alpha_{n+1}-a\\delta_{n+1}+2\\beta_{n+1}L) f(X_n)  + (\\epsilon_{n+1}-\\frac{b}{2} \\delta_{n+1}) |V_n|^2)].\r\n\t\t\\end{split}\r\n\t\\end{align}\r\n","sha256_utf8":"c2ed8f2a22ff9f9ace15734e1148db071ced2177ce850664abff905eece7301d"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"利用非负性迭代局部能量界","source_units":[{"start":56193,"end":56756,"start_line":811,"end_line":818,"raw_tex":"\tWith the assumptions on $(\\alpha_{n+1}-a\\delta_{n+1}+2\\beta_{n+1}L)_{n \\in \\N_0}$ and $(\\epsilon_{n+1}-\\frac{b}{2} \\delta_{n+1})_{n \\in \\N_0}$ we have\r\n\t$\r\n\t\\E[\\1_{\\IA_{n}} E_{n+1}] \\le \\delta_{n+1} \\E[\\1_{\\IA_{n}} E_n].\r\n\t$\r\n\tFor $n\\in \\N$, we use Lemma~\\ref{lem:Epositive} and the monotonicity of $(\\IA_n)_{n \\in \\N_0}$ in order to show that $\\E[\\1_{\\IA_{n}} E_n]\\le \\E[\\1_{\\IA_{n-1}} E_n]$ so that, iteratively, \r\n\t\\begin{align} \\label{eq:2776362562}\r\n\t\t\\E[\\1_{\\IA_{n-1}} E_{n}] \\le \\Bigl( \\prod_{i=1}^{n}\\delta_{i} \\Bigr)\\E[\\1_{\\IA_{0}} E_0].\r\n\t\\end{align}\r\n","sha256_utf8":"986e53c43686fe4bde7922e056b444509e3b1b23be19251771b546d2833d6b01"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"反向泰勒估计联系损失与能量","source_units":[{"start":56756,"end":57414,"start_line":819,"end_line":827,"raw_tex":"\tNext, we bound the expectation of $(f(X_n))_{n \\in \\N_0}$ using \\eqref{eq:2776362562}. Analogously to \\eqref{eq:273856230}, we have for all $n \\in \\N_0$ that\r\n\t\\begin{align*}\r\n\t\t\\E[\\1_{\\IA_{n}}f(X_{n+1})]\\le \\E\\Big[\\1_{\\IA_{n}} \\bigl( f(X_{n})+\\gamma_{n+1}\\langle \\grad f(X_{n+1}),V_{n+1}\\rangle + \\frac{C_L}{2} \\gamma_{n+1}^2 \\bigl|V_{n+1}\\bigr|^2\\bigr) \\Big]\r\n\t\\end{align*}\r\n\tso that, by definition of $E_{n+1}$,\r\n\t\\begin{align*} \r\n\t\t&(1+a\\gamma_{n+1})\\E[\\1_{\\IA_{n}}f(X_{n+1})]\\\\\r\n\t\t&\\le \\E\\Big[ \\1_{\\IA_{n}} \\bigl(f(X_{n})+\\gamma_{n+1} E_{n+1} + (\\frac{C_L}{2} \\gamma_{n+1}^2- \\frac{b}{2}\\gamma_{n+1}) \\bigl|V_{n+1}\\bigr|^2 \\bigr)\\Big].\r\n\t\\end{align*}\r\n","sha256_utf8":"30417b5ffcd9871ff32a23c27d2fce9314cf3a5336779cc6ce321503a36ee056"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"得到目标函数的一步递推","source_units":[{"start":57414,"end":57834,"start_line":828,"end_line":831,"raw_tex":"\tBy assumption, $(\\frac{C_L}{2} \\gamma_{n}^2- \\frac{b}{2}\\gamma_{n})_{n \\in \\N}$ is a sequence of non-positive reals. Therefore, we can neglect the last term in the upper bound above and get\r\n\t\\begin{align} \\label{eq:21378861}\r\n\t\t\\E[\\1_{\\IA_{n}}f(X_{n+1})] \\le (1+a\\gamma_{n+1})^{-1} \\E[\\1_{\\IA_n} f(X_{n})]+\\frac{\\gamma_{n+1}}{1+a\\gamma_{n+1}} \\Bigl( \\prod_{i=1}^{n+1}\\delta_{i} \\Bigr)\\E[\\1_{\\IA_0}E_0].\r\n\t\\end{align}\r\n","sha256_utf8":"f046f860fae954d7033e0ea64083003f70696b4c1ba0133967176624ac2bccf8"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"归纳得到完整乘积求和界","source_units":[{"start":57834,"end":58305,"start_line":832,"end_line":836,"raw_tex":"\tUsing the non-negativity of $f(X_n)$, the monotonicity of $(\\IA_n)_{n \\in \\N_0}$ and \\eqref{eq:21378861}, one can inductively show that, for all $n \\in \\N$,\r\n\t\\begin{align*}\r\n\t\t&\\E[\\1_{\\IA_{n-1}} f(X_{n})] \\\\\r\n\t\t&\\le \\Bigl(\\prod_{i=1}^{n} (1+a\\gamma_{i})^{-1}\\Bigr) \\Bigl( \\E[\\1_{\\IA_0} f(X_0)] + \\sum_{i=1}^{n}  \\frac{\\gamma_{i}}{1+a\\gamma_{i}} \\Bigl(\\prod_{j=1}^{i} (1+a\\gamma_{j})\\Bigr) \\Bigl(\\prod_{j=1}^{i}\\delta_{j} \\Bigr)\\E[\\1_{\\IA_0}E_0] \\Bigr).\r\n\t\\end{align*}\r\n","sha256_utf8":"fa6eba477cc09a17d46cc5e1fcdaf27b68d9f9456c128544619bbde64e298fdd"}]},{"id":"S056","semantic_type":"lemma","kind":"lemma","title":"李雅普诺夫函数的非负性","source_units":[{"start":50533,"end":50838,"start_line":716,"end_line":722,"raw_tex":"\\begin{lemma} \\label{lem:Epositive}\r\n\tLet $a, b, C_L>0$ and\r\n\t\\begin{align*}\r\n\t\tE(x,y) = a f(x) + \\langle \\grad f(x), y \\rangle + \\frac{b}{2} |y|^2.\r\n\t\\end{align*}\r\n\tIf $ab\\ge C_L$ then for all $x \\in \\R^d$ satisfying $2C_L f(x)\\ge |\\grad f(x)|^2$ we have $E(x,y)\\ge 0$, for all $y \\in \\R^d$.\r\n\\end{lemma}","sha256_utf8":"ea59f082829b35d294ac1566438591d1f2f958d2e6a190bb98b48fe1aa02fdd4"}]},{"id":"S057","semantic_type":"proof","kind":"proof","title":"能量非负性引理完整证明","source_units":[{"start":50842,"end":51341,"start_line":724,"end_line":733,"raw_tex":"\\begin{proof}\r\n\tIf $\\grad f(x)=0$ the statement is trivial. If $\\grad f(x) \\neq 0$, we denote $\\rho=\\frac{|y|}{|\\grad f(x)|}$ and get\r\n\t$$\r\n\tE(x,y) \\ge \\Bigl(\\frac{a}{2C_L}-\\rho+\\frac{b}{2}\\rho^2 \\Bigr) |\\grad f(x)|^2.\r\n\t$$\r\n\tThe quadratic function $\\varphi(\\rho)=\\frac{a}{2C_L}-\\rho+\\frac{b}{2}\\rho^2$ attains its global minimum at $\\rho=\\frac{1}{b}$ and using $ab \\ge C_L$ we deduce that\r\n\t\\begin{align*}\r\n\t\t\\varphi\\Bigl(\\frac 1b\\Bigr)=\\frac{a}{2C_L}-\\frac{1}{2b}\\ge 0.\r\n\t\\end{align*}\r\n\\end{proof}","sha256_utf8":"38bc2643eb6cc437d01430f9995845a656dc39feffd8b802dfccfe15d7edde2a"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"将能量下界化为二次函数","source_units":[{"start":50857,"end":51067,"start_line":725,"end_line":728,"raw_tex":"\tIf $\\grad f(x)=0$ the statement is trivial. If $\\grad f(x) \\neq 0$, we denote $\\rho=\\frac{|y|}{|\\grad f(x)|}$ and get\r\n\t$$\r\n\tE(x,y) \\ge \\Bigl(\\frac{a}{2C_L}-\\rho+\\frac{b}{2}\\rho^2 \\Bigr) |\\grad f(x)|^2.\r\n\t$$\r\n","sha256_utf8":"861f11411cfe47ea1c6e6795dd6cd18895d7bb4b988aefab164bf6fb73044fc3"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"计算二次函数的非负最小值","source_units":[{"start":51067,"end":51330,"start_line":729,"end_line":732,"raw_tex":"\tThe quadratic function $\\varphi(\\rho)=\\frac{a}{2C_L}-\\rho+\\frac{b}{2}\\rho^2$ attains its global minimum at $\\rho=\\frac{1}{b}$ and using $ab \\ge C_L$ we deduce that\r\n\t\\begin{align*}\r\n\t\t\\varphi\\Bigl(\\frac 1b\\Bigr)=\\frac{a}{2C_L}-\\frac{1}{2b}\\ge 0.\r\n\t\\end{align*}\r\n","sha256_utf8":"e9ea1da35e85edd98d03a3ee1a591879b8c366cd94b536b8acc171f57608c329"}]},{"id":"S060","semantic_type":"lemma","kind":"lemma","title":"光滑函数的逆梯度不等式","source_units":[{"start":33667,"end":33984,"start_line":457,"end_line":462,"raw_tex":"\\begin{lemma} \\label{rem:Lipschitz}\r\n\tLet $r>0$, $y \\in \\R^d$ and assume that $\\grad f$ is $C_L$-Lipschitz continuous on $B_r(y)$ and $\\inf_{x \\in B_r(y)}f(x)=f(y)$. Then, for all $x \\in B_{r/2}(y)$ it holds that\r\n\t\\begin{align} \\label{eq:inversePL}\r\n\t\t|\\grad f(x)|^2 \\le 2 C_L (f(x)-f(y)).\r\n\t\\end{align}\r\n\\end{lemma}","sha256_utf8":"fd1f3e7394a69ba188e3d9bbe9724741a79213317219580f24e27fad8d53fa7a"}]},{"id":"S061","semantic_type":"proof","kind":"proof","title":"逆梯度不等式完整证明","source_units":[{"start":33988,"end":34798,"start_line":464,"end_line":479,"raw_tex":"\\begin{proof}\r\n\tSince $y$ is a critical point of $f$ we have for all $x \\in B_{r/2}(y)$\r\n\t$$\r\n\t|\\grad f(x)| = |\\grad f(x)-\\grad f(y)| \\le \\frac{C_L r}{2}.\r\n\t$$\r\n\tIf $\\grad f(x)=0$ the statement is obviously true. \r\n\tIf $\\grad f(x) \\neq 0$ consider the function \r\n\t$$\r\n\tg(t) = f\\Bigl(x-t \\frac{\\grad f(x)}{|\\grad f(x)|}\\Bigr).\r\n\t$$\r\n\tNote that for $x \\in B_{r/2}(y)$ and all $t \\in [0, \\frac{|\\grad f(x)|}{C_L}]$ we have $x-t \\frac{\\grad f(x)}{|\\grad f(x)|} \\in B_r(y)$ so that with the Lipschitz continuity of $\\grad f$ and since $y$ is a local minimum\r\n\t\\begin{align*}\r\n\t\tf(y)-f(x) &\\le g\\Bigl(\\frac{|\\grad f(x)|}{C_L}\\Bigr)-g(0) = \\int_0^{\\frac{|\\grad f(x)|}{C_L}} g'(s)  \\, ds \\\\\r\n\t\t& \\le \\frac{|\\grad f(x)|}{C_L} g'(0) +\\frac{|\\grad f(x)|^2}{2C_L}= -\\frac{|\\grad f(x)|^2}{2C_L}.\r\n\t\\end{align*}\r\n\\end{proof}","sha256_utf8":"ca6cec0bf84bdfb2ec95b5c08d6592d1c718b6cbe86f9dba1252d79938e72355"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"控制梯度并处理零梯度情形","source_units":[{"start":34003,"end":34203,"start_line":465,"end_line":469,"raw_tex":"\tSince $y$ is a critical point of $f$ we have for all $x \\in B_{r/2}(y)$\r\n\t$$\r\n\t|\\grad f(x)| = |\\grad f(x)-\\grad f(y)| \\le \\frac{C_L r}{2}.\r\n\t$$\r\n\tIf $\\grad f(x)=0$ the statement is obviously true. \r\n","sha256_utf8":"a8f2729d44d2408f9baa591d62bc786293c456d64644e645121ba3c556d4db1c"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"定义负梯度方向的一维曲线","source_units":[{"start":34203,"end":34320,"start_line":470,"end_line":473,"raw_tex":"\tIf $\\grad f(x) \\neq 0$ consider the function \r\n\t$$\r\n\tg(t) = f\\Bigl(x-t \\frac{\\grad f(x)}{|\\grad f(x)|}\\Bigr).\r\n\t$$\r\n","sha256_utf8":"fc1af27b1c291e30d15f2e8907f029a57c392ed8a2d6e6a28afcfcb564d9a847"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"沿曲线积分得到逆梯度界","source_units":[{"start":34320,"end":34787,"start_line":474,"end_line":478,"raw_tex":"\tNote that for $x \\in B_{r/2}(y)$ and all $t \\in [0, \\frac{|\\grad f(x)|}{C_L}]$ we have $x-t \\frac{\\grad f(x)}{|\\grad f(x)|} \\in B_r(y)$ so that with the Lipschitz continuity of $\\grad f$ and since $y$ is a local minimum\r\n\t\\begin{align*}\r\n\t\tf(y)-f(x) &\\le g\\Bigl(\\frac{|\\grad f(x)|}{C_L}\\Bigr)-g(0) = \\int_0^{\\frac{|\\grad f(x)|}{C_L}} g'(s)  \\, ds \\\\\r\n\t\t& \\le \\frac{|\\grad f(x)|}{C_L} g'(0) +\\frac{|\\grad f(x)|^2}{2C_L}= -\\frac{|\\grad f(x)|^2}{2C_L}.\r\n\t\\end{align*}\r\n","sha256_utf8":"5b5e93d836f0fca034fa3fac7af1b0c1903520e5fe1a3accb8a4b8e789fbace3"}]},{"id":"S065","semantic_type":"external_fact","kind":"external_fact","title":"柯西与杨氏不等式","source_units":[],"declared_external":true,"declaration_text":["欧氏内积满足 |⟨u,v⟩|≤|u||v|；随机变量满足 E[|YZ|]≤E[|Y|²]^{1/2}E[|Z|²]^{1/2}；对任意 η>0，有 uv≤u²/(2η)+ηv²/2。"]},{"id":"S066","semantic_type":"external_fact","kind":"external_fact","title":"马尔可夫与尾事件估计","source_units":[],"declared_external":true,"declaration_text":["对非负随机变量 Y 和 c>0，P(Y≥c)≤E[Y]/c；P(⋃_i A_i)≤Σ_i P(A_i)；对递减事件序列，P(⋂_n A_n)=lim_n P(A_n)。"]},{"id":"S067","semantic_type":"external_fact","kind":"external_fact","title":"有限期望长度与路径收敛","source_units":[],"declared_external":true,"declaration_text":["非负可测函数的求和或积分可以按Tonelli定理与期望交换；非负随机变量具有有限期望则几乎处处有限；若 Σ_n |x_n−x_{n−1}|<∞，则欧氏空间中的序列 (x_n) 收敛。"]},{"id":"S068","semantic_type":"assumption","kind":"assumption","title":"小步长命题的技术约束","source_units":[{"start":59908,"end":60158,"start_line":863,"end_line":867,"raw_tex":"\tAssume that\r\n\t\\begin{align} \\label{eq:238877}\r\n\t\t\\mu-a+b>0 \\quad , \\quad a\\mu-a^2+ab-2L <0 \\quad \\text{and} \\quad C_L -\\frac{b}{2}(\\mu+a-b)<0.\r\n\t\\end{align}\r\n\tIf $\\P(\\bigcap_{n \\in \\N_0} \\IA_n)< \\P(\\IA_0)$ we additionally assume that $ab \\ge C_L$.\r\n","sha256_utf8":"305a7ce34bad8233c2ada6d7fa8e9b60c96d532c5f801d49eef1d381d6aa3cf2"}]},{"id":"S069","semantic_type":"assumption","kind":"assumption","title":"能量递推的事件与系数条件","source_units":[{"start":51514,"end":52394,"start_line":738,"end_line":743,"raw_tex":"\tLet $L,a,b>0, \\sigma \\ge 0$. Let $(\\IA_n)_{n \\in \\N_0}$ be a decreasing sequence of events such that, for all $n \\in \\N_0$, $\\IA_n \\in \\cF_n$ and on $\\IA_n$ it holds that\r\n\t\\begin{align} \\label{eq:1293182}\r\n\t\t|\\grad f(X_n)|^2 \\ge 2L f(X_n) \\quad \\text{and} \\quad \\E[|D_{n+1}|^2 | \\cF_n] \\le \\sigma f(X_n).\r\n\t\\end{align}\r\n\tLet $(\\alpha_n)_{n \\in \\N}, (\\beta_n)_{n \\in \\N}, (\\delta_n)_{n \\in \\N}$ and $(\\epsilon_n)_{n \\in \\N}$ be given by $\\eqref{eq:alpha}$. Assume that $(\\beta_n)_{n \\in \\N}$, $(\\alpha_{n+1}-a\\delta_{n+1}+2\\beta_{n+1}L)_{n \\in \\N_0}$, $(\\epsilon_{n+1}-\\frac{b}{2} \\delta_{n+1})_{n \\in \\N_0}$ and $(\\frac{C_L}{2} \\gamma_{n}^2- \\frac{b}{2}\\gamma_{n})_{n \\in \\N_0}$ are sequences of non-positive reals and $(\\delta_n)_{n \\in \\N}$ is a sequence of non-negative reals.\r\n\tMoreover, if $\\P(\\bigcap_{n \\in \\N_0} \\IA_n)<\\P(\\IA_0)$ additionally assume that $ab \\ge C_L$.\r\n","sha256_utf8":"8e5dfc69023fd65ea4fb79b8ec5e79e1cadb36f1172dbc6f7703cbe5498f8b37"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S006","edges":[{"source_id":"S008","target_id":"S006","kind":"supports"},{"source_id":"S009","target_id":"S008","kind":"supports"},{"source_id":"S010","target_id":"S008","kind":"supports"},{"source_id":"S011","target_id":"S008","kind":"supports"},{"source_id":"S012","target_id":"S008","kind":"supports"},{"source_id":"S013","target_id":"S008","kind":"supports"},{"source_id":"S014","target_id":"S008","kind":"supports"},{"source_id":"S015","target_id":"S008","kind":"supports"},{"source_id":"S016","target_id":"S008","kind":"supports"},{"source_id":"S017","target_id":"S008","kind":"supports"},{"source_id":"S018","target_id":"S008","kind":"supports"},{"source_id":"S019","target_id":"S008","kind":"supports"},{"source_id":"S020","target_id":"S008","kind":"supports"},{"source_id":"S021","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S009","kind":"supports"},{"source_id":"S007","target_id":"S017","kind":"supports"},{"source_id":"S007","target_id":"S018","kind":"supports"},{"source_id":"S004","target_id":"S010","kind":"supports"},{"source_id":"S004","target_id":"S018","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S013","kind":"supports"},{"source_id":"S022","target_id":"S010","kind":"supports"},{"source_id":"S022","target_id":"S013","kind":"supports"},{"source_id":"S022","target_id":"S018","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S016","kind":"supports"},{"source_id":"S001","target_id":"S017","kind":"supports"},{"source_id":"S011","target_id":"S017","kind":"supports"},{"source_id":"S012","target_id":"S017","kind":"supports"},{"source_id":"S014","target_id":"S017","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S029","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S017","target_id":"S021","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S023","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S030","target_id":"S024","kind":"supports"},{"source_id":"S032","target_id":"S024","kind":"supports"},{"source_id":"S004","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S029","target_id":"S022","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S027","kind":"supports"},{"source_id":"S031","target_id":"S027","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S060","target_id":"S029","kind":"supports"},{"source_id":"S004","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S035","target_id":"S033","kind":"supports"},{"source_id":"S036","target_id":"S033","kind":"supports"},{"source_id":"S037","target_id":"S033","kind":"supports"},{"source_id":"S038","target_id":"S033","kind":"supports"},{"source_id":"S039","target_id":"S033","kind":"supports"},{"source_id":"S040","target_id":"S033","kind":"supports"},{"source_id":"S041","target_id":"S033","kind":"supports"},{"source_id":"S042","target_id":"S033","kind":"supports"},{"source_id":"S043","target_id":"S033","kind":"supports"},{"source_id":"S050","target_id":"S035","kind":"supports"},{"source_id":"S050","target_id":"S036","kind":"supports"},{"source_id":"S068","target_id":"S035","kind":"supports"},{"source_id":"S068","target_id":"S036","kind":"supports"},{"source_id":"S034","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S044","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S039","kind":"supports"},{"source_id":"S060","target_id":"S040","kind":"supports"},{"source_id":"S005","target_id":"S040","kind":"supports"},{"source_id":"S065","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S037","target_id":"S041","kind":"supports"},{"source_id":"S038","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S052","target_id":"S041","kind":"supports"},{"source_id":"S035","target_id":"S041","kind":"supports"},{"source_id":"S037","target_id":"S042","kind":"supports"},{"source_id":"S038","target_id":"S042","kind":"supports"},{"source_id":"S039","target_id":"S042","kind":"supports"},{"source_id":"S066","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S043","kind":"supports"},{"source_id":"S002","target_id":"S043","kind":"supports"},{"source_id":"S065","target_id":"S043","kind":"supports"},{"source_id":"S067","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S048","target_id":"S045","kind":"supports"},{"source_id":"S049","target_id":"S045","kind":"supports"},{"source_id":"S050","target_id":"S045","kind":"supports"},{"source_id":"S051","target_id":"S045","kind":"supports"},{"source_id":"S052","target_id":"S045","kind":"supports"},{"source_id":"S053","target_id":"S045","kind":"supports"},{"source_id":"S054","target_id":"S045","kind":"supports"},{"source_id":"S055","target_id":"S045","kind":"supports"},{"source_id":"S001","target_id":"S046","kind":"supports"},{"source_id":"S003","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S002","target_id":"S047","kind":"supports"},{"source_id":"S002","target_id":"S048","kind":"supports"},{"source_id":"S069","target_id":"S048","kind":"supports"},{"source_id":"S002","target_id":"S049","kind":"supports"},{"source_id":"S001","target_id":"S049","kind":"supports"},{"source_id":"S047","target_id":"S050","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S005","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S005","target_id":"S051","kind":"supports"},{"source_id":"S069","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S069","target_id":"S052","kind":"supports"},{"source_id":"S056","target_id":"S052","kind":"supports"},{"source_id":"S060","target_id":"S052","kind":"supports"},{"source_id":"S046","target_id":"S053","kind":"supports"},{"source_id":"S002","target_id":"S053","kind":"supports"},{"source_id":"S005","target_id":"S053","kind":"supports"},{"source_id":"S052","target_id":"S054","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S069","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S059","target_id":"S057","kind":"supports"},{"source_id":"S065","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S063","target_id":"S061","kind":"supports"},{"source_id":"S064","target_id":"S061","kind":"supports"},{"source_id":"S001","target_id":"S062","kind":"supports"},{"source_id":"S062","target_id":"S064","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"}]}}
{"record_id":"2302.05875v1","paper":{"title":"A tensor optimization algorithm for computing Lagrangians of hypergraphs","authors":["Jingya Chang","Bin Xiao","Xin Zhang"],"arxiv_id":"2302.05875","version":"v1","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-023-02215-2","abstract_url":"https://arxiv.org/abs/2302.05875v1","pdf_url":"https://arxiv.org/pdf/2302.05875v1"},"context":{"raw_tex":"An $r$-uniform hypergraph ($r$-graph) $G=(V,E)$ consists of a vertex set $V:=\\{1,2,\\dots,n\\}$ and an edge set $E\\subseteq V^{(r)}$, where $V^{(r)}$ denotes the collection of all subsets of $V$ of size $r$,\n$m=|E|$ is the number of edges. For example the sunflower hypergraph in Figure \\ref{sunflower} is a $4$-uniform hypergraph with $10$ vertices. The weight polynomial for $G$ is defined as $w(G,\\xv):=\\sum_{e\\in E}\\prod_{i\\in e}x_i$, where legal weighting for $G$ satisfies $x_i\\ge0$ for all $i\\in V$ and $\\sum_{i\\in V}x_i=1$.\nLet $\\Delta:=\\{\\xv\\in \\Re^n_+:\\one^T\\xv=1\\}$ be the legal weighting set that is a simplex. Here $\\one$ is the all one vector. The Lagrangian of an $r$-graph $G$ \\cite{MS-65,Ta-02} is to maximize the weight polynomial $w(G,\\xv)$ under the simplex constraint\n\\begin{equation}\\label{Lagr}\n  \\lambda(G) := \\max\\left\\{w(G,\\xv)=\\sum_{e\\in E}\\prod_{i\\in e}x_i~:~\\one^T\\xv=1,~\\xv\\in\\Re^n_+\\right\\}.\n\\end{equation}\n\n\\begin{Def}[Symmetry tensor \\cite{QL-book}]\n  A tensor $$\\mathcal{T}=(t_{i_1\\ldots i_r})\\in \\Re^{[r,n]}, \\qquad \\text{for} \\ i_j = 1,\\ldots, n, j = 1,\\ldots,r$$ is an $r$th order $n$ dimensional symmetric tensor if the value of $t_{i_1\\ldots i_k}$ is invariable under any permutation of its indices.\n\\end{Def}\n\n\\begin{Def}[Adjacency tensor \\cite{CoD-12}]\n  Let $G=(V,E)$ be an $r$-graph with $n$ vertices. The adjacency tensor of $G$ is an $r$th order $n$-dimensional symmetric tensor $\\At=[a_{i_1 \\cdots i_r}]$, of which elements are\n  \\begin{equation*}\n    a_{i_1 \\cdots i_r}=\\left\\{\\begin{aligned}\n      &\\frac{1}{(r-1)!} && \\quad\\text{ if }\\{i_1,\\ldots,i_r\\}\\in E, \\\\\n      &0                && \\quad\\text{ otherwise. }\n    \\end{aligned}\\right.\n  \\end{equation*}\n\\end{Def}\n\nHence, to compute the Lagrangian of an $r$-graph $G$, we will solve the following optimization problem\n\\begin{equation}\\label{OPM}\n\\lambda(G)=\\left\\{\\begin{aligned}\n  & \\max && f(\\xv)=\\frac{1}{r}\\At\\xv^r \\\\\n  & ~\\mathrm{s.t.}&& \\xv\\in\\Delta,\n\\end{aligned}\\right.\n\\end{equation}\n\nwhich is equivalent to \\eqref{Lagr}. Since the adjacency tensor $\\At$ of $G$ is symmetric, it holds \\cite{QL-book,CDQY18} that $\\nabla f(\\xv)=\\At\\xv^{r-1}$, where elements of the vector $\\At\\xv^{r-1}$ are\n\\begin{equation*}\n  (\\At\\xv^{r-1})_j=\\sum_{i_2=1}^n\\cdots\\sum_{i_r=1}^n a_{ji_2\\cdots i_r}x_{i_2}\\cdots x_{i_r} \\qquad\n  \\forall~j=1,\\dots,n.\n\\end{equation*}\n\nIt is easy to see that the feasible region $\\Delta$ is a closed convex set. The Euclidean projection\n\\begin{equation*}\n  \\pi_{\\Delta}(\\av) := \\arg\\min\\limits_{\\xv\\in\\Delta} \\|\\xv-\\av\\|\n\\end{equation*}\nexists and is unique for any vector $\\av\\in\\Re^n$. Algorithm \\ref{Alg:Proj} provides an $\\mathcal{O}(n\\log n)$ approach for computing the Euclidean projection onto $\\Delta$ \\cite{WC'13,CY'11}.\n\n\n\\begin{algorithm}[t]\n\\caption{Euclidean projection $\\pi_{\\Delta}(\\av)$.}\\label{Alg:Proj}\n\\begin{algorithmic}[1]\n  \\STATE Sort the input vector $\\av\\in\\Re^n$ into an auxiliary vector $\\bv$ such that $b_1\\ge b_2\\ge \\dots\\ge b_n$.\n  \\STATE Find the largest index $\\ell=\\max\\{j: jb_j+1-\\sum_{i=1}^j b_i>0, 1\\le j\\le n\\}$.\n  \\STATE Set $\\lambda = (1-\\sum_{i=1}^{\\ell} b_i)/{\\ell}$.\n  \\STATE Output $\\xv$ with $x_i=\\max\\{a_i+\\lambda,0\\}$.\n\\end{algorithmic}\n\\end{algorithm}\n\n\\begin{algorithm}[t]\n\\caption{A gradient projection algorithm (GPA).}\\label{Alg:pGrad}\n\\begin{algorithmic}[1]\n  \\STATE Choose $0<\\underline{\\alpha}\\le\\alpha_0\\le\\overline{\\alpha}$, $\\beta\\in(0,1)$, $\\eta\\in(0,1/2]$, and $\\xv_0\\in\\Delta$. Set $c\\gets0$.\n  \\WHILE{the sequence of iterates does not converge}\n    \\STATE Set $\\gv_c = \\pi_{\\Delta}(\\xv_c + \\alpha_c\\nabla f(\\xv_c))-\\xv_c$.\n    \\STATE Find the smallest nonnegative integer $j$ such that\n      \\begin{equation}\\label{LineSch}\n        f(\\xv_c+\\beta^j\\gv_c)-f(\\xv_c) \\ge \\eta\\beta^j \\gv_c^T \\nabla f(\\xv_c).\n      \\end{equation}\n    \\STATE Set $\\rho_c=\\beta^j$ and $\\xv_{c+1}=\\xv_c+\\rho_c\\gv_c$.\n    \\STATE Choose $\\alpha_{c+1}\\in[\\underline{\\alpha},\\overline{\\alpha}]$ and set $c\\gets c+1$.\n  \\ENDWHILE\n\\end{algorithmic}\n\\end{algorithm}\n\nNow, we analyze the convergence of the proposed GPA. Part of our analysis on basic theory is similar to \\cite{Ber-book,HZ'06} and we present here for the purpose of completion. First, we call $\\xv_*\\in\\Delta$ a critical point of \\eqref{OPM} if\n\\begin{equation*}\n  \\pi_{\\Delta}(\\xv_*+\\nabla f(\\xv_*))=\\xv_*.\n\\end{equation*}\n\n\nBecause $f(\\xv)=\\frac{1}{r}\\At\\xv^r$ is a polynomial and the simplex $\\Delta$ is compact, the Hessian $\\nabla^2 f(\\xv)$ is bounded, i.e., there exists a constant $M\\ge1$ such that\n\\begin{equation*}\n  \\|\\nabla^2 f(\\xv)\\| \\le M  \\qquad \\forall \\xv\\in\\Delta.\n\\end{equation*}\n\n\nBecause the objective function and the constraint set of \\eqref{OPM} are semi-algebraic, the following {\\L}ojasiewicz gradient inequality is valid \\cite{Loj-63,BDL-07}. The analysis of this subsection is based on work in \\cite{ABRS10}.\n","units":[{"start":5130,"end":6065,"start_line":111,"end_line":116,"raw_tex":"An $r$-uniform hypergraph ($r$-graph) $G=(V,E)$ consists of a vertex set $V:=\\{1,2,\\dots,n\\}$ and an edge set $E\\subseteq V^{(r)}$, where $V^{(r)}$ denotes the collection of all subsets of $V$ of size $r$,\n$m=|E|$ is the number of edges. For example the sunflower hypergraph in Figure \\ref{sunflower} is a $4$-uniform hypergraph with $10$ vertices. The weight polynomial for $G$ is defined as $w(G,\\xv):=\\sum_{e\\in E}\\prod_{i\\in e}x_i$, where legal weighting for $G$ satisfies $x_i\\ge0$ for all $i\\in V$ and $\\sum_{i\\in V}x_i=1$.\nLet $\\Delta:=\\{\\xv\\in \\Re^n_+:\\one^T\\xv=1\\}$ be the legal weighting set that is a simplex. Here $\\one$ is the all one vector. The Lagrangian of an $r$-graph $G$ \\cite{MS-65,Ta-02} is to maximize the weight polynomial $w(G,\\xv)$ under the simplex constraint\n\\begin{equation}\\label{Lagr}\n  \\lambda(G) := \\max\\left\\{w(G,\\xv)=\\sum_{e\\in E}\\prod_{i\\in e}x_i~:~\\one^T\\xv=1,~\\xv\\in\\Re^n_+\\right\\}.\n\\end{equation}","sha256_utf8":"6663d0ba0a57519cbb6897d59bc7c11c19034ae71dc4ff332d6c9e3402fa655a"},{"start":11121,"end":11430,"start_line":151,"end_line":153,"raw_tex":"\\begin{Def}[Symmetry tensor \\cite{QL-book}]\n  A tensor $$\\mathcal{T}=(t_{i_1\\ldots i_r})\\in \\Re^{[r,n]}, \\qquad \\text{for} \\ i_j = 1,\\ldots, n, j = 1,\\ldots,r$$ is an $r$th order $n$ dimensional symmetric tensor if the value of $t_{i_1\\ldots i_k}$ is invariable under any permutation of its indices.\n\\end{Def}","sha256_utf8":"9393170f003fac761b131b5f43c34e75936ecde2b30e8597c94ea83553420930"},{"start":11432,"end":11897,"start_line":155,"end_line":163,"raw_tex":"\\begin{Def}[Adjacency tensor \\cite{CoD-12}]\n  Let $G=(V,E)$ be an $r$-graph with $n$ vertices. The adjacency tensor of $G$ is an $r$th order $n$-dimensional symmetric tensor $\\At=[a_{i_1 \\cdots i_r}]$, of which elements are\n  \\begin{equation*}\n    a_{i_1 \\cdots i_r}=\\left\\{\\begin{aligned}\n      &\\frac{1}{(r-1)!} && \\quad\\text{ if }\\{i_1,\\ldots,i_r\\}\\in E, \\\\\n      &0                && \\quad\\text{ otherwise. }\n    \\end{aligned}\\right.\n  \\end{equation*}\n\\end{Def}","sha256_utf8":"99f0acff87bc4cfabc2ce22b365718ede699e1ddd13381c9b50053bb176139bc"},{"start":12725,"end":13002,"start_line":184,"end_line":190,"raw_tex":"Hence, to compute the Lagrangian of an $r$-graph $G$, we will solve the following optimization problem\n\\begin{equation}\\label{OPM}\n\\lambda(G)=\\left\\{\\begin{aligned}\n  & \\max && f(\\xv)=\\frac{1}{r}\\At\\xv^r \\\\\n  & ~\\mathrm{s.t.}&& \\xv\\in\\Delta,\n\\end{aligned}\\right.\n\\end{equation}","sha256_utf8":"4076209a46f20112bdbb95cf15d705ea602ad4dbaccfe0ebe06bd4bb0c6a5909"},{"start":13003,"end":13365,"start_line":191,"end_line":195,"raw_tex":"which is equivalent to \\eqref{Lagr}. Since the adjacency tensor $\\At$ of $G$ is symmetric, it holds \\cite{QL-book,CDQY18} that $\\nabla f(\\xv)=\\At\\xv^{r-1}$, where elements of the vector $\\At\\xv^{r-1}$ are\n\\begin{equation*}\n  (\\At\\xv^{r-1})_j=\\sum_{i_2=1}^n\\cdots\\sum_{i_r=1}^n a_{ji_2\\cdots i_r}x_{i_2}\\cdots x_{i_r} \\qquad\n  \\forall~j=1,\\dots,n.\n\\end{equation*}","sha256_utf8":"f3db71f28a7553d58e3c3aec223373ce8014737c04cc07769bc3e2022cb0d21c"},{"start":15676,"end":16070,"start_line":247,"end_line":251,"raw_tex":"It is easy to see that the feasible region $\\Delta$ is a closed convex set. The Euclidean projection\n\\begin{equation*}\n  \\pi_{\\Delta}(\\av) := \\arg\\min\\limits_{\\xv\\in\\Delta} \\|\\xv-\\av\\|\n\\end{equation*}\nexists and is unique for any vector $\\av\\in\\Re^n$. Algorithm \\ref{Alg:Proj} provides an $\\mathcal{O}(n\\log n)$ approach for computing the Euclidean projection onto $\\Delta$ \\cite{WC'13,CY'11}.\n","sha256_utf8":"27e29b4fb2c386b6e42184c33b6834555991a72a93f6093e7966031a7f54812c"},{"start":16072,"end":16538,"start_line":254,"end_line":262,"raw_tex":"\\begin{algorithm}[t]\n\\caption{Euclidean projection $\\pi_{\\Delta}(\\av)$.}\\label{Alg:Proj}\n\\begin{algorithmic}[1]\n  \\STATE Sort the input vector $\\av\\in\\Re^n$ into an auxiliary vector $\\bv$ such that $b_1\\ge b_2\\ge \\dots\\ge b_n$.\n  \\STATE Find the largest index $\\ell=\\max\\{j: jb_j+1-\\sum_{i=1}^j b_i>0, 1\\le j\\le n\\}$.\n  \\STATE Set $\\lambda = (1-\\sum_{i=1}^{\\ell} b_i)/{\\ell}$.\n  \\STATE Output $\\xv$ with $x_i=\\max\\{a_i+\\lambda,0\\}$.\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"da0fb1330554044f45193fe0a3250c512ba08d2ec9dfaca191d85b9f36ffc068"},{"start":17262,"end":18056,"start_line":268,"end_line":282,"raw_tex":"\\begin{algorithm}[t]\n\\caption{A gradient projection algorithm (GPA).}\\label{Alg:pGrad}\n\\begin{algorithmic}[1]\n  \\STATE Choose $0<\\underline{\\alpha}\\le\\alpha_0\\le\\overline{\\alpha}$, $\\beta\\in(0,1)$, $\\eta\\in(0,1/2]$, and $\\xv_0\\in\\Delta$. Set $c\\gets0$.\n  \\WHILE{the sequence of iterates does not converge}\n    \\STATE Set $\\gv_c = \\pi_{\\Delta}(\\xv_c + \\alpha_c\\nabla f(\\xv_c))-\\xv_c$.\n    \\STATE Find the smallest nonnegative integer $j$ such that\n      \\begin{equation}\\label{LineSch}\n        f(\\xv_c+\\beta^j\\gv_c)-f(\\xv_c) \\ge \\eta\\beta^j \\gv_c^T \\nabla f(\\xv_c).\n      \\end{equation}\n    \\STATE Set $\\rho_c=\\beta^j$ and $\\xv_{c+1}=\\xv_c+\\rho_c\\gv_c$.\n    \\STATE Choose $\\alpha_{c+1}\\in[\\underline{\\alpha},\\overline{\\alpha}]$ and set $c\\gets c+1$.\n  \\ENDWHILE\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"665febb6e98a95c9be8f8d707d6b3c8b5a488beb0ef9ceb7902a84a656da8d3e"},{"start":18121,"end":18444,"start_line":290,"end_line":293,"raw_tex":"Now, we analyze the convergence of the proposed GPA. Part of our analysis on basic theory is similar to \\cite{Ber-book,HZ'06} and we present here for the purpose of completion. First, we call $\\xv_*\\in\\Delta$ a critical point of \\eqref{OPM} if\n\\begin{equation*}\n  \\pi_{\\Delta}(\\xv_*+\\nabla f(\\xv_*))=\\xv_*.\n\\end{equation*}\n","sha256_utf8":"2829364d2eb405c057201267bf27907d975c0637558e62f9120ccf3c762cc10d"},{"start":19113,"end":19385,"start_line":305,"end_line":308,"raw_tex":"Because $f(\\xv)=\\frac{1}{r}\\At\\xv^r$ is a polynomial and the simplex $\\Delta$ is compact, the Hessian $\\nabla^2 f(\\xv)$ is bounded, i.e., there exists a constant $M\\ge1$ such that\n\\begin{equation*}\n  \\|\\nabla^2 f(\\xv)\\| \\le M  \\qquad \\forall \\xv\\in\\Delta.\n\\end{equation*}\n","sha256_utf8":"98b34dd27131a4c9df570e741c015b2b1d5e914409529a2c1ee75681c278d8d9"},{"start":22061,"end":22297,"start_line":362,"end_line":362,"raw_tex":"Because the objective function and the constraint set of \\eqref{OPM} are semi-algebraic, the following {\\L}ojasiewicz gradient inequality is valid \\cite{Loj-63,BDL-07}. The analysis of this subsection is based on work in \\cite{ABRS10}.\n","sha256_utf8":"bd70b37079727b73d7988b9e615f1544182173a4c216c73c8eb035cb519d8055"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{Thm}\\label{Thm:5-4}\n  Suppose GPA generates an infinite sequence of iteration points $\\{\\xv_c\\}$ that converges to a critical point $\\xv_*$. Then, we have the following estimations on convergence rate.\n  \\begin{itemize}\n    \\item If $\\theta=1/2$, there exist $\\gamma>0$ and $\\mu\\in(0,1)$ such that\n      \\begin{equation*}\n        \\|\\xv_c-\\xv_*\\| \\le \\gamma\\mu^c.\n      \\end{equation*}\n    \\item If $\\theta\\in(1/2,1)$, there exist $\\gamma_1>0$ and $\\gamma_2>0$ such that\n      \\begin{equation*}\n        |f(\\xv_c)-f(\\xv_*)| \\le \\gamma_1 c^{-\\frac{1}{2\\theta-1}} \\qquad\\text{ and }\\qquad\n        \\|\\xv_c-\\xv_*\\| \\le \\gamma_2 c^{-\\frac{1-\\theta}{2\\theta-1}}.\n      \\end{equation*}\n  \\end{itemize}\n\\end{Thm}","units":[{"start":27222,"end":27931,"start_line":439,"end_line":452,"raw_tex":"\\begin{Thm}\\label{Thm:5-4}\n  Suppose GPA generates an infinite sequence of iteration points $\\{\\xv_c\\}$ that converges to a critical point $\\xv_*$. Then, we have the following estimations on convergence rate.\n  \\begin{itemize}\n    \\item If $\\theta=1/2$, there exist $\\gamma>0$ and $\\mu\\in(0,1)$ such that\n      \\begin{equation*}\n        \\|\\xv_c-\\xv_*\\| \\le \\gamma\\mu^c.\n      \\end{equation*}\n    \\item If $\\theta\\in(1/2,1)$, there exist $\\gamma_1>0$ and $\\gamma_2>0$ such that\n      \\begin{equation*}\n        |f(\\xv_c)-f(\\xv_*)| \\le \\gamma_1 c^{-\\frac{1}{2\\theta-1}} \\qquad\\text{ and }\\qquad\n        \\|\\xv_c-\\xv_*\\| \\le \\gamma_2 c^{-\\frac{1-\\theta}{2\\theta-1}}.\n      \\end{equation*}\n  \\end{itemize}\n\\end{Thm}","sha256_utf8":"d8c63ca1fabe403de815b82f2369c09a54a574e21946623a18fec9b30b063130"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{Lem}\n  Let $G$ be an $r$-graph with  $n$ vertices. Its weight polynomial $w(G,\\xv)$ could be rewritten as\n  \\begin{equation*}\n    w(G,\\xv) = \\frac{1}{r}\\At\\xv^r,\n  \\end{equation*}\n  where $\\At\\xv^r := \\sum_{i_1=1}^n\\cdots\\sum_{i_r=1}^n a_{i_1\\cdots i_r}x_{i_1}\\cdots x_{i_r}$.\n\\end{Lem}\n\n\\begin{proof}\n  By direct calculations, it yields that\n  \\begin{eqnarray}\n  \\frac{1}{r}\\At\\xv^r &=& \\frac{1}{r}\\sum_{i_1=1}^n\\cdots\\sum_{i_r=1}^n a_{i_1\\cdots i_r}x_{i_1}\\cdots x_{i_r} \\nonumber\\\\\n    &=& \\frac{1}{r}\\sum_{{\\{i_1,\\dots,i_r\\}\\in E}\\atop i_1<\\dots<i_r} \\frac{1}{(r-1)!} r! x_{i_1}\\cdots x_{i_r} \\nonumber\\\\\n    &=& \\sum_{e\\in E}\\prod_{i\\in e}x_i = w(G,\\xv). \\label{aaabbb}\n  \\end{eqnarray}\n  The proof is completed.\n\\end{proof}\n\n\\begin{Lem}\\label{Lem:4-1}\n  Let $\\gv^{\\alpha}(\\xv)=\\pi_{\\Delta}(\\xv+\\alpha\\nabla f(\\xv))-\\xv$. Then, \\\\\n  1) $\\|\\gv^{\\alpha}(\\xv)\\|$ is nondecreasing in $\\alpha>0$ for all $\\xv\\in\\Delta$; \\\\\n  2) $\\|\\gv^{\\alpha}(\\xv)\\|/\\alpha$ is nonincreasing in $\\alpha>0$ for all $\\xv\\in\\Delta$; \\\\\n  3) $\\gv^{\\alpha}(\\xv)^T\\nabla f(\\xv)\\ge\\|\\gv^{\\alpha}(\\xv)\\|^2/\\alpha$ for all $\\xv\\in\\Delta$ and $\\alpha>0$; \\\\\n  4) for any $\\xv\\in\\Delta$ and $\\alpha>0$, $\\gv^{\\alpha}(\\xv)=0$ if and only if $\\xv$ is a \\cl{stationary} point for \\eqref{OPM}.\n\\end{Lem}\n\n\\begin{Lem}\\label{Lem:4-2}\n  There exists a constant $\\kappa_1>0$ such that for any $\\rho_c$ generated by Algorithm \\ref{Alg:pGrad}\n  \\begin{equation*}\n    \\rho_c\\ge \\kappa_1.\n  \\end{equation*}\n\n\\end{Lem}\n\n\\begin{proof}\n  We will prove that the inequality \\eqref{LineSch} is valid\n  %\\begin{equation*}\n  %  f(\\xv_c+\\rho\\gv_c) \\le f(\\xv_c)+\\eta\\rho\\gv_c^T \\nabla f(\\xv_c)\n  %\\end{equation*}\n  when $\\rho\\in[0, \\frac{2(1-\\eta)}{\\overline{\\alpha}M}]$. By Lemma \\ref{Lem:4-1}, it yields that $\\|\\gv_c\\|^2\\le\\alpha_c\\gv_c^T\\nabla f(\\xv_c)$. From Taylor's formula, we have\n  \\begin{eqnarray*}\n    f(\\xv_c+\\rho\\gv_c) &\\ge& f(\\xv_c)+\\rho \\gv_c^T\\nabla f(\\xv_c)-\\frac{M}{2}\\rho^2\\|\\gv_c\\|^2 \\\\\n      &\\ge& f(\\xv_c)+\\rho \\gv_c^T\\nabla f(\\xv_c) -\\frac{M}{2}\\rho^2\\alpha_c\\gv_c^T\\nabla f(\\xv_c) \\\\\n      &\\ge& f(\\xv_c)+\\rho \\gv_c^T\\nabla f(\\xv_c) -(1-\\eta)\\rho\\gv_c^T\\nabla f(\\xv_c) \\\\\n      &=& f(\\xv_c)+\\eta\\rho \\gv_c^T\\nabla f(\\xv_c),\n  \\end{eqnarray*}\n  where the last inequality holds owing to the assumption $0\\le\\rho\\alpha_cM\\le \\rho\\overline{\\alpha}M\\le 2(1-\\eta)$. According to Step 4 of Algorithm GPA,  this lemma is valid if $\\kappa_1:=\\frac{2\\beta(1-\\eta)}{\\overline{\\alpha}M}$.\n\\end{proof}\n\n\\begin{Thm}\\label{Thm:4-3}\n  Suppose that $\\{\\xv_c\\}$ is an infinity sequence of iterates generated by GPA. Then, we have\n  \\begin{equation*}\n    \\lim_{c\\to\\infty} \\|\\pi_{\\Delta}(\\xv_c + \\nabla f(\\xv_c))-\\xv_c\\| = 0.\n  \\end{equation*}\n  That is to say, every limit point of $\\{\\xv_c\\}$ is a critical point.\n\\end{Thm}\n\n\\begin{proof}\n  From \\eqref{LineSch}, Lemmas \\ref{Lem:4-1} and \\ref{Lem:4-2}, we obtain\n  \\begin{equation}\\label{fun-lowbd}\n    f(\\xv_{c+1})-f(\\xv_c) \\ge \\eta\\rho_c\\gv_c^T\\nabla f(\\xv_c) \\ge \\frac{\\eta\\rho_c}{\\alpha_c}\\|\\gv_c\\|^2\n      \\ge \\frac{\\eta\\kappa_1}{\\overline{\\alpha}}\\|\\gv_c\\|^2.\n  \\end{equation}\n  Hence, we have\n  \\begin{equation*}\n    \\sum_{c=1}^{\\infty} \\|\\gv_c\\|^2 \\le \\frac{\\overline{\\alpha}}{\\eta\\kappa_1}\\sum_{c=1}^{\\infty}[f(\\xv_{c+1})-f(\\xv_c)]\n       \\le \\frac{\\overline{\\alpha}}{\\eta\\kappa_1}\\lambda(G),\n  \\end{equation*}\n  which means that $\\|\\gv_c\\| \\to0$ as $c\\to\\infty$.\n\n  On one hand, if $\\alpha_c\\ge1$, we get $\\|\\gv^{\\alpha_c}(\\xv_c)\\|\\ge\\|\\gv^1(\\xv_c)\\|$ by Lemma \\ref{Lem:4-1}. On the other hand, $\\alpha_c<1$ and hence $\\|\\gv^{\\alpha_c}(\\xv_c)\\|/\\alpha_c\\ge\\|\\gv^1(\\xv_c)\\|$. Hence, we have\n  \\begin{equation}\\label{gc-lowbd}\n    \\|\\gv_c\\|=\\|\\gv^{\\alpha_c}(\\xv_c)\\|\\ge \\min\\{\\alpha_c,1\\}\\|\\gv^1(\\xv_c)\\| \\ge \\min\\{\\underline{\\alpha},1\\}\\|\\gv^1(\\xv_c)\\|.\n  \\end{equation}\n Therefore $\\|\\gv^1(\\xv_c)\\|$ tends to zero. The proof is then completed.\n\\end{proof}\n\n\\begin{Thm}[{\\L}ojasiewicz property]\\label{Thm:5-1}\n  Suppose that $\\xv_*$ is a critical point of \\eqref{OPM}, i.e., $\\gv^1(\\xv_*)=0$. Then, there exist a neighborhood of $\\xv_*$ denoted as $\\mathds{U}(\\xv_*)$, an exponent $\\theta\\in[1/2,1)$, and a positive constant $C$ such that the following inequality\n  \\begin{equation*}\n    \\|\\gv^1(\\xv)\\| \\ge C|f(\\xv)-f(\\xv_*)|^{\\theta}\n  \\end{equation*}\n  holds for all $\\xv\\in \\mathds{U}(\\xv_*) \\cap\\Delta$.\n\\end{Thm}\n\n\\begin{Lem}\\label{Lem:5-2}\n  Let $\\xv_*$ be a limiting point of $\\{\\xv_c\\}$. The initial iterate $\\xv_0$ is sufficiently close to $\\xv_*$ in the sense that $\\xv_0\\in\\mathds{B}(\\xv_*,\\sigma)\\cap\\Delta$, where $\\mathds{B}(\\xv_*,\\sigma):=\\{\\xv\\in\\Re^n:\\|\\xv-\\xv_*\\|<\\sigma\\}\\subseteq\\mathds{U}(\\xv_*)$ and $\\sigma\\ge \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C(1-\\theta)}|f(\\xv_0)-f(\\xv_*)|^{1-\\theta}+\\|\\xv_0-\\xv_*\\|$. Then, we have the following two assertions:\n  \\begin{equation*}\n    \\xv_c\\in\\mathds{B}(\\xv_*,\\sigma)\\cap\\Delta \\qquad\\text{ for }c=0,1,2,\\dots,\n  \\end{equation*}\n  and\n  \\begin{equation*}\n    \\sum_{c=0}^{\\infty}\\|\\xv_c-\\xv_{c+1}\\| \\le \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C(1-\\theta)}|f(\\xv_0)-f(\\xv_*)|^{1-\\theta}.\n  \\end{equation*}\n\\end{Lem}\n\n\\begin{proof}\nThe conclusions can be  proved by  induction. It is obvious to see that $\\xv_0\\in\\mathds{B}(\\xv_*,\\sigma)\\cap\\Delta$. Next, by supposing that there exists a positive integer $\\ell$ such that\n  \\begin{equation*}\n    \\xv_c\\in\\mathds{B}(\\xv_*,\\sigma)\\cap\\Delta \\qquad\\text{ for }c=0,1,\\dots,\\ell,\n  \\end{equation*}\n  we show $\\xv_{\\ell+1}\\in\\mathds{B}(\\xv_*,\\sigma)\\cap\\Delta$.\n\n  Define $\\phi(t):=\\frac{1}{C(1-\\theta)}|t-f(\\xv_*)|^{1-\\theta}$. It is easy to see that $\\phi(t)$ is a monotonically decreasing and  \\cl{concave} function for  $t<f(\\xv_*)$. Then, for $0\\le c\\le \\ell$, we have\n  \\begin{eqnarray*}\n    \\phi(f(\\xv_c))-\\phi(f(\\xv_{c+1}))\n      &\\ge& \\phi'(f(\\xv_c))(f(\\xv_c)-f(\\xv_{c+1})) \\\\\n      &=& \\frac{-1}{C|f(\\xv_c)-f(\\xv_*)|^{\\theta}} (f(\\xv_c)-f(\\xv_{c+1})) \\\\\n      &\\ge& \\frac{1}{\\|\\gv^1(\\xv_c)\\|}(f(\\xv_{c+1})-f(\\xv_c)) \\\\\n      &\\ge& \\frac{1}{\\|\\gv^1(\\xv_c)\\|}\\frac{\\eta\\rho_c}{\\alpha_c}\\|\\gv_c\\|^2 \\\\\n      &\\ge& \\frac{1}{\\|\\gv^1(\\xv_c)\\|}\\frac{\\eta\\rho_c}{\\alpha_c}\\|\\gv_c\\|\\min\\{\\alpha_c,1\\}\\|\\gv^1(\\xv_c)\\| \\\\\n      &=& \\frac{\\eta\\min\\{\\alpha_c,1\\}}{\\alpha_c}\\|\\rho_c\\gv_c\\| \\\\\n      &=& \\eta\\min\\{1,\\alpha_c^{-1}\\}\\|\\xv_c-\\xv_{c+1}\\|,\n  \\end{eqnarray*}\n  where the second inequality is obtained based on Theorem \\ref{Thm:5-1}, the third inequality is deduced from \\eqref{fun-lowbd}, and the fourth inequality is valid because of \\eqref{gc-lowbd}. The above inequalities indicate that\n  \\begin{equation*}\n    \\|\\xv_c-\\xv_{c+1}\\| \\le \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta}\\left(\\phi(f(\\xv_c))-\\phi(f(\\xv_{c+1}))\\right),\n  \\end{equation*}\n  for $0\\le c\\le \\ell$. Hence, it holds that\n  \\begin{equation}\\label{iterate-length}\n    \\sum_{c=0}^{\\ell} \\|\\xv_c-\\xv_{c+1}\\|\n      \\le \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta}\\sum_{c=0}^{\\ell}\\left(\\phi(f(\\xv_c))-\\phi(f(\\xv_{c+1}))\\right)\n      \\le \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta}\\phi(f(\\xv_0)).\n  \\end{equation}\n  Therefore, we have\n  \\begin{equation*}\n    \\|\\xv_{\\ell+1}-\\xv_*\\| \\le \\|\\xv_0-\\xv_*\\|+\\sum_{c=0}^{\\ell} \\|\\xv_c-\\xv_{c+1}\\|\n      \\le \\|\\xv_0-\\xv_*\\|+\\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta}\\phi(f(\\xv_0)) <\\sigma,\n  \\end{equation*}\n  which implies that $\\xv_{\\ell+1}\\in\\mathds{B}(\\xv_*,\\sigma)$. On the other hand, every iterate is feasible by the mechanism of GPA.  Hence, we obtain the first assertion. The second  conclusion comes from \\eqref{iterate-length} straightforwardly by setting $\\ell\\to\\infty$.\n\\end{proof}\n\n\\begin{proof}\n  Without loss of generality, we assume $\\xv_0\\in\\mathds{B}(\\xv_*,\\sigma)$. Define\n  \\begin{equation*}\n    \\zeta_c:=\\sum_{\\ell=c}^{\\infty} \\|\\xv_{\\ell}-\\xv_{\\ell+1}\\| \\ge \\|\\xv_c-\\xv_*\\|.\n  \\end{equation*}\n  From Lemma \\ref{Lem:5-2} and Theorem \\ref{Thm:5-1}, we have\n  \\begin{eqnarray*}\n    \\zeta_c &\\le& \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C(1-\\theta)}|f(\\xv_c)-f(\\xv_*)|^{1-\\theta} \\\\\n      &=& \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C^{1/\\theta}(1-\\theta)}\\left(C|f(\\xv_c)-f(\\xv_*)|^{\\theta}\\right)^{(1-\\theta)/\\theta} \\\\\n      &\\le& \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C^{1/\\theta}(1-\\theta)}\\left(\\|\\gv^1(\\xv_c)\\|\\right)^{(1-\\theta)/\\theta}.\n  \\end{eqnarray*}\n  It can be obtained from Lemma \\ref{Lem:4-2} and \\eqref{gc-lowbd} that\n  \\begin{equation*}\n    \\|\\xv_c-\\xv_{c+1}\\| = \\rho_c\\|\\gv_c\\| \\ge \\kappa_1\\min\\{\\underline{\\alpha},1\\}\\|\\gv^1(\\xv_c)\\|.\n  \\end{equation*}\n  Hence, by denoting $\\kappa_2:=\\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C^{1/\\theta}(1-\\theta)}(\\kappa_1\\min\\{\\underline{\\alpha},1\\})^{-(1-\\theta)/\\theta}$, we get\n  \\begin{equation}\\label{zeta-1}\n    \\zeta_c \\le \\kappa_2\\|\\xv_c-\\xv_{c+1}\\|^{(1-\\theta)/\\theta} = \\kappa_2(\\zeta_c-\\zeta_{c+1})^{(1-\\theta)/\\theta}.\n  \\end{equation}\n  If $\\theta=1/2$, the inequality \\eqref{zeta-1} means\n  \\begin{equation*}\n    \\zeta_{c+1} \\le \\frac{\\kappa_2-1}{\\kappa_2}\\zeta_c.\n  \\end{equation*}\n  Hence, the first estimation holds with $\\gamma=\\zeta_0$ and $\\mu=(\\kappa_2-1)/\\kappa_2$.\n\n  Next, we consider the case $\\theta\\in(1/2,1)$. Let $\\varphi(t):=t^{-\\theta/(1-\\theta)}$ be a decreasing function for $t>0$. It yields from \\eqref{zeta-1} that\n  \\begin{eqnarray*}\n    \\kappa_2^{-\\theta/(1-\\theta)} &\\le& \\varphi(\\zeta_c)(\\zeta_c-\\zeta_{c+1}) \\\\\n      &=& \\int_{\\zeta_{c+1}}^{\\zeta_c} \\varphi(\\zeta_c) \\mathrm{d}t \\\\\n      &\\le& \\int_{\\zeta_{c+1}}^{\\zeta_c} \\varphi(t) \\mathrm{d}t \\\\\n      &=& \\cl{\\frac{1-\\theta}{1-2\\theta}}\\left(\\zeta_{c+1}^{-(2\\theta-1)/(1-\\theta)}-\\zeta_c^{-(2\\theta-1)/(1-\\theta)}\\right),\n  \\end{eqnarray*}\n  which implies\n  \\begin{equation*}\n    \\zeta_{c+1}^{-(2\\theta-1)/(1-\\theta)}-\\zeta_c^{-(2\\theta-1)/(1-\\theta)}\n      \\ge \\cl{\\frac{1-2\\theta}{1-\\theta}}\\kappa_2^{-\\theta/(1-\\theta)} := \\kappa_3>0.\n  \\end{equation*}\nThen, we have\n  \\begin{equation*}\n    \\zeta_c^{-(2\\theta-1)/(1-\\theta)} \\ge \\kappa_3+\\zeta_{c-1}^{-(2\\theta-1)/(1-\\theta)} \\ge\\dots\n      \\ge \\kappa_3c+\\zeta_0^{-(2\\theta-1)/(1-\\theta)},\n  \\end{equation*}\n  which means\n  \\begin{equation*}\n    \\zeta_c \\le \\left(\\kappa_3c+\\zeta_0^{-(2\\theta-1)/(1-\\theta)}\\right)^{-(1-\\theta)/(2\\theta-1)}\n      \\le (\\kappa_3c)^{-(1-\\theta)/(2\\theta-1)}.\n  \\end{equation*}\nThe last estimation holds by taking $\\gamma_2=\\kappa_3^{-(1-\\theta)/(2\\theta-1)}$.\n\n   Let $\\xi_c:=|f(\\xv_c)-f(\\xv_*)|$. From \\eqref{fun-lowbd}, \\eqref{gc-lowbd}, and Theorem \\ref{Thm:5-1}, we have\n  \\begin{eqnarray*}\n    \\xi_c-\\xi_{c+1} &=& f(\\xv_{c+1})-f(\\xv_c) \\\\\n      &\\ge& \\frac{\\eta\\kappa_1}{\\overline{\\alpha}}\\|\\gv_c\\|^2 \\\\\n      &\\ge& \\frac{\\eta\\kappa_1(\\min\\{\\underline{\\alpha},1\\})^2}{\\overline{\\alpha}}\\|\\gv^1(\\xv_c)\\|^2 \\\\\n      &\\ge& \\frac{\\eta\\kappa_1C^2(\\min\\{\\underline{\\alpha},1\\})^2}{\\overline{\\alpha}}\\xi_c^{2\\theta}\n      := \\kappa_4 \\xi_c^{2\\theta}.\n  \\end{eqnarray*}\nDenote $\\chi(t):=t^{-2\\theta}$ as a decreasing function for $t>0$. It is deduced from the above inequality that\n  \\begin{equation*}\n    \\kappa_4 \\le \\chi(\\xi_c)(\\xi_c-\\xi_{c+1})\n      = \\int_{\\xi_{c+1}}^{\\xi_c} \\chi(\\xi_c) \\mathrm{d}t\n      \\le \\int_{\\xi_{c+1}}^{\\xi_c} \\chi(t) \\mathrm{d}t\n      = \\frac{1}{2\\theta-1}\\left(\\xi_{c+1}^{-(2\\theta-1)}-\\xi_c^{-(2\\theta-1)}\\right).\n  \\end{equation*}\n  Let $\\kappa_5:=(2\\theta-1)\\kappa_4>0$. We have\n  \\begin{equation*}\n    \\xi_c^{-(2\\theta-1)} \\ge \\kappa_5+\\xi_{c-1}^{-(2\\theta-1)} \\ge \\dots \\ge\\kappa_5c+\\xi_0^{-(2\\theta-1)},\n  \\end{equation*}\n  which means\n  \\begin{equation*}\n    \\xi_c \\le \\left(\\kappa_5c+\\xi_0^{-(2\\theta-1)}\\right)^{-1/(2\\theta-1)}\n      \\le (\\kappa_5c)^{-1/(2\\theta-1)}.\n  \\end{equation*}\nThe second estimation is valid if we take $\\gamma_1=\\kappa_5^{-1/(2\\theta-1)}$.\n\\end{proof}","units":[{"start":11988,"end":12281,"start_line":167,"end_line":173,"raw_tex":"\\begin{Lem}\n  Let $G$ be an $r$-graph with  $n$ vertices. Its weight polynomial $w(G,\\xv)$ could be rewritten as\n  \\begin{equation*}\n    w(G,\\xv) = \\frac{1}{r}\\At\\xv^r,\n  \\end{equation*}\n  where $\\At\\xv^r := \\sum_{i_1=1}^n\\cdots\\sum_{i_r=1}^n a_{i_1\\cdots i_r}x_{i_1}\\cdots x_{i_r}$.\n\\end{Lem}","sha256_utf8":"5e614e437f9c95d1a72818dbaabd914c6b708b6048f1452dc287790a3c1092fa"},{"start":12282,"end":12723,"start_line":174,"end_line":182,"raw_tex":"\\begin{proof}\n  By direct calculations, it yields that\n  \\begin{eqnarray}\n  \\frac{1}{r}\\At\\xv^r &=& \\frac{1}{r}\\sum_{i_1=1}^n\\cdots\\sum_{i_r=1}^n a_{i_1\\cdots i_r}x_{i_1}\\cdots x_{i_r} \\nonumber\\\\\n    &=& \\frac{1}{r}\\sum_{{\\{i_1,\\dots,i_r\\}\\in E}\\atop i_1<\\dots<i_r} \\frac{1}{(r-1)!} r! x_{i_1}\\cdots x_{i_r} \\nonumber\\\\\n    &=& \\sum_{e\\in E}\\prod_{i\\in e}x_i = w(G,\\xv). \\label{aaabbb}\n  \\end{eqnarray}\n  The proof is completed.\n\\end{proof}","sha256_utf8":"8adc7657c29ea29aa50541418454102721922e72d2440f12a06c39654e08fbfb"},{"start":18570,"end":19111,"start_line":297,"end_line":303,"raw_tex":"\\begin{Lem}\\label{Lem:4-1}\n  Let $\\gv^{\\alpha}(\\xv)=\\pi_{\\Delta}(\\xv+\\alpha\\nabla f(\\xv))-\\xv$. Then, \\\\\n  1) $\\|\\gv^{\\alpha}(\\xv)\\|$ is nondecreasing in $\\alpha>0$ for all $\\xv\\in\\Delta$; \\\\\n  2) $\\|\\gv^{\\alpha}(\\xv)\\|/\\alpha$ is nonincreasing in $\\alpha>0$ for all $\\xv\\in\\Delta$; \\\\\n  3) $\\gv^{\\alpha}(\\xv)^T\\nabla f(\\xv)\\ge\\|\\gv^{\\alpha}(\\xv)\\|^2/\\alpha$ for all $\\xv\\in\\Delta$ and $\\alpha>0$; \\\\\n  4) for any $\\xv\\in\\Delta$ and $\\alpha>0$, $\\gv^{\\alpha}(\\xv)=0$ if and only if $\\xv$ is a \\cl{stationary} point for \\eqref{OPM}.\n\\end{Lem}","sha256_utf8":"fa0449afc2ac531baeb5d5225b45970684b144293665805a832d1ec0ce2b0f7b"},{"start":19386,"end":19590,"start_line":310,"end_line":316,"raw_tex":"\\begin{Lem}\\label{Lem:4-2}\n  There exists a constant $\\kappa_1>0$ such that for any $\\rho_c$ generated by Algorithm \\ref{Alg:pGrad}\n  \\begin{equation*}\n    \\rho_c\\ge \\kappa_1.\n  \\end{equation*}\n\n\\end{Lem}","sha256_utf8":"fe6acc361db6e12c25984980b9e408f64332754da98b1f1ec8ff03b34649d62d"},{"start":19591,"end":20575,"start_line":317,"end_line":330,"raw_tex":"\\begin{proof}\n  We will prove that the inequality \\eqref{LineSch} is valid\n  %\\begin{equation*}\n  %  f(\\xv_c+\\rho\\gv_c) \\le f(\\xv_c)+\\eta\\rho\\gv_c^T \\nabla f(\\xv_c)\n  %\\end{equation*}\n  when $\\rho\\in[0, \\frac{2(1-\\eta)}{\\overline{\\alpha}M}]$. By Lemma \\ref{Lem:4-1}, it yields that $\\|\\gv_c\\|^2\\le\\alpha_c\\gv_c^T\\nabla f(\\xv_c)$. From Taylor's formula, we have\n  \\begin{eqnarray*}\n    f(\\xv_c+\\rho\\gv_c) &\\ge& f(\\xv_c)+\\rho \\gv_c^T\\nabla f(\\xv_c)-\\frac{M}{2}\\rho^2\\|\\gv_c\\|^2 \\\\\n      &\\ge& f(\\xv_c)+\\rho \\gv_c^T\\nabla f(\\xv_c) -\\frac{M}{2}\\rho^2\\alpha_c\\gv_c^T\\nabla f(\\xv_c) \\\\\n      &\\ge& f(\\xv_c)+\\rho \\gv_c^T\\nabla f(\\xv_c) -(1-\\eta)\\rho\\gv_c^T\\nabla f(\\xv_c) \\\\\n      &=& f(\\xv_c)+\\eta\\rho \\gv_c^T\\nabla f(\\xv_c),\n  \\end{eqnarray*}\n  where the last inequality holds owing to the assumption $0\\le\\rho\\alpha_cM\\le \\rho\\overline{\\alpha}M\\le 2(1-\\eta)$. According to Step 4 of Algorithm GPA,  this lemma is valid if $\\kappa_1:=\\frac{2\\beta(1-\\eta)}{\\overline{\\alpha}M}$.\n\\end{proof}","sha256_utf8":"6fcb18cae177c9c3a104f3f4fba0d2f90f29c8c9c914a3980f94bc5f901fdf7d"},{"start":20577,"end":20893,"start_line":332,"end_line":338,"raw_tex":"\\begin{Thm}\\label{Thm:4-3}\n  Suppose that $\\{\\xv_c\\}$ is an infinity sequence of iterates generated by GPA. Then, we have\n  \\begin{equation*}\n    \\lim_{c\\to\\infty} \\|\\pi_{\\Delta}(\\xv_c + \\nabla f(\\xv_c))-\\xv_c\\| = 0.\n  \\end{equation*}\n  That is to say, every limit point of $\\{\\xv_c\\}$ is a critical point.\n\\end{Thm}","sha256_utf8":"02dd25e57fae188776cc4da320c12e09990e15c1489d246b7f8a2f1eee6b7f81"},{"start":20894,"end":21984,"start_line":339,"end_line":357,"raw_tex":"\\begin{proof}\n  From \\eqref{LineSch}, Lemmas \\ref{Lem:4-1} and \\ref{Lem:4-2}, we obtain\n  \\begin{equation}\\label{fun-lowbd}\n    f(\\xv_{c+1})-f(\\xv_c) \\ge \\eta\\rho_c\\gv_c^T\\nabla f(\\xv_c) \\ge \\frac{\\eta\\rho_c}{\\alpha_c}\\|\\gv_c\\|^2\n      \\ge \\frac{\\eta\\kappa_1}{\\overline{\\alpha}}\\|\\gv_c\\|^2.\n  \\end{equation}\n  Hence, we have\n  \\begin{equation*}\n    \\sum_{c=1}^{\\infty} \\|\\gv_c\\|^2 \\le \\frac{\\overline{\\alpha}}{\\eta\\kappa_1}\\sum_{c=1}^{\\infty}[f(\\xv_{c+1})-f(\\xv_c)]\n       \\le \\frac{\\overline{\\alpha}}{\\eta\\kappa_1}\\lambda(G),\n  \\end{equation*}\n  which means that $\\|\\gv_c\\| \\to0$ as $c\\to\\infty$.\n\n  On one hand, if $\\alpha_c\\ge1$, we get $\\|\\gv^{\\alpha_c}(\\xv_c)\\|\\ge\\|\\gv^1(\\xv_c)\\|$ by Lemma \\ref{Lem:4-1}. On the other hand, $\\alpha_c<1$ and hence $\\|\\gv^{\\alpha_c}(\\xv_c)\\|/\\alpha_c\\ge\\|\\gv^1(\\xv_c)\\|$. Hence, we have\n  \\begin{equation}\\label{gc-lowbd}\n    \\|\\gv_c\\|=\\|\\gv^{\\alpha_c}(\\xv_c)\\|\\ge \\min\\{\\alpha_c,1\\}\\|\\gv^1(\\xv_c)\\| \\ge \\min\\{\\underline{\\alpha},1\\}\\|\\gv^1(\\xv_c)\\|.\n  \\end{equation}\n Therefore $\\|\\gv^1(\\xv_c)\\|$ tends to zero. The proof is then completed.\n\\end{proof}","sha256_utf8":"d9aba4fbbf0c724bcee59b09aae1583fcbc45d8d82fd41b85a34509e3e8fd2b0"},{"start":22298,"end":22757,"start_line":364,"end_line":370,"raw_tex":"\\begin{Thm}[{\\L}ojasiewicz property]\\label{Thm:5-1}\n  Suppose that $\\xv_*$ is a critical point of \\eqref{OPM}, i.e., $\\gv^1(\\xv_*)=0$. Then, there exist a neighborhood of $\\xv_*$ denoted as $\\mathds{U}(\\xv_*)$, an exponent $\\theta\\in[1/2,1)$, and a positive constant $C$ such that the following inequality\n  \\begin{equation*}\n    \\|\\gv^1(\\xv)\\| \\ge C|f(\\xv)-f(\\xv_*)|^{\\theta}\n  \\end{equation*}\n  holds for all $\\xv\\in \\mathds{U}(\\xv_*) \\cap\\Delta$.\n\\end{Thm}","sha256_utf8":"769b5e90c15a5b93f7cf2588d436671fb92246b3fc185c89c728f80a61026b5a"},{"start":22760,"end":23521,"start_line":373,"end_line":382,"raw_tex":"\\begin{Lem}\\label{Lem:5-2}\n  Let $\\xv_*$ be a limiting point of $\\{\\xv_c\\}$. The initial iterate $\\xv_0$ is sufficiently close to $\\xv_*$ in the sense that $\\xv_0\\in\\mathds{B}(\\xv_*,\\sigma)\\cap\\Delta$, where $\\mathds{B}(\\xv_*,\\sigma):=\\{\\xv\\in\\Re^n:\\|\\xv-\\xv_*\\|<\\sigma\\}\\subseteq\\mathds{U}(\\xv_*)$ and $\\sigma\\ge \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C(1-\\theta)}|f(\\xv_0)-f(\\xv_*)|^{1-\\theta}+\\|\\xv_0-\\xv_*\\|$. Then, we have the following two assertions:\n  \\begin{equation*}\n    \\xv_c\\in\\mathds{B}(\\xv_*,\\sigma)\\cap\\Delta \\qquad\\text{ for }c=0,1,2,\\dots,\n  \\end{equation*}\n  and\n  \\begin{equation*}\n    \\sum_{c=0}^{\\infty}\\|\\xv_c-\\xv_{c+1}\\| \\le \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C(1-\\theta)}|f(\\xv_0)-f(\\xv_*)|^{1-\\theta}.\n  \\end{equation*}\n\\end{Lem}","sha256_utf8":"6003f1f0161564b289bb0e06e9b9384bb910d80e3757f0d44d31d821751057d5"},{"start":23522,"end":25945,"start_line":383,"end_line":417,"raw_tex":"\\begin{proof}\nThe conclusions can be  proved by  induction. It is obvious to see that $\\xv_0\\in\\mathds{B}(\\xv_*,\\sigma)\\cap\\Delta$. Next, by supposing that there exists a positive integer $\\ell$ such that\n  \\begin{equation*}\n    \\xv_c\\in\\mathds{B}(\\xv_*,\\sigma)\\cap\\Delta \\qquad\\text{ for }c=0,1,\\dots,\\ell,\n  \\end{equation*}\n  we show $\\xv_{\\ell+1}\\in\\mathds{B}(\\xv_*,\\sigma)\\cap\\Delta$.\n\n  Define $\\phi(t):=\\frac{1}{C(1-\\theta)}|t-f(\\xv_*)|^{1-\\theta}$. It is easy to see that $\\phi(t)$ is a monotonically decreasing and  \\cl{concave} function for  $t<f(\\xv_*)$. Then, for $0\\le c\\le \\ell$, we have\n  \\begin{eqnarray*}\n    \\phi(f(\\xv_c))-\\phi(f(\\xv_{c+1}))\n      &\\ge& \\phi'(f(\\xv_c))(f(\\xv_c)-f(\\xv_{c+1})) \\\\\n      &=& \\frac{-1}{C|f(\\xv_c)-f(\\xv_*)|^{\\theta}} (f(\\xv_c)-f(\\xv_{c+1})) \\\\\n      &\\ge& \\frac{1}{\\|\\gv^1(\\xv_c)\\|}(f(\\xv_{c+1})-f(\\xv_c)) \\\\\n      &\\ge& \\frac{1}{\\|\\gv^1(\\xv_c)\\|}\\frac{\\eta\\rho_c}{\\alpha_c}\\|\\gv_c\\|^2 \\\\\n      &\\ge& \\frac{1}{\\|\\gv^1(\\xv_c)\\|}\\frac{\\eta\\rho_c}{\\alpha_c}\\|\\gv_c\\|\\min\\{\\alpha_c,1\\}\\|\\gv^1(\\xv_c)\\| \\\\\n      &=& \\frac{\\eta\\min\\{\\alpha_c,1\\}}{\\alpha_c}\\|\\rho_c\\gv_c\\| \\\\\n      &=& \\eta\\min\\{1,\\alpha_c^{-1}\\}\\|\\xv_c-\\xv_{c+1}\\|,\n  \\end{eqnarray*}\n  where the second inequality is obtained based on Theorem \\ref{Thm:5-1}, the third inequality is deduced from \\eqref{fun-lowbd}, and the fourth inequality is valid because of \\eqref{gc-lowbd}. The above inequalities indicate that\n  \\begin{equation*}\n    \\|\\xv_c-\\xv_{c+1}\\| \\le \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta}\\left(\\phi(f(\\xv_c))-\\phi(f(\\xv_{c+1}))\\right),\n  \\end{equation*}\n  for $0\\le c\\le \\ell$. Hence, it holds that\n  \\begin{equation}\\label{iterate-length}\n    \\sum_{c=0}^{\\ell} \\|\\xv_c-\\xv_{c+1}\\|\n      \\le \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta}\\sum_{c=0}^{\\ell}\\left(\\phi(f(\\xv_c))-\\phi(f(\\xv_{c+1}))\\right)\n      \\le \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta}\\phi(f(\\xv_0)).\n  \\end{equation}\n  Therefore, we have\n  \\begin{equation*}\n    \\|\\xv_{\\ell+1}-\\xv_*\\| \\le \\|\\xv_0-\\xv_*\\|+\\sum_{c=0}^{\\ell} \\|\\xv_c-\\xv_{c+1}\\|\n      \\le \\|\\xv_0-\\xv_*\\|+\\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta}\\phi(f(\\xv_0)) <\\sigma,\n  \\end{equation*}\n  which implies that $\\xv_{\\ell+1}\\in\\mathds{B}(\\xv_*,\\sigma)$. On the other hand, every iterate is feasible by the mechanism of GPA.  Hence, we obtain the first assertion. The second  conclusion comes from \\eqref{iterate-length} straightforwardly by setting $\\ell\\to\\infty$.\n\\end{proof}","sha256_utf8":"58c05f451e3677264edd649fcc78f972cc05ef41f8d0db6311f15b96f7959b58"},{"start":27932,"end":32007,"start_line":453,"end_line":527,"raw_tex":"\\begin{proof}\n  Without loss of generality, we assume $\\xv_0\\in\\mathds{B}(\\xv_*,\\sigma)$. Define\n  \\begin{equation*}\n    \\zeta_c:=\\sum_{\\ell=c}^{\\infty} \\|\\xv_{\\ell}-\\xv_{\\ell+1}\\| \\ge \\|\\xv_c-\\xv_*\\|.\n  \\end{equation*}\n  From Lemma \\ref{Lem:5-2} and Theorem \\ref{Thm:5-1}, we have\n  \\begin{eqnarray*}\n    \\zeta_c &\\le& \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C(1-\\theta)}|f(\\xv_c)-f(\\xv_*)|^{1-\\theta} \\\\\n      &=& \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C^{1/\\theta}(1-\\theta)}\\left(C|f(\\xv_c)-f(\\xv_*)|^{\\theta}\\right)^{(1-\\theta)/\\theta} \\\\\n      &\\le& \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C^{1/\\theta}(1-\\theta)}\\left(\\|\\gv^1(\\xv_c)\\|\\right)^{(1-\\theta)/\\theta}.\n  \\end{eqnarray*}\n  It can be obtained from Lemma \\ref{Lem:4-2} and \\eqref{gc-lowbd} that\n  \\begin{equation*}\n    \\|\\xv_c-\\xv_{c+1}\\| = \\rho_c\\|\\gv_c\\| \\ge \\kappa_1\\min\\{\\underline{\\alpha},1\\}\\|\\gv^1(\\xv_c)\\|.\n  \\end{equation*}\n  Hence, by denoting $\\kappa_2:=\\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C^{1/\\theta}(1-\\theta)}(\\kappa_1\\min\\{\\underline{\\alpha},1\\})^{-(1-\\theta)/\\theta}$, we get\n  \\begin{equation}\\label{zeta-1}\n    \\zeta_c \\le \\kappa_2\\|\\xv_c-\\xv_{c+1}\\|^{(1-\\theta)/\\theta} = \\kappa_2(\\zeta_c-\\zeta_{c+1})^{(1-\\theta)/\\theta}.\n  \\end{equation}\n  If $\\theta=1/2$, the inequality \\eqref{zeta-1} means\n  \\begin{equation*}\n    \\zeta_{c+1} \\le \\frac{\\kappa_2-1}{\\kappa_2}\\zeta_c.\n  \\end{equation*}\n  Hence, the first estimation holds with $\\gamma=\\zeta_0$ and $\\mu=(\\kappa_2-1)/\\kappa_2$.\n\n  Next, we consider the case $\\theta\\in(1/2,1)$. Let $\\varphi(t):=t^{-\\theta/(1-\\theta)}$ be a decreasing function for $t>0$. It yields from \\eqref{zeta-1} that\n  \\begin{eqnarray*}\n    \\kappa_2^{-\\theta/(1-\\theta)} &\\le& \\varphi(\\zeta_c)(\\zeta_c-\\zeta_{c+1}) \\\\\n      &=& \\int_{\\zeta_{c+1}}^{\\zeta_c} \\varphi(\\zeta_c) \\mathrm{d}t \\\\\n      &\\le& \\int_{\\zeta_{c+1}}^{\\zeta_c} \\varphi(t) \\mathrm{d}t \\\\\n      &=& \\cl{\\frac{1-\\theta}{1-2\\theta}}\\left(\\zeta_{c+1}^{-(2\\theta-1)/(1-\\theta)}-\\zeta_c^{-(2\\theta-1)/(1-\\theta)}\\right),\n  \\end{eqnarray*}\n  which implies\n  \\begin{equation*}\n    \\zeta_{c+1}^{-(2\\theta-1)/(1-\\theta)}-\\zeta_c^{-(2\\theta-1)/(1-\\theta)}\n      \\ge \\cl{\\frac{1-2\\theta}{1-\\theta}}\\kappa_2^{-\\theta/(1-\\theta)} := \\kappa_3>0.\n  \\end{equation*}\nThen, we have\n  \\begin{equation*}\n    \\zeta_c^{-(2\\theta-1)/(1-\\theta)} \\ge \\kappa_3+\\zeta_{c-1}^{-(2\\theta-1)/(1-\\theta)} \\ge\\dots\n      \\ge \\kappa_3c+\\zeta_0^{-(2\\theta-1)/(1-\\theta)},\n  \\end{equation*}\n  which means\n  \\begin{equation*}\n    \\zeta_c \\le \\left(\\kappa_3c+\\zeta_0^{-(2\\theta-1)/(1-\\theta)}\\right)^{-(1-\\theta)/(2\\theta-1)}\n      \\le (\\kappa_3c)^{-(1-\\theta)/(2\\theta-1)}.\n  \\end{equation*}\nThe last estimation holds by taking $\\gamma_2=\\kappa_3^{-(1-\\theta)/(2\\theta-1)}$.\n\n   Let $\\xi_c:=|f(\\xv_c)-f(\\xv_*)|$. From \\eqref{fun-lowbd}, \\eqref{gc-lowbd}, and Theorem \\ref{Thm:5-1}, we have\n  \\begin{eqnarray*}\n    \\xi_c-\\xi_{c+1} &=& f(\\xv_{c+1})-f(\\xv_c) \\\\\n      &\\ge& \\frac{\\eta\\kappa_1}{\\overline{\\alpha}}\\|\\gv_c\\|^2 \\\\\n      &\\ge& \\frac{\\eta\\kappa_1(\\min\\{\\underline{\\alpha},1\\})^2}{\\overline{\\alpha}}\\|\\gv^1(\\xv_c)\\|^2 \\\\\n      &\\ge& \\frac{\\eta\\kappa_1C^2(\\min\\{\\underline{\\alpha},1\\})^2}{\\overline{\\alpha}}\\xi_c^{2\\theta}\n      := \\kappa_4 \\xi_c^{2\\theta}.\n  \\end{eqnarray*}\nDenote $\\chi(t):=t^{-2\\theta}$ as a decreasing function for $t>0$. It is deduced from the above inequality that\n  \\begin{equation*}\n    \\kappa_4 \\le \\chi(\\xi_c)(\\xi_c-\\xi_{c+1})\n      = \\int_{\\xi_{c+1}}^{\\xi_c} \\chi(\\xi_c) \\mathrm{d}t\n      \\le \\int_{\\xi_{c+1}}^{\\xi_c} \\chi(t) \\mathrm{d}t\n      = \\frac{1}{2\\theta-1}\\left(\\xi_{c+1}^{-(2\\theta-1)}-\\xi_c^{-(2\\theta-1)}\\right).\n  \\end{equation*}\n  Let $\\kappa_5:=(2\\theta-1)\\kappa_4>0$. We have\n  \\begin{equation*}\n    \\xi_c^{-(2\\theta-1)} \\ge \\kappa_5+\\xi_{c-1}^{-(2\\theta-1)} \\ge \\dots \\ge\\kappa_5c+\\xi_0^{-(2\\theta-1)},\n  \\end{equation*}\n  which means\n  \\begin{equation*}\n    \\xi_c \\le \\left(\\kappa_5c+\\xi_0^{-(2\\theta-1)}\\right)^{-1/(2\\theta-1)}\n      \\le (\\kappa_5c)^{-1/(2\\theta-1)}.\n  \\end{equation*}\nThe second estimation is valid if we take $\\gamma_1=\\kappa_5^{-1/(2\\theta-1)}$.\n\\end{proof}","sha256_utf8":"db6d0df09293056c55798421701a5aef8857d13e127d37d905263706732f9950"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"超图拉格朗日问题","source_units":[{"start":5130,"end":6065,"start_line":111,"end_line":116,"raw_tex":"An $r$-uniform hypergraph ($r$-graph) $G=(V,E)$ consists of a vertex set $V:=\\{1,2,\\dots,n\\}$ and an edge set $E\\subseteq V^{(r)}$, where $V^{(r)}$ denotes the collection of all subsets of $V$ of size $r$,\n$m=|E|$ is the number of edges. For example the sunflower hypergraph in Figure \\ref{sunflower} is a $4$-uniform hypergraph with $10$ vertices. The weight polynomial for $G$ is defined as $w(G,\\xv):=\\sum_{e\\in E}\\prod_{i\\in e}x_i$, where legal weighting for $G$ satisfies $x_i\\ge0$ for all $i\\in V$ and $\\sum_{i\\in V}x_i=1$.\nLet $\\Delta:=\\{\\xv\\in \\Re^n_+:\\one^T\\xv=1\\}$ be the legal weighting set that is a simplex. Here $\\one$ is the all one vector. The Lagrangian of an $r$-graph $G$ \\cite{MS-65,Ta-02} is to maximize the weight polynomial $w(G,\\xv)$ under the simplex constraint\n\\begin{equation}\\label{Lagr}\n  \\lambda(G) := \\max\\left\\{w(G,\\xv)=\\sum_{e\\in E}\\prod_{i\\in e}x_i~:~\\one^T\\xv=1,~\\xv\\in\\Re^n_+\\right\\}.\n\\end{equation}","sha256_utf8":"6663d0ba0a57519cbb6897d59bc7c11c19034ae71dc4ff332d6c9e3402fa655a"}]},{"id":"S002","semantic_type":"definition","kind":"definition","title":"对称张量","source_units":[{"start":11121,"end":11430,"start_line":151,"end_line":153,"raw_tex":"\\begin{Def}[Symmetry tensor \\cite{QL-book}]\n  A tensor $$\\mathcal{T}=(t_{i_1\\ldots i_r})\\in \\Re^{[r,n]}, \\qquad \\text{for} \\ i_j = 1,\\ldots, n, j = 1,\\ldots,r$$ is an $r$th order $n$ dimensional symmetric tensor if the value of $t_{i_1\\ldots i_k}$ is invariable under any permutation of its indices.\n\\end{Def}","sha256_utf8":"9393170f003fac761b131b5f43c34e75936ecde2b30e8597c94ea83553420930"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"邻接张量","source_units":[{"start":11432,"end":11897,"start_line":155,"end_line":163,"raw_tex":"\\begin{Def}[Adjacency tensor \\cite{CoD-12}]\n  Let $G=(V,E)$ be an $r$-graph with $n$ vertices. The adjacency tensor of $G$ is an $r$th order $n$-dimensional symmetric tensor $\\At=[a_{i_1 \\cdots i_r}]$, of which elements are\n  \\begin{equation*}\n    a_{i_1 \\cdots i_r}=\\left\\{\\begin{aligned}\n      &\\frac{1}{(r-1)!} && \\quad\\text{ if }\\{i_1,\\ldots,i_r\\}\\in E, \\\\\n      &0                && \\quad\\text{ otherwise. }\n    \\end{aligned}\\right.\n  \\end{equation*}\n\\end{Def}","sha256_utf8":"99f0acff87bc4cfabc2ce22b365718ede699e1ddd13381c9b50053bb176139bc"}]},{"id":"S004","semantic_type":"lemma","kind":"lemma","title":"权多项式张量表示","source_units":[{"start":11988,"end":12281,"start_line":167,"end_line":173,"raw_tex":"\\begin{Lem}\n  Let $G$ be an $r$-graph with  $n$ vertices. Its weight polynomial $w(G,\\xv)$ could be rewritten as\n  \\begin{equation*}\n    w(G,\\xv) = \\frac{1}{r}\\At\\xv^r,\n  \\end{equation*}\n  where $\\At\\xv^r := \\sum_{i_1=1}^n\\cdots\\sum_{i_r=1}^n a_{i_1\\cdots i_r}x_{i_1}\\cdots x_{i_r}$.\n\\end{Lem}","sha256_utf8":"5e614e437f9c95d1a72818dbaabd914c6b708b6048f1452dc287790a3c1092fa"}]},{"id":"S005","semantic_type":"proof","kind":"proof","title":"张量表示证明","source_units":[{"start":12282,"end":12723,"start_line":174,"end_line":182,"raw_tex":"\\begin{proof}\n  By direct calculations, it yields that\n  \\begin{eqnarray}\n  \\frac{1}{r}\\At\\xv^r &=& \\frac{1}{r}\\sum_{i_1=1}^n\\cdots\\sum_{i_r=1}^n a_{i_1\\cdots i_r}x_{i_1}\\cdots x_{i_r} \\nonumber\\\\\n    &=& \\frac{1}{r}\\sum_{{\\{i_1,\\dots,i_r\\}\\in E}\\atop i_1<\\dots<i_r} \\frac{1}{(r-1)!} r! x_{i_1}\\cdots x_{i_r} \\nonumber\\\\\n    &=& \\sum_{e\\in E}\\prod_{i\\in e}x_i = w(G,\\xv). \\label{aaabbb}\n  \\end{eqnarray}\n  The proof is completed.\n\\end{proof}","sha256_utf8":"8adc7657c29ea29aa50541418454102721922e72d2440f12a06c39654e08fbfb"}]},{"id":"S006","semantic_type":"proof_step","kind":"proof_step","title":"按边合并张量求和","source_units":[{"start":12282,"end":12723,"start_line":174,"end_line":182,"raw_tex":"\\begin{proof}\n  By direct calculations, it yields that\n  \\begin{eqnarray}\n  \\frac{1}{r}\\At\\xv^r &=& \\frac{1}{r}\\sum_{i_1=1}^n\\cdots\\sum_{i_r=1}^n a_{i_1\\cdots i_r}x_{i_1}\\cdots x_{i_r} \\nonumber\\\\\n    &=& \\frac{1}{r}\\sum_{{\\{i_1,\\dots,i_r\\}\\in E}\\atop i_1<\\dots<i_r} \\frac{1}{(r-1)!} r! x_{i_1}\\cdots x_{i_r} \\nonumber\\\\\n    &=& \\sum_{e\\in E}\\prod_{i\\in e}x_i = w(G,\\xv). \\label{aaabbb}\n  \\end{eqnarray}\n  The proof is completed.\n\\end{proof}","sha256_utf8":"8adc7657c29ea29aa50541418454102721922e72d2440f12a06c39654e08fbfb"}]},{"id":"S007","semantic_type":"problem_setting","kind":"problem_setting","title":"张量优化模型与梯度","source_units":[{"start":12725,"end":13002,"start_line":184,"end_line":190,"raw_tex":"Hence, to compute the Lagrangian of an $r$-graph $G$, we will solve the following optimization problem\n\\begin{equation}\\label{OPM}\n\\lambda(G)=\\left\\{\\begin{aligned}\n  & \\max && f(\\xv)=\\frac{1}{r}\\At\\xv^r \\\\\n  & ~\\mathrm{s.t.}&& \\xv\\in\\Delta,\n\\end{aligned}\\right.\n\\end{equation}","sha256_utf8":"4076209a46f20112bdbb95cf15d705ea602ad4dbaccfe0ebe06bd4bb0c6a5909"},{"start":13003,"end":13365,"start_line":191,"end_line":195,"raw_tex":"which is equivalent to \\eqref{Lagr}. Since the adjacency tensor $\\At$ of $G$ is symmetric, it holds \\cite{QL-book,CDQY18} that $\\nabla f(\\xv)=\\At\\xv^{r-1}$, where elements of the vector $\\At\\xv^{r-1}$ are\n\\begin{equation*}\n  (\\At\\xv^{r-1})_j=\\sum_{i_2=1}^n\\cdots\\sum_{i_r=1}^n a_{ji_2\\cdots i_r}x_{i_2}\\cdots x_{i_r} \\qquad\n  \\forall~j=1,\\dots,n.\n\\end{equation*}","sha256_utf8":"f3db71f28a7553d58e3c3aec223373ce8014737c04cc07769bc3e2022cb0d21c"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"单纯形欧氏投影","source_units":[{"start":15676,"end":16070,"start_line":247,"end_line":251,"raw_tex":"It is easy to see that the feasible region $\\Delta$ is a closed convex set. The Euclidean projection\n\\begin{equation*}\n  \\pi_{\\Delta}(\\av) := \\arg\\min\\limits_{\\xv\\in\\Delta} \\|\\xv-\\av\\|\n\\end{equation*}\nexists and is unique for any vector $\\av\\in\\Re^n$. Algorithm \\ref{Alg:Proj} provides an $\\mathcal{O}(n\\log n)$ approach for computing the Euclidean projection onto $\\Delta$ \\cite{WC'13,CY'11}.\n","sha256_utf8":"27e29b4fb2c386b6e42184c33b6834555991a72a93f6093e7966031a7f54812c"}]},{"id":"S009","semantic_type":"algorithm","kind":"algorithm","title":"梯度投影迭代","source_units":[{"start":17262,"end":18056,"start_line":268,"end_line":282,"raw_tex":"\\begin{algorithm}[t]\n\\caption{A gradient projection algorithm (GPA).}\\label{Alg:pGrad}\n\\begin{algorithmic}[1]\n  \\STATE Choose $0<\\underline{\\alpha}\\le\\alpha_0\\le\\overline{\\alpha}$, $\\beta\\in(0,1)$, $\\eta\\in(0,1/2]$, and $\\xv_0\\in\\Delta$. Set $c\\gets0$.\n  \\WHILE{the sequence of iterates does not converge}\n    \\STATE Set $\\gv_c = \\pi_{\\Delta}(\\xv_c + \\alpha_c\\nabla f(\\xv_c))-\\xv_c$.\n    \\STATE Find the smallest nonnegative integer $j$ such that\n      \\begin{equation}\\label{LineSch}\n        f(\\xv_c+\\beta^j\\gv_c)-f(\\xv_c) \\ge \\eta\\beta^j \\gv_c^T \\nabla f(\\xv_c).\n      \\end{equation}\n    \\STATE Set $\\rho_c=\\beta^j$ and $\\xv_{c+1}=\\xv_c+\\rho_c\\gv_c$.\n    \\STATE Choose $\\alpha_{c+1}\\in[\\underline{\\alpha},\\overline{\\alpha}]$ and set $c\\gets c+1$.\n  \\ENDWHILE\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"665febb6e98a95c9be8f8d707d6b3c8b5a488beb0ef9ceb7902a84a656da8d3e"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"临界点定义","source_units":[{"start":18121,"end":18444,"start_line":290,"end_line":293,"raw_tex":"Now, we analyze the convergence of the proposed GPA. Part of our analysis on basic theory is similar to \\cite{Ber-book,HZ'06} and we present here for the purpose of completion. First, we call $\\xv_*\\in\\Delta$ a critical point of \\eqref{OPM} if\n\\begin{equation*}\n  \\pi_{\\Delta}(\\xv_*+\\nabla f(\\xv_*))=\\xv_*.\n\\end{equation*}\n","sha256_utf8":"2829364d2eb405c057201267bf27907d975c0637558e62f9120ccf3c762cc10d"}]},{"id":"S011","semantic_type":"derived_fact","kind":"derived_fact","title":"海森矩阵一致有界","source_units":[{"start":19113,"end":19385,"start_line":305,"end_line":308,"raw_tex":"Because $f(\\xv)=\\frac{1}{r}\\At\\xv^r$ is a polynomial and the simplex $\\Delta$ is compact, the Hessian $\\nabla^2 f(\\xv)$ is bounded, i.e., there exists a constant $M\\ge1$ such that\n\\begin{equation*}\n  \\|\\nabla^2 f(\\xv)\\| \\le M  \\qquad \\forall \\xv\\in\\Delta.\n\\end{equation*}\n","sha256_utf8":"98b34dd27131a4c9df570e741c015b2b1d5e914409529a2c1ee75681c278d8d9"}]},{"id":"S012","semantic_type":"lemma","kind":"lemma","title":"投影方向基本性质","source_units":[{"start":18570,"end":19111,"start_line":297,"end_line":303,"raw_tex":"\\begin{Lem}\\label{Lem:4-1}\n  Let $\\gv^{\\alpha}(\\xv)=\\pi_{\\Delta}(\\xv+\\alpha\\nabla f(\\xv))-\\xv$. Then, \\\\\n  1) $\\|\\gv^{\\alpha}(\\xv)\\|$ is nondecreasing in $\\alpha>0$ for all $\\xv\\in\\Delta$; \\\\\n  2) $\\|\\gv^{\\alpha}(\\xv)\\|/\\alpha$ is nonincreasing in $\\alpha>0$ for all $\\xv\\in\\Delta$; \\\\\n  3) $\\gv^{\\alpha}(\\xv)^T\\nabla f(\\xv)\\ge\\|\\gv^{\\alpha}(\\xv)\\|^2/\\alpha$ for all $\\xv\\in\\Delta$ and $\\alpha>0$; \\\\\n  4) for any $\\xv\\in\\Delta$ and $\\alpha>0$, $\\gv^{\\alpha}(\\xv)=0$ if and only if $\\xv$ is a \\cl{stationary} point for \\eqref{OPM}.\n\\end{Lem}","sha256_utf8":"fa0449afc2ac531baeb5d5225b45970684b144293665805a832d1ec0ce2b0f7b"}]},{"id":"S013","semantic_type":"external_fact","kind":"external_fact","title":"凸集投影标准性质","source_units":[],"declared_external":true,"declaration_text":["Euclidean projection onto a nonempty closed convex set is unique and continuous. For g^alpha(x)=pi_Delta(x+alpha grad f(x))-x, ||g^alpha(x)|| is nondecreasing in alpha>0, ||g^alpha(x)||/alpha is nonincreasing, g^alpha(x)^T grad f(x)>=||g^alpha(x)||^2/alpha, and g^alpha(x)=0 is equivalent to constrained stationarity; the source attributes these properties to HZ'06, Proposition 2.1."]},{"id":"S014","semantic_type":"theorem","kind":"theorem","title":"洛雅谢维奇梯度不等式","source_units":[{"start":22298,"end":22757,"start_line":364,"end_line":370,"raw_tex":"\\begin{Thm}[{\\L}ojasiewicz property]\\label{Thm:5-1}\n  Suppose that $\\xv_*$ is a critical point of \\eqref{OPM}, i.e., $\\gv^1(\\xv_*)=0$. Then, there exist a neighborhood of $\\xv_*$ denoted as $\\mathds{U}(\\xv_*)$, an exponent $\\theta\\in[1/2,1)$, and a positive constant $C$ such that the following inequality\n  \\begin{equation*}\n    \\|\\gv^1(\\xv)\\| \\ge C|f(\\xv)-f(\\xv_*)|^{\\theta}\n  \\end{equation*}\n  holds for all $\\xv\\in \\mathds{U}(\\xv_*) \\cap\\Delta$.\n\\end{Thm}","sha256_utf8":"769b5e90c15a5b93f7cf2588d436671fb92246b3fc185c89c728f80a61026b5a"}]},{"id":"S015","semantic_type":"derived_fact","kind":"derived_fact","title":"目标与约束的半代数性","source_units":[{"start":22061,"end":22297,"start_line":362,"end_line":362,"raw_tex":"Because the objective function and the constraint set of \\eqref{OPM} are semi-algebraic, the following {\\L}ojasiewicz gradient inequality is valid \\cite{Loj-63,BDL-07}. The analysis of this subsection is based on work in \\cite{ABRS10}.\n","sha256_utf8":"bd70b37079727b73d7988b9e615f1544182173a4c216c73c8eb035cb519d8055"}]},{"id":"S016","semantic_type":"external_fact","kind":"external_fact","title":"引用的局部梯度界","source_units":[],"declared_external":true,"declaration_text":["The cited Lojasiewicz property is taken in the exact form used by the authors: for each critical point x_*, some neighborhood U(x_*), exponent theta in [1/2,1), and C>0 satisfy ||g^1(x)||>=C|f(x)-f(x_*)|^theta for every x in U(x_*) intersected with Delta. The source attributes this boundary fact to Loj-63 and BDL-07."]},{"id":"S017","semantic_type":"external_fact","kind":"external_fact","title":"对称张量多项式求导","source_units":[],"declared_external":true,"declaration_text":["For a symmetric order-r tensor A, the polynomial f(x)=(1/r)Ax^r has gradient Ax^(r-1), whose jth component is the contraction sum specified in lines 192-195; this differentiation identity is cited to QL-book and CDQY18."]},{"id":"S018","semantic_type":"lemma","kind":"lemma","title":"回溯步长正下界","source_units":[{"start":19386,"end":19590,"start_line":310,"end_line":316,"raw_tex":"\\begin{Lem}\\label{Lem:4-2}\n  There exists a constant $\\kappa_1>0$ such that for any $\\rho_c$ generated by Algorithm \\ref{Alg:pGrad}\n  \\begin{equation*}\n    \\rho_c\\ge \\kappa_1.\n  \\end{equation*}\n\n\\end{Lem}","sha256_utf8":"fe6acc361db6e12c25984980b9e408f64332754da98b1f1ec8ff03b34649d62d"}]},{"id":"S019","semantic_type":"proof","kind":"proof","title":"步长下界证明","source_units":[{"start":19591,"end":20575,"start_line":317,"end_line":330,"raw_tex":"\\begin{proof}\n  We will prove that the inequality \\eqref{LineSch} is valid\n  %\\begin{equation*}\n  %  f(\\xv_c+\\rho\\gv_c) \\le f(\\xv_c)+\\eta\\rho\\gv_c^T \\nabla f(\\xv_c)\n  %\\end{equation*}\n  when $\\rho\\in[0, \\frac{2(1-\\eta)}{\\overline{\\alpha}M}]$. By Lemma \\ref{Lem:4-1}, it yields that $\\|\\gv_c\\|^2\\le\\alpha_c\\gv_c^T\\nabla f(\\xv_c)$. From Taylor's formula, we have\n  \\begin{eqnarray*}\n    f(\\xv_c+\\rho\\gv_c) &\\ge& f(\\xv_c)+\\rho \\gv_c^T\\nabla f(\\xv_c)-\\frac{M}{2}\\rho^2\\|\\gv_c\\|^2 \\\\\n      &\\ge& f(\\xv_c)+\\rho \\gv_c^T\\nabla f(\\xv_c) -\\frac{M}{2}\\rho^2\\alpha_c\\gv_c^T\\nabla f(\\xv_c) \\\\\n      &\\ge& f(\\xv_c)+\\rho \\gv_c^T\\nabla f(\\xv_c) -(1-\\eta)\\rho\\gv_c^T\\nabla f(\\xv_c) \\\\\n      &=& f(\\xv_c)+\\eta\\rho \\gv_c^T\\nabla f(\\xv_c),\n  \\end{eqnarray*}\n  where the last inequality holds owing to the assumption $0\\le\\rho\\alpha_cM\\le \\rho\\overline{\\alpha}M\\le 2(1-\\eta)$. According to Step 4 of Algorithm GPA,  this lemma is valid if $\\kappa_1:=\\frac{2\\beta(1-\\eta)}{\\overline{\\alpha}M}$.\n\\end{proof}","sha256_utf8":"6fcb18cae177c9c3a104f3f4fba0d2f90f29c8c9c914a3980f94bc5f901fdf7d"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"泰勒展开保证充分上升","source_units":[{"start":19591,"end":20329,"start_line":317,"end_line":328,"raw_tex":"\\begin{proof}\n  We will prove that the inequality \\eqref{LineSch} is valid\n  %\\begin{equation*}\n  %  f(\\xv_c+\\rho\\gv_c) \\le f(\\xv_c)+\\eta\\rho\\gv_c^T \\nabla f(\\xv_c)\n  %\\end{equation*}\n  when $\\rho\\in[0, \\frac{2(1-\\eta)}{\\overline{\\alpha}M}]$. By Lemma \\ref{Lem:4-1}, it yields that $\\|\\gv_c\\|^2\\le\\alpha_c\\gv_c^T\\nabla f(\\xv_c)$. From Taylor's formula, we have\n  \\begin{eqnarray*}\n    f(\\xv_c+\\rho\\gv_c) &\\ge& f(\\xv_c)+\\rho \\gv_c^T\\nabla f(\\xv_c)-\\frac{M}{2}\\rho^2\\|\\gv_c\\|^2 \\\\\n      &\\ge& f(\\xv_c)+\\rho \\gv_c^T\\nabla f(\\xv_c) -\\frac{M}{2}\\rho^2\\alpha_c\\gv_c^T\\nabla f(\\xv_c) \\\\\n      &\\ge& f(\\xv_c)+\\rho \\gv_c^T\\nabla f(\\xv_c) -(1-\\eta)\\rho\\gv_c^T\\nabla f(\\xv_c) \\\\\n      &=& f(\\xv_c)+\\eta\\rho \\gv_c^T\\nabla f(\\xv_c),\n  \\end{eqnarray*}\n","sha256_utf8":"3ee3e8e86ebe0f5153ad7e05c2423ff6368c3ecfdec9519b57219a622f7b9e4e"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"由回溯规则确定常数","source_units":[{"start":20329,"end":20575,"start_line":329,"end_line":330,"raw_tex":"  where the last inequality holds owing to the assumption $0\\le\\rho\\alpha_cM\\le \\rho\\overline{\\alpha}M\\le 2(1-\\eta)$. According to Step 4 of Algorithm GPA,  this lemma is valid if $\\kappa_1:=\\frac{2\\beta(1-\\eta)}{\\overline{\\alpha}M}$.\n\\end{proof}","sha256_utf8":"9a76d591c42d4bd0b898cb73710d0773b739edbb10883823578f99b36273ad42"}]},{"id":"S022","semantic_type":"theorem","kind":"theorem","title":"投影残差趋零","source_units":[{"start":20577,"end":20893,"start_line":332,"end_line":338,"raw_tex":"\\begin{Thm}\\label{Thm:4-3}\n  Suppose that $\\{\\xv_c\\}$ is an infinity sequence of iterates generated by GPA. Then, we have\n  \\begin{equation*}\n    \\lim_{c\\to\\infty} \\|\\pi_{\\Delta}(\\xv_c + \\nabla f(\\xv_c))-\\xv_c\\| = 0.\n  \\end{equation*}\n  That is to say, every limit point of $\\{\\xv_c\\}$ is a critical point.\n\\end{Thm}","sha256_utf8":"02dd25e57fae188776cc4da320c12e09990e15c1489d246b7f8a2f1eee6b7f81"}]},{"id":"S023","semantic_type":"proof","kind":"proof","title":"残差趋零证明","source_units":[{"start":20894,"end":21984,"start_line":339,"end_line":357,"raw_tex":"\\begin{proof}\n  From \\eqref{LineSch}, Lemmas \\ref{Lem:4-1} and \\ref{Lem:4-2}, we obtain\n  \\begin{equation}\\label{fun-lowbd}\n    f(\\xv_{c+1})-f(\\xv_c) \\ge \\eta\\rho_c\\gv_c^T\\nabla f(\\xv_c) \\ge \\frac{\\eta\\rho_c}{\\alpha_c}\\|\\gv_c\\|^2\n      \\ge \\frac{\\eta\\kappa_1}{\\overline{\\alpha}}\\|\\gv_c\\|^2.\n  \\end{equation}\n  Hence, we have\n  \\begin{equation*}\n    \\sum_{c=1}^{\\infty} \\|\\gv_c\\|^2 \\le \\frac{\\overline{\\alpha}}{\\eta\\kappa_1}\\sum_{c=1}^{\\infty}[f(\\xv_{c+1})-f(\\xv_c)]\n       \\le \\frac{\\overline{\\alpha}}{\\eta\\kappa_1}\\lambda(G),\n  \\end{equation*}\n  which means that $\\|\\gv_c\\| \\to0$ as $c\\to\\infty$.\n\n  On one hand, if $\\alpha_c\\ge1$, we get $\\|\\gv^{\\alpha_c}(\\xv_c)\\|\\ge\\|\\gv^1(\\xv_c)\\|$ by Lemma \\ref{Lem:4-1}. On the other hand, $\\alpha_c<1$ and hence $\\|\\gv^{\\alpha_c}(\\xv_c)\\|/\\alpha_c\\ge\\|\\gv^1(\\xv_c)\\|$. Hence, we have\n  \\begin{equation}\\label{gc-lowbd}\n    \\|\\gv_c\\|=\\|\\gv^{\\alpha_c}(\\xv_c)\\|\\ge \\min\\{\\alpha_c,1\\}\\|\\gv^1(\\xv_c)\\| \\ge \\min\\{\\underline{\\alpha},1\\}\\|\\gv^1(\\xv_c)\\|.\n  \\end{equation}\n Therefore $\\|\\gv^1(\\xv_c)\\|$ tends to zero. The proof is then completed.\n\\end{proof}","sha256_utf8":"d9aba4fbbf0c724bcee59b09aae1583fcbc45d8d82fd41b85a34509e3e8fd2b0"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"目标增量控制方向平方","source_units":[{"start":20894,"end":21202,"start_line":339,"end_line":344,"raw_tex":"\\begin{proof}\n  From \\eqref{LineSch}, Lemmas \\ref{Lem:4-1} and \\ref{Lem:4-2}, we obtain\n  \\begin{equation}\\label{fun-lowbd}\n    f(\\xv_{c+1})-f(\\xv_c) \\ge \\eta\\rho_c\\gv_c^T\\nabla f(\\xv_c) \\ge \\frac{\\eta\\rho_c}{\\alpha_c}\\|\\gv_c\\|^2\n      \\ge \\frac{\\eta\\kappa_1}{\\overline{\\alpha}}\\|\\gv_c\\|^2.\n  \\end{equation}\n","sha256_utf8":"d0171df8309a7aa00640db08b21717050938e876dee88692716e29e048fe2c7f"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"方向平方可求和","source_units":[{"start":21202,"end":21492,"start_line":345,"end_line":350,"raw_tex":"  Hence, we have\n  \\begin{equation*}\n    \\sum_{c=1}^{\\infty} \\|\\gv_c\\|^2 \\le \\frac{\\overline{\\alpha}}{\\eta\\kappa_1}\\sum_{c=1}^{\\infty}[f(\\xv_{c+1})-f(\\xv_c)]\n       \\le \\frac{\\overline{\\alpha}}{\\eta\\kappa_1}\\lambda(G),\n  \\end{equation*}\n  which means that $\\|\\gv_c\\| \\to0$ as $c\\to\\infty$.\n","sha256_utf8":"3c451ac7fc442274fe4649c5a607d1f9bc7fcfd9cd9eee0aa3bf28e4beda3b4a"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"比较不同尺度投影残差","source_units":[{"start":21493,"end":21984,"start_line":352,"end_line":357,"raw_tex":"  On one hand, if $\\alpha_c\\ge1$, we get $\\|\\gv^{\\alpha_c}(\\xv_c)\\|\\ge\\|\\gv^1(\\xv_c)\\|$ by Lemma \\ref{Lem:4-1}. On the other hand, $\\alpha_c<1$ and hence $\\|\\gv^{\\alpha_c}(\\xv_c)\\|/\\alpha_c\\ge\\|\\gv^1(\\xv_c)\\|$. Hence, we have\n  \\begin{equation}\\label{gc-lowbd}\n    \\|\\gv_c\\|=\\|\\gv^{\\alpha_c}(\\xv_c)\\|\\ge \\min\\{\\alpha_c,1\\}\\|\\gv^1(\\xv_c)\\| \\ge \\min\\{\\underline{\\alpha},1\\}\\|\\gv^1(\\xv_c)\\|.\n  \\end{equation}\n Therefore $\\|\\gv^1(\\xv_c)\\|$ tends to zero. The proof is then completed.\n\\end{proof}","sha256_utf8":"342cb2c4a126e6151511b01a60691a737232edb75ec9d82415cb4a326d6847a5"}]},{"id":"S027","semantic_type":"lemma","kind":"lemma","title":"局部封闭与有限路径长","source_units":[{"start":22760,"end":23521,"start_line":373,"end_line":382,"raw_tex":"\\begin{Lem}\\label{Lem:5-2}\n  Let $\\xv_*$ be a limiting point of $\\{\\xv_c\\}$. The initial iterate $\\xv_0$ is sufficiently close to $\\xv_*$ in the sense that $\\xv_0\\in\\mathds{B}(\\xv_*,\\sigma)\\cap\\Delta$, where $\\mathds{B}(\\xv_*,\\sigma):=\\{\\xv\\in\\Re^n:\\|\\xv-\\xv_*\\|<\\sigma\\}\\subseteq\\mathds{U}(\\xv_*)$ and $\\sigma\\ge \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C(1-\\theta)}|f(\\xv_0)-f(\\xv_*)|^{1-\\theta}+\\|\\xv_0-\\xv_*\\|$. Then, we have the following two assertions:\n  \\begin{equation*}\n    \\xv_c\\in\\mathds{B}(\\xv_*,\\sigma)\\cap\\Delta \\qquad\\text{ for }c=0,1,2,\\dots,\n  \\end{equation*}\n  and\n  \\begin{equation*}\n    \\sum_{c=0}^{\\infty}\\|\\xv_c-\\xv_{c+1}\\| \\le \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C(1-\\theta)}|f(\\xv_0)-f(\\xv_*)|^{1-\\theta}.\n  \\end{equation*}\n\\end{Lem}","sha256_utf8":"6003f1f0161564b289bb0e06e9b9384bb910d80e3757f0d44d31d821751057d5"}]},{"id":"S028","semantic_type":"assumption","kind":"assumption","title":"局部初值与半径条件","source_units":[{"start":22787,"end":23219,"start_line":374,"end_line":374,"raw_tex":"  Let $\\xv_*$ be a limiting point of $\\{\\xv_c\\}$. The initial iterate $\\xv_0$ is sufficiently close to $\\xv_*$ in the sense that $\\xv_0\\in\\mathds{B}(\\xv_*,\\sigma)\\cap\\Delta$, where $\\mathds{B}(\\xv_*,\\sigma):=\\{\\xv\\in\\Re^n:\\|\\xv-\\xv_*\\|<\\sigma\\}\\subseteq\\mathds{U}(\\xv_*)$ and $\\sigma\\ge \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C(1-\\theta)}|f(\\xv_0)-f(\\xv_*)|^{1-\\theta}+\\|\\xv_0-\\xv_*\\|$. Then, we have the following two assertions:\n","sha256_utf8":"4dfe3866cd66253ad34edd6e8f0ee0e29e1f7c1a95c5bfb7b33a6d5f3cc45a8b"}]},{"id":"S029","semantic_type":"proof","kind":"proof","title":"有限路径长引理证明","source_units":[{"start":23522,"end":25945,"start_line":383,"end_line":417,"raw_tex":"\\begin{proof}\nThe conclusions can be  proved by  induction. It is obvious to see that $\\xv_0\\in\\mathds{B}(\\xv_*,\\sigma)\\cap\\Delta$. Next, by supposing that there exists a positive integer $\\ell$ such that\n  \\begin{equation*}\n    \\xv_c\\in\\mathds{B}(\\xv_*,\\sigma)\\cap\\Delta \\qquad\\text{ for }c=0,1,\\dots,\\ell,\n  \\end{equation*}\n  we show $\\xv_{\\ell+1}\\in\\mathds{B}(\\xv_*,\\sigma)\\cap\\Delta$.\n\n  Define $\\phi(t):=\\frac{1}{C(1-\\theta)}|t-f(\\xv_*)|^{1-\\theta}$. It is easy to see that $\\phi(t)$ is a monotonically decreasing and  \\cl{concave} function for  $t<f(\\xv_*)$. Then, for $0\\le c\\le \\ell$, we have\n  \\begin{eqnarray*}\n    \\phi(f(\\xv_c))-\\phi(f(\\xv_{c+1}))\n      &\\ge& \\phi'(f(\\xv_c))(f(\\xv_c)-f(\\xv_{c+1})) \\\\\n      &=& \\frac{-1}{C|f(\\xv_c)-f(\\xv_*)|^{\\theta}} (f(\\xv_c)-f(\\xv_{c+1})) \\\\\n      &\\ge& \\frac{1}{\\|\\gv^1(\\xv_c)\\|}(f(\\xv_{c+1})-f(\\xv_c)) \\\\\n      &\\ge& \\frac{1}{\\|\\gv^1(\\xv_c)\\|}\\frac{\\eta\\rho_c}{\\alpha_c}\\|\\gv_c\\|^2 \\\\\n      &\\ge& \\frac{1}{\\|\\gv^1(\\xv_c)\\|}\\frac{\\eta\\rho_c}{\\alpha_c}\\|\\gv_c\\|\\min\\{\\alpha_c,1\\}\\|\\gv^1(\\xv_c)\\| \\\\\n      &=& \\frac{\\eta\\min\\{\\alpha_c,1\\}}{\\alpha_c}\\|\\rho_c\\gv_c\\| \\\\\n      &=& \\eta\\min\\{1,\\alpha_c^{-1}\\}\\|\\xv_c-\\xv_{c+1}\\|,\n  \\end{eqnarray*}\n  where the second inequality is obtained based on Theorem \\ref{Thm:5-1}, the third inequality is deduced from \\eqref{fun-lowbd}, and the fourth inequality is valid because of \\eqref{gc-lowbd}. The above inequalities indicate that\n  \\begin{equation*}\n    \\|\\xv_c-\\xv_{c+1}\\| \\le \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta}\\left(\\phi(f(\\xv_c))-\\phi(f(\\xv_{c+1}))\\right),\n  \\end{equation*}\n  for $0\\le c\\le \\ell$. Hence, it holds that\n  \\begin{equation}\\label{iterate-length}\n    \\sum_{c=0}^{\\ell} \\|\\xv_c-\\xv_{c+1}\\|\n      \\le \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta}\\sum_{c=0}^{\\ell}\\left(\\phi(f(\\xv_c))-\\phi(f(\\xv_{c+1}))\\right)\n      \\le \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta}\\phi(f(\\xv_0)).\n  \\end{equation}\n  Therefore, we have\n  \\begin{equation*}\n    \\|\\xv_{\\ell+1}-\\xv_*\\| \\le \\|\\xv_0-\\xv_*\\|+\\sum_{c=0}^{\\ell} \\|\\xv_c-\\xv_{c+1}\\|\n      \\le \\|\\xv_0-\\xv_*\\|+\\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta}\\phi(f(\\xv_0)) <\\sigma,\n  \\end{equation*}\n  which implies that $\\xv_{\\ell+1}\\in\\mathds{B}(\\xv_*,\\sigma)$. On the other hand, every iterate is feasible by the mechanism of GPA.  Hence, we obtain the first assertion. The second  conclusion comes from \\eqref{iterate-length} straightforwardly by setting $\\ell\\to\\infty$.\n\\end{proof}","sha256_utf8":"58c05f451e3677264edd649fcc78f972cc05ef41f8d0db6311f15b96f7959b58"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"建立邻域归纳假设","source_units":[{"start":23522,"end":23911,"start_line":383,"end_line":388,"raw_tex":"\\begin{proof}\nThe conclusions can be  proved by  induction. It is obvious to see that $\\xv_0\\in\\mathds{B}(\\xv_*,\\sigma)\\cap\\Delta$. Next, by supposing that there exists a positive integer $\\ell$ such that\n  \\begin{equation*}\n    \\xv_c\\in\\mathds{B}(\\xv_*,\\sigma)\\cap\\Delta \\qquad\\text{ for }c=0,1,\\dots,\\ell,\n  \\end{equation*}\n  we show $\\xv_{\\ell+1}\\in\\mathds{B}(\\xv_*,\\sigma)\\cap\\Delta$.\n","sha256_utf8":"8ceed4a072648b69ba35c374d7b0695ecc6a947207c57f7a6a7e3982ac6029cf"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"凹函数差控制迭代位移","source_units":[{"start":23912,"end":25099,"start_line":390,"end_line":404,"raw_tex":"  Define $\\phi(t):=\\frac{1}{C(1-\\theta)}|t-f(\\xv_*)|^{1-\\theta}$. It is easy to see that $\\phi(t)$ is a monotonically decreasing and  \\cl{concave} function for  $t<f(\\xv_*)$. Then, for $0\\le c\\le \\ell$, we have\n  \\begin{eqnarray*}\n    \\phi(f(\\xv_c))-\\phi(f(\\xv_{c+1}))\n      &\\ge& \\phi'(f(\\xv_c))(f(\\xv_c)-f(\\xv_{c+1})) \\\\\n      &=& \\frac{-1}{C|f(\\xv_c)-f(\\xv_*)|^{\\theta}} (f(\\xv_c)-f(\\xv_{c+1})) \\\\\n      &\\ge& \\frac{1}{\\|\\gv^1(\\xv_c)\\|}(f(\\xv_{c+1})-f(\\xv_c)) \\\\\n      &\\ge& \\frac{1}{\\|\\gv^1(\\xv_c)\\|}\\frac{\\eta\\rho_c}{\\alpha_c}\\|\\gv_c\\|^2 \\\\\n      &\\ge& \\frac{1}{\\|\\gv^1(\\xv_c)\\|}\\frac{\\eta\\rho_c}{\\alpha_c}\\|\\gv_c\\|\\min\\{\\alpha_c,1\\}\\|\\gv^1(\\xv_c)\\| \\\\\n      &=& \\frac{\\eta\\min\\{\\alpha_c,1\\}}{\\alpha_c}\\|\\rho_c\\gv_c\\| \\\\\n      &=& \\eta\\min\\{1,\\alpha_c^{-1}\\}\\|\\xv_c-\\xv_{c+1}\\|,\n  \\end{eqnarray*}\n  where the second inequality is obtained based on Theorem \\ref{Thm:5-1}, the third inequality is deduced from \\eqref{fun-lowbd}, and the fourth inequality is valid because of \\eqref{gc-lowbd}. The above inequalities indicate that\n  \\begin{equation*}\n    \\|\\xv_c-\\xv_{c+1}\\| \\le \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta}\\left(\\phi(f(\\xv_c))-\\phi(f(\\xv_{c+1}))\\right),\n  \\end{equation*}\n","sha256_utf8":"daa48bf22e44f0a6b467df3d52e0000179d75f0546fb834babd5af3f09d6938c"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"望远镜求和控制路径长","source_units":[{"start":25099,"end":25424,"start_line":405,"end_line":410,"raw_tex":"  for $0\\le c\\le \\ell$. Hence, it holds that\n  \\begin{equation}\\label{iterate-length}\n    \\sum_{c=0}^{\\ell} \\|\\xv_c-\\xv_{c+1}\\|\n      \\le \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta}\\sum_{c=0}^{\\ell}\\left(\\phi(f(\\xv_c))-\\phi(f(\\xv_{c+1}))\\right)\n      \\le \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta}\\phi(f(\\xv_0)).\n  \\end{equation}\n","sha256_utf8":"ca56be3417203b1e72d730ad18ba58c8b73b4f8448831a4a32726837dafa7759"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"保持邻域并取无限和","source_units":[{"start":25424,"end":25945,"start_line":411,"end_line":417,"raw_tex":"  Therefore, we have\n  \\begin{equation*}\n    \\|\\xv_{\\ell+1}-\\xv_*\\| \\le \\|\\xv_0-\\xv_*\\|+\\sum_{c=0}^{\\ell} \\|\\xv_c-\\xv_{c+1}\\|\n      \\le \\|\\xv_0-\\xv_*\\|+\\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta}\\phi(f(\\xv_0)) <\\sigma,\n  \\end{equation*}\n  which implies that $\\xv_{\\ell+1}\\in\\mathds{B}(\\xv_*,\\sigma)$. On the other hand, every iterate is feasible by the mechanism of GPA.  Hence, we obtain the first assertion. The second  conclusion comes from \\eqref{iterate-length} straightforwardly by setting $\\ell\\to\\infty$.\n\\end{proof}","sha256_utf8":"20056316e8e2badf7ce81d349b47cea5c79261d27126e6a37daae063da48b3dc"}]},{"id":"S034","semantic_type":"theorem","kind":"theorem","title":"梯度投影法收敛速率","source_units":[{"start":27222,"end":27931,"start_line":439,"end_line":452,"raw_tex":"\\begin{Thm}\\label{Thm:5-4}\n  Suppose GPA generates an infinite sequence of iteration points $\\{\\xv_c\\}$ that converges to a critical point $\\xv_*$. Then, we have the following estimations on convergence rate.\n  \\begin{itemize}\n    \\item If $\\theta=1/2$, there exist $\\gamma>0$ and $\\mu\\in(0,1)$ such that\n      \\begin{equation*}\n        \\|\\xv_c-\\xv_*\\| \\le \\gamma\\mu^c.\n      \\end{equation*}\n    \\item If $\\theta\\in(1/2,1)$, there exist $\\gamma_1>0$ and $\\gamma_2>0$ such that\n      \\begin{equation*}\n        |f(\\xv_c)-f(\\xv_*)| \\le \\gamma_1 c^{-\\frac{1}{2\\theta-1}} \\qquad\\text{ and }\\qquad\n        \\|\\xv_c-\\xv_*\\| \\le \\gamma_2 c^{-\\frac{1-\\theta}{2\\theta-1}}.\n      \\end{equation*}\n  \\end{itemize}\n\\end{Thm}","sha256_utf8":"d8c63ca1fabe403de815b82f2369c09a54a574e21946623a18fec9b30b063130"}]},{"id":"S035","semantic_type":"assumption","kind":"assumption","title":"无限迭代收敛至临界点","source_units":[{"start":27249,"end":27431,"start_line":440,"end_line":440,"raw_tex":"  Suppose GPA generates an infinite sequence of iteration points $\\{\\xv_c\\}$ that converges to a critical point $\\xv_*$. Then, we have the following estimations on convergence rate.\n","sha256_utf8":"1cc757b2816588e4e3f963a589ac3744da783befc68c0af72919231f7ce2d1b7"}]},{"id":"S036","semantic_type":"proof","kind":"proof","title":"收敛速率完整证明","source_units":[{"start":27932,"end":32007,"start_line":453,"end_line":527,"raw_tex":"\\begin{proof}\n  Without loss of generality, we assume $\\xv_0\\in\\mathds{B}(\\xv_*,\\sigma)$. Define\n  \\begin{equation*}\n    \\zeta_c:=\\sum_{\\ell=c}^{\\infty} \\|\\xv_{\\ell}-\\xv_{\\ell+1}\\| \\ge \\|\\xv_c-\\xv_*\\|.\n  \\end{equation*}\n  From Lemma \\ref{Lem:5-2} and Theorem \\ref{Thm:5-1}, we have\n  \\begin{eqnarray*}\n    \\zeta_c &\\le& \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C(1-\\theta)}|f(\\xv_c)-f(\\xv_*)|^{1-\\theta} \\\\\n      &=& \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C^{1/\\theta}(1-\\theta)}\\left(C|f(\\xv_c)-f(\\xv_*)|^{\\theta}\\right)^{(1-\\theta)/\\theta} \\\\\n      &\\le& \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C^{1/\\theta}(1-\\theta)}\\left(\\|\\gv^1(\\xv_c)\\|\\right)^{(1-\\theta)/\\theta}.\n  \\end{eqnarray*}\n  It can be obtained from Lemma \\ref{Lem:4-2} and \\eqref{gc-lowbd} that\n  \\begin{equation*}\n    \\|\\xv_c-\\xv_{c+1}\\| = \\rho_c\\|\\gv_c\\| \\ge \\kappa_1\\min\\{\\underline{\\alpha},1\\}\\|\\gv^1(\\xv_c)\\|.\n  \\end{equation*}\n  Hence, by denoting $\\kappa_2:=\\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C^{1/\\theta}(1-\\theta)}(\\kappa_1\\min\\{\\underline{\\alpha},1\\})^{-(1-\\theta)/\\theta}$, we get\n  \\begin{equation}\\label{zeta-1}\n    \\zeta_c \\le \\kappa_2\\|\\xv_c-\\xv_{c+1}\\|^{(1-\\theta)/\\theta} = \\kappa_2(\\zeta_c-\\zeta_{c+1})^{(1-\\theta)/\\theta}.\n  \\end{equation}\n  If $\\theta=1/2$, the inequality \\eqref{zeta-1} means\n  \\begin{equation*}\n    \\zeta_{c+1} \\le \\frac{\\kappa_2-1}{\\kappa_2}\\zeta_c.\n  \\end{equation*}\n  Hence, the first estimation holds with $\\gamma=\\zeta_0$ and $\\mu=(\\kappa_2-1)/\\kappa_2$.\n\n  Next, we consider the case $\\theta\\in(1/2,1)$. Let $\\varphi(t):=t^{-\\theta/(1-\\theta)}$ be a decreasing function for $t>0$. It yields from \\eqref{zeta-1} that\n  \\begin{eqnarray*}\n    \\kappa_2^{-\\theta/(1-\\theta)} &\\le& \\varphi(\\zeta_c)(\\zeta_c-\\zeta_{c+1}) \\\\\n      &=& \\int_{\\zeta_{c+1}}^{\\zeta_c} \\varphi(\\zeta_c) \\mathrm{d}t \\\\\n      &\\le& \\int_{\\zeta_{c+1}}^{\\zeta_c} \\varphi(t) \\mathrm{d}t \\\\\n      &=& \\cl{\\frac{1-\\theta}{1-2\\theta}}\\left(\\zeta_{c+1}^{-(2\\theta-1)/(1-\\theta)}-\\zeta_c^{-(2\\theta-1)/(1-\\theta)}\\right),\n  \\end{eqnarray*}\n  which implies\n  \\begin{equation*}\n    \\zeta_{c+1}^{-(2\\theta-1)/(1-\\theta)}-\\zeta_c^{-(2\\theta-1)/(1-\\theta)}\n      \\ge \\cl{\\frac{1-2\\theta}{1-\\theta}}\\kappa_2^{-\\theta/(1-\\theta)} := \\kappa_3>0.\n  \\end{equation*}\nThen, we have\n  \\begin{equation*}\n    \\zeta_c^{-(2\\theta-1)/(1-\\theta)} \\ge \\kappa_3+\\zeta_{c-1}^{-(2\\theta-1)/(1-\\theta)} \\ge\\dots\n      \\ge \\kappa_3c+\\zeta_0^{-(2\\theta-1)/(1-\\theta)},\n  \\end{equation*}\n  which means\n  \\begin{equation*}\n    \\zeta_c \\le \\left(\\kappa_3c+\\zeta_0^{-(2\\theta-1)/(1-\\theta)}\\right)^{-(1-\\theta)/(2\\theta-1)}\n      \\le (\\kappa_3c)^{-(1-\\theta)/(2\\theta-1)}.\n  \\end{equation*}\nThe last estimation holds by taking $\\gamma_2=\\kappa_3^{-(1-\\theta)/(2\\theta-1)}$.\n\n   Let $\\xi_c:=|f(\\xv_c)-f(\\xv_*)|$. From \\eqref{fun-lowbd}, \\eqref{gc-lowbd}, and Theorem \\ref{Thm:5-1}, we have\n  \\begin{eqnarray*}\n    \\xi_c-\\xi_{c+1} &=& f(\\xv_{c+1})-f(\\xv_c) \\\\\n      &\\ge& \\frac{\\eta\\kappa_1}{\\overline{\\alpha}}\\|\\gv_c\\|^2 \\\\\n      &\\ge& \\frac{\\eta\\kappa_1(\\min\\{\\underline{\\alpha},1\\})^2}{\\overline{\\alpha}}\\|\\gv^1(\\xv_c)\\|^2 \\\\\n      &\\ge& \\frac{\\eta\\kappa_1C^2(\\min\\{\\underline{\\alpha},1\\})^2}{\\overline{\\alpha}}\\xi_c^{2\\theta}\n      := \\kappa_4 \\xi_c^{2\\theta}.\n  \\end{eqnarray*}\nDenote $\\chi(t):=t^{-2\\theta}$ as a decreasing function for $t>0$. It is deduced from the above inequality that\n  \\begin{equation*}\n    \\kappa_4 \\le \\chi(\\xi_c)(\\xi_c-\\xi_{c+1})\n      = \\int_{\\xi_{c+1}}^{\\xi_c} \\chi(\\xi_c) \\mathrm{d}t\n      \\le \\int_{\\xi_{c+1}}^{\\xi_c} \\chi(t) \\mathrm{d}t\n      = \\frac{1}{2\\theta-1}\\left(\\xi_{c+1}^{-(2\\theta-1)}-\\xi_c^{-(2\\theta-1)}\\right).\n  \\end{equation*}\n  Let $\\kappa_5:=(2\\theta-1)\\kappa_4>0$. We have\n  \\begin{equation*}\n    \\xi_c^{-(2\\theta-1)} \\ge \\kappa_5+\\xi_{c-1}^{-(2\\theta-1)} \\ge \\dots \\ge\\kappa_5c+\\xi_0^{-(2\\theta-1)},\n  \\end{equation*}\n  which means\n  \\begin{equation*}\n    \\xi_c \\le \\left(\\kappa_5c+\\xi_0^{-(2\\theta-1)}\\right)^{-1/(2\\theta-1)}\n      \\le (\\kappa_5c)^{-1/(2\\theta-1)}.\n  \\end{equation*}\nThe second estimation is valid if we take $\\gamma_1=\\kappa_5^{-1/(2\\theta-1)}$.\n\\end{proof}","sha256_utf8":"db6d0df09293056c55798421701a5aef8857d13e127d37d905263706732f9950"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"定义尾部路径长","source_units":[{"start":27932,"end":28152,"start_line":453,"end_line":457,"raw_tex":"\\begin{proof}\n  Without loss of generality, we assume $\\xv_0\\in\\mathds{B}(\\xv_*,\\sigma)$. Define\n  \\begin{equation*}\n    \\zeta_c:=\\sum_{\\ell=c}^{\\infty} \\|\\xv_{\\ell}-\\xv_{\\ell+1}\\| \\ge \\|\\xv_c-\\xv_*\\|.\n  \\end{equation*}\n","sha256_utf8":"8d581ed1c9f1a9fcef8c89d0a6c2ccd5ae4ecf6ff5fc026f1b5f1678f38f4f2e"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"尾路径长由残差控制","source_units":[{"start":28152,"end":28621,"start_line":458,"end_line":463,"raw_tex":"  From Lemma \\ref{Lem:5-2} and Theorem \\ref{Thm:5-1}, we have\n  \\begin{eqnarray*}\n    \\zeta_c &\\le& \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C(1-\\theta)}|f(\\xv_c)-f(\\xv_*)|^{1-\\theta} \\\\\n      &=& \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C^{1/\\theta}(1-\\theta)}\\left(C|f(\\xv_c)-f(\\xv_*)|^{\\theta}\\right)^{(1-\\theta)/\\theta} \\\\\n      &\\le& \\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C^{1/\\theta}(1-\\theta)}\\left(\\|\\gv^1(\\xv_c)\\|\\right)^{(1-\\theta)/\\theta}.\n  \\end{eqnarray*}\n","sha256_utf8":"a2cf68a3dfe921e8d4293384b65a2af6f9f5106ccc81aaf357b7812dda76b44d"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"单步位移控制投影残差","source_units":[{"start":28621,"end":28831,"start_line":464,"end_line":467,"raw_tex":"  It can be obtained from Lemma \\ref{Lem:4-2} and \\eqref{gc-lowbd} that\n  \\begin{equation*}\n    \\|\\xv_c-\\xv_{c+1}\\| = \\rho_c\\|\\gv_c\\| \\ge \\kappa_1\\min\\{\\underline{\\alpha},1\\}\\|\\gv^1(\\xv_c)\\|.\n  \\end{equation*}\n","sha256_utf8":"feedd0e12b81c5cc9fcfb4b220f2c1e0d0281b269ed7e7ff620b013db464c4ca"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"建立尾路径递推不等式","source_units":[{"start":28831,"end":29162,"start_line":468,"end_line":471,"raw_tex":"  Hence, by denoting $\\kappa_2:=\\frac{\\max\\{1,\\overline{\\alpha}\\}}{\\eta C^{1/\\theta}(1-\\theta)}(\\kappa_1\\min\\{\\underline{\\alpha},1\\})^{-(1-\\theta)/\\theta}$, we get\n  \\begin{equation}\\label{zeta-1}\n    \\zeta_c \\le \\kappa_2\\|\\xv_c-\\xv_{c+1}\\|^{(1-\\theta)/\\theta} = \\kappa_2(\\zeta_c-\\zeta_{c+1})^{(1-\\theta)/\\theta}.\n  \\end{equation}\n","sha256_utf8":"72b58cd63ca66f495e494a660d3c593ba59948b639d49022f1a82b5a6b92e3d2"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"二分之一指数给出线性率","source_units":[{"start":29162,"end":29402,"start_line":472,"end_line":476,"raw_tex":"  If $\\theta=1/2$, the inequality \\eqref{zeta-1} means\n  \\begin{equation*}\n    \\zeta_{c+1} \\le \\frac{\\kappa_2-1}{\\kappa_2}\\zeta_c.\n  \\end{equation*}\n  Hence, the first estimation holds with $\\gamma=\\zeta_0$ and $\\mu=(\\kappa_2-1)/\\kappa_2$.\n","sha256_utf8":"999e8b3e17e778b46717b54845dd228dd1afeb60d49dc431eb107c14c696e986"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"积分比较得到幂次差界","source_units":[{"start":29403,"end":30164,"start_line":478,"end_line":489,"raw_tex":"  Next, we consider the case $\\theta\\in(1/2,1)$. Let $\\varphi(t):=t^{-\\theta/(1-\\theta)}$ be a decreasing function for $t>0$. It yields from \\eqref{zeta-1} that\n  \\begin{eqnarray*}\n    \\kappa_2^{-\\theta/(1-\\theta)} &\\le& \\varphi(\\zeta_c)(\\zeta_c-\\zeta_{c+1}) \\\\\n      &=& \\int_{\\zeta_{c+1}}^{\\zeta_c} \\varphi(\\zeta_c) \\mathrm{d}t \\\\\n      &\\le& \\int_{\\zeta_{c+1}}^{\\zeta_c} \\varphi(t) \\mathrm{d}t \\\\\n      &=& \\cl{\\frac{1-\\theta}{1-2\\theta}}\\left(\\zeta_{c+1}^{-(2\\theta-1)/(1-\\theta)}-\\zeta_c^{-(2\\theta-1)/(1-\\theta)}\\right),\n  \\end{eqnarray*}\n  which implies\n  \\begin{equation*}\n    \\zeta_{c+1}^{-(2\\theta-1)/(1-\\theta)}-\\zeta_c^{-(2\\theta-1)/(1-\\theta)}\n      \\ge \\cl{\\frac{1-2\\theta}{1-\\theta}}\\kappa_2^{-\\theta/(1-\\theta)} := \\kappa_3>0.\n  \\end{equation*}\n","sha256_utf8":"15e22df6d1ed05a8f2a7239d2f4e5cc95b26e8da446e7f55908d04a0e42180c1"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"累积尾路径幂次差","source_units":[{"start":30164,"end":30369,"start_line":490,"end_line":494,"raw_tex":"Then, we have\n  \\begin{equation*}\n    \\zeta_c^{-(2\\theta-1)/(1-\\theta)} \\ge \\kappa_3+\\zeta_{c-1}^{-(2\\theta-1)/(1-\\theta)} \\ge\\dots\n      \\ge \\kappa_3c+\\zeta_0^{-(2\\theta-1)/(1-\\theta)},\n  \\end{equation*}\n","sha256_utf8":"de7d84ec934593186edaf702381a5a0c0e92f82be077ee4273f1793ce408aa62"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"得到迭代点次线性率","source_units":[{"start":30369,"end":30652,"start_line":495,"end_line":500,"raw_tex":"  which means\n  \\begin{equation*}\n    \\zeta_c \\le \\left(\\kappa_3c+\\zeta_0^{-(2\\theta-1)/(1-\\theta)}\\right)^{-(1-\\theta)/(2\\theta-1)}\n      \\le (\\kappa_3c)^{-(1-\\theta)/(2\\theta-1)}.\n  \\end{equation*}\nThe last estimation holds by taking $\\gamma_2=\\kappa_3^{-(1-\\theta)/(2\\theta-1)}$.\n","sha256_utf8":"d1488eb8ec7b789ba5048e4a0ced093e82bee6d8d37d3b56bf28020c386abebe"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"建立目标误差递推界","source_units":[{"start":30653,"end":31159,"start_line":502,"end_line":509,"raw_tex":"   Let $\\xi_c:=|f(\\xv_c)-f(\\xv_*)|$. From \\eqref{fun-lowbd}, \\eqref{gc-lowbd}, and Theorem \\ref{Thm:5-1}, we have\n  \\begin{eqnarray*}\n    \\xi_c-\\xi_{c+1} &=& f(\\xv_{c+1})-f(\\xv_c) \\\\\n      &\\ge& \\frac{\\eta\\kappa_1}{\\overline{\\alpha}}\\|\\gv_c\\|^2 \\\\\n      &\\ge& \\frac{\\eta\\kappa_1(\\min\\{\\underline{\\alpha},1\\})^2}{\\overline{\\alpha}}\\|\\gv^1(\\xv_c)\\|^2 \\\\\n      &\\ge& \\frac{\\eta\\kappa_1C^2(\\min\\{\\underline{\\alpha},1\\})^2}{\\overline{\\alpha}}\\xi_c^{2\\theta}\n      := \\kappa_4 \\xi_c^{2\\theta}.\n  \\end{eqnarray*}\n","sha256_utf8":"7338fc2b221c2349bfbdc6dc546a2680f64a217dcc82aefd6b20fca11e19eb9b"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"积分比较目标误差幂次","source_units":[{"start":31159,"end":31554,"start_line":510,"end_line":516,"raw_tex":"Denote $\\chi(t):=t^{-2\\theta}$ as a decreasing function for $t>0$. It is deduced from the above inequality that\n  \\begin{equation*}\n    \\kappa_4 \\le \\chi(\\xi_c)(\\xi_c-\\xi_{c+1})\n      = \\int_{\\xi_{c+1}}^{\\xi_c} \\chi(\\xi_c) \\mathrm{d}t\n      \\le \\int_{\\xi_{c+1}}^{\\xi_c} \\chi(t) \\mathrm{d}t\n      = \\frac{1}{2\\theta-1}\\left(\\xi_{c+1}^{-(2\\theta-1)}-\\xi_c^{-(2\\theta-1)}\\right).\n  \\end{equation*}\n","sha256_utf8":"205a73d5f68f9bb5052badad679de6ab23e614b8b9f158646f5e36a799cf4506"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"累积目标误差幂次差","source_units":[{"start":31554,"end":31749,"start_line":517,"end_line":520,"raw_tex":"  Let $\\kappa_5:=(2\\theta-1)\\kappa_4>0$. We have\n  \\begin{equation*}\n    \\xi_c^{-(2\\theta-1)} \\ge \\kappa_5+\\xi_{c-1}^{-(2\\theta-1)} \\ge \\dots \\ge\\kappa_5c+\\xi_0^{-(2\\theta-1)},\n  \\end{equation*}\n","sha256_utf8":"ecd653f7ce2f3bac7a62ddfd833d6a5cac0d48d684f404600b2441ba7d961ed7"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"得到目标值次线性率","source_units":[{"start":31749,"end":32007,"start_line":521,"end_line":527,"raw_tex":"  which means\n  \\begin{equation*}\n    \\xi_c \\le \\left(\\kappa_5c+\\xi_0^{-(2\\theta-1)}\\right)^{-1/(2\\theta-1)}\n      \\le (\\kappa_5c)^{-1/(2\\theta-1)}.\n  \\end{equation*}\nThe second estimation is valid if we take $\\gamma_1=\\kappa_5^{-1/(2\\theta-1)}$.\n\\end{proof}","sha256_utf8":"53055fca9e569ff39040ecc5bc2501a7f235ac9212275be3f41c479e347be42a"}]},{"id":"S049","semantic_type":"algorithm","kind":"algorithm","title":"排序阈值投影算法","source_units":[{"start":16072,"end":16538,"start_line":254,"end_line":262,"raw_tex":"\\begin{algorithm}[t]\n\\caption{Euclidean projection $\\pi_{\\Delta}(\\av)$.}\\label{Alg:Proj}\n\\begin{algorithmic}[1]\n  \\STATE Sort the input vector $\\av\\in\\Re^n$ into an auxiliary vector $\\bv$ such that $b_1\\ge b_2\\ge \\dots\\ge b_n$.\n  \\STATE Find the largest index $\\ell=\\max\\{j: jb_j+1-\\sum_{i=1}^j b_i>0, 1\\le j\\le n\\}$.\n  \\STATE Set $\\lambda = (1-\\sum_{i=1}^{\\ell} b_i)/{\\ell}$.\n  \\STATE Output $\\xv$ with $x_i=\\max\\{a_i+\\lambda,0\\}$.\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"da0fb1330554044f45193fe0a3250c512ba08d2ec9dfaca191d85b9f36ffc068"}]},{"id":"S050","semantic_type":"external_fact","kind":"external_fact","title":"单纯形投影算法正确性","source_units":[],"declared_external":true,"declaration_text":["Sorting a vector, selecting the largest index satisfying the threshold test, and applying the common threshold as in B0024 computes its Euclidean projection onto Delta in O(n log n) time; the source cites WC'13 and CY'11."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S034","edges":[{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S001","target_id":"S006","kind":"supports"},{"source_id":"S003","target_id":"S006","kind":"supports"},{"source_id":"S006","target_id":"S005","kind":"supports"},{"source_id":"S005","target_id":"S004","kind":"supports"},{"source_id":"S004","target_id":"S007","kind":"supports"},{"source_id":"S001","target_id":"S007","kind":"supports"},{"source_id":"S003","target_id":"S007","kind":"supports"},{"source_id":"S017","target_id":"S007","kind":"supports"},{"source_id":"S001","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S049","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S009","kind":"supports"},{"source_id":"S007","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S007","target_id":"S010","kind":"supports"},{"source_id":"S008","target_id":"S010","kind":"supports"},{"source_id":"S001","target_id":"S011","kind":"supports"},{"source_id":"S007","target_id":"S011","kind":"supports"},{"source_id":"S007","target_id":"S012","kind":"supports"},{"source_id":"S008","target_id":"S012","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S001","target_id":"S015","kind":"supports"},{"source_id":"S007","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S016","target_id":"S014","kind":"supports"},{"source_id":"S010","target_id":"S014","kind":"supports"},{"source_id":"S012","target_id":"S014","kind":"supports"},{"source_id":"S009","target_id":"S020","kind":"supports"},{"source_id":"S011","target_id":"S020","kind":"supports"},{"source_id":"S012","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S009","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S009","target_id":"S024","kind":"supports"},{"source_id":"S012","target_id":"S024","kind":"supports"},{"source_id":"S018","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S001","target_id":"S025","kind":"supports"},{"source_id":"S007","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S012","target_id":"S026","kind":"supports"},{"source_id":"S009","target_id":"S026","kind":"supports"},{"source_id":"S010","target_id":"S026","kind":"supports"},{"source_id":"S013","target_id":"S026","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S023","kind":"supports"},{"source_id":"S026","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S014","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S022","target_id":"S031","kind":"supports"},{"source_id":"S014","target_id":"S031","kind":"supports"},{"source_id":"S024","target_id":"S031","kind":"supports"},{"source_id":"S026","target_id":"S031","kind":"supports"},{"source_id":"S009","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S028","target_id":"S033","kind":"supports"},{"source_id":"S009","target_id":"S033","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S032","target_id":"S029","kind":"supports"},{"source_id":"S033","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S027","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S027","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S027","target_id":"S038","kind":"supports"},{"source_id":"S014","target_id":"S038","kind":"supports"},{"source_id":"S018","target_id":"S039","kind":"supports"},{"source_id":"S026","target_id":"S039","kind":"supports"},{"source_id":"S009","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S037","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S037","target_id":"S044","kind":"supports"},{"source_id":"S035","target_id":"S045","kind":"supports"},{"source_id":"S024","target_id":"S045","kind":"supports"},{"source_id":"S026","target_id":"S045","kind":"supports"},{"source_id":"S014","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S039","target_id":"S036","kind":"supports"},{"source_id":"S040","target_id":"S036","kind":"supports"},{"source_id":"S041","target_id":"S036","kind":"supports"},{"source_id":"S042","target_id":"S036","kind":"supports"},{"source_id":"S043","target_id":"S036","kind":"supports"},{"source_id":"S044","target_id":"S036","kind":"supports"},{"source_id":"S045","target_id":"S036","kind":"supports"},{"source_id":"S046","target_id":"S036","kind":"supports"},{"source_id":"S047","target_id":"S036","kind":"supports"},{"source_id":"S048","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S034","kind":"supports"}]}}
{"record_id":"2303.06838v3","paper":{"title":"Sample Complexity Analysis for Adaptive Optimization Algorithms with Stochastic Oracles","authors":["Billy Jin","Katya Scheinberg","Miaolan Xie"],"arxiv_id":"2303.06838","version":"v3","journal":"Mathematical Programming","doi":"10.1007/s10107-024-02078-z","abstract_url":"https://arxiv.org/abs/2303.06838v3","pdf_url":"https://arxiv.org/pdf/2303.06838v3"},"context":{"raw_tex":"We consider a continuous optimization problem of the form\n\\begin{equation}\\label{prob.opt}\n  \\min_{x \\in \\R^{m}}\\ \\phi(x),\n\\end{equation}\n\nwhere $\\phi$ is possibly non-convex, (twice-)continuously differentiable with Lipschitz continuous derivatives.\nNeither function values $\\phi(x)$, nor gradients $\\nabla \\phi(x)$ are assumed to be directly computable. Instead, given any $x \\in \\R^{m}$, it is assumed that stochastic estimates of $\\phi(x)$, $\\nabla \\phi(x)$, and possibly $\\nabla^2 \\phi(x)$  can be computed, and these estimates may possess different levels of \\emph{accuracy} and \\emph{reliability} depending on the particular setting of interest.  \n\n\n\\begin{algorithm}\n\t\\caption{~\\textbf{Algorithmic Framework for Adaptive Stochastic Optimization}}\n\t\\label{alg:generic_stochastic}\n\t\n\t{\\bf 0. Initialization} \n\t\n\t\\vspace{0.5mm}\n\t\n\t\\hspace{0.5cm} Choose $\\mytheta \\in (0,1)$, $\\gamma\\in (0, 1)$, $\\alpha_{\\max} \\in (0,\\infty)$, $x_0 \\in \\R^{m}$, and $\\alpha_0 \\in (0,\\alpha_{\\max}]$.  Set $k \\gets 0$.\n\t\\vspace{0.5mm}\n\t\n\t{\\bf 1. Determine model and compute step}\n\t\n\t\\vspace{0.5mm}\n\t\n\t\\hspace{0.5cm} Construct a stochastic model $m_k$ of $\\phi$ at $x_k$ using $f(x_k,\\xi_{0,k})$, $g(x_k,\\xi_{1,k})$, and (optionally) $H(x_k,\\xi_{2,k})$ from \\emph{probabilistic zeroth-, first-, and (optionally) second-order oracles}.  Compute $s_k(\\alpha_k)$ such that the model reduction $m_k(x_k) - m_k(x_k+s_k(\\alpha_k)) \\geq 0$ is sufficiently large.\n\t\\vspace{0.5mm}\n\t\n\t{\\bf 2. Check for sufficient reduction}\n\t\n\t\\vspace{0.5mm}\n\t\n\t\\hspace{0.5cm} Set $x_k^+ \\gets x_k + s_k(\\alpha_k)$ and compute $f(x_k^+,\\xi_{0,k}^+)$ as a stochastic estimate of $\\phi(x_k^+)$ using a \\emph{probabilistic zeroth-order oracle}.  Check if $ f(x_k,\\xi_{0,k})- f(x_k^+,\\xi_{0,k}^+)$ is sufficiently large (e.g., relative to the model reduction $m_k(x_k)-m_k(x_k^+)$) using a condition parameterized by~$\\mytheta$.\n\t\n\t\\vspace{0.5mm}\n\t{\\bf 3. Successful iteration}\n\t\n\t\\vspace{0.5mm}\n\t\n\t\\hspace{0.5cm} If sufficient reduction has been attained (along with other potential requirements), then set $x_{k+1} \\gets x_k^+ $ and $\\alpha_{k+1} \\gets \\min\\{\\gamma^{-1}\\alpha_k,\\alpha_{\\max}\\}$.\n\t\\vspace{0.5mm}\n\t\n\t{\\bf 4. Unsuccessful iteration}\n\t\n\t\\vspace{0.5mm}\n\t\n\t\\hspace{0.5cm} Otherwise, set $x_{k+1} \\gets x_k$ and $\\alpha_{k+1} \\gets \\gamma \\alpha_k$.\n\t\\vspace{0.5mm}\n\t\n\t{\\bf 5. Next iteration}\n\t\n\t\\vspace{0.5mm}\n\t\n\t\\hspace{0.5cm}  Set $k \\gets k + 1$ and go to Step~1.\n\t\\vspace{0.5mm}\n\\end{algorithm}\n\n\\begin{definition}{\\bf Stochastic $j$th-order oracle.} Given an input  $x \\in \\R^{n}$ and the step size parameter $\\alpha$, the oracle computes $\\varphi_j(x,\\xi_j)$, an estimate of the $j^{\\mathrm{th}}$-order derivative $\\nabla^j \\phi(x)$. In all the algorithms we consider, $\\|\\varphi_j(x,\\xi_j)- \\nabla^j\\phi(x)\\|$ is assumed to be bounded by some quantity (which will be a function of  $\\alpha$), with probability at least $1-\\delta_j$.\n%  where $\\delta_j$ is an intrinsic value. \n   Here, $\\xi_j$ is a random variable defined on probability space $(\\Omega_j, {\\cal F}_j, P_j)$ whose distribution depends on the input  $x$ and $\\alpha$, and $\\delta_j$ is intrinsic to the oracle.\n%    and $\\P_{\\xi_j}$ denotes probability w.r.t.~that distribution. \n    The cost of the oracle is also a  function of $x$ and $\\alpha$. \n  \\end{definition}\n\nSpecifically, let $(X_k)_{k\\geq 0}$ be the random iterates with realizations $x_k$, let $(G_k)_{k\\geq 0}$ be the gradient estimates with realizations $g_k$, and let $(\\mathcal{A}_k)_{k \\geq 0}$ be the step size parameter values with realizations $\\alpha_k$. The prior works that analyze different methods \\rev{belonging to the framework of Algorithm~\\ref{alg:generic_stochastic}}  define this stochastic process rigorously, with appropriate filtrations. Here for brevity, we will omit those details, as we do not use them in the analysis. \n\n\n\\begin{defn}[Stopping time]\n\tFor $\\varepsilon > 0$, let $T_\\varepsilon$ be the first time such that a specified optimality condition is satisfied. For all the settings considered in this paper,\n\t${T_\\varepsilon}=\\min\\{k: \\norm{\\nabla \\phi(x_k)} \\leq \\varepsilon\\}$ if $\\phi$ is non-convex, and ${T_\\varepsilon}=\\min\\{k: \\phi(x_k) - \\inf_x \\phi(x)  \\leq \\varepsilon\\}$ if $\\phi$ is strongly convex. We will refer to $T_\\varepsilon$ as the \\emph{stopping time} of the algorithm.\n\\end{defn}\n\n\\begin{assumption}\n\t[Properties of the stochastic process generated by the adaptive stochastic algorithm]\n\t\\label{ass:alg_behave}\n\tThe random sequence of parameters $\\mathcal{A}_k$ generated by the algorithm satisfies the following: \n\t\\begin{itemize}\n\t\t\\item[(i)] For all $k$, $\\mathcal{A}_k \\in \\{\\gamma \\mathcal{A}_{k-1}, \\min\\{\\amax, \\gamma^{-1}\\mathcal{A}_{k-1}\\}\\}$, and\n\t\t\\item[(ii)] There exist constants $\\bar\\alpha > 0$, and $p>\\frac 12$, such that \\rev{for all iterations $k$,\n%\t\t for all iterations  $k < T_\\varepsilon$, if  $\\mathcal{A}_k  \\leq  \\bar \\alpha$ then\n\t\t$$\\P(\\mathcal{A}_{k+1} = \\gamma^{-1} \\mathcal{A}_{k} \\mid \\mathcal{F}_{k}, \\, k < T_\\varepsilon, \\, \\mathcal{A}_k  \\leq  \\bar \\alpha ) \\geq p.$$}\n\t\tHere, $\\mathcal{F}_k$ denotes the filtration generated by the algorithm up to iteration $k$.\n\t\\end{itemize}       \n\n\\end{assumption}\n\nThroughout the remainder of this paper, we will use $q$ to denote $1-p$.\n\n\nLet $\\{\\A_k\\}_{k=0}^\\infty$ denote the random sequence of parameter values (whose realization is $\\{\\alpha_k\\}_{k=0}^\\infty$), for  Algorithm \\ref{alg:generic_stochastic}.  Let us assume, WLOG, that $\\A_0 = \\gamma^j \\bar{\\alpha}$, for some integer $j\\leq0$.  (Recall here that $0 <\\gamma < 1$.) Then we observe that $\\mathcal{A}_k = \\gamma^{\\bar{Y}_k}\\bar{\\alpha}$,   where  $\\{\\bar{Y}_k\\}_{k=0}^\\infty$ is a random sequence of  integers, with $\\bar{Y}_0 =j\\leq 0$, which increases by one on every unsuccessful step, and decreases by one on every successful step. Moreover, by \nAssumption  \\ref{ass:alg_behave}, whenever $k < T_\\eps$ and $\\bar{Y}_k \\geq 0$, the probability that it decreases by one is at least $p$. Define $Y_k$ as follows:\n\\begin{equation}\n\\label{eq:Yk}\nY_k = \\bar{Y}_k \\quad \\text{if $k \\leq T_\\eps$},\n\\qquad\nY_k = \n\\begin{cases}\nY_{k-1} - 1 &\\text{w.p. $p$} \\\\\nY_{k-1} + 1 &\\text{w.p. $1-p$} \n\\end{cases}\n\\quad\n\\text{if $k > T_\\eps$.} \n\\end{equation}\n\nIn other words, $Y_k$ follows the algorithm until $T_\\eps$, and then behaves like a random walk with downward drift $p$ after $T_\\eps$.\nWe now couple  $\\{Y_k\\}_{k=0}^\\infty$  with a random walk $\\{Z_k\\}_{k=0}^\\infty$ which stochastically dominates $Y_k$. \n\n\nConsider the following one-sided random walk $\\{Z_k\\}_{k=0}^\\infty$, defined  on the non-negative integers. \n\\begin{equation}\n\\label{eq:randwalk_update}\nZ_0 = 0, \\qquad Z_{k+1} = \\begin{cases}\n\tZ_k +1, &\\text{w.p. $1-p$,}\\\\\n\tZ_k-1, &\\text{w.p. $p$, if $Z_k \\geq 1$,}\\\\\n\t0, &\\text{w.p. $p$, if $Z_k = 0$.}\n\\end{cases}\n\\end{equation}\n\n\\begin{defn}\nLet $N(l, n)$ be the random variable that denotes the number of times $Z_k=l$  in the first $n$ steps of the random walk.  \n%When $N(l, n)>0$, we say state $l$ is visited by the random walk. \n\\end{defn}\n\nBy definition of $N(l, n)$, we have $N(l, n) > 0$ if and only if state $l$ is visited in the first $n$ steps of the random walk. The next proposition  upper bounds the probability that $N(l, n) > 0$. \n\n\nThe cost of an oracle call may depend on the step size parameter $\\alpha$ and the probability parameter $1-\\delta$, thus we denote the cost by $\\sc(\\alpha, 1-\\delta)$. \nWe will use $\\sc(\\alpha)$ in the paper to simplify the notation because for all algorithms in the class, $\\delta$ can be treated as a constant.\n Moreover, the cost of an oracle call is a non-increasing function of  $\\alpha$ for all algorithms developed so far that fit into the framework. \n\n\n\\begin{assumption}\n\t\\label{ass:monotone}\n%\t\\item For a fixed set of probability parameters of the stochastic oracles, the oracle costs of all the oracle calls at iteration $k$ only depend on the parameter $\\alpha_k$. We assume $\\sc(\\alpha)$ increases as $\\alpha$ decreases.\n  $\\sc(\\alpha)$ is non-increasing in $\\alpha$.\n\\end{assumption}\n\n\\begin{defn}[Total Oracle Complexity]\nFor a positive integer $n$, let $\\SC(n)$ be the random variable which denotes the total oracle complexity of running the algorithm for $n$ iterations. In other words,\n$$\\SC(n) = \\sum_{k=1}^n \\sc(\\mathcal{A}_k).$$\n\\end{defn}","units":[{"start":15221,"end":15358,"start_line":221,"end_line":224,"raw_tex":"We consider a continuous optimization problem of the form\n\\begin{equation}\\label{prob.opt}\n  \\min_{x \\in \\R^{m}}\\ \\phi(x),\n\\end{equation}","sha256_utf8":"ca12e075dd569515355d9a00e84c5c70eb634a2a5fdeecf009587fb823fa40ae"},{"start":15359,"end":15875,"start_line":225,"end_line":226,"raw_tex":"where $\\phi$ is possibly non-convex, (twice-)continuously differentiable with Lipschitz continuous derivatives.\nNeither function values $\\phi(x)$, nor gradients $\\nabla \\phi(x)$ are assumed to be directly computable. Instead, given any $x \\in \\R^{m}$, it is assumed that stochastic estimates of $\\phi(x)$, $\\nabla \\phi(x)$, and possibly $\\nabla^2 \\phi(x)$  can be computed, and these estimates may possess different levels of \\emph{accuracy} and \\emph{reliability} depending on the particular setting of interest.  \n","sha256_utf8":"8a04084eb594e9e0567a14dd385843594c6d24595c8b7001c8a913da3d509b37"},{"start":19267,"end":21078,"start_line":251,"end_line":296,"raw_tex":"\\begin{algorithm}\n\t\\caption{~\\textbf{Algorithmic Framework for Adaptive Stochastic Optimization}}\n\t\\label{alg:generic_stochastic}\n\t\n\t{\\bf 0. Initialization} \n\t\n\t\\vspace{0.5mm}\n\t\n\t\\hspace{0.5cm} Choose $\\mytheta \\in (0,1)$, $\\gamma\\in (0, 1)$, $\\alpha_{\\max} \\in (0,\\infty)$, $x_0 \\in \\R^{m}$, and $\\alpha_0 \\in (0,\\alpha_{\\max}]$.  Set $k \\gets 0$.\n\t\\vspace{0.5mm}\n\t\n\t{\\bf 1. Determine model and compute step}\n\t\n\t\\vspace{0.5mm}\n\t\n\t\\hspace{0.5cm} Construct a stochastic model $m_k$ of $\\phi$ at $x_k$ using $f(x_k,\\xi_{0,k})$, $g(x_k,\\xi_{1,k})$, and (optionally) $H(x_k,\\xi_{2,k})$ from \\emph{probabilistic zeroth-, first-, and (optionally) second-order oracles}.  Compute $s_k(\\alpha_k)$ such that the model reduction $m_k(x_k) - m_k(x_k+s_k(\\alpha_k)) \\geq 0$ is sufficiently large.\n\t\\vspace{0.5mm}\n\t\n\t{\\bf 2. Check for sufficient reduction}\n\t\n\t\\vspace{0.5mm}\n\t\n\t\\hspace{0.5cm} Set $x_k^+ \\gets x_k + s_k(\\alpha_k)$ and compute $f(x_k^+,\\xi_{0,k}^+)$ as a stochastic estimate of $\\phi(x_k^+)$ using a \\emph{probabilistic zeroth-order oracle}.  Check if $ f(x_k,\\xi_{0,k})- f(x_k^+,\\xi_{0,k}^+)$ is sufficiently large (e.g., relative to the model reduction $m_k(x_k)-m_k(x_k^+)$) using a condition parameterized by~$\\mytheta$.\n\t\n\t\\vspace{0.5mm}\n\t{\\bf 3. Successful iteration}\n\t\n\t\\vspace{0.5mm}\n\t\n\t\\hspace{0.5cm} If sufficient reduction has been attained (along with other potential requirements), then set $x_{k+1} \\gets x_k^+ $ and $\\alpha_{k+1} \\gets \\min\\{\\gamma^{-1}\\alpha_k,\\alpha_{\\max}\\}$.\n\t\\vspace{0.5mm}\n\t\n\t{\\bf 4. Unsuccessful iteration}\n\t\n\t\\vspace{0.5mm}\n\t\n\t\\hspace{0.5cm} Otherwise, set $x_{k+1} \\gets x_k$ and $\\alpha_{k+1} \\gets \\gamma \\alpha_k$.\n\t\\vspace{0.5mm}\n\t\n\t{\\bf 5. Next iteration}\n\t\n\t\\vspace{0.5mm}\n\t\n\t\\hspace{0.5cm}  Set $k \\gets k + 1$ and go to Step~1.\n\t\\vspace{0.5mm}\n\\end{algorithm}","sha256_utf8":"c910b1d372e891b39ba121814a4cf65d26f5564979ed536704c17ff7c53260a6"},{"start":24865,"end":25704,"start_line":309,"end_line":314,"raw_tex":"\\begin{definition}{\\bf Stochastic $j$th-order oracle.} Given an input  $x \\in \\R^{n}$ and the step size parameter $\\alpha$, the oracle computes $\\varphi_j(x,\\xi_j)$, an estimate of the $j^{\\mathrm{th}}$-order derivative $\\nabla^j \\phi(x)$. In all the algorithms we consider, $\\|\\varphi_j(x,\\xi_j)- \\nabla^j\\phi(x)\\|$ is assumed to be bounded by some quantity (which will be a function of  $\\alpha$), with probability at least $1-\\delta_j$.\n%  where $\\delta_j$ is an intrinsic value. \n   Here, $\\xi_j$ is a random variable defined on probability space $(\\Omega_j, {\\cal F}_j, P_j)$ whose distribution depends on the input  $x$ and $\\alpha$, and $\\delta_j$ is intrinsic to the oracle.\n%    and $\\P_{\\xi_j}$ denotes probability w.r.t.~that distribution. \n    The cost of the oracle is also a  function of $x$ and $\\alpha$. \n  \\end{definition}","sha256_utf8":"473785e5e6b1280dfb57b059b82bb78d27338517cc6fc3d0f9dc127d02a9dcc5"},{"start":38834,"end":39374,"start_line":458,"end_line":458,"raw_tex":"Specifically, let $(X_k)_{k\\geq 0}$ be the random iterates with realizations $x_k$, let $(G_k)_{k\\geq 0}$ be the gradient estimates with realizations $g_k$, and let $(\\mathcal{A}_k)_{k \\geq 0}$ be the step size parameter values with realizations $\\alpha_k$. The prior works that analyze different methods \\rev{belonging to the framework of Algorithm~\\ref{alg:generic_stochastic}}  define this stochastic process rigorously, with appropriate filtrations. Here for brevity, we will omit those details, as we do not use them in the analysis. \n","sha256_utf8":"362c5191121502b6332fc127fedfef6764e4e2cce385bcba848575f63a842dad"},{"start":40236,"end":40723,"start_line":463,"end_line":466,"raw_tex":"\\begin{defn}[Stopping time]\n\tFor $\\varepsilon > 0$, let $T_\\varepsilon$ be the first time such that a specified optimality condition is satisfied. For all the settings considered in this paper,\n\t${T_\\varepsilon}=\\min\\{k: \\norm{\\nabla \\phi(x_k)} \\leq \\varepsilon\\}$ if $\\phi$ is non-convex, and ${T_\\varepsilon}=\\min\\{k: \\phi(x_k) - \\inf_x \\phi(x)  \\leq \\varepsilon\\}$ if $\\phi$ is strongly convex. We will refer to $T_\\varepsilon$ as the \\emph{stopping time} of the algorithm.\n\\end{defn}","sha256_utf8":"bad78ef4ecab607f80c6d54d5154912a72c1e4b946aec17bbb6f5c2144e8f6d5"},{"start":41015,"end":41873,"start_line":473,"end_line":485,"raw_tex":"\\begin{assumption}\n\t[Properties of the stochastic process generated by the adaptive stochastic algorithm]\n\t\\label{ass:alg_behave}\n\tThe random sequence of parameters $\\mathcal{A}_k$ generated by the algorithm satisfies the following: \n\t\\begin{itemize}\n\t\t\\item[(i)] For all $k$, $\\mathcal{A}_k \\in \\{\\gamma \\mathcal{A}_{k-1}, \\min\\{\\amax, \\gamma^{-1}\\mathcal{A}_{k-1}\\}\\}$, and\n\t\t\\item[(ii)] There exist constants $\\bar\\alpha > 0$, and $p>\\frac 12$, such that \\rev{for all iterations $k$,\n%\t\t for all iterations  $k < T_\\varepsilon$, if  $\\mathcal{A}_k  \\leq  \\bar \\alpha$ then\n\t\t$$\\P(\\mathcal{A}_{k+1} = \\gamma^{-1} \\mathcal{A}_{k} \\mid \\mathcal{F}_{k}, \\, k < T_\\varepsilon, \\, \\mathcal{A}_k  \\leq  \\bar \\alpha ) \\geq p.$$}\n\t\tHere, $\\mathcal{F}_k$ denotes the filtration generated by the algorithm up to iteration $k$.\n\t\\end{itemize}       \n\n\\end{assumption}","sha256_utf8":"f87ed7e2ca5505734438270bf9d5bc5dd120ba93e96df91326805099e98c4974"},{"start":42574,"end":42647,"start_line":489,"end_line":489,"raw_tex":"Throughout the remainder of this paper, we will use $q$ to denote $1-p$.\n","sha256_utf8":"7cfdb5da62148199301114a4cbf875e85e24efaa4ba2fcebf09bdb9fa99344ee"},{"start":50689,"end":51659,"start_line":595,"end_line":608,"raw_tex":"Let $\\{\\A_k\\}_{k=0}^\\infty$ denote the random sequence of parameter values (whose realization is $\\{\\alpha_k\\}_{k=0}^\\infty$), for  Algorithm \\ref{alg:generic_stochastic}.  Let us assume, WLOG, that $\\A_0 = \\gamma^j \\bar{\\alpha}$, for some integer $j\\leq0$.  (Recall here that $0 <\\gamma < 1$.) Then we observe that $\\mathcal{A}_k = \\gamma^{\\bar{Y}_k}\\bar{\\alpha}$,   where  $\\{\\bar{Y}_k\\}_{k=0}^\\infty$ is a random sequence of  integers, with $\\bar{Y}_0 =j\\leq 0$, which increases by one on every unsuccessful step, and decreases by one on every successful step. Moreover, by \nAssumption  \\ref{ass:alg_behave}, whenever $k < T_\\eps$ and $\\bar{Y}_k \\geq 0$, the probability that it decreases by one is at least $p$. Define $Y_k$ as follows:\n\\begin{equation}\n\\label{eq:Yk}\nY_k = \\bar{Y}_k \\quad \\text{if $k \\leq T_\\eps$},\n\\qquad\nY_k = \n\\begin{cases}\nY_{k-1} - 1 &\\text{w.p. $p$} \\\\\nY_{k-1} + 1 &\\text{w.p. $1-p$} \n\\end{cases}\n\\quad\n\\text{if $k > T_\\eps$.} \n\\end{equation}","sha256_utf8":"8b0457af2f0afcdbc177c49f85035d8470ea0858b6bbc52e18ef2389d1b84b43"},{"start":51660,"end":51916,"start_line":609,"end_line":610,"raw_tex":"In other words, $Y_k$ follows the algorithm until $T_\\eps$, and then behaves like a random walk with downward drift $p$ after $T_\\eps$.\nWe now couple  $\\{Y_k\\}_{k=0}^\\infty$  with a random walk $\\{Z_k\\}_{k=0}^\\infty$ which stochastically dominates $Y_k$. \n","sha256_utf8":"0c926b55a7b10391ef9afeb5bdbc22a079b75ebf44c8f41e01aba7cd73460563"},{"start":51918,"end":52249,"start_line":613,"end_line":621,"raw_tex":"Consider the following one-sided random walk $\\{Z_k\\}_{k=0}^\\infty$, defined  on the non-negative integers. \n\\begin{equation}\n\\label{eq:randwalk_update}\nZ_0 = 0, \\qquad Z_{k+1} = \\begin{cases}\n\tZ_k +1, &\\text{w.p. $1-p$,}\\\\\n\tZ_k-1, &\\text{w.p. $p$, if $Z_k \\geq 1$,}\\\\\n\t0, &\\text{w.p. $p$, if $Z_k = 0$.}\n\\end{cases}\n\\end{equation}","sha256_utf8":"c59148814327126b2c7cf489ebdf13739f70386bec6b3e530e177e02fc02cbfd"},{"start":55206,"end":55421,"start_line":650,"end_line":653,"raw_tex":"\\begin{defn}\nLet $N(l, n)$ be the random variable that denotes the number of times $Z_k=l$  in the first $n$ steps of the random walk.  \n%When $N(l, n)>0$, we say state $l$ is visited by the random walk. \n\\end{defn}","sha256_utf8":"df7131852603e5895616b4d45949895b159d0a386e52fdedbafec923d4f19534"},{"start":55422,"end":55623,"start_line":654,"end_line":654,"raw_tex":"By definition of $N(l, n)$, we have $N(l, n) > 0$ if and only if state $l$ is visited in the first $n$ steps of the random walk. The next proposition  upper bounds the probability that $N(l, n) > 0$. \n","sha256_utf8":"309dce448502e53f52a4b187768aabbbfcf03a2a7a61163e363f8f02855cbb0f"},{"start":61589,"end":62048,"start_line":742,"end_line":744,"raw_tex":"The cost of an oracle call may depend on the step size parameter $\\alpha$ and the probability parameter $1-\\delta$, thus we denote the cost by $\\sc(\\alpha, 1-\\delta)$. \nWe will use $\\sc(\\alpha)$ in the paper to simplify the notation because for all algorithms in the class, $\\delta$ can be treated as a constant.\n Moreover, the cost of an oracle call is a non-increasing function of  $\\alpha$ for all algorithms developed so far that fit into the framework. \n","sha256_utf8":"90e23e2e8781b2469d16f1f2338bd6feb11d9070adba49b5a8770811f0d80b75"},{"start":62076,"end":62413,"start_line":747,"end_line":751,"raw_tex":"\\begin{assumption}\n\t\\label{ass:monotone}\n%\t\\item For a fixed set of probability parameters of the stochastic oracles, the oracle costs of all the oracle calls at iteration $k$ only depend on the parameter $\\alpha_k$. We assume $\\sc(\\alpha)$ increases as $\\alpha$ decreases.\n  $\\sc(\\alpha)$ is non-increasing in $\\alpha$.\n\\end{assumption}","sha256_utf8":"b0dc53a76b673fd1751abcd88f62048060ad1e02df67a5f87edd6b967fec052d"},{"start":62414,"end":62675,"start_line":752,"end_line":755,"raw_tex":"\\begin{defn}[Total Oracle Complexity]\nFor a positive integer $n$, let $\\SC(n)$ be the random variable which denotes the total oracle complexity of running the algorithm for $n$ iterations. In other words,\n$$\\SC(n) = \\sum_{k=1}^n \\sc(\\mathcal{A}_k).$$\n\\end{defn}","sha256_utf8":"c6f472532529deecf9f481a2626903826e5a7b7fc7aa35bc55a07c1d338d3213"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\n\\label{cor:sc_highprob}\n\\begin{sloppypar}\n\tLet \\Cref{ass:alg_behave} and \\Cref{ass:monotone} hold in Algorithm \\ref{alg:generic_stochastic}.\n\t For any $\\omega  > 0$ and positive integer $n$, with probability at least $1-\\P(T_\\eps> n)- n^{-\\omega } - cn^{-(1+\\omega )}$, $$\\SC(T_{\\eps})\\leq n\\cdot \\sc(\\alpha^*(n)),$$ \n\t where $\\alpha^*(n)= \\bar{\\alpha}\\gamma   n^{-{(1+\\omega)\\log_{1/2q} 1/\\gamma}}$ , and $c$ is as defined in \\Cref{thm:stepsize_bound}.\n\\end{sloppypar}\n\t If $\\gamma $ is chosen to be at least $\\max\\left\\{\\frac12, \\left(\\frac{1}{2q}\\right)^{\\frac{\\log(2\\beta)}{(1+\\omega)\\log n} } \\right\\}$ for some $\\beta < \\frac12$, then $\\alpha^*(n) \\geq \\beta \\bar{\\alpha}$, thus \\rev{with probability at least $1-\\P(T_\\eps> n)- n^{-\\omega } - cn^{-(1+\\omega )}$,}\n\t $$\\SC(T_{\\eps})\\leq n\\cdot \\sc(\\beta \\bar{\\alpha}).$$ \n\\end{theorem}","units":[{"start":66206,"end":67062,"start_line":813,"end_line":822,"raw_tex":"\\begin{theorem}\n\\label{cor:sc_highprob}\n\\begin{sloppypar}\n\tLet \\Cref{ass:alg_behave} and \\Cref{ass:monotone} hold in Algorithm \\ref{alg:generic_stochastic}.\n\t For any $\\omega  > 0$ and positive integer $n$, with probability at least $1-\\P(T_\\eps> n)- n^{-\\omega } - cn^{-(1+\\omega )}$, $$\\SC(T_{\\eps})\\leq n\\cdot \\sc(\\alpha^*(n)),$$ \n\t where $\\alpha^*(n)= \\bar{\\alpha}\\gamma   n^{-{(1+\\omega)\\log_{1/2q} 1/\\gamma}}$ , and $c$ is as defined in \\Cref{thm:stepsize_bound}.\n\\end{sloppypar}\n\t If $\\gamma $ is chosen to be at least $\\max\\left\\{\\frac12, \\left(\\frac{1}{2q}\\right)^{\\frac{\\log(2\\beta)}{(1+\\omega)\\log n} } \\right\\}$ for some $\\beta < \\frac12$, then $\\alpha^*(n) \\geq \\beta \\bar{\\alpha}$, thus \\rev{with probability at least $1-\\P(T_\\eps> n)- n^{-\\omega } - cn^{-(1+\\omega )}$,}\n\t $$\\SC(T_{\\eps})\\leq n\\cdot \\sc(\\beta \\bar{\\alpha}).$$ \n\\end{theorem}","sha256_utf8":"189a89a684aae492cdeb401ba96f5fc9a802bdfd0deedfe96320c6c4212ad69e"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{theorem}\n\\label{thm:stepsize_bound}\n\tSuppose Assumption \\ref{ass:alg_behave} holds for Algorithm \\ref{alg:generic_stochastic}. For any positive integer $n$, any $\\omega  > 0$, with probability at least  $1 - n^{-\\omega } - cn^{-(1+\\omega )}$, we have \n\t$$\\text{either  $T_\\eps < n$ ~or~ $\\min_{1 \\leq k \\leq n} \\alpha_k  \n\t\\geq \\rev{\\alpha^*(n)} :=  \\bar{\\alpha}\\gamma  \\gamma^{{(1+\\omega)\\log_{1/2q} n}}\n=\\bar{\\alpha}\\gamma   n^{-{(1+\\omega)\\log_{1/2q} 1/\\gamma}}$},$$ \twhere $c = \\frac{2\\sqrt{pq}}{(1-2\\sqrt{pq})^2}$ and $q = 1-p$.\n\t\n% \t In other words, with high probability the algorithm will not reach any step size smaller or equal to $ \\gamma^{\\ceil{(1+\\nu )\\log_{p/q} n}}\\bar{\\alpha}$ in the first $n$ steps. \n\\end{theorem}\n\n\\begin{lemma}\n\t\\label{lem:coupling_new}\n\tThere exists a coupling between $Z_k$  and $Y_k$, where $Z_k$ stochastically dominates $Y_k$. \n\\end{lemma}\n\n\\begin{proof}[Proof of Lemma \\ref{lem:coupling_new}]\nInitially, $Z_0=0$ and $Y_0 \\leq 0$. For each $k$, we show how to update $Z_k$ to $Z_{k+1}$ according to how $Y_k$ changes to $Y_{k+1}$. We consider two cases depending on whether $k < T_\\eps$ or $k \\geq T_\\eps$. \n\n\\underline{Case 1: $k < T_\\eps$}.\n If $Y_k\\leq -1$, we update $Z_{k+1}$ from $Z_{k}$ according to \\Cref{eq:randwalk_update}, independently of how $Y_k$ changes to $Y_{k+1}$. If $Y_k \\geq 0$, then we first check if $Y_k$ increased or decreased. Let $p'$ be the probability that $Y_{k+1} = Y_k -1$ on this sample path. Since $Y_k \\geq 0$, we know by \\Cref{ass:alg_behave} that $p' \\geq p$. Now, if $Y_{k+1} = Y_k + 1$, then we set $Z_{k+1} = Z_{k} + 1$. On the other hand, if $Y_{k+1} = Y_k - 1$, then we set $Z_{k+1} = Z_k + 1$ with probability $1 - \\frac{p}{p'}$, and $Z_{k+1} = \\max\\{Z_k-1, 0\\}$ with probability $\\frac{p}{p'}$. Note that these probabilities are well-defined because $p' \\geq p$. \n \n\\underline{Case 2: $k \\geq T_\\eps$}. If $Y_{k+1} = Y_k + 1$, then set $Z_{k+1} = Z_{k}+1$. Otherwise, if $Y_{k+1} = Y_{k}-1$, then set $Z_{k+1} = \\max\\{Z_k-1, 0\\}$. \n\nObserve that under this coupling, $Z_k \\geq Y_k$ on every sample path. Moreover, $\\{Z_k\\}$ and $\\{Y_k\\}$ have the correct marginal distributions. For $Y_k$, this is easy to see, since it evolves according to its true distribution and we are constructing $Z_k$ from it. For $Z_k$, on any step with $k \\geq T_\\eps$, $Z_{k+1}$ evolves from $Z_k$ correctly according to \\Cref{eq:randwalk_update} by construction. On a step with $k < T_\\eps$ there are two cases: 1) $Y_k \\leq -1$,  and 2) $Y_k \\geq 0$. In the first case, the update from $Z_k$ to $Z_{k+1}$ clearly follows \\Cref{eq:randwalk_update}. This is also true in the second case, since there the probability that $Z_{k}$ increases is $(1-p') + p'(1-\\frac{p}{p'}) = 1-p$.\n\nTo summarize, we have exhibited a coupling between $\\{Z_k\\}$ and $\\{Y_k\\}$, under which $Z_k \\geq Y_k$ on any sample path. \n\n%Thus, $\\sA_k = \\gamma^{Y_k}\\bar{\\alpha}$ the step size of the algorithm at iteration $k$, stochastically dominates $\\gamma^{Z_k}\\bar{\\alpha}$.  Thus, to obtain a lower bound on the step sizes of the algorithm, it suffices to obtain an upper bound on the random walk $\\{Z_k\\}$. \n\\end{proof}\n\n\\begin{prop}\\label{prop:Fill}\n\tLet $q=1-p$. We have\n%\t$$\\P(N(l,n) >0 ) \\leq(n-l+1) \\frac{1-(q / p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l}  +\n%\t\\frac{2 p \\rev{2^l(pq)^{l/2}}}{(1-2 \\sqrt{p q})^{2}}\\left(\\frac{q}{p}\\right)^{(l+1) / 2}.$$\n\t$$\\P(N(l,n) >0 ) \\leq(n-l+1) \\frac{1-(q / p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l}  +\n\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}(2q)^l.$$\n\\end{prop}\n\n\\begin{proof}\n%\tLet $q=1-p .$ \n%\tDenote by $p_{l, N}$ the probability that state $l$ is reached in the first $N$ steps. Note that $p_{l, N} = \\P(\\mathcal{N}(l, N) > 0)$ by definition. \n\tFirst, observe that $ \\P({N}(l, n) > 0)$ remains unchanged if we change the state space from $\\{0, 1,2, \\ldots\\}$ to $\\{0, 1,2, \\ldots, l\\}$ and modify the walk to hold in state $l$ with probability $q$ (instead of moving from $l$ to $l+1$ with that probability). This defines a Markov chain on $\\{0,1,2,\\ldots, l\\}$, and let $P$ be its transition matrix. Noting that $P_{0,l}^m$ is the probability that the Markov chain is in state $l$ at time $m$, we see that\n\t\\begin{equation}\n\t\t\\label{eq:subadd}\n\t\t\\P(N(l,n) >0 ) \\leq \\sum_{m=l}^{n} P_{0, l}^{m}.\n\t\\end{equation}\n\t\n\tThe matrix $P$ is explicitly diagonalized in \\cite{feller_book} (Section XVI.3). By $(3.16)$ in that section,\n\t\\begin{equation}\n\t\t\\label{eq:transprob}\n\t\tP_{0,l}^{m}=\\frac{1-(q/p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l} \n\t\t-\\frac{2 q}{l+1}\\left(\\frac{q}{p}\\right)^{\\frac{l-1}{2}} \\sum_{r=1}^{l} \\frac{\\left[\\sin \\frac{\\pi r}{l+1}\\right]\\left[\\sin \\frac{\\pi rl}{l+1}\\right]\\left[2 \\sqrt{pq} \\cos \\frac{\\pi r}{l+1}\\right]^{m}}{1-2 \\sqrt{p q} \\cos \\frac{\\pi r}{l+1}}.\n\t\\end{equation}\n\tThe absolute value of the sum appearing in (\\ref{eq:transprob}) can of course be bounded above by\n\t$$\n\t\\sum_{r=1}^{l} \\frac{(2 \\sqrt{p q})^{m}}{1-2 \\sqrt{p q}}=l \\frac{(2 \\sqrt{p q})^{m}}{1-2 \\sqrt{p q}}\n\t$$\n\tand this readily yields\n\t\\begin{equation}\n\t\t\\label{eq:fillprop_final}\n\t\tP_{0, l}^{m} \\leq \\frac{1-(q/p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l} \n\t\t+\n\t\t\\frac{2 p}{1-2 \\sqrt{p q}}\\left(\\frac{q}{p}\\right)^{\\frac{l+1}{2}}(2 \\sqrt{p q})^{m}.\n\t\\end{equation}\n\tSumming (\\ref{eq:fillprop_final}) over $m=l, \\ldots, n$ and using (\\ref{eq:subadd}), we obtain the bound on $\\P(N(l,n) >0 )$ claimed in the proposition.\n \n\\end{proof}\n\n\\begin{proof}[Proof of \\Cref{thm:stepsize_bound}]\n\tLet $q=1-p$. By Proposition \\ref{prop:Fill}, we have:\n\t$$\\P(N(l,n) >0 ) \\leq(n-l+1) \\frac{1-(q / p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l}  +\n\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}(2q)^l.$$\n\tIn other words,\n\t\\begin{align*}\n\t\t\\P(N(l,n) =0 ) &\\geq 1-(n-l+1) \\frac{1-(q / p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l}  -\n\t\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}(2q)^l \\\\\n\t\t&\\geq 1- n \\left(\\frac{q}{p}\\right)^{l}  -\n\t\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}(2q)^l.\n\t\\end{align*}\n\tLet $a>0$ be a parameter to be set later, and take $l=\\ceil{a\\log(n)}$. Then, the above inequality implies:\n\t\\begin{align*}\n\t\\P(N(l,n) =0 ) &\\geq 1- n \\left(\\frac{q}{p}\\right)^{a\\log(n)}  -\n\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}(2q)^{a\\log(n)} \\\\\n\t&= 1- n^{1-a\\log(\\frac{p}{q})}  -\n\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}n^{-a\\log(1/(2q))}.\n\t\\end{align*}\n%\tNote that $\\left(\\frac{q}{p}\\right)^{\\log_a(n)}=n^{\\log_a(\\frac{q}{p})}$, hence\n%\t$$\\P(N(k,n) =0 )\\geq \n%\t1- n^{1-\\log_a(\\frac{p}{q})}   -\n%\t\\frac{2 p\\sqrt{q}}{\\sqrt{p}(1-2 \\sqrt{p q})^{2}}n^{-\\frac 1 2\\log_a(\\frac p q)}.$$\n\t\\rev{In going from the first line to the second, we used the following fact about logarithms: $x^{a \\log n} = n^{a \\log x} = n^{-a\\log (1/x)}$.} Let $a = \\frac{1+\\omega}{\\log (1/2q)}$. Then, $a \\log(\\frac{p}{q})  \\geq  a \\log(\\frac{1}{2q}) = 1+\\omega$, so \n\t$$\\P(N(l,n) =0 )\\geq \n\t1- n^{-\\omega}   -\n\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}n^{-(1+\\omega)}.$$\n\tIn other words, with probability at least $1 - n^{-\\omega} - cn^{-(1+\\omega)}$ with $c = \\frac{2\\sqrt{pq}}{(1-2\\sqrt{pq})^2}$, the random walk will remain below $l=\\ceil{(1+\\omega)\\log_{1/2q} n}$ in the first $n$ steps. By construction of the coupling, with the above probability, we know the  $\\alpha$ parameter in the algorithm either remains above $\\gamma^{\\ceil{a\\log n}}\\bar{\\alpha} = \\gamma^{\\ceil{(1+\\omega)\\log_{1/2q} n}}\\bar{\\alpha}$  throughout the first $n$ steps, or the algorithm has reached its stopping time in $n$ steps. \n \n\\end{proof}\n\n\\begin{proof}\n%\tWe now want to derive an upper bound on the sample complexity $\\SC(T_{\\eps})$ that holds with high probability.  With probability at least $ 1 - \\P(T_\\eps> N)$, the algorithm reaches the stopping time in $N$ iterations, in other words $T_{\\eps}\\leq N$ with high probability. \n\t\n\tLet $\\SC_{\\mathsf{rw}}(n)$ be the total oracle complexity of the first $n$ iterations with the corresponding sequence of parameters $\\alpha_k$ induced by the one-sided random walk (that is, the sequence defined by $\\alpha_k = \\bar{\\alpha}\\gamma^{Z_k}$, where $Z_k$ is defined in \\Cref{sec:randwalk}). In other words, \n\t$$\\SC_{\\mathsf{rw}}(n) = \\sum_{k=1}^n \\sc(\\bar{\\alpha}\\gamma^{Z_k}).$$\n\tWith probability $1 - \\P(T_\\eps > n)$, we have $T_\\eps \\leq n$, which implies\n\t$$\\SC(T_{\\eps})\\leq \\SC_{\\mathsf{rw}}(T_\\eps) \\leq \\SC_{\\mathsf{rw}}(n).$$\n\tHere, the first inequality is by \\Cref{lem:coupling_new} and \\Cref{ass:monotone}, and the second inequality is by $T_\\eps \\leq n$.\n\t\n\tThe same arguments used in the proof of \\Cref{thm:stepsize_bound} show that with probability at least $1 - n^{-\\omega} - c n^{-(1+\\omega)}$,  we have $\\min_{1 \\leq k \\leq n} \\bar{\\alpha}\\gamma^{Z_k} \\geq \\alpha^*(n)$. Thus, with at least this probability, $\\SC_{\\mathsf{rw}}(n) \\leq n\\cdot\\sc(\\alpha^*(n))$. \n\t\n\tPutting these together with a union bound, the result follows.\n\t\n\tThe second part of the theorem follows from substituting in the specific choice of $\\gamma$.\n%\tTogether with Theorem \\ref{thm:stepsize_bound}, the result follows.\n\\end{proof}","units":[{"start":42822,"end":43560,"start_line":499,"end_line":507,"raw_tex":"\\begin{theorem}\n\\label{thm:stepsize_bound}\n\tSuppose Assumption \\ref{ass:alg_behave} holds for Algorithm \\ref{alg:generic_stochastic}. For any positive integer $n$, any $\\omega  > 0$, with probability at least  $1 - n^{-\\omega } - cn^{-(1+\\omega )}$, we have \n\t$$\\text{either  $T_\\eps < n$ ~or~ $\\min_{1 \\leq k \\leq n} \\alpha_k  \n\t\\geq \\rev{\\alpha^*(n)} :=  \\bar{\\alpha}\\gamma  \\gamma^{{(1+\\omega)\\log_{1/2q} n}}\n=\\bar{\\alpha}\\gamma   n^{-{(1+\\omega)\\log_{1/2q} 1/\\gamma}}$},$$ \twhere $c = \\frac{2\\sqrt{pq}}{(1-2\\sqrt{pq})^2}$ and $q = 1-p$.\n\t\n% \t In other words, with high probability the algorithm will not reach any step size smaller or equal to $ \\gamma^{\\ceil{(1+\\nu )\\log_{p/q} n}}\\bar{\\alpha}$ in the first $n$ steps. \n\\end{theorem}","sha256_utf8":"d838e75b11a5c14cc9c10622c80d8e5d8844eabf1255ea300486de6e4bafef1a"},{"start":52584,"end":52731,"start_line":625,"end_line":628,"raw_tex":"\\begin{lemma}\n\t\\label{lem:coupling_new}\n\tThere exists a coupling between $Z_k$  and $Y_k$, where $Z_k$ stochastically dominates $Y_k$. \n\\end{lemma}","sha256_utf8":"195d06de1d31783bacdb2d54ebe0fe7cd5f2b9081adf1f1ffc6a10dcc41fc76a"},{"start":52732,"end":55008,"start_line":629,"end_line":642,"raw_tex":"\\begin{proof}[Proof of Lemma \\ref{lem:coupling_new}]\nInitially, $Z_0=0$ and $Y_0 \\leq 0$. For each $k$, we show how to update $Z_k$ to $Z_{k+1}$ according to how $Y_k$ changes to $Y_{k+1}$. We consider two cases depending on whether $k < T_\\eps$ or $k \\geq T_\\eps$. \n\n\\underline{Case 1: $k < T_\\eps$}.\n If $Y_k\\leq -1$, we update $Z_{k+1}$ from $Z_{k}$ according to \\Cref{eq:randwalk_update}, independently of how $Y_k$ changes to $Y_{k+1}$. If $Y_k \\geq 0$, then we first check if $Y_k$ increased or decreased. Let $p'$ be the probability that $Y_{k+1} = Y_k -1$ on this sample path. Since $Y_k \\geq 0$, we know by \\Cref{ass:alg_behave} that $p' \\geq p$. Now, if $Y_{k+1} = Y_k + 1$, then we set $Z_{k+1} = Z_{k} + 1$. On the other hand, if $Y_{k+1} = Y_k - 1$, then we set $Z_{k+1} = Z_k + 1$ with probability $1 - \\frac{p}{p'}$, and $Z_{k+1} = \\max\\{Z_k-1, 0\\}$ with probability $\\frac{p}{p'}$. Note that these probabilities are well-defined because $p' \\geq p$. \n \n\\underline{Case 2: $k \\geq T_\\eps$}. If $Y_{k+1} = Y_k + 1$, then set $Z_{k+1} = Z_{k}+1$. Otherwise, if $Y_{k+1} = Y_{k}-1$, then set $Z_{k+1} = \\max\\{Z_k-1, 0\\}$. \n\nObserve that under this coupling, $Z_k \\geq Y_k$ on every sample path. Moreover, $\\{Z_k\\}$ and $\\{Y_k\\}$ have the correct marginal distributions. For $Y_k$, this is easy to see, since it evolves according to its true distribution and we are constructing $Z_k$ from it. For $Z_k$, on any step with $k \\geq T_\\eps$, $Z_{k+1}$ evolves from $Z_k$ correctly according to \\Cref{eq:randwalk_update} by construction. On a step with $k < T_\\eps$ there are two cases: 1) $Y_k \\leq -1$,  and 2) $Y_k \\geq 0$. In the first case, the update from $Z_k$ to $Z_{k+1}$ clearly follows \\Cref{eq:randwalk_update}. This is also true in the second case, since there the probability that $Z_{k}$ increases is $(1-p') + p'(1-\\frac{p}{p'}) = 1-p$.\n\nTo summarize, we have exhibited a coupling between $\\{Z_k\\}$ and $\\{Y_k\\}$, under which $Z_k \\geq Y_k$ on any sample path. \n\n%Thus, $\\sA_k = \\gamma^{Y_k}\\bar{\\alpha}$ the step size of the algorithm at iteration $k$, stochastically dominates $\\gamma^{Z_k}\\bar{\\alpha}$.  Thus, to obtain a lower bound on the step sizes of the algorithm, it suffices to obtain an upper bound on the random walk $\\{Z_k\\}$. \n\\end{proof}","sha256_utf8":"a9820f1bf6b29b638d0dd07351d1cdfb4782c75e2942c439eb1bd0d37ac965fe"},{"start":55624,"end":56022,"start_line":656,"end_line":662,"raw_tex":"\\begin{prop}\\label{prop:Fill}\n\tLet $q=1-p$. We have\n%\t$$\\P(N(l,n) >0 ) \\leq(n-l+1) \\frac{1-(q / p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l}  +\n%\t\\frac{2 p \\rev{2^l(pq)^{l/2}}}{(1-2 \\sqrt{p q})^{2}}\\left(\\frac{q}{p}\\right)^{(l+1) / 2}.$$\n\t$$\\P(N(l,n) >0 ) \\leq(n-l+1) \\frac{1-(q / p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l}  +\n\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}(2q)^l.$$\n\\end{prop}","sha256_utf8":"ef05580e6892e8ae239a9608415cbed60b3e68e0c9e30fecda7cd3ef5a18e156"},{"start":56024,"end":57902,"start_line":664,"end_line":692,"raw_tex":"\\begin{proof}\n%\tLet $q=1-p .$ \n%\tDenote by $p_{l, N}$ the probability that state $l$ is reached in the first $N$ steps. Note that $p_{l, N} = \\P(\\mathcal{N}(l, N) > 0)$ by definition. \n\tFirst, observe that $ \\P({N}(l, n) > 0)$ remains unchanged if we change the state space from $\\{0, 1,2, \\ldots\\}$ to $\\{0, 1,2, \\ldots, l\\}$ and modify the walk to hold in state $l$ with probability $q$ (instead of moving from $l$ to $l+1$ with that probability). This defines a Markov chain on $\\{0,1,2,\\ldots, l\\}$, and let $P$ be its transition matrix. Noting that $P_{0,l}^m$ is the probability that the Markov chain is in state $l$ at time $m$, we see that\n\t\\begin{equation}\n\t\t\\label{eq:subadd}\n\t\t\\P(N(l,n) >0 ) \\leq \\sum_{m=l}^{n} P_{0, l}^{m}.\n\t\\end{equation}\n\t\n\tThe matrix $P$ is explicitly diagonalized in \\cite{feller_book} (Section XVI.3). By $(3.16)$ in that section,\n\t\\begin{equation}\n\t\t\\label{eq:transprob}\n\t\tP_{0,l}^{m}=\\frac{1-(q/p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l} \n\t\t-\\frac{2 q}{l+1}\\left(\\frac{q}{p}\\right)^{\\frac{l-1}{2}} \\sum_{r=1}^{l} \\frac{\\left[\\sin \\frac{\\pi r}{l+1}\\right]\\left[\\sin \\frac{\\pi rl}{l+1}\\right]\\left[2 \\sqrt{pq} \\cos \\frac{\\pi r}{l+1}\\right]^{m}}{1-2 \\sqrt{p q} \\cos \\frac{\\pi r}{l+1}}.\n\t\\end{equation}\n\tThe absolute value of the sum appearing in (\\ref{eq:transprob}) can of course be bounded above by\n\t$$\n\t\\sum_{r=1}^{l} \\frac{(2 \\sqrt{p q})^{m}}{1-2 \\sqrt{p q}}=l \\frac{(2 \\sqrt{p q})^{m}}{1-2 \\sqrt{p q}}\n\t$$\n\tand this readily yields\n\t\\begin{equation}\n\t\t\\label{eq:fillprop_final}\n\t\tP_{0, l}^{m} \\leq \\frac{1-(q/p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l} \n\t\t+\n\t\t\\frac{2 p}{1-2 \\sqrt{p q}}\\left(\\frac{q}{p}\\right)^{\\frac{l+1}{2}}(2 \\sqrt{p q})^{m}.\n\t\\end{equation}\n\tSumming (\\ref{eq:fillprop_final}) over $m=l, \\ldots, n$ and using (\\ref{eq:subadd}), we obtain the bound on $\\P(N(l,n) >0 )$ claimed in the proposition.\n \n\\end{proof}","sha256_utf8":"ed2f94bfff95c6925b91e33aeeb8742920fe02798b777b8b564e61c7aea904a7"},{"start":58374,"end":60417,"start_line":700,"end_line":728,"raw_tex":"\\begin{proof}[Proof of \\Cref{thm:stepsize_bound}]\n\tLet $q=1-p$. By Proposition \\ref{prop:Fill}, we have:\n\t$$\\P(N(l,n) >0 ) \\leq(n-l+1) \\frac{1-(q / p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l}  +\n\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}(2q)^l.$$\n\tIn other words,\n\t\\begin{align*}\n\t\t\\P(N(l,n) =0 ) &\\geq 1-(n-l+1) \\frac{1-(q / p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l}  -\n\t\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}(2q)^l \\\\\n\t\t&\\geq 1- n \\left(\\frac{q}{p}\\right)^{l}  -\n\t\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}(2q)^l.\n\t\\end{align*}\n\tLet $a>0$ be a parameter to be set later, and take $l=\\ceil{a\\log(n)}$. Then, the above inequality implies:\n\t\\begin{align*}\n\t\\P(N(l,n) =0 ) &\\geq 1- n \\left(\\frac{q}{p}\\right)^{a\\log(n)}  -\n\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}(2q)^{a\\log(n)} \\\\\n\t&= 1- n^{1-a\\log(\\frac{p}{q})}  -\n\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}n^{-a\\log(1/(2q))}.\n\t\\end{align*}\n%\tNote that $\\left(\\frac{q}{p}\\right)^{\\log_a(n)}=n^{\\log_a(\\frac{q}{p})}$, hence\n%\t$$\\P(N(k,n) =0 )\\geq \n%\t1- n^{1-\\log_a(\\frac{p}{q})}   -\n%\t\\frac{2 p\\sqrt{q}}{\\sqrt{p}(1-2 \\sqrt{p q})^{2}}n^{-\\frac 1 2\\log_a(\\frac p q)}.$$\n\t\\rev{In going from the first line to the second, we used the following fact about logarithms: $x^{a \\log n} = n^{a \\log x} = n^{-a\\log (1/x)}$.} Let $a = \\frac{1+\\omega}{\\log (1/2q)}$. Then, $a \\log(\\frac{p}{q})  \\geq  a \\log(\\frac{1}{2q}) = 1+\\omega$, so \n\t$$\\P(N(l,n) =0 )\\geq \n\t1- n^{-\\omega}   -\n\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}n^{-(1+\\omega)}.$$\n\tIn other words, with probability at least $1 - n^{-\\omega} - cn^{-(1+\\omega)}$ with $c = \\frac{2\\sqrt{pq}}{(1-2\\sqrt{pq})^2}$, the random walk will remain below $l=\\ceil{(1+\\omega)\\log_{1/2q} n}$ in the first $n$ steps. By construction of the coupling, with the above probability, we know the  $\\alpha$ parameter in the algorithm either remains above $\\gamma^{\\ceil{a\\log n}}\\bar{\\alpha} = \\gamma^{\\ceil{(1+\\omega)\\log_{1/2q} n}}\\bar{\\alpha}$  throughout the first $n$ steps, or the algorithm has reached its stopping time in $n$ steps. \n \n\\end{proof}","sha256_utf8":"b03f2e15aa17bafe6fbbefb2586d6cd38dd62d89e5535b2f900f28459947748e"},{"start":67063,"end":68590,"start_line":823,"end_line":838,"raw_tex":"\\begin{proof}\n%\tWe now want to derive an upper bound on the sample complexity $\\SC(T_{\\eps})$ that holds with high probability.  With probability at least $ 1 - \\P(T_\\eps> N)$, the algorithm reaches the stopping time in $N$ iterations, in other words $T_{\\eps}\\leq N$ with high probability. \n\t\n\tLet $\\SC_{\\mathsf{rw}}(n)$ be the total oracle complexity of the first $n$ iterations with the corresponding sequence of parameters $\\alpha_k$ induced by the one-sided random walk (that is, the sequence defined by $\\alpha_k = \\bar{\\alpha}\\gamma^{Z_k}$, where $Z_k$ is defined in \\Cref{sec:randwalk}). In other words, \n\t$$\\SC_{\\mathsf{rw}}(n) = \\sum_{k=1}^n \\sc(\\bar{\\alpha}\\gamma^{Z_k}).$$\n\tWith probability $1 - \\P(T_\\eps > n)$, we have $T_\\eps \\leq n$, which implies\n\t$$\\SC(T_{\\eps})\\leq \\SC_{\\mathsf{rw}}(T_\\eps) \\leq \\SC_{\\mathsf{rw}}(n).$$\n\tHere, the first inequality is by \\Cref{lem:coupling_new} and \\Cref{ass:monotone}, and the second inequality is by $T_\\eps \\leq n$.\n\t\n\tThe same arguments used in the proof of \\Cref{thm:stepsize_bound} show that with probability at least $1 - n^{-\\omega} - c n^{-(1+\\omega)}$,  we have $\\min_{1 \\leq k \\leq n} \\bar{\\alpha}\\gamma^{Z_k} \\geq \\alpha^*(n)$. Thus, with at least this probability, $\\SC_{\\mathsf{rw}}(n) \\leq n\\cdot\\sc(\\alpha^*(n))$. \n\t\n\tPutting these together with a union bound, the result follows.\n\t\n\tThe second part of the theorem follows from substituting in the specific choice of $\\gamma$.\n%\tTogether with Theorem \\ref{thm:stepsize_bound}, the result follows.\n\\end{proof}","sha256_utf8":"349dceecf2e9e35c51802d86b32b13804fdfb99334af3ba1d193f58cbd23563c"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"总预言复杂度的高概率界","source_units":[{"start":66206,"end":67062,"start_line":813,"end_line":822,"raw_tex":"\\begin{theorem}\n\\label{cor:sc_highprob}\n\\begin{sloppypar}\n\tLet \\Cref{ass:alg_behave} and \\Cref{ass:monotone} hold in Algorithm \\ref{alg:generic_stochastic}.\n\t For any $\\omega  > 0$ and positive integer $n$, with probability at least $1-\\P(T_\\eps> n)- n^{-\\omega } - cn^{-(1+\\omega )}$, $$\\SC(T_{\\eps})\\leq n\\cdot \\sc(\\alpha^*(n)),$$ \n\t where $\\alpha^*(n)= \\bar{\\alpha}\\gamma   n^{-{(1+\\omega)\\log_{1/2q} 1/\\gamma}}$ , and $c$ is as defined in \\Cref{thm:stepsize_bound}.\n\\end{sloppypar}\n\t If $\\gamma $ is chosen to be at least $\\max\\left\\{\\frac12, \\left(\\frac{1}{2q}\\right)^{\\frac{\\log(2\\beta)}{(1+\\omega)\\log n} } \\right\\}$ for some $\\beta < \\frac12$, then $\\alpha^*(n) \\geq \\beta \\bar{\\alpha}$, thus \\rev{with probability at least $1-\\P(T_\\eps> n)- n^{-\\omega } - cn^{-(1+\\omega )}$,}\n\t $$\\SC(T_{\\eps})\\leq n\\cdot \\sc(\\beta \\bar{\\alpha}).$$ \n\\end{theorem}","sha256_utf8":"189a89a684aae492cdeb401ba96f5fc9a802bdfd0deedfe96320c6c4212ad69e"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"高概率复杂度界的完整证明","source_units":[{"start":67063,"end":68590,"start_line":823,"end_line":838,"raw_tex":"\\begin{proof}\n%\tWe now want to derive an upper bound on the sample complexity $\\SC(T_{\\eps})$ that holds with high probability.  With probability at least $ 1 - \\P(T_\\eps> N)$, the algorithm reaches the stopping time in $N$ iterations, in other words $T_{\\eps}\\leq N$ with high probability. \n\t\n\tLet $\\SC_{\\mathsf{rw}}(n)$ be the total oracle complexity of the first $n$ iterations with the corresponding sequence of parameters $\\alpha_k$ induced by the one-sided random walk (that is, the sequence defined by $\\alpha_k = \\bar{\\alpha}\\gamma^{Z_k}$, where $Z_k$ is defined in \\Cref{sec:randwalk}). In other words, \n\t$$\\SC_{\\mathsf{rw}}(n) = \\sum_{k=1}^n \\sc(\\bar{\\alpha}\\gamma^{Z_k}).$$\n\tWith probability $1 - \\P(T_\\eps > n)$, we have $T_\\eps \\leq n$, which implies\n\t$$\\SC(T_{\\eps})\\leq \\SC_{\\mathsf{rw}}(T_\\eps) \\leq \\SC_{\\mathsf{rw}}(n).$$\n\tHere, the first inequality is by \\Cref{lem:coupling_new} and \\Cref{ass:monotone}, and the second inequality is by $T_\\eps \\leq n$.\n\t\n\tThe same arguments used in the proof of \\Cref{thm:stepsize_bound} show that with probability at least $1 - n^{-\\omega} - c n^{-(1+\\omega)}$,  we have $\\min_{1 \\leq k \\leq n} \\bar{\\alpha}\\gamma^{Z_k} \\geq \\alpha^*(n)$. Thus, with at least this probability, $\\SC_{\\mathsf{rw}}(n) \\leq n\\cdot\\sc(\\alpha^*(n))$. \n\t\n\tPutting these together with a union bound, the result follows.\n\t\n\tThe second part of the theorem follows from substituting in the specific choice of $\\gamma$.\n%\tTogether with Theorem \\ref{thm:stepsize_bound}, the result follows.\n\\end{proof}","sha256_utf8":"349dceecf2e9e35c51802d86b32b13804fdfb99334af3ba1d193f58cbd23563c"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"随机预言优化问题","source_units":[{"start":15221,"end":15358,"start_line":221,"end_line":224,"raw_tex":"We consider a continuous optimization problem of the form\n\\begin{equation}\\label{prob.opt}\n  \\min_{x \\in \\R^{m}}\\ \\phi(x),\n\\end{equation}","sha256_utf8":"ca12e075dd569515355d9a00e84c5c70eb634a2a5fdeecf009587fb823fa40ae"},{"start":15359,"end":15875,"start_line":225,"end_line":226,"raw_tex":"where $\\phi$ is possibly non-convex, (twice-)continuously differentiable with Lipschitz continuous derivatives.\nNeither function values $\\phi(x)$, nor gradients $\\nabla \\phi(x)$ are assumed to be directly computable. Instead, given any $x \\in \\R^{m}$, it is assumed that stochastic estimates of $\\phi(x)$, $\\nabla \\phi(x)$, and possibly $\\nabla^2 \\phi(x)$  can be computed, and these estimates may possess different levels of \\emph{accuracy} and \\emph{reliability} depending on the particular setting of interest.  \n","sha256_utf8":"8a04084eb594e9e0567a14dd385843594c6d24595c8b7001c8a913da3d509b37"}]},{"id":"S004","semantic_type":"algorithm","kind":"algorithm","title":"自适应随机优化框架","source_units":[{"start":19267,"end":21078,"start_line":251,"end_line":296,"raw_tex":"\\begin{algorithm}\n\t\\caption{~\\textbf{Algorithmic Framework for Adaptive Stochastic Optimization}}\n\t\\label{alg:generic_stochastic}\n\t\n\t{\\bf 0. Initialization} \n\t\n\t\\vspace{0.5mm}\n\t\n\t\\hspace{0.5cm} Choose $\\mytheta \\in (0,1)$, $\\gamma\\in (0, 1)$, $\\alpha_{\\max} \\in (0,\\infty)$, $x_0 \\in \\R^{m}$, and $\\alpha_0 \\in (0,\\alpha_{\\max}]$.  Set $k \\gets 0$.\n\t\\vspace{0.5mm}\n\t\n\t{\\bf 1. Determine model and compute step}\n\t\n\t\\vspace{0.5mm}\n\t\n\t\\hspace{0.5cm} Construct a stochastic model $m_k$ of $\\phi$ at $x_k$ using $f(x_k,\\xi_{0,k})$, $g(x_k,\\xi_{1,k})$, and (optionally) $H(x_k,\\xi_{2,k})$ from \\emph{probabilistic zeroth-, first-, and (optionally) second-order oracles}.  Compute $s_k(\\alpha_k)$ such that the model reduction $m_k(x_k) - m_k(x_k+s_k(\\alpha_k)) \\geq 0$ is sufficiently large.\n\t\\vspace{0.5mm}\n\t\n\t{\\bf 2. Check for sufficient reduction}\n\t\n\t\\vspace{0.5mm}\n\t\n\t\\hspace{0.5cm} Set $x_k^+ \\gets x_k + s_k(\\alpha_k)$ and compute $f(x_k^+,\\xi_{0,k}^+)$ as a stochastic estimate of $\\phi(x_k^+)$ using a \\emph{probabilistic zeroth-order oracle}.  Check if $ f(x_k,\\xi_{0,k})- f(x_k^+,\\xi_{0,k}^+)$ is sufficiently large (e.g., relative to the model reduction $m_k(x_k)-m_k(x_k^+)$) using a condition parameterized by~$\\mytheta$.\n\t\n\t\\vspace{0.5mm}\n\t{\\bf 3. Successful iteration}\n\t\n\t\\vspace{0.5mm}\n\t\n\t\\hspace{0.5cm} If sufficient reduction has been attained (along with other potential requirements), then set $x_{k+1} \\gets x_k^+ $ and $\\alpha_{k+1} \\gets \\min\\{\\gamma^{-1}\\alpha_k,\\alpha_{\\max}\\}$.\n\t\\vspace{0.5mm}\n\t\n\t{\\bf 4. Unsuccessful iteration}\n\t\n\t\\vspace{0.5mm}\n\t\n\t\\hspace{0.5cm} Otherwise, set $x_{k+1} \\gets x_k$ and $\\alpha_{k+1} \\gets \\gamma \\alpha_k$.\n\t\\vspace{0.5mm}\n\t\n\t{\\bf 5. Next iteration}\n\t\n\t\\vspace{0.5mm}\n\t\n\t\\hspace{0.5cm}  Set $k \\gets k + 1$ and go to Step~1.\n\t\\vspace{0.5mm}\n\\end{algorithm}","sha256_utf8":"c910b1d372e891b39ba121814a4cf65d26f5564979ed536704c17ff7c53260a6"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"随机预言及其成本","source_units":[{"start":24865,"end":25704,"start_line":309,"end_line":314,"raw_tex":"\\begin{definition}{\\bf Stochastic $j$th-order oracle.} Given an input  $x \\in \\R^{n}$ and the step size parameter $\\alpha$, the oracle computes $\\varphi_j(x,\\xi_j)$, an estimate of the $j^{\\mathrm{th}}$-order derivative $\\nabla^j \\phi(x)$. In all the algorithms we consider, $\\|\\varphi_j(x,\\xi_j)- \\nabla^j\\phi(x)\\|$ is assumed to be bounded by some quantity (which will be a function of  $\\alpha$), with probability at least $1-\\delta_j$.\n%  where $\\delta_j$ is an intrinsic value. \n   Here, $\\xi_j$ is a random variable defined on probability space $(\\Omega_j, {\\cal F}_j, P_j)$ whose distribution depends on the input  $x$ and $\\alpha$, and $\\delta_j$ is intrinsic to the oracle.\n%    and $\\P_{\\xi_j}$ denotes probability w.r.t.~that distribution. \n    The cost of the oracle is also a  function of $x$ and $\\alpha$. \n  \\end{definition}","sha256_utf8":"473785e5e6b1280dfb57b059b82bb78d27338517cc6fc3d0f9dc127d02a9dcc5"}]},{"id":"S006","semantic_type":"notation","kind":"notation","title":"随机迭代与步长记号","source_units":[{"start":38834,"end":39374,"start_line":458,"end_line":458,"raw_tex":"Specifically, let $(X_k)_{k\\geq 0}$ be the random iterates with realizations $x_k$, let $(G_k)_{k\\geq 0}$ be the gradient estimates with realizations $g_k$, and let $(\\mathcal{A}_k)_{k \\geq 0}$ be the step size parameter values with realizations $\\alpha_k$. The prior works that analyze different methods \\rev{belonging to the framework of Algorithm~\\ref{alg:generic_stochastic}}  define this stochastic process rigorously, with appropriate filtrations. Here for brevity, we will omit those details, as we do not use them in the analysis. \n","sha256_utf8":"362c5191121502b6332fc127fedfef6764e4e2cce385bcba848575f63a842dad"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"精度停止时间","source_units":[{"start":40236,"end":40723,"start_line":463,"end_line":466,"raw_tex":"\\begin{defn}[Stopping time]\n\tFor $\\varepsilon > 0$, let $T_\\varepsilon$ be the first time such that a specified optimality condition is satisfied. For all the settings considered in this paper,\n\t${T_\\varepsilon}=\\min\\{k: \\norm{\\nabla \\phi(x_k)} \\leq \\varepsilon\\}$ if $\\phi$ is non-convex, and ${T_\\varepsilon}=\\min\\{k: \\phi(x_k) - \\inf_x \\phi(x)  \\leq \\varepsilon\\}$ if $\\phi$ is strongly convex. We will refer to $T_\\varepsilon$ as the \\emph{stopping time} of the algorithm.\n\\end{defn}","sha256_utf8":"bad78ef4ecab607f80c6d54d5154912a72c1e4b946aec17bbb6f5c2144e8f6d5"}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","title":"步长更新与成功概率假设","source_units":[{"start":41015,"end":41873,"start_line":473,"end_line":485,"raw_tex":"\\begin{assumption}\n\t[Properties of the stochastic process generated by the adaptive stochastic algorithm]\n\t\\label{ass:alg_behave}\n\tThe random sequence of parameters $\\mathcal{A}_k$ generated by the algorithm satisfies the following: \n\t\\begin{itemize}\n\t\t\\item[(i)] For all $k$, $\\mathcal{A}_k \\in \\{\\gamma \\mathcal{A}_{k-1}, \\min\\{\\amax, \\gamma^{-1}\\mathcal{A}_{k-1}\\}\\}$, and\n\t\t\\item[(ii)] There exist constants $\\bar\\alpha > 0$, and $p>\\frac 12$, such that \\rev{for all iterations $k$,\n%\t\t for all iterations  $k < T_\\varepsilon$, if  $\\mathcal{A}_k  \\leq  \\bar \\alpha$ then\n\t\t$$\\P(\\mathcal{A}_{k+1} = \\gamma^{-1} \\mathcal{A}_{k} \\mid \\mathcal{F}_{k}, \\, k < T_\\varepsilon, \\, \\mathcal{A}_k  \\leq  \\bar \\alpha ) \\geq p.$$}\n\t\tHere, $\\mathcal{F}_k$ denotes the filtration generated by the algorithm up to iteration $k$.\n\t\\end{itemize}       \n\n\\end{assumption}","sha256_utf8":"f87ed7e2ca5505734438270bf9d5bc5dd120ba93e96df91326805099e98c4974"}]},{"id":"S009","semantic_type":"notation","kind":"notation","title":"失败概率记号","source_units":[{"start":42574,"end":42647,"start_line":489,"end_line":489,"raw_tex":"Throughout the remainder of this paper, we will use $q$ to denote $1-p$.\n","sha256_utf8":"7cfdb5da62148199301114a4cbf875e85e24efaa4ba2fcebf09bdb9fa99344ee"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"步长指数过程及停止后延拓","source_units":[{"start":50689,"end":51659,"start_line":595,"end_line":608,"raw_tex":"Let $\\{\\A_k\\}_{k=0}^\\infty$ denote the random sequence of parameter values (whose realization is $\\{\\alpha_k\\}_{k=0}^\\infty$), for  Algorithm \\ref{alg:generic_stochastic}.  Let us assume, WLOG, that $\\A_0 = \\gamma^j \\bar{\\alpha}$, for some integer $j\\leq0$.  (Recall here that $0 <\\gamma < 1$.) Then we observe that $\\mathcal{A}_k = \\gamma^{\\bar{Y}_k}\\bar{\\alpha}$,   where  $\\{\\bar{Y}_k\\}_{k=0}^\\infty$ is a random sequence of  integers, with $\\bar{Y}_0 =j\\leq 0$, which increases by one on every unsuccessful step, and decreases by one on every successful step. Moreover, by \nAssumption  \\ref{ass:alg_behave}, whenever $k < T_\\eps$ and $\\bar{Y}_k \\geq 0$, the probability that it decreases by one is at least $p$. Define $Y_k$ as follows:\n\\begin{equation}\n\\label{eq:Yk}\nY_k = \\bar{Y}_k \\quad \\text{if $k \\leq T_\\eps$},\n\\qquad\nY_k = \n\\begin{cases}\nY_{k-1} - 1 &\\text{w.p. $p$} \\\\\nY_{k-1} + 1 &\\text{w.p. $1-p$} \n\\end{cases}\n\\quad\n\\text{if $k > T_\\eps$.} \n\\end{equation}","sha256_utf8":"8b0457af2f0afcdbc177c49f85035d8470ea0858b6bbc52e18ef2389d1b84b43"},{"start":51660,"end":51916,"start_line":609,"end_line":610,"raw_tex":"In other words, $Y_k$ follows the algorithm until $T_\\eps$, and then behaves like a random walk with downward drift $p$ after $T_\\eps$.\nWe now couple  $\\{Y_k\\}_{k=0}^\\infty$  with a random walk $\\{Z_k\\}_{k=0}^\\infty$ which stochastically dominates $Y_k$. \n","sha256_utf8":"0c926b55a7b10391ef9afeb5bdbc22a079b75ebf44c8f41e01aba7cd73460563"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"单边反射随机游走","source_units":[{"start":51918,"end":52249,"start_line":613,"end_line":621,"raw_tex":"Consider the following one-sided random walk $\\{Z_k\\}_{k=0}^\\infty$, defined  on the non-negative integers. \n\\begin{equation}\n\\label{eq:randwalk_update}\nZ_0 = 0, \\qquad Z_{k+1} = \\begin{cases}\n\tZ_k +1, &\\text{w.p. $1-p$,}\\\\\n\tZ_k-1, &\\text{w.p. $p$, if $Z_k \\geq 1$,}\\\\\n\t0, &\\text{w.p. $p$, if $Z_k = 0$.}\n\\end{cases}\n\\end{equation}","sha256_utf8":"c59148814327126b2c7cf489ebdf13739f70386bec6b3e530e177e02fc02cbfd"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"状态访问次数与访问事件","source_units":[{"start":55206,"end":55421,"start_line":650,"end_line":653,"raw_tex":"\\begin{defn}\nLet $N(l, n)$ be the random variable that denotes the number of times $Z_k=l$  in the first $n$ steps of the random walk.  \n%When $N(l, n)>0$, we say state $l$ is visited by the random walk. \n\\end{defn}","sha256_utf8":"df7131852603e5895616b4d45949895b159d0a386e52fdedbafec923d4f19534"},{"start":55422,"end":55623,"start_line":654,"end_line":654,"raw_tex":"By definition of $N(l, n)$, we have $N(l, n) > 0$ if and only if state $l$ is visited in the first $n$ steps of the random walk. The next proposition  upper bounds the probability that $N(l, n) > 0$. \n","sha256_utf8":"309dce448502e53f52a4b187768aabbbfcf03a2a7a61163e363f8f02855cbb0f"}]},{"id":"S013","semantic_type":"notation","kind":"notation","title":"单次预言成本函数","source_units":[{"start":61589,"end":62048,"start_line":742,"end_line":744,"raw_tex":"The cost of an oracle call may depend on the step size parameter $\\alpha$ and the probability parameter $1-\\delta$, thus we denote the cost by $\\sc(\\alpha, 1-\\delta)$. \nWe will use $\\sc(\\alpha)$ in the paper to simplify the notation because for all algorithms in the class, $\\delta$ can be treated as a constant.\n Moreover, the cost of an oracle call is a non-increasing function of  $\\alpha$ for all algorithms developed so far that fit into the framework. \n","sha256_utf8":"90e23e2e8781b2469d16f1f2338bd6feb11d9070adba49b5a8770811f0d80b75"}]},{"id":"S014","semantic_type":"assumption","kind":"assumption","title":"成本关于步长单调","source_units":[{"start":62076,"end":62413,"start_line":747,"end_line":751,"raw_tex":"\\begin{assumption}\n\t\\label{ass:monotone}\n%\t\\item For a fixed set of probability parameters of the stochastic oracles, the oracle costs of all the oracle calls at iteration $k$ only depend on the parameter $\\alpha_k$. We assume $\\sc(\\alpha)$ increases as $\\alpha$ decreases.\n  $\\sc(\\alpha)$ is non-increasing in $\\alpha$.\n\\end{assumption}","sha256_utf8":"b0dc53a76b673fd1751abcd88f62048060ad1e02df67a5f87edd6b967fec052d"}]},{"id":"S015","semantic_type":"definition","kind":"definition","title":"总预言复杂度","source_units":[{"start":62414,"end":62675,"start_line":752,"end_line":755,"raw_tex":"\\begin{defn}[Total Oracle Complexity]\nFor a positive integer $n$, let $\\SC(n)$ be the random variable which denotes the total oracle complexity of running the algorithm for $n$ iterations. In other words,\n$$\\SC(n) = \\sum_{k=1}^n \\sc(\\mathcal{A}_k).$$\n\\end{defn}","sha256_utf8":"c6f472532529deecf9f481a2626903826e5a7b7fc7aa35bc55a07c1d338d3213"}]},{"id":"S016","semantic_type":"lemma","kind":"lemma","title":"随机游走支配耦合","source_units":[{"start":52584,"end":52731,"start_line":625,"end_line":628,"raw_tex":"\\begin{lemma}\n\t\\label{lem:coupling_new}\n\tThere exists a coupling between $Z_k$  and $Y_k$, where $Z_k$ stochastically dominates $Y_k$. \n\\end{lemma}","sha256_utf8":"195d06de1d31783bacdb2d54ebe0fe7cd5f2b9081adf1f1ffc6a10dcc41fc76a"}]},{"id":"S017","semantic_type":"proof","kind":"proof","title":"支配耦合的完整证明","source_units":[{"start":52732,"end":55008,"start_line":629,"end_line":642,"raw_tex":"\\begin{proof}[Proof of Lemma \\ref{lem:coupling_new}]\nInitially, $Z_0=0$ and $Y_0 \\leq 0$. For each $k$, we show how to update $Z_k$ to $Z_{k+1}$ according to how $Y_k$ changes to $Y_{k+1}$. We consider two cases depending on whether $k < T_\\eps$ or $k \\geq T_\\eps$. \n\n\\underline{Case 1: $k < T_\\eps$}.\n If $Y_k\\leq -1$, we update $Z_{k+1}$ from $Z_{k}$ according to \\Cref{eq:randwalk_update}, independently of how $Y_k$ changes to $Y_{k+1}$. If $Y_k \\geq 0$, then we first check if $Y_k$ increased or decreased. Let $p'$ be the probability that $Y_{k+1} = Y_k -1$ on this sample path. Since $Y_k \\geq 0$, we know by \\Cref{ass:alg_behave} that $p' \\geq p$. Now, if $Y_{k+1} = Y_k + 1$, then we set $Z_{k+1} = Z_{k} + 1$. On the other hand, if $Y_{k+1} = Y_k - 1$, then we set $Z_{k+1} = Z_k + 1$ with probability $1 - \\frac{p}{p'}$, and $Z_{k+1} = \\max\\{Z_k-1, 0\\}$ with probability $\\frac{p}{p'}$. Note that these probabilities are well-defined because $p' \\geq p$. \n \n\\underline{Case 2: $k \\geq T_\\eps$}. If $Y_{k+1} = Y_k + 1$, then set $Z_{k+1} = Z_{k}+1$. Otherwise, if $Y_{k+1} = Y_{k}-1$, then set $Z_{k+1} = \\max\\{Z_k-1, 0\\}$. \n\nObserve that under this coupling, $Z_k \\geq Y_k$ on every sample path. Moreover, $\\{Z_k\\}$ and $\\{Y_k\\}$ have the correct marginal distributions. For $Y_k$, this is easy to see, since it evolves according to its true distribution and we are constructing $Z_k$ from it. For $Z_k$, on any step with $k \\geq T_\\eps$, $Z_{k+1}$ evolves from $Z_k$ correctly according to \\Cref{eq:randwalk_update} by construction. On a step with $k < T_\\eps$ there are two cases: 1) $Y_k \\leq -1$,  and 2) $Y_k \\geq 0$. In the first case, the update from $Z_k$ to $Z_{k+1}$ clearly follows \\Cref{eq:randwalk_update}. This is also true in the second case, since there the probability that $Z_{k}$ increases is $(1-p') + p'(1-\\frac{p}{p'}) = 1-p$.\n\nTo summarize, we have exhibited a coupling between $\\{Z_k\\}$ and $\\{Y_k\\}$, under which $Z_k \\geq Y_k$ on any sample path. \n\n%Thus, $\\sA_k = \\gamma^{Y_k}\\bar{\\alpha}$ the step size of the algorithm at iteration $k$, stochastically dominates $\\gamma^{Z_k}\\bar{\\alpha}$.  Thus, to obtain a lower bound on the step sizes of the algorithm, it suffices to obtain an upper bound on the random walk $\\{Z_k\\}$. \n\\end{proof}","sha256_utf8":"a9820f1bf6b29b638d0dd07351d1cdfb4782c75e2942c439eb1bd0d37ac965fe"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"初始化耦合并划分时段","source_units":[{"start":52732,"end":52999,"start_line":629,"end_line":630,"raw_tex":"\\begin{proof}[Proof of Lemma \\ref{lem:coupling_new}]\nInitially, $Z_0=0$ and $Y_0 \\leq 0$. For each $k$, we show how to update $Z_k$ to $Z_{k+1}$ according to how $Y_k$ changes to $Y_{k+1}$. We consider two cases depending on whether $k < T_\\eps$ or $k \\geq T_\\eps$. \n","sha256_utf8":"1e4d0b9f1832890c9737b795c09b4366b368347144dc958d0f028d9d81546ffb"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"构造停止前的耦合更新","source_units":[{"start":52999,"end":53699,"start_line":631,"end_line":633,"raw_tex":"\n\\underline{Case 1: $k < T_\\eps$}.\n If $Y_k\\leq -1$, we update $Z_{k+1}$ from $Z_{k}$ according to \\Cref{eq:randwalk_update}, independently of how $Y_k$ changes to $Y_{k+1}$. If $Y_k \\geq 0$, then we first check if $Y_k$ increased or decreased. Let $p'$ be the probability that $Y_{k+1} = Y_k -1$ on this sample path. Since $Y_k \\geq 0$, we know by \\Cref{ass:alg_behave} that $p' \\geq p$. Now, if $Y_{k+1} = Y_k + 1$, then we set $Z_{k+1} = Z_{k} + 1$. On the other hand, if $Y_{k+1} = Y_k - 1$, then we set $Z_{k+1} = Z_k + 1$ with probability $1 - \\frac{p}{p'}$, and $Z_{k+1} = \\max\\{Z_k-1, 0\\}$ with probability $\\frac{p}{p'}$. Note that these probabilities are well-defined because $p' \\geq p$. \n","sha256_utf8":"1be5aeae18ffeee7e9e349a832cdd17c340ce6ff87391517cca3e987fdc3acea"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"构造停止后的同步更新","source_units":[{"start":53699,"end":53867,"start_line":634,"end_line":635,"raw_tex":" \n\\underline{Case 2: $k \\geq T_\\eps$}. If $Y_{k+1} = Y_k + 1$, then set $Z_{k+1} = Z_{k}+1$. Otherwise, if $Y_{k+1} = Y_{k}-1$, then set $Z_{k+1} = \\max\\{Z_k-1, 0\\}$. \n","sha256_utf8":"be6164058c5e5df77ea833071eb46df2dd0cedcff608d0f9dead13f239feb3e7"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"验证路径支配与边缘分布","source_units":[{"start":53867,"end":54592,"start_line":636,"end_line":637,"raw_tex":"\nObserve that under this coupling, $Z_k \\geq Y_k$ on every sample path. Moreover, $\\{Z_k\\}$ and $\\{Y_k\\}$ have the correct marginal distributions. For $Y_k$, this is easy to see, since it evolves according to its true distribution and we are constructing $Z_k$ from it. For $Z_k$, on any step with $k \\geq T_\\eps$, $Z_{k+1}$ evolves from $Z_k$ correctly according to \\Cref{eq:randwalk_update} by construction. On a step with $k < T_\\eps$ there are two cases: 1) $Y_k \\leq -1$,  and 2) $Y_k \\geq 0$. In the first case, the update from $Z_k$ to $Z_{k+1}$ clearly follows \\Cref{eq:randwalk_update}. This is also true in the second case, since there the probability that $Z_{k}$ increases is $(1-p') + p'(1-\\frac{p}{p'}) = 1-p$.\n","sha256_utf8":"6a722a159cc1142503adc8daabfe8176fe072d3b307aa79052260b69b4d4d915"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"完成路径支配耦合","source_units":[{"start":54592,"end":55008,"start_line":638,"end_line":642,"raw_tex":"\nTo summarize, we have exhibited a coupling between $\\{Z_k\\}$ and $\\{Y_k\\}$, under which $Z_k \\geq Y_k$ on any sample path. \n\n%Thus, $\\sA_k = \\gamma^{Y_k}\\bar{\\alpha}$ the step size of the algorithm at iteration $k$, stochastically dominates $\\gamma^{Z_k}\\bar{\\alpha}$.  Thus, to obtain a lower bound on the step sizes of the algorithm, it suffices to obtain an upper bound on the random walk $\\{Z_k\\}$. \n\\end{proof}","sha256_utf8":"d5526aeb65fc4196f42656026d41f8b99f45488736e8061e5ef6c51215d24648"}]},{"id":"S023","semantic_type":"proposition","kind":"proposition","title":"随机游走访问概率上界","source_units":[{"start":55624,"end":56022,"start_line":656,"end_line":662,"raw_tex":"\\begin{prop}\\label{prop:Fill}\n\tLet $q=1-p$. We have\n%\t$$\\P(N(l,n) >0 ) \\leq(n-l+1) \\frac{1-(q / p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l}  +\n%\t\\frac{2 p \\rev{2^l(pq)^{l/2}}}{(1-2 \\sqrt{p q})^{2}}\\left(\\frac{q}{p}\\right)^{(l+1) / 2}.$$\n\t$$\\P(N(l,n) >0 ) \\leq(n-l+1) \\frac{1-(q / p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l}  +\n\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}(2q)^l.$$\n\\end{prop}","sha256_utf8":"ef05580e6892e8ae239a9608415cbed60b3e68e0c9e30fecda7cd3ef5a18e156"}]},{"id":"S024","semantic_type":"proof","kind":"proof","title":"访问概率界的完整证明","source_units":[{"start":56024,"end":57902,"start_line":664,"end_line":692,"raw_tex":"\\begin{proof}\n%\tLet $q=1-p .$ \n%\tDenote by $p_{l, N}$ the probability that state $l$ is reached in the first $N$ steps. Note that $p_{l, N} = \\P(\\mathcal{N}(l, N) > 0)$ by definition. \n\tFirst, observe that $ \\P({N}(l, n) > 0)$ remains unchanged if we change the state space from $\\{0, 1,2, \\ldots\\}$ to $\\{0, 1,2, \\ldots, l\\}$ and modify the walk to hold in state $l$ with probability $q$ (instead of moving from $l$ to $l+1$ with that probability). This defines a Markov chain on $\\{0,1,2,\\ldots, l\\}$, and let $P$ be its transition matrix. Noting that $P_{0,l}^m$ is the probability that the Markov chain is in state $l$ at time $m$, we see that\n\t\\begin{equation}\n\t\t\\label{eq:subadd}\n\t\t\\P(N(l,n) >0 ) \\leq \\sum_{m=l}^{n} P_{0, l}^{m}.\n\t\\end{equation}\n\t\n\tThe matrix $P$ is explicitly diagonalized in \\cite{feller_book} (Section XVI.3). By $(3.16)$ in that section,\n\t\\begin{equation}\n\t\t\\label{eq:transprob}\n\t\tP_{0,l}^{m}=\\frac{1-(q/p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l} \n\t\t-\\frac{2 q}{l+1}\\left(\\frac{q}{p}\\right)^{\\frac{l-1}{2}} \\sum_{r=1}^{l} \\frac{\\left[\\sin \\frac{\\pi r}{l+1}\\right]\\left[\\sin \\frac{\\pi rl}{l+1}\\right]\\left[2 \\sqrt{pq} \\cos \\frac{\\pi r}{l+1}\\right]^{m}}{1-2 \\sqrt{p q} \\cos \\frac{\\pi r}{l+1}}.\n\t\\end{equation}\n\tThe absolute value of the sum appearing in (\\ref{eq:transprob}) can of course be bounded above by\n\t$$\n\t\\sum_{r=1}^{l} \\frac{(2 \\sqrt{p q})^{m}}{1-2 \\sqrt{p q}}=l \\frac{(2 \\sqrt{p q})^{m}}{1-2 \\sqrt{p q}}\n\t$$\n\tand this readily yields\n\t\\begin{equation}\n\t\t\\label{eq:fillprop_final}\n\t\tP_{0, l}^{m} \\leq \\frac{1-(q/p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l} \n\t\t+\n\t\t\\frac{2 p}{1-2 \\sqrt{p q}}\\left(\\frac{q}{p}\\right)^{\\frac{l+1}{2}}(2 \\sqrt{p q})^{m}.\n\t\\end{equation}\n\tSumming (\\ref{eq:fillprop_final}) over $m=l, \\ldots, n$ and using (\\ref{eq:subadd}), we obtain the bound on $\\P(N(l,n) >0 )$ claimed in the proposition.\n \n\\end{proof}","sha256_utf8":"ed2f94bfff95c6925b91e33aeeb8742920fe02798b777b8b564e61c7aea904a7"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"截断状态空间并作联合界","source_units":[{"start":56024,"end":56777,"start_line":664,"end_line":671,"raw_tex":"\\begin{proof}\n%\tLet $q=1-p .$ \n%\tDenote by $p_{l, N}$ the probability that state $l$ is reached in the first $N$ steps. Note that $p_{l, N} = \\P(\\mathcal{N}(l, N) > 0)$ by definition. \n\tFirst, observe that $ \\P({N}(l, n) > 0)$ remains unchanged if we change the state space from $\\{0, 1,2, \\ldots\\}$ to $\\{0, 1,2, \\ldots, l\\}$ and modify the walk to hold in state $l$ with probability $q$ (instead of moving from $l$ to $l+1$ with that probability). This defines a Markov chain on $\\{0,1,2,\\ldots, l\\}$, and let $P$ be its transition matrix. Noting that $P_{0,l}^m$ is the probability that the Markov chain is in state $l$ at time $m$, we see that\n\t\\begin{equation}\n\t\t\\label{eq:subadd}\n\t\t\\P(N(l,n) >0 ) \\leq \\sum_{m=l}^{n} P_{0, l}^{m}.\n\t\\end{equation}\n","sha256_utf8":"c33c231b80b75f1cd93398627ae42841f6525976c6de72105c507bf804c9c38a"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"引用有限链转移概率公式","source_units":[{"start":56777,"end":57266,"start_line":672,"end_line":678,"raw_tex":"\t\n\tThe matrix $P$ is explicitly diagonalized in \\cite{feller_book} (Section XVI.3). By $(3.16)$ in that section,\n\t\\begin{equation}\n\t\t\\label{eq:transprob}\n\t\tP_{0,l}^{m}=\\frac{1-(q/p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l} \n\t\t-\\frac{2 q}{l+1}\\left(\\frac{q}{p}\\right)^{\\frac{l-1}{2}} \\sum_{r=1}^{l} \\frac{\\left[\\sin \\frac{\\pi r}{l+1}\\right]\\left[\\sin \\frac{\\pi rl}{l+1}\\right]\\left[2 \\sqrt{pq} \\cos \\frac{\\pi r}{l+1}\\right]^{m}}{1-2 \\sqrt{p q} \\cos \\frac{\\pi r}{l+1}}.\n\t\\end{equation}\n","sha256_utf8":"0a3a1814d503437883f57d9c6e4e4e87e8a6f69e714fb2c5af8134cbcedd8fa9"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"控制谱展开中的振荡项","source_units":[{"start":57266,"end":57475,"start_line":679,"end_line":682,"raw_tex":"\tThe absolute value of the sum appearing in (\\ref{eq:transprob}) can of course be bounded above by\n\t$$\n\t\\sum_{r=1}^{l} \\frac{(2 \\sqrt{p q})^{m}}{1-2 \\sqrt{p q}}=l \\frac{(2 \\sqrt{p q})^{m}}{1-2 \\sqrt{p q}}\n\t$$\n","sha256_utf8":"32ee78a5c9f9639819520ada90f6fd4425d577e0b01e10c97add128698f5cbc2"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"得到单时刻转移概率界","source_units":[{"start":57475,"end":57735,"start_line":683,"end_line":689,"raw_tex":"\tand this readily yields\n\t\\begin{equation}\n\t\t\\label{eq:fillprop_final}\n\t\tP_{0, l}^{m} \\leq \\frac{1-(q/p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l} \n\t\t+\n\t\t\\frac{2 p}{1-2 \\sqrt{p q}}\\left(\\frac{q}{p}\\right)^{\\frac{l+1}{2}}(2 \\sqrt{p q})^{m}.\n\t\\end{equation}\n","sha256_utf8":"3023c9985f7594a1c90532e3df26ac781cf6411ade96220ee516ed35fdfa1ab3"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"对时间求和完成访问界","source_units":[{"start":57735,"end":57902,"start_line":690,"end_line":692,"raw_tex":"\tSumming (\\ref{eq:fillprop_final}) over $m=l, \\ldots, n$ and using (\\ref{eq:subadd}), we obtain the bound on $\\P(N(l,n) >0 )$ claimed in the proposition.\n \n\\end{proof}","sha256_utf8":"e1291510cfeede9f21623bf531d41898f04d82f52fcbe611f4ae08942302f2f1"}]},{"id":"S030","semantic_type":"external_fact","kind":"external_fact","title":"有限反射链的谱公式","source_units":[],"declared_external":true,"declaration_text":["For the Markov chain on {0,1,...,l} with downward probability p, upward probability q=1-p, and reflection by holding at the endpoints, Feller, Section XVI.3, equation (3.16), gives the transition probability P^m_{0,l} by the formula reproduced in eq:transprob at lines 674–678."]},{"id":"S031","semantic_type":"theorem","kind":"theorem","title":"步长的高概率下界","source_units":[{"start":42822,"end":43560,"start_line":499,"end_line":507,"raw_tex":"\\begin{theorem}\n\\label{thm:stepsize_bound}\n\tSuppose Assumption \\ref{ass:alg_behave} holds for Algorithm \\ref{alg:generic_stochastic}. For any positive integer $n$, any $\\omega  > 0$, with probability at least  $1 - n^{-\\omega } - cn^{-(1+\\omega )}$, we have \n\t$$\\text{either  $T_\\eps < n$ ~or~ $\\min_{1 \\leq k \\leq n} \\alpha_k  \n\t\\geq \\rev{\\alpha^*(n)} :=  \\bar{\\alpha}\\gamma  \\gamma^{{(1+\\omega)\\log_{1/2q} n}}\n=\\bar{\\alpha}\\gamma   n^{-{(1+\\omega)\\log_{1/2q} 1/\\gamma}}$},$$ \twhere $c = \\frac{2\\sqrt{pq}}{(1-2\\sqrt{pq})^2}$ and $q = 1-p$.\n\t\n% \t In other words, with high probability the algorithm will not reach any step size smaller or equal to $ \\gamma^{\\ceil{(1+\\nu )\\log_{p/q} n}}\\bar{\\alpha}$ in the first $n$ steps. \n\\end{theorem}","sha256_utf8":"d838e75b11a5c14cc9c10622c80d8e5d8844eabf1255ea300486de6e4bafef1a"}]},{"id":"S032","semantic_type":"proof","kind":"proof","title":"步长下界的完整证明","source_units":[{"start":58374,"end":60417,"start_line":700,"end_line":728,"raw_tex":"\\begin{proof}[Proof of \\Cref{thm:stepsize_bound}]\n\tLet $q=1-p$. By Proposition \\ref{prop:Fill}, we have:\n\t$$\\P(N(l,n) >0 ) \\leq(n-l+1) \\frac{1-(q / p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l}  +\n\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}(2q)^l.$$\n\tIn other words,\n\t\\begin{align*}\n\t\t\\P(N(l,n) =0 ) &\\geq 1-(n-l+1) \\frac{1-(q / p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l}  -\n\t\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}(2q)^l \\\\\n\t\t&\\geq 1- n \\left(\\frac{q}{p}\\right)^{l}  -\n\t\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}(2q)^l.\n\t\\end{align*}\n\tLet $a>0$ be a parameter to be set later, and take $l=\\ceil{a\\log(n)}$. Then, the above inequality implies:\n\t\\begin{align*}\n\t\\P(N(l,n) =0 ) &\\geq 1- n \\left(\\frac{q}{p}\\right)^{a\\log(n)}  -\n\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}(2q)^{a\\log(n)} \\\\\n\t&= 1- n^{1-a\\log(\\frac{p}{q})}  -\n\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}n^{-a\\log(1/(2q))}.\n\t\\end{align*}\n%\tNote that $\\left(\\frac{q}{p}\\right)^{\\log_a(n)}=n^{\\log_a(\\frac{q}{p})}$, hence\n%\t$$\\P(N(k,n) =0 )\\geq \n%\t1- n^{1-\\log_a(\\frac{p}{q})}   -\n%\t\\frac{2 p\\sqrt{q}}{\\sqrt{p}(1-2 \\sqrt{p q})^{2}}n^{-\\frac 1 2\\log_a(\\frac p q)}.$$\n\t\\rev{In going from the first line to the second, we used the following fact about logarithms: $x^{a \\log n} = n^{a \\log x} = n^{-a\\log (1/x)}$.} Let $a = \\frac{1+\\omega}{\\log (1/2q)}$. Then, $a \\log(\\frac{p}{q})  \\geq  a \\log(\\frac{1}{2q}) = 1+\\omega$, so \n\t$$\\P(N(l,n) =0 )\\geq \n\t1- n^{-\\omega}   -\n\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}n^{-(1+\\omega)}.$$\n\tIn other words, with probability at least $1 - n^{-\\omega} - cn^{-(1+\\omega)}$ with $c = \\frac{2\\sqrt{pq}}{(1-2\\sqrt{pq})^2}$, the random walk will remain below $l=\\ceil{(1+\\omega)\\log_{1/2q} n}$ in the first $n$ steps. By construction of the coupling, with the above probability, we know the  $\\alpha$ parameter in the algorithm either remains above $\\gamma^{\\ceil{a\\log n}}\\bar{\\alpha} = \\gamma^{\\ceil{(1+\\omega)\\log_{1/2q} n}}\\bar{\\alpha}$  throughout the first $n$ steps, or the algorithm has reached its stopping time in $n$ steps. \n \n\\end{proof}","sha256_utf8":"b03f2e15aa17bafe6fbbefb2586d6cd38dd62d89e5535b2f900f28459947748e"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"调用状态访问概率界","source_units":[{"start":58374,"end":58625,"start_line":700,"end_line":703,"raw_tex":"\\begin{proof}[Proof of \\Cref{thm:stepsize_bound}]\n\tLet $q=1-p$. By Proposition \\ref{prop:Fill}, we have:\n\t$$\\P(N(l,n) >0 ) \\leq(n-l+1) \\frac{1-(q / p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l}  +\n\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}(2q)^l.$$\n","sha256_utf8":"4084f0bbbd478e3ac6bff061a2847213c37ebb5b020d5b667ffb8a43b67cd484"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"估计未访问指定高度的概率","source_units":[{"start":58625,"end":58917,"start_line":704,"end_line":710,"raw_tex":"\tIn other words,\n\t\\begin{align*}\n\t\t\\P(N(l,n) =0 ) &\\geq 1-(n-l+1) \\frac{1-(q / p)}{1-(q / p)^{l+1}}\\left(\\frac{q}{p}\\right)^{l}  -\n\t\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}(2q)^l \\\\\n\t\t&\\geq 1- n \\left(\\frac{q}{p}\\right)^{l}  -\n\t\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}(2q)^l.\n\t\\end{align*}\n","sha256_utf8":"89d4e5412915a0d2df7655d480fa51168b82017fe7f80b7694f13054874bc034"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"选取对数高度并改写尾界","source_units":[{"start":58917,"end":59278,"start_line":711,"end_line":717,"raw_tex":"\tLet $a>0$ be a parameter to be set later, and take $l=\\ceil{a\\log(n)}$. Then, the above inequality implies:\n\t\\begin{align*}\n\t\\P(N(l,n) =0 ) &\\geq 1- n \\left(\\frac{q}{p}\\right)^{a\\log(n)}  -\n\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}(2q)^{a\\log(n)} \\\\\n\t&= 1- n^{1-a\\log(\\frac{p}{q})}  -\n\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}n^{-a\\log(1/(2q))}.\n\t\\end{align*}\n","sha256_utf8":"73a7f38cbaf868d414d26183ede659d930352fdcf19467497562fa4efd9b6fab"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"设定高度系数获得高概率","source_units":[{"start":59278,"end":59865,"start_line":718,"end_line":725,"raw_tex":"%\tNote that $\\left(\\frac{q}{p}\\right)^{\\log_a(n)}=n^{\\log_a(\\frac{q}{p})}$, hence\n%\t$$\\P(N(k,n) =0 )\\geq \n%\t1- n^{1-\\log_a(\\frac{p}{q})}   -\n%\t\\frac{2 p\\sqrt{q}}{\\sqrt{p}(1-2 \\sqrt{p q})^{2}}n^{-\\frac 1 2\\log_a(\\frac p q)}.$$\n\t\\rev{In going from the first line to the second, we used the following fact about logarithms: $x^{a \\log n} = n^{a \\log x} = n^{-a\\log (1/x)}$.} Let $a = \\frac{1+\\omega}{\\log (1/2q)}$. Then, $a \\log(\\frac{p}{q})  \\geq  a \\log(\\frac{1}{2q}) = 1+\\omega$, so \n\t$$\\P(N(l,n) =0 )\\geq \n\t1- n^{-\\omega}   -\n\t\\frac{2 \\sqrt{pq}}{(1-2 \\sqrt{p q})^{2}}n^{-(1+\\omega)}.$$\n","sha256_utf8":"a5e6942c9a0f50d5544d3166f1db9d0f61178dd957a22d3cac99de8cf1a7ed3c"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"由耦合转回算法步长下界","source_units":[{"start":59865,"end":60417,"start_line":726,"end_line":728,"raw_tex":"\tIn other words, with probability at least $1 - n^{-\\omega} - cn^{-(1+\\omega)}$ with $c = \\frac{2\\sqrt{pq}}{(1-2\\sqrt{pq})^2}$, the random walk will remain below $l=\\ceil{(1+\\omega)\\log_{1/2q} n}$ in the first $n$ steps. By construction of the coupling, with the above probability, we know the  $\\alpha$ parameter in the algorithm either remains above $\\gamma^{\\ceil{a\\log n}}\\bar{\\alpha} = \\gamma^{\\ceil{(1+\\omega)\\log_{1/2q} n}}\\bar{\\alpha}$  throughout the first $n$ steps, or the algorithm has reached its stopping time in $n$ steps. \n \n\\end{proof}","sha256_utf8":"c9738401e774afd8c36c6d22d99addeb9f4b72fdb1a435eeb7669916263f819d"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"定义随机游走诱导的总成本","source_units":[{"start":67063,"end":67748,"start_line":823,"end_line":827,"raw_tex":"\\begin{proof}\n%\tWe now want to derive an upper bound on the sample complexity $\\SC(T_{\\eps})$ that holds with high probability.  With probability at least $ 1 - \\P(T_\\eps> N)$, the algorithm reaches the stopping time in $N$ iterations, in other words $T_{\\eps}\\leq N$ with high probability. \n\t\n\tLet $\\SC_{\\mathsf{rw}}(n)$ be the total oracle complexity of the first $n$ iterations with the corresponding sequence of parameters $\\alpha_k$ induced by the one-sided random walk (that is, the sequence defined by $\\alpha_k = \\bar{\\alpha}\\gamma^{Z_k}$, where $Z_k$ is defined in \\Cref{sec:randwalk}). In other words, \n\t$$\\SC_{\\mathsf{rw}}(n) = \\sum_{k=1}^n \\sc(\\bar{\\alpha}\\gamma^{Z_k}).$$\n","sha256_utf8":"5b8fa706a24fcd90e09f334bb5d2ac4f2069fc7b4dd28479f9ec2397a8e8387e"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"用耦合控制停止时总成本","source_units":[{"start":67748,"end":68035,"start_line":828,"end_line":830,"raw_tex":"\tWith probability $1 - \\P(T_\\eps > n)$, we have $T_\\eps \\leq n$, which implies\n\t$$\\SC(T_{\\eps})\\leq \\SC_{\\mathsf{rw}}(T_\\eps) \\leq \\SC_{\\mathsf{rw}}(n).$$\n\tHere, the first inequality is by \\Cref{lem:coupling_new} and \\Cref{ass:monotone}, and the second inequality is by $T_\\eps \\leq n$.\n","sha256_utf8":"3897b187613439a5583171981ff70d90853bf51a91347c040b663c8c949ba37b"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"由步长下界控制游走成本","source_units":[{"start":68035,"end":68347,"start_line":831,"end_line":832,"raw_tex":"\t\n\tThe same arguments used in the proof of \\Cref{thm:stepsize_bound} show that with probability at least $1 - n^{-\\omega} - c n^{-(1+\\omega)}$,  we have $\\min_{1 \\leq k \\leq n} \\bar{\\alpha}\\gamma^{Z_k} \\geq \\alpha^*(n)$. Thus, with at least this probability, $\\SC_{\\mathsf{rw}}(n) \\leq n\\cdot\\sc(\\alpha^*(n))$. \n","sha256_utf8":"311936c7a935f3eea8be0cb9e3d68025163f1ea4e26431878bb2ef956d36fb78"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"联合两个事件得到复杂度界","source_units":[{"start":68347,"end":68413,"start_line":833,"end_line":834,"raw_tex":"\t\n\tPutting these together with a union bound, the result follows.\n","sha256_utf8":"de07dcb449e8145d176e40a393de399db32a96814707ae1afe187966beb0cbde"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"代入步长因子得到常数下界情形","source_units":[{"start":68413,"end":68590,"start_line":835,"end_line":838,"raw_tex":"\t\n\tThe second part of the theorem follows from substituting in the specific choice of $\\gamma$.\n%\tTogether with Theorem \\ref{thm:stepsize_bound}, the result follows.\n\\end{proof}","sha256_utf8":"d987dc776f5401201544e082d7896d66d66701c85fa476351ca50a7da6ac1fc7"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S013","kind":"supports"},{"source_id":"S004","target_id":"S010","kind":"supports"},{"source_id":"S006","target_id":"S010","kind":"supports"},{"source_id":"S007","target_id":"S010","kind":"supports"},{"source_id":"S008","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S018","kind":"supports"},{"source_id":"S011","target_id":"S018","kind":"supports"},{"source_id":"S008","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S019","kind":"supports"},{"source_id":"S011","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S020","kind":"supports"},{"source_id":"S011","target_id":"S020","kind":"supports"},{"source_id":"S018","target_id":"S021","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S017","kind":"supports"},{"source_id":"S020","target_id":"S017","kind":"supports"},{"source_id":"S021","target_id":"S017","kind":"supports"},{"source_id":"S022","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S011","target_id":"S025","kind":"supports"},{"source_id":"S012","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S030","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S025","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S027","target_id":"S024","kind":"supports"},{"source_id":"S028","target_id":"S024","kind":"supports"},{"source_id":"S029","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S016","target_id":"S037","kind":"supports"},{"source_id":"S010","target_id":"S037","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S032","kind":"supports"},{"source_id":"S035","target_id":"S032","kind":"supports"},{"source_id":"S036","target_id":"S032","kind":"supports"},{"source_id":"S037","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S011","target_id":"S038","kind":"supports"},{"source_id":"S013","target_id":"S038","kind":"supports"},{"source_id":"S015","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S007","target_id":"S039","kind":"supports"},{"source_id":"S016","target_id":"S039","kind":"supports"},{"source_id":"S014","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S031","target_id":"S040","kind":"supports"},{"source_id":"S032","target_id":"S040","kind":"supports"},{"source_id":"S014","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S014","target_id":"S042","kind":"supports"},{"source_id":"S038","target_id":"S002","kind":"supports"},{"source_id":"S039","target_id":"S002","kind":"supports"},{"source_id":"S040","target_id":"S002","kind":"supports"},{"source_id":"S041","target_id":"S002","kind":"supports"},{"source_id":"S042","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S001","kind":"supports"}]}}
{"record_id":"2305.03559v3","paper":{"title":"On the convergence of proximal gradient methods for convex simple bilevel optimization","authors":["Puya Latafat","Andreas Themelis","Silvia Villa","Panagiotis Patrinos"],"arxiv_id":"2305.03559","version":"v3","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-024-02564-6","abstract_url":"https://arxiv.org/abs/2305.03559v3","pdf_url":"https://arxiv.org/pdf/2305.03559v3"},"context":{"raw_tex":"\\begin{align}\n\t\t\t\t\\label{eq:P1}\n\t\t\t\t\\minimize_{x\\in\\R^n}{} &~ \\cost_1(x)\\coloneqq \\f_1(x)+\\g_1(x)\n\t\t\t\\\\\n\t\t\t\t\\label{eq:P2}\n\t\t\t\t\\stt{} &~ x\\in\\X_2\\coloneqq\\argmin_{w\\in\\R^n}\\set{\\cost_2(w)\\coloneqq \\f_2(w)+\\g_2(w)},\n\t\t\t\\end{align}\n\n\\begin{equation}\\label{eq:phik}\n\t\t\t\\fk\\coloneqq\\sigk\\f_1+\\f_2,\n\t\t\t\\quad\\gk\\coloneqq\\sigk\\g_1+\\g_2,\n\t\t\\quad\\text{and}\\quad\n\t\t\t\\phik\\coloneqq \\fk + \\gk = \\sigk \\cost_1 + \\cost_2,\n\t\t\\end{equation}\n\n\t\t\tThe sets of natural, real, and extended-real numbers are \\(\\N\\), \\(\\R\\coloneqq(-\\infty,\\infty)\\) and \\(\\Rinf\\coloneqq\\R\\cup\\set\\infty\\), respectively, while the positive and strictly positive reals are \\(\\R_+\\coloneqq[0,\\infty)\\) and \\(\\R_{++}\\coloneqq(0,\\infty)\\).\n\t\t\tWe adopt the conventions that \\(0\\in\\N\\) and \\(1/0=\\infty\\).\n\t\t\tGiven \\(x\\in\\Rinf\\), its positive part is indicated as \\([x]_+\\coloneqq\\max\\set{0,x}\\).\n\t\t\tWe denote by \\(\\innprod{\\cdot}{\\cdot}\\) and \\(\\|{}\\cdot{}\\|\\) the standard Euclidean inner product and the induced norm, and with \\(\\id\\) the identity function defined on a suitable space.\n\t\t\tThe closed Euclidean ball of radius \\(r>0\\) and centered at \\(\\bar x\\in\\R^n\\) is denoted as \\(\\cball{\\bar x}{r}\\coloneqq\\set{x\\in\\R^n}[\\|x-\\bar x\\|\\leq r]\\).\n\t\t\tGiven two nonempty sets \\(U,V\\subseteq\\R^n\\), with \\(U+V\\coloneqq\\set{u+v}[u\\in U,v\\in V]\\) we indicate their Minkowski sum, while \\(\\conv V\\) is used to denote the convex hull of \\(V\\).\n\n\n\t\t\tThe \\DEF{domain} and \\DEF{epigraph} of an extended real--valued function \\(\\func{h}{\\R^n}{\\Rinf}\\) are, respectively, the sets\n\t\t\t\\(\n\t\t\t\t\\dom h\n\t\t\t\\coloneqq\n\t\t\t\t\\set{x\\in\\R^n}[h(x)<\\infty]\n\t\t\t\\)\n\t\t\tand\n\t\t\t\\(\n\t\t\t\t\\epi h\n\t\t\t\\coloneqq\n\t\t\t\t\\set{(x, c )\\in\\R^n\\times\\R}[h(x)\\leq c]\n\t\t\t\\).\n\t\t\tFunction \\(h\\) is said to be \\DEF{proper} if \\(\\dom h\\neq\\emptyset\\), and \\DEF{lower semicontinuous (lsc)} if \\(\\epi h\\) is a closed subset of \\(\\R^{n+1}\\).\n\t\t\tWe say that \\(h\\) is \\emph{level bounded} if its \\( c \\)-sublevel set\n\t\t\t\\(\n\t\t\t\t\\lev_{\\leq c }h\n\t\t\t{}\\coloneqq{}\n\t\t\t\t\\set{x\\in\\R^n}[\n\t\t\t\t\th(x)\\leq c\n\t\t\t\t]\n\t\t\t\\)\n\t\t\tis bounded for all \\( c \\in\\R\\).\n\n\n\t\t\tThe projection onto and the distance from $E$ are respectively denoted by\n\t\t\t\\[\n\t\t\t\t\\proj_E(x)\\coloneqq\\argmin_{z\\in E}\\|z-x\\|\n\t\t\t\\quad\\text{and}\\quad\n\t\t\t\t\\dist(x,E)\\coloneqq\\inf_{z\\in E}\\|z-x\\|.\n\t\t\t\\]\n\t\t\tThe (convex) \\DEF{subdifferential} of a proper lsc convex function \\(\\func{h}{\\R^n}{\\Rinf}\\) at a point \\(\\bar x\\) is the set\n\t\t\t\\(\n\t\t\t\t\\partial h(\\bar x)\n\t\t\t\\coloneqq\n\t\t\t\t\\set{v\\in\\R^n}[h(x)\\geq h(\\bar x)+\\innprod{v}{x-\\bar x}\\ \\forall x\\in\\R^n]\n\t\t\t\\).\n\t\t\tThe \\DEF{proximal mapping} of \\(h\\) is \\(\\func{\\prox_h}{\\R^n}{\\R^n}\\) defined by\n\t\t\t\\[\n\t\t\t\t\\prox_h(x)\n\t\t\t=\n\t\t\t\t\\argmin_{w\\in\\R^n}\\set{\n\t\t\t\t\th(w)+\\tfrac12\\|w-x\\|^2\n\t\t\t\t},\n\t\t\t\\]\n\t\t\tand is characterized by the implicit subdifferential inclusion \\cite[Eq. (24.2)]{bauschke2017convex}\n\t\t\t\\begin{equation}\\label{eq:proxsubgrad}\n\t\t\t\tx-\\prox_h(x)\n\t\t\t\\in\n\t\t\t\t\\partial h\\bigl(\\prox_h(x)\\bigr).\n\t\t\t\\end{equation}\n\n\n\\begin{assumption}[basic requirements]\\label{ass:basic}%\n\t\t\t\tThe following hold in problem \\eqref{eq:P}:\n\t\t\t\t\\begin{enumeratass}\n\t\t\t\t\\item \\label{ass:f}%\n\t\t\t\t\t\\(\\f_1,\\f_2:\\R^n\\to \\R\\) are convex and have locally Lipschitz-continuous gradients;\n\t\t\t\t\\item \\label{ass:g}%\n\t\t\t\t\t\\(\\g_1,\\g_2:\\R^n \\to \\Rinf\\) are proper lsc convex functions with easy to compute proximal mappings;\n\t\t\t\t\\item \\label{ass:cost1}%\n\t\t\t\t\tthe upper level problem restricted to \\(\\dom\\cost_2\\) is lower bounded:\n\t\t\t\t\t\\begin{equation}\\label{eq:costinf_1}\n\t\t\t\t\t\t\\costinf_1\n\t\t\t\t\t{}\\coloneqq{}\n\t\t\t\t\t\t\\inf_{x\\in\\dom\\cost_2}\\set{\\f_1(x)+\\g_1(x)}\n\t\t\t\t\t{}>{}\n\t\t\t\t\t\t-\\infty;\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\\item \\label{ass:X1}%\n\t\t\t\t\tthe set of solutions \\(\\X_1\\coloneqq\\argmin\\set{\\cost_1(x)}[x\\in\\argmin\\cost_2]\\) is nonempty and bounded; in particular,\n\t\t\t\t\t\\begin{equation}\\label{eq:costinf_2}\n\t\t\t\t\t\t\\costinf_2\n\t\t\t\t\t{}\\coloneqq{}\n\t\t\t\t\t\t\\inf_{\\fillwidthof[c]{x\\in\\dom \\g_2}{x\\in\\R^n}}\\set{\\f_2(x)+\\g_2(x)}\n\t\t\t\t\t{}>{}\n\t\t\t\t\t\t-\\infty\\mathrlap{.}\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\\end{enumeratass}\n\t\t\t\\end{assumption}\n\n\\begin{equation}\\label{eq:PG}\n\t\t\tx_{k+1}\n\t\t=\n\t\t\t\\prox_{\\alphk*\\gk*}(x_k-\\alphk*\\nabla\\fk*(x_k)).\n\t\t\\end{equation}\n\n\t\t\tSpecifically, we use local Lipschitz estimates of the differentiable functions \\(\\fk=\\sigk\\f_1+\\f_2\\) at the previous iterates \\(x_k, x_{k-1}\\) as%\n\t\t\t\\begin{subequations}\\label{eq:lL}\n\t\t\t\t\\begin{align}\n\t\t\t\t\t\\lk\n\t\t\t\t{}\\coloneqq{} &\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\\nabla\\fk(x_{k-1})-\\nabla\\fk(x_k)}{x_{k-1}-x_k}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t}\n\t\t\t\t\t,\\quad\n\t\t\t\t\t\\Lk\n\t\t\t\t\t{}\\coloneqq {}\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\|\\nabla\\fk(x_{k-1})-\\nabla\\fk(x_k)\\|^2\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t},\n\t\t\t\t\\shortintertext{%\n\t\t\t\t\tand for each \\(\\f_i\\) with\n\t\t\t\t}\n\t\t\t\t\\label{eq:Lk2}\n\t\t\t\t\t\\lk^{(i)}\n\t\t\t\t{}\\coloneqq{} &\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\\nabla\\f_i(x_{k-1})-\\nabla\\f_i(x_k)}{x_{k-1}-x_k}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t},\n\t\t\t\t\\quad\n\t\t\t\t\ti=1,2,\n\t\t\t\t\\end{align}\n\t\t\t\tso that \\(\\lk=\\sigk\\lk^{(1)}+\\lk^{(2)}\\).\n\t\t\t\\end{subequations}\n\t\t\tNoting that the denominator of \\(\\lk\\), \\(\\Lk\\), or \\(\\lk^{(i)}\\) is zero iff \\(\\nabla f(x^k)-\\nabla f(x^{k-1})=0\\), we use the convention \\(\\nicefrac00=0\\) so that both \\(\\lk\\) and \\(\\Lk\\) are (well-defined, positive) real numbers.\n\n\n\t\t\t\tand denoting \\(\\rhok\\coloneqq\\frac{\\sigk\\alphk}{\\sigma_{k-1}\\alpha_{k-1}}\\) initialize the next stepsize as\\algfootnotemark\\textsuperscript{,}\\algfootnotemark\n\t\t\t\t\\begin{equation}\\label{eq:adabim:alphk*_0}\n\t\t\t\t\t\\balphk*\n\t\t\t\t=\n\t\t\t\t\t\\tfrac{\\sigk}{\\sigk*}\n\t\t\t\t\t\\alphk\n\t\t\t\t\t\\min\\set{\n\t\t\t\t\t\t\\sqrt{\\tfrac{\\sigk}{\\sigma_{k-1}}\\bigl(1+\\rhok\\bigr)}\n\t\t\t\t\t\t,\\,\n\t\t\t\t\t\t\\tfrac{\n\t\t\t\t\t\t\t\\sqrt{\n\t\t\t\t\t\t\t\t1\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t4\\left(1-\\frac{\\sigk}{\\sigma_{k-1}}\\right)\n\t\t\t\t\t\t\t\t\\alphk\\lk^{(2)}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t2\\sqrt{[\\alphk^2 \\Lk^2 - \\alphk\\lk]_+}\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\\end{equation}\n\n\n\\begin{remark}[Bar notation for minima and shifted costs]\\label{rem:cost}%\n\t\t\t\tLet us define\n\t\t\t\t\\[\n\t\t\t\t\t\\bcost_i\\coloneqq\\cost_i-\\costinf_i, \\; i=1,2,\n\t\t\t\t\\quad\\text{and}\\quad\n\t\t\t\t\t\\bphik \\coloneqq \\sigk \\bcost_1 + \\bcost_2,\n\t\t\t\t\\]\n\t\t\t\twhere \\(\\costinf_i=\\inf_{\\dom\\cost_2}\\cost_i\\) as in \\cref{ass:basic} (restricting to \\(\\dom\\cost_2\\) is superfluous for \\(i=2\\)), and let\n\t\t\t\t\\begin{equation}\\label{eq:phi*}\n\t\t\t\t\t\\costinf*\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\inf_{\\X_2}\\cost_1\n\t\t\t\t\\end{equation}\n\t\t\t\tbe the optimal cost of problem \\eqref{eq:P}.\n\t\t\t\tThen,\n\t\t\t\t\\begin{enumerate}\n\t\t\t\t\\item\\label{thm:bar>=0}%\n\t\t\t\t\t\\(\\bcost_1(x)\\geq0\\) for any \\(x\\in\\dom\\cost_2\\) and \\(\\bcost_2(x)\\geq0\\) for any \\(x\\in\\R^n\\);\n\t\t\t\t\\item \\label{thm:Phi*}%\n\t\t\t\t\t\\( \\tfrac1{\\sigk}\\left(\\phik(x_\\star)-\\costinf_2\\right)=\\costinf*\\) for any \\(x_\\star\\in\\X_1\\) and \\(k\\in\\N\\).\n\t\t\t\t\\qedhere\n\t\t\t\t\\end{enumerate}\n\t\t\t\\end{remark}\n\n\t\t\tA key step in our convergence analysis based on adaptive stepsizes relies on  the introduction of the quantities in \\eqref{eq:lL}.\n\t\t\tWe define the shorhand notation for the forward operator \\(\\Hk = \\id - \\alphk\\nabla\\fk\\), and note that by optimality conditions of the prox-grad update \\eqref{eq:PG}\n\t\t\t\\begin{equation}\\label{eq:Hk:phi}\n\t\t\t\t\\tfrac1{\\alphk}(\\Hk(x_{k-1})-\\Hk(x_k)) \\in \\partial \\phik(x_k).\n\t\t\t\\end{equation}\n\n\n\\begin{fact}[{\\cite[Lem. 2.1]{latafat2023adaptive}}]\\label{thm:lL}%\n\t\t\t\tSuppose that \\cref{ass:f} holds, and for \\(x_{k-1},x_k\\in\\R^n\\) and \\(\\alphk>0\\) let \\(\\lk\\) and \\(\\Lk\\) be as in \\eqref{eq:lL} and \\(\\Hk\\coloneqq\\id-\\alphk\\nabla\\fk\\).\n\t\t\t\tThen, the following hold:\n\t\t\t\t\\begin{enumerate}\n\t\t\t\t\\item \\label{thm:H}%\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\|\\Hk(x_{k-1})-\\Hk(x_k)\\|\n\t\t\t\t\t=\n\t\t\t\t\t\t\\Mk\n\t\t\t\t\t\t\\|x_{k-1}-x_k\\|\n\t\t\t\t\t\\),\n\t\t\t\t\twhere\n\t\t\t\t\t\\[\n\t\t\t\t\t\t|1-\\alphk\\Lk|\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\Mk\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\sqrt{1 + \\alphk^2 \\Lk^2 -2\\alphk\\lk}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t1+\\alphk\\Lk.\n\t\t\t\t\t\\]\n\t\t\t\t\\item\\label{thm:l<=L}%\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\lk\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\Lk\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\sigk L_{\\f_1,\\V} + L_{\\f_2,\\V}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\sigma_0 L_{\\f_1,\\V} + L_{\\f_2,\\V}\n\t\t\t\t\t\\),\n\t\t\t\t\twhere \\(L_{\\f_i,\\V}\\) is a Lipschitz modulus for \\(\\nabla\\f_i\\) on a convex set \\(\\V\\) containing \\(x_{k-1}\\) and \\(x_k\\), \\(i=1,2\\).\\footnote{%\n\t\t\t\t\t\tThe first inequality owes to Cauchy-Schwarz; the last one to convexity of \\(\\f_1\\) together with the fact that \\(\\sigk\\leq\\sigma_0\\).%\n\t\t\t\t\t}%\n\t\t\t\t\\end{enumerate}\n\t\t\t\\end{fact}\n\n\t\t\t\\begin{equation}\\label{eq:uppercase}\n\t\t\t\t\\Phik=\\Fk+\\Gk\n\t\t\t\\quad\\text{with}\\quad\n\t\t\t\t\\Fk=\\tfrac1{\\sigk}\\fk=\\f_1+\\tfrac1{\\sigk}\\f_2\n\t\t\t\\quad\\text{and}\\quad\n\t\t\t\t\\Gk=\\tfrac1{\\sigk}\\gk=\\g_1+\\tfrac1{\\sigk}\\g_2,\n\t\t\t\\end{equation}\n\t\t\tnoticing that \\(\\Phik=\\cost_1+\\frac{1}{\\sigk}\\cost_2=\\frac{1}{\\sigk}\\phik\\).\n\t\t\tDoing so allows us to express the difference\n\t\t\t\\begin{equation}\n\t\t\t\t\\Phik*-\\Phik\n\t\t\t{}={}\n\t\t\t\t\\left(\\tfrac1{\\sigk*}-\\tfrac1{\\sigk}\\right)\\cost_2\n\t\t\t\\end{equation}\n\t\t\tas a multiple of the lower-level cost \\(\\cost_2\\), rather than of the upper-level cost \\(\\cost_1\\).\n\n\n\t\t\t\t\tThere exist \\(\\alpha_{\\rm max}\\geq\\alpha_{\\rm min}>0\\) and \\(\\nu\\in(0,1)\\) such that, for every \\(k\\in\\N\\),\n\n\t\t\t\t\t\\begin{minipage}[t]{0.57\\linewidth}\n\t\t\t\t\t\t\\vspace*{-.5\\baselineskip}%\n\t\t\t\t\t\t\\begin{alphaproperties}\n\t\t\t\t\t\t\\item \\label{cond:alphk}%\n\t\t\t\t\t\t\t\\(\\alphk*\\leq\\min\\set{\\balphk*,\\alpha_{\\rm max}}\\) with \\(\\balphk*\\) as in \\eqref{eq:adabim:alphk*_0}%\n\n\t\t\t\t\t\t\\item \\label{cond:alphk:nu}%\n\t\t\t\t\t\t\t\\(\\alphk*\\lk*\\leq\\nu\\)\n\n\t\t\t\t\t\t\\item \\label{cond:alphamin}%\n\t\t\t\t\t\t\t\\(\\alphk*\\geq\\alpha_{\\rm min}\\)\n\t\t\t\t\t\t\\end{alphaproperties}\n\t\t\t\t\t\\end{minipage}\n\t\t\t\t\t\\hfill\\vline\\hfill\n\t\t\t\t\t\\begin{minipage}[t]{0.35\\linewidth}\n\t\t\t\t\t\t\\vspace*{-.5\\baselineskip}%\n\t\t\t\t\t\t\\begin{sigmaproperties}\n\t\t\t\t\t\t\\item \\label{cond:monotone}%\n\t\t\t\t\t\t\t\\(0<\\sigk* \\leq \\sigk\\)\n\n\t\t\t\t\t\t\\item \\label{cond:slowcontrol}%\n\t\t\t\t\t\t\t\\(\\sigk\\to 0\\) and \\(\\sum_{k\\in\\N}\\sigk=\\infty\\)\n\t\t\t\t\t\t\\end{sigmaproperties}\n","units":[{"start":16831,"end":17059,"start_line":499,"end_line":505,"raw_tex":"\\begin{align}\n\t\t\t\t\\label{eq:P1}\n\t\t\t\t\\minimize_{x\\in\\R^n}{} &~ \\cost_1(x)\\coloneqq \\f_1(x)+\\g_1(x)\n\t\t\t\\\\\n\t\t\t\t\\label{eq:P2}\n\t\t\t\t\\stt{} &~ x\\in\\X_2\\coloneqq\\argmin_{w\\in\\R^n}\\set{\\cost_2(w)\\coloneqq \\f_2(w)+\\g_2(w)},\n\t\t\t\\end{align}","sha256_utf8":"c50e7bed98b4ef3178e35810923f34d29029979b97eb3f0058e2dc38762e6002"},{"start":18197,"end":18390,"start_line":512,"end_line":517,"raw_tex":"\\begin{equation}\\label{eq:phik}\n\t\t\t\\fk\\coloneqq\\sigk\\f_1+\\f_2,\n\t\t\t\\quad\\gk\\coloneqq\\sigk\\g_1+\\g_2,\n\t\t\\quad\\text{and}\\quad\n\t\t\t\\phik\\coloneqq \\fk + \\gk = \\sigk \\cost_1 + \\cost_2,\n\t\t\\end{equation}","sha256_utf8":"1861b24ab77898557c9f0a1455b76bc2bfcfba16a732b2fdbdcf3b5631e18299"},{"start":31980,"end":32947,"start_line":681,"end_line":686,"raw_tex":"\t\t\tThe sets of natural, real, and extended-real numbers are \\(\\N\\), \\(\\R\\coloneqq(-\\infty,\\infty)\\) and \\(\\Rinf\\coloneqq\\R\\cup\\set\\infty\\), respectively, while the positive and strictly positive reals are \\(\\R_+\\coloneqq[0,\\infty)\\) and \\(\\R_{++}\\coloneqq(0,\\infty)\\).\n\t\t\tWe adopt the conventions that \\(0\\in\\N\\) and \\(1/0=\\infty\\).\n\t\t\tGiven \\(x\\in\\Rinf\\), its positive part is indicated as \\([x]_+\\coloneqq\\max\\set{0,x}\\).\n\t\t\tWe denote by \\(\\innprod{\\cdot}{\\cdot}\\) and \\(\\|{}\\cdot{}\\|\\) the standard Euclidean inner product and the induced norm, and with \\(\\id\\) the identity function defined on a suitable space.\n\t\t\tThe closed Euclidean ball of radius \\(r>0\\) and centered at \\(\\bar x\\in\\R^n\\) is denoted as \\(\\cball{\\bar x}{r}\\coloneqq\\set{x\\in\\R^n}[\\|x-\\bar x\\|\\leq r]\\).\n\t\t\tGiven two nonempty sets \\(U,V\\subseteq\\R^n\\), with \\(U+V\\coloneqq\\set{u+v}[u\\in U,v\\in V]\\) we indicate their Minkowski sum, while \\(\\conv V\\) is used to denote the convex hull of \\(V\\).\n","sha256_utf8":"c7b26999d96c2104cd617a32fa2d383670d04a402e5d2a39f05491d50f6ef42f"},{"start":33060,"end":33707,"start_line":690,"end_line":711,"raw_tex":"\t\t\tThe \\DEF{domain} and \\DEF{epigraph} of an extended real--valued function \\(\\func{h}{\\R^n}{\\Rinf}\\) are, respectively, the sets\n\t\t\t\\(\n\t\t\t\t\\dom h\n\t\t\t\\coloneqq\n\t\t\t\t\\set{x\\in\\R^n}[h(x)<\\infty]\n\t\t\t\\)\n\t\t\tand\n\t\t\t\\(\n\t\t\t\t\\epi h\n\t\t\t\\coloneqq\n\t\t\t\t\\set{(x, c )\\in\\R^n\\times\\R}[h(x)\\leq c]\n\t\t\t\\).\n\t\t\tFunction \\(h\\) is said to be \\DEF{proper} if \\(\\dom h\\neq\\emptyset\\), and \\DEF{lower semicontinuous (lsc)} if \\(\\epi h\\) is a closed subset of \\(\\R^{n+1}\\).\n\t\t\tWe say that \\(h\\) is \\emph{level bounded} if its \\( c \\)-sublevel set\n\t\t\t\\(\n\t\t\t\t\\lev_{\\leq c }h\n\t\t\t{}\\coloneqq{}\n\t\t\t\t\\set{x\\in\\R^n}[\n\t\t\t\t\th(x)\\leq c\n\t\t\t\t]\n\t\t\t\\)\n\t\t\tis bounded for all \\( c \\in\\R\\).\n","sha256_utf8":"f10a016c164d710ab9047aa9e8dc7880175bd2a5cc9d0af84e3023c863ce06e1"},{"start":33867,"end":34734,"start_line":714,"end_line":739,"raw_tex":"\t\t\tThe projection onto and the distance from $E$ are respectively denoted by\n\t\t\t\\[\n\t\t\t\t\\proj_E(x)\\coloneqq\\argmin_{z\\in E}\\|z-x\\|\n\t\t\t\\quad\\text{and}\\quad\n\t\t\t\t\\dist(x,E)\\coloneqq\\inf_{z\\in E}\\|z-x\\|.\n\t\t\t\\]\n\t\t\tThe (convex) \\DEF{subdifferential} of a proper lsc convex function \\(\\func{h}{\\R^n}{\\Rinf}\\) at a point \\(\\bar x\\) is the set\n\t\t\t\\(\n\t\t\t\t\\partial h(\\bar x)\n\t\t\t\\coloneqq\n\t\t\t\t\\set{v\\in\\R^n}[h(x)\\geq h(\\bar x)+\\innprod{v}{x-\\bar x}\\ \\forall x\\in\\R^n]\n\t\t\t\\).\n\t\t\tThe \\DEF{proximal mapping} of \\(h\\) is \\(\\func{\\prox_h}{\\R^n}{\\R^n}\\) defined by\n\t\t\t\\[\n\t\t\t\t\\prox_h(x)\n\t\t\t=\n\t\t\t\t\\argmin_{w\\in\\R^n}\\set{\n\t\t\t\t\th(w)+\\tfrac12\\|w-x\\|^2\n\t\t\t\t},\n\t\t\t\\]\n\t\t\tand is characterized by the implicit subdifferential inclusion \\cite[Eq. (24.2)]{bauschke2017convex}\n\t\t\t\\begin{equation}\\label{eq:proxsubgrad}\n\t\t\t\tx-\\prox_h(x)\n\t\t\t\\in\n\t\t\t\t\\partial h\\bigl(\\prox_h(x)\\bigr).\n\t\t\t\\end{equation}\n","sha256_utf8":"8564864873ff9fad7272f3a439d67e1f9725ac99711e960c54fffc25ce36da85"},{"start":35375,"end":36434,"start_line":751,"end_line":777,"raw_tex":"\\begin{assumption}[basic requirements]\\label{ass:basic}%\n\t\t\t\tThe following hold in problem \\eqref{eq:P}:\n\t\t\t\t\\begin{enumeratass}\n\t\t\t\t\\item \\label{ass:f}%\n\t\t\t\t\t\\(\\f_1,\\f_2:\\R^n\\to \\R\\) are convex and have locally Lipschitz-continuous gradients;\n\t\t\t\t\\item \\label{ass:g}%\n\t\t\t\t\t\\(\\g_1,\\g_2:\\R^n \\to \\Rinf\\) are proper lsc convex functions with easy to compute proximal mappings;\n\t\t\t\t\\item \\label{ass:cost1}%\n\t\t\t\t\tthe upper level problem restricted to \\(\\dom\\cost_2\\) is lower bounded:\n\t\t\t\t\t\\begin{equation}\\label{eq:costinf_1}\n\t\t\t\t\t\t\\costinf_1\n\t\t\t\t\t{}\\coloneqq{}\n\t\t\t\t\t\t\\inf_{x\\in\\dom\\cost_2}\\set{\\f_1(x)+\\g_1(x)}\n\t\t\t\t\t{}>{}\n\t\t\t\t\t\t-\\infty;\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\\item \\label{ass:X1}%\n\t\t\t\t\tthe set of solutions \\(\\X_1\\coloneqq\\argmin\\set{\\cost_1(x)}[x\\in\\argmin\\cost_2]\\) is nonempty and bounded; in particular,\n\t\t\t\t\t\\begin{equation}\\label{eq:costinf_2}\n\t\t\t\t\t\t\\costinf_2\n\t\t\t\t\t{}\\coloneqq{}\n\t\t\t\t\t\t\\inf_{\\fillwidthof[c]{x\\in\\dom \\g_2}{x\\in\\R^n}}\\set{\\f_2(x)+\\g_2(x)}\n\t\t\t\t\t{}>{}\n\t\t\t\t\t\t-\\infty\\mathrlap{.}\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\\end{enumeratass}\n\t\t\t\\end{assumption}","sha256_utf8":"0fac96d3016b3ce3a46d6e5e2907951efe58c051f021fc660494351dca7eedb9"},{"start":37008,"end":37121,"start_line":784,"end_line":788,"raw_tex":"\\begin{equation}\\label{eq:PG}\n\t\t\tx_{k+1}\n\t\t=\n\t\t\t\\prox_{\\alphk*\\gk*}(x_k-\\alphk*\\nabla\\fk*(x_k)).\n\t\t\\end{equation}","sha256_utf8":"48b7bf49e3cf04c306cc995b25fcaa305f7b004b1c8f5a5467e4aceffc3ddd31"},{"start":48362,"end":49423,"start_line":944,"end_line":978,"raw_tex":"\t\t\tSpecifically, we use local Lipschitz estimates of the differentiable functions \\(\\fk=\\sigk\\f_1+\\f_2\\) at the previous iterates \\(x_k, x_{k-1}\\) as%\n\t\t\t\\begin{subequations}\\label{eq:lL}\n\t\t\t\t\\begin{align}\n\t\t\t\t\t\\lk\n\t\t\t\t{}\\coloneqq{} &\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\\nabla\\fk(x_{k-1})-\\nabla\\fk(x_k)}{x_{k-1}-x_k}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t}\n\t\t\t\t\t,\\quad\n\t\t\t\t\t\\Lk\n\t\t\t\t\t{}\\coloneqq {}\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\|\\nabla\\fk(x_{k-1})-\\nabla\\fk(x_k)\\|^2\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t},\n\t\t\t\t\\shortintertext{%\n\t\t\t\t\tand for each \\(\\f_i\\) with\n\t\t\t\t}\n\t\t\t\t\\label{eq:Lk2}\n\t\t\t\t\t\\lk^{(i)}\n\t\t\t\t{}\\coloneqq{} &\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\\nabla\\f_i(x_{k-1})-\\nabla\\f_i(x_k)}{x_{k-1}-x_k}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t},\n\t\t\t\t\\quad\n\t\t\t\t\ti=1,2,\n\t\t\t\t\\end{align}\n\t\t\t\tso that \\(\\lk=\\sigk\\lk^{(1)}+\\lk^{(2)}\\).\n\t\t\t\\end{subequations}\n\t\t\tNoting that the denominator of \\(\\lk\\), \\(\\Lk\\), or \\(\\lk^{(i)}\\) is zero iff \\(\\nabla f(x^k)-\\nabla f(x^{k-1})=0\\), we use the convention \\(\\nicefrac00=0\\) so that both \\(\\lk\\) and \\(\\Lk\\) are (well-defined, positive) real numbers.\n","sha256_utf8":"b175d3349de60e87dd3207faf82dc5cbd142150fd93583ec88a7d312e0b98966"},{"start":54814,"end":55389,"start_line":1088,"end_line":1108,"raw_tex":"\t\t\t\tand denoting \\(\\rhok\\coloneqq\\frac{\\sigk\\alphk}{\\sigma_{k-1}\\alpha_{k-1}}\\) initialize the next stepsize as\\algfootnotemark\\textsuperscript{,}\\algfootnotemark\n\t\t\t\t\\begin{equation}\\label{eq:adabim:alphk*_0}\n\t\t\t\t\t\\balphk*\n\t\t\t\t=\n\t\t\t\t\t\\tfrac{\\sigk}{\\sigk*}\n\t\t\t\t\t\\alphk\n\t\t\t\t\t\\min\\set{\n\t\t\t\t\t\t\\sqrt{\\tfrac{\\sigk}{\\sigma_{k-1}}\\bigl(1+\\rhok\\bigr)}\n\t\t\t\t\t\t,\\,\n\t\t\t\t\t\t\\tfrac{\n\t\t\t\t\t\t\t\\sqrt{\n\t\t\t\t\t\t\t\t1\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t4\\left(1-\\frac{\\sigk}{\\sigma_{k-1}}\\right)\n\t\t\t\t\t\t\t\t\\alphk\\lk^{(2)}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t2\\sqrt{[\\alphk^2 \\Lk^2 - \\alphk\\lk]_+}\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\\end{equation}\n","sha256_utf8":"5398aa5cc629c2d04d4d31ac59b6733c9a4befa10123d20495d8b61c4953b64c"},{"start":63303,"end":64190,"start_line":1256,"end_line":1278,"raw_tex":"\\begin{remark}[Bar notation for minima and shifted costs]\\label{rem:cost}%\n\t\t\t\tLet us define\n\t\t\t\t\\[\n\t\t\t\t\t\\bcost_i\\coloneqq\\cost_i-\\costinf_i, \\; i=1,2,\n\t\t\t\t\\quad\\text{and}\\quad\n\t\t\t\t\t\\bphik \\coloneqq \\sigk \\bcost_1 + \\bcost_2,\n\t\t\t\t\\]\n\t\t\t\twhere \\(\\costinf_i=\\inf_{\\dom\\cost_2}\\cost_i\\) as in \\cref{ass:basic} (restricting to \\(\\dom\\cost_2\\) is superfluous for \\(i=2\\)), and let\n\t\t\t\t\\begin{equation}\\label{eq:phi*}\n\t\t\t\t\t\\costinf*\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\inf_{\\X_2}\\cost_1\n\t\t\t\t\\end{equation}\n\t\t\t\tbe the optimal cost of problem \\eqref{eq:P}.\n\t\t\t\tThen,\n\t\t\t\t\\begin{enumerate}\n\t\t\t\t\\item\\label{thm:bar>=0}%\n\t\t\t\t\t\\(\\bcost_1(x)\\geq0\\) for any \\(x\\in\\dom\\cost_2\\) and \\(\\bcost_2(x)\\geq0\\) for any \\(x\\in\\R^n\\);\n\t\t\t\t\\item \\label{thm:Phi*}%\n\t\t\t\t\t\\( \\tfrac1{\\sigk}\\left(\\phik(x_\\star)-\\costinf_2\\right)=\\costinf*\\) for any \\(x_\\star\\in\\X_1\\) and \\(k\\in\\N\\).\n\t\t\t\t\\qedhere\n\t\t\t\t\\end{enumerate}\n\t\t\t\\end{remark}","sha256_utf8":"576fb59e20e4ddedf2b4bcaf5a5de28afc9e42a130fca626508fcc19c5e85802"},{"start":64192,"end":64619,"start_line":1280,"end_line":1284,"raw_tex":"\t\t\tA key step in our convergence analysis based on adaptive stepsizes relies on  the introduction of the quantities in \\eqref{eq:lL}.\n\t\t\tWe define the shorhand notation for the forward operator \\(\\Hk = \\id - \\alphk\\nabla\\fk\\), and note that by optimality conditions of the prox-grad update \\eqref{eq:PG}\n\t\t\t\\begin{equation}\\label{eq:Hk:phi}\n\t\t\t\t\\tfrac1{\\alphk}(\\Hk(x_{k-1})-\\Hk(x_k)) \\in \\partial \\phik(x_k).\n\t\t\t\\end{equation}\n","sha256_utf8":"7f54e089a24f51148b12a9b73cb16468f3d17e3e9af0f5377bfd54471da62aa1"},{"start":64827,"end":65893,"start_line":1287,"end_line":1322,"raw_tex":"\\begin{fact}[{\\cite[Lem. 2.1]{latafat2023adaptive}}]\\label{thm:lL}%\n\t\t\t\tSuppose that \\cref{ass:f} holds, and for \\(x_{k-1},x_k\\in\\R^n\\) and \\(\\alphk>0\\) let \\(\\lk\\) and \\(\\Lk\\) be as in \\eqref{eq:lL} and \\(\\Hk\\coloneqq\\id-\\alphk\\nabla\\fk\\).\n\t\t\t\tThen, the following hold:\n\t\t\t\t\\begin{enumerate}\n\t\t\t\t\\item \\label{thm:H}%\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\|\\Hk(x_{k-1})-\\Hk(x_k)\\|\n\t\t\t\t\t=\n\t\t\t\t\t\t\\Mk\n\t\t\t\t\t\t\\|x_{k-1}-x_k\\|\n\t\t\t\t\t\\),\n\t\t\t\t\twhere\n\t\t\t\t\t\\[\n\t\t\t\t\t\t|1-\\alphk\\Lk|\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\Mk\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\sqrt{1 + \\alphk^2 \\Lk^2 -2\\alphk\\lk}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t1+\\alphk\\Lk.\n\t\t\t\t\t\\]\n\t\t\t\t\\item\\label{thm:l<=L}%\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\lk\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\Lk\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\sigk L_{\\f_1,\\V} + L_{\\f_2,\\V}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\sigma_0 L_{\\f_1,\\V} + L_{\\f_2,\\V}\n\t\t\t\t\t\\),\n\t\t\t\t\twhere \\(L_{\\f_i,\\V}\\) is a Lipschitz modulus for \\(\\nabla\\f_i\\) on a convex set \\(\\V\\) containing \\(x_{k-1}\\) and \\(x_k\\), \\(i=1,2\\).\\footnote{%\n\t\t\t\t\t\tThe first inequality owes to Cauchy-Schwarz; the last one to convexity of \\(\\f_1\\) together with the fact that \\(\\sigk\\leq\\sigma_0\\).%\n\t\t\t\t\t}%\n\t\t\t\t\\end{enumerate}\n\t\t\t\\end{fact}","sha256_utf8":"0452940b569d1ceaf9a1e081ff565ef6b0e92a60f0b404979437dc6a7409690f"},{"start":71699,"end":72275,"start_line":1425,"end_line":1439,"raw_tex":"\t\t\t\\begin{equation}\\label{eq:uppercase}\n\t\t\t\t\\Phik=\\Fk+\\Gk\n\t\t\t\\quad\\text{with}\\quad\n\t\t\t\t\\Fk=\\tfrac1{\\sigk}\\fk=\\f_1+\\tfrac1{\\sigk}\\f_2\n\t\t\t\\quad\\text{and}\\quad\n\t\t\t\t\\Gk=\\tfrac1{\\sigk}\\gk=\\g_1+\\tfrac1{\\sigk}\\g_2,\n\t\t\t\\end{equation}\n\t\t\tnoticing that \\(\\Phik=\\cost_1+\\frac{1}{\\sigk}\\cost_2=\\frac{1}{\\sigk}\\phik\\).\n\t\t\tDoing so allows us to express the difference\n\t\t\t\\begin{equation}\n\t\t\t\t\\Phik*-\\Phik\n\t\t\t{}={}\n\t\t\t\t\\left(\\tfrac1{\\sigk*}-\\tfrac1{\\sigk}\\right)\\cost_2\n\t\t\t\\end{equation}\n\t\t\tas a multiple of the lower-level cost \\(\\cost_2\\), rather than of the upper-level cost \\(\\cost_1\\).\n","sha256_utf8":"d9a42be4b8e19028e5d1b4f3d6c06492413d2c719a1a939bddd6f2ea0cbb2358"},{"start":85191,"end":86059,"start_line":1919,"end_line":1943,"raw_tex":"\t\t\t\t\tThere exist \\(\\alpha_{\\rm max}\\geq\\alpha_{\\rm min}>0\\) and \\(\\nu\\in(0,1)\\) such that, for every \\(k\\in\\N\\),\n\n\t\t\t\t\t\\begin{minipage}[t]{0.57\\linewidth}\n\t\t\t\t\t\t\\vspace*{-.5\\baselineskip}%\n\t\t\t\t\t\t\\begin{alphaproperties}\n\t\t\t\t\t\t\\item \\label{cond:alphk}%\n\t\t\t\t\t\t\t\\(\\alphk*\\leq\\min\\set{\\balphk*,\\alpha_{\\rm max}}\\) with \\(\\balphk*\\) as in \\eqref{eq:adabim:alphk*_0}%\n\n\t\t\t\t\t\t\\item \\label{cond:alphk:nu}%\n\t\t\t\t\t\t\t\\(\\alphk*\\lk*\\leq\\nu\\)\n\n\t\t\t\t\t\t\\item \\label{cond:alphamin}%\n\t\t\t\t\t\t\t\\(\\alphk*\\geq\\alpha_{\\rm min}\\)\n\t\t\t\t\t\t\\end{alphaproperties}\n\t\t\t\t\t\\end{minipage}\n\t\t\t\t\t\\hfill\\vline\\hfill\n\t\t\t\t\t\\begin{minipage}[t]{0.35\\linewidth}\n\t\t\t\t\t\t\\vspace*{-.5\\baselineskip}%\n\t\t\t\t\t\t\\begin{sigmaproperties}\n\t\t\t\t\t\t\\item \\label{cond:monotone}%\n\t\t\t\t\t\t\t\\(0<\\sigk* \\leq \\sigk\\)\n\n\t\t\t\t\t\t\\item \\label{cond:slowcontrol}%\n\t\t\t\t\t\t\t\\(\\sigk\\to 0\\) and \\(\\sum_{k\\in\\N}\\sigk=\\infty\\)\n\t\t\t\t\t\t\\end{sigmaproperties}\n","sha256_utf8":"45e046d6cea53513e69b70f61a2383c2bd8aa851d70822ca9fc367160405c19a"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}[convergence recipe for proximal gradient iterations]\\label{thm:recipe}%\n\t\t\t\tSuppose that \\cref{ass:basic} holds, and consider the proximal gradient iterations \\eqref{eq:PG} with \\(\\seq{\\sigk}\\) and \\(\\seq{\\alphk}\\) complying with all \\cref{cond:alphk,cond:alphk:nu,cond:alphamin,cond:monotone,cond:slowcontrol}.\n\t\t\t\tThen,\n\t\t\t\t\\begin{enumerate}\n\t\t\t\t\\item \\label{thm:recipe:solution}%\n\t\t\t\t\t (\\(\\seq{x_k}\\) is bounded and) \\(\\seq{\\dist(x_k, \\X_1)}\\) converges to zero.\n\t\t\t\t\t% \\(\\seq{x_k}\\) is bounded and all its limit points are solutions to the bilevel problem \\eqref{eq:P};\n\t\t\t\t\\item \\label{thm:recipe:rate}%\n\t\t\t\t\tboth \\(\\bphik(x_k)\\) and \\(W_k\\) as in \\cref{thm:LW} converge to 0 as \\(k\\to\\infty\\), and the following worst-case rate holds\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\min_{k\\leq K}\\dist^2(0, \\partial \\phik*(x_{k+1})\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\tfrac{ (1+ \\alpha_{\\rm max}L_{f_0, \\V})^2}{\\alpha_{\\rm min}(1-\\nu)(K+1)}\\bar{\\originalvarphi}_0(x_0),\n\t\t\t\t\t\\]\n\t\t\t\t\twhere \\(L_{f_0, \\V}\\) is a Lipschitz modulus for \\(\\sigma_0\\nabla\\f_1+\\nabla\\f_2\\) on \\(\\V\\coloneqq\\conv\\set{x^k}[k\\in\\N]\\).\n\t\t\t\t\\end{enumerate}\n\t\t\t\\end{theorem}","units":[{"start":96793,"end":97906,"start_line":2249,"end_line":2265,"raw_tex":"\\begin{theorem}[convergence recipe for proximal gradient iterations]\\label{thm:recipe}%\n\t\t\t\tSuppose that \\cref{ass:basic} holds, and consider the proximal gradient iterations \\eqref{eq:PG} with \\(\\seq{\\sigk}\\) and \\(\\seq{\\alphk}\\) complying with all \\cref{cond:alphk,cond:alphk:nu,cond:alphamin,cond:monotone,cond:slowcontrol}.\n\t\t\t\tThen,\n\t\t\t\t\\begin{enumerate}\n\t\t\t\t\\item \\label{thm:recipe:solution}%\n\t\t\t\t\t (\\(\\seq{x_k}\\) is bounded and) \\(\\seq{\\dist(x_k, \\X_1)}\\) converges to zero.\n\t\t\t\t\t% \\(\\seq{x_k}\\) is bounded and all its limit points are solutions to the bilevel problem \\eqref{eq:P};\n\t\t\t\t\\item \\label{thm:recipe:rate}%\n\t\t\t\t\tboth \\(\\bphik(x_k)\\) and \\(W_k\\) as in \\cref{thm:LW} converge to 0 as \\(k\\to\\infty\\), and the following worst-case rate holds\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\min_{k\\leq K}\\dist^2(0, \\partial \\phik*(x_{k+1})\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\tfrac{ (1+ \\alpha_{\\rm max}L_{f_0, \\V})^2}{\\alpha_{\\rm min}(1-\\nu)(K+1)}\\bar{\\originalvarphi}_0(x_0),\n\t\t\t\t\t\\]\n\t\t\t\t\twhere \\(L_{f_0, \\V}\\) is a Lipschitz modulus for \\(\\sigma_0\\nabla\\f_1+\\nabla\\f_2\\) on \\(\\V\\coloneqq\\conv\\set{x^k}[k\\in\\N]\\).\n\t\t\t\t\\end{enumerate}\n\t\t\t\\end{theorem}","sha256_utf8":"f5862a82c0e9215b4976c00d6b385522b714346360451f382a48e874498ba4d5"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}[quasi-descent inequality]\\label{thm:LW}%\n\t\t\t\tSuppose that \\cref{ass:basic} holds and consider iterations \\eqref{eq:PG} with \\(0<\\sigk*\\leq\\sigk\\).\n\t\t\t\tLet \\(\\bcost_2\\coloneqq\\cost_2-\\costinf_2\\geq0\\) be as in \\cref{rem:cost}, and define \\(\\Delk\\coloneqq\\frac{1}{\\sigk}-\\frac{1}{\\sigma_{k-1}}\\), \\(\\rhok*\\coloneqq\\frac{\\sigk*\\alphk*}{\\sigk\\alphk}\\), and\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\mathcal L_k(x_\\star)\n\t\t\t\t\\coloneqq{} &\n\t\t\t\t\t\\tfrac12\\|x_k-x_\\star\\|^2\n\t\t\t\t\t+\n\t\t\t\t\tW_k\n\t\t\t\t\\shortintertext{with}\n\t\t\t\t\tW_k\n\t\t\t\t\\coloneqq{} &\n\t\t\t\t\t\\tfrac{1-4\\alphk\\sigk\\lk^{(2)}\\Delk}{4}\\|x_k-x_{k-1}\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\sigk*}{\\sigk}\\alphk*\\rhok*\n\t\t\t\t\t\\bcost_2(x_{k-1})\n\t\t\t\t\\\\\n\t\t\t\t&\n\t\t\t\t\t+\n\t\t\t\t\t\\sigk\\alphk\\Delk\\bcost_2(x_k)\n\t\t\t\t\t+\n\t\t\t\t\t\\sigk\\alphk\\left(1+\\rhok\\right)\n\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\\cost_1(x_{k-1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\costinf*\n\t\t\t\t\t\\bigr).\n\t\t\t\t\\end{align*}\n\t\t\t\tThen, for every \\(k\\in\\N\\) and \\(x_\\star\\in\\X_1\\) it holds that\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\mathcal L_{k+1}(x_\\star)\n\t\t\t\t\t-\n\t\t\t\t\t\\mathcal L_k(x_\\star)\n\t\t\t\t\\leq{} &\n\t\t\t\t\t-\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t\\tfrac{1}{4}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\rhok*^2(\\alphk\\lk-\\alphk^2\\Lk^2)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\sigk\\alphk\\lk^{(2)}\\Delk\n\t\t\t\t\t\\right)\n\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\\\\\n\t\t\t\t&\n\t\t\t\t\t-\n\t\t\t\t\t\\sigk\\alphk\n\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t1+\\rhok-\\rhok*^2\n\t\t\t\t\t\\bigr)\n\t\t\t\t\t\\bigl(\\cost_1(x_{k-1})-\\costinf*\\bigr)\n\t\t\t\t\\\\\n\t\t\t\t&\n\t\t\t\t\t-\n\t\t\t\t\t\\sigk*\\alphk*\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t\\tfrac{\\Delk}{\\rhok*}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{\\sigk}\\left(\n\t\t\t\t\t\t\t1+\\rhok*\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\rho_{k+2}^2\\tfrac{\\sigk}{\\sigk*}\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\\right)\n\t\t\t\t\t\\bcost_2(x_k).\n\t\t\t\t\\numberthis\\label{eq:puya:muk:descent}\n\t\t\t\t\\end{align*}\n\t\t\t\t\\begin{proof}\n\t\t\t\tLet\n\t\t\t\t\\begin{equation}\n\t\t\t\t\tp_k%\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\phik(x_k)-\\phik(x_\\star)\n\t\t\t\t=\n\t\t\t\t\t\\sigk\\bigl(\\cost_1(x_k)-\\costinf*\\bigr)\n\t\t\t\t\t+\n\t\t\t\t\t\\bcost_2(x_k)\n\t\t\t\t\\end{equation}\n\t\t\t\t(which is not necessarily positive).\n\t\t\t\tWe will prove the claim using the uppercase notation of \\eqref{eq:uppercase} and, consistently, set\n\t\t\t\t\t\\(P_k%\n\t\t\t\t\t\\coloneqq{\\tfrac{1}{\\sigk}}p_k%\n\t\t\t\t\t=\\Phik(x_k)-\\Phik(x_\\star)\n\t\t\t\t\t\\).\n\t\t\t\t\tBeing a simple matter of multiplicative constants, note that proximal gradient iterations \\eqref{eq:PG} can equivalently be expressed in the penalized cost reformulation \\(\\Phik*=\\Fk*+\\Gk*\\) up to suitable scaling of the stepsize, namely,\n\t\t\t\t\t\\[\n\t\t\t\t\t\tx_{k+1}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\prox_{\\sigk*\\alphk*\\Gk*}(x_k-\\sigk*\\alphk*\\nabla\\Fk*(x_k)).\n\t\t\t\t\t\\]\n\t\t\t\t\tThe subgradient characterization of the proximal mapping as in \\eqref{eq:Hk:phi} yields\n\t\t\t\t\t\\begin{equation}\\label{eq:subgrad}\n\t\t\t\t\t\t\\tfrac{\\Hk(x_{k-1})-x_k}{\\sigk\\alphk}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\tfrac{x_{k-1}-x_k}{\\sigk\\alphk}-\\nabla\\Fk(x_{k-1})\n\t\t\t\t\t\\in\n\t\t\t\t\t\t\\partial\\Gk(x_k),\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\twhere we remind that \\(\\Hk=\\id-\\alphk\\nabla\\fk\\), cf. \\eqref{eq:Hk:phi}.\n\t\t\t\t\tHence, since \\(\\partial\\Phik=\\nabla\\Fk+\\partial\\Gk\\),\n\t\t\t\t\t\\begin{subequations}\n\t\t\t\t\t\t\\begin{align}\n\t\t\t\t\t\t\\nonumber\n\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\Phik(x_{k-1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Phik(x_k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\\innprod{x_{k-1}-x_k}{x_{k-1}-x_k}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk(x_{k-1})-\\nabla\\Fk(x_k)}{x_{k-1}-x_k}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\nonumber\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\Phik(x_{k-1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Phik(x_k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac1{\\sigk}\\innprod{\\nabla\\fk(x_{k-1})-\\nabla\\fk(x_k)}{x_{k-1}-x_k}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\Phik(x_{k-1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Phik(x_k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1-\\alphk\\lk}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t\t\\label{eq:phikdescent}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\tP_{k-1}%\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\tP_k%\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t(\\tfrac{1}{\\sigk}-\\tfrac{1}{\\sigma_{k-1}})\\bcost_2(x_{k-1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1-\\alphk\\lk}{\\sigk\\alphk}\\|x_{k-1}-x_k\\|^2.\n\t\t\t\t\t\t\\label{eq:ineqsubgrad}\n\t\t\t\t\t\t\\end{align}\n\t\t\t\t\t\\end{subequations}\n\t\t\t\t\tAgain from \\eqref{eq:subgrad}, this time with \\(k\\gets k+1\\), we have\n\t\t\t\t\t\\begin{subequations}\\label{subeq:3ineq}\n\t\t\t\t\t\t{\\mathtight[0.6]%\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\Gk*(x_\\star)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Gk*(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_\\star-x_{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\innprod{x_k-x_{k+1}}{x_\\star-x_{k+1}}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\Gk*(x_\\star)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Gk*(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_\\star-x_{k+1}}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_\\star\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_\\star-x_{k+1}\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t&\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\Gk*(x_\\star)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Gk*(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_\\star-x_k}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_\\star\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t&\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_\\star-x_{k+1}\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\Gk*(x_\\star)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Gk*(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\fillwidthof[c]{\n\t\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_\\star-x_k}\n\t\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\t\\Fk*(x_\\star)-\\Fk*(x_k)\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\underbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t}_{\\text{(A)}}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_\\star\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t&\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_\\star-x_{k+1}\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_{k+1}\\|^2,\n\t\t\t\t\t\t\\numberthis\\label{eq:ineq1}\n\t\t\t\t\t\t\\end{align*}}%\n\t\t\t\t\t\twhere the last inequality uses convexity of \\(\\Fk*\\).\n\t\t\t\t\t\tAs to term (A), we have\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\text{(A)}\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\\innprod{\\Hk(x_{k-1})-x_k}{x_{k+1}-x_k}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\\innprod{\\Hk(x_{k-1})-x_k+\\sigk\\alphk\\nabla\\Fk*(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:subgrad}}{} &\n\t\t\t\t\t\t\t\\fillwidthof[c]{\n\t\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\t\\innprod{\\Hk(x_{k-1})-x_k}{x_{k+1}-x_k}\n\t\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\t\\Gk(x_{k+1})-\\Gk(x_k)\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)-\\nabla\\Fk(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t&\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\t\\innprod{\\Hk(x_{k-1})-\\Hk(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t}_{\\text{(B)}}.\n\t\t\t\t\t\t\\numberthis\\label{eq:ineqA}\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\tNext, we bound the term (B) by \\(\\epsk*\\)-Young's inequality as\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\text{(B)}\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\tfrac{\\epsk*}{2\\sigk\\alphk}\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\epsk*\\sigk\\alphk}\\|\\Hk(x_{k-1})-\\Hk(x_k)\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\overrel{\\ref{thm:H}}{} &\n\t\t\t\t\t\t\t\\tfrac{\\epsk*}{2\\sigk\\alphk}\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{\\Mk^2}{2\\epsk*\\sigk\\alphk}\\|x_{k-1}-x_k\\|^2.\n\t\t\t\t\t\t\\numberthis\\label{eq:ineqB}\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\\end{subequations}\n\t\t\t\t\tCombining the three inequalities \\eqref{subeq:3ineq} yields\n\t\t\t\t\t{\\mathtight[0.9]%\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t0\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\Phik*(x_\\star)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\Gk*(x_{k+1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\Fk*(x_k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\Gk(x_{k+1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\Gk(x_k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t(\\tfrac1{\\sigk*}-\\tfrac{1}{\\sigk})\n\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\Biggl\\{\n\t\t\t\t\t\t\t\\tfrac{\\Mk^2}{2\\epsk*\\sigk\\alphk}\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\tfrac{\\epsk*}{2\\sigk\\alphk}\n\t\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\t\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_\\star-x_{k+1}\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t\\hphantom{+\\Biggl\\{{}}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_\\star\\|^2\n\t\t\t\t\t\t\\Biggr\\}\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\overrel{\\ref{thm:Phi*}}{} &\n\t\t\t\t\t\t\\costinf*\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac1{\\sigk*}\\costinf_2\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\Fk(x_k)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\Gk(x_k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\bigl(\\tfrac1{\\sigk*}-\\tfrac{1}{\\sigk}\\bigr)\n\t\t\t\t\t\t\\vphantom{\\f^2}\n\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\f_2(x_k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\g_2(x_{k+1})\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\{\\cdots\\}\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t-P_k%\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\t\\tfrac1{\\sigk*}-\\tfrac{1}{\\sigk}\n\t\t\t\t\t\t\t\t\\vphantom{\\f^2}\n\t\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\\,\n\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\f_2(x_k)\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\g_2(x_{k+1})\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\costinf_2\n\t\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t}^{\\text{(D)}}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\Bigl\\{\\cdots\\Bigr\\}.\n\t\t\t\t\t\\end{align*}}%\n\t\t\t\t\tBy using convexity of \\(\\f_2\\) we can bound the term (D) as\n\t\t\t\t\t{\\mathtight[0.75]%\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\text{(D)}\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_{k+1})}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\f_2(x_k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\g_2(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_{k+1})-\\nabla \\f_2(x_k)}{x_{k+1}-x_k}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\costinf_2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\fillwidthof[c]{\n\t\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_{k+1})}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\f_2(x_k)\n\t\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\t-\\f_2(x_{k+1})\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\g_2(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\fillwidthof[c]{\n\t\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_{k+1})-\\nabla \\f_2(x_k)}{x_{k+1}-x_k}\n\t\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\t\\lk*^{(2)}\\|x_{k+1}-x_k\\|^2\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\costinf_2,\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t}%\n\t\t\t\t\twhich plugged in the previous inequality results in\n\t\t\t\t\t{\\mathtight[0.75]%\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t0\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t-P_k%\n\t\t\t\t\t\t-\n\t\t\t\t\t\t(\\tfrac1{\\sigk*}-\\tfrac{1}{\\sigk})\\bcost_2(x_{k+1})\n\t\t\t\t\t\t+\n\t\t\t\t\t\t(\\tfrac1{\\sigk*}-\\tfrac{1}{\\sigk})\n\t\t\t\t\t\t\\lk*^{(2)}\\|x_{k+1}-x_k\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{\\Mk^2}{2\\epsk*\\sigk\\alphk}\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{\\epsk*}{2\\sigk\\alphk}\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\\|x_\\star-x_{k+1}\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\\|x_k-x_\\star\\|^2.\n\t\t\t\t\t\\numberthis\\label{eq:3ineqs}\n\t\t\t\t\t\\end{align*}}%\n\t\t\t\t\tSumming \\eqref{eq:3ineqs}\\(+\\betk*\\)\\eqref{eq:ineqsubgrad}, multiplying by \\(\\sigk*\\alphk*\\), and rearranging yields that for every \\(\\betk*\\geq0\\) and \\(\\epsk*>0\\) the following hold.\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t& \\tfrac{1}{2}\n\t\t\t\t\t\\|x_{k+1}-x_\\star\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\sigk*}{\\sigk}\\alphk*(1+\\betk*)p_k%\n\t\t\t\t\t+\n\t\t\t\t\t\\alphk*\\sigk*\\Delk*\\bcost_2(x_{k+1})\n\t\t\t\t\\\\\n\t\t\t\t\t+ {}&\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t\\tfrac{1-\\rhok*\\epsk*}{2}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\alphk*\\sigk*\\Delk*%\n\t\t\t\t\t\t\\lk*^{(2)}\n\t\t\t\t\t\\right)\n\t\t\t\t\t\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\\leq\n\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t\\|x_k-x_\\star\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\sigk*}{\\sigma_{k-1}}\\betk*\\alphk*p_{k-1}%\n\t\t\t\t\\\\\n\t\t\t\t\t-{}&\n\t\t\t\t\t\\rhok*\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t\\betk*(1-\\alphk\\lk)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{\\Mk^2}{2\\epsk*}\n\t\t\t\t\t\\right)\n\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\alphk*\\sigk*\\betk*\\Delk\\bcost_2(x_{k-1}).\n\t\t\t\t\\numberthis\n\t\t\t\t\\end{align*}\n\t\t\t\tBy selecting \\(\\betk=\\rhok\\) and \\(\\epsk\\coloneqq\\nicefrac{1}{2\\rhok}\\) the inequality reduces to the claimed quasi-descent in terms of \\(\\mathcal L_{k}(x^\\star)\\).\n\t\t\t\t\\end{proof}\n\t\t\t\\end{lemma}\n\n\\begin{lemma}\\label{thm:PG}%\n\t\t\t\tSuppose that \\cref{ass:basic} holds, and consider proximal gradient iterations \\eqref{eq:PG} with \\(\\seq{\\alphk}\\) and \\(\\seq{\\sigk}\\) complying with \\cref{cond:alphk,cond:alphk:nu,cond:monotone}.\n\t\t\t\tThen, the following hold:\n\t\t\t\t\\begin{enumerate}\n\t\t\t\t\\item \\label{thm:PG:rhomax}%\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\rhok\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\frac{\\sigk\\alphk}{\\sigma_{k-1}\\alpha_{k-1}}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\max\\set{\n\t\t\t\t\t\t\t\\frac{\\alpha_0}{\\alpha_{-1}},\\,\n\t\t\t\t\t\t\t\\frac{1+\\sqrt5}{2}\n\t\t\t\t\t\t}\n\t\t\t\t\t\\)\n\t\t\t\t\tfor every \\(k\\in\\N\\).\n\n\t\t\t\t\\item \\label{thm:PG:qSD}%\n\t\t\t\t\tFor \\(\\mathcal L_k\\) as in \\cref{thm:LW} it holds that\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\mathcal L_{k+1}(x_\\star)\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\mathcal L_k(x_\\star)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\sigk\\alphk(1+ \\rhok -\\rhok*^2)\n\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\\cost_1(x_{k-1})-\\costinf*\n\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\\)\n\t\t\t\t\tfor all \\(k\\in\\N\\) and \\(x_\\star\\in\\X_1\\).\n\n\t\t\t\t\\item \\label{thm:PG:barPhi}%\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\bphik*(x_{k+1})\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\bphik(x_k)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{1-\\nu}{\\alphk*}\\|x_{k+1}-x_k\\|^2\n\t\t\t\t\t\\)\n\t\t\t\t\tholds for every \\(k\\in\\N\\).\n\t\t\t\t\tIn particular, \\(\\|x_k-x_{k-1}\\|\\to0\\) as \\(k\\to \\infty\\), \\(\\seq{\\bphik(x_k)}\\) is convergent, the following  worst-case rate holds\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\min_{k\\leq K} \\|x_{k+1} - x_k\\|^2 \\leq\n\t\t\t\t\t\t\\tfrac{\\alpha_{\\rm max} \\bar{\\originalvarphi}_0(x_{0})}{(1-\\nu)(1+K)},\n\t\t\t\t\t\\]\n\t\t\t\t\tand both \\(\\seq{\\sigk\\cost_1(x_k)}\\) and \\(\\seq{\\cost_2(x_k)}\\) are bounded.\n\n\t\t\t\t\\item \\label{thm:PG:boundedness}%\n\t\t\t\t\tThe sequence \\(\\seq{x^k}\\) is bounded.\n\t\t\t\t\\end{enumerate}\n\t\t\t\\end{lemma}\n\n\\begin{proof}~\n\t\t\t\t\\begin{proofitemize}\n\t\t\t\t\\item \\ref{thm:PG:rhomax}~\n\t\t\t\t\tObserving that\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\rhok*\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\sqrt{\\tfrac{\\sigk}{\\sigma_{k-1}}\\bigl(1+\\rhok\\bigr)}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\sqrt{1+\\rhok}\n\t\t\t\t\t\\)\n\t\t\t\t\tby \\cref{cond:alphk}, the assertion follows from a trivial induction argument.\n\n\t\t\t\t\\item \\ref{thm:PG:qSD}~\n\t\t\t\t\tSince \\(\\alphk\\leq\\balphk\\), from the definition of \\(\\balphk\\) in \\eqref{eq:adabim:alphk*_0} it follows that the multiplying coefficients in \\eqref{eq:puya:muk:descent} are positive:\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\mathcal L_{k+1}(x_\\star) - \\mathcal L_k(x_\\star)\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\sigk*\\alphk*\n\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\\overbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{\\Delk}{\\rhok*}\n\t\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk}\n\t\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\t\\overbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\t1+\\rhok*\n\t\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\t\\rho_{k+2}^2\\tfrac{\\sigk}{\\sigk*}\n\t\t\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\\bcost_2(x_k)\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\sigk\\alphk\n\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\t1+\\rhok-\\rhok*^2\n\t\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\t\\bigl(\\cost_1(x_{k-1})-\\costinf*\\bigr)\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\\underbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{1}{4}\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\rhok*^2(\\alphk\\lk-\\alphk^2\\Lk^2)\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\alphk\\sigk\\lk^{(2)}\\Delk\n\t\t\t\t\t\t\t}_{\\geq0}\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\\|x_{k-1}-x_k\\|^2.\n\t\t\t\t\t\t\\vphantom{\n\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{1}{4}\n\t\t\t\t\t\t\t}_0\n\t\t\t\t\t\t}\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tThe claim then readily follows from the fact that \\(\\bcost_2\\geq0\\), cf. \\cref{thm:bar>=0}.\n\n\t\t\t\t\\item \\ref{thm:PG:barPhi}~\n\t\t\t\t\tWe have\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\bphik*(x_{k+1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\bphik(x_k)\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\bphik*(x_{k+1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\bphik*(x_k)\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\phik*(x_{k+1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\phik*(x_k)\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\sigk*\\left(\n\t\t\t\t\t\t\t\\Phik*(x_{k+1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Phik*(x_k)\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\\overrel*[\\leq]{\\eqref{eq:phikdescent}}\n\t\t\t\t\t\t- \\tfrac{1-\\nu}{\\alphk*}\n\t\t\t\t\t\t\\|x_{k+1}-x_k\\|^2.\n\t\t\t\t\t\t\\numberthis \\label{eq:descent:bphik}\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tHere, the first inequality uses the fact that \\(\\sigk\\geq\\sigk*\\), and therefore \\(\\bphik=\\sigk\\bcost_1+\\bcost_2\\) is smaller than \\(\\bphik*=\\sigk*\\bcost_1+\\bcost_2\\) on \\(\\dom\\cost_2\\), cf. \\cref{thm:bar>=0}, and \\(x_k\\in\\dom\\gk=\\dom\\cost_1\\cap\\dom\\cost_2\\subseteq\\dom\\cost_2\\);\n\t\t\t\t\tthe second inequality follows from the fact that \\(\\alphk*\\lk*\\leq\\nu\\).\n\t\t\t\t\tThis shows the sought inequality.\n\t\t\t\t\tIn turn, since \\(\\nu<1\\), the sequence \\(\\seq{\\bphik(x_k)}\\) is decreasing; since \\(x_k\\in\\dom\\cost_2\\), it follows from \\cref{thm:bar>=0} that \\(\\bphik(x_k)=\\sigk\\bcost_1(x^k)+\\bcost_2(x^k)\\geq0\\), hence that it is convergent and that the positive-valued sequences \\(\\seq{\\sigk\\bcost_1(x^k)}\\) and \\(\\seq{\\bcost_2(x^k)}\\) are bounded.\n\n\t\t\t\t\tThe convergence rate result follows immediately by telescoping \\eqref{eq:descent:bphik}\n\t\t\t\t\t\\begin{equation}\\label{eq:sublin:res}\n\t\t\t\t\t\t\\tfrac{(1-\\nu)(K+1)}{\\alpha_{\\rm max}}\\min_{k\\leq K}\\|x_{k+1}-x_k\\|^2\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\sum_{k=0}^{K}\\tfrac{1-\\nu}{\\alphk*}\\|x_{k+1}-x_k\\|^2\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\bar{\\originalvarphi}_0(x_{0}),\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\twhere \\cref{cond:alphk} was used to bound \\(\\alphk*\\leq\\alpha_{\\rm max}\\).\n\n\t\t\t\t\\item \\ref{thm:PG:boundedness}~\n\t\t\t\t\tWe pattern the proof structure of \\cite[Thm. 3.2]{solodov2007explicit}, thereby considering two mutually exclusive cases.\n\t\t\t\t\t\\begin{proofitemize}[leftmargin=*,topsep=0pt]\n\t\t\t\t\t\\item\n\t\t\t\t\t\t{\\it Case 1: \\(\\cost_1(x_k)\\geq\\costinf*\\) holds for \\(k\\) large enough.}\n\n\t\t\t\t\t\tRecall the definition of \\(W_k\\) and \\(\\mathcal L_k\\) in \\cref{thm:LW}.\n\t\t\t\t\t\tObserving that \\(W_k\\geq0\\) and \\(\\frac12\\|x_k-x_\\star\\|^2\\leq\\mathcal L_k(x_\\star)\\), \\cref{thm:PG:qSD} implies that \\(\\seq{\\mathcal L_k(x_\\star)}\\) converges and that consequently \\(\\seq{x_k}\\) is bounded.\n\n\t\t\t\t\t\\item\n\t\t\t\t\t\t{\\it Case 2: \\(\\cost_1(x_k)<\\costinf*\\) holds infinitely often.}\n\n\t\t\t\t\t\tIn this case, for every \\(k\\) large enough the index\n\t\t\t\t\t\t\\begin{equation}\\label{eq:ik}\n\t\t\t\t\t\t\ti_k\n\t\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\t\\max\\set{i\\leq k}[\\cost_1(x_i)<\\costinf*]\n\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\tis well defined.\n\t\t\t\t\t\tWe proceed by intermediate claims.\n\t\t\t\t\t\t\\renewcommand{\\claimscounter}{\\ref*{thm:PG}}%\n\t\t\t\t\t\t\\begin{claims}\n\t\t\t\t\t\t\\item\\label{claim:convergence:ik}%\n\t\t\t\t\t\t\t{\\it the sequences \\(\\seq{x_{i_k}}\\) and \\(\\seq{x_{i_k+1}}\\) are bounded}.\n\n\t\t\t\t\t\t\tThe optimal set\n\t\t\t\t\t\t\t\\[\n\t\t\t\t\t\t\t\t\\X_1\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\set{x\\in\\X_2}[\\cost_1(x)\\leq \\costinf*]\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\set{x}[\\cost_1(x)\\leq\\costinf*,\\ \\bcost_2(x)\\leq0]\n\t\t\t\t\t\t\t\\]\n\t\t\t\t\t\t\tcoincides with a sublevel set of the convex function \\(h\\coloneqq\\max\\set{\\cost_1-\\costinf*,\\bcost_2}\\).\n\t\t\t\t\t\t\tSince it is nonempty and bounded by \\cref{ass:X1}, \\(h\\) is level bounded; see, e.g., \\cite[Prop. 11.13]{bauschke2017convex} or \\cite[Lem. 1]{themelis2019acceleration}.\n\t\t\t\t\t\t\tNote that \\cref{thm:PG:barPhi} implies that \\(\\seq{\\bcost_2(x_k)}\\) is (upper) bounded, which combined with the fact that \\(\\cost_1(x_{i_k})<\\costinf*\\) implies that \\(\\seq{x_{i_k}}\\) lies in a sublevel set of \\(h\\), and is therefore bounded.\n\t\t\t\t\t\t\tIn turn, \\cref{thm:PG:barPhi} implies that so is \\(\\seq{x_{i_k+1}}\\).\n\t\t\t\t\t\t\\item\n\t\t\t\t\t\t\t{\\it the whole sequence \\(\\seq{x_k}\\) is bounded}.\n\n\t\t\t\t\t\t\tTo this end, it remains to show that \\(\\seq{x_{k'}}[k'\\in K']\\) is bounded, where\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:K'}\n\t\t\t\t\t\t\t\tK'\\coloneqq\\set{k'\\in\\N}[k'\\geq i_{k'}+2].\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tWith \\(\\rho_{\\rm max}\\) as in \\cref{thm:PG:rhomax}, for every \\(k\\in\\N\\)\n\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\\mathcal L_{i_k+1}(x_\\star)\n\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\tfrac12\n\t\t\t\t\t\t\t\t\t\\|x_{i_k+1}-x_\\star\\|^2\n\t\t\t\t\t\t\t\t\t\\vphantom{\\left(\\tfrac{\\sigma_{i_k+1}}{\\sigma_{i_k}}\\right)}\n\t\t\t\t\t\t\t\t}^{\\text{bounded by \\cref{claim:convergence:ik}}}\n\t\t\t\t\t\t\t\t+~~\n\t\t\t\t\t\t\t\t\\overbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\tfrac{1}{4}\\|x_{i_k+1}-x_{i_k}\\|^2\n\t\t\t\t\t\t\t\t\t\\vphantom{\\left(\\tfrac{\\sigma_{i_k+1}}{\\sigma_{i_k}}\\right)}\n\t\t\t\t\t\t\t\t}^{\\mathclap{\\to0 \\text{ by \\cref{thm:PG:barPhi}}}}\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\sigma_{i_k+1}\\alpha_{i_k+1}(1+\\rho_{i_k+1})\n\t\t\t\t\t\t\t\t\t(\\cost_1(x_{i_k})-\\costinf*)\n\t\t\t\t\t\t\t\t\t\\vphantom{\\left(\\tfrac{\\sigma_{i_k+1}}{\\sigma_{i_k}}\\right)}\n\t\t\t\t\t\t\t\t}^{<0}\n\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t&\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\alpha_{\\rm max}\\rho_{\\rm max}^2\n\t\t\t\t\t\t\t\t\t\\bcost_2(x_{i_k})\n\t\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\t\\alpha_{\\rm max}\\left(1-\\tfrac{\\sigma_{i_k+1}}{\\sigma_{i_k}}\\right)\n\t\t\t\t\t\t\t\t\t\\bcost_2(x_{i_k+1})\n\t\t\t\t\t\t\t\t}_{\\text{bounded by \\cref{thm:PG:barPhi}}}.\n\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\tIn particular, we have\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:supL}\n\t\t\t\t\t\t\t\t\\sup_{k\\in\\N}\\mathcal L_{i_k+1}(x_\\star)<\\infty.\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tLet now \\(k'\\in K'\\).\n\t\t\t\t\t\t\tSince \\(\\cost_1(x_j)-\\costinf*\\geq0\\) for \\(j=i_{k'}+1,\\dots,k'\\), observe that\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:Lk'geq}\n\t\t\t\t\t\t\t\t\\tfrac12\\|x_{k'}-x_\\star\\|^2\n\t\t\t\t\t\t\t\\leq\n\t\t\t\t\t\t\t\t\\mathcal L_{k'}(x_\\star)\n\t\t\t\t\t\t\t\\quad\n\t\t\t\t\t\t\t\t\\forall k'\\in K',\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tand \\cref{thm:PG:qSD} yields that\n\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\\mathcal L_{k'}(x_\\star)\n\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\\mathcal L_{k'-1}(x_\\star)\n\t\t\t\t\t\t\t\\leq\\dots\\leq\n\t\t\t\t\t\t\t\t\\mathcal L_{i_{k'}+2}(x_\\star)\n\t\t\t\t\t\t\t\\numberthis\\label{eq:Lk'descent}\n\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\\mathcal L_{i_{k'}+1}(x_\\star)\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\sigma_{i_{k'}+1}\\alpha_{i_{k'}+1}(1+\\rho_{i_{k'}+1}-\\rho_{i_{k'}+2}^2)\n\t\t\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\t\t\\cost_1(x_{i_{k'}})-\\costinf*\n\t\t\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\\mathcal L_{i_{k'}+1}(x_\\star)\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\tfrac{\\sigma_{i_{k'}+1}}{\\sigma_{i_{k'}}}\n\t\t\t\t\t\t\t\t}_{\\leq1}\n\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\vphantom{\\tfrac{\\sigma_{i_{k'}+1}}{\\sigma_{i_{k'}}}}\n\t\t\t\t\t\t\t\t\t\\alpha_{\\rm max}(1+\\rho_{i_{k'}+1}-\\rho_{i_{k'}+2}^2)\n\t\t\t\t\t\t\t\t}_{\\text{\\clap{bounded by \\cref{thm:PG:rhomax}}}}\n\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\vphantom{\\tfrac{\\sigma_{i_{k'}+1}}{\\sigma_{i_{k'}}}}\n\t\t\t\t\t\t\t\t\t\\sigma_{i_{k'}}\n\t\t\t\t\t\t\t\t\t\\bigl|\n\t\t\t\t\t\t\t\t\t\t\\cost_1(x_{i_{k'}})-\\costinf*\n\t\t\t\t\t\t\t\t\t\\bigr|\n\t\t\t\t\t\t\t\t}_{\\text{\\clap{bounded by \\cref{thm:PG:barPhi}}}}\n\t\t\t\t\t\t\t\t< \\infty\n\t\t\t\t\t\t\t\\qquad\n\t\t\t\t\t\t\t\t\\forall k'\\in K',\n\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\twhere \\eqref{eq:supL} was used in the last inequality.\n\t\t\t\t\t\t\tHere, boundedness of the under-bracketed term follows from boundedness of \\(\\seq{x_{i_{k'}}}[k'\\in K']\\) and lower semicontinuity of \\(\\cost_1\\).\n\t\t\t\t\t\t\tThen, \\eqref{eq:Lk'geq} implies that the sequence \\(\\seq{x_{k'}}[k'\\in K']\\) is bounded.\n\t\t\t\t\t\t\tCombined with \\cref{claim:convergence:ik} and the fact that the index set \\(K'\\) is the complement of the indices therein, the claim follows.\n\t\t\t\t\t\t\\qedhere\n\t\t\t\t\t\t\\end{claims}\n\t\t\t\t\t\\end{proofitemize}\n\t\t\t\t\\end{proofitemize}\n\t\t\t\\end{proof}\n\n\\begin{proof}\n\t\t\t\tWe begin by remarking that boundedness of the sequence \\(\\seq{x_k}\\) is ensured by \\cref{cond:alphk,cond:alphk:nu}, as shown in \\cref{thm:PG:boundedness}.\n\t\t\t\tWe next prove each claim individually.\n\t\t\t\t\\begin{proofitemize}\n\t\t\t\t\\item \\ref{thm:recipe:rate}~\n\t\t\t\t\tConsider a convergent subsequence \\(x_{k_j}\\to x_\\infty\\), so that \\(x_{k_j+1}\\to x_\\infty\\) by \\cref{thm:PG:barPhi}.\n\t\t\t\t\tUp to further extracting if necessary we have that\n\t\t\t\t\t\\(\\alpha_{k_j+1}\\to\\alpha_\\infty\\geq\\alpha_{\\rm min}>0\\) and \\(\\sigma_{k_j}\\to0\\).\n\t\t\t\t\tObserve that\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\tx_{k+1}\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\argmin h({}\\cdot{};x_k,\\alpha_{k+1},\\sigk*),\n\t\t\t\t\t\\shortintertext{where}\n\t\t\t\t\t\th(w;x,\\alpha,\\sigma)\n\t\t\t\t\t\\coloneqq{} &\n\t\t\t\t\t\t(\\sigma \\g_1+\\g_2)(w)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{2\\alpha}\n\t\t\t\t\t\t\\bigl\\|w-x+\\alpha\\nabla\\bigl(\\sigma \\f_1+\\f_2\\bigr)(x)\\bigr\\|^2\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tis level bounded in \\(w\\) locally uniformly in \\((x,\\alpha,\\sigma)\\), as a function from \\(\\R^n\\times\\bigl(\\R^n\\times[\\alpha_{\\rm min},\\infty)\\times[0,\\infty)\\bigr)\\) to \\(\\Rinf\\).\n\t\t\t\t\tSince \\(h\\) is continuous in \\((x,\\alpha,\\sigma)\\), it follows from \\cite[Thm. 1.17]{rockafellar2011variational} that\n\t\t\t\t\t\\[\n\t\t\t\t\t\tx_\\infty\n\t\t\t\t\t\\in\n\t\t\t\t\t\t\\argmin h({}\\cdot{};x_\\infty,\\alpha_\\infty,0)\n\t\t\t\t\t=\n\t\t\t\t\t\t\\prox_{\\alpha_\\infty \\g_2}\\bigl(x_\\infty-\\alpha_\\infty\\nabla \\f_2(x_\\infty)\\bigr),\n\t\t\t\t\t\\]\n\t\t\t\t\tthis condition being equivalent to \\(x_\\infty\\in\\argmin(\\f_2+\\g_2)\\defeq\\X_2\\).\n\t\t\t\t\tWe next show that \\(\\bphik(x_k)\\to0\\).\n\n\t\t\t\t\tFor any \\(x_\\star\\in\\X_1\\), the subdifferential characterization of \\(x_{k+1}\\in\\prox_{\\alphk*\\gk*}(x_k-\\alphk*\\nabla\\fk*(x_k))\\)\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\tfrac{x_{k-1}-x_k}{\\alphk}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t(\\nabla\\fk(x_{k-1})-\\nabla\\fk(x_k))\n\t\t\t\t\t\\in\n\t\t\t\t\t\t\\partial\\phik(x_k)\n\t\t\t\t\t\\]\n\t\t\t\t\timplies that\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\phik(x_\\star)\n\t\t\t\t\t\\geq\n\t\t\t\t\t\t\\phik(x_k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\overbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{x_{k-1}-x_k}{\\alphk}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t(\\nabla\\fk(x_{k-1})-\\nabla\\fk(x_k))\n\t\t\t\t\t\t\t}^{\\to0 \\text{ by \\cref{thm:PG:barPhi}}}\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\\overbracket*[0.5pt]{\n\t\t\t\t\t\t\t\tx_\\star-x_k\n\t\t\t\t\t\t\t\t\\vphantom{\\tfrac{x_{k-1}-x_k}{\\alphk}}\n\t\t\t\t\t\t\t}^{\\text{\\clap{bounded}}}\n\t\t\t\t\t\t},\n\t\t\t\t\t\\]\n\t\t\t\t\thence that\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\costinf_2\n\t\t\t\t\t=\n\t\t\t\t\t\t\\lim_{k\\to\\infty}\\phik(x_\\star)\n\t\t\t\t\t\\geq\n\t\t\t\t\t\t\\limsup_{k\\to\\infty}\\phik(x_k)\n\t\t\t\t\t=\n\t\t\t\t\t\t\\limsup_{k\\to\\infty}\\left(\\sigk\\bcost_1(x_k)+\\overbracket*[0.5pt]{\\cost_2(x_k)}^{\\geq\\costinf_2}\\right)\n\t\t\t\t\t\\geq\n\t\t\t\t\t\t\\limsup_{k\\to\\infty}\\sigk\\bcost_1(x_k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\costinf_2.\n\t\t\t\t\t\\]\n\t\t\t\t\tHere, the second equality is obtained by adding and subtracting \\(\\sigk\\costinf_1\\) along with the fact that \\(\\sigk\\to0\\).\n\t\t\t\t\tSince \\(\\sigk\\bcost_1\\geq0\\), necessarily \\(\\sigk\\bcost_1(x_k)\\to0\\) and consequently \\(\\limsup_{k\\to\\infty}\\cost_2(x_k)=\\costinf_2\\).\n\t\t\t\t\tSimilarly, since \\(\\cost_2\\geq\\costinf_2\\) this necessarily implies that \\(\\cost_2(x_k)\\to\\costinf_2\\).\n\t\t\t\t\tThis shows that \\(\\bcost_2(x^k)\\to0\\) as claimed.\n\n\t\t\t\t\tWe next show that \\(\\seq{W_k}\\) too is vanishing.\n\t\t\t\t\tObserve that\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\tW_k\n\t\t\t\t\t\\defeq{} &\n\t\t\t\t\t\t\\tfrac{1-4\\alphk\\sigk\\lk^{(2)}\\Delk}{4}\n\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\\|x_k-x_{k-1}\\|^2\n\t\t\t\t\t\t}^{\\mathclap{\\to 0\\text{ by \\cref{thm:PG:barPhi}}}}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{\\sigk*}{\\sigk}\\alphk*\\rhok*\n\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\\bcost_2(x_{k-1})\n\t\t\t\t\t\t}^{\\mathclap{\\to 0}}\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\\sigk\\alphk\\Delk\\bcost_2(x_k)\n\t\t\t\t\t\t}_{\\to 0}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\alphk\n\t\t\t\t\t\t\\left(1+\\rhok\\right)\n\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\\sigk\n\t\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\t\\cost_1(x_{k-1})\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\costinf*\n\t\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\t}_{\\to0}\n\t\t\t\t\t\\to\n\t\t\t\t\t\t0\n\t\t\t\t\t\\quad\\text{as \\(k\\to\\infty\\)},\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\twhere we also used the fact that \\(\\seq{\\alphk}\\leq\\alpha_{\\rm max}\\) and \\(\\seq{\\rhok}\\leq\\rho_{\\rm max}\\) as in \\cref{thm:PG:rhomax}.\n\n\t\t\t\t\tThe asserted sublinear rate is a consequence of \\eqref{eq:sublin:res}, the characterization of the residual in \\cref{thm:H} (see \\eqref{eq:Hk:phi}), and the upper bounds for \\(\\lk\\), \\(\\Lk\\) in \\cref{thm:l<=L}:\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\textstyle\\sum_{k=0}^K\\dist^2(0, \\partial \\phik*(x_{k+1})\n\t\t\t\t\t\\leq{}&\n\t\t\t\t\t\t\\textstyle\\sum_{k=0}^K\\tfrac{1}{\\alphk*^2}\\|\\Hk*(x_{k})-\\Hk*(x_{k+1}) \\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\leq {}&\n\t\t\t\t\t\t\\tfrac{M_{\\rm max}^2}{\\alpha_{\\rm min}}\\textstyle\\sum_{k=0}^K \\tfrac{1}{\\alphk*}\\|x_{k}-x_{k+1}\\|^2\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\tfrac{M_{\\rm max}^2}{\\alpha_{\\rm min}(1-\\nu)}\\bar{\\originalvarphi}_0(x_{0}),\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\twhere \\(M_{\\rm max} = \\sup_{k\\in\\N}M_k\\leq 1+ \\alpha_{\\rm max}L_{f_0, \\V}\\), see \\cref{thm:lL}.\n\n\n\t\t\t\t\\item \\ref{thm:recipe:solution}~\n\t\t\t\t\tWe begin by observing that, since \\(\\bcost_2(x^k)\\to0\\), by lower semicontinuity all limit points of \\(\\seq{x^k}\\) belong to \\(\\X_2\\).\n\t\t\t\t\tAs done in the proof of \\cref{thm:PG:boundedness} we consider two possible cases.\n\t\t\t\t\t\\begin{proofitemize}[leftmargin=*,topsep=0pt]\n\t\t\t\t\t\\item\n\t\t\t\t\t\t{\\it Case 1: \\(\\cost_1(x_k)\\geq\\costinf*\\) holds for \\(k\\) large enough.}\n\n\t\t\t\t\t\tIn this case, we will actually show that the sequence \\(\\seq{x_k}\\) converges to a solution of \\eqref{eq:P}.\n\t\t\t\t\t\tWe first argue that there exists an optimal limit point; to this end, since all limit points are feasible and because of lower semicontinuity, it suffices to show that \\(\\liminf_{k\\to\\infty}\\cost_1(x_k)=\\costinf*\\).\n\t\t\t\t\t\tA telescoping argument on \\cref{thm:PG:qSD} along with \\cref{cond:alphamin} yields\n\t\t\t\t\t\t\\begin{equation}\n\t\t\t\t\t\t\t\\alpha_{\\rm min}\\sum_{k\\in \\N}\\sigk(1+\\rhok-\\rhok*^2)\\bigl(\\cost_1(x_{k-1})-\\costinf*\\bigr)\n\t\t\t\t\t\t\\leq\n\t\t\t\t\t\t\t\\sum_{k\\in \\N}\\sigk\\alphk(1+\\rhok-\\rhok*^2)\\bigl(\\cost_1(x_{k-1})-\\costinf*\\bigr)\n\t\t\t\t\t\t<\n\t\t\t\t\t\t\t\\infty.\n\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\tSince \\(\\sum_{k\\in\\N}\\sigk=\\infty\\), necessarily\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\liminf_{k\\to\\infty}(1+\\rhok-\\rhok*^2)(\\cost_1(x_{k-1})-\\costinf*)\n\t\t\t\t\t\t=\n\t\t\t\t\t\t\t0.\n\t\t\t\t\t\t\t\\numberthis\\label{eq:liminf:cost1}\n\t\t\t\t\t\t\\end{align*}\n\t\t\t% \t\t\tObserve first that\n\t\t\t% \t\t\t\\begin{align*}\n\t\t\t% \t\t\t\t\\rhok*\n\t\t\t% \t\t\t=\n\t\t\t% \t\t\t\t\\tfrac{\\sigk*\\alphk*}{\\sigk\\alphk}\n\t\t\t% \t\t\t\\leq\n\t\t\t% \t\t\t\t\\tfrac{\\sigk*\\balphk*}{\\sigk\\alphk}\n\t\t\t% \t\t\t\\leq\n\t\t\t% \t\t\t\t\\tfrac{\\sigk*}{\\sigk}\n\t\t\t% \t\t\t\t\\sqrt{\\tfrac{\\sigk}{\\sigma_{k-1}}\\bigl(1+\\rhok\\bigr)}\n\t\t\t% \t\t\t\t\\tfrac{\\sigk}{\\sigk*}\n\t\t\t% \t\t\t\\leq\n\t\t\t% \t\t\t\t\\sqrt{1+\\rhok}\n\t\t\t% \t\t\t,\n\t\t\t% \t\t\t\\numberthis\\label{eq:rhok:plus}\n\t\t\t% \t\t\t\\end{align*}\n\t\t\t% \t\t\twhere the first inequality holds by \\cref{cond:alphk}, the second is due to the first term in \\eqref{eq:adabim:alphk*_0}, and the last inequality follows from \\eqref{eq:sigk}.\n\t\t\t\t\t\tNotice that \\(\\limsup_{k\\to\\infty}(1+\\rhok-\\rhok*^2)>0\\), for otherwise \\(1+\\rhok-\\rhok*^2 \\to 0\\), implying that \\(\\liminf_{k \\to \\infty} \\rhok >0\\), and consequently that \\(\\alphk\\sigk\\) eventually increases exponentially, which contradicts \\(\\sigk\\alphk \\leq \\sigk\\alpha_{\\rm max} \\to 0\\).\n\t\t\t\t\t\tHence, \\(\\limsup_{k\\to\\infty}(1+\\rhok-\\rhok*^2)>0\\), which along with \\eqref{eq:liminf:cost1} implies that \\(\\liminf_{k\\to\\infty}\\cost_1(x_{k-1})=\\costinf*\\).\n\t\t\t\t\t\tTherefore, an optimal limit point exists, be it \\(x_\\infty\\).\n\t\t\t\t\t\tSince \\(W_k\\to0\\) by assertion \\ref{thm:recipe:rate} and since \\(\\mathcal L_k(x_\\infty)\\) converges, it follows that \\(\\frac12\\|x_k-x_\\infty\\|^2 = \\mathcal L_k(x_\\infty) - W_k\\) converges as well.\n\t\t\t\t\t\tSince along a subsequence \\(\\frac12\\|x_k-x_\\infty\\|^2\\) converges to zero, necessarily \\(\\lim_{k\\to\\infty}\\frac12\\|x_k-x_\\infty\\|^2=0\\), proving that the entire sequence \\(\\seq{x_k}\\) converges to \\(x_\\infty\\).\n\t\t\t\t\t\\item\n\t\t\t\t\t\t{\\it Case 2: \\(\\cost_1(x_k)<\\costinf*\\) holds infinitely often.}\n\n\t\t\t\t\t\tRecall the index \\(i_k\\) and the set \\(K'\\) defined in \\eqref{eq:ik} and \\eqref{eq:K'}.\n\t\t\t% \t\t\tOnce we establish that all the limit points of \\(\\seq{x_{i_k}}\\) and \\(\\seq{x_{i_k+1}}\\) are optimal, {\\red it only remains to show that\n\t\t\t% \t\t\t\\(\n\t\t\t% \t\t\t\t\\dist(x_{k'},\\X_1)\n\t\t\t% \t\t\t\\to\n\t\t\t% \t\t\t\t0\n\t\t\t% \t\t\t\\)\n\t\t\t% \t\t\tas \\(K'\\ni k'\\to\\infty\\)\\note{Why}}.\n\t\t\t\t\t\tHaving established boundedness of the entire sequence, limit points of \\(\\seq{x_{i_k}}\\) exist and, as shown above, belong to \\(\\X_2\\).\n\t\t\t\t\t\tMoreover, it follows from \\eqref{eq:ik} that \\(\\limsup_{k\\to\\infty}\\cost_1(x_{i_k})\\leq\\costinf*\\), and a lower semicontinuity argument then yields that all the limit points of \\(\\seq{x_{i_k}}\\) attain the optimal cost and are therefore optimal.\n\t\t\t\t\t\tSince \\(x_{i_k}-x_{i_k+1}\\to0\\) by \\cref{thm:PG:barPhi}, the same is also true for \\(\\seq{x_{i_k+1}}\\), and in particular\n\t\t\t\t\t\t\\(\\dist(x_{i_{k}+1},\\X_1) \\to 0\\) as \\(k \\to \\infty\\).\n\n\t\t\t\t\t\tFor each \\(k\\) let \\(\\bar x_k\\coloneqq\\proj_{\\X_1}\\!x_k\\), which is well defined since \\(\\X_1\\neq\\emptyset\\) is closed and convex.\n\t\t\t\t\t\tRecalling that \\(\\mathcal L_k(x_\\star)=W_k + \\tfrac12\\|x_k-x_\\star\\|^2\\), for \\(k'\\in K'\\) we have\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\tfrac12\\dist(x_{k'},\\X_1)^2\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\tfrac12\\|x_{k'}-\\bar x_{i_{k'}+1}\\|^2\n\t\t\t\t\t\t={}\n\t\t\t\t\t\t\t\\mathcal L_{k'}(\\bar x_{i_{k'}+1})-W_{k'}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\\dueto{\\scriptsize by \\eqref{eq:Lk'descent}}\n\t\t\t\t\t\t\t\\\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\mathcal L_{i_{k'}+2}(\\bar x_{i_{k'}+1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\tW_{k'}\n\t\t\t\t\t\t={}\n\t\t\t\t\t\t\t% \\tfrac12\\dist(x_{i_{k'}+2},\\X_1)^2\n\t\t\t\t\t\t\t\\tfrac12\\|x_{i_{k'}+2} - \\bar x_{i_{k'}+1}\\|^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\tW_{i_{k'}+2}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\tW_{k'}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\tfrac12\\big(\\dist(x_{i_{k'}+1},\\X_1) + \\|x_{i_{k'}+1} - x_{i_{k'}+2}\\|\\big)^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\tW_{i_{k'}+2}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\tW_{k'}\n\t\t\t\t\t\t\\to\n\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\tas \\(K'\\ni k'\\to\\infty\\), where the limit follows from \\cref{thm:PG:barPhi}, the vanishing of \\(\\seq{W_k}\\) established in assertion \\ref{thm:recipe:rate}, and the fact that \\(i_k\\to\\infty\\) as \\(k\\to\\infty\\).\n\t\t\t\t\t\tSince \\(K'\\cup\\set{i_k,i_k+1}[k\\in\\N]=\\N\\), we conclude that all the limit points of \\(\\seq{x^k}\\) are optimal.\n\t\t\t\t\t\\qedhere\n\t\t\t\t\t\\end{proofitemize}\n\t\t\t\t\\end{proofitemize}\n\t\t\t\\end{proof}","units":[{"start":72279,"end":83240,"start_line":1441,"end_line":1899,"raw_tex":"\\begin{lemma}[quasi-descent inequality]\\label{thm:LW}%\n\t\t\t\tSuppose that \\cref{ass:basic} holds and consider iterations \\eqref{eq:PG} with \\(0<\\sigk*\\leq\\sigk\\).\n\t\t\t\tLet \\(\\bcost_2\\coloneqq\\cost_2-\\costinf_2\\geq0\\) be as in \\cref{rem:cost}, and define \\(\\Delk\\coloneqq\\frac{1}{\\sigk}-\\frac{1}{\\sigma_{k-1}}\\), \\(\\rhok*\\coloneqq\\frac{\\sigk*\\alphk*}{\\sigk\\alphk}\\), and\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\mathcal L_k(x_\\star)\n\t\t\t\t\\coloneqq{} &\n\t\t\t\t\t\\tfrac12\\|x_k-x_\\star\\|^2\n\t\t\t\t\t+\n\t\t\t\t\tW_k\n\t\t\t\t\\shortintertext{with}\n\t\t\t\t\tW_k\n\t\t\t\t\\coloneqq{} &\n\t\t\t\t\t\\tfrac{1-4\\alphk\\sigk\\lk^{(2)}\\Delk}{4}\\|x_k-x_{k-1}\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\sigk*}{\\sigk}\\alphk*\\rhok*\n\t\t\t\t\t\\bcost_2(x_{k-1})\n\t\t\t\t\\\\\n\t\t\t\t&\n\t\t\t\t\t+\n\t\t\t\t\t\\sigk\\alphk\\Delk\\bcost_2(x_k)\n\t\t\t\t\t+\n\t\t\t\t\t\\sigk\\alphk\\left(1+\\rhok\\right)\n\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\\cost_1(x_{k-1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\costinf*\n\t\t\t\t\t\\bigr).\n\t\t\t\t\\end{align*}\n\t\t\t\tThen, for every \\(k\\in\\N\\) and \\(x_\\star\\in\\X_1\\) it holds that\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\mathcal L_{k+1}(x_\\star)\n\t\t\t\t\t-\n\t\t\t\t\t\\mathcal L_k(x_\\star)\n\t\t\t\t\\leq{} &\n\t\t\t\t\t-\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t\\tfrac{1}{4}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\rhok*^2(\\alphk\\lk-\\alphk^2\\Lk^2)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\sigk\\alphk\\lk^{(2)}\\Delk\n\t\t\t\t\t\\right)\n\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\\\\\n\t\t\t\t&\n\t\t\t\t\t-\n\t\t\t\t\t\\sigk\\alphk\n\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t1+\\rhok-\\rhok*^2\n\t\t\t\t\t\\bigr)\n\t\t\t\t\t\\bigl(\\cost_1(x_{k-1})-\\costinf*\\bigr)\n\t\t\t\t\\\\\n\t\t\t\t&\n\t\t\t\t\t-\n\t\t\t\t\t\\sigk*\\alphk*\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t\\tfrac{\\Delk}{\\rhok*}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{\\sigk}\\left(\n\t\t\t\t\t\t\t1+\\rhok*\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\rho_{k+2}^2\\tfrac{\\sigk}{\\sigk*}\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\\right)\n\t\t\t\t\t\\bcost_2(x_k).\n\t\t\t\t\\numberthis\\label{eq:puya:muk:descent}\n\t\t\t\t\\end{align*}\n\t\t\t\t\\begin{proof}\n\t\t\t\tLet\n\t\t\t\t\\begin{equation}\n\t\t\t\t\tp_k%\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\phik(x_k)-\\phik(x_\\star)\n\t\t\t\t=\n\t\t\t\t\t\\sigk\\bigl(\\cost_1(x_k)-\\costinf*\\bigr)\n\t\t\t\t\t+\n\t\t\t\t\t\\bcost_2(x_k)\n\t\t\t\t\\end{equation}\n\t\t\t\t(which is not necessarily positive).\n\t\t\t\tWe will prove the claim using the uppercase notation of \\eqref{eq:uppercase} and, consistently, set\n\t\t\t\t\t\\(P_k%\n\t\t\t\t\t\\coloneqq{\\tfrac{1}{\\sigk}}p_k%\n\t\t\t\t\t=\\Phik(x_k)-\\Phik(x_\\star)\n\t\t\t\t\t\\).\n\t\t\t\t\tBeing a simple matter of multiplicative constants, note that proximal gradient iterations \\eqref{eq:PG} can equivalently be expressed in the penalized cost reformulation \\(\\Phik*=\\Fk*+\\Gk*\\) up to suitable scaling of the stepsize, namely,\n\t\t\t\t\t\\[\n\t\t\t\t\t\tx_{k+1}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\prox_{\\sigk*\\alphk*\\Gk*}(x_k-\\sigk*\\alphk*\\nabla\\Fk*(x_k)).\n\t\t\t\t\t\\]\n\t\t\t\t\tThe subgradient characterization of the proximal mapping as in \\eqref{eq:Hk:phi} yields\n\t\t\t\t\t\\begin{equation}\\label{eq:subgrad}\n\t\t\t\t\t\t\\tfrac{\\Hk(x_{k-1})-x_k}{\\sigk\\alphk}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\tfrac{x_{k-1}-x_k}{\\sigk\\alphk}-\\nabla\\Fk(x_{k-1})\n\t\t\t\t\t\\in\n\t\t\t\t\t\t\\partial\\Gk(x_k),\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\twhere we remind that \\(\\Hk=\\id-\\alphk\\nabla\\fk\\), cf. \\eqref{eq:Hk:phi}.\n\t\t\t\t\tHence, since \\(\\partial\\Phik=\\nabla\\Fk+\\partial\\Gk\\),\n\t\t\t\t\t\\begin{subequations}\n\t\t\t\t\t\t\\begin{align}\n\t\t\t\t\t\t\\nonumber\n\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\Phik(x_{k-1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Phik(x_k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\\innprod{x_{k-1}-x_k}{x_{k-1}-x_k}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk(x_{k-1})-\\nabla\\Fk(x_k)}{x_{k-1}-x_k}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\nonumber\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\Phik(x_{k-1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Phik(x_k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac1{\\sigk}\\innprod{\\nabla\\fk(x_{k-1})-\\nabla\\fk(x_k)}{x_{k-1}-x_k}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\Phik(x_{k-1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Phik(x_k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1-\\alphk\\lk}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t\t\\label{eq:phikdescent}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\tP_{k-1}%\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\tP_k%\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t(\\tfrac{1}{\\sigk}-\\tfrac{1}{\\sigma_{k-1}})\\bcost_2(x_{k-1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1-\\alphk\\lk}{\\sigk\\alphk}\\|x_{k-1}-x_k\\|^2.\n\t\t\t\t\t\t\\label{eq:ineqsubgrad}\n\t\t\t\t\t\t\\end{align}\n\t\t\t\t\t\\end{subequations}\n\t\t\t\t\tAgain from \\eqref{eq:subgrad}, this time with \\(k\\gets k+1\\), we have\n\t\t\t\t\t\\begin{subequations}\\label{subeq:3ineq}\n\t\t\t\t\t\t{\\mathtight[0.6]%\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\Gk*(x_\\star)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Gk*(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_\\star-x_{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\innprod{x_k-x_{k+1}}{x_\\star-x_{k+1}}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\Gk*(x_\\star)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Gk*(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_\\star-x_{k+1}}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_\\star\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_\\star-x_{k+1}\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t&\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\Gk*(x_\\star)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Gk*(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_\\star-x_k}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_\\star\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t&\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_\\star-x_{k+1}\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\Gk*(x_\\star)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Gk*(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\fillwidthof[c]{\n\t\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_\\star-x_k}\n\t\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\t\\Fk*(x_\\star)-\\Fk*(x_k)\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\underbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t}_{\\text{(A)}}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_\\star\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t&\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_\\star-x_{k+1}\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_{k+1}\\|^2,\n\t\t\t\t\t\t\\numberthis\\label{eq:ineq1}\n\t\t\t\t\t\t\\end{align*}}%\n\t\t\t\t\t\twhere the last inequality uses convexity of \\(\\Fk*\\).\n\t\t\t\t\t\tAs to term (A), we have\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\text{(A)}\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\\innprod{\\Hk(x_{k-1})-x_k}{x_{k+1}-x_k}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\\innprod{\\Hk(x_{k-1})-x_k+\\sigk\\alphk\\nabla\\Fk*(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:subgrad}}{} &\n\t\t\t\t\t\t\t\\fillwidthof[c]{\n\t\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\t\\innprod{\\Hk(x_{k-1})-x_k}{x_{k+1}-x_k}\n\t\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\t\\Gk(x_{k+1})-\\Gk(x_k)\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)-\\nabla\\Fk(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t&\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\t\\innprod{\\Hk(x_{k-1})-\\Hk(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t}_{\\text{(B)}}.\n\t\t\t\t\t\t\\numberthis\\label{eq:ineqA}\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\tNext, we bound the term (B) by \\(\\epsk*\\)-Young's inequality as\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\text{(B)}\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\tfrac{\\epsk*}{2\\sigk\\alphk}\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\epsk*\\sigk\\alphk}\\|\\Hk(x_{k-1})-\\Hk(x_k)\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\overrel{\\ref{thm:H}}{} &\n\t\t\t\t\t\t\t\\tfrac{\\epsk*}{2\\sigk\\alphk}\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{\\Mk^2}{2\\epsk*\\sigk\\alphk}\\|x_{k-1}-x_k\\|^2.\n\t\t\t\t\t\t\\numberthis\\label{eq:ineqB}\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\\end{subequations}\n\t\t\t\t\tCombining the three inequalities \\eqref{subeq:3ineq} yields\n\t\t\t\t\t{\\mathtight[0.9]%\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t0\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\Phik*(x_\\star)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\Gk*(x_{k+1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\Fk*(x_k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\Gk(x_{k+1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\Gk(x_k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t(\\tfrac1{\\sigk*}-\\tfrac{1}{\\sigk})\n\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\Biggl\\{\n\t\t\t\t\t\t\t\\tfrac{\\Mk^2}{2\\epsk*\\sigk\\alphk}\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\tfrac{\\epsk*}{2\\sigk\\alphk}\n\t\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\t\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_\\star-x_{k+1}\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t\\hphantom{+\\Biggl\\{{}}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_\\star\\|^2\n\t\t\t\t\t\t\\Biggr\\}\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\overrel{\\ref{thm:Phi*}}{} &\n\t\t\t\t\t\t\\costinf*\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac1{\\sigk*}\\costinf_2\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\Fk(x_k)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\Gk(x_k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\bigl(\\tfrac1{\\sigk*}-\\tfrac{1}{\\sigk}\\bigr)\n\t\t\t\t\t\t\\vphantom{\\f^2}\n\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\f_2(x_k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\g_2(x_{k+1})\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\{\\cdots\\}\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t-P_k%\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\t\\tfrac1{\\sigk*}-\\tfrac{1}{\\sigk}\n\t\t\t\t\t\t\t\t\\vphantom{\\f^2}\n\t\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\\,\n\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\f_2(x_k)\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\g_2(x_{k+1})\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\costinf_2\n\t\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t}^{\\text{(D)}}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\Bigl\\{\\cdots\\Bigr\\}.\n\t\t\t\t\t\\end{align*}}%\n\t\t\t\t\tBy using convexity of \\(\\f_2\\) we can bound the term (D) as\n\t\t\t\t\t{\\mathtight[0.75]%\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\text{(D)}\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_{k+1})}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\f_2(x_k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\g_2(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_{k+1})-\\nabla \\f_2(x_k)}{x_{k+1}-x_k}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\costinf_2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\fillwidthof[c]{\n\t\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_{k+1})}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\f_2(x_k)\n\t\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\t-\\f_2(x_{k+1})\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\g_2(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\fillwidthof[c]{\n\t\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_{k+1})-\\nabla \\f_2(x_k)}{x_{k+1}-x_k}\n\t\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\t\\lk*^{(2)}\\|x_{k+1}-x_k\\|^2\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\costinf_2,\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t}%\n\t\t\t\t\twhich plugged in the previous inequality results in\n\t\t\t\t\t{\\mathtight[0.75]%\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t0\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t-P_k%\n\t\t\t\t\t\t-\n\t\t\t\t\t\t(\\tfrac1{\\sigk*}-\\tfrac{1}{\\sigk})\\bcost_2(x_{k+1})\n\t\t\t\t\t\t+\n\t\t\t\t\t\t(\\tfrac1{\\sigk*}-\\tfrac{1}{\\sigk})\n\t\t\t\t\t\t\\lk*^{(2)}\\|x_{k+1}-x_k\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{\\Mk^2}{2\\epsk*\\sigk\\alphk}\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{\\epsk*}{2\\sigk\\alphk}\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\\|x_\\star-x_{k+1}\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\\|x_k-x_\\star\\|^2.\n\t\t\t\t\t\\numberthis\\label{eq:3ineqs}\n\t\t\t\t\t\\end{align*}}%\n\t\t\t\t\tSumming \\eqref{eq:3ineqs}\\(+\\betk*\\)\\eqref{eq:ineqsubgrad}, multiplying by \\(\\sigk*\\alphk*\\), and rearranging yields that for every \\(\\betk*\\geq0\\) and \\(\\epsk*>0\\) the following hold.\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t& \\tfrac{1}{2}\n\t\t\t\t\t\\|x_{k+1}-x_\\star\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\sigk*}{\\sigk}\\alphk*(1+\\betk*)p_k%\n\t\t\t\t\t+\n\t\t\t\t\t\\alphk*\\sigk*\\Delk*\\bcost_2(x_{k+1})\n\t\t\t\t\\\\\n\t\t\t\t\t+ {}&\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t\\tfrac{1-\\rhok*\\epsk*}{2}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\alphk*\\sigk*\\Delk*%\n\t\t\t\t\t\t\\lk*^{(2)}\n\t\t\t\t\t\\right)\n\t\t\t\t\t\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\\leq\n\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t\\|x_k-x_\\star\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\sigk*}{\\sigma_{k-1}}\\betk*\\alphk*p_{k-1}%\n\t\t\t\t\\\\\n\t\t\t\t\t-{}&\n\t\t\t\t\t\\rhok*\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t\\betk*(1-\\alphk\\lk)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{\\Mk^2}{2\\epsk*}\n\t\t\t\t\t\\right)\n\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\alphk*\\sigk*\\betk*\\Delk\\bcost_2(x_{k-1}).\n\t\t\t\t\\numberthis\n\t\t\t\t\\end{align*}\n\t\t\t\tBy selecting \\(\\betk=\\rhok\\) and \\(\\epsk\\coloneqq\\nicefrac{1}{2\\rhok}\\) the inequality reduces to the claimed quasi-descent in terms of \\(\\mathcal L_{k}(x^\\star)\\).\n\t\t\t\t\\end{proof}\n\t\t\t\\end{lemma}","sha256_utf8":"b82e8a9d66a1a3feabe466f6a136300ce01a6a3d97cb691d212b58331a92937a"},{"start":86462,"end":88028,"start_line":1951,"end_line":2003,"raw_tex":"\\begin{lemma}\\label{thm:PG}%\n\t\t\t\tSuppose that \\cref{ass:basic} holds, and consider proximal gradient iterations \\eqref{eq:PG} with \\(\\seq{\\alphk}\\) and \\(\\seq{\\sigk}\\) complying with \\cref{cond:alphk,cond:alphk:nu,cond:monotone}.\n\t\t\t\tThen, the following hold:\n\t\t\t\t\\begin{enumerate}\n\t\t\t\t\\item \\label{thm:PG:rhomax}%\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\rhok\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\frac{\\sigk\\alphk}{\\sigma_{k-1}\\alpha_{k-1}}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\max\\set{\n\t\t\t\t\t\t\t\\frac{\\alpha_0}{\\alpha_{-1}},\\,\n\t\t\t\t\t\t\t\\frac{1+\\sqrt5}{2}\n\t\t\t\t\t\t}\n\t\t\t\t\t\\)\n\t\t\t\t\tfor every \\(k\\in\\N\\).\n\n\t\t\t\t\\item \\label{thm:PG:qSD}%\n\t\t\t\t\tFor \\(\\mathcal L_k\\) as in \\cref{thm:LW} it holds that\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\mathcal L_{k+1}(x_\\star)\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\mathcal L_k(x_\\star)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\sigk\\alphk(1+ \\rhok -\\rhok*^2)\n\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\\cost_1(x_{k-1})-\\costinf*\n\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\\)\n\t\t\t\t\tfor all \\(k\\in\\N\\) and \\(x_\\star\\in\\X_1\\).\n\n\t\t\t\t\\item \\label{thm:PG:barPhi}%\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\bphik*(x_{k+1})\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\bphik(x_k)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{1-\\nu}{\\alphk*}\\|x_{k+1}-x_k\\|^2\n\t\t\t\t\t\\)\n\t\t\t\t\tholds for every \\(k\\in\\N\\).\n\t\t\t\t\tIn particular, \\(\\|x_k-x_{k-1}\\|\\to0\\) as \\(k\\to \\infty\\), \\(\\seq{\\bphik(x_k)}\\) is convergent, the following  worst-case rate holds\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\min_{k\\leq K} \\|x_{k+1} - x_k\\|^2 \\leq\n\t\t\t\t\t\t\\tfrac{\\alpha_{\\rm max} \\bar{\\originalvarphi}_0(x_{0})}{(1-\\nu)(1+K)},\n\t\t\t\t\t\\]\n\t\t\t\t\tand both \\(\\seq{\\sigk\\cost_1(x_k)}\\) and \\(\\seq{\\cost_2(x_k)}\\) are bounded.\n\n\t\t\t\t\\item \\label{thm:PG:boundedness}%\n\t\t\t\t\tThe sequence \\(\\seq{x^k}\\) is bounded.\n\t\t\t\t\\end{enumerate}\n\t\t\t\\end{lemma}","sha256_utf8":"cb8e5607cd3bdddc36f57f93af97a042e427f7081807ce51ef94f122e3bfa25e"},{"start":88032,"end":96787,"start_line":2004,"end_line":2246,"raw_tex":"\\begin{proof}~\n\t\t\t\t\\begin{proofitemize}\n\t\t\t\t\\item \\ref{thm:PG:rhomax}~\n\t\t\t\t\tObserving that\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\rhok*\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\sqrt{\\tfrac{\\sigk}{\\sigma_{k-1}}\\bigl(1+\\rhok\\bigr)}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\sqrt{1+\\rhok}\n\t\t\t\t\t\\)\n\t\t\t\t\tby \\cref{cond:alphk}, the assertion follows from a trivial induction argument.\n\n\t\t\t\t\\item \\ref{thm:PG:qSD}~\n\t\t\t\t\tSince \\(\\alphk\\leq\\balphk\\), from the definition of \\(\\balphk\\) in \\eqref{eq:adabim:alphk*_0} it follows that the multiplying coefficients in \\eqref{eq:puya:muk:descent} are positive:\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\mathcal L_{k+1}(x_\\star) - \\mathcal L_k(x_\\star)\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\sigk*\\alphk*\n\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\\overbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{\\Delk}{\\rhok*}\n\t\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk}\n\t\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\t\\overbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\t1+\\rhok*\n\t\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\t\\rho_{k+2}^2\\tfrac{\\sigk}{\\sigk*}\n\t\t\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\\bcost_2(x_k)\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\sigk\\alphk\n\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\t1+\\rhok-\\rhok*^2\n\t\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\t\\bigl(\\cost_1(x_{k-1})-\\costinf*\\bigr)\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\\underbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{1}{4}\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\rhok*^2(\\alphk\\lk-\\alphk^2\\Lk^2)\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\alphk\\sigk\\lk^{(2)}\\Delk\n\t\t\t\t\t\t\t}_{\\geq0}\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\\|x_{k-1}-x_k\\|^2.\n\t\t\t\t\t\t\\vphantom{\n\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{1}{4}\n\t\t\t\t\t\t\t}_0\n\t\t\t\t\t\t}\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tThe claim then readily follows from the fact that \\(\\bcost_2\\geq0\\), cf. \\cref{thm:bar>=0}.\n\n\t\t\t\t\\item \\ref{thm:PG:barPhi}~\n\t\t\t\t\tWe have\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\bphik*(x_{k+1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\bphik(x_k)\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\bphik*(x_{k+1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\bphik*(x_k)\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\phik*(x_{k+1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\phik*(x_k)\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\sigk*\\left(\n\t\t\t\t\t\t\t\\Phik*(x_{k+1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Phik*(x_k)\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\\overrel*[\\leq]{\\eqref{eq:phikdescent}}\n\t\t\t\t\t\t- \\tfrac{1-\\nu}{\\alphk*}\n\t\t\t\t\t\t\\|x_{k+1}-x_k\\|^2.\n\t\t\t\t\t\t\\numberthis \\label{eq:descent:bphik}\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tHere, the first inequality uses the fact that \\(\\sigk\\geq\\sigk*\\), and therefore \\(\\bphik=\\sigk\\bcost_1+\\bcost_2\\) is smaller than \\(\\bphik*=\\sigk*\\bcost_1+\\bcost_2\\) on \\(\\dom\\cost_2\\), cf. \\cref{thm:bar>=0}, and \\(x_k\\in\\dom\\gk=\\dom\\cost_1\\cap\\dom\\cost_2\\subseteq\\dom\\cost_2\\);\n\t\t\t\t\tthe second inequality follows from the fact that \\(\\alphk*\\lk*\\leq\\nu\\).\n\t\t\t\t\tThis shows the sought inequality.\n\t\t\t\t\tIn turn, since \\(\\nu<1\\), the sequence \\(\\seq{\\bphik(x_k)}\\) is decreasing; since \\(x_k\\in\\dom\\cost_2\\), it follows from \\cref{thm:bar>=0} that \\(\\bphik(x_k)=\\sigk\\bcost_1(x^k)+\\bcost_2(x^k)\\geq0\\), hence that it is convergent and that the positive-valued sequences \\(\\seq{\\sigk\\bcost_1(x^k)}\\) and \\(\\seq{\\bcost_2(x^k)}\\) are bounded.\n\n\t\t\t\t\tThe convergence rate result follows immediately by telescoping \\eqref{eq:descent:bphik}\n\t\t\t\t\t\\begin{equation}\\label{eq:sublin:res}\n\t\t\t\t\t\t\\tfrac{(1-\\nu)(K+1)}{\\alpha_{\\rm max}}\\min_{k\\leq K}\\|x_{k+1}-x_k\\|^2\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\sum_{k=0}^{K}\\tfrac{1-\\nu}{\\alphk*}\\|x_{k+1}-x_k\\|^2\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\bar{\\originalvarphi}_0(x_{0}),\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\twhere \\cref{cond:alphk} was used to bound \\(\\alphk*\\leq\\alpha_{\\rm max}\\).\n\n\t\t\t\t\\item \\ref{thm:PG:boundedness}~\n\t\t\t\t\tWe pattern the proof structure of \\cite[Thm. 3.2]{solodov2007explicit}, thereby considering two mutually exclusive cases.\n\t\t\t\t\t\\begin{proofitemize}[leftmargin=*,topsep=0pt]\n\t\t\t\t\t\\item\n\t\t\t\t\t\t{\\it Case 1: \\(\\cost_1(x_k)\\geq\\costinf*\\) holds for \\(k\\) large enough.}\n\n\t\t\t\t\t\tRecall the definition of \\(W_k\\) and \\(\\mathcal L_k\\) in \\cref{thm:LW}.\n\t\t\t\t\t\tObserving that \\(W_k\\geq0\\) and \\(\\frac12\\|x_k-x_\\star\\|^2\\leq\\mathcal L_k(x_\\star)\\), \\cref{thm:PG:qSD} implies that \\(\\seq{\\mathcal L_k(x_\\star)}\\) converges and that consequently \\(\\seq{x_k}\\) is bounded.\n\n\t\t\t\t\t\\item\n\t\t\t\t\t\t{\\it Case 2: \\(\\cost_1(x_k)<\\costinf*\\) holds infinitely often.}\n\n\t\t\t\t\t\tIn this case, for every \\(k\\) large enough the index\n\t\t\t\t\t\t\\begin{equation}\\label{eq:ik}\n\t\t\t\t\t\t\ti_k\n\t\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\t\\max\\set{i\\leq k}[\\cost_1(x_i)<\\costinf*]\n\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\tis well defined.\n\t\t\t\t\t\tWe proceed by intermediate claims.\n\t\t\t\t\t\t\\renewcommand{\\claimscounter}{\\ref*{thm:PG}}%\n\t\t\t\t\t\t\\begin{claims}\n\t\t\t\t\t\t\\item\\label{claim:convergence:ik}%\n\t\t\t\t\t\t\t{\\it the sequences \\(\\seq{x_{i_k}}\\) and \\(\\seq{x_{i_k+1}}\\) are bounded}.\n\n\t\t\t\t\t\t\tThe optimal set\n\t\t\t\t\t\t\t\\[\n\t\t\t\t\t\t\t\t\\X_1\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\set{x\\in\\X_2}[\\cost_1(x)\\leq \\costinf*]\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\set{x}[\\cost_1(x)\\leq\\costinf*,\\ \\bcost_2(x)\\leq0]\n\t\t\t\t\t\t\t\\]\n\t\t\t\t\t\t\tcoincides with a sublevel set of the convex function \\(h\\coloneqq\\max\\set{\\cost_1-\\costinf*,\\bcost_2}\\).\n\t\t\t\t\t\t\tSince it is nonempty and bounded by \\cref{ass:X1}, \\(h\\) is level bounded; see, e.g., \\cite[Prop. 11.13]{bauschke2017convex} or \\cite[Lem. 1]{themelis2019acceleration}.\n\t\t\t\t\t\t\tNote that \\cref{thm:PG:barPhi} implies that \\(\\seq{\\bcost_2(x_k)}\\) is (upper) bounded, which combined with the fact that \\(\\cost_1(x_{i_k})<\\costinf*\\) implies that \\(\\seq{x_{i_k}}\\) lies in a sublevel set of \\(h\\), and is therefore bounded.\n\t\t\t\t\t\t\tIn turn, \\cref{thm:PG:barPhi} implies that so is \\(\\seq{x_{i_k+1}}\\).\n\t\t\t\t\t\t\\item\n\t\t\t\t\t\t\t{\\it the whole sequence \\(\\seq{x_k}\\) is bounded}.\n\n\t\t\t\t\t\t\tTo this end, it remains to show that \\(\\seq{x_{k'}}[k'\\in K']\\) is bounded, where\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:K'}\n\t\t\t\t\t\t\t\tK'\\coloneqq\\set{k'\\in\\N}[k'\\geq i_{k'}+2].\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tWith \\(\\rho_{\\rm max}\\) as in \\cref{thm:PG:rhomax}, for every \\(k\\in\\N\\)\n\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\\mathcal L_{i_k+1}(x_\\star)\n\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\tfrac12\n\t\t\t\t\t\t\t\t\t\\|x_{i_k+1}-x_\\star\\|^2\n\t\t\t\t\t\t\t\t\t\\vphantom{\\left(\\tfrac{\\sigma_{i_k+1}}{\\sigma_{i_k}}\\right)}\n\t\t\t\t\t\t\t\t}^{\\text{bounded by \\cref{claim:convergence:ik}}}\n\t\t\t\t\t\t\t\t+~~\n\t\t\t\t\t\t\t\t\\overbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\tfrac{1}{4}\\|x_{i_k+1}-x_{i_k}\\|^2\n\t\t\t\t\t\t\t\t\t\\vphantom{\\left(\\tfrac{\\sigma_{i_k+1}}{\\sigma_{i_k}}\\right)}\n\t\t\t\t\t\t\t\t}^{\\mathclap{\\to0 \\text{ by \\cref{thm:PG:barPhi}}}}\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\sigma_{i_k+1}\\alpha_{i_k+1}(1+\\rho_{i_k+1})\n\t\t\t\t\t\t\t\t\t(\\cost_1(x_{i_k})-\\costinf*)\n\t\t\t\t\t\t\t\t\t\\vphantom{\\left(\\tfrac{\\sigma_{i_k+1}}{\\sigma_{i_k}}\\right)}\n\t\t\t\t\t\t\t\t}^{<0}\n\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t&\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\alpha_{\\rm max}\\rho_{\\rm max}^2\n\t\t\t\t\t\t\t\t\t\\bcost_2(x_{i_k})\n\t\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\t\\alpha_{\\rm max}\\left(1-\\tfrac{\\sigma_{i_k+1}}{\\sigma_{i_k}}\\right)\n\t\t\t\t\t\t\t\t\t\\bcost_2(x_{i_k+1})\n\t\t\t\t\t\t\t\t}_{\\text{bounded by \\cref{thm:PG:barPhi}}}.\n\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\tIn particular, we have\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:supL}\n\t\t\t\t\t\t\t\t\\sup_{k\\in\\N}\\mathcal L_{i_k+1}(x_\\star)<\\infty.\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tLet now \\(k'\\in K'\\).\n\t\t\t\t\t\t\tSince \\(\\cost_1(x_j)-\\costinf*\\geq0\\) for \\(j=i_{k'}+1,\\dots,k'\\), observe that\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:Lk'geq}\n\t\t\t\t\t\t\t\t\\tfrac12\\|x_{k'}-x_\\star\\|^2\n\t\t\t\t\t\t\t\\leq\n\t\t\t\t\t\t\t\t\\mathcal L_{k'}(x_\\star)\n\t\t\t\t\t\t\t\\quad\n\t\t\t\t\t\t\t\t\\forall k'\\in K',\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tand \\cref{thm:PG:qSD} yields that\n\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\\mathcal L_{k'}(x_\\star)\n\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\\mathcal L_{k'-1}(x_\\star)\n\t\t\t\t\t\t\t\\leq\\dots\\leq\n\t\t\t\t\t\t\t\t\\mathcal L_{i_{k'}+2}(x_\\star)\n\t\t\t\t\t\t\t\\numberthis\\label{eq:Lk'descent}\n\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\\mathcal L_{i_{k'}+1}(x_\\star)\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\sigma_{i_{k'}+1}\\alpha_{i_{k'}+1}(1+\\rho_{i_{k'}+1}-\\rho_{i_{k'}+2}^2)\n\t\t\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\t\t\\cost_1(x_{i_{k'}})-\\costinf*\n\t\t\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\\mathcal L_{i_{k'}+1}(x_\\star)\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\tfrac{\\sigma_{i_{k'}+1}}{\\sigma_{i_{k'}}}\n\t\t\t\t\t\t\t\t}_{\\leq1}\n\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\vphantom{\\tfrac{\\sigma_{i_{k'}+1}}{\\sigma_{i_{k'}}}}\n\t\t\t\t\t\t\t\t\t\\alpha_{\\rm max}(1+\\rho_{i_{k'}+1}-\\rho_{i_{k'}+2}^2)\n\t\t\t\t\t\t\t\t}_{\\text{\\clap{bounded by \\cref{thm:PG:rhomax}}}}\n\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\vphantom{\\tfrac{\\sigma_{i_{k'}+1}}{\\sigma_{i_{k'}}}}\n\t\t\t\t\t\t\t\t\t\\sigma_{i_{k'}}\n\t\t\t\t\t\t\t\t\t\\bigl|\n\t\t\t\t\t\t\t\t\t\t\\cost_1(x_{i_{k'}})-\\costinf*\n\t\t\t\t\t\t\t\t\t\\bigr|\n\t\t\t\t\t\t\t\t}_{\\text{\\clap{bounded by \\cref{thm:PG:barPhi}}}}\n\t\t\t\t\t\t\t\t< \\infty\n\t\t\t\t\t\t\t\\qquad\n\t\t\t\t\t\t\t\t\\forall k'\\in K',\n\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\twhere \\eqref{eq:supL} was used in the last inequality.\n\t\t\t\t\t\t\tHere, boundedness of the under-bracketed term follows from boundedness of \\(\\seq{x_{i_{k'}}}[k'\\in K']\\) and lower semicontinuity of \\(\\cost_1\\).\n\t\t\t\t\t\t\tThen, \\eqref{eq:Lk'geq} implies that the sequence \\(\\seq{x_{k'}}[k'\\in K']\\) is bounded.\n\t\t\t\t\t\t\tCombined with \\cref{claim:convergence:ik} and the fact that the index set \\(K'\\) is the complement of the indices therein, the claim follows.\n\t\t\t\t\t\t\\qedhere\n\t\t\t\t\t\t\\end{claims}\n\t\t\t\t\t\\end{proofitemize}\n\t\t\t\t\\end{proofitemize}\n\t\t\t\\end{proof}","sha256_utf8":"e674617d3f3eeb36082c45fc0b5e999d1cb70342846552ff86268da054777010"},{"start":97910,"end":107776,"start_line":2266,"end_line":2495,"raw_tex":"\\begin{proof}\n\t\t\t\tWe begin by remarking that boundedness of the sequence \\(\\seq{x_k}\\) is ensured by \\cref{cond:alphk,cond:alphk:nu}, as shown in \\cref{thm:PG:boundedness}.\n\t\t\t\tWe next prove each claim individually.\n\t\t\t\t\\begin{proofitemize}\n\t\t\t\t\\item \\ref{thm:recipe:rate}~\n\t\t\t\t\tConsider a convergent subsequence \\(x_{k_j}\\to x_\\infty\\), so that \\(x_{k_j+1}\\to x_\\infty\\) by \\cref{thm:PG:barPhi}.\n\t\t\t\t\tUp to further extracting if necessary we have that\n\t\t\t\t\t\\(\\alpha_{k_j+1}\\to\\alpha_\\infty\\geq\\alpha_{\\rm min}>0\\) and \\(\\sigma_{k_j}\\to0\\).\n\t\t\t\t\tObserve that\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\tx_{k+1}\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\argmin h({}\\cdot{};x_k,\\alpha_{k+1},\\sigk*),\n\t\t\t\t\t\\shortintertext{where}\n\t\t\t\t\t\th(w;x,\\alpha,\\sigma)\n\t\t\t\t\t\\coloneqq{} &\n\t\t\t\t\t\t(\\sigma \\g_1+\\g_2)(w)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{2\\alpha}\n\t\t\t\t\t\t\\bigl\\|w-x+\\alpha\\nabla\\bigl(\\sigma \\f_1+\\f_2\\bigr)(x)\\bigr\\|^2\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tis level bounded in \\(w\\) locally uniformly in \\((x,\\alpha,\\sigma)\\), as a function from \\(\\R^n\\times\\bigl(\\R^n\\times[\\alpha_{\\rm min},\\infty)\\times[0,\\infty)\\bigr)\\) to \\(\\Rinf\\).\n\t\t\t\t\tSince \\(h\\) is continuous in \\((x,\\alpha,\\sigma)\\), it follows from \\cite[Thm. 1.17]{rockafellar2011variational} that\n\t\t\t\t\t\\[\n\t\t\t\t\t\tx_\\infty\n\t\t\t\t\t\\in\n\t\t\t\t\t\t\\argmin h({}\\cdot{};x_\\infty,\\alpha_\\infty,0)\n\t\t\t\t\t=\n\t\t\t\t\t\t\\prox_{\\alpha_\\infty \\g_2}\\bigl(x_\\infty-\\alpha_\\infty\\nabla \\f_2(x_\\infty)\\bigr),\n\t\t\t\t\t\\]\n\t\t\t\t\tthis condition being equivalent to \\(x_\\infty\\in\\argmin(\\f_2+\\g_2)\\defeq\\X_2\\).\n\t\t\t\t\tWe next show that \\(\\bphik(x_k)\\to0\\).\n\n\t\t\t\t\tFor any \\(x_\\star\\in\\X_1\\), the subdifferential characterization of \\(x_{k+1}\\in\\prox_{\\alphk*\\gk*}(x_k-\\alphk*\\nabla\\fk*(x_k))\\)\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\tfrac{x_{k-1}-x_k}{\\alphk}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t(\\nabla\\fk(x_{k-1})-\\nabla\\fk(x_k))\n\t\t\t\t\t\\in\n\t\t\t\t\t\t\\partial\\phik(x_k)\n\t\t\t\t\t\\]\n\t\t\t\t\timplies that\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\phik(x_\\star)\n\t\t\t\t\t\\geq\n\t\t\t\t\t\t\\phik(x_k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\overbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{x_{k-1}-x_k}{\\alphk}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t(\\nabla\\fk(x_{k-1})-\\nabla\\fk(x_k))\n\t\t\t\t\t\t\t}^{\\to0 \\text{ by \\cref{thm:PG:barPhi}}}\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\\overbracket*[0.5pt]{\n\t\t\t\t\t\t\t\tx_\\star-x_k\n\t\t\t\t\t\t\t\t\\vphantom{\\tfrac{x_{k-1}-x_k}{\\alphk}}\n\t\t\t\t\t\t\t}^{\\text{\\clap{bounded}}}\n\t\t\t\t\t\t},\n\t\t\t\t\t\\]\n\t\t\t\t\thence that\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\costinf_2\n\t\t\t\t\t=\n\t\t\t\t\t\t\\lim_{k\\to\\infty}\\phik(x_\\star)\n\t\t\t\t\t\\geq\n\t\t\t\t\t\t\\limsup_{k\\to\\infty}\\phik(x_k)\n\t\t\t\t\t=\n\t\t\t\t\t\t\\limsup_{k\\to\\infty}\\left(\\sigk\\bcost_1(x_k)+\\overbracket*[0.5pt]{\\cost_2(x_k)}^{\\geq\\costinf_2}\\right)\n\t\t\t\t\t\\geq\n\t\t\t\t\t\t\\limsup_{k\\to\\infty}\\sigk\\bcost_1(x_k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\costinf_2.\n\t\t\t\t\t\\]\n\t\t\t\t\tHere, the second equality is obtained by adding and subtracting \\(\\sigk\\costinf_1\\) along with the fact that \\(\\sigk\\to0\\).\n\t\t\t\t\tSince \\(\\sigk\\bcost_1\\geq0\\), necessarily \\(\\sigk\\bcost_1(x_k)\\to0\\) and consequently \\(\\limsup_{k\\to\\infty}\\cost_2(x_k)=\\costinf_2\\).\n\t\t\t\t\tSimilarly, since \\(\\cost_2\\geq\\costinf_2\\) this necessarily implies that \\(\\cost_2(x_k)\\to\\costinf_2\\).\n\t\t\t\t\tThis shows that \\(\\bcost_2(x^k)\\to0\\) as claimed.\n\n\t\t\t\t\tWe next show that \\(\\seq{W_k}\\) too is vanishing.\n\t\t\t\t\tObserve that\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\tW_k\n\t\t\t\t\t\\defeq{} &\n\t\t\t\t\t\t\\tfrac{1-4\\alphk\\sigk\\lk^{(2)}\\Delk}{4}\n\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\\|x_k-x_{k-1}\\|^2\n\t\t\t\t\t\t}^{\\mathclap{\\to 0\\text{ by \\cref{thm:PG:barPhi}}}}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{\\sigk*}{\\sigk}\\alphk*\\rhok*\n\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\\bcost_2(x_{k-1})\n\t\t\t\t\t\t}^{\\mathclap{\\to 0}}\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\\sigk\\alphk\\Delk\\bcost_2(x_k)\n\t\t\t\t\t\t}_{\\to 0}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\alphk\n\t\t\t\t\t\t\\left(1+\\rhok\\right)\n\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\\sigk\n\t\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\t\\cost_1(x_{k-1})\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\costinf*\n\t\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\t}_{\\to0}\n\t\t\t\t\t\\to\n\t\t\t\t\t\t0\n\t\t\t\t\t\\quad\\text{as \\(k\\to\\infty\\)},\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\twhere we also used the fact that \\(\\seq{\\alphk}\\leq\\alpha_{\\rm max}\\) and \\(\\seq{\\rhok}\\leq\\rho_{\\rm max}\\) as in \\cref{thm:PG:rhomax}.\n\n\t\t\t\t\tThe asserted sublinear rate is a consequence of \\eqref{eq:sublin:res}, the characterization of the residual in \\cref{thm:H} (see \\eqref{eq:Hk:phi}), and the upper bounds for \\(\\lk\\), \\(\\Lk\\) in \\cref{thm:l<=L}:\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\textstyle\\sum_{k=0}^K\\dist^2(0, \\partial \\phik*(x_{k+1})\n\t\t\t\t\t\\leq{}&\n\t\t\t\t\t\t\\textstyle\\sum_{k=0}^K\\tfrac{1}{\\alphk*^2}\\|\\Hk*(x_{k})-\\Hk*(x_{k+1}) \\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\leq {}&\n\t\t\t\t\t\t\\tfrac{M_{\\rm max}^2}{\\alpha_{\\rm min}}\\textstyle\\sum_{k=0}^K \\tfrac{1}{\\alphk*}\\|x_{k}-x_{k+1}\\|^2\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\tfrac{M_{\\rm max}^2}{\\alpha_{\\rm min}(1-\\nu)}\\bar{\\originalvarphi}_0(x_{0}),\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\twhere \\(M_{\\rm max} = \\sup_{k\\in\\N}M_k\\leq 1+ \\alpha_{\\rm max}L_{f_0, \\V}\\), see \\cref{thm:lL}.\n\n\n\t\t\t\t\\item \\ref{thm:recipe:solution}~\n\t\t\t\t\tWe begin by observing that, since \\(\\bcost_2(x^k)\\to0\\), by lower semicontinuity all limit points of \\(\\seq{x^k}\\) belong to \\(\\X_2\\).\n\t\t\t\t\tAs done in the proof of \\cref{thm:PG:boundedness} we consider two possible cases.\n\t\t\t\t\t\\begin{proofitemize}[leftmargin=*,topsep=0pt]\n\t\t\t\t\t\\item\n\t\t\t\t\t\t{\\it Case 1: \\(\\cost_1(x_k)\\geq\\costinf*\\) holds for \\(k\\) large enough.}\n\n\t\t\t\t\t\tIn this case, we will actually show that the sequence \\(\\seq{x_k}\\) converges to a solution of \\eqref{eq:P}.\n\t\t\t\t\t\tWe first argue that there exists an optimal limit point; to this end, since all limit points are feasible and because of lower semicontinuity, it suffices to show that \\(\\liminf_{k\\to\\infty}\\cost_1(x_k)=\\costinf*\\).\n\t\t\t\t\t\tA telescoping argument on \\cref{thm:PG:qSD} along with \\cref{cond:alphamin} yields\n\t\t\t\t\t\t\\begin{equation}\n\t\t\t\t\t\t\t\\alpha_{\\rm min}\\sum_{k\\in \\N}\\sigk(1+\\rhok-\\rhok*^2)\\bigl(\\cost_1(x_{k-1})-\\costinf*\\bigr)\n\t\t\t\t\t\t\\leq\n\t\t\t\t\t\t\t\\sum_{k\\in \\N}\\sigk\\alphk(1+\\rhok-\\rhok*^2)\\bigl(\\cost_1(x_{k-1})-\\costinf*\\bigr)\n\t\t\t\t\t\t<\n\t\t\t\t\t\t\t\\infty.\n\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\tSince \\(\\sum_{k\\in\\N}\\sigk=\\infty\\), necessarily\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\liminf_{k\\to\\infty}(1+\\rhok-\\rhok*^2)(\\cost_1(x_{k-1})-\\costinf*)\n\t\t\t\t\t\t=\n\t\t\t\t\t\t\t0.\n\t\t\t\t\t\t\t\\numberthis\\label{eq:liminf:cost1}\n\t\t\t\t\t\t\\end{align*}\n\t\t\t% \t\t\tObserve first that\n\t\t\t% \t\t\t\\begin{align*}\n\t\t\t% \t\t\t\t\\rhok*\n\t\t\t% \t\t\t=\n\t\t\t% \t\t\t\t\\tfrac{\\sigk*\\alphk*}{\\sigk\\alphk}\n\t\t\t% \t\t\t\\leq\n\t\t\t% \t\t\t\t\\tfrac{\\sigk*\\balphk*}{\\sigk\\alphk}\n\t\t\t% \t\t\t\\leq\n\t\t\t% \t\t\t\t\\tfrac{\\sigk*}{\\sigk}\n\t\t\t% \t\t\t\t\\sqrt{\\tfrac{\\sigk}{\\sigma_{k-1}}\\bigl(1+\\rhok\\bigr)}\n\t\t\t% \t\t\t\t\\tfrac{\\sigk}{\\sigk*}\n\t\t\t% \t\t\t\\leq\n\t\t\t% \t\t\t\t\\sqrt{1+\\rhok}\n\t\t\t% \t\t\t,\n\t\t\t% \t\t\t\\numberthis\\label{eq:rhok:plus}\n\t\t\t% \t\t\t\\end{align*}\n\t\t\t% \t\t\twhere the first inequality holds by \\cref{cond:alphk}, the second is due to the first term in \\eqref{eq:adabim:alphk*_0}, and the last inequality follows from \\eqref{eq:sigk}.\n\t\t\t\t\t\tNotice that \\(\\limsup_{k\\to\\infty}(1+\\rhok-\\rhok*^2)>0\\), for otherwise \\(1+\\rhok-\\rhok*^2 \\to 0\\), implying that \\(\\liminf_{k \\to \\infty} \\rhok >0\\), and consequently that \\(\\alphk\\sigk\\) eventually increases exponentially, which contradicts \\(\\sigk\\alphk \\leq \\sigk\\alpha_{\\rm max} \\to 0\\).\n\t\t\t\t\t\tHence, \\(\\limsup_{k\\to\\infty}(1+\\rhok-\\rhok*^2)>0\\), which along with \\eqref{eq:liminf:cost1} implies that \\(\\liminf_{k\\to\\infty}\\cost_1(x_{k-1})=\\costinf*\\).\n\t\t\t\t\t\tTherefore, an optimal limit point exists, be it \\(x_\\infty\\).\n\t\t\t\t\t\tSince \\(W_k\\to0\\) by assertion \\ref{thm:recipe:rate} and since \\(\\mathcal L_k(x_\\infty)\\) converges, it follows that \\(\\frac12\\|x_k-x_\\infty\\|^2 = \\mathcal L_k(x_\\infty) - W_k\\) converges as well.\n\t\t\t\t\t\tSince along a subsequence \\(\\frac12\\|x_k-x_\\infty\\|^2\\) converges to zero, necessarily \\(\\lim_{k\\to\\infty}\\frac12\\|x_k-x_\\infty\\|^2=0\\), proving that the entire sequence \\(\\seq{x_k}\\) converges to \\(x_\\infty\\).\n\t\t\t\t\t\\item\n\t\t\t\t\t\t{\\it Case 2: \\(\\cost_1(x_k)<\\costinf*\\) holds infinitely often.}\n\n\t\t\t\t\t\tRecall the index \\(i_k\\) and the set \\(K'\\) defined in \\eqref{eq:ik} and \\eqref{eq:K'}.\n\t\t\t% \t\t\tOnce we establish that all the limit points of \\(\\seq{x_{i_k}}\\) and \\(\\seq{x_{i_k+1}}\\) are optimal, {\\red it only remains to show that\n\t\t\t% \t\t\t\\(\n\t\t\t% \t\t\t\t\\dist(x_{k'},\\X_1)\n\t\t\t% \t\t\t\\to\n\t\t\t% \t\t\t\t0\n\t\t\t% \t\t\t\\)\n\t\t\t% \t\t\tas \\(K'\\ni k'\\to\\infty\\)\\note{Why}}.\n\t\t\t\t\t\tHaving established boundedness of the entire sequence, limit points of \\(\\seq{x_{i_k}}\\) exist and, as shown above, belong to \\(\\X_2\\).\n\t\t\t\t\t\tMoreover, it follows from \\eqref{eq:ik} that \\(\\limsup_{k\\to\\infty}\\cost_1(x_{i_k})\\leq\\costinf*\\), and a lower semicontinuity argument then yields that all the limit points of \\(\\seq{x_{i_k}}\\) attain the optimal cost and are therefore optimal.\n\t\t\t\t\t\tSince \\(x_{i_k}-x_{i_k+1}\\to0\\) by \\cref{thm:PG:barPhi}, the same is also true for \\(\\seq{x_{i_k+1}}\\), and in particular\n\t\t\t\t\t\t\\(\\dist(x_{i_{k}+1},\\X_1) \\to 0\\) as \\(k \\to \\infty\\).\n\n\t\t\t\t\t\tFor each \\(k\\) let \\(\\bar x_k\\coloneqq\\proj_{\\X_1}\\!x_k\\), which is well defined since \\(\\X_1\\neq\\emptyset\\) is closed and convex.\n\t\t\t\t\t\tRecalling that \\(\\mathcal L_k(x_\\star)=W_k + \\tfrac12\\|x_k-x_\\star\\|^2\\), for \\(k'\\in K'\\) we have\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\tfrac12\\dist(x_{k'},\\X_1)^2\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\tfrac12\\|x_{k'}-\\bar x_{i_{k'}+1}\\|^2\n\t\t\t\t\t\t={}\n\t\t\t\t\t\t\t\\mathcal L_{k'}(\\bar x_{i_{k'}+1})-W_{k'}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\\dueto{\\scriptsize by \\eqref{eq:Lk'descent}}\n\t\t\t\t\t\t\t\\\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\mathcal L_{i_{k'}+2}(\\bar x_{i_{k'}+1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\tW_{k'}\n\t\t\t\t\t\t={}\n\t\t\t\t\t\t\t% \\tfrac12\\dist(x_{i_{k'}+2},\\X_1)^2\n\t\t\t\t\t\t\t\\tfrac12\\|x_{i_{k'}+2} - \\bar x_{i_{k'}+1}\\|^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\tW_{i_{k'}+2}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\tW_{k'}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\tfrac12\\big(\\dist(x_{i_{k'}+1},\\X_1) + \\|x_{i_{k'}+1} - x_{i_{k'}+2}\\|\\big)^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\tW_{i_{k'}+2}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\tW_{k'}\n\t\t\t\t\t\t\\to\n\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\tas \\(K'\\ni k'\\to\\infty\\), where the limit follows from \\cref{thm:PG:barPhi}, the vanishing of \\(\\seq{W_k}\\) established in assertion \\ref{thm:recipe:rate}, and the fact that \\(i_k\\to\\infty\\) as \\(k\\to\\infty\\).\n\t\t\t\t\t\tSince \\(K'\\cup\\set{i_k,i_k+1}[k\\in\\N]=\\N\\), we conclude that all the limit points of \\(\\seq{x^k}\\) are optimal.\n\t\t\t\t\t\\qedhere\n\t\t\t\t\t\\end{proofitemize}\n\t\t\t\t\\end{proofitemize}\n\t\t\t\\end{proof}","sha256_utf8":"1ec28c371435769c142616a64a34eb36247e10526ec60cb785f6d6f55c333bef"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"凸简单双层问题","source_units":[{"start":16831,"end":17059,"start_line":499,"end_line":505,"raw_tex":"\\begin{align}\n\t\t\t\t\\label{eq:P1}\n\t\t\t\t\\minimize_{x\\in\\R^n}{} &~ \\cost_1(x)\\coloneqq \\f_1(x)+\\g_1(x)\n\t\t\t\\\\\n\t\t\t\t\\label{eq:P2}\n\t\t\t\t\\stt{} &~ x\\in\\X_2\\coloneqq\\argmin_{w\\in\\R^n}\\set{\\cost_2(w)\\coloneqq \\f_2(w)+\\g_2(w)},\n\t\t\t\\end{align}","sha256_utf8":"c50e7bed98b4ef3178e35810923f34d29029979b97eb3f0058e2dc38762e6002"}]},{"id":"S002","semantic_type":"definition","kind":"definition","title":"逆罚目标族","source_units":[{"start":18197,"end":18390,"start_line":512,"end_line":517,"raw_tex":"\\begin{equation}\\label{eq:phik}\n\t\t\t\\fk\\coloneqq\\sigk\\f_1+\\f_2,\n\t\t\t\\quad\\gk\\coloneqq\\sigk\\g_1+\\g_2,\n\t\t\\quad\\text{and}\\quad\n\t\t\t\\phik\\coloneqq \\fk + \\gk = \\sigk \\cost_1 + \\cost_2,\n\t\t\\end{equation}","sha256_utf8":"1861b24ab77898557c9f0a1455b76bc2bfcfba16a732b2fdbdcf3b5631e18299"}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","title":"基本凸性与解集假设","source_units":[{"start":35375,"end":36434,"start_line":751,"end_line":777,"raw_tex":"\\begin{assumption}[basic requirements]\\label{ass:basic}%\n\t\t\t\tThe following hold in problem \\eqref{eq:P}:\n\t\t\t\t\\begin{enumeratass}\n\t\t\t\t\\item \\label{ass:f}%\n\t\t\t\t\t\\(\\f_1,\\f_2:\\R^n\\to \\R\\) are convex and have locally Lipschitz-continuous gradients;\n\t\t\t\t\\item \\label{ass:g}%\n\t\t\t\t\t\\(\\g_1,\\g_2:\\R^n \\to \\Rinf\\) are proper lsc convex functions with easy to compute proximal mappings;\n\t\t\t\t\\item \\label{ass:cost1}%\n\t\t\t\t\tthe upper level problem restricted to \\(\\dom\\cost_2\\) is lower bounded:\n\t\t\t\t\t\\begin{equation}\\label{eq:costinf_1}\n\t\t\t\t\t\t\\costinf_1\n\t\t\t\t\t{}\\coloneqq{}\n\t\t\t\t\t\t\\inf_{x\\in\\dom\\cost_2}\\set{\\f_1(x)+\\g_1(x)}\n\t\t\t\t\t{}>{}\n\t\t\t\t\t\t-\\infty;\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\\item \\label{ass:X1}%\n\t\t\t\t\tthe set of solutions \\(\\X_1\\coloneqq\\argmin\\set{\\cost_1(x)}[x\\in\\argmin\\cost_2]\\) is nonempty and bounded; in particular,\n\t\t\t\t\t\\begin{equation}\\label{eq:costinf_2}\n\t\t\t\t\t\t\\costinf_2\n\t\t\t\t\t{}\\coloneqq{}\n\t\t\t\t\t\t\\inf_{\\fillwidthof[c]{x\\in\\dom \\g_2}{x\\in\\R^n}}\\set{\\f_2(x)+\\g_2(x)}\n\t\t\t\t\t{}>{}\n\t\t\t\t\t\t-\\infty\\mathrlap{.}\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\\end{enumeratass}\n\t\t\t\\end{assumption}","sha256_utf8":"0fac96d3016b3ce3a46d6e5e2907951efe58c051f021fc660494351dca7eedb9"}]},{"id":"S004","semantic_type":"notation","kind":"notation","title":"数值与集合记号","source_units":[{"start":31980,"end":32947,"start_line":681,"end_line":686,"raw_tex":"\t\t\tThe sets of natural, real, and extended-real numbers are \\(\\N\\), \\(\\R\\coloneqq(-\\infty,\\infty)\\) and \\(\\Rinf\\coloneqq\\R\\cup\\set\\infty\\), respectively, while the positive and strictly positive reals are \\(\\R_+\\coloneqq[0,\\infty)\\) and \\(\\R_{++}\\coloneqq(0,\\infty)\\).\n\t\t\tWe adopt the conventions that \\(0\\in\\N\\) and \\(1/0=\\infty\\).\n\t\t\tGiven \\(x\\in\\Rinf\\), its positive part is indicated as \\([x]_+\\coloneqq\\max\\set{0,x}\\).\n\t\t\tWe denote by \\(\\innprod{\\cdot}{\\cdot}\\) and \\(\\|{}\\cdot{}\\|\\) the standard Euclidean inner product and the induced norm, and with \\(\\id\\) the identity function defined on a suitable space.\n\t\t\tThe closed Euclidean ball of radius \\(r>0\\) and centered at \\(\\bar x\\in\\R^n\\) is denoted as \\(\\cball{\\bar x}{r}\\coloneqq\\set{x\\in\\R^n}[\\|x-\\bar x\\|\\leq r]\\).\n\t\t\tGiven two nonempty sets \\(U,V\\subseteq\\R^n\\), with \\(U+V\\coloneqq\\set{u+v}[u\\in U,v\\in V]\\) we indicate their Minkowski sum, while \\(\\conv V\\) is used to denote the convex hull of \\(V\\).\n","sha256_utf8":"c7b26999d96c2104cd617a32fa2d383670d04a402e5d2a39f05491d50f6ef42f"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"定义域与水平有界性","source_units":[{"start":33060,"end":33707,"start_line":690,"end_line":711,"raw_tex":"\t\t\tThe \\DEF{domain} and \\DEF{epigraph} of an extended real--valued function \\(\\func{h}{\\R^n}{\\Rinf}\\) are, respectively, the sets\n\t\t\t\\(\n\t\t\t\t\\dom h\n\t\t\t\\coloneqq\n\t\t\t\t\\set{x\\in\\R^n}[h(x)<\\infty]\n\t\t\t\\)\n\t\t\tand\n\t\t\t\\(\n\t\t\t\t\\epi h\n\t\t\t\\coloneqq\n\t\t\t\t\\set{(x, c )\\in\\R^n\\times\\R}[h(x)\\leq c]\n\t\t\t\\).\n\t\t\tFunction \\(h\\) is said to be \\DEF{proper} if \\(\\dom h\\neq\\emptyset\\), and \\DEF{lower semicontinuous (lsc)} if \\(\\epi h\\) is a closed subset of \\(\\R^{n+1}\\).\n\t\t\tWe say that \\(h\\) is \\emph{level bounded} if its \\( c \\)-sublevel set\n\t\t\t\\(\n\t\t\t\t\\lev_{\\leq c }h\n\t\t\t{}\\coloneqq{}\n\t\t\t\t\\set{x\\in\\R^n}[\n\t\t\t\t\th(x)\\leq c\n\t\t\t\t]\n\t\t\t\\)\n\t\t\tis bounded for all \\( c \\in\\R\\).\n","sha256_utf8":"f10a016c164d710ab9047aa9e8dc7880175bd2a5cc9d0af84e3023c863ce06e1"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"投影与集合距离","source_units":[{"start":33867,"end":34072,"start_line":714,"end_line":719,"raw_tex":"\t\t\tThe projection onto and the distance from $E$ are respectively denoted by\n\t\t\t\\[\n\t\t\t\t\\proj_E(x)\\coloneqq\\argmin_{z\\in E}\\|z-x\\|\n\t\t\t\\quad\\text{and}\\quad\n\t\t\t\t\\dist(x,E)\\coloneqq\\inf_{z\\in E}\\|z-x\\|.\n\t\t\t\\]\n","sha256_utf8":"aa24851d4a10a5bddaf23d14a5c92c0259a0037a897e30b8446d993faab6b86b"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"次微分与近端映射","source_units":[{"start":34072,"end":34734,"start_line":720,"end_line":739,"raw_tex":"\t\t\tThe (convex) \\DEF{subdifferential} of a proper lsc convex function \\(\\func{h}{\\R^n}{\\Rinf}\\) at a point \\(\\bar x\\) is the set\n\t\t\t\\(\n\t\t\t\t\\partial h(\\bar x)\n\t\t\t\\coloneqq\n\t\t\t\t\\set{v\\in\\R^n}[h(x)\\geq h(\\bar x)+\\innprod{v}{x-\\bar x}\\ \\forall x\\in\\R^n]\n\t\t\t\\).\n\t\t\tThe \\DEF{proximal mapping} of \\(h\\) is \\(\\func{\\prox_h}{\\R^n}{\\R^n}\\) defined by\n\t\t\t\\[\n\t\t\t\t\\prox_h(x)\n\t\t\t=\n\t\t\t\t\\argmin_{w\\in\\R^n}\\set{\n\t\t\t\t\th(w)+\\tfrac12\\|w-x\\|^2\n\t\t\t\t},\n\t\t\t\\]\n\t\t\tand is characterized by the implicit subdifferential inclusion \\cite[Eq. (24.2)]{bauschke2017convex}\n\t\t\t\\begin{equation}\\label{eq:proxsubgrad}\n\t\t\t\tx-\\prox_h(x)\n\t\t\t\\in\n\t\t\t\t\\partial h\\bigl(\\prox_h(x)\\bigr).\n\t\t\t\\end{equation}\n","sha256_utf8":"bbbd2d019fd930f932b4d62d6905de83fbe4f60ae924b8771cc37a494ed8e992"}]},{"id":"S008","semantic_type":"algorithm","kind":"algorithm","title":"变参数近端梯度迭代","source_units":[{"start":37008,"end":37121,"start_line":784,"end_line":788,"raw_tex":"\\begin{equation}\\label{eq:PG}\n\t\t\tx_{k+1}\n\t\t=\n\t\t\t\\prox_{\\alphk*\\gk*}(x_k-\\alphk*\\nabla\\fk*(x_k)).\n\t\t\\end{equation}","sha256_utf8":"48b7bf49e3cf04c306cc995b25fcaa305f7b004b1c8f5a5467e4aceffc3ddd31"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"局部曲率估计","source_units":[{"start":48362,"end":49423,"start_line":944,"end_line":978,"raw_tex":"\t\t\tSpecifically, we use local Lipschitz estimates of the differentiable functions \\(\\fk=\\sigk\\f_1+\\f_2\\) at the previous iterates \\(x_k, x_{k-1}\\) as%\n\t\t\t\\begin{subequations}\\label{eq:lL}\n\t\t\t\t\\begin{align}\n\t\t\t\t\t\\lk\n\t\t\t\t{}\\coloneqq{} &\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\\nabla\\fk(x_{k-1})-\\nabla\\fk(x_k)}{x_{k-1}-x_k}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t}\n\t\t\t\t\t,\\quad\n\t\t\t\t\t\\Lk\n\t\t\t\t\t{}\\coloneqq {}\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\|\\nabla\\fk(x_{k-1})-\\nabla\\fk(x_k)\\|^2\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t},\n\t\t\t\t\\shortintertext{%\n\t\t\t\t\tand for each \\(\\f_i\\) with\n\t\t\t\t}\n\t\t\t\t\\label{eq:Lk2}\n\t\t\t\t\t\\lk^{(i)}\n\t\t\t\t{}\\coloneqq{} &\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\\nabla\\f_i(x_{k-1})-\\nabla\\f_i(x_k)}{x_{k-1}-x_k}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t},\n\t\t\t\t\\quad\n\t\t\t\t\ti=1,2,\n\t\t\t\t\\end{align}\n\t\t\t\tso that \\(\\lk=\\sigk\\lk^{(1)}+\\lk^{(2)}\\).\n\t\t\t\\end{subequations}\n\t\t\tNoting that the denominator of \\(\\lk\\), \\(\\Lk\\), or \\(\\lk^{(i)}\\) is zero iff \\(\\nabla f(x^k)-\\nabla f(x^{k-1})=0\\), we use the convention \\(\\nicefrac00=0\\) so that both \\(\\lk\\) and \\(\\Lk\\) are (well-defined, positive) real numbers.\n","sha256_utf8":"b175d3349de60e87dd3207faf82dc5cbd142150fd93583ec88a7d312e0b98966"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"步长上界与参数比","source_units":[{"start":54814,"end":55389,"start_line":1088,"end_line":1108,"raw_tex":"\t\t\t\tand denoting \\(\\rhok\\coloneqq\\frac{\\sigk\\alphk}{\\sigma_{k-1}\\alpha_{k-1}}\\) initialize the next stepsize as\\algfootnotemark\\textsuperscript{,}\\algfootnotemark\n\t\t\t\t\\begin{equation}\\label{eq:adabim:alphk*_0}\n\t\t\t\t\t\\balphk*\n\t\t\t\t=\n\t\t\t\t\t\\tfrac{\\sigk}{\\sigk*}\n\t\t\t\t\t\\alphk\n\t\t\t\t\t\\min\\set{\n\t\t\t\t\t\t\\sqrt{\\tfrac{\\sigk}{\\sigma_{k-1}}\\bigl(1+\\rhok\\bigr)}\n\t\t\t\t\t\t,\\,\n\t\t\t\t\t\t\\tfrac{\n\t\t\t\t\t\t\t\\sqrt{\n\t\t\t\t\t\t\t\t1\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t4\\left(1-\\frac{\\sigk}{\\sigma_{k-1}}\\right)\n\t\t\t\t\t\t\t\t\\alphk\\lk^{(2)}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t2\\sqrt{[\\alphk^2 \\Lk^2 - \\alphk\\lk]_+}\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\\end{equation}\n","sha256_utf8":"5398aa5cc629c2d04d4d31ac59b6733c9a4befa10123d20495d8b61c4953b64c"}]},{"id":"S011","semantic_type":"derived_fact","kind":"derived_fact","title":"平移目标与最优值","source_units":[{"start":63303,"end":64190,"start_line":1256,"end_line":1278,"raw_tex":"\\begin{remark}[Bar notation for minima and shifted costs]\\label{rem:cost}%\n\t\t\t\tLet us define\n\t\t\t\t\\[\n\t\t\t\t\t\\bcost_i\\coloneqq\\cost_i-\\costinf_i, \\; i=1,2,\n\t\t\t\t\\quad\\text{and}\\quad\n\t\t\t\t\t\\bphik \\coloneqq \\sigk \\bcost_1 + \\bcost_2,\n\t\t\t\t\\]\n\t\t\t\twhere \\(\\costinf_i=\\inf_{\\dom\\cost_2}\\cost_i\\) as in \\cref{ass:basic} (restricting to \\(\\dom\\cost_2\\) is superfluous for \\(i=2\\)), and let\n\t\t\t\t\\begin{equation}\\label{eq:phi*}\n\t\t\t\t\t\\costinf*\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\inf_{\\X_2}\\cost_1\n\t\t\t\t\\end{equation}\n\t\t\t\tbe the optimal cost of problem \\eqref{eq:P}.\n\t\t\t\tThen,\n\t\t\t\t\\begin{enumerate}\n\t\t\t\t\\item\\label{thm:bar>=0}%\n\t\t\t\t\t\\(\\bcost_1(x)\\geq0\\) for any \\(x\\in\\dom\\cost_2\\) and \\(\\bcost_2(x)\\geq0\\) for any \\(x\\in\\R^n\\);\n\t\t\t\t\\item \\label{thm:Phi*}%\n\t\t\t\t\t\\( \\tfrac1{\\sigk}\\left(\\phik(x_\\star)-\\costinf_2\\right)=\\costinf*\\) for any \\(x_\\star\\in\\X_1\\) and \\(k\\in\\N\\).\n\t\t\t\t\\qedhere\n\t\t\t\t\\end{enumerate}\n\t\t\t\\end{remark}","sha256_utf8":"576fb59e20e4ddedf2b4bcaf5a5de28afc9e42a130fca626508fcc19c5e85802"}]},{"id":"S012","semantic_type":"derived_fact","kind":"derived_fact","title":"前向算子与残差包含","source_units":[{"start":64192,"end":64619,"start_line":1280,"end_line":1284,"raw_tex":"\t\t\tA key step in our convergence analysis based on adaptive stepsizes relies on  the introduction of the quantities in \\eqref{eq:lL}.\n\t\t\tWe define the shorhand notation for the forward operator \\(\\Hk = \\id - \\alphk\\nabla\\fk\\), and note that by optimality conditions of the prox-grad update \\eqref{eq:PG}\n\t\t\t\\begin{equation}\\label{eq:Hk:phi}\n\t\t\t\t\\tfrac1{\\alphk}(\\Hk(x_{k-1})-\\Hk(x_k)) \\in \\partial \\phik(x_k).\n\t\t\t\\end{equation}\n","sha256_utf8":"7f54e089a24f51148b12a9b73cb16468f3d17e3e9af0f5377bfd54471da62aa1"}]},{"id":"S013","semantic_type":"derived_fact","kind":"derived_fact","title":"前向算子局部范数界","source_units":[{"start":64827,"end":65893,"start_line":1287,"end_line":1322,"raw_tex":"\\begin{fact}[{\\cite[Lem. 2.1]{latafat2023adaptive}}]\\label{thm:lL}%\n\t\t\t\tSuppose that \\cref{ass:f} holds, and for \\(x_{k-1},x_k\\in\\R^n\\) and \\(\\alphk>0\\) let \\(\\lk\\) and \\(\\Lk\\) be as in \\eqref{eq:lL} and \\(\\Hk\\coloneqq\\id-\\alphk\\nabla\\fk\\).\n\t\t\t\tThen, the following hold:\n\t\t\t\t\\begin{enumerate}\n\t\t\t\t\\item \\label{thm:H}%\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\|\\Hk(x_{k-1})-\\Hk(x_k)\\|\n\t\t\t\t\t=\n\t\t\t\t\t\t\\Mk\n\t\t\t\t\t\t\\|x_{k-1}-x_k\\|\n\t\t\t\t\t\\),\n\t\t\t\t\twhere\n\t\t\t\t\t\\[\n\t\t\t\t\t\t|1-\\alphk\\Lk|\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\Mk\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\sqrt{1 + \\alphk^2 \\Lk^2 -2\\alphk\\lk}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t1+\\alphk\\Lk.\n\t\t\t\t\t\\]\n\t\t\t\t\\item\\label{thm:l<=L}%\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\lk\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\Lk\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\sigk L_{\\f_1,\\V} + L_{\\f_2,\\V}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\sigma_0 L_{\\f_1,\\V} + L_{\\f_2,\\V}\n\t\t\t\t\t\\),\n\t\t\t\t\twhere \\(L_{\\f_i,\\V}\\) is a Lipschitz modulus for \\(\\nabla\\f_i\\) on a convex set \\(\\V\\) containing \\(x_{k-1}\\) and \\(x_k\\), \\(i=1,2\\).\\footnote{%\n\t\t\t\t\t\tThe first inequality owes to Cauchy-Schwarz; the last one to convexity of \\(\\f_1\\) together with the fact that \\(\\sigk\\leq\\sigma_0\\).%\n\t\t\t\t\t}%\n\t\t\t\t\\end{enumerate}\n\t\t\t\\end{fact}","sha256_utf8":"0452940b569d1ceaf9a1e081ff565ef6b0e92a60f0b404979437dc6a7409690f"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"罚目标及参数差分","source_units":[{"start":71699,"end":72275,"start_line":1425,"end_line":1439,"raw_tex":"\t\t\t\\begin{equation}\\label{eq:uppercase}\n\t\t\t\t\\Phik=\\Fk+\\Gk\n\t\t\t\\quad\\text{with}\\quad\n\t\t\t\t\\Fk=\\tfrac1{\\sigk}\\fk=\\f_1+\\tfrac1{\\sigk}\\f_2\n\t\t\t\\quad\\text{and}\\quad\n\t\t\t\t\\Gk=\\tfrac1{\\sigk}\\gk=\\g_1+\\tfrac1{\\sigk}\\g_2,\n\t\t\t\\end{equation}\n\t\t\tnoticing that \\(\\Phik=\\cost_1+\\frac{1}{\\sigk}\\cost_2=\\frac{1}{\\sigk}\\phik\\).\n\t\t\tDoing so allows us to express the difference\n\t\t\t\\begin{equation}\n\t\t\t\t\\Phik*-\\Phik\n\t\t\t{}={}\n\t\t\t\t\\left(\\tfrac1{\\sigk*}-\\tfrac1{\\sigk}\\right)\\cost_2\n\t\t\t\\end{equation}\n\t\t\tas a multiple of the lower-level cost \\(\\cost_2\\), rather than of the upper-level cost \\(\\cost_1\\).\n","sha256_utf8":"d9a42be4b8e19028e5d1b4f3d6c06492413d2c719a1a939bddd6f2ea0cbb2358"}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","title":"统一收敛参数条件","source_units":[{"start":85191,"end":86059,"start_line":1919,"end_line":1943,"raw_tex":"\t\t\t\t\tThere exist \\(\\alpha_{\\rm max}\\geq\\alpha_{\\rm min}>0\\) and \\(\\nu\\in(0,1)\\) such that, for every \\(k\\in\\N\\),\n\n\t\t\t\t\t\\begin{minipage}[t]{0.57\\linewidth}\n\t\t\t\t\t\t\\vspace*{-.5\\baselineskip}%\n\t\t\t\t\t\t\\begin{alphaproperties}\n\t\t\t\t\t\t\\item \\label{cond:alphk}%\n\t\t\t\t\t\t\t\\(\\alphk*\\leq\\min\\set{\\balphk*,\\alpha_{\\rm max}}\\) with \\(\\balphk*\\) as in \\eqref{eq:adabim:alphk*_0}%\n\n\t\t\t\t\t\t\\item \\label{cond:alphk:nu}%\n\t\t\t\t\t\t\t\\(\\alphk*\\lk*\\leq\\nu\\)\n\n\t\t\t\t\t\t\\item \\label{cond:alphamin}%\n\t\t\t\t\t\t\t\\(\\alphk*\\geq\\alpha_{\\rm min}\\)\n\t\t\t\t\t\t\\end{alphaproperties}\n\t\t\t\t\t\\end{minipage}\n\t\t\t\t\t\\hfill\\vline\\hfill\n\t\t\t\t\t\\begin{minipage}[t]{0.35\\linewidth}\n\t\t\t\t\t\t\\vspace*{-.5\\baselineskip}%\n\t\t\t\t\t\t\\begin{sigmaproperties}\n\t\t\t\t\t\t\\item \\label{cond:monotone}%\n\t\t\t\t\t\t\t\\(0<\\sigk* \\leq \\sigk\\)\n\n\t\t\t\t\t\t\\item \\label{cond:slowcontrol}%\n\t\t\t\t\t\t\t\\(\\sigk\\to 0\\) and \\(\\sum_{k\\in\\N}\\sigk=\\infty\\)\n\t\t\t\t\t\t\\end{sigmaproperties}\n","sha256_utf8":"45e046d6cea53513e69b70f61a2383c2bd8aa851d70822ca9fc367160405c19a"}]},{"id":"S016","semantic_type":"theorem","kind":"theorem","title":"统一近端梯度收敛定理","source_units":[{"start":96793,"end":97906,"start_line":2249,"end_line":2265,"raw_tex":"\\begin{theorem}[convergence recipe for proximal gradient iterations]\\label{thm:recipe}%\n\t\t\t\tSuppose that \\cref{ass:basic} holds, and consider the proximal gradient iterations \\eqref{eq:PG} with \\(\\seq{\\sigk}\\) and \\(\\seq{\\alphk}\\) complying with all \\cref{cond:alphk,cond:alphk:nu,cond:alphamin,cond:monotone,cond:slowcontrol}.\n\t\t\t\tThen,\n\t\t\t\t\\begin{enumerate}\n\t\t\t\t\\item \\label{thm:recipe:solution}%\n\t\t\t\t\t (\\(\\seq{x_k}\\) is bounded and) \\(\\seq{\\dist(x_k, \\X_1)}\\) converges to zero.\n\t\t\t\t\t% \\(\\seq{x_k}\\) is bounded and all its limit points are solutions to the bilevel problem \\eqref{eq:P};\n\t\t\t\t\\item \\label{thm:recipe:rate}%\n\t\t\t\t\tboth \\(\\bphik(x_k)\\) and \\(W_k\\) as in \\cref{thm:LW} converge to 0 as \\(k\\to\\infty\\), and the following worst-case rate holds\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\min_{k\\leq K}\\dist^2(0, \\partial \\phik*(x_{k+1})\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\tfrac{ (1+ \\alpha_{\\rm max}L_{f_0, \\V})^2}{\\alpha_{\\rm min}(1-\\nu)(K+1)}\\bar{\\originalvarphi}_0(x_0),\n\t\t\t\t\t\\]\n\t\t\t\t\twhere \\(L_{f_0, \\V}\\) is a Lipschitz modulus for \\(\\sigma_0\\nabla\\f_1+\\nabla\\f_2\\) on \\(\\V\\coloneqq\\conv\\set{x^k}[k\\in\\N]\\).\n\t\t\t\t\\end{enumerate}\n\t\t\t\\end{theorem}","sha256_utf8":"f5862a82c0e9215b4976c00d6b385522b714346360451f382a48e874498ba4d5"}]},{"id":"S017","semantic_type":"proof","kind":"proof","title":"统一收敛定理完整证明","source_units":[{"start":97910,"end":107776,"start_line":2266,"end_line":2495,"raw_tex":"\\begin{proof}\n\t\t\t\tWe begin by remarking that boundedness of the sequence \\(\\seq{x_k}\\) is ensured by \\cref{cond:alphk,cond:alphk:nu}, as shown in \\cref{thm:PG:boundedness}.\n\t\t\t\tWe next prove each claim individually.\n\t\t\t\t\\begin{proofitemize}\n\t\t\t\t\\item \\ref{thm:recipe:rate}~\n\t\t\t\t\tConsider a convergent subsequence \\(x_{k_j}\\to x_\\infty\\), so that \\(x_{k_j+1}\\to x_\\infty\\) by \\cref{thm:PG:barPhi}.\n\t\t\t\t\tUp to further extracting if necessary we have that\n\t\t\t\t\t\\(\\alpha_{k_j+1}\\to\\alpha_\\infty\\geq\\alpha_{\\rm min}>0\\) and \\(\\sigma_{k_j}\\to0\\).\n\t\t\t\t\tObserve that\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\tx_{k+1}\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\argmin h({}\\cdot{};x_k,\\alpha_{k+1},\\sigk*),\n\t\t\t\t\t\\shortintertext{where}\n\t\t\t\t\t\th(w;x,\\alpha,\\sigma)\n\t\t\t\t\t\\coloneqq{} &\n\t\t\t\t\t\t(\\sigma \\g_1+\\g_2)(w)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{2\\alpha}\n\t\t\t\t\t\t\\bigl\\|w-x+\\alpha\\nabla\\bigl(\\sigma \\f_1+\\f_2\\bigr)(x)\\bigr\\|^2\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tis level bounded in \\(w\\) locally uniformly in \\((x,\\alpha,\\sigma)\\), as a function from \\(\\R^n\\times\\bigl(\\R^n\\times[\\alpha_{\\rm min},\\infty)\\times[0,\\infty)\\bigr)\\) to \\(\\Rinf\\).\n\t\t\t\t\tSince \\(h\\) is continuous in \\((x,\\alpha,\\sigma)\\), it follows from \\cite[Thm. 1.17]{rockafellar2011variational} that\n\t\t\t\t\t\\[\n\t\t\t\t\t\tx_\\infty\n\t\t\t\t\t\\in\n\t\t\t\t\t\t\\argmin h({}\\cdot{};x_\\infty,\\alpha_\\infty,0)\n\t\t\t\t\t=\n\t\t\t\t\t\t\\prox_{\\alpha_\\infty \\g_2}\\bigl(x_\\infty-\\alpha_\\infty\\nabla \\f_2(x_\\infty)\\bigr),\n\t\t\t\t\t\\]\n\t\t\t\t\tthis condition being equivalent to \\(x_\\infty\\in\\argmin(\\f_2+\\g_2)\\defeq\\X_2\\).\n\t\t\t\t\tWe next show that \\(\\bphik(x_k)\\to0\\).\n\n\t\t\t\t\tFor any \\(x_\\star\\in\\X_1\\), the subdifferential characterization of \\(x_{k+1}\\in\\prox_{\\alphk*\\gk*}(x_k-\\alphk*\\nabla\\fk*(x_k))\\)\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\tfrac{x_{k-1}-x_k}{\\alphk}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t(\\nabla\\fk(x_{k-1})-\\nabla\\fk(x_k))\n\t\t\t\t\t\\in\n\t\t\t\t\t\t\\partial\\phik(x_k)\n\t\t\t\t\t\\]\n\t\t\t\t\timplies that\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\phik(x_\\star)\n\t\t\t\t\t\\geq\n\t\t\t\t\t\t\\phik(x_k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\overbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{x_{k-1}-x_k}{\\alphk}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t(\\nabla\\fk(x_{k-1})-\\nabla\\fk(x_k))\n\t\t\t\t\t\t\t}^{\\to0 \\text{ by \\cref{thm:PG:barPhi}}}\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\\overbracket*[0.5pt]{\n\t\t\t\t\t\t\t\tx_\\star-x_k\n\t\t\t\t\t\t\t\t\\vphantom{\\tfrac{x_{k-1}-x_k}{\\alphk}}\n\t\t\t\t\t\t\t}^{\\text{\\clap{bounded}}}\n\t\t\t\t\t\t},\n\t\t\t\t\t\\]\n\t\t\t\t\thence that\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\costinf_2\n\t\t\t\t\t=\n\t\t\t\t\t\t\\lim_{k\\to\\infty}\\phik(x_\\star)\n\t\t\t\t\t\\geq\n\t\t\t\t\t\t\\limsup_{k\\to\\infty}\\phik(x_k)\n\t\t\t\t\t=\n\t\t\t\t\t\t\\limsup_{k\\to\\infty}\\left(\\sigk\\bcost_1(x_k)+\\overbracket*[0.5pt]{\\cost_2(x_k)}^{\\geq\\costinf_2}\\right)\n\t\t\t\t\t\\geq\n\t\t\t\t\t\t\\limsup_{k\\to\\infty}\\sigk\\bcost_1(x_k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\costinf_2.\n\t\t\t\t\t\\]\n\t\t\t\t\tHere, the second equality is obtained by adding and subtracting \\(\\sigk\\costinf_1\\) along with the fact that \\(\\sigk\\to0\\).\n\t\t\t\t\tSince \\(\\sigk\\bcost_1\\geq0\\), necessarily \\(\\sigk\\bcost_1(x_k)\\to0\\) and consequently \\(\\limsup_{k\\to\\infty}\\cost_2(x_k)=\\costinf_2\\).\n\t\t\t\t\tSimilarly, since \\(\\cost_2\\geq\\costinf_2\\) this necessarily implies that \\(\\cost_2(x_k)\\to\\costinf_2\\).\n\t\t\t\t\tThis shows that \\(\\bcost_2(x^k)\\to0\\) as claimed.\n\n\t\t\t\t\tWe next show that \\(\\seq{W_k}\\) too is vanishing.\n\t\t\t\t\tObserve that\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\tW_k\n\t\t\t\t\t\\defeq{} &\n\t\t\t\t\t\t\\tfrac{1-4\\alphk\\sigk\\lk^{(2)}\\Delk}{4}\n\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\\|x_k-x_{k-1}\\|^2\n\t\t\t\t\t\t}^{\\mathclap{\\to 0\\text{ by \\cref{thm:PG:barPhi}}}}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{\\sigk*}{\\sigk}\\alphk*\\rhok*\n\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\\bcost_2(x_{k-1})\n\t\t\t\t\t\t}^{\\mathclap{\\to 0}}\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\\sigk\\alphk\\Delk\\bcost_2(x_k)\n\t\t\t\t\t\t}_{\\to 0}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\alphk\n\t\t\t\t\t\t\\left(1+\\rhok\\right)\n\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\\sigk\n\t\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\t\\cost_1(x_{k-1})\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\costinf*\n\t\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\t}_{\\to0}\n\t\t\t\t\t\\to\n\t\t\t\t\t\t0\n\t\t\t\t\t\\quad\\text{as \\(k\\to\\infty\\)},\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\twhere we also used the fact that \\(\\seq{\\alphk}\\leq\\alpha_{\\rm max}\\) and \\(\\seq{\\rhok}\\leq\\rho_{\\rm max}\\) as in \\cref{thm:PG:rhomax}.\n\n\t\t\t\t\tThe asserted sublinear rate is a consequence of \\eqref{eq:sublin:res}, the characterization of the residual in \\cref{thm:H} (see \\eqref{eq:Hk:phi}), and the upper bounds for \\(\\lk\\), \\(\\Lk\\) in \\cref{thm:l<=L}:\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\textstyle\\sum_{k=0}^K\\dist^2(0, \\partial \\phik*(x_{k+1})\n\t\t\t\t\t\\leq{}&\n\t\t\t\t\t\t\\textstyle\\sum_{k=0}^K\\tfrac{1}{\\alphk*^2}\\|\\Hk*(x_{k})-\\Hk*(x_{k+1}) \\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\leq {}&\n\t\t\t\t\t\t\\tfrac{M_{\\rm max}^2}{\\alpha_{\\rm min}}\\textstyle\\sum_{k=0}^K \\tfrac{1}{\\alphk*}\\|x_{k}-x_{k+1}\\|^2\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\tfrac{M_{\\rm max}^2}{\\alpha_{\\rm min}(1-\\nu)}\\bar{\\originalvarphi}_0(x_{0}),\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\twhere \\(M_{\\rm max} = \\sup_{k\\in\\N}M_k\\leq 1+ \\alpha_{\\rm max}L_{f_0, \\V}\\), see \\cref{thm:lL}.\n\n\n\t\t\t\t\\item \\ref{thm:recipe:solution}~\n\t\t\t\t\tWe begin by observing that, since \\(\\bcost_2(x^k)\\to0\\), by lower semicontinuity all limit points of \\(\\seq{x^k}\\) belong to \\(\\X_2\\).\n\t\t\t\t\tAs done in the proof of \\cref{thm:PG:boundedness} we consider two possible cases.\n\t\t\t\t\t\\begin{proofitemize}[leftmargin=*,topsep=0pt]\n\t\t\t\t\t\\item\n\t\t\t\t\t\t{\\it Case 1: \\(\\cost_1(x_k)\\geq\\costinf*\\) holds for \\(k\\) large enough.}\n\n\t\t\t\t\t\tIn this case, we will actually show that the sequence \\(\\seq{x_k}\\) converges to a solution of \\eqref{eq:P}.\n\t\t\t\t\t\tWe first argue that there exists an optimal limit point; to this end, since all limit points are feasible and because of lower semicontinuity, it suffices to show that \\(\\liminf_{k\\to\\infty}\\cost_1(x_k)=\\costinf*\\).\n\t\t\t\t\t\tA telescoping argument on \\cref{thm:PG:qSD} along with \\cref{cond:alphamin} yields\n\t\t\t\t\t\t\\begin{equation}\n\t\t\t\t\t\t\t\\alpha_{\\rm min}\\sum_{k\\in \\N}\\sigk(1+\\rhok-\\rhok*^2)\\bigl(\\cost_1(x_{k-1})-\\costinf*\\bigr)\n\t\t\t\t\t\t\\leq\n\t\t\t\t\t\t\t\\sum_{k\\in \\N}\\sigk\\alphk(1+\\rhok-\\rhok*^2)\\bigl(\\cost_1(x_{k-1})-\\costinf*\\bigr)\n\t\t\t\t\t\t<\n\t\t\t\t\t\t\t\\infty.\n\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\tSince \\(\\sum_{k\\in\\N}\\sigk=\\infty\\), necessarily\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\liminf_{k\\to\\infty}(1+\\rhok-\\rhok*^2)(\\cost_1(x_{k-1})-\\costinf*)\n\t\t\t\t\t\t=\n\t\t\t\t\t\t\t0.\n\t\t\t\t\t\t\t\\numberthis\\label{eq:liminf:cost1}\n\t\t\t\t\t\t\\end{align*}\n\t\t\t% \t\t\tObserve first that\n\t\t\t% \t\t\t\\begin{align*}\n\t\t\t% \t\t\t\t\\rhok*\n\t\t\t% \t\t\t=\n\t\t\t% \t\t\t\t\\tfrac{\\sigk*\\alphk*}{\\sigk\\alphk}\n\t\t\t% \t\t\t\\leq\n\t\t\t% \t\t\t\t\\tfrac{\\sigk*\\balphk*}{\\sigk\\alphk}\n\t\t\t% \t\t\t\\leq\n\t\t\t% \t\t\t\t\\tfrac{\\sigk*}{\\sigk}\n\t\t\t% \t\t\t\t\\sqrt{\\tfrac{\\sigk}{\\sigma_{k-1}}\\bigl(1+\\rhok\\bigr)}\n\t\t\t% \t\t\t\t\\tfrac{\\sigk}{\\sigk*}\n\t\t\t% \t\t\t\\leq\n\t\t\t% \t\t\t\t\\sqrt{1+\\rhok}\n\t\t\t% \t\t\t,\n\t\t\t% \t\t\t\\numberthis\\label{eq:rhok:plus}\n\t\t\t% \t\t\t\\end{align*}\n\t\t\t% \t\t\twhere the first inequality holds by \\cref{cond:alphk}, the second is due to the first term in \\eqref{eq:adabim:alphk*_0}, and the last inequality follows from \\eqref{eq:sigk}.\n\t\t\t\t\t\tNotice that \\(\\limsup_{k\\to\\infty}(1+\\rhok-\\rhok*^2)>0\\), for otherwise \\(1+\\rhok-\\rhok*^2 \\to 0\\), implying that \\(\\liminf_{k \\to \\infty} \\rhok >0\\), and consequently that \\(\\alphk\\sigk\\) eventually increases exponentially, which contradicts \\(\\sigk\\alphk \\leq \\sigk\\alpha_{\\rm max} \\to 0\\).\n\t\t\t\t\t\tHence, \\(\\limsup_{k\\to\\infty}(1+\\rhok-\\rhok*^2)>0\\), which along with \\eqref{eq:liminf:cost1} implies that \\(\\liminf_{k\\to\\infty}\\cost_1(x_{k-1})=\\costinf*\\).\n\t\t\t\t\t\tTherefore, an optimal limit point exists, be it \\(x_\\infty\\).\n\t\t\t\t\t\tSince \\(W_k\\to0\\) by assertion \\ref{thm:recipe:rate} and since \\(\\mathcal L_k(x_\\infty)\\) converges, it follows that \\(\\frac12\\|x_k-x_\\infty\\|^2 = \\mathcal L_k(x_\\infty) - W_k\\) converges as well.\n\t\t\t\t\t\tSince along a subsequence \\(\\frac12\\|x_k-x_\\infty\\|^2\\) converges to zero, necessarily \\(\\lim_{k\\to\\infty}\\frac12\\|x_k-x_\\infty\\|^2=0\\), proving that the entire sequence \\(\\seq{x_k}\\) converges to \\(x_\\infty\\).\n\t\t\t\t\t\\item\n\t\t\t\t\t\t{\\it Case 2: \\(\\cost_1(x_k)<\\costinf*\\) holds infinitely often.}\n\n\t\t\t\t\t\tRecall the index \\(i_k\\) and the set \\(K'\\) defined in \\eqref{eq:ik} and \\eqref{eq:K'}.\n\t\t\t% \t\t\tOnce we establish that all the limit points of \\(\\seq{x_{i_k}}\\) and \\(\\seq{x_{i_k+1}}\\) are optimal, {\\red it only remains to show that\n\t\t\t% \t\t\t\\(\n\t\t\t% \t\t\t\t\\dist(x_{k'},\\X_1)\n\t\t\t% \t\t\t\\to\n\t\t\t% \t\t\t\t0\n\t\t\t% \t\t\t\\)\n\t\t\t% \t\t\tas \\(K'\\ni k'\\to\\infty\\)\\note{Why}}.\n\t\t\t\t\t\tHaving established boundedness of the entire sequence, limit points of \\(\\seq{x_{i_k}}\\) exist and, as shown above, belong to \\(\\X_2\\).\n\t\t\t\t\t\tMoreover, it follows from \\eqref{eq:ik} that \\(\\limsup_{k\\to\\infty}\\cost_1(x_{i_k})\\leq\\costinf*\\), and a lower semicontinuity argument then yields that all the limit points of \\(\\seq{x_{i_k}}\\) attain the optimal cost and are therefore optimal.\n\t\t\t\t\t\tSince \\(x_{i_k}-x_{i_k+1}\\to0\\) by \\cref{thm:PG:barPhi}, the same is also true for \\(\\seq{x_{i_k+1}}\\), and in particular\n\t\t\t\t\t\t\\(\\dist(x_{i_{k}+1},\\X_1) \\to 0\\) as \\(k \\to \\infty\\).\n\n\t\t\t\t\t\tFor each \\(k\\) let \\(\\bar x_k\\coloneqq\\proj_{\\X_1}\\!x_k\\), which is well defined since \\(\\X_1\\neq\\emptyset\\) is closed and convex.\n\t\t\t\t\t\tRecalling that \\(\\mathcal L_k(x_\\star)=W_k + \\tfrac12\\|x_k-x_\\star\\|^2\\), for \\(k'\\in K'\\) we have\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\tfrac12\\dist(x_{k'},\\X_1)^2\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\tfrac12\\|x_{k'}-\\bar x_{i_{k'}+1}\\|^2\n\t\t\t\t\t\t={}\n\t\t\t\t\t\t\t\\mathcal L_{k'}(\\bar x_{i_{k'}+1})-W_{k'}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\\dueto{\\scriptsize by \\eqref{eq:Lk'descent}}\n\t\t\t\t\t\t\t\\\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\mathcal L_{i_{k'}+2}(\\bar x_{i_{k'}+1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\tW_{k'}\n\t\t\t\t\t\t={}\n\t\t\t\t\t\t\t% \\tfrac12\\dist(x_{i_{k'}+2},\\X_1)^2\n\t\t\t\t\t\t\t\\tfrac12\\|x_{i_{k'}+2} - \\bar x_{i_{k'}+1}\\|^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\tW_{i_{k'}+2}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\tW_{k'}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\tfrac12\\big(\\dist(x_{i_{k'}+1},\\X_1) + \\|x_{i_{k'}+1} - x_{i_{k'}+2}\\|\\big)^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\tW_{i_{k'}+2}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\tW_{k'}\n\t\t\t\t\t\t\\to\n\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\tas \\(K'\\ni k'\\to\\infty\\), where the limit follows from \\cref{thm:PG:barPhi}, the vanishing of \\(\\seq{W_k}\\) established in assertion \\ref{thm:recipe:rate}, and the fact that \\(i_k\\to\\infty\\) as \\(k\\to\\infty\\).\n\t\t\t\t\t\tSince \\(K'\\cup\\set{i_k,i_k+1}[k\\in\\N]=\\N\\), we conclude that all the limit points of \\(\\seq{x^k}\\) are optimal.\n\t\t\t\t\t\\qedhere\n\t\t\t\t\t\\end{proofitemize}\n\t\t\t\t\\end{proofitemize}\n\t\t\t\\end{proof}","sha256_utf8":"1ec28c371435769c142616a64a34eb36247e10526ec60cb785f6d6f55c333bef"}]},{"id":"S018","semantic_type":"lemma","kind":"lemma","title":"李雅普诺夫拟下降引理","source_units":[{"start":72279,"end":73874,"start_line":1441,"end_line":1507,"raw_tex":"\\begin{lemma}[quasi-descent inequality]\\label{thm:LW}%\n\t\t\t\tSuppose that \\cref{ass:basic} holds and consider iterations \\eqref{eq:PG} with \\(0<\\sigk*\\leq\\sigk\\).\n\t\t\t\tLet \\(\\bcost_2\\coloneqq\\cost_2-\\costinf_2\\geq0\\) be as in \\cref{rem:cost}, and define \\(\\Delk\\coloneqq\\frac{1}{\\sigk}-\\frac{1}{\\sigma_{k-1}}\\), \\(\\rhok*\\coloneqq\\frac{\\sigk*\\alphk*}{\\sigk\\alphk}\\), and\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\mathcal L_k(x_\\star)\n\t\t\t\t\\coloneqq{} &\n\t\t\t\t\t\\tfrac12\\|x_k-x_\\star\\|^2\n\t\t\t\t\t+\n\t\t\t\t\tW_k\n\t\t\t\t\\shortintertext{with}\n\t\t\t\t\tW_k\n\t\t\t\t\\coloneqq{} &\n\t\t\t\t\t\\tfrac{1-4\\alphk\\sigk\\lk^{(2)}\\Delk}{4}\\|x_k-x_{k-1}\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\sigk*}{\\sigk}\\alphk*\\rhok*\n\t\t\t\t\t\\bcost_2(x_{k-1})\n\t\t\t\t\\\\\n\t\t\t\t&\n\t\t\t\t\t+\n\t\t\t\t\t\\sigk\\alphk\\Delk\\bcost_2(x_k)\n\t\t\t\t\t+\n\t\t\t\t\t\\sigk\\alphk\\left(1+\\rhok\\right)\n\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\\cost_1(x_{k-1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\costinf*\n\t\t\t\t\t\\bigr).\n\t\t\t\t\\end{align*}\n\t\t\t\tThen, for every \\(k\\in\\N\\) and \\(x_\\star\\in\\X_1\\) it holds that\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\mathcal L_{k+1}(x_\\star)\n\t\t\t\t\t-\n\t\t\t\t\t\\mathcal L_k(x_\\star)\n\t\t\t\t\\leq{} &\n\t\t\t\t\t-\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t\\tfrac{1}{4}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\rhok*^2(\\alphk\\lk-\\alphk^2\\Lk^2)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\sigk\\alphk\\lk^{(2)}\\Delk\n\t\t\t\t\t\\right)\n\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\\\\\n\t\t\t\t&\n\t\t\t\t\t-\n\t\t\t\t\t\\sigk\\alphk\n\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t1+\\rhok-\\rhok*^2\n\t\t\t\t\t\\bigr)\n\t\t\t\t\t\\bigl(\\cost_1(x_{k-1})-\\costinf*\\bigr)\n\t\t\t\t\\\\\n\t\t\t\t&\n\t\t\t\t\t-\n\t\t\t\t\t\\sigk*\\alphk*\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t\\tfrac{\\Delk}{\\rhok*}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{\\sigk}\\left(\n\t\t\t\t\t\t\t1+\\rhok*\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\rho_{k+2}^2\\tfrac{\\sigk}{\\sigk*}\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\\right)\n\t\t\t\t\t\\bcost_2(x_k).\n\t\t\t\t\\numberthis\\label{eq:puya:muk:descent}\n\t\t\t\t\\end{align*}\n","sha256_utf8":"e9e246976e47f6eea0f1b6e6fc99fd17e7f31368283239f411d8a6580a4edbc1"},{"start":83225,"end":83240,"start_line":1898,"end_line":1899,"raw_tex":"\n\t\t\t\\end{lemma}","sha256_utf8":"89722f535bb9a4a7116c0f15610164ec8ca7063e5e49cf48287165d918fc35e7"}]},{"id":"S019","semantic_type":"proof","kind":"proof","title":"拟下降引理完整证明","source_units":[{"start":73878,"end":83225,"start_line":1508,"end_line":1898,"raw_tex":"\\begin{proof}\n\t\t\t\tLet\n\t\t\t\t\\begin{equation}\n\t\t\t\t\tp_k%\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\phik(x_k)-\\phik(x_\\star)\n\t\t\t\t=\n\t\t\t\t\t\\sigk\\bigl(\\cost_1(x_k)-\\costinf*\\bigr)\n\t\t\t\t\t+\n\t\t\t\t\t\\bcost_2(x_k)\n\t\t\t\t\\end{equation}\n\t\t\t\t(which is not necessarily positive).\n\t\t\t\tWe will prove the claim using the uppercase notation of \\eqref{eq:uppercase} and, consistently, set\n\t\t\t\t\t\\(P_k%\n\t\t\t\t\t\\coloneqq{\\tfrac{1}{\\sigk}}p_k%\n\t\t\t\t\t=\\Phik(x_k)-\\Phik(x_\\star)\n\t\t\t\t\t\\).\n\t\t\t\t\tBeing a simple matter of multiplicative constants, note that proximal gradient iterations \\eqref{eq:PG} can equivalently be expressed in the penalized cost reformulation \\(\\Phik*=\\Fk*+\\Gk*\\) up to suitable scaling of the stepsize, namely,\n\t\t\t\t\t\\[\n\t\t\t\t\t\tx_{k+1}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\prox_{\\sigk*\\alphk*\\Gk*}(x_k-\\sigk*\\alphk*\\nabla\\Fk*(x_k)).\n\t\t\t\t\t\\]\n\t\t\t\t\tThe subgradient characterization of the proximal mapping as in \\eqref{eq:Hk:phi} yields\n\t\t\t\t\t\\begin{equation}\\label{eq:subgrad}\n\t\t\t\t\t\t\\tfrac{\\Hk(x_{k-1})-x_k}{\\sigk\\alphk}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\tfrac{x_{k-1}-x_k}{\\sigk\\alphk}-\\nabla\\Fk(x_{k-1})\n\t\t\t\t\t\\in\n\t\t\t\t\t\t\\partial\\Gk(x_k),\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\twhere we remind that \\(\\Hk=\\id-\\alphk\\nabla\\fk\\), cf. \\eqref{eq:Hk:phi}.\n\t\t\t\t\tHence, since \\(\\partial\\Phik=\\nabla\\Fk+\\partial\\Gk\\),\n\t\t\t\t\t\\begin{subequations}\n\t\t\t\t\t\t\\begin{align}\n\t\t\t\t\t\t\\nonumber\n\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\Phik(x_{k-1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Phik(x_k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\\innprod{x_{k-1}-x_k}{x_{k-1}-x_k}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk(x_{k-1})-\\nabla\\Fk(x_k)}{x_{k-1}-x_k}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\nonumber\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\Phik(x_{k-1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Phik(x_k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac1{\\sigk}\\innprod{\\nabla\\fk(x_{k-1})-\\nabla\\fk(x_k)}{x_{k-1}-x_k}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\Phik(x_{k-1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Phik(x_k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1-\\alphk\\lk}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t\t\\label{eq:phikdescent}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\tP_{k-1}%\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\tP_k%\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t(\\tfrac{1}{\\sigk}-\\tfrac{1}{\\sigma_{k-1}})\\bcost_2(x_{k-1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1-\\alphk\\lk}{\\sigk\\alphk}\\|x_{k-1}-x_k\\|^2.\n\t\t\t\t\t\t\\label{eq:ineqsubgrad}\n\t\t\t\t\t\t\\end{align}\n\t\t\t\t\t\\end{subequations}\n\t\t\t\t\tAgain from \\eqref{eq:subgrad}, this time with \\(k\\gets k+1\\), we have\n\t\t\t\t\t\\begin{subequations}\\label{subeq:3ineq}\n\t\t\t\t\t\t{\\mathtight[0.6]%\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\Gk*(x_\\star)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Gk*(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_\\star-x_{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\innprod{x_k-x_{k+1}}{x_\\star-x_{k+1}}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\Gk*(x_\\star)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Gk*(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_\\star-x_{k+1}}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_\\star\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_\\star-x_{k+1}\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t&\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\Gk*(x_\\star)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Gk*(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_\\star-x_k}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_\\star\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t&\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_\\star-x_{k+1}\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\Gk*(x_\\star)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Gk*(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\fillwidthof[c]{\n\t\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_\\star-x_k}\n\t\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\t\\Fk*(x_\\star)-\\Fk*(x_k)\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\underbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t}_{\\text{(A)}}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_\\star\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t&\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_\\star-x_{k+1}\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_{k+1}\\|^2,\n\t\t\t\t\t\t\\numberthis\\label{eq:ineq1}\n\t\t\t\t\t\t\\end{align*}}%\n\t\t\t\t\t\twhere the last inequality uses convexity of \\(\\Fk*\\).\n\t\t\t\t\t\tAs to term (A), we have\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\text{(A)}\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\\innprod{\\Hk(x_{k-1})-x_k}{x_{k+1}-x_k}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\\innprod{\\Hk(x_{k-1})-x_k+\\sigk\\alphk\\nabla\\Fk*(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:subgrad}}{} &\n\t\t\t\t\t\t\t\\fillwidthof[c]{\n\t\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\t\\innprod{\\Hk(x_{k-1})-x_k}{x_{k+1}-x_k}\n\t\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\t\\Gk(x_{k+1})-\\Gk(x_k)\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)-\\nabla\\Fk(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t&\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\t\\innprod{\\Hk(x_{k-1})-\\Hk(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t}_{\\text{(B)}}.\n\t\t\t\t\t\t\\numberthis\\label{eq:ineqA}\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\tNext, we bound the term (B) by \\(\\epsk*\\)-Young's inequality as\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\text{(B)}\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\tfrac{\\epsk*}{2\\sigk\\alphk}\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\epsk*\\sigk\\alphk}\\|\\Hk(x_{k-1})-\\Hk(x_k)\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\overrel{\\ref{thm:H}}{} &\n\t\t\t\t\t\t\t\\tfrac{\\epsk*}{2\\sigk\\alphk}\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{\\Mk^2}{2\\epsk*\\sigk\\alphk}\\|x_{k-1}-x_k\\|^2.\n\t\t\t\t\t\t\\numberthis\\label{eq:ineqB}\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\\end{subequations}\n\t\t\t\t\tCombining the three inequalities \\eqref{subeq:3ineq} yields\n\t\t\t\t\t{\\mathtight[0.9]%\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t0\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\Phik*(x_\\star)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\Gk*(x_{k+1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\Fk*(x_k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\Gk(x_{k+1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\Gk(x_k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t(\\tfrac1{\\sigk*}-\\tfrac{1}{\\sigk})\n\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\Biggl\\{\n\t\t\t\t\t\t\t\\tfrac{\\Mk^2}{2\\epsk*\\sigk\\alphk}\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\tfrac{\\epsk*}{2\\sigk\\alphk}\n\t\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\t\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_\\star-x_{k+1}\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t\\hphantom{+\\Biggl\\{{}}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_\\star\\|^2\n\t\t\t\t\t\t\\Biggr\\}\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\overrel{\\ref{thm:Phi*}}{} &\n\t\t\t\t\t\t\\costinf*\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac1{\\sigk*}\\costinf_2\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\Fk(x_k)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\Gk(x_k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\bigl(\\tfrac1{\\sigk*}-\\tfrac{1}{\\sigk}\\bigr)\n\t\t\t\t\t\t\\vphantom{\\f^2}\n\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\f_2(x_k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\g_2(x_{k+1})\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\{\\cdots\\}\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t-P_k%\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\t\\tfrac1{\\sigk*}-\\tfrac{1}{\\sigk}\n\t\t\t\t\t\t\t\t\\vphantom{\\f^2}\n\t\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\\,\n\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\f_2(x_k)\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\g_2(x_{k+1})\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\costinf_2\n\t\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t}^{\\text{(D)}}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\Bigl\\{\\cdots\\Bigr\\}.\n\t\t\t\t\t\\end{align*}}%\n\t\t\t\t\tBy using convexity of \\(\\f_2\\) we can bound the term (D) as\n\t\t\t\t\t{\\mathtight[0.75]%\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\text{(D)}\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_{k+1})}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\f_2(x_k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\g_2(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_{k+1})-\\nabla \\f_2(x_k)}{x_{k+1}-x_k}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\costinf_2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\fillwidthof[c]{\n\t\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_{k+1})}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\f_2(x_k)\n\t\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\t-\\f_2(x_{k+1})\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\g_2(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\fillwidthof[c]{\n\t\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_{k+1})-\\nabla \\f_2(x_k)}{x_{k+1}-x_k}\n\t\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\t\\lk*^{(2)}\\|x_{k+1}-x_k\\|^2\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\costinf_2,\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t}%\n\t\t\t\t\twhich plugged in the previous inequality results in\n\t\t\t\t\t{\\mathtight[0.75]%\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t0\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t-P_k%\n\t\t\t\t\t\t-\n\t\t\t\t\t\t(\\tfrac1{\\sigk*}-\\tfrac{1}{\\sigk})\\bcost_2(x_{k+1})\n\t\t\t\t\t\t+\n\t\t\t\t\t\t(\\tfrac1{\\sigk*}-\\tfrac{1}{\\sigk})\n\t\t\t\t\t\t\\lk*^{(2)}\\|x_{k+1}-x_k\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{\\Mk^2}{2\\epsk*\\sigk\\alphk}\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{\\epsk*}{2\\sigk\\alphk}\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\\|x_\\star-x_{k+1}\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\\|x_k-x_\\star\\|^2.\n\t\t\t\t\t\\numberthis\\label{eq:3ineqs}\n\t\t\t\t\t\\end{align*}}%\n\t\t\t\t\tSumming \\eqref{eq:3ineqs}\\(+\\betk*\\)\\eqref{eq:ineqsubgrad}, multiplying by \\(\\sigk*\\alphk*\\), and rearranging yields that for every \\(\\betk*\\geq0\\) and \\(\\epsk*>0\\) the following hold.\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t& \\tfrac{1}{2}\n\t\t\t\t\t\\|x_{k+1}-x_\\star\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\sigk*}{\\sigk}\\alphk*(1+\\betk*)p_k%\n\t\t\t\t\t+\n\t\t\t\t\t\\alphk*\\sigk*\\Delk*\\bcost_2(x_{k+1})\n\t\t\t\t\\\\\n\t\t\t\t\t+ {}&\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t\\tfrac{1-\\rhok*\\epsk*}{2}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\alphk*\\sigk*\\Delk*%\n\t\t\t\t\t\t\\lk*^{(2)}\n\t\t\t\t\t\\right)\n\t\t\t\t\t\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\\leq\n\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t\\|x_k-x_\\star\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\sigk*}{\\sigma_{k-1}}\\betk*\\alphk*p_{k-1}%\n\t\t\t\t\\\\\n\t\t\t\t\t-{}&\n\t\t\t\t\t\\rhok*\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t\\betk*(1-\\alphk\\lk)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{\\Mk^2}{2\\epsk*}\n\t\t\t\t\t\\right)\n\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\alphk*\\sigk*\\betk*\\Delk\\bcost_2(x_{k-1}).\n\t\t\t\t\\numberthis\n\t\t\t\t\\end{align*}\n\t\t\t\tBy selecting \\(\\betk=\\rhok\\) and \\(\\epsk\\coloneqq\\nicefrac{1}{2\\rhok}\\) the inequality reduces to the claimed quasi-descent in terms of \\(\\mathcal L_{k}(x^\\star)\\).\n\t\t\t\t\\end{proof}","sha256_utf8":"2dec07737832a07ab7d6c25cce01e02c7d0460c224d7c77ca35f6a38465a8dbb"}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","title":"下降性与迭代有界引理","source_units":[{"start":86462,"end":88028,"start_line":1951,"end_line":2003,"raw_tex":"\\begin{lemma}\\label{thm:PG}%\n\t\t\t\tSuppose that \\cref{ass:basic} holds, and consider proximal gradient iterations \\eqref{eq:PG} with \\(\\seq{\\alphk}\\) and \\(\\seq{\\sigk}\\) complying with \\cref{cond:alphk,cond:alphk:nu,cond:monotone}.\n\t\t\t\tThen, the following hold:\n\t\t\t\t\\begin{enumerate}\n\t\t\t\t\\item \\label{thm:PG:rhomax}%\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\rhok\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\frac{\\sigk\\alphk}{\\sigma_{k-1}\\alpha_{k-1}}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\rho_{\\rm max}\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\max\\set{\n\t\t\t\t\t\t\t\\frac{\\alpha_0}{\\alpha_{-1}},\\,\n\t\t\t\t\t\t\t\\frac{1+\\sqrt5}{2}\n\t\t\t\t\t\t}\n\t\t\t\t\t\\)\n\t\t\t\t\tfor every \\(k\\in\\N\\).\n\n\t\t\t\t\\item \\label{thm:PG:qSD}%\n\t\t\t\t\tFor \\(\\mathcal L_k\\) as in \\cref{thm:LW} it holds that\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\mathcal L_{k+1}(x_\\star)\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\mathcal L_k(x_\\star)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\sigk\\alphk(1+ \\rhok -\\rhok*^2)\n\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\\cost_1(x_{k-1})-\\costinf*\n\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\\)\n\t\t\t\t\tfor all \\(k\\in\\N\\) and \\(x_\\star\\in\\X_1\\).\n\n\t\t\t\t\\item \\label{thm:PG:barPhi}%\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\bphik*(x_{k+1})\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\bphik(x_k)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{1-\\nu}{\\alphk*}\\|x_{k+1}-x_k\\|^2\n\t\t\t\t\t\\)\n\t\t\t\t\tholds for every \\(k\\in\\N\\).\n\t\t\t\t\tIn particular, \\(\\|x_k-x_{k-1}\\|\\to0\\) as \\(k\\to \\infty\\), \\(\\seq{\\bphik(x_k)}\\) is convergent, the following  worst-case rate holds\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\min_{k\\leq K} \\|x_{k+1} - x_k\\|^2 \\leq\n\t\t\t\t\t\t\\tfrac{\\alpha_{\\rm max} \\bar{\\originalvarphi}_0(x_{0})}{(1-\\nu)(1+K)},\n\t\t\t\t\t\\]\n\t\t\t\t\tand both \\(\\seq{\\sigk\\cost_1(x_k)}\\) and \\(\\seq{\\cost_2(x_k)}\\) are bounded.\n\n\t\t\t\t\\item \\label{thm:PG:boundedness}%\n\t\t\t\t\tThe sequence \\(\\seq{x^k}\\) is bounded.\n\t\t\t\t\\end{enumerate}\n\t\t\t\\end{lemma}","sha256_utf8":"cb8e5607cd3bdddc36f57f93af97a042e427f7081807ce51ef94f122e3bfa25e"}]},{"id":"S021","semantic_type":"proof","kind":"proof","title":"下降与有界引理完整证明","source_units":[{"start":88032,"end":96787,"start_line":2004,"end_line":2246,"raw_tex":"\\begin{proof}~\n\t\t\t\t\\begin{proofitemize}\n\t\t\t\t\\item \\ref{thm:PG:rhomax}~\n\t\t\t\t\tObserving that\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\rhok*\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\sqrt{\\tfrac{\\sigk}{\\sigma_{k-1}}\\bigl(1+\\rhok\\bigr)}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\sqrt{1+\\rhok}\n\t\t\t\t\t\\)\n\t\t\t\t\tby \\cref{cond:alphk}, the assertion follows from a trivial induction argument.\n\n\t\t\t\t\\item \\ref{thm:PG:qSD}~\n\t\t\t\t\tSince \\(\\alphk\\leq\\balphk\\), from the definition of \\(\\balphk\\) in \\eqref{eq:adabim:alphk*_0} it follows that the multiplying coefficients in \\eqref{eq:puya:muk:descent} are positive:\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\mathcal L_{k+1}(x_\\star) - \\mathcal L_k(x_\\star)\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\sigk*\\alphk*\n\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\\overbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{\\Delk}{\\rhok*}\n\t\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk}\n\t\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\t\\overbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\t1+\\rhok*\n\t\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\t\\rho_{k+2}^2\\tfrac{\\sigk}{\\sigk*}\n\t\t\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\\bcost_2(x_k)\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\sigk\\alphk\n\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\t1+\\rhok-\\rhok*^2\n\t\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\t\\bigl(\\cost_1(x_{k-1})-\\costinf*\\bigr)\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\\underbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{1}{4}\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\rhok*^2(\\alphk\\lk-\\alphk^2\\Lk^2)\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\alphk\\sigk\\lk^{(2)}\\Delk\n\t\t\t\t\t\t\t}_{\\geq0}\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\\|x_{k-1}-x_k\\|^2.\n\t\t\t\t\t\t\\vphantom{\n\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{1}{4}\n\t\t\t\t\t\t\t}_0\n\t\t\t\t\t\t}\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tThe claim then readily follows from the fact that \\(\\bcost_2\\geq0\\), cf. \\cref{thm:bar>=0}.\n\n\t\t\t\t\\item \\ref{thm:PG:barPhi}~\n\t\t\t\t\tWe have\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\bphik*(x_{k+1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\bphik(x_k)\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\bphik*(x_{k+1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\bphik*(x_k)\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\phik*(x_{k+1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\phik*(x_k)\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\sigk*\\left(\n\t\t\t\t\t\t\t\\Phik*(x_{k+1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Phik*(x_k)\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\\overrel*[\\leq]{\\eqref{eq:phikdescent}}\n\t\t\t\t\t\t- \\tfrac{1-\\nu}{\\alphk*}\n\t\t\t\t\t\t\\|x_{k+1}-x_k\\|^2.\n\t\t\t\t\t\t\\numberthis \\label{eq:descent:bphik}\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tHere, the first inequality uses the fact that \\(\\sigk\\geq\\sigk*\\), and therefore \\(\\bphik=\\sigk\\bcost_1+\\bcost_2\\) is smaller than \\(\\bphik*=\\sigk*\\bcost_1+\\bcost_2\\) on \\(\\dom\\cost_2\\), cf. \\cref{thm:bar>=0}, and \\(x_k\\in\\dom\\gk=\\dom\\cost_1\\cap\\dom\\cost_2\\subseteq\\dom\\cost_2\\);\n\t\t\t\t\tthe second inequality follows from the fact that \\(\\alphk*\\lk*\\leq\\nu\\).\n\t\t\t\t\tThis shows the sought inequality.\n\t\t\t\t\tIn turn, since \\(\\nu<1\\), the sequence \\(\\seq{\\bphik(x_k)}\\) is decreasing; since \\(x_k\\in\\dom\\cost_2\\), it follows from \\cref{thm:bar>=0} that \\(\\bphik(x_k)=\\sigk\\bcost_1(x^k)+\\bcost_2(x^k)\\geq0\\), hence that it is convergent and that the positive-valued sequences \\(\\seq{\\sigk\\bcost_1(x^k)}\\) and \\(\\seq{\\bcost_2(x^k)}\\) are bounded.\n\n\t\t\t\t\tThe convergence rate result follows immediately by telescoping \\eqref{eq:descent:bphik}\n\t\t\t\t\t\\begin{equation}\\label{eq:sublin:res}\n\t\t\t\t\t\t\\tfrac{(1-\\nu)(K+1)}{\\alpha_{\\rm max}}\\min_{k\\leq K}\\|x_{k+1}-x_k\\|^2\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\sum_{k=0}^{K}\\tfrac{1-\\nu}{\\alphk*}\\|x_{k+1}-x_k\\|^2\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\bar{\\originalvarphi}_0(x_{0}),\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\twhere \\cref{cond:alphk} was used to bound \\(\\alphk*\\leq\\alpha_{\\rm max}\\).\n\n\t\t\t\t\\item \\ref{thm:PG:boundedness}~\n\t\t\t\t\tWe pattern the proof structure of \\cite[Thm. 3.2]{solodov2007explicit}, thereby considering two mutually exclusive cases.\n\t\t\t\t\t\\begin{proofitemize}[leftmargin=*,topsep=0pt]\n\t\t\t\t\t\\item\n\t\t\t\t\t\t{\\it Case 1: \\(\\cost_1(x_k)\\geq\\costinf*\\) holds for \\(k\\) large enough.}\n\n\t\t\t\t\t\tRecall the definition of \\(W_k\\) and \\(\\mathcal L_k\\) in \\cref{thm:LW}.\n\t\t\t\t\t\tObserving that \\(W_k\\geq0\\) and \\(\\frac12\\|x_k-x_\\star\\|^2\\leq\\mathcal L_k(x_\\star)\\), \\cref{thm:PG:qSD} implies that \\(\\seq{\\mathcal L_k(x_\\star)}\\) converges and that consequently \\(\\seq{x_k}\\) is bounded.\n\n\t\t\t\t\t\\item\n\t\t\t\t\t\t{\\it Case 2: \\(\\cost_1(x_k)<\\costinf*\\) holds infinitely often.}\n\n\t\t\t\t\t\tIn this case, for every \\(k\\) large enough the index\n\t\t\t\t\t\t\\begin{equation}\\label{eq:ik}\n\t\t\t\t\t\t\ti_k\n\t\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\t\\max\\set{i\\leq k}[\\cost_1(x_i)<\\costinf*]\n\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\tis well defined.\n\t\t\t\t\t\tWe proceed by intermediate claims.\n\t\t\t\t\t\t\\renewcommand{\\claimscounter}{\\ref*{thm:PG}}%\n\t\t\t\t\t\t\\begin{claims}\n\t\t\t\t\t\t\\item\\label{claim:convergence:ik}%\n\t\t\t\t\t\t\t{\\it the sequences \\(\\seq{x_{i_k}}\\) and \\(\\seq{x_{i_k+1}}\\) are bounded}.\n\n\t\t\t\t\t\t\tThe optimal set\n\t\t\t\t\t\t\t\\[\n\t\t\t\t\t\t\t\t\\X_1\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\set{x\\in\\X_2}[\\cost_1(x)\\leq \\costinf*]\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\set{x}[\\cost_1(x)\\leq\\costinf*,\\ \\bcost_2(x)\\leq0]\n\t\t\t\t\t\t\t\\]\n\t\t\t\t\t\t\tcoincides with a sublevel set of the convex function \\(h\\coloneqq\\max\\set{\\cost_1-\\costinf*,\\bcost_2}\\).\n\t\t\t\t\t\t\tSince it is nonempty and bounded by \\cref{ass:X1}, \\(h\\) is level bounded; see, e.g., \\cite[Prop. 11.13]{bauschke2017convex} or \\cite[Lem. 1]{themelis2019acceleration}.\n\t\t\t\t\t\t\tNote that \\cref{thm:PG:barPhi} implies that \\(\\seq{\\bcost_2(x_k)}\\) is (upper) bounded, which combined with the fact that \\(\\cost_1(x_{i_k})<\\costinf*\\) implies that \\(\\seq{x_{i_k}}\\) lies in a sublevel set of \\(h\\), and is therefore bounded.\n\t\t\t\t\t\t\tIn turn, \\cref{thm:PG:barPhi} implies that so is \\(\\seq{x_{i_k+1}}\\).\n\t\t\t\t\t\t\\item\n\t\t\t\t\t\t\t{\\it the whole sequence \\(\\seq{x_k}\\) is bounded}.\n\n\t\t\t\t\t\t\tTo this end, it remains to show that \\(\\seq{x_{k'}}[k'\\in K']\\) is bounded, where\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:K'}\n\t\t\t\t\t\t\t\tK'\\coloneqq\\set{k'\\in\\N}[k'\\geq i_{k'}+2].\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tWith \\(\\rho_{\\rm max}\\) as in \\cref{thm:PG:rhomax}, for every \\(k\\in\\N\\)\n\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\\mathcal L_{i_k+1}(x_\\star)\n\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\tfrac12\n\t\t\t\t\t\t\t\t\t\\|x_{i_k+1}-x_\\star\\|^2\n\t\t\t\t\t\t\t\t\t\\vphantom{\\left(\\tfrac{\\sigma_{i_k+1}}{\\sigma_{i_k}}\\right)}\n\t\t\t\t\t\t\t\t}^{\\text{bounded by \\cref{claim:convergence:ik}}}\n\t\t\t\t\t\t\t\t+~~\n\t\t\t\t\t\t\t\t\\overbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\tfrac{1}{4}\\|x_{i_k+1}-x_{i_k}\\|^2\n\t\t\t\t\t\t\t\t\t\\vphantom{\\left(\\tfrac{\\sigma_{i_k+1}}{\\sigma_{i_k}}\\right)}\n\t\t\t\t\t\t\t\t}^{\\mathclap{\\to0 \\text{ by \\cref{thm:PG:barPhi}}}}\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\sigma_{i_k+1}\\alpha_{i_k+1}(1+\\rho_{i_k+1})\n\t\t\t\t\t\t\t\t\t(\\cost_1(x_{i_k})-\\costinf*)\n\t\t\t\t\t\t\t\t\t\\vphantom{\\left(\\tfrac{\\sigma_{i_k+1}}{\\sigma_{i_k}}\\right)}\n\t\t\t\t\t\t\t\t}^{<0}\n\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t&\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\alpha_{\\rm max}\\rho_{\\rm max}^2\n\t\t\t\t\t\t\t\t\t\\bcost_2(x_{i_k})\n\t\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\t\\alpha_{\\rm max}\\left(1-\\tfrac{\\sigma_{i_k+1}}{\\sigma_{i_k}}\\right)\n\t\t\t\t\t\t\t\t\t\\bcost_2(x_{i_k+1})\n\t\t\t\t\t\t\t\t}_{\\text{bounded by \\cref{thm:PG:barPhi}}}.\n\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\tIn particular, we have\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:supL}\n\t\t\t\t\t\t\t\t\\sup_{k\\in\\N}\\mathcal L_{i_k+1}(x_\\star)<\\infty.\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tLet now \\(k'\\in K'\\).\n\t\t\t\t\t\t\tSince \\(\\cost_1(x_j)-\\costinf*\\geq0\\) for \\(j=i_{k'}+1,\\dots,k'\\), observe that\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:Lk'geq}\n\t\t\t\t\t\t\t\t\\tfrac12\\|x_{k'}-x_\\star\\|^2\n\t\t\t\t\t\t\t\\leq\n\t\t\t\t\t\t\t\t\\mathcal L_{k'}(x_\\star)\n\t\t\t\t\t\t\t\\quad\n\t\t\t\t\t\t\t\t\\forall k'\\in K',\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tand \\cref{thm:PG:qSD} yields that\n\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\\mathcal L_{k'}(x_\\star)\n\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\\mathcal L_{k'-1}(x_\\star)\n\t\t\t\t\t\t\t\\leq\\dots\\leq\n\t\t\t\t\t\t\t\t\\mathcal L_{i_{k'}+2}(x_\\star)\n\t\t\t\t\t\t\t\\numberthis\\label{eq:Lk'descent}\n\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\\mathcal L_{i_{k'}+1}(x_\\star)\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\sigma_{i_{k'}+1}\\alpha_{i_{k'}+1}(1+\\rho_{i_{k'}+1}-\\rho_{i_{k'}+2}^2)\n\t\t\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\t\t\\cost_1(x_{i_{k'}})-\\costinf*\n\t\t\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\\mathcal L_{i_{k'}+1}(x_\\star)\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\tfrac{\\sigma_{i_{k'}+1}}{\\sigma_{i_{k'}}}\n\t\t\t\t\t\t\t\t}_{\\leq1}\n\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\vphantom{\\tfrac{\\sigma_{i_{k'}+1}}{\\sigma_{i_{k'}}}}\n\t\t\t\t\t\t\t\t\t\\alpha_{\\rm max}(1+\\rho_{i_{k'}+1}-\\rho_{i_{k'}+2}^2)\n\t\t\t\t\t\t\t\t}_{\\text{\\clap{bounded by \\cref{thm:PG:rhomax}}}}\n\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\vphantom{\\tfrac{\\sigma_{i_{k'}+1}}{\\sigma_{i_{k'}}}}\n\t\t\t\t\t\t\t\t\t\\sigma_{i_{k'}}\n\t\t\t\t\t\t\t\t\t\\bigl|\n\t\t\t\t\t\t\t\t\t\t\\cost_1(x_{i_{k'}})-\\costinf*\n\t\t\t\t\t\t\t\t\t\\bigr|\n\t\t\t\t\t\t\t\t}_{\\text{\\clap{bounded by \\cref{thm:PG:barPhi}}}}\n\t\t\t\t\t\t\t\t< \\infty\n\t\t\t\t\t\t\t\\qquad\n\t\t\t\t\t\t\t\t\\forall k'\\in K',\n\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\twhere \\eqref{eq:supL} was used in the last inequality.\n\t\t\t\t\t\t\tHere, boundedness of the under-bracketed term follows from boundedness of \\(\\seq{x_{i_{k'}}}[k'\\in K']\\) and lower semicontinuity of \\(\\cost_1\\).\n\t\t\t\t\t\t\tThen, \\eqref{eq:Lk'geq} implies that the sequence \\(\\seq{x_{k'}}[k'\\in K']\\) is bounded.\n\t\t\t\t\t\t\tCombined with \\cref{claim:convergence:ik} and the fact that the index set \\(K'\\) is the complement of the indices therein, the claim follows.\n\t\t\t\t\t\t\\qedhere\n\t\t\t\t\t\t\\end{claims}\n\t\t\t\t\t\\end{proofitemize}\n\t\t\t\t\\end{proofitemize}\n\t\t\t\\end{proof}","sha256_utf8":"e674617d3f3eeb36082c45fc0b5e999d1cb70342846552ff86268da054777010"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"引入目标差并缩放迭代","source_units":[{"start":73892,"end":74655,"start_line":1509,"end_line":1530,"raw_tex":"\t\t\t\tLet\n\t\t\t\t\\begin{equation}\n\t\t\t\t\tp_k%\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\phik(x_k)-\\phik(x_\\star)\n\t\t\t\t=\n\t\t\t\t\t\\sigk\\bigl(\\cost_1(x_k)-\\costinf*\\bigr)\n\t\t\t\t\t+\n\t\t\t\t\t\\bcost_2(x_k)\n\t\t\t\t\\end{equation}\n\t\t\t\t(which is not necessarily positive).\n\t\t\t\tWe will prove the claim using the uppercase notation of \\eqref{eq:uppercase} and, consistently, set\n\t\t\t\t\t\\(P_k%\n\t\t\t\t\t\\coloneqq{\\tfrac{1}{\\sigk}}p_k%\n\t\t\t\t\t=\\Phik(x_k)-\\Phik(x_\\star)\n\t\t\t\t\t\\).\n\t\t\t\t\tBeing a simple matter of multiplicative constants, note that proximal gradient iterations \\eqref{eq:PG} can equivalently be expressed in the penalized cost reformulation \\(\\Phik*=\\Fk*+\\Gk*\\) up to suitable scaling of the stepsize, namely,\n\t\t\t\t\t\\[\n\t\t\t\t\t\tx_{k+1}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\prox_{\\sigk*\\alphk*\\Gk*}(x_k-\\sigk*\\alphk*\\nabla\\Fk*(x_k)).\n\t\t\t\t\t\\]\n","sha256_utf8":"64257f98e01e98cea129509e4114eb326546f765a301781b32a5ac202aa213e7"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"取得近端次梯度包含","source_units":[{"start":74655,"end":75028,"start_line":1531,"end_line":1539,"raw_tex":"\t\t\t\t\tThe subgradient characterization of the proximal mapping as in \\eqref{eq:Hk:phi} yields\n\t\t\t\t\t\\begin{equation}\\label{eq:subgrad}\n\t\t\t\t\t\t\\tfrac{\\Hk(x_{k-1})-x_k}{\\sigk\\alphk}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\tfrac{x_{k-1}-x_k}{\\sigk\\alphk}-\\nabla\\Fk(x_{k-1})\n\t\t\t\t\t\\in\n\t\t\t\t\t\t\\partial\\Gk(x_k),\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\twhere we remind that \\(\\Hk=\\id-\\alphk\\nabla\\fk\\), cf. \\eqref{eq:Hk:phi}.\n","sha256_utf8":"00fa744d1b9fcd8f257529ed347edead2729040ba0a68a3c1930d6488abcbe87"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"建立罚目标单步下降式","source_units":[{"start":75028,"end":76058,"start_line":1540,"end_line":1585,"raw_tex":"\t\t\t\t\tHence, since \\(\\partial\\Phik=\\nabla\\Fk+\\partial\\Gk\\),\n\t\t\t\t\t\\begin{subequations}\n\t\t\t\t\t\t\\begin{align}\n\t\t\t\t\t\t\\nonumber\n\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\Phik(x_{k-1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Phik(x_k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\\innprod{x_{k-1}-x_k}{x_{k-1}-x_k}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk(x_{k-1})-\\nabla\\Fk(x_k)}{x_{k-1}-x_k}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\nonumber\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\Phik(x_{k-1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Phik(x_k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac1{\\sigk}\\innprod{\\nabla\\fk(x_{k-1})-\\nabla\\fk(x_k)}{x_{k-1}-x_k}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\Phik(x_{k-1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Phik(x_k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1-\\alphk\\lk}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t\t\\label{eq:phikdescent}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\tP_{k-1}%\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\tP_k%\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t(\\tfrac{1}{\\sigk}-\\tfrac{1}{\\sigma_{k-1}})\\bcost_2(x_{k-1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1-\\alphk\\lk}{\\sigk\\alphk}\\|x_{k-1}-x_k\\|^2.\n\t\t\t\t\t\t\\label{eq:ineqsubgrad}\n\t\t\t\t\t\t\\end{align}\n\t\t\t\t\t\\end{subequations}\n","sha256_utf8":"15cc4d4f46e7e2ea6eb3bbf1d8e1426b183b25bf7538362a9fc8deae9fe8b247"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"展开解点距离不等式","source_units":[{"start":76058,"end":77853,"start_line":1586,"end_line":1666,"raw_tex":"\t\t\t\t\tAgain from \\eqref{eq:subgrad}, this time with \\(k\\gets k+1\\), we have\n\t\t\t\t\t\\begin{subequations}\\label{subeq:3ineq}\n\t\t\t\t\t\t{\\mathtight[0.6]%\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\Gk*(x_\\star)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Gk*(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_\\star-x_{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\innprod{x_k-x_{k+1}}{x_\\star-x_{k+1}}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\Gk*(x_\\star)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Gk*(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_\\star-x_{k+1}}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_\\star\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_\\star-x_{k+1}\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t&\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\Gk*(x_\\star)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Gk*(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_\\star-x_k}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_\\star\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t&\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_\\star-x_{k+1}\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\Gk*(x_\\star)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Gk*(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\fillwidthof[c]{\n\t\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_\\star-x_k}\n\t\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\t\\Fk*(x_\\star)-\\Fk*(x_k)\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\underbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t}_{\\text{(A)}}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_\\star\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t&\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_\\star-x_{k+1}\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_{k+1}\\|^2,\n\t\t\t\t\t\t\\numberthis\\label{eq:ineq1}\n\t\t\t\t\t\t\\end{align*}}%\n\t\t\t\t\t\twhere the last inequality uses convexity of \\(\\Fk*\\).\n","sha256_utf8":"22c8ea4b8e0028328ab33cc0e9c067d0ca24a46bf3d8c44e6a8032c7176f47b3"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"分解并估计交叉项甲","source_units":[{"start":77853,"end":78615,"start_line":1667,"end_line":1694,"raw_tex":"\t\t\t\t\t\tAs to term (A), we have\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\text{(A)}\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\\innprod{\\Hk(x_{k-1})-x_k}{x_{k+1}-x_k}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\\innprod{\\Hk(x_{k-1})-x_k+\\sigk\\alphk\\nabla\\Fk*(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\overrel[\\leq]{\\eqref{eq:subgrad}}{} &\n\t\t\t\t\t\t\t\\fillwidthof[c]{\n\t\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\t\\innprod{\\Hk(x_{k-1})-x_k}{x_{k+1}-x_k}\n\t\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\t\\Gk(x_{k+1})-\\Gk(x_k)\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla\\Fk*(x_k)-\\nabla\\Fk(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t&\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{1}{\\sigk\\alphk}\n\t\t\t\t\t\t\t\t\\innprod{\\Hk(x_{k-1})-\\Hk(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t}_{\\text{(B)}}.\n\t\t\t\t\t\t\\numberthis\\label{eq:ineqA}\n\t\t\t\t\t\t\\end{align*}\n","sha256_utf8":"42b4b6d0261d85b249c7749b04918e004148513b460790322157fc557c1e92ac"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"用杨氏不等式控制交叉项","source_units":[{"start":78615,"end":79104,"start_line":1695,"end_line":1709,"raw_tex":"\t\t\t\t\t\tNext, we bound the term (B) by \\(\\epsk*\\)-Young's inequality as\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\text{(B)}\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\tfrac{\\epsk*}{2\\sigk\\alphk}\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\epsk*\\sigk\\alphk}\\|\\Hk(x_{k-1})-\\Hk(x_k)\\|^2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\overrel{\\ref{thm:H}}{} &\n\t\t\t\t\t\t\t\\tfrac{\\epsk*}{2\\sigk\\alphk}\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{\\Mk^2}{2\\epsk*\\sigk\\alphk}\\|x_{k-1}-x_k\\|^2.\n\t\t\t\t\t\t\\numberthis\\label{eq:ineqB}\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\\end{subequations}\n","sha256_utf8":"3467084bb587d1fd6b1e22240f2a37b85ad0a16962824058fd06322d84724e08"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"合并三式并分离下层项","source_units":[{"start":79104,"end":80740,"start_line":1710,"end_line":1796,"raw_tex":"\t\t\t\t\tCombining the three inequalities \\eqref{subeq:3ineq} yields\n\t\t\t\t\t{\\mathtight[0.9]%\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t0\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\Phik*(x_\\star)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\Gk*(x_{k+1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\Fk*(x_k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\Gk(x_{k+1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\Gk(x_k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t(\\tfrac1{\\sigk*}-\\tfrac{1}{\\sigk})\n\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\Biggl\\{\n\t\t\t\t\t\t\t\\tfrac{\\Mk^2}{2\\epsk*\\sigk\\alphk}\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\tfrac{\\epsk*}{2\\sigk\\alphk}\n\t\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\t\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_\\star-x_{k+1}\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t\\hphantom{+\\Biggl\\{{}}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t\\|x_k-x_\\star\\|^2\n\t\t\t\t\t\t\\Biggr\\}\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\overrel{\\ref{thm:Phi*}}{} &\n\t\t\t\t\t\t\\costinf*\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac1{\\sigk*}\\costinf_2\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\Fk(x_k)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\Gk(x_k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\bigl(\\tfrac1{\\sigk*}-\\tfrac{1}{\\sigk}\\bigr)\n\t\t\t\t\t\t\\vphantom{\\f^2}\n\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\f_2(x_k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\g_2(x_{k+1})\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\{\\cdots\\}\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t-P_k%\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\t\\tfrac1{\\sigk*}-\\tfrac{1}{\\sigk}\n\t\t\t\t\t\t\t\t\\vphantom{\\f^2}\n\t\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\\,\n\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_k)}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\f_2(x_k)\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\g_2(x_{k+1})\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\costinf_2\n\t\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t}^{\\text{(D)}}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\Bigl\\{\\cdots\\Bigr\\}.\n\t\t\t\t\t\\end{align*}}%\n","sha256_utf8":"a9dc7832d3b518737d693b293f10275af4d0da8d710002a128aa53087964e114"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"利用下层凸性控制余项","source_units":[{"start":80740,"end":81503,"start_line":1797,"end_line":1831,"raw_tex":"\t\t\t\t\tBy using convexity of \\(\\f_2\\) we can bound the term (D) as\n\t\t\t\t\t{\\mathtight[0.75]%\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\text{(D)}\n\t\t\t\t\t\t={} &\n\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_{k+1})}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\f_2(x_k)\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\g_2(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_{k+1})-\\nabla \\f_2(x_k)}{x_{k+1}-x_k}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\costinf_2\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\fillwidthof[c]{\n\t\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_{k+1})}{x_k-x_{k+1}}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\f_2(x_k)\n\t\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\t-\\f_2(x_{k+1})\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\g_2(x_{k+1})\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\fillwidthof[c]{\n\t\t\t\t\t\t\t\t\\innprod{\\nabla \\f_2(x_{k+1})-\\nabla \\f_2(x_k)}{x_{k+1}-x_k}\n\t\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\t\\lk*^{(2)}\\|x_{k+1}-x_k\\|^2\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\costinf_2,\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t}%\n","sha256_utf8":"91c239db1c10740f295c466490d1f22811ecf443ace44f77cfb69380f488ebaf"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"得到含距离与残差的界","source_units":[{"start":81503,"end":82191,"start_line":1832,"end_line":1861,"raw_tex":"\t\t\t\t\twhich plugged in the previous inequality results in\n\t\t\t\t\t{\\mathtight[0.75]%\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t0\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t-P_k%\n\t\t\t\t\t\t-\n\t\t\t\t\t\t(\\tfrac1{\\sigk*}-\\tfrac{1}{\\sigk})\\bcost_2(x_{k+1})\n\t\t\t\t\t\t+\n\t\t\t\t\t\t(\\tfrac1{\\sigk*}-\\tfrac{1}{\\sigk})\n\t\t\t\t\t\t\\lk*^{(2)}\\|x_{k+1}-x_k\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{\\Mk^2}{2\\epsk*\\sigk\\alphk}\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\tfrac{\\epsk*}{2\\sigk\\alphk}\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\\|x_\\star-x_{k+1}\\|^2\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{2\\sigk*\\alphk*}\n\t\t\t\t\t\t\\|x_k-x_\\star\\|^2.\n\t\t\t\t\t\\numberthis\\label{eq:3ineqs}\n\t\t\t\t\t\\end{align*}}%\n","sha256_utf8":"0f245a2d356d94e307dd9ff69fc97f6098daae62951b397fa7bf9509690d933c"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"加权合并并选取参数","source_units":[{"start":82191,"end":83210,"start_line":1862,"end_line":1897,"raw_tex":"\t\t\t\t\tSumming \\eqref{eq:3ineqs}\\(+\\betk*\\)\\eqref{eq:ineqsubgrad}, multiplying by \\(\\sigk*\\alphk*\\), and rearranging yields that for every \\(\\betk*\\geq0\\) and \\(\\epsk*>0\\) the following hold.\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t& \\tfrac{1}{2}\n\t\t\t\t\t\\|x_{k+1}-x_\\star\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\sigk*}{\\sigk}\\alphk*(1+\\betk*)p_k%\n\t\t\t\t\t+\n\t\t\t\t\t\\alphk*\\sigk*\\Delk*\\bcost_2(x_{k+1})\n\t\t\t\t\\\\\n\t\t\t\t\t+ {}&\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t\\tfrac{1-\\rhok*\\epsk*}{2}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\alphk*\\sigk*\\Delk*%\n\t\t\t\t\t\t\\lk*^{(2)}\n\t\t\t\t\t\\right)\n\t\t\t\t\t\\|x_k-x_{k+1}\\|^2\n\t\t\t\t\\leq\n\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t\\|x_k-x_\\star\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\sigk*}{\\sigma_{k-1}}\\betk*\\alphk*p_{k-1}%\n\t\t\t\t\\\\\n\t\t\t\t\t-{}&\n\t\t\t\t\t\\rhok*\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t\\betk*(1-\\alphk\\lk)\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\tfrac{\\Mk^2}{2\\epsk*}\n\t\t\t\t\t\\right)\n\t\t\t\t\t\\|x_{k-1}-x_k\\|^2\n\t\t\t\t\t+\n\t\t\t\t\t\\alphk*\\sigk*\\betk*\\Delk\\bcost_2(x_{k-1}).\n\t\t\t\t\\numberthis\n\t\t\t\t\\end{align*}\n\t\t\t\tBy selecting \\(\\betk=\\rhok\\) and \\(\\epsk\\coloneqq\\nicefrac{1}{2\\rhok}\\) the inequality reduces to the claimed quasi-descent in terms of \\(\\mathcal L_{k}(x^\\star)\\).\n","sha256_utf8":"7fc11f80b951f12d13f3a808ca32bfff597fe22169efb420e78ae7473383c0f1"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"归纳控制相邻参数比","source_units":[{"start":88045,"end":88337,"start_line":2004,"end_line":2015,"raw_tex":"~\n\t\t\t\t\\begin{proofitemize}\n\t\t\t\t\\item \\ref{thm:PG:rhomax}~\n\t\t\t\t\tObserving that\n\t\t\t\t\t\\(\n\t\t\t\t\t\t\\rhok*\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\sqrt{\\tfrac{\\sigk}{\\sigma_{k-1}}\\bigl(1+\\rhok\\bigr)}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\sqrt{1+\\rhok}\n\t\t\t\t\t\\)\n\t\t\t\t\tby \\cref{cond:alphk}, the assertion follows from a trivial induction argument.\n","sha256_utf8":"f4dc083c3aacaf000dc5707d288b8a61251afbebc8012b019ed01b582ef96435"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"验证拟下降各系数非负","source_units":[{"start":88338,"end":89599,"start_line":2017,"end_line":2068,"raw_tex":"\t\t\t\t\\item \\ref{thm:PG:qSD}~\n\t\t\t\t\tSince \\(\\alphk\\leq\\balphk\\), from the definition of \\(\\balphk\\) in \\eqref{eq:adabim:alphk*_0} it follows that the multiplying coefficients in \\eqref{eq:puya:muk:descent} are positive:\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\mathcal L_{k+1}(x_\\star) - \\mathcal L_k(x_\\star)\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\sigk*\\alphk*\n\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\\overbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{\\Delk}{\\rhok*}\n\t\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\\tfrac{1}{\\sigk}\n\t\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\t\\overbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\t1+\\rhok*\n\t\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\t\\rho_{k+2}^2\\tfrac{\\sigk}{\\sigk*}\n\t\t\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\\bcost_2(x_k)\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\sigk\\alphk\n\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\t1+\\rhok-\\rhok*^2\n\t\t\t\t\t\t\t}^{\\geq0}\n\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\t\\bigl(\\cost_1(x_{k-1})-\\costinf*\\bigr)\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\left(\n\t\t\t\t\t\t\t\\underbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{1}{4}\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\rhok*^2(\\alphk\\lk-\\alphk^2\\Lk^2)\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\alphk\\sigk\\lk^{(2)}\\Delk\n\t\t\t\t\t\t\t}_{\\geq0}\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\t\\|x_{k-1}-x_k\\|^2.\n\t\t\t\t\t\t\\vphantom{\n\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{1}{4}\n\t\t\t\t\t\t\t}_0\n\t\t\t\t\t\t}\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tThe claim then readily follows from the fact that \\(\\bcost_2\\geq0\\), cf. \\cref{thm:bar>=0}.\n","sha256_utf8":"ad2e7397c7ff631d7b6178e008e8e443e60ad9c175c878b7227674809fb37e49"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"证明平移目标下降与有界","source_units":[{"start":89600,"end":90852,"start_line":2070,"end_line":2100,"raw_tex":"\t\t\t\t\\item \\ref{thm:PG:barPhi}~\n\t\t\t\t\tWe have\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\bphik*(x_{k+1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\bphik(x_k)\n\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\\bphik*(x_{k+1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\bphik*(x_k)\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\phik*(x_{k+1})\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\phik*(x_k)\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\sigk*\\left(\n\t\t\t\t\t\t\t\\Phik*(x_{k+1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\\Phik*(x_k)\n\t\t\t\t\t\t\\right)\n\t\t\t\t\t\\overrel*[\\leq]{\\eqref{eq:phikdescent}}\n\t\t\t\t\t\t- \\tfrac{1-\\nu}{\\alphk*}\n\t\t\t\t\t\t\\|x_{k+1}-x_k\\|^2.\n\t\t\t\t\t\t\\numberthis \\label{eq:descent:bphik}\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tHere, the first inequality uses the fact that \\(\\sigk\\geq\\sigk*\\), and therefore \\(\\bphik=\\sigk\\bcost_1+\\bcost_2\\) is smaller than \\(\\bphik*=\\sigk*\\bcost_1+\\bcost_2\\) on \\(\\dom\\cost_2\\), cf. \\cref{thm:bar>=0}, and \\(x_k\\in\\dom\\gk=\\dom\\cost_1\\cap\\dom\\cost_2\\subseteq\\dom\\cost_2\\);\n\t\t\t\t\tthe second inequality follows from the fact that \\(\\alphk*\\lk*\\leq\\nu\\).\n\t\t\t\t\tThis shows the sought inequality.\n\t\t\t\t\tIn turn, since \\(\\nu<1\\), the sequence \\(\\seq{\\bphik(x_k)}\\) is decreasing; since \\(x_k\\in\\dom\\cost_2\\), it follows from \\cref{thm:bar>=0} that \\(\\bphik(x_k)=\\sigk\\bcost_1(x^k)+\\bcost_2(x^k)\\geq0\\), hence that it is convergent and that the positive-valued sequences \\(\\seq{\\sigk\\bcost_1(x^k)}\\) and \\(\\seq{\\bcost_2(x^k)}\\) are bounded.\n","sha256_utf8":"bd557c9aa350846acca0ea713933f2797d8d83a8d9e6dca25f46de793d001cb7"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"望远镜求和得到残差率","source_units":[{"start":90853,"end":91283,"start_line":2102,"end_line":2110,"raw_tex":"\t\t\t\t\tThe convergence rate result follows immediately by telescoping \\eqref{eq:descent:bphik}\n\t\t\t\t\t\\begin{equation}\\label{eq:sublin:res}\n\t\t\t\t\t\t\\tfrac{(1-\\nu)(K+1)}{\\alpha_{\\rm max}}\\min_{k\\leq K}\\|x_{k+1}-x_k\\|^2\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\sum_{k=0}^{K}\\tfrac{1-\\nu}{\\alphk*}\\|x_{k+1}-x_k\\|^2\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\bar{\\originalvarphi}_0(x_{0}),\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\twhere \\cref{cond:alphk} was used to bound \\(\\alphk*\\leq\\alpha_{\\rm max}\\).\n","sha256_utf8":"b26ed75246ee2eb1c3d29eb407c8241d6af43c621b8b0795e042fc7ac7596fb4"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"首种情形直接推出有界","source_units":[{"start":91284,"end":91883,"start_line":2112,"end_line":2120,"raw_tex":"\t\t\t\t\\item \\ref{thm:PG:boundedness}~\n\t\t\t\t\tWe pattern the proof structure of \\cite[Thm. 3.2]{solodov2007explicit}, thereby considering two mutually exclusive cases.\n\t\t\t\t\t\\begin{proofitemize}[leftmargin=*,topsep=0pt]\n\t\t\t\t\t\\item\n\t\t\t\t\t\t{\\it Case 1: \\(\\cost_1(x_k)\\geq\\costinf*\\) holds for \\(k\\) large enough.}\n\n\t\t\t\t\t\tRecall the definition of \\(W_k\\) and \\(\\mathcal L_k\\) in \\cref{thm:LW}.\n\t\t\t\t\t\tObserving that \\(W_k\\geq0\\) and \\(\\frac12\\|x_k-x_\\star\\|^2\\leq\\mathcal L_k(x_\\star)\\), \\cref{thm:PG:qSD} implies that \\(\\seq{\\mathcal L_k(x_\\star)}\\) converges and that consequently \\(\\seq{x_k}\\) is bounded.\n\n","sha256_utf8":"d3f6ddec6c59e9d955a0342a0d0f395c4648f64d33d1678cd63cf07abb53ce88"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"定义低于最优值的末次索引","source_units":[{"start":91883,"end":92295,"start_line":2121,"end_line":2133,"raw_tex":"\t\t\t\t\t\\item\n\t\t\t\t\t\t{\\it Case 2: \\(\\cost_1(x_k)<\\costinf*\\) holds infinitely often.}\n\n\t\t\t\t\t\tIn this case, for every \\(k\\) large enough the index\n\t\t\t\t\t\t\\begin{equation}\\label{eq:ik}\n\t\t\t\t\t\t\ti_k\n\t\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\t\\max\\set{i\\leq k}[\\cost_1(x_i)<\\costinf*]\n\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\tis well defined.\n\t\t\t\t\t\tWe proceed by intermediate claims.\n\t\t\t\t\t\t\\renewcommand{\\claimscounter}{\\ref*{thm:PG}}%\n\t\t\t\t\t\t\\begin{claims}\n","sha256_utf8":"8e502d01dda50041c75413d79056b70952fdd5fd0de724147f4f488a7c3cd9ac"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"水平有界性控制选定子列","source_units":[{"start":92295,"end":93217,"start_line":2134,"end_line":2148,"raw_tex":"\t\t\t\t\t\t\\item\\label{claim:convergence:ik}%\n\t\t\t\t\t\t\t{\\it the sequences \\(\\seq{x_{i_k}}\\) and \\(\\seq{x_{i_k+1}}\\) are bounded}.\n\n\t\t\t\t\t\t\tThe optimal set\n\t\t\t\t\t\t\t\\[\n\t\t\t\t\t\t\t\t\\X_1\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\set{x\\in\\X_2}[\\cost_1(x)\\leq \\costinf*]\n\t\t\t\t\t\t\t=\n\t\t\t\t\t\t\t\t\\set{x}[\\cost_1(x)\\leq\\costinf*,\\ \\bcost_2(x)\\leq0]\n\t\t\t\t\t\t\t\\]\n\t\t\t\t\t\t\tcoincides with a sublevel set of the convex function \\(h\\coloneqq\\max\\set{\\cost_1-\\costinf*,\\bcost_2}\\).\n\t\t\t\t\t\t\tSince it is nonempty and bounded by \\cref{ass:X1}, \\(h\\) is level bounded; see, e.g., \\cite[Prop. 11.13]{bauschke2017convex} or \\cite[Lem. 1]{themelis2019acceleration}.\n\t\t\t\t\t\t\tNote that \\cref{thm:PG:barPhi} implies that \\(\\seq{\\bcost_2(x_k)}\\) is (upper) bounded, which combined with the fact that \\(\\cost_1(x_{i_k})<\\costinf*\\) implies that \\(\\seq{x_{i_k}}\\) lies in a sublevel set of \\(h\\), and is therefore bounded.\n\t\t\t\t\t\t\tIn turn, \\cref{thm:PG:barPhi} implies that so is \\(\\seq{x_{i_k+1}}\\).\n","sha256_utf8":"64be7cce02d461e0962d23ebb3c76a568d761c4a77dbc81477539a5df32c8f59"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"控制子列处李雅普诺夫值","source_units":[{"start":93217,"end":94747,"start_line":2149,"end_line":2190,"raw_tex":"\t\t\t\t\t\t\\item\n\t\t\t\t\t\t\t{\\it the whole sequence \\(\\seq{x_k}\\) is bounded}.\n\n\t\t\t\t\t\t\tTo this end, it remains to show that \\(\\seq{x_{k'}}[k'\\in K']\\) is bounded, where\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:K'}\n\t\t\t\t\t\t\t\tK'\\coloneqq\\set{k'\\in\\N}[k'\\geq i_{k'}+2].\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tWith \\(\\rho_{\\rm max}\\) as in \\cref{thm:PG:rhomax}, for every \\(k\\in\\N\\)\n\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\\mathcal L_{i_k+1}(x_\\star)\n\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\tfrac12\n\t\t\t\t\t\t\t\t\t\\|x_{i_k+1}-x_\\star\\|^2\n\t\t\t\t\t\t\t\t\t\\vphantom{\\left(\\tfrac{\\sigma_{i_k+1}}{\\sigma_{i_k}}\\right)}\n\t\t\t\t\t\t\t\t}^{\\text{bounded by \\cref{claim:convergence:ik}}}\n\t\t\t\t\t\t\t\t+~~\n\t\t\t\t\t\t\t\t\\overbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\tfrac{1}{4}\\|x_{i_k+1}-x_{i_k}\\|^2\n\t\t\t\t\t\t\t\t\t\\vphantom{\\left(\\tfrac{\\sigma_{i_k+1}}{\\sigma_{i_k}}\\right)}\n\t\t\t\t\t\t\t\t}^{\\mathclap{\\to0 \\text{ by \\cref{thm:PG:barPhi}}}}\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\sigma_{i_k+1}\\alpha_{i_k+1}(1+\\rho_{i_k+1})\n\t\t\t\t\t\t\t\t\t(\\cost_1(x_{i_k})-\\costinf*)\n\t\t\t\t\t\t\t\t\t\\vphantom{\\left(\\tfrac{\\sigma_{i_k+1}}{\\sigma_{i_k}}\\right)}\n\t\t\t\t\t\t\t\t}^{<0}\n\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t&\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\alpha_{\\rm max}\\rho_{\\rm max}^2\n\t\t\t\t\t\t\t\t\t\\bcost_2(x_{i_k})\n\t\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\t\\alpha_{\\rm max}\\left(1-\\tfrac{\\sigma_{i_k+1}}{\\sigma_{i_k}}\\right)\n\t\t\t\t\t\t\t\t\t\\bcost_2(x_{i_k+1})\n\t\t\t\t\t\t\t\t}_{\\text{bounded by \\cref{thm:PG:barPhi}}}.\n\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\tIn particular, we have\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:supL}\n\t\t\t\t\t\t\t\t\\sup_{k\\in\\N}\\mathcal L_{i_k+1}(x_\\star)<\\infty.\n\t\t\t\t\t\t\t\\end{equation}\n","sha256_utf8":"72d11ba03f5d463209f0963c593f4a009b42a8b4bf42c9edf0cb27b360566e26"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"沿索引间隙控制整个序列","source_units":[{"start":94747,"end":96776,"start_line":2191,"end_line":2246,"raw_tex":"\t\t\t\t\t\t\tLet now \\(k'\\in K'\\).\n\t\t\t\t\t\t\tSince \\(\\cost_1(x_j)-\\costinf*\\geq0\\) for \\(j=i_{k'}+1,\\dots,k'\\), observe that\n\t\t\t\t\t\t\t\\begin{equation}\\label{eq:Lk'geq}\n\t\t\t\t\t\t\t\t\\tfrac12\\|x_{k'}-x_\\star\\|^2\n\t\t\t\t\t\t\t\\leq\n\t\t\t\t\t\t\t\t\\mathcal L_{k'}(x_\\star)\n\t\t\t\t\t\t\t\\quad\n\t\t\t\t\t\t\t\t\\forall k'\\in K',\n\t\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\t\tand \\cref{thm:PG:qSD} yields that\n\t\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\t\\mathcal L_{k'}(x_\\star)\n\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\\mathcal L_{k'-1}(x_\\star)\n\t\t\t\t\t\t\t\\leq\\dots\\leq\n\t\t\t\t\t\t\t\t\\mathcal L_{i_{k'}+2}(x_\\star)\n\t\t\t\t\t\t\t\\numberthis\\label{eq:Lk'descent}\n\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\\mathcal L_{i_{k'}+1}(x_\\star)\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\sigma_{i_{k'}+1}\\alpha_{i_{k'}+1}(1+\\rho_{i_{k'}+1}-\\rho_{i_{k'}+2}^2)\n\t\t\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\t\t\\cost_1(x_{i_{k'}})-\\costinf*\n\t\t\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\t\\mathcal L_{i_{k'}+1}(x_\\star)\n\t\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\tfrac{\\sigma_{i_{k'}+1}}{\\sigma_{i_{k'}}}\n\t\t\t\t\t\t\t\t}_{\\leq1}\n\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\vphantom{\\tfrac{\\sigma_{i_{k'}+1}}{\\sigma_{i_{k'}}}}\n\t\t\t\t\t\t\t\t\t\\alpha_{\\rm max}(1+\\rho_{i_{k'}+1}-\\rho_{i_{k'}+2}^2)\n\t\t\t\t\t\t\t\t}_{\\text{\\clap{bounded by \\cref{thm:PG:rhomax}}}}\n\t\t\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\t\t\\vphantom{\\tfrac{\\sigma_{i_{k'}+1}}{\\sigma_{i_{k'}}}}\n\t\t\t\t\t\t\t\t\t\\sigma_{i_{k'}}\n\t\t\t\t\t\t\t\t\t\\bigl|\n\t\t\t\t\t\t\t\t\t\t\\cost_1(x_{i_{k'}})-\\costinf*\n\t\t\t\t\t\t\t\t\t\\bigr|\n\t\t\t\t\t\t\t\t}_{\\text{\\clap{bounded by \\cref{thm:PG:barPhi}}}}\n\t\t\t\t\t\t\t\t< \\infty\n\t\t\t\t\t\t\t\\qquad\n\t\t\t\t\t\t\t\t\\forall k'\\in K',\n\t\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t\twhere \\eqref{eq:supL} was used in the last inequality.\n\t\t\t\t\t\t\tHere, boundedness of the under-bracketed term follows from boundedness of \\(\\seq{x_{i_{k'}}}[k'\\in K']\\) and lower semicontinuity of \\(\\cost_1\\).\n\t\t\t\t\t\t\tThen, \\eqref{eq:Lk'geq} implies that the sequence \\(\\seq{x_{k'}}[k'\\in K']\\) is bounded.\n\t\t\t\t\t\t\tCombined with \\cref{claim:convergence:ik} and the fact that the index set \\(K'\\) is the complement of the indices therein, the claim follows.\n\t\t\t\t\t\t\\qedhere\n\t\t\t\t\t\t\\end{claims}\n\t\t\t\t\t\\end{proofitemize}\n\t\t\t\t\\end{proofitemize}\n\t\t\t","sha256_utf8":"dd91989bdfb74af9db0134d2e146287ccd13ee16ffce2295ce2d0f91bb6647dd"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"调用迭代有界性","source_units":[{"start":97924,"end":98151,"start_line":2267,"end_line":2269,"raw_tex":"\t\t\t\tWe begin by remarking that boundedness of the sequence \\(\\seq{x_k}\\) is ensured by \\cref{cond:alphk,cond:alphk:nu}, as shown in \\cref{thm:PG:boundedness}.\n\t\t\t\tWe next prove each claim individually.\n\t\t\t\t\\begin{proofitemize}\n","sha256_utf8":"9f5e900d5f4910d55114a51a7066837a507423315d6ec4a92007f3ce05fd9c8f"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"通过极限近端问题证明可行性","source_units":[{"start":98151,"end":99371,"start_line":2270,"end_line":2296,"raw_tex":"\t\t\t\t\\item \\ref{thm:recipe:rate}~\n\t\t\t\t\tConsider a convergent subsequence \\(x_{k_j}\\to x_\\infty\\), so that \\(x_{k_j+1}\\to x_\\infty\\) by \\cref{thm:PG:barPhi}.\n\t\t\t\t\tUp to further extracting if necessary we have that\n\t\t\t\t\t\\(\\alpha_{k_j+1}\\to\\alpha_\\infty\\geq\\alpha_{\\rm min}>0\\) and \\(\\sigma_{k_j}\\to0\\).\n\t\t\t\t\tObserve that\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\tx_{k+1}\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\argmin h({}\\cdot{};x_k,\\alpha_{k+1},\\sigk*),\n\t\t\t\t\t\\shortintertext{where}\n\t\t\t\t\t\th(w;x,\\alpha,\\sigma)\n\t\t\t\t\t\\coloneqq{} &\n\t\t\t\t\t\t(\\sigma \\g_1+\\g_2)(w)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{2\\alpha}\n\t\t\t\t\t\t\\bigl\\|w-x+\\alpha\\nabla\\bigl(\\sigma \\f_1+\\f_2\\bigr)(x)\\bigr\\|^2\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tis level bounded in \\(w\\) locally uniformly in \\((x,\\alpha,\\sigma)\\), as a function from \\(\\R^n\\times\\bigl(\\R^n\\times[\\alpha_{\\rm min},\\infty)\\times[0,\\infty)\\bigr)\\) to \\(\\Rinf\\).\n\t\t\t\t\tSince \\(h\\) is continuous in \\((x,\\alpha,\\sigma)\\), it follows from \\cite[Thm. 1.17]{rockafellar2011variational} that\n\t\t\t\t\t\\[\n\t\t\t\t\t\tx_\\infty\n\t\t\t\t\t\\in\n\t\t\t\t\t\t\\argmin h({}\\cdot{};x_\\infty,\\alpha_\\infty,0)\n\t\t\t\t\t=\n\t\t\t\t\t\t\\prox_{\\alpha_\\infty \\g_2}\\bigl(x_\\infty-\\alpha_\\infty\\nabla \\f_2(x_\\infty)\\bigr),\n\t\t\t\t\t\\]\n\t\t\t\t\tthis condition being equivalent to \\(x_\\infty\\in\\argmin(\\f_2+\\g_2)\\defeq\\X_2\\).\n","sha256_utf8":"98c6d2190cf8e5570100f43242c8bb249c876b8add7a92ab535758079042f35a"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"证明平移目标趋于零","source_units":[{"start":99371,"end":100878,"start_line":2297,"end_line":2344,"raw_tex":"\t\t\t\t\tWe next show that \\(\\bphik(x_k)\\to0\\).\n\n\t\t\t\t\tFor any \\(x_\\star\\in\\X_1\\), the subdifferential characterization of \\(x_{k+1}\\in\\prox_{\\alphk*\\gk*}(x_k-\\alphk*\\nabla\\fk*(x_k))\\)\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\tfrac{x_{k-1}-x_k}{\\alphk}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t(\\nabla\\fk(x_{k-1})-\\nabla\\fk(x_k))\n\t\t\t\t\t\\in\n\t\t\t\t\t\t\\partial\\phik(x_k)\n\t\t\t\t\t\\]\n\t\t\t\t\timplies that\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\phik(x_\\star)\n\t\t\t\t\t\\geq\n\t\t\t\t\t\t\\phik(x_k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\overbracket*[0.5pt]{\n\t\t\t\t\t\t\t\t\\tfrac{x_{k-1}-x_k}{\\alphk}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t(\\nabla\\fk(x_{k-1})-\\nabla\\fk(x_k))\n\t\t\t\t\t\t\t}^{\\to0 \\text{ by \\cref{thm:PG:barPhi}}}\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\\overbracket*[0.5pt]{\n\t\t\t\t\t\t\t\tx_\\star-x_k\n\t\t\t\t\t\t\t\t\\vphantom{\\tfrac{x_{k-1}-x_k}{\\alphk}}\n\t\t\t\t\t\t\t}^{\\text{\\clap{bounded}}}\n\t\t\t\t\t\t},\n\t\t\t\t\t\\]\n\t\t\t\t\thence that\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\costinf_2\n\t\t\t\t\t=\n\t\t\t\t\t\t\\lim_{k\\to\\infty}\\phik(x_\\star)\n\t\t\t\t\t\\geq\n\t\t\t\t\t\t\\limsup_{k\\to\\infty}\\phik(x_k)\n\t\t\t\t\t=\n\t\t\t\t\t\t\\limsup_{k\\to\\infty}\\left(\\sigk\\bcost_1(x_k)+\\overbracket*[0.5pt]{\\cost_2(x_k)}^{\\geq\\costinf_2}\\right)\n\t\t\t\t\t\\geq\n\t\t\t\t\t\t\\limsup_{k\\to\\infty}\\sigk\\bcost_1(x_k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\costinf_2.\n\t\t\t\t\t\\]\n\t\t\t\t\tHere, the second equality is obtained by adding and subtracting \\(\\sigk\\costinf_1\\) along with the fact that \\(\\sigk\\to0\\).\n\t\t\t\t\tSince \\(\\sigk\\bcost_1\\geq0\\), necessarily \\(\\sigk\\bcost_1(x_k)\\to0\\) and consequently \\(\\limsup_{k\\to\\infty}\\cost_2(x_k)=\\costinf_2\\).\n\t\t\t\t\tSimilarly, since \\(\\cost_2\\geq\\costinf_2\\) this necessarily implies that \\(\\cost_2(x_k)\\to\\costinf_2\\).\n\t\t\t\t\tThis shows that \\(\\bcost_2(x^k)\\to0\\) as claimed.\n\n","sha256_utf8":"483b78c1c6fdc2ee0836e026748d33a9389ee753f7ee1bdc4b4d05a76a909790"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"逐项证明辅助能量消失","source_units":[{"start":100878,"end":101783,"start_line":2345,"end_line":2381,"raw_tex":"\t\t\t\t\tWe next show that \\(\\seq{W_k}\\) too is vanishing.\n\t\t\t\t\tObserve that\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\tW_k\n\t\t\t\t\t\\defeq{} &\n\t\t\t\t\t\t\\tfrac{1-4\\alphk\\sigk\\lk^{(2)}\\Delk}{4}\n\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\\|x_k-x_{k-1}\\|^2\n\t\t\t\t\t\t}^{\\mathclap{\\to 0\\text{ by \\cref{thm:PG:barPhi}}}}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{\\sigk*}{\\sigk}\\alphk*\\rhok*\n\t\t\t\t\t\t\\overbracket[0.5pt]{\n\t\t\t\t\t\t\t\\bcost_2(x_{k-1})\n\t\t\t\t\t\t}^{\\mathclap{\\to 0}}\n\t\t\t\t\t\\\\\n\t\t\t\t\t&\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\\sigk\\alphk\\Delk\\bcost_2(x_k)\n\t\t\t\t\t\t}_{\\to 0}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\alphk\n\t\t\t\t\t\t\\left(1+\\rhok\\right)\n\t\t\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\t\t\\sigk\n\t\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\t\t\\cost_1(x_{k-1})\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\costinf*\n\t\t\t\t\t\t\t\\bigr)\n\t\t\t\t\t\t}_{\\to0}\n\t\t\t\t\t\\to\n\t\t\t\t\t\t0\n\t\t\t\t\t\\quad\\text{as \\(k\\to\\infty\\)},\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\twhere we also used the fact that \\(\\seq{\\alphk}\\leq\\alpha_{\\rm max}\\) and \\(\\seq{\\rhok}\\leq\\rho_{\\rm max}\\) as in \\cref{thm:PG:rhomax}.\n\n","sha256_utf8":"3f21e725981446778b069f646f7dd1c2684aba05846a8e60d9ef9eae0e9580e5"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"推导次梯度残差次线性率","source_units":[{"start":101783,"end":102520,"start_line":2382,"end_line":2395,"raw_tex":"\t\t\t\t\tThe asserted sublinear rate is a consequence of \\eqref{eq:sublin:res}, the characterization of the residual in \\cref{thm:H} (see \\eqref{eq:Hk:phi}), and the upper bounds for \\(\\lk\\), \\(\\Lk\\) in \\cref{thm:l<=L}:\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\textstyle\\sum_{k=0}^K\\dist^2(0, \\partial \\phik*(x_{k+1})\n\t\t\t\t\t\\leq{}&\n\t\t\t\t\t\t\\textstyle\\sum_{k=0}^K\\tfrac{1}{\\alphk*^2}\\|\\Hk*(x_{k})-\\Hk*(x_{k+1}) \\|^2\n\t\t\t\t\t\\\\\n\t\t\t\t\t\\leq {}&\n\t\t\t\t\t\t\\tfrac{M_{\\rm max}^2}{\\alpha_{\\rm min}}\\textstyle\\sum_{k=0}^K \\tfrac{1}{\\alphk*}\\|x_{k}-x_{k+1}\\|^2\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\tfrac{M_{\\rm max}^2}{\\alpha_{\\rm min}(1-\\nu)}\\bar{\\originalvarphi}_0(x_{0}),\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\twhere \\(M_{\\rm max} = \\sup_{k\\in\\N}M_k\\leq 1+ \\alpha_{\\rm max}L_{f_0, \\V}\\), see \\cref{thm:lL}.\n\n\n","sha256_utf8":"411f231359cca5b487cade03229559e519ed66ebccd937953c8690761d1436ad"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"确认所有聚点下层可行","source_units":[{"start":102520,"end":102835,"start_line":2396,"end_line":2399,"raw_tex":"\t\t\t\t\\item \\ref{thm:recipe:solution}~\n\t\t\t\t\tWe begin by observing that, since \\(\\bcost_2(x^k)\\to0\\), by lower semicontinuity all limit points of \\(\\seq{x^k}\\) belong to \\(\\X_2\\).\n\t\t\t\t\tAs done in the proof of \\cref{thm:PG:boundedness} we consider two possible cases.\n\t\t\t\t\t\\begin{proofitemize}[leftmargin=*,topsep=0pt]\n","sha256_utf8":"c2b6daf44c911fec42eed1e31b106e5a36017e0a293197df98531d9bbfc8f544"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"首种情形的加权差值求和","source_units":[{"start":102835,"end":104456,"start_line":2400,"end_line":2436,"raw_tex":"\t\t\t\t\t\\item\n\t\t\t\t\t\t{\\it Case 1: \\(\\cost_1(x_k)\\geq\\costinf*\\) holds for \\(k\\) large enough.}\n\n\t\t\t\t\t\tIn this case, we will actually show that the sequence \\(\\seq{x_k}\\) converges to a solution of \\eqref{eq:P}.\n\t\t\t\t\t\tWe first argue that there exists an optimal limit point; to this end, since all limit points are feasible and because of lower semicontinuity, it suffices to show that \\(\\liminf_{k\\to\\infty}\\cost_1(x_k)=\\costinf*\\).\n\t\t\t\t\t\tA telescoping argument on \\cref{thm:PG:qSD} along with \\cref{cond:alphamin} yields\n\t\t\t\t\t\t\\begin{equation}\n\t\t\t\t\t\t\t\\alpha_{\\rm min}\\sum_{k\\in \\N}\\sigk(1+\\rhok-\\rhok*^2)\\bigl(\\cost_1(x_{k-1})-\\costinf*\\bigr)\n\t\t\t\t\t\t\\leq\n\t\t\t\t\t\t\t\\sum_{k\\in \\N}\\sigk\\alphk(1+\\rhok-\\rhok*^2)\\bigl(\\cost_1(x_{k-1})-\\costinf*\\bigr)\n\t\t\t\t\t\t<\n\t\t\t\t\t\t\t\\infty.\n\t\t\t\t\t\t\\end{equation}\n\t\t\t\t\t\tSince \\(\\sum_{k\\in\\N}\\sigk=\\infty\\), necessarily\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\liminf_{k\\to\\infty}(1+\\rhok-\\rhok*^2)(\\cost_1(x_{k-1})-\\costinf*)\n\t\t\t\t\t\t=\n\t\t\t\t\t\t\t0.\n\t\t\t\t\t\t\t\\numberthis\\label{eq:liminf:cost1}\n\t\t\t\t\t\t\\end{align*}\n\t\t\t% \t\t\tObserve first that\n\t\t\t% \t\t\t\\begin{align*}\n\t\t\t% \t\t\t\t\\rhok*\n\t\t\t% \t\t\t=\n\t\t\t% \t\t\t\t\\tfrac{\\sigk*\\alphk*}{\\sigk\\alphk}\n\t\t\t% \t\t\t\\leq\n\t\t\t% \t\t\t\t\\tfrac{\\sigk*\\balphk*}{\\sigk\\alphk}\n\t\t\t% \t\t\t\\leq\n\t\t\t% \t\t\t\t\\tfrac{\\sigk*}{\\sigk}\n\t\t\t% \t\t\t\t\\sqrt{\\tfrac{\\sigk}{\\sigma_{k-1}}\\bigl(1+\\rhok\\bigr)}\n\t\t\t% \t\t\t\t\\tfrac{\\sigk}{\\sigk*}\n\t\t\t% \t\t\t\\leq\n\t\t\t% \t\t\t\t\\sqrt{1+\\rhok}\n\t\t\t% \t\t\t,\n\t\t\t% \t\t\t\\numberthis\\label{eq:rhok:plus}\n\t\t\t% \t\t\t\\end{align*}\n\t\t\t% \t\t\twhere the first inequality holds by \\cref{cond:alphk}, the second is due to the first term in \\eqref{eq:adabim:alphk*_0}, and the last inequality follows from \\eqref{eq:sigk}.\n","sha256_utf8":"ebba6540fb9dc8f3d9042dc6ce28c533c4d80f2a4c37d0916f4749383535c9a3"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"证明首种情形收敛到最优点","source_units":[{"start":104456,"end":105408,"start_line":2437,"end_line":2441,"raw_tex":"\t\t\t\t\t\tNotice that \\(\\limsup_{k\\to\\infty}(1+\\rhok-\\rhok*^2)>0\\), for otherwise \\(1+\\rhok-\\rhok*^2 \\to 0\\), implying that \\(\\liminf_{k \\to \\infty} \\rhok >0\\), and consequently that \\(\\alphk\\sigk\\) eventually increases exponentially, which contradicts \\(\\sigk\\alphk \\leq \\sigk\\alpha_{\\rm max} \\to 0\\).\n\t\t\t\t\t\tHence, \\(\\limsup_{k\\to\\infty}(1+\\rhok-\\rhok*^2)>0\\), which along with \\eqref{eq:liminf:cost1} implies that \\(\\liminf_{k\\to\\infty}\\cost_1(x_{k-1})=\\costinf*\\).\n\t\t\t\t\t\tTherefore, an optimal limit point exists, be it \\(x_\\infty\\).\n\t\t\t\t\t\tSince \\(W_k\\to0\\) by assertion \\ref{thm:recipe:rate} and since \\(\\mathcal L_k(x_\\infty)\\) converges, it follows that \\(\\frac12\\|x_k-x_\\infty\\|^2 = \\mathcal L_k(x_\\infty) - W_k\\) converges as well.\n\t\t\t\t\t\tSince along a subsequence \\(\\frac12\\|x_k-x_\\infty\\|^2\\) converges to zero, necessarily \\(\\lim_{k\\to\\infty}\\frac12\\|x_k-x_\\infty\\|^2=0\\), proving that the entire sequence \\(\\seq{x_k}\\) converges to \\(x_\\infty\\).\n","sha256_utf8":"6cdb0f09316e35616bfcc66b850cb97e5b9782718db84677e21d28a442334a58"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"证明第二种情形子列最优","source_units":[{"start":105408,"end":106432,"start_line":2442,"end_line":2457,"raw_tex":"\t\t\t\t\t\\item\n\t\t\t\t\t\t{\\it Case 2: \\(\\cost_1(x_k)<\\costinf*\\) holds infinitely often.}\n\n\t\t\t\t\t\tRecall the index \\(i_k\\) and the set \\(K'\\) defined in \\eqref{eq:ik} and \\eqref{eq:K'}.\n\t\t\t% \t\t\tOnce we establish that all the limit points of \\(\\seq{x_{i_k}}\\) and \\(\\seq{x_{i_k+1}}\\) are optimal, {\\red it only remains to show that\n\t\t\t% \t\t\t\\(\n\t\t\t% \t\t\t\t\\dist(x_{k'},\\X_1)\n\t\t\t% \t\t\t\\to\n\t\t\t% \t\t\t\t0\n\t\t\t% \t\t\t\\)\n\t\t\t% \t\t\tas \\(K'\\ni k'\\to\\infty\\)\\note{Why}}.\n\t\t\t\t\t\tHaving established boundedness of the entire sequence, limit points of \\(\\seq{x_{i_k}}\\) exist and, as shown above, belong to \\(\\X_2\\).\n\t\t\t\t\t\tMoreover, it follows from \\eqref{eq:ik} that \\(\\limsup_{k\\to\\infty}\\cost_1(x_{i_k})\\leq\\costinf*\\), and a lower semicontinuity argument then yields that all the limit points of \\(\\seq{x_{i_k}}\\) attain the optimal cost and are therefore optimal.\n\t\t\t\t\t\tSince \\(x_{i_k}-x_{i_k+1}\\to0\\) by \\cref{thm:PG:barPhi}, the same is also true for \\(\\seq{x_{i_k+1}}\\), and in particular\n\t\t\t\t\t\t\\(\\dist(x_{i_{k}+1},\\X_1) \\to 0\\) as \\(k \\to \\infty\\).\n\n","sha256_utf8":"24bcfe3e599fc5edb1aab558228a992ff9cec4965a7b8acc8d9ef17424017696"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"投影比较推出全序列逼近解集","source_units":[{"start":106432,"end":107765,"start_line":2458,"end_line":2495,"raw_tex":"\t\t\t\t\t\tFor each \\(k\\) let \\(\\bar x_k\\coloneqq\\proj_{\\X_1}\\!x_k\\), which is well defined since \\(\\X_1\\neq\\emptyset\\) is closed and convex.\n\t\t\t\t\t\tRecalling that \\(\\mathcal L_k(x_\\star)=W_k + \\tfrac12\\|x_k-x_\\star\\|^2\\), for \\(k'\\in K'\\) we have\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\tfrac12\\dist(x_{k'},\\X_1)^2\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\tfrac12\\|x_{k'}-\\bar x_{i_{k'}+1}\\|^2\n\t\t\t\t\t\t={}\n\t\t\t\t\t\t\t\\mathcal L_{k'}(\\bar x_{i_{k'}+1})-W_{k'}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\t\\dueto{\\scriptsize by \\eqref{eq:Lk'descent}}\n\t\t\t\t\t\t\t\\\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\mathcal L_{i_{k'}+2}(\\bar x_{i_{k'}+1})\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\tW_{k'}\n\t\t\t\t\t\t={}\n\t\t\t\t\t\t\t% \\tfrac12\\dist(x_{i_{k'}+2},\\X_1)^2\n\t\t\t\t\t\t\t\\tfrac12\\|x_{i_{k'}+2} - \\bar x_{i_{k'}+1}\\|^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\tW_{i_{k'}+2}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\tW_{k'}\n\t\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\leq{} &\n\t\t\t\t\t\t\t\\tfrac12\\big(\\dist(x_{i_{k'}+1},\\X_1) + \\|x_{i_{k'}+1} - x_{i_{k'}+2}\\|\\big)^2\n\t\t\t\t\t\t\t+\n\t\t\t\t\t\t\tW_{i_{k'}+2}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\tW_{k'}\n\t\t\t\t\t\t\\to\n\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\tas \\(K'\\ni k'\\to\\infty\\), where the limit follows from \\cref{thm:PG:barPhi}, the vanishing of \\(\\seq{W_k}\\) established in assertion \\ref{thm:recipe:rate}, and the fact that \\(i_k\\to\\infty\\) as \\(k\\to\\infty\\).\n\t\t\t\t\t\tSince \\(K'\\cup\\set{i_k,i_k+1}[k\\in\\N]=\\N\\), we conclude that all the limit points of \\(\\seq{x^k}\\) are optimal.\n\t\t\t\t\t\\qedhere\n\t\t\t\t\t\\end{proofitemize}\n\t\t\t\t\\end{proofitemize}\n\t\t\t","sha256_utf8":"82fbc25a901296a3e6ce0043147b0ac6aff26d125f948086182f3d9bfde42e82"}]},{"id":"S051","semantic_type":"external_fact","kind":"external_fact","title":"标准凸近端最优性","source_units":[],"declared_external":true,"declaration_text":["For proper lower semicontinuous convex h, z=prox_{alpha h}(x) iff (x-z)/alpha belongs to partial h(z). Convex subgradients satisfy the supporting-hyperplane inequality; adding a finite differentiable convex function adds its gradient to the subdifferential. Consequently, proximal-gradient fixed points minimize the corresponding convex composite objective."]},{"id":"S052","semantic_type":"external_fact","kind":"external_fact","title":"已引局部算子估计","source_units":[],"declared_external":true,"declaration_text":["The forward-operator norm identity, the expression and bounds for M_k, and the local Lipschitz estimates stated in B0044 are the cited boundary facts from Latafat et al., Lemma 2.1."]},{"id":"S053","semantic_type":"external_fact","kind":"external_fact","title":"凸函数水平有界判据","source_units":[],"declared_external":true,"declaration_text":["If a proper lower semicontinuous convex function on a finite-dimensional space has a nonempty bounded sublevel set, then all its sublevel sets are bounded; this is the criterion invoked at lines 2145–2146."]},{"id":"S054","semantic_type":"external_fact","kind":"external_fact","title":"参数极小点稳定性","source_units":[],"declared_external":true,"declaration_text":["Under the continuity and local uniform level-boundedness hypotheses invoked by the author for h(w;x,alpha,sigma), limits of minimizers along convergent parameter sequences minimize the limiting objective; Rockafellar–Wets, Theorem 1.17."]},{"id":"S055","semantic_type":"external_fact","kind":"external_fact","title":"标准有限维分析工具","source_units":[],"declared_external":true,"declaration_text":["Use finite-dimensional subsequence compactness, lower semicontinuity along convergent sequences, existence of projections onto nonempty closed convex sets, Cauchy–Schwarz and Young inequalities, squared-norm identities and triangle inequalities, and elementary induction, monotone convergence, and telescoping arguments."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S016","edges":[{"source_id":"S001","target_id":"S003","kind":"supports"},{"source_id":"S001","target_id":"S011","kind":"supports"},{"source_id":"S002","target_id":"S008","kind":"supports"},{"source_id":"S002","target_id":"S011","kind":"supports"},{"source_id":"S002","target_id":"S014","kind":"supports"},{"source_id":"S003","target_id":"S018","kind":"supports"},{"source_id":"S003","target_id":"S020","kind":"supports"},{"source_id":"S003","target_id":"S016","kind":"supports"},{"source_id":"S004","target_id":"S010","kind":"supports"},{"source_id":"S005","target_id":"S038","kind":"supports"},{"source_id":"S005","target_id":"S042","kind":"supports"},{"source_id":"S006","target_id":"S050","kind":"supports"},{"source_id":"S007","target_id":"S012","kind":"supports"},{"source_id":"S007","target_id":"S023","kind":"supports"},{"source_id":"S007","target_id":"S043","kind":"supports"},{"source_id":"S008","target_id":"S012","kind":"supports"},{"source_id":"S008","target_id":"S022","kind":"supports"},{"source_id":"S008","target_id":"S042","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S013","kind":"supports"},{"source_id":"S009","target_id":"S024","kind":"supports"},{"source_id":"S009","target_id":"S029","kind":"supports"},{"source_id":"S010","target_id":"S015","kind":"supports"},{"source_id":"S010","target_id":"S032","kind":"supports"},{"source_id":"S010","target_id":"S033","kind":"supports"},{"source_id":"S011","target_id":"S018","kind":"supports"},{"source_id":"S011","target_id":"S022","kind":"supports"},{"source_id":"S011","target_id":"S028","kind":"supports"},{"source_id":"S011","target_id":"S033","kind":"supports"},{"source_id":"S011","target_id":"S034","kind":"supports"},{"source_id":"S011","target_id":"S043","kind":"supports"},{"source_id":"S012","target_id":"S023","kind":"supports"},{"source_id":"S012","target_id":"S045","kind":"supports"},{"source_id":"S013","target_id":"S027","kind":"supports"},{"source_id":"S013","target_id":"S043","kind":"supports"},{"source_id":"S013","target_id":"S045","kind":"supports"},{"source_id":"S014","target_id":"S022","kind":"supports"},{"source_id":"S014","target_id":"S024","kind":"supports"},{"source_id":"S014","target_id":"S028","kind":"supports"},{"source_id":"S015","target_id":"S020","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S032","kind":"supports"},{"source_id":"S015","target_id":"S033","kind":"supports"},{"source_id":"S015","target_id":"S034","kind":"supports"},{"source_id":"S015","target_id":"S035","kind":"supports"},{"source_id":"S015","target_id":"S042","kind":"supports"},{"source_id":"S015","target_id":"S043","kind":"supports"},{"source_id":"S015","target_id":"S044","kind":"supports"},{"source_id":"S015","target_id":"S045","kind":"supports"},{"source_id":"S015","target_id":"S047","kind":"supports"},{"source_id":"S015","target_id":"S048","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S033","kind":"supports"},{"source_id":"S018","target_id":"S036","kind":"supports"},{"source_id":"S018","target_id":"S039","kind":"supports"},{"source_id":"S018","target_id":"S044","kind":"supports"},{"source_id":"S018","target_id":"S050","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S041","kind":"supports"},{"source_id":"S020","target_id":"S042","kind":"supports"},{"source_id":"S020","target_id":"S043","kind":"supports"},{"source_id":"S020","target_id":"S044","kind":"supports"},{"source_id":"S020","target_id":"S045","kind":"supports"},{"source_id":"S020","target_id":"S047","kind":"supports"},{"source_id":"S020","target_id":"S048","kind":"supports"},{"source_id":"S020","target_id":"S049","kind":"supports"},{"source_id":"S020","target_id":"S050","kind":"supports"},{"source_id":"S022","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S028","kind":"supports"},{"source_id":"S026","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S024","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S022","target_id":"S019","kind":"supports"},{"source_id":"S023","target_id":"S019","kind":"supports"},{"source_id":"S024","target_id":"S019","kind":"supports"},{"source_id":"S025","target_id":"S019","kind":"supports"},{"source_id":"S026","target_id":"S019","kind":"supports"},{"source_id":"S027","target_id":"S019","kind":"supports"},{"source_id":"S028","target_id":"S019","kind":"supports"},{"source_id":"S029","target_id":"S019","kind":"supports"},{"source_id":"S030","target_id":"S019","kind":"supports"},{"source_id":"S031","target_id":"S019","kind":"supports"},{"source_id":"S024","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S033","target_id":"S036","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S034","target_id":"S038","kind":"supports"},{"source_id":"S032","target_id":"S039","kind":"supports"},{"source_id":"S034","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S033","target_id":"S040","kind":"supports"},{"source_id":"S032","target_id":"S040","kind":"supports"},{"source_id":"S034","target_id":"S040","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S032","target_id":"S021","kind":"supports"},{"source_id":"S033","target_id":"S021","kind":"supports"},{"source_id":"S034","target_id":"S021","kind":"supports"},{"source_id":"S035","target_id":"S021","kind":"supports"},{"source_id":"S036","target_id":"S021","kind":"supports"},{"source_id":"S037","target_id":"S021","kind":"supports"},{"source_id":"S038","target_id":"S021","kind":"supports"},{"source_id":"S039","target_id":"S021","kind":"supports"},{"source_id":"S040","target_id":"S021","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S044","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S049","kind":"supports"},{"source_id":"S037","target_id":"S049","kind":"supports"},{"source_id":"S040","target_id":"S050","kind":"supports"},{"source_id":"S044","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S041","target_id":"S017","kind":"supports"},{"source_id":"S042","target_id":"S017","kind":"supports"},{"source_id":"S043","target_id":"S017","kind":"supports"},{"source_id":"S044","target_id":"S017","kind":"supports"},{"source_id":"S045","target_id":"S017","kind":"supports"},{"source_id":"S046","target_id":"S017","kind":"supports"},{"source_id":"S047","target_id":"S017","kind":"supports"},{"source_id":"S048","target_id":"S017","kind":"supports"},{"source_id":"S049","target_id":"S017","kind":"supports"},{"source_id":"S050","target_id":"S017","kind":"supports"},{"source_id":"S051","target_id":"S007","kind":"supports"},{"source_id":"S051","target_id":"S023","kind":"supports"},{"source_id":"S051","target_id":"S024","kind":"supports"},{"source_id":"S051","target_id":"S025","kind":"supports"},{"source_id":"S051","target_id":"S026","kind":"supports"},{"source_id":"S051","target_id":"S029","kind":"supports"},{"source_id":"S051","target_id":"S042","kind":"supports"},{"source_id":"S051","target_id":"S043","kind":"supports"},{"source_id":"S052","target_id":"S013","kind":"supports"},{"source_id":"S053","target_id":"S038","kind":"supports"},{"source_id":"S054","target_id":"S042","kind":"supports"},{"source_id":"S055","target_id":"S025","kind":"supports"},{"source_id":"S055","target_id":"S027","kind":"supports"},{"source_id":"S055","target_id":"S032","kind":"supports"},{"source_id":"S055","target_id":"S035","kind":"supports"},{"source_id":"S055","target_id":"S040","kind":"supports"},{"source_id":"S055","target_id":"S042","kind":"supports"},{"source_id":"S055","target_id":"S046","kind":"supports"},{"source_id":"S055","target_id":"S048","kind":"supports"},{"source_id":"S055","target_id":"S049","kind":"supports"},{"source_id":"S055","target_id":"S050","kind":"supports"}]}}
{"record_id":"2305.05828v3","paper":{"title":"A Normal Map-Based Proximal Stochastic Gradient Method: Convergence and Identification Properties","authors":["Junwen Qiu","Li Jiang","Andre Milzarek"],"arxiv_id":"2305.05828","version":"v3","journal":"SIAM Journal on Optimization","doi":"10.1137/25m1757332","abstract_url":"https://arxiv.org/abs/2305.05828v3","pdf_url":"https://arxiv.org/pdf/2305.05828v3"},"context":{"raw_tex":"In this work, we propose and investigate a novel normal map-based proximal stochastic gradient method for the composite problem\n%\n\\begin{equation}\n\t\\label{eq:SO} \\min_{x\\in \\Rn}~\\psi(x) := f(x) + \\vp(x),\n\\end{equation}\n%\nwhere $\\vp: \\Rn \\to \\Rex$ is a convex, lower semicontinuous (lsc.), and proper mapping and $f: \\Rn \\to \\R$ is continuously differentiable on an open set containing the effective domain $\\dom{\\vp} := \\{x \\in \\Rn: \\vp(x) < \\infty\\}$. \n\n\nwhere the sequence $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}\\subseteq \\R_{++}$ denotes the selected step sizes, $g^k$ is a (stochastic) approximation of $\\nabla f(x^k)$, and the well-known proximity operator of $\\vp$ is defined as $\\prox{\\alpha \\vp}(x) := {\\argmin}_{y\\in\\Rn} \\, \\vp(y) + \\frac{1}{2\\alpha}\\|x-y\\|^2$, $\\alpha >0$. \\vspace{1mm}   \n\n\n    \\begin{enumerate}[label=\\textup{\\textrm{(H.\\arabic*)}},leftmargin=3em,topsep=0ex,itemsep=1ex,partopsep=0ex]\n        \\item \\label{A1} The gradient $\\nabla f : \\Rn \\to \\Rn$ is Lipschitz continuous on $\\dom{\\vp}$ with modulus $\\sL>0$ and $\\vp$ is convex, lsc., and proper.\n        \\item \\label{A2} The objective function $\\psi$ is bounded from below, i.e., $\\bar \\psi := \\inf_{x \\in \\Rn}\\,\\psi(x) > -\\infty$. \n    \\end{enumerate}\n\n\nIn addition, we assume that there is a filtered probability space $(\\Omega,\\mathcal F,\\revise{\\{\\mathcal F_k\\}_{k\\in\\N}},\\Prob)$ that can describe the stochastic components of the update \\cref{eq:update-PSGD} and of other stochastic algorithms studied in this work in a unified way. In particular, each approximation $g^k$ is understood as a realization of an $\\mathcal F_{k+1}$-measurable random vector $\\sg^k : \\Omega \\to \\Rn$. Thus, the stochastic process $\\revise{\\{\\sx^k\\}_{k\\in\\N}}$, associated with \\cref{eq:update-PSGD} (or other stochastic methods), is adapted to the filtration $\\revise{\\{\\mathcal F_k\\}_{k\\in\\N}}$. We further define the stochastic error terms $\\se^k : \\Omega \\to \\Rn$ via $\\se^k := \\sg^k - \\nabla f(\\sx^k)$ and set $\\Exp_k[\\cdot] := \\Exp[\\cdot | \\cF_k]$. We mainly work with the following conditions on the errors $\\revise{\\{\\se^k\\}_{k\\in\\N}}$ and step sizes $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}$:\n%\n\\vspace{1.5mm}\n%\n\\begin{enumerate}[label=\\textup{\\textrm{(S.\\arabic*)}},leftmargin=3em,topsep=0ex,itemsep=1ex,partopsep=0ex]\n        \\item \\label{B1} The stochastic error $\\se^k$ has mean zero, i.e., $\\Exp_k[\\se^k]=0$ almost surely (a.s.).\n        \\item \\label{B2} There exists $\\revise{\\{\\sigma_k\\}_{k\\in\\N}}\\subseteq \\R_+$ such that $\\Exp_k[\\|\\se^k\\|^2]\\leq \\sigma_k^2$ a.s. for all $k$. \n        \\item \\label{B3} The step sizes $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}$ satisfy $\\alpha_k\\to0$, $\\sum_{k=0}^\\infty \\alpha_k = \\infty$, and $\\sum_{k=0}^{\\infty}\\alpha_k^2\\sigma_k^2<\\infty$.\n    \\end{enumerate}\n\n\nLet $\\lambda>0$, $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}\\subseteq \\R_{++}$ be given. Choose $z^0\\in\\Rn$ and set $x^0=\\proxl(z^0)$. \\\\[1mm]\n\n\n\\be\\label{eq:update-short}\n    \\sz^{k+1} = \\sz^k - \\alpha_k(\\sg^k + \\lambda^{-1}(\\sz^k-\\sx^k)) \\quad \\text{and} \\quad \\sx^{k+1} = \\proxl(\\sz^{k+1}),\n\\ee\n\nwhere $\\sg^k$ is a (unbiased) stochastic approximation of $\\nabla f(\\sx^k)$.\n\n\nWe now briefly discuss common stationarity measures for proximal gradient-type methods and for the problem \\cref{eq:SO}. The first-order optimality conditions associated with \\cref{eq:SO} are given by $0 \\in \\partial \\psi(x) = \\nabla f(x) + \\partial \\vp(x)$. Here, $\\partial \\psi$ denotes the limiting subdifferential of $\\psi$ and $\\partial \\vp$ is the standard subdifferential for convex functions, cf. \\cite{rocwet98,baucom11}. A point $\\bar x \\in \\dom{\\vp}$ is called a stationary point of \\cref{eq:SO} if $0 \\in \\partial \\psi(\\bar x)$ and we use $\\crit(\\psi)$ to denote the set of all stationary points of $\\psi$. We further introduce the notation $\\|\\partial \\psi(x)\\|_{-} := \\dist(0,\\partial\\psi(x)) := {\\min}_{v\\in\\partial \\psi(x)} \\;\\|v\\|$. A point $\\bar x \\in \\dom{\\vp}$ is an $\\varepsilon$-stationary point of $\\psi$, $\\varepsilon \\geq 0$, if we have $\\|\\partial\\psi(\\bar x)\\|_- \\leq \\varepsilon$. %\\\\[-2.5ex]\n\n\n\\[ x \\in \\crit(\\psi) \\quad \\iff \\quad \\Fnat(x) := \\lambda^{-1}(x-\\proxl(x-\\lambda \\nabla f(x))) = 0, \\quad \\lambda > 0. \\]\n\n\\begin{equation} \\label{eq:norm-subdiff} \\Fnor(z) = \\nabla f(\\proxl(z)) + \\lambda^{-1}(z-\\proxl(z)) \\in \\partial\\psi(\\proxl(z)) \\quad \\forall~z\\in\\Rn. \\end{equation}\n\n\\begin{equation} \\label{eq:stat-am} \\|\\Fnat(x)\\| \\leq \\|\\partial\\psi(x)\\|_- \\quad \\text{and} \\quad \\|\\partial\\psi(\\prox{\\lambda\\vp}(z))\\|_- \\leq \\|\\Fnor(z)\\| \\end{equation}\n\nWe first present several basic estimates and properties that facilitate the analysis of $\\NSGD$. Since $\\vp$ is assumed to be convex, lsc., and proper (cf$.$ \\ref{A1}), the proximity operator $\\proxl$ is firmly nonexpansive, \\cite{mor65,baucom11}. Hence, recalling $\\sx^k = \\proxl(\\sz^k)$, it holds that\n%\n\\begin{equation}\\label{prox-prop}\n    \\|\\sx^n-\\sx^m\\|^2 \\leq \\iprod{\\sz^n-\\sz^m}{\\sx^n-\\sx^m}\\leq \\norm{\\sz^n-\\sz^m}^2 \\quad \\forall~\\revise{m<n} \\quad \\text{(a$.$s$.$)}.\n\\end{equation}\n\n\\begin{align}\n    \\nonumber \\sz^n&=\\sz^{n-1} -\\alpha_{n-1}[\\sg^{n-1}+ \\tfrac{1}{\\lambda}(\\sz^{n-1} - \\sx^{n-1})] =\\sz^{n-1} - \\alpha_{n-1}[\\Fnor(\\sz^{n-1}) +\\se^{n-1}]\t\\\\\n     \\nonumber &=\\sz^m - {\\sum}_{i=m}^{n-1}\\alpha_i\\Fnor(\\sz^i) - {\\sum}_{i=m}^{n-1} \\alpha_i\\se^i \\\\\n    \\nonumber &=\\sz^m - {{\\sum}_{i=m}^{n-1}\\alpha_i}\\Fnor(\\sz^m) - {\\sum}_{i=m}^{n-1} \\alpha_i[\\Fnor(\\sz^i)-\\Fnor(\\sz^m)] - {\\sum}_{i=m}^{n-1} \\alpha_i\\se^i \\\\ & = \\sz^m - \\tau_{m,n} \\Fnor(\\sz^m) + \\scp^{m,n} \\revise{\\quad \\forall~m<n,}\\label{eq:pretty-important}\n\\end{align}\n\nwhere $\\tau_{m,n}:= {\\sum}_{i=m}^{n-1}\\alpha_i$ and $\\scp^{m,n}:=-{\\sum}_{i=m}^{n-1} \\alpha_i[\\Fnor(\\sz^i)-\\Fnor(\\sz^m)] - {\\sum}_{i=m}^{n-1} \\alpha_i\\se^i$.\n\n\n\\[\n\\scs_{m,n} :=\\max_{m<j\\leq n}\\Big\\|{\\sum}_{i=m}^{j-1} \\alpha_i \\se^i\\Big\\|.\n\\]\n\n\\begin{align}\n         \\su^{m,n}  & :=  \\lambda^{-1}(\\sz^n-\\sz^m) + \\Fnor(\\sz^m) \\label{eq:u-def1}\\\\  &\\;=  (\\lambda^{-1}-\\tau_{m,n}^{-1})(\\sz^n-\\sz^m) + \\tau_{m,n}^{-1} \\scp^{m,n} \\label{eq:u-def2}\\\\\n          &\\;=  (1-\\lambda^{-1}\\tau_{m,n}) \\Fnor(\\sz^m) + \\lambda^{-1}\\scp^{m,n}, \\quad \\revise{m < n}.\\label{eq:u-def3}\n     \\end{align}\n\n\\begin{definition}[Merit function]\\label{def:mer-fun} Given $\\lambda > 0$, we define the merit function: \n \\[\\mer(z):=\\psi(\\prox{\\lambda\\vp}(z)) + \\frac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2,\\quad \\text{where} \\quad \\xi:={(2+2\\lambda^2 \\sL^2)^{-1}}.\\]\n\\end{definition}\n\n\\begin{definition}[Time indices]\\label{def:time win}\nLet $\\sT>0$ and $\\delta\\in(0,1)$ be given. We introduce the mapping $\\varpi:\\N \\to \\N\\cup \\{\\pm\\infty\\}$, $\\varpi(k):=\\sup\\{n > k: \\tau_{k,n} \\leq \\sT\\}$, where $\\tau_{m,n} = {\\sum}_{k=m}^{n-1} \\alpha_k$. The \\emph{time indices} $\\revise{\\{t_k\\}_{k\\in\\N}}$, associated with the \\emph{time window} $\\sT$ \\revise{(and the step sizes $\\{\\alpha_k\\}_{k \\in \\N}$)}, are defined as \n\\[\nt_0 := \\inf\\{i: \\alpha_j \\leq (1-\\delta)\\sT \\text{ for all $j\\geq i$}\\}\\quad \\text{and} \\quad t_{k+1} := \\varpi(t_k).\n\\]\n\\end{definition}\n\n\\begin{equation}\\label{index-def}\n\\fk(i) = \\max\\{k: t_k < i\\}\\quad \\text{for all $i>t_0$}.\n\\end{equation}\n\nIt is evident that we have $t_{\\fk(i)}<i\\leq t_{\\fk(i)+1}$ for all $i>t_0$.\n\n\n\\begin{equation}\n    \\label{eq:def delta and T}\n    \\sr_k := \\scs_{t_{k},t_{k+1}}, \\;\\; \\delta := \\frac45, \\;\\;  \\sT := \\frac{\\sqrt{\\xi}}{8\\Llambda},\\quad \\text{where} \\quad \\Llambda =\\sL+\\frac{2}{\\lambda}, \\;\\; \\xi =\\frac{1}{2+2\\lambda^2 \\sL^2}.\n\\end{equation}\n\n\\begin{enumerate}[label=\\textup{\\textrm{(H.3)}},leftmargin=3em,topsep=2mm,itemsep=1ex,partopsep=0ex]\n        \\item \\label{A3} The objective function $\\psi : \\Rn \\to \\Rex$ is definable. %\\vspace{2mm}\n\\end{enumerate}\n\n\n\\[ \\cX := \\{\\omega\\in\\Omega: {\\liminf}_{k\\to\\infty}\\; \\|\\sx^k(\\omega)\\| < \\infty\\}. %\\vspace{-.25ex} \n\\]\n\nClearly, if $\\omega\\in\\cX$, then the realization $\\revise{\\{x^k\\}_{k\\in\\N}} = \\revise{\\{\\sx^k(\\omega)\\}_{k\\in\\N}}$ has at least one bounded subsequence and accumulation point. We will work with the following assumption: \n%\\vspace{1.5mm}\n \\begin{enumerate}[label=\\textup{\\textrm{(S.4)}},leftmargin=3em,topsep=1.5mm,itemsep=1ex,partopsep=0ex]\n        \\item \\label{B4} The event $\\cX \\in \\mathcal F$ occurs with probability $1$, i.e., $\\Prob(\\cX)=1$.\n    \\end{enumerate}\n","units":[{"start":12150,"end":12604,"start_line":354,"end_line":360,"raw_tex":"In this work, we propose and investigate a novel normal map-based proximal stochastic gradient method for the composite problem\n%\n\\begin{equation}\n\t\\label{eq:SO} \\min_{x\\in \\Rn}~\\psi(x) := f(x) + \\vp(x),\n\\end{equation}\n%\nwhere $\\vp: \\Rn \\to \\Rex$ is a convex, lower semicontinuous (lsc.), and proper mapping and $f: \\Rn \\to \\R$ is continuously differentiable on an open set containing the effective domain $\\dom{\\vp} := \\{x \\in \\Rn: \\vp(x) < \\infty\\}$. \n","sha256_utf8":"198c6cf2bbfd40a2d23b1322c713c1b0a743b19eba4600b945ebb415aabbb99a"},{"start":14223,"end":14559,"start_line":376,"end_line":376,"raw_tex":"where the sequence $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}\\subseteq \\R_{++}$ denotes the selected step sizes, $g^k$ is a (stochastic) approximation of $\\nabla f(x^k)$, and the well-known proximity operator of $\\vp$ is defined as $\\prox{\\alpha \\vp}(x) := {\\argmin}_{y\\in\\Rn} \\, \\vp(y) + \\frac{1}{2\\alpha}\\|x-y\\|^2$, $\\alpha >0$. \\vspace{1mm}   \n","sha256_utf8":"d34619e4fd665e10b21a2a8493baad24f500449e36276710f9391b8af93bbf83"},{"start":14664,"end":15095,"start_line":380,"end_line":383,"raw_tex":"    \\begin{enumerate}[label=\\textup{\\textrm{(H.\\arabic*)}},leftmargin=3em,topsep=0ex,itemsep=1ex,partopsep=0ex]\n        \\item \\label{A1} The gradient $\\nabla f : \\Rn \\to \\Rn$ is Lipschitz continuous on $\\dom{\\vp}$ with modulus $\\sL>0$ and $\\vp$ is convex, lsc., and proper.\n        \\item \\label{A2} The objective function $\\psi$ is bounded from below, i.e., $\\bar \\psi := \\inf_{x \\in \\Rn}\\,\\psi(x) > -\\infty$. \n    \\end{enumerate}\n","sha256_utf8":"9d80ee432f4dcace3b2588536ece1e9505549eed90153ede8c21d4cdd3d121bf"},{"start":15115,"end":16639,"start_line":386,"end_line":394,"raw_tex":"In addition, we assume that there is a filtered probability space $(\\Omega,\\mathcal F,\\revise{\\{\\mathcal F_k\\}_{k\\in\\N}},\\Prob)$ that can describe the stochastic components of the update \\cref{eq:update-PSGD} and of other stochastic algorithms studied in this work in a unified way. In particular, each approximation $g^k$ is understood as a realization of an $\\mathcal F_{k+1}$-measurable random vector $\\sg^k : \\Omega \\to \\Rn$. Thus, the stochastic process $\\revise{\\{\\sx^k\\}_{k\\in\\N}}$, associated with \\cref{eq:update-PSGD} (or other stochastic methods), is adapted to the filtration $\\revise{\\{\\mathcal F_k\\}_{k\\in\\N}}$. We further define the stochastic error terms $\\se^k : \\Omega \\to \\Rn$ via $\\se^k := \\sg^k - \\nabla f(\\sx^k)$ and set $\\Exp_k[\\cdot] := \\Exp[\\cdot | \\cF_k]$. We mainly work with the following conditions on the errors $\\revise{\\{\\se^k\\}_{k\\in\\N}}$ and step sizes $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}$:\n%\n\\vspace{1.5mm}\n%\n\\begin{enumerate}[label=\\textup{\\textrm{(S.\\arabic*)}},leftmargin=3em,topsep=0ex,itemsep=1ex,partopsep=0ex]\n        \\item \\label{B1} The stochastic error $\\se^k$ has mean zero, i.e., $\\Exp_k[\\se^k]=0$ almost surely (a.s.).\n        \\item \\label{B2} There exists $\\revise{\\{\\sigma_k\\}_{k\\in\\N}}\\subseteq \\R_+$ such that $\\Exp_k[\\|\\se^k\\|^2]\\leq \\sigma_k^2$ a.s. for all $k$. \n        \\item \\label{B3} The step sizes $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}$ satisfy $\\alpha_k\\to0$, $\\sum_{k=0}^\\infty \\alpha_k = \\infty$, and $\\sum_{k=0}^{\\infty}\\alpha_k^2\\sigma_k^2<\\infty$.\n    \\end{enumerate}\n","sha256_utf8":"82e8f55db135ad101755fe6fd7952d2268261eceeeaed40d992fef88fd29ec6c"},{"start":22953,"end":23084,"start_line":427,"end_line":427,"raw_tex":"Let $\\lambda>0$, $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}\\subseteq \\R_{++}$ be given. Choose $z^0\\in\\Rn$ and set $x^0=\\proxl(z^0)$. \\\\[1mm]\n","sha256_utf8":"ef66c340c1f8227414503a689bc82e2c1755bc5f1e7b87782dddea0c6d4681ee"},{"start":29442,"end":29594,"start_line":493,"end_line":495,"raw_tex":"\\be\\label{eq:update-short}\n    \\sz^{k+1} = \\sz^k - \\alpha_k(\\sg^k + \\lambda^{-1}(\\sz^k-\\sx^k)) \\quad \\text{and} \\quad \\sx^{k+1} = \\proxl(\\sz^{k+1}),\n\\ee","sha256_utf8":"547f15e2948261f8d9e524f40dbe14dfc143839962c7f9e87a645eb4104a69b1"},{"start":29597,"end":29674,"start_line":497,"end_line":497,"raw_tex":"where $\\sg^k$ is a (unbiased) stochastic approximation of $\\nabla f(\\sx^k)$.\n","sha256_utf8":"4d34e269514fa2266a4a2872e9076a6069030d14d6e8dcf970f015bb1ac9771a"},{"start":33916,"end":34837,"start_line":525,"end_line":525,"raw_tex":"We now briefly discuss common stationarity measures for proximal gradient-type methods and for the problem \\cref{eq:SO}. The first-order optimality conditions associated with \\cref{eq:SO} are given by $0 \\in \\partial \\psi(x) = \\nabla f(x) + \\partial \\vp(x)$. Here, $\\partial \\psi$ denotes the limiting subdifferential of $\\psi$ and $\\partial \\vp$ is the standard subdifferential for convex functions, cf. \\cite{rocwet98,baucom11}. A point $\\bar x \\in \\dom{\\vp}$ is called a stationary point of \\cref{eq:SO} if $0 \\in \\partial \\psi(\\bar x)$ and we use $\\crit(\\psi)$ to denote the set of all stationary points of $\\psi$. We further introduce the notation $\\|\\partial \\psi(x)\\|_{-} := \\dist(0,\\partial\\psi(x)) := {\\min}_{v\\in\\partial \\psi(x)} \\;\\|v\\|$. A point $\\bar x \\in \\dom{\\vp}$ is an $\\varepsilon$-stationary point of $\\psi$, $\\varepsilon \\geq 0$, if we have $\\|\\partial\\psi(\\bar x)\\|_- \\leq \\varepsilon$. %\\\\[-2.5ex]\n","sha256_utf8":"e2ead5bfb786a9396cb45e2583cf40c393eeafd026c10106e1c49e7785626502"},{"start":35018,"end":35140,"start_line":529,"end_line":529,"raw_tex":"\\[ x \\in \\crit(\\psi) \\quad \\iff \\quad \\Fnat(x) := \\lambda^{-1}(x-\\proxl(x-\\lambda \\nabla f(x))) = 0, \\quad \\lambda > 0. \\]","sha256_utf8":"ee30b40d5bda9340e4f2830bcf4363b8475de307780d6a3a668c27ee0825a39f"},{"start":35759,"end":35924,"start_line":538,"end_line":538,"raw_tex":"\\begin{equation} \\label{eq:norm-subdiff} \\Fnor(z) = \\nabla f(\\proxl(z)) + \\lambda^{-1}(z-\\proxl(z)) \\in \\partial\\psi(\\proxl(z)) \\quad \\forall~z\\in\\Rn. \\end{equation}","sha256_utf8":"c1c0f6bd9e221ef3304d5f3ae3c4a24a04b75ce25669fb2e1b63f0c7a16cdbda"},{"start":36337,"end":36509,"start_line":542,"end_line":542,"raw_tex":"\\begin{equation} \\label{eq:stat-am} \\|\\Fnat(x)\\| \\leq \\|\\partial\\psi(x)\\|_- \\quad \\text{and} \\quad \\|\\partial\\psi(\\prox{\\lambda\\vp}(z))\\|_- \\leq \\|\\Fnor(z)\\| \\end{equation}","sha256_utf8":"cf886f3ff670c02f2357faddfa27dcfdabad9fbac3c0d35f7efdbcaf476d373c"},{"start":42380,"end":42871,"start_line":572,"end_line":576,"raw_tex":"We first present several basic estimates and properties that facilitate the analysis of $\\NSGD$. Since $\\vp$ is assumed to be convex, lsc., and proper (cf$.$ \\ref{A1}), the proximity operator $\\proxl$ is firmly nonexpansive, \\cite{mor65,baucom11}. Hence, recalling $\\sx^k = \\proxl(\\sz^k)$, it holds that\n%\n\\begin{equation}\\label{prox-prop}\n    \\|\\sx^n-\\sx^m\\|^2 \\leq \\iprod{\\sz^n-\\sz^m}{\\sx^n-\\sx^m}\\leq \\norm{\\sz^n-\\sz^m}^2 \\quad \\forall~\\revise{m<n} \\quad \\text{(a$.$s$.$)}.\n\\end{equation}","sha256_utf8":"91d06bdc512dd60a9a816819342ebb8395d720d43b9f388a02a117e56aea2b07"},{"start":42969,"end":43517,"start_line":580,"end_line":584,"raw_tex":"\\begin{align}\n    \\nonumber \\sz^n&=\\sz^{n-1} -\\alpha_{n-1}[\\sg^{n-1}+ \\tfrac{1}{\\lambda}(\\sz^{n-1} - \\sx^{n-1})] =\\sz^{n-1} - \\alpha_{n-1}[\\Fnor(\\sz^{n-1}) +\\se^{n-1}]\t\\\\\n     \\nonumber &=\\sz^m - {\\sum}_{i=m}^{n-1}\\alpha_i\\Fnor(\\sz^i) - {\\sum}_{i=m}^{n-1} \\alpha_i\\se^i \\\\\n    \\nonumber &=\\sz^m - {{\\sum}_{i=m}^{n-1}\\alpha_i}\\Fnor(\\sz^m) - {\\sum}_{i=m}^{n-1} \\alpha_i[\\Fnor(\\sz^i)-\\Fnor(\\sz^m)] - {\\sum}_{i=m}^{n-1} \\alpha_i\\se^i \\\\ & = \\sz^m - \\tau_{m,n} \\Fnor(\\sz^m) + \\scp^{m,n} \\revise{\\quad \\forall~m<n,}\\label{eq:pretty-important}\n\\end{align}","sha256_utf8":"1e515b720b2f8ea0531a04db0220f502c08fceb28f210be061c277e53b05c816"},{"start":43520,"end":43678,"start_line":586,"end_line":586,"raw_tex":"where $\\tau_{m,n}:= {\\sum}_{i=m}^{n-1}\\alpha_i$ and $\\scp^{m,n}:=-{\\sum}_{i=m}^{n-1} \\alpha_i[\\Fnor(\\sz^i)-\\Fnor(\\sz^m)] - {\\sum}_{i=m}^{n-1} \\alpha_i\\se^i$.\n","sha256_utf8":"16ede40c04a9b858ce2c3e399a6e27d74007e218ba56826b8df35a69415a23e5"},{"start":43805,"end":43886,"start_line":590,"end_line":592,"raw_tex":"\\[\n\\scs_{m,n} :=\\max_{m<j\\leq n}\\Big\\|{\\sum}_{i=m}^{j-1} \\alpha_i \\se^i\\Big\\|.\n\\]","sha256_utf8":"28dc33e24776febb80c43b347557e2c2e52565cf56317159da52651480fc4390"},{"start":46035,"end":46373,"start_line":620,"end_line":623,"raw_tex":"\\begin{align}\n         \\su^{m,n}  & :=  \\lambda^{-1}(\\sz^n-\\sz^m) + \\Fnor(\\sz^m) \\label{eq:u-def1}\\\\  &\\;=  (\\lambda^{-1}-\\tau_{m,n}^{-1})(\\sz^n-\\sz^m) + \\tau_{m,n}^{-1} \\scp^{m,n} \\label{eq:u-def2}\\\\\n          &\\;=  (1-\\lambda^{-1}\\tau_{m,n}) \\Fnor(\\sz^m) + \\lambda^{-1}\\scp^{m,n}, \\quad \\revise{m < n}.\\label{eq:u-def3}\n     \\end{align}","sha256_utf8":"22c6fb6f8f2eeb5cb51e25a180748abf93e0247e842fa89beb9ba1abe075eff8"},{"start":48902,"end":49161,"start_line":650,"end_line":652,"raw_tex":"\\begin{definition}[Merit function]\\label{def:mer-fun} Given $\\lambda > 0$, we define the merit function: \n \\[\\mer(z):=\\psi(\\prox{\\lambda\\vp}(z)) + \\frac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2,\\quad \\text{where} \\quad \\xi:={(2+2\\lambda^2 \\sL^2)^{-1}}.\\]\n\\end{definition}","sha256_utf8":"b372819dba69078abbcb01cd35b0b153be721a37ebfff734443da03c68c9bf44"},{"start":52842,"end":53411,"start_line":701,"end_line":706,"raw_tex":"\\begin{definition}[Time indices]\\label{def:time win}\nLet $\\sT>0$ and $\\delta\\in(0,1)$ be given. We introduce the mapping $\\varpi:\\N \\to \\N\\cup \\{\\pm\\infty\\}$, $\\varpi(k):=\\sup\\{n > k: \\tau_{k,n} \\leq \\sT\\}$, where $\\tau_{m,n} = {\\sum}_{k=m}^{n-1} \\alpha_k$. The \\emph{time indices} $\\revise{\\{t_k\\}_{k\\in\\N}}$, associated with the \\emph{time window} $\\sT$ \\revise{(and the step sizes $\\{\\alpha_k\\}_{k \\in \\N}$)}, are defined as \n\\[\nt_0 := \\inf\\{i: \\alpha_j \\leq (1-\\delta)\\sT \\text{ for all $j\\geq i$}\\}\\quad \\text{and} \\quad t_{k+1} := \\varpi(t_k).\n\\]\n\\end{definition}","sha256_utf8":"fe7864be7f073298722b7decc2473092405f013aaa9a48fc9402eb83e597b387"},{"start":53768,"end":53873,"start_line":710,"end_line":712,"raw_tex":"\\begin{equation}\\label{index-def}\n\\fk(i) = \\max\\{k: t_k < i\\}\\quad \\text{for all $i>t_0$}.\n\\end{equation}","sha256_utf8":"34e07636aa872d5ade2f8b7226be0daddc562d58d718d0290b631f2c2cc9dcfe"},{"start":53876,"end":53952,"start_line":714,"end_line":714,"raw_tex":"It is evident that we have $t_{\\fk(i)}<i\\leq t_{\\fk(i)+1}$ for all $i>t_0$.\n","sha256_utf8":"b723ada71d7f7b48a5f9bff53d368cd0e03ac97c5bffc32e094761bbce8e5390"},{"start":55823,"end":56084,"start_line":734,"end_line":737,"raw_tex":"\\begin{equation}\n    \\label{eq:def delta and T}\n    \\sr_k := \\scs_{t_{k},t_{k+1}}, \\;\\; \\delta := \\frac45, \\;\\;  \\sT := \\frac{\\sqrt{\\xi}}{8\\Llambda},\\quad \\text{where} \\quad \\Llambda =\\sL+\\frac{2}{\\lambda}, \\;\\; \\xi =\\frac{1}{2+2\\lambda^2 \\sL^2}.\n\\end{equation}","sha256_utf8":"6abb8241167d3523b886fe17a84777e7e17c43de2da4542e8e2b9c935f61694b"},{"start":70069,"end":70284,"start_line":908,"end_line":910,"raw_tex":"\\begin{enumerate}[label=\\textup{\\textrm{(H.3)}},leftmargin=3em,topsep=2mm,itemsep=1ex,partopsep=0ex]\n        \\item \\label{A3} The objective function $\\psi : \\Rn \\to \\Rex$ is definable. %\\vspace{2mm}\n\\end{enumerate}\n","sha256_utf8":"3a427f5e17af18cff74c7bd6e1a4fc1e68d2db92e6e186d676a681e277395ddc"},{"start":77013,"end":77117,"start_line":963,"end_line":964,"raw_tex":"\\[ \\cX := \\{\\omega\\in\\Omega: {\\liminf}_{k\\to\\infty}\\; \\|\\sx^k(\\omega)\\| < \\infty\\}. %\\vspace{-.25ex} \n\\]","sha256_utf8":"133b898aad9c47e016adced155a258eaf20e0bc212df6664639d7ce3bab809ab"},{"start":77120,"end":77588,"start_line":966,"end_line":970,"raw_tex":"Clearly, if $\\omega\\in\\cX$, then the realization $\\revise{\\{x^k\\}_{k\\in\\N}} = \\revise{\\{\\sx^k(\\omega)\\}_{k\\in\\N}}$ has at least one bounded subsequence and accumulation point. We will work with the following assumption: \n%\\vspace{1.5mm}\n \\begin{enumerate}[label=\\textup{\\textrm{(S.4)}},leftmargin=3em,topsep=1.5mm,itemsep=1ex,partopsep=0ex]\n        \\item \\label{B4} The event $\\cX \\in \\mathcal F$ occurs with probability $1$, i.e., $\\Prob(\\cX)=1$.\n    \\end{enumerate}\n","sha256_utf8":"6aa10f26b1df8f2d36cfa0078faa8d12b18cad29ef852dbd639287564e5c2168"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\n\t\\label{thm:iter converge}\n\t Let \\ref{A1}--\\ref{A3}, \\ref{B1}--\\ref{B2}, and \\ref{B4} hold and let $\\revise{\\{\\sx^k\\}_{k\\in\\N}}$ be generated by $\\NSGD$ with step sizes $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}\\subset \\R_{++}$ satisfying: %\\vspace{-0.25ex}\n     %\n\t\\begin{equation}\\label{eq:kl-step}  \n\t\t\\lim_{k\\to\\infty}\\alpha_k=0,\\quad {\\sum}_{k=0}^\\infty \\alpha_k = \\infty \\quad \\text{and} \\quad {\\sum}_{k=0}^{\\infty} \\alpha_k^2 \\sigma_k^2 \\Big( {\\sum}_{i=0}^k\\alpha_i \\Big)^{\\mu}< \\infty,% \\quad \\text{for some $\\mu>\\frac12$}.\n        %\\vspace{-0.25ex}\n\t\\end{equation}\nfor some $\\mu>1$. Then, we have $\\sx^k \\to \\sx^*$ a$.$s$.$ for some $\\sx^*:\\Omega\\to\\crit(\\psi)$.\n\\end{theorem}","units":[{"start":78004,"end":78694,"start_line":977,"end_line":986,"raw_tex":"\\begin{theorem}\n\t\\label{thm:iter converge}\n\t Let \\ref{A1}--\\ref{A3}, \\ref{B1}--\\ref{B2}, and \\ref{B4} hold and let $\\revise{\\{\\sx^k\\}_{k\\in\\N}}$ be generated by $\\NSGD$ with step sizes $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}\\subset \\R_{++}$ satisfying: %\\vspace{-0.25ex}\n     %\n\t\\begin{equation}\\label{eq:kl-step}  \n\t\t\\lim_{k\\to\\infty}\\alpha_k=0,\\quad {\\sum}_{k=0}^\\infty \\alpha_k = \\infty \\quad \\text{and} \\quad {\\sum}_{k=0}^{\\infty} \\alpha_k^2 \\sigma_k^2 \\Big( {\\sum}_{i=0}^k\\alpha_i \\Big)^{\\mu}< \\infty,% \\quad \\text{for some $\\mu>\\frac12$}.\n        %\\vspace{-0.25ex}\n\t\\end{equation}\nfor some $\\mu>1$. Then, we have $\\sx^k \\to \\sx^*$ a$.$s$.$ for some $\\sx^*:\\Omega\\to\\crit(\\psi)$.\n\\end{theorem}","sha256_utf8":"c1c141f0fbfdcdd5b4d698baca8a3ec8c9cab29dbe4576ab644b6fc18bc16f81"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}[Iterate bounds]\n    \\label{lem:est-err}\n     Let \\ref{A1}--\\ref{A2} and \\ref{B1}--\\ref{B2} hold and let $\\revise{\\{\\sx^k\\}_{k\\in\\N}}$ and $\\revise{\\{\\sz^k\\}_{k\\in\\N}}$ be generated by $\\NSGD$. Let us set $\\Llambda := \\sL+2\\lambda^{-1}$. For all $0\\leq m<n$ with $\\tau_{m,n} \\leq 1/(2\\Llambda)$, we have\n     %\n    \\begin{align*}\n        \\max_{m<i\\leq n}\\|\\sx^i-\\sx^m\\| &\\leq \\max_{m<i\\leq n}\\|\\sz^i-\\sz^m\\| \\leq 2\\tau_{m,n} \\norm{\\Fnor(\\sz^m)} + 2\\scs_{m,n}\\quad \\text{and}\\\\\n        \\|{\\scp^{m,n}}\\| &\\leq 2\\Llambda\\tau_{m,n}^2\\norm{\\Fnor(\\sz^m)} + 2\\scs_{m,n}. \n    \\end{align*}\n\\end{lemma}\n\n\\begin{lemma}\n    \\label{lem:err_estimate}\n    Let \\ref{B1}--\\ref{B2} hold and let $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}$ be given. Then for any (arbitrary) subsequence $\\revise{\\{t_k\\}_{k\\in\\N}} \\subseteq \\N$, we have $\\Exp[{{\\scs_{t_k,t_{k+1}}^2}}] \\leq 4 \\sum_{i=t_k}^{t_{k+1}-1}\\alpha_i^2\\sigma_i^2$. Moreover, if there is a non-decreasing sequence $\\revise{\\{\\beta_k\\}_{k\\in\\N}}\\subset\\R_{++}$ such that ${\\sum}_{k=0}^\\infty \\alpha_k^2\\beta_k^2\\sigma_k^2 < \\infty$, it holds that\n    %\n    \\[ {\\sum}_{k=0}^\\infty \\beta_{t_k}^2 \\Exp[{{\\scs_{t_k,t_{k+1}}^2}}] < \\infty \\quad \\text{and}\\quad {\\sum}_{k=0}^\\infty \\beta_{t_k}^2 \\scs_{t_k,t_{k+1}}^2 < \\infty \\quad \\text{a$.$s$.$.}\\] \n    %\n\\end{lemma}\n\n\\begin{lemma}\n\t\\label{lem:basic bounds}\n\tLet the assumptions \\ref{A1}--\\ref{A2} and \\ref{B1}--\\ref{B2} be satisfied and let the sequences $\\revise{\\{\\sx^k\\}_{k\\in\\N}}$, $\\revise{\\{\\sz^k\\}_{k\\in\\N}}$ be generated by $\\NSGD$. Let $0 \\leq m<n$ be given with $\\tau_{m,n} < \\lambda$. Then, the following two bounds are valid:\n    \\begin{enumerate}[label=\\textup{\\textrm{(\\alph*)}},leftmargin=3em,topsep=2pt,itemsep=.5ex,partopsep=0ex]\n    \\item It holds that $\\norm{\\Fnor(\\sz^n)}^2 \\leq  ({1-\\frac{\\tau_{m,n}}{2\\lambda}})\\norm{\\Fnor(\\sz^m)}^2 - \\frac{2}{\\lambda}\\iprod{\\su^{m,n}}{\\sx^n-\\sx^m} + [({\\sL+\\frac{1}{\\lambda}})^2 + \\frac{2\\lambda \\sL^2}{\\tau_{m,n}}] \\norm{\\sx^n-\\sx^m}^2 + \\frac{3}{2\\lambda\\tau_{m,n}} \\norm{\\scp^{m,n}}^2$.\n    \\item We have $\\psi(\\sx^n) \\leq \\psi(\\sx^m) + ({\\frac{\\sL}{2}-\\frac{1}{\\lambda}})\\norm{\\sx^n-\\sx^m}^2 + \\iprod{\\su^{m,n}}{\\sx^n-\\sx^m}.$\t\n    \\end{enumerate}\n\\end{lemma}\n\nThe proof of \\cref{lem:basic bounds} is relegated to \\cref{app:lem:basic bounds}. Based on \\cref{lem:basic bounds}, we see that proper control of the inner product $\\langle \\su^{m,n}, \\sx^n-\\sx^m \\rangle$ is crucial to achieve descent (of $\\psi$). The following estimate illustrates that while this term can generate a desirable decrease, it also introduces an additional nonnegative error term: \n%\n    \\begin{align}\n    \\nonumber \\iprod{\\su^{m,n}}{\\sx^n-\\sx^m} & = (\\lambda^{-1}-\\tau_{m,n}^{-1}) \\iprod{\\sz^{n}-\\sz^m}{\\sx^n-\\sx^m} + \\tau_{m,n}^{-1}\\iprod{\\scp^{m,n}}{\\sx^n-\\sx^m}\\\\\n    \\nonumber & \\leq (\\lambda^{-1}-\\tau_{m,n}^{-1}) \\norm{\\sx^n-\\sx^m}^2 + \\tau_{m,n}^{-1} ({\\norm{\\scp^{m,n}}^2 + 0.25\\norm{\\sx^n-\\sx^m}^2})\\\\\n    \\label{u-descent} & = (\\lambda^{-1}-0.75\\tau_{m,n}^{-1}) \\norm{\\sx^n-\\sx^m}^2 + \\tau_{m,n}^{-1}\\norm{\\scp^{m,n}}^2.\n    \\end{align}\n%\nHere, we applied \\cref{eq:u-def2}, \\cref{prox-prop}, and Young's inequality. Thankfully and as we will verify in \\Cref{subsec:merit function}, the extra term $\\tau_{m,n}^{-1}\\norm{\\scp^{m,n}}^2$ in \\cref{u-descent} can be effectively compensated for by leveraging the normal map bound and choosing $\\tau_{m,n}$ sufficiently small. \n\n\n\\begin{proposition}[Approximate descent property]\n\\label{lem:merit-descent-1}\nLet \\ref{A1}--\\ref{A2} and \\ref{B1}--\\ref{B2} hold and let $\\revise{\\{\\sz^k\\}_{k\\in\\N}}$ be generated by $\\NSGD$. Suppose, \\revise{for $0\\leq m <n$, the} step sizes $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}$ satisfy $\\tau_{m,n} = {\\sum}_{i=m}^{n-1}\\alpha_i\\leq \\frac{\\sqrt{\\xi}}{8\\Llambda}$ where $\\Llambda := \\sL+\\frac{2}{\\lambda}$. We then have{}%\\vspace{-2ex}\n%\n\\begin{equation}\n    H(\\sz^n) - H(\\sz^m) \\leq  -\\frac{\\xi\\tau_{m,n}}{8} \\norm{\\Fnor(\\sz^m)}^2 + \\frac{8}{\\tau_{m,n}} \\scs_{m,n}^2.\n\\end{equation}\n\\end{proposition}\n\n\\begin{proof}\nBy \\cref{lem:basic bounds} and setting $\\tau = \\tau_{m,n}$, $\\su = \\su^{m,n}$, $\\scp = \\scp^{m,n}$, and $\\scs = \\scs_{m,n}$, it holds that\n%\n\\begin{align}\n \\nonumber   H(\\sz^n) - H(\\sz^m) & \\leq  -\\frac{\\xi\\tau}{4} \\norm{\\Fnor(\\sz^m)}^2 + (1-\\xi) \\iprod{\\su}{\\sx^n-\\sx^m} + \\frac{3\\xi}{4\\tau} \\norm{\\scp}^2\\\\\n \\label{H-descent1}   &\\hspace{4ex} + \\Big[{\\Big({\\frac{\\sL}{2}-\\frac{1}{\\lambda}}\\Big) + \\frac{\\xi\\lambda}{2} \\Big({\\sL+\\frac{1}{\\lambda}}\\Big)^2 + \\frac{\\xi \\lambda^2\\sL^2}{\\tau}}\\Big] \\norm{\\sx^n-\\sx^m}^2.\n\\end{align}\n%\nInvoking the upper bound for $\\iprod{\\su}{\\sx^n-\\sx^m}$ in \\cref{u-descent}, this yields \n%\n\\begin{align}\n\\nonumber    H(\\sz^n) - H(\\sz^m) & \\leq -\\frac{\\xi\\tau}{4} \\norm{\\Fnor(\\sz^m)}^2 + \\frac{4-\\xi}{4\\tau}\\norm{\\scp}^2\\\\\n\\label{H-descent2}    &\\hspace{-6ex} + \\underbracket{\\Big[\\frac{\\sL}{2}-\\frac{\\xi}{\\lambda} + \\frac{\\xi\\lambda}{2} \\Big({\\sL+\\frac{1}{\\lambda}}\\Big)^2 + \\frac{4\\xi \\lambda^2\\sL^2 - 3(1-\\xi)}{4\\tau}\\Big]}_{=:T_1} \\norm{\\sx^n-\\sx^m}^2.\n\\end{align}\n%\nNext, we bound $T_1$. Using $\\xi= {(2+2\\lambda^2\\sL^2)^{-1}}$ and $(a+b)^2 \\leq 2(a^2+b^2)$, $a,b \\in \\R$, we have $\\frac12\\xi\\lambda(\\sL+\\lambda^{-1})^2 \\leq \\xi \\lambda (\\sL^2+\\lambda^{-2}) = \\frac{1}{2\\lambda}$ and $4\\xi \\lambda^2\\sL^2 - 3(1-\\xi) \\leq 4\\xi(\\lambda^2\\sL^2+1)-3 = -1$. Applying the bound $\\tau \\leq \\frac{\\sqrt{\\xi}}{8\\Llambda} \\leq \\frac{1}{2(\\sL+\\lambda^{-1})}$, we then obtain $T_1 \\leq \\frac{\\sL+\\lambda^{-1}}{2} - \\frac{\\xi}{\\lambda} - \\frac{1}{4\\tau} \\leq - \\frac{\\xi}{\\lambda} < 0$. Therefore, due to $\\frac{4-\\xi}{4\\tau}\\leq \\frac{1}{\\tau}$, the estimate \\cref{H-descent2} simplifies to\n%\n\\[\nH(\\sz^n) - H(\\sz^m) \\leq  -{(\\xi\\tau/4)} \\norm{\\Fnor(\\sz^m)}^2 + {\\tau^{-1}} \\norm{\\scp}^2.\n\\]\n%\nBy \\cref{lem:est-err}, we further have $\\|\\scp\\|^2 \\leq 8\\Llambda^2 \\tau^4 \\|\\Fnor(\\sz^m)\\|^2+8\\scs^2$. This implies\n%\n\\begin{align*}\n    H(\\sz^n) - H(\\sz^m) \\leq  - \\Big[\\frac{\\xi}{4} - 8\\Llambda^2 \\tau^2 \\Big]\\tau\\|{\\Fnor(\\sz^m)}\\|^2 + \\frac{8\\scs^2}{\\tau} \n    \\leq  -\\frac{\\xi\\tau}{8} \\norm{\\Fnor(\\sz^m)}^2 + \\frac{8\\scs^2}{\\tau} ,\n\\end{align*}\n%\nwhere the last inequality is again due to $\\tau \\leq \\frac{\\sqrt{\\xi}}{8\\Llambda}$.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:exist time}\nLet $\\sT > 0$, $\\delta \\in (0,1)$ be given. The following assertions are valid.\n%\n \\begin{enumerate}[label=\\textup{\\textrm{(\\alph*)}},leftmargin=3em,topsep=0pt,itemsep=.5ex,partopsep=0ex]\n \\item Assume $\\alpha_k \\in (0,(1-\\delta) \\sT]$, $k< K$, and $\\alpha_k = \\alpha \\in(0,(1-\\delta)\\sT]$ for all $k \\geq K$ and some $K\\in\\N$. We then have $t_k \\to \\infty$ and $\\delta\\sT \\leq \\tau_{t_k,t_{k+1}} \\leq \\sT$ for all $k \\in\\N$.\n    \n \\item\n    If $\\lim_{k\\to\\infty}\\alpha_k = 0$ and $\\sum_{k=0}^\\infty \\alpha_k = \\infty$, it holds that $t_k \\to \\infty$ and $\\delta \\sT \\leq \\tau_{t_k,t_{k+1}} \\leq \\sT$ for all $k\\in\\N$.\n    \\end{enumerate}\n\\end{lemma}\n\n\\begin{proof} \nHere, we use the convention $\\sup \\emptyset = -\\infty$. According to the construction of $\\revise{\\{t_k\\}_{k\\in\\N}}$ in \\cref{def:time win} and $\\alpha_k \\leq (1-\\delta) \\sT$, we have $t_0 = 0$ and $t_{k+1}=\\varpi(t_k) \\neq -\\infty$ for all $k\\in\\N$. Furthermore, we have $t_{k+1} \\neq \\infty$ since $\\sum_{k=0}^{\\infty} \\alpha_k = \\infty$ implies $\\tau_{t_k,n}\\rightarrow \\infty$ as $n\\rightarrow \\infty$. This also yields $t_{k} \\to \\infty$, i.e., $\\revise{\\{t_k\\}_{k\\in\\N}}$ is an infinite subsequence.  By the definition of $\\varpi(k)$, it follows $\\tau_{t_k,t_{k+1}} =  \\tau_{t_k,\\varpi(t_{k})} \\leq \\sT$. In addition, by the optimality of $\\varpi(t_k)$, we obtain $\\tau_{t_k,t_{k+1}} =  \\tau_{t_k,\\varpi(t_{k})} \\geq \\sT - \\alpha_{\\varpi(t_k)} \\geq \\delta \\sT$ for all $k$.\n% \nWe continue with part (b). Due to $\\alpha_k \\to 0$ and the definition of $t_0$, we have $\\alpha_k \\leq (1-\\delta)\\sT$ for all $k\\geq  t_0$. Hence, all previous derivations are applicable in this case.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:index-time bound}\n    Let \\ref{A1}--\\ref{A2} and \\ref{B1}--\\ref{B2} hold and let $\\revise{\\{\\sx^k\\}_{k\\in\\N}}$ and $\\revise{\\{\\sz^k\\}_{k\\in\\N}}$ be generated by $\\NSGD$. Recalling $\\Llambda := \\sL+2\\lambda^{-1}$, we have for all $i>t_0$:\n%\n    \\begin{subequations}\n    \\begin{align}\n    \\label{eq:tech1}\\|\\Fnor(\\sz^i)\\|^2 &\\leq 2\\prt{2\\Llambda\\sT +1}^2\\| \\Fnor(\\sz^{t_{\\fk(i)}})\\|^2 + 8\\Llambda^2 \\sr_{\\fk(i)}^2, \\\\\n    \\label{eq:tech2}\\psi(\\sx^i) &\\leq \\psi(\\sx^{t_{\\fk(i)}})+(4\\Llambda\\sT^2+{\\lambda}/{4})\\norm{\\Fnor(\\sz^{t_{\\fk(i)}})}^2 + 4\\Llambda \\sr_{\\fk(i)}^2,\\\\\n    \\label{eq:tech3}\\psi(\\sx^i) &\\geq \\psi(\\sx^{t_{\\fk(i)+1}})-(4\\Llambda\\sT^2+{\\lambda}/{4})\\norm{\\Fnor(\\sz^{t_{\\fk(i)}})}^2 - 16\\Llambda \\sr_{\\fk(i)}^2.\n\\end{align}\n \\end{subequations}\n\\end{lemma}\n\n\\begin{theorem}[Global convergence]\n\\label{thm:global_convergence-2}\nLet \\ref{A1}--\\ref{A2}, \\ref{B1}--\\ref{B3} hold and let $\\revise{\\{\\sx^k\\}_{k\\in\\N}}$, $\\revise{\\{\\sz^k\\}_{k\\in\\N}}$ be generated by $\\NSGD$. Then, there is $\\spsi^* : \\Omega \\to \\R$ such that: \n%\n \\begin{enumerate}[label=\\textup{\\textrm{(\\alph*)}},leftmargin=3em,topsep=0.5ex,itemsep=.5ex,partopsep=0ex]\n        \\item $\\Fnor(\\sz^k)\\to 0$, $\\|\\partial \\psi(\\sx^k)\\|_{-} \\to 0$, and $\\psi(\\sx^k)\\to \\spsi^*$ a.s..\n\t\\item $\\Exp[\\norm{\\Fnor(\\sz^k)}^2]\\to 0$, $\\Exp[\\|\\partial \\psi(\\sx^k)\\|_{-}^2]\\to0$, and $\\Exp[\\psi(\\sx^k)] \\to \\Exp[\\spsi^*]$.\n\\end{enumerate}\n\\end{theorem}\n\n\\begin{proof}\nBy \\cref{lem:merit-descent-1}, \\cref{lem:exist time} (b) (with $\\delta = 4/5$; cf$.$ \\cref{eq:def delta and T}), we have \n%\n\\begin{equation}\n    \\label{eq:thm:global-1}\n    \\mer(\\sz^{t_{k+1}}) - \\mer(\\sz^{t_{k}}) \\leq -\\frac{\\xi\\sT}{{10}}\\|\\Fnor(\\sz^{t_k})\\|^2 + \\frac{10}{\\sT}\\cdot\\sr_{k}^2\\quad \\forall \\,k\\in\\N.\n\\end{equation}\n%\nThe condition $\\sum_{k=0}^{\\infty}\\alpha_k^2\\sigma_k^2<\\infty$ and \\cref{lem:err_estimate} imply that ${\\sum}_{k=0}^\\infty \\sr_{k}^2<\\infty$ a$.$s$.$ and ${\\sum}_{k=0}^\\infty\\Exp[{\\sr_{k}^2}]<\\infty$. Setting $\\su_k:=\\frac{10}{\\sT}\\, \\sum_{i=k}^\\infty \\sr_{i}^2$, it follows\n%\n\\begin{equation}\n\t\\label{eq:strong-0}\n\t\\mer(\\sz^{t_{k+1}}) + \\su_{k+1} \\leq  \\mer(\\sz^{t_{k}}) + \\su_{k}- ({\\xi \\sT}/{10}) \\|\\Fnor(\\sz^{t_k})\\|^2.\n\\end{equation}\t\n%\nSumming \\cref{eq:thm:global-1} for $k=0,1,\\ldots$ and using the bound $\\mer(z)\\geq \\bar \\psi$, this yields\n%\n\\begin{equation}\\label{eq:norm-sum}\n{\\xi\\sT}\\,{\\sum}_{k=0}^\\infty \\|\\Fnor(\\sz^{t_k})\\|^2 \\leq 10[\\mer(\\sz^{t_{0}}) + \\su_{0} - \\bar \\psi].\n\\end{equation}\n%\nHence, due to ${\\sum}_{k=0}^\\infty \\sr_{k}^2<\\infty$ a$.$s$.$, we can conclude $\\|\\Fnor(\\sz^{t_k})\\|\\to 0$, $\\sr_{k}\\to0$, and $\\su_{k}\\to0$ a$.$s$.$. Moreover, based on the definition \\cref{index-def} of $\\fk(i)$, we can infer $\\|\\Fnor(\\sz^{t_{\\fk(i)}})\\|\\to 0$ and $\\sr_{\\fk(i)}\\to0$ as $i\\to\\infty$ a$.$s$.$. This proves $\\|\\Fnor(\\sz^{i})\\|\\to 0$ a$.$s$.$ owing to \\cref{eq:tech1}. \n\nTaking expectation in \\cref{eq:norm-sum} and using $\\sum_{k=0}^\\infty\\Exp{[\\sr_{k}^2]}<\\infty$, we further obtain $\\Exp[\\sr_{k}^2]\\to0$, $\\Exp[\\su_{k}^2]\\to0$, and $\\Exp[{\\|\\Fnor(\\sz^{t_k})\\|^2}]\\to 0$. Thus, we can infer $\\Exp[\\|\\Fnor(\\sz^{i})\\|^2]\\to 0$ by taking expectation in \\cref{eq:tech1} and passing $i\\to\\infty$. By \\cref{eq:stat-am}, we have $\\|\\partial \\psi(\\proxl(z))\\|_{-} \\leq \\|\\Fnor(z)\\|$, which implies $\\|\\partial \\psi(\\sx^k)\\|_{-} \\to 0$ a$.$s$.$ and $\\Exp[{\\|\\partial \\psi(\\sx^k)\\|^2_{-}}] \\to 0$.\n\nNext, we show $\\psi(\\sx^{t_k}) \\to \\spsi^*$ a$.$s$.$. By \\cref{eq:strong-0}, $\\{\\mer(\\sz^{t_k})+\\su_k\\}_k$ is monotonically non-increasing, and thus, we conclude that $\\mer(\\sz^{t_k})+\\su_k \\to \\spsi^*$ a$.$s$.$ for some random variable $\\spsi^*$. Using $\\|\\Fnor(\\sz^{t_k})\\|\\to0$, $\\su_k\\to 0$, and $\\psi(\\sx^{t_k}) = \\mer(\\sz^{t_k}) + \\su_k - (\\frac{\\xi\\lambda}{2}\\|\\Fnor(\\sz^{t_k})\\|^2+\\su_k)$, we obtain $\\psi(\\sx^{t_k}) \\to \\spsi^*$ a$.$s$.$. Taking $i\\to \\infty$ in \\cref{eq:tech2} and \\cref{eq:tech3}, it follows %\n\\begin{align*}\n     \\psi(\\sx^i) &\\leq \\psi(\\sx^{t_{\\fk(i)}})+(4\\Llambda\\sT^2+\\lambda/4)\\norm{\\Fnor(\\sz^{t_{\\fk(i)}})}^2 + 4\\Llambda \\sr_{\\fk(i)}^2 \\to \\spsi^*,\\\\\n    \\psi(\\sx^i) &\\geq \\psi(\\sx^{t_{\\fk(i)+1}})-(4\\Llambda\\sT^2+\\lambda/4)\\norm{\\Fnor(\\sz^{t_{\\fk(i)}})}^2 - 16\\Llambda \\sr_{\\fk(i)}^2 \\to \\spsi^*, \\quad \\text{a.s.},\n \\end{align*}\n%\nand hence $\\psi(\\sx^i) \\to \\spsi^*$. Finally, we verify $\\Exp[\\psi(\\sx^{i})] \\to \\Exp[\\spsi^*]$. Since $\\{\\mer(\\sz^{t_k}) + \\su_k\\}_k$ is bounded below by $\\bar\\psi$, non-increasing, and converges to $\\spsi^*$, we can obtain $\\Exp[\\mer(\\sz^{t_k}) + \\su_k]\\to \\Exp[\\spsi^*]$ by the monotone convergence theorem. Using similar arguments as before, this yields $\\Exp[\\psi(\\sx^{t_k})] \\to \\Exp[\\spsi^*]$. Let us now define $\\ssigma_i := \\psi(\\sx^{t_{\\fk(i)}})+(4\\Llambda\\sT^2+\\lambda/4)\\norm{\\Fnor(\\sz^{t_{\\fk(i)}})}^2 + 4\\Llambda \\sr_{\\fk(i)}^2$.\n%\nWe then have $\\psi(\\sx^i) \\leq \\ssigma_i$ (see \\cref{eq:tech2}), $\\ssigma_i\\to \\spsi^*$ a$.$s$.$, and $\\Exp[\\ssigma_i]\\to \\Exp[\\spsi^*]$. Applying Fatou's Lemma for $\\psi(\\sx^i)-\\bar{\\psi}$ and $\\ssigma_i-\\psi(\\sx^i)$, it follows \n%\n \\begin{align*}\n      \\Exp[\\spsi^*]-\\bar{\\psi} &= \\Exp[{{\\lim}_{i\\to\\infty}({\\psi(\\sx^i)-\\bar{\\psi}}})] \\leq {\\liminf}_{i\\to\\infty}{\\Exp{[\\psi(\\sx^i)]}}-\\bar{\\psi}\\quad \n \\end{align*}\n%\nand $0 = \\Exp[{{\\lim}_{i\\to\\infty} (\\ssigma_i-\\psi(\\sx^i))}] \\leq \\Exp[{\\spsi^*}] - {\\limsup}_{i\\to\\infty} {\\Exp{[\\psi(\\sx^i)]}}$. Consequently, we can infer $\\Exp{[\\psi(\\sx^i)]}\\to \\Exp[{\\spsi^*}]$. This finishes the proof. \n\\end{proof}\n\n\\begin{lemma}[Kurdyka-{\\L}ojasiewicz-type inequality]\n\\label{lem:KL-mer}\n    Let \\ref{A3} hold and let $\\bar z$ be given with $\\bar{x}=\\proxl(\\bar{z}) \\in \\crit(\\psi)$. For all $\\theta\\in[\\frac12,1)$, there then are $\\varsigma, \\rho > 0, \\eta\\in(0,1]$ and a continuous, concave function $\\varrho:[0,\\eta) \\to \\R_+$, which is $C^1$ on $(0,\\eta)$, satisfying \n      \\begin{enumerate}[label=\\textup{\\textrm{(\\alph*)}},leftmargin=3em,topsep=1pt,itemsep=.5ex,partopsep=0ex]\n         \\item $1/\\varrho^\\prime(s+t) \\leq 1/\\varrho^\\prime(s) + \\varsigma t^{\\theta}$ for all $s,t >0$ with $s+t<\\eta$,\n         \\item $\\varrho(0) = 0$ and $\\varrho^\\prime(t) \\geq t^{-\\theta}$ for all $t\\in(0,\\eta)$,\n     \\end{enumerate}\n     such that, for all $z\\in\\cB_\\rho(\\bar z) \\cap\\{z:0<|\\mer(z)-\\mer(\\bar z)| + \\frac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2<\\eta\\}$, we have\n     \\begin{equation}\n         \\label{eq:mer-kl}\n         \\varrho^\\prime(|\\mer(z)-\\mer(\\bar z)|) \\cdot  \\|\\Fnor(z)\\|\\geq 1.\n     \\end{equation} \n\\end{lemma}\n\n\\begin{proof}\nThe proof is divided into two steps. First, we show that $\\psi$ satisfies the KL property, with the desingularization function $\\varrho$ being concave, differentiable, and fulfilling properties (a) and (b). This step closely follows the proof of \\cite[Lemma 4.11]{josz2024proximal}. Although the differences are minor, we include a complete proof for self-containedness. In the second step, we show that these properties can be transferred to $\\mer$. %\\\\[1mm]\n\n\\noindent\\textbf{Step 1.}  \n%\nApplying \\cite[Corollary 15]{boldanlewshi07} and \\cite[Theorem 4.1]{AttBolRedSou10}, there are $\\rho,\\eta_1 >0$ and a definable, concave, continuous function $\\varrho_1:[0,\\eta_1)\\to\\R_+$, $\\varrho_1 \\in C^1((0,\\eta_1))$, such that \n%\n\\begin{equation}\n\\varrho_1^\\prime(|\\psi(x)-\\psi(\\bar x)|) \\cdot \\|\\partial \\psi(x)\\|_{-}\\geq 1, \\quad \\forall\\; x\\in\\cB_{\\rho}(\\bar x) \\cap\\{x:0<|\\psi(x)-\\psi(\\bar x)|<\\eta_1\\}. \\label{eq:kl-1}\n\\end{equation}\n%\nWe now set $\\varrho(t)=\\int_{0}^t \\varrho^\\prime(\\tau)\\,\\rmd \\tau$, $\\varrho^\\prime(t):=\\max\\{\\varrho_1^\\prime(t),t^{-\\theta}\\}$, and $\\bar \\eta:=\\min\\{\\eta_1,1\\}$. Then, $\\varrho(0)=0$ and \\cref{eq:kl-1} remains to hold with $\\varrho_1$ and $\\eta_1$ being replaced by $\\varrho$ and $\\bar \\eta$, respectively.\nThe function $\\varrho$ is continuous, $C^1$ on $(0,\\bar\\eta)$, and concave as $\\varrho^\\prime$ is non-increasing. \n%\nMoreover, $\\varrho^\\prime$ is definable because $t \\mapsto t^{-\\theta}$ and $\\varrho_1^\\prime$ are definable \\cite[Lemma 6.1]{coste2000introduction}, so is $t\\mapsto 1/\\varrho^\\prime(t)$ \\cite[Lemma 2.3 p.14]{van1998tame} (see also \\cite[Section 4.3]{AttBolRedSou10}). According to \\cite[Theorem 6.4]{coste2000introduction}, the mapping $t\\mapsto 1/\\varrho^\\prime(t)$ is differentiable for all but finitely many points in $(0,\\bar \\eta)$. This result, along with the monotonicity theorem \\cite[(1.2) p.43]{van1998tame}, indicates that $t\\mapsto 1/\\varrho^\\prime(t)$ is differentiable and monotone on the interval $(0,\\eta)$ for some $\\eta\\in(0,\\bar \\eta)$.  \n\nIf $(1/\\varrho^\\prime)^\\prime(t)\\to \\infty$ as $t\\searrow0$, then $(1/\\varrho^\\prime)^\\prime$ is decreasing on $(0,\\eta)$ and $1/\\varrho^\\prime$ is concave. By \\cite[Lemma 3.5]{josz2023convergence}, this yields $1/\\varrho^\\prime(s+t) \\leq 1/\\varrho^\\prime(s) + 1/\\varrho^\\prime(t) \\leq 1/\\varrho^\\prime(s) + t^\\theta$ for all $s,t>0$ and $s+t<\\eta$. Otherwise, we have $\\bar\\varsigma:=\\max_{\\tau\\in(0,\\eta)}(1/\\varrho^\\prime)^\\prime(\\tau) < \\infty$ and by the mean value theorem, it follows $1/\\varrho^\\prime(s+t) - 1/\\varrho^\\prime(s)= (1/\\varrho^\\prime)^\\prime(\\tau) t \\leq \\bar\\varsigma t \\leq \\bar\\varsigma t^{\\theta}$, $\\tau \\in (0,\\eta)$, where we used $\\theta < 1$ and $t \\in (0,1)$. Note that $\\bar\\varsigma$ is nonnegative since $(1/\\varrho^\\prime)^\\prime(\\tau) = -\\varrho^{\\prime\\prime}(\\tau)/(\\varrho^\\prime(\\tau))^2 \\geq 0$ (by the concavity of $\\varrho$). We may now set $\\varsigma := \\max\\{\\bar\\varsigma,1\\}$. %\\\\[1mm]\n\n\\noindent\\textbf{Step 2:} \\emph{Property \\cref{eq:mer-kl} for $\\mer$.} Let $z\\in\\cB_\\rho(\\bar z) \\cap\\{z:0<|\\mer(z)-\\mer(\\bar z)|+ \\frac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2<\\eta\\}$ be arbitrary. Setting $x=\\proxl(z)$ and using the nonexpansiveness of $\\proxl$, we have $\\|x-\\bar x\\| \\leq \\|z-\\bar z\\| \\leq \\rho$ and $|\\psi(x)-\\psi(\\bar x)| \\leq |\\mer(z)-\\mer(\\bar z)|+ \\frac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2<\\eta$.\n\nThus, if $x$ satisfies $\\psi(x) \\neq \\psi(\\bar x)$, then \\textbf{Step 1} implies $\\varrho^\\prime(|\\psi(x)-\\psi(\\bar x)|) \\|\\partial \\psi(x)\\|_{-}\\geq 1$. Furthermore, since $1/\\varrho^\\prime$ is non-decreasing and applying the bound (a), we can infer\n%\n\\begin{align*}\n          1/\\varrho^\\prime(|\\mer(z)-\\mer(\\bar z)|) \n          &\\leq  1/\\varrho^\\prime(|\\psi(x)-\\psi(\\bar x)| + \\tfrac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2)\\\\\n          &\\leq 1/\\varrho^\\prime(|\\psi(x)-\\psi(\\bar x)|) + \\varsigma (\\tfrac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2)^\\theta \\\\ &\\leq \\|\\partial \\psi(x)\\|_{-} + \\varsigma \\sqrt{\\xi\\lambda/2}\\|\\Fnor(z)\\| \\leq (1+\\varsigma \\sqrt{\\xi\\lambda/2})\\|\\Fnor(z)\\|.\n\\end{align*}\n%\nHere, the third inequality is due to $\\tfrac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2<1$ and $\\theta\\in[\\frac12,1)$, and the last step follows from $\\Fnor(z)\\in\\partial \\psi(x)$. If $\\psi(x) = \\psi(\\bar x)$, then we obtain $1/\\varrho^\\prime(|H(z)-H(\\bar z)|) \\leq (\\tfrac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2)^\\theta \\leq \\sqrt{\\xi\\lambda/2}\\|\\Fnor(z)\\|$ by (b). We can conclude the proof after rescaling $\\varrho$ to $(1+\\max\\{1,\\varsigma\\} \\sqrt{\\xi\\lambda/2}) \\varrho$ (this does not affect the properties (a)--(b)).\n\\end{proof}\n\n\\subsubsection{\\texorpdfstring{Proof of \\cref{thm:iter converge}}{Proof of Theorem 4.2}}\n\nWe start the proof by deriving certain summability results for the error terms $\\su_k$ and $\\sr_k$. %\\\\[1mm]\n\n\\noindent\\textbf{Step 1:} \\emph{Summability of error terms.} Due to condition \\cref{eq:kl-step} (i.e., $\\sum_{k=0}^\\infty \\alpha_k^2 \\tau_{0,k}^{\\mu}\\sigma_k^2  <\\infty$) and \\cref{lem:err_estimate}, we have $\\bar \\sr:=\\sum_{k=0}^\\infty \\tau_{0,t_k}^{\\mu}\\sr_k^2<\\infty$ a$.$s$.$. This yields \\vspace{-0.5ex}\n\\[\n\\sT\\su_k = {10}\\, {\\sum}_{i=k}^\\infty \\sr_{i}^2 =  {10}\\, {\\sum}_{i=k}^\\infty \\tau_{0,t_i}^{-\\mu} \\tau_{0,t_i}^{\\mu} \\sr_{i}^2 \\leq {10} \\tau_{0,t_k}^{-\\mu} \\cdot {\\sum}_{i=k}^\\infty  \\tau_{0,t_i}^{\\mu} \\sr_{i}^2 \\leq {10\\bar \\sr}  \\cdot \\tau_{0,t_k}^{-\\mu}.\n\\]\n%\nUsing $\\delta \\sT \\leq \\tau_{t_k,t_{k+1}}$, it follows $\\tau_{0,t_k} \\geq  \\tau_{0,t_0} + \\delta \\sT k$ and $\\su_k\\to 0$ a$.$s$.$, which implies \n%\n\\begin{equation}\n    \\label{eq:summability u_k}\n    {\\sum}_{k=0}^\\infty \\su_k^\\theta \\leq (10\\bar\\sr/\\sT)^\\theta  \\cdot {\\sum}_{k=0}^\\infty (\\tau_{0,t_0} + \\delta \\sT k)^{-\\mu\\theta}<\\infty \\quad \\text{for all}\\quad \\theta > \\tfrac{1}{\\mu}.\n\\end{equation}\n%\nHence, we have $\\sum_{k=j}^\\infty \\su_k^\\theta \\to 0$ as $j\\to\\infty$ for all $\\theta > {\\mu}^{-1}$. Moreover, it holds that \\vspace{-1ex}\n%\n\\begin{equation}\\label{eq:rk-sum}\n{\\sum}_{k=0}^\\infty \\sr_k = {\\sum}_{k=0}^\\infty \\tau_{0,t_k}^{-\\frac{\\mu}{2}}\\cdot \\tau_{0,t_k}^{\\frac{\\mu}{2}} \\sr_k \\leq \\Big({ \\bar{\\sr} \\cdot {\\sum}_{k=0}^\\infty (\\tau_{0,t_0} + \\delta \\sT  k)^{-\\mu} }\\Big)^{1/2} < \\infty \\quad \\text{a$.$s$.$.}\n\\end{equation}\n%\nLet us fix $\\theta\\in(\\max\\{\\frac{1}{\\mu},\\frac12\\},1)$ and let us introduce the events $\\mathcal E := \\{\\omega : \\sum_{k=0}^\\infty \\su_k(\\omega)^\\theta < \\infty, \\, \\sum_{k=0}^\\infty \\sr_k(\\omega) < \\infty\\}$ and $\\mathcal C := \\mathcal X \\cap \\{\\omega : \\Fnor(\\sz^k(\\omega)) \\to 0, \\, \\psi(\\sx^k(\\omega))\\to\\spsi^*(\\omega)\\}$. Condition \\cref{eq:kl-step} implies that \\ref{B3} is satisfied. Thus, by \\cref{thm:global_convergence-2}, \\cref{eq:summability u_k}, \\cref{eq:rk-sum}, and \\ref{B4}, we can infer $\\Prob(\\mathcal C) = \\Prob(\\mathcal E) = \\Prob(\\mathcal D)=1$, where $\\mathcal D = \\mathcal C \\cap \\mathcal E$. %\\\\[1mm]\n\n\\noindent\\textbf{Step 2:} \\emph{Subsequence convergence and roadmap.} We first present an auxiliary result.\n \n\\begin{lemma}[Subsequence convergence]\\label{lem:subsequence converge}\nLet \\ref{A1}--\\ref{A2} and \\ref{B1}--\\ref{B4} hold and let $\\omega\\in\\mathcal C$ be arbitrary. Then, there exist $z^*\\in\\Rn$ and a subsequence $\\revise{\\{q_k\\}_{k\\in\\N}}\\subseteq \\N$ such that $x^*:=\\proxl(z^*)\\in\\crit(\\psi)$, $x^{q_k} \\to x^*$, $z^{q_k} \\to z^*$, $\\psi(x^k) \\to \\psi(x^*) = \\psi^*$, and $\\mer(z^{k}) \\to \\mer(z^*) = \\psi^*$ as $k\\to\\infty$, where  \n$\\revise{\\{x^k\\}_{k\\in\\N}} = \\revise{\\{\\sx^k(\\omega)\\}_{k\\in\\N}}$ and $\\revise{\\{z^k\\}_{k\\in\\N}}= \\revise{\\{\\sz^k(\\omega)\\}_{k\\in\\N}}$. \n\\end{lemma}\n\nThe proof of \\cref{lem:subsequence converge} can be found in \\cref{proof:lem:subsequence converge}. To simplify the notation, we set\n%\n$\\sd_k := {\\max}_{t_k<i\\leq t_{k+1}} \\max\\{\\|\\sx^i-\\sx^{t_k}\\|,\\|\\sz^i-\\sz^{t_k}\\|\\}$.\n\n\\cref{lem:est-err} and $\\tau_{t_k,t_{k+1}} \\leq \\sT$ then imply \n%\n\\begin{equation}\\label{dk-bound}\n    % \\sd_k^2 \\leq 4 \\sT^2 \\|\\Fnor(\\sz^{t_k})\\|^2 + 4 \\sr_k^2.\n    \\sd_k \\leq 2 \\sT \\|\\Fnor(\\sz^{t_k})\\| + 2 \\sr_k.\n\\end{equation}\n%\nOur goal is to show $\\sx^k \\to \\sx^*$ a$.$s$.$. Thanks to $\\Prob(\\mathcal D)=1$, it suffices to prove that $\\revise{\\{\\sx^k(\\omega)\\}_{k\\in\\N}}$ is a Cauchy sequence for all $\\omega \\in \\mathcal D$. Let $\\omega \\in \\mathcal D$ be arbitrary and let $\\revise{\\{x^k\\}_{k\\in\\N}} = \\revise{\\{\\sx^k(\\omega)\\}_{k\\in\\N}}$,  $\\revise{\\{z^k\\}_{k\\in\\N}} = \\revise{\\{\\sz^k(\\omega)\\}_{k\\in\\N}}$, $\\revise{\\{r_k\\}_{k\\in\\N}} = \\revise{\\{\\sr_k(\\omega)\\}_{k\\in\\N}}$, etc. denote the corresponding realizations. For given $\\varepsilon>0$, we need to show that there is $k$ such that $\\|x^m - x^n\\|<\\varepsilon$ for all $k\\leq m<n$. \n\nWe claim that it suffices to establish ${\\sum}_{k=0}^{\\infty}\\,d_k< \\infty$. In this case, there is $k^\\prime\\in\\N$ such that ${\\sum}_{j=k}^{\\infty}\\,d_j<\\varepsilon/2$ for all $k \\geq k^\\prime$. For any $n > m > t_{k^\\prime}$, we have $k^\\prime \\leq \\fk(m)\\leq \\fk(n)$. Therefore, using the triangle inequality and the above bound, we obtain %\\vspace{-1.5ex}\n%\n\\begin{equation*}\n\\|x^m - x^n\\| \\leq d_{\\fk(m)} + d_{\\fk(n)} + {\\sum}_{j=\\fk(m)}^{{\\fk(n)-1}}\\|x^{t_{j+1}}-x^{t_j}\\| \\leq 2 {\\sum}_{j=\\fk(m)}^{{\\fk(n)}} d_j< \\varepsilon, \n\\end{equation*}\n%\nwhich proves our claim. In the remaining steps, we establish summability of $\\revise{\\{d_k\\}_{k\\in\\N}}$.%\\\\[1mm]\n\n\\noindent\\textbf{Step 3:} \\emph{Entering the local region and a KL-based bound for $\\revise{\\{d_k\\}_{k\\in\\N}}$.} By \\cref{lem:subsequence converge}, there exists a subsequence $\\revise{\\{q_k\\}_{k\\in\\N}}$ such that $x^{q_k} \\to x^*$, $z^{q_k} \\to z^*$ for some $z^*$ with $x^* = \\proxl(z^*) \\in\\crit(\\psi)$, and $\\mer(z^{k}) \\to \\mer(z^*) = \\psi^*$. Based on the definition of $\\fk$ and $d_k$, it holds that $\\max\\{\\|x^{q_k} - x^{t_{\\fk(q_k)}}\\|,\\|z^{q_k} - z^{t_{\\fk(q_k)}}\\|\\} \\leq d_{\\fk(q_k)}$. Due to $\\|\\Fnor(z^{t_{{\\fk(q_k)}}})\\|\\to0$, $r_{{\\fk(q_k)}}\\to0$, and \\cref{dk-bound}, we then have $d_{{\\fk(q_k)}} \\to 0$, $x^{t_{\\fk(q_k)}} \\to x^*$, and $z^{t_{\\fk(q_k)}} \\to z^*$.\n\n\nApplying \\cref{lem:KL-mer}, there exist $\\rho>0$, $\\eta\\in(0,1]$, and a concave function $\\varrho:[0,\\eta)\\to\\R_+$, $\\varrho \\in C^1((0,\\eta))$ (satisfying properties (a)--(b) in \\cref{lem:KL-mer}) such that\n%\n\\begin{equation}\n         \\label{eq:super-proof-kl}\n         \\varrho^\\prime(|\\mer(z)-\\psi^*|) \\cdot  \\|\\Fnor(z)\\|\\geq 1\\quad %\\text{holds}\n\\end{equation}\n%\nfor all $z\\in\\cB_\\rho(z^*) \\cap\\{z \\in \\Rn:0<|\\mer(z)-\\psi^*| + \\frac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2<\\eta\\}$.\n    \nNext, we define $\\Delta_k:= \\varrho(\\mer(z^{t_k}) -\\psi^* + u_k)$.\nSince \\revise{$\\{\\mer(z^{t_k}) -\\psi^* + u_k\\}_{k \\in \\N}$} is monotonically decreasing (see \\cref{eq:strong-0}) with $\\mer(z^{t_k}) + u_k \\to \\psi^*$, $\\Delta_k$ is well-defined. By $\\omega \\in \\mathcal D$, $\\varrho(0) = 0$, and by the previous discussion, there then is $\\hat k \\in \\N$ such that\n%\n\\begin{subequations}\n\\label{eq:choose rho eta}\n    \\begin{align}\n        \\label{eq:choose rho eta-1} &\\|z^{t_{\\hat k}} - z^*\\| + 20\\xi^{-1}\\Delta_{\\hat{k}}  + 2\\varsigma\\sT {\\sum}_{i=\\hat k}^\\infty\\, u_i^\\theta + 2{\\sum}_{i=\\hat k}^\\infty\\, r_i< \\rho \\quad \\text{and} \\quad \\\\ \\label{eq:choose rho eta-2} &|\\mer(z^{t_k})-\\psi^*| + \\max\\{\\tfrac{\\xi\\lambda}{2}\\|\\Fnor(z^{t_k})\\|^2, u_k\\} \\leq  \\eta\\quad\\text{for all} \\quad k \\geq \\hat k. \n    \\end{align}\n\\end{subequations}\n\nLet $k \\geq \\hat k$ be now given with $z^{t_k} \\in \\mathcal B_\\rho(z^*)$. (In particular, this is true for $k = \\hat k$). Hence, by \\cref{eq:choose rho eta-2}, the KL inequality \\cref{eq:super-proof-kl} is applicable and we have\n%\n\\begin{subequations}\n    \\begin{align}\n\t\\label{eq:long-equation-1}\\Delta_k - \\Delta_{k+1}\n    & \\geq \\varrho^\\prime(\\mer(z^{t_k}) - \\psi^*+u_k)(\\mer(z^{t_k}) +u_k - [\\mer(z^{t_{k+1}})+u_{k+1}]) \\\\\n\t\\label{eq:long-equation-2}& \\geq \\varrho^\\prime(|\\mer(z^{t_k}) - \\psi^*|+u_k) (\\xi \\sT \\|\\Fnor(z^{t_k})\\|^2/10) \\\\%[1mm]\n\t\\label{eq:long-equation-3}&\\geq (1/\\varrho^\\prime({|\\mer(z^{t_k}) - \\psi^*|})  + \\varsigma u_k^\\theta)^{-1} (\\xi \\sT \\|\\Fnor(z^{t_k})\\|^2/10) \\\\\n    \\label{eq:long-equation-4}&\\geq \\frac{\\xi\\sT}{10}\\cdot \\frac{\\|\\Fnor(z^{t_k})\\|^2}{\\|\\Fnor(z^{t_k})\\| +  \\varsigma u_k^\\theta}\\geq \\frac{\\xi \\sT}{10}\\srt{\\|\\Fnor(z^{t_k})\\| -  \\varsigma \n u_k^\\theta}.\n\\end{align}\n\\end{subequations}\n%\nThe inequality \\cref{eq:long-equation-1} is due to the concavity of $\\varrho$, \\cref{eq:long-equation-2} follows from the descent-type property \\cref{eq:strong-0} and the fact that $\\varrho^\\prime$ is non-increasing, \\cref{eq:long-equation-3} utilizes $\\varrho^\\prime(s+t) \\geq [1/\\varrho^\\prime(s) + \\varsigma t^{\\theta}]^{-1}$ (see \\cref{lem:KL-mer} (a)), and \\cref{eq:long-equation-4} is based on \\cref{eq:super-proof-kl} and ${a^2}\\geq a^2-b^2$, $a,b\\in \\R$. Rearranging yields $\\sT\\|\\Fnor(z^{t_k})\\| \\leq 10\\xi^{-1}(\\Delta_k - \\Delta_{k+1}) + \\varsigma \\sT u_k^\\theta$. Combining this estimate with $d_k \\leq 2 \\sT \\|\\Fnor(z^{t_k})\\| + 2 r_k$ (see \\cref{dk-bound}), we can infer\n%\n\\begin{equation}\n    \\label{eq:d bound}\nd_k \\leq 20\\xi^{-1}(\\Delta_k - \\Delta_{k+1}) + 2\\varsigma\\sT u_k^\\theta + 2r_k.\n\\end{equation}\n\n\\noindent\\textbf{Step 4:} \\emph{Attracted by the local region.}  We now show that the following statements hold for all $k \\geq \\hat k$: \n%\n\t\\begin{enumerate}[label=(\\alph*),topsep=4pt,itemsep=0.5ex,partopsep=0ex]\n\t\\item $z^{t_k} \\in \\mathcal{B}_{\\rho}(z^*)$ and $|\\mer(z^{t_k}) - \\psi^*| + \\max\\{\\frac{\\xi\\lambda}{2}\\|\\Fnor(z^{t_k})\\|^2,u_k\\}< \\eta$.\n %\\vspace{1mm}\n\t\\item $\\sum_{i=\\hat k}^{k}d_i  \\leq 20\\xi^{-1}[\\Delta_{\\hat k} - \\Delta_{k+1}] + 2\\varsigma\\sT \\sum_{i=\\hat k}^k u_i^\\theta + 2\\sum_{i=\\hat k}^k r_i.$\n\t\\end{enumerate}\n% \nWe will prove these statements by induction. First, (a) and (b) are satisfied for $k=\\hat k$ thanks to \\cref{eq:choose rho eta} and \\cref{eq:d bound}. To conduct the induction step, let us assume that (a)--(b) hold for some $k =n \\geq \\hat k$. We first verify (a) for $k=n+1$. Due to \\cref{eq:choose rho eta-2}, the bound $|\\mer(z^{t_{n+1}}) - \\psi^*| + \\max\\{\\frac{\\xi\\lambda}{2}\\|\\Fnor(z^{t_{n+1}})\\|^2,u_{n+1}\\} < \\eta$ is immediate. Moreover, we have\n%\n\\begin{align*}\n    \\|z^{t_{n+1}} - z^*\\| &\\leq \\|z^{t_{\\hat k}} - z^*\\| + {\\sum}_{i=\\hat{k}}^n \\|z^{t_{i+1}} - z^{t_{i}}\\| \\leq  \\|z^{t_{\\hat k}} - z^*\\| + {\\sum}_{i=\\hat k}^n \\,d_i\\\\\n    &\\leq \\|z^{t_{\\hat k}} - z^*\\| +  20\\xi^{-1}[\\Delta_{\\hat k} - \\Delta_{n+1}] + 2\\varsigma\\sT {\\sum}_{i=\\hat k}^\\infty\\, u_i^\\theta + 2{\\sum}_{i=\\hat k}^\\infty\\, r_i < \\rho,\n\\end{align*}\n%\nwhere the second inequality follows from the definition of $d_k$, the third step uses (b) for $k=n$, and the last step holds due to \\cref{eq:choose rho eta-1} and $\\Delta_{n+1}\\geq 0$. Next, we prove that (b) holds for $k=n+1$. Invoking \\cref{eq:d bound} with $k=n+1$ and (b) with $k=n$, we obtain\n%\n\\[\n{\\sum}_{i=\\hat k}^{n+1}\\,d_i = d_{n+1} + {\\sum}_{i=\\hat k}^{n}\\,d_i \\leq 20\\xi^{-1}[\\Delta_{\\hat k} - \\Delta_{n+2}] + 2\\varsigma\\sT {\\sum}_{i=\\hat k}^{n+1}\\, u_i^\\theta + 2{\\sum}_{i=\\hat k}^{n+1}\\, r_i.\n\\]\n%\nThis completes the induction. Thus, (a), (b), and \\cref{eq:d bound} hold for all $k \\geq \\hat k$. %\\\\[1mm]\n\n\\noindent\\textbf{Step 5:} \\emph{Summability of $\\revise{\\{d_k\\}_{k\\in\\N}}$.} Taking $k\\to\\infty$ in \\textbf{Step 4} (b) and due to $\\Delta_k \\geq 0$, we have \\vspace{-1.5ex}\n%\n\\begin{equation*} \n{\\sum}_{i=\\hat k}^{\\infty}d_i  \\leq 20\\xi^{-1}\\Delta_{\\hat k} + 2\\varsigma\\sT {\\sum}_{i=\\hat k}^{\\infty} u_i^\\theta + 2{\\sum}_{i=\\hat k}^{\\infty} r_i < \\rho < \\infty.\n\\end{equation*}\n%\nThis yields $\\sum_{k=0}^{\\infty}d_k < \\infty$ and, following \\textbf{Step 2}, this completes the proof. \\hfill \\proofbox\n\n\n\n\\begin{proof} Let $\\omega \\in \\Omega$ be arbitrary and let us set $\\tau = \\tau_{m,n}$, $p = p^{m,n} = \\scp^{m,n}(\\omega)$, $s = s_{m,n} = \\scs_{m,n}(\\omega)$, $x^k = \\sx^k(\\omega)$, $z^k = \\sz^k(\\omega)$, etc. It follows from the definition of $p^{m,i}$ \\revise{and $s_{m,n}$} and the triangle inequality that\n%\n\\begin{align}\n        \\nonumber \\|p^{m,i}\\| &\\leq {\\sum}_{j=m}^{i-1}\\alpha_j\\|{\\Fnor(z^j) - \\Fnor(z^m)}\\| + \\Big\\|{{\\sum}_{j=m}^{i-1} \\alpha_j e^j}\\Big\\| \\\\ &\\leq {\\sum}_{j=m}^{i-1}\\alpha_j\\|{\\Fnor(z^j) - \\Fnor(z^m)}\\| + s,\\qquad \\text{for all $i\\in(m,n] \\cap \\N$.} \\label{eq:lem-4-est-1}\n\\end{align}\n% \nRecalling $\\Fnor(z) = \\nabla f(x) + \\lambda^{-1}(z-x)$ and using \\ref{A1} and the nonexpansiveness of the proximity operator (i.e., $\\|x^i-x^m\\| \\leq \\|z^i-z^m\\|$), we have\n%\n\\begin{equation}\n\\label{eq:nor is Lip}\n     \\|\\Fnor(z^j) - \\Fnor(z^m)\\| \\leq (\\sL+\\lambda^{-1})\\|x^j-x^m\\| + \\lambda^{-1}\\|z^j-z^m\\| \\leq  \\Llambda \\|z^j-z^m\\|,% =: C_\\lambda\\|z^i-z^m\\|.\n\\end{equation} \n%\nwhere $\\Llambda =\\sL+2\\lambda^{-1}$. Thus, combining the relation $z^i = z^m - \\tau_{m,i}\\Fnor(z^m) + p^{m,i}$ (cf$.$ \\cref{eq:pretty-important})\\revise{, $\\tau_{m,i} \\leq \\tau$,} and the inequality \\cref{eq:lem-4-est-1}, we can infer\n% \n\\begin{align}\n    \\nonumber \\|z^i-z^m\\| &\\leq \\tau_{m,i} \\|\\Fnor(z^m)\\| + \\|p^{m,i}\\| \\\\\n    \\nonumber &\\leq {\\sum}_{j=m}^{i-1} \\alpha_j \\|\\Fnor(z^j)-\\Fnor(z^m)\\| + \\tau \\|\\Fnor(z^m)\\| + s\\\\ \n    \\label{eq:lem-4-est-3} &\\leq \\Llambda {\\sum}_{j=m}^{i-1} \\alpha_j  \\|z^j-z^m\\| + \\tau \\|\\Fnor(z^m)\\| + s, \\qquad \\text{for all $i\\in(m,n] \\cap \\N$}.\n\\end{align}\n%\nWe now apply Gronwall's inequality (\\cref{Thm:G}) upon setting $b_1 :=\\tau \\|\\Fnor(z^m)\\| + s$, $b_2 := \\Llambda$, $a_k:=\\alpha_{m+k}$, $y_k:=\\|z^{m+k} - z^m\\|$, and $t:=i-m-1$. For all $i\\in(m,n] \\cap \\N$, this establishes the following upper bound  \n%\n\\begin{equation}\n            \\|z^i - z^m\\|  \\leq \\exp(\\tau \\Llambda) \\cdot [\\tau\\|\\Fnor(z^m)\\|+s]\\leq 2\\tau\\|\\Fnor(z^m)\\| + 2s,\n            \\label{eq:lem-4-est-5}\n\\end{equation}\n%\nwhere the last inequality is due to $\\tau\\Llambda \\leq \\frac12$ and $\\exp(\\frac12)\\leq 2$. Moreover, it follows from $\\|x^i-x^m\\| \\leq \\|z^i-z^m\\|$ that ${\\max}_{m<i\\leq n}\\|x^i-x^m\\| \\leq {\\max}_{m<i\\leq n}\\|z^i-z^m\\| \\leq 2\\tau\\|\\Fnor(z^m)\\| + 2s$. This verifies the first bound in \\cref{lem:est-err}. \n     \nNext, combining \\cref{eq:nor is Lip} and \\cref{eq:lem-4-est-5}, we obtain\n%\n\\[\n{\\max}_{m<i\\leq n} \\|\\Fnor(z^i) - \\Fnor(z^m)\\|\\leq \\Llambda{\\max}_{m<i\\leq n}\\|z^i-z^m\\| \\leq 2\\Llambda[\\tau\\|\\Fnor(z^m)\\| + s].\n\\]\n%\nUsing this bound in \\cref{eq:lem-4-est-1} with $i=n$ and recalling $\\tau={\\sum}_{i=m}^{n-1} \\alpha_i$, this yields $\\|p\\|\\leq s+ 2\\Llambda (\\tau\\|\\Fnor(z^m)\\| + s) {\\sum}_{j=m}^{n-1} \\alpha_j \\leq 2\\Llambda\\tau^2\\|\\Fnor(z^m)\\| + 2s$.\n\\end{proof}\n\n\\begin{proof}  Let us define the filtration $\\mathcal{U}_\\ell:=\\mathcal{F}_{t_k+\\ell}$ and introduce the sequence \\revise{$\\{\\sy^\\ell\\}_{\\ell\\in\\N}$} as follows $\\sy^0:=0$, $\\sy^\\ell := {\\sum}_{i=t_k}^{\\min\\{t_k+\\ell,t_{k+1}\\}-1}\\alpha_i \\se^i$ for all $\\ell \\geq 1$.\n%\nThen, $\\sy_\\ell$ is $\\mathcal U_\\ell$-measurable and for all $\\ell \\in \\{1,\\dots,t_{k+1}-t_k-1\\}$, we have \n%\n\\[ {\\Exp[{\\sy^{\\ell+1} \\mid \\mathcal U_\\ell}] = {\\sum}_{i=t_k}^{t_k+\\ell} \\alpha_i\\Exp[{\\se^i \\mid \\mathcal U_\\ell}] = \\sy^\\ell + \\alpha_{t_k+\\ell} \\Exp[{\\se^{t_k+\\ell} \\mid \\mathcal{F}_{t_k+\\ell}}] = \\sy^\\ell.} \\]\n% \n(Similarly for $\\ell \\geq t_{k+1}-t_k$). Hence, \\revise{$\\{\\sy^\\ell\\}_{\\ell\\in\\N}$} defines a \\revise{$\\{\\mathcal{U}_{\\ell}\\}_{\\ell\\in\\N}$}-martingale. Let us further set $\\sr_k = \\scs_{t_k,t_{k+1}}$. By the Burkholder-Davis-Gundy inequality (\\cref{Thm:BDG} with $p=2$ and $C_p\\leq 4$) and assumption \\ref{B1}, we have $\\Exp[{\\sr_k^2}] =\\Exp[{{\\sup}_{\\ell>0} \\|\\sy^\\ell\\|^2}]$ and\n%\n\\begin{align*} \n\\nonumber\\Exp[{\\sr_k^2}] \\leq 4 \\Exp\\Big[{ {\\sum}_{\\ell=1}^{\\infty}\\|\\sy^\\ell-\\sy^{\\ell-1}\\|^2}\\Big] & = 4 {\\sum}_{i=t_k}^{t_{k+1}-1}\\alpha_i^2 \\Exp[{ \\|\\se^i\\|^2}]  \\leq 4 {\\sum}_{i=t_k}^{t_{k+1}-1} \\alpha_i^2\\sigma_i^2. \n\\end{align*}\n%\nThis proves the first claim in \\Cref{lem:err_estimate}. Since $\\revise{\\{\\beta_k\\}_{k\\in\\N}}$ is non-decreasing, we obtain%\n%\n\\begin{equation}\\label{eq:app-bdg}\n\t\\beta_{t_k}^2 \\Exp[\\sr_k^2] \\leq 4 \\beta_{t_k}^2 {\\sum}_{i=t_k}^{t_{k+1}-1} \\alpha_i^2\\sigma_i^2 \\leq 4  {\\sum}_{i=t_k}^{t_{k+1}-1} \\alpha_i^2\\beta_{i}^2\\sigma_i^2\n\\end{equation}\n%\nfor all $k \\in \\N$. The monotone convergence theorem and \\cref{eq:app-bdg} further imply\n%\n\\begin{align*}\n \\Exp\\Big[{ {\\sum}_{k=0}^\\infty \\beta_{t_k}^2{\\sr_k^2}}\\Big] = {\\sum}_{k=0}^{\\infty}\\beta_{t_k}^2 \\Exp[\\sr_k^2] \n \\leq \n    {4} {\\sum}_{i=t_{0}}^\\infty \\alpha_i^2\\beta_i^2\\sigma_i^2 < \\infty,\n\\end{align*}\n%\nand consequently, it holds that $\\sum_{k=0}^\\infty \\beta_{t_k}^2{\\sr_k^2} < \\infty$ a$.$s$.$.\n\\end{proof}\n\n\\begin{proof}[Proof of (a)]\nLet $\\omega\\in\\Omega$ be arbitrary and denote $x^k = \\sx^k(\\omega)$, $z^k = \\sz^k(\\omega)$, $p = \\scp^{m,n}(\\omega)$, $u = \\su^{m,n}(\\omega)$, $\\tau = \\tau_{m,n}$, etc$.$. Recalling $\\Fnor(z) = \\nabla f(x) + \\lambda^{-1}(z-x)$ and by the definition \\cref{eq:u-def1} of $\\su^{m,n}$, we can express $\\Fnor(z^n)$ as\n%\n\\begin{align*} \n    \\Fnor(z^n) \n    & = \\Fnor(z^m) + [\\nabla f(x^n)-\\nabla f(x^m)] + \\lambda^{-1}\\prt{z^n-z^m+x^m-x^n}\\\\\n    & = u + [\\nabla f(x^n)-\\nabla f(x^m)] - \\lambda^{-1}(x^n-x^m).\n\\end{align*}\t\n%\nTaking squares and using the Lipschitz continuity of $\\nabla f$, this yields\n%\n\\begin{align}\n            \\nonumber \\norm{\\Fnor(z^n)}^2 & = \\norm{u}^2 + 2\\iprod{u}{(\\nabla f(x^n)-\\nabla f(x^m)) - \\lambda^{-1}(x^n-x^m)} \\\\\n            \\nonumber&\\hspace{4ex}  +  \\norm{(\\nabla f(x^n)-\\nabla f(x^m)) - \\lambda^{-1}(x^n-x^m)}^2\\\\\n            \\label{C1} & \\hspace{-6ex} \\leq \\norm{u}^2 + 2\\iprod{u}{(\\nabla f(x^n)-\\nabla f(x^m))-\\lambda^{-1}(x^n-x^m)} + (\\sL+{\\lambda}^{-1})^2 \\norm{x^n-x^m}^2.\n\\end{align}\n%\nUsing $u=(1-\\lambda^{-1}\\tau)\\Fnor(z^m) + \\lambda^{-1}p$ (cf$.$ \\cref{eq:u-def3}) and $\\norm{a+b}^2 \\leq (1+\\varepsilon) \\norm{a}^2 + (1+\\frac{1}{\\varepsilon})\\norm{b}^2$ with $a= (1-\\lambda^{-1}\\tau) \\Fnor(z^m)$, $b = \\lambda^{-1}p$, and $\\varepsilon = \\frac{\\lambda^{-1}\\tau}{1-\\lambda^{-1}\\tau}$, we have\n%\n\\begin{equation}\\label{C2}\n        \\norm{u}^2 \\leq (1-\\lambda^{-1}{\\tau}) \\|{\\Fnor(z^m)}\\|^2 + (\\lambda\\tau)^{-1} \\|{p}\\|^2.\n\\end{equation}\n%\nUtilizing Young's inequality for $2\\iprod{u}{\\nabla f(x^n)-\\nabla f(x^m)}$ and \\cref{C2}, we obtain \n%\n\\begin{align}\n            \\nonumber 2\\iprod{u}{\\nabla f(x^n)-\\nabla f(x^m)} & \\leq \\tfrac{\\tau}{2\\lambda} \\norm{u}^2 + \\tfrac{2\\lambda}{\\tau} \\norm{\\nabla f(x^n)-\\nabla f(x^m)}^2\\\\\n            \\label{C3} & \\leq  \\tfrac{\\tau}{2\\lambda}\\norm{\\Fnor(z^m)}^2 + \\tfrac{1}{2\\lambda^2} \\norm{p}^2 + \\tfrac{2\\lambda \\sL^2}{\\tau} \\norm{x^n-x^m}^2,\n\\end{align}\n%\nSumming up the estimates \\cref{C1}--\\cref{C3}, and noticing $\\tau \\leq \\lambda$, it follows\n%\n\\begin{align*}\n        \\norm{\\Fnor(z^n)}^2  & \\leq  ({1-\\tfrac{\\tau}{2\\lambda}})\\|{\\Fnor(z^m)}\\|^2 - \\tfrac{2}{\\lambda}\\iprod{u}{x^n-x^m} \\\\\n        & \\hspace{4ex} + [(\\sL+\\lambda^{-1})^2 + {2\\sL^2\\lambda}{\\tau^{-1}}] \\norm{x^n-x^m}^2 + \\tfrac{3}{2\\lambda\\tau} \\norm{p}^2.\n\\end{align*}\n\n\\emph{Proof of (b).} By \\ref{A1}--\\ref{A2}, $\\nabla f$ is $\\sL$-continuous and $\\vp$ is convex. Thus, we have\n%\n\\begin{equation*}\n        \\psi(x^n)-\\psi(x^m) \\leq \\iprod{\\nabla f(x^m)}{x^n-x^m}+\\tfrac{\\sL}{2}\\|x^n-x^m\\|^2 + \\tfrac{1}{\\lambda}\\iprod{z^n-x^n}{x^n-x^m},\n\\end{equation*}\n%\nwhere we used $\\lambda^{-1}(z^n-x^n) \\in\\partial \\vp(x^n)$ in the last inequality. Moreover, based on the definition of $\\Fnor$, we can reformulate %\\cref{eq:lem-1-est-1} \nthe above inequality as\n%\n\\begin{align*}\n\\psi(x^n)-\\psi(x^m) & \\leq \\iprod{\\nabla f(x^m)+\\lambda^{-1}(z^m-x^m)}{x^n-x^m}+\\tfrac{\\sL}{2}\\|{x^n-x^m}\\|^2\\\\\n& \\hspace{4ex} + \\lambda^{-1}\\iprod{z^n-x^n}{x^n-x^m} - \\lambda^{-1}\\iprod{z^m-x^m}{x^n-x^m}\\\\\n& \\hspace{-4ex} = [{\\tfrac{\\sL}{2}-\\tfrac{1}{\\lambda}}]\\|{x^n-x^m}\\|^2 + \t\\iprod{\\Fnor(z^m)}{x^n-x^m} + \\tfrac{1}{\\lambda}\\iprod{z^n-z^m}{x^n-x^m}.\n\\end{align*}\n%\nThis concludes the proof by substituting \\cref{eq:u-def1} into the last expression.\n\\end{proof}\n\n\\begin{proof}\nAs shown in \\cref{eq:nor is Lip}, $\\Fnor$ is Lipschitz with modulus $\\Llambda$ and we have \n%\n\\begin{align*}\n        \\|\\Fnor(\\sz^i)\\| &\\leq \\|\\Fnor(\\sz^i) - \\Fnor(\\sz^{t_{\\fk(i)}})\\| + \\| \\Fnor(\\sz^{t_{\\fk(i)}})\\|\\\\ &\\leq  \\Llambda \\|\\sz^i-\\sz^{t_{\\fk(i)}}\\|+ \\| \\Fnor(\\sz^{t_{\\fk(i)}})\\| \\leq (2\\Llambda\\sT+1)\\| \\Fnor(\\sz^{t_{\\fk(i)}})\\| + 2\\Llambda \\scs_{t_{\\fk(i)},i}\\\\\n        &\\leq (2\\Llambda\\sT+1)\\| \\Fnor(\\sz^{t_{\\fk(i)}})\\| + 2\\Llambda \\sr_{\\fk(i)},\n\\end{align*}\n%\nwhere the \\revise{second to last} inequality follows from \\cref{lem:est-err} by setting $m=t_{\\fk(i)}$, $n=t_{\\fk(i)+1}$ (the condition $\\tau_{t_{\\fk(i)},t_{\\fk(i)+1}}\\leq \\sT \\leq 1/(2\\Llambda)$ is due to \\cref{eq:def delta and T}). Squaring both sides of the above inequality and using $(a+b)^2\\leq 2a^2+2b^2$, we obtain \\cref{eq:tech1}.\n\nBased on \\cref{lem:basic bounds} (b) and the definition of $\\su^{m,n}$ (see \\cref{eq:u-def1}), it holds that\n%\n\\[\n    \\psi(\\sx^n) \\leq \\psi(\\sx^m) + (\\sL/2-\\lambda^{-1})\\norm{\\sx^n-\\sx^m}^2 + \\iprod{\\lambda^{-1}(\\sz^n-\\sz^m) + \\Fnor(\\sz^m)}{\\sx^n-\\sx^m}.\n\\]\n%\nApplying the nonexpansiveness of the proximity operator, \\cref{prox-prop}, and $\\iprod{\\Fnor(\\sz^m)}{\\sx^n-\\sx^m} \\leq \\frac{\\lambda}{4} \\norm{\\Fnor(\\sz^m)}^2 + \\lambda^{-1} \\norm{\\sx^n-\\sx^m}^2$, we have\n%\n\\[\n    \\psi(\\sx^n)\\leq  \\psi(\\sx^m) + \\tfrac{\\Llambda}{2} \\norm{\\sz^n-\\sz^m}^2 + \\tfrac{\\lambda}{4} \\norm{\\Fnor(\\sz^m)}^2.\n\\]\n%\nInvoking \\Cref{lem:est-err} (for $\\norm{\\sz^n-\\sz^m}$) and $(a+b)^2\\leq 2a^2+2b^2$,  we obtain\n%\n\\[\n    \\psi(\\sx^n)\\leq  \\psi(\\sx^m) + (4\\Llambda\\tau_{m,n}^2+\\tfrac{\\lambda}{4}) \\norm{\\Fnor(\\sz^m)}^2 + 4\\Llambda \\scs_{m,n}^2.\n\\]\n%\nThen, \\cref{eq:tech2} readily follows by setting $n = i$, $m = t_{\\fk(i)}$ and using $\\tau_{t_{\\fk(i)},i}\\leq \\sT$. Similarly, \\cref{eq:tech3} can be obtained by setting $n = t_{\\fk(i)+1}$, $m = i$ and using  $\\tau_{i,t_{\\fk(i)+1}}\\leq \\sT$ and $\\scs_{i,t_{\\fk(i)+1}} = \\max_{i<j\\leq t_{\\fk(i)+1}}\\|{\\sum}_{\\ell=i}^{\\revise{j}-1} \\alpha_\\ell \\se^\\ell\\| \\leq \\scs_{t_{\\fk(i)},i}+  \\scs_{t_{\\fk(i)},t_{\\fk(i)+1}}  \\leq 2 \\sr_{\\fk(i)}$.\n\\end{proof}\n\n\\begin{proof}\nLet $\\omega\\in\\mathcal C$ be arbitrary and let $x^k = \\sx^k(\\omega)$,  $z^k = \\sz^k(\\omega)$, etc$.$ be the corresponding realizations. By the definition of the event $\\mathcal C$, there exists $\\revise{\\{p_k\\}_{k\\in\\N}}\\subseteq\\N$ such that \\revise{$\\{x^{p_k}\\}_{k\\in\\N}$} is bounded and we have $\\Fnor(z^k) \\to 0$, $\\psi(x^k) \\to \\psi^*$. Using the definition of the normal map, we can infer $\\|z^{p_k}\\| \\leq \\lambda(\\|\\Fnor(z^{p_k})\\|+\\|\\nabla f(x^{p_k})\\|)+\\|x^{p_k}\\|$. Hence, \\revise{$\\{z^{p_k}\\}_{k\\in\\N}$} is also bounded and there is a convergent subsequence \\revise{$\\{z^{q_k}\\}_{k\\in\\N}$} of \\revise{$\\{z^{p_k}\\}_{k\\in\\N}$} such that $\\lim_{k\\to\\infty} z^{q_k} = z^*$ for some $z^*$ satisfying $\\|\\Fnor(z^{*})\\|=0$ and $x^*:=\\proxl(z^*) \\in \\crit(\\psi)$. Due to $\\|x^k-x^*\\| \\leq \\|z^k - z^*\\|$, we then have $\\lim_{k\\to\\infty} x^{q_k}=x^*$. Using the continuity of the Moreau envelope, this yields\n%\n\\begin{align*} \\psi^* & = {\\lim}_{k \\to \\infty}\\,\\psi(x^{q_k})  = {\\lim}_{k \\to \\infty}\\,\\vp(\\proxi{\\lambda}(z^{q_k})) + f(x^{q_k}) \\\\ & = {\\lim}_{k\\to\\infty}\\,\\envi{\\lambda}(z^{q_k}) - \\tfrac{1}{2\\lambda} \\|z^{q_k}- x^{q_k}\\|^2 + f(x^*) = \\psi(x^*). \\end{align*}\n%\nThis readily implies $H(z^k) \\to \\psi^* = H(z^*)$ and finishes the proof.\n\\end{proof}","units":[{"start":44046,"end":44652,"start_line":596,"end_line":604,"raw_tex":"\\begin{lemma}[Iterate bounds]\n    \\label{lem:est-err}\n     Let \\ref{A1}--\\ref{A2} and \\ref{B1}--\\ref{B2} hold and let $\\revise{\\{\\sx^k\\}_{k\\in\\N}}$ and $\\revise{\\{\\sz^k\\}_{k\\in\\N}}$ be generated by $\\NSGD$. Let us set $\\Llambda := \\sL+2\\lambda^{-1}$. For all $0\\leq m<n$ with $\\tau_{m,n} \\leq 1/(2\\Llambda)$, we have\n     %\n    \\begin{align*}\n        \\max_{m<i\\leq n}\\|\\sx^i-\\sx^m\\| &\\leq \\max_{m<i\\leq n}\\|\\sz^i-\\sz^m\\| \\leq 2\\tau_{m,n} \\norm{\\Fnor(\\sz^m)} + 2\\scs_{m,n}\\quad \\text{and}\\\\\n        \\|{\\scp^{m,n}}\\| &\\leq 2\\Llambda\\tau_{m,n}^2\\norm{\\Fnor(\\sz^m)} + 2\\scs_{m,n}. \n    \\end{align*}\n\\end{lemma}","sha256_utf8":"4c85434bb548b9530035fb781f0c1fb11514d5a26015ee65cfaacf1f22454363"},{"start":45036,"end":45731,"start_line":608,"end_line":614,"raw_tex":"\\begin{lemma}\n    \\label{lem:err_estimate}\n    Let \\ref{B1}--\\ref{B2} hold and let $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}$ be given. Then for any (arbitrary) subsequence $\\revise{\\{t_k\\}_{k\\in\\N}} \\subseteq \\N$, we have $\\Exp[{{\\scs_{t_k,t_{k+1}}^2}}] \\leq 4 \\sum_{i=t_k}^{t_{k+1}-1}\\alpha_i^2\\sigma_i^2$. Moreover, if there is a non-decreasing sequence $\\revise{\\{\\beta_k\\}_{k\\in\\N}}\\subset\\R_{++}$ such that ${\\sum}_{k=0}^\\infty \\alpha_k^2\\beta_k^2\\sigma_k^2 < \\infty$, it holds that\n    %\n    \\[ {\\sum}_{k=0}^\\infty \\beta_{t_k}^2 \\Exp[{{\\scs_{t_k,t_{k+1}}^2}}] < \\infty \\quad \\text{and}\\quad {\\sum}_{k=0}^\\infty \\beta_{t_k}^2 \\scs_{t_k,t_{k+1}}^2 < \\infty \\quad \\text{a$.$s$.$.}\\] \n    %\n\\end{lemma}","sha256_utf8":"f4ae0ff751342677812ca826ae3018f5df697990f7df39766845a18fc74dc14f"},{"start":46494,"end":47397,"start_line":628,"end_line":635,"raw_tex":"\\begin{lemma}\n\t\\label{lem:basic bounds}\n\tLet the assumptions \\ref{A1}--\\ref{A2} and \\ref{B1}--\\ref{B2} be satisfied and let the sequences $\\revise{\\{\\sx^k\\}_{k\\in\\N}}$, $\\revise{\\{\\sz^k\\}_{k\\in\\N}}$ be generated by $\\NSGD$. Let $0 \\leq m<n$ be given with $\\tau_{m,n} < \\lambda$. Then, the following two bounds are valid:\n    \\begin{enumerate}[label=\\textup{\\textrm{(\\alph*)}},leftmargin=3em,topsep=2pt,itemsep=.5ex,partopsep=0ex]\n    \\item It holds that $\\norm{\\Fnor(\\sz^n)}^2 \\leq  ({1-\\frac{\\tau_{m,n}}{2\\lambda}})\\norm{\\Fnor(\\sz^m)}^2 - \\frac{2}{\\lambda}\\iprod{\\su^{m,n}}{\\sx^n-\\sx^m} + [({\\sL+\\frac{1}{\\lambda}})^2 + \\frac{2\\lambda \\sL^2}{\\tau_{m,n}}] \\norm{\\sx^n-\\sx^m}^2 + \\frac{3}{2\\lambda\\tau_{m,n}} \\norm{\\scp^{m,n}}^2$.\n    \\item We have $\\psi(\\sx^n) \\leq \\psi(\\sx^m) + ({\\frac{\\sL}{2}-\\frac{1}{\\lambda}})\\norm{\\sx^n-\\sx^m}^2 + \\iprod{\\su^{m,n}}{\\sx^n-\\sx^m}.$\t\n    \\end{enumerate}\n\\end{lemma}","sha256_utf8":"c9684ddc2757aef12331c4298a567f45e97f4b318adb92e5845faddfb1dac0ba"},{"start":47399,"end":48596,"start_line":637,"end_line":645,"raw_tex":"The proof of \\cref{lem:basic bounds} is relegated to \\cref{app:lem:basic bounds}. Based on \\cref{lem:basic bounds}, we see that proper control of the inner product $\\langle \\su^{m,n}, \\sx^n-\\sx^m \\rangle$ is crucial to achieve descent (of $\\psi$). The following estimate illustrates that while this term can generate a desirable decrease, it also introduces an additional nonnegative error term: \n%\n    \\begin{align}\n    \\nonumber \\iprod{\\su^{m,n}}{\\sx^n-\\sx^m} & = (\\lambda^{-1}-\\tau_{m,n}^{-1}) \\iprod{\\sz^{n}-\\sz^m}{\\sx^n-\\sx^m} + \\tau_{m,n}^{-1}\\iprod{\\scp^{m,n}}{\\sx^n-\\sx^m}\\\\\n    \\nonumber & \\leq (\\lambda^{-1}-\\tau_{m,n}^{-1}) \\norm{\\sx^n-\\sx^m}^2 + \\tau_{m,n}^{-1} ({\\norm{\\scp^{m,n}}^2 + 0.25\\norm{\\sx^n-\\sx^m}^2})\\\\\n    \\label{u-descent} & = (\\lambda^{-1}-0.75\\tau_{m,n}^{-1}) \\norm{\\sx^n-\\sx^m}^2 + \\tau_{m,n}^{-1}\\norm{\\scp^{m,n}}^2.\n    \\end{align}\n%\nHere, we applied \\cref{eq:u-def2}, \\cref{prox-prop}, and Young's inequality. Thankfully and as we will verify in \\Cref{subsec:merit function}, the extra term $\\tau_{m,n}^{-1}\\norm{\\scp^{m,n}}^2$ in \\cref{u-descent} can be effectively compensated for by leveraging the normal map bound and choosing $\\tau_{m,n}$ sufficiently small. \n","sha256_utf8":"a7a05f5d9b90f9c657d7a2897783d51421fa58ec6211bd505ab80f42450dbe27"},{"start":49235,"end":49831,"start_line":656,"end_line":663,"raw_tex":"\\begin{proposition}[Approximate descent property]\n\\label{lem:merit-descent-1}\nLet \\ref{A1}--\\ref{A2} and \\ref{B1}--\\ref{B2} hold and let $\\revise{\\{\\sz^k\\}_{k\\in\\N}}$ be generated by $\\NSGD$. Suppose, \\revise{for $0\\leq m <n$, the} step sizes $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}$ satisfy $\\tau_{m,n} = {\\sum}_{i=m}^{n-1}\\alpha_i\\leq \\frac{\\sqrt{\\xi}}{8\\Llambda}$ where $\\Llambda := \\sL+\\frac{2}{\\lambda}$. We then have{}%\\vspace{-2ex}\n%\n\\begin{equation}\n    H(\\sz^n) - H(\\sz^m) \\leq  -\\frac{\\xi\\tau_{m,n}}{8} \\norm{\\Fnor(\\sz^m)}^2 + \\frac{8}{\\tau_{m,n}} \\scs_{m,n}^2.\n\\end{equation}\n\\end{proposition}","sha256_utf8":"2d4f29c31ecd211e290cd893da86feff0911ad497ce128ba2c341654f7efdfcc"},{"start":49833,"end":52012,"start_line":665,"end_line":694,"raw_tex":"\\begin{proof}\nBy \\cref{lem:basic bounds} and setting $\\tau = \\tau_{m,n}$, $\\su = \\su^{m,n}$, $\\scp = \\scp^{m,n}$, and $\\scs = \\scs_{m,n}$, it holds that\n%\n\\begin{align}\n \\nonumber   H(\\sz^n) - H(\\sz^m) & \\leq  -\\frac{\\xi\\tau}{4} \\norm{\\Fnor(\\sz^m)}^2 + (1-\\xi) \\iprod{\\su}{\\sx^n-\\sx^m} + \\frac{3\\xi}{4\\tau} \\norm{\\scp}^2\\\\\n \\label{H-descent1}   &\\hspace{4ex} + \\Big[{\\Big({\\frac{\\sL}{2}-\\frac{1}{\\lambda}}\\Big) + \\frac{\\xi\\lambda}{2} \\Big({\\sL+\\frac{1}{\\lambda}}\\Big)^2 + \\frac{\\xi \\lambda^2\\sL^2}{\\tau}}\\Big] \\norm{\\sx^n-\\sx^m}^2.\n\\end{align}\n%\nInvoking the upper bound for $\\iprod{\\su}{\\sx^n-\\sx^m}$ in \\cref{u-descent}, this yields \n%\n\\begin{align}\n\\nonumber    H(\\sz^n) - H(\\sz^m) & \\leq -\\frac{\\xi\\tau}{4} \\norm{\\Fnor(\\sz^m)}^2 + \\frac{4-\\xi}{4\\tau}\\norm{\\scp}^2\\\\\n\\label{H-descent2}    &\\hspace{-6ex} + \\underbracket{\\Big[\\frac{\\sL}{2}-\\frac{\\xi}{\\lambda} + \\frac{\\xi\\lambda}{2} \\Big({\\sL+\\frac{1}{\\lambda}}\\Big)^2 + \\frac{4\\xi \\lambda^2\\sL^2 - 3(1-\\xi)}{4\\tau}\\Big]}_{=:T_1} \\norm{\\sx^n-\\sx^m}^2.\n\\end{align}\n%\nNext, we bound $T_1$. Using $\\xi= {(2+2\\lambda^2\\sL^2)^{-1}}$ and $(a+b)^2 \\leq 2(a^2+b^2)$, $a,b \\in \\R$, we have $\\frac12\\xi\\lambda(\\sL+\\lambda^{-1})^2 \\leq \\xi \\lambda (\\sL^2+\\lambda^{-2}) = \\frac{1}{2\\lambda}$ and $4\\xi \\lambda^2\\sL^2 - 3(1-\\xi) \\leq 4\\xi(\\lambda^2\\sL^2+1)-3 = -1$. Applying the bound $\\tau \\leq \\frac{\\sqrt{\\xi}}{8\\Llambda} \\leq \\frac{1}{2(\\sL+\\lambda^{-1})}$, we then obtain $T_1 \\leq \\frac{\\sL+\\lambda^{-1}}{2} - \\frac{\\xi}{\\lambda} - \\frac{1}{4\\tau} \\leq - \\frac{\\xi}{\\lambda} < 0$. Therefore, due to $\\frac{4-\\xi}{4\\tau}\\leq \\frac{1}{\\tau}$, the estimate \\cref{H-descent2} simplifies to\n%\n\\[\nH(\\sz^n) - H(\\sz^m) \\leq  -{(\\xi\\tau/4)} \\norm{\\Fnor(\\sz^m)}^2 + {\\tau^{-1}} \\norm{\\scp}^2.\n\\]\n%\nBy \\cref{lem:est-err}, we further have $\\|\\scp\\|^2 \\leq 8\\Llambda^2 \\tau^4 \\|\\Fnor(\\sz^m)\\|^2+8\\scs^2$. This implies\n%\n\\begin{align*}\n    H(\\sz^n) - H(\\sz^m) \\leq  - \\Big[\\frac{\\xi}{4} - 8\\Llambda^2 \\tau^2 \\Big]\\tau\\|{\\Fnor(\\sz^m)}\\|^2 + \\frac{8\\scs^2}{\\tau} \n    \\leq  -\\frac{\\xi\\tau}{8} \\norm{\\Fnor(\\sz^m)}^2 + \\frac{8\\scs^2}{\\tau} ,\n\\end{align*}\n%\nwhere the last inequality is again due to $\\tau \\leq \\frac{\\sqrt{\\xi}}{8\\Llambda}$.\n\\end{proof}","sha256_utf8":"01fa7195f33d01f8c47f15cd570f37075e24b25f43b8ea8879f4913ad852deef"},{"start":53953,"end":54639,"start_line":716,"end_line":725,"raw_tex":"\\begin{lemma}\\label{lem:exist time}\nLet $\\sT > 0$, $\\delta \\in (0,1)$ be given. The following assertions are valid.\n%\n \\begin{enumerate}[label=\\textup{\\textrm{(\\alph*)}},leftmargin=3em,topsep=0pt,itemsep=.5ex,partopsep=0ex]\n \\item Assume $\\alpha_k \\in (0,(1-\\delta) \\sT]$, $k< K$, and $\\alpha_k = \\alpha \\in(0,(1-\\delta)\\sT]$ for all $k \\geq K$ and some $K\\in\\N$. We then have $t_k \\to \\infty$ and $\\delta\\sT \\leq \\tau_{t_k,t_{k+1}} \\leq \\sT$ for all $k \\in\\N$.\n    \n \\item\n    If $\\lim_{k\\to\\infty}\\alpha_k = 0$ and $\\sum_{k=0}^\\infty \\alpha_k = \\infty$, it holds that $t_k \\to \\infty$ and $\\delta \\sT \\leq \\tau_{t_k,t_{k+1}} \\leq \\sT$ for all $k\\in\\N$.\n    \\end{enumerate}\n\\end{lemma}","sha256_utf8":"156243ff24a3cea843545906d900f7f649c44fddd66a931b7d3aaf2c6c0f11c0"},{"start":54641,"end":55651,"start_line":727,"end_line":731,"raw_tex":"\\begin{proof} \nHere, we use the convention $\\sup \\emptyset = -\\infty$. According to the construction of $\\revise{\\{t_k\\}_{k\\in\\N}}$ in \\cref{def:time win} and $\\alpha_k \\leq (1-\\delta) \\sT$, we have $t_0 = 0$ and $t_{k+1}=\\varpi(t_k) \\neq -\\infty$ for all $k\\in\\N$. Furthermore, we have $t_{k+1} \\neq \\infty$ since $\\sum_{k=0}^{\\infty} \\alpha_k = \\infty$ implies $\\tau_{t_k,n}\\rightarrow \\infty$ as $n\\rightarrow \\infty$. This also yields $t_{k} \\to \\infty$, i.e., $\\revise{\\{t_k\\}_{k\\in\\N}}$ is an infinite subsequence.  By the definition of $\\varpi(k)$, it follows $\\tau_{t_k,t_{k+1}} =  \\tau_{t_k,\\varpi(t_{k})} \\leq \\sT$. In addition, by the optimality of $\\varpi(t_k)$, we obtain $\\tau_{t_k,t_{k+1}} =  \\tau_{t_k,\\varpi(t_{k})} \\geq \\sT - \\alpha_{\\varpi(t_k)} \\geq \\delta \\sT$ for all $k$.\n% \nWe continue with part (b). Due to $\\alpha_k \\to 0$ and the definition of $t_0$, we have $\\alpha_k \\leq (1-\\delta)\\sT$ for all $k\\geq  t_0$. Hence, all previous derivations are applicable in this case.\n\\end{proof}","sha256_utf8":"a1ec43b33488ad9a874c25c7bfe7d5257050c4781cb21a2069ca268db028d045"},{"start":62893,"end":63685,"start_line":833,"end_line":843,"raw_tex":"\\begin{lemma}\\label{lem:index-time bound}\n    Let \\ref{A1}--\\ref{A2} and \\ref{B1}--\\ref{B2} hold and let $\\revise{\\{\\sx^k\\}_{k\\in\\N}}$ and $\\revise{\\{\\sz^k\\}_{k\\in\\N}}$ be generated by $\\NSGD$. Recalling $\\Llambda := \\sL+2\\lambda^{-1}$, we have for all $i>t_0$:\n%\n    \\begin{subequations}\n    \\begin{align}\n    \\label{eq:tech1}\\|\\Fnor(\\sz^i)\\|^2 &\\leq 2\\prt{2\\Llambda\\sT +1}^2\\| \\Fnor(\\sz^{t_{\\fk(i)}})\\|^2 + 8\\Llambda^2 \\sr_{\\fk(i)}^2, \\\\\n    \\label{eq:tech2}\\psi(\\sx^i) &\\leq \\psi(\\sx^{t_{\\fk(i)}})+(4\\Llambda\\sT^2+{\\lambda}/{4})\\norm{\\Fnor(\\sz^{t_{\\fk(i)}})}^2 + 4\\Llambda \\sr_{\\fk(i)}^2,\\\\\n    \\label{eq:tech3}\\psi(\\sx^i) &\\geq \\psi(\\sx^{t_{\\fk(i)+1}})-(4\\Llambda\\sT^2+{\\lambda}/{4})\\norm{\\Fnor(\\sz^{t_{\\fk(i)}})}^2 - 16\\Llambda \\sr_{\\fk(i)}^2.\n\\end{align}\n \\end{subequations}\n\\end{lemma}","sha256_utf8":"95c8f79d220c9af1491cc05f105bfd8352f6630dae9837ddcad18e010e512201"},{"start":64359,"end":65000,"start_line":847,"end_line":855,"raw_tex":"\\begin{theorem}[Global convergence]\n\\label{thm:global_convergence-2}\nLet \\ref{A1}--\\ref{A2}, \\ref{B1}--\\ref{B3} hold and let $\\revise{\\{\\sx^k\\}_{k\\in\\N}}$, $\\revise{\\{\\sz^k\\}_{k\\in\\N}}$ be generated by $\\NSGD$. Then, there is $\\spsi^* : \\Omega \\to \\R$ such that: \n%\n \\begin{enumerate}[label=\\textup{\\textrm{(\\alph*)}},leftmargin=3em,topsep=0.5ex,itemsep=.5ex,partopsep=0ex]\n        \\item $\\Fnor(\\sz^k)\\to 0$, $\\|\\partial \\psi(\\sx^k)\\|_{-} \\to 0$, and $\\psi(\\sx^k)\\to \\spsi^*$ a.s..\n\t\\item $\\Exp[\\norm{\\Fnor(\\sz^k)}^2]\\to 0$, $\\Exp[\\|\\partial \\psi(\\sx^k)\\|_{-}^2]\\to0$, and $\\Exp[\\psi(\\sx^k)] \\to \\Exp[\\spsi^*]$.\n\\end{enumerate}\n\\end{theorem}","sha256_utf8":"7fc12b623366288c28c73878a177da01ef099bcdf6c6dc2644052e28ea5d50ec"},{"start":65566,"end":69594,"start_line":859,"end_line":899,"raw_tex":"\\begin{proof}\nBy \\cref{lem:merit-descent-1}, \\cref{lem:exist time} (b) (with $\\delta = 4/5$; cf$.$ \\cref{eq:def delta and T}), we have \n%\n\\begin{equation}\n    \\label{eq:thm:global-1}\n    \\mer(\\sz^{t_{k+1}}) - \\mer(\\sz^{t_{k}}) \\leq -\\frac{\\xi\\sT}{{10}}\\|\\Fnor(\\sz^{t_k})\\|^2 + \\frac{10}{\\sT}\\cdot\\sr_{k}^2\\quad \\forall \\,k\\in\\N.\n\\end{equation}\n%\nThe condition $\\sum_{k=0}^{\\infty}\\alpha_k^2\\sigma_k^2<\\infty$ and \\cref{lem:err_estimate} imply that ${\\sum}_{k=0}^\\infty \\sr_{k}^2<\\infty$ a$.$s$.$ and ${\\sum}_{k=0}^\\infty\\Exp[{\\sr_{k}^2}]<\\infty$. Setting $\\su_k:=\\frac{10}{\\sT}\\, \\sum_{i=k}^\\infty \\sr_{i}^2$, it follows\n%\n\\begin{equation}\n\t\\label{eq:strong-0}\n\t\\mer(\\sz^{t_{k+1}}) + \\su_{k+1} \\leq  \\mer(\\sz^{t_{k}}) + \\su_{k}- ({\\xi \\sT}/{10}) \\|\\Fnor(\\sz^{t_k})\\|^2.\n\\end{equation}\t\n%\nSumming \\cref{eq:thm:global-1} for $k=0,1,\\ldots$ and using the bound $\\mer(z)\\geq \\bar \\psi$, this yields\n%\n\\begin{equation}\\label{eq:norm-sum}\n{\\xi\\sT}\\,{\\sum}_{k=0}^\\infty \\|\\Fnor(\\sz^{t_k})\\|^2 \\leq 10[\\mer(\\sz^{t_{0}}) + \\su_{0} - \\bar \\psi].\n\\end{equation}\n%\nHence, due to ${\\sum}_{k=0}^\\infty \\sr_{k}^2<\\infty$ a$.$s$.$, we can conclude $\\|\\Fnor(\\sz^{t_k})\\|\\to 0$, $\\sr_{k}\\to0$, and $\\su_{k}\\to0$ a$.$s$.$. Moreover, based on the definition \\cref{index-def} of $\\fk(i)$, we can infer $\\|\\Fnor(\\sz^{t_{\\fk(i)}})\\|\\to 0$ and $\\sr_{\\fk(i)}\\to0$ as $i\\to\\infty$ a$.$s$.$. This proves $\\|\\Fnor(\\sz^{i})\\|\\to 0$ a$.$s$.$ owing to \\cref{eq:tech1}. \n\nTaking expectation in \\cref{eq:norm-sum} and using $\\sum_{k=0}^\\infty\\Exp{[\\sr_{k}^2]}<\\infty$, we further obtain $\\Exp[\\sr_{k}^2]\\to0$, $\\Exp[\\su_{k}^2]\\to0$, and $\\Exp[{\\|\\Fnor(\\sz^{t_k})\\|^2}]\\to 0$. Thus, we can infer $\\Exp[\\|\\Fnor(\\sz^{i})\\|^2]\\to 0$ by taking expectation in \\cref{eq:tech1} and passing $i\\to\\infty$. By \\cref{eq:stat-am}, we have $\\|\\partial \\psi(\\proxl(z))\\|_{-} \\leq \\|\\Fnor(z)\\|$, which implies $\\|\\partial \\psi(\\sx^k)\\|_{-} \\to 0$ a$.$s$.$ and $\\Exp[{\\|\\partial \\psi(\\sx^k)\\|^2_{-}}] \\to 0$.\n\nNext, we show $\\psi(\\sx^{t_k}) \\to \\spsi^*$ a$.$s$.$. By \\cref{eq:strong-0}, $\\{\\mer(\\sz^{t_k})+\\su_k\\}_k$ is monotonically non-increasing, and thus, we conclude that $\\mer(\\sz^{t_k})+\\su_k \\to \\spsi^*$ a$.$s$.$ for some random variable $\\spsi^*$. Using $\\|\\Fnor(\\sz^{t_k})\\|\\to0$, $\\su_k\\to 0$, and $\\psi(\\sx^{t_k}) = \\mer(\\sz^{t_k}) + \\su_k - (\\frac{\\xi\\lambda}{2}\\|\\Fnor(\\sz^{t_k})\\|^2+\\su_k)$, we obtain $\\psi(\\sx^{t_k}) \\to \\spsi^*$ a$.$s$.$. Taking $i\\to \\infty$ in \\cref{eq:tech2} and \\cref{eq:tech3}, it follows %\n\\begin{align*}\n     \\psi(\\sx^i) &\\leq \\psi(\\sx^{t_{\\fk(i)}})+(4\\Llambda\\sT^2+\\lambda/4)\\norm{\\Fnor(\\sz^{t_{\\fk(i)}})}^2 + 4\\Llambda \\sr_{\\fk(i)}^2 \\to \\spsi^*,\\\\\n    \\psi(\\sx^i) &\\geq \\psi(\\sx^{t_{\\fk(i)+1}})-(4\\Llambda\\sT^2+\\lambda/4)\\norm{\\Fnor(\\sz^{t_{\\fk(i)}})}^2 - 16\\Llambda \\sr_{\\fk(i)}^2 \\to \\spsi^*, \\quad \\text{a.s.},\n \\end{align*}\n%\nand hence $\\psi(\\sx^i) \\to \\spsi^*$. Finally, we verify $\\Exp[\\psi(\\sx^{i})] \\to \\Exp[\\spsi^*]$. Since $\\{\\mer(\\sz^{t_k}) + \\su_k\\}_k$ is bounded below by $\\bar\\psi$, non-increasing, and converges to $\\spsi^*$, we can obtain $\\Exp[\\mer(\\sz^{t_k}) + \\su_k]\\to \\Exp[\\spsi^*]$ by the monotone convergence theorem. Using similar arguments as before, this yields $\\Exp[\\psi(\\sx^{t_k})] \\to \\Exp[\\spsi^*]$. Let us now define $\\ssigma_i := \\psi(\\sx^{t_{\\fk(i)}})+(4\\Llambda\\sT^2+\\lambda/4)\\norm{\\Fnor(\\sz^{t_{\\fk(i)}})}^2 + 4\\Llambda \\sr_{\\fk(i)}^2$.\n%\nWe then have $\\psi(\\sx^i) \\leq \\ssigma_i$ (see \\cref{eq:tech2}), $\\ssigma_i\\to \\spsi^*$ a$.$s$.$, and $\\Exp[\\ssigma_i]\\to \\Exp[\\spsi^*]$. Applying Fatou's Lemma for $\\psi(\\sx^i)-\\bar{\\psi}$ and $\\ssigma_i-\\psi(\\sx^i)$, it follows \n%\n \\begin{align*}\n      \\Exp[\\spsi^*]-\\bar{\\psi} &= \\Exp[{{\\lim}_{i\\to\\infty}({\\psi(\\sx^i)-\\bar{\\psi}}})] \\leq {\\liminf}_{i\\to\\infty}{\\Exp{[\\psi(\\sx^i)]}}-\\bar{\\psi}\\quad \n \\end{align*}\n%\nand $0 = \\Exp[{{\\lim}_{i\\to\\infty} (\\ssigma_i-\\psi(\\sx^i))}] \\leq \\Exp[{\\spsi^*}] - {\\limsup}_{i\\to\\infty} {\\Exp{[\\psi(\\sx^i)]}}$. Consequently, we can infer $\\Exp{[\\psi(\\sx^i)]}\\to \\Exp[{\\spsi^*}]$. This finishes the proof. \n\\end{proof}","sha256_utf8":"2c7572f7762a6b9b1ede9757c46f4b3953cad3de2e43ca63969659cc3d815acd"},{"start":71369,"end":72366,"start_line":916,"end_line":928,"raw_tex":"\\begin{lemma}[Kurdyka-{\\L}ojasiewicz-type inequality]\n\\label{lem:KL-mer}\n    Let \\ref{A3} hold and let $\\bar z$ be given with $\\bar{x}=\\proxl(\\bar{z}) \\in \\crit(\\psi)$. For all $\\theta\\in[\\frac12,1)$, there then are $\\varsigma, \\rho > 0, \\eta\\in(0,1]$ and a continuous, concave function $\\varrho:[0,\\eta) \\to \\R_+$, which is $C^1$ on $(0,\\eta)$, satisfying \n      \\begin{enumerate}[label=\\textup{\\textrm{(\\alph*)}},leftmargin=3em,topsep=1pt,itemsep=.5ex,partopsep=0ex]\n         \\item $1/\\varrho^\\prime(s+t) \\leq 1/\\varrho^\\prime(s) + \\varsigma t^{\\theta}$ for all $s,t >0$ with $s+t<\\eta$,\n         \\item $\\varrho(0) = 0$ and $\\varrho^\\prime(t) \\geq t^{-\\theta}$ for all $t\\in(0,\\eta)$,\n     \\end{enumerate}\n     such that, for all $z\\in\\cB_\\rho(\\bar z) \\cap\\{z:0<|\\mer(z)-\\mer(\\bar z)| + \\frac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2<\\eta\\}$, we have\n     \\begin{equation}\n         \\label{eq:mer-kl}\n         \\varrho^\\prime(|\\mer(z)-\\mer(\\bar z)|) \\cdot  \\|\\Fnor(z)\\|\\geq 1.\n     \\end{equation} \n\\end{lemma}","sha256_utf8":"2fa99b5be46c65628ac911a9c40883d739dd65db3746d8e248df579b118feaf8"},{"start":72367,"end":76932,"start_line":929,"end_line":958,"raw_tex":"\\begin{proof}\nThe proof is divided into two steps. First, we show that $\\psi$ satisfies the KL property, with the desingularization function $\\varrho$ being concave, differentiable, and fulfilling properties (a) and (b). This step closely follows the proof of \\cite[Lemma 4.11]{josz2024proximal}. Although the differences are minor, we include a complete proof for self-containedness. In the second step, we show that these properties can be transferred to $\\mer$. %\\\\[1mm]\n\n\\noindent\\textbf{Step 1.}  \n%\nApplying \\cite[Corollary 15]{boldanlewshi07} and \\cite[Theorem 4.1]{AttBolRedSou10}, there are $\\rho,\\eta_1 >0$ and a definable, concave, continuous function $\\varrho_1:[0,\\eta_1)\\to\\R_+$, $\\varrho_1 \\in C^1((0,\\eta_1))$, such that \n%\n\\begin{equation}\n\\varrho_1^\\prime(|\\psi(x)-\\psi(\\bar x)|) \\cdot \\|\\partial \\psi(x)\\|_{-}\\geq 1, \\quad \\forall\\; x\\in\\cB_{\\rho}(\\bar x) \\cap\\{x:0<|\\psi(x)-\\psi(\\bar x)|<\\eta_1\\}. \\label{eq:kl-1}\n\\end{equation}\n%\nWe now set $\\varrho(t)=\\int_{0}^t \\varrho^\\prime(\\tau)\\,\\rmd \\tau$, $\\varrho^\\prime(t):=\\max\\{\\varrho_1^\\prime(t),t^{-\\theta}\\}$, and $\\bar \\eta:=\\min\\{\\eta_1,1\\}$. Then, $\\varrho(0)=0$ and \\cref{eq:kl-1} remains to hold with $\\varrho_1$ and $\\eta_1$ being replaced by $\\varrho$ and $\\bar \\eta$, respectively.\nThe function $\\varrho$ is continuous, $C^1$ on $(0,\\bar\\eta)$, and concave as $\\varrho^\\prime$ is non-increasing. \n%\nMoreover, $\\varrho^\\prime$ is definable because $t \\mapsto t^{-\\theta}$ and $\\varrho_1^\\prime$ are definable \\cite[Lemma 6.1]{coste2000introduction}, so is $t\\mapsto 1/\\varrho^\\prime(t)$ \\cite[Lemma 2.3 p.14]{van1998tame} (see also \\cite[Section 4.3]{AttBolRedSou10}). According to \\cite[Theorem 6.4]{coste2000introduction}, the mapping $t\\mapsto 1/\\varrho^\\prime(t)$ is differentiable for all but finitely many points in $(0,\\bar \\eta)$. This result, along with the monotonicity theorem \\cite[(1.2) p.43]{van1998tame}, indicates that $t\\mapsto 1/\\varrho^\\prime(t)$ is differentiable and monotone on the interval $(0,\\eta)$ for some $\\eta\\in(0,\\bar \\eta)$.  \n\nIf $(1/\\varrho^\\prime)^\\prime(t)\\to \\infty$ as $t\\searrow0$, then $(1/\\varrho^\\prime)^\\prime$ is decreasing on $(0,\\eta)$ and $1/\\varrho^\\prime$ is concave. By \\cite[Lemma 3.5]{josz2023convergence}, this yields $1/\\varrho^\\prime(s+t) \\leq 1/\\varrho^\\prime(s) + 1/\\varrho^\\prime(t) \\leq 1/\\varrho^\\prime(s) + t^\\theta$ for all $s,t>0$ and $s+t<\\eta$. Otherwise, we have $\\bar\\varsigma:=\\max_{\\tau\\in(0,\\eta)}(1/\\varrho^\\prime)^\\prime(\\tau) < \\infty$ and by the mean value theorem, it follows $1/\\varrho^\\prime(s+t) - 1/\\varrho^\\prime(s)= (1/\\varrho^\\prime)^\\prime(\\tau) t \\leq \\bar\\varsigma t \\leq \\bar\\varsigma t^{\\theta}$, $\\tau \\in (0,\\eta)$, where we used $\\theta < 1$ and $t \\in (0,1)$. Note that $\\bar\\varsigma$ is nonnegative since $(1/\\varrho^\\prime)^\\prime(\\tau) = -\\varrho^{\\prime\\prime}(\\tau)/(\\varrho^\\prime(\\tau))^2 \\geq 0$ (by the concavity of $\\varrho$). We may now set $\\varsigma := \\max\\{\\bar\\varsigma,1\\}$. %\\\\[1mm]\n\n\\noindent\\textbf{Step 2:} \\emph{Property \\cref{eq:mer-kl} for $\\mer$.} Let $z\\in\\cB_\\rho(\\bar z) \\cap\\{z:0<|\\mer(z)-\\mer(\\bar z)|+ \\frac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2<\\eta\\}$ be arbitrary. Setting $x=\\proxl(z)$ and using the nonexpansiveness of $\\proxl$, we have $\\|x-\\bar x\\| \\leq \\|z-\\bar z\\| \\leq \\rho$ and $|\\psi(x)-\\psi(\\bar x)| \\leq |\\mer(z)-\\mer(\\bar z)|+ \\frac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2<\\eta$.\n\nThus, if $x$ satisfies $\\psi(x) \\neq \\psi(\\bar x)$, then \\textbf{Step 1} implies $\\varrho^\\prime(|\\psi(x)-\\psi(\\bar x)|) \\|\\partial \\psi(x)\\|_{-}\\geq 1$. Furthermore, since $1/\\varrho^\\prime$ is non-decreasing and applying the bound (a), we can infer\n%\n\\begin{align*}\n          1/\\varrho^\\prime(|\\mer(z)-\\mer(\\bar z)|) \n          &\\leq  1/\\varrho^\\prime(|\\psi(x)-\\psi(\\bar x)| + \\tfrac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2)\\\\\n          &\\leq 1/\\varrho^\\prime(|\\psi(x)-\\psi(\\bar x)|) + \\varsigma (\\tfrac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2)^\\theta \\\\ &\\leq \\|\\partial \\psi(x)\\|_{-} + \\varsigma \\sqrt{\\xi\\lambda/2}\\|\\Fnor(z)\\| \\leq (1+\\varsigma \\sqrt{\\xi\\lambda/2})\\|\\Fnor(z)\\|.\n\\end{align*}\n%\nHere, the third inequality is due to $\\tfrac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2<1$ and $\\theta\\in[\\frac12,1)$, and the last step follows from $\\Fnor(z)\\in\\partial \\psi(x)$. If $\\psi(x) = \\psi(\\bar x)$, then we obtain $1/\\varrho^\\prime(|H(z)-H(\\bar z)|) \\leq (\\tfrac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2)^\\theta \\leq \\sqrt{\\xi\\lambda/2}\\|\\Fnor(z)\\|$ by (b). We can conclude the proof after rescaling $\\varrho$ to $(1+\\max\\{1,\\varsigma\\} \\sqrt{\\xi\\lambda/2}) \\varrho$ (this does not affect the properties (a)--(b)).\n\\end{proof}","sha256_utf8":"1bb2303942eb99d0248f36ef8c4c1708160155f93e5411ae842de379dcb7e8a6"},{"start":79828,"end":90769,"start_line":997,"end_line":1121,"raw_tex":"\\subsubsection{\\texorpdfstring{Proof of \\cref{thm:iter converge}}{Proof of Theorem 4.2}}\n\nWe start the proof by deriving certain summability results for the error terms $\\su_k$ and $\\sr_k$. %\\\\[1mm]\n\n\\noindent\\textbf{Step 1:} \\emph{Summability of error terms.} Due to condition \\cref{eq:kl-step} (i.e., $\\sum_{k=0}^\\infty \\alpha_k^2 \\tau_{0,k}^{\\mu}\\sigma_k^2  <\\infty$) and \\cref{lem:err_estimate}, we have $\\bar \\sr:=\\sum_{k=0}^\\infty \\tau_{0,t_k}^{\\mu}\\sr_k^2<\\infty$ a$.$s$.$. This yields \\vspace{-0.5ex}\n\\[\n\\sT\\su_k = {10}\\, {\\sum}_{i=k}^\\infty \\sr_{i}^2 =  {10}\\, {\\sum}_{i=k}^\\infty \\tau_{0,t_i}^{-\\mu} \\tau_{0,t_i}^{\\mu} \\sr_{i}^2 \\leq {10} \\tau_{0,t_k}^{-\\mu} \\cdot {\\sum}_{i=k}^\\infty  \\tau_{0,t_i}^{\\mu} \\sr_{i}^2 \\leq {10\\bar \\sr}  \\cdot \\tau_{0,t_k}^{-\\mu}.\n\\]\n%\nUsing $\\delta \\sT \\leq \\tau_{t_k,t_{k+1}}$, it follows $\\tau_{0,t_k} \\geq  \\tau_{0,t_0} + \\delta \\sT k$ and $\\su_k\\to 0$ a$.$s$.$, which implies \n%\n\\begin{equation}\n    \\label{eq:summability u_k}\n    {\\sum}_{k=0}^\\infty \\su_k^\\theta \\leq (10\\bar\\sr/\\sT)^\\theta  \\cdot {\\sum}_{k=0}^\\infty (\\tau_{0,t_0} + \\delta \\sT k)^{-\\mu\\theta}<\\infty \\quad \\text{for all}\\quad \\theta > \\tfrac{1}{\\mu}.\n\\end{equation}\n%\nHence, we have $\\sum_{k=j}^\\infty \\su_k^\\theta \\to 0$ as $j\\to\\infty$ for all $\\theta > {\\mu}^{-1}$. Moreover, it holds that \\vspace{-1ex}\n%\n\\begin{equation}\\label{eq:rk-sum}\n{\\sum}_{k=0}^\\infty \\sr_k = {\\sum}_{k=0}^\\infty \\tau_{0,t_k}^{-\\frac{\\mu}{2}}\\cdot \\tau_{0,t_k}^{\\frac{\\mu}{2}} \\sr_k \\leq \\Big({ \\bar{\\sr} \\cdot {\\sum}_{k=0}^\\infty (\\tau_{0,t_0} + \\delta \\sT  k)^{-\\mu} }\\Big)^{1/2} < \\infty \\quad \\text{a$.$s$.$.}\n\\end{equation}\n%\nLet us fix $\\theta\\in(\\max\\{\\frac{1}{\\mu},\\frac12\\},1)$ and let us introduce the events $\\mathcal E := \\{\\omega : \\sum_{k=0}^\\infty \\su_k(\\omega)^\\theta < \\infty, \\, \\sum_{k=0}^\\infty \\sr_k(\\omega) < \\infty\\}$ and $\\mathcal C := \\mathcal X \\cap \\{\\omega : \\Fnor(\\sz^k(\\omega)) \\to 0, \\, \\psi(\\sx^k(\\omega))\\to\\spsi^*(\\omega)\\}$. Condition \\cref{eq:kl-step} implies that \\ref{B3} is satisfied. Thus, by \\cref{thm:global_convergence-2}, \\cref{eq:summability u_k}, \\cref{eq:rk-sum}, and \\ref{B4}, we can infer $\\Prob(\\mathcal C) = \\Prob(\\mathcal E) = \\Prob(\\mathcal D)=1$, where $\\mathcal D = \\mathcal C \\cap \\mathcal E$. %\\\\[1mm]\n\n\\noindent\\textbf{Step 2:} \\emph{Subsequence convergence and roadmap.} We first present an auxiliary result.\n \n\\begin{lemma}[Subsequence convergence]\\label{lem:subsequence converge}\nLet \\ref{A1}--\\ref{A2} and \\ref{B1}--\\ref{B4} hold and let $\\omega\\in\\mathcal C$ be arbitrary. Then, there exist $z^*\\in\\Rn$ and a subsequence $\\revise{\\{q_k\\}_{k\\in\\N}}\\subseteq \\N$ such that $x^*:=\\proxl(z^*)\\in\\crit(\\psi)$, $x^{q_k} \\to x^*$, $z^{q_k} \\to z^*$, $\\psi(x^k) \\to \\psi(x^*) = \\psi^*$, and $\\mer(z^{k}) \\to \\mer(z^*) = \\psi^*$ as $k\\to\\infty$, where  \n$\\revise{\\{x^k\\}_{k\\in\\N}} = \\revise{\\{\\sx^k(\\omega)\\}_{k\\in\\N}}$ and $\\revise{\\{z^k\\}_{k\\in\\N}}= \\revise{\\{\\sz^k(\\omega)\\}_{k\\in\\N}}$. \n\\end{lemma}\n\nThe proof of \\cref{lem:subsequence converge} can be found in \\cref{proof:lem:subsequence converge}. To simplify the notation, we set\n%\n$\\sd_k := {\\max}_{t_k<i\\leq t_{k+1}} \\max\\{\\|\\sx^i-\\sx^{t_k}\\|,\\|\\sz^i-\\sz^{t_k}\\|\\}$.\n\n\\cref{lem:est-err} and $\\tau_{t_k,t_{k+1}} \\leq \\sT$ then imply \n%\n\\begin{equation}\\label{dk-bound}\n    % \\sd_k^2 \\leq 4 \\sT^2 \\|\\Fnor(\\sz^{t_k})\\|^2 + 4 \\sr_k^2.\n    \\sd_k \\leq 2 \\sT \\|\\Fnor(\\sz^{t_k})\\| + 2 \\sr_k.\n\\end{equation}\n%\nOur goal is to show $\\sx^k \\to \\sx^*$ a$.$s$.$. Thanks to $\\Prob(\\mathcal D)=1$, it suffices to prove that $\\revise{\\{\\sx^k(\\omega)\\}_{k\\in\\N}}$ is a Cauchy sequence for all $\\omega \\in \\mathcal D$. Let $\\omega \\in \\mathcal D$ be arbitrary and let $\\revise{\\{x^k\\}_{k\\in\\N}} = \\revise{\\{\\sx^k(\\omega)\\}_{k\\in\\N}}$,  $\\revise{\\{z^k\\}_{k\\in\\N}} = \\revise{\\{\\sz^k(\\omega)\\}_{k\\in\\N}}$, $\\revise{\\{r_k\\}_{k\\in\\N}} = \\revise{\\{\\sr_k(\\omega)\\}_{k\\in\\N}}$, etc. denote the corresponding realizations. For given $\\varepsilon>0$, we need to show that there is $k$ such that $\\|x^m - x^n\\|<\\varepsilon$ for all $k\\leq m<n$. \n\nWe claim that it suffices to establish ${\\sum}_{k=0}^{\\infty}\\,d_k< \\infty$. In this case, there is $k^\\prime\\in\\N$ such that ${\\sum}_{j=k}^{\\infty}\\,d_j<\\varepsilon/2$ for all $k \\geq k^\\prime$. For any $n > m > t_{k^\\prime}$, we have $k^\\prime \\leq \\fk(m)\\leq \\fk(n)$. Therefore, using the triangle inequality and the above bound, we obtain %\\vspace{-1.5ex}\n%\n\\begin{equation*}\n\\|x^m - x^n\\| \\leq d_{\\fk(m)} + d_{\\fk(n)} + {\\sum}_{j=\\fk(m)}^{{\\fk(n)-1}}\\|x^{t_{j+1}}-x^{t_j}\\| \\leq 2 {\\sum}_{j=\\fk(m)}^{{\\fk(n)}} d_j< \\varepsilon, \n\\end{equation*}\n%\nwhich proves our claim. In the remaining steps, we establish summability of $\\revise{\\{d_k\\}_{k\\in\\N}}$.%\\\\[1mm]\n\n\\noindent\\textbf{Step 3:} \\emph{Entering the local region and a KL-based bound for $\\revise{\\{d_k\\}_{k\\in\\N}}$.} By \\cref{lem:subsequence converge}, there exists a subsequence $\\revise{\\{q_k\\}_{k\\in\\N}}$ such that $x^{q_k} \\to x^*$, $z^{q_k} \\to z^*$ for some $z^*$ with $x^* = \\proxl(z^*) \\in\\crit(\\psi)$, and $\\mer(z^{k}) \\to \\mer(z^*) = \\psi^*$. Based on the definition of $\\fk$ and $d_k$, it holds that $\\max\\{\\|x^{q_k} - x^{t_{\\fk(q_k)}}\\|,\\|z^{q_k} - z^{t_{\\fk(q_k)}}\\|\\} \\leq d_{\\fk(q_k)}$. Due to $\\|\\Fnor(z^{t_{{\\fk(q_k)}}})\\|\\to0$, $r_{{\\fk(q_k)}}\\to0$, and \\cref{dk-bound}, we then have $d_{{\\fk(q_k)}} \\to 0$, $x^{t_{\\fk(q_k)}} \\to x^*$, and $z^{t_{\\fk(q_k)}} \\to z^*$.\n\n\nApplying \\cref{lem:KL-mer}, there exist $\\rho>0$, $\\eta\\in(0,1]$, and a concave function $\\varrho:[0,\\eta)\\to\\R_+$, $\\varrho \\in C^1((0,\\eta))$ (satisfying properties (a)--(b) in \\cref{lem:KL-mer}) such that\n%\n\\begin{equation}\n         \\label{eq:super-proof-kl}\n         \\varrho^\\prime(|\\mer(z)-\\psi^*|) \\cdot  \\|\\Fnor(z)\\|\\geq 1\\quad %\\text{holds}\n\\end{equation}\n%\nfor all $z\\in\\cB_\\rho(z^*) \\cap\\{z \\in \\Rn:0<|\\mer(z)-\\psi^*| + \\frac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2<\\eta\\}$.\n    \nNext, we define $\\Delta_k:= \\varrho(\\mer(z^{t_k}) -\\psi^* + u_k)$.\nSince \\revise{$\\{\\mer(z^{t_k}) -\\psi^* + u_k\\}_{k \\in \\N}$} is monotonically decreasing (see \\cref{eq:strong-0}) with $\\mer(z^{t_k}) + u_k \\to \\psi^*$, $\\Delta_k$ is well-defined. By $\\omega \\in \\mathcal D$, $\\varrho(0) = 0$, and by the previous discussion, there then is $\\hat k \\in \\N$ such that\n%\n\\begin{subequations}\n\\label{eq:choose rho eta}\n    \\begin{align}\n        \\label{eq:choose rho eta-1} &\\|z^{t_{\\hat k}} - z^*\\| + 20\\xi^{-1}\\Delta_{\\hat{k}}  + 2\\varsigma\\sT {\\sum}_{i=\\hat k}^\\infty\\, u_i^\\theta + 2{\\sum}_{i=\\hat k}^\\infty\\, r_i< \\rho \\quad \\text{and} \\quad \\\\ \\label{eq:choose rho eta-2} &|\\mer(z^{t_k})-\\psi^*| + \\max\\{\\tfrac{\\xi\\lambda}{2}\\|\\Fnor(z^{t_k})\\|^2, u_k\\} \\leq  \\eta\\quad\\text{for all} \\quad k \\geq \\hat k. \n    \\end{align}\n\\end{subequations}\n\nLet $k \\geq \\hat k$ be now given with $z^{t_k} \\in \\mathcal B_\\rho(z^*)$. (In particular, this is true for $k = \\hat k$). Hence, by \\cref{eq:choose rho eta-2}, the KL inequality \\cref{eq:super-proof-kl} is applicable and we have\n%\n\\begin{subequations}\n    \\begin{align}\n\t\\label{eq:long-equation-1}\\Delta_k - \\Delta_{k+1}\n    & \\geq \\varrho^\\prime(\\mer(z^{t_k}) - \\psi^*+u_k)(\\mer(z^{t_k}) +u_k - [\\mer(z^{t_{k+1}})+u_{k+1}]) \\\\\n\t\\label{eq:long-equation-2}& \\geq \\varrho^\\prime(|\\mer(z^{t_k}) - \\psi^*|+u_k) (\\xi \\sT \\|\\Fnor(z^{t_k})\\|^2/10) \\\\%[1mm]\n\t\\label{eq:long-equation-3}&\\geq (1/\\varrho^\\prime({|\\mer(z^{t_k}) - \\psi^*|})  + \\varsigma u_k^\\theta)^{-1} (\\xi \\sT \\|\\Fnor(z^{t_k})\\|^2/10) \\\\\n    \\label{eq:long-equation-4}&\\geq \\frac{\\xi\\sT}{10}\\cdot \\frac{\\|\\Fnor(z^{t_k})\\|^2}{\\|\\Fnor(z^{t_k})\\| +  \\varsigma u_k^\\theta}\\geq \\frac{\\xi \\sT}{10}\\srt{\\|\\Fnor(z^{t_k})\\| -  \\varsigma \n u_k^\\theta}.\n\\end{align}\n\\end{subequations}\n%\nThe inequality \\cref{eq:long-equation-1} is due to the concavity of $\\varrho$, \\cref{eq:long-equation-2} follows from the descent-type property \\cref{eq:strong-0} and the fact that $\\varrho^\\prime$ is non-increasing, \\cref{eq:long-equation-3} utilizes $\\varrho^\\prime(s+t) \\geq [1/\\varrho^\\prime(s) + \\varsigma t^{\\theta}]^{-1}$ (see \\cref{lem:KL-mer} (a)), and \\cref{eq:long-equation-4} is based on \\cref{eq:super-proof-kl} and ${a^2}\\geq a^2-b^2$, $a,b\\in \\R$. Rearranging yields $\\sT\\|\\Fnor(z^{t_k})\\| \\leq 10\\xi^{-1}(\\Delta_k - \\Delta_{k+1}) + \\varsigma \\sT u_k^\\theta$. Combining this estimate with $d_k \\leq 2 \\sT \\|\\Fnor(z^{t_k})\\| + 2 r_k$ (see \\cref{dk-bound}), we can infer\n%\n\\begin{equation}\n    \\label{eq:d bound}\nd_k \\leq 20\\xi^{-1}(\\Delta_k - \\Delta_{k+1}) + 2\\varsigma\\sT u_k^\\theta + 2r_k.\n\\end{equation}\n\n\\noindent\\textbf{Step 4:} \\emph{Attracted by the local region.}  We now show that the following statements hold for all $k \\geq \\hat k$: \n%\n\t\\begin{enumerate}[label=(\\alph*),topsep=4pt,itemsep=0.5ex,partopsep=0ex]\n\t\\item $z^{t_k} \\in \\mathcal{B}_{\\rho}(z^*)$ and $|\\mer(z^{t_k}) - \\psi^*| + \\max\\{\\frac{\\xi\\lambda}{2}\\|\\Fnor(z^{t_k})\\|^2,u_k\\}< \\eta$.\n %\\vspace{1mm}\n\t\\item $\\sum_{i=\\hat k}^{k}d_i  \\leq 20\\xi^{-1}[\\Delta_{\\hat k} - \\Delta_{k+1}] + 2\\varsigma\\sT \\sum_{i=\\hat k}^k u_i^\\theta + 2\\sum_{i=\\hat k}^k r_i.$\n\t\\end{enumerate}\n% \nWe will prove these statements by induction. First, (a) and (b) are satisfied for $k=\\hat k$ thanks to \\cref{eq:choose rho eta} and \\cref{eq:d bound}. To conduct the induction step, let us assume that (a)--(b) hold for some $k =n \\geq \\hat k$. We first verify (a) for $k=n+1$. Due to \\cref{eq:choose rho eta-2}, the bound $|\\mer(z^{t_{n+1}}) - \\psi^*| + \\max\\{\\frac{\\xi\\lambda}{2}\\|\\Fnor(z^{t_{n+1}})\\|^2,u_{n+1}\\} < \\eta$ is immediate. Moreover, we have\n%\n\\begin{align*}\n    \\|z^{t_{n+1}} - z^*\\| &\\leq \\|z^{t_{\\hat k}} - z^*\\| + {\\sum}_{i=\\hat{k}}^n \\|z^{t_{i+1}} - z^{t_{i}}\\| \\leq  \\|z^{t_{\\hat k}} - z^*\\| + {\\sum}_{i=\\hat k}^n \\,d_i\\\\\n    &\\leq \\|z^{t_{\\hat k}} - z^*\\| +  20\\xi^{-1}[\\Delta_{\\hat k} - \\Delta_{n+1}] + 2\\varsigma\\sT {\\sum}_{i=\\hat k}^\\infty\\, u_i^\\theta + 2{\\sum}_{i=\\hat k}^\\infty\\, r_i < \\rho,\n\\end{align*}\n%\nwhere the second inequality follows from the definition of $d_k$, the third step uses (b) for $k=n$, and the last step holds due to \\cref{eq:choose rho eta-1} and $\\Delta_{n+1}\\geq 0$. Next, we prove that (b) holds for $k=n+1$. Invoking \\cref{eq:d bound} with $k=n+1$ and (b) with $k=n$, we obtain\n%\n\\[\n{\\sum}_{i=\\hat k}^{n+1}\\,d_i = d_{n+1} + {\\sum}_{i=\\hat k}^{n}\\,d_i \\leq 20\\xi^{-1}[\\Delta_{\\hat k} - \\Delta_{n+2}] + 2\\varsigma\\sT {\\sum}_{i=\\hat k}^{n+1}\\, u_i^\\theta + 2{\\sum}_{i=\\hat k}^{n+1}\\, r_i.\n\\]\n%\nThis completes the induction. Thus, (a), (b), and \\cref{eq:d bound} hold for all $k \\geq \\hat k$. %\\\\[1mm]\n\n\\noindent\\textbf{Step 5:} \\emph{Summability of $\\revise{\\{d_k\\}_{k\\in\\N}}$.} Taking $k\\to\\infty$ in \\textbf{Step 4} (b) and due to $\\Delta_k \\geq 0$, we have \\vspace{-1.5ex}\n%\n\\begin{equation*} \n{\\sum}_{i=\\hat k}^{\\infty}d_i  \\leq 20\\xi^{-1}\\Delta_{\\hat k} + 2\\varsigma\\sT {\\sum}_{i=\\hat k}^{\\infty} u_i^\\theta + 2{\\sum}_{i=\\hat k}^{\\infty} r_i < \\rho < \\infty.\n\\end{equation*}\n%\nThis yields $\\sum_{k=0}^{\\infty}d_k < \\infty$ and, following \\textbf{Step 2}, this completes the proof. \\hfill \\proofbox\n\n","sha256_utf8":"09d3329a215fd3dcdeca55b9e6c2c1b3162950d66c1b86b6cde9171b7044e390"},{"start":143517,"end":146313,"start_line":1716,"end_line":1753,"raw_tex":"\\begin{proof} Let $\\omega \\in \\Omega$ be arbitrary and let us set $\\tau = \\tau_{m,n}$, $p = p^{m,n} = \\scp^{m,n}(\\omega)$, $s = s_{m,n} = \\scs_{m,n}(\\omega)$, $x^k = \\sx^k(\\omega)$, $z^k = \\sz^k(\\omega)$, etc. It follows from the definition of $p^{m,i}$ \\revise{and $s_{m,n}$} and the triangle inequality that\n%\n\\begin{align}\n        \\nonumber \\|p^{m,i}\\| &\\leq {\\sum}_{j=m}^{i-1}\\alpha_j\\|{\\Fnor(z^j) - \\Fnor(z^m)}\\| + \\Big\\|{{\\sum}_{j=m}^{i-1} \\alpha_j e^j}\\Big\\| \\\\ &\\leq {\\sum}_{j=m}^{i-1}\\alpha_j\\|{\\Fnor(z^j) - \\Fnor(z^m)}\\| + s,\\qquad \\text{for all $i\\in(m,n] \\cap \\N$.} \\label{eq:lem-4-est-1}\n\\end{align}\n% \nRecalling $\\Fnor(z) = \\nabla f(x) + \\lambda^{-1}(z-x)$ and using \\ref{A1} and the nonexpansiveness of the proximity operator (i.e., $\\|x^i-x^m\\| \\leq \\|z^i-z^m\\|$), we have\n%\n\\begin{equation}\n\\label{eq:nor is Lip}\n     \\|\\Fnor(z^j) - \\Fnor(z^m)\\| \\leq (\\sL+\\lambda^{-1})\\|x^j-x^m\\| + \\lambda^{-1}\\|z^j-z^m\\| \\leq  \\Llambda \\|z^j-z^m\\|,% =: C_\\lambda\\|z^i-z^m\\|.\n\\end{equation} \n%\nwhere $\\Llambda =\\sL+2\\lambda^{-1}$. Thus, combining the relation $z^i = z^m - \\tau_{m,i}\\Fnor(z^m) + p^{m,i}$ (cf$.$ \\cref{eq:pretty-important})\\revise{, $\\tau_{m,i} \\leq \\tau$,} and the inequality \\cref{eq:lem-4-est-1}, we can infer\n% \n\\begin{align}\n    \\nonumber \\|z^i-z^m\\| &\\leq \\tau_{m,i} \\|\\Fnor(z^m)\\| + \\|p^{m,i}\\| \\\\\n    \\nonumber &\\leq {\\sum}_{j=m}^{i-1} \\alpha_j \\|\\Fnor(z^j)-\\Fnor(z^m)\\| + \\tau \\|\\Fnor(z^m)\\| + s\\\\ \n    \\label{eq:lem-4-est-3} &\\leq \\Llambda {\\sum}_{j=m}^{i-1} \\alpha_j  \\|z^j-z^m\\| + \\tau \\|\\Fnor(z^m)\\| + s, \\qquad \\text{for all $i\\in(m,n] \\cap \\N$}.\n\\end{align}\n%\nWe now apply Gronwall's inequality (\\cref{Thm:G}) upon setting $b_1 :=\\tau \\|\\Fnor(z^m)\\| + s$, $b_2 := \\Llambda$, $a_k:=\\alpha_{m+k}$, $y_k:=\\|z^{m+k} - z^m\\|$, and $t:=i-m-1$. For all $i\\in(m,n] \\cap \\N$, this establishes the following upper bound  \n%\n\\begin{equation}\n            \\|z^i - z^m\\|  \\leq \\exp(\\tau \\Llambda) \\cdot [\\tau\\|\\Fnor(z^m)\\|+s]\\leq 2\\tau\\|\\Fnor(z^m)\\| + 2s,\n            \\label{eq:lem-4-est-5}\n\\end{equation}\n%\nwhere the last inequality is due to $\\tau\\Llambda \\leq \\frac12$ and $\\exp(\\frac12)\\leq 2$. Moreover, it follows from $\\|x^i-x^m\\| \\leq \\|z^i-z^m\\|$ that ${\\max}_{m<i\\leq n}\\|x^i-x^m\\| \\leq {\\max}_{m<i\\leq n}\\|z^i-z^m\\| \\leq 2\\tau\\|\\Fnor(z^m)\\| + 2s$. This verifies the first bound in \\cref{lem:est-err}. \n     \nNext, combining \\cref{eq:nor is Lip} and \\cref{eq:lem-4-est-5}, we obtain\n%\n\\[\n{\\max}_{m<i\\leq n} \\|\\Fnor(z^i) - \\Fnor(z^m)\\|\\leq \\Llambda{\\max}_{m<i\\leq n}\\|z^i-z^m\\| \\leq 2\\Llambda[\\tau\\|\\Fnor(z^m)\\| + s].\n\\]\n%\nUsing this bound in \\cref{eq:lem-4-est-1} with $i=n$ and recalling $\\tau={\\sum}_{i=m}^{n-1} \\alpha_i$, this yields $\\|p\\|\\leq s+ 2\\Llambda (\\tau\\|\\Fnor(z^m)\\| + s) {\\sum}_{j=m}^{n-1} \\alpha_j \\leq 2\\Llambda\\tau^2\\|\\Fnor(z^m)\\| + 2s$.\n\\end{proof}","sha256_utf8":"4ff76d49f208465cf80e007ab7e80f21b15c7edf9008ca6d87d9ff69e0050810"},{"start":146429,"end":148425,"start_line":1757,"end_line":1784,"raw_tex":"\\begin{proof}  Let us define the filtration $\\mathcal{U}_\\ell:=\\mathcal{F}_{t_k+\\ell}$ and introduce the sequence \\revise{$\\{\\sy^\\ell\\}_{\\ell\\in\\N}$} as follows $\\sy^0:=0$, $\\sy^\\ell := {\\sum}_{i=t_k}^{\\min\\{t_k+\\ell,t_{k+1}\\}-1}\\alpha_i \\se^i$ for all $\\ell \\geq 1$.\n%\nThen, $\\sy_\\ell$ is $\\mathcal U_\\ell$-measurable and for all $\\ell \\in \\{1,\\dots,t_{k+1}-t_k-1\\}$, we have \n%\n\\[ {\\Exp[{\\sy^{\\ell+1} \\mid \\mathcal U_\\ell}] = {\\sum}_{i=t_k}^{t_k+\\ell} \\alpha_i\\Exp[{\\se^i \\mid \\mathcal U_\\ell}] = \\sy^\\ell + \\alpha_{t_k+\\ell} \\Exp[{\\se^{t_k+\\ell} \\mid \\mathcal{F}_{t_k+\\ell}}] = \\sy^\\ell.} \\]\n% \n(Similarly for $\\ell \\geq t_{k+1}-t_k$). Hence, \\revise{$\\{\\sy^\\ell\\}_{\\ell\\in\\N}$} defines a \\revise{$\\{\\mathcal{U}_{\\ell}\\}_{\\ell\\in\\N}$}-martingale. Let us further set $\\sr_k = \\scs_{t_k,t_{k+1}}$. By the Burkholder-Davis-Gundy inequality (\\cref{Thm:BDG} with $p=2$ and $C_p\\leq 4$) and assumption \\ref{B1}, we have $\\Exp[{\\sr_k^2}] =\\Exp[{{\\sup}_{\\ell>0} \\|\\sy^\\ell\\|^2}]$ and\n%\n\\begin{align*} \n\\nonumber\\Exp[{\\sr_k^2}] \\leq 4 \\Exp\\Big[{ {\\sum}_{\\ell=1}^{\\infty}\\|\\sy^\\ell-\\sy^{\\ell-1}\\|^2}\\Big] & = 4 {\\sum}_{i=t_k}^{t_{k+1}-1}\\alpha_i^2 \\Exp[{ \\|\\se^i\\|^2}]  \\leq 4 {\\sum}_{i=t_k}^{t_{k+1}-1} \\alpha_i^2\\sigma_i^2. \n\\end{align*}\n%\nThis proves the first claim in \\Cref{lem:err_estimate}. Since $\\revise{\\{\\beta_k\\}_{k\\in\\N}}$ is non-decreasing, we obtain%\n%\n\\begin{equation}\\label{eq:app-bdg}\n\t\\beta_{t_k}^2 \\Exp[\\sr_k^2] \\leq 4 \\beta_{t_k}^2 {\\sum}_{i=t_k}^{t_{k+1}-1} \\alpha_i^2\\sigma_i^2 \\leq 4  {\\sum}_{i=t_k}^{t_{k+1}-1} \\alpha_i^2\\beta_{i}^2\\sigma_i^2\n\\end{equation}\n%\nfor all $k \\in \\N$. The monotone convergence theorem and \\cref{eq:app-bdg} further imply\n%\n\\begin{align*}\n \\Exp\\Big[{ {\\sum}_{k=0}^\\infty \\beta_{t_k}^2{\\sr_k^2}}\\Big] = {\\sum}_{k=0}^{\\infty}\\beta_{t_k}^2 \\Exp[\\sr_k^2] \n \\leq \n    {4} {\\sum}_{i=t_{0}}^\\infty \\alpha_i^2\\beta_i^2\\sigma_i^2 < \\infty,\n\\end{align*}\n%\nand consequently, it holds that $\\sum_{k=0}^\\infty \\beta_{t_k}^2{\\sr_k^2} < \\infty$ a$.$s$.$.\n\\end{proof}","sha256_utf8":"b7367b1fc1166c88890bfe83c3cdfe4d74e3d30a6726f180400490b2057e4eea"},{"start":148542,"end":151844,"start_line":1788,"end_line":1841,"raw_tex":"\\begin{proof}[Proof of (a)]\nLet $\\omega\\in\\Omega$ be arbitrary and denote $x^k = \\sx^k(\\omega)$, $z^k = \\sz^k(\\omega)$, $p = \\scp^{m,n}(\\omega)$, $u = \\su^{m,n}(\\omega)$, $\\tau = \\tau_{m,n}$, etc$.$. Recalling $\\Fnor(z) = \\nabla f(x) + \\lambda^{-1}(z-x)$ and by the definition \\cref{eq:u-def1} of $\\su^{m,n}$, we can express $\\Fnor(z^n)$ as\n%\n\\begin{align*} \n    \\Fnor(z^n) \n    & = \\Fnor(z^m) + [\\nabla f(x^n)-\\nabla f(x^m)] + \\lambda^{-1}\\prt{z^n-z^m+x^m-x^n}\\\\\n    & = u + [\\nabla f(x^n)-\\nabla f(x^m)] - \\lambda^{-1}(x^n-x^m).\n\\end{align*}\t\n%\nTaking squares and using the Lipschitz continuity of $\\nabla f$, this yields\n%\n\\begin{align}\n            \\nonumber \\norm{\\Fnor(z^n)}^2 & = \\norm{u}^2 + 2\\iprod{u}{(\\nabla f(x^n)-\\nabla f(x^m)) - \\lambda^{-1}(x^n-x^m)} \\\\\n            \\nonumber&\\hspace{4ex}  +  \\norm{(\\nabla f(x^n)-\\nabla f(x^m)) - \\lambda^{-1}(x^n-x^m)}^2\\\\\n            \\label{C1} & \\hspace{-6ex} \\leq \\norm{u}^2 + 2\\iprod{u}{(\\nabla f(x^n)-\\nabla f(x^m))-\\lambda^{-1}(x^n-x^m)} + (\\sL+{\\lambda}^{-1})^2 \\norm{x^n-x^m}^2.\n\\end{align}\n%\nUsing $u=(1-\\lambda^{-1}\\tau)\\Fnor(z^m) + \\lambda^{-1}p$ (cf$.$ \\cref{eq:u-def3}) and $\\norm{a+b}^2 \\leq (1+\\varepsilon) \\norm{a}^2 + (1+\\frac{1}{\\varepsilon})\\norm{b}^2$ with $a= (1-\\lambda^{-1}\\tau) \\Fnor(z^m)$, $b = \\lambda^{-1}p$, and $\\varepsilon = \\frac{\\lambda^{-1}\\tau}{1-\\lambda^{-1}\\tau}$, we have\n%\n\\begin{equation}\\label{C2}\n        \\norm{u}^2 \\leq (1-\\lambda^{-1}{\\tau}) \\|{\\Fnor(z^m)}\\|^2 + (\\lambda\\tau)^{-1} \\|{p}\\|^2.\n\\end{equation}\n%\nUtilizing Young's inequality for $2\\iprod{u}{\\nabla f(x^n)-\\nabla f(x^m)}$ and \\cref{C2}, we obtain \n%\n\\begin{align}\n            \\nonumber 2\\iprod{u}{\\nabla f(x^n)-\\nabla f(x^m)} & \\leq \\tfrac{\\tau}{2\\lambda} \\norm{u}^2 + \\tfrac{2\\lambda}{\\tau} \\norm{\\nabla f(x^n)-\\nabla f(x^m)}^2\\\\\n            \\label{C3} & \\leq  \\tfrac{\\tau}{2\\lambda}\\norm{\\Fnor(z^m)}^2 + \\tfrac{1}{2\\lambda^2} \\norm{p}^2 + \\tfrac{2\\lambda \\sL^2}{\\tau} \\norm{x^n-x^m}^2,\n\\end{align}\n%\nSumming up the estimates \\cref{C1}--\\cref{C3}, and noticing $\\tau \\leq \\lambda$, it follows\n%\n\\begin{align*}\n        \\norm{\\Fnor(z^n)}^2  & \\leq  ({1-\\tfrac{\\tau}{2\\lambda}})\\|{\\Fnor(z^m)}\\|^2 - \\tfrac{2}{\\lambda}\\iprod{u}{x^n-x^m} \\\\\n        & \\hspace{4ex} + [(\\sL+\\lambda^{-1})^2 + {2\\sL^2\\lambda}{\\tau^{-1}}] \\norm{x^n-x^m}^2 + \\tfrac{3}{2\\lambda\\tau} \\norm{p}^2.\n\\end{align*}\n\n\\emph{Proof of (b).} By \\ref{A1}--\\ref{A2}, $\\nabla f$ is $\\sL$-continuous and $\\vp$ is convex. Thus, we have\n%\n\\begin{equation*}\n        \\psi(x^n)-\\psi(x^m) \\leq \\iprod{\\nabla f(x^m)}{x^n-x^m}+\\tfrac{\\sL}{2}\\|x^n-x^m\\|^2 + \\tfrac{1}{\\lambda}\\iprod{z^n-x^n}{x^n-x^m},\n\\end{equation*}\n%\nwhere we used $\\lambda^{-1}(z^n-x^n) \\in\\partial \\vp(x^n)$ in the last inequality. Moreover, based on the definition of $\\Fnor$, we can reformulate %\\cref{eq:lem-1-est-1} \nthe above inequality as\n%\n\\begin{align*}\n\\psi(x^n)-\\psi(x^m) & \\leq \\iprod{\\nabla f(x^m)+\\lambda^{-1}(z^m-x^m)}{x^n-x^m}+\\tfrac{\\sL}{2}\\|{x^n-x^m}\\|^2\\\\\n& \\hspace{4ex} + \\lambda^{-1}\\iprod{z^n-x^n}{x^n-x^m} - \\lambda^{-1}\\iprod{z^m-x^m}{x^n-x^m}\\\\\n& \\hspace{-4ex} = [{\\tfrac{\\sL}{2}-\\tfrac{1}{\\lambda}}]\\|{x^n-x^m}\\|^2 + \t\\iprod{\\Fnor(z^m)}{x^n-x^m} + \\tfrac{1}{\\lambda}\\iprod{z^n-z^m}{x^n-x^m}.\n\\end{align*}\n%\nThis concludes the proof by substituting \\cref{eq:u-def1} into the last expression.\n\\end{proof}","sha256_utf8":"28b313bcfd772a1400df629169d71184c332684b44841c1b3cef9a6e413f6adf"},{"start":151969,"end":154068,"start_line":1845,"end_line":1874,"raw_tex":"\\begin{proof}\nAs shown in \\cref{eq:nor is Lip}, $\\Fnor$ is Lipschitz with modulus $\\Llambda$ and we have \n%\n\\begin{align*}\n        \\|\\Fnor(\\sz^i)\\| &\\leq \\|\\Fnor(\\sz^i) - \\Fnor(\\sz^{t_{\\fk(i)}})\\| + \\| \\Fnor(\\sz^{t_{\\fk(i)}})\\|\\\\ &\\leq  \\Llambda \\|\\sz^i-\\sz^{t_{\\fk(i)}}\\|+ \\| \\Fnor(\\sz^{t_{\\fk(i)}})\\| \\leq (2\\Llambda\\sT+1)\\| \\Fnor(\\sz^{t_{\\fk(i)}})\\| + 2\\Llambda \\scs_{t_{\\fk(i)},i}\\\\\n        &\\leq (2\\Llambda\\sT+1)\\| \\Fnor(\\sz^{t_{\\fk(i)}})\\| + 2\\Llambda \\sr_{\\fk(i)},\n\\end{align*}\n%\nwhere the \\revise{second to last} inequality follows from \\cref{lem:est-err} by setting $m=t_{\\fk(i)}$, $n=t_{\\fk(i)+1}$ (the condition $\\tau_{t_{\\fk(i)},t_{\\fk(i)+1}}\\leq \\sT \\leq 1/(2\\Llambda)$ is due to \\cref{eq:def delta and T}). Squaring both sides of the above inequality and using $(a+b)^2\\leq 2a^2+2b^2$, we obtain \\cref{eq:tech1}.\n\nBased on \\cref{lem:basic bounds} (b) and the definition of $\\su^{m,n}$ (see \\cref{eq:u-def1}), it holds that\n%\n\\[\n    \\psi(\\sx^n) \\leq \\psi(\\sx^m) + (\\sL/2-\\lambda^{-1})\\norm{\\sx^n-\\sx^m}^2 + \\iprod{\\lambda^{-1}(\\sz^n-\\sz^m) + \\Fnor(\\sz^m)}{\\sx^n-\\sx^m}.\n\\]\n%\nApplying the nonexpansiveness of the proximity operator, \\cref{prox-prop}, and $\\iprod{\\Fnor(\\sz^m)}{\\sx^n-\\sx^m} \\leq \\frac{\\lambda}{4} \\norm{\\Fnor(\\sz^m)}^2 + \\lambda^{-1} \\norm{\\sx^n-\\sx^m}^2$, we have\n%\n\\[\n    \\psi(\\sx^n)\\leq  \\psi(\\sx^m) + \\tfrac{\\Llambda}{2} \\norm{\\sz^n-\\sz^m}^2 + \\tfrac{\\lambda}{4} \\norm{\\Fnor(\\sz^m)}^2.\n\\]\n%\nInvoking \\Cref{lem:est-err} (for $\\norm{\\sz^n-\\sz^m}$) and $(a+b)^2\\leq 2a^2+2b^2$,  we obtain\n%\n\\[\n    \\psi(\\sx^n)\\leq  \\psi(\\sx^m) + (4\\Llambda\\tau_{m,n}^2+\\tfrac{\\lambda}{4}) \\norm{\\Fnor(\\sz^m)}^2 + 4\\Llambda \\scs_{m,n}^2.\n\\]\n%\nThen, \\cref{eq:tech2} readily follows by setting $n = i$, $m = t_{\\fk(i)}$ and using $\\tau_{t_{\\fk(i)},i}\\leq \\sT$. Similarly, \\cref{eq:tech3} can be obtained by setting $n = t_{\\fk(i)+1}$, $m = i$ and using  $\\tau_{i,t_{\\fk(i)+1}}\\leq \\sT$ and $\\scs_{i,t_{\\fk(i)+1}} = \\max_{i<j\\leq t_{\\fk(i)+1}}\\|{\\sum}_{\\ell=i}^{\\revise{j}-1} \\alpha_\\ell \\se^\\ell\\| \\leq \\scs_{t_{\\fk(i)},i}+  \\scs_{t_{\\fk(i)},t_{\\fk(i)+1}}  \\leq 2 \\sr_{\\fk(i)}$.\n\\end{proof}","sha256_utf8":"fb1b7761e946a0d2bd78179a791495390a6a1f48ea6be35396cec83053266b2a"},{"start":154201,"end":155480,"start_line":1877,"end_line":1883,"raw_tex":"\\begin{proof}\nLet $\\omega\\in\\mathcal C$ be arbitrary and let $x^k = \\sx^k(\\omega)$,  $z^k = \\sz^k(\\omega)$, etc$.$ be the corresponding realizations. By the definition of the event $\\mathcal C$, there exists $\\revise{\\{p_k\\}_{k\\in\\N}}\\subseteq\\N$ such that \\revise{$\\{x^{p_k}\\}_{k\\in\\N}$} is bounded and we have $\\Fnor(z^k) \\to 0$, $\\psi(x^k) \\to \\psi^*$. Using the definition of the normal map, we can infer $\\|z^{p_k}\\| \\leq \\lambda(\\|\\Fnor(z^{p_k})\\|+\\|\\nabla f(x^{p_k})\\|)+\\|x^{p_k}\\|$. Hence, \\revise{$\\{z^{p_k}\\}_{k\\in\\N}$} is also bounded and there is a convergent subsequence \\revise{$\\{z^{q_k}\\}_{k\\in\\N}$} of \\revise{$\\{z^{p_k}\\}_{k\\in\\N}$} such that $\\lim_{k\\to\\infty} z^{q_k} = z^*$ for some $z^*$ satisfying $\\|\\Fnor(z^{*})\\|=0$ and $x^*:=\\proxl(z^*) \\in \\crit(\\psi)$. Due to $\\|x^k-x^*\\| \\leq \\|z^k - z^*\\|$, we then have $\\lim_{k\\to\\infty} x^{q_k}=x^*$. Using the continuity of the Moreau envelope, this yields\n%\n\\begin{align*} \\psi^* & = {\\lim}_{k \\to \\infty}\\,\\psi(x^{q_k})  = {\\lim}_{k \\to \\infty}\\,\\vp(\\proxi{\\lambda}(z^{q_k})) + f(x^{q_k}) \\\\ & = {\\lim}_{k\\to\\infty}\\,\\envi{\\lambda}(z^{q_k}) - \\tfrac{1}{2\\lambda} \\|z^{q_k}- x^{q_k}\\|^2 + f(x^*) = \\psi(x^*). \\end{align*}\n%\nThis readily implies $H(z^k) \\to \\psi^* = H(z^*)$ and finishes the proof.\n\\end{proof}","sha256_utf8":"3ee0323b4adf7e8f478ca3f089c2b505b67326c32465e756d74ed0d7ecdd28c1"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"随机迭代几乎必然收敛","source_units":[{"start":78004,"end":78694,"start_line":977,"end_line":986,"raw_tex":"\\begin{theorem}\n\t\\label{thm:iter converge}\n\t Let \\ref{A1}--\\ref{A3}, \\ref{B1}--\\ref{B2}, and \\ref{B4} hold and let $\\revise{\\{\\sx^k\\}_{k\\in\\N}}$ be generated by $\\NSGD$ with step sizes $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}\\subset \\R_{++}$ satisfying: %\\vspace{-0.25ex}\n     %\n\t\\begin{equation}\\label{eq:kl-step}  \n\t\t\\lim_{k\\to\\infty}\\alpha_k=0,\\quad {\\sum}_{k=0}^\\infty \\alpha_k = \\infty \\quad \\text{and} \\quad {\\sum}_{k=0}^{\\infty} \\alpha_k^2 \\sigma_k^2 \\Big( {\\sum}_{i=0}^k\\alpha_i \\Big)^{\\mu}< \\infty,% \\quad \\text{for some $\\mu>\\frac12$}.\n        %\\vspace{-0.25ex}\n\t\\end{equation}\nfor some $\\mu>1$. Then, we have $\\sx^k \\to \\sx^*$ a$.$s$.$ for some $\\sx^*:\\Omega\\to\\crit(\\psi)$.\n\\end{theorem}","sha256_utf8":"c1c141f0fbfdcdd5b4d698baca8a3ec8c9cab29dbe4576ab644b6fc18bc16f81"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"迭代收敛的完整证明","source_units":[{"start":79828,"end":90769,"start_line":997,"end_line":1121,"raw_tex":"\\subsubsection{\\texorpdfstring{Proof of \\cref{thm:iter converge}}{Proof of Theorem 4.2}}\n\nWe start the proof by deriving certain summability results for the error terms $\\su_k$ and $\\sr_k$. %\\\\[1mm]\n\n\\noindent\\textbf{Step 1:} \\emph{Summability of error terms.} Due to condition \\cref{eq:kl-step} (i.e., $\\sum_{k=0}^\\infty \\alpha_k^2 \\tau_{0,k}^{\\mu}\\sigma_k^2  <\\infty$) and \\cref{lem:err_estimate}, we have $\\bar \\sr:=\\sum_{k=0}^\\infty \\tau_{0,t_k}^{\\mu}\\sr_k^2<\\infty$ a$.$s$.$. This yields \\vspace{-0.5ex}\n\\[\n\\sT\\su_k = {10}\\, {\\sum}_{i=k}^\\infty \\sr_{i}^2 =  {10}\\, {\\sum}_{i=k}^\\infty \\tau_{0,t_i}^{-\\mu} \\tau_{0,t_i}^{\\mu} \\sr_{i}^2 \\leq {10} \\tau_{0,t_k}^{-\\mu} \\cdot {\\sum}_{i=k}^\\infty  \\tau_{0,t_i}^{\\mu} \\sr_{i}^2 \\leq {10\\bar \\sr}  \\cdot \\tau_{0,t_k}^{-\\mu}.\n\\]\n%\nUsing $\\delta \\sT \\leq \\tau_{t_k,t_{k+1}}$, it follows $\\tau_{0,t_k} \\geq  \\tau_{0,t_0} + \\delta \\sT k$ and $\\su_k\\to 0$ a$.$s$.$, which implies \n%\n\\begin{equation}\n    \\label{eq:summability u_k}\n    {\\sum}_{k=0}^\\infty \\su_k^\\theta \\leq (10\\bar\\sr/\\sT)^\\theta  \\cdot {\\sum}_{k=0}^\\infty (\\tau_{0,t_0} + \\delta \\sT k)^{-\\mu\\theta}<\\infty \\quad \\text{for all}\\quad \\theta > \\tfrac{1}{\\mu}.\n\\end{equation}\n%\nHence, we have $\\sum_{k=j}^\\infty \\su_k^\\theta \\to 0$ as $j\\to\\infty$ for all $\\theta > {\\mu}^{-1}$. Moreover, it holds that \\vspace{-1ex}\n%\n\\begin{equation}\\label{eq:rk-sum}\n{\\sum}_{k=0}^\\infty \\sr_k = {\\sum}_{k=0}^\\infty \\tau_{0,t_k}^{-\\frac{\\mu}{2}}\\cdot \\tau_{0,t_k}^{\\frac{\\mu}{2}} \\sr_k \\leq \\Big({ \\bar{\\sr} \\cdot {\\sum}_{k=0}^\\infty (\\tau_{0,t_0} + \\delta \\sT  k)^{-\\mu} }\\Big)^{1/2} < \\infty \\quad \\text{a$.$s$.$.}\n\\end{equation}\n%\nLet us fix $\\theta\\in(\\max\\{\\frac{1}{\\mu},\\frac12\\},1)$ and let us introduce the events $\\mathcal E := \\{\\omega : \\sum_{k=0}^\\infty \\su_k(\\omega)^\\theta < \\infty, \\, \\sum_{k=0}^\\infty \\sr_k(\\omega) < \\infty\\}$ and $\\mathcal C := \\mathcal X \\cap \\{\\omega : \\Fnor(\\sz^k(\\omega)) \\to 0, \\, \\psi(\\sx^k(\\omega))\\to\\spsi^*(\\omega)\\}$. Condition \\cref{eq:kl-step} implies that \\ref{B3} is satisfied. Thus, by \\cref{thm:global_convergence-2}, \\cref{eq:summability u_k}, \\cref{eq:rk-sum}, and \\ref{B4}, we can infer $\\Prob(\\mathcal C) = \\Prob(\\mathcal E) = \\Prob(\\mathcal D)=1$, where $\\mathcal D = \\mathcal C \\cap \\mathcal E$. %\\\\[1mm]\n\n\\noindent\\textbf{Step 2:} \\emph{Subsequence convergence and roadmap.} We first present an auxiliary result.\n \n\\begin{lemma}[Subsequence convergence]\\label{lem:subsequence converge}\nLet \\ref{A1}--\\ref{A2} and \\ref{B1}--\\ref{B4} hold and let $\\omega\\in\\mathcal C$ be arbitrary. Then, there exist $z^*\\in\\Rn$ and a subsequence $\\revise{\\{q_k\\}_{k\\in\\N}}\\subseteq \\N$ such that $x^*:=\\proxl(z^*)\\in\\crit(\\psi)$, $x^{q_k} \\to x^*$, $z^{q_k} \\to z^*$, $\\psi(x^k) \\to \\psi(x^*) = \\psi^*$, and $\\mer(z^{k}) \\to \\mer(z^*) = \\psi^*$ as $k\\to\\infty$, where  \n$\\revise{\\{x^k\\}_{k\\in\\N}} = \\revise{\\{\\sx^k(\\omega)\\}_{k\\in\\N}}$ and $\\revise{\\{z^k\\}_{k\\in\\N}}= \\revise{\\{\\sz^k(\\omega)\\}_{k\\in\\N}}$. \n\\end{lemma}\n\nThe proof of \\cref{lem:subsequence converge} can be found in \\cref{proof:lem:subsequence converge}. To simplify the notation, we set\n%\n$\\sd_k := {\\max}_{t_k<i\\leq t_{k+1}} \\max\\{\\|\\sx^i-\\sx^{t_k}\\|,\\|\\sz^i-\\sz^{t_k}\\|\\}$.\n\n\\cref{lem:est-err} and $\\tau_{t_k,t_{k+1}} \\leq \\sT$ then imply \n%\n\\begin{equation}\\label{dk-bound}\n    % \\sd_k^2 \\leq 4 \\sT^2 \\|\\Fnor(\\sz^{t_k})\\|^2 + 4 \\sr_k^2.\n    \\sd_k \\leq 2 \\sT \\|\\Fnor(\\sz^{t_k})\\| + 2 \\sr_k.\n\\end{equation}\n%\nOur goal is to show $\\sx^k \\to \\sx^*$ a$.$s$.$. Thanks to $\\Prob(\\mathcal D)=1$, it suffices to prove that $\\revise{\\{\\sx^k(\\omega)\\}_{k\\in\\N}}$ is a Cauchy sequence for all $\\omega \\in \\mathcal D$. Let $\\omega \\in \\mathcal D$ be arbitrary and let $\\revise{\\{x^k\\}_{k\\in\\N}} = \\revise{\\{\\sx^k(\\omega)\\}_{k\\in\\N}}$,  $\\revise{\\{z^k\\}_{k\\in\\N}} = \\revise{\\{\\sz^k(\\omega)\\}_{k\\in\\N}}$, $\\revise{\\{r_k\\}_{k\\in\\N}} = \\revise{\\{\\sr_k(\\omega)\\}_{k\\in\\N}}$, etc. denote the corresponding realizations. For given $\\varepsilon>0$, we need to show that there is $k$ such that $\\|x^m - x^n\\|<\\varepsilon$ for all $k\\leq m<n$. \n\nWe claim that it suffices to establish ${\\sum}_{k=0}^{\\infty}\\,d_k< \\infty$. In this case, there is $k^\\prime\\in\\N$ such that ${\\sum}_{j=k}^{\\infty}\\,d_j<\\varepsilon/2$ for all $k \\geq k^\\prime$. For any $n > m > t_{k^\\prime}$, we have $k^\\prime \\leq \\fk(m)\\leq \\fk(n)$. Therefore, using the triangle inequality and the above bound, we obtain %\\vspace{-1.5ex}\n%\n\\begin{equation*}\n\\|x^m - x^n\\| \\leq d_{\\fk(m)} + d_{\\fk(n)} + {\\sum}_{j=\\fk(m)}^{{\\fk(n)-1}}\\|x^{t_{j+1}}-x^{t_j}\\| \\leq 2 {\\sum}_{j=\\fk(m)}^{{\\fk(n)}} d_j< \\varepsilon, \n\\end{equation*}\n%\nwhich proves our claim. In the remaining steps, we establish summability of $\\revise{\\{d_k\\}_{k\\in\\N}}$.%\\\\[1mm]\n\n\\noindent\\textbf{Step 3:} \\emph{Entering the local region and a KL-based bound for $\\revise{\\{d_k\\}_{k\\in\\N}}$.} By \\cref{lem:subsequence converge}, there exists a subsequence $\\revise{\\{q_k\\}_{k\\in\\N}}$ such that $x^{q_k} \\to x^*$, $z^{q_k} \\to z^*$ for some $z^*$ with $x^* = \\proxl(z^*) \\in\\crit(\\psi)$, and $\\mer(z^{k}) \\to \\mer(z^*) = \\psi^*$. Based on the definition of $\\fk$ and $d_k$, it holds that $\\max\\{\\|x^{q_k} - x^{t_{\\fk(q_k)}}\\|,\\|z^{q_k} - z^{t_{\\fk(q_k)}}\\|\\} \\leq d_{\\fk(q_k)}$. Due to $\\|\\Fnor(z^{t_{{\\fk(q_k)}}})\\|\\to0$, $r_{{\\fk(q_k)}}\\to0$, and \\cref{dk-bound}, we then have $d_{{\\fk(q_k)}} \\to 0$, $x^{t_{\\fk(q_k)}} \\to x^*$, and $z^{t_{\\fk(q_k)}} \\to z^*$.\n\n\nApplying \\cref{lem:KL-mer}, there exist $\\rho>0$, $\\eta\\in(0,1]$, and a concave function $\\varrho:[0,\\eta)\\to\\R_+$, $\\varrho \\in C^1((0,\\eta))$ (satisfying properties (a)--(b) in \\cref{lem:KL-mer}) such that\n%\n\\begin{equation}\n         \\label{eq:super-proof-kl}\n         \\varrho^\\prime(|\\mer(z)-\\psi^*|) \\cdot  \\|\\Fnor(z)\\|\\geq 1\\quad %\\text{holds}\n\\end{equation}\n%\nfor all $z\\in\\cB_\\rho(z^*) \\cap\\{z \\in \\Rn:0<|\\mer(z)-\\psi^*| + \\frac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2<\\eta\\}$.\n    \nNext, we define $\\Delta_k:= \\varrho(\\mer(z^{t_k}) -\\psi^* + u_k)$.\nSince \\revise{$\\{\\mer(z^{t_k}) -\\psi^* + u_k\\}_{k \\in \\N}$} is monotonically decreasing (see \\cref{eq:strong-0}) with $\\mer(z^{t_k}) + u_k \\to \\psi^*$, $\\Delta_k$ is well-defined. By $\\omega \\in \\mathcal D$, $\\varrho(0) = 0$, and by the previous discussion, there then is $\\hat k \\in \\N$ such that\n%\n\\begin{subequations}\n\\label{eq:choose rho eta}\n    \\begin{align}\n        \\label{eq:choose rho eta-1} &\\|z^{t_{\\hat k}} - z^*\\| + 20\\xi^{-1}\\Delta_{\\hat{k}}  + 2\\varsigma\\sT {\\sum}_{i=\\hat k}^\\infty\\, u_i^\\theta + 2{\\sum}_{i=\\hat k}^\\infty\\, r_i< \\rho \\quad \\text{and} \\quad \\\\ \\label{eq:choose rho eta-2} &|\\mer(z^{t_k})-\\psi^*| + \\max\\{\\tfrac{\\xi\\lambda}{2}\\|\\Fnor(z^{t_k})\\|^2, u_k\\} \\leq  \\eta\\quad\\text{for all} \\quad k \\geq \\hat k. \n    \\end{align}\n\\end{subequations}\n\nLet $k \\geq \\hat k$ be now given with $z^{t_k} \\in \\mathcal B_\\rho(z^*)$. (In particular, this is true for $k = \\hat k$). Hence, by \\cref{eq:choose rho eta-2}, the KL inequality \\cref{eq:super-proof-kl} is applicable and we have\n%\n\\begin{subequations}\n    \\begin{align}\n\t\\label{eq:long-equation-1}\\Delta_k - \\Delta_{k+1}\n    & \\geq \\varrho^\\prime(\\mer(z^{t_k}) - \\psi^*+u_k)(\\mer(z^{t_k}) +u_k - [\\mer(z^{t_{k+1}})+u_{k+1}]) \\\\\n\t\\label{eq:long-equation-2}& \\geq \\varrho^\\prime(|\\mer(z^{t_k}) - \\psi^*|+u_k) (\\xi \\sT \\|\\Fnor(z^{t_k})\\|^2/10) \\\\%[1mm]\n\t\\label{eq:long-equation-3}&\\geq (1/\\varrho^\\prime({|\\mer(z^{t_k}) - \\psi^*|})  + \\varsigma u_k^\\theta)^{-1} (\\xi \\sT \\|\\Fnor(z^{t_k})\\|^2/10) \\\\\n    \\label{eq:long-equation-4}&\\geq \\frac{\\xi\\sT}{10}\\cdot \\frac{\\|\\Fnor(z^{t_k})\\|^2}{\\|\\Fnor(z^{t_k})\\| +  \\varsigma u_k^\\theta}\\geq \\frac{\\xi \\sT}{10}\\srt{\\|\\Fnor(z^{t_k})\\| -  \\varsigma \n u_k^\\theta}.\n\\end{align}\n\\end{subequations}\n%\nThe inequality \\cref{eq:long-equation-1} is due to the concavity of $\\varrho$, \\cref{eq:long-equation-2} follows from the descent-type property \\cref{eq:strong-0} and the fact that $\\varrho^\\prime$ is non-increasing, \\cref{eq:long-equation-3} utilizes $\\varrho^\\prime(s+t) \\geq [1/\\varrho^\\prime(s) + \\varsigma t^{\\theta}]^{-1}$ (see \\cref{lem:KL-mer} (a)), and \\cref{eq:long-equation-4} is based on \\cref{eq:super-proof-kl} and ${a^2}\\geq a^2-b^2$, $a,b\\in \\R$. Rearranging yields $\\sT\\|\\Fnor(z^{t_k})\\| \\leq 10\\xi^{-1}(\\Delta_k - \\Delta_{k+1}) + \\varsigma \\sT u_k^\\theta$. Combining this estimate with $d_k \\leq 2 \\sT \\|\\Fnor(z^{t_k})\\| + 2 r_k$ (see \\cref{dk-bound}), we can infer\n%\n\\begin{equation}\n    \\label{eq:d bound}\nd_k \\leq 20\\xi^{-1}(\\Delta_k - \\Delta_{k+1}) + 2\\varsigma\\sT u_k^\\theta + 2r_k.\n\\end{equation}\n\n\\noindent\\textbf{Step 4:} \\emph{Attracted by the local region.}  We now show that the following statements hold for all $k \\geq \\hat k$: \n%\n\t\\begin{enumerate}[label=(\\alph*),topsep=4pt,itemsep=0.5ex,partopsep=0ex]\n\t\\item $z^{t_k} \\in \\mathcal{B}_{\\rho}(z^*)$ and $|\\mer(z^{t_k}) - \\psi^*| + \\max\\{\\frac{\\xi\\lambda}{2}\\|\\Fnor(z^{t_k})\\|^2,u_k\\}< \\eta$.\n %\\vspace{1mm}\n\t\\item $\\sum_{i=\\hat k}^{k}d_i  \\leq 20\\xi^{-1}[\\Delta_{\\hat k} - \\Delta_{k+1}] + 2\\varsigma\\sT \\sum_{i=\\hat k}^k u_i^\\theta + 2\\sum_{i=\\hat k}^k r_i.$\n\t\\end{enumerate}\n% \nWe will prove these statements by induction. First, (a) and (b) are satisfied for $k=\\hat k$ thanks to \\cref{eq:choose rho eta} and \\cref{eq:d bound}. To conduct the induction step, let us assume that (a)--(b) hold for some $k =n \\geq \\hat k$. We first verify (a) for $k=n+1$. Due to \\cref{eq:choose rho eta-2}, the bound $|\\mer(z^{t_{n+1}}) - \\psi^*| + \\max\\{\\frac{\\xi\\lambda}{2}\\|\\Fnor(z^{t_{n+1}})\\|^2,u_{n+1}\\} < \\eta$ is immediate. Moreover, we have\n%\n\\begin{align*}\n    \\|z^{t_{n+1}} - z^*\\| &\\leq \\|z^{t_{\\hat k}} - z^*\\| + {\\sum}_{i=\\hat{k}}^n \\|z^{t_{i+1}} - z^{t_{i}}\\| \\leq  \\|z^{t_{\\hat k}} - z^*\\| + {\\sum}_{i=\\hat k}^n \\,d_i\\\\\n    &\\leq \\|z^{t_{\\hat k}} - z^*\\| +  20\\xi^{-1}[\\Delta_{\\hat k} - \\Delta_{n+1}] + 2\\varsigma\\sT {\\sum}_{i=\\hat k}^\\infty\\, u_i^\\theta + 2{\\sum}_{i=\\hat k}^\\infty\\, r_i < \\rho,\n\\end{align*}\n%\nwhere the second inequality follows from the definition of $d_k$, the third step uses (b) for $k=n$, and the last step holds due to \\cref{eq:choose rho eta-1} and $\\Delta_{n+1}\\geq 0$. Next, we prove that (b) holds for $k=n+1$. Invoking \\cref{eq:d bound} with $k=n+1$ and (b) with $k=n$, we obtain\n%\n\\[\n{\\sum}_{i=\\hat k}^{n+1}\\,d_i = d_{n+1} + {\\sum}_{i=\\hat k}^{n}\\,d_i \\leq 20\\xi^{-1}[\\Delta_{\\hat k} - \\Delta_{n+2}] + 2\\varsigma\\sT {\\sum}_{i=\\hat k}^{n+1}\\, u_i^\\theta + 2{\\sum}_{i=\\hat k}^{n+1}\\, r_i.\n\\]\n%\nThis completes the induction. Thus, (a), (b), and \\cref{eq:d bound} hold for all $k \\geq \\hat k$. %\\\\[1mm]\n\n\\noindent\\textbf{Step 5:} \\emph{Summability of $\\revise{\\{d_k\\}_{k\\in\\N}}$.} Taking $k\\to\\infty$ in \\textbf{Step 4} (b) and due to $\\Delta_k \\geq 0$, we have \\vspace{-1.5ex}\n%\n\\begin{equation*} \n{\\sum}_{i=\\hat k}^{\\infty}d_i  \\leq 20\\xi^{-1}\\Delta_{\\hat k} + 2\\varsigma\\sT {\\sum}_{i=\\hat k}^{\\infty} u_i^\\theta + 2{\\sum}_{i=\\hat k}^{\\infty} r_i < \\rho < \\infty.\n\\end{equation*}\n%\nThis yields $\\sum_{k=0}^{\\infty}d_k < \\infty$ and, following \\textbf{Step 2}, this completes the proof. \\hfill \\proofbox\n\n","sha256_utf8":"09d3329a215fd3dcdeca55b9e6c2c1b3162950d66c1b86b6cde9171b7044e390"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"非凸复合优化问题","source_units":[{"start":12150,"end":12604,"start_line":354,"end_line":360,"raw_tex":"In this work, we propose and investigate a novel normal map-based proximal stochastic gradient method for the composite problem\n%\n\\begin{equation}\n\t\\label{eq:SO} \\min_{x\\in \\Rn}~\\psi(x) := f(x) + \\vp(x),\n\\end{equation}\n%\nwhere $\\vp: \\Rn \\to \\Rex$ is a convex, lower semicontinuous (lsc.), and proper mapping and $f: \\Rn \\to \\R$ is continuously differentiable on an open set containing the effective domain $\\dom{\\vp} := \\{x \\in \\Rn: \\vp(x) < \\infty\\}$. \n","sha256_utf8":"198c6cf2bbfd40a2d23b1322c713c1b0a743b19eba4600b945ebb415aabbb99a"}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","title":"光滑性与目标下界","source_units":[{"start":14664,"end":15095,"start_line":380,"end_line":383,"raw_tex":"    \\begin{enumerate}[label=\\textup{\\textrm{(H.\\arabic*)}},leftmargin=3em,topsep=0ex,itemsep=1ex,partopsep=0ex]\n        \\item \\label{A1} The gradient $\\nabla f : \\Rn \\to \\Rn$ is Lipschitz continuous on $\\dom{\\vp}$ with modulus $\\sL>0$ and $\\vp$ is convex, lsc., and proper.\n        \\item \\label{A2} The objective function $\\psi$ is bounded from below, i.e., $\\bar \\psi := \\inf_{x \\in \\Rn}\\,\\psi(x) > -\\infty$. \n    \\end{enumerate}\n","sha256_utf8":"9d80ee432f4dcace3b2588536ece1e9505549eed90153ede8c21d4cdd3d121bf"}]},{"id":"S005","semantic_type":"assumption","kind":"assumption","title":"随机误差与基本步长条件","source_units":[{"start":15115,"end":16639,"start_line":386,"end_line":394,"raw_tex":"In addition, we assume that there is a filtered probability space $(\\Omega,\\mathcal F,\\revise{\\{\\mathcal F_k\\}_{k\\in\\N}},\\Prob)$ that can describe the stochastic components of the update \\cref{eq:update-PSGD} and of other stochastic algorithms studied in this work in a unified way. In particular, each approximation $g^k$ is understood as a realization of an $\\mathcal F_{k+1}$-measurable random vector $\\sg^k : \\Omega \\to \\Rn$. Thus, the stochastic process $\\revise{\\{\\sx^k\\}_{k\\in\\N}}$, associated with \\cref{eq:update-PSGD} (or other stochastic methods), is adapted to the filtration $\\revise{\\{\\mathcal F_k\\}_{k\\in\\N}}$. We further define the stochastic error terms $\\se^k : \\Omega \\to \\Rn$ via $\\se^k := \\sg^k - \\nabla f(\\sx^k)$ and set $\\Exp_k[\\cdot] := \\Exp[\\cdot | \\cF_k]$. We mainly work with the following conditions on the errors $\\revise{\\{\\se^k\\}_{k\\in\\N}}$ and step sizes $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}$:\n%\n\\vspace{1.5mm}\n%\n\\begin{enumerate}[label=\\textup{\\textrm{(S.\\arabic*)}},leftmargin=3em,topsep=0ex,itemsep=1ex,partopsep=0ex]\n        \\item \\label{B1} The stochastic error $\\se^k$ has mean zero, i.e., $\\Exp_k[\\se^k]=0$ almost surely (a.s.).\n        \\item \\label{B2} There exists $\\revise{\\{\\sigma_k\\}_{k\\in\\N}}\\subseteq \\R_+$ such that $\\Exp_k[\\|\\se^k\\|^2]\\leq \\sigma_k^2$ a.s. for all $k$. \n        \\item \\label{B3} The step sizes $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}$ satisfy $\\alpha_k\\to0$, $\\sum_{k=0}^\\infty \\alpha_k = \\infty$, and $\\sum_{k=0}^{\\infty}\\alpha_k^2\\sigma_k^2<\\infty$.\n    \\end{enumerate}\n","sha256_utf8":"82e8f55db135ad101755fe6fd7952d2268261eceeeaed40d992fef88fd29ec6c"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"邻近算子","source_units":[{"start":14223,"end":14559,"start_line":376,"end_line":376,"raw_tex":"where the sequence $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}\\subseteq \\R_{++}$ denotes the selected step sizes, $g^k$ is a (stochastic) approximation of $\\nabla f(x^k)$, and the well-known proximity operator of $\\vp$ is defined as $\\prox{\\alpha \\vp}(x) := {\\argmin}_{y\\in\\Rn} \\, \\vp(y) + \\frac{1}{2\\alpha}\\|x-y\\|^2$, $\\alpha >0$. \\vspace{1mm}   \n","sha256_utf8":"d34619e4fd665e10b21a2a8493baad24f500449e36276710f9391b8af93bbf83"}]},{"id":"S007","semantic_type":"algorithm","kind":"algorithm","title":"法映射随机梯度迭代","source_units":[{"start":22953,"end":23084,"start_line":427,"end_line":427,"raw_tex":"Let $\\lambda>0$, $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}\\subseteq \\R_{++}$ be given. Choose $z^0\\in\\Rn$ and set $x^0=\\proxl(z^0)$. \\\\[1mm]\n","sha256_utf8":"ef66c340c1f8227414503a689bc82e2c1755bc5f1e7b87782dddea0c6d4681ee"},{"start":29442,"end":29594,"start_line":493,"end_line":495,"raw_tex":"\\be\\label{eq:update-short}\n    \\sz^{k+1} = \\sz^k - \\alpha_k(\\sg^k + \\lambda^{-1}(\\sz^k-\\sx^k)) \\quad \\text{and} \\quad \\sx^{k+1} = \\proxl(\\sz^{k+1}),\n\\ee","sha256_utf8":"547f15e2948261f8d9e524f40dbe14dfc143839962c7f9e87a645eb4104a69b1"},{"start":29597,"end":29674,"start_line":497,"end_line":497,"raw_tex":"where $\\sg^k$ is a (unbiased) stochastic approximation of $\\nabla f(\\sx^k)$.\n","sha256_utf8":"4d34e269514fa2266a4a2872e9076a6069030d14d6e8dcf970f015bb1ac9771a"}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","title":"法映射及其次梯度归属","source_units":[{"start":35759,"end":35924,"start_line":538,"end_line":538,"raw_tex":"\\begin{equation} \\label{eq:norm-subdiff} \\Fnor(z) = \\nabla f(\\proxl(z)) + \\lambda^{-1}(z-\\proxl(z)) \\in \\partial\\psi(\\proxl(z)) \\quad \\forall~z\\in\\Rn. \\end{equation}","sha256_utf8":"c1c0f6bd9e221ef3304d5f3ae3c4a24a04b75ce25669fb2e1b63f0c7a16cdbda"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"驻点与驻性度量","source_units":[{"start":33916,"end":34837,"start_line":525,"end_line":525,"raw_tex":"We now briefly discuss common stationarity measures for proximal gradient-type methods and for the problem \\cref{eq:SO}. The first-order optimality conditions associated with \\cref{eq:SO} are given by $0 \\in \\partial \\psi(x) = \\nabla f(x) + \\partial \\vp(x)$. Here, $\\partial \\psi$ denotes the limiting subdifferential of $\\psi$ and $\\partial \\vp$ is the standard subdifferential for convex functions, cf. \\cite{rocwet98,baucom11}. A point $\\bar x \\in \\dom{\\vp}$ is called a stationary point of \\cref{eq:SO} if $0 \\in \\partial \\psi(\\bar x)$ and we use $\\crit(\\psi)$ to denote the set of all stationary points of $\\psi$. We further introduce the notation $\\|\\partial \\psi(x)\\|_{-} := \\dist(0,\\partial\\psi(x)) := {\\min}_{v\\in\\partial \\psi(x)} \\;\\|v\\|$. A point $\\bar x \\in \\dom{\\vp}$ is an $\\varepsilon$-stationary point of $\\psi$, $\\varepsilon \\geq 0$, if we have $\\|\\partial\\psi(\\bar x)\\|_- \\leq \\varepsilon$. %\\\\[-2.5ex]\n","sha256_utf8":"e2ead5bfb786a9396cb45e2583cf40c393eeafd026c10106e1c49e7785626502"},{"start":35018,"end":35140,"start_line":529,"end_line":529,"raw_tex":"\\[ x \\in \\crit(\\psi) \\quad \\iff \\quad \\Fnat(x) := \\lambda^{-1}(x-\\proxl(x-\\lambda \\nabla f(x))) = 0, \\quad \\lambda > 0. \\]","sha256_utf8":"ee30b40d5bda9340e4f2830bcf4363b8475de307780d6a3a668c27ee0825a39f"},{"start":36337,"end":36509,"start_line":542,"end_line":542,"raw_tex":"\\begin{equation} \\label{eq:stat-am} \\|\\Fnat(x)\\| \\leq \\|\\partial\\psi(x)\\|_- \\quad \\text{and} \\quad \\|\\partial\\psi(\\prox{\\lambda\\vp}(z))\\|_- \\leq \\|\\Fnor(z)\\| \\end{equation}","sha256_utf8":"cf886f3ff670c02f2357faddfa27dcfdabad9fbac3c0d35f7efdbcaf476d373c"}]},{"id":"S010","semantic_type":"derived_fact","kind":"derived_fact","title":"邻近算子的坚定非扩张性","source_units":[{"start":42380,"end":42871,"start_line":572,"end_line":576,"raw_tex":"We first present several basic estimates and properties that facilitate the analysis of $\\NSGD$. Since $\\vp$ is assumed to be convex, lsc., and proper (cf$.$ \\ref{A1}), the proximity operator $\\proxl$ is firmly nonexpansive, \\cite{mor65,baucom11}. Hence, recalling $\\sx^k = \\proxl(\\sz^k)$, it holds that\n%\n\\begin{equation}\\label{prox-prop}\n    \\|\\sx^n-\\sx^m\\|^2 \\leq \\iprod{\\sz^n-\\sz^m}{\\sx^n-\\sx^m}\\leq \\norm{\\sz^n-\\sz^m}^2 \\quad \\forall~\\revise{m<n} \\quad \\text{(a$.$s$.$)}.\n\\end{equation}","sha256_utf8":"91d06bdc512dd60a9a816819342ebb8395d720d43b9f388a02a117e56aea2b07"}]},{"id":"S011","semantic_type":"notation","kind":"notation","title":"累计步长与聚合误差","source_units":[{"start":42969,"end":43517,"start_line":580,"end_line":584,"raw_tex":"\\begin{align}\n    \\nonumber \\sz^n&=\\sz^{n-1} -\\alpha_{n-1}[\\sg^{n-1}+ \\tfrac{1}{\\lambda}(\\sz^{n-1} - \\sx^{n-1})] =\\sz^{n-1} - \\alpha_{n-1}[\\Fnor(\\sz^{n-1}) +\\se^{n-1}]\t\\\\\n     \\nonumber &=\\sz^m - {\\sum}_{i=m}^{n-1}\\alpha_i\\Fnor(\\sz^i) - {\\sum}_{i=m}^{n-1} \\alpha_i\\se^i \\\\\n    \\nonumber &=\\sz^m - {{\\sum}_{i=m}^{n-1}\\alpha_i}\\Fnor(\\sz^m) - {\\sum}_{i=m}^{n-1} \\alpha_i[\\Fnor(\\sz^i)-\\Fnor(\\sz^m)] - {\\sum}_{i=m}^{n-1} \\alpha_i\\se^i \\\\ & = \\sz^m - \\tau_{m,n} \\Fnor(\\sz^m) + \\scp^{m,n} \\revise{\\quad \\forall~m<n,}\\label{eq:pretty-important}\n\\end{align}","sha256_utf8":"1e515b720b2f8ea0531a04db0220f502c08fceb28f210be061c277e53b05c816"},{"start":43520,"end":43678,"start_line":586,"end_line":586,"raw_tex":"where $\\tau_{m,n}:= {\\sum}_{i=m}^{n-1}\\alpha_i$ and $\\scp^{m,n}:=-{\\sum}_{i=m}^{n-1} \\alpha_i[\\Fnor(\\sz^i)-\\Fnor(\\sz^m)] - {\\sum}_{i=m}^{n-1} \\alpha_i\\se^i$.\n","sha256_utf8":"16ede40c04a9b858ce2c3e399a6e27d74007e218ba56826b8df35a69415a23e5"},{"start":43805,"end":43886,"start_line":590,"end_line":592,"raw_tex":"\\[\n\\scs_{m,n} :=\\max_{m<j\\leq n}\\Big\\|{\\sum}_{i=m}^{j-1} \\alpha_i \\se^i\\Big\\|.\n\\]","sha256_utf8":"28dc33e24776febb80c43b347557e2c2e52565cf56317159da52651480fc4390"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"辅助向量的等价表达","source_units":[{"start":46035,"end":46373,"start_line":620,"end_line":623,"raw_tex":"\\begin{align}\n         \\su^{m,n}  & :=  \\lambda^{-1}(\\sz^n-\\sz^m) + \\Fnor(\\sz^m) \\label{eq:u-def1}\\\\  &\\;=  (\\lambda^{-1}-\\tau_{m,n}^{-1})(\\sz^n-\\sz^m) + \\tau_{m,n}^{-1} \\scp^{m,n} \\label{eq:u-def2}\\\\\n          &\\;=  (1-\\lambda^{-1}\\tau_{m,n}) \\Fnor(\\sz^m) + \\lambda^{-1}\\scp^{m,n}, \\quad \\revise{m < n}.\\label{eq:u-def3}\n     \\end{align}","sha256_utf8":"22c6fb6f8f2eeb5cb51e25a180748abf93e0247e842fa89beb9ba1abe075eff8"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"法映射评价函数","source_units":[{"start":48902,"end":49161,"start_line":650,"end_line":652,"raw_tex":"\\begin{definition}[Merit function]\\label{def:mer-fun} Given $\\lambda > 0$, we define the merit function: \n \\[\\mer(z):=\\psi(\\prox{\\lambda\\vp}(z)) + \\frac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2,\\quad \\text{where} \\quad \\xi:={(2+2\\lambda^2 \\sL^2)^{-1}}.\\]\n\\end{definition}","sha256_utf8":"b372819dba69078abbcb01cd35b0b153be721a37ebfff734443da03c68c9bf44"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"时间窗口与时间指标","source_units":[{"start":52842,"end":53411,"start_line":701,"end_line":706,"raw_tex":"\\begin{definition}[Time indices]\\label{def:time win}\nLet $\\sT>0$ and $\\delta\\in(0,1)$ be given. We introduce the mapping $\\varpi:\\N \\to \\N\\cup \\{\\pm\\infty\\}$, $\\varpi(k):=\\sup\\{n > k: \\tau_{k,n} \\leq \\sT\\}$, where $\\tau_{m,n} = {\\sum}_{k=m}^{n-1} \\alpha_k$. The \\emph{time indices} $\\revise{\\{t_k\\}_{k\\in\\N}}$, associated with the \\emph{time window} $\\sT$ \\revise{(and the step sizes $\\{\\alpha_k\\}_{k \\in \\N}$)}, are defined as \n\\[\nt_0 := \\inf\\{i: \\alpha_j \\leq (1-\\delta)\\sT \\text{ for all $j\\geq i$}\\}\\quad \\text{and} \\quad t_{k+1} := \\varpi(t_k).\n\\]\n\\end{definition}","sha256_utf8":"fe7864be7f073298722b7decc2473092405f013aaa9a48fc9402eb83e597b387"}]},{"id":"S015","semantic_type":"notation","kind":"notation","title":"普通指标的窗口归属","source_units":[{"start":53768,"end":53873,"start_line":710,"end_line":712,"raw_tex":"\\begin{equation}\\label{index-def}\n\\fk(i) = \\max\\{k: t_k < i\\}\\quad \\text{for all $i>t_0$}.\n\\end{equation}","sha256_utf8":"34e07636aa872d5ade2f8b7226be0daddc562d58d718d0290b631f2c2cc9dcfe"},{"start":53876,"end":53952,"start_line":714,"end_line":714,"raw_tex":"It is evident that we have $t_{\\fk(i)}<i\\leq t_{\\fk(i)+1}$ for all $i>t_0$.\n","sha256_utf8":"b723ada71d7f7b48a5f9bff53d368cd0e03ac97c5bffc32e094761bbce8e5390"}]},{"id":"S016","semantic_type":"notation","kind":"notation","title":"固定窗口与误差常数","source_units":[{"start":55823,"end":56084,"start_line":734,"end_line":737,"raw_tex":"\\begin{equation}\n    \\label{eq:def delta and T}\n    \\sr_k := \\scs_{t_{k},t_{k+1}}, \\;\\; \\delta := \\frac45, \\;\\;  \\sT := \\frac{\\sqrt{\\xi}}{8\\Llambda},\\quad \\text{where} \\quad \\Llambda =\\sL+\\frac{2}{\\lambda}, \\;\\; \\xi =\\frac{1}{2+2\\lambda^2 \\sL^2}.\n\\end{equation}","sha256_utf8":"6abb8241167d3523b886fe17a84777e7e17c43de2da4542e8e2b9c935f61694b"}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","title":"目标函数可定义性","source_units":[{"start":70069,"end":70284,"start_line":908,"end_line":910,"raw_tex":"\\begin{enumerate}[label=\\textup{\\textrm{(H.3)}},leftmargin=3em,topsep=2mm,itemsep=1ex,partopsep=0ex]\n        \\item \\label{A3} The objective function $\\psi : \\Rn \\to \\Rex$ is definable. %\\vspace{2mm}\n\\end{enumerate}\n","sha256_utf8":"3a427f5e17af18cff74c7bd6e1a4fc1e68d2db92e6e186d676a681e277395ddc"}]},{"id":"S018","semantic_type":"assumption","kind":"assumption","title":"有界子序列的满概率事件","source_units":[{"start":77013,"end":77117,"start_line":963,"end_line":964,"raw_tex":"\\[ \\cX := \\{\\omega\\in\\Omega: {\\liminf}_{k\\to\\infty}\\; \\|\\sx^k(\\omega)\\| < \\infty\\}. %\\vspace{-.25ex} \n\\]","sha256_utf8":"133b898aad9c47e016adced155a258eaf20e0bc212df6664639d7ce3bab809ab"},{"start":77120,"end":77588,"start_line":966,"end_line":970,"raw_tex":"Clearly, if $\\omega\\in\\cX$, then the realization $\\revise{\\{x^k\\}_{k\\in\\N}} = \\revise{\\{\\sx^k(\\omega)\\}_{k\\in\\N}}$ has at least one bounded subsequence and accumulation point. We will work with the following assumption: \n%\\vspace{1.5mm}\n \\begin{enumerate}[label=\\textup{\\textrm{(S.4)}},leftmargin=3em,topsep=1.5mm,itemsep=1ex,partopsep=0ex]\n        \\item \\label{B4} The event $\\cX \\in \\mathcal F$ occurs with probability $1$, i.e., $\\Prob(\\cX)=1$.\n    \\end{enumerate}\n","sha256_utf8":"6aa10f26b1df8f2d36cfa0078faa8d12b18cad29ef852dbd639287564e5c2168"}]},{"id":"S019","semantic_type":"lemma","kind":"lemma","title":"窗口内迭代与误差界","source_units":[{"start":44046,"end":44652,"start_line":596,"end_line":604,"raw_tex":"\\begin{lemma}[Iterate bounds]\n    \\label{lem:est-err}\n     Let \\ref{A1}--\\ref{A2} and \\ref{B1}--\\ref{B2} hold and let $\\revise{\\{\\sx^k\\}_{k\\in\\N}}$ and $\\revise{\\{\\sz^k\\}_{k\\in\\N}}$ be generated by $\\NSGD$. Let us set $\\Llambda := \\sL+2\\lambda^{-1}$. For all $0\\leq m<n$ with $\\tau_{m,n} \\leq 1/(2\\Llambda)$, we have\n     %\n    \\begin{align*}\n        \\max_{m<i\\leq n}\\|\\sx^i-\\sx^m\\| &\\leq \\max_{m<i\\leq n}\\|\\sz^i-\\sz^m\\| \\leq 2\\tau_{m,n} \\norm{\\Fnor(\\sz^m)} + 2\\scs_{m,n}\\quad \\text{and}\\\\\n        \\|{\\scp^{m,n}}\\| &\\leq 2\\Llambda\\tau_{m,n}^2\\norm{\\Fnor(\\sz^m)} + 2\\scs_{m,n}. \n    \\end{align*}\n\\end{lemma}","sha256_utf8":"4c85434bb548b9530035fb781f0c1fb11514d5a26015ee65cfaacf1f22454363"}]},{"id":"S020","semantic_type":"proof","kind":"proof","title":"窗口内迭代界的证明","source_units":[{"start":143517,"end":146313,"start_line":1716,"end_line":1753,"raw_tex":"\\begin{proof} Let $\\omega \\in \\Omega$ be arbitrary and let us set $\\tau = \\tau_{m,n}$, $p = p^{m,n} = \\scp^{m,n}(\\omega)$, $s = s_{m,n} = \\scs_{m,n}(\\omega)$, $x^k = \\sx^k(\\omega)$, $z^k = \\sz^k(\\omega)$, etc. It follows from the definition of $p^{m,i}$ \\revise{and $s_{m,n}$} and the triangle inequality that\n%\n\\begin{align}\n        \\nonumber \\|p^{m,i}\\| &\\leq {\\sum}_{j=m}^{i-1}\\alpha_j\\|{\\Fnor(z^j) - \\Fnor(z^m)}\\| + \\Big\\|{{\\sum}_{j=m}^{i-1} \\alpha_j e^j}\\Big\\| \\\\ &\\leq {\\sum}_{j=m}^{i-1}\\alpha_j\\|{\\Fnor(z^j) - \\Fnor(z^m)}\\| + s,\\qquad \\text{for all $i\\in(m,n] \\cap \\N$.} \\label{eq:lem-4-est-1}\n\\end{align}\n% \nRecalling $\\Fnor(z) = \\nabla f(x) + \\lambda^{-1}(z-x)$ and using \\ref{A1} and the nonexpansiveness of the proximity operator (i.e., $\\|x^i-x^m\\| \\leq \\|z^i-z^m\\|$), we have\n%\n\\begin{equation}\n\\label{eq:nor is Lip}\n     \\|\\Fnor(z^j) - \\Fnor(z^m)\\| \\leq (\\sL+\\lambda^{-1})\\|x^j-x^m\\| + \\lambda^{-1}\\|z^j-z^m\\| \\leq  \\Llambda \\|z^j-z^m\\|,% =: C_\\lambda\\|z^i-z^m\\|.\n\\end{equation} \n%\nwhere $\\Llambda =\\sL+2\\lambda^{-1}$. Thus, combining the relation $z^i = z^m - \\tau_{m,i}\\Fnor(z^m) + p^{m,i}$ (cf$.$ \\cref{eq:pretty-important})\\revise{, $\\tau_{m,i} \\leq \\tau$,} and the inequality \\cref{eq:lem-4-est-1}, we can infer\n% \n\\begin{align}\n    \\nonumber \\|z^i-z^m\\| &\\leq \\tau_{m,i} \\|\\Fnor(z^m)\\| + \\|p^{m,i}\\| \\\\\n    \\nonumber &\\leq {\\sum}_{j=m}^{i-1} \\alpha_j \\|\\Fnor(z^j)-\\Fnor(z^m)\\| + \\tau \\|\\Fnor(z^m)\\| + s\\\\ \n    \\label{eq:lem-4-est-3} &\\leq \\Llambda {\\sum}_{j=m}^{i-1} \\alpha_j  \\|z^j-z^m\\| + \\tau \\|\\Fnor(z^m)\\| + s, \\qquad \\text{for all $i\\in(m,n] \\cap \\N$}.\n\\end{align}\n%\nWe now apply Gronwall's inequality (\\cref{Thm:G}) upon setting $b_1 :=\\tau \\|\\Fnor(z^m)\\| + s$, $b_2 := \\Llambda$, $a_k:=\\alpha_{m+k}$, $y_k:=\\|z^{m+k} - z^m\\|$, and $t:=i-m-1$. For all $i\\in(m,n] \\cap \\N$, this establishes the following upper bound  \n%\n\\begin{equation}\n            \\|z^i - z^m\\|  \\leq \\exp(\\tau \\Llambda) \\cdot [\\tau\\|\\Fnor(z^m)\\|+s]\\leq 2\\tau\\|\\Fnor(z^m)\\| + 2s,\n            \\label{eq:lem-4-est-5}\n\\end{equation}\n%\nwhere the last inequality is due to $\\tau\\Llambda \\leq \\frac12$ and $\\exp(\\frac12)\\leq 2$. Moreover, it follows from $\\|x^i-x^m\\| \\leq \\|z^i-z^m\\|$ that ${\\max}_{m<i\\leq n}\\|x^i-x^m\\| \\leq {\\max}_{m<i\\leq n}\\|z^i-z^m\\| \\leq 2\\tau\\|\\Fnor(z^m)\\| + 2s$. This verifies the first bound in \\cref{lem:est-err}. \n     \nNext, combining \\cref{eq:nor is Lip} and \\cref{eq:lem-4-est-5}, we obtain\n%\n\\[\n{\\max}_{m<i\\leq n} \\|\\Fnor(z^i) - \\Fnor(z^m)\\|\\leq \\Llambda{\\max}_{m<i\\leq n}\\|z^i-z^m\\| \\leq 2\\Llambda[\\tau\\|\\Fnor(z^m)\\| + s].\n\\]\n%\nUsing this bound in \\cref{eq:lem-4-est-1} with $i=n$ and recalling $\\tau={\\sum}_{i=m}^{n-1} \\alpha_i$, this yields $\\|p\\|\\leq s+ 2\\Llambda (\\tau\\|\\Fnor(z^m)\\| + s) {\\sum}_{j=m}^{n-1} \\alpha_j \\leq 2\\Llambda\\tau^2\\|\\Fnor(z^m)\\| + 2s$.\n\\end{proof}","sha256_utf8":"4ff76d49f208465cf80e007ab7e80f21b15c7edf9008ca6d87d9ff69e0050810"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"聚合误差的三角估计","source_units":[{"start":143517,"end":144130,"start_line":1716,"end_line":1720,"raw_tex":"\\begin{proof} Let $\\omega \\in \\Omega$ be arbitrary and let us set $\\tau = \\tau_{m,n}$, $p = p^{m,n} = \\scp^{m,n}(\\omega)$, $s = s_{m,n} = \\scs_{m,n}(\\omega)$, $x^k = \\sx^k(\\omega)$, $z^k = \\sz^k(\\omega)$, etc. It follows from the definition of $p^{m,i}$ \\revise{and $s_{m,n}$} and the triangle inequality that\n%\n\\begin{align}\n        \\nonumber \\|p^{m,i}\\| &\\leq {\\sum}_{j=m}^{i-1}\\alpha_j\\|{\\Fnor(z^j) - \\Fnor(z^m)}\\| + \\Big\\|{{\\sum}_{j=m}^{i-1} \\alpha_j e^j}\\Big\\| \\\\ &\\leq {\\sum}_{j=m}^{i-1}\\alpha_j\\|{\\Fnor(z^j) - \\Fnor(z^m)}\\| + s,\\qquad \\text{for all $i\\in(m,n] \\cap \\N$.} \\label{eq:lem-4-est-1}\n\\end{align}\n","sha256_utf8":"32a07d647b23aa5af4342fb8bbc1472b059edd820ec8b32f416bed039175cffe"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"法映射利普希茨性与递推","source_units":[{"start":144133,"end":145108,"start_line":1722,"end_line":1735,"raw_tex":"Recalling $\\Fnor(z) = \\nabla f(x) + \\lambda^{-1}(z-x)$ and using \\ref{A1} and the nonexpansiveness of the proximity operator (i.e., $\\|x^i-x^m\\| \\leq \\|z^i-z^m\\|$), we have\n%\n\\begin{equation}\n\\label{eq:nor is Lip}\n     \\|\\Fnor(z^j) - \\Fnor(z^m)\\| \\leq (\\sL+\\lambda^{-1})\\|x^j-x^m\\| + \\lambda^{-1}\\|z^j-z^m\\| \\leq  \\Llambda \\|z^j-z^m\\|,% =: C_\\lambda\\|z^i-z^m\\|.\n\\end{equation} \n%\nwhere $\\Llambda =\\sL+2\\lambda^{-1}$. Thus, combining the relation $z^i = z^m - \\tau_{m,i}\\Fnor(z^m) + p^{m,i}$ (cf$.$ \\cref{eq:pretty-important})\\revise{, $\\tau_{m,i} \\leq \\tau$,} and the inequality \\cref{eq:lem-4-est-1}, we can infer\n% \n\\begin{align}\n    \\nonumber \\|z^i-z^m\\| &\\leq \\tau_{m,i} \\|\\Fnor(z^m)\\| + \\|p^{m,i}\\| \\\\\n    \\nonumber &\\leq {\\sum}_{j=m}^{i-1} \\alpha_j \\|\\Fnor(z^j)-\\Fnor(z^m)\\| + \\tau \\|\\Fnor(z^m)\\| + s\\\\ \n    \\label{eq:lem-4-est-3} &\\leq \\Llambda {\\sum}_{j=m}^{i-1} \\alpha_j  \\|z^j-z^m\\| + \\tau \\|\\Fnor(z^m)\\| + s, \\qquad \\text{for all $i\\in(m,n] \\cap \\N$}.\n\\end{align}\n","sha256_utf8":"44229a634e2c89e64d0ccb99ed23ea79292171d25d51af534517ebe0a218cc05"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"格朗沃尔控制迭代位移","source_units":[{"start":145110,"end":145849,"start_line":1737,"end_line":1744,"raw_tex":"We now apply Gronwall's inequality (\\cref{Thm:G}) upon setting $b_1 :=\\tau \\|\\Fnor(z^m)\\| + s$, $b_2 := \\Llambda$, $a_k:=\\alpha_{m+k}$, $y_k:=\\|z^{m+k} - z^m\\|$, and $t:=i-m-1$. For all $i\\in(m,n] \\cap \\N$, this establishes the following upper bound  \n%\n\\begin{equation}\n            \\|z^i - z^m\\|  \\leq \\exp(\\tau \\Llambda) \\cdot [\\tau\\|\\Fnor(z^m)\\|+s]\\leq 2\\tau\\|\\Fnor(z^m)\\| + 2s,\n            \\label{eq:lem-4-est-5}\n\\end{equation}\n%\nwhere the last inequality is due to $\\tau\\Llambda \\leq \\frac12$ and $\\exp(\\frac12)\\leq 2$. Moreover, it follows from $\\|x^i-x^m\\| \\leq \\|z^i-z^m\\|$ that ${\\max}_{m<i\\leq n}\\|x^i-x^m\\| \\leq {\\max}_{m<i\\leq n}\\|z^i-z^m\\| \\leq 2\\tau\\|\\Fnor(z^m)\\| + 2s$. This verifies the first bound in \\cref{lem:est-err}. \n","sha256_utf8":"3c1df1ea8e3ec64c07cd6af9504e5f20bbf2aecd818f1430dcaec8fca5679dbb"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"回代得到聚合误差界","source_units":[{"start":145855,"end":146313,"start_line":1746,"end_line":1753,"raw_tex":"Next, combining \\cref{eq:nor is Lip} and \\cref{eq:lem-4-est-5}, we obtain\n%\n\\[\n{\\max}_{m<i\\leq n} \\|\\Fnor(z^i) - \\Fnor(z^m)\\|\\leq \\Llambda{\\max}_{m<i\\leq n}\\|z^i-z^m\\| \\leq 2\\Llambda[\\tau\\|\\Fnor(z^m)\\| + s].\n\\]\n%\nUsing this bound in \\cref{eq:lem-4-est-1} with $i=n$ and recalling $\\tau={\\sum}_{i=m}^{n-1} \\alpha_i$, this yields $\\|p\\|\\leq s+ 2\\Llambda (\\tau\\|\\Fnor(z^m)\\| + s) {\\sum}_{j=m}^{n-1} \\alpha_j \\leq 2\\Llambda\\tau^2\\|\\Fnor(z^m)\\| + 2s$.\n\\end{proof}","sha256_utf8":"f2f44959fd4ab91c8501190ce448ca832bc952c70f032e63d53a1e53fa4139fa"}]},{"id":"S025","semantic_type":"lemma","kind":"lemma","title":"聚合噪声的矩与可和性","source_units":[{"start":45036,"end":45731,"start_line":608,"end_line":614,"raw_tex":"\\begin{lemma}\n    \\label{lem:err_estimate}\n    Let \\ref{B1}--\\ref{B2} hold and let $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}$ be given. Then for any (arbitrary) subsequence $\\revise{\\{t_k\\}_{k\\in\\N}} \\subseteq \\N$, we have $\\Exp[{{\\scs_{t_k,t_{k+1}}^2}}] \\leq 4 \\sum_{i=t_k}^{t_{k+1}-1}\\alpha_i^2\\sigma_i^2$. Moreover, if there is a non-decreasing sequence $\\revise{\\{\\beta_k\\}_{k\\in\\N}}\\subset\\R_{++}$ such that ${\\sum}_{k=0}^\\infty \\alpha_k^2\\beta_k^2\\sigma_k^2 < \\infty$, it holds that\n    %\n    \\[ {\\sum}_{k=0}^\\infty \\beta_{t_k}^2 \\Exp[{{\\scs_{t_k,t_{k+1}}^2}}] < \\infty \\quad \\text{and}\\quad {\\sum}_{k=0}^\\infty \\beta_{t_k}^2 \\scs_{t_k,t_{k+1}}^2 < \\infty \\quad \\text{a$.$s$.$.}\\] \n    %\n\\end{lemma}","sha256_utf8":"f4ae0ff751342677812ca826ae3018f5df697990f7df39766845a18fc74dc14f"}]},{"id":"S026","semantic_type":"proof","kind":"proof","title":"聚合噪声估计的证明","source_units":[{"start":146429,"end":148425,"start_line":1757,"end_line":1784,"raw_tex":"\\begin{proof}  Let us define the filtration $\\mathcal{U}_\\ell:=\\mathcal{F}_{t_k+\\ell}$ and introduce the sequence \\revise{$\\{\\sy^\\ell\\}_{\\ell\\in\\N}$} as follows $\\sy^0:=0$, $\\sy^\\ell := {\\sum}_{i=t_k}^{\\min\\{t_k+\\ell,t_{k+1}\\}-1}\\alpha_i \\se^i$ for all $\\ell \\geq 1$.\n%\nThen, $\\sy_\\ell$ is $\\mathcal U_\\ell$-measurable and for all $\\ell \\in \\{1,\\dots,t_{k+1}-t_k-1\\}$, we have \n%\n\\[ {\\Exp[{\\sy^{\\ell+1} \\mid \\mathcal U_\\ell}] = {\\sum}_{i=t_k}^{t_k+\\ell} \\alpha_i\\Exp[{\\se^i \\mid \\mathcal U_\\ell}] = \\sy^\\ell + \\alpha_{t_k+\\ell} \\Exp[{\\se^{t_k+\\ell} \\mid \\mathcal{F}_{t_k+\\ell}}] = \\sy^\\ell.} \\]\n% \n(Similarly for $\\ell \\geq t_{k+1}-t_k$). Hence, \\revise{$\\{\\sy^\\ell\\}_{\\ell\\in\\N}$} defines a \\revise{$\\{\\mathcal{U}_{\\ell}\\}_{\\ell\\in\\N}$}-martingale. Let us further set $\\sr_k = \\scs_{t_k,t_{k+1}}$. By the Burkholder-Davis-Gundy inequality (\\cref{Thm:BDG} with $p=2$ and $C_p\\leq 4$) and assumption \\ref{B1}, we have $\\Exp[{\\sr_k^2}] =\\Exp[{{\\sup}_{\\ell>0} \\|\\sy^\\ell\\|^2}]$ and\n%\n\\begin{align*} \n\\nonumber\\Exp[{\\sr_k^2}] \\leq 4 \\Exp\\Big[{ {\\sum}_{\\ell=1}^{\\infty}\\|\\sy^\\ell-\\sy^{\\ell-1}\\|^2}\\Big] & = 4 {\\sum}_{i=t_k}^{t_{k+1}-1}\\alpha_i^2 \\Exp[{ \\|\\se^i\\|^2}]  \\leq 4 {\\sum}_{i=t_k}^{t_{k+1}-1} \\alpha_i^2\\sigma_i^2. \n\\end{align*}\n%\nThis proves the first claim in \\Cref{lem:err_estimate}. Since $\\revise{\\{\\beta_k\\}_{k\\in\\N}}$ is non-decreasing, we obtain%\n%\n\\begin{equation}\\label{eq:app-bdg}\n\t\\beta_{t_k}^2 \\Exp[\\sr_k^2] \\leq 4 \\beta_{t_k}^2 {\\sum}_{i=t_k}^{t_{k+1}-1} \\alpha_i^2\\sigma_i^2 \\leq 4  {\\sum}_{i=t_k}^{t_{k+1}-1} \\alpha_i^2\\beta_{i}^2\\sigma_i^2\n\\end{equation}\n%\nfor all $k \\in \\N$. The monotone convergence theorem and \\cref{eq:app-bdg} further imply\n%\n\\begin{align*}\n \\Exp\\Big[{ {\\sum}_{k=0}^\\infty \\beta_{t_k}^2{\\sr_k^2}}\\Big] = {\\sum}_{k=0}^{\\infty}\\beta_{t_k}^2 \\Exp[\\sr_k^2] \n \\leq \n    {4} {\\sum}_{i=t_{0}}^\\infty \\alpha_i^2\\beta_i^2\\sigma_i^2 < \\infty,\n\\end{align*}\n%\nand consequently, it holds that $\\sum_{k=0}^\\infty \\beta_{t_k}^2{\\sr_k^2} < \\infty$ a$.$s$.$.\n\\end{proof}","sha256_utf8":"b7367b1fc1166c88890bfe83c3cdfe4d74e3d30a6726f180400490b2057e4eea"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"构造窗口内误差鞅","source_units":[{"start":146429,"end":147024,"start_line":1757,"end_line":1761,"raw_tex":"\\begin{proof}  Let us define the filtration $\\mathcal{U}_\\ell:=\\mathcal{F}_{t_k+\\ell}$ and introduce the sequence \\revise{$\\{\\sy^\\ell\\}_{\\ell\\in\\N}$} as follows $\\sy^0:=0$, $\\sy^\\ell := {\\sum}_{i=t_k}^{\\min\\{t_k+\\ell,t_{k+1}\\}-1}\\alpha_i \\se^i$ for all $\\ell \\geq 1$.\n%\nThen, $\\sy_\\ell$ is $\\mathcal U_\\ell$-measurable and for all $\\ell \\in \\{1,\\dots,t_{k+1}-t_k-1\\}$, we have \n%\n\\[ {\\Exp[{\\sy^{\\ell+1} \\mid \\mathcal U_\\ell}] = {\\sum}_{i=t_k}^{t_k+\\ell} \\alpha_i\\Exp[{\\se^i \\mid \\mathcal U_\\ell}] = \\sy^\\ell + \\alpha_{t_k+\\ell} \\Exp[{\\se^{t_k+\\ell} \\mid \\mathcal{F}_{t_k+\\ell}}] = \\sy^\\ell.} \\]\n","sha256_utf8":"c158a1d3e5a3cac2c6a2f4b494bcc8b8756923072724e4d169212df92b914152"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"鞅最大不等式给出二阶矩","source_units":[{"start":147027,"end":147662,"start_line":1763,"end_line":1767,"raw_tex":"(Similarly for $\\ell \\geq t_{k+1}-t_k$). Hence, \\revise{$\\{\\sy^\\ell\\}_{\\ell\\in\\N}$} defines a \\revise{$\\{\\mathcal{U}_{\\ell}\\}_{\\ell\\in\\N}$}-martingale. Let us further set $\\sr_k = \\scs_{t_k,t_{k+1}}$. By the Burkholder-Davis-Gundy inequality (\\cref{Thm:BDG} with $p=2$ and $C_p\\leq 4$) and assumption \\ref{B1}, we have $\\Exp[{\\sr_k^2}] =\\Exp[{{\\sup}_{\\ell>0} \\|\\sy^\\ell\\|^2}]$ and\n%\n\\begin{align*} \n\\nonumber\\Exp[{\\sr_k^2}] \\leq 4 \\Exp\\Big[{ {\\sum}_{\\ell=1}^{\\infty}\\|\\sy^\\ell-\\sy^{\\ell-1}\\|^2}\\Big] & = 4 {\\sum}_{i=t_k}^{t_{k+1}-1}\\alpha_i^2 \\Exp[{ \\|\\se^i\\|^2}]  \\leq 4 {\\sum}_{i=t_k}^{t_{k+1}-1} \\alpha_i^2\\sigma_i^2. \n\\end{align*}\n","sha256_utf8":"ad2c6f1880927eaa3088eed8f3a5fd3a6eddc343adc9249a7f5b6e967b358ccb"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"单调权重控制加权误差","source_units":[{"start":147664,"end":148005,"start_line":1769,"end_line":1773,"raw_tex":"This proves the first claim in \\Cref{lem:err_estimate}. Since $\\revise{\\{\\beta_k\\}_{k\\in\\N}}$ is non-decreasing, we obtain%\n%\n\\begin{equation}\\label{eq:app-bdg}\n\t\\beta_{t_k}^2 \\Exp[\\sr_k^2] \\leq 4 \\beta_{t_k}^2 {\\sum}_{i=t_k}^{t_{k+1}-1} \\alpha_i^2\\sigma_i^2 \\leq 4  {\\sum}_{i=t_k}^{t_{k+1}-1} \\alpha_i^2\\beta_{i}^2\\sigma_i^2\n\\end{equation}\n","sha256_utf8":"8896e9eb291e79e3f6656cf4495158ee258717a57ee14f4f3a4490c1fc661b54"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"单调收敛推出几乎必然可和","source_units":[{"start":148007,"end":148425,"start_line":1775,"end_line":1784,"raw_tex":"for all $k \\in \\N$. The monotone convergence theorem and \\cref{eq:app-bdg} further imply\n%\n\\begin{align*}\n \\Exp\\Big[{ {\\sum}_{k=0}^\\infty \\beta_{t_k}^2{\\sr_k^2}}\\Big] = {\\sum}_{k=0}^{\\infty}\\beta_{t_k}^2 \\Exp[\\sr_k^2] \n \\leq \n    {4} {\\sum}_{i=t_{0}}^\\infty \\alpha_i^2\\beta_i^2\\sigma_i^2 < \\infty,\n\\end{align*}\n%\nand consequently, it holds that $\\sum_{k=0}^\\infty \\beta_{t_k}^2{\\sr_k^2} < \\infty$ a$.$s$.$.\n\\end{proof}","sha256_utf8":"2710476c54df5458c3bec51c203b8ee2a204c333e78dbdef52dc765ba276b8ed"}]},{"id":"S031","semantic_type":"lemma","kind":"lemma","title":"法映射与目标值基本界","source_units":[{"start":46494,"end":47397,"start_line":628,"end_line":635,"raw_tex":"\\begin{lemma}\n\t\\label{lem:basic bounds}\n\tLet the assumptions \\ref{A1}--\\ref{A2} and \\ref{B1}--\\ref{B2} be satisfied and let the sequences $\\revise{\\{\\sx^k\\}_{k\\in\\N}}$, $\\revise{\\{\\sz^k\\}_{k\\in\\N}}$ be generated by $\\NSGD$. Let $0 \\leq m<n$ be given with $\\tau_{m,n} < \\lambda$. Then, the following two bounds are valid:\n    \\begin{enumerate}[label=\\textup{\\textrm{(\\alph*)}},leftmargin=3em,topsep=2pt,itemsep=.5ex,partopsep=0ex]\n    \\item It holds that $\\norm{\\Fnor(\\sz^n)}^2 \\leq  ({1-\\frac{\\tau_{m,n}}{2\\lambda}})\\norm{\\Fnor(\\sz^m)}^2 - \\frac{2}{\\lambda}\\iprod{\\su^{m,n}}{\\sx^n-\\sx^m} + [({\\sL+\\frac{1}{\\lambda}})^2 + \\frac{2\\lambda \\sL^2}{\\tau_{m,n}}] \\norm{\\sx^n-\\sx^m}^2 + \\frac{3}{2\\lambda\\tau_{m,n}} \\norm{\\scp^{m,n}}^2$.\n    \\item We have $\\psi(\\sx^n) \\leq \\psi(\\sx^m) + ({\\frac{\\sL}{2}-\\frac{1}{\\lambda}})\\norm{\\sx^n-\\sx^m}^2 + \\iprod{\\su^{m,n}}{\\sx^n-\\sx^m}.$\t\n    \\end{enumerate}\n\\end{lemma}","sha256_utf8":"c9684ddc2757aef12331c4298a567f45e97f4b318adb92e5845faddfb1dac0ba"}]},{"id":"S032","semantic_type":"proof","kind":"proof","title":"两个基本界的完整证明","source_units":[{"start":148542,"end":151844,"start_line":1788,"end_line":1841,"raw_tex":"\\begin{proof}[Proof of (a)]\nLet $\\omega\\in\\Omega$ be arbitrary and denote $x^k = \\sx^k(\\omega)$, $z^k = \\sz^k(\\omega)$, $p = \\scp^{m,n}(\\omega)$, $u = \\su^{m,n}(\\omega)$, $\\tau = \\tau_{m,n}$, etc$.$. Recalling $\\Fnor(z) = \\nabla f(x) + \\lambda^{-1}(z-x)$ and by the definition \\cref{eq:u-def1} of $\\su^{m,n}$, we can express $\\Fnor(z^n)$ as\n%\n\\begin{align*} \n    \\Fnor(z^n) \n    & = \\Fnor(z^m) + [\\nabla f(x^n)-\\nabla f(x^m)] + \\lambda^{-1}\\prt{z^n-z^m+x^m-x^n}\\\\\n    & = u + [\\nabla f(x^n)-\\nabla f(x^m)] - \\lambda^{-1}(x^n-x^m).\n\\end{align*}\t\n%\nTaking squares and using the Lipschitz continuity of $\\nabla f$, this yields\n%\n\\begin{align}\n            \\nonumber \\norm{\\Fnor(z^n)}^2 & = \\norm{u}^2 + 2\\iprod{u}{(\\nabla f(x^n)-\\nabla f(x^m)) - \\lambda^{-1}(x^n-x^m)} \\\\\n            \\nonumber&\\hspace{4ex}  +  \\norm{(\\nabla f(x^n)-\\nabla f(x^m)) - \\lambda^{-1}(x^n-x^m)}^2\\\\\n            \\label{C1} & \\hspace{-6ex} \\leq \\norm{u}^2 + 2\\iprod{u}{(\\nabla f(x^n)-\\nabla f(x^m))-\\lambda^{-1}(x^n-x^m)} + (\\sL+{\\lambda}^{-1})^2 \\norm{x^n-x^m}^2.\n\\end{align}\n%\nUsing $u=(1-\\lambda^{-1}\\tau)\\Fnor(z^m) + \\lambda^{-1}p$ (cf$.$ \\cref{eq:u-def3}) and $\\norm{a+b}^2 \\leq (1+\\varepsilon) \\norm{a}^2 + (1+\\frac{1}{\\varepsilon})\\norm{b}^2$ with $a= (1-\\lambda^{-1}\\tau) \\Fnor(z^m)$, $b = \\lambda^{-1}p$, and $\\varepsilon = \\frac{\\lambda^{-1}\\tau}{1-\\lambda^{-1}\\tau}$, we have\n%\n\\begin{equation}\\label{C2}\n        \\norm{u}^2 \\leq (1-\\lambda^{-1}{\\tau}) \\|{\\Fnor(z^m)}\\|^2 + (\\lambda\\tau)^{-1} \\|{p}\\|^2.\n\\end{equation}\n%\nUtilizing Young's inequality for $2\\iprod{u}{\\nabla f(x^n)-\\nabla f(x^m)}$ and \\cref{C2}, we obtain \n%\n\\begin{align}\n            \\nonumber 2\\iprod{u}{\\nabla f(x^n)-\\nabla f(x^m)} & \\leq \\tfrac{\\tau}{2\\lambda} \\norm{u}^2 + \\tfrac{2\\lambda}{\\tau} \\norm{\\nabla f(x^n)-\\nabla f(x^m)}^2\\\\\n            \\label{C3} & \\leq  \\tfrac{\\tau}{2\\lambda}\\norm{\\Fnor(z^m)}^2 + \\tfrac{1}{2\\lambda^2} \\norm{p}^2 + \\tfrac{2\\lambda \\sL^2}{\\tau} \\norm{x^n-x^m}^2,\n\\end{align}\n%\nSumming up the estimates \\cref{C1}--\\cref{C3}, and noticing $\\tau \\leq \\lambda$, it follows\n%\n\\begin{align*}\n        \\norm{\\Fnor(z^n)}^2  & \\leq  ({1-\\tfrac{\\tau}{2\\lambda}})\\|{\\Fnor(z^m)}\\|^2 - \\tfrac{2}{\\lambda}\\iprod{u}{x^n-x^m} \\\\\n        & \\hspace{4ex} + [(\\sL+\\lambda^{-1})^2 + {2\\sL^2\\lambda}{\\tau^{-1}}] \\norm{x^n-x^m}^2 + \\tfrac{3}{2\\lambda\\tau} \\norm{p}^2.\n\\end{align*}\n\n\\emph{Proof of (b).} By \\ref{A1}--\\ref{A2}, $\\nabla f$ is $\\sL$-continuous and $\\vp$ is convex. Thus, we have\n%\n\\begin{equation*}\n        \\psi(x^n)-\\psi(x^m) \\leq \\iprod{\\nabla f(x^m)}{x^n-x^m}+\\tfrac{\\sL}{2}\\|x^n-x^m\\|^2 + \\tfrac{1}{\\lambda}\\iprod{z^n-x^n}{x^n-x^m},\n\\end{equation*}\n%\nwhere we used $\\lambda^{-1}(z^n-x^n) \\in\\partial \\vp(x^n)$ in the last inequality. Moreover, based on the definition of $\\Fnor$, we can reformulate %\\cref{eq:lem-1-est-1} \nthe above inequality as\n%\n\\begin{align*}\n\\psi(x^n)-\\psi(x^m) & \\leq \\iprod{\\nabla f(x^m)+\\lambda^{-1}(z^m-x^m)}{x^n-x^m}+\\tfrac{\\sL}{2}\\|{x^n-x^m}\\|^2\\\\\n& \\hspace{4ex} + \\lambda^{-1}\\iprod{z^n-x^n}{x^n-x^m} - \\lambda^{-1}\\iprod{z^m-x^m}{x^n-x^m}\\\\\n& \\hspace{-4ex} = [{\\tfrac{\\sL}{2}-\\tfrac{1}{\\lambda}}]\\|{x^n-x^m}\\|^2 + \t\\iprod{\\Fnor(z^m)}{x^n-x^m} + \\tfrac{1}{\\lambda}\\iprod{z^n-z^m}{x^n-x^m}.\n\\end{align*}\n%\nThis concludes the proof by substituting \\cref{eq:u-def1} into the last expression.\n\\end{proof}","sha256_utf8":"28b313bcfd772a1400df629169d71184c332684b44841c1b3cef9a6e413f6adf"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"展开法映射平方范数","source_units":[{"start":148542,"end":149590,"start_line":1788,"end_line":1803,"raw_tex":"\\begin{proof}[Proof of (a)]\nLet $\\omega\\in\\Omega$ be arbitrary and denote $x^k = \\sx^k(\\omega)$, $z^k = \\sz^k(\\omega)$, $p = \\scp^{m,n}(\\omega)$, $u = \\su^{m,n}(\\omega)$, $\\tau = \\tau_{m,n}$, etc$.$. Recalling $\\Fnor(z) = \\nabla f(x) + \\lambda^{-1}(z-x)$ and by the definition \\cref{eq:u-def1} of $\\su^{m,n}$, we can express $\\Fnor(z^n)$ as\n%\n\\begin{align*} \n    \\Fnor(z^n) \n    & = \\Fnor(z^m) + [\\nabla f(x^n)-\\nabla f(x^m)] + \\lambda^{-1}\\prt{z^n-z^m+x^m-x^n}\\\\\n    & = u + [\\nabla f(x^n)-\\nabla f(x^m)] - \\lambda^{-1}(x^n-x^m).\n\\end{align*}\t\n%\nTaking squares and using the Lipschitz continuity of $\\nabla f$, this yields\n%\n\\begin{align}\n            \\nonumber \\norm{\\Fnor(z^n)}^2 & = \\norm{u}^2 + 2\\iprod{u}{(\\nabla f(x^n)-\\nabla f(x^m)) - \\lambda^{-1}(x^n-x^m)} \\\\\n            \\nonumber&\\hspace{4ex}  +  \\norm{(\\nabla f(x^n)-\\nabla f(x^m)) - \\lambda^{-1}(x^n-x^m)}^2\\\\\n            \\label{C1} & \\hspace{-6ex} \\leq \\norm{u}^2 + 2\\iprod{u}{(\\nabla f(x^n)-\\nabla f(x^m))-\\lambda^{-1}(x^n-x^m)} + (\\sL+{\\lambda}^{-1})^2 \\norm{x^n-x^m}^2.\n\\end{align}\n","sha256_utf8":"a9f7b7fc0f5ce5edff3979a882c49690bf3cc17df5e5c96baac9d2eb8957a817"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"控制辅助向量范数","source_units":[{"start":149592,"end":150042,"start_line":1805,"end_line":1809,"raw_tex":"Using $u=(1-\\lambda^{-1}\\tau)\\Fnor(z^m) + \\lambda^{-1}p$ (cf$.$ \\cref{eq:u-def3}) and $\\norm{a+b}^2 \\leq (1+\\varepsilon) \\norm{a}^2 + (1+\\frac{1}{\\varepsilon})\\norm{b}^2$ with $a= (1-\\lambda^{-1}\\tau) \\Fnor(z^m)$, $b = \\lambda^{-1}p$, and $\\varepsilon = \\frac{\\lambda^{-1}\\tau}{1-\\lambda^{-1}\\tau}$, we have\n%\n\\begin{equation}\\label{C2}\n        \\norm{u}^2 \\leq (1-\\lambda^{-1}{\\tau}) \\|{\\Fnor(z^m)}\\|^2 + (\\lambda\\tau)^{-1} \\|{p}\\|^2.\n\\end{equation}\n","sha256_utf8":"8a574a168ab856912bb9cb1f60b67c0e646db733860e199173ff618ff0892be9"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"合并杨氏估计得到法映射界","source_units":[{"start":150044,"end":150879,"start_line":1811,"end_line":1823,"raw_tex":"Utilizing Young's inequality for $2\\iprod{u}{\\nabla f(x^n)-\\nabla f(x^m)}$ and \\cref{C2}, we obtain \n%\n\\begin{align}\n            \\nonumber 2\\iprod{u}{\\nabla f(x^n)-\\nabla f(x^m)} & \\leq \\tfrac{\\tau}{2\\lambda} \\norm{u}^2 + \\tfrac{2\\lambda}{\\tau} \\norm{\\nabla f(x^n)-\\nabla f(x^m)}^2\\\\\n            \\label{C3} & \\leq  \\tfrac{\\tau}{2\\lambda}\\norm{\\Fnor(z^m)}^2 + \\tfrac{1}{2\\lambda^2} \\norm{p}^2 + \\tfrac{2\\lambda \\sL^2}{\\tau} \\norm{x^n-x^m}^2,\n\\end{align}\n%\nSumming up the estimates \\cref{C1}--\\cref{C3}, and noticing $\\tau \\leq \\lambda$, it follows\n%\n\\begin{align*}\n        \\norm{\\Fnor(z^n)}^2  & \\leq  ({1-\\tfrac{\\tau}{2\\lambda}})\\|{\\Fnor(z^m)}\\|^2 - \\tfrac{2}{\\lambda}\\iprod{u}{x^n-x^m} \\\\\n        & \\hspace{4ex} + [(\\sL+\\lambda^{-1})^2 + {2\\sL^2\\lambda}{\\tau^{-1}}] \\norm{x^n-x^m}^2 + \\tfrac{3}{2\\lambda\\tau} \\norm{p}^2.\n\\end{align*}\n","sha256_utf8":"d439771dcfc09c0b81fab7807cddce6dad106f061c6dc544ff5c1c9498375a0e"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"光滑性与凸性给出目标值界","source_units":[{"start":150880,"end":151844,"start_line":1825,"end_line":1841,"raw_tex":"\\emph{Proof of (b).} By \\ref{A1}--\\ref{A2}, $\\nabla f$ is $\\sL$-continuous and $\\vp$ is convex. Thus, we have\n%\n\\begin{equation*}\n        \\psi(x^n)-\\psi(x^m) \\leq \\iprod{\\nabla f(x^m)}{x^n-x^m}+\\tfrac{\\sL}{2}\\|x^n-x^m\\|^2 + \\tfrac{1}{\\lambda}\\iprod{z^n-x^n}{x^n-x^m},\n\\end{equation*}\n%\nwhere we used $\\lambda^{-1}(z^n-x^n) \\in\\partial \\vp(x^n)$ in the last inequality. Moreover, based on the definition of $\\Fnor$, we can reformulate %\\cref{eq:lem-1-est-1} \nthe above inequality as\n%\n\\begin{align*}\n\\psi(x^n)-\\psi(x^m) & \\leq \\iprod{\\nabla f(x^m)+\\lambda^{-1}(z^m-x^m)}{x^n-x^m}+\\tfrac{\\sL}{2}\\|{x^n-x^m}\\|^2\\\\\n& \\hspace{4ex} + \\lambda^{-1}\\iprod{z^n-x^n}{x^n-x^m} - \\lambda^{-1}\\iprod{z^m-x^m}{x^n-x^m}\\\\\n& \\hspace{-4ex} = [{\\tfrac{\\sL}{2}-\\tfrac{1}{\\lambda}}]\\|{x^n-x^m}\\|^2 + \t\\iprod{\\Fnor(z^m)}{x^n-x^m} + \\tfrac{1}{\\lambda}\\iprod{z^n-z^m}{x^n-x^m}.\n\\end{align*}\n%\nThis concludes the proof by substituting \\cref{eq:u-def1} into the last expression.\n\\end{proof}","sha256_utf8":"6b009663468290c153ef091dbd320f4a4fdb3fcb5809cded51a93dce74de4353"}]},{"id":"S037","semantic_type":"proof","kind":"proof","title":"辅助内积下降估计的推导","source_units":[{"start":47399,"end":48596,"start_line":637,"end_line":645,"raw_tex":"The proof of \\cref{lem:basic bounds} is relegated to \\cref{app:lem:basic bounds}. Based on \\cref{lem:basic bounds}, we see that proper control of the inner product $\\langle \\su^{m,n}, \\sx^n-\\sx^m \\rangle$ is crucial to achieve descent (of $\\psi$). The following estimate illustrates that while this term can generate a desirable decrease, it also introduces an additional nonnegative error term: \n%\n    \\begin{align}\n    \\nonumber \\iprod{\\su^{m,n}}{\\sx^n-\\sx^m} & = (\\lambda^{-1}-\\tau_{m,n}^{-1}) \\iprod{\\sz^{n}-\\sz^m}{\\sx^n-\\sx^m} + \\tau_{m,n}^{-1}\\iprod{\\scp^{m,n}}{\\sx^n-\\sx^m}\\\\\n    \\nonumber & \\leq (\\lambda^{-1}-\\tau_{m,n}^{-1}) \\norm{\\sx^n-\\sx^m}^2 + \\tau_{m,n}^{-1} ({\\norm{\\scp^{m,n}}^2 + 0.25\\norm{\\sx^n-\\sx^m}^2})\\\\\n    \\label{u-descent} & = (\\lambda^{-1}-0.75\\tau_{m,n}^{-1}) \\norm{\\sx^n-\\sx^m}^2 + \\tau_{m,n}^{-1}\\norm{\\scp^{m,n}}^2.\n    \\end{align}\n%\nHere, we applied \\cref{eq:u-def2}, \\cref{prox-prop}, and Young's inequality. Thankfully and as we will verify in \\Cref{subsec:merit function}, the extra term $\\tau_{m,n}^{-1}\\norm{\\scp^{m,n}}^2$ in \\cref{u-descent} can be effectively compensated for by leveraging the normal map bound and choosing $\\tau_{m,n}$ sufficiently small. \n","sha256_utf8":"a7a05f5d9b90f9c657d7a2897783d51421fa58ec6211bd505ab80f42450dbe27"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"非扩张性与杨氏不等式控制内积","source_units":[{"start":47399,"end":48596,"start_line":637,"end_line":645,"raw_tex":"The proof of \\cref{lem:basic bounds} is relegated to \\cref{app:lem:basic bounds}. Based on \\cref{lem:basic bounds}, we see that proper control of the inner product $\\langle \\su^{m,n}, \\sx^n-\\sx^m \\rangle$ is crucial to achieve descent (of $\\psi$). The following estimate illustrates that while this term can generate a desirable decrease, it also introduces an additional nonnegative error term: \n%\n    \\begin{align}\n    \\nonumber \\iprod{\\su^{m,n}}{\\sx^n-\\sx^m} & = (\\lambda^{-1}-\\tau_{m,n}^{-1}) \\iprod{\\sz^{n}-\\sz^m}{\\sx^n-\\sx^m} + \\tau_{m,n}^{-1}\\iprod{\\scp^{m,n}}{\\sx^n-\\sx^m}\\\\\n    \\nonumber & \\leq (\\lambda^{-1}-\\tau_{m,n}^{-1}) \\norm{\\sx^n-\\sx^m}^2 + \\tau_{m,n}^{-1} ({\\norm{\\scp^{m,n}}^2 + 0.25\\norm{\\sx^n-\\sx^m}^2})\\\\\n    \\label{u-descent} & = (\\lambda^{-1}-0.75\\tau_{m,n}^{-1}) \\norm{\\sx^n-\\sx^m}^2 + \\tau_{m,n}^{-1}\\norm{\\scp^{m,n}}^2.\n    \\end{align}\n%\nHere, we applied \\cref{eq:u-def2}, \\cref{prox-prop}, and Young's inequality. Thankfully and as we will verify in \\Cref{subsec:merit function}, the extra term $\\tau_{m,n}^{-1}\\norm{\\scp^{m,n}}^2$ in \\cref{u-descent} can be effectively compensated for by leveraging the normal map bound and choosing $\\tau_{m,n}$ sufficiently small. \n","sha256_utf8":"a7a05f5d9b90f9c657d7a2897783d51421fa58ec6211bd505ab80f42450dbe27"}]},{"id":"S039","semantic_type":"proposition","kind":"proposition","title":"评价函数的近似下降","source_units":[{"start":49235,"end":49831,"start_line":656,"end_line":663,"raw_tex":"\\begin{proposition}[Approximate descent property]\n\\label{lem:merit-descent-1}\nLet \\ref{A1}--\\ref{A2} and \\ref{B1}--\\ref{B2} hold and let $\\revise{\\{\\sz^k\\}_{k\\in\\N}}$ be generated by $\\NSGD$. Suppose, \\revise{for $0\\leq m <n$, the} step sizes $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}$ satisfy $\\tau_{m,n} = {\\sum}_{i=m}^{n-1}\\alpha_i\\leq \\frac{\\sqrt{\\xi}}{8\\Llambda}$ where $\\Llambda := \\sL+\\frac{2}{\\lambda}$. We then have{}%\\vspace{-2ex}\n%\n\\begin{equation}\n    H(\\sz^n) - H(\\sz^m) \\leq  -\\frac{\\xi\\tau_{m,n}}{8} \\norm{\\Fnor(\\sz^m)}^2 + \\frac{8}{\\tau_{m,n}} \\scs_{m,n}^2.\n\\end{equation}\n\\end{proposition}","sha256_utf8":"2d4f29c31ecd211e290cd893da86feff0911ad497ce128ba2c341654f7efdfcc"}]},{"id":"S040","semantic_type":"proof","kind":"proof","title":"近似下降的完整证明","source_units":[{"start":49833,"end":52012,"start_line":665,"end_line":694,"raw_tex":"\\begin{proof}\nBy \\cref{lem:basic bounds} and setting $\\tau = \\tau_{m,n}$, $\\su = \\su^{m,n}$, $\\scp = \\scp^{m,n}$, and $\\scs = \\scs_{m,n}$, it holds that\n%\n\\begin{align}\n \\nonumber   H(\\sz^n) - H(\\sz^m) & \\leq  -\\frac{\\xi\\tau}{4} \\norm{\\Fnor(\\sz^m)}^2 + (1-\\xi) \\iprod{\\su}{\\sx^n-\\sx^m} + \\frac{3\\xi}{4\\tau} \\norm{\\scp}^2\\\\\n \\label{H-descent1}   &\\hspace{4ex} + \\Big[{\\Big({\\frac{\\sL}{2}-\\frac{1}{\\lambda}}\\Big) + \\frac{\\xi\\lambda}{2} \\Big({\\sL+\\frac{1}{\\lambda}}\\Big)^2 + \\frac{\\xi \\lambda^2\\sL^2}{\\tau}}\\Big] \\norm{\\sx^n-\\sx^m}^2.\n\\end{align}\n%\nInvoking the upper bound for $\\iprod{\\su}{\\sx^n-\\sx^m}$ in \\cref{u-descent}, this yields \n%\n\\begin{align}\n\\nonumber    H(\\sz^n) - H(\\sz^m) & \\leq -\\frac{\\xi\\tau}{4} \\norm{\\Fnor(\\sz^m)}^2 + \\frac{4-\\xi}{4\\tau}\\norm{\\scp}^2\\\\\n\\label{H-descent2}    &\\hspace{-6ex} + \\underbracket{\\Big[\\frac{\\sL}{2}-\\frac{\\xi}{\\lambda} + \\frac{\\xi\\lambda}{2} \\Big({\\sL+\\frac{1}{\\lambda}}\\Big)^2 + \\frac{4\\xi \\lambda^2\\sL^2 - 3(1-\\xi)}{4\\tau}\\Big]}_{=:T_1} \\norm{\\sx^n-\\sx^m}^2.\n\\end{align}\n%\nNext, we bound $T_1$. Using $\\xi= {(2+2\\lambda^2\\sL^2)^{-1}}$ and $(a+b)^2 \\leq 2(a^2+b^2)$, $a,b \\in \\R$, we have $\\frac12\\xi\\lambda(\\sL+\\lambda^{-1})^2 \\leq \\xi \\lambda (\\sL^2+\\lambda^{-2}) = \\frac{1}{2\\lambda}$ and $4\\xi \\lambda^2\\sL^2 - 3(1-\\xi) \\leq 4\\xi(\\lambda^2\\sL^2+1)-3 = -1$. Applying the bound $\\tau \\leq \\frac{\\sqrt{\\xi}}{8\\Llambda} \\leq \\frac{1}{2(\\sL+\\lambda^{-1})}$, we then obtain $T_1 \\leq \\frac{\\sL+\\lambda^{-1}}{2} - \\frac{\\xi}{\\lambda} - \\frac{1}{4\\tau} \\leq - \\frac{\\xi}{\\lambda} < 0$. Therefore, due to $\\frac{4-\\xi}{4\\tau}\\leq \\frac{1}{\\tau}$, the estimate \\cref{H-descent2} simplifies to\n%\n\\[\nH(\\sz^n) - H(\\sz^m) \\leq  -{(\\xi\\tau/4)} \\norm{\\Fnor(\\sz^m)}^2 + {\\tau^{-1}} \\norm{\\scp}^2.\n\\]\n%\nBy \\cref{lem:est-err}, we further have $\\|\\scp\\|^2 \\leq 8\\Llambda^2 \\tau^4 \\|\\Fnor(\\sz^m)\\|^2+8\\scs^2$. This implies\n%\n\\begin{align*}\n    H(\\sz^n) - H(\\sz^m) \\leq  - \\Big[\\frac{\\xi}{4} - 8\\Llambda^2 \\tau^2 \\Big]\\tau\\|{\\Fnor(\\sz^m)}\\|^2 + \\frac{8\\scs^2}{\\tau} \n    \\leq  -\\frac{\\xi\\tau}{8} \\norm{\\Fnor(\\sz^m)}^2 + \\frac{8\\scs^2}{\\tau} ,\n\\end{align*}\n%\nwhere the last inequality is again due to $\\tau \\leq \\frac{\\sqrt{\\xi}}{8\\Llambda}$.\n\\end{proof}","sha256_utf8":"01fa7195f33d01f8c47f15cd570f37075e24b25f43b8ea8879f4913ad852deef"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"合并目标值与法映射基本界","source_units":[{"start":49833,"end":50377,"start_line":665,"end_line":671,"raw_tex":"\\begin{proof}\nBy \\cref{lem:basic bounds} and setting $\\tau = \\tau_{m,n}$, $\\su = \\su^{m,n}$, $\\scp = \\scp^{m,n}$, and $\\scs = \\scs_{m,n}$, it holds that\n%\n\\begin{align}\n \\nonumber   H(\\sz^n) - H(\\sz^m) & \\leq  -\\frac{\\xi\\tau}{4} \\norm{\\Fnor(\\sz^m)}^2 + (1-\\xi) \\iprod{\\su}{\\sx^n-\\sx^m} + \\frac{3\\xi}{4\\tau} \\norm{\\scp}^2\\\\\n \\label{H-descent1}   &\\hspace{4ex} + \\Big[{\\Big({\\frac{\\sL}{2}-\\frac{1}{\\lambda}}\\Big) + \\frac{\\xi\\lambda}{2} \\Big({\\sL+\\frac{1}{\\lambda}}\\Big)^2 + \\frac{\\xi \\lambda^2\\sL^2}{\\tau}}\\Big] \\norm{\\sx^n-\\sx^m}^2.\n\\end{align}\n","sha256_utf8":"efa5752c57b96bf747a54df851483ed43415e138adfd9ee2a5ccf92467d19501"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"代入辅助内积估计","source_units":[{"start":50379,"end":50849,"start_line":673,"end_line":678,"raw_tex":"Invoking the upper bound for $\\iprod{\\su}{\\sx^n-\\sx^m}$ in \\cref{u-descent}, this yields \n%\n\\begin{align}\n\\nonumber    H(\\sz^n) - H(\\sz^m) & \\leq -\\frac{\\xi\\tau}{4} \\norm{\\Fnor(\\sz^m)}^2 + \\frac{4-\\xi}{4\\tau}\\norm{\\scp}^2\\\\\n\\label{H-descent2}    &\\hspace{-6ex} + \\underbracket{\\Big[\\frac{\\sL}{2}-\\frac{\\xi}{\\lambda} + \\frac{\\xi\\lambda}{2} \\Big({\\sL+\\frac{1}{\\lambda}}\\Big)^2 + \\frac{4\\xi \\lambda^2\\sL^2 - 3(1-\\xi)}{4\\tau}\\Big]}_{=:T_1} \\norm{\\sx^n-\\sx^m}^2.\n\\end{align}\n","sha256_utf8":"19d3b9998d4e232886aa9514072868fc8b78258be32e70e459df8d480f1b53f9"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"步长限制使位移系数为负","source_units":[{"start":50851,"end":51564,"start_line":680,"end_line":684,"raw_tex":"Next, we bound $T_1$. Using $\\xi= {(2+2\\lambda^2\\sL^2)^{-1}}$ and $(a+b)^2 \\leq 2(a^2+b^2)$, $a,b \\in \\R$, we have $\\frac12\\xi\\lambda(\\sL+\\lambda^{-1})^2 \\leq \\xi \\lambda (\\sL^2+\\lambda^{-2}) = \\frac{1}{2\\lambda}$ and $4\\xi \\lambda^2\\sL^2 - 3(1-\\xi) \\leq 4\\xi(\\lambda^2\\sL^2+1)-3 = -1$. Applying the bound $\\tau \\leq \\frac{\\sqrt{\\xi}}{8\\Llambda} \\leq \\frac{1}{2(\\sL+\\lambda^{-1})}$, we then obtain $T_1 \\leq \\frac{\\sL+\\lambda^{-1}}{2} - \\frac{\\xi}{\\lambda} - \\frac{1}{4\\tau} \\leq - \\frac{\\xi}{\\lambda} < 0$. Therefore, due to $\\frac{4-\\xi}{4\\tau}\\leq \\frac{1}{\\tau}$, the estimate \\cref{H-descent2} simplifies to\n%\n\\[\nH(\\sz^n) - H(\\sz^m) \\leq  -{(\\xi\\tau/4)} \\norm{\\Fnor(\\sz^m)}^2 + {\\tau^{-1}} \\norm{\\scp}^2.\n\\]\n","sha256_utf8":"7b87de351c21b5e4293fdcc6afd00d3f19806113367d564337832f8524947181"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"吸收聚合误差得到近似下降","source_units":[{"start":51566,"end":52012,"start_line":686,"end_line":694,"raw_tex":"By \\cref{lem:est-err}, we further have $\\|\\scp\\|^2 \\leq 8\\Llambda^2 \\tau^4 \\|\\Fnor(\\sz^m)\\|^2+8\\scs^2$. This implies\n%\n\\begin{align*}\n    H(\\sz^n) - H(\\sz^m) \\leq  - \\Big[\\frac{\\xi}{4} - 8\\Llambda^2 \\tau^2 \\Big]\\tau\\|{\\Fnor(\\sz^m)}\\|^2 + \\frac{8\\scs^2}{\\tau} \n    \\leq  -\\frac{\\xi\\tau}{8} \\norm{\\Fnor(\\sz^m)}^2 + \\frac{8\\scs^2}{\\tau} ,\n\\end{align*}\n%\nwhere the last inequality is again due to $\\tau \\leq \\frac{\\sqrt{\\xi}}{8\\Llambda}$.\n\\end{proof}","sha256_utf8":"7c27b4cf4f96925b170d5809c1c1aff30b0035803d827effb235c52240f7fccb"}]},{"id":"S045","semantic_type":"lemma","kind":"lemma","title":"时间窗口存在与双边界","source_units":[{"start":53953,"end":54639,"start_line":716,"end_line":725,"raw_tex":"\\begin{lemma}\\label{lem:exist time}\nLet $\\sT > 0$, $\\delta \\in (0,1)$ be given. The following assertions are valid.\n%\n \\begin{enumerate}[label=\\textup{\\textrm{(\\alph*)}},leftmargin=3em,topsep=0pt,itemsep=.5ex,partopsep=0ex]\n \\item Assume $\\alpha_k \\in (0,(1-\\delta) \\sT]$, $k< K$, and $\\alpha_k = \\alpha \\in(0,(1-\\delta)\\sT]$ for all $k \\geq K$ and some $K\\in\\N$. We then have $t_k \\to \\infty$ and $\\delta\\sT \\leq \\tau_{t_k,t_{k+1}} \\leq \\sT$ for all $k \\in\\N$.\n    \n \\item\n    If $\\lim_{k\\to\\infty}\\alpha_k = 0$ and $\\sum_{k=0}^\\infty \\alpha_k = \\infty$, it holds that $t_k \\to \\infty$ and $\\delta \\sT \\leq \\tau_{t_k,t_{k+1}} \\leq \\sT$ for all $k\\in\\N$.\n    \\end{enumerate}\n\\end{lemma}","sha256_utf8":"156243ff24a3cea843545906d900f7f649c44fddd66a931b7d3aaf2c6c0f11c0"}]},{"id":"S046","semantic_type":"proof","kind":"proof","title":"时间窗口性质的证明","source_units":[{"start":54641,"end":55651,"start_line":727,"end_line":731,"raw_tex":"\\begin{proof} \nHere, we use the convention $\\sup \\emptyset = -\\infty$. According to the construction of $\\revise{\\{t_k\\}_{k\\in\\N}}$ in \\cref{def:time win} and $\\alpha_k \\leq (1-\\delta) \\sT$, we have $t_0 = 0$ and $t_{k+1}=\\varpi(t_k) \\neq -\\infty$ for all $k\\in\\N$. Furthermore, we have $t_{k+1} \\neq \\infty$ since $\\sum_{k=0}^{\\infty} \\alpha_k = \\infty$ implies $\\tau_{t_k,n}\\rightarrow \\infty$ as $n\\rightarrow \\infty$. This also yields $t_{k} \\to \\infty$, i.e., $\\revise{\\{t_k\\}_{k\\in\\N}}$ is an infinite subsequence.  By the definition of $\\varpi(k)$, it follows $\\tau_{t_k,t_{k+1}} =  \\tau_{t_k,\\varpi(t_{k})} \\leq \\sT$. In addition, by the optimality of $\\varpi(t_k)$, we obtain $\\tau_{t_k,t_{k+1}} =  \\tau_{t_k,\\varpi(t_{k})} \\geq \\sT - \\alpha_{\\varpi(t_k)} \\geq \\delta \\sT$ for all $k$.\n% \nWe continue with part (b). Due to $\\alpha_k \\to 0$ and the definition of $t_0$, we have $\\alpha_k \\leq (1-\\delta)\\sT$ for all $k\\geq  t_0$. Hence, all previous derivations are applicable in this case.\n\\end{proof}","sha256_utf8":"a1ec43b33488ad9a874c25c7bfe7d5257050c4781cb21a2069ca268db028d045"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"常步长延拓下窗口界","source_units":[{"start":54641,"end":55436,"start_line":727,"end_line":728,"raw_tex":"\\begin{proof} \nHere, we use the convention $\\sup \\emptyset = -\\infty$. According to the construction of $\\revise{\\{t_k\\}_{k\\in\\N}}$ in \\cref{def:time win} and $\\alpha_k \\leq (1-\\delta) \\sT$, we have $t_0 = 0$ and $t_{k+1}=\\varpi(t_k) \\neq -\\infty$ for all $k\\in\\N$. Furthermore, we have $t_{k+1} \\neq \\infty$ since $\\sum_{k=0}^{\\infty} \\alpha_k = \\infty$ implies $\\tau_{t_k,n}\\rightarrow \\infty$ as $n\\rightarrow \\infty$. This also yields $t_{k} \\to \\infty$, i.e., $\\revise{\\{t_k\\}_{k\\in\\N}}$ is an infinite subsequence.  By the definition of $\\varpi(k)$, it follows $\\tau_{t_k,t_{k+1}} =  \\tau_{t_k,\\varpi(t_{k})} \\leq \\sT$. In addition, by the optimality of $\\varpi(t_k)$, we obtain $\\tau_{t_k,t_{k+1}} =  \\tau_{t_k,\\varpi(t_{k})} \\geq \\sT - \\alpha_{\\varpi(t_k)} \\geq \\delta \\sT$ for all $k$.\n","sha256_utf8":"abaaeb357b1182497649c54590dffddc7e3dae47448a158a472bd4f6392c38f8"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"递减步长下沿用窗口论证","source_units":[{"start":55439,"end":55651,"start_line":730,"end_line":731,"raw_tex":"We continue with part (b). Due to $\\alpha_k \\to 0$ and the definition of $t_0$, we have $\\alpha_k \\leq (1-\\delta)\\sT$ for all $k\\geq  t_0$. Hence, all previous derivations are applicable in this case.\n\\end{proof}","sha256_utf8":"e1d728538302d8d930da598a8e899154f2b61aa27415fe92b92832b87d723450"}]},{"id":"S049","semantic_type":"lemma","kind":"lemma","title":"窗口指标到全序列的定量界","source_units":[{"start":62893,"end":63685,"start_line":833,"end_line":843,"raw_tex":"\\begin{lemma}\\label{lem:index-time bound}\n    Let \\ref{A1}--\\ref{A2} and \\ref{B1}--\\ref{B2} hold and let $\\revise{\\{\\sx^k\\}_{k\\in\\N}}$ and $\\revise{\\{\\sz^k\\}_{k\\in\\N}}$ be generated by $\\NSGD$. Recalling $\\Llambda := \\sL+2\\lambda^{-1}$, we have for all $i>t_0$:\n%\n    \\begin{subequations}\n    \\begin{align}\n    \\label{eq:tech1}\\|\\Fnor(\\sz^i)\\|^2 &\\leq 2\\prt{2\\Llambda\\sT +1}^2\\| \\Fnor(\\sz^{t_{\\fk(i)}})\\|^2 + 8\\Llambda^2 \\sr_{\\fk(i)}^2, \\\\\n    \\label{eq:tech2}\\psi(\\sx^i) &\\leq \\psi(\\sx^{t_{\\fk(i)}})+(4\\Llambda\\sT^2+{\\lambda}/{4})\\norm{\\Fnor(\\sz^{t_{\\fk(i)}})}^2 + 4\\Llambda \\sr_{\\fk(i)}^2,\\\\\n    \\label{eq:tech3}\\psi(\\sx^i) &\\geq \\psi(\\sx^{t_{\\fk(i)+1}})-(4\\Llambda\\sT^2+{\\lambda}/{4})\\norm{\\Fnor(\\sz^{t_{\\fk(i)}})}^2 - 16\\Llambda \\sr_{\\fk(i)}^2.\n\\end{align}\n \\end{subequations}\n\\end{lemma}","sha256_utf8":"95c8f79d220c9af1491cc05f105bfd8352f6630dae9837ddcad18e010e512201"}]},{"id":"S050","semantic_type":"proof","kind":"proof","title":"全序列定量界的证明","source_units":[{"start":151969,"end":154068,"start_line":1845,"end_line":1874,"raw_tex":"\\begin{proof}\nAs shown in \\cref{eq:nor is Lip}, $\\Fnor$ is Lipschitz with modulus $\\Llambda$ and we have \n%\n\\begin{align*}\n        \\|\\Fnor(\\sz^i)\\| &\\leq \\|\\Fnor(\\sz^i) - \\Fnor(\\sz^{t_{\\fk(i)}})\\| + \\| \\Fnor(\\sz^{t_{\\fk(i)}})\\|\\\\ &\\leq  \\Llambda \\|\\sz^i-\\sz^{t_{\\fk(i)}}\\|+ \\| \\Fnor(\\sz^{t_{\\fk(i)}})\\| \\leq (2\\Llambda\\sT+1)\\| \\Fnor(\\sz^{t_{\\fk(i)}})\\| + 2\\Llambda \\scs_{t_{\\fk(i)},i}\\\\\n        &\\leq (2\\Llambda\\sT+1)\\| \\Fnor(\\sz^{t_{\\fk(i)}})\\| + 2\\Llambda \\sr_{\\fk(i)},\n\\end{align*}\n%\nwhere the \\revise{second to last} inequality follows from \\cref{lem:est-err} by setting $m=t_{\\fk(i)}$, $n=t_{\\fk(i)+1}$ (the condition $\\tau_{t_{\\fk(i)},t_{\\fk(i)+1}}\\leq \\sT \\leq 1/(2\\Llambda)$ is due to \\cref{eq:def delta and T}). Squaring both sides of the above inequality and using $(a+b)^2\\leq 2a^2+2b^2$, we obtain \\cref{eq:tech1}.\n\nBased on \\cref{lem:basic bounds} (b) and the definition of $\\su^{m,n}$ (see \\cref{eq:u-def1}), it holds that\n%\n\\[\n    \\psi(\\sx^n) \\leq \\psi(\\sx^m) + (\\sL/2-\\lambda^{-1})\\norm{\\sx^n-\\sx^m}^2 + \\iprod{\\lambda^{-1}(\\sz^n-\\sz^m) + \\Fnor(\\sz^m)}{\\sx^n-\\sx^m}.\n\\]\n%\nApplying the nonexpansiveness of the proximity operator, \\cref{prox-prop}, and $\\iprod{\\Fnor(\\sz^m)}{\\sx^n-\\sx^m} \\leq \\frac{\\lambda}{4} \\norm{\\Fnor(\\sz^m)}^2 + \\lambda^{-1} \\norm{\\sx^n-\\sx^m}^2$, we have\n%\n\\[\n    \\psi(\\sx^n)\\leq  \\psi(\\sx^m) + \\tfrac{\\Llambda}{2} \\norm{\\sz^n-\\sz^m}^2 + \\tfrac{\\lambda}{4} \\norm{\\Fnor(\\sz^m)}^2.\n\\]\n%\nInvoking \\Cref{lem:est-err} (for $\\norm{\\sz^n-\\sz^m}$) and $(a+b)^2\\leq 2a^2+2b^2$,  we obtain\n%\n\\[\n    \\psi(\\sx^n)\\leq  \\psi(\\sx^m) + (4\\Llambda\\tau_{m,n}^2+\\tfrac{\\lambda}{4}) \\norm{\\Fnor(\\sz^m)}^2 + 4\\Llambda \\scs_{m,n}^2.\n\\]\n%\nThen, \\cref{eq:tech2} readily follows by setting $n = i$, $m = t_{\\fk(i)}$ and using $\\tau_{t_{\\fk(i)},i}\\leq \\sT$. Similarly, \\cref{eq:tech3} can be obtained by setting $n = t_{\\fk(i)+1}$, $m = i$ and using  $\\tau_{i,t_{\\fk(i)+1}}\\leq \\sT$ and $\\scs_{i,t_{\\fk(i)+1}} = \\max_{i<j\\leq t_{\\fk(i)+1}}\\|{\\sum}_{\\ell=i}^{\\revise{j}-1} \\alpha_\\ell \\se^\\ell\\| \\leq \\scs_{t_{\\fk(i)},i}+  \\scs_{t_{\\fk(i)},t_{\\fk(i)+1}}  \\leq 2 \\sr_{\\fk(i)}$.\n\\end{proof}","sha256_utf8":"fb1b7761e946a0d2bd78179a791495390a6a1f48ea6be35396cec83053266b2a"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"利普希茨性传递法映射界","source_units":[{"start":151969,"end":152796,"start_line":1845,"end_line":1853,"raw_tex":"\\begin{proof}\nAs shown in \\cref{eq:nor is Lip}, $\\Fnor$ is Lipschitz with modulus $\\Llambda$ and we have \n%\n\\begin{align*}\n        \\|\\Fnor(\\sz^i)\\| &\\leq \\|\\Fnor(\\sz^i) - \\Fnor(\\sz^{t_{\\fk(i)}})\\| + \\| \\Fnor(\\sz^{t_{\\fk(i)}})\\|\\\\ &\\leq  \\Llambda \\|\\sz^i-\\sz^{t_{\\fk(i)}}\\|+ \\| \\Fnor(\\sz^{t_{\\fk(i)}})\\| \\leq (2\\Llambda\\sT+1)\\| \\Fnor(\\sz^{t_{\\fk(i)}})\\| + 2\\Llambda \\scs_{t_{\\fk(i)},i}\\\\\n        &\\leq (2\\Llambda\\sT+1)\\| \\Fnor(\\sz^{t_{\\fk(i)}})\\| + 2\\Llambda \\sr_{\\fk(i)},\n\\end{align*}\n%\nwhere the \\revise{second to last} inequality follows from \\cref{lem:est-err} by setting $m=t_{\\fk(i)}$, $n=t_{\\fk(i)+1}$ (the condition $\\tau_{t_{\\fk(i)},t_{\\fk(i)+1}}\\leq \\sT \\leq 1/(2\\Llambda)$ is due to \\cref{eq:def delta and T}). Squaring both sides of the above inequality and using $(a+b)^2\\leq 2a^2+2b^2$, we obtain \\cref{eq:tech1}.\n","sha256_utf8":"91484ad1a5f072592eca4c48c83e2911f43c6ef8e5dd3cbcac474806012298f3"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"基本目标界化为位移平方界","source_units":[{"start":152797,"end":153390,"start_line":1855,"end_line":1865,"raw_tex":"Based on \\cref{lem:basic bounds} (b) and the definition of $\\su^{m,n}$ (see \\cref{eq:u-def1}), it holds that\n%\n\\[\n    \\psi(\\sx^n) \\leq \\psi(\\sx^m) + (\\sL/2-\\lambda^{-1})\\norm{\\sx^n-\\sx^m}^2 + \\iprod{\\lambda^{-1}(\\sz^n-\\sz^m) + \\Fnor(\\sz^m)}{\\sx^n-\\sx^m}.\n\\]\n%\nApplying the nonexpansiveness of the proximity operator, \\cref{prox-prop}, and $\\iprod{\\Fnor(\\sz^m)}{\\sx^n-\\sx^m} \\leq \\frac{\\lambda}{4} \\norm{\\Fnor(\\sz^m)}^2 + \\lambda^{-1} \\norm{\\sx^n-\\sx^m}^2$, we have\n%\n\\[\n    \\psi(\\sx^n)\\leq  \\psi(\\sx^m) + \\tfrac{\\Llambda}{2} \\norm{\\sz^n-\\sz^m}^2 + \\tfrac{\\lambda}{4} \\norm{\\Fnor(\\sz^m)}^2.\n\\]\n","sha256_utf8":"fc5af4a544feb4ee0685e6db19bab91bc5dba966e21f045e75531478d85fa56f"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"代入窗口内位移估计","source_units":[{"start":153392,"end":153621,"start_line":1867,"end_line":1871,"raw_tex":"Invoking \\Cref{lem:est-err} (for $\\norm{\\sz^n-\\sz^m}$) and $(a+b)^2\\leq 2a^2+2b^2$,  we obtain\n%\n\\[\n    \\psi(\\sx^n)\\leq  \\psi(\\sx^m) + (4\\Llambda\\tau_{m,n}^2+\\tfrac{\\lambda}{4}) \\norm{\\Fnor(\\sz^m)}^2 + 4\\Llambda \\scs_{m,n}^2.\n\\]\n","sha256_utf8":"8efc34b80117034ea0dbcadda9b1e5747416393526b85f9ab7963a8380427670"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"选择端点得到目标值双边界","source_units":[{"start":153623,"end":154068,"start_line":1873,"end_line":1874,"raw_tex":"Then, \\cref{eq:tech2} readily follows by setting $n = i$, $m = t_{\\fk(i)}$ and using $\\tau_{t_{\\fk(i)},i}\\leq \\sT$. Similarly, \\cref{eq:tech3} can be obtained by setting $n = t_{\\fk(i)+1}$, $m = i$ and using  $\\tau_{i,t_{\\fk(i)+1}}\\leq \\sT$ and $\\scs_{i,t_{\\fk(i)+1}} = \\max_{i<j\\leq t_{\\fk(i)+1}}\\|{\\sum}_{\\ell=i}^{\\revise{j}-1} \\alpha_\\ell \\se^\\ell\\| \\leq \\scs_{t_{\\fk(i)},i}+  \\scs_{t_{\\fk(i)},t_{\\fk(i)+1}}  \\leq 2 \\sr_{\\fk(i)}$.\n\\end{proof}","sha256_utf8":"2806afe4ef74b8b282e6f84425e30eac93fc943158264e195412c5cdcf3896e8"}]},{"id":"S055","semantic_type":"theorem","kind":"theorem","title":"驻性与目标值的全局收敛","source_units":[{"start":64359,"end":65000,"start_line":847,"end_line":855,"raw_tex":"\\begin{theorem}[Global convergence]\n\\label{thm:global_convergence-2}\nLet \\ref{A1}--\\ref{A2}, \\ref{B1}--\\ref{B3} hold and let $\\revise{\\{\\sx^k\\}_{k\\in\\N}}$, $\\revise{\\{\\sz^k\\}_{k\\in\\N}}$ be generated by $\\NSGD$. Then, there is $\\spsi^* : \\Omega \\to \\R$ such that: \n%\n \\begin{enumerate}[label=\\textup{\\textrm{(\\alph*)}},leftmargin=3em,topsep=0.5ex,itemsep=.5ex,partopsep=0ex]\n        \\item $\\Fnor(\\sz^k)\\to 0$, $\\|\\partial \\psi(\\sx^k)\\|_{-} \\to 0$, and $\\psi(\\sx^k)\\to \\spsi^*$ a.s..\n\t\\item $\\Exp[\\norm{\\Fnor(\\sz^k)}^2]\\to 0$, $\\Exp[\\|\\partial \\psi(\\sx^k)\\|_{-}^2]\\to0$, and $\\Exp[\\psi(\\sx^k)] \\to \\Exp[\\spsi^*]$.\n\\end{enumerate}\n\\end{theorem}","sha256_utf8":"7fc12b623366288c28c73878a177da01ef099bcdf6c6dc2644052e28ea5d50ec"}]},{"id":"S056","semantic_type":"proof","kind":"proof","title":"全局收敛的完整证明","source_units":[{"start":65566,"end":69594,"start_line":859,"end_line":899,"raw_tex":"\\begin{proof}\nBy \\cref{lem:merit-descent-1}, \\cref{lem:exist time} (b) (with $\\delta = 4/5$; cf$.$ \\cref{eq:def delta and T}), we have \n%\n\\begin{equation}\n    \\label{eq:thm:global-1}\n    \\mer(\\sz^{t_{k+1}}) - \\mer(\\sz^{t_{k}}) \\leq -\\frac{\\xi\\sT}{{10}}\\|\\Fnor(\\sz^{t_k})\\|^2 + \\frac{10}{\\sT}\\cdot\\sr_{k}^2\\quad \\forall \\,k\\in\\N.\n\\end{equation}\n%\nThe condition $\\sum_{k=0}^{\\infty}\\alpha_k^2\\sigma_k^2<\\infty$ and \\cref{lem:err_estimate} imply that ${\\sum}_{k=0}^\\infty \\sr_{k}^2<\\infty$ a$.$s$.$ and ${\\sum}_{k=0}^\\infty\\Exp[{\\sr_{k}^2}]<\\infty$. Setting $\\su_k:=\\frac{10}{\\sT}\\, \\sum_{i=k}^\\infty \\sr_{i}^2$, it follows\n%\n\\begin{equation}\n\t\\label{eq:strong-0}\n\t\\mer(\\sz^{t_{k+1}}) + \\su_{k+1} \\leq  \\mer(\\sz^{t_{k}}) + \\su_{k}- ({\\xi \\sT}/{10}) \\|\\Fnor(\\sz^{t_k})\\|^2.\n\\end{equation}\t\n%\nSumming \\cref{eq:thm:global-1} for $k=0,1,\\ldots$ and using the bound $\\mer(z)\\geq \\bar \\psi$, this yields\n%\n\\begin{equation}\\label{eq:norm-sum}\n{\\xi\\sT}\\,{\\sum}_{k=0}^\\infty \\|\\Fnor(\\sz^{t_k})\\|^2 \\leq 10[\\mer(\\sz^{t_{0}}) + \\su_{0} - \\bar \\psi].\n\\end{equation}\n%\nHence, due to ${\\sum}_{k=0}^\\infty \\sr_{k}^2<\\infty$ a$.$s$.$, we can conclude $\\|\\Fnor(\\sz^{t_k})\\|\\to 0$, $\\sr_{k}\\to0$, and $\\su_{k}\\to0$ a$.$s$.$. Moreover, based on the definition \\cref{index-def} of $\\fk(i)$, we can infer $\\|\\Fnor(\\sz^{t_{\\fk(i)}})\\|\\to 0$ and $\\sr_{\\fk(i)}\\to0$ as $i\\to\\infty$ a$.$s$.$. This proves $\\|\\Fnor(\\sz^{i})\\|\\to 0$ a$.$s$.$ owing to \\cref{eq:tech1}. \n\nTaking expectation in \\cref{eq:norm-sum} and using $\\sum_{k=0}^\\infty\\Exp{[\\sr_{k}^2]}<\\infty$, we further obtain $\\Exp[\\sr_{k}^2]\\to0$, $\\Exp[\\su_{k}^2]\\to0$, and $\\Exp[{\\|\\Fnor(\\sz^{t_k})\\|^2}]\\to 0$. Thus, we can infer $\\Exp[\\|\\Fnor(\\sz^{i})\\|^2]\\to 0$ by taking expectation in \\cref{eq:tech1} and passing $i\\to\\infty$. By \\cref{eq:stat-am}, we have $\\|\\partial \\psi(\\proxl(z))\\|_{-} \\leq \\|\\Fnor(z)\\|$, which implies $\\|\\partial \\psi(\\sx^k)\\|_{-} \\to 0$ a$.$s$.$ and $\\Exp[{\\|\\partial \\psi(\\sx^k)\\|^2_{-}}] \\to 0$.\n\nNext, we show $\\psi(\\sx^{t_k}) \\to \\spsi^*$ a$.$s$.$. By \\cref{eq:strong-0}, $\\{\\mer(\\sz^{t_k})+\\su_k\\}_k$ is monotonically non-increasing, and thus, we conclude that $\\mer(\\sz^{t_k})+\\su_k \\to \\spsi^*$ a$.$s$.$ for some random variable $\\spsi^*$. Using $\\|\\Fnor(\\sz^{t_k})\\|\\to0$, $\\su_k\\to 0$, and $\\psi(\\sx^{t_k}) = \\mer(\\sz^{t_k}) + \\su_k - (\\frac{\\xi\\lambda}{2}\\|\\Fnor(\\sz^{t_k})\\|^2+\\su_k)$, we obtain $\\psi(\\sx^{t_k}) \\to \\spsi^*$ a$.$s$.$. Taking $i\\to \\infty$ in \\cref{eq:tech2} and \\cref{eq:tech3}, it follows %\n\\begin{align*}\n     \\psi(\\sx^i) &\\leq \\psi(\\sx^{t_{\\fk(i)}})+(4\\Llambda\\sT^2+\\lambda/4)\\norm{\\Fnor(\\sz^{t_{\\fk(i)}})}^2 + 4\\Llambda \\sr_{\\fk(i)}^2 \\to \\spsi^*,\\\\\n    \\psi(\\sx^i) &\\geq \\psi(\\sx^{t_{\\fk(i)+1}})-(4\\Llambda\\sT^2+\\lambda/4)\\norm{\\Fnor(\\sz^{t_{\\fk(i)}})}^2 - 16\\Llambda \\sr_{\\fk(i)}^2 \\to \\spsi^*, \\quad \\text{a.s.},\n \\end{align*}\n%\nand hence $\\psi(\\sx^i) \\to \\spsi^*$. Finally, we verify $\\Exp[\\psi(\\sx^{i})] \\to \\Exp[\\spsi^*]$. Since $\\{\\mer(\\sz^{t_k}) + \\su_k\\}_k$ is bounded below by $\\bar\\psi$, non-increasing, and converges to $\\spsi^*$, we can obtain $\\Exp[\\mer(\\sz^{t_k}) + \\su_k]\\to \\Exp[\\spsi^*]$ by the monotone convergence theorem. Using similar arguments as before, this yields $\\Exp[\\psi(\\sx^{t_k})] \\to \\Exp[\\spsi^*]$. Let us now define $\\ssigma_i := \\psi(\\sx^{t_{\\fk(i)}})+(4\\Llambda\\sT^2+\\lambda/4)\\norm{\\Fnor(\\sz^{t_{\\fk(i)}})}^2 + 4\\Llambda \\sr_{\\fk(i)}^2$.\n%\nWe then have $\\psi(\\sx^i) \\leq \\ssigma_i$ (see \\cref{eq:tech2}), $\\ssigma_i\\to \\spsi^*$ a$.$s$.$, and $\\Exp[\\ssigma_i]\\to \\Exp[\\spsi^*]$. Applying Fatou's Lemma for $\\psi(\\sx^i)-\\bar{\\psi}$ and $\\ssigma_i-\\psi(\\sx^i)$, it follows \n%\n \\begin{align*}\n      \\Exp[\\spsi^*]-\\bar{\\psi} &= \\Exp[{{\\lim}_{i\\to\\infty}({\\psi(\\sx^i)-\\bar{\\psi}}})] \\leq {\\liminf}_{i\\to\\infty}{\\Exp{[\\psi(\\sx^i)]}}-\\bar{\\psi}\\quad \n \\end{align*}\n%\nand $0 = \\Exp[{{\\lim}_{i\\to\\infty} (\\ssigma_i-\\psi(\\sx^i))}] \\leq \\Exp[{\\spsi^*}] - {\\limsup}_{i\\to\\infty} {\\Exp{[\\psi(\\sx^i)]}}$. Consequently, we can infer $\\Exp{[\\psi(\\sx^i)]}\\to \\Exp[{\\spsi^*}]$. This finishes the proof. \n\\end{proof}","sha256_utf8":"2c7572f7762a6b9b1ede9757c46f4b3953cad3de2e43ca63969659cc3d815acd"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"时间窗口上的近似下降","source_units":[{"start":65566,"end":65910,"start_line":859,"end_line":865,"raw_tex":"\\begin{proof}\nBy \\cref{lem:merit-descent-1}, \\cref{lem:exist time} (b) (with $\\delta = 4/5$; cf$.$ \\cref{eq:def delta and T}), we have \n%\n\\begin{equation}\n    \\label{eq:thm:global-1}\n    \\mer(\\sz^{t_{k+1}}) - \\mer(\\sz^{t_{k}}) \\leq -\\frac{\\xi\\sT}{{10}}\\|\\Fnor(\\sz^{t_k})\\|^2 + \\frac{10}{\\sT}\\cdot\\sr_{k}^2\\quad \\forall \\,k\\in\\N.\n\\end{equation}\n","sha256_utf8":"b02de46830ffaa684a3e42846c45444034625ea7ff6b64895b0d0c949f6eec8c"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"噪声尾和修正为单调下降","source_units":[{"start":65912,"end":66352,"start_line":867,"end_line":872,"raw_tex":"The condition $\\sum_{k=0}^{\\infty}\\alpha_k^2\\sigma_k^2<\\infty$ and \\cref{lem:err_estimate} imply that ${\\sum}_{k=0}^\\infty \\sr_{k}^2<\\infty$ a$.$s$.$ and ${\\sum}_{k=0}^\\infty\\Exp[{\\sr_{k}^2}]<\\infty$. Setting $\\su_k:=\\frac{10}{\\sT}\\, \\sum_{i=k}^\\infty \\sr_{i}^2$, it follows\n%\n\\begin{equation}\n\t\\label{eq:strong-0}\n\t\\mer(\\sz^{t_{k+1}}) + \\su_{k+1} \\leq  \\mer(\\sz^{t_{k}}) + \\su_{k}- ({\\xi \\sT}/{10}) \\|\\Fnor(\\sz^{t_k})\\|^2.\n\\end{equation}\t\n","sha256_utf8":"88295cf9e8a13304b5f94fd81007a22b03ba0ab036308f0e04adba41aacf5a66"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"求和得到法映射平方可和","source_units":[{"start":66354,"end":66617,"start_line":874,"end_line":878,"raw_tex":"Summing \\cref{eq:thm:global-1} for $k=0,1,\\ldots$ and using the bound $\\mer(z)\\geq \\bar \\psi$, this yields\n%\n\\begin{equation}\\label{eq:norm-sum}\n{\\xi\\sT}\\,{\\sum}_{k=0}^\\infty \\|\\Fnor(\\sz^{t_k})\\|^2 \\leq 10[\\mer(\\sz^{t_{0}}) + \\su_{0} - \\bar \\psi].\n\\end{equation}\n","sha256_utf8":"0046bc35de7e1391ef8d0f6243ed67437b70ee8d5cd4ee79b0fde702bb00511b"}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","title":"驻性几乎必然传递到全序列","source_units":[{"start":66619,"end":67005,"start_line":880,"end_line":880,"raw_tex":"Hence, due to ${\\sum}_{k=0}^\\infty \\sr_{k}^2<\\infty$ a$.$s$.$, we can conclude $\\|\\Fnor(\\sz^{t_k})\\|\\to 0$, $\\sr_{k}\\to0$, and $\\su_{k}\\to0$ a$.$s$.$. Moreover, based on the definition \\cref{index-def} of $\\fk(i)$, we can infer $\\|\\Fnor(\\sz^{t_{\\fk(i)}})\\|\\to 0$ and $\\sr_{\\fk(i)}\\to0$ as $i\\to\\infty$ a$.$s$.$. This proves $\\|\\Fnor(\\sz^{i})\\|\\to 0$ a$.$s$.$ owing to \\cref{eq:tech1}. \n","sha256_utf8":"1c450aa2e41be448ddd6cbff26b303a9e331c1008997d6ff8f67b7f802460166"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"期望驻性与次梯度范数收敛","source_units":[{"start":67006,"end":67525,"start_line":882,"end_line":882,"raw_tex":"Taking expectation in \\cref{eq:norm-sum} and using $\\sum_{k=0}^\\infty\\Exp{[\\sr_{k}^2]}<\\infty$, we further obtain $\\Exp[\\sr_{k}^2]\\to0$, $\\Exp[\\su_{k}^2]\\to0$, and $\\Exp[{\\|\\Fnor(\\sz^{t_k})\\|^2}]\\to 0$. Thus, we can infer $\\Exp[\\|\\Fnor(\\sz^{i})\\|^2]\\to 0$ by taking expectation in \\cref{eq:tech1} and passing $i\\to\\infty$. By \\cref{eq:stat-am}, we have $\\|\\partial \\psi(\\proxl(z))\\|_{-} \\leq \\|\\Fnor(z)\\|$, which implies $\\|\\partial \\psi(\\sx^k)\\|_{-} \\to 0$ a$.$s$.$ and $\\Exp[{\\|\\partial \\psi(\\sx^k)\\|^2_{-}}] \\to 0$.\n","sha256_utf8":"df20ca8437ac4b1e01ed7d0546a50b62531f6f118141dd7f4211c9d31d764203"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"目标值几乎必然收敛","source_units":[{"start":67526,"end":68390,"start_line":884,"end_line":888,"raw_tex":"Next, we show $\\psi(\\sx^{t_k}) \\to \\spsi^*$ a$.$s$.$. By \\cref{eq:strong-0}, $\\{\\mer(\\sz^{t_k})+\\su_k\\}_k$ is monotonically non-increasing, and thus, we conclude that $\\mer(\\sz^{t_k})+\\su_k \\to \\spsi^*$ a$.$s$.$ for some random variable $\\spsi^*$. Using $\\|\\Fnor(\\sz^{t_k})\\|\\to0$, $\\su_k\\to 0$, and $\\psi(\\sx^{t_k}) = \\mer(\\sz^{t_k}) + \\su_k - (\\frac{\\xi\\lambda}{2}\\|\\Fnor(\\sz^{t_k})\\|^2+\\su_k)$, we obtain $\\psi(\\sx^{t_k}) \\to \\spsi^*$ a$.$s$.$. Taking $i\\to \\infty$ in \\cref{eq:tech2} and \\cref{eq:tech3}, it follows %\n\\begin{align*}\n     \\psi(\\sx^i) &\\leq \\psi(\\sx^{t_{\\fk(i)}})+(4\\Llambda\\sT^2+\\lambda/4)\\norm{\\Fnor(\\sz^{t_{\\fk(i)}})}^2 + 4\\Llambda \\sr_{\\fk(i)}^2 \\to \\spsi^*,\\\\\n    \\psi(\\sx^i) &\\geq \\psi(\\sx^{t_{\\fk(i)+1}})-(4\\Llambda\\sT^2+\\lambda/4)\\norm{\\Fnor(\\sz^{t_{\\fk(i)}})}^2 - 16\\Llambda \\sr_{\\fk(i)}^2 \\to \\spsi^*, \\quad \\text{a.s.},\n \\end{align*}\n","sha256_utf8":"1063e5859b85dd9ad126f38b0e961928e929f2d7642d0ed76f7349c4ae1663dd"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"单调收敛与法图引理控制期望","source_units":[{"start":68392,"end":69594,"start_line":890,"end_line":899,"raw_tex":"and hence $\\psi(\\sx^i) \\to \\spsi^*$. Finally, we verify $\\Exp[\\psi(\\sx^{i})] \\to \\Exp[\\spsi^*]$. Since $\\{\\mer(\\sz^{t_k}) + \\su_k\\}_k$ is bounded below by $\\bar\\psi$, non-increasing, and converges to $\\spsi^*$, we can obtain $\\Exp[\\mer(\\sz^{t_k}) + \\su_k]\\to \\Exp[\\spsi^*]$ by the monotone convergence theorem. Using similar arguments as before, this yields $\\Exp[\\psi(\\sx^{t_k})] \\to \\Exp[\\spsi^*]$. Let us now define $\\ssigma_i := \\psi(\\sx^{t_{\\fk(i)}})+(4\\Llambda\\sT^2+\\lambda/4)\\norm{\\Fnor(\\sz^{t_{\\fk(i)}})}^2 + 4\\Llambda \\sr_{\\fk(i)}^2$.\n%\nWe then have $\\psi(\\sx^i) \\leq \\ssigma_i$ (see \\cref{eq:tech2}), $\\ssigma_i\\to \\spsi^*$ a$.$s$.$, and $\\Exp[\\ssigma_i]\\to \\Exp[\\spsi^*]$. Applying Fatou's Lemma for $\\psi(\\sx^i)-\\bar{\\psi}$ and $\\ssigma_i-\\psi(\\sx^i)$, it follows \n%\n \\begin{align*}\n      \\Exp[\\spsi^*]-\\bar{\\psi} &= \\Exp[{{\\lim}_{i\\to\\infty}({\\psi(\\sx^i)-\\bar{\\psi}}})] \\leq {\\liminf}_{i\\to\\infty}{\\Exp{[\\psi(\\sx^i)]}}-\\bar{\\psi}\\quad \n \\end{align*}\n%\nand $0 = \\Exp[{{\\lim}_{i\\to\\infty} (\\ssigma_i-\\psi(\\sx^i))}] \\leq \\Exp[{\\spsi^*}] - {\\limsup}_{i\\to\\infty} {\\Exp{[\\psi(\\sx^i)]}}$. Consequently, we can infer $\\Exp{[\\psi(\\sx^i)]}\\to \\Exp[{\\spsi^*}]$. This finishes the proof. \n\\end{proof}","sha256_utf8":"4ad05ca77fde16439cffff46ca3d1d5b8f46c71f031a9fbd94269f0c182a33db"}]},{"id":"S064","semantic_type":"lemma","kind":"lemma","title":"评价函数的加强型KL不等式","source_units":[{"start":71369,"end":72366,"start_line":916,"end_line":928,"raw_tex":"\\begin{lemma}[Kurdyka-{\\L}ojasiewicz-type inequality]\n\\label{lem:KL-mer}\n    Let \\ref{A3} hold and let $\\bar z$ be given with $\\bar{x}=\\proxl(\\bar{z}) \\in \\crit(\\psi)$. For all $\\theta\\in[\\frac12,1)$, there then are $\\varsigma, \\rho > 0, \\eta\\in(0,1]$ and a continuous, concave function $\\varrho:[0,\\eta) \\to \\R_+$, which is $C^1$ on $(0,\\eta)$, satisfying \n      \\begin{enumerate}[label=\\textup{\\textrm{(\\alph*)}},leftmargin=3em,topsep=1pt,itemsep=.5ex,partopsep=0ex]\n         \\item $1/\\varrho^\\prime(s+t) \\leq 1/\\varrho^\\prime(s) + \\varsigma t^{\\theta}$ for all $s,t >0$ with $s+t<\\eta$,\n         \\item $\\varrho(0) = 0$ and $\\varrho^\\prime(t) \\geq t^{-\\theta}$ for all $t\\in(0,\\eta)$,\n     \\end{enumerate}\n     such that, for all $z\\in\\cB_\\rho(\\bar z) \\cap\\{z:0<|\\mer(z)-\\mer(\\bar z)| + \\frac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2<\\eta\\}$, we have\n     \\begin{equation}\n         \\label{eq:mer-kl}\n         \\varrho^\\prime(|\\mer(z)-\\mer(\\bar z)|) \\cdot  \\|\\Fnor(z)\\|\\geq 1.\n     \\end{equation} \n\\end{lemma}","sha256_utf8":"2fa99b5be46c65628ac911a9c40883d739dd65db3746d8e248df579b118feaf8"}]},{"id":"S065","semantic_type":"proof","kind":"proof","title":"加强型KL不等式的完整证明","source_units":[{"start":72367,"end":76932,"start_line":929,"end_line":958,"raw_tex":"\\begin{proof}\nThe proof is divided into two steps. First, we show that $\\psi$ satisfies the KL property, with the desingularization function $\\varrho$ being concave, differentiable, and fulfilling properties (a) and (b). This step closely follows the proof of \\cite[Lemma 4.11]{josz2024proximal}. Although the differences are minor, we include a complete proof for self-containedness. In the second step, we show that these properties can be transferred to $\\mer$. %\\\\[1mm]\n\n\\noindent\\textbf{Step 1.}  \n%\nApplying \\cite[Corollary 15]{boldanlewshi07} and \\cite[Theorem 4.1]{AttBolRedSou10}, there are $\\rho,\\eta_1 >0$ and a definable, concave, continuous function $\\varrho_1:[0,\\eta_1)\\to\\R_+$, $\\varrho_1 \\in C^1((0,\\eta_1))$, such that \n%\n\\begin{equation}\n\\varrho_1^\\prime(|\\psi(x)-\\psi(\\bar x)|) \\cdot \\|\\partial \\psi(x)\\|_{-}\\geq 1, \\quad \\forall\\; x\\in\\cB_{\\rho}(\\bar x) \\cap\\{x:0<|\\psi(x)-\\psi(\\bar x)|<\\eta_1\\}. \\label{eq:kl-1}\n\\end{equation}\n%\nWe now set $\\varrho(t)=\\int_{0}^t \\varrho^\\prime(\\tau)\\,\\rmd \\tau$, $\\varrho^\\prime(t):=\\max\\{\\varrho_1^\\prime(t),t^{-\\theta}\\}$, and $\\bar \\eta:=\\min\\{\\eta_1,1\\}$. Then, $\\varrho(0)=0$ and \\cref{eq:kl-1} remains to hold with $\\varrho_1$ and $\\eta_1$ being replaced by $\\varrho$ and $\\bar \\eta$, respectively.\nThe function $\\varrho$ is continuous, $C^1$ on $(0,\\bar\\eta)$, and concave as $\\varrho^\\prime$ is non-increasing. \n%\nMoreover, $\\varrho^\\prime$ is definable because $t \\mapsto t^{-\\theta}$ and $\\varrho_1^\\prime$ are definable \\cite[Lemma 6.1]{coste2000introduction}, so is $t\\mapsto 1/\\varrho^\\prime(t)$ \\cite[Lemma 2.3 p.14]{van1998tame} (see also \\cite[Section 4.3]{AttBolRedSou10}). According to \\cite[Theorem 6.4]{coste2000introduction}, the mapping $t\\mapsto 1/\\varrho^\\prime(t)$ is differentiable for all but finitely many points in $(0,\\bar \\eta)$. This result, along with the monotonicity theorem \\cite[(1.2) p.43]{van1998tame}, indicates that $t\\mapsto 1/\\varrho^\\prime(t)$ is differentiable and monotone on the interval $(0,\\eta)$ for some $\\eta\\in(0,\\bar \\eta)$.  \n\nIf $(1/\\varrho^\\prime)^\\prime(t)\\to \\infty$ as $t\\searrow0$, then $(1/\\varrho^\\prime)^\\prime$ is decreasing on $(0,\\eta)$ and $1/\\varrho^\\prime$ is concave. By \\cite[Lemma 3.5]{josz2023convergence}, this yields $1/\\varrho^\\prime(s+t) \\leq 1/\\varrho^\\prime(s) + 1/\\varrho^\\prime(t) \\leq 1/\\varrho^\\prime(s) + t^\\theta$ for all $s,t>0$ and $s+t<\\eta$. Otherwise, we have $\\bar\\varsigma:=\\max_{\\tau\\in(0,\\eta)}(1/\\varrho^\\prime)^\\prime(\\tau) < \\infty$ and by the mean value theorem, it follows $1/\\varrho^\\prime(s+t) - 1/\\varrho^\\prime(s)= (1/\\varrho^\\prime)^\\prime(\\tau) t \\leq \\bar\\varsigma t \\leq \\bar\\varsigma t^{\\theta}$, $\\tau \\in (0,\\eta)$, where we used $\\theta < 1$ and $t \\in (0,1)$. Note that $\\bar\\varsigma$ is nonnegative since $(1/\\varrho^\\prime)^\\prime(\\tau) = -\\varrho^{\\prime\\prime}(\\tau)/(\\varrho^\\prime(\\tau))^2 \\geq 0$ (by the concavity of $\\varrho$). We may now set $\\varsigma := \\max\\{\\bar\\varsigma,1\\}$. %\\\\[1mm]\n\n\\noindent\\textbf{Step 2:} \\emph{Property \\cref{eq:mer-kl} for $\\mer$.} Let $z\\in\\cB_\\rho(\\bar z) \\cap\\{z:0<|\\mer(z)-\\mer(\\bar z)|+ \\frac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2<\\eta\\}$ be arbitrary. Setting $x=\\proxl(z)$ and using the nonexpansiveness of $\\proxl$, we have $\\|x-\\bar x\\| \\leq \\|z-\\bar z\\| \\leq \\rho$ and $|\\psi(x)-\\psi(\\bar x)| \\leq |\\mer(z)-\\mer(\\bar z)|+ \\frac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2<\\eta$.\n\nThus, if $x$ satisfies $\\psi(x) \\neq \\psi(\\bar x)$, then \\textbf{Step 1} implies $\\varrho^\\prime(|\\psi(x)-\\psi(\\bar x)|) \\|\\partial \\psi(x)\\|_{-}\\geq 1$. Furthermore, since $1/\\varrho^\\prime$ is non-decreasing and applying the bound (a), we can infer\n%\n\\begin{align*}\n          1/\\varrho^\\prime(|\\mer(z)-\\mer(\\bar z)|) \n          &\\leq  1/\\varrho^\\prime(|\\psi(x)-\\psi(\\bar x)| + \\tfrac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2)\\\\\n          &\\leq 1/\\varrho^\\prime(|\\psi(x)-\\psi(\\bar x)|) + \\varsigma (\\tfrac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2)^\\theta \\\\ &\\leq \\|\\partial \\psi(x)\\|_{-} + \\varsigma \\sqrt{\\xi\\lambda/2}\\|\\Fnor(z)\\| \\leq (1+\\varsigma \\sqrt{\\xi\\lambda/2})\\|\\Fnor(z)\\|.\n\\end{align*}\n%\nHere, the third inequality is due to $\\tfrac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2<1$ and $\\theta\\in[\\frac12,1)$, and the last step follows from $\\Fnor(z)\\in\\partial \\psi(x)$. If $\\psi(x) = \\psi(\\bar x)$, then we obtain $1/\\varrho^\\prime(|H(z)-H(\\bar z)|) \\leq (\\tfrac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2)^\\theta \\leq \\sqrt{\\xi\\lambda/2}\\|\\Fnor(z)\\|$ by (b). We can conclude the proof after rescaling $\\varrho$ to $(1+\\max\\{1,\\varsigma\\} \\sqrt{\\xi\\lambda/2}) \\varrho$ (this does not affect the properties (a)--(b)).\n\\end{proof}","sha256_utf8":"1bb2303942eb99d0248f36ef8c4c1708160155f93e5411ae842de379dcb7e8a6"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"调用可定义目标的局部KL性质","source_units":[{"start":72367,"end":73316,"start_line":929,"end_line":938,"raw_tex":"\\begin{proof}\nThe proof is divided into two steps. First, we show that $\\psi$ satisfies the KL property, with the desingularization function $\\varrho$ being concave, differentiable, and fulfilling properties (a) and (b). This step closely follows the proof of \\cite[Lemma 4.11]{josz2024proximal}. Although the differences are minor, we include a complete proof for self-containedness. In the second step, we show that these properties can be transferred to $\\mer$. %\\\\[1mm]\n\n\\noindent\\textbf{Step 1.}  \n%\nApplying \\cite[Corollary 15]{boldanlewshi07} and \\cite[Theorem 4.1]{AttBolRedSou10}, there are $\\rho,\\eta_1 >0$ and a definable, concave, continuous function $\\varrho_1:[0,\\eta_1)\\to\\R_+$, $\\varrho_1 \\in C^1((0,\\eta_1))$, such that \n%\n\\begin{equation}\n\\varrho_1^\\prime(|\\psi(x)-\\psi(\\bar x)|) \\cdot \\|\\partial \\psi(x)\\|_{-}\\geq 1, \\quad \\forall\\; x\\in\\cB_{\\rho}(\\bar x) \\cap\\{x:0<|\\psi(x)-\\psi(\\bar x)|<\\eta_1\\}. \\label{eq:kl-1}\n\\end{equation}\n","sha256_utf8":"840bfdcb3edc88b43d4e27f261113c3695f999ffb6940bc8ec3a92b12cf902db"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"构造导数具有幂次下界的函数","source_units":[{"start":73318,"end":73743,"start_line":940,"end_line":941,"raw_tex":"We now set $\\varrho(t)=\\int_{0}^t \\varrho^\\prime(\\tau)\\,\\rmd \\tau$, $\\varrho^\\prime(t):=\\max\\{\\varrho_1^\\prime(t),t^{-\\theta}\\}$, and $\\bar \\eta:=\\min\\{\\eta_1,1\\}$. Then, $\\varrho(0)=0$ and \\cref{eq:kl-1} remains to hold with $\\varrho_1$ and $\\eta_1$ being replaced by $\\varrho$ and $\\bar \\eta$, respectively.\nThe function $\\varrho$ is continuous, $C^1$ on $(0,\\bar\\eta)$, and concave as $\\varrho^\\prime$ is non-increasing. \n","sha256_utf8":"2674d09e0ac9f008d315067e4179036d3c889c5176bf89b4840330bf0751264b"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"可定义性保证局部光滑与单调","source_units":[{"start":73745,"end":74404,"start_line":943,"end_line":943,"raw_tex":"Moreover, $\\varrho^\\prime$ is definable because $t \\mapsto t^{-\\theta}$ and $\\varrho_1^\\prime$ are definable \\cite[Lemma 6.1]{coste2000introduction}, so is $t\\mapsto 1/\\varrho^\\prime(t)$ \\cite[Lemma 2.3 p.14]{van1998tame} (see also \\cite[Section 4.3]{AttBolRedSou10}). According to \\cite[Theorem 6.4]{coste2000introduction}, the mapping $t\\mapsto 1/\\varrho^\\prime(t)$ is differentiable for all but finitely many points in $(0,\\bar \\eta)$. This result, along with the monotonicity theorem \\cite[(1.2) p.43]{van1998tame}, indicates that $t\\mapsto 1/\\varrho^\\prime(t)$ is differentiable and monotone on the interval $(0,\\eta)$ for some $\\eta\\in(0,\\bar \\eta)$.  \n","sha256_utf8":"01aa8d78894e161caaad72d28db531203ac96edae68e1ac62062e1bde1b7fbc3"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"分情形证明倒数导数增量界","source_units":[{"start":74405,"end":75338,"start_line":945,"end_line":945,"raw_tex":"If $(1/\\varrho^\\prime)^\\prime(t)\\to \\infty$ as $t\\searrow0$, then $(1/\\varrho^\\prime)^\\prime$ is decreasing on $(0,\\eta)$ and $1/\\varrho^\\prime$ is concave. By \\cite[Lemma 3.5]{josz2023convergence}, this yields $1/\\varrho^\\prime(s+t) \\leq 1/\\varrho^\\prime(s) + 1/\\varrho^\\prime(t) \\leq 1/\\varrho^\\prime(s) + t^\\theta$ for all $s,t>0$ and $s+t<\\eta$. Otherwise, we have $\\bar\\varsigma:=\\max_{\\tau\\in(0,\\eta)}(1/\\varrho^\\prime)^\\prime(\\tau) < \\infty$ and by the mean value theorem, it follows $1/\\varrho^\\prime(s+t) - 1/\\varrho^\\prime(s)= (1/\\varrho^\\prime)^\\prime(\\tau) t \\leq \\bar\\varsigma t \\leq \\bar\\varsigma t^{\\theta}$, $\\tau \\in (0,\\eta)$, where we used $\\theta < 1$ and $t \\in (0,1)$. Note that $\\bar\\varsigma$ is nonnegative since $(1/\\varrho^\\prime)^\\prime(\\tau) = -\\varrho^{\\prime\\prime}(\\tau)/(\\varrho^\\prime(\\tau))^2 \\geq 0$ (by the concavity of $\\varrho$). We may now set $\\varsigma := \\max\\{\\bar\\varsigma,1\\}$. %\\\\[1mm]\n","sha256_utf8":"c5891cb4d0108a535f0899b0f5bccaf3de95717d0c3b7ea78b45e5170f561a16"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"邻近映射传递局部适用区域","source_units":[{"start":75339,"end":75743,"start_line":947,"end_line":947,"raw_tex":"\\noindent\\textbf{Step 2:} \\emph{Property \\cref{eq:mer-kl} for $\\mer$.} Let $z\\in\\cB_\\rho(\\bar z) \\cap\\{z:0<|\\mer(z)-\\mer(\\bar z)|+ \\frac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2<\\eta\\}$ be arbitrary. Setting $x=\\proxl(z)$ and using the nonexpansiveness of $\\proxl$, we have $\\|x-\\bar x\\| \\leq \\|z-\\bar z\\| \\leq \\rho$ and $|\\psi(x)-\\psi(\\bar x)| \\leq |\\mer(z)-\\mer(\\bar z)|+ \\frac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2<\\eta$.\n","sha256_utf8":"3b945a2bc31cfa4aaa1199cf4ab558667c017066c12316112f321c79bc5ae07e"}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","title":"将目标KL界转移到评价函数","source_units":[{"start":75744,"end":76419,"start_line":949,"end_line":955,"raw_tex":"Thus, if $x$ satisfies $\\psi(x) \\neq \\psi(\\bar x)$, then \\textbf{Step 1} implies $\\varrho^\\prime(|\\psi(x)-\\psi(\\bar x)|) \\|\\partial \\psi(x)\\|_{-}\\geq 1$. Furthermore, since $1/\\varrho^\\prime$ is non-decreasing and applying the bound (a), we can infer\n%\n\\begin{align*}\n          1/\\varrho^\\prime(|\\mer(z)-\\mer(\\bar z)|) \n          &\\leq  1/\\varrho^\\prime(|\\psi(x)-\\psi(\\bar x)| + \\tfrac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2)\\\\\n          &\\leq 1/\\varrho^\\prime(|\\psi(x)-\\psi(\\bar x)|) + \\varsigma (\\tfrac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2)^\\theta \\\\ &\\leq \\|\\partial \\psi(x)\\|_{-} + \\varsigma \\sqrt{\\xi\\lambda/2}\\|\\Fnor(z)\\| \\leq (1+\\varsigma \\sqrt{\\xi\\lambda/2})\\|\\Fnor(z)\\|.\n\\end{align*}\n","sha256_utf8":"2c5e258d7b6ae6b5c070d991c1b3df4284285e1c5052ea7fbc3d0496f842fb3e"}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","title":"处理等值情形并缩放去奇异化函数","source_units":[{"start":76421,"end":76932,"start_line":957,"end_line":958,"raw_tex":"Here, the third inequality is due to $\\tfrac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2<1$ and $\\theta\\in[\\frac12,1)$, and the last step follows from $\\Fnor(z)\\in\\partial \\psi(x)$. If $\\psi(x) = \\psi(\\bar x)$, then we obtain $1/\\varrho^\\prime(|H(z)-H(\\bar z)|) \\leq (\\tfrac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2)^\\theta \\leq \\sqrt{\\xi\\lambda/2}\\|\\Fnor(z)\\|$ by (b). We can conclude the proof after rescaling $\\varrho$ to $(1+\\max\\{1,\\varsigma\\} \\sqrt{\\xi\\lambda/2}) \\varrho$ (this does not affect the properties (a)--(b)).\n\\end{proof}","sha256_utf8":"9f03ed8e337a5231a0c67782dd11ec04c6aded638741cd0463f16d8b3d13463b"}]},{"id":"S073","semantic_type":"lemma","kind":"lemma","title":"驻点子序列与极限值一致","source_units":[{"start":82190,"end":82776,"start_line":1023,"end_line":1026,"raw_tex":"\\begin{lemma}[Subsequence convergence]\\label{lem:subsequence converge}\nLet \\ref{A1}--\\ref{A2} and \\ref{B1}--\\ref{B4} hold and let $\\omega\\in\\mathcal C$ be arbitrary. Then, there exist $z^*\\in\\Rn$ and a subsequence $\\revise{\\{q_k\\}_{k\\in\\N}}\\subseteq \\N$ such that $x^*:=\\proxl(z^*)\\in\\crit(\\psi)$, $x^{q_k} \\to x^*$, $z^{q_k} \\to z^*$, $\\psi(x^k) \\to \\psi(x^*) = \\psi^*$, and $\\mer(z^{k}) \\to \\mer(z^*) = \\psi^*$ as $k\\to\\infty$, where  \n$\\revise{\\{x^k\\}_{k\\in\\N}} = \\revise{\\{\\sx^k(\\omega)\\}_{k\\in\\N}}$ and $\\revise{\\{z^k\\}_{k\\in\\N}}= \\revise{\\{\\sz^k(\\omega)\\}_{k\\in\\N}}$. \n\\end{lemma}","sha256_utf8":"3ff07acb534f8e78ccbae0baf527cf19eaf706a240dc70c84ddf7bd68b7ac3b0"}]},{"id":"S074","semantic_type":"proof","kind":"proof","title":"驻点子序列的完整证明","source_units":[{"start":154201,"end":155480,"start_line":1877,"end_line":1883,"raw_tex":"\\begin{proof}\nLet $\\omega\\in\\mathcal C$ be arbitrary and let $x^k = \\sx^k(\\omega)$,  $z^k = \\sz^k(\\omega)$, etc$.$ be the corresponding realizations. By the definition of the event $\\mathcal C$, there exists $\\revise{\\{p_k\\}_{k\\in\\N}}\\subseteq\\N$ such that \\revise{$\\{x^{p_k}\\}_{k\\in\\N}$} is bounded and we have $\\Fnor(z^k) \\to 0$, $\\psi(x^k) \\to \\psi^*$. Using the definition of the normal map, we can infer $\\|z^{p_k}\\| \\leq \\lambda(\\|\\Fnor(z^{p_k})\\|+\\|\\nabla f(x^{p_k})\\|)+\\|x^{p_k}\\|$. Hence, \\revise{$\\{z^{p_k}\\}_{k\\in\\N}$} is also bounded and there is a convergent subsequence \\revise{$\\{z^{q_k}\\}_{k\\in\\N}$} of \\revise{$\\{z^{p_k}\\}_{k\\in\\N}$} such that $\\lim_{k\\to\\infty} z^{q_k} = z^*$ for some $z^*$ satisfying $\\|\\Fnor(z^{*})\\|=0$ and $x^*:=\\proxl(z^*) \\in \\crit(\\psi)$. Due to $\\|x^k-x^*\\| \\leq \\|z^k - z^*\\|$, we then have $\\lim_{k\\to\\infty} x^{q_k}=x^*$. Using the continuity of the Moreau envelope, this yields\n%\n\\begin{align*} \\psi^* & = {\\lim}_{k \\to \\infty}\\,\\psi(x^{q_k})  = {\\lim}_{k \\to \\infty}\\,\\vp(\\proxi{\\lambda}(z^{q_k})) + f(x^{q_k}) \\\\ & = {\\lim}_{k\\to\\infty}\\,\\envi{\\lambda}(z^{q_k}) - \\tfrac{1}{2\\lambda} \\|z^{q_k}- x^{q_k}\\|^2 + f(x^*) = \\psi(x^*). \\end{align*}\n%\nThis readily implies $H(z^k) \\to \\psi^* = H(z^*)$ and finishes the proof.\n\\end{proof}","sha256_utf8":"3ee0323b4adf7e8f478ca3f089c2b505b67326c32465e756d74ed0d7ecdd28c1"}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","title":"有界子序列提升为法映射零点","source_units":[{"start":154201,"end":155127,"start_line":1877,"end_line":1878,"raw_tex":"\\begin{proof}\nLet $\\omega\\in\\mathcal C$ be arbitrary and let $x^k = \\sx^k(\\omega)$,  $z^k = \\sz^k(\\omega)$, etc$.$ be the corresponding realizations. By the definition of the event $\\mathcal C$, there exists $\\revise{\\{p_k\\}_{k\\in\\N}}\\subseteq\\N$ such that \\revise{$\\{x^{p_k}\\}_{k\\in\\N}$} is bounded and we have $\\Fnor(z^k) \\to 0$, $\\psi(x^k) \\to \\psi^*$. Using the definition of the normal map, we can infer $\\|z^{p_k}\\| \\leq \\lambda(\\|\\Fnor(z^{p_k})\\|+\\|\\nabla f(x^{p_k})\\|)+\\|x^{p_k}\\|$. Hence, \\revise{$\\{z^{p_k}\\}_{k\\in\\N}$} is also bounded and there is a convergent subsequence \\revise{$\\{z^{q_k}\\}_{k\\in\\N}$} of \\revise{$\\{z^{p_k}\\}_{k\\in\\N}$} such that $\\lim_{k\\to\\infty} z^{q_k} = z^*$ for some $z^*$ satisfying $\\|\\Fnor(z^{*})\\|=0$ and $x^*:=\\proxl(z^*) \\in \\crit(\\psi)$. Due to $\\|x^k-x^*\\| \\leq \\|z^k - z^*\\|$, we then have $\\lim_{k\\to\\infty} x^{q_k}=x^*$. Using the continuity of the Moreau envelope, this yields\n","sha256_utf8":"983976fd5ba6a8035939c3ab950c422a3218f80e700f03b629c27fb93430ca45"}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","title":"包络连续性确定目标极限","source_units":[{"start":155129,"end":155480,"start_line":1880,"end_line":1883,"raw_tex":"\\begin{align*} \\psi^* & = {\\lim}_{k \\to \\infty}\\,\\psi(x^{q_k})  = {\\lim}_{k \\to \\infty}\\,\\vp(\\proxi{\\lambda}(z^{q_k})) + f(x^{q_k}) \\\\ & = {\\lim}_{k\\to\\infty}\\,\\envi{\\lambda}(z^{q_k}) - \\tfrac{1}{2\\lambda} \\|z^{q_k}- x^{q_k}\\|^2 + f(x^*) = \\psi(x^*). \\end{align*}\n%\nThis readily implies $H(z^k) \\to \\psi^* = H(z^*)$ and finishes the proof.\n\\end{proof}","sha256_utf8":"57bd96b950fd420b2f07cc43249ae3f897659361c940a1465f148f2b46f7e933"}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","title":"加权噪声控制修正尾和","source_units":[{"start":79828,"end":81008,"start_line":997,"end_line":1011,"raw_tex":"\\subsubsection{\\texorpdfstring{Proof of \\cref{thm:iter converge}}{Proof of Theorem 4.2}}\n\nWe start the proof by deriving certain summability results for the error terms $\\su_k$ and $\\sr_k$. %\\\\[1mm]\n\n\\noindent\\textbf{Step 1:} \\emph{Summability of error terms.} Due to condition \\cref{eq:kl-step} (i.e., $\\sum_{k=0}^\\infty \\alpha_k^2 \\tau_{0,k}^{\\mu}\\sigma_k^2  <\\infty$) and \\cref{lem:err_estimate}, we have $\\bar \\sr:=\\sum_{k=0}^\\infty \\tau_{0,t_k}^{\\mu}\\sr_k^2<\\infty$ a$.$s$.$. This yields \\vspace{-0.5ex}\n\\[\n\\sT\\su_k = {10}\\, {\\sum}_{i=k}^\\infty \\sr_{i}^2 =  {10}\\, {\\sum}_{i=k}^\\infty \\tau_{0,t_i}^{-\\mu} \\tau_{0,t_i}^{\\mu} \\sr_{i}^2 \\leq {10} \\tau_{0,t_k}^{-\\mu} \\cdot {\\sum}_{i=k}^\\infty  \\tau_{0,t_i}^{\\mu} \\sr_{i}^2 \\leq {10\\bar \\sr}  \\cdot \\tau_{0,t_k}^{-\\mu}.\n\\]\n%\nUsing $\\delta \\sT \\leq \\tau_{t_k,t_{k+1}}$, it follows $\\tau_{0,t_k} \\geq  \\tau_{0,t_0} + \\delta \\sT k$ and $\\su_k\\to 0$ a$.$s$.$, which implies \n%\n\\begin{equation}\n    \\label{eq:summability u_k}\n    {\\sum}_{k=0}^\\infty \\su_k^\\theta \\leq (10\\bar\\sr/\\sT)^\\theta  \\cdot {\\sum}_{k=0}^\\infty (\\tau_{0,t_0} + \\delta \\sT k)^{-\\mu\\theta}<\\infty \\quad \\text{for all}\\quad \\theta > \\tfrac{1}{\\mu}.\n\\end{equation}\n","sha256_utf8":"0ec1eecf7153901f081350bd8e516645041227c9953d9166d846753e860c0650"}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","title":"柯西施瓦茨推出噪声范数可和","source_units":[{"start":81010,"end":81449,"start_line":1013,"end_line":1017,"raw_tex":"Hence, we have $\\sum_{k=j}^\\infty \\su_k^\\theta \\to 0$ as $j\\to\\infty$ for all $\\theta > {\\mu}^{-1}$. Moreover, it holds that \\vspace{-1ex}\n%\n\\begin{equation}\\label{eq:rk-sum}\n{\\sum}_{k=0}^\\infty \\sr_k = {\\sum}_{k=0}^\\infty \\tau_{0,t_k}^{-\\frac{\\mu}{2}}\\cdot \\tau_{0,t_k}^{\\frac{\\mu}{2}} \\sr_k \\leq \\Big({ \\bar{\\sr} \\cdot {\\sum}_{k=0}^\\infty (\\tau_{0,t_0} + \\delta \\sT  k)^{-\\mu} }\\Big)^{1/2} < \\infty \\quad \\text{a$.$s$.$.}\n\\end{equation}\n","sha256_utf8":"2d16214eeb734161f432d967336f5d01331bcce87c6d769bd676424b729314a1"}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","title":"确定共同满概率分析事件","source_units":[{"start":81451,"end":82188,"start_line":1019,"end_line":1021,"raw_tex":"Let us fix $\\theta\\in(\\max\\{\\frac{1}{\\mu},\\frac12\\},1)$ and let us introduce the events $\\mathcal E := \\{\\omega : \\sum_{k=0}^\\infty \\su_k(\\omega)^\\theta < \\infty, \\, \\sum_{k=0}^\\infty \\sr_k(\\omega) < \\infty\\}$ and $\\mathcal C := \\mathcal X \\cap \\{\\omega : \\Fnor(\\sz^k(\\omega)) \\to 0, \\, \\psi(\\sx^k(\\omega))\\to\\spsi^*(\\omega)\\}$. Condition \\cref{eq:kl-step} implies that \\ref{B3} is satisfied. Thus, by \\cref{thm:global_convergence-2}, \\cref{eq:summability u_k}, \\cref{eq:rk-sum}, and \\ref{B4}, we can infer $\\Prob(\\mathcal C) = \\Prob(\\mathcal E) = \\Prob(\\mathcal D)=1$, where $\\mathcal D = \\mathcal C \\cap \\mathcal E$. %\\\\[1mm]\n\n\\noindent\\textbf{Step 2:} \\emph{Subsequence convergence and roadmap.} We first present an auxiliary result.\n","sha256_utf8":"af9af8c6186dc5885f8389539f2fe46c261f95b607fbb7002df4d488804cad55"}]},{"id":"S080","semantic_type":"proof_step","kind":"proof_step","title":"定义窗口最大位移并估计","source_units":[{"start":82778,"end":83232,"start_line":1028,"end_line":1037,"raw_tex":"The proof of \\cref{lem:subsequence converge} can be found in \\cref{proof:lem:subsequence converge}. To simplify the notation, we set\n%\n$\\sd_k := {\\max}_{t_k<i\\leq t_{k+1}} \\max\\{\\|\\sx^i-\\sx^{t_k}\\|,\\|\\sz^i-\\sz^{t_k}\\|\\}$.\n\n\\cref{lem:est-err} and $\\tau_{t_k,t_{k+1}} \\leq \\sT$ then imply \n%\n\\begin{equation}\\label{dk-bound}\n    % \\sd_k^2 \\leq 4 \\sT^2 \\|\\Fnor(\\sz^{t_k})\\|^2 + 4 \\sr_k^2.\n    \\sd_k \\leq 2 \\sT \\|\\Fnor(\\sz^{t_k})\\| + 2 \\sr_k.\n\\end{equation}\n","sha256_utf8":"764e21569b8f2cd7478fa0c005cb3c5edb09bddb7d2922b179ae202bfc5e6985"}]},{"id":"S081","semantic_type":"proof_step","kind":"proof_step","title":"将收敛归结为窗口位移可和","source_units":[{"start":83234,"end":84515,"start_line":1039,"end_line":1047,"raw_tex":"Our goal is to show $\\sx^k \\to \\sx^*$ a$.$s$.$. Thanks to $\\Prob(\\mathcal D)=1$, it suffices to prove that $\\revise{\\{\\sx^k(\\omega)\\}_{k\\in\\N}}$ is a Cauchy sequence for all $\\omega \\in \\mathcal D$. Let $\\omega \\in \\mathcal D$ be arbitrary and let $\\revise{\\{x^k\\}_{k\\in\\N}} = \\revise{\\{\\sx^k(\\omega)\\}_{k\\in\\N}}$,  $\\revise{\\{z^k\\}_{k\\in\\N}} = \\revise{\\{\\sz^k(\\omega)\\}_{k\\in\\N}}$, $\\revise{\\{r_k\\}_{k\\in\\N}} = \\revise{\\{\\sr_k(\\omega)\\}_{k\\in\\N}}$, etc. denote the corresponding realizations. For given $\\varepsilon>0$, we need to show that there is $k$ such that $\\|x^m - x^n\\|<\\varepsilon$ for all $k\\leq m<n$. \n\nWe claim that it suffices to establish ${\\sum}_{k=0}^{\\infty}\\,d_k< \\infty$. In this case, there is $k^\\prime\\in\\N$ such that ${\\sum}_{j=k}^{\\infty}\\,d_j<\\varepsilon/2$ for all $k \\geq k^\\prime$. For any $n > m > t_{k^\\prime}$, we have $k^\\prime \\leq \\fk(m)\\leq \\fk(n)$. Therefore, using the triangle inequality and the above bound, we obtain %\\vspace{-1.5ex}\n%\n\\begin{equation*}\n\\|x^m - x^n\\| \\leq d_{\\fk(m)} + d_{\\fk(n)} + {\\sum}_{j=\\fk(m)}^{{\\fk(n)-1}}\\|x^{t_{j+1}}-x^{t_j}\\| \\leq 2 {\\sum}_{j=\\fk(m)}^{{\\fk(n)}} d_j< \\varepsilon, \n\\end{equation*}\n%\nwhich proves our claim. In the remaining steps, we establish summability of $\\revise{\\{d_k\\}_{k\\in\\N}}$.%\\\\[1mm]\n","sha256_utf8":"6e13b9f2081bc60c21fc71225705912ea6de5cd654db8c85cfb4f56c4b77671b"}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","title":"窗口端点子序列进入驻点邻域","source_units":[{"start":84516,"end":85198,"start_line":1049,"end_line":1049,"raw_tex":"\\noindent\\textbf{Step 3:} \\emph{Entering the local region and a KL-based bound for $\\revise{\\{d_k\\}_{k\\in\\N}}$.} By \\cref{lem:subsequence converge}, there exists a subsequence $\\revise{\\{q_k\\}_{k\\in\\N}}$ such that $x^{q_k} \\to x^*$, $z^{q_k} \\to z^*$ for some $z^*$ with $x^* = \\proxl(z^*) \\in\\crit(\\psi)$, and $\\mer(z^{k}) \\to \\mer(z^*) = \\psi^*$. Based on the definition of $\\fk$ and $d_k$, it holds that $\\max\\{\\|x^{q_k} - x^{t_{\\fk(q_k)}}\\|,\\|z^{q_k} - z^{t_{\\fk(q_k)}}\\|\\} \\leq d_{\\fk(q_k)}$. Due to $\\|\\Fnor(z^{t_{{\\fk(q_k)}}})\\|\\to0$, $r_{{\\fk(q_k)}}\\to0$, and \\cref{dk-bound}, we then have $d_{{\\fk(q_k)}} \\to 0$, $x^{t_{\\fk(q_k)}} \\to x^*$, and $z^{t_{\\fk(q_k)}} \\to z^*$.\n","sha256_utf8":"6a9b7c038b6ef585dd459274eb85477ba6c1c9564e6e9d9c4a0f787965db4710"}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","title":"在极限驻点处应用加强KL界","source_units":[{"start":85200,"end":85674,"start_line":1052,"end_line":1059,"raw_tex":"Applying \\cref{lem:KL-mer}, there exist $\\rho>0$, $\\eta\\in(0,1]$, and a concave function $\\varrho:[0,\\eta)\\to\\R_+$, $\\varrho \\in C^1((0,\\eta))$ (satisfying properties (a)--(b) in \\cref{lem:KL-mer}) such that\n%\n\\begin{equation}\n         \\label{eq:super-proof-kl}\n         \\varrho^\\prime(|\\mer(z)-\\psi^*|) \\cdot  \\|\\Fnor(z)\\|\\geq 1\\quad %\\text{holds}\n\\end{equation}\n%\nfor all $z\\in\\cB_\\rho(z^*) \\cap\\{z \\in \\Rn:0<|\\mer(z)-\\psi^*| + \\frac{\\xi\\lambda}{2}\\|\\Fnor(z)\\|^2<\\eta\\}$.\n","sha256_utf8":"6daaa77bcc55e291306c85fe1da732b2fb3c669eb9417a692a581dd2eac789dc"}]},{"id":"S084","semantic_type":"proof_step","kind":"proof_step","title":"选择局部区域与足够晚指标","source_units":[{"start":85679,"end":86519,"start_line":1061,"end_line":1069,"raw_tex":"Next, we define $\\Delta_k:= \\varrho(\\mer(z^{t_k}) -\\psi^* + u_k)$.\nSince \\revise{$\\{\\mer(z^{t_k}) -\\psi^* + u_k\\}_{k \\in \\N}$} is monotonically decreasing (see \\cref{eq:strong-0}) with $\\mer(z^{t_k}) + u_k \\to \\psi^*$, $\\Delta_k$ is well-defined. By $\\omega \\in \\mathcal D$, $\\varrho(0) = 0$, and by the previous discussion, there then is $\\hat k \\in \\N$ such that\n%\n\\begin{subequations}\n\\label{eq:choose rho eta}\n    \\begin{align}\n        \\label{eq:choose rho eta-1} &\\|z^{t_{\\hat k}} - z^*\\| + 20\\xi^{-1}\\Delta_{\\hat{k}}  + 2\\varsigma\\sT {\\sum}_{i=\\hat k}^\\infty\\, u_i^\\theta + 2{\\sum}_{i=\\hat k}^\\infty\\, r_i< \\rho \\quad \\text{and} \\quad \\\\ \\label{eq:choose rho eta-2} &|\\mer(z^{t_k})-\\psi^*| + \\max\\{\\tfrac{\\xi\\lambda}{2}\\|\\Fnor(z^{t_k})\\|^2, u_k\\} \\leq  \\eta\\quad\\text{for all} \\quad k \\geq \\hat k. \n    \\end{align}\n\\end{subequations}\n","sha256_utf8":"b5ef5cb8f43d4cf108055bc5530f8965d92a1cbec418f081cedda6769a507b34"}]},{"id":"S085","semantic_type":"proof_step","kind":"proof_step","title":"KL与修正下降给出位移望远镜界","source_units":[{"start":86520,"end":88275,"start_line":1071,"end_line":1089,"raw_tex":"Let $k \\geq \\hat k$ be now given with $z^{t_k} \\in \\mathcal B_\\rho(z^*)$. (In particular, this is true for $k = \\hat k$). Hence, by \\cref{eq:choose rho eta-2}, the KL inequality \\cref{eq:super-proof-kl} is applicable and we have\n%\n\\begin{subequations}\n    \\begin{align}\n\t\\label{eq:long-equation-1}\\Delta_k - \\Delta_{k+1}\n    & \\geq \\varrho^\\prime(\\mer(z^{t_k}) - \\psi^*+u_k)(\\mer(z^{t_k}) +u_k - [\\mer(z^{t_{k+1}})+u_{k+1}]) \\\\\n\t\\label{eq:long-equation-2}& \\geq \\varrho^\\prime(|\\mer(z^{t_k}) - \\psi^*|+u_k) (\\xi \\sT \\|\\Fnor(z^{t_k})\\|^2/10) \\\\%[1mm]\n\t\\label{eq:long-equation-3}&\\geq (1/\\varrho^\\prime({|\\mer(z^{t_k}) - \\psi^*|})  + \\varsigma u_k^\\theta)^{-1} (\\xi \\sT \\|\\Fnor(z^{t_k})\\|^2/10) \\\\\n    \\label{eq:long-equation-4}&\\geq \\frac{\\xi\\sT}{10}\\cdot \\frac{\\|\\Fnor(z^{t_k})\\|^2}{\\|\\Fnor(z^{t_k})\\| +  \\varsigma u_k^\\theta}\\geq \\frac{\\xi \\sT}{10}\\srt{\\|\\Fnor(z^{t_k})\\| -  \\varsigma \n u_k^\\theta}.\n\\end{align}\n\\end{subequations}\n%\nThe inequality \\cref{eq:long-equation-1} is due to the concavity of $\\varrho$, \\cref{eq:long-equation-2} follows from the descent-type property \\cref{eq:strong-0} and the fact that $\\varrho^\\prime$ is non-increasing, \\cref{eq:long-equation-3} utilizes $\\varrho^\\prime(s+t) \\geq [1/\\varrho^\\prime(s) + \\varsigma t^{\\theta}]^{-1}$ (see \\cref{lem:KL-mer} (a)), and \\cref{eq:long-equation-4} is based on \\cref{eq:super-proof-kl} and ${a^2}\\geq a^2-b^2$, $a,b\\in \\R$. Rearranging yields $\\sT\\|\\Fnor(z^{t_k})\\| \\leq 10\\xi^{-1}(\\Delta_k - \\Delta_{k+1}) + \\varsigma \\sT u_k^\\theta$. Combining this estimate with $d_k \\leq 2 \\sT \\|\\Fnor(z^{t_k})\\| + 2 r_k$ (see \\cref{dk-bound}), we can infer\n%\n\\begin{equation}\n    \\label{eq:d bound}\nd_k \\leq 20\\xi^{-1}(\\Delta_k - \\Delta_{k+1}) + 2\\varsigma\\sT u_k^\\theta + 2r_k.\n\\end{equation}\n","sha256_utf8":"9f10f85f93054f40fa0121c9f4d14a8e3eceb96b868aeb90718e37301d39c73b"}]},{"id":"S086","semantic_type":"proof_step","kind":"proof_step","title":"建立局部不变性与累积位移归纳","source_units":[{"start":88276,"end":89270,"start_line":1091,"end_line":1099,"raw_tex":"\\noindent\\textbf{Step 4:} \\emph{Attracted by the local region.}  We now show that the following statements hold for all $k \\geq \\hat k$: \n%\n\t\\begin{enumerate}[label=(\\alph*),topsep=4pt,itemsep=0.5ex,partopsep=0ex]\n\t\\item $z^{t_k} \\in \\mathcal{B}_{\\rho}(z^*)$ and $|\\mer(z^{t_k}) - \\psi^*| + \\max\\{\\frac{\\xi\\lambda}{2}\\|\\Fnor(z^{t_k})\\|^2,u_k\\}< \\eta$.\n %\\vspace{1mm}\n\t\\item $\\sum_{i=\\hat k}^{k}d_i  \\leq 20\\xi^{-1}[\\Delta_{\\hat k} - \\Delta_{k+1}] + 2\\varsigma\\sT \\sum_{i=\\hat k}^k u_i^\\theta + 2\\sum_{i=\\hat k}^k r_i.$\n\t\\end{enumerate}\n% \nWe will prove these statements by induction. First, (a) and (b) are satisfied for $k=\\hat k$ thanks to \\cref{eq:choose rho eta} and \\cref{eq:d bound}. To conduct the induction step, let us assume that (a)--(b) hold for some $k =n \\geq \\hat k$. We first verify (a) for $k=n+1$. Due to \\cref{eq:choose rho eta-2}, the bound $|\\mer(z^{t_{n+1}}) - \\psi^*| + \\max\\{\\frac{\\xi\\lambda}{2}\\|\\Fnor(z^{t_{n+1}})\\|^2,u_{n+1}\\} < \\eta$ is immediate. Moreover, we have\n","sha256_utf8":"e27c53aa1b46347a4dbf7a5f796df8e9661bf965ecbd3e3ec1333b9db61e483b"}]},{"id":"S087","semantic_type":"proof_step","kind":"proof_step","title":"完成邻域保持与位移累积归纳","source_units":[{"start":89272,"end":90266,"start_line":1101,"end_line":1112,"raw_tex":"\\begin{align*}\n    \\|z^{t_{n+1}} - z^*\\| &\\leq \\|z^{t_{\\hat k}} - z^*\\| + {\\sum}_{i=\\hat{k}}^n \\|z^{t_{i+1}} - z^{t_{i}}\\| \\leq  \\|z^{t_{\\hat k}} - z^*\\| + {\\sum}_{i=\\hat k}^n \\,d_i\\\\\n    &\\leq \\|z^{t_{\\hat k}} - z^*\\| +  20\\xi^{-1}[\\Delta_{\\hat k} - \\Delta_{n+1}] + 2\\varsigma\\sT {\\sum}_{i=\\hat k}^\\infty\\, u_i^\\theta + 2{\\sum}_{i=\\hat k}^\\infty\\, r_i < \\rho,\n\\end{align*}\n%\nwhere the second inequality follows from the definition of $d_k$, the third step uses (b) for $k=n$, and the last step holds due to \\cref{eq:choose rho eta-1} and $\\Delta_{n+1}\\geq 0$. Next, we prove that (b) holds for $k=n+1$. Invoking \\cref{eq:d bound} with $k=n+1$ and (b) with $k=n$, we obtain\n%\n\\[\n{\\sum}_{i=\\hat k}^{n+1}\\,d_i = d_{n+1} + {\\sum}_{i=\\hat k}^{n}\\,d_i \\leq 20\\xi^{-1}[\\Delta_{\\hat k} - \\Delta_{n+2}] + 2\\varsigma\\sT {\\sum}_{i=\\hat k}^{n+1}\\, u_i^\\theta + 2{\\sum}_{i=\\hat k}^{n+1}\\, r_i.\n\\]\n%\nThis completes the induction. Thus, (a), (b), and \\cref{eq:d bound} hold for all $k \\geq \\hat k$. %\\\\[1mm]\n","sha256_utf8":"40633969ea2d746926ca5a655aa07feb0b3890dae6f7fd835d7607e3c8d73e6c"}]},{"id":"S088","semantic_type":"proof_step","kind":"proof_step","title":"取极限得有限位移并完成收敛","source_units":[{"start":90267,"end":90769,"start_line":1114,"end_line":1121,"raw_tex":"\\noindent\\textbf{Step 5:} \\emph{Summability of $\\revise{\\{d_k\\}_{k\\in\\N}}$.} Taking $k\\to\\infty$ in \\textbf{Step 4} (b) and due to $\\Delta_k \\geq 0$, we have \\vspace{-1.5ex}\n%\n\\begin{equation*} \n{\\sum}_{i=\\hat k}^{\\infty}d_i  \\leq 20\\xi^{-1}\\Delta_{\\hat k} + 2\\varsigma\\sT {\\sum}_{i=\\hat k}^{\\infty} u_i^\\theta + 2{\\sum}_{i=\\hat k}^{\\infty} r_i < \\rho < \\infty.\n\\end{equation*}\n%\nThis yields $\\sum_{k=0}^{\\infty}d_k < \\infty$ and, following \\textbf{Step 2}, this completes the proof. \\hfill \\proofbox\n\n","sha256_utf8":"e87b045c17d21af9f1da0eab10d7e90025bf68ed084dc84ad25288fd571bd758"}]},{"id":"S089","semantic_type":"assumption","kind":"assumption","title":"主定理的加权步长假设","source_units":[{"start":78047,"end":78681,"start_line":979,"end_line":985,"raw_tex":"\t Let \\ref{A1}--\\ref{A3}, \\ref{B1}--\\ref{B2}, and \\ref{B4} hold and let $\\revise{\\{\\sx^k\\}_{k\\in\\N}}$ be generated by $\\NSGD$ with step sizes $\\revise{\\{\\alpha_k\\}_{k\\in\\N}}\\subset \\R_{++}$ satisfying: %\\vspace{-0.25ex}\n     %\n\t\\begin{equation}\\label{eq:kl-step}  \n\t\t\\lim_{k\\to\\infty}\\alpha_k=0,\\quad {\\sum}_{k=0}^\\infty \\alpha_k = \\infty \\quad \\text{and} \\quad {\\sum}_{k=0}^{\\infty} \\alpha_k^2 \\sigma_k^2 \\Big( {\\sum}_{i=0}^k\\alpha_i \\Big)^{\\mu}< \\infty,% \\quad \\text{for some $\\mu>\\frac12$}.\n        %\\vspace{-0.25ex}\n\t\\end{equation}\nfor some $\\mu>1$. Then, we have $\\sx^k \\to \\sx^*$ a$.$s$.$ for some $\\sx^*:\\Omega\\to\\crit(\\psi)$.\n","sha256_utf8":"e4a202b3ea49bca622653983ef7139477d92f4b750af978fee2b56d9951040ab"}]},{"id":"S090","semantic_type":"external_fact","kind":"external_fact","title":"向量鞅最大二阶矩不等式","source_units":[],"declared_external":true,"declaration_text":["For a vector-valued martingale M with M_0=0, E[sup_k ||M_k||^2] <= 4 E[sum_{k>=1} ||M_k-M_{k-1}||^2]. This is the p=2 case of the cited Burkholder–Davis–Gundy inequality."]},{"id":"S091","semantic_type":"external_fact","kind":"external_fact","title":"离散格朗沃尔不等式","source_units":[],"declared_external":true,"declaration_text":["If y_0<=b_1 and y_{t+1}<=b_1+b_2 sum_{k=0}^t a_k y_k, with nonnegative y_k,b_1,b_2 and positive a_k, then y_{t+1}<=b_1 exp(b_2 sum_{k=0}^t a_k), as stated in B0130."]},{"id":"S092","semantic_type":"external_fact","kind":"external_fact","title":"凸邻近映射与包络标准性质","source_units":[],"declared_external":true,"declaration_text":["For proper lower semicontinuous convex phi and lambda>0, prox_{lambda phi} is firmly nonexpansive, x=prox_{lambda phi}(z) iff (z-x)/lambda belongs to partial phi(x), and env_{lambda phi}(z)=phi(prox_{lambda phi}(z))+||z-prox_{lambda phi}(z)||^2/(2lambda) is continuous. For continuously differentiable f, partial(f+phi)=grad f+partial phi."]},{"id":"S093","semantic_type":"external_fact","kind":"external_fact","title":"可定义函数的局部KL定理","source_units":[],"declared_external":true,"declaration_text":["Near a stationary point of a proper lower semicontinuous definable function psi, there is a definable continuous concave desingularization function rho_1, C^1 away from zero, satisfying rho_1'(|psi(x)-psi(x*)|) dist(0,partial psi(x))>=1 whenever the nonzero value difference is sufficiently small. The cited KL results and desingularization framework supply this boundary fact."]},{"id":"S094","semantic_type":"external_fact","kind":"external_fact","title":"单变量可定义函数的结构性质","source_units":[],"declared_external":true,"declaration_text":["The definable functions used in the proof remain definable under differentiation, maxima, reciprocal, and the stated power operations. One-variable definable functions admit finite differentiability partitions and are eventually monotone near an endpoint; these properties also apply to their definable derivatives."]},{"id":"S095","semantic_type":"external_fact","kind":"external_fact","title":"凹函数的次可加性","source_units":[],"declared_external":true,"declaration_text":["If h is nonnegative and concave on an interval starting at zero and h(0)=0, then h(s+t)<=h(s)+h(t) whenever s,t>=0 and s+t lies in that interval; the author invokes josz2023convergence, Lemma 3.5."]},{"id":"S096","semantic_type":"external_fact","kind":"external_fact","title":"积分极限定理","source_units":[],"declared_external":true,"declaration_text":["Monotone convergence permits exchanging expectations and increasing sums of nonnegative random variables; Fatou's lemma bounds the expectation of a nonnegative liminf; a nonnegative random variable with finite expectation is finite almost surely. An integrable decreasing sequence bounded below by a constant has convergent expectations."]},{"id":"S097","semantic_type":"external_fact","kind":"external_fact","title":"基础分析与范数不等式","source_units":[],"declared_external":true,"declaration_text":["Triangle, Young, and Cauchy–Schwarz inequalities, the mean value theorem, convergence of sum k^{-p} for p>1, existence of convergent subsequences of bounded sequences in finite-dimensional Euclidean space, and completeness of Euclidean space."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S008","kind":"supports"},{"source_id":"S003","target_id":"S009","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S010","kind":"supports"},{"source_id":"S007","target_id":"S011","kind":"supports"},{"source_id":"S004","target_id":"S022","kind":"supports"},{"source_id":"S004","target_id":"S033","kind":"supports"},{"source_id":"S004","target_id":"S036","kind":"supports"},{"source_id":"S004","target_id":"S059","kind":"supports"},{"source_id":"S005","target_id":"S027","kind":"supports"},{"source_id":"S005","target_id":"S028","kind":"supports"},{"source_id":"S005","target_id":"S048","kind":"supports"},{"source_id":"S005","target_id":"S058","kind":"supports"},{"source_id":"S008","target_id":"S022","kind":"supports"},{"source_id":"S008","target_id":"S033","kind":"supports"},{"source_id":"S008","target_id":"S071","kind":"supports"},{"source_id":"S008","target_id":"S075","kind":"supports"},{"source_id":"S009","target_id":"S061","kind":"supports"},{"source_id":"S009","target_id":"S075","kind":"supports"},{"source_id":"S010","target_id":"S022","kind":"supports"},{"source_id":"S010","target_id":"S023","kind":"supports"},{"source_id":"S010","target_id":"S038","kind":"supports"},{"source_id":"S010","target_id":"S052","kind":"supports"},{"source_id":"S010","target_id":"S070","kind":"supports"},{"source_id":"S010","target_id":"S075","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S021","kind":"supports"},{"source_id":"S011","target_id":"S022","kind":"supports"},{"source_id":"S011","target_id":"S027","kind":"supports"},{"source_id":"S012","target_id":"S033","kind":"supports"},{"source_id":"S012","target_id":"S034","kind":"supports"},{"source_id":"S012","target_id":"S036","kind":"supports"},{"source_id":"S012","target_id":"S038","kind":"supports"},{"source_id":"S012","target_id":"S052","kind":"supports"},{"source_id":"S013","target_id":"S041","kind":"supports"},{"source_id":"S013","target_id":"S059","kind":"supports"},{"source_id":"S013","target_id":"S062","kind":"supports"},{"source_id":"S013","target_id":"S070","kind":"supports"},{"source_id":"S013","target_id":"S071","kind":"supports"},{"source_id":"S013","target_id":"S076","kind":"supports"},{"source_id":"S014","target_id":"S047","kind":"supports"},{"source_id":"S014","target_id":"S048","kind":"supports"},{"source_id":"S015","target_id":"S051","kind":"supports"},{"source_id":"S015","target_id":"S054","kind":"supports"},{"source_id":"S015","target_id":"S060","kind":"supports"},{"source_id":"S015","target_id":"S081","kind":"supports"},{"source_id":"S015","target_id":"S082","kind":"supports"},{"source_id":"S016","target_id":"S043","kind":"supports"},{"source_id":"S016","target_id":"S051","kind":"supports"},{"source_id":"S016","target_id":"S057","kind":"supports"},{"source_id":"S016","target_id":"S077","kind":"supports"},{"source_id":"S016","target_id":"S080","kind":"supports"},{"source_id":"S017","target_id":"S066","kind":"supports"},{"source_id":"S018","target_id":"S079","kind":"supports"},{"source_id":"S018","target_id":"S075","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S020","kind":"supports"},{"source_id":"S023","target_id":"S020","kind":"supports"},{"source_id":"S024","target_id":"S020","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S029","target_id":"S026","kind":"supports"},{"source_id":"S030","target_id":"S026","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S032","kind":"supports"},{"source_id":"S035","target_id":"S032","kind":"supports"},{"source_id":"S036","target_id":"S032","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S031","target_id":"S041","kind":"supports"},{"source_id":"S037","target_id":"S042","kind":"supports"},{"source_id":"S019","target_id":"S044","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S042","target_id":"S040","kind":"supports"},{"source_id":"S043","target_id":"S040","kind":"supports"},{"source_id":"S044","target_id":"S040","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S022","target_id":"S051","kind":"supports"},{"source_id":"S019","target_id":"S051","kind":"supports"},{"source_id":"S031","target_id":"S052","kind":"supports"},{"source_id":"S019","target_id":"S053","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S050","kind":"supports"},{"source_id":"S054","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S039","target_id":"S057","kind":"supports"},{"source_id":"S045","target_id":"S057","kind":"supports"},{"source_id":"S025","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S049","target_id":"S060","kind":"supports"},{"source_id":"S059","target_id":"S061","kind":"supports"},{"source_id":"S058","target_id":"S061","kind":"supports"},{"source_id":"S049","target_id":"S061","kind":"supports"},{"source_id":"S058","target_id":"S062","kind":"supports"},{"source_id":"S060","target_id":"S062","kind":"supports"},{"source_id":"S049","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S063","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S049","target_id":"S063","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S058","target_id":"S056","kind":"supports"},{"source_id":"S059","target_id":"S056","kind":"supports"},{"source_id":"S060","target_id":"S056","kind":"supports"},{"source_id":"S061","target_id":"S056","kind":"supports"},{"source_id":"S062","target_id":"S056","kind":"supports"},{"source_id":"S063","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S055","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S067","target_id":"S069","kind":"supports"},{"source_id":"S066","target_id":"S071","kind":"supports"},{"source_id":"S069","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S067","target_id":"S072","kind":"supports"},{"source_id":"S069","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"},{"source_id":"S067","target_id":"S065","kind":"supports"},{"source_id":"S068","target_id":"S065","kind":"supports"},{"source_id":"S069","target_id":"S065","kind":"supports"},{"source_id":"S070","target_id":"S065","kind":"supports"},{"source_id":"S071","target_id":"S065","kind":"supports"},{"source_id":"S072","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S079","target_id":"S075","kind":"supports"},{"source_id":"S075","target_id":"S076","kind":"supports"},{"source_id":"S075","target_id":"S074","kind":"supports"},{"source_id":"S076","target_id":"S074","kind":"supports"},{"source_id":"S074","target_id":"S073","kind":"supports"},{"source_id":"S089","target_id":"S077","kind":"supports"},{"source_id":"S089","target_id":"S078","kind":"supports"},{"source_id":"S089","target_id":"S079","kind":"supports"},{"source_id":"S025","target_id":"S077","kind":"supports"},{"source_id":"S045","target_id":"S077","kind":"supports"},{"source_id":"S077","target_id":"S078","kind":"supports"},{"source_id":"S077","target_id":"S079","kind":"supports"},{"source_id":"S078","target_id":"S079","kind":"supports"},{"source_id":"S055","target_id":"S079","kind":"supports"},{"source_id":"S019","target_id":"S080","kind":"supports"},{"source_id":"S080","target_id":"S081","kind":"supports"},{"source_id":"S079","target_id":"S081","kind":"supports"},{"source_id":"S073","target_id":"S082","kind":"supports"},{"source_id":"S080","target_id":"S082","kind":"supports"},{"source_id":"S079","target_id":"S082","kind":"supports"},{"source_id":"S073","target_id":"S083","kind":"supports"},{"source_id":"S064","target_id":"S083","kind":"supports"},{"source_id":"S058","target_id":"S084","kind":"supports"},{"source_id":"S062","target_id":"S084","kind":"supports"},{"source_id":"S077","target_id":"S084","kind":"supports"},{"source_id":"S078","target_id":"S084","kind":"supports"},{"source_id":"S082","target_id":"S084","kind":"supports"},{"source_id":"S083","target_id":"S084","kind":"supports"},{"source_id":"S058","target_id":"S085","kind":"supports"},{"source_id":"S064","target_id":"S085","kind":"supports"},{"source_id":"S080","target_id":"S085","kind":"supports"},{"source_id":"S083","target_id":"S085","kind":"supports"},{"source_id":"S084","target_id":"S085","kind":"supports"},{"source_id":"S084","target_id":"S086","kind":"supports"},{"source_id":"S085","target_id":"S086","kind":"supports"},{"source_id":"S086","target_id":"S087","kind":"supports"},{"source_id":"S084","target_id":"S087","kind":"supports"},{"source_id":"S085","target_id":"S087","kind":"supports"},{"source_id":"S077","target_id":"S088","kind":"supports"},{"source_id":"S078","target_id":"S088","kind":"supports"},{"source_id":"S081","target_id":"S088","kind":"supports"},{"source_id":"S073","target_id":"S088","kind":"supports"},{"source_id":"S087","target_id":"S088","kind":"supports"},{"source_id":"S077","target_id":"S002","kind":"supports"},{"source_id":"S078","target_id":"S002","kind":"supports"},{"source_id":"S079","target_id":"S002","kind":"supports"},{"source_id":"S080","target_id":"S002","kind":"supports"},{"source_id":"S081","target_id":"S002","kind":"supports"},{"source_id":"S082","target_id":"S002","kind":"supports"},{"source_id":"S083","target_id":"S002","kind":"supports"},{"source_id":"S084","target_id":"S002","kind":"supports"},{"source_id":"S085","target_id":"S002","kind":"supports"},{"source_id":"S086","target_id":"S002","kind":"supports"},{"source_id":"S087","target_id":"S002","kind":"supports"},{"source_id":"S088","target_id":"S002","kind":"supports"},{"source_id":"S090","target_id":"S028","kind":"supports"},{"source_id":"S091","target_id":"S023","kind":"supports"},{"source_id":"S092","target_id":"S008","kind":"supports"},{"source_id":"S092","target_id":"S009","kind":"supports"},{"source_id":"S092","target_id":"S010","kind":"supports"},{"source_id":"S092","target_id":"S036","kind":"supports"},{"source_id":"S092","target_id":"S076","kind":"supports"},{"source_id":"S093","target_id":"S066","kind":"supports"},{"source_id":"S094","target_id":"S068","kind":"supports"},{"source_id":"S094","target_id":"S069","kind":"supports"},{"source_id":"S095","target_id":"S069","kind":"supports"},{"source_id":"S096","target_id":"S030","kind":"supports"},{"source_id":"S096","target_id":"S063","kind":"supports"},{"source_id":"S097","target_id":"S021","kind":"supports"},{"source_id":"S097","target_id":"S034","kind":"supports"},{"source_id":"S097","target_id":"S035","kind":"supports"},{"source_id":"S097","target_id":"S038","kind":"supports"},{"source_id":"S097","target_id":"S069","kind":"supports"},{"source_id":"S097","target_id":"S075","kind":"supports"},{"source_id":"S097","target_id":"S077","kind":"supports"},{"source_id":"S097","target_id":"S078","kind":"supports"},{"source_id":"S097","target_id":"S081","kind":"supports"},{"source_id":"S097","target_id":"S088","kind":"supports"}]}}
{"record_id":"2305.17323v5","paper":{"title":"Some Primal-Dual Theory for Subgradient Methods for Strongly Convex Optimization","authors":["Benjamin Grimmer","Danlin Li"],"arxiv_id":"2305.17323","version":"v5","journal":"Mathematical Programming","doi":"10.1007/s10107-025-02201-8","abstract_url":"https://arxiv.org/abs/2305.17323v5","pdf_url":"https://arxiv.org/pdf/2305.17323v5"},"context":{"raw_tex":"\\begin{equation}\n    p_\\star = \\begin{cases}\n        \\min & f_0(x) + r(x)\\\\\n        \\mathrm{s.t.} & f_s(x) \\leq 0 \\qquad \\forall s=1\\dots m\n    \\end{cases}\\label{eq:super-problem}\n\\end{equation}\n\nwhere the functions $f_s\\colon \\mathcal{E}\\rightarrow \\mathbb{R}$, $s=0,\\dots, m,$ are (strongly) convex but may be nonsmooth and not globally Lipschitz continuous and $r\\colon \\mathcal{E}\\rightarrow \\mathbb{R}\\cup\\{\\infty\\}$ is convex, closed, and simple, all defined over a finite-dimensional Euclidean space $\\varSpace$. We will consider iteratively solving problems of this general form via a stochastic switching proximal subgradient method. This general method corresponds to the classic subgradient method when $r=0$ and $m=0$, the proximal subgradient method when $m=0$, and the switching subgradient method of~\\cite{Polyak1967} when $r=0$. Formal assumptions and definitions are deferred to Section~\\ref{sec:prelim}. \n\n\n$g_0(x)\\in \\partial f_0(x) := \\{  g \\mid f_0(y) \\geq f_0(x) + \\langle g , y-x \\rangle \\quad \\forall y\\in\\varSpace\\}$. Note when $f_0$ is $\\mu$-strongly convex (that is, $f(x)-\\frac{\\mu}{2}\\|x\\|_2^2$ is convex), each subgradient $g_0(x)\\in \\partial f_0(x)$, provides a quadratic lower bound\n\\begin{equation}\n    f_0(y) \\geq f_0(x) + \\langle g_0(x) , y-x \\rangle +\\frac{\\mu}{2}\\|y-x\\|_2^2 \\quad \\forall y\\in\\varSpace \\ .\n\\label{eq:subgradient-lower-bound}\n\\end{equation}\n\nFirst, we allow for additive composite objectives $f_0+r$ for any {\\color{blue} proper} closed convex $r\\colon \\varSpace\\rightarrow \\mathbb{R}\\cup\\{\\infty\\}$. We address this added term by assuming $r$ is sufficiently simple that its proximal operator\n$ \\mathrm{prox}_{\\alpha_k, r}(z) := \\argmin_x\\left\\{r(x) + \\frac{1}{2\\alpha_k}\\|x-z\\|_2^2\\right\\} $\ncan be evaluated at each iteration. For example, if $r$ is an indicator function for a closed convex constraint set, its proximal operator corresponds to projection onto that set. Since $r(x) + \\frac{1}{2\\alpha_k}\\|x-z\\|_2^2$ is strongly convex, it has a unique minimizer described by the following equivalence\n\\begin{equation}\n    z_+ = \\mathrm{prox}_{\\alpha_k, r}(z) \\quad \\iff \\quad \\frac{1}{\\alpha_k}(z-z_+)\\in \\partial r(z_+) \\ . \\label{eq:proximal-equivalence}\n\\end{equation}\n\nSecond, we allow for $m$ strongly convex functional constraints $f_s(x)\\leq 0$ for $s=1,\\dots, m$. We address these added terms by ``switching'': Given a current iterate $x$, only one function $f_{s(x)}$ will be considered in that iteration. This function will be chosen as $s(x)=0$ if $x$ is feasible (that is, $f_s(x)\\leq 0$ for all $s=1\\dots m$), otherwise $s(x)$ can be chosen generically as any violated constraint $f_{s(x)}(x)>0$.\nThird, we allow for stochasticity in our subgradient oracles for each function, denoted by $g_s(x;\\xi)$ such that $\\mathbb{E}_\\xi g_{s}(x;\\xi) \\in \\partial f_{s}(x)$. Note this trivially captures deterministic methods by selecting $g_s(x;\\xi)$ constant with respect to $\\xi$.\n\n\nAs a primal algorithm, consider the {\\it Stochastic Switching Proximal Subgradient Method} with stepsizes $\\alpha_k>0$ and sequence of iterates $\\{x_k\\}$ defined by\n\\begin{equation}\\label{eq:super-primal-method}\n    x_{k+1} = \\begin{cases}\n        \\mathrm{prox}_{\\alpha_k, r} (x_k-\\alpha_k g_{0}(x_k;\\xi_k)) & \\text{if $x_k$ is feasible}\\\\\n        x_k-\\alpha_k g_{s(x_k)}(x_k;\\xi_k) & \\text{otherwise.}        \n    \\end{cases}\n\\end{equation}\n\nfor i.i.d.~sampled $\\xi_k$. Throughout, we always assume $\\alpha_k>0$, strictly.\n\n\n\\begin{align*}\n    p_\\star &= \\min_{x\\in\\varSpace}\\max_{u_1,\\dots u_s\\geq 0} f_0(x) + \\sum_{s=1}^m u_sf_s(x) + r(x) \\\\\n    &=\\min_{x\\in\\varSpace}\\max_{\\substack{y_0,\\dots, y_{T}\\in\\varSpace\\\\ \\lambda_0,\\dots, \\lambda_{T-1}\\geq 0 \\\\ \\sum_{k<T\\colon s(y_k)=0} \\lambda_k >0\\\\ n_{k+1}\\in\\partial r(y_{k+1})}} \\mathbb{E}_{\\xi}\\Bigg(\\sum_{s=0}^m\\sum_{\\substack{k<T \\\\ s(y_k)=s}} \\lambda_k\\left(f_{s(y_k)}(y_k) + \\langle g_{s(y_k)}(y_k; \\xi_k), x-y_k \\rangle + \\frac{\\mu}{2}\\|x-y_k\\|_2^2\\right)\\\\\n    &\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad +\\sum_{\\substack{k<T\\\\ s(y_k)=0}} \\lambda_k\\left(r(y_{k+1}) + \\langle n_{k+1}, x - y_{k+1}\\rangle\\right)\\Bigg)\\frac{1}{\\sum_{k<T\\colon s(y_k)=0} \\lambda_k}\n\\end{align*}\n\nAs a dual algorithm, consider the following {\\it Stochastic Lagrangian Proximal Dual Averaging} with dual weights $\\lambda_k>0$ and regularization parameters $\\beta_k\\geq 0$. Throughout, we always assume $\\lambda_k>0$, strictly. Its sequence of iterates $\\{y_k\\}$ based on i.i.d.~sampled $\\xi_k$ is defined as follows: At iteration $k$, construct the following (unnormalized) aggregations for each function based on the previous $k-1$ iterations as\n\n\nwhere $n_{i+1} = \\frac{-1}{\\lambda_i}(\\nabla M^{(i-1)}(y_{i+1}) + \\lambda_i(g_0(y_i;\\xi_i) +\\mu(y_{i+1}-y_i)) + \\beta_i(y_{i+1} - y_0)) \\in \\partial r(y_{i+1})$ (see Lemma~\\ref{lem:Mk-minimization} for verification of this subdifferential containment).\nAt iteration $k=0$, these empty summations are understood to take value zero.\nThen, based on the switching selection $s(y_k)$, a new weighted model is constructed as\n$$ U^{(k)}(y) := \\begin{cases} \\lambda_k\\left(f_{0}(y_k) + \\langle g_0(y_k;\\xi_k), y-y_k\\rangle + \\frac{\\mu}{2}\\|y-y_k\\|_2^2 + r(y)\\right) & \\text{if $y_k$ is feasible}\\\\\n\\lambda_k\\left(f_{s(y_k)}(y_k) + \\langle g_{s(y_k)}(y_k;\\xi_k), y-y_k\\rangle + \\frac{\\mu}{2}\\|y-y_k\\|_2^2\\right) & \\text{otherwise.}\n\\end{cases} $$\n\nThe Lagrangian proximal dual averaging method then iterates by minimizing the aggregation of past models $M^{(k-1)}$ plus the new model $U^{(k)}$ (and an optional extra regularization term)\n\\begin{equation}\\label{eq:super-dual-method}\n    y_{k+1} = \\argmin\\left\\{M^{(k-1)}(y) + U^{(k)}(y) +\\frac{\\beta_k}{2}\\|y-y_0\\|_2^2 \\right\\} \\ .\n\\end{equation}\n\n\\noindent {\\bf Assumption A.} The functions $f_s$ for $s=0\\dots m$ are each $\\mu>0$-strongly convex.\\\\\n\\noindent {\\bf Assumption B.} There exists some $x_\\mathtt{SL} \\in \\mathrm{dom\\ }\\partial r$ with $f_s(x_\\mathtt{SL}) <0$ for all \n $s = 1\\dots m$.\n\n\nNote strong convexity ensures there exists a unique minimizer $x_\\mathtt{OPT} \\in \\mathrm{dom\\ }\\partial r$ of~\\eqref{eq:super-problem}. These two points $x_\\mathtt{OPT},x_\\mathtt{SL} \\in \\mathrm{dom\\ }\\partial r$ serve as important references that our analysis is done with respect to. We fix two subgradients of $r$ at these points: The subgradient $n_{x_\\mathtt{OPT}}\\in \\partial r(x_\\mathtt{OPT})$ is chosen such that $f_0(x) + r(x_\\mathtt{OPT}) + \\langle n_{x_\\mathtt{OPT}}, x-x_\\mathtt{OPT}\\rangle$ is minimized over $f_s(x)\\leq 0$ at $x_\\mathtt{OPT}$ for all $ s=1\\ldots m$. The subgradient $n_{x_\\mathtt{SL}} \\in \\partial r(x_\\mathtt{SL})$ can be chosen freely. %When $r$ is an indicator function for a closed convex set, $n_{x_\\mathtt{SL}}=0$ is a nice simplifying choice.\n\nThese two reference subgradients of $r$ facilitate considering two lower bounds of the objective $f_0+r$ for either $y\\in\\{x_{\\mathtt{OPT}}, x_{\\mathtt{SL}}\\}$, denoted by\n$$ h_{y}(x) := f_0(x) + r(y) + \\langle n_{y}, x-y\\rangle \\ . $$\n\nAt each iteration $k$, we denote the relative difference between $x_k$ and $y\\in\\{x_{\\mathtt{OPT}}, x_{\\mathtt{SL}}\\}$ in (relaxed) objective value or feasibility on the selected constraint function $f_{s(x_k)}$ by\n\\begin{align}\n    \\delta_k(y) &:=\\begin{cases}\n        h_y(x_k) - h_y(y) & \\mathrm{if\\ } x_k \\mathrm{\\ is\\ feasible}\\\\\n        f_{s(x_k)}(x_k) - f_{s(x_k)}(y) & \\mathrm{otherwise.}\n    \\end{cases}\\label{eq:delta_k-definition}\n\\end{align}\n\nThe sign of $\\delta_k(y)$ may vary. If $y = x_{\\mathtt{OPT}}$, then $\\delta_k(x_\\mathtt{OPT})$ is nonnegative, being lower bounded by the level of suboptimality or current infeasibility\n$$ \\delta_k(x_\\mathtt{OPT}) \\geq \\begin{cases}\n        h_{x_\\mathtt{OPT}}(x_k) - p_\\star & \\mathrm{if\\ } x_k \\mathrm{\\ is\\ feasible}\\\\\n        f_{s(x_k)}(x_k) & \\mathrm{otherwise}\n    \\end{cases} \\ \\geq 0 \\ . $$\nWhen $y = x_{\\mathtt{SL}}$ and $x_k$ is feasible, $\\delta_k(y)$ may be negative but is bounded below by\n$$ \\delta_k(x_\\mathtt{SL}) \\geq \\inf h_{x_{\\mathtt{SL}}} -h_{x_{\\mathtt{SL}}}(x_{\\mathtt{SL}}) > -\\infty \\ . $$ When $y = x_{\\mathtt{SL}}$ and $x_k$ is infeasible, $\\delta_k(y)$ is strictly positive, being bounded below by $$\\delta_k(x_\\mathtt{SL}) \\geq 0-\\max_{s=1\\dots m} f_s(x_\\mathtt{SL}) >0 \\ . $$\n\n\n\\noindent {\\bf Assumption C.} For both $y\\in\\{x_{\\mathtt{OPT}}, x_{\\mathtt{SL}}\\}$, there exist constants $L_0$, $L_1{\\color{blue}\\geq 0}$ such that every iterate $x_k$ has\n\\begin{equation}\\label{eq:NonLipschitz-condition}\n    \\begin{cases}\n        x_k \\mathrm{\\ is\\ feasible} & \\implies \\mathbb{E}_{\\xi_k}\\|g_0(x_k;\\xi_k) + n_{y}\\|_2^2 \\leq L_0^2 + L_1\\delta_k(y)\\\\\n        x_k \\mathrm{\\ is\\ not\\ feasible} & \\implies \\mathbb{E}_{\\xi_k}\\|g_{s(x_k)}(x_k;\\xi_k)\\|_2^2 \\leq L_0^2 + L_1\\delta_k(y) \\ . \n    \\end{cases}\n\\end{equation}\n\nTo understand and bound such behaviors, we introduce the following two constants, dependent on the choice of stepsizes $\\alpha_k$ and associated dual weights $\\lambda_k$,\n\\begin{align}\n    T_0 &:= \\sup\\left\\{ k\\in\\{0,1,2\\dots\\} \\mid L_1\\alpha_k > 1\\right\\} \\ ,\\label{eq:T0} \\\\\n    C_0 &:= \\sum_{k=0}^{T_0}\\lambda_k \\max\\left\\{L_1 \\alpha_k - 1, 0 \\right\\}\\max\\{\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}), \\mathbb{E}_\\xi\\delta_k(x_\\mathtt{SL})\\} \\ . \\label{eq:C0}\n\\end{align}\n\nWe prove a uniform convergence guarantee in terms of the primal gap\n\\begin{equation} \\label{eq:primalgap-definition}\n    \\mathtt{primal\\mbox{-}gap}_T := \\frac{\\sum_{k< T: s(x_k)=0} \\lambda_k h_{x_\\mathtt{OPT}}(x_k)}{\\sum_{k< T: s(x_k)=0} \\lambda_k } - p_\\star \\ ,\n\\end{equation}\n\nwhich utilizes a combination of the feasible objective values seen, the dual gap\n\\begin{equation} \\label{eq:dualgap-definition}\n    \\mathtt{dual\\mbox{-}gap}_T := p_\\star - \\inf \\frac{M^{(T-1)}}{\\sum_{k< T: s(x_k)=0} \\lambda_k }\\ ,\n\\end{equation}\n\nwhich utilizes a combination of the subgradient lower bounds seen, and the distance to optimal.\nFor the primal and dual gaps to be well-defined, at least one feasible iterate must have been seen (i.e., $\\sum_{k< T: s(x_k)=0} \\lambda_k >0$). Our assumptions facilitate a bound on how long it takes for this to occur. We show the expected fraction of the dual weight that occurs on iterations with a feasible iterate (i.e., $s(x_k)=0$) is bounded below.\n\n\nOnce a feasible iterate occurs, our primal and dual convergence measures are well-defined. Alternatively, one could assume the initialization $x_0$ is feasible to ensure these quantities are well-defined. In either case, Subsection~\\ref{subsec:proof-main-result} proves the following primal-dual convergence guarantee as our main result.\n\n\n\\begin{equation} \\label{eq:Rk-definition}\nR_k(y) := \\left(\\frac{\\mu}{2}\\sum_{i=0}^{{k-1}} \\lambda_i\\right) \\mathbb{E}_\\xi\\|x_k-y\\|_2^2\n\\end{equation}\n\nTo simplify notations, throughout our analysis, we denote $g_k = g_{s(x_k)}(x_k;\\xi_k)$ and $w_k = \\mu\\sum_{i=0}^{k} \\lambda_i$ (with the convention that $w_{-1}=0$ as the given summation is empty).\n\n\n\\begin{equation} \\label{eq:Dk-definition}\n    D_k := \\mathbb{E}_\\xi\\left[\\left(\\sum_{i< k:s(x_i)=0} \\lambda_i\\right) p_\\star - \\inf M^{(k-1)}\\right] \\ .\n\\end{equation}","units":[{"start":8570,"end":8764,"start_line":230,"end_line":235,"raw_tex":"\\begin{equation}\n    p_\\star = \\begin{cases}\n        \\min & f_0(x) + r(x)\\\\\n        \\mathrm{s.t.} & f_s(x) \\leq 0 \\qquad \\forall s=1\\dots m\n    \\end{cases}\\label{eq:super-problem}\n\\end{equation}","sha256_utf8":"395f763533be34434b9ead45725e9659026aec06813b7535d0754009da523250"},{"start":8765,"end":9492,"start_line":236,"end_line":236,"raw_tex":"where the functions $f_s\\colon \\mathcal{E}\\rightarrow \\mathbb{R}$, $s=0,\\dots, m,$ are (strongly) convex but may be nonsmooth and not globally Lipschitz continuous and $r\\colon \\mathcal{E}\\rightarrow \\mathbb{R}\\cup\\{\\infty\\}$ is convex, closed, and simple, all defined over a finite-dimensional Euclidean space $\\varSpace$. We will consider iteratively solving problems of this general form via a stochastic switching proximal subgradient method. This general method corresponds to the classic subgradient method when $r=0$ and $m=0$, the proximal subgradient method when $m=0$, and the switching subgradient method of~\\cite{Polyak1967} when $r=0$. Formal assumptions and definitions are deferred to Section~\\ref{sec:prelim}. \n","sha256_utf8":"c1bd6e6a1d732e4b6d55b5aab472a44f7ab5c1c339a521f0b3fd828e6275c71e"},{"start":10157,"end":10625,"start_line":243,"end_line":247,"raw_tex":"$g_0(x)\\in \\partial f_0(x) := \\{  g \\mid f_0(y) \\geq f_0(x) + \\langle g , y-x \\rangle \\quad \\forall y\\in\\varSpace\\}$. Note when $f_0$ is $\\mu$-strongly convex (that is, $f(x)-\\frac{\\mu}{2}\\|x\\|_2^2$ is convex), each subgradient $g_0(x)\\in \\partial f_0(x)$, provides a quadratic lower bound\n\\begin{equation}\n    f_0(y) \\geq f_0(x) + \\langle g_0(x) , y-x \\rangle +\\frac{\\mu}{2}\\|y-x\\|_2^2 \\quad \\forall y\\in\\varSpace \\ .\n\\label{eq:subgradient-lower-bound}\n\\end{equation}","sha256_utf8":"75df2ca926bf80cd0531c5930f7538e0ba2b3c8d706629bdd82178cf0961b15a"},{"start":19194,"end":20027,"start_line":305,"end_line":310,"raw_tex":"First, we allow for additive composite objectives $f_0+r$ for any {\\color{blue} proper} closed convex $r\\colon \\varSpace\\rightarrow \\mathbb{R}\\cup\\{\\infty\\}$. We address this added term by assuming $r$ is sufficiently simple that its proximal operator\n$ \\mathrm{prox}_{\\alpha_k, r}(z) := \\argmin_x\\left\\{r(x) + \\frac{1}{2\\alpha_k}\\|x-z\\|_2^2\\right\\} $\ncan be evaluated at each iteration. For example, if $r$ is an indicator function for a closed convex constraint set, its proximal operator corresponds to projection onto that set. Since $r(x) + \\frac{1}{2\\alpha_k}\\|x-z\\|_2^2$ is strongly convex, it has a unique minimizer described by the following equivalence\n\\begin{equation}\n    z_+ = \\mathrm{prox}_{\\alpha_k, r}(z) \\quad \\iff \\quad \\frac{1}{\\alpha_k}(z-z_+)\\in \\partial r(z_+) \\ . \\label{eq:proximal-equivalence}\n\\end{equation}","sha256_utf8":"1832f1b4bac6131c873cfe4b5c1180d659f94ec1cb66ece1ec86dc58e47517ba"},{"start":20028,"end":20741,"start_line":311,"end_line":312,"raw_tex":"Second, we allow for $m$ strongly convex functional constraints $f_s(x)\\leq 0$ for $s=1,\\dots, m$. We address these added terms by ``switching'': Given a current iterate $x$, only one function $f_{s(x)}$ will be considered in that iteration. This function will be chosen as $s(x)=0$ if $x$ is feasible (that is, $f_s(x)\\leq 0$ for all $s=1\\dots m$), otherwise $s(x)$ can be chosen generically as any violated constraint $f_{s(x)}(x)>0$.\nThird, we allow for stochasticity in our subgradient oracles for each function, denoted by $g_s(x;\\xi)$ such that $\\mathbb{E}_\\xi g_{s}(x;\\xi) \\in \\partial f_{s}(x)$. Note this trivially captures deterministic methods by selecting $g_s(x;\\xi)$ constant with respect to $\\xi$.\n","sha256_utf8":"e9dd8bdd6bb4ba69c8c3251361f0bfff4e059d4bc62825f551e1d1519c4fc29b"},{"start":21077,"end":21518,"start_line":317,"end_line":323,"raw_tex":"As a primal algorithm, consider the {\\it Stochastic Switching Proximal Subgradient Method} with stepsizes $\\alpha_k>0$ and sequence of iterates $\\{x_k\\}$ defined by\n\\begin{equation}\\label{eq:super-primal-method}\n    x_{k+1} = \\begin{cases}\n        \\mathrm{prox}_{\\alpha_k, r} (x_k-\\alpha_k g_{0}(x_k;\\xi_k)) & \\text{if $x_k$ is feasible}\\\\\n        x_k-\\alpha_k g_{s(x_k)}(x_k;\\xi_k) & \\text{otherwise.}        \n    \\end{cases}\n\\end{equation}","sha256_utf8":"f03c6cd42c3e9174aa1621903f67f46981f5f1364d0cf06823111311bec5962a"},{"start":21519,"end":21600,"start_line":324,"end_line":324,"raw_tex":"for i.i.d.~sampled $\\xi_k$. Throughout, we always assume $\\alpha_k>0$, strictly.\n","sha256_utf8":"71166c253ad8e6feee5fdec9789c8662d1ad13bad30c51ce222908d96a85d013"},{"start":22432,"end":23132,"start_line":330,"end_line":334,"raw_tex":"\\begin{align*}\n    p_\\star &= \\min_{x\\in\\varSpace}\\max_{u_1,\\dots u_s\\geq 0} f_0(x) + \\sum_{s=1}^m u_sf_s(x) + r(x) \\\\\n    &=\\min_{x\\in\\varSpace}\\max_{\\substack{y_0,\\dots, y_{T}\\in\\varSpace\\\\ \\lambda_0,\\dots, \\lambda_{T-1}\\geq 0 \\\\ \\sum_{k<T\\colon s(y_k)=0} \\lambda_k >0\\\\ n_{k+1}\\in\\partial r(y_{k+1})}} \\mathbb{E}_{\\xi}\\Bigg(\\sum_{s=0}^m\\sum_{\\substack{k<T \\\\ s(y_k)=s}} \\lambda_k\\left(f_{s(y_k)}(y_k) + \\langle g_{s(y_k)}(y_k; \\xi_k), x-y_k \\rangle + \\frac{\\mu}{2}\\|x-y_k\\|_2^2\\right)\\\\\n    &\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad +\\sum_{\\substack{k<T\\\\ s(y_k)=0}} \\lambda_k\\left(r(y_{k+1}) + \\langle n_{k+1}, x - y_{k+1}\\rangle\\right)\\Bigg)\\frac{1}{\\sum_{k<T\\colon s(y_k)=0} \\lambda_k}\n\\end{align*}","sha256_utf8":"2d8c1547ea63c0c2286fb85cd53df42121db054337fcd16a7118ef4d2ea0f0c1"},{"start":23400,"end":23849,"start_line":337,"end_line":337,"raw_tex":"As a dual algorithm, consider the following {\\it Stochastic Lagrangian Proximal Dual Averaging} with dual weights $\\lambda_k>0$ and regularization parameters $\\beta_k\\geq 0$. Throughout, we always assume $\\lambda_k>0$, strictly. Its sequence of iterates $\\{y_k\\}$ based on i.i.d.~sampled $\\xi_k$ is defined as follows: At iteration $k$, construct the following (unnormalized) aggregations for each function based on the previous $k-1$ iterations as\n","sha256_utf8":"405790e8630f63f115b2b06be339aaee8d50bb5297305bc61c1e3e56d9ea2a80"},{"start":24229,"end":24966,"start_line":343,"end_line":348,"raw_tex":"where $n_{i+1} = \\frac{-1}{\\lambda_i}(\\nabla M^{(i-1)}(y_{i+1}) + \\lambda_i(g_0(y_i;\\xi_i) +\\mu(y_{i+1}-y_i)) + \\beta_i(y_{i+1} - y_0)) \\in \\partial r(y_{i+1})$ (see Lemma~\\ref{lem:Mk-minimization} for verification of this subdifferential containment).\nAt iteration $k=0$, these empty summations are understood to take value zero.\nThen, based on the switching selection $s(y_k)$, a new weighted model is constructed as\n$$ U^{(k)}(y) := \\begin{cases} \\lambda_k\\left(f_{0}(y_k) + \\langle g_0(y_k;\\xi_k), y-y_k\\rangle + \\frac{\\mu}{2}\\|y-y_k\\|_2^2 + r(y)\\right) & \\text{if $y_k$ is feasible}\\\\\n\\lambda_k\\left(f_{s(y_k)}(y_k) + \\langle g_{s(y_k)}(y_k;\\xi_k), y-y_k\\rangle + \\frac{\\mu}{2}\\|y-y_k\\|_2^2\\right) & \\text{otherwise.}\n\\end{cases} $$","sha256_utf8":"56b372da01c51aa45e36c71706fbe387c995b99ad45aa6332fefc3d893f3bf78"},{"start":24967,"end":25315,"start_line":349,"end_line":352,"raw_tex":"The Lagrangian proximal dual averaging method then iterates by minimizing the aggregation of past models $M^{(k-1)}$ plus the new model $U^{(k)}$ (and an optional extra regularization term)\n\\begin{equation}\\label{eq:super-dual-method}\n    y_{k+1} = \\argmin\\left\\{M^{(k-1)}(y) + U^{(k)}(y) +\\frac{\\beta_k}{2}\\|y-y_0\\|_2^2 \\right\\} \\ .\n\\end{equation}","sha256_utf8":"a7aa07dd435930609e7401d110c4b67438cbf041490f2a9048ba57729cc59435"},{"start":35575,"end":35826,"start_line":429,"end_line":431,"raw_tex":"\\noindent {\\bf Assumption A.} The functions $f_s$ for $s=0\\dots m$ are each $\\mu>0$-strongly convex.\\\\\n\\noindent {\\bf Assumption B.} There exists some $x_\\mathtt{SL} \\in \\mathrm{dom\\ }\\partial r$ with $f_s(x_\\mathtt{SL}) <0$ for all \n $s = 1\\dots m$.\n","sha256_utf8":"00e07aacb9d00f0260ad6ac97961b1f57106122a27a49e142ebe39952df7869f"},{"start":35827,"end":36845,"start_line":433,"end_line":436,"raw_tex":"Note strong convexity ensures there exists a unique minimizer $x_\\mathtt{OPT} \\in \\mathrm{dom\\ }\\partial r$ of~\\eqref{eq:super-problem}. These two points $x_\\mathtt{OPT},x_\\mathtt{SL} \\in \\mathrm{dom\\ }\\partial r$ serve as important references that our analysis is done with respect to. We fix two subgradients of $r$ at these points: The subgradient $n_{x_\\mathtt{OPT}}\\in \\partial r(x_\\mathtt{OPT})$ is chosen such that $f_0(x) + r(x_\\mathtt{OPT}) + \\langle n_{x_\\mathtt{OPT}}, x-x_\\mathtt{OPT}\\rangle$ is minimized over $f_s(x)\\leq 0$ at $x_\\mathtt{OPT}$ for all $ s=1\\ldots m$. The subgradient $n_{x_\\mathtt{SL}} \\in \\partial r(x_\\mathtt{SL})$ can be chosen freely. %When $r$ is an indicator function for a closed convex set, $n_{x_\\mathtt{SL}}=0$ is a nice simplifying choice.\n\nThese two reference subgradients of $r$ facilitate considering two lower bounds of the objective $f_0+r$ for either $y\\in\\{x_{\\mathtt{OPT}}, x_{\\mathtt{SL}}\\}$, denoted by\n$$ h_{y}(x) := f_0(x) + r(y) + \\langle n_{y}, x-y\\rangle \\ . $$","sha256_utf8":"4fa3b118bac84773a5baff070abe2a9522ea6e5adce329c76b943dbae55d48d8"},{"start":36950,"end":37402,"start_line":438,"end_line":444,"raw_tex":"At each iteration $k$, we denote the relative difference between $x_k$ and $y\\in\\{x_{\\mathtt{OPT}}, x_{\\mathtt{SL}}\\}$ in (relaxed) objective value or feasibility on the selected constraint function $f_{s(x_k)}$ by\n\\begin{align}\n    \\delta_k(y) &:=\\begin{cases}\n        h_y(x_k) - h_y(y) & \\mathrm{if\\ } x_k \\mathrm{\\ is\\ feasible}\\\\\n        f_{s(x_k)}(x_k) - f_{s(x_k)}(y) & \\mathrm{otherwise.}\n    \\end{cases}\\label{eq:delta_k-definition}\n\\end{align}","sha256_utf8":"f28d0b17b78d4a93ed35b2d6c9ea50a43d08773692f6f4c038b3950e0395b10a"},{"start":37915,"end":38720,"start_line":447,"end_line":453,"raw_tex":"The sign of $\\delta_k(y)$ may vary. If $y = x_{\\mathtt{OPT}}$, then $\\delta_k(x_\\mathtt{OPT})$ is nonnegative, being lower bounded by the level of suboptimality or current infeasibility\n$$ \\delta_k(x_\\mathtt{OPT}) \\geq \\begin{cases}\n        h_{x_\\mathtt{OPT}}(x_k) - p_\\star & \\mathrm{if\\ } x_k \\mathrm{\\ is\\ feasible}\\\\\n        f_{s(x_k)}(x_k) & \\mathrm{otherwise}\n    \\end{cases} \\ \\geq 0 \\ . $$\nWhen $y = x_{\\mathtt{SL}}$ and $x_k$ is feasible, $\\delta_k(y)$ may be negative but is bounded below by\n$$ \\delta_k(x_\\mathtt{SL}) \\geq \\inf h_{x_{\\mathtt{SL}}} -h_{x_{\\mathtt{SL}}}(x_{\\mathtt{SL}}) > -\\infty \\ . $$ When $y = x_{\\mathtt{SL}}$ and $x_k$ is infeasible, $\\delta_k(y)$ is strictly positive, being bounded below by $$\\delta_k(x_\\mathtt{SL}) \\geq 0-\\max_{s=1\\dots m} f_s(x_\\mathtt{SL}) >0 \\ . $$\n","sha256_utf8":"32ea4dd85f3b6728fcd622b9806a66282605934ce8b54ce7b75b6f80fbb54bd0"},{"start":39311,"end":39841,"start_line":456,"end_line":462,"raw_tex":"\\noindent {\\bf Assumption C.} For both $y\\in\\{x_{\\mathtt{OPT}}, x_{\\mathtt{SL}}\\}$, there exist constants $L_0$, $L_1{\\color{blue}\\geq 0}$ such that every iterate $x_k$ has\n\\begin{equation}\\label{eq:NonLipschitz-condition}\n    \\begin{cases}\n        x_k \\mathrm{\\ is\\ feasible} & \\implies \\mathbb{E}_{\\xi_k}\\|g_0(x_k;\\xi_k) + n_{y}\\|_2^2 \\leq L_0^2 + L_1\\delta_k(y)\\\\\n        x_k \\mathrm{\\ is\\ not\\ feasible} & \\implies \\mathbb{E}_{\\xi_k}\\|g_{s(x_k)}(x_k;\\xi_k)\\|_2^2 \\leq L_0^2 + L_1\\delta_k(y) \\ . \n    \\end{cases}\n\\end{equation}","sha256_utf8":"1c37af056a250aa8c656538d03fbe0dff2a67345bfd2ac1bcb13394ab50eb6c5"},{"start":42699,"end":43168,"start_line":483,"end_line":487,"raw_tex":"To understand and bound such behaviors, we introduce the following two constants, dependent on the choice of stepsizes $\\alpha_k$ and associated dual weights $\\lambda_k$,\n\\begin{align}\n    T_0 &:= \\sup\\left\\{ k\\in\\{0,1,2\\dots\\} \\mid L_1\\alpha_k > 1\\right\\} \\ ,\\label{eq:T0} \\\\\n    C_0 &:= \\sum_{k=0}^{T_0}\\lambda_k \\max\\left\\{L_1 \\alpha_k - 1, 0 \\right\\}\\max\\{\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}), \\mathbb{E}_\\xi\\delta_k(x_\\mathtt{SL})\\} \\ . \\label{eq:C0}\n\\end{align}","sha256_utf8":"2cae97c3b85e29a042975d99d5068bf2190282bc79c3dbbb387213f6addc0df1"},{"start":55910,"end":56188,"start_line":596,"end_line":599,"raw_tex":"We prove a uniform convergence guarantee in terms of the primal gap\n\\begin{equation} \\label{eq:primalgap-definition}\n    \\mathtt{primal\\mbox{-}gap}_T := \\frac{\\sum_{k< T: s(x_k)=0} \\lambda_k h_{x_\\mathtt{OPT}}(x_k)}{\\sum_{k< T: s(x_k)=0} \\lambda_k } - p_\\star \\ ,\n\\end{equation}","sha256_utf8":"100397a4cd0fc6435a7d88369894c3dad8261148a1b461c84b2edd1b4db37f69"},{"start":56189,"end":56434,"start_line":600,"end_line":603,"raw_tex":"which utilizes a combination of the feasible objective values seen, the dual gap\n\\begin{equation} \\label{eq:dualgap-definition}\n    \\mathtt{dual\\mbox{-}gap}_T := p_\\star - \\inf \\frac{M^{(T-1)}}{\\sum_{k< T: s(x_k)=0} \\lambda_k }\\ ,\n\\end{equation}","sha256_utf8":"81f6b011d7c532737e9ce25657182ace10ba2017bf0b0f3474e328601b8867cb"},{"start":56435,"end":56887,"start_line":604,"end_line":605,"raw_tex":"which utilizes a combination of the subgradient lower bounds seen, and the distance to optimal.\nFor the primal and dual gaps to be well-defined, at least one feasible iterate must have been seen (i.e., $\\sum_{k< T: s(x_k)=0} \\lambda_k >0$). Our assumptions facilitate a bound on how long it takes for this to occur. We show the expected fraction of the dual weight that occurs on iterations with a feasible iterate (i.e., $s(x_k)=0$) is bounded below.\n","sha256_utf8":"afca0580e747fa77c0620c3f3bf52141506be72fa73ad0a39d4b5421f852c559"},{"start":58098,"end":58436,"start_line":614,"end_line":614,"raw_tex":"Once a feasible iterate occurs, our primal and dual convergence measures are well-defined. Alternatively, one could assume the initialization $x_0$ is feasible to ensure these quantities are well-defined. In either case, Subsection~\\ref{subsec:proof-main-result} proves the following primal-dual convergence guarantee as our main result.\n","sha256_utf8":"2be9270a8b422548f2963a9814a0c602e2c443483846748c5fea27ffc48c68f6"},{"start":70214,"end":70363,"start_line":704,"end_line":706,"raw_tex":"\\begin{equation} \\label{eq:Rk-definition}\nR_k(y) := \\left(\\frac{\\mu}{2}\\sum_{i=0}^{{k-1}} \\lambda_i\\right) \\mathbb{E}_\\xi\\|x_k-y\\|_2^2\n\\end{equation}","sha256_utf8":"2db352d4f3eac09e5ac88b8506fd22a7df5a65614173ddfc25ec86ddbb16e63f"},{"start":70364,"end":70563,"start_line":707,"end_line":707,"raw_tex":"To simplify notations, throughout our analysis, we denote $g_k = g_{s(x_k)}(x_k;\\xi_k)$ and $w_k = \\mu\\sum_{i=0}^{k} \\lambda_i$ (with the convention that $w_{-1}=0$ as the given summation is empty).\n","sha256_utf8":"fd027c96445ae9dce42d48498a3441c4805781551dbe4cd01da0103f36199980"},{"start":73297,"end":73464,"start_line":733,"end_line":735,"raw_tex":"\\begin{equation} \\label{eq:Dk-definition}\n    D_k := \\mathbb{E}_\\xi\\left[\\left(\\sum_{i< k:s(x_i)=0} \\lambda_i\\right) p_\\star - \\inf M^{(k-1)}\\right] \\ .\n\\end{equation}","sha256_utf8":"adde246f983d1ff20823e030a669ddfb38f93cc95cd713684b7ef283ea26b05f"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{thm:super-rate}\n    Under Assumptions A-C, for any primal stepsizes $\\alpha_k>0$ and dual weights $\\lambda_k>0$ satisfying~\\eqref{eq:alpha-lambda} with $\\bar\\beta=0$, the stochastic switching proximal subgradient method~\\eqref{eq:super-primal-method} has\n    \\begin{align*}\n        &\\mathbb{E}_\\xi \\left[\\left(\\frac{\\sum_{k< T: s(x_k)=0} \\lambda_k}{\\sum_{k=0}^{T-1} \\lambda_k}\\right)\\left(\\mathtt{primal\\mbox{-}gap}_T + \\mathtt{dual\\mbox{-}gap}_T\\right) + \\frac{\\mu}{2}\\|x_{T}-x_\\mathtt{OPT}\\|_2^2\\right] \\\\\n        &\\qquad \\qquad \\leq \\frac{ L_0^2\\sum_{k=0}^{T-1}\\lambda_k\\alpha_k + C_0}{\\sum_{k=0}^{T-1}\\lambda_k} \\ .\n    \\end{align*}\n\\end{theorem}","units":[{"start":58436,"end":59108,"start_line":615,"end_line":621,"raw_tex":"\\begin{theorem}\\label{thm:super-rate}\n    Under Assumptions A-C, for any primal stepsizes $\\alpha_k>0$ and dual weights $\\lambda_k>0$ satisfying~\\eqref{eq:alpha-lambda} with $\\bar\\beta=0$, the stochastic switching proximal subgradient method~\\eqref{eq:super-primal-method} has\n    \\begin{align*}\n        &\\mathbb{E}_\\xi \\left[\\left(\\frac{\\sum_{k< T: s(x_k)=0} \\lambda_k}{\\sum_{k=0}^{T-1} \\lambda_k}\\right)\\left(\\mathtt{primal\\mbox{-}gap}_T + \\mathtt{dual\\mbox{-}gap}_T\\right) + \\frac{\\mu}{2}\\|x_{T}-x_\\mathtt{OPT}\\|_2^2\\right] \\\\\n        &\\qquad \\qquad \\leq \\frac{ L_0^2\\sum_{k=0}^{T-1}\\lambda_k\\alpha_k + C_0}{\\sum_{k=0}^{T-1}\\lambda_k} \\ .\n    \\end{align*}\n\\end{theorem}","sha256_utf8":"8e8b9ecf8123876ef5a2a8b04c476ae720afe10571f35b9817db10bc1295b90d"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\\label{lem:Mk-minimization}\n    $y_{k+1}$ is the unique minimizer of $M^{(k)}(y)+\\frac{\\beta_k}{2}\\|y-y_0\\|_2^2$.\n\\end{lemma}\n\n\\begin{proof}\n% {\\color{blue}\nFirst note that when $y_k$ is infeasible (and so $s(y_k)\\neq 0$), $M^{(k)} = M^{(k-1)} + U^{(k)}$. From this, the result is immediate.\nNow consider when $y_k$ is feasible (and so $s(y_k)=0$). Since $y_{k+1}$ is the unique minimizer of $M^{(k-1)}(y)+U^{(k)}(y)+\\frac{\\beta_k}{2}\\|y-y_0\\|_2^2$, the necessary and sufficient optimality condition ensures $y_{k+1}$ is the unique solution to\n$$ 0 \\in \\nabla M^{(k-1)}(y_{k+1}) + \\lambda_k(g_0(y_k; \\xi_k) + \\mu(y_{k+1}-y_k) + \\partial r(y_{k+1})) + \\beta_k(y_{k+1}-y_0) \\ . $$\nRewriting this as $ \\nabla M^{(k-1)}(y_{k+1}) + \\lambda_k (g_0(y_k; \\xi_k)+\\mu(y_{k+1}-y_k)) + \\beta_k(y_{k+1}-y_0) \\in -\\lambda_k \\partial r(y_{k+1})$, we see that $n_{k+1}$ is the element of $\\partial r(y_{k+1})$ certifying the minimization's optimality. Therefore\n\\begin{align*}\n    0 &= \\nabla M^{(k-1)}(y_{k+1}) + \\lambda_k(g_0(y_k; \\xi_k) + \\mu(y_{k+1}-y_k)+n_{k+1}) + \\beta_k(y_{k+1}-y_0)\\\\\n    &= \\nabla M^{(k)}(y_{k+1}) + \\beta_k(y_{k+1}-y_0)\n\\end{align*}\nand so $y_{k+1}$ is a the unique minimizer of $M^{(k)}(y)+\\frac{\\beta_k}{2}\\|y-y_0\\|_2^2$. \n\n\\end{proof}\n\n\\begin{lemma}\\label{lem:standard-quadratic}\n    Any quadratic function of the form $Q(z) = c + \\langle d,z\\rangle + \\frac{b}{2}\\|z-\\hat z\\|^2_2$ with $b>0$ is equal to $Q(z) = \\min Q + \\frac{b}{2}\\|z - \\argmin Q\\|^2_2$.\n\\end{lemma}\n\n\\begin{lemma}\\label{lem:quad-calc}\n    The sum of quadratics $Q_i(z) = a_i + \\frac{b_i}{2}\\|z-z_i\\|_2^2$ for $i=1,2$ with $b_i > 0$ equals\n    $$ (Q_1+Q_2)(z) = a_{1+2} + \\frac{b_{1+2}}{2}\\|z-z_{1+2}\\|_2^2$$\n    where $a_{1+2} = a_1 + a_2 + \\frac{b_1b_2}{2(b_1+b_2)}\\|z_1-z_2\\|_2^2$, $b_{1+2} = b_1+b_2$, and $z_{1+2} = \\frac{b_1}{b_1+b_2}z_1 + \\frac{b_2}{b_1+b_2}z_2$.\n\\end{lemma}\n\n\\begin{theorem} \\label{thm:super-equivalence}\n    Let $\\{x_k\\}$ be the sequence of iterates of the primal method~\\eqref{eq:super-primal-method} with stepsizes $\\alpha_k>0$ and $\\{y_k\\}$ be the sequence of iterates of the dual method~\\eqref{eq:super-dual-method} for some $\\lambda_k,\\beta_k,\\mu\\geq 0$. If $x_0=y_0$, $\\beta_k=\\bar\\beta\\geq 0$ is constant, and\n    \\begin{equation}\n        \\alpha_k = \\frac{\\lambda_k}{\\mu\\sum_{i=0}^k\\lambda_i + \\bar\\beta}\\ , \\label{eq:alpha-lambda}\n    \\end{equation} \n    then these methods are equivalent, that is $x_k=y_k$.\n\\end{theorem}\n\n\\begin{proof}\n        We prove this by inductively showing that $x_k=y_k$ and for any iteration with $s(x_k)=0$ that $n_{k+1} = \\frac{1}{\\alpha_k}(x_k-\\alpha_k g_{0}(x_k;\\xi_k) -x_{k+1})$ (i.e., the subgradients of $r$  produced by the dual method are exactly those produced by the primal method via~\\eqref{eq:proximal-equivalence}).\n    By assumption, $x_0=y_0$. Suppose for induction that $x_i=y_i$ for all $i=0,\\dots,k$ and for all $i=0,\\dots,k-1$ with $s(x_i)=0$ that $n_{i+1} = \\frac{1}{\\alpha_i}(x_i-\\alpha_i g_{0}(x_i;\\xi_i) -x_{i+1})$. Let $\\bar g_i = g_0(y_i;\\xi_i) + n_{i+1}$ if $y_i$ is feasible and $g_{s(y_i)}(y_i;\\xi_i)$ otherwise. For each $i<k$, we claim $x_{i+1} =  x_i - \\alpha_i \\bar g_i$. If $x_i$ is infeasible, this is immediate. If $x_i$ is feasible, {\\color{blue} rearrangement of our inductive hypothesis of on $n_{i+1}$ gives the claim}. Inductively, we conclude for $i<k$ that $ x_{i+1} - x_0 = - \\sum_{t=0}^i \\alpha_t \\bar g_t. $\n    \n    By Lemma~\\ref{lem:Mk-minimization}, $y_{k+1}$ is the unique solution to\n    $\\sum_{i=0}^k \\lambda_i\\left(\\bar g_i + \\mu(y_{k+1}-y_i) \\right) + \\bar\\beta(y_{k+1}-y_0) = 0. $\n    Rearranging and simplifying this, it follows that\n    \\begin{align*}\n    y_{k+1} &= y_0 + \\frac{\\sum_{i=0}^k\\lambda_i\\mu(y_i-y_0)-\\sum_{i=0}^k\\lambda_i\\bar g_i}{\\sum_{i=0}^k\\lambda_i\\mu+\\bar\\beta}\\\\\n    &= y_0 -\\frac{\\sum_{i=0}^{k} \\lambda_{i}\\mu\\left( \\sum_{t=0}^{i-1} \\frac{\\lambda_{t} \\bar g_t}{\\sum_{s=0}^{t}\\lambda_{s}\\mu + \\bar \\beta} \\right) + \\sum_{i=0}^{k}\\lambda_i \\bar g_i}{\\sum_{i=0}^{k}\\lambda_i\\mu + \\bar\\beta}\\\\\n    &= y_0 -\\frac{\\sum_{t=0}^{k-1} \\lambda_t \\bar g_t\\frac{\\sum_{i=t+1}^k \\mu\\lambda_i}{\\sum_{i=0}^t \\mu\\lambda_{{\\color{blue} i}} +\\bar\\beta} + \\sum_{i=0}^{k}\\lambda_i \\bar g_i}{\\sum_{i=0}^{k}\\lambda_i\\mu+\\bar\\beta}\\\\\n    &= y_0 -\\frac{\\sum_{t=0}^{k-1} \\lambda_t \\bar g_t\\left(\\frac{\\sum_{i=t+1}^k \\mu\\lambda_i}{\\sum_{i=0}^t \\mu\\lambda_{{\\color{blue} i}} +\\bar\\beta}+1\\right)}{\\sum_{i=0}^{k}\\lambda_i\\mu+\\bar\\beta} - \\frac{\\lambda_k \\bar g_k}{\\sum_{i=0}^{k}\\lambda_i\\mu + \\bar\\beta}\\\\\n    &=  y_0 -\\sum_{t=0}^{k-1} \\frac{\\lambda_t \\bar g_t}{\\sum_{i=0}^{t}\\lambda_{{\\color{blue} i}}\\mu+\\bar\\beta} - \\frac{\\lambda_k \\bar g_k}{\\sum_{i=0}^{k}\\lambda_i\\mu + \\bar\\beta}\\\\\n    &= x_k - \\alpha_k\\bar g_k\n    \\end{align*}\n    where the second equality uses the inductive hypothesis for each $y_i-y_0 = x_i-x_0 = - \\sum_{t=0}^{i-1} \\alpha_t\\bar g_t$, the third exchanges summands, and the remainder combines and simplifies terms. If $x_k$ is infeasible, {\\color{blue} it is} immediate that $x_{k+1}=x_k-\\alpha_k\\bar g_k=y_{k+1}$. If $x_k$ is feasible, the above equality ensures\n    $$ \\frac{1}{\\alpha_k}(x_k - \\alpha_k g_0(x_k;\\xi_k) -y_{k+1}) = n_{k+1} \\in\\partial r(y_{k+1}) \\ . $$\n    Noting by~\\eqref{eq:proximal-equivalence}, $x_{k+1}$ is the unique solution to $\\frac{1}{\\alpha_k}(x_k-\\alpha_kg_0(x_k;\\xi_k) -z)\\in\\partial r(z)$, we must have $x_{k+1} {\\color{blue} = x_k - \\alpha_k(g_k + n_{k+1})} = y_{k+1}$ and $n_{k+1} =  \\frac{1}{\\alpha_k}(x_k - \\alpha_k g_0(x_k;\\xi_k) -x_{k+1})$ as required.\n%     } \n%     Recall the optimality condition for the proximal switching subgradient subproblem ensures $x_{k+1}$ is the unique solution to\n%     \\begin{equation*}\n%         \\begin{cases}\n%         x_{k+1} = x_k - \\alpha_k(g_0(x_k;\\xi_k)+n_{k+1}) & \\mathrm{if\\ } f_s(x_k) \\leq 0 \\ \\ \\forall s = 1...m \\\\\n%         x_{k+1}=x_k - \\alpha_k g_{s(x_k)}(x_k;\\xi_k) & \\mathrm{for\\ some}\\ f_s(x_k)>0\\ \\mathrm{otherwise} \\\\\n%         n_{k+1} \\in \\partial r(x_{k+1}) \\ . \n%     \\end{cases}\n%     \\end{equation*}\n%     Now we define \n%     \\begin{equation*}\n%         \\bar g_i(y_i;\\xi_i)=\\begin{cases}\n%         g_0(y_i;\\xi_i)+n_{i+1} & \\mathrm{if}\\ f_s(y_i)\\leq 0 \\quad \\forall s=1\\dots m,\\\\\n%         &n_{i+1} \\in \\partial r(y_{i+1})\\\\\n%         g_{s(y_i)}(y_i;\\xi_i) & \\mathrm{for\\ some}\\ f_s(y_i)>0\\ \\mathrm{otherwise} \\ .\n%     \\end{cases}\n%     \\end{equation*}\n%    Following Lemma~\\ref{lem:uniqness}, we first consider when $y_k$ is feasible,\n%     \\begin{align*}\n%           y_{k+1} & =\\argmin \\left\\{ M^{(k-1)}(y)+U^{k}(y)+\\frac{\\beta_k}{2}\\|y-y_0\\|_2^2\\right\\}\\\\\n%                   &=\\argmin \\left\\{\\sum_{i=0}^{k-1}\\lambda_i\\left( \\langle \\bar g_i(y_i;\\xi_i), y-y_i \\rangle + \\frac{\\mu}{2}\\|y-y_i\\|_2^2 \\right) \\right.\\\\ & \\left.\n%                    + \\lambda_k \\left(\\langle g_0(y_k;\\xi_k), y-y_k \\rangle +\\frac{\\mu}{2}\\|y-y_k\\|_2^2 + r(y)\\right) + \\frac{\\bar \\beta}{2}\\|y-y_0\\|_2^2 \\right\\},\n%     \\end{align*}\n% where the second equality holds by removing some constant terms in $M^{(k-1)}(y)$ and $U^{k}(y)$.\n%     \\begin{align*}\n%           y_{k+1} & =\\argmin \\left\\{ M^{(k-1)}(y)+U^{k}(y)+\\frac{\\beta_k}{2}\\|y-y_0\\|_2^2\\right\\}\\\\\n%                   &=\\argmin \\left\\{\\sum_{i=0}^{k-1}\\lambda_i\\left(\\langle g_i(y_i;\\xi_i), y-y_i \\rangle + \\frac{\\mu}{2}\\|y-y_i\\|_2^2 \\right)+  \\sum_{i<k \\colon  s(y_i)=0} \\lambda_i\\langle n_{i+1}, y-y_{i+1}\\rangle \\right.\n%                   \\\\ & \\qquad \\qquad \\left.\n%                    + \\lambda_k \\left(\\langle g_0(y_k;\\xi_k), y-y_k \\rangle +\\frac{\\mu}{2}\\|y-y_k\\|_2^2 + r(y)\\right) + \\frac{\\bar \\beta}{2}\\|y-y_0\\|_2^2 \\right\\},\n%     \\end{align*}\n%     where the second equality holds by removing some constant terms in $M^{(k-1)}(y)$ and $U^{(k)}(y)$.\n%    Taking the subgradient of the right hand side and let the it be zero, and combining with $\\bar g_k(y_k;\\xi_k) = g_0(y_k;\\xi_i)+n_{k+1}$, we have\n%    \\begin{equation*}\n%        \\sum_{i=0}^k \\lambda_i\\left(\\bar g_i(y_i;\\xi_i) + \\mu(y_{k+1}-y_i) \\right) + \\beta(y_{k+1}-y_0) = 0,\n%    \\end{equation*}\n%    which remains same when $y_k$ is infeasible since $\\bar g_k(y_k;\\xi_k)=g_{s(y_k)}(y_k;\\xi_k)$.  \n%    By rearranging the equation, we have\n%    \\begin{equation}\\label{eq:dual-iterates-formula}\n%        y_{k+1}-y_0 = \\frac{\\sum_{i=0}^k\\lambda_i\\mu(y_i-y_0)-\\sum_{i=0}^k\\lambda_i\\bar g_i(y_i;\\xi_i)}{\\sum_{i=0}^k\\lambda_i\\mu+\\bar\\beta}\n%    \\end{equation}\n%  We prove the sequence of iterates $y_k$ given by the dual method is equivalent to the sequence of iterates $x_k$ given by the primal method by inductively showing the formula\n%  \\begin{equation}\\label{eq:recursive-formula}\n%      y_{k+1}-y_0 = -\\sum_{t=0}^k \\frac{\\lambda_t \\bar g_t(y_t;\\xi_t)}{\\sum_{s=0}^t \\lambda_s\\mu+\\bar\\beta},\n%  \\end{equation}\n%    and it is easy to see that the right hand side is the closed form solution of $x_{k+1}-x_0$.\n% One can readily verify this holds $k=0$. So we inductively assume the formula~\\eqref{eq:recursive-formula} holds for all $y_i-y_0$ with $i \\leq k$ and derive it for $y_{k+1}-y_0$.\n% Combining the assumptions with~\\eqref{eq:dual-iterates-formula}, the claimed result follows as\n% \\begin{align*}\n%     y_{k+1} -y_0 &= -\\frac{\\sum_{i=0}^{k} \\lambda_{i}\\mu\\left( \\sum_{t=0}^{i-1} \\frac{\\lambda_{t} \\bar g_t(y_t;\\xi_t)}{\\sum_{s=0}^{t}\\lambda_{s}\\mu + \\bar \\beta} \\right) + \\sum_{i=0}^{k}\\lambda_i \\bar g_i(y_i;\\xi_i)}{\\sum_{i=0}^{k}\\lambda_i\\mu + \\bar\\beta}\\\\\n%     &= -\\frac{\\sum_{t=0}^{k-1} \\lambda_t \\bar g_t(y_t;\\xi_t)\\frac{\\sum_{i=t+1}^k \\mu\\lambda_i}{\\sum_{i=0}^t \\mu\\lambda_t +\\bar\\beta} + \\sum_{i=0}^{k}\\lambda_i \\bar g_i(y_i;\\xi_i)}{\\sum_{i=0}^{k}\\lambda_i\\mu+\\bar\\beta}\\\\\n%     &= -\\frac{\\sum_{t=0}^{k-1} \\lambda_t \\bar g_t(y_t;\\xi_t)\\left(\\frac{\\sum_{i=t+1}^k \\mu\\lambda_i}{\\sum_{i=0}^t \\mu\\lambda_t +\\bar\\beta}+1\\right)}{\\sum_{i=0}^{k}\\lambda_i\\mu+\\bar\\beta} - \\frac{\\lambda_k \\bar g_k(y_k;\\xi_k)}{\\sum_{i=0}^{k}\\lambda_i\\mu + \\bar\\beta}\\\\\n%     &=  -\\sum_{t=0}^{k-1} \\frac{\\lambda_t \\bar g_t(y_t;\\xi_t)}{\\sum_{i=0}^{t}\\lambda_t\\mu+\\bar\\beta} - \\frac{\\lambda_k \\bar g_k(y_k;\\xi_k)}{\\sum_{i=0}^{k}\\lambda_i\\mu + \\bar\\beta} \n% \\end{align*}\n% where the first equality uses the inductive hypothesis for each $y_i-y_0$, the second exchanges summands, and the third and fourth combine and simplify terms.\n% Which tells us $y_k=x_k$ for each $k$, hence the primal method~\\eqref{eq:super-primal-method}and dual method~\\eqref{eq:super-dual-method}are equivalent. \n\\end{proof}\n\n\\begin{lemma}\\label{lem:switching-proximal-primal-induction}\n    Under Assumptions A-C, the switching proximal subgradient method~\\eqref{eq:super-primal-method} with $\\alpha_k=\\lambda_k/\\mu\\sum_{i=0}^k\\lambda_i$ has for either $y\\in\\{x_{\\mathtt{OPT}}, x_{\\mathtt{SL}}\\}$\n    $$R_{k+1}(y) \\leq R_k(y) - \\frac{\\lambda_k}{2}\\left(\\left(2-L_1\\alpha_k\\right)\\mathbb{E}_\\xi\\delta_{k}(y) - L_0^2\\alpha_k\\right) \\ . $$\n\\end{lemma}\n\n\\begin{proof}\n    This proof follows a standard analysis technique, directly expanding the definition of $R_{k+1}(y)$. First, suppose $x_k$ is feasible. Then\n    \\begin{align*}\n    R_{k+1}(y) &= \\frac{w_k}{2}\\mathbb{E}_\\xi\\|\\mathrm{prox}_{\\alpha_k,r}(x_k - \\alpha_k g_k) - \\mathrm{prox}_{\\alpha_k,r}(y +\\alpha_k n_{y})\\|_2^2\\\\\n    &\\leq \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k - \\alpha_k(g_k+n_{y}) - y\\|_2^2\\\\\n    &= \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k-y\\|_2^2 - \\lambda_k\\mathbb{E}_\\xi\\langle g_k + n_{y}, x_k-y\\rangle +\\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{y}\\|_2^2\\\\\n    &\\leq \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k-y\\|_2^2 - \\lambda_k\\mathbb{E}_\\xi(h_y(x_k)-h_y(y)+\\frac{\\mu}{2}\\|x_k - y\\|_2^2) + \\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{y}\\|_2^2\\\\\n    &= R_k(y) - \\lambda_k\\mathbb{E}_\\xi\\delta_k(y)+\\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{y}\\|_2^2\n   \\end{align*}\n    where the first line uses that $\\mathrm{prox}_{\\alpha_k,r}(y +\\alpha_k n_{y}) = y$, the second uses the nonexpansiveness of the proximal operator~\\cite[Proposition 12.19]{RockWets98}, the third factors the norm squared and uses $\\alpha_k = \\lambda_k/w_k$, the fourth uses the strong convexity of $h_y$, {\\color{blue} and the fifth follows from definitions as $R_k(y) = \\frac{w_k - \\lambda_k\\mu}{2}\\mathbb{E}_\\xi\\|x_k-y\\|_2^2$ and $\\delta_k(y)=h_y(x_k)-h_y(y)$}.\n    Then, applying the bound~\\eqref{eq:NonLipschitz-condition} gives the claim.\n    Similarly, supposing $x_k$ is infeasible,\n    \\begin{align*}\n    R_{k+1}(y) &= \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k - \\alpha_k g_k - y\\|_2^2\\\\\n    &= \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k-y\\|_2^2 - \\lambda_k\\mathbb{E}_\\xi\\langle g_k, x_k-y\\rangle + \\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k\\|_2^2\\\\\n    &\\leq \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k-y\\|_2^2- \\lambda_k\\mathbb{E}_\\xi(f_{s(x_k)}(x_k)-f_{s(x_k)}(y) +\\frac{\\mu}{2}\\|x_k - y\\|_2^2) + \\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k\\|_2^2\\\\\n    &= R_k(y) - \\lambda_k\\mathbb{E}_\\xi\\delta_k(y)+\\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k\\|_2^2 \n   \\end{align*}\n   using strong convexity of $f_{s(x_k)}$. Applying~\\eqref{eq:NonLipschitz-condition} completes the proof.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:switching-proximal-dual-induction}\n    Under Assumptions A-C, the switching proximal subgradient method~\\eqref{eq:super-primal-method} with $\\alpha_k=\\lambda_k/\\mu\\sum_{i=0}^k\\lambda_i$ has\n     $$ D_{k+1} \\leq D_k - \\frac{\\lambda_k}{2}\\begin{cases}\n         \\left(2-L_1\\alpha_k\\right)\\mathbb{E}_\\xi\\delta_{k}(x_\\mathtt{OPT}) - L_0^2\\alpha_k & \\text{if } x_k \\text{ is feasible}\\\\\n         {\\color{blue}\\left(2-L_1\\alpha_k\\right)\\mathbb{E}_\\xi\\delta_{k}(x_\\mathtt{OPT}) + 2\\mathbb{E}_\\xi f_{s(x_k)}(x_\\mathtt{OPT}) - L_0^2\\alpha_k} & \\text{otherwise.}\n     \\end{cases} $$\n\\end{lemma}\n\n\\begin{proof}\nObserve that one can rewrite $M^{(k)}(x) = Q_1(x) + Q_2(x)$ as the sum of two quadratics where $Q_1(x) = M^{(k-1)}(x)$ and $Q_2$ depends on whether $x_k$ is feasible. In the notation of Lemma~\\ref{lem:quad-calc}, Lemmas~\\ref{lem:Mk-minimization} {\\color{blue} and~\\ref{lem:standard-quadratic}} ensure $Q_1$ has $a_1 = \\inf M^{(k-1)}$, $b_1 = \\mu\\sum_{i=0}^{k-1}\\lambda_i$, and $z_1 = x_{k}${\\color{blue}, since $\\bar \\beta = 0$ here}. To determine $Q_2$, first suppose $x_k$ is feasible. Then we have\n$$Q_2(x) = \\lambda_k \\left(f_{0}(x_k) + \\langle g_k, x-x_k \\rangle +\\frac{\\mu}{2}\\|x - x_k\\|_2^2 + r(x_{k+1}) + \\langle n_{k+1}, x-x_{k+1}\\rangle\\right) \\ . $$\nThis quadratic can be written in the form $Q_2(z) = a_2 + \\frac{b_2}{2}\\|z-z_2\\|_2^2$ with\n\\begin{align*}\n    a_2 &= \\lambda_k\\left(f_0(x_k) + r(x_{k+1}) + \\alpha_k\\langle n_{k+1}, g_{k}+n_{k+1}\\rangle - \\frac{1}{2\\mu}\\|g_k + n_{k+1}\\|_2^2 \\right)\\ , \\\\\n    b_2 &= \\mu\\lambda_k \\ , \\qquad z_2 = x_k - \\frac{g_k+n_{k+1}}{\\mu} \\ . \n\\end{align*}\nBy Lemma~\\ref{lem:quad-calc}, the expected minimum value of the updated model $\\mathbb{E}_\\xi \\inf M^{(k)}$ is\n\\begin{align*}\n    &\\mathbb{E}_\\xi \\inf M^{(k-1)} + \\lambda_k \\mathbb{E}_\\xi\\Bigg(f_0(x_k) + r(x_{k+1}) + \\alpha_k \\langle n_{k+1}, g_{k}+n_{k+1}\\rangle - \\frac{1}{2\\mu}\\|g_k + n_{k+1}\\|_2^2\\\\\n    & \\qquad\\qquad\\qquad\\qquad\\qquad + \\frac{\\sum_{i=0}^{k-1}\\lambda_i }{2\\mu \\sum_{i=0}^k \\lambda_i}\\left\\|g_{k}+n_{k+1}\\right\\|_2^2\\Bigg) \\ . \n\\end{align*}\nFrom this, we conclude the lower bound\n\\begin{align*}\n    \\mathbb{E}_\\xi \\inf M^{(k)} &\\geq \\mathbb{E}_\\xi \\inf M^{(k-1)} + \\lambda_k \\mathbb{E}_\\xi\\Bigg(f_0(x_k) + r(x_\\mathtt{OPT}) + \\langle n_{x_\\mathtt{OPT}}, x_{k+1}-x_\\mathtt{OPT}\\rangle  \\\\\n    & \\quad + \\alpha_k \\langle n_{k+1}, g_{k}+n_{k+1}\\rangle - \\frac{1}{2\\mu}\\|g_k + n_{k+1}\\|_2^2 + \\frac{\\sum_{i=0}^{k-1}\\lambda_i }{2\\mu \\sum_{i=0}^k \\lambda_i}\\left\\|g_{k}+n_{k+1}\\right\\|_2^2\\Bigg)\\\\\n    &= \\mathbb{E}_\\xi \\inf M^{(k-1)} + \\lambda_k \\mathbb{E}_\\xi\\Bigg(\\delta_k(x_\\mathtt{OPT}) + p_\\star + \\langle n_{x_\\mathtt{OPT}}, -\\alpha_k(g_k + n_{k+1})\\rangle  \\\\\n    & \\quad + \\alpha_k \\langle n_{k+1}, g_{k}+n_{k+1}\\rangle - \\frac{\\alpha_k}{2}\\|g_k + n_{k+1}\\|_2^2\\Bigg)\\\\\n    &= \\mathbb{E}_\\xi\\inf M^{(k-1)} + \\lambda_k \\left(\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) + p_\\star - \\frac{\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{x_\\mathtt{OPT}}\\|_2^2 { \\color{blue} + \\frac{\\alpha_k}{2}\\|n_{k+1}-n_{x_\\mathtt{OPT}} \\|_2^2}\\right)\\\\\n    & {\\color{blue} \\geq \\mathbb{E}_\\xi\\inf M^{(k-1)} + \\lambda_k \\left(\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) + p_\\star - \\frac{\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{x_\\mathtt{OPT}}\\|_2^2\\right)}\n\\end{align*}\nwhere the first inequality lower bounds $r(x_{k+1})$ by $r(x_\\mathtt{OPT}) + \\langle n_{x_\\mathtt{OPT}}, x_{k+1}-x_\\mathtt{OPT}\\rangle$, the first equality applies the definitions of $\\delta_k(x_\\mathtt{OPT})$ in~\\eqref{eq:delta_k-definition}, $\\alpha_k$ in~\\eqref{eq:alpha-lambda} {\\color{blue} with $\\bar \\beta = 0$, and that $x_{k+1}-x_k=-\\alpha_k(g_k+n_{k+1})$}, and the {\\color{blue} second} equality combines and simplifies terms.\nIn terms of $D_k = \\mathbb{E}_\\xi[\\sum_{i< k:s(x_i)=0} \\lambda_i p_\\star - \\inf M^{(k-1)}]$, this gives the following \nrecurrence\n$$ D_{k+1} \\leq D_k - \\lambda_k \\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) + \\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{x_\\mathtt{OPT}}\\|_2^2 \\ . $$\nApplying~\\eqref{eq:NonLipschitz-condition} gives the claim in this case.\nNow suppose $x_k$ is infeasible. Then, noting $Q_2$ minimizes at $x_k - \\frac{g_{k}}{\\mu}$, Lemma~\\ref{lem:standard-quadratic} ensures\n\\begin{align*}\n    Q_2(x) &= \\lambda_k\\left(f_{s(x_k)}(x_k) - \\frac{1}{2\\mu}\\|g_k \\|_2^2 + \\frac{\\mu}{2}\\left\\|x - \\left(x_k - \\frac{g_{k}}{\\mu}\\right)\\right\\|_2^2\\right) \\ .\n\\end{align*}\nThen by Lemma~\\ref{lem:quad-calc}, the expected minimum value of the updated model $\\mathbb{E}_\\xi \\inf M^{(k)}$ is given by\n\\begin{align*}\n    &\\mathbb{E}_\\xi\\inf  M^{(k-1)} + \\lambda_k\\mathbb{E}_\\xi\\left(f_{s(x_k)}(x_k) - \\frac{1}{2\\mu}\\|g_k \\|_2^2\\right) + \\frac{\\lambda_k\\sum_{i=0}^{k-1}\\lambda_i }{{\\color{blue} 2}\\mu \\sum_{i=0}^k \\lambda_i}\\mathbb{E}_\\xi\\left\\|g_{k}\\right\\|_2^2 \\\\\n    &= \\mathbb{E}_\\xi\\inf M^{(k-1)} + \\lambda_k\\mathbb{E}_\\xi f_{s(x_k)}(x_k) - \\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k\\|_2^2 \\ . \n\\end{align*}\n{\\color{blue}\nIn terms of $D_k = \\mathbb{E}_\\xi[\\sum_{i< k:s(x_i)=0} \\lambda_i p_\\star - \\mathbb{E}_\\xi\\inf M^{(k-1)}]$, this gives the recurrence\n$$ D_{k+1} = D_k - \\lambda_k \\mathbb{E}_\\xi\\left[\\delta_{k}(x_\\mathtt{OPT}) + f_{s(x_k)}(x_{\\mathtt{OPT}})\\right] + \\frac{\\lambda_k\\alpha_k}{2} \\mathbb{E}_\\xi\\|g_k\\|_2^2 \\ . $$\nBounding $\\mathbb{E}_\\xi\\|g_k\\|_2^2$ by~\\eqref{eq:NonLipschitz-condition} gives the claim in this last case.\n}\n% Noting $f_{s(x_k)}(x_\\mathtt{OPT})\\leq 0$, we conclude the lower bound\n% $$\\mathbb{E}_\\xi \\inf M^{(k)} \\geq \\mathbb{E}_\\xi\\inf M^{(k-1)} + \\lambda_k \\left(\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) - \\frac{\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k\\|_2^2\\right) \\ .$$\n\\end{proof}\n\n\\subsubsection{Proof of Theorem~\\ref{thm:super-rate}} \\label{subsec:proof-main-result}\nApplying Lemma~\\ref{lem:switching-proximal-primal-induction} with $y=x_\\mathtt{OPT}$ from $k=0$ to $T-1$ yields\n$$R_T(x_\\mathtt{OPT}) + \\sum_{k=0}^{T-1} \\frac{\\lambda_k}{2}\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) \\leq R_0(x_\\mathtt{OPT}) + \\sum_{k=0}^{T-1}\\frac{L_0^2\\lambda_k\\alpha_k}{2} + \\sum_{k=0}^{T-1} \\frac{\\lambda_k}{2}\\left(L_1\\alpha_k - 1 \\right)\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT})\\ .$$\nSimilarly, applying Lemma~\\ref{lem:switching-proximal-dual-induction} from $k=0$ to $T-1$ yields\n\\begin{align*}\n    D_T &+ \\sum_{k=0}^{T-1} \\frac{\\lambda_k}{2}(\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT})) + {\\color{blue}\\sum_{k<T:s(x_k)>0} \\lambda_k\\mathbb{E}_\\xi f_{s(x_k)}(x_\\mathtt{OPT})}\\\\\n    &\\leq D_0 + \\sum_{k=0}^{T-1}\\frac{L_0^2\\lambda_k\\alpha_k}{2} + \\sum_{k=0}^{T-1} \\frac{\\lambda_k}{2}\\left(L_1\\alpha_k - 1 \\right)\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT})\\ .\n\\end{align*}\nNoting $R_0(x_\\mathtt{OPT})=D_0=0$ and the last summation in each bound is at most half our initial blow-up constant $\\frac{1}{2}C_0$, the sum of these inequalities provides a bound of\n$$ R_T(x_\\mathtt{OPT}) + D_T + \\sum_{k=0}^{T-1} \\lambda_k\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) {\\color{blue} + \\sum_{k<T:s(x_k)>0} \\lambda_k\\mathbb{E}_\\xi}f_{s(x_k)}(x_\\mathtt{OPT}) \\leq \\sum_{k=0}^{T-1}L_0^2\\lambda_k\\alpha_k + C_0 \\ . $$\nLower bounding each $\\mathbb{E}_\\xi[\\delta_k(x_\\mathtt{OPT}) {\\color{blue}+  f_{s(x_k)}(x_\\mathtt{OPT})}]$ with $s(x_k)\\neq 0$ by zero {\\color{blue} and dividing through by $\\sum_{k=0}^{T-1} \\lambda_k$ establishes that\n$$ \\frac{1}{\\sum_{k=0}^{T-1} \\lambda_k}\\left(R_T(x_\\mathtt{OPT}) + D_T + \\sum_{k<T:s(x_k)=0} \\lambda_k\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT})\\right) \\leq \\frac{\\sum_{k=0}^{T-1}L_0^2\\lambda_k\\alpha_k + C_0}{\\sum_{k=0}^{T-1} \\lambda_k} \\ . $$\nSubstituting the definitions of $R_T$ and $D_T$ from~\\eqref{eq:Rk-definition} and~\\eqref{eq:Dk-definition}, the left-hand side above is equal to\n$$ \\mathbb{E}_\\xi\\left[\\frac{\\mu}{2}\\|x_T-x_\\mathtt{OPT}\\|^2_2 + \\frac{1}{{\\sum_{k=0}^{T-1} \\lambda_k}}\\left(\\sum_{k<T:s(x_k)=0} \\lambda_k h_{x_\\mathtt{OPT}}(x_k)- \\inf M^{(T-1)}\\right)\\right]\\ . $$\nRecalling the primal and dual gaps definitions in~\\eqref{eq:primalgap-definition} and~\\eqref{eq:dualgap-definition} gives our main theorem.}\n\n    ","units":[{"start":25495,"end":25633,"start_line":356,"end_line":358,"raw_tex":"\\begin{lemma}\\label{lem:Mk-minimization}\n    $y_{k+1}$ is the unique minimizer of $M^{(k)}(y)+\\frac{\\beta_k}{2}\\|y-y_0\\|_2^2$.\n\\end{lemma}","sha256_utf8":"45cd133fdd2935beff93321f67d650300459a1f80d298c853692fc925aef678d"},{"start":25634,"end":26755,"start_line":359,"end_line":371,"raw_tex":"\\begin{proof}\n% {\\color{blue}\nFirst note that when $y_k$ is infeasible (and so $s(y_k)\\neq 0$), $M^{(k)} = M^{(k-1)} + U^{(k)}$. From this, the result is immediate.\nNow consider when $y_k$ is feasible (and so $s(y_k)=0$). Since $y_{k+1}$ is the unique minimizer of $M^{(k-1)}(y)+U^{(k)}(y)+\\frac{\\beta_k}{2}\\|y-y_0\\|_2^2$, the necessary and sufficient optimality condition ensures $y_{k+1}$ is the unique solution to\n$$ 0 \\in \\nabla M^{(k-1)}(y_{k+1}) + \\lambda_k(g_0(y_k; \\xi_k) + \\mu(y_{k+1}-y_k) + \\partial r(y_{k+1})) + \\beta_k(y_{k+1}-y_0) \\ . $$\nRewriting this as $ \\nabla M^{(k-1)}(y_{k+1}) + \\lambda_k (g_0(y_k; \\xi_k)+\\mu(y_{k+1}-y_k)) + \\beta_k(y_{k+1}-y_0) \\in -\\lambda_k \\partial r(y_{k+1})$, we see that $n_{k+1}$ is the element of $\\partial r(y_{k+1})$ certifying the minimization's optimality. Therefore\n\\begin{align*}\n    0 &= \\nabla M^{(k-1)}(y_{k+1}) + \\lambda_k(g_0(y_k; \\xi_k) + \\mu(y_{k+1}-y_k)+n_{k+1}) + \\beta_k(y_{k+1}-y_0)\\\\\n    &= \\nabla M^{(k)}(y_{k+1}) + \\beta_k(y_{k+1}-y_0)\n\\end{align*}\nand so $y_{k+1}$ is a the unique minimizer of $M^{(k)}(y)+\\frac{\\beta_k}{2}\\|y-y_0\\|_2^2$. \n\n\\end{proof}","sha256_utf8":"e391eb312d7a1c554657a50bb24c5e898798bf6b7e8f799b3d640033e19528f9"},{"start":27230,"end":27461,"start_line":375,"end_line":377,"raw_tex":"\\begin{lemma}\\label{lem:standard-quadratic}\n    Any quadratic function of the form $Q(z) = c + \\langle d,z\\rangle + \\frac{b}{2}\\|z-\\hat z\\|^2_2$ with $b>0$ is equal to $Q(z) = \\min Q + \\frac{b}{2}\\|z - \\argmin Q\\|^2_2$.\n\\end{lemma}","sha256_utf8":"94e87a9d279c7ef2b78321320d813bce7acb85f8229b96503dbf316d9807ddda"},{"start":27464,"end":27845,"start_line":380,"end_line":384,"raw_tex":"\\begin{lemma}\\label{lem:quad-calc}\n    The sum of quadratics $Q_i(z) = a_i + \\frac{b_i}{2}\\|z-z_i\\|_2^2$ for $i=1,2$ with $b_i > 0$ equals\n    $$ (Q_1+Q_2)(z) = a_{1+2} + \\frac{b_{1+2}}{2}\\|z-z_{1+2}\\|_2^2$$\n    where $a_{1+2} = a_1 + a_2 + \\frac{b_1b_2}{2(b_1+b_2)}\\|z_1-z_2\\|_2^2$, $b_{1+2} = b_1+b_2$, and $z_{1+2} = \\frac{b_1}{b_1+b_2}z_1 + \\frac{b_2}{b_1+b_2}z_2$.\n\\end{lemma}","sha256_utf8":"8a8870c7a9aca263897eb9bcef04a69fad1c8246254d85367c3544b38f321dd4"},{"start":44451,"end":45023,"start_line":494,"end_line":500,"raw_tex":"\\begin{theorem} \\label{thm:super-equivalence}\n    Let $\\{x_k\\}$ be the sequence of iterates of the primal method~\\eqref{eq:super-primal-method} with stepsizes $\\alpha_k>0$ and $\\{y_k\\}$ be the sequence of iterates of the dual method~\\eqref{eq:super-dual-method} for some $\\lambda_k,\\beta_k,\\mu\\geq 0$. If $x_0=y_0$, $\\beta_k=\\bar\\beta\\geq 0$ is constant, and\n    \\begin{equation}\n        \\alpha_k = \\frac{\\lambda_k}{\\mu\\sum_{i=0}^k\\lambda_i + \\bar\\beta}\\ , \\label{eq:alpha-lambda}\n    \\end{equation} \n    then these methods are equivalent, that is $x_k=y_k$.\n\\end{theorem}","sha256_utf8":"44405636a25282de182c671cdf60b8de28614d5286a6c50fe45d75e4c7767b14"},{"start":45024,"end":52980,"start_line":501,"end_line":574,"raw_tex":"\\begin{proof}\n        We prove this by inductively showing that $x_k=y_k$ and for any iteration with $s(x_k)=0$ that $n_{k+1} = \\frac{1}{\\alpha_k}(x_k-\\alpha_k g_{0}(x_k;\\xi_k) -x_{k+1})$ (i.e., the subgradients of $r$  produced by the dual method are exactly those produced by the primal method via~\\eqref{eq:proximal-equivalence}).\n    By assumption, $x_0=y_0$. Suppose for induction that $x_i=y_i$ for all $i=0,\\dots,k$ and for all $i=0,\\dots,k-1$ with $s(x_i)=0$ that $n_{i+1} = \\frac{1}{\\alpha_i}(x_i-\\alpha_i g_{0}(x_i;\\xi_i) -x_{i+1})$. Let $\\bar g_i = g_0(y_i;\\xi_i) + n_{i+1}$ if $y_i$ is feasible and $g_{s(y_i)}(y_i;\\xi_i)$ otherwise. For each $i<k$, we claim $x_{i+1} =  x_i - \\alpha_i \\bar g_i$. If $x_i$ is infeasible, this is immediate. If $x_i$ is feasible, {\\color{blue} rearrangement of our inductive hypothesis of on $n_{i+1}$ gives the claim}. Inductively, we conclude for $i<k$ that $ x_{i+1} - x_0 = - \\sum_{t=0}^i \\alpha_t \\bar g_t. $\n    \n    By Lemma~\\ref{lem:Mk-minimization}, $y_{k+1}$ is the unique solution to\n    $\\sum_{i=0}^k \\lambda_i\\left(\\bar g_i + \\mu(y_{k+1}-y_i) \\right) + \\bar\\beta(y_{k+1}-y_0) = 0. $\n    Rearranging and simplifying this, it follows that\n    \\begin{align*}\n    y_{k+1} &= y_0 + \\frac{\\sum_{i=0}^k\\lambda_i\\mu(y_i-y_0)-\\sum_{i=0}^k\\lambda_i\\bar g_i}{\\sum_{i=0}^k\\lambda_i\\mu+\\bar\\beta}\\\\\n    &= y_0 -\\frac{\\sum_{i=0}^{k} \\lambda_{i}\\mu\\left( \\sum_{t=0}^{i-1} \\frac{\\lambda_{t} \\bar g_t}{\\sum_{s=0}^{t}\\lambda_{s}\\mu + \\bar \\beta} \\right) + \\sum_{i=0}^{k}\\lambda_i \\bar g_i}{\\sum_{i=0}^{k}\\lambda_i\\mu + \\bar\\beta}\\\\\n    &= y_0 -\\frac{\\sum_{t=0}^{k-1} \\lambda_t \\bar g_t\\frac{\\sum_{i=t+1}^k \\mu\\lambda_i}{\\sum_{i=0}^t \\mu\\lambda_{{\\color{blue} i}} +\\bar\\beta} + \\sum_{i=0}^{k}\\lambda_i \\bar g_i}{\\sum_{i=0}^{k}\\lambda_i\\mu+\\bar\\beta}\\\\\n    &= y_0 -\\frac{\\sum_{t=0}^{k-1} \\lambda_t \\bar g_t\\left(\\frac{\\sum_{i=t+1}^k \\mu\\lambda_i}{\\sum_{i=0}^t \\mu\\lambda_{{\\color{blue} i}} +\\bar\\beta}+1\\right)}{\\sum_{i=0}^{k}\\lambda_i\\mu+\\bar\\beta} - \\frac{\\lambda_k \\bar g_k}{\\sum_{i=0}^{k}\\lambda_i\\mu + \\bar\\beta}\\\\\n    &=  y_0 -\\sum_{t=0}^{k-1} \\frac{\\lambda_t \\bar g_t}{\\sum_{i=0}^{t}\\lambda_{{\\color{blue} i}}\\mu+\\bar\\beta} - \\frac{\\lambda_k \\bar g_k}{\\sum_{i=0}^{k}\\lambda_i\\mu + \\bar\\beta}\\\\\n    &= x_k - \\alpha_k\\bar g_k\n    \\end{align*}\n    where the second equality uses the inductive hypothesis for each $y_i-y_0 = x_i-x_0 = - \\sum_{t=0}^{i-1} \\alpha_t\\bar g_t$, the third exchanges summands, and the remainder combines and simplifies terms. If $x_k$ is infeasible, {\\color{blue} it is} immediate that $x_{k+1}=x_k-\\alpha_k\\bar g_k=y_{k+1}$. If $x_k$ is feasible, the above equality ensures\n    $$ \\frac{1}{\\alpha_k}(x_k - \\alpha_k g_0(x_k;\\xi_k) -y_{k+1}) = n_{k+1} \\in\\partial r(y_{k+1}) \\ . $$\n    Noting by~\\eqref{eq:proximal-equivalence}, $x_{k+1}$ is the unique solution to $\\frac{1}{\\alpha_k}(x_k-\\alpha_kg_0(x_k;\\xi_k) -z)\\in\\partial r(z)$, we must have $x_{k+1} {\\color{blue} = x_k - \\alpha_k(g_k + n_{k+1})} = y_{k+1}$ and $n_{k+1} =  \\frac{1}{\\alpha_k}(x_k - \\alpha_k g_0(x_k;\\xi_k) -x_{k+1})$ as required.\n%     } \n%     Recall the optimality condition for the proximal switching subgradient subproblem ensures $x_{k+1}$ is the unique solution to\n%     \\begin{equation*}\n%         \\begin{cases}\n%         x_{k+1} = x_k - \\alpha_k(g_0(x_k;\\xi_k)+n_{k+1}) & \\mathrm{if\\ } f_s(x_k) \\leq 0 \\ \\ \\forall s = 1...m \\\\\n%         x_{k+1}=x_k - \\alpha_k g_{s(x_k)}(x_k;\\xi_k) & \\mathrm{for\\ some}\\ f_s(x_k)>0\\ \\mathrm{otherwise} \\\\\n%         n_{k+1} \\in \\partial r(x_{k+1}) \\ . \n%     \\end{cases}\n%     \\end{equation*}\n%     Now we define \n%     \\begin{equation*}\n%         \\bar g_i(y_i;\\xi_i)=\\begin{cases}\n%         g_0(y_i;\\xi_i)+n_{i+1} & \\mathrm{if}\\ f_s(y_i)\\leq 0 \\quad \\forall s=1\\dots m,\\\\\n%         &n_{i+1} \\in \\partial r(y_{i+1})\\\\\n%         g_{s(y_i)}(y_i;\\xi_i) & \\mathrm{for\\ some}\\ f_s(y_i)>0\\ \\mathrm{otherwise} \\ .\n%     \\end{cases}\n%     \\end{equation*}\n%    Following Lemma~\\ref{lem:uniqness}, we first consider when $y_k$ is feasible,\n%     \\begin{align*}\n%           y_{k+1} & =\\argmin \\left\\{ M^{(k-1)}(y)+U^{k}(y)+\\frac{\\beta_k}{2}\\|y-y_0\\|_2^2\\right\\}\\\\\n%                   &=\\argmin \\left\\{\\sum_{i=0}^{k-1}\\lambda_i\\left( \\langle \\bar g_i(y_i;\\xi_i), y-y_i \\rangle + \\frac{\\mu}{2}\\|y-y_i\\|_2^2 \\right) \\right.\\\\ & \\left.\n%                    + \\lambda_k \\left(\\langle g_0(y_k;\\xi_k), y-y_k \\rangle +\\frac{\\mu}{2}\\|y-y_k\\|_2^2 + r(y)\\right) + \\frac{\\bar \\beta}{2}\\|y-y_0\\|_2^2 \\right\\},\n%     \\end{align*}\n% where the second equality holds by removing some constant terms in $M^{(k-1)}(y)$ and $U^{k}(y)$.\n%     \\begin{align*}\n%           y_{k+1} & =\\argmin \\left\\{ M^{(k-1)}(y)+U^{k}(y)+\\frac{\\beta_k}{2}\\|y-y_0\\|_2^2\\right\\}\\\\\n%                   &=\\argmin \\left\\{\\sum_{i=0}^{k-1}\\lambda_i\\left(\\langle g_i(y_i;\\xi_i), y-y_i \\rangle + \\frac{\\mu}{2}\\|y-y_i\\|_2^2 \\right)+  \\sum_{i<k \\colon  s(y_i)=0} \\lambda_i\\langle n_{i+1}, y-y_{i+1}\\rangle \\right.\n%                   \\\\ & \\qquad \\qquad \\left.\n%                    + \\lambda_k \\left(\\langle g_0(y_k;\\xi_k), y-y_k \\rangle +\\frac{\\mu}{2}\\|y-y_k\\|_2^2 + r(y)\\right) + \\frac{\\bar \\beta}{2}\\|y-y_0\\|_2^2 \\right\\},\n%     \\end{align*}\n%     where the second equality holds by removing some constant terms in $M^{(k-1)}(y)$ and $U^{(k)}(y)$.\n%    Taking the subgradient of the right hand side and let the it be zero, and combining with $\\bar g_k(y_k;\\xi_k) = g_0(y_k;\\xi_i)+n_{k+1}$, we have\n%    \\begin{equation*}\n%        \\sum_{i=0}^k \\lambda_i\\left(\\bar g_i(y_i;\\xi_i) + \\mu(y_{k+1}-y_i) \\right) + \\beta(y_{k+1}-y_0) = 0,\n%    \\end{equation*}\n%    which remains same when $y_k$ is infeasible since $\\bar g_k(y_k;\\xi_k)=g_{s(y_k)}(y_k;\\xi_k)$.  \n%    By rearranging the equation, we have\n%    \\begin{equation}\\label{eq:dual-iterates-formula}\n%        y_{k+1}-y_0 = \\frac{\\sum_{i=0}^k\\lambda_i\\mu(y_i-y_0)-\\sum_{i=0}^k\\lambda_i\\bar g_i(y_i;\\xi_i)}{\\sum_{i=0}^k\\lambda_i\\mu+\\bar\\beta}\n%    \\end{equation}\n%  We prove the sequence of iterates $y_k$ given by the dual method is equivalent to the sequence of iterates $x_k$ given by the primal method by inductively showing the formula\n%  \\begin{equation}\\label{eq:recursive-formula}\n%      y_{k+1}-y_0 = -\\sum_{t=0}^k \\frac{\\lambda_t \\bar g_t(y_t;\\xi_t)}{\\sum_{s=0}^t \\lambda_s\\mu+\\bar\\beta},\n%  \\end{equation}\n%    and it is easy to see that the right hand side is the closed form solution of $x_{k+1}-x_0$.\n% One can readily verify this holds $k=0$. So we inductively assume the formula~\\eqref{eq:recursive-formula} holds for all $y_i-y_0$ with $i \\leq k$ and derive it for $y_{k+1}-y_0$.\n% Combining the assumptions with~\\eqref{eq:dual-iterates-formula}, the claimed result follows as\n% \\begin{align*}\n%     y_{k+1} -y_0 &= -\\frac{\\sum_{i=0}^{k} \\lambda_{i}\\mu\\left( \\sum_{t=0}^{i-1} \\frac{\\lambda_{t} \\bar g_t(y_t;\\xi_t)}{\\sum_{s=0}^{t}\\lambda_{s}\\mu + \\bar \\beta} \\right) + \\sum_{i=0}^{k}\\lambda_i \\bar g_i(y_i;\\xi_i)}{\\sum_{i=0}^{k}\\lambda_i\\mu + \\bar\\beta}\\\\\n%     &= -\\frac{\\sum_{t=0}^{k-1} \\lambda_t \\bar g_t(y_t;\\xi_t)\\frac{\\sum_{i=t+1}^k \\mu\\lambda_i}{\\sum_{i=0}^t \\mu\\lambda_t +\\bar\\beta} + \\sum_{i=0}^{k}\\lambda_i \\bar g_i(y_i;\\xi_i)}{\\sum_{i=0}^{k}\\lambda_i\\mu+\\bar\\beta}\\\\\n%     &= -\\frac{\\sum_{t=0}^{k-1} \\lambda_t \\bar g_t(y_t;\\xi_t)\\left(\\frac{\\sum_{i=t+1}^k \\mu\\lambda_i}{\\sum_{i=0}^t \\mu\\lambda_t +\\bar\\beta}+1\\right)}{\\sum_{i=0}^{k}\\lambda_i\\mu+\\bar\\beta} - \\frac{\\lambda_k \\bar g_k(y_k;\\xi_k)}{\\sum_{i=0}^{k}\\lambda_i\\mu + \\bar\\beta}\\\\\n%     &=  -\\sum_{t=0}^{k-1} \\frac{\\lambda_t \\bar g_t(y_t;\\xi_t)}{\\sum_{i=0}^{t}\\lambda_t\\mu+\\bar\\beta} - \\frac{\\lambda_k \\bar g_k(y_k;\\xi_k)}{\\sum_{i=0}^{k}\\lambda_i\\mu + \\bar\\beta} \n% \\end{align*}\n% where the first equality uses the inductive hypothesis for each $y_i-y_0$, the second exchanges summands, and the third and fourth combine and simplify terms.\n% Which tells us $y_k=x_k$ for each $k$, hence the primal method~\\eqref{eq:super-primal-method}and dual method~\\eqref{eq:super-dual-method}are equivalent. \n\\end{proof}","sha256_utf8":"9c477eec0db3705ec664f8c9c93b5dfcf0f53fb44d2ceb66a1acfda5a6bd81ed"},{"start":70563,"end":70985,"start_line":708,"end_line":711,"raw_tex":"\\begin{lemma}\\label{lem:switching-proximal-primal-induction}\n    Under Assumptions A-C, the switching proximal subgradient method~\\eqref{eq:super-primal-method} with $\\alpha_k=\\lambda_k/\\mu\\sum_{i=0}^k\\lambda_i$ has for either $y\\in\\{x_{\\mathtt{OPT}}, x_{\\mathtt{SL}}\\}$\n    $$R_{k+1}(y) \\leq R_k(y) - \\frac{\\lambda_k}{2}\\left(\\left(2-L_1\\alpha_k\\right)\\mathbb{E}_\\xi\\delta_{k}(y) - L_0^2\\alpha_k\\right) \\ . $$\n\\end{lemma}","sha256_utf8":"8dc7d8324cc0db65adbd6fe499d108a2e5288928c8b875ad0e62524ae3a47542"},{"start":70986,"end":73149,"start_line":712,"end_line":731,"raw_tex":"\\begin{proof}\n    This proof follows a standard analysis technique, directly expanding the definition of $R_{k+1}(y)$. First, suppose $x_k$ is feasible. Then\n    \\begin{align*}\n    R_{k+1}(y) &= \\frac{w_k}{2}\\mathbb{E}_\\xi\\|\\mathrm{prox}_{\\alpha_k,r}(x_k - \\alpha_k g_k) - \\mathrm{prox}_{\\alpha_k,r}(y +\\alpha_k n_{y})\\|_2^2\\\\\n    &\\leq \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k - \\alpha_k(g_k+n_{y}) - y\\|_2^2\\\\\n    &= \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k-y\\|_2^2 - \\lambda_k\\mathbb{E}_\\xi\\langle g_k + n_{y}, x_k-y\\rangle +\\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{y}\\|_2^2\\\\\n    &\\leq \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k-y\\|_2^2 - \\lambda_k\\mathbb{E}_\\xi(h_y(x_k)-h_y(y)+\\frac{\\mu}{2}\\|x_k - y\\|_2^2) + \\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{y}\\|_2^2\\\\\n    &= R_k(y) - \\lambda_k\\mathbb{E}_\\xi\\delta_k(y)+\\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{y}\\|_2^2\n   \\end{align*}\n    where the first line uses that $\\mathrm{prox}_{\\alpha_k,r}(y +\\alpha_k n_{y}) = y$, the second uses the nonexpansiveness of the proximal operator~\\cite[Proposition 12.19]{RockWets98}, the third factors the norm squared and uses $\\alpha_k = \\lambda_k/w_k$, the fourth uses the strong convexity of $h_y$, {\\color{blue} and the fifth follows from definitions as $R_k(y) = \\frac{w_k - \\lambda_k\\mu}{2}\\mathbb{E}_\\xi\\|x_k-y\\|_2^2$ and $\\delta_k(y)=h_y(x_k)-h_y(y)$}.\n    Then, applying the bound~\\eqref{eq:NonLipschitz-condition} gives the claim.\n    Similarly, supposing $x_k$ is infeasible,\n    \\begin{align*}\n    R_{k+1}(y) &= \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k - \\alpha_k g_k - y\\|_2^2\\\\\n    &= \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k-y\\|_2^2 - \\lambda_k\\mathbb{E}_\\xi\\langle g_k, x_k-y\\rangle + \\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k\\|_2^2\\\\\n    &\\leq \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k-y\\|_2^2- \\lambda_k\\mathbb{E}_\\xi(f_{s(x_k)}(x_k)-f_{s(x_k)}(y) +\\frac{\\mu}{2}\\|x_k - y\\|_2^2) + \\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k\\|_2^2\\\\\n    &= R_k(y) - \\lambda_k\\mathbb{E}_\\xi\\delta_k(y)+\\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k\\|_2^2 \n   \\end{align*}\n   using strong convexity of $f_{s(x_k)}$. Applying~\\eqref{eq:NonLipschitz-condition} completes the proof.\n\\end{proof}","sha256_utf8":"2b078c268a3c911011fb6cdca03764e36e8a86f4cf0cf723007c4581f326914d"},{"start":73714,"end":74321,"start_line":737,"end_line":743,"raw_tex":"\\begin{lemma}\\label{lem:switching-proximal-dual-induction}\n    Under Assumptions A-C, the switching proximal subgradient method~\\eqref{eq:super-primal-method} with $\\alpha_k=\\lambda_k/\\mu\\sum_{i=0}^k\\lambda_i$ has\n     $$ D_{k+1} \\leq D_k - \\frac{\\lambda_k}{2}\\begin{cases}\n         \\left(2-L_1\\alpha_k\\right)\\mathbb{E}_\\xi\\delta_{k}(x_\\mathtt{OPT}) - L_0^2\\alpha_k & \\text{if } x_k \\text{ is feasible}\\\\\n         {\\color{blue}\\left(2-L_1\\alpha_k\\right)\\mathbb{E}_\\xi\\delta_{k}(x_\\mathtt{OPT}) + 2\\mathbb{E}_\\xi f_{s(x_k)}(x_\\mathtt{OPT}) - L_0^2\\alpha_k} & \\text{otherwise.}\n     \\end{cases} $$\n\\end{lemma}","sha256_utf8":"18e0aaaea366f08386c4db87a20db3cb6f0777fccae1a33f1245558d0375cdea"},{"start":74322,"end":79345,"start_line":744,"end_line":787,"raw_tex":"\\begin{proof}\nObserve that one can rewrite $M^{(k)}(x) = Q_1(x) + Q_2(x)$ as the sum of two quadratics where $Q_1(x) = M^{(k-1)}(x)$ and $Q_2$ depends on whether $x_k$ is feasible. In the notation of Lemma~\\ref{lem:quad-calc}, Lemmas~\\ref{lem:Mk-minimization} {\\color{blue} and~\\ref{lem:standard-quadratic}} ensure $Q_1$ has $a_1 = \\inf M^{(k-1)}$, $b_1 = \\mu\\sum_{i=0}^{k-1}\\lambda_i$, and $z_1 = x_{k}${\\color{blue}, since $\\bar \\beta = 0$ here}. To determine $Q_2$, first suppose $x_k$ is feasible. Then we have\n$$Q_2(x) = \\lambda_k \\left(f_{0}(x_k) + \\langle g_k, x-x_k \\rangle +\\frac{\\mu}{2}\\|x - x_k\\|_2^2 + r(x_{k+1}) + \\langle n_{k+1}, x-x_{k+1}\\rangle\\right) \\ . $$\nThis quadratic can be written in the form $Q_2(z) = a_2 + \\frac{b_2}{2}\\|z-z_2\\|_2^2$ with\n\\begin{align*}\n    a_2 &= \\lambda_k\\left(f_0(x_k) + r(x_{k+1}) + \\alpha_k\\langle n_{k+1}, g_{k}+n_{k+1}\\rangle - \\frac{1}{2\\mu}\\|g_k + n_{k+1}\\|_2^2 \\right)\\ , \\\\\n    b_2 &= \\mu\\lambda_k \\ , \\qquad z_2 = x_k - \\frac{g_k+n_{k+1}}{\\mu} \\ . \n\\end{align*}\nBy Lemma~\\ref{lem:quad-calc}, the expected minimum value of the updated model $\\mathbb{E}_\\xi \\inf M^{(k)}$ is\n\\begin{align*}\n    &\\mathbb{E}_\\xi \\inf M^{(k-1)} + \\lambda_k \\mathbb{E}_\\xi\\Bigg(f_0(x_k) + r(x_{k+1}) + \\alpha_k \\langle n_{k+1}, g_{k}+n_{k+1}\\rangle - \\frac{1}{2\\mu}\\|g_k + n_{k+1}\\|_2^2\\\\\n    & \\qquad\\qquad\\qquad\\qquad\\qquad + \\frac{\\sum_{i=0}^{k-1}\\lambda_i }{2\\mu \\sum_{i=0}^k \\lambda_i}\\left\\|g_{k}+n_{k+1}\\right\\|_2^2\\Bigg) \\ . \n\\end{align*}\nFrom this, we conclude the lower bound\n\\begin{align*}\n    \\mathbb{E}_\\xi \\inf M^{(k)} &\\geq \\mathbb{E}_\\xi \\inf M^{(k-1)} + \\lambda_k \\mathbb{E}_\\xi\\Bigg(f_0(x_k) + r(x_\\mathtt{OPT}) + \\langle n_{x_\\mathtt{OPT}}, x_{k+1}-x_\\mathtt{OPT}\\rangle  \\\\\n    & \\quad + \\alpha_k \\langle n_{k+1}, g_{k}+n_{k+1}\\rangle - \\frac{1}{2\\mu}\\|g_k + n_{k+1}\\|_2^2 + \\frac{\\sum_{i=0}^{k-1}\\lambda_i }{2\\mu \\sum_{i=0}^k \\lambda_i}\\left\\|g_{k}+n_{k+1}\\right\\|_2^2\\Bigg)\\\\\n    &= \\mathbb{E}_\\xi \\inf M^{(k-1)} + \\lambda_k \\mathbb{E}_\\xi\\Bigg(\\delta_k(x_\\mathtt{OPT}) + p_\\star + \\langle n_{x_\\mathtt{OPT}}, -\\alpha_k(g_k + n_{k+1})\\rangle  \\\\\n    & \\quad + \\alpha_k \\langle n_{k+1}, g_{k}+n_{k+1}\\rangle - \\frac{\\alpha_k}{2}\\|g_k + n_{k+1}\\|_2^2\\Bigg)\\\\\n    &= \\mathbb{E}_\\xi\\inf M^{(k-1)} + \\lambda_k \\left(\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) + p_\\star - \\frac{\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{x_\\mathtt{OPT}}\\|_2^2 { \\color{blue} + \\frac{\\alpha_k}{2}\\|n_{k+1}-n_{x_\\mathtt{OPT}} \\|_2^2}\\right)\\\\\n    & {\\color{blue} \\geq \\mathbb{E}_\\xi\\inf M^{(k-1)} + \\lambda_k \\left(\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) + p_\\star - \\frac{\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{x_\\mathtt{OPT}}\\|_2^2\\right)}\n\\end{align*}\nwhere the first inequality lower bounds $r(x_{k+1})$ by $r(x_\\mathtt{OPT}) + \\langle n_{x_\\mathtt{OPT}}, x_{k+1}-x_\\mathtt{OPT}\\rangle$, the first equality applies the definitions of $\\delta_k(x_\\mathtt{OPT})$ in~\\eqref{eq:delta_k-definition}, $\\alpha_k$ in~\\eqref{eq:alpha-lambda} {\\color{blue} with $\\bar \\beta = 0$, and that $x_{k+1}-x_k=-\\alpha_k(g_k+n_{k+1})$}, and the {\\color{blue} second} equality combines and simplifies terms.\nIn terms of $D_k = \\mathbb{E}_\\xi[\\sum_{i< k:s(x_i)=0} \\lambda_i p_\\star - \\inf M^{(k-1)}]$, this gives the following \nrecurrence\n$$ D_{k+1} \\leq D_k - \\lambda_k \\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) + \\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{x_\\mathtt{OPT}}\\|_2^2 \\ . $$\nApplying~\\eqref{eq:NonLipschitz-condition} gives the claim in this case.\nNow suppose $x_k$ is infeasible. Then, noting $Q_2$ minimizes at $x_k - \\frac{g_{k}}{\\mu}$, Lemma~\\ref{lem:standard-quadratic} ensures\n\\begin{align*}\n    Q_2(x) &= \\lambda_k\\left(f_{s(x_k)}(x_k) - \\frac{1}{2\\mu}\\|g_k \\|_2^2 + \\frac{\\mu}{2}\\left\\|x - \\left(x_k - \\frac{g_{k}}{\\mu}\\right)\\right\\|_2^2\\right) \\ .\n\\end{align*}\nThen by Lemma~\\ref{lem:quad-calc}, the expected minimum value of the updated model $\\mathbb{E}_\\xi \\inf M^{(k)}$ is given by\n\\begin{align*}\n    &\\mathbb{E}_\\xi\\inf  M^{(k-1)} + \\lambda_k\\mathbb{E}_\\xi\\left(f_{s(x_k)}(x_k) - \\frac{1}{2\\mu}\\|g_k \\|_2^2\\right) + \\frac{\\lambda_k\\sum_{i=0}^{k-1}\\lambda_i }{{\\color{blue} 2}\\mu \\sum_{i=0}^k \\lambda_i}\\mathbb{E}_\\xi\\left\\|g_{k}\\right\\|_2^2 \\\\\n    &= \\mathbb{E}_\\xi\\inf M^{(k-1)} + \\lambda_k\\mathbb{E}_\\xi f_{s(x_k)}(x_k) - \\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k\\|_2^2 \\ . \n\\end{align*}\n{\\color{blue}\nIn terms of $D_k = \\mathbb{E}_\\xi[\\sum_{i< k:s(x_i)=0} \\lambda_i p_\\star - \\mathbb{E}_\\xi\\inf M^{(k-1)}]$, this gives the recurrence\n$$ D_{k+1} = D_k - \\lambda_k \\mathbb{E}_\\xi\\left[\\delta_{k}(x_\\mathtt{OPT}) + f_{s(x_k)}(x_{\\mathtt{OPT}})\\right] + \\frac{\\lambda_k\\alpha_k}{2} \\mathbb{E}_\\xi\\|g_k\\|_2^2 \\ . $$\nBounding $\\mathbb{E}_\\xi\\|g_k\\|_2^2$ by~\\eqref{eq:NonLipschitz-condition} gives the claim in this last case.\n}\n% Noting $f_{s(x_k)}(x_\\mathtt{OPT})\\leq 0$, we conclude the lower bound\n% $$\\mathbb{E}_\\xi \\inf M^{(k)} \\geq \\mathbb{E}_\\xi\\inf M^{(k-1)} + \\lambda_k \\left(\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) - \\frac{\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k\\|_2^2\\right) \\ .$$\n\\end{proof}","sha256_utf8":"d194add877daf2151cbcc164ac5c2a41b9947aaaf1faf0624d77df3567d427bd"},{"start":84932,"end":87267,"start_line":832,"end_line":848,"raw_tex":"\\subsubsection{Proof of Theorem~\\ref{thm:super-rate}} \\label{subsec:proof-main-result}\nApplying Lemma~\\ref{lem:switching-proximal-primal-induction} with $y=x_\\mathtt{OPT}$ from $k=0$ to $T-1$ yields\n$$R_T(x_\\mathtt{OPT}) + \\sum_{k=0}^{T-1} \\frac{\\lambda_k}{2}\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) \\leq R_0(x_\\mathtt{OPT}) + \\sum_{k=0}^{T-1}\\frac{L_0^2\\lambda_k\\alpha_k}{2} + \\sum_{k=0}^{T-1} \\frac{\\lambda_k}{2}\\left(L_1\\alpha_k - 1 \\right)\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT})\\ .$$\nSimilarly, applying Lemma~\\ref{lem:switching-proximal-dual-induction} from $k=0$ to $T-1$ yields\n\\begin{align*}\n    D_T &+ \\sum_{k=0}^{T-1} \\frac{\\lambda_k}{2}(\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT})) + {\\color{blue}\\sum_{k<T:s(x_k)>0} \\lambda_k\\mathbb{E}_\\xi f_{s(x_k)}(x_\\mathtt{OPT})}\\\\\n    &\\leq D_0 + \\sum_{k=0}^{T-1}\\frac{L_0^2\\lambda_k\\alpha_k}{2} + \\sum_{k=0}^{T-1} \\frac{\\lambda_k}{2}\\left(L_1\\alpha_k - 1 \\right)\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT})\\ .\n\\end{align*}\nNoting $R_0(x_\\mathtt{OPT})=D_0=0$ and the last summation in each bound is at most half our initial blow-up constant $\\frac{1}{2}C_0$, the sum of these inequalities provides a bound of\n$$ R_T(x_\\mathtt{OPT}) + D_T + \\sum_{k=0}^{T-1} \\lambda_k\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) {\\color{blue} + \\sum_{k<T:s(x_k)>0} \\lambda_k\\mathbb{E}_\\xi}f_{s(x_k)}(x_\\mathtt{OPT}) \\leq \\sum_{k=0}^{T-1}L_0^2\\lambda_k\\alpha_k + C_0 \\ . $$\nLower bounding each $\\mathbb{E}_\\xi[\\delta_k(x_\\mathtt{OPT}) {\\color{blue}+  f_{s(x_k)}(x_\\mathtt{OPT})}]$ with $s(x_k)\\neq 0$ by zero {\\color{blue} and dividing through by $\\sum_{k=0}^{T-1} \\lambda_k$ establishes that\n$$ \\frac{1}{\\sum_{k=0}^{T-1} \\lambda_k}\\left(R_T(x_\\mathtt{OPT}) + D_T + \\sum_{k<T:s(x_k)=0} \\lambda_k\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT})\\right) \\leq \\frac{\\sum_{k=0}^{T-1}L_0^2\\lambda_k\\alpha_k + C_0}{\\sum_{k=0}^{T-1} \\lambda_k} \\ . $$\nSubstituting the definitions of $R_T$ and $D_T$ from~\\eqref{eq:Rk-definition} and~\\eqref{eq:Dk-definition}, the left-hand side above is equal to\n$$ \\mathbb{E}_\\xi\\left[\\frac{\\mu}{2}\\|x_T-x_\\mathtt{OPT}\\|^2_2 + \\frac{1}{{\\sum_{k=0}^{T-1} \\lambda_k}}\\left(\\sum_{k<T:s(x_k)=0} \\lambda_k h_{x_\\mathtt{OPT}}(x_k)- \\inf M^{(T-1)}\\right)\\right]\\ . $$\nRecalling the primal and dual gaps definitions in~\\eqref{eq:primalgap-definition} and~\\eqref{eq:dualgap-definition} gives our main theorem.}\n\n    ","sha256_utf8":"91fe0ce6b94e67409f79499c8fa2cd93d07c583e46a80a76f0b368ca4606cc56"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"复合约束优化问题","source_units":[{"start":8570,"end":8764,"start_line":230,"end_line":235,"raw_tex":"\\begin{equation}\n    p_\\star = \\begin{cases}\n        \\min & f_0(x) + r(x)\\\\\n        \\mathrm{s.t.} & f_s(x) \\leq 0 \\qquad \\forall s=1\\dots m\n    \\end{cases}\\label{eq:super-problem}\n\\end{equation}","sha256_utf8":"395f763533be34434b9ead45725e9659026aec06813b7535d0754009da523250"},{"start":8765,"end":9492,"start_line":236,"end_line":236,"raw_tex":"where the functions $f_s\\colon \\mathcal{E}\\rightarrow \\mathbb{R}$, $s=0,\\dots, m,$ are (strongly) convex but may be nonsmooth and not globally Lipschitz continuous and $r\\colon \\mathcal{E}\\rightarrow \\mathbb{R}\\cup\\{\\infty\\}$ is convex, closed, and simple, all defined over a finite-dimensional Euclidean space $\\varSpace$. We will consider iteratively solving problems of this general form via a stochastic switching proximal subgradient method. This general method corresponds to the classic subgradient method when $r=0$ and $m=0$, the proximal subgradient method when $m=0$, and the switching subgradient method of~\\cite{Polyak1967} when $r=0$. Formal assumptions and definitions are deferred to Section~\\ref{sec:prelim}. \n","sha256_utf8":"c1bd6e6a1d732e4b6d55b5aab472a44f7ab5c1c339a521f0b3fd828e6275c71e"}]},{"id":"S002","semantic_type":"definition","kind":"definition","title":"次梯度与强凸下界","source_units":[{"start":10157,"end":10625,"start_line":243,"end_line":247,"raw_tex":"$g_0(x)\\in \\partial f_0(x) := \\{  g \\mid f_0(y) \\geq f_0(x) + \\langle g , y-x \\rangle \\quad \\forall y\\in\\varSpace\\}$. Note when $f_0$ is $\\mu$-strongly convex (that is, $f(x)-\\frac{\\mu}{2}\\|x\\|_2^2$ is convex), each subgradient $g_0(x)\\in \\partial f_0(x)$, provides a quadratic lower bound\n\\begin{equation}\n    f_0(y) \\geq f_0(x) + \\langle g_0(x) , y-x \\rangle +\\frac{\\mu}{2}\\|y-x\\|_2^2 \\quad \\forall y\\in\\varSpace \\ .\n\\label{eq:subgradient-lower-bound}\n\\end{equation}","sha256_utf8":"75df2ca926bf80cd0531c5930f7538e0ba2b3c8d706629bdd82178cf0961b15a"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"近端算子及最优条件","source_units":[{"start":19194,"end":20027,"start_line":305,"end_line":310,"raw_tex":"First, we allow for additive composite objectives $f_0+r$ for any {\\color{blue} proper} closed convex $r\\colon \\varSpace\\rightarrow \\mathbb{R}\\cup\\{\\infty\\}$. We address this added term by assuming $r$ is sufficiently simple that its proximal operator\n$ \\mathrm{prox}_{\\alpha_k, r}(z) := \\argmin_x\\left\\{r(x) + \\frac{1}{2\\alpha_k}\\|x-z\\|_2^2\\right\\} $\ncan be evaluated at each iteration. For example, if $r$ is an indicator function for a closed convex constraint set, its proximal operator corresponds to projection onto that set. Since $r(x) + \\frac{1}{2\\alpha_k}\\|x-z\\|_2^2$ is strongly convex, it has a unique minimizer described by the following equivalence\n\\begin{equation}\n    z_+ = \\mathrm{prox}_{\\alpha_k, r}(z) \\quad \\iff \\quad \\frac{1}{\\alpha_k}(z-z_+)\\in \\partial r(z_+) \\ . \\label{eq:proximal-equivalence}\n\\end{equation}","sha256_utf8":"1832f1b4bac6131c873cfe4b5c1180d659f94ec1cb66ece1ec86dc58e47517ba"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"切换规则与随机预言机","source_units":[{"start":20028,"end":20741,"start_line":311,"end_line":312,"raw_tex":"Second, we allow for $m$ strongly convex functional constraints $f_s(x)\\leq 0$ for $s=1,\\dots, m$. We address these added terms by ``switching'': Given a current iterate $x$, only one function $f_{s(x)}$ will be considered in that iteration. This function will be chosen as $s(x)=0$ if $x$ is feasible (that is, $f_s(x)\\leq 0$ for all $s=1\\dots m$), otherwise $s(x)$ can be chosen generically as any violated constraint $f_{s(x)}(x)>0$.\nThird, we allow for stochasticity in our subgradient oracles for each function, denoted by $g_s(x;\\xi)$ such that $\\mathbb{E}_\\xi g_{s}(x;\\xi) \\in \\partial f_{s}(x)$. Note this trivially captures deterministic methods by selecting $g_s(x;\\xi)$ constant with respect to $\\xi$.\n","sha256_utf8":"e9dd8bdd6bb4ba69c8c3251361f0bfff4e059d4bc62825f551e1d1519c4fc29b"}]},{"id":"S005","semantic_type":"algorithm","kind":"algorithm","title":"随机切换近端次梯度法","source_units":[{"start":21077,"end":21518,"start_line":317,"end_line":323,"raw_tex":"As a primal algorithm, consider the {\\it Stochastic Switching Proximal Subgradient Method} with stepsizes $\\alpha_k>0$ and sequence of iterates $\\{x_k\\}$ defined by\n\\begin{equation}\\label{eq:super-primal-method}\n    x_{k+1} = \\begin{cases}\n        \\mathrm{prox}_{\\alpha_k, r} (x_k-\\alpha_k g_{0}(x_k;\\xi_k)) & \\text{if $x_k$ is feasible}\\\\\n        x_k-\\alpha_k g_{s(x_k)}(x_k;\\xi_k) & \\text{otherwise.}        \n    \\end{cases}\n\\end{equation}","sha256_utf8":"f03c6cd42c3e9174aa1621903f67f46981f5f1364d0cf06823111311bec5962a"},{"start":21519,"end":21600,"start_line":324,"end_line":324,"raw_tex":"for i.i.d.~sampled $\\xi_k$. Throughout, we always assume $\\alpha_k>0$, strictly.\n","sha256_utf8":"71166c253ad8e6feee5fdec9789c8662d1ad13bad30c51ce222908d96a85d013"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"对偶聚合模型","source_units":[{"start":22432,"end":23132,"start_line":330,"end_line":334,"raw_tex":"\\begin{align*}\n    p_\\star &= \\min_{x\\in\\varSpace}\\max_{u_1,\\dots u_s\\geq 0} f_0(x) + \\sum_{s=1}^m u_sf_s(x) + r(x) \\\\\n    &=\\min_{x\\in\\varSpace}\\max_{\\substack{y_0,\\dots, y_{T}\\in\\varSpace\\\\ \\lambda_0,\\dots, \\lambda_{T-1}\\geq 0 \\\\ \\sum_{k<T\\colon s(y_k)=0} \\lambda_k >0\\\\ n_{k+1}\\in\\partial r(y_{k+1})}} \\mathbb{E}_{\\xi}\\Bigg(\\sum_{s=0}^m\\sum_{\\substack{k<T \\\\ s(y_k)=s}} \\lambda_k\\left(f_{s(y_k)}(y_k) + \\langle g_{s(y_k)}(y_k; \\xi_k), x-y_k \\rangle + \\frac{\\mu}{2}\\|x-y_k\\|_2^2\\right)\\\\\n    &\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad +\\sum_{\\substack{k<T\\\\ s(y_k)=0}} \\lambda_k\\left(r(y_{k+1}) + \\langle n_{k+1}, x - y_{k+1}\\rangle\\right)\\Bigg)\\frac{1}{\\sum_{k<T\\colon s(y_k)=0} \\lambda_k}\n\\end{align*}","sha256_utf8":"2d8c1547ea63c0c2286fb85cd53df42121db054337fcd16a7118ef4d2ea0f0c1"},{"start":23400,"end":23849,"start_line":337,"end_line":337,"raw_tex":"As a dual algorithm, consider the following {\\it Stochastic Lagrangian Proximal Dual Averaging} with dual weights $\\lambda_k>0$ and regularization parameters $\\beta_k\\geq 0$. Throughout, we always assume $\\lambda_k>0$, strictly. Its sequence of iterates $\\{y_k\\}$ based on i.i.d.~sampled $\\xi_k$ is defined as follows: At iteration $k$, construct the following (unnormalized) aggregations for each function based on the previous $k-1$ iterations as\n","sha256_utf8":"405790e8630f63f115b2b06be339aaee8d50bb5297305bc61c1e3e56d9ea2a80"},{"start":24229,"end":24648,"start_line":343,"end_line":345,"raw_tex":"where $n_{i+1} = \\frac{-1}{\\lambda_i}(\\nabla M^{(i-1)}(y_{i+1}) + \\lambda_i(g_0(y_i;\\xi_i) +\\mu(y_{i+1}-y_i)) + \\beta_i(y_{i+1} - y_0)) \\in \\partial r(y_{i+1})$ (see Lemma~\\ref{lem:Mk-minimization} for verification of this subdifferential containment).\nAt iteration $k=0$, these empty summations are understood to take value zero.\nThen, based on the switching selection $s(y_k)$, a new weighted model is constructed as\n","sha256_utf8":"aba86c1112411a4b240bdab0bcdc13c7c0e77a19ca901ef54a7caa6fb2f3b675"}]},{"id":"S007","semantic_type":"algorithm","kind":"algorithm","title":"拉格朗日近端对偶平均","source_units":[{"start":24648,"end":24966,"start_line":346,"end_line":348,"raw_tex":"$$ U^{(k)}(y) := \\begin{cases} \\lambda_k\\left(f_{0}(y_k) + \\langle g_0(y_k;\\xi_k), y-y_k\\rangle + \\frac{\\mu}{2}\\|y-y_k\\|_2^2 + r(y)\\right) & \\text{if $y_k$ is feasible}\\\\\n\\lambda_k\\left(f_{s(y_k)}(y_k) + \\langle g_{s(y_k)}(y_k;\\xi_k), y-y_k\\rangle + \\frac{\\mu}{2}\\|y-y_k\\|_2^2\\right) & \\text{otherwise.}\n\\end{cases} $$","sha256_utf8":"476295eddd665ebd3694b47b962826fb6f59977bfdd8c1469fd1b88c3519cf28"},{"start":24967,"end":25315,"start_line":349,"end_line":352,"raw_tex":"The Lagrangian proximal dual averaging method then iterates by minimizing the aggregation of past models $M^{(k-1)}$ plus the new model $U^{(k)}$ (and an optional extra regularization term)\n\\begin{equation}\\label{eq:super-dual-method}\n    y_{k+1} = \\argmin\\left\\{M^{(k-1)}(y) + U^{(k)}(y) +\\frac{\\beta_k}{2}\\|y-y_0\\|_2^2 \\right\\} \\ .\n\\end{equation}","sha256_utf8":"a7aa07dd435930609e7401d110c4b67438cbf041490f2a9048ba57729cc59435"}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","title":"强凸性与严格可行点","source_units":[{"start":35575,"end":35826,"start_line":429,"end_line":431,"raw_tex":"\\noindent {\\bf Assumption A.} The functions $f_s$ for $s=0\\dots m$ are each $\\mu>0$-strongly convex.\\\\\n\\noindent {\\bf Assumption B.} There exists some $x_\\mathtt{SL} \\in \\mathrm{dom\\ }\\partial r$ with $f_s(x_\\mathtt{SL}) <0$ for all \n $s = 1\\dots m$.\n","sha256_utf8":"00e07aacb9d00f0260ad6ac97961b1f57106122a27a49e142ebe39952df7869f"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"参考点与松弛目标","source_units":[{"start":35827,"end":36845,"start_line":433,"end_line":436,"raw_tex":"Note strong convexity ensures there exists a unique minimizer $x_\\mathtt{OPT} \\in \\mathrm{dom\\ }\\partial r$ of~\\eqref{eq:super-problem}. These two points $x_\\mathtt{OPT},x_\\mathtt{SL} \\in \\mathrm{dom\\ }\\partial r$ serve as important references that our analysis is done with respect to. We fix two subgradients of $r$ at these points: The subgradient $n_{x_\\mathtt{OPT}}\\in \\partial r(x_\\mathtt{OPT})$ is chosen such that $f_0(x) + r(x_\\mathtt{OPT}) + \\langle n_{x_\\mathtt{OPT}}, x-x_\\mathtt{OPT}\\rangle$ is minimized over $f_s(x)\\leq 0$ at $x_\\mathtt{OPT}$ for all $ s=1\\ldots m$. The subgradient $n_{x_\\mathtt{SL}} \\in \\partial r(x_\\mathtt{SL})$ can be chosen freely. %When $r$ is an indicator function for a closed convex set, $n_{x_\\mathtt{SL}}=0$ is a nice simplifying choice.\n\nThese two reference subgradients of $r$ facilitate considering two lower bounds of the objective $f_0+r$ for either $y\\in\\{x_{\\mathtt{OPT}}, x_{\\mathtt{SL}}\\}$, denoted by\n$$ h_{y}(x) := f_0(x) + r(y) + \\langle n_{y}, x-y\\rangle \\ . $$","sha256_utf8":"4fa3b118bac84773a5baff070abe2a9522ea6e5adce329c76b943dbae55d48d8"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"相对差值及符号界","source_units":[{"start":36950,"end":37402,"start_line":438,"end_line":444,"raw_tex":"At each iteration $k$, we denote the relative difference between $x_k$ and $y\\in\\{x_{\\mathtt{OPT}}, x_{\\mathtt{SL}}\\}$ in (relaxed) objective value or feasibility on the selected constraint function $f_{s(x_k)}$ by\n\\begin{align}\n    \\delta_k(y) &:=\\begin{cases}\n        h_y(x_k) - h_y(y) & \\mathrm{if\\ } x_k \\mathrm{\\ is\\ feasible}\\\\\n        f_{s(x_k)}(x_k) - f_{s(x_k)}(y) & \\mathrm{otherwise.}\n    \\end{cases}\\label{eq:delta_k-definition}\n\\end{align}","sha256_utf8":"f28d0b17b78d4a93ed35b2d6c9ea50a43d08773692f6f4c038b3950e0395b10a"},{"start":37915,"end":38720,"start_line":447,"end_line":453,"raw_tex":"The sign of $\\delta_k(y)$ may vary. If $y = x_{\\mathtt{OPT}}$, then $\\delta_k(x_\\mathtt{OPT})$ is nonnegative, being lower bounded by the level of suboptimality or current infeasibility\n$$ \\delta_k(x_\\mathtt{OPT}) \\geq \\begin{cases}\n        h_{x_\\mathtt{OPT}}(x_k) - p_\\star & \\mathrm{if\\ } x_k \\mathrm{\\ is\\ feasible}\\\\\n        f_{s(x_k)}(x_k) & \\mathrm{otherwise}\n    \\end{cases} \\ \\geq 0 \\ . $$\nWhen $y = x_{\\mathtt{SL}}$ and $x_k$ is feasible, $\\delta_k(y)$ may be negative but is bounded below by\n$$ \\delta_k(x_\\mathtt{SL}) \\geq \\inf h_{x_{\\mathtt{SL}}} -h_{x_{\\mathtt{SL}}}(x_{\\mathtt{SL}}) > -\\infty \\ . $$ When $y = x_{\\mathtt{SL}}$ and $x_k$ is infeasible, $\\delta_k(y)$ is strictly positive, being bounded below by $$\\delta_k(x_\\mathtt{SL}) \\geq 0-\\max_{s=1\\dots m} f_s(x_\\mathtt{SL}) >0 \\ . $$\n","sha256_utf8":"32ea4dd85f3b6728fcd622b9806a66282605934ce8b54ce7b75b6f80fbb54bd0"}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","title":"随机次梯度增长条件","source_units":[{"start":39311,"end":39841,"start_line":456,"end_line":462,"raw_tex":"\\noindent {\\bf Assumption C.} For both $y\\in\\{x_{\\mathtt{OPT}}, x_{\\mathtt{SL}}\\}$, there exist constants $L_0$, $L_1{\\color{blue}\\geq 0}$ such that every iterate $x_k$ has\n\\begin{equation}\\label{eq:NonLipschitz-condition}\n    \\begin{cases}\n        x_k \\mathrm{\\ is\\ feasible} & \\implies \\mathbb{E}_{\\xi_k}\\|g_0(x_k;\\xi_k) + n_{y}\\|_2^2 \\leq L_0^2 + L_1\\delta_k(y)\\\\\n        x_k \\mathrm{\\ is\\ not\\ feasible} & \\implies \\mathbb{E}_{\\xi_k}\\|g_{s(x_k)}(x_k;\\xi_k)\\|_2^2 \\leq L_0^2 + L_1\\delta_k(y) \\ . \n    \\end{cases}\n\\end{equation}","sha256_utf8":"1c37af056a250aa8c656538d03fbe0dff2a67345bfd2ac1bcb13394ab50eb6c5"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"初期发散常数","source_units":[{"start":42699,"end":43168,"start_line":483,"end_line":487,"raw_tex":"To understand and bound such behaviors, we introduce the following two constants, dependent on the choice of stepsizes $\\alpha_k$ and associated dual weights $\\lambda_k$,\n\\begin{align}\n    T_0 &:= \\sup\\left\\{ k\\in\\{0,1,2\\dots\\} \\mid L_1\\alpha_k > 1\\right\\} \\ ,\\label{eq:T0} \\\\\n    C_0 &:= \\sum_{k=0}^{T_0}\\lambda_k \\max\\left\\{L_1 \\alpha_k - 1, 0 \\right\\}\\max\\{\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}), \\mathbb{E}_\\xi\\delta_k(x_\\mathtt{SL})\\} \\ . \\label{eq:C0}\n\\end{align}","sha256_utf8":"2cae97c3b85e29a042975d99d5068bf2190282bc79c3dbbb387213f6addc0df1"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"原始对偶间隙及定义域","source_units":[{"start":55910,"end":56188,"start_line":596,"end_line":599,"raw_tex":"We prove a uniform convergence guarantee in terms of the primal gap\n\\begin{equation} \\label{eq:primalgap-definition}\n    \\mathtt{primal\\mbox{-}gap}_T := \\frac{\\sum_{k< T: s(x_k)=0} \\lambda_k h_{x_\\mathtt{OPT}}(x_k)}{\\sum_{k< T: s(x_k)=0} \\lambda_k } - p_\\star \\ ,\n\\end{equation}","sha256_utf8":"100397a4cd0fc6435a7d88369894c3dad8261148a1b461c84b2edd1b4db37f69"},{"start":56189,"end":56434,"start_line":600,"end_line":603,"raw_tex":"which utilizes a combination of the feasible objective values seen, the dual gap\n\\begin{equation} \\label{eq:dualgap-definition}\n    \\mathtt{dual\\mbox{-}gap}_T := p_\\star - \\inf \\frac{M^{(T-1)}}{\\sum_{k< T: s(x_k)=0} \\lambda_k }\\ ,\n\\end{equation}","sha256_utf8":"81f6b011d7c532737e9ce25657182ace10ba2017bf0b0f3474e328601b8867cb"},{"start":56435,"end":56887,"start_line":604,"end_line":605,"raw_tex":"which utilizes a combination of the subgradient lower bounds seen, and the distance to optimal.\nFor the primal and dual gaps to be well-defined, at least one feasible iterate must have been seen (i.e., $\\sum_{k< T: s(x_k)=0} \\lambda_k >0$). Our assumptions facilitate a bound on how long it takes for this to occur. We show the expected fraction of the dual weight that occurs on iterations with a feasible iterate (i.e., $s(x_k)=0$) is bounded below.\n","sha256_utf8":"afca0580e747fa77c0620c3f3bf52141506be72fa73ad0a39d4b5421f852c559"},{"start":58098,"end":58436,"start_line":614,"end_line":614,"raw_tex":"Once a feasible iterate occurs, our primal and dual convergence measures are well-defined. Alternatively, one could assume the initialization $x_0$ is feasible to ensure these quantities are well-defined. In either case, Subsection~\\ref{subsec:proof-main-result} proves the following primal-dual convergence guarantee as our main result.\n","sha256_utf8":"2be9270a8b422548f2963a9814a0c602e2c443483846748c5fea27ffc48c68f6"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"原始对偶势函数与记号","source_units":[{"start":70214,"end":70363,"start_line":704,"end_line":706,"raw_tex":"\\begin{equation} \\label{eq:Rk-definition}\nR_k(y) := \\left(\\frac{\\mu}{2}\\sum_{i=0}^{{k-1}} \\lambda_i\\right) \\mathbb{E}_\\xi\\|x_k-y\\|_2^2\n\\end{equation}","sha256_utf8":"2db352d4f3eac09e5ac88b8506fd22a7df5a65614173ddfc25ec86ddbb16e63f"},{"start":70364,"end":70563,"start_line":707,"end_line":707,"raw_tex":"To simplify notations, throughout our analysis, we denote $g_k = g_{s(x_k)}(x_k;\\xi_k)$ and $w_k = \\mu\\sum_{i=0}^{k} \\lambda_i$ (with the convention that $w_{-1}=0$ as the given summation is empty).\n","sha256_utf8":"fd027c96445ae9dce42d48498a3441c4805781551dbe4cd01da0103f36199980"},{"start":73297,"end":73464,"start_line":733,"end_line":735,"raw_tex":"\\begin{equation} \\label{eq:Dk-definition}\n    D_k := \\mathbb{E}_\\xi\\left[\\left(\\sum_{i< k:s(x_i)=0} \\lambda_i\\right) p_\\star - \\inf M^{(k-1)}\\right] \\ .\n\\end{equation}","sha256_utf8":"adde246f983d1ff20823e030a669ddfb38f93cc95cd713684b7ef283ea26b05f"}]},{"id":"S015","semantic_type":"theorem","kind":"theorem","title":"随机方法原始对偶收敛界","source_units":[{"start":58436,"end":59108,"start_line":615,"end_line":621,"raw_tex":"\\begin{theorem}\\label{thm:super-rate}\n    Under Assumptions A-C, for any primal stepsizes $\\alpha_k>0$ and dual weights $\\lambda_k>0$ satisfying~\\eqref{eq:alpha-lambda} with $\\bar\\beta=0$, the stochastic switching proximal subgradient method~\\eqref{eq:super-primal-method} has\n    \\begin{align*}\n        &\\mathbb{E}_\\xi \\left[\\left(\\frac{\\sum_{k< T: s(x_k)=0} \\lambda_k}{\\sum_{k=0}^{T-1} \\lambda_k}\\right)\\left(\\mathtt{primal\\mbox{-}gap}_T + \\mathtt{dual\\mbox{-}gap}_T\\right) + \\frac{\\mu}{2}\\|x_{T}-x_\\mathtt{OPT}\\|_2^2\\right] \\\\\n        &\\qquad \\qquad \\leq \\frac{ L_0^2\\sum_{k=0}^{T-1}\\lambda_k\\alpha_k + C_0}{\\sum_{k=0}^{T-1}\\lambda_k} \\ .\n    \\end{align*}\n\\end{theorem}","sha256_utf8":"8e8b9ecf8123876ef5a2a8b04c476ae720afe10571f35b9817db10bc1295b90d"}]},{"id":"S016","semantic_type":"assumption","kind":"assumption","title":"主定理步长权重条件","source_units":[{"start":58474,"end":58713,"start_line":616,"end_line":616,"raw_tex":"    Under Assumptions A-C, for any primal stepsizes $\\alpha_k>0$ and dual weights $\\lambda_k>0$ satisfying~\\eqref{eq:alpha-lambda} with $\\bar\\beta=0$, the stochastic switching proximal subgradient method~\\eqref{eq:super-primal-method} has\n","sha256_utf8":"a533329ae9d042380aaf27060465ea71d095bd3c2cf44bd1ce729950b8e91f03"}]},{"id":"S017","semantic_type":"proof","kind":"proof","title":"原始对偶收敛完整证明","source_units":[{"start":84932,"end":87267,"start_line":832,"end_line":848,"raw_tex":"\\subsubsection{Proof of Theorem~\\ref{thm:super-rate}} \\label{subsec:proof-main-result}\nApplying Lemma~\\ref{lem:switching-proximal-primal-induction} with $y=x_\\mathtt{OPT}$ from $k=0$ to $T-1$ yields\n$$R_T(x_\\mathtt{OPT}) + \\sum_{k=0}^{T-1} \\frac{\\lambda_k}{2}\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) \\leq R_0(x_\\mathtt{OPT}) + \\sum_{k=0}^{T-1}\\frac{L_0^2\\lambda_k\\alpha_k}{2} + \\sum_{k=0}^{T-1} \\frac{\\lambda_k}{2}\\left(L_1\\alpha_k - 1 \\right)\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT})\\ .$$\nSimilarly, applying Lemma~\\ref{lem:switching-proximal-dual-induction} from $k=0$ to $T-1$ yields\n\\begin{align*}\n    D_T &+ \\sum_{k=0}^{T-1} \\frac{\\lambda_k}{2}(\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT})) + {\\color{blue}\\sum_{k<T:s(x_k)>0} \\lambda_k\\mathbb{E}_\\xi f_{s(x_k)}(x_\\mathtt{OPT})}\\\\\n    &\\leq D_0 + \\sum_{k=0}^{T-1}\\frac{L_0^2\\lambda_k\\alpha_k}{2} + \\sum_{k=0}^{T-1} \\frac{\\lambda_k}{2}\\left(L_1\\alpha_k - 1 \\right)\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT})\\ .\n\\end{align*}\nNoting $R_0(x_\\mathtt{OPT})=D_0=0$ and the last summation in each bound is at most half our initial blow-up constant $\\frac{1}{2}C_0$, the sum of these inequalities provides a bound of\n$$ R_T(x_\\mathtt{OPT}) + D_T + \\sum_{k=0}^{T-1} \\lambda_k\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) {\\color{blue} + \\sum_{k<T:s(x_k)>0} \\lambda_k\\mathbb{E}_\\xi}f_{s(x_k)}(x_\\mathtt{OPT}) \\leq \\sum_{k=0}^{T-1}L_0^2\\lambda_k\\alpha_k + C_0 \\ . $$\nLower bounding each $\\mathbb{E}_\\xi[\\delta_k(x_\\mathtt{OPT}) {\\color{blue}+  f_{s(x_k)}(x_\\mathtt{OPT})}]$ with $s(x_k)\\neq 0$ by zero {\\color{blue} and dividing through by $\\sum_{k=0}^{T-1} \\lambda_k$ establishes that\n$$ \\frac{1}{\\sum_{k=0}^{T-1} \\lambda_k}\\left(R_T(x_\\mathtt{OPT}) + D_T + \\sum_{k<T:s(x_k)=0} \\lambda_k\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT})\\right) \\leq \\frac{\\sum_{k=0}^{T-1}L_0^2\\lambda_k\\alpha_k + C_0}{\\sum_{k=0}^{T-1} \\lambda_k} \\ . $$\nSubstituting the definitions of $R_T$ and $D_T$ from~\\eqref{eq:Rk-definition} and~\\eqref{eq:Dk-definition}, the left-hand side above is equal to\n$$ \\mathbb{E}_\\xi\\left[\\frac{\\mu}{2}\\|x_T-x_\\mathtt{OPT}\\|^2_2 + \\frac{1}{{\\sum_{k=0}^{T-1} \\lambda_k}}\\left(\\sum_{k<T:s(x_k)=0} \\lambda_k h_{x_\\mathtt{OPT}}(x_k)- \\inf M^{(T-1)}\\right)\\right]\\ . $$\nRecalling the primal and dual gaps definitions in~\\eqref{eq:primalgap-definition} and~\\eqref{eq:dualgap-definition} gives our main theorem.}\n\n    ","sha256_utf8":"91fe0ce6b94e67409f79499c8fa2cd93d07c583e46a80a76f0b368ca4606cc56"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"累加原始递推式","source_units":[{"start":84985,"end":85417,"start_line":832,"end_line":834,"raw_tex":" \\label{subsec:proof-main-result}\nApplying Lemma~\\ref{lem:switching-proximal-primal-induction} with $y=x_\\mathtt{OPT}$ from $k=0$ to $T-1$ yields\n$$R_T(x_\\mathtt{OPT}) + \\sum_{k=0}^{T-1} \\frac{\\lambda_k}{2}\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) \\leq R_0(x_\\mathtt{OPT}) + \\sum_{k=0}^{T-1}\\frac{L_0^2\\lambda_k\\alpha_k}{2} + \\sum_{k=0}^{T-1} \\frac{\\lambda_k}{2}\\left(L_1\\alpha_k - 1 \\right)\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT})\\ .$$\n","sha256_utf8":"102d039c45ce1be0a5cd64f587408f738c1f5e9844add8a72b8d0c7aa710a128"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"累加对偶递推式","source_units":[{"start":85417,"end":85893,"start_line":835,"end_line":839,"raw_tex":"Similarly, applying Lemma~\\ref{lem:switching-proximal-dual-induction} from $k=0$ to $T-1$ yields\n\\begin{align*}\n    D_T &+ \\sum_{k=0}^{T-1} \\frac{\\lambda_k}{2}(\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT})) + {\\color{blue}\\sum_{k<T:s(x_k)>0} \\lambda_k\\mathbb{E}_\\xi f_{s(x_k)}(x_\\mathtt{OPT})}\\\\\n    &\\leq D_0 + \\sum_{k=0}^{T-1}\\frac{L_0^2\\lambda_k\\alpha_k}{2} + \\sum_{k=0}^{T-1} \\frac{\\lambda_k}{2}\\left(L_1\\alpha_k - 1 \\right)\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT})\\ .\n\\end{align*}\n","sha256_utf8":"6c263636ff73ef7c448b2e790075bccfc70a7d182cc3d0fdabc772aa2a16bcd7"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"合并递推并控制初期项","source_units":[{"start":85893,"end":86318,"start_line":840,"end_line":841,"raw_tex":"Noting $R_0(x_\\mathtt{OPT})=D_0=0$ and the last summation in each bound is at most half our initial blow-up constant $\\frac{1}{2}C_0$, the sum of these inequalities provides a bound of\n$$ R_T(x_\\mathtt{OPT}) + D_T + \\sum_{k=0}^{T-1} \\lambda_k\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) {\\color{blue} + \\sum_{k<T:s(x_k)>0} \\lambda_k\\mathbb{E}_\\xi}f_{s(x_k)}(x_\\mathtt{OPT}) \\leq \\sum_{k=0}^{T-1}L_0^2\\lambda_k\\alpha_k + C_0 \\ . $$\n","sha256_utf8":"c2d418e2bd2ff0337ed2b6a219fb3aac979fa3f7973d60c7e8701789892d0aff"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"舍去非负约束项并归一化","source_units":[{"start":86318,"end":86777,"start_line":842,"end_line":843,"raw_tex":"Lower bounding each $\\mathbb{E}_\\xi[\\delta_k(x_\\mathtt{OPT}) {\\color{blue}+  f_{s(x_k)}(x_\\mathtt{OPT})}]$ with $s(x_k)\\neq 0$ by zero {\\color{blue} and dividing through by $\\sum_{k=0}^{T-1} \\lambda_k$ establishes that\n$$ \\frac{1}{\\sum_{k=0}^{T-1} \\lambda_k}\\left(R_T(x_\\mathtt{OPT}) + D_T + \\sum_{k<T:s(x_k)=0} \\lambda_k\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT})\\right) \\leq \\frac{\\sum_{k=0}^{T-1}L_0^2\\lambda_k\\alpha_k + C_0}{\\sum_{k=0}^{T-1} \\lambda_k} \\ . $$\n","sha256_utf8":"2cd324761dfdb034e8dca92ecca55c5862b7c5d54a07e1fa4c42f4f23f659549"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"代回势函数与间隙定义","source_units":[{"start":86777,"end":87262,"start_line":844,"end_line":846,"raw_tex":"Substituting the definitions of $R_T$ and $D_T$ from~\\eqref{eq:Rk-definition} and~\\eqref{eq:Dk-definition}, the left-hand side above is equal to\n$$ \\mathbb{E}_\\xi\\left[\\frac{\\mu}{2}\\|x_T-x_\\mathtt{OPT}\\|^2_2 + \\frac{1}{{\\sum_{k=0}^{T-1} \\lambda_k}}\\left(\\sum_{k<T:s(x_k)=0} \\lambda_k h_{x_\\mathtt{OPT}}(x_k)- \\inf M^{(T-1)}\\right)\\right]\\ . $$\nRecalling the primal and dual gaps definitions in~\\eqref{eq:primalgap-definition} and~\\eqref{eq:dualgap-definition} gives our main theorem.}\n","sha256_utf8":"9142fc16d2a976985297d20dd0d6f5136b6c6147b78f52404f1d7c81e5e9160e"}]},{"id":"S023","semantic_type":"lemma","kind":"lemma","title":"原始距离递推引理","source_units":[{"start":70563,"end":70985,"start_line":708,"end_line":711,"raw_tex":"\\begin{lemma}\\label{lem:switching-proximal-primal-induction}\n    Under Assumptions A-C, the switching proximal subgradient method~\\eqref{eq:super-primal-method} with $\\alpha_k=\\lambda_k/\\mu\\sum_{i=0}^k\\lambda_i$ has for either $y\\in\\{x_{\\mathtt{OPT}}, x_{\\mathtt{SL}}\\}$\n    $$R_{k+1}(y) \\leq R_k(y) - \\frac{\\lambda_k}{2}\\left(\\left(2-L_1\\alpha_k\\right)\\mathbb{E}_\\xi\\delta_{k}(y) - L_0^2\\alpha_k\\right) \\ . $$\n\\end{lemma}","sha256_utf8":"8dc7d8324cc0db65adbd6fe499d108a2e5288928c8b875ad0e62524ae3a47542"}]},{"id":"S024","semantic_type":"proof","kind":"proof","title":"原始递推完整证明","source_units":[{"start":70986,"end":73149,"start_line":712,"end_line":731,"raw_tex":"\\begin{proof}\n    This proof follows a standard analysis technique, directly expanding the definition of $R_{k+1}(y)$. First, suppose $x_k$ is feasible. Then\n    \\begin{align*}\n    R_{k+1}(y) &= \\frac{w_k}{2}\\mathbb{E}_\\xi\\|\\mathrm{prox}_{\\alpha_k,r}(x_k - \\alpha_k g_k) - \\mathrm{prox}_{\\alpha_k,r}(y +\\alpha_k n_{y})\\|_2^2\\\\\n    &\\leq \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k - \\alpha_k(g_k+n_{y}) - y\\|_2^2\\\\\n    &= \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k-y\\|_2^2 - \\lambda_k\\mathbb{E}_\\xi\\langle g_k + n_{y}, x_k-y\\rangle +\\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{y}\\|_2^2\\\\\n    &\\leq \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k-y\\|_2^2 - \\lambda_k\\mathbb{E}_\\xi(h_y(x_k)-h_y(y)+\\frac{\\mu}{2}\\|x_k - y\\|_2^2) + \\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{y}\\|_2^2\\\\\n    &= R_k(y) - \\lambda_k\\mathbb{E}_\\xi\\delta_k(y)+\\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{y}\\|_2^2\n   \\end{align*}\n    where the first line uses that $\\mathrm{prox}_{\\alpha_k,r}(y +\\alpha_k n_{y}) = y$, the second uses the nonexpansiveness of the proximal operator~\\cite[Proposition 12.19]{RockWets98}, the third factors the norm squared and uses $\\alpha_k = \\lambda_k/w_k$, the fourth uses the strong convexity of $h_y$, {\\color{blue} and the fifth follows from definitions as $R_k(y) = \\frac{w_k - \\lambda_k\\mu}{2}\\mathbb{E}_\\xi\\|x_k-y\\|_2^2$ and $\\delta_k(y)=h_y(x_k)-h_y(y)$}.\n    Then, applying the bound~\\eqref{eq:NonLipschitz-condition} gives the claim.\n    Similarly, supposing $x_k$ is infeasible,\n    \\begin{align*}\n    R_{k+1}(y) &= \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k - \\alpha_k g_k - y\\|_2^2\\\\\n    &= \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k-y\\|_2^2 - \\lambda_k\\mathbb{E}_\\xi\\langle g_k, x_k-y\\rangle + \\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k\\|_2^2\\\\\n    &\\leq \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k-y\\|_2^2- \\lambda_k\\mathbb{E}_\\xi(f_{s(x_k)}(x_k)-f_{s(x_k)}(y) +\\frac{\\mu}{2}\\|x_k - y\\|_2^2) + \\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k\\|_2^2\\\\\n    &= R_k(y) - \\lambda_k\\mathbb{E}_\\xi\\delta_k(y)+\\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k\\|_2^2 \n   \\end{align*}\n   using strong convexity of $f_{s(x_k)}$. Applying~\\eqref{eq:NonLipschitz-condition} completes the proof.\n\\end{proof}","sha256_utf8":"2b078c268a3c911011fb6cdca03764e36e8a86f4cf0cf723007c4581f326914d"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"可行步近端距离展开","source_units":[{"start":71000,"end":72337,"start_line":713,"end_line":721,"raw_tex":"    This proof follows a standard analysis technique, directly expanding the definition of $R_{k+1}(y)$. First, suppose $x_k$ is feasible. Then\n    \\begin{align*}\n    R_{k+1}(y) &= \\frac{w_k}{2}\\mathbb{E}_\\xi\\|\\mathrm{prox}_{\\alpha_k,r}(x_k - \\alpha_k g_k) - \\mathrm{prox}_{\\alpha_k,r}(y +\\alpha_k n_{y})\\|_2^2\\\\\n    &\\leq \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k - \\alpha_k(g_k+n_{y}) - y\\|_2^2\\\\\n    &= \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k-y\\|_2^2 - \\lambda_k\\mathbb{E}_\\xi\\langle g_k + n_{y}, x_k-y\\rangle +\\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{y}\\|_2^2\\\\\n    &\\leq \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k-y\\|_2^2 - \\lambda_k\\mathbb{E}_\\xi(h_y(x_k)-h_y(y)+\\frac{\\mu}{2}\\|x_k - y\\|_2^2) + \\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{y}\\|_2^2\\\\\n    &= R_k(y) - \\lambda_k\\mathbb{E}_\\xi\\delta_k(y)+\\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{y}\\|_2^2\n   \\end{align*}\n    where the first line uses that $\\mathrm{prox}_{\\alpha_k,r}(y +\\alpha_k n_{y}) = y$, the second uses the nonexpansiveness of the proximal operator~\\cite[Proposition 12.19]{RockWets98}, the third factors the norm squared and uses $\\alpha_k = \\lambda_k/w_k$, the fourth uses the strong convexity of $h_y$, {\\color{blue} and the fifth follows from definitions as $R_k(y) = \\frac{w_k - \\lambda_k\\mu}{2}\\mathbb{E}_\\xi\\|x_k-y\\|_2^2$ and $\\delta_k(y)=h_y(x_k)-h_y(y)$}.\n","sha256_utf8":"c29ec34099120b83eb86e78251ef80001cbdfcba4d1a2e0f0b8c8fdc84ea676a"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"可行步应用增长界","source_units":[{"start":72337,"end":72417,"start_line":722,"end_line":722,"raw_tex":"    Then, applying the bound~\\eqref{eq:NonLipschitz-condition} gives the claim.\n","sha256_utf8":"d6686de0d441faf5ed73d85f8cb44b678da61298929d78cf34e7d44b0ce55b20"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"不可行步展开与增长界","source_units":[{"start":72417,"end":73138,"start_line":723,"end_line":730,"raw_tex":"    Similarly, supposing $x_k$ is infeasible,\n    \\begin{align*}\n    R_{k+1}(y) &= \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k - \\alpha_k g_k - y\\|_2^2\\\\\n    &= \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k-y\\|_2^2 - \\lambda_k\\mathbb{E}_\\xi\\langle g_k, x_k-y\\rangle + \\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k\\|_2^2\\\\\n    &\\leq \\frac{w_k}{2}\\mathbb{E}_\\xi\\|x_k-y\\|_2^2- \\lambda_k\\mathbb{E}_\\xi(f_{s(x_k)}(x_k)-f_{s(x_k)}(y) +\\frac{\\mu}{2}\\|x_k - y\\|_2^2) + \\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k\\|_2^2\\\\\n    &= R_k(y) - \\lambda_k\\mathbb{E}_\\xi\\delta_k(y)+\\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k\\|_2^2 \n   \\end{align*}\n   using strong convexity of $f_{s(x_k)}$. Applying~\\eqref{eq:NonLipschitz-condition} completes the proof.\n","sha256_utf8":"e89a84d837c54ed602cf577e243ac736aa2ff17a1f7e1c75af92c13847b16689"}]},{"id":"S028","semantic_type":"lemma","kind":"lemma","title":"对偶间隙递推引理","source_units":[{"start":73714,"end":74321,"start_line":737,"end_line":743,"raw_tex":"\\begin{lemma}\\label{lem:switching-proximal-dual-induction}\n    Under Assumptions A-C, the switching proximal subgradient method~\\eqref{eq:super-primal-method} with $\\alpha_k=\\lambda_k/\\mu\\sum_{i=0}^k\\lambda_i$ has\n     $$ D_{k+1} \\leq D_k - \\frac{\\lambda_k}{2}\\begin{cases}\n         \\left(2-L_1\\alpha_k\\right)\\mathbb{E}_\\xi\\delta_{k}(x_\\mathtt{OPT}) - L_0^2\\alpha_k & \\text{if } x_k \\text{ is feasible}\\\\\n         {\\color{blue}\\left(2-L_1\\alpha_k\\right)\\mathbb{E}_\\xi\\delta_{k}(x_\\mathtt{OPT}) + 2\\mathbb{E}_\\xi f_{s(x_k)}(x_\\mathtt{OPT}) - L_0^2\\alpha_k} & \\text{otherwise.}\n     \\end{cases} $$\n\\end{lemma}","sha256_utf8":"18e0aaaea366f08386c4db87a20db3cb6f0777fccae1a33f1245558d0375cdea"}]},{"id":"S029","semantic_type":"proof","kind":"proof","title":"对偶递推完整证明","source_units":[{"start":74322,"end":79345,"start_line":744,"end_line":787,"raw_tex":"\\begin{proof}\nObserve that one can rewrite $M^{(k)}(x) = Q_1(x) + Q_2(x)$ as the sum of two quadratics where $Q_1(x) = M^{(k-1)}(x)$ and $Q_2$ depends on whether $x_k$ is feasible. In the notation of Lemma~\\ref{lem:quad-calc}, Lemmas~\\ref{lem:Mk-minimization} {\\color{blue} and~\\ref{lem:standard-quadratic}} ensure $Q_1$ has $a_1 = \\inf M^{(k-1)}$, $b_1 = \\mu\\sum_{i=0}^{k-1}\\lambda_i$, and $z_1 = x_{k}${\\color{blue}, since $\\bar \\beta = 0$ here}. To determine $Q_2$, first suppose $x_k$ is feasible. Then we have\n$$Q_2(x) = \\lambda_k \\left(f_{0}(x_k) + \\langle g_k, x-x_k \\rangle +\\frac{\\mu}{2}\\|x - x_k\\|_2^2 + r(x_{k+1}) + \\langle n_{k+1}, x-x_{k+1}\\rangle\\right) \\ . $$\nThis quadratic can be written in the form $Q_2(z) = a_2 + \\frac{b_2}{2}\\|z-z_2\\|_2^2$ with\n\\begin{align*}\n    a_2 &= \\lambda_k\\left(f_0(x_k) + r(x_{k+1}) + \\alpha_k\\langle n_{k+1}, g_{k}+n_{k+1}\\rangle - \\frac{1}{2\\mu}\\|g_k + n_{k+1}\\|_2^2 \\right)\\ , \\\\\n    b_2 &= \\mu\\lambda_k \\ , \\qquad z_2 = x_k - \\frac{g_k+n_{k+1}}{\\mu} \\ . \n\\end{align*}\nBy Lemma~\\ref{lem:quad-calc}, the expected minimum value of the updated model $\\mathbb{E}_\\xi \\inf M^{(k)}$ is\n\\begin{align*}\n    &\\mathbb{E}_\\xi \\inf M^{(k-1)} + \\lambda_k \\mathbb{E}_\\xi\\Bigg(f_0(x_k) + r(x_{k+1}) + \\alpha_k \\langle n_{k+1}, g_{k}+n_{k+1}\\rangle - \\frac{1}{2\\mu}\\|g_k + n_{k+1}\\|_2^2\\\\\n    & \\qquad\\qquad\\qquad\\qquad\\qquad + \\frac{\\sum_{i=0}^{k-1}\\lambda_i }{2\\mu \\sum_{i=0}^k \\lambda_i}\\left\\|g_{k}+n_{k+1}\\right\\|_2^2\\Bigg) \\ . \n\\end{align*}\nFrom this, we conclude the lower bound\n\\begin{align*}\n    \\mathbb{E}_\\xi \\inf M^{(k)} &\\geq \\mathbb{E}_\\xi \\inf M^{(k-1)} + \\lambda_k \\mathbb{E}_\\xi\\Bigg(f_0(x_k) + r(x_\\mathtt{OPT}) + \\langle n_{x_\\mathtt{OPT}}, x_{k+1}-x_\\mathtt{OPT}\\rangle  \\\\\n    & \\quad + \\alpha_k \\langle n_{k+1}, g_{k}+n_{k+1}\\rangle - \\frac{1}{2\\mu}\\|g_k + n_{k+1}\\|_2^2 + \\frac{\\sum_{i=0}^{k-1}\\lambda_i }{2\\mu \\sum_{i=0}^k \\lambda_i}\\left\\|g_{k}+n_{k+1}\\right\\|_2^2\\Bigg)\\\\\n    &= \\mathbb{E}_\\xi \\inf M^{(k-1)} + \\lambda_k \\mathbb{E}_\\xi\\Bigg(\\delta_k(x_\\mathtt{OPT}) + p_\\star + \\langle n_{x_\\mathtt{OPT}}, -\\alpha_k(g_k + n_{k+1})\\rangle  \\\\\n    & \\quad + \\alpha_k \\langle n_{k+1}, g_{k}+n_{k+1}\\rangle - \\frac{\\alpha_k}{2}\\|g_k + n_{k+1}\\|_2^2\\Bigg)\\\\\n    &= \\mathbb{E}_\\xi\\inf M^{(k-1)} + \\lambda_k \\left(\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) + p_\\star - \\frac{\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{x_\\mathtt{OPT}}\\|_2^2 { \\color{blue} + \\frac{\\alpha_k}{2}\\|n_{k+1}-n_{x_\\mathtt{OPT}} \\|_2^2}\\right)\\\\\n    & {\\color{blue} \\geq \\mathbb{E}_\\xi\\inf M^{(k-1)} + \\lambda_k \\left(\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) + p_\\star - \\frac{\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{x_\\mathtt{OPT}}\\|_2^2\\right)}\n\\end{align*}\nwhere the first inequality lower bounds $r(x_{k+1})$ by $r(x_\\mathtt{OPT}) + \\langle n_{x_\\mathtt{OPT}}, x_{k+1}-x_\\mathtt{OPT}\\rangle$, the first equality applies the definitions of $\\delta_k(x_\\mathtt{OPT})$ in~\\eqref{eq:delta_k-definition}, $\\alpha_k$ in~\\eqref{eq:alpha-lambda} {\\color{blue} with $\\bar \\beta = 0$, and that $x_{k+1}-x_k=-\\alpha_k(g_k+n_{k+1})$}, and the {\\color{blue} second} equality combines and simplifies terms.\nIn terms of $D_k = \\mathbb{E}_\\xi[\\sum_{i< k:s(x_i)=0} \\lambda_i p_\\star - \\inf M^{(k-1)}]$, this gives the following \nrecurrence\n$$ D_{k+1} \\leq D_k - \\lambda_k \\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) + \\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{x_\\mathtt{OPT}}\\|_2^2 \\ . $$\nApplying~\\eqref{eq:NonLipschitz-condition} gives the claim in this case.\nNow suppose $x_k$ is infeasible. Then, noting $Q_2$ minimizes at $x_k - \\frac{g_{k}}{\\mu}$, Lemma~\\ref{lem:standard-quadratic} ensures\n\\begin{align*}\n    Q_2(x) &= \\lambda_k\\left(f_{s(x_k)}(x_k) - \\frac{1}{2\\mu}\\|g_k \\|_2^2 + \\frac{\\mu}{2}\\left\\|x - \\left(x_k - \\frac{g_{k}}{\\mu}\\right)\\right\\|_2^2\\right) \\ .\n\\end{align*}\nThen by Lemma~\\ref{lem:quad-calc}, the expected minimum value of the updated model $\\mathbb{E}_\\xi \\inf M^{(k)}$ is given by\n\\begin{align*}\n    &\\mathbb{E}_\\xi\\inf  M^{(k-1)} + \\lambda_k\\mathbb{E}_\\xi\\left(f_{s(x_k)}(x_k) - \\frac{1}{2\\mu}\\|g_k \\|_2^2\\right) + \\frac{\\lambda_k\\sum_{i=0}^{k-1}\\lambda_i }{{\\color{blue} 2}\\mu \\sum_{i=0}^k \\lambda_i}\\mathbb{E}_\\xi\\left\\|g_{k}\\right\\|_2^2 \\\\\n    &= \\mathbb{E}_\\xi\\inf M^{(k-1)} + \\lambda_k\\mathbb{E}_\\xi f_{s(x_k)}(x_k) - \\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k\\|_2^2 \\ . \n\\end{align*}\n{\\color{blue}\nIn terms of $D_k = \\mathbb{E}_\\xi[\\sum_{i< k:s(x_i)=0} \\lambda_i p_\\star - \\mathbb{E}_\\xi\\inf M^{(k-1)}]$, this gives the recurrence\n$$ D_{k+1} = D_k - \\lambda_k \\mathbb{E}_\\xi\\left[\\delta_{k}(x_\\mathtt{OPT}) + f_{s(x_k)}(x_{\\mathtt{OPT}})\\right] + \\frac{\\lambda_k\\alpha_k}{2} \\mathbb{E}_\\xi\\|g_k\\|_2^2 \\ . $$\nBounding $\\mathbb{E}_\\xi\\|g_k\\|_2^2$ by~\\eqref{eq:NonLipschitz-condition} gives the claim in this last case.\n}\n% Noting $f_{s(x_k)}(x_\\mathtt{OPT})\\leq 0$, we conclude the lower bound\n% $$\\mathbb{E}_\\xi \\inf M^{(k)} \\geq \\mathbb{E}_\\xi\\inf M^{(k-1)} + \\lambda_k \\left(\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) - \\frac{\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k\\|_2^2\\right) \\ .$$\n\\end{proof}","sha256_utf8":"d194add877daf2151cbcc164ac5c2a41b9947aaaf1faf0624d77df3567d427bd"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"可行步二次模型分解","source_units":[{"start":74336,"end":75340,"start_line":745,"end_line":751,"raw_tex":"Observe that one can rewrite $M^{(k)}(x) = Q_1(x) + Q_2(x)$ as the sum of two quadratics where $Q_1(x) = M^{(k-1)}(x)$ and $Q_2$ depends on whether $x_k$ is feasible. In the notation of Lemma~\\ref{lem:quad-calc}, Lemmas~\\ref{lem:Mk-minimization} {\\color{blue} and~\\ref{lem:standard-quadratic}} ensure $Q_1$ has $a_1 = \\inf M^{(k-1)}$, $b_1 = \\mu\\sum_{i=0}^{k-1}\\lambda_i$, and $z_1 = x_{k}${\\color{blue}, since $\\bar \\beta = 0$ here}. To determine $Q_2$, first suppose $x_k$ is feasible. Then we have\n$$Q_2(x) = \\lambda_k \\left(f_{0}(x_k) + \\langle g_k, x-x_k \\rangle +\\frac{\\mu}{2}\\|x - x_k\\|_2^2 + r(x_{k+1}) + \\langle n_{k+1}, x-x_{k+1}\\rangle\\right) \\ . $$\nThis quadratic can be written in the form $Q_2(z) = a_2 + \\frac{b_2}{2}\\|z-z_2\\|_2^2$ with\n\\begin{align*}\n    a_2 &= \\lambda_k\\left(f_0(x_k) + r(x_{k+1}) + \\alpha_k\\langle n_{k+1}, g_{k}+n_{k+1}\\rangle - \\frac{1}{2\\mu}\\|g_k + n_{k+1}\\|_2^2 \\right)\\ , \\\\\n    b_2 &= \\mu\\lambda_k \\ , \\qquad z_2 = x_k - \\frac{g_k+n_{k+1}}{\\mu} \\ . \n\\end{align*}\n","sha256_utf8":"768645b439869fe418a80cf2e6a53bf3d8350cfa8abe277e9f60d7e8af8d6c97"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"计算可行步模型最小值","source_units":[{"start":75340,"end":75802,"start_line":752,"end_line":756,"raw_tex":"By Lemma~\\ref{lem:quad-calc}, the expected minimum value of the updated model $\\mathbb{E}_\\xi \\inf M^{(k)}$ is\n\\begin{align*}\n    &\\mathbb{E}_\\xi \\inf M^{(k-1)} + \\lambda_k \\mathbb{E}_\\xi\\Bigg(f_0(x_k) + r(x_{k+1}) + \\alpha_k \\langle n_{k+1}, g_{k}+n_{k+1}\\rangle - \\frac{1}{2\\mu}\\|g_k + n_{k+1}\\|_2^2\\\\\n    & \\qquad\\qquad\\qquad\\qquad\\qquad + \\frac{\\sum_{i=0}^{k-1}\\lambda_i }{2\\mu \\sum_{i=0}^k \\lambda_i}\\left\\|g_{k}+n_{k+1}\\right\\|_2^2\\Bigg) \\ . \n\\end{align*}\n","sha256_utf8":"1818e375a7da7fe6e25bc4f329b152c36b43eb4bec3ccbab3cb8094bcf4bd1e8"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"参考次梯度给出模型下界","source_units":[{"start":75802,"end":77427,"start_line":757,"end_line":766,"raw_tex":"From this, we conclude the lower bound\n\\begin{align*}\n    \\mathbb{E}_\\xi \\inf M^{(k)} &\\geq \\mathbb{E}_\\xi \\inf M^{(k-1)} + \\lambda_k \\mathbb{E}_\\xi\\Bigg(f_0(x_k) + r(x_\\mathtt{OPT}) + \\langle n_{x_\\mathtt{OPT}}, x_{k+1}-x_\\mathtt{OPT}\\rangle  \\\\\n    & \\quad + \\alpha_k \\langle n_{k+1}, g_{k}+n_{k+1}\\rangle - \\frac{1}{2\\mu}\\|g_k + n_{k+1}\\|_2^2 + \\frac{\\sum_{i=0}^{k-1}\\lambda_i }{2\\mu \\sum_{i=0}^k \\lambda_i}\\left\\|g_{k}+n_{k+1}\\right\\|_2^2\\Bigg)\\\\\n    &= \\mathbb{E}_\\xi \\inf M^{(k-1)} + \\lambda_k \\mathbb{E}_\\xi\\Bigg(\\delta_k(x_\\mathtt{OPT}) + p_\\star + \\langle n_{x_\\mathtt{OPT}}, -\\alpha_k(g_k + n_{k+1})\\rangle  \\\\\n    & \\quad + \\alpha_k \\langle n_{k+1}, g_{k}+n_{k+1}\\rangle - \\frac{\\alpha_k}{2}\\|g_k + n_{k+1}\\|_2^2\\Bigg)\\\\\n    &= \\mathbb{E}_\\xi\\inf M^{(k-1)} + \\lambda_k \\left(\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) + p_\\star - \\frac{\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{x_\\mathtt{OPT}}\\|_2^2 { \\color{blue} + \\frac{\\alpha_k}{2}\\|n_{k+1}-n_{x_\\mathtt{OPT}} \\|_2^2}\\right)\\\\\n    & {\\color{blue} \\geq \\mathbb{E}_\\xi\\inf M^{(k-1)} + \\lambda_k \\left(\\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) + p_\\star - \\frac{\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{x_\\mathtt{OPT}}\\|_2^2\\right)}\n\\end{align*}\nwhere the first inequality lower bounds $r(x_{k+1})$ by $r(x_\\mathtt{OPT}) + \\langle n_{x_\\mathtt{OPT}}, x_{k+1}-x_\\mathtt{OPT}\\rangle$, the first equality applies the definitions of $\\delta_k(x_\\mathtt{OPT})$ in~\\eqref{eq:delta_k-definition}, $\\alpha_k$ in~\\eqref{eq:alpha-lambda} {\\color{blue} with $\\bar \\beta = 0$, and that $x_{k+1}-x_k=-\\alpha_k(g_k+n_{k+1})$}, and the {\\color{blue} second} equality combines and simplifies terms.\n","sha256_utf8":"d78f8a77614472acac9fee1c5703fec58a9af33d4377ad46e8a9bfbbd62aa2be"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"得到可行步对偶递推","source_units":[{"start":77427,"end":77782,"start_line":767,"end_line":770,"raw_tex":"In terms of $D_k = \\mathbb{E}_\\xi[\\sum_{i< k:s(x_i)=0} \\lambda_i p_\\star - \\inf M^{(k-1)}]$, this gives the following \nrecurrence\n$$ D_{k+1} \\leq D_k - \\lambda_k \\mathbb{E}_\\xi\\delta_k(x_\\mathtt{OPT}) + \\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k+n_{x_\\mathtt{OPT}}\\|_2^2 \\ . $$\nApplying~\\eqref{eq:NonLipschitz-condition} gives the claim in this case.\n","sha256_utf8":"bc8284e0ba1691b359ec7fba0a04b05815d7333c1802f3a9718b84b82e512714"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"不可行步模型配方","source_units":[{"start":77782,"end":78105,"start_line":771,"end_line":774,"raw_tex":"Now suppose $x_k$ is infeasible. Then, noting $Q_2$ minimizes at $x_k - \\frac{g_{k}}{\\mu}$, Lemma~\\ref{lem:standard-quadratic} ensures\n\\begin{align*}\n    Q_2(x) &= \\lambda_k\\left(f_{s(x_k)}(x_k) - \\frac{1}{2\\mu}\\|g_k \\|_2^2 + \\frac{\\mu}{2}\\left\\|x - \\left(x_k - \\frac{g_{k}}{\\mu}\\right)\\right\\|_2^2\\right) \\ .\n\\end{align*}\n","sha256_utf8":"3e1b53a14313de983ba2adf9e246a4c0b2d3fb74a38302786a8e294c24351d83"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"计算不可行步模型最小值","source_units":[{"start":78105,"end":78644,"start_line":775,"end_line":779,"raw_tex":"Then by Lemma~\\ref{lem:quad-calc}, the expected minimum value of the updated model $\\mathbb{E}_\\xi \\inf M^{(k)}$ is given by\n\\begin{align*}\n    &\\mathbb{E}_\\xi\\inf  M^{(k-1)} + \\lambda_k\\mathbb{E}_\\xi\\left(f_{s(x_k)}(x_k) - \\frac{1}{2\\mu}\\|g_k \\|_2^2\\right) + \\frac{\\lambda_k\\sum_{i=0}^{k-1}\\lambda_i }{{\\color{blue} 2}\\mu \\sum_{i=0}^k \\lambda_i}\\mathbb{E}_\\xi\\left\\|g_{k}\\right\\|_2^2 \\\\\n    &= \\mathbb{E}_\\xi\\inf M^{(k-1)} + \\lambda_k\\mathbb{E}_\\xi f_{s(x_k)}(x_k) - \\frac{\\lambda_k\\alpha_k}{2}\\mathbb{E}_\\xi\\|g_k\\|_2^2 \\ . \n\\end{align*}\n","sha256_utf8":"c0fc387274ac4e2105b7b437a8c923a2fd80b29d27823d614fb235702220442c"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"得到不可行步对偶递推","source_units":[{"start":78644,"end":79079,"start_line":780,"end_line":784,"raw_tex":"{\\color{blue}\nIn terms of $D_k = \\mathbb{E}_\\xi[\\sum_{i< k:s(x_i)=0} \\lambda_i p_\\star - \\mathbb{E}_\\xi\\inf M^{(k-1)}]$, this gives the recurrence\n$$ D_{k+1} = D_k - \\lambda_k \\mathbb{E}_\\xi\\left[\\delta_{k}(x_\\mathtt{OPT}) + f_{s(x_k)}(x_{\\mathtt{OPT}})\\right] + \\frac{\\lambda_k\\alpha_k}{2} \\mathbb{E}_\\xi\\|g_k\\|_2^2 \\ . $$\nBounding $\\mathbb{E}_\\xi\\|g_k\\|_2^2$ by~\\eqref{eq:NonLipschitz-condition} gives the claim in this last case.\n}\n","sha256_utf8":"b26fcc777328a265ba2bc495d5c109944ce64e03f61237c591c887f0de01b0a5"}]},{"id":"S037","semantic_type":"theorem","kind":"theorem","title":"原始对偶迭代等价性","source_units":[{"start":44451,"end":45023,"start_line":494,"end_line":500,"raw_tex":"\\begin{theorem} \\label{thm:super-equivalence}\n    Let $\\{x_k\\}$ be the sequence of iterates of the primal method~\\eqref{eq:super-primal-method} with stepsizes $\\alpha_k>0$ and $\\{y_k\\}$ be the sequence of iterates of the dual method~\\eqref{eq:super-dual-method} for some $\\lambda_k,\\beta_k,\\mu\\geq 0$. If $x_0=y_0$, $\\beta_k=\\bar\\beta\\geq 0$ is constant, and\n    \\begin{equation}\n        \\alpha_k = \\frac{\\lambda_k}{\\mu\\sum_{i=0}^k\\lambda_i + \\bar\\beta}\\ , \\label{eq:alpha-lambda}\n    \\end{equation} \n    then these methods are equivalent, that is $x_k=y_k$.\n\\end{theorem}","sha256_utf8":"44405636a25282de182c671cdf60b8de28614d5286a6c50fe45d75e4c7767b14"}]},{"id":"S038","semantic_type":"assumption","kind":"assumption","title":"等价性初值与参数条件","source_units":[{"start":44497,"end":44952,"start_line":495,"end_line":498,"raw_tex":"    Let $\\{x_k\\}$ be the sequence of iterates of the primal method~\\eqref{eq:super-primal-method} with stepsizes $\\alpha_k>0$ and $\\{y_k\\}$ be the sequence of iterates of the dual method~\\eqref{eq:super-dual-method} for some $\\lambda_k,\\beta_k,\\mu\\geq 0$. If $x_0=y_0$, $\\beta_k=\\bar\\beta\\geq 0$ is constant, and\n    \\begin{equation}\n        \\alpha_k = \\frac{\\lambda_k}{\\mu\\sum_{i=0}^k\\lambda_i + \\bar\\beta}\\ , \\label{eq:alpha-lambda}\n    \\end{equation} \n","sha256_utf8":"262f142854ed3a97e46354f45c2b59243a5f00d1b3390b4f5dab427429afcbe5"}]},{"id":"S039","semantic_type":"proof","kind":"proof","title":"迭代等价性完整证明","source_units":[{"start":45024,"end":52980,"start_line":501,"end_line":574,"raw_tex":"\\begin{proof}\n        We prove this by inductively showing that $x_k=y_k$ and for any iteration with $s(x_k)=0$ that $n_{k+1} = \\frac{1}{\\alpha_k}(x_k-\\alpha_k g_{0}(x_k;\\xi_k) -x_{k+1})$ (i.e., the subgradients of $r$  produced by the dual method are exactly those produced by the primal method via~\\eqref{eq:proximal-equivalence}).\n    By assumption, $x_0=y_0$. Suppose for induction that $x_i=y_i$ for all $i=0,\\dots,k$ and for all $i=0,\\dots,k-1$ with $s(x_i)=0$ that $n_{i+1} = \\frac{1}{\\alpha_i}(x_i-\\alpha_i g_{0}(x_i;\\xi_i) -x_{i+1})$. Let $\\bar g_i = g_0(y_i;\\xi_i) + n_{i+1}$ if $y_i$ is feasible and $g_{s(y_i)}(y_i;\\xi_i)$ otherwise. For each $i<k$, we claim $x_{i+1} =  x_i - \\alpha_i \\bar g_i$. If $x_i$ is infeasible, this is immediate. If $x_i$ is feasible, {\\color{blue} rearrangement of our inductive hypothesis of on $n_{i+1}$ gives the claim}. Inductively, we conclude for $i<k$ that $ x_{i+1} - x_0 = - \\sum_{t=0}^i \\alpha_t \\bar g_t. $\n    \n    By Lemma~\\ref{lem:Mk-minimization}, $y_{k+1}$ is the unique solution to\n    $\\sum_{i=0}^k \\lambda_i\\left(\\bar g_i + \\mu(y_{k+1}-y_i) \\right) + \\bar\\beta(y_{k+1}-y_0) = 0. $\n    Rearranging and simplifying this, it follows that\n    \\begin{align*}\n    y_{k+1} &= y_0 + \\frac{\\sum_{i=0}^k\\lambda_i\\mu(y_i-y_0)-\\sum_{i=0}^k\\lambda_i\\bar g_i}{\\sum_{i=0}^k\\lambda_i\\mu+\\bar\\beta}\\\\\n    &= y_0 -\\frac{\\sum_{i=0}^{k} \\lambda_{i}\\mu\\left( \\sum_{t=0}^{i-1} \\frac{\\lambda_{t} \\bar g_t}{\\sum_{s=0}^{t}\\lambda_{s}\\mu + \\bar \\beta} \\right) + \\sum_{i=0}^{k}\\lambda_i \\bar g_i}{\\sum_{i=0}^{k}\\lambda_i\\mu + \\bar\\beta}\\\\\n    &= y_0 -\\frac{\\sum_{t=0}^{k-1} \\lambda_t \\bar g_t\\frac{\\sum_{i=t+1}^k \\mu\\lambda_i}{\\sum_{i=0}^t \\mu\\lambda_{{\\color{blue} i}} +\\bar\\beta} + \\sum_{i=0}^{k}\\lambda_i \\bar g_i}{\\sum_{i=0}^{k}\\lambda_i\\mu+\\bar\\beta}\\\\\n    &= y_0 -\\frac{\\sum_{t=0}^{k-1} \\lambda_t \\bar g_t\\left(\\frac{\\sum_{i=t+1}^k \\mu\\lambda_i}{\\sum_{i=0}^t \\mu\\lambda_{{\\color{blue} i}} +\\bar\\beta}+1\\right)}{\\sum_{i=0}^{k}\\lambda_i\\mu+\\bar\\beta} - \\frac{\\lambda_k \\bar g_k}{\\sum_{i=0}^{k}\\lambda_i\\mu + \\bar\\beta}\\\\\n    &=  y_0 -\\sum_{t=0}^{k-1} \\frac{\\lambda_t \\bar g_t}{\\sum_{i=0}^{t}\\lambda_{{\\color{blue} i}}\\mu+\\bar\\beta} - \\frac{\\lambda_k \\bar g_k}{\\sum_{i=0}^{k}\\lambda_i\\mu + \\bar\\beta}\\\\\n    &= x_k - \\alpha_k\\bar g_k\n    \\end{align*}\n    where the second equality uses the inductive hypothesis for each $y_i-y_0 = x_i-x_0 = - \\sum_{t=0}^{i-1} \\alpha_t\\bar g_t$, the third exchanges summands, and the remainder combines and simplifies terms. If $x_k$ is infeasible, {\\color{blue} it is} immediate that $x_{k+1}=x_k-\\alpha_k\\bar g_k=y_{k+1}$. If $x_k$ is feasible, the above equality ensures\n    $$ \\frac{1}{\\alpha_k}(x_k - \\alpha_k g_0(x_k;\\xi_k) -y_{k+1}) = n_{k+1} \\in\\partial r(y_{k+1}) \\ . $$\n    Noting by~\\eqref{eq:proximal-equivalence}, $x_{k+1}$ is the unique solution to $\\frac{1}{\\alpha_k}(x_k-\\alpha_kg_0(x_k;\\xi_k) -z)\\in\\partial r(z)$, we must have $x_{k+1} {\\color{blue} = x_k - \\alpha_k(g_k + n_{k+1})} = y_{k+1}$ and $n_{k+1} =  \\frac{1}{\\alpha_k}(x_k - \\alpha_k g_0(x_k;\\xi_k) -x_{k+1})$ as required.\n%     } \n%     Recall the optimality condition for the proximal switching subgradient subproblem ensures $x_{k+1}$ is the unique solution to\n%     \\begin{equation*}\n%         \\begin{cases}\n%         x_{k+1} = x_k - \\alpha_k(g_0(x_k;\\xi_k)+n_{k+1}) & \\mathrm{if\\ } f_s(x_k) \\leq 0 \\ \\ \\forall s = 1...m \\\\\n%         x_{k+1}=x_k - \\alpha_k g_{s(x_k)}(x_k;\\xi_k) & \\mathrm{for\\ some}\\ f_s(x_k)>0\\ \\mathrm{otherwise} \\\\\n%         n_{k+1} \\in \\partial r(x_{k+1}) \\ . \n%     \\end{cases}\n%     \\end{equation*}\n%     Now we define \n%     \\begin{equation*}\n%         \\bar g_i(y_i;\\xi_i)=\\begin{cases}\n%         g_0(y_i;\\xi_i)+n_{i+1} & \\mathrm{if}\\ f_s(y_i)\\leq 0 \\quad \\forall s=1\\dots m,\\\\\n%         &n_{i+1} \\in \\partial r(y_{i+1})\\\\\n%         g_{s(y_i)}(y_i;\\xi_i) & \\mathrm{for\\ some}\\ f_s(y_i)>0\\ \\mathrm{otherwise} \\ .\n%     \\end{cases}\n%     \\end{equation*}\n%    Following Lemma~\\ref{lem:uniqness}, we first consider when $y_k$ is feasible,\n%     \\begin{align*}\n%           y_{k+1} & =\\argmin \\left\\{ M^{(k-1)}(y)+U^{k}(y)+\\frac{\\beta_k}{2}\\|y-y_0\\|_2^2\\right\\}\\\\\n%                   &=\\argmin \\left\\{\\sum_{i=0}^{k-1}\\lambda_i\\left( \\langle \\bar g_i(y_i;\\xi_i), y-y_i \\rangle + \\frac{\\mu}{2}\\|y-y_i\\|_2^2 \\right) \\right.\\\\ & \\left.\n%                    + \\lambda_k \\left(\\langle g_0(y_k;\\xi_k), y-y_k \\rangle +\\frac{\\mu}{2}\\|y-y_k\\|_2^2 + r(y)\\right) + \\frac{\\bar \\beta}{2}\\|y-y_0\\|_2^2 \\right\\},\n%     \\end{align*}\n% where the second equality holds by removing some constant terms in $M^{(k-1)}(y)$ and $U^{k}(y)$.\n%     \\begin{align*}\n%           y_{k+1} & =\\argmin \\left\\{ M^{(k-1)}(y)+U^{k}(y)+\\frac{\\beta_k}{2}\\|y-y_0\\|_2^2\\right\\}\\\\\n%                   &=\\argmin \\left\\{\\sum_{i=0}^{k-1}\\lambda_i\\left(\\langle g_i(y_i;\\xi_i), y-y_i \\rangle + \\frac{\\mu}{2}\\|y-y_i\\|_2^2 \\right)+  \\sum_{i<k \\colon  s(y_i)=0} \\lambda_i\\langle n_{i+1}, y-y_{i+1}\\rangle \\right.\n%                   \\\\ & \\qquad \\qquad \\left.\n%                    + \\lambda_k \\left(\\langle g_0(y_k;\\xi_k), y-y_k \\rangle +\\frac{\\mu}{2}\\|y-y_k\\|_2^2 + r(y)\\right) + \\frac{\\bar \\beta}{2}\\|y-y_0\\|_2^2 \\right\\},\n%     \\end{align*}\n%     where the second equality holds by removing some constant terms in $M^{(k-1)}(y)$ and $U^{(k)}(y)$.\n%    Taking the subgradient of the right hand side and let the it be zero, and combining with $\\bar g_k(y_k;\\xi_k) = g_0(y_k;\\xi_i)+n_{k+1}$, we have\n%    \\begin{equation*}\n%        \\sum_{i=0}^k \\lambda_i\\left(\\bar g_i(y_i;\\xi_i) + \\mu(y_{k+1}-y_i) \\right) + \\beta(y_{k+1}-y_0) = 0,\n%    \\end{equation*}\n%    which remains same when $y_k$ is infeasible since $\\bar g_k(y_k;\\xi_k)=g_{s(y_k)}(y_k;\\xi_k)$.  \n%    By rearranging the equation, we have\n%    \\begin{equation}\\label{eq:dual-iterates-formula}\n%        y_{k+1}-y_0 = \\frac{\\sum_{i=0}^k\\lambda_i\\mu(y_i-y_0)-\\sum_{i=0}^k\\lambda_i\\bar g_i(y_i;\\xi_i)}{\\sum_{i=0}^k\\lambda_i\\mu+\\bar\\beta}\n%    \\end{equation}\n%  We prove the sequence of iterates $y_k$ given by the dual method is equivalent to the sequence of iterates $x_k$ given by the primal method by inductively showing the formula\n%  \\begin{equation}\\label{eq:recursive-formula}\n%      y_{k+1}-y_0 = -\\sum_{t=0}^k \\frac{\\lambda_t \\bar g_t(y_t;\\xi_t)}{\\sum_{s=0}^t \\lambda_s\\mu+\\bar\\beta},\n%  \\end{equation}\n%    and it is easy to see that the right hand side is the closed form solution of $x_{k+1}-x_0$.\n% One can readily verify this holds $k=0$. So we inductively assume the formula~\\eqref{eq:recursive-formula} holds for all $y_i-y_0$ with $i \\leq k$ and derive it for $y_{k+1}-y_0$.\n% Combining the assumptions with~\\eqref{eq:dual-iterates-formula}, the claimed result follows as\n% \\begin{align*}\n%     y_{k+1} -y_0 &= -\\frac{\\sum_{i=0}^{k} \\lambda_{i}\\mu\\left( \\sum_{t=0}^{i-1} \\frac{\\lambda_{t} \\bar g_t(y_t;\\xi_t)}{\\sum_{s=0}^{t}\\lambda_{s}\\mu + \\bar \\beta} \\right) + \\sum_{i=0}^{k}\\lambda_i \\bar g_i(y_i;\\xi_i)}{\\sum_{i=0}^{k}\\lambda_i\\mu + \\bar\\beta}\\\\\n%     &= -\\frac{\\sum_{t=0}^{k-1} \\lambda_t \\bar g_t(y_t;\\xi_t)\\frac{\\sum_{i=t+1}^k \\mu\\lambda_i}{\\sum_{i=0}^t \\mu\\lambda_t +\\bar\\beta} + \\sum_{i=0}^{k}\\lambda_i \\bar g_i(y_i;\\xi_i)}{\\sum_{i=0}^{k}\\lambda_i\\mu+\\bar\\beta}\\\\\n%     &= -\\frac{\\sum_{t=0}^{k-1} \\lambda_t \\bar g_t(y_t;\\xi_t)\\left(\\frac{\\sum_{i=t+1}^k \\mu\\lambda_i}{\\sum_{i=0}^t \\mu\\lambda_t +\\bar\\beta}+1\\right)}{\\sum_{i=0}^{k}\\lambda_i\\mu+\\bar\\beta} - \\frac{\\lambda_k \\bar g_k(y_k;\\xi_k)}{\\sum_{i=0}^{k}\\lambda_i\\mu + \\bar\\beta}\\\\\n%     &=  -\\sum_{t=0}^{k-1} \\frac{\\lambda_t \\bar g_t(y_t;\\xi_t)}{\\sum_{i=0}^{t}\\lambda_t\\mu+\\bar\\beta} - \\frac{\\lambda_k \\bar g_k(y_k;\\xi_k)}{\\sum_{i=0}^{k}\\lambda_i\\mu + \\bar\\beta} \n% \\end{align*}\n% where the first equality uses the inductive hypothesis for each $y_i-y_0$, the second exchanges summands, and the third and fourth combine and simplify terms.\n% Which tells us $y_k=x_k$ for each $k$, hence the primal method~\\eqref{eq:super-primal-method}and dual method~\\eqref{eq:super-dual-method}are equivalent. \n\\end{proof}","sha256_utf8":"9c477eec0db3705ec664f8c9c93b5dfcf0f53fb44d2ceb66a1acfda5a6bd81ed"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"建立联合归纳命题","source_units":[{"start":45038,"end":45982,"start_line":502,"end_line":503,"raw_tex":"        We prove this by inductively showing that $x_k=y_k$ and for any iteration with $s(x_k)=0$ that $n_{k+1} = \\frac{1}{\\alpha_k}(x_k-\\alpha_k g_{0}(x_k;\\xi_k) -x_{k+1})$ (i.e., the subgradients of $r$  produced by the dual method are exactly those produced by the primal method via~\\eqref{eq:proximal-equivalence}).\n    By assumption, $x_0=y_0$. Suppose for induction that $x_i=y_i$ for all $i=0,\\dots,k$ and for all $i=0,\\dots,k-1$ with $s(x_i)=0$ that $n_{i+1} = \\frac{1}{\\alpha_i}(x_i-\\alpha_i g_{0}(x_i;\\xi_i) -x_{i+1})$. Let $\\bar g_i = g_0(y_i;\\xi_i) + n_{i+1}$ if $y_i$ is feasible and $g_{s(y_i)}(y_i;\\xi_i)$ otherwise. For each $i<k$, we claim $x_{i+1} =  x_i - \\alpha_i \\bar g_i$. If $x_i$ is infeasible, this is immediate. If $x_i$ is feasible, {\\color{blue} rearrangement of our inductive hypothesis of on $n_{i+1}$ gives the claim}. Inductively, we conclude for $i<k$ that $ x_{i+1} - x_0 = - \\sum_{t=0}^i \\alpha_t \\bar g_t. $\n","sha256_utf8":"bdde0b2598ab308c698a641126c47e89b90d207e1faeecf7f106f9782ce090ba"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"对偶最小化的一阶条件","source_units":[{"start":45987,"end":46164,"start_line":505,"end_line":506,"raw_tex":"    By Lemma~\\ref{lem:Mk-minimization}, $y_{k+1}$ is the unique solution to\n    $\\sum_{i=0}^k \\lambda_i\\left(\\bar g_i + \\mu(y_{k+1}-y_i) \\right) + \\bar\\beta(y_{k+1}-y_0) = 0. $\n","sha256_utf8":"aff67672a661d960ac83a58b025fd7a4c3ad87d84e12fcaecd272d76e56304ed"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"交换求和得到原始更新","source_units":[{"start":46164,"end":47309,"start_line":507,"end_line":515,"raw_tex":"    Rearranging and simplifying this, it follows that\n    \\begin{align*}\n    y_{k+1} &= y_0 + \\frac{\\sum_{i=0}^k\\lambda_i\\mu(y_i-y_0)-\\sum_{i=0}^k\\lambda_i\\bar g_i}{\\sum_{i=0}^k\\lambda_i\\mu+\\bar\\beta}\\\\\n    &= y_0 -\\frac{\\sum_{i=0}^{k} \\lambda_{i}\\mu\\left( \\sum_{t=0}^{i-1} \\frac{\\lambda_{t} \\bar g_t}{\\sum_{s=0}^{t}\\lambda_{s}\\mu + \\bar \\beta} \\right) + \\sum_{i=0}^{k}\\lambda_i \\bar g_i}{\\sum_{i=0}^{k}\\lambda_i\\mu + \\bar\\beta}\\\\\n    &= y_0 -\\frac{\\sum_{t=0}^{k-1} \\lambda_t \\bar g_t\\frac{\\sum_{i=t+1}^k \\mu\\lambda_i}{\\sum_{i=0}^t \\mu\\lambda_{{\\color{blue} i}} +\\bar\\beta} + \\sum_{i=0}^{k}\\lambda_i \\bar g_i}{\\sum_{i=0}^{k}\\lambda_i\\mu+\\bar\\beta}\\\\\n    &= y_0 -\\frac{\\sum_{t=0}^{k-1} \\lambda_t \\bar g_t\\left(\\frac{\\sum_{i=t+1}^k \\mu\\lambda_i}{\\sum_{i=0}^t \\mu\\lambda_{{\\color{blue} i}} +\\bar\\beta}+1\\right)}{\\sum_{i=0}^{k}\\lambda_i\\mu+\\bar\\beta} - \\frac{\\lambda_k \\bar g_k}{\\sum_{i=0}^{k}\\lambda_i\\mu + \\bar\\beta}\\\\\n    &=  y_0 -\\sum_{t=0}^{k-1} \\frac{\\lambda_t \\bar g_t}{\\sum_{i=0}^{t}\\lambda_{{\\color{blue} i}}\\mu+\\bar\\beta} - \\frac{\\lambda_k \\bar g_k}{\\sum_{i=0}^{k}\\lambda_i\\mu + \\bar\\beta}\\\\\n    &= x_k - \\alpha_k\\bar g_k\n    \\end{align*}\n","sha256_utf8":"6f3b08f270f9cb29857ebe6516f3a92bccd72a733b3c9b2c9c8dda54719ad3ce"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"分支验证并闭合归纳","source_units":[{"start":47309,"end":48092,"start_line":516,"end_line":518,"raw_tex":"    where the second equality uses the inductive hypothesis for each $y_i-y_0 = x_i-x_0 = - \\sum_{t=0}^{i-1} \\alpha_t\\bar g_t$, the third exchanges summands, and the remainder combines and simplifies terms. If $x_k$ is infeasible, {\\color{blue} it is} immediate that $x_{k+1}=x_k-\\alpha_k\\bar g_k=y_{k+1}$. If $x_k$ is feasible, the above equality ensures\n    $$ \\frac{1}{\\alpha_k}(x_k - \\alpha_k g_0(x_k;\\xi_k) -y_{k+1}) = n_{k+1} \\in\\partial r(y_{k+1}) \\ . $$\n    Noting by~\\eqref{eq:proximal-equivalence}, $x_{k+1}$ is the unique solution to $\\frac{1}{\\alpha_k}(x_k-\\alpha_kg_0(x_k;\\xi_k) -z)\\in\\partial r(z)$, we must have $x_{k+1} {\\color{blue} = x_k - \\alpha_k(g_k + n_{k+1})} = y_{k+1}$ and $n_{k+1} =  \\frac{1}{\\alpha_k}(x_k - \\alpha_k g_0(x_k;\\xi_k) -x_{k+1})$ as required.\n","sha256_utf8":"5ed7118670ca4551668e54ccb5e063e80ce185e7d8e4efc33f068ee603926740"}]},{"id":"S044","semantic_type":"lemma","kind":"lemma","title":"更新聚合模型的最小点","source_units":[{"start":25495,"end":25633,"start_line":356,"end_line":358,"raw_tex":"\\begin{lemma}\\label{lem:Mk-minimization}\n    $y_{k+1}$ is the unique minimizer of $M^{(k)}(y)+\\frac{\\beta_k}{2}\\|y-y_0\\|_2^2$.\n\\end{lemma}","sha256_utf8":"45cd133fdd2935beff93321f67d650300459a1f80d298c853692fc925aef678d"}]},{"id":"S045","semantic_type":"proof","kind":"proof","title":"聚合模型最小化完整证明","source_units":[{"start":25634,"end":26755,"start_line":359,"end_line":371,"raw_tex":"\\begin{proof}\n% {\\color{blue}\nFirst note that when $y_k$ is infeasible (and so $s(y_k)\\neq 0$), $M^{(k)} = M^{(k-1)} + U^{(k)}$. From this, the result is immediate.\nNow consider when $y_k$ is feasible (and so $s(y_k)=0$). Since $y_{k+1}$ is the unique minimizer of $M^{(k-1)}(y)+U^{(k)}(y)+\\frac{\\beta_k}{2}\\|y-y_0\\|_2^2$, the necessary and sufficient optimality condition ensures $y_{k+1}$ is the unique solution to\n$$ 0 \\in \\nabla M^{(k-1)}(y_{k+1}) + \\lambda_k(g_0(y_k; \\xi_k) + \\mu(y_{k+1}-y_k) + \\partial r(y_{k+1})) + \\beta_k(y_{k+1}-y_0) \\ . $$\nRewriting this as $ \\nabla M^{(k-1)}(y_{k+1}) + \\lambda_k (g_0(y_k; \\xi_k)+\\mu(y_{k+1}-y_k)) + \\beta_k(y_{k+1}-y_0) \\in -\\lambda_k \\partial r(y_{k+1})$, we see that $n_{k+1}$ is the element of $\\partial r(y_{k+1})$ certifying the minimization's optimality. Therefore\n\\begin{align*}\n    0 &= \\nabla M^{(k-1)}(y_{k+1}) + \\lambda_k(g_0(y_k; \\xi_k) + \\mu(y_{k+1}-y_k)+n_{k+1}) + \\beta_k(y_{k+1}-y_0)\\\\\n    &= \\nabla M^{(k)}(y_{k+1}) + \\beta_k(y_{k+1}-y_0)\n\\end{align*}\nand so $y_{k+1}$ is a the unique minimizer of $M^{(k)}(y)+\\frac{\\beta_k}{2}\\|y-y_0\\|_2^2$. \n\n\\end{proof}","sha256_utf8":"e391eb312d7a1c554657a50bb24c5e898798bf6b7e8f799b3d640033e19528f9"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"不可行分支模型直接相加","source_units":[{"start":25664,"end":25799,"start_line":361,"end_line":361,"raw_tex":"First note that when $y_k$ is infeasible (and so $s(y_k)\\neq 0$), $M^{(k)} = M^{(k-1)} + U^{(k)}$. From this, the result is immediate.\n","sha256_utf8":"0270ec4684eb08250e38f82e6a949ba3e5394ef924baf49c57b9e232f1facdcc"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"可行分支确定正则项次梯度","source_units":[{"start":25799,"end":26453,"start_line":362,"end_line":364,"raw_tex":"Now consider when $y_k$ is feasible (and so $s(y_k)=0$). Since $y_{k+1}$ is the unique minimizer of $M^{(k-1)}(y)+U^{(k)}(y)+\\frac{\\beta_k}{2}\\|y-y_0\\|_2^2$, the necessary and sufficient optimality condition ensures $y_{k+1}$ is the unique solution to\n$$ 0 \\in \\nabla M^{(k-1)}(y_{k+1}) + \\lambda_k(g_0(y_k; \\xi_k) + \\mu(y_{k+1}-y_k) + \\partial r(y_{k+1})) + \\beta_k(y_{k+1}-y_0) \\ . $$\nRewriting this as $ \\nabla M^{(k-1)}(y_{k+1}) + \\lambda_k (g_0(y_k; \\xi_k)+\\mu(y_{k+1}-y_k)) + \\beta_k(y_{k+1}-y_0) \\in -\\lambda_k \\partial r(y_{k+1})$, we see that $n_{k+1}$ is the element of $\\partial r(y_{k+1})$ certifying the minimization's optimality. Therefore\n","sha256_utf8":"ef1fba284b9e14408cd14f2e8a34ab58dd33d82e7d5e92a580d9a77ad0ad9285"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"验证更新模型梯度为零","source_units":[{"start":26453,"end":26743,"start_line":365,"end_line":369,"raw_tex":"\\begin{align*}\n    0 &= \\nabla M^{(k-1)}(y_{k+1}) + \\lambda_k(g_0(y_k; \\xi_k) + \\mu(y_{k+1}-y_k)+n_{k+1}) + \\beta_k(y_{k+1}-y_0)\\\\\n    &= \\nabla M^{(k)}(y_{k+1}) + \\beta_k(y_{k+1}-y_0)\n\\end{align*}\nand so $y_{k+1}$ is a the unique minimizer of $M^{(k)}(y)+\\frac{\\beta_k}{2}\\|y-y_0\\|_2^2$. \n","sha256_utf8":"3858037b88827c1b4f083045becae1c7f0563a79d93f0c84724febac87455198"}]},{"id":"S049","semantic_type":"lemma","kind":"lemma","title":"二次函数标准形式","source_units":[{"start":27230,"end":27461,"start_line":375,"end_line":377,"raw_tex":"\\begin{lemma}\\label{lem:standard-quadratic}\n    Any quadratic function of the form $Q(z) = c + \\langle d,z\\rangle + \\frac{b}{2}\\|z-\\hat z\\|^2_2$ with $b>0$ is equal to $Q(z) = \\min Q + \\frac{b}{2}\\|z - \\argmin Q\\|^2_2$.\n\\end{lemma}","sha256_utf8":"94e87a9d279c7ef2b78321320d813bce7acb85f8229b96503dbf316d9807ddda"}]},{"id":"S050","semantic_type":"lemma","kind":"lemma","title":"两个二次函数合并公式","source_units":[{"start":27464,"end":27845,"start_line":380,"end_line":384,"raw_tex":"\\begin{lemma}\\label{lem:quad-calc}\n    The sum of quadratics $Q_i(z) = a_i + \\frac{b_i}{2}\\|z-z_i\\|_2^2$ for $i=1,2$ with $b_i > 0$ equals\n    $$ (Q_1+Q_2)(z) = a_{1+2} + \\frac{b_{1+2}}{2}\\|z-z_{1+2}\\|_2^2$$\n    where $a_{1+2} = a_1 + a_2 + \\frac{b_1b_2}{2(b_1+b_2)}\\|z_1-z_2\\|_2^2$, $b_{1+2} = b_1+b_2$, and $z_{1+2} = \\frac{b_1}{b_1+b_2}z_1 + \\frac{b_2}{b_1+b_2}z_2$.\n\\end{lemma}","sha256_utf8":"8a8870c7a9aca263897eb9bcef04a69fad1c8246254d85367c3544b38f321dd4"}]},{"id":"S051","semantic_type":"external_fact","kind":"external_fact","title":"标准二次配方恒等式","source_units":[],"declared_external":true,"declaration_text":["For Q(z)=c+<d,z>+(b/2)||z-zhat||^2 with b>0, completing the square gives Q(z)=min Q+(b/2)||z-argmin Q||^2."]},{"id":"S052","semantic_type":"external_fact","kind":"external_fact","title":"二次函数求和恒等式","source_units":[],"declared_external":true,"declaration_text":["For Q_i(z)=a_i+(b_i/2)||z-z_i||^2 with b_i>0, their sum has curvature b_1+b_2, center (b_1 z_1+b_2 z_2)/(b_1+b_2), and minimum a_1+a_2+b_1 b_2||z_1-z_2||^2/(2(b_1+b_2))."]},{"id":"S053","semantic_type":"external_fact","kind":"external_fact","title":"近端算子非扩张性","source_units":[],"declared_external":true,"declaration_text":["For proper closed convex r and alpha>0, ||prox_{alpha,r}(u)-prox_{alpha,r}(v)||_2 <= ||u-v||_2; cited as Rockafellar and Wets, Proposition 12.19."]},{"id":"S054","semantic_type":"external_fact","kind":"external_fact","title":"凸函数最优性与唯一性","source_units":[],"declared_external":true,"declaration_text":["A point minimizes a proper convex function exactly when its subdifferential contains zero. If the function is strongly convex, the minimizer is unique."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S015","edges":[{"source_id":"S001","target_id":"S004","kind":"supports"},{"source_id":"S001","target_id":"S009","kind":"supports"},{"source_id":"S002","target_id":"S025","kind":"supports"},{"source_id":"S002","target_id":"S027","kind":"supports"},{"source_id":"S003","target_id":"S025","kind":"supports"},{"source_id":"S003","target_id":"S043","kind":"supports"},{"source_id":"S004","target_id":"S025","kind":"supports"},{"source_id":"S004","target_id":"S027","kind":"supports"},{"source_id":"S004","target_id":"S021","kind":"supports"},{"source_id":"S005","target_id":"S025","kind":"supports"},{"source_id":"S005","target_id":"S027","kind":"supports"},{"source_id":"S005","target_id":"S040","kind":"supports"},{"source_id":"S005","target_id":"S043","kind":"supports"},{"source_id":"S006","target_id":"S046","kind":"supports"},{"source_id":"S006","target_id":"S047","kind":"supports"},{"source_id":"S006","target_id":"S048","kind":"supports"},{"source_id":"S006","target_id":"S030","kind":"supports"},{"source_id":"S006","target_id":"S034","kind":"supports"},{"source_id":"S007","target_id":"S046","kind":"supports"},{"source_id":"S007","target_id":"S047","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S025","kind":"supports"},{"source_id":"S008","target_id":"S027","kind":"supports"},{"source_id":"S008","target_id":"S047","kind":"supports"},{"source_id":"S008","target_id":"S048","kind":"supports"},{"source_id":"S009","target_id":"S025","kind":"supports"},{"source_id":"S009","target_id":"S032","kind":"supports"},{"source_id":"S009","target_id":"S021","kind":"supports"},{"source_id":"S009","target_id":"S022","kind":"supports"},{"source_id":"S010","target_id":"S025","kind":"supports"},{"source_id":"S010","target_id":"S027","kind":"supports"},{"source_id":"S010","target_id":"S032","kind":"supports"},{"source_id":"S010","target_id":"S036","kind":"supports"},{"source_id":"S010","target_id":"S020","kind":"supports"},{"source_id":"S010","target_id":"S021","kind":"supports"},{"source_id":"S010","target_id":"S022","kind":"supports"},{"source_id":"S011","target_id":"S026","kind":"supports"},{"source_id":"S011","target_id":"S027","kind":"supports"},{"source_id":"S011","target_id":"S033","kind":"supports"},{"source_id":"S011","target_id":"S036","kind":"supports"},{"source_id":"S012","target_id":"S020","kind":"supports"},{"source_id":"S013","target_id":"S022","kind":"supports"},{"source_id":"S014","target_id":"S018","kind":"supports"},{"source_id":"S014","target_id":"S019","kind":"supports"},{"source_id":"S014","target_id":"S025","kind":"supports"},{"source_id":"S014","target_id":"S027","kind":"supports"},{"source_id":"S014","target_id":"S033","kind":"supports"},{"source_id":"S014","target_id":"S036","kind":"supports"},{"source_id":"S014","target_id":"S020","kind":"supports"},{"source_id":"S014","target_id":"S022","kind":"supports"},{"source_id":"S016","target_id":"S018","kind":"supports"},{"source_id":"S016","target_id":"S019","kind":"supports"},{"source_id":"S016","target_id":"S030","kind":"supports"},{"source_id":"S016","target_id":"S032","kind":"supports"},{"source_id":"S023","target_id":"S018","kind":"supports"},{"source_id":"S028","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S017","kind":"supports"},{"source_id":"S020","target_id":"S017","kind":"supports"},{"source_id":"S021","target_id":"S017","kind":"supports"},{"source_id":"S022","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S015","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S027","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S032","target_id":"S029","kind":"supports"},{"source_id":"S033","target_id":"S029","kind":"supports"},{"source_id":"S034","target_id":"S029","kind":"supports"},{"source_id":"S035","target_id":"S029","kind":"supports"},{"source_id":"S036","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S037","target_id":"S030","kind":"supports"},{"source_id":"S037","target_id":"S032","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S038","target_id":"S041","kind":"supports"},{"source_id":"S038","target_id":"S042","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S041","target_id":"S039","kind":"supports"},{"source_id":"S042","target_id":"S039","kind":"supports"},{"source_id":"S043","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S037","kind":"supports"},{"source_id":"S044","target_id":"S041","kind":"supports"},{"source_id":"S044","target_id":"S030","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S048","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S049","target_id":"S030","kind":"supports"},{"source_id":"S049","target_id":"S034","kind":"supports"},{"source_id":"S050","target_id":"S031","kind":"supports"},{"source_id":"S050","target_id":"S035","kind":"supports"},{"source_id":"S051","target_id":"S049","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S025","kind":"supports"},{"source_id":"S054","target_id":"S047","kind":"supports"},{"source_id":"S054","target_id":"S048","kind":"supports"}]}}
{"record_id":"2308.05887v2","paper":{"title":"A search-free $O(1/k^{3/2})$ homotopy inexact proximal-Newton extragradient algorithm for monotone variational inequalities","authors":["M. Marques Alves","João M. Pereira","Benar F. Svaiter"],"arxiv_id":"2308.05887","version":"v2","journal":"SIAM Journal on Optimization","doi":"10.1137/23m1593000","abstract_url":"https://arxiv.org/abs/2308.05887v2","pdf_url":"https://arxiv.org/pdf/2308.05887v2"},"context":{"raw_tex":"Let $\\HH$ be a real Hilbert space with inner product $\\inner{\\cdot}{\\cdot}$ and induced norm \n$\\norm{\\cdot} = \\sqrt{\\inner{\\cdot}{\\cdot}}$.\n%\nWe use the same notation $\\norm{A}$ to denote the norm of a bounded linear operator $A$.\n%\nThe \\emph{normal cone} to $C\\subset \\HH$ at $x\\in C$ is defined as\n%\n%\\[\n$N_C(x) = \\left\\{\\nu\\in \\HH\\;\\;|\\;\\;\\inner{\\nu}{y-x}\\leq 0\\quad \\forall y\\in C\\right\\}.$\n%\\]\n%\nWe also set $N_C(x) := \\emptyset$ if $x\\notin C$. The $\\varepsilon$-enlargement of the set-valued map $T:\\HH\\tos \\HH$ (see, e.g., \\cite{bur.ius.sva-enl.svva97}) is the (set-valued map) $T^\\varepsilon:\\HH\\tos \\HH$ defined as\n%\n\\begin{align}\\lab{eq:def.enl}\n%\nT^{\\varepsilon}(x) = \\left\\{v\\in \\HH\\;\\;|\\;\\;\\inner{v - u}{x - z}\\geq -\\varepsilon\\quad \\forall u\\in T(z)\\right\\}. \n%\n\\end{align}\n%\nWe also use the notation \n%\n\\begin{align}\\lab{eq:def.lin}\n%\nF_y(x) := F(y) + F'(y)(x-y)\\qquad \\forall x\\in C\n%\n\\end{align}\n%\nto indicate the linearization of a smooth map $F:C\\subset \\HH\\to \\HH$ at $y\\in C$.\n%\nThe set of natural numbers is $\\N = \\{1,2,\\dots\\}$ and $\\# S$ stands for the number of elements of\na set $S$. We also define $\\log^+(t) = \\max\\{\\log(t), 0\\}$ for $t>0$.\n\n\n\\begin{algorithm}[H]\n\\lab{alg:rHPE}\n \\caption{A special instance of the large-step under-relaxed HPE method for \\eqref{eq:probim}}\n%\n\\lab{alg_gen}\n%\n\\SetAlgoLined\n\\KwInput{$x_0\\in \\HH$, $0<\\tau<1$, $0\\leq \\sigma <1$, $\\eta>0$ and $N\\in \\N\\cup \\{\\infty\\}$}\n \\For{$k = 1,2,\\dots, N$}{\n  Choose $\\lambda_k>0$ and find $y_k\\in \\HH$ and $\\nu_k\\in N_C(y_k)$ such that\\\\\n  %\n  \\begin{align}\n  %\n  \\lab{eq:rel.al1}\n   &\\left\\|\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}\\right\\|\\leq \\sigma\\norm{y_k - x_{k-1}}\\\\[2mm]\n   \\lab{eq:lar.ste}\n   &\\lambda_k\\norm{y_k - x_{k-1}}\\geq \\eta\n  %\n  \\end{align}\n  %\n  Set\n  %\n  \\begin{align}\\lab{eq:ext.al1}\n  %\n   x_k = x_{k-1} - \\tau\\lambda_k \\left(F(y_k) + \\nu_k\\right)\n  %\n  \\end{align}\n  }\n\\end{algorithm}\n\nThe \\emph{ergodic sequences} associated to the sequences generated by Algorithm \\ref{alg:rHPE} are defined as \nfollows (see, e.g., \\cite{mon.sva-hpe.siam10}):\n\n\\begin{align}\\lab{eq:def.erg1}\n%\n& y_k^a = \\dfrac{1}{\\sum_{i=1}^k\\,\\lambda_i}\\,\\sum_{i=1}^k\\,\\lambda_i  y_i,\\\\[2mm]\n%\n\\lab{eq:def.erg2}\n& v_k^a = \\dfrac{1}{\\sum_{i=1}^k\\,\\lambda_i}\\,\\sum_{i=1}^k\\,\\lambda_i  \\left(F(y_i) + \\nu_i\\right),\\\\[2mm]\n%\n\\lab{eq:def.erg3}\n& \\varepsilon_k^a = \\dfrac{1}{\\sum_{i=1}^k\\,\\lambda_i}\\,\\sum_{i=1}^k\\,\\lambda_i \\inner{y_i - y_k^a}{F(y_i) + \\nu_i- v_k^a}.\n%\n\\end{align}\n\n\nConsider the (monotone) VIP given in \\eqref{eq:probi}, i.e.,\n%\n\\begin{align} \\lab{eq:prob}\n%\n\\mbox{Find}\\;\\;x\\in C\\;\\;\\mbox{such that}\\;\\; \\inner{F(x)}{y-x}\\geq 0\\qquad \\forall y\\in C,\n%\n\\end{align}\n%\nwhere $C$ is a (nonempty) closed and convex subset of $\\HH$ and \n$F:C\\to \\HH$ is assumed to be (point-to-point) \\emph{monotone}\nand continuously differentiable with a $L$-Lipschitz continuous derivative, i.e., $F$ and $F'$ satisfy the following conditions for all\n$x,y\\in C$:\n%\n\\begin{align}\\lab{eq:mon.lip}\n\\begin{aligned}\n%\n& \\inner{F(x) - F(y)}{x - y}\\geq 0,\\\\[2mm]\n& \\norm{F'(x) - F'(y)}\\leq L\\norm{x-y}.\n %\n\\end{aligned}\n\\end{align}\n\nRecall that \\eqref{eq:prob} is equivalent to \\eqref{eq:probim}, i.e.,\n%\n\\begin{align}\\lab{eq:probm}\n%\n 0 \\in F(x) + N_C(x).\n%\n\\end{align}\n% \nWe assume throughout this paper that the solution set $(F+N_C)^{-1}(0)$ of \\eqref{eq:probm} \n(or, equivalently, of \\eqref{eq:prob}) is nonempty.\n\n\nOur main algorithm \n(Algorithm \\ref{alg:main} below) will be defined by solving, at each iteration, a similar inclusion:\n%\n\\begin{align}\\lab{eq:farias}\n%\n0\\in \\lambda_k \\big(F_{y_{k-1}}(x) + N_C(x)\\big) + x - x_{k-1},\n%\n\\end{align}\n\n\n\\bdefi[$\\hat\\sigma$-approximate solution of \\eqref{eq:farias}]\n \\lab{def:app_sol}\n%\nFor $y_{k-1}\\in C$, $x_{k-1}\\in \\HH$ and $\\lambda_k>0$, a pair $(y_k,\\nu_k)\\in \\HH\\times \\HH$ is said to be a \n$\\hat\\sigma$-approximate solution of \\eqref{eq:farias} \nif $\\hat\\sigma\\in [0,1/2)$ and\n%\n\\begin{align}\\lab{eq:app_sol}\n\\begin{aligned}\n%\n\\begin{cases}\n%\n\\nu_k\\in N_C(y_k),\\\\[3mm]\n%\n\\norm{\\lambda_k\\left(F_{y_{k-1}}(y_k) + \\nu_k\\right) + y_k - x_{k-1}} \\leq \\hat\\sigma\\norm{y_k - y_{k-1}}.\n%\n\\end{cases}\n%\n\\end{aligned}\n\\end{align}\n%\n\\edefi\n\nLet \n\\begin{align}\n  \\lab{eq:hsigma}\n  0 \\leq \\hat\\sigma <1/2\n\\end{align}\nbe the parameter of approximate solution of \\eqref{eq:farias}, as\nin Definition~\\ref{def:app_sol}.\n%\nThis $\\hat\\sigma$ can be regarded as the inexactness of an (inexact) oracle which\nsolves approximately the strongly monotone linear VIP~\\eqref{eq:farias}.\n\n\nBefore stating the main algorithm, \nchoose and define the parameters $\\theta$, $\\hat\\theta$, $\\eta$, $\\tau$\nas follows:\n% consider\n\\begin{align}\\lab{eq:def.hth}\n  0 < \\theta<(1-\\hat\\sigma)(1-2\\hat\\sigma),\n  \\quad \n  \\hat\\theta & := \\theta\\left(\\dfrac{\\hat\\sigma}{1-\\hat\\sigma}\n+ \\dfrac{\\theta}{(1-\\hat\\sigma)^2}\\right)\n\\end{align}\nand \n\\begin{align}\\lab{eq:def.tau.alg}\n%\n    \\eta & > \\dfrac{2\\hat\\theta}{L},\n  \\quad \\tau  := \\dfrac{2 \\left(\\theta - \\hat\\theta\\right)}\n{2\\theta + \\dfrac{\\eta L}{2} + \\sqrt{\\left(2\\theta + \\dfrac{\\eta L}{2}\\right)^2 \n- 4\\theta\\left(\\theta - \\hat\\theta\\right)}}.\n%\n\\end{align}\n\n\n\\begin{algorithm}[H]\n%\n \\caption{A relative-error inexact proximal-Newton algorithm for solving \\eqref{eq:probm}}\n%\n\\lab{alg:main}\n%\n\\SetAlgoLined\n\\KwInput{$x_0\\in C$,$\\;y_0:=x_0$, $\\nu_0 := 0$, the scalars \n $\\hat\\sigma$, $\\theta$, $\\;\\hat\\theta$ $\\eta$, $\\tau$\n as in \\eqref{eq:hsigma}--\\eqref{eq:def.tau.alg}\nand $\\lambda_1>0$ such that $\\lambda_1^2\\norm{F(y_0)}\\leq 2\\theta/L$\n%\n}\n \\For{$k=1,2,\\dots$}{\n %\n \\If{$F(y_{k-1})+\\nu_{k-1}=0$}\n \t{\\Return $y_{k-1}$\\\\\\lab{eq:return}}\n%\n \\eIf{$\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_{k-1}) + \\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}\\leq \n  \\hat\\theta$}\n  {$y_k = y_{k-1}$\\\\ \\lab{eq:n.cond}\n   $\\nu_k =\\nu_{k-1}$ \\\\ \\lab{eq:n.cond02}}\n  {Find a $\\hat\\sigma$-approximate solution $(y_k,\\nu_k)$ of \\eqref{eq:farias} (in the sense of\n   Definition \\ref{def:app_sol}), i.e., find $(y_k,\\nu_k)$ satisfying\n   %\n   \\begin{align}\\lab{eq:err.cri}\n    %\n    \\begin{cases}\n    %\n   \\nu_k\\in N_C(y_k),\\\\[3mm]\n   %\n   \\norm{\\lambda_k\\left(F_{y_{k-1}}(y_k) + \\nu_k\\right) + y_k - x_{k-1}} \\leq \\hat\\sigma\\norm{y_k - y_{k-1}}\n   %\n   \\end{cases}\n  %\n  \\end{align} \\lab{eq:do.newton}}\n  %\n %  \n   \\eIf{$\\lambda_k\\norm{y_k - x_{k-1}}\\geq \\eta$}\n      {\n      \\vspace{0.2cm}\n      $x_{k} = x_{k-1} -\\tau\\lambda_k \\left(F(y_k) + \\nu_k\\right)$\\\\[2mm]\\lab{eq:ext.step}\n      \\vspace{0.1cm}\n      $\\lambda_{k+1} = (1 - \\tau)\\lambda_k$\\\\[3mm]\\lab{eq:red.lam}\n      }\n     {$x_k = x_{k-1}$\\\\[2mm]\\lab{eq:null.step}\n     $\\lambda_{k+1} = \\dfrac{\\lambda_k}{1 - \\tau}$\\\\\\lab{eq:inc.lam}\n     }\n  %\n  }\n\\end{algorithm}\n\n\\item[(ii)] We mention that if Algorithm \\ref{alg:main} stops at step \\ref{eq:return}, then it follows that $y_{k-1}$ is a solution of\n\\eqref{eq:probm}. So from now on in this paper, we assume that\n%\n%\n\\[\n%\n\\fbox{\nAlgorithm \\ref{alg:main} doesn't stop at step \\ref{eq:return}.\n}\n%\n\\]\n\n\nTo simplify the presentation, define for $k \\geq 1$\n%\n\\begin{align}\\lab{eq:def.ek}\n%\n  e_k & := \\lambda_k\\left(F_{y_{k-1}}(y_k) + \\nu_k\\right) + y_k - x_{k-1}\\;.\n%\n\\end{align} \n\n\nFor every $j\\geq 1$, define\n%\n\\begin{align}\n  \\lab{eq:def.ab}\n  \\begin{aligned}\n    A_j & =\\left\\{1\\leq k \\leq j\\;|\\;\\; \\text{$\\lambda_k\\norm{y_k - x_{k-1}}\\geq \\eta$\\; holds at iteration $k$}\\right\\},\n\t  & a_j =\\# A_j\\,,\\\\\n    B_j & =\\{1\\leq k\\leq j\\;|\\;\\; \n   \\text{$\\lambda_k\\norm{y_k - x_{k-1}} < \\eta$\\; holds at iteration $k$}\\},\n    & b_j =\\# B_j\\,,\n  \\end{aligned}\n\\end{align}\n%\n\\begin{align}\n \\lab{eq:uni.ab}\n A =\\bigcup_{j=1}^\\infty A_j\\;\\;\\mbox{and}\\;\\; B =\\bigcup_{j=1}^\\infty B_j.\n\\end{align}\n%\nWe also define\n%\n\\begin{align}\n\\lab{eq:s70}\n  K=\\set{k\\geq 1\\;|\\; k\\leq \\# A},\\quad\n  i_0=0,\\quad i_k=\\text{$k$-th element of }A,\n\\end{align}\n%\nand note that \n%\n\\begin{align}\n \\lab{eq:car.a}\n  i_0<i_1<i_2<\\cdots,\\qquad A=\\set{i_k\\;|\\; k\\in K}.\n\\end{align}\n","units":[{"start":27874,"end":29044,"start_line":547,"end_line":577,"raw_tex":"Let $\\HH$ be a real Hilbert space with inner product $\\inner{\\cdot}{\\cdot}$ and induced norm \n$\\norm{\\cdot} = \\sqrt{\\inner{\\cdot}{\\cdot}}$.\n%\nWe use the same notation $\\norm{A}$ to denote the norm of a bounded linear operator $A$.\n%\nThe \\emph{normal cone} to $C\\subset \\HH$ at $x\\in C$ is defined as\n%\n%\\[\n$N_C(x) = \\left\\{\\nu\\in \\HH\\;\\;|\\;\\;\\inner{\\nu}{y-x}\\leq 0\\quad \\forall y\\in C\\right\\}.$\n%\\]\n%\nWe also set $N_C(x) := \\emptyset$ if $x\\notin C$. The $\\varepsilon$-enlargement of the set-valued map $T:\\HH\\tos \\HH$ (see, e.g., \\cite{bur.ius.sva-enl.svva97}) is the (set-valued map) $T^\\varepsilon:\\HH\\tos \\HH$ defined as\n%\n\\begin{align}\\lab{eq:def.enl}\n%\nT^{\\varepsilon}(x) = \\left\\{v\\in \\HH\\;\\;|\\;\\;\\inner{v - u}{x - z}\\geq -\\varepsilon\\quad \\forall u\\in T(z)\\right\\}. \n%\n\\end{align}\n%\nWe also use the notation \n%\n\\begin{align}\\lab{eq:def.lin}\n%\nF_y(x) := F(y) + F'(y)(x-y)\\qquad \\forall x\\in C\n%\n\\end{align}\n%\nto indicate the linearization of a smooth map $F:C\\subset \\HH\\to \\HH$ at $y\\in C$.\n%\nThe set of natural numbers is $\\N = \\{1,2,\\dots\\}$ and $\\# S$ stands for the number of elements of\na set $S$. We also define $\\log^+(t) = \\max\\{\\log(t), 0\\}$ for $t>0$.\n","sha256_utf8":"60eb24e1f243b0d242012b2ee2df7b844eb69482036fb677ce4a8085eb600bca"},{"start":29521,"end":30268,"start_line":592,"end_line":620,"raw_tex":"\\begin{algorithm}[H]\n\\lab{alg:rHPE}\n \\caption{A special instance of the large-step under-relaxed HPE method for \\eqref{eq:probim}}\n%\n\\lab{alg_gen}\n%\n\\SetAlgoLined\n\\KwInput{$x_0\\in \\HH$, $0<\\tau<1$, $0\\leq \\sigma <1$, $\\eta>0$ and $N\\in \\N\\cup \\{\\infty\\}$}\n \\For{$k = 1,2,\\dots, N$}{\n  Choose $\\lambda_k>0$ and find $y_k\\in \\HH$ and $\\nu_k\\in N_C(y_k)$ such that\\\\\n  %\n  \\begin{align}\n  %\n  \\lab{eq:rel.al1}\n   &\\left\\|\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}\\right\\|\\leq \\sigma\\norm{y_k - x_{k-1}}\\\\[2mm]\n   \\lab{eq:lar.ste}\n   &\\lambda_k\\norm{y_k - x_{k-1}}\\geq \\eta\n  %\n  \\end{align}\n  %\n  Set\n  %\n  \\begin{align}\\lab{eq:ext.al1}\n  %\n   x_k = x_{k-1} - \\tau\\lambda_k \\left(F(y_k) + \\nu_k\\right)\n  %\n  \\end{align}\n  }\n\\end{algorithm}","sha256_utf8":"f12d61a999c661ce4e225fef985eba2089fa65aa2903a79d9880191d63b68cde"},{"start":33670,"end":34231,"start_line":685,"end_line":698,"raw_tex":"The \\emph{ergodic sequences} associated to the sequences generated by Algorithm \\ref{alg:rHPE} are defined as \nfollows (see, e.g., \\cite{mon.sva-hpe.siam10}):\n\n\\begin{align}\\lab{eq:def.erg1}\n%\n& y_k^a = \\dfrac{1}{\\sum_{i=1}^k\\,\\lambda_i}\\,\\sum_{i=1}^k\\,\\lambda_i  y_i,\\\\[2mm]\n%\n\\lab{eq:def.erg2}\n& v_k^a = \\dfrac{1}{\\sum_{i=1}^k\\,\\lambda_i}\\,\\sum_{i=1}^k\\,\\lambda_i  \\left(F(y_i) + \\nu_i\\right),\\\\[2mm]\n%\n\\lab{eq:def.erg3}\n& \\varepsilon_k^a = \\dfrac{1}{\\sum_{i=1}^k\\,\\lambda_i}\\,\\sum_{i=1}^k\\,\\lambda_i \\inner{y_i - y_k^a}{F(y_i) + \\nu_i- v_k^a}.\n%\n\\end{align}\n","sha256_utf8":"a2bbe2a2d3e8715d79427d04d031bd415b5286a4aa08bd28482d842907367862"},{"start":36257,"end":37183,"start_line":782,"end_line":813,"raw_tex":"Consider the (monotone) VIP given in \\eqref{eq:probi}, i.e.,\n%\n\\begin{align} \\lab{eq:prob}\n%\n\\mbox{Find}\\;\\;x\\in C\\;\\;\\mbox{such that}\\;\\; \\inner{F(x)}{y-x}\\geq 0\\qquad \\forall y\\in C,\n%\n\\end{align}\n%\nwhere $C$ is a (nonempty) closed and convex subset of $\\HH$ and \n$F:C\\to \\HH$ is assumed to be (point-to-point) \\emph{monotone}\nand continuously differentiable with a $L$-Lipschitz continuous derivative, i.e., $F$ and $F'$ satisfy the following conditions for all\n$x,y\\in C$:\n%\n\\begin{align}\\lab{eq:mon.lip}\n\\begin{aligned}\n%\n& \\inner{F(x) - F(y)}{x - y}\\geq 0,\\\\[2mm]\n& \\norm{F'(x) - F'(y)}\\leq L\\norm{x-y}.\n %\n\\end{aligned}\n\\end{align}\n\nRecall that \\eqref{eq:prob} is equivalent to \\eqref{eq:probim}, i.e.,\n%\n\\begin{align}\\lab{eq:probm}\n%\n 0 \\in F(x) + N_C(x).\n%\n\\end{align}\n% \nWe assume throughout this paper that the solution set $(F+N_C)^{-1}(0)$ of \\eqref{eq:probm} \n(or, equivalently, of \\eqref{eq:prob}) is nonempty.\n","sha256_utf8":"c0561b5b13e19e00b526b65d529983676403790f42449df45dec99b649c602b8"},{"start":37819,"end":38051,"start_line":834,"end_line":841,"raw_tex":"Our main algorithm \n(Algorithm \\ref{alg:main} below) will be defined by solving, at each iteration, a similar inclusion:\n%\n\\begin{align}\\lab{eq:farias}\n%\n0\\in \\lambda_k \\big(F_{y_{k-1}}(x) + N_C(x)\\big) + x - x_{k-1},\n%\n\\end{align}\n","sha256_utf8":"181c7b94921e5a270523b5850101025c207cd2ef58402d65db0497f06dc76a7a"},{"start":38716,"end":39249,"start_line":860,"end_line":881,"raw_tex":"\\bdefi[$\\hat\\sigma$-approximate solution of \\eqref{eq:farias}]\n \\lab{def:app_sol}\n%\nFor $y_{k-1}\\in C$, $x_{k-1}\\in \\HH$ and $\\lambda_k>0$, a pair $(y_k,\\nu_k)\\in \\HH\\times \\HH$ is said to be a \n$\\hat\\sigma$-approximate solution of \\eqref{eq:farias} \nif $\\hat\\sigma\\in [0,1/2)$ and\n%\n\\begin{align}\\lab{eq:app_sol}\n\\begin{aligned}\n%\n\\begin{cases}\n%\n\\nu_k\\in N_C(y_k),\\\\[3mm]\n%\n\\norm{\\lambda_k\\left(F_{y_{k-1}}(y_k) + \\nu_k\\right) + y_k - x_{k-1}} \\leq \\hat\\sigma\\norm{y_k - y_{k-1}}.\n%\n\\end{cases}\n%\n\\end{aligned}\n\\end{align}\n%\n\\edefi","sha256_utf8":"8c827d9e2322cffbb4a98d707c1f6fc2d7760a990128fdd7461c00844cc692f7"},{"start":39259,"end":40200,"start_line":885,"end_line":915,"raw_tex":"Let \n\\begin{align}\n  \\lab{eq:hsigma}\n  0 \\leq \\hat\\sigma <1/2\n\\end{align}\nbe the parameter of approximate solution of \\eqref{eq:farias}, as\nin Definition~\\ref{def:app_sol}.\n%\nThis $\\hat\\sigma$ can be regarded as the inexactness of an (inexact) oracle which\nsolves approximately the strongly monotone linear VIP~\\eqref{eq:farias}.\n\n\nBefore stating the main algorithm, \nchoose and define the parameters $\\theta$, $\\hat\\theta$, $\\eta$, $\\tau$\nas follows:\n% consider\n\\begin{align}\\lab{eq:def.hth}\n  0 < \\theta<(1-\\hat\\sigma)(1-2\\hat\\sigma),\n  \\quad \n  \\hat\\theta & := \\theta\\left(\\dfrac{\\hat\\sigma}{1-\\hat\\sigma}\n+ \\dfrac{\\theta}{(1-\\hat\\sigma)^2}\\right)\n\\end{align}\nand \n\\begin{align}\\lab{eq:def.tau.alg}\n%\n    \\eta & > \\dfrac{2\\hat\\theta}{L},\n  \\quad \\tau  := \\dfrac{2 \\left(\\theta - \\hat\\theta\\right)}\n{2\\theta + \\dfrac{\\eta L}{2} + \\sqrt{\\left(2\\theta + \\dfrac{\\eta L}{2}\\right)^2 \n- 4\\theta\\left(\\theta - \\hat\\theta\\right)}}.\n%\n\\end{align}\n","sha256_utf8":"4050e7fa69bb64cdc19630b0b7df2f988ea11cc3b7ed00967386f39f444cfce5"},{"start":40257,"end":41798,"start_line":926,"end_line":976,"raw_tex":"\\begin{algorithm}[H]\n%\n \\caption{A relative-error inexact proximal-Newton algorithm for solving \\eqref{eq:probm}}\n%\n\\lab{alg:main}\n%\n\\SetAlgoLined\n\\KwInput{$x_0\\in C$,$\\;y_0:=x_0$, $\\nu_0 := 0$, the scalars \n $\\hat\\sigma$, $\\theta$, $\\;\\hat\\theta$ $\\eta$, $\\tau$\n as in \\eqref{eq:hsigma}--\\eqref{eq:def.tau.alg}\nand $\\lambda_1>0$ such that $\\lambda_1^2\\norm{F(y_0)}\\leq 2\\theta/L$\n%\n}\n \\For{$k=1,2,\\dots$}{\n %\n \\If{$F(y_{k-1})+\\nu_{k-1}=0$}\n \t{\\Return $y_{k-1}$\\\\\\lab{eq:return}}\n%\n \\eIf{$\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_{k-1}) + \\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}\\leq \n  \\hat\\theta$}\n  {$y_k = y_{k-1}$\\\\ \\lab{eq:n.cond}\n   $\\nu_k =\\nu_{k-1}$ \\\\ \\lab{eq:n.cond02}}\n  {Find a $\\hat\\sigma$-approximate solution $(y_k,\\nu_k)$ of \\eqref{eq:farias} (in the sense of\n   Definition \\ref{def:app_sol}), i.e., find $(y_k,\\nu_k)$ satisfying\n   %\n   \\begin{align}\\lab{eq:err.cri}\n    %\n    \\begin{cases}\n    %\n   \\nu_k\\in N_C(y_k),\\\\[3mm]\n   %\n   \\norm{\\lambda_k\\left(F_{y_{k-1}}(y_k) + \\nu_k\\right) + y_k - x_{k-1}} \\leq \\hat\\sigma\\norm{y_k - y_{k-1}}\n   %\n   \\end{cases}\n  %\n  \\end{align} \\lab{eq:do.newton}}\n  %\n %  \n   \\eIf{$\\lambda_k\\norm{y_k - x_{k-1}}\\geq \\eta$}\n      {\n      \\vspace{0.2cm}\n      $x_{k} = x_{k-1} -\\tau\\lambda_k \\left(F(y_k) + \\nu_k\\right)$\\\\[2mm]\\lab{eq:ext.step}\n      \\vspace{0.1cm}\n      $\\lambda_{k+1} = (1 - \\tau)\\lambda_k$\\\\[3mm]\\lab{eq:red.lam}\n      }\n     {$x_k = x_{k-1}$\\\\[2mm]\\lab{eq:null.step}\n     $\\lambda_{k+1} = \\dfrac{\\lambda_k}{1 - \\tau}$\\\\\\lab{eq:inc.lam}\n     }\n  %\n  }\n\\end{algorithm}","sha256_utf8":"12a2a10d9a18cc399f39b98118920ad2dfd73d57bfabebed4a57d5a201a98c03"},{"start":42148,"end":42432,"start_line":989,"end_line":999,"raw_tex":"\\item[(ii)] We mention that if Algorithm \\ref{alg:main} stops at step \\ref{eq:return}, then it follows that $y_{k-1}$ is a solution of\n\\eqref{eq:probm}. So from now on in this paper, we assume that\n%\n%\n\\[\n%\n\\fbox{\nAlgorithm \\ref{alg:main} doesn't stop at step \\ref{eq:return}.\n}\n%\n\\]\n","sha256_utf8":"9a0d043a7e22188e3df635602a3dbd59eac83bb446a55ed9c9abfcc686704d43"},{"start":55435,"end":55612,"start_line":1364,"end_line":1370,"raw_tex":"To simplify the presentation, define for $k \\geq 1$\n%\n\\begin{align}\\lab{eq:def.ek}\n%\n  e_k & := \\lambda_k\\left(F_{y_{k-1}}(y_k) + \\nu_k\\right) + y_k - x_{k-1}\\;.\n%\n\\end{align} \n","sha256_utf8":"d3768841a64216eb6d1c728ded35f34dd29205baffd721e809a99afc0306e666"},{"start":65300,"end":66064,"start_line":1678,"end_line":1709,"raw_tex":"For every $j\\geq 1$, define\n%\n\\begin{align}\n  \\lab{eq:def.ab}\n  \\begin{aligned}\n    A_j & =\\left\\{1\\leq k \\leq j\\;|\\;\\; \\text{$\\lambda_k\\norm{y_k - x_{k-1}}\\geq \\eta$\\; holds at iteration $k$}\\right\\},\n\t  & a_j =\\# A_j\\,,\\\\\n    B_j & =\\{1\\leq k\\leq j\\;|\\;\\; \n   \\text{$\\lambda_k\\norm{y_k - x_{k-1}} < \\eta$\\; holds at iteration $k$}\\},\n    & b_j =\\# B_j\\,,\n  \\end{aligned}\n\\end{align}\n%\n\\begin{align}\n \\lab{eq:uni.ab}\n A =\\bigcup_{j=1}^\\infty A_j\\;\\;\\mbox{and}\\;\\; B =\\bigcup_{j=1}^\\infty B_j.\n\\end{align}\n%\nWe also define\n%\n\\begin{align}\n\\lab{eq:s70}\n  K=\\set{k\\geq 1\\;|\\; k\\leq \\# A},\\quad\n  i_0=0,\\quad i_k=\\text{$k$-th element of }A,\n\\end{align}\n%\nand note that \n%\n\\begin{align}\n \\lab{eq:car.a}\n  i_0<i_1<i_2<\\cdots,\\qquad A=\\set{i_k\\;|\\; k\\in K}.\n\\end{align}\n","sha256_utf8":"13a87a28178ca981e0a93ad9f6fb7ad6997fb3a96b8218111c1a8b8a462bd62d"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}[{\\bf First result on the iteration-complexity of Algorithm \\ref{alg:main}}] \\lab{th:main}\n%\nLet $\\rho>0$ be a given tolerance and let $\\hat\\theta>0$, $\\eta>0$, $0<\\tau<1$ and $\\lambda_1>0$ be as in\nthe input of \\emph{Algorithm \\ref{alg:main}}. \nLet also $d_0$ be the distance of $x_0$ to the solution set $(F + N_C)^{-1}(0)\\neq \\emptyset$ of \\eqref{eq:probm}, let\n$L>0$ be as in \\eqref{eq:mon.lip}\nand let $0<\\sigma<1$ be as in \\eqref{eq:def.sigma}.\n%\nThe following holds:\n%\n\\begin{itemize}\n%\n\\item[\\emph{(a)}] \\emph{Algorithm \\ref{alg:main}} finds $y\\in \\HH$ and $\\nu\\in N_C(y)$ such that\n%\n\\begin{align}\\lab{eq:pointw}\n%\n\\norm{F(y) + \\nu}\\leq \\rho\n%\n\\end{align}\n%\nin at most \n%\n\\begin{align}\\lab{eq:def.m}\n%\n \\left\\lceil{\\left(\\dfrac{2}{\\tau \\eta(1-\\sigma)}\\right)\\dfrac{d_0^2}{\\rho}}\\right\\rceil + \n \\left\\lceil{\\dfrac{1}{2\\tau}\\log^+\\left(\\dfrac{\\eta + 2\\hat\\theta/L}{\\lambda_1^2\\rho}\\right)}\\right\\rceil\n%\n\\end{align}\n%\niterations.\n%\n\\item[\\emph{(b)}] \\emph{Algorithm \\ref{alg:main}} finds a triple $(y,v,\\varepsilon)\\in \\HH\\times \\HH\\times \\R_+$  such\nthat\n%\n\\begin{align}\\lab{eq:erg}\n%\nv\\in (F + N_C)^{\\varepsilon}(y)\\;\\;\\emph{and}\\;\\; \\max\\left\\{\\norm{v},\\varepsilon\\right\\}\\leq \\rho\n%\n\\end{align}\n%\nin at most\n%\n\\begin{align}\\lab{eq:def.mt}\n%\n\\begin{aligned}\n& \\max\\left\\{\\left\\lceil{\\left(\\dfrac{2\\sqrt[3]{4}}\n      {\\tau \\eta^{2/3}(1-\\sigma^2)^{1/3}}\\right)\\left(\\dfrac{d_0^2}{\\rho}\\right)^{2/3}}\\right\\rceil, \n      \\left\\lceil{\\left(\\dfrac{2\\sqrt[3]{4}}\n      {\\tau \\eta^{2/3}(1-\\sigma^2)^{2/3}}\\right)\\left(\\dfrac{d_0^3}{\\rho}\\right)^{2/3}}\\right\\rceil\\right\\}\\\\[4mm]\n&\\hspace{3cm}+\\left\\lceil{\\dfrac{1}{2\\tau}\\log^+\\left(\\dfrac{\\eta + 2\\hat\\theta/L}{\\lambda_1^2\\,\\rho}\\right)}\\right\\rceil\n%\n\\end{aligned}\n%\n\\end{align}\n%\niterations.\n%\n\\end{itemize}\n%\n\\end{theorem}","units":[{"start":72037,"end":73832,"start_line":1915,"end_line":1974,"raw_tex":"\\begin{theorem}[{\\bf First result on the iteration-complexity of Algorithm \\ref{alg:main}}] \\lab{th:main}\n%\nLet $\\rho>0$ be a given tolerance and let $\\hat\\theta>0$, $\\eta>0$, $0<\\tau<1$ and $\\lambda_1>0$ be as in\nthe input of \\emph{Algorithm \\ref{alg:main}}. \nLet also $d_0$ be the distance of $x_0$ to the solution set $(F + N_C)^{-1}(0)\\neq \\emptyset$ of \\eqref{eq:probm}, let\n$L>0$ be as in \\eqref{eq:mon.lip}\nand let $0<\\sigma<1$ be as in \\eqref{eq:def.sigma}.\n%\nThe following holds:\n%\n\\begin{itemize}\n%\n\\item[\\emph{(a)}] \\emph{Algorithm \\ref{alg:main}} finds $y\\in \\HH$ and $\\nu\\in N_C(y)$ such that\n%\n\\begin{align}\\lab{eq:pointw}\n%\n\\norm{F(y) + \\nu}\\leq \\rho\n%\n\\end{align}\n%\nin at most \n%\n\\begin{align}\\lab{eq:def.m}\n%\n \\left\\lceil{\\left(\\dfrac{2}{\\tau \\eta(1-\\sigma)}\\right)\\dfrac{d_0^2}{\\rho}}\\right\\rceil + \n \\left\\lceil{\\dfrac{1}{2\\tau}\\log^+\\left(\\dfrac{\\eta + 2\\hat\\theta/L}{\\lambda_1^2\\rho}\\right)}\\right\\rceil\n%\n\\end{align}\n%\niterations.\n%\n\\item[\\emph{(b)}] \\emph{Algorithm \\ref{alg:main}} finds a triple $(y,v,\\varepsilon)\\in \\HH\\times \\HH\\times \\R_+$  such\nthat\n%\n\\begin{align}\\lab{eq:erg}\n%\nv\\in (F + N_C)^{\\varepsilon}(y)\\;\\;\\emph{and}\\;\\; \\max\\left\\{\\norm{v},\\varepsilon\\right\\}\\leq \\rho\n%\n\\end{align}\n%\nin at most\n%\n\\begin{align}\\lab{eq:def.mt}\n%\n\\begin{aligned}\n& \\max\\left\\{\\left\\lceil{\\left(\\dfrac{2\\sqrt[3]{4}}\n      {\\tau \\eta^{2/3}(1-\\sigma^2)^{1/3}}\\right)\\left(\\dfrac{d_0^2}{\\rho}\\right)^{2/3}}\\right\\rceil, \n      \\left\\lceil{\\left(\\dfrac{2\\sqrt[3]{4}}\n      {\\tau \\eta^{2/3}(1-\\sigma^2)^{2/3}}\\right)\\left(\\dfrac{d_0^3}{\\rho}\\right)^{2/3}}\\right\\rceil\\right\\}\\\\[4mm]\n&\\hspace{3cm}+\\left\\lceil{\\dfrac{1}{2\\tau}\\log^+\\left(\\dfrac{\\eta + 2\\hat\\theta/L}{\\lambda_1^2\\,\\rho}\\right)}\\right\\rceil\n%\n\\end{aligned}\n%\n\\end{align}\n%\niterations.\n%\n\\end{itemize}\n%\n\\end{theorem}","sha256_utf8":"f8ffff1db2d0dfb2fe368a30c7b1a849132077c762d5084fd63c4b288fdcb73c"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{theorem} \n \\lab{th:rhpe}\n%\nConsider the sequences evolved by \\emph{Algorithm \\ref{alg:rHPE}} and the ergodic sequences as in \\eqref{eq:def.erg1}--\\eqref{eq:def.erg3}. Let also $d_0$ denote the distance of $x_0$ to the solution set $(F + N_C)^{-1}(0)\\neq \\emptyset$ of \\eqref{eq:probim}. The following hold for all $1\\leq k\\leq N$:\n%\n\\begin{itemize}\n%\n\\item[\\emph{(a)}] There exists $1\\leq j\\leq k$ such that $\\nu_j\\in N_C(y_j)$ and\n%\n\\begin{align*}\n%\n\\norm{F(y_j) + \\nu_j}\\leq \\dfrac{d_0^2}{\\tau \\eta (1-\\sigma)\\,k}\\,.\n%\n\\end{align*}\n%\n\\item[\\emph{(b)}] $v_k^a\\in (F + N_C)^{\\varepsilon_k^a}(y_k^a)$ and\n%\n\\begin{align*}\n%\n\\norm{v_k^a}&\\leq \\dfrac{2d_0^2}{\\tau^{3/2}\\eta\\sqrt{1-\\sigma^2}\\,k^{3/2}}\\,,\\\\[3mm]\n%\n\\varepsilon_k^a&\\leq \\dfrac{2d_0^3}{\\tau^{3/2}\\eta(1-\\sigma^2)\\,k^{3/2}}.\n%\n\\end{align*}\n%\n\\end{itemize}\n%\n\\end{theorem}\n\n\\bproo\n%\nThe proof follows from \\cite[Theorem 2.4]{pre-print-benar}) applied to the maximal monotone operator \n$T = F + N_C$.\n%\n\\eproo\n\n\\brema \\lab{rem:theta}\n%\n\\emph{We mention that\n%\n\\begin{align}\\lab{eq:tau.ineq}\n%\n  0<\\hat\\theta<\\theta\\;\\;\\;\\mbox{and}\\;\\;\\;0<\\dfrac{\\theta -\\hat\\theta}{2\\theta + \\dfrac{\\eta L}{2}}<\\tau<1.\n%\n\\end{align}\n%\nThe fact that $\\theta>\\hat \\theta$ is a direct consequence of the assumption $0<\\theta<(1-\\hat\\sigma)(1-2\\hat\\sigma)$\n(see the input in Algorithm \\ref{alg:main}) and \\eqref{eq:def.hth}. On the other hand, by defining the quadratic function \n$q:\\R\\to \\R$,\n%\n\\begin{align}\\lab{eq:def.q}\n%\nq(t) := \\theta t^2 - \\left(2\\theta + \\dfrac{\\eta L}{2}\\right)t + \\theta -\\hat\\theta,\\qquad t\\in \\R,\n%\n\\end{align}\n%\nwe see that $q(0) = \\theta -\\hat\\theta >0$ and $q(1) = -\\frac{\\eta L}{2} - \\hat\\theta<0$. Then, since $\\tau$ is clearly the smallest root of $q(\\cdot)$, it follows that\n%\n\\begin{align}\\lab{eq:root.q}\n%\n q(\\tau) =0\n%\n\\end{align}\n%\nand\n%\n\\begin{align*}\n%\n-\\dfrac{q(0)}{q'(0)}<\\tau< 1,\n%\n\\end{align*}\n%\nwhich is equivalent to \\eqref{eq:tau.ineq}.}\n%\n\\erema\n\n\n\\begin{proposition}\\lab{prop:stepb}\n%\nConsider the sequences evolved by \\emph{Algorithm \\ref{alg:main}}. Then, for all $k\\geq 1$,\n%\n\\begin{align*}\n%\n\\norm{y_k - y_{k-1}}\\leq \\dfrac{1}{1-\\hat\\sigma}\\norm{\\lambda_k \\left(F(y_{k-1})+\\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}.\n%\n\\end{align*}\n%\n\\end{proposition}\n\n\\bproo\n%\nFirst note that if $y_k$ is computed as in line \\ref{eq:n.cond} of Algorithm \\ref{alg:main}, then the inequality holds trivially. \n%\nAssume now that $y_k$ is computed as in line \\ref{eq:do.newton} of Algorithm \\ref{alg:main}, so that \\eqref{eq:err.cri} holds.\n%\n%\nIn this case, note that from Algorithm \\ref{alg:main}'s definition we have that $\\nu_k\\in N_C(y_k)$ for all $k\\geq 0$. Hence, in view of the monotonicity of $N_C$, we obtain\n%\n\\begin{align}\\lab{eq:lego}\n%\n\\inner{\\nu_k - \\nu_{k-1}}{y_k - y_{k-1}}\\geq 0\\qquad \\forall k\\geq 1.\n%\n\\end{align}\n%\nNow by multiplying the inequality in \\eqref{eq:lego} by $\\lambda_k>0$ \nand using \\eqref{eq:def.ek}\nwe have\n%\n\\begin{align}\\lab{eq:mouse}\n%\n\\inner{e_k + x_{k-1} - y_k - \\lambda_k F_{y_{k-1}}(y_k) - \\lambda_k\\nu_{k-1}}{y_k - y_{k-1}}\\geq 0.\n%\n\\end{align}\n\nNote also that using \\eqref{eq:def.ek} and the inequality in \\eqref{eq:err.cri} we also find\n%\n\\begin{align}\\lab{eq:lego3}\n%\n\\norm{e_k}\\leq \\hat\\sigma \\norm{y_k - y_{k-1}}\n\\qquad \\forall k\\geq 1.\n%\n\\end{align}\n%\nAfter some simple algebraic manipulations with \\eqref{eq:mouse} we obtain\n%\n\\begin{align}\\lab{eq:lego2}\n%\n\\nonumber\n\\normq{y_k - y_{k-1}}&\\leq \\inner{e_k + x_{k-1} - y_{k-1}- \\lambda_k F_{y_{k-1}}(y_k) - \\lambda_k\\nu_{k-1}}{y_k - y_{k-1}}\\\\[2mm]\n%\n\\nonumber\n&=\\inner{e_k + x_{k-1} - y_{k-1}- \\lambda_k (F(y_{k-1})+\\nu_{k-1})}{y_k - y_{k-1}}\\\\[2mm]\n%\n\\nonumber\n&\\hspace{2cm}- \\lambda_k \\inner{F'(y_{k-1})(y_k - y_{k-1})}{y_k - y_{k-1}}\n\\\\[2mm]\n%\n\\nonumber\n&\\leq \\inner{e_k + x_{k-1} - y_{k-1}- \\lambda_k (F(y_{k-1})+\\nu_{k-1})}{y_k - y_{k-1}}\\\\[2mm]\n%\n&\\leq \\big(\\norm{e_k} + \\norm{x_{k-1} - y_{k-1}- \\lambda_k (F(y_{k-1})+\\nu_{k-1})}\\big)\\norm{y_k - y_{k-1}},\n%\n\\end{align}\n%\nwhere we also used the definition of $F_{y_{k-1}}(y_k)$ -- see \\eqref{eq:def.lin} -- and the fact that $F'(y_{k-1})\\geq 0$ (see, e.g., \\cite[Proposition 12.3]{roc.wet-book}).\n\nTo finish the proof of the proposition, note that from \\eqref{eq:lego3} and \\eqref{eq:lego2} we find\n%\n\\begin{align*}\n%\n\\norm{y_k - y_{k-1}}\\leq \\hat\\sigma\\norm{y_k - y_{k-1}} + \\norm{x_{k-1} - y_{k-1}- \\lambda_k (F(y_{k-1})+\\nu_{k-1})},\n%\n\\end{align*}\n%\nwhich is clearly equivalent to the desired inequality.\n%\n\\eproo\n\n\\bprop \\lab{prop:imp}\n%\nConsider the sequences evolved by \\emph{Algorithm \\ref{alg:main}}, let $\\hat\\theta>0$ be as in \\eqref{eq:def.hth} and assume that at iteration $k\\ge1$, $y_k$ and $\\nu_k$ are computed as in line \\ref{eq:do.newton} of \\emph{Algorithm \\ref{alg:main}}, so that  \\eqref{eq:err.cri} holds. Then:\n%\n\\begin{align*}\n%\n\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_{k-1}) + \\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}\\leq \n\\theta\\;\\;\\Rightarrow\\;\\;\n%\n\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\leq \n\\hat\\theta.\n%\n\\end{align*}\n%\n\\eprop\n\n\\bproo\n%\nAssume that $y_k$ is computed as in line \\ref{eq:do.newton} of Algorithm \\ref{alg:main} and\n%\n\\begin{align}\\lab{eq:coffee}\n%\n\\frac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_{k-1}) + \\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}\\leq \n\\theta.\n%\n\\end{align}  \nLetting $e_k$ be as in \\eqref{eq:def.ek}\n%\nand using its definition, some simple algebraic manipulations \nand the inequality in \\eqref{eq:err.cri}\n we find\n%\n\\begin{align*}\n%\n\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}} &= \n               \\norm{e_k + \\lambda_k\\left(F(y_k) - F_{y_{k-1}}(y_k)\\right)}\\\\[2mm]\n               &\\leq \\norm{e_k} + \\lambda_k\\norm{F(y_k) - F_{y_{k-1}}(y_k)}\\\\[2mm]\n               &\\leq \\hat\\sigma \\norm{y_k - y_{k-1}} + \\dfrac{\\lambda_k L}{2}\\normq{y_k - y_{k-1}},\n%\n\\end{align*}\n%\nwhere in the second inequality we also used Lemma \\ref{lm:lip} below. \n\nAs a consequence, from Proposition \\ref{prop:stepb},\n%\n\\begin{align*}\n%\n\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}} &\\leq \n\\dfrac{\\hat\\sigma}{1-\\hat\\sigma}\\norm{\\lambda_k \\left(F(y_{k-1}) + \\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}\\\\[2mm]\n& \\hspace{2cm}+\\dfrac{1}{(1-\\hat\\sigma)^2}\\dfrac{\\lambda_k L}{2}\n\\norm{\\lambda_k \\left(F(y_{k-1}) + \\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}^2.\n%\n\\end{align*}\n%\nBy multiplying the latter inequality by $\\frac{\\lambda_k L}{2}$, using the assumption \n\\eqref{eq:coffee} and \\eqref{eq:def.hth} we obtain\n%\n\\begin{align*}\n%\n\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k F(y_k) + y_k - x_{k-1}} &\\leq \\theta\\left(\\dfrac{\\hat\\sigma}{1-\\hat\\sigma}\n+ \\dfrac{\\theta}{(1-\\hat\\sigma)^2}\\right)\\\\[2mm]\n & = \\hat \\theta,\n%\n\\end{align*}\n%\nwhich finishes the proof of the proposition.\n%\n\\eproo\n\n\\bprop \\lab{prop:inc.theta}\n%\nConsider the sequences evolved by \\emph{Algorithm \\ref{alg:main}} and let $\\hat\\theta$ be as in \\eqref{eq:def.hth}. The following holds:\n%\n\\begin{itemize}\n%\n\\item[\\emph{(a)}] For all $k\\geq 1$,\n%\n\\begin{align}\\lab{eq:inc.theta}\n%\n\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_{k-1}) + \\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}\\leq \n\\theta.\n%\n\\end{align}\n%\n\\item[\\emph{(b)}] For all $k\\geq 1$,\n%\n\\begin{align}\n%\n\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\leq \n\\hat\\theta.\n%\n\\end{align}\n%\n\\end{itemize}\n%\n\\eprop\n\n\\bproo \n%\n(a) Let us proceed by induction on $k\\geq 1$. Note first that \\eqref{eq:inc.theta} is true for $k=1$ in view of \nthe facts that $y_0=x_0$, $\\nu_0 =0$ and $\\frac{\\lambda_1 L}{2}\\norm{\\lambda_1 F(y_0)}\\leq \\theta$ (see the\ninput in Algorithm \\ref{alg:main}).\n%\nAssume now that \\eqref{eq:inc.theta} holds for some $k\\geq 1$. \n%\nWe claim that \n%\n\\begin{align}\\lab{eq:sophia}\n%\n\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\leq \\hat\\theta.\n%\n\\end{align}\n%\nIndeed, if $y_k$ and $\\nu_k$ are given as in lines \\ref{eq:n.cond} and \\ref{eq:n.cond02} of Algorithm \\ref{alg:main}, then we \nreadily obtain \\eqref{eq:sophia}. Otherwise, \\eqref{eq:sophia} follows from the induction hypothesis and Proposition \\ref{prop:imp}.\n\nNext we will consider two cases: $\\lambda_k\\norm{y_k - x_{k-1}}\\geq \\eta$ and\n$\\lambda_k\\norm{y_k - x_{k-1}} < \\eta$. In the first case, using lines \\ref{eq:ext.step} and \\ref{eq:red.lam} of Algorithm \\ref{alg:main} and \\eqref{eq:sophia}, we find\n\n\\begin{align}\\lab{eq:sophia2}\n%\n\\nonumber\n\\dfrac{\\lambda_{k+1} L}{2}\\norm{\\lambda_{k+1} \\left(F(y_k) + \\nu_k\\right) + y_k - x_k} \n%\n\\nonumber\n& = \\dfrac{\\lambda_{k+1} L}{2}\\norm{\\lambda_{k+1} \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1} + x_{k-1} - x_k}\\\\[2mm]\n% \n\\nonumber\n& =\\dfrac{\\lambda_{k+1} L}{2}\\norm{\\lambda_{k+1} \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1} + \\tau\\lambda_k\n   \\left(F(y_k) + \\nu_k\\right)}\\\\[2mm]\n%\n\\nonumber\n& = \\dfrac{\\lambda_{k+1} L}{2}\\norm{\\left(\\lambda_{k+1} + \\tau\\lambda_k\\right) \\left(F(y_k) + \\nu_k\\right) \n+ y_k - x_{k-1}}\\\\[2mm]\n%\n\\nonumber\n& = (1-\\tau)\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\\\[2mm]\n%\n\\nonumber\n& \\leq (1 - \\tau)\\hat \\theta\\\\[2mm]\n%\n&\\leq \\theta,\n %\n\\end{align}\n%\nwhere the latter inequality comes from the fact that $0<\\tau<1$ and \n$\\hat\\theta < \\theta$ (because of the assumption that $0<\\theta<(1-\\hat\\sigma)(1 - 2\\hat\\sigma)$). \n%\nNote that \\eqref{eq:sophia2} finishes the induction argument in the case $\\lambda_k\\norm{y_k - x_{k-1}}\\geq \\eta$.\n\nLet us now consider the case $\\lambda_k\\norm{y_k - x_{k-1}} < \\eta$. Note first that, using the latter inequality, lines \\ref{eq:null.step} and \\ref{eq:inc.lam} of Algorithm \\ref{alg:main} and \\eqref{eq:sophia} combined with the induction hypothesis, we find\n%\n\\begin{align*}\n%\n&\\dfrac{\\lambda_{k+1} L}{2}\\norm{\\lambda_{k+1} \\left(F(y_k) + \\nu_k\\right) + y_k - x_k}\\\\[3mm] \n%\n = &\\dfrac{\\lambda_{k+1} L}{2}\\norm{\\lambda_{k+1} \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\\\[2mm]\n%\n = &\\dfrac{\\lambda_{k+1} L}{2}\\Big\\|\\dfrac{\\lambda_{k+1}}{\\lambda_k}\\big[\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}\\big] \\\\[2mm]\n% \n&\\hspace{2cm} + \\left(1 - \\dfrac{\\lambda_{k+1}}{\\lambda_k}\\right)(y_k - x_{k-1})\\Big\\|\\\\[2mm]\n%\n\\leq &\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k}\\right)^2\\dfrac{\\lambda_k L}{2}\n \\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\\\[2mm] \n%\n&\\hspace{2cm}+ \\dfrac{\\lambda_{k+1} L}{2}\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k} - 1\\right)\\norm{y_k - x_{k-1}}\\\\[2mm]\n%\n< &\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k}\\right)^2 \\hat \\theta + \\dfrac{\\lambda_{k+1} L}{2}\n\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k} - 1\\right)\n\\dfrac{\\eta}{\\lambda_k}\\\\[2mm]\n%\n= &\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k}\\right)^2 \\hat \\theta + \\dfrac{\\lambda_{k+1}}{\\lambda_k}\n\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k} - 1\\right)\n\\dfrac{\\eta L}{2}\\\\[2mm]\n%\n = &\\dfrac{\\hat\\theta}{(1 - \\tau)^2} + \\dfrac{\\tau}{(1-\\tau)^2}\\dfrac{\\eta L}{2}.\n %\n\\end{align*}\n%\nAs a consequence, to complete the induction argument, it is sufficient to show that\n%\n\\begin{align*}\n%\n\\dfrac{\\hat\\theta}{(1 - \\tau)^2} + \\dfrac{\\tau}{(1-\\tau)^2}\\dfrac{\\eta L}{2} = \\theta,\n%\n\\end{align*}\n%\nor, equivalently,\n%\n\\begin{align*}\n%\n(1-\\tau)^2\\theta - \\dfrac{\\eta L}{2}\\tau - \\hat \\theta = 0.\n%\n\\end{align*}\n%\nNote that the latter identity is clearly equivalent to\n%\n\\begin{align*}\n%\n\\theta \\tau^2 - \\left(2\\theta + \\dfrac{\\eta L}{2}\\right)\\tau + \\theta -\\hat\\theta = 0.\n%\n\\end{align*}\n%\nHence, the result follows from \\eqref{eq:def.q} and \\eqref{eq:root.q}. This completes the induction argument and finishes the proof of item (a).\n\n\\mgap\n\n(b) This is a direct consequence of item (a), Proposition \\ref{prop:imp} and lines \\ref{eq:n.cond} and \\ref{eq:n.cond02}\nof Algorithm \\ref{alg:main}.\n%\n\\eproo\n\n\\brema \\lab{rm:rm}\n%\n\\emph{\nWe claim that\nif $(x_k)$ is generated by Algorithm \\ref{alg:main} then\n%\n\\begin{align}\n\\lab{eq:ixk}\n x_{i_k-1}=x_{i_{k-1}}\\qquad \\forall k\\in K,\n\\end{align}\n%\nwhere $K$ is as in \\eqref{eq:s70} -- see also \\eqref{eq:car.a}.\n%\nIndeed, for any $k\\in K$, by \\eqref{eq:uni.ab} and \\eqref{eq:car.a} we have \n$\\{k\\geq 1\\,|\\, i_{k-1}< k <i_k\\}\\subset B$. Consequently, by the definition \nof $B$ as in \\eqref{eq:uni.ab},  and line \\ref{eq:null.step} we conclude that \n$x_i=x_{i_{k-1}}$ whenever $i_{k-1}\\leq i<  i_k$. \nAs a consequence, we obtain that \\eqref{eq:ixk} follows\nfrom the fact that $i_{k-1}\\leq i_k-1<i_k$.}\n%\n\\erema\n\n\n\\bprop \\lab{prop:special}\n%\nConsider the sequences evolved by \\emph{Algorithm \\ref{alg:main}} and let the set $K$ be as in \\eqref{eq:s70} \nand \\eqref{eq:car.a}.\n%\nThen, for all $k\\in K$,\n%\n\\begin{align}\n%\\begin{aligned}\n  %\n  \\lab{eq:stick00}\n  &\\nu_{i_k}\\in N_C(y_{i_k}),\\\\[2mm]\n  \\lab{eq:stick01}\n   &\\left\\|\\lambda_{i_k} \\left(F(y_{i_k}) + \\nu_{i_k}\\right) + y_{i_k} - x_{i_{k-1}}\\right\\|\\leq \\sigma\\norm{y_{i_k} - x_{i_{k-1}}},\\\\[2mm]\n   %\n   \\lab{eq:stick02}\n   &\\lambda_{i_k}\\norm{y_{i_k} - x_{i_{k-1}}}\\geq \\eta,\\\\[2mm]\n   %\n   \\lab{eq:stick03}\n   & x_{i_k} = x_{i_{k-1}} - \\tau\\lambda_{i_k} \\left(F(y_{i_k}) + \\nu_{i_k}\\right),\n  %\n  %\\end{aligned}\n  \\end{align}\n  %\n  where\n %\n \\begin{align}\\lab{eq:def.sigma}\n %\n \\sigma:=\\dfrac{2\\hat\\theta}{\\eta L}<1.\n%\n \\end{align}\n %\n As a consequence, the sequences $(x_{i_k})_{k\\in K}$, $(y_{i_k})_{k\\in K}$ and $(\\lambda_{i_k})_{k\\in K}$ are generated\n by \\emph{Algorithm \\ref{alg:rHPE}} \\emph{(}with $N=\\# A$\\emph{)} for solving \\eqref{eq:probm}.\n%\n\\eprop\n\n\\bproo\n%\nNote first that \\eqref{eq:stick00} is a direct consequence of Algorithm \\ref{alg:main}'s definition and the definition \nof $A$ as in \\eqref{eq:uni.ab}.\n%\nMoreover, note that \\eqref{eq:stick02} and \\eqref{eq:stick03} follow directly from \\eqref{eq:def.ab}--\\eqref{eq:car.a}\nand the assumption that $k\\in K$ (see also line \\ref{eq:ext.step} in Algorithm \\ref{alg:main}).\n%\nLet's now prove the inequality \\eqref{eq:stick01}.\n%\nFrom Proposition \\ref{prop:inc.theta}(b) we obtain\n%\n\\[\n\\dfrac{\\lambda_{i_k}L}{2}\\left\\|\\lambda_{i_k} \\left(F(y_{i_k}) + \\nu_{i_k}\\right) + y_{i_k} - x_{i_k-1}\\right\\|\\leq \\hat\\theta,\n\\]\n%\nwhich when combined with \\eqref{eq:ixk}, \\eqref{eq:def.sigma} and \\eqref{eq:stick02} yields\n%\n\\begin{align*}\n%\n\\left\\|\\lambda_{i_k} \\left(F(y_{i_k}) + \\nu_{i_k}\\right) + y_{i_k} - x_{i_{k-1}}\\right\\|&\\leq \\dfrac{2\\hat\\theta}{\\lambda_{i_k} L}\\\\[2mm]\n    & = \\sigma\\dfrac{\\eta}{\\lambda_{i_k}}\\\\[2mm]\n    & \\leq \\sigma \\norm{y_{i_k} - x_{i_{k-1}}},\n%\n\\end{align*}\n%\nwhich gives exactly \\eqref{eq:stick01}. The last statement in the proposition is a direct consequence of \n\\eqref{eq:stick01}--\\eqref{eq:stick03}, Algorithm \\ref{alg:rHPE}'s definition and the definition of $K$ as in \\eqref{eq:s70}.\n%\n\\eproo\n\n\\bcoro \\lab{cor:rate}\n%\nConsider the sequences evolved by \\emph{Algorithm \\ref{alg:main}}, and the ergodic sequences as in \\eqref{eq:def.erg1}--\\eqref{eq:def.erg3} -- for the sequences $(y_{i_k})_{k\\in K}$ and $(\\lambda_{i_k})_{k\\in K}$, where the set $K$ is as in \\eqref{eq:s70}. \n%\nLet $d_0$ denote the distance of $x_0$ to the solution set $(F + N_C)^{-1}(0)\\neq \\emptyset$ of \\eqref{eq:probm}. \n%\nLet also $0<\\tau<1$ and $0\\leq\\sigma<1$ be as in \\eqref{eq:def.tau.alg} and \\eqref{eq:def.sigma}, respectively.\n\nThe following hold for all $k\\in K$:\n%\n\\begin{itemize}\n%\n\\item[\\emph{(a)}] There exists $1\\leq j\\leq k$ such that $\\nu_{i_j}\\in N_C(y_{i_j})$ and\n%\n\\begin{align*}\n%\n\\norm{F(y_{i_j}) + \\nu_{i_j}}\\leq \\dfrac{d_0^2}{\\tau \\eta(1-\\sigma)\\,k},\n%\n\\end{align*}\n%\n\\item[\\emph{(b)}] $v_k^a\\in (F + N_C)^{\\varepsilon_k^a}(y_k^a)$ and\n%\n\\begin{align*}\n%\n\\norm{v_k^a}&\\leq \\dfrac{2d_0^2}{\\tau^{3/2}\\eta\\sqrt{1-\\sigma^2}\\,k^{3/2}},\\\\[3mm]\n%\n\\varepsilon_k^a&\\leq \\dfrac{2d_0^3}{\\tau^{3/2}\\eta(1-\\sigma^2)\\,k^{3/2}}.\n%\n\\end{align*}\n%\n\\end{itemize}\n%\n\\ecoro\n\n\\bproo\n%\nThe proof is a direct consequence of the last statement in Proposition \\ref{prop:special} and Theorem \\ref{th:rhpe}.\n%\n\\eproo\n\n\\bprop \\lab{prop:iteb}\n%\nConsider the sequences evolved by \\emph{Algorithm \\ref{alg:main}} and let the set $B$ be as in \\eqref{eq:def.ab} and\n\\eqref{eq:uni.ab}. Then, for all $k\\in B$,\n%\n\\begin{align}\\lab{eq:res.b}\n%\n\\nu_k\\in N_C(y_k)\\;\\;\\mbox{and}\\;\\;\\norm{F(y_k) + \\nu_k}\\leq \\dfrac{2\\hat\\theta/L + \\eta}{\\lambda_k^2}, \n%\n\\end{align}\n%\nwhere $\\eta > 0$ is as in the input of \\emph{Algorithm \\ref{alg:main}} and $\\hat\\theta>0$ is as in \\eqref{eq:def.hth}. \n%\n\\eprop\n\n\\bproo\n%\nNote first that the inclusion in \\eqref{eq:res.b} follows directly from Algorithm \\ref{alg:main}'s definition.\n%\nNow, from Proposition \\ref{prop:inc.theta}(b), the definition of $B$ and the triangle inequality, we obtain\n%\n\\begin{align*}\n%\n\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right)} &\\leq \\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}+\\norm{y_k - x_{k-1}}\\\\[2mm]\n   &\\leq \\dfrac{2\\hat\\theta}{\\lambda_k L} + \\dfrac{\\eta}{\\lambda_k},\n%\n\\end{align*}\n%\nwhich clearly gives the desired inequality in \\eqref{eq:res.b}.\n%\n\\eproo\n\n\\begin{lemma} \\lab{lm:geo}\n%\nLet $(\\lambda_k)$ be generated by \\emph{Algorithm \\ref{alg:main}} and let $(a_k)$ and $(b_k)$ be as in \\eqref{eq:def.ab}.\nThen, for all $k\\geq 1$,\n%\n\\begin{align}\n%\n\\lambda_{k+1} = (1 - \\tau)^{a_{k} - b_{k}}\\lambda_1.\n%\n\\end{align}\n%\n\\end{lemma}\n\n\\bproo\n%\nThe result follows from lines \\ref{eq:red.lam} and \\ref{eq:inc.lam} of Algorithm \\ref{alg:main} and the definitions of\n$a_k$ and $b_k$ as in \\eqref{eq:def.ab}.\n%\n\\eproo\n\n\\bproo\n%\n(a) Define\n%\n\\begin{align}\\lab{eq:def.m1}\n%\nM_1 = \\left\\lceil{\\left(\\dfrac{1}{\\tau \\eta(1-\\sigma)}\\right)\\dfrac{d_0^2}{\\rho}}\\right\\rceil\\;\\;\\mbox{and}\\;\\; \nM_2 = \\left\\lceil{\\dfrac{1}{2\\tau}\\log^+\\left(\\dfrac{\\eta + 2\\hat\\theta/L}{\\lambda_1^2\\,\\rho}\\right)}\\right\\rceil \n%\n\\end{align}\n%\nand note that, in view of \\eqref{eq:def.m}, we have\n%\n\\begin{align}\\lab{eq:m1m2}\n%\n M = 2M_1 + M_2,\n%\n\\end{align}\n%\nwhere $M$ is defined as the quantity given in \\eqref{eq:def.m}.\n%`\nWe will consider two cases: (i) $\\#A \\geq M_1$ and (ii) $\\#A < M_1$, where $A$ is as in \\eqref{eq:uni.ab}.\nIn the first case, it follows directly from Corollary \\ref{cor:rate}(a) and the definition of $M_1$ as in \\eqref{eq:def.m1} that\nAlgorithm \\ref{alg:main} finds $y\\in \\HH$ satisfying \\eqref{eq:pointw} in at most $M_1$ iterations. Since $M\\geq M_1$, we have the desired result.\n\nConsider now the case (ii), i.e., $\\#A < M_1$. Let $\\ell\\geq 1$ be such that $\\#A = a_\\ell = a_k$ for all $k\\geq \\ell$ (see\n\\eqref{eq:def.ab} and \\eqref{eq:uni.ab}). Hence, if $b_{k-1}\\geq M_1 + M_2$, for some $k\\geq \\ell + 1$, then it follows that\n%\n\\begin{align}\\lab{eq:bma}\n%\nb_{k-1} - a_{k-1} = b_{k-1} - \\#A > b_{k-1} - M_1 \\geq M_2,\n%\n\\end{align}\n%'\nwhere in the latter inequality we also used \nthe assumption that $b_{k-1}\\geq M_1 + M_2$.\n%\n\nTo conclude that proof of (a), note that \\eqref{eq:bma}, Lemma \\ref{lm:geo}, Proposition \\ref{prop:iteb}, the definition of\n$M_2$ as in \\eqref{eq:m1m2} and some simple algebraic manipulation give that $\\norm{F(y_k) + \\nu_k}\\leq \\rho$.\n%\nAltogether, still in the case (ii), we proved that Algorithm \\ref{alg:main} finds $y\\in \\HH$ satisfying \\eqref{eq:pointw} in at most\n$M_1 + (M_1 + M_2)$ iterations, that is, in at most $M$ iterations.\n\n\\mgap\n\n(b) The proof follows the same outline of the proof in item (a), now using Corollary \\ref{cor:rate}(b) instead of Corollary \\ref{cor:rate}(a).\n%\n\\eproo\n\nA proof of the lemma below in finite-dimensional spaces can be found in \\cite[Lemma 4.1.12]{den.sch-numbook.siam96}. \nA proof in general Hilbert spaces follows the same outline as in the finite-dimensional case.\n\n\n\\begin{lemma}\n \\lab{lm:lip}\n %\n Let $F:\\HH\\to \\HH$ be a continuously differentiable mapping\n  with a $L$-Lipschitz continuous derivative $F'(\\cdot)$, i.e., let $F$ be such that there exits $L\\geq 0$ satisfying\n%\n\\begin{align}\n \\lab{eq:lip.app}\n \\norm{F'(x) - F'(y)}\\leq L\\norm{x - y}\\qquad  \\forall x,y \\in \\HH.\n\\end{align}\n%\nThen,\n%\n  \\[\n   \\norm{F(x) - F_y(x)}\\leq \\dfrac{L}{2}\\norm{x - y}^2 \\qquad \\forall x,y\\in \\HH,\n  \\]\n  %\n  where $F_y(x)$ is as in \\eqref{eq:def.lin}.\n  %\n\\end{lemma}","units":[{"start":34314,"end":35151,"start_line":707,"end_line":734,"raw_tex":"\\begin{theorem} \n \\lab{th:rhpe}\n%\nConsider the sequences evolved by \\emph{Algorithm \\ref{alg:rHPE}} and the ergodic sequences as in \\eqref{eq:def.erg1}--\\eqref{eq:def.erg3}. Let also $d_0$ denote the distance of $x_0$ to the solution set $(F + N_C)^{-1}(0)\\neq \\emptyset$ of \\eqref{eq:probim}. The following hold for all $1\\leq k\\leq N$:\n%\n\\begin{itemize}\n%\n\\item[\\emph{(a)}] There exists $1\\leq j\\leq k$ such that $\\nu_j\\in N_C(y_j)$ and\n%\n\\begin{align*}\n%\n\\norm{F(y_j) + \\nu_j}\\leq \\dfrac{d_0^2}{\\tau \\eta (1-\\sigma)\\,k}\\,.\n%\n\\end{align*}\n%\n\\item[\\emph{(b)}] $v_k^a\\in (F + N_C)^{\\varepsilon_k^a}(y_k^a)$ and\n%\n\\begin{align*}\n%\n\\norm{v_k^a}&\\leq \\dfrac{2d_0^2}{\\tau^{3/2}\\eta\\sqrt{1-\\sigma^2}\\,k^{3/2}}\\,,\\\\[3mm]\n%\n\\varepsilon_k^a&\\leq \\dfrac{2d_0^3}{\\tau^{3/2}\\eta(1-\\sigma^2)\\,k^{3/2}}.\n%\n\\end{align*}\n%\n\\end{itemize}\n%\n\\end{theorem}","sha256_utf8":"9f6dc2d5199c123830a1dca211bf78e25c09dcfea4cbca1902c0a2964ff78d3e"},{"start":35154,"end":35288,"start_line":736,"end_line":741,"raw_tex":"\\bproo\n%\nThe proof follows from \\cite[Theorem 2.4]{pre-print-benar}) applied to the maximal monotone operator \n$T = F + N_C$.\n%\n\\eproo","sha256_utf8":"7b9c4eacec0d7dccd6dc0148ef1b955f4db44f1165604caa876da232b3fd50af"},{"start":44595,"end":45559,"start_line":1038,"end_line":1076,"raw_tex":"\\brema \\lab{rem:theta}\n%\n\\emph{We mention that\n%\n\\begin{align}\\lab{eq:tau.ineq}\n%\n  0<\\hat\\theta<\\theta\\;\\;\\;\\mbox{and}\\;\\;\\;0<\\dfrac{\\theta -\\hat\\theta}{2\\theta + \\dfrac{\\eta L}{2}}<\\tau<1.\n%\n\\end{align}\n%\nThe fact that $\\theta>\\hat \\theta$ is a direct consequence of the assumption $0<\\theta<(1-\\hat\\sigma)(1-2\\hat\\sigma)$\n(see the input in Algorithm \\ref{alg:main}) and \\eqref{eq:def.hth}. On the other hand, by defining the quadratic function \n$q:\\R\\to \\R$,\n%\n\\begin{align}\\lab{eq:def.q}\n%\nq(t) := \\theta t^2 - \\left(2\\theta + \\dfrac{\\eta L}{2}\\right)t + \\theta -\\hat\\theta,\\qquad t\\in \\R,\n%\n\\end{align}\n%\nwe see that $q(0) = \\theta -\\hat\\theta >0$ and $q(1) = -\\frac{\\eta L}{2} - \\hat\\theta<0$. Then, since $\\tau$ is clearly the smallest root of $q(\\cdot)$, it follows that\n%\n\\begin{align}\\lab{eq:root.q}\n%\n q(\\tau) =0\n%\n\\end{align}\n%\nand\n%\n\\begin{align*}\n%\n-\\dfrac{q(0)}{q'(0)}<\\tau< 1,\n%\n\\end{align*}\n%\nwhich is equivalent to \\eqref{eq:tau.ineq}.}\n%\n\\erema\n","sha256_utf8":"bd3d4a6d8387354f9309b2397f8f1144ee51df085089501610b372d2782e51a0"},{"start":55622,"end":55925,"start_line":1376,"end_line":1386,"raw_tex":"\\begin{proposition}\\lab{prop:stepb}\n%\nConsider the sequences evolved by \\emph{Algorithm \\ref{alg:main}}. Then, for all $k\\geq 1$,\n%\n\\begin{align*}\n%\n\\norm{y_k - y_{k-1}}\\leq \\dfrac{1}{1-\\hat\\sigma}\\norm{\\lambda_k \\left(F(y_{k-1})+\\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}.\n%\n\\end{align*}\n%\n\\end{proposition}","sha256_utf8":"faa9e85eb3ab54e27e77eb99aa2cd68900ca18f600d18feaffb80cc9038f4b4b"},{"start":55928,"end":58128,"start_line":1388,"end_line":1455,"raw_tex":"\\bproo\n%\nFirst note that if $y_k$ is computed as in line \\ref{eq:n.cond} of Algorithm \\ref{alg:main}, then the inequality holds trivially. \n%\nAssume now that $y_k$ is computed as in line \\ref{eq:do.newton} of Algorithm \\ref{alg:main}, so that \\eqref{eq:err.cri} holds.\n%\n%\nIn this case, note that from Algorithm \\ref{alg:main}'s definition we have that $\\nu_k\\in N_C(y_k)$ for all $k\\geq 0$. Hence, in view of the monotonicity of $N_C$, we obtain\n%\n\\begin{align}\\lab{eq:lego}\n%\n\\inner{\\nu_k - \\nu_{k-1}}{y_k - y_{k-1}}\\geq 0\\qquad \\forall k\\geq 1.\n%\n\\end{align}\n%\nNow by multiplying the inequality in \\eqref{eq:lego} by $\\lambda_k>0$ \nand using \\eqref{eq:def.ek}\nwe have\n%\n\\begin{align}\\lab{eq:mouse}\n%\n\\inner{e_k + x_{k-1} - y_k - \\lambda_k F_{y_{k-1}}(y_k) - \\lambda_k\\nu_{k-1}}{y_k - y_{k-1}}\\geq 0.\n%\n\\end{align}\n\nNote also that using \\eqref{eq:def.ek} and the inequality in \\eqref{eq:err.cri} we also find\n%\n\\begin{align}\\lab{eq:lego3}\n%\n\\norm{e_k}\\leq \\hat\\sigma \\norm{y_k - y_{k-1}}\n\\qquad \\forall k\\geq 1.\n%\n\\end{align}\n%\nAfter some simple algebraic manipulations with \\eqref{eq:mouse} we obtain\n%\n\\begin{align}\\lab{eq:lego2}\n%\n\\nonumber\n\\normq{y_k - y_{k-1}}&\\leq \\inner{e_k + x_{k-1} - y_{k-1}- \\lambda_k F_{y_{k-1}}(y_k) - \\lambda_k\\nu_{k-1}}{y_k - y_{k-1}}\\\\[2mm]\n%\n\\nonumber\n&=\\inner{e_k + x_{k-1} - y_{k-1}- \\lambda_k (F(y_{k-1})+\\nu_{k-1})}{y_k - y_{k-1}}\\\\[2mm]\n%\n\\nonumber\n&\\hspace{2cm}- \\lambda_k \\inner{F'(y_{k-1})(y_k - y_{k-1})}{y_k - y_{k-1}}\n\\\\[2mm]\n%\n\\nonumber\n&\\leq \\inner{e_k + x_{k-1} - y_{k-1}- \\lambda_k (F(y_{k-1})+\\nu_{k-1})}{y_k - y_{k-1}}\\\\[2mm]\n%\n&\\leq \\big(\\norm{e_k} + \\norm{x_{k-1} - y_{k-1}- \\lambda_k (F(y_{k-1})+\\nu_{k-1})}\\big)\\norm{y_k - y_{k-1}},\n%\n\\end{align}\n%\nwhere we also used the definition of $F_{y_{k-1}}(y_k)$ -- see \\eqref{eq:def.lin} -- and the fact that $F'(y_{k-1})\\geq 0$ (see, e.g., \\cite[Proposition 12.3]{roc.wet-book}).\n\nTo finish the proof of the proposition, note that from \\eqref{eq:lego3} and \\eqref{eq:lego2} we find\n%\n\\begin{align*}\n%\n\\norm{y_k - y_{k-1}}\\leq \\hat\\sigma\\norm{y_k - y_{k-1}} + \\norm{x_{k-1} - y_{k-1}- \\lambda_k (F(y_{k-1})+\\nu_{k-1})},\n%\n\\end{align*}\n%\nwhich is clearly equivalent to the desired inequality.\n%\n\\eproo","sha256_utf8":"f9365e3a151bb47378fb410917bb2bef2425e0795bfe9b76b32cc057a744f4c7"},{"start":58139,"end":58723,"start_line":1461,"end_line":1475,"raw_tex":"\\bprop \\lab{prop:imp}\n%\nConsider the sequences evolved by \\emph{Algorithm \\ref{alg:main}}, let $\\hat\\theta>0$ be as in \\eqref{eq:def.hth} and assume that at iteration $k\\ge1$, $y_k$ and $\\nu_k$ are computed as in line \\ref{eq:do.newton} of \\emph{Algorithm \\ref{alg:main}}, so that  \\eqref{eq:err.cri} holds. Then:\n%\n\\begin{align*}\n%\n\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_{k-1}) + \\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}\\leq \n\\theta\\;\\;\\Rightarrow\\;\\;\n%\n\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\leq \n\\hat\\theta.\n%\n\\end{align*}\n%\n\\eprop","sha256_utf8":"28e57fd29480869dd0371303607e669d8905651195420d27ef2dc3abb6db45d4"},{"start":58726,"end":60395,"start_line":1477,"end_line":1528,"raw_tex":"\\bproo\n%\nAssume that $y_k$ is computed as in line \\ref{eq:do.newton} of Algorithm \\ref{alg:main} and\n%\n\\begin{align}\\lab{eq:coffee}\n%\n\\frac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_{k-1}) + \\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}\\leq \n\\theta.\n%\n\\end{align}  \nLetting $e_k$ be as in \\eqref{eq:def.ek}\n%\nand using its definition, some simple algebraic manipulations \nand the inequality in \\eqref{eq:err.cri}\n we find\n%\n\\begin{align*}\n%\n\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}} &= \n               \\norm{e_k + \\lambda_k\\left(F(y_k) - F_{y_{k-1}}(y_k)\\right)}\\\\[2mm]\n               &\\leq \\norm{e_k} + \\lambda_k\\norm{F(y_k) - F_{y_{k-1}}(y_k)}\\\\[2mm]\n               &\\leq \\hat\\sigma \\norm{y_k - y_{k-1}} + \\dfrac{\\lambda_k L}{2}\\normq{y_k - y_{k-1}},\n%\n\\end{align*}\n%\nwhere in the second inequality we also used Lemma \\ref{lm:lip} below. \n\nAs a consequence, from Proposition \\ref{prop:stepb},\n%\n\\begin{align*}\n%\n\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}} &\\leq \n\\dfrac{\\hat\\sigma}{1-\\hat\\sigma}\\norm{\\lambda_k \\left(F(y_{k-1}) + \\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}\\\\[2mm]\n& \\hspace{2cm}+\\dfrac{1}{(1-\\hat\\sigma)^2}\\dfrac{\\lambda_k L}{2}\n\\norm{\\lambda_k \\left(F(y_{k-1}) + \\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}^2.\n%\n\\end{align*}\n%\nBy multiplying the latter inequality by $\\frac{\\lambda_k L}{2}$, using the assumption \n\\eqref{eq:coffee} and \\eqref{eq:def.hth} we obtain\n%\n\\begin{align*}\n%\n\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k F(y_k) + y_k - x_{k-1}} &\\leq \\theta\\left(\\dfrac{\\hat\\sigma}{1-\\hat\\sigma}\n+ \\dfrac{\\theta}{(1-\\hat\\sigma)^2}\\right)\\\\[2mm]\n & = \\hat \\theta,\n%\n\\end{align*}\n%\nwhich finishes the proof of the proposition.\n%\n\\eproo","sha256_utf8":"a23db971a0dd2eee2d1c1ef980896f6288687005ee7981119947828a47f210a7"},{"start":60404,"end":60981,"start_line":1532,"end_line":1558,"raw_tex":"\\bprop \\lab{prop:inc.theta}\n%\nConsider the sequences evolved by \\emph{Algorithm \\ref{alg:main}} and let $\\hat\\theta$ be as in \\eqref{eq:def.hth}. The following holds:\n%\n\\begin{itemize}\n%\n\\item[\\emph{(a)}] For all $k\\geq 1$,\n%\n\\begin{align}\\lab{eq:inc.theta}\n%\n\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_{k-1}) + \\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}\\leq \n\\theta.\n%\n\\end{align}\n%\n\\item[\\emph{(b)}] For all $k\\geq 1$,\n%\n\\begin{align}\n%\n\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\leq \n\\hat\\theta.\n%\n\\end{align}\n%\n\\end{itemize}\n%\n\\eprop","sha256_utf8":"8e13cd61b69a4e43674dae7bbfc48ba1fe06caedb1e9c226ad690d18f3f4d2d4"},{"start":60984,"end":65285,"start_line":1560,"end_line":1673,"raw_tex":"\\bproo \n%\n(a) Let us proceed by induction on $k\\geq 1$. Note first that \\eqref{eq:inc.theta} is true for $k=1$ in view of \nthe facts that $y_0=x_0$, $\\nu_0 =0$ and $\\frac{\\lambda_1 L}{2}\\norm{\\lambda_1 F(y_0)}\\leq \\theta$ (see the\ninput in Algorithm \\ref{alg:main}).\n%\nAssume now that \\eqref{eq:inc.theta} holds for some $k\\geq 1$. \n%\nWe claim that \n%\n\\begin{align}\\lab{eq:sophia}\n%\n\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\leq \\hat\\theta.\n%\n\\end{align}\n%\nIndeed, if $y_k$ and $\\nu_k$ are given as in lines \\ref{eq:n.cond} and \\ref{eq:n.cond02} of Algorithm \\ref{alg:main}, then we \nreadily obtain \\eqref{eq:sophia}. Otherwise, \\eqref{eq:sophia} follows from the induction hypothesis and Proposition \\ref{prop:imp}.\n\nNext we will consider two cases: $\\lambda_k\\norm{y_k - x_{k-1}}\\geq \\eta$ and\n$\\lambda_k\\norm{y_k - x_{k-1}} < \\eta$. In the first case, using lines \\ref{eq:ext.step} and \\ref{eq:red.lam} of Algorithm \\ref{alg:main} and \\eqref{eq:sophia}, we find\n\n\\begin{align}\\lab{eq:sophia2}\n%\n\\nonumber\n\\dfrac{\\lambda_{k+1} L}{2}\\norm{\\lambda_{k+1} \\left(F(y_k) + \\nu_k\\right) + y_k - x_k} \n%\n\\nonumber\n& = \\dfrac{\\lambda_{k+1} L}{2}\\norm{\\lambda_{k+1} \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1} + x_{k-1} - x_k}\\\\[2mm]\n% \n\\nonumber\n& =\\dfrac{\\lambda_{k+1} L}{2}\\norm{\\lambda_{k+1} \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1} + \\tau\\lambda_k\n   \\left(F(y_k) + \\nu_k\\right)}\\\\[2mm]\n%\n\\nonumber\n& = \\dfrac{\\lambda_{k+1} L}{2}\\norm{\\left(\\lambda_{k+1} + \\tau\\lambda_k\\right) \\left(F(y_k) + \\nu_k\\right) \n+ y_k - x_{k-1}}\\\\[2mm]\n%\n\\nonumber\n& = (1-\\tau)\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\\\[2mm]\n%\n\\nonumber\n& \\leq (1 - \\tau)\\hat \\theta\\\\[2mm]\n%\n&\\leq \\theta,\n %\n\\end{align}\n%\nwhere the latter inequality comes from the fact that $0<\\tau<1$ and \n$\\hat\\theta < \\theta$ (because of the assumption that $0<\\theta<(1-\\hat\\sigma)(1 - 2\\hat\\sigma)$). \n%\nNote that \\eqref{eq:sophia2} finishes the induction argument in the case $\\lambda_k\\norm{y_k - x_{k-1}}\\geq \\eta$.\n\nLet us now consider the case $\\lambda_k\\norm{y_k - x_{k-1}} < \\eta$. Note first that, using the latter inequality, lines \\ref{eq:null.step} and \\ref{eq:inc.lam} of Algorithm \\ref{alg:main} and \\eqref{eq:sophia} combined with the induction hypothesis, we find\n%\n\\begin{align*}\n%\n&\\dfrac{\\lambda_{k+1} L}{2}\\norm{\\lambda_{k+1} \\left(F(y_k) + \\nu_k\\right) + y_k - x_k}\\\\[3mm] \n%\n = &\\dfrac{\\lambda_{k+1} L}{2}\\norm{\\lambda_{k+1} \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\\\[2mm]\n%\n = &\\dfrac{\\lambda_{k+1} L}{2}\\Big\\|\\dfrac{\\lambda_{k+1}}{\\lambda_k}\\big[\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}\\big] \\\\[2mm]\n% \n&\\hspace{2cm} + \\left(1 - \\dfrac{\\lambda_{k+1}}{\\lambda_k}\\right)(y_k - x_{k-1})\\Big\\|\\\\[2mm]\n%\n\\leq &\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k}\\right)^2\\dfrac{\\lambda_k L}{2}\n \\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\\\[2mm] \n%\n&\\hspace{2cm}+ \\dfrac{\\lambda_{k+1} L}{2}\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k} - 1\\right)\\norm{y_k - x_{k-1}}\\\\[2mm]\n%\n< &\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k}\\right)^2 \\hat \\theta + \\dfrac{\\lambda_{k+1} L}{2}\n\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k} - 1\\right)\n\\dfrac{\\eta}{\\lambda_k}\\\\[2mm]\n%\n= &\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k}\\right)^2 \\hat \\theta + \\dfrac{\\lambda_{k+1}}{\\lambda_k}\n\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k} - 1\\right)\n\\dfrac{\\eta L}{2}\\\\[2mm]\n%\n = &\\dfrac{\\hat\\theta}{(1 - \\tau)^2} + \\dfrac{\\tau}{(1-\\tau)^2}\\dfrac{\\eta L}{2}.\n %\n\\end{align*}\n%\nAs a consequence, to complete the induction argument, it is sufficient to show that\n%\n\\begin{align*}\n%\n\\dfrac{\\hat\\theta}{(1 - \\tau)^2} + \\dfrac{\\tau}{(1-\\tau)^2}\\dfrac{\\eta L}{2} = \\theta,\n%\n\\end{align*}\n%\nor, equivalently,\n%\n\\begin{align*}\n%\n(1-\\tau)^2\\theta - \\dfrac{\\eta L}{2}\\tau - \\hat \\theta = 0.\n%\n\\end{align*}\n%\nNote that the latter identity is clearly equivalent to\n%\n\\begin{align*}\n%\n\\theta \\tau^2 - \\left(2\\theta + \\dfrac{\\eta L}{2}\\right)\\tau + \\theta -\\hat\\theta = 0.\n%\n\\end{align*}\n%\nHence, the result follows from \\eqref{eq:def.q} and \\eqref{eq:root.q}. This completes the induction argument and finishes the proof of item (a).\n\n\\mgap\n\n(b) This is a direct consequence of item (a), Proposition \\ref{prop:imp} and lines \\ref{eq:n.cond} and \\ref{eq:n.cond02}\nof Algorithm \\ref{alg:main}.\n%\n\\eproo","sha256_utf8":"2224e74f98e6ab0d0055cb38990562fdf84c478d314530550d213e05439b093e"},{"start":66074,"end":66722,"start_line":1714,"end_line":1734,"raw_tex":"\\brema \\lab{rm:rm}\n%\n\\emph{\nWe claim that\nif $(x_k)$ is generated by Algorithm \\ref{alg:main} then\n%\n\\begin{align}\n\\lab{eq:ixk}\n x_{i_k-1}=x_{i_{k-1}}\\qquad \\forall k\\in K,\n\\end{align}\n%\nwhere $K$ is as in \\eqref{eq:s70} -- see also \\eqref{eq:car.a}.\n%\nIndeed, for any $k\\in K$, by \\eqref{eq:uni.ab} and \\eqref{eq:car.a} we have \n$\\{k\\geq 1\\,|\\, i_{k-1}< k <i_k\\}\\subset B$. Consequently, by the definition \nof $B$ as in \\eqref{eq:uni.ab},  and line \\ref{eq:null.step} we conclude that \n$x_i=x_{i_{k-1}}$ whenever $i_{k-1}\\leq i<  i_k$. \nAs a consequence, we obtain that \\eqref{eq:ixk} follows\nfrom the fact that $i_{k-1}\\leq i_k-1<i_k$.}\n%\n\\erema\n","sha256_utf8":"240232d52a7d13d7f05afb06f712befd7f47f9dcc8b137eacd1cc6360c798b6c"},{"start":66730,"end":67735,"start_line":1738,"end_line":1773,"raw_tex":"\\bprop \\lab{prop:special}\n%\nConsider the sequences evolved by \\emph{Algorithm \\ref{alg:main}} and let the set $K$ be as in \\eqref{eq:s70} \nand \\eqref{eq:car.a}.\n%\nThen, for all $k\\in K$,\n%\n\\begin{align}\n%\\begin{aligned}\n  %\n  \\lab{eq:stick00}\n  &\\nu_{i_k}\\in N_C(y_{i_k}),\\\\[2mm]\n  \\lab{eq:stick01}\n   &\\left\\|\\lambda_{i_k} \\left(F(y_{i_k}) + \\nu_{i_k}\\right) + y_{i_k} - x_{i_{k-1}}\\right\\|\\leq \\sigma\\norm{y_{i_k} - x_{i_{k-1}}},\\\\[2mm]\n   %\n   \\lab{eq:stick02}\n   &\\lambda_{i_k}\\norm{y_{i_k} - x_{i_{k-1}}}\\geq \\eta,\\\\[2mm]\n   %\n   \\lab{eq:stick03}\n   & x_{i_k} = x_{i_{k-1}} - \\tau\\lambda_{i_k} \\left(F(y_{i_k}) + \\nu_{i_k}\\right),\n  %\n  %\\end{aligned}\n  \\end{align}\n  %\n  where\n %\n \\begin{align}\\lab{eq:def.sigma}\n %\n \\sigma:=\\dfrac{2\\hat\\theta}{\\eta L}<1.\n%\n \\end{align}\n %\n As a consequence, the sequences $(x_{i_k})_{k\\in K}$, $(y_{i_k})_{k\\in K}$ and $(\\lambda_{i_k})_{k\\in K}$ are generated\n by \\emph{Algorithm \\ref{alg:rHPE}} \\emph{(}with $N=\\# A$\\emph{)} for solving \\eqref{eq:probm}.\n%\n\\eprop","sha256_utf8":"decdcf5f247ed7a7977a024fe5f4b976886088019ae668cdec30cda9fe2eb1c4"},{"start":67738,"end":68963,"start_line":1775,"end_line":1804,"raw_tex":"\\bproo\n%\nNote first that \\eqref{eq:stick00} is a direct consequence of Algorithm \\ref{alg:main}'s definition and the definition \nof $A$ as in \\eqref{eq:uni.ab}.\n%\nMoreover, note that \\eqref{eq:stick02} and \\eqref{eq:stick03} follow directly from \\eqref{eq:def.ab}--\\eqref{eq:car.a}\nand the assumption that $k\\in K$ (see also line \\ref{eq:ext.step} in Algorithm \\ref{alg:main}).\n%\nLet's now prove the inequality \\eqref{eq:stick01}.\n%\nFrom Proposition \\ref{prop:inc.theta}(b) we obtain\n%\n\\[\n\\dfrac{\\lambda_{i_k}L}{2}\\left\\|\\lambda_{i_k} \\left(F(y_{i_k}) + \\nu_{i_k}\\right) + y_{i_k} - x_{i_k-1}\\right\\|\\leq \\hat\\theta,\n\\]\n%\nwhich when combined with \\eqref{eq:ixk}, \\eqref{eq:def.sigma} and \\eqref{eq:stick02} yields\n%\n\\begin{align*}\n%\n\\left\\|\\lambda_{i_k} \\left(F(y_{i_k}) + \\nu_{i_k}\\right) + y_{i_k} - x_{i_{k-1}}\\right\\|&\\leq \\dfrac{2\\hat\\theta}{\\lambda_{i_k} L}\\\\[2mm]\n    & = \\sigma\\dfrac{\\eta}{\\lambda_{i_k}}\\\\[2mm]\n    & \\leq \\sigma \\norm{y_{i_k} - x_{i_{k-1}}},\n%\n\\end{align*}\n%\nwhich gives exactly \\eqref{eq:stick01}. The last statement in the proposition is a direct consequence of \n\\eqref{eq:stick01}--\\eqref{eq:stick03}, Algorithm \\ref{alg:rHPE}'s definition and the definition of $K$ as in \\eqref{eq:s70}.\n%\n\\eproo","sha256_utf8":"471ea62a098f1ea5201e77c4b3049997c23832d2386ebf970e70a08bd768fb9c"},{"start":68972,"end":70026,"start_line":1808,"end_line":1840,"raw_tex":"\\bcoro \\lab{cor:rate}\n%\nConsider the sequences evolved by \\emph{Algorithm \\ref{alg:main}}, and the ergodic sequences as in \\eqref{eq:def.erg1}--\\eqref{eq:def.erg3} -- for the sequences $(y_{i_k})_{k\\in K}$ and $(\\lambda_{i_k})_{k\\in K}$, where the set $K$ is as in \\eqref{eq:s70}. \n%\nLet $d_0$ denote the distance of $x_0$ to the solution set $(F + N_C)^{-1}(0)\\neq \\emptyset$ of \\eqref{eq:probm}. \n%\nLet also $0<\\tau<1$ and $0\\leq\\sigma<1$ be as in \\eqref{eq:def.tau.alg} and \\eqref{eq:def.sigma}, respectively.\n\nThe following hold for all $k\\in K$:\n%\n\\begin{itemize}\n%\n\\item[\\emph{(a)}] There exists $1\\leq j\\leq k$ such that $\\nu_{i_j}\\in N_C(y_{i_j})$ and\n%\n\\begin{align*}\n%\n\\norm{F(y_{i_j}) + \\nu_{i_j}}\\leq \\dfrac{d_0^2}{\\tau \\eta(1-\\sigma)\\,k},\n%\n\\end{align*}\n%\n\\item[\\emph{(b)}] $v_k^a\\in (F + N_C)^{\\varepsilon_k^a}(y_k^a)$ and\n%\n\\begin{align*}\n%\n\\norm{v_k^a}&\\leq \\dfrac{2d_0^2}{\\tau^{3/2}\\eta\\sqrt{1-\\sigma^2}\\,k^{3/2}},\\\\[3mm]\n%\n\\varepsilon_k^a&\\leq \\dfrac{2d_0^3}{\\tau^{3/2}\\eta(1-\\sigma^2)\\,k^{3/2}}.\n%\n\\end{align*}\n%\n\\end{itemize}\n%\n\\ecoro","sha256_utf8":"f0e7018058d8bcf4e3075495b595a8c5461f3d31f1276fece30fbc1e55847304"},{"start":70029,"end":70163,"start_line":1842,"end_line":1846,"raw_tex":"\\bproo\n%\nThe proof is a direct consequence of the last statement in Proposition \\ref{prop:special} and Theorem \\ref{th:rhpe}.\n%\n\\eproo","sha256_utf8":"c4a0efc993b231680b53a4d2ad625e3bdd0949c2f64c5fc7650a789f04e6fe92"},{"start":70173,"end":70639,"start_line":1851,"end_line":1864,"raw_tex":"\\bprop \\lab{prop:iteb}\n%\nConsider the sequences evolved by \\emph{Algorithm \\ref{alg:main}} and let the set $B$ be as in \\eqref{eq:def.ab} and\n\\eqref{eq:uni.ab}. Then, for all $k\\in B$,\n%\n\\begin{align}\\lab{eq:res.b}\n%\n\\nu_k\\in N_C(y_k)\\;\\;\\mbox{and}\\;\\;\\norm{F(y_k) + \\nu_k}\\leq \\dfrac{2\\hat\\theta/L + \\eta}{\\lambda_k^2}, \n%\n\\end{align}\n%\nwhere $\\eta > 0$ is as in the input of \\emph{Algorithm \\ref{alg:main}} and $\\hat\\theta>0$ is as in \\eqref{eq:def.hth}. \n%\n\\eprop","sha256_utf8":"28f6a96d0b1c8273eb7d230c41028528a0205f5781d1a1ed0d0788d872249b98"},{"start":70642,"end":71189,"start_line":1866,"end_line":1881,"raw_tex":"\\bproo\n%\nNote first that the inclusion in \\eqref{eq:res.b} follows directly from Algorithm \\ref{alg:main}'s definition.\n%\nNow, from Proposition \\ref{prop:inc.theta}(b), the definition of $B$ and the triangle inequality, we obtain\n%\n\\begin{align*}\n%\n\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right)} &\\leq \\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}+\\norm{y_k - x_{k-1}}\\\\[2mm]\n   &\\leq \\dfrac{2\\hat\\theta}{\\lambda_k L} + \\dfrac{\\eta}{\\lambda_k},\n%\n\\end{align*}\n%\nwhich clearly gives the desired inequality in \\eqref{eq:res.b}.\n%\n\\eproo","sha256_utf8":"92497f18280d0bacddc33a559afb6ef17dc4d84416f836770f89819bd8a4e97f"},{"start":71199,"end":71473,"start_line":1886,"end_line":1897,"raw_tex":"\\begin{lemma} \\lab{lm:geo}\n%\nLet $(\\lambda_k)$ be generated by \\emph{Algorithm \\ref{alg:main}} and let $(a_k)$ and $(b_k)$ be as in \\eqref{eq:def.ab}.\nThen, for all $k\\geq 1$,\n%\n\\begin{align}\n%\n\\lambda_{k+1} = (1 - \\tau)^{a_{k} - b_{k}}\\lambda_1.\n%\n\\end{align}\n%\n\\end{lemma}","sha256_utf8":"ab99688cea5390aae3f9f192d58dc554ff0ce4cdd65849491fd4cf349e5cc6f7"},{"start":71476,"end":71653,"start_line":1899,"end_line":1904,"raw_tex":"\\bproo\n%\nThe result follows from lines \\ref{eq:red.lam} and \\ref{eq:inc.lam} of Algorithm \\ref{alg:main} and the definitions of\n$a_k$ and $b_k$ as in \\eqref{eq:def.ab}.\n%\n\\eproo","sha256_utf8":"8b8fb8cff337c3426811ad96000caedcf1ff332e0bc86ec212a9abe608f76875"},{"start":73835,"end":75745,"start_line":1976,"end_line":2024,"raw_tex":"\\bproo\n%\n(a) Define\n%\n\\begin{align}\\lab{eq:def.m1}\n%\nM_1 = \\left\\lceil{\\left(\\dfrac{1}{\\tau \\eta(1-\\sigma)}\\right)\\dfrac{d_0^2}{\\rho}}\\right\\rceil\\;\\;\\mbox{and}\\;\\; \nM_2 = \\left\\lceil{\\dfrac{1}{2\\tau}\\log^+\\left(\\dfrac{\\eta + 2\\hat\\theta/L}{\\lambda_1^2\\,\\rho}\\right)}\\right\\rceil \n%\n\\end{align}\n%\nand note that, in view of \\eqref{eq:def.m}, we have\n%\n\\begin{align}\\lab{eq:m1m2}\n%\n M = 2M_1 + M_2,\n%\n\\end{align}\n%\nwhere $M$ is defined as the quantity given in \\eqref{eq:def.m}.\n%`\nWe will consider two cases: (i) $\\#A \\geq M_1$ and (ii) $\\#A < M_1$, where $A$ is as in \\eqref{eq:uni.ab}.\nIn the first case, it follows directly from Corollary \\ref{cor:rate}(a) and the definition of $M_1$ as in \\eqref{eq:def.m1} that\nAlgorithm \\ref{alg:main} finds $y\\in \\HH$ satisfying \\eqref{eq:pointw} in at most $M_1$ iterations. Since $M\\geq M_1$, we have the desired result.\n\nConsider now the case (ii), i.e., $\\#A < M_1$. Let $\\ell\\geq 1$ be such that $\\#A = a_\\ell = a_k$ for all $k\\geq \\ell$ (see\n\\eqref{eq:def.ab} and \\eqref{eq:uni.ab}). Hence, if $b_{k-1}\\geq M_1 + M_2$, for some $k\\geq \\ell + 1$, then it follows that\n%\n\\begin{align}\\lab{eq:bma}\n%\nb_{k-1} - a_{k-1} = b_{k-1} - \\#A > b_{k-1} - M_1 \\geq M_2,\n%\n\\end{align}\n%'\nwhere in the latter inequality we also used \nthe assumption that $b_{k-1}\\geq M_1 + M_2$.\n%\n\nTo conclude that proof of (a), note that \\eqref{eq:bma}, Lemma \\ref{lm:geo}, Proposition \\ref{prop:iteb}, the definition of\n$M_2$ as in \\eqref{eq:m1m2} and some simple algebraic manipulation give that $\\norm{F(y_k) + \\nu_k}\\leq \\rho$.\n%\nAltogether, still in the case (ii), we proved that Algorithm \\ref{alg:main} finds $y\\in \\HH$ satisfying \\eqref{eq:pointw} in at most\n$M_1 + (M_1 + M_2)$ iterations, that is, in at most $M$ iterations.\n\n\\mgap\n\n(b) The proof follows the same outline of the proof in item (a), now using Corollary \\ref{cor:rate}(b) instead of Corollary \\ref{cor:rate}(a).\n%\n\\eproo","sha256_utf8":"53380f10cdef558a309ef7f242cdfdab2c9e97e480293ab2d3394eab843e0db5"},{"start":86642,"end":86854,"start_line":2301,"end_line":2302,"raw_tex":"A proof of the lemma below in finite-dimensional spaces can be found in \\cite[Lemma 4.1.12]{den.sch-numbook.siam96}. \nA proof in general Hilbert spaces follows the same outline as in the finite-dimensional case.\n","sha256_utf8":"40731d4ae91190b36374093106d52bb41d68c65a084cfb46f32d687121252b29"},{"start":86862,"end":87353,"start_line":2306,"end_line":2325,"raw_tex":"\\begin{lemma}\n \\lab{lm:lip}\n %\n Let $F:\\HH\\to \\HH$ be a continuously differentiable mapping\n  with a $L$-Lipschitz continuous derivative $F'(\\cdot)$, i.e., let $F$ be such that there exits $L\\geq 0$ satisfying\n%\n\\begin{align}\n \\lab{eq:lip.app}\n \\norm{F'(x) - F'(y)}\\leq L\\norm{x - y}\\qquad  \\forall x,y \\in \\HH.\n\\end{align}\n%\nThen,\n%\n  \\[\n   \\norm{F(x) - F_y(x)}\\leq \\dfrac{L}{2}\\norm{x - y}^2 \\qquad \\forall x,y\\in \\HH,\n  \\]\n  %\n  where $F_y(x)$ is as in \\eqref{eq:def.lin}.\n  %\n\\end{lemma}","sha256_utf8":"5f4cb205aab6443d3c02568429bca6e721b0ad430077007093d4c7579be2c1cf"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"希尔伯特空间与变分不等式","source_units":[{"start":27874,"end":28105,"start_line":547,"end_line":550,"raw_tex":"Let $\\HH$ be a real Hilbert space with inner product $\\inner{\\cdot}{\\cdot}$ and induced norm \n$\\norm{\\cdot} = \\sqrt{\\inner{\\cdot}{\\cdot}}$.\n%\nWe use the same notation $\\norm{A}$ to denote the norm of a bounded linear operator $A$.\n","sha256_utf8":"80d510fa38c3a0e12febd004dcf1cf0f732242a8af7da49a47497d5b3e16c9c6"},{"start":36257,"end":36456,"start_line":782,"end_line":788,"raw_tex":"Consider the (monotone) VIP given in \\eqref{eq:probi}, i.e.,\n%\n\\begin{align} \\lab{eq:prob}\n%\n\\mbox{Find}\\;\\;x\\in C\\;\\;\\mbox{such that}\\;\\; \\inner{F(x)}{y-x}\\geq 0\\qquad \\forall y\\in C,\n%\n\\end{align}\n","sha256_utf8":"9747e93827faf8e5c12334d840b1dbf094f3ded62d97153099c3018dd634f384"},{"start":36897,"end":37035,"start_line":804,"end_line":810,"raw_tex":"Recall that \\eqref{eq:prob} is equivalent to \\eqref{eq:probim}, i.e.,\n%\n\\begin{align}\\lab{eq:probm}\n%\n 0 \\in F(x) + N_C(x).\n%\n\\end{align}\n","sha256_utf8":"cc333bdaf244ca5b487f9d4c49df6f27ea7ea7611a964f7274cc792548fb686c"}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","title":"单调光滑性与解存在","source_units":[{"start":36458,"end":36896,"start_line":790,"end_line":802,"raw_tex":"where $C$ is a (nonempty) closed and convex subset of $\\HH$ and \n$F:C\\to \\HH$ is assumed to be (point-to-point) \\emph{monotone}\nand continuously differentiable with a $L$-Lipschitz continuous derivative, i.e., $F$ and $F'$ satisfy the following conditions for all\n$x,y\\in C$:\n%\n\\begin{align}\\lab{eq:mon.lip}\n\\begin{aligned}\n%\n& \\inner{F(x) - F(y)}{x - y}\\geq 0,\\\\[2mm]\n& \\norm{F'(x) - F'(y)}\\leq L\\norm{x-y}.\n %\n\\end{aligned}\n\\end{align}\n","sha256_utf8":"8256267b9077f4b0e5c60d359d2472575d6227d1086c011612d98c990f8fd9cd"},{"start":37038,"end":37183,"start_line":812,"end_line":813,"raw_tex":"We assume throughout this paper that the solution set $(F+N_C)^{-1}(0)$ of \\eqref{eq:probm} \n(or, equivalently, of \\eqref{eq:prob}) is nonempty.\n","sha256_utf8":"241e29d9cc4e43195c4a9870f1cbb05abd2be3d3dacf50c33d4ff2761fb64ecf"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"法锥与算子扩张","source_units":[{"start":28107,"end":28663,"start_line":552,"end_line":564,"raw_tex":"The \\emph{normal cone} to $C\\subset \\HH$ at $x\\in C$ is defined as\n%\n%\\[\n$N_C(x) = \\left\\{\\nu\\in \\HH\\;\\;|\\;\\;\\inner{\\nu}{y-x}\\leq 0\\quad \\forall y\\in C\\right\\}.$\n%\\]\n%\nWe also set $N_C(x) := \\emptyset$ if $x\\notin C$. The $\\varepsilon$-enlargement of the set-valued map $T:\\HH\\tos \\HH$ (see, e.g., \\cite{bur.ius.sva-enl.svva97}) is the (set-valued map) $T^\\varepsilon:\\HH\\tos \\HH$ defined as\n%\n\\begin{align}\\lab{eq:def.enl}\n%\nT^{\\varepsilon}(x) = \\left\\{v\\in \\HH\\;\\;|\\;\\;\\inner{v - u}{x - z}\\geq -\\varepsilon\\quad \\forall u\\in T(z)\\right\\}. \n%\n\\end{align}\n","sha256_utf8":"25c211cd7e517c698702e3565e1ead852ec14998435d3d986e3bcbcdbfbdf682"}]},{"id":"S004","semantic_type":"notation","kind":"notation","title":"线性化及计数记号","source_units":[{"start":28665,"end":29044,"start_line":566,"end_line":577,"raw_tex":"We also use the notation \n%\n\\begin{align}\\lab{eq:def.lin}\n%\nF_y(x) := F(y) + F'(y)(x-y)\\qquad \\forall x\\in C\n%\n\\end{align}\n%\nto indicate the linearization of a smooth map $F:C\\subset \\HH\\to \\HH$ at $y\\in C$.\n%\nThe set of natural numbers is $\\N = \\{1,2,\\dots\\}$ and $\\# S$ stands for the number of elements of\na set $S$. We also define $\\log^+(t) = \\max\\{\\log(t), 0\\}$ for $t>0$.\n","sha256_utf8":"85a713fcee26638705c0c6e857b36cf13b4e8e45cd56807e3fd435fa2e2fb3eb"}]},{"id":"S005","semantic_type":"algorithm","kind":"algorithm","title":"大步长欠松弛HPE框架","source_units":[{"start":29521,"end":30268,"start_line":592,"end_line":620,"raw_tex":"\\begin{algorithm}[H]\n\\lab{alg:rHPE}\n \\caption{A special instance of the large-step under-relaxed HPE method for \\eqref{eq:probim}}\n%\n\\lab{alg_gen}\n%\n\\SetAlgoLined\n\\KwInput{$x_0\\in \\HH$, $0<\\tau<1$, $0\\leq \\sigma <1$, $\\eta>0$ and $N\\in \\N\\cup \\{\\infty\\}$}\n \\For{$k = 1,2,\\dots, N$}{\n  Choose $\\lambda_k>0$ and find $y_k\\in \\HH$ and $\\nu_k\\in N_C(y_k)$ such that\\\\\n  %\n  \\begin{align}\n  %\n  \\lab{eq:rel.al1}\n   &\\left\\|\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}\\right\\|\\leq \\sigma\\norm{y_k - x_{k-1}}\\\\[2mm]\n   \\lab{eq:lar.ste}\n   &\\lambda_k\\norm{y_k - x_{k-1}}\\geq \\eta\n  %\n  \\end{align}\n  %\n  Set\n  %\n  \\begin{align}\\lab{eq:ext.al1}\n  %\n   x_k = x_{k-1} - \\tau\\lambda_k \\left(F(y_k) + \\nu_k\\right)\n  %\n  \\end{align}\n  }\n\\end{algorithm}","sha256_utf8":"f12d61a999c661ce4e225fef985eba2089fa65aa2903a79d9880191d63b68cde"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"加权遍历序列","source_units":[{"start":33670,"end":34231,"start_line":685,"end_line":698,"raw_tex":"The \\emph{ergodic sequences} associated to the sequences generated by Algorithm \\ref{alg:rHPE} are defined as \nfollows (see, e.g., \\cite{mon.sva-hpe.siam10}):\n\n\\begin{align}\\lab{eq:def.erg1}\n%\n& y_k^a = \\dfrac{1}{\\sum_{i=1}^k\\,\\lambda_i}\\,\\sum_{i=1}^k\\,\\lambda_i  y_i,\\\\[2mm]\n%\n\\lab{eq:def.erg2}\n& v_k^a = \\dfrac{1}{\\sum_{i=1}^k\\,\\lambda_i}\\,\\sum_{i=1}^k\\,\\lambda_i  \\left(F(y_i) + \\nu_i\\right),\\\\[2mm]\n%\n\\lab{eq:def.erg3}\n& \\varepsilon_k^a = \\dfrac{1}{\\sum_{i=1}^k\\,\\lambda_i}\\,\\sum_{i=1}^k\\,\\lambda_i \\inner{y_i - y_k^a}{F(y_i) + \\nu_i- v_k^a}.\n%\n\\end{align}\n","sha256_utf8":"a2bbe2a2d3e8715d79427d04d031bd415b5286a4aa08bd28482d842907367862"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"线性化近端子问题","source_units":[{"start":37819,"end":38051,"start_line":834,"end_line":841,"raw_tex":"Our main algorithm \n(Algorithm \\ref{alg:main} below) will be defined by solving, at each iteration, a similar inclusion:\n%\n\\begin{align}\\lab{eq:farias}\n%\n0\\in \\lambda_k \\big(F_{y_{k-1}}(x) + N_C(x)\\big) + x - x_{k-1},\n%\n\\end{align}\n","sha256_utf8":"181c7b94921e5a270523b5850101025c207cd2ef58402d65db0497f06dc76a7a"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"子问题相对误差解","source_units":[{"start":38716,"end":39249,"start_line":860,"end_line":881,"raw_tex":"\\bdefi[$\\hat\\sigma$-approximate solution of \\eqref{eq:farias}]\n \\lab{def:app_sol}\n%\nFor $y_{k-1}\\in C$, $x_{k-1}\\in \\HH$ and $\\lambda_k>0$, a pair $(y_k,\\nu_k)\\in \\HH\\times \\HH$ is said to be a \n$\\hat\\sigma$-approximate solution of \\eqref{eq:farias} \nif $\\hat\\sigma\\in [0,1/2)$ and\n%\n\\begin{align}\\lab{eq:app_sol}\n\\begin{aligned}\n%\n\\begin{cases}\n%\n\\nu_k\\in N_C(y_k),\\\\[3mm]\n%\n\\norm{\\lambda_k\\left(F_{y_{k-1}}(y_k) + \\nu_k\\right) + y_k - x_{k-1}} \\leq \\hat\\sigma\\norm{y_k - y_{k-1}}.\n%\n\\end{cases}\n%\n\\end{aligned}\n\\end{align}\n%\n\\edefi","sha256_utf8":"8c827d9e2322cffbb4a98d707c1f6fc2d7760a990128fdd7461c00844cc692f7"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"误差与同伦参数","source_units":[{"start":39259,"end":40200,"start_line":885,"end_line":915,"raw_tex":"Let \n\\begin{align}\n  \\lab{eq:hsigma}\n  0 \\leq \\hat\\sigma <1/2\n\\end{align}\nbe the parameter of approximate solution of \\eqref{eq:farias}, as\nin Definition~\\ref{def:app_sol}.\n%\nThis $\\hat\\sigma$ can be regarded as the inexactness of an (inexact) oracle which\nsolves approximately the strongly monotone linear VIP~\\eqref{eq:farias}.\n\n\nBefore stating the main algorithm, \nchoose and define the parameters $\\theta$, $\\hat\\theta$, $\\eta$, $\\tau$\nas follows:\n% consider\n\\begin{align}\\lab{eq:def.hth}\n  0 < \\theta<(1-\\hat\\sigma)(1-2\\hat\\sigma),\n  \\quad \n  \\hat\\theta & := \\theta\\left(\\dfrac{\\hat\\sigma}{1-\\hat\\sigma}\n+ \\dfrac{\\theta}{(1-\\hat\\sigma)^2}\\right)\n\\end{align}\nand \n\\begin{align}\\lab{eq:def.tau.alg}\n%\n    \\eta & > \\dfrac{2\\hat\\theta}{L},\n  \\quad \\tau  := \\dfrac{2 \\left(\\theta - \\hat\\theta\\right)}\n{2\\theta + \\dfrac{\\eta L}{2} + \\sqrt{\\left(2\\theta + \\dfrac{\\eta L}{2}\\right)^2 \n- 4\\theta\\left(\\theta - \\hat\\theta\\right)}}.\n%\n\\end{align}\n","sha256_utf8":"4050e7fa69bb64cdc19630b0b7df2f988ea11cc3b7ed00967386f39f444cfce5"}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","title":"无搜索非精确近端牛顿法","source_units":[{"start":40257,"end":41798,"start_line":926,"end_line":976,"raw_tex":"\\begin{algorithm}[H]\n%\n \\caption{A relative-error inexact proximal-Newton algorithm for solving \\eqref{eq:probm}}\n%\n\\lab{alg:main}\n%\n\\SetAlgoLined\n\\KwInput{$x_0\\in C$,$\\;y_0:=x_0$, $\\nu_0 := 0$, the scalars \n $\\hat\\sigma$, $\\theta$, $\\;\\hat\\theta$ $\\eta$, $\\tau$\n as in \\eqref{eq:hsigma}--\\eqref{eq:def.tau.alg}\nand $\\lambda_1>0$ such that $\\lambda_1^2\\norm{F(y_0)}\\leq 2\\theta/L$\n%\n}\n \\For{$k=1,2,\\dots$}{\n %\n \\If{$F(y_{k-1})+\\nu_{k-1}=0$}\n \t{\\Return $y_{k-1}$\\\\\\lab{eq:return}}\n%\n \\eIf{$\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_{k-1}) + \\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}\\leq \n  \\hat\\theta$}\n  {$y_k = y_{k-1}$\\\\ \\lab{eq:n.cond}\n   $\\nu_k =\\nu_{k-1}$ \\\\ \\lab{eq:n.cond02}}\n  {Find a $\\hat\\sigma$-approximate solution $(y_k,\\nu_k)$ of \\eqref{eq:farias} (in the sense of\n   Definition \\ref{def:app_sol}), i.e., find $(y_k,\\nu_k)$ satisfying\n   %\n   \\begin{align}\\lab{eq:err.cri}\n    %\n    \\begin{cases}\n    %\n   \\nu_k\\in N_C(y_k),\\\\[3mm]\n   %\n   \\norm{\\lambda_k\\left(F_{y_{k-1}}(y_k) + \\nu_k\\right) + y_k - x_{k-1}} \\leq \\hat\\sigma\\norm{y_k - y_{k-1}}\n   %\n   \\end{cases}\n  %\n  \\end{align} \\lab{eq:do.newton}}\n  %\n %  \n   \\eIf{$\\lambda_k\\norm{y_k - x_{k-1}}\\geq \\eta$}\n      {\n      \\vspace{0.2cm}\n      $x_{k} = x_{k-1} -\\tau\\lambda_k \\left(F(y_k) + \\nu_k\\right)$\\\\[2mm]\\lab{eq:ext.step}\n      \\vspace{0.1cm}\n      $\\lambda_{k+1} = (1 - \\tau)\\lambda_k$\\\\[3mm]\\lab{eq:red.lam}\n      }\n     {$x_k = x_{k-1}$\\\\[2mm]\\lab{eq:null.step}\n     $\\lambda_{k+1} = \\dfrac{\\lambda_k}{1 - \\tau}$\\\\\\lab{eq:inc.lam}\n     }\n  %\n  }\n\\end{algorithm}","sha256_utf8":"12a2a10d9a18cc399f39b98118920ad2dfd73d57bfabebed4a57d5a201a98c03"}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","title":"排除已精确终止情形","source_units":[{"start":42148,"end":42432,"start_line":989,"end_line":999,"raw_tex":"\\item[(ii)] We mention that if Algorithm \\ref{alg:main} stops at step \\ref{eq:return}, then it follows that $y_{k-1}$ is a solution of\n\\eqref{eq:probm}. So from now on in this paper, we assume that\n%\n%\n\\[\n%\n\\fbox{\nAlgorithm \\ref{alg:main} doesn't stop at step \\ref{eq:return}.\n}\n%\n\\]\n","sha256_utf8":"9a0d043a7e22188e3df635602a3dbd59eac83bb446a55ed9c9abfcc686704d43"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"线性化残差","source_units":[{"start":55435,"end":55612,"start_line":1364,"end_line":1370,"raw_tex":"To simplify the presentation, define for $k \\geq 1$\n%\n\\begin{align}\\lab{eq:def.ek}\n%\n  e_k & := \\lambda_k\\left(F_{y_{k-1}}(y_k) + \\nu_k\\right) + y_k - x_{k-1}\\;.\n%\n\\end{align} \n","sha256_utf8":"d3768841a64216eb6d1c728ded35f34dd29205baffd721e809a99afc0306e666"}]},{"id":"S013","semantic_type":"notation","kind":"notation","title":"大步与空步索引","source_units":[{"start":65300,"end":66064,"start_line":1678,"end_line":1709,"raw_tex":"For every $j\\geq 1$, define\n%\n\\begin{align}\n  \\lab{eq:def.ab}\n  \\begin{aligned}\n    A_j & =\\left\\{1\\leq k \\leq j\\;|\\;\\; \\text{$\\lambda_k\\norm{y_k - x_{k-1}}\\geq \\eta$\\; holds at iteration $k$}\\right\\},\n\t  & a_j =\\# A_j\\,,\\\\\n    B_j & =\\{1\\leq k\\leq j\\;|\\;\\; \n   \\text{$\\lambda_k\\norm{y_k - x_{k-1}} < \\eta$\\; holds at iteration $k$}\\},\n    & b_j =\\# B_j\\,,\n  \\end{aligned}\n\\end{align}\n%\n\\begin{align}\n \\lab{eq:uni.ab}\n A =\\bigcup_{j=1}^\\infty A_j\\;\\;\\mbox{and}\\;\\; B =\\bigcup_{j=1}^\\infty B_j.\n\\end{align}\n%\nWe also define\n%\n\\begin{align}\n\\lab{eq:s70}\n  K=\\set{k\\geq 1\\;|\\; k\\leq \\# A},\\quad\n  i_0=0,\\quad i_k=\\text{$k$-th element of }A,\n\\end{align}\n%\nand note that \n%\n\\begin{align}\n \\lab{eq:car.a}\n  i_0<i_1<i_2<\\cdots,\\qquad A=\\set{i_k\\;|\\; k\\in K}.\n\\end{align}\n","sha256_utf8":"13a87a28178ca981e0a93ad9f6fb7ad6997fb3a96b8218111c1a8b8a462bd62d"}]},{"id":"S014","semantic_type":"theorem","kind":"theorem","title":"主算法逐点与遍历复杂度","source_units":[{"start":72037,"end":73832,"start_line":1915,"end_line":1974,"raw_tex":"\\begin{theorem}[{\\bf First result on the iteration-complexity of Algorithm \\ref{alg:main}}] \\lab{th:main}\n%\nLet $\\rho>0$ be a given tolerance and let $\\hat\\theta>0$, $\\eta>0$, $0<\\tau<1$ and $\\lambda_1>0$ be as in\nthe input of \\emph{Algorithm \\ref{alg:main}}. \nLet also $d_0$ be the distance of $x_0$ to the solution set $(F + N_C)^{-1}(0)\\neq \\emptyset$ of \\eqref{eq:probm}, let\n$L>0$ be as in \\eqref{eq:mon.lip}\nand let $0<\\sigma<1$ be as in \\eqref{eq:def.sigma}.\n%\nThe following holds:\n%\n\\begin{itemize}\n%\n\\item[\\emph{(a)}] \\emph{Algorithm \\ref{alg:main}} finds $y\\in \\HH$ and $\\nu\\in N_C(y)$ such that\n%\n\\begin{align}\\lab{eq:pointw}\n%\n\\norm{F(y) + \\nu}\\leq \\rho\n%\n\\end{align}\n%\nin at most \n%\n\\begin{align}\\lab{eq:def.m}\n%\n \\left\\lceil{\\left(\\dfrac{2}{\\tau \\eta(1-\\sigma)}\\right)\\dfrac{d_0^2}{\\rho}}\\right\\rceil + \n \\left\\lceil{\\dfrac{1}{2\\tau}\\log^+\\left(\\dfrac{\\eta + 2\\hat\\theta/L}{\\lambda_1^2\\rho}\\right)}\\right\\rceil\n%\n\\end{align}\n%\niterations.\n%\n\\item[\\emph{(b)}] \\emph{Algorithm \\ref{alg:main}} finds a triple $(y,v,\\varepsilon)\\in \\HH\\times \\HH\\times \\R_+$  such\nthat\n%\n\\begin{align}\\lab{eq:erg}\n%\nv\\in (F + N_C)^{\\varepsilon}(y)\\;\\;\\emph{and}\\;\\; \\max\\left\\{\\norm{v},\\varepsilon\\right\\}\\leq \\rho\n%\n\\end{align}\n%\nin at most\n%\n\\begin{align}\\lab{eq:def.mt}\n%\n\\begin{aligned}\n& \\max\\left\\{\\left\\lceil{\\left(\\dfrac{2\\sqrt[3]{4}}\n      {\\tau \\eta^{2/3}(1-\\sigma^2)^{1/3}}\\right)\\left(\\dfrac{d_0^2}{\\rho}\\right)^{2/3}}\\right\\rceil, \n      \\left\\lceil{\\left(\\dfrac{2\\sqrt[3]{4}}\n      {\\tau \\eta^{2/3}(1-\\sigma^2)^{2/3}}\\right)\\left(\\dfrac{d_0^3}{\\rho}\\right)^{2/3}}\\right\\rceil\\right\\}\\\\[4mm]\n&\\hspace{3cm}+\\left\\lceil{\\dfrac{1}{2\\tau}\\log^+\\left(\\dfrac{\\eta + 2\\hat\\theta/L}{\\lambda_1^2\\,\\rho}\\right)}\\right\\rceil\n%\n\\end{aligned}\n%\n\\end{align}\n%\niterations.\n%\n\\end{itemize}\n%\n\\end{theorem}","sha256_utf8":"f8ffff1db2d0dfb2fe368a30c7b1a849132077c762d5084fd63c4b288fdcb73c"}]},{"id":"S015","semantic_type":"proof","kind":"proof","title":"复杂度定理完整证明","source_units":[{"start":73835,"end":75745,"start_line":1976,"end_line":2024,"raw_tex":"\\bproo\n%\n(a) Define\n%\n\\begin{align}\\lab{eq:def.m1}\n%\nM_1 = \\left\\lceil{\\left(\\dfrac{1}{\\tau \\eta(1-\\sigma)}\\right)\\dfrac{d_0^2}{\\rho}}\\right\\rceil\\;\\;\\mbox{and}\\;\\; \nM_2 = \\left\\lceil{\\dfrac{1}{2\\tau}\\log^+\\left(\\dfrac{\\eta + 2\\hat\\theta/L}{\\lambda_1^2\\,\\rho}\\right)}\\right\\rceil \n%\n\\end{align}\n%\nand note that, in view of \\eqref{eq:def.m}, we have\n%\n\\begin{align}\\lab{eq:m1m2}\n%\n M = 2M_1 + M_2,\n%\n\\end{align}\n%\nwhere $M$ is defined as the quantity given in \\eqref{eq:def.m}.\n%`\nWe will consider two cases: (i) $\\#A \\geq M_1$ and (ii) $\\#A < M_1$, where $A$ is as in \\eqref{eq:uni.ab}.\nIn the first case, it follows directly from Corollary \\ref{cor:rate}(a) and the definition of $M_1$ as in \\eqref{eq:def.m1} that\nAlgorithm \\ref{alg:main} finds $y\\in \\HH$ satisfying \\eqref{eq:pointw} in at most $M_1$ iterations. Since $M\\geq M_1$, we have the desired result.\n\nConsider now the case (ii), i.e., $\\#A < M_1$. Let $\\ell\\geq 1$ be such that $\\#A = a_\\ell = a_k$ for all $k\\geq \\ell$ (see\n\\eqref{eq:def.ab} and \\eqref{eq:uni.ab}). Hence, if $b_{k-1}\\geq M_1 + M_2$, for some $k\\geq \\ell + 1$, then it follows that\n%\n\\begin{align}\\lab{eq:bma}\n%\nb_{k-1} - a_{k-1} = b_{k-1} - \\#A > b_{k-1} - M_1 \\geq M_2,\n%\n\\end{align}\n%'\nwhere in the latter inequality we also used \nthe assumption that $b_{k-1}\\geq M_1 + M_2$.\n%\n\nTo conclude that proof of (a), note that \\eqref{eq:bma}, Lemma \\ref{lm:geo}, Proposition \\ref{prop:iteb}, the definition of\n$M_2$ as in \\eqref{eq:m1m2} and some simple algebraic manipulation give that $\\norm{F(y_k) + \\nu_k}\\leq \\rho$.\n%\nAltogether, still in the case (ii), we proved that Algorithm \\ref{alg:main} finds $y\\in \\HH$ satisfying \\eqref{eq:pointw} in at most\n$M_1 + (M_1 + M_2)$ iterations, that is, in at most $M$ iterations.\n\n\\mgap\n\n(b) The proof follows the same outline of the proof in item (a), now using Corollary \\ref{cor:rate}(b) instead of Corollary \\ref{cor:rate}(a).\n%\n\\eproo","sha256_utf8":"53380f10cdef558a309ef7f242cdfdab2c9e97e480293ab2d3394eab843e0db5"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"设定两类迭代预算","source_units":[{"start":73835,"end":74312,"start_line":1976,"end_line":1995,"raw_tex":"\\bproo\n%\n(a) Define\n%\n\\begin{align}\\lab{eq:def.m1}\n%\nM_1 = \\left\\lceil{\\left(\\dfrac{1}{\\tau \\eta(1-\\sigma)}\\right)\\dfrac{d_0^2}{\\rho}}\\right\\rceil\\;\\;\\mbox{and}\\;\\; \nM_2 = \\left\\lceil{\\dfrac{1}{2\\tau}\\log^+\\left(\\dfrac{\\eta + 2\\hat\\theta/L}{\\lambda_1^2\\,\\rho}\\right)}\\right\\rceil \n%\n\\end{align}\n%\nand note that, in view of \\eqref{eq:def.m}, we have\n%\n\\begin{align}\\lab{eq:m1m2}\n%\n M = 2M_1 + M_2,\n%\n\\end{align}\n%\nwhere $M$ is defined as the quantity given in \\eqref{eq:def.m}.\n","sha256_utf8":"11d96c8a601ffc5296ac30905865d6f94ad45f154192dd6b688306b298693736"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"大步充足时使用逐点界","source_units":[{"start":74315,"end":74698,"start_line":1997,"end_line":1999,"raw_tex":"We will consider two cases: (i) $\\#A \\geq M_1$ and (ii) $\\#A < M_1$, where $A$ is as in \\eqref{eq:uni.ab}.\nIn the first case, it follows directly from Corollary \\ref{cor:rate}(a) and the definition of $M_1$ as in \\eqref{eq:def.m1} that\nAlgorithm \\ref{alg:main} finds $y\\in \\HH$ satisfying \\eqref{eq:pointw} in at most $M_1$ iterations. Since $M\\geq M_1$, we have the desired result.\n","sha256_utf8":"0aa60f88459a83897c171700231ac8989d9c4d11be389c87fa39f2f0ef434a52"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"大步不足时比较计数","source_units":[{"start":74699,"end":75145,"start_line":2001,"end_line":2011,"raw_tex":"Consider now the case (ii), i.e., $\\#A < M_1$. Let $\\ell\\geq 1$ be such that $\\#A = a_\\ell = a_k$ for all $k\\geq \\ell$ (see\n\\eqref{eq:def.ab} and \\eqref{eq:uni.ab}). Hence, if $b_{k-1}\\geq M_1 + M_2$, for some $k\\geq \\ell + 1$, then it follows that\n%\n\\begin{align}\\lab{eq:bma}\n%\nb_{k-1} - a_{k-1} = b_{k-1} - \\#A > b_{k-1} - M_1 \\geq M_2,\n%\n\\end{align}\n%'\nwhere in the latter inequality we also used \nthe assumption that $b_{k-1}\\geq M_1 + M_2$.\n","sha256_utf8":"8934018c19d3b2d45b7bca6fa085ba3211fefbe5de2331b1611655aee031bd9c"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"空步残差与总预算","source_units":[{"start":75148,"end":75586,"start_line":2014,"end_line":2018,"raw_tex":"To conclude that proof of (a), note that \\eqref{eq:bma}, Lemma \\ref{lm:geo}, Proposition \\ref{prop:iteb}, the definition of\n$M_2$ as in \\eqref{eq:m1m2} and some simple algebraic manipulation give that $\\norm{F(y_k) + \\nu_k}\\leq \\rho$.\n%\nAltogether, still in the case (ii), we proved that Algorithm \\ref{alg:main} finds $y\\in \\HH$ satisfying \\eqref{eq:pointw} in at most\n$M_1 + (M_1 + M_2)$ iterations, that is, in at most $M$ iterations.\n","sha256_utf8":"714c9d51062572f06ab6c9b0e880cc16c8d8bc90939717bc5951f762a6930d06"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"替换为遍历界完成证明","source_units":[{"start":75587,"end":75745,"start_line":2020,"end_line":2024,"raw_tex":"\\mgap\n\n(b) The proof follows the same outline of the proof in item (a), now using Corollary \\ref{cor:rate}(b) instead of Corollary \\ref{cor:rate}(a).\n%\n\\eproo","sha256_utf8":"aa85dd3ac4f134e74d9f152383be197bdbba55846f2e313c4e9bb5490fb7fc89"}]},{"id":"S021","semantic_type":"theorem","kind":"theorem","title":"HPE逐点与遍历速率","source_units":[{"start":34314,"end":35151,"start_line":707,"end_line":734,"raw_tex":"\\begin{theorem} \n \\lab{th:rhpe}\n%\nConsider the sequences evolved by \\emph{Algorithm \\ref{alg:rHPE}} and the ergodic sequences as in \\eqref{eq:def.erg1}--\\eqref{eq:def.erg3}. Let also $d_0$ denote the distance of $x_0$ to the solution set $(F + N_C)^{-1}(0)\\neq \\emptyset$ of \\eqref{eq:probim}. The following hold for all $1\\leq k\\leq N$:\n%\n\\begin{itemize}\n%\n\\item[\\emph{(a)}] There exists $1\\leq j\\leq k$ such that $\\nu_j\\in N_C(y_j)$ and\n%\n\\begin{align*}\n%\n\\norm{F(y_j) + \\nu_j}\\leq \\dfrac{d_0^2}{\\tau \\eta (1-\\sigma)\\,k}\\,.\n%\n\\end{align*}\n%\n\\item[\\emph{(b)}] $v_k^a\\in (F + N_C)^{\\varepsilon_k^a}(y_k^a)$ and\n%\n\\begin{align*}\n%\n\\norm{v_k^a}&\\leq \\dfrac{2d_0^2}{\\tau^{3/2}\\eta\\sqrt{1-\\sigma^2}\\,k^{3/2}}\\,,\\\\[3mm]\n%\n\\varepsilon_k^a&\\leq \\dfrac{2d_0^3}{\\tau^{3/2}\\eta(1-\\sigma^2)\\,k^{3/2}}.\n%\n\\end{align*}\n%\n\\end{itemize}\n%\n\\end{theorem}","sha256_utf8":"9f6dc2d5199c123830a1dca211bf78e25c09dcfea4cbca1902c0a2964ff78d3e"}]},{"id":"S022","semantic_type":"proof","kind":"proof","title":"HPE速率引用证明","source_units":[{"start":35154,"end":35288,"start_line":736,"end_line":741,"raw_tex":"\\bproo\n%\nThe proof follows from \\cite[Theorem 2.4]{pre-print-benar}) applied to the maximal monotone operator \n$T = F + N_C$.\n%\n\\eproo","sha256_utf8":"7b9c4eacec0d7dccd6dc0148ef1b955f4db44f1165604caa876da232b3fd50af"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"应用已有极大单调HPE定理","source_units":[{"start":35154,"end":35288,"start_line":736,"end_line":741,"raw_tex":"\\bproo\n%\nThe proof follows from \\cite[Theorem 2.4]{pre-print-benar}) applied to the maximal monotone operator \n$T = F + N_C$.\n%\n\\eproo","sha256_utf8":"7b9c4eacec0d7dccd6dc0148ef1b955f4db44f1165604caa876da232b3fd50af"}]},{"id":"S024","semantic_type":"proposition","kind":"proposition","title":"牛顿子步位移估计","source_units":[{"start":55622,"end":55925,"start_line":1376,"end_line":1386,"raw_tex":"\\begin{proposition}\\lab{prop:stepb}\n%\nConsider the sequences evolved by \\emph{Algorithm \\ref{alg:main}}. Then, for all $k\\geq 1$,\n%\n\\begin{align*}\n%\n\\norm{y_k - y_{k-1}}\\leq \\dfrac{1}{1-\\hat\\sigma}\\norm{\\lambda_k \\left(F(y_{k-1})+\\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}.\n%\n\\end{align*}\n%\n\\end{proposition}","sha256_utf8":"faa9e85eb3ab54e27e77eb99aa2cd68900ca18f600d18feaffb80cc9038f4b4b"}]},{"id":"S025","semantic_type":"proof","kind":"proof","title":"子步位移完整证明","source_units":[{"start":55928,"end":58128,"start_line":1388,"end_line":1455,"raw_tex":"\\bproo\n%\nFirst note that if $y_k$ is computed as in line \\ref{eq:n.cond} of Algorithm \\ref{alg:main}, then the inequality holds trivially. \n%\nAssume now that $y_k$ is computed as in line \\ref{eq:do.newton} of Algorithm \\ref{alg:main}, so that \\eqref{eq:err.cri} holds.\n%\n%\nIn this case, note that from Algorithm \\ref{alg:main}'s definition we have that $\\nu_k\\in N_C(y_k)$ for all $k\\geq 0$. Hence, in view of the monotonicity of $N_C$, we obtain\n%\n\\begin{align}\\lab{eq:lego}\n%\n\\inner{\\nu_k - \\nu_{k-1}}{y_k - y_{k-1}}\\geq 0\\qquad \\forall k\\geq 1.\n%\n\\end{align}\n%\nNow by multiplying the inequality in \\eqref{eq:lego} by $\\lambda_k>0$ \nand using \\eqref{eq:def.ek}\nwe have\n%\n\\begin{align}\\lab{eq:mouse}\n%\n\\inner{e_k + x_{k-1} - y_k - \\lambda_k F_{y_{k-1}}(y_k) - \\lambda_k\\nu_{k-1}}{y_k - y_{k-1}}\\geq 0.\n%\n\\end{align}\n\nNote also that using \\eqref{eq:def.ek} and the inequality in \\eqref{eq:err.cri} we also find\n%\n\\begin{align}\\lab{eq:lego3}\n%\n\\norm{e_k}\\leq \\hat\\sigma \\norm{y_k - y_{k-1}}\n\\qquad \\forall k\\geq 1.\n%\n\\end{align}\n%\nAfter some simple algebraic manipulations with \\eqref{eq:mouse} we obtain\n%\n\\begin{align}\\lab{eq:lego2}\n%\n\\nonumber\n\\normq{y_k - y_{k-1}}&\\leq \\inner{e_k + x_{k-1} - y_{k-1}- \\lambda_k F_{y_{k-1}}(y_k) - \\lambda_k\\nu_{k-1}}{y_k - y_{k-1}}\\\\[2mm]\n%\n\\nonumber\n&=\\inner{e_k + x_{k-1} - y_{k-1}- \\lambda_k (F(y_{k-1})+\\nu_{k-1})}{y_k - y_{k-1}}\\\\[2mm]\n%\n\\nonumber\n&\\hspace{2cm}- \\lambda_k \\inner{F'(y_{k-1})(y_k - y_{k-1})}{y_k - y_{k-1}}\n\\\\[2mm]\n%\n\\nonumber\n&\\leq \\inner{e_k + x_{k-1} - y_{k-1}- \\lambda_k (F(y_{k-1})+\\nu_{k-1})}{y_k - y_{k-1}}\\\\[2mm]\n%\n&\\leq \\big(\\norm{e_k} + \\norm{x_{k-1} - y_{k-1}- \\lambda_k (F(y_{k-1})+\\nu_{k-1})}\\big)\\norm{y_k - y_{k-1}},\n%\n\\end{align}\n%\nwhere we also used the definition of $F_{y_{k-1}}(y_k)$ -- see \\eqref{eq:def.lin} -- and the fact that $F'(y_{k-1})\\geq 0$ (see, e.g., \\cite[Proposition 12.3]{roc.wet-book}).\n\nTo finish the proof of the proposition, note that from \\eqref{eq:lego3} and \\eqref{eq:lego2} we find\n%\n\\begin{align*}\n%\n\\norm{y_k - y_{k-1}}\\leq \\hat\\sigma\\norm{y_k - y_{k-1}} + \\norm{x_{k-1} - y_{k-1}- \\lambda_k (F(y_{k-1})+\\nu_{k-1})},\n%\n\\end{align*}\n%\nwhich is clearly equivalent to the desired inequality.\n%\n\\eproo","sha256_utf8":"f9365e3a151bb47378fb410917bb2bef2425e0795bfe9b76b32cc057a744f4c7"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"区分复用与牛顿子步","source_units":[{"start":55928,"end":56197,"start_line":1388,"end_line":1392,"raw_tex":"\\bproo\n%\nFirst note that if $y_k$ is computed as in line \\ref{eq:n.cond} of Algorithm \\ref{alg:main}, then the inequality holds trivially. \n%\nAssume now that $y_k$ is computed as in line \\ref{eq:do.newton} of Algorithm \\ref{alg:main}, so that \\eqref{eq:err.cri} holds.\n","sha256_utf8":"423e7a8d1e3e28c5e26db1d3d98aeb138b9e31a22761d6156d9a516f8cdca5e4"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"法锥单调性给出内积界","source_units":[{"start":56201,"end":56745,"start_line":1395,"end_line":1411,"raw_tex":"In this case, note that from Algorithm \\ref{alg:main}'s definition we have that $\\nu_k\\in N_C(y_k)$ for all $k\\geq 0$. Hence, in view of the monotonicity of $N_C$, we obtain\n%\n\\begin{align}\\lab{eq:lego}\n%\n\\inner{\\nu_k - \\nu_{k-1}}{y_k - y_{k-1}}\\geq 0\\qquad \\forall k\\geq 1.\n%\n\\end{align}\n%\nNow by multiplying the inequality in \\eqref{eq:lego} by $\\lambda_k>0$ \nand using \\eqref{eq:def.ek}\nwe have\n%\n\\begin{align}\\lab{eq:mouse}\n%\n\\inner{e_k + x_{k-1} - y_k - \\lambda_k F_{y_{k-1}}(y_k) - \\lambda_k\\nu_{k-1}}{y_k - y_{k-1}}\\geq 0.\n%\n\\end{align}\n","sha256_utf8":"b260edff27065822b805ea918defd8ced62b9f098a3178632846095dc1bc2075"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"控制线性化误差","source_units":[{"start":56746,"end":56956,"start_line":1413,"end_line":1420,"raw_tex":"Note also that using \\eqref{eq:def.ek} and the inequality in \\eqref{eq:err.cri} we also find\n%\n\\begin{align}\\lab{eq:lego3}\n%\n\\norm{e_k}\\leq \\hat\\sigma \\norm{y_k - y_{k-1}}\n\\qquad \\forall k\\geq 1.\n%\n\\end{align}\n","sha256_utf8":"43644b019cb6d515d0868fac21afa4e57a89c43b4d02c31f17e8def63786c5b3"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"利用导数非负性估计位移","source_units":[{"start":56958,"end":57809,"start_line":1422,"end_line":1443,"raw_tex":"After some simple algebraic manipulations with \\eqref{eq:mouse} we obtain\n%\n\\begin{align}\\lab{eq:lego2}\n%\n\\nonumber\n\\normq{y_k - y_{k-1}}&\\leq \\inner{e_k + x_{k-1} - y_{k-1}- \\lambda_k F_{y_{k-1}}(y_k) - \\lambda_k\\nu_{k-1}}{y_k - y_{k-1}}\\\\[2mm]\n%\n\\nonumber\n&=\\inner{e_k + x_{k-1} - y_{k-1}- \\lambda_k (F(y_{k-1})+\\nu_{k-1})}{y_k - y_{k-1}}\\\\[2mm]\n%\n\\nonumber\n&\\hspace{2cm}- \\lambda_k \\inner{F'(y_{k-1})(y_k - y_{k-1})}{y_k - y_{k-1}}\n\\\\[2mm]\n%\n\\nonumber\n&\\leq \\inner{e_k + x_{k-1} - y_{k-1}- \\lambda_k (F(y_{k-1})+\\nu_{k-1})}{y_k - y_{k-1}}\\\\[2mm]\n%\n&\\leq \\big(\\norm{e_k} + \\norm{x_{k-1} - y_{k-1}- \\lambda_k (F(y_{k-1})+\\nu_{k-1})}\\big)\\norm{y_k - y_{k-1}},\n%\n\\end{align}\n%\nwhere we also used the definition of $F_{y_{k-1}}(y_k)$ -- see \\eqref{eq:def.lin} -- and the fact that $F'(y_{k-1})\\geq 0$ (see, e.g., \\cite[Proposition 12.3]{roc.wet-book}).\n","sha256_utf8":"2476d7abadab58acf4e6303e0f78b418e56f77a59a73de5f14bad950227cae2c"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"吸收相对误差项","source_units":[{"start":57810,"end":58128,"start_line":1445,"end_line":1455,"raw_tex":"To finish the proof of the proposition, note that from \\eqref{eq:lego3} and \\eqref{eq:lego2} we find\n%\n\\begin{align*}\n%\n\\norm{y_k - y_{k-1}}\\leq \\hat\\sigma\\norm{y_k - y_{k-1}} + \\norm{x_{k-1} - y_{k-1}- \\lambda_k (F(y_{k-1})+\\nu_{k-1})},\n%\n\\end{align*}\n%\nwhich is clearly equivalent to the desired inequality.\n%\n\\eproo","sha256_utf8":"b1a75086c6aeb951f45a239ce9be6a5e524cd84c11e10797b5d5f834a3af1238"}]},{"id":"S031","semantic_type":"proposition","kind":"proposition","title":"牛顿子步改进残差","source_units":[{"start":58139,"end":58723,"start_line":1461,"end_line":1475,"raw_tex":"\\bprop \\lab{prop:imp}\n%\nConsider the sequences evolved by \\emph{Algorithm \\ref{alg:main}}, let $\\hat\\theta>0$ be as in \\eqref{eq:def.hth} and assume that at iteration $k\\ge1$, $y_k$ and $\\nu_k$ are computed as in line \\ref{eq:do.newton} of \\emph{Algorithm \\ref{alg:main}}, so that  \\eqref{eq:err.cri} holds. Then:\n%\n\\begin{align*}\n%\n\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_{k-1}) + \\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}\\leq \n\\theta\\;\\;\\Rightarrow\\;\\;\n%\n\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\leq \n\\hat\\theta.\n%\n\\end{align*}\n%\n\\eprop","sha256_utf8":"28e57fd29480869dd0371303607e669d8905651195420d27ef2dc3abb6db45d4"}]},{"id":"S032","semantic_type":"proof","kind":"proof","title":"残差改进完整证明","source_units":[{"start":58726,"end":60395,"start_line":1477,"end_line":1528,"raw_tex":"\\bproo\n%\nAssume that $y_k$ is computed as in line \\ref{eq:do.newton} of Algorithm \\ref{alg:main} and\n%\n\\begin{align}\\lab{eq:coffee}\n%\n\\frac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_{k-1}) + \\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}\\leq \n\\theta.\n%\n\\end{align}  \nLetting $e_k$ be as in \\eqref{eq:def.ek}\n%\nand using its definition, some simple algebraic manipulations \nand the inequality in \\eqref{eq:err.cri}\n we find\n%\n\\begin{align*}\n%\n\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}} &= \n               \\norm{e_k + \\lambda_k\\left(F(y_k) - F_{y_{k-1}}(y_k)\\right)}\\\\[2mm]\n               &\\leq \\norm{e_k} + \\lambda_k\\norm{F(y_k) - F_{y_{k-1}}(y_k)}\\\\[2mm]\n               &\\leq \\hat\\sigma \\norm{y_k - y_{k-1}} + \\dfrac{\\lambda_k L}{2}\\normq{y_k - y_{k-1}},\n%\n\\end{align*}\n%\nwhere in the second inequality we also used Lemma \\ref{lm:lip} below. \n\nAs a consequence, from Proposition \\ref{prop:stepb},\n%\n\\begin{align*}\n%\n\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}} &\\leq \n\\dfrac{\\hat\\sigma}{1-\\hat\\sigma}\\norm{\\lambda_k \\left(F(y_{k-1}) + \\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}\\\\[2mm]\n& \\hspace{2cm}+\\dfrac{1}{(1-\\hat\\sigma)^2}\\dfrac{\\lambda_k L}{2}\n\\norm{\\lambda_k \\left(F(y_{k-1}) + \\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}^2.\n%\n\\end{align*}\n%\nBy multiplying the latter inequality by $\\frac{\\lambda_k L}{2}$, using the assumption \n\\eqref{eq:coffee} and \\eqref{eq:def.hth} we obtain\n%\n\\begin{align*}\n%\n\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k F(y_k) + y_k - x_{k-1}} &\\leq \\theta\\left(\\dfrac{\\hat\\sigma}{1-\\hat\\sigma}\n+ \\dfrac{\\theta}{(1-\\hat\\sigma)^2}\\right)\\\\[2mm]\n & = \\hat \\theta,\n%\n\\end{align*}\n%\nwhich finishes the proof of the proposition.\n%\n\\eproo","sha256_utf8":"a23db971a0dd2eee2d1c1ef980896f6288687005ee7981119947828a47f210a7"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"引入当前残差假设","source_units":[{"start":58726,"end":58983,"start_line":1477,"end_line":1486,"raw_tex":"\\bproo\n%\nAssume that $y_k$ is computed as in line \\ref{eq:do.newton} of Algorithm \\ref{alg:main} and\n%\n\\begin{align}\\lab{eq:coffee}\n%\n\\frac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_{k-1}) + \\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}\\leq \n\\theta.\n%\n\\end{align}  \n","sha256_utf8":"1e1e76dc2c43c1c23ed883568c5ab040ca84ef8840edd0fd26cb20082c523632"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"泰勒余项控制真实残差","source_units":[{"start":58983,"end":59577,"start_line":1487,"end_line":1502,"raw_tex":"Letting $e_k$ be as in \\eqref{eq:def.ek}\n%\nand using its definition, some simple algebraic manipulations \nand the inequality in \\eqref{eq:err.cri}\n we find\n%\n\\begin{align*}\n%\n\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}} &= \n               \\norm{e_k + \\lambda_k\\left(F(y_k) - F_{y_{k-1}}(y_k)\\right)}\\\\[2mm]\n               &\\leq \\norm{e_k} + \\lambda_k\\norm{F(y_k) - F_{y_{k-1}}(y_k)}\\\\[2mm]\n               &\\leq \\hat\\sigma \\norm{y_k - y_{k-1}} + \\dfrac{\\lambda_k L}{2}\\normq{y_k - y_{k-1}},\n%\n\\end{align*}\n%\nwhere in the second inequality we also used Lemma \\ref{lm:lip} below. \n","sha256_utf8":"411d3052d36fe17392f2e248a0f2d1523e04c88942be0fc0b8913ea150360930"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"代入子步位移界","source_units":[{"start":59578,"end":59986,"start_line":1504,"end_line":1513,"raw_tex":"As a consequence, from Proposition \\ref{prop:stepb},\n%\n\\begin{align*}\n%\n\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}} &\\leq \n\\dfrac{\\hat\\sigma}{1-\\hat\\sigma}\\norm{\\lambda_k \\left(F(y_{k-1}) + \\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}\\\\[2mm]\n& \\hspace{2cm}+\\dfrac{1}{(1-\\hat\\sigma)^2}\\dfrac{\\lambda_k L}{2}\n\\norm{\\lambda_k \\left(F(y_{k-1}) + \\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}^2.\n%\n\\end{align*}\n","sha256_utf8":"c81abbfecd1ec95bc2f422665bbb5b702d5ba25a08a5effeed7a7b220a9dad1b"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"缩放并识别改进阈值","source_units":[{"start":59988,"end":60395,"start_line":1515,"end_line":1528,"raw_tex":"By multiplying the latter inequality by $\\frac{\\lambda_k L}{2}$, using the assumption \n\\eqref{eq:coffee} and \\eqref{eq:def.hth} we obtain\n%\n\\begin{align*}\n%\n\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k F(y_k) + y_k - x_{k-1}} &\\leq \\theta\\left(\\dfrac{\\hat\\sigma}{1-\\hat\\sigma}\n+ \\dfrac{\\theta}{(1-\\hat\\sigma)^2}\\right)\\\\[2mm]\n & = \\hat \\theta,\n%\n\\end{align*}\n%\nwhich finishes the proof of the proposition.\n%\n\\eproo","sha256_utf8":"e2bbd7bbc6ce072f1af32bd8ff8b8aafb446ecb8548e2d9cf93101a338bab266"}]},{"id":"S037","semantic_type":"proposition","kind":"proposition","title":"全迭代残差不变量","source_units":[{"start":60404,"end":60981,"start_line":1532,"end_line":1558,"raw_tex":"\\bprop \\lab{prop:inc.theta}\n%\nConsider the sequences evolved by \\emph{Algorithm \\ref{alg:main}} and let $\\hat\\theta$ be as in \\eqref{eq:def.hth}. The following holds:\n%\n\\begin{itemize}\n%\n\\item[\\emph{(a)}] For all $k\\geq 1$,\n%\n\\begin{align}\\lab{eq:inc.theta}\n%\n\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_{k-1}) + \\nu_{k-1}\\right) + y_{k-1} - x_{k-1}}\\leq \n\\theta.\n%\n\\end{align}\n%\n\\item[\\emph{(b)}] For all $k\\geq 1$,\n%\n\\begin{align}\n%\n\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\leq \n\\hat\\theta.\n%\n\\end{align}\n%\n\\end{itemize}\n%\n\\eprop","sha256_utf8":"8e13cd61b69a4e43674dae7bbfc48ba1fe06caedb1e9c226ad690d18f3f4d2d4"}]},{"id":"S038","semantic_type":"proof","kind":"proof","title":"残差不变量完整证明","source_units":[{"start":60984,"end":65285,"start_line":1560,"end_line":1673,"raw_tex":"\\bproo \n%\n(a) Let us proceed by induction on $k\\geq 1$. Note first that \\eqref{eq:inc.theta} is true for $k=1$ in view of \nthe facts that $y_0=x_0$, $\\nu_0 =0$ and $\\frac{\\lambda_1 L}{2}\\norm{\\lambda_1 F(y_0)}\\leq \\theta$ (see the\ninput in Algorithm \\ref{alg:main}).\n%\nAssume now that \\eqref{eq:inc.theta} holds for some $k\\geq 1$. \n%\nWe claim that \n%\n\\begin{align}\\lab{eq:sophia}\n%\n\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\leq \\hat\\theta.\n%\n\\end{align}\n%\nIndeed, if $y_k$ and $\\nu_k$ are given as in lines \\ref{eq:n.cond} and \\ref{eq:n.cond02} of Algorithm \\ref{alg:main}, then we \nreadily obtain \\eqref{eq:sophia}. Otherwise, \\eqref{eq:sophia} follows from the induction hypothesis and Proposition \\ref{prop:imp}.\n\nNext we will consider two cases: $\\lambda_k\\norm{y_k - x_{k-1}}\\geq \\eta$ and\n$\\lambda_k\\norm{y_k - x_{k-1}} < \\eta$. In the first case, using lines \\ref{eq:ext.step} and \\ref{eq:red.lam} of Algorithm \\ref{alg:main} and \\eqref{eq:sophia}, we find\n\n\\begin{align}\\lab{eq:sophia2}\n%\n\\nonumber\n\\dfrac{\\lambda_{k+1} L}{2}\\norm{\\lambda_{k+1} \\left(F(y_k) + \\nu_k\\right) + y_k - x_k} \n%\n\\nonumber\n& = \\dfrac{\\lambda_{k+1} L}{2}\\norm{\\lambda_{k+1} \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1} + x_{k-1} - x_k}\\\\[2mm]\n% \n\\nonumber\n& =\\dfrac{\\lambda_{k+1} L}{2}\\norm{\\lambda_{k+1} \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1} + \\tau\\lambda_k\n   \\left(F(y_k) + \\nu_k\\right)}\\\\[2mm]\n%\n\\nonumber\n& = \\dfrac{\\lambda_{k+1} L}{2}\\norm{\\left(\\lambda_{k+1} + \\tau\\lambda_k\\right) \\left(F(y_k) + \\nu_k\\right) \n+ y_k - x_{k-1}}\\\\[2mm]\n%\n\\nonumber\n& = (1-\\tau)\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\\\[2mm]\n%\n\\nonumber\n& \\leq (1 - \\tau)\\hat \\theta\\\\[2mm]\n%\n&\\leq \\theta,\n %\n\\end{align}\n%\nwhere the latter inequality comes from the fact that $0<\\tau<1$ and \n$\\hat\\theta < \\theta$ (because of the assumption that $0<\\theta<(1-\\hat\\sigma)(1 - 2\\hat\\sigma)$). \n%\nNote that \\eqref{eq:sophia2} finishes the induction argument in the case $\\lambda_k\\norm{y_k - x_{k-1}}\\geq \\eta$.\n\nLet us now consider the case $\\lambda_k\\norm{y_k - x_{k-1}} < \\eta$. Note first that, using the latter inequality, lines \\ref{eq:null.step} and \\ref{eq:inc.lam} of Algorithm \\ref{alg:main} and \\eqref{eq:sophia} combined with the induction hypothesis, we find\n%\n\\begin{align*}\n%\n&\\dfrac{\\lambda_{k+1} L}{2}\\norm{\\lambda_{k+1} \\left(F(y_k) + \\nu_k\\right) + y_k - x_k}\\\\[3mm] \n%\n = &\\dfrac{\\lambda_{k+1} L}{2}\\norm{\\lambda_{k+1} \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\\\[2mm]\n%\n = &\\dfrac{\\lambda_{k+1} L}{2}\\Big\\|\\dfrac{\\lambda_{k+1}}{\\lambda_k}\\big[\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}\\big] \\\\[2mm]\n% \n&\\hspace{2cm} + \\left(1 - \\dfrac{\\lambda_{k+1}}{\\lambda_k}\\right)(y_k - x_{k-1})\\Big\\|\\\\[2mm]\n%\n\\leq &\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k}\\right)^2\\dfrac{\\lambda_k L}{2}\n \\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\\\[2mm] \n%\n&\\hspace{2cm}+ \\dfrac{\\lambda_{k+1} L}{2}\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k} - 1\\right)\\norm{y_k - x_{k-1}}\\\\[2mm]\n%\n< &\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k}\\right)^2 \\hat \\theta + \\dfrac{\\lambda_{k+1} L}{2}\n\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k} - 1\\right)\n\\dfrac{\\eta}{\\lambda_k}\\\\[2mm]\n%\n= &\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k}\\right)^2 \\hat \\theta + \\dfrac{\\lambda_{k+1}}{\\lambda_k}\n\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k} - 1\\right)\n\\dfrac{\\eta L}{2}\\\\[2mm]\n%\n = &\\dfrac{\\hat\\theta}{(1 - \\tau)^2} + \\dfrac{\\tau}{(1-\\tau)^2}\\dfrac{\\eta L}{2}.\n %\n\\end{align*}\n%\nAs a consequence, to complete the induction argument, it is sufficient to show that\n%\n\\begin{align*}\n%\n\\dfrac{\\hat\\theta}{(1 - \\tau)^2} + \\dfrac{\\tau}{(1-\\tau)^2}\\dfrac{\\eta L}{2} = \\theta,\n%\n\\end{align*}\n%\nor, equivalently,\n%\n\\begin{align*}\n%\n(1-\\tau)^2\\theta - \\dfrac{\\eta L}{2}\\tau - \\hat \\theta = 0.\n%\n\\end{align*}\n%\nNote that the latter identity is clearly equivalent to\n%\n\\begin{align*}\n%\n\\theta \\tau^2 - \\left(2\\theta + \\dfrac{\\eta L}{2}\\right)\\tau + \\theta -\\hat\\theta = 0.\n%\n\\end{align*}\n%\nHence, the result follows from \\eqref{eq:def.q} and \\eqref{eq:root.q}. This completes the induction argument and finishes the proof of item (a).\n\n\\mgap\n\n(b) This is a direct consequence of item (a), Proposition \\ref{prop:imp} and lines \\ref{eq:n.cond} and \\ref{eq:n.cond02}\nof Algorithm \\ref{alg:main}.\n%\n\\eproo","sha256_utf8":"2224e74f98e6ab0d0055cb38990562fdf84c478d314530550d213e05439b093e"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"验证归纳初始条件","source_units":[{"start":60984,"end":61251,"start_line":1560,"end_line":1564,"raw_tex":"\\bproo \n%\n(a) Let us proceed by induction on $k\\geq 1$. Note first that \\eqref{eq:inc.theta} is true for $k=1$ in view of \nthe facts that $y_0=x_0$, $\\nu_0 =0$ and $\\frac{\\lambda_1 L}{2}\\norm{\\lambda_1 F(y_0)}\\leq \\theta$ (see the\ninput in Algorithm \\ref{alg:main}).\n","sha256_utf8":"8f0ff3f283863af07ae8dcb22a65f78f3ad2dd500fdb9bdaed53f736f29c2ebf"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"归纳假设推出当前改进界","source_units":[{"start":61253,"end":61742,"start_line":1566,"end_line":1577,"raw_tex":"Assume now that \\eqref{eq:inc.theta} holds for some $k\\geq 1$. \n%\nWe claim that \n%\n\\begin{align}\\lab{eq:sophia}\n%\n\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\leq \\hat\\theta.\n%\n\\end{align}\n%\nIndeed, if $y_k$ and $\\nu_k$ are given as in lines \\ref{eq:n.cond} and \\ref{eq:n.cond02} of Algorithm \\ref{alg:main}, then we \nreadily obtain \\eqref{eq:sophia}. Otherwise, \\eqref{eq:sophia} follows from the induction hypothesis and Proposition \\ref{prop:imp}.\n","sha256_utf8":"26fc898d69bbec9e5189311bc3bd160941af6af5469d4f2921919f2be54f4f80"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"大步更新保持不变量","source_units":[{"start":61743,"end":63037,"start_line":1579,"end_line":1611,"raw_tex":"Next we will consider two cases: $\\lambda_k\\norm{y_k - x_{k-1}}\\geq \\eta$ and\n$\\lambda_k\\norm{y_k - x_{k-1}} < \\eta$. In the first case, using lines \\ref{eq:ext.step} and \\ref{eq:red.lam} of Algorithm \\ref{alg:main} and \\eqref{eq:sophia}, we find\n\n\\begin{align}\\lab{eq:sophia2}\n%\n\\nonumber\n\\dfrac{\\lambda_{k+1} L}{2}\\norm{\\lambda_{k+1} \\left(F(y_k) + \\nu_k\\right) + y_k - x_k} \n%\n\\nonumber\n& = \\dfrac{\\lambda_{k+1} L}{2}\\norm{\\lambda_{k+1} \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1} + x_{k-1} - x_k}\\\\[2mm]\n% \n\\nonumber\n& =\\dfrac{\\lambda_{k+1} L}{2}\\norm{\\lambda_{k+1} \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1} + \\tau\\lambda_k\n   \\left(F(y_k) + \\nu_k\\right)}\\\\[2mm]\n%\n\\nonumber\n& = \\dfrac{\\lambda_{k+1} L}{2}\\norm{\\left(\\lambda_{k+1} + \\tau\\lambda_k\\right) \\left(F(y_k) + \\nu_k\\right) \n+ y_k - x_{k-1}}\\\\[2mm]\n%\n\\nonumber\n& = (1-\\tau)\\dfrac{\\lambda_k L}{2}\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\\\[2mm]\n%\n\\nonumber\n& \\leq (1 - \\tau)\\hat \\theta\\\\[2mm]\n%\n&\\leq \\theta,\n %\n\\end{align}\n%\nwhere the latter inequality comes from the fact that $0<\\tau<1$ and \n$\\hat\\theta < \\theta$ (because of the assumption that $0<\\theta<(1-\\hat\\sigma)(1 - 2\\hat\\sigma)$). \n%\nNote that \\eqref{eq:sophia2} finishes the induction argument in the case $\\lambda_k\\norm{y_k - x_{k-1}}\\geq \\eta$.\n","sha256_utf8":"99eadbcb1e980d2d74d553f391982e549244d6bc4b1cebf19000a3b048d518b0"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"空步更新的残差估计","source_units":[{"start":63038,"end":64473,"start_line":1613,"end_line":1640,"raw_tex":"Let us now consider the case $\\lambda_k\\norm{y_k - x_{k-1}} < \\eta$. Note first that, using the latter inequality, lines \\ref{eq:null.step} and \\ref{eq:inc.lam} of Algorithm \\ref{alg:main} and \\eqref{eq:sophia} combined with the induction hypothesis, we find\n%\n\\begin{align*}\n%\n&\\dfrac{\\lambda_{k+1} L}{2}\\norm{\\lambda_{k+1} \\left(F(y_k) + \\nu_k\\right) + y_k - x_k}\\\\[3mm] \n%\n = &\\dfrac{\\lambda_{k+1} L}{2}\\norm{\\lambda_{k+1} \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\\\[2mm]\n%\n = &\\dfrac{\\lambda_{k+1} L}{2}\\Big\\|\\dfrac{\\lambda_{k+1}}{\\lambda_k}\\big[\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}\\big] \\\\[2mm]\n% \n&\\hspace{2cm} + \\left(1 - \\dfrac{\\lambda_{k+1}}{\\lambda_k}\\right)(y_k - x_{k-1})\\Big\\|\\\\[2mm]\n%\n\\leq &\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k}\\right)^2\\dfrac{\\lambda_k L}{2}\n \\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}\\\\[2mm] \n%\n&\\hspace{2cm}+ \\dfrac{\\lambda_{k+1} L}{2}\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k} - 1\\right)\\norm{y_k - x_{k-1}}\\\\[2mm]\n%\n< &\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k}\\right)^2 \\hat \\theta + \\dfrac{\\lambda_{k+1} L}{2}\n\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k} - 1\\right)\n\\dfrac{\\eta}{\\lambda_k}\\\\[2mm]\n%\n= &\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k}\\right)^2 \\hat \\theta + \\dfrac{\\lambda_{k+1}}{\\lambda_k}\n\\left(\\dfrac{\\lambda_{k+1}}{\\lambda_k} - 1\\right)\n\\dfrac{\\eta L}{2}\\\\[2mm]\n%\n = &\\dfrac{\\hat\\theta}{(1 - \\tau)^2} + \\dfrac{\\tau}{(1-\\tau)^2}\\dfrac{\\eta L}{2}.\n %\n\\end{align*}\n","sha256_utf8":"e935759922401fef589a1ce762c28f71044d7e789bfd78697317daa9d8ba7a41"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"二次方程闭合归纳","source_units":[{"start":64475,"end":65127,"start_line":1642,"end_line":1669,"raw_tex":"As a consequence, to complete the induction argument, it is sufficient to show that\n%\n\\begin{align*}\n%\n\\dfrac{\\hat\\theta}{(1 - \\tau)^2} + \\dfrac{\\tau}{(1-\\tau)^2}\\dfrac{\\eta L}{2} = \\theta,\n%\n\\end{align*}\n%\nor, equivalently,\n%\n\\begin{align*}\n%\n(1-\\tau)^2\\theta - \\dfrac{\\eta L}{2}\\tau - \\hat \\theta = 0.\n%\n\\end{align*}\n%\nNote that the latter identity is clearly equivalent to\n%\n\\begin{align*}\n%\n\\theta \\tau^2 - \\left(2\\theta + \\dfrac{\\eta L}{2}\\right)\\tau + \\theta -\\hat\\theta = 0.\n%\n\\end{align*}\n%\nHence, the result follows from \\eqref{eq:def.q} and \\eqref{eq:root.q}. This completes the induction argument and finishes the proof of item (a).\n\n\\mgap\n\n","sha256_utf8":"9a70d00423e1c359faca1abc869e87d6c23caf275f4e80f40044aca1e7ed360c"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"推出第二个残差不变量","source_units":[{"start":65127,"end":65285,"start_line":1670,"end_line":1673,"raw_tex":"(b) This is a direct consequence of item (a), Proposition \\ref{prop:imp} and lines \\ref{eq:n.cond} and \\ref{eq:n.cond02}\nof Algorithm \\ref{alg:main}.\n%\n\\eproo","sha256_utf8":"1b2f9824163e135f3883528bb107451ca83f62eef97b61fb65c6ec642e381ef0"}]},{"id":"S045","semantic_type":"proof","kind":"proof","title":"参数性质完整论证","source_units":[{"start":44595,"end":45559,"start_line":1038,"end_line":1076,"raw_tex":"\\brema \\lab{rem:theta}\n%\n\\emph{We mention that\n%\n\\begin{align}\\lab{eq:tau.ineq}\n%\n  0<\\hat\\theta<\\theta\\;\\;\\;\\mbox{and}\\;\\;\\;0<\\dfrac{\\theta -\\hat\\theta}{2\\theta + \\dfrac{\\eta L}{2}}<\\tau<1.\n%\n\\end{align}\n%\nThe fact that $\\theta>\\hat \\theta$ is a direct consequence of the assumption $0<\\theta<(1-\\hat\\sigma)(1-2\\hat\\sigma)$\n(see the input in Algorithm \\ref{alg:main}) and \\eqref{eq:def.hth}. On the other hand, by defining the quadratic function \n$q:\\R\\to \\R$,\n%\n\\begin{align}\\lab{eq:def.q}\n%\nq(t) := \\theta t^2 - \\left(2\\theta + \\dfrac{\\eta L}{2}\\right)t + \\theta -\\hat\\theta,\\qquad t\\in \\R,\n%\n\\end{align}\n%\nwe see that $q(0) = \\theta -\\hat\\theta >0$ and $q(1) = -\\frac{\\eta L}{2} - \\hat\\theta<0$. Then, since $\\tau$ is clearly the smallest root of $q(\\cdot)$, it follows that\n%\n\\begin{align}\\lab{eq:root.q}\n%\n q(\\tau) =0\n%\n\\end{align}\n%\nand\n%\n\\begin{align*}\n%\n-\\dfrac{q(0)}{q'(0)}<\\tau< 1,\n%\n\\end{align*}\n%\nwhich is equivalent to \\eqref{eq:tau.ineq}.}\n%\n\\erema\n","sha256_utf8":"bd3d4a6d8387354f9309b2397f8f1144ee51df085089501610b372d2782e51a0"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"参数界与阈值严格次序","source_units":[{"start":44595,"end":45043,"start_line":1038,"end_line":1049,"raw_tex":"\\brema \\lab{rem:theta}\n%\n\\emph{We mention that\n%\n\\begin{align}\\lab{eq:tau.ineq}\n%\n  0<\\hat\\theta<\\theta\\;\\;\\;\\mbox{and}\\;\\;\\;0<\\dfrac{\\theta -\\hat\\theta}{2\\theta + \\dfrac{\\eta L}{2}}<\\tau<1.\n%\n\\end{align}\n%\nThe fact that $\\theta>\\hat \\theta$ is a direct consequence of the assumption $0<\\theta<(1-\\hat\\sigma)(1-2\\hat\\sigma)$\n(see the input in Algorithm \\ref{alg:main}) and \\eqref{eq:def.hth}. On the other hand, by defining the quadratic function \n","sha256_utf8":"7464d57d04c196241bfafda585f252d40aee0613317dc44017aa4c390645b7d7"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"构造多项式并确认根","source_units":[{"start":45043,"end":45433,"start_line":1050,"end_line":1064,"raw_tex":"$q:\\R\\to \\R$,\n%\n\\begin{align}\\lab{eq:def.q}\n%\nq(t) := \\theta t^2 - \\left(2\\theta + \\dfrac{\\eta L}{2}\\right)t + \\theta -\\hat\\theta,\\qquad t\\in \\R,\n%\n\\end{align}\n%\nwe see that $q(0) = \\theta -\\hat\\theta >0$ and $q(1) = -\\frac{\\eta L}{2} - \\hat\\theta<0$. Then, since $\\tau$ is clearly the smallest root of $q(\\cdot)$, it follows that\n%\n\\begin{align}\\lab{eq:root.q}\n%\n q(\\tau) =0\n%\n\\end{align}\n","sha256_utf8":"3c855907cf4b33d0f6ca153e44905de1559916d01961ef01b886bc2c356d477a"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"切线界给出松弛参数范围","source_units":[{"start":45435,"end":45559,"start_line":1066,"end_line":1076,"raw_tex":"and\n%\n\\begin{align*}\n%\n-\\dfrac{q(0)}{q'(0)}<\\tau< 1,\n%\n\\end{align*}\n%\nwhich is equivalent to \\eqref{eq:tau.ineq}.}\n%\n\\erema\n","sha256_utf8":"191d739ff8a5760ee831966bfb59de4260f608f6b47da749654db02f9110b217"}]},{"id":"S049","semantic_type":"proof","kind":"proof","title":"大步间状态恒等式证明","source_units":[{"start":66074,"end":66722,"start_line":1714,"end_line":1734,"raw_tex":"\\brema \\lab{rm:rm}\n%\n\\emph{\nWe claim that\nif $(x_k)$ is generated by Algorithm \\ref{alg:main} then\n%\n\\begin{align}\n\\lab{eq:ixk}\n x_{i_k-1}=x_{i_{k-1}}\\qquad \\forall k\\in K,\n\\end{align}\n%\nwhere $K$ is as in \\eqref{eq:s70} -- see also \\eqref{eq:car.a}.\n%\nIndeed, for any $k\\in K$, by \\eqref{eq:uni.ab} and \\eqref{eq:car.a} we have \n$\\{k\\geq 1\\,|\\, i_{k-1}< k <i_k\\}\\subset B$. Consequently, by the definition \nof $B$ as in \\eqref{eq:uni.ab},  and line \\ref{eq:null.step} we conclude that \n$x_i=x_{i_{k-1}}$ whenever $i_{k-1}\\leq i<  i_k$. \nAs a consequence, we obtain that \\eqref{eq:ixk} follows\nfrom the fact that $i_{k-1}\\leq i_k-1<i_k$.}\n%\n\\erema\n","sha256_utf8":"240232d52a7d13d7f05afb06f712befd7f47f9dcc8b137eacd1cc6360c798b6c"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"提出相邻大步状态恒等式","source_units":[{"start":66074,"end":66325,"start_line":1714,"end_line":1725,"raw_tex":"\\brema \\lab{rm:rm}\n%\n\\emph{\nWe claim that\nif $(x_k)$ is generated by Algorithm \\ref{alg:main} then\n%\n\\begin{align}\n\\lab{eq:ixk}\n x_{i_k-1}=x_{i_{k-1}}\\qquad \\forall k\\in K,\n\\end{align}\n%\nwhere $K$ is as in \\eqref{eq:s70} -- see also \\eqref{eq:car.a}.\n","sha256_utf8":"e1b6739c2ed6fa650ee7a46d5ee0d2a2cd6cad21b4cc9ad74398548732feef78"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"中间空步保持状态不变","source_units":[{"start":66327,"end":66722,"start_line":1727,"end_line":1734,"raw_tex":"Indeed, for any $k\\in K$, by \\eqref{eq:uni.ab} and \\eqref{eq:car.a} we have \n$\\{k\\geq 1\\,|\\, i_{k-1}< k <i_k\\}\\subset B$. Consequently, by the definition \nof $B$ as in \\eqref{eq:uni.ab},  and line \\ref{eq:null.step} we conclude that \n$x_i=x_{i_{k-1}}$ whenever $i_{k-1}\\leq i<  i_k$. \nAs a consequence, we obtain that \\eqref{eq:ixk} follows\nfrom the fact that $i_{k-1}\\leq i_k-1<i_k$.}\n%\n\\erema\n","sha256_utf8":"621fe3c5a3c8277082978b001cef2ea72ffbc7e0eda0cbfad6a02043046a42f5"}]},{"id":"S052","semantic_type":"proposition","kind":"proposition","title":"大步子序列归入HPE","source_units":[{"start":66730,"end":67735,"start_line":1738,"end_line":1773,"raw_tex":"\\bprop \\lab{prop:special}\n%\nConsider the sequences evolved by \\emph{Algorithm \\ref{alg:main}} and let the set $K$ be as in \\eqref{eq:s70} \nand \\eqref{eq:car.a}.\n%\nThen, for all $k\\in K$,\n%\n\\begin{align}\n%\\begin{aligned}\n  %\n  \\lab{eq:stick00}\n  &\\nu_{i_k}\\in N_C(y_{i_k}),\\\\[2mm]\n  \\lab{eq:stick01}\n   &\\left\\|\\lambda_{i_k} \\left(F(y_{i_k}) + \\nu_{i_k}\\right) + y_{i_k} - x_{i_{k-1}}\\right\\|\\leq \\sigma\\norm{y_{i_k} - x_{i_{k-1}}},\\\\[2mm]\n   %\n   \\lab{eq:stick02}\n   &\\lambda_{i_k}\\norm{y_{i_k} - x_{i_{k-1}}}\\geq \\eta,\\\\[2mm]\n   %\n   \\lab{eq:stick03}\n   & x_{i_k} = x_{i_{k-1}} - \\tau\\lambda_{i_k} \\left(F(y_{i_k}) + \\nu_{i_k}\\right),\n  %\n  %\\end{aligned}\n  \\end{align}\n  %\n  where\n %\n \\begin{align}\\lab{eq:def.sigma}\n %\n \\sigma:=\\dfrac{2\\hat\\theta}{\\eta L}<1.\n%\n \\end{align}\n %\n As a consequence, the sequences $(x_{i_k})_{k\\in K}$, $(y_{i_k})_{k\\in K}$ and $(\\lambda_{i_k})_{k\\in K}$ are generated\n by \\emph{Algorithm \\ref{alg:rHPE}} \\emph{(}with $N=\\# A$\\emph{)} for solving \\eqref{eq:probm}.\n%\n\\eprop","sha256_utf8":"decdcf5f247ed7a7977a024fe5f4b976886088019ae668cdec30cda9fe2eb1c4"}]},{"id":"S053","semantic_type":"proof","kind":"proof","title":"HPE归约完整证明","source_units":[{"start":67738,"end":68963,"start_line":1775,"end_line":1804,"raw_tex":"\\bproo\n%\nNote first that \\eqref{eq:stick00} is a direct consequence of Algorithm \\ref{alg:main}'s definition and the definition \nof $A$ as in \\eqref{eq:uni.ab}.\n%\nMoreover, note that \\eqref{eq:stick02} and \\eqref{eq:stick03} follow directly from \\eqref{eq:def.ab}--\\eqref{eq:car.a}\nand the assumption that $k\\in K$ (see also line \\ref{eq:ext.step} in Algorithm \\ref{alg:main}).\n%\nLet's now prove the inequality \\eqref{eq:stick01}.\n%\nFrom Proposition \\ref{prop:inc.theta}(b) we obtain\n%\n\\[\n\\dfrac{\\lambda_{i_k}L}{2}\\left\\|\\lambda_{i_k} \\left(F(y_{i_k}) + \\nu_{i_k}\\right) + y_{i_k} - x_{i_k-1}\\right\\|\\leq \\hat\\theta,\n\\]\n%\nwhich when combined with \\eqref{eq:ixk}, \\eqref{eq:def.sigma} and \\eqref{eq:stick02} yields\n%\n\\begin{align*}\n%\n\\left\\|\\lambda_{i_k} \\left(F(y_{i_k}) + \\nu_{i_k}\\right) + y_{i_k} - x_{i_{k-1}}\\right\\|&\\leq \\dfrac{2\\hat\\theta}{\\lambda_{i_k} L}\\\\[2mm]\n    & = \\sigma\\dfrac{\\eta}{\\lambda_{i_k}}\\\\[2mm]\n    & \\leq \\sigma \\norm{y_{i_k} - x_{i_{k-1}}},\n%\n\\end{align*}\n%\nwhich gives exactly \\eqref{eq:stick01}. The last statement in the proposition is a direct consequence of \n\\eqref{eq:stick01}--\\eqref{eq:stick03}, Algorithm \\ref{alg:rHPE}'s definition and the definition of $K$ as in \\eqref{eq:s70}.\n%\n\\eproo","sha256_utf8":"471ea62a098f1ea5201e77c4b3049997c23832d2386ebf970e70a08bd768fb9c"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"确认法锥包含与大步更新","source_units":[{"start":67738,"end":68116,"start_line":1775,"end_line":1781,"raw_tex":"\\bproo\n%\nNote first that \\eqref{eq:stick00} is a direct consequence of Algorithm \\ref{alg:main}'s definition and the definition \nof $A$ as in \\eqref{eq:uni.ab}.\n%\nMoreover, note that \\eqref{eq:stick02} and \\eqref{eq:stick03} follow directly from \\eqref{eq:def.ab}--\\eqref{eq:car.a}\nand the assumption that $k\\in K$ (see also line \\ref{eq:ext.step} in Algorithm \\ref{alg:main}).\n","sha256_utf8":"a37959ef098c4d436fc03262570dc6675e48fd8f20826635285f3f603e10beee"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"提取大步残差不变量","source_units":[{"start":68118,"end":68358,"start_line":1783,"end_line":1789,"raw_tex":"Let's now prove the inequality \\eqref{eq:stick01}.\n%\nFrom Proposition \\ref{prop:inc.theta}(b) we obtain\n%\n\\[\n\\dfrac{\\lambda_{i_k}L}{2}\\left\\|\\lambda_{i_k} \\left(F(y_{i_k}) + \\nu_{i_k}\\right) + y_{i_k} - x_{i_k-1}\\right\\|\\leq \\hat\\theta,\n\\]\n","sha256_utf8":"8c1c71a1a155152ac0a66d39f3ea58de8139451ea14cf92c7d8232cfe589067b"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"转化为HPE相对误差条件","source_units":[{"start":68360,"end":68721,"start_line":1791,"end_line":1799,"raw_tex":"which when combined with \\eqref{eq:ixk}, \\eqref{eq:def.sigma} and \\eqref{eq:stick02} yields\n%\n\\begin{align*}\n%\n\\left\\|\\lambda_{i_k} \\left(F(y_{i_k}) + \\nu_{i_k}\\right) + y_{i_k} - x_{i_{k-1}}\\right\\|&\\leq \\dfrac{2\\hat\\theta}{\\lambda_{i_k} L}\\\\[2mm]\n    & = \\sigma\\dfrac{\\eta}{\\lambda_{i_k}}\\\\[2mm]\n    & \\leq \\sigma \\norm{y_{i_k} - x_{i_{k-1}}},\n%\n\\end{align*}\n","sha256_utf8":"dc13c5dc05c5c07eebd7df5b98c3665a450191474d79d4dd6ae1334c688bc171"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"识别完整HPE迭代","source_units":[{"start":68723,"end":68963,"start_line":1801,"end_line":1804,"raw_tex":"which gives exactly \\eqref{eq:stick01}. The last statement in the proposition is a direct consequence of \n\\eqref{eq:stick01}--\\eqref{eq:stick03}, Algorithm \\ref{alg:rHPE}'s definition and the definition of $K$ as in \\eqref{eq:s70}.\n%\n\\eproo","sha256_utf8":"ecd2225dcf5dc81b14e47dbbe773a5bd804b229a19c4abc92bddeead5572f6fa"}]},{"id":"S058","semantic_type":"corollary","kind":"corollary","title":"大步子序列收敛速率","source_units":[{"start":68972,"end":70026,"start_line":1808,"end_line":1840,"raw_tex":"\\bcoro \\lab{cor:rate}\n%\nConsider the sequences evolved by \\emph{Algorithm \\ref{alg:main}}, and the ergodic sequences as in \\eqref{eq:def.erg1}--\\eqref{eq:def.erg3} -- for the sequences $(y_{i_k})_{k\\in K}$ and $(\\lambda_{i_k})_{k\\in K}$, where the set $K$ is as in \\eqref{eq:s70}. \n%\nLet $d_0$ denote the distance of $x_0$ to the solution set $(F + N_C)^{-1}(0)\\neq \\emptyset$ of \\eqref{eq:probm}. \n%\nLet also $0<\\tau<1$ and $0\\leq\\sigma<1$ be as in \\eqref{eq:def.tau.alg} and \\eqref{eq:def.sigma}, respectively.\n\nThe following hold for all $k\\in K$:\n%\n\\begin{itemize}\n%\n\\item[\\emph{(a)}] There exists $1\\leq j\\leq k$ such that $\\nu_{i_j}\\in N_C(y_{i_j})$ and\n%\n\\begin{align*}\n%\n\\norm{F(y_{i_j}) + \\nu_{i_j}}\\leq \\dfrac{d_0^2}{\\tau \\eta(1-\\sigma)\\,k},\n%\n\\end{align*}\n%\n\\item[\\emph{(b)}] $v_k^a\\in (F + N_C)^{\\varepsilon_k^a}(y_k^a)$ and\n%\n\\begin{align*}\n%\n\\norm{v_k^a}&\\leq \\dfrac{2d_0^2}{\\tau^{3/2}\\eta\\sqrt{1-\\sigma^2}\\,k^{3/2}},\\\\[3mm]\n%\n\\varepsilon_k^a&\\leq \\dfrac{2d_0^3}{\\tau^{3/2}\\eta(1-\\sigma^2)\\,k^{3/2}}.\n%\n\\end{align*}\n%\n\\end{itemize}\n%\n\\ecoro","sha256_utf8":"f0e7018058d8bcf4e3075495b595a8c5461f3d31f1276fece30fbc1e55847304"}]},{"id":"S059","semantic_type":"proof","kind":"proof","title":"子序列速率完整证明","source_units":[{"start":70029,"end":70163,"start_line":1842,"end_line":1846,"raw_tex":"\\bproo\n%\nThe proof is a direct consequence of the last statement in Proposition \\ref{prop:special} and Theorem \\ref{th:rhpe}.\n%\n\\eproo","sha256_utf8":"c4a0efc993b231680b53a4d2ad625e3bdd0949c2f64c5fc7650a789f04e6fe92"}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","title":"通过归约继承HPE速率","source_units":[{"start":70029,"end":70163,"start_line":1842,"end_line":1846,"raw_tex":"\\bproo\n%\nThe proof is a direct consequence of the last statement in Proposition \\ref{prop:special} and Theorem \\ref{th:rhpe}.\n%\n\\eproo","sha256_utf8":"c4a0efc993b231680b53a4d2ad625e3bdd0949c2f64c5fc7650a789f04e6fe92"}]},{"id":"S061","semantic_type":"proposition","kind":"proposition","title":"空步残差界","source_units":[{"start":70173,"end":70639,"start_line":1851,"end_line":1864,"raw_tex":"\\bprop \\lab{prop:iteb}\n%\nConsider the sequences evolved by \\emph{Algorithm \\ref{alg:main}} and let the set $B$ be as in \\eqref{eq:def.ab} and\n\\eqref{eq:uni.ab}. Then, for all $k\\in B$,\n%\n\\begin{align}\\lab{eq:res.b}\n%\n\\nu_k\\in N_C(y_k)\\;\\;\\mbox{and}\\;\\;\\norm{F(y_k) + \\nu_k}\\leq \\dfrac{2\\hat\\theta/L + \\eta}{\\lambda_k^2}, \n%\n\\end{align}\n%\nwhere $\\eta > 0$ is as in the input of \\emph{Algorithm \\ref{alg:main}} and $\\hat\\theta>0$ is as in \\eqref{eq:def.hth}. \n%\n\\eprop","sha256_utf8":"28f6a96d0b1c8273eb7d230c41028528a0205f5781d1a1ed0d0788d872249b98"}]},{"id":"S062","semantic_type":"proof","kind":"proof","title":"空步残差完整证明","source_units":[{"start":70642,"end":71189,"start_line":1866,"end_line":1881,"raw_tex":"\\bproo\n%\nNote first that the inclusion in \\eqref{eq:res.b} follows directly from Algorithm \\ref{alg:main}'s definition.\n%\nNow, from Proposition \\ref{prop:inc.theta}(b), the definition of $B$ and the triangle inequality, we obtain\n%\n\\begin{align*}\n%\n\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right)} &\\leq \\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}+\\norm{y_k - x_{k-1}}\\\\[2mm]\n   &\\leq \\dfrac{2\\hat\\theta}{\\lambda_k L} + \\dfrac{\\eta}{\\lambda_k},\n%\n\\end{align*}\n%\nwhich clearly gives the desired inequality in \\eqref{eq:res.b}.\n%\n\\eproo","sha256_utf8":"92497f18280d0bacddc33a559afb6ef17dc4d84416f836770f89819bd8a4e97f"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"确认空步法锥包含","source_units":[{"start":70642,"end":70762,"start_line":1866,"end_line":1868,"raw_tex":"\\bproo\n%\nNote first that the inclusion in \\eqref{eq:res.b} follows directly from Algorithm \\ref{alg:main}'s definition.\n","sha256_utf8":"68dd380f872cd9d91fb06d905471bc96c0710279ccbaf1b9916e2cce8522e9ed"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"三角不等式导出残差界","source_units":[{"start":70764,"end":71189,"start_line":1870,"end_line":1881,"raw_tex":"Now, from Proposition \\ref{prop:inc.theta}(b), the definition of $B$ and the triangle inequality, we obtain\n%\n\\begin{align*}\n%\n\\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right)} &\\leq \\norm{\\lambda_k \\left(F(y_k) + \\nu_k\\right) + y_k - x_{k-1}}+\\norm{y_k - x_{k-1}}\\\\[2mm]\n   &\\leq \\dfrac{2\\hat\\theta}{\\lambda_k L} + \\dfrac{\\eta}{\\lambda_k},\n%\n\\end{align*}\n%\nwhich clearly gives the desired inequality in \\eqref{eq:res.b}.\n%\n\\eproo","sha256_utf8":"b003f81d43010a7b62adf778388973d107e9071a578374ff740680c9a6c0ed30"}]},{"id":"S065","semantic_type":"lemma","kind":"lemma","title":"近端参数的计数表达","source_units":[{"start":71199,"end":71473,"start_line":1886,"end_line":1897,"raw_tex":"\\begin{lemma} \\lab{lm:geo}\n%\nLet $(\\lambda_k)$ be generated by \\emph{Algorithm \\ref{alg:main}} and let $(a_k)$ and $(b_k)$ be as in \\eqref{eq:def.ab}.\nThen, for all $k\\geq 1$,\n%\n\\begin{align}\n%\n\\lambda_{k+1} = (1 - \\tau)^{a_{k} - b_{k}}\\lambda_1.\n%\n\\end{align}\n%\n\\end{lemma}","sha256_utf8":"ab99688cea5390aae3f9f192d58dc554ff0ce4cdd65849491fd4cf349e5cc6f7"}]},{"id":"S066","semantic_type":"proof","kind":"proof","title":"参数表达完整证明","source_units":[{"start":71476,"end":71653,"start_line":1899,"end_line":1904,"raw_tex":"\\bproo\n%\nThe result follows from lines \\ref{eq:red.lam} and \\ref{eq:inc.lam} of Algorithm \\ref{alg:main} and the definitions of\n$a_k$ and $b_k$ as in \\eqref{eq:def.ab}.\n%\n\\eproo","sha256_utf8":"8b8fb8cff337c3426811ad96000caedcf1ff332e0bc86ec212a9abe608f76875"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"累乘两类参数更新","source_units":[{"start":71476,"end":71653,"start_line":1899,"end_line":1904,"raw_tex":"\\bproo\n%\nThe result follows from lines \\ref{eq:red.lam} and \\ref{eq:inc.lam} of Algorithm \\ref{alg:main} and the definitions of\n$a_k$ and $b_k$ as in \\eqref{eq:def.ab}.\n%\n\\eproo","sha256_utf8":"8b8fb8cff337c3426811ad96000caedcf1ff332e0bc86ec212a9abe608f76875"}]},{"id":"S068","semantic_type":"lemma","kind":"lemma","title":"线性化二次余项界","source_units":[{"start":86862,"end":87353,"start_line":2306,"end_line":2325,"raw_tex":"\\begin{lemma}\n \\lab{lm:lip}\n %\n Let $F:\\HH\\to \\HH$ be a continuously differentiable mapping\n  with a $L$-Lipschitz continuous derivative $F'(\\cdot)$, i.e., let $F$ be such that there exits $L\\geq 0$ satisfying\n%\n\\begin{align}\n \\lab{eq:lip.app}\n \\norm{F'(x) - F'(y)}\\leq L\\norm{x - y}\\qquad  \\forall x,y \\in \\HH.\n\\end{align}\n%\nThen,\n%\n  \\[\n   \\norm{F(x) - F_y(x)}\\leq \\dfrac{L}{2}\\norm{x - y}^2 \\qquad \\forall x,y\\in \\HH,\n  \\]\n  %\n  where $F_y(x)$ is as in \\eqref{eq:def.lin}.\n  %\n\\end{lemma}","sha256_utf8":"5f4cb205aab6443d3c02568429bca6e721b0ad430077007093d4c7579be2c1cf"}]},{"id":"S069","semantic_type":"external_fact","kind":"external_fact","title":"标准泰勒余项估计","source_units":[],"declared_external":true,"declaration_text":["For a continuously differentiable map F with L-Lipschitz derivative on the relevant line segment, ||F(x)-F(y)-F'(y)(x-y)|| <= (L/2)||x-y||^2. This standard estimate holds in real Hilbert spaces; the paper cites Dennis and Schnabel, Lemma 4.1.12, for its finite-dimensional version."]},{"id":"S070","semantic_type":"proof","kind":"proof","title":"余项引理的作者引用说明","source_units":[{"start":86642,"end":86854,"start_line":2301,"end_line":2302,"raw_tex":"A proof of the lemma below in finite-dimensional spaces can be found in \\cite[Lemma 4.1.12]{den.sch-numbook.siam96}. \nA proof in general Hilbert spaces follows the same outline as in the finite-dimensional case.\n","sha256_utf8":"40731d4ae91190b36374093106d52bb41d68c65a084cfb46f32d687121252b29"}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","title":"引用标准证明及空间扩展","source_units":[{"start":86642,"end":86854,"start_line":2301,"end_line":2302,"raw_tex":"A proof of the lemma below in finite-dimensional spaces can be found in \\cite[Lemma 4.1.12]{den.sch-numbook.siam96}. \nA proof in general Hilbert spaces follows the same outline as in the finite-dimensional case.\n","sha256_utf8":"40731d4ae91190b36374093106d52bb41d68c65a084cfb46f32d687121252b29"}]},{"id":"S072","semantic_type":"external_fact","kind":"external_fact","title":"单调算子标准性质","source_units":[],"declared_external":true,"declaration_text":["For nonempty closed convex C, its normal cone is monotone. Differentiability and monotonicity of F give <F'(y)h,h> >= 0 in the directions used in the proof, as cited to Rockafellar and Wets, Proposition 12.3. The continuous monotone VIP operator F+N_C is maximal monotone in the setting used for the cited HPE theorem."]},{"id":"S073","semantic_type":"external_fact","kind":"external_fact","title":"已有大步欠松弛HPE定理","source_units":[],"declared_external":true,"declaration_text":["Svaiter (2023), Theorem 2.4: for the large-step under-relaxed HPE method on a maximal monotone T with relaxation tau in (0,1), relative error sigma in [0,1), large-step parameter eta>0 and initial solution distance d0, some j<=k has ||v_j|| <= d0^2/[tau*eta*(1-sigma)*k]; its weighted ergodic triple satisfies v_k^a in T^{epsilon_k^a}(y_k^a), ||v_k^a|| <= 2*d0^2/[tau^{3/2}*eta*sqrt(1-sigma^2)*k^{3/2}] and epsilon_k^a <= 2*d0^3/[tau^{3/2}*eta*(1-sigma^2)*k^{3/2}]."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S014","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S023","kind":"supports"},{"source_id":"S002","target_id":"S029","kind":"supports"},{"source_id":"S002","target_id":"S034","kind":"supports"},{"source_id":"S003","target_id":"S008","kind":"supports"},{"source_id":"S003","target_id":"S027","kind":"supports"},{"source_id":"S004","target_id":"S007","kind":"supports"},{"source_id":"S004","target_id":"S029","kind":"supports"},{"source_id":"S004","target_id":"S068","kind":"supports"},{"source_id":"S005","target_id":"S023","kind":"supports"},{"source_id":"S005","target_id":"S057","kind":"supports"},{"source_id":"S006","target_id":"S021","kind":"supports"},{"source_id":"S006","target_id":"S058","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S036","kind":"supports"},{"source_id":"S009","target_id":"S046","kind":"supports"},{"source_id":"S009","target_id":"S047","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S026","kind":"supports"},{"source_id":"S010","target_id":"S027","kind":"supports"},{"source_id":"S010","target_id":"S028","kind":"supports"},{"source_id":"S010","target_id":"S033","kind":"supports"},{"source_id":"S010","target_id":"S039","kind":"supports"},{"source_id":"S010","target_id":"S040","kind":"supports"},{"source_id":"S010","target_id":"S041","kind":"supports"},{"source_id":"S010","target_id":"S042","kind":"supports"},{"source_id":"S010","target_id":"S051","kind":"supports"},{"source_id":"S010","target_id":"S054","kind":"supports"},{"source_id":"S010","target_id":"S063","kind":"supports"},{"source_id":"S010","target_id":"S067","kind":"supports"},{"source_id":"S011","target_id":"S018","kind":"supports"},{"source_id":"S012","target_id":"S027","kind":"supports"},{"source_id":"S012","target_id":"S028","kind":"supports"},{"source_id":"S012","target_id":"S034","kind":"supports"},{"source_id":"S013","target_id":"S017","kind":"supports"},{"source_id":"S013","target_id":"S018","kind":"supports"},{"source_id":"S013","target_id":"S050","kind":"supports"},{"source_id":"S013","target_id":"S051","kind":"supports"},{"source_id":"S013","target_id":"S054","kind":"supports"},{"source_id":"S013","target_id":"S064","kind":"supports"},{"source_id":"S013","target_id":"S067","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S018","kind":"supports"},{"source_id":"S016","target_id":"S019","kind":"supports"},{"source_id":"S017","target_id":"S015","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S015","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S018","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S060","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S027","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S027","target_id":"S025","kind":"supports"},{"source_id":"S028","target_id":"S025","kind":"supports"},{"source_id":"S029","target_id":"S025","kind":"supports"},{"source_id":"S030","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S035","kind":"supports"},{"source_id":"S033","target_id":"S036","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S032","kind":"supports"},{"source_id":"S035","target_id":"S032","kind":"supports"},{"source_id":"S036","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S040","kind":"supports"},{"source_id":"S031","target_id":"S044","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S041","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S041","target_id":"S038","kind":"supports"},{"source_id":"S042","target_id":"S038","kind":"supports"},{"source_id":"S043","target_id":"S038","kind":"supports"},{"source_id":"S044","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S055","kind":"supports"},{"source_id":"S037","target_id":"S064","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S048","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S041","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S051","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S054","kind":"supports"},{"source_id":"S049","target_id":"S056","kind":"supports"},{"source_id":"S054","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S009","target_id":"S056","kind":"supports"},{"source_id":"S054","target_id":"S057","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S055","target_id":"S053","kind":"supports"},{"source_id":"S056","target_id":"S053","kind":"supports"},{"source_id":"S057","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S017","kind":"supports"},{"source_id":"S058","target_id":"S020","kind":"supports"},{"source_id":"S063","target_id":"S062","kind":"supports"},{"source_id":"S064","target_id":"S062","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S019","kind":"supports"},{"source_id":"S067","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S019","kind":"supports"},{"source_id":"S068","target_id":"S034","kind":"supports"},{"source_id":"S069","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S070","kind":"supports"},{"source_id":"S070","target_id":"S068","kind":"supports"},{"source_id":"S072","target_id":"S023","kind":"supports"},{"source_id":"S072","target_id":"S027","kind":"supports"},{"source_id":"S072","target_id":"S029","kind":"supports"},{"source_id":"S073","target_id":"S023","kind":"supports"}]}}
{"record_id":"2310.12139v3","paper":{"title":"Optimal and parameter-free gradient minimization methods for convex and nonconvex optimization","authors":["Guanghui Lan","Yuyuan Ouyang","Zhe Zhang"],"arxiv_id":"2310.12139","version":"v3","journal":"Mathematical Programming","doi":"10.1007/s10107-026-02352-2","abstract_url":"https://arxiv.org/abs/2310.12139v3","pdf_url":"https://arxiv.org/pdf/2310.12139v3"},"context":{"raw_tex":"The basic problem of interest in this paper is to compute an\napproximate stationary point $\\hat x$ to  \nthe unconstrained optimization problem\n\\begin{equation} \\label{eq:basic_problem}\n\tf^* = \\min_{x\\in\\R^n}f(x),\n\\end{equation}\n\nwhere $f$ is $L$-smooth over $\\R^n$, meaning that $f$ is differentiable and its gradient $\\nabla f$ is Lipschitz continuous, i.e., $\\|\\nabla f(x) - \\nabla f(y)\\|\n\\le L \\|x - y\\|$ for any $x, y \\in \\R^n$. \\cmo{ Here and throughout this paper, the norm $\\|\\cdot\\|$ measured over points $x\\in \\R^n$ and gradients $\\nabla f(x)\\in\\R^n$ are all Euclidean norm.}\nMoreover, we consider three important types of $f$ as follows: (a) $f$ is convex, i.e.,\n$f(x) \\ge f(y) +\\langle \\nabla f(y), x-y\\rangle$ for any $x, y \\in \\R^n$; (b) $f$ is $\\mu$-strongly convex, i.e., $f(x) \\ge f(y) +\\langle \\nabla f(y), x-y\\rangle + \\mu \\|x-y\\|^2/2$ $\\forall x, y \\in \\R^n$ for some $\\mu \\in (0, L)$;\n\n\n\\begin{algorithm}[h]\n\t\\caption{\\label{alg:proposed_unconstrained} An accumulative regularization method for gradient minimization}\n\t\\begin{algorithmic}\n\t\t\\Require Total number of subproblems $S$, strictly increasing regularization parameters $\\{\\sigma_s\\}_{s=0}^{S}$ with $\\sigma_0=0$, and initial point $x_0\\in\\R^n$. \\State Set initial prox-center to $\\xu_0 := x_0$.\n\t\t\\For {$s=1,\\ldots, S$}\n\t\t\\State Set\n\t\t\\begin{align}\n\t\t\t\\label{eq:xus_unconstrained}\n\t\t\t\\xu_s = (1-\\gamma_s) \\xu_{s-1} + \\gamma_s x_{s-1}\\text{ with }\\gamma_s =1 - \\sigma_{s-1}/\\sigma_s.\n\t\t\\end{align}\n\t\t\\State Compute an approximate solution $x_s$ of the proximal mapping subproblem\n\t\t\\begin{align}\n\t\t\t\\label{eq:subproblem_unconstrained}\n\t\t\tx_s^*:=\\argmin_{x\\in \\R^n} \\{f_s(x):=f(x) + \\frac{\\sigma_s}{2}\\|x - \\xu_s\\|^2\\}\n\t\t\\end{align}\n\t\tby running subroutine $\\cA(f, \\sigma_s, \\xu_s, x_{s-1}, N_s)$ with $N_s$ gradient evaluations of $\\nabla f$ and initial point $x_{s-1}$.\n\t\t\\EndFor\n\t\t\\State Output $x_S$.\n\t\\end{algorithmic}\n\\end{algorithm}\n\nWe are now ready to describe the details for proving our claim \\eqref{eq:claim_on_distance_unconstrained} and estimating the number of gradient evaluations required to solve subproblems \\eqref{eq:subproblem_unconstrained}. For convenience, we allow that the definition   \\eqref{eq:subproblem_unconstrained} applies for $s=0$, and in such case we have $\\sigma_0 = 0$ and hence $x_0^*=x^*$ is an optimal solution to the original problem. The proof of our claim \\eqref{eq:claim_on_distance_unconstrained} is described in the following proposition. Note that the result \\eqref{eq:dist_xu_xstarsig_unconstrained} below is exactly our claim\n\\eqref{eq:claim_on_distance_unconstrained} since $\\sigma_0 = 0$.\n\n\nIn this section, we show that the accumulative regularization technique we introduce in the previous section can be adapted to problems with constrained feasible sets, and (nested) composite objective function. We consider the problem with simple feasible set $X$ and composite function $\\phi$:\n\\begin{align}\n\t\\label{eq:problem} \\min_{x\\in X}\\{ F(x):=f(x) + \\phi(x)\\}.\n\\end{align}\n\nHere $X \\subset \\R^n$ is a closed convex set and $f$ and $\\phi$ are closed convex functions. We still assume that $\\nabla f$ is $L$-smooth. In addition, we assume that $X$ and $\\phi$ are  relatively simple such that for any $\\sigma, \\eta\\ge 0$ and any $x,\\xu\\in\\R^n$, the following optimization problem can be computed efficiently:\n\\begin{align}\n\t\\label{eq:xpp}\n\tx^{++}(\\eta, \\sigma, \\xu; x):=\\argmin_{u\\in X}\\langle \\nabla f(x), u\\rangle + \\phi(u) + \\frac{\\sigma}{2}\\|u - \\xu\\|^2 + \\frac{\\eta}{2}\\|u - x\\|^2.\n\\end{align}\n\nThroughout this paper we assume that an optimal solution $x^*$ to problem \\eqref{eq:problem} exists. \nClearly, $x^*$ is an optimal solution of \\eqref{eq:problem} if and only if the \\emph{projected gradient} $G_\\eta(x^*) = 0$ for some $\\eta>0$, where\n\\begin{align}\n\t\\label{eq:G}\n\tG_{\\eta}(x) :=\\ & \\eta(x - x^+(\\eta;x))\\text{ and }\n\t\\\\\n\t\\label{eq:xp}\n\tx^+(\\eta;x):=\\ &x^{++}(\\eta, 0, x;x) = \\argmin_{u\\in X}\\langle\\nabla f(x), u\\rangle + \\phi(u) + \\frac{\\eta}{2}\\|u - x\\|^2\\ \\forall x\\in \\R^n.\n\\end{align}\n\n\\begin{algorithm*}[h] \n\t\\caption{$M$=Backtracking($g$, $\\sigma$, $x$, $M_0$) \\label{alg:_searchL}}\n\t\\begin{algorithmic}\n\t\t\\For{$j=0,1,\\ldots,$}\n\t\t\\State Set\n\t\t$\t\t\t\\xpp = x - ({1}/{(2(M_j + \\sigma))})\\nabla g(x).\n\t\t$\n\t\t\\State If \n\t\t\\begin{align}\n\t\t\t\\label{eq:linesearch_estL_success_criterion}\n\t\t\tg(\\xpp) - g(x) - \\langle \\nabla g(x), \\xpp - x\\rangle \\le ({(M_j + \\sigma)}/{2})\\|\\xpp - x\\|^2, \n\t\t\\end{align}\n\t\tthen {\\bf terminate} with $M=M_j$. Otherwise, set $M_{j+1} = 2M_{j}$. \n\t\t\\EndFor\n\t\\end{algorithmic}\n\\end{algorithm*}\n\n\\begin{algorithm}[!h]\n\t\\caption{\\label{alg:proposed_unconstrained_searchL} Accumulative regularization (AR) without the knowledge of $L$}\n\t\\begin{algorithmic}\n\t\t\\Function{$(\\hat x, M)=\\ $AR}{$f, x_0, \\sigma_1, M_0$}\n\t\t\\State Set the initial prox-center $\\xu_0 = x_0$, and denote $\\sigma_0=0$.\n\t\t\\For {$s=1,2,\\ldots$}\n\t\t\\State If $s>1$, then set $\\sigma_s=4\\sigma_{s-1}$.\n\t\t\\State Set $\\xu_s =  (1-\\gamma_s) \\xu_{s-1} + \\gamma_s x_{s-1}$ with $\\gamma_s=1 - \\sigma_{s-1}/\\sigma_s$.\n\t\t\n\t\t\\State Compute an approximate solution $x_s\\in \\R^n$ of\n\t\t\\begin{align}\n\t\t\t\\label{eq:subproblem_unconstrained_searchL}\n\t\t\tx_s^*:=\\argmin_{x\\in \\R^n} \\{ f_s(x):=f(x) + \\dfrac{\\sigma_s}{2}\\|x - \\xu_s\\|^2\\}\n\t\t\\end{align}\n\t\tby running subroutine $x_s = \\cA(f, \\sigma_s, \\xu_s, x_{s-1})$ with initial point $x_{s-1}$. \n\t\t\n\t\t\\State Set $M_s=$ Backtracking ($f_s$, $\\sigma_s$, $x_s$, $M_{s-1}/2$).\n\t\t\\State If $\\sigma_s\\ge M_s$, then {\\bf terminate} with $\\hat x=x_s$ and $M=M_s$ . \n\t\t\\EndFor\n\t\t\\EndFunction\n\t\\end{algorithmic}\n\\end{algorithm}\n\n\\begin{assumption}\n\t\\label{assum:A_unconstrained_searchL}\n\tThe subroutine $x_s =\\cA(f, \\sigma_s, \\xu_s, x_{s-1})$ has the following performance guarantees after the $k$-th gradient evaluation of $\\nabla f$:\n\t\\begin{align}\n\t\t\\label{eq:A_cond_unconstrained_searchL}\n\t\tf_s(x_s^k) - f_s(x_s^*) \\le ({L_s^k}/{k^2})\\|x_{s-1} - x_s^*\\|^2.\n\t\\end{align} \n\tHere $x_s^k$ is the computed approximate solution and $L_s^k$ is an estimate of Lipschitz constant $L$ such that $L_s^k\\le c_\\cA L$, where $c_\\cA$ is a universal constant that depends on algorithm $\\cA$. \n\tSubroutine $\\cA$ will terminate whenever $k\\ge 8\\sqrt{2L_s^k/\\sigma_s}$ and output the approximate solution $x_s = x_s^k$.\n\\end{assumption}\n\n\tOur goal in this section is to develop an optimal and parameter-free algorithm for $\\min_{x\\in\\R^n}f(x)$, in which $f$ is both $L$-smooth and $\\mu$-strongly convex. \n\n\n\\begin{algorithm}[h]\n\t\t\\caption{\\label{alg:proposed_SCAR_noflag} A parameter-free strongly convex accumulative regularization (SCAR) algorithm}\n\t\t\\begin{algorithmic}\n\t\t\t\\Function{($\\hat x$, $\\hat M$) = SCAR}{$f$, $\\varepsilon$, $y_0$, $\\mu_0$, $M_0$}\n\t\t\t\\For {$t=1,2,\\ldots$}\n\t\t\t\\State Set\n\t\t\t$\n\t\t\t(y_{t},M_{t}) = \\text{AR}(f, y_{t-1}, \\mu_{t-1}/10, M_{t-1})\n\t\t\t$ (see Algorithm~\\ref{alg:proposed_unconstrained_searchL}).\n\t\t\t\\State If {$\\|\\nabla f(y_t)\\|>\\|\\nabla f(y_{t-1})\\|/2$}, then set $\\mu_t = \\mu_{t-1}/4$ and $y_t = y_{t-1}$. \t\t\n\t\t\t\\State If $\\|\\nabla f(y_{t})\\|\\le \\varepsilon$, then \\textbf{terminate} with $\\hat x = y_{t}$ and $\\hat M = M_t$.\\EndFor\n\t\t\t\\EndFunction\n\t\t\\end{algorithmic}\n\t\\end{algorithm}","units":[{"start":6290,"end":6517,"start_line":152,"end_line":157,"raw_tex":"The basic problem of interest in this paper is to compute an\napproximate stationary point $\\hat x$ to  \nthe unconstrained optimization problem\n\\begin{equation} \\label{eq:basic_problem}\n\tf^* = \\min_{x\\in\\R^n}f(x),\n\\end{equation}","sha256_utf8":"36bd7176ac4c439d60c8a05e58fa50df078385daae896fd0c33efaf710f2b279"},{"start":6518,"end":7194,"start_line":158,"end_line":161,"raw_tex":"where $f$ is $L$-smooth over $\\R^n$, meaning that $f$ is differentiable and its gradient $\\nabla f$ is Lipschitz continuous, i.e., $\\|\\nabla f(x) - \\nabla f(y)\\|\n\\le L \\|x - y\\|$ for any $x, y \\in \\R^n$. \\cmo{ Here and throughout this paper, the norm $\\|\\cdot\\|$ measured over points $x\\in \\R^n$ and gradients $\\nabla f(x)\\in\\R^n$ are all Euclidean norm.}\nMoreover, we consider three important types of $f$ as follows: (a) $f$ is convex, i.e.,\n$f(x) \\ge f(y) +\\langle \\nabla f(y), x-y\\rangle$ for any $x, y \\in \\R^n$; (b) $f$ is $\\mu$-strongly convex, i.e., $f(x) \\ge f(y) +\\langle \\nabla f(y), x-y\\rangle + \\mu \\|x-y\\|^2/2$ $\\forall x, y \\in \\R^n$ for some $\\mu \\in (0, L)$;\n","sha256_utf8":"b9279492760d578d418a5933df4dc7d17c609d5991bf41715edd560c32365b8a"},{"start":22844,"end":23854,"start_line":277,"end_line":296,"raw_tex":"\\begin{algorithm}[h]\n\t\\caption{\\label{alg:proposed_unconstrained} An accumulative regularization method for gradient minimization}\n\t\\begin{algorithmic}\n\t\t\\Require Total number of subproblems $S$, strictly increasing regularization parameters $\\{\\sigma_s\\}_{s=0}^{S}$ with $\\sigma_0=0$, and initial point $x_0\\in\\R^n$. \\State Set initial prox-center to $\\xu_0 := x_0$.\n\t\t\\For {$s=1,\\ldots, S$}\n\t\t\\State Set\n\t\t\\begin{align}\n\t\t\t\\label{eq:xus_unconstrained}\n\t\t\t\\xu_s = (1-\\gamma_s) \\xu_{s-1} + \\gamma_s x_{s-1}\\text{ with }\\gamma_s =1 - \\sigma_{s-1}/\\sigma_s.\n\t\t\\end{align}\n\t\t\\State Compute an approximate solution $x_s$ of the proximal mapping subproblem\n\t\t\\begin{align}\n\t\t\t\\label{eq:subproblem_unconstrained}\n\t\t\tx_s^*:=\\argmin_{x\\in \\R^n} \\{f_s(x):=f(x) + \\frac{\\sigma_s}{2}\\|x - \\xu_s\\|^2\\}\n\t\t\\end{align}\n\t\tby running subroutine $\\cA(f, \\sigma_s, \\xu_s, x_{s-1}, N_s)$ with $N_s$ gradient evaluations of $\\nabla f$ and initial point $x_{s-1}$.\n\t\t\\EndFor\n\t\t\\State Output $x_S$.\n\t\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"a19c99fb5736563126cb6e840233a73707989504cd06d9917a77c0777c5c6ac7"},{"start":27503,"end":28203,"start_line":357,"end_line":358,"raw_tex":"We are now ready to describe the details for proving our claim \\eqref{eq:claim_on_distance_unconstrained} and estimating the number of gradient evaluations required to solve subproblems \\eqref{eq:subproblem_unconstrained}. For convenience, we allow that the definition   \\eqref{eq:subproblem_unconstrained} applies for $s=0$, and in such case we have $\\sigma_0 = 0$ and hence $x_0^*=x^*$ is an optimal solution to the original problem. The proof of our claim \\eqref{eq:claim_on_distance_unconstrained} is described in the following proposition. Note that the result \\eqref{eq:dist_xu_xstarsig_unconstrained} below is exactly our claim\n\\eqref{eq:claim_on_distance_unconstrained} since $\\sigma_0 = 0$.\n","sha256_utf8":"591475ce38be85586e460c7530271ab09ec1bda074d3accff0d8a68f0f6923e5"},{"start":35894,"end":36274,"start_line":472,"end_line":475,"raw_tex":"In this section, we show that the accumulative regularization technique we introduce in the previous section can be adapted to problems with constrained feasible sets, and (nested) composite objective function. We consider the problem with simple feasible set $X$ and composite function $\\phi$:\n\\begin{align}\n\t\\label{eq:problem} \\min_{x\\in X}\\{ F(x):=f(x) + \\phi(x)\\}.\n\\end{align}","sha256_utf8":"e4352f090d65b3812d33cd46f4746ff1ebefd5c53fcdef8c614b54003b8dedaa"},{"start":36275,"end":36796,"start_line":476,"end_line":480,"raw_tex":"Here $X \\subset \\R^n$ is a closed convex set and $f$ and $\\phi$ are closed convex functions. We still assume that $\\nabla f$ is $L$-smooth. In addition, we assume that $X$ and $\\phi$ are  relatively simple such that for any $\\sigma, \\eta\\ge 0$ and any $x,\\xu\\in\\R^n$, the following optimization problem can be computed efficiently:\n\\begin{align}\n\t\\label{eq:xpp}\n\tx^{++}(\\eta, \\sigma, \\xu; x):=\\argmin_{u\\in X}\\langle \\nabla f(x), u\\rangle + \\phi(u) + \\frac{\\sigma}{2}\\|u - \\xu\\|^2 + \\frac{\\eta}{2}\\|u - x\\|^2.\n\\end{align}","sha256_utf8":"1a795dab4b6133c1ea03660b81c6dd1cd4c2d3417ee6114e6f8dc03f3e2dc1c3"},{"start":36797,"end":37301,"start_line":481,"end_line":489,"raw_tex":"Throughout this paper we assume that an optimal solution $x^*$ to problem \\eqref{eq:problem} exists. \nClearly, $x^*$ is an optimal solution of \\eqref{eq:problem} if and only if the \\emph{projected gradient} $G_\\eta(x^*) = 0$ for some $\\eta>0$, where\n\\begin{align}\n\t\\label{eq:G}\n\tG_{\\eta}(x) :=\\ & \\eta(x - x^+(\\eta;x))\\text{ and }\n\t\\\\\n\t\\label{eq:xp}\n\tx^+(\\eta;x):=\\ &x^{++}(\\eta, 0, x;x) = \\argmin_{u\\in X}\\langle\\nabla f(x), u\\rangle + \\phi(u) + \\frac{\\eta}{2}\\|u - x\\|^2\\ \\forall x\\in \\R^n.\n\\end{align}","sha256_utf8":"8a7b1a8eb082d383cd55fba10e7a2ac4557a2da29fcd6c9dff26af80eed04af6"},{"start":60211,"end":60736,"start_line":853,"end_line":868,"raw_tex":"\\begin{algorithm*}[h] \n\t\\caption{$M$=Backtracking($g$, $\\sigma$, $x$, $M_0$) \\label{alg:_searchL}}\n\t\\begin{algorithmic}\n\t\t\\For{$j=0,1,\\ldots,$}\n\t\t\\State Set\n\t\t$\t\t\t\\xpp = x - ({1}/{(2(M_j + \\sigma))})\\nabla g(x).\n\t\t$\n\t\t\\State If \n\t\t\\begin{align}\n\t\t\t\\label{eq:linesearch_estL_success_criterion}\n\t\t\tg(\\xpp) - g(x) - \\langle \\nabla g(x), \\xpp - x\\rangle \\le ({(M_j + \\sigma)}/{2})\\|\\xpp - x\\|^2, \n\t\t\\end{align}\n\t\tthen {\\bf terminate} with $M=M_j$. Otherwise, set $M_{j+1} = 2M_{j}$. \n\t\t\\EndFor\n\t\\end{algorithmic}\n\\end{algorithm*}","sha256_utf8":"eed46d770744b9aaf4368450b56ed183025ed2b4ed735451362538e3171741aa"},{"start":62469,"end":63489,"start_line":890,"end_line":911,"raw_tex":"\\begin{algorithm}[!h]\n\t\\caption{\\label{alg:proposed_unconstrained_searchL} Accumulative regularization (AR) without the knowledge of $L$}\n\t\\begin{algorithmic}\n\t\t\\Function{$(\\hat x, M)=\\ $AR}{$f, x_0, \\sigma_1, M_0$}\n\t\t\\State Set the initial prox-center $\\xu_0 = x_0$, and denote $\\sigma_0=0$.\n\t\t\\For {$s=1,2,\\ldots$}\n\t\t\\State If $s>1$, then set $\\sigma_s=4\\sigma_{s-1}$.\n\t\t\\State Set $\\xu_s =  (1-\\gamma_s) \\xu_{s-1} + \\gamma_s x_{s-1}$ with $\\gamma_s=1 - \\sigma_{s-1}/\\sigma_s$.\n\t\t\n\t\t\\State Compute an approximate solution $x_s\\in \\R^n$ of\n\t\t\\begin{align}\n\t\t\t\\label{eq:subproblem_unconstrained_searchL}\n\t\t\tx_s^*:=\\argmin_{x\\in \\R^n} \\{ f_s(x):=f(x) + \\dfrac{\\sigma_s}{2}\\|x - \\xu_s\\|^2\\}\n\t\t\\end{align}\n\t\tby running subroutine $x_s = \\cA(f, \\sigma_s, \\xu_s, x_{s-1})$ with initial point $x_{s-1}$. \n\t\t\n\t\t\\State Set $M_s=$ Backtracking ($f_s$, $\\sigma_s$, $x_s$, $M_{s-1}/2$).\n\t\t\\State If $\\sigma_s\\ge M_s$, then {\\bf terminate} with $\\hat x=x_s$ and $M=M_s$ . \n\t\t\\EndFor\n\t\t\\EndFunction\n\t\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"7145164813347259e38f8361e3e9f50093e91622f9db9293bc85429785671a45"},{"start":64684,"end":65376,"start_line":925,"end_line":934,"raw_tex":"\\begin{assumption}\n\t\\label{assum:A_unconstrained_searchL}\n\tThe subroutine $x_s =\\cA(f, \\sigma_s, \\xu_s, x_{s-1})$ has the following performance guarantees after the $k$-th gradient evaluation of $\\nabla f$:\n\t\\begin{align}\n\t\t\\label{eq:A_cond_unconstrained_searchL}\n\t\tf_s(x_s^k) - f_s(x_s^*) \\le ({L_s^k}/{k^2})\\|x_{s-1} - x_s^*\\|^2.\n\t\\end{align} \n\tHere $x_s^k$ is the computed approximate solution and $L_s^k$ is an estimate of Lipschitz constant $L$ such that $L_s^k\\le c_\\cA L$, where $c_\\cA$ is a universal constant that depends on algorithm $\\cA$. \n\tSubroutine $\\cA$ will terminate whenever $k\\ge 8\\sqrt{2L_s^k/\\sigma_s}$ and output the approximate solution $x_s = x_s^k$.\n\\end{assumption}","sha256_utf8":"c5b9b14809a1054177642055aa3ed6782417b9303add09cd5275268d156f5e20"},{"start":79304,"end":79471,"start_line":1082,"end_line":1082,"raw_tex":"\tOur goal in this section is to develop an optimal and parameter-free algorithm for $\\min_{x\\in\\R^n}f(x)$, in which $f$ is both $L$-smooth and $\\mu$-strongly convex. \n","sha256_utf8":"5e9b275cd7f6f62fffd56cbb2218c84ecdd413644859d35cc4a3a6fb5b95f925"},{"start":82498,"end":83211,"start_line":1094,"end_line":1107,"raw_tex":"\\begin{algorithm}[h]\n\t\t\\caption{\\label{alg:proposed_SCAR_noflag} A parameter-free strongly convex accumulative regularization (SCAR) algorithm}\n\t\t\\begin{algorithmic}\n\t\t\t\\Function{($\\hat x$, $\\hat M$) = SCAR}{$f$, $\\varepsilon$, $y_0$, $\\mu_0$, $M_0$}\n\t\t\t\\For {$t=1,2,\\ldots$}\n\t\t\t\\State Set\n\t\t\t$\n\t\t\t(y_{t},M_{t}) = \\text{AR}(f, y_{t-1}, \\mu_{t-1}/10, M_{t-1})\n\t\t\t$ (see Algorithm~\\ref{alg:proposed_unconstrained_searchL}).\n\t\t\t\\State If {$\\|\\nabla f(y_t)\\|>\\|\\nabla f(y_{t-1})\\|/2$}, then set $\\mu_t = \\mu_{t-1}/4$ and $y_t = y_{t-1}$. \t\t\n\t\t\t\\State If $\\|\\nabla f(y_{t})\\|\\le \\varepsilon$, then \\textbf{terminate} with $\\hat x = y_{t}$ and $\\hat M = M_t$.\\EndFor\n\t\t\t\\EndFunction\n\t\t\\end{algorithmic}\n\t\\end{algorithm}","sha256_utf8":"90de2a7abe482c90844bdda2e53ba7ea7c1ef889ffea3dceb9193a3d32836ebf"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\n\t\t\\label{thm:conv_proposed_unconstrained_sc_noflag}\n\t\tSuppose that $\\|\\nabla f(y_0)\\|> \\varepsilon$. For any $\\varepsilon>0$, $\\mu_0\\ge \\mu$ and $M_0\\le 4L$, Algorithm~\\ref{alg:proposed_SCAR_noflag} terminates with $\\|\\nabla f(\\hat x)\\|\\le\\varepsilon$. The total number of gradient evaluations is bounded by  $(4+8\\sqrt{5}C_1)\\lceil\\log_4(\\mu_0/\\mu)\\rceil\\sqrt{L/\\mu}  + (4+8\\sqrt{5}C_1)\\sqrt{L/\\mu}\\lceil\\log_2(\\|\\nabla f(y_0)\\|/\\varepsilon)\\rceil$\n\t\t.\n\t\t\\cmo{\n\t\tMoreover, if we have $\\mu_0\\le 2^{j}\\varepsilon/\\|y_0-x^*\\|$ for some integer $j$, an alternative bound of gradient evaluations is $(4+8\\sqrt{5}C_1)\\lceil\\log_4(\\mu_0/\\mu)\\rceil\\sqrt{L/\\mu}  +(4+C_1\\sqrt{10L/\\mu})\\max\\{1,j\\}$.}\n\t\\end{theorem}","units":[{"start":83828,"end":84550,"start_line":1113,"end_line":1119,"raw_tex":"\\begin{theorem}\n\t\t\\label{thm:conv_proposed_unconstrained_sc_noflag}\n\t\tSuppose that $\\|\\nabla f(y_0)\\|> \\varepsilon$. For any $\\varepsilon>0$, $\\mu_0\\ge \\mu$ and $M_0\\le 4L$, Algorithm~\\ref{alg:proposed_SCAR_noflag} terminates with $\\|\\nabla f(\\hat x)\\|\\le\\varepsilon$. The total number of gradient evaluations is bounded by  $(4+8\\sqrt{5}C_1)\\lceil\\log_4(\\mu_0/\\mu)\\rceil\\sqrt{L/\\mu}  + (4+8\\sqrt{5}C_1)\\sqrt{L/\\mu}\\lceil\\log_2(\\|\\nabla f(y_0)\\|/\\varepsilon)\\rceil$\n\t\t.\n\t\t\\cmo{\n\t\tMoreover, if we have $\\mu_0\\le 2^{j}\\varepsilon/\\|y_0-x^*\\|$ for some integer $j$, an alternative bound of gradient evaluations is $(4+8\\sqrt{5}C_1)\\lceil\\log_4(\\mu_0/\\mu)\\rceil\\sqrt{L/\\mu}  +(4+C_1\\sqrt{10L/\\mu})\\max\\{1,j\\}$.}\n\t\\end{theorem}","sha256_utf8":"e7f9702d4590c189d9d852ddb4282057bea2381f5f5a8d0c3c6c78841d5c8087"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"Specifically, \nnoting the $L$-Lipschitz continuity of $\\nabla f(x)$ and using the optimality condition $\\nabla f_s(x_s^*)=\\nabla f(x_s^*) + \\sigma_s(x_s^* - \\xu_s) = 0$ at $x_s^*$,\nwe have\n\\begin{align}\n\t\\label{eq:xs_norm_bound_unconstrained}\n\t\\begin{aligned}\n\t\t& \\|\\nabla f(x_s)\\| = \\|\\nabla f(x_s) - \\nabla f(x_s^*) - \\sigma_s(x_s^* - \\xu_s)\\| \n\t\t\\\\\n\t\t\\le \\ & \\|\\nabla f(x_s) - \\nabla f(x_s^*)\\| + \\sigma_s\\|x_s^* - \\xu_s\\| \n\t\t\\\\\n\t\t\\le\\ &  L\\|x_s - x_s^*\\| + \\sigma_s\\|x_s^* - \\xu_s\\|.\n\t\\end{aligned}\n\\end{align}\nWe will show later that with the increasing regularization parameter $\\sigma_s$ and convex combination design of prox-center in \\eqref{eq:xus_unconstrained}, the distance $\\sigma_s\\|x_s^* - \\xu_s\\|$ can be further bounded:\n\\begin{align}\n\t\\label{eq:claim_on_distance_unconstrained}\n\t\\sigma_s\\|x_s^* - \\xu_s\\| \\le \\sigma_1\\|x^* - x_0\\| + \\sum_{i=2}^s(\\sigma_{i-1} + \\sigma_i)\\|x_{i-1}^* - x_{i-1}\\|.\n\\end{align}\nTherefore, $\\|\\nabla f(x_s)\\|$ can be controled by\n\\begin{align}\n\t\\label{eq:gradS_estimate_unconstrained}\n\t\\|\\nabla f(x_s)\\| \\le\\ &  L\\|x_s - x_s^*\\| + \\sigma_1\\|x^* - x_0\\| + \\sum_{i=2}^s(\\sigma_{i-1} + \\sigma_i)\\|x_{i-1}^* - x_{i-1}\\|.\n\\end{align}\nNote that the right-hand-side of the above estimate is the summation of terms of form $\\|x_i - x_i^*\\|$, i.e., the distance from an approximate solution $x_i$ to the optimal solution $x_i^*$ of the $i$-th strongly convex subproblem.\nBy studying the convergence properties of subroutine $\\cA$ for minimizing smooth and strongly convex functions in subproblem \\eqref{eq:subproblem_unconstrained}, we will then bound the total number of gradient evaluations needed to reduce the right-hand-side of the above estimate to the accuracy $\\varepsilon$. \n\n\n\\begin{proposition}\n\t\\label{pro:xstar_dist_shrink_unconstrained}\n\tIn  Algorithm~\\ref{alg:proposed_unconstrained} we have for all $s\\ge 1$ \n\t\\begin{align}\n\t\t\\label{eq:dist_subproblem_unconstrained}\n\t\t& \\|x_{s-1} - x_s^*\\|\\le \\|x_{s-1} - x_{s-1}^*\\|, \n\t\t\\\\\n\t\t\\label{eq:dist_xu_xstarsig_unconstrained}\n\t\t&\\sigma_s\\|\\xu_s - x_s^*\\| \n\t\t\\le \\sum_{i=1}^{s}(\\sigma_{i-1} + \\sigma_{i})\\|x_{i-1}^* - x_{i-1}\\|\n\t\t.\n\t\\end{align}\n\\end{proposition}\n\n\\begin{proof}\n\tWith the selection of $\\gamma_s = 1 - \\sigma_{s-1}/\\sigma_s$ in the definition of prox-center $\\xu_s$ in   \\eqref{eq:xus_unconstrained}, the subproblem \\eqref{eq:subproblem_unconstrained} is equivalent to\n\t\\begin{align*}\n\t\tx_s^* = \\argmin_{x\\in \\R^n} f(x) + \\dfrac{\\sigma_{s-1}}{2}\\|x - \\xu_{s-1}\\|^2 + \\dfrac{\\sigma_s - \\sigma_{s-1}}{2}\\|x - x_{s-1}\\|^2,\\ \\forall s \\ge 1.\n\t\\end{align*}\n\tNoting the optimality of $x_{s-1}^*$ in \\eqref{eq:subproblem_unconstrained}  and $x_s^*$ in the above subproblem we have\n\t\\begin{align*}\n\t\t& f(x_{s-1}^*) + \\dfrac{\\sigma_{s-1}}{2}\\|x_{s-1}^* - \\xu_{s-1}\\|^2 + \\dfrac{\\sigma_s - \\sigma_{s-1}}{2}\\|x_s^* - x_{s-1}\\|^2\n\t\t\\\\\n\t\t\\le\\ & f(x_s^*) + \\dfrac{\\sigma_{s-1}}{2}\\|x_s^* - \\xu_{s-1}\\|^2 + \\dfrac{\\sigma_s - \\sigma_{s-1}}{2}\\|x_s^* - x_{s-1}\\|^2 \n\t\t\\\\\n\t\t\\le\\ & f(x_{s-1}^*) + \\dfrac{\\sigma_{s-1}}{2}\\|x_{s-1}^* - \\xu_{s-1}\\|^2 + \\dfrac{\\sigma_s - \\sigma_{s-1}}{2}\\|x_{s-1}^* - x_{s-1}\\|^2.\n\t\\end{align*}\n\tThe conclusion in \\eqref{eq:dist_subproblem_unconstrained} then follows from the above relation and the assumption $\\sigma_s>\\sigma_{s-1}$ required by Algorithm~\\ref{alg:proposed_unconstrained}.\n\t\n\tWe move to prove the next result \\eqref{eq:dist_xu_xstarsig_unconstrained}. Denoting $\\alpha_s:=\\sigma_s - \\sigma_{s-1}$ for all $s$ and noting that $\\gamma_s = \\alpha_s/\\sigma_s$ in \\eqref{eq:xus_unconstrained}, we can rewrite the definition of $\\xu_s$ in \\eqref{eq:xus_unconstrained} to\n\t$\n\t\\sigma_s \\xu_s = \\sigma_{s-1}\\xu_{s-1} + \\alpha_s x_{s-1}.\n\t$\n\tRecalling that $\\sigma_0=0$, the above recursive relation yields\n\t$\n\t\\sigma_s\\xu_s = \\sum_{i=1}^{s}\\alpha_ix_{i-1}.\n\t$\n\tIn other words, $\\xu_s$ is a convex combination of $x_0,\\ldots, x_{s-1}$ with weights $\\alpha_i/\\sigma_s$  since $\\sigma_s = \\sum_{i=1}^{s}\\alpha_i$. Thus \n\t\\begin{align*}\n\t\t\\sigma_s(\\xu_s - x_s^*) =\\ & \\sum_{i=1}^{s}\\alpha_i(x_{i-1} - x_s^*) \n\t\t\\\\\n\t\t=\\ & \\alpha_s(x_{s-1} - x_s^*) + \\sum_{i=1}^{s-1}\\alpha_i(x_{i-1} - x_{s-1}^*)  + \\left(\\sum_{i=1}^{s-1}\\alpha_i\\right)(x_{s-1}^* - x_s^*)\n\t\t\\\\\n\t\t=\\ & \\alpha_s(x_{s-1} - x_s^*) + \\sigma_{s-1}(\\xu_{s-1} - x_{s-1}^*) + \\sigma_{s-1}(x_{s-1}^* - x_{s-1}) + \\sigma_{s-1}(x_{s-1} - x_s^*)\n\t\t\\\\\n\t\t=\\ & \\sigma_{s-1}(\\xu_{s-1} - x_{s-1}^*) + \\sigma_{s-1}(x_{s-1}^* - x_{s-1}) + \\sigma_s(x_{s-1} - x_s^*).\n\t\\end{align*}\n\tThe above recursive relation yields\n\t$\n\t\\sigma_s(\\xu_s - x_s^*) = \\sum_{i=1}^{s}[\\sigma_{i-1}(x_{i-1}^* - x_{i-1}) + \\sigma_i(x_{i-1} - x_i^*)].\n\t$\n\tBy the above equality and our previous result \\eqref{eq:dist_subproblem_unconstrained}, we conclude \\eqref{eq:dist_xu_xstarsig_unconstrained}.\n\\qedsymbol\\end{proof}\n\n\\begin{lem}\n\t\\label{lem:grad_map_properties}\n\tFor any $x, \\xu\\in\\R^n$ and $\\eta>0$, $\\sigma\\ge0$, we have the following properties for the (perturbed) gradient mappings:\n\t\\begin{align}\n\t\t\\label{eq:xp_lip}\n\t\t& \\|x^+(\\eta+\\sigma; x) - x^{++}(\\eta, \\sigma, \\xu; x)\\| \\le \\frac{\\sigma}{\\eta+\\sigma}\\|x - \\xu\\|;\n\t\t\\\\\n\t\t\\label{eq:xp_monotonicity}\n\t\t& \\|G_\\eta(x)\\| \\le \\|G_{\\sigma + \\eta}(x)\\|.\n\t\\end{align}\n\tMoreover, if $x^{++}:=x^{++}(\\eta, \\sigma, \\xu; x)$ satisfies \n\t\\begin{align}\n\t\t\\label{eq:local_lip}\n\t\tf(x^{++}) - f(x) - \\langle \\nabla f(x), x^{++} - x\\rangle \\le \\frac{M}{2}\\|x^{++} - x\\|^2\n\t\\end{align}\n\tfor some $M>0$,  we have\n\t\\begin{align}\n\t\t\\label{eq:3point}\n\t\t\\begin{aligned}\n\t\t\t&  \\left[f(x^{++}) + \\phi(x^{++}) + \\frac{\\sigma}{2}\\|x^{++} - \\xu\\|^2\\right] - \\left[f(u) + \\phi(u) + \\frac{\\sigma}{2}\\|u - \\xu\\|^2\\right]\n\t\t\t\\\\\n\t\t\t\\le\\ & \\frac{\\eta}{2}\\|u - x\\|^2 - \\frac{\\sigma + \\eta}{2}\\|u - x^{++}\\|^2 - \\frac{\\eta-M}{2}\\|x^{++} - x\\|^2 ,\\ \\forall u\\in X.\n\t\t\\end{aligned}\n\t\\end{align}\n\\end{lem}\n\n\\begin{proof}\n\tLet us denote $x^+:=x^+(\\eta+\\sigma; x)$, $x^{++}:=x^{++}(\\eta, \\sigma, \\xu; x)$ and $x' := x^+(\\eta; x)$. By the optimality conditions of \\eqref{eq:xp} and \\eqref{eq:xpp} we have \n\t\\begin{align}\n\t\t\\langle \\nabla f(x) + (\\eta+\\sigma)(x^+ - x), u - x^{+}\\rangle + \\phi(u) - \\phi(x^{+}) \\ge 0, \\label{eq:oc_simple_1} \\\\\n\t\t\\langle \\nabla f(x) + \\sigma (x^{++}-\\bar x) +\\eta (x^{++} - x), u - x^{++}\\rangle + \\phi(u) - \\phi(x^{++})\\ge 0, \\label{eq:oc_simple_2} \\\\\n\t\t\\langle \\nabla f(x) + \\eta (x' - x), u - x'\\rangle + \\phi(u) - \\phi(x') \\ge 0 \\label{eq:oc_simple_3}\n\t\\end{align}\n\tfor any $u \\in X$.\n\tTo prove \\eqref{eq:xp_lip}, let us set $u = x^{++}$ in \\eqref{eq:oc_simple_1}, $u=\\xp$ in \\eqref{eq:oc_simple_2}, and add the two relations together. It is easy to see that\n\t$\n\t\\langle (\\eta+\\sigma) (x^+ - x^{++})\n\t+ \\sigma (\\bar x - x), x^{++} - x^+\\rangle  \\ge 0 \n\t$, or equivalently,\n\t$\n\t\\sigma \\langle \\bar x - x, x^{++} - x^+\\rangle \\ge (\\eta+\\sigma) \\|x^+ - x^{++}\\|^2,\n\t$\n\twhich implies \\eqref{eq:xp_lip} by the Cauchy-Schwarz inequality.\n\tNow notice that by definition\n\tthe relation \\eqref{eq:xp_monotonicity} is equivalent to $\\eta\\|x - x'\\|\\le (\\sigma + \\eta)\\|x - x^{+}\\|$.\n\tLet us set $u = x'$ in \\eqref{eq:oc_simple_1}, $u=\\xp$ in \\eqref{eq:oc_simple_3}, and add the two relations together.\n\tThen we have \n\t$\\sigma \\langle x^+ - x, x' - x^+\\rangle\n\t- \\eta \\|x^+ - x'\\|^2 \\ge 0$,\n\twhich implies \n\t$\\eta \\|x^+ - x'\\| \\le \\sigma \\|x - x^+ \\|$ again by the Cauchy-Schwarz inequality.\n\tThe previous inequality together with the triangular inequality then immediately implies\n\t\\[\n\t\\eta\\|x - x'\\| \n\t\\le \\eta (\\|x - x^+\\|\n\t+\\|x^+ - x'\\|) \\le (\\sigma + \\eta) \\|x - x^+ \\|.\n\t\\]\n\tIn order to show \\eqref{eq:3point}, we first notice that\n\t\\eqref{eq:oc_simple_2} is equivalent to \n\t\n\t\\begin{align*}\n\t\t\\begin{aligned}\n\t\t\t\\langle \\nabla f(x), x^{++} - u\\rangle + \\phi(x^{++}) - \\phi(u)\n\t\t\t&\\le ({\\sigma}/{2})(\\|u - \\xu\\|^2 - \\|x^{++} - \\xu\\|^2 - \\|u - x^{++}\\|^2) \\\\\n\t\t\t& + ({\\eta}/{2})(\\|u - x\\|^2 - \\|x^{++} - x\\|^2 - \\|u - x^{++}\\|^2)\n\t\t\\end{aligned}\n\t\\end{align*}\t\n\tfor any $u\\in X$.\n\tMoreover, from \\eqref{eq:local_lip} and the convexity of $f$ we have\n\t\\begin{align*}\n\t\tf(\\xpp) &\\le f(x) + \\langle \\nabla f(x), \\xpp - x\\rangle + ({M}/{2})\\|\\xpp - x\\|^2\\\\\n\t\t&\\le f(x) + \\langle \\nabla f(x), u - x\\rangle \n\t\t+ \\langle \\nabla f(x), \\xpp - u\\rangle\n\t\t+({M}/{2})\\|\\xpp - x\\|^2\\\\\n\t\t&\\le f(u) + \\langle \\nabla f(x), \\xpp - u\\rangle\n\t\t+({M}/{2})\\|\\xpp - x\\|^2.\n\t\\end{align*} \n\tCombining the above two relations we conclude \\eqref{eq:3point}.\n\\qedsymbol\\end{proof}\n\n\\begin{lemma}\n\t\\label{thm:Mlemma}\n\tSuppose that $g:\\R^n\\to\\R$ is a convex smooth function.\n\tFor any $x\\in\\R^n$, $M>0$, $\\eta\\ge 2M$, and $\\xpp:=x - (1/\\eta)\\nabla g(x)$,\n\tif $\\xpp$ and $x$ satisfy\n\t\\begin{align}\n\t\t\\label{eq:M}\n\t\tg(\\xpp) - g(x) - \\langle \\nabla g(x), \\xpp - x\\rangle \\le \\frac{M}{2}\\|\\xpp - x\\|^2,\n\t\\end{align}\n\tthen we have $\\|\\nabla g(x)\\|\\le \\sqrt{2}\\eta \\|x - x^*\\|$, where $x^*$ is any minimizer of function $g$.\n\\end{lemma}\n\n\\begin{proof}\n\tNote that our definition of $\\xpp$ is exactly $\\xpp(\\eta,0,\\xu;x)$ described in \\eqref{eq:xpp} (with $X\\in\\R^n$, $\\sigma=0$, and $\\phi\\equiv 0$). Applying result \\eqref{eq:3point} in Lemma~\\ref{lem:grad_map_properties} and noting that $\\eta\\ge 2M$ we have\n\t\\begin{align*}\n\t\tg(\\xpp) - g(u) \\le \\frac{\\eta}{2}\\|u - x\\|^2 - \\frac{\\eta}{2}\\|u - \\xpp\\|^2 - \\frac{\\eta}{4}\\|\\xpp - x\\|^2, \\ \\forall u\\in \\R^n.\n\t\\end{align*}\n\tLetting $u=x^*$ in the above relation, using the optimality of $x^*$ and the fact that $\\xpp - x = -(1/\\eta)\\nabla g(x)$ we conclude that \n\t$\t\\|\\nabla g(x)\\|^2 \\le 2\\eta^2\\|x^* -x\\|^2$.\n\\qedsymbol\\end{proof}\n\nThe above lemma allows us to design a backtracking function described in Algorithm~\\ref{alg:_searchL}. Applying Lemma~\\ref{thm:Mlemma} with $x = x_s$, $x^* = x_s^*$, $g = f_s$, $M=M_s+\\sigma_s$, and $\\eta = 2(M_s+\\sigma_s)$, we conclude that whenever \\eqref{eq:M} holds, \nwe have $\\|\\nabla f_s(x_s)\\|\\le 2\\sqrt{2}(M_s+\\sigma_s)\\|x_s - x_s^*\\|$. As a consequence,\n\\begin{align}\n\t\\label{eq:xs_norm_bound_unconstrained_est}\n\t\\begin{aligned}\n\t\t& \\|\\nabla f(x_s)\\| = \\|\\nabla f_s(x_s) - \\sigma_s(x_s - \\xu_s)\\| \n\t\t=  \\|\\nabla f_s(x_s) - \\sigma_s(x_s - x_s^*) - \\sigma_s(\\xu_s-x_s^*)\\| \n\t\t\\\\\n\t\t\\le &  (2\\sqrt{2}M_s+(2\\sqrt{2}+1)\\sigma_s)\\|x_s - x_s^*\\| + \\sigma_s\\|x_s^* - \\xu_s\\|,\n\t\\end{aligned}\n\\end{align}\nwhich is analogous to the gradient-to-distance relation \\eqref{eq:xs_norm_bound_unconstrained} but with $L$ replaced by $M_s$.\nApparently, we can apply one single gradient descent step with backtracking, i.e., Algorithm~\\ref{alg:_searchL} with input $g=f_s$, $\\sigma =\\sigma_s$, $x= x_s$ and initial guess $M_0$, to estimate the local Lipschitz constant $M_s$.\n\n\n\\begin{theorem}\n\t\\label{thm:conv_proposed_unconstrained_searchL} \n\t\tFor any input argument $M_0, \\sigma_1>0$, the AR method outputs an $\\hat x$ with $\\|\\nabla f(\\hat x)\\|\\le 5\\sigma_1\\operatorname{dist}(x_0, X^*)$ within $4 + 3\\sqrt{\\max\\{M_0/2, 2L\\}/\\sigma_1}+16\\sqrt{2c_\\cA }\\sqrt{L/\\sigma_1}$ gradient evaluations of $\\nabla f$.\n\t\tSpecially, if $M_0\\le 4L$, the total number of gradient evaluations is bounded by $4 + C_1\\sqrt{L/\\sigma_1}$, where $C_1:= \\sqrt{2}(3 + 16\\sqrt{2c_\\cA})$ is a universal constant. \n\t\tIf in addition $\\sigma_1 = \\varepsilon/(5D)$ for some $D>0$, then an approximate solution $\\hat x$ satisfying $\\|\\nabla f(\\hat x)\\|\\le \\varepsilon\\operatorname{dist}(x_0, X^*)/D$  can be computed within at most  $4 + \\sqrt{5}C_1\\sqrt{LD/\\varepsilon}$ gradient evaluations of $\\nabla f$. \n\t\\end{theorem}\n\n\\begin{proof}\n\t\tFirst note that by \\eqref{eq:dist_xu_xstarsig_unconstrained} in Proposition~\\ref{pro:xstar_dist_shrink_unconstrained} and the relation in \\eqref{eq:xs_norm_bound_unconstrained_est}, we have \n\t\t\\begin{align}\n\t\t\t\\label{eq:gradS_estimate_unconstrained_searchL}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\|\\nabla f(x_s)\\|\n\t\t\t\t&\t\\le (2\\sqrt{2}M_s + (1+2\\sqrt{2})\\sigma_s)\\|x_s -x_s^*\\| \\\\\n\t\t\t\t&\\quad + \\sigma_1\\|x^* - x_0\\| + \\sum_{i=2}^s(\\sigma_{i-1} + \\sigma_i)\\|x_{i-1}^* - x_{i-1}\\|,\n\t\t\t\\end{aligned}\n\t\t\\end{align}\n\t\twhich is analogous to \\eqref{eq:gradS_estimate_unconstrained}.\n\t\tConsider the call to the subroutine $\\cA$ for minimizing $f_s(x)$ in \\eqref{eq:subproblem_unconstrained_searchL}. Noting that $f_s$ is $\\sigma_s$-strongly convex, we have $\\|x - x_s^*\\|^2 \\le ({2}/{\\sigma_s})(f_s(x) - f_s(x_s^*)),\\ \\forall x\\in\\R^n$. In view of this observation and Assumption~\\ref{assum:A_unconstrained_searchL}, after the $k$-th gradient evaluation of $\\nabla f$, subroutine $\\cA$ will compute an approximation solution $x_s^k$ such that\n\t\t\\begin{align}\n\t\t\t\\label{eq:A_dist_shrink}\n\t\t\t\\|x_s^k - x_s^*\\|\\le\\dfrac{1}{k}\\sqrt{\\dfrac{2L_s^k}{\\sigma_s}}\\|x_{s-1} - x_{s}^*\\|\\le  \\dfrac{1}{k}\\sqrt{\\dfrac{2c_\\cA L}{\\sigma_s}}\\|x_{s-1} - x_{s}^*\\|.\n\t\t\\end{align}\n\t\tAlso by\tAssumption~\\ref{assum:A_unconstrained_searchL}, the number of gradient evaluations that $\\cA$ performs is bounded by \n\t\t\\begin{align}\n\t\t\t\\label{eq:Ns}\n\t\t\tN_s:=\\lceil8\\sqrt{2L_s/\\sigma_s}\\rceil\\le \\lceil8\\sqrt{2c_\\cA L/\\sigma_s}\\rceil.\n\t\t\\end{align} \n\t\tBy the above two relations and \\eqref{eq:dist_subproblem_unconstrained} in  \n\t\tProposition~\\ref{pro:xstar_dist_shrink_unconstrained}, the output solution $x_s$ computed at the termination of subroutine $\\cA$ applied to minimize $f_s$ satisfies\n\t\t\\begin{align*}\n\t\t\t\\|x_s - x_s^*\\| \\le \\frac{1}{8}\\|x_{s-1} - x_s^*\\| \\le \\frac{1}{8}\\|x_{s-1} - x_{s-1}^*\\|,\\ \\forall s = 1,\\ldots, S.\n\t\t\\end{align*}\n\t\tUsing the above result and the fact that $\\sigma_s = 4\\sigma_{s-1}$ for all $s>1$ in our previous estimate of $\\|\\nabla f(x_s)\\|$ in  \\eqref{eq:gradS_estimate_unconstrained_searchL}, we have\n\t\t\\begin{align}\n\t\t\t\\label{eq:tmp_grad_est}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\|\\nabla f(x_s)\\| \\le\\ & (2\\sqrt{2}M_s + (1+2\\sqrt{2})\\sigma_s)8^{-s}\\|x^* - x_0\\|  \n\t\t\t\t\\\\\n\t\t\t\t&\\ + \\sigma_1\\|x^* - x_0\\| + \\sum_{i=2}^{s}5\\cdot 4^{i-2}\\sigma_1 8^{-i+1}\\|x^* - x_0\\|\n\t\t\t\t\\\\\n\t\t\t\t\\le\\ & (\\sqrt{2}M_s + (1/2+\\sqrt{2})\\sigma_s) 4^{-s}\\|x_0 - x^*\\| + (9/4)\\sigma_1\\|x_0 - x^*\\|.\n\t\t\t\\end{aligned}\n\t\t\\end{align}\n\t\tHere $x^*$ is any solution in the optimal solution set $X^*$. \n\t\t\n\t\tWe will now estimate $\\sigma_s$ and $4^{-s}$ in the above relation when the termination criterion $\\sigma_s\\ge M_s$ is satisfied. \n\t\tIf the criterion is satisfied when $s=1$, by \\eqref{eq:tmp_grad_est} the output $\\hat x=x_1$ satisfies\n\t\t\\begin{align*}\n\t\t\t\\|\\nabla f(\\hat x)\\| \\le (1/2+2\\sqrt{2})\\sigma_1 4^{-1}\\|x_0 - x^*\\| + (9/4)\\sigma_1\\|x_0 - x^*\\|\\le 5\\sigma_1\\|x_0 - x^*\\|.\n\t\t\\end{align*}\n\t\tOtherwise, let $S>1$ be the smallest $s$ when the termination criterion $\\sigma_s\\ge M_s$ is satisfied. Observe in the backtracking function that $M_s\\le\\max\\{M_{s-1}/2, 2L\\}$. Thus we have $M_s\\le \\max\\{M_0/2, 2L\\}$ for $s=1,\\ldots, S$. Use this observation and recalling that $\\sigma_s = 4\\sigma_{s-1}$, we have \n\t\t\\begin{align}\n\t\t\t\\label{eq:sigmaS_bound}\n\t\t\tM_S\\le 4^{S-1}\\sigma_1 = \\sigma_{S} = 4\\sigma_{S-1}<4M_{S-1}\\le \\max\\{2M_0, 8L\\}.\n\t\t\\end{align}\n\t\tThus by \\eqref{eq:tmp_grad_est}, during termination the output $\\hat x = x_S$ satisfies\n\t\t\\begin{align*}\n\t\t\t\\|\\nabla f(\\hat x)\\| \\le & (1/2+2\\sqrt{2})\\sigma_S 4^{-S}\\|x_0 - x^*\\| + (9/4)\\sigma_1\\|x_0 - x^*\\|\n\t\t\t\\le 5\\sigma_1\\|x_0 - x^*\\|.\n\t\t\\end{align*}\n\t\tSince $x^*$ here is an arbitrary optimal solution, we have $\\|\\nabla f(\\hat x)\\|\\le 5\\sigma_1\\operatorname{dist}(x_0, X^*)$.\t\n\t\t\n\t\tNext we will bound the total number of gradient estimations. \n\t\tIf the termination criterion is satisfied when $s=1$,  bounding the numbers of gradient evaluations required for backtracking line-search and subroutine $\\cA$ respectively, we have the bound $1 + N_1$. Recalling the value of $N_s$ and its bound in \\eqref{eq:Ns}, this number is bounded by $ 1 + 8\\sqrt{2c_A L/\\sigma_1}$. If the termination criterion is satisfied at some $s=S>1$,  the total number of gradient evaluations is $S+\\sum_{s=1}^{S}N_s$. Here by \\eqref{eq:sigmaS_bound} we have $S\\le 2 + \\log_4(\\max\\{M_0/2, 2L\\}/\\sigma_1)$. Also, recalling the value of $N_s$ and its bound in \\eqref{eq:Ns} and noting that $\\sigma_s = 4\\sigma_{s-1}$ for all $s$ we have\n\t\t\\begin{align*}\n\t\t\t\\sum_{s=1}^{S}N_s \\le \\sum_{s=1}^{S}1 + 8\\sqrt{2c_\\cA L/\\sigma_s} \\le & S + 8\\sqrt{2c_\\cA L/\\sigma_1}\\sum_{s=1}^{\\infty}2^{-(s-1)}  \\le S + 16\\sqrt{2c_\\cA L/\\sigma_1}.\n\t\t\\end{align*}      \n\t\tSummarizing the above we can bound the total number of gradient evaluations by\n\t\t\\begin{align}\n\t\t\t\\label{eq:total_gradient}\n\t\t\t\\begin{aligned}\n\t\t\t\t& S+\\sum_{s=1}^{S}N_s \n\t\t\t\t\\le 2S + 16\\sqrt{2c_\\cA L/\\sigma_1} \\le 2(2 + \\log_4(\\max\\{M_0/2, 2L\\}/\\sigma_1)) + 16\\sqrt{2c_\\cA L/\\sigma_1}\n\t\t\t\t\\\\ \n\t\t\t\t\\le\\ & 4 + 3\\sqrt{\\max\\{M_0/2, 2L\\}/\\sigma_1}+16\\sqrt{2c_\\cA }\\sqrt{L/\\sigma_1}. \n\t\t\t\\end{aligned}\n\t\t\\end{align}\n\t\tHere in the last inequality we use the fact that $2\\log_4 t \\cmo{<} 3\\sqrt{t}$ for all $t>0$.\n\t\tNote that the above bound is also an upper bound of number of gradient evaluations if the termination criterion is satisfied when $s=1$. \n\t\tThe remainder parts of result then immediately follows from \\eqref{eq:total_gradient}.\n\t\\qedsymbol\\end{proof}\n\n\\begin{proof}\n\t\tWe start with three observations. First, by Theorem~\\ref{thm:conv_proposed_unconstrained_searchL}, \n\t\tthe output $y_{t}$ of the AR method satisfies $\\|\\nabla f(y_{t})\\|\\le \\mu_{t-1}\\|y_{t-1}-x^*\\|/2$ and the total number of gradient evaluations for computing $y_t$ is bounded by\n\t\t$4 +  C_1\\sqrt{10L/\\mu_{t-1}}$.\n\t\tSecond, using the previous observation and \n\t\tthe relation \n  \\begin{equation}\\label{eq:growth}\n  \\|y_{t-1}-x^*\\|\\le \\tfrac{1}{\\mu}\\|\\nabla f(y_{t-1})\\|\n  \\end{equation}\n  due to the strong convexity of $f$, we conclude that $\\|\\nabla f(y_{t})\\|\\le \\|\\nabla f(y_{t-1})\\|/2$  and hence that $\\mu_t$ will no longer change whenever $\\mu_{t-1}\\le \\mu$. Therefore $\\mu_t\\ge \\mu/4$ for all $t$.\n\t\tThird, the algorithm will terminate with $\\|\\nabla f(y_t)\\|\\le \\varepsilon$ whenever $t\\ge P:=\\lceil\\log_4(\\mu_0/\\mu)\\rceil + \\lceil\\log_2(\\|\\nabla f(y_0)\\|/\\varepsilon)\\rceil$. Here $P$ is the sum of two parts: $\\lceil\\log_4(\\mu_0/\\mu)\\rceil$ counts the number of times when $\\|\\nabla f(y_t)\\|>\\|\\nabla f(y_{t-1})\\|/2$ and the value of $\\mu_t$ is reduced from $\\mu_{t-1}$; $\\lceil\\log_2(\\|\\nabla f(y_0)\\|/\\varepsilon)\\rceil$ counts the number of times when $\\|\\nabla f(y_t)\\|\\le\\|\\nabla f(y_{t-1})\\|/2$. Summarizing the above observations, the total number of gradient evaluations $N$ is bounded by\n\t\t\\begin{align*}\n\t\t\tN \\le\\ & 4P + C_1P\\sqrt{40L/\\mu} \\le (4+8\\sqrt{5}C_1)P\\sqrt{L/\\mu}\n\t\t\t\\\\\n\t\t\t\\le\\ & (4+8\\sqrt{5}C_1)\\lceil\\log_4(\\mu_0/\\mu)\\rceil\\sqrt{L/\\mu}  + (4+8\\sqrt{5}C_1)\\sqrt{L/\\mu}\\lceil\\log_2(\\|\\nabla f(y_0)\\|/\\varepsilon)\\rceil.\n\t\t\\end{align*}\n\t\n\t\\cmo{\n\t\tWe may also analyze the total number of gradient evaluations from an alternative perspective when we know that  $\\mu_0\\le 2^{j}\\varepsilon/\\operatorname{dist}(y_0,X^*)$. By   Theorem~\\ref{thm:conv_proposed_unconstrained_searchL}, the first AR call is guaranteed to yield $y_1$ such that $\\|\\nabla f(y_1)\\| \\le 2^{j-1}\\varepsilon$. If $j\\le 1$, then $y_1$ already satisfies $\\|\\nabla f(y_1)\\|\\le\\varepsilon$. If $j>1$,\n\t\tthen after $(j-1)$ occurrences of $\\|\\nabla f(y_t)\\| \\le \\|\\nabla f(y_{t-1})\\|/2$ we will have $\\|\\nabla f(y_t)\\|\\le \\varepsilon$. Therefore, the algorithm will terminate with $\\|\\nabla f(y_t)\\|\\le \\varepsilon$ whenever $t\\ge P:=\\lceil\\log_4(\\mu_0/\\mu)\\rceil + j$ and we can bound $N$ by $N \\le (4+8\\sqrt{5}C_1)P\\sqrt{L/\\mu}$ similar as the proof of the previous bound.\n\t}\t\n\t\n\t\\qedsymbol\n\t\\end{proof}","units":[{"start":25781,"end":27502,"start_line":331,"end_line":355,"raw_tex":"Specifically, \nnoting the $L$-Lipschitz continuity of $\\nabla f(x)$ and using the optimality condition $\\nabla f_s(x_s^*)=\\nabla f(x_s^*) + \\sigma_s(x_s^* - \\xu_s) = 0$ at $x_s^*$,\nwe have\n\\begin{align}\n\t\\label{eq:xs_norm_bound_unconstrained}\n\t\\begin{aligned}\n\t\t& \\|\\nabla f(x_s)\\| = \\|\\nabla f(x_s) - \\nabla f(x_s^*) - \\sigma_s(x_s^* - \\xu_s)\\| \n\t\t\\\\\n\t\t\\le \\ & \\|\\nabla f(x_s) - \\nabla f(x_s^*)\\| + \\sigma_s\\|x_s^* - \\xu_s\\| \n\t\t\\\\\n\t\t\\le\\ &  L\\|x_s - x_s^*\\| + \\sigma_s\\|x_s^* - \\xu_s\\|.\n\t\\end{aligned}\n\\end{align}\nWe will show later that with the increasing regularization parameter $\\sigma_s$ and convex combination design of prox-center in \\eqref{eq:xus_unconstrained}, the distance $\\sigma_s\\|x_s^* - \\xu_s\\|$ can be further bounded:\n\\begin{align}\n\t\\label{eq:claim_on_distance_unconstrained}\n\t\\sigma_s\\|x_s^* - \\xu_s\\| \\le \\sigma_1\\|x^* - x_0\\| + \\sum_{i=2}^s(\\sigma_{i-1} + \\sigma_i)\\|x_{i-1}^* - x_{i-1}\\|.\n\\end{align}\nTherefore, $\\|\\nabla f(x_s)\\|$ can be controled by\n\\begin{align}\n\t\\label{eq:gradS_estimate_unconstrained}\n\t\\|\\nabla f(x_s)\\| \\le\\ &  L\\|x_s - x_s^*\\| + \\sigma_1\\|x^* - x_0\\| + \\sum_{i=2}^s(\\sigma_{i-1} + \\sigma_i)\\|x_{i-1}^* - x_{i-1}\\|.\n\\end{align}\nNote that the right-hand-side of the above estimate is the summation of terms of form $\\|x_i - x_i^*\\|$, i.e., the distance from an approximate solution $x_i$ to the optimal solution $x_i^*$ of the $i$-th strongly convex subproblem.\nBy studying the convergence properties of subroutine $\\cA$ for minimizing smooth and strongly convex functions in subproblem \\eqref{eq:subproblem_unconstrained}, we will then bound the total number of gradient evaluations needed to reduce the right-hand-side of the above estimate to the accuracy $\\varepsilon$. \n","sha256_utf8":"e1ac878f85392e104bf9d9c6a7a60f9407301820d98585dc277b9d8073e290c2"},{"start":28211,"end":28645,"start_line":362,"end_line":374,"raw_tex":"\\begin{proposition}\n\t\\label{pro:xstar_dist_shrink_unconstrained}\n\tIn  Algorithm~\\ref{alg:proposed_unconstrained} we have for all $s\\ge 1$ \n\t\\begin{align}\n\t\t\\label{eq:dist_subproblem_unconstrained}\n\t\t& \\|x_{s-1} - x_s^*\\|\\le \\|x_{s-1} - x_{s-1}^*\\|, \n\t\t\\\\\n\t\t\\label{eq:dist_xu_xstarsig_unconstrained}\n\t\t&\\sigma_s\\|\\xu_s - x_s^*\\| \n\t\t\\le \\sum_{i=1}^{s}(\\sigma_{i-1} + \\sigma_{i})\\|x_{i-1}^* - x_{i-1}\\|\n\t\t.\n\t\\end{align}\n\\end{proposition}","sha256_utf8":"5cdd65ecaffcd0143febadb7d3e3c11f63201010690779e98cd9b7d041898d80"},{"start":28646,"end":31253,"start_line":375,"end_line":413,"raw_tex":"\\begin{proof}\n\tWith the selection of $\\gamma_s = 1 - \\sigma_{s-1}/\\sigma_s$ in the definition of prox-center $\\xu_s$ in   \\eqref{eq:xus_unconstrained}, the subproblem \\eqref{eq:subproblem_unconstrained} is equivalent to\n\t\\begin{align*}\n\t\tx_s^* = \\argmin_{x\\in \\R^n} f(x) + \\dfrac{\\sigma_{s-1}}{2}\\|x - \\xu_{s-1}\\|^2 + \\dfrac{\\sigma_s - \\sigma_{s-1}}{2}\\|x - x_{s-1}\\|^2,\\ \\forall s \\ge 1.\n\t\\end{align*}\n\tNoting the optimality of $x_{s-1}^*$ in \\eqref{eq:subproblem_unconstrained}  and $x_s^*$ in the above subproblem we have\n\t\\begin{align*}\n\t\t& f(x_{s-1}^*) + \\dfrac{\\sigma_{s-1}}{2}\\|x_{s-1}^* - \\xu_{s-1}\\|^2 + \\dfrac{\\sigma_s - \\sigma_{s-1}}{2}\\|x_s^* - x_{s-1}\\|^2\n\t\t\\\\\n\t\t\\le\\ & f(x_s^*) + \\dfrac{\\sigma_{s-1}}{2}\\|x_s^* - \\xu_{s-1}\\|^2 + \\dfrac{\\sigma_s - \\sigma_{s-1}}{2}\\|x_s^* - x_{s-1}\\|^2 \n\t\t\\\\\n\t\t\\le\\ & f(x_{s-1}^*) + \\dfrac{\\sigma_{s-1}}{2}\\|x_{s-1}^* - \\xu_{s-1}\\|^2 + \\dfrac{\\sigma_s - \\sigma_{s-1}}{2}\\|x_{s-1}^* - x_{s-1}\\|^2.\n\t\\end{align*}\n\tThe conclusion in \\eqref{eq:dist_subproblem_unconstrained} then follows from the above relation and the assumption $\\sigma_s>\\sigma_{s-1}$ required by Algorithm~\\ref{alg:proposed_unconstrained}.\n\t\n\tWe move to prove the next result \\eqref{eq:dist_xu_xstarsig_unconstrained}. Denoting $\\alpha_s:=\\sigma_s - \\sigma_{s-1}$ for all $s$ and noting that $\\gamma_s = \\alpha_s/\\sigma_s$ in \\eqref{eq:xus_unconstrained}, we can rewrite the definition of $\\xu_s$ in \\eqref{eq:xus_unconstrained} to\n\t$\n\t\\sigma_s \\xu_s = \\sigma_{s-1}\\xu_{s-1} + \\alpha_s x_{s-1}.\n\t$\n\tRecalling that $\\sigma_0=0$, the above recursive relation yields\n\t$\n\t\\sigma_s\\xu_s = \\sum_{i=1}^{s}\\alpha_ix_{i-1}.\n\t$\n\tIn other words, $\\xu_s$ is a convex combination of $x_0,\\ldots, x_{s-1}$ with weights $\\alpha_i/\\sigma_s$  since $\\sigma_s = \\sum_{i=1}^{s}\\alpha_i$. Thus \n\t\\begin{align*}\n\t\t\\sigma_s(\\xu_s - x_s^*) =\\ & \\sum_{i=1}^{s}\\alpha_i(x_{i-1} - x_s^*) \n\t\t\\\\\n\t\t=\\ & \\alpha_s(x_{s-1} - x_s^*) + \\sum_{i=1}^{s-1}\\alpha_i(x_{i-1} - x_{s-1}^*)  + \\left(\\sum_{i=1}^{s-1}\\alpha_i\\right)(x_{s-1}^* - x_s^*)\n\t\t\\\\\n\t\t=\\ & \\alpha_s(x_{s-1} - x_s^*) + \\sigma_{s-1}(\\xu_{s-1} - x_{s-1}^*) + \\sigma_{s-1}(x_{s-1}^* - x_{s-1}) + \\sigma_{s-1}(x_{s-1} - x_s^*)\n\t\t\\\\\n\t\t=\\ & \\sigma_{s-1}(\\xu_{s-1} - x_{s-1}^*) + \\sigma_{s-1}(x_{s-1}^* - x_{s-1}) + \\sigma_s(x_{s-1} - x_s^*).\n\t\\end{align*}\n\tThe above recursive relation yields\n\t$\n\t\\sigma_s(\\xu_s - x_s^*) = \\sum_{i=1}^{s}[\\sigma_{i-1}(x_{i-1}^* - x_{i-1}) + \\sigma_i(x_{i-1} - x_i^*)].\n\t$\n\tBy the above equality and our previous result \\eqref{eq:dist_subproblem_unconstrained}, we conclude \\eqref{eq:dist_xu_xstarsig_unconstrained}.\n\\qedsymbol\\end{proof}","sha256_utf8":"69683e541c1088e3028a966b08a6d86bef2c7687a25d2987fe549d717fc51854"},{"start":40532,"end":41539,"start_line":520,"end_line":544,"raw_tex":"\\begin{lem}\n\t\\label{lem:grad_map_properties}\n\tFor any $x, \\xu\\in\\R^n$ and $\\eta>0$, $\\sigma\\ge0$, we have the following properties for the (perturbed) gradient mappings:\n\t\\begin{align}\n\t\t\\label{eq:xp_lip}\n\t\t& \\|x^+(\\eta+\\sigma; x) - x^{++}(\\eta, \\sigma, \\xu; x)\\| \\le \\frac{\\sigma}{\\eta+\\sigma}\\|x - \\xu\\|;\n\t\t\\\\\n\t\t\\label{eq:xp_monotonicity}\n\t\t& \\|G_\\eta(x)\\| \\le \\|G_{\\sigma + \\eta}(x)\\|.\n\t\\end{align}\n\tMoreover, if $x^{++}:=x^{++}(\\eta, \\sigma, \\xu; x)$ satisfies \n\t\\begin{align}\n\t\t\\label{eq:local_lip}\n\t\tf(x^{++}) - f(x) - \\langle \\nabla f(x), x^{++} - x\\rangle \\le \\frac{M}{2}\\|x^{++} - x\\|^2\n\t\\end{align}\n\tfor some $M>0$,  we have\n\t\\begin{align}\n\t\t\\label{eq:3point}\n\t\t\\begin{aligned}\n\t\t\t&  \\left[f(x^{++}) + \\phi(x^{++}) + \\frac{\\sigma}{2}\\|x^{++} - \\xu\\|^2\\right] - \\left[f(u) + \\phi(u) + \\frac{\\sigma}{2}\\|u - \\xu\\|^2\\right]\n\t\t\t\\\\\n\t\t\t\\le\\ & \\frac{\\eta}{2}\\|u - x\\|^2 - \\frac{\\sigma + \\eta}{2}\\|u - x^{++}\\|^2 - \\frac{\\eta-M}{2}\\|x^{++} - x\\|^2 ,\\ \\forall u\\in X.\n\t\t\\end{aligned}\n\t\\end{align}\n\\end{lem}","sha256_utf8":"cecb9526383e2e776a3af7ffd9ea131b74e904869ca13e1fb20b60526c2f7ae6"},{"start":41542,"end":44113,"start_line":547,"end_line":599,"raw_tex":"\\begin{proof}\n\tLet us denote $x^+:=x^+(\\eta+\\sigma; x)$, $x^{++}:=x^{++}(\\eta, \\sigma, \\xu; x)$ and $x' := x^+(\\eta; x)$. By the optimality conditions of \\eqref{eq:xp} and \\eqref{eq:xpp} we have \n\t\\begin{align}\n\t\t\\langle \\nabla f(x) + (\\eta+\\sigma)(x^+ - x), u - x^{+}\\rangle + \\phi(u) - \\phi(x^{+}) \\ge 0, \\label{eq:oc_simple_1} \\\\\n\t\t\\langle \\nabla f(x) + \\sigma (x^{++}-\\bar x) +\\eta (x^{++} - x), u - x^{++}\\rangle + \\phi(u) - \\phi(x^{++})\\ge 0, \\label{eq:oc_simple_2} \\\\\n\t\t\\langle \\nabla f(x) + \\eta (x' - x), u - x'\\rangle + \\phi(u) - \\phi(x') \\ge 0 \\label{eq:oc_simple_3}\n\t\\end{align}\n\tfor any $u \\in X$.\n\tTo prove \\eqref{eq:xp_lip}, let us set $u = x^{++}$ in \\eqref{eq:oc_simple_1}, $u=\\xp$ in \\eqref{eq:oc_simple_2}, and add the two relations together. It is easy to see that\n\t$\n\t\\langle (\\eta+\\sigma) (x^+ - x^{++})\n\t+ \\sigma (\\bar x - x), x^{++} - x^+\\rangle  \\ge 0 \n\t$, or equivalently,\n\t$\n\t\\sigma \\langle \\bar x - x, x^{++} - x^+\\rangle \\ge (\\eta+\\sigma) \\|x^+ - x^{++}\\|^2,\n\t$\n\twhich implies \\eqref{eq:xp_lip} by the Cauchy-Schwarz inequality.\n\tNow notice that by definition\n\tthe relation \\eqref{eq:xp_monotonicity} is equivalent to $\\eta\\|x - x'\\|\\le (\\sigma + \\eta)\\|x - x^{+}\\|$.\n\tLet us set $u = x'$ in \\eqref{eq:oc_simple_1}, $u=\\xp$ in \\eqref{eq:oc_simple_3}, and add the two relations together.\n\tThen we have \n\t$\\sigma \\langle x^+ - x, x' - x^+\\rangle\n\t- \\eta \\|x^+ - x'\\|^2 \\ge 0$,\n\twhich implies \n\t$\\eta \\|x^+ - x'\\| \\le \\sigma \\|x - x^+ \\|$ again by the Cauchy-Schwarz inequality.\n\tThe previous inequality together with the triangular inequality then immediately implies\n\t\\[\n\t\\eta\\|x - x'\\| \n\t\\le \\eta (\\|x - x^+\\|\n\t+\\|x^+ - x'\\|) \\le (\\sigma + \\eta) \\|x - x^+ \\|.\n\t\\]\n\tIn order to show \\eqref{eq:3point}, we first notice that\n\t\\eqref{eq:oc_simple_2} is equivalent to \n\t\n\t\\begin{align*}\n\t\t\\begin{aligned}\n\t\t\t\\langle \\nabla f(x), x^{++} - u\\rangle + \\phi(x^{++}) - \\phi(u)\n\t\t\t&\\le ({\\sigma}/{2})(\\|u - \\xu\\|^2 - \\|x^{++} - \\xu\\|^2 - \\|u - x^{++}\\|^2) \\\\\n\t\t\t& + ({\\eta}/{2})(\\|u - x\\|^2 - \\|x^{++} - x\\|^2 - \\|u - x^{++}\\|^2)\n\t\t\\end{aligned}\n\t\\end{align*}\t\n\tfor any $u\\in X$.\n\tMoreover, from \\eqref{eq:local_lip} and the convexity of $f$ we have\n\t\\begin{align*}\n\t\tf(\\xpp) &\\le f(x) + \\langle \\nabla f(x), \\xpp - x\\rangle + ({M}/{2})\\|\\xpp - x\\|^2\\\\\n\t\t&\\le f(x) + \\langle \\nabla f(x), u - x\\rangle \n\t\t+ \\langle \\nabla f(x), \\xpp - u\\rangle\n\t\t+({M}/{2})\\|\\xpp - x\\|^2\\\\\n\t\t&\\le f(u) + \\langle \\nabla f(x), \\xpp - u\\rangle\n\t\t+({M}/{2})\\|\\xpp - x\\|^2.\n\t\\end{align*} \n\tCombining the above two relations we conclude \\eqref{eq:3point}.\n\\qedsymbol\\end{proof}","sha256_utf8":"bf1f2bafede87f0876197b199659513f98e08959d30f4e45f2e3ba67be98e57c"},{"start":59115,"end":59560,"start_line":830,"end_line":840,"raw_tex":"\\begin{lemma}\n\t\\label{thm:Mlemma}\n\tSuppose that $g:\\R^n\\to\\R$ is a convex smooth function.\n\tFor any $x\\in\\R^n$, $M>0$, $\\eta\\ge 2M$, and $\\xpp:=x - (1/\\eta)\\nabla g(x)$,\n\tif $\\xpp$ and $x$ satisfy\n\t\\begin{align}\n\t\t\\label{eq:M}\n\t\tg(\\xpp) - g(x) - \\langle \\nabla g(x), \\xpp - x\\rangle \\le \\frac{M}{2}\\|\\xpp - x\\|^2,\n\t\\end{align}\n\tthen we have $\\|\\nabla g(x)\\|\\le \\sqrt{2}\\eta \\|x - x^*\\|$, where $x^*$ is any minimizer of function $g$.\n\\end{lemma}","sha256_utf8":"fa42cd6899a828c4204c08d0d5f74d2e578c7745a3da5df536e8157a141a932a"},{"start":59561,"end":60201,"start_line":841,"end_line":848,"raw_tex":"\\begin{proof}\n\tNote that our definition of $\\xpp$ is exactly $\\xpp(\\eta,0,\\xu;x)$ described in \\eqref{eq:xpp} (with $X\\in\\R^n$, $\\sigma=0$, and $\\phi\\equiv 0$). Applying result \\eqref{eq:3point} in Lemma~\\ref{lem:grad_map_properties} and noting that $\\eta\\ge 2M$ we have\n\t\\begin{align*}\n\t\tg(\\xpp) - g(u) \\le \\frac{\\eta}{2}\\|u - x\\|^2 - \\frac{\\eta}{2}\\|u - \\xpp\\|^2 - \\frac{\\eta}{4}\\|\\xpp - x\\|^2, \\ \\forall u\\in \\R^n.\n\t\\end{align*}\n\tLetting $u=x^*$ in the above relation, using the optimality of $x^*$ and the fact that $\\xpp - x = -(1/\\eta)\\nabla g(x)$ we conclude that \n\t$\t\\|\\nabla g(x)\\|^2 \\le 2\\eta^2\\|x^* -x\\|^2$.\n\\qedsymbol\\end{proof}","sha256_utf8":"ec211c1012c4728acb1710fb54d71cd8420b7d31ae56fbca4177da7808604aec"},{"start":60739,"end":61803,"start_line":871,"end_line":883,"raw_tex":"The above lemma allows us to design a backtracking function described in Algorithm~\\ref{alg:_searchL}. Applying Lemma~\\ref{thm:Mlemma} with $x = x_s$, $x^* = x_s^*$, $g = f_s$, $M=M_s+\\sigma_s$, and $\\eta = 2(M_s+\\sigma_s)$, we conclude that whenever \\eqref{eq:M} holds, \nwe have $\\|\\nabla f_s(x_s)\\|\\le 2\\sqrt{2}(M_s+\\sigma_s)\\|x_s - x_s^*\\|$. As a consequence,\n\\begin{align}\n\t\\label{eq:xs_norm_bound_unconstrained_est}\n\t\\begin{aligned}\n\t\t& \\|\\nabla f(x_s)\\| = \\|\\nabla f_s(x_s) - \\sigma_s(x_s - \\xu_s)\\| \n\t\t=  \\|\\nabla f_s(x_s) - \\sigma_s(x_s - x_s^*) - \\sigma_s(\\xu_s-x_s^*)\\| \n\t\t\\\\\n\t\t\\le &  (2\\sqrt{2}M_s+(2\\sqrt{2}+1)\\sigma_s)\\|x_s - x_s^*\\| + \\sigma_s\\|x_s^* - \\xu_s\\|,\n\t\\end{aligned}\n\\end{align}\nwhich is analogous to the gradient-to-distance relation \\eqref{eq:xs_norm_bound_unconstrained} but with $L$ replaced by $M_s$.\nApparently, we can apply one single gradient descent step with backtracking, i.e., Algorithm~\\ref{alg:_searchL} with input $g=f_s$, $\\sigma =\\sigma_s$, $x= x_s$ and initial guess $M_0$, to estimate the local Lipschitz constant $M_s$.\n","sha256_utf8":"c1b4fce0593b98f5a22e2bc64d1796f95f5112ed10cc1df39dabe1fd62448d4a"},{"start":66112,"end":66930,"start_line":948,"end_line":953,"raw_tex":"\\begin{theorem}\n\t\\label{thm:conv_proposed_unconstrained_searchL} \n\t\tFor any input argument $M_0, \\sigma_1>0$, the AR method outputs an $\\hat x$ with $\\|\\nabla f(\\hat x)\\|\\le 5\\sigma_1\\operatorname{dist}(x_0, X^*)$ within $4 + 3\\sqrt{\\max\\{M_0/2, 2L\\}/\\sigma_1}+16\\sqrt{2c_\\cA }\\sqrt{L/\\sigma_1}$ gradient evaluations of $\\nabla f$.\n\t\tSpecially, if $M_0\\le 4L$, the total number of gradient evaluations is bounded by $4 + C_1\\sqrt{L/\\sigma_1}$, where $C_1:= \\sqrt{2}(3 + 16\\sqrt{2c_\\cA})$ is a universal constant. \n\t\tIf in addition $\\sigma_1 = \\varepsilon/(5D)$ for some $D>0$, then an approximate solution $\\hat x$ satisfying $\\|\\nabla f(\\hat x)\\|\\le \\varepsilon\\operatorname{dist}(x_0, X^*)/D$  can be computed within at most  $4 + \\sqrt{5}C_1\\sqrt{LD/\\varepsilon}$ gradient evaluations of $\\nabla f$. \n\t\\end{theorem}","sha256_utf8":"c42a3faf8ba4bcab2c45f4cbc68ac107fa53153ebdb68fc501d4181c4c9d7eee"},{"start":66932,"end":72434,"start_line":954,"end_line":1028,"raw_tex":"\\begin{proof}\n\t\tFirst note that by \\eqref{eq:dist_xu_xstarsig_unconstrained} in Proposition~\\ref{pro:xstar_dist_shrink_unconstrained} and the relation in \\eqref{eq:xs_norm_bound_unconstrained_est}, we have \n\t\t\\begin{align}\n\t\t\t\\label{eq:gradS_estimate_unconstrained_searchL}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\|\\nabla f(x_s)\\|\n\t\t\t\t&\t\\le (2\\sqrt{2}M_s + (1+2\\sqrt{2})\\sigma_s)\\|x_s -x_s^*\\| \\\\\n\t\t\t\t&\\quad + \\sigma_1\\|x^* - x_0\\| + \\sum_{i=2}^s(\\sigma_{i-1} + \\sigma_i)\\|x_{i-1}^* - x_{i-1}\\|,\n\t\t\t\\end{aligned}\n\t\t\\end{align}\n\t\twhich is analogous to \\eqref{eq:gradS_estimate_unconstrained}.\n\t\tConsider the call to the subroutine $\\cA$ for minimizing $f_s(x)$ in \\eqref{eq:subproblem_unconstrained_searchL}. Noting that $f_s$ is $\\sigma_s$-strongly convex, we have $\\|x - x_s^*\\|^2 \\le ({2}/{\\sigma_s})(f_s(x) - f_s(x_s^*)),\\ \\forall x\\in\\R^n$. In view of this observation and Assumption~\\ref{assum:A_unconstrained_searchL}, after the $k$-th gradient evaluation of $\\nabla f$, subroutine $\\cA$ will compute an approximation solution $x_s^k$ such that\n\t\t\\begin{align}\n\t\t\t\\label{eq:A_dist_shrink}\n\t\t\t\\|x_s^k - x_s^*\\|\\le\\dfrac{1}{k}\\sqrt{\\dfrac{2L_s^k}{\\sigma_s}}\\|x_{s-1} - x_{s}^*\\|\\le  \\dfrac{1}{k}\\sqrt{\\dfrac{2c_\\cA L}{\\sigma_s}}\\|x_{s-1} - x_{s}^*\\|.\n\t\t\\end{align}\n\t\tAlso by\tAssumption~\\ref{assum:A_unconstrained_searchL}, the number of gradient evaluations that $\\cA$ performs is bounded by \n\t\t\\begin{align}\n\t\t\t\\label{eq:Ns}\n\t\t\tN_s:=\\lceil8\\sqrt{2L_s/\\sigma_s}\\rceil\\le \\lceil8\\sqrt{2c_\\cA L/\\sigma_s}\\rceil.\n\t\t\\end{align} \n\t\tBy the above two relations and \\eqref{eq:dist_subproblem_unconstrained} in  \n\t\tProposition~\\ref{pro:xstar_dist_shrink_unconstrained}, the output solution $x_s$ computed at the termination of subroutine $\\cA$ applied to minimize $f_s$ satisfies\n\t\t\\begin{align*}\n\t\t\t\\|x_s - x_s^*\\| \\le \\frac{1}{8}\\|x_{s-1} - x_s^*\\| \\le \\frac{1}{8}\\|x_{s-1} - x_{s-1}^*\\|,\\ \\forall s = 1,\\ldots, S.\n\t\t\\end{align*}\n\t\tUsing the above result and the fact that $\\sigma_s = 4\\sigma_{s-1}$ for all $s>1$ in our previous estimate of $\\|\\nabla f(x_s)\\|$ in  \\eqref{eq:gradS_estimate_unconstrained_searchL}, we have\n\t\t\\begin{align}\n\t\t\t\\label{eq:tmp_grad_est}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\|\\nabla f(x_s)\\| \\le\\ & (2\\sqrt{2}M_s + (1+2\\sqrt{2})\\sigma_s)8^{-s}\\|x^* - x_0\\|  \n\t\t\t\t\\\\\n\t\t\t\t&\\ + \\sigma_1\\|x^* - x_0\\| + \\sum_{i=2}^{s}5\\cdot 4^{i-2}\\sigma_1 8^{-i+1}\\|x^* - x_0\\|\n\t\t\t\t\\\\\n\t\t\t\t\\le\\ & (\\sqrt{2}M_s + (1/2+\\sqrt{2})\\sigma_s) 4^{-s}\\|x_0 - x^*\\| + (9/4)\\sigma_1\\|x_0 - x^*\\|.\n\t\t\t\\end{aligned}\n\t\t\\end{align}\n\t\tHere $x^*$ is any solution in the optimal solution set $X^*$. \n\t\t\n\t\tWe will now estimate $\\sigma_s$ and $4^{-s}$ in the above relation when the termination criterion $\\sigma_s\\ge M_s$ is satisfied. \n\t\tIf the criterion is satisfied when $s=1$, by \\eqref{eq:tmp_grad_est} the output $\\hat x=x_1$ satisfies\n\t\t\\begin{align*}\n\t\t\t\\|\\nabla f(\\hat x)\\| \\le (1/2+2\\sqrt{2})\\sigma_1 4^{-1}\\|x_0 - x^*\\| + (9/4)\\sigma_1\\|x_0 - x^*\\|\\le 5\\sigma_1\\|x_0 - x^*\\|.\n\t\t\\end{align*}\n\t\tOtherwise, let $S>1$ be the smallest $s$ when the termination criterion $\\sigma_s\\ge M_s$ is satisfied. Observe in the backtracking function that $M_s\\le\\max\\{M_{s-1}/2, 2L\\}$. Thus we have $M_s\\le \\max\\{M_0/2, 2L\\}$ for $s=1,\\ldots, S$. Use this observation and recalling that $\\sigma_s = 4\\sigma_{s-1}$, we have \n\t\t\\begin{align}\n\t\t\t\\label{eq:sigmaS_bound}\n\t\t\tM_S\\le 4^{S-1}\\sigma_1 = \\sigma_{S} = 4\\sigma_{S-1}<4M_{S-1}\\le \\max\\{2M_0, 8L\\}.\n\t\t\\end{align}\n\t\tThus by \\eqref{eq:tmp_grad_est}, during termination the output $\\hat x = x_S$ satisfies\n\t\t\\begin{align*}\n\t\t\t\\|\\nabla f(\\hat x)\\| \\le & (1/2+2\\sqrt{2})\\sigma_S 4^{-S}\\|x_0 - x^*\\| + (9/4)\\sigma_1\\|x_0 - x^*\\|\n\t\t\t\\le 5\\sigma_1\\|x_0 - x^*\\|.\n\t\t\\end{align*}\n\t\tSince $x^*$ here is an arbitrary optimal solution, we have $\\|\\nabla f(\\hat x)\\|\\le 5\\sigma_1\\operatorname{dist}(x_0, X^*)$.\t\n\t\t\n\t\tNext we will bound the total number of gradient estimations. \n\t\tIf the termination criterion is satisfied when $s=1$,  bounding the numbers of gradient evaluations required for backtracking line-search and subroutine $\\cA$ respectively, we have the bound $1 + N_1$. Recalling the value of $N_s$ and its bound in \\eqref{eq:Ns}, this number is bounded by $ 1 + 8\\sqrt{2c_A L/\\sigma_1}$. If the termination criterion is satisfied at some $s=S>1$,  the total number of gradient evaluations is $S+\\sum_{s=1}^{S}N_s$. Here by \\eqref{eq:sigmaS_bound} we have $S\\le 2 + \\log_4(\\max\\{M_0/2, 2L\\}/\\sigma_1)$. Also, recalling the value of $N_s$ and its bound in \\eqref{eq:Ns} and noting that $\\sigma_s = 4\\sigma_{s-1}$ for all $s$ we have\n\t\t\\begin{align*}\n\t\t\t\\sum_{s=1}^{S}N_s \\le \\sum_{s=1}^{S}1 + 8\\sqrt{2c_\\cA L/\\sigma_s} \\le & S + 8\\sqrt{2c_\\cA L/\\sigma_1}\\sum_{s=1}^{\\infty}2^{-(s-1)}  \\le S + 16\\sqrt{2c_\\cA L/\\sigma_1}.\n\t\t\\end{align*}      \n\t\tSummarizing the above we can bound the total number of gradient evaluations by\n\t\t\\begin{align}\n\t\t\t\\label{eq:total_gradient}\n\t\t\t\\begin{aligned}\n\t\t\t\t& S+\\sum_{s=1}^{S}N_s \n\t\t\t\t\\le 2S + 16\\sqrt{2c_\\cA L/\\sigma_1} \\le 2(2 + \\log_4(\\max\\{M_0/2, 2L\\}/\\sigma_1)) + 16\\sqrt{2c_\\cA L/\\sigma_1}\n\t\t\t\t\\\\ \n\t\t\t\t\\le\\ & 4 + 3\\sqrt{\\max\\{M_0/2, 2L\\}/\\sigma_1}+16\\sqrt{2c_\\cA }\\sqrt{L/\\sigma_1}. \n\t\t\t\\end{aligned}\n\t\t\\end{align}\n\t\tHere in the last inequality we use the fact that $2\\log_4 t \\cmo{<} 3\\sqrt{t}$ for all $t>0$.\n\t\tNote that the above bound is also an upper bound of number of gradient evaluations if the termination criterion is satisfied when $s=1$. \n\t\tThe remainder parts of result then immediately follows from \\eqref{eq:total_gradient}.\n\t\\qedsymbol\\end{proof}","sha256_utf8":"16c13686df43bceb701b689ea62343d9007b7f709036477b13f1e51c5e25d4b5"},{"start":84552,"end":86963,"start_line":1120,"end_line":1143,"raw_tex":"\\begin{proof}\n\t\tWe start with three observations. First, by Theorem~\\ref{thm:conv_proposed_unconstrained_searchL}, \n\t\tthe output $y_{t}$ of the AR method satisfies $\\|\\nabla f(y_{t})\\|\\le \\mu_{t-1}\\|y_{t-1}-x^*\\|/2$ and the total number of gradient evaluations for computing $y_t$ is bounded by\n\t\t$4 +  C_1\\sqrt{10L/\\mu_{t-1}}$.\n\t\tSecond, using the previous observation and \n\t\tthe relation \n  \\begin{equation}\\label{eq:growth}\n  \\|y_{t-1}-x^*\\|\\le \\tfrac{1}{\\mu}\\|\\nabla f(y_{t-1})\\|\n  \\end{equation}\n  due to the strong convexity of $f$, we conclude that $\\|\\nabla f(y_{t})\\|\\le \\|\\nabla f(y_{t-1})\\|/2$  and hence that $\\mu_t$ will no longer change whenever $\\mu_{t-1}\\le \\mu$. Therefore $\\mu_t\\ge \\mu/4$ for all $t$.\n\t\tThird, the algorithm will terminate with $\\|\\nabla f(y_t)\\|\\le \\varepsilon$ whenever $t\\ge P:=\\lceil\\log_4(\\mu_0/\\mu)\\rceil + \\lceil\\log_2(\\|\\nabla f(y_0)\\|/\\varepsilon)\\rceil$. Here $P$ is the sum of two parts: $\\lceil\\log_4(\\mu_0/\\mu)\\rceil$ counts the number of times when $\\|\\nabla f(y_t)\\|>\\|\\nabla f(y_{t-1})\\|/2$ and the value of $\\mu_t$ is reduced from $\\mu_{t-1}$; $\\lceil\\log_2(\\|\\nabla f(y_0)\\|/\\varepsilon)\\rceil$ counts the number of times when $\\|\\nabla f(y_t)\\|\\le\\|\\nabla f(y_{t-1})\\|/2$. Summarizing the above observations, the total number of gradient evaluations $N$ is bounded by\n\t\t\\begin{align*}\n\t\t\tN \\le\\ & 4P + C_1P\\sqrt{40L/\\mu} \\le (4+8\\sqrt{5}C_1)P\\sqrt{L/\\mu}\n\t\t\t\\\\\n\t\t\t\\le\\ & (4+8\\sqrt{5}C_1)\\lceil\\log_4(\\mu_0/\\mu)\\rceil\\sqrt{L/\\mu}  + (4+8\\sqrt{5}C_1)\\sqrt{L/\\mu}\\lceil\\log_2(\\|\\nabla f(y_0)\\|/\\varepsilon)\\rceil.\n\t\t\\end{align*}\n\t\n\t\\cmo{\n\t\tWe may also analyze the total number of gradient evaluations from an alternative perspective when we know that  $\\mu_0\\le 2^{j}\\varepsilon/\\operatorname{dist}(y_0,X^*)$. By   Theorem~\\ref{thm:conv_proposed_unconstrained_searchL}, the first AR call is guaranteed to yield $y_1$ such that $\\|\\nabla f(y_1)\\| \\le 2^{j-1}\\varepsilon$. If $j\\le 1$, then $y_1$ already satisfies $\\|\\nabla f(y_1)\\|\\le\\varepsilon$. If $j>1$,\n\t\tthen after $(j-1)$ occurrences of $\\|\\nabla f(y_t)\\| \\le \\|\\nabla f(y_{t-1})\\|/2$ we will have $\\|\\nabla f(y_t)\\|\\le \\varepsilon$. Therefore, the algorithm will terminate with $\\|\\nabla f(y_t)\\|\\le \\varepsilon$ whenever $t\\ge P:=\\lceil\\log_4(\\mu_0/\\mu)\\rceil + j$ and we can bound $N$ by $N \\le (4+8\\sqrt{5}C_1)P\\sqrt{L/\\mu}$ similar as the proof of the previous bound.\n\t}\t\n\t\n\t\\qedsymbol\n\t\\end{proof}","sha256_utf8":"23c995f59d01dc36e7d22b36c612191c442ce1d20577423cdcb5d794eade4749"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"SCAR收敛与梯度复杂度","source_units":[{"start":83828,"end":84550,"start_line":1113,"end_line":1119,"raw_tex":"\\begin{theorem}\n\t\t\\label{thm:conv_proposed_unconstrained_sc_noflag}\n\t\tSuppose that $\\|\\nabla f(y_0)\\|> \\varepsilon$. For any $\\varepsilon>0$, $\\mu_0\\ge \\mu$ and $M_0\\le 4L$, Algorithm~\\ref{alg:proposed_SCAR_noflag} terminates with $\\|\\nabla f(\\hat x)\\|\\le\\varepsilon$. The total number of gradient evaluations is bounded by  $(4+8\\sqrt{5}C_1)\\lceil\\log_4(\\mu_0/\\mu)\\rceil\\sqrt{L/\\mu}  + (4+8\\sqrt{5}C_1)\\sqrt{L/\\mu}\\lceil\\log_2(\\|\\nabla f(y_0)\\|/\\varepsilon)\\rceil$\n\t\t.\n\t\t\\cmo{\n\t\tMoreover, if we have $\\mu_0\\le 2^{j}\\varepsilon/\\|y_0-x^*\\|$ for some integer $j$, an alternative bound of gradient evaluations is $(4+8\\sqrt{5}C_1)\\lceil\\log_4(\\mu_0/\\mu)\\rceil\\sqrt{L/\\mu}  +(4+C_1\\sqrt{10L/\\mu})\\max\\{1,j\\}$.}\n\t\\end{theorem}","sha256_utf8":"e7f9702d4590c189d9d852ddb4282057bea2381f5f5a8d0c3c6c78841d5c8087"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"SCAR定理完整证明","source_units":[{"start":84552,"end":86963,"start_line":1120,"end_line":1143,"raw_tex":"\\begin{proof}\n\t\tWe start with three observations. First, by Theorem~\\ref{thm:conv_proposed_unconstrained_searchL}, \n\t\tthe output $y_{t}$ of the AR method satisfies $\\|\\nabla f(y_{t})\\|\\le \\mu_{t-1}\\|y_{t-1}-x^*\\|/2$ and the total number of gradient evaluations for computing $y_t$ is bounded by\n\t\t$4 +  C_1\\sqrt{10L/\\mu_{t-1}}$.\n\t\tSecond, using the previous observation and \n\t\tthe relation \n  \\begin{equation}\\label{eq:growth}\n  \\|y_{t-1}-x^*\\|\\le \\tfrac{1}{\\mu}\\|\\nabla f(y_{t-1})\\|\n  \\end{equation}\n  due to the strong convexity of $f$, we conclude that $\\|\\nabla f(y_{t})\\|\\le \\|\\nabla f(y_{t-1})\\|/2$  and hence that $\\mu_t$ will no longer change whenever $\\mu_{t-1}\\le \\mu$. Therefore $\\mu_t\\ge \\mu/4$ for all $t$.\n\t\tThird, the algorithm will terminate with $\\|\\nabla f(y_t)\\|\\le \\varepsilon$ whenever $t\\ge P:=\\lceil\\log_4(\\mu_0/\\mu)\\rceil + \\lceil\\log_2(\\|\\nabla f(y_0)\\|/\\varepsilon)\\rceil$. Here $P$ is the sum of two parts: $\\lceil\\log_4(\\mu_0/\\mu)\\rceil$ counts the number of times when $\\|\\nabla f(y_t)\\|>\\|\\nabla f(y_{t-1})\\|/2$ and the value of $\\mu_t$ is reduced from $\\mu_{t-1}$; $\\lceil\\log_2(\\|\\nabla f(y_0)\\|/\\varepsilon)\\rceil$ counts the number of times when $\\|\\nabla f(y_t)\\|\\le\\|\\nabla f(y_{t-1})\\|/2$. Summarizing the above observations, the total number of gradient evaluations $N$ is bounded by\n\t\t\\begin{align*}\n\t\t\tN \\le\\ & 4P + C_1P\\sqrt{40L/\\mu} \\le (4+8\\sqrt{5}C_1)P\\sqrt{L/\\mu}\n\t\t\t\\\\\n\t\t\t\\le\\ & (4+8\\sqrt{5}C_1)\\lceil\\log_4(\\mu_0/\\mu)\\rceil\\sqrt{L/\\mu}  + (4+8\\sqrt{5}C_1)\\sqrt{L/\\mu}\\lceil\\log_2(\\|\\nabla f(y_0)\\|/\\varepsilon)\\rceil.\n\t\t\\end{align*}\n\t\n\t\\cmo{\n\t\tWe may also analyze the total number of gradient evaluations from an alternative perspective when we know that  $\\mu_0\\le 2^{j}\\varepsilon/\\operatorname{dist}(y_0,X^*)$. By   Theorem~\\ref{thm:conv_proposed_unconstrained_searchL}, the first AR call is guaranteed to yield $y_1$ such that $\\|\\nabla f(y_1)\\| \\le 2^{j-1}\\varepsilon$. If $j\\le 1$, then $y_1$ already satisfies $\\|\\nabla f(y_1)\\|\\le\\varepsilon$. If $j>1$,\n\t\tthen after $(j-1)$ occurrences of $\\|\\nabla f(y_t)\\| \\le \\|\\nabla f(y_{t-1})\\|/2$ we will have $\\|\\nabla f(y_t)\\|\\le \\varepsilon$. Therefore, the algorithm will terminate with $\\|\\nabla f(y_t)\\|\\le \\varepsilon$ whenever $t\\ge P:=\\lceil\\log_4(\\mu_0/\\mu)\\rceil + j$ and we can bound $N$ by $N \\le (4+8\\sqrt{5}C_1)P\\sqrt{L/\\mu}$ similar as the proof of the previous bound.\n\t}\t\n\t\n\t\\qedsymbol\n\t\\end{proof}","sha256_utf8":"23c995f59d01dc36e7d22b36c612191c442ce1d20577423cdcb5d794eade4749"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"单次AR输出与代价","source_units":[{"start":84552,"end":84881,"start_line":1120,"end_line":1123,"raw_tex":"\\begin{proof}\n\t\tWe start with three observations. First, by Theorem~\\ref{thm:conv_proposed_unconstrained_searchL}, \n\t\tthe output $y_{t}$ of the AR method satisfies $\\|\\nabla f(y_{t})\\|\\le \\mu_{t-1}\\|y_{t-1}-x^*\\|/2$ and the total number of gradient evaluations for computing $y_t$ is bounded by\n\t\t$4 +  C_1\\sqrt{10L/\\mu_{t-1}}$.\n","sha256_utf8":"39388d95b876c5645c5b6874b753a4be479bf4ec4e0ca36a835ac1e2d901a172"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"强凸误差界控制猜测","source_units":[{"start":84881,"end":85272,"start_line":1124,"end_line":1129,"raw_tex":"\t\tSecond, using the previous observation and \n\t\tthe relation \n  \\begin{equation}\\label{eq:growth}\n  \\|y_{t-1}-x^*\\|\\le \\tfrac{1}{\\mu}\\|\\nabla f(y_{t-1})\\|\n  \\end{equation}\n  due to the strong convexity of $f$, we conclude that $\\|\\nabla f(y_{t})\\|\\le \\|\\nabla f(y_{t-1})\\|/2$  and hence that $\\mu_t$ will no longer change whenever $\\mu_{t-1}\\le \\mu$. Therefore $\\mu_t\\ge \\mu/4$ for all $t$.\n","sha256_utf8":"b2d87dd51daf4614959a30134f4ba445929efb2b0b1cd8de115d9d1eb90f05c7"}]},{"id":"S005","semantic_type":"proof_step","kind":"proof_step","title":"统计重启次数与总代价","source_units":[{"start":85272,"end":86132,"start_line":1130,"end_line":1135,"raw_tex":"\t\tThird, the algorithm will terminate with $\\|\\nabla f(y_t)\\|\\le \\varepsilon$ whenever $t\\ge P:=\\lceil\\log_4(\\mu_0/\\mu)\\rceil + \\lceil\\log_2(\\|\\nabla f(y_0)\\|/\\varepsilon)\\rceil$. Here $P$ is the sum of two parts: $\\lceil\\log_4(\\mu_0/\\mu)\\rceil$ counts the number of times when $\\|\\nabla f(y_t)\\|>\\|\\nabla f(y_{t-1})\\|/2$ and the value of $\\mu_t$ is reduced from $\\mu_{t-1}$; $\\lceil\\log_2(\\|\\nabla f(y_0)\\|/\\varepsilon)\\rceil$ counts the number of times when $\\|\\nabla f(y_t)\\|\\le\\|\\nabla f(y_{t-1})\\|/2$. Summarizing the above observations, the total number of gradient evaluations $N$ is bounded by\n\t\t\\begin{align*}\n\t\t\tN \\le\\ & 4P + C_1P\\sqrt{40L/\\mu} \\le (4+8\\sqrt{5}C_1)P\\sqrt{L/\\mu}\n\t\t\t\\\\\n\t\t\t\\le\\ & (4+8\\sqrt{5}C_1)\\lceil\\log_4(\\mu_0/\\mu)\\rceil\\sqrt{L/\\mu}  + (4+8\\sqrt{5}C_1)\\sqrt{L/\\mu}\\lceil\\log_2(\\|\\nabla f(y_0)\\|/\\varepsilon)\\rceil.\n\t\t\\end{align*}\n","sha256_utf8":"9af73c5099a849bdda3c23fe166776aa29c8199fee8d06498e33a7a12708045a"}]},{"id":"S006","semantic_type":"proof_step","kind":"proof_step","title":"推导替代复杂度界","source_units":[{"start":86134,"end":86963,"start_line":1137,"end_line":1143,"raw_tex":"\t\\cmo{\n\t\tWe may also analyze the total number of gradient evaluations from an alternative perspective when we know that  $\\mu_0\\le 2^{j}\\varepsilon/\\operatorname{dist}(y_0,X^*)$. By   Theorem~\\ref{thm:conv_proposed_unconstrained_searchL}, the first AR call is guaranteed to yield $y_1$ such that $\\|\\nabla f(y_1)\\| \\le 2^{j-1}\\varepsilon$. If $j\\le 1$, then $y_1$ already satisfies $\\|\\nabla f(y_1)\\|\\le\\varepsilon$. If $j>1$,\n\t\tthen after $(j-1)$ occurrences of $\\|\\nabla f(y_t)\\| \\le \\|\\nabla f(y_{t-1})\\|/2$ we will have $\\|\\nabla f(y_t)\\|\\le \\varepsilon$. Therefore, the algorithm will terminate with $\\|\\nabla f(y_t)\\|\\le \\varepsilon$ whenever $t\\ge P:=\\lceil\\log_4(\\mu_0/\\mu)\\rceil + j$ and we can bound $N$ by $N \\le (4+8\\sqrt{5}C_1)P\\sqrt{L/\\mu}$ similar as the proof of the previous bound.\n\t}\t\n\t\n\t\\qedsymbol\n\t\\end{proof}","sha256_utf8":"24095d942439782eb3947ba227f2d920ac614e8bf007879011256140a4dbe4c1"}]},{"id":"S007","semantic_type":"theorem","kind":"theorem","title":"自适应AR收敛界","source_units":[{"start":66112,"end":66930,"start_line":948,"end_line":953,"raw_tex":"\\begin{theorem}\n\t\\label{thm:conv_proposed_unconstrained_searchL} \n\t\tFor any input argument $M_0, \\sigma_1>0$, the AR method outputs an $\\hat x$ with $\\|\\nabla f(\\hat x)\\|\\le 5\\sigma_1\\operatorname{dist}(x_0, X^*)$ within $4 + 3\\sqrt{\\max\\{M_0/2, 2L\\}/\\sigma_1}+16\\sqrt{2c_\\cA }\\sqrt{L/\\sigma_1}$ gradient evaluations of $\\nabla f$.\n\t\tSpecially, if $M_0\\le 4L$, the total number of gradient evaluations is bounded by $4 + C_1\\sqrt{L/\\sigma_1}$, where $C_1:= \\sqrt{2}(3 + 16\\sqrt{2c_\\cA})$ is a universal constant. \n\t\tIf in addition $\\sigma_1 = \\varepsilon/(5D)$ for some $D>0$, then an approximate solution $\\hat x$ satisfying $\\|\\nabla f(\\hat x)\\|\\le \\varepsilon\\operatorname{dist}(x_0, X^*)/D$  can be computed within at most  $4 + \\sqrt{5}C_1\\sqrt{LD/\\varepsilon}$ gradient evaluations of $\\nabla f$. \n\t\\end{theorem}","sha256_utf8":"c42a3faf8ba4bcab2c45f4cbc68ac107fa53153ebdb68fc501d4181c4c9d7eee"}]},{"id":"S008","semantic_type":"proof","kind":"proof","title":"AR定理完整证明","source_units":[{"start":66932,"end":72434,"start_line":954,"end_line":1028,"raw_tex":"\\begin{proof}\n\t\tFirst note that by \\eqref{eq:dist_xu_xstarsig_unconstrained} in Proposition~\\ref{pro:xstar_dist_shrink_unconstrained} and the relation in \\eqref{eq:xs_norm_bound_unconstrained_est}, we have \n\t\t\\begin{align}\n\t\t\t\\label{eq:gradS_estimate_unconstrained_searchL}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\|\\nabla f(x_s)\\|\n\t\t\t\t&\t\\le (2\\sqrt{2}M_s + (1+2\\sqrt{2})\\sigma_s)\\|x_s -x_s^*\\| \\\\\n\t\t\t\t&\\quad + \\sigma_1\\|x^* - x_0\\| + \\sum_{i=2}^s(\\sigma_{i-1} + \\sigma_i)\\|x_{i-1}^* - x_{i-1}\\|,\n\t\t\t\\end{aligned}\n\t\t\\end{align}\n\t\twhich is analogous to \\eqref{eq:gradS_estimate_unconstrained}.\n\t\tConsider the call to the subroutine $\\cA$ for minimizing $f_s(x)$ in \\eqref{eq:subproblem_unconstrained_searchL}. Noting that $f_s$ is $\\sigma_s$-strongly convex, we have $\\|x - x_s^*\\|^2 \\le ({2}/{\\sigma_s})(f_s(x) - f_s(x_s^*)),\\ \\forall x\\in\\R^n$. In view of this observation and Assumption~\\ref{assum:A_unconstrained_searchL}, after the $k$-th gradient evaluation of $\\nabla f$, subroutine $\\cA$ will compute an approximation solution $x_s^k$ such that\n\t\t\\begin{align}\n\t\t\t\\label{eq:A_dist_shrink}\n\t\t\t\\|x_s^k - x_s^*\\|\\le\\dfrac{1}{k}\\sqrt{\\dfrac{2L_s^k}{\\sigma_s}}\\|x_{s-1} - x_{s}^*\\|\\le  \\dfrac{1}{k}\\sqrt{\\dfrac{2c_\\cA L}{\\sigma_s}}\\|x_{s-1} - x_{s}^*\\|.\n\t\t\\end{align}\n\t\tAlso by\tAssumption~\\ref{assum:A_unconstrained_searchL}, the number of gradient evaluations that $\\cA$ performs is bounded by \n\t\t\\begin{align}\n\t\t\t\\label{eq:Ns}\n\t\t\tN_s:=\\lceil8\\sqrt{2L_s/\\sigma_s}\\rceil\\le \\lceil8\\sqrt{2c_\\cA L/\\sigma_s}\\rceil.\n\t\t\\end{align} \n\t\tBy the above two relations and \\eqref{eq:dist_subproblem_unconstrained} in  \n\t\tProposition~\\ref{pro:xstar_dist_shrink_unconstrained}, the output solution $x_s$ computed at the termination of subroutine $\\cA$ applied to minimize $f_s$ satisfies\n\t\t\\begin{align*}\n\t\t\t\\|x_s - x_s^*\\| \\le \\frac{1}{8}\\|x_{s-1} - x_s^*\\| \\le \\frac{1}{8}\\|x_{s-1} - x_{s-1}^*\\|,\\ \\forall s = 1,\\ldots, S.\n\t\t\\end{align*}\n\t\tUsing the above result and the fact that $\\sigma_s = 4\\sigma_{s-1}$ for all $s>1$ in our previous estimate of $\\|\\nabla f(x_s)\\|$ in  \\eqref{eq:gradS_estimate_unconstrained_searchL}, we have\n\t\t\\begin{align}\n\t\t\t\\label{eq:tmp_grad_est}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\|\\nabla f(x_s)\\| \\le\\ & (2\\sqrt{2}M_s + (1+2\\sqrt{2})\\sigma_s)8^{-s}\\|x^* - x_0\\|  \n\t\t\t\t\\\\\n\t\t\t\t&\\ + \\sigma_1\\|x^* - x_0\\| + \\sum_{i=2}^{s}5\\cdot 4^{i-2}\\sigma_1 8^{-i+1}\\|x^* - x_0\\|\n\t\t\t\t\\\\\n\t\t\t\t\\le\\ & (\\sqrt{2}M_s + (1/2+\\sqrt{2})\\sigma_s) 4^{-s}\\|x_0 - x^*\\| + (9/4)\\sigma_1\\|x_0 - x^*\\|.\n\t\t\t\\end{aligned}\n\t\t\\end{align}\n\t\tHere $x^*$ is any solution in the optimal solution set $X^*$. \n\t\t\n\t\tWe will now estimate $\\sigma_s$ and $4^{-s}$ in the above relation when the termination criterion $\\sigma_s\\ge M_s$ is satisfied. \n\t\tIf the criterion is satisfied when $s=1$, by \\eqref{eq:tmp_grad_est} the output $\\hat x=x_1$ satisfies\n\t\t\\begin{align*}\n\t\t\t\\|\\nabla f(\\hat x)\\| \\le (1/2+2\\sqrt{2})\\sigma_1 4^{-1}\\|x_0 - x^*\\| + (9/4)\\sigma_1\\|x_0 - x^*\\|\\le 5\\sigma_1\\|x_0 - x^*\\|.\n\t\t\\end{align*}\n\t\tOtherwise, let $S>1$ be the smallest $s$ when the termination criterion $\\sigma_s\\ge M_s$ is satisfied. Observe in the backtracking function that $M_s\\le\\max\\{M_{s-1}/2, 2L\\}$. Thus we have $M_s\\le \\max\\{M_0/2, 2L\\}$ for $s=1,\\ldots, S$. Use this observation and recalling that $\\sigma_s = 4\\sigma_{s-1}$, we have \n\t\t\\begin{align}\n\t\t\t\\label{eq:sigmaS_bound}\n\t\t\tM_S\\le 4^{S-1}\\sigma_1 = \\sigma_{S} = 4\\sigma_{S-1}<4M_{S-1}\\le \\max\\{2M_0, 8L\\}.\n\t\t\\end{align}\n\t\tThus by \\eqref{eq:tmp_grad_est}, during termination the output $\\hat x = x_S$ satisfies\n\t\t\\begin{align*}\n\t\t\t\\|\\nabla f(\\hat x)\\| \\le & (1/2+2\\sqrt{2})\\sigma_S 4^{-S}\\|x_0 - x^*\\| + (9/4)\\sigma_1\\|x_0 - x^*\\|\n\t\t\t\\le 5\\sigma_1\\|x_0 - x^*\\|.\n\t\t\\end{align*}\n\t\tSince $x^*$ here is an arbitrary optimal solution, we have $\\|\\nabla f(\\hat x)\\|\\le 5\\sigma_1\\operatorname{dist}(x_0, X^*)$.\t\n\t\t\n\t\tNext we will bound the total number of gradient estimations. \n\t\tIf the termination criterion is satisfied when $s=1$,  bounding the numbers of gradient evaluations required for backtracking line-search and subroutine $\\cA$ respectively, we have the bound $1 + N_1$. Recalling the value of $N_s$ and its bound in \\eqref{eq:Ns}, this number is bounded by $ 1 + 8\\sqrt{2c_A L/\\sigma_1}$. If the termination criterion is satisfied at some $s=S>1$,  the total number of gradient evaluations is $S+\\sum_{s=1}^{S}N_s$. Here by \\eqref{eq:sigmaS_bound} we have $S\\le 2 + \\log_4(\\max\\{M_0/2, 2L\\}/\\sigma_1)$. Also, recalling the value of $N_s$ and its bound in \\eqref{eq:Ns} and noting that $\\sigma_s = 4\\sigma_{s-1}$ for all $s$ we have\n\t\t\\begin{align*}\n\t\t\t\\sum_{s=1}^{S}N_s \\le \\sum_{s=1}^{S}1 + 8\\sqrt{2c_\\cA L/\\sigma_s} \\le & S + 8\\sqrt{2c_\\cA L/\\sigma_1}\\sum_{s=1}^{\\infty}2^{-(s-1)}  \\le S + 16\\sqrt{2c_\\cA L/\\sigma_1}.\n\t\t\\end{align*}      \n\t\tSummarizing the above we can bound the total number of gradient evaluations by\n\t\t\\begin{align}\n\t\t\t\\label{eq:total_gradient}\n\t\t\t\\begin{aligned}\n\t\t\t\t& S+\\sum_{s=1}^{S}N_s \n\t\t\t\t\\le 2S + 16\\sqrt{2c_\\cA L/\\sigma_1} \\le 2(2 + \\log_4(\\max\\{M_0/2, 2L\\}/\\sigma_1)) + 16\\sqrt{2c_\\cA L/\\sigma_1}\n\t\t\t\t\\\\ \n\t\t\t\t\\le\\ & 4 + 3\\sqrt{\\max\\{M_0/2, 2L\\}/\\sigma_1}+16\\sqrt{2c_\\cA }\\sqrt{L/\\sigma_1}. \n\t\t\t\\end{aligned}\n\t\t\\end{align}\n\t\tHere in the last inequality we use the fact that $2\\log_4 t \\cmo{<} 3\\sqrt{t}$ for all $t>0$.\n\t\tNote that the above bound is also an upper bound of number of gradient evaluations if the termination criterion is satisfied when $s=1$. \n\t\tThe remainder parts of result then immediately follows from \\eqref{eq:total_gradient}.\n\t\\qedsymbol\\end{proof}","sha256_utf8":"16c13686df43bceb701b689ea62343d9007b7f709036477b13f1e51c5e25d4b5"}]},{"id":"S009","semantic_type":"proof_step","kind":"proof_step","title":"建立自适应梯度距离界","source_units":[{"start":66932,"end":67508,"start_line":954,"end_line":964,"raw_tex":"\\begin{proof}\n\t\tFirst note that by \\eqref{eq:dist_xu_xstarsig_unconstrained} in Proposition~\\ref{pro:xstar_dist_shrink_unconstrained} and the relation in \\eqref{eq:xs_norm_bound_unconstrained_est}, we have \n\t\t\\begin{align}\n\t\t\t\\label{eq:gradS_estimate_unconstrained_searchL}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\|\\nabla f(x_s)\\|\n\t\t\t\t&\t\\le (2\\sqrt{2}M_s + (1+2\\sqrt{2})\\sigma_s)\\|x_s -x_s^*\\| \\\\\n\t\t\t\t&\\quad + \\sigma_1\\|x^* - x_0\\| + \\sum_{i=2}^s(\\sigma_{i-1} + \\sigma_i)\\|x_{i-1}^* - x_{i-1}\\|,\n\t\t\t\\end{aligned}\n\t\t\\end{align}\n\t\twhich is analogous to \\eqref{eq:gradS_estimate_unconstrained}.\n","sha256_utf8":"57a031265b6ece2fc873cf65636d0170b7296db8845f0eb3570a7e119ada31d9"}]},{"id":"S010","semantic_type":"proof_step","kind":"proof_step","title":"子程序函数误差转距离","source_units":[{"start":67508,"end":68185,"start_line":965,"end_line":969,"raw_tex":"\t\tConsider the call to the subroutine $\\cA$ for minimizing $f_s(x)$ in \\eqref{eq:subproblem_unconstrained_searchL}. Noting that $f_s$ is $\\sigma_s$-strongly convex, we have $\\|x - x_s^*\\|^2 \\le ({2}/{\\sigma_s})(f_s(x) - f_s(x_s^*)),\\ \\forall x\\in\\R^n$. In view of this observation and Assumption~\\ref{assum:A_unconstrained_searchL}, after the $k$-th gradient evaluation of $\\nabla f$, subroutine $\\cA$ will compute an approximation solution $x_s^k$ such that\n\t\t\\begin{align}\n\t\t\t\\label{eq:A_dist_shrink}\n\t\t\t\\|x_s^k - x_s^*\\|\\le\\dfrac{1}{k}\\sqrt{\\dfrac{2L_s^k}{\\sigma_s}}\\|x_{s-1} - x_{s}^*\\|\\le  \\dfrac{1}{k}\\sqrt{\\dfrac{2c_\\cA L}{\\sigma_s}}\\|x_{s-1} - x_{s}^*\\|.\n\t\t\\end{align}\n","sha256_utf8":"5b7ae01b1c44e0992b4e1e722e0954bd6a59beccfb9482d0199d1589d6cf66d3"}]},{"id":"S011","semantic_type":"proof_step","kind":"proof_step","title":"界定子程序梯度次数","source_units":[{"start":68185,"end":68445,"start_line":970,"end_line":974,"raw_tex":"\t\tAlso by\tAssumption~\\ref{assum:A_unconstrained_searchL}, the number of gradient evaluations that $\\cA$ performs is bounded by \n\t\t\\begin{align}\n\t\t\t\\label{eq:Ns}\n\t\t\tN_s:=\\lceil8\\sqrt{2L_s/\\sigma_s}\\rceil\\le \\lceil8\\sqrt{2c_\\cA L/\\sigma_s}\\rceil.\n\t\t\\end{align} \n","sha256_utf8":"3ccfac96bd3e990260e22e2a822141693b2bf3dcff1b1763759ea17c26421dd7"}]},{"id":"S012","semantic_type":"proof_step","kind":"proof_step","title":"得到八分之一距离收缩","source_units":[{"start":68445,"end":68843,"start_line":975,"end_line":979,"raw_tex":"\t\tBy the above two relations and \\eqref{eq:dist_subproblem_unconstrained} in  \n\t\tProposition~\\ref{pro:xstar_dist_shrink_unconstrained}, the output solution $x_s$ computed at the termination of subroutine $\\cA$ applied to minimize $f_s$ satisfies\n\t\t\\begin{align*}\n\t\t\t\\|x_s - x_s^*\\| \\le \\frac{1}{8}\\|x_{s-1} - x_s^*\\| \\le \\frac{1}{8}\\|x_{s-1} - x_{s-1}^*\\|,\\ \\forall s = 1,\\ldots, S.\n\t\t\\end{align*}\n","sha256_utf8":"57a5cf36ad8495ebbd8834a9964a7f607a48f9c64ba921f42d0d1e77cc612dd4"}]},{"id":"S013","semantic_type":"proof_step","kind":"proof_step","title":"累加收缩后的梯度误差","source_units":[{"start":68843,"end":69489,"start_line":980,"end_line":991,"raw_tex":"\t\tUsing the above result and the fact that $\\sigma_s = 4\\sigma_{s-1}$ for all $s>1$ in our previous estimate of $\\|\\nabla f(x_s)\\|$ in  \\eqref{eq:gradS_estimate_unconstrained_searchL}, we have\n\t\t\\begin{align}\n\t\t\t\\label{eq:tmp_grad_est}\n\t\t\t\\begin{aligned}\n\t\t\t\t\\|\\nabla f(x_s)\\| \\le\\ & (2\\sqrt{2}M_s + (1+2\\sqrt{2})\\sigma_s)8^{-s}\\|x^* - x_0\\|  \n\t\t\t\t\\\\\n\t\t\t\t&\\ + \\sigma_1\\|x^* - x_0\\| + \\sum_{i=2}^{s}5\\cdot 4^{i-2}\\sigma_1 8^{-i+1}\\|x^* - x_0\\|\n\t\t\t\t\\\\\n\t\t\t\t\\le\\ & (\\sqrt{2}M_s + (1/2+\\sqrt{2})\\sigma_s) 4^{-s}\\|x_0 - x^*\\| + (9/4)\\sigma_1\\|x_0 - x^*\\|.\n\t\t\t\\end{aligned}\n\t\t\\end{align}\n\t\tHere $x^*$ is any solution in the optimal solution set $X^*$. \n","sha256_utf8":"9102944b483496564b1e6c14dbc7f6f7df46e78761548438558eacdca4729c6f"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"处理首次迭代终止","source_units":[{"start":69492,"end":69890,"start_line":993,"end_line":997,"raw_tex":"\t\tWe will now estimate $\\sigma_s$ and $4^{-s}$ in the above relation when the termination criterion $\\sigma_s\\ge M_s$ is satisfied. \n\t\tIf the criterion is satisfied when $s=1$, by \\eqref{eq:tmp_grad_est} the output $\\hat x=x_1$ satisfies\n\t\t\\begin{align*}\n\t\t\t\\|\\nabla f(\\hat x)\\| \\le (1/2+2\\sqrt{2})\\sigma_1 4^{-1}\\|x_0 - x^*\\| + (9/4)\\sigma_1\\|x_0 - x^*\\|\\le 5\\sigma_1\\|x_0 - x^*\\|.\n\t\t\\end{align*}\n","sha256_utf8":"3c60f1098ce107e5025ca1649cbd0fbb77cd26e5a1bec5f36097abd1607579db"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"控制终止尺度与输出梯度","source_units":[{"start":69890,"end":70733,"start_line":998,"end_line":1008,"raw_tex":"\t\tOtherwise, let $S>1$ be the smallest $s$ when the termination criterion $\\sigma_s\\ge M_s$ is satisfied. Observe in the backtracking function that $M_s\\le\\max\\{M_{s-1}/2, 2L\\}$. Thus we have $M_s\\le \\max\\{M_0/2, 2L\\}$ for $s=1,\\ldots, S$. Use this observation and recalling that $\\sigma_s = 4\\sigma_{s-1}$, we have \n\t\t\\begin{align}\n\t\t\t\\label{eq:sigmaS_bound}\n\t\t\tM_S\\le 4^{S-1}\\sigma_1 = \\sigma_{S} = 4\\sigma_{S-1}<4M_{S-1}\\le \\max\\{2M_0, 8L\\}.\n\t\t\\end{align}\n\t\tThus by \\eqref{eq:tmp_grad_est}, during termination the output $\\hat x = x_S$ satisfies\n\t\t\\begin{align*}\n\t\t\t\\|\\nabla f(\\hat x)\\| \\le & (1/2+2\\sqrt{2})\\sigma_S 4^{-S}\\|x_0 - x^*\\| + (9/4)\\sigma_1\\|x_0 - x^*\\|\n\t\t\t\\le 5\\sigma_1\\|x_0 - x^*\\|.\n\t\t\\end{align*}\n\t\tSince $x^*$ here is an arbitrary optimal solution, we have $\\|\\nabla f(\\hat x)\\|\\le 5\\sigma_1\\operatorname{dist}(x_0, X^*)$.\t\n","sha256_utf8":"71ba60a046ad4bf461bf2caa033632a9bf83a870e4383c9efb321f00b5cf4a81"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"累加回溯与子程序代价","source_units":[{"start":70736,"end":71675,"start_line":1010,"end_line":1014,"raw_tex":"\t\tNext we will bound the total number of gradient estimations. \n\t\tIf the termination criterion is satisfied when $s=1$,  bounding the numbers of gradient evaluations required for backtracking line-search and subroutine $\\cA$ respectively, we have the bound $1 + N_1$. Recalling the value of $N_s$ and its bound in \\eqref{eq:Ns}, this number is bounded by $ 1 + 8\\sqrt{2c_A L/\\sigma_1}$. If the termination criterion is satisfied at some $s=S>1$,  the total number of gradient evaluations is $S+\\sum_{s=1}^{S}N_s$. Here by \\eqref{eq:sigmaS_bound} we have $S\\le 2 + \\log_4(\\max\\{M_0/2, 2L\\}/\\sigma_1)$. Also, recalling the value of $N_s$ and its bound in \\eqref{eq:Ns} and noting that $\\sigma_s = 4\\sigma_{s-1}$ for all $s$ we have\n\t\t\\begin{align*}\n\t\t\t\\sum_{s=1}^{S}N_s \\le \\sum_{s=1}^{S}1 + 8\\sqrt{2c_\\cA L/\\sigma_s} \\le & S + 8\\sqrt{2c_\\cA L/\\sigma_1}\\sum_{s=1}^{\\infty}2^{-(s-1)}  \\le S + 16\\sqrt{2c_\\cA L/\\sigma_1}.\n\t\t\\end{align*}      \n","sha256_utf8":"a3e824635967c199440c6d1155e137aa5c928fcccc6b2a3396127f8005c083f1"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"将迭代对数转为复杂度界","source_units":[{"start":71675,"end":72183,"start_line":1015,"end_line":1025,"raw_tex":"\t\tSummarizing the above we can bound the total number of gradient evaluations by\n\t\t\\begin{align}\n\t\t\t\\label{eq:total_gradient}\n\t\t\t\\begin{aligned}\n\t\t\t\t& S+\\sum_{s=1}^{S}N_s \n\t\t\t\t\\le 2S + 16\\sqrt{2c_\\cA L/\\sigma_1} \\le 2(2 + \\log_4(\\max\\{M_0/2, 2L\\}/\\sigma_1)) + 16\\sqrt{2c_\\cA L/\\sigma_1}\n\t\t\t\t\\\\ \n\t\t\t\t\\le\\ & 4 + 3\\sqrt{\\max\\{M_0/2, 2L\\}/\\sigma_1}+16\\sqrt{2c_\\cA }\\sqrt{L/\\sigma_1}. \n\t\t\t\\end{aligned}\n\t\t\\end{align}\n\t\tHere in the last inequality we use the fact that $2\\log_4 t \\cmo{<} 3\\sqrt{t}$ for all $t>0$.\n","sha256_utf8":"378cd53ee5b36264290d54d157f1b3b0658915ead0dda74940b9993b8e0c6566"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"统一终止情形并推出特例","source_units":[{"start":72183,"end":72434,"start_line":1026,"end_line":1028,"raw_tex":"\t\tNote that the above bound is also an upper bound of number of gradient evaluations if the termination criterion is satisfied when $s=1$. \n\t\tThe remainder parts of result then immediately follows from \\eqref{eq:total_gradient}.\n\t\\qedsymbol\\end{proof}","sha256_utf8":"2ab1e7d8ad1f97f167c4ad26302652044eaf7f8453ab09185c36282d0c101038"}]},{"id":"S019","semantic_type":"proposition","kind":"proposition","title":"累积正则化距离关系","source_units":[{"start":28211,"end":28645,"start_line":362,"end_line":374,"raw_tex":"\\begin{proposition}\n\t\\label{pro:xstar_dist_shrink_unconstrained}\n\tIn  Algorithm~\\ref{alg:proposed_unconstrained} we have for all $s\\ge 1$ \n\t\\begin{align}\n\t\t\\label{eq:dist_subproblem_unconstrained}\n\t\t& \\|x_{s-1} - x_s^*\\|\\le \\|x_{s-1} - x_{s-1}^*\\|, \n\t\t\\\\\n\t\t\\label{eq:dist_xu_xstarsig_unconstrained}\n\t\t&\\sigma_s\\|\\xu_s - x_s^*\\| \n\t\t\\le \\sum_{i=1}^{s}(\\sigma_{i-1} + \\sigma_{i})\\|x_{i-1}^* - x_{i-1}\\|\n\t\t.\n\t\\end{align}\n\\end{proposition}","sha256_utf8":"5cdd65ecaffcd0143febadb7d3e3c11f63201010690779e98cd9b7d041898d80"}]},{"id":"S020","semantic_type":"proof","kind":"proof","title":"距离关系完整证明","source_units":[{"start":28646,"end":31253,"start_line":375,"end_line":413,"raw_tex":"\\begin{proof}\n\tWith the selection of $\\gamma_s = 1 - \\sigma_{s-1}/\\sigma_s$ in the definition of prox-center $\\xu_s$ in   \\eqref{eq:xus_unconstrained}, the subproblem \\eqref{eq:subproblem_unconstrained} is equivalent to\n\t\\begin{align*}\n\t\tx_s^* = \\argmin_{x\\in \\R^n} f(x) + \\dfrac{\\sigma_{s-1}}{2}\\|x - \\xu_{s-1}\\|^2 + \\dfrac{\\sigma_s - \\sigma_{s-1}}{2}\\|x - x_{s-1}\\|^2,\\ \\forall s \\ge 1.\n\t\\end{align*}\n\tNoting the optimality of $x_{s-1}^*$ in \\eqref{eq:subproblem_unconstrained}  and $x_s^*$ in the above subproblem we have\n\t\\begin{align*}\n\t\t& f(x_{s-1}^*) + \\dfrac{\\sigma_{s-1}}{2}\\|x_{s-1}^* - \\xu_{s-1}\\|^2 + \\dfrac{\\sigma_s - \\sigma_{s-1}}{2}\\|x_s^* - x_{s-1}\\|^2\n\t\t\\\\\n\t\t\\le\\ & f(x_s^*) + \\dfrac{\\sigma_{s-1}}{2}\\|x_s^* - \\xu_{s-1}\\|^2 + \\dfrac{\\sigma_s - \\sigma_{s-1}}{2}\\|x_s^* - x_{s-1}\\|^2 \n\t\t\\\\\n\t\t\\le\\ & f(x_{s-1}^*) + \\dfrac{\\sigma_{s-1}}{2}\\|x_{s-1}^* - \\xu_{s-1}\\|^2 + \\dfrac{\\sigma_s - \\sigma_{s-1}}{2}\\|x_{s-1}^* - x_{s-1}\\|^2.\n\t\\end{align*}\n\tThe conclusion in \\eqref{eq:dist_subproblem_unconstrained} then follows from the above relation and the assumption $\\sigma_s>\\sigma_{s-1}$ required by Algorithm~\\ref{alg:proposed_unconstrained}.\n\t\n\tWe move to prove the next result \\eqref{eq:dist_xu_xstarsig_unconstrained}. Denoting $\\alpha_s:=\\sigma_s - \\sigma_{s-1}$ for all $s$ and noting that $\\gamma_s = \\alpha_s/\\sigma_s$ in \\eqref{eq:xus_unconstrained}, we can rewrite the definition of $\\xu_s$ in \\eqref{eq:xus_unconstrained} to\n\t$\n\t\\sigma_s \\xu_s = \\sigma_{s-1}\\xu_{s-1} + \\alpha_s x_{s-1}.\n\t$\n\tRecalling that $\\sigma_0=0$, the above recursive relation yields\n\t$\n\t\\sigma_s\\xu_s = \\sum_{i=1}^{s}\\alpha_ix_{i-1}.\n\t$\n\tIn other words, $\\xu_s$ is a convex combination of $x_0,\\ldots, x_{s-1}$ with weights $\\alpha_i/\\sigma_s$  since $\\sigma_s = \\sum_{i=1}^{s}\\alpha_i$. Thus \n\t\\begin{align*}\n\t\t\\sigma_s(\\xu_s - x_s^*) =\\ & \\sum_{i=1}^{s}\\alpha_i(x_{i-1} - x_s^*) \n\t\t\\\\\n\t\t=\\ & \\alpha_s(x_{s-1} - x_s^*) + \\sum_{i=1}^{s-1}\\alpha_i(x_{i-1} - x_{s-1}^*)  + \\left(\\sum_{i=1}^{s-1}\\alpha_i\\right)(x_{s-1}^* - x_s^*)\n\t\t\\\\\n\t\t=\\ & \\alpha_s(x_{s-1} - x_s^*) + \\sigma_{s-1}(\\xu_{s-1} - x_{s-1}^*) + \\sigma_{s-1}(x_{s-1}^* - x_{s-1}) + \\sigma_{s-1}(x_{s-1} - x_s^*)\n\t\t\\\\\n\t\t=\\ & \\sigma_{s-1}(\\xu_{s-1} - x_{s-1}^*) + \\sigma_{s-1}(x_{s-1}^* - x_{s-1}) + \\sigma_s(x_{s-1} - x_s^*).\n\t\\end{align*}\n\tThe above recursive relation yields\n\t$\n\t\\sigma_s(\\xu_s - x_s^*) = \\sum_{i=1}^{s}[\\sigma_{i-1}(x_{i-1}^* - x_{i-1}) + \\sigma_i(x_{i-1} - x_i^*)].\n\t$\n\tBy the above equality and our previous result \\eqref{eq:dist_subproblem_unconstrained}, we conclude \\eqref{eq:dist_xu_xstarsig_unconstrained}.\n\\qedsymbol\\end{proof}","sha256_utf8":"69683e541c1088e3028a966b08a6d86bef2c7687a25d2987fe549d717fc51854"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"改写累积正则化子问题","source_units":[{"start":28646,"end":29049,"start_line":375,"end_line":379,"raw_tex":"\\begin{proof}\n\tWith the selection of $\\gamma_s = 1 - \\sigma_{s-1}/\\sigma_s$ in the definition of prox-center $\\xu_s$ in   \\eqref{eq:xus_unconstrained}, the subproblem \\eqref{eq:subproblem_unconstrained} is equivalent to\n\t\\begin{align*}\n\t\tx_s^* = \\argmin_{x\\in \\R^n} f(x) + \\dfrac{\\sigma_{s-1}}{2}\\|x - \\xu_{s-1}\\|^2 + \\dfrac{\\sigma_s - \\sigma_{s-1}}{2}\\|x - x_{s-1}\\|^2,\\ \\forall s \\ge 1.\n\t\\end{align*}\n","sha256_utf8":"a309a57e7522405ad0dde9dc5a6c084242f9b7e806b188fb7096cf12cbf7cdfe"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"比较相邻最优值得到距离界","source_units":[{"start":29049,"end":29799,"start_line":380,"end_line":388,"raw_tex":"\tNoting the optimality of $x_{s-1}^*$ in \\eqref{eq:subproblem_unconstrained}  and $x_s^*$ in the above subproblem we have\n\t\\begin{align*}\n\t\t& f(x_{s-1}^*) + \\dfrac{\\sigma_{s-1}}{2}\\|x_{s-1}^* - \\xu_{s-1}\\|^2 + \\dfrac{\\sigma_s - \\sigma_{s-1}}{2}\\|x_s^* - x_{s-1}\\|^2\n\t\t\\\\\n\t\t\\le\\ & f(x_s^*) + \\dfrac{\\sigma_{s-1}}{2}\\|x_s^* - \\xu_{s-1}\\|^2 + \\dfrac{\\sigma_s - \\sigma_{s-1}}{2}\\|x_s^* - x_{s-1}\\|^2 \n\t\t\\\\\n\t\t\\le\\ & f(x_{s-1}^*) + \\dfrac{\\sigma_{s-1}}{2}\\|x_{s-1}^* - \\xu_{s-1}\\|^2 + \\dfrac{\\sigma_s - \\sigma_{s-1}}{2}\\|x_{s-1}^* - x_{s-1}\\|^2.\n\t\\end{align*}\n\tThe conclusion in \\eqref{eq:dist_subproblem_unconstrained} then follows from the above relation and the assumption $\\sigma_s>\\sigma_{s-1}$ required by Algorithm~\\ref{alg:proposed_unconstrained}.\n","sha256_utf8":"8a05c0be0e2b4eacdb81961d97798646a7eed2576ee2bc0b8134f04294be1c48"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"展开近端中心的累积权重","source_units":[{"start":29801,"end":30434,"start_line":390,"end_line":398,"raw_tex":"\tWe move to prove the next result \\eqref{eq:dist_xu_xstarsig_unconstrained}. Denoting $\\alpha_s:=\\sigma_s - \\sigma_{s-1}$ for all $s$ and noting that $\\gamma_s = \\alpha_s/\\sigma_s$ in \\eqref{eq:xus_unconstrained}, we can rewrite the definition of $\\xu_s$ in \\eqref{eq:xus_unconstrained} to\n\t$\n\t\\sigma_s \\xu_s = \\sigma_{s-1}\\xu_{s-1} + \\alpha_s x_{s-1}.\n\t$\n\tRecalling that $\\sigma_0=0$, the above recursive relation yields\n\t$\n\t\\sigma_s\\xu_s = \\sum_{i=1}^{s}\\alpha_ix_{i-1}.\n\t$\n\tIn other words, $\\xu_s$ is a convex combination of $x_0,\\ldots, x_{s-1}$ with weights $\\alpha_i/\\sigma_s$  since $\\sigma_s = \\sum_{i=1}^{s}\\alpha_i$. Thus \n","sha256_utf8":"02f0de6a27c04e85b3ef53a17435213bc6e8caed5eb77e39c23c0697b975cac6"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"建立中心偏差递推恒等式","source_units":[{"start":30434,"end":30939,"start_line":399,"end_line":407,"raw_tex":"\t\\begin{align*}\n\t\t\\sigma_s(\\xu_s - x_s^*) =\\ & \\sum_{i=1}^{s}\\alpha_i(x_{i-1} - x_s^*) \n\t\t\\\\\n\t\t=\\ & \\alpha_s(x_{s-1} - x_s^*) + \\sum_{i=1}^{s-1}\\alpha_i(x_{i-1} - x_{s-1}^*)  + \\left(\\sum_{i=1}^{s-1}\\alpha_i\\right)(x_{s-1}^* - x_s^*)\n\t\t\\\\\n\t\t=\\ & \\alpha_s(x_{s-1} - x_s^*) + \\sigma_{s-1}(\\xu_{s-1} - x_{s-1}^*) + \\sigma_{s-1}(x_{s-1}^* - x_{s-1}) + \\sigma_{s-1}(x_{s-1} - x_s^*)\n\t\t\\\\\n\t\t=\\ & \\sigma_{s-1}(\\xu_{s-1} - x_{s-1}^*) + \\sigma_{s-1}(x_{s-1}^* - x_{s-1}) + \\sigma_s(x_{s-1} - x_s^*).\n\t\\end{align*}\n","sha256_utf8":"96f50a04b96fa21d17a59de74bdd5eb85f2f47489b125da3542108e31e75565b"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"展开递推并界定累积偏差","source_units":[{"start":30939,"end":31253,"start_line":408,"end_line":413,"raw_tex":"\tThe above recursive relation yields\n\t$\n\t\\sigma_s(\\xu_s - x_s^*) = \\sum_{i=1}^{s}[\\sigma_{i-1}(x_{i-1}^* - x_{i-1}) + \\sigma_i(x_{i-1} - x_i^*)].\n\t$\n\tBy the above equality and our previous result \\eqref{eq:dist_subproblem_unconstrained}, we conclude \\eqref{eq:dist_xu_xstarsig_unconstrained}.\n\\qedsymbol\\end{proof}","sha256_utf8":"211b121c8e8e27e8715ed3a6f34eadd17b56d18f1cfc2074ea585c8633becb31"}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","title":"局部光滑性梯度距离界","source_units":[{"start":59115,"end":59560,"start_line":830,"end_line":840,"raw_tex":"\\begin{lemma}\n\t\\label{thm:Mlemma}\n\tSuppose that $g:\\R^n\\to\\R$ is a convex smooth function.\n\tFor any $x\\in\\R^n$, $M>0$, $\\eta\\ge 2M$, and $\\xpp:=x - (1/\\eta)\\nabla g(x)$,\n\tif $\\xpp$ and $x$ satisfy\n\t\\begin{align}\n\t\t\\label{eq:M}\n\t\tg(\\xpp) - g(x) - \\langle \\nabla g(x), \\xpp - x\\rangle \\le \\frac{M}{2}\\|\\xpp - x\\|^2,\n\t\\end{align}\n\tthen we have $\\|\\nabla g(x)\\|\\le \\sqrt{2}\\eta \\|x - x^*\\|$, where $x^*$ is any minimizer of function $g$.\n\\end{lemma}","sha256_utf8":"fa42cd6899a828c4204c08d0d5f74d2e578c7745a3da5df536e8157a141a932a"}]},{"id":"S027","semantic_type":"proof","kind":"proof","title":"局部梯度界完整证明","source_units":[{"start":59561,"end":60201,"start_line":841,"end_line":848,"raw_tex":"\\begin{proof}\n\tNote that our definition of $\\xpp$ is exactly $\\xpp(\\eta,0,\\xu;x)$ described in \\eqref{eq:xpp} (with $X\\in\\R^n$, $\\sigma=0$, and $\\phi\\equiv 0$). Applying result \\eqref{eq:3point} in Lemma~\\ref{lem:grad_map_properties} and noting that $\\eta\\ge 2M$ we have\n\t\\begin{align*}\n\t\tg(\\xpp) - g(u) \\le \\frac{\\eta}{2}\\|u - x\\|^2 - \\frac{\\eta}{2}\\|u - \\xpp\\|^2 - \\frac{\\eta}{4}\\|\\xpp - x\\|^2, \\ \\forall u\\in \\R^n.\n\t\\end{align*}\n\tLetting $u=x^*$ in the above relation, using the optimality of $x^*$ and the fact that $\\xpp - x = -(1/\\eta)\\nabla g(x)$ we conclude that \n\t$\t\\|\\nabla g(x)\\|^2 \\le 2\\eta^2\\|x^* -x\\|^2$.\n\\qedsymbol\\end{proof}","sha256_utf8":"ec211c1012c4728acb1710fb54d71cd8420b7d31ae56fbca4177da7808604aec"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"特化三点不等式","source_units":[{"start":59561,"end":59993,"start_line":841,"end_line":845,"raw_tex":"\\begin{proof}\n\tNote that our definition of $\\xpp$ is exactly $\\xpp(\\eta,0,\\xu;x)$ described in \\eqref{eq:xpp} (with $X\\in\\R^n$, $\\sigma=0$, and $\\phi\\equiv 0$). Applying result \\eqref{eq:3point} in Lemma~\\ref{lem:grad_map_properties} and noting that $\\eta\\ge 2M$ we have\n\t\\begin{align*}\n\t\tg(\\xpp) - g(u) \\le \\frac{\\eta}{2}\\|u - x\\|^2 - \\frac{\\eta}{2}\\|u - \\xpp\\|^2 - \\frac{\\eta}{4}\\|\\xpp - x\\|^2, \\ \\forall u\\in \\R^n.\n\t\\end{align*}\n","sha256_utf8":"e7adca047ba55514a11df458261637733d607b84e2e115f840a75ac27dcbbb37"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"代入最优点控制梯度","source_units":[{"start":59993,"end":60201,"start_line":846,"end_line":848,"raw_tex":"\tLetting $u=x^*$ in the above relation, using the optimality of $x^*$ and the fact that $\\xpp - x = -(1/\\eta)\\nabla g(x)$ we conclude that \n\t$\t\\|\\nabla g(x)\\|^2 \\le 2\\eta^2\\|x^* -x\\|^2$.\n\\qedsymbol\\end{proof}","sha256_utf8":"883e59d2459e539181cb454ab0b9a173328c1f9ad2e203a5cef3ef0548855d7c"}]},{"id":"S030","semantic_type":"lemma","kind":"lemma","title":"梯度映射基本性质","source_units":[{"start":40532,"end":41539,"start_line":520,"end_line":544,"raw_tex":"\\begin{lem}\n\t\\label{lem:grad_map_properties}\n\tFor any $x, \\xu\\in\\R^n$ and $\\eta>0$, $\\sigma\\ge0$, we have the following properties for the (perturbed) gradient mappings:\n\t\\begin{align}\n\t\t\\label{eq:xp_lip}\n\t\t& \\|x^+(\\eta+\\sigma; x) - x^{++}(\\eta, \\sigma, \\xu; x)\\| \\le \\frac{\\sigma}{\\eta+\\sigma}\\|x - \\xu\\|;\n\t\t\\\\\n\t\t\\label{eq:xp_monotonicity}\n\t\t& \\|G_\\eta(x)\\| \\le \\|G_{\\sigma + \\eta}(x)\\|.\n\t\\end{align}\n\tMoreover, if $x^{++}:=x^{++}(\\eta, \\sigma, \\xu; x)$ satisfies \n\t\\begin{align}\n\t\t\\label{eq:local_lip}\n\t\tf(x^{++}) - f(x) - \\langle \\nabla f(x), x^{++} - x\\rangle \\le \\frac{M}{2}\\|x^{++} - x\\|^2\n\t\\end{align}\n\tfor some $M>0$,  we have\n\t\\begin{align}\n\t\t\\label{eq:3point}\n\t\t\\begin{aligned}\n\t\t\t&  \\left[f(x^{++}) + \\phi(x^{++}) + \\frac{\\sigma}{2}\\|x^{++} - \\xu\\|^2\\right] - \\left[f(u) + \\phi(u) + \\frac{\\sigma}{2}\\|u - \\xu\\|^2\\right]\n\t\t\t\\\\\n\t\t\t\\le\\ & \\frac{\\eta}{2}\\|u - x\\|^2 - \\frac{\\sigma + \\eta}{2}\\|u - x^{++}\\|^2 - \\frac{\\eta-M}{2}\\|x^{++} - x\\|^2 ,\\ \\forall u\\in X.\n\t\t\\end{aligned}\n\t\\end{align}\n\\end{lem}","sha256_utf8":"cecb9526383e2e776a3af7ffd9ea131b74e904869ca13e1fb20b60526c2f7ae6"}]},{"id":"S031","semantic_type":"proof","kind":"proof","title":"梯度映射性质完整证明","source_units":[{"start":41542,"end":44113,"start_line":547,"end_line":599,"raw_tex":"\\begin{proof}\n\tLet us denote $x^+:=x^+(\\eta+\\sigma; x)$, $x^{++}:=x^{++}(\\eta, \\sigma, \\xu; x)$ and $x' := x^+(\\eta; x)$. By the optimality conditions of \\eqref{eq:xp} and \\eqref{eq:xpp} we have \n\t\\begin{align}\n\t\t\\langle \\nabla f(x) + (\\eta+\\sigma)(x^+ - x), u - x^{+}\\rangle + \\phi(u) - \\phi(x^{+}) \\ge 0, \\label{eq:oc_simple_1} \\\\\n\t\t\\langle \\nabla f(x) + \\sigma (x^{++}-\\bar x) +\\eta (x^{++} - x), u - x^{++}\\rangle + \\phi(u) - \\phi(x^{++})\\ge 0, \\label{eq:oc_simple_2} \\\\\n\t\t\\langle \\nabla f(x) + \\eta (x' - x), u - x'\\rangle + \\phi(u) - \\phi(x') \\ge 0 \\label{eq:oc_simple_3}\n\t\\end{align}\n\tfor any $u \\in X$.\n\tTo prove \\eqref{eq:xp_lip}, let us set $u = x^{++}$ in \\eqref{eq:oc_simple_1}, $u=\\xp$ in \\eqref{eq:oc_simple_2}, and add the two relations together. It is easy to see that\n\t$\n\t\\langle (\\eta+\\sigma) (x^+ - x^{++})\n\t+ \\sigma (\\bar x - x), x^{++} - x^+\\rangle  \\ge 0 \n\t$, or equivalently,\n\t$\n\t\\sigma \\langle \\bar x - x, x^{++} - x^+\\rangle \\ge (\\eta+\\sigma) \\|x^+ - x^{++}\\|^2,\n\t$\n\twhich implies \\eqref{eq:xp_lip} by the Cauchy-Schwarz inequality.\n\tNow notice that by definition\n\tthe relation \\eqref{eq:xp_monotonicity} is equivalent to $\\eta\\|x - x'\\|\\le (\\sigma + \\eta)\\|x - x^{+}\\|$.\n\tLet us set $u = x'$ in \\eqref{eq:oc_simple_1}, $u=\\xp$ in \\eqref{eq:oc_simple_3}, and add the two relations together.\n\tThen we have \n\t$\\sigma \\langle x^+ - x, x' - x^+\\rangle\n\t- \\eta \\|x^+ - x'\\|^2 \\ge 0$,\n\twhich implies \n\t$\\eta \\|x^+ - x'\\| \\le \\sigma \\|x - x^+ \\|$ again by the Cauchy-Schwarz inequality.\n\tThe previous inequality together with the triangular inequality then immediately implies\n\t\\[\n\t\\eta\\|x - x'\\| \n\t\\le \\eta (\\|x - x^+\\|\n\t+\\|x^+ - x'\\|) \\le (\\sigma + \\eta) \\|x - x^+ \\|.\n\t\\]\n\tIn order to show \\eqref{eq:3point}, we first notice that\n\t\\eqref{eq:oc_simple_2} is equivalent to \n\t\n\t\\begin{align*}\n\t\t\\begin{aligned}\n\t\t\t\\langle \\nabla f(x), x^{++} - u\\rangle + \\phi(x^{++}) - \\phi(u)\n\t\t\t&\\le ({\\sigma}/{2})(\\|u - \\xu\\|^2 - \\|x^{++} - \\xu\\|^2 - \\|u - x^{++}\\|^2) \\\\\n\t\t\t& + ({\\eta}/{2})(\\|u - x\\|^2 - \\|x^{++} - x\\|^2 - \\|u - x^{++}\\|^2)\n\t\t\\end{aligned}\n\t\\end{align*}\t\n\tfor any $u\\in X$.\n\tMoreover, from \\eqref{eq:local_lip} and the convexity of $f$ we have\n\t\\begin{align*}\n\t\tf(\\xpp) &\\le f(x) + \\langle \\nabla f(x), \\xpp - x\\rangle + ({M}/{2})\\|\\xpp - x\\|^2\\\\\n\t\t&\\le f(x) + \\langle \\nabla f(x), u - x\\rangle \n\t\t+ \\langle \\nabla f(x), \\xpp - u\\rangle\n\t\t+({M}/{2})\\|\\xpp - x\\|^2\\\\\n\t\t&\\le f(u) + \\langle \\nabla f(x), \\xpp - u\\rangle\n\t\t+({M}/{2})\\|\\xpp - x\\|^2.\n\t\\end{align*} \n\tCombining the above two relations we conclude \\eqref{eq:3point}.\n\\qedsymbol\\end{proof}","sha256_utf8":"bf1f2bafede87f0876197b199659513f98e08959d30f4e45f2e3ba67be98e57c"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"写出三个映射最优性条件","source_units":[{"start":41542,"end":42153,"start_line":547,"end_line":554,"raw_tex":"\\begin{proof}\n\tLet us denote $x^+:=x^+(\\eta+\\sigma; x)$, $x^{++}:=x^{++}(\\eta, \\sigma, \\xu; x)$ and $x' := x^+(\\eta; x)$. By the optimality conditions of \\eqref{eq:xp} and \\eqref{eq:xpp} we have \n\t\\begin{align}\n\t\t\\langle \\nabla f(x) + (\\eta+\\sigma)(x^+ - x), u - x^{+}\\rangle + \\phi(u) - \\phi(x^{+}) \\ge 0, \\label{eq:oc_simple_1} \\\\\n\t\t\\langle \\nabla f(x) + \\sigma (x^{++}-\\bar x) +\\eta (x^{++} - x), u - x^{++}\\rangle + \\phi(u) - \\phi(x^{++})\\ge 0, \\label{eq:oc_simple_2} \\\\\n\t\t\\langle \\nabla f(x) + \\eta (x' - x), u - x'\\rangle + \\phi(u) - \\phi(x') \\ge 0 \\label{eq:oc_simple_3}\n\t\\end{align}\n\tfor any $u \\in X$.\n","sha256_utf8":"95ee96434a1a13393e7238cede8260821ce5baf3bf8a5811fd36df9b83186353"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"证明扰动映射距离界","source_units":[{"start":42153,"end":42600,"start_line":555,"end_line":563,"raw_tex":"\tTo prove \\eqref{eq:xp_lip}, let us set $u = x^{++}$ in \\eqref{eq:oc_simple_1}, $u=\\xp$ in \\eqref{eq:oc_simple_2}, and add the two relations together. It is easy to see that\n\t$\n\t\\langle (\\eta+\\sigma) (x^+ - x^{++})\n\t+ \\sigma (\\bar x - x), x^{++} - x^+\\rangle  \\ge 0 \n\t$, or equivalently,\n\t$\n\t\\sigma \\langle \\bar x - x, x^{++} - x^+\\rangle \\ge (\\eta+\\sigma) \\|x^+ - x^{++}\\|^2,\n\t$\n\twhich implies \\eqref{eq:xp_lip} by the Cauchy-Schwarz inequality.\n","sha256_utf8":"f998c00954822827770f2ab76989154af89c3900538c16773198a251ae8a6085"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"证明投影梯度单调性","source_units":[{"start":42600,"end":43235,"start_line":564,"end_line":577,"raw_tex":"\tNow notice that by definition\n\tthe relation \\eqref{eq:xp_monotonicity} is equivalent to $\\eta\\|x - x'\\|\\le (\\sigma + \\eta)\\|x - x^{+}\\|$.\n\tLet us set $u = x'$ in \\eqref{eq:oc_simple_1}, $u=\\xp$ in \\eqref{eq:oc_simple_3}, and add the two relations together.\n\tThen we have \n\t$\\sigma \\langle x^+ - x, x' - x^+\\rangle\n\t- \\eta \\|x^+ - x'\\|^2 \\ge 0$,\n\twhich implies \n\t$\\eta \\|x^+ - x'\\| \\le \\sigma \\|x - x^+ \\|$ again by the Cauchy-Schwarz inequality.\n\tThe previous inequality together with the triangular inequality then immediately implies\n\t\\[\n\t\\eta\\|x - x'\\| \n\t\\le \\eta (\\|x - x^+\\|\n\t+\\|x^+ - x'\\|) \\le (\\sigma + \\eta) \\|x - x^+ \\|.\n\t\\]\n","sha256_utf8":"99e99c81f3f2fc2f82829918988e5b5508da6d13c4c179152a7f352503392f56"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"将最优性条件转为距离差","source_units":[{"start":43235,"end":43640,"start_line":578,"end_line":588,"raw_tex":"\tIn order to show \\eqref{eq:3point}, we first notice that\n\t\\eqref{eq:oc_simple_2} is equivalent to \n\t\n\t\\begin{align*}\n\t\t\\begin{aligned}\n\t\t\t\\langle \\nabla f(x), x^{++} - u\\rangle + \\phi(x^{++}) - \\phi(u)\n\t\t\t&\\le ({\\sigma}/{2})(\\|u - \\xu\\|^2 - \\|x^{++} - \\xu\\|^2 - \\|u - x^{++}\\|^2) \\\\\n\t\t\t& + ({\\eta}/{2})(\\|u - x\\|^2 - \\|x^{++} - x\\|^2 - \\|u - x^{++}\\|^2)\n\t\t\\end{aligned}\n\t\\end{align*}\t\n\tfor any $u\\in X$.\n","sha256_utf8":"ac38cfb544792522933ba39a8d239bec7037bdba4aa2c6b04f26a128b23ce1ba"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"结合凸性得到三点界","source_units":[{"start":43640,"end":44113,"start_line":589,"end_line":599,"raw_tex":"\tMoreover, from \\eqref{eq:local_lip} and the convexity of $f$ we have\n\t\\begin{align*}\n\t\tf(\\xpp) &\\le f(x) + \\langle \\nabla f(x), \\xpp - x\\rangle + ({M}/{2})\\|\\xpp - x\\|^2\\\\\n\t\t&\\le f(x) + \\langle \\nabla f(x), u - x\\rangle \n\t\t+ \\langle \\nabla f(x), \\xpp - u\\rangle\n\t\t+({M}/{2})\\|\\xpp - x\\|^2\\\\\n\t\t&\\le f(u) + \\langle \\nabla f(x), \\xpp - u\\rangle\n\t\t+({M}/{2})\\|\\xpp - x\\|^2.\n\t\\end{align*} \n\tCombining the above two relations we conclude \\eqref{eq:3point}.\n\\qedsymbol\\end{proof}","sha256_utf8":"f4310660b5e7fe8fb7daf673a54551d49918e6a69a44b5e00172a61ddb354bb0"}]},{"id":"S037","semantic_type":"proof","kind":"proof","title":"原始梯度距离估计推导","source_units":[{"start":25781,"end":27502,"start_line":331,"end_line":355,"raw_tex":"Specifically, \nnoting the $L$-Lipschitz continuity of $\\nabla f(x)$ and using the optimality condition $\\nabla f_s(x_s^*)=\\nabla f(x_s^*) + \\sigma_s(x_s^* - \\xu_s) = 0$ at $x_s^*$,\nwe have\n\\begin{align}\n\t\\label{eq:xs_norm_bound_unconstrained}\n\t\\begin{aligned}\n\t\t& \\|\\nabla f(x_s)\\| = \\|\\nabla f(x_s) - \\nabla f(x_s^*) - \\sigma_s(x_s^* - \\xu_s)\\| \n\t\t\\\\\n\t\t\\le \\ & \\|\\nabla f(x_s) - \\nabla f(x_s^*)\\| + \\sigma_s\\|x_s^* - \\xu_s\\| \n\t\t\\\\\n\t\t\\le\\ &  L\\|x_s - x_s^*\\| + \\sigma_s\\|x_s^* - \\xu_s\\|.\n\t\\end{aligned}\n\\end{align}\nWe will show later that with the increasing regularization parameter $\\sigma_s$ and convex combination design of prox-center in \\eqref{eq:xus_unconstrained}, the distance $\\sigma_s\\|x_s^* - \\xu_s\\|$ can be further bounded:\n\\begin{align}\n\t\\label{eq:claim_on_distance_unconstrained}\n\t\\sigma_s\\|x_s^* - \\xu_s\\| \\le \\sigma_1\\|x^* - x_0\\| + \\sum_{i=2}^s(\\sigma_{i-1} + \\sigma_i)\\|x_{i-1}^* - x_{i-1}\\|.\n\\end{align}\nTherefore, $\\|\\nabla f(x_s)\\|$ can be controled by\n\\begin{align}\n\t\\label{eq:gradS_estimate_unconstrained}\n\t\\|\\nabla f(x_s)\\| \\le\\ &  L\\|x_s - x_s^*\\| + \\sigma_1\\|x^* - x_0\\| + \\sum_{i=2}^s(\\sigma_{i-1} + \\sigma_i)\\|x_{i-1}^* - x_{i-1}\\|.\n\\end{align}\nNote that the right-hand-side of the above estimate is the summation of terms of form $\\|x_i - x_i^*\\|$, i.e., the distance from an approximate solution $x_i$ to the optimal solution $x_i^*$ of the $i$-th strongly convex subproblem.\nBy studying the convergence properties of subroutine $\\cA$ for minimizing smooth and strongly convex functions in subproblem \\eqref{eq:subproblem_unconstrained}, we will then bound the total number of gradient evaluations needed to reduce the right-hand-side of the above estimate to the accuracy $\\varepsilon$. \n","sha256_utf8":"e1ac878f85392e104bf9d9c6a7a60f9407301820d98585dc277b9d8073e290c2"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"利用最优性分解梯度","source_units":[{"start":25781,"end":26296,"start_line":331,"end_line":343,"raw_tex":"Specifically, \nnoting the $L$-Lipschitz continuity of $\\nabla f(x)$ and using the optimality condition $\\nabla f_s(x_s^*)=\\nabla f(x_s^*) + \\sigma_s(x_s^* - \\xu_s) = 0$ at $x_s^*$,\nwe have\n\\begin{align}\n\t\\label{eq:xs_norm_bound_unconstrained}\n\t\\begin{aligned}\n\t\t& \\|\\nabla f(x_s)\\| = \\|\\nabla f(x_s) - \\nabla f(x_s^*) - \\sigma_s(x_s^* - \\xu_s)\\| \n\t\t\\\\\n\t\t\\le \\ & \\|\\nabla f(x_s) - \\nabla f(x_s^*)\\| + \\sigma_s\\|x_s^* - \\xu_s\\| \n\t\t\\\\\n\t\t\\le\\ &  L\\|x_s - x_s^*\\| + \\sigma_s\\|x_s^* - \\xu_s\\|.\n\t\\end{aligned}\n\\end{align}\n","sha256_utf8":"80e80697150ea3c7472a6ba15e82549f145e546626307d19ba70a7f3cd761e33"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"引入累积中心偏差界","source_units":[{"start":26296,"end":26706,"start_line":344,"end_line":348,"raw_tex":"We will show later that with the increasing regularization parameter $\\sigma_s$ and convex combination design of prox-center in \\eqref{eq:xus_unconstrained}, the distance $\\sigma_s\\|x_s^* - \\xu_s\\|$ can be further bounded:\n\\begin{align}\n\t\\label{eq:claim_on_distance_unconstrained}\n\t\\sigma_s\\|x_s^* - \\xu_s\\| \\le \\sigma_1\\|x^* - x_0\\| + \\sum_{i=2}^s(\\sigma_{i-1} + \\sigma_i)\\|x_{i-1}^* - x_{i-1}\\|.\n\\end{align}\n","sha256_utf8":"b3f7af1ad61d863c76127562738c1a5cc06e97ee6350899a104f786217e45610"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"得到梯度误差累加形式","source_units":[{"start":26706,"end":27502,"start_line":349,"end_line":355,"raw_tex":"Therefore, $\\|\\nabla f(x_s)\\|$ can be controled by\n\\begin{align}\n\t\\label{eq:gradS_estimate_unconstrained}\n\t\\|\\nabla f(x_s)\\| \\le\\ &  L\\|x_s - x_s^*\\| + \\sigma_1\\|x^* - x_0\\| + \\sum_{i=2}^s(\\sigma_{i-1} + \\sigma_i)\\|x_{i-1}^* - x_{i-1}\\|.\n\\end{align}\nNote that the right-hand-side of the above estimate is the summation of terms of form $\\|x_i - x_i^*\\|$, i.e., the distance from an approximate solution $x_i$ to the optimal solution $x_i^*$ of the $i$-th strongly convex subproblem.\nBy studying the convergence properties of subroutine $\\cA$ for minimizing smooth and strongly convex functions in subproblem \\eqref{eq:subproblem_unconstrained}, we will then bound the total number of gradient evaluations needed to reduce the right-hand-side of the above estimate to the accuracy $\\varepsilon$. \n","sha256_utf8":"fe33951db7658bcdd042c1df0271adff26ae95c95edf57544c73adc211a1113f"}]},{"id":"S041","semantic_type":"proof","kind":"proof","title":"局部自适应梯度估计推导","source_units":[{"start":60739,"end":61803,"start_line":871,"end_line":883,"raw_tex":"The above lemma allows us to design a backtracking function described in Algorithm~\\ref{alg:_searchL}. Applying Lemma~\\ref{thm:Mlemma} with $x = x_s$, $x^* = x_s^*$, $g = f_s$, $M=M_s+\\sigma_s$, and $\\eta = 2(M_s+\\sigma_s)$, we conclude that whenever \\eqref{eq:M} holds, \nwe have $\\|\\nabla f_s(x_s)\\|\\le 2\\sqrt{2}(M_s+\\sigma_s)\\|x_s - x_s^*\\|$. As a consequence,\n\\begin{align}\n\t\\label{eq:xs_norm_bound_unconstrained_est}\n\t\\begin{aligned}\n\t\t& \\|\\nabla f(x_s)\\| = \\|\\nabla f_s(x_s) - \\sigma_s(x_s - \\xu_s)\\| \n\t\t=  \\|\\nabla f_s(x_s) - \\sigma_s(x_s - x_s^*) - \\sigma_s(\\xu_s-x_s^*)\\| \n\t\t\\\\\n\t\t\\le &  (2\\sqrt{2}M_s+(2\\sqrt{2}+1)\\sigma_s)\\|x_s - x_s^*\\| + \\sigma_s\\|x_s^* - \\xu_s\\|,\n\t\\end{aligned}\n\\end{align}\nwhich is analogous to the gradient-to-distance relation \\eqref{eq:xs_norm_bound_unconstrained} but with $L$ replaced by $M_s$.\nApparently, we can apply one single gradient descent step with backtracking, i.e., Algorithm~\\ref{alg:_searchL} with input $g=f_s$, $\\sigma =\\sigma_s$, $x= x_s$ and initial guess $M_0$, to estimate the local Lipschitz constant $M_s$.\n","sha256_utf8":"c1b4fce0593b98f5a22e2bc64d1796f95f5112ed10cc1df39dabe1fd62448d4a"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"对正则化函数应用局部界","source_units":[{"start":60739,"end":61102,"start_line":871,"end_line":872,"raw_tex":"The above lemma allows us to design a backtracking function described in Algorithm~\\ref{alg:_searchL}. Applying Lemma~\\ref{thm:Mlemma} with $x = x_s$, $x^* = x_s^*$, $g = f_s$, $M=M_s+\\sigma_s$, and $\\eta = 2(M_s+\\sigma_s)$, we conclude that whenever \\eqref{eq:M} holds, \nwe have $\\|\\nabla f_s(x_s)\\|\\le 2\\sqrt{2}(M_s+\\sigma_s)\\|x_s - x_s^*\\|$. As a consequence,\n","sha256_utf8":"cec404b5d94a1a371709e516a57a64774a68228f0c35a56dc3de1e708f7bd5e6"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"分离梯度误差与中心偏差","source_units":[{"start":61102,"end":61442,"start_line":873,"end_line":881,"raw_tex":"\\begin{align}\n\t\\label{eq:xs_norm_bound_unconstrained_est}\n\t\\begin{aligned}\n\t\t& \\|\\nabla f(x_s)\\| = \\|\\nabla f_s(x_s) - \\sigma_s(x_s - \\xu_s)\\| \n\t\t=  \\|\\nabla f_s(x_s) - \\sigma_s(x_s - x_s^*) - \\sigma_s(\\xu_s-x_s^*)\\| \n\t\t\\\\\n\t\t\\le &  (2\\sqrt{2}M_s+(2\\sqrt{2}+1)\\sigma_s)\\|x_s - x_s^*\\| + \\sigma_s\\|x_s^* - \\xu_s\\|,\n\t\\end{aligned}\n\\end{align}\n","sha256_utf8":"1ca1ebd9b2d4f7674b357b8e886483a34b4f3c61143f379c1646af3850af90f1"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"以回溯获得局部估计","source_units":[{"start":61442,"end":61803,"start_line":882,"end_line":883,"raw_tex":"which is analogous to the gradient-to-distance relation \\eqref{eq:xs_norm_bound_unconstrained} but with $L$ replaced by $M_s$.\nApparently, we can apply one single gradient descent step with backtracking, i.e., Algorithm~\\ref{alg:_searchL} with input $g=f_s$, $\\sigma =\\sigma_s$, $x= x_s$ and initial guess $M_0$, to estimate the local Lipschitz constant $M_s$.\n","sha256_utf8":"7c1969d513160e891299221ab9e872b1c85e0de54eb5235c4882d6a7ef6d9483"}]},{"id":"S045","semantic_type":"problem_setting","kind":"problem_setting","title":"无约束问题与光滑凸性","source_units":[{"start":6290,"end":6517,"start_line":152,"end_line":157,"raw_tex":"The basic problem of interest in this paper is to compute an\napproximate stationary point $\\hat x$ to  \nthe unconstrained optimization problem\n\\begin{equation} \\label{eq:basic_problem}\n\tf^* = \\min_{x\\in\\R^n}f(x),\n\\end{equation}","sha256_utf8":"36bd7176ac4c439d60c8a05e58fa50df078385daae896fd0c33efaf710f2b279"},{"start":6518,"end":7194,"start_line":158,"end_line":161,"raw_tex":"where $f$ is $L$-smooth over $\\R^n$, meaning that $f$ is differentiable and its gradient $\\nabla f$ is Lipschitz continuous, i.e., $\\|\\nabla f(x) - \\nabla f(y)\\|\n\\le L \\|x - y\\|$ for any $x, y \\in \\R^n$. \\cmo{ Here and throughout this paper, the norm $\\|\\cdot\\|$ measured over points $x\\in \\R^n$ and gradients $\\nabla f(x)\\in\\R^n$ are all Euclidean norm.}\nMoreover, we consider three important types of $f$ as follows: (a) $f$ is convex, i.e.,\n$f(x) \\ge f(y) +\\langle \\nabla f(y), x-y\\rangle$ for any $x, y \\in \\R^n$; (b) $f$ is $\\mu$-strongly convex, i.e., $f(x) \\ge f(y) +\\langle \\nabla f(y), x-y\\rangle + \\mu \\|x-y\\|^2/2$ $\\forall x, y \\in \\R^n$ for some $\\mu \\in (0, L)$;\n","sha256_utf8":"b9279492760d578d418a5933df4dc7d17c609d5991bf41715edd560c32365b8a"}]},{"id":"S046","semantic_type":"algorithm","kind":"algorithm","title":"累积正则化基本算法","source_units":[{"start":22844,"end":23854,"start_line":277,"end_line":296,"raw_tex":"\\begin{algorithm}[h]\n\t\\caption{\\label{alg:proposed_unconstrained} An accumulative regularization method for gradient minimization}\n\t\\begin{algorithmic}\n\t\t\\Require Total number of subproblems $S$, strictly increasing regularization parameters $\\{\\sigma_s\\}_{s=0}^{S}$ with $\\sigma_0=0$, and initial point $x_0\\in\\R^n$. \\State Set initial prox-center to $\\xu_0 := x_0$.\n\t\t\\For {$s=1,\\ldots, S$}\n\t\t\\State Set\n\t\t\\begin{align}\n\t\t\t\\label{eq:xus_unconstrained}\n\t\t\t\\xu_s = (1-\\gamma_s) \\xu_{s-1} + \\gamma_s x_{s-1}\\text{ with }\\gamma_s =1 - \\sigma_{s-1}/\\sigma_s.\n\t\t\\end{align}\n\t\t\\State Compute an approximate solution $x_s$ of the proximal mapping subproblem\n\t\t\\begin{align}\n\t\t\t\\label{eq:subproblem_unconstrained}\n\t\t\tx_s^*:=\\argmin_{x\\in \\R^n} \\{f_s(x):=f(x) + \\frac{\\sigma_s}{2}\\|x - \\xu_s\\|^2\\}\n\t\t\\end{align}\n\t\tby running subroutine $\\cA(f, \\sigma_s, \\xu_s, x_{s-1}, N_s)$ with $N_s$ gradient evaluations of $\\nabla f$ and initial point $x_{s-1}$.\n\t\t\\EndFor\n\t\t\\State Output $x_S$.\n\t\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"a19c99fb5736563126cb6e840233a73707989504cd06d9917a77c0777c5c6ac7"}]},{"id":"S047","semantic_type":"notation","kind":"notation","title":"零阶段最优点约定","source_units":[{"start":27503,"end":28203,"start_line":357,"end_line":358,"raw_tex":"We are now ready to describe the details for proving our claim \\eqref{eq:claim_on_distance_unconstrained} and estimating the number of gradient evaluations required to solve subproblems \\eqref{eq:subproblem_unconstrained}. For convenience, we allow that the definition   \\eqref{eq:subproblem_unconstrained} applies for $s=0$, and in such case we have $\\sigma_0 = 0$ and hence $x_0^*=x^*$ is an optimal solution to the original problem. The proof of our claim \\eqref{eq:claim_on_distance_unconstrained} is described in the following proposition. Note that the result \\eqref{eq:dist_xu_xstarsig_unconstrained} below is exactly our claim\n\\eqref{eq:claim_on_distance_unconstrained} since $\\sigma_0 = 0$.\n","sha256_utf8":"591475ce38be85586e460c7530271ab09ec1bda074d3accff0d8a68f0f6923e5"}]},{"id":"S048","semantic_type":"problem_setting","kind":"problem_setting","title":"复合凸问题及可行集","source_units":[{"start":35894,"end":36274,"start_line":472,"end_line":475,"raw_tex":"In this section, we show that the accumulative regularization technique we introduce in the previous section can be adapted to problems with constrained feasible sets, and (nested) composite objective function. We consider the problem with simple feasible set $X$ and composite function $\\phi$:\n\\begin{align}\n\t\\label{eq:problem} \\min_{x\\in X}\\{ F(x):=f(x) + \\phi(x)\\}.\n\\end{align}","sha256_utf8":"e4352f090d65b3812d33cd46f4746ff1ebefd5c53fcdef8c614b54003b8dedaa"},{"start":36275,"end":36607,"start_line":476,"end_line":476,"raw_tex":"Here $X \\subset \\R^n$ is a closed convex set and $f$ and $\\phi$ are closed convex functions. We still assume that $\\nabla f$ is $L$-smooth. In addition, we assume that $X$ and $\\phi$ are  relatively simple such that for any $\\sigma, \\eta\\ge 0$ and any $x,\\xu\\in\\R^n$, the following optimization problem can be computed efficiently:\n","sha256_utf8":"29314342c10002a3f3699a2404df38b94b24054ac83a0469b63267f2f54ea34a"},{"start":36797,"end":36899,"start_line":481,"end_line":481,"raw_tex":"Throughout this paper we assume that an optimal solution $x^*$ to problem \\eqref{eq:problem} exists. \n","sha256_utf8":"8b1decb3d833caa1906fcfc1a327eb0ea8f34ec355583c05087afdde2e768f76"}]},{"id":"S049","semantic_type":"definition","kind":"definition","title":"扰动映射与投影梯度","source_units":[{"start":36607,"end":36796,"start_line":477,"end_line":480,"raw_tex":"\\begin{align}\n\t\\label{eq:xpp}\n\tx^{++}(\\eta, \\sigma, \\xu; x):=\\argmin_{u\\in X}\\langle \\nabla f(x), u\\rangle + \\phi(u) + \\frac{\\sigma}{2}\\|u - \\xu\\|^2 + \\frac{\\eta}{2}\\|u - x\\|^2.\n\\end{align}","sha256_utf8":"a8d4f4351861861508f67c27fb2da8841dbd13e81496134331d59a9217611a16"},{"start":36899,"end":37301,"start_line":482,"end_line":489,"raw_tex":"Clearly, $x^*$ is an optimal solution of \\eqref{eq:problem} if and only if the \\emph{projected gradient} $G_\\eta(x^*) = 0$ for some $\\eta>0$, where\n\\begin{align}\n\t\\label{eq:G}\n\tG_{\\eta}(x) :=\\ & \\eta(x - x^+(\\eta;x))\\text{ and }\n\t\\\\\n\t\\label{eq:xp}\n\tx^+(\\eta;x):=\\ &x^{++}(\\eta, 0, x;x) = \\argmin_{u\\in X}\\langle\\nabla f(x), u\\rangle + \\phi(u) + \\frac{\\eta}{2}\\|u - x\\|^2\\ \\forall x\\in \\R^n.\n\\end{align}","sha256_utf8":"7923fcf7ff71af8db5ac6b08f26aba6685c1716ceb1f6f5a075c1fa47a325f70"}]},{"id":"S050","semantic_type":"algorithm","kind":"algorithm","title":"局部光滑常数回溯","source_units":[{"start":60211,"end":60736,"start_line":853,"end_line":868,"raw_tex":"\\begin{algorithm*}[h] \n\t\\caption{$M$=Backtracking($g$, $\\sigma$, $x$, $M_0$) \\label{alg:_searchL}}\n\t\\begin{algorithmic}\n\t\t\\For{$j=0,1,\\ldots,$}\n\t\t\\State Set\n\t\t$\t\t\t\\xpp = x - ({1}/{(2(M_j + \\sigma))})\\nabla g(x).\n\t\t$\n\t\t\\State If \n\t\t\\begin{align}\n\t\t\t\\label{eq:linesearch_estL_success_criterion}\n\t\t\tg(\\xpp) - g(x) - \\langle \\nabla g(x), \\xpp - x\\rangle \\le ({(M_j + \\sigma)}/{2})\\|\\xpp - x\\|^2, \n\t\t\\end{align}\n\t\tthen {\\bf terminate} with $M=M_j$. Otherwise, set $M_{j+1} = 2M_{j}$. \n\t\t\\EndFor\n\t\\end{algorithmic}\n\\end{algorithm*}","sha256_utf8":"eed46d770744b9aaf4368450b56ed183025ed2b4ed735451362538e3171741aa"}]},{"id":"S051","semantic_type":"algorithm","kind":"algorithm","title":"无需光滑常数的AR算法","source_units":[{"start":62469,"end":63489,"start_line":890,"end_line":911,"raw_tex":"\\begin{algorithm}[!h]\n\t\\caption{\\label{alg:proposed_unconstrained_searchL} Accumulative regularization (AR) without the knowledge of $L$}\n\t\\begin{algorithmic}\n\t\t\\Function{$(\\hat x, M)=\\ $AR}{$f, x_0, \\sigma_1, M_0$}\n\t\t\\State Set the initial prox-center $\\xu_0 = x_0$, and denote $\\sigma_0=0$.\n\t\t\\For {$s=1,2,\\ldots$}\n\t\t\\State If $s>1$, then set $\\sigma_s=4\\sigma_{s-1}$.\n\t\t\\State Set $\\xu_s =  (1-\\gamma_s) \\xu_{s-1} + \\gamma_s x_{s-1}$ with $\\gamma_s=1 - \\sigma_{s-1}/\\sigma_s$.\n\t\t\n\t\t\\State Compute an approximate solution $x_s\\in \\R^n$ of\n\t\t\\begin{align}\n\t\t\t\\label{eq:subproblem_unconstrained_searchL}\n\t\t\tx_s^*:=\\argmin_{x\\in \\R^n} \\{ f_s(x):=f(x) + \\dfrac{\\sigma_s}{2}\\|x - \\xu_s\\|^2\\}\n\t\t\\end{align}\n\t\tby running subroutine $x_s = \\cA(f, \\sigma_s, \\xu_s, x_{s-1})$ with initial point $x_{s-1}$. \n\t\t\n\t\t\\State Set $M_s=$ Backtracking ($f_s$, $\\sigma_s$, $x_s$, $M_{s-1}/2$).\n\t\t\\State If $\\sigma_s\\ge M_s$, then {\\bf terminate} with $\\hat x=x_s$ and $M=M_s$ . \n\t\t\\EndFor\n\t\t\\EndFunction\n\t\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"7145164813347259e38f8361e3e9f50093e91622f9db9293bc85429785671a45"}]},{"id":"S052","semantic_type":"assumption","kind":"assumption","title":"自适应子程序收敛假设","source_units":[{"start":64684,"end":65376,"start_line":925,"end_line":934,"raw_tex":"\\begin{assumption}\n\t\\label{assum:A_unconstrained_searchL}\n\tThe subroutine $x_s =\\cA(f, \\sigma_s, \\xu_s, x_{s-1})$ has the following performance guarantees after the $k$-th gradient evaluation of $\\nabla f$:\n\t\\begin{align}\n\t\t\\label{eq:A_cond_unconstrained_searchL}\n\t\tf_s(x_s^k) - f_s(x_s^*) \\le ({L_s^k}/{k^2})\\|x_{s-1} - x_s^*\\|^2.\n\t\\end{align} \n\tHere $x_s^k$ is the computed approximate solution and $L_s^k$ is an estimate of Lipschitz constant $L$ such that $L_s^k\\le c_\\cA L$, where $c_\\cA$ is a universal constant that depends on algorithm $\\cA$. \n\tSubroutine $\\cA$ will terminate whenever $k\\ge 8\\sqrt{2L_s^k/\\sigma_s}$ and output the approximate solution $x_s = x_s^k$.\n\\end{assumption}","sha256_utf8":"c5b9b14809a1054177642055aa3ed6782417b9303add09cd5275268d156f5e20"}]},{"id":"S053","semantic_type":"algorithm","kind":"algorithm","title":"SCAR重启与猜测更新","source_units":[{"start":82498,"end":83211,"start_line":1094,"end_line":1107,"raw_tex":"\\begin{algorithm}[h]\n\t\t\\caption{\\label{alg:proposed_SCAR_noflag} A parameter-free strongly convex accumulative regularization (SCAR) algorithm}\n\t\t\\begin{algorithmic}\n\t\t\t\\Function{($\\hat x$, $\\hat M$) = SCAR}{$f$, $\\varepsilon$, $y_0$, $\\mu_0$, $M_0$}\n\t\t\t\\For {$t=1,2,\\ldots$}\n\t\t\t\\State Set\n\t\t\t$\n\t\t\t(y_{t},M_{t}) = \\text{AR}(f, y_{t-1}, \\mu_{t-1}/10, M_{t-1})\n\t\t\t$ (see Algorithm~\\ref{alg:proposed_unconstrained_searchL}).\n\t\t\t\\State If {$\\|\\nabla f(y_t)\\|>\\|\\nabla f(y_{t-1})\\|/2$}, then set $\\mu_t = \\mu_{t-1}/4$ and $y_t = y_{t-1}$. \t\t\n\t\t\t\\State If $\\|\\nabla f(y_{t})\\|\\le \\varepsilon$, then \\textbf{terminate} with $\\hat x = y_{t}$ and $\\hat M = M_t$.\\EndFor\n\t\t\t\\EndFunction\n\t\t\\end{algorithmic}\n\t\\end{algorithm}","sha256_utf8":"90de2a7abe482c90844bdda2e53ba7ea7c1ef889ffea3dceb9193a3d32836ebf"}]},{"id":"S054","semantic_type":"problem_setting","kind":"problem_setting","title":"光滑强凸优化设定","source_units":[{"start":79304,"end":79471,"start_line":1082,"end_line":1082,"raw_tex":"\tOur goal in this section is to develop an optimal and parameter-free algorithm for $\\min_{x\\in\\R^n}f(x)$, in which $f$ is both $L$-smooth and $\\mu$-strongly convex. \n","sha256_utf8":"5e9b275cd7f6f62fffd56cbb2218c84ecdd413644859d35cc4a3a6fb5b95f925"}]},{"id":"S055","semantic_type":"external_fact","kind":"external_fact","title":"凸最优性与欧氏恒等式","source_units":[],"declared_external":true,"declaration_text":["Convex minimization optimality conditions, including the variational inequality for a convex proximal subproblem and vanishing gradient at an unconstrained differentiable minimizer; Euclidean Cauchy–Schwarz, triangle inequality, and squared-distance identities."]},{"id":"S056","semantic_type":"external_fact","kind":"external_fact","title":"强凸函数标准误差界","source_units":[],"declared_external":true,"declaration_text":["For a differentiable m-strongly convex function h with minimizer z*, h(z)-h(z*) is at least (m/2)||z-z*||^2, and ||z-z*|| is at most ||∇h(z)||/m."]},{"id":"S057","semantic_type":"external_fact","kind":"external_fact","title":"光滑函数二次上界","source_units":[],"declared_external":true,"declaration_text":["An L-Lipschitz gradient implies h(v) ≤ h(u)+〈∇h(u),v-u〉+(L/2)||v-u||^2; adding (σ/2)||x-a||^2 increases the smoothness constant by σ."]},{"id":"S058","semantic_type":"external_fact","kind":"external_fact","title":"几何级数与对数估计","source_units":[],"declared_external":true,"declaration_text":["Geometric-series identities, counting repeated halvings by base-two logarithms, elementary ceiling estimates, and the inequality 2 log_4(t) < 3 sqrt(t) for t>0."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S006","target_id":"S002","kind":"supports"},{"source_id":"S007","target_id":"S003","kind":"supports"},{"source_id":"S007","target_id":"S006","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S008","target_id":"S007","kind":"supports"},{"source_id":"S009","target_id":"S008","kind":"supports"},{"source_id":"S010","target_id":"S008","kind":"supports"},{"source_id":"S011","target_id":"S008","kind":"supports"},{"source_id":"S012","target_id":"S008","kind":"supports"},{"source_id":"S013","target_id":"S008","kind":"supports"},{"source_id":"S014","target_id":"S008","kind":"supports"},{"source_id":"S015","target_id":"S008","kind":"supports"},{"source_id":"S016","target_id":"S008","kind":"supports"},{"source_id":"S017","target_id":"S008","kind":"supports"},{"source_id":"S018","target_id":"S008","kind":"supports"},{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S009","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S015","kind":"supports"},{"source_id":"S011","target_id":"S016","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S014","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S019","target_id":"S009","kind":"supports"},{"source_id":"S019","target_id":"S012","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S020","kind":"supports"},{"source_id":"S023","target_id":"S020","kind":"supports"},{"source_id":"S024","target_id":"S020","kind":"supports"},{"source_id":"S025","target_id":"S020","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S022","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S027","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S030","target_id":"S028","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S034","target_id":"S031","kind":"supports"},{"source_id":"S035","target_id":"S031","kind":"supports"},{"source_id":"S036","target_id":"S031","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S032","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S037","kind":"supports"},{"source_id":"S040","target_id":"S037","kind":"supports"},{"source_id":"S019","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S037","target_id":"S009","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S043","target_id":"S041","kind":"supports"},{"source_id":"S044","target_id":"S041","kind":"supports"},{"source_id":"S026","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S041","target_id":"S009","kind":"supports"},{"source_id":"S045","target_id":"S038","kind":"supports"},{"source_id":"S045","target_id":"S021","kind":"supports"},{"source_id":"S045","target_id":"S010","kind":"supports"},{"source_id":"S046","target_id":"S021","kind":"supports"},{"source_id":"S046","target_id":"S023","kind":"supports"},{"source_id":"S046","target_id":"S038","kind":"supports"},{"source_id":"S047","target_id":"S022","kind":"supports"},{"source_id":"S047","target_id":"S025","kind":"supports"},{"source_id":"S047","target_id":"S039","kind":"supports"},{"source_id":"S047","target_id":"S013","kind":"supports"},{"source_id":"S048","target_id":"S032","kind":"supports"},{"source_id":"S048","target_id":"S036","kind":"supports"},{"source_id":"S049","target_id":"S032","kind":"supports"},{"source_id":"S049","target_id":"S028","kind":"supports"},{"source_id":"S050","target_id":"S042","kind":"supports"},{"source_id":"S050","target_id":"S044","kind":"supports"},{"source_id":"S050","target_id":"S015","kind":"supports"},{"source_id":"S050","target_id":"S016","kind":"supports"},{"source_id":"S051","target_id":"S009","kind":"supports"},{"source_id":"S051","target_id":"S011","kind":"supports"},{"source_id":"S051","target_id":"S013","kind":"supports"},{"source_id":"S051","target_id":"S014","kind":"supports"},{"source_id":"S051","target_id":"S015","kind":"supports"},{"source_id":"S051","target_id":"S016","kind":"supports"},{"source_id":"S052","target_id":"S010","kind":"supports"},{"source_id":"S052","target_id":"S011","kind":"supports"},{"source_id":"S052","target_id":"S012","kind":"supports"},{"source_id":"S053","target_id":"S003","kind":"supports"},{"source_id":"S053","target_id":"S004","kind":"supports"},{"source_id":"S053","target_id":"S005","kind":"supports"},{"source_id":"S053","target_id":"S006","kind":"supports"},{"source_id":"S054","target_id":"S004","kind":"supports"},{"source_id":"S055","target_id":"S032","kind":"supports"},{"source_id":"S055","target_id":"S033","kind":"supports"},{"source_id":"S055","target_id":"S034","kind":"supports"},{"source_id":"S055","target_id":"S035","kind":"supports"},{"source_id":"S055","target_id":"S036","kind":"supports"},{"source_id":"S055","target_id":"S038","kind":"supports"},{"source_id":"S055","target_id":"S043","kind":"supports"},{"source_id":"S055","target_id":"S025","kind":"supports"},{"source_id":"S056","target_id":"S010","kind":"supports"},{"source_id":"S056","target_id":"S004","kind":"supports"},{"source_id":"S057","target_id":"S015","kind":"supports"},{"source_id":"S058","target_id":"S013","kind":"supports"},{"source_id":"S058","target_id":"S016","kind":"supports"},{"source_id":"S058","target_id":"S017","kind":"supports"},{"source_id":"S058","target_id":"S005","kind":"supports"},{"source_id":"S058","target_id":"S006","kind":"supports"}]}}
{"record_id":"2401.00108v2","paper":{"title":"Stochastic Optimization under Hidden Convexity","authors":["Ilyas Fatkhullin","Niao He","Yifan Hu"],"arxiv_id":"2401.00108","version":"v2","journal":"SIAM Journal on Optimization","doi":"10.1137/22m1708903","abstract_url":"https://arxiv.org/abs/2401.00108v2","pdf_url":"https://arxiv.org/pdf/2401.00108v2"},"context":{"raw_tex":"\\begin{equation}\n\t\t\\label{problem:original}\n\t\t\\min_{x\\in\\cX}~F(x) := \\Expu{\\xi \\sim \\cD}{f(x, \\xi)}, \n\t\\end{equation}\n\n\twhere $\\cX $ is a closed convex subset of $\\R^d$, $\\xi$ is a random variable following an unknown distribution $\\cD$, and $F(\\cdot)$ is possibly non-convex in $x$. Our central structural assumption about \\eqref{problem:original} is that it admits a \\textit{convex reformulation} of the form\n\n\n\\begin{equation}\n\t\t\\label{problem:reformulation}\n\t\t\\min_{u\\in \\cU}~H(u):= F(c^{-1}(u)), %\\text{with  } g(x) = u,\n\t\\end{equation}\n\n\tThroughout, we denote by $\\ve{\\cdot}{\\cdot}$ the inner product in $\\R^d$ along with its induced Euclidean norm $\\norm{\\cdot}$. For a real valued matrix $A\\in \\R^{m\\times n}$, we denote by $\\opnorm{\\cdot}$ its operator norm, i.e.,  $\\opnorm{A} := \\max_{\\norm{x}\\leq 1}\\norm{A x}$. {\\color{black} We denote the interior and the boundary of $\\cX$ as $\\operatorname{int} \\cX$ and $\\partial \\cX$ respectively.} %For any map $c: \\cX \\rightarrow \\cU$, we say $c$ is $L$-Lipschitz continuous on the set $\\cX \\subset \\R^d$ if there exists a constant $L < +\\infty$ such that for all $x, y\\in \\cX$ it holds $\\norm{c(x) - c(y)} \\leq L \\norm{x - y}. $\n    %We denote $\\text{Ker}( A ) := \\left\\{ x \\in \\R^n | A x = 0 \\right\\} $.\n\tThe map $c: \\cX \\rightarrow \\cU$ is called invertible if there exists a map $c^{-1}: \\cU \\rightarrow \\cX$ (called inverse) such that $c^{-1}(c(x)) = x$ for any $x \\in \\cX$ and $c( c^{-1}(u) ) = u$ for any $u \\in \\cU$. For any $u, v\\in\\cU$ and any $\\lambda\\in[0,1]$, if  $(1-\\lambda) u + \\lambda v \\in \\cU$, we say $\\mathcal{U}$ is convex. We denote the diameter of $\\cU$ as $D_{\\cU}:= \\sup_{u, v\\in \\cU}\\pnorm{u - v}$.\n\n\nA function $F: \\cX \\rightarrow\\R \\cup \\{+\\infty\\}$ is $\\ell$-weakly convex ($\\ell$-WC) if for any fixed $y\\in \\cX$, $F_{\\ell}(x,y) := F(x) + \\frac{\\ell}{2} \\, \\sqnorm{x- y}$ is convex in $x \\in \\cX$.\nIf $x\\notin \\cX$, we assign $F(x) = +\\infty$. The (Fr\\'echet) sub-differential of $F$ at $x\\in \\cX$ is $\\partial F(x) := \\left\\{g \\in \\mathbb{R}^d \\mid  F(y) \\geq F(x)+\\langle g, y-x\\rangle+o(\\|y-x\\|), \\forall y \\in \\R^d  \\right\\}.$ The elements $g \\in \\partial F(x)$ are called sub-gradients of $F$ at $x$, see \\cite{davis2019proximally} for alternative equivalent definitions of the sub-differential set for $\\ell$-WC functions. A differentiable function $F: \\cX \\rightarrow \\R$ is $L$-smooth on $\\cX \\subset \\R^d$ if its gradient is $L$-Lipschitz continuous on the set $\\cX$, i.e., it holds $\\norm{\\nabla F(x) - \\nabla F(y)} \\leq L \\norm{x - y}$ for all $x, y\\in \\cX$. For a convex set $\\cX \\subset \\R^d$, the projection of a point $y\\in \\R^d$ onto $\\cX$ is  $\\Pi_{\\cX}(y) := \\argmin_{x\\in\\cX}\\norm{y - x}$. We denote $\\delta_{\\cX}$ as the indicator function of a set $\\cX \\subset \\R^d$ and define $\\delta_{\\cX}(x)  = 0$ if $x\\in \\cX$ and $\\delta_{\\cX}(x)  = +\\infty$ otherwise. \n\n\n    We define by $\\cX^* \\subset \\cX$ the set of optimal points of $\\min_{x\\in\\cX} F(x)$ and by $F^*$ its optimal value. A point $\\bar{x} \\in \\cX$ is called a stationary point of a weakly convex function $F: \\cX \\rightarrow \\R$ if $0 \\in \\partial (F +  \\delta_{\\cX})(\\bar{x})$.\n\n\n\\begin{equation}\n\t\t\\label{problem:original_composition}\n\t\t\\min_{x\\in\\cX}~F(x) := H(c(x)) . \n\t\\end{equation}\n\n\\begin{definition}\nThe  above problem is called {hidden convex} with modulus $\\mu_c>0, \\mu_H\\geq 0$ (or function $F$ is hidden convex on $\\cX$) if its components satisfy the following underlying conditions. \n\t\\begin{enumerate}%[ref=\\theenumi{}~(\\alph*)]\n\t\t%\\item The set $\\cX \\subset \\R^d $ is closed and convex. \\label{ass:convex_cX}\n\t\t\\myitem{C.1}\\label{C1} The domain $\\mathcal{U}=c(\\mathcal{X})$ is convex, the function $H: \\cU \\rightarrow \\R$ satisfies that for all $u, v \\in \\cU$ and $ \\lambda \\in [0,1]$, \n  \\begin{equation}\\label{assumption:H_conv}\n  \tH( (1-\\lambda)u + \\lambda v ) \\leq  (1- \\lambda) H(u) +  \\lambda  H(v)  -   \\fr{(1-\\lambda)\\lambda \\mu_H}{2} \\sqpnorm{ u - v }, \n  \\end{equation}\n  and \\eqref{problem:reformulation} admits a solution $u^* \\in \\cU$.  %\\label{ass:convex_H} \n  \n\t\t\\myitem{C.2} The map $c: \\cX \\rightarrow \\cU$ is invertible. There exists $\\mu_c>0$ such that  for all $x, y \\in \\cX$ it holds  \\label{C2}\n\t\t\\begin{equation}\\label{assumption:g_inv}\n\t\t\t\\pnorm{c(x) - c(y) } \\geq \\mu_c \\pnorm{x - y }.\n\t\t\\end{equation}\n\t\t%\\item The function $F: \\cX \\rightarrow \\R$ is differentiable on $\\cX$ and its gradient $\\nabla F(x)$ is $L$-Lipschitz continuous. \n\t\\end{enumerate}\n\t%\\IF{Check if A.2. is connected to metric subregularity condition from the works of Russel Luke. }\n\t%\\end{assumption}\nIn particular, if $\\mu_H>0$, we say the above problem is {$(\\mu_c,\\mu_H)$-hidden strongly convex}.  \n\\end{definition}\n\n    \\begin{enumerate}\n    \\myitem{A.1'}\\label{A1'} The function $F: \\cX \\rightarrow \\R$ is differentiable on a closed, convex set $\\cX$ and its gradient $\\nabla F(x)$ is $L$-Lipschitz continuous. \n    \n    \\myitem{A.2'}\\label{A2'} We have access to an unbiased stochastic gradient oracle with bounded variance $\\sigma > 0$, i.e. for any $x\\in \\cX$: \n\t$\\Exp{\\nabla f(x,\\xi)} = \\nabla F(x)$, and \n\t$$\\Exp{\\sqnorm{\\nabla f(x,\\xi) - \\nabla F(x)}} \\leq \\sigma^2, $$where expectations are with respect to the random variable $\\xi \\sim \\cD$.  \n    \\end{enumerate}\n\n\n\\begin{align}\n\t%\\boxed{ \n\t\tx^{t+1} = \\Pi_{\\mathcal{X}}(x^t-\\stepsize \\, g^t) , \\qquad  \\label{eq:PSGDM_x_upd}\n\t\tg^{t+1} = (1-\\momentum) \\, g^{t} + \\momentum \\, \\nabla f(x^{t+1}, \\xi^{t+1}) . \n\t\t%} %\\label{eq:PSGDM_g_upd} \n\\end{align}\n\nLet $x^* \\in \\cX^*$, for any $x^t \\in \\cX$, we define the Lyapunov function\n\\begin{equation}\\label{eq:Lyapunov_HB}\n\t\\Lambda_t^{HB} := \\Exp{ F(x^t) - F(x^*) + \\frac{\\stepsize}{\\momentum} \\sqnorm{g^t - \\nabla F(x^t)} } .\n\\end{equation}","units":[{"start":6335,"end":6452,"start_line":204,"end_line":207,"raw_tex":"\\begin{equation}\n\t\t\\label{problem:original}\n\t\t\\min_{x\\in\\cX}~F(x) := \\Expu{\\xi \\sim \\cD}{f(x, \\xi)}, \n\t\\end{equation}","sha256_utf8":"6f680574a187677e43fd96fb691d83dad3271699ec39907df7ea3f149dabdee7"},{"start":6453,"end":6745,"start_line":208,"end_line":208,"raw_tex":"\twhere $\\cX $ is a closed convex subset of $\\R^d$, $\\xi$ is a random variable following an unknown distribution $\\cD$, and $F(\\cdot)$ is possibly non-convex in $x$. Our central structural assumption about \\eqref{problem:original} is that it admits a \\textit{convex reformulation} of the form\n","sha256_utf8":"0b9dd01cb184ca71b343f3c8ac39d4be905d41a67ce7a52f632b6d49328733a9"},{"start":6746,"end":6874,"start_line":209,"end_line":212,"raw_tex":"\\begin{equation}\n\t\t\\label{problem:reformulation}\n\t\t\\min_{u\\in \\cU}~H(u):= F(c^{-1}(u)), %\\text{with  } g(x) = u,\n\t\\end{equation}","sha256_utf8":"544a22a5643866770ae48a1f48aed0c395ee79b47115a4a57c3cafaaa0249975"},{"start":19048,"end":20184,"start_line":267,"end_line":269,"raw_tex":"\tThroughout, we denote by $\\ve{\\cdot}{\\cdot}$ the inner product in $\\R^d$ along with its induced Euclidean norm $\\norm{\\cdot}$. For a real valued matrix $A\\in \\R^{m\\times n}$, we denote by $\\opnorm{\\cdot}$ its operator norm, i.e.,  $\\opnorm{A} := \\max_{\\norm{x}\\leq 1}\\norm{A x}$. {\\color{black} We denote the interior and the boundary of $\\cX$ as $\\operatorname{int} \\cX$ and $\\partial \\cX$ respectively.} %For any map $c: \\cX \\rightarrow \\cU$, we say $c$ is $L$-Lipschitz continuous on the set $\\cX \\subset \\R^d$ if there exists a constant $L < +\\infty$ such that for all $x, y\\in \\cX$ it holds $\\norm{c(x) - c(y)} \\leq L \\norm{x - y}. $\n    %We denote $\\text{Ker}( A ) := \\left\\{ x \\in \\R^n | A x = 0 \\right\\} $.\n\tThe map $c: \\cX \\rightarrow \\cU$ is called invertible if there exists a map $c^{-1}: \\cU \\rightarrow \\cX$ (called inverse) such that $c^{-1}(c(x)) = x$ for any $x \\in \\cX$ and $c( c^{-1}(u) ) = u$ for any $u \\in \\cU$. For any $u, v\\in\\cU$ and any $\\lambda\\in[0,1]$, if  $(1-\\lambda) u + \\lambda v \\in \\cU$, we say $\\mathcal{U}$ is convex. We denote the diameter of $\\cU$ as $D_{\\cU}:= \\sup_{u, v\\in \\cU}\\pnorm{u - v}$.\n","sha256_utf8":"7f348807d991ef6eb361368e117eac6f9045b730781a456e600d934291967400"},{"start":20628,"end":21811,"start_line":278,"end_line":279,"raw_tex":"A function $F: \\cX \\rightarrow\\R \\cup \\{+\\infty\\}$ is $\\ell$-weakly convex ($\\ell$-WC) if for any fixed $y\\in \\cX$, $F_{\\ell}(x,y) := F(x) + \\frac{\\ell}{2} \\, \\sqnorm{x- y}$ is convex in $x \\in \\cX$.\nIf $x\\notin \\cX$, we assign $F(x) = +\\infty$. The (Fr\\'echet) sub-differential of $F$ at $x\\in \\cX$ is $\\partial F(x) := \\left\\{g \\in \\mathbb{R}^d \\mid  F(y) \\geq F(x)+\\langle g, y-x\\rangle+o(\\|y-x\\|), \\forall y \\in \\R^d  \\right\\}.$ The elements $g \\in \\partial F(x)$ are called sub-gradients of $F$ at $x$, see \\cite{davis2019proximally} for alternative equivalent definitions of the sub-differential set for $\\ell$-WC functions. A differentiable function $F: \\cX \\rightarrow \\R$ is $L$-smooth on $\\cX \\subset \\R^d$ if its gradient is $L$-Lipschitz continuous on the set $\\cX$, i.e., it holds $\\norm{\\nabla F(x) - \\nabla F(y)} \\leq L \\norm{x - y}$ for all $x, y\\in \\cX$. For a convex set $\\cX \\subset \\R^d$, the projection of a point $y\\in \\R^d$ onto $\\cX$ is  $\\Pi_{\\cX}(y) := \\argmin_{x\\in\\cX}\\norm{y - x}$. We denote $\\delta_{\\cX}$ as the indicator function of a set $\\cX \\subset \\R^d$ and define $\\delta_{\\cX}(x)  = 0$ if $x\\in \\cX$ and $\\delta_{\\cX}(x)  = +\\infty$ otherwise. \n","sha256_utf8":"d1a41cac097b9027d5f1fb09c6a5d2e341ab9389635e4de3c2d925b20591ac1a"},{"start":22294,"end":22571,"start_line":282,"end_line":282,"raw_tex":"    We define by $\\cX^* \\subset \\cX$ the set of optimal points of $\\min_{x\\in\\cX} F(x)$ and by $F^*$ its optimal value. A point $\\bar{x} \\in \\cX$ is called a stationary point of a weakly convex function $F: \\cX \\rightarrow \\R$ if $0 \\in \\partial (F +  \\delta_{\\cX})(\\bar{x})$.\n","sha256_utf8":"9e6f3ebacf6a3f47e363863697af64aeef9563c5ead2b1ef8801a8b709cceac9"},{"start":24101,"end":24208,"start_line":300,"end_line":303,"raw_tex":"\\begin{equation}\n\t\t\\label{problem:original_composition}\n\t\t\\min_{x\\in\\cX}~F(x) := H(c(x)) . \n\t\\end{equation}","sha256_utf8":"821b19dd4efac2c09c98d350313b4deeb9822a8eef82922abe7602af99ed56f3"},{"start":24255,"end":25696,"start_line":305,"end_line":324,"raw_tex":"\\begin{definition}\nThe  above problem is called {hidden convex} with modulus $\\mu_c>0, \\mu_H\\geq 0$ (or function $F$ is hidden convex on $\\cX$) if its components satisfy the following underlying conditions. \n\t\\begin{enumerate}%[ref=\\theenumi{}~(\\alph*)]\n\t\t%\\item The set $\\cX \\subset \\R^d $ is closed and convex. \\label{ass:convex_cX}\n\t\t\\myitem{C.1}\\label{C1} The domain $\\mathcal{U}=c(\\mathcal{X})$ is convex, the function $H: \\cU \\rightarrow \\R$ satisfies that for all $u, v \\in \\cU$ and $ \\lambda \\in [0,1]$, \n  \\begin{equation}\\label{assumption:H_conv}\n  \tH( (1-\\lambda)u + \\lambda v ) \\leq  (1- \\lambda) H(u) +  \\lambda  H(v)  -   \\fr{(1-\\lambda)\\lambda \\mu_H}{2} \\sqpnorm{ u - v }, \n  \\end{equation}\n  and \\eqref{problem:reformulation} admits a solution $u^* \\in \\cU$.  %\\label{ass:convex_H} \n  \n\t\t\\myitem{C.2} The map $c: \\cX \\rightarrow \\cU$ is invertible. There exists $\\mu_c>0$ such that  for all $x, y \\in \\cX$ it holds  \\label{C2}\n\t\t\\begin{equation}\\label{assumption:g_inv}\n\t\t\t\\pnorm{c(x) - c(y) } \\geq \\mu_c \\pnorm{x - y }.\n\t\t\\end{equation}\n\t\t%\\item The function $F: \\cX \\rightarrow \\R$ is differentiable on $\\cX$ and its gradient $\\nabla F(x)$ is $L$-Lipschitz continuous. \n\t\\end{enumerate}\n\t%\\IF{Check if A.2. is connected to metric subregularity condition from the works of Russel Luke. }\n\t%\\end{assumption}\nIn particular, if $\\mu_H>0$, we say the above problem is {$(\\mu_c,\\mu_H)$-hidden strongly convex}.  \n\\end{definition}","sha256_utf8":"bf53394b4ca8bc2f02aa50e73c2d1649a36c582db4fad5b395056b42f3ecad3e"},{"start":70448,"end":71005,"start_line":795,"end_line":801,"raw_tex":"    \\begin{enumerate}\n    \\myitem{A.1'}\\label{A1'} The function $F: \\cX \\rightarrow \\R$ is differentiable on a closed, convex set $\\cX$ and its gradient $\\nabla F(x)$ is $L$-Lipschitz continuous. \n    \n    \\myitem{A.2'}\\label{A2'} We have access to an unbiased stochastic gradient oracle with bounded variance $\\sigma > 0$, i.e. for any $x\\in \\cX$: \n\t$\\Exp{\\nabla f(x,\\xi)} = \\nabla F(x)$, and \n\t$$\\Exp{\\sqnorm{\\nabla f(x,\\xi) - \\nabla F(x)}} \\leq \\sigma^2, $$where expectations are with respect to the random variable $\\xi \\sim \\cD$.  \n    \\end{enumerate}\n","sha256_utf8":"65ee038a68dd9ce403ab7bc818e10e9bcd3b0c35af9a2da12a9ed7ccbba9afe3"},{"start":80394,"end":80627,"start_line":909,"end_line":914,"raw_tex":"\\begin{align}\n\t%\\boxed{ \n\t\tx^{t+1} = \\Pi_{\\mathcal{X}}(x^t-\\stepsize \\, g^t) , \\qquad  \\label{eq:PSGDM_x_upd}\n\t\tg^{t+1} = (1-\\momentum) \\, g^{t} + \\momentum \\, \\nabla f(x^{t+1}, \\xi^{t+1}) . \n\t\t%} %\\label{eq:PSGDM_g_upd} \n\\end{align}","sha256_utf8":"d3ddbda259be9ccf149c99048286074364ee097b958fa4714d80e621672a2c3a"},{"start":80862,"end":81095,"start_line":918,"end_line":921,"raw_tex":"Let $x^* \\in \\cX^*$, for any $x^t \\in \\cX$, we define the Lyapunov function\n\\begin{equation}\\label{eq:Lyapunov_HB}\n\t\\Lambda_t^{HB} := \\Exp{ F(x^t) - F(x^*) + \\frac{\\stepsize}{\\momentum} \\sqnorm{g^t - \\nabla F(x^t)} } .\n\\end{equation}","sha256_utf8":"83ca4a545a56f2b1b11f767b273b0e49e0e168f06ae44d146d39cebcdbdbcfc7"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{thm:SGDM}\n\tLet \\ref{C1}, \\ref{C2}, \\ref{A1'}, \\ref{A2'} hold with $\\mu_H = 0$, and the set $\\cU$ be bounded by a diameter $D_{\\cU}$. Then for any $\\stepsize \\leq \\frac{\\momentum}{4 L}$, $\\momentum \\in (0, 1]$, and $\\alpha \\leq  \\frac{\\momentum}{2} $, we have for any $T \\geq 0$\n\t$$\n\t\\Lambda_{T}^{\\text{HB}} \\leq (1 -  \\alpha )^T \\Lambda_0^{\\text{HB}}  + \\fr{ \\alpha D_{\\cU}^2}{ \\mu_c^2   \\stepsize }   +  \\fr{\\momentum \\stepsize  \\sigma^2 }{  \\alpha  } ,\n\t$$\n\twhere $\\Lambda_t^{\\text{HB}}$ is given by \\eqref{eq:Lyapunov_HB}.\n\tFix $\\varepsilon > 0$, and set the parameters of algorithm \\eqref{eq:PSGDM_x_upd} as \n\t$$\n\t\\stepsize = \\frac{\\momentum}{4 L}, \\quad \\beta = \\min\\left\\{1, \\frac{\\mu_c^2  }{9 D_{\\cU}^2 \\sigma^2} \\varepsilon^2 \\right\\}  .\n\t$$\n\tThen the scheme \\eqref{eq:PSGDM_x_upd} returns a point $x^T\\in\\cX$ with $\\Exp{F(x^{T}) - F(x^*) } \\leq \\varepsilon$ when\n\t\\begin{eqnarray} \\notag\n\t  %\\boxed{ \n\t  \tT = \\wt \\cO\\rb{  \\frac{L D_{\\cU}^2 }{\\mu_c^2  } \\frac{1}{\\varepsilon} + \\frac{ L D_{\\cU}^4 \\sigma^2 }{\\mu_c^4  } \\frac{1}{\\varepsilon^3}  } . \n\t  \t%}\n\t\\end{eqnarray}\n\t%iterations.\n\\end{theorem}","units":[{"start":86200,"end":87329,"start_line":985,"end_line":1002,"raw_tex":"\\begin{theorem}\\label{thm:SGDM}\n\tLet \\ref{C1}, \\ref{C2}, \\ref{A1'}, \\ref{A2'} hold with $\\mu_H = 0$, and the set $\\cU$ be bounded by a diameter $D_{\\cU}$. Then for any $\\stepsize \\leq \\frac{\\momentum}{4 L}$, $\\momentum \\in (0, 1]$, and $\\alpha \\leq  \\frac{\\momentum}{2} $, we have for any $T \\geq 0$\n\t$$\n\t\\Lambda_{T}^{\\text{HB}} \\leq (1 -  \\alpha )^T \\Lambda_0^{\\text{HB}}  + \\fr{ \\alpha D_{\\cU}^2}{ \\mu_c^2   \\stepsize }   +  \\fr{\\momentum \\stepsize  \\sigma^2 }{  \\alpha  } ,\n\t$$\n\twhere $\\Lambda_t^{\\text{HB}}$ is given by \\eqref{eq:Lyapunov_HB}.\n\tFix $\\varepsilon > 0$, and set the parameters of algorithm \\eqref{eq:PSGDM_x_upd} as \n\t$$\n\t\\stepsize = \\frac{\\momentum}{4 L}, \\quad \\beta = \\min\\left\\{1, \\frac{\\mu_c^2  }{9 D_{\\cU}^2 \\sigma^2} \\varepsilon^2 \\right\\}  .\n\t$$\n\tThen the scheme \\eqref{eq:PSGDM_x_upd} returns a point $x^T\\in\\cX$ with $\\Exp{F(x^{T}) - F(x^*) } \\leq \\varepsilon$ when\n\t\\begin{eqnarray} \\notag\n\t  %\\boxed{ \n\t  \tT = \\wt \\cO\\rb{  \\frac{L D_{\\cU}^2 }{\\mu_c^2  } \\frac{1}{\\varepsilon} + \\frac{ L D_{\\cU}^4 \\sigma^2 }{\\mu_c^4  } \\frac{1}{\\varepsilon^3}  } . \n\t  \t%}\n\t\\end{eqnarray}\n\t%iterations.\n\\end{theorem}","sha256_utf8":"64adb5812183a15dca345b826c64c205d89de864ecd9065a120d47cfc34420ee"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{proposition}\\label{prop:HC}\n\t\tLet $F(\\cdot)$ be hidden convex with $\\mu_c>0, \\mu_H\\geq0$. For any $\\alpha \\in [0,1]$, $x^* \\in \\cX^*$ and $x\\in \\cX$ , define $x_{\\alpha} := c^{-1} \\rb{(1-\\alpha) c(x) + \\alpha c(x^*)}$. Then\n\t\t\\begin{eqnarray}\\label{prop:F_x_alpha_strong_conv}\n\t\t\tF(x_{\\alpha}) \\leq  (1- \\alpha) F(x) + \\alpha  F(x^*)  -   \\fr{(1-\\alpha) \\alpha  \\mu_H}{2}  \\sqpnorm{c(x) - c(x^*) } ,\n\t\t\\end{eqnarray}\t\\begin{eqnarray}\\label{prop:x_alpha_x_bound}\n\t\t\t\\pnorm{ x_{\\alpha}-x } \\leq \\fr{\\alpha }{\\mu_c} \\pnorm{c(x) - c(x^*) } .\n\t\t\\end{eqnarray}\n\t\t\n\t\\end{proposition}\n\n\\begin{proof}\n\t\t%\\textbf{Part I: \\eqref{prop:F_x_alpha_strong_conv}.} \n\t\tBy the (strong) convexity of $H(\\cdot)$ and the convexity of $\\cU$, we have\n\t\t\\begin{eqnarray*}\n\t\t\tF(x_{\\alpha}) &=& F( c^{-1} \\rb{(1-\\alpha) c(x) + \\alpha c(x^*)} ) \\\\\n\t\t\t& = & H( (1-\\alpha) c(x) + \\alpha c(x^*) ) \\\\ \n\t\t\t& \\leq  &  (1-\\alpha) H( c(x) ) + \\alpha H( c(x^*) ) -   \\fr{(1-\\alpha) \\alpha  \\mu_H}{2} \\sqpnorm{c(x) - c(x^*) }\\\\\n\t\t\t& =  &  (1- \\alpha) F(x)  +  \\alpha F(x^*) -  \\fr{(1-\\alpha)  \\alpha   \\mu_H}{2} \\sqpnorm{c(x) - c(x^*) }.\n\t\t\\end{eqnarray*}\n  where the inequality uses the fact that $\\cU$ is a convex set and that $(1-\\alpha) c(x) + \\alpha c(x^*) \\in \\cU$ for any $x \\in \\cX$. By definition of $x_{\\alpha}$ and \\eqref{assumption:g_inv}, we derive\n\t\t$$\n\t\t\\pnorm{ x_{\\alpha}-x } = \\pnorm{ c^{-1} \\rb{(1-\\alpha) c(x) + \\alpha c(x^*)} - c^{-1}(c(x)) } \\leq \\fr{1}{\\mu_c} \\pnorm{\\alpha(c(x) - c(x^*)) } .\n\t\t$$\n\t\t\\end{proof}\n\n\\begin{lemma}[Lemma 2 in \\cite{fatkhullin2023momentum}]\\label{le:key_HB_recursion}\n\t\tLet $\\momentum \\in (0,1]$ and $g^t$ be updated via \\eqref{eq:PSGDM_x_upd}. Then\n\t\t\\begin{eqnarray}\\notag %\\label{eq:key_HB_recursion}\n\t\t\t\\Exp{ \\sqnorm{g^{t+1} - \\nabla F(x^{t+1})} }  \\leq (1 - \\momentum) \\Exp{ \\sqnorm{  g^{t} - \\nabla F(x^{t}) } }  + \\fr{3 L^2}{\\momentum} \\Exp{ \\sqnorm{  x^{t} - x^{t+1}  } } + \\momentum^2 \\sigma^2 .\n\t\t\\end{eqnarray}\n\t\\end{lemma}\n\n\\begin{theorem}\\label{le:descent_HB_HC}\n\tSuppose that \\ref{C1}, \\ref{C2}, \\ref{A1'}, \\ref{A2'} hold with $\\mu_H\\geq0$, and the step-size in \\eqref{eq:PSGDM_x_upd} satisfies $\\stepsize \\leq 1 / L$. For any $\\alpha \\in [0, 1] $, it holds that\n\t\\begin{eqnarray}\\label{eq:descent_HC}\n\t\tF(x^{t+1})  &\\leq&  (1- \\alpha) F(x^t) +  \\alpha F(x^*)  + \\rb{  \\fr{ \\alpha^{2} }{ \\mu_c^2 \\stepsize } -  \\fr{(1-\\alpha) \\alpha  \\mu_H}{2} } \\sqnorm{ c(x^t) - c(x^*) },  \\notag \\\\\n\t\t&& \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } + \\frac{\\stepsize}{2} \\sqnorm{g^t - \\nabla F(x^t) } .\n\t\\end{eqnarray}\n\\end{theorem}\n\n\\begin{proof}\n\tBy the update rule of $x^{t+1}$ and following the standard descent inequality (cf. \\cref{le:bregman_properties_optimality}), we have for any $z \\in \\cX$ that\n\t\\begin{eqnarray}\\label{eq:projection_upd}\n\t\t\\langle g^t , x^{t+1} - z \\rangle  + \\frac{1}{2\\stepsize} \\sqnorm{ x^{t+1} - x^t} \\leq \\frac{1}{2\\stepsize} \\sqnorm{ z - x^t } -\\frac{1}{2\\stepsize} \\sqnorm{ z - x^{t+1} }  .\n\t\\end{eqnarray}\n\t\n\tBy the smoothness of $F(\\cdot)$, we derive\n\t\\begin{eqnarray}\n\t\tF(x^{t+1}) &\\leq& F(x^t) + \\langle \\nabla F(x^t), x^{t+1}- x^t \\rangle + \\fr{L}{2} \\sqnorm{x^{t+1}-x^t } \\notag \\\\\n\t\t&=& F(x^t) + \\langle g^t, x^{t+1}- x^t \\rangle + \\fr{1}{2 \\stepsize } \\sqnorm{x^{t+1}-x^t } \\notag \\\\\n\t\t&& \\qquad  + \\langle \\nabla F(x^t) - g^t , x^{t+1}- x^t \\rangle - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } \\notag \\\\\n\t\t&\\overset{(i)}{\\leq}&  F(x^t) + \\langle g^t, z - x^t \\rangle + \\fr{1}{2 \\stepsize } \\sqnorm{z - x^t } - \\fr{1}{2 \\stepsize } \\sqnorm{z - x^{t+1} }  \\notag \\\\\n\t\t&& \\qquad  + \\langle \\nabla F(x^t) - g^t , x^{t+1}- x^t \\rangle - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t }   \\notag \\\\\n\t\t& { = } & F(x^t) + \\langle \\nabla F(x^t), z - x^t \\rangle + \\fr{1}{2 \\stepsize } \\sqnorm{ z - x^t } - \\fr{1}{2 \\stepsize } \\sqnorm{ z - x^{t+1} }  \\notag \\\\  \n\t\t&& \\qquad + \\langle \\nabla F(x^t) - g^t , x^{t+1}- z \\rangle - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } \\notag  \\\\\n\t\t& \\overset{(ii)}{ \\leq } & F(x^t) + \\langle \\nabla F(x^t), z - x^t \\rangle + \\fr{1}{2 \\stepsize } \\sqnorm{z - x^t }   + \\frac{\\stepsize}{2} \\sqnorm{g^t - \\nabla F(x^t) }  \\notag \\\\\n\t\t&& \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } \\notag  \\\\\n\t\t&\\overset{(iii)}{\\leq}& F(z) + \\fr{L}{2}\\sqnorm{ z - x^t } + \\fr{1}{2 \\stepsize } \\sqnorm{z - x^t }  + \\frac{\\stepsize}{2} \\sqnorm{g^t - \\nabla F(x^t) }  \\notag \\\\\n\t\t&& \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } \\notag  ,\n\t\\end{eqnarray}\n\twhere $(i)$ follows from \\eqref{eq:projection_upd}, $(ii)$ holds by Young's inequality, i.e., $\\langle a, b\\rangle \\leq \\frac{\\stepsize}{2}\\sqnorm{a} + \\frac{1}{2 \\stepsize} \\sqnorm{b} $ with $a = \\nabla F(x^t) - g^t$, $b = x^{t+1} - z $, $(iii)$ holds by the smoothness of $F(\\cdot)$, i.e., $-\\frac{L}{2}\\sqnorm{z - x^t } \\leq F(x^t) - F(z) - \\langle \\nabla F(x^t) , z - x^t \\rangle $. \n\t\n\tWe are now ready to utilize the properties of hidden convex functions to bound $F(z)$ and $\\sqnorm{z - x^t}$ for some specific choice of $z \\in \\cX$. We select $z := x_{\\alpha}^t = c^{-1}((1-{\\alpha})c(x^t)+ {\\alpha} c(x^*)) \\in \\cX$, for some $\\alpha \\in [0,1]$, and $x^* \\in \\cX^* $. By \\Cref{prop:HC}, we have for $\\mu_H \\geq 0$, \n\t$$\n\tF(z) \\leq (1- \\alpha) F(x^t) +  \\alpha F(x^*) -    \\fr{(1-\\alpha) \\alpha  \\mu_H}{2} \\sqnorm{ c(x^t) - c(x^*) } ,\n\t$$\n and\n\t$$\n\t\\sqnorm{z - x^t } \\leq \\fr{ \\alpha^{2} }{ \\mu_c^2  } \\sqnorm{ c(x^t) - c(x^*) }.\n\t$$\n\tCombining the three inequalities above  and utilizing the assumption $\\stepsize \\leq 1/L$, we complete the proof.\n\t%\\begin{eqnarray}F(x^{t+1})  &\\leq&  (1- \\alpha) F(x^t) +  \\alpha F(x^*)  + \\rb{  \\fr{ \\alpha^{2} }{ \\mu_c^2 \\stepsize } -   \\fr{ (1-\\alpha)\\alpha   \\mu_H}{2} } \\sqnorm{ c(x^t) - c(x^*) }  \\notag \\\\ && \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } + \\frac{\\stepsize}{2} \\sqnorm{g^t - \\nabla F(x^t) } . \\end{eqnarray}\n\\end{proof}\n\n\\begin{proof} \n\t%\\textbf{Step I. Descent-like lemma for hidden convex objectives.} We start the proof by using our descent like Lemma~\\ref{le:descent_HB_HC}, which guarantees that for any $\\stepsize \\leq 1/L$\n\tBy \\Cref{le:descent_HB_HC}, subtracting $F(x^*)$ from both sides of  \\eqref{eq:descent_HC}, setting $\\mu_H=0$, and taking the expectation, we have for any $\\stepsize \\leq 1/L$ that\n\t\\begin{eqnarray*}\n\t\t\\Exp{F(x^{t+1}) - F(x^*)}  &\\leq&  (1-  \\alpha) \\Exp{ F(x^t)  - F(x^*) }  +  \\fr{ \\alpha^{2} }{ \\mu_c^2 \\stepsize } \\Exp{ \\sqnorm{ c(x^t) - c(x^*) } } \\notag \\\\\n\t\t&& \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\Exp{\\sqnorm{x^{t+1}-x^t }} + \\frac{\\stepsize}{2} \\Exp{\\sqnorm{g^t - \\nabla F(x^t) }} \\\\\n\t\t&\\leq&  (1-  \\alpha) \\Exp{ F(x^t)  - F(x^*) }  +  \\fr{ \\alpha^{2} D_{\\cU}^2  }{ \\mu_c^2 \\stepsize }  \\notag \\\\\n\t\t&& \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\Exp{\\sqnorm{x^{t+1}-x^t }} + \\frac{\\stepsize}{2} \\Exp{\\sqnorm{g^t - \\nabla F(x^t) }} ,\n\t\\end{eqnarray*}\n\twhere the second inequality uses boundedness of  $\\cU$.\n\t%Define $\\delta_t := \\Exp{ F(x^t) - F(x^*) }$, $V_t : = \\Exp{\\sqnorm{g^t - \\nabla F(x^t)}}$ , $R_t := \\Exp{\\sqnorm{x^{t+1} - x^t }}$. Then taking expectation, we have\n\t%\\begin{eqnarray}\\label{eq:descent_delta_conv} \\delta_{t+1} \\leq (1-  \\alpha) \\delta_t + \\fr{ \\alpha^{2} D_{\\cU}^2 }{ \\mu_c^2 \\stepsize }   - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } R_t + \\frac{\\stepsize}{2} V_t .\\end{eqnarray}\n\t\n\t%\\textbf{Step II. Controlling the error term of momentum $V_t$.} To control the error term in the above inequality, we use a recursive bound given by Lemma~\\ref{le:key_HB_recursion}\n%\t\\begin{eqnarray}\\label{eq:HB_recursion} V_{t+1}  \\leq (1 - \\momentum) V_t  + \\fr{3 L^2}{\\momentum} R_t + \\momentum^2 \\sigma^2 . \\end{eqnarray}\n\nSumming up the inequality  above with a $\\frac{\\stepsize}{\\momentum}$ multiple of the result of \\Cref{le:key_HB_recursion}, we recognize the Lyapunov function $\\Lambda_t^{\\text{HB}} $ defined in \\eqref{eq:Lyapunov_HB}, and derive\n\t\\begin{eqnarray*}\n\t\t\\Lambda_{t+1}^{\\text{HB}} &\\leq& \\Lambda_t^{\\text{HB}} -  \\alpha \\Exp{ F(x^t)  - F(x^*) }   - \\frac{\\stepsize}{2} \\Exp{\\sqnorm{g^t - \\nabla F(x^t)}} \\\\\n\t\t&&\\qquad  + \\fr{ \\alpha^{2} D_{\\cU}^2 }{ \\mu_c^2 \\stepsize }   - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} - \\frac{3 L^2 \\stepsize }{\\momentum^2} } \\Exp{\\sqnorm{x^{t+1}-x^t }} + \\momentum \\stepsize \\sigma^2 \\\\\n\t\t&\\leq& (1 -  \\alpha) \\Lambda_t^{\\text{HB}} + \\fr{ \\alpha^{2} D_{\\cU}^2 }{ \\mu_c^2 \\stepsize }  + \\momentum \\stepsize \\sigma^2 ,\n\t\\end{eqnarray*}\n\twhere the last step holds for $ \\alpha \\leq \\momentum / 2$ and $\\stepsize \\leq \\frac{\\momentum}{4 L}$. Unrolling the recursion from $t = 0$ to $t = T-1$ and choosing $\\stepsize = \\fr{\\momentum}{ 4 L }$, we obtain\n\t\\begin{eqnarray}\n\t\t\\Lambda_{T}^{\\text{HB}} &\\leq& (1 -  \\alpha )^T \\Lambda_0^{\\text{HB}}  + \\fr{ \\alpha D_{\\cU}^2}{ \\mu_c^2   \\stepsize }   +  \\fr{\\momentum \\stepsize  \\sigma^2 }{  \\alpha  } \\notag \\\\\n\t\t&\\leq& (1 -  \\alpha )^T \\Lambda_0^{\\text{HB}}  + \\fr{ 4 L  D_{\\cU}^2  }{ \\mu_c^2  } \\frac{\\alpha}{\\momentum}  +  \\fr{ \\sigma^2 }{ 4 L   } \\frac{\\momentum^2}{\\alpha} \\leq \\varepsilon  \\notag ,\n\t\\end{eqnarray}\n\twhere the last inequality holds by setting $\\alpha =  \\min\\left\\{ \\frac{\\momentum}{2}, \\frac{3 \\mu_c^2   }{2 L D_{\\cU}^2} \\momentum \\varepsilon , \\frac{\\sigma \\mu_c }{4 L D_{\\cU}} \\beta^{\\frac{3}{2}} \\right\\}$ , $\\beta = \\min\\left\\{1, \\frac{\\mu_c^2  }{9 D_{\\cU}^2 \\sigma^2} \\varepsilon^2 \\right\\} $, and the number of iterations as \n\t$$\n\tT = \\frac{1}{ \\alpha} \\log\\rb{\\frac{3 \\Lambda_0^{\\text{HB}}}{\\varepsilon} } = \\wt \\cO\\rb{ \\frac{L D_{\\cU}^2 }{\\mu_c^2  } \\frac{1}{\\varepsilon}  + \\frac{ L D_{\\cU}^4 \\sigma^2 }{\\mu_c^4 } \\frac{1}{\\varepsilon^3}  } .\n\t$$\n\\end{proof}\n\n\\begin{customproof}{\\Cref{le:key_HB_recursion}}\n\t\tUsing the update rule of $g^{t+1}$ and the unbiasedness of stochastic gradients, we have\n  $$\n\t\t\\begin{aligned}\n\t\t\t\\mathbb{E}&\\left[ \\sqnorm{g^{t+1} - \\nabla F(x^{t+1})} \\right] \n\t\t\t =  \\Exp{ \\sqnorm{(1 - \\momentum) g^{t} + \\momentum \\nabla f(x^{t+1}, \\xi^{t+1}) - \\nabla F(x^{t+1})}  } \\notag \\\\\n\t\t\t& =  (1-\\momentum)^2 \\Exp{ \\sqnorm{  g^{t} - \\nabla F(x^{t+1}) } }  +  \\momentum^2 \\Exp{\\sqnorm{\\nabla f(x^{t+1}, \\xi^{t+1}) - \\nabla F(x^{t+1}) }} \\notag \\\\\n\t\t\t& \\leq   (1 - \\momentum)^2 \\rb{ 1 + \\nfr{\\momentum}{2}} \\Exp{ \\sqnorm{  g^{t} - \\nabla F(x^{t}) } }  + \\rb{1+\\nfr{2}{\\momentum} } \\Exp{ \\sqnorm{  \\nabla F(x^{t}) - \\nabla F(x^{t+1}) }  } + \\momentum^2 \\sigma^2  \\notag \\\\\n\t\t\t& \\leq   (1 - \\momentum) \\Exp{ \\sqnorm{  g^{t} - \\nabla F(x^{t}) } }  + \\fr{3 L^2}{\\momentum} \\Exp{ \\sqnorm{  x^{t} - x^{t+1}  } } \n   + \\momentum^2 \\sigma^2  \\notag ,\n\t\t\\end{aligned}\n  $$\n\t\twhere the first inequality uses Young's inequality and the bound of the variance of stochastic gradients, and the last step uses the Lipschitz continuity of the gradient and the fact that $(1 - \\momentum) \\rb{ 1 + \\nfr{\\momentum}{2}} \\leq 1 $ for all $\\momentum \\in (0,1]$.\n\t\\end{customproof}\n\n\\begin{lemma}\\label{le:bregman_properties}\n\t\t\tLet $\\phi(\\cdot)$ be convex and for any $\\stepsize>0$, $x\\in \\cX$, define $$x^{+} := \\argmin_{y\\in \\cX} \\left\\{ \\phi(y) + \\fr{1}{2 \\stepsize} \\sqnorm{y - x} \\right\\}. $$ \n        Then\n\t\t\t$$\n\t\t\t\\phi(y) + \\fr{1}{2 \\stepsize} \\sqnorm{y - x} \\geq \\phi(x^{+}) + \\fr{1}{2 \\stepsize} \\sqnorm{x^{+} - x }  + \\fr{1}{2\\stepsize} \\sqnorm{ y  - x^{+} }  \\quad \\text{for all } y \\in \\cX. \n\t\t\t$$\\label{le:bregman_properties_optimality}\n\t\\end{lemma}","units":[{"start":52224,"end":52807,"start_line":554,"end_line":562,"raw_tex":"\\begin{proposition}\\label{prop:HC}\n\t\tLet $F(\\cdot)$ be hidden convex with $\\mu_c>0, \\mu_H\\geq0$. For any $\\alpha \\in [0,1]$, $x^* \\in \\cX^*$ and $x\\in \\cX$ , define $x_{\\alpha} := c^{-1} \\rb{(1-\\alpha) c(x) + \\alpha c(x^*)}$. Then\n\t\t\\begin{eqnarray}\\label{prop:F_x_alpha_strong_conv}\n\t\t\tF(x_{\\alpha}) \\leq  (1- \\alpha) F(x) + \\alpha  F(x^*)  -   \\fr{(1-\\alpha) \\alpha  \\mu_H}{2}  \\sqpnorm{c(x) - c(x^*) } ,\n\t\t\\end{eqnarray}\t\\begin{eqnarray}\\label{prop:x_alpha_x_bound}\n\t\t\t\\pnorm{ x_{\\alpha}-x } \\leq \\fr{\\alpha }{\\mu_c} \\pnorm{c(x) - c(x^*) } .\n\t\t\\end{eqnarray}\n\t\t\n\t\\end{proposition}","sha256_utf8":"1ded5b6d9b4420e7b2d65a84b7408522943cb31fa83845380452c2f1de0e5ce4"},{"start":52811,"end":53728,"start_line":564,"end_line":577,"raw_tex":"\\begin{proof}\n\t\t%\\textbf{Part I: \\eqref{prop:F_x_alpha_strong_conv}.} \n\t\tBy the (strong) convexity of $H(\\cdot)$ and the convexity of $\\cU$, we have\n\t\t\\begin{eqnarray*}\n\t\t\tF(x_{\\alpha}) &=& F( c^{-1} \\rb{(1-\\alpha) c(x) + \\alpha c(x^*)} ) \\\\\n\t\t\t& = & H( (1-\\alpha) c(x) + \\alpha c(x^*) ) \\\\ \n\t\t\t& \\leq  &  (1-\\alpha) H( c(x) ) + \\alpha H( c(x^*) ) -   \\fr{(1-\\alpha) \\alpha  \\mu_H}{2} \\sqpnorm{c(x) - c(x^*) }\\\\\n\t\t\t& =  &  (1- \\alpha) F(x)  +  \\alpha F(x^*) -  \\fr{(1-\\alpha)  \\alpha   \\mu_H}{2} \\sqpnorm{c(x) - c(x^*) }.\n\t\t\\end{eqnarray*}\n  where the inequality uses the fact that $\\cU$ is a convex set and that $(1-\\alpha) c(x) + \\alpha c(x^*) \\in \\cU$ for any $x \\in \\cX$. By definition of $x_{\\alpha}$ and \\eqref{assumption:g_inv}, we derive\n\t\t$$\n\t\t\\pnorm{ x_{\\alpha}-x } = \\pnorm{ c^{-1} \\rb{(1-\\alpha) c(x) + \\alpha c(x^*)} - c^{-1}(c(x)) } \\leq \\fr{1}{\\mu_c} \\pnorm{\\alpha(c(x) - c(x^*)) } .\n\t\t$$\n\t\t\\end{proof}","sha256_utf8":"e45d1d9e7cda0ea49f635e1da67acd47b47d8463ac9d1ef879d8f0da843e5813"},{"start":81393,"end":81842,"start_line":925,"end_line":930,"raw_tex":"\\begin{lemma}[Lemma 2 in \\cite{fatkhullin2023momentum}]\\label{le:key_HB_recursion}\n\t\tLet $\\momentum \\in (0,1]$ and $g^t$ be updated via \\eqref{eq:PSGDM_x_upd}. Then\n\t\t\\begin{eqnarray}\\notag %\\label{eq:key_HB_recursion}\n\t\t\t\\Exp{ \\sqnorm{g^{t+1} - \\nabla F(x^{t+1})} }  \\leq (1 - \\momentum) \\Exp{ \\sqnorm{  g^{t} - \\nabla F(x^{t}) } }  + \\fr{3 L^2}{\\momentum} \\Exp{ \\sqnorm{  x^{t} - x^{t+1}  } } + \\momentum^2 \\sigma^2 .\n\t\t\\end{eqnarray}\n\t\\end{lemma}","sha256_utf8":"2907926b42ef9d0ed0686af4e5c9fe4b02baac0646c9ea41a9f7322d5922a14e"},{"start":81974,"end":82596,"start_line":935,"end_line":941,"raw_tex":"\\begin{theorem}\\label{le:descent_HB_HC}\n\tSuppose that \\ref{C1}, \\ref{C2}, \\ref{A1'}, \\ref{A2'} hold with $\\mu_H\\geq0$, and the step-size in \\eqref{eq:PSGDM_x_upd} satisfies $\\stepsize \\leq 1 / L$. For any $\\alpha \\in [0, 1] $, it holds that\n\t\\begin{eqnarray}\\label{eq:descent_HC}\n\t\tF(x^{t+1})  &\\leq&  (1- \\alpha) F(x^t) +  \\alpha F(x^*)  + \\rb{  \\fr{ \\alpha^{2} }{ \\mu_c^2 \\stepsize } -  \\fr{(1-\\alpha) \\alpha  \\mu_H}{2} } \\sqnorm{ c(x^t) - c(x^*) },  \\notag \\\\\n\t\t&& \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } + \\frac{\\stepsize}{2} \\sqnorm{g^t - \\nabla F(x^t) } .\n\t\\end{eqnarray}\n\\end{theorem}","sha256_utf8":"4f25afd564ab980ef4cf8f8a622c74178e4fd1e150f3d9b6f45832faa77e1424"},{"start":82598,"end":85983,"start_line":943,"end_line":975,"raw_tex":"\\begin{proof}\n\tBy the update rule of $x^{t+1}$ and following the standard descent inequality (cf. \\cref{le:bregman_properties_optimality}), we have for any $z \\in \\cX$ that\n\t\\begin{eqnarray}\\label{eq:projection_upd}\n\t\t\\langle g^t , x^{t+1} - z \\rangle  + \\frac{1}{2\\stepsize} \\sqnorm{ x^{t+1} - x^t} \\leq \\frac{1}{2\\stepsize} \\sqnorm{ z - x^t } -\\frac{1}{2\\stepsize} \\sqnorm{ z - x^{t+1} }  .\n\t\\end{eqnarray}\n\t\n\tBy the smoothness of $F(\\cdot)$, we derive\n\t\\begin{eqnarray}\n\t\tF(x^{t+1}) &\\leq& F(x^t) + \\langle \\nabla F(x^t), x^{t+1}- x^t \\rangle + \\fr{L}{2} \\sqnorm{x^{t+1}-x^t } \\notag \\\\\n\t\t&=& F(x^t) + \\langle g^t, x^{t+1}- x^t \\rangle + \\fr{1}{2 \\stepsize } \\sqnorm{x^{t+1}-x^t } \\notag \\\\\n\t\t&& \\qquad  + \\langle \\nabla F(x^t) - g^t , x^{t+1}- x^t \\rangle - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } \\notag \\\\\n\t\t&\\overset{(i)}{\\leq}&  F(x^t) + \\langle g^t, z - x^t \\rangle + \\fr{1}{2 \\stepsize } \\sqnorm{z - x^t } - \\fr{1}{2 \\stepsize } \\sqnorm{z - x^{t+1} }  \\notag \\\\\n\t\t&& \\qquad  + \\langle \\nabla F(x^t) - g^t , x^{t+1}- x^t \\rangle - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t }   \\notag \\\\\n\t\t& { = } & F(x^t) + \\langle \\nabla F(x^t), z - x^t \\rangle + \\fr{1}{2 \\stepsize } \\sqnorm{ z - x^t } - \\fr{1}{2 \\stepsize } \\sqnorm{ z - x^{t+1} }  \\notag \\\\  \n\t\t&& \\qquad + \\langle \\nabla F(x^t) - g^t , x^{t+1}- z \\rangle - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } \\notag  \\\\\n\t\t& \\overset{(ii)}{ \\leq } & F(x^t) + \\langle \\nabla F(x^t), z - x^t \\rangle + \\fr{1}{2 \\stepsize } \\sqnorm{z - x^t }   + \\frac{\\stepsize}{2} \\sqnorm{g^t - \\nabla F(x^t) }  \\notag \\\\\n\t\t&& \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } \\notag  \\\\\n\t\t&\\overset{(iii)}{\\leq}& F(z) + \\fr{L}{2}\\sqnorm{ z - x^t } + \\fr{1}{2 \\stepsize } \\sqnorm{z - x^t }  + \\frac{\\stepsize}{2} \\sqnorm{g^t - \\nabla F(x^t) }  \\notag \\\\\n\t\t&& \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } \\notag  ,\n\t\\end{eqnarray}\n\twhere $(i)$ follows from \\eqref{eq:projection_upd}, $(ii)$ holds by Young's inequality, i.e., $\\langle a, b\\rangle \\leq \\frac{\\stepsize}{2}\\sqnorm{a} + \\frac{1}{2 \\stepsize} \\sqnorm{b} $ with $a = \\nabla F(x^t) - g^t$, $b = x^{t+1} - z $, $(iii)$ holds by the smoothness of $F(\\cdot)$, i.e., $-\\frac{L}{2}\\sqnorm{z - x^t } \\leq F(x^t) - F(z) - \\langle \\nabla F(x^t) , z - x^t \\rangle $. \n\t\n\tWe are now ready to utilize the properties of hidden convex functions to bound $F(z)$ and $\\sqnorm{z - x^t}$ for some specific choice of $z \\in \\cX$. We select $z := x_{\\alpha}^t = c^{-1}((1-{\\alpha})c(x^t)+ {\\alpha} c(x^*)) \\in \\cX$, for some $\\alpha \\in [0,1]$, and $x^* \\in \\cX^* $. By \\Cref{prop:HC}, we have for $\\mu_H \\geq 0$, \n\t$$\n\tF(z) \\leq (1- \\alpha) F(x^t) +  \\alpha F(x^*) -    \\fr{(1-\\alpha) \\alpha  \\mu_H}{2} \\sqnorm{ c(x^t) - c(x^*) } ,\n\t$$\n and\n\t$$\n\t\\sqnorm{z - x^t } \\leq \\fr{ \\alpha^{2} }{ \\mu_c^2  } \\sqnorm{ c(x^t) - c(x^*) }.\n\t$$\n\tCombining the three inequalities above  and utilizing the assumption $\\stepsize \\leq 1/L$, we complete the proof.\n\t%\\begin{eqnarray}F(x^{t+1})  &\\leq&  (1- \\alpha) F(x^t) +  \\alpha F(x^*)  + \\rb{  \\fr{ \\alpha^{2} }{ \\mu_c^2 \\stepsize } -   \\fr{ (1-\\alpha)\\alpha   \\mu_H}{2} } \\sqnorm{ c(x^t) - c(x^*) }  \\notag \\\\ && \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } + \\frac{\\stepsize}{2} \\sqnorm{g^t - \\nabla F(x^t) } . \\end{eqnarray}\n\\end{proof}","sha256_utf8":"b218f4e8d311ee8253e6e47ec230b5d715552362f2fed0f1a86d9ed37002890b"},{"start":87686,"end":91412,"start_line":1008,"end_line":1039,"raw_tex":"\\begin{proof} \n\t%\\textbf{Step I. Descent-like lemma for hidden convex objectives.} We start the proof by using our descent like Lemma~\\ref{le:descent_HB_HC}, which guarantees that for any $\\stepsize \\leq 1/L$\n\tBy \\Cref{le:descent_HB_HC}, subtracting $F(x^*)$ from both sides of  \\eqref{eq:descent_HC}, setting $\\mu_H=0$, and taking the expectation, we have for any $\\stepsize \\leq 1/L$ that\n\t\\begin{eqnarray*}\n\t\t\\Exp{F(x^{t+1}) - F(x^*)}  &\\leq&  (1-  \\alpha) \\Exp{ F(x^t)  - F(x^*) }  +  \\fr{ \\alpha^{2} }{ \\mu_c^2 \\stepsize } \\Exp{ \\sqnorm{ c(x^t) - c(x^*) } } \\notag \\\\\n\t\t&& \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\Exp{\\sqnorm{x^{t+1}-x^t }} + \\frac{\\stepsize}{2} \\Exp{\\sqnorm{g^t - \\nabla F(x^t) }} \\\\\n\t\t&\\leq&  (1-  \\alpha) \\Exp{ F(x^t)  - F(x^*) }  +  \\fr{ \\alpha^{2} D_{\\cU}^2  }{ \\mu_c^2 \\stepsize }  \\notag \\\\\n\t\t&& \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\Exp{\\sqnorm{x^{t+1}-x^t }} + \\frac{\\stepsize}{2} \\Exp{\\sqnorm{g^t - \\nabla F(x^t) }} ,\n\t\\end{eqnarray*}\n\twhere the second inequality uses boundedness of  $\\cU$.\n\t%Define $\\delta_t := \\Exp{ F(x^t) - F(x^*) }$, $V_t : = \\Exp{\\sqnorm{g^t - \\nabla F(x^t)}}$ , $R_t := \\Exp{\\sqnorm{x^{t+1} - x^t }}$. Then taking expectation, we have\n\t%\\begin{eqnarray}\\label{eq:descent_delta_conv} \\delta_{t+1} \\leq (1-  \\alpha) \\delta_t + \\fr{ \\alpha^{2} D_{\\cU}^2 }{ \\mu_c^2 \\stepsize }   - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } R_t + \\frac{\\stepsize}{2} V_t .\\end{eqnarray}\n\t\n\t%\\textbf{Step II. Controlling the error term of momentum $V_t$.} To control the error term in the above inequality, we use a recursive bound given by Lemma~\\ref{le:key_HB_recursion}\n%\t\\begin{eqnarray}\\label{eq:HB_recursion} V_{t+1}  \\leq (1 - \\momentum) V_t  + \\fr{3 L^2}{\\momentum} R_t + \\momentum^2 \\sigma^2 . \\end{eqnarray}\n\nSumming up the inequality  above with a $\\frac{\\stepsize}{\\momentum}$ multiple of the result of \\Cref{le:key_HB_recursion}, we recognize the Lyapunov function $\\Lambda_t^{\\text{HB}} $ defined in \\eqref{eq:Lyapunov_HB}, and derive\n\t\\begin{eqnarray*}\n\t\t\\Lambda_{t+1}^{\\text{HB}} &\\leq& \\Lambda_t^{\\text{HB}} -  \\alpha \\Exp{ F(x^t)  - F(x^*) }   - \\frac{\\stepsize}{2} \\Exp{\\sqnorm{g^t - \\nabla F(x^t)}} \\\\\n\t\t&&\\qquad  + \\fr{ \\alpha^{2} D_{\\cU}^2 }{ \\mu_c^2 \\stepsize }   - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} - \\frac{3 L^2 \\stepsize }{\\momentum^2} } \\Exp{\\sqnorm{x^{t+1}-x^t }} + \\momentum \\stepsize \\sigma^2 \\\\\n\t\t&\\leq& (1 -  \\alpha) \\Lambda_t^{\\text{HB}} + \\fr{ \\alpha^{2} D_{\\cU}^2 }{ \\mu_c^2 \\stepsize }  + \\momentum \\stepsize \\sigma^2 ,\n\t\\end{eqnarray*}\n\twhere the last step holds for $ \\alpha \\leq \\momentum / 2$ and $\\stepsize \\leq \\frac{\\momentum}{4 L}$. Unrolling the recursion from $t = 0$ to $t = T-1$ and choosing $\\stepsize = \\fr{\\momentum}{ 4 L }$, we obtain\n\t\\begin{eqnarray}\n\t\t\\Lambda_{T}^{\\text{HB}} &\\leq& (1 -  \\alpha )^T \\Lambda_0^{\\text{HB}}  + \\fr{ \\alpha D_{\\cU}^2}{ \\mu_c^2   \\stepsize }   +  \\fr{\\momentum \\stepsize  \\sigma^2 }{  \\alpha  } \\notag \\\\\n\t\t&\\leq& (1 -  \\alpha )^T \\Lambda_0^{\\text{HB}}  + \\fr{ 4 L  D_{\\cU}^2  }{ \\mu_c^2  } \\frac{\\alpha}{\\momentum}  +  \\fr{ \\sigma^2 }{ 4 L   } \\frac{\\momentum^2}{\\alpha} \\leq \\varepsilon  \\notag ,\n\t\\end{eqnarray}\n\twhere the last inequality holds by setting $\\alpha =  \\min\\left\\{ \\frac{\\momentum}{2}, \\frac{3 \\mu_c^2   }{2 L D_{\\cU}^2} \\momentum \\varepsilon , \\frac{\\sigma \\mu_c }{4 L D_{\\cU}} \\beta^{\\frac{3}{2}} \\right\\}$ , $\\beta = \\min\\left\\{1, \\frac{\\mu_c^2  }{9 D_{\\cU}^2 \\sigma^2} \\varepsilon^2 \\right\\} $, and the number of iterations as \n\t$$\n\tT = \\frac{1}{ \\alpha} \\log\\rb{\\frac{3 \\Lambda_0^{\\text{HB}}}{\\varepsilon} } = \\wt \\cO\\rb{ \\frac{L D_{\\cU}^2 }{\\mu_c^2  } \\frac{1}{\\varepsilon}  + \\frac{ L D_{\\cU}^4 \\sigma^2 }{\\mu_c^4 } \\frac{1}{\\varepsilon^3}  } .\n\t$$\n\\end{proof}","sha256_utf8":"4ccfbacca27e02bf59bb6f8f588e22e79b384378d69514f2d5e1872382c90ea2"},{"start":103239,"end":104457,"start_line":1171,"end_line":1184,"raw_tex":"\\begin{customproof}{\\Cref{le:key_HB_recursion}}\n\t\tUsing the update rule of $g^{t+1}$ and the unbiasedness of stochastic gradients, we have\n  $$\n\t\t\\begin{aligned}\n\t\t\t\\mathbb{E}&\\left[ \\sqnorm{g^{t+1} - \\nabla F(x^{t+1})} \\right] \n\t\t\t =  \\Exp{ \\sqnorm{(1 - \\momentum) g^{t} + \\momentum \\nabla f(x^{t+1}, \\xi^{t+1}) - \\nabla F(x^{t+1})}  } \\notag \\\\\n\t\t\t& =  (1-\\momentum)^2 \\Exp{ \\sqnorm{  g^{t} - \\nabla F(x^{t+1}) } }  +  \\momentum^2 \\Exp{\\sqnorm{\\nabla f(x^{t+1}, \\xi^{t+1}) - \\nabla F(x^{t+1}) }} \\notag \\\\\n\t\t\t& \\leq   (1 - \\momentum)^2 \\rb{ 1 + \\nfr{\\momentum}{2}} \\Exp{ \\sqnorm{  g^{t} - \\nabla F(x^{t}) } }  + \\rb{1+\\nfr{2}{\\momentum} } \\Exp{ \\sqnorm{  \\nabla F(x^{t}) - \\nabla F(x^{t+1}) }  } + \\momentum^2 \\sigma^2  \\notag \\\\\n\t\t\t& \\leq   (1 - \\momentum) \\Exp{ \\sqnorm{  g^{t} - \\nabla F(x^{t}) } }  + \\fr{3 L^2}{\\momentum} \\Exp{ \\sqnorm{  x^{t} - x^{t+1}  } } \n   + \\momentum^2 \\sigma^2  \\notag ,\n\t\t\\end{aligned}\n  $$\n\t\twhere the first inequality uses Young's inequality and the bound of the variance of stochastic gradients, and the last step uses the Lipschitz continuity of the gradient and the fact that $(1 - \\momentum) \\rb{ 1 + \\nfr{\\momentum}{2}} \\leq 1 $ for all $\\momentum \\in (0,1]$.\n\t\\end{customproof}","sha256_utf8":"db50730580a7e86c3d03048d0f025490aa723c981490e3d83907af0a4f9ee965"},{"start":104547,"end":105027,"start_line":1190,"end_line":1196,"raw_tex":"\\begin{lemma}\\label{le:bregman_properties}\n\t\t\tLet $\\phi(\\cdot)$ be convex and for any $\\stepsize>0$, $x\\in \\cX$, define $$x^{+} := \\argmin_{y\\in \\cX} \\left\\{ \\phi(y) + \\fr{1}{2 \\stepsize} \\sqnorm{y - x} \\right\\}. $$ \n        Then\n\t\t\t$$\n\t\t\t\\phi(y) + \\fr{1}{2 \\stepsize} \\sqnorm{y - x} \\geq \\phi(x^{+}) + \\fr{1}{2 \\stepsize} \\sqnorm{x^{+} - x }  + \\fr{1}{2\\stepsize} \\sqnorm{ y  - x^{+} }  \\quad \\text{for all } y \\in \\cX. \n\t\t\t$$\\label{le:bregman_properties_optimality}\n\t\\end{lemma}","sha256_utf8":"07ae1b5e5af47b3dba956904203ef8b6f1d118e9dec1783bbecb642754cb1a7b"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"投影动量随机梯度的末次迭代复杂度","source_units":[{"start":86200,"end":87329,"start_line":985,"end_line":1002,"raw_tex":"\\begin{theorem}\\label{thm:SGDM}\n\tLet \\ref{C1}, \\ref{C2}, \\ref{A1'}, \\ref{A2'} hold with $\\mu_H = 0$, and the set $\\cU$ be bounded by a diameter $D_{\\cU}$. Then for any $\\stepsize \\leq \\frac{\\momentum}{4 L}$, $\\momentum \\in (0, 1]$, and $\\alpha \\leq  \\frac{\\momentum}{2} $, we have for any $T \\geq 0$\n\t$$\n\t\\Lambda_{T}^{\\text{HB}} \\leq (1 -  \\alpha )^T \\Lambda_0^{\\text{HB}}  + \\fr{ \\alpha D_{\\cU}^2}{ \\mu_c^2   \\stepsize }   +  \\fr{\\momentum \\stepsize  \\sigma^2 }{  \\alpha  } ,\n\t$$\n\twhere $\\Lambda_t^{\\text{HB}}$ is given by \\eqref{eq:Lyapunov_HB}.\n\tFix $\\varepsilon > 0$, and set the parameters of algorithm \\eqref{eq:PSGDM_x_upd} as \n\t$$\n\t\\stepsize = \\frac{\\momentum}{4 L}, \\quad \\beta = \\min\\left\\{1, \\frac{\\mu_c^2  }{9 D_{\\cU}^2 \\sigma^2} \\varepsilon^2 \\right\\}  .\n\t$$\n\tThen the scheme \\eqref{eq:PSGDM_x_upd} returns a point $x^T\\in\\cX$ with $\\Exp{F(x^{T}) - F(x^*) } \\leq \\varepsilon$ when\n\t\\begin{eqnarray} \\notag\n\t  %\\boxed{ \n\t  \tT = \\wt \\cO\\rb{  \\frac{L D_{\\cU}^2 }{\\mu_c^2  } \\frac{1}{\\varepsilon} + \\frac{ L D_{\\cU}^4 \\sigma^2 }{\\mu_c^4  } \\frac{1}{\\varepsilon^3}  } . \n\t  \t%}\n\t\\end{eqnarray}\n\t%iterations.\n\\end{theorem}","sha256_utf8":"64adb5812183a15dca345b826c64c205d89de864ecd9065a120d47cfc34420ee"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"末次迭代复杂度的完整证明","source_units":[{"start":87686,"end":91412,"start_line":1008,"end_line":1039,"raw_tex":"\\begin{proof} \n\t%\\textbf{Step I. Descent-like lemma for hidden convex objectives.} We start the proof by using our descent like Lemma~\\ref{le:descent_HB_HC}, which guarantees that for any $\\stepsize \\leq 1/L$\n\tBy \\Cref{le:descent_HB_HC}, subtracting $F(x^*)$ from both sides of  \\eqref{eq:descent_HC}, setting $\\mu_H=0$, and taking the expectation, we have for any $\\stepsize \\leq 1/L$ that\n\t\\begin{eqnarray*}\n\t\t\\Exp{F(x^{t+1}) - F(x^*)}  &\\leq&  (1-  \\alpha) \\Exp{ F(x^t)  - F(x^*) }  +  \\fr{ \\alpha^{2} }{ \\mu_c^2 \\stepsize } \\Exp{ \\sqnorm{ c(x^t) - c(x^*) } } \\notag \\\\\n\t\t&& \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\Exp{\\sqnorm{x^{t+1}-x^t }} + \\frac{\\stepsize}{2} \\Exp{\\sqnorm{g^t - \\nabla F(x^t) }} \\\\\n\t\t&\\leq&  (1-  \\alpha) \\Exp{ F(x^t)  - F(x^*) }  +  \\fr{ \\alpha^{2} D_{\\cU}^2  }{ \\mu_c^2 \\stepsize }  \\notag \\\\\n\t\t&& \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\Exp{\\sqnorm{x^{t+1}-x^t }} + \\frac{\\stepsize}{2} \\Exp{\\sqnorm{g^t - \\nabla F(x^t) }} ,\n\t\\end{eqnarray*}\n\twhere the second inequality uses boundedness of  $\\cU$.\n\t%Define $\\delta_t := \\Exp{ F(x^t) - F(x^*) }$, $V_t : = \\Exp{\\sqnorm{g^t - \\nabla F(x^t)}}$ , $R_t := \\Exp{\\sqnorm{x^{t+1} - x^t }}$. Then taking expectation, we have\n\t%\\begin{eqnarray}\\label{eq:descent_delta_conv} \\delta_{t+1} \\leq (1-  \\alpha) \\delta_t + \\fr{ \\alpha^{2} D_{\\cU}^2 }{ \\mu_c^2 \\stepsize }   - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } R_t + \\frac{\\stepsize}{2} V_t .\\end{eqnarray}\n\t\n\t%\\textbf{Step II. Controlling the error term of momentum $V_t$.} To control the error term in the above inequality, we use a recursive bound given by Lemma~\\ref{le:key_HB_recursion}\n%\t\\begin{eqnarray}\\label{eq:HB_recursion} V_{t+1}  \\leq (1 - \\momentum) V_t  + \\fr{3 L^2}{\\momentum} R_t + \\momentum^2 \\sigma^2 . \\end{eqnarray}\n\nSumming up the inequality  above with a $\\frac{\\stepsize}{\\momentum}$ multiple of the result of \\Cref{le:key_HB_recursion}, we recognize the Lyapunov function $\\Lambda_t^{\\text{HB}} $ defined in \\eqref{eq:Lyapunov_HB}, and derive\n\t\\begin{eqnarray*}\n\t\t\\Lambda_{t+1}^{\\text{HB}} &\\leq& \\Lambda_t^{\\text{HB}} -  \\alpha \\Exp{ F(x^t)  - F(x^*) }   - \\frac{\\stepsize}{2} \\Exp{\\sqnorm{g^t - \\nabla F(x^t)}} \\\\\n\t\t&&\\qquad  + \\fr{ \\alpha^{2} D_{\\cU}^2 }{ \\mu_c^2 \\stepsize }   - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} - \\frac{3 L^2 \\stepsize }{\\momentum^2} } \\Exp{\\sqnorm{x^{t+1}-x^t }} + \\momentum \\stepsize \\sigma^2 \\\\\n\t\t&\\leq& (1 -  \\alpha) \\Lambda_t^{\\text{HB}} + \\fr{ \\alpha^{2} D_{\\cU}^2 }{ \\mu_c^2 \\stepsize }  + \\momentum \\stepsize \\sigma^2 ,\n\t\\end{eqnarray*}\n\twhere the last step holds for $ \\alpha \\leq \\momentum / 2$ and $\\stepsize \\leq \\frac{\\momentum}{4 L}$. Unrolling the recursion from $t = 0$ to $t = T-1$ and choosing $\\stepsize = \\fr{\\momentum}{ 4 L }$, we obtain\n\t\\begin{eqnarray}\n\t\t\\Lambda_{T}^{\\text{HB}} &\\leq& (1 -  \\alpha )^T \\Lambda_0^{\\text{HB}}  + \\fr{ \\alpha D_{\\cU}^2}{ \\mu_c^2   \\stepsize }   +  \\fr{\\momentum \\stepsize  \\sigma^2 }{  \\alpha  } \\notag \\\\\n\t\t&\\leq& (1 -  \\alpha )^T \\Lambda_0^{\\text{HB}}  + \\fr{ 4 L  D_{\\cU}^2  }{ \\mu_c^2  } \\frac{\\alpha}{\\momentum}  +  \\fr{ \\sigma^2 }{ 4 L   } \\frac{\\momentum^2}{\\alpha} \\leq \\varepsilon  \\notag ,\n\t\\end{eqnarray}\n\twhere the last inequality holds by setting $\\alpha =  \\min\\left\\{ \\frac{\\momentum}{2}, \\frac{3 \\mu_c^2   }{2 L D_{\\cU}^2} \\momentum \\varepsilon , \\frac{\\sigma \\mu_c }{4 L D_{\\cU}} \\beta^{\\frac{3}{2}} \\right\\}$ , $\\beta = \\min\\left\\{1, \\frac{\\mu_c^2  }{9 D_{\\cU}^2 \\sigma^2} \\varepsilon^2 \\right\\} $, and the number of iterations as \n\t$$\n\tT = \\frac{1}{ \\alpha} \\log\\rb{\\frac{3 \\Lambda_0^{\\text{HB}}}{\\varepsilon} } = \\wt \\cO\\rb{ \\frac{L D_{\\cU}^2 }{\\mu_c^2  } \\frac{1}{\\varepsilon}  + \\frac{ L D_{\\cU}^4 \\sigma^2 }{\\mu_c^4 } \\frac{1}{\\varepsilon^3}  } .\n\t$$\n\\end{proof}","sha256_utf8":"4ccfbacca27e02bf59bb6f8f588e22e79b384378d69514f2d5e1872382c90ea2"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"取期望并用直径控制函数差递推","source_units":[{"start":87686,"end":89459,"start_line":1008,"end_line":1023,"raw_tex":"\\begin{proof} \n\t%\\textbf{Step I. Descent-like lemma for hidden convex objectives.} We start the proof by using our descent like Lemma~\\ref{le:descent_HB_HC}, which guarantees that for any $\\stepsize \\leq 1/L$\n\tBy \\Cref{le:descent_HB_HC}, subtracting $F(x^*)$ from both sides of  \\eqref{eq:descent_HC}, setting $\\mu_H=0$, and taking the expectation, we have for any $\\stepsize \\leq 1/L$ that\n\t\\begin{eqnarray*}\n\t\t\\Exp{F(x^{t+1}) - F(x^*)}  &\\leq&  (1-  \\alpha) \\Exp{ F(x^t)  - F(x^*) }  +  \\fr{ \\alpha^{2} }{ \\mu_c^2 \\stepsize } \\Exp{ \\sqnorm{ c(x^t) - c(x^*) } } \\notag \\\\\n\t\t&& \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\Exp{\\sqnorm{x^{t+1}-x^t }} + \\frac{\\stepsize}{2} \\Exp{\\sqnorm{g^t - \\nabla F(x^t) }} \\\\\n\t\t&\\leq&  (1-  \\alpha) \\Exp{ F(x^t)  - F(x^*) }  +  \\fr{ \\alpha^{2} D_{\\cU}^2  }{ \\mu_c^2 \\stepsize }  \\notag \\\\\n\t\t&& \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\Exp{\\sqnorm{x^{t+1}-x^t }} + \\frac{\\stepsize}{2} \\Exp{\\sqnorm{g^t - \\nabla F(x^t) }} ,\n\t\\end{eqnarray*}\n\twhere the second inequality uses boundedness of  $\\cU$.\n\t%Define $\\delta_t := \\Exp{ F(x^t) - F(x^*) }$, $V_t : = \\Exp{\\sqnorm{g^t - \\nabla F(x^t)}}$ , $R_t := \\Exp{\\sqnorm{x^{t+1} - x^t }}$. Then taking expectation, we have\n\t%\\begin{eqnarray}\\label{eq:descent_delta_conv} \\delta_{t+1} \\leq (1-  \\alpha) \\delta_t + \\fr{ \\alpha^{2} D_{\\cU}^2 }{ \\mu_c^2 \\stepsize }   - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } R_t + \\frac{\\stepsize}{2} V_t .\\end{eqnarray}\n\t\n\t%\\textbf{Step II. Controlling the error term of momentum $V_t$.} To control the error term in the above inequality, we use a recursive bound given by Lemma~\\ref{le:key_HB_recursion}\n%\t\\begin{eqnarray}\\label{eq:HB_recursion} V_{t+1}  \\leq (1 - \\momentum) V_t  + \\fr{3 L^2}{\\momentum} R_t + \\momentum^2 \\sigma^2 . \\end{eqnarray}\n\n","sha256_utf8":"73e5052d29dbff7d809fc557d51339dfd3d95fe10569612e3b36a3be5c4d6e76"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"合并动量误差并建立势函数收缩","source_units":[{"start":89459,"end":90431,"start_line":1024,"end_line":1030,"raw_tex":"Summing up the inequality  above with a $\\frac{\\stepsize}{\\momentum}$ multiple of the result of \\Cref{le:key_HB_recursion}, we recognize the Lyapunov function $\\Lambda_t^{\\text{HB}} $ defined in \\eqref{eq:Lyapunov_HB}, and derive\n\t\\begin{eqnarray*}\n\t\t\\Lambda_{t+1}^{\\text{HB}} &\\leq& \\Lambda_t^{\\text{HB}} -  \\alpha \\Exp{ F(x^t)  - F(x^*) }   - \\frac{\\stepsize}{2} \\Exp{\\sqnorm{g^t - \\nabla F(x^t)}} \\\\\n\t\t&&\\qquad  + \\fr{ \\alpha^{2} D_{\\cU}^2 }{ \\mu_c^2 \\stepsize }   - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} - \\frac{3 L^2 \\stepsize }{\\momentum^2} } \\Exp{\\sqnorm{x^{t+1}-x^t }} + \\momentum \\stepsize \\sigma^2 \\\\\n\t\t&\\leq& (1 -  \\alpha) \\Lambda_t^{\\text{HB}} + \\fr{ \\alpha^{2} D_{\\cU}^2 }{ \\mu_c^2 \\stepsize }  + \\momentum \\stepsize \\sigma^2 ,\n\t\\end{eqnarray*}\n\twhere the last step holds for $ \\alpha \\leq \\momentum / 2$ and $\\stepsize \\leq \\frac{\\momentum}{4 L}$. Unrolling the recursion from $t = 0$ to $t = T-1$ and choosing $\\stepsize = \\fr{\\momentum}{ 4 L }$, we obtain\n","sha256_utf8":"e500a4f680d660421ab3a798cfaf1ddc9508b309fc5caa3f80734223fbdfb20d"}]},{"id":"S005","semantic_type":"proof_step","kind":"proof_step","title":"展开收缩递推得到有限步界","source_units":[{"start":90431,"end":90843,"start_line":1031,"end_line":1034,"raw_tex":"\t\\begin{eqnarray}\n\t\t\\Lambda_{T}^{\\text{HB}} &\\leq& (1 -  \\alpha )^T \\Lambda_0^{\\text{HB}}  + \\fr{ \\alpha D_{\\cU}^2}{ \\mu_c^2   \\stepsize }   +  \\fr{\\momentum \\stepsize  \\sigma^2 }{  \\alpha  } \\notag \\\\\n\t\t&\\leq& (1 -  \\alpha )^T \\Lambda_0^{\\text{HB}}  + \\fr{ 4 L  D_{\\cU}^2  }{ \\mu_c^2  } \\frac{\\alpha}{\\momentum}  +  \\fr{ \\sigma^2 }{ 4 L   } \\frac{\\momentum^2}{\\alpha} \\leq \\varepsilon  \\notag ,\n\t\\end{eqnarray}\n","sha256_utf8":"fa3dd8ac0f16a0ec56727a5507b27292e9453b30059bde952f312a6a5e2c3142"}]},{"id":"S006","semantic_type":"proof_step","kind":"proof_step","title":"选择参数并计算样本复杂度","source_units":[{"start":90843,"end":91412,"start_line":1035,"end_line":1039,"raw_tex":"\twhere the last inequality holds by setting $\\alpha =  \\min\\left\\{ \\frac{\\momentum}{2}, \\frac{3 \\mu_c^2   }{2 L D_{\\cU}^2} \\momentum \\varepsilon , \\frac{\\sigma \\mu_c }{4 L D_{\\cU}} \\beta^{\\frac{3}{2}} \\right\\}$ , $\\beta = \\min\\left\\{1, \\frac{\\mu_c^2  }{9 D_{\\cU}^2 \\sigma^2} \\varepsilon^2 \\right\\} $, and the number of iterations as \n\t$$\n\tT = \\frac{1}{ \\alpha} \\log\\rb{\\frac{3 \\Lambda_0^{\\text{HB}}}{\\varepsilon} } = \\wt \\cO\\rb{ \\frac{L D_{\\cU}^2 }{\\mu_c^2  } \\frac{1}{\\varepsilon}  + \\frac{ L D_{\\cU}^4 \\sigma^2 }{\\mu_c^4 } \\frac{1}{\\varepsilon^3}  } .\n\t$$\n\\end{proof}","sha256_utf8":"098bcf76e65ca17bce4d37b227b6480f9a0b0af61f3750e3d5ef6008268872f3"}]},{"id":"S007","semantic_type":"theorem","kind":"theorem","title":"隐藏凸性下的动量下降不等式","source_units":[{"start":81974,"end":82596,"start_line":935,"end_line":941,"raw_tex":"\\begin{theorem}\\label{le:descent_HB_HC}\n\tSuppose that \\ref{C1}, \\ref{C2}, \\ref{A1'}, \\ref{A2'} hold with $\\mu_H\\geq0$, and the step-size in \\eqref{eq:PSGDM_x_upd} satisfies $\\stepsize \\leq 1 / L$. For any $\\alpha \\in [0, 1] $, it holds that\n\t\\begin{eqnarray}\\label{eq:descent_HC}\n\t\tF(x^{t+1})  &\\leq&  (1- \\alpha) F(x^t) +  \\alpha F(x^*)  + \\rb{  \\fr{ \\alpha^{2} }{ \\mu_c^2 \\stepsize } -  \\fr{(1-\\alpha) \\alpha  \\mu_H}{2} } \\sqnorm{ c(x^t) - c(x^*) },  \\notag \\\\\n\t\t&& \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } + \\frac{\\stepsize}{2} \\sqnorm{g^t - \\nabla F(x^t) } .\n\t\\end{eqnarray}\n\\end{theorem}","sha256_utf8":"4f25afd564ab980ef4cf8f8a622c74178e4fd1e150f3d9b6f45832faa77e1424"}]},{"id":"S008","semantic_type":"proof","kind":"proof","title":"动量下降不等式的完整证明","source_units":[{"start":82598,"end":85983,"start_line":943,"end_line":975,"raw_tex":"\\begin{proof}\n\tBy the update rule of $x^{t+1}$ and following the standard descent inequality (cf. \\cref{le:bregman_properties_optimality}), we have for any $z \\in \\cX$ that\n\t\\begin{eqnarray}\\label{eq:projection_upd}\n\t\t\\langle g^t , x^{t+1} - z \\rangle  + \\frac{1}{2\\stepsize} \\sqnorm{ x^{t+1} - x^t} \\leq \\frac{1}{2\\stepsize} \\sqnorm{ z - x^t } -\\frac{1}{2\\stepsize} \\sqnorm{ z - x^{t+1} }  .\n\t\\end{eqnarray}\n\t\n\tBy the smoothness of $F(\\cdot)$, we derive\n\t\\begin{eqnarray}\n\t\tF(x^{t+1}) &\\leq& F(x^t) + \\langle \\nabla F(x^t), x^{t+1}- x^t \\rangle + \\fr{L}{2} \\sqnorm{x^{t+1}-x^t } \\notag \\\\\n\t\t&=& F(x^t) + \\langle g^t, x^{t+1}- x^t \\rangle + \\fr{1}{2 \\stepsize } \\sqnorm{x^{t+1}-x^t } \\notag \\\\\n\t\t&& \\qquad  + \\langle \\nabla F(x^t) - g^t , x^{t+1}- x^t \\rangle - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } \\notag \\\\\n\t\t&\\overset{(i)}{\\leq}&  F(x^t) + \\langle g^t, z - x^t \\rangle + \\fr{1}{2 \\stepsize } \\sqnorm{z - x^t } - \\fr{1}{2 \\stepsize } \\sqnorm{z - x^{t+1} }  \\notag \\\\\n\t\t&& \\qquad  + \\langle \\nabla F(x^t) - g^t , x^{t+1}- x^t \\rangle - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t }   \\notag \\\\\n\t\t& { = } & F(x^t) + \\langle \\nabla F(x^t), z - x^t \\rangle + \\fr{1}{2 \\stepsize } \\sqnorm{ z - x^t } - \\fr{1}{2 \\stepsize } \\sqnorm{ z - x^{t+1} }  \\notag \\\\  \n\t\t&& \\qquad + \\langle \\nabla F(x^t) - g^t , x^{t+1}- z \\rangle - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } \\notag  \\\\\n\t\t& \\overset{(ii)}{ \\leq } & F(x^t) + \\langle \\nabla F(x^t), z - x^t \\rangle + \\fr{1}{2 \\stepsize } \\sqnorm{z - x^t }   + \\frac{\\stepsize}{2} \\sqnorm{g^t - \\nabla F(x^t) }  \\notag \\\\\n\t\t&& \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } \\notag  \\\\\n\t\t&\\overset{(iii)}{\\leq}& F(z) + \\fr{L}{2}\\sqnorm{ z - x^t } + \\fr{1}{2 \\stepsize } \\sqnorm{z - x^t }  + \\frac{\\stepsize}{2} \\sqnorm{g^t - \\nabla F(x^t) }  \\notag \\\\\n\t\t&& \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } \\notag  ,\n\t\\end{eqnarray}\n\twhere $(i)$ follows from \\eqref{eq:projection_upd}, $(ii)$ holds by Young's inequality, i.e., $\\langle a, b\\rangle \\leq \\frac{\\stepsize}{2}\\sqnorm{a} + \\frac{1}{2 \\stepsize} \\sqnorm{b} $ with $a = \\nabla F(x^t) - g^t$, $b = x^{t+1} - z $, $(iii)$ holds by the smoothness of $F(\\cdot)$, i.e., $-\\frac{L}{2}\\sqnorm{z - x^t } \\leq F(x^t) - F(z) - \\langle \\nabla F(x^t) , z - x^t \\rangle $. \n\t\n\tWe are now ready to utilize the properties of hidden convex functions to bound $F(z)$ and $\\sqnorm{z - x^t}$ for some specific choice of $z \\in \\cX$. We select $z := x_{\\alpha}^t = c^{-1}((1-{\\alpha})c(x^t)+ {\\alpha} c(x^*)) \\in \\cX$, for some $\\alpha \\in [0,1]$, and $x^* \\in \\cX^* $. By \\Cref{prop:HC}, we have for $\\mu_H \\geq 0$, \n\t$$\n\tF(z) \\leq (1- \\alpha) F(x^t) +  \\alpha F(x^*) -    \\fr{(1-\\alpha) \\alpha  \\mu_H}{2} \\sqnorm{ c(x^t) - c(x^*) } ,\n\t$$\n and\n\t$$\n\t\\sqnorm{z - x^t } \\leq \\fr{ \\alpha^{2} }{ \\mu_c^2  } \\sqnorm{ c(x^t) - c(x^*) }.\n\t$$\n\tCombining the three inequalities above  and utilizing the assumption $\\stepsize \\leq 1/L$, we complete the proof.\n\t%\\begin{eqnarray}F(x^{t+1})  &\\leq&  (1- \\alpha) F(x^t) +  \\alpha F(x^*)  + \\rb{  \\fr{ \\alpha^{2} }{ \\mu_c^2 \\stepsize } -   \\fr{ (1-\\alpha)\\alpha   \\mu_H}{2} } \\sqnorm{ c(x^t) - c(x^*) }  \\notag \\\\ && \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } + \\frac{\\stepsize}{2} \\sqnorm{g^t - \\nabla F(x^t) } . \\end{eqnarray}\n\\end{proof}","sha256_utf8":"b218f4e8d311ee8253e6e47ec230b5d715552362f2fed0f1a86d9ed37002890b"}]},{"id":"S009","semantic_type":"proof_step","kind":"proof_step","title":"由投影更新得到三点不等式","source_units":[{"start":82598,"end":83009,"start_line":943,"end_line":948,"raw_tex":"\\begin{proof}\n\tBy the update rule of $x^{t+1}$ and following the standard descent inequality (cf. \\cref{le:bregman_properties_optimality}), we have for any $z \\in \\cX$ that\n\t\\begin{eqnarray}\\label{eq:projection_upd}\n\t\t\\langle g^t , x^{t+1} - z \\rangle  + \\frac{1}{2\\stepsize} \\sqnorm{ x^{t+1} - x^t} \\leq \\frac{1}{2\\stepsize} \\sqnorm{ z - x^t } -\\frac{1}{2\\stepsize} \\sqnorm{ z - x^{t+1} }  .\n\t\\end{eqnarray}\n\t\n","sha256_utf8":"0e0d1048a794fa49341e10fee4a471a0148cc244bfa45c253446e92a2e81abd9"}]},{"id":"S010","semantic_type":"proof_step","kind":"proof_step","title":"光滑性与杨氏不等式控制估计误差","source_units":[{"start":83009,"end":84962,"start_line":949,"end_line":964,"raw_tex":"\tBy the smoothness of $F(\\cdot)$, we derive\n\t\\begin{eqnarray}\n\t\tF(x^{t+1}) &\\leq& F(x^t) + \\langle \\nabla F(x^t), x^{t+1}- x^t \\rangle + \\fr{L}{2} \\sqnorm{x^{t+1}-x^t } \\notag \\\\\n\t\t&=& F(x^t) + \\langle g^t, x^{t+1}- x^t \\rangle + \\fr{1}{2 \\stepsize } \\sqnorm{x^{t+1}-x^t } \\notag \\\\\n\t\t&& \\qquad  + \\langle \\nabla F(x^t) - g^t , x^{t+1}- x^t \\rangle - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } \\notag \\\\\n\t\t&\\overset{(i)}{\\leq}&  F(x^t) + \\langle g^t, z - x^t \\rangle + \\fr{1}{2 \\stepsize } \\sqnorm{z - x^t } - \\fr{1}{2 \\stepsize } \\sqnorm{z - x^{t+1} }  \\notag \\\\\n\t\t&& \\qquad  + \\langle \\nabla F(x^t) - g^t , x^{t+1}- x^t \\rangle - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t }   \\notag \\\\\n\t\t& { = } & F(x^t) + \\langle \\nabla F(x^t), z - x^t \\rangle + \\fr{1}{2 \\stepsize } \\sqnorm{ z - x^t } - \\fr{1}{2 \\stepsize } \\sqnorm{ z - x^{t+1} }  \\notag \\\\  \n\t\t&& \\qquad + \\langle \\nabla F(x^t) - g^t , x^{t+1}- z \\rangle - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } \\notag  \\\\\n\t\t& \\overset{(ii)}{ \\leq } & F(x^t) + \\langle \\nabla F(x^t), z - x^t \\rangle + \\fr{1}{2 \\stepsize } \\sqnorm{z - x^t }   + \\frac{\\stepsize}{2} \\sqnorm{g^t - \\nabla F(x^t) }  \\notag \\\\\n\t\t&& \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } \\notag  \\\\\n\t\t&\\overset{(iii)}{\\leq}& F(z) + \\fr{L}{2}\\sqnorm{ z - x^t } + \\fr{1}{2 \\stepsize } \\sqnorm{z - x^t }  + \\frac{\\stepsize}{2} \\sqnorm{g^t - \\nabla F(x^t) }  \\notag \\\\\n\t\t&& \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } \\notag  ,\n\t\\end{eqnarray}\n\twhere $(i)$ follows from \\eqref{eq:projection_upd}, $(ii)$ holds by Young's inequality, i.e., $\\langle a, b\\rangle \\leq \\frac{\\stepsize}{2}\\sqnorm{a} + \\frac{1}{2 \\stepsize} \\sqnorm{b} $ with $a = \\nabla F(x^t) - g^t$, $b = x^{t+1} - z $, $(iii)$ holds by the smoothness of $F(\\cdot)$, i.e., $-\\frac{L}{2}\\sqnorm{z - x^t } \\leq F(x^t) - F(z) - \\langle \\nabla F(x^t) , z - x^t \\rangle $. \n\t\n","sha256_utf8":"3096114c4900c8365254f1302b6b7ddc29533c11c2ad053aa1b23be9f175ac74"}]},{"id":"S011","semantic_type":"proof_step","kind":"proof_step","title":"选取隐藏凸插值点并控制其距离","source_units":[{"start":84962,"end":85514,"start_line":965,"end_line":972,"raw_tex":"\tWe are now ready to utilize the properties of hidden convex functions to bound $F(z)$ and $\\sqnorm{z - x^t}$ for some specific choice of $z \\in \\cX$. We select $z := x_{\\alpha}^t = c^{-1}((1-{\\alpha})c(x^t)+ {\\alpha} c(x^*)) \\in \\cX$, for some $\\alpha \\in [0,1]$, and $x^* \\in \\cX^* $. By \\Cref{prop:HC}, we have for $\\mu_H \\geq 0$, \n\t$$\n\tF(z) \\leq (1- \\alpha) F(x^t) +  \\alpha F(x^*) -    \\fr{(1-\\alpha) \\alpha  \\mu_H}{2} \\sqnorm{ c(x^t) - c(x^*) } ,\n\t$$\n and\n\t$$\n\t\\sqnorm{z - x^t } \\leq \\fr{ \\alpha^{2} }{ \\mu_c^2  } \\sqnorm{ c(x^t) - c(x^*) }.\n\t$$\n","sha256_utf8":"7f947b861b8e793b4f4946cbd19dc19979d6cbc91c0de6ec3326f46e5a291e03"}]},{"id":"S012","semantic_type":"proof_step","kind":"proof_step","title":"合并比较界并使用步长条件","source_units":[{"start":85514,"end":85983,"start_line":973,"end_line":975,"raw_tex":"\tCombining the three inequalities above  and utilizing the assumption $\\stepsize \\leq 1/L$, we complete the proof.\n\t%\\begin{eqnarray}F(x^{t+1})  &\\leq&  (1- \\alpha) F(x^t) +  \\alpha F(x^*)  + \\rb{  \\fr{ \\alpha^{2} }{ \\mu_c^2 \\stepsize } -   \\fr{ (1-\\alpha)\\alpha   \\mu_H}{2} } \\sqnorm{ c(x^t) - c(x^*) }  \\notag \\\\ && \\qquad - \\rb{ \\fr{1}{2 \\stepsize } - \\fr{L}{2} } \\sqnorm{x^{t+1}-x^t } + \\frac{\\stepsize}{2} \\sqnorm{g^t - \\nabla F(x^t) } . \\end{eqnarray}\n\\end{proof}","sha256_utf8":"668fcaecc1edbbe8c7d4c7a6a17d42ca87e825538e579ab3319746e9824d20c2"}]},{"id":"S013","semantic_type":"proposition","kind":"proposition","title":"隐藏凸插值的函数值与距离界","source_units":[{"start":52224,"end":52807,"start_line":554,"end_line":562,"raw_tex":"\\begin{proposition}\\label{prop:HC}\n\t\tLet $F(\\cdot)$ be hidden convex with $\\mu_c>0, \\mu_H\\geq0$. For any $\\alpha \\in [0,1]$, $x^* \\in \\cX^*$ and $x\\in \\cX$ , define $x_{\\alpha} := c^{-1} \\rb{(1-\\alpha) c(x) + \\alpha c(x^*)}$. Then\n\t\t\\begin{eqnarray}\\label{prop:F_x_alpha_strong_conv}\n\t\t\tF(x_{\\alpha}) \\leq  (1- \\alpha) F(x) + \\alpha  F(x^*)  -   \\fr{(1-\\alpha) \\alpha  \\mu_H}{2}  \\sqpnorm{c(x) - c(x^*) } ,\n\t\t\\end{eqnarray}\t\\begin{eqnarray}\\label{prop:x_alpha_x_bound}\n\t\t\t\\pnorm{ x_{\\alpha}-x } \\leq \\fr{\\alpha }{\\mu_c} \\pnorm{c(x) - c(x^*) } .\n\t\t\\end{eqnarray}\n\t\t\n\t\\end{proposition}","sha256_utf8":"1ded5b6d9b4420e7b2d65a84b7408522943cb31fa83845380452c2f1de0e5ce4"}]},{"id":"S014","semantic_type":"proof","kind":"proof","title":"隐藏凸插值界的完整证明","source_units":[{"start":52811,"end":53728,"start_line":564,"end_line":577,"raw_tex":"\\begin{proof}\n\t\t%\\textbf{Part I: \\eqref{prop:F_x_alpha_strong_conv}.} \n\t\tBy the (strong) convexity of $H(\\cdot)$ and the convexity of $\\cU$, we have\n\t\t\\begin{eqnarray*}\n\t\t\tF(x_{\\alpha}) &=& F( c^{-1} \\rb{(1-\\alpha) c(x) + \\alpha c(x^*)} ) \\\\\n\t\t\t& = & H( (1-\\alpha) c(x) + \\alpha c(x^*) ) \\\\ \n\t\t\t& \\leq  &  (1-\\alpha) H( c(x) ) + \\alpha H( c(x^*) ) -   \\fr{(1-\\alpha) \\alpha  \\mu_H}{2} \\sqpnorm{c(x) - c(x^*) }\\\\\n\t\t\t& =  &  (1- \\alpha) F(x)  +  \\alpha F(x^*) -  \\fr{(1-\\alpha)  \\alpha   \\mu_H}{2} \\sqpnorm{c(x) - c(x^*) }.\n\t\t\\end{eqnarray*}\n  where the inequality uses the fact that $\\cU$ is a convex set and that $(1-\\alpha) c(x) + \\alpha c(x^*) \\in \\cU$ for any $x \\in \\cX$. By definition of $x_{\\alpha}$ and \\eqref{assumption:g_inv}, we derive\n\t\t$$\n\t\t\\pnorm{ x_{\\alpha}-x } = \\pnorm{ c^{-1} \\rb{(1-\\alpha) c(x) + \\alpha c(x^*)} - c^{-1}(c(x)) } \\leq \\fr{1}{\\mu_c} \\pnorm{\\alpha(c(x) - c(x^*)) } .\n\t\t$$\n\t\t\\end{proof}","sha256_utf8":"e45d1d9e7cda0ea49f635e1da67acd47b47d8463ac9d1ef879d8f0da843e5813"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"在重参数空间应用强凸插值","source_units":[{"start":52811,"end":53351,"start_line":564,"end_line":572,"raw_tex":"\\begin{proof}\n\t\t%\\textbf{Part I: \\eqref{prop:F_x_alpha_strong_conv}.} \n\t\tBy the (strong) convexity of $H(\\cdot)$ and the convexity of $\\cU$, we have\n\t\t\\begin{eqnarray*}\n\t\t\tF(x_{\\alpha}) &=& F( c^{-1} \\rb{(1-\\alpha) c(x) + \\alpha c(x^*)} ) \\\\\n\t\t\t& = & H( (1-\\alpha) c(x) + \\alpha c(x^*) ) \\\\ \n\t\t\t& \\leq  &  (1-\\alpha) H( c(x) ) + \\alpha H( c(x^*) ) -   \\fr{(1-\\alpha) \\alpha  \\mu_H}{2} \\sqpnorm{c(x) - c(x^*) }\\\\\n\t\t\t& =  &  (1- \\alpha) F(x)  +  \\alpha F(x^*) -  \\fr{(1-\\alpha)  \\alpha   \\mu_H}{2} \\sqpnorm{c(x) - c(x^*) }.\n\t\t\\end{eqnarray*}\n","sha256_utf8":"2114b16283c7fc004c99164aedb02fb12acfc8a95a13c58f8030176659482802"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"利用逆映射条件控制插值距离","source_units":[{"start":53351,"end":53728,"start_line":573,"end_line":577,"raw_tex":"  where the inequality uses the fact that $\\cU$ is a convex set and that $(1-\\alpha) c(x) + \\alpha c(x^*) \\in \\cU$ for any $x \\in \\cX$. By definition of $x_{\\alpha}$ and \\eqref{assumption:g_inv}, we derive\n\t\t$$\n\t\t\\pnorm{ x_{\\alpha}-x } = \\pnorm{ c^{-1} \\rb{(1-\\alpha) c(x) + \\alpha c(x^*)} - c^{-1}(c(x)) } \\leq \\fr{1}{\\mu_c} \\pnorm{\\alpha(c(x) - c(x^*)) } .\n\t\t$$\n\t\t\\end{proof}","sha256_utf8":"e1488113d801443bc37e5f89457edb6f560ea7c11655367c2ddad54980eb1fc6"}]},{"id":"S017","semantic_type":"lemma","kind":"lemma","title":"动量梯度估计误差的递推界","source_units":[{"start":81393,"end":81842,"start_line":925,"end_line":930,"raw_tex":"\\begin{lemma}[Lemma 2 in \\cite{fatkhullin2023momentum}]\\label{le:key_HB_recursion}\n\t\tLet $\\momentum \\in (0,1]$ and $g^t$ be updated via \\eqref{eq:PSGDM_x_upd}. Then\n\t\t\\begin{eqnarray}\\notag %\\label{eq:key_HB_recursion}\n\t\t\t\\Exp{ \\sqnorm{g^{t+1} - \\nabla F(x^{t+1})} }  \\leq (1 - \\momentum) \\Exp{ \\sqnorm{  g^{t} - \\nabla F(x^{t}) } }  + \\fr{3 L^2}{\\momentum} \\Exp{ \\sqnorm{  x^{t} - x^{t+1}  } } + \\momentum^2 \\sigma^2 .\n\t\t\\end{eqnarray}\n\t\\end{lemma}","sha256_utf8":"2907926b42ef9d0ed0686af4e5c9fe4b02baac0646c9ea41a9f7322d5922a14e"}]},{"id":"S018","semantic_type":"proof","kind":"proof","title":"动量误差递推的完整附录证明","source_units":[{"start":103239,"end":104457,"start_line":1171,"end_line":1184,"raw_tex":"\\begin{customproof}{\\Cref{le:key_HB_recursion}}\n\t\tUsing the update rule of $g^{t+1}$ and the unbiasedness of stochastic gradients, we have\n  $$\n\t\t\\begin{aligned}\n\t\t\t\\mathbb{E}&\\left[ \\sqnorm{g^{t+1} - \\nabla F(x^{t+1})} \\right] \n\t\t\t =  \\Exp{ \\sqnorm{(1 - \\momentum) g^{t} + \\momentum \\nabla f(x^{t+1}, \\xi^{t+1}) - \\nabla F(x^{t+1})}  } \\notag \\\\\n\t\t\t& =  (1-\\momentum)^2 \\Exp{ \\sqnorm{  g^{t} - \\nabla F(x^{t+1}) } }  +  \\momentum^2 \\Exp{\\sqnorm{\\nabla f(x^{t+1}, \\xi^{t+1}) - \\nabla F(x^{t+1}) }} \\notag \\\\\n\t\t\t& \\leq   (1 - \\momentum)^2 \\rb{ 1 + \\nfr{\\momentum}{2}} \\Exp{ \\sqnorm{  g^{t} - \\nabla F(x^{t}) } }  + \\rb{1+\\nfr{2}{\\momentum} } \\Exp{ \\sqnorm{  \\nabla F(x^{t}) - \\nabla F(x^{t+1}) }  } + \\momentum^2 \\sigma^2  \\notag \\\\\n\t\t\t& \\leq   (1 - \\momentum) \\Exp{ \\sqnorm{  g^{t} - \\nabla F(x^{t}) } }  + \\fr{3 L^2}{\\momentum} \\Exp{ \\sqnorm{  x^{t} - x^{t+1}  } } \n   + \\momentum^2 \\sigma^2  \\notag ,\n\t\t\\end{aligned}\n  $$\n\t\twhere the first inequality uses Young's inequality and the bound of the variance of stochastic gradients, and the last step uses the Lipschitz continuity of the gradient and the fact that $(1 - \\momentum) \\rb{ 1 + \\nfr{\\momentum}{2}} \\leq 1 $ for all $\\momentum \\in (0,1]$.\n\t\\end{customproof}","sha256_utf8":"db50730580a7e86c3d03048d0f025490aa723c981490e3d83907af0a4f9ee965"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"展开随机误差并推导二阶矩界","source_units":[{"start":103239,"end":104163,"start_line":1171,"end_line":1182,"raw_tex":"\\begin{customproof}{\\Cref{le:key_HB_recursion}}\n\t\tUsing the update rule of $g^{t+1}$ and the unbiasedness of stochastic gradients, we have\n  $$\n\t\t\\begin{aligned}\n\t\t\t\\mathbb{E}&\\left[ \\sqnorm{g^{t+1} - \\nabla F(x^{t+1})} \\right] \n\t\t\t =  \\Exp{ \\sqnorm{(1 - \\momentum) g^{t} + \\momentum \\nabla f(x^{t+1}, \\xi^{t+1}) - \\nabla F(x^{t+1})}  } \\notag \\\\\n\t\t\t& =  (1-\\momentum)^2 \\Exp{ \\sqnorm{  g^{t} - \\nabla F(x^{t+1}) } }  +  \\momentum^2 \\Exp{\\sqnorm{\\nabla f(x^{t+1}, \\xi^{t+1}) - \\nabla F(x^{t+1}) }} \\notag \\\\\n\t\t\t& \\leq   (1 - \\momentum)^2 \\rb{ 1 + \\nfr{\\momentum}{2}} \\Exp{ \\sqnorm{  g^{t} - \\nabla F(x^{t}) } }  + \\rb{1+\\nfr{2}{\\momentum} } \\Exp{ \\sqnorm{  \\nabla F(x^{t}) - \\nabla F(x^{t+1}) }  } + \\momentum^2 \\sigma^2  \\notag \\\\\n\t\t\t& \\leq   (1 - \\momentum) \\Exp{ \\sqnorm{  g^{t} - \\nabla F(x^{t}) } }  + \\fr{3 L^2}{\\momentum} \\Exp{ \\sqnorm{  x^{t} - x^{t+1}  } } \n   + \\momentum^2 \\sigma^2  \\notag ,\n\t\t\\end{aligned}\n  $$\n","sha256_utf8":"933ebceda14c8190fb1e61f449b5948a0df0c3a9f529e9e83923e49e22014b0c"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"核验方差光滑性与动量系数估计","source_units":[{"start":104163,"end":104457,"start_line":1183,"end_line":1184,"raw_tex":"\t\twhere the first inequality uses Young's inequality and the bound of the variance of stochastic gradients, and the last step uses the Lipschitz continuity of the gradient and the fact that $(1 - \\momentum) \\rb{ 1 + \\nfr{\\momentum}{2}} \\leq 1 $ for all $\\momentum \\in (0,1]$.\n\t\\end{customproof}","sha256_utf8":"27f5d67061dfc53358b624ac48809531fed5beb36567f03fe6aa3b8f3828e1b0"}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","title":"凸近端子问题的标准三点界","source_units":[{"start":104547,"end":105027,"start_line":1190,"end_line":1196,"raw_tex":"\\begin{lemma}\\label{le:bregman_properties}\n\t\t\tLet $\\phi(\\cdot)$ be convex and for any $\\stepsize>0$, $x\\in \\cX$, define $$x^{+} := \\argmin_{y\\in \\cX} \\left\\{ \\phi(y) + \\fr{1}{2 \\stepsize} \\sqnorm{y - x} \\right\\}. $$ \n        Then\n\t\t\t$$\n\t\t\t\\phi(y) + \\fr{1}{2 \\stepsize} \\sqnorm{y - x} \\geq \\phi(x^{+}) + \\fr{1}{2 \\stepsize} \\sqnorm{x^{+} - x }  + \\fr{1}{2\\stepsize} \\sqnorm{ y  - x^{+} }  \\quad \\text{for all } y \\in \\cX. \n\t\t\t$$\\label{le:bregman_properties_optimality}\n\t\\end{lemma}","sha256_utf8":"07ae1b5e5af47b3dba956904203ef8b6f1d118e9dec1783bbecb642754cb1a7b"}]},{"id":"S022","semantic_type":"external_fact","kind":"external_fact","title":"强凸子问题的最优性增长界","source_units":[],"declared_external":true,"declaration_text":["For an r-strongly convex function Q on a convex set, a minimizer x+ satisfies Q(y) >= Q(x+) + (r/2)||y-x+||^2. Applied to the convex proximal subproblem, this is the standard three-point inequality stated in B0137 and cited to Guler_PPM."]},{"id":"S023","semantic_type":"external_fact","kind":"external_fact","title":"杨氏不等式与光滑函数比较界","source_units":[],"declared_external":true,"declaration_text":["Young's inequality gives <a,b> <= (s/2)||a||^2 + (1/(2s))||b||^2 for s>0. An L-Lipschitz gradient gives |F(y)-F(x)-<grad F(x),y-x>| <= (L/2)||y-x||^2."]},{"id":"S024","semantic_type":"problem_setting","kind":"problem_setting","title":"约束随机目标及其凸重参数化","source_units":[{"start":6335,"end":6452,"start_line":204,"end_line":207,"raw_tex":"\\begin{equation}\n\t\t\\label{problem:original}\n\t\t\\min_{x\\in\\cX}~F(x) := \\Expu{\\xi \\sim \\cD}{f(x, \\xi)}, \n\t\\end{equation}","sha256_utf8":"6f680574a187677e43fd96fb691d83dad3271699ec39907df7ea3f149dabdee7"},{"start":6453,"end":6745,"start_line":208,"end_line":208,"raw_tex":"\twhere $\\cX $ is a closed convex subset of $\\R^d$, $\\xi$ is a random variable following an unknown distribution $\\cD$, and $F(\\cdot)$ is possibly non-convex in $x$. Our central structural assumption about \\eqref{problem:original} is that it admits a \\textit{convex reformulation} of the form\n","sha256_utf8":"0b9dd01cb184ca71b343f3c8ac39d4be905d41a67ce7a52f632b6d49328733a9"},{"start":6746,"end":6874,"start_line":209,"end_line":212,"raw_tex":"\\begin{equation}\n\t\t\\label{problem:reformulation}\n\t\t\\min_{u\\in \\cU}~H(u):= F(c^{-1}(u)), %\\text{with  } g(x) = u,\n\t\\end{equation}","sha256_utf8":"544a22a5643866770ae48a1f48aed0c395ee79b47115a4a57c3cafaaa0249975"},{"start":24101,"end":24208,"start_line":300,"end_line":303,"raw_tex":"\\begin{equation}\n\t\t\\label{problem:original_composition}\n\t\t\\min_{x\\in\\cX}~F(x) := H(c(x)) . \n\t\\end{equation}","sha256_utf8":"821b19dd4efac2c09c98d350313b4deeb9822a8eef82922abe7602af99ed56f3"}]},{"id":"S025","semantic_type":"notation","kind":"notation","title":"欧氏范数逆映射与集合直径","source_units":[{"start":19048,"end":20184,"start_line":267,"end_line":269,"raw_tex":"\tThroughout, we denote by $\\ve{\\cdot}{\\cdot}$ the inner product in $\\R^d$ along with its induced Euclidean norm $\\norm{\\cdot}$. For a real valued matrix $A\\in \\R^{m\\times n}$, we denote by $\\opnorm{\\cdot}$ its operator norm, i.e.,  $\\opnorm{A} := \\max_{\\norm{x}\\leq 1}\\norm{A x}$. {\\color{black} We denote the interior and the boundary of $\\cX$ as $\\operatorname{int} \\cX$ and $\\partial \\cX$ respectively.} %For any map $c: \\cX \\rightarrow \\cU$, we say $c$ is $L$-Lipschitz continuous on the set $\\cX \\subset \\R^d$ if there exists a constant $L < +\\infty$ such that for all $x, y\\in \\cX$ it holds $\\norm{c(x) - c(y)} \\leq L \\norm{x - y}. $\n    %We denote $\\text{Ker}( A ) := \\left\\{ x \\in \\R^n | A x = 0 \\right\\} $.\n\tThe map $c: \\cX \\rightarrow \\cU$ is called invertible if there exists a map $c^{-1}: \\cU \\rightarrow \\cX$ (called inverse) such that $c^{-1}(c(x)) = x$ for any $x \\in \\cX$ and $c( c^{-1}(u) ) = u$ for any $u \\in \\cU$. For any $u, v\\in\\cU$ and any $\\lambda\\in[0,1]$, if  $(1-\\lambda) u + \\lambda v \\in \\cU$, we say $\\mathcal{U}$ is convex. We denote the diameter of $\\cU$ as $D_{\\cU}:= \\sup_{u, v\\in \\cU}\\pnorm{u - v}$.\n","sha256_utf8":"7f348807d991ef6eb361368e117eac6f9045b730781a456e600d934291967400"}]},{"id":"S026","semantic_type":"definition","kind":"definition","title":"弱凸性次梯度光滑性与投影","source_units":[{"start":20628,"end":21811,"start_line":278,"end_line":279,"raw_tex":"A function $F: \\cX \\rightarrow\\R \\cup \\{+\\infty\\}$ is $\\ell$-weakly convex ($\\ell$-WC) if for any fixed $y\\in \\cX$, $F_{\\ell}(x,y) := F(x) + \\frac{\\ell}{2} \\, \\sqnorm{x- y}$ is convex in $x \\in \\cX$.\nIf $x\\notin \\cX$, we assign $F(x) = +\\infty$. The (Fr\\'echet) sub-differential of $F$ at $x\\in \\cX$ is $\\partial F(x) := \\left\\{g \\in \\mathbb{R}^d \\mid  F(y) \\geq F(x)+\\langle g, y-x\\rangle+o(\\|y-x\\|), \\forall y \\in \\R^d  \\right\\}.$ The elements $g \\in \\partial F(x)$ are called sub-gradients of $F$ at $x$, see \\cite{davis2019proximally} for alternative equivalent definitions of the sub-differential set for $\\ell$-WC functions. A differentiable function $F: \\cX \\rightarrow \\R$ is $L$-smooth on $\\cX \\subset \\R^d$ if its gradient is $L$-Lipschitz continuous on the set $\\cX$, i.e., it holds $\\norm{\\nabla F(x) - \\nabla F(y)} \\leq L \\norm{x - y}$ for all $x, y\\in \\cX$. For a convex set $\\cX \\subset \\R^d$, the projection of a point $y\\in \\R^d$ onto $\\cX$ is  $\\Pi_{\\cX}(y) := \\argmin_{x\\in\\cX}\\norm{y - x}$. We denote $\\delta_{\\cX}$ as the indicator function of a set $\\cX \\subset \\R^d$ and define $\\delta_{\\cX}(x)  = 0$ if $x\\in \\cX$ and $\\delta_{\\cX}(x)  = +\\infty$ otherwise. \n","sha256_utf8":"d1a41cac097b9027d5f1fb09c6a5d2e341ab9389635e4de3c2d925b20591ac1a"}]},{"id":"S027","semantic_type":"notation","kind":"notation","title":"最优解最优值与驻点记号","source_units":[{"start":22294,"end":22571,"start_line":282,"end_line":282,"raw_tex":"    We define by $\\cX^* \\subset \\cX$ the set of optimal points of $\\min_{x\\in\\cX} F(x)$ and by $F^*$ its optimal value. A point $\\bar{x} \\in \\cX$ is called a stationary point of a weakly convex function $F: \\cX \\rightarrow \\R$ if $0 \\in \\partial (F +  \\delta_{\\cX})(\\bar{x})$.\n","sha256_utf8":"9e6f3ebacf6a3f47e363863697af64aeef9563c5ead2b1ef8801a8b709cceac9"}]},{"id":"S028","semantic_type":"definition","kind":"definition","title":"隐藏凸性及逆映射模量条件","source_units":[{"start":24255,"end":25696,"start_line":305,"end_line":324,"raw_tex":"\\begin{definition}\nThe  above problem is called {hidden convex} with modulus $\\mu_c>0, \\mu_H\\geq 0$ (or function $F$ is hidden convex on $\\cX$) if its components satisfy the following underlying conditions. \n\t\\begin{enumerate}%[ref=\\theenumi{}~(\\alph*)]\n\t\t%\\item The set $\\cX \\subset \\R^d $ is closed and convex. \\label{ass:convex_cX}\n\t\t\\myitem{C.1}\\label{C1} The domain $\\mathcal{U}=c(\\mathcal{X})$ is convex, the function $H: \\cU \\rightarrow \\R$ satisfies that for all $u, v \\in \\cU$ and $ \\lambda \\in [0,1]$, \n  \\begin{equation}\\label{assumption:H_conv}\n  \tH( (1-\\lambda)u + \\lambda v ) \\leq  (1- \\lambda) H(u) +  \\lambda  H(v)  -   \\fr{(1-\\lambda)\\lambda \\mu_H}{2} \\sqpnorm{ u - v }, \n  \\end{equation}\n  and \\eqref{problem:reformulation} admits a solution $u^* \\in \\cU$.  %\\label{ass:convex_H} \n  \n\t\t\\myitem{C.2} The map $c: \\cX \\rightarrow \\cU$ is invertible. There exists $\\mu_c>0$ such that  for all $x, y \\in \\cX$ it holds  \\label{C2}\n\t\t\\begin{equation}\\label{assumption:g_inv}\n\t\t\t\\pnorm{c(x) - c(y) } \\geq \\mu_c \\pnorm{x - y }.\n\t\t\\end{equation}\n\t\t%\\item The function $F: \\cX \\rightarrow \\R$ is differentiable on $\\cX$ and its gradient $\\nabla F(x)$ is $L$-Lipschitz continuous. \n\t\\end{enumerate}\n\t%\\IF{Check if A.2. is connected to metric subregularity condition from the works of Russel Luke. }\n\t%\\end{assumption}\nIn particular, if $\\mu_H>0$, we say the above problem is {$(\\mu_c,\\mu_H)$-hidden strongly convex}.  \n\\end{definition}","sha256_utf8":"bf53394b4ca8bc2f02aa50e73c2d1649a36c582db4fad5b395056b42f3ecad3e"}]},{"id":"S029","semantic_type":"assumption","kind":"assumption","title":"光滑性与无偏有界方差假设","source_units":[{"start":70448,"end":71005,"start_line":795,"end_line":801,"raw_tex":"    \\begin{enumerate}\n    \\myitem{A.1'}\\label{A1'} The function $F: \\cX \\rightarrow \\R$ is differentiable on a closed, convex set $\\cX$ and its gradient $\\nabla F(x)$ is $L$-Lipschitz continuous. \n    \n    \\myitem{A.2'}\\label{A2'} We have access to an unbiased stochastic gradient oracle with bounded variance $\\sigma > 0$, i.e. for any $x\\in \\cX$: \n\t$\\Exp{\\nabla f(x,\\xi)} = \\nabla F(x)$, and \n\t$$\\Exp{\\sqnorm{\\nabla f(x,\\xi) - \\nabla F(x)}} \\leq \\sigma^2, $$where expectations are with respect to the random variable $\\xi \\sim \\cD$.  \n    \\end{enumerate}\n","sha256_utf8":"65ee038a68dd9ce403ab7bc818e10e9bcd3b0c35af9a2da12a9ed7ccbba9afe3"}]},{"id":"S030","semantic_type":"algorithm","kind":"algorithm","title":"投影随机梯度与动量更新","source_units":[{"start":80394,"end":80627,"start_line":909,"end_line":914,"raw_tex":"\\begin{align}\n\t%\\boxed{ \n\t\tx^{t+1} = \\Pi_{\\mathcal{X}}(x^t-\\stepsize \\, g^t) , \\qquad  \\label{eq:PSGDM_x_upd}\n\t\tg^{t+1} = (1-\\momentum) \\, g^{t} + \\momentum \\, \\nabla f(x^{t+1}, \\xi^{t+1}) . \n\t\t%} %\\label{eq:PSGDM_g_upd} \n\\end{align}","sha256_utf8":"d3ddbda259be9ccf149c99048286074364ee097b958fa4714d80e621672a2c3a"}]},{"id":"S031","semantic_type":"definition","kind":"definition","title":"函数差与动量误差联合势函数","source_units":[{"start":80862,"end":81095,"start_line":918,"end_line":921,"raw_tex":"Let $x^* \\in \\cX^*$, for any $x^t \\in \\cX$, we define the Lyapunov function\n\\begin{equation}\\label{eq:Lyapunov_HB}\n\t\\Lambda_t^{HB} := \\Exp{ F(x^t) - F(x^*) + \\frac{\\stepsize}{\\momentum} \\sqnorm{g^t - \\nabla F(x^t)} } .\n\\end{equation}","sha256_utf8":"83ca4a545a56f2b1b11f767b273b0e49e0e168f06ae44d146d39cebcdbdbcfc7"}]},{"id":"S032","semantic_type":"external_fact","kind":"external_fact","title":"几何收缩递推的有限步展开","source_units":[],"declared_external":true,"declaration_text":["For 0<alpha<=1, a recursion a_(t+1) <= (1-alpha)a_t+b implies a_T <= (1-alpha)^T a_0+b/alpha; also (1-alpha)^T <= exp(-alpha T)."]},{"id":"S033","semantic_type":"assumption","kind":"assumption","title":"主定理的凸性直径与参数条件","source_units":[{"start":86232,"end":86500,"start_line":986,"end_line":986,"raw_tex":"\tLet \\ref{C1}, \\ref{C2}, \\ref{A1'}, \\ref{A2'} hold with $\\mu_H = 0$, and the set $\\cU$ be bounded by a diameter $D_{\\cU}$. Then for any $\\stepsize \\leq \\frac{\\momentum}{4 L}$, $\\momentum \\in (0, 1]$, and $\\alpha \\leq  \\frac{\\momentum}{2} $, we have for any $T \\geq 0$\n","sha256_utf8":"fd28aa5d74121133d0e5418eed73bd4aba3efd2f1cdca41141dd0934e59b5e74"}]},{"id":"S034","semantic_type":"assumption","kind":"assumption","title":"下降分析的结构与步长条件","source_units":[{"start":82014,"end":82215,"start_line":936,"end_line":936,"raw_tex":"\tSuppose that \\ref{C1}, \\ref{C2}, \\ref{A1'}, \\ref{A2'} hold with $\\mu_H\\geq0$, and the step-size in \\eqref{eq:PSGDM_x_upd} satisfies $\\stepsize \\leq 1 / L$. For any $\\alpha \\in [0, 1] $, it holds that\n","sha256_utf8":"09159d041e4f8d020a8cf4ccdcf0235ae8ae3b50ba2dc061419648a5a258284a"}]},{"id":"S035","semantic_type":"assumption","kind":"assumption","title":"插值参数最优点与插值点设定","source_units":[{"start":52259,"end":52455,"start_line":555,"end_line":555,"raw_tex":"\t\tLet $F(\\cdot)$ be hidden convex with $\\mu_c>0, \\mu_H\\geq0$. For any $\\alpha \\in [0,1]$, $x^* \\in \\cX^*$ and $x\\in \\cX$ , define $x_{\\alpha} := c^{-1} \\rb{(1-\\alpha) c(x) + \\alpha c(x^*)}$. Then\n","sha256_utf8":"2682775daec0a68b51877a4042b78d1f8892555efbf32a00b41e81864dd76c75"}]},{"id":"S036","semantic_type":"assumption","kind":"assumption","title":"动量参数范围与更新条件","source_units":[{"start":81476,"end":81558,"start_line":926,"end_line":926,"raw_tex":"\t\tLet $\\momentum \\in (0,1]$ and $g^t$ be updated via \\eqref{eq:PSGDM_x_upd}. Then\n","sha256_utf8":"f65217b1dfe89b03a8a6c17e0225aa6a348ea1e3ef2a63df8b081d59818a9eb0"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S006","target_id":"S002","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S007","target_id":"S003","kind":"supports"},{"source_id":"S008","target_id":"S007","kind":"supports"},{"source_id":"S009","target_id":"S008","kind":"supports"},{"source_id":"S010","target_id":"S008","kind":"supports"},{"source_id":"S011","target_id":"S008","kind":"supports"},{"source_id":"S012","target_id":"S008","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S013","target_id":"S011","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S016","target_id":"S014","kind":"supports"},{"source_id":"S017","target_id":"S004","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S020","target_id":"S018","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S009","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S023","target_id":"S010","kind":"supports"},{"source_id":"S023","target_id":"S019","kind":"supports"},{"source_id":"S023","target_id":"S020","kind":"supports"},{"source_id":"S024","target_id":"S028","kind":"supports"},{"source_id":"S024","target_id":"S029","kind":"supports"},{"source_id":"S025","target_id":"S003","kind":"supports"},{"source_id":"S025","target_id":"S016","kind":"supports"},{"source_id":"S026","target_id":"S009","kind":"supports"},{"source_id":"S026","target_id":"S010","kind":"supports"},{"source_id":"S027","target_id":"S031","kind":"supports"},{"source_id":"S027","target_id":"S035","kind":"supports"},{"source_id":"S028","target_id":"S015","kind":"supports"},{"source_id":"S028","target_id":"S016","kind":"supports"},{"source_id":"S029","target_id":"S010","kind":"supports"},{"source_id":"S029","target_id":"S019","kind":"supports"},{"source_id":"S029","target_id":"S020","kind":"supports"},{"source_id":"S030","target_id":"S009","kind":"supports"},{"source_id":"S030","target_id":"S019","kind":"supports"},{"source_id":"S031","target_id":"S004","kind":"supports"},{"source_id":"S031","target_id":"S005","kind":"supports"},{"source_id":"S031","target_id":"S006","kind":"supports"},{"source_id":"S032","target_id":"S005","kind":"supports"},{"source_id":"S032","target_id":"S006","kind":"supports"},{"source_id":"S033","target_id":"S003","kind":"supports"},{"source_id":"S033","target_id":"S004","kind":"supports"},{"source_id":"S034","target_id":"S011","kind":"supports"},{"source_id":"S034","target_id":"S012","kind":"supports"},{"source_id":"S035","target_id":"S015","kind":"supports"},{"source_id":"S035","target_id":"S016","kind":"supports"},{"source_id":"S036","target_id":"S019","kind":"supports"},{"source_id":"S036","target_id":"S020","kind":"supports"}]}}
{"record_id":"2401.03155v4","paper":{"title":"Stochastic Bregman Proximal Gradient Method Revisited: Kernel Conditioning and Painless Variance Reduction","authors":["Junyu Zhang"],"arxiv_id":"2401.03155","version":"v4","journal":"Mathematical Programming","doi":"10.1007/s10107-025-02285-2","abstract_url":"https://arxiv.org/abs/2401.03155v4","pdf_url":"https://arxiv.org/pdf/2401.03155v4"},"context":{"raw_tex":"\\begin{equation}\n    \\label{prob:main-det}\n    \\min_{x\\in\\RR^d} \\Psi(x) = f(x) + \\phi(x) \\qquad\\mbox{with}\\qquad f(x) = \\frac{1}{n}\\sum_{i=1}^n f_i(x),\n\\end{equation}\n\nwhere $\\phi(x)$ is a convex but possibly non-differentiable function, while $f$ and each $f_i$ are nonconvex and continuously differentiable.\n\n\n\\begin{equation}\n    \\label{defn:BPG-standard}\n    x_{k+1} = \\mathbf{T}_{\\phi,h}^\\lambda\\big(x_k, \\nabla f(x_k)\\big)\\qquad\\mbox{with}\\qquad \\mathbf{T}_{\\phi,h}^\\lambda (x,v) := \\argmin_{y\\in\\RR^d}\\,\\, y^\\top v + \\phi(y) + \\lambda^{-1}D_h(y,x),\\vspace{-0.1cm}\n\\end{equation}\n\nwhere $D_h(y,x):=h(y)-\\nabla h(x)^\\top(y-x)-h(x)$ stands for the Bregman divergence induced by $h$. \n\n\n\\begin{equation}\n    \\label{defn:GradMap-BPG}\n    \\mathcal{G}_{\\phi,h}^\\lambda(x):= \\frac{x-\\mathbf{T}_{\\phi,h}^\\lambda\\big(x,\\nabla f(x)\\big)}{\\lambda}.  \n\\end{equation}\n\n\\noindent\\textbf{Notations.} For $\\forall x\\in\\RR^d$, We default $\\|x\\|:=\\sqrt{x^\\top x}$  the $\\ell_2$-norm, and we denote $\\|x\\|_1:=\\sum_i|x_i|$ as the $\\ell_1$-norm. For a matrix $X\\in\\RR^{d\\times d}$, we use $\\lambda_{\\max}(X)$ and $\\lambda_{\\min}(X)$ to denote the maximal and minimal eigenvalues of $X$, respectively. And we use $\\|X\\|$ to denote the $\\ell_2$-operator norm of $X$. When $X$ is positive semidefinite, we write $X\\succeq0$, and we have $\\|X\\| = \\lambda_{\\max}(X)$ in this case. For any set $\\cX$, \nwe denote $\\mathrm{id}_{\\cX}(\\cdot)$ as the indicator function of the set.  Namely, $\\mathrm{id}_{\\cX}(x)=0$ if $x\\in\\cX$ and $\\mathrm{id}_{\\cX}(x) = +\\infty$ if $x\\notin\\cX$. We denote the interior of $\\cX$ as $\\mathrm{int}(\\cX)$ and we denote the boundary of $\\cX$ as $\\partial \\cX$. We denote $[n]:=\\{1,2,\\cdots,n\\}$. For any vector $x\\in\\RR^d$, the function $\\mathrm{sign}: \\RR^d\\to\\RR^d$ returns a sign vector of $x$. That is, for any $s=\\mathrm{sign}(x)$, its $i$-th element satisfies $s_i=1$ if $x_i\\geq0$ and $s_i=-1$ if $x_i<0$, for any $1\\leq i\\leq d$. Because many literature use the terminology $L$-smooth to denote $L$-Lipschitz continuity of the gradient, we will use ``continuously differentiable'' instead of ``smooth''  to avoid confusion. \n\n\n\\begin{assumption}[Smooth adaptability, \\cite{bolte2018first}]\n    \\label{assumption:L-smad}\n    Let $f$ and $h$ be twice continuously differentiable in $\\mathbb{R}^d$, and let $h$ be strictly convex. Then we assume $f$ is $L$-smooth adaptable to $h$ for some positive constant $L>0$. In other words, both  $Lh+f$ and $Lh-f$ are convex functions. \n\\end{assumption}\n\n\\noindent Given the twice continuous differentiability of $f$ and $h$, Assumption \\ref{assumption:L-smad} can be equivalently written  as \n\\begin{equation}\n    \\label{asmp:L-smad-alt}\n    -L\\nabla^2h(x)\\preceq \\nabla^2 f(x)\\preceq L\\nabla^2h(x),\\,\\, \\forall x\\in\\RR^d.\n\\end{equation}\n\n\\begin{equation}\n    \\label{defn:condition}\n    \\mu_h(\\mathcal{X}) \\,=\\, \\min_{x\\in\\cX}\\,\\lambda_{\\min}\\big(\\nabla^2 h(x)\\big),\\quad L_h(\\mathcal{X}) \\,=\\, \\max_{x\\in\\cX}\\,\\lambda_{\\max}\\big(\\nabla^2 h(x)\\big),\\quad\\mbox{and}\\quad \\kappa_h(\\cX) := \\frac{L_h(\\cX)}{\\mu_h(\\cX)}.\n\\end{equation}\n\n\\begin{assumption}[Kernel-conditioning]\n    \\label{assumption:kernel-conditioning}\n    For any $\\delta>0$, there exists a constant $\\kappa_h^\\delta>0$ such that  $$\\sup_{\\mathcal{X}\\subseteq\\mathbb{R}^d}\\Big\\{\\kappa_h(\\mathcal{X}):\\mathrm{diam}(\\mathcal{X})\\leq \\delta\\Big\\}\\leq \\kappa_h^\\delta,$$\nwhere $\\mathrm{diam}(\\cX):=\\sup\\{\\|x-y\\|: x,y\\in\\cX\\}$ denotes the diameter of the set $\\cX$.\n\\end{assumption}\n\n\\begin{assumption}[Strong convexity]\n    \\label{assumption:SC}\n    The kernel $h$ is $\\mu$-strongly convex for some $\\mu>0$.\n\\end{assumption}\n\n\\begin{assumption}\n    \\label{assumption:L-smad-finite-sum}\n    For each $i\\in[n]$, $f_i$ is $L_i$-smooth adaptable to $h$ for some positive constant $L_{i}>0$. Denote $L:=\\sqrt{\\frac{1}{n}\\sum_i^nL_i^2}$, then $f$ is $L$-smooth adaptable to $h$. \n\\end{assumption}\n\nThroughout Section \\ref{section:VR}, we will use Assumption \\ref{assumption:L-smad-finite-sum}, Assumption \\ref{assumption:kernel-conditioning}, and Assumption \\ref{assumption:SC}, and we propose a stochastic variance reduced BPG method with epoch-wise bounds  in Algorithm \\ref{algorithm:proxSARAH}. \n\n\n\\begin{algorithm2e}\n\\caption{Stochastic variance reduced BPG method with epoch bounds} \n\\label{algorithm:proxSARAH}\n\\textbf{Input:} Initial point $x_{1,0}$, constant $\\delta$, epoch length $\\tau$, step size $\\eta$,  interpolation factor $\\gamma\\in(0,1]$.   \\\\\n\\For{$s=1,2,3,\\cdots,S$}{\n    Construct a convex set $\\mathcal{X}_s\\supseteq {B}(x_{s,0},\\delta/2)$ such that $\\kappa_h(\\mathcal{X}_s)\\leq \\kappa_h^\\delta$\\,.  \\\\\\textcolor{gray}{//**Proposition \\ref{proposition: Poly-kernel-condition} suggests $\\mathcal{X}_s= B\\left(x_{s,0},\\max\\left\\{\\frac{1}{2r},\\frac{\\|x_{s,0}\\|}{2r+1}\\right\\}\\right)$ for polynomial kernel**//}\n    \n    \\For{$k = 0,1,2,\\cdots,\\tau-1$}{  \n    If $k==0$, compute $v_{s,0}= \\nabla f(x_{s,0}) = \\frac{1}{n}\\sum_{i=1}^n\\nabla f_i(x_{s,0})$.\\\\\n    If $k\\geq 1$, uniformly sample a mini-batch $\\mathcal{B}_{s,k}\\subseteq[n]$  with replacement, compute  \n    \\begin{equation}\n        \\label{alg:sarah-small}\n        v_{s,k} = v_{s,k-1} + \\frac{1}{|\\mathcal{B}_{s,k}|}\\sum_{\\xi\\in\\mathcal{B}_{s,k}} \\Big(\\nabla f_\\xi(x_{s,k})-\\nabla f_\\xi(x_{s,k-1})\\Big)\\,.\n    \\end{equation}\\\\\n    \n    Denote $\\mathrm{id}_{\\cX_s}$ the indicator function of $\\cX_s$. Compute the BPG update with \n\n   \n    \\begin{equation}\n        \\label{alg:proxSARAH-update}\n        \\bar{x}_{s,k+1} =  \\Tshe(x_{s,k},v_{s,k})\\qquad\\mbox{and}\\qquad x_{s,k+1} = (1-\\gamma)x_{s,k}+\\gamma\\bar{x}_{s,k+1}\\vspace{-0.1cm}\n    \\end{equation}  \\\\ \n    \\textbf{if}\\,\\,\\,$\\mathrm{dist}(x_{s,k+1},\\partial \\cX_s) \\leq \\delta/4$\\,\\,\\,\\textbf{then}\\,\\,\\,\\textbf{break} the inner forloop.\n    } \n    Set $\\tau_s=k+1$ and $x_{s+1,0} = x_{s,\\tau_s}$. \n    } \n\\end{algorithm2e}\n\n\\begin{equation} \n        \\label{alg:sarah-update}\n        \\bar{x}_{s,k+1} = \\argmin_{x\\in\\mathcal{X}_s}\\,\\, \\langle v_{s,k}, x\\rangle + \\phi(x) +  \\frac{1}{\\eta}D_h(x,x_{s,k}) . \n    \\end{equation}\n\nDefine the true \\emph{restricted} primal gradient mapping in epoch $s$ as  \n\\begin{equation}\n    \\label{defn:restricted-grd-mapping}\n    \\cGshe(x_{s,k}):=\\frac{x_{s,k}-\\hat{x}_{s,k+1}}{\\eta}\\quad\\mbox{with}\\quad\\hat{x}_{s,k+1}:=\\Tshe\\big(x_{s,k},\\nabla f(x_{s,k})\\big),\n\\end{equation}\n\nwhere by ``restricted'' we means this primal gradient mapping incorporates the indicator function of the set constraint $x\\in\\cX_s$.\nCompared to the $\\bar{x}_{s,k+1}$ in Algorithm \\ref{algorithm:proxSARAH}, $\\hat{x}_{s,k+1}$ is constructed with the true gradient $\\nabla f(x_{s,k})$.  For this restricted primal gradient mapping, the following lemma holds true. \n","units":[{"start":5423,"end":5589,"start_line":114,"end_line":117,"raw_tex":"\\begin{equation}\n    \\label{prob:main-det}\n    \\min_{x\\in\\RR^d} \\Psi(x) = f(x) + \\phi(x) \\qquad\\mbox{with}\\qquad f(x) = \\frac{1}{n}\\sum_{i=1}^n f_i(x),\n\\end{equation}","sha256_utf8":"0983a3f59fd47f2eed32528fe77b0e8c2d89cbb635eb8c8d57bbaaab21b08b92"},{"start":5591,"end":5733,"start_line":118,"end_line":118,"raw_tex":"where $\\phi(x)$ is a convex but possibly non-differentiable function, while $f$ and each $f_i$ are nonconvex and continuously differentiable.\n","sha256_utf8":"f4392c520bad102052cfb6d7b6528aea078982abb4c79465b3c771f40453f052"},{"start":8314,"end":8587,"start_line":133,"end_line":136,"raw_tex":"\\begin{equation}\n    \\label{defn:BPG-standard}\n    x_{k+1} = \\mathbf{T}_{\\phi,h}^\\lambda\\big(x_k, \\nabla f(x_k)\\big)\\qquad\\mbox{with}\\qquad \\mathbf{T}_{\\phi,h}^\\lambda (x,v) := \\argmin_{y\\in\\RR^d}\\,\\, y^\\top v + \\phi(y) + \\lambda^{-1}D_h(y,x),\\vspace{-0.1cm}\n\\end{equation}","sha256_utf8":"b0074ee8aeb87fb512e45858626fb7ee0528c359305264c4a59f844883a98d13"},{"start":8589,"end":8690,"start_line":137,"end_line":137,"raw_tex":"where $D_h(y,x):=h(y)-\\nabla h(x)^\\top(y-x)-h(x)$ stands for the Bregman divergence induced by $h$. \n","sha256_utf8":"b1c021a620cc9512af44d83c56735568fdcdac234c3236e5bc92bdc269a5cc54"},{"start":9209,"end":9379,"start_line":140,"end_line":143,"raw_tex":"\\begin{equation}\n    \\label{defn:GradMap-BPG}\n    \\mathcal{G}_{\\phi,h}^\\lambda(x):= \\frac{x-\\mathbf{T}_{\\phi,h}^\\lambda\\big(x,\\nabla f(x)\\big)}{\\lambda}.  \n\\end{equation}","sha256_utf8":"95cc0544ecfee1175167689f35df178460b3d2e8812bc3789131bb83be12eab4"},{"start":24224,"end":25501,"start_line":190,"end_line":191,"raw_tex":"\\noindent\\textbf{Notations.} For $\\forall x\\in\\RR^d$, We default $\\|x\\|:=\\sqrt{x^\\top x}$  the $\\ell_2$-norm, and we denote $\\|x\\|_1:=\\sum_i|x_i|$ as the $\\ell_1$-norm. For a matrix $X\\in\\RR^{d\\times d}$, we use $\\lambda_{\\max}(X)$ and $\\lambda_{\\min}(X)$ to denote the maximal and minimal eigenvalues of $X$, respectively. And we use $\\|X\\|$ to denote the $\\ell_2$-operator norm of $X$. When $X$ is positive semidefinite, we write $X\\succeq0$, and we have $\\|X\\| = \\lambda_{\\max}(X)$ in this case. For any set $\\cX$, \nwe denote $\\mathrm{id}_{\\cX}(\\cdot)$ as the indicator function of the set.  Namely, $\\mathrm{id}_{\\cX}(x)=0$ if $x\\in\\cX$ and $\\mathrm{id}_{\\cX}(x) = +\\infty$ if $x\\notin\\cX$. We denote the interior of $\\cX$ as $\\mathrm{int}(\\cX)$ and we denote the boundary of $\\cX$ as $\\partial \\cX$. We denote $[n]:=\\{1,2,\\cdots,n\\}$. For any vector $x\\in\\RR^d$, the function $\\mathrm{sign}: \\RR^d\\to\\RR^d$ returns a sign vector of $x$. That is, for any $s=\\mathrm{sign}(x)$, its $i$-th element satisfies $s_i=1$ if $x_i\\geq0$ and $s_i=-1$ if $x_i<0$, for any $1\\leq i\\leq d$. Because many literature use the terminology $L$-smooth to denote $L$-Lipschitz continuity of the gradient, we will use ``continuously differentiable'' instead of ``smooth''  to avoid confusion. \n","sha256_utf8":"20ef8faed682a35b98a43eea5e7a7f3085f395b26f9a2e24868e2804a6c823aa"},{"start":25864,"end":26228,"start_line":197,"end_line":200,"raw_tex":"\\begin{assumption}[Smooth adaptability, \\cite{bolte2018first}]\n    \\label{assumption:L-smad}\n    Let $f$ and $h$ be twice continuously differentiable in $\\mathbb{R}^d$, and let $h$ be strictly convex. Then we assume $f$ is $L$-smooth adaptable to $h$ for some positive constant $L>0$. In other words, both  $Lh+f$ and $Lh-f$ are convex functions. \n\\end{assumption}","sha256_utf8":"514dc06f19b508b513187105fd0287dca989847e8341ac3e47c8c7e22e2a6150"},{"start":26230,"end":26513,"start_line":201,"end_line":205,"raw_tex":"\\noindent Given the twice continuous differentiability of $f$ and $h$, Assumption \\ref{assumption:L-smad} can be equivalently written  as \n\\begin{equation}\n    \\label{asmp:L-smad-alt}\n    -L\\nabla^2h(x)\\preceq \\nabla^2 f(x)\\preceq L\\nabla^2h(x),\\,\\, \\forall x\\in\\RR^d.\n\\end{equation}","sha256_utf8":"e73cba67c59ace0c75bf0b5bbcfd07b63d02424e17c3053ec145de86c3a5e299"},{"start":34136,"end":34427,"start_line":254,"end_line":257,"raw_tex":"\\begin{equation}\n    \\label{defn:condition}\n    \\mu_h(\\mathcal{X}) \\,=\\, \\min_{x\\in\\cX}\\,\\lambda_{\\min}\\big(\\nabla^2 h(x)\\big),\\quad L_h(\\mathcal{X}) \\,=\\, \\max_{x\\in\\cX}\\,\\lambda_{\\max}\\big(\\nabla^2 h(x)\\big),\\quad\\mbox{and}\\quad \\kappa_h(\\cX) := \\frac{L_h(\\cX)}{\\mu_h(\\cX)}.\n\\end{equation}","sha256_utf8":"4787b10967943f916f52c7c605e19277e0223527ee86a8c7302414633da2f1a7"},{"start":52599,"end":53007,"start_line":389,"end_line":393,"raw_tex":"\\begin{assumption}[Kernel-conditioning]\n    \\label{assumption:kernel-conditioning}\n    For any $\\delta>0$, there exists a constant $\\kappa_h^\\delta>0$ such that  $$\\sup_{\\mathcal{X}\\subseteq\\mathbb{R}^d}\\Big\\{\\kappa_h(\\mathcal{X}):\\mathrm{diam}(\\mathcal{X})\\leq \\delta\\Big\\}\\leq \\kappa_h^\\delta,$$\nwhere $\\mathrm{diam}(\\cX):=\\sup\\{\\|x-y\\|: x,y\\in\\cX\\}$ denotes the diameter of the set $\\cX$.\n\\end{assumption}","sha256_utf8":"6e19f811d05f3c7444bf253f42158c7d82af2f4b64da1ac2a0006e86eb470017"},{"start":54290,"end":54431,"start_line":397,"end_line":400,"raw_tex":"\\begin{assumption}[Strong convexity]\n    \\label{assumption:SC}\n    The kernel $h$ is $\\mu$-strongly convex for some $\\mu>0$.\n\\end{assumption}","sha256_utf8":"ca7a37d84a755acfa9f7ffe259bb9ef012bab51ebb0eb00d21871f75be02044f"},{"start":69833,"end":70097,"start_line":517,"end_line":520,"raw_tex":"\\begin{assumption}\n    \\label{assumption:L-smad-finite-sum}\n    For each $i\\in[n]$, $f_i$ is $L_i$-smooth adaptable to $h$ for some positive constant $L_{i}>0$. Denote $L:=\\sqrt{\\frac{1}{n}\\sum_i^nL_i^2}$, then $f$ is $L$-smooth adaptable to $h$. \n\\end{assumption}","sha256_utf8":"e04dfacee4978f7df4cbf6347a5043e1ea6878ab0e22099e6f758bb0b34d2723"},{"start":70098,"end":70400,"start_line":521,"end_line":521,"raw_tex":"Throughout Section \\ref{section:VR}, we will use Assumption \\ref{assumption:L-smad-finite-sum}, Assumption \\ref{assumption:kernel-conditioning}, and Assumption \\ref{assumption:SC}, and we propose a stochastic variance reduced BPG method with epoch-wise bounds  in Algorithm \\ref{algorithm:proxSARAH}. \n","sha256_utf8":"d0ed302a6dcd11f5bbbe4cae8c6814678c2f60d4ef06375ff16ac6f81c904bf1"},{"start":70402,"end":72054,"start_line":523,"end_line":549,"raw_tex":"\\begin{algorithm2e}\n\\caption{Stochastic variance reduced BPG method with epoch bounds} \n\\label{algorithm:proxSARAH}\n\\textbf{Input:} Initial point $x_{1,0}$, constant $\\delta$, epoch length $\\tau$, step size $\\eta$,  interpolation factor $\\gamma\\in(0,1]$.   \\\\\n\\For{$s=1,2,3,\\cdots,S$}{\n    Construct a convex set $\\mathcal{X}_s\\supseteq {B}(x_{s,0},\\delta/2)$ such that $\\kappa_h(\\mathcal{X}_s)\\leq \\kappa_h^\\delta$\\,.  \\\\\\textcolor{gray}{//**Proposition \\ref{proposition: Poly-kernel-condition} suggests $\\mathcal{X}_s= B\\left(x_{s,0},\\max\\left\\{\\frac{1}{2r},\\frac{\\|x_{s,0}\\|}{2r+1}\\right\\}\\right)$ for polynomial kernel**//}\n    \n    \\For{$k = 0,1,2,\\cdots,\\tau-1$}{  \n    If $k==0$, compute $v_{s,0}= \\nabla f(x_{s,0}) = \\frac{1}{n}\\sum_{i=1}^n\\nabla f_i(x_{s,0})$.\\\\\n    If $k\\geq 1$, uniformly sample a mini-batch $\\mathcal{B}_{s,k}\\subseteq[n]$  with replacement, compute  \n    \\begin{equation}\n        \\label{alg:sarah-small}\n        v_{s,k} = v_{s,k-1} + \\frac{1}{|\\mathcal{B}_{s,k}|}\\sum_{\\xi\\in\\mathcal{B}_{s,k}} \\Big(\\nabla f_\\xi(x_{s,k})-\\nabla f_\\xi(x_{s,k-1})\\Big)\\,.\n    \\end{equation}\\\\\n    \n    Denote $\\mathrm{id}_{\\cX_s}$ the indicator function of $\\cX_s$. Compute the BPG update with \n\n   \n    \\begin{equation}\n        \\label{alg:proxSARAH-update}\n        \\bar{x}_{s,k+1} =  \\Tshe(x_{s,k},v_{s,k})\\qquad\\mbox{and}\\qquad x_{s,k+1} = (1-\\gamma)x_{s,k}+\\gamma\\bar{x}_{s,k+1}\\vspace{-0.1cm}\n    \\end{equation}  \\\\ \n    \\textbf{if}\\,\\,\\,$\\mathrm{dist}(x_{s,k+1},\\partial \\cX_s) \\leq \\delta/4$\\,\\,\\,\\textbf{then}\\,\\,\\,\\textbf{break} the inner forloop.\n    } \n    Set $\\tau_s=k+1$ and $x_{s+1,0} = x_{s,\\tau_s}$. \n    } \n\\end{algorithm2e}","sha256_utf8":"0653268726e7ca2227f98c80ef830b20c9e28918440031d50522db647f30974a"},{"start":72645,"end":72843,"start_line":552,"end_line":555,"raw_tex":"\\begin{equation} \n        \\label{alg:sarah-update}\n        \\bar{x}_{s,k+1} = \\argmin_{x\\in\\mathcal{X}_s}\\,\\, \\langle v_{s,k}, x\\rangle + \\phi(x) +  \\frac{1}{\\eta}D_h(x,x_{s,k}) . \n    \\end{equation}","sha256_utf8":"1daa6876ce4dd853f31ccbfdbd9e84ed527c7a7194d4c9c693e01316db3bd63a"},{"start":73756,"end":74040,"start_line":558,"end_line":562,"raw_tex":"Define the true \\emph{restricted} primal gradient mapping in epoch $s$ as  \n\\begin{equation}\n    \\label{defn:restricted-grd-mapping}\n    \\cGshe(x_{s,k}):=\\frac{x_{s,k}-\\hat{x}_{s,k+1}}{\\eta}\\quad\\mbox{with}\\quad\\hat{x}_{s,k+1}:=\\Tshe\\big(x_{s,k},\\nabla f(x_{s,k})\\big),\n\\end{equation}","sha256_utf8":"ef6fd25a1045a355bb59d29f0a7cf5c46392fa87ba46b37d108ef184a8c3e09a"},{"start":74042,"end":74405,"start_line":563,"end_line":564,"raw_tex":"where by ``restricted'' we means this primal gradient mapping incorporates the indicator function of the set constraint $x\\in\\cX_s$.\nCompared to the $\\bar{x}_{s,k+1}$ in Algorithm \\ref{algorithm:proxSARAH}, $\\hat{x}_{s,k+1}$ is constructed with the true gradient $\\nabla f(x_{s,k})$.  For this restricted primal gradient mapping, the following lemma holds true. \n","sha256_utf8":"81d4ceef79c290158f906a6b503952da2fb95c8f1f4da4e03e3aacd35a874eab"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\n    \\label{theorem:proxSARAH-finite}\n    For any constant batch size $|\\mathcal{B}_{s,k}|=b\\in[n]$, let us set the epoch length as $\\tau = \\lceil n/b\\rceil$, step size $\\eta = \\frac{\\sqrt{2\\tau}}{\\sqrt{7\\tau}+\\sqrt{2b}}$, interpolation coefficient $\\gamma = \\frac{\\sqrt{b}}{L\\kappa_h^\\delta\\sqrt{\\tau}}$, and total epoch number $S = \\big\\lceil\\frac{16\\Delta_\\Psi}{\\tau\\gamma\\eta\\mu\\epsilon}\\big\\rceil$. Suppose the target accuracy satisfies $\\epsilon\\leq \\frac{\\delta^2}{16}\\cdot\\min\\big\\{\\frac{L^2(\\kappa_h^\\delta)^2}{b\\tau},\\frac{1}{9\\eta^2}\\big\\} = O(1/n)$ and let $x_{\\mathrm{out}}$ be uniformly randomly selected from all iterations, then there is a high probability event $\\mathcal{A}$ such that $$\\mathbb{E}\\Big[\\big\\|\\cGphe(x_{\\mathrm{out}})\\big\\|^2 \\,\\big|\\, \\mathcal{A}\\Big]\\leq 4\\epsilon\\qquad\\mbox{and}\\qquad \\mathrm{Prob}\\left(\\mathcal{A}\\right)\\geq1 - \\frac{8\\eta\\tau b\\epsilon}{L^2(\\kappa_h^\\delta)^2\\delta^2}-\\frac{4\\sqrt{\\epsilon}}{\\delta}.$$\n    In particular, $\\mathrm{Prob}\\left(\\mathcal{A}\\right) \\geq 1-O(n\\epsilon+\\sqrt{\\epsilon})\\to1$ as $\\epsilon\\to0.$ Suppose we take the batch size $b=O(n^\\alpha)$, $\\alpha\\in[0,1]$, then the total number of samples consumed is $O\\big(\\epsilon^{-1}\\cdot n^{\\max\\{\\alpha,\\frac{1}{2}\\}}\\big)$.\n\\end{theorem}","units":[{"start":86067,"end":87349,"start_line":691,"end_line":695,"raw_tex":"\\begin{theorem}\n    \\label{theorem:proxSARAH-finite}\n    For any constant batch size $|\\mathcal{B}_{s,k}|=b\\in[n]$, let us set the epoch length as $\\tau = \\lceil n/b\\rceil$, step size $\\eta = \\frac{\\sqrt{2\\tau}}{\\sqrt{7\\tau}+\\sqrt{2b}}$, interpolation coefficient $\\gamma = \\frac{\\sqrt{b}}{L\\kappa_h^\\delta\\sqrt{\\tau}}$, and total epoch number $S = \\big\\lceil\\frac{16\\Delta_\\Psi}{\\tau\\gamma\\eta\\mu\\epsilon}\\big\\rceil$. Suppose the target accuracy satisfies $\\epsilon\\leq \\frac{\\delta^2}{16}\\cdot\\min\\big\\{\\frac{L^2(\\kappa_h^\\delta)^2}{b\\tau},\\frac{1}{9\\eta^2}\\big\\} = O(1/n)$ and let $x_{\\mathrm{out}}$ be uniformly randomly selected from all iterations, then there is a high probability event $\\mathcal{A}$ such that $$\\mathbb{E}\\Big[\\big\\|\\cGphe(x_{\\mathrm{out}})\\big\\|^2 \\,\\big|\\, \\mathcal{A}\\Big]\\leq 4\\epsilon\\qquad\\mbox{and}\\qquad \\mathrm{Prob}\\left(\\mathcal{A}\\right)\\geq1 - \\frac{8\\eta\\tau b\\epsilon}{L^2(\\kappa_h^\\delta)^2\\delta^2}-\\frac{4\\sqrt{\\epsilon}}{\\delta}.$$\n    In particular, $\\mathrm{Prob}\\left(\\mathcal{A}\\right) \\geq 1-O(n\\epsilon+\\sqrt{\\epsilon})\\to1$ as $\\epsilon\\to0.$ Suppose we take the batch size $b=O(n^\\alpha)$, $\\alpha\\in[0,1]$, then the total number of samples consumed is $O\\big(\\epsilon^{-1}\\cdot n^{\\max\\{\\alpha,\\frac{1}{2}\\}}\\big)$.\n\\end{theorem}","sha256_utf8":"6f5a8b479014edf0b284fb189db40b3881e258edf6e357b01fce4df05862651d"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}[Extended descent lemma, \\cite{bolte2018first}]\n    \\label{lemma:descent-lemma}\n    Suppose $f$ and $h$ satisfy Assumption \\ref{assumption:L-smad} for some constant $L>0$, then for any $\\forall x,y\\in\\RR^d$, it holds that \n    $$|f(x)-f(y)-\\langle\\nabla f(y),x-y\\rangle|\\leq LD_h(x,y).$$\n\\end{lemma}\n\n\\begin{lemma}\n    \\label{lemma:grd-vs-grdmap}\n    For any $x,v\\in\\RR^d$ and any strictly convex kernel $h$, denote $x_\\lambda^+(v) := \\mathbf{T}_{\\phi,h}^\\lambda(x,v)$.  Let $[x,x_\\lambda^+(v)]$ be the line segment between $x_\\lambda^+(v)$ and $x$, then there exists $u\\in\\partial\\phi(x_\\lambda^+(v))$ such that  \n    \\begin{equation}\n        \\mu_h\\big([x,x_\\lambda^+(v)]\\big)\\cdot \\|x-x_\\lambda^+(v)\\|\\leq\\lambda\\|v+u\\|\\leq L_h\\big([x,x_\\lambda^+(v)]\\big)\\cdot \\|x-x_\\lambda^+(v)\\|\\,,\\nonumber\n    \\end{equation} \n    \\begin{equation}\n        \\sqrt{2\\mu_h\\big([x,x_\\lambda^+(v)]\\big)\\cdot D_h(x_\\lambda^+(v),x)}\\leq \\lambda\\|v+u\\|\\leq \\sqrt{2L_h\\big([x,x_\\lambda^+(v)]\\big)\\cdot D_h(x_\\lambda^+(v),x)}\\,\\,.\\nonumber\n    \\end{equation}   \n\\end{lemma}\n\n\\begin{proof}\n    By the optimality condition of the subproblem $x_\\lambda^+(v)=\\argmin_{y\\in\\RR^d}\\, y^\\top v + \\phi(y) + \\frac{1}{\\lambda}D_h(y,x)$, we have $0\\in \\partial\\phi(x_\\lambda^+(v))+v+\\frac{1}{\\lambda}\\nabla_y D_h(y,x)|_{y=x_\\lambda^+(v)}$. Namely, there exists  $u\\in\\partial\\phi(x_\\lambda^+(v))$ such that \n    \\begin{equation}\n        \\label{lm:grd-vs-grdmap-1}\n        \\lambda(v + u) +  \\big(\\nabla h(x_\\lambda^+(v))-\\nabla h(x)\\big)=0.\n    \\end{equation}\n    Then \\cite[Theorem 2.1.9]{nesterov2018lectures}, together with the definition of $\\mu_h(\\cdot)$ and $L_h(\\cdot)$, indicates that \n    $$\\mu_h\\big([x,x_\\lambda^+(v)]\\big)\\cdot \\|x-x_\\lambda^+(v)\\|\\leq\\|\\nabla h(x_\\lambda^+(v))-\\nabla h(x)\\|\\leq L_h\\big([x,x_\\lambda^+(v)]\\big)\\cdot \\|x-x_\\lambda^+(v)\\|.$$\n    Combining this bound with equation \\eqref{lm:grd-vs-grdmap-1} proves the first inequality of Lemma \\ref{lemma:grd-vs-grdmap}. Also observe that $\\nabla_y^2 D_h(y,x) = \\nabla^2h(y)$, we have \n    $$\\mu_h([x,x_\\lambda^+(v)])\\cdot I \\preceq \\nabla^2_y D_h(y,x) \\preceq L_h([x,x_\\lambda^+(v)])\\cdot I\\quad\\mbox{for}\\quad \\forall y\\in[x,x_\\lambda^+(v)].$$ \n    Combined with the fact that $\\nabla_y D_h(y,x)|_{y=x} = 0$, then   \\cite[Theorem 2.1.5, Eq.(2.1.10)]{nesterov2018lectures} and \\cite[Theorem 2.1.10, Eq.(2.1.24)]{nesterov2018lectures} immediately indicates \n    $$\\frac{\\left\\|\\nabla_y D_h(y,x)|_{y=x_\\lambda^+(v)}\\right\\|^2}{2L_h([x,x_\\lambda^+(v)])}\\leq D_h(x_\\lambda^+(v),x)-D_h(x,x)\\leq \\frac{\\left\\|\\nabla_y D_h(y,x)|_{y=x_\\lambda^+(v)}\\right\\|^2}{2\\mu_h([x,x_\\lambda^+(v)])}.$$\n    Then substituting $D_h(x,x)=0$ and $\\lambda(v+u) = \\nabla h(x)-\\nabla h(x_\\lambda^+(v)) = -\\nabla_z D_h(y,x)\\mid_{y=x_\\lambda^+(v)}$ to the above bound proves the second inequality of Lemma \\ref{lemma:grd-vs-grdmap}.  \n\\end{proof}\n\n\\begin{proposition}[Lipschitz-like bound] \n    \\label{proposition:Lipschitz} \n    Suppose $f$ is $L$-smooth adaptable to some KC-regular kernel $h$. Let $\\delta>0$ and let $\\cX$ be any convex set with $\\mathrm{diam}(\\cX)\\leq \\delta$. Then for any $x,y\\in\\cX$, and an arbitrary interpolation point $z_\\theta := \\theta x + (1-\\theta)y$ with $ \\theta\\in[0,1]$, we have  \n    \\begin{equation}\n        \\label{prop:Lipschitz}\n        \\frac{\\|\\nabla f(z_\\theta)-\\nabla f(y)\\|^2}{2L^2\\mu_h(\\cX)}\\leq  \\theta^2\\kappa_h^2(\\cX) D_h(x,y),\n    \\end{equation}\n    where by KC-regularity, we know the constant upper bound $\\kappa_h(\\cX)\\leq \\kappa_h^\\delta$ always hold. \n\\end{proposition}\n\n\\begin{lemma}\n    \\label{lemma:restricted-grd-mapping} \n    Let $\\cGshe(x_{s,k})$, $\\hat{x}_{s,k+1}$ and $\\bar{x}_{s,k+1}$ be defined by \\eqref{defn:restricted-grd-mapping} and \\eqref{alg:sarah-update}, respectively. Then \n    \\begin{eqnarray}\n    \\label{lm:grad-mapping} \n        \\big\\|\\cGshe(x_{s,k})\\big\\|^2 \\leq \\frac{2\\|x_{s,k}-\\bar{x}_{s,k+1}\\|^2}{\\eta^2} + \\frac{2\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h^2(\\mathcal{X}_s)},\n    \\end{eqnarray} \n    where $\\mathcal{E}_{s,k} = \\nabla f(x_{s,k})-v_{s,k}$ denotes the gradient estimation error at $x_{s,k}$.\n\\end{lemma}\n\n\\begin{lemma}\n\\label{lemma:descent-proxSARAH-finite}\n    Let $\\{x_{s,k}\\}_{k=0}^{\\tau_s}$ be the $s$-th epoch of Algorithm \\ref{algorithm:proxSARAH}, then  we have \n    \\begin{eqnarray}\n    \\Psi(x_{s,k+1})\\leq \\Psi(x_{s,k})-\\frac{\\gamma\\eta\\mu_h(\\mathcal{X}_s)}{8}\\big\\|\\cGshe(x_{s,k})\\big\\|^2\\!-\\!\\bigg(\\frac{\\gamma}{\\eta}- L\\kappa_h^\\delta\\gamma^2\\!\\bigg)\\!D_h(\\bar{x}_{s,k+1},x_{s,k}) \\!+\\!\\frac{5\\gamma\\eta\\|\\mathcal{E}_{s,k}\\|^2}{4\\mu_h(\\mathcal{X}_s)}.\\,\\nonumber\n    \\end{eqnarray} \n\\end{lemma}\n\n\\begin{lemma}\n    \\label{lemma:err-sarah} \n    Let $\\{x_{s,k}\\}_{k=0}^{\\tau_s}$ be the $s$-th epoch of Algorithm \\ref{algorithm:proxSARAH}. Given any batch size $b>0$, if we set $|\\mathcal{B}_{s,k}|=b$ for $k=1,\\cdots,\\tau_s-1$. Then conditioning on the initial point $x_{s,0}$ of the epoch, we have\n    \\begin{eqnarray}\n    \\label{lm:err-sarah-0}\n    \\mathbb{E}\\left[\\frac{\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h(\\mathcal{X}_s)}\\,\\Big|\\, x_{s,0}\\right] \\leq  \\frac{2\\gamma^2L^2(\\kappa_h^\\delta)^2}{b}\\mathbb{E}\\bigg[\\sum_{j=0}^{k-1} D_h(\\bar{x}_{s,j+1},x_{s,j})\\,\\big|\\,x_{s,0}\\bigg]\\,, \n\\end{eqnarray}\nas long as $h$ satisfies the kernel conditioning regularity assumption in $\\cX_s$. \n\\end{lemma}\n\n\\begin{proof}\n    By \\cite[Lemma 2]{pham2020proxsarah}, it is not hard to obtain that \n    \\begin{equation}\n        \\label{lm:err-sarah-0.5}\n        \\mathbb{E}\\Big[\\|\\mathcal{E}_{s,k}\\|^2\\,\\big|\\,x_{s,0}\\Big]  \\leq  \\sum_{j=0}^{k-1} \\mathbb{E}\\bigg[\\frac{1}{bn}\\sum_{i=1}^n\\|\\nabla f_i(x_{s,j+1})-\\nabla f_i(x_{s,j})\\|^2\\,\\big|\\,x_{s,0}\\bigg].\n    \\end{equation}\n    Because each $f_i$ is $L_i$-smooth adaptable to the KC-regular kernel $h$, applying the Lipschitz-like bound \\eqref{prop:Lipschitz} to each $f_i$ on   $\\bar{x}_{s,j+1}, x_{s,j}$ and their interpolation ${x}_{s,j+1} = \\gamma\\bar{x}_{s,j+1} + (1-\\gamma)x_{s,j}$ immediately yields \n    \\begin{eqnarray}\n    \\label{lm:err-sarah-1}\n    \\mathbb{E}\\left[\\frac{\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h(\\mathcal{X}_s)}\\,\\Big|\\, x_{s,0}\\right] \n    &\\leq&  \\sum_{j=0}^{k-1}\\mathbb{E}\\bigg[ \\frac{\\sum_{i=1}^nL_i^2}{n}\\cdot\\frac{2\\gamma^2  \\kappa_h^2(\\cX_s)}{b}D_h(\\bar{x}_{s,j+1},x_{s,j})\\,\\big|\\,x_{s,0}\\bigg]\\nonumber.\n\\end{eqnarray}\nUsing the fact that $L^2 = \\frac{1}{n}\\sum_{i=1}^nL_i^2$ and the fact that $\\kappa_h(\\cX_s)\\leq \\kappa_h^\\delta$ leads to \\eqref{lm:err-sarah-0}. \n\\end{proof}\n\n\\begin{lemma}\n    \\label{lemma:proxSARAH-grad-map-mid}\n    For any $b,\\tau\\in\\mathbb{Z}_+$, set $\\eta = \\frac{\\sqrt{2\\tau}}{\\sqrt{7\\tau}+\\sqrt{2b}}$, $\\gamma = \\frac{\\sqrt{b}}{L\\kappa_h^\\delta\\sqrt{\\tau}}$, and $|\\mathcal{B}_{s,k}|=b$, $\\forall s,k\\geq1$, then\n    $$\\mathbb{E}\\left[\\sum_{s=1}^S\\sum_{k=0}^{\\tau_s-1}\\frac{\\mu_h(\\mathcal{X}_s)}{8}\\big\\|\\cGshe(x_{s,k})\\big\\|^2 +D_h(\\bar{x}_{s,k+1},x_{s,k}) \\right]\\leq \\frac{\\Delta_\\Psi}{\\gamma\\eta},$$\n    where $\\Delta_\\Psi:=\\Psi(x_{1,0})-\\inf\\{\\Psi(x):x\\in\\RR^d\\}$ denotes the initial function value gap. \n\\end{lemma}\n\n\\begin{lemma}\n\\label{lemma:event-halway} \nFor any $S$ epochs  generated by Algorithm \\ref{algorithm:proxSARAH}, define the set $\\mathcal{I}_1$ and event $\\mathcal{A}_1(m_1)$ as  \n$$\\mathcal{I}_1:=\\big\\{s\\in[S] :  \\tau_s < \\tau\\big\\}\\qquad\\mbox{and}\\qquad\\mathcal{A}_1(m_1):=\\left\\{\\omega: |\\mathcal{I}_1|\\geq m_1\\right\\}.$$\nwhere $m_1>0$ is an arbitrary positive number. Then it holds that \n$$\\mathrm{Prob}\\left(\\mathcal{A}_1(m_1)\\right)\\leq \\frac{32\\gamma\\tau\\Delta_\\Psi}{\\eta\\mu\\delta^2\\cdot m_1}\\,.$$\n\\end{lemma}\n\n\\begin{proof}\nBy lemma \\ref{lemma:proxSARAH-grad-map-mid}, ignoring the restricted primal gradient mapping terms yields   \n\\begin{eqnarray}\n    \\label{lm:event-halfway-1}\n    \\frac{\\Delta_\\Psi}{\\gamma\\eta} &\\geq& \\mathbb{E}\\left[\\sum_{s=1}^S\\sum_{k=0}^{\\tau_s-1}D_h(\\bar{x}_{s,k+1},x_{s,k}) \\right]\\nonumber\\\\ \n    & \\geq & \\mathbb{E}\\left[\\frac{\\mu}{2\\gamma^2}\\sum_{s\\in\\mathcal{I}_1}\\sum_{k=0}^{\\tau_s-1}\\|{x}_{s,k+1}-x_{s,k}\\|^2 \\right] \\\\\n    & \\geq & \\frac{\\mu}{2\\gamma^2}\\cdot \\mathrm{Prob}\\left(\\mathcal{A}_1(m_1)\\right) \\cdot\\mathbb{E}\\left[\\sum_{s\\in\\mathcal{I}_1}\\sum_{k=0}^{\\tau_s-1}\\|{x}_{s,k+1}-x_{s,k}\\|^2\\,\\Big|\\, \\mathcal{A}_1(m_1) \\right] \\nonumber,\n\\end{eqnarray}\nwhere the last inequality is due to the fact that   \n$$\\EE[X] = \\mathrm{Prob}(\\cA)\\cdot\\EE\\big[X\\mid\\cA\\big] + \\mathrm{Prob}(\\cA^c)\\cdot\\EE\\big[X\\mid\\cA^c\\big]\\geq \\mathrm{Prob}(\\cA)\\cdot\\EE\\big[X\\mid\\cA\\big]$$\nfor any non-negative random variable $X\\geq0$ and any event $\\cA$. Therefore, conditioning on the event $\\mathcal{A}_1(m_1)$, for any epoch $s\\in\\mathcal{I}_1$, we will have $\\|x_{s,\\tau_s}-x_{s,0}\\|\\geq\\frac{\\delta}{4}$ because of Line 3 and Line 9 of Algorithm \\ref{algorithm:proxSARAH}. Consequently, for $\\forall s\\in\\mathcal{I}_1$, the triangle inequality and the arithmetic inequality indicate that\n\\begin{equation}\n    \\label{lm:event-halfway-2}\n    \\frac{\\delta}{4\\tau_s}\\leq \\frac{\\|x_{s,\\tau_s}-x_{s,0}\\|}{\\tau_s}\\leq\\frac{\\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|}{\\tau_s}\\leq  \\sqrt{\\frac{\\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|^2}{\\tau_s}}\\,.\n\\end{equation} \nBecause $\\tau_s\\leq\\tau$ always holds, we have \n\\begin{align}\n    \\label{lm:event-halfway-3}\n    \\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|^2  \\geq  \\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|^2\\geq\\frac{\\delta^2}{16\\tau_s}\\geq\\frac{\\delta^2}{16\\tau}\\,.\n\\end{align}\nNote that the above inequalities hold w.p. 1 conditioning on $\\cA_1(m_1)$. Combined with \\eqref{lm:event-halfway-1}, we have \n$$\\frac{\\Delta_\\Psi}{\\gamma\\eta} \\geq\\frac{\\mu}{2\\gamma^2}\\cdot \\mathrm{Prob}\\left(\\mathcal{A}_1(m_1)\\right)\\cdot\\frac{m_1\\delta^2}{16\\tau}.$$\nRearranging the terms completes the proof. \n\\end{proof}\n\n\\begin{lemma}\n    \\label{lemma:event-grad-map}\n    For any $S$ epochs generated by Algorithm \\ref{algorithm:proxSARAH}, define the set $\\mathcal{I}_2$ and event $\\mathcal{A}_2(m_2)$ as\n    $$\\mathcal{I}_2:=\\left\\{(s,k) :  \\hat{x}_{s,k+1}\\in\\partial \\mathcal{X}_s, 1\\leq s\\leq S, 0\\leq k\\leq \\tau_s-1\\right\\}\\quad\\mbox{and}\\quad\\mathcal{A}_2(m_2):=\\big\\{\\omega : |\\mathcal{I}_2|\\geq m_2\\big\\}$$\n    where  $m_2>0$ is an arbitrary positive number. Then it holds that \n    $$\\mathrm{Prob}\\left(\\mathcal{A}_2(m_2)\\right)\\leq \\frac{128\\eta\\Delta_\\Psi}{\\gamma\\mu\\delta^2\\cdot m_2}\\,.$$\n\\end{lemma}\n\n\\begin{proof}\nSimilar to the proof of Lemma \\ref{lemma:event-halway}, ignoring the Bregman divergence terms in Lemma \\ref{lemma:proxSARAH-grad-map-mid} yields   \n\\begin{eqnarray}\n    \\label{lm:event-grad-map-1}\n    \\frac{\\Delta_\\Psi}{\\gamma\\eta} & \\geq & \\mathbb{E}\\left[\\sum_{s=1}^S\\sum_{k=0}^{\\tau_s-1}\\frac{\\mu}{8}\\big\\|\\cGshe(x_{s,k})\\big\\|^2\\right]\\nonumber\\\\\n    & = & \\frac{\\mu}{8\\eta^2}\\mathbb{E}\\left[\\sum_{s=1}^S\\sum_{k=0}^{\\tau_s-1}\\|\\hat{x}_{s,k+1}-x_{s,k}\\|^2\\right]\\\\\n    & \\geq & \\frac{\\mu}{8\\eta^2}\\mathbb{E}\\Bigg[\\sum_{(s,k)\\in\\mathcal{I}_2} \\|\\hat{x}_{s,k+1}-x_{s,k}\\|^2\\Bigg]\\nonumber\\\\\n    & \\geq & \\frac{\\mu}{8\\eta^2}\\cdot\\mathrm{Prob}\\left(\\mathcal{A}_2(m_2)\\right)\\cdot\\mathbb{E}\\Bigg[\\sum_{(s,k)\\in\\mathcal{I}_2}\\|\\hat{x}_{s,k+1}-x_{s,k}\\|^2\\,\\Big|\\,\\mathcal{A}_2(m_2)\\Bigg]\\nonumber.\n\\end{eqnarray}\nNote that for  $\\forall(s,k)\\in\\mathcal{I}_2$, we have $\\mathrm{dist}(x_{s,k},\\partial\\cX_s)\\geq\\delta/4$ while $\\hat{x}_{s,k+1}\\in\\partial\\cX_s$. Consequently, one must have $\\|\\hat{x}_{s,k+1}-{x}_{s,k}\\|\\geq \n\\frac{\\delta}{4}$. Conditioning on the event $\\cA_2(m_2)$, we have $|\\mathcal{I}_2|\\geq m_2$ and \n$$\\mathbb{E}\\Bigg[\\sum_{(s,k)\\in\\mathcal{I}_2}\\|\\hat{x}_{s,k+1}-x_{s,k}\\|^2\\,\\Big|\\,\\mathcal{A}_2(m_2)\\Bigg]\\geq \\frac{m_2\\delta^2}{16}.$$\nSubstitute this bound to \\eqref{lm:event-grad-map-1} proves lemma.  \n\\end{proof}\n\n\\begin{proof}\nBy lemma \\ref{lemma:proxSARAH-grad-map-mid}, ignoring the restricted primal gradient mapping terms yields  \n\\begin{eqnarray} \n\\frac{\\Delta_\\Psi}{\\gamma\\eta} \\geq\n\\mathbb{E}\\bigg[\\sum_{s=1}^{S}\\sum_{k=0}^{\\tau_s-1}D_h(\\bar{x}_{s,k+1},x_{s,k})\\bigg] \n\\geq \\frac{\\mu}{2\\gamma^2}\\mathbb{E}\\bigg[\\sum_{s=1}^{S}\\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|^2\\bigg]\\,. \\nonumber\n\\end{eqnarray}\n\nBy expanding the expectation over all possible $\\mathcal{I}_1, \\mathcal{I}_2,$ and $\\{\\tau_s\\}$, we have for all $m_1,m_2>0$ that \n\\begin{eqnarray} \n\\label{thm:proxSARAH-finite-1}\n\\frac{2\\gamma\\Delta_\\Psi}{\\mu\\eta} \\!\\!\\!\\!& \\geq & \\!\\!\\!\\!\\mathbb{E}\\bigg[\\sum_{s=1}^{S}\\sum_{k=0}^{\\tau_s-1}\\!\\|x_{s,k+1}\\!-\\!x_{s,k}\\|^2\\bigg] \\\\\n& \\geq &\\!\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\!\\!\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\mathbb{E}\\bigg[\\!\\sum_{s=1}^S\\!\\sum_{k=0}^{\\tau_s-1}\\!\\|x_{s,k+1}-x_{s,k}\\|^2 \\,\\Big|\\, \\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\!\\bigg]\\nonumber\\\\\n& \\geq &\\!\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\!\\!\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\mathbb{E}\\bigg[\\!\\sum_{s\\in\\mathcal{I}_1}\\!\\!\\sum_{k=0}^{\\tau_s-1}\\!\\|x_{s,k+1}-x_{s,k}\\|^2 \\,\\Big|\\, \\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\!\\bigg]\\nonumber \n\\end{eqnarray}\nNote that for $\\forall s\\in\\mathcal{I}_1$, according to the discussion in Lemma \\ref{lemma:event-grad-map}, we have  $\\|x_{s,\\tau_s}-x_{s,0}\\|  \\geq \\delta/4.$\nRepeating the analysis of \\eqref{lm:event-halfway-2} and \\eqref{lm:event-halfway-3} yields $\\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|^2  \\geq \\frac{\\delta^2}{16\\tau_s}$, which always holds true. Then substituting this lower bound to \\eqref{thm:proxSARAH-finite-1} gives \n\\begin{equation}\n    \\label{thm:proxSARAH-finite-2}\n    \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\cdot\\sum_{s\\in\\mathcal{I}_1}\\tau_s^{-1}\\leq\\frac{32\\gamma\\Delta_\\Psi}{\\mu\\eta\\delta^2}\\,.\n\\end{equation} \nDefine the events  \n\\begin{equation}\n    \\label{defn:event-A3}\n    \\mathcal{A}_3:=\\big\\{\\omega: \\mbox{the index of } x_{\\mathrm{out}}\\mbox{ is from }\\mathcal{I}_2^c\\big\\}\\quad\\mbox{and}\\quad \\mathcal{A}:=\\mathcal{A}_1^c(m_1)\\cap\\mathcal{A}_2^c(m_2)\\cap\\mathcal{A}_3.\n\\end{equation} \nThen we have\n\\begin{eqnarray}\n    \\label{thm:proxSARAH-finite-33}\n    \\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\mathrm{Prob}\\left(\\mathcal{A}\\right)\\!\\!\\! &=&\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\cdot\\mathrm{Prob}\\big(\\cA_3\\mid\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\\\\n    &=&\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\cdot\\left(1-\\frac{|\\mathcal{I}_2|}{(S-|\\mathcal{I}_1|)\\tau+\\sum_{s\\in\\mathcal{I}_1}\\tau_s}\\right)\\nonumber\\\\\n    & \\geq &\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\cdot\\left(1-\\frac{m_2}{(S-|\\mathcal{I}_1|)\\tau+\\sum_{s\\in\\mathcal{I}_1}\\tau_s}\\right).\\nonumber\n\\end{eqnarray}\nNote that for any positive numbers $y_1,\\cdots,y_S>0$,  the arithmetic-harmonic inequality states that $$\\frac{y_1+y_2+\\cdots+y_S}{S}\\geq\\frac{S}{\\frac{1}{y_1}+\\frac{1}{y_2}+\\cdots+\\frac{1}{y_S}}\\,.$$ \nApplying this inequality gives \n$$\\frac{1}{\\tau} + \\frac{\\sum_{s\\in\\mathcal{I}_1}\\tau_s^{-1}}{S}= \\frac{(S-|\\mathcal{I}|_1)\\tau^{-1} + \\sum_{s\\in\\mathcal{I}_1}\\tau_s^{-1}}{S}\\geq \\frac{S}{(S-|\\mathcal{I}|_1)\\tau+ \\sum_{s\\in\\mathcal{I}_1}\\tau_s}\\,,$$\nwhich implies that \n$$1-\\frac{m_2}{(S-|\\mathcal{I}_1|)\\tau+\\sum_{s\\in\\mathcal{I}_1}\\tau_s}\\geq 1-\\frac{m_2}{S\\tau}-\\frac{m_2}{S^2}\\sum_{s\\in\\mathcal{I}_1}\\tau_s^{-1}\\,.$$\nSubstituting this bound to \\eqref{thm:proxSARAH-finite-33} and setting $m_1^* = S/4$ yields\n\\begin{eqnarray}\n    \\label{thm:proxSARAH-finite-44}\n    \\!\\!\\!\\mathrm{Prob}\\left(\\mathcal{A}\\right) \\!\\!\\!\\!& \\geq &\\!\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1^*}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\cdot\\left(1-\\frac{m_2}{S\\tau}-\\frac{m_2}{S^2}\\sum_{s\\in\\mathcal{I}_1}\\tau_s^{-1}\\right)\\nonumber\\\\\n    & \\overset{(i)}{\\geq} &\\!\\!\\!\\! \\left(1-\\frac{m_2}{S\\tau}\\right)\\mathrm{Prob}\\left(\\mathcal{A}_1^c(m_1^*)\\cap\\mathcal{A}_2^c(m_2)\\right) - \\frac{m_2}{S^2}\\cdot\\frac{32\\gamma\\Delta_\\Psi}{\\mu\\eta\\delta^2}\\\\\n    & \\geq & \\!\\!\\!\\! \\left(1-\\frac{m_2}{S\\tau}\\right)\\big(1-\\mathrm{Prob}\\left(\\mathcal{A}_1(m_1^*)\\right)-\\mathrm{Prob}\\left(\\mathcal{A}_2(m_2)\\right)\\big) - \\frac{m_2}{S^2}\\cdot\\frac{32\\gamma\\Delta_\\Psi}{\\mu\\eta\\delta^2}\\nonumber\\\\\n    & \\overset{(ii)}{\\geq} & \\!\\!\\!\\! 1 - \\frac{8\\eta\\tau b\\epsilon}{L^2(\\kappa_h^\\delta)^2\\delta^2} - \\frac{128\\eta\\Delta_\\Psi}{\\gamma\\mu\\delta^2\\cdot m_2} - \\frac{m_2}{S\\tau} - \\frac{m_2}{S^2}\\cdot\\frac{32\\gamma\\Delta_\\Psi}{\\mu\\eta\\delta^2}\\nonumber\\\\\n    & \\overset{(iii)}{\\geq} & \\!\\!\\!\\! 1 - \\frac{8\\eta\\tau b\\epsilon}{L^2(\\kappa_h^\\delta)^2\\delta^2} - \\frac{128\\eta\\Delta_\\Psi}{m_2\\gamma\\mu\\delta^2} - \\frac{9m_2}{8S\\tau}\\nonumber\n\\end{eqnarray}\nwhere (i) is due to \\eqref{thm:proxSARAH-finite-2} and the fact that \n\\begin{equation}\n    \\label{thm:proxSARAH-finite-55}\n    \\sum_{|\\mathcal{I}_1|<m_1^*}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big) = \\mathrm{Prob}\\big(\\mathcal{A}_1^c(m_1^*)\\cap\\mathcal{A}_2^c(m_2)\\big)\n\\end{equation}\n(ii) is due to Lemma \\ref{lemma:event-halway} with $m_1^*=S/4$ and $S = \\big\\lceil\\frac{16\\Delta_\\Psi}{\\tau\\gamma\\eta\\mu\\epsilon}\\big\\rceil$: \n$$\\mathrm{Prob}\\big(\\mathcal{A}_1(S/4)\\big)\\leq \\frac{32\\gamma\\tau\\Delta_\\Psi}{\\mu\\delta^2S/4} = \\frac{16\\Delta_\\Psi}{S\\tau\\gamma\\eta\\mu}\\cdot\\frac{8\\gamma^2\\tau^2\\eta}{\\delta^2} \\leq \\frac{8\\tau b\\epsilon}{L^2(\\kappa_h^\\delta)^2\\delta^2},$$\nand (iii) is because we require $\\epsilon\\leq \\frac{\\delta^2}{16}\\cdot\\min\\left\\{\\frac{L^2(\\kappa_h^\\delta)^2}{b\\tau},\\frac{1}{9\\eta^2}\\right\\} = O(1/n)$ such that \n$$\\frac{m_2}{S^2}\\cdot\\frac{32\\gamma\\Delta_\\Psi}{\\mu\\eta\\delta^2} = \\frac{m_2}{S\\tau}\\cdot\\frac{16\\Delta_\\Psi}{S\\tau\\gamma\\eta\\mu}\\cdot\\frac{2\\gamma^2\\tau^2}{\\delta^2}\\leq \\frac{m_2}{8S\\tau}$$\nTherefore, to maximize the above probability, we can choose $m_2^*=\\sqrt{\\frac{S\\tau\\eta\\Delta_\\Psi}{\\gamma\\mu}}\\cdot\\frac{32}{3\\delta}$ such that\n$$\\frac{128\\eta\\Delta_\\Psi}{m_2^*\\gamma\\mu\\delta^2} + \\frac{9m_2^*}{8S\\tau} = 2\\sqrt{\\frac{128\\eta\\Delta_\\Psi}{m_2^*\\gamma\\mu\\delta^2} \\cdot  \\frac{9m_2^*}{8S\\tau}} = \\sqrt{\\frac{\\eta\\Delta_\\Psi}{\\gamma\\mu S\\tau}}\\cdot\\frac{24}{\\delta}\\leq \\frac{6\\eta\\sqrt{\\epsilon}}{\\delta}\\leq \\frac{4\\sqrt{\\epsilon}}{\\delta}\\,, $$\nwhere the last inequality is due to the fact that $\\eta = \\frac{\\sqrt{2\\tau}}{\\sqrt{7\\tau}+\\sqrt{2b}}\\leq\\sqrt{2/7}.$ Combining all the above discussion, we can conclude that  \n$$\\mathrm{Prob}\\left(\\mathcal{A}\\right)\\geq1 - \\frac{8\\eta\\tau b\\epsilon}{L^2(\\kappa_h^\\delta)^2\\delta^2}-\\frac{4\\sqrt{\\epsilon}}{\\delta} = 1-O(n\\epsilon+\\sqrt{\\epsilon})$$\nwhen taking $m_1^* = S/4$ and $m_2^* = \\sqrt{\\frac{S\\tau\\eta\\Delta_\\Psi}{\\gamma\\mu}}\\cdot\\frac{32}{3\\delta}.$ Note that the requirement on $\\epsilon$ further implies that $\\mathrm{Prob}(\\cA_1(m_1^*))\\leq 1/2$ and $\\mathrm{Prob}(\\cA_2(m_2^*))\\leq 1/4$, and hence $\\mathrm{Prob}\\big(\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)\\big)\\geq1/4$. By ignoring the Bregman divergence terms of Lemma \\ref{lemma:proxSARAH-grad-map-mid}, we obtain\n\\begin{eqnarray*}\n    \\frac{\\Delta_\\Psi}{\\gamma\\eta} &\\geq& \\frac{\\mu}{8} \\mathbb{E}\\bigg[\\sum_{s=1}^S\\sum_{k=0}^{\\tau_s-1}\\big\\|\\cGshe(x_{s,k})\\big\\|^2\\bigg]  \\geq \\frac{\\mu}{32}\\mathbb{E}\\bigg[\\sum_{(s,k)\\in\\mathcal{I}_2^c}\\big\\|\\cGshe(x_{s,k})\\big\\|^2\\,\\Big|\\,\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)\\bigg] \\nonumber.\n\\end{eqnarray*}\nConditioning on $\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)$, regardless of the random sets $\\mathcal{I}_1, \\mathcal{I}_2$ and $\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}$, we have  \n\\begin{eqnarray}\n    |\\mathcal{I}_2^c| = (S-|\\mathcal{I}_1|)\\tau+\\sum_{s\\in\\mathcal{I}_1}\\tau_s - |\\mathcal{I}_2|\\geq  \\frac{3S\\tau}{4} - m_2^*\\geq S\\tau\\Big(\\frac{3}{4}-\\frac{8\\eta\\sqrt{\\epsilon}}{3\\delta}\\Big) \\geq \\frac{S\\tau}{2}.\\nonumber\n\\end{eqnarray}\nwhere the second last inequality is because $|\\mathcal{I}_2|\\leq m_2^*\\leq \\frac{8S\\tau\\eta\\sqrt{\\epsilon}}{3\\delta}$, and the last inequality is because  $\\frac{8\\eta\\sqrt{\\epsilon}}{3\\delta}\\leq 2/9< 1/4$ since we require $\\epsilon\\leq\\frac{\\delta^2}{16}\\cdot\\frac{1}{9\\eta^2}$. As a result, \n\\begin{eqnarray}\n\\label{thm:proxSARAH-finite-66}\n    \\frac{\\Delta_\\Psi}{\\gamma\\eta} &\\geq& \\frac{\\mu}{32}\\cdot\\frac{S\\tau}{2}\\cdot\\mathbb{E}\\bigg[\\frac{|\\mathcal{I}_2^c|}{S\\tau/2}\\cdot\\frac{\\sum_{(s,k)\\in\\mathcal{I}_2^c}\\big\\|\\cGshe(x_{s,k})\\big\\|^2}{|\\mathcal{I}_2^c|}\\,\\Big|\\,\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)\\bigg] \\\\\n    & \\geq & \\frac{\\mu S\\tau}{64}\\mathbb{E}\\bigg[\\frac{\\sum_{(s,k)\\in\\mathcal{I}_2^c}\\big\\|\\cGshe(x_{s,k})\\big\\|^2}{|\\mathcal{I}_2^c|}\\,\\Big|\\,\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)\\bigg] \\nonumber\\\\\n    & = & \\frac{\\mu S\\tau}{64}\\mathbb{E}\\Big[\\big\\|\\cGphe(x_{\\mathrm{out}})\\big\\|^2\\,\\big|\\,\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)\\cap\\cA_3\\Big]\\,, \\nonumber\n\\end{eqnarray}\nwhere the last equality is due to the definition of $x_{\\mathrm{out}}$, $\\cA_3$, and the fact that $\\cGshe(\\cdot)$ coincides with $\\cGphe(\\cdot)$ in $\\mathcal{I}_2^c$. As a result, we have\n$$\\mathbb{E}\\Big[\\big\\|\\cGphe(x_{\\mathrm{out}})\\big\\|^2\\,\\big|\\,\\cA\\Big]\\leq \\frac{64\\Delta_\\Psi}{\\gamma\\eta\\mu\\tau S}\\leq 4\\epsilon.$$\nGiven the choice of $S$, $\\gamma$, $\\eta$, and the fact that $\\tau = \\lceil n/b\\rceil$, the total sample complexity will be \n$$S(n+b\\tau) = (n+b\\tau)\\cdot\\bigg\\lceil\\frac{16\\Delta_\\Psi}{\\tau\\gamma\\eta\\mu\\epsilon}\\bigg\\rceil = O\\left(\\frac{L\\kappa_h^\\delta\\Delta_\\Psi\\sqrt{n}}{\\mu\\epsilon}\\cdot\\Big(1+\\sqrt{b/\\tau}\\Big)\\right),$$\nwhich indicates an $O(\\sqrt{n}\\epsilon^{-1})$ sample complexity for all $b\\leq O(\\tau)$ (or equivalently, $b\\leq O(\\lceil\\sqrt{n}\\rceil)$). If larger batch size $b = \\lceil n^\\alpha\\rceil$ with $\\alpha\\in(1/2,1]$ is taken, then we obtain an $O(n^\\alpha\\epsilon^{-1})$ complexity. \n\\end{proof}\n\n\\begin{lemma}[Three-Point Property of Tseng \\cite{tseng2008accelerated}]\n    \\label{lemma:Tseng-3point}\n    Let $\\phi(x)$ be a convex function, and let $D_h(\\cdot,\\cdot)$ be the Bregman distance for $h(\\cdot)$. For a given vector $z$, let\n    $z_+:=\\argmin_{x\\in Q} \\phi(x) + D_h(x,z).$\n    Then \n    $$\\phi(x)+D_h(x,z)\\geq\\phi(z_+) + D_h(z_+,z) + D_h(x,z_+), \\quad\\forall x\\in Q.$$\n\\end{lemma}\n\n\\begin{lemma}[Lemma 2 finite-sum case of \\cite{pham2020proxsarah}]\n\\label{lemma:proxSARAH-finite-sum}\n    Let $v_{s,k}$ be generated by \\eqref{alg:sarah-small}, suppose $|\\mathcal{B}_{s,k}| = b$ and the sampled index are uniformly randomly picked from $[n]$ with replacement, then  \n    \\begin{eqnarray}\n        \\mathbb{E}\\Big[\\|\\nabla f(x_{s,k})-v_{s,k}\\|^2\\,\\big|\\,x_{s,0}\\Big] \\leq \\sum_{j=0}^{k-1} \\mathbb{E}\\Big[\\|v_{s,j+1}-v_{s,j}\\|^2-\\|\\nabla f(x_{s,j+1})-\\nabla f(x_{s,j})\\|^2\\,\\big|\\, x_{s,0}\\Big], \\nonumber \n    \\end{eqnarray}\n    where the expectation term of $\\|v_{s,j+1}-v_{s,j}\\|^2$ satisfies\n    \\begin{eqnarray}\n        \\mathbb{E}\\Big[\\|v_{s,j+1}-v_{s,j}\\|^2\\,\\big|\\,x_{s,0}\\Big] \\leq  \\mathbb{E}\\bigg[\\|\\nabla f(x_{s,j+1})-\\nabla f(x_{s,j})\\|^2 + \\frac{1}{bn}\\sum_{i=1}^n\\|\\nabla f_i(x_{s,j+1})-\\nabla f_i(x_{s,j})\\|^2\\,\\big|\\,x_{s,0}\\bigg].  \\nonumber\n    \\end{eqnarray}  \n    In particular, we have slightly modified the second inequality to suit our analysis. \n\\end{lemma}\n\n\\begin{lemma}\n    \\label{lemma:B.1}\n    Let $A$ be a symmetric matrix, and let $B\\succeq0$ be positive semidefinite matrix. Then $-B\\preceq A\\preceq B$ indicates that $\\|A\\|\\leq \\|B\\|$.\n\\end{lemma}\n\n\\begin{proof}\n    For any symmetric but not semidefinite matrix $A$, it is easy to verify that \n    \\begin{equation}\n        \\label{lm:B.1-1}\n        \\|A\\| = \\max\\big\\{|\\lambda_{\\max}(A)|,|\\lambda_{\\min}(A)|\\big\\}.\n    \\end{equation}\n    Let $u\\neq0$ be the eigenvector of $A$ associated with the maximum eigenvalue.  Then by Rayleigh’s principle for maximum eigenvalue, $B\\succeq A$ indicates that $B-A\\succeq0$ and hence \n    \\begin{equation}\n        \\label{lm:B.1-2}\n        0\\leq \\frac{u^\\top(B-A)u}{u^\\top u} \\leq \\max_{x\\neq0}\\left\\{\\frac{x^\\top Bx}{x^\\top x}\\right\\} - \\frac{u^\\top(B-A)u}{u^\\top u} = \\lambda_{\\max}(B) - \\lambda_{\\max}(A). \n    \\end{equation}\n    Similarly, $A\\succeq-B$ indicates that $B-(-A)\\succeq0$, then \\eqref{lm:B.1-2} immediately gives\n    \\begin{equation}\n        \\label{lm:B.1-3}\n        0\\leq  \\lambda_{\\max}(B) - \\lambda_{\\max}(-A) = \\lambda_{\\max}(B) + \\lambda_{\\min}(A).\n    \\end{equation}\n    Combining \\eqref{lm:B.1-1}-\\eqref{lm:B.1-3} and the fact that $\\|B\\| = \\lambda_{\\max}(B)$ for p.s.d. matrix proves the lemma.\n\\end{proof}\n\n\\begin{proof}\n     By Lemma \\ref{lemma:B.1} and the assumption that $f$ is $L$-smooth adaptable to $h$, we know \n     $$\\max\\left\\{\\|\\nabla^2f(x)\\|: x\\in\\cX\\right\\} \\leq \\max\\left\\{L\\|\\nabla^2h(x)\\|: x\\in\\cX\\right\\} = L\\cdot L_h(\\cX).$$\n     As $\\cX$ is convex, the line segment $[x,y]\\subseteq\\cX$ and hence \n     $$\\|\\nabla f(z_\\theta)-\\nabla f(y)\\|^2\\leq L^2L_h^2(\\cX)\\|z_\\theta-y\\|^2 = L^2\\theta^2 L_h^2(\\cX)\\|x-y\\|^2.$$\n     Combined with    the fact that $$D_h(x,y)\\geq\\frac{\\mu_h([x,y])}{2}\\|x-y\\|^2\\geq  \\frac{\\mu_h(\\cX)}{2}\\|x-y\\|^2,$$\n     we obtain\n     $$\\frac{\\|\\nabla f(z_\\theta)-\\nabla f(y)\\|^2}{2L^2\\mu_h(\\cX)} \\leq \\theta^2\\cdot\\frac{L_h^2(\\cX)}{\\mu_h^2(\\cX)}\\cdot\\frac{\\mu_h(\\cX)}{2}\\|x-y\\|^2\\leq \\theta^2\\kappa_h^2(\\cX)D_h(x,y),$$\n     where KC-regularity guarantees that $\\kappa_h(\\cX)$ is always upper bounded by $\\kappa_h^\\delta$.\n\\end{proof}\n\n\\begin{proof}\n    Denote $\\widetilde{\\mathcal{G}}=\\frac{x_{s,k}-\\bar{x}_{s,k+1}}{\\eta}$, then $\\|\\cGshe(x_{s,k})-\\widetilde{\\mathcal{G}}\\| = \\frac{1}{\\eta}\\|\\bar{x}_{s,k+1}-\\hat{x}_{s,k+1}\\|$ holds by definition. \n    By the optimality of $\\hat{x}_{s,k+1}$ and $\\bar{x}_{s,k+1}$ for the corresponding subproblems, Tseng's three point property (Lemma \\ref{lemma:Tseng-3point}) indicates that\n    \\begin{eqnarray*}\n        &&\\langle v_{s,k}, \\bar{x}_{s,k+1}\\rangle + \\phi(\\bar{x}_{s,k+1}) + \\frac{D_h(\\bar{x}_{s,k+1},x_{s,k})}{\\eta} + \\frac{D_h(\\hat{x}_{s,k+1},\\bar{x}_{s,k+1})}{\\eta}\\qquad\\,\\,\\,\\\\\n        &\\leq& \\langle v_{s,k}, \\hat{x}_{s,k+1}\\rangle + \\phi(\\hat{x}_{s,k+1})+ \\frac{D_h(\\hat{x}_{s,k+1},x_{s,k})}{\\eta} \n    \\end{eqnarray*} \n    and \n    \\begin{eqnarray*}\n        && \\langle \\nabla f(x_{s,k}), \\hat{x}_{s,k+1}\\rangle + \\phi(\\hat{x}_{s,k+1})+\\frac{D_h(\\hat{x}_{s,k+1},x_{s,k})}{\\eta} + \\frac{D_h(\\bar{x}_{s,k+1},\\hat{x}_{s,k+1})}{\\eta}\\\\\n        &\\leq& \\langle \\nabla f(x_{s,k}), \\bar{x}_{s,k+1}\\rangle + \\phi(\\bar{x}_{s,k+1})+\\frac{D_h(\\bar{x}_{s,k+1},x_{s,k})}{\\eta}\n    \\end{eqnarray*} \n    Summing up the two inequalities and applying Lemma \\ref{lemma:grd-vs-grdmap} gives \n    \\begin{align} &\\frac{\\mu_h([\\bar{x}_{s,k+1},\\hat{x}_{s,k+1}])}{\\eta}\\cdot\\|\\bar{x}_{s,k+1}-\\hat{x}_{s,k+1}\\|^2 \\leq \\frac{D_h(\\bar{x}_{s,k+1},\\hat{x}_{s,k+1})}{\\eta} +  \\frac{D_h(\\hat{x}_{s,k+1},\\bar{x}_{s,k+1})}{\\eta}\\nonumber\\\\\n    &\\qquad\\qquad\\qquad\\qquad\\qquad\\,\\,\\,\\, \\leq  \\langle v_{s,k}-\\nabla f(x_{s,k}), \\hat{x}_{s,k+1}-\\bar{x}_{s,k+1}\\rangle \\leq  \\|v_{s,k}-\\nabla f(x_{s,k})\\|\\cdot\\|\\hat{x}_{s,k+1}-\\bar{x}_{s,k+1}\\|\\nonumber.\n    \\end{align} \n    Hence $\\|\\hat{x}_{s,k+1}-\\bar{x}_{s,k+1}\\|\\leq \\frac{\\eta\\|v_{s,k}-\\nabla f(x_{s,k})\\|}{\\mu_h([\\bar{x}_{s,k+1},\\hat{x}_{s,k+1}])}$, and $\\big\\|\\cGshe(x_{s,k})-\\widetilde{\\mathcal{G}}\\big\\|\\leq\\frac{\\|v_{s,k}-\\nabla f(x_{s,k})\\|}{\\mu_h([\\bar{x}_{s,k+1},\\hat{x}_{s,k+1}])}$. As a result, \n    \\begin{eqnarray*}\n        \\big\\|\\cGshe(x_{s,k})\\big\\|^2 \\leq \\left(\\big\\|\\widetilde{\\mathcal{G}}\\big\\|+\\big\\|\\cGshe(x_{s,k})-\\widetilde{\\mathcal{G}}\\big\\|\\right)^2\\leq  \\frac{2\\|x_{s,k}-\\bar{x}_{s,k+1}\\|^2}{\\eta^2} + \\frac{2\\|v_{s,k}-\\nabla f(x_{s,k})\\|^2}{\\mu_h^2(\\mathcal{X}_s)},\n    \\end{eqnarray*} \n    where the last inequality is because $(a+b)^2\\leq 2a^2+2b^2$ and $\\mu_h^2\\big([\\bar{x}_{s,k+1},\\hat{x}_{s,k+1}]\\big)\\geq\\mu_h^2(\\cX_s)$. \n\\end{proof}\n\n\\begin{proof} \n    First of all, by the update rule of $x_{s,k+1}$, we have   \n    \\begin{eqnarray}\n    \\label{lm:descent-proxSARAH-finite-1}\n        \\Psi(x_{s,k+1}) \n        \\!\\!\\!& = & \\!\\!\\!f\\big(x_{s,k} + \\gamma (\\bar{x}_{s,k+1}-x_{s,k})\\big) + \\phi\\big((1-\\gamma)x_{s,k} + \\gamma\\bar{x}_{s,k+1}\\big) \\\\\n        &\\overset{(i)}{\\leq}& \\!\\!\\!f(x_{s,k}) + \\gamma\\langle\\nabla f(x_{s,k}),\\bar{x}_{s,k+1}\\!-x_{s,k}\\rangle \\!+\\! LD_h\\left(x_{s,k+1},x_{s,k}\\right) \\!+\\! (1-\\gamma)\\phi(x_{s,k})+\\gamma\\phi(\\bar{x}_{s,k+1})\\nonumber\\\\\n        & \\overset{(ii)}{\\leq} & \\!\\!\\!\\Psi(x_{s,k}) + L\\kappa_h^\\delta\\gamma^2D_h(\\bar{x}_{s,k+1},x_{s,k}) + \\gamma\\langle\\mathcal{E}_{s,k}+v_{s,k},\\bar{x}_{s,k+1}-x_{s,k}\\rangle  + \\gamma\\left(\\phi(\\bar{x}_{s,k+1})-\\phi(x_{s,k})\\right)\\nonumber\\\\\n        &\\overset{(iii)}{\\leq}&\\!\\!\\! \\Psi(x_{s,k}) + L\\kappa_h^\\delta\\gamma^2D_h(\\bar{x}_{s,k+1},x_{s,k}) + \\frac{\\gamma\\eta\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h(\\mathcal{X}_s)} + \\frac{\\gamma\\mu_h(\\mathcal{X}_s)}{4\\eta} \\|\\bar{x}_{s,k+1}-x_{s,k}\\|^2\\nonumber\\\\\n        & & \\!\\!\\!-\\frac{\\gamma}{\\eta}D_h(\\bar{x}_{s,k+1},x_{s,k}) - \\frac{\\gamma}{\\eta}D_h(x_{s,k},\\bar{x}_{s,k+1})\\nonumber\\\\\n        &\\leq & \\!\\!\\!\\Psi(x_{s,k}) - \\left(\\frac{\\gamma}{\\eta}- L\\kappa_h^\\delta\\gamma^2\\right)D_h(\\bar{x}_{s,k+1},x_{s,k}) -\\frac{\\gamma}{2\\eta}D_h(x_{s,k},\\bar{x}_{s,k+1})+ \\frac{\\gamma\\eta\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h(\\mathcal{X}_s)} \\,,\\nonumber\n    \\end{eqnarray}\n    where (i) is due to Assumption \\ref{assumption:L-smad-finite-sum}, Lemma \\ref{lemma:descent-lemma}, and the convexity of $\\phi$, (ii) is due to the definition of $\\mathcal{E}_{s,k}$ in Lemma \\ref{lemma:restricted-grd-mapping} and the following scaling property\n    \\begin{eqnarray}\n        D_h\\left(x_{s,k+1},x_{s,k}\\right)  \\leq \\frac{L_h(\\mathcal{X}_s)}{2}\\|x_{s,k+1}-x_{s,k}\\|^2 = \\frac{\\gamma^2L_h(\\mathcal{X}_s)}{2}\\|\\bar{x}_{s,k+1}-x_{s,k}\\|^2 \\leq \\gamma^2\\kappa_h^\\delta D_h(\\bar{x}_{s,k+1},x_{s,k})\\,,\\nonumber\n    \\end{eqnarray}\n    and (iii) is due to the optimality of  $\\bar{x}_{s,k+1}$ to the corresponding subproblem and Tseng's three point property (Lemma \\ref{lemma:Tseng-3point}). Finally, by Lemma \\ref{lemma:restricted-grd-mapping}, we also have\n    $$\\big\\|\\cGshe(x_{s,k})\\big\\|^2 \\leq \\frac{2\\|x_{s,k}-\\bar{x}_{s,k+1}\\|^2}{\\eta^2} + \\frac{2\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h^2(\\mathcal{X}_s)}\\,.$$\n    Multiplying both sides of the above inequality by $\\frac{\\gamma\\eta\\mu_h(\\mathcal{X}_s)}{8}$ and add it to \\eqref{lm:descent-proxSARAH-finite-1} proves the lemma. \n\\end{proof}\n\n\\begin{proof}\n    First, substituting the gradient estimation bound in Lemma \\ref{lemma:err-sarah} to Lemma \\ref{lemma:descent-proxSARAH-finite}, we have the following descent result throughout the $s$-th epoch\n\n\\begin{align}\n    \\label{lm:proxSARAH-grad-map-mid-1}\n    &\\mathbb{E}\\Big[\\Psi(x_{s,\\tau_s}) \\,\\big|\\, x_{s,0}\\Big] \\!\\leq \\Psi(x_{s,0}) \\!-\\! \\mathbb{E}\\bigg[\\frac{\\gamma\\eta\\mu_h(\\mathcal{X}_s)}{8}\\!\\sum_{k=0}^{\\tau_s-1}\\!\\big\\|\\cGshe(x_{s,k})\\big\\|^2 \\!+\\!\\bigg(\\!\\frac{\\gamma}{\\eta}- L\\kappa_h^\\delta\\gamma^2\\!\\bigg)D_h(\\bar{x}_{s,k+1},x_{s,k}) \\,\\Big|\\, x_{s,0}\\bigg]\\nonumber\\\\\n    &\\qquad\\qquad\\qquad\\qquad\\,\\, +\\frac{5\\gamma\\eta}{4\\mu_h(\\mathcal{X}_s)}\\cdot\\frac{2\\gamma^2L^2\\kappa_h^\\delta L_h(\\mathcal{X}_s)}{b}\\mathbb{E}\\bigg[\\sum_{k=0}^{\\tau_s-1}\\sum_{j=0}^{k-1} D_h(\\bar{x}_{s,j+1},x_{s,j}) \\,\\Big|\\, x_{s,0}\\bigg]\\\\\n    &\\quad\\,\\leq \\Psi(x_{s,0}) \\!-\\! \\mathbb{E}\\bigg[\\frac{\\gamma\\eta\\mu_h(\\mathcal{X}_s)}{8}\\!\\sum_{k=0}^{\\tau_s-1}\\big\\|\\mathcal{G}_{s,k}^\\eta(x_{s,k})\\big\\|^2 \\!+\\bigg(\\frac{\\gamma}{\\eta}- L\\kappa_h^\\delta\\gamma^2-\\frac{5\\tau\\gamma^3\\eta L^2(\\kappa_h^\\delta)^2}{2b}\\bigg)D_h(\\bar{x}_{s,k+1},x_{s,k}) \\,\\Big|\\, x_{s,0}\\bigg]\\nonumber\n\\end{align}\n\n\\noindent Suppose we choose $\\eta  = \\frac{\\sqrt{2\\tau}}{\\sqrt{7\\tau}+\\sqrt{2b}}$ and we choose $\\gamma = \\frac{\\sqrt{b}}{L\\kappa_h^\\delta\\sqrt{\\tau}}$. Then we have \n\\begin{eqnarray}\n    \\label{lm:proxSARAH-grad-map-mid-2}\n    &&\\frac{\\gamma}{\\eta} - L\\kappa_h^\\delta\\gamma^2 - \\frac{5\\tau\\gamma^3\\eta L^2(\\kappa_h^\\delta)^2}{2b}\\nonumber\\\\\n    & = & \\gamma\\eta\\bigg(\\frac{1}{\\eta^2} - \\frac{L\\kappa_h^\\delta\\gamma}{\\eta} - \\frac{5\\tau\\gamma^2 L^2(\\kappa_h^\\delta)^2}{2b}\\bigg)\\\\\n    &=& \\gamma\\eta\\bigg(1+\\frac{\\sqrt{14} -\\sqrt{3.5}}{\\sqrt{\\tau/b}}\\bigg)\\nonumber \\\\\n    &\\geq& \\gamma\\eta\\,.\\nonumber\n\\end{eqnarray} \nSubstitute the bound to the previous inequality, summing up over all epochs, and taking the expectation over all randomness proves the lemma. \n\\end{proof}","units":[{"start":27746,"end":28058,"start_line":212,"end_line":216,"raw_tex":"\\begin{lemma}[Extended descent lemma, \\cite{bolte2018first}]\n    \\label{lemma:descent-lemma}\n    Suppose $f$ and $h$ satisfy Assumption \\ref{assumption:L-smad} for some constant $L>0$, then for any $\\forall x,y\\in\\RR^d$, it holds that \n    $$|f(x)-f(y)-\\langle\\nabla f(y),x-y\\rangle|\\leq LD_h(x,y).$$\n\\end{lemma}","sha256_utf8":"710102daa3d19b5c5c60c685263426e4d87fa6e94fed02ea3dd34d527a53c6bd"},{"start":34849,"end":35599,"start_line":259,"end_line":268,"raw_tex":"\\begin{lemma}\n    \\label{lemma:grd-vs-grdmap}\n    For any $x,v\\in\\RR^d$ and any strictly convex kernel $h$, denote $x_\\lambda^+(v) := \\mathbf{T}_{\\phi,h}^\\lambda(x,v)$.  Let $[x,x_\\lambda^+(v)]$ be the line segment between $x_\\lambda^+(v)$ and $x$, then there exists $u\\in\\partial\\phi(x_\\lambda^+(v))$ such that  \n    \\begin{equation}\n        \\mu_h\\big([x,x_\\lambda^+(v)]\\big)\\cdot \\|x-x_\\lambda^+(v)\\|\\leq\\lambda\\|v+u\\|\\leq L_h\\big([x,x_\\lambda^+(v)]\\big)\\cdot \\|x-x_\\lambda^+(v)\\|\\,,\\nonumber\n    \\end{equation} \n    \\begin{equation}\n        \\sqrt{2\\mu_h\\big([x,x_\\lambda^+(v)]\\big)\\cdot D_h(x_\\lambda^+(v),x)}\\leq \\lambda\\|v+u\\|\\leq \\sqrt{2L_h\\big([x,x_\\lambda^+(v)]\\big)\\cdot D_h(x_\\lambda^+(v),x)}\\,\\,.\\nonumber\n    \\end{equation}   \n\\end{lemma}","sha256_utf8":"d4596091c9b5c9f98934627ae3f77875b8037b95f10e5e853a9c9db3bade1b6c"},{"start":35600,"end":37408,"start_line":269,"end_line":282,"raw_tex":"\\begin{proof}\n    By the optimality condition of the subproblem $x_\\lambda^+(v)=\\argmin_{y\\in\\RR^d}\\, y^\\top v + \\phi(y) + \\frac{1}{\\lambda}D_h(y,x)$, we have $0\\in \\partial\\phi(x_\\lambda^+(v))+v+\\frac{1}{\\lambda}\\nabla_y D_h(y,x)|_{y=x_\\lambda^+(v)}$. Namely, there exists  $u\\in\\partial\\phi(x_\\lambda^+(v))$ such that \n    \\begin{equation}\n        \\label{lm:grd-vs-grdmap-1}\n        \\lambda(v + u) +  \\big(\\nabla h(x_\\lambda^+(v))-\\nabla h(x)\\big)=0.\n    \\end{equation}\n    Then \\cite[Theorem 2.1.9]{nesterov2018lectures}, together with the definition of $\\mu_h(\\cdot)$ and $L_h(\\cdot)$, indicates that \n    $$\\mu_h\\big([x,x_\\lambda^+(v)]\\big)\\cdot \\|x-x_\\lambda^+(v)\\|\\leq\\|\\nabla h(x_\\lambda^+(v))-\\nabla h(x)\\|\\leq L_h\\big([x,x_\\lambda^+(v)]\\big)\\cdot \\|x-x_\\lambda^+(v)\\|.$$\n    Combining this bound with equation \\eqref{lm:grd-vs-grdmap-1} proves the first inequality of Lemma \\ref{lemma:grd-vs-grdmap}. Also observe that $\\nabla_y^2 D_h(y,x) = \\nabla^2h(y)$, we have \n    $$\\mu_h([x,x_\\lambda^+(v)])\\cdot I \\preceq \\nabla^2_y D_h(y,x) \\preceq L_h([x,x_\\lambda^+(v)])\\cdot I\\quad\\mbox{for}\\quad \\forall y\\in[x,x_\\lambda^+(v)].$$ \n    Combined with the fact that $\\nabla_y D_h(y,x)|_{y=x} = 0$, then   \\cite[Theorem 2.1.5, Eq.(2.1.10)]{nesterov2018lectures} and \\cite[Theorem 2.1.10, Eq.(2.1.24)]{nesterov2018lectures} immediately indicates \n    $$\\frac{\\left\\|\\nabla_y D_h(y,x)|_{y=x_\\lambda^+(v)}\\right\\|^2}{2L_h([x,x_\\lambda^+(v)])}\\leq D_h(x_\\lambda^+(v),x)-D_h(x,x)\\leq \\frac{\\left\\|\\nabla_y D_h(y,x)|_{y=x_\\lambda^+(v)}\\right\\|^2}{2\\mu_h([x,x_\\lambda^+(v)])}.$$\n    Then substituting $D_h(x,x)=0$ and $\\lambda(v+u) = \\nabla h(x)-\\nabla h(x_\\lambda^+(v)) = -\\nabla_z D_h(y,x)\\mid_{y=x_\\lambda^+(v)}$ to the above bound proves the second inequality of Lemma \\ref{lemma:grd-vs-grdmap}.  \n\\end{proof}","sha256_utf8":"75277b6d90d035677a22667faac189fb6a39f15b5856d2c527432ec52e420f4e"},{"start":64631,"end":65305,"start_line":486,"end_line":494,"raw_tex":"\\begin{proposition}[Lipschitz-like bound] \n    \\label{proposition:Lipschitz} \n    Suppose $f$ is $L$-smooth adaptable to some KC-regular kernel $h$. Let $\\delta>0$ and let $\\cX$ be any convex set with $\\mathrm{diam}(\\cX)\\leq \\delta$. Then for any $x,y\\in\\cX$, and an arbitrary interpolation point $z_\\theta := \\theta x + (1-\\theta)y$ with $ \\theta\\in[0,1]$, we have  \n    \\begin{equation}\n        \\label{prop:Lipschitz}\n        \\frac{\\|\\nabla f(z_\\theta)-\\nabla f(y)\\|^2}{2L^2\\mu_h(\\cX)}\\leq  \\theta^2\\kappa_h^2(\\cX) D_h(x,y),\n    \\end{equation}\n    where by KC-regularity, we know the constant upper bound $\\kappa_h(\\cX)\\leq \\kappa_h^\\delta$ always hold. \n\\end{proposition}","sha256_utf8":"eb79dcf9d4c4a8313947e0a797f891ff732ee2855771af4424d4dc05622c6895"},{"start":74405,"end":74967,"start_line":565,"end_line":573,"raw_tex":"\\begin{lemma}\n    \\label{lemma:restricted-grd-mapping} \n    Let $\\cGshe(x_{s,k})$, $\\hat{x}_{s,k+1}$ and $\\bar{x}_{s,k+1}$ be defined by \\eqref{defn:restricted-grd-mapping} and \\eqref{alg:sarah-update}, respectively. Then \n    \\begin{eqnarray}\n    \\label{lm:grad-mapping} \n        \\big\\|\\cGshe(x_{s,k})\\big\\|^2 \\leq \\frac{2\\|x_{s,k}-\\bar{x}_{s,k+1}\\|^2}{\\eta^2} + \\frac{2\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h^2(\\mathcal{X}_s)},\n    \\end{eqnarray} \n    where $\\mathcal{E}_{s,k} = \\nabla f(x_{s,k})-v_{s,k}$ denotes the gradient estimation error at $x_{s,k}$.\n\\end{lemma}","sha256_utf8":"676da4e437df7f7700c2ee2c0b95858798eb7e07e7f240bb54a8221e9bd6ca7f"},{"start":75608,"end":76109,"start_line":578,"end_line":584,"raw_tex":"\\begin{lemma}\n\\label{lemma:descent-proxSARAH-finite}\n    Let $\\{x_{s,k}\\}_{k=0}^{\\tau_s}$ be the $s$-th epoch of Algorithm \\ref{algorithm:proxSARAH}, then  we have \n    \\begin{eqnarray}\n    \\Psi(x_{s,k+1})\\leq \\Psi(x_{s,k})-\\frac{\\gamma\\eta\\mu_h(\\mathcal{X}_s)}{8}\\big\\|\\cGshe(x_{s,k})\\big\\|^2\\!-\\!\\bigg(\\frac{\\gamma}{\\eta}- L\\kappa_h^\\delta\\gamma^2\\!\\bigg)\\!D_h(\\bar{x}_{s,k+1},x_{s,k}) \\!+\\!\\frac{5\\gamma\\eta\\|\\mathcal{E}_{s,k}\\|^2}{4\\mu_h(\\mathcal{X}_s)}.\\,\\nonumber\n    \\end{eqnarray} \n\\end{lemma}","sha256_utf8":"391aaf1793853678afc2b91ddc9bd5168d24d4dfe437a2159dcdfedbb9065dac"},{"start":76378,"end":77068,"start_line":588,"end_line":596,"raw_tex":"\\begin{lemma}\n    \\label{lemma:err-sarah} \n    Let $\\{x_{s,k}\\}_{k=0}^{\\tau_s}$ be the $s$-th epoch of Algorithm \\ref{algorithm:proxSARAH}. Given any batch size $b>0$, if we set $|\\mathcal{B}_{s,k}|=b$ for $k=1,\\cdots,\\tau_s-1$. Then conditioning on the initial point $x_{s,0}$ of the epoch, we have\n    \\begin{eqnarray}\n    \\label{lm:err-sarah-0}\n    \\mathbb{E}\\left[\\frac{\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h(\\mathcal{X}_s)}\\,\\Big|\\, x_{s,0}\\right] \\leq  \\frac{2\\gamma^2L^2(\\kappa_h^\\delta)^2}{b}\\mathbb{E}\\bigg[\\sum_{j=0}^{k-1} D_h(\\bar{x}_{s,j+1},x_{s,j})\\,\\big|\\,x_{s,0}\\bigg]\\,, \n\\end{eqnarray}\nas long as $h$ satisfies the kernel conditioning regularity assumption in $\\cX_s$. \n\\end{lemma}","sha256_utf8":"0aeda5fbcfcd5add3fbf82e49c56adf6a065dd2470ddab0be0d82ddeafa46bbc"},{"start":77069,"end":78211,"start_line":597,"end_line":610,"raw_tex":"\\begin{proof}\n    By \\cite[Lemma 2]{pham2020proxsarah}, it is not hard to obtain that \n    \\begin{equation}\n        \\label{lm:err-sarah-0.5}\n        \\mathbb{E}\\Big[\\|\\mathcal{E}_{s,k}\\|^2\\,\\big|\\,x_{s,0}\\Big]  \\leq  \\sum_{j=0}^{k-1} \\mathbb{E}\\bigg[\\frac{1}{bn}\\sum_{i=1}^n\\|\\nabla f_i(x_{s,j+1})-\\nabla f_i(x_{s,j})\\|^2\\,\\big|\\,x_{s,0}\\bigg].\n    \\end{equation}\n    Because each $f_i$ is $L_i$-smooth adaptable to the KC-regular kernel $h$, applying the Lipschitz-like bound \\eqref{prop:Lipschitz} to each $f_i$ on   $\\bar{x}_{s,j+1}, x_{s,j}$ and their interpolation ${x}_{s,j+1} = \\gamma\\bar{x}_{s,j+1} + (1-\\gamma)x_{s,j}$ immediately yields \n    \\begin{eqnarray}\n    \\label{lm:err-sarah-1}\n    \\mathbb{E}\\left[\\frac{\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h(\\mathcal{X}_s)}\\,\\Big|\\, x_{s,0}\\right] \n    &\\leq&  \\sum_{j=0}^{k-1}\\mathbb{E}\\bigg[ \\frac{\\sum_{i=1}^nL_i^2}{n}\\cdot\\frac{2\\gamma^2  \\kappa_h^2(\\cX_s)}{b}D_h(\\bar{x}_{s,j+1},x_{s,j})\\,\\big|\\,x_{s,0}\\bigg]\\nonumber.\n\\end{eqnarray}\nUsing the fact that $L^2 = \\frac{1}{n}\\sum_{i=1}^nL_i^2$ and the fact that $\\kappa_h(\\cX_s)\\leq \\kappa_h^\\delta$ leads to \\eqref{lm:err-sarah-0}. \n\\end{proof}","sha256_utf8":"6872660aa0afff840ee399b2d1df71895b101dbf8c5bd18c209576aee670664b"},{"start":78427,"end":78996,"start_line":614,"end_line":619,"raw_tex":"\\begin{lemma}\n    \\label{lemma:proxSARAH-grad-map-mid}\n    For any $b,\\tau\\in\\mathbb{Z}_+$, set $\\eta = \\frac{\\sqrt{2\\tau}}{\\sqrt{7\\tau}+\\sqrt{2b}}$, $\\gamma = \\frac{\\sqrt{b}}{L\\kappa_h^\\delta\\sqrt{\\tau}}$, and $|\\mathcal{B}_{s,k}|=b$, $\\forall s,k\\geq1$, then\n    $$\\mathbb{E}\\left[\\sum_{s=1}^S\\sum_{k=0}^{\\tau_s-1}\\frac{\\mu_h(\\mathcal{X}_s)}{8}\\big\\|\\cGshe(x_{s,k})\\big\\|^2 +D_h(\\bar{x}_{s,k+1},x_{s,k}) \\right]\\leq \\frac{\\Delta_\\Psi}{\\gamma\\eta},$$\n    where $\\Delta_\\Psi:=\\Psi(x_{1,0})-\\inf\\{\\Psi(x):x\\in\\RR^d\\}$ denotes the initial function value gap. \n\\end{lemma}","sha256_utf8":"6d58ca64d2655627a9ea2343cacad156b82fdc256b36206b194e73126745e65c"},{"start":81024,"end":81539,"start_line":634,"end_line":640,"raw_tex":"\\begin{lemma}\n\\label{lemma:event-halway} \nFor any $S$ epochs  generated by Algorithm \\ref{algorithm:proxSARAH}, define the set $\\mathcal{I}_1$ and event $\\mathcal{A}_1(m_1)$ as  \n$$\\mathcal{I}_1:=\\big\\{s\\in[S] :  \\tau_s < \\tau\\big\\}\\qquad\\mbox{and}\\qquad\\mathcal{A}_1(m_1):=\\left\\{\\omega: |\\mathcal{I}_1|\\geq m_1\\right\\}.$$\nwhere $m_1>0$ is an arbitrary positive number. Then it holds that \n$$\\mathrm{Prob}\\left(\\mathcal{A}_1(m_1)\\right)\\leq \\frac{32\\gamma\\tau\\Delta_\\Psi}{\\eta\\mu\\delta^2\\cdot m_1}\\,.$$\n\\end{lemma}","sha256_utf8":"4ba09c16ec16e2b38117f191316a1b1e73b53953e20028359129e9755f9aea60"},{"start":81540,"end":83698,"start_line":641,"end_line":664,"raw_tex":"\\begin{proof}\nBy lemma \\ref{lemma:proxSARAH-grad-map-mid}, ignoring the restricted primal gradient mapping terms yields   \n\\begin{eqnarray}\n    \\label{lm:event-halfway-1}\n    \\frac{\\Delta_\\Psi}{\\gamma\\eta} &\\geq& \\mathbb{E}\\left[\\sum_{s=1}^S\\sum_{k=0}^{\\tau_s-1}D_h(\\bar{x}_{s,k+1},x_{s,k}) \\right]\\nonumber\\\\ \n    & \\geq & \\mathbb{E}\\left[\\frac{\\mu}{2\\gamma^2}\\sum_{s\\in\\mathcal{I}_1}\\sum_{k=0}^{\\tau_s-1}\\|{x}_{s,k+1}-x_{s,k}\\|^2 \\right] \\\\\n    & \\geq & \\frac{\\mu}{2\\gamma^2}\\cdot \\mathrm{Prob}\\left(\\mathcal{A}_1(m_1)\\right) \\cdot\\mathbb{E}\\left[\\sum_{s\\in\\mathcal{I}_1}\\sum_{k=0}^{\\tau_s-1}\\|{x}_{s,k+1}-x_{s,k}\\|^2\\,\\Big|\\, \\mathcal{A}_1(m_1) \\right] \\nonumber,\n\\end{eqnarray}\nwhere the last inequality is due to the fact that   \n$$\\EE[X] = \\mathrm{Prob}(\\cA)\\cdot\\EE\\big[X\\mid\\cA\\big] + \\mathrm{Prob}(\\cA^c)\\cdot\\EE\\big[X\\mid\\cA^c\\big]\\geq \\mathrm{Prob}(\\cA)\\cdot\\EE\\big[X\\mid\\cA\\big]$$\nfor any non-negative random variable $X\\geq0$ and any event $\\cA$. Therefore, conditioning on the event $\\mathcal{A}_1(m_1)$, for any epoch $s\\in\\mathcal{I}_1$, we will have $\\|x_{s,\\tau_s}-x_{s,0}\\|\\geq\\frac{\\delta}{4}$ because of Line 3 and Line 9 of Algorithm \\ref{algorithm:proxSARAH}. Consequently, for $\\forall s\\in\\mathcal{I}_1$, the triangle inequality and the arithmetic inequality indicate that\n\\begin{equation}\n    \\label{lm:event-halfway-2}\n    \\frac{\\delta}{4\\tau_s}\\leq \\frac{\\|x_{s,\\tau_s}-x_{s,0}\\|}{\\tau_s}\\leq\\frac{\\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|}{\\tau_s}\\leq  \\sqrt{\\frac{\\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|^2}{\\tau_s}}\\,.\n\\end{equation} \nBecause $\\tau_s\\leq\\tau$ always holds, we have \n\\begin{align}\n    \\label{lm:event-halfway-3}\n    \\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|^2  \\geq  \\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|^2\\geq\\frac{\\delta^2}{16\\tau_s}\\geq\\frac{\\delta^2}{16\\tau}\\,.\n\\end{align}\nNote that the above inequalities hold w.p. 1 conditioning on $\\cA_1(m_1)$. Combined with \\eqref{lm:event-halfway-1}, we have \n$$\\frac{\\Delta_\\Psi}{\\gamma\\eta} \\geq\\frac{\\mu}{2\\gamma^2}\\cdot \\mathrm{Prob}\\left(\\mathcal{A}_1(m_1)\\right)\\cdot\\frac{m_1\\delta^2}{16\\tau}.$$\nRearranging the terms completes the proof. \n\\end{proof}","sha256_utf8":"14ca1ba56ad86c0d3ebf174be9e539543205eb8d97a269b8d17510ef92e65a90"},{"start":84050,"end":84641,"start_line":667,"end_line":673,"raw_tex":"\\begin{lemma}\n    \\label{lemma:event-grad-map}\n    For any $S$ epochs generated by Algorithm \\ref{algorithm:proxSARAH}, define the set $\\mathcal{I}_2$ and event $\\mathcal{A}_2(m_2)$ as\n    $$\\mathcal{I}_2:=\\left\\{(s,k) :  \\hat{x}_{s,k+1}\\in\\partial \\mathcal{X}_s, 1\\leq s\\leq S, 0\\leq k\\leq \\tau_s-1\\right\\}\\quad\\mbox{and}\\quad\\mathcal{A}_2(m_2):=\\big\\{\\omega : |\\mathcal{I}_2|\\geq m_2\\big\\}$$\n    where  $m_2>0$ is an arbitrary positive number. Then it holds that \n    $$\\mathrm{Prob}\\left(\\mathcal{A}_2(m_2)\\right)\\leq \\frac{128\\eta\\Delta_\\Psi}{\\gamma\\mu\\delta^2\\cdot m_2}\\,.$$\n\\end{lemma}","sha256_utf8":"618ec4348b6ea90a212e7dca649b371eff23ea4fe89b95b33da6d835413d9b6c"},{"start":84642,"end":85994,"start_line":674,"end_line":687,"raw_tex":"\\begin{proof}\nSimilar to the proof of Lemma \\ref{lemma:event-halway}, ignoring the Bregman divergence terms in Lemma \\ref{lemma:proxSARAH-grad-map-mid} yields   \n\\begin{eqnarray}\n    \\label{lm:event-grad-map-1}\n    \\frac{\\Delta_\\Psi}{\\gamma\\eta} & \\geq & \\mathbb{E}\\left[\\sum_{s=1}^S\\sum_{k=0}^{\\tau_s-1}\\frac{\\mu}{8}\\big\\|\\cGshe(x_{s,k})\\big\\|^2\\right]\\nonumber\\\\\n    & = & \\frac{\\mu}{8\\eta^2}\\mathbb{E}\\left[\\sum_{s=1}^S\\sum_{k=0}^{\\tau_s-1}\\|\\hat{x}_{s,k+1}-x_{s,k}\\|^2\\right]\\\\\n    & \\geq & \\frac{\\mu}{8\\eta^2}\\mathbb{E}\\Bigg[\\sum_{(s,k)\\in\\mathcal{I}_2} \\|\\hat{x}_{s,k+1}-x_{s,k}\\|^2\\Bigg]\\nonumber\\\\\n    & \\geq & \\frac{\\mu}{8\\eta^2}\\cdot\\mathrm{Prob}\\left(\\mathcal{A}_2(m_2)\\right)\\cdot\\mathbb{E}\\Bigg[\\sum_{(s,k)\\in\\mathcal{I}_2}\\|\\hat{x}_{s,k+1}-x_{s,k}\\|^2\\,\\Big|\\,\\mathcal{A}_2(m_2)\\Bigg]\\nonumber.\n\\end{eqnarray}\nNote that for  $\\forall(s,k)\\in\\mathcal{I}_2$, we have $\\mathrm{dist}(x_{s,k},\\partial\\cX_s)\\geq\\delta/4$ while $\\hat{x}_{s,k+1}\\in\\partial\\cX_s$. Consequently, one must have $\\|\\hat{x}_{s,k+1}-{x}_{s,k}\\|\\geq \n\\frac{\\delta}{4}$. Conditioning on the event $\\cA_2(m_2)$, we have $|\\mathcal{I}_2|\\geq m_2$ and \n$$\\mathbb{E}\\Bigg[\\sum_{(s,k)\\in\\mathcal{I}_2}\\|\\hat{x}_{s,k+1}-x_{s,k}\\|^2\\,\\Big|\\,\\mathcal{A}_2(m_2)\\Bigg]\\geq \\frac{m_2\\delta^2}{16}.$$\nSubstitute this bound to \\eqref{lm:event-grad-map-1} proves lemma.  \n\\end{proof}","sha256_utf8":"207db1d169db39a36dd8a25328ac225a5949d88ab1c24d83569830c9e94b9019"},{"start":88120,"end":98744,"start_line":702,"end_line":782,"raw_tex":"\\begin{proof}\nBy lemma \\ref{lemma:proxSARAH-grad-map-mid}, ignoring the restricted primal gradient mapping terms yields  \n\\begin{eqnarray} \n\\frac{\\Delta_\\Psi}{\\gamma\\eta} \\geq\n\\mathbb{E}\\bigg[\\sum_{s=1}^{S}\\sum_{k=0}^{\\tau_s-1}D_h(\\bar{x}_{s,k+1},x_{s,k})\\bigg] \n\\geq \\frac{\\mu}{2\\gamma^2}\\mathbb{E}\\bigg[\\sum_{s=1}^{S}\\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|^2\\bigg]\\,. \\nonumber\n\\end{eqnarray}\n\nBy expanding the expectation over all possible $\\mathcal{I}_1, \\mathcal{I}_2,$ and $\\{\\tau_s\\}$, we have for all $m_1,m_2>0$ that \n\\begin{eqnarray} \n\\label{thm:proxSARAH-finite-1}\n\\frac{2\\gamma\\Delta_\\Psi}{\\mu\\eta} \\!\\!\\!\\!& \\geq & \\!\\!\\!\\!\\mathbb{E}\\bigg[\\sum_{s=1}^{S}\\sum_{k=0}^{\\tau_s-1}\\!\\|x_{s,k+1}\\!-\\!x_{s,k}\\|^2\\bigg] \\\\\n& \\geq &\\!\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\!\\!\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\mathbb{E}\\bigg[\\!\\sum_{s=1}^S\\!\\sum_{k=0}^{\\tau_s-1}\\!\\|x_{s,k+1}-x_{s,k}\\|^2 \\,\\Big|\\, \\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\!\\bigg]\\nonumber\\\\\n& \\geq &\\!\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\!\\!\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\mathbb{E}\\bigg[\\!\\sum_{s\\in\\mathcal{I}_1}\\!\\!\\sum_{k=0}^{\\tau_s-1}\\!\\|x_{s,k+1}-x_{s,k}\\|^2 \\,\\Big|\\, \\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\!\\bigg]\\nonumber \n\\end{eqnarray}\nNote that for $\\forall s\\in\\mathcal{I}_1$, according to the discussion in Lemma \\ref{lemma:event-grad-map}, we have  $\\|x_{s,\\tau_s}-x_{s,0}\\|  \\geq \\delta/4.$\nRepeating the analysis of \\eqref{lm:event-halfway-2} and \\eqref{lm:event-halfway-3} yields $\\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|^2  \\geq \\frac{\\delta^2}{16\\tau_s}$, which always holds true. Then substituting this lower bound to \\eqref{thm:proxSARAH-finite-1} gives \n\\begin{equation}\n    \\label{thm:proxSARAH-finite-2}\n    \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\cdot\\sum_{s\\in\\mathcal{I}_1}\\tau_s^{-1}\\leq\\frac{32\\gamma\\Delta_\\Psi}{\\mu\\eta\\delta^2}\\,.\n\\end{equation} \nDefine the events  \n\\begin{equation}\n    \\label{defn:event-A3}\n    \\mathcal{A}_3:=\\big\\{\\omega: \\mbox{the index of } x_{\\mathrm{out}}\\mbox{ is from }\\mathcal{I}_2^c\\big\\}\\quad\\mbox{and}\\quad \\mathcal{A}:=\\mathcal{A}_1^c(m_1)\\cap\\mathcal{A}_2^c(m_2)\\cap\\mathcal{A}_3.\n\\end{equation} \nThen we have\n\\begin{eqnarray}\n    \\label{thm:proxSARAH-finite-33}\n    \\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\mathrm{Prob}\\left(\\mathcal{A}\\right)\\!\\!\\! &=&\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\cdot\\mathrm{Prob}\\big(\\cA_3\\mid\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\\\\n    &=&\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\cdot\\left(1-\\frac{|\\mathcal{I}_2|}{(S-|\\mathcal{I}_1|)\\tau+\\sum_{s\\in\\mathcal{I}_1}\\tau_s}\\right)\\nonumber\\\\\n    & \\geq &\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\cdot\\left(1-\\frac{m_2}{(S-|\\mathcal{I}_1|)\\tau+\\sum_{s\\in\\mathcal{I}_1}\\tau_s}\\right).\\nonumber\n\\end{eqnarray}\nNote that for any positive numbers $y_1,\\cdots,y_S>0$,  the arithmetic-harmonic inequality states that $$\\frac{y_1+y_2+\\cdots+y_S}{S}\\geq\\frac{S}{\\frac{1}{y_1}+\\frac{1}{y_2}+\\cdots+\\frac{1}{y_S}}\\,.$$ \nApplying this inequality gives \n$$\\frac{1}{\\tau} + \\frac{\\sum_{s\\in\\mathcal{I}_1}\\tau_s^{-1}}{S}= \\frac{(S-|\\mathcal{I}|_1)\\tau^{-1} + \\sum_{s\\in\\mathcal{I}_1}\\tau_s^{-1}}{S}\\geq \\frac{S}{(S-|\\mathcal{I}|_1)\\tau+ \\sum_{s\\in\\mathcal{I}_1}\\tau_s}\\,,$$\nwhich implies that \n$$1-\\frac{m_2}{(S-|\\mathcal{I}_1|)\\tau+\\sum_{s\\in\\mathcal{I}_1}\\tau_s}\\geq 1-\\frac{m_2}{S\\tau}-\\frac{m_2}{S^2}\\sum_{s\\in\\mathcal{I}_1}\\tau_s^{-1}\\,.$$\nSubstituting this bound to \\eqref{thm:proxSARAH-finite-33} and setting $m_1^* = S/4$ yields\n\\begin{eqnarray}\n    \\label{thm:proxSARAH-finite-44}\n    \\!\\!\\!\\mathrm{Prob}\\left(\\mathcal{A}\\right) \\!\\!\\!\\!& \\geq &\\!\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1^*}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\cdot\\left(1-\\frac{m_2}{S\\tau}-\\frac{m_2}{S^2}\\sum_{s\\in\\mathcal{I}_1}\\tau_s^{-1}\\right)\\nonumber\\\\\n    & \\overset{(i)}{\\geq} &\\!\\!\\!\\! \\left(1-\\frac{m_2}{S\\tau}\\right)\\mathrm{Prob}\\left(\\mathcal{A}_1^c(m_1^*)\\cap\\mathcal{A}_2^c(m_2)\\right) - \\frac{m_2}{S^2}\\cdot\\frac{32\\gamma\\Delta_\\Psi}{\\mu\\eta\\delta^2}\\\\\n    & \\geq & \\!\\!\\!\\! \\left(1-\\frac{m_2}{S\\tau}\\right)\\big(1-\\mathrm{Prob}\\left(\\mathcal{A}_1(m_1^*)\\right)-\\mathrm{Prob}\\left(\\mathcal{A}_2(m_2)\\right)\\big) - \\frac{m_2}{S^2}\\cdot\\frac{32\\gamma\\Delta_\\Psi}{\\mu\\eta\\delta^2}\\nonumber\\\\\n    & \\overset{(ii)}{\\geq} & \\!\\!\\!\\! 1 - \\frac{8\\eta\\tau b\\epsilon}{L^2(\\kappa_h^\\delta)^2\\delta^2} - \\frac{128\\eta\\Delta_\\Psi}{\\gamma\\mu\\delta^2\\cdot m_2} - \\frac{m_2}{S\\tau} - \\frac{m_2}{S^2}\\cdot\\frac{32\\gamma\\Delta_\\Psi}{\\mu\\eta\\delta^2}\\nonumber\\\\\n    & \\overset{(iii)}{\\geq} & \\!\\!\\!\\! 1 - \\frac{8\\eta\\tau b\\epsilon}{L^2(\\kappa_h^\\delta)^2\\delta^2} - \\frac{128\\eta\\Delta_\\Psi}{m_2\\gamma\\mu\\delta^2} - \\frac{9m_2}{8S\\tau}\\nonumber\n\\end{eqnarray}\nwhere (i) is due to \\eqref{thm:proxSARAH-finite-2} and the fact that \n\\begin{equation}\n    \\label{thm:proxSARAH-finite-55}\n    \\sum_{|\\mathcal{I}_1|<m_1^*}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big) = \\mathrm{Prob}\\big(\\mathcal{A}_1^c(m_1^*)\\cap\\mathcal{A}_2^c(m_2)\\big)\n\\end{equation}\n(ii) is due to Lemma \\ref{lemma:event-halway} with $m_1^*=S/4$ and $S = \\big\\lceil\\frac{16\\Delta_\\Psi}{\\tau\\gamma\\eta\\mu\\epsilon}\\big\\rceil$: \n$$\\mathrm{Prob}\\big(\\mathcal{A}_1(S/4)\\big)\\leq \\frac{32\\gamma\\tau\\Delta_\\Psi}{\\mu\\delta^2S/4} = \\frac{16\\Delta_\\Psi}{S\\tau\\gamma\\eta\\mu}\\cdot\\frac{8\\gamma^2\\tau^2\\eta}{\\delta^2} \\leq \\frac{8\\tau b\\epsilon}{L^2(\\kappa_h^\\delta)^2\\delta^2},$$\nand (iii) is because we require $\\epsilon\\leq \\frac{\\delta^2}{16}\\cdot\\min\\left\\{\\frac{L^2(\\kappa_h^\\delta)^2}{b\\tau},\\frac{1}{9\\eta^2}\\right\\} = O(1/n)$ such that \n$$\\frac{m_2}{S^2}\\cdot\\frac{32\\gamma\\Delta_\\Psi}{\\mu\\eta\\delta^2} = \\frac{m_2}{S\\tau}\\cdot\\frac{16\\Delta_\\Psi}{S\\tau\\gamma\\eta\\mu}\\cdot\\frac{2\\gamma^2\\tau^2}{\\delta^2}\\leq \\frac{m_2}{8S\\tau}$$\nTherefore, to maximize the above probability, we can choose $m_2^*=\\sqrt{\\frac{S\\tau\\eta\\Delta_\\Psi}{\\gamma\\mu}}\\cdot\\frac{32}{3\\delta}$ such that\n$$\\frac{128\\eta\\Delta_\\Psi}{m_2^*\\gamma\\mu\\delta^2} + \\frac{9m_2^*}{8S\\tau} = 2\\sqrt{\\frac{128\\eta\\Delta_\\Psi}{m_2^*\\gamma\\mu\\delta^2} \\cdot  \\frac{9m_2^*}{8S\\tau}} = \\sqrt{\\frac{\\eta\\Delta_\\Psi}{\\gamma\\mu S\\tau}}\\cdot\\frac{24}{\\delta}\\leq \\frac{6\\eta\\sqrt{\\epsilon}}{\\delta}\\leq \\frac{4\\sqrt{\\epsilon}}{\\delta}\\,, $$\nwhere the last inequality is due to the fact that $\\eta = \\frac{\\sqrt{2\\tau}}{\\sqrt{7\\tau}+\\sqrt{2b}}\\leq\\sqrt{2/7}.$ Combining all the above discussion, we can conclude that  \n$$\\mathrm{Prob}\\left(\\mathcal{A}\\right)\\geq1 - \\frac{8\\eta\\tau b\\epsilon}{L^2(\\kappa_h^\\delta)^2\\delta^2}-\\frac{4\\sqrt{\\epsilon}}{\\delta} = 1-O(n\\epsilon+\\sqrt{\\epsilon})$$\nwhen taking $m_1^* = S/4$ and $m_2^* = \\sqrt{\\frac{S\\tau\\eta\\Delta_\\Psi}{\\gamma\\mu}}\\cdot\\frac{32}{3\\delta}.$ Note that the requirement on $\\epsilon$ further implies that $\\mathrm{Prob}(\\cA_1(m_1^*))\\leq 1/2$ and $\\mathrm{Prob}(\\cA_2(m_2^*))\\leq 1/4$, and hence $\\mathrm{Prob}\\big(\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)\\big)\\geq1/4$. By ignoring the Bregman divergence terms of Lemma \\ref{lemma:proxSARAH-grad-map-mid}, we obtain\n\\begin{eqnarray*}\n    \\frac{\\Delta_\\Psi}{\\gamma\\eta} &\\geq& \\frac{\\mu}{8} \\mathbb{E}\\bigg[\\sum_{s=1}^S\\sum_{k=0}^{\\tau_s-1}\\big\\|\\cGshe(x_{s,k})\\big\\|^2\\bigg]  \\geq \\frac{\\mu}{32}\\mathbb{E}\\bigg[\\sum_{(s,k)\\in\\mathcal{I}_2^c}\\big\\|\\cGshe(x_{s,k})\\big\\|^2\\,\\Big|\\,\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)\\bigg] \\nonumber.\n\\end{eqnarray*}\nConditioning on $\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)$, regardless of the random sets $\\mathcal{I}_1, \\mathcal{I}_2$ and $\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}$, we have  \n\\begin{eqnarray}\n    |\\mathcal{I}_2^c| = (S-|\\mathcal{I}_1|)\\tau+\\sum_{s\\in\\mathcal{I}_1}\\tau_s - |\\mathcal{I}_2|\\geq  \\frac{3S\\tau}{4} - m_2^*\\geq S\\tau\\Big(\\frac{3}{4}-\\frac{8\\eta\\sqrt{\\epsilon}}{3\\delta}\\Big) \\geq \\frac{S\\tau}{2}.\\nonumber\n\\end{eqnarray}\nwhere the second last inequality is because $|\\mathcal{I}_2|\\leq m_2^*\\leq \\frac{8S\\tau\\eta\\sqrt{\\epsilon}}{3\\delta}$, and the last inequality is because  $\\frac{8\\eta\\sqrt{\\epsilon}}{3\\delta}\\leq 2/9< 1/4$ since we require $\\epsilon\\leq\\frac{\\delta^2}{16}\\cdot\\frac{1}{9\\eta^2}$. As a result, \n\\begin{eqnarray}\n\\label{thm:proxSARAH-finite-66}\n    \\frac{\\Delta_\\Psi}{\\gamma\\eta} &\\geq& \\frac{\\mu}{32}\\cdot\\frac{S\\tau}{2}\\cdot\\mathbb{E}\\bigg[\\frac{|\\mathcal{I}_2^c|}{S\\tau/2}\\cdot\\frac{\\sum_{(s,k)\\in\\mathcal{I}_2^c}\\big\\|\\cGshe(x_{s,k})\\big\\|^2}{|\\mathcal{I}_2^c|}\\,\\Big|\\,\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)\\bigg] \\\\\n    & \\geq & \\frac{\\mu S\\tau}{64}\\mathbb{E}\\bigg[\\frac{\\sum_{(s,k)\\in\\mathcal{I}_2^c}\\big\\|\\cGshe(x_{s,k})\\big\\|^2}{|\\mathcal{I}_2^c|}\\,\\Big|\\,\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)\\bigg] \\nonumber\\\\\n    & = & \\frac{\\mu S\\tau}{64}\\mathbb{E}\\Big[\\big\\|\\cGphe(x_{\\mathrm{out}})\\big\\|^2\\,\\big|\\,\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)\\cap\\cA_3\\Big]\\,, \\nonumber\n\\end{eqnarray}\nwhere the last equality is due to the definition of $x_{\\mathrm{out}}$, $\\cA_3$, and the fact that $\\cGshe(\\cdot)$ coincides with $\\cGphe(\\cdot)$ in $\\mathcal{I}_2^c$. As a result, we have\n$$\\mathbb{E}\\Big[\\big\\|\\cGphe(x_{\\mathrm{out}})\\big\\|^2\\,\\big|\\,\\cA\\Big]\\leq \\frac{64\\Delta_\\Psi}{\\gamma\\eta\\mu\\tau S}\\leq 4\\epsilon.$$\nGiven the choice of $S$, $\\gamma$, $\\eta$, and the fact that $\\tau = \\lceil n/b\\rceil$, the total sample complexity will be \n$$S(n+b\\tau) = (n+b\\tau)\\cdot\\bigg\\lceil\\frac{16\\Delta_\\Psi}{\\tau\\gamma\\eta\\mu\\epsilon}\\bigg\\rceil = O\\left(\\frac{L\\kappa_h^\\delta\\Delta_\\Psi\\sqrt{n}}{\\mu\\epsilon}\\cdot\\Big(1+\\sqrt{b/\\tau}\\Big)\\right),$$\nwhich indicates an $O(\\sqrt{n}\\epsilon^{-1})$ sample complexity for all $b\\leq O(\\tau)$ (or equivalently, $b\\leq O(\\lceil\\sqrt{n}\\rceil)$). If larger batch size $b = \\lceil n^\\alpha\\rceil$ with $\\alpha\\in(1/2,1]$ is taken, then we obtain an $O(n^\\alpha\\epsilon^{-1})$ complexity. \n\\end{proof}","sha256_utf8":"108b6829d4d57bd55db3baaeddbbd13d80dea851488f2a34b218809d2dd8afb0"},{"start":168597,"end":168991,"start_line":1325,"end_line":1331,"raw_tex":"\\begin{lemma}[Three-Point Property of Tseng \\cite{tseng2008accelerated}]\n    \\label{lemma:Tseng-3point}\n    Let $\\phi(x)$ be a convex function, and let $D_h(\\cdot,\\cdot)$ be the Bregman distance for $h(\\cdot)$. For a given vector $z$, let\n    $z_+:=\\argmin_{x\\in Q} \\phi(x) + D_h(x,z).$\n    Then \n    $$\\phi(x)+D_h(x,z)\\geq\\phi(z_+) + D_h(z_+,z) + D_h(x,z_+), \\quad\\forall x\\in Q.$$\n\\end{lemma}","sha256_utf8":"5823b5632fa1a6033768c40d2564fc95b78dbc2a3fa158cf9b8ff8706a63c97a"},{"start":168995,"end":169988,"start_line":1335,"end_line":1346,"raw_tex":"\\begin{lemma}[Lemma 2 finite-sum case of \\cite{pham2020proxsarah}]\n\\label{lemma:proxSARAH-finite-sum}\n    Let $v_{s,k}$ be generated by \\eqref{alg:sarah-small}, suppose $|\\mathcal{B}_{s,k}| = b$ and the sampled index are uniformly randomly picked from $[n]$ with replacement, then  \n    \\begin{eqnarray}\n        \\mathbb{E}\\Big[\\|\\nabla f(x_{s,k})-v_{s,k}\\|^2\\,\\big|\\,x_{s,0}\\Big] \\leq \\sum_{j=0}^{k-1} \\mathbb{E}\\Big[\\|v_{s,j+1}-v_{s,j}\\|^2-\\|\\nabla f(x_{s,j+1})-\\nabla f(x_{s,j})\\|^2\\,\\big|\\, x_{s,0}\\Big], \\nonumber \n    \\end{eqnarray}\n    where the expectation term of $\\|v_{s,j+1}-v_{s,j}\\|^2$ satisfies\n    \\begin{eqnarray}\n        \\mathbb{E}\\Big[\\|v_{s,j+1}-v_{s,j}\\|^2\\,\\big|\\,x_{s,0}\\Big] \\leq  \\mathbb{E}\\bigg[\\|\\nabla f(x_{s,j+1})-\\nabla f(x_{s,j})\\|^2 + \\frac{1}{bn}\\sum_{i=1}^n\\|\\nabla f_i(x_{s,j+1})-\\nabla f_i(x_{s,j})\\|^2\\,\\big|\\,x_{s,0}\\bigg].  \\nonumber\n    \\end{eqnarray}  \n    In particular, we have slightly modified the second inequality to suit our analysis. \n\\end{lemma}","sha256_utf8":"c43893b1ab8a02d0fa702237517ce42ccbbf5073d88d77aa96c8bf667f1546d8"},{"start":173330,"end":173527,"start_line":1390,"end_line":1393,"raw_tex":"\\begin{lemma}\n    \\label{lemma:B.1}\n    Let $A$ be a symmetric matrix, and let $B\\succeq0$ be positive semidefinite matrix. Then $-B\\preceq A\\preceq B$ indicates that $\\|A\\|\\leq \\|B\\|$.\n\\end{lemma}","sha256_utf8":"dda4533c88b205b0723dde19beb68cfa8be43e44b6c0fed6a8b7b223f581945b"},{"start":173529,"end":174598,"start_line":1394,"end_line":1411,"raw_tex":"\\begin{proof}\n    For any symmetric but not semidefinite matrix $A$, it is easy to verify that \n    \\begin{equation}\n        \\label{lm:B.1-1}\n        \\|A\\| = \\max\\big\\{|\\lambda_{\\max}(A)|,|\\lambda_{\\min}(A)|\\big\\}.\n    \\end{equation}\n    Let $u\\neq0$ be the eigenvector of $A$ associated with the maximum eigenvalue.  Then by Rayleigh’s principle for maximum eigenvalue, $B\\succeq A$ indicates that $B-A\\succeq0$ and hence \n    \\begin{equation}\n        \\label{lm:B.1-2}\n        0\\leq \\frac{u^\\top(B-A)u}{u^\\top u} \\leq \\max_{x\\neq0}\\left\\{\\frac{x^\\top Bx}{x^\\top x}\\right\\} - \\frac{u^\\top(B-A)u}{u^\\top u} = \\lambda_{\\max}(B) - \\lambda_{\\max}(A). \n    \\end{equation}\n    Similarly, $A\\succeq-B$ indicates that $B-(-A)\\succeq0$, then \\eqref{lm:B.1-2} immediately gives\n    \\begin{equation}\n        \\label{lm:B.1-3}\n        0\\leq  \\lambda_{\\max}(B) - \\lambda_{\\max}(-A) = \\lambda_{\\max}(B) + \\lambda_{\\min}(A).\n    \\end{equation}\n    Combining \\eqref{lm:B.1-1}-\\eqref{lm:B.1-3} and the fact that $\\|B\\| = \\lambda_{\\max}(B)$ for p.s.d. matrix proves the lemma.\n\\end{proof}","sha256_utf8":"8b610ab764f277eea976008a056ddfec5c13c3d7b715186259df34980d1d7a80"},{"start":174721,"end":175585,"start_line":1413,"end_line":1422,"raw_tex":"\\begin{proof}\n     By Lemma \\ref{lemma:B.1} and the assumption that $f$ is $L$-smooth adaptable to $h$, we know \n     $$\\max\\left\\{\\|\\nabla^2f(x)\\|: x\\in\\cX\\right\\} \\leq \\max\\left\\{L\\|\\nabla^2h(x)\\|: x\\in\\cX\\right\\} = L\\cdot L_h(\\cX).$$\n     As $\\cX$ is convex, the line segment $[x,y]\\subseteq\\cX$ and hence \n     $$\\|\\nabla f(z_\\theta)-\\nabla f(y)\\|^2\\leq L^2L_h^2(\\cX)\\|z_\\theta-y\\|^2 = L^2\\theta^2 L_h^2(\\cX)\\|x-y\\|^2.$$\n     Combined with    the fact that $$D_h(x,y)\\geq\\frac{\\mu_h([x,y])}{2}\\|x-y\\|^2\\geq  \\frac{\\mu_h(\\cX)}{2}\\|x-y\\|^2,$$\n     we obtain\n     $$\\frac{\\|\\nabla f(z_\\theta)-\\nabla f(y)\\|^2}{2L^2\\mu_h(\\cX)} \\leq \\theta^2\\cdot\\frac{L_h^2(\\cX)}{\\mu_h^2(\\cX)}\\cdot\\frac{\\mu_h(\\cX)}{2}\\|x-y\\|^2\\leq \\theta^2\\kappa_h^2(\\cX)D_h(x,y),$$\n     where KC-regularity guarantees that $\\kappa_h(\\cX)$ is always upper bounded by $\\kappa_h^\\delta$.\n\\end{proof}","sha256_utf8":"8e06f09b0e70bd32ead6a1847f2b4e3bd888124a74b81b8e4e20fc3422ecdd5c"},{"start":175721,"end":178126,"start_line":1430,"end_line":1451,"raw_tex":"\\begin{proof}\n    Denote $\\widetilde{\\mathcal{G}}=\\frac{x_{s,k}-\\bar{x}_{s,k+1}}{\\eta}$, then $\\|\\cGshe(x_{s,k})-\\widetilde{\\mathcal{G}}\\| = \\frac{1}{\\eta}\\|\\bar{x}_{s,k+1}-\\hat{x}_{s,k+1}\\|$ holds by definition. \n    By the optimality of $\\hat{x}_{s,k+1}$ and $\\bar{x}_{s,k+1}$ for the corresponding subproblems, Tseng's three point property (Lemma \\ref{lemma:Tseng-3point}) indicates that\n    \\begin{eqnarray*}\n        &&\\langle v_{s,k}, \\bar{x}_{s,k+1}\\rangle + \\phi(\\bar{x}_{s,k+1}) + \\frac{D_h(\\bar{x}_{s,k+1},x_{s,k})}{\\eta} + \\frac{D_h(\\hat{x}_{s,k+1},\\bar{x}_{s,k+1})}{\\eta}\\qquad\\,\\,\\,\\\\\n        &\\leq& \\langle v_{s,k}, \\hat{x}_{s,k+1}\\rangle + \\phi(\\hat{x}_{s,k+1})+ \\frac{D_h(\\hat{x}_{s,k+1},x_{s,k})}{\\eta} \n    \\end{eqnarray*} \n    and \n    \\begin{eqnarray*}\n        && \\langle \\nabla f(x_{s,k}), \\hat{x}_{s,k+1}\\rangle + \\phi(\\hat{x}_{s,k+1})+\\frac{D_h(\\hat{x}_{s,k+1},x_{s,k})}{\\eta} + \\frac{D_h(\\bar{x}_{s,k+1},\\hat{x}_{s,k+1})}{\\eta}\\\\\n        &\\leq& \\langle \\nabla f(x_{s,k}), \\bar{x}_{s,k+1}\\rangle + \\phi(\\bar{x}_{s,k+1})+\\frac{D_h(\\bar{x}_{s,k+1},x_{s,k})}{\\eta}\n    \\end{eqnarray*} \n    Summing up the two inequalities and applying Lemma \\ref{lemma:grd-vs-grdmap} gives \n    \\begin{align} &\\frac{\\mu_h([\\bar{x}_{s,k+1},\\hat{x}_{s,k+1}])}{\\eta}\\cdot\\|\\bar{x}_{s,k+1}-\\hat{x}_{s,k+1}\\|^2 \\leq \\frac{D_h(\\bar{x}_{s,k+1},\\hat{x}_{s,k+1})}{\\eta} +  \\frac{D_h(\\hat{x}_{s,k+1},\\bar{x}_{s,k+1})}{\\eta}\\nonumber\\\\\n    &\\qquad\\qquad\\qquad\\qquad\\qquad\\,\\,\\,\\, \\leq  \\langle v_{s,k}-\\nabla f(x_{s,k}), \\hat{x}_{s,k+1}-\\bar{x}_{s,k+1}\\rangle \\leq  \\|v_{s,k}-\\nabla f(x_{s,k})\\|\\cdot\\|\\hat{x}_{s,k+1}-\\bar{x}_{s,k+1}\\|\\nonumber.\n    \\end{align} \n    Hence $\\|\\hat{x}_{s,k+1}-\\bar{x}_{s,k+1}\\|\\leq \\frac{\\eta\\|v_{s,k}-\\nabla f(x_{s,k})\\|}{\\mu_h([\\bar{x}_{s,k+1},\\hat{x}_{s,k+1}])}$, and $\\big\\|\\cGshe(x_{s,k})-\\widetilde{\\mathcal{G}}\\big\\|\\leq\\frac{\\|v_{s,k}-\\nabla f(x_{s,k})\\|}{\\mu_h([\\bar{x}_{s,k+1},\\hat{x}_{s,k+1}])}$. As a result, \n    \\begin{eqnarray*}\n        \\big\\|\\cGshe(x_{s,k})\\big\\|^2 \\leq \\left(\\big\\|\\widetilde{\\mathcal{G}}\\big\\|+\\big\\|\\cGshe(x_{s,k})-\\widetilde{\\mathcal{G}}\\big\\|\\right)^2\\leq  \\frac{2\\|x_{s,k}-\\bar{x}_{s,k+1}\\|^2}{\\eta^2} + \\frac{2\\|v_{s,k}-\\nabla f(x_{s,k})\\|^2}{\\mu_h^2(\\mathcal{X}_s)},\n    \\end{eqnarray*} \n    where the last inequality is because $(a+b)^2\\leq 2a^2+2b^2$ and $\\mu_h^2\\big([\\bar{x}_{s,k+1},\\hat{x}_{s,k+1}]\\big)\\geq\\mu_h^2(\\cX_s)$. \n\\end{proof}","sha256_utf8":"a27d2e1e52c5307995bb9607dc0e5dd01faee21e490052c0a0f1688f6fd0346b"},{"start":178195,"end":180729,"start_line":1456,"end_line":1475,"raw_tex":"\\begin{proof} \n    First of all, by the update rule of $x_{s,k+1}$, we have   \n    \\begin{eqnarray}\n    \\label{lm:descent-proxSARAH-finite-1}\n        \\Psi(x_{s,k+1}) \n        \\!\\!\\!& = & \\!\\!\\!f\\big(x_{s,k} + \\gamma (\\bar{x}_{s,k+1}-x_{s,k})\\big) + \\phi\\big((1-\\gamma)x_{s,k} + \\gamma\\bar{x}_{s,k+1}\\big) \\\\\n        &\\overset{(i)}{\\leq}& \\!\\!\\!f(x_{s,k}) + \\gamma\\langle\\nabla f(x_{s,k}),\\bar{x}_{s,k+1}\\!-x_{s,k}\\rangle \\!+\\! LD_h\\left(x_{s,k+1},x_{s,k}\\right) \\!+\\! (1-\\gamma)\\phi(x_{s,k})+\\gamma\\phi(\\bar{x}_{s,k+1})\\nonumber\\\\\n        & \\overset{(ii)}{\\leq} & \\!\\!\\!\\Psi(x_{s,k}) + L\\kappa_h^\\delta\\gamma^2D_h(\\bar{x}_{s,k+1},x_{s,k}) + \\gamma\\langle\\mathcal{E}_{s,k}+v_{s,k},\\bar{x}_{s,k+1}-x_{s,k}\\rangle  + \\gamma\\left(\\phi(\\bar{x}_{s,k+1})-\\phi(x_{s,k})\\right)\\nonumber\\\\\n        &\\overset{(iii)}{\\leq}&\\!\\!\\! \\Psi(x_{s,k}) + L\\kappa_h^\\delta\\gamma^2D_h(\\bar{x}_{s,k+1},x_{s,k}) + \\frac{\\gamma\\eta\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h(\\mathcal{X}_s)} + \\frac{\\gamma\\mu_h(\\mathcal{X}_s)}{4\\eta} \\|\\bar{x}_{s,k+1}-x_{s,k}\\|^2\\nonumber\\\\\n        & & \\!\\!\\!-\\frac{\\gamma}{\\eta}D_h(\\bar{x}_{s,k+1},x_{s,k}) - \\frac{\\gamma}{\\eta}D_h(x_{s,k},\\bar{x}_{s,k+1})\\nonumber\\\\\n        &\\leq & \\!\\!\\!\\Psi(x_{s,k}) - \\left(\\frac{\\gamma}{\\eta}- L\\kappa_h^\\delta\\gamma^2\\right)D_h(\\bar{x}_{s,k+1},x_{s,k}) -\\frac{\\gamma}{2\\eta}D_h(x_{s,k},\\bar{x}_{s,k+1})+ \\frac{\\gamma\\eta\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h(\\mathcal{X}_s)} \\,,\\nonumber\n    \\end{eqnarray}\n    where (i) is due to Assumption \\ref{assumption:L-smad-finite-sum}, Lemma \\ref{lemma:descent-lemma}, and the convexity of $\\phi$, (ii) is due to the definition of $\\mathcal{E}_{s,k}$ in Lemma \\ref{lemma:restricted-grd-mapping} and the following scaling property\n    \\begin{eqnarray}\n        D_h\\left(x_{s,k+1},x_{s,k}\\right)  \\leq \\frac{L_h(\\mathcal{X}_s)}{2}\\|x_{s,k+1}-x_{s,k}\\|^2 = \\frac{\\gamma^2L_h(\\mathcal{X}_s)}{2}\\|\\bar{x}_{s,k+1}-x_{s,k}\\|^2 \\leq \\gamma^2\\kappa_h^\\delta D_h(\\bar{x}_{s,k+1},x_{s,k})\\,,\\nonumber\n    \\end{eqnarray}\n    and (iii) is due to the optimality of  $\\bar{x}_{s,k+1}$ to the corresponding subproblem and Tseng's three point property (Lemma \\ref{lemma:Tseng-3point}). Finally, by Lemma \\ref{lemma:restricted-grd-mapping}, we also have\n    $$\\big\\|\\cGshe(x_{s,k})\\big\\|^2 \\leq \\frac{2\\|x_{s,k}-\\bar{x}_{s,k+1}\\|^2}{\\eta^2} + \\frac{2\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h^2(\\mathcal{X}_s)}\\,.$$\n    Multiplying both sides of the above inequality by $\\frac{\\gamma\\eta\\mu_h(\\mathcal{X}_s)}{8}$ and add it to \\eqref{lm:descent-proxSARAH-finite-1} proves the lemma. \n\\end{proof}","sha256_utf8":"98a4d486ed34efde9b645ee1043966351f32a6555eb1e8d290f7b315e92ca678"},{"start":180796,"end":182760,"start_line":1480,"end_line":1499,"raw_tex":"\\begin{proof}\n    First, substituting the gradient estimation bound in Lemma \\ref{lemma:err-sarah} to Lemma \\ref{lemma:descent-proxSARAH-finite}, we have the following descent result throughout the $s$-th epoch\n\n\\begin{align}\n    \\label{lm:proxSARAH-grad-map-mid-1}\n    &\\mathbb{E}\\Big[\\Psi(x_{s,\\tau_s}) \\,\\big|\\, x_{s,0}\\Big] \\!\\leq \\Psi(x_{s,0}) \\!-\\! \\mathbb{E}\\bigg[\\frac{\\gamma\\eta\\mu_h(\\mathcal{X}_s)}{8}\\!\\sum_{k=0}^{\\tau_s-1}\\!\\big\\|\\cGshe(x_{s,k})\\big\\|^2 \\!+\\!\\bigg(\\!\\frac{\\gamma}{\\eta}- L\\kappa_h^\\delta\\gamma^2\\!\\bigg)D_h(\\bar{x}_{s,k+1},x_{s,k}) \\,\\Big|\\, x_{s,0}\\bigg]\\nonumber\\\\\n    &\\qquad\\qquad\\qquad\\qquad\\,\\, +\\frac{5\\gamma\\eta}{4\\mu_h(\\mathcal{X}_s)}\\cdot\\frac{2\\gamma^2L^2\\kappa_h^\\delta L_h(\\mathcal{X}_s)}{b}\\mathbb{E}\\bigg[\\sum_{k=0}^{\\tau_s-1}\\sum_{j=0}^{k-1} D_h(\\bar{x}_{s,j+1},x_{s,j}) \\,\\Big|\\, x_{s,0}\\bigg]\\\\\n    &\\quad\\,\\leq \\Psi(x_{s,0}) \\!-\\! \\mathbb{E}\\bigg[\\frac{\\gamma\\eta\\mu_h(\\mathcal{X}_s)}{8}\\!\\sum_{k=0}^{\\tau_s-1}\\big\\|\\mathcal{G}_{s,k}^\\eta(x_{s,k})\\big\\|^2 \\!+\\bigg(\\frac{\\gamma}{\\eta}- L\\kappa_h^\\delta\\gamma^2-\\frac{5\\tau\\gamma^3\\eta L^2(\\kappa_h^\\delta)^2}{2b}\\bigg)D_h(\\bar{x}_{s,k+1},x_{s,k}) \\,\\Big|\\, x_{s,0}\\bigg]\\nonumber\n\\end{align}\n\n\\noindent Suppose we choose $\\eta  = \\frac{\\sqrt{2\\tau}}{\\sqrt{7\\tau}+\\sqrt{2b}}$ and we choose $\\gamma = \\frac{\\sqrt{b}}{L\\kappa_h^\\delta\\sqrt{\\tau}}$. Then we have \n\\begin{eqnarray}\n    \\label{lm:proxSARAH-grad-map-mid-2}\n    &&\\frac{\\gamma}{\\eta} - L\\kappa_h^\\delta\\gamma^2 - \\frac{5\\tau\\gamma^3\\eta L^2(\\kappa_h^\\delta)^2}{2b}\\nonumber\\\\\n    & = & \\gamma\\eta\\bigg(\\frac{1}{\\eta^2} - \\frac{L\\kappa_h^\\delta\\gamma}{\\eta} - \\frac{5\\tau\\gamma^2 L^2(\\kappa_h^\\delta)^2}{2b}\\bigg)\\\\\n    &=& \\gamma\\eta\\bigg(1+\\frac{\\sqrt{14} -\\sqrt{3.5}}{\\sqrt{\\tau/b}}\\bigg)\\nonumber \\\\\n    &\\geq& \\gamma\\eta\\,.\\nonumber\n\\end{eqnarray} \nSubstitute the bound to the previous inequality, summing up over all epochs, and taking the expectation over all randomness proves the lemma. \n\\end{proof}","sha256_utf8":"aea62fd8fd421846f7c091ad48fb237928438baca9eb03a258b0e26aa08cbe28"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"随机方差缩减复杂度","source_units":[{"start":86067,"end":87349,"start_line":691,"end_line":695,"raw_tex":"\\begin{theorem}\n    \\label{theorem:proxSARAH-finite}\n    For any constant batch size $|\\mathcal{B}_{s,k}|=b\\in[n]$, let us set the epoch length as $\\tau = \\lceil n/b\\rceil$, step size $\\eta = \\frac{\\sqrt{2\\tau}}{\\sqrt{7\\tau}+\\sqrt{2b}}$, interpolation coefficient $\\gamma = \\frac{\\sqrt{b}}{L\\kappa_h^\\delta\\sqrt{\\tau}}$, and total epoch number $S = \\big\\lceil\\frac{16\\Delta_\\Psi}{\\tau\\gamma\\eta\\mu\\epsilon}\\big\\rceil$. Suppose the target accuracy satisfies $\\epsilon\\leq \\frac{\\delta^2}{16}\\cdot\\min\\big\\{\\frac{L^2(\\kappa_h^\\delta)^2}{b\\tau},\\frac{1}{9\\eta^2}\\big\\} = O(1/n)$ and let $x_{\\mathrm{out}}$ be uniformly randomly selected from all iterations, then there is a high probability event $\\mathcal{A}$ such that $$\\mathbb{E}\\Big[\\big\\|\\cGphe(x_{\\mathrm{out}})\\big\\|^2 \\,\\big|\\, \\mathcal{A}\\Big]\\leq 4\\epsilon\\qquad\\mbox{and}\\qquad \\mathrm{Prob}\\left(\\mathcal{A}\\right)\\geq1 - \\frac{8\\eta\\tau b\\epsilon}{L^2(\\kappa_h^\\delta)^2\\delta^2}-\\frac{4\\sqrt{\\epsilon}}{\\delta}.$$\n    In particular, $\\mathrm{Prob}\\left(\\mathcal{A}\\right) \\geq 1-O(n\\epsilon+\\sqrt{\\epsilon})\\to1$ as $\\epsilon\\to0.$ Suppose we take the batch size $b=O(n^\\alpha)$, $\\alpha\\in[0,1]$, then the total number of samples consumed is $O\\big(\\epsilon^{-1}\\cdot n^{\\max\\{\\alpha,\\frac{1}{2}\\}}\\big)$.\n\\end{theorem}","sha256_utf8":"6f5a8b479014edf0b284fb189db40b3881e258edf6e357b01fce4df05862651d"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"主定理完整证明","source_units":[{"start":88120,"end":98744,"start_line":702,"end_line":782,"raw_tex":"\\begin{proof}\nBy lemma \\ref{lemma:proxSARAH-grad-map-mid}, ignoring the restricted primal gradient mapping terms yields  \n\\begin{eqnarray} \n\\frac{\\Delta_\\Psi}{\\gamma\\eta} \\geq\n\\mathbb{E}\\bigg[\\sum_{s=1}^{S}\\sum_{k=0}^{\\tau_s-1}D_h(\\bar{x}_{s,k+1},x_{s,k})\\bigg] \n\\geq \\frac{\\mu}{2\\gamma^2}\\mathbb{E}\\bigg[\\sum_{s=1}^{S}\\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|^2\\bigg]\\,. \\nonumber\n\\end{eqnarray}\n\nBy expanding the expectation over all possible $\\mathcal{I}_1, \\mathcal{I}_2,$ and $\\{\\tau_s\\}$, we have for all $m_1,m_2>0$ that \n\\begin{eqnarray} \n\\label{thm:proxSARAH-finite-1}\n\\frac{2\\gamma\\Delta_\\Psi}{\\mu\\eta} \\!\\!\\!\\!& \\geq & \\!\\!\\!\\!\\mathbb{E}\\bigg[\\sum_{s=1}^{S}\\sum_{k=0}^{\\tau_s-1}\\!\\|x_{s,k+1}\\!-\\!x_{s,k}\\|^2\\bigg] \\\\\n& \\geq &\\!\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\!\\!\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\mathbb{E}\\bigg[\\!\\sum_{s=1}^S\\!\\sum_{k=0}^{\\tau_s-1}\\!\\|x_{s,k+1}-x_{s,k}\\|^2 \\,\\Big|\\, \\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\!\\bigg]\\nonumber\\\\\n& \\geq &\\!\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\!\\!\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\mathbb{E}\\bigg[\\!\\sum_{s\\in\\mathcal{I}_1}\\!\\!\\sum_{k=0}^{\\tau_s-1}\\!\\|x_{s,k+1}-x_{s,k}\\|^2 \\,\\Big|\\, \\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\!\\bigg]\\nonumber \n\\end{eqnarray}\nNote that for $\\forall s\\in\\mathcal{I}_1$, according to the discussion in Lemma \\ref{lemma:event-grad-map}, we have  $\\|x_{s,\\tau_s}-x_{s,0}\\|  \\geq \\delta/4.$\nRepeating the analysis of \\eqref{lm:event-halfway-2} and \\eqref{lm:event-halfway-3} yields $\\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|^2  \\geq \\frac{\\delta^2}{16\\tau_s}$, which always holds true. Then substituting this lower bound to \\eqref{thm:proxSARAH-finite-1} gives \n\\begin{equation}\n    \\label{thm:proxSARAH-finite-2}\n    \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\cdot\\sum_{s\\in\\mathcal{I}_1}\\tau_s^{-1}\\leq\\frac{32\\gamma\\Delta_\\Psi}{\\mu\\eta\\delta^2}\\,.\n\\end{equation} \nDefine the events  \n\\begin{equation}\n    \\label{defn:event-A3}\n    \\mathcal{A}_3:=\\big\\{\\omega: \\mbox{the index of } x_{\\mathrm{out}}\\mbox{ is from }\\mathcal{I}_2^c\\big\\}\\quad\\mbox{and}\\quad \\mathcal{A}:=\\mathcal{A}_1^c(m_1)\\cap\\mathcal{A}_2^c(m_2)\\cap\\mathcal{A}_3.\n\\end{equation} \nThen we have\n\\begin{eqnarray}\n    \\label{thm:proxSARAH-finite-33}\n    \\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\mathrm{Prob}\\left(\\mathcal{A}\\right)\\!\\!\\! &=&\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\cdot\\mathrm{Prob}\\big(\\cA_3\\mid\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\\\\n    &=&\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\cdot\\left(1-\\frac{|\\mathcal{I}_2|}{(S-|\\mathcal{I}_1|)\\tau+\\sum_{s\\in\\mathcal{I}_1}\\tau_s}\\right)\\nonumber\\\\\n    & \\geq &\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\cdot\\left(1-\\frac{m_2}{(S-|\\mathcal{I}_1|)\\tau+\\sum_{s\\in\\mathcal{I}_1}\\tau_s}\\right).\\nonumber\n\\end{eqnarray}\nNote that for any positive numbers $y_1,\\cdots,y_S>0$,  the arithmetic-harmonic inequality states that $$\\frac{y_1+y_2+\\cdots+y_S}{S}\\geq\\frac{S}{\\frac{1}{y_1}+\\frac{1}{y_2}+\\cdots+\\frac{1}{y_S}}\\,.$$ \nApplying this inequality gives \n$$\\frac{1}{\\tau} + \\frac{\\sum_{s\\in\\mathcal{I}_1}\\tau_s^{-1}}{S}= \\frac{(S-|\\mathcal{I}|_1)\\tau^{-1} + \\sum_{s\\in\\mathcal{I}_1}\\tau_s^{-1}}{S}\\geq \\frac{S}{(S-|\\mathcal{I}|_1)\\tau+ \\sum_{s\\in\\mathcal{I}_1}\\tau_s}\\,,$$\nwhich implies that \n$$1-\\frac{m_2}{(S-|\\mathcal{I}_1|)\\tau+\\sum_{s\\in\\mathcal{I}_1}\\tau_s}\\geq 1-\\frac{m_2}{S\\tau}-\\frac{m_2}{S^2}\\sum_{s\\in\\mathcal{I}_1}\\tau_s^{-1}\\,.$$\nSubstituting this bound to \\eqref{thm:proxSARAH-finite-33} and setting $m_1^* = S/4$ yields\n\\begin{eqnarray}\n    \\label{thm:proxSARAH-finite-44}\n    \\!\\!\\!\\mathrm{Prob}\\left(\\mathcal{A}\\right) \\!\\!\\!\\!& \\geq &\\!\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1^*}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\cdot\\left(1-\\frac{m_2}{S\\tau}-\\frac{m_2}{S^2}\\sum_{s\\in\\mathcal{I}_1}\\tau_s^{-1}\\right)\\nonumber\\\\\n    & \\overset{(i)}{\\geq} &\\!\\!\\!\\! \\left(1-\\frac{m_2}{S\\tau}\\right)\\mathrm{Prob}\\left(\\mathcal{A}_1^c(m_1^*)\\cap\\mathcal{A}_2^c(m_2)\\right) - \\frac{m_2}{S^2}\\cdot\\frac{32\\gamma\\Delta_\\Psi}{\\mu\\eta\\delta^2}\\\\\n    & \\geq & \\!\\!\\!\\! \\left(1-\\frac{m_2}{S\\tau}\\right)\\big(1-\\mathrm{Prob}\\left(\\mathcal{A}_1(m_1^*)\\right)-\\mathrm{Prob}\\left(\\mathcal{A}_2(m_2)\\right)\\big) - \\frac{m_2}{S^2}\\cdot\\frac{32\\gamma\\Delta_\\Psi}{\\mu\\eta\\delta^2}\\nonumber\\\\\n    & \\overset{(ii)}{\\geq} & \\!\\!\\!\\! 1 - \\frac{8\\eta\\tau b\\epsilon}{L^2(\\kappa_h^\\delta)^2\\delta^2} - \\frac{128\\eta\\Delta_\\Psi}{\\gamma\\mu\\delta^2\\cdot m_2} - \\frac{m_2}{S\\tau} - \\frac{m_2}{S^2}\\cdot\\frac{32\\gamma\\Delta_\\Psi}{\\mu\\eta\\delta^2}\\nonumber\\\\\n    & \\overset{(iii)}{\\geq} & \\!\\!\\!\\! 1 - \\frac{8\\eta\\tau b\\epsilon}{L^2(\\kappa_h^\\delta)^2\\delta^2} - \\frac{128\\eta\\Delta_\\Psi}{m_2\\gamma\\mu\\delta^2} - \\frac{9m_2}{8S\\tau}\\nonumber\n\\end{eqnarray}\nwhere (i) is due to \\eqref{thm:proxSARAH-finite-2} and the fact that \n\\begin{equation}\n    \\label{thm:proxSARAH-finite-55}\n    \\sum_{|\\mathcal{I}_1|<m_1^*}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big) = \\mathrm{Prob}\\big(\\mathcal{A}_1^c(m_1^*)\\cap\\mathcal{A}_2^c(m_2)\\big)\n\\end{equation}\n(ii) is due to Lemma \\ref{lemma:event-halway} with $m_1^*=S/4$ and $S = \\big\\lceil\\frac{16\\Delta_\\Psi}{\\tau\\gamma\\eta\\mu\\epsilon}\\big\\rceil$: \n$$\\mathrm{Prob}\\big(\\mathcal{A}_1(S/4)\\big)\\leq \\frac{32\\gamma\\tau\\Delta_\\Psi}{\\mu\\delta^2S/4} = \\frac{16\\Delta_\\Psi}{S\\tau\\gamma\\eta\\mu}\\cdot\\frac{8\\gamma^2\\tau^2\\eta}{\\delta^2} \\leq \\frac{8\\tau b\\epsilon}{L^2(\\kappa_h^\\delta)^2\\delta^2},$$\nand (iii) is because we require $\\epsilon\\leq \\frac{\\delta^2}{16}\\cdot\\min\\left\\{\\frac{L^2(\\kappa_h^\\delta)^2}{b\\tau},\\frac{1}{9\\eta^2}\\right\\} = O(1/n)$ such that \n$$\\frac{m_2}{S^2}\\cdot\\frac{32\\gamma\\Delta_\\Psi}{\\mu\\eta\\delta^2} = \\frac{m_2}{S\\tau}\\cdot\\frac{16\\Delta_\\Psi}{S\\tau\\gamma\\eta\\mu}\\cdot\\frac{2\\gamma^2\\tau^2}{\\delta^2}\\leq \\frac{m_2}{8S\\tau}$$\nTherefore, to maximize the above probability, we can choose $m_2^*=\\sqrt{\\frac{S\\tau\\eta\\Delta_\\Psi}{\\gamma\\mu}}\\cdot\\frac{32}{3\\delta}$ such that\n$$\\frac{128\\eta\\Delta_\\Psi}{m_2^*\\gamma\\mu\\delta^2} + \\frac{9m_2^*}{8S\\tau} = 2\\sqrt{\\frac{128\\eta\\Delta_\\Psi}{m_2^*\\gamma\\mu\\delta^2} \\cdot  \\frac{9m_2^*}{8S\\tau}} = \\sqrt{\\frac{\\eta\\Delta_\\Psi}{\\gamma\\mu S\\tau}}\\cdot\\frac{24}{\\delta}\\leq \\frac{6\\eta\\sqrt{\\epsilon}}{\\delta}\\leq \\frac{4\\sqrt{\\epsilon}}{\\delta}\\,, $$\nwhere the last inequality is due to the fact that $\\eta = \\frac{\\sqrt{2\\tau}}{\\sqrt{7\\tau}+\\sqrt{2b}}\\leq\\sqrt{2/7}.$ Combining all the above discussion, we can conclude that  \n$$\\mathrm{Prob}\\left(\\mathcal{A}\\right)\\geq1 - \\frac{8\\eta\\tau b\\epsilon}{L^2(\\kappa_h^\\delta)^2\\delta^2}-\\frac{4\\sqrt{\\epsilon}}{\\delta} = 1-O(n\\epsilon+\\sqrt{\\epsilon})$$\nwhen taking $m_1^* = S/4$ and $m_2^* = \\sqrt{\\frac{S\\tau\\eta\\Delta_\\Psi}{\\gamma\\mu}}\\cdot\\frac{32}{3\\delta}.$ Note that the requirement on $\\epsilon$ further implies that $\\mathrm{Prob}(\\cA_1(m_1^*))\\leq 1/2$ and $\\mathrm{Prob}(\\cA_2(m_2^*))\\leq 1/4$, and hence $\\mathrm{Prob}\\big(\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)\\big)\\geq1/4$. By ignoring the Bregman divergence terms of Lemma \\ref{lemma:proxSARAH-grad-map-mid}, we obtain\n\\begin{eqnarray*}\n    \\frac{\\Delta_\\Psi}{\\gamma\\eta} &\\geq& \\frac{\\mu}{8} \\mathbb{E}\\bigg[\\sum_{s=1}^S\\sum_{k=0}^{\\tau_s-1}\\big\\|\\cGshe(x_{s,k})\\big\\|^2\\bigg]  \\geq \\frac{\\mu}{32}\\mathbb{E}\\bigg[\\sum_{(s,k)\\in\\mathcal{I}_2^c}\\big\\|\\cGshe(x_{s,k})\\big\\|^2\\,\\Big|\\,\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)\\bigg] \\nonumber.\n\\end{eqnarray*}\nConditioning on $\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)$, regardless of the random sets $\\mathcal{I}_1, \\mathcal{I}_2$ and $\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}$, we have  \n\\begin{eqnarray}\n    |\\mathcal{I}_2^c| = (S-|\\mathcal{I}_1|)\\tau+\\sum_{s\\in\\mathcal{I}_1}\\tau_s - |\\mathcal{I}_2|\\geq  \\frac{3S\\tau}{4} - m_2^*\\geq S\\tau\\Big(\\frac{3}{4}-\\frac{8\\eta\\sqrt{\\epsilon}}{3\\delta}\\Big) \\geq \\frac{S\\tau}{2}.\\nonumber\n\\end{eqnarray}\nwhere the second last inequality is because $|\\mathcal{I}_2|\\leq m_2^*\\leq \\frac{8S\\tau\\eta\\sqrt{\\epsilon}}{3\\delta}$, and the last inequality is because  $\\frac{8\\eta\\sqrt{\\epsilon}}{3\\delta}\\leq 2/9< 1/4$ since we require $\\epsilon\\leq\\frac{\\delta^2}{16}\\cdot\\frac{1}{9\\eta^2}$. As a result, \n\\begin{eqnarray}\n\\label{thm:proxSARAH-finite-66}\n    \\frac{\\Delta_\\Psi}{\\gamma\\eta} &\\geq& \\frac{\\mu}{32}\\cdot\\frac{S\\tau}{2}\\cdot\\mathbb{E}\\bigg[\\frac{|\\mathcal{I}_2^c|}{S\\tau/2}\\cdot\\frac{\\sum_{(s,k)\\in\\mathcal{I}_2^c}\\big\\|\\cGshe(x_{s,k})\\big\\|^2}{|\\mathcal{I}_2^c|}\\,\\Big|\\,\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)\\bigg] \\\\\n    & \\geq & \\frac{\\mu S\\tau}{64}\\mathbb{E}\\bigg[\\frac{\\sum_{(s,k)\\in\\mathcal{I}_2^c}\\big\\|\\cGshe(x_{s,k})\\big\\|^2}{|\\mathcal{I}_2^c|}\\,\\Big|\\,\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)\\bigg] \\nonumber\\\\\n    & = & \\frac{\\mu S\\tau}{64}\\mathbb{E}\\Big[\\big\\|\\cGphe(x_{\\mathrm{out}})\\big\\|^2\\,\\big|\\,\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)\\cap\\cA_3\\Big]\\,, \\nonumber\n\\end{eqnarray}\nwhere the last equality is due to the definition of $x_{\\mathrm{out}}$, $\\cA_3$, and the fact that $\\cGshe(\\cdot)$ coincides with $\\cGphe(\\cdot)$ in $\\mathcal{I}_2^c$. As a result, we have\n$$\\mathbb{E}\\Big[\\big\\|\\cGphe(x_{\\mathrm{out}})\\big\\|^2\\,\\big|\\,\\cA\\Big]\\leq \\frac{64\\Delta_\\Psi}{\\gamma\\eta\\mu\\tau S}\\leq 4\\epsilon.$$\nGiven the choice of $S$, $\\gamma$, $\\eta$, and the fact that $\\tau = \\lceil n/b\\rceil$, the total sample complexity will be \n$$S(n+b\\tau) = (n+b\\tau)\\cdot\\bigg\\lceil\\frac{16\\Delta_\\Psi}{\\tau\\gamma\\eta\\mu\\epsilon}\\bigg\\rceil = O\\left(\\frac{L\\kappa_h^\\delta\\Delta_\\Psi\\sqrt{n}}{\\mu\\epsilon}\\cdot\\Big(1+\\sqrt{b/\\tau}\\Big)\\right),$$\nwhich indicates an $O(\\sqrt{n}\\epsilon^{-1})$ sample complexity for all $b\\leq O(\\tau)$ (or equivalently, $b\\leq O(\\lceil\\sqrt{n}\\rceil)$). If larger batch size $b = \\lceil n^\\alpha\\rceil$ with $\\alpha\\in(1/2,1]$ is taken, then we obtain an $O(n^\\alpha\\epsilon^{-1})$ complexity. \n\\end{proof}","sha256_utf8":"108b6829d4d57bd55db3baaeddbbd13d80dea851488f2a34b218809d2dd8afb0"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"累计位移能量","source_units":[{"start":88120,"end":88518,"start_line":702,"end_line":708,"raw_tex":"\\begin{proof}\nBy lemma \\ref{lemma:proxSARAH-grad-map-mid}, ignoring the restricted primal gradient mapping terms yields  \n\\begin{eqnarray} \n\\frac{\\Delta_\\Psi}{\\gamma\\eta} \\geq\n\\mathbb{E}\\bigg[\\sum_{s=1}^{S}\\sum_{k=0}^{\\tau_s-1}D_h(\\bar{x}_{s,k+1},x_{s,k})\\bigg] \n\\geq \\frac{\\mu}{2\\gamma^2}\\mathbb{E}\\bigg[\\sum_{s=1}^{S}\\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|^2\\bigg]\\,. \\nonumber\n\\end{eqnarray}\n","sha256_utf8":"59d121dd7062f8f123bef4e8c3baf2821cf8496d0d9d192102c0dd55b018f5b7"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"按随机历程展开","source_units":[{"start":88519,"end":89605,"start_line":710,"end_line":716,"raw_tex":"By expanding the expectation over all possible $\\mathcal{I}_1, \\mathcal{I}_2,$ and $\\{\\tau_s\\}$, we have for all $m_1,m_2>0$ that \n\\begin{eqnarray} \n\\label{thm:proxSARAH-finite-1}\n\\frac{2\\gamma\\Delta_\\Psi}{\\mu\\eta} \\!\\!\\!\\!& \\geq & \\!\\!\\!\\!\\mathbb{E}\\bigg[\\sum_{s=1}^{S}\\sum_{k=0}^{\\tau_s-1}\\!\\|x_{s,k+1}\\!-\\!x_{s,k}\\|^2\\bigg] \\\\\n& \\geq &\\!\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\!\\!\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\mathbb{E}\\bigg[\\!\\sum_{s=1}^S\\!\\sum_{k=0}^{\\tau_s-1}\\!\\|x_{s,k+1}-x_{s,k}\\|^2 \\,\\Big|\\, \\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\!\\bigg]\\nonumber\\\\\n& \\geq &\\!\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\!\\!\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\mathbb{E}\\bigg[\\!\\sum_{s\\in\\mathcal{I}_1}\\!\\!\\sum_{k=0}^{\\tau_s-1}\\!\\|x_{s,k+1}-x_{s,k}\\|^2 \\,\\Big|\\, \\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\!\\bigg]\\nonumber \n\\end{eqnarray}\n","sha256_utf8":"b0b073f67235f23215b0c81aa941850fe142c62c4d0585a8a20fbb131b3498cd"}]},{"id":"S005","semantic_type":"proof_step","kind":"proof_step","title":"提前终止轮次代价","source_units":[{"start":89605,"end":90376,"start_line":717,"end_line":722,"raw_tex":"Note that for $\\forall s\\in\\mathcal{I}_1$, according to the discussion in Lemma \\ref{lemma:event-grad-map}, we have  $\\|x_{s,\\tau_s}-x_{s,0}\\|  \\geq \\delta/4.$\nRepeating the analysis of \\eqref{lm:event-halfway-2} and \\eqref{lm:event-halfway-3} yields $\\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|^2  \\geq \\frac{\\delta^2}{16\\tau_s}$, which always holds true. Then substituting this lower bound to \\eqref{thm:proxSARAH-finite-1} gives \n\\begin{equation}\n    \\label{thm:proxSARAH-finite-2}\n    \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\cdot\\sum_{s\\in\\mathcal{I}_1}\\tau_s^{-1}\\leq\\frac{32\\gamma\\Delta_\\Psi}{\\mu\\eta\\delta^2}\\,.\n\\end{equation} \n","sha256_utf8":"aebcfef4fb911c0fb91f87e900ad941661c07dc3561a7c90c4adc228cb7db5e6"}]},{"id":"S006","semantic_type":"proof_step","kind":"proof_step","title":"定义输出成功事件","source_units":[{"start":90376,"end":91684,"start_line":723,"end_line":734,"raw_tex":"Define the events  \n\\begin{equation}\n    \\label{defn:event-A3}\n    \\mathcal{A}_3:=\\big\\{\\omega: \\mbox{the index of } x_{\\mathrm{out}}\\mbox{ is from }\\mathcal{I}_2^c\\big\\}\\quad\\mbox{and}\\quad \\mathcal{A}:=\\mathcal{A}_1^c(m_1)\\cap\\mathcal{A}_2^c(m_2)\\cap\\mathcal{A}_3.\n\\end{equation} \nThen we have\n\\begin{eqnarray}\n    \\label{thm:proxSARAH-finite-33}\n    \\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\mathrm{Prob}\\left(\\mathcal{A}\\right)\\!\\!\\! &=&\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\cdot\\mathrm{Prob}\\big(\\cA_3\\mid\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\\\\n    &=&\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\cdot\\left(1-\\frac{|\\mathcal{I}_2|}{(S-|\\mathcal{I}_1|)\\tau+\\sum_{s\\in\\mathcal{I}_1}\\tau_s}\\right)\\nonumber\\\\\n    & \\geq &\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\cdot\\left(1-\\frac{m_2}{(S-|\\mathcal{I}_1|)\\tau+\\sum_{s\\in\\mathcal{I}_1}\\tau_s}\\right).\\nonumber\n\\end{eqnarray}\n","sha256_utf8":"772defc2bc77a665aacbae67aa88b6e1c4f766f11fb236af3b514ae5e4ca31ba"}]},{"id":"S007","semantic_type":"proof_step","kind":"proof_step","title":"调和平均控制轮长","source_units":[{"start":91684,"end":92307,"start_line":735,"end_line":739,"raw_tex":"Note that for any positive numbers $y_1,\\cdots,y_S>0$,  the arithmetic-harmonic inequality states that $$\\frac{y_1+y_2+\\cdots+y_S}{S}\\geq\\frac{S}{\\frac{1}{y_1}+\\frac{1}{y_2}+\\cdots+\\frac{1}{y_S}}\\,.$$ \nApplying this inequality gives \n$$\\frac{1}{\\tau} + \\frac{\\sum_{s\\in\\mathcal{I}_1}\\tau_s^{-1}}{S}= \\frac{(S-|\\mathcal{I}|_1)\\tau^{-1} + \\sum_{s\\in\\mathcal{I}_1}\\tau_s^{-1}}{S}\\geq \\frac{S}{(S-|\\mathcal{I}|_1)\\tau+ \\sum_{s\\in\\mathcal{I}_1}\\tau_s}\\,,$$\nwhich implies that \n$$1-\\frac{m_2}{(S-|\\mathcal{I}_1|)\\tau+\\sum_{s\\in\\mathcal{I}_1}\\tau_s}\\geq 1-\\frac{m_2}{S\\tau}-\\frac{m_2}{S^2}\\sum_{s\\in\\mathcal{I}_1}\\tau_s^{-1}\\,.$$\n","sha256_utf8":"1d504cbd72e2a9e0f4459e5986b54569ac30f1789ace1d1d2754091c55d2c7de"}]},{"id":"S008","semantic_type":"proof_step","kind":"proof_step","title":"合并失败概率","source_units":[{"start":92307,"end":94092,"start_line":740,"end_line":753,"raw_tex":"Substituting this bound to \\eqref{thm:proxSARAH-finite-33} and setting $m_1^* = S/4$ yields\n\\begin{eqnarray}\n    \\label{thm:proxSARAH-finite-44}\n    \\!\\!\\!\\mathrm{Prob}\\left(\\mathcal{A}\\right) \\!\\!\\!\\!& \\geq &\\!\\!\\!\\! \\sum_{|\\mathcal{I}_1|<m_1^*}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big)\\cdot\\left(1-\\frac{m_2}{S\\tau}-\\frac{m_2}{S^2}\\sum_{s\\in\\mathcal{I}_1}\\tau_s^{-1}\\right)\\nonumber\\\\\n    & \\overset{(i)}{\\geq} &\\!\\!\\!\\! \\left(1-\\frac{m_2}{S\\tau}\\right)\\mathrm{Prob}\\left(\\mathcal{A}_1^c(m_1^*)\\cap\\mathcal{A}_2^c(m_2)\\right) - \\frac{m_2}{S^2}\\cdot\\frac{32\\gamma\\Delta_\\Psi}{\\mu\\eta\\delta^2}\\\\\n    & \\geq & \\!\\!\\!\\! \\left(1-\\frac{m_2}{S\\tau}\\right)\\big(1-\\mathrm{Prob}\\left(\\mathcal{A}_1(m_1^*)\\right)-\\mathrm{Prob}\\left(\\mathcal{A}_2(m_2)\\right)\\big) - \\frac{m_2}{S^2}\\cdot\\frac{32\\gamma\\Delta_\\Psi}{\\mu\\eta\\delta^2}\\nonumber\\\\\n    & \\overset{(ii)}{\\geq} & \\!\\!\\!\\! 1 - \\frac{8\\eta\\tau b\\epsilon}{L^2(\\kappa_h^\\delta)^2\\delta^2} - \\frac{128\\eta\\Delta_\\Psi}{\\gamma\\mu\\delta^2\\cdot m_2} - \\frac{m_2}{S\\tau} - \\frac{m_2}{S^2}\\cdot\\frac{32\\gamma\\Delta_\\Psi}{\\mu\\eta\\delta^2}\\nonumber\\\\\n    & \\overset{(iii)}{\\geq} & \\!\\!\\!\\! 1 - \\frac{8\\eta\\tau b\\epsilon}{L^2(\\kappa_h^\\delta)^2\\delta^2} - \\frac{128\\eta\\Delta_\\Psi}{m_2\\gamma\\mu\\delta^2} - \\frac{9m_2}{8S\\tau}\\nonumber\n\\end{eqnarray}\nwhere (i) is due to \\eqref{thm:proxSARAH-finite-2} and the fact that \n\\begin{equation}\n    \\label{thm:proxSARAH-finite-55}\n    \\sum_{|\\mathcal{I}_1|<m_1^*}\\sum_{|\\mathcal{I}_2|< m_2}\\sum_{\\tau_s\\in[\\tau-1],s\\in\\mathcal{I}_1}\\mathrm{Prob}\\big(\\mathcal{I}_1,\\mathcal{I}_2,\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}\\big) = \\mathrm{Prob}\\big(\\mathcal{A}_1^c(m_1^*)\\cap\\mathcal{A}_2^c(m_2)\\big)\n\\end{equation}\n","sha256_utf8":"e6803fa916f181700b3e583aeb579ec2c9eb57d246317d910a791b3ab05741dd"}]},{"id":"S009","semantic_type":"proof_step","kind":"proof_step","title":"代入精度与轮数","source_units":[{"start":94092,"end":94835,"start_line":754,"end_line":757,"raw_tex":"(ii) is due to Lemma \\ref{lemma:event-halway} with $m_1^*=S/4$ and $S = \\big\\lceil\\frac{16\\Delta_\\Psi}{\\tau\\gamma\\eta\\mu\\epsilon}\\big\\rceil$: \n$$\\mathrm{Prob}\\big(\\mathcal{A}_1(S/4)\\big)\\leq \\frac{32\\gamma\\tau\\Delta_\\Psi}{\\mu\\delta^2S/4} = \\frac{16\\Delta_\\Psi}{S\\tau\\gamma\\eta\\mu}\\cdot\\frac{8\\gamma^2\\tau^2\\eta}{\\delta^2} \\leq \\frac{8\\tau b\\epsilon}{L^2(\\kappa_h^\\delta)^2\\delta^2},$$\nand (iii) is because we require $\\epsilon\\leq \\frac{\\delta^2}{16}\\cdot\\min\\left\\{\\frac{L^2(\\kappa_h^\\delta)^2}{b\\tau},\\frac{1}{9\\eta^2}\\right\\} = O(1/n)$ such that \n$$\\frac{m_2}{S^2}\\cdot\\frac{32\\gamma\\Delta_\\Psi}{\\mu\\eta\\delta^2} = \\frac{m_2}{S\\tau}\\cdot\\frac{16\\Delta_\\Psi}{S\\tau\\gamma\\eta\\mu}\\cdot\\frac{2\\gamma^2\\tau^2}{\\delta^2}\\leq \\frac{m_2}{8S\\tau}$$\n","sha256_utf8":"2048f428076d2ece65fb6ac4acddb386ae495e525fb9ffc3b75d75272c8e39b0"}]},{"id":"S010","semantic_type":"proof_step","kind":"proof_step","title":"优化阈值与成功率","source_units":[{"start":94835,"end":96074,"start_line":758,"end_line":762,"raw_tex":"Therefore, to maximize the above probability, we can choose $m_2^*=\\sqrt{\\frac{S\\tau\\eta\\Delta_\\Psi}{\\gamma\\mu}}\\cdot\\frac{32}{3\\delta}$ such that\n$$\\frac{128\\eta\\Delta_\\Psi}{m_2^*\\gamma\\mu\\delta^2} + \\frac{9m_2^*}{8S\\tau} = 2\\sqrt{\\frac{128\\eta\\Delta_\\Psi}{m_2^*\\gamma\\mu\\delta^2} \\cdot  \\frac{9m_2^*}{8S\\tau}} = \\sqrt{\\frac{\\eta\\Delta_\\Psi}{\\gamma\\mu S\\tau}}\\cdot\\frac{24}{\\delta}\\leq \\frac{6\\eta\\sqrt{\\epsilon}}{\\delta}\\leq \\frac{4\\sqrt{\\epsilon}}{\\delta}\\,, $$\nwhere the last inequality is due to the fact that $\\eta = \\frac{\\sqrt{2\\tau}}{\\sqrt{7\\tau}+\\sqrt{2b}}\\leq\\sqrt{2/7}.$ Combining all the above discussion, we can conclude that  \n$$\\mathrm{Prob}\\left(\\mathcal{A}\\right)\\geq1 - \\frac{8\\eta\\tau b\\epsilon}{L^2(\\kappa_h^\\delta)^2\\delta^2}-\\frac{4\\sqrt{\\epsilon}}{\\delta} = 1-O(n\\epsilon+\\sqrt{\\epsilon})$$\nwhen taking $m_1^* = S/4$ and $m_2^* = \\sqrt{\\frac{S\\tau\\eta\\Delta_\\Psi}{\\gamma\\mu}}\\cdot\\frac{32}{3\\delta}.$ Note that the requirement on $\\epsilon$ further implies that $\\mathrm{Prob}(\\cA_1(m_1^*))\\leq 1/2$ and $\\mathrm{Prob}(\\cA_2(m_2^*))\\leq 1/4$, and hence $\\mathrm{Prob}\\big(\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)\\big)\\geq1/4$. By ignoring the Bregman divergence terms of Lemma \\ref{lemma:proxSARAH-grad-map-mid}, we obtain\n","sha256_utf8":"92d94b6b38dc08fbd5150fcfa11be05a734f64632c997446d83019a20e109a8f"}]},{"id":"S011","semantic_type":"proof_step","kind":"proof_step","title":"条件映射平方和","source_units":[{"start":96074,"end":96403,"start_line":763,"end_line":765,"raw_tex":"\\begin{eqnarray*}\n    \\frac{\\Delta_\\Psi}{\\gamma\\eta} &\\geq& \\frac{\\mu}{8} \\mathbb{E}\\bigg[\\sum_{s=1}^S\\sum_{k=0}^{\\tau_s-1}\\big\\|\\cGshe(x_{s,k})\\big\\|^2\\bigg]  \\geq \\frac{\\mu}{32}\\mathbb{E}\\bigg[\\sum_{(s,k)\\in\\mathcal{I}_2^c}\\big\\|\\cGshe(x_{s,k})\\big\\|^2\\,\\Big|\\,\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)\\bigg] \\nonumber.\n\\end{eqnarray*}\n","sha256_utf8":"52876d73f273a3660eba3589dbdce263d0bd702ea3fd802d3c13d3c704895cf1"}]},{"id":"S012","semantic_type":"proof_step","kind":"proof_step","title":"有效迭代数下界","source_units":[{"start":96403,"end":97117,"start_line":766,"end_line":770,"raw_tex":"Conditioning on $\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)$, regardless of the random sets $\\mathcal{I}_1, \\mathcal{I}_2$ and $\\{\\tau_s\\}_{s\\in\\mathcal{I}_1}$, we have  \n\\begin{eqnarray}\n    |\\mathcal{I}_2^c| = (S-|\\mathcal{I}_1|)\\tau+\\sum_{s\\in\\mathcal{I}_1}\\tau_s - |\\mathcal{I}_2|\\geq  \\frac{3S\\tau}{4} - m_2^*\\geq S\\tau\\Big(\\frac{3}{4}-\\frac{8\\eta\\sqrt{\\epsilon}}{3\\delta}\\Big) \\geq \\frac{S\\tau}{2}.\\nonumber\n\\end{eqnarray}\nwhere the second last inequality is because $|\\mathcal{I}_2|\\leq m_2^*\\leq \\frac{8S\\tau\\eta\\sqrt{\\epsilon}}{3\\delta}$, and the last inequality is because  $\\frac{8\\eta\\sqrt{\\epsilon}}{3\\delta}\\leq 2/9< 1/4$ since we require $\\epsilon\\leq\\frac{\\delta^2}{16}\\cdot\\frac{1}{9\\eta^2}$. As a result, \n","sha256_utf8":"f161f3e53030086bf18196ae07568b1152aaa4adc8cd5d777f354b6d451186c3"}]},{"id":"S013","semantic_type":"proof_step","kind":"proof_step","title":"输出原始映射界","source_units":[{"start":97117,"end":98123,"start_line":771,"end_line":778,"raw_tex":"\\begin{eqnarray}\n\\label{thm:proxSARAH-finite-66}\n    \\frac{\\Delta_\\Psi}{\\gamma\\eta} &\\geq& \\frac{\\mu}{32}\\cdot\\frac{S\\tau}{2}\\cdot\\mathbb{E}\\bigg[\\frac{|\\mathcal{I}_2^c|}{S\\tau/2}\\cdot\\frac{\\sum_{(s,k)\\in\\mathcal{I}_2^c}\\big\\|\\cGshe(x_{s,k})\\big\\|^2}{|\\mathcal{I}_2^c|}\\,\\Big|\\,\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)\\bigg] \\\\\n    & \\geq & \\frac{\\mu S\\tau}{64}\\mathbb{E}\\bigg[\\frac{\\sum_{(s,k)\\in\\mathcal{I}_2^c}\\big\\|\\cGshe(x_{s,k})\\big\\|^2}{|\\mathcal{I}_2^c|}\\,\\Big|\\,\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)\\bigg] \\nonumber\\\\\n    & = & \\frac{\\mu S\\tau}{64}\\mathbb{E}\\Big[\\big\\|\\cGphe(x_{\\mathrm{out}})\\big\\|^2\\,\\big|\\,\\cA_1^c(m_1^*)\\cap\\cA_2^c(m_2^*)\\cap\\cA_3\\Big]\\,, \\nonumber\n\\end{eqnarray}\nwhere the last equality is due to the definition of $x_{\\mathrm{out}}$, $\\cA_3$, and the fact that $\\cGshe(\\cdot)$ coincides with $\\cGphe(\\cdot)$ in $\\mathcal{I}_2^c$. As a result, we have\n$$\\mathbb{E}\\Big[\\big\\|\\cGphe(x_{\\mathrm{out}})\\big\\|^2\\,\\big|\\,\\cA\\Big]\\leq \\frac{64\\Delta_\\Psi}{\\gamma\\eta\\mu\\tau S}\\leq 4\\epsilon.$$\n","sha256_utf8":"745cc1eea7e9e853c19d37f926f1ea5655af2c1edaba64eaf9444871a4c8bf33"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"计算采样复杂度","source_units":[{"start":98123,"end":98744,"start_line":779,"end_line":782,"raw_tex":"Given the choice of $S$, $\\gamma$, $\\eta$, and the fact that $\\tau = \\lceil n/b\\rceil$, the total sample complexity will be \n$$S(n+b\\tau) = (n+b\\tau)\\cdot\\bigg\\lceil\\frac{16\\Delta_\\Psi}{\\tau\\gamma\\eta\\mu\\epsilon}\\bigg\\rceil = O\\left(\\frac{L\\kappa_h^\\delta\\Delta_\\Psi\\sqrt{n}}{\\mu\\epsilon}\\cdot\\Big(1+\\sqrt{b/\\tau}\\Big)\\right),$$\nwhich indicates an $O(\\sqrt{n}\\epsilon^{-1})$ sample complexity for all $b\\leq O(\\tau)$ (or equivalently, $b\\leq O(\\lceil\\sqrt{n}\\rceil)$). If larger batch size $b = \\lceil n^\\alpha\\rceil$ with $\\alpha\\in(1/2,1]$ is taken, then we obtain an $O(n^\\alpha\\epsilon^{-1})$ complexity. \n\\end{proof}","sha256_utf8":"ea160d1038f2ec14f0a445f39e1d9b37353c74515a22aa12b451e459c928020a"}]},{"id":"S015","semantic_type":"problem_setting","kind":"problem_setting","title":"非凸有限和问题","source_units":[{"start":5423,"end":5589,"start_line":114,"end_line":117,"raw_tex":"\\begin{equation}\n    \\label{prob:main-det}\n    \\min_{x\\in\\RR^d} \\Psi(x) = f(x) + \\phi(x) \\qquad\\mbox{with}\\qquad f(x) = \\frac{1}{n}\\sum_{i=1}^n f_i(x),\n\\end{equation}","sha256_utf8":"0983a3f59fd47f2eed32528fe77b0e8c2d89cbb635eb8c8d57bbaaab21b08b92"},{"start":5591,"end":5733,"start_line":118,"end_line":118,"raw_tex":"where $\\phi(x)$ is a convex but possibly non-differentiable function, while $f$ and each $f_i$ are nonconvex and continuously differentiable.\n","sha256_utf8":"f4392c520bad102052cfb6d7b6528aea078982abb4c79465b3c771f40453f052"}]},{"id":"S016","semantic_type":"definition","kind":"definition","title":"布雷格曼近端算子","source_units":[{"start":8314,"end":8587,"start_line":133,"end_line":136,"raw_tex":"\\begin{equation}\n    \\label{defn:BPG-standard}\n    x_{k+1} = \\mathbf{T}_{\\phi,h}^\\lambda\\big(x_k, \\nabla f(x_k)\\big)\\qquad\\mbox{with}\\qquad \\mathbf{T}_{\\phi,h}^\\lambda (x,v) := \\argmin_{y\\in\\RR^d}\\,\\, y^\\top v + \\phi(y) + \\lambda^{-1}D_h(y,x),\\vspace{-0.1cm}\n\\end{equation}","sha256_utf8":"b0074ee8aeb87fb512e45858626fb7ee0528c359305264c4a59f844883a98d13"},{"start":8589,"end":8690,"start_line":137,"end_line":137,"raw_tex":"where $D_h(y,x):=h(y)-\\nabla h(x)^\\top(y-x)-h(x)$ stands for the Bregman divergence induced by $h$. \n","sha256_utf8":"b1c021a620cc9512af44d83c56735568fdcdac234c3236e5bc92bdc269a5cc54"}]},{"id":"S017","semantic_type":"definition","kind":"definition","title":"原始梯度映射","source_units":[{"start":9209,"end":9379,"start_line":140,"end_line":143,"raw_tex":"\\begin{equation}\n    \\label{defn:GradMap-BPG}\n    \\mathcal{G}_{\\phi,h}^\\lambda(x):= \\frac{x-\\mathbf{T}_{\\phi,h}^\\lambda\\big(x,\\nabla f(x)\\big)}{\\lambda}.  \n\\end{equation}","sha256_utf8":"95cc0544ecfee1175167689f35df178460b3d2e8812bc3789131bb83be12eab4"}]},{"id":"S018","semantic_type":"assumption","kind":"assumption","title":"光滑适应性","source_units":[{"start":25864,"end":26228,"start_line":197,"end_line":200,"raw_tex":"\\begin{assumption}[Smooth adaptability, \\cite{bolte2018first}]\n    \\label{assumption:L-smad}\n    Let $f$ and $h$ be twice continuously differentiable in $\\mathbb{R}^d$, and let $h$ be strictly convex. Then we assume $f$ is $L$-smooth adaptable to $h$ for some positive constant $L>0$. In other words, both  $Lh+f$ and $Lh-f$ are convex functions. \n\\end{assumption}","sha256_utf8":"514dc06f19b508b513187105fd0287dca989847e8341ac3e47c8c7e22e2a6150"},{"start":26230,"end":26513,"start_line":201,"end_line":205,"raw_tex":"\\noindent Given the twice continuous differentiability of $f$ and $h$, Assumption \\ref{assumption:L-smad} can be equivalently written  as \n\\begin{equation}\n    \\label{asmp:L-smad-alt}\n    -L\\nabla^2h(x)\\preceq \\nabla^2 f(x)\\preceq L\\nabla^2h(x),\\,\\, \\forall x\\in\\RR^d.\n\\end{equation}","sha256_utf8":"e73cba67c59ace0c75bf0b5bbcfd07b63d02424e17c3053ec145de86c3a5e299"}]},{"id":"S019","semantic_type":"notation","kind":"notation","title":"局部曲率与条件数","source_units":[{"start":34136,"end":34427,"start_line":254,"end_line":257,"raw_tex":"\\begin{equation}\n    \\label{defn:condition}\n    \\mu_h(\\mathcal{X}) \\,=\\, \\min_{x\\in\\cX}\\,\\lambda_{\\min}\\big(\\nabla^2 h(x)\\big),\\quad L_h(\\mathcal{X}) \\,=\\, \\max_{x\\in\\cX}\\,\\lambda_{\\max}\\big(\\nabla^2 h(x)\\big),\\quad\\mbox{and}\\quad \\kappa_h(\\cX) := \\frac{L_h(\\cX)}{\\mu_h(\\cX)}.\n\\end{equation}","sha256_utf8":"4787b10967943f916f52c7c605e19277e0223527ee86a8c7302414633da2f1a7"}]},{"id":"S020","semantic_type":"assumption","kind":"assumption","title":"核条件正则性","source_units":[{"start":52599,"end":53007,"start_line":389,"end_line":393,"raw_tex":"\\begin{assumption}[Kernel-conditioning]\n    \\label{assumption:kernel-conditioning}\n    For any $\\delta>0$, there exists a constant $\\kappa_h^\\delta>0$ such that  $$\\sup_{\\mathcal{X}\\subseteq\\mathbb{R}^d}\\Big\\{\\kappa_h(\\mathcal{X}):\\mathrm{diam}(\\mathcal{X})\\leq \\delta\\Big\\}\\leq \\kappa_h^\\delta,$$\nwhere $\\mathrm{diam}(\\cX):=\\sup\\{\\|x-y\\|: x,y\\in\\cX\\}$ denotes the diameter of the set $\\cX$.\n\\end{assumption}","sha256_utf8":"6e19f811d05f3c7444bf253f42158c7d82af2f4b64da1ac2a0006e86eb470017"}]},{"id":"S021","semantic_type":"assumption","kind":"assumption","title":"核的全局强凸性","source_units":[{"start":54290,"end":54431,"start_line":397,"end_line":400,"raw_tex":"\\begin{assumption}[Strong convexity]\n    \\label{assumption:SC}\n    The kernel $h$ is $\\mu$-strongly convex for some $\\mu>0$.\n\\end{assumption}","sha256_utf8":"ca7a37d84a755acfa9f7ffe259bb9ef012bab51ebb0eb00d21871f75be02044f"}]},{"id":"S022","semantic_type":"assumption","kind":"assumption","title":"分量光滑适应性","source_units":[{"start":69833,"end":70097,"start_line":517,"end_line":520,"raw_tex":"\\begin{assumption}\n    \\label{assumption:L-smad-finite-sum}\n    For each $i\\in[n]$, $f_i$ is $L_i$-smooth adaptable to $h$ for some positive constant $L_{i}>0$. Denote $L:=\\sqrt{\\frac{1}{n}\\sum_i^nL_i^2}$, then $f$ is $L$-smooth adaptable to $h$. \n\\end{assumption}","sha256_utf8":"e04dfacee4978f7df4cbf6347a5043e1ea6878ab0e22099e6f758bb0b34d2723"}]},{"id":"S023","semantic_type":"algorithm","kind":"algorithm","title":"带轮次边界的算法","source_units":[{"start":70402,"end":72054,"start_line":523,"end_line":549,"raw_tex":"\\begin{algorithm2e}\n\\caption{Stochastic variance reduced BPG method with epoch bounds} \n\\label{algorithm:proxSARAH}\n\\textbf{Input:} Initial point $x_{1,0}$, constant $\\delta$, epoch length $\\tau$, step size $\\eta$,  interpolation factor $\\gamma\\in(0,1]$.   \\\\\n\\For{$s=1,2,3,\\cdots,S$}{\n    Construct a convex set $\\mathcal{X}_s\\supseteq {B}(x_{s,0},\\delta/2)$ such that $\\kappa_h(\\mathcal{X}_s)\\leq \\kappa_h^\\delta$\\,.  \\\\\\textcolor{gray}{//**Proposition \\ref{proposition: Poly-kernel-condition} suggests $\\mathcal{X}_s= B\\left(x_{s,0},\\max\\left\\{\\frac{1}{2r},\\frac{\\|x_{s,0}\\|}{2r+1}\\right\\}\\right)$ for polynomial kernel**//}\n    \n    \\For{$k = 0,1,2,\\cdots,\\tau-1$}{  \n    If $k==0$, compute $v_{s,0}= \\nabla f(x_{s,0}) = \\frac{1}{n}\\sum_{i=1}^n\\nabla f_i(x_{s,0})$.\\\\\n    If $k\\geq 1$, uniformly sample a mini-batch $\\mathcal{B}_{s,k}\\subseteq[n]$  with replacement, compute  \n    \\begin{equation}\n        \\label{alg:sarah-small}\n        v_{s,k} = v_{s,k-1} + \\frac{1}{|\\mathcal{B}_{s,k}|}\\sum_{\\xi\\in\\mathcal{B}_{s,k}} \\Big(\\nabla f_\\xi(x_{s,k})-\\nabla f_\\xi(x_{s,k-1})\\Big)\\,.\n    \\end{equation}\\\\\n    \n    Denote $\\mathrm{id}_{\\cX_s}$ the indicator function of $\\cX_s$. Compute the BPG update with \n\n   \n    \\begin{equation}\n        \\label{alg:proxSARAH-update}\n        \\bar{x}_{s,k+1} =  \\Tshe(x_{s,k},v_{s,k})\\qquad\\mbox{and}\\qquad x_{s,k+1} = (1-\\gamma)x_{s,k}+\\gamma\\bar{x}_{s,k+1}\\vspace{-0.1cm}\n    \\end{equation}  \\\\ \n    \\textbf{if}\\,\\,\\,$\\mathrm{dist}(x_{s,k+1},\\partial \\cX_s) \\leq \\delta/4$\\,\\,\\,\\textbf{then}\\,\\,\\,\\textbf{break} the inner forloop.\n    } \n    Set $\\tau_s=k+1$ and $x_{s+1,0} = x_{s,\\tau_s}$. \n    } \n\\end{algorithm2e}","sha256_utf8":"0653268726e7ca2227f98c80ef830b20c9e28918440031d50522db647f30974a"}]},{"id":"S024","semantic_type":"algorithm","kind":"algorithm","title":"受限近端子问题","source_units":[{"start":72645,"end":72843,"start_line":552,"end_line":555,"raw_tex":"\\begin{equation} \n        \\label{alg:sarah-update}\n        \\bar{x}_{s,k+1} = \\argmin_{x\\in\\mathcal{X}_s}\\,\\, \\langle v_{s,k}, x\\rangle + \\phi(x) +  \\frac{1}{\\eta}D_h(x,x_{s,k}) . \n    \\end{equation}","sha256_utf8":"1daa6876ce4dd853f31ccbfdbd9e84ed527c7a7194d4c9c693e01316db3bd63a"}]},{"id":"S025","semantic_type":"definition","kind":"definition","title":"真实受限梯度映射","source_units":[{"start":73756,"end":74040,"start_line":558,"end_line":562,"raw_tex":"Define the true \\emph{restricted} primal gradient mapping in epoch $s$ as  \n\\begin{equation}\n    \\label{defn:restricted-grd-mapping}\n    \\cGshe(x_{s,k}):=\\frac{x_{s,k}-\\hat{x}_{s,k+1}}{\\eta}\\quad\\mbox{with}\\quad\\hat{x}_{s,k+1}:=\\Tshe\\big(x_{s,k},\\nabla f(x_{s,k})\\big),\n\\end{equation}","sha256_utf8":"ef6fd25a1045a355bb59d29f0a7cf5c46392fa87ba46b37d108ef184a8c3e09a"},{"start":74042,"end":74405,"start_line":563,"end_line":564,"raw_tex":"where by ``restricted'' we means this primal gradient mapping incorporates the indicator function of the set constraint $x\\in\\cX_s$.\nCompared to the $\\bar{x}_{s,k+1}$ in Algorithm \\ref{algorithm:proxSARAH}, $\\hat{x}_{s,k+1}$ is constructed with the true gradient $\\nabla f(x_{s,k})$.  For this restricted primal gradient mapping, the following lemma holds true. \n","sha256_utf8":"81d4ceef79c290158f906a6b503952da2fb95c8f1f4da4e03e3aacd35a874eab"}]},{"id":"S026","semantic_type":"notation","kind":"notation","title":"范数与集合记号","source_units":[{"start":24224,"end":25501,"start_line":190,"end_line":191,"raw_tex":"\\noindent\\textbf{Notations.} For $\\forall x\\in\\RR^d$, We default $\\|x\\|:=\\sqrt{x^\\top x}$  the $\\ell_2$-norm, and we denote $\\|x\\|_1:=\\sum_i|x_i|$ as the $\\ell_1$-norm. For a matrix $X\\in\\RR^{d\\times d}$, we use $\\lambda_{\\max}(X)$ and $\\lambda_{\\min}(X)$ to denote the maximal and minimal eigenvalues of $X$, respectively. And we use $\\|X\\|$ to denote the $\\ell_2$-operator norm of $X$. When $X$ is positive semidefinite, we write $X\\succeq0$, and we have $\\|X\\| = \\lambda_{\\max}(X)$ in this case. For any set $\\cX$, \nwe denote $\\mathrm{id}_{\\cX}(\\cdot)$ as the indicator function of the set.  Namely, $\\mathrm{id}_{\\cX}(x)=0$ if $x\\in\\cX$ and $\\mathrm{id}_{\\cX}(x) = +\\infty$ if $x\\notin\\cX$. We denote the interior of $\\cX$ as $\\mathrm{int}(\\cX)$ and we denote the boundary of $\\cX$ as $\\partial \\cX$. We denote $[n]:=\\{1,2,\\cdots,n\\}$. For any vector $x\\in\\RR^d$, the function $\\mathrm{sign}: \\RR^d\\to\\RR^d$ returns a sign vector of $x$. That is, for any $s=\\mathrm{sign}(x)$, its $i$-th element satisfies $s_i=1$ if $x_i\\geq0$ and $s_i=-1$ if $x_i<0$, for any $1\\leq i\\leq d$. Because many literature use the terminology $L$-smooth to denote $L$-Lipschitz continuity of the gradient, we will use ``continuously differentiable'' instead of ``smooth''  to avoid confusion. \n","sha256_utf8":"20ef8faed682a35b98a43eea5e7a7f3085f395b26f9a2e24868e2804a6c823aa"}]},{"id":"S027","semantic_type":"assumption","kind":"assumption","title":"算法分析共同假设","source_units":[{"start":70098,"end":70400,"start_line":521,"end_line":521,"raw_tex":"Throughout Section \\ref{section:VR}, we will use Assumption \\ref{assumption:L-smad-finite-sum}, Assumption \\ref{assumption:kernel-conditioning}, and Assumption \\ref{assumption:SC}, and we propose a stochastic variance reduced BPG method with epoch-wise bounds  in Algorithm \\ref{algorithm:proxSARAH}. \n","sha256_utf8":"d0ed302a6dcd11f5bbbe4cae8c6814678c2f60d4ef06375ff16ac6f81c904bf1"}]},{"id":"S028","semantic_type":"lemma","kind":"lemma","title":"映射与散度可和界","source_units":[{"start":78427,"end":78996,"start_line":614,"end_line":619,"raw_tex":"\\begin{lemma}\n    \\label{lemma:proxSARAH-grad-map-mid}\n    For any $b,\\tau\\in\\mathbb{Z}_+$, set $\\eta = \\frac{\\sqrt{2\\tau}}{\\sqrt{7\\tau}+\\sqrt{2b}}$, $\\gamma = \\frac{\\sqrt{b}}{L\\kappa_h^\\delta\\sqrt{\\tau}}$, and $|\\mathcal{B}_{s,k}|=b$, $\\forall s,k\\geq1$, then\n    $$\\mathbb{E}\\left[\\sum_{s=1}^S\\sum_{k=0}^{\\tau_s-1}\\frac{\\mu_h(\\mathcal{X}_s)}{8}\\big\\|\\cGshe(x_{s,k})\\big\\|^2 +D_h(\\bar{x}_{s,k+1},x_{s,k}) \\right]\\leq \\frac{\\Delta_\\Psi}{\\gamma\\eta},$$\n    where $\\Delta_\\Psi:=\\Psi(x_{1,0})-\\inf\\{\\Psi(x):x\\in\\RR^d\\}$ denotes the initial function value gap. \n\\end{lemma}","sha256_utf8":"6d58ca64d2655627a9ea2343cacad156b82fdc256b36206b194e73126745e65c"}]},{"id":"S029","semantic_type":"proof","kind":"proof","title":"可和界完整证明","source_units":[{"start":180796,"end":182760,"start_line":1480,"end_line":1499,"raw_tex":"\\begin{proof}\n    First, substituting the gradient estimation bound in Lemma \\ref{lemma:err-sarah} to Lemma \\ref{lemma:descent-proxSARAH-finite}, we have the following descent result throughout the $s$-th epoch\n\n\\begin{align}\n    \\label{lm:proxSARAH-grad-map-mid-1}\n    &\\mathbb{E}\\Big[\\Psi(x_{s,\\tau_s}) \\,\\big|\\, x_{s,0}\\Big] \\!\\leq \\Psi(x_{s,0}) \\!-\\! \\mathbb{E}\\bigg[\\frac{\\gamma\\eta\\mu_h(\\mathcal{X}_s)}{8}\\!\\sum_{k=0}^{\\tau_s-1}\\!\\big\\|\\cGshe(x_{s,k})\\big\\|^2 \\!+\\!\\bigg(\\!\\frac{\\gamma}{\\eta}- L\\kappa_h^\\delta\\gamma^2\\!\\bigg)D_h(\\bar{x}_{s,k+1},x_{s,k}) \\,\\Big|\\, x_{s,0}\\bigg]\\nonumber\\\\\n    &\\qquad\\qquad\\qquad\\qquad\\,\\, +\\frac{5\\gamma\\eta}{4\\mu_h(\\mathcal{X}_s)}\\cdot\\frac{2\\gamma^2L^2\\kappa_h^\\delta L_h(\\mathcal{X}_s)}{b}\\mathbb{E}\\bigg[\\sum_{k=0}^{\\tau_s-1}\\sum_{j=0}^{k-1} D_h(\\bar{x}_{s,j+1},x_{s,j}) \\,\\Big|\\, x_{s,0}\\bigg]\\\\\n    &\\quad\\,\\leq \\Psi(x_{s,0}) \\!-\\! \\mathbb{E}\\bigg[\\frac{\\gamma\\eta\\mu_h(\\mathcal{X}_s)}{8}\\!\\sum_{k=0}^{\\tau_s-1}\\big\\|\\mathcal{G}_{s,k}^\\eta(x_{s,k})\\big\\|^2 \\!+\\bigg(\\frac{\\gamma}{\\eta}- L\\kappa_h^\\delta\\gamma^2-\\frac{5\\tau\\gamma^3\\eta L^2(\\kappa_h^\\delta)^2}{2b}\\bigg)D_h(\\bar{x}_{s,k+1},x_{s,k}) \\,\\Big|\\, x_{s,0}\\bigg]\\nonumber\n\\end{align}\n\n\\noindent Suppose we choose $\\eta  = \\frac{\\sqrt{2\\tau}}{\\sqrt{7\\tau}+\\sqrt{2b}}$ and we choose $\\gamma = \\frac{\\sqrt{b}}{L\\kappa_h^\\delta\\sqrt{\\tau}}$. Then we have \n\\begin{eqnarray}\n    \\label{lm:proxSARAH-grad-map-mid-2}\n    &&\\frac{\\gamma}{\\eta} - L\\kappa_h^\\delta\\gamma^2 - \\frac{5\\tau\\gamma^3\\eta L^2(\\kappa_h^\\delta)^2}{2b}\\nonumber\\\\\n    & = & \\gamma\\eta\\bigg(\\frac{1}{\\eta^2} - \\frac{L\\kappa_h^\\delta\\gamma}{\\eta} - \\frac{5\\tau\\gamma^2 L^2(\\kappa_h^\\delta)^2}{2b}\\bigg)\\\\\n    &=& \\gamma\\eta\\bigg(1+\\frac{\\sqrt{14} -\\sqrt{3.5}}{\\sqrt{\\tau/b}}\\bigg)\\nonumber \\\\\n    &\\geq& \\gamma\\eta\\,.\\nonumber\n\\end{eqnarray} \nSubstitute the bound to the previous inequality, summing up over all epochs, and taking the expectation over all randomness proves the lemma. \n\\end{proof}","sha256_utf8":"aea62fd8fd421846f7c091ad48fb237928438baca9eb03a258b0e26aa08cbe28"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"轮内累计误差下降","source_units":[{"start":180796,"end":181986,"start_line":1480,"end_line":1488,"raw_tex":"\\begin{proof}\n    First, substituting the gradient estimation bound in Lemma \\ref{lemma:err-sarah} to Lemma \\ref{lemma:descent-proxSARAH-finite}, we have the following descent result throughout the $s$-th epoch\n\n\\begin{align}\n    \\label{lm:proxSARAH-grad-map-mid-1}\n    &\\mathbb{E}\\Big[\\Psi(x_{s,\\tau_s}) \\,\\big|\\, x_{s,0}\\Big] \\!\\leq \\Psi(x_{s,0}) \\!-\\! \\mathbb{E}\\bigg[\\frac{\\gamma\\eta\\mu_h(\\mathcal{X}_s)}{8}\\!\\sum_{k=0}^{\\tau_s-1}\\!\\big\\|\\cGshe(x_{s,k})\\big\\|^2 \\!+\\!\\bigg(\\!\\frac{\\gamma}{\\eta}- L\\kappa_h^\\delta\\gamma^2\\!\\bigg)D_h(\\bar{x}_{s,k+1},x_{s,k}) \\,\\Big|\\, x_{s,0}\\bigg]\\nonumber\\\\\n    &\\qquad\\qquad\\qquad\\qquad\\,\\, +\\frac{5\\gamma\\eta}{4\\mu_h(\\mathcal{X}_s)}\\cdot\\frac{2\\gamma^2L^2\\kappa_h^\\delta L_h(\\mathcal{X}_s)}{b}\\mathbb{E}\\bigg[\\sum_{k=0}^{\\tau_s-1}\\sum_{j=0}^{k-1} D_h(\\bar{x}_{s,j+1},x_{s,j}) \\,\\Big|\\, x_{s,0}\\bigg]\\\\\n    &\\quad\\,\\leq \\Psi(x_{s,0}) \\!-\\! \\mathbb{E}\\bigg[\\frac{\\gamma\\eta\\mu_h(\\mathcal{X}_s)}{8}\\!\\sum_{k=0}^{\\tau_s-1}\\big\\|\\mathcal{G}_{s,k}^\\eta(x_{s,k})\\big\\|^2 \\!+\\bigg(\\frac{\\gamma}{\\eta}- L\\kappa_h^\\delta\\gamma^2-\\frac{5\\tau\\gamma^3\\eta L^2(\\kappa_h^\\delta)^2}{2b}\\bigg)D_h(\\bar{x}_{s,k+1},x_{s,k}) \\,\\Big|\\, x_{s,0}\\bigg]\\nonumber\n\\end{align}\n","sha256_utf8":"483e528fef5e22f320818f7aca17588c8d6b6ec17fb58339a1228fca41b0e525"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"参数确保正下降","source_units":[{"start":181987,"end":182606,"start_line":1490,"end_line":1497,"raw_tex":"\\noindent Suppose we choose $\\eta  = \\frac{\\sqrt{2\\tau}}{\\sqrt{7\\tau}+\\sqrt{2b}}$ and we choose $\\gamma = \\frac{\\sqrt{b}}{L\\kappa_h^\\delta\\sqrt{\\tau}}$. Then we have \n\\begin{eqnarray}\n    \\label{lm:proxSARAH-grad-map-mid-2}\n    &&\\frac{\\gamma}{\\eta} - L\\kappa_h^\\delta\\gamma^2 - \\frac{5\\tau\\gamma^3\\eta L^2(\\kappa_h^\\delta)^2}{2b}\\nonumber\\\\\n    & = & \\gamma\\eta\\bigg(\\frac{1}{\\eta^2} - \\frac{L\\kappa_h^\\delta\\gamma}{\\eta} - \\frac{5\\tau\\gamma^2 L^2(\\kappa_h^\\delta)^2}{2b}\\bigg)\\\\\n    &=& \\gamma\\eta\\bigg(1+\\frac{\\sqrt{14} -\\sqrt{3.5}}{\\sqrt{\\tau/b}}\\bigg)\\nonumber \\\\\n    &\\geq& \\gamma\\eta\\,.\\nonumber\n\\end{eqnarray} \n","sha256_utf8":"e13d51f5aeb478e470b6011f834c0aece7973c642b665691f45146cc90e4ba9a"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"跨轮求和取期望","source_units":[{"start":182606,"end":182760,"start_line":1498,"end_line":1499,"raw_tex":"Substitute the bound to the previous inequality, summing up over all epochs, and taking the expectation over all randomness proves the lemma. \n\\end{proof}","sha256_utf8":"236ecb6a57dc8d1324460a829c8706a8acd7f62a852519df8bcd6d6ee7f3847b"}]},{"id":"S033","semantic_type":"lemma","kind":"lemma","title":"随机近端单步下降","source_units":[{"start":75608,"end":76109,"start_line":578,"end_line":584,"raw_tex":"\\begin{lemma}\n\\label{lemma:descent-proxSARAH-finite}\n    Let $\\{x_{s,k}\\}_{k=0}^{\\tau_s}$ be the $s$-th epoch of Algorithm \\ref{algorithm:proxSARAH}, then  we have \n    \\begin{eqnarray}\n    \\Psi(x_{s,k+1})\\leq \\Psi(x_{s,k})-\\frac{\\gamma\\eta\\mu_h(\\mathcal{X}_s)}{8}\\big\\|\\cGshe(x_{s,k})\\big\\|^2\\!-\\!\\bigg(\\frac{\\gamma}{\\eta}- L\\kappa_h^\\delta\\gamma^2\\!\\bigg)\\!D_h(\\bar{x}_{s,k+1},x_{s,k}) \\!+\\!\\frac{5\\gamma\\eta\\|\\mathcal{E}_{s,k}\\|^2}{4\\mu_h(\\mathcal{X}_s)}.\\,\\nonumber\n    \\end{eqnarray} \n\\end{lemma}","sha256_utf8":"391aaf1793853678afc2b91ddc9bd5168d24d4dfe437a2159dcdfedbb9065dac"}]},{"id":"S034","semantic_type":"proof","kind":"proof","title":"单步下降完整证明","source_units":[{"start":178195,"end":180729,"start_line":1456,"end_line":1475,"raw_tex":"\\begin{proof} \n    First of all, by the update rule of $x_{s,k+1}$, we have   \n    \\begin{eqnarray}\n    \\label{lm:descent-proxSARAH-finite-1}\n        \\Psi(x_{s,k+1}) \n        \\!\\!\\!& = & \\!\\!\\!f\\big(x_{s,k} + \\gamma (\\bar{x}_{s,k+1}-x_{s,k})\\big) + \\phi\\big((1-\\gamma)x_{s,k} + \\gamma\\bar{x}_{s,k+1}\\big) \\\\\n        &\\overset{(i)}{\\leq}& \\!\\!\\!f(x_{s,k}) + \\gamma\\langle\\nabla f(x_{s,k}),\\bar{x}_{s,k+1}\\!-x_{s,k}\\rangle \\!+\\! LD_h\\left(x_{s,k+1},x_{s,k}\\right) \\!+\\! (1-\\gamma)\\phi(x_{s,k})+\\gamma\\phi(\\bar{x}_{s,k+1})\\nonumber\\\\\n        & \\overset{(ii)}{\\leq} & \\!\\!\\!\\Psi(x_{s,k}) + L\\kappa_h^\\delta\\gamma^2D_h(\\bar{x}_{s,k+1},x_{s,k}) + \\gamma\\langle\\mathcal{E}_{s,k}+v_{s,k},\\bar{x}_{s,k+1}-x_{s,k}\\rangle  + \\gamma\\left(\\phi(\\bar{x}_{s,k+1})-\\phi(x_{s,k})\\right)\\nonumber\\\\\n        &\\overset{(iii)}{\\leq}&\\!\\!\\! \\Psi(x_{s,k}) + L\\kappa_h^\\delta\\gamma^2D_h(\\bar{x}_{s,k+1},x_{s,k}) + \\frac{\\gamma\\eta\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h(\\mathcal{X}_s)} + \\frac{\\gamma\\mu_h(\\mathcal{X}_s)}{4\\eta} \\|\\bar{x}_{s,k+1}-x_{s,k}\\|^2\\nonumber\\\\\n        & & \\!\\!\\!-\\frac{\\gamma}{\\eta}D_h(\\bar{x}_{s,k+1},x_{s,k}) - \\frac{\\gamma}{\\eta}D_h(x_{s,k},\\bar{x}_{s,k+1})\\nonumber\\\\\n        &\\leq & \\!\\!\\!\\Psi(x_{s,k}) - \\left(\\frac{\\gamma}{\\eta}- L\\kappa_h^\\delta\\gamma^2\\right)D_h(\\bar{x}_{s,k+1},x_{s,k}) -\\frac{\\gamma}{2\\eta}D_h(x_{s,k},\\bar{x}_{s,k+1})+ \\frac{\\gamma\\eta\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h(\\mathcal{X}_s)} \\,,\\nonumber\n    \\end{eqnarray}\n    where (i) is due to Assumption \\ref{assumption:L-smad-finite-sum}, Lemma \\ref{lemma:descent-lemma}, and the convexity of $\\phi$, (ii) is due to the definition of $\\mathcal{E}_{s,k}$ in Lemma \\ref{lemma:restricted-grd-mapping} and the following scaling property\n    \\begin{eqnarray}\n        D_h\\left(x_{s,k+1},x_{s,k}\\right)  \\leq \\frac{L_h(\\mathcal{X}_s)}{2}\\|x_{s,k+1}-x_{s,k}\\|^2 = \\frac{\\gamma^2L_h(\\mathcal{X}_s)}{2}\\|\\bar{x}_{s,k+1}-x_{s,k}\\|^2 \\leq \\gamma^2\\kappa_h^\\delta D_h(\\bar{x}_{s,k+1},x_{s,k})\\,,\\nonumber\n    \\end{eqnarray}\n    and (iii) is due to the optimality of  $\\bar{x}_{s,k+1}$ to the corresponding subproblem and Tseng's three point property (Lemma \\ref{lemma:Tseng-3point}). Finally, by Lemma \\ref{lemma:restricted-grd-mapping}, we also have\n    $$\\big\\|\\cGshe(x_{s,k})\\big\\|^2 \\leq \\frac{2\\|x_{s,k}-\\bar{x}_{s,k+1}\\|^2}{\\eta^2} + \\frac{2\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h^2(\\mathcal{X}_s)}\\,.$$\n    Multiplying both sides of the above inequality by $\\frac{\\gamma\\eta\\mu_h(\\mathcal{X}_s)}{8}$ and add it to \\eqref{lm:descent-proxSARAH-finite-1} proves the lemma. \n\\end{proof}","sha256_utf8":"98a4d486ed34efde9b645ee1043966351f32a6555eb1e8d290f7b315e92ca678"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"展开目标并控制误差","source_units":[{"start":178195,"end":179630,"start_line":1456,"end_line":1467,"raw_tex":"\\begin{proof} \n    First of all, by the update rule of $x_{s,k+1}$, we have   \n    \\begin{eqnarray}\n    \\label{lm:descent-proxSARAH-finite-1}\n        \\Psi(x_{s,k+1}) \n        \\!\\!\\!& = & \\!\\!\\!f\\big(x_{s,k} + \\gamma (\\bar{x}_{s,k+1}-x_{s,k})\\big) + \\phi\\big((1-\\gamma)x_{s,k} + \\gamma\\bar{x}_{s,k+1}\\big) \\\\\n        &\\overset{(i)}{\\leq}& \\!\\!\\!f(x_{s,k}) + \\gamma\\langle\\nabla f(x_{s,k}),\\bar{x}_{s,k+1}\\!-x_{s,k}\\rangle \\!+\\! LD_h\\left(x_{s,k+1},x_{s,k}\\right) \\!+\\! (1-\\gamma)\\phi(x_{s,k})+\\gamma\\phi(\\bar{x}_{s,k+1})\\nonumber\\\\\n        & \\overset{(ii)}{\\leq} & \\!\\!\\!\\Psi(x_{s,k}) + L\\kappa_h^\\delta\\gamma^2D_h(\\bar{x}_{s,k+1},x_{s,k}) + \\gamma\\langle\\mathcal{E}_{s,k}+v_{s,k},\\bar{x}_{s,k+1}-x_{s,k}\\rangle  + \\gamma\\left(\\phi(\\bar{x}_{s,k+1})-\\phi(x_{s,k})\\right)\\nonumber\\\\\n        &\\overset{(iii)}{\\leq}&\\!\\!\\! \\Psi(x_{s,k}) + L\\kappa_h^\\delta\\gamma^2D_h(\\bar{x}_{s,k+1},x_{s,k}) + \\frac{\\gamma\\eta\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h(\\mathcal{X}_s)} + \\frac{\\gamma\\mu_h(\\mathcal{X}_s)}{4\\eta} \\|\\bar{x}_{s,k+1}-x_{s,k}\\|^2\\nonumber\\\\\n        & & \\!\\!\\!-\\frac{\\gamma}{\\eta}D_h(\\bar{x}_{s,k+1},x_{s,k}) - \\frac{\\gamma}{\\eta}D_h(x_{s,k},\\bar{x}_{s,k+1})\\nonumber\\\\\n        &\\leq & \\!\\!\\!\\Psi(x_{s,k}) - \\left(\\frac{\\gamma}{\\eta}- L\\kappa_h^\\delta\\gamma^2\\right)D_h(\\bar{x}_{s,k+1},x_{s,k}) -\\frac{\\gamma}{2\\eta}D_h(x_{s,k},\\bar{x}_{s,k+1})+ \\frac{\\gamma\\eta\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h(\\mathcal{X}_s)} \\,,\\nonumber\n    \\end{eqnarray}\n","sha256_utf8":"4900813bc41db12fb9055981e82b32bac96f1e5662b74ab331b46eb6edcd7ee7"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"解释散度插值缩放","source_units":[{"start":179630,"end":180173,"start_line":1468,"end_line":1471,"raw_tex":"    where (i) is due to Assumption \\ref{assumption:L-smad-finite-sum}, Lemma \\ref{lemma:descent-lemma}, and the convexity of $\\phi$, (ii) is due to the definition of $\\mathcal{E}_{s,k}$ in Lemma \\ref{lemma:restricted-grd-mapping} and the following scaling property\n    \\begin{eqnarray}\n        D_h\\left(x_{s,k+1},x_{s,k}\\right)  \\leq \\frac{L_h(\\mathcal{X}_s)}{2}\\|x_{s,k+1}-x_{s,k}\\|^2 = \\frac{\\gamma^2L_h(\\mathcal{X}_s)}{2}\\|\\bar{x}_{s,k+1}-x_{s,k}\\|^2 \\leq \\gamma^2\\kappa_h^\\delta D_h(\\bar{x}_{s,k+1},x_{s,k})\\,,\\nonumber\n    \\end{eqnarray}\n","sha256_utf8":"8b6a6d9cee084b92634883e9f5fc54c68d21ee0773abee3454d3e47236acdc85"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"加入受限映射下降","source_units":[{"start":180173,"end":180729,"start_line":1472,"end_line":1475,"raw_tex":"    and (iii) is due to the optimality of  $\\bar{x}_{s,k+1}$ to the corresponding subproblem and Tseng's three point property (Lemma \\ref{lemma:Tseng-3point}). Finally, by Lemma \\ref{lemma:restricted-grd-mapping}, we also have\n    $$\\big\\|\\cGshe(x_{s,k})\\big\\|^2 \\leq \\frac{2\\|x_{s,k}-\\bar{x}_{s,k+1}\\|^2}{\\eta^2} + \\frac{2\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h^2(\\mathcal{X}_s)}\\,.$$\n    Multiplying both sides of the above inequality by $\\frac{\\gamma\\eta\\mu_h(\\mathcal{X}_s)}{8}$ and add it to \\eqref{lm:descent-proxSARAH-finite-1} proves the lemma. \n\\end{proof}","sha256_utf8":"2f0b55d4d38a06f4fa50bc82bd9fdbbb2b4a1ca488f35eaa7716a280e4dd20e3"}]},{"id":"S038","semantic_type":"lemma","kind":"lemma","title":"递归估计误差界","source_units":[{"start":76378,"end":77068,"start_line":588,"end_line":596,"raw_tex":"\\begin{lemma}\n    \\label{lemma:err-sarah} \n    Let $\\{x_{s,k}\\}_{k=0}^{\\tau_s}$ be the $s$-th epoch of Algorithm \\ref{algorithm:proxSARAH}. Given any batch size $b>0$, if we set $|\\mathcal{B}_{s,k}|=b$ for $k=1,\\cdots,\\tau_s-1$. Then conditioning on the initial point $x_{s,0}$ of the epoch, we have\n    \\begin{eqnarray}\n    \\label{lm:err-sarah-0}\n    \\mathbb{E}\\left[\\frac{\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h(\\mathcal{X}_s)}\\,\\Big|\\, x_{s,0}\\right] \\leq  \\frac{2\\gamma^2L^2(\\kappa_h^\\delta)^2}{b}\\mathbb{E}\\bigg[\\sum_{j=0}^{k-1} D_h(\\bar{x}_{s,j+1},x_{s,j})\\,\\big|\\,x_{s,0}\\bigg]\\,, \n\\end{eqnarray}\nas long as $h$ satisfies the kernel conditioning regularity assumption in $\\cX_s$. \n\\end{lemma}","sha256_utf8":"0aeda5fbcfcd5add3fbf82e49c56adf6a065dd2470ddab0be0d82ddeafa46bbc"}]},{"id":"S039","semantic_type":"proof","kind":"proof","title":"误差界完整证明","source_units":[{"start":77069,"end":78211,"start_line":597,"end_line":610,"raw_tex":"\\begin{proof}\n    By \\cite[Lemma 2]{pham2020proxsarah}, it is not hard to obtain that \n    \\begin{equation}\n        \\label{lm:err-sarah-0.5}\n        \\mathbb{E}\\Big[\\|\\mathcal{E}_{s,k}\\|^2\\,\\big|\\,x_{s,0}\\Big]  \\leq  \\sum_{j=0}^{k-1} \\mathbb{E}\\bigg[\\frac{1}{bn}\\sum_{i=1}^n\\|\\nabla f_i(x_{s,j+1})-\\nabla f_i(x_{s,j})\\|^2\\,\\big|\\,x_{s,0}\\bigg].\n    \\end{equation}\n    Because each $f_i$ is $L_i$-smooth adaptable to the KC-regular kernel $h$, applying the Lipschitz-like bound \\eqref{prop:Lipschitz} to each $f_i$ on   $\\bar{x}_{s,j+1}, x_{s,j}$ and their interpolation ${x}_{s,j+1} = \\gamma\\bar{x}_{s,j+1} + (1-\\gamma)x_{s,j}$ immediately yields \n    \\begin{eqnarray}\n    \\label{lm:err-sarah-1}\n    \\mathbb{E}\\left[\\frac{\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h(\\mathcal{X}_s)}\\,\\Big|\\, x_{s,0}\\right] \n    &\\leq&  \\sum_{j=0}^{k-1}\\mathbb{E}\\bigg[ \\frac{\\sum_{i=1}^nL_i^2}{n}\\cdot\\frac{2\\gamma^2  \\kappa_h^2(\\cX_s)}{b}D_h(\\bar{x}_{s,j+1},x_{s,j})\\,\\big|\\,x_{s,0}\\bigg]\\nonumber.\n\\end{eqnarray}\nUsing the fact that $L^2 = \\frac{1}{n}\\sum_{i=1}^nL_i^2$ and the fact that $\\kappa_h(\\cX_s)\\leq \\kappa_h^\\delta$ leads to \\eqref{lm:err-sarah-0}. \n\\end{proof}","sha256_utf8":"6872660aa0afff840ee399b2d1df71895b101dbf8c5bd18c209576aee670664b"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"递归方差累计","source_units":[{"start":77069,"end":77432,"start_line":597,"end_line":602,"raw_tex":"\\begin{proof}\n    By \\cite[Lemma 2]{pham2020proxsarah}, it is not hard to obtain that \n    \\begin{equation}\n        \\label{lm:err-sarah-0.5}\n        \\mathbb{E}\\Big[\\|\\mathcal{E}_{s,k}\\|^2\\,\\big|\\,x_{s,0}\\Big]  \\leq  \\sum_{j=0}^{k-1} \\mathbb{E}\\bigg[\\frac{1}{bn}\\sum_{i=1}^n\\|\\nabla f_i(x_{s,j+1})-\\nabla f_i(x_{s,j})\\|^2\\,\\big|\\,x_{s,0}\\bigg].\n    \\end{equation}\n","sha256_utf8":"cc331855bafe22389960b04d37473badaabb689557eb918d0c0307b00f79056f"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"梯度差转为散度","source_units":[{"start":77432,"end":78053,"start_line":603,"end_line":608,"raw_tex":"    Because each $f_i$ is $L_i$-smooth adaptable to the KC-regular kernel $h$, applying the Lipschitz-like bound \\eqref{prop:Lipschitz} to each $f_i$ on   $\\bar{x}_{s,j+1}, x_{s,j}$ and their interpolation ${x}_{s,j+1} = \\gamma\\bar{x}_{s,j+1} + (1-\\gamma)x_{s,j}$ immediately yields \n    \\begin{eqnarray}\n    \\label{lm:err-sarah-1}\n    \\mathbb{E}\\left[\\frac{\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h(\\mathcal{X}_s)}\\,\\Big|\\, x_{s,0}\\right] \n    &\\leq&  \\sum_{j=0}^{k-1}\\mathbb{E}\\bigg[ \\frac{\\sum_{i=1}^nL_i^2}{n}\\cdot\\frac{2\\gamma^2  \\kappa_h^2(\\cX_s)}{b}D_h(\\bar{x}_{s,j+1},x_{s,j})\\,\\big|\\,x_{s,0}\\bigg]\\nonumber.\n\\end{eqnarray}\n","sha256_utf8":"f9eb7860b7bd2432e23bd6ed5629ed3bf724b0899044835e9e7043a430497079"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"代入分量均方常数","source_units":[{"start":78053,"end":78211,"start_line":609,"end_line":610,"raw_tex":"Using the fact that $L^2 = \\frac{1}{n}\\sum_{i=1}^nL_i^2$ and the fact that $\\kappa_h(\\cX_s)\\leq \\kappa_h^\\delta$ leads to \\eqref{lm:err-sarah-0}. \n\\end{proof}","sha256_utf8":"b7b23bd26efbf4decc0258275015fe7f7bb4239e5c24745cc2a7b99feb39e5c2"}]},{"id":"S043","semantic_type":"lemma","kind":"lemma","title":"受限映射误差比较","source_units":[{"start":74405,"end":74967,"start_line":565,"end_line":573,"raw_tex":"\\begin{lemma}\n    \\label{lemma:restricted-grd-mapping} \n    Let $\\cGshe(x_{s,k})$, $\\hat{x}_{s,k+1}$ and $\\bar{x}_{s,k+1}$ be defined by \\eqref{defn:restricted-grd-mapping} and \\eqref{alg:sarah-update}, respectively. Then \n    \\begin{eqnarray}\n    \\label{lm:grad-mapping} \n        \\big\\|\\cGshe(x_{s,k})\\big\\|^2 \\leq \\frac{2\\|x_{s,k}-\\bar{x}_{s,k+1}\\|^2}{\\eta^2} + \\frac{2\\|\\mathcal{E}_{s,k}\\|^2}{\\mu_h^2(\\mathcal{X}_s)},\n    \\end{eqnarray} \n    where $\\mathcal{E}_{s,k} = \\nabla f(x_{s,k})-v_{s,k}$ denotes the gradient estimation error at $x_{s,k}$.\n\\end{lemma}","sha256_utf8":"676da4e437df7f7700c2ee2c0b95858798eb7e07e7f240bb54a8221e9bd6ca7f"}]},{"id":"S044","semantic_type":"proof","kind":"proof","title":"映射比较完整证明","source_units":[{"start":175721,"end":178126,"start_line":1430,"end_line":1451,"raw_tex":"\\begin{proof}\n    Denote $\\widetilde{\\mathcal{G}}=\\frac{x_{s,k}-\\bar{x}_{s,k+1}}{\\eta}$, then $\\|\\cGshe(x_{s,k})-\\widetilde{\\mathcal{G}}\\| = \\frac{1}{\\eta}\\|\\bar{x}_{s,k+1}-\\hat{x}_{s,k+1}\\|$ holds by definition. \n    By the optimality of $\\hat{x}_{s,k+1}$ and $\\bar{x}_{s,k+1}$ for the corresponding subproblems, Tseng's three point property (Lemma \\ref{lemma:Tseng-3point}) indicates that\n    \\begin{eqnarray*}\n        &&\\langle v_{s,k}, \\bar{x}_{s,k+1}\\rangle + \\phi(\\bar{x}_{s,k+1}) + \\frac{D_h(\\bar{x}_{s,k+1},x_{s,k})}{\\eta} + \\frac{D_h(\\hat{x}_{s,k+1},\\bar{x}_{s,k+1})}{\\eta}\\qquad\\,\\,\\,\\\\\n        &\\leq& \\langle v_{s,k}, \\hat{x}_{s,k+1}\\rangle + \\phi(\\hat{x}_{s,k+1})+ \\frac{D_h(\\hat{x}_{s,k+1},x_{s,k})}{\\eta} \n    \\end{eqnarray*} \n    and \n    \\begin{eqnarray*}\n        && \\langle \\nabla f(x_{s,k}), \\hat{x}_{s,k+1}\\rangle + \\phi(\\hat{x}_{s,k+1})+\\frac{D_h(\\hat{x}_{s,k+1},x_{s,k})}{\\eta} + \\frac{D_h(\\bar{x}_{s,k+1},\\hat{x}_{s,k+1})}{\\eta}\\\\\n        &\\leq& \\langle \\nabla f(x_{s,k}), \\bar{x}_{s,k+1}\\rangle + \\phi(\\bar{x}_{s,k+1})+\\frac{D_h(\\bar{x}_{s,k+1},x_{s,k})}{\\eta}\n    \\end{eqnarray*} \n    Summing up the two inequalities and applying Lemma \\ref{lemma:grd-vs-grdmap} gives \n    \\begin{align} &\\frac{\\mu_h([\\bar{x}_{s,k+1},\\hat{x}_{s,k+1}])}{\\eta}\\cdot\\|\\bar{x}_{s,k+1}-\\hat{x}_{s,k+1}\\|^2 \\leq \\frac{D_h(\\bar{x}_{s,k+1},\\hat{x}_{s,k+1})}{\\eta} +  \\frac{D_h(\\hat{x}_{s,k+1},\\bar{x}_{s,k+1})}{\\eta}\\nonumber\\\\\n    &\\qquad\\qquad\\qquad\\qquad\\qquad\\,\\,\\,\\, \\leq  \\langle v_{s,k}-\\nabla f(x_{s,k}), \\hat{x}_{s,k+1}-\\bar{x}_{s,k+1}\\rangle \\leq  \\|v_{s,k}-\\nabla f(x_{s,k})\\|\\cdot\\|\\hat{x}_{s,k+1}-\\bar{x}_{s,k+1}\\|\\nonumber.\n    \\end{align} \n    Hence $\\|\\hat{x}_{s,k+1}-\\bar{x}_{s,k+1}\\|\\leq \\frac{\\eta\\|v_{s,k}-\\nabla f(x_{s,k})\\|}{\\mu_h([\\bar{x}_{s,k+1},\\hat{x}_{s,k+1}])}$, and $\\big\\|\\cGshe(x_{s,k})-\\widetilde{\\mathcal{G}}\\big\\|\\leq\\frac{\\|v_{s,k}-\\nabla f(x_{s,k})\\|}{\\mu_h([\\bar{x}_{s,k+1},\\hat{x}_{s,k+1}])}$. As a result, \n    \\begin{eqnarray*}\n        \\big\\|\\cGshe(x_{s,k})\\big\\|^2 \\leq \\left(\\big\\|\\widetilde{\\mathcal{G}}\\big\\|+\\big\\|\\cGshe(x_{s,k})-\\widetilde{\\mathcal{G}}\\big\\|\\right)^2\\leq  \\frac{2\\|x_{s,k}-\\bar{x}_{s,k+1}\\|^2}{\\eta^2} + \\frac{2\\|v_{s,k}-\\nabla f(x_{s,k})\\|^2}{\\mu_h^2(\\mathcal{X}_s)},\n    \\end{eqnarray*} \n    where the last inequality is because $(a+b)^2\\leq 2a^2+2b^2$ and $\\mu_h^2\\big([\\bar{x}_{s,k+1},\\hat{x}_{s,k+1}]\\big)\\geq\\mu_h^2(\\cX_s)$. \n\\end{proof}","sha256_utf8":"a27d2e1e52c5307995bb9607dc0e5dd01faee21e490052c0a0f1688f6fd0346b"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"随机子问题三点界","source_units":[{"start":175721,"end":176462,"start_line":1430,"end_line":1436,"raw_tex":"\\begin{proof}\n    Denote $\\widetilde{\\mathcal{G}}=\\frac{x_{s,k}-\\bar{x}_{s,k+1}}{\\eta}$, then $\\|\\cGshe(x_{s,k})-\\widetilde{\\mathcal{G}}\\| = \\frac{1}{\\eta}\\|\\bar{x}_{s,k+1}-\\hat{x}_{s,k+1}\\|$ holds by definition. \n    By the optimality of $\\hat{x}_{s,k+1}$ and $\\bar{x}_{s,k+1}$ for the corresponding subproblems, Tseng's three point property (Lemma \\ref{lemma:Tseng-3point}) indicates that\n    \\begin{eqnarray*}\n        &&\\langle v_{s,k}, \\bar{x}_{s,k+1}\\rangle + \\phi(\\bar{x}_{s,k+1}) + \\frac{D_h(\\bar{x}_{s,k+1},x_{s,k})}{\\eta} + \\frac{D_h(\\hat{x}_{s,k+1},\\bar{x}_{s,k+1})}{\\eta}\\qquad\\,\\,\\,\\\\\n        &\\leq& \\langle v_{s,k}, \\hat{x}_{s,k+1}\\rangle + \\phi(\\hat{x}_{s,k+1})+ \\frac{D_h(\\hat{x}_{s,k+1},x_{s,k})}{\\eta} \n    \\end{eqnarray*} \n","sha256_utf8":"9c7ddd39e1bdd15c04c0715f83b94be546472065a69300b1079d1da8a097cd7f"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"真实子问题三点界","source_units":[{"start":176462,"end":176826,"start_line":1437,"end_line":1441,"raw_tex":"    and \n    \\begin{eqnarray*}\n        && \\langle \\nabla f(x_{s,k}), \\hat{x}_{s,k+1}\\rangle + \\phi(\\hat{x}_{s,k+1})+\\frac{D_h(\\hat{x}_{s,k+1},x_{s,k})}{\\eta} + \\frac{D_h(\\bar{x}_{s,k+1},\\hat{x}_{s,k+1})}{\\eta}\\\\\n        &\\leq& \\langle \\nabla f(x_{s,k}), \\bar{x}_{s,k+1}\\rangle + \\phi(\\bar{x}_{s,k+1})+\\frac{D_h(\\bar{x}_{s,k+1},x_{s,k})}{\\eta}\n    \\end{eqnarray*} \n","sha256_utf8":"db1cb46b65bdee6da66eb5aefaa3e0dd846361969ec0545e62c6fdab8f3dca32"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"合并最优性不等式","source_units":[{"start":176826,"end":177375,"start_line":1442,"end_line":1445,"raw_tex":"    Summing up the two inequalities and applying Lemma \\ref{lemma:grd-vs-grdmap} gives \n    \\begin{align} &\\frac{\\mu_h([\\bar{x}_{s,k+1},\\hat{x}_{s,k+1}])}{\\eta}\\cdot\\|\\bar{x}_{s,k+1}-\\hat{x}_{s,k+1}\\|^2 \\leq \\frac{D_h(\\bar{x}_{s,k+1},\\hat{x}_{s,k+1})}{\\eta} +  \\frac{D_h(\\hat{x}_{s,k+1},\\bar{x}_{s,k+1})}{\\eta}\\nonumber\\\\\n    &\\qquad\\qquad\\qquad\\qquad\\qquad\\,\\,\\,\\, \\leq  \\langle v_{s,k}-\\nabla f(x_{s,k}), \\hat{x}_{s,k+1}-\\bar{x}_{s,k+1}\\rangle \\leq  \\|v_{s,k}-\\nabla f(x_{s,k})\\|\\cdot\\|\\hat{x}_{s,k+1}-\\bar{x}_{s,k+1}\\|\\nonumber.\n    \\end{align} \n","sha256_utf8":"553c0fbccdd9aba42823d0ab386642bcff49b88b5beba9da88ef02654246b6e7"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"近端点差控制映射","source_units":[{"start":177375,"end":178126,"start_line":1446,"end_line":1451,"raw_tex":"    Hence $\\|\\hat{x}_{s,k+1}-\\bar{x}_{s,k+1}\\|\\leq \\frac{\\eta\\|v_{s,k}-\\nabla f(x_{s,k})\\|}{\\mu_h([\\bar{x}_{s,k+1},\\hat{x}_{s,k+1}])}$, and $\\big\\|\\cGshe(x_{s,k})-\\widetilde{\\mathcal{G}}\\big\\|\\leq\\frac{\\|v_{s,k}-\\nabla f(x_{s,k})\\|}{\\mu_h([\\bar{x}_{s,k+1},\\hat{x}_{s,k+1}])}$. As a result, \n    \\begin{eqnarray*}\n        \\big\\|\\cGshe(x_{s,k})\\big\\|^2 \\leq \\left(\\big\\|\\widetilde{\\mathcal{G}}\\big\\|+\\big\\|\\cGshe(x_{s,k})-\\widetilde{\\mathcal{G}}\\big\\|\\right)^2\\leq  \\frac{2\\|x_{s,k}-\\bar{x}_{s,k+1}\\|^2}{\\eta^2} + \\frac{2\\|v_{s,k}-\\nabla f(x_{s,k})\\|^2}{\\mu_h^2(\\mathcal{X}_s)},\n    \\end{eqnarray*} \n    where the last inequality is because $(a+b)^2\\leq 2a^2+2b^2$ and $\\mu_h^2\\big([\\bar{x}_{s,k+1},\\hat{x}_{s,k+1}]\\big)\\geq\\mu_h^2(\\cX_s)$. \n\\end{proof}","sha256_utf8":"bec590afd9fae69fb87734baeaeff166c9f97a017640ea31ba10f23e4d2c2feb"}]},{"id":"S049","semantic_type":"proposition","kind":"proposition","title":"梯度差的类利普希茨界","source_units":[{"start":64631,"end":65305,"start_line":486,"end_line":494,"raw_tex":"\\begin{proposition}[Lipschitz-like bound] \n    \\label{proposition:Lipschitz} \n    Suppose $f$ is $L$-smooth adaptable to some KC-regular kernel $h$. Let $\\delta>0$ and let $\\cX$ be any convex set with $\\mathrm{diam}(\\cX)\\leq \\delta$. Then for any $x,y\\in\\cX$, and an arbitrary interpolation point $z_\\theta := \\theta x + (1-\\theta)y$ with $ \\theta\\in[0,1]$, we have  \n    \\begin{equation}\n        \\label{prop:Lipschitz}\n        \\frac{\\|\\nabla f(z_\\theta)-\\nabla f(y)\\|^2}{2L^2\\mu_h(\\cX)}\\leq  \\theta^2\\kappa_h^2(\\cX) D_h(x,y),\n    \\end{equation}\n    where by KC-regularity, we know the constant upper bound $\\kappa_h(\\cX)\\leq \\kappa_h^\\delta$ always hold. \n\\end{proposition}","sha256_utf8":"eb79dcf9d4c4a8313947e0a797f891ff732ee2855771af4424d4dc05622c6895"}]},{"id":"S050","semantic_type":"proof","kind":"proof","title":"梯度差界完整证明","source_units":[{"start":174721,"end":175585,"start_line":1413,"end_line":1422,"raw_tex":"\\begin{proof}\n     By Lemma \\ref{lemma:B.1} and the assumption that $f$ is $L$-smooth adaptable to $h$, we know \n     $$\\max\\left\\{\\|\\nabla^2f(x)\\|: x\\in\\cX\\right\\} \\leq \\max\\left\\{L\\|\\nabla^2h(x)\\|: x\\in\\cX\\right\\} = L\\cdot L_h(\\cX).$$\n     As $\\cX$ is convex, the line segment $[x,y]\\subseteq\\cX$ and hence \n     $$\\|\\nabla f(z_\\theta)-\\nabla f(y)\\|^2\\leq L^2L_h^2(\\cX)\\|z_\\theta-y\\|^2 = L^2\\theta^2 L_h^2(\\cX)\\|x-y\\|^2.$$\n     Combined with    the fact that $$D_h(x,y)\\geq\\frac{\\mu_h([x,y])}{2}\\|x-y\\|^2\\geq  \\frac{\\mu_h(\\cX)}{2}\\|x-y\\|^2,$$\n     we obtain\n     $$\\frac{\\|\\nabla f(z_\\theta)-\\nabla f(y)\\|^2}{2L^2\\mu_h(\\cX)} \\leq \\theta^2\\cdot\\frac{L_h^2(\\cX)}{\\mu_h^2(\\cX)}\\cdot\\frac{\\mu_h(\\cX)}{2}\\|x-y\\|^2\\leq \\theta^2\\kappa_h^2(\\cX)D_h(x,y),$$\n     where KC-regularity guarantees that $\\kappa_h(\\cX)$ is always upper bounded by $\\kappa_h^\\delta$.\n\\end{proof}","sha256_utf8":"8e06f09b0e70bd32ead6a1847f2b4e3bd888124a74b81b8e4e20fc3422ecdd5c"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"海森矩阵范数界","source_units":[{"start":174721,"end":174958,"start_line":1413,"end_line":1415,"raw_tex":"\\begin{proof}\n     By Lemma \\ref{lemma:B.1} and the assumption that $f$ is $L$-smooth adaptable to $h$, we know \n     $$\\max\\left\\{\\|\\nabla^2f(x)\\|: x\\in\\cX\\right\\} \\leq \\max\\left\\{L\\|\\nabla^2h(x)\\|: x\\in\\cX\\right\\} = L\\cdot L_h(\\cX).$$\n","sha256_utf8":"fb4cbf03a3947585cb6be636d0f2a94f0076e16e1c97468551276410ac14438c"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"线段梯度差与散度界","source_units":[{"start":174958,"end":175266,"start_line":1416,"end_line":1418,"raw_tex":"     As $\\cX$ is convex, the line segment $[x,y]\\subseteq\\cX$ and hence \n     $$\\|\\nabla f(z_\\theta)-\\nabla f(y)\\|^2\\leq L^2L_h^2(\\cX)\\|z_\\theta-y\\|^2 = L^2\\theta^2 L_h^2(\\cX)\\|x-y\\|^2.$$\n     Combined with    the fact that $$D_h(x,y)\\geq\\frac{\\mu_h([x,y])}{2}\\|x-y\\|^2\\geq  \\frac{\\mu_h(\\cX)}{2}\\|x-y\\|^2,$$\n","sha256_utf8":"d1a6f2e99e412fc0c1c5fdfda475ccbf9b1707cb956589825e1f3bdad908c3e4"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"合并局部条件数","source_units":[{"start":175266,"end":175585,"start_line":1419,"end_line":1422,"raw_tex":"     we obtain\n     $$\\frac{\\|\\nabla f(z_\\theta)-\\nabla f(y)\\|^2}{2L^2\\mu_h(\\cX)} \\leq \\theta^2\\cdot\\frac{L_h^2(\\cX)}{\\mu_h^2(\\cX)}\\cdot\\frac{\\mu_h(\\cX)}{2}\\|x-y\\|^2\\leq \\theta^2\\kappa_h^2(\\cX)D_h(x,y),$$\n     where KC-regularity guarantees that $\\kappa_h(\\cX)$ is always upper bounded by $\\kappa_h^\\delta$.\n\\end{proof}","sha256_utf8":"9edabf21af2677411ac26ab6e3601e4595503a8803c44cd92db4e7384db810d1"}]},{"id":"S054","semantic_type":"lemma","kind":"lemma","title":"矩阵序推出范数界","source_units":[{"start":173330,"end":173527,"start_line":1390,"end_line":1393,"raw_tex":"\\begin{lemma}\n    \\label{lemma:B.1}\n    Let $A$ be a symmetric matrix, and let $B\\succeq0$ be positive semidefinite matrix. Then $-B\\preceq A\\preceq B$ indicates that $\\|A\\|\\leq \\|B\\|$.\n\\end{lemma}","sha256_utf8":"dda4533c88b205b0723dde19beb68cfa8be43e44b6c0fed6a8b7b223f581945b"}]},{"id":"S055","semantic_type":"proof","kind":"proof","title":"矩阵范数界完整证明","source_units":[{"start":173529,"end":174598,"start_line":1394,"end_line":1411,"raw_tex":"\\begin{proof}\n    For any symmetric but not semidefinite matrix $A$, it is easy to verify that \n    \\begin{equation}\n        \\label{lm:B.1-1}\n        \\|A\\| = \\max\\big\\{|\\lambda_{\\max}(A)|,|\\lambda_{\\min}(A)|\\big\\}.\n    \\end{equation}\n    Let $u\\neq0$ be the eigenvector of $A$ associated with the maximum eigenvalue.  Then by Rayleigh’s principle for maximum eigenvalue, $B\\succeq A$ indicates that $B-A\\succeq0$ and hence \n    \\begin{equation}\n        \\label{lm:B.1-2}\n        0\\leq \\frac{u^\\top(B-A)u}{u^\\top u} \\leq \\max_{x\\neq0}\\left\\{\\frac{x^\\top Bx}{x^\\top x}\\right\\} - \\frac{u^\\top(B-A)u}{u^\\top u} = \\lambda_{\\max}(B) - \\lambda_{\\max}(A). \n    \\end{equation}\n    Similarly, $A\\succeq-B$ indicates that $B-(-A)\\succeq0$, then \\eqref{lm:B.1-2} immediately gives\n    \\begin{equation}\n        \\label{lm:B.1-3}\n        0\\leq  \\lambda_{\\max}(B) - \\lambda_{\\max}(-A) = \\lambda_{\\max}(B) + \\lambda_{\\min}(A).\n    \\end{equation}\n    Combining \\eqref{lm:B.1-1}-\\eqref{lm:B.1-3} and the fact that $\\|B\\| = \\lambda_{\\max}(B)$ for p.s.d. matrix proves the lemma.\n\\end{proof}","sha256_utf8":"8b610ab764f277eea976008a056ddfec5c13c3d7b715186259df34980d1d7a80"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"对称矩阵谱范数","source_units":[{"start":173529,"end":173763,"start_line":1394,"end_line":1399,"raw_tex":"\\begin{proof}\n    For any symmetric but not semidefinite matrix $A$, it is easy to verify that \n    \\begin{equation}\n        \\label{lm:B.1-1}\n        \\|A\\| = \\max\\big\\{|\\lambda_{\\max}(A)|,|\\lambda_{\\min}(A)|\\big\\}.\n    \\end{equation}\n","sha256_utf8":"ca32dc5a5d89a04a1139ba5ef9d1a51b7de618f43f56a84dc583b8699895670c"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"最大特征值比较","source_units":[{"start":173763,"end":174196,"start_line":1400,"end_line":1404,"raw_tex":"    Let $u\\neq0$ be the eigenvector of $A$ associated with the maximum eigenvalue.  Then by Rayleigh’s principle for maximum eigenvalue, $B\\succeq A$ indicates that $B-A\\succeq0$ and hence \n    \\begin{equation}\n        \\label{lm:B.1-2}\n        0\\leq \\frac{u^\\top(B-A)u}{u^\\top u} \\leq \\max_{x\\neq0}\\left\\{\\frac{x^\\top Bx}{x^\\top x}\\right\\} - \\frac{u^\\top(B-A)u}{u^\\top u} = \\lambda_{\\max}(B) - \\lambda_{\\max}(A). \n    \\end{equation}\n","sha256_utf8":"05e26a37c2c9e62e177ade640c18c9ebe67da56d0bdbe069eee902fd2d38f2e5"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"最小特征值与结论","source_units":[{"start":174196,"end":174598,"start_line":1405,"end_line":1411,"raw_tex":"    Similarly, $A\\succeq-B$ indicates that $B-(-A)\\succeq0$, then \\eqref{lm:B.1-2} immediately gives\n    \\begin{equation}\n        \\label{lm:B.1-3}\n        0\\leq  \\lambda_{\\max}(B) - \\lambda_{\\max}(-A) = \\lambda_{\\max}(B) + \\lambda_{\\min}(A).\n    \\end{equation}\n    Combining \\eqref{lm:B.1-1}-\\eqref{lm:B.1-3} and the fact that $\\|B\\| = \\lambda_{\\max}(B)$ for p.s.d. matrix proves the lemma.\n\\end{proof}","sha256_utf8":"002c2d218304d880360b3bafbbb54e2a12f6d0b4a4680b885ea69033fc4d2b72"}]},{"id":"S059","semantic_type":"lemma","kind":"lemma","title":"近端残差曲率比较","source_units":[{"start":34849,"end":35599,"start_line":259,"end_line":268,"raw_tex":"\\begin{lemma}\n    \\label{lemma:grd-vs-grdmap}\n    For any $x,v\\in\\RR^d$ and any strictly convex kernel $h$, denote $x_\\lambda^+(v) := \\mathbf{T}_{\\phi,h}^\\lambda(x,v)$.  Let $[x,x_\\lambda^+(v)]$ be the line segment between $x_\\lambda^+(v)$ and $x$, then there exists $u\\in\\partial\\phi(x_\\lambda^+(v))$ such that  \n    \\begin{equation}\n        \\mu_h\\big([x,x_\\lambda^+(v)]\\big)\\cdot \\|x-x_\\lambda^+(v)\\|\\leq\\lambda\\|v+u\\|\\leq L_h\\big([x,x_\\lambda^+(v)]\\big)\\cdot \\|x-x_\\lambda^+(v)\\|\\,,\\nonumber\n    \\end{equation} \n    \\begin{equation}\n        \\sqrt{2\\mu_h\\big([x,x_\\lambda^+(v)]\\big)\\cdot D_h(x_\\lambda^+(v),x)}\\leq \\lambda\\|v+u\\|\\leq \\sqrt{2L_h\\big([x,x_\\lambda^+(v)]\\big)\\cdot D_h(x_\\lambda^+(v),x)}\\,\\,.\\nonumber\n    \\end{equation}   \n\\end{lemma}","sha256_utf8":"d4596091c9b5c9f98934627ae3f77875b8037b95f10e5e853a9c9db3bade1b6c"}]},{"id":"S060","semantic_type":"proof","kind":"proof","title":"残差比较完整证明","source_units":[{"start":35600,"end":37408,"start_line":269,"end_line":282,"raw_tex":"\\begin{proof}\n    By the optimality condition of the subproblem $x_\\lambda^+(v)=\\argmin_{y\\in\\RR^d}\\, y^\\top v + \\phi(y) + \\frac{1}{\\lambda}D_h(y,x)$, we have $0\\in \\partial\\phi(x_\\lambda^+(v))+v+\\frac{1}{\\lambda}\\nabla_y D_h(y,x)|_{y=x_\\lambda^+(v)}$. Namely, there exists  $u\\in\\partial\\phi(x_\\lambda^+(v))$ such that \n    \\begin{equation}\n        \\label{lm:grd-vs-grdmap-1}\n        \\lambda(v + u) +  \\big(\\nabla h(x_\\lambda^+(v))-\\nabla h(x)\\big)=0.\n    \\end{equation}\n    Then \\cite[Theorem 2.1.9]{nesterov2018lectures}, together with the definition of $\\mu_h(\\cdot)$ and $L_h(\\cdot)$, indicates that \n    $$\\mu_h\\big([x,x_\\lambda^+(v)]\\big)\\cdot \\|x-x_\\lambda^+(v)\\|\\leq\\|\\nabla h(x_\\lambda^+(v))-\\nabla h(x)\\|\\leq L_h\\big([x,x_\\lambda^+(v)]\\big)\\cdot \\|x-x_\\lambda^+(v)\\|.$$\n    Combining this bound with equation \\eqref{lm:grd-vs-grdmap-1} proves the first inequality of Lemma \\ref{lemma:grd-vs-grdmap}. Also observe that $\\nabla_y^2 D_h(y,x) = \\nabla^2h(y)$, we have \n    $$\\mu_h([x,x_\\lambda^+(v)])\\cdot I \\preceq \\nabla^2_y D_h(y,x) \\preceq L_h([x,x_\\lambda^+(v)])\\cdot I\\quad\\mbox{for}\\quad \\forall y\\in[x,x_\\lambda^+(v)].$$ \n    Combined with the fact that $\\nabla_y D_h(y,x)|_{y=x} = 0$, then   \\cite[Theorem 2.1.5, Eq.(2.1.10)]{nesterov2018lectures} and \\cite[Theorem 2.1.10, Eq.(2.1.24)]{nesterov2018lectures} immediately indicates \n    $$\\frac{\\left\\|\\nabla_y D_h(y,x)|_{y=x_\\lambda^+(v)}\\right\\|^2}{2L_h([x,x_\\lambda^+(v)])}\\leq D_h(x_\\lambda^+(v),x)-D_h(x,x)\\leq \\frac{\\left\\|\\nabla_y D_h(y,x)|_{y=x_\\lambda^+(v)}\\right\\|^2}{2\\mu_h([x,x_\\lambda^+(v)])}.$$\n    Then substituting $D_h(x,x)=0$ and $\\lambda(v+u) = \\nabla h(x)-\\nabla h(x_\\lambda^+(v)) = -\\nabla_z D_h(y,x)\\mid_{y=x_\\lambda^+(v)}$ to the above bound proves the second inequality of Lemma \\ref{lemma:grd-vs-grdmap}.  \n\\end{proof}","sha256_utf8":"75277b6d90d035677a22667faac189fb6a39f15b5856d2c527432ec52e420f4e"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"近端最优性恒等式","source_units":[{"start":35600,"end":36072,"start_line":269,"end_line":274,"raw_tex":"\\begin{proof}\n    By the optimality condition of the subproblem $x_\\lambda^+(v)=\\argmin_{y\\in\\RR^d}\\, y^\\top v + \\phi(y) + \\frac{1}{\\lambda}D_h(y,x)$, we have $0\\in \\partial\\phi(x_\\lambda^+(v))+v+\\frac{1}{\\lambda}\\nabla_y D_h(y,x)|_{y=x_\\lambda^+(v)}$. Namely, there exists  $u\\in\\partial\\phi(x_\\lambda^+(v))$ such that \n    \\begin{equation}\n        \\label{lm:grd-vs-grdmap-1}\n        \\lambda(v + u) +  \\big(\\nabla h(x_\\lambda^+(v))-\\nabla h(x)\\big)=0.\n    \\end{equation}\n","sha256_utf8":"d9dccfe901a677dec1f153ea24c08ff4559e793b6a6801c055f2e06b6ad80c32"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"残差与位移比较","source_units":[{"start":36072,"end":36576,"start_line":275,"end_line":277,"raw_tex":"    Then \\cite[Theorem 2.1.9]{nesterov2018lectures}, together with the definition of $\\mu_h(\\cdot)$ and $L_h(\\cdot)$, indicates that \n    $$\\mu_h\\big([x,x_\\lambda^+(v)]\\big)\\cdot \\|x-x_\\lambda^+(v)\\|\\leq\\|\\nabla h(x_\\lambda^+(v))-\\nabla h(x)\\|\\leq L_h\\big([x,x_\\lambda^+(v)]\\big)\\cdot \\|x-x_\\lambda^+(v)\\|.$$\n    Combining this bound with equation \\eqref{lm:grd-vs-grdmap-1} proves the first inequality of Lemma \\ref{lemma:grd-vs-grdmap}. Also observe that $\\nabla_y^2 D_h(y,x) = \\nabla^2h(y)$, we have \n","sha256_utf8":"29a50af5ebcd48bd0451b9506db59205ad3c1fefe65aaa3b5cbd45cef11e5b60"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"残差与散度比较","source_units":[{"start":36576,"end":37408,"start_line":278,"end_line":282,"raw_tex":"    $$\\mu_h([x,x_\\lambda^+(v)])\\cdot I \\preceq \\nabla^2_y D_h(y,x) \\preceq L_h([x,x_\\lambda^+(v)])\\cdot I\\quad\\mbox{for}\\quad \\forall y\\in[x,x_\\lambda^+(v)].$$ \n    Combined with the fact that $\\nabla_y D_h(y,x)|_{y=x} = 0$, then   \\cite[Theorem 2.1.5, Eq.(2.1.10)]{nesterov2018lectures} and \\cite[Theorem 2.1.10, Eq.(2.1.24)]{nesterov2018lectures} immediately indicates \n    $$\\frac{\\left\\|\\nabla_y D_h(y,x)|_{y=x_\\lambda^+(v)}\\right\\|^2}{2L_h([x,x_\\lambda^+(v)])}\\leq D_h(x_\\lambda^+(v),x)-D_h(x,x)\\leq \\frac{\\left\\|\\nabla_y D_h(y,x)|_{y=x_\\lambda^+(v)}\\right\\|^2}{2\\mu_h([x,x_\\lambda^+(v)])}.$$\n    Then substituting $D_h(x,x)=0$ and $\\lambda(v+u) = \\nabla h(x)-\\nabla h(x_\\lambda^+(v)) = -\\nabla_z D_h(y,x)\\mid_{y=x_\\lambda^+(v)}$ to the above bound proves the second inequality of Lemma \\ref{lemma:grd-vs-grdmap}.  \n\\end{proof}","sha256_utf8":"383cca497688364f15af73318be3f61f51cbad2cae5acac42a44a9e6e1fd9823"}]},{"id":"S064","semantic_type":"lemma","kind":"lemma","title":"提前终止事件概率","source_units":[{"start":81024,"end":81539,"start_line":634,"end_line":640,"raw_tex":"\\begin{lemma}\n\\label{lemma:event-halway} \nFor any $S$ epochs  generated by Algorithm \\ref{algorithm:proxSARAH}, define the set $\\mathcal{I}_1$ and event $\\mathcal{A}_1(m_1)$ as  \n$$\\mathcal{I}_1:=\\big\\{s\\in[S] :  \\tau_s < \\tau\\big\\}\\qquad\\mbox{and}\\qquad\\mathcal{A}_1(m_1):=\\left\\{\\omega: |\\mathcal{I}_1|\\geq m_1\\right\\}.$$\nwhere $m_1>0$ is an arbitrary positive number. Then it holds that \n$$\\mathrm{Prob}\\left(\\mathcal{A}_1(m_1)\\right)\\leq \\frac{32\\gamma\\tau\\Delta_\\Psi}{\\eta\\mu\\delta^2\\cdot m_1}\\,.$$\n\\end{lemma}","sha256_utf8":"4ba09c16ec16e2b38117f191316a1b1e73b53953e20028359129e9755f9aea60"}]},{"id":"S065","semantic_type":"proof","kind":"proof","title":"终止事件完整证明","source_units":[{"start":81540,"end":83698,"start_line":641,"end_line":664,"raw_tex":"\\begin{proof}\nBy lemma \\ref{lemma:proxSARAH-grad-map-mid}, ignoring the restricted primal gradient mapping terms yields   \n\\begin{eqnarray}\n    \\label{lm:event-halfway-1}\n    \\frac{\\Delta_\\Psi}{\\gamma\\eta} &\\geq& \\mathbb{E}\\left[\\sum_{s=1}^S\\sum_{k=0}^{\\tau_s-1}D_h(\\bar{x}_{s,k+1},x_{s,k}) \\right]\\nonumber\\\\ \n    & \\geq & \\mathbb{E}\\left[\\frac{\\mu}{2\\gamma^2}\\sum_{s\\in\\mathcal{I}_1}\\sum_{k=0}^{\\tau_s-1}\\|{x}_{s,k+1}-x_{s,k}\\|^2 \\right] \\\\\n    & \\geq & \\frac{\\mu}{2\\gamma^2}\\cdot \\mathrm{Prob}\\left(\\mathcal{A}_1(m_1)\\right) \\cdot\\mathbb{E}\\left[\\sum_{s\\in\\mathcal{I}_1}\\sum_{k=0}^{\\tau_s-1}\\|{x}_{s,k+1}-x_{s,k}\\|^2\\,\\Big|\\, \\mathcal{A}_1(m_1) \\right] \\nonumber,\n\\end{eqnarray}\nwhere the last inequality is due to the fact that   \n$$\\EE[X] = \\mathrm{Prob}(\\cA)\\cdot\\EE\\big[X\\mid\\cA\\big] + \\mathrm{Prob}(\\cA^c)\\cdot\\EE\\big[X\\mid\\cA^c\\big]\\geq \\mathrm{Prob}(\\cA)\\cdot\\EE\\big[X\\mid\\cA\\big]$$\nfor any non-negative random variable $X\\geq0$ and any event $\\cA$. Therefore, conditioning on the event $\\mathcal{A}_1(m_1)$, for any epoch $s\\in\\mathcal{I}_1$, we will have $\\|x_{s,\\tau_s}-x_{s,0}\\|\\geq\\frac{\\delta}{4}$ because of Line 3 and Line 9 of Algorithm \\ref{algorithm:proxSARAH}. Consequently, for $\\forall s\\in\\mathcal{I}_1$, the triangle inequality and the arithmetic inequality indicate that\n\\begin{equation}\n    \\label{lm:event-halfway-2}\n    \\frac{\\delta}{4\\tau_s}\\leq \\frac{\\|x_{s,\\tau_s}-x_{s,0}\\|}{\\tau_s}\\leq\\frac{\\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|}{\\tau_s}\\leq  \\sqrt{\\frac{\\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|^2}{\\tau_s}}\\,.\n\\end{equation} \nBecause $\\tau_s\\leq\\tau$ always holds, we have \n\\begin{align}\n    \\label{lm:event-halfway-3}\n    \\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|^2  \\geq  \\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|^2\\geq\\frac{\\delta^2}{16\\tau_s}\\geq\\frac{\\delta^2}{16\\tau}\\,.\n\\end{align}\nNote that the above inequalities hold w.p. 1 conditioning on $\\cA_1(m_1)$. Combined with \\eqref{lm:event-halfway-1}, we have \n$$\\frac{\\Delta_\\Psi}{\\gamma\\eta} \\geq\\frac{\\mu}{2\\gamma^2}\\cdot \\mathrm{Prob}\\left(\\mathcal{A}_1(m_1)\\right)\\cdot\\frac{m_1\\delta^2}{16\\tau}.$$\nRearranging the terms completes the proof. \n\\end{proof}","sha256_utf8":"14ca1ba56ad86c0d3ebf174be9e539543205eb8d97a269b8d17510ef92e65a90"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"条件位移能量下界","source_units":[{"start":81540,"end":82433,"start_line":641,"end_line":650,"raw_tex":"\\begin{proof}\nBy lemma \\ref{lemma:proxSARAH-grad-map-mid}, ignoring the restricted primal gradient mapping terms yields   \n\\begin{eqnarray}\n    \\label{lm:event-halfway-1}\n    \\frac{\\Delta_\\Psi}{\\gamma\\eta} &\\geq& \\mathbb{E}\\left[\\sum_{s=1}^S\\sum_{k=0}^{\\tau_s-1}D_h(\\bar{x}_{s,k+1},x_{s,k}) \\right]\\nonumber\\\\ \n    & \\geq & \\mathbb{E}\\left[\\frac{\\mu}{2\\gamma^2}\\sum_{s\\in\\mathcal{I}_1}\\sum_{k=0}^{\\tau_s-1}\\|{x}_{s,k+1}-x_{s,k}\\|^2 \\right] \\\\\n    & \\geq & \\frac{\\mu}{2\\gamma^2}\\cdot \\mathrm{Prob}\\left(\\mathcal{A}_1(m_1)\\right) \\cdot\\mathbb{E}\\left[\\sum_{s\\in\\mathcal{I}_1}\\sum_{k=0}^{\\tau_s-1}\\|{x}_{s,k+1}-x_{s,k}\\|^2\\,\\Big|\\, \\mathcal{A}_1(m_1) \\right] \\nonumber,\n\\end{eqnarray}\nwhere the last inequality is due to the fact that   \n$$\\EE[X] = \\mathrm{Prob}(\\cA)\\cdot\\EE\\big[X\\mid\\cA\\big] + \\mathrm{Prob}(\\cA^c)\\cdot\\EE\\big[X\\mid\\cA^c\\big]\\geq \\mathrm{Prob}(\\cA)\\cdot\\EE\\big[X\\mid\\cA\\big]$$\n","sha256_utf8":"a179d5bf71b6d7c09351b8692b06393e060a57fcf62d2da51207cbb38c458f03"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"轮次终止必有位移","source_units":[{"start":82433,"end":83109,"start_line":651,"end_line":655,"raw_tex":"for any non-negative random variable $X\\geq0$ and any event $\\cA$. Therefore, conditioning on the event $\\mathcal{A}_1(m_1)$, for any epoch $s\\in\\mathcal{I}_1$, we will have $\\|x_{s,\\tau_s}-x_{s,0}\\|\\geq\\frac{\\delta}{4}$ because of Line 3 and Line 9 of Algorithm \\ref{algorithm:proxSARAH}. Consequently, for $\\forall s\\in\\mathcal{I}_1$, the triangle inequality and the arithmetic inequality indicate that\n\\begin{equation}\n    \\label{lm:event-halfway-2}\n    \\frac{\\delta}{4\\tau_s}\\leq \\frac{\\|x_{s,\\tau_s}-x_{s,0}\\|}{\\tau_s}\\leq\\frac{\\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|}{\\tau_s}\\leq  \\sqrt{\\frac{\\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|^2}{\\tau_s}}\\,.\n\\end{equation} \n","sha256_utf8":"278f6fd4192fbbd159f236dd7244faae2d82243535e15bea5bf6b92b463a2917"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"每轮位移平方和","source_units":[{"start":83109,"end":83374,"start_line":656,"end_line":660,"raw_tex":"Because $\\tau_s\\leq\\tau$ always holds, we have \n\\begin{align}\n    \\label{lm:event-halfway-3}\n    \\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|^2  \\geq  \\sum_{k=0}^{\\tau_s-1}\\|x_{s,k+1}-x_{s,k}\\|^2\\geq\\frac{\\delta^2}{16\\tau_s}\\geq\\frac{\\delta^2}{16\\tau}\\,.\n\\end{align}\n","sha256_utf8":"e3514a1302eace3ece43753c225758b9238de71056bedb980ea5b1081cd3dd26"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"解出提前终止概率","source_units":[{"start":83374,"end":83698,"start_line":661,"end_line":664,"raw_tex":"Note that the above inequalities hold w.p. 1 conditioning on $\\cA_1(m_1)$. Combined with \\eqref{lm:event-halfway-1}, we have \n$$\\frac{\\Delta_\\Psi}{\\gamma\\eta} \\geq\\frac{\\mu}{2\\gamma^2}\\cdot \\mathrm{Prob}\\left(\\mathcal{A}_1(m_1)\\right)\\cdot\\frac{m_1\\delta^2}{16\\tau}.$$\nRearranging the terms completes the proof. \n\\end{proof}","sha256_utf8":"7ad0ebf8c49804ad1f56495143dd70be94a0f07a39e01a396a25515d7a39d0b7"}]},{"id":"S070","semantic_type":"lemma","kind":"lemma","title":"受限映射触边概率","source_units":[{"start":84050,"end":84641,"start_line":667,"end_line":673,"raw_tex":"\\begin{lemma}\n    \\label{lemma:event-grad-map}\n    For any $S$ epochs generated by Algorithm \\ref{algorithm:proxSARAH}, define the set $\\mathcal{I}_2$ and event $\\mathcal{A}_2(m_2)$ as\n    $$\\mathcal{I}_2:=\\left\\{(s,k) :  \\hat{x}_{s,k+1}\\in\\partial \\mathcal{X}_s, 1\\leq s\\leq S, 0\\leq k\\leq \\tau_s-1\\right\\}\\quad\\mbox{and}\\quad\\mathcal{A}_2(m_2):=\\big\\{\\omega : |\\mathcal{I}_2|\\geq m_2\\big\\}$$\n    where  $m_2>0$ is an arbitrary positive number. Then it holds that \n    $$\\mathrm{Prob}\\left(\\mathcal{A}_2(m_2)\\right)\\leq \\frac{128\\eta\\Delta_\\Psi}{\\gamma\\mu\\delta^2\\cdot m_2}\\,.$$\n\\end{lemma}","sha256_utf8":"618ec4348b6ea90a212e7dca649b371eff23ea4fe89b95b33da6d835413d9b6c"}]},{"id":"S071","semantic_type":"proof","kind":"proof","title":"触边事件完整证明","source_units":[{"start":84642,"end":85994,"start_line":674,"end_line":687,"raw_tex":"\\begin{proof}\nSimilar to the proof of Lemma \\ref{lemma:event-halway}, ignoring the Bregman divergence terms in Lemma \\ref{lemma:proxSARAH-grad-map-mid} yields   \n\\begin{eqnarray}\n    \\label{lm:event-grad-map-1}\n    \\frac{\\Delta_\\Psi}{\\gamma\\eta} & \\geq & \\mathbb{E}\\left[\\sum_{s=1}^S\\sum_{k=0}^{\\tau_s-1}\\frac{\\mu}{8}\\big\\|\\cGshe(x_{s,k})\\big\\|^2\\right]\\nonumber\\\\\n    & = & \\frac{\\mu}{8\\eta^2}\\mathbb{E}\\left[\\sum_{s=1}^S\\sum_{k=0}^{\\tau_s-1}\\|\\hat{x}_{s,k+1}-x_{s,k}\\|^2\\right]\\\\\n    & \\geq & \\frac{\\mu}{8\\eta^2}\\mathbb{E}\\Bigg[\\sum_{(s,k)\\in\\mathcal{I}_2} \\|\\hat{x}_{s,k+1}-x_{s,k}\\|^2\\Bigg]\\nonumber\\\\\n    & \\geq & \\frac{\\mu}{8\\eta^2}\\cdot\\mathrm{Prob}\\left(\\mathcal{A}_2(m_2)\\right)\\cdot\\mathbb{E}\\Bigg[\\sum_{(s,k)\\in\\mathcal{I}_2}\\|\\hat{x}_{s,k+1}-x_{s,k}\\|^2\\,\\Big|\\,\\mathcal{A}_2(m_2)\\Bigg]\\nonumber.\n\\end{eqnarray}\nNote that for  $\\forall(s,k)\\in\\mathcal{I}_2$, we have $\\mathrm{dist}(x_{s,k},\\partial\\cX_s)\\geq\\delta/4$ while $\\hat{x}_{s,k+1}\\in\\partial\\cX_s$. Consequently, one must have $\\|\\hat{x}_{s,k+1}-{x}_{s,k}\\|\\geq \n\\frac{\\delta}{4}$. Conditioning on the event $\\cA_2(m_2)$, we have $|\\mathcal{I}_2|\\geq m_2$ and \n$$\\mathbb{E}\\Bigg[\\sum_{(s,k)\\in\\mathcal{I}_2}\\|\\hat{x}_{s,k+1}-x_{s,k}\\|^2\\,\\Big|\\,\\mathcal{A}_2(m_2)\\Bigg]\\geq \\frac{m_2\\delta^2}{16}.$$\nSubstitute this bound to \\eqref{lm:event-grad-map-1} proves lemma.  \n\\end{proof}","sha256_utf8":"207db1d169db39a36dd8a25328ac225a5949d88ab1c24d83569830c9e94b9019"}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","title":"触边事件条件能量","source_units":[{"start":84642,"end":85466,"start_line":674,"end_line":682,"raw_tex":"\\begin{proof}\nSimilar to the proof of Lemma \\ref{lemma:event-halway}, ignoring the Bregman divergence terms in Lemma \\ref{lemma:proxSARAH-grad-map-mid} yields   \n\\begin{eqnarray}\n    \\label{lm:event-grad-map-1}\n    \\frac{\\Delta_\\Psi}{\\gamma\\eta} & \\geq & \\mathbb{E}\\left[\\sum_{s=1}^S\\sum_{k=0}^{\\tau_s-1}\\frac{\\mu}{8}\\big\\|\\cGshe(x_{s,k})\\big\\|^2\\right]\\nonumber\\\\\n    & = & \\frac{\\mu}{8\\eta^2}\\mathbb{E}\\left[\\sum_{s=1}^S\\sum_{k=0}^{\\tau_s-1}\\|\\hat{x}_{s,k+1}-x_{s,k}\\|^2\\right]\\\\\n    & \\geq & \\frac{\\mu}{8\\eta^2}\\mathbb{E}\\Bigg[\\sum_{(s,k)\\in\\mathcal{I}_2} \\|\\hat{x}_{s,k+1}-x_{s,k}\\|^2\\Bigg]\\nonumber\\\\\n    & \\geq & \\frac{\\mu}{8\\eta^2}\\cdot\\mathrm{Prob}\\left(\\mathcal{A}_2(m_2)\\right)\\cdot\\mathbb{E}\\Bigg[\\sum_{(s,k)\\in\\mathcal{I}_2}\\|\\hat{x}_{s,k+1}-x_{s,k}\\|^2\\,\\Big|\\,\\mathcal{A}_2(m_2)\\Bigg]\\nonumber.\n\\end{eqnarray}\n","sha256_utf8":"a2e6b6691ed0b31e4b6d2f1f5c0df9d44355903fffebf11c7ccdbd751d473a2d"}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","title":"触边距离推出概率","source_units":[{"start":85466,"end":85994,"start_line":683,"end_line":687,"raw_tex":"Note that for  $\\forall(s,k)\\in\\mathcal{I}_2$, we have $\\mathrm{dist}(x_{s,k},\\partial\\cX_s)\\geq\\delta/4$ while $\\hat{x}_{s,k+1}\\in\\partial\\cX_s$. Consequently, one must have $\\|\\hat{x}_{s,k+1}-{x}_{s,k}\\|\\geq \n\\frac{\\delta}{4}$. Conditioning on the event $\\cA_2(m_2)$, we have $|\\mathcal{I}_2|\\geq m_2$ and \n$$\\mathbb{E}\\Bigg[\\sum_{(s,k)\\in\\mathcal{I}_2}\\|\\hat{x}_{s,k+1}-x_{s,k}\\|^2\\,\\Big|\\,\\mathcal{A}_2(m_2)\\Bigg]\\geq \\frac{m_2\\delta^2}{16}.$$\nSubstitute this bound to \\eqref{lm:event-grad-map-1} proves lemma.  \n\\end{proof}","sha256_utf8":"591f281f5e7d3d7fc0d7202bee7d41e932531cd899631faf032d093d3c6a86ac"}]},{"id":"S074","semantic_type":"lemma","kind":"lemma","title":"广义下降引理","source_units":[{"start":27746,"end":28058,"start_line":212,"end_line":216,"raw_tex":"\\begin{lemma}[Extended descent lemma, \\cite{bolte2018first}]\n    \\label{lemma:descent-lemma}\n    Suppose $f$ and $h$ satisfy Assumption \\ref{assumption:L-smad} for some constant $L>0$, then for any $\\forall x,y\\in\\RR^d$, it holds that \n    $$|f(x)-f(y)-\\langle\\nabla f(y),x-y\\rangle|\\leq LD_h(x,y).$$\n\\end{lemma}","sha256_utf8":"710102daa3d19b5c5c60c685263426e4d87fa6e94fed02ea3dd34d527a53c6bd"}]},{"id":"S075","semantic_type":"lemma","kind":"lemma","title":"三点性质","source_units":[{"start":168597,"end":168991,"start_line":1325,"end_line":1331,"raw_tex":"\\begin{lemma}[Three-Point Property of Tseng \\cite{tseng2008accelerated}]\n    \\label{lemma:Tseng-3point}\n    Let $\\phi(x)$ be a convex function, and let $D_h(\\cdot,\\cdot)$ be the Bregman distance for $h(\\cdot)$. For a given vector $z$, let\n    $z_+:=\\argmin_{x\\in Q} \\phi(x) + D_h(x,z).$\n    Then \n    $$\\phi(x)+D_h(x,z)\\geq\\phi(z_+) + D_h(z_+,z) + D_h(x,z_+), \\quad\\forall x\\in Q.$$\n\\end{lemma}","sha256_utf8":"5823b5632fa1a6033768c40d2564fc95b78dbc2a3fa158cf9b8ff8706a63c97a"}]},{"id":"S076","semantic_type":"lemma","kind":"lemma","title":"有限和递归方差界","source_units":[{"start":168995,"end":169988,"start_line":1335,"end_line":1346,"raw_tex":"\\begin{lemma}[Lemma 2 finite-sum case of \\cite{pham2020proxsarah}]\n\\label{lemma:proxSARAH-finite-sum}\n    Let $v_{s,k}$ be generated by \\eqref{alg:sarah-small}, suppose $|\\mathcal{B}_{s,k}| = b$ and the sampled index are uniformly randomly picked from $[n]$ with replacement, then  \n    \\begin{eqnarray}\n        \\mathbb{E}\\Big[\\|\\nabla f(x_{s,k})-v_{s,k}\\|^2\\,\\big|\\,x_{s,0}\\Big] \\leq \\sum_{j=0}^{k-1} \\mathbb{E}\\Big[\\|v_{s,j+1}-v_{s,j}\\|^2-\\|\\nabla f(x_{s,j+1})-\\nabla f(x_{s,j})\\|^2\\,\\big|\\, x_{s,0}\\Big], \\nonumber \n    \\end{eqnarray}\n    where the expectation term of $\\|v_{s,j+1}-v_{s,j}\\|^2$ satisfies\n    \\begin{eqnarray}\n        \\mathbb{E}\\Big[\\|v_{s,j+1}-v_{s,j}\\|^2\\,\\big|\\,x_{s,0}\\Big] \\leq  \\mathbb{E}\\bigg[\\|\\nabla f(x_{s,j+1})-\\nabla f(x_{s,j})\\|^2 + \\frac{1}{bn}\\sum_{i=1}^n\\|\\nabla f_i(x_{s,j+1})-\\nabla f_i(x_{s,j})\\|^2\\,\\big|\\,x_{s,0}\\bigg].  \\nonumber\n    \\end{eqnarray}  \n    In particular, we have slightly modified the second inequality to suit our analysis. \n\\end{lemma}","sha256_utf8":"c43893b1ab8a02d0fa702237517ce42ccbbf5073d88d77aa96c8bf667f1546d8"}]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","title":"光滑强凸标准估计","source_units":[],"declared_external":true,"declaration_text":["For a differentiable convex function g with L-Lipschitz gradient and strong convexity modulus μ>0, the standard gradient Lipschitz, strong monotonicity, and gradient-function-gap estimates apply. In particular, at a minimizer x with ∇g(x)=0, ||∇g(y)||²/(2L) ≤ g(y)-g(x) ≤ ||∇g(y)||²/(2μ). These are the Nesterov estimates invoked by the author for the Bregman function in B0027."]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","title":"光滑适应下降事实","source_units":[],"declared_external":true,"declaration_text":["The cited extended descent lemma of Bolte et al. states that convexity of Lh+f and Lh-f implies |f(x)-f(y)-<∇f(y),x-y>| ≤ L D_h(x,y)."]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","title":"凸近端三点事实","source_units":[],"declared_external":true,"declaration_text":["Tseng's cited three-point property: if z+ minimizes a convex function a(x)+D_h(x,z) over a convex set Q, then a(x)+D_h(x,z) ≥ a(z+)+D_h(z+,z)+D_h(x,z+) for every x∈Q."]},{"id":"S080","semantic_type":"external_fact","kind":"external_fact","title":"递归估计方差事实","source_units":[],"declared_external":true,"declaration_text":["For the finite-sum SARAH recursion initialized by an exact gradient and using independent uniform sampling with replacement, the estimation-error second moment accumulates the conditional variances of the sampled gradient-difference increments. For a batch of size b, its increment variance is at most (bn)^{-1} times the sum of squared component gradient differences. These are the cited Pham et al. Lemma 2 and standard sample-average variance identities underlying B0218 and B0088."]},{"id":"S081","semantic_type":"external_fact","kind":"external_fact","title":"对称矩阵谱事实","source_units":[],"declared_external":true,"declaration_text":["For a real symmetric matrix A, its operator norm is the largest absolute eigenvalue and λmax(A)=max_{x≠0}(xᵀAx)/(xᵀx). For B positive semidefinite, ||B||=λmax(B)."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S006","target_id":"S002","kind":"supports"},{"source_id":"S007","target_id":"S002","kind":"supports"},{"source_id":"S008","target_id":"S002","kind":"supports"},{"source_id":"S009","target_id":"S002","kind":"supports"},{"source_id":"S010","target_id":"S002","kind":"supports"},{"source_id":"S011","target_id":"S002","kind":"supports"},{"source_id":"S012","target_id":"S002","kind":"supports"},{"source_id":"S013","target_id":"S002","kind":"supports"},{"source_id":"S014","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S005","target_id":"S008","kind":"supports"},{"source_id":"S006","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S015","target_id":"S035","kind":"supports"},{"source_id":"S016","target_id":"S061","kind":"supports"},{"source_id":"S016","target_id":"S045","kind":"supports"},{"source_id":"S016","target_id":"S046","kind":"supports"},{"source_id":"S016","target_id":"S035","kind":"supports"},{"source_id":"S017","target_id":"S013","kind":"supports"},{"source_id":"S018","target_id":"S051","kind":"supports"},{"source_id":"S018","target_id":"S074","kind":"supports"},{"source_id":"S019","target_id":"S051","kind":"supports"},{"source_id":"S019","target_id":"S052","kind":"supports"},{"source_id":"S019","target_id":"S053","kind":"supports"},{"source_id":"S019","target_id":"S062","kind":"supports"},{"source_id":"S019","target_id":"S063","kind":"supports"},{"source_id":"S019","target_id":"S047","kind":"supports"},{"source_id":"S019","target_id":"S048","kind":"supports"},{"source_id":"S019","target_id":"S036","kind":"supports"},{"source_id":"S020","target_id":"S053","kind":"supports"},{"source_id":"S020","target_id":"S041","kind":"supports"},{"source_id":"S020","target_id":"S042","kind":"supports"},{"source_id":"S020","target_id":"S036","kind":"supports"},{"source_id":"S020","target_id":"S031","kind":"supports"},{"source_id":"S021","target_id":"S003","kind":"supports"},{"source_id":"S021","target_id":"S066","kind":"supports"},{"source_id":"S021","target_id":"S072","kind":"supports"},{"source_id":"S022","target_id":"S035","kind":"supports"},{"source_id":"S022","target_id":"S036","kind":"supports"},{"source_id":"S022","target_id":"S041","kind":"supports"},{"source_id":"S022","target_id":"S042","kind":"supports"},{"source_id":"S023","target_id":"S003","kind":"supports"},{"source_id":"S023","target_id":"S014","kind":"supports"},{"source_id":"S023","target_id":"S035","kind":"supports"},{"source_id":"S023","target_id":"S040","kind":"supports"},{"source_id":"S023","target_id":"S045","kind":"supports"},{"source_id":"S023","target_id":"S067","kind":"supports"},{"source_id":"S023","target_id":"S073","kind":"supports"},{"source_id":"S024","target_id":"S045","kind":"supports"},{"source_id":"S024","target_id":"S046","kind":"supports"},{"source_id":"S024","target_id":"S035","kind":"supports"},{"source_id":"S025","target_id":"S045","kind":"supports"},{"source_id":"S025","target_id":"S072","kind":"supports"},{"source_id":"S025","target_id":"S013","kind":"supports"},{"source_id":"S026","target_id":"S023","kind":"supports"},{"source_id":"S026","target_id":"S045","kind":"supports"},{"source_id":"S026","target_id":"S051","kind":"supports"},{"source_id":"S026","target_id":"S056","kind":"supports"},{"source_id":"S026","target_id":"S067","kind":"supports"},{"source_id":"S027","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S032","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S028","target_id":"S003","kind":"supports"},{"source_id":"S028","target_id":"S010","kind":"supports"},{"source_id":"S028","target_id":"S011","kind":"supports"},{"source_id":"S028","target_id":"S066","kind":"supports"},{"source_id":"S028","target_id":"S072","kind":"supports"},{"source_id":"S033","target_id":"S030","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S034","kind":"supports"},{"source_id":"S037","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S038","target_id":"S030","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S041","target_id":"S039","kind":"supports"},{"source_id":"S042","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S043","target_id":"S037","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S049","target_id":"S041","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S051","target_id":"S053","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S054","target_id":"S051","kind":"supports"},{"source_id":"S056","target_id":"S055","kind":"supports"},{"source_id":"S057","target_id":"S055","kind":"supports"},{"source_id":"S058","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S054","kind":"supports"},{"source_id":"S056","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S059","target_id":"S047","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S060","kind":"supports"},{"source_id":"S063","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S063","kind":"supports"},{"source_id":"S064","target_id":"S008","kind":"supports"},{"source_id":"S064","target_id":"S009","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"},{"source_id":"S067","target_id":"S065","kind":"supports"},{"source_id":"S068","target_id":"S065","kind":"supports"},{"source_id":"S069","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S069","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S067","target_id":"S005","kind":"supports"},{"source_id":"S068","target_id":"S005","kind":"supports"},{"source_id":"S070","target_id":"S008","kind":"supports"},{"source_id":"S072","target_id":"S071","kind":"supports"},{"source_id":"S073","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S070","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S074","target_id":"S035","kind":"supports"},{"source_id":"S075","target_id":"S045","kind":"supports"},{"source_id":"S075","target_id":"S046","kind":"supports"},{"source_id":"S075","target_id":"S035","kind":"supports"},{"source_id":"S075","target_id":"S037","kind":"supports"},{"source_id":"S076","target_id":"S040","kind":"supports"},{"source_id":"S077","target_id":"S062","kind":"supports"},{"source_id":"S077","target_id":"S063","kind":"supports"},{"source_id":"S078","target_id":"S074","kind":"supports"},{"source_id":"S079","target_id":"S075","kind":"supports"},{"source_id":"S080","target_id":"S076","kind":"supports"},{"source_id":"S081","target_id":"S056","kind":"supports"},{"source_id":"S081","target_id":"S057","kind":"supports"},{"source_id":"S081","target_id":"S058","kind":"supports"}]}}
{"record_id":"2401.08665v3","paper":{"title":"Zeroth-order Gradient and Quasi-Newton Methods for Nonsmooth Nonconvex Stochastic Optimization","authors":["Luke Marrinan","Uday V. Shanbhag","Farzad Yousefian"],"arxiv_id":"2401.08665","version":"v3","journal":"SIAM Journal on Optimization","doi":"10.1137/23m1626724","abstract_url":"https://arxiv.org/abs/2401.08665v3","pdf_url":"https://arxiv.org/pdf/2401.08665v3"},"context":{"raw_tex":"\\begin{align}\\label{eqn:prob}\n\t\\min_{\\x \\, \\in \\, \\Xscr} & \\quad f(\\x) \\, \\triangleq \\, \\mathbb{E}\\left[\\, \\f(\\x,\\bxi)\\, \\right], \n\\end{align}\n\nwhere $f:\\mathbb{R}^n\\to \\mathbb{R}$ is a real-valued, nonsmooth, and nonconvex\nfunction, $\\Xscr \\subseteq \\mathbb{R}^{n} $ is a closed and convex set, and $\\bxi:\\Omega \\to \\mathbb{R}^d$ denotes\na random variable associated with the probability space $(\\Omega,\n\\mathcal{F},\\mathbb{P})$. {Our interest lies in the case when $f$ is\n\tLipschitz continuous and at any $\\x \\in \\mathbb{R}^n$,  $f(\\x)\\triangleq\n\t\\mathbb{E}[{\\f}(\\x,\\bxi)]$ and $\\bxi: \\Omega \\mapsto \\Xi$ is a $d$-dimensional\n\trandom variable with $\\Xi \\, \\subseteq \\, \\Real^d$. We denote a realization of\n\t$\\bxi$ and $\\f({\\x},\\bxi)$ by $\\xi$ and $\\f({\\x},\\xi)$, respectively}. Recall that\nfunction $f$ is $L_0$-Lipschitz continuous on the set $\\Xscr$ if there exists a\nscalar $L_0>0$ such that for all $\\x,\\y \\in \\Xscr$ we have $|f(\\x)-f(\\y)| \\leq\nL_0\\|\\x-\\y\\|$. Our goal lies in devising randomized zeroth-order\n\n\n\t\t\t\t\\smallskip {\\bf Notation.} We use $\\x$, $\\x^\\top$, and $\\|\\x\\|$ to denote a column vector,\n\t\t\t\tits transpose, and its Euclidean norm, respectively. We\n\t\t\t\tdefine $f^*\\triangleq {\\displaystyle \\inf_{\\x \\in \\Xscr}} f(\\x)$ and $f^*_\\eta\\triangleq\n\t\t\t\t{\\displaystyle \\inf_{\\x \\in \\Xscr}} f_\\eta(\\x)$, where $f_\\eta(\\x)$ denotes the smoothed\n\t\t\t\tapproximation of $f$. Given a continuous function, i.e., $f \\in C^{0}$,\n\t\t\t\twe write $f \\in C^{0,0}(\\Xscr)$ if $f$ is Lipschitz on \n\t\t\t\t$\\Xscr$ with parameter $L_0$. Given a continuously differentiable\n\t\t\t\tfunction, i.e., $f \\in C^{1}$, we write $f \\in C^{1,1}(\\Xscr)$ if $\\nabla f$ is\n\t\t\t\tLipschitz continuous on $\\Xscr$ with parameter $L_1$. We\n\t\t\t\twrite a.s. for ``almost surely” and {$\\mathbb{E}[h(\\x,\\bxi)]$} denotes the\n\t\t\t\texpectation {with respect to a random variable $\\bxi$}. %Given a scalar $u$, $[u]_+\\triangleq \\max\\{0,u\\}$. \n\t\t\t\t$\\Pi_{\\Xscr}[\\vv]$ denotes the Euclidean projection of $\\vv$ onto the closed convex set $\\Xscr$.\n\n\n\t\t\t\twe address nonsmoothness by considering a locally randomized smoothing technique. Given a function $h:\\mathbb{R}^n \\to \\mathbb{R}$ and a scalar $\\eta>0$, a spherically smoothed approximation of $h$, denoted by $h_\\eta$, is defined as \n\n\n\\begin{align}\\label{def-smooth}\n\t\t\t\t\th_\\eta (\\x) \\, \\triangleq \\, \\mathbb{E}_{\\uu \\in \\mathbb{B}}\\left[\\, h(\\x+\\eta \\uu)\\, \\right],\n\t\t\t\t\\end{align} where {$\\uu$ denotes a random variable uniformly distributed on a unit ball $\\mathbb{B}$, defined as  $\\mathbb{B}\\triangleq \\{u \\in \\mathbb{R}^n\\mid \\|u\\|\\leq 1\\}$, with realizations denoted by $u \\in \\mathbb{B}$}.\n\t\t\t\tWe now recall a concentration inequality and subsequently present properties for spherical smoothing, where\n\t\t\t\t$\\mathbb{S}\\triangleq \\{v \\in \\mathbb{R}^n \\mid\n\t\t\t\t\\|v\\|=1\\}$ is the surface of $\\mathbb{B}$.\n\n\nIn developing \\texttt{VRG-ZO}, we minimize an $L_0$-Lipschitz\nfunction on $\\Xscr_{\\eta_0}$, where  $\\Xscr_{\\eta_0} \\triangleq\n\\Xscr + \\eta_0 \\mathbb{B}$ represents the Minkowski sum of $\\Xscr$\nand $\\eta \\mathbb{B}$.  Further, we assume that the random\nvariable $\\f(\\x,\\bxi) - f(\\x)$ admits suitable bias and moment\nproperties.  We develop schemes for computing approximate\nstationary points of \\eqref{eqn:prob} by an iterative scheme.\n\n\n\\begin{definition}[{\\bf The residual mapping}]\\label{def:res_maps}\\em Given $\\beta>0$ and a smoothing parameter $\\eta>0$, for any $\\x \\in \\mathbb{R}^n$ and an arbitrary vector $\\tilde e \\in \\mathbb{R}^n$,\n\t\t\t\t\tlet the residual mappings $G_{\\eta,\\beta}(\\x)$ and $\\tilde{G}_{\\eta,\\beta}(\\x, {\\tilde e})$ be defined as $G_{\\eta,\\beta}(\\x)  \\triangleq \\beta {\\left(\\x - \\Pi_{\\Xscr}\\left[\\x - \\tfrac{1}{\\beta} \\nabla_x {f_{\\eta}(\\x)}\\right]\\right)}$ and $\\tilde{G}_{\\eta,\\beta}(\\x, {\\tilde e})  \\triangleq \\beta\\left( \\x - \\Pi_{\\Xscr}\\left[\\x - \\tfrac{1}{\\beta} ({\\nabla_x f_{\\eta}(\\x)}+\\tilde{e}) \\right]\\right)$, respectively.\n\t\t\t\t\t%\\begin{align}\n\t\t\t\t\t%    G_{\\eta,\\beta}(\\x) &\\triangleq \\beta {\\left(\\x - \\Pi_{\\Xscr}\\left[\\x - \\tfrac{1}{\\beta} \\nabla_x {f_{\\eta}(\\x)}\\right]\\right)} \\mbox{ and }\\\\\n\t\t\t\t\t%    \\hspace{-0.05in}\\tilde{G}_{\\eta,\\beta}(\\x, {\\tilde e}) &\\triangleq \\beta\\left( \\x - \\Pi_{\\Xscr}\\left[\\x - \\tfrac{1}{\\beta} ({\\nabla_x f_{\\eta}(\\x)}+\\tilde{e}) \\right]\\right). \\hspace{-0.1in} \\qed\n\t\t\t\t\t%\\end{align}\n\t\t\t\t\\end{definition}\n\n\\begin{algorithm}[hbt]\n\t\t\t\t\t\\caption{\\texttt{{VRG-ZO}}: Variance-reduced randomized zeroth-order {gradient} method}\\label{algorithm:zo_nonconvex}\n\t\t\t\t\t%\\begin{boxedminipage}{165mm}\n\t\t\t\t\t{   \\begin{algorithmic}[1]\n\t\t\t\t\t\t\t\\STATE\\textbf{input:}  Given $\\x_0 \\in \\Xscr$, stepsize $\\gamma>0$, smoothing parameter $\\eta>0$, mini-batch sequence $\\{N_k\\}$, and an integer $R_{\\ell,K}$ randomly selected from $\\{\\ell\\triangleq\\lceil\\lambda K\\rceil ,\\ldots,K\\}$ using a discrete uniform distribution where $\\lambda \\in (0,1)$ and $K\\geq 1$.\n\t\t\t\t\t\t\t\\FOR {$k=0,1,\\ldots,{K}-1$}\n\t\t\t\t\t\t\t\\STATE   Generate a random {mini-}{batch $v_{j,k} \\in \\eta \\mathbb{S}$ for $j=1,\\ldots,N_k$}  \n\t\t\t\t\t\t\t% \\STATE Generate a random {batch $\\xi_{j,k}$ for $j=1,\\ldots,N_k$}  \n\t\t\t\t\t\t\t\\STATE  Compute $\\{g_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k})\\}_{j=1}^{N_k}$, where for all $j=1,\\ldots,N_k$,\\\\ $g_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k}) :=\\left(\\tfrac{n}{\\us{2}\\eta}\\right)\\left({\\f}(\\x_k+ v_{j,k},\\xi_{j,k}) - {\\f}(\\x_k- v_{j,k},\\xi_{j,k})\\right)\\tfrac{v_{j,k}}{\\left\\|v_{j,k}\\right\\|}$ \n\t\t\t\t\t\t\t\\STATE   {Compute} mini-batch inexact {ZO} gradient. $\n\t\t\t\t\t\t\tg_{\\eta,N_k}(\\x_k) := \\tfrac{\\sum_{j=1}^{N_k}  g_{\\eta}\\left(\\x_k,v_{j,k},\\xi_{j,k}\\right)}{N_k}$\n\t\t\t\t\t\t\t\\STATE   {Update $\\x_k$ as follows.}\n\t\t\t\t\t\t\t$\\x_{k+1}:= \\Pi_{\\Xscr}\\left[ \\x_{k}-\\gamma g_{\\eta,N_k}(\\x_k) \\right]$\n\t\t\t\t\t\t\t\\ENDFOR\n\t\t\t\t\t\t\t\\STATE Return $\\x_{R_{\\ell,K}}$ \n\t\t\t\t\t\\end{algorithmic}}\n\t\t\t\t\t%\\end{boxedminipage}\n\t\t\t\t\t% \\vskip17.5pt\n\t\t\t\t\\end{algorithm}\n\n\\begin{assumption}[{\\bf Problem properties}] \n\t\t\\label{ass-1}\\em\n\t\t\\noindent Consider problem \\eqref{eqn:prob}.\n\t\t\n\t\t\\noindent (i) {$\\tilde{f}(\\bullet,\\bxi)$ is $\\tilde{L}_0(\\bxi)$}-Lipschitz on $\\Xscr + \\eta_0 \\mathbb{B}$ for some $\\eta_0 > 0$, a.s. for all $\\bxi$,  where $L_0\\triangleq \\sqrt{\\mathbb{E}[{\\tilde{L}_0}(\\bxi)^2]} <\\infty$.  \n\t\t\\noindent (ii) $\\Xscr \\subseteq \\Real^n$ is a nonempty, closed, and convex set.\n\t\t\\noindent (iii) For any $\\x \\in \\Xscr+ \\eta_0 \\us{\\mathbb{B}}$, $\\mathbb{E}[\\, {\\f(\\x,\\bxi)} \\, \\mid \\,  \\x\\, ] \\, =\\, f(\\x)$.\n\t\t%\\noindent (iv) For all $\\x \\, \\in \\, \\Xscr+ \\eta_0 \\mathbb{S}$, $\\mathbb{E}[\\, |\\, \\f(\\x,{\\bxi})- f(\\x) \\, |^2 \\, \\mid \\,  \\x\\, ] \\, \\leq \\, \\nu^2$ for some $\\nu > 0$.\n\\end{assumption}\n\n\t\t\t\tWe introduce a variance-reduced randomized zeroth-order scheme presented by\n\t\t\t\tAlgorithm \\ref{algorithm:zo_nonconvex} and define a zeroth-order gradient\n\t\t\t\testimate of  $\\f(\\x_k, \\xi_{j,k})$ as \n\n\n\\begin{align*}\n\t\t\t\t\tg_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k}) \\triangleq \\left(\\tfrac{n}{\\eta}\\right)\\left({\\f}(\\x_k+ v_{j,k},\\xi_{j,k}) - {\\f}({\\x_k-v_{j,k}},\\xi_{j,k})\\right)\\tfrac{v_{j,k}}{\\left\\|v_{j,k}\\right\\|},\n\t\t\t\t\\end{align*}\n\n\t\t\t\twhere $v_{j,k} \\in \\eta\\mathbb{S}$.  {Intuitively, $g_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k})$ generates a gradient estimate by employing sampled function evaluations $\\tilde{f}(\\x_k{+v_{j,k}},\\xi_{j,k})$ and $\\tilde{f}(\\x_k{-v_{j,k}},\\xi_{j,k})$; in short, the zeroth-order oracle, given an $\\x_k$ and a perturbation vector $v_{j,k}$, produces two evaluations, \\fy{i.e.,}   \n\t\t\t\t\t$\\tilde{f}(\\x_k +v_{j,k},\\xi_{j,k})$ and $\\tilde{f}(\\x_k-v_{j,k},\\xi_{j,k})$.}\n\t\t\t\t%\\lm{We then employ a mini-batch approximation of this gradient estimate.} \n\t\t\t\tWe formally define the stochastic errors emergent from the randomized scheme based on a mini-batch approximation.\n\n\n\\begin{definition}\\label{def:stoch_errors} \\em\n\t\t\t\t\tFor all $k\\geq 0,j=1,\\ldots,N_k$, let \n\t\t\t\t\t$\\mathbf{e}_k\\triangleq \\tfrac{\\sum_{j=1}^{N_k}\\mathbf{e}_{j,k}}{N_k}$, %, $\\bm{\\theta}_k\\triangleq \\tfrac{\\sum_{j=1}^{N_k}\\bm{\\theta}_{j,k}}{N_k}$, \n\t\t\t\t\t\\begin{align}\n\t\t\t\t\t\te_{j,k} & \\triangleq {g_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k}) -\\nabla f_\\eta(\\x_k)}, \\ \\mbox{and } {\\mathbf{e}_{j,k}  \\triangleq g_{\\eta}(\\x_k,\\vv_{j,k},\\bxi_{j,k})}-\\nabla f_\\eta(\\x_k)\\label{def:stoch_errors1}. \\mbox{$\\ \\Box$}\n\t\t\t\t\t\t%\\\\ \\theta_{j,k } & \\triangleq g_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k}) -\\nabla {\\f}_\\eta(\\x_k,\\xi_{j,k}), \\mbox{ and } {\\bm{\\theta}_{j,k}  \\triangleq g_{\\eta}(\\x_k,\\vv,\\bxi_{\\lm{j}}) -\\nabla {\\f}_\\eta(\\x_k,\\bxi_{\\lm{j}})}.\\label{def:stoch_errors2}\n\t\t\t\t\t\t%&\\delta_{j,k} \\triangleq  g_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k}) -  g_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k}),\\label{def:stoch_errors3}\n\t\t\t\t\t\\end{align}\n\t\t\t\t\\end{definition}\n\n\t\t\t\tWe employ boldface  to denote random variables (such as $\\mathbf{e}_{j,k}$)  while their realizations are not emboldened (such as $e_{j,k}$).\n\t\t\t\tThe history of Algorithm \\ref{algorithm:zo_nonconvex} at iteration $k$ is denoted by $\\mathcal{F}_k$, defined as\n\n\n\\begin{align}\\label{eqn:history}\n\t\t\t\t\t\\mathcal{F}_0 \\triangleq \\{x_0\\},\\quad \n\t\t\t\t\t\\mathcal{F}_k \\triangleq \\mathcal{F}_{k-1} \\, \\bigcup \\, \\left\\{ \\, \\bigcup_{j=1}^{N_{k-1}}\\, \\{\\xi_{j,k-1}, v_{j,k-1}\\} \\, \\right\\}, \\qquad \\hbox{for } k \\geq 1.\n\t\t\t\t\\end{align}\n\n\\begin{assumption}[{\\bf Independence}]\\label{assum:random_vars}\\em\n\t\t\t\t\tRandom samples $\\xi_{j,k}$, $v_{j,k}$ are generated independent of each other for all $k\\geq 0$ and $1\\leq j \\leq N_k$.\n\t\t\t\t\\end{assumption}","units":[{"start":14959,"end":15101,"start_line":420,"end_line":422,"raw_tex":"\\begin{align}\\label{eqn:prob}\n\t\\min_{\\x \\, \\in \\, \\Xscr} & \\quad f(\\x) \\, \\triangleq \\, \\mathbb{E}\\left[\\, \\f(\\x,\\bxi)\\, \\right], \n\\end{align}","sha256_utf8":"12d57532dfdff5f60a53da5bacfe00cde1520f47de2c074b06c3e4502da07c09"},{"start":15102,"end":15974,"start_line":423,"end_line":433,"raw_tex":"where $f:\\mathbb{R}^n\\to \\mathbb{R}$ is a real-valued, nonsmooth, and nonconvex\nfunction, $\\Xscr \\subseteq \\mathbb{R}^{n} $ is a closed and convex set, and $\\bxi:\\Omega \\to \\mathbb{R}^d$ denotes\na random variable associated with the probability space $(\\Omega,\n\\mathcal{F},\\mathbb{P})$. {Our interest lies in the case when $f$ is\n\tLipschitz continuous and at any $\\x \\in \\mathbb{R}^n$,  $f(\\x)\\triangleq\n\t\\mathbb{E}[{\\f}(\\x,\\bxi)]$ and $\\bxi: \\Omega \\mapsto \\Xi$ is a $d$-dimensional\n\trandom variable with $\\Xi \\, \\subseteq \\, \\Real^d$. We denote a realization of\n\t$\\bxi$ and $\\f({\\x},\\bxi)$ by $\\xi$ and $\\f({\\x},\\xi)$, respectively}. Recall that\nfunction $f$ is $L_0$-Lipschitz continuous on the set $\\Xscr$ if there exists a\nscalar $L_0>0$ such that for all $\\x,\\y \\in \\Xscr$ we have $|f(\\x)-f(\\y)| \\leq\nL_0\\|\\x-\\y\\|$. Our goal lies in devising randomized zeroth-order\n","sha256_utf8":"fd24de3719ee02c8c7f3642c01a91f885388c542d668c7af099694a70b8ca703"},{"start":37269,"end":38254,"start_line":675,"end_line":686,"raw_tex":"\t\t\t\t\\smallskip {\\bf Notation.} We use $\\x$, $\\x^\\top$, and $\\|\\x\\|$ to denote a column vector,\n\t\t\t\tits transpose, and its Euclidean norm, respectively. We\n\t\t\t\tdefine $f^*\\triangleq {\\displaystyle \\inf_{\\x \\in \\Xscr}} f(\\x)$ and $f^*_\\eta\\triangleq\n\t\t\t\t{\\displaystyle \\inf_{\\x \\in \\Xscr}} f_\\eta(\\x)$, where $f_\\eta(\\x)$ denotes the smoothed\n\t\t\t\tapproximation of $f$. Given a continuous function, i.e., $f \\in C^{0}$,\n\t\t\t\twe write $f \\in C^{0,0}(\\Xscr)$ if $f$ is Lipschitz on \n\t\t\t\t$\\Xscr$ with parameter $L_0$. Given a continuously differentiable\n\t\t\t\tfunction, i.e., $f \\in C^{1}$, we write $f \\in C^{1,1}(\\Xscr)$ if $\\nabla f$ is\n\t\t\t\tLipschitz continuous on $\\Xscr$ with parameter $L_1$. We\n\t\t\t\twrite a.s. for ``almost surely” and {$\\mathbb{E}[h(\\x,\\bxi)]$} denotes the\n\t\t\t\texpectation {with respect to a random variable $\\bxi$}. %Given a scalar $u$, $[u]_+\\triangleq \\max\\{0,u\\}$. \n\t\t\t\t$\\Pi_{\\Xscr}[\\vv]$ denotes the Euclidean projection of $\\vv$ onto the closed convex set $\\Xscr$.\n","sha256_utf8":"ad12195a30f0bd1f8406bbc45dee78d4ac2b2edf9a42a7c46bf18c6085781949"},{"start":41901,"end":42140,"start_line":730,"end_line":730,"raw_tex":"\t\t\t\twe address nonsmoothness by considering a locally randomized smoothing technique. Given a function $h:\\mathbb{R}^n \\to \\mathbb{R}$ and a scalar $\\eta>0$, a spherically smoothed approximation of $h$, denoted by $h_\\eta$, is defined as \n","sha256_utf8":"1d5c202cbe71b1754cfa5f158d3731309c8cd84ccc267722838da81f8b0cb0a5"},{"start":42144,"end":42718,"start_line":731,"end_line":736,"raw_tex":"\\begin{align}\\label{def-smooth}\n\t\t\t\t\th_\\eta (\\x) \\, \\triangleq \\, \\mathbb{E}_{\\uu \\in \\mathbb{B}}\\left[\\, h(\\x+\\eta \\uu)\\, \\right],\n\t\t\t\t\\end{align} where {$\\uu$ denotes a random variable uniformly distributed on a unit ball $\\mathbb{B}$, defined as  $\\mathbb{B}\\triangleq \\{u \\in \\mathbb{R}^n\\mid \\|u\\|\\leq 1\\}$, with realizations denoted by $u \\in \\mathbb{B}$}.\n\t\t\t\tWe now recall a concentration inequality and subsequently present properties for spherical smoothing, where\n\t\t\t\t$\\mathbb{S}\\triangleq \\{v \\in \\mathbb{R}^n \\mid\n\t\t\t\t\\|v\\|=1\\}$ is the surface of $\\mathbb{B}$.\n","sha256_utf8":"9317b218e2efd60b000af8e0b3c502fd0f08d88f4e1e849289ae4510cc1d94c4"},{"start":50503,"end":50938,"start_line":846,"end_line":852,"raw_tex":"In developing \\texttt{VRG-ZO}, we minimize an $L_0$-Lipschitz\nfunction on $\\Xscr_{\\eta_0}$, where  $\\Xscr_{\\eta_0} \\triangleq\n\\Xscr + \\eta_0 \\mathbb{B}$ represents the Minkowski sum of $\\Xscr$\nand $\\eta \\mathbb{B}$.  Further, we assume that the random\nvariable $\\f(\\x,\\bxi) - f(\\x)$ admits suitable bias and moment\nproperties.  We develop schemes for computing approximate\nstationary points of \\eqref{eqn:prob} by an iterative scheme.\n","sha256_utf8":"2f684e639853ed60d10a70a8d1064c044d1ae831b8694101cccbc43e5b8fdff1"},{"start":53534,"end":54574,"start_line":877,"end_line":883,"raw_tex":"\\begin{definition}[{\\bf The residual mapping}]\\label{def:res_maps}\\em Given $\\beta>0$ and a smoothing parameter $\\eta>0$, for any $\\x \\in \\mathbb{R}^n$ and an arbitrary vector $\\tilde e \\in \\mathbb{R}^n$,\n\t\t\t\t\tlet the residual mappings $G_{\\eta,\\beta}(\\x)$ and $\\tilde{G}_{\\eta,\\beta}(\\x, {\\tilde e})$ be defined as $G_{\\eta,\\beta}(\\x)  \\triangleq \\beta {\\left(\\x - \\Pi_{\\Xscr}\\left[\\x - \\tfrac{1}{\\beta} \\nabla_x {f_{\\eta}(\\x)}\\right]\\right)}$ and $\\tilde{G}_{\\eta,\\beta}(\\x, {\\tilde e})  \\triangleq \\beta\\left( \\x - \\Pi_{\\Xscr}\\left[\\x - \\tfrac{1}{\\beta} ({\\nabla_x f_{\\eta}(\\x)}+\\tilde{e}) \\right]\\right)$, respectively.\n\t\t\t\t\t%\\begin{align}\n\t\t\t\t\t%    G_{\\eta,\\beta}(\\x) &\\triangleq \\beta {\\left(\\x - \\Pi_{\\Xscr}\\left[\\x - \\tfrac{1}{\\beta} \\nabla_x {f_{\\eta}(\\x)}\\right]\\right)} \\mbox{ and }\\\\\n\t\t\t\t\t%    \\hspace{-0.05in}\\tilde{G}_{\\eta,\\beta}(\\x, {\\tilde e}) &\\triangleq \\beta\\left( \\x - \\Pi_{\\Xscr}\\left[\\x - \\tfrac{1}{\\beta} ({\\nabla_x f_{\\eta}(\\x)}+\\tilde{e}) \\right]\\right). \\hspace{-0.1in} \\qed\n\t\t\t\t\t%\\end{align}\n\t\t\t\t\\end{definition}","sha256_utf8":"eb85a9654c484c5cc82ee47982f9c4299f7538e55e9838e335855d2e1528e999"},{"start":56164,"end":57638,"start_line":905,"end_line":923,"raw_tex":"\\begin{algorithm}[hbt]\n\t\t\t\t\t\\caption{\\texttt{{VRG-ZO}}: Variance-reduced randomized zeroth-order {gradient} method}\\label{algorithm:zo_nonconvex}\n\t\t\t\t\t%\\begin{boxedminipage}{165mm}\n\t\t\t\t\t{   \\begin{algorithmic}[1]\n\t\t\t\t\t\t\t\\STATE\\textbf{input:}  Given $\\x_0 \\in \\Xscr$, stepsize $\\gamma>0$, smoothing parameter $\\eta>0$, mini-batch sequence $\\{N_k\\}$, and an integer $R_{\\ell,K}$ randomly selected from $\\{\\ell\\triangleq\\lceil\\lambda K\\rceil ,\\ldots,K\\}$ using a discrete uniform distribution where $\\lambda \\in (0,1)$ and $K\\geq 1$.\n\t\t\t\t\t\t\t\\FOR {$k=0,1,\\ldots,{K}-1$}\n\t\t\t\t\t\t\t\\STATE   Generate a random {mini-}{batch $v_{j,k} \\in \\eta \\mathbb{S}$ for $j=1,\\ldots,N_k$}  \n\t\t\t\t\t\t\t% \\STATE Generate a random {batch $\\xi_{j,k}$ for $j=1,\\ldots,N_k$}  \n\t\t\t\t\t\t\t\\STATE  Compute $\\{g_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k})\\}_{j=1}^{N_k}$, where for all $j=1,\\ldots,N_k$,\\\\ $g_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k}) :=\\left(\\tfrac{n}{\\us{2}\\eta}\\right)\\left({\\f}(\\x_k+ v_{j,k},\\xi_{j,k}) - {\\f}(\\x_k- v_{j,k},\\xi_{j,k})\\right)\\tfrac{v_{j,k}}{\\left\\|v_{j,k}\\right\\|}$ \n\t\t\t\t\t\t\t\\STATE   {Compute} mini-batch inexact {ZO} gradient. $\n\t\t\t\t\t\t\tg_{\\eta,N_k}(\\x_k) := \\tfrac{\\sum_{j=1}^{N_k}  g_{\\eta}\\left(\\x_k,v_{j,k},\\xi_{j,k}\\right)}{N_k}$\n\t\t\t\t\t\t\t\\STATE   {Update $\\x_k$ as follows.}\n\t\t\t\t\t\t\t$\\x_{k+1}:= \\Pi_{\\Xscr}\\left[ \\x_{k}-\\gamma g_{\\eta,N_k}(\\x_k) \\right]$\n\t\t\t\t\t\t\t\\ENDFOR\n\t\t\t\t\t\t\t\\STATE Return $\\x_{R_{\\ell,K}}$ \n\t\t\t\t\t\\end{algorithmic}}\n\t\t\t\t\t%\\end{boxedminipage}\n\t\t\t\t\t% \\vskip17.5pt\n\t\t\t\t\\end{algorithm}","sha256_utf8":"6b5541c3ff38f352fb69571f1c8d2017690c293fbbc96fb6e223ff1783e37e8f"},{"start":57909,"end":58649,"start_line":928,"end_line":936,"raw_tex":"\\begin{assumption}[{\\bf Problem properties}] \n\t\t\\label{ass-1}\\em\n\t\t\\noindent Consider problem \\eqref{eqn:prob}.\n\t\t\n\t\t\\noindent (i) {$\\tilde{f}(\\bullet,\\bxi)$ is $\\tilde{L}_0(\\bxi)$}-Lipschitz on $\\Xscr + \\eta_0 \\mathbb{B}$ for some $\\eta_0 > 0$, a.s. for all $\\bxi$,  where $L_0\\triangleq \\sqrt{\\mathbb{E}[{\\tilde{L}_0}(\\bxi)^2]} <\\infty$.  \n\t\t\\noindent (ii) $\\Xscr \\subseteq \\Real^n$ is a nonempty, closed, and convex set.\n\t\t\\noindent (iii) For any $\\x \\in \\Xscr+ \\eta_0 \\us{\\mathbb{B}}$, $\\mathbb{E}[\\, {\\f(\\x,\\bxi)} \\, \\mid \\,  \\x\\, ] \\, =\\, f(\\x)$.\n\t\t%\\noindent (iv) For all $\\x \\, \\in \\, \\Xscr+ \\eta_0 \\mathbb{S}$, $\\mathbb{E}[\\, |\\, \\f(\\x,{\\bxi})- f(\\x) \\, |^2 \\, \\mid \\,  \\x\\, ] \\, \\leq \\, \\nu^2$ for some $\\nu > 0$.\n\\end{assumption}","sha256_utf8":"a65c0758ad38f90b2fa9329a763fb2071022fec3dd8e129833516d9a55095ba0"},{"start":58650,"end":58851,"start_line":937,"end_line":939,"raw_tex":"\t\t\t\tWe introduce a variance-reduced randomized zeroth-order scheme presented by\n\t\t\t\tAlgorithm \\ref{algorithm:zo_nonconvex} and define a zeroth-order gradient\n\t\t\t\testimate of  $\\f(\\x_k, \\xi_{j,k})$ as \n","sha256_utf8":"cead9ee3de03f3f767989c6f745567e5e5a821f5abc22526c4dd74443bbf10d8"},{"start":59195,"end":59419,"start_line":945,"end_line":947,"raw_tex":"\\begin{align*}\n\t\t\t\t\tg_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k}) \\triangleq \\left(\\tfrac{n}{\\eta}\\right)\\left({\\f}(\\x_k+ v_{j,k},\\xi_{j,k}) - {\\f}({\\x_k-v_{j,k}},\\xi_{j,k})\\right)\\tfrac{v_{j,k}}{\\left\\|v_{j,k}\\right\\|},\n\t\t\t\t\\end{align*}","sha256_utf8":"4a7bc714c30cd8dd8ece5bf720d70da2817bd103f1bc3b91ebedc10ec29c1216"},{"start":59420,"end":60073,"start_line":948,"end_line":951,"raw_tex":"\t\t\t\twhere $v_{j,k} \\in \\eta\\mathbb{S}$.  {Intuitively, $g_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k})$ generates a gradient estimate by employing sampled function evaluations $\\tilde{f}(\\x_k{+v_{j,k}},\\xi_{j,k})$ and $\\tilde{f}(\\x_k{-v_{j,k}},\\xi_{j,k})$; in short, the zeroth-order oracle, given an $\\x_k$ and a perturbation vector $v_{j,k}$, produces two evaluations, \\fy{i.e.,}   \n\t\t\t\t\t$\\tilde{f}(\\x_k +v_{j,k},\\xi_{j,k})$ and $\\tilde{f}(\\x_k-v_{j,k},\\xi_{j,k})$.}\n\t\t\t\t%\\lm{We then employ a mini-batch approximation of this gradient estimate.} \n\t\t\t\tWe formally define the stochastic errors emergent from the randomized scheme based on a mini-batch approximation.\n","sha256_utf8":"36c51c3327c13e6237dc7bc96d8aae93e3553cef6bea555cfa7fe094ff4f38ba"},{"start":60077,"end":60981,"start_line":952,"end_line":960,"raw_tex":"\\begin{definition}\\label{def:stoch_errors} \\em\n\t\t\t\t\tFor all $k\\geq 0,j=1,\\ldots,N_k$, let \n\t\t\t\t\t$\\mathbf{e}_k\\triangleq \\tfrac{\\sum_{j=1}^{N_k}\\mathbf{e}_{j,k}}{N_k}$, %, $\\bm{\\theta}_k\\triangleq \\tfrac{\\sum_{j=1}^{N_k}\\bm{\\theta}_{j,k}}{N_k}$, \n\t\t\t\t\t\\begin{align}\n\t\t\t\t\t\te_{j,k} & \\triangleq {g_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k}) -\\nabla f_\\eta(\\x_k)}, \\ \\mbox{and } {\\mathbf{e}_{j,k}  \\triangleq g_{\\eta}(\\x_k,\\vv_{j,k},\\bxi_{j,k})}-\\nabla f_\\eta(\\x_k)\\label{def:stoch_errors1}. \\mbox{$\\ \\Box$}\n\t\t\t\t\t\t%\\\\ \\theta_{j,k } & \\triangleq g_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k}) -\\nabla {\\f}_\\eta(\\x_k,\\xi_{j,k}), \\mbox{ and } {\\bm{\\theta}_{j,k}  \\triangleq g_{\\eta}(\\x_k,\\vv,\\bxi_{\\lm{j}}) -\\nabla {\\f}_\\eta(\\x_k,\\bxi_{\\lm{j}})}.\\label{def:stoch_errors2}\n\t\t\t\t\t\t%&\\delta_{j,k} \\triangleq  g_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k}) -  g_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k}),\\label{def:stoch_errors3}\n\t\t\t\t\t\\end{align}\n\t\t\t\t\\end{definition}","sha256_utf8":"06c89c1a3b4176adf9862f0ce160bb11f887ca6f69178e64e502661df5834546"},{"start":60982,"end":61245,"start_line":961,"end_line":962,"raw_tex":"\t\t\t\tWe employ boldface  to denote random variables (such as $\\mathbf{e}_{j,k}$)  while their realizations are not emboldened (such as $e_{j,k}$).\n\t\t\t\tThe history of Algorithm \\ref{algorithm:zo_nonconvex} at iteration $k$ is denoted by $\\mathcal{F}_k$, defined as\n","sha256_utf8":"548d40a416e17d9a21b1740d7b2137f83c77bbbd8ca83710b535074dc9a6aa30"},{"start":61249,"end":61510,"start_line":963,"end_line":966,"raw_tex":"\\begin{align}\\label{eqn:history}\n\t\t\t\t\t\\mathcal{F}_0 \\triangleq \\{x_0\\},\\quad \n\t\t\t\t\t\\mathcal{F}_k \\triangleq \\mathcal{F}_{k-1} \\, \\bigcup \\, \\left\\{ \\, \\bigcup_{j=1}^{N_{k-1}}\\, \\{\\xi_{j,k-1}, v_{j,k-1}\\} \\, \\right\\}, \\qquad \\hbox{for } k \\geq 1.\n\t\t\t\t\\end{align}","sha256_utf8":"3f2433c87b29955398c6a17b7d88aaace104ef77db40a2b84962e6de47f21cd6"},{"start":61612,"end":61824,"start_line":968,"end_line":970,"raw_tex":"\\begin{assumption}[{\\bf Independence}]\\label{assum:random_vars}\\em\n\t\t\t\t\tRandom samples $\\xi_{j,k}$, $v_{j,k}$ are generated independent of each other for all $k\\geq 0$ and $1\\leq j \\leq N_k$.\n\t\t\t\t\\end{assumption}","sha256_utf8":"6f1924b4f71565648f49b02ce68bc7472bc0e0c0e75c2e14ef7361ad5634ce28"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}[{\\bf Rate and complexity statements for \\texttt{VRG-ZO}}]\\label{thm:vr-rb-zo}%Alg.~\\ref{algorithm:zo_nonconvex}\n\t\t\t\t\t\\em\n\t\t\t\t\tConsider Algorithm \\ref{algorithm:zo_nonconvex}. Let Assumptions~\\ref{ass-1} and \\ref{assum:random_vars} hold.  Let $a > 0$ be an arbitrary scalar.\n\t\t\t\t\t\n\t\t\t\t\t{\\noindent {\\bf  (a)  [Constant step]}} Let $\\gamma_k:=\\gamma<{\\frac{\\eta}{\\sqrt{n} L_0}}$ and $N_k:=  \\lceil  a \\sqrt{n}L_0 (k+1) \\rceil$ for $k \\geq 0$.  %\\farz{{\\bf TODO:} $a$ cannot be zero, because we use $a^{-1}$ in the proof. I changed $a \\geq 0$ to $a >0$. With this change, the addition of 1 in $N_k:= n L_0 \\lceil 1+a(k+1) \\rceil$ seems unnecessary because $\\lceil a(k+1) \\rceil \\geq 1$ anyway. So I removed 1 in $N_k:= n L_0 \\lceil 1+a(k+1) \\rceil$. I also moved $L_0$ inside the ceiling term because it could be non-integer. Please check.}\n\t\t\t\t\t\n\t\t\t\t\t\\noindent {\\bf{(a-i)}} For $K> \\tfrac{2}{1-\\lambda}$ with $\\ell\\triangleq  \\lceil \\lambda K\\rceil$, we have\n\t\t\t\t\t\\begin{align}\\label{eqn:GBound_constant_gamma}\n\t\t\t\t\t\t\\mathbb{E}\\left[ \\|G_{\\eta,1/\\gamma} (\\x_{R_{\\ell,K}})\\|^2\\right] \\leq %\\left(\\left( 1-\\tfrac{\\lm{\\sqrt{n}}L_0\\gamma_{\\lm{0}}}{\\eta}\\right) \\tfrac{\\gamma}{4}(1-\\lambda)K \\right)^{-1}\\\\\n\t\t\t\t\t\t\\tfrac{\\left(\\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left( {16 \\sqrt{n}} \\sqrt{2\\pi} L_0 \\right)(0.5-\\ln(\\lambda)) {{a^{-1}}}\\gamma\\right)}{\\left(\\left( 1-\\tfrac{{\\sqrt{n}}L_0\\gamma}{\\eta}\\right) \\tfrac{\\gamma}{4}(1-\\lambda)K \\right)}.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\t\\noindent {\\bf{(a-ii)}} Let $\\gamma= {\\tfrac{\\eta}{2{\\sqrt{n}}L_0}}$,  $\\epsilon>0$, and $K_{\\epsilon}$ satisfy $ \\mathbb{E}\\left[ \\|G_{\\eta,1/\\gamma} (\\x_{R_{\\ell,K_{\\epsilon}}})\\|\\right]   \\leq \\epsilon$.  Then the following hold.  {{(a-ii-1)}} The total no. of projection steps is $K_{\\epsilon}=\\mathcal{O}\\left({n^{1/2}}{(L_0\\eta^{-1} +L_0^2)} \\epsilon^{-2}\\right)$.%\\farz{{\\bf TODO:} I don't think $\\eta^{-1}$ will appear in the complexity result here. $\\gamma$ seems to cancel out from denominator in $\\tfrac{\\gamma}{4}(K-\\ell)$ and numerator in $\\gamma\\sum_{k=\\ell}^{K-1} \\tfrac{\\lm{1}}{N_k}$ and there won't remain any $\\eta^{-1}$. Please check. }\n\t\t\t\t\t\n\t\t\t\t\t\\noindent {{(a-ii-2)}} The overall sample complexity is $\\mathcal{O}\\left({n^{3/2}(L_0^3\\eta^{-2}+L_0^5)} \\epsilon^{-4}\\right)$.\n\t\t\t\t\t\n\t\t\t\t\t\\noindent {\\bf  (b)  [Diminishing step]} Let  $\\gamma_k = \\tfrac{\\gamma_0}{\\sqrt{k+1}}$ and {$N_k:=  {\\lceil a{\\sqrt{n}} L_0{\\sqrt{k+1}} \\rceil }$}, where {$\\gamma_0<{{\\eta}{{\\sqrt{n}} L_0}}$}. \n\t\t\t\t {\\bf{(b-i)}}  For $K> \\tfrac{2}{1-\\lambda}$ with $\\ell\\triangleq  \\lceil \\lambda K\\rceil$, we have\n\t\t\t\t\t\\begin{align}\\label{eqn:GBound_diminish_gamma}\n\t\t\t\t\t\t\\mathbb{E}\\left[ \\|G_{\\eta,1/\\gamma_0} (\\x_{R_{\\ell,K}})\\|^2\\right] \\leq \\tfrac{\\left(\\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+  \\left({16  \\sqrt{2\\pi} L_0\\sqrt{n}}\\right)  (0.5-\\ln(\\lambda))\\lm{a^{-1}}\\gamma_0\\right)}{\\left(\\left( 1-\\tfrac{{\\sqrt{n}} L_0\\gamma_0}{\\eta}\\right) {\\tfrac{\\gamma_0}{2}}(1-\\sqrt{\\lambda})\\sqrt{K}\\right)}.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\t\\noindent {\\bf{(b-ii)}}  Let $\\epsilon>0$ and $K_{\\epsilon}$ be such that $ \\mathbb{E}\\left[ \\|G_{\\eta,1/{\\gamma_0}} (\\x_{R_{\\ell,K_\\epsilon}})\\|\\right]   \\leq \\epsilon$. Then the following hold. \n\t\t\t\t\t {{(b-ii-1)}} The total number of projection steps is  $K_{\\epsilon}=\\mathcal{O}\\left({n(L_0^2\\eta^{-2} +L_0^4)} \\epsilon^{-4}\\right).$ {{(b-ii-2)}} The total sample complexity is $\\mathcal{O}\\left({n^{2}(L_0^4\\eta^{-3}+L_0^7)} \\epsilon^{-6}\\right)$.\n\t\t\t\t\\end{theorem}","units":[{"start":76138,"end":79631,"start_line":1142,"end_line":1164,"raw_tex":"\\begin{theorem}[{\\bf Rate and complexity statements for \\texttt{VRG-ZO}}]\\label{thm:vr-rb-zo}%Alg.~\\ref{algorithm:zo_nonconvex}\n\t\t\t\t\t\\em\n\t\t\t\t\tConsider Algorithm \\ref{algorithm:zo_nonconvex}. Let Assumptions~\\ref{ass-1} and \\ref{assum:random_vars} hold.  Let $a > 0$ be an arbitrary scalar.\n\t\t\t\t\t\n\t\t\t\t\t{\\noindent {\\bf  (a)  [Constant step]}} Let $\\gamma_k:=\\gamma<{\\frac{\\eta}{\\sqrt{n} L_0}}$ and $N_k:=  \\lceil  a \\sqrt{n}L_0 (k+1) \\rceil$ for $k \\geq 0$.  %\\farz{{\\bf TODO:} $a$ cannot be zero, because we use $a^{-1}$ in the proof. I changed $a \\geq 0$ to $a >0$. With this change, the addition of 1 in $N_k:= n L_0 \\lceil 1+a(k+1) \\rceil$ seems unnecessary because $\\lceil a(k+1) \\rceil \\geq 1$ anyway. So I removed 1 in $N_k:= n L_0 \\lceil 1+a(k+1) \\rceil$. I also moved $L_0$ inside the ceiling term because it could be non-integer. Please check.}\n\t\t\t\t\t\n\t\t\t\t\t\\noindent {\\bf{(a-i)}} For $K> \\tfrac{2}{1-\\lambda}$ with $\\ell\\triangleq  \\lceil \\lambda K\\rceil$, we have\n\t\t\t\t\t\\begin{align}\\label{eqn:GBound_constant_gamma}\n\t\t\t\t\t\t\\mathbb{E}\\left[ \\|G_{\\eta,1/\\gamma} (\\x_{R_{\\ell,K}})\\|^2\\right] \\leq %\\left(\\left( 1-\\tfrac{\\lm{\\sqrt{n}}L_0\\gamma_{\\lm{0}}}{\\eta}\\right) \\tfrac{\\gamma}{4}(1-\\lambda)K \\right)^{-1}\\\\\n\t\t\t\t\t\t\\tfrac{\\left(\\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left( {16 \\sqrt{n}} \\sqrt{2\\pi} L_0 \\right)(0.5-\\ln(\\lambda)) {{a^{-1}}}\\gamma\\right)}{\\left(\\left( 1-\\tfrac{{\\sqrt{n}}L_0\\gamma}{\\eta}\\right) \\tfrac{\\gamma}{4}(1-\\lambda)K \\right)}.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\t\\noindent {\\bf{(a-ii)}} Let $\\gamma= {\\tfrac{\\eta}{2{\\sqrt{n}}L_0}}$,  $\\epsilon>0$, and $K_{\\epsilon}$ satisfy $ \\mathbb{E}\\left[ \\|G_{\\eta,1/\\gamma} (\\x_{R_{\\ell,K_{\\epsilon}}})\\|\\right]   \\leq \\epsilon$.  Then the following hold.  {{(a-ii-1)}} The total no. of projection steps is $K_{\\epsilon}=\\mathcal{O}\\left({n^{1/2}}{(L_0\\eta^{-1} +L_0^2)} \\epsilon^{-2}\\right)$.%\\farz{{\\bf TODO:} I don't think $\\eta^{-1}$ will appear in the complexity result here. $\\gamma$ seems to cancel out from denominator in $\\tfrac{\\gamma}{4}(K-\\ell)$ and numerator in $\\gamma\\sum_{k=\\ell}^{K-1} \\tfrac{\\lm{1}}{N_k}$ and there won't remain any $\\eta^{-1}$. Please check. }\n\t\t\t\t\t\n\t\t\t\t\t\\noindent {{(a-ii-2)}} The overall sample complexity is $\\mathcal{O}\\left({n^{3/2}(L_0^3\\eta^{-2}+L_0^5)} \\epsilon^{-4}\\right)$.\n\t\t\t\t\t\n\t\t\t\t\t\\noindent {\\bf  (b)  [Diminishing step]} Let  $\\gamma_k = \\tfrac{\\gamma_0}{\\sqrt{k+1}}$ and {$N_k:=  {\\lceil a{\\sqrt{n}} L_0{\\sqrt{k+1}} \\rceil }$}, where {$\\gamma_0<{{\\eta}{{\\sqrt{n}} L_0}}$}. \n\t\t\t\t {\\bf{(b-i)}}  For $K> \\tfrac{2}{1-\\lambda}$ with $\\ell\\triangleq  \\lceil \\lambda K\\rceil$, we have\n\t\t\t\t\t\\begin{align}\\label{eqn:GBound_diminish_gamma}\n\t\t\t\t\t\t\\mathbb{E}\\left[ \\|G_{\\eta,1/\\gamma_0} (\\x_{R_{\\ell,K}})\\|^2\\right] \\leq \\tfrac{\\left(\\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+  \\left({16  \\sqrt{2\\pi} L_0\\sqrt{n}}\\right)  (0.5-\\ln(\\lambda))\\lm{a^{-1}}\\gamma_0\\right)}{\\left(\\left( 1-\\tfrac{{\\sqrt{n}} L_0\\gamma_0}{\\eta}\\right) {\\tfrac{\\gamma_0}{2}}(1-\\sqrt{\\lambda})\\sqrt{K}\\right)}.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\t\\noindent {\\bf{(b-ii)}}  Let $\\epsilon>0$ and $K_{\\epsilon}$ be such that $ \\mathbb{E}\\left[ \\|G_{\\eta,1/{\\gamma_0}} (\\x_{R_{\\ell,K_\\epsilon}})\\|\\right]   \\leq \\epsilon$. Then the following hold. \n\t\t\t\t\t {{(b-ii-1)}} The total number of projection steps is  $K_{\\epsilon}=\\mathcal{O}\\left({n(L_0^2\\eta^{-2} +L_0^4)} \\epsilon^{-4}\\right).$ {{(b-ii-2)}} The total sample complexity is $\\mathcal{O}\\left({n^{2}(L_0^4\\eta^{-3}+L_0^7)} \\epsilon^{-6}\\right)$.\n\t\t\t\t\\end{theorem}","sha256_utf8":"deccf645d4e26f524b1c362ead6c3d17f68fae4371a87168650c8693a3c58991"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}[L\\'{e}vy concentration on $ \\mathbb{S}^n $~{\\cite[Prop. 3.11 and Example 3.12]{Wainwright2019}}]\n\t\t\t\t\t\\label{lemma:levy-concentration}\\em\n\t\t\t\t\tLet ${h}: \\mathbb{R}^n \\to \\mathbb{R}$ be $L$-Lipschitz and $\\tilde{\\vv}$ be uniformly distributed on {$\\mathbb{S}$}. Then \n\t\t\t\t\t\\begin{align}\n\t\t\t\t\t\t\\mathbb{P}\\left[\\, \\left|\\, h(\\tilde{\\vv}) - \\mathbb{E}[h(\\tilde{\\vv})]\\, \\right|\\, \\geq \\, \\alpha \\, \\right] &\\, \\leq \\, 2 \\sqrt{2\\pi} e^{-\\frac{n \\alpha^2}{8L^2}}.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\\end{lemma}\n\n\\begin{lemma}[{\\bf Properties of spherical smoothing}] \\label{lemma:props_local_smoothing}\\em Suppose $h:\\mathbb{R}^n \\mapsto \\mathbb{R}$ is {a continuous function and its smoothed counterpart $h_{\\eta}$ is defined as \\eqref{def-smooth}, where $\\eta>0$ is a given scalar}. Then the following hold for any $\\x, \\y \\in {\\cal X}$.\n\t\t\t\t\t\n\\begin{enumerate}\n\t\t\t\t\t\t\\item[ (i)] $h_{\\eta}$ is $C^1$ over $\\Xscr$ and $\\nabla_{\\x} h_{\\eta}(\\x) = \\left(\\tfrac{n}{2\\eta}\\right) \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\left[\\left(h(\\x+\\vv)-h(\\x-\\vv)\\right) \\tfrac{\\vv}{\\|\\vv\\|}\\right]. $ \n\t\t\t\t\t\\end{enumerate}\n\t\t\t\t\t%    \\begin{align}\n\t\t\t\t\t\t%        \\nabla_{\\x} h_{\\eta}(\\x) = \\left(\\tfrac{n}{\\eta}\\right) \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\left[\\us{h(\\x+\\vv) \\tfrac{\\vv}{\\|\\vv\\|}}\\right] \\quad \\forall \\x \\in \\Xscr. \n\t\t\t\t\t\t%    \\end{align}\nSuppose $h \\in C^{0,0}(\\Xscr_{\\eta})$ with parameter $L_0$. For any $\\x, \\y \\in \\Xscr$,  the following hold. \n\t\t\t\t\t\n\t\t\t\t\t\\begin{enumerate}\n\t\t\t\t\t\t\\item[ (ii)] $| h_{\\eta}(\\x)-h_{\\eta}(\\y) | \\leq L_0 \\|\\x-\\y\\|.$  (iii) $| h_{\\eta}(\\x) - h(\\x)| \\leq L_0 \\eta.$ \n\t\t\t\t\t\t\n\t\t\t\t\t\t\\item[ (iv)] $\\| \\nabla_{\\x} h_{\\eta}(\\x) -\\nabla_{\\x} h_{\\eta}(\\y)\\| \\leq {\n\t\t\t\t\t\t\t\\tfrac{L_0 \\sqrt{n}}{\\eta}}\\|\\x-\\y\\|.$\n\t\t\t\t\t\t%    \\item[(v)] If $h$ is convex {and $h \\in C^{0,0}(\\Xscr_{\\eta})$ with parameter $L_0$}, then $h_{\\eta}$ is convex and satisfies the following for any $\\x \\in \\Xscr$. \n\t\t\t\t\t\t%        \\begin{align}\n\t\t\t\t\t\t\t%                h(\\x) \\leq h_{\\eta}(\\x) \\leq h(\\x) + \\eta L_0.\n\t\t\t\t\t\t\t%            \\end{align} \n\t\t\t\t\t\t% \\item[(vi)] $h_{\\eta} \\in C^{0,0}$.  \\us{Prove//cite} FY: This seems redundant. It is already shown in (ii).\n\t\t\t\t\t\t%  \\item[(iv)] If \\fy{$\\eta>0$, then $ h_{\\eta}(\\x)$ is continuously differentiable and} $\\nabla h_{\\eta}(\\x)$ is defined as $$ \\nabla h_{\\eta}(\\x) \\triangleq \\tfrac{1}{\\kappa} \\int_{\\fy{\\mathbb{R}^n}} \\tfrac{f(\\x+\\eta u) - f(\\x)}{\\eta} e^{-\\frac{\\|u\\|^2}{2}} Bu du. $$ \\fy{Moreover}, $h_{\\eta} \\in C^{1,1}$ and $L_1(h_{\\eta}) = \\tfrac{\\sqrt{n}}{\\eta} L_0(h).$  \n\t\t\t\t\t\t%    \\item[(vi)] If $h$ is convex {and $h \\in C^{0,0}(\\Xscr_{\\eta})$ with parameter $L_0$}, then $\\nabla_x h_{\\eta}(\\x) \\in \\partial_{\\epsilon} h(\\x)$ where {$\\epsilon \\triangleq \\eta L_0$}.\n\t\t\t\t\t\t\n\t\t\t\t\t\t%\\item[(vii)] If $h \\in C^{0,0}$ and $\\eta > 0$, then $h_{\\eta} \\in C^{1,1}$ and $L_1(f_{\\eta}) = \\tfrac{\\sqrt{n} L_0(f)}{\\eta}.$ \n\t\t\t\t\t\t%     \\us{Modify for spherical/Prove/Cite} FY: I think the version we use in the paper is already shown in (iv)\n\t\t\t\t\t\t\n\t\t\t\t\t\t%    \\item[(vii)] If {$h \\in C^{1,1}(\\Xscr_{\\eta})$} with constant {$L_1$}, then $$\\|\\nabla_x h_{\\eta}(\\x)-\\nabla_x h(\\x) \\| \\leq {\\eta L_1 n}.$$\n\t\t\t\t\t\t\n\t\t\t\t\t\t\\item [(v)] {Suppose $h \\in C^{0,0}(\\Xscr_{\\eta})$ with parameter $L_0$ {and} \n\t\t\t\t\t\t\t$g_{\\eta}(\\x,\\vv)  \\triangleq \\left(\\tfrac{n(h(\\x+\\vv) - h(\\x-\\vv))\\vv}{2\\eta\\|\\vv\\|}\\right)$\n\t\t\t\t\t\t\tfor $\\vv \\in \\eta\\mathbb{S}$.  Then for} any $\\x \\in \\Xscr$, we have that ${\\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}}}[{\\|{g_\\eta}(\\x,\\vv)\\|^2}] \\leq 16\\sqrt{2\\pi}L_0^2 n$. $\\hfill \\Box$ \n\t\t\t\t\t\\end{enumerate}\n\t\t\t\t\\end{lemma}\n\n\\begin{proof}\n\t\t\t\t\t{(i) By Stoke's theorem and spherical symmetry of $\\vv$, we have that \n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\nabla_{\\x} h_{\\eta}(\\x)& = \\tfrac{n}{\\eta} \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\ \\left[ \\tfrac{h(\\x+\\vv) \\vv }{\\|\\vv\\|} \\right] = \n\t\t\t\t\t\t\t\\tfrac{n}{2\\eta} \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\ \\left[ \\tfrac{(h(\\x+\\vv)+h(\\x+\\vv)) \\vv }{\\|\\vv\\|} \\right] \\\\\n\t\t\t\t\t\t\t& = \\tfrac{n}{2\\eta} \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\ \\left[ \\tfrac{(h(\\x+\\vv)) \\vv }{\\|\\vv\\|}+\\tfrac{(h(\\x+\\vv)) \\vv }{\\|\\vv\\|} \\right] = \n\t\t\t\t\t\t\t\\tfrac{n}{2\\eta} \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\ \\left[ \\tfrac{(h(\\x+\\vv)) \\vv }{\\|\\vv\\|}+\\tfrac{(h(\\x-\\vv)) (-\\vv) }{\\|\\vv\\|} \\right] \\\\\n\t\t\t\t\t\t\t& = \\tfrac{n}{2\\eta} \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\ \\left[ \\tfrac{((h(\\x+\\vv))-(h(\\x-\\vv)) \\vv) }{\\|\\vv\\|} \\right]. \n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t(ii,iii) follow akin to ~\\cite[Lemma 1]{CSY2021MPEC}. \n\t\t\t\t\t\t(iv)\nRecall from~\\cite[Prop. 2.2]{lin2022} that $\\|\n\\nabla_{\\x} h_{\\eta}(\\x) -\\nabla_{\\x}\nh_{\\eta}(\\y)\\| \\leq \\tfrac{C_n\nL_0}{\\eta}\\|\\x-\\y\\|,$ for $n\\geq 1$, where\n$C_n \\triangleq\n\\left(\\tfrac{2}{\\pi}\\right)^{p_n}\n\\tfrac{n!!}{(n-1)!!}$ and $p_n = 1$ if $n$\nis even, and $p_n =0$ otherwise. It suffices to show that $C_n \\le 1$ for $n \\ge 1$. For $n \\ge\n0$, let $w_n$ denote the Wallis'\nintegrals~\\cite{spivak2006calculus}, given as\n$w_n = \\int_{0}^{\\pi/2}\\sin^n(u)du$. Then we\nhave $w_n =\\left(\\tfrac{n-1}{n}\\right)w_{n-2}$ for $n \\geq\n2$ where $w_0=\\tfrac{\\pi}{2}$ and $w_1=1$\n(cf.~\\cite[Ch. 19]{spivak2006calculus}). This\nimplies (via mathematical induction) that\n$w_n =\n\\left(\\tfrac{\\pi}{2}\\right)\\tfrac{(n-1)!!}{n!!}$\nfor even $n$, while $w_n =\n\\tfrac{(n-1)!!}{n!!}$ for odd $n$. Let us\ndefine $v_n \\,\\triangleq \\, (n+1)w_n w_{n+1}$ for $n\\geq 0$.\nWe obtain $v_n =\n(n+1)\\left(\\tfrac{\\pi}{2}\\right)\n\\tfrac{(n-1)!!}{n!!}\\tfrac{n!!}{(n+1)!!} =\n\\tfrac{\\pi}{2}$, implying that $v_n =\n\\tfrac{\\pi}{2}$ for all $n\\geq 0$. Recall\nthat from~\\cite{spivak2006calculus} we have\n$w_{n+1} < w_n$ for all $n\\geq 0$. Thus we\nobtain $\\tfrac{\\pi}{2} = v_n  =(n+1)w_n\nw_{n+1} < (n+1)w_n^2$ for all $n\\geq 0$, i.e., $w_n < \\tfrac{\\pi}{2(n+1)}$ for $n \\ge 0$.  Note\nthat $C_1 =1$. Suppose $n \\geq 2$. We have\n$C_n= \\tfrac{1}{w_n} <\n\\sqrt{\\tfrac{2(n+1)}{\\pi}} =\n\\sqrt{\\tfrac{2(1+1/n)}{\\pi}} \\sqrt{n} \\leq\n\\sqrt{\\tfrac{2(1+1/2)}{\\pi}} \\sqrt{n} \\leq\n0.98 \\sqrt{n} $. This implies that $C_n  \\leq\n1$ for all $n\\geq 1$.} {(v) We have that}\n\\begin{align*}\n\t\t\t\t\t\t\t\\mathbb{E}\\left[ \\|g_{\\eta}(\\x,\\vv)\\|^2 \\right] & \\le \n\t\t\t\t\t\t\t\\tfrac{n^2}{4\\eta^2} \\mathbb{E}\\left[ \\|\\tfrac{(h(\\x+\\vv)-h(\\x-\\vv))\\vv}{\\|\\vv\\|}\\|^2 \\right] \\le \n\t\t\t\t\t\t\t\\tfrac{n^2}{4\\eta^2} \\mathbb{E}\\left[  (h(\\x+\\vv)-h(\\x-\\vv))^2 \\tfrac{\\|\\vv\\|^2}{\\|\\vv\\|^2} \\right] \\\\\n\t\t\t\t\t\t\t& \\le \n\t\t\t\t\t\t\t\\tfrac{n^2}{4\\eta^2} \\mathbb{E}\\left[ \\left(h(\\x+\\vv)-\\mathbb{E}\\left[h(\\x+\\vv)\\right] + \\mathbb{E}\\left[ h(\\x+\\vv)\\right]-h(\\x-\\vv)\\right)^2  \\right]. \n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\tBut $\\mathbb{E}\\left[ h(\\x+\\vv)\\right] = \\mathbb{E}\\left[ h(\\x-\\vv)\\right]$ by spherical symmetry. Consequently, \n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\mathbb{E}\\left[ \\|g_{\\eta}(\\x,\\vv)\\|^2 \\right] & \\le \n\t\t\t\t\t\t\t\\tfrac{n^2}{4\\eta^2} \\mathbb{E}\\left[  \\left(h(\\x+\\vv)-\\mathbb{E}\\left[h(\\x+\\vv)\\right] + \\mathbb{E}\\left[ h(\\x-\\vv)\\right]-h(\\x-\\vv)\\right)^2  \\right] \\\\\n\t\t\t\t\t\t\t& \\le \\tfrac{2n^2}{4\\eta^2} \\mathbb{E}\\left[ \\left|(h(\\x+\\vv)-\\mathbb{E}\\left[h(\\x+\\vv)\\right]\\right|^2  + \\left|\\mathbb{E}\\left[ h(\\x-\\vv)\\right]-h(\\x-\\vv))\\right|^2 \\right]. \n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\tThen by invoking L\\'{e}vy's inequality (cf. Lemma~\\ref{lemma:levy-concentration}), we have that \n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\mathbb{E}\\left[ |(h(\\x+\\vv)-\\mathbb{E}\\left[h(\\x+\\vv)\\right]|^2 \\right] \\le 2\\sqrt{2\\pi}\\int_0^{\\infty} e^{-\\alpha n}{8 \\eta^2 L_0^2} d\\alpha = \\tfrac{16\\sqrt{2\\pi} \\eta^2 L_0^2}{n}. \n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t{By noting that $\\mathbb{E}\\left[ \\left|(h(\\x+\\vv)-\\mathbb{E}\\left[h(\\x+\\vv)\\right]\\right|^2  \\right] = \\mathbb{E}\\left[  \\left|\\mathbb{E}\\left[ h(\\x-\\vv)\\right]-h(\\x-\\vv))\\right|^2 \\right]$ and leveraging the preceding bound, we get}  \n\t\t\t\t\t\t$\\mathbb{E}\\left[ \\|g_{\\eta}(\\x,\\vv)\\|^2 \\right] \\le \\tfrac{n^2}{\\eta^2} \\tfrac{16\\sqrt{2\\pi} \\eta^2 L_0^2}{n} = 16\\sqrt{2\\pi}L_0^2 n.$\n\t\t\t\t\t\n\t\t\t\t\\end{proof}\n\n\\begin{lemma}[{\\cite[Lemma 6]{CSY2021MPEC}}]\\label{lem:inexact_proj_2}\\em\n\t\t\t\t\tFor any $\\beta,\\eta>0, \\x \\in \\Xscr, \\tilde{e} \\in\\mathbb{R}^n$, \n\t\t\t\t\t$$\\|G_{\\eta,\\beta}(\\x) \\|^2   \\leq  {2} \\| \\tilde{G}_{\\eta,\\beta}(\\x, {\\tilde e}) \\|^2 + 2 \\|\\tilde{e}\\|^2.$$  \n\t\t\t\t\\end{lemma}\n\n\\begin{lemma}\\label{lemma:mon_G} \\em Suppose $\\eta > 0$ and $\\{\\gamma_k\\}$ is a diminishing sequence. Then, for any $\\x$ and $k \\geq 0$, $\\| G_{\\eta,1/\\gamma_k} (\\x)\\| \\, \\leq \\,\\| G_{\\eta,1/\\gamma_{k+1}} (\\x)\\|$. \n\t\t\t\t\\end{lemma}\n\n\\begin{lemma}[{\\bf Bias and moment properties of ${\\bf e}$}] \\label{lem:stoch_error_var}\\em\n\t\t\t\t\t%Consider Definition \\ref{def:stoch_errors}. \n\t\t\t\t\tLet Assumptions \\ref{ass-1} and \\ref{assum:random_vars} hold. Then (i) and (ii) hold a.s. for $k\\geq 0$, $N_k \\geq 1$, and $j = 1, \\hdots, N_k$.\n\t\t\t\t\t\n\t\t\t\t\t\\noindent (i) $\\mathbb{E}[\\mathbf{e}_{j,k}\\mid \\mathcal{F}_k] = 0$;  \\noindent (ii) $\\mathbb{E}[\\|\\mathbf{e}_{k}\\|^2\\mid \\mathcal{F}_k] \\leq  \\tfrac{16\\sqrt{2\\pi} L_0^2 n}{N_k}$.\n\t\t\t\t\\end{lemma}\n\n\\begin{proof}\n\t\t\t\t\t\t(i) To show the conditional unbiasedness of $\\mathbf{e}_{j,k}$, we proceed as follows.\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\mathbb{E}_{\\bxi,\\! \\vv}\\left[ {g_{\\eta}(\\x_k,\\!\\vv_{j,k},\\!\\bxi_{j,k})}  \\mid {\\cal F}_k\\right]\\! & =\\mathbb{E}_{\\vv}\\!   \n\t\t\t\t\t\t\t\\left[\\mathbb{E}_{\\bxi}\\left[ \\tfrac{n(\\f(\\x_k+v_{j,k},\\xi_{j,k})-\\f(\\x_k-v_{j,k},\\xi_{j,k}))v_{j,k}}{2\\eta \\|v_{j,k}\\|} \\hspace{-0.05in} \\mid {\\cal F}_k \\cup \\{v_{j,k}\\} \\right] \\right] \\\\\n\t\t\t\t\t\t\t&\\hspace{-1in}  \\overset{\\tiny \\mbox{Lemma~\\ref{lemma:props_local_smoothing}~{(i)}}}{=}\n\t\t\t\t\t\t\t\\mathbb{E}_{\\vv}\\left[ \\tfrac{n}{2\\eta}\\tfrac{(f(\\x_k+v_{j,k})-f(\\x_k\\lm{-}v_{j,k}))v_{j,k}}{\\|v_{j,k}\\|} \\mid {\\cal F}_k \\right]  = \\nabla_{\\x} f_{\\eta}(\\x_k).    \n\t\t\t\t\t\t\\end{align*} \n\t\t\t\t\t\tBy leveraging Lemma~\\ref{lemma:props_local_smoothing}~{(v)}, we have that \n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\mathbb{E}_{{\\bxi, \\vv}}\\left[\\, \\|\\mathbf{e}_{j,k}\\|^2 \\mid {\\cal F}_k \\, \\right] &= \n\t\t\t\t\t\t\t\\mathbb{E}_{{\\bxi}}\\left[\\, \\mathbb{E}_{ {\\vv}}\\left[\\, \\|\\mathbf{e}_{j,k}\\|^2  \\, \\mid{\\cal F}_{k}\\cup\\{\\xi_{j,k}\\} \\right] \\right]\\\\\n\t\t\t\t\t\t\t& \\le \n\t\t\t\t\t\t\t\\mathbb{E}_{{\\bxi}}\\left[\\, 16\\sqrt{2\\pi}\\tilde{L}_0(\\bxi_{{j,k}})^2n \\right] \\le 16\\sqrt{2\\pi} L_0^2 n. \n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\tIt follows that $\\mathbb{E}\\left[\\, \\|\\mathbf{e}_{k}\\|^2  \\, \\mid {\\cal F}_k\\right] \\le \\tfrac{16\\sqrt{2\\pi} L_0^2 n}{N_k}.$\n\t\t\t\t\\end{proof}\n\n\\begin{lemma}\\label{lemma:zog_ineq}\\em \n\t\t\t\t\tLet Assumption \\ref{ass-1} hold {and suppose} $\\left\\{\\x_k\\right\\}$ is generated by\n\t\t\t\t\tAlgorithm~\\ref{algorithm:zo_nonconvex}. Let $\\{\\gamma_k\\}$ be a\n\t\t\t\t\tnon-increasing stepsize sequence (e.g., constant or diminishing) where\n\t\t\t\t\t$\\gamma_0 \\in (0,{\\tfrac{\\eta}{{\\sqrt{n}}L_0}})$ for $\\eta>0$. Then  the\n\t\t\t\t\tfollowing hold. \n\t\t\t\t\t\n\t\t\t\t\t\\noindent (i) For $k\\geq 0$, we have \n\t\t\t\t\t\\begin{align}\\label{eqn:recursive_ineq_lemma5} \n\t\t\t\t\t\t\\left( 1-\\tfrac{{\\sqrt{n}}L_0\\gamma_k}{\\eta}\\right)\\tfrac{\\gamma_k \\|G_{\\eta,1/\\gamma_0} (\\x_k)\\|^2}{4}     &   \\leq   f_{\\eta}(\\x_k)-      f_{\\eta}(\\x_{k+1}) \n\t\t\t\t\t\t+{\\left( 1-\\tfrac{{\\sqrt{n}}L_0\\gamma_k}{2\\eta}\\right)} \\gamma_k \\|e_k\\|^2.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\t\n\t\t\t\t\t\\noindent (ii) Let $R_{\\ell,K}$ be a random integer taking values in $\\{\\ell, \\ldots, K-1\\}$ with mass function $\\mathbb{P}{[} R_{\\ell,K} = j {]} = \\tfrac{\\gamma_j}{\\sum_{i=\\ell}^{K-1} \\gamma_i}$ for any $j \\in \\{\\ell, \\ldots, K-1\\}$.  Then, for all $k\\geq 0$ we have \n\t\t\t\t\t\\begin{align}\\label{ineq:residual_map_ineq_VRGZO_2}\n\t\t\t\t\t\t\\mathbb{E}\\left[ \\|G_{\\eta,{1/\\gamma_0}} (\\x_{R_{\\ell,K}})\\|^2\\right]\n\t\t\t\t\t\t&\\leq \n\t\t\t\t\t\t\\tfrac{\\left( \\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left({16 \\sqrt{2\\pi} L_0^2{n} }\\right)\\sum_{k=\\ell}^{K-1} \\tfrac{\\gamma_k}{N_k}\\right)}{\\left(\\left( \\tfrac{1}{4}-\\tfrac{{\\sqrt{n}} L_0\\gamma_0}{4\\eta}\\right) \\sum_{k=\\ell}^{K-1} \\gamma_k\\right)} .\n\t\t\t\t\t\\end{align}\n\t\t\t\t\\end{lemma}\n\n\\begin{proof}\n\t\t\t\t\t(i) From Lemma~\\ref{lemma:props_local_smoothing} (iv), the gradient mapping $\\nabla f_\\eta$ is Lipschitz continuous with the parameter $L_{\\eta}\\triangleq \\tfrac{\\sqrt{n}L_0}{\\eta}$. From the descent lemma, \n\t\t\t\t\t\\begin{align}\\label{eqn:descent_lem_proof_eq1}\n\t\t\t\t\t\\notag\tf_{\\eta}(\\x_{k+1}) & \\leq f_{\\eta}(\\x_{k}) + \\nabla f_{\\eta}(\\x_k)^\\top(\\x_{k+1}-\\x_k)\n\t\t\t\t\t\t+\\tfrac{L_{\\eta}}{2}\\|\\x_{k+1}-\\x_k\\|^2\\\\\n\t\t\t\t\t\\notag\t&=f_\\eta(\\x_k) + \\left(\\nabla f_{\\eta}(\\x_k)+{e_k}\\right)^\\top(\\x_{k+1}-\\x_k)\\\\\n\t\t& -{e_k}^\\top(\\x_{k+1}-\\x_k)+\\tfrac{L_{\\eta}}{2}\\|\\x_{k+1}-\\x_k\\|^2.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\tInvoking %Lemma \\ref{eqn:proof_comapct_eq1}, \n\t\t\t\t\t\tthe properties of the Euclidean projection, we have \\\\$ \n\t\t\t\t\t(\\x_k-\\gamma_k\\left(\\nabla f_\\eta(\\x_k)+e_k\\right)-\\x_{k+1})^\\top(\\x_k-\\x_{k+1})\\leq 0$. This implies that \n\t\t\t\t\t\\begin{align}\\label{eqn:descent_lem_proof_eq2}\n\t\t\t\t\t\t\\left(\\nabla f_\\eta(\\x_k)+{e_k}\\right)^\\top(\\x_{k+1}-x_k)\n\t\t\t\t\t\t\\leq -\\tfrac{1}{\\gamma_k}\\|\\x_{k+1}-\\x_k\\|^2.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\tIn addition, we may also express $e_k^{\\top}(\\x_{k+1}-\\x_k)$ as follows. \n\t\t\t\t\t\\begin{align}\\label{eqn:descent_lem_proof_eq3}\n\t\t\t\t\t\t-e_k^{\\top}(\\x_{k+1}-\\x_k)\n\t\t\t\t\t\t\\leq \\tfrac{{\\gamma_k}}{2}\\|e_k\\|^2+\\tfrac{1}{2{\\gamma_k}}\\|\\x_{k+1}-\\x_k\\|^2.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\tCombining the inequalities \\eqref{eqn:descent_lem_proof_eq1}, \\eqref{eqn:descent_lem_proof_eq2}, and \\eqref{eqn:descent_lem_proof_eq3} we obtain \n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\tf_{\\eta}(\\x_{k+1})  & \\leq   f_{\\eta}(\\x_k)+ \\left( -\\tfrac{1}{2\\gamma_k}+\\tfrac{L_{\\eta}}{2}\\right) \\|\\x_{k+1}-\\x_k\\|^2 +\\tfrac{\\gamma_k}{2} \\|e_k\\|^2 \\\\\n\t& \\overset{\\tiny \\mbox{Def.~\\ref{def:res_maps}}}{=} f_{\\eta}(\\x_k)- \\left( \\tfrac{1}{2\\gamma_k}-\\tfrac{L_{\\eta}}{2}\\right)\\gamma_k^2 \\left\\|\\tilde{G}_{\\eta,1/\\gamma_k}(\\x_k, {e_k})\\right\\|^2+\\tfrac{\\gamma_k}{2} \\|e_k\\|^2\\\\\n\t& = f_{\\eta}(\\x_k)- \\left( 1-L_{\\eta}\\gamma_k\\right)\\tfrac{\\gamma_k}{2} \\left\\|\\tilde{G}_{\\eta,1/\\gamma_k}(\\x_k, {e_k})\\right\\|^2+\\tfrac{\\gamma_k}{2} \\|e_k\\|^2\\\\\n\t& \\hspace{-0.2in}\\overset{\\tiny \\mbox{Lemma~\\ref{lem:inexact_proj_2}}}{\\le} f_{\\eta}(\\x_k)- \\left( 1-L_\\eta \\gamma_k\\right)\\tfrac{\\gamma_k}{4} \\left\\|G_{\\eta,1/\\gamma_k}(\\x_k)\\right\\|^2+\\left(\\tfrac{\\gamma_k}{2} + \\left( 1-L_{\\eta} \\gamma_k\\right)\\tfrac{\\gamma_k}{2}\\right) \\|e_k\\|^2,\n\t\t\t\t\t\\end{align*}\nwhere $1-L_{\\eta}\\gamma_k > 0$ for any $k$  since $\\gamma_0 < \\tfrac{1}{L_{\\eta}}$. \n\t\t\t\t\tFrom the preceding relation and invoking Lemma~\\ref{lemma:mon_G}, we obtain~\\eqref{eqn:recursive_ineq_lemma5}, i.e.,\n\\begin{align*}\n\\left( 1-L_\\eta \\gamma_k\\right)\\tfrac{\\gamma_k}{4} \\left\\|G_{\\eta,1/\\gamma_0}(\\x_k)\\right\\|^2 \\, \\le \\, f_{\\eta}(\\x_k)- f_{\\eta}(\\x_{k+1}) + \\left(1-\\tfrac{L_{\\eta} \\gamma_k}{2}\\right)\\gamma_k \\|e_k\\|^2.\n\\end{align*} \n\t\t\t\t\t\n\t\t\t\t\t\\noindent (ii) Summing \\eqref{eqn:recursive_ineq_lemma5}  from $k =\\ell, \\ldots, K-1$ where $\\ell\\triangleq  \\lceil \\lambda K\\rceil$ and taking expectations with respect to the iterate trajectory, we have\n\t\t\t\t\t\\begin{align}\n\t\t\t\t\t\t\\notag \\sum_{{k=\\ell}}^{K-1} \\left( 1-\\tfrac{ \\sqrt{n} L_0\\gamma_k}{\\eta}\\right) \\tfrac{\\gamma_k}{4}\\EXP{\\, \\|G_{\\eta,1/\\gamma_k} (\\x_k)\\|^2\\, }  \n\t\t\t\t\t\t& \\leq   \\EXP{\\, f_{\\eta}(\\x_\\ell)\\, }-      \\EXP{\\, f_{\\eta}(\\x_{K})\\, }\\\\ \n\t\t\t\t\t\t&+\\sum_{{k=\\ell}}^{K-1}\\left( 1-\\tfrac{\\sqrt{n} L_0\\gamma_k}{2\\eta}\\right) \\gamma_k \\EXP{\\, \\|\\mathbf{e}_k\\|^2\\, }.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\tInvoking the definition of $R_{\\ell,K}$ and Lemma~\\ref{lemma:mon_G}, we have \n\t\t\t\t\t\\begin{align} \n\t\t\t\t\t\t\\notag & \\quad \\left( \\tfrac{1}{4}-\\tfrac{\\sqrt{n} L_0\\gamma_0}{4\\eta}\\right) \\left(\\sum_{k=\\ell}^{K-1} \\gamma_k\\right)\\EXP{\\mathbb{E}_{R_{\\ell,K}}\\left[\\|G_{\\eta,1/\\gamma_0} ({\\x_{R_{\\ell,K}}})\\|^2\\right]}\\\\  \n\t\t\t\t\t\t& \\notag \\leq   \\left( \\tfrac{1}{4}-\\tfrac{\\sqrt{n} L_0\\gamma_0}{4\\eta}\\right)\\sum_{{k=\\ell}}^{K-1} \\gamma_k \\mathbb{E}\\left[\\|G_{\\eta,1/\\gamma_0} ({\\x_k})\\|^2\\right]   \\leq\t\t\t\t\t\t   \\EXP{f_{\\eta}(\\x_\\ell)-f^*_{\\eta}}+\\sum_{{k=\\ell}}^{K-1} \\gamma_k \\EXP{\\|\\mathbf{e}_k\\|^2}, \n\t\t\t\t\t\\end{align}\n\t\t\t\t\twhere the last inequality follows from $\\left( 1-\\tfrac{\\sqrt{n} L_0\\gamma_k}{2\\eta}\\right) \\leq 1$ for any $k$. \n\t\t\t\t\t{Note that we have} \n\t\t\t\t\t\\begin{align}\\notag\n\t\t\t\t\t\t\\mathbb{E}\\left[f_{\\eta}(\\x_{\\ell})\\right] - {f^*_{\\eta}} & =   \\mathbb{E}\\left[ f(\\x_{\\ell}) + f_{\\eta}(\\x_{\\ell})-f(\\x_{\\ell})\\right]-f^*_{\\eta} +f^*\n\t\t\t\t\t\t-f^*\\\\\n\t\t\t\t\t\t\\notag & = \\mathbb{E}\\left[{f(\\x_{\\ell})}\\right]-f^*+ \\mathbb{E}\\left[\\left | f_{\\eta}(\\x_{\\ell})-f(\\x_{\\ell})\\right|\\right] + \\left| f^*-f^*_\\eta\\right| \\\\\n\t\t\t\t\t\t& \\stackrel{\\tiny{\\hbox{Lemma~\\ref{lemma:props_local_smoothing} (iii)}}}{\\leq}\n\t\t\t\t\t\t\\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta.\n\t\t\t\t\t\t\\label{bd_feta}\n\t\t\t\t\t\\end{align}\n\t\t\t\t\tFrom Lemma \\ref{lem:stoch_error_var} (ii), \n\t\t\t\t\t$\\mathbb{E}\\left[\\|\\mathbf{e}_k\\|^2\\right] \\leq \\mathbb{E}\\left[\\mathbb{E}\\left[\\|\\mathbf{e}_k\\|^2\\mid \\mathcal{F}_k\\right]\\right] \n\t\t\t\t\t\\leq \\tfrac{ 16\\sqrt{2\\pi}  L_0^2n}{N_k}.$\n\t\t\t\t\tFrom the preceding relations, we obtain the inequality \\eqref{ineq:residual_map_ineq_VRGZO_2}.\n\t\t\t\t\\end{proof}\n\n\\begin{proof}\n\t\t\t\t\t\\noindent {\\bf (a-i)} From~\\eqref{ineq:residual_map_ineq_VRGZO_2}, we obtain\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\mathbb{E}\\left[ \\|G_{\\eta,{1/\\gamma}} ({\\x_{R_{\\ell,K}}})\\|^2\\right]\n\t\t\t\t\t\t&\\leq  \n\t\t\t\t\t\t\\tfrac{\\left( \\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left(16 \\sqrt{2\\pi} L_0^2{n} \\right)\\gamma\\sum_{k=\\ell}^{K-1} \\tfrac{1}{N_k}\\right)}{\\left(\\left( 1-\\tfrac{\\sqrt{n}L_0\\gamma}{\\eta}\\right) \\tfrac{\\gamma}{4}(K-\\ell)\\right)} \\\\\n\t\t\t\t\t\t& \\leq \\tfrac{\\left( \\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left({16 \\sqrt{2\\pi} L_0^2 n}\\right){\\gamma}\\sum_{k=\\ell}^{K-1} \\tfrac{a^{-1}}{{L_0}{{\\sqrt{n}}{(k+1)}}}\\right)}{\\left(\\left( 1-\\tfrac{\\sqrt{n}L_0\\gamma}{\\eta}\\right) \\tfrac{\\gamma}{4}(K-\\ell)\\right)},  \n\t\t\t\t\t\\end{align*}\n\t\t\t\t\twhere the expectation is with respect to both the iterate trajectory and the random integer $R_{\\ell,K}$, while the last inequality  follows from the definition of $N_k$. Recall that $K> \\tfrac{2}{1-\\lambda}$ implies $\\ell \\leq K-1$ and  \n\t\t\t\t\t$ \\textstyle\\sum_{k={\\ell}}^{K-1}\\tfrac{1}{k+1}%\\leq  \\tfrac{1}{\\ell+1}+\\ln\\left( \\tfrac{K}{\\ell+1}\\right) \n\t\t\t\t\t\\leq 0.5 +\\ln\\left(\\tfrac{N}{\\lambda N+1}\\right)\\leq 0.5-\\ln(\\lambda).$\n\t\t\t\t\t{Further}, $K-\\ell \\geq K-\\lambda K=(1-\\lambda)K$ {implying that} \\eqref{eqn:GBound_constant_gamma} holds.\n\t\t\t\t\t%\\begin{align*}\n\t\t\t\t\t%    \\mathbb{E}\\left[ \\|G_{\\eta,1/\\gamma} ({\\x_{R_{\\ell,K}}})\\|^2\\right] \\leq \\tfrac{\\left(\\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left(\\uss{16 \\farz{\\sqrt{n}} \\sqrt{2\\pi} L_0}\\right)  (0.5-\\ln(\\lambda)){\\lm{a^{-1}}}\\gamma\\right)}{\\left(\\left( 1-\\tfrac{\\farz{\\sqrt{n}}L_0\\uss{\\gamma}}{\\eta}\\right) \\tfrac{\\gamma}{4}(1-\\lambda)K\\right)}.\n\t\t\t\t\t%\\end{align*}\n\t\t\t\t\t\n\t\t\t\t\t\\noindent {\\bf (a-ii)} To show (ii-1), using the relation in part (i) and by $\\gamma:= \\tfrac{\\eta}{2\\sqrt{n}L_0}$, we obtain\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t& \\mathbb{E}\\left[ \\|G_{\\eta,1/\\gamma} (\\x_{R_{\\ell,K}})\\|^2\\right] \\leq  \n\t\t\t\t\t\t\\tfrac{\\left(\\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left(8\\eta \\sqrt{2\\pi} \\right)  (0.5-\\ln(\\lambda)){a^{-1}}\\right)}{\\left(\\eta(1-\\lambda)K\\right)/(16\\sqrt{n}L_0)}.\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tFrom Jensen's inequality,  $\\mathbb{E}\\left[ \\|G_{\\eta,1/\\gamma} (\\x_{R_{\\ell,K}})\\|\\right] \\leq \\sqrt{\\mathcal{O}\\left({n^{1/2}}{(L_0\\eta^{-1}+L_0^2)}/K\\right)}$, implying that $K_{\\epsilon}=\\mathcal{O}\\left({n^{1/2}}{(L_0\\eta^{-1} +L_0^2)} \\epsilon^{-2}\\right)$. Next, we derive sample complexity as per (ii-2). %The sample complexity of {function evaluations} is as follows.\n\t\t\t\t\t\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\sum_{k=0}^{K_\\epsilon} N_k & = \\sum_{k=0}^{K_\\epsilon}  \\lceil  {a{\\sqrt{n} L_0}(k+1)}\\rceil \n\t\t\t\t\t\t\\leq {\\sqrt{n} L_0}  {\\mathcal{O}(K_{\\epsilon}+{K_\\epsilon^2})}\n\t\t\t\t\t\t\\leq \\mathcal{O}\\left({n^{3/2}(L_0^3\\eta^{-2}+L_0^5)} \\epsilon^{-4}\\right). %\\uss{ \\mbox{CHECK...exponents of $n,L_0$}} \\\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\t%\\farz{{\\bf TODO:} I don't think $\\eta^{-2}$ will appear in the complexity result here. Please see my earlier comment on (a-ii-1).  }\n\t\t\t\t\t\n\t\t\t\t\t\\noindent {\\bf (b-i)}\n\t\t\t\t\t{Consider the inequality~\\eqref{ineq:residual_map_ineq_VRGZO_2}.  Using $\\tfrac{\\gamma_k}{N_k}\\leq \\tfrac{\\gamma_0}{{a{\\sqrt{n}} L_0}(k+1)}$, we obtain} \n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\mathbb{E}\\left[ \\|G_{\\eta,{1/\\gamma_0}} (\\x_{R_{\\ell,K}})\\|^2\\right]\n\t\t\t\t\t\t& \\leq \\tfrac{\\left( \\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left({16  \\sqrt{2\\pi} L_0^2{n}} \\right)\\sum_{k=\\ell}^{K-1} \\tfrac{\\gamma_0}{{a \\sqrt{n} L_0}(k+1)}\\right)}{\\left(\\left( 1-\\tfrac{{\\sqrt{n}} L_0\\gamma_0}{\\eta}\\right) {\\tfrac{\\gamma_0}{2}}(1-\\sqrt{\\lambda})\\sqrt{K}\\right)},\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\twhere we note that $\\textstyle\\sum_{k=\\ell}^{K-1} \\gamma_k \\geq \\textstyle\\int_{\\ell-1}^{K-1} \\tfrac{\\gamma_0}{\\sqrt{x+1}} dx = 2\\gamma_0(\\sqrt{K}-\\sqrt{\\ell}) \\geq 2\\gamma_0(1-\\sqrt{\\lambda})\\sqrt{K}.$ As earlier, we have that  \n\t\t\t\t\t$ \\textstyle\\sum_{k={\\ell}}^{K-1}\\tfrac{1}{k+1} \\leq 0.5-\\ln(\\lambda),$ leading to the inequality in (b-i). \n\t\t\t\t\t\n\t\t\t\t\t\\noindent {\\bf (b-ii)} From Jensen's\n\t\t\t\t\tinequality and by setting $\\gamma_0 =\n\t\t\t\t\t\\tfrac{\\eta}{2 {\\sqrt{n}} L_0}$, we obtain\n\t\t\t\t\tthat \\\\$ \\mathbb{E}\\left[\n\t\t\t\t\t\\|G_{\\eta,1/\\gamma_0}\n\t\t\t\t\t(\\x_{R_{\\ell,K}})\\|\\right] \\leq\n\t\t\t\t\t\\sqrt{\\mathcal{O}\\left({(L_0\\eta^{-1}+L_0^2)\\sqrt{n/K}}\\right)}$, thus\n\t\t\t\t\t$K_{\\epsilon}=\\mathcal{O}\\left(\\tfrac{n(L_0^2\\eta^{-2} +L_0^4)}{\\epsilon^{4}}\\right)$.\\\\ Next, (b-ii-2) holds by\n\t\t\t\t\tbounding the sample complexity as follows.\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\sum_{k=0}^{K_\\epsilon} N_k & = \\sum_{k=0}^{K_\\epsilon}  \\lceil  {a\\sqrt{n} L_0}\\sqrt{k+1}\\rceil \n\t\t\t\t\t\t\\leq {\\sqrt{n} L_0}  {\\mathcal{O}(K_{\\epsilon}+{K_\\epsilon^{3/2}})}\n\t\t\t\t\t\t\\leq \\mathcal{O}\\left({n^{2}(L_0^4\\eta^{-3}+L_0^7)} \\epsilon^{-6}\\right).\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\\end{proof}","units":[{"start":42722,"end":43226,"start_line":737,"end_line":743,"raw_tex":"\\begin{lemma}[L\\'{e}vy concentration on $ \\mathbb{S}^n $~{\\cite[Prop. 3.11 and Example 3.12]{Wainwright2019}}]\n\t\t\t\t\t\\label{lemma:levy-concentration}\\em\n\t\t\t\t\tLet ${h}: \\mathbb{R}^n \\to \\mathbb{R}$ be $L$-Lipschitz and $\\tilde{\\vv}$ be uniformly distributed on {$\\mathbb{S}$}. Then \n\t\t\t\t\t\\begin{align}\n\t\t\t\t\t\t\\mathbb{P}\\left[\\, \\left|\\, h(\\tilde{\\vv}) - \\mathbb{E}[h(\\tilde{\\vv})]\\, \\right|\\, \\geq \\, \\alpha \\, \\right] &\\, \\leq \\, 2 \\sqrt{2\\pi} e^{-\\frac{n \\alpha^2}{8L^2}}.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\\end{lemma}","sha256_utf8":"b7204e2b186b730a8086dbccbec1841dde5876f9d61155f9d289949441ec725e"},{"start":43233,"end":46287,"start_line":745,"end_line":777,"raw_tex":"\\begin{lemma}[{\\bf Properties of spherical smoothing}] \\label{lemma:props_local_smoothing}\\em Suppose $h:\\mathbb{R}^n \\mapsto \\mathbb{R}$ is {a continuous function and its smoothed counterpart $h_{\\eta}$ is defined as \\eqref{def-smooth}, where $\\eta>0$ is a given scalar}. Then the following hold for any $\\x, \\y \\in {\\cal X}$.\n\t\t\t\t\t\n\\begin{enumerate}\n\t\t\t\t\t\t\\item[ (i)] $h_{\\eta}$ is $C^1$ over $\\Xscr$ and $\\nabla_{\\x} h_{\\eta}(\\x) = \\left(\\tfrac{n}{2\\eta}\\right) \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\left[\\left(h(\\x+\\vv)-h(\\x-\\vv)\\right) \\tfrac{\\vv}{\\|\\vv\\|}\\right]. $ \n\t\t\t\t\t\\end{enumerate}\n\t\t\t\t\t%    \\begin{align}\n\t\t\t\t\t\t%        \\nabla_{\\x} h_{\\eta}(\\x) = \\left(\\tfrac{n}{\\eta}\\right) \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\left[\\us{h(\\x+\\vv) \\tfrac{\\vv}{\\|\\vv\\|}}\\right] \\quad \\forall \\x \\in \\Xscr. \n\t\t\t\t\t\t%    \\end{align}\nSuppose $h \\in C^{0,0}(\\Xscr_{\\eta})$ with parameter $L_0$. For any $\\x, \\y \\in \\Xscr$,  the following hold. \n\t\t\t\t\t\n\t\t\t\t\t\\begin{enumerate}\n\t\t\t\t\t\t\\item[ (ii)] $| h_{\\eta}(\\x)-h_{\\eta}(\\y) | \\leq L_0 \\|\\x-\\y\\|.$  (iii) $| h_{\\eta}(\\x) - h(\\x)| \\leq L_0 \\eta.$ \n\t\t\t\t\t\t\n\t\t\t\t\t\t\\item[ (iv)] $\\| \\nabla_{\\x} h_{\\eta}(\\x) -\\nabla_{\\x} h_{\\eta}(\\y)\\| \\leq {\n\t\t\t\t\t\t\t\\tfrac{L_0 \\sqrt{n}}{\\eta}}\\|\\x-\\y\\|.$\n\t\t\t\t\t\t%    \\item[(v)] If $h$ is convex {and $h \\in C^{0,0}(\\Xscr_{\\eta})$ with parameter $L_0$}, then $h_{\\eta}$ is convex and satisfies the following for any $\\x \\in \\Xscr$. \n\t\t\t\t\t\t%        \\begin{align}\n\t\t\t\t\t\t\t%                h(\\x) \\leq h_{\\eta}(\\x) \\leq h(\\x) + \\eta L_0.\n\t\t\t\t\t\t\t%            \\end{align} \n\t\t\t\t\t\t% \\item[(vi)] $h_{\\eta} \\in C^{0,0}$.  \\us{Prove//cite} FY: This seems redundant. It is already shown in (ii).\n\t\t\t\t\t\t%  \\item[(iv)] If \\fy{$\\eta>0$, then $ h_{\\eta}(\\x)$ is continuously differentiable and} $\\nabla h_{\\eta}(\\x)$ is defined as $$ \\nabla h_{\\eta}(\\x) \\triangleq \\tfrac{1}{\\kappa} \\int_{\\fy{\\mathbb{R}^n}} \\tfrac{f(\\x+\\eta u) - f(\\x)}{\\eta} e^{-\\frac{\\|u\\|^2}{2}} Bu du. $$ \\fy{Moreover}, $h_{\\eta} \\in C^{1,1}$ and $L_1(h_{\\eta}) = \\tfrac{\\sqrt{n}}{\\eta} L_0(h).$  \n\t\t\t\t\t\t%    \\item[(vi)] If $h$ is convex {and $h \\in C^{0,0}(\\Xscr_{\\eta})$ with parameter $L_0$}, then $\\nabla_x h_{\\eta}(\\x) \\in \\partial_{\\epsilon} h(\\x)$ where {$\\epsilon \\triangleq \\eta L_0$}.\n\t\t\t\t\t\t\n\t\t\t\t\t\t%\\item[(vii)] If $h \\in C^{0,0}$ and $\\eta > 0$, then $h_{\\eta} \\in C^{1,1}$ and $L_1(f_{\\eta}) = \\tfrac{\\sqrt{n} L_0(f)}{\\eta}.$ \n\t\t\t\t\t\t%     \\us{Modify for spherical/Prove/Cite} FY: I think the version we use in the paper is already shown in (iv)\n\t\t\t\t\t\t\n\t\t\t\t\t\t%    \\item[(vii)] If {$h \\in C^{1,1}(\\Xscr_{\\eta})$} with constant {$L_1$}, then $$\\|\\nabla_x h_{\\eta}(\\x)-\\nabla_x h(\\x) \\| \\leq {\\eta L_1 n}.$$\n\t\t\t\t\t\t\n\t\t\t\t\t\t\\item [(v)] {Suppose $h \\in C^{0,0}(\\Xscr_{\\eta})$ with parameter $L_0$ {and} \n\t\t\t\t\t\t\t$g_{\\eta}(\\x,\\vv)  \\triangleq \\left(\\tfrac{n(h(\\x+\\vv) - h(\\x-\\vv))\\vv}{2\\eta\\|\\vv\\|}\\right)$\n\t\t\t\t\t\t\tfor $\\vv \\in \\eta\\mathbb{S}$.  Then for} any $\\x \\in \\Xscr$, we have that ${\\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}}}[{\\|{g_\\eta}(\\x,\\vv)\\|^2}] \\leq 16\\sqrt{2\\pi}L_0^2 n$. $\\hfill \\Box$ \n\t\t\t\t\t\\end{enumerate}\n\t\t\t\t\\end{lemma}","sha256_utf8":"e1f73b5f1c1018970c960013cb9e88a0faaba9399abbaf902074e9c3d243e542"},{"start":46292,"end":50502,"start_line":778,"end_line":845,"raw_tex":"\\begin{proof}\n\t\t\t\t\t{(i) By Stoke's theorem and spherical symmetry of $\\vv$, we have that \n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\nabla_{\\x} h_{\\eta}(\\x)& = \\tfrac{n}{\\eta} \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\ \\left[ \\tfrac{h(\\x+\\vv) \\vv }{\\|\\vv\\|} \\right] = \n\t\t\t\t\t\t\t\\tfrac{n}{2\\eta} \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\ \\left[ \\tfrac{(h(\\x+\\vv)+h(\\x+\\vv)) \\vv }{\\|\\vv\\|} \\right] \\\\\n\t\t\t\t\t\t\t& = \\tfrac{n}{2\\eta} \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\ \\left[ \\tfrac{(h(\\x+\\vv)) \\vv }{\\|\\vv\\|}+\\tfrac{(h(\\x+\\vv)) \\vv }{\\|\\vv\\|} \\right] = \n\t\t\t\t\t\t\t\\tfrac{n}{2\\eta} \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\ \\left[ \\tfrac{(h(\\x+\\vv)) \\vv }{\\|\\vv\\|}+\\tfrac{(h(\\x-\\vv)) (-\\vv) }{\\|\\vv\\|} \\right] \\\\\n\t\t\t\t\t\t\t& = \\tfrac{n}{2\\eta} \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\ \\left[ \\tfrac{((h(\\x+\\vv))-(h(\\x-\\vv)) \\vv) }{\\|\\vv\\|} \\right]. \n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t(ii,iii) follow akin to ~\\cite[Lemma 1]{CSY2021MPEC}. \n\t\t\t\t\t\t(iv)\nRecall from~\\cite[Prop. 2.2]{lin2022} that $\\|\n\\nabla_{\\x} h_{\\eta}(\\x) -\\nabla_{\\x}\nh_{\\eta}(\\y)\\| \\leq \\tfrac{C_n\nL_0}{\\eta}\\|\\x-\\y\\|,$ for $n\\geq 1$, where\n$C_n \\triangleq\n\\left(\\tfrac{2}{\\pi}\\right)^{p_n}\n\\tfrac{n!!}{(n-1)!!}$ and $p_n = 1$ if $n$\nis even, and $p_n =0$ otherwise. It suffices to show that $C_n \\le 1$ for $n \\ge 1$. For $n \\ge\n0$, let $w_n$ denote the Wallis'\nintegrals~\\cite{spivak2006calculus}, given as\n$w_n = \\int_{0}^{\\pi/2}\\sin^n(u)du$. Then we\nhave $w_n =\\left(\\tfrac{n-1}{n}\\right)w_{n-2}$ for $n \\geq\n2$ where $w_0=\\tfrac{\\pi}{2}$ and $w_1=1$\n(cf.~\\cite[Ch. 19]{spivak2006calculus}). This\nimplies (via mathematical induction) that\n$w_n =\n\\left(\\tfrac{\\pi}{2}\\right)\\tfrac{(n-1)!!}{n!!}$\nfor even $n$, while $w_n =\n\\tfrac{(n-1)!!}{n!!}$ for odd $n$. Let us\ndefine $v_n \\,\\triangleq \\, (n+1)w_n w_{n+1}$ for $n\\geq 0$.\nWe obtain $v_n =\n(n+1)\\left(\\tfrac{\\pi}{2}\\right)\n\\tfrac{(n-1)!!}{n!!}\\tfrac{n!!}{(n+1)!!} =\n\\tfrac{\\pi}{2}$, implying that $v_n =\n\\tfrac{\\pi}{2}$ for all $n\\geq 0$. Recall\nthat from~\\cite{spivak2006calculus} we have\n$w_{n+1} < w_n$ for all $n\\geq 0$. Thus we\nobtain $\\tfrac{\\pi}{2} = v_n  =(n+1)w_n\nw_{n+1} < (n+1)w_n^2$ for all $n\\geq 0$, i.e., $w_n < \\tfrac{\\pi}{2(n+1)}$ for $n \\ge 0$.  Note\nthat $C_1 =1$. Suppose $n \\geq 2$. We have\n$C_n= \\tfrac{1}{w_n} <\n\\sqrt{\\tfrac{2(n+1)}{\\pi}} =\n\\sqrt{\\tfrac{2(1+1/n)}{\\pi}} \\sqrt{n} \\leq\n\\sqrt{\\tfrac{2(1+1/2)}{\\pi}} \\sqrt{n} \\leq\n0.98 \\sqrt{n} $. This implies that $C_n  \\leq\n1$ for all $n\\geq 1$.} {(v) We have that}\n\\begin{align*}\n\t\t\t\t\t\t\t\\mathbb{E}\\left[ \\|g_{\\eta}(\\x,\\vv)\\|^2 \\right] & \\le \n\t\t\t\t\t\t\t\\tfrac{n^2}{4\\eta^2} \\mathbb{E}\\left[ \\|\\tfrac{(h(\\x+\\vv)-h(\\x-\\vv))\\vv}{\\|\\vv\\|}\\|^2 \\right] \\le \n\t\t\t\t\t\t\t\\tfrac{n^2}{4\\eta^2} \\mathbb{E}\\left[  (h(\\x+\\vv)-h(\\x-\\vv))^2 \\tfrac{\\|\\vv\\|^2}{\\|\\vv\\|^2} \\right] \\\\\n\t\t\t\t\t\t\t& \\le \n\t\t\t\t\t\t\t\\tfrac{n^2}{4\\eta^2} \\mathbb{E}\\left[ \\left(h(\\x+\\vv)-\\mathbb{E}\\left[h(\\x+\\vv)\\right] + \\mathbb{E}\\left[ h(\\x+\\vv)\\right]-h(\\x-\\vv)\\right)^2  \\right]. \n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\tBut $\\mathbb{E}\\left[ h(\\x+\\vv)\\right] = \\mathbb{E}\\left[ h(\\x-\\vv)\\right]$ by spherical symmetry. Consequently, \n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\mathbb{E}\\left[ \\|g_{\\eta}(\\x,\\vv)\\|^2 \\right] & \\le \n\t\t\t\t\t\t\t\\tfrac{n^2}{4\\eta^2} \\mathbb{E}\\left[  \\left(h(\\x+\\vv)-\\mathbb{E}\\left[h(\\x+\\vv)\\right] + \\mathbb{E}\\left[ h(\\x-\\vv)\\right]-h(\\x-\\vv)\\right)^2  \\right] \\\\\n\t\t\t\t\t\t\t& \\le \\tfrac{2n^2}{4\\eta^2} \\mathbb{E}\\left[ \\left|(h(\\x+\\vv)-\\mathbb{E}\\left[h(\\x+\\vv)\\right]\\right|^2  + \\left|\\mathbb{E}\\left[ h(\\x-\\vv)\\right]-h(\\x-\\vv))\\right|^2 \\right]. \n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\tThen by invoking L\\'{e}vy's inequality (cf. Lemma~\\ref{lemma:levy-concentration}), we have that \n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\mathbb{E}\\left[ |(h(\\x+\\vv)-\\mathbb{E}\\left[h(\\x+\\vv)\\right]|^2 \\right] \\le 2\\sqrt{2\\pi}\\int_0^{\\infty} e^{-\\alpha n}{8 \\eta^2 L_0^2} d\\alpha = \\tfrac{16\\sqrt{2\\pi} \\eta^2 L_0^2}{n}. \n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t{By noting that $\\mathbb{E}\\left[ \\left|(h(\\x+\\vv)-\\mathbb{E}\\left[h(\\x+\\vv)\\right]\\right|^2  \\right] = \\mathbb{E}\\left[  \\left|\\mathbb{E}\\left[ h(\\x-\\vv)\\right]-h(\\x-\\vv))\\right|^2 \\right]$ and leveraging the preceding bound, we get}  \n\t\t\t\t\t\t$\\mathbb{E}\\left[ \\|g_{\\eta}(\\x,\\vv)\\|^2 \\right] \\le \\tfrac{n^2}{\\eta^2} \\tfrac{16\\sqrt{2\\pi} \\eta^2 L_0^2}{n} = 16\\sqrt{2\\pi}L_0^2 n.$\n\t\t\t\t\t\n\t\t\t\t\\end{proof}","sha256_utf8":"80392213ea0a702560c6388da5cd2af2aa4d55b51c802fd206d726835d1a2c03"},{"start":54721,"end":54998,"start_line":885,"end_line":888,"raw_tex":"\\begin{lemma}[{\\cite[Lemma 6]{CSY2021MPEC}}]\\label{lem:inexact_proj_2}\\em\n\t\t\t\t\tFor any $\\beta,\\eta>0, \\x \\in \\Xscr, \\tilde{e} \\in\\mathbb{R}^n$, \n\t\t\t\t\t$$\\|G_{\\eta,\\beta}(\\x) \\|^2   \\leq  {2} \\| \\tilde{G}_{\\eta,\\beta}(\\x, {\\tilde e}) \\|^2 + 2 \\|\\tilde{e}\\|^2.$$  \n\t\t\t\t\\end{lemma}","sha256_utf8":"e4894eccb50bb709a38b9feaffc0b62c2009c307603b2366c04b5ef1859b3590"},{"start":55114,"end":55344,"start_line":890,"end_line":891,"raw_tex":"\\begin{lemma}\\label{lemma:mon_G} \\em Suppose $\\eta > 0$ and $\\{\\gamma_k\\}$ is a diminishing sequence. Then, for any $\\x$ and $k \\geq 0$, $\\| G_{\\eta,1/\\gamma_k} (\\x)\\| \\, \\leq \\,\\| G_{\\eta,1/\\gamma_{k+1}} (\\x)\\|$. \n\t\t\t\t\\end{lemma}","sha256_utf8":"ba926915bebdfbeefdaa6147f7ee4394f4945b31b8cb7b9cf123b445b38a3ca4"},{"start":61904,"end":62402,"start_line":972,"end_line":977,"raw_tex":"\\begin{lemma}[{\\bf Bias and moment properties of ${\\bf e}$}] \\label{lem:stoch_error_var}\\em\n\t\t\t\t\t%Consider Definition \\ref{def:stoch_errors}. \n\t\t\t\t\tLet Assumptions \\ref{ass-1} and \\ref{assum:random_vars} hold. Then (i) and (ii) hold a.s. for $k\\geq 0$, $N_k \\geq 1$, and $j = 1, \\hdots, N_k$.\n\t\t\t\t\t\n\t\t\t\t\t\\noindent (i) $\\mathbb{E}[\\mathbf{e}_{j,k}\\mid \\mathcal{F}_k] = 0$;  \\noindent (ii) $\\mathbb{E}[\\|\\mathbf{e}_{k}\\|^2\\mid \\mathcal{F}_k] \\leq  \\tfrac{16\\sqrt{2\\pi} L_0^2 n}{N_k}$.\n\t\t\t\t\\end{lemma}","sha256_utf8":"570b7129d6998a6ad43c4950287c260e351b07e9ace0c7206f45b69ee6b77578"},{"start":62407,"end":63780,"start_line":978,"end_line":994,"raw_tex":"\\begin{proof}\n\t\t\t\t\t\t(i) To show the conditional unbiasedness of $\\mathbf{e}_{j,k}$, we proceed as follows.\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\mathbb{E}_{\\bxi,\\! \\vv}\\left[ {g_{\\eta}(\\x_k,\\!\\vv_{j,k},\\!\\bxi_{j,k})}  \\mid {\\cal F}_k\\right]\\! & =\\mathbb{E}_{\\vv}\\!   \n\t\t\t\t\t\t\t\\left[\\mathbb{E}_{\\bxi}\\left[ \\tfrac{n(\\f(\\x_k+v_{j,k},\\xi_{j,k})-\\f(\\x_k-v_{j,k},\\xi_{j,k}))v_{j,k}}{2\\eta \\|v_{j,k}\\|} \\hspace{-0.05in} \\mid {\\cal F}_k \\cup \\{v_{j,k}\\} \\right] \\right] \\\\\n\t\t\t\t\t\t\t&\\hspace{-1in}  \\overset{\\tiny \\mbox{Lemma~\\ref{lemma:props_local_smoothing}~{(i)}}}{=}\n\t\t\t\t\t\t\t\\mathbb{E}_{\\vv}\\left[ \\tfrac{n}{2\\eta}\\tfrac{(f(\\x_k+v_{j,k})-f(\\x_k\\lm{-}v_{j,k}))v_{j,k}}{\\|v_{j,k}\\|} \\mid {\\cal F}_k \\right]  = \\nabla_{\\x} f_{\\eta}(\\x_k).    \n\t\t\t\t\t\t\\end{align*} \n\t\t\t\t\t\tBy leveraging Lemma~\\ref{lemma:props_local_smoothing}~{(v)}, we have that \n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\mathbb{E}_{{\\bxi, \\vv}}\\left[\\, \\|\\mathbf{e}_{j,k}\\|^2 \\mid {\\cal F}_k \\, \\right] &= \n\t\t\t\t\t\t\t\\mathbb{E}_{{\\bxi}}\\left[\\, \\mathbb{E}_{ {\\vv}}\\left[\\, \\|\\mathbf{e}_{j,k}\\|^2  \\, \\mid{\\cal F}_{k}\\cup\\{\\xi_{j,k}\\} \\right] \\right]\\\\\n\t\t\t\t\t\t\t& \\le \n\t\t\t\t\t\t\t\\mathbb{E}_{{\\bxi}}\\left[\\, 16\\sqrt{2\\pi}\\tilde{L}_0(\\bxi_{{j,k}})^2n \\right] \\le 16\\sqrt{2\\pi} L_0^2 n. \n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\tIt follows that $\\mathbb{E}\\left[\\, \\|\\mathbf{e}_{k}\\|^2  \\, \\mid {\\cal F}_k\\right] \\le \\tfrac{16\\sqrt{2\\pi} L_0^2 n}{N_k}.$\n\t\t\t\t\\end{proof}","sha256_utf8":"c3e08b02f9ec547a0185430436b422913e571c5c632652be2a4f883580be4613"},{"start":63960,"end":65430,"start_line":997,"end_line":1016,"raw_tex":"\\begin{lemma}\\label{lemma:zog_ineq}\\em \n\t\t\t\t\tLet Assumption \\ref{ass-1} hold {and suppose} $\\left\\{\\x_k\\right\\}$ is generated by\n\t\t\t\t\tAlgorithm~\\ref{algorithm:zo_nonconvex}. Let $\\{\\gamma_k\\}$ be a\n\t\t\t\t\tnon-increasing stepsize sequence (e.g., constant or diminishing) where\n\t\t\t\t\t$\\gamma_0 \\in (0,{\\tfrac{\\eta}{{\\sqrt{n}}L_0}})$ for $\\eta>0$. Then  the\n\t\t\t\t\tfollowing hold. \n\t\t\t\t\t\n\t\t\t\t\t\\noindent (i) For $k\\geq 0$, we have \n\t\t\t\t\t\\begin{align}\\label{eqn:recursive_ineq_lemma5} \n\t\t\t\t\t\t\\left( 1-\\tfrac{{\\sqrt{n}}L_0\\gamma_k}{\\eta}\\right)\\tfrac{\\gamma_k \\|G_{\\eta,1/\\gamma_0} (\\x_k)\\|^2}{4}     &   \\leq   f_{\\eta}(\\x_k)-      f_{\\eta}(\\x_{k+1}) \n\t\t\t\t\t\t+{\\left( 1-\\tfrac{{\\sqrt{n}}L_0\\gamma_k}{2\\eta}\\right)} \\gamma_k \\|e_k\\|^2.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\t\n\t\t\t\t\t\\noindent (ii) Let $R_{\\ell,K}$ be a random integer taking values in $\\{\\ell, \\ldots, K-1\\}$ with mass function $\\mathbb{P}{[} R_{\\ell,K} = j {]} = \\tfrac{\\gamma_j}{\\sum_{i=\\ell}^{K-1} \\gamma_i}$ for any $j \\in \\{\\ell, \\ldots, K-1\\}$.  Then, for all $k\\geq 0$ we have \n\t\t\t\t\t\\begin{align}\\label{ineq:residual_map_ineq_VRGZO_2}\n\t\t\t\t\t\t\\mathbb{E}\\left[ \\|G_{\\eta,{1/\\gamma_0}} (\\x_{R_{\\ell,K}})\\|^2\\right]\n\t\t\t\t\t\t&\\leq \n\t\t\t\t\t\t\\tfrac{\\left( \\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left({16 \\sqrt{2\\pi} L_0^2{n} }\\right)\\sum_{k=\\ell}^{K-1} \\tfrac{\\gamma_k}{N_k}\\right)}{\\left(\\left( \\tfrac{1}{4}-\\tfrac{{\\sqrt{n}} L_0\\gamma_0}{4\\eta}\\right) \\sum_{k=\\ell}^{K-1} \\gamma_k\\right)} .\n\t\t\t\t\t\\end{align}\n\t\t\t\t\\end{lemma}","sha256_utf8":"7112fa466645f204c43a065146dc4f7b3ba92981a9ab9f25340f3b435d63adc9"},{"start":65440,"end":70450,"start_line":1018,"end_line":1076,"raw_tex":"\\begin{proof}\n\t\t\t\t\t(i) From Lemma~\\ref{lemma:props_local_smoothing} (iv), the gradient mapping $\\nabla f_\\eta$ is Lipschitz continuous with the parameter $L_{\\eta}\\triangleq \\tfrac{\\sqrt{n}L_0}{\\eta}$. From the descent lemma, \n\t\t\t\t\t\\begin{align}\\label{eqn:descent_lem_proof_eq1}\n\t\t\t\t\t\\notag\tf_{\\eta}(\\x_{k+1}) & \\leq f_{\\eta}(\\x_{k}) + \\nabla f_{\\eta}(\\x_k)^\\top(\\x_{k+1}-\\x_k)\n\t\t\t\t\t\t+\\tfrac{L_{\\eta}}{2}\\|\\x_{k+1}-\\x_k\\|^2\\\\\n\t\t\t\t\t\\notag\t&=f_\\eta(\\x_k) + \\left(\\nabla f_{\\eta}(\\x_k)+{e_k}\\right)^\\top(\\x_{k+1}-\\x_k)\\\\\n\t\t& -{e_k}^\\top(\\x_{k+1}-\\x_k)+\\tfrac{L_{\\eta}}{2}\\|\\x_{k+1}-\\x_k\\|^2.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\tInvoking %Lemma \\ref{eqn:proof_comapct_eq1}, \n\t\t\t\t\t\tthe properties of the Euclidean projection, we have \\\\$ \n\t\t\t\t\t(\\x_k-\\gamma_k\\left(\\nabla f_\\eta(\\x_k)+e_k\\right)-\\x_{k+1})^\\top(\\x_k-\\x_{k+1})\\leq 0$. This implies that \n\t\t\t\t\t\\begin{align}\\label{eqn:descent_lem_proof_eq2}\n\t\t\t\t\t\t\\left(\\nabla f_\\eta(\\x_k)+{e_k}\\right)^\\top(\\x_{k+1}-x_k)\n\t\t\t\t\t\t\\leq -\\tfrac{1}{\\gamma_k}\\|\\x_{k+1}-\\x_k\\|^2.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\tIn addition, we may also express $e_k^{\\top}(\\x_{k+1}-\\x_k)$ as follows. \n\t\t\t\t\t\\begin{align}\\label{eqn:descent_lem_proof_eq3}\n\t\t\t\t\t\t-e_k^{\\top}(\\x_{k+1}-\\x_k)\n\t\t\t\t\t\t\\leq \\tfrac{{\\gamma_k}}{2}\\|e_k\\|^2+\\tfrac{1}{2{\\gamma_k}}\\|\\x_{k+1}-\\x_k\\|^2.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\tCombining the inequalities \\eqref{eqn:descent_lem_proof_eq1}, \\eqref{eqn:descent_lem_proof_eq2}, and \\eqref{eqn:descent_lem_proof_eq3} we obtain \n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\tf_{\\eta}(\\x_{k+1})  & \\leq   f_{\\eta}(\\x_k)+ \\left( -\\tfrac{1}{2\\gamma_k}+\\tfrac{L_{\\eta}}{2}\\right) \\|\\x_{k+1}-\\x_k\\|^2 +\\tfrac{\\gamma_k}{2} \\|e_k\\|^2 \\\\\n\t& \\overset{\\tiny \\mbox{Def.~\\ref{def:res_maps}}}{=} f_{\\eta}(\\x_k)- \\left( \\tfrac{1}{2\\gamma_k}-\\tfrac{L_{\\eta}}{2}\\right)\\gamma_k^2 \\left\\|\\tilde{G}_{\\eta,1/\\gamma_k}(\\x_k, {e_k})\\right\\|^2+\\tfrac{\\gamma_k}{2} \\|e_k\\|^2\\\\\n\t& = f_{\\eta}(\\x_k)- \\left( 1-L_{\\eta}\\gamma_k\\right)\\tfrac{\\gamma_k}{2} \\left\\|\\tilde{G}_{\\eta,1/\\gamma_k}(\\x_k, {e_k})\\right\\|^2+\\tfrac{\\gamma_k}{2} \\|e_k\\|^2\\\\\n\t& \\hspace{-0.2in}\\overset{\\tiny \\mbox{Lemma~\\ref{lem:inexact_proj_2}}}{\\le} f_{\\eta}(\\x_k)- \\left( 1-L_\\eta \\gamma_k\\right)\\tfrac{\\gamma_k}{4} \\left\\|G_{\\eta,1/\\gamma_k}(\\x_k)\\right\\|^2+\\left(\\tfrac{\\gamma_k}{2} + \\left( 1-L_{\\eta} \\gamma_k\\right)\\tfrac{\\gamma_k}{2}\\right) \\|e_k\\|^2,\n\t\t\t\t\t\\end{align*}\nwhere $1-L_{\\eta}\\gamma_k > 0$ for any $k$  since $\\gamma_0 < \\tfrac{1}{L_{\\eta}}$. \n\t\t\t\t\tFrom the preceding relation and invoking Lemma~\\ref{lemma:mon_G}, we obtain~\\eqref{eqn:recursive_ineq_lemma5}, i.e.,\n\\begin{align*}\n\\left( 1-L_\\eta \\gamma_k\\right)\\tfrac{\\gamma_k}{4} \\left\\|G_{\\eta,1/\\gamma_0}(\\x_k)\\right\\|^2 \\, \\le \\, f_{\\eta}(\\x_k)- f_{\\eta}(\\x_{k+1}) + \\left(1-\\tfrac{L_{\\eta} \\gamma_k}{2}\\right)\\gamma_k \\|e_k\\|^2.\n\\end{align*} \n\t\t\t\t\t\n\t\t\t\t\t\\noindent (ii) Summing \\eqref{eqn:recursive_ineq_lemma5}  from $k =\\ell, \\ldots, K-1$ where $\\ell\\triangleq  \\lceil \\lambda K\\rceil$ and taking expectations with respect to the iterate trajectory, we have\n\t\t\t\t\t\\begin{align}\n\t\t\t\t\t\t\\notag \\sum_{{k=\\ell}}^{K-1} \\left( 1-\\tfrac{ \\sqrt{n} L_0\\gamma_k}{\\eta}\\right) \\tfrac{\\gamma_k}{4}\\EXP{\\, \\|G_{\\eta,1/\\gamma_k} (\\x_k)\\|^2\\, }  \n\t\t\t\t\t\t& \\leq   \\EXP{\\, f_{\\eta}(\\x_\\ell)\\, }-      \\EXP{\\, f_{\\eta}(\\x_{K})\\, }\\\\ \n\t\t\t\t\t\t&+\\sum_{{k=\\ell}}^{K-1}\\left( 1-\\tfrac{\\sqrt{n} L_0\\gamma_k}{2\\eta}\\right) \\gamma_k \\EXP{\\, \\|\\mathbf{e}_k\\|^2\\, }.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\tInvoking the definition of $R_{\\ell,K}$ and Lemma~\\ref{lemma:mon_G}, we have \n\t\t\t\t\t\\begin{align} \n\t\t\t\t\t\t\\notag & \\quad \\left( \\tfrac{1}{4}-\\tfrac{\\sqrt{n} L_0\\gamma_0}{4\\eta}\\right) \\left(\\sum_{k=\\ell}^{K-1} \\gamma_k\\right)\\EXP{\\mathbb{E}_{R_{\\ell,K}}\\left[\\|G_{\\eta,1/\\gamma_0} ({\\x_{R_{\\ell,K}}})\\|^2\\right]}\\\\  \n\t\t\t\t\t\t& \\notag \\leq   \\left( \\tfrac{1}{4}-\\tfrac{\\sqrt{n} L_0\\gamma_0}{4\\eta}\\right)\\sum_{{k=\\ell}}^{K-1} \\gamma_k \\mathbb{E}\\left[\\|G_{\\eta,1/\\gamma_0} ({\\x_k})\\|^2\\right]   \\leq\t\t\t\t\t\t   \\EXP{f_{\\eta}(\\x_\\ell)-f^*_{\\eta}}+\\sum_{{k=\\ell}}^{K-1} \\gamma_k \\EXP{\\|\\mathbf{e}_k\\|^2}, \n\t\t\t\t\t\\end{align}\n\t\t\t\t\twhere the last inequality follows from $\\left( 1-\\tfrac{\\sqrt{n} L_0\\gamma_k}{2\\eta}\\right) \\leq 1$ for any $k$. \n\t\t\t\t\t{Note that we have} \n\t\t\t\t\t\\begin{align}\\notag\n\t\t\t\t\t\t\\mathbb{E}\\left[f_{\\eta}(\\x_{\\ell})\\right] - {f^*_{\\eta}} & =   \\mathbb{E}\\left[ f(\\x_{\\ell}) + f_{\\eta}(\\x_{\\ell})-f(\\x_{\\ell})\\right]-f^*_{\\eta} +f^*\n\t\t\t\t\t\t-f^*\\\\\n\t\t\t\t\t\t\\notag & = \\mathbb{E}\\left[{f(\\x_{\\ell})}\\right]-f^*+ \\mathbb{E}\\left[\\left | f_{\\eta}(\\x_{\\ell})-f(\\x_{\\ell})\\right|\\right] + \\left| f^*-f^*_\\eta\\right| \\\\\n\t\t\t\t\t\t& \\stackrel{\\tiny{\\hbox{Lemma~\\ref{lemma:props_local_smoothing} (iii)}}}{\\leq}\n\t\t\t\t\t\t\\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta.\n\t\t\t\t\t\t\\label{bd_feta}\n\t\t\t\t\t\\end{align}\n\t\t\t\t\tFrom Lemma \\ref{lem:stoch_error_var} (ii), \n\t\t\t\t\t$\\mathbb{E}\\left[\\|\\mathbf{e}_k\\|^2\\right] \\leq \\mathbb{E}\\left[\\mathbb{E}\\left[\\|\\mathbf{e}_k\\|^2\\mid \\mathcal{F}_k\\right]\\right] \n\t\t\t\t\t\\leq \\tfrac{ 16\\sqrt{2\\pi}  L_0^2n}{N_k}.$\n\t\t\t\t\tFrom the preceding relations, we obtain the inequality \\eqref{ineq:residual_map_ineq_VRGZO_2}.\n\t\t\t\t\\end{proof}","sha256_utf8":"0030f8527fb82bbfa0eca4ff7597b105f9ba734be6891340692eaa1a0474eba5"},{"start":79636,"end":84387,"start_line":1165,"end_line":1218,"raw_tex":"\\begin{proof}\n\t\t\t\t\t\\noindent {\\bf (a-i)} From~\\eqref{ineq:residual_map_ineq_VRGZO_2}, we obtain\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\mathbb{E}\\left[ \\|G_{\\eta,{1/\\gamma}} ({\\x_{R_{\\ell,K}}})\\|^2\\right]\n\t\t\t\t\t\t&\\leq  \n\t\t\t\t\t\t\\tfrac{\\left( \\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left(16 \\sqrt{2\\pi} L_0^2{n} \\right)\\gamma\\sum_{k=\\ell}^{K-1} \\tfrac{1}{N_k}\\right)}{\\left(\\left( 1-\\tfrac{\\sqrt{n}L_0\\gamma}{\\eta}\\right) \\tfrac{\\gamma}{4}(K-\\ell)\\right)} \\\\\n\t\t\t\t\t\t& \\leq \\tfrac{\\left( \\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left({16 \\sqrt{2\\pi} L_0^2 n}\\right){\\gamma}\\sum_{k=\\ell}^{K-1} \\tfrac{a^{-1}}{{L_0}{{\\sqrt{n}}{(k+1)}}}\\right)}{\\left(\\left( 1-\\tfrac{\\sqrt{n}L_0\\gamma}{\\eta}\\right) \\tfrac{\\gamma}{4}(K-\\ell)\\right)},  \n\t\t\t\t\t\\end{align*}\n\t\t\t\t\twhere the expectation is with respect to both the iterate trajectory and the random integer $R_{\\ell,K}$, while the last inequality  follows from the definition of $N_k$. Recall that $K> \\tfrac{2}{1-\\lambda}$ implies $\\ell \\leq K-1$ and  \n\t\t\t\t\t$ \\textstyle\\sum_{k={\\ell}}^{K-1}\\tfrac{1}{k+1}%\\leq  \\tfrac{1}{\\ell+1}+\\ln\\left( \\tfrac{K}{\\ell+1}\\right) \n\t\t\t\t\t\\leq 0.5 +\\ln\\left(\\tfrac{N}{\\lambda N+1}\\right)\\leq 0.5-\\ln(\\lambda).$\n\t\t\t\t\t{Further}, $K-\\ell \\geq K-\\lambda K=(1-\\lambda)K$ {implying that} \\eqref{eqn:GBound_constant_gamma} holds.\n\t\t\t\t\t%\\begin{align*}\n\t\t\t\t\t%    \\mathbb{E}\\left[ \\|G_{\\eta,1/\\gamma} ({\\x_{R_{\\ell,K}}})\\|^2\\right] \\leq \\tfrac{\\left(\\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left(\\uss{16 \\farz{\\sqrt{n}} \\sqrt{2\\pi} L_0}\\right)  (0.5-\\ln(\\lambda)){\\lm{a^{-1}}}\\gamma\\right)}{\\left(\\left( 1-\\tfrac{\\farz{\\sqrt{n}}L_0\\uss{\\gamma}}{\\eta}\\right) \\tfrac{\\gamma}{4}(1-\\lambda)K\\right)}.\n\t\t\t\t\t%\\end{align*}\n\t\t\t\t\t\n\t\t\t\t\t\\noindent {\\bf (a-ii)} To show (ii-1), using the relation in part (i) and by $\\gamma:= \\tfrac{\\eta}{2\\sqrt{n}L_0}$, we obtain\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t& \\mathbb{E}\\left[ \\|G_{\\eta,1/\\gamma} (\\x_{R_{\\ell,K}})\\|^2\\right] \\leq  \n\t\t\t\t\t\t\\tfrac{\\left(\\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left(8\\eta \\sqrt{2\\pi} \\right)  (0.5-\\ln(\\lambda)){a^{-1}}\\right)}{\\left(\\eta(1-\\lambda)K\\right)/(16\\sqrt{n}L_0)}.\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tFrom Jensen's inequality,  $\\mathbb{E}\\left[ \\|G_{\\eta,1/\\gamma} (\\x_{R_{\\ell,K}})\\|\\right] \\leq \\sqrt{\\mathcal{O}\\left({n^{1/2}}{(L_0\\eta^{-1}+L_0^2)}/K\\right)}$, implying that $K_{\\epsilon}=\\mathcal{O}\\left({n^{1/2}}{(L_0\\eta^{-1} +L_0^2)} \\epsilon^{-2}\\right)$. Next, we derive sample complexity as per (ii-2). %The sample complexity of {function evaluations} is as follows.\n\t\t\t\t\t\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\sum_{k=0}^{K_\\epsilon} N_k & = \\sum_{k=0}^{K_\\epsilon}  \\lceil  {a{\\sqrt{n} L_0}(k+1)}\\rceil \n\t\t\t\t\t\t\\leq {\\sqrt{n} L_0}  {\\mathcal{O}(K_{\\epsilon}+{K_\\epsilon^2})}\n\t\t\t\t\t\t\\leq \\mathcal{O}\\left({n^{3/2}(L_0^3\\eta^{-2}+L_0^5)} \\epsilon^{-4}\\right). %\\uss{ \\mbox{CHECK...exponents of $n,L_0$}} \\\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\t%\\farz{{\\bf TODO:} I don't think $\\eta^{-2}$ will appear in the complexity result here. Please see my earlier comment on (a-ii-1).  }\n\t\t\t\t\t\n\t\t\t\t\t\\noindent {\\bf (b-i)}\n\t\t\t\t\t{Consider the inequality~\\eqref{ineq:residual_map_ineq_VRGZO_2}.  Using $\\tfrac{\\gamma_k}{N_k}\\leq \\tfrac{\\gamma_0}{{a{\\sqrt{n}} L_0}(k+1)}$, we obtain} \n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\mathbb{E}\\left[ \\|G_{\\eta,{1/\\gamma_0}} (\\x_{R_{\\ell,K}})\\|^2\\right]\n\t\t\t\t\t\t& \\leq \\tfrac{\\left( \\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left({16  \\sqrt{2\\pi} L_0^2{n}} \\right)\\sum_{k=\\ell}^{K-1} \\tfrac{\\gamma_0}{{a \\sqrt{n} L_0}(k+1)}\\right)}{\\left(\\left( 1-\\tfrac{{\\sqrt{n}} L_0\\gamma_0}{\\eta}\\right) {\\tfrac{\\gamma_0}{2}}(1-\\sqrt{\\lambda})\\sqrt{K}\\right)},\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\twhere we note that $\\textstyle\\sum_{k=\\ell}^{K-1} \\gamma_k \\geq \\textstyle\\int_{\\ell-1}^{K-1} \\tfrac{\\gamma_0}{\\sqrt{x+1}} dx = 2\\gamma_0(\\sqrt{K}-\\sqrt{\\ell}) \\geq 2\\gamma_0(1-\\sqrt{\\lambda})\\sqrt{K}.$ As earlier, we have that  \n\t\t\t\t\t$ \\textstyle\\sum_{k={\\ell}}^{K-1}\\tfrac{1}{k+1} \\leq 0.5-\\ln(\\lambda),$ leading to the inequality in (b-i). \n\t\t\t\t\t\n\t\t\t\t\t\\noindent {\\bf (b-ii)} From Jensen's\n\t\t\t\t\tinequality and by setting $\\gamma_0 =\n\t\t\t\t\t\\tfrac{\\eta}{2 {\\sqrt{n}} L_0}$, we obtain\n\t\t\t\t\tthat \\\\$ \\mathbb{E}\\left[\n\t\t\t\t\t\\|G_{\\eta,1/\\gamma_0}\n\t\t\t\t\t(\\x_{R_{\\ell,K}})\\|\\right] \\leq\n\t\t\t\t\t\\sqrt{\\mathcal{O}\\left({(L_0\\eta^{-1}+L_0^2)\\sqrt{n/K}}\\right)}$, thus\n\t\t\t\t\t$K_{\\epsilon}=\\mathcal{O}\\left(\\tfrac{n(L_0^2\\eta^{-2} +L_0^4)}{\\epsilon^{4}}\\right)$.\\\\ Next, (b-ii-2) holds by\n\t\t\t\t\tbounding the sample complexity as follows.\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\sum_{k=0}^{K_\\epsilon} N_k & = \\sum_{k=0}^{K_\\epsilon}  \\lceil  {a\\sqrt{n} L_0}\\sqrt{k+1}\\rceil \n\t\t\t\t\t\t\\leq {\\sqrt{n} L_0}  {\\mathcal{O}(K_{\\epsilon}+{K_\\epsilon^{3/2}})}\n\t\t\t\t\t\t\\leq \\mathcal{O}\\left({n^{2}(L_0^4\\eta^{-3}+L_0^7)} \\epsilon^{-6}\\right).\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\\end{proof}","sha256_utf8":"0c2470117038a6cea7e259648ff1c1913a971d8750a00933c757f4c2790d23e8"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"随机约束优化问题","source_units":[{"start":14959,"end":15101,"start_line":420,"end_line":422,"raw_tex":"\\begin{align}\\label{eqn:prob}\n\t\\min_{\\x \\, \\in \\, \\Xscr} & \\quad f(\\x) \\, \\triangleq \\, \\mathbb{E}\\left[\\, \\f(\\x,\\bxi)\\, \\right], \n\\end{align}","sha256_utf8":"12d57532dfdff5f60a53da5bacfe00cde1520f47de2c074b06c3e4502da07c09"},{"start":15102,"end":15974,"start_line":423,"end_line":433,"raw_tex":"where $f:\\mathbb{R}^n\\to \\mathbb{R}$ is a real-valued, nonsmooth, and nonconvex\nfunction, $\\Xscr \\subseteq \\mathbb{R}^{n} $ is a closed and convex set, and $\\bxi:\\Omega \\to \\mathbb{R}^d$ denotes\na random variable associated with the probability space $(\\Omega,\n\\mathcal{F},\\mathbb{P})$. {Our interest lies in the case when $f$ is\n\tLipschitz continuous and at any $\\x \\in \\mathbb{R}^n$,  $f(\\x)\\triangleq\n\t\\mathbb{E}[{\\f}(\\x,\\bxi)]$ and $\\bxi: \\Omega \\mapsto \\Xi$ is a $d$-dimensional\n\trandom variable with $\\Xi \\, \\subseteq \\, \\Real^d$. We denote a realization of\n\t$\\bxi$ and $\\f({\\x},\\bxi)$ by $\\xi$ and $\\f({\\x},\\xi)$, respectively}. Recall that\nfunction $f$ is $L_0$-Lipschitz continuous on the set $\\Xscr$ if there exists a\nscalar $L_0>0$ such that for all $\\x,\\y \\in \\Xscr$ we have $|f(\\x)-f(\\y)| \\leq\nL_0\\|\\x-\\y\\|$. Our goal lies in devising randomized zeroth-order\n","sha256_utf8":"fd24de3719ee02c8c7f3642c01a91f885388c542d668c7af099694a70b8ca703"}]},{"id":"S002","semantic_type":"notation","kind":"notation","title":"最优值与投影记号","source_units":[{"start":37269,"end":38254,"start_line":675,"end_line":686,"raw_tex":"\t\t\t\t\\smallskip {\\bf Notation.} We use $\\x$, $\\x^\\top$, and $\\|\\x\\|$ to denote a column vector,\n\t\t\t\tits transpose, and its Euclidean norm, respectively. We\n\t\t\t\tdefine $f^*\\triangleq {\\displaystyle \\inf_{\\x \\in \\Xscr}} f(\\x)$ and $f^*_\\eta\\triangleq\n\t\t\t\t{\\displaystyle \\inf_{\\x \\in \\Xscr}} f_\\eta(\\x)$, where $f_\\eta(\\x)$ denotes the smoothed\n\t\t\t\tapproximation of $f$. Given a continuous function, i.e., $f \\in C^{0}$,\n\t\t\t\twe write $f \\in C^{0,0}(\\Xscr)$ if $f$ is Lipschitz on \n\t\t\t\t$\\Xscr$ with parameter $L_0$. Given a continuously differentiable\n\t\t\t\tfunction, i.e., $f \\in C^{1}$, we write $f \\in C^{1,1}(\\Xscr)$ if $\\nabla f$ is\n\t\t\t\tLipschitz continuous on $\\Xscr$ with parameter $L_1$. We\n\t\t\t\twrite a.s. for ``almost surely” and {$\\mathbb{E}[h(\\x,\\bxi)]$} denotes the\n\t\t\t\texpectation {with respect to a random variable $\\bxi$}. %Given a scalar $u$, $[u]_+\\triangleq \\max\\{0,u\\}$. \n\t\t\t\t$\\Pi_{\\Xscr}[\\vv]$ denotes the Euclidean projection of $\\vv$ onto the closed convex set $\\Xscr$.\n","sha256_utf8":"ad12195a30f0bd1f8406bbc45dee78d4ac2b2edf9a42a7c46bf18c6085781949"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"球内平滑与球面","source_units":[{"start":41901,"end":42140,"start_line":730,"end_line":730,"raw_tex":"\t\t\t\twe address nonsmoothness by considering a locally randomized smoothing technique. Given a function $h:\\mathbb{R}^n \\to \\mathbb{R}$ and a scalar $\\eta>0$, a spherically smoothed approximation of $h$, denoted by $h_\\eta$, is defined as \n","sha256_utf8":"1d5c202cbe71b1754cfa5f158d3731309c8cd84ccc267722838da81f8b0cb0a5"},{"start":42144,"end":42718,"start_line":731,"end_line":736,"raw_tex":"\\begin{align}\\label{def-smooth}\n\t\t\t\t\th_\\eta (\\x) \\, \\triangleq \\, \\mathbb{E}_{\\uu \\in \\mathbb{B}}\\left[\\, h(\\x+\\eta \\uu)\\, \\right],\n\t\t\t\t\\end{align} where {$\\uu$ denotes a random variable uniformly distributed on a unit ball $\\mathbb{B}$, defined as  $\\mathbb{B}\\triangleq \\{u \\in \\mathbb{R}^n\\mid \\|u\\|\\leq 1\\}$, with realizations denoted by $u \\in \\mathbb{B}$}.\n\t\t\t\tWe now recall a concentration inequality and subsequently present properties for spherical smoothing, where\n\t\t\t\t$\\mathbb{S}\\triangleq \\{v \\in \\mathbb{R}^n \\mid\n\t\t\t\t\\|v\\|=1\\}$ is the surface of $\\mathbb{B}$.\n","sha256_utf8":"9317b218e2efd60b000af8e0b3c502fd0f08d88f4e1e849289ae4510cc1d94c4"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"扩张定义域","source_units":[{"start":50503,"end":50938,"start_line":846,"end_line":852,"raw_tex":"In developing \\texttt{VRG-ZO}, we minimize an $L_0$-Lipschitz\nfunction on $\\Xscr_{\\eta_0}$, where  $\\Xscr_{\\eta_0} \\triangleq\n\\Xscr + \\eta_0 \\mathbb{B}$ represents the Minkowski sum of $\\Xscr$\nand $\\eta \\mathbb{B}$.  Further, we assume that the random\nvariable $\\f(\\x,\\bxi) - f(\\x)$ admits suitable bias and moment\nproperties.  We develop schemes for computing approximate\nstationary points of \\eqref{eqn:prob} by an iterative scheme.\n","sha256_utf8":"2f684e639853ed60d10a70a8d1064c044d1ae831b8694101cccbc43e5b8fdff1"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"精确与含误差残差","source_units":[{"start":53534,"end":54574,"start_line":877,"end_line":883,"raw_tex":"\\begin{definition}[{\\bf The residual mapping}]\\label{def:res_maps}\\em Given $\\beta>0$ and a smoothing parameter $\\eta>0$, for any $\\x \\in \\mathbb{R}^n$ and an arbitrary vector $\\tilde e \\in \\mathbb{R}^n$,\n\t\t\t\t\tlet the residual mappings $G_{\\eta,\\beta}(\\x)$ and $\\tilde{G}_{\\eta,\\beta}(\\x, {\\tilde e})$ be defined as $G_{\\eta,\\beta}(\\x)  \\triangleq \\beta {\\left(\\x - \\Pi_{\\Xscr}\\left[\\x - \\tfrac{1}{\\beta} \\nabla_x {f_{\\eta}(\\x)}\\right]\\right)}$ and $\\tilde{G}_{\\eta,\\beta}(\\x, {\\tilde e})  \\triangleq \\beta\\left( \\x - \\Pi_{\\Xscr}\\left[\\x - \\tfrac{1}{\\beta} ({\\nabla_x f_{\\eta}(\\x)}+\\tilde{e}) \\right]\\right)$, respectively.\n\t\t\t\t\t%\\begin{align}\n\t\t\t\t\t%    G_{\\eta,\\beta}(\\x) &\\triangleq \\beta {\\left(\\x - \\Pi_{\\Xscr}\\left[\\x - \\tfrac{1}{\\beta} \\nabla_x {f_{\\eta}(\\x)}\\right]\\right)} \\mbox{ and }\\\\\n\t\t\t\t\t%    \\hspace{-0.05in}\\tilde{G}_{\\eta,\\beta}(\\x, {\\tilde e}) &\\triangleq \\beta\\left( \\x - \\Pi_{\\Xscr}\\left[\\x - \\tfrac{1}{\\beta} ({\\nabla_x f_{\\eta}(\\x)}+\\tilde{e}) \\right]\\right). \\hspace{-0.1in} \\qed\n\t\t\t\t\t%\\end{align}\n\t\t\t\t\\end{definition}","sha256_utf8":"eb85a9654c484c5cc82ee47982f9c4299f7538e55e9838e335855d2e1528e999"}]},{"id":"S006","semantic_type":"algorithm","kind":"algorithm","title":"零阶投影梯度算法","source_units":[{"start":56164,"end":57638,"start_line":905,"end_line":923,"raw_tex":"\\begin{algorithm}[hbt]\n\t\t\t\t\t\\caption{\\texttt{{VRG-ZO}}: Variance-reduced randomized zeroth-order {gradient} method}\\label{algorithm:zo_nonconvex}\n\t\t\t\t\t%\\begin{boxedminipage}{165mm}\n\t\t\t\t\t{   \\begin{algorithmic}[1]\n\t\t\t\t\t\t\t\\STATE\\textbf{input:}  Given $\\x_0 \\in \\Xscr$, stepsize $\\gamma>0$, smoothing parameter $\\eta>0$, mini-batch sequence $\\{N_k\\}$, and an integer $R_{\\ell,K}$ randomly selected from $\\{\\ell\\triangleq\\lceil\\lambda K\\rceil ,\\ldots,K\\}$ using a discrete uniform distribution where $\\lambda \\in (0,1)$ and $K\\geq 1$.\n\t\t\t\t\t\t\t\\FOR {$k=0,1,\\ldots,{K}-1$}\n\t\t\t\t\t\t\t\\STATE   Generate a random {mini-}{batch $v_{j,k} \\in \\eta \\mathbb{S}$ for $j=1,\\ldots,N_k$}  \n\t\t\t\t\t\t\t% \\STATE Generate a random {batch $\\xi_{j,k}$ for $j=1,\\ldots,N_k$}  \n\t\t\t\t\t\t\t\\STATE  Compute $\\{g_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k})\\}_{j=1}^{N_k}$, where for all $j=1,\\ldots,N_k$,\\\\ $g_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k}) :=\\left(\\tfrac{n}{\\us{2}\\eta}\\right)\\left({\\f}(\\x_k+ v_{j,k},\\xi_{j,k}) - {\\f}(\\x_k- v_{j,k},\\xi_{j,k})\\right)\\tfrac{v_{j,k}}{\\left\\|v_{j,k}\\right\\|}$ \n\t\t\t\t\t\t\t\\STATE   {Compute} mini-batch inexact {ZO} gradient. $\n\t\t\t\t\t\t\tg_{\\eta,N_k}(\\x_k) := \\tfrac{\\sum_{j=1}^{N_k}  g_{\\eta}\\left(\\x_k,v_{j,k},\\xi_{j,k}\\right)}{N_k}$\n\t\t\t\t\t\t\t\\STATE   {Update $\\x_k$ as follows.}\n\t\t\t\t\t\t\t$\\x_{k+1}:= \\Pi_{\\Xscr}\\left[ \\x_{k}-\\gamma g_{\\eta,N_k}(\\x_k) \\right]$\n\t\t\t\t\t\t\t\\ENDFOR\n\t\t\t\t\t\t\t\\STATE Return $\\x_{R_{\\ell,K}}$ \n\t\t\t\t\t\\end{algorithmic}}\n\t\t\t\t\t%\\end{boxedminipage}\n\t\t\t\t\t% \\vskip17.5pt\n\t\t\t\t\\end{algorithm}","sha256_utf8":"6b5541c3ff38f352fb69571f1c8d2017690c293fbbc96fb6e223ff1783e37e8f"}]},{"id":"S007","semantic_type":"assumption","kind":"assumption","title":"随机目标与可行集假设","source_units":[{"start":57909,"end":58649,"start_line":928,"end_line":936,"raw_tex":"\\begin{assumption}[{\\bf Problem properties}] \n\t\t\\label{ass-1}\\em\n\t\t\\noindent Consider problem \\eqref{eqn:prob}.\n\t\t\n\t\t\\noindent (i) {$\\tilde{f}(\\bullet,\\bxi)$ is $\\tilde{L}_0(\\bxi)$}-Lipschitz on $\\Xscr + \\eta_0 \\mathbb{B}$ for some $\\eta_0 > 0$, a.s. for all $\\bxi$,  where $L_0\\triangleq \\sqrt{\\mathbb{E}[{\\tilde{L}_0}(\\bxi)^2]} <\\infty$.  \n\t\t\\noindent (ii) $\\Xscr \\subseteq \\Real^n$ is a nonempty, closed, and convex set.\n\t\t\\noindent (iii) For any $\\x \\in \\Xscr+ \\eta_0 \\us{\\mathbb{B}}$, $\\mathbb{E}[\\, {\\f(\\x,\\bxi)} \\, \\mid \\,  \\x\\, ] \\, =\\, f(\\x)$.\n\t\t%\\noindent (iv) For all $\\x \\, \\in \\, \\Xscr+ \\eta_0 \\mathbb{S}$, $\\mathbb{E}[\\, |\\, \\f(\\x,{\\bxi})- f(\\x) \\, |^2 \\, \\mid \\,  \\x\\, ] \\, \\leq \\, \\nu^2$ for some $\\nu > 0$.\n\\end{assumption}","sha256_utf8":"a65c0758ad38f90b2fa9329a763fb2071022fec3dd8e129833516d9a55095ba0"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"零阶估计与查询成本","source_units":[{"start":58650,"end":58851,"start_line":937,"end_line":939,"raw_tex":"\t\t\t\tWe introduce a variance-reduced randomized zeroth-order scheme presented by\n\t\t\t\tAlgorithm \\ref{algorithm:zo_nonconvex} and define a zeroth-order gradient\n\t\t\t\testimate of  $\\f(\\x_k, \\xi_{j,k})$ as \n","sha256_utf8":"cead9ee3de03f3f767989c6f745567e5e5a821f5abc22526c4dd74443bbf10d8"},{"start":59195,"end":59419,"start_line":945,"end_line":947,"raw_tex":"\\begin{align*}\n\t\t\t\t\tg_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k}) \\triangleq \\left(\\tfrac{n}{\\eta}\\right)\\left({\\f}(\\x_k+ v_{j,k},\\xi_{j,k}) - {\\f}({\\x_k-v_{j,k}},\\xi_{j,k})\\right)\\tfrac{v_{j,k}}{\\left\\|v_{j,k}\\right\\|},\n\t\t\t\t\\end{align*}","sha256_utf8":"4a7bc714c30cd8dd8ece5bf720d70da2817bd103f1bc3b91ebedc10ec29c1216"},{"start":59420,"end":60073,"start_line":948,"end_line":951,"raw_tex":"\t\t\t\twhere $v_{j,k} \\in \\eta\\mathbb{S}$.  {Intuitively, $g_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k})$ generates a gradient estimate by employing sampled function evaluations $\\tilde{f}(\\x_k{+v_{j,k}},\\xi_{j,k})$ and $\\tilde{f}(\\x_k{-v_{j,k}},\\xi_{j,k})$; in short, the zeroth-order oracle, given an $\\x_k$ and a perturbation vector $v_{j,k}$, produces two evaluations, \\fy{i.e.,}   \n\t\t\t\t\t$\\tilde{f}(\\x_k +v_{j,k},\\xi_{j,k})$ and $\\tilde{f}(\\x_k-v_{j,k},\\xi_{j,k})$.}\n\t\t\t\t%\\lm{We then employ a mini-batch approximation of this gradient estimate.} \n\t\t\t\tWe formally define the stochastic errors emergent from the randomized scheme based on a mini-batch approximation.\n","sha256_utf8":"36c51c3327c13e6237dc7bc96d8aae93e3553cef6bea555cfa7fe094ff4f38ba"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"随机梯度误差","source_units":[{"start":60077,"end":60981,"start_line":952,"end_line":960,"raw_tex":"\\begin{definition}\\label{def:stoch_errors} \\em\n\t\t\t\t\tFor all $k\\geq 0,j=1,\\ldots,N_k$, let \n\t\t\t\t\t$\\mathbf{e}_k\\triangleq \\tfrac{\\sum_{j=1}^{N_k}\\mathbf{e}_{j,k}}{N_k}$, %, $\\bm{\\theta}_k\\triangleq \\tfrac{\\sum_{j=1}^{N_k}\\bm{\\theta}_{j,k}}{N_k}$, \n\t\t\t\t\t\\begin{align}\n\t\t\t\t\t\te_{j,k} & \\triangleq {g_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k}) -\\nabla f_\\eta(\\x_k)}, \\ \\mbox{and } {\\mathbf{e}_{j,k}  \\triangleq g_{\\eta}(\\x_k,\\vv_{j,k},\\bxi_{j,k})}-\\nabla f_\\eta(\\x_k)\\label{def:stoch_errors1}. \\mbox{$\\ \\Box$}\n\t\t\t\t\t\t%\\\\ \\theta_{j,k } & \\triangleq g_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k}) -\\nabla {\\f}_\\eta(\\x_k,\\xi_{j,k}), \\mbox{ and } {\\bm{\\theta}_{j,k}  \\triangleq g_{\\eta}(\\x_k,\\vv,\\bxi_{\\lm{j}}) -\\nabla {\\f}_\\eta(\\x_k,\\bxi_{\\lm{j}})}.\\label{def:stoch_errors2}\n\t\t\t\t\t\t%&\\delta_{j,k} \\triangleq  g_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k}) -  g_{\\eta}(\\x_k,v_{j,k},\\xi_{j,k}),\\label{def:stoch_errors3}\n\t\t\t\t\t\\end{align}\n\t\t\t\t\\end{definition}","sha256_utf8":"06c89c1a3b4176adf9862f0ce160bb11f887ca6f69178e64e502661df5834546"}]},{"id":"S010","semantic_type":"notation","kind":"notation","title":"随机变量与算法历史","source_units":[{"start":60982,"end":61245,"start_line":961,"end_line":962,"raw_tex":"\t\t\t\tWe employ boldface  to denote random variables (such as $\\mathbf{e}_{j,k}$)  while their realizations are not emboldened (such as $e_{j,k}$).\n\t\t\t\tThe history of Algorithm \\ref{algorithm:zo_nonconvex} at iteration $k$ is denoted by $\\mathcal{F}_k$, defined as\n","sha256_utf8":"548d40a416e17d9a21b1740d7b2137f83c77bbbd8ca83710b535074dc9a6aa30"},{"start":61249,"end":61510,"start_line":963,"end_line":966,"raw_tex":"\\begin{align}\\label{eqn:history}\n\t\t\t\t\t\\mathcal{F}_0 \\triangleq \\{x_0\\},\\quad \n\t\t\t\t\t\\mathcal{F}_k \\triangleq \\mathcal{F}_{k-1} \\, \\bigcup \\, \\left\\{ \\, \\bigcup_{j=1}^{N_{k-1}}\\, \\{\\xi_{j,k-1}, v_{j,k-1}\\} \\, \\right\\}, \\qquad \\hbox{for } k \\geq 1.\n\t\t\t\t\\end{align}","sha256_utf8":"3f2433c87b29955398c6a17b7d88aaace104ef77db40a2b84962e6de47f21cd6"}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","title":"采样独立性","source_units":[{"start":61612,"end":61824,"start_line":968,"end_line":970,"raw_tex":"\\begin{assumption}[{\\bf Independence}]\\label{assum:random_vars}\\em\n\t\t\t\t\tRandom samples $\\xi_{j,k}$, $v_{j,k}$ are generated independent of each other for all $k\\geq 0$ and $1\\leq j \\leq N_k$.\n\t\t\t\t\\end{assumption}","sha256_utf8":"6f1924b4f71565648f49b02ce68bc7472bc0e0c0e75c2e14ef7361ad5634ce28"}]},{"id":"S012","semantic_type":"lemma","kind":"lemma","title":"球面集中不等式","source_units":[{"start":42722,"end":43226,"start_line":737,"end_line":743,"raw_tex":"\\begin{lemma}[L\\'{e}vy concentration on $ \\mathbb{S}^n $~{\\cite[Prop. 3.11 and Example 3.12]{Wainwright2019}}]\n\t\t\t\t\t\\label{lemma:levy-concentration}\\em\n\t\t\t\t\tLet ${h}: \\mathbb{R}^n \\to \\mathbb{R}$ be $L$-Lipschitz and $\\tilde{\\vv}$ be uniformly distributed on {$\\mathbb{S}$}. Then \n\t\t\t\t\t\\begin{align}\n\t\t\t\t\t\t\\mathbb{P}\\left[\\, \\left|\\, h(\\tilde{\\vv}) - \\mathbb{E}[h(\\tilde{\\vv})]\\, \\right|\\, \\geq \\, \\alpha \\, \\right] &\\, \\leq \\, 2 \\sqrt{2\\pi} e^{-\\frac{n \\alpha^2}{8L^2}}.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\\end{lemma}","sha256_utf8":"b7204e2b186b730a8086dbccbec1841dde5876f9d61155f9d289949441ec725e"}]},{"id":"S013","semantic_type":"lemma","kind":"lemma","title":"球内平滑的基本性质","source_units":[{"start":43233,"end":46287,"start_line":745,"end_line":777,"raw_tex":"\\begin{lemma}[{\\bf Properties of spherical smoothing}] \\label{lemma:props_local_smoothing}\\em Suppose $h:\\mathbb{R}^n \\mapsto \\mathbb{R}$ is {a continuous function and its smoothed counterpart $h_{\\eta}$ is defined as \\eqref{def-smooth}, where $\\eta>0$ is a given scalar}. Then the following hold for any $\\x, \\y \\in {\\cal X}$.\n\t\t\t\t\t\n\\begin{enumerate}\n\t\t\t\t\t\t\\item[ (i)] $h_{\\eta}$ is $C^1$ over $\\Xscr$ and $\\nabla_{\\x} h_{\\eta}(\\x) = \\left(\\tfrac{n}{2\\eta}\\right) \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\left[\\left(h(\\x+\\vv)-h(\\x-\\vv)\\right) \\tfrac{\\vv}{\\|\\vv\\|}\\right]. $ \n\t\t\t\t\t\\end{enumerate}\n\t\t\t\t\t%    \\begin{align}\n\t\t\t\t\t\t%        \\nabla_{\\x} h_{\\eta}(\\x) = \\left(\\tfrac{n}{\\eta}\\right) \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\left[\\us{h(\\x+\\vv) \\tfrac{\\vv}{\\|\\vv\\|}}\\right] \\quad \\forall \\x \\in \\Xscr. \n\t\t\t\t\t\t%    \\end{align}\nSuppose $h \\in C^{0,0}(\\Xscr_{\\eta})$ with parameter $L_0$. For any $\\x, \\y \\in \\Xscr$,  the following hold. \n\t\t\t\t\t\n\t\t\t\t\t\\begin{enumerate}\n\t\t\t\t\t\t\\item[ (ii)] $| h_{\\eta}(\\x)-h_{\\eta}(\\y) | \\leq L_0 \\|\\x-\\y\\|.$  (iii) $| h_{\\eta}(\\x) - h(\\x)| \\leq L_0 \\eta.$ \n\t\t\t\t\t\t\n\t\t\t\t\t\t\\item[ (iv)] $\\| \\nabla_{\\x} h_{\\eta}(\\x) -\\nabla_{\\x} h_{\\eta}(\\y)\\| \\leq {\n\t\t\t\t\t\t\t\\tfrac{L_0 \\sqrt{n}}{\\eta}}\\|\\x-\\y\\|.$\n\t\t\t\t\t\t%    \\item[(v)] If $h$ is convex {and $h \\in C^{0,0}(\\Xscr_{\\eta})$ with parameter $L_0$}, then $h_{\\eta}$ is convex and satisfies the following for any $\\x \\in \\Xscr$. \n\t\t\t\t\t\t%        \\begin{align}\n\t\t\t\t\t\t\t%                h(\\x) \\leq h_{\\eta}(\\x) \\leq h(\\x) + \\eta L_0.\n\t\t\t\t\t\t\t%            \\end{align} \n\t\t\t\t\t\t% \\item[(vi)] $h_{\\eta} \\in C^{0,0}$.  \\us{Prove//cite} FY: This seems redundant. It is already shown in (ii).\n\t\t\t\t\t\t%  \\item[(iv)] If \\fy{$\\eta>0$, then $ h_{\\eta}(\\x)$ is continuously differentiable and} $\\nabla h_{\\eta}(\\x)$ is defined as $$ \\nabla h_{\\eta}(\\x) \\triangleq \\tfrac{1}{\\kappa} \\int_{\\fy{\\mathbb{R}^n}} \\tfrac{f(\\x+\\eta u) - f(\\x)}{\\eta} e^{-\\frac{\\|u\\|^2}{2}} Bu du. $$ \\fy{Moreover}, $h_{\\eta} \\in C^{1,1}$ and $L_1(h_{\\eta}) = \\tfrac{\\sqrt{n}}{\\eta} L_0(h).$  \n\t\t\t\t\t\t%    \\item[(vi)] If $h$ is convex {and $h \\in C^{0,0}(\\Xscr_{\\eta})$ with parameter $L_0$}, then $\\nabla_x h_{\\eta}(\\x) \\in \\partial_{\\epsilon} h(\\x)$ where {$\\epsilon \\triangleq \\eta L_0$}.\n\t\t\t\t\t\t\n\t\t\t\t\t\t%\\item[(vii)] If $h \\in C^{0,0}$ and $\\eta > 0$, then $h_{\\eta} \\in C^{1,1}$ and $L_1(f_{\\eta}) = \\tfrac{\\sqrt{n} L_0(f)}{\\eta}.$ \n\t\t\t\t\t\t%     \\us{Modify for spherical/Prove/Cite} FY: I think the version we use in the paper is already shown in (iv)\n\t\t\t\t\t\t\n\t\t\t\t\t\t%    \\item[(vii)] If {$h \\in C^{1,1}(\\Xscr_{\\eta})$} with constant {$L_1$}, then $$\\|\\nabla_x h_{\\eta}(\\x)-\\nabla_x h(\\x) \\| \\leq {\\eta L_1 n}.$$\n\t\t\t\t\t\t\n\t\t\t\t\t\t\\item [(v)] {Suppose $h \\in C^{0,0}(\\Xscr_{\\eta})$ with parameter $L_0$ {and} \n\t\t\t\t\t\t\t$g_{\\eta}(\\x,\\vv)  \\triangleq \\left(\\tfrac{n(h(\\x+\\vv) - h(\\x-\\vv))\\vv}{2\\eta\\|\\vv\\|}\\right)$\n\t\t\t\t\t\t\tfor $\\vv \\in \\eta\\mathbb{S}$.  Then for} any $\\x \\in \\Xscr$, we have that ${\\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}}}[{\\|{g_\\eta}(\\x,\\vv)\\|^2}] \\leq 16\\sqrt{2\\pi}L_0^2 n$. $\\hfill \\Box$ \n\t\t\t\t\t\\end{enumerate}\n\t\t\t\t\\end{lemma}","sha256_utf8":"e1f73b5f1c1018970c960013cb9e88a0faaba9399abbaf902074e9c3d243e542"}]},{"id":"S014","semantic_type":"proof","kind":"proof","title":"平滑性质完整证明","source_units":[{"start":46292,"end":50502,"start_line":778,"end_line":845,"raw_tex":"\\begin{proof}\n\t\t\t\t\t{(i) By Stoke's theorem and spherical symmetry of $\\vv$, we have that \n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\nabla_{\\x} h_{\\eta}(\\x)& = \\tfrac{n}{\\eta} \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\ \\left[ \\tfrac{h(\\x+\\vv) \\vv }{\\|\\vv\\|} \\right] = \n\t\t\t\t\t\t\t\\tfrac{n}{2\\eta} \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\ \\left[ \\tfrac{(h(\\x+\\vv)+h(\\x+\\vv)) \\vv }{\\|\\vv\\|} \\right] \\\\\n\t\t\t\t\t\t\t& = \\tfrac{n}{2\\eta} \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\ \\left[ \\tfrac{(h(\\x+\\vv)) \\vv }{\\|\\vv\\|}+\\tfrac{(h(\\x+\\vv)) \\vv }{\\|\\vv\\|} \\right] = \n\t\t\t\t\t\t\t\\tfrac{n}{2\\eta} \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\ \\left[ \\tfrac{(h(\\x+\\vv)) \\vv }{\\|\\vv\\|}+\\tfrac{(h(\\x-\\vv)) (-\\vv) }{\\|\\vv\\|} \\right] \\\\\n\t\t\t\t\t\t\t& = \\tfrac{n}{2\\eta} \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\ \\left[ \\tfrac{((h(\\x+\\vv))-(h(\\x-\\vv)) \\vv) }{\\|\\vv\\|} \\right]. \n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\t(ii,iii) follow akin to ~\\cite[Lemma 1]{CSY2021MPEC}. \n\t\t\t\t\t\t(iv)\nRecall from~\\cite[Prop. 2.2]{lin2022} that $\\|\n\\nabla_{\\x} h_{\\eta}(\\x) -\\nabla_{\\x}\nh_{\\eta}(\\y)\\| \\leq \\tfrac{C_n\nL_0}{\\eta}\\|\\x-\\y\\|,$ for $n\\geq 1$, where\n$C_n \\triangleq\n\\left(\\tfrac{2}{\\pi}\\right)^{p_n}\n\\tfrac{n!!}{(n-1)!!}$ and $p_n = 1$ if $n$\nis even, and $p_n =0$ otherwise. It suffices to show that $C_n \\le 1$ for $n \\ge 1$. For $n \\ge\n0$, let $w_n$ denote the Wallis'\nintegrals~\\cite{spivak2006calculus}, given as\n$w_n = \\int_{0}^{\\pi/2}\\sin^n(u)du$. Then we\nhave $w_n =\\left(\\tfrac{n-1}{n}\\right)w_{n-2}$ for $n \\geq\n2$ where $w_0=\\tfrac{\\pi}{2}$ and $w_1=1$\n(cf.~\\cite[Ch. 19]{spivak2006calculus}). This\nimplies (via mathematical induction) that\n$w_n =\n\\left(\\tfrac{\\pi}{2}\\right)\\tfrac{(n-1)!!}{n!!}$\nfor even $n$, while $w_n =\n\\tfrac{(n-1)!!}{n!!}$ for odd $n$. Let us\ndefine $v_n \\,\\triangleq \\, (n+1)w_n w_{n+1}$ for $n\\geq 0$.\nWe obtain $v_n =\n(n+1)\\left(\\tfrac{\\pi}{2}\\right)\n\\tfrac{(n-1)!!}{n!!}\\tfrac{n!!}{(n+1)!!} =\n\\tfrac{\\pi}{2}$, implying that $v_n =\n\\tfrac{\\pi}{2}$ for all $n\\geq 0$. Recall\nthat from~\\cite{spivak2006calculus} we have\n$w_{n+1} < w_n$ for all $n\\geq 0$. Thus we\nobtain $\\tfrac{\\pi}{2} = v_n  =(n+1)w_n\nw_{n+1} < (n+1)w_n^2$ for all $n\\geq 0$, i.e., $w_n < \\tfrac{\\pi}{2(n+1)}$ for $n \\ge 0$.  Note\nthat $C_1 =1$. Suppose $n \\geq 2$. We have\n$C_n= \\tfrac{1}{w_n} <\n\\sqrt{\\tfrac{2(n+1)}{\\pi}} =\n\\sqrt{\\tfrac{2(1+1/n)}{\\pi}} \\sqrt{n} \\leq\n\\sqrt{\\tfrac{2(1+1/2)}{\\pi}} \\sqrt{n} \\leq\n0.98 \\sqrt{n} $. This implies that $C_n  \\leq\n1$ for all $n\\geq 1$.} {(v) We have that}\n\\begin{align*}\n\t\t\t\t\t\t\t\\mathbb{E}\\left[ \\|g_{\\eta}(\\x,\\vv)\\|^2 \\right] & \\le \n\t\t\t\t\t\t\t\\tfrac{n^2}{4\\eta^2} \\mathbb{E}\\left[ \\|\\tfrac{(h(\\x+\\vv)-h(\\x-\\vv))\\vv}{\\|\\vv\\|}\\|^2 \\right] \\le \n\t\t\t\t\t\t\t\\tfrac{n^2}{4\\eta^2} \\mathbb{E}\\left[  (h(\\x+\\vv)-h(\\x-\\vv))^2 \\tfrac{\\|\\vv\\|^2}{\\|\\vv\\|^2} \\right] \\\\\n\t\t\t\t\t\t\t& \\le \n\t\t\t\t\t\t\t\\tfrac{n^2}{4\\eta^2} \\mathbb{E}\\left[ \\left(h(\\x+\\vv)-\\mathbb{E}\\left[h(\\x+\\vv)\\right] + \\mathbb{E}\\left[ h(\\x+\\vv)\\right]-h(\\x-\\vv)\\right)^2  \\right]. \n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\tBut $\\mathbb{E}\\left[ h(\\x+\\vv)\\right] = \\mathbb{E}\\left[ h(\\x-\\vv)\\right]$ by spherical symmetry. Consequently, \n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\mathbb{E}\\left[ \\|g_{\\eta}(\\x,\\vv)\\|^2 \\right] & \\le \n\t\t\t\t\t\t\t\\tfrac{n^2}{4\\eta^2} \\mathbb{E}\\left[  \\left(h(\\x+\\vv)-\\mathbb{E}\\left[h(\\x+\\vv)\\right] + \\mathbb{E}\\left[ h(\\x-\\vv)\\right]-h(\\x-\\vv)\\right)^2  \\right] \\\\\n\t\t\t\t\t\t\t& \\le \\tfrac{2n^2}{4\\eta^2} \\mathbb{E}\\left[ \\left|(h(\\x+\\vv)-\\mathbb{E}\\left[h(\\x+\\vv)\\right]\\right|^2  + \\left|\\mathbb{E}\\left[ h(\\x-\\vv)\\right]-h(\\x-\\vv))\\right|^2 \\right]. \n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\tThen by invoking L\\'{e}vy's inequality (cf. Lemma~\\ref{lemma:levy-concentration}), we have that \n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\mathbb{E}\\left[ |(h(\\x+\\vv)-\\mathbb{E}\\left[h(\\x+\\vv)\\right]|^2 \\right] \\le 2\\sqrt{2\\pi}\\int_0^{\\infty} e^{-\\alpha n}{8 \\eta^2 L_0^2} d\\alpha = \\tfrac{16\\sqrt{2\\pi} \\eta^2 L_0^2}{n}. \n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t{By noting that $\\mathbb{E}\\left[ \\left|(h(\\x+\\vv)-\\mathbb{E}\\left[h(\\x+\\vv)\\right]\\right|^2  \\right] = \\mathbb{E}\\left[  \\left|\\mathbb{E}\\left[ h(\\x-\\vv)\\right]-h(\\x-\\vv))\\right|^2 \\right]$ and leveraging the preceding bound, we get}  \n\t\t\t\t\t\t$\\mathbb{E}\\left[ \\|g_{\\eta}(\\x,\\vv)\\|^2 \\right] \\le \\tfrac{n^2}{\\eta^2} \\tfrac{16\\sqrt{2\\pi} \\eta^2 L_0^2}{n} = 16\\sqrt{2\\pi}L_0^2 n.$\n\t\t\t\t\t\n\t\t\t\t\\end{proof}","sha256_utf8":"80392213ea0a702560c6388da5cd2af2aa4d55b51c802fd206d726835d1a2c03"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"对称化梯度积分公式","source_units":[{"start":46292,"end":47124,"start_line":778,"end_line":786,"raw_tex":"\\begin{proof}\n\t\t\t\t\t{(i) By Stoke's theorem and spherical symmetry of $\\vv$, we have that \n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\nabla_{\\x} h_{\\eta}(\\x)& = \\tfrac{n}{\\eta} \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\ \\left[ \\tfrac{h(\\x+\\vv) \\vv }{\\|\\vv\\|} \\right] = \n\t\t\t\t\t\t\t\\tfrac{n}{2\\eta} \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\ \\left[ \\tfrac{(h(\\x+\\vv)+h(\\x+\\vv)) \\vv }{\\|\\vv\\|} \\right] \\\\\n\t\t\t\t\t\t\t& = \\tfrac{n}{2\\eta} \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\ \\left[ \\tfrac{(h(\\x+\\vv)) \\vv }{\\|\\vv\\|}+\\tfrac{(h(\\x+\\vv)) \\vv }{\\|\\vv\\|} \\right] = \n\t\t\t\t\t\t\t\\tfrac{n}{2\\eta} \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\ \\left[ \\tfrac{(h(\\x+\\vv)) \\vv }{\\|\\vv\\|}+\\tfrac{(h(\\x-\\vv)) (-\\vv) }{\\|\\vv\\|} \\right] \\\\\n\t\t\t\t\t\t\t& = \\tfrac{n}{2\\eta} \\mathbb{E}_{\\vv \\in \\eta \\mathbb{S}} \\ \\left[ \\tfrac{((h(\\x+\\vv))-(h(\\x-\\vv)) \\vv) }{\\|\\vv\\|} \\right]. \n\t\t\t\t\t\t\\end{align*}\n","sha256_utf8":"65d5d40ecdb35f5f8e5226589c43c239aa745c864e6cbac1fbdd5960a710c5ca"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"引用平滑逼近界","source_units":[{"start":47124,"end":47185,"start_line":787,"end_line":787,"raw_tex":"\t\t\t\t\t\t(ii,iii) follow akin to ~\\cite[Lemma 1]{CSY2021MPEC}. \n","sha256_utf8":"a28ca744a95be27537876d63c7d6f7e48ca450b7b8a5797dbbbbc710eb771ac1"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"引入梯度界与沃利斯积分","source_units":[{"start":47185,"end":47982,"start_line":788,"end_line":807,"raw_tex":"\t\t\t\t\t\t(iv)\nRecall from~\\cite[Prop. 2.2]{lin2022} that $\\|\n\\nabla_{\\x} h_{\\eta}(\\x) -\\nabla_{\\x}\nh_{\\eta}(\\y)\\| \\leq \\tfrac{C_n\nL_0}{\\eta}\\|\\x-\\y\\|,$ for $n\\geq 1$, where\n$C_n \\triangleq\n\\left(\\tfrac{2}{\\pi}\\right)^{p_n}\n\\tfrac{n!!}{(n-1)!!}$ and $p_n = 1$ if $n$\nis even, and $p_n =0$ otherwise. It suffices to show that $C_n \\le 1$ for $n \\ge 1$. For $n \\ge\n0$, let $w_n$ denote the Wallis'\nintegrals~\\cite{spivak2006calculus}, given as\n$w_n = \\int_{0}^{\\pi/2}\\sin^n(u)du$. Then we\nhave $w_n =\\left(\\tfrac{n-1}{n}\\right)w_{n-2}$ for $n \\geq\n2$ where $w_0=\\tfrac{\\pi}{2}$ and $w_1=1$\n(cf.~\\cite[Ch. 19]{spivak2006calculus}). This\nimplies (via mathematical induction) that\n$w_n =\n\\left(\\tfrac{\\pi}{2}\\right)\\tfrac{(n-1)!!}{n!!}$\nfor even $n$, while $w_n =\n\\tfrac{(n-1)!!}{n!!}$ for odd $n$. Let us\n","sha256_utf8":"171df7a31e0e1dd6f7d0956d5d1a536698cb4d66c7e088ccdcc97c4e7ac0c037"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"估计维数相关系数","source_units":[{"start":47982,"end":48708,"start_line":808,"end_line":824,"raw_tex":"define $v_n \\,\\triangleq \\, (n+1)w_n w_{n+1}$ for $n\\geq 0$.\nWe obtain $v_n =\n(n+1)\\left(\\tfrac{\\pi}{2}\\right)\n\\tfrac{(n-1)!!}{n!!}\\tfrac{n!!}{(n+1)!!} =\n\\tfrac{\\pi}{2}$, implying that $v_n =\n\\tfrac{\\pi}{2}$ for all $n\\geq 0$. Recall\nthat from~\\cite{spivak2006calculus} we have\n$w_{n+1} < w_n$ for all $n\\geq 0$. Thus we\nobtain $\\tfrac{\\pi}{2} = v_n  =(n+1)w_n\nw_{n+1} < (n+1)w_n^2$ for all $n\\geq 0$, i.e., $w_n < \\tfrac{\\pi}{2(n+1)}$ for $n \\ge 0$.  Note\nthat $C_1 =1$. Suppose $n \\geq 2$. We have\n$C_n= \\tfrac{1}{w_n} <\n\\sqrt{\\tfrac{2(n+1)}{\\pi}} =\n\\sqrt{\\tfrac{2(1+1/n)}{\\pi}} \\sqrt{n} \\leq\n\\sqrt{\\tfrac{2(1+1/2)}{\\pi}} \\sqrt{n} \\leq\n0.98 \\sqrt{n} $. This implies that $C_n  \\leq\n1$ for all $n\\geq 1$.} {(v) We have that}\n","sha256_utf8":"a323d19ad5addf41579f7f66d5d231c6f738cd21e0918192f325bbdba0d0acd1"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"展开估计量二阶矩","source_units":[{"start":48708,"end":49194,"start_line":825,"end_line":831,"raw_tex":"\\begin{align*}\n\t\t\t\t\t\t\t\\mathbb{E}\\left[ \\|g_{\\eta}(\\x,\\vv)\\|^2 \\right] & \\le \n\t\t\t\t\t\t\t\\tfrac{n^2}{4\\eta^2} \\mathbb{E}\\left[ \\|\\tfrac{(h(\\x+\\vv)-h(\\x-\\vv))\\vv}{\\|\\vv\\|}\\|^2 \\right] \\le \n\t\t\t\t\t\t\t\\tfrac{n^2}{4\\eta^2} \\mathbb{E}\\left[  (h(\\x+\\vv)-h(\\x-\\vv))^2 \\tfrac{\\|\\vv\\|^2}{\\|\\vv\\|^2} \\right] \\\\\n\t\t\t\t\t\t\t& \\le \n\t\t\t\t\t\t\t\\tfrac{n^2}{4\\eta^2} \\mathbb{E}\\left[ \\left(h(\\x+\\vv)-\\mathbb{E}\\left[h(\\x+\\vv)\\right] + \\mathbb{E}\\left[ h(\\x+\\vv)\\right]-h(\\x-\\vv)\\right)^2  \\right]. \n\t\t\t\t\t\t\\end{align*}\n","sha256_utf8":"85dc07d93bd508f23eeb00bcb4109ce6198222061e65a21923e9569ad32720cb"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"利用球面对称中心化","source_units":[{"start":49194,"end":49762,"start_line":832,"end_line":837,"raw_tex":"\t\t\t\t\t\tBut $\\mathbb{E}\\left[ h(\\x+\\vv)\\right] = \\mathbb{E}\\left[ h(\\x-\\vv)\\right]$ by spherical symmetry. Consequently, \n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\mathbb{E}\\left[ \\|g_{\\eta}(\\x,\\vv)\\|^2 \\right] & \\le \n\t\t\t\t\t\t\t\\tfrac{n^2}{4\\eta^2} \\mathbb{E}\\left[  \\left(h(\\x+\\vv)-\\mathbb{E}\\left[h(\\x+\\vv)\\right] + \\mathbb{E}\\left[ h(\\x-\\vv)\\right]-h(\\x-\\vv)\\right)^2  \\right] \\\\\n\t\t\t\t\t\t\t& \\le \\tfrac{2n^2}{4\\eta^2} \\mathbb{E}\\left[ \\left|(h(\\x+\\vv)-\\mathbb{E}\\left[h(\\x+\\vv)\\right]\\right|^2  + \\left|\\mathbb{E}\\left[ h(\\x-\\vv)\\right]-h(\\x-\\vv))\\right|^2 \\right]. \n\t\t\t\t\t\t\\end{align*}\n","sha256_utf8":"a97d332c3b31cca2003094f8381ccf93c9507f3adaafcb0b7282ab4535247132"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"集中界推出二阶矩界","source_units":[{"start":49762,"end":50502,"start_line":838,"end_line":845,"raw_tex":"\t\t\t\t\t\tThen by invoking L\\'{e}vy's inequality (cf. Lemma~\\ref{lemma:levy-concentration}), we have that \n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\mathbb{E}\\left[ |(h(\\x+\\vv)-\\mathbb{E}\\left[h(\\x+\\vv)\\right]|^2 \\right] \\le 2\\sqrt{2\\pi}\\int_0^{\\infty} e^{-\\alpha n}{8 \\eta^2 L_0^2} d\\alpha = \\tfrac{16\\sqrt{2\\pi} \\eta^2 L_0^2}{n}. \n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t{By noting that $\\mathbb{E}\\left[ \\left|(h(\\x+\\vv)-\\mathbb{E}\\left[h(\\x+\\vv)\\right]\\right|^2  \\right] = \\mathbb{E}\\left[  \\left|\\mathbb{E}\\left[ h(\\x-\\vv)\\right]-h(\\x-\\vv))\\right|^2 \\right]$ and leveraging the preceding bound, we get}  \n\t\t\t\t\t\t$\\mathbb{E}\\left[ \\|g_{\\eta}(\\x,\\vv)\\|^2 \\right] \\le \\tfrac{n^2}{\\eta^2} \\tfrac{16\\sqrt{2\\pi} \\eta^2 L_0^2}{n} = 16\\sqrt{2\\pi}L_0^2 n.$\n\t\t\t\t\t\n\t\t\t\t\\end{proof}","sha256_utf8":"329c640b574a2385992c19b84db714e4c8f0e9ae40243c0458e069e463a80d42"}]},{"id":"S022","semantic_type":"lemma","kind":"lemma","title":"随机误差无偏性与方差界","source_units":[{"start":61904,"end":62402,"start_line":972,"end_line":977,"raw_tex":"\\begin{lemma}[{\\bf Bias and moment properties of ${\\bf e}$}] \\label{lem:stoch_error_var}\\em\n\t\t\t\t\t%Consider Definition \\ref{def:stoch_errors}. \n\t\t\t\t\tLet Assumptions \\ref{ass-1} and \\ref{assum:random_vars} hold. Then (i) and (ii) hold a.s. for $k\\geq 0$, $N_k \\geq 1$, and $j = 1, \\hdots, N_k$.\n\t\t\t\t\t\n\t\t\t\t\t\\noindent (i) $\\mathbb{E}[\\mathbf{e}_{j,k}\\mid \\mathcal{F}_k] = 0$;  \\noindent (ii) $\\mathbb{E}[\\|\\mathbf{e}_{k}\\|^2\\mid \\mathcal{F}_k] \\leq  \\tfrac{16\\sqrt{2\\pi} L_0^2 n}{N_k}$.\n\t\t\t\t\\end{lemma}","sha256_utf8":"570b7129d6998a6ad43c4950287c260e351b07e9ace0c7206f45b69ee6b77578"}]},{"id":"S023","semantic_type":"proof","kind":"proof","title":"随机误差性质完整证明","source_units":[{"start":62407,"end":63780,"start_line":978,"end_line":994,"raw_tex":"\\begin{proof}\n\t\t\t\t\t\t(i) To show the conditional unbiasedness of $\\mathbf{e}_{j,k}$, we proceed as follows.\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\mathbb{E}_{\\bxi,\\! \\vv}\\left[ {g_{\\eta}(\\x_k,\\!\\vv_{j,k},\\!\\bxi_{j,k})}  \\mid {\\cal F}_k\\right]\\! & =\\mathbb{E}_{\\vv}\\!   \n\t\t\t\t\t\t\t\\left[\\mathbb{E}_{\\bxi}\\left[ \\tfrac{n(\\f(\\x_k+v_{j,k},\\xi_{j,k})-\\f(\\x_k-v_{j,k},\\xi_{j,k}))v_{j,k}}{2\\eta \\|v_{j,k}\\|} \\hspace{-0.05in} \\mid {\\cal F}_k \\cup \\{v_{j,k}\\} \\right] \\right] \\\\\n\t\t\t\t\t\t\t&\\hspace{-1in}  \\overset{\\tiny \\mbox{Lemma~\\ref{lemma:props_local_smoothing}~{(i)}}}{=}\n\t\t\t\t\t\t\t\\mathbb{E}_{\\vv}\\left[ \\tfrac{n}{2\\eta}\\tfrac{(f(\\x_k+v_{j,k})-f(\\x_k\\lm{-}v_{j,k}))v_{j,k}}{\\|v_{j,k}\\|} \\mid {\\cal F}_k \\right]  = \\nabla_{\\x} f_{\\eta}(\\x_k).    \n\t\t\t\t\t\t\\end{align*} \n\t\t\t\t\t\tBy leveraging Lemma~\\ref{lemma:props_local_smoothing}~{(v)}, we have that \n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\mathbb{E}_{{\\bxi, \\vv}}\\left[\\, \\|\\mathbf{e}_{j,k}\\|^2 \\mid {\\cal F}_k \\, \\right] &= \n\t\t\t\t\t\t\t\\mathbb{E}_{{\\bxi}}\\left[\\, \\mathbb{E}_{ {\\vv}}\\left[\\, \\|\\mathbf{e}_{j,k}\\|^2  \\, \\mid{\\cal F}_{k}\\cup\\{\\xi_{j,k}\\} \\right] \\right]\\\\\n\t\t\t\t\t\t\t& \\le \n\t\t\t\t\t\t\t\\mathbb{E}_{{\\bxi}}\\left[\\, 16\\sqrt{2\\pi}\\tilde{L}_0(\\bxi_{{j,k}})^2n \\right] \\le 16\\sqrt{2\\pi} L_0^2 n. \n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\tIt follows that $\\mathbb{E}\\left[\\, \\|\\mathbf{e}_{k}\\|^2  \\, \\mid {\\cal F}_k\\right] \\le \\tfrac{16\\sqrt{2\\pi} L_0^2 n}{N_k}.$\n\t\t\t\t\\end{proof}","sha256_utf8":"c3e08b02f9ec547a0185430436b422913e571c5c632652be2a4f883580be4613"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"条件期望验证无偏性","source_units":[{"start":62407,"end":63150,"start_line":978,"end_line":985,"raw_tex":"\\begin{proof}\n\t\t\t\t\t\t(i) To show the conditional unbiasedness of $\\mathbf{e}_{j,k}$, we proceed as follows.\n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\mathbb{E}_{\\bxi,\\! \\vv}\\left[ {g_{\\eta}(\\x_k,\\!\\vv_{j,k},\\!\\bxi_{j,k})}  \\mid {\\cal F}_k\\right]\\! & =\\mathbb{E}_{\\vv}\\!   \n\t\t\t\t\t\t\t\\left[\\mathbb{E}_{\\bxi}\\left[ \\tfrac{n(\\f(\\x_k+v_{j,k},\\xi_{j,k})-\\f(\\x_k-v_{j,k},\\xi_{j,k}))v_{j,k}}{2\\eta \\|v_{j,k}\\|} \\hspace{-0.05in} \\mid {\\cal F}_k \\cup \\{v_{j,k}\\} \\right] \\right] \\\\\n\t\t\t\t\t\t\t&\\hspace{-1in}  \\overset{\\tiny \\mbox{Lemma~\\ref{lemma:props_local_smoothing}~{(i)}}}{=}\n\t\t\t\t\t\t\t\\mathbb{E}_{\\vv}\\left[ \\tfrac{n}{2\\eta}\\tfrac{(f(\\x_k+v_{j,k})-f(\\x_k\\lm{-}v_{j,k}))v_{j,k}}{\\|v_{j,k}\\|} \\mid {\\cal F}_k \\right]  = \\nabla_{\\x} f_{\\eta}(\\x_k).    \n\t\t\t\t\t\t\\end{align*} \n","sha256_utf8":"add85ba2ac6dc9edeb2e65c859e59f247392086ef994c861d1b876f20fde8c7d"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"单样本与批均值方差界","source_units":[{"start":63150,"end":63780,"start_line":986,"end_line":994,"raw_tex":"\t\t\t\t\t\tBy leveraging Lemma~\\ref{lemma:props_local_smoothing}~{(v)}, we have that \n\t\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\t\\mathbb{E}_{{\\bxi, \\vv}}\\left[\\, \\|\\mathbf{e}_{j,k}\\|^2 \\mid {\\cal F}_k \\, \\right] &= \n\t\t\t\t\t\t\t\\mathbb{E}_{{\\bxi}}\\left[\\, \\mathbb{E}_{ {\\vv}}\\left[\\, \\|\\mathbf{e}_{j,k}\\|^2  \\, \\mid{\\cal F}_{k}\\cup\\{\\xi_{j,k}\\} \\right] \\right]\\\\\n\t\t\t\t\t\t\t& \\le \n\t\t\t\t\t\t\t\\mathbb{E}_{{\\bxi}}\\left[\\, 16\\sqrt{2\\pi}\\tilde{L}_0(\\bxi_{{j,k}})^2n \\right] \\le 16\\sqrt{2\\pi} L_0^2 n. \n\t\t\t\t\t\t\\end{align*}\n\t\t\t\t\t\tIt follows that $\\mathbb{E}\\left[\\, \\|\\mathbf{e}_{k}\\|^2  \\, \\mid {\\cal F}_k\\right] \\le \\tfrac{16\\sqrt{2\\pi} L_0^2 n}{N_k}.$\n\t\t\t\t\\end{proof}","sha256_utf8":"193002aedba66de855d349073931262cd0451887c3cb6a988a1c6984c2f2d6a3"}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","title":"含误差残差比较界","source_units":[{"start":54721,"end":54998,"start_line":885,"end_line":888,"raw_tex":"\\begin{lemma}[{\\cite[Lemma 6]{CSY2021MPEC}}]\\label{lem:inexact_proj_2}\\em\n\t\t\t\t\tFor any $\\beta,\\eta>0, \\x \\in \\Xscr, \\tilde{e} \\in\\mathbb{R}^n$, \n\t\t\t\t\t$$\\|G_{\\eta,\\beta}(\\x) \\|^2   \\leq  {2} \\| \\tilde{G}_{\\eta,\\beta}(\\x, {\\tilde e}) \\|^2 + 2 \\|\\tilde{e}\\|^2.$$  \n\t\t\t\t\\end{lemma}","sha256_utf8":"e4894eccb50bb709a38b9feaffc0b62c2009c307603b2366c04b5ef1859b3590"}]},{"id":"S027","semantic_type":"lemma","kind":"lemma","title":"残差随逆步长单调","source_units":[{"start":55114,"end":55344,"start_line":890,"end_line":891,"raw_tex":"\\begin{lemma}\\label{lemma:mon_G} \\em Suppose $\\eta > 0$ and $\\{\\gamma_k\\}$ is a diminishing sequence. Then, for any $\\x$ and $k \\geq 0$, $\\| G_{\\eta,1/\\gamma_k} (\\x)\\| \\, \\leq \\,\\| G_{\\eta,1/\\gamma_{k+1}} (\\x)\\|$. \n\t\t\t\t\\end{lemma}","sha256_utf8":"ba926915bebdfbeefdaa6147f7ee4394f4945b31b8cb7b9cf123b445b38a3ca4"}]},{"id":"S028","semantic_type":"lemma","kind":"lemma","title":"单步下降与随机残差界","source_units":[{"start":63960,"end":65430,"start_line":997,"end_line":1016,"raw_tex":"\\begin{lemma}\\label{lemma:zog_ineq}\\em \n\t\t\t\t\tLet Assumption \\ref{ass-1} hold {and suppose} $\\left\\{\\x_k\\right\\}$ is generated by\n\t\t\t\t\tAlgorithm~\\ref{algorithm:zo_nonconvex}. Let $\\{\\gamma_k\\}$ be a\n\t\t\t\t\tnon-increasing stepsize sequence (e.g., constant or diminishing) where\n\t\t\t\t\t$\\gamma_0 \\in (0,{\\tfrac{\\eta}{{\\sqrt{n}}L_0}})$ for $\\eta>0$. Then  the\n\t\t\t\t\tfollowing hold. \n\t\t\t\t\t\n\t\t\t\t\t\\noindent (i) For $k\\geq 0$, we have \n\t\t\t\t\t\\begin{align}\\label{eqn:recursive_ineq_lemma5} \n\t\t\t\t\t\t\\left( 1-\\tfrac{{\\sqrt{n}}L_0\\gamma_k}{\\eta}\\right)\\tfrac{\\gamma_k \\|G_{\\eta,1/\\gamma_0} (\\x_k)\\|^2}{4}     &   \\leq   f_{\\eta}(\\x_k)-      f_{\\eta}(\\x_{k+1}) \n\t\t\t\t\t\t+{\\left( 1-\\tfrac{{\\sqrt{n}}L_0\\gamma_k}{2\\eta}\\right)} \\gamma_k \\|e_k\\|^2.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\t\n\t\t\t\t\t\\noindent (ii) Let $R_{\\ell,K}$ be a random integer taking values in $\\{\\ell, \\ldots, K-1\\}$ with mass function $\\mathbb{P}{[} R_{\\ell,K} = j {]} = \\tfrac{\\gamma_j}{\\sum_{i=\\ell}^{K-1} \\gamma_i}$ for any $j \\in \\{\\ell, \\ldots, K-1\\}$.  Then, for all $k\\geq 0$ we have \n\t\t\t\t\t\\begin{align}\\label{ineq:residual_map_ineq_VRGZO_2}\n\t\t\t\t\t\t\\mathbb{E}\\left[ \\|G_{\\eta,{1/\\gamma_0}} (\\x_{R_{\\ell,K}})\\|^2\\right]\n\t\t\t\t\t\t&\\leq \n\t\t\t\t\t\t\\tfrac{\\left( \\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left({16 \\sqrt{2\\pi} L_0^2{n} }\\right)\\sum_{k=\\ell}^{K-1} \\tfrac{\\gamma_k}{N_k}\\right)}{\\left(\\left( \\tfrac{1}{4}-\\tfrac{{\\sqrt{n}} L_0\\gamma_0}{4\\eta}\\right) \\sum_{k=\\ell}^{K-1} \\gamma_k\\right)} .\n\t\t\t\t\t\\end{align}\n\t\t\t\t\\end{lemma}","sha256_utf8":"7112fa466645f204c43a065146dc4f7b3ba92981a9ab9f25340f3b435d63adc9"}]},{"id":"S029","semantic_type":"proof","kind":"proof","title":"下降与残差界完整证明","source_units":[{"start":65440,"end":70450,"start_line":1018,"end_line":1076,"raw_tex":"\\begin{proof}\n\t\t\t\t\t(i) From Lemma~\\ref{lemma:props_local_smoothing} (iv), the gradient mapping $\\nabla f_\\eta$ is Lipschitz continuous with the parameter $L_{\\eta}\\triangleq \\tfrac{\\sqrt{n}L_0}{\\eta}$. From the descent lemma, \n\t\t\t\t\t\\begin{align}\\label{eqn:descent_lem_proof_eq1}\n\t\t\t\t\t\\notag\tf_{\\eta}(\\x_{k+1}) & \\leq f_{\\eta}(\\x_{k}) + \\nabla f_{\\eta}(\\x_k)^\\top(\\x_{k+1}-\\x_k)\n\t\t\t\t\t\t+\\tfrac{L_{\\eta}}{2}\\|\\x_{k+1}-\\x_k\\|^2\\\\\n\t\t\t\t\t\\notag\t&=f_\\eta(\\x_k) + \\left(\\nabla f_{\\eta}(\\x_k)+{e_k}\\right)^\\top(\\x_{k+1}-\\x_k)\\\\\n\t\t& -{e_k}^\\top(\\x_{k+1}-\\x_k)+\\tfrac{L_{\\eta}}{2}\\|\\x_{k+1}-\\x_k\\|^2.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\tInvoking %Lemma \\ref{eqn:proof_comapct_eq1}, \n\t\t\t\t\t\tthe properties of the Euclidean projection, we have \\\\$ \n\t\t\t\t\t(\\x_k-\\gamma_k\\left(\\nabla f_\\eta(\\x_k)+e_k\\right)-\\x_{k+1})^\\top(\\x_k-\\x_{k+1})\\leq 0$. This implies that \n\t\t\t\t\t\\begin{align}\\label{eqn:descent_lem_proof_eq2}\n\t\t\t\t\t\t\\left(\\nabla f_\\eta(\\x_k)+{e_k}\\right)^\\top(\\x_{k+1}-x_k)\n\t\t\t\t\t\t\\leq -\\tfrac{1}{\\gamma_k}\\|\\x_{k+1}-\\x_k\\|^2.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\tIn addition, we may also express $e_k^{\\top}(\\x_{k+1}-\\x_k)$ as follows. \n\t\t\t\t\t\\begin{align}\\label{eqn:descent_lem_proof_eq3}\n\t\t\t\t\t\t-e_k^{\\top}(\\x_{k+1}-\\x_k)\n\t\t\t\t\t\t\\leq \\tfrac{{\\gamma_k}}{2}\\|e_k\\|^2+\\tfrac{1}{2{\\gamma_k}}\\|\\x_{k+1}-\\x_k\\|^2.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\tCombining the inequalities \\eqref{eqn:descent_lem_proof_eq1}, \\eqref{eqn:descent_lem_proof_eq2}, and \\eqref{eqn:descent_lem_proof_eq3} we obtain \n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\tf_{\\eta}(\\x_{k+1})  & \\leq   f_{\\eta}(\\x_k)+ \\left( -\\tfrac{1}{2\\gamma_k}+\\tfrac{L_{\\eta}}{2}\\right) \\|\\x_{k+1}-\\x_k\\|^2 +\\tfrac{\\gamma_k}{2} \\|e_k\\|^2 \\\\\n\t& \\overset{\\tiny \\mbox{Def.~\\ref{def:res_maps}}}{=} f_{\\eta}(\\x_k)- \\left( \\tfrac{1}{2\\gamma_k}-\\tfrac{L_{\\eta}}{2}\\right)\\gamma_k^2 \\left\\|\\tilde{G}_{\\eta,1/\\gamma_k}(\\x_k, {e_k})\\right\\|^2+\\tfrac{\\gamma_k}{2} \\|e_k\\|^2\\\\\n\t& = f_{\\eta}(\\x_k)- \\left( 1-L_{\\eta}\\gamma_k\\right)\\tfrac{\\gamma_k}{2} \\left\\|\\tilde{G}_{\\eta,1/\\gamma_k}(\\x_k, {e_k})\\right\\|^2+\\tfrac{\\gamma_k}{2} \\|e_k\\|^2\\\\\n\t& \\hspace{-0.2in}\\overset{\\tiny \\mbox{Lemma~\\ref{lem:inexact_proj_2}}}{\\le} f_{\\eta}(\\x_k)- \\left( 1-L_\\eta \\gamma_k\\right)\\tfrac{\\gamma_k}{4} \\left\\|G_{\\eta,1/\\gamma_k}(\\x_k)\\right\\|^2+\\left(\\tfrac{\\gamma_k}{2} + \\left( 1-L_{\\eta} \\gamma_k\\right)\\tfrac{\\gamma_k}{2}\\right) \\|e_k\\|^2,\n\t\t\t\t\t\\end{align*}\nwhere $1-L_{\\eta}\\gamma_k > 0$ for any $k$  since $\\gamma_0 < \\tfrac{1}{L_{\\eta}}$. \n\t\t\t\t\tFrom the preceding relation and invoking Lemma~\\ref{lemma:mon_G}, we obtain~\\eqref{eqn:recursive_ineq_lemma5}, i.e.,\n\\begin{align*}\n\\left( 1-L_\\eta \\gamma_k\\right)\\tfrac{\\gamma_k}{4} \\left\\|G_{\\eta,1/\\gamma_0}(\\x_k)\\right\\|^2 \\, \\le \\, f_{\\eta}(\\x_k)- f_{\\eta}(\\x_{k+1}) + \\left(1-\\tfrac{L_{\\eta} \\gamma_k}{2}\\right)\\gamma_k \\|e_k\\|^2.\n\\end{align*} \n\t\t\t\t\t\n\t\t\t\t\t\\noindent (ii) Summing \\eqref{eqn:recursive_ineq_lemma5}  from $k =\\ell, \\ldots, K-1$ where $\\ell\\triangleq  \\lceil \\lambda K\\rceil$ and taking expectations with respect to the iterate trajectory, we have\n\t\t\t\t\t\\begin{align}\n\t\t\t\t\t\t\\notag \\sum_{{k=\\ell}}^{K-1} \\left( 1-\\tfrac{ \\sqrt{n} L_0\\gamma_k}{\\eta}\\right) \\tfrac{\\gamma_k}{4}\\EXP{\\, \\|G_{\\eta,1/\\gamma_k} (\\x_k)\\|^2\\, }  \n\t\t\t\t\t\t& \\leq   \\EXP{\\, f_{\\eta}(\\x_\\ell)\\, }-      \\EXP{\\, f_{\\eta}(\\x_{K})\\, }\\\\ \n\t\t\t\t\t\t&+\\sum_{{k=\\ell}}^{K-1}\\left( 1-\\tfrac{\\sqrt{n} L_0\\gamma_k}{2\\eta}\\right) \\gamma_k \\EXP{\\, \\|\\mathbf{e}_k\\|^2\\, }.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\tInvoking the definition of $R_{\\ell,K}$ and Lemma~\\ref{lemma:mon_G}, we have \n\t\t\t\t\t\\begin{align} \n\t\t\t\t\t\t\\notag & \\quad \\left( \\tfrac{1}{4}-\\tfrac{\\sqrt{n} L_0\\gamma_0}{4\\eta}\\right) \\left(\\sum_{k=\\ell}^{K-1} \\gamma_k\\right)\\EXP{\\mathbb{E}_{R_{\\ell,K}}\\left[\\|G_{\\eta,1/\\gamma_0} ({\\x_{R_{\\ell,K}}})\\|^2\\right]}\\\\  \n\t\t\t\t\t\t& \\notag \\leq   \\left( \\tfrac{1}{4}-\\tfrac{\\sqrt{n} L_0\\gamma_0}{4\\eta}\\right)\\sum_{{k=\\ell}}^{K-1} \\gamma_k \\mathbb{E}\\left[\\|G_{\\eta,1/\\gamma_0} ({\\x_k})\\|^2\\right]   \\leq\t\t\t\t\t\t   \\EXP{f_{\\eta}(\\x_\\ell)-f^*_{\\eta}}+\\sum_{{k=\\ell}}^{K-1} \\gamma_k \\EXP{\\|\\mathbf{e}_k\\|^2}, \n\t\t\t\t\t\\end{align}\n\t\t\t\t\twhere the last inequality follows from $\\left( 1-\\tfrac{\\sqrt{n} L_0\\gamma_k}{2\\eta}\\right) \\leq 1$ for any $k$. \n\t\t\t\t\t{Note that we have} \n\t\t\t\t\t\\begin{align}\\notag\n\t\t\t\t\t\t\\mathbb{E}\\left[f_{\\eta}(\\x_{\\ell})\\right] - {f^*_{\\eta}} & =   \\mathbb{E}\\left[ f(\\x_{\\ell}) + f_{\\eta}(\\x_{\\ell})-f(\\x_{\\ell})\\right]-f^*_{\\eta} +f^*\n\t\t\t\t\t\t-f^*\\\\\n\t\t\t\t\t\t\\notag & = \\mathbb{E}\\left[{f(\\x_{\\ell})}\\right]-f^*+ \\mathbb{E}\\left[\\left | f_{\\eta}(\\x_{\\ell})-f(\\x_{\\ell})\\right|\\right] + \\left| f^*-f^*_\\eta\\right| \\\\\n\t\t\t\t\t\t& \\stackrel{\\tiny{\\hbox{Lemma~\\ref{lemma:props_local_smoothing} (iii)}}}{\\leq}\n\t\t\t\t\t\t\\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta.\n\t\t\t\t\t\t\\label{bd_feta}\n\t\t\t\t\t\\end{align}\n\t\t\t\t\tFrom Lemma \\ref{lem:stoch_error_var} (ii), \n\t\t\t\t\t$\\mathbb{E}\\left[\\|\\mathbf{e}_k\\|^2\\right] \\leq \\mathbb{E}\\left[\\mathbb{E}\\left[\\|\\mathbf{e}_k\\|^2\\mid \\mathcal{F}_k\\right]\\right] \n\t\t\t\t\t\\leq \\tfrac{ 16\\sqrt{2\\pi}  L_0^2n}{N_k}.$\n\t\t\t\t\tFrom the preceding relations, we obtain the inequality \\eqref{ineq:residual_map_ineq_VRGZO_2}.\n\t\t\t\t\\end{proof}","sha256_utf8":"0030f8527fb82bbfa0eca4ff7597b105f9ba734be6891340692eaa1a0474eba5"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"平滑目标的下降展开","source_units":[{"start":65440,"end":66046,"start_line":1018,"end_line":1025,"raw_tex":"\\begin{proof}\n\t\t\t\t\t(i) From Lemma~\\ref{lemma:props_local_smoothing} (iv), the gradient mapping $\\nabla f_\\eta$ is Lipschitz continuous with the parameter $L_{\\eta}\\triangleq \\tfrac{\\sqrt{n}L_0}{\\eta}$. From the descent lemma, \n\t\t\t\t\t\\begin{align}\\label{eqn:descent_lem_proof_eq1}\n\t\t\t\t\t\\notag\tf_{\\eta}(\\x_{k+1}) & \\leq f_{\\eta}(\\x_{k}) + \\nabla f_{\\eta}(\\x_k)^\\top(\\x_{k+1}-\\x_k)\n\t\t\t\t\t\t+\\tfrac{L_{\\eta}}{2}\\|\\x_{k+1}-\\x_k\\|^2\\\\\n\t\t\t\t\t\\notag\t&=f_\\eta(\\x_k) + \\left(\\nabla f_{\\eta}(\\x_k)+{e_k}\\right)^\\top(\\x_{k+1}-\\x_k)\\\\\n\t\t& -{e_k}^\\top(\\x_{k+1}-\\x_k)+\\tfrac{L_{\\eta}}{2}\\|\\x_{k+1}-\\x_k\\|^2.\n\t\t\t\t\t\\end{align}\n","sha256_utf8":"911788cfaa50a7c1b06df793ad6021b646e1a339f2e96bda6f5d9697c2167bfb"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"投影性质控制更新内积","source_units":[{"start":66046,"end":66458,"start_line":1026,"end_line":1032,"raw_tex":"\t\t\t\t\tInvoking %Lemma \\ref{eqn:proof_comapct_eq1}, \n\t\t\t\t\t\tthe properties of the Euclidean projection, we have \\\\$ \n\t\t\t\t\t(\\x_k-\\gamma_k\\left(\\nabla f_\\eta(\\x_k)+e_k\\right)-\\x_{k+1})^\\top(\\x_k-\\x_{k+1})\\leq 0$. This implies that \n\t\t\t\t\t\\begin{align}\\label{eqn:descent_lem_proof_eq2}\n\t\t\t\t\t\t\\left(\\nabla f_\\eta(\\x_k)+{e_k}\\right)^\\top(\\x_{k+1}-x_k)\n\t\t\t\t\t\t\\leq -\\tfrac{1}{\\gamma_k}\\|\\x_{k+1}-\\x_k\\|^2.\n\t\t\t\t\t\\end{align}\n","sha256_utf8":"d4c3f060a216fc7de3b8d51d6f18bc73f6aa7a2f62b8690e66a13b1eae4b61d5"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"控制误差交叉项","source_units":[{"start":66458,"end":66724,"start_line":1033,"end_line":1037,"raw_tex":"\t\t\t\t\tIn addition, we may also express $e_k^{\\top}(\\x_{k+1}-\\x_k)$ as follows. \n\t\t\t\t\t\\begin{align}\\label{eqn:descent_lem_proof_eq3}\n\t\t\t\t\t\t-e_k^{\\top}(\\x_{k+1}-\\x_k)\n\t\t\t\t\t\t\\leq \\tfrac{{\\gamma_k}}{2}\\|e_k\\|^2+\\tfrac{1}{2{\\gamma_k}}\\|\\x_{k+1}-\\x_k\\|^2.\n\t\t\t\t\t\\end{align}\n","sha256_utf8":"f8c9e78ee0ccc08d05c276592b31f16e3b6c87c514cd0a456f690da58d31d698"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"下降界改写为残差界","source_units":[{"start":66724,"end":67832,"start_line":1038,"end_line":1045,"raw_tex":"\t\t\t\t\tCombining the inequalities \\eqref{eqn:descent_lem_proof_eq1}, \\eqref{eqn:descent_lem_proof_eq2}, and \\eqref{eqn:descent_lem_proof_eq3} we obtain \n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\tf_{\\eta}(\\x_{k+1})  & \\leq   f_{\\eta}(\\x_k)+ \\left( -\\tfrac{1}{2\\gamma_k}+\\tfrac{L_{\\eta}}{2}\\right) \\|\\x_{k+1}-\\x_k\\|^2 +\\tfrac{\\gamma_k}{2} \\|e_k\\|^2 \\\\\n\t& \\overset{\\tiny \\mbox{Def.~\\ref{def:res_maps}}}{=} f_{\\eta}(\\x_k)- \\left( \\tfrac{1}{2\\gamma_k}-\\tfrac{L_{\\eta}}{2}\\right)\\gamma_k^2 \\left\\|\\tilde{G}_{\\eta,1/\\gamma_k}(\\x_k, {e_k})\\right\\|^2+\\tfrac{\\gamma_k}{2} \\|e_k\\|^2\\\\\n\t& = f_{\\eta}(\\x_k)- \\left( 1-L_{\\eta}\\gamma_k\\right)\\tfrac{\\gamma_k}{2} \\left\\|\\tilde{G}_{\\eta,1/\\gamma_k}(\\x_k, {e_k})\\right\\|^2+\\tfrac{\\gamma_k}{2} \\|e_k\\|^2\\\\\n\t& \\hspace{-0.2in}\\overset{\\tiny \\mbox{Lemma~\\ref{lem:inexact_proj_2}}}{\\le} f_{\\eta}(\\x_k)- \\left( 1-L_\\eta \\gamma_k\\right)\\tfrac{\\gamma_k}{4} \\left\\|G_{\\eta,1/\\gamma_k}(\\x_k)\\right\\|^2+\\left(\\tfrac{\\gamma_k}{2} + \\left( 1-L_{\\eta} \\gamma_k\\right)\\tfrac{\\gamma_k}{2}\\right) \\|e_k\\|^2,\n\t\t\t\t\t\\end{align*}\nwhere $1-L_{\\eta}\\gamma_k > 0$ for any $k$  since $\\gamma_0 < \\tfrac{1}{L_{\\eta}}$. \n","sha256_utf8":"982cffe4aa00d99e6b9225970effb7c5415c03f52d9a1079cc2dd524bf5c64f4"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"统一残差步长参数","source_units":[{"start":67832,"end":68193,"start_line":1046,"end_line":1050,"raw_tex":"\t\t\t\t\tFrom the preceding relation and invoking Lemma~\\ref{lemma:mon_G}, we obtain~\\eqref{eqn:recursive_ineq_lemma5}, i.e.,\n\\begin{align*}\n\\left( 1-L_\\eta \\gamma_k\\right)\\tfrac{\\gamma_k}{4} \\left\\|G_{\\eta,1/\\gamma_0}(\\x_k)\\right\\|^2 \\, \\le \\, f_{\\eta}(\\x_k)- f_{\\eta}(\\x_{k+1}) + \\left(1-\\tfrac{L_{\\eta} \\gamma_k}{2}\\right)\\gamma_k \\|e_k\\|^2.\n\\end{align*} \n\t\t\t\t\t\n","sha256_utf8":"010e0fb661f913539ad291228f301eb32c0ae8d0565977fa116760dbfef6f5b0"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"累加下降界并取期望","source_units":[{"start":68193,"end":68797,"start_line":1051,"end_line":1056,"raw_tex":"\t\t\t\t\t\\noindent (ii) Summing \\eqref{eqn:recursive_ineq_lemma5}  from $k =\\ell, \\ldots, K-1$ where $\\ell\\triangleq  \\lceil \\lambda K\\rceil$ and taking expectations with respect to the iterate trajectory, we have\n\t\t\t\t\t\\begin{align}\n\t\t\t\t\t\t\\notag \\sum_{{k=\\ell}}^{K-1} \\left( 1-\\tfrac{ \\sqrt{n} L_0\\gamma_k}{\\eta}\\right) \\tfrac{\\gamma_k}{4}\\EXP{\\, \\|G_{\\eta,1/\\gamma_k} (\\x_k)\\|^2\\, }  \n\t\t\t\t\t\t& \\leq   \\EXP{\\, f_{\\eta}(\\x_\\ell)\\, }-      \\EXP{\\, f_{\\eta}(\\x_{K})\\, }\\\\ \n\t\t\t\t\t\t&+\\sum_{{k=\\ell}}^{K-1}\\left( 1-\\tfrac{\\sqrt{n} L_0\\gamma_k}{2\\eta}\\right) \\gamma_k \\EXP{\\, \\|\\mathbf{e}_k\\|^2\\, }.\n\t\t\t\t\t\\end{align}\n","sha256_utf8":"449427744b16bb7da3f61b923d2631f7f9b354dd4936c914218f1bd5455b1a10"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"转为随机输出残差","source_units":[{"start":68797,"end":69534,"start_line":1057,"end_line":1062,"raw_tex":"\t\t\t\t\tInvoking the definition of $R_{\\ell,K}$ and Lemma~\\ref{lemma:mon_G}, we have \n\t\t\t\t\t\\begin{align} \n\t\t\t\t\t\t\\notag & \\quad \\left( \\tfrac{1}{4}-\\tfrac{\\sqrt{n} L_0\\gamma_0}{4\\eta}\\right) \\left(\\sum_{k=\\ell}^{K-1} \\gamma_k\\right)\\EXP{\\mathbb{E}_{R_{\\ell,K}}\\left[\\|G_{\\eta,1/\\gamma_0} ({\\x_{R_{\\ell,K}}})\\|^2\\right]}\\\\  \n\t\t\t\t\t\t& \\notag \\leq   \\left( \\tfrac{1}{4}-\\tfrac{\\sqrt{n} L_0\\gamma_0}{4\\eta}\\right)\\sum_{{k=\\ell}}^{K-1} \\gamma_k \\mathbb{E}\\left[\\|G_{\\eta,1/\\gamma_0} ({\\x_k})\\|^2\\right]   \\leq\t\t\t\t\t\t   \\EXP{f_{\\eta}(\\x_\\ell)-f^*_{\\eta}}+\\sum_{{k=\\ell}}^{K-1} \\gamma_k \\EXP{\\|\\mathbf{e}_k\\|^2}, \n\t\t\t\t\t\\end{align}\n\t\t\t\t\twhere the last inequality follows from $\\left( 1-\\tfrac{\\sqrt{n} L_0\\gamma_k}{2\\eta}\\right) \\leq 1$ for any $k$. \n","sha256_utf8":"c9472cb0b558e3ff40476a1420b925c1404a96cd147f79b9e988f6745a4fec72"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"控制平滑目标值差","source_units":[{"start":69534,"end":70101,"start_line":1063,"end_line":1071,"raw_tex":"\t\t\t\t\t{Note that we have} \n\t\t\t\t\t\\begin{align}\\notag\n\t\t\t\t\t\t\\mathbb{E}\\left[f_{\\eta}(\\x_{\\ell})\\right] - {f^*_{\\eta}} & =   \\mathbb{E}\\left[ f(\\x_{\\ell}) + f_{\\eta}(\\x_{\\ell})-f(\\x_{\\ell})\\right]-f^*_{\\eta} +f^*\n\t\t\t\t\t\t-f^*\\\\\n\t\t\t\t\t\t\\notag & = \\mathbb{E}\\left[{f(\\x_{\\ell})}\\right]-f^*+ \\mathbb{E}\\left[\\left | f_{\\eta}(\\x_{\\ell})-f(\\x_{\\ell})\\right|\\right] + \\left| f^*-f^*_\\eta\\right| \\\\\n\t\t\t\t\t\t& \\stackrel{\\tiny{\\hbox{Lemma~\\ref{lemma:props_local_smoothing} (iii)}}}{\\leq}\n\t\t\t\t\t\t\\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta.\n\t\t\t\t\t\t\\label{bd_feta}\n\t\t\t\t\t\\end{align}\n","sha256_utf8":"5a216c733300f28605348a55421460a99a414440e1120d9c41f91822bba785a1"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"代入误差方差完成界定","source_units":[{"start":70101,"end":70450,"start_line":1072,"end_line":1076,"raw_tex":"\t\t\t\t\tFrom Lemma \\ref{lem:stoch_error_var} (ii), \n\t\t\t\t\t$\\mathbb{E}\\left[\\|\\mathbf{e}_k\\|^2\\right] \\leq \\mathbb{E}\\left[\\mathbb{E}\\left[\\|\\mathbf{e}_k\\|^2\\mid \\mathcal{F}_k\\right]\\right] \n\t\t\t\t\t\\leq \\tfrac{ 16\\sqrt{2\\pi}  L_0^2n}{N_k}.$\n\t\t\t\t\tFrom the preceding relations, we obtain the inequality \\eqref{ineq:residual_map_ineq_VRGZO_2}.\n\t\t\t\t\\end{proof}","sha256_utf8":"8fc51985abae59269b81446f694f5dfdf005ea72435df346efb7b6b7ad3f10c6"}]},{"id":"S039","semantic_type":"theorem","kind":"theorem","title":"零阶梯度法速率与复杂度","source_units":[{"start":76138,"end":79631,"start_line":1142,"end_line":1164,"raw_tex":"\\begin{theorem}[{\\bf Rate and complexity statements for \\texttt{VRG-ZO}}]\\label{thm:vr-rb-zo}%Alg.~\\ref{algorithm:zo_nonconvex}\n\t\t\t\t\t\\em\n\t\t\t\t\tConsider Algorithm \\ref{algorithm:zo_nonconvex}. Let Assumptions~\\ref{ass-1} and \\ref{assum:random_vars} hold.  Let $a > 0$ be an arbitrary scalar.\n\t\t\t\t\t\n\t\t\t\t\t{\\noindent {\\bf  (a)  [Constant step]}} Let $\\gamma_k:=\\gamma<{\\frac{\\eta}{\\sqrt{n} L_0}}$ and $N_k:=  \\lceil  a \\sqrt{n}L_0 (k+1) \\rceil$ for $k \\geq 0$.  %\\farz{{\\bf TODO:} $a$ cannot be zero, because we use $a^{-1}$ in the proof. I changed $a \\geq 0$ to $a >0$. With this change, the addition of 1 in $N_k:= n L_0 \\lceil 1+a(k+1) \\rceil$ seems unnecessary because $\\lceil a(k+1) \\rceil \\geq 1$ anyway. So I removed 1 in $N_k:= n L_0 \\lceil 1+a(k+1) \\rceil$. I also moved $L_0$ inside the ceiling term because it could be non-integer. Please check.}\n\t\t\t\t\t\n\t\t\t\t\t\\noindent {\\bf{(a-i)}} For $K> \\tfrac{2}{1-\\lambda}$ with $\\ell\\triangleq  \\lceil \\lambda K\\rceil$, we have\n\t\t\t\t\t\\begin{align}\\label{eqn:GBound_constant_gamma}\n\t\t\t\t\t\t\\mathbb{E}\\left[ \\|G_{\\eta,1/\\gamma} (\\x_{R_{\\ell,K}})\\|^2\\right] \\leq %\\left(\\left( 1-\\tfrac{\\lm{\\sqrt{n}}L_0\\gamma_{\\lm{0}}}{\\eta}\\right) \\tfrac{\\gamma}{4}(1-\\lambda)K \\right)^{-1}\\\\\n\t\t\t\t\t\t\\tfrac{\\left(\\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left( {16 \\sqrt{n}} \\sqrt{2\\pi} L_0 \\right)(0.5-\\ln(\\lambda)) {{a^{-1}}}\\gamma\\right)}{\\left(\\left( 1-\\tfrac{{\\sqrt{n}}L_0\\gamma}{\\eta}\\right) \\tfrac{\\gamma}{4}(1-\\lambda)K \\right)}.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\t\\noindent {\\bf{(a-ii)}} Let $\\gamma= {\\tfrac{\\eta}{2{\\sqrt{n}}L_0}}$,  $\\epsilon>0$, and $K_{\\epsilon}$ satisfy $ \\mathbb{E}\\left[ \\|G_{\\eta,1/\\gamma} (\\x_{R_{\\ell,K_{\\epsilon}}})\\|\\right]   \\leq \\epsilon$.  Then the following hold.  {{(a-ii-1)}} The total no. of projection steps is $K_{\\epsilon}=\\mathcal{O}\\left({n^{1/2}}{(L_0\\eta^{-1} +L_0^2)} \\epsilon^{-2}\\right)$.%\\farz{{\\bf TODO:} I don't think $\\eta^{-1}$ will appear in the complexity result here. $\\gamma$ seems to cancel out from denominator in $\\tfrac{\\gamma}{4}(K-\\ell)$ and numerator in $\\gamma\\sum_{k=\\ell}^{K-1} \\tfrac{\\lm{1}}{N_k}$ and there won't remain any $\\eta^{-1}$. Please check. }\n\t\t\t\t\t\n\t\t\t\t\t\\noindent {{(a-ii-2)}} The overall sample complexity is $\\mathcal{O}\\left({n^{3/2}(L_0^3\\eta^{-2}+L_0^5)} \\epsilon^{-4}\\right)$.\n\t\t\t\t\t\n\t\t\t\t\t\\noindent {\\bf  (b)  [Diminishing step]} Let  $\\gamma_k = \\tfrac{\\gamma_0}{\\sqrt{k+1}}$ and {$N_k:=  {\\lceil a{\\sqrt{n}} L_0{\\sqrt{k+1}} \\rceil }$}, where {$\\gamma_0<{{\\eta}{{\\sqrt{n}} L_0}}$}. \n\t\t\t\t {\\bf{(b-i)}}  For $K> \\tfrac{2}{1-\\lambda}$ with $\\ell\\triangleq  \\lceil \\lambda K\\rceil$, we have\n\t\t\t\t\t\\begin{align}\\label{eqn:GBound_diminish_gamma}\n\t\t\t\t\t\t\\mathbb{E}\\left[ \\|G_{\\eta,1/\\gamma_0} (\\x_{R_{\\ell,K}})\\|^2\\right] \\leq \\tfrac{\\left(\\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+  \\left({16  \\sqrt{2\\pi} L_0\\sqrt{n}}\\right)  (0.5-\\ln(\\lambda))\\lm{a^{-1}}\\gamma_0\\right)}{\\left(\\left( 1-\\tfrac{{\\sqrt{n}} L_0\\gamma_0}{\\eta}\\right) {\\tfrac{\\gamma_0}{2}}(1-\\sqrt{\\lambda})\\sqrt{K}\\right)}.\n\t\t\t\t\t\\end{align}\n\t\t\t\t\t\\noindent {\\bf{(b-ii)}}  Let $\\epsilon>0$ and $K_{\\epsilon}$ be such that $ \\mathbb{E}\\left[ \\|G_{\\eta,1/{\\gamma_0}} (\\x_{R_{\\ell,K_\\epsilon}})\\|\\right]   \\leq \\epsilon$. Then the following hold. \n\t\t\t\t\t {{(b-ii-1)}} The total number of projection steps is  $K_{\\epsilon}=\\mathcal{O}\\left({n(L_0^2\\eta^{-2} +L_0^4)} \\epsilon^{-4}\\right).$ {{(b-ii-2)}} The total sample complexity is $\\mathcal{O}\\left({n^{2}(L_0^4\\eta^{-3}+L_0^7)} \\epsilon^{-6}\\right)$.\n\t\t\t\t\\end{theorem}","sha256_utf8":"deccf645d4e26f524b1c362ead6c3d17f68fae4371a87168650c8693a3c58991"}]},{"id":"S040","semantic_type":"proof","kind":"proof","title":"速率与复杂度完整证明","source_units":[{"start":79636,"end":84387,"start_line":1165,"end_line":1218,"raw_tex":"\\begin{proof}\n\t\t\t\t\t\\noindent {\\bf (a-i)} From~\\eqref{ineq:residual_map_ineq_VRGZO_2}, we obtain\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\mathbb{E}\\left[ \\|G_{\\eta,{1/\\gamma}} ({\\x_{R_{\\ell,K}}})\\|^2\\right]\n\t\t\t\t\t\t&\\leq  \n\t\t\t\t\t\t\\tfrac{\\left( \\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left(16 \\sqrt{2\\pi} L_0^2{n} \\right)\\gamma\\sum_{k=\\ell}^{K-1} \\tfrac{1}{N_k}\\right)}{\\left(\\left( 1-\\tfrac{\\sqrt{n}L_0\\gamma}{\\eta}\\right) \\tfrac{\\gamma}{4}(K-\\ell)\\right)} \\\\\n\t\t\t\t\t\t& \\leq \\tfrac{\\left( \\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left({16 \\sqrt{2\\pi} L_0^2 n}\\right){\\gamma}\\sum_{k=\\ell}^{K-1} \\tfrac{a^{-1}}{{L_0}{{\\sqrt{n}}{(k+1)}}}\\right)}{\\left(\\left( 1-\\tfrac{\\sqrt{n}L_0\\gamma}{\\eta}\\right) \\tfrac{\\gamma}{4}(K-\\ell)\\right)},  \n\t\t\t\t\t\\end{align*}\n\t\t\t\t\twhere the expectation is with respect to both the iterate trajectory and the random integer $R_{\\ell,K}$, while the last inequality  follows from the definition of $N_k$. Recall that $K> \\tfrac{2}{1-\\lambda}$ implies $\\ell \\leq K-1$ and  \n\t\t\t\t\t$ \\textstyle\\sum_{k={\\ell}}^{K-1}\\tfrac{1}{k+1}%\\leq  \\tfrac{1}{\\ell+1}+\\ln\\left( \\tfrac{K}{\\ell+1}\\right) \n\t\t\t\t\t\\leq 0.5 +\\ln\\left(\\tfrac{N}{\\lambda N+1}\\right)\\leq 0.5-\\ln(\\lambda).$\n\t\t\t\t\t{Further}, $K-\\ell \\geq K-\\lambda K=(1-\\lambda)K$ {implying that} \\eqref{eqn:GBound_constant_gamma} holds.\n\t\t\t\t\t%\\begin{align*}\n\t\t\t\t\t%    \\mathbb{E}\\left[ \\|G_{\\eta,1/\\gamma} ({\\x_{R_{\\ell,K}}})\\|^2\\right] \\leq \\tfrac{\\left(\\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left(\\uss{16 \\farz{\\sqrt{n}} \\sqrt{2\\pi} L_0}\\right)  (0.5-\\ln(\\lambda)){\\lm{a^{-1}}}\\gamma\\right)}{\\left(\\left( 1-\\tfrac{\\farz{\\sqrt{n}}L_0\\uss{\\gamma}}{\\eta}\\right) \\tfrac{\\gamma}{4}(1-\\lambda)K\\right)}.\n\t\t\t\t\t%\\end{align*}\n\t\t\t\t\t\n\t\t\t\t\t\\noindent {\\bf (a-ii)} To show (ii-1), using the relation in part (i) and by $\\gamma:= \\tfrac{\\eta}{2\\sqrt{n}L_0}$, we obtain\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t& \\mathbb{E}\\left[ \\|G_{\\eta,1/\\gamma} (\\x_{R_{\\ell,K}})\\|^2\\right] \\leq  \n\t\t\t\t\t\t\\tfrac{\\left(\\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left(8\\eta \\sqrt{2\\pi} \\right)  (0.5-\\ln(\\lambda)){a^{-1}}\\right)}{\\left(\\eta(1-\\lambda)K\\right)/(16\\sqrt{n}L_0)}.\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tFrom Jensen's inequality,  $\\mathbb{E}\\left[ \\|G_{\\eta,1/\\gamma} (\\x_{R_{\\ell,K}})\\|\\right] \\leq \\sqrt{\\mathcal{O}\\left({n^{1/2}}{(L_0\\eta^{-1}+L_0^2)}/K\\right)}$, implying that $K_{\\epsilon}=\\mathcal{O}\\left({n^{1/2}}{(L_0\\eta^{-1} +L_0^2)} \\epsilon^{-2}\\right)$. Next, we derive sample complexity as per (ii-2). %The sample complexity of {function evaluations} is as follows.\n\t\t\t\t\t\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\sum_{k=0}^{K_\\epsilon} N_k & = \\sum_{k=0}^{K_\\epsilon}  \\lceil  {a{\\sqrt{n} L_0}(k+1)}\\rceil \n\t\t\t\t\t\t\\leq {\\sqrt{n} L_0}  {\\mathcal{O}(K_{\\epsilon}+{K_\\epsilon^2})}\n\t\t\t\t\t\t\\leq \\mathcal{O}\\left({n^{3/2}(L_0^3\\eta^{-2}+L_0^5)} \\epsilon^{-4}\\right). %\\uss{ \\mbox{CHECK...exponents of $n,L_0$}} \\\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\t%\\farz{{\\bf TODO:} I don't think $\\eta^{-2}$ will appear in the complexity result here. Please see my earlier comment on (a-ii-1).  }\n\t\t\t\t\t\n\t\t\t\t\t\\noindent {\\bf (b-i)}\n\t\t\t\t\t{Consider the inequality~\\eqref{ineq:residual_map_ineq_VRGZO_2}.  Using $\\tfrac{\\gamma_k}{N_k}\\leq \\tfrac{\\gamma_0}{{a{\\sqrt{n}} L_0}(k+1)}$, we obtain} \n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\mathbb{E}\\left[ \\|G_{\\eta,{1/\\gamma_0}} (\\x_{R_{\\ell,K}})\\|^2\\right]\n\t\t\t\t\t\t& \\leq \\tfrac{\\left( \\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left({16  \\sqrt{2\\pi} L_0^2{n}} \\right)\\sum_{k=\\ell}^{K-1} \\tfrac{\\gamma_0}{{a \\sqrt{n} L_0}(k+1)}\\right)}{\\left(\\left( 1-\\tfrac{{\\sqrt{n}} L_0\\gamma_0}{\\eta}\\right) {\\tfrac{\\gamma_0}{2}}(1-\\sqrt{\\lambda})\\sqrt{K}\\right)},\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\twhere we note that $\\textstyle\\sum_{k=\\ell}^{K-1} \\gamma_k \\geq \\textstyle\\int_{\\ell-1}^{K-1} \\tfrac{\\gamma_0}{\\sqrt{x+1}} dx = 2\\gamma_0(\\sqrt{K}-\\sqrt{\\ell}) \\geq 2\\gamma_0(1-\\sqrt{\\lambda})\\sqrt{K}.$ As earlier, we have that  \n\t\t\t\t\t$ \\textstyle\\sum_{k={\\ell}}^{K-1}\\tfrac{1}{k+1} \\leq 0.5-\\ln(\\lambda),$ leading to the inequality in (b-i). \n\t\t\t\t\t\n\t\t\t\t\t\\noindent {\\bf (b-ii)} From Jensen's\n\t\t\t\t\tinequality and by setting $\\gamma_0 =\n\t\t\t\t\t\\tfrac{\\eta}{2 {\\sqrt{n}} L_0}$, we obtain\n\t\t\t\t\tthat \\\\$ \\mathbb{E}\\left[\n\t\t\t\t\t\\|G_{\\eta,1/\\gamma_0}\n\t\t\t\t\t(\\x_{R_{\\ell,K}})\\|\\right] \\leq\n\t\t\t\t\t\\sqrt{\\mathcal{O}\\left({(L_0\\eta^{-1}+L_0^2)\\sqrt{n/K}}\\right)}$, thus\n\t\t\t\t\t$K_{\\epsilon}=\\mathcal{O}\\left(\\tfrac{n(L_0^2\\eta^{-2} +L_0^4)}{\\epsilon^{4}}\\right)$.\\\\ Next, (b-ii-2) holds by\n\t\t\t\t\tbounding the sample complexity as follows.\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\sum_{k=0}^{K_\\epsilon} N_k & = \\sum_{k=0}^{K_\\epsilon}  \\lceil  {a\\sqrt{n} L_0}\\sqrt{k+1}\\rceil \n\t\t\t\t\t\t\\leq {\\sqrt{n} L_0}  {\\mathcal{O}(K_{\\epsilon}+{K_\\epsilon^{3/2}})}\n\t\t\t\t\t\t\\leq \\mathcal{O}\\left({n^{2}(L_0^4\\eta^{-3}+L_0^7)} \\epsilon^{-6}\\right).\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\\end{proof}","sha256_utf8":"0c2470117038a6cea7e259648ff1c1913a971d8750a00933c757f4c2790d23e8"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"常步长随机残差速率","source_units":[{"start":79636,"end":81340,"start_line":1165,"end_line":1180,"raw_tex":"\\begin{proof}\n\t\t\t\t\t\\noindent {\\bf (a-i)} From~\\eqref{ineq:residual_map_ineq_VRGZO_2}, we obtain\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\mathbb{E}\\left[ \\|G_{\\eta,{1/\\gamma}} ({\\x_{R_{\\ell,K}}})\\|^2\\right]\n\t\t\t\t\t\t&\\leq  \n\t\t\t\t\t\t\\tfrac{\\left( \\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left(16 \\sqrt{2\\pi} L_0^2{n} \\right)\\gamma\\sum_{k=\\ell}^{K-1} \\tfrac{1}{N_k}\\right)}{\\left(\\left( 1-\\tfrac{\\sqrt{n}L_0\\gamma}{\\eta}\\right) \\tfrac{\\gamma}{4}(K-\\ell)\\right)} \\\\\n\t\t\t\t\t\t& \\leq \\tfrac{\\left( \\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left({16 \\sqrt{2\\pi} L_0^2 n}\\right){\\gamma}\\sum_{k=\\ell}^{K-1} \\tfrac{a^{-1}}{{L_0}{{\\sqrt{n}}{(k+1)}}}\\right)}{\\left(\\left( 1-\\tfrac{\\sqrt{n}L_0\\gamma}{\\eta}\\right) \\tfrac{\\gamma}{4}(K-\\ell)\\right)},  \n\t\t\t\t\t\\end{align*}\n\t\t\t\t\twhere the expectation is with respect to both the iterate trajectory and the random integer $R_{\\ell,K}$, while the last inequality  follows from the definition of $N_k$. Recall that $K> \\tfrac{2}{1-\\lambda}$ implies $\\ell \\leq K-1$ and  \n\t\t\t\t\t$ \\textstyle\\sum_{k={\\ell}}^{K-1}\\tfrac{1}{k+1}%\\leq  \\tfrac{1}{\\ell+1}+\\ln\\left( \\tfrac{K}{\\ell+1}\\right) \n\t\t\t\t\t\\leq 0.5 +\\ln\\left(\\tfrac{N}{\\lambda N+1}\\right)\\leq 0.5-\\ln(\\lambda).$\n\t\t\t\t\t{Further}, $K-\\ell \\geq K-\\lambda K=(1-\\lambda)K$ {implying that} \\eqref{eqn:GBound_constant_gamma} holds.\n\t\t\t\t\t%\\begin{align*}\n\t\t\t\t\t%    \\mathbb{E}\\left[ \\|G_{\\eta,1/\\gamma} ({\\x_{R_{\\ell,K}}})\\|^2\\right] \\leq \\tfrac{\\left(\\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left(\\uss{16 \\farz{\\sqrt{n}} \\sqrt{2\\pi} L_0}\\right)  (0.5-\\ln(\\lambda)){\\lm{a^{-1}}}\\gamma\\right)}{\\left(\\left( 1-\\tfrac{\\farz{\\sqrt{n}}L_0\\uss{\\gamma}}{\\eta}\\right) \\tfrac{\\gamma}{4}(1-\\lambda)K\\right)}.\n\t\t\t\t\t%\\end{align*}\n\t\t\t\t\t\n","sha256_utf8":"db8bd42043b780a285315e19e3f9781089d309242152e410386a78f24a1f80a6"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"常步长投影次数复杂度","source_units":[{"start":81340,"end":82160,"start_line":1181,"end_line":1186,"raw_tex":"\t\t\t\t\t\\noindent {\\bf (a-ii)} To show (ii-1), using the relation in part (i) and by $\\gamma:= \\tfrac{\\eta}{2\\sqrt{n}L_0}$, we obtain\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t& \\mathbb{E}\\left[ \\|G_{\\eta,1/\\gamma} (\\x_{R_{\\ell,K}})\\|^2\\right] \\leq  \n\t\t\t\t\t\t\\tfrac{\\left(\\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left(8\\eta \\sqrt{2\\pi} \\right)  (0.5-\\ln(\\lambda)){a^{-1}}\\right)}{\\left(\\eta(1-\\lambda)K\\right)/(16\\sqrt{n}L_0)}.\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\tFrom Jensen's inequality,  $\\mathbb{E}\\left[ \\|G_{\\eta,1/\\gamma} (\\x_{R_{\\ell,K}})\\|\\right] \\leq \\sqrt{\\mathcal{O}\\left({n^{1/2}}{(L_0\\eta^{-1}+L_0^2)}/K\\right)}$, implying that $K_{\\epsilon}=\\mathcal{O}\\left({n^{1/2}}{(L_0\\eta^{-1} +L_0^2)} \\epsilon^{-2}\\right)$. Next, we derive sample complexity as per (ii-2). %The sample complexity of {function evaluations} is as follows.\n","sha256_utf8":"28251998a6580bb17adeeebe268d1aada8cf2d6199838e62b201a3c24ac0df4f"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"常步长采样复杂度","source_units":[{"start":82160,"end":82648,"start_line":1187,"end_line":1194,"raw_tex":"\t\t\t\t\t\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\sum_{k=0}^{K_\\epsilon} N_k & = \\sum_{k=0}^{K_\\epsilon}  \\lceil  {a{\\sqrt{n} L_0}(k+1)}\\rceil \n\t\t\t\t\t\t\\leq {\\sqrt{n} L_0}  {\\mathcal{O}(K_{\\epsilon}+{K_\\epsilon^2})}\n\t\t\t\t\t\t\\leq \\mathcal{O}\\left({n^{3/2}(L_0^3\\eta^{-2}+L_0^5)} \\epsilon^{-4}\\right). %\\uss{ \\mbox{CHECK...exponents of $n,L_0$}} \\\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\t%\\farz{{\\bf TODO:} I don't think $\\eta^{-2}$ will appear in the complexity result here. Please see my earlier comment on (a-ii-1).  }\n\t\t\t\t\t\n","sha256_utf8":"a82540fdb70aec18ffb3dea1afe8baa837d1b7042a9435bdb0bbaaa9194c8128"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"递减步长随机残差速率","source_units":[{"start":82648,"end":83606,"start_line":1195,"end_line":1203,"raw_tex":"\t\t\t\t\t\\noindent {\\bf (b-i)}\n\t\t\t\t\t{Consider the inequality~\\eqref{ineq:residual_map_ineq_VRGZO_2}.  Using $\\tfrac{\\gamma_k}{N_k}\\leq \\tfrac{\\gamma_0}{{a{\\sqrt{n}} L_0}(k+1)}$, we obtain} \n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\mathbb{E}\\left[ \\|G_{\\eta,{1/\\gamma_0}} (\\x_{R_{\\ell,K}})\\|^2\\right]\n\t\t\t\t\t\t& \\leq \\tfrac{\\left( \\mathbb{E}\\left[f(\\x_{\\ell})\\right] -f^* +2L_0\\eta+ \\left({16  \\sqrt{2\\pi} L_0^2{n}} \\right)\\sum_{k=\\ell}^{K-1} \\tfrac{\\gamma_0}{{a \\sqrt{n} L_0}(k+1)}\\right)}{\\left(\\left( 1-\\tfrac{{\\sqrt{n}} L_0\\gamma_0}{\\eta}\\right) {\\tfrac{\\gamma_0}{2}}(1-\\sqrt{\\lambda})\\sqrt{K}\\right)},\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\twhere we note that $\\textstyle\\sum_{k=\\ell}^{K-1} \\gamma_k \\geq \\textstyle\\int_{\\ell-1}^{K-1} \\tfrac{\\gamma_0}{\\sqrt{x+1}} dx = 2\\gamma_0(\\sqrt{K}-\\sqrt{\\ell}) \\geq 2\\gamma_0(1-\\sqrt{\\lambda})\\sqrt{K}.$ As earlier, we have that  \n\t\t\t\t\t$ \\textstyle\\sum_{k={\\ell}}^{K-1}\\tfrac{1}{k+1} \\leq 0.5-\\ln(\\lambda),$ leading to the inequality in (b-i). \n\t\t\t\t\t\n","sha256_utf8":"ee6edb7676e3b4be56b6a9828e6bdde001047b96c346399c4859d6d7194b6dd8"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"递减步长投影次数复杂度","source_units":[{"start":83606,"end":84076,"start_line":1204,"end_line":1212,"raw_tex":"\t\t\t\t\t\\noindent {\\bf (b-ii)} From Jensen's\n\t\t\t\t\tinequality and by setting $\\gamma_0 =\n\t\t\t\t\t\\tfrac{\\eta}{2 {\\sqrt{n}} L_0}$, we obtain\n\t\t\t\t\tthat \\\\$ \\mathbb{E}\\left[\n\t\t\t\t\t\\|G_{\\eta,1/\\gamma_0}\n\t\t\t\t\t(\\x_{R_{\\ell,K}})\\|\\right] \\leq\n\t\t\t\t\t\\sqrt{\\mathcal{O}\\left({(L_0\\eta^{-1}+L_0^2)\\sqrt{n/K}}\\right)}$, thus\n\t\t\t\t\t$K_{\\epsilon}=\\mathcal{O}\\left(\\tfrac{n(L_0^2\\eta^{-2} +L_0^4)}{\\epsilon^{4}}\\right)$.\\\\ Next, (b-ii-2) holds by\n\t\t\t\t\tbounding the sample complexity as follows.\n","sha256_utf8":"9b72c86da1955f318ceed9b9547943e946d2e12cea66c8a4f6b016863a8ddc59"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"递减步长采样复杂度","source_units":[{"start":84076,"end":84387,"start_line":1213,"end_line":1218,"raw_tex":"\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t\t\\sum_{k=0}^{K_\\epsilon} N_k & = \\sum_{k=0}^{K_\\epsilon}  \\lceil  {a\\sqrt{n} L_0}\\sqrt{k+1}\\rceil \n\t\t\t\t\t\t\\leq {\\sqrt{n} L_0}  {\\mathcal{O}(K_{\\epsilon}+{K_\\epsilon^{3/2}})}\n\t\t\t\t\t\t\\leq \\mathcal{O}\\left({n^{2}(L_0^4\\eta^{-3}+L_0^7)} \\epsilon^{-6}\\right).\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\\end{proof}","sha256_utf8":"dfca593986e5923604f35f18a6d28004e4fd1f7440d1d89ffae09d2eafd96aac"}]},{"id":"S047","semantic_type":"external_fact","kind":"external_fact","title":"球面集中标准定理","source_units":[],"declared_external":true,"declaration_text":["For an L-Lipschitz function h and V uniform on the unit sphere in R^n, P(|h(V)-E[h(V)]| >= alpha) <= 2 sqrt(2 pi) exp(-n alpha^2/(8 L^2)); this is the cited boundary fact recalled in B0013."]},{"id":"S048","semantic_type":"external_fact","kind":"external_fact","title":"斯托克斯公式与球面对称","source_units":[],"declared_external":true,"declaration_text":["Stokes' theorem gives the gradient of uniform ball smoothing as a spherical boundary integral: grad h_eta(x) = (n/eta) E[h(x+V)V/||V||] for V uniform on eta times the unit sphere. Uniform spherical measure is invariant under V -> -V."]},{"id":"S049","semantic_type":"external_fact","kind":"external_fact","title":"平滑保利普希茨与逼近界","source_units":[],"declared_external":true,"declaration_text":["If h is L0-Lipschitz on X + eta B, its uniform ball smoothing satisfies |h_eta(x)-h_eta(y)| <= L0 ||x-y|| and |h_eta(x)-h(x)| <= L0 eta for x,y in X, as cited from CSY2021MPEC, Lemma 1."]},{"id":"S050","semantic_type":"external_fact","kind":"external_fact","title":"平滑梯度的维数相关界","source_units":[],"declared_external":true,"declaration_text":["The spherical smoothing gradient estimate cited from lin2022, Proposition 2.2, is ||grad h_eta(x)-grad h_eta(y)|| <= (C_n L0/eta)||x-y||, where C_n=(2/pi)^{p_n} n!!/(n-1)!!, p_n=1 for even n and p_n=0 for odd n."]},{"id":"S051","semantic_type":"external_fact","kind":"external_fact","title":"沃利斯积分标准性质","source_units":[],"declared_external":true,"declaration_text":["For w_n = integral from 0 to pi/2 of sin(u)^n du, w_0=pi/2, w_1=1, w_n=((n-1)/n)w_{n-2} for n>=2, and w_{n+1}<w_n. These standard Wallis integral properties and their double-factorial expressions are cited from spivak2006calculus."]},{"id":"S052","semantic_type":"external_fact","kind":"external_fact","title":"投影残差扰动标准界","source_units":[],"declared_external":true,"declaration_text":["For the exact and inexact projected residuals defined in B0023, ||G_{eta,beta}(x)||^2 <= 2||tilde G_{eta,beta}(x,e)||^2 + 2||e||^2 for beta,eta>0 and x in X. This is CSY2021MPEC, Lemma 6, recalled in B0024."]},{"id":"S053","semantic_type":"external_fact","kind":"external_fact","title":"投影残差单调性","source_units":[],"declared_external":true,"declaration_text":["For fixed x and eta>0, the norm of beta(x-Pi_X[x-grad f_eta(x)/beta]) is nondecreasing in beta>0. Thus a nonincreasing positive stepsize sequence gives the residual comparison stated in B0026; the paper attributes this recalled fact to beck23introduction."]},{"id":"S054","semantic_type":"external_fact","kind":"external_fact","title":"下降引理与投影不等式","source_units":[],"declared_external":true,"declaration_text":["For an L-smooth function F, F(y)<=F(x)+grad F(x)^T(y-x)+(L/2)||y-x||^2. For a nonempty closed convex X and p=Pi_X(z), (z-p)^T(x-p)<=0 for x in X. Young's inequality gives |a^T b| <= (t/2)||a||^2+(1/(2t))||b||^2 for t>0."]},{"id":"S055","semantic_type":"external_fact","kind":"external_fact","title":"期望运算与求和标准性质","source_units":[],"declared_external":true,"declaration_text":["Conditional expectations obey the tower property. Independent centered sample vectors have zero cross moments, so their average has variance equal to the sum of individual variances divided by the squared sample count. Jensen's inequality gives E||Z|| <= sqrt(E||Z||^2) and ||EZ||^2 <= E||Z||^2. For Y>=0, EY=integral_0^infinity P(Y>t)dt. Integral comparisons and elementary power-sum estimates bound harmonic, inverse-square-root, linear, and square-root sums."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S039","edges":[{"source_id":"S001","target_id":"S007","kind":"supports"},{"source_id":"S002","target_id":"S037","kind":"supports"},{"source_id":"S002","target_id":"S041","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S003","target_id":"S015","kind":"supports"},{"source_id":"S003","target_id":"S016","kind":"supports"},{"source_id":"S003","target_id":"S017","kind":"supports"},{"source_id":"S004","target_id":"S013","kind":"supports"},{"source_id":"S005","target_id":"S026","kind":"supports"},{"source_id":"S005","target_id":"S027","kind":"supports"},{"source_id":"S005","target_id":"S033","kind":"supports"},{"source_id":"S006","target_id":"S024","kind":"supports"},{"source_id":"S006","target_id":"S031","kind":"supports"},{"source_id":"S006","target_id":"S033","kind":"supports"},{"source_id":"S006","target_id":"S043","kind":"supports"},{"source_id":"S006","target_id":"S046","kind":"supports"},{"source_id":"S007","target_id":"S024","kind":"supports"},{"source_id":"S007","target_id":"S025","kind":"supports"},{"source_id":"S007","target_id":"S030","kind":"supports"},{"source_id":"S007","target_id":"S031","kind":"supports"},{"source_id":"S007","target_id":"S037","kind":"supports"},{"source_id":"S008","target_id":"S024","kind":"supports"},{"source_id":"S008","target_id":"S025","kind":"supports"},{"source_id":"S008","target_id":"S043","kind":"supports"},{"source_id":"S008","target_id":"S046","kind":"supports"},{"source_id":"S009","target_id":"S024","kind":"supports"},{"source_id":"S009","target_id":"S025","kind":"supports"},{"source_id":"S009","target_id":"S030","kind":"supports"},{"source_id":"S009","target_id":"S033","kind":"supports"},{"source_id":"S010","target_id":"S024","kind":"supports"},{"source_id":"S010","target_id":"S025","kind":"supports"},{"source_id":"S011","target_id":"S024","kind":"supports"},{"source_id":"S011","target_id":"S025","kind":"supports"},{"source_id":"S012","target_id":"S021","kind":"supports"},{"source_id":"S013","target_id":"S024","kind":"supports"},{"source_id":"S013","target_id":"S025","kind":"supports"},{"source_id":"S013","target_id":"S030","kind":"supports"},{"source_id":"S013","target_id":"S037","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S016","target_id":"S014","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S014","kind":"supports"},{"source_id":"S018","target_id":"S014","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S014","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S014","kind":"supports"},{"source_id":"S021","target_id":"S014","kind":"supports"},{"source_id":"S022","target_id":"S038","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S023","kind":"supports"},{"source_id":"S026","target_id":"S033","kind":"supports"},{"source_id":"S027","target_id":"S034","kind":"supports"},{"source_id":"S027","target_id":"S036","kind":"supports"},{"source_id":"S028","target_id":"S041","kind":"supports"},{"source_id":"S028","target_id":"S044","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S033","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S033","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S029","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S033","target_id":"S029","kind":"supports"},{"source_id":"S034","target_id":"S036","kind":"supports"},{"source_id":"S034","target_id":"S029","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S029","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S029","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S029","kind":"supports"},{"source_id":"S038","target_id":"S029","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S040","kind":"supports"},{"source_id":"S043","target_id":"S040","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S040","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S040","kind":"supports"},{"source_id":"S046","target_id":"S040","kind":"supports"},{"source_id":"S047","target_id":"S012","kind":"supports"},{"source_id":"S048","target_id":"S015","kind":"supports"},{"source_id":"S048","target_id":"S020","kind":"supports"},{"source_id":"S049","target_id":"S016","kind":"supports"},{"source_id":"S050","target_id":"S017","kind":"supports"},{"source_id":"S051","target_id":"S017","kind":"supports"},{"source_id":"S051","target_id":"S018","kind":"supports"},{"source_id":"S052","target_id":"S026","kind":"supports"},{"source_id":"S053","target_id":"S027","kind":"supports"},{"source_id":"S054","target_id":"S019","kind":"supports"},{"source_id":"S054","target_id":"S020","kind":"supports"},{"source_id":"S054","target_id":"S030","kind":"supports"},{"source_id":"S054","target_id":"S031","kind":"supports"},{"source_id":"S054","target_id":"S032","kind":"supports"},{"source_id":"S055","target_id":"S021","kind":"supports"},{"source_id":"S055","target_id":"S024","kind":"supports"},{"source_id":"S055","target_id":"S025","kind":"supports"},{"source_id":"S055","target_id":"S035","kind":"supports"},{"source_id":"S055","target_id":"S036","kind":"supports"},{"source_id":"S055","target_id":"S038","kind":"supports"},{"source_id":"S055","target_id":"S041","kind":"supports"},{"source_id":"S055","target_id":"S042","kind":"supports"},{"source_id":"S055","target_id":"S043","kind":"supports"},{"source_id":"S055","target_id":"S044","kind":"supports"},{"source_id":"S055","target_id":"S045","kind":"supports"},{"source_id":"S055","target_id":"S046","kind":"supports"}]}}
{"record_id":"2401.17899v1","paper":{"title":"On Tractability, Complexity, and Mixed-Integer Convex Programming Representability of Distributionally Favorable Optimization","authors":["Nan Jiang","Weijun Xie"],"arxiv_id":"2401.17899","version":"v1","journal":"Mathematical Programming","doi":"10.1007/s10107-025-02299-w","abstract_url":"https://arxiv.org/abs/2401.17899v1","pdf_url":"https://arxiv.org/pdf/2401.17899v1"},"context":{"raw_tex":"\\begin{equation}\nv^*=\\min_{\\bm x \\in \\X}\\inf_{\\Pr\\in\\P}\\E_{\\Pr}\\left[Q(\\bm x, \\trxi)\\right], \\label{dfo}\n\\end{equation}\n\nwhere \n% the linear function $\\bm c^\\top \\bm x$ represents the deterministic objective function, \nset $\\X\\subseteq \\Re^n$ is deterministic, set $\\P$ denotes an ambiguity set with $\\P \\subseteq \\{ \\Pr\\colon\\Pr\\{\\trxi\\in {\\U}\\}=1 \\}$ and support $\\U\\subseteq\\Re^m$ (also called ``uncertainty set\" throughout this paper), and function $Q(\\bm x, {\\trxi}):\\Re^n\\times \\U\\to \\Re$. \n\n\nNote that if only support information $\\U$ is available (i.e., $\\P = \\{ \\Pr\\colon\\Pr\\{\\trxi\\in {\\U}\\}=1 \\}$), then DFO \\eqref{dfo} simplifies to a singular DFO (sDFO), i.e.,\n\\begin{align}\nv^*=\\min_{\\bm x\\in\\X}  \\inf_{\\rxi\\in\\U} Q(\\bm x,\\rxi).\\label{eq_uq_simple}\n\\end{align}\n\n\\begin{assumption}\\label{assum_0}\n Set $\\X$ is compact and has a non-empty relative interior.\n\\end{assumption}\n\n\\noindent\\textbf{Notation.} The following notation is used throughout the paper. We use bold letters (e.g., $\\bm{x}, \\bm{A}$) to denote vectors and matrices and use corresponding non-bold letters to denote their components. %Given a vector or matrix $\\bm{x}$, its zero norm $\\left \\| \\bm{x}\\right \\|_0$ denotes the number of its nonzero elements. \nWe let $\\|\\cdot\\|_*$ denote the dual norm of a general norm $\\|\\cdot\\|$. We let $\\bm{e}$ be the vector or matrix of all ones, and let $\\bm{e}_i$ be the $i$th standard basis vector.\nGiven an integer $n$, we let $[n]:=\\{1,2,\\ldots,n\\}$, and use $\\Re_+^n:=\\{\\bm {x}\\in \\Re^n:x_i\\geq0, \\forall i\\in [n]\\}$. Given a real number $t$, we let $(t)_+:=\\max\\{t,0\\}$ and $(t)_-:=\\min\\{t,0\\}$. \nGiven a set $I$, we use $\\mathrm{ext}\\{I\\}$ to represent its extreme points. \nWe let $\\tilde{\\bm\\xi}$ denote a random vector and denote its realizations by $\\bm\\xi$. \n\n\n\\begin{definition}\\label{def_tract}\n(Tractability, theorem A.3.3 in \\cite{ben2009robust}) Suppose that for any given compact set $\\X\\subseteq \\Re^n$, which has a nonempty relative interior and is contained in a Euclidean ball with radius $R$ and is containing a Euclidean ball with radius $r$, then there exists an efficient algorithm to solve the favorable problem $\\min_{\\bm x\\in\\X}\\inf_{\\Pr\\in\\P} \\E_{\\Pr}[Q(\\bm x,\\trxi)]$ to $\\hat\\varepsilon>0$ accuracy, whose running time is polynomial in $n,m,\\ln(R/r),\\ln(1/\\hat\\varepsilon)$, and the encoding length of $\\min_{\\bm x\\in\\X}\\inf_{\\Pr\\in\\P} \\E_{\\Pr}[Q(\\bm x,\\trxi)]$.\n\\end{definition}\n\nfunction $Q(\\bm x, \\rxi)$ is the minimum of $K$ piecewise affine functions $\\rxi^\\top \\bm a_k(\\bm x)+b_k(\\bm x)$ with affine mappings $\\bm a_k(\\bm x)=\\hat{\\bm{A}}_k \\bm{x}+\\hat{\\bm a}_k\\in \\Re^{m}$ with $\\hat{\\bm{A}}_k\\in \\Re^{m\\times n},\\hat{\\bm a}_k\\in \\Re^m$ and $b_k(\\bm x)=\\hat{\\bm{B}}_k^\\top \\bm{x}+\\hat{b}_k\\in \\Re$ with $\\hat{\\bm{B}}_k\\in \\Re^{ n},\\hat{ b}_k\\in \\Re$ for each $k\\in[K]$, i.e., $Q(\\bm x, \\rxi)=\\min_{k\\in[K]}[\\rxi^\\top \\bm a_k(\\bm x)+b_k(\\bm x)]$. Suppose that the uncertainty set is defined as the ball $\\U=\\{\\rxi : \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\}$ with the known parameter $\\rxi^0$ and the radius $\\theta\\geq 0$. It is worth mentioning that the reformulations and complexity analyses can be simply extended to more general uncertainty sets, such as polyhedral and ellipsoidal (see, e.g., \\cite{ben2009robust}). In this subsection, for brevity, we focus on the ball uncertainty set $\\U$. In this setting, sDFO \\eqref{eq_uq_simple} becomes\n\n\nIn this subsection, we follow the same notation and uncertainty set as the previous subsection (Section~\\ref{eq_uq_simple_cases_concave}) and consider the maximum of piecewise affine function, that is, the function $Q(\\bm x, \\rxi)$ is defined as the maximum of $K$ piecewise affine function $\\rxi^\\top \\bm a_k(\\bm x)+b_k(\\bm x)$, i.e., $Q(\\bm x, \\rxi)=\\max_{k\\in[K]}[\\rxi^\\top \\bm a_k(\\bm x)+b_k(\\bm x)]$.  \nIn this setting, sDFO \\eqref{eq_uq_simple} can be recast as\n\\begin{align}\nv^* = \\min_{\\bm x\\in\\X}\\min_{\\rxi\\in\\U} \\max_{k\\in[K]}  \\left\\{\\rxi^\\top \\bm a_k(\\bm x)+b_k(\\bm x)\\right\\}. \\label{robust_max_b}\n\\end{align}","units":[{"start":11643,"end":11762,"start_line":366,"end_line":368,"raw_tex":"\\begin{equation}\nv^*=\\min_{\\bm x \\in \\X}\\inf_{\\Pr\\in\\P}\\E_{\\Pr}\\left[Q(\\bm x, \\trxi)\\right], \\label{dfo}\n\\end{equation}","sha256_utf8":"96f905705ff96f6856ebb7b4dc2000a1ff19cfa7812fa6432f65630efb29b042"},{"start":11763,"end":12139,"start_line":369,"end_line":371,"raw_tex":"where \n% the linear function $\\bm c^\\top \\bm x$ represents the deterministic objective function, \nset $\\X\\subseteq \\Re^n$ is deterministic, set $\\P$ denotes an ambiguity set with $\\P \\subseteq \\{ \\Pr\\colon\\Pr\\{\\trxi\\in {\\U}\\}=1 \\}$ and support $\\U\\subseteq\\Re^m$ (also called ``uncertainty set\" throughout this paper), and function $Q(\\bm x, {\\trxi}):\\Re^n\\times \\U\\to \\Re$. \n","sha256_utf8":"3e1eef04590b3d4aee6c0d18719d907da82d02cf7a7499a8bbd406b5e9165190"},{"start":12172,"end":12446,"start_line":373,"end_line":376,"raw_tex":"Note that if only support information $\\U$ is available (i.e., $\\P = \\{ \\Pr\\colon\\Pr\\{\\trxi\\in {\\U}\\}=1 \\}$), then DFO \\eqref{dfo} simplifies to a singular DFO (sDFO), i.e.,\n\\begin{align}\nv^*=\\min_{\\bm x\\in\\X}  \\inf_{\\rxi\\in\\U} Q(\\bm x,\\rxi).\\label{eq_uq_simple}\n\\end{align}","sha256_utf8":"739657c0d6048741964313baabc2473a7109e29da73a06d18c058ab5caf8f285"},{"start":16666,"end":16776,"start_line":404,"end_line":406,"raw_tex":"\\begin{assumption}\\label{assum_0}\n Set $\\X$ is compact and has a non-empty relative interior.\n\\end{assumption}","sha256_utf8":"9f293f85f006974d2d437712e0a391f2ed61b77fde11fd8ea366923e09dc44af"},{"start":25657,"end":26555,"start_line":456,"end_line":460,"raw_tex":"\\noindent\\textbf{Notation.} The following notation is used throughout the paper. We use bold letters (e.g., $\\bm{x}, \\bm{A}$) to denote vectors and matrices and use corresponding non-bold letters to denote their components. %Given a vector or matrix $\\bm{x}$, its zero norm $\\left \\| \\bm{x}\\right \\|_0$ denotes the number of its nonzero elements. \nWe let $\\|\\cdot\\|_*$ denote the dual norm of a general norm $\\|\\cdot\\|$. We let $\\bm{e}$ be the vector or matrix of all ones, and let $\\bm{e}_i$ be the $i$th standard basis vector.\nGiven an integer $n$, we let $[n]:=\\{1,2,\\ldots,n\\}$, and use $\\Re_+^n:=\\{\\bm {x}\\in \\Re^n:x_i\\geq0, \\forall i\\in [n]\\}$. Given a real number $t$, we let $(t)_+:=\\max\\{t,0\\}$ and $(t)_-:=\\min\\{t,0\\}$. \nGiven a set $I$, we use $\\mathrm{ext}\\{I\\}$ to represent its extreme points. \nWe let $\\tilde{\\bm\\xi}$ denote a random vector and denote its realizations by $\\bm\\xi$. \n","sha256_utf8":"e3c84349526ac38f917f12f767e4fd144373e9197016279b087bf343f046f729"},{"start":32340,"end":32978,"start_line":515,"end_line":517,"raw_tex":"\\begin{definition}\\label{def_tract}\n(Tractability, theorem A.3.3 in \\cite{ben2009robust}) Suppose that for any given compact set $\\X\\subseteq \\Re^n$, which has a nonempty relative interior and is contained in a Euclidean ball with radius $R$ and is containing a Euclidean ball with radius $r$, then there exists an efficient algorithm to solve the favorable problem $\\min_{\\bm x\\in\\X}\\inf_{\\Pr\\in\\P} \\E_{\\Pr}[Q(\\bm x,\\trxi)]$ to $\\hat\\varepsilon>0$ accuracy, whose running time is polynomial in $n,m,\\ln(R/r),\\ln(1/\\hat\\varepsilon)$, and the encoding length of $\\min_{\\bm x\\in\\X}\\inf_{\\Pr\\in\\P} \\E_{\\Pr}[Q(\\bm x,\\trxi)]$.\n\\end{definition}","sha256_utf8":"e66c995da00e8b01424a082d24230aa67ba4cce5f3466339133fe493807af631"},{"start":42303,"end":43265,"start_line":632,"end_line":632,"raw_tex":"function $Q(\\bm x, \\rxi)$ is the minimum of $K$ piecewise affine functions $\\rxi^\\top \\bm a_k(\\bm x)+b_k(\\bm x)$ with affine mappings $\\bm a_k(\\bm x)=\\hat{\\bm{A}}_k \\bm{x}+\\hat{\\bm a}_k\\in \\Re^{m}$ with $\\hat{\\bm{A}}_k\\in \\Re^{m\\times n},\\hat{\\bm a}_k\\in \\Re^m$ and $b_k(\\bm x)=\\hat{\\bm{B}}_k^\\top \\bm{x}+\\hat{b}_k\\in \\Re$ with $\\hat{\\bm{B}}_k\\in \\Re^{ n},\\hat{ b}_k\\in \\Re$ for each $k\\in[K]$, i.e., $Q(\\bm x, \\rxi)=\\min_{k\\in[K]}[\\rxi^\\top \\bm a_k(\\bm x)+b_k(\\bm x)]$. Suppose that the uncertainty set is defined as the ball $\\U=\\{\\rxi : \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\}$ with the known parameter $\\rxi^0$ and the radius $\\theta\\geq 0$. It is worth mentioning that the reformulations and complexity analyses can be simply extended to more general uncertainty sets, such as polyhedral and ellipsoidal (see, e.g., \\cite{ben2009robust}). In this subsection, for brevity, we focus on the ball uncertainty set $\\U$. In this setting, sDFO \\eqref{eq_uq_simple} becomes\n","sha256_utf8":"b6d3a560721bee15e6e904bb2b8669a350d5f1950cd5cf1107c9a6793ad51ca6"},{"start":54180,"end":54802,"start_line":774,"end_line":778,"raw_tex":"In this subsection, we follow the same notation and uncertainty set as the previous subsection (Section~\\ref{eq_uq_simple_cases_concave}) and consider the maximum of piecewise affine function, that is, the function $Q(\\bm x, \\rxi)$ is defined as the maximum of $K$ piecewise affine function $\\rxi^\\top \\bm a_k(\\bm x)+b_k(\\bm x)$, i.e., $Q(\\bm x, \\rxi)=\\max_{k\\in[K]}[\\rxi^\\top \\bm a_k(\\bm x)+b_k(\\bm x)]$.  \nIn this setting, sDFO \\eqref{eq_uq_simple} can be recast as\n\\begin{align}\nv^* = \\min_{\\bm x\\in\\X}\\min_{\\rxi\\in\\U} \\max_{k\\in[K]}  \\left\\{\\rxi^\\top \\bm a_k(\\bm x)+b_k(\\bm x)\\right\\}. \\label{robust_max_b}\n\\end{align}","sha256_utf8":"6d75e334f197ce84db08f5dcd3996629de00f758bfb1baf55b51dc4fa9f3d16f"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{restatable}{theorem}{proprobustmaxtractable}\\label{prop_robust_max_tractable} \nsDFO \\eqref{robust_max_b} can be tractable if any of the following conditions holds:\n\\begin{itemize}\n\\item[(i)] When $ \\bm a_k(\\bm x):=\\bar{\\bm a}_k$ is constant for all $k\\in[K]$ and $\\bm{x}\\in \\X$,  sDFO \\eqref{robust_max_b} is equivalent to\n\\begin{align*}\nv^*=\\min_{\\bm x\\in\\X,\\rxi,\\eta}  \\left\\{\\eta\\colon\\eta\\geq \\rxi^\\top \\bar{\\bm a}_k+b_k(\\bm x), \\forall k\\in[K], \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\right\\};\n\\end{align*}\n\\item[(ii)] When $p=1$ and  $a_k(\\bm x)=a_1(\\bm x)$ for each $k\\in[K]$, sDFO  \\eqref{robust_max_b} is equivalent to solving $2m$ tractable convex programs, and selecting the lowest optimal value, i.e., $v^*=\\min_{i\\in[m],\\ell\\in [2]}v_{i\\ell}^*$, where for each $i\\in[m]$, we have\n\\begin{align*}\n& v_{i1}^*=\\min_{\\bm x\\in\\X,\\eta}  \\left\\{\\eta\\colon \\eta\\geq  b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x)- \\theta a_{1i}(\\bm x),\\forall k\\in[K] \\right\\},\\\\\n& v_{i2}^*=\\min_{\\bm x\\in\\X,\\eta}  \\left\\{\\eta\\colon \\eta\\geq  b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x) + \\theta a_{1i}(\\bm x),\\forall k\\in[K] \\right\\};\n\\end{align*}\n\\item[(iii)] Suppose that $p=1$,  and $\\rxi:=[\\rxi_1,\\ldots,\\rxi_K]$ such that $\\rxi^i$ and $\\rxi^j$ do not overlap for each $i\\neq j$, and $\\rxi^\\top \\bm a_k(\\bm x)= {\\rxi^k}^\\top \\bar{\\bm a}_{k}(\\bm x) $ for each $k\\in[K]$ such that $\\|\\bar{\\bm a}_k(\\bm x) \\|_{\\infty}=C_k$ is constant for each $k\\in[K]$, where $\\rxi_k\\in \\Re^{m_k}$, $\\bar{\\bm a}_k(\\bm x)=\\bar{\\bm{A}}_k \\bm{x}+\\bar{\\bm a}_k\\in \\Re^{m_k}$ with $\\bar{\\bm{A}}_k\\in \\Re^{m_k\\times n},\\bar{\\bm a}_k\\in \\Re^{m_k}$ such that $\\bar{\\bm{A}}_i$ and $\\bar{\\bm{A}}_j$, $\\bar{\\bm{a}}_i$ and $\\bar{\\bm{a}}_j$ do not overlap each $i\\neq j$ with $\\sum_{k\\in[K]}m_k=m$ and each $m_k$ is nonnegative. \nThen,  sDFO \\eqref{robust_max_b} is equivalent to solving\n\\begin{align*}\nv^*=\\min_{\\bm x\\in\\X,\\beta,\\bm\\gamma\\geq \\bm 0} \\left\\{\\beta\\colon \\sum_{k\\in[K]} \\gamma_{k}= \\theta,\\beta \\geq  {\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x)+b_k(\\bm x)-\\gamma_{k}C_k,\\forall k\\in[K]  \\right\\}.\n\\end{align*}\n\\end{itemize}\n\\end{restatable}","units":[{"start":57458,"end":59557,"start_line":817,"end_line":835,"raw_tex":"\\begin{restatable}{theorem}{proprobustmaxtractable}\\label{prop_robust_max_tractable} \nsDFO \\eqref{robust_max_b} can be tractable if any of the following conditions holds:\n\\begin{itemize}\n\\item[(i)] When $ \\bm a_k(\\bm x):=\\bar{\\bm a}_k$ is constant for all $k\\in[K]$ and $\\bm{x}\\in \\X$,  sDFO \\eqref{robust_max_b} is equivalent to\n\\begin{align*}\nv^*=\\min_{\\bm x\\in\\X,\\rxi,\\eta}  \\left\\{\\eta\\colon\\eta\\geq \\rxi^\\top \\bar{\\bm a}_k+b_k(\\bm x), \\forall k\\in[K], \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\right\\};\n\\end{align*}\n\\item[(ii)] When $p=1$ and  $a_k(\\bm x)=a_1(\\bm x)$ for each $k\\in[K]$, sDFO  \\eqref{robust_max_b} is equivalent to solving $2m$ tractable convex programs, and selecting the lowest optimal value, i.e., $v^*=\\min_{i\\in[m],\\ell\\in [2]}v_{i\\ell}^*$, where for each $i\\in[m]$, we have\n\\begin{align*}\n& v_{i1}^*=\\min_{\\bm x\\in\\X,\\eta}  \\left\\{\\eta\\colon \\eta\\geq  b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x)- \\theta a_{1i}(\\bm x),\\forall k\\in[K] \\right\\},\\\\\n& v_{i2}^*=\\min_{\\bm x\\in\\X,\\eta}  \\left\\{\\eta\\colon \\eta\\geq  b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x) + \\theta a_{1i}(\\bm x),\\forall k\\in[K] \\right\\};\n\\end{align*}\n\\item[(iii)] Suppose that $p=1$,  and $\\rxi:=[\\rxi_1,\\ldots,\\rxi_K]$ such that $\\rxi^i$ and $\\rxi^j$ do not overlap for each $i\\neq j$, and $\\rxi^\\top \\bm a_k(\\bm x)= {\\rxi^k}^\\top \\bar{\\bm a}_{k}(\\bm x) $ for each $k\\in[K]$ such that $\\|\\bar{\\bm a}_k(\\bm x) \\|_{\\infty}=C_k$ is constant for each $k\\in[K]$, where $\\rxi_k\\in \\Re^{m_k}$, $\\bar{\\bm a}_k(\\bm x)=\\bar{\\bm{A}}_k \\bm{x}+\\bar{\\bm a}_k\\in \\Re^{m_k}$ with $\\bar{\\bm{A}}_k\\in \\Re^{m_k\\times n},\\bar{\\bm a}_k\\in \\Re^{m_k}$ such that $\\bar{\\bm{A}}_i$ and $\\bar{\\bm{A}}_j$, $\\bar{\\bm{a}}_i$ and $\\bar{\\bm{a}}_j$ do not overlap each $i\\neq j$ with $\\sum_{k\\in[K]}m_k=m$ and each $m_k$ is nonnegative. \nThen,  sDFO \\eqref{robust_max_b} is equivalent to solving\n\\begin{align*}\nv^*=\\min_{\\bm x\\in\\X,\\beta,\\bm\\gamma\\geq \\bm 0} \\left\\{\\beta\\colon \\sum_{k\\in[K]} \\gamma_{k}= \\theta,\\beta \\geq  {\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x)+b_k(\\bm x)-\\gamma_{k}C_k,\\forall k\\in[K]  \\right\\}.\n\\end{align*}\n\\end{itemize}\n\\end{restatable}","sha256_utf8":"83dc7e42a5ff163788a3a8689f2bb1f389030fb16db3c5dd5a37fb82ca4bfbba"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{restatable}{lemma}{robustmaxrefor}\\label{robust_max_refor} \nsDFO \\eqref{robust_max_b} is equivalent to \n\\begin{align}\nv^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top \\bm a_k(\\bm x) +b_k(\\bm x) \\right] -  \\theta \\left\\|\\sum_{k\\in[K]} \\lambda_k \\bm a_k(\\bm x) \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}. \\label{robust_max_eq}\n\\end{align}\n\\end{restatable}\n\n\\begin{proof}\nLet us first consider the inner minimax of sDFO \\eqref{robust_max_b} as\n\\begin{align*}\n\\min_{\\rxi} \\left\\{\\max_{k\\in[K]}  \\rxi^\\top \\bm a_k(\\bm x)+b_k(\\bm x)\\colon \\|\\bm \\xi-\\bm \\xi^0\\|_p\\leq \\theta\\right\\}.\n\\end{align*}\nIntroducing auxiliary nonnegative variables $\\bm\\lambda$, the inner minimax of sDFO \\eqref{robust_max_b} is equivalent to\n\\begin{align*}\n\\min_{\\rxi}\\left\\{\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi}^\\top \\bm a_k(\\bm x)+b_k(\\bm x)\\right]\\colon  \\sum_{k\\in[K]} \\lambda_k =1\\right\\}: \\|\\bm \\xi-\\bm \\xi^0\\|_p\\leq \\theta\\right\\}.\n\\end{align*} \nAccording to Sion's minimax theorem \\cite{sion1958general}, we can interchange the maximum operator with the minimum one as\n\\begin{align*}\n\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\min_{\\rxi}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi}^\\top \\bm a_k(\\bm x)+b_k(\\bm x)\\right]\\colon \\|\\bm \\xi-\\bm \\xi^0\\|_p\\leq \\theta  \\right\\}:\\sum_{k\\in[K]} \\lambda_k =1\\right\\}.\n\\end{align*}\nInvoking the definition of dual norm, we have \n\\begin{align*}\n\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top \\bm a_k(\\bm x) +b_k(\\bm x) \\right] -  \\theta \\left\\|\\sum_{k\\in[K]} \\lambda_k \\bm a_k(\\bm x) \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}.\n\\end{align*}\n% and DFO \\eqref{robust_max_b} can be written as\n% \\begin{align*}\n% v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top \\bm a_k(\\bm x) +b_k(\\bm x) \\right] -  \\theta \\left\\|\\sum_{k\\in[K]} \\lambda_k \\bm a_k(\\bm x) \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}.\n% \\end{align*}\nThis completes the proof.\n\\QEDA\n\\end{proof}\n\n\\begin{proof}\nWe split the proof into three parts accordingly.\n\\begin{itemize}\n\\item[(i)] When $\\bm a_k(\\bm x)=\\bar{\\bm a}_k$ is constant for all $k\\in[K]$, sDFO \\eqref{robust_max_b} can be written as\n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top \\bm C_k +b_k(\\bm x) \\right] -  \\theta \\left\\|\\sum_{k\\in[K]} \\lambda_k \\bm C_k \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\},\n% \\end{align*}\n% or\n\\begin{align*}\nv^*=\\min_{\\bm x\\in\\X} \\min_{\\rxi} \\max_{k\\in[K]}  \\left\\{\\rxi^\\top \\bar{\\bm a}_k+b_k(\\bm x)\\colon \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\right\\}.\n\\end{align*}\nIntroducing an auxiliary variable $\\eta$ to linearize the inner maximum, we arrive at Part (i).\n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X,\\rxi,\\eta}  \\left\\{\\eta\\colon\\eta\\geq \\rxi^\\top \\bar{\\bm a}_k+b_k(\\bm x), \\forall k\\in[K], \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\right\\},\n% \\end{align*}\n% which can be a tractable convex optimization problem.\n% Since the decision $\\bm x$ does not influence the objective value, we can drop the first minimum and the problem can be written as\n% \\begin{align*}\n%  v^*= \\min_{\\rxi} \\max_{k\\in[K]}  \\left\\{\\rxi^\\top \\bm C_k+b_k\\colon \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\right\\}.\n% \\end{align*}\n% Letting $\\hat\\rxi=\\rxi-\\rxi^0$, then we have\n% \\begin{align*}\n%  v^*=\\min_{\\hat\\rxi} \\max_{k\\in[K]}  \\left\\{\\hat\\rxi^\\top \\bm C_k + {\\rxi^0}^\\top \\bm C_k  +b_k\\colon \\|\\hat\\rxi\\|_p\\leq \\theta\\right\\},\n% \\end{align*}\n% and the optimal objective value can found by solving the optimization problem on each piece and select the largest one, i.e., $v^* = \\max_{k\\in[K]}v_k^*$, where\n% \\begin{align*}\n%   v_k^*= \\min_{\\rxi} \\left\\{\\rxi^\\top \\bm C_k+b_k\\colon \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\right\\}.\n% \\end{align*}\n\n\n% and\n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq \\bm 0}  \\min_{\\hat\\rxi}  \\left\\{\\sum_{k\\in[K]}\\lambda_k\\left[{\\rxi^0}^\\top \\bm C_k +\\hat\\rxi^\\top \\bm C_k +b_k(\\bm x)\\right]\\colon \\|\\hat\\rxi\\|_p\\leq \\theta,\\sum_{k\\in[K]}\\lambda_k=1\\right\\},\n% \\end{align*}\n% which can be simplified as\n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top \\bm C_k +b_k(\\bm x) \\right] -  \\theta\\left\\| \\sum_{k\\in[K]} \\lambda_k \\bm C_k \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}.\n% \\end{align*}\n% If $\\bm C_k=\\bm C_1$ for all $k\\in[K]$, then \n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{  \\sum_{k\\in[K]} \\lambda_k  b_k(\\bm x) + {\\rxi^0}^\\top \\bm C_1 -  \\theta\\left\\| \\bm C_1 \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\},\n% \\end{align*}\n% and the problem is equivalent to \n% % \\begin{align*}\n% %   \\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{  \\sum_{k\\in[K]} \\lambda_k  b_k(\\bm x) \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\},\n% % \\end{align*}\n% % and \n% \\begin{align*}\n%   \\min_{\\bm x\\in\\X,\\eta}\\left\\{  \\eta\\colon \\eta\\geq b_k(\\bm x)-{\\rxi^0}^\\top \\bm C_1 -  \\theta\\left\\| \\bm C_1 \\right\\|_{p^*}   \\right\\}.\n% \\end{align*}\n\n% Otherwise, let $\\rxi^0=\\bm 0 $ and $b_k(\\bm x) =0$, then the problem reduces to \n% \\begin{align}\n%  \\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{ -  \\theta\\left\\| \\sum_{k\\in[K]} \\lambda_k \\bm C_k \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}\n% \\end{align}\n% and\n% \\begin{align*}\n%  \\min_{\\bm\\lambda\\geq  \\bm 0}\\left\\{ \\left\\| \\sum_{k\\in[K]} \\lambda_k \\bm C_k \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}\n% \\end{align*}\n% According to theorem 1 \\cite{ge2011note}, this problem can be NP-hard with general norm. \n\n% and\n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top C_k +b_k(\\bm x) -  \\theta C_k   \\right] \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\},\n% \\end{align*}\n% Taking the dual and  using strong duality from linear programming, we have \n% \\begin{align*}\n%   \\min_{\\bm x\\in\\X,\\eta}\\left\\{\\eta\\colon \\eta\\geq \\left[{\\rxi^0}^\\top \\bm a_k(\\bm x) +b_k(\\bm x) \\right] -  \\theta C,\\forall k\\in[K]  \\right\\},  \n% \\end{align*}\n% which can be tractable.\n\\item[(ii)] When $\\bm a_k(\\bm x)=\\bm a_1(\\bm x)$  for each $k\\in[K]$, we rewrite sDFO  \\eqref{robust_max_eq} as\n\\begin{align*}\nv^*=\\min_{\\bm x\\in\\X} \\max_{\\bm\\lambda\\geq  \\bm 0} \\left\\{\\sum_{k\\in[K]} \\lambda_k b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x) - \\theta \\| \\bm a_1(\\bm x) \\|_{p^*}\\colon  \\sum_{k\\in[K]} \\lambda_k =1  \\right\\}.\n\\end{align*}\nTaking the dual of the inner maximization problem and using strong duality from linear programming, we have \n\\begin{align*}\nv^*=\\min_{\\bm x\\in\\X,\\eta}  \\left\\{\\eta\\colon \\eta\\geq  b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x)- \\theta \\| \\bm a_1(\\bm x) \\|_{p^*}, \\forall k\\in[K] \\right\\}.\n%\\label{robust_max_micp_special_tractable_1}\n\\end{align*}\nWhen $p=1$, i.e., when the dual norm is $L_\\infty$, \nthen $\\theta \\|\\bm a_1(\\bm x) \\|_\\infty =  \\theta \\max_{i\\in [m]}\\max\\{ a_{1i}(\\bm x),-a_{1i}(\\bm x)\\}$. Therefore, sDFO \\eqref{robust_max_eq} is equivalent to solving $2m$ tractable problems, and selecting the best one with the lowest optimal value, i.e., $v^*=\\min_{i\\in[m],\\ell\\in [2]}v_{i\\ell}^*$, where for each $i\\in[m]$, we have\n\\begin{align*}\n& v_{i1}^*=\\min_{\\bm x\\in\\X,\\eta}  \\left\\{\\eta\\colon \\eta\\geq  b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x)- \\theta a_{1i}(\\bm x),\\forall k\\in[K] \\right\\},\\\\\n& v_{i2}^*=\\min_{\\bm x\\in\\X,\\eta}  \\left\\{\\eta\\colon \\eta\\geq  b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x) + \\theta a_{1i}(\\bm x),\\forall k\\in[K] \\right\\}.\n\\end{align*}\n% which can be derived from \\eqref{robust_min_micp} by letting $k=K=1$. \n% Therefore, according to the results in \\Cref{prop_robust_min_tractable}, when $p=1$, DFO \\eqref{robust_max_micp_special_case_1} can be tractable.\n\\item[(iii)] Since $\\rxi^i$ and $\\rxi^j$ do not overlap for each $i\\neq j$ and $\\bar{\\bm a}_{j}(\\bm x) $ and $\\bar{\\bm a}_{j}(\\bm x) $ do not overlap for each $i\\neq j$ as well, when $p=1$ and $\\|\\bar{\\bm a}_k(\\bm x) \\|_{\\infty}=C_k$ is constant for each $k\\in[K]$, the dual norm term in \\eqref{robust_max_eq} can be simplified as\n\\begin{align*}\n\\left\\|\\sum_{k\\in[K]} \\lambda_k \\bm a_k(\\bm x) \\right\\|_{\\infty} = \\max_{k\\in[K]} \\lambda_k\\left\\|\\bar{\\bm a}_k(\\bm x) \\right\\|_{\\infty}=\\max_{k\\in[K]} \\lambda_k C_k.\n\\end{align*}\nThen, sDFO \\eqref{robust_max_eq} can be written as\n\\begin{align*}\n v^*=  \\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x) \\right] + \\sum_{k\\in[K]} \\lambda_k  b_k(\\bm x) -  \\theta \\max_{k\\in[K]} \\lambda_kC_k \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}.\n\\end{align*}\n% which is\n% \\begin{align*}\n% \\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x) \\right] + \\sum_{k\\in[K]} \\lambda_k  b_k(\\bm x) -  \\theta \\max_{k\\in[K]}\\max_{i\\in[n]} \\lambda_k|\\bar{a}_{ki}(\\bm x)| \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}.\n% \\end{align*}\nIntroducing one variable $\\eta$ to linearize the term $\\max_{k\\in[K]} \\lambda_kC_k$, then we have\n\\begin{align*}\n\\max_{\\bm\\lambda\\geq  \\bm 0,\\eta}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x) \\right] + \\sum_{k\\in[K]} \\lambda_k  b_k(\\bm x) -  \\theta \\eta  \\colon  \\sum_{k\\in[K]} \\lambda_k =1,   \\lambda_kC_k-\\eta\\leq 0, \\forall k\\in[K] \\right\\}. \n\\end{align*}\nTaking the dual of the inner maximization problem with dual variables $\\beta,\\bm\\gamma$ and using strong duality from linear programming, we have \n\\begin{align*}\n\\min_{\\beta,\\bm\\gamma\\geq \\bm 0} \\left\\{\\beta\\colon \\sum_{k\\in[K]} \\gamma_{k}= \\theta,\\beta \\geq  {\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x)+b_k(\\bm x)-\\gamma_{k}C_k,\\forall k\\in[K]  \\right\\}.\n\\end{align*}\nThis completes the proof.\\QEDA\n% \\begin{align*}\n%  \\min_{\\beta,\\bm\\gamma\\geq \\bm 0}\\quad & \\beta,\\\\\n%  \\textup{s.t.} \\quad & \\sum_{k\\in[K]} \\gamma_{k}\\leq \\theta,\\\\\n%  & \\gamma_{k}\\left\\|\\bar{\\bm a}_k(\\bm x) \\right\\|_{\\infty}+\\beta \\geq  {\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x)+b_k(\\bm x),\\forall k\\in[K].\n% \\end{align*}\n% Taking the dual of the inner maximization problem and using strong duality from linear programming, DFO \\eqref{robust_max_eq} is equivalent to solving $2nK$ tractable programs and selecting the best one with the lowest optimal value, i.e., $v^*=\\min_{k\\in [K],i\\in[n],\\ell\\in [2]}v_{ik\\ell}^*$, where for each $k\\in [K]$ and $i\\in[n]$, we have\n%   \\begin{align*}\n% & v_{ik1}^*=\\min_{\\bm x\\in\\X,\\eta}\\left\\{\\eta\\colon \\eta\\geq{\\rxi^0}^\\top \\bar{\\bm a}_{k}(\\bm x) + b_k(\\bm x) - \\theta \\bar{a}_{ki}(\\bm x), \\eta\\geq{\\rxi^0}^\\top \\bar{\\bm a}_{j}(\\bm x) + b_j(\\bm x), \\forall j\\in[K], j\\neq k\\right\\},\\\\\n% & v_{ik2}^*=\\min_{\\bm x\\in\\X,\\eta }\\left\\{\\eta\\colon \\eta\\geq{\\rxi^0}^\\top \\bar{\\bm a}_{k}(\\bm x) + b_k(\\bm x) + \\theta \\bar{a}_{ki}(\\bm x), \\eta\\geq{\\rxi^0}^\\top \\bar{\\bm a}_{j}(\\bm x) + b_j(\\bm x), \\forall j\\in[K], j\\neq k\\right\\}.\n%   \\end{align*}\n\\end{itemize}\n\\end{proof}","units":[{"start":54973,"end":55403,"start_line":780,"end_line":785,"raw_tex":"\\begin{restatable}{lemma}{robustmaxrefor}\\label{robust_max_refor} \nsDFO \\eqref{robust_max_b} is equivalent to \n\\begin{align}\nv^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top \\bm a_k(\\bm x) +b_k(\\bm x) \\right] -  \\theta \\left\\|\\sum_{k\\in[K]} \\lambda_k \\bm a_k(\\bm x) \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}. \\label{robust_max_eq}\n\\end{align}\n\\end{restatable}","sha256_utf8":"15c2bce0c95f1e70aa630b4be026a5ace5807e8e37ed3b14b21ae80908248c7e"},{"start":55487,"end":57149,"start_line":790,"end_line":813,"raw_tex":"\\begin{proof}\nLet us first consider the inner minimax of sDFO \\eqref{robust_max_b} as\n\\begin{align*}\n\\min_{\\rxi} \\left\\{\\max_{k\\in[K]}  \\rxi^\\top \\bm a_k(\\bm x)+b_k(\\bm x)\\colon \\|\\bm \\xi-\\bm \\xi^0\\|_p\\leq \\theta\\right\\}.\n\\end{align*}\nIntroducing auxiliary nonnegative variables $\\bm\\lambda$, the inner minimax of sDFO \\eqref{robust_max_b} is equivalent to\n\\begin{align*}\n\\min_{\\rxi}\\left\\{\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi}^\\top \\bm a_k(\\bm x)+b_k(\\bm x)\\right]\\colon  \\sum_{k\\in[K]} \\lambda_k =1\\right\\}: \\|\\bm \\xi-\\bm \\xi^0\\|_p\\leq \\theta\\right\\}.\n\\end{align*} \nAccording to Sion's minimax theorem \\cite{sion1958general}, we can interchange the maximum operator with the minimum one as\n\\begin{align*}\n\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\min_{\\rxi}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi}^\\top \\bm a_k(\\bm x)+b_k(\\bm x)\\right]\\colon \\|\\bm \\xi-\\bm \\xi^0\\|_p\\leq \\theta  \\right\\}:\\sum_{k\\in[K]} \\lambda_k =1\\right\\}.\n\\end{align*}\nInvoking the definition of dual norm, we have \n\\begin{align*}\n\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top \\bm a_k(\\bm x) +b_k(\\bm x) \\right] -  \\theta \\left\\|\\sum_{k\\in[K]} \\lambda_k \\bm a_k(\\bm x) \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}.\n\\end{align*}\n% and DFO \\eqref{robust_max_b} can be written as\n% \\begin{align*}\n% v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top \\bm a_k(\\bm x) +b_k(\\bm x) \\right] -  \\theta \\left\\|\\sum_{k\\in[K]} \\lambda_k \\bm a_k(\\bm x) \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}.\n% \\end{align*}\nThis completes the proof.\n\\QEDA\n\\end{proof}","sha256_utf8":"8a530e858f1dd553a6792e77fb973b575a93ae6edec29fc46e2fd1803b0f792e"},{"start":59650,"end":68513,"start_line":840,"end_line":959,"raw_tex":"\\begin{proof}\nWe split the proof into three parts accordingly.\n\\begin{itemize}\n\\item[(i)] When $\\bm a_k(\\bm x)=\\bar{\\bm a}_k$ is constant for all $k\\in[K]$, sDFO \\eqref{robust_max_b} can be written as\n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top \\bm C_k +b_k(\\bm x) \\right] -  \\theta \\left\\|\\sum_{k\\in[K]} \\lambda_k \\bm C_k \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\},\n% \\end{align*}\n% or\n\\begin{align*}\nv^*=\\min_{\\bm x\\in\\X} \\min_{\\rxi} \\max_{k\\in[K]}  \\left\\{\\rxi^\\top \\bar{\\bm a}_k+b_k(\\bm x)\\colon \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\right\\}.\n\\end{align*}\nIntroducing an auxiliary variable $\\eta$ to linearize the inner maximum, we arrive at Part (i).\n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X,\\rxi,\\eta}  \\left\\{\\eta\\colon\\eta\\geq \\rxi^\\top \\bar{\\bm a}_k+b_k(\\bm x), \\forall k\\in[K], \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\right\\},\n% \\end{align*}\n% which can be a tractable convex optimization problem.\n% Since the decision $\\bm x$ does not influence the objective value, we can drop the first minimum and the problem can be written as\n% \\begin{align*}\n%  v^*= \\min_{\\rxi} \\max_{k\\in[K]}  \\left\\{\\rxi^\\top \\bm C_k+b_k\\colon \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\right\\}.\n% \\end{align*}\n% Letting $\\hat\\rxi=\\rxi-\\rxi^0$, then we have\n% \\begin{align*}\n%  v^*=\\min_{\\hat\\rxi} \\max_{k\\in[K]}  \\left\\{\\hat\\rxi^\\top \\bm C_k + {\\rxi^0}^\\top \\bm C_k  +b_k\\colon \\|\\hat\\rxi\\|_p\\leq \\theta\\right\\},\n% \\end{align*}\n% and the optimal objective value can found by solving the optimization problem on each piece and select the largest one, i.e., $v^* = \\max_{k\\in[K]}v_k^*$, where\n% \\begin{align*}\n%   v_k^*= \\min_{\\rxi} \\left\\{\\rxi^\\top \\bm C_k+b_k\\colon \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\right\\}.\n% \\end{align*}\n\n\n% and\n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq \\bm 0}  \\min_{\\hat\\rxi}  \\left\\{\\sum_{k\\in[K]}\\lambda_k\\left[{\\rxi^0}^\\top \\bm C_k +\\hat\\rxi^\\top \\bm C_k +b_k(\\bm x)\\right]\\colon \\|\\hat\\rxi\\|_p\\leq \\theta,\\sum_{k\\in[K]}\\lambda_k=1\\right\\},\n% \\end{align*}\n% which can be simplified as\n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top \\bm C_k +b_k(\\bm x) \\right] -  \\theta\\left\\| \\sum_{k\\in[K]} \\lambda_k \\bm C_k \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}.\n% \\end{align*}\n% If $\\bm C_k=\\bm C_1$ for all $k\\in[K]$, then \n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{  \\sum_{k\\in[K]} \\lambda_k  b_k(\\bm x) + {\\rxi^0}^\\top \\bm C_1 -  \\theta\\left\\| \\bm C_1 \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\},\n% \\end{align*}\n% and the problem is equivalent to \n% % \\begin{align*}\n% %   \\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{  \\sum_{k\\in[K]} \\lambda_k  b_k(\\bm x) \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\},\n% % \\end{align*}\n% % and \n% \\begin{align*}\n%   \\min_{\\bm x\\in\\X,\\eta}\\left\\{  \\eta\\colon \\eta\\geq b_k(\\bm x)-{\\rxi^0}^\\top \\bm C_1 -  \\theta\\left\\| \\bm C_1 \\right\\|_{p^*}   \\right\\}.\n% \\end{align*}\n\n% Otherwise, let $\\rxi^0=\\bm 0 $ and $b_k(\\bm x) =0$, then the problem reduces to \n% \\begin{align}\n%  \\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{ -  \\theta\\left\\| \\sum_{k\\in[K]} \\lambda_k \\bm C_k \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}\n% \\end{align}\n% and\n% \\begin{align*}\n%  \\min_{\\bm\\lambda\\geq  \\bm 0}\\left\\{ \\left\\| \\sum_{k\\in[K]} \\lambda_k \\bm C_k \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}\n% \\end{align*}\n% According to theorem 1 \\cite{ge2011note}, this problem can be NP-hard with general norm. \n\n% and\n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top C_k +b_k(\\bm x) -  \\theta C_k   \\right] \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\},\n% \\end{align*}\n% Taking the dual and  using strong duality from linear programming, we have \n% \\begin{align*}\n%   \\min_{\\bm x\\in\\X,\\eta}\\left\\{\\eta\\colon \\eta\\geq \\left[{\\rxi^0}^\\top \\bm a_k(\\bm x) +b_k(\\bm x) \\right] -  \\theta C,\\forall k\\in[K]  \\right\\},  \n% \\end{align*}\n% which can be tractable.\n\\item[(ii)] When $\\bm a_k(\\bm x)=\\bm a_1(\\bm x)$  for each $k\\in[K]$, we rewrite sDFO  \\eqref{robust_max_eq} as\n\\begin{align*}\nv^*=\\min_{\\bm x\\in\\X} \\max_{\\bm\\lambda\\geq  \\bm 0} \\left\\{\\sum_{k\\in[K]} \\lambda_k b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x) - \\theta \\| \\bm a_1(\\bm x) \\|_{p^*}\\colon  \\sum_{k\\in[K]} \\lambda_k =1  \\right\\}.\n\\end{align*}\nTaking the dual of the inner maximization problem and using strong duality from linear programming, we have \n\\begin{align*}\nv^*=\\min_{\\bm x\\in\\X,\\eta}  \\left\\{\\eta\\colon \\eta\\geq  b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x)- \\theta \\| \\bm a_1(\\bm x) \\|_{p^*}, \\forall k\\in[K] \\right\\}.\n%\\label{robust_max_micp_special_tractable_1}\n\\end{align*}\nWhen $p=1$, i.e., when the dual norm is $L_\\infty$, \nthen $\\theta \\|\\bm a_1(\\bm x) \\|_\\infty =  \\theta \\max_{i\\in [m]}\\max\\{ a_{1i}(\\bm x),-a_{1i}(\\bm x)\\}$. Therefore, sDFO \\eqref{robust_max_eq} is equivalent to solving $2m$ tractable problems, and selecting the best one with the lowest optimal value, i.e., $v^*=\\min_{i\\in[m],\\ell\\in [2]}v_{i\\ell}^*$, where for each $i\\in[m]$, we have\n\\begin{align*}\n& v_{i1}^*=\\min_{\\bm x\\in\\X,\\eta}  \\left\\{\\eta\\colon \\eta\\geq  b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x)- \\theta a_{1i}(\\bm x),\\forall k\\in[K] \\right\\},\\\\\n& v_{i2}^*=\\min_{\\bm x\\in\\X,\\eta}  \\left\\{\\eta\\colon \\eta\\geq  b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x) + \\theta a_{1i}(\\bm x),\\forall k\\in[K] \\right\\}.\n\\end{align*}\n% which can be derived from \\eqref{robust_min_micp} by letting $k=K=1$. \n% Therefore, according to the results in \\Cref{prop_robust_min_tractable}, when $p=1$, DFO \\eqref{robust_max_micp_special_case_1} can be tractable.\n\\item[(iii)] Since $\\rxi^i$ and $\\rxi^j$ do not overlap for each $i\\neq j$ and $\\bar{\\bm a}_{j}(\\bm x) $ and $\\bar{\\bm a}_{j}(\\bm x) $ do not overlap for each $i\\neq j$ as well, when $p=1$ and $\\|\\bar{\\bm a}_k(\\bm x) \\|_{\\infty}=C_k$ is constant for each $k\\in[K]$, the dual norm term in \\eqref{robust_max_eq} can be simplified as\n\\begin{align*}\n\\left\\|\\sum_{k\\in[K]} \\lambda_k \\bm a_k(\\bm x) \\right\\|_{\\infty} = \\max_{k\\in[K]} \\lambda_k\\left\\|\\bar{\\bm a}_k(\\bm x) \\right\\|_{\\infty}=\\max_{k\\in[K]} \\lambda_k C_k.\n\\end{align*}\nThen, sDFO \\eqref{robust_max_eq} can be written as\n\\begin{align*}\n v^*=  \\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x) \\right] + \\sum_{k\\in[K]} \\lambda_k  b_k(\\bm x) -  \\theta \\max_{k\\in[K]} \\lambda_kC_k \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}.\n\\end{align*}\n% which is\n% \\begin{align*}\n% \\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x) \\right] + \\sum_{k\\in[K]} \\lambda_k  b_k(\\bm x) -  \\theta \\max_{k\\in[K]}\\max_{i\\in[n]} \\lambda_k|\\bar{a}_{ki}(\\bm x)| \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}.\n% \\end{align*}\nIntroducing one variable $\\eta$ to linearize the term $\\max_{k\\in[K]} \\lambda_kC_k$, then we have\n\\begin{align*}\n\\max_{\\bm\\lambda\\geq  \\bm 0,\\eta}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x) \\right] + \\sum_{k\\in[K]} \\lambda_k  b_k(\\bm x) -  \\theta \\eta  \\colon  \\sum_{k\\in[K]} \\lambda_k =1,   \\lambda_kC_k-\\eta\\leq 0, \\forall k\\in[K] \\right\\}. \n\\end{align*}\nTaking the dual of the inner maximization problem with dual variables $\\beta,\\bm\\gamma$ and using strong duality from linear programming, we have \n\\begin{align*}\n\\min_{\\beta,\\bm\\gamma\\geq \\bm 0} \\left\\{\\beta\\colon \\sum_{k\\in[K]} \\gamma_{k}= \\theta,\\beta \\geq  {\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x)+b_k(\\bm x)-\\gamma_{k}C_k,\\forall k\\in[K]  \\right\\}.\n\\end{align*}\nThis completes the proof.\\QEDA\n% \\begin{align*}\n%  \\min_{\\beta,\\bm\\gamma\\geq \\bm 0}\\quad & \\beta,\\\\\n%  \\textup{s.t.} \\quad & \\sum_{k\\in[K]} \\gamma_{k}\\leq \\theta,\\\\\n%  & \\gamma_{k}\\left\\|\\bar{\\bm a}_k(\\bm x) \\right\\|_{\\infty}+\\beta \\geq  {\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x)+b_k(\\bm x),\\forall k\\in[K].\n% \\end{align*}\n% Taking the dual of the inner maximization problem and using strong duality from linear programming, DFO \\eqref{robust_max_eq} is equivalent to solving $2nK$ tractable programs and selecting the best one with the lowest optimal value, i.e., $v^*=\\min_{k\\in [K],i\\in[n],\\ell\\in [2]}v_{ik\\ell}^*$, where for each $k\\in [K]$ and $i\\in[n]$, we have\n%   \\begin{align*}\n% & v_{ik1}^*=\\min_{\\bm x\\in\\X,\\eta}\\left\\{\\eta\\colon \\eta\\geq{\\rxi^0}^\\top \\bar{\\bm a}_{k}(\\bm x) + b_k(\\bm x) - \\theta \\bar{a}_{ki}(\\bm x), \\eta\\geq{\\rxi^0}^\\top \\bar{\\bm a}_{j}(\\bm x) + b_j(\\bm x), \\forall j\\in[K], j\\neq k\\right\\},\\\\\n% & v_{ik2}^*=\\min_{\\bm x\\in\\X,\\eta }\\left\\{\\eta\\colon \\eta\\geq{\\rxi^0}^\\top \\bar{\\bm a}_{k}(\\bm x) + b_k(\\bm x) + \\theta \\bar{a}_{ki}(\\bm x), \\eta\\geq{\\rxi^0}^\\top \\bar{\\bm a}_{j}(\\bm x) + b_j(\\bm x), \\forall j\\in[K], j\\neq k\\right\\}.\n%   \\end{align*}\n\\end{itemize}\n\\end{proof}","sha256_utf8":"7bad4d24ec64d80dd6e6a9c2d641b72f4bee40e9a5ed1d548d44ec9ac9ae7153"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"分布有利优化模型","source_units":[{"start":11643,"end":11762,"start_line":366,"end_line":368,"raw_tex":"\\begin{equation}\nv^*=\\min_{\\bm x \\in \\X}\\inf_{\\Pr\\in\\P}\\E_{\\Pr}\\left[Q(\\bm x, \\trxi)\\right], \\label{dfo}\n\\end{equation}","sha256_utf8":"96f905705ff96f6856ebb7b4dc2000a1ff19cfa7812fa6432f65630efb29b042"},{"start":11763,"end":12139,"start_line":369,"end_line":371,"raw_tex":"where \n% the linear function $\\bm c^\\top \\bm x$ represents the deterministic objective function, \nset $\\X\\subseteq \\Re^n$ is deterministic, set $\\P$ denotes an ambiguity set with $\\P \\subseteq \\{ \\Pr\\colon\\Pr\\{\\trxi\\in {\\U}\\}=1 \\}$ and support $\\U\\subseteq\\Re^m$ (also called ``uncertainty set\" throughout this paper), and function $Q(\\bm x, {\\trxi}):\\Re^n\\times \\U\\to \\Re$. \n","sha256_utf8":"3e1eef04590b3d4aee6c0d18719d907da82d02cf7a7499a8bbd406b5e9165190"}]},{"id":"S002","semantic_type":"problem_setting","kind":"problem_setting","title":"仅含支撑信息的模型","source_units":[{"start":12172,"end":12446,"start_line":373,"end_line":376,"raw_tex":"Note that if only support information $\\U$ is available (i.e., $\\P = \\{ \\Pr\\colon\\Pr\\{\\trxi\\in {\\U}\\}=1 \\}$), then DFO \\eqref{dfo} simplifies to a singular DFO (sDFO), i.e.,\n\\begin{align}\nv^*=\\min_{\\bm x\\in\\X}  \\inf_{\\rxi\\in\\U} Q(\\bm x,\\rxi).\\label{eq_uq_simple}\n\\end{align}","sha256_utf8":"739657c0d6048741964313baabc2473a7109e29da73a06d18c058ab5caf8f285"}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","title":"决策域的紧性假设","source_units":[{"start":16666,"end":16776,"start_line":404,"end_line":406,"raw_tex":"\\begin{assumption}\\label{assum_0}\n Set $\\X$ is compact and has a non-empty relative interior.\n\\end{assumption}","sha256_utf8":"9f293f85f006974d2d437712e0a391f2ed61b77fde11fd8ea366923e09dc44af"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"计算可解性的定义","source_units":[{"start":32340,"end":32978,"start_line":515,"end_line":517,"raw_tex":"\\begin{definition}\\label{def_tract}\n(Tractability, theorem A.3.3 in \\cite{ben2009robust}) Suppose that for any given compact set $\\X\\subseteq \\Re^n$, which has a nonempty relative interior and is contained in a Euclidean ball with radius $R$ and is containing a Euclidean ball with radius $r$, then there exists an efficient algorithm to solve the favorable problem $\\min_{\\bm x\\in\\X}\\inf_{\\Pr\\in\\P} \\E_{\\Pr}[Q(\\bm x,\\trxi)]$ to $\\hat\\varepsilon>0$ accuracy, whose running time is polynomial in $n,m,\\ln(R/r),\\ln(1/\\hat\\varepsilon)$, and the encoding length of $\\min_{\\bm x\\in\\X}\\inf_{\\Pr\\in\\P} \\E_{\\Pr}[Q(\\bm x,\\trxi)]$.\n\\end{definition}","sha256_utf8":"e66c995da00e8b01424a082d24230aa67ba4cce5f3466339133fe493807af631"}]},{"id":"S005","semantic_type":"notation","kind":"notation","title":"向量与对偶范数记号","source_units":[{"start":25657,"end":26555,"start_line":456,"end_line":460,"raw_tex":"\\noindent\\textbf{Notation.} The following notation is used throughout the paper. We use bold letters (e.g., $\\bm{x}, \\bm{A}$) to denote vectors and matrices and use corresponding non-bold letters to denote their components. %Given a vector or matrix $\\bm{x}$, its zero norm $\\left \\| \\bm{x}\\right \\|_0$ denotes the number of its nonzero elements. \nWe let $\\|\\cdot\\|_*$ denote the dual norm of a general norm $\\|\\cdot\\|$. We let $\\bm{e}$ be the vector or matrix of all ones, and let $\\bm{e}_i$ be the $i$th standard basis vector.\nGiven an integer $n$, we let $[n]:=\\{1,2,\\ldots,n\\}$, and use $\\Re_+^n:=\\{\\bm {x}\\in \\Re^n:x_i\\geq0, \\forall i\\in [n]\\}$. Given a real number $t$, we let $(t)_+:=\\max\\{t,0\\}$ and $(t)_-:=\\min\\{t,0\\}$. \nGiven a set $I$, we use $\\mathrm{ext}\\{I\\}$ to represent its extreme points. \nWe let $\\tilde{\\bm\\xi}$ denote a random vector and denote its realizations by $\\bm\\xi$. \n","sha256_utf8":"e3c84349526ac38f917f12f767e4fd144373e9197016279b087bf343f046f729"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"仿射系数与范数球","source_units":[{"start":42303,"end":43265,"start_line":632,"end_line":632,"raw_tex":"function $Q(\\bm x, \\rxi)$ is the minimum of $K$ piecewise affine functions $\\rxi^\\top \\bm a_k(\\bm x)+b_k(\\bm x)$ with affine mappings $\\bm a_k(\\bm x)=\\hat{\\bm{A}}_k \\bm{x}+\\hat{\\bm a}_k\\in \\Re^{m}$ with $\\hat{\\bm{A}}_k\\in \\Re^{m\\times n},\\hat{\\bm a}_k\\in \\Re^m$ and $b_k(\\bm x)=\\hat{\\bm{B}}_k^\\top \\bm{x}+\\hat{b}_k\\in \\Re$ with $\\hat{\\bm{B}}_k\\in \\Re^{ n},\\hat{ b}_k\\in \\Re$ for each $k\\in[K]$, i.e., $Q(\\bm x, \\rxi)=\\min_{k\\in[K]}[\\rxi^\\top \\bm a_k(\\bm x)+b_k(\\bm x)]$. Suppose that the uncertainty set is defined as the ball $\\U=\\{\\rxi : \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\}$ with the known parameter $\\rxi^0$ and the radius $\\theta\\geq 0$. It is worth mentioning that the reformulations and complexity analyses can be simply extended to more general uncertainty sets, such as polyhedral and ellipsoidal (see, e.g., \\cite{ben2009robust}). In this subsection, for brevity, we focus on the ball uncertainty set $\\U$. In this setting, sDFO \\eqref{eq_uq_simple} becomes\n","sha256_utf8":"b6d3a560721bee15e6e904bb2b8669a350d5f1950cd5cf1107c9a6793ad51ca6"}]},{"id":"S007","semantic_type":"problem_setting","kind":"problem_setting","title":"凸分段仿射有利优化","source_units":[{"start":54180,"end":54802,"start_line":774,"end_line":778,"raw_tex":"In this subsection, we follow the same notation and uncertainty set as the previous subsection (Section~\\ref{eq_uq_simple_cases_concave}) and consider the maximum of piecewise affine function, that is, the function $Q(\\bm x, \\rxi)$ is defined as the maximum of $K$ piecewise affine function $\\rxi^\\top \\bm a_k(\\bm x)+b_k(\\bm x)$, i.e., $Q(\\bm x, \\rxi)=\\max_{k\\in[K]}[\\rxi^\\top \\bm a_k(\\bm x)+b_k(\\bm x)]$.  \nIn this setting, sDFO \\eqref{eq_uq_simple} can be recast as\n\\begin{align}\nv^* = \\min_{\\bm x\\in\\X}\\min_{\\rxi\\in\\U} \\max_{k\\in[K]}  \\left\\{\\rxi^\\top \\bm a_k(\\bm x)+b_k(\\bm x)\\right\\}. \\label{robust_max_b}\n\\end{align}","sha256_utf8":"6d75e334f197ce84db08f5dcd3996629de00f758bfb1baf55b51dc4fa9f3d16f"}]},{"id":"S008","semantic_type":"theorem","kind":"theorem","title":"三类情形的可解重构","source_units":[{"start":57458,"end":59557,"start_line":817,"end_line":835,"raw_tex":"\\begin{restatable}{theorem}{proprobustmaxtractable}\\label{prop_robust_max_tractable} \nsDFO \\eqref{robust_max_b} can be tractable if any of the following conditions holds:\n\\begin{itemize}\n\\item[(i)] When $ \\bm a_k(\\bm x):=\\bar{\\bm a}_k$ is constant for all $k\\in[K]$ and $\\bm{x}\\in \\X$,  sDFO \\eqref{robust_max_b} is equivalent to\n\\begin{align*}\nv^*=\\min_{\\bm x\\in\\X,\\rxi,\\eta}  \\left\\{\\eta\\colon\\eta\\geq \\rxi^\\top \\bar{\\bm a}_k+b_k(\\bm x), \\forall k\\in[K], \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\right\\};\n\\end{align*}\n\\item[(ii)] When $p=1$ and  $a_k(\\bm x)=a_1(\\bm x)$ for each $k\\in[K]$, sDFO  \\eqref{robust_max_b} is equivalent to solving $2m$ tractable convex programs, and selecting the lowest optimal value, i.e., $v^*=\\min_{i\\in[m],\\ell\\in [2]}v_{i\\ell}^*$, where for each $i\\in[m]$, we have\n\\begin{align*}\n& v_{i1}^*=\\min_{\\bm x\\in\\X,\\eta}  \\left\\{\\eta\\colon \\eta\\geq  b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x)- \\theta a_{1i}(\\bm x),\\forall k\\in[K] \\right\\},\\\\\n& v_{i2}^*=\\min_{\\bm x\\in\\X,\\eta}  \\left\\{\\eta\\colon \\eta\\geq  b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x) + \\theta a_{1i}(\\bm x),\\forall k\\in[K] \\right\\};\n\\end{align*}\n\\item[(iii)] Suppose that $p=1$,  and $\\rxi:=[\\rxi_1,\\ldots,\\rxi_K]$ such that $\\rxi^i$ and $\\rxi^j$ do not overlap for each $i\\neq j$, and $\\rxi^\\top \\bm a_k(\\bm x)= {\\rxi^k}^\\top \\bar{\\bm a}_{k}(\\bm x) $ for each $k\\in[K]$ such that $\\|\\bar{\\bm a}_k(\\bm x) \\|_{\\infty}=C_k$ is constant for each $k\\in[K]$, where $\\rxi_k\\in \\Re^{m_k}$, $\\bar{\\bm a}_k(\\bm x)=\\bar{\\bm{A}}_k \\bm{x}+\\bar{\\bm a}_k\\in \\Re^{m_k}$ with $\\bar{\\bm{A}}_k\\in \\Re^{m_k\\times n},\\bar{\\bm a}_k\\in \\Re^{m_k}$ such that $\\bar{\\bm{A}}_i$ and $\\bar{\\bm{A}}_j$, $\\bar{\\bm{a}}_i$ and $\\bar{\\bm{a}}_j$ do not overlap each $i\\neq j$ with $\\sum_{k\\in[K]}m_k=m$ and each $m_k$ is nonnegative. \nThen,  sDFO \\eqref{robust_max_b} is equivalent to solving\n\\begin{align*}\nv^*=\\min_{\\bm x\\in\\X,\\beta,\\bm\\gamma\\geq \\bm 0} \\left\\{\\beta\\colon \\sum_{k\\in[K]} \\gamma_{k}= \\theta,\\beta \\geq  {\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x)+b_k(\\bm x)-\\gamma_{k}C_k,\\forall k\\in[K]  \\right\\}.\n\\end{align*}\n\\end{itemize}\n\\end{restatable}","sha256_utf8":"83dc7e42a5ff163788a3a8689f2bb1f389030fb16db3c5dd5a37fb82ca4bfbba"}]},{"id":"S009","semantic_type":"proof","kind":"proof","title":"三类可解情形的完整证明","source_units":[{"start":59650,"end":68513,"start_line":840,"end_line":959,"raw_tex":"\\begin{proof}\nWe split the proof into three parts accordingly.\n\\begin{itemize}\n\\item[(i)] When $\\bm a_k(\\bm x)=\\bar{\\bm a}_k$ is constant for all $k\\in[K]$, sDFO \\eqref{robust_max_b} can be written as\n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top \\bm C_k +b_k(\\bm x) \\right] -  \\theta \\left\\|\\sum_{k\\in[K]} \\lambda_k \\bm C_k \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\},\n% \\end{align*}\n% or\n\\begin{align*}\nv^*=\\min_{\\bm x\\in\\X} \\min_{\\rxi} \\max_{k\\in[K]}  \\left\\{\\rxi^\\top \\bar{\\bm a}_k+b_k(\\bm x)\\colon \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\right\\}.\n\\end{align*}\nIntroducing an auxiliary variable $\\eta$ to linearize the inner maximum, we arrive at Part (i).\n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X,\\rxi,\\eta}  \\left\\{\\eta\\colon\\eta\\geq \\rxi^\\top \\bar{\\bm a}_k+b_k(\\bm x), \\forall k\\in[K], \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\right\\},\n% \\end{align*}\n% which can be a tractable convex optimization problem.\n% Since the decision $\\bm x$ does not influence the objective value, we can drop the first minimum and the problem can be written as\n% \\begin{align*}\n%  v^*= \\min_{\\rxi} \\max_{k\\in[K]}  \\left\\{\\rxi^\\top \\bm C_k+b_k\\colon \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\right\\}.\n% \\end{align*}\n% Letting $\\hat\\rxi=\\rxi-\\rxi^0$, then we have\n% \\begin{align*}\n%  v^*=\\min_{\\hat\\rxi} \\max_{k\\in[K]}  \\left\\{\\hat\\rxi^\\top \\bm C_k + {\\rxi^0}^\\top \\bm C_k  +b_k\\colon \\|\\hat\\rxi\\|_p\\leq \\theta\\right\\},\n% \\end{align*}\n% and the optimal objective value can found by solving the optimization problem on each piece and select the largest one, i.e., $v^* = \\max_{k\\in[K]}v_k^*$, where\n% \\begin{align*}\n%   v_k^*= \\min_{\\rxi} \\left\\{\\rxi^\\top \\bm C_k+b_k\\colon \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\right\\}.\n% \\end{align*}\n\n\n% and\n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq \\bm 0}  \\min_{\\hat\\rxi}  \\left\\{\\sum_{k\\in[K]}\\lambda_k\\left[{\\rxi^0}^\\top \\bm C_k +\\hat\\rxi^\\top \\bm C_k +b_k(\\bm x)\\right]\\colon \\|\\hat\\rxi\\|_p\\leq \\theta,\\sum_{k\\in[K]}\\lambda_k=1\\right\\},\n% \\end{align*}\n% which can be simplified as\n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top \\bm C_k +b_k(\\bm x) \\right] -  \\theta\\left\\| \\sum_{k\\in[K]} \\lambda_k \\bm C_k \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}.\n% \\end{align*}\n% If $\\bm C_k=\\bm C_1$ for all $k\\in[K]$, then \n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{  \\sum_{k\\in[K]} \\lambda_k  b_k(\\bm x) + {\\rxi^0}^\\top \\bm C_1 -  \\theta\\left\\| \\bm C_1 \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\},\n% \\end{align*}\n% and the problem is equivalent to \n% % \\begin{align*}\n% %   \\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{  \\sum_{k\\in[K]} \\lambda_k  b_k(\\bm x) \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\},\n% % \\end{align*}\n% % and \n% \\begin{align*}\n%   \\min_{\\bm x\\in\\X,\\eta}\\left\\{  \\eta\\colon \\eta\\geq b_k(\\bm x)-{\\rxi^0}^\\top \\bm C_1 -  \\theta\\left\\| \\bm C_1 \\right\\|_{p^*}   \\right\\}.\n% \\end{align*}\n\n% Otherwise, let $\\rxi^0=\\bm 0 $ and $b_k(\\bm x) =0$, then the problem reduces to \n% \\begin{align}\n%  \\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{ -  \\theta\\left\\| \\sum_{k\\in[K]} \\lambda_k \\bm C_k \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}\n% \\end{align}\n% and\n% \\begin{align*}\n%  \\min_{\\bm\\lambda\\geq  \\bm 0}\\left\\{ \\left\\| \\sum_{k\\in[K]} \\lambda_k \\bm C_k \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}\n% \\end{align*}\n% According to theorem 1 \\cite{ge2011note}, this problem can be NP-hard with general norm. \n\n% and\n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top C_k +b_k(\\bm x) -  \\theta C_k   \\right] \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\},\n% \\end{align*}\n% Taking the dual and  using strong duality from linear programming, we have \n% \\begin{align*}\n%   \\min_{\\bm x\\in\\X,\\eta}\\left\\{\\eta\\colon \\eta\\geq \\left[{\\rxi^0}^\\top \\bm a_k(\\bm x) +b_k(\\bm x) \\right] -  \\theta C,\\forall k\\in[K]  \\right\\},  \n% \\end{align*}\n% which can be tractable.\n\\item[(ii)] When $\\bm a_k(\\bm x)=\\bm a_1(\\bm x)$  for each $k\\in[K]$, we rewrite sDFO  \\eqref{robust_max_eq} as\n\\begin{align*}\nv^*=\\min_{\\bm x\\in\\X} \\max_{\\bm\\lambda\\geq  \\bm 0} \\left\\{\\sum_{k\\in[K]} \\lambda_k b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x) - \\theta \\| \\bm a_1(\\bm x) \\|_{p^*}\\colon  \\sum_{k\\in[K]} \\lambda_k =1  \\right\\}.\n\\end{align*}\nTaking the dual of the inner maximization problem and using strong duality from linear programming, we have \n\\begin{align*}\nv^*=\\min_{\\bm x\\in\\X,\\eta}  \\left\\{\\eta\\colon \\eta\\geq  b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x)- \\theta \\| \\bm a_1(\\bm x) \\|_{p^*}, \\forall k\\in[K] \\right\\}.\n%\\label{robust_max_micp_special_tractable_1}\n\\end{align*}\nWhen $p=1$, i.e., when the dual norm is $L_\\infty$, \nthen $\\theta \\|\\bm a_1(\\bm x) \\|_\\infty =  \\theta \\max_{i\\in [m]}\\max\\{ a_{1i}(\\bm x),-a_{1i}(\\bm x)\\}$. Therefore, sDFO \\eqref{robust_max_eq} is equivalent to solving $2m$ tractable problems, and selecting the best one with the lowest optimal value, i.e., $v^*=\\min_{i\\in[m],\\ell\\in [2]}v_{i\\ell}^*$, where for each $i\\in[m]$, we have\n\\begin{align*}\n& v_{i1}^*=\\min_{\\bm x\\in\\X,\\eta}  \\left\\{\\eta\\colon \\eta\\geq  b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x)- \\theta a_{1i}(\\bm x),\\forall k\\in[K] \\right\\},\\\\\n& v_{i2}^*=\\min_{\\bm x\\in\\X,\\eta}  \\left\\{\\eta\\colon \\eta\\geq  b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x) + \\theta a_{1i}(\\bm x),\\forall k\\in[K] \\right\\}.\n\\end{align*}\n% which can be derived from \\eqref{robust_min_micp} by letting $k=K=1$. \n% Therefore, according to the results in \\Cref{prop_robust_min_tractable}, when $p=1$, DFO \\eqref{robust_max_micp_special_case_1} can be tractable.\n\\item[(iii)] Since $\\rxi^i$ and $\\rxi^j$ do not overlap for each $i\\neq j$ and $\\bar{\\bm a}_{j}(\\bm x) $ and $\\bar{\\bm a}_{j}(\\bm x) $ do not overlap for each $i\\neq j$ as well, when $p=1$ and $\\|\\bar{\\bm a}_k(\\bm x) \\|_{\\infty}=C_k$ is constant for each $k\\in[K]$, the dual norm term in \\eqref{robust_max_eq} can be simplified as\n\\begin{align*}\n\\left\\|\\sum_{k\\in[K]} \\lambda_k \\bm a_k(\\bm x) \\right\\|_{\\infty} = \\max_{k\\in[K]} \\lambda_k\\left\\|\\bar{\\bm a}_k(\\bm x) \\right\\|_{\\infty}=\\max_{k\\in[K]} \\lambda_k C_k.\n\\end{align*}\nThen, sDFO \\eqref{robust_max_eq} can be written as\n\\begin{align*}\n v^*=  \\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x) \\right] + \\sum_{k\\in[K]} \\lambda_k  b_k(\\bm x) -  \\theta \\max_{k\\in[K]} \\lambda_kC_k \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}.\n\\end{align*}\n% which is\n% \\begin{align*}\n% \\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x) \\right] + \\sum_{k\\in[K]} \\lambda_k  b_k(\\bm x) -  \\theta \\max_{k\\in[K]}\\max_{i\\in[n]} \\lambda_k|\\bar{a}_{ki}(\\bm x)| \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}.\n% \\end{align*}\nIntroducing one variable $\\eta$ to linearize the term $\\max_{k\\in[K]} \\lambda_kC_k$, then we have\n\\begin{align*}\n\\max_{\\bm\\lambda\\geq  \\bm 0,\\eta}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x) \\right] + \\sum_{k\\in[K]} \\lambda_k  b_k(\\bm x) -  \\theta \\eta  \\colon  \\sum_{k\\in[K]} \\lambda_k =1,   \\lambda_kC_k-\\eta\\leq 0, \\forall k\\in[K] \\right\\}. \n\\end{align*}\nTaking the dual of the inner maximization problem with dual variables $\\beta,\\bm\\gamma$ and using strong duality from linear programming, we have \n\\begin{align*}\n\\min_{\\beta,\\bm\\gamma\\geq \\bm 0} \\left\\{\\beta\\colon \\sum_{k\\in[K]} \\gamma_{k}= \\theta,\\beta \\geq  {\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x)+b_k(\\bm x)-\\gamma_{k}C_k,\\forall k\\in[K]  \\right\\}.\n\\end{align*}\nThis completes the proof.\\QEDA\n% \\begin{align*}\n%  \\min_{\\beta,\\bm\\gamma\\geq \\bm 0}\\quad & \\beta,\\\\\n%  \\textup{s.t.} \\quad & \\sum_{k\\in[K]} \\gamma_{k}\\leq \\theta,\\\\\n%  & \\gamma_{k}\\left\\|\\bar{\\bm a}_k(\\bm x) \\right\\|_{\\infty}+\\beta \\geq  {\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x)+b_k(\\bm x),\\forall k\\in[K].\n% \\end{align*}\n% Taking the dual of the inner maximization problem and using strong duality from linear programming, DFO \\eqref{robust_max_eq} is equivalent to solving $2nK$ tractable programs and selecting the best one with the lowest optimal value, i.e., $v^*=\\min_{k\\in [K],i\\in[n],\\ell\\in [2]}v_{ik\\ell}^*$, where for each $k\\in [K]$ and $i\\in[n]$, we have\n%   \\begin{align*}\n% & v_{ik1}^*=\\min_{\\bm x\\in\\X,\\eta}\\left\\{\\eta\\colon \\eta\\geq{\\rxi^0}^\\top \\bar{\\bm a}_{k}(\\bm x) + b_k(\\bm x) - \\theta \\bar{a}_{ki}(\\bm x), \\eta\\geq{\\rxi^0}^\\top \\bar{\\bm a}_{j}(\\bm x) + b_j(\\bm x), \\forall j\\in[K], j\\neq k\\right\\},\\\\\n% & v_{ik2}^*=\\min_{\\bm x\\in\\X,\\eta }\\left\\{\\eta\\colon \\eta\\geq{\\rxi^0}^\\top \\bar{\\bm a}_{k}(\\bm x) + b_k(\\bm x) + \\theta \\bar{a}_{ki}(\\bm x), \\eta\\geq{\\rxi^0}^\\top \\bar{\\bm a}_{j}(\\bm x) + b_j(\\bm x), \\forall j\\in[K], j\\neq k\\right\\}.\n%   \\end{align*}\n\\end{itemize}\n\\end{proof}","sha256_utf8":"7bad4d24ec64d80dd6e6a9c2d641b72f4bee40e9a5ed1d548d44ec9ac9ae7153"}]},{"id":"S010","semantic_type":"lemma","kind":"lemma","title":"单纯形对偶范数重构","source_units":[{"start":54973,"end":55403,"start_line":780,"end_line":785,"raw_tex":"\\begin{restatable}{lemma}{robustmaxrefor}\\label{robust_max_refor} \nsDFO \\eqref{robust_max_b} is equivalent to \n\\begin{align}\nv^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top \\bm a_k(\\bm x) +b_k(\\bm x) \\right] -  \\theta \\left\\|\\sum_{k\\in[K]} \\lambda_k \\bm a_k(\\bm x) \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}. \\label{robust_max_eq}\n\\end{align}\n\\end{restatable}","sha256_utf8":"15c2bce0c95f1e70aa630b4be026a5ace5807e8e37ed3b14b21ae80908248c7e"}]},{"id":"S011","semantic_type":"proof","kind":"proof","title":"极小极大重构的完整证明","source_units":[{"start":55487,"end":57149,"start_line":790,"end_line":813,"raw_tex":"\\begin{proof}\nLet us first consider the inner minimax of sDFO \\eqref{robust_max_b} as\n\\begin{align*}\n\\min_{\\rxi} \\left\\{\\max_{k\\in[K]}  \\rxi^\\top \\bm a_k(\\bm x)+b_k(\\bm x)\\colon \\|\\bm \\xi-\\bm \\xi^0\\|_p\\leq \\theta\\right\\}.\n\\end{align*}\nIntroducing auxiliary nonnegative variables $\\bm\\lambda$, the inner minimax of sDFO \\eqref{robust_max_b} is equivalent to\n\\begin{align*}\n\\min_{\\rxi}\\left\\{\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi}^\\top \\bm a_k(\\bm x)+b_k(\\bm x)\\right]\\colon  \\sum_{k\\in[K]} \\lambda_k =1\\right\\}: \\|\\bm \\xi-\\bm \\xi^0\\|_p\\leq \\theta\\right\\}.\n\\end{align*} \nAccording to Sion's minimax theorem \\cite{sion1958general}, we can interchange the maximum operator with the minimum one as\n\\begin{align*}\n\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\min_{\\rxi}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi}^\\top \\bm a_k(\\bm x)+b_k(\\bm x)\\right]\\colon \\|\\bm \\xi-\\bm \\xi^0\\|_p\\leq \\theta  \\right\\}:\\sum_{k\\in[K]} \\lambda_k =1\\right\\}.\n\\end{align*}\nInvoking the definition of dual norm, we have \n\\begin{align*}\n\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top \\bm a_k(\\bm x) +b_k(\\bm x) \\right] -  \\theta \\left\\|\\sum_{k\\in[K]} \\lambda_k \\bm a_k(\\bm x) \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}.\n\\end{align*}\n% and DFO \\eqref{robust_max_b} can be written as\n% \\begin{align*}\n% v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top \\bm a_k(\\bm x) +b_k(\\bm x) \\right] -  \\theta \\left\\|\\sum_{k\\in[K]} \\lambda_k \\bm a_k(\\bm x) \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}.\n% \\end{align*}\nThis completes the proof.\n\\QEDA\n\\end{proof}","sha256_utf8":"8a530e858f1dd553a6792e77fb973b575a93ae6edec29fc46e2fd1803b0f792e"}]},{"id":"S012","semantic_type":"proof_step","kind":"proof_step","title":"提取内层极小极大问题","source_units":[{"start":55487,"end":55722,"start_line":790,"end_line":794,"raw_tex":"\\begin{proof}\nLet us first consider the inner minimax of sDFO \\eqref{robust_max_b} as\n\\begin{align*}\n\\min_{\\rxi} \\left\\{\\max_{k\\in[K]}  \\rxi^\\top \\bm a_k(\\bm x)+b_k(\\bm x)\\colon \\|\\bm \\xi-\\bm \\xi^0\\|_p\\leq \\theta\\right\\}.\n\\end{align*}\n","sha256_utf8":"33dee44892045d20e49a348e5271b75cb449beb606b8bd4527600cdf4535a020"}]},{"id":"S013","semantic_type":"proof_step","kind":"proof_step","title":"以单纯形表示有限最大值","source_units":[{"start":55722,"end":56090,"start_line":795,"end_line":798,"raw_tex":"Introducing auxiliary nonnegative variables $\\bm\\lambda$, the inner minimax of sDFO \\eqref{robust_max_b} is equivalent to\n\\begin{align*}\n\\min_{\\rxi}\\left\\{\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi}^\\top \\bm a_k(\\bm x)+b_k(\\bm x)\\right]\\colon  \\sum_{k\\in[K]} \\lambda_k =1\\right\\}: \\|\\bm \\xi-\\bm \\xi^0\\|_p\\leq \\theta\\right\\}.\n\\end{align*} \n","sha256_utf8":"8dc973603ef6a0b003cc45d2e8b5de85eb069f2e4f9425534225f3d4cf525f48"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"交换极大与极小算子","source_units":[{"start":56090,"end":56459,"start_line":799,"end_line":802,"raw_tex":"According to Sion's minimax theorem \\cite{sion1958general}, we can interchange the maximum operator with the minimum one as\n\\begin{align*}\n\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\min_{\\rxi}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi}^\\top \\bm a_k(\\bm x)+b_k(\\bm x)\\right]\\colon \\|\\bm \\xi-\\bm \\xi^0\\|_p\\leq \\theta  \\right\\}:\\sum_{k\\in[K]} \\lambda_k =1\\right\\}.\n\\end{align*}\n","sha256_utf8":"1c6531064e3d168d742b8a31bf19945c6411db418edc558260a675b7c667449f"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"用对偶范数消去不确定量","source_units":[{"start":56459,"end":57149,"start_line":803,"end_line":813,"raw_tex":"Invoking the definition of dual norm, we have \n\\begin{align*}\n\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top \\bm a_k(\\bm x) +b_k(\\bm x) \\right] -  \\theta \\left\\|\\sum_{k\\in[K]} \\lambda_k \\bm a_k(\\bm x) \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}.\n\\end{align*}\n% and DFO \\eqref{robust_max_b} can be written as\n% \\begin{align*}\n% v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top \\bm a_k(\\bm x) +b_k(\\bm x) \\right] -  \\theta \\left\\|\\sum_{k\\in[K]} \\lambda_k \\bm a_k(\\bm x) \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}.\n% \\end{align*}\nThis completes the proof.\n\\QEDA\n\\end{proof}","sha256_utf8":"2ea2c42ba452513f96a510874bf4461da69ab64b9229c66c6813b5c413e89eb6"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"常系数情形的上图重构","source_units":[{"start":59650,"end":63732,"start_line":840,"end_line":909,"raw_tex":"\\begin{proof}\nWe split the proof into three parts accordingly.\n\\begin{itemize}\n\\item[(i)] When $\\bm a_k(\\bm x)=\\bar{\\bm a}_k$ is constant for all $k\\in[K]$, sDFO \\eqref{robust_max_b} can be written as\n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top \\bm C_k +b_k(\\bm x) \\right] -  \\theta \\left\\|\\sum_{k\\in[K]} \\lambda_k \\bm C_k \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\},\n% \\end{align*}\n% or\n\\begin{align*}\nv^*=\\min_{\\bm x\\in\\X} \\min_{\\rxi} \\max_{k\\in[K]}  \\left\\{\\rxi^\\top \\bar{\\bm a}_k+b_k(\\bm x)\\colon \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\right\\}.\n\\end{align*}\nIntroducing an auxiliary variable $\\eta$ to linearize the inner maximum, we arrive at Part (i).\n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X,\\rxi,\\eta}  \\left\\{\\eta\\colon\\eta\\geq \\rxi^\\top \\bar{\\bm a}_k+b_k(\\bm x), \\forall k\\in[K], \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\right\\},\n% \\end{align*}\n% which can be a tractable convex optimization problem.\n% Since the decision $\\bm x$ does not influence the objective value, we can drop the first minimum and the problem can be written as\n% \\begin{align*}\n%  v^*= \\min_{\\rxi} \\max_{k\\in[K]}  \\left\\{\\rxi^\\top \\bm C_k+b_k\\colon \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\right\\}.\n% \\end{align*}\n% Letting $\\hat\\rxi=\\rxi-\\rxi^0$, then we have\n% \\begin{align*}\n%  v^*=\\min_{\\hat\\rxi} \\max_{k\\in[K]}  \\left\\{\\hat\\rxi^\\top \\bm C_k + {\\rxi^0}^\\top \\bm C_k  +b_k\\colon \\|\\hat\\rxi\\|_p\\leq \\theta\\right\\},\n% \\end{align*}\n% and the optimal objective value can found by solving the optimization problem on each piece and select the largest one, i.e., $v^* = \\max_{k\\in[K]}v_k^*$, where\n% \\begin{align*}\n%   v_k^*= \\min_{\\rxi} \\left\\{\\rxi^\\top \\bm C_k+b_k\\colon \\|\\rxi-\\rxi^0\\|_p\\leq \\theta\\right\\}.\n% \\end{align*}\n\n\n% and\n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq \\bm 0}  \\min_{\\hat\\rxi}  \\left\\{\\sum_{k\\in[K]}\\lambda_k\\left[{\\rxi^0}^\\top \\bm C_k +\\hat\\rxi^\\top \\bm C_k +b_k(\\bm x)\\right]\\colon \\|\\hat\\rxi\\|_p\\leq \\theta,\\sum_{k\\in[K]}\\lambda_k=1\\right\\},\n% \\end{align*}\n% which can be simplified as\n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top \\bm C_k +b_k(\\bm x) \\right] -  \\theta\\left\\| \\sum_{k\\in[K]} \\lambda_k \\bm C_k \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}.\n% \\end{align*}\n% If $\\bm C_k=\\bm C_1$ for all $k\\in[K]$, then \n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{  \\sum_{k\\in[K]} \\lambda_k  b_k(\\bm x) + {\\rxi^0}^\\top \\bm C_1 -  \\theta\\left\\| \\bm C_1 \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\},\n% \\end{align*}\n% and the problem is equivalent to \n% % \\begin{align*}\n% %   \\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{  \\sum_{k\\in[K]} \\lambda_k  b_k(\\bm x) \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\},\n% % \\end{align*}\n% % and \n% \\begin{align*}\n%   \\min_{\\bm x\\in\\X,\\eta}\\left\\{  \\eta\\colon \\eta\\geq b_k(\\bm x)-{\\rxi^0}^\\top \\bm C_1 -  \\theta\\left\\| \\bm C_1 \\right\\|_{p^*}   \\right\\}.\n% \\end{align*}\n\n% Otherwise, let $\\rxi^0=\\bm 0 $ and $b_k(\\bm x) =0$, then the problem reduces to \n% \\begin{align}\n%  \\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{ -  \\theta\\left\\| \\sum_{k\\in[K]} \\lambda_k \\bm C_k \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}\n% \\end{align}\n% and\n% \\begin{align*}\n%  \\min_{\\bm\\lambda\\geq  \\bm 0}\\left\\{ \\left\\| \\sum_{k\\in[K]} \\lambda_k \\bm C_k \\right\\|_{p^*}  \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}\n% \\end{align*}\n% According to theorem 1 \\cite{ge2011note}, this problem can be NP-hard with general norm. \n\n% and\n% \\begin{align*}\n%  v^*=\\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi^0}^\\top C_k +b_k(\\bm x) -  \\theta C_k   \\right] \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\},\n% \\end{align*}\n% Taking the dual and  using strong duality from linear programming, we have \n% \\begin{align*}\n%   \\min_{\\bm x\\in\\X,\\eta}\\left\\{\\eta\\colon \\eta\\geq \\left[{\\rxi^0}^\\top \\bm a_k(\\bm x) +b_k(\\bm x) \\right] -  \\theta C,\\forall k\\in[K]  \\right\\},  \n% \\end{align*}\n% which can be tractable.\n","sha256_utf8":"13eee25bd5301235b1f19b23df7134379892190be2afdfe39a9c0cf1e144cec2"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"公共系数情形的线性对偶","source_units":[{"start":63732,"end":64419,"start_line":910,"end_line":918,"raw_tex":"\\item[(ii)] When $\\bm a_k(\\bm x)=\\bm a_1(\\bm x)$  for each $k\\in[K]$, we rewrite sDFO  \\eqref{robust_max_eq} as\n\\begin{align*}\nv^*=\\min_{\\bm x\\in\\X} \\max_{\\bm\\lambda\\geq  \\bm 0} \\left\\{\\sum_{k\\in[K]} \\lambda_k b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x) - \\theta \\| \\bm a_1(\\bm x) \\|_{p^*}\\colon  \\sum_{k\\in[K]} \\lambda_k =1  \\right\\}.\n\\end{align*}\nTaking the dual of the inner maximization problem and using strong duality from linear programming, we have \n\\begin{align*}\nv^*=\\min_{\\bm x\\in\\X,\\eta}  \\left\\{\\eta\\colon \\eta\\geq  b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x)- \\theta \\| \\bm a_1(\\bm x) \\|_{p^*}, \\forall k\\in[K] \\right\\}.\n%\\label{robust_max_micp_special_tractable_1}\n\\end{align*}\n","sha256_utf8":"6dc51c3eead39f75cac15b7ad89e3757aaed26e7ef5f13225f3ad7b0bee251b0"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"枚举无穷范数的符号分支","source_units":[{"start":64419,"end":65364,"start_line":919,"end_line":926,"raw_tex":"When $p=1$, i.e., when the dual norm is $L_\\infty$, \nthen $\\theta \\|\\bm a_1(\\bm x) \\|_\\infty =  \\theta \\max_{i\\in [m]}\\max\\{ a_{1i}(\\bm x),-a_{1i}(\\bm x)\\}$. Therefore, sDFO \\eqref{robust_max_eq} is equivalent to solving $2m$ tractable problems, and selecting the best one with the lowest optimal value, i.e., $v^*=\\min_{i\\in[m],\\ell\\in [2]}v_{i\\ell}^*$, where for each $i\\in[m]$, we have\n\\begin{align*}\n& v_{i1}^*=\\min_{\\bm x\\in\\X,\\eta}  \\left\\{\\eta\\colon \\eta\\geq  b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x)- \\theta a_{1i}(\\bm x),\\forall k\\in[K] \\right\\},\\\\\n& v_{i2}^*=\\min_{\\bm x\\in\\X,\\eta}  \\left\\{\\eta\\colon \\eta\\geq  b_k(\\bm x)+ {\\rxi^0}^\\top \\bm a_1(\\bm x) + \\theta a_{1i}(\\bm x),\\forall k\\in[K] \\right\\}.\n\\end{align*}\n% which can be derived from \\eqref{robust_min_micp} by letting $k=K=1$. \n% Therefore, according to the results in \\Cref{prop_robust_min_tractable}, when $p=1$, DFO \\eqref{robust_max_micp_special_case_1} can be tractable.\n","sha256_utf8":"4c05e971242d75c9f708ec20da8c8d6879917bebb29d95bd0a42cdbc2ccdf68e"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"分离系数的范数简化","source_units":[{"start":65364,"end":66559,"start_line":927,"end_line":938,"raw_tex":"\\item[(iii)] Since $\\rxi^i$ and $\\rxi^j$ do not overlap for each $i\\neq j$ and $\\bar{\\bm a}_{j}(\\bm x) $ and $\\bar{\\bm a}_{j}(\\bm x) $ do not overlap for each $i\\neq j$ as well, when $p=1$ and $\\|\\bar{\\bm a}_k(\\bm x) \\|_{\\infty}=C_k$ is constant for each $k\\in[K]$, the dual norm term in \\eqref{robust_max_eq} can be simplified as\n\\begin{align*}\n\\left\\|\\sum_{k\\in[K]} \\lambda_k \\bm a_k(\\bm x) \\right\\|_{\\infty} = \\max_{k\\in[K]} \\lambda_k\\left\\|\\bar{\\bm a}_k(\\bm x) \\right\\|_{\\infty}=\\max_{k\\in[K]} \\lambda_k C_k.\n\\end{align*}\nThen, sDFO \\eqref{robust_max_eq} can be written as\n\\begin{align*}\n v^*=  \\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x) \\right] + \\sum_{k\\in[K]} \\lambda_k  b_k(\\bm x) -  \\theta \\max_{k\\in[K]} \\lambda_kC_k \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}.\n\\end{align*}\n% which is\n% \\begin{align*}\n% \\min_{\\bm x\\in\\X}\\max_{\\bm\\lambda\\geq  \\bm 0}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x) \\right] + \\sum_{k\\in[K]} \\lambda_k  b_k(\\bm x) -  \\theta \\max_{k\\in[K]}\\max_{i\\in[n]} \\lambda_k|\\bar{a}_{ki}(\\bm x)| \\colon  \\sum_{k\\in[K]} \\lambda_k =1 \\right\\}.\n% \\end{align*}\n","sha256_utf8":"b8c05f155bef9f15bf72e99edeed87998c7566afd83bccc1ccf5285f2ea4b580"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"线性化加权常数最大值","source_units":[{"start":66559,"end":66949,"start_line":939,"end_line":942,"raw_tex":"Introducing one variable $\\eta$ to linearize the term $\\max_{k\\in[K]} \\lambda_kC_k$, then we have\n\\begin{align*}\n\\max_{\\bm\\lambda\\geq  \\bm 0,\\eta}\\left\\{\\sum_{k\\in[K]} \\lambda_k \\left[{\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x) \\right] + \\sum_{k\\in[K]} \\lambda_k  b_k(\\bm x) -  \\theta \\eta  \\colon  \\sum_{k\\in[K]} \\lambda_k =1,   \\lambda_kC_k-\\eta\\leq 0, \\forall k\\in[K] \\right\\}. \n\\end{align*}\n","sha256_utf8":"74fee3aecf676aa7af1ad07b1bbcff966503989b044c5b4c39310f4286f5c95c"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"对偶得到第三类凸重构","source_units":[{"start":66949,"end":68513,"start_line":943,"end_line":959,"raw_tex":"Taking the dual of the inner maximization problem with dual variables $\\beta,\\bm\\gamma$ and using strong duality from linear programming, we have \n\\begin{align*}\n\\min_{\\beta,\\bm\\gamma\\geq \\bm 0} \\left\\{\\beta\\colon \\sum_{k\\in[K]} \\gamma_{k}= \\theta,\\beta \\geq  {\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x)+b_k(\\bm x)-\\gamma_{k}C_k,\\forall k\\in[K]  \\right\\}.\n\\end{align*}\nThis completes the proof.\\QEDA\n% \\begin{align*}\n%  \\min_{\\beta,\\bm\\gamma\\geq \\bm 0}\\quad & \\beta,\\\\\n%  \\textup{s.t.} \\quad & \\sum_{k\\in[K]} \\gamma_{k}\\leq \\theta,\\\\\n%  & \\gamma_{k}\\left\\|\\bar{\\bm a}_k(\\bm x) \\right\\|_{\\infty}+\\beta \\geq  {\\rxi_k^0}^\\top \\bar{\\bm a}_{k}(\\bm x)+b_k(\\bm x),\\forall k\\in[K].\n% \\end{align*}\n% Taking the dual of the inner maximization problem and using strong duality from linear programming, DFO \\eqref{robust_max_eq} is equivalent to solving $2nK$ tractable programs and selecting the best one with the lowest optimal value, i.e., $v^*=\\min_{k\\in [K],i\\in[n],\\ell\\in [2]}v_{ik\\ell}^*$, where for each $k\\in [K]$ and $i\\in[n]$, we have\n%   \\begin{align*}\n% & v_{ik1}^*=\\min_{\\bm x\\in\\X,\\eta}\\left\\{\\eta\\colon \\eta\\geq{\\rxi^0}^\\top \\bar{\\bm a}_{k}(\\bm x) + b_k(\\bm x) - \\theta \\bar{a}_{ki}(\\bm x), \\eta\\geq{\\rxi^0}^\\top \\bar{\\bm a}_{j}(\\bm x) + b_j(\\bm x), \\forall j\\in[K], j\\neq k\\right\\},\\\\\n% & v_{ik2}^*=\\min_{\\bm x\\in\\X,\\eta }\\left\\{\\eta\\colon \\eta\\geq{\\rxi^0}^\\top \\bar{\\bm a}_{k}(\\bm x) + b_k(\\bm x) + \\theta \\bar{a}_{ki}(\\bm x), \\eta\\geq{\\rxi^0}^\\top \\bar{\\bm a}_{j}(\\bm x) + b_j(\\bm x), \\forall j\\in[K], j\\neq k\\right\\}.\n%   \\end{align*}\n\\end{itemize}\n\\end{proof}","sha256_utf8":"207963d906b1821ab74d2dfbc15c0704c9a02dbec1a0ac33629a4751fef3f25a"}]},{"id":"S022","semantic_type":"external_fact","kind":"external_fact","title":"极小极大交换定理","source_units":[],"declared_external":true,"declaration_text":["Sion's minimax theorem applies to the continuous bilinear objective on the compact convex norm ball and probability simplex, allowing interchange of its minimum and maximum."]},{"id":"S023","semantic_type":"external_fact","kind":"external_fact","title":"范数球上的线性最小值","source_units":[],"declared_external":true,"declaration_text":["For θ≥0, min{ξᵀc:‖ξ−ξ⁰‖p≤θ}=(ξ⁰)ᵀc−θ‖c‖p*, where p* is the dual exponent; in particular, the dual of L₁ is L∞."]},{"id":"S024","semantic_type":"external_fact","kind":"external_fact","title":"线性规划强对偶","source_units":[],"declared_external":true,"declaration_text":["A feasible linear program with a finite optimum has the same optimal value as its dual. This applies to the inner simplex maximizations and the auxiliary-variable linear maximization used in the author proof."]},{"id":"S025","semantic_type":"external_fact","kind":"external_fact","title":"标准凸规划的计算可解性","source_units":[],"declared_external":true,"declaration_text":["Under the representation and regularity conventions of theorem A.3.3 in ben2009robust, standard tractable convex programs admit polynomial-time approximation algorithms in the parameters specified by the paper's tractability definition."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S008","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S001","target_id":"S004","kind":"supports"},{"source_id":"S002","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S007","target_id":"S012","kind":"supports"},{"source_id":"S005","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S022","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S023","target_id":"S015","kind":"supports"},{"source_id":"S005","target_id":"S015","kind":"supports"},{"source_id":"S012","target_id":"S011","kind":"supports"},{"source_id":"S013","target_id":"S011","kind":"supports"},{"source_id":"S014","target_id":"S011","kind":"supports"},{"source_id":"S015","target_id":"S011","kind":"supports"},{"source_id":"S011","target_id":"S010","kind":"supports"},{"source_id":"S007","target_id":"S016","kind":"supports"},{"source_id":"S006","target_id":"S016","kind":"supports"},{"source_id":"S010","target_id":"S017","kind":"supports"},{"source_id":"S024","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S023","target_id":"S018","kind":"supports"},{"source_id":"S010","target_id":"S019","kind":"supports"},{"source_id":"S005","target_id":"S019","kind":"supports"},{"source_id":"S023","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S024","target_id":"S021","kind":"supports"},{"source_id":"S016","target_id":"S009","kind":"supports"},{"source_id":"S017","target_id":"S009","kind":"supports"},{"source_id":"S018","target_id":"S009","kind":"supports"},{"source_id":"S019","target_id":"S009","kind":"supports"},{"source_id":"S020","target_id":"S009","kind":"supports"},{"source_id":"S021","target_id":"S009","kind":"supports"},{"source_id":"S003","target_id":"S009","kind":"supports"},{"source_id":"S004","target_id":"S009","kind":"supports"},{"source_id":"S025","target_id":"S009","kind":"supports"},{"source_id":"S009","target_id":"S008","kind":"supports"}]}}
{"record_id":"2402.06033v3","paper":{"title":"An Inexact Halpern Iteration with Application to Distributionally Robust Optimization","authors":["Ling Liang","Zusen Xu","Kim-Chuan Toh","Jia-Jie Zhu"],"arxiv_id":"2402.06033","version":"v3","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-025-02744-y","abstract_url":"https://arxiv.org/abs/2402.06033v3","pdf_url":"https://arxiv.org/pdf/2402.06033v3"},"context":{"raw_tex":"\\begin{equation}\n    \\label{eq-inexact-halpern}\n    \\textrm{Find $\\tilde{z}^k$ such that } \\norm{G(z^k) - \\tilde{z}^k}\\leq \\gamma_k,\\; z^{k+1} = \\beta_k z^0 + (1-\\beta_k)z^k - \\eta_k \\tilde{z}^k,\n    \\tag{\\textrm{iHI}}\n\\end{equation}\n\nWe use $\\bb{X}$,  $\\bb{Y}$ and $\\bb{Z}$ to denote finite dimensional Euclidean spaces equipped with the standard inner product $\\inprod{\\cdot}{\\cdot}$ and the induced Euclidean norm $\\norm{\\cdot}$. Moreover, we use $\\|\\cdot\\|_\\infty$ to denote the infinity norm. Let $n$ be a positive integer, the standard basis for $\\bb{R}^n$ is denoted as $\\{e_1,\\dots, e_n\\}$ and the vector of all ones in $\\bb{R}^n$ is denoted as $\\mathds{1}_n$. We use $\\overline{\\bb{R}}$ to represent the extended reals $\\bb{R}\\cup\\{\\pm\\infty\\}$. Let $f:\\bb{X}\\to\\bb{R}$ be a differentiable function, its gradient at a point $x\\in \\bb{X}$ is denoted as $\\nabla f(x)$. If $\\bb{X} = \\bb{R}$, i.e., $f$ is a univariate function, then its derivative is denoted as $f'$. If $f:\\bb{X}\\times\\bb{Y}\\to\\bb{R}$ is a differentiable {function} of two vector variables, we use $\\nabla_xf(x,y)$ and $\\nabla_yf(x,y)$ to denote the partial derivatives with respect to $x\\in \\bb{X}$ and $y\\in \\bb{Y}$, respectively. In this way, the gradient of $f$ at the point $(x,y)$ is denoted as $\\nabla f(x,y) = [\\nabla_xf(x,y);\\nabla_yf(x,y)]$, where $[\\cdot;\\dots;\\cdot]$ is the column vector stacked from its components.\n\n\n\\textbf{Set-valued mappings.} Let $G:\\bb{X}\\rightrightarrows 2^\\bb{X}$ be a set-valued mapping, we use $\\dom{G}:=\\left\\{x\\in \\bb{X}:G(x)\\neq \\emptyset\\right\\}$ to denote its domain and use $\\mathrm{graph}(G):=\\left\\{(x,x')\\in \\bb{X}\\times \\bb{X}: x'\\in G(x)\\right\\}$ to denote its graph. Then the inverse of $G$ is denoted as $G^{-1}(x'):=\\left\\{x\\in\\bb{X}: x'\\in G(x)\\right\\}$. We say that $G$ is single-valued if $G(x)$ is a singleton for any $x\\in \\bb{X}$.\n\n\n\\textbf{Resolvent operators.} For a monotone operator $G:\\bb{X}\\rightrightarrows 2^\\bb{X}$, we denote its resolvent operator as $J_G:=(I+G)^{-1}$, where $I$ denotes the identity mapping. It is easy to verify that when $G$ is monotone, $J_G$ is single-valued with $\\dom{G} = \\bb{X}$. Note also that for a closed, proper and convex function $f:\\bb{X}\\to \\overline{\\bb{R}}$ and any positive scalar $\\alpha\\in \\bb{R}$, it holds that $J_{\\alpha \\partial f}(x) = \\prox{\\alpha f}{x}$. In particular, for any closed convex set $\\cal{C}\\subseteq \\bb{X}$ and any positive scalar $\\alpha\\in \\bb{R}$, it holds that $J_{\\alpha\\partial\\cal{I}_{\\cal{C}}}(\\cdot) = \\proj{\\cal{C}}{\\cdot}$, i.e., the projection operator onto $\\cal{C}$.\n\n\nLet $G:\\bb{X}\\rightrightarrows 2^\\bb{X}$ be a single-valued mapping, if there exists $L\\geq 0$ such that\n\\[\n    \\left\\lVert G(x_1) - G(x_2)\\right\\rVert \\leq L\\left\\lVert x_1 - x_2\\right\\rVert,\\quad \\forall x_1,x_2\\in \\dom{G},\n\\]\nthen $G$ is said to be $L$-Lipschitz continuous. \n\n\\textbf{Co-coerciveness.} Let $L > 0$. A set-valued mapping $G:\\bb{X}\\rightrightarrows 2^\\bb{X}$ is said to be $\\frac{1}{L}$-co-coercive if the following relation holds:\n\\[\n    \\left\\langle G(x_1) - G(x_2), x_1 - x_2\\right\\rangle \\geq \\frac{1}{L}\\left\\lVert G(x_1) - G(x_2)\\right\\rVert^2,\\quad \\forall x_1,x_2\\in \\dom{G}.\n\\]\n\n\nAs mentioned in the introduction, in the ideal case when $G(z)$ can be evaluated exactly for any $z\\in \\bb{Z}$, the exact Halpern iterative scheme \\eqref{eq-halpern} is guaranteed to possess an $O(k^{-1})$ rate. In this section, we illustrate that when the inexactness tolerance $\\gamma_k$ satisfies some mild conditions, the inexact version \\eqref{eq-inexact-halpern} shares a similar convergence rate. We assume for the rest of this paper that the problem \\eqref{eq-inclusion-G} admits at least one solution.\n\n\nIn many modern applications in operation research and machine learning, $G$ usually admits a finite-sum structure. To see this, let $E:\\bb{Z}\\to 2^\\bb{Z}$ and $F_i:\\bb{Z}\\to 2^\\bb{Z}$, for $i = 1,\\dots, N$, be maximal monotone operators. In addition, $F_i$ is assumed to be $\\frac{1}{L_0}$-co-coercive for any $i = 1,\\dots, N$. Then, many real-world applications can be reformulated as the following inclusion problem: \n\\begin{equation}\n\t0\\in E(z) + F(z):= E(z) + \\frac{1}{N}\\sum_{i = 1}^N F_i(z),\\quad z\\in \\bb{Z},\n\t\\label{eq-finitesum}\t\n\\end{equation}\n\nwhich is equivalent to the following new problem involving the resolvent operator of $E$:\n\\begin{equation}\n\t\\label{eq-finitesum-G}\n\t0\\in G(z):= \\frac{1}{\\alpha}\\left(z - J_{\\alpha E}\\left(z -  \\frac{\\alpha}{N}\\sum_{i = 1}^NF_i(z)\\right)\\right),\\quad z\\in \\bb{Z}.\n\\end{equation}\n\nHowever, when $N$ is large, which is typically the case in practical applications, evaluating the sum $ \\frac{1}{N}\\sum_{i = 1}^NF_i(z) $ can be costly. This motivates us to consider stochastic/randomized variants of the inexact Halpern fixed-point iterative scheme \\eqref{eq-inexact-halpern}. Specifically, given $z^0\\in \\bb{Z}$, it is natural to consider performing the following steps for $k\\geq 0$:\n\\begin{equation}\n\\label{eq-isHalpern}\n\\left\\{\n\t\\begin{aligned}\n\t\t&\\; \\textrm{Find a stochastic estimator $ \\widetilde{F}(z^k) $ satisfying $ \\bb{E}\\left[\\norm{F(z^k) - \\widetilde{F}(z^k)}^2\\right] \\leq \\frac{1}{2} \\sigma_k^2 $}, \\\\ \n\t\t&\\; \t\\textrm{Find $\\bar{z}^k$ such that } \\norm{J_{\\alpha E}\\left(z^k - \\alpha \\widetilde{F}(z^k) \\right) - \\bar{z}^k}\\leq \\frac{\\sqrt{2}}{2}\\alpha \\gamma_k,\\\\\n\t\t&\\; \\tilde{z}^k = \\frac{1}{\\alpha}(z^k - \\bar{z}^k), \\quad  z^{k+1} = \\beta_k z^0 + (1-\\beta_k)z^k - \\eta_k \\tilde{z}^k, \n\t\\end{aligned}\n\t\\right.\n    \\tag{\\textrm{isHI}}\n\\end{equation}\n\nwhere $ \\alpha\\in (0,4/L_0)$, $\\beta_k = 1/(k+2)$, $\\eta_k  = (1-\\beta_k)/L$ with $L:= 4/(\\alpha(4-\\alpha L_0))$, $\\gamma_k \\geq 0$ and $\\sigma_k\\geq 0$ are chosen appropriately. Here, we use $\\bb{E}$ to denote the expectation with respect to all the randomness at any iteration of \\eqref{eq-isHalpern}.\n","units":[{"start":10348,"end":10581,"start_line":118,"end_line":122,"raw_tex":"\\begin{equation}\n    \\label{eq-inexact-halpern}\n    \\textrm{Find $\\tilde{z}^k$ such that } \\norm{G(z^k) - \\tilde{z}^k}\\leq \\gamma_k,\\; z^{k+1} = \\beta_k z^0 + (1-\\beta_k)z^k - \\eta_k \\tilde{z}^k,\n    \\tag{\\textrm{iHI}}\n\\end{equation}","sha256_utf8":"b9f923c05f65f24ae4f24a56e4d5e24b56a965b000ae9c530c417a928123b6e6"},{"start":20370,"end":21539,"start_line":160,"end_line":160,"raw_tex":"We use $\\bb{X}$,  $\\bb{Y}$ and $\\bb{Z}$ to denote finite dimensional Euclidean spaces equipped with the standard inner product $\\inprod{\\cdot}{\\cdot}$ and the induced Euclidean norm $\\norm{\\cdot}$. Moreover, we use $\\|\\cdot\\|_\\infty$ to denote the infinity norm. Let $n$ be a positive integer, the standard basis for $\\bb{R}^n$ is denoted as $\\{e_1,\\dots, e_n\\}$ and the vector of all ones in $\\bb{R}^n$ is denoted as $\\mathds{1}_n$. We use $\\overline{\\bb{R}}$ to represent the extended reals $\\bb{R}\\cup\\{\\pm\\infty\\}$. Let $f:\\bb{X}\\to\\bb{R}$ be a differentiable function, its gradient at a point $x\\in \\bb{X}$ is denoted as $\\nabla f(x)$. If $\\bb{X} = \\bb{R}$, i.e., $f$ is a univariate function, then its derivative is denoted as $f'$. If $f:\\bb{X}\\times\\bb{Y}\\to\\bb{R}$ is a differentiable {function} of two vector variables, we use $\\nabla_xf(x,y)$ and $\\nabla_yf(x,y)$ to denote the partial derivatives with respect to $x\\in \\bb{X}$ and $y\\in \\bb{Y}$, respectively. In this way, the gradient of $f$ at the point $(x,y)$ is denoted as $\\nabla f(x,y) = [\\nabla_xf(x,y);\\nabla_yf(x,y)]$, where $[\\cdot;\\dots;\\cdot]$ is the column vector stacked from its components.\n","sha256_utf8":"9109e1a412dce0545df498e88c07cbe0f01b0937c3c5f91cfafa6b95ae140a10"},{"start":22757,"end":23217,"start_line":164,"end_line":164,"raw_tex":"\\textbf{Set-valued mappings.} Let $G:\\bb{X}\\rightrightarrows 2^\\bb{X}$ be a set-valued mapping, we use $\\dom{G}:=\\left\\{x\\in \\bb{X}:G(x)\\neq \\emptyset\\right\\}$ to denote its domain and use $\\mathrm{graph}(G):=\\left\\{(x,x')\\in \\bb{X}\\times \\bb{X}: x'\\in G(x)\\right\\}$ to denote its graph. Then the inverse of $G$ is denoted as $G^{-1}(x'):=\\left\\{x\\in\\bb{X}: x'\\in G(x)\\right\\}$. We say that $G$ is single-valued if $G(x)$ is a singleton for any $x\\in \\bb{X}$.\n","sha256_utf8":"087e408e9d02157ddaf09eff8eb697a2d08fe21a3646da3cfcb984cb97e60af8"},{"start":24036,"end":24755,"start_line":176,"end_line":176,"raw_tex":"\\textbf{Resolvent operators.} For a monotone operator $G:\\bb{X}\\rightrightarrows 2^\\bb{X}$, we denote its resolvent operator as $J_G:=(I+G)^{-1}$, where $I$ denotes the identity mapping. It is easy to verify that when $G$ is monotone, $J_G$ is single-valued with $\\dom{G} = \\bb{X}$. Note also that for a closed, proper and convex function $f:\\bb{X}\\to \\overline{\\bb{R}}$ and any positive scalar $\\alpha\\in \\bb{R}$, it holds that $J_{\\alpha \\partial f}(x) = \\prox{\\alpha f}{x}$. In particular, for any closed convex set $\\cal{C}\\subseteq \\bb{X}$ and any positive scalar $\\alpha\\in \\bb{R}$, it holds that $J_{\\alpha\\partial\\cal{I}_{\\cal{C}}}(\\cdot) = \\proj{\\cal{C}}{\\cdot}$, i.e., the projection operator onto $\\cal{C}$.\n","sha256_utf8":"facc499081512a8b72d71a172dce192c281e36a19a6a40676e3cf2157c756930"},{"start":25267,"end":25872,"start_line":186,"end_line":195,"raw_tex":"Let $G:\\bb{X}\\rightrightarrows 2^\\bb{X}$ be a single-valued mapping, if there exists $L\\geq 0$ such that\n\\[\n    \\left\\lVert G(x_1) - G(x_2)\\right\\rVert \\leq L\\left\\lVert x_1 - x_2\\right\\rVert,\\quad \\forall x_1,x_2\\in \\dom{G},\n\\]\nthen $G$ is said to be $L$-Lipschitz continuous. \n\n\\textbf{Co-coerciveness.} Let $L > 0$. A set-valued mapping $G:\\bb{X}\\rightrightarrows 2^\\bb{X}$ is said to be $\\frac{1}{L}$-co-coercive if the following relation holds:\n\\[\n    \\left\\langle G(x_1) - G(x_2), x_1 - x_2\\right\\rangle \\geq \\frac{1}{L}\\left\\lVert G(x_1) - G(x_2)\\right\\rVert^2,\\quad \\forall x_1,x_2\\in \\dom{G}.\n\\]\n","sha256_utf8":"29678bbfdbab9bacd18a1a1a4e6e57ed95a71f0d4e5fafd7e86e4884f0e7f632"},{"start":27626,"end":28137,"start_line":223,"end_line":223,"raw_tex":"As mentioned in the introduction, in the ideal case when $G(z)$ can be evaluated exactly for any $z\\in \\bb{Z}$, the exact Halpern iterative scheme \\eqref{eq-halpern} is guaranteed to possess an $O(k^{-1})$ rate. In this section, we illustrate that when the inexactness tolerance $\\gamma_k$ satisfies some mild conditions, the inexact version \\eqref{eq-inexact-halpern} shares a similar convergence rate. We assume for the rest of this paper that the problem \\eqref{eq-inclusion-G} admits at least one solution.\n","sha256_utf8":"aeb5043384b78d6b2dec86838ae50bb504246be37215bca9b1e53485cc50866d"},{"start":32187,"end":32740,"start_line":264,"end_line":268,"raw_tex":"In many modern applications in operation research and machine learning, $G$ usually admits a finite-sum structure. To see this, let $E:\\bb{Z}\\to 2^\\bb{Z}$ and $F_i:\\bb{Z}\\to 2^\\bb{Z}$, for $i = 1,\\dots, N$, be maximal monotone operators. In addition, $F_i$ is assumed to be $\\frac{1}{L_0}$-co-coercive for any $i = 1,\\dots, N$. Then, many real-world applications can be reformulated as the following inclusion problem: \n\\begin{equation}\n\t0\\in E(z) + F(z):= E(z) + \\frac{1}{N}\\sum_{i = 1}^N F_i(z),\\quad z\\in \\bb{Z},\n\t\\label{eq-finitesum}\t\n\\end{equation}","sha256_utf8":"fbbbcb0587c92f07d1fe5dfe188bb674762c34e0cda8fc18116f12078b68cb16"},{"start":32741,"end":33018,"start_line":269,"end_line":273,"raw_tex":"which is equivalent to the following new problem involving the resolvent operator of $E$:\n\\begin{equation}\n\t\\label{eq-finitesum-G}\n\t0\\in G(z):= \\frac{1}{\\alpha}\\left(z - J_{\\alpha E}\\left(z -  \\frac{\\alpha}{N}\\sum_{i = 1}^NF_i(z)\\right)\\right),\\quad z\\in \\bb{Z}.\n\\end{equation}","sha256_utf8":"ea406e46c147dcb0c6cd1db9ab94dc269194197d4e11a11eac3c9d16d6801b13"},{"start":33395,"end":34379,"start_line":277,"end_line":288,"raw_tex":"However, when $N$ is large, which is typically the case in practical applications, evaluating the sum $ \\frac{1}{N}\\sum_{i = 1}^NF_i(z) $ can be costly. This motivates us to consider stochastic/randomized variants of the inexact Halpern fixed-point iterative scheme \\eqref{eq-inexact-halpern}. Specifically, given $z^0\\in \\bb{Z}$, it is natural to consider performing the following steps for $k\\geq 0$:\n\\begin{equation}\n\\label{eq-isHalpern}\n\\left\\{\n\t\\begin{aligned}\n\t\t&\\; \\textrm{Find a stochastic estimator $ \\widetilde{F}(z^k) $ satisfying $ \\bb{E}\\left[\\norm{F(z^k) - \\widetilde{F}(z^k)}^2\\right] \\leq \\frac{1}{2} \\sigma_k^2 $}, \\\\ \n\t\t&\\; \t\\textrm{Find $\\bar{z}^k$ such that } \\norm{J_{\\alpha E}\\left(z^k - \\alpha \\widetilde{F}(z^k) \\right) - \\bar{z}^k}\\leq \\frac{\\sqrt{2}}{2}\\alpha \\gamma_k,\\\\\n\t\t&\\; \\tilde{z}^k = \\frac{1}{\\alpha}(z^k - \\bar{z}^k), \\quad  z^{k+1} = \\beta_k z^0 + (1-\\beta_k)z^k - \\eta_k \\tilde{z}^k, \n\t\\end{aligned}\n\t\\right.\n    \\tag{\\textrm{isHI}}\n\\end{equation}","sha256_utf8":"5a018b6edab8c1d656e28ffb9471910f33abf027e0fb0d614a27ed3cc6496830"},{"start":34380,"end":34684,"start_line":289,"end_line":289,"raw_tex":"where $ \\alpha\\in (0,4/L_0)$, $\\beta_k = 1/(k+2)$, $\\eta_k  = (1-\\beta_k)/L$ with $L:= 4/(\\alpha(4-\\alpha L_0))$, $\\gamma_k \\geq 0$ and $\\sigma_k\\geq 0$ are chosen appropriately. Here, we use $\\bb{E}$ to denote the expectation with respect to all the randomness at any iteration of \\eqref{eq-isHalpern}.\n","sha256_utf8":"740ed0c376ebebbcf6b2613fdb412803da41af07081784f56dd35ac5761addea"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\n\t\\label{thm-rate-ishi}\n\tSuppose that $G$ is $\\frac{1}{L}$-co-coercive for a constant $L>0$. Let $\\{z^k\\}$ be the sequence generated by \\eqref{eq-isHalpern} for solving \\eqref{eq-finitesum-G}. Then it holds that \n\t\\begin{align}\n\t\t\\bb{E}\\left[\\norm{G(z^k)}^2\\right] \\leq &\\;  \\frac{\\left(7L\\norm{z^0 - z^*} + 10\\sqrt{\\sum_{i = 0}^{k-1}(i+1)^2(\\sigma_i + \\gamma_i)^2}\\right)^2}{(k+1)(k+2)},\\quad k\\geq 0, \\label{eq-thm-rate-ishi-G}\t\\\\\n\t\t\\bb{E}\\left[\\norm{z^{k+1} - z^k}^2\\right] \\leq &\\; \\frac{8\\left(7L\\norm{z^0 - z^*} + 11\\sqrt{\\sum_{i = 0}^{k}(i+1)^2(\\sigma_i + \\gamma_i)^2}\\right)^2 }{L^2(k+1)(k+2)}, \\quad k\\geq 0,\\label{eq-thm-rate-ishi-Z}\n\t\\end{align}\n\twhere $z^*$ is a solution such that $G(z^*) = 0$.\n\\end{theorem}","units":[{"start":35353,"end":36089,"start_line":297,"end_line":305,"raw_tex":"\\begin{theorem}\n\t\\label{thm-rate-ishi}\n\tSuppose that $G$ is $\\frac{1}{L}$-co-coercive for a constant $L>0$. Let $\\{z^k\\}$ be the sequence generated by \\eqref{eq-isHalpern} for solving \\eqref{eq-finitesum-G}. Then it holds that \n\t\\begin{align}\n\t\t\\bb{E}\\left[\\norm{G(z^k)}^2\\right] \\leq &\\;  \\frac{\\left(7L\\norm{z^0 - z^*} + 10\\sqrt{\\sum_{i = 0}^{k-1}(i+1)^2(\\sigma_i + \\gamma_i)^2}\\right)^2}{(k+1)(k+2)},\\quad k\\geq 0, \\label{eq-thm-rate-ishi-G}\t\\\\\n\t\t\\bb{E}\\left[\\norm{z^{k+1} - z^k}^2\\right] \\leq &\\; \\frac{8\\left(7L\\norm{z^0 - z^*} + 11\\sqrt{\\sum_{i = 0}^{k}(i+1)^2(\\sigma_i + \\gamma_i)^2}\\right)^2 }{L^2(k+1)(k+2)}, \\quad k\\geq 0,\\label{eq-thm-rate-ishi-Z}\n\t\\end{align}\n\twhere $z^*$ is a solution such that $G(z^*) = 0$.\n\\end{theorem}","sha256_utf8":"14c8558d02e9ec0d9314f60af42f4fb60904c1b27c35606ff2d2efa7c3127b6a"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{theorem}\n\t\\label{thm-convergence-rate}\n\tSuppose that $G$ is $\\frac{1}{L}$-co-coercive for a constant $L>0$. Let $\\{z^k\\}$ be generated by \\eqref{eq-inexact-halpern} with $\\beta_k = 1/(k+2)$ and $\\eta_k = (1-\\beta_k)/L$, then it holds that\n\t\\begin{align}\n\t\t\\norm{G(z^k)}^2 \\leq &\\;  \\frac{\\left(7L\\norm{z^0 - z^*} + 10\\sqrt{\\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2}\\right)^2}{(k+1)(k+2)},\\quad k\\geq 0, \\label{thm-rate-1}\\\\\n\t\t\\norm{z^{k+1} - z^k}^2 \\leq &\\; \\frac{8\\left(7L\\norm{z^0 - z^*} + 11\\sqrt{\\sum_{i = 0}^{k}(i+1)^2\\gamma_i^2}\\right)^2 }{L^2(k+1)(k+2)}, \\quad k\\geq 0,\\label{thm-rate-2}\n\t\\end{align}\n\twhere $z^* \\in \\bb{Z}$ is any solution such that $G(z^*) = 0$.\n\\end{theorem}\n\n\\begin{lemma}\n    \\label{lemma-zdiff}\n    Let $\\{z^k\\}$ be generated by \\eqref{eq-inexact-halpern} with $\\eta_k = (1-\\beta_k)/L$, then the following expressions for $z^{k+1} - z^k$ for $k\\geq 0$ hold:\n    \\begin{enumerate}\n        \\item $z^{k+1} - z^k = \\beta_k(z^0 - z^k) - \\frac{1-\\beta_k}{L}\\tilde{z}^k$;\n        \\item $z^{k+1} - z^k = \\frac{\\beta_k}{1-\\beta_k}(z^0 - z^{k+1}) - \\frac{1}{L}\\tilde{z}^k$;\n        \\item $z^{k+1} - z^k = -\\frac{1-\\beta_k}{L}\\tilde{z}^k + \\sum_{i = 0}^{k-1}\\frac{\\beta_k}{L}\\left( \\prod_{j = i}^{k-1} (1-\\beta_j)\\right) \\tilde{z}^i$, for $k\\geq 0$. Moreover, if $\\beta_k:= 1/(k+2)$, it holds that \n    \\begin{equation}\n        \\label{eq-lemma-zdiff-1}\n        \\begin{aligned}\n            &\\; \\norm{z^{k+1} - z^k}^2 \\\\\n            \\leq &\\; \\left\\{\n            \\begin{array}{ll}\n                \\frac{1}{4L^2} \\norm{\\tilde{z}^0}^2, & \\textrm{if } k = 0,\\\\[5pt]\n                \\frac{2(k+1)^2}{L^2(k+2)^2}\\norm{\\tilde{z}^k}^2 + \\frac{2k}{L^2(k+1)^2(k+2)^2}\\sum_{i = 0}^{k-1}(i+1)^2\\norm{\\tilde{z}^i}^2,    & \\textrm{if } k\\geq 1. \n            \\end{array}\n        \\right.\n        \\end{aligned}\n    \\end{equation}\n    \\end{enumerate}\n\\end{lemma}\n\n\\begin{proof}\n    The first statement of the lemma is obvious. For the second statement, we first note that \n    \\[\n        z^0 - z^{k+1} = (1 - \\beta_k)(z^0 - z^k) + \\frac{1-\\beta_k}{L}\\tilde{z}^k.\n    \\]\n    Hence, it holds that\n    \\[\n        z^0 - z^k = \\frac{1}{1 - \\beta_k} (z^0 - z^{k+1}) - \\frac{1}{L} \\tilde{z}^k,\n    \\]\n    which together with part 1 further implies that \n    \\[\n        z^{k+1} - z^k = \\beta_k(z^0 - z^k) - \\frac{1-\\beta_k}{L}\\tilde{z}^k = \\frac{\\beta_k}{1 - \\beta_k} (z^0 - z^{k+1}) - \\frac{1}{L} \\tilde{z}^k.\n    \\]\n    Therefore, the second statement of the lemma holds true. We next show the third statement via mathematical induction. For $k = 0$, we see from the first statement of the lemma that \n    \\[\n        z^1 - z^0 = -\\frac{{1-\\beta_0}}{L}\\tilde{z}^0,\n    \\]\n    which implies that the third statement of the lemma holds for $k = 0$. Now suppose that the third statement of the lemma holds for $k-1$, i.e., \n    \\[\n        z^{k} - z^{k-1} = -\\frac{1-\\beta_{k-1}}{L}\\tilde{z}^{k-1} + \\sum_{i = 0}^{k-2}\\frac{\\beta_{k-1}}{L}\\left( \\prod_{j = i}^{k-2} (1-\\beta_j)\\right) \\tilde{z}^i.\n    \\]\n    Then, we can see that\n        \\begin{align*}\n             &\\; z^{k+1} - z^k \\\\\n            = &\\;  - \\frac{1-\\beta_k}{L}\\tilde{z}^k + \\beta_k(z^0 - z^k) \\\\\n            = &\\; - \\frac{1-\\beta_k}{L}\\tilde{z}^k + \\beta_k\\left( \\frac{1-\\beta_{k-1}}{\\beta_{k-1}}(z^k - z^{k-1}) + \\frac{1-\\beta_{k-1}}{L\\beta_{k-1}}\\tilde{z}^{k-1}  \\right) \\\\\n             = &\\; - \\frac{1-\\beta_k}{L}\\tilde{z}^k +  \\frac{\\beta_k(1-\\beta_{k-1})}{\\beta_{k-1}}\\left( -\\frac{1-\\beta_{k-1}}{L}\\tilde{z}^{k-1} + \\sum_{i = 0}^{k-2}\\frac{\\beta_{k-1}}{L}\\left( \\prod_{j = i}^{k-2} (1-\\beta_j)\\right) \\tilde{z}^i\\right) \\\\\n             &\\; + \\frac{\\beta_k(1-\\beta_{k-1})}{L\\beta_{k-1}}\\tilde{z}^{k-1}  \\\\\n             =&\\; - \\frac{1-\\beta_k}{L}\\tilde{z}^k + \\sum_{i = 0}^{k-2}\\frac{\\beta_k(1-\\beta_{k-1})}{L}\\left( \\prod_{j = i}^{k-2} (1-\\beta_j)\\right) \\tilde{z}^i + \\frac{\\beta_{k}(1 - \\beta_{k-1})}{L}\\tilde{z}^{k-1} \\\\\n             =&\\; -\\frac{1-\\beta_k}{L}\\tilde{z}^k + \\sum_{i = 0}^{k-1}\\frac{\\beta_k}{L}\\left( \\prod_{j = i}^{k-1} (1-\\beta_j)\\right) \\tilde{z}^i.\n        \\end{align*}\n    Here, we use the first statement of the lemma in the first equality and use the second statement of the lemma in the second equality. Moreover, the third equality is due to the induction assumption and the remaining inequalities are derived from some direct simplifications. Therefore, the statement is true for $k$. By induction, we see that it is true for all $k\\geq 0$.\n    \n    Finally, if $\\beta_k = 1/(k+2)$, we have that $\\beta_0 = 1/2$ and \n    \\[\n        z^1 - z^0 = \\frac{1}{2L}\\tilde{z}^0,\n    \\]\n    which implies that \n    \\[\n        \\norm{z^1 - z^0}^2 = {\\frac{1}{4L^2}}\\norm{\\tilde{z}^0}^2.\n    \\]\n    For $k\\geq 1$, substituting $\\beta_k = 1/(k+2)$ into the  expression of $z^{k+1} - z^k$ in the third statement yields that \n    \\begin{align*}\n        z^{k+1} - z^k = &\\; -\\frac{k+1}{L(k+2)}\\tilde{z}^k + \\sum_{i = 0}^{k-1}\\frac{1}{L(k+2)}\\left( \\prod_{j = i}^{k-1} \\frac{j+1}{j+2}\\right) \\tilde{z}^i \\\\\n        =&\\;  -\\frac{k+1}{L(k+2)}\\tilde{z}^k + \\sum_{i = 0}^{k-1}\\frac{i+1}{L(k+1)(k+2)}\\tilde{z}^i.\n    \\end{align*}\n    Therefore, by applying the Cauchy-Schwarz inequality twice, we see that \n        \\begin{align*}\n            \\norm{z^{k+1} - z^k}^2 = &\\; \\norm{-\\frac{k+1}{L(k+2)}\\tilde{z}^k + \\sum_{i = 0}^{k-1}\\frac{i+1}{L(k+1)(k+2)}\\tilde{z}^i}^2 \\\\\n            \\leq &\\; \\frac{2(k+1)^2}{L^2(k+2)^2}\\norm{\\tilde{z}^k}^2 + \\frac{2}{L^2(k+1)^2(k+2)^2}\\norm{\\sum_{i = 0}^{k-1}(i+1)\\tilde{z}^i}^2 \\\\\n            \\leq &\\; \\frac{2(k+1)^2}{L^2(k+2)^2}\\norm{\\tilde{z}^k}^2 + \\frac{2k}{L^2(k+1)^2(k+2)^2}\\sum_{i = 0}^{k-1}(i+1)^2\\norm{\\tilde{z}^i}^2,\n        \\end{align*}    \n    which proves \\eqref{eq-lemma-zdiff-1}. Hence, the proof is completed.\n\\end{proof}\n\n\\begin{lemma}\n    \\label{lemma-key-estimate}\n    Let $\\{z^k\\}$ be generated by \\eqref{eq-inexact-halpern} with $\\eta_k = (1-\\beta_k)/L$, then for $k\\geq 0$, it holds that \n    \\begin{equation}\n        \\label{eq-lemma-key-estimate-1}\n        \\begin{aligned}\n            &\\; \\frac{1}{2L}\\norm{G(z^{k+1})}^2 - \\frac{\\beta_k}{1-\\beta_k}\\inprod{G(z^{k+1})}{z^0 - z^{k+1}} \\\\ \\leq &\\;   \\frac{1 - 2\\beta_k}{2L}\\norm{G(z^k)}^2 - \\beta_k\\inprod{G(z^k)}{z^0 - z^k} + \\frac{\\beta_k}{L}\\inprod{G(z^k)}{G(z^k) - \\tilde{z}^k}\\\\\n            &\\; + \\frac{1}{2L}\\norm{G(z^k) - \\tilde{z}^k}^2.\n        \\end{aligned}\n    \\end{equation}\n    In particular, if $\\beta_k:=1/(k+2)$, then for $k\\geq 0$,\n    \\begin{equation}\n        \\label{eq-lemma-key-estimate-2}\n        \\mathcal{L}_{k+1} \\leq \\mathcal{L}_k  + \\frac{1}{12L}\\norm{G(z^k)}^2 + \\frac{4(k+1)^2}{L}\\norm{G(z^k) - \\tilde{z}^k}^2,\n    \\end{equation}\n    where $\\mathcal{L}_k$ is the potential function that is defined as \n    \\[\n        \\mathcal{L}_k:= \\frac{k(k+1)}{2L}\\norm{G(z^k)}^2 - (k+1)\\inprod{G(z^k)}{z^0 - z^k},\\quad \\forall k\\geq 0.\n    \\]\n\\end{lemma}\n\n\\begin{proof}\n    By the co-coerciveness of the mapping $G$, we see that \n    \\[\n        \\frac{1}{L}\\norm{G(z^{k+1}) - G(z^k)}^2 \\leq \\inprod{G(z^{k+1}) - G(z^k)}{z^{k+1} - z^k},\n    \\]  \n    which implies that \n    \\begin{equation}\n        \\label{eq-lemma-key-estimate-3}\n        \\begin{aligned}\n            &\\; \\frac{1}{2L}\\norm{G(z^{k+1})}^2 + \\frac{1}{2L}\\norm{G(z^{k})}^2 +  \\frac{1}{2L}\\norm{G(z^{k+1}) - G(z^k)}^2 \\\\\n            \\leq &\\;  \\inprod{G(z^{k+1})}{z^{k+1} - z^k + \\frac{1}{L}G(z^k)} - \\inprod{G(z^k)}{z^{k+1} - z^k}.\n        \\end{aligned}\n    \\end{equation}\n    The first two expressions for $z^{k+1} - z^k$ in Lemma \\ref{lemma-zdiff} together with \\eqref{eq-lemma-key-estimate-3} and the Cauchy-Schwarz inequality imply that \n        \\begin{align*}\n            &\\; \\frac{1}{2L}\\norm{G(z^{k+1})}^2 + \\frac{1}{2L}\\norm{G(z^{k})}^2 +  \\frac{1}{2L}\\norm{G(z^{k+1}) - G(z^k)}^2 \\\\\n            \\leq &\\; \\inprod{G(z^{k+1})}{\\frac{\\beta_k}{1-\\beta_k}(z^0 - z^{k+1}) - \\frac{1}{L}\\tilde{z}^k + \\frac{1}{L}G(z^k)} - \\inprod{G(z^k)}{\\beta_k(z^0 - z^k) - \\frac{1-\\beta_k}{L}\\tilde{z}^k} \\\\\n            = &\\; \\frac{\\beta_k}{1-\\beta_k}\\inprod{G(z^{k+1})}{z^0 - z^{k+1}} - \\beta_k\\inprod{G(z^k)}{z^0 - z^k} + \\frac{1}{L}\\inprod{G(z^{k+1})}{G(z^k) - \\tilde{z}^k} \\\\\n            &\\; + \\frac{1-\\beta_k}{L}\\inprod{G(z^k)}{\\tilde{z}^k}\\\\\n            = &\\; \\frac{\\beta_k}{1-\\beta_k}\\inprod{G(z^{k+1})}{z^0 - z^{k+1}} - \\beta_k\\inprod{G(z^k)}{z^0 - z^k}  + \\frac{1}{L}\\norm{G(z^k)}^2 \\\\\n            &\\; + \\frac{1}{L}\\inprod{G(z^{k+1}) - G(z^k)}{G(z^k) - \\tilde{z}^k} - \\frac{\\beta_k}{L}\\inprod{G(z^k)}{\\tilde{z}^k} \\\\\n            \\leq &\\; \\frac{\\beta_k}{1-\\beta_k}\\inprod{G(z^{k+1})}{z^0 - z^{k+1}} - \\beta_k\\inprod{G(z^k)}{z^0 - z^k}  + \\frac{1 - \\beta_k}{L}\\norm{G(z^k)}^2 \\\\\n            &\\; + \\frac{1}{2L}\\norm{G(z^{k+1}) - G(z^k)}^2 + \\frac{1}{2L}\\norm{G(z^k) - \\tilde{z}^k}^2 + \\frac{\\beta_k}{L}\\inprod{G(z^k)}{G(z^k) - \\tilde{z}^k}.\n        \\end{align*}\n    Therefore, the inequality \\eqref{eq-lemma-key-estimate-1} holds by rearranging terms in the above inequality. By substituting $ \\beta_k:= 1 / (k+2) $ into \\eqref{eq-lemma-key-estimate-1}, and using the Cauchy-Schwarz inequality and the definition of the potential function $ \\mathcal{L}_k $, for $k\\geq 0$, we see that \n    \t\\begin{align*}\n    \t\t\\mathcal{L}_{k+1} \\leq &\\;  \\mathcal{L}_k  + \\frac{k+1}{L}\\inprod{G(z^k)}{G(z^k) - \\tilde{z}^k} + \\frac{(k+1)(k+2)}{2L}\\norm{G(z^k) - \\tilde{z}^k}^2 \\\\\n    \t\t\\leq &\\; \\mathcal{L}_k  + \\frac{k+1}{L}\\inprod{G(z^k)}{G(z^k) - \\tilde{z}^k} + \\frac{(k+1)^2}{L}\\norm{G(z^k) - \\tilde{z}^k}^2 \\\\\n    \t\t\\leq &\\; \\mathcal{L}_k  + \\frac{k+1}{2L}\\left(\\frac{1}{6(k+1)}\\norm{G(z^k)}^2 + 6(k+1) \\norm{G(z^k) - \\tilde{z}^k}^2 \\right) \\\\\n            &\\; + \\frac{(k+1)^2}{L}\\norm{G(z^k) - \\tilde{z}^k}^2 \\\\\n    \t\t= &\\; \\mathcal{L}_k  + \\frac{1}{12L}\\norm{G(z^k)}^2 + \\frac{4(k+1)^2}{L}\\norm{G(z^k) - \\tilde{z}^k}^2,\n    \t\\end{align*}\n    which proves the inequality \\eqref{eq-lemma-key-estimate-2}. Therefore, the proof is completed.\n\\end{proof}\n\n\\begin{lemma}\n\t\\label{lemma-exp-error}\n\tLet $\\{z^k\\}$ be the sequence generated by \\eqref{eq-isHalpern}, then it holds that\n\t\\[\n\t\t\\bb{E} \\left[\\norm{G(z^k) - \\tilde{z}^k}^2\\right] \\leq \\gamma_k + \\sigma_k,\\quad \\forall k\\geq 0.\n\t\\]\n\\end{lemma}\n\n\\begin{proof}\n\tSince the resolvent operator $J_{\\alpha E}$ is firmly non-expansive (hence $1$-Lipschitz continuous), we see that \n\t\t\\begin{align*}\n\t\t\t&\\; \\norm{G(z^k) - \\tilde{z}^k}^2 \\\\\n\t\t\t= &\\; \\norm{\\frac{1}{\\alpha}\\left(z^k - J_{\\alpha E}\\left(z^k -  \\alpha F(z^k)\\right)\\right)- \\tilde{z}^k}^2 \\\\\n\t\t\t= &\\; \\frac{1}{\\alpha^2}\\norm{J_{\\alpha E}\\left(z^k -  \\alpha F(z^k)\\right) - \\bar{z}^k}^2 \\\\\n\t\t\t\\leq &\\; \\frac{2}{\\alpha^2}\\norm{J_{\\alpha E}\\left(z^k -  \\alpha F(z^k)\\right) - J_{\\alpha E}\\left(z^k -  \\alpha \\widetilde{F}(z^k)\\right)}^2 + \\frac{2}{\\alpha^2}\\norm{J_{\\alpha E}\\left(z^k - \\alpha \\widetilde{F}(z^k)\\right) - \\bar{z}^k}^2 \\\\\n\t\t\t\\leq &\\; 2\\norm{F(z^k) - \\widetilde{F}(z^k)}^2 + \\gamma_k,\n\t\t\\end{align*}\n    where the first equality uses the definition of $G$ and the second equality uses the fact that $\\tilde z^k = \\frac{1}{\\alpha}(z^k - \\bar{z}^k)$, and the first inequality is due to the Cauchy-Schwarz inequality.\n\tThen, it follows that: \n\t\\[\n\t\t\\bb{E} \\left[\\norm{G(z^k) - \\tilde{z}^k}^2\\right] \\leq  2\\bb{E}\\left[\\norm{F(z^k) - \\widetilde{F}(z^k)}^2\\right] +  \\gamma_k \n\t\t\t\\leq    \\sigma_k + \\gamma_k,\n\t\\]\n\twhich completes the proof.\n\\end{proof}\n\n\\begin{proof}\n\tFrom Lemma \\ref{lemma-key-estimate} and notice that $\\mathcal{L}_0 = 0$, we can see that  \n\t\\[\n\t\t\\mathcal{L}_k \\leq \\frac{1}{12L}\\sum_{i = 0}^{k-1}\\norm{G(z^i)}^2 + \\frac{4}{L}\\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2, \\quad k\\geq 1.\n\t\\]\n\tBy the definition of the potential function $\\mathcal{L}_k$, we can see that \n\t\t\\begin{align*}\n\t\t\t\\frac{k(k+1)}{2L}\\norm{G(z^k)}^2 \\leq &\\; \\frac{1}{12L}\\sum_{i = 0}^{k-1}\\norm{G(z^i)}^2 + \\frac{4}{L}\\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2 + (k+1)\\inprod{G(z^k)}{z^0 - z^k} \\\\\n\t\t\t\\leq &\\; \\frac{1}{12L}\\sum_{i = 0}^{k-1}\\norm{G(z^i)}^2 + \\frac{4}{L}\\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2 + (k+1)\\norm{G(z^k)}\\norm{z^0 - z^*}\n\t\t\\end{align*} \n\twhere the last inequality is due to the fact that \n\t\\begin{align*}\n\t    \\inprod{G(z^k)}{z^0 - z^k} = &\\; \\inprod{G(z^k)}{z^0 - z^*} + \\inprod{G(z^k)}{z^* - z^k} \\\\\n     \\leq &\\;  \\inprod{G(z^k)}{z^0 - z^*} \\\\\n     \\leq &\\;  \\norm{G(z^k)}\\norm{z^0 - z^*},\n\t\\end{align*}\n\tand $z^*$ is any solution such that $G(z^*) = 0$.\n The first inequality holds, since $G$ is co-coercive. As a consequence, it holds that \n\t\\begin{equation}\n\t\\label{thm-rate-proof-1}\n\t\t\\norm{G(z^k)}^2 \\leq \\frac{1}{6k(k+1)}\\sum_{i = 0}^{k-1}\\norm{G(z^i)}^2 + \\frac{8}{k(k+1)}\\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2 + \\frac{2L}{k}\\norm{G(z^k)}\\norm{z^0 - z^*},  \n\t\\end{equation}\n\tfor $k\\geq 1$. For notational simplicity, let us denote \n\t\\[\n\t\ta_k:= \\norm{G(z^k)}, \\quad k\\geq 0, \\quad b_k:= \\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2,\\quad k\\geq 1,\\quad D:= L\\norm{z^0 - z^*}.\n\t\\]\n\tThen, from \\eqref{thm-rate-proof-1}, we see that, for $k\\geq 1$, \n\t\\begin{align*}\n\t\ta_k^2 \\leq &\\;  \\frac{1}{6k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{8}{k(k+1)}b_k + \\frac{2D}{k}a_k \\\\\n\t\t\\leq &\\; \\frac{1}{6k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{8}{k(k+1)}b_k + \\frac{1}{2}\\left(\\frac{4D^2}{k^2} + a_k^2 \\right),\n\t\\end{align*}\n\twhich implies that, for $k\\geq 1$,\n\t\\begin{equation}\n\t\\label{thm-rate-proof-2}\n\t\\begin{aligned}\n\t    a_k^2 \\leq &\\; \\frac{1}{3k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{16}{k(k+1)}b_k + \\frac{4D^2}{k^2} \\\\\n     \\leq &\\; \\frac{1}{3k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)}.\n\t\\end{aligned}\n\t\\end{equation}\n\tWe next claim that the following inequality holds:\n\t\\begin{equation}\n\t\t\\label{thm-rate-proof-3}\n\t\ta_k^2 \\leq \\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)},\\quad k\\geq 1.\n\t\\end{equation}\n\tWe shall prove the above claim by induction. First, from \\eqref{thm-rate-proof-2} and notice that $a_0 \\leq D$ by the $L$-Lipschitz continuity of $G$, we see that\n\t\\[\n\t\ta_1^2 \\leq \\frac{1}{6}a_0^2 + 8b_1 + 4D^2 \\leq  \\frac{1}{6}D^2 + 8b_1 + 4D^2 \\leq \\frac{1}{6}(7D+10\\sqrt{b_1})^2.\n\t\\]\n\tSo \\eqref{thm-rate-proof-3} holds for $k = 1$. Suppose now that \\eqref{thm-rate-proof-3} holds for indices $\\{1,\\dots, k-1\\}$. Again, from \\eqref{thm-rate-proof-2} and the fact that $b_k$ is non-decreasing, we deduce that \n\t\t\\begin{align*}\n\t\t\ta_k^2 \\leq &\\; \\frac{1}{3k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)} \\\\\n\t\t\t\\leq &\\; \\frac{1}{3k(k+1)}\\sum_{i = 0}^{k-1} \\frac{(7D + 10\\sqrt{b_i})^2}{(i+1)(i+2)} + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)} \\\\\n\t\t\t\\leq &\\; \\frac{(7D + 10\\sqrt{b_k})^2}{3k(k+1)}\\sum_{i = 0}^{k-1} \\frac{1}{(i+1)(i+2)} + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)} \\\\\n\t\t\t= &\\; \\frac{(7D + 10\\sqrt{b_k})^2}{3(k+1)^2} + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)} \\\\\n\t\t\t\\leq &\\; \\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)},\n\t\t\\end{align*}\n\twhich shows that \\eqref{thm-rate-proof-3} holds for index $k$. Thus, by induction, \\eqref{thm-rate-proof-3} holds for all $k\\geq 1$. Moreover, it is clear that \\eqref{thm-rate-proof-3} trivially holds for $k = 0$. This proves \\eqref{thm-rate-1}.\n\t\n\tTo prove  \\eqref{thm-rate-2}, we first observe that \n\t\\[\n\t\t\\norm{\\tilde{z}^k}^2 \\leq 2\\norm{G(z^k)}^2 + 2\\norm{G(z^k) - \\tilde{z}^k}^2 \\leq 2a_k^2 + 2\\gamma_k^2 \\leq  \\frac{2(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)} + 2\\gamma_k^2,\\quad k\\geq 0.\n\t\\]\n\tThen, by the third statement of Lemma \\ref{lemma-zdiff} and the fact that $b_k$ is non-decreasing, we see that for $k\\geq 0$, \n\t\t\\begin{align*}\n\t\t\t\\norm{z^{k+1} - z^k}^2\n\t\t\t\\leq &\\; \\frac{2(k+1)^2}{L^2(k+2)^2}\\norm{\\tilde{z}^k}^2 + \\frac{2k}{L^2(k+1)^2(k+2)^2}\\sum_{i = 0}^{k-1}(i+1)^2\\norm{\\tilde{z}^i}^2 \\\\\n\t\t\t% \\leq &\\;  \\frac{4(k+1)^2}{L^2(k+2)^2}\\left(\\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)} + \\gamma_k^2 \\right) \\\\\n   % &\\; + \\frac{4k}{L^2(k+1)^2(k+2)^2}\\sum_{i = 0}^{k-1}\\left(\\frac{(i+1)(7D + 10\\sqrt{b_i})^2}{i+2} + (i+1)^2 \\gamma_i^2 \\right) \\\\\n\t\t\t\\leq &\\;  \\frac{4(k+1)^2}{L^2(k+2)^2}\\left(\\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)} + \\gamma_k^2 \\right) \\\\\n   &\\; + \\frac{4k}{L^2(k+1)^2(k+2)^2}\\sum_{i = 0}^{k-1}\\left(\\frac{(i+1)(7D + 10\\sqrt{b_k})^2}{i+2} + (i+1)^2 \\gamma_i^2 \\right) \\\\\n\t\t\t\\leq &\\; \\frac{4(7D + 10\\sqrt{b_k})^2}{L^2}\\left(\\frac{k+1}{(k+2)^3} + \\frac{k^2}{(k+1)^2(k+2)^2}\\right) \\\\\n   &\\; + \\frac{4(k+1)^2}{L^2(k+2)^2}\\gamma_k^2 + \\frac{4k}{L^2(k+1)^2(k+2)^2}b_k \\\\\n\t\t\t\\leq &\\; \\frac{8(7D + 10\\sqrt{b_{k+1}})^2 + 8b_{k+1}}{L^2(k+1)(k+2)}\\\\\n\t\t\t\\leq &\\; \\frac{8(7D + 11\\sqrt{b_{k+1}})^2 }{L^2(k+1)(k+2)},\n\t\t\\end{align*}\n\twhich proves \\eqref{thm-rate-2}. Therefore, the proof is completed.\n\\end{proof}\n\n\\begin{proof}\n\tWe recall from Lemma \\ref{lemma-key-estimate} that \n\t\\[\n\t\t\\mathcal{L}_{k+1} \\leq \\mathcal{L}_k  + \\frac{1}{12L}\\norm{G(z^k)}^2 + \\frac{4(k+1)^2}{L}\\norm{G(z^k) - \\tilde{z}^k}^2,\\quad k\\geq 0.\n\t\\]\n\tTaking the full expectation $\\bb{E}$ (with respect to all the randomness) on both side of the above inequality yields \n\t\\[\n\t\t\\bb{E}[\\mathcal{L}_{k+1}] \\leq \\bb{E}[\\mathcal{L}_k]  + \\frac{1}{12L}\\bb{E}\\left[\\norm{G(z^k)}^2\\right] + \\frac{4(k+1)^2}{L}(\\sigma_k + \\gamma_k)^2,\\quad k\\geq 0,\n\t\\]\n\twhere the third term is due to Lemma \\ref{lemma-exp-error}. Then, by recursively applying the above and summing up the resulted inequalities gives:\n\t\\begin{equation}\\label{eq-thm-ishi-1}\n\t\\bb{E}[\\mathcal{L}_k] \\leq \\frac{1}{12L}\\sum_{i = 0}^{k-1}\\bb{E}\\norm{G(z^i)}^2 + \\frac{4}{L}\\sum_{i = 1}^{k-1}(i+1)^2(\\sigma_i + \\gamma_i)^2,\\quad k\\geq 0.\t\n\t\\end{equation}\n\tHere, the case when $k= 0$ holds trivially since $\\mathcal{L}_0 = 0$. By the definition of the potential function $\\mathcal{L}_k$ \n\twe see that \n\t\\begin{equation}\n\t\\label{eq-thm-ishi-2}\n\t\t\\bb{E}[\\mathcal{L}_k] = \\frac{k(k+1)}{2L}\\bb{E}\\left[\\norm{G(z^k)}^2\\right] - (k+1)\\bb{E}\\left[\\inprod{G(z^k)}{z^0 - z^k}\\right],\\quad k\\geq 0.\t\n\t\\end{equation}\n\tMoreover, let $z^*$ be any solution such that $G(z^*) = 0$, since \n\t\\begin{align*}\n\t    \\bb{E}\\left[\\inprod{G(z^k)}{z^0 - z^k}\\right] = &\\; \\bb{E}\\left[\\inprod{G(z^k)}{z^0 - z^*}\\right] + \\bb{E}\\left[\\inprod{G(z^k)}{z^* - z^k}\\right] \\\\\n        \\leq &\\;  \\bb{E}\\left[\\inprod{G(z^k)}{z^0 - z^*}\\right],\n\t\\end{align*}\n\tit holds that \n\t\\begin{equation}\n\t\\label{eq-thm-ishi-3}\n\t\t\\bb{E}\\left[\\inprod{G(z^k)}{z^0 - z^k}\\right] \\leq \\bb{E}\\left[\\norm{G(z^k)}\\norm{z^0 - z^*}\\right] = \\norm{z^0 - z^*}\\bb{E}\\left[\\norm{G(z^k)}\\right].\t\n\t\\end{equation}\n\tCombining \\eqref{eq-thm-ishi-1}, \\eqref{eq-thm-ishi-2} and \\eqref{eq-thm-ishi-3}, we get\n\t\\begin{equation*}\n\t% \\label{eq-thm-ishi-4}\n\t\t\\begin{aligned}\n\t\t\t&\\; \\bb{E}\\left[\\norm{G(z^k)}^2 \\right] \\\\\n\t\t\t\\leq &\\; \\frac{1}{6k(k+1)}\\sum_{i = 0}^{k-1}\\bb{E}\\left[\\norm{G(z^i)}^2\\right] + \\frac{2L}{k}\\norm{z^0 - z^*}\\bb{E}\\left[\\norm{G(z^k)}\\right] + \\frac{8}{k(k+1)}\\sum_{i = 0}^{k-1}(i+1)^2(\\sigma_i + \\gamma_i)^2 \\\\\n\t\t\t\\leq &\\; \\frac{1}{6k(k+1)}\\sum_{i = 0}^{k-1}\\bb{E}\\left[\\norm{G(z^i)}^2\\right] + \\frac{2L}{k}\\norm{z^0 - z^*}\\left(\\bb{E}\\left[\\norm{G(z^k)}^2\\right]\\right)^{\\frac{1}{2}} \\\\\n   &\\; + \\frac{8}{k(k+1)}\\sum_{i = 0}^{k-1}(i+1)^2(\\sigma_i + \\gamma_i)^2,\n\t\t\\end{aligned}\n\t\\end{equation*}\n\tfor all $ k\\geq 1 $. Now denote \n\t\\[\n\t\ta_k:= \\left(\\bb{E}\\left[\\norm{G(z^k)}^2\\right]\\right)^{\\frac{1}{2}}, \\quad k\\geq 0, \\quad b_k:= \\sum_{i = 0}^{k-1}(i+1)^2(\\sigma_i + \\gamma_i)^2,\\quad k\\geq 1,\\quad D:= L\\norm{z^0 - z^*}.\n\t\\]\n\tSimilar to \\eqref{thm-rate-proof-2}, we can derive \n\t\\[\n\t\ta_k^2 \\leq \\frac{1}{3k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)}.\n\t\\]\n\tThen, by following the proof of Theorem \\ref{thm-convergence-rate} directly, we see that \n\t\\[\n\t\ta_k^2\\leq \\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)}, \\quad k\\geq 1,\n\t\\]\n\twhich proves \\eqref{eq-thm-rate-ishi-G} since the case for $k = 0$ is trivial. Moreover, we can also verify that\n\t\t\\begin{align*}\n\t\t    \\bb{E}\\left[\\norm{\\tilde{z}^k}^2\\right] \\leq &\\;  2\\bb{E}\\left[\\norm{G(z^k)}^2 + \\norm{\\tilde{z}^k - G(z^k)}^2\\right] \\\\\n      \\leq &\\; 2a_k^2 + 2(\\sigma_k + \\gamma_k)^2 \\\\\n      \\leq  &\\; \\frac{2(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)} + 2(\\sigma_k + \\gamma_k)^2.\n\t\t\\end{align*}\n\tThen, by following the proof of Theorem \\ref{thm-convergence-rate} again, we can show that \n\t\\[\n\t\ta_k^2\\leq \\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)},\\quad \\bb{E}\\left[\\norm{z^{k+1} - z^k}^2\\right] \\leq \\frac{8(7D + 11\\sqrt{b_{k+1}})^2 }{L^2(k+1)(k+2)},\\quad k\\geq 0,\n\t\\]\n\twhich is exactly \\eqref{eq-thm-rate-ishi-Z}. Hence, the proof is completed.\n\\end{proof}","units":[{"start":28329,"end":29015,"start_line":227,"end_line":235,"raw_tex":"\\begin{theorem}\n\t\\label{thm-convergence-rate}\n\tSuppose that $G$ is $\\frac{1}{L}$-co-coercive for a constant $L>0$. Let $\\{z^k\\}$ be generated by \\eqref{eq-inexact-halpern} with $\\beta_k = 1/(k+2)$ and $\\eta_k = (1-\\beta_k)/L$, then it holds that\n\t\\begin{align}\n\t\t\\norm{G(z^k)}^2 \\leq &\\;  \\frac{\\left(7L\\norm{z^0 - z^*} + 10\\sqrt{\\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2}\\right)^2}{(k+1)(k+2)},\\quad k\\geq 0, \\label{thm-rate-1}\\\\\n\t\t\\norm{z^{k+1} - z^k}^2 \\leq &\\; \\frac{8\\left(7L\\norm{z^0 - z^*} + 11\\sqrt{\\sum_{i = 0}^{k}(i+1)^2\\gamma_i^2}\\right)^2 }{L^2(k+1)(k+2)}, \\quad k\\geq 0,\\label{thm-rate-2}\n\t\\end{align}\n\twhere $z^* \\in \\bb{Z}$ is any solution such that $G(z^*) = 0$.\n\\end{theorem}","sha256_utf8":"39bc3b28370d8f1fe03926787ae5ac5d6ca73c54c544fd4162146b24e8a7f061"},{"start":77704,"end":78877,"start_line":753,"end_line":773,"raw_tex":"\\begin{lemma}\n    \\label{lemma-zdiff}\n    Let $\\{z^k\\}$ be generated by \\eqref{eq-inexact-halpern} with $\\eta_k = (1-\\beta_k)/L$, then the following expressions for $z^{k+1} - z^k$ for $k\\geq 0$ hold:\n    \\begin{enumerate}\n        \\item $z^{k+1} - z^k = \\beta_k(z^0 - z^k) - \\frac{1-\\beta_k}{L}\\tilde{z}^k$;\n        \\item $z^{k+1} - z^k = \\frac{\\beta_k}{1-\\beta_k}(z^0 - z^{k+1}) - \\frac{1}{L}\\tilde{z}^k$;\n        \\item $z^{k+1} - z^k = -\\frac{1-\\beta_k}{L}\\tilde{z}^k + \\sum_{i = 0}^{k-1}\\frac{\\beta_k}{L}\\left( \\prod_{j = i}^{k-1} (1-\\beta_j)\\right) \\tilde{z}^i$, for $k\\geq 0$. Moreover, if $\\beta_k:= 1/(k+2)$, it holds that \n    \\begin{equation}\n        \\label{eq-lemma-zdiff-1}\n        \\begin{aligned}\n            &\\; \\norm{z^{k+1} - z^k}^2 \\\\\n            \\leq &\\; \\left\\{\n            \\begin{array}{ll}\n                \\frac{1}{4L^2} \\norm{\\tilde{z}^0}^2, & \\textrm{if } k = 0,\\\\[5pt]\n                \\frac{2(k+1)^2}{L^2(k+2)^2}\\norm{\\tilde{z}^k}^2 + \\frac{2k}{L^2(k+1)^2(k+2)^2}\\sum_{i = 0}^{k-1}(i+1)^2\\norm{\\tilde{z}^i}^2,    & \\textrm{if } k\\geq 1. \n            \\end{array}\n        \\right.\n        \\end{aligned}\n    \\end{equation}\n    \\end{enumerate}\n\\end{lemma}","sha256_utf8":"bd0f78652dde6f9f1a0807251a0ac9d74b0e257031c31371d56ed83806c073b9"},{"start":78879,"end":82750,"start_line":775,"end_line":828,"raw_tex":"\\begin{proof}\n    The first statement of the lemma is obvious. For the second statement, we first note that \n    \\[\n        z^0 - z^{k+1} = (1 - \\beta_k)(z^0 - z^k) + \\frac{1-\\beta_k}{L}\\tilde{z}^k.\n    \\]\n    Hence, it holds that\n    \\[\n        z^0 - z^k = \\frac{1}{1 - \\beta_k} (z^0 - z^{k+1}) - \\frac{1}{L} \\tilde{z}^k,\n    \\]\n    which together with part 1 further implies that \n    \\[\n        z^{k+1} - z^k = \\beta_k(z^0 - z^k) - \\frac{1-\\beta_k}{L}\\tilde{z}^k = \\frac{\\beta_k}{1 - \\beta_k} (z^0 - z^{k+1}) - \\frac{1}{L} \\tilde{z}^k.\n    \\]\n    Therefore, the second statement of the lemma holds true. We next show the third statement via mathematical induction. For $k = 0$, we see from the first statement of the lemma that \n    \\[\n        z^1 - z^0 = -\\frac{{1-\\beta_0}}{L}\\tilde{z}^0,\n    \\]\n    which implies that the third statement of the lemma holds for $k = 0$. Now suppose that the third statement of the lemma holds for $k-1$, i.e., \n    \\[\n        z^{k} - z^{k-1} = -\\frac{1-\\beta_{k-1}}{L}\\tilde{z}^{k-1} + \\sum_{i = 0}^{k-2}\\frac{\\beta_{k-1}}{L}\\left( \\prod_{j = i}^{k-2} (1-\\beta_j)\\right) \\tilde{z}^i.\n    \\]\n    Then, we can see that\n        \\begin{align*}\n             &\\; z^{k+1} - z^k \\\\\n            = &\\;  - \\frac{1-\\beta_k}{L}\\tilde{z}^k + \\beta_k(z^0 - z^k) \\\\\n            = &\\; - \\frac{1-\\beta_k}{L}\\tilde{z}^k + \\beta_k\\left( \\frac{1-\\beta_{k-1}}{\\beta_{k-1}}(z^k - z^{k-1}) + \\frac{1-\\beta_{k-1}}{L\\beta_{k-1}}\\tilde{z}^{k-1}  \\right) \\\\\n             = &\\; - \\frac{1-\\beta_k}{L}\\tilde{z}^k +  \\frac{\\beta_k(1-\\beta_{k-1})}{\\beta_{k-1}}\\left( -\\frac{1-\\beta_{k-1}}{L}\\tilde{z}^{k-1} + \\sum_{i = 0}^{k-2}\\frac{\\beta_{k-1}}{L}\\left( \\prod_{j = i}^{k-2} (1-\\beta_j)\\right) \\tilde{z}^i\\right) \\\\\n             &\\; + \\frac{\\beta_k(1-\\beta_{k-1})}{L\\beta_{k-1}}\\tilde{z}^{k-1}  \\\\\n             =&\\; - \\frac{1-\\beta_k}{L}\\tilde{z}^k + \\sum_{i = 0}^{k-2}\\frac{\\beta_k(1-\\beta_{k-1})}{L}\\left( \\prod_{j = i}^{k-2} (1-\\beta_j)\\right) \\tilde{z}^i + \\frac{\\beta_{k}(1 - \\beta_{k-1})}{L}\\tilde{z}^{k-1} \\\\\n             =&\\; -\\frac{1-\\beta_k}{L}\\tilde{z}^k + \\sum_{i = 0}^{k-1}\\frac{\\beta_k}{L}\\left( \\prod_{j = i}^{k-1} (1-\\beta_j)\\right) \\tilde{z}^i.\n        \\end{align*}\n    Here, we use the first statement of the lemma in the first equality and use the second statement of the lemma in the second equality. Moreover, the third equality is due to the induction assumption and the remaining inequalities are derived from some direct simplifications. Therefore, the statement is true for $k$. By induction, we see that it is true for all $k\\geq 0$.\n    \n    Finally, if $\\beta_k = 1/(k+2)$, we have that $\\beta_0 = 1/2$ and \n    \\[\n        z^1 - z^0 = \\frac{1}{2L}\\tilde{z}^0,\n    \\]\n    which implies that \n    \\[\n        \\norm{z^1 - z^0}^2 = {\\frac{1}{4L^2}}\\norm{\\tilde{z}^0}^2.\n    \\]\n    For $k\\geq 1$, substituting $\\beta_k = 1/(k+2)$ into the  expression of $z^{k+1} - z^k$ in the third statement yields that \n    \\begin{align*}\n        z^{k+1} - z^k = &\\; -\\frac{k+1}{L(k+2)}\\tilde{z}^k + \\sum_{i = 0}^{k-1}\\frac{1}{L(k+2)}\\left( \\prod_{j = i}^{k-1} \\frac{j+1}{j+2}\\right) \\tilde{z}^i \\\\\n        =&\\;  -\\frac{k+1}{L(k+2)}\\tilde{z}^k + \\sum_{i = 0}^{k-1}\\frac{i+1}{L(k+1)(k+2)}\\tilde{z}^i.\n    \\end{align*}\n    Therefore, by applying the Cauchy-Schwarz inequality twice, we see that \n        \\begin{align*}\n            \\norm{z^{k+1} - z^k}^2 = &\\; \\norm{-\\frac{k+1}{L(k+2)}\\tilde{z}^k + \\sum_{i = 0}^{k-1}\\frac{i+1}{L(k+1)(k+2)}\\tilde{z}^i}^2 \\\\\n            \\leq &\\; \\frac{2(k+1)^2}{L^2(k+2)^2}\\norm{\\tilde{z}^k}^2 + \\frac{2}{L^2(k+1)^2(k+2)^2}\\norm{\\sum_{i = 0}^{k-1}(i+1)\\tilde{z}^i}^2 \\\\\n            \\leq &\\; \\frac{2(k+1)^2}{L^2(k+2)^2}\\norm{\\tilde{z}^k}^2 + \\frac{2k}{L^2(k+1)^2(k+2)^2}\\sum_{i = 0}^{k-1}(i+1)^2\\norm{\\tilde{z}^i}^2,\n        \\end{align*}    \n    which proves \\eqref{eq-lemma-zdiff-1}. Hence, the proof is completed.\n\\end{proof}","sha256_utf8":"ce1e93f34e137711d70e37200054da35e263ed34e940833a2cab44c707947538"},{"start":82951,"end":84049,"start_line":833,"end_line":852,"raw_tex":"\\begin{lemma}\n    \\label{lemma-key-estimate}\n    Let $\\{z^k\\}$ be generated by \\eqref{eq-inexact-halpern} with $\\eta_k = (1-\\beta_k)/L$, then for $k\\geq 0$, it holds that \n    \\begin{equation}\n        \\label{eq-lemma-key-estimate-1}\n        \\begin{aligned}\n            &\\; \\frac{1}{2L}\\norm{G(z^{k+1})}^2 - \\frac{\\beta_k}{1-\\beta_k}\\inprod{G(z^{k+1})}{z^0 - z^{k+1}} \\\\ \\leq &\\;   \\frac{1 - 2\\beta_k}{2L}\\norm{G(z^k)}^2 - \\beta_k\\inprod{G(z^k)}{z^0 - z^k} + \\frac{\\beta_k}{L}\\inprod{G(z^k)}{G(z^k) - \\tilde{z}^k}\\\\\n            &\\; + \\frac{1}{2L}\\norm{G(z^k) - \\tilde{z}^k}^2.\n        \\end{aligned}\n    \\end{equation}\n    In particular, if $\\beta_k:=1/(k+2)$, then for $k\\geq 0$,\n    \\begin{equation}\n        \\label{eq-lemma-key-estimate-2}\n        \\mathcal{L}_{k+1} \\leq \\mathcal{L}_k  + \\frac{1}{12L}\\norm{G(z^k)}^2 + \\frac{4(k+1)^2}{L}\\norm{G(z^k) - \\tilde{z}^k}^2,\n    \\end{equation}\n    where $\\mathcal{L}_k$ is the potential function that is defined as \n    \\[\n        \\mathcal{L}_k:= \\frac{k(k+1)}{2L}\\norm{G(z^k)}^2 - (k+1)\\inprod{G(z^k)}{z^0 - z^k},\\quad \\forall k\\geq 0.\n    \\]\n\\end{lemma}","sha256_utf8":"e27dc8b1e001355f818e0606f3e87c41ce6208f728be6b5193fadf968ae75a0f"},{"start":84051,"end":87085,"start_line":854,"end_line":887,"raw_tex":"\\begin{proof}\n    By the co-coerciveness of the mapping $G$, we see that \n    \\[\n        \\frac{1}{L}\\norm{G(z^{k+1}) - G(z^k)}^2 \\leq \\inprod{G(z^{k+1}) - G(z^k)}{z^{k+1} - z^k},\n    \\]  \n    which implies that \n    \\begin{equation}\n        \\label{eq-lemma-key-estimate-3}\n        \\begin{aligned}\n            &\\; \\frac{1}{2L}\\norm{G(z^{k+1})}^2 + \\frac{1}{2L}\\norm{G(z^{k})}^2 +  \\frac{1}{2L}\\norm{G(z^{k+1}) - G(z^k)}^2 \\\\\n            \\leq &\\;  \\inprod{G(z^{k+1})}{z^{k+1} - z^k + \\frac{1}{L}G(z^k)} - \\inprod{G(z^k)}{z^{k+1} - z^k}.\n        \\end{aligned}\n    \\end{equation}\n    The first two expressions for $z^{k+1} - z^k$ in Lemma \\ref{lemma-zdiff} together with \\eqref{eq-lemma-key-estimate-3} and the Cauchy-Schwarz inequality imply that \n        \\begin{align*}\n            &\\; \\frac{1}{2L}\\norm{G(z^{k+1})}^2 + \\frac{1}{2L}\\norm{G(z^{k})}^2 +  \\frac{1}{2L}\\norm{G(z^{k+1}) - G(z^k)}^2 \\\\\n            \\leq &\\; \\inprod{G(z^{k+1})}{\\frac{\\beta_k}{1-\\beta_k}(z^0 - z^{k+1}) - \\frac{1}{L}\\tilde{z}^k + \\frac{1}{L}G(z^k)} - \\inprod{G(z^k)}{\\beta_k(z^0 - z^k) - \\frac{1-\\beta_k}{L}\\tilde{z}^k} \\\\\n            = &\\; \\frac{\\beta_k}{1-\\beta_k}\\inprod{G(z^{k+1})}{z^0 - z^{k+1}} - \\beta_k\\inprod{G(z^k)}{z^0 - z^k} + \\frac{1}{L}\\inprod{G(z^{k+1})}{G(z^k) - \\tilde{z}^k} \\\\\n            &\\; + \\frac{1-\\beta_k}{L}\\inprod{G(z^k)}{\\tilde{z}^k}\\\\\n            = &\\; \\frac{\\beta_k}{1-\\beta_k}\\inprod{G(z^{k+1})}{z^0 - z^{k+1}} - \\beta_k\\inprod{G(z^k)}{z^0 - z^k}  + \\frac{1}{L}\\norm{G(z^k)}^2 \\\\\n            &\\; + \\frac{1}{L}\\inprod{G(z^{k+1}) - G(z^k)}{G(z^k) - \\tilde{z}^k} - \\frac{\\beta_k}{L}\\inprod{G(z^k)}{\\tilde{z}^k} \\\\\n            \\leq &\\; \\frac{\\beta_k}{1-\\beta_k}\\inprod{G(z^{k+1})}{z^0 - z^{k+1}} - \\beta_k\\inprod{G(z^k)}{z^0 - z^k}  + \\frac{1 - \\beta_k}{L}\\norm{G(z^k)}^2 \\\\\n            &\\; + \\frac{1}{2L}\\norm{G(z^{k+1}) - G(z^k)}^2 + \\frac{1}{2L}\\norm{G(z^k) - \\tilde{z}^k}^2 + \\frac{\\beta_k}{L}\\inprod{G(z^k)}{G(z^k) - \\tilde{z}^k}.\n        \\end{align*}\n    Therefore, the inequality \\eqref{eq-lemma-key-estimate-1} holds by rearranging terms in the above inequality. By substituting $ \\beta_k:= 1 / (k+2) $ into \\eqref{eq-lemma-key-estimate-1}, and using the Cauchy-Schwarz inequality and the definition of the potential function $ \\mathcal{L}_k $, for $k\\geq 0$, we see that \n    \t\\begin{align*}\n    \t\t\\mathcal{L}_{k+1} \\leq &\\;  \\mathcal{L}_k  + \\frac{k+1}{L}\\inprod{G(z^k)}{G(z^k) - \\tilde{z}^k} + \\frac{(k+1)(k+2)}{2L}\\norm{G(z^k) - \\tilde{z}^k}^2 \\\\\n    \t\t\\leq &\\; \\mathcal{L}_k  + \\frac{k+1}{L}\\inprod{G(z^k)}{G(z^k) - \\tilde{z}^k} + \\frac{(k+1)^2}{L}\\norm{G(z^k) - \\tilde{z}^k}^2 \\\\\n    \t\t\\leq &\\; \\mathcal{L}_k  + \\frac{k+1}{2L}\\left(\\frac{1}{6(k+1)}\\norm{G(z^k)}^2 + 6(k+1) \\norm{G(z^k) - \\tilde{z}^k}^2 \\right) \\\\\n            &\\; + \\frac{(k+1)^2}{L}\\norm{G(z^k) - \\tilde{z}^k}^2 \\\\\n    \t\t= &\\; \\mathcal{L}_k  + \\frac{1}{12L}\\norm{G(z^k)}^2 + \\frac{4(k+1)^2}{L}\\norm{G(z^k) - \\tilde{z}^k}^2,\n    \t\\end{align*}\n    which proves the inequality \\eqref{eq-lemma-key-estimate-2}. Therefore, the proof is completed.\n\\end{proof}","sha256_utf8":"8b008c0655193148155a52442d0a0e3da8dbdbd14fd4daac39a67701227ca558"},{"start":87377,"end":87620,"start_line":890,"end_line":896,"raw_tex":"\\begin{lemma}\n\t\\label{lemma-exp-error}\n\tLet $\\{z^k\\}$ be the sequence generated by \\eqref{eq-isHalpern}, then it holds that\n\t\\[\n\t\t\\bb{E} \\left[\\norm{G(z^k) - \\tilde{z}^k}^2\\right] \\leq \\gamma_k + \\sigma_k,\\quad \\forall k\\geq 0.\n\t\\]\n\\end{lemma}","sha256_utf8":"c8f8d8d4dfede2ae1b83f5d882bb666db4f8bbba54cd15fc10a112f71809bdbc"},{"start":87624,"end":88793,"start_line":898,"end_line":914,"raw_tex":"\\begin{proof}\n\tSince the resolvent operator $J_{\\alpha E}$ is firmly non-expansive (hence $1$-Lipschitz continuous), we see that \n\t\t\\begin{align*}\n\t\t\t&\\; \\norm{G(z^k) - \\tilde{z}^k}^2 \\\\\n\t\t\t= &\\; \\norm{\\frac{1}{\\alpha}\\left(z^k - J_{\\alpha E}\\left(z^k -  \\alpha F(z^k)\\right)\\right)- \\tilde{z}^k}^2 \\\\\n\t\t\t= &\\; \\frac{1}{\\alpha^2}\\norm{J_{\\alpha E}\\left(z^k -  \\alpha F(z^k)\\right) - \\bar{z}^k}^2 \\\\\n\t\t\t\\leq &\\; \\frac{2}{\\alpha^2}\\norm{J_{\\alpha E}\\left(z^k -  \\alpha F(z^k)\\right) - J_{\\alpha E}\\left(z^k -  \\alpha \\widetilde{F}(z^k)\\right)}^2 + \\frac{2}{\\alpha^2}\\norm{J_{\\alpha E}\\left(z^k - \\alpha \\widetilde{F}(z^k)\\right) - \\bar{z}^k}^2 \\\\\n\t\t\t\\leq &\\; 2\\norm{F(z^k) - \\widetilde{F}(z^k)}^2 + \\gamma_k,\n\t\t\\end{align*}\n    where the first equality uses the definition of $G$ and the second equality uses the fact that $\\tilde z^k = \\frac{1}{\\alpha}(z^k - \\bar{z}^k)$, and the first inequality is due to the Cauchy-Schwarz inequality.\n\tThen, it follows that: \n\t\\[\n\t\t\\bb{E} \\left[\\norm{G(z^k) - \\tilde{z}^k}^2\\right] \\leq  2\\bb{E}\\left[\\norm{F(z^k) - \\widetilde{F}(z^k)}^2\\right] +  \\gamma_k \n\t\t\t\\leq    \\sigma_k + \\gamma_k,\n\t\\]\n\twhich completes the proof.\n\\end{proof}","sha256_utf8":"e173a24e8c7d76147bac08de221ef6ec65989d62df692ab392c823d2bd6f7724"},{"start":93353,"end":98561,"start_line":970,"end_line":1046,"raw_tex":"\\begin{proof}\n\tFrom Lemma \\ref{lemma-key-estimate} and notice that $\\mathcal{L}_0 = 0$, we can see that  \n\t\\[\n\t\t\\mathcal{L}_k \\leq \\frac{1}{12L}\\sum_{i = 0}^{k-1}\\norm{G(z^i)}^2 + \\frac{4}{L}\\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2, \\quad k\\geq 1.\n\t\\]\n\tBy the definition of the potential function $\\mathcal{L}_k$, we can see that \n\t\t\\begin{align*}\n\t\t\t\\frac{k(k+1)}{2L}\\norm{G(z^k)}^2 \\leq &\\; \\frac{1}{12L}\\sum_{i = 0}^{k-1}\\norm{G(z^i)}^2 + \\frac{4}{L}\\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2 + (k+1)\\inprod{G(z^k)}{z^0 - z^k} \\\\\n\t\t\t\\leq &\\; \\frac{1}{12L}\\sum_{i = 0}^{k-1}\\norm{G(z^i)}^2 + \\frac{4}{L}\\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2 + (k+1)\\norm{G(z^k)}\\norm{z^0 - z^*}\n\t\t\\end{align*} \n\twhere the last inequality is due to the fact that \n\t\\begin{align*}\n\t    \\inprod{G(z^k)}{z^0 - z^k} = &\\; \\inprod{G(z^k)}{z^0 - z^*} + \\inprod{G(z^k)}{z^* - z^k} \\\\\n     \\leq &\\;  \\inprod{G(z^k)}{z^0 - z^*} \\\\\n     \\leq &\\;  \\norm{G(z^k)}\\norm{z^0 - z^*},\n\t\\end{align*}\n\tand $z^*$ is any solution such that $G(z^*) = 0$.\n The first inequality holds, since $G$ is co-coercive. As a consequence, it holds that \n\t\\begin{equation}\n\t\\label{thm-rate-proof-1}\n\t\t\\norm{G(z^k)}^2 \\leq \\frac{1}{6k(k+1)}\\sum_{i = 0}^{k-1}\\norm{G(z^i)}^2 + \\frac{8}{k(k+1)}\\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2 + \\frac{2L}{k}\\norm{G(z^k)}\\norm{z^0 - z^*},  \n\t\\end{equation}\n\tfor $k\\geq 1$. For notational simplicity, let us denote \n\t\\[\n\t\ta_k:= \\norm{G(z^k)}, \\quad k\\geq 0, \\quad b_k:= \\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2,\\quad k\\geq 1,\\quad D:= L\\norm{z^0 - z^*}.\n\t\\]\n\tThen, from \\eqref{thm-rate-proof-1}, we see that, for $k\\geq 1$, \n\t\\begin{align*}\n\t\ta_k^2 \\leq &\\;  \\frac{1}{6k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{8}{k(k+1)}b_k + \\frac{2D}{k}a_k \\\\\n\t\t\\leq &\\; \\frac{1}{6k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{8}{k(k+1)}b_k + \\frac{1}{2}\\left(\\frac{4D^2}{k^2} + a_k^2 \\right),\n\t\\end{align*}\n\twhich implies that, for $k\\geq 1$,\n\t\\begin{equation}\n\t\\label{thm-rate-proof-2}\n\t\\begin{aligned}\n\t    a_k^2 \\leq &\\; \\frac{1}{3k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{16}{k(k+1)}b_k + \\frac{4D^2}{k^2} \\\\\n     \\leq &\\; \\frac{1}{3k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)}.\n\t\\end{aligned}\n\t\\end{equation}\n\tWe next claim that the following inequality holds:\n\t\\begin{equation}\n\t\t\\label{thm-rate-proof-3}\n\t\ta_k^2 \\leq \\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)},\\quad k\\geq 1.\n\t\\end{equation}\n\tWe shall prove the above claim by induction. First, from \\eqref{thm-rate-proof-2} and notice that $a_0 \\leq D$ by the $L$-Lipschitz continuity of $G$, we see that\n\t\\[\n\t\ta_1^2 \\leq \\frac{1}{6}a_0^2 + 8b_1 + 4D^2 \\leq  \\frac{1}{6}D^2 + 8b_1 + 4D^2 \\leq \\frac{1}{6}(7D+10\\sqrt{b_1})^2.\n\t\\]\n\tSo \\eqref{thm-rate-proof-3} holds for $k = 1$. Suppose now that \\eqref{thm-rate-proof-3} holds for indices $\\{1,\\dots, k-1\\}$. Again, from \\eqref{thm-rate-proof-2} and the fact that $b_k$ is non-decreasing, we deduce that \n\t\t\\begin{align*}\n\t\t\ta_k^2 \\leq &\\; \\frac{1}{3k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)} \\\\\n\t\t\t\\leq &\\; \\frac{1}{3k(k+1)}\\sum_{i = 0}^{k-1} \\frac{(7D + 10\\sqrt{b_i})^2}{(i+1)(i+2)} + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)} \\\\\n\t\t\t\\leq &\\; \\frac{(7D + 10\\sqrt{b_k})^2}{3k(k+1)}\\sum_{i = 0}^{k-1} \\frac{1}{(i+1)(i+2)} + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)} \\\\\n\t\t\t= &\\; \\frac{(7D + 10\\sqrt{b_k})^2}{3(k+1)^2} + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)} \\\\\n\t\t\t\\leq &\\; \\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)},\n\t\t\\end{align*}\n\twhich shows that \\eqref{thm-rate-proof-3} holds for index $k$. Thus, by induction, \\eqref{thm-rate-proof-3} holds for all $k\\geq 1$. Moreover, it is clear that \\eqref{thm-rate-proof-3} trivially holds for $k = 0$. This proves \\eqref{thm-rate-1}.\n\t\n\tTo prove  \\eqref{thm-rate-2}, we first observe that \n\t\\[\n\t\t\\norm{\\tilde{z}^k}^2 \\leq 2\\norm{G(z^k)}^2 + 2\\norm{G(z^k) - \\tilde{z}^k}^2 \\leq 2a_k^2 + 2\\gamma_k^2 \\leq  \\frac{2(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)} + 2\\gamma_k^2,\\quad k\\geq 0.\n\t\\]\n\tThen, by the third statement of Lemma \\ref{lemma-zdiff} and the fact that $b_k$ is non-decreasing, we see that for $k\\geq 0$, \n\t\t\\begin{align*}\n\t\t\t\\norm{z^{k+1} - z^k}^2\n\t\t\t\\leq &\\; \\frac{2(k+1)^2}{L^2(k+2)^2}\\norm{\\tilde{z}^k}^2 + \\frac{2k}{L^2(k+1)^2(k+2)^2}\\sum_{i = 0}^{k-1}(i+1)^2\\norm{\\tilde{z}^i}^2 \\\\\n\t\t\t% \\leq &\\;  \\frac{4(k+1)^2}{L^2(k+2)^2}\\left(\\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)} + \\gamma_k^2 \\right) \\\\\n   % &\\; + \\frac{4k}{L^2(k+1)^2(k+2)^2}\\sum_{i = 0}^{k-1}\\left(\\frac{(i+1)(7D + 10\\sqrt{b_i})^2}{i+2} + (i+1)^2 \\gamma_i^2 \\right) \\\\\n\t\t\t\\leq &\\;  \\frac{4(k+1)^2}{L^2(k+2)^2}\\left(\\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)} + \\gamma_k^2 \\right) \\\\\n   &\\; + \\frac{4k}{L^2(k+1)^2(k+2)^2}\\sum_{i = 0}^{k-1}\\left(\\frac{(i+1)(7D + 10\\sqrt{b_k})^2}{i+2} + (i+1)^2 \\gamma_i^2 \\right) \\\\\n\t\t\t\\leq &\\; \\frac{4(7D + 10\\sqrt{b_k})^2}{L^2}\\left(\\frac{k+1}{(k+2)^3} + \\frac{k^2}{(k+1)^2(k+2)^2}\\right) \\\\\n   &\\; + \\frac{4(k+1)^2}{L^2(k+2)^2}\\gamma_k^2 + \\frac{4k}{L^2(k+1)^2(k+2)^2}b_k \\\\\n\t\t\t\\leq &\\; \\frac{8(7D + 10\\sqrt{b_{k+1}})^2 + 8b_{k+1}}{L^2(k+1)(k+2)}\\\\\n\t\t\t\\leq &\\; \\frac{8(7D + 11\\sqrt{b_{k+1}})^2 }{L^2(k+1)(k+2)},\n\t\t\\end{align*}\n\twhich proves \\eqref{thm-rate-2}. Therefore, the proof is completed.\n\\end{proof}","sha256_utf8":"8535fa2e85d718cbd77cc8de5e70b4221826cc0296ff3fce9fc6afb790f3f5d6"},{"start":98641,"end":102443,"start_line":1050,"end_line":1112,"raw_tex":"\\begin{proof}\n\tWe recall from Lemma \\ref{lemma-key-estimate} that \n\t\\[\n\t\t\\mathcal{L}_{k+1} \\leq \\mathcal{L}_k  + \\frac{1}{12L}\\norm{G(z^k)}^2 + \\frac{4(k+1)^2}{L}\\norm{G(z^k) - \\tilde{z}^k}^2,\\quad k\\geq 0.\n\t\\]\n\tTaking the full expectation $\\bb{E}$ (with respect to all the randomness) on both side of the above inequality yields \n\t\\[\n\t\t\\bb{E}[\\mathcal{L}_{k+1}] \\leq \\bb{E}[\\mathcal{L}_k]  + \\frac{1}{12L}\\bb{E}\\left[\\norm{G(z^k)}^2\\right] + \\frac{4(k+1)^2}{L}(\\sigma_k + \\gamma_k)^2,\\quad k\\geq 0,\n\t\\]\n\twhere the third term is due to Lemma \\ref{lemma-exp-error}. Then, by recursively applying the above and summing up the resulted inequalities gives:\n\t\\begin{equation}\\label{eq-thm-ishi-1}\n\t\\bb{E}[\\mathcal{L}_k] \\leq \\frac{1}{12L}\\sum_{i = 0}^{k-1}\\bb{E}\\norm{G(z^i)}^2 + \\frac{4}{L}\\sum_{i = 1}^{k-1}(i+1)^2(\\sigma_i + \\gamma_i)^2,\\quad k\\geq 0.\t\n\t\\end{equation}\n\tHere, the case when $k= 0$ holds trivially since $\\mathcal{L}_0 = 0$. By the definition of the potential function $\\mathcal{L}_k$ \n\twe see that \n\t\\begin{equation}\n\t\\label{eq-thm-ishi-2}\n\t\t\\bb{E}[\\mathcal{L}_k] = \\frac{k(k+1)}{2L}\\bb{E}\\left[\\norm{G(z^k)}^2\\right] - (k+1)\\bb{E}\\left[\\inprod{G(z^k)}{z^0 - z^k}\\right],\\quad k\\geq 0.\t\n\t\\end{equation}\n\tMoreover, let $z^*$ be any solution such that $G(z^*) = 0$, since \n\t\\begin{align*}\n\t    \\bb{E}\\left[\\inprod{G(z^k)}{z^0 - z^k}\\right] = &\\; \\bb{E}\\left[\\inprod{G(z^k)}{z^0 - z^*}\\right] + \\bb{E}\\left[\\inprod{G(z^k)}{z^* - z^k}\\right] \\\\\n        \\leq &\\;  \\bb{E}\\left[\\inprod{G(z^k)}{z^0 - z^*}\\right],\n\t\\end{align*}\n\tit holds that \n\t\\begin{equation}\n\t\\label{eq-thm-ishi-3}\n\t\t\\bb{E}\\left[\\inprod{G(z^k)}{z^0 - z^k}\\right] \\leq \\bb{E}\\left[\\norm{G(z^k)}\\norm{z^0 - z^*}\\right] = \\norm{z^0 - z^*}\\bb{E}\\left[\\norm{G(z^k)}\\right].\t\n\t\\end{equation}\n\tCombining \\eqref{eq-thm-ishi-1}, \\eqref{eq-thm-ishi-2} and \\eqref{eq-thm-ishi-3}, we get\n\t\\begin{equation*}\n\t% \\label{eq-thm-ishi-4}\n\t\t\\begin{aligned}\n\t\t\t&\\; \\bb{E}\\left[\\norm{G(z^k)}^2 \\right] \\\\\n\t\t\t\\leq &\\; \\frac{1}{6k(k+1)}\\sum_{i = 0}^{k-1}\\bb{E}\\left[\\norm{G(z^i)}^2\\right] + \\frac{2L}{k}\\norm{z^0 - z^*}\\bb{E}\\left[\\norm{G(z^k)}\\right] + \\frac{8}{k(k+1)}\\sum_{i = 0}^{k-1}(i+1)^2(\\sigma_i + \\gamma_i)^2 \\\\\n\t\t\t\\leq &\\; \\frac{1}{6k(k+1)}\\sum_{i = 0}^{k-1}\\bb{E}\\left[\\norm{G(z^i)}^2\\right] + \\frac{2L}{k}\\norm{z^0 - z^*}\\left(\\bb{E}\\left[\\norm{G(z^k)}^2\\right]\\right)^{\\frac{1}{2}} \\\\\n   &\\; + \\frac{8}{k(k+1)}\\sum_{i = 0}^{k-1}(i+1)^2(\\sigma_i + \\gamma_i)^2,\n\t\t\\end{aligned}\n\t\\end{equation*}\n\tfor all $ k\\geq 1 $. Now denote \n\t\\[\n\t\ta_k:= \\left(\\bb{E}\\left[\\norm{G(z^k)}^2\\right]\\right)^{\\frac{1}{2}}, \\quad k\\geq 0, \\quad b_k:= \\sum_{i = 0}^{k-1}(i+1)^2(\\sigma_i + \\gamma_i)^2,\\quad k\\geq 1,\\quad D:= L\\norm{z^0 - z^*}.\n\t\\]\n\tSimilar to \\eqref{thm-rate-proof-2}, we can derive \n\t\\[\n\t\ta_k^2 \\leq \\frac{1}{3k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)}.\n\t\\]\n\tThen, by following the proof of Theorem \\ref{thm-convergence-rate} directly, we see that \n\t\\[\n\t\ta_k^2\\leq \\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)}, \\quad k\\geq 1,\n\t\\]\n\twhich proves \\eqref{eq-thm-rate-ishi-G} since the case for $k = 0$ is trivial. Moreover, we can also verify that\n\t\t\\begin{align*}\n\t\t    \\bb{E}\\left[\\norm{\\tilde{z}^k}^2\\right] \\leq &\\;  2\\bb{E}\\left[\\norm{G(z^k)}^2 + \\norm{\\tilde{z}^k - G(z^k)}^2\\right] \\\\\n      \\leq &\\; 2a_k^2 + 2(\\sigma_k + \\gamma_k)^2 \\\\\n      \\leq  &\\; \\frac{2(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)} + 2(\\sigma_k + \\gamma_k)^2.\n\t\t\\end{align*}\n\tThen, by following the proof of Theorem \\ref{thm-convergence-rate} again, we can show that \n\t\\[\n\t\ta_k^2\\leq \\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)},\\quad \\bb{E}\\left[\\norm{z^{k+1} - z^k}^2\\right] \\leq \\frac{8(7D + 11\\sqrt{b_{k+1}})^2 }{L^2(k+1)(k+2)},\\quad k\\geq 0,\n\t\\]\n\twhich is exactly \\eqref{eq-thm-rate-ishi-Z}. Hence, the proof is completed.\n\\end{proof}","sha256_utf8":"30d87796067e690a8b9b41a7a5c63ca6f6eb84dca593d19a743558db68d31b89"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"随机非精确迭代的期望收敛界","source_units":[{"start":35353,"end":36089,"start_line":297,"end_line":305,"raw_tex":"\\begin{theorem}\n\t\\label{thm-rate-ishi}\n\tSuppose that $G$ is $\\frac{1}{L}$-co-coercive for a constant $L>0$. Let $\\{z^k\\}$ be the sequence generated by \\eqref{eq-isHalpern} for solving \\eqref{eq-finitesum-G}. Then it holds that \n\t\\begin{align}\n\t\t\\bb{E}\\left[\\norm{G(z^k)}^2\\right] \\leq &\\;  \\frac{\\left(7L\\norm{z^0 - z^*} + 10\\sqrt{\\sum_{i = 0}^{k-1}(i+1)^2(\\sigma_i + \\gamma_i)^2}\\right)^2}{(k+1)(k+2)},\\quad k\\geq 0, \\label{eq-thm-rate-ishi-G}\t\\\\\n\t\t\\bb{E}\\left[\\norm{z^{k+1} - z^k}^2\\right] \\leq &\\; \\frac{8\\left(7L\\norm{z^0 - z^*} + 11\\sqrt{\\sum_{i = 0}^{k}(i+1)^2(\\sigma_i + \\gamma_i)^2}\\right)^2 }{L^2(k+1)(k+2)}, \\quad k\\geq 0,\\label{eq-thm-rate-ishi-Z}\n\t\\end{align}\n\twhere $z^*$ is a solution such that $G(z^*) = 0$.\n\\end{theorem}","sha256_utf8":"14c8558d02e9ec0d9314f60af42f4fb60904c1b27c35606ff2d2efa7c3127b6a"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"期望收敛界的完整证明","source_units":[{"start":98641,"end":102443,"start_line":1050,"end_line":1112,"raw_tex":"\\begin{proof}\n\tWe recall from Lemma \\ref{lemma-key-estimate} that \n\t\\[\n\t\t\\mathcal{L}_{k+1} \\leq \\mathcal{L}_k  + \\frac{1}{12L}\\norm{G(z^k)}^2 + \\frac{4(k+1)^2}{L}\\norm{G(z^k) - \\tilde{z}^k}^2,\\quad k\\geq 0.\n\t\\]\n\tTaking the full expectation $\\bb{E}$ (with respect to all the randomness) on both side of the above inequality yields \n\t\\[\n\t\t\\bb{E}[\\mathcal{L}_{k+1}] \\leq \\bb{E}[\\mathcal{L}_k]  + \\frac{1}{12L}\\bb{E}\\left[\\norm{G(z^k)}^2\\right] + \\frac{4(k+1)^2}{L}(\\sigma_k + \\gamma_k)^2,\\quad k\\geq 0,\n\t\\]\n\twhere the third term is due to Lemma \\ref{lemma-exp-error}. Then, by recursively applying the above and summing up the resulted inequalities gives:\n\t\\begin{equation}\\label{eq-thm-ishi-1}\n\t\\bb{E}[\\mathcal{L}_k] \\leq \\frac{1}{12L}\\sum_{i = 0}^{k-1}\\bb{E}\\norm{G(z^i)}^2 + \\frac{4}{L}\\sum_{i = 1}^{k-1}(i+1)^2(\\sigma_i + \\gamma_i)^2,\\quad k\\geq 0.\t\n\t\\end{equation}\n\tHere, the case when $k= 0$ holds trivially since $\\mathcal{L}_0 = 0$. By the definition of the potential function $\\mathcal{L}_k$ \n\twe see that \n\t\\begin{equation}\n\t\\label{eq-thm-ishi-2}\n\t\t\\bb{E}[\\mathcal{L}_k] = \\frac{k(k+1)}{2L}\\bb{E}\\left[\\norm{G(z^k)}^2\\right] - (k+1)\\bb{E}\\left[\\inprod{G(z^k)}{z^0 - z^k}\\right],\\quad k\\geq 0.\t\n\t\\end{equation}\n\tMoreover, let $z^*$ be any solution such that $G(z^*) = 0$, since \n\t\\begin{align*}\n\t    \\bb{E}\\left[\\inprod{G(z^k)}{z^0 - z^k}\\right] = &\\; \\bb{E}\\left[\\inprod{G(z^k)}{z^0 - z^*}\\right] + \\bb{E}\\left[\\inprod{G(z^k)}{z^* - z^k}\\right] \\\\\n        \\leq &\\;  \\bb{E}\\left[\\inprod{G(z^k)}{z^0 - z^*}\\right],\n\t\\end{align*}\n\tit holds that \n\t\\begin{equation}\n\t\\label{eq-thm-ishi-3}\n\t\t\\bb{E}\\left[\\inprod{G(z^k)}{z^0 - z^k}\\right] \\leq \\bb{E}\\left[\\norm{G(z^k)}\\norm{z^0 - z^*}\\right] = \\norm{z^0 - z^*}\\bb{E}\\left[\\norm{G(z^k)}\\right].\t\n\t\\end{equation}\n\tCombining \\eqref{eq-thm-ishi-1}, \\eqref{eq-thm-ishi-2} and \\eqref{eq-thm-ishi-3}, we get\n\t\\begin{equation*}\n\t% \\label{eq-thm-ishi-4}\n\t\t\\begin{aligned}\n\t\t\t&\\; \\bb{E}\\left[\\norm{G(z^k)}^2 \\right] \\\\\n\t\t\t\\leq &\\; \\frac{1}{6k(k+1)}\\sum_{i = 0}^{k-1}\\bb{E}\\left[\\norm{G(z^i)}^2\\right] + \\frac{2L}{k}\\norm{z^0 - z^*}\\bb{E}\\left[\\norm{G(z^k)}\\right] + \\frac{8}{k(k+1)}\\sum_{i = 0}^{k-1}(i+1)^2(\\sigma_i + \\gamma_i)^2 \\\\\n\t\t\t\\leq &\\; \\frac{1}{6k(k+1)}\\sum_{i = 0}^{k-1}\\bb{E}\\left[\\norm{G(z^i)}^2\\right] + \\frac{2L}{k}\\norm{z^0 - z^*}\\left(\\bb{E}\\left[\\norm{G(z^k)}^2\\right]\\right)^{\\frac{1}{2}} \\\\\n   &\\; + \\frac{8}{k(k+1)}\\sum_{i = 0}^{k-1}(i+1)^2(\\sigma_i + \\gamma_i)^2,\n\t\t\\end{aligned}\n\t\\end{equation*}\n\tfor all $ k\\geq 1 $. Now denote \n\t\\[\n\t\ta_k:= \\left(\\bb{E}\\left[\\norm{G(z^k)}^2\\right]\\right)^{\\frac{1}{2}}, \\quad k\\geq 0, \\quad b_k:= \\sum_{i = 0}^{k-1}(i+1)^2(\\sigma_i + \\gamma_i)^2,\\quad k\\geq 1,\\quad D:= L\\norm{z^0 - z^*}.\n\t\\]\n\tSimilar to \\eqref{thm-rate-proof-2}, we can derive \n\t\\[\n\t\ta_k^2 \\leq \\frac{1}{3k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)}.\n\t\\]\n\tThen, by following the proof of Theorem \\ref{thm-convergence-rate} directly, we see that \n\t\\[\n\t\ta_k^2\\leq \\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)}, \\quad k\\geq 1,\n\t\\]\n\twhich proves \\eqref{eq-thm-rate-ishi-G} since the case for $k = 0$ is trivial. Moreover, we can also verify that\n\t\t\\begin{align*}\n\t\t    \\bb{E}\\left[\\norm{\\tilde{z}^k}^2\\right] \\leq &\\;  2\\bb{E}\\left[\\norm{G(z^k)}^2 + \\norm{\\tilde{z}^k - G(z^k)}^2\\right] \\\\\n      \\leq &\\; 2a_k^2 + 2(\\sigma_k + \\gamma_k)^2 \\\\\n      \\leq  &\\; \\frac{2(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)} + 2(\\sigma_k + \\gamma_k)^2.\n\t\t\\end{align*}\n\tThen, by following the proof of Theorem \\ref{thm-convergence-rate} again, we can show that \n\t\\[\n\t\ta_k^2\\leq \\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)},\\quad \\bb{E}\\left[\\norm{z^{k+1} - z^k}^2\\right] \\leq \\frac{8(7D + 11\\sqrt{b_{k+1}})^2 }{L^2(k+1)(k+2)},\\quad k\\geq 0,\n\t\\]\n\twhich is exactly \\eqref{eq-thm-rate-ishi-Z}. Hence, the proof is completed.\n\\end{proof}","sha256_utf8":"30d87796067e690a8b9b41a7a5c63ca6f6eb84dca593d19a743558db68d31b89"}]},{"id":"S003","semantic_type":"notation","kind":"notation","title":"欧氏空间与范数记号","source_units":[{"start":20370,"end":21539,"start_line":160,"end_line":160,"raw_tex":"We use $\\bb{X}$,  $\\bb{Y}$ and $\\bb{Z}$ to denote finite dimensional Euclidean spaces equipped with the standard inner product $\\inprod{\\cdot}{\\cdot}$ and the induced Euclidean norm $\\norm{\\cdot}$. Moreover, we use $\\|\\cdot\\|_\\infty$ to denote the infinity norm. Let $n$ be a positive integer, the standard basis for $\\bb{R}^n$ is denoted as $\\{e_1,\\dots, e_n\\}$ and the vector of all ones in $\\bb{R}^n$ is denoted as $\\mathds{1}_n$. We use $\\overline{\\bb{R}}$ to represent the extended reals $\\bb{R}\\cup\\{\\pm\\infty\\}$. Let $f:\\bb{X}\\to\\bb{R}$ be a differentiable function, its gradient at a point $x\\in \\bb{X}$ is denoted as $\\nabla f(x)$. If $\\bb{X} = \\bb{R}$, i.e., $f$ is a univariate function, then its derivative is denoted as $f'$. If $f:\\bb{X}\\times\\bb{Y}\\to\\bb{R}$ is a differentiable {function} of two vector variables, we use $\\nabla_xf(x,y)$ and $\\nabla_yf(x,y)$ to denote the partial derivatives with respect to $x\\in \\bb{X}$ and $y\\in \\bb{Y}$, respectively. In this way, the gradient of $f$ at the point $(x,y)$ is denoted as $\\nabla f(x,y) = [\\nabla_xf(x,y);\\nabla_yf(x,y)]$, where $[\\cdot;\\dots;\\cdot]$ is the column vector stacked from its components.\n","sha256_utf8":"9109e1a412dce0545df498e88c07cbe0f01b0937c3c5f91cfafa6b95ae140a10"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"集合值映射及其逆","source_units":[{"start":22757,"end":23217,"start_line":164,"end_line":164,"raw_tex":"\\textbf{Set-valued mappings.} Let $G:\\bb{X}\\rightrightarrows 2^\\bb{X}$ be a set-valued mapping, we use $\\dom{G}:=\\left\\{x\\in \\bb{X}:G(x)\\neq \\emptyset\\right\\}$ to denote its domain and use $\\mathrm{graph}(G):=\\left\\{(x,x')\\in \\bb{X}\\times \\bb{X}: x'\\in G(x)\\right\\}$ to denote its graph. Then the inverse of $G$ is denoted as $G^{-1}(x'):=\\left\\{x\\in\\bb{X}: x'\\in G(x)\\right\\}$. We say that $G$ is single-valued if $G(x)$ is a singleton for any $x\\in \\bb{X}$.\n","sha256_utf8":"087e408e9d02157ddaf09eff8eb697a2d08fe21a3646da3cfcb984cb97e60af8"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"预解算子及近端映射","source_units":[{"start":24036,"end":24755,"start_line":176,"end_line":176,"raw_tex":"\\textbf{Resolvent operators.} For a monotone operator $G:\\bb{X}\\rightrightarrows 2^\\bb{X}$, we denote its resolvent operator as $J_G:=(I+G)^{-1}$, where $I$ denotes the identity mapping. It is easy to verify that when $G$ is monotone, $J_G$ is single-valued with $\\dom{G} = \\bb{X}$. Note also that for a closed, proper and convex function $f:\\bb{X}\\to \\overline{\\bb{R}}$ and any positive scalar $\\alpha\\in \\bb{R}$, it holds that $J_{\\alpha \\partial f}(x) = \\prox{\\alpha f}{x}$. In particular, for any closed convex set $\\cal{C}\\subseteq \\bb{X}$ and any positive scalar $\\alpha\\in \\bb{R}$, it holds that $J_{\\alpha\\partial\\cal{I}_{\\cal{C}}}(\\cdot) = \\proj{\\cal{C}}{\\cdot}$, i.e., the projection operator onto $\\cal{C}$.\n","sha256_utf8":"facc499081512a8b72d71a172dce192c281e36a19a6a40676e3cf2157c756930"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"映射的利普希茨连续性","source_units":[{"start":25267,"end":25546,"start_line":186,"end_line":190,"raw_tex":"Let $G:\\bb{X}\\rightrightarrows 2^\\bb{X}$ be a single-valued mapping, if there exists $L\\geq 0$ such that\n\\[\n    \\left\\lVert G(x_1) - G(x_2)\\right\\rVert \\leq L\\left\\lVert x_1 - x_2\\right\\rVert,\\quad \\forall x_1,x_2\\in \\dom{G},\n\\]\nthen $G$ is said to be $L$-Lipschitz continuous. \n","sha256_utf8":"9bd7d590399117d315c6538c229bdf64d3811d192ab66f8f624c53c74c386cc3"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"余强制性","source_units":[{"start":25547,"end":25872,"start_line":192,"end_line":195,"raw_tex":"\\textbf{Co-coerciveness.} Let $L > 0$. A set-valued mapping $G:\\bb{X}\\rightrightarrows 2^\\bb{X}$ is said to be $\\frac{1}{L}$-co-coercive if the following relation holds:\n\\[\n    \\left\\langle G(x_1) - G(x_2), x_1 - x_2\\right\\rangle \\geq \\frac{1}{L}\\left\\lVert G(x_1) - G(x_2)\\right\\rVert^2,\\quad \\forall x_1,x_2\\in \\dom{G}.\n\\]\n","sha256_utf8":"6f80e1201c10e42b49c9d17c28f063d2e038b9d9548f51fde561cf8807ed59f6"}]},{"id":"S008","semantic_type":"external_fact","kind":"external_fact","title":"预解算子的坚固非扩张性","source_units":[],"declared_external":true,"declaration_text":["For maximally monotone E and α>0, J_{αE}=(I+αE)^{-1} is everywhere defined and firmly nonexpansive; in particular, ||J_{αE}(u)-J_{αE}(v)||≤||u-v||."]},{"id":"S009","semantic_type":"algorithm","kind":"algorithm","title":"确定性非精确哈尔彭迭代","source_units":[{"start":10348,"end":10581,"start_line":118,"end_line":122,"raw_tex":"\\begin{equation}\n    \\label{eq-inexact-halpern}\n    \\textrm{Find $\\tilde{z}^k$ such that } \\norm{G(z^k) - \\tilde{z}^k}\\leq \\gamma_k,\\; z^{k+1} = \\beta_k z^0 + (1-\\beta_k)z^k - \\eta_k \\tilde{z}^k,\n    \\tag{\\textrm{iHI}}\n\\end{equation}","sha256_utf8":"b9f923c05f65f24ae4f24a56e4d5e24b56a965b000ae9c530c417a928123b6e6"}]},{"id":"S010","semantic_type":"assumption","kind":"assumption","title":"包含问题存在解","source_units":[{"start":27626,"end":28137,"start_line":223,"end_line":223,"raw_tex":"As mentioned in the introduction, in the ideal case when $G(z)$ can be evaluated exactly for any $z\\in \\bb{Z}$, the exact Halpern iterative scheme \\eqref{eq-halpern} is guaranteed to possess an $O(k^{-1})$ rate. In this section, we illustrate that when the inexactness tolerance $\\gamma_k$ satisfies some mild conditions, the inexact version \\eqref{eq-inexact-halpern} shares a similar convergence rate. We assume for the rest of this paper that the problem \\eqref{eq-inclusion-G} admits at least one solution.\n","sha256_utf8":"aeb5043384b78d6b2dec86838ae50bb504246be37215bca9b1e53485cc50866d"}]},{"id":"S011","semantic_type":"problem_setting","kind":"problem_setting","title":"有限和包含问题与残差映射","source_units":[{"start":32187,"end":32740,"start_line":264,"end_line":268,"raw_tex":"In many modern applications in operation research and machine learning, $G$ usually admits a finite-sum structure. To see this, let $E:\\bb{Z}\\to 2^\\bb{Z}$ and $F_i:\\bb{Z}\\to 2^\\bb{Z}$, for $i = 1,\\dots, N$, be maximal monotone operators. In addition, $F_i$ is assumed to be $\\frac{1}{L_0}$-co-coercive for any $i = 1,\\dots, N$. Then, many real-world applications can be reformulated as the following inclusion problem: \n\\begin{equation}\n\t0\\in E(z) + F(z):= E(z) + \\frac{1}{N}\\sum_{i = 1}^N F_i(z),\\quad z\\in \\bb{Z},\n\t\\label{eq-finitesum}\t\n\\end{equation}","sha256_utf8":"fbbbcb0587c92f07d1fe5dfe188bb674762c34e0cda8fc18116f12078b68cb16"},{"start":32741,"end":33018,"start_line":269,"end_line":273,"raw_tex":"which is equivalent to the following new problem involving the resolvent operator of $E$:\n\\begin{equation}\n\t\\label{eq-finitesum-G}\n\t0\\in G(z):= \\frac{1}{\\alpha}\\left(z - J_{\\alpha E}\\left(z -  \\frac{\\alpha}{N}\\sum_{i = 1}^NF_i(z)\\right)\\right),\\quad z\\in \\bb{Z}.\n\\end{equation}","sha256_utf8":"ea406e46c147dcb0c6cd1db9ab94dc269194197d4e11a11eac3c9d16d6801b13"}]},{"id":"S012","semantic_type":"algorithm","kind":"algorithm","title":"随机估计与非精确预解更新","source_units":[{"start":33395,"end":34379,"start_line":277,"end_line":288,"raw_tex":"However, when $N$ is large, which is typically the case in practical applications, evaluating the sum $ \\frac{1}{N}\\sum_{i = 1}^NF_i(z) $ can be costly. This motivates us to consider stochastic/randomized variants of the inexact Halpern fixed-point iterative scheme \\eqref{eq-inexact-halpern}. Specifically, given $z^0\\in \\bb{Z}$, it is natural to consider performing the following steps for $k\\geq 0$:\n\\begin{equation}\n\\label{eq-isHalpern}\n\\left\\{\n\t\\begin{aligned}\n\t\t&\\; \\textrm{Find a stochastic estimator $ \\widetilde{F}(z^k) $ satisfying $ \\bb{E}\\left[\\norm{F(z^k) - \\widetilde{F}(z^k)}^2\\right] \\leq \\frac{1}{2} \\sigma_k^2 $}, \\\\ \n\t\t&\\; \t\\textrm{Find $\\bar{z}^k$ such that } \\norm{J_{\\alpha E}\\left(z^k - \\alpha \\widetilde{F}(z^k) \\right) - \\bar{z}^k}\\leq \\frac{\\sqrt{2}}{2}\\alpha \\gamma_k,\\\\\n\t\t&\\; \\tilde{z}^k = \\frac{1}{\\alpha}(z^k - \\bar{z}^k), \\quad  z^{k+1} = \\beta_k z^0 + (1-\\beta_k)z^k - \\eta_k \\tilde{z}^k, \n\t\\end{aligned}\n\t\\right.\n    \\tag{\\textrm{isHI}}\n\\end{equation}","sha256_utf8":"5a018b6edab8c1d656e28ffb9471910f33abf027e0fb0d614a27ed3cc6496830"},{"start":34380,"end":34684,"start_line":289,"end_line":289,"raw_tex":"where $ \\alpha\\in (0,4/L_0)$, $\\beta_k = 1/(k+2)$, $\\eta_k  = (1-\\beta_k)/L$ with $L:= 4/(\\alpha(4-\\alpha L_0))$, $\\gamma_k \\geq 0$ and $\\sigma_k\\geq 0$ are chosen appropriately. Here, we use $\\bb{E}$ to denote the expectation with respect to all the randomness at any iteration of \\eqref{eq-isHalpern}.\n","sha256_utf8":"740ed0c376ebebbcf6b2613fdb412803da41af07081784f56dd35ac5761addea"}]},{"id":"S013","semantic_type":"assumption","kind":"assumption","title":"主定理的余强制性与解条件","source_units":[{"start":35392,"end":35581,"start_line":299,"end_line":299,"raw_tex":"\tSuppose that $G$ is $\\frac{1}{L}$-co-coercive for a constant $L>0$. Let $\\{z^k\\}$ be the sequence generated by \\eqref{eq-isHalpern} for solving \\eqref{eq-finitesum-G}. Then it holds that \n","sha256_utf8":"3cff315f3a75253c855c4223d6ff11accb7c1fd4b5cd95b978d60496091a6b4e"},{"start":36025,"end":36076,"start_line":304,"end_line":304,"raw_tex":"\twhere $z^*$ is a solution such that $G(z^*) = 0$.\n","sha256_utf8":"0d06a64887f75b11b27a222a03b18bda8a86ff94aaad82841fe5363f7bd7fa94"}]},{"id":"S014","semantic_type":"lemma","kind":"lemma","title":"迭代差分恒等式及平方界","source_units":[{"start":77704,"end":78877,"start_line":753,"end_line":773,"raw_tex":"\\begin{lemma}\n    \\label{lemma-zdiff}\n    Let $\\{z^k\\}$ be generated by \\eqref{eq-inexact-halpern} with $\\eta_k = (1-\\beta_k)/L$, then the following expressions for $z^{k+1} - z^k$ for $k\\geq 0$ hold:\n    \\begin{enumerate}\n        \\item $z^{k+1} - z^k = \\beta_k(z^0 - z^k) - \\frac{1-\\beta_k}{L}\\tilde{z}^k$;\n        \\item $z^{k+1} - z^k = \\frac{\\beta_k}{1-\\beta_k}(z^0 - z^{k+1}) - \\frac{1}{L}\\tilde{z}^k$;\n        \\item $z^{k+1} - z^k = -\\frac{1-\\beta_k}{L}\\tilde{z}^k + \\sum_{i = 0}^{k-1}\\frac{\\beta_k}{L}\\left( \\prod_{j = i}^{k-1} (1-\\beta_j)\\right) \\tilde{z}^i$, for $k\\geq 0$. Moreover, if $\\beta_k:= 1/(k+2)$, it holds that \n    \\begin{equation}\n        \\label{eq-lemma-zdiff-1}\n        \\begin{aligned}\n            &\\; \\norm{z^{k+1} - z^k}^2 \\\\\n            \\leq &\\; \\left\\{\n            \\begin{array}{ll}\n                \\frac{1}{4L^2} \\norm{\\tilde{z}^0}^2, & \\textrm{if } k = 0,\\\\[5pt]\n                \\frac{2(k+1)^2}{L^2(k+2)^2}\\norm{\\tilde{z}^k}^2 + \\frac{2k}{L^2(k+1)^2(k+2)^2}\\sum_{i = 0}^{k-1}(i+1)^2\\norm{\\tilde{z}^i}^2,    & \\textrm{if } k\\geq 1. \n            \\end{array}\n        \\right.\n        \\end{aligned}\n    \\end{equation}\n    \\end{enumerate}\n\\end{lemma}","sha256_utf8":"bd0f78652dde6f9f1a0807251a0ac9d74b0e257031c31371d56ed83806c073b9"}]},{"id":"S015","semantic_type":"proof","kind":"proof","title":"差分引理的完整证明","source_units":[{"start":78879,"end":82750,"start_line":775,"end_line":828,"raw_tex":"\\begin{proof}\n    The first statement of the lemma is obvious. For the second statement, we first note that \n    \\[\n        z^0 - z^{k+1} = (1 - \\beta_k)(z^0 - z^k) + \\frac{1-\\beta_k}{L}\\tilde{z}^k.\n    \\]\n    Hence, it holds that\n    \\[\n        z^0 - z^k = \\frac{1}{1 - \\beta_k} (z^0 - z^{k+1}) - \\frac{1}{L} \\tilde{z}^k,\n    \\]\n    which together with part 1 further implies that \n    \\[\n        z^{k+1} - z^k = \\beta_k(z^0 - z^k) - \\frac{1-\\beta_k}{L}\\tilde{z}^k = \\frac{\\beta_k}{1 - \\beta_k} (z^0 - z^{k+1}) - \\frac{1}{L} \\tilde{z}^k.\n    \\]\n    Therefore, the second statement of the lemma holds true. We next show the third statement via mathematical induction. For $k = 0$, we see from the first statement of the lemma that \n    \\[\n        z^1 - z^0 = -\\frac{{1-\\beta_0}}{L}\\tilde{z}^0,\n    \\]\n    which implies that the third statement of the lemma holds for $k = 0$. Now suppose that the third statement of the lemma holds for $k-1$, i.e., \n    \\[\n        z^{k} - z^{k-1} = -\\frac{1-\\beta_{k-1}}{L}\\tilde{z}^{k-1} + \\sum_{i = 0}^{k-2}\\frac{\\beta_{k-1}}{L}\\left( \\prod_{j = i}^{k-2} (1-\\beta_j)\\right) \\tilde{z}^i.\n    \\]\n    Then, we can see that\n        \\begin{align*}\n             &\\; z^{k+1} - z^k \\\\\n            = &\\;  - \\frac{1-\\beta_k}{L}\\tilde{z}^k + \\beta_k(z^0 - z^k) \\\\\n            = &\\; - \\frac{1-\\beta_k}{L}\\tilde{z}^k + \\beta_k\\left( \\frac{1-\\beta_{k-1}}{\\beta_{k-1}}(z^k - z^{k-1}) + \\frac{1-\\beta_{k-1}}{L\\beta_{k-1}}\\tilde{z}^{k-1}  \\right) \\\\\n             = &\\; - \\frac{1-\\beta_k}{L}\\tilde{z}^k +  \\frac{\\beta_k(1-\\beta_{k-1})}{\\beta_{k-1}}\\left( -\\frac{1-\\beta_{k-1}}{L}\\tilde{z}^{k-1} + \\sum_{i = 0}^{k-2}\\frac{\\beta_{k-1}}{L}\\left( \\prod_{j = i}^{k-2} (1-\\beta_j)\\right) \\tilde{z}^i\\right) \\\\\n             &\\; + \\frac{\\beta_k(1-\\beta_{k-1})}{L\\beta_{k-1}}\\tilde{z}^{k-1}  \\\\\n             =&\\; - \\frac{1-\\beta_k}{L}\\tilde{z}^k + \\sum_{i = 0}^{k-2}\\frac{\\beta_k(1-\\beta_{k-1})}{L}\\left( \\prod_{j = i}^{k-2} (1-\\beta_j)\\right) \\tilde{z}^i + \\frac{\\beta_{k}(1 - \\beta_{k-1})}{L}\\tilde{z}^{k-1} \\\\\n             =&\\; -\\frac{1-\\beta_k}{L}\\tilde{z}^k + \\sum_{i = 0}^{k-1}\\frac{\\beta_k}{L}\\left( \\prod_{j = i}^{k-1} (1-\\beta_j)\\right) \\tilde{z}^i.\n        \\end{align*}\n    Here, we use the first statement of the lemma in the first equality and use the second statement of the lemma in the second equality. Moreover, the third equality is due to the induction assumption and the remaining inequalities are derived from some direct simplifications. Therefore, the statement is true for $k$. By induction, we see that it is true for all $k\\geq 0$.\n    \n    Finally, if $\\beta_k = 1/(k+2)$, we have that $\\beta_0 = 1/2$ and \n    \\[\n        z^1 - z^0 = \\frac{1}{2L}\\tilde{z}^0,\n    \\]\n    which implies that \n    \\[\n        \\norm{z^1 - z^0}^2 = {\\frac{1}{4L^2}}\\norm{\\tilde{z}^0}^2.\n    \\]\n    For $k\\geq 1$, substituting $\\beta_k = 1/(k+2)$ into the  expression of $z^{k+1} - z^k$ in the third statement yields that \n    \\begin{align*}\n        z^{k+1} - z^k = &\\; -\\frac{k+1}{L(k+2)}\\tilde{z}^k + \\sum_{i = 0}^{k-1}\\frac{1}{L(k+2)}\\left( \\prod_{j = i}^{k-1} \\frac{j+1}{j+2}\\right) \\tilde{z}^i \\\\\n        =&\\;  -\\frac{k+1}{L(k+2)}\\tilde{z}^k + \\sum_{i = 0}^{k-1}\\frac{i+1}{L(k+1)(k+2)}\\tilde{z}^i.\n    \\end{align*}\n    Therefore, by applying the Cauchy-Schwarz inequality twice, we see that \n        \\begin{align*}\n            \\norm{z^{k+1} - z^k}^2 = &\\; \\norm{-\\frac{k+1}{L(k+2)}\\tilde{z}^k + \\sum_{i = 0}^{k-1}\\frac{i+1}{L(k+1)(k+2)}\\tilde{z}^i}^2 \\\\\n            \\leq &\\; \\frac{2(k+1)^2}{L^2(k+2)^2}\\norm{\\tilde{z}^k}^2 + \\frac{2}{L^2(k+1)^2(k+2)^2}\\norm{\\sum_{i = 0}^{k-1}(i+1)\\tilde{z}^i}^2 \\\\\n            \\leq &\\; \\frac{2(k+1)^2}{L^2(k+2)^2}\\norm{\\tilde{z}^k}^2 + \\frac{2k}{L^2(k+1)^2(k+2)^2}\\sum_{i = 0}^{k-1}(i+1)^2\\norm{\\tilde{z}^i}^2,\n        \\end{align*}    \n    which proves \\eqref{eq-lemma-zdiff-1}. Hence, the proof is completed.\n\\end{proof}","sha256_utf8":"ce1e93f34e137711d70e37200054da35e263ed34e940833a2cab44c707947538"}]},{"id":"S016","semantic_type":"lemma","kind":"lemma","title":"势函数的关键递推估计","source_units":[{"start":82951,"end":84049,"start_line":833,"end_line":852,"raw_tex":"\\begin{lemma}\n    \\label{lemma-key-estimate}\n    Let $\\{z^k\\}$ be generated by \\eqref{eq-inexact-halpern} with $\\eta_k = (1-\\beta_k)/L$, then for $k\\geq 0$, it holds that \n    \\begin{equation}\n        \\label{eq-lemma-key-estimate-1}\n        \\begin{aligned}\n            &\\; \\frac{1}{2L}\\norm{G(z^{k+1})}^2 - \\frac{\\beta_k}{1-\\beta_k}\\inprod{G(z^{k+1})}{z^0 - z^{k+1}} \\\\ \\leq &\\;   \\frac{1 - 2\\beta_k}{2L}\\norm{G(z^k)}^2 - \\beta_k\\inprod{G(z^k)}{z^0 - z^k} + \\frac{\\beta_k}{L}\\inprod{G(z^k)}{G(z^k) - \\tilde{z}^k}\\\\\n            &\\; + \\frac{1}{2L}\\norm{G(z^k) - \\tilde{z}^k}^2.\n        \\end{aligned}\n    \\end{equation}\n    In particular, if $\\beta_k:=1/(k+2)$, then for $k\\geq 0$,\n    \\begin{equation}\n        \\label{eq-lemma-key-estimate-2}\n        \\mathcal{L}_{k+1} \\leq \\mathcal{L}_k  + \\frac{1}{12L}\\norm{G(z^k)}^2 + \\frac{4(k+1)^2}{L}\\norm{G(z^k) - \\tilde{z}^k}^2,\n    \\end{equation}\n    where $\\mathcal{L}_k$ is the potential function that is defined as \n    \\[\n        \\mathcal{L}_k:= \\frac{k(k+1)}{2L}\\norm{G(z^k)}^2 - (k+1)\\inprod{G(z^k)}{z^0 - z^k},\\quad \\forall k\\geq 0.\n    \\]\n\\end{lemma}","sha256_utf8":"e27dc8b1e001355f818e0606f3e87c41ce6208f728be6b5193fadf968ae75a0f"}]},{"id":"S017","semantic_type":"proof","kind":"proof","title":"关键递推估计的完整证明","source_units":[{"start":84051,"end":87085,"start_line":854,"end_line":887,"raw_tex":"\\begin{proof}\n    By the co-coerciveness of the mapping $G$, we see that \n    \\[\n        \\frac{1}{L}\\norm{G(z^{k+1}) - G(z^k)}^2 \\leq \\inprod{G(z^{k+1}) - G(z^k)}{z^{k+1} - z^k},\n    \\]  \n    which implies that \n    \\begin{equation}\n        \\label{eq-lemma-key-estimate-3}\n        \\begin{aligned}\n            &\\; \\frac{1}{2L}\\norm{G(z^{k+1})}^2 + \\frac{1}{2L}\\norm{G(z^{k})}^2 +  \\frac{1}{2L}\\norm{G(z^{k+1}) - G(z^k)}^2 \\\\\n            \\leq &\\;  \\inprod{G(z^{k+1})}{z^{k+1} - z^k + \\frac{1}{L}G(z^k)} - \\inprod{G(z^k)}{z^{k+1} - z^k}.\n        \\end{aligned}\n    \\end{equation}\n    The first two expressions for $z^{k+1} - z^k$ in Lemma \\ref{lemma-zdiff} together with \\eqref{eq-lemma-key-estimate-3} and the Cauchy-Schwarz inequality imply that \n        \\begin{align*}\n            &\\; \\frac{1}{2L}\\norm{G(z^{k+1})}^2 + \\frac{1}{2L}\\norm{G(z^{k})}^2 +  \\frac{1}{2L}\\norm{G(z^{k+1}) - G(z^k)}^2 \\\\\n            \\leq &\\; \\inprod{G(z^{k+1})}{\\frac{\\beta_k}{1-\\beta_k}(z^0 - z^{k+1}) - \\frac{1}{L}\\tilde{z}^k + \\frac{1}{L}G(z^k)} - \\inprod{G(z^k)}{\\beta_k(z^0 - z^k) - \\frac{1-\\beta_k}{L}\\tilde{z}^k} \\\\\n            = &\\; \\frac{\\beta_k}{1-\\beta_k}\\inprod{G(z^{k+1})}{z^0 - z^{k+1}} - \\beta_k\\inprod{G(z^k)}{z^0 - z^k} + \\frac{1}{L}\\inprod{G(z^{k+1})}{G(z^k) - \\tilde{z}^k} \\\\\n            &\\; + \\frac{1-\\beta_k}{L}\\inprod{G(z^k)}{\\tilde{z}^k}\\\\\n            = &\\; \\frac{\\beta_k}{1-\\beta_k}\\inprod{G(z^{k+1})}{z^0 - z^{k+1}} - \\beta_k\\inprod{G(z^k)}{z^0 - z^k}  + \\frac{1}{L}\\norm{G(z^k)}^2 \\\\\n            &\\; + \\frac{1}{L}\\inprod{G(z^{k+1}) - G(z^k)}{G(z^k) - \\tilde{z}^k} - \\frac{\\beta_k}{L}\\inprod{G(z^k)}{\\tilde{z}^k} \\\\\n            \\leq &\\; \\frac{\\beta_k}{1-\\beta_k}\\inprod{G(z^{k+1})}{z^0 - z^{k+1}} - \\beta_k\\inprod{G(z^k)}{z^0 - z^k}  + \\frac{1 - \\beta_k}{L}\\norm{G(z^k)}^2 \\\\\n            &\\; + \\frac{1}{2L}\\norm{G(z^{k+1}) - G(z^k)}^2 + \\frac{1}{2L}\\norm{G(z^k) - \\tilde{z}^k}^2 + \\frac{\\beta_k}{L}\\inprod{G(z^k)}{G(z^k) - \\tilde{z}^k}.\n        \\end{align*}\n    Therefore, the inequality \\eqref{eq-lemma-key-estimate-1} holds by rearranging terms in the above inequality. By substituting $ \\beta_k:= 1 / (k+2) $ into \\eqref{eq-lemma-key-estimate-1}, and using the Cauchy-Schwarz inequality and the definition of the potential function $ \\mathcal{L}_k $, for $k\\geq 0$, we see that \n    \t\\begin{align*}\n    \t\t\\mathcal{L}_{k+1} \\leq &\\;  \\mathcal{L}_k  + \\frac{k+1}{L}\\inprod{G(z^k)}{G(z^k) - \\tilde{z}^k} + \\frac{(k+1)(k+2)}{2L}\\norm{G(z^k) - \\tilde{z}^k}^2 \\\\\n    \t\t\\leq &\\; \\mathcal{L}_k  + \\frac{k+1}{L}\\inprod{G(z^k)}{G(z^k) - \\tilde{z}^k} + \\frac{(k+1)^2}{L}\\norm{G(z^k) - \\tilde{z}^k}^2 \\\\\n    \t\t\\leq &\\; \\mathcal{L}_k  + \\frac{k+1}{2L}\\left(\\frac{1}{6(k+1)}\\norm{G(z^k)}^2 + 6(k+1) \\norm{G(z^k) - \\tilde{z}^k}^2 \\right) \\\\\n            &\\; + \\frac{(k+1)^2}{L}\\norm{G(z^k) - \\tilde{z}^k}^2 \\\\\n    \t\t= &\\; \\mathcal{L}_k  + \\frac{1}{12L}\\norm{G(z^k)}^2 + \\frac{4(k+1)^2}{L}\\norm{G(z^k) - \\tilde{z}^k}^2,\n    \t\\end{align*}\n    which proves the inequality \\eqref{eq-lemma-key-estimate-2}. Therefore, the proof is completed.\n\\end{proof}","sha256_utf8":"8b008c0655193148155a52442d0a0e3da8dbdbd14fd4daac39a67701227ca558"}]},{"id":"S018","semantic_type":"lemma","kind":"lemma","title":"随机残差近似的误差界","source_units":[{"start":87377,"end":87620,"start_line":890,"end_line":896,"raw_tex":"\\begin{lemma}\n\t\\label{lemma-exp-error}\n\tLet $\\{z^k\\}$ be the sequence generated by \\eqref{eq-isHalpern}, then it holds that\n\t\\[\n\t\t\\bb{E} \\left[\\norm{G(z^k) - \\tilde{z}^k}^2\\right] \\leq \\gamma_k + \\sigma_k,\\quad \\forall k\\geq 0.\n\t\\]\n\\end{lemma}","sha256_utf8":"c8f8d8d4dfede2ae1b83f5d882bb666db4f8bbba54cd15fc10a112f71809bdbc"}]},{"id":"S019","semantic_type":"proof","kind":"proof","title":"随机误差引理的完整证明","source_units":[{"start":87624,"end":88793,"start_line":898,"end_line":914,"raw_tex":"\\begin{proof}\n\tSince the resolvent operator $J_{\\alpha E}$ is firmly non-expansive (hence $1$-Lipschitz continuous), we see that \n\t\t\\begin{align*}\n\t\t\t&\\; \\norm{G(z^k) - \\tilde{z}^k}^2 \\\\\n\t\t\t= &\\; \\norm{\\frac{1}{\\alpha}\\left(z^k - J_{\\alpha E}\\left(z^k -  \\alpha F(z^k)\\right)\\right)- \\tilde{z}^k}^2 \\\\\n\t\t\t= &\\; \\frac{1}{\\alpha^2}\\norm{J_{\\alpha E}\\left(z^k -  \\alpha F(z^k)\\right) - \\bar{z}^k}^2 \\\\\n\t\t\t\\leq &\\; \\frac{2}{\\alpha^2}\\norm{J_{\\alpha E}\\left(z^k -  \\alpha F(z^k)\\right) - J_{\\alpha E}\\left(z^k -  \\alpha \\widetilde{F}(z^k)\\right)}^2 + \\frac{2}{\\alpha^2}\\norm{J_{\\alpha E}\\left(z^k - \\alpha \\widetilde{F}(z^k)\\right) - \\bar{z}^k}^2 \\\\\n\t\t\t\\leq &\\; 2\\norm{F(z^k) - \\widetilde{F}(z^k)}^2 + \\gamma_k,\n\t\t\\end{align*}\n    where the first equality uses the definition of $G$ and the second equality uses the fact that $\\tilde z^k = \\frac{1}{\\alpha}(z^k - \\bar{z}^k)$, and the first inequality is due to the Cauchy-Schwarz inequality.\n\tThen, it follows that: \n\t\\[\n\t\t\\bb{E} \\left[\\norm{G(z^k) - \\tilde{z}^k}^2\\right] \\leq  2\\bb{E}\\left[\\norm{F(z^k) - \\widetilde{F}(z^k)}^2\\right] +  \\gamma_k \n\t\t\t\\leq    \\sigma_k + \\gamma_k,\n\t\\]\n\twhich completes the proof.\n\\end{proof}","sha256_utf8":"e173a24e8c7d76147bac08de221ef6ec65989d62df692ab392c823d2bd6f7724"}]},{"id":"S020","semantic_type":"theorem","kind":"theorem","title":"确定性残差与步长收敛界","source_units":[{"start":28329,"end":29015,"start_line":227,"end_line":235,"raw_tex":"\\begin{theorem}\n\t\\label{thm-convergence-rate}\n\tSuppose that $G$ is $\\frac{1}{L}$-co-coercive for a constant $L>0$. Let $\\{z^k\\}$ be generated by \\eqref{eq-inexact-halpern} with $\\beta_k = 1/(k+2)$ and $\\eta_k = (1-\\beta_k)/L$, then it holds that\n\t\\begin{align}\n\t\t\\norm{G(z^k)}^2 \\leq &\\;  \\frac{\\left(7L\\norm{z^0 - z^*} + 10\\sqrt{\\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2}\\right)^2}{(k+1)(k+2)},\\quad k\\geq 0, \\label{thm-rate-1}\\\\\n\t\t\\norm{z^{k+1} - z^k}^2 \\leq &\\; \\frac{8\\left(7L\\norm{z^0 - z^*} + 11\\sqrt{\\sum_{i = 0}^{k}(i+1)^2\\gamma_i^2}\\right)^2 }{L^2(k+1)(k+2)}, \\quad k\\geq 0,\\label{thm-rate-2}\n\t\\end{align}\n\twhere $z^* \\in \\bb{Z}$ is any solution such that $G(z^*) = 0$.\n\\end{theorem}","sha256_utf8":"39bc3b28370d8f1fe03926787ae5ac5d6ca73c54c544fd4162146b24e8a7f061"}]},{"id":"S021","semantic_type":"proof","kind":"proof","title":"确定性收敛界的完整证明","source_units":[{"start":93353,"end":98561,"start_line":970,"end_line":1046,"raw_tex":"\\begin{proof}\n\tFrom Lemma \\ref{lemma-key-estimate} and notice that $\\mathcal{L}_0 = 0$, we can see that  \n\t\\[\n\t\t\\mathcal{L}_k \\leq \\frac{1}{12L}\\sum_{i = 0}^{k-1}\\norm{G(z^i)}^2 + \\frac{4}{L}\\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2, \\quad k\\geq 1.\n\t\\]\n\tBy the definition of the potential function $\\mathcal{L}_k$, we can see that \n\t\t\\begin{align*}\n\t\t\t\\frac{k(k+1)}{2L}\\norm{G(z^k)}^2 \\leq &\\; \\frac{1}{12L}\\sum_{i = 0}^{k-1}\\norm{G(z^i)}^2 + \\frac{4}{L}\\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2 + (k+1)\\inprod{G(z^k)}{z^0 - z^k} \\\\\n\t\t\t\\leq &\\; \\frac{1}{12L}\\sum_{i = 0}^{k-1}\\norm{G(z^i)}^2 + \\frac{4}{L}\\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2 + (k+1)\\norm{G(z^k)}\\norm{z^0 - z^*}\n\t\t\\end{align*} \n\twhere the last inequality is due to the fact that \n\t\\begin{align*}\n\t    \\inprod{G(z^k)}{z^0 - z^k} = &\\; \\inprod{G(z^k)}{z^0 - z^*} + \\inprod{G(z^k)}{z^* - z^k} \\\\\n     \\leq &\\;  \\inprod{G(z^k)}{z^0 - z^*} \\\\\n     \\leq &\\;  \\norm{G(z^k)}\\norm{z^0 - z^*},\n\t\\end{align*}\n\tand $z^*$ is any solution such that $G(z^*) = 0$.\n The first inequality holds, since $G$ is co-coercive. As a consequence, it holds that \n\t\\begin{equation}\n\t\\label{thm-rate-proof-1}\n\t\t\\norm{G(z^k)}^2 \\leq \\frac{1}{6k(k+1)}\\sum_{i = 0}^{k-1}\\norm{G(z^i)}^2 + \\frac{8}{k(k+1)}\\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2 + \\frac{2L}{k}\\norm{G(z^k)}\\norm{z^0 - z^*},  \n\t\\end{equation}\n\tfor $k\\geq 1$. For notational simplicity, let us denote \n\t\\[\n\t\ta_k:= \\norm{G(z^k)}, \\quad k\\geq 0, \\quad b_k:= \\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2,\\quad k\\geq 1,\\quad D:= L\\norm{z^0 - z^*}.\n\t\\]\n\tThen, from \\eqref{thm-rate-proof-1}, we see that, for $k\\geq 1$, \n\t\\begin{align*}\n\t\ta_k^2 \\leq &\\;  \\frac{1}{6k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{8}{k(k+1)}b_k + \\frac{2D}{k}a_k \\\\\n\t\t\\leq &\\; \\frac{1}{6k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{8}{k(k+1)}b_k + \\frac{1}{2}\\left(\\frac{4D^2}{k^2} + a_k^2 \\right),\n\t\\end{align*}\n\twhich implies that, for $k\\geq 1$,\n\t\\begin{equation}\n\t\\label{thm-rate-proof-2}\n\t\\begin{aligned}\n\t    a_k^2 \\leq &\\; \\frac{1}{3k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{16}{k(k+1)}b_k + \\frac{4D^2}{k^2} \\\\\n     \\leq &\\; \\frac{1}{3k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)}.\n\t\\end{aligned}\n\t\\end{equation}\n\tWe next claim that the following inequality holds:\n\t\\begin{equation}\n\t\t\\label{thm-rate-proof-3}\n\t\ta_k^2 \\leq \\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)},\\quad k\\geq 1.\n\t\\end{equation}\n\tWe shall prove the above claim by induction. First, from \\eqref{thm-rate-proof-2} and notice that $a_0 \\leq D$ by the $L$-Lipschitz continuity of $G$, we see that\n\t\\[\n\t\ta_1^2 \\leq \\frac{1}{6}a_0^2 + 8b_1 + 4D^2 \\leq  \\frac{1}{6}D^2 + 8b_1 + 4D^2 \\leq \\frac{1}{6}(7D+10\\sqrt{b_1})^2.\n\t\\]\n\tSo \\eqref{thm-rate-proof-3} holds for $k = 1$. Suppose now that \\eqref{thm-rate-proof-3} holds for indices $\\{1,\\dots, k-1\\}$. Again, from \\eqref{thm-rate-proof-2} and the fact that $b_k$ is non-decreasing, we deduce that \n\t\t\\begin{align*}\n\t\t\ta_k^2 \\leq &\\; \\frac{1}{3k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)} \\\\\n\t\t\t\\leq &\\; \\frac{1}{3k(k+1)}\\sum_{i = 0}^{k-1} \\frac{(7D + 10\\sqrt{b_i})^2}{(i+1)(i+2)} + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)} \\\\\n\t\t\t\\leq &\\; \\frac{(7D + 10\\sqrt{b_k})^2}{3k(k+1)}\\sum_{i = 0}^{k-1} \\frac{1}{(i+1)(i+2)} + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)} \\\\\n\t\t\t= &\\; \\frac{(7D + 10\\sqrt{b_k})^2}{3(k+1)^2} + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)} \\\\\n\t\t\t\\leq &\\; \\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)},\n\t\t\\end{align*}\n\twhich shows that \\eqref{thm-rate-proof-3} holds for index $k$. Thus, by induction, \\eqref{thm-rate-proof-3} holds for all $k\\geq 1$. Moreover, it is clear that \\eqref{thm-rate-proof-3} trivially holds for $k = 0$. This proves \\eqref{thm-rate-1}.\n\t\n\tTo prove  \\eqref{thm-rate-2}, we first observe that \n\t\\[\n\t\t\\norm{\\tilde{z}^k}^2 \\leq 2\\norm{G(z^k)}^2 + 2\\norm{G(z^k) - \\tilde{z}^k}^2 \\leq 2a_k^2 + 2\\gamma_k^2 \\leq  \\frac{2(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)} + 2\\gamma_k^2,\\quad k\\geq 0.\n\t\\]\n\tThen, by the third statement of Lemma \\ref{lemma-zdiff} and the fact that $b_k$ is non-decreasing, we see that for $k\\geq 0$, \n\t\t\\begin{align*}\n\t\t\t\\norm{z^{k+1} - z^k}^2\n\t\t\t\\leq &\\; \\frac{2(k+1)^2}{L^2(k+2)^2}\\norm{\\tilde{z}^k}^2 + \\frac{2k}{L^2(k+1)^2(k+2)^2}\\sum_{i = 0}^{k-1}(i+1)^2\\norm{\\tilde{z}^i}^2 \\\\\n\t\t\t% \\leq &\\;  \\frac{4(k+1)^2}{L^2(k+2)^2}\\left(\\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)} + \\gamma_k^2 \\right) \\\\\n   % &\\; + \\frac{4k}{L^2(k+1)^2(k+2)^2}\\sum_{i = 0}^{k-1}\\left(\\frac{(i+1)(7D + 10\\sqrt{b_i})^2}{i+2} + (i+1)^2 \\gamma_i^2 \\right) \\\\\n\t\t\t\\leq &\\;  \\frac{4(k+1)^2}{L^2(k+2)^2}\\left(\\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)} + \\gamma_k^2 \\right) \\\\\n   &\\; + \\frac{4k}{L^2(k+1)^2(k+2)^2}\\sum_{i = 0}^{k-1}\\left(\\frac{(i+1)(7D + 10\\sqrt{b_k})^2}{i+2} + (i+1)^2 \\gamma_i^2 \\right) \\\\\n\t\t\t\\leq &\\; \\frac{4(7D + 10\\sqrt{b_k})^2}{L^2}\\left(\\frac{k+1}{(k+2)^3} + \\frac{k^2}{(k+1)^2(k+2)^2}\\right) \\\\\n   &\\; + \\frac{4(k+1)^2}{L^2(k+2)^2}\\gamma_k^2 + \\frac{4k}{L^2(k+1)^2(k+2)^2}b_k \\\\\n\t\t\t\\leq &\\; \\frac{8(7D + 10\\sqrt{b_{k+1}})^2 + 8b_{k+1}}{L^2(k+1)(k+2)}\\\\\n\t\t\t\\leq &\\; \\frac{8(7D + 11\\sqrt{b_{k+1}})^2 }{L^2(k+1)(k+2)},\n\t\t\\end{align*}\n\twhich proves \\eqref{thm-rate-2}. Therefore, the proof is completed.\n\\end{proof}","sha256_utf8":"8535fa2e85d718cbd77cc8de5e70b4221826cc0296ff3fce9fc6afb790f3f5d6"}]},{"id":"S022","semantic_type":"external_fact","kind":"external_fact","title":"内积与期望的基本不等式","source_units":[],"declared_external":true,"declaration_text":["Cauchy–Schwarz gives |⟨u,v⟩|≤||u||||v||; Young's inequality gives 2ab≤t a²+t^{-1}b² for t>0. Expectations are linear and preserve inequalities, and E[X]≤sqrt(E[X²]) for nonnegative square-integrable X."]},{"id":"S023","semantic_type":"external_fact","kind":"external_fact","title":"余强制性的连续性推论","source_units":[],"declared_external":true,"declaration_text":["If ⟨G(x)-G(y),x-y⟩≥L^{-1}||G(x)-G(y)||² with L>0, then G is monotone and ||G(x)-G(y)||≤L||x-y||."]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"对势函数递推取全期望","source_units":[{"start":98641,"end":99294,"start_line":1050,"end_line":1059,"raw_tex":"\\begin{proof}\n\tWe recall from Lemma \\ref{lemma-key-estimate} that \n\t\\[\n\t\t\\mathcal{L}_{k+1} \\leq \\mathcal{L}_k  + \\frac{1}{12L}\\norm{G(z^k)}^2 + \\frac{4(k+1)^2}{L}\\norm{G(z^k) - \\tilde{z}^k}^2,\\quad k\\geq 0.\n\t\\]\n\tTaking the full expectation $\\bb{E}$ (with respect to all the randomness) on both side of the above inequality yields \n\t\\[\n\t\t\\bb{E}[\\mathcal{L}_{k+1}] \\leq \\bb{E}[\\mathcal{L}_k]  + \\frac{1}{12L}\\bb{E}\\left[\\norm{G(z^k)}^2\\right] + \\frac{4(k+1)^2}{L}(\\sigma_k + \\gamma_k)^2,\\quad k\\geq 0,\n\t\\]\n\twhere the third term is due to Lemma \\ref{lemma-exp-error}. Then, by recursively applying the above and summing up the resulted inequalities gives:\n","sha256_utf8":"184852238a392e90041b3fc121bea5269e25fe54d825c1f40daad366bbb195e2"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"累加递推并展开期望势函数","source_units":[{"start":99294,"end":99858,"start_line":1060,"end_line":1068,"raw_tex":"\t\\begin{equation}\\label{eq-thm-ishi-1}\n\t\\bb{E}[\\mathcal{L}_k] \\leq \\frac{1}{12L}\\sum_{i = 0}^{k-1}\\bb{E}\\norm{G(z^i)}^2 + \\frac{4}{L}\\sum_{i = 1}^{k-1}(i+1)^2(\\sigma_i + \\gamma_i)^2,\\quad k\\geq 0.\t\n\t\\end{equation}\n\tHere, the case when $k= 0$ holds trivially since $\\mathcal{L}_0 = 0$. By the definition of the potential function $\\mathcal{L}_k$ \n\twe see that \n\t\\begin{equation}\n\t\\label{eq-thm-ishi-2}\n\t\t\\bb{E}[\\mathcal{L}_k] = \\frac{k(k+1)}{2L}\\bb{E}\\left[\\norm{G(z^k)}^2\\right] - (k+1)\\bb{E}\\left[\\inprod{G(z^k)}{z^0 - z^k}\\right],\\quad k\\geq 0.\t\n\t\\end{equation}\n","sha256_utf8":"8d0552e8d5e3856d7381bb0c62f30ab40fc0b3b0d031bbf1a63cc8a956ebf01c"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"利用解点控制交叉内积","source_units":[{"start":99858,"end":100403,"start_line":1069,"end_line":1078,"raw_tex":"\tMoreover, let $z^*$ be any solution such that $G(z^*) = 0$, since \n\t\\begin{align*}\n\t    \\bb{E}\\left[\\inprod{G(z^k)}{z^0 - z^k}\\right] = &\\; \\bb{E}\\left[\\inprod{G(z^k)}{z^0 - z^*}\\right] + \\bb{E}\\left[\\inprod{G(z^k)}{z^* - z^k}\\right] \\\\\n        \\leq &\\;  \\bb{E}\\left[\\inprod{G(z^k)}{z^0 - z^*}\\right],\n\t\\end{align*}\n\tit holds that \n\t\\begin{equation}\n\t\\label{eq-thm-ishi-3}\n\t\t\\bb{E}\\left[\\inprod{G(z^k)}{z^0 - z^k}\\right] \\leq \\bb{E}\\left[\\norm{G(z^k)}\\norm{z^0 - z^*}\\right] = \\norm{z^0 - z^*}\\bb{E}\\left[\\norm{G(z^k)}\\right].\t\n\t\\end{equation}\n","sha256_utf8":"ac639ba3e2ce2223f611a24cbdad13bc63a6580633cb326046dc4b5b8816e1d8"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"建立均方残差的标量递推","source_units":[{"start":100403,"end":101333,"start_line":1079,"end_line":1092,"raw_tex":"\tCombining \\eqref{eq-thm-ishi-1}, \\eqref{eq-thm-ishi-2} and \\eqref{eq-thm-ishi-3}, we get\n\t\\begin{equation*}\n\t% \\label{eq-thm-ishi-4}\n\t\t\\begin{aligned}\n\t\t\t&\\; \\bb{E}\\left[\\norm{G(z^k)}^2 \\right] \\\\\n\t\t\t\\leq &\\; \\frac{1}{6k(k+1)}\\sum_{i = 0}^{k-1}\\bb{E}\\left[\\norm{G(z^i)}^2\\right] + \\frac{2L}{k}\\norm{z^0 - z^*}\\bb{E}\\left[\\norm{G(z^k)}\\right] + \\frac{8}{k(k+1)}\\sum_{i = 0}^{k-1}(i+1)^2(\\sigma_i + \\gamma_i)^2 \\\\\n\t\t\t\\leq &\\; \\frac{1}{6k(k+1)}\\sum_{i = 0}^{k-1}\\bb{E}\\left[\\norm{G(z^i)}^2\\right] + \\frac{2L}{k}\\norm{z^0 - z^*}\\left(\\bb{E}\\left[\\norm{G(z^k)}^2\\right]\\right)^{\\frac{1}{2}} \\\\\n   &\\; + \\frac{8}{k(k+1)}\\sum_{i = 0}^{k-1}(i+1)^2(\\sigma_i + \\gamma_i)^2,\n\t\t\\end{aligned}\n\t\\end{equation*}\n\tfor all $ k\\geq 1 $. Now denote \n\t\\[\n\t\ta_k:= \\left(\\bb{E}\\left[\\norm{G(z^k)}^2\\right]\\right)^{\\frac{1}{2}}, \\quad k\\geq 0, \\quad b_k:= \\sum_{i = 0}^{k-1}(i+1)^2(\\sigma_i + \\gamma_i)^2,\\quad k\\geq 1,\\quad D:= L\\norm{z^0 - z^*}.\n\t\\]\n","sha256_utf8":"c87a8ae3290aff29a39c2ff3a59546893198149afeb589f55b90a8b6fe548fd9"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"沿用确定性归纳得到残差界","source_units":[{"start":101333,"end":101786,"start_line":1093,"end_line":1101,"raw_tex":"\tSimilar to \\eqref{thm-rate-proof-2}, we can derive \n\t\\[\n\t\ta_k^2 \\leq \\frac{1}{3k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)}.\n\t\\]\n\tThen, by following the proof of Theorem \\ref{thm-convergence-rate} directly, we see that \n\t\\[\n\t\ta_k^2\\leq \\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)}, \\quad k\\geq 1,\n\t\\]\n\twhich proves \\eqref{eq-thm-rate-ishi-G} since the case for $k = 0$ is trivial. Moreover, we can also verify that\n","sha256_utf8":"48defe8d6c82622cb3377483403e9fb3995196ae7ef05e0102a53830b4a5f775"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"控制近似残差的二阶矩","source_units":[{"start":101786,"end":102083,"start_line":1102,"end_line":1106,"raw_tex":"\t\t\\begin{align*}\n\t\t    \\bb{E}\\left[\\norm{\\tilde{z}^k}^2\\right] \\leq &\\;  2\\bb{E}\\left[\\norm{G(z^k)}^2 + \\norm{\\tilde{z}^k - G(z^k)}^2\\right] \\\\\n      \\leq &\\; 2a_k^2 + 2(\\sigma_k + \\gamma_k)^2 \\\\\n      \\leq  &\\; \\frac{2(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)} + 2(\\sigma_k + \\gamma_k)^2.\n\t\t\\end{align*}\n","sha256_utf8":"865ed899cbccdd84cc19aab1497af11ceb3730b9b5c808f21fa1b2f763c56607"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"沿用差分估计完成期望步长界","source_units":[{"start":102083,"end":102443,"start_line":1107,"end_line":1112,"raw_tex":"\tThen, by following the proof of Theorem \\ref{thm-convergence-rate} again, we can show that \n\t\\[\n\t\ta_k^2\\leq \\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)},\\quad \\bb{E}\\left[\\norm{z^{k+1} - z^k}^2\\right] \\leq \\frac{8(7D + 11\\sqrt{b_{k+1}})^2 }{L^2(k+1)(k+2)},\\quad k\\geq 0,\n\t\\]\n\twhich is exactly \\eqref{eq-thm-rate-ishi-Z}. Hence, the proof is completed.\n\\end{proof}","sha256_utf8":"46760a9ad94bafc74d23c1f2149cfca424bff5239ca0886c6eba57796904b841"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"由更新式导出前两个差分恒等式","source_units":[{"start":78879,"end":79425,"start_line":775,"end_line":787,"raw_tex":"\\begin{proof}\n    The first statement of the lemma is obvious. For the second statement, we first note that \n    \\[\n        z^0 - z^{k+1} = (1 - \\beta_k)(z^0 - z^k) + \\frac{1-\\beta_k}{L}\\tilde{z}^k.\n    \\]\n    Hence, it holds that\n    \\[\n        z^0 - z^k = \\frac{1}{1 - \\beta_k} (z^0 - z^{k+1}) - \\frac{1}{L} \\tilde{z}^k,\n    \\]\n    which together with part 1 further implies that \n    \\[\n        z^{k+1} - z^k = \\beta_k(z^0 - z^k) - \\frac{1-\\beta_k}{L}\\tilde{z}^k = \\frac{\\beta_k}{1 - \\beta_k} (z^0 - z^{k+1}) - \\frac{1}{L} \\tilde{z}^k.\n    \\]\n","sha256_utf8":"b7ef3025b4fe4d65c9e247a61e42b00820d8c36c93878073e65cd7b35424a786"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"验证展开式基例并设归纳假设","source_units":[{"start":79425,"end":80009,"start_line":788,"end_line":795,"raw_tex":"    Therefore, the second statement of the lemma holds true. We next show the third statement via mathematical induction. For $k = 0$, we see from the first statement of the lemma that \n    \\[\n        z^1 - z^0 = -\\frac{{1-\\beta_0}}{L}\\tilde{z}^0,\n    \\]\n    which implies that the third statement of the lemma holds for $k = 0$. Now suppose that the third statement of the lemma holds for $k-1$, i.e., \n    \\[\n        z^{k} - z^{k-1} = -\\frac{1-\\beta_{k-1}}{L}\\tilde{z}^{k-1} + \\sum_{i = 0}^{k-2}\\frac{\\beta_{k-1}}{L}\\left( \\prod_{j = i}^{k-2} (1-\\beta_j)\\right) \\tilde{z}^i.\n    \\]\n","sha256_utf8":"8f41520f33df73397aa82094adb07993ed20bd61dae030f8cf06faf39b8ed372"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"归纳推导历史残差展开式","source_units":[{"start":80009,"end":81445,"start_line":796,"end_line":806,"raw_tex":"    Then, we can see that\n        \\begin{align*}\n             &\\; z^{k+1} - z^k \\\\\n            = &\\;  - \\frac{1-\\beta_k}{L}\\tilde{z}^k + \\beta_k(z^0 - z^k) \\\\\n            = &\\; - \\frac{1-\\beta_k}{L}\\tilde{z}^k + \\beta_k\\left( \\frac{1-\\beta_{k-1}}{\\beta_{k-1}}(z^k - z^{k-1}) + \\frac{1-\\beta_{k-1}}{L\\beta_{k-1}}\\tilde{z}^{k-1}  \\right) \\\\\n             = &\\; - \\frac{1-\\beta_k}{L}\\tilde{z}^k +  \\frac{\\beta_k(1-\\beta_{k-1})}{\\beta_{k-1}}\\left( -\\frac{1-\\beta_{k-1}}{L}\\tilde{z}^{k-1} + \\sum_{i = 0}^{k-2}\\frac{\\beta_{k-1}}{L}\\left( \\prod_{j = i}^{k-2} (1-\\beta_j)\\right) \\tilde{z}^i\\right) \\\\\n             &\\; + \\frac{\\beta_k(1-\\beta_{k-1})}{L\\beta_{k-1}}\\tilde{z}^{k-1}  \\\\\n             =&\\; - \\frac{1-\\beta_k}{L}\\tilde{z}^k + \\sum_{i = 0}^{k-2}\\frac{\\beta_k(1-\\beta_{k-1})}{L}\\left( \\prod_{j = i}^{k-2} (1-\\beta_j)\\right) \\tilde{z}^i + \\frac{\\beta_{k}(1 - \\beta_{k-1})}{L}\\tilde{z}^{k-1} \\\\\n             =&\\; -\\frac{1-\\beta_k}{L}\\tilde{z}^k + \\sum_{i = 0}^{k-1}\\frac{\\beta_k}{L}\\left( \\prod_{j = i}^{k-1} (1-\\beta_j)\\right) \\tilde{z}^i.\n        \\end{align*}\n    Here, we use the first statement of the lemma in the first equality and use the second statement of the lemma in the second equality. Moreover, the third equality is due to the induction assumption and the remaining inequalities are derived from some direct simplifications. Therefore, the statement is true for $k$. By induction, we see that it is true for all $k\\geq 0$.\n","sha256_utf8":"55f1e215b2e90ccb8be35b48708c14981e97b56cd45df86af77c17d6b25dd345"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"计算初始差分的平方范数","source_units":[{"start":81450,"end":81685,"start_line":808,"end_line":815,"raw_tex":"    Finally, if $\\beta_k = 1/(k+2)$, we have that $\\beta_0 = 1/2$ and \n    \\[\n        z^1 - z^0 = \\frac{1}{2L}\\tilde{z}^0,\n    \\]\n    which implies that \n    \\[\n        \\norm{z^1 - z^0}^2 = {\\frac{1}{4L^2}}\\norm{\\tilde{z}^0}^2.\n    \\]\n","sha256_utf8":"58681e6f75a901c7d5b535aaabbccba644f13a392030ea55d641c14d31606b98"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"代入参数化简历史权重","source_units":[{"start":81685,"end":82110,"start_line":816,"end_line":820,"raw_tex":"    For $k\\geq 1$, substituting $\\beta_k = 1/(k+2)$ into the  expression of $z^{k+1} - z^k$ in the third statement yields that \n    \\begin{align*}\n        z^{k+1} - z^k = &\\; -\\frac{k+1}{L(k+2)}\\tilde{z}^k + \\sum_{i = 0}^{k-1}\\frac{1}{L(k+2)}\\left( \\prod_{j = i}^{k-1} \\frac{j+1}{j+2}\\right) \\tilde{z}^i \\\\\n        =&\\;  -\\frac{k+1}{L(k+2)}\\tilde{z}^k + \\sum_{i = 0}^{k-1}\\frac{i+1}{L(k+1)(k+2)}\\tilde{z}^i.\n    \\end{align*}\n","sha256_utf8":"e501865811854f4bd9fe13ad30a687ac3a78748f03c066fb78d4860a0f099f4d"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"两次柯西不等式得到差分界","source_units":[{"start":82110,"end":82750,"start_line":821,"end_line":828,"raw_tex":"    Therefore, by applying the Cauchy-Schwarz inequality twice, we see that \n        \\begin{align*}\n            \\norm{z^{k+1} - z^k}^2 = &\\; \\norm{-\\frac{k+1}{L(k+2)}\\tilde{z}^k + \\sum_{i = 0}^{k-1}\\frac{i+1}{L(k+1)(k+2)}\\tilde{z}^i}^2 \\\\\n            \\leq &\\; \\frac{2(k+1)^2}{L^2(k+2)^2}\\norm{\\tilde{z}^k}^2 + \\frac{2}{L^2(k+1)^2(k+2)^2}\\norm{\\sum_{i = 0}^{k-1}(i+1)\\tilde{z}^i}^2 \\\\\n            \\leq &\\; \\frac{2(k+1)^2}{L^2(k+2)^2}\\norm{\\tilde{z}^k}^2 + \\frac{2k}{L^2(k+1)^2(k+2)^2}\\sum_{i = 0}^{k-1}(i+1)^2\\norm{\\tilde{z}^i}^2,\n        \\end{align*}    \n    which proves \\eqref{eq-lemma-zdiff-1}. Hence, the proof is completed.\n\\end{proof}","sha256_utf8":"7a02af75f67c4e9a8fa31fb60431bf10a201c4335aff70aee316e51d67cefc98"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"将余强制性改写为能量不等式","source_units":[{"start":84051,"end":84627,"start_line":854,"end_line":866,"raw_tex":"\\begin{proof}\n    By the co-coerciveness of the mapping $G$, we see that \n    \\[\n        \\frac{1}{L}\\norm{G(z^{k+1}) - G(z^k)}^2 \\leq \\inprod{G(z^{k+1}) - G(z^k)}{z^{k+1} - z^k},\n    \\]  \n    which implies that \n    \\begin{equation}\n        \\label{eq-lemma-key-estimate-3}\n        \\begin{aligned}\n            &\\; \\frac{1}{2L}\\norm{G(z^{k+1})}^2 + \\frac{1}{2L}\\norm{G(z^{k})}^2 +  \\frac{1}{2L}\\norm{G(z^{k+1}) - G(z^k)}^2 \\\\\n            \\leq &\\;  \\inprod{G(z^{k+1})}{z^{k+1} - z^k + \\frac{1}{L}G(z^k)} - \\inprod{G(z^k)}{z^{k+1} - z^k}.\n        \\end{aligned}\n    \\end{equation}\n","sha256_utf8":"d30e82a6628fd8ece7978b71badc755529b68101dd3ee969a757c6baf3d7221d"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"代入差分恒等式控制误差内积","source_units":[{"start":84627,"end":86008,"start_line":867,"end_line":877,"raw_tex":"    The first two expressions for $z^{k+1} - z^k$ in Lemma \\ref{lemma-zdiff} together with \\eqref{eq-lemma-key-estimate-3} and the Cauchy-Schwarz inequality imply that \n        \\begin{align*}\n            &\\; \\frac{1}{2L}\\norm{G(z^{k+1})}^2 + \\frac{1}{2L}\\norm{G(z^{k})}^2 +  \\frac{1}{2L}\\norm{G(z^{k+1}) - G(z^k)}^2 \\\\\n            \\leq &\\; \\inprod{G(z^{k+1})}{\\frac{\\beta_k}{1-\\beta_k}(z^0 - z^{k+1}) - \\frac{1}{L}\\tilde{z}^k + \\frac{1}{L}G(z^k)} - \\inprod{G(z^k)}{\\beta_k(z^0 - z^k) - \\frac{1-\\beta_k}{L}\\tilde{z}^k} \\\\\n            = &\\; \\frac{\\beta_k}{1-\\beta_k}\\inprod{G(z^{k+1})}{z^0 - z^{k+1}} - \\beta_k\\inprod{G(z^k)}{z^0 - z^k} + \\frac{1}{L}\\inprod{G(z^{k+1})}{G(z^k) - \\tilde{z}^k} \\\\\n            &\\; + \\frac{1-\\beta_k}{L}\\inprod{G(z^k)}{\\tilde{z}^k}\\\\\n            = &\\; \\frac{\\beta_k}{1-\\beta_k}\\inprod{G(z^{k+1})}{z^0 - z^{k+1}} - \\beta_k\\inprod{G(z^k)}{z^0 - z^k}  + \\frac{1}{L}\\norm{G(z^k)}^2 \\\\\n            &\\; + \\frac{1}{L}\\inprod{G(z^{k+1}) - G(z^k)}{G(z^k) - \\tilde{z}^k} - \\frac{\\beta_k}{L}\\inprod{G(z^k)}{\\tilde{z}^k} \\\\\n            \\leq &\\; \\frac{\\beta_k}{1-\\beta_k}\\inprod{G(z^{k+1})}{z^0 - z^{k+1}} - \\beta_k\\inprod{G(z^k)}{z^0 - z^k}  + \\frac{1 - \\beta_k}{L}\\norm{G(z^k)}^2 \\\\\n            &\\; + \\frac{1}{2L}\\norm{G(z^{k+1}) - G(z^k)}^2 + \\frac{1}{2L}\\norm{G(z^k) - \\tilde{z}^k}^2 + \\frac{\\beta_k}{L}\\inprod{G(z^k)}{G(z^k) - \\tilde{z}^k}.\n        \\end{align*}\n","sha256_utf8":"7aee81a9f922e06584e2ed824c6cc27292bb81e46a3d5a7ba406a6b4e5e3bfb8"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"重排并缩放得到势函数递推","source_units":[{"start":86008,"end":87085,"start_line":878,"end_line":887,"raw_tex":"    Therefore, the inequality \\eqref{eq-lemma-key-estimate-1} holds by rearranging terms in the above inequality. By substituting $ \\beta_k:= 1 / (k+2) $ into \\eqref{eq-lemma-key-estimate-1}, and using the Cauchy-Schwarz inequality and the definition of the potential function $ \\mathcal{L}_k $, for $k\\geq 0$, we see that \n    \t\\begin{align*}\n    \t\t\\mathcal{L}_{k+1} \\leq &\\;  \\mathcal{L}_k  + \\frac{k+1}{L}\\inprod{G(z^k)}{G(z^k) - \\tilde{z}^k} + \\frac{(k+1)(k+2)}{2L}\\norm{G(z^k) - \\tilde{z}^k}^2 \\\\\n    \t\t\\leq &\\; \\mathcal{L}_k  + \\frac{k+1}{L}\\inprod{G(z^k)}{G(z^k) - \\tilde{z}^k} + \\frac{(k+1)^2}{L}\\norm{G(z^k) - \\tilde{z}^k}^2 \\\\\n    \t\t\\leq &\\; \\mathcal{L}_k  + \\frac{k+1}{2L}\\left(\\frac{1}{6(k+1)}\\norm{G(z^k)}^2 + 6(k+1) \\norm{G(z^k) - \\tilde{z}^k}^2 \\right) \\\\\n            &\\; + \\frac{(k+1)^2}{L}\\norm{G(z^k) - \\tilde{z}^k}^2 \\\\\n    \t\t= &\\; \\mathcal{L}_k  + \\frac{1}{12L}\\norm{G(z^k)}^2 + \\frac{4(k+1)^2}{L}\\norm{G(z^k) - \\tilde{z}^k}^2,\n    \t\\end{align*}\n    which proves the inequality \\eqref{eq-lemma-key-estimate-2}. Therefore, the proof is completed.\n\\end{proof}","sha256_utf8":"70b35e8150f80504d35ae3c81bd687870e582e5c3cc99d77391dc4fd66363b8d"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"利用预解非扩张性分解近似误差","source_units":[{"start":87624,"end":88561,"start_line":898,"end_line":907,"raw_tex":"\\begin{proof}\n\tSince the resolvent operator $J_{\\alpha E}$ is firmly non-expansive (hence $1$-Lipschitz continuous), we see that \n\t\t\\begin{align*}\n\t\t\t&\\; \\norm{G(z^k) - \\tilde{z}^k}^2 \\\\\n\t\t\t= &\\; \\norm{\\frac{1}{\\alpha}\\left(z^k - J_{\\alpha E}\\left(z^k -  \\alpha F(z^k)\\right)\\right)- \\tilde{z}^k}^2 \\\\\n\t\t\t= &\\; \\frac{1}{\\alpha^2}\\norm{J_{\\alpha E}\\left(z^k -  \\alpha F(z^k)\\right) - \\bar{z}^k}^2 \\\\\n\t\t\t\\leq &\\; \\frac{2}{\\alpha^2}\\norm{J_{\\alpha E}\\left(z^k -  \\alpha F(z^k)\\right) - J_{\\alpha E}\\left(z^k -  \\alpha \\widetilde{F}(z^k)\\right)}^2 + \\frac{2}{\\alpha^2}\\norm{J_{\\alpha E}\\left(z^k - \\alpha \\widetilde{F}(z^k)\\right) - \\bar{z}^k}^2 \\\\\n\t\t\t\\leq &\\; 2\\norm{F(z^k) - \\widetilde{F}(z^k)}^2 + \\gamma_k,\n\t\t\\end{align*}\n    where the first equality uses the definition of $G$ and the second equality uses the fact that $\\tilde z^k = \\frac{1}{\\alpha}(z^k - \\bar{z}^k)$, and the first inequality is due to the Cauchy-Schwarz inequality.\n","sha256_utf8":"051102e0b00381c45b0c9f83da2f01ade0744de2cc30b2964b83555ff63488d5"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"取期望并代入随机估计条件","source_units":[{"start":88561,"end":88793,"start_line":908,"end_line":914,"raw_tex":"\tThen, it follows that: \n\t\\[\n\t\t\\bb{E} \\left[\\norm{G(z^k) - \\tilde{z}^k}^2\\right] \\leq  2\\bb{E}\\left[\\norm{F(z^k) - \\widetilde{F}(z^k)}^2\\right] +  \\gamma_k \n\t\t\t\\leq    \\sigma_k + \\gamma_k,\n\t\\]\n\twhich completes the proof.\n\\end{proof}","sha256_utf8":"01374bf90e05a45ee94ace6956bd89b63324116bfdcfe4439e8f0fa79b7ea796"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"累加确定性势函数递推","source_units":[{"start":93353,"end":93600,"start_line":970,"end_line":974,"raw_tex":"\\begin{proof}\n\tFrom Lemma \\ref{lemma-key-estimate} and notice that $\\mathcal{L}_0 = 0$, we can see that  \n\t\\[\n\t\t\\mathcal{L}_k \\leq \\frac{1}{12L}\\sum_{i = 0}^{k-1}\\norm{G(z^i)}^2 + \\frac{4}{L}\\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2, \\quad k\\geq 1.\n\t\\]\n","sha256_utf8":"28f3a04c30c732ddba9f552ac0f6a2c1bd23fcd2ef09cbac4ce392bd06ba9551"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"借助解点得到残差递推界","source_units":[{"start":93600,"end":94679,"start_line":975,"end_line":991,"raw_tex":"\tBy the definition of the potential function $\\mathcal{L}_k$, we can see that \n\t\t\\begin{align*}\n\t\t\t\\frac{k(k+1)}{2L}\\norm{G(z^k)}^2 \\leq &\\; \\frac{1}{12L}\\sum_{i = 0}^{k-1}\\norm{G(z^i)}^2 + \\frac{4}{L}\\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2 + (k+1)\\inprod{G(z^k)}{z^0 - z^k} \\\\\n\t\t\t\\leq &\\; \\frac{1}{12L}\\sum_{i = 0}^{k-1}\\norm{G(z^i)}^2 + \\frac{4}{L}\\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2 + (k+1)\\norm{G(z^k)}\\norm{z^0 - z^*}\n\t\t\\end{align*} \n\twhere the last inequality is due to the fact that \n\t\\begin{align*}\n\t    \\inprod{G(z^k)}{z^0 - z^k} = &\\; \\inprod{G(z^k)}{z^0 - z^*} + \\inprod{G(z^k)}{z^* - z^k} \\\\\n     \\leq &\\;  \\inprod{G(z^k)}{z^0 - z^*} \\\\\n     \\leq &\\;  \\norm{G(z^k)}\\norm{z^0 - z^*},\n\t\\end{align*}\n\tand $z^*$ is any solution such that $G(z^*) = 0$.\n The first inequality holds, since $G$ is co-coercive. As a consequence, it holds that \n\t\\begin{equation}\n\t\\label{thm-rate-proof-1}\n\t\t\\norm{G(z^k)}^2 \\leq \\frac{1}{6k(k+1)}\\sum_{i = 0}^{k-1}\\norm{G(z^i)}^2 + \\frac{8}{k(k+1)}\\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2 + \\frac{2L}{k}\\norm{G(z^k)}\\norm{z^0 - z^*},  \n\t\\end{equation}\n","sha256_utf8":"c7d8f8e0c82d84de0627e5eca357ac41ee4e6b0e74b248f45b9ffacc222b02e4"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"引入标量记号并吸收线性项","source_units":[{"start":94679,"end":95548,"start_line":992,"end_line":1008,"raw_tex":"\tfor $k\\geq 1$. For notational simplicity, let us denote \n\t\\[\n\t\ta_k:= \\norm{G(z^k)}, \\quad k\\geq 0, \\quad b_k:= \\sum_{i = 0}^{k-1}(i+1)^2\\gamma_i^2,\\quad k\\geq 1,\\quad D:= L\\norm{z^0 - z^*}.\n\t\\]\n\tThen, from \\eqref{thm-rate-proof-1}, we see that, for $k\\geq 1$, \n\t\\begin{align*}\n\t\ta_k^2 \\leq &\\;  \\frac{1}{6k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{8}{k(k+1)}b_k + \\frac{2D}{k}a_k \\\\\n\t\t\\leq &\\; \\frac{1}{6k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{8}{k(k+1)}b_k + \\frac{1}{2}\\left(\\frac{4D^2}{k^2} + a_k^2 \\right),\n\t\\end{align*}\n\twhich implies that, for $k\\geq 1$,\n\t\\begin{equation}\n\t\\label{thm-rate-proof-2}\n\t\\begin{aligned}\n\t    a_k^2 \\leq &\\; \\frac{1}{3k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{16}{k(k+1)}b_k + \\frac{4D^2}{k^2} \\\\\n     \\leq &\\; \\frac{1}{3k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)}.\n\t\\end{aligned}\n\t\\end{equation}\n","sha256_utf8":"98479d0fe0e76c3d15ed632ee2529b38b51a4fbda56576acc3c4ec3d4955428a"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"提出残差界并验证归纳基例","source_units":[{"start":95548,"end":96018,"start_line":1009,"end_line":1017,"raw_tex":"\tWe next claim that the following inequality holds:\n\t\\begin{equation}\n\t\t\\label{thm-rate-proof-3}\n\t\ta_k^2 \\leq \\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)},\\quad k\\geq 1.\n\t\\end{equation}\n\tWe shall prove the above claim by induction. First, from \\eqref{thm-rate-proof-2} and notice that $a_0 \\leq D$ by the $L$-Lipschitz continuity of $G$, we see that\n\t\\[\n\t\ta_1^2 \\leq \\frac{1}{6}a_0^2 + 8b_1 + 4D^2 \\leq  \\frac{1}{6}D^2 + 8b_1 + 4D^2 \\leq \\frac{1}{6}(7D+10\\sqrt{b_1})^2.\n\t\\]\n","sha256_utf8":"b13d63f17c0020bf74b53d92682328e93f4b6e7a0c7f702d66914b3fc7502037"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"归纳与望远镜求和证明残差界","source_units":[{"start":96018,"end":97089,"start_line":1018,"end_line":1026,"raw_tex":"\tSo \\eqref{thm-rate-proof-3} holds for $k = 1$. Suppose now that \\eqref{thm-rate-proof-3} holds for indices $\\{1,\\dots, k-1\\}$. Again, from \\eqref{thm-rate-proof-2} and the fact that $b_k$ is non-decreasing, we deduce that \n\t\t\\begin{align*}\n\t\t\ta_k^2 \\leq &\\; \\frac{1}{3k(k+1)}\\sum_{i = 0}^{k-1} a_i^2 + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)} \\\\\n\t\t\t\\leq &\\; \\frac{1}{3k(k+1)}\\sum_{i = 0}^{k-1} \\frac{(7D + 10\\sqrt{b_i})^2}{(i+1)(i+2)} + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)} \\\\\n\t\t\t\\leq &\\; \\frac{(7D + 10\\sqrt{b_k})^2}{3k(k+1)}\\sum_{i = 0}^{k-1} \\frac{1}{(i+1)(i+2)} + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)} \\\\\n\t\t\t= &\\; \\frac{(7D + 10\\sqrt{b_k})^2}{3(k+1)^2} + \\frac{48}{(k+1)(k+2)}b_k + \\frac{24D^2}{(k+1)(k+2)} \\\\\n\t\t\t\\leq &\\; \\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)},\n\t\t\\end{align*}\n\twhich shows that \\eqref{thm-rate-proof-3} holds for index $k$. Thus, by induction, \\eqref{thm-rate-proof-3} holds for all $k\\geq 1$. Moreover, it is clear that \\eqref{thm-rate-proof-3} trivially holds for $k = 0$. This proves \\eqref{thm-rate-1}.\n","sha256_utf8":"fb9ba6e147d5df92b66feb37c7bd9572e7eb27144e263cf630c6c56f44cdac55"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"由残差界控制近似算子范数","source_units":[{"start":97091,"end":97334,"start_line":1028,"end_line":1031,"raw_tex":"\tTo prove  \\eqref{thm-rate-2}, we first observe that \n\t\\[\n\t\t\\norm{\\tilde{z}^k}^2 \\leq 2\\norm{G(z^k)}^2 + 2\\norm{G(z^k) - \\tilde{z}^k}^2 \\leq 2a_k^2 + 2\\gamma_k^2 \\leq  \\frac{2(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)} + 2\\gamma_k^2,\\quad k\\geq 0.\n\t\\]\n","sha256_utf8":"c11efe58a99001dc6fd53a23802cf88e17156986faae10c08117c30b9e38ecce"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"代入差分引理完成确定性步长界","source_units":[{"start":97334,"end":98561,"start_line":1032,"end_line":1046,"raw_tex":"\tThen, by the third statement of Lemma \\ref{lemma-zdiff} and the fact that $b_k$ is non-decreasing, we see that for $k\\geq 0$, \n\t\t\\begin{align*}\n\t\t\t\\norm{z^{k+1} - z^k}^2\n\t\t\t\\leq &\\; \\frac{2(k+1)^2}{L^2(k+2)^2}\\norm{\\tilde{z}^k}^2 + \\frac{2k}{L^2(k+1)^2(k+2)^2}\\sum_{i = 0}^{k-1}(i+1)^2\\norm{\\tilde{z}^i}^2 \\\\\n\t\t\t% \\leq &\\;  \\frac{4(k+1)^2}{L^2(k+2)^2}\\left(\\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)} + \\gamma_k^2 \\right) \\\\\n   % &\\; + \\frac{4k}{L^2(k+1)^2(k+2)^2}\\sum_{i = 0}^{k-1}\\left(\\frac{(i+1)(7D + 10\\sqrt{b_i})^2}{i+2} + (i+1)^2 \\gamma_i^2 \\right) \\\\\n\t\t\t\\leq &\\;  \\frac{4(k+1)^2}{L^2(k+2)^2}\\left(\\frac{(7D + 10\\sqrt{b_k})^2}{(k+1)(k+2)} + \\gamma_k^2 \\right) \\\\\n   &\\; + \\frac{4k}{L^2(k+1)^2(k+2)^2}\\sum_{i = 0}^{k-1}\\left(\\frac{(i+1)(7D + 10\\sqrt{b_k})^2}{i+2} + (i+1)^2 \\gamma_i^2 \\right) \\\\\n\t\t\t\\leq &\\; \\frac{4(7D + 10\\sqrt{b_k})^2}{L^2}\\left(\\frac{k+1}{(k+2)^3} + \\frac{k^2}{(k+1)^2(k+2)^2}\\right) \\\\\n   &\\; + \\frac{4(k+1)^2}{L^2(k+2)^2}\\gamma_k^2 + \\frac{4k}{L^2(k+1)^2(k+2)^2}b_k \\\\\n\t\t\t\\leq &\\; \\frac{8(7D + 10\\sqrt{b_{k+1}})^2 + 8b_{k+1}}{L^2(k+1)(k+2)}\\\\\n\t\t\t\\leq &\\; \\frac{8(7D + 11\\sqrt{b_{k+1}})^2 }{L^2(k+1)(k+2)},\n\t\t\\end{align*}\n\twhich proves \\eqref{thm-rate-2}. Therefore, the proof is completed.\n\\end{proof}","sha256_utf8":"bf32a55706095236b5490058a4fdc25502c06f22b5c88b08f39ca70a2bb65d7a"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S006","kind":"supports"},{"source_id":"S003","target_id":"S007","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S011","kind":"supports"},{"source_id":"S005","target_id":"S040","kind":"supports"},{"source_id":"S006","target_id":"S045","kind":"supports"},{"source_id":"S007","target_id":"S037","kind":"supports"},{"source_id":"S007","target_id":"S043","kind":"supports"},{"source_id":"S007","target_id":"S026","kind":"supports"},{"source_id":"S008","target_id":"S040","kind":"supports"},{"source_id":"S009","target_id":"S031","kind":"supports"},{"source_id":"S009","target_id":"S032","kind":"supports"},{"source_id":"S009","target_id":"S033","kind":"supports"},{"source_id":"S009","target_id":"S034","kind":"supports"},{"source_id":"S009","target_id":"S035","kind":"supports"},{"source_id":"S009","target_id":"S042","kind":"supports"},{"source_id":"S009","target_id":"S047","kind":"supports"},{"source_id":"S010","target_id":"S043","kind":"supports"},{"source_id":"S010","target_id":"S026","kind":"supports"},{"source_id":"S011","target_id":"S040","kind":"supports"},{"source_id":"S012","target_id":"S024","kind":"supports"},{"source_id":"S012","target_id":"S040","kind":"supports"},{"source_id":"S012","target_id":"S041","kind":"supports"},{"source_id":"S012","target_id":"S030","kind":"supports"},{"source_id":"S013","target_id":"S024","kind":"supports"},{"source_id":"S013","target_id":"S026","kind":"supports"},{"source_id":"S013","target_id":"S028","kind":"supports"},{"source_id":"S014","target_id":"S038","kind":"supports"},{"source_id":"S014","target_id":"S048","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S016","target_id":"S024","kind":"supports"},{"source_id":"S016","target_id":"S025","kind":"supports"},{"source_id":"S016","target_id":"S042","kind":"supports"},{"source_id":"S016","target_id":"S043","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S024","kind":"supports"},{"source_id":"S018","target_id":"S029","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S021","target_id":"S028","kind":"supports"},{"source_id":"S021","target_id":"S030","kind":"supports"},{"source_id":"S022","target_id":"S026","kind":"supports"},{"source_id":"S022","target_id":"S027","kind":"supports"},{"source_id":"S022","target_id":"S029","kind":"supports"},{"source_id":"S022","target_id":"S036","kind":"supports"},{"source_id":"S022","target_id":"S038","kind":"supports"},{"source_id":"S022","target_id":"S039","kind":"supports"},{"source_id":"S022","target_id":"S040","kind":"supports"},{"source_id":"S022","target_id":"S041","kind":"supports"},{"source_id":"S022","target_id":"S043","kind":"supports"},{"source_id":"S022","target_id":"S044","kind":"supports"},{"source_id":"S022","target_id":"S047","kind":"supports"},{"source_id":"S023","target_id":"S026","kind":"supports"},{"source_id":"S023","target_id":"S043","kind":"supports"},{"source_id":"S023","target_id":"S045","kind":"supports"},{"source_id":"S024","target_id":"S002","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S002","kind":"supports"},{"source_id":"S025","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S002","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S002","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S002","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S002","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S002","kind":"supports"},{"source_id":"S031","target_id":"S015","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S015","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S015","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S015","kind":"supports"},{"source_id":"S035","target_id":"S015","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S015","kind":"supports"},{"source_id":"S037","target_id":"S017","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S017","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S017","kind":"supports"},{"source_id":"S040","target_id":"S019","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S019","kind":"supports"},{"source_id":"S042","target_id":"S021","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S021","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S021","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S046","kind":"supports"},{"source_id":"S044","target_id":"S028","kind":"supports"},{"source_id":"S045","target_id":"S021","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S021","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S028","kind":"supports"},{"source_id":"S047","target_id":"S021","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S021","kind":"supports"},{"source_id":"S048","target_id":"S030","kind":"supports"}]}}
{"record_id":"2402.10898v3","paper":{"title":"The Price of Adaptivity in Stochastic Convex Optimization","authors":["Yair Carmon","Oliver Hinder"],"arxiv_id":"2402.10898","version":"v3","journal":"Mathematical Programming","doi":"10.1007/s10107-025-02268-3","abstract_url":"https://arxiv.org/abs/2402.10898v3","pdf_url":"https://arxiv.org/pdf/2402.10898v3"},"context":{"raw_tex":"\\paragraph{Notation.}\nThroughout the paper, $\\xset$ denotes a closed convex set (our lower bounds use $\\xset=\\R$) and $\\| \\cdot \\|$ denotes the Euclidean norm of a vector, and the infimum Euclidean norm in a set of vectors (or infinity if the set is empty).\n%\nFor convex function $h:\\xset \\to \\R$ we write (with minor abuse of notation) $\\grad h(x)$ for an arbitrary subgradient of $h$ at $x$, and let $\\del h(x)$ denote the set of all subgradients at $x$. \nWe say that $h$ is $L$-Lipschitz if $\\abs{ h(x) - h(y) } \\le L \\| x - y \\|$ for all $u, v \\in \\xset$ or, equivalently, $\\| \\grad h(x) \\| \\le L$ for almost-all $x\\in\\xset$. We use $[n]\\defeq\\{1,\\ldots,n\\}$ and $\\indic{\\cdot}$ for the indicator function.\n%\nThroughout, $\\log(\\cdot)$ denotes the natural logarithm and $\\N$ starts from 1.\n\n\nThe developments in this section use the standard notation\n\\[\n    I(X;Y) = H(X)-H(X\\mid Y)=H(Y)-H(Y\\mid X)\n\\]\nfor mutual information $I(X;Y)$, entropy $H(X)=\\E_{x\\sim X}\\log \\frac{1}{\\Pr(X=x)}$ and conditional entropy $H(X\\mid Y) = \\E_{x,y\\sim X,Y}\\log \\frac{1}{\\Pr(X=x | Y=y)}$.  We also write\n\\[\n    h_2(q) \\defeq H(\\bernoulli(q)) = q \\log\\frac{1}{q} + (1-q) \\log\\frac{1}{1-q}    \n\\]\nfor the binary entropy function.\n\n\n\\paragraph{Stochastic optimization (SO) problems.}\nA SO problem instance is a tuple $(f,P)$ containing a distribution $P$ over sample space $\\sset$ and a sample objective $f:\\xset\\times \\sset \\to \\R$. We write\n\\[\nF_{f,P}(x) \\defeq \\E_{S\\sim P} f(x;S) \n~~\\mbox{and}~~\nX^\\star_{f,P} \\defeq \\argmin_{x\\in\\xset} F_{f,P}(x)\n\\]\nfor the population objective (which we wish to minimize) and its set of minimizers, respectively. Let $\\Iclass$ denote a class of SO problem instances. We consider two fundamental classes of convex functions with a minimizer at most $R$ away from the origin.\\footnote{This implicitly, and without loss of generality, assumes that optimization methods are initialized at $x_0=0$.} The first class contains problems with bounded stochastic gradient norm (i.e., where each sample function is $L$-Lipschitz),\n\\begin{equation*}\n\t\\Iclass[\\Lip]^{L,R} \\defeq \\crl*{(f,P) \\mid f(\\cdot; s) \\text{ is convex and $L$-Lipschitz for all $s \\in \\sset$, and } \\norm*{X^\\star_{f,P}} \\le R }.\n\\end{equation*}\nThe second class contains problems with bounded stochastic gradient second moment, \n%\n%\n\\begin{equation*}\n\t\\Iclass[\\SMLip]^{L,R} \\defeq \n\t\\crl*{(f,P) \\mid f(\\cdot; s) \\text{ is convex $\\forall s\\in\\sset$ and }\n\t\\E_{S\\sim P}\\norm{\\grad f(\\cdot; S)}^2 \\le L^2,\\text{ and } \\norm*{X^\\star_{f,P}} \\le R }.\n\t%\n\\end{equation*}\nClearly, $\\Iclass[\\Lip]^{L,R} \\subset \\Iclass[\\SMLip]^{L,R}$, with $\\Iclass[\\SMLip]^{L,R}$ also including problems with heavy-tailed gradient noise.\\yair{consider defining $\\Iclass[*\\text{-}Lip]$ and $\\Mclass[*\\text{-}Lip]$ as a catch-both term.}\n\n\n\\paragraph*{Optimization algorithms.}\nGeneral SO algorithms have unrestricted access to the function $f(\\cdot;\\cdot)$ but observe $P$ only through samples $S_1,S_2,\\ldots\\simiid P$. The output of general SO algorithms is therefore of the form\n\\[\n%\nx_{t} = \\alg_t\\prn[\\big]{ f(\\cdot; S_1) , \\dots,  f(\\cdot; S_T) ; \\xi } \\in \\xset,\n%\n\\]\nwhere $\\alg_t$ is some measurable mapping and $\\xi \\sim \\uniform([0,1])\\perp S_1, \\ldots, S_T$ allows randomization.\n\nStochastic first-order (SFO) optimization algorithms are SO algorithms that depend on $f$ only through gradients observed at past iterates. That is, their output sequence takes the form \n\\[\n%\nx_{t} = \\alg_{t}\\prn*{\\grad f(x_0; S_1) , \\dots,  \\grad f(x_{t-1}; S_t); \\xi} \\in \\xset,\n%\n\\]\nwith $\\xi \\sim \\uniform([0,1])\\perp S_1, \\ldots, S_T$ as before.\n%\nWe write $\\Aall$ and $\\AFO$ for the sets of all SO and all SFO algorithms, respectively, so that $\\AFO\\subset \\Aall$. All the algorithms from \\Cref{tab:poa-upper-lower-bounds} are in $\\AFO$. \n\n\n\\paragraph{Error metrics.}\nFor an algorithm $\\alg$, SO instance $(f,P)$, and budget $T$, we let $x_T$ denote the algorithm's output given $T$ samples and define the expected error to be\n\\begin{equation*}\n\t\\ErrE{\\alg}{(f,P)} \\defeq \\E F_{f,P}(x_T) - \\inf_{\\xopt\\in\\xset} F_{f,P}(\\xopt),\n\\end{equation*}\nwhere the expectation is with respect to $x_T$ and hence $S_1,\\ldots,S_T$ and $\\xi$. \n%\nWe similarly define the high-probability error at confidence level $\\delta$ as\n\\begin{equation*}\n\t\\ErrHP{\\alg}{(f,P)} \\defeq Q_{1-\\delta}( F_{f,P}(x_T) ) - \\inf_{\\xopt\\in\\xset} F_{f,P}(\\xopt),\n\\end{equation*}\nwhere $Q_p(Y)=\\min\\{ y : \\Pr( Y \\le y ) \\ge p\\}$ denotes the $p$'th quantile of random variable $Y$ \\cite[Ch. 3]{borovkov1999probability}.\n\n\nThroughout, we drop the superscripts $\\E$  and $\\delta$ when making statements that apply to both.\n\n\\paragraph{Minimax error.}\nGiven an instance class $\\Iclass$ and an algorithm $\\alg$, we overload the notation above to denote worst-case error over the class,\n\\begin{equation*}\n\t\\Err{\\alg}{\\Iclass} \\defeq \\sup_{(f,P)\\in\\Iclass} \\Err{\\alg}{(f,P)}.\n\\end{equation*}\n%\nGiven an algorithm class $\\Aclass$, we further overload our notation to express minimax optimal worst-case error\n\\begin{equation}\n\t\\Err{\\Aclass}{\\Iclass} \\defeq \\inf_{\\alg\\in\\Aclass} \\Err{\\alg}{\\Iclass}\n\t= \\inf_{\\alg\\in\\Aclass} \\sup_{(f,P)\\in\\Iclass} \\Err{\\alg}{(f,P)}.\n\t\\label{eq:minimax-err-def}\n\\end{equation}\n\n\nGiven stochastic optimization instance ($f,P$), step size parameter $\\eta$, clipping parameter $\\gclip$ and initial point $x_0 \\in \\xset$, clipped SGD consists of the following recursion:\n\\begin{equation*}\n    x_{t+1} = \\Pi_{\\xset'}\\prn*{x_t - \\eta \\gbar_t},~~\\mbox{where}~~\\gbar_t \\defeq \\frac{g_t}{\\max\\{1, \\norm{g_t} / \\gclip\\}}~~\\mbox{and}~~g_t \\in \\del f(x_t; S_{t+1}).\n    \\numberthis \\label{eq:clipped-SGD}\n\\end{equation*}\n\nHere $\\Pi_{\\xset'}$ is the Euclidean projection onto $\\xset' = \\xset \\cap \\{ x \\mid \\norm{x-x_0} \\le R\\}$.\n\n\nWe begin with the notion of a \\emph{meta-class}, i.e., a class of classes, which we denote by $\\Mclass$. For any $\\ell,\\rho \\ge 1$, we define\n\\begin{equation*}\n\t\\Mclass[\\Lip]^{\\ell, \\rho} \\defeq \n\t\\crl*{\\Iclass[\\Lip]^{L,R} \\mid L\\in[1, \\ell]\\text{ and }R\\in[1,\\rho]}\n\\end{equation*}\n\nThe PoA of algorithm $\\alg$ competing against all SO algorithms on meta-class $\\Mclass$ is\n\\begin{equation*}\n\t\\PoA{\\alg}{\\Mclass} \\defeq \\sup_{\\Iclass \\in \\Mclass} \n\t\\frac{\\Err{\\alg}{\\Iclass}}{\\Err{\\Aall}{\\Iclass}}.\n\\end{equation*}\n\nWe use $\\PoAE{\\alg}{\\Mclass}$ and $\\PoAHP{\\alg}{\\Mclass}$ to denote the price of adaptivity w.r.t.\\ expected and quantile error, respectively (i.e., with $\\ErrBlank$ standing for $\\ErrBlank^{\\E}$ and $\\ErrBlank^{\\delta}$, respectively). \nWe may also consider the price of adaptivity in competing against algorithms in $\\Aclass\\subset\\Aall$, which we denote \n\\begin{equation}\n\t\\PoAGen{\\alg}{\\Mclass}{\\Aclass} \\defeq \\sup_{\\Iclass \\in \\Mclass} \n\t\\frac{\\Err{\\alg}{\\Iclass}}{\\Err{\\Aclass}{\\Iclass}}.\n\t\\label{eq:PoAGen-def}\n\\end{equation}\n\nOur strategy to lower bound the PoA in this paper is to carefully construct a collection of $n$ `hard' problem instances $(f_1, P_1), \\dots, (f_n, P_n)$ such that $(f_k, P_k) \\in {\\Iclass}_k$ with ${\\Iclass}_k \\in \\Mclass$  and then utilize the following observation\n\\begin{flalign}\\label{eq:PoA-lower-bounding-strategy}\n\\PoAGen{\\alg}{\\Mclass}{\\Aclass} \\ge \\max_{k \\in [n]} \n\\frac{\\Err{\\alg}{{\\Iclass}_k}}{\\Err{\\Aclass}{{\\Iclass}_k}} \\ge \\max_{k \\in [n]} \n\\frac{\\Err{\\alg}{(f_k, P_k)}}{\\Err{\\Aclass}{{\\Iclass}_k}}.\n\\end{flalign}\n\nFor brevity we define $g_t = \\grad f(x_t; S_{t+1})$ and $\\Delta_t = f(x_t; S_{t+1}) - f(x_\\star; S_{t+1})$.\n","units":[{"start":25149,"end":25942,"start_line":689,"end_line":695,"raw_tex":"\\paragraph{Notation.}\nThroughout the paper, $\\xset$ denotes a closed convex set (our lower bounds use $\\xset=\\R$) and $\\| \\cdot \\|$ denotes the Euclidean norm of a vector, and the infimum Euclidean norm in a set of vectors (or infinity if the set is empty).\n%\nFor convex function $h:\\xset \\to \\R$ we write (with minor abuse of notation) $\\grad h(x)$ for an arbitrary subgradient of $h$ at $x$, and let $\\del h(x)$ denote the set of all subgradients at $x$. \nWe say that $h$ is $L$-Lipschitz if $\\abs{ h(x) - h(y) } \\le L \\| x - y \\|$ for all $u, v \\in \\xset$ or, equivalently, $\\| \\grad h(x) \\| \\le L$ for almost-all $x\\in\\xset$. We use $[n]\\defeq\\{1,\\ldots,n\\}$ and $\\indic{\\cdot}$ for the indicator function.\n%\nThroughout, $\\log(\\cdot)$ denotes the natural logarithm and $\\N$ starts from 1.\n","sha256_utf8":"edc5e1a4660c0779f8cbaba70700fe5d204be49320c091e1ef87a01efdfc6f40"},{"start":38281,"end":38700,"start_line":772,"end_line":780,"raw_tex":"The developments in this section use the standard notation\n\\[\n    I(X;Y) = H(X)-H(X\\mid Y)=H(Y)-H(Y\\mid X)\n\\]\nfor mutual information $I(X;Y)$, entropy $H(X)=\\E_{x\\sim X}\\log \\frac{1}{\\Pr(X=x)}$ and conditional entropy $H(X\\mid Y) = \\E_{x,y\\sim X,Y}\\log \\frac{1}{\\Pr(X=x | Y=y)}$.  We also write\n\\[\n    h_2(q) \\defeq H(\\bernoulli(q)) = q \\log\\frac{1}{q} + (1-q) \\log\\frac{1}{1-q}    \n\\]\nfor the binary entropy function.\n","sha256_utf8":"f717d5a1baf3594e9eb58df14b5f0afcf3dcfe59c7351ba3f9ff8303b3481637"},{"start":46229,"end":47810,"start_line":944,"end_line":964,"raw_tex":"\\paragraph{Stochastic optimization (SO) problems.}\nA SO problem instance is a tuple $(f,P)$ containing a distribution $P$ over sample space $\\sset$ and a sample objective $f:\\xset\\times \\sset \\to \\R$. We write\n\\[\nF_{f,P}(x) \\defeq \\E_{S\\sim P} f(x;S) \n~~\\mbox{and}~~\nX^\\star_{f,P} \\defeq \\argmin_{x\\in\\xset} F_{f,P}(x)\n\\]\nfor the population objective (which we wish to minimize) and its set of minimizers, respectively. Let $\\Iclass$ denote a class of SO problem instances. We consider two fundamental classes of convex functions with a minimizer at most $R$ away from the origin.\\footnote{This implicitly, and without loss of generality, assumes that optimization methods are initialized at $x_0=0$.} The first class contains problems with bounded stochastic gradient norm (i.e., where each sample function is $L$-Lipschitz),\n\\begin{equation*}\n\t\\Iclass[\\Lip]^{L,R} \\defeq \\crl*{(f,P) \\mid f(\\cdot; s) \\text{ is convex and $L$-Lipschitz for all $s \\in \\sset$, and } \\norm*{X^\\star_{f,P}} \\le R }.\n\\end{equation*}\nThe second class contains problems with bounded stochastic gradient second moment, \n%\n%\n\\begin{equation*}\n\t\\Iclass[\\SMLip]^{L,R} \\defeq \n\t\\crl*{(f,P) \\mid f(\\cdot; s) \\text{ is convex $\\forall s\\in\\sset$ and }\n\t\\E_{S\\sim P}\\norm{\\grad f(\\cdot; S)}^2 \\le L^2,\\text{ and } \\norm*{X^\\star_{f,P}} \\le R }.\n\t%\n\\end{equation*}\nClearly, $\\Iclass[\\Lip]^{L,R} \\subset \\Iclass[\\SMLip]^{L,R}$, with $\\Iclass[\\SMLip]^{L,R}$ also including problems with heavy-tailed gradient noise.\\yair{consider defining $\\Iclass[*\\text{-}Lip]$ and $\\Mclass[*\\text{-}Lip]$ as a catch-both term.}\n","sha256_utf8":"40ffb65746bfc7eba381d05aa142ed2ba63dadbe23c94a82b39d3241a4261334"},{"start":47811,"end":48810,"start_line":966,"end_line":983,"raw_tex":"\\paragraph*{Optimization algorithms.}\nGeneral SO algorithms have unrestricted access to the function $f(\\cdot;\\cdot)$ but observe $P$ only through samples $S_1,S_2,\\ldots\\simiid P$. The output of general SO algorithms is therefore of the form\n\\[\n%\nx_{t} = \\alg_t\\prn[\\big]{ f(\\cdot; S_1) , \\dots,  f(\\cdot; S_T) ; \\xi } \\in \\xset,\n%\n\\]\nwhere $\\alg_t$ is some measurable mapping and $\\xi \\sim \\uniform([0,1])\\perp S_1, \\ldots, S_T$ allows randomization.\n\nStochastic first-order (SFO) optimization algorithms are SO algorithms that depend on $f$ only through gradients observed at past iterates. That is, their output sequence takes the form \n\\[\n%\nx_{t} = \\alg_{t}\\prn*{\\grad f(x_0; S_1) , \\dots,  \\grad f(x_{t-1}; S_t); \\xi} \\in \\xset,\n%\n\\]\nwith $\\xi \\sim \\uniform([0,1])\\perp S_1, \\ldots, S_T$ as before.\n%\nWe write $\\Aall$ and $\\AFO$ for the sets of all SO and all SFO algorithms, respectively, so that $\\AFO\\subset \\Aall$. All the algorithms from \\Cref{tab:poa-upper-lower-bounds} are in $\\AFO$. \n","sha256_utf8":"5c5618ce8011a41902e33036585a4f92199d1f8f397a4a6f736b199af06d95cb"},{"start":48812,"end":49550,"start_line":986,"end_line":997,"raw_tex":"\\paragraph{Error metrics.}\nFor an algorithm $\\alg$, SO instance $(f,P)$, and budget $T$, we let $x_T$ denote the algorithm's output given $T$ samples and define the expected error to be\n\\begin{equation*}\n\t\\ErrE{\\alg}{(f,P)} \\defeq \\E F_{f,P}(x_T) - \\inf_{\\xopt\\in\\xset} F_{f,P}(\\xopt),\n\\end{equation*}\nwhere the expectation is with respect to $x_T$ and hence $S_1,\\ldots,S_T$ and $\\xi$. \n%\nWe similarly define the high-probability error at confidence level $\\delta$ as\n\\begin{equation*}\n\t\\ErrHP{\\alg}{(f,P)} \\defeq Q_{1-\\delta}( F_{f,P}(x_T) ) - \\inf_{\\xopt\\in\\xset} F_{f,P}(\\xopt),\n\\end{equation*}\nwhere $Q_p(Y)=\\min\\{ y : \\Pr( Y \\le y ) \\ge p\\}$ denotes the $p$'th quantile of random variable $Y$ \\cite[Ch. 3]{borovkov1999probability}.\n","sha256_utf8":"9e80943e9f825e51754952abad8e881c89ea57dd1a288355e149fdaacf881a70"},{"start":49564,"end":50243,"start_line":1005,"end_line":1018,"raw_tex":"Throughout, we drop the superscripts $\\E$  and $\\delta$ when making statements that apply to both.\n\n\\paragraph{Minimax error.}\nGiven an instance class $\\Iclass$ and an algorithm $\\alg$, we overload the notation above to denote worst-case error over the class,\n\\begin{equation*}\n\t\\Err{\\alg}{\\Iclass} \\defeq \\sup_{(f,P)\\in\\Iclass} \\Err{\\alg}{(f,P)}.\n\\end{equation*}\n%\nGiven an algorithm class $\\Aclass$, we further overload our notation to express minimax optimal worst-case error\n\\begin{equation}\n\t\\Err{\\Aclass}{\\Iclass} \\defeq \\inf_{\\alg\\in\\Aclass} \\Err{\\alg}{\\Iclass}\n\t= \\inf_{\\alg\\in\\Aclass} \\sup_{(f,P)\\in\\Iclass} \\Err{\\alg}{(f,P)}.\n\t\\label{eq:minimax-err-def}\n\\end{equation}\n","sha256_utf8":"e33a98b9b7743cd28f734d4f0ec610213eb59b5ed87f8e8c467933f66098de93"},{"start":53432,"end":53861,"start_line":1063,"end_line":1067,"raw_tex":"Given stochastic optimization instance ($f,P$), step size parameter $\\eta$, clipping parameter $\\gclip$ and initial point $x_0 \\in \\xset$, clipped SGD consists of the following recursion:\n\\begin{equation*}\n    x_{t+1} = \\Pi_{\\xset'}\\prn*{x_t - \\eta \\gbar_t},~~\\mbox{where}~~\\gbar_t \\defeq \\frac{g_t}{\\max\\{1, \\norm{g_t} / \\gclip\\}}~~\\mbox{and}~~g_t \\in \\del f(x_t; S_{t+1}).\n    \\numberthis \\label{eq:clipped-SGD}\n\\end{equation*}","sha256_utf8":"957330ec84fac7db1653d88431c7edfd9bef6dd54d795903f90e6673d9ed6519"},{"start":53862,"end":53969,"start_line":1068,"end_line":1068,"raw_tex":"Here $\\Pi_{\\xset'}$ is the Euclidean projection onto $\\xset' = \\xset \\cap \\{ x \\mid \\norm{x-x_0} \\le R\\}$.\n","sha256_utf8":"31804100500cea760894a7fca2bf70ea6d6588ecaf538e76e076a72deaf57375"},{"start":63882,"end":64164,"start_line":1223,"end_line":1227,"raw_tex":"We begin with the notion of a \\emph{meta-class}, i.e., a class of classes, which we denote by $\\Mclass$. For any $\\ell,\\rho \\ge 1$, we define\n\\begin{equation*}\n\t\\Mclass[\\Lip]^{\\ell, \\rho} \\defeq \n\t\\crl*{\\Iclass[\\Lip]^{L,R} \\mid L\\in[1, \\ell]\\text{ and }R\\in[1,\\rho]}\n\\end{equation*}","sha256_utf8":"346e3f537c72bc34e2e92c6f6111148bcd7578e3494621274cd8ed06b6f3f468"},{"start":64603,"end":64834,"start_line":1230,"end_line":1234,"raw_tex":"The PoA of algorithm $\\alg$ competing against all SO algorithms on meta-class $\\Mclass$ is\n\\begin{equation*}\n\t\\PoA{\\alg}{\\Mclass} \\defeq \\sup_{\\Iclass \\in \\Mclass} \n\t\\frac{\\Err{\\alg}{\\Iclass}}{\\Err{\\Aall}{\\Iclass}}.\n\\end{equation*}","sha256_utf8":"ad357ee2c3629da69c4ed8ca2a0f798bd0e2328b461cb0d5ec99127164b0f03b"},{"start":64835,"end":65368,"start_line":1235,"end_line":1241,"raw_tex":"We use $\\PoAE{\\alg}{\\Mclass}$ and $\\PoAHP{\\alg}{\\Mclass}$ to denote the price of adaptivity w.r.t.\\ expected and quantile error, respectively (i.e., with $\\ErrBlank$ standing for $\\ErrBlank^{\\E}$ and $\\ErrBlank^{\\delta}$, respectively). \nWe may also consider the price of adaptivity in competing against algorithms in $\\Aclass\\subset\\Aall$, which we denote \n\\begin{equation}\n\t\\PoAGen{\\alg}{\\Mclass}{\\Aclass} \\defeq \\sup_{\\Iclass \\in \\Mclass} \n\t\\frac{\\Err{\\alg}{\\Iclass}}{\\Err{\\Aclass}{\\Iclass}}.\n\t\\label{eq:PoAGen-def}\n\\end{equation}","sha256_utf8":"3065a6195a4e9ae4330abee71e4d3d43449e96d1c07b08eb25251008dcff16ec"},{"start":65369,"end":65897,"start_line":1242,"end_line":1247,"raw_tex":"Our strategy to lower bound the PoA in this paper is to carefully construct a collection of $n$ `hard' problem instances $(f_1, P_1), \\dots, (f_n, P_n)$ such that $(f_k, P_k) \\in {\\Iclass}_k$ with ${\\Iclass}_k \\in \\Mclass$  and then utilize the following observation\n\\begin{flalign}\\label{eq:PoA-lower-bounding-strategy}\n\\PoAGen{\\alg}{\\Mclass}{\\Aclass} \\ge \\max_{k \\in [n]} \n\\frac{\\Err{\\alg}{{\\Iclass}_k}}{\\Err{\\Aclass}{{\\Iclass}_k}} \\ge \\max_{k \\in [n]} \n\\frac{\\Err{\\alg}{(f_k, P_k)}}{\\Err{\\Aclass}{{\\Iclass}_k}}.\n\\end{flalign}","sha256_utf8":"6a1b569b2dffac1f75284cd88e85314440b9e00c7abeddba10161d0286fda546"},{"start":102232,"end":102340,"start_line":2013,"end_line":2013,"raw_tex":"For brevity we define $g_t = \\grad f(x_t; S_{t+1})$ and $\\Delta_t = f(x_t; S_{t+1}) - f(x_\\star; S_{t+1})$.\n","sha256_utf8":"0af93e20974a0c3c25ca77d43ee12250880102e5a7e019baf7d9cd9eccc551c9"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{thm:log-PoA-lb}\nFor all $T\\in\\N$, $\\rho \\ge 1$, $\\alg\\in\\Aall$ we have \n%\n%\n$\\bigPoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \\ge \\Omega\\prn[\\big]{\\min\\crl[\\big]{\\sqrt{\\log (\\rho)}, \\sqrt{T}}}$.\n%\n%\n%\n\\end{theorem}","units":[{"start":68508,"end":68734,"start_line":1283,"end_line":1291,"raw_tex":"\\begin{theorem}\\label{thm:log-PoA-lb}\nFor all $T\\in\\N$, $\\rho \\ge 1$, $\\alg\\in\\Aall$ we have \n%\n%\n$\\bigPoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \\ge \\Omega\\prn[\\big]{\\min\\crl[\\big]{\\sqrt{\\log (\\rho)}, \\sqrt{T}}}$.\n%\n%\n%\n\\end{theorem}","sha256_utf8":"8f81ebdec28583b8736ff6295e9fc86711f816f8712a716a21530b79618e9d0a"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lem}\\label{lem:general-mi-bound}\n    For any $T\\in \\N$ and $\\eps \\in [0,1]$ and any random variable $V$, let $S_1,\\ldots, S_T$ be a sequence of binary random variables such that, for all $t\\le T$ \\yair{could also write it as a TV distance bound}\n    %\n    \\[\n        \\abs*{\\Pr(S_t = 0| S_1, \\ldots, S_{t-1}, V) - \\half} \\le \\frac{\\eps}{2}\n    \\]\n     with probability 1 w.r.t.\\ $S_1,\\ldots, S_{t-1}$ and $V$. Then, for every randomized estimator $\\hat{V}$ that depends on $V$ only through $S_1, \\ldots, S_T$, we have\n    \\[\n        I(V;\\hat{V}) \\le \\eps^2 T.\\numberthis \\label{eq:general-mi-bound-1}\n    \\]\n    If in addition $\\eps \\le \\half$ and, for some $p\\in [0,\\half]$, we have that\n    \\[\n        \\abs*{\\Pr(S_t = 0) - \\half} \\ge \\eps \\prn*{\\frac{1}{2} - p}\n    \\]\n    for every $t\\le T$, then\n    \\[\n        I(V;\\hat{V}) \\le 4p \\eps^2 T.\\numberthis \\label{eq:general-mi-bound-2}\n    \\]\n\\end{lem}\n\n\\begin{proof}\n    By the data processing inequality \\citep[Theorem 2.8.1]{cover1991elements} and chain rule for mutual information \\cite[Theorem 2.5.2]{cover1991elements}, \n    %\n\t\\begin{flalign*}\n\t\tI(\\hat{V}; V) & \\le I(S_1, \\ldots, S_T; V) \n\t\t= \\sum_{t=1}^{T} \\brk[\\Big]{ H(S_t \\mid S_1, \\ldots, S_{t-1}) - H(S_t \\mid S_1, \\ldots, S_{t-1}, V) }. \\numberthis \\label{eq:log-mi-expression}\n\t\\end{flalign*}\n\tUsing the facts conditioning decreases entropy and that binary random variables have entropy at most $\\log 2$, we get\n    \\[\n\t\tH(S_t \\mid S_1, \\ldots, S_{t-1}) \\le H(S_t) \\le \\log 2.\n\t\\]\n    Next, we have \n    \\[\n\t\tH(S_t \\mid S_1, \\ldots, S_{t-1}, V) = \\E_{S_1, \\ldots, S_{t-1}, V} \\brk*{h_2 \\prn[\\Big]{\\Pr(S_t = 0| S_1, \\ldots, S_{t-1}, V) }}\n         \\ge  h_2\\prn*{\\frac{1-\\eps}{2}},\n    \\]\n    by the assumption that $\\abs*{\\Pr(S_t = 0| S_1, \\ldots, S_{t-1}, V) - \\half} \\le \\frac{\\eps}{2}$ with probability 1 and the fact that $h_2(x)$ has a single maximum at $x=\\half$.\n\n    Combining the last two displays and using $\\log(1+x)\\le x$, we obtain\n    \\begin{flalign*}\n        H(S_t \\mid S_1, \\ldots, S_{t-1}) &- H(S_t \\mid S_1, \\ldots, S_{t-1}, V) \\\\& \n        \\le \\log 2 - h_2\\prn*{\\frac{1-\\eps}{2}}\n        \\\\ &=\n\t\t\\frac{1+\\veps}{2} \\log(1+\\veps) + \\frac{1-\\veps}{2} \\log(1-\\veps) \n        \\le \n\t\t\\veps^2,\n    \\end{flalign*}\n    where the last inequality is due to $\\log(1+x)\\le x$.\n    Substituting back into~\\cref{eq:log-mi-expression} yields the first claimed bound, $I(\\hat{V}; V) \\le \\eps^2 T$. \n\n    To obtain the second claimed bound we use the assumption that $\\abs*{\\Pr(S_t = 0) - \\half} \\ge \\eps \\prn*{\\frac{1}{2} - p}$ to write\n    \\[\n\t\tH(S_t \\mid S_1, \\ldots, S_{t-1}) \\le H(S_t) \n        \\le h_2\\prn*{\\frac{1-(1-2p)\\eps}{2}},\n    \\]\n    improving the bound on $H(S_t \\mid S_1, \\ldots, S_{t-1}) - H(S_t \\mid S_1, \\ldots, S_{t-1}, V)$ to \n    \\begin{flalign*}\n        h_2\\prn*{\\frac{1-(1-2p)\\eps}{2}} - h_2\\prn*{\\frac{1-\\eps}{2}}\n         & \\overle{(i)} \n        \\prn*{\\frac{1-(1-2p)\\eps}{2} - \\frac{1-\\eps}{2}} h_2'\\prn*{\\frac{1-\\eps}{2}}\n        \\\\ & \\overeq{(ii)} \n        p \\eps \\log\\prn*{1+\\frac{2\\eps}{1-\\eps}} \\overle{(iii)} \\frac{2p\\eps^2}{1-\\eps} \\overle{(iv)} 4p\\eps^2,\n    \\end{flalign*}\n    due to $(i)$ concavity of binary entropy, $(ii)$ the fact that $h_2'(x) = \\log\\prn*{\\frac{1-x}{x}}$, $(iii)$ the inequality $\\log(1+x)\\le x$ and $(iv)$ our assumption that $\\veps \\le \\half$. Substituting back into~\\eqref{eq:log-mi-expression}, we get that $I(\\hat{V}; V) \\le 4p\\eps^2 T$, as required.\n\\end{proof}\n\n\\begin{lem}\\label{lem:skewed-binary-bound}\n\tFor any $p\\in(0,\\frac{1}{2}]$ and $T\\in \\N$, let $V\\sim \\bernoulli(p)$ and\n\t%\n\t%\n\t%\n\t\\[\n\t\\eps = \\min\\crl*{\\sqrt{\\frac{\\log \\frac{1-p}{p}}{8T}}, \\half}.\t\n\t\\]\n\tLet $P_0 = \\bernoulli(\\frac{1-\\eps}{2})$ and $P_1 = \\bernoulli(\\frac{1+\\eps}{2})$. Then, for $S_1, \\ldots, S_T \\simiid P_V$ and (potentially randomized) estimator $\\hat{V}$ that depends on $V$ only through $S_1, \\ldots, S_T$, we have %\n\t\\[\n\t\t\\Pr(\\hat{V} \\ne V) \\ge \\frac{p}{2}.\n\t\\]\n\\end{lem}\n\n\\begin{proof}\n\tTo streamline notation, we write \n\t\\[\n\t\t%\n\t\t%\n\t\tq \\defeq \\Pr(\\hat{V} \\ne V). \n\t\\] \n\tBy Fano's inequality (\\Cref{fact:fano}) with $n=2$,\n\t\\[\n\t\th_2(q) \\ge H(V\\mid \\hat{V}) =  H(V) - I(V;\\hat{V}).\t\n\t\\numberthis \\label{eq:skewed-fano}\n\t\\]\n\tby \\cref{eq:general-mi-bound-2} in \\Cref{lem:general-mi-bound}, \n    \\[\n\t\tI(\\hat{V}; V) \\le 4p\\eps^2 T \\overle{(i)} \\frac{p}{2} \\log\\frac{1-p}{p} \\overeq{(ii)} \\prn*{p - \\frac{p}{2}}h_2'(p) \\overle{(iii)} h_2(p) - h_2(p/2), \n\t\t%\n\t\\]\n\tdue to $(i)$ our choice of $\\eps \\le \\sqrt{\\frac{\\log \\frac{1-p}{p}}{8T}}$, (ii) the fact that $h_2'(x) = \\log\\frac{1-x}{x}$ and $(iii)$ concavity of entropy. Noting that $H(V) = h_2(p)$ and substituting back to~\\eqref{eq:skewed-fano}, we get that $h_2(q) \\ge h_2(p/2)$, which implies $q\\ge p/2$ since $h_2$ is monotone in $[0, \\half]$. \n\\end{proof}\n\n\\begin{restatable}{prop}{restatePropStandardErrorBounds}\\label{prop:standard-error-bounds}\nFor all $R, L > 0$, instance class $\\Iclass \\in \\{ \\Iclass[\\Lip]^{L,R}, \\Iclass[\\SMLip]^{L,R} \\}$ and \n$\\Aclass \\in \\{ \\Aall, \\AFO \\}$, we have \n$\\ErrE{\\Aclass}{\\Iclass}=\\Theta\\left( \\frac{LR}{\\sqrt{T} } \\right)$. Moreover, for any $\\delta\\in(0,\\half)$ we have $\\ErrHP{\\Aclass}{\\Iclass}=\\Theta\\left( \\frac{LR}{\\sqrt{1+ {T}/{\\log \\frac{1}{\\delta}}}} \\right)$.\n\\end{restatable}\n\n\\begin{proof}\nSince $\\Iclass[\\Lip]^{L,R}\\subset \\Iclass[\\SMLip]^{L,R}$ and $\\AFO\\subset\\Aall$, we only need to prove minimax error lower bounds on $(\\Aall,\\Iclass[\\Lip]^{L,R})$ and minimax upper bounds on $(\\AFO, \\Iclass[\\SMLip]^{L,R})$. \n\nFor expected error these results are well-known. The upper bound on $\\ErrE{\\AFO}{ \\Iclass[\\SMLip]^{L,R}}$ follows, for example, by setting $\\eta = \\frac{R}{L\\sqrt{T}}$ in \\cref{eq:sgd-expected-err}. For the lower bound on $\\ErrE{\\Aall}{ \\Iclass[\\Lip]^{L,R}}$, originally due to~\\citet{nemirovski1983problem}, see, e.g., \\citet[Theorem 5.2.10.]{duchi2018introductory}.\n\nFor error quantiles, however, we could not find direct proof of the required upper and lower bounds, and we prove them in \\Cref{app:high-prob-ms-lip,app:high-prop-lip-lb} below.\n\\end{proof}\n\n\\begin{customprop}{1a}\\label{prop:ms-lip-hp-bound}\n    For any $\\delta \\in (0,\\half)$, any $L,R>0$, and any instance in $\\Iclass[\\SMLip]^{L, R}$, we have that $T$ steps of clipped SGD~\\eqref{eq:clipped-SGD} with step size $\\eta = \\frac{R}{L\\sqrt{T}}$ and $\\gclip = \\frac{L\\sqrt{T}}{\\sqrt{\\log (1/\\delta)}}$ produce iterates $x_0, \\ldots, x_{T-1}$ whose average $\\bar{x}_T = \\frac{1}{T} \\sum_{t=0}^{T-1} x_t$ has suboptimality $O\\prn*{\\frac{LR}{\\sqrt{T}}\\sqrt{\\log\\frac{1}{\\delta}}}$ with probability at least $1-\\delta$.\n\\end{customprop}\n\n\\begin{proof}\n    Let $F(x) = \\E_{S\\sim P} f(x;S)$ be the objective function and let $\\xopt$ be its minimizer in $\\xset'$ (and hence, by assumption, also in $\\xset$). The standard SGD regret bound, e.g., \\cite[Lemma 2.12.]{orabona2021modern}, gives\n    \\[\n    \\sum_{t=0}^{T-1} \\inner{\\gbar_t}{x_t - \\xopt} \\le \\frac{R^2}{2\\eta} + \\frac{\\eta}{2} \\sum_{t=0}^{T-1} \\norm{\\gbar_t}^2\n    \\]\n    with probability 1. \n    Writing $\\E_t Z = \\Ex*{Z | x_0, \\ldots, x_t}$ to streamline notation, \n    and using $\\gbar_t = \\E_t g_t - (\\E_t g_t - \\E_t \\gbar_t) - (\\E_t \\gbar_t - \\gbar_t)$ we rearrange the above display to read,\n    \\[\n        \\sum_{t=0}^{T-1} \\inner{\\E_t g_t}{x_t - \\xopt} \n        \\le \\frac{R^2}{2\\eta} \n        + \\sum_{t=0}^{T-1} \\inner{\\E_t g_t - \\E_t \\gbar_t}{x_t - \\xopt} \n        + \\sum_{t=0}^{T-1} \\inner{\\E_t \\gbar_t - \\gbar_t}{x_t - \\xopt}\n        + \\frac{\\eta}{2} \\sum_{t=0}^{T-1} \\norm{\\gbar_t}^2. \n        \\numberthis \\label{eq:clipped-sgd-regret-rearranged}\n    \\]\n    \n    Since $\\E_t g_t\\in\\del F(x_t)$, we have $\\inner{\\E_t g_t}{x_t - \\xopt} \\ge F(x_t) - F(\\xopt)$ and therefore, by Jensen's inequality we obtain the following lower bound on the LHS of \\Cref{eq:clipped-sgd-regret-rearranged}\n    \\[\n        \\sum_{t=0}^{T-1} \\inner{\\E_t g_t}{x_t - \\xopt}  \\ge T \\brk*{F(\\bar{x}_T) - F(\\xopt)}.\n        \\numberthis \\label{eq:clipped-sgd-bound-1}\n    \\]\n    We have $R^2 / (2 \\eta) = O(LR\\sqrt{T})$ by our choice of $\\eta$, and it remains to bound the three sums in the RHS of \\Cref{eq:clipped-sgd-regret-rearranged}.\n\n    To bound the first sum, we observe that\n    \\begin{flalign*}\n        \\norm{\\E_t g_t - \\E_t \\gbar_t} &\\overle{(i)} \\E_t \\norm{g_t - \\gbar_t} \\overeq{(ii)} \\E_t \\brk*{\\norm*{g_t} \\prn*{1 - \\frac{\\gclip}{\\| g_t \\|} } \\indic{\\norm{g_t} > \\gclip}} \n       \\le \n        \\E_t \\brk*{\\norm{g_t}\\indic{\\norm{g_t} > \\gclip}}  \\\\ &\n        \\le \\frac{1}{\\gclip} \\E_t \\norm{g_t}^2 \\overle{(iii)} \\frac{L^2}{\\gclip},\n    \\end{flalign*}\n    where $(i)$ is from Jensen's inequality applied to the Euclidean norm, $(ii)$ from the definition of $\\gbar_t$ and $(iii)$ is from the definition of $\\Iclass[\\SMLip]^{L, R}$. \n    Therefore, using the triangle inequality, Cauchy-Schwarz, the bound $\\norm{x_t-\\xopt}\\le 2R$, and our setting of $\\gclip$, we get that\n    \\[\n    \\abs*{\\sum_{t=0}^{T-1} \\inner{\\E_t \\gbar_t - \\E_t g_t}{x_t - \\xopt} } \\le \\sum_{t=0}^{T-1} \\norm{\\E_t g_t - \\E_t \\gbar_t} \\norm{x_t - \\xopt}  \\le \\frac{2L^2R T}{\\gclip} = 2LR \\sqrt{T\\log\\frac{1}{\\delta}}.\n    \\numberthis \\label{eq:clipped-sgd-bound-2}\n    \\]\n\n    To bound the second sum, we note that $\\E_t \\brk*{\\inner{\\gbar_t - \\E_t \\gbar_t}{x_t - \\xopt}} = 0$ for all $t$. Moreover, again by Cauchy-Schwarz and $\\norm{x_t-\\xopt}\\le 2R$, we have that \n    \\[\n        \\E_t \\brk*{\\inner{\\gbar_t - \\E_t \\gbar_t}{x_t - \\xopt}}^2\n        \\le 4R^2 \\E_t \\norm{\\gbar_t - \\E_t \\gbar_t}^2 \n        \\le 4R^2 \\E_t \\norm{\\gbar_t}^2 \n        \\le 4R^2 \\E_t \\norm{g_t}^2 \n        \\le 4L^2 R^2,\n    \\]\n    as well as\n    \\[\n        \\abs*{\\inner{\\gbar_t - \\E_t \\gbar_t}{x_t - \\xopt}}\n        \\le \n        \\norm{\\gbar_t - \\E_t \\gbar_t} \\norm{x_t - \\xopt}\n        \\le 4\\gclip R,\n    \\]\n    due to the gradient clipping.\n    Therefore, we may apply \\Cref{fact:freedman} on $X_t = \\inner{\\E_t\\gbar_t -  \\gbar_t}{x_t - \\xopt}$ to conclude that, with probability at least $1-\\frac{\\delta}{2}$,\n    \\[\n        \\sum_{t=0}^{T-1} \\inner{\\E_t \\gbar_t - \\gbar_t}{x_t - \\xopt}\n        \\le LR \\sqrt{8T \\log\\frac{2}{\\delta}} + \\frac{4}{3} \\gclip R \\log\\frac{2}{\\delta} = O\\prn*{LR \\sqrt{T \\log\\frac{1}{\\delta}} }.\n        \\numberthis \\label{eq:clipped-sgd-bound-3}\n    \\]\n\n    For the third and final sum, we observe that $\\E_t \\norm{\\gbar_t}^4 \\le \\gclip^2 \\E_t \\norm{\\gbar_t}^2 \\le (\\gclip L)^2$, and that $\\norm{\\gbar_t}^2 \\le \\gclip^2$ with probability 1. We then apply \\Cref{fact:freedman} again, this time with $X_t = \\norm{\\gbar_t}^2$ to obtain that, with probability at least $1-\\frac{\\delta}{2}$,\n    \\begin{flalign*}\n        \\frac{\\eta}{2} \\sum_{t=0}^{T-1} \\norm{\\gbar_t}^2 \n        & \\le \\frac{\\eta}{2} \\sum_{t=0}^{T-1} \\E_t\\norm{\\gbar_t}^2 \n        + \\frac{\\eta}{2} \\gclip L \\sqrt{ 2T \\log \\frac{1}{\\delta}} \n        + \\frac{\\eta}{6} \\gclip^2 \\log \\frac{1}{\\delta}\n        \\\\ &\n        \\overle{(i)} \\frac{\\eta}{2} L^2 T \\cdot \\prn*{1 + \\sqrt{2} + \\frac{1}{3} } \\overle{(ii)} O\\prn*{LR \\sqrt{T }},\n        \\numberthis \\label{eq:clipped-sgd-bound-4}\n    \\end{flalign*}\n    due to $(i)$ the choice $\\gclip$ and $\\E_t\\norm{\\gbar_t}^2 \\le \\E_t\\norm{g_t}^2 \\le L^2$, and $(ii)$ the choice of $\\eta$. \n\n    Substituting \\cref{eq:clipped-sgd-bound-1,eq:clipped-sgd-bound-2,eq:clipped-sgd-bound-3,eq:clipped-sgd-bound-4} into \\cref{eq:clipped-sgd-regret-rearranged} and using again the choice of $\\eta$ and a union bound over the two events holding with probability at least $1-\\frac{\\delta}{2}$, we conclude that $T \\brk*{F(\\bar{x}_T) - F(\\xopt)} \\le O\\prn*{LR \\sqrt{T \\log\\frac{1}{\\delta}} }$ with probability at least $1-\\delta$, leading to the claimed rate of convergence.\n\\end{proof}\n\n\\begin{customprop}{1b}\\label{prop:high-prop-lip-lb}\n    For all $L,R>0$ and $\\delta\\in (0,\\frac{1}{2})$ we have $\\ErrHP{\\Aall}{\\Iclass[\\Lip]^{L,R}} \\ge \n    \\Omega\\prn*{  \n        LR \\min\\crl*{ \\sqrt{\\frac{\\log\\frac{1}{\\delta}}{T}}, 1 }\n        }$.\n\\end{customprop}\n\n\\begin{proof}\n    We prove \\Cref{prop:high-prop-lip-lb} with a construction almost identical to the one in the proof of \\Cref{thm:log-PoA-lb}. \n    We let $\\xset=\\R$, $\\sset = \\{0,1\\}$ and $f:\\xset\\times\\sset\\to\\R$ be\n\t\\begin{equation*}\n\t\tf(x;s) = \\begin{cases}\n\t\t\tL|x| & s = 0 \\\\\n\t\t\tL|x-R| & s = 1,\\\\\n\t\t\\end{cases}\n\t\\end{equation*}\n\tand we set, assuming for now $\\delta \\le \\frac{1}{5}$,\n\t\\begin{equation*}\n\t\t\\eps \\defeq \\min\\crl*{\\sqrt{\\frac{\\log\\frac{1-2\\delta}{2\\delta}}{8T}}, \\frac{1}{2}}~~\\mbox{and}~~\\Pv \\defeq \\bernoulli\\prn*{\\frac{1+(2v-1)\\eps}{2}}.\n\t\\end{equation*}\n\tClearly, $f(\\cdot;s)$ is $L$-Lipschitz for both $s\\in\\{0,1\\}$ and moreover,\n    \\begin{flalign*}\n        F_{f,\\Pv}(x) - \\inf_{\\xopt\\in\\R} F_{f,\\Pv}(\\xopt) \\ge \n        \\begin{cases}\n        \\eps L |x+R| & v = 0 \\\\\n        \\eps L |x-R| & v = 1.\\\\\n        \\end{cases} \\numberthis \\label{eq:minimax-lb-error-expression}\n    \\end{flalign*} \n    Consequently,  $(f, \\Pv) \\in \\Iclass[\\Lip]^{L,R}$ for $v\\in \\{0,1\\}$.\n    \n    For any $\\alg\\in\\Aall$, let $x_T^v$ be the output of $\\alg$ when interacting with $(f,\\Pv)$ by observing $S_1,\\ldots,S_T\\simiid \\Pv$, and let $\\hat{v}(x) \\defeq \\indic{x>0}$. Substituting into~\\eqref{eq:minimax-lb-error-expression}, we have \n    \\[\n    \\ErrHP{\\alg}{(f,\\Pv)} \\ge \\eps LR \\indicb{ \\Pr(\\hat{v}\\prn*{x_T^v} \\ne v) \\ge \\delta }\n    ~~\n    \\mbox{for}\n    ~~\n    v\\in \\{0,1\\}.\n    \\]\n    Therefore, since $ \\max_{v\\in\\{0,1\\}}\\Pr(\\hat{v}\\prn*{x_T^v} \\ne v) \\ge \\E_{V\\sim \\bernoulli(2\\delta)}\\Pr(\\hat{v}\\prn*{x_T^V} \\ne V)$ we have\n    \\[\n    \\ErrHP{\\alg}{\\Iclass[\\Lip]^{L,R}} \\ge \\eps LR \\indicb{ \\max_{v\\in\\{0,1\\}}\\Pr(\\hat{v}\\prn*{x_T^v} \\ne v) \\ge \\delta } \n    \\ge \n    \\eps LR \\indicb{ \\E_{V\\sim \\bernoulli(2\\delta)}\\Pr(\\hat{v}\\prn*{x_T^V} \\ne V) \\ge \\delta }.\n    \\]\n    Let $\\hat{V} \\defeq \\hat{v}\\prn*{x_T^V}$ and note that it depends on $V$ only through $x_T^V$ and hence through $S_1,\\ldots,S_T\\simiid P_V$. By our choice of $\\eps$ and the current assumption that $2 \\delta < 2/5 \\le 1/2$, we may apply \\Cref{lem:skewed-binary-bound} with $p=2\\delta$ and conclude that $\\Pr(\\hat{V} \\ne V) \\ge \\delta$, implying\n    \\[\n        \\ErrHP{\\alg}{\\Iclass[\\Lip]^{L,R}} \\ge \\eps LR = \\Omega\\prn*{  \n            LR \\min\\crl*{ \\sqrt{\\frac{\\log\\frac{1}{\\delta}}{T}}, 1 }\n            }\n    \\]\n    for all $\\delta \\le \\frac{1}{5}$. \n\n    To extend the lower bound to $\\delta \\in (\\frac{1}{5}, \\frac{1}{2})$,\n    %\n    we only need to show that $\\ErrCustom{1/2}{\\alg}{\\Iclass[\\Lip]^{L,R}} = \\Omega(LR/\\sqrt{T})$. We do this with a simple ``anti-boosting'' argument. Let $k=4$ and consider $k$ copies of the construction above for $\\delta'=\\frac{1}{5}$, each operating on independent samples and independent coordinates, scaled so that the sum of the copies is in $\\Iclass[\\Lip]^{L,R}$. Concretely, the construction uses $\\xset=\\R^k$ and $\\sset=\\{0,1\\}^k$, and given $f:\\R\\times \\{0,1\\}\\to \\R$ described above, defines $\\tilde{f}: \\xset\\times\\sset\\to\\R$ such that\n    \\[\n        \\tilde{f}(x,s) = k^{-3/2}\\sum_{i \\in [k]} f(x_i \\sqrt{k};s_i).\n    \\]\n    For $v=(v_1,\\ldots,v_k)\\in\\{0,1\\}^k$ and $P=P_{v_1}\\times \\cdots \\times P_{v_k}$, this construction has optimality gap\n    \\[\n        F_{f,\\Pv}(x) - \\inf_{\\xopt\\in\\xset} F_{f,\\Pv}(\\xopt)\n        \\ge \\tfrac{1}{k^{3/2}}\\eps LR \\cdot  \\norm*{x \\sqrt{k} - (2v-1)R}_1.\n    \\]\n    Therefore, disagreeing with the sign of $\\xopt$ in even a single coordinate implies optimality gap of at least $k^{-3/2}\\eps LR = \\Omega(LR/\\sqrt{T})$ by our choice of $\\eps$. By the discussion above, the probability of mistaking at least one of the coordinates is at least $1-(1-\\delta')^k > \\half$ for $\\delta' = 1/5$ and $k=4$, completing the proof.\n\\end{proof}\n\n\\begin{proof}\n\t%\n\t%\n\t%\n\t%\n\t%\n\t %\n\tWe pick two elements from $\\Mclass[\\Lip]^{1,\\rho}$, namely $\\IclassMinus = \\Iclass[\\Lip]^{1, 1}$ and $\\IclassPlus = \\Iclass[\\Lip]^{1, \\rho}$. We then pick two instances $(f,\\Pminus)$ and $(f,\\Pplus)$ such that $(f, \\Pv) \\in \\IclassV$ for $v\\in \\{0,1\\}$. From \\cref{eq:PoA-lower-bounding-strategy}, the PoA is bounded from below by\n\t\\begin{equation*}\n\t\t\\PoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \n\t\t\\ge\n\t\t\\max_{v\\in\\{0,1\\}} \\frac{\\ErrE{\\alg}{(f,\\Pv)}}{\\ErrE{\\Aall}{\\IclassV}} \\ge \\half \\cdot \\frac{\\ErrE{\\alg}{(f,\\Pminus)}}{\\ErrE{\\Aall}{\\IclassMinus}}\n\t\t+  \\half \\cdot \\frac{\\ErrE{\\alg}{(f,\\Pplus)}}{\\ErrE{\\Aall}{\\IclassPlus}}\n\t\t.\n\t\\end{equation*}\n\tSubstituting $\\ErrE{\\Aall}{\\Iclass[\\Lip]^{L, R}} = O(L R / \\sqrt{T} )$ as per Lemma~\\ref{prop:standard-error-bounds}, we obtain\n\t\\begin{equation}\\label{eq:poa-as-expectation}\n\t\t\\PoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \\ge \\frac{\\sqrt{T}}{2} \\ErrE{\\alg}{(f,\\Pminus)} + \\frac{\\sqrt{T}}{2\\rho} \\ErrE{\\alg}{(f,\\Pplus)}.\n\t\\end{equation}\n\tIt remains to specify $f$, $\\Pminus$ and $\\Pplus$, argue that $(f, \\Pv) \\in \\IclassV$ for $v\\in \\{0,1\\}$ holds, and lower bound $\\ErrE{\\alg}{(f,\\Pv)}$. \n\tWe let $\\xset=\\R$, $\\sset = \\{0,1\\}$ and $f:\\xset\\times\\sset\\to\\R$ be\n\t\\begin{equation*}\n\t\tf(x;s) = \\begin{cases}\n\t\t\t|x| & s = 0 \\\\\n\t\t\t|x-\\rho| & s = 1.\\\\\n\t\t\\end{cases}\n\t\\end{equation*}\n\tFurthermore, we let \n\t\\begin{equation*}\n\t\t\\eps \\defeq \\min\\crl*{\\sqrt{\\frac{\\log\\rho}{8T}}, \\frac{1}{2}}~~\\mbox{and}~~\\Pv \\defeq \\bernoulli\\prn*{\\frac{1+(2v-1)\\eps}{2}}.\n\t\\end{equation*}\n\tClearly, $f(\\cdot;s)$ is 1-Lipschitz for both $s\\in\\{0,1\\}$ and moreover,\n%\n%\n%\n%\n%\n%\n%\n%\n\\begin{flalign}\n\tF_{f,\\Pv}(x) - \\inf_{\\xopt\\in\\R} F_{f,\\Pv}(\\xopt) \\ge \n\t\\begin{cases}\n\t\\eps |x| & v = 0 \\\\\n\t\\eps |x-\\rho| & v = 1.\\\\\n\t\\end{cases} \\label{eq:expectation-lb-error-expression}\n\t\\end{flalign} \n\tConsequently,  $(f, \\Pv) \\in \\IclassV$ for $v\\in \\{0,1\\}$ as required. \n\n\tLet $S_1, \\dots, S_T\\sim P_v$, yielding the  sample functions $f(\\cdot, S_1), \\dots, f(\\cdot, S_T)$.\n\tTo lower bound the error, let $x_T$ be the output of the algorithm after $T$ oracle queries. Using the expression~\\eqref{eq:expectation-lb-error-expression}, we have\n\t\\begin{equation*}\n\t\t\\ErrE{\\alg}{(f,\\Pminus)} \\ge \\frac{\\eps\\rho}{2} \\cdot \n\t\t%\n\t\t\\Pminus\\prn*{x_T \\ge \\frac{\\rho}{2}}\n\t\\quad \\text{and} \\quad \n\t\t\\ErrE{\\alg}{(f,\\Pplus)} \\ge \\frac{\\eps\\rho}{2} \\cdot \n\t\t%\n\t\t\\Pplus\\prn*{x_T < \\frac{\\rho}{2}}.\n\t\\end{equation*}\n\tTo translate these lower bounds into the language of hypothesis testing, let $\\hat{V} \\defeq \\indicb{x_T \\ge \\frac{\\rho}{2}}$\n\tand note that, for $v\\in\\{0,1\\}$,\n\t\\[\n\t\t%\n\t\t\\ErrE{\\alg}{(f,\\Pv)} \\ge \\frac{\\eps\\rho}{2} \\Pv(\\hat{V} \\ne v ).\n\t\t%\n\t\\]\n\tSubstituting into~\\eqref{eq:poa-as-expectation}, we have that\n\t%\n\t\t$\\bigPoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \\ge \\frac{\\eps\\sqrt{T}}{4}\\prn[\\big]{\n\t\t\t\\rho \\Pminus(\\hat{V} \\ne 0) + \\Pplus(\\hat{V} \\ne 1)\n\t\t}$.\n\t%\n\tWriting $p \\defeq \\frac{1}{1+\\rho}$ and letting $V\\sim \\bernoulli(p)$, we may rewrite the above PoA as\n\t\\[\n\t%\n\t\\PoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \\ge \\frac{\\eps\\sqrt{T}}{4p} \n\t\t\\prn[\\Big]{\n\t\t\t(1-p) \\Pminus(\\hat{V} \\ne 0) + p \\Pplus(\\hat{V} \\ne 1)\n\t\t}\n\t\t= \\frac{\\eps\\sqrt{T}}{4p} \\P(\\hat{V} \\ne V).\n\t%\n\t\\]\n\tNoting that $p\\le \\half$ since $\\rho \\ge 1$ and that $\\rho = \\frac{1-p}{p}$, we invoke \\Cref{lem:skewed-binary-bound} (in \\Cref{app:skewed-binary-bound}) to conclude that $\\P(\\hat{V} \\ne V) \\ge \\frac{p}{2}$ and therefore (recalling our choice of $\\veps$)\n\t$\\PoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \\ge \\frac{\\eps \\sqrt{T}}{8} \\ge \n\t\t\\frac{1}{32}\\min\\crl{\n\t\t\\sqrt{\\log\\rho}, \\sqrt{T}\n\t\t}$\n\tas required.\n\\end{proof}\n\nLet $\\sgd$ denote fixed-steps size SGD with step size $\\eta$ (outputting the iterate average). Then, for $(f,P) \\in \\Iclass[\\SMLip]^{L,R}$ it is well-known the iterates of SGD satisfy \\cite[see, e.g.,][Theorem 2.13]{orabona2021modern}:\n\\[\n\\frac{1}{T} \\sum_{t=0}^{T-1} \\Delta_t \\le \\frac{\\| x_0 - x_\\star \\|^2}{2 \\eta T} + \\frac{\\eta}{2 T} \\sum_{t=0}^{T-1} \\norm{g_t}^2\n\\] \nwhere $x_\\star$ is any optimal solution.\nBy online-to-batch conversion \\cite[Chapter 3]{orabona2021modern} we get\n\\[\n\\ErrE{\\sgd}{\\Iclass[\\SMLip]^{L,R}} \\le \\frac{R^2}{2\\eta T} + \\frac{\\eta L^2}{2}.\n\\numberthis \\label{eq:sgd-expected-err}\n\\]\n","units":[{"start":39460,"end":40368,"start_line":798,"end_line":816,"raw_tex":"\\begin{lem}\\label{lem:general-mi-bound}\n    For any $T\\in \\N$ and $\\eps \\in [0,1]$ and any random variable $V$, let $S_1,\\ldots, S_T$ be a sequence of binary random variables such that, for all $t\\le T$ \\yair{could also write it as a TV distance bound}\n    %\n    \\[\n        \\abs*{\\Pr(S_t = 0| S_1, \\ldots, S_{t-1}, V) - \\half} \\le \\frac{\\eps}{2}\n    \\]\n     with probability 1 w.r.t.\\ $S_1,\\ldots, S_{t-1}$ and $V$. Then, for every randomized estimator $\\hat{V}$ that depends on $V$ only through $S_1, \\ldots, S_T$, we have\n    \\[\n        I(V;\\hat{V}) \\le \\eps^2 T.\\numberthis \\label{eq:general-mi-bound-1}\n    \\]\n    If in addition $\\eps \\le \\half$ and, for some $p\\in [0,\\half]$, we have that\n    \\[\n        \\abs*{\\Pr(S_t = 0) - \\half} \\ge \\eps \\prn*{\\frac{1}{2} - p}\n    \\]\n    for every $t\\le T$, then\n    \\[\n        I(V;\\hat{V}) \\le 4p \\eps^2 T.\\numberthis \\label{eq:general-mi-bound-2}\n    \\]\n\\end{lem}","sha256_utf8":"c870125d66ba52735ad04a246b1e548ac3bec6887bc24ddf23d4af44ed5d6a14"},{"start":40370,"end":42907,"start_line":818,"end_line":862,"raw_tex":"\\begin{proof}\n    By the data processing inequality \\citep[Theorem 2.8.1]{cover1991elements} and chain rule for mutual information \\cite[Theorem 2.5.2]{cover1991elements}, \n    %\n\t\\begin{flalign*}\n\t\tI(\\hat{V}; V) & \\le I(S_1, \\ldots, S_T; V) \n\t\t= \\sum_{t=1}^{T} \\brk[\\Big]{ H(S_t \\mid S_1, \\ldots, S_{t-1}) - H(S_t \\mid S_1, \\ldots, S_{t-1}, V) }. \\numberthis \\label{eq:log-mi-expression}\n\t\\end{flalign*}\n\tUsing the facts conditioning decreases entropy and that binary random variables have entropy at most $\\log 2$, we get\n    \\[\n\t\tH(S_t \\mid S_1, \\ldots, S_{t-1}) \\le H(S_t) \\le \\log 2.\n\t\\]\n    Next, we have \n    \\[\n\t\tH(S_t \\mid S_1, \\ldots, S_{t-1}, V) = \\E_{S_1, \\ldots, S_{t-1}, V} \\brk*{h_2 \\prn[\\Big]{\\Pr(S_t = 0| S_1, \\ldots, S_{t-1}, V) }}\n         \\ge  h_2\\prn*{\\frac{1-\\eps}{2}},\n    \\]\n    by the assumption that $\\abs*{\\Pr(S_t = 0| S_1, \\ldots, S_{t-1}, V) - \\half} \\le \\frac{\\eps}{2}$ with probability 1 and the fact that $h_2(x)$ has a single maximum at $x=\\half$.\n\n    Combining the last two displays and using $\\log(1+x)\\le x$, we obtain\n    \\begin{flalign*}\n        H(S_t \\mid S_1, \\ldots, S_{t-1}) &- H(S_t \\mid S_1, \\ldots, S_{t-1}, V) \\\\& \n        \\le \\log 2 - h_2\\prn*{\\frac{1-\\eps}{2}}\n        \\\\ &=\n\t\t\\frac{1+\\veps}{2} \\log(1+\\veps) + \\frac{1-\\veps}{2} \\log(1-\\veps) \n        \\le \n\t\t\\veps^2,\n    \\end{flalign*}\n    where the last inequality is due to $\\log(1+x)\\le x$.\n    Substituting back into~\\cref{eq:log-mi-expression} yields the first claimed bound, $I(\\hat{V}; V) \\le \\eps^2 T$. \n\n    To obtain the second claimed bound we use the assumption that $\\abs*{\\Pr(S_t = 0) - \\half} \\ge \\eps \\prn*{\\frac{1}{2} - p}$ to write\n    \\[\n\t\tH(S_t \\mid S_1, \\ldots, S_{t-1}) \\le H(S_t) \n        \\le h_2\\prn*{\\frac{1-(1-2p)\\eps}{2}},\n    \\]\n    improving the bound on $H(S_t \\mid S_1, \\ldots, S_{t-1}) - H(S_t \\mid S_1, \\ldots, S_{t-1}, V)$ to \n    \\begin{flalign*}\n        h_2\\prn*{\\frac{1-(1-2p)\\eps}{2}} - h_2\\prn*{\\frac{1-\\eps}{2}}\n         & \\overle{(i)} \n        \\prn*{\\frac{1-(1-2p)\\eps}{2} - \\frac{1-\\eps}{2}} h_2'\\prn*{\\frac{1-\\eps}{2}}\n        \\\\ & \\overeq{(ii)} \n        p \\eps \\log\\prn*{1+\\frac{2\\eps}{1-\\eps}} \\overle{(iii)} \\frac{2p\\eps^2}{1-\\eps} \\overle{(iv)} 4p\\eps^2,\n    \\end{flalign*}\n    due to $(i)$ concavity of binary entropy, $(ii)$ the fact that $h_2'(x) = \\log\\prn*{\\frac{1-x}{x}}$, $(iii)$ the inequality $\\log(1+x)\\le x$ and $(iv)$ our assumption that $\\veps \\le \\half$. Substituting back into~\\eqref{eq:log-mi-expression}, we get that $I(\\hat{V}; V) \\le 4p\\eps^2 T$, as required.\n\\end{proof}","sha256_utf8":"0a404dbb60a6dc759d96d65c11c8d5eec29a0104d5c1fa1fc9cb391d649e38cb"},{"start":43002,"end":43495,"start_line":866,"end_line":878,"raw_tex":"\\begin{lem}\\label{lem:skewed-binary-bound}\n\tFor any $p\\in(0,\\frac{1}{2}]$ and $T\\in \\N$, let $V\\sim \\bernoulli(p)$ and\n\t%\n\t%\n\t%\n\t\\[\n\t\\eps = \\min\\crl*{\\sqrt{\\frac{\\log \\frac{1-p}{p}}{8T}}, \\half}.\t\n\t\\]\n\tLet $P_0 = \\bernoulli(\\frac{1-\\eps}{2})$ and $P_1 = \\bernoulli(\\frac{1+\\eps}{2})$. Then, for $S_1, \\ldots, S_T \\simiid P_V$ and (potentially randomized) estimator $\\hat{V}$ that depends on $V$ only through $S_1, \\ldots, S_T$, we have %\n\t\\[\n\t\t\\Pr(\\hat{V} \\ne V) \\ge \\frac{p}{2}.\n\t\\]\n\\end{lem}","sha256_utf8":"937913e8d3aae8ca814564552748eebe478e13538dfd8800338f18db5058678b"},{"start":43496,"end":44330,"start_line":879,"end_line":897,"raw_tex":"\\begin{proof}\n\tTo streamline notation, we write \n\t\\[\n\t\t%\n\t\t%\n\t\tq \\defeq \\Pr(\\hat{V} \\ne V). \n\t\\] \n\tBy Fano's inequality (\\Cref{fact:fano}) with $n=2$,\n\t\\[\n\t\th_2(q) \\ge H(V\\mid \\hat{V}) =  H(V) - I(V;\\hat{V}).\t\n\t\\numberthis \\label{eq:skewed-fano}\n\t\\]\n\tby \\cref{eq:general-mi-bound-2} in \\Cref{lem:general-mi-bound}, \n    \\[\n\t\tI(\\hat{V}; V) \\le 4p\\eps^2 T \\overle{(i)} \\frac{p}{2} \\log\\frac{1-p}{p} \\overeq{(ii)} \\prn*{p - \\frac{p}{2}}h_2'(p) \\overle{(iii)} h_2(p) - h_2(p/2), \n\t\t%\n\t\\]\n\tdue to $(i)$ our choice of $\\eps \\le \\sqrt{\\frac{\\log \\frac{1-p}{p}}{8T}}$, (ii) the fact that $h_2'(x) = \\log\\frac{1-x}{x}$ and $(iii)$ concavity of entropy. Noting that $H(V) = h_2(p)$ and substituting back to~\\eqref{eq:skewed-fano}, we get that $h_2(q) \\ge h_2(p/2)$, which implies $q\\ge p/2$ since $h_2$ is monotone in $[0, \\half]$. \n\\end{proof}","sha256_utf8":"5a4fd246e85e2a50897f6b3d179168662d6964cb3806522d1bd92a5fa83bbac4"},{"start":50394,"end":50860,"start_line":1026,"end_line":1030,"raw_tex":"\\begin{restatable}{prop}{restatePropStandardErrorBounds}\\label{prop:standard-error-bounds}\nFor all $R, L > 0$, instance class $\\Iclass \\in \\{ \\Iclass[\\Lip]^{L,R}, \\Iclass[\\SMLip]^{L,R} \\}$ and \n$\\Aclass \\in \\{ \\Aall, \\AFO \\}$, we have \n$\\ErrE{\\Aclass}{\\Iclass}=\\Theta\\left( \\frac{LR}{\\sqrt{T} } \\right)$. Moreover, for any $\\delta\\in(0,\\half)$ we have $\\ErrHP{\\Aclass}{\\Iclass}=\\Theta\\left( \\frac{LR}{\\sqrt{1+ {T}/{\\log \\frac{1}{\\delta}}}} \\right)$.\n\\end{restatable}","sha256_utf8":"be047a1b983dbaa352a3df2ab77623308f9d814ebdb902d53ee768a0633025a9"},{"start":50862,"end":51660,"start_line":1032,"end_line":1038,"raw_tex":"\\begin{proof}\nSince $\\Iclass[\\Lip]^{L,R}\\subset \\Iclass[\\SMLip]^{L,R}$ and $\\AFO\\subset\\Aall$, we only need to prove minimax error lower bounds on $(\\Aall,\\Iclass[\\Lip]^{L,R})$ and minimax upper bounds on $(\\AFO, \\Iclass[\\SMLip]^{L,R})$. \n\nFor expected error these results are well-known. The upper bound on $\\ErrE{\\AFO}{ \\Iclass[\\SMLip]^{L,R}}$ follows, for example, by setting $\\eta = \\frac{R}{L\\sqrt{T}}$ in \\cref{eq:sgd-expected-err}. For the lower bound on $\\ErrE{\\Aall}{ \\Iclass[\\Lip]^{L,R}}$, originally due to~\\citet{nemirovski1983problem}, see, e.g., \\citet[Theorem 5.2.10.]{duchi2018introductory}.\n\nFor error quantiles, however, we could not find direct proof of the required upper and lower bounds, and we prove them in \\Cref{app:high-prob-ms-lip,app:high-prop-lip-lb} below.\n\\end{proof}","sha256_utf8":"8b1ec9c9354e24906767a152c691cd72d7e9e71c54d38d9d93bcb318cd18176c"},{"start":54056,"end":54593,"start_line":1071,"end_line":1073,"raw_tex":"\\begin{customprop}{1a}\\label{prop:ms-lip-hp-bound}\n    For any $\\delta \\in (0,\\half)$, any $L,R>0$, and any instance in $\\Iclass[\\SMLip]^{L, R}$, we have that $T$ steps of clipped SGD~\\eqref{eq:clipped-SGD} with step size $\\eta = \\frac{R}{L\\sqrt{T}}$ and $\\gclip = \\frac{L\\sqrt{T}}{\\sqrt{\\log (1/\\delta)}}$ produce iterates $x_0, \\ldots, x_{T-1}$ whose average $\\bar{x}_T = \\frac{1}{T} \\sum_{t=0}^{T-1} x_t$ has suboptimality $O\\prn*{\\frac{LR}{\\sqrt{T}}\\sqrt{\\log\\frac{1}{\\delta}}}$ with probability at least $1-\\delta$.\n\\end{customprop}","sha256_utf8":"f2254604f6b2c068b8e75a37c403c1642721ce6117993cf8dfb53fadeaddb515"},{"start":54595,"end":59649,"start_line":1075,"end_line":1149,"raw_tex":"\\begin{proof}\n    Let $F(x) = \\E_{S\\sim P} f(x;S)$ be the objective function and let $\\xopt$ be its minimizer in $\\xset'$ (and hence, by assumption, also in $\\xset$). The standard SGD regret bound, e.g., \\cite[Lemma 2.12.]{orabona2021modern}, gives\n    \\[\n    \\sum_{t=0}^{T-1} \\inner{\\gbar_t}{x_t - \\xopt} \\le \\frac{R^2}{2\\eta} + \\frac{\\eta}{2} \\sum_{t=0}^{T-1} \\norm{\\gbar_t}^2\n    \\]\n    with probability 1. \n    Writing $\\E_t Z = \\Ex*{Z | x_0, \\ldots, x_t}$ to streamline notation, \n    and using $\\gbar_t = \\E_t g_t - (\\E_t g_t - \\E_t \\gbar_t) - (\\E_t \\gbar_t - \\gbar_t)$ we rearrange the above display to read,\n    \\[\n        \\sum_{t=0}^{T-1} \\inner{\\E_t g_t}{x_t - \\xopt} \n        \\le \\frac{R^2}{2\\eta} \n        + \\sum_{t=0}^{T-1} \\inner{\\E_t g_t - \\E_t \\gbar_t}{x_t - \\xopt} \n        + \\sum_{t=0}^{T-1} \\inner{\\E_t \\gbar_t - \\gbar_t}{x_t - \\xopt}\n        + \\frac{\\eta}{2} \\sum_{t=0}^{T-1} \\norm{\\gbar_t}^2. \n        \\numberthis \\label{eq:clipped-sgd-regret-rearranged}\n    \\]\n    \n    Since $\\E_t g_t\\in\\del F(x_t)$, we have $\\inner{\\E_t g_t}{x_t - \\xopt} \\ge F(x_t) - F(\\xopt)$ and therefore, by Jensen's inequality we obtain the following lower bound on the LHS of \\Cref{eq:clipped-sgd-regret-rearranged}\n    \\[\n        \\sum_{t=0}^{T-1} \\inner{\\E_t g_t}{x_t - \\xopt}  \\ge T \\brk*{F(\\bar{x}_T) - F(\\xopt)}.\n        \\numberthis \\label{eq:clipped-sgd-bound-1}\n    \\]\n    We have $R^2 / (2 \\eta) = O(LR\\sqrt{T})$ by our choice of $\\eta$, and it remains to bound the three sums in the RHS of \\Cref{eq:clipped-sgd-regret-rearranged}.\n\n    To bound the first sum, we observe that\n    \\begin{flalign*}\n        \\norm{\\E_t g_t - \\E_t \\gbar_t} &\\overle{(i)} \\E_t \\norm{g_t - \\gbar_t} \\overeq{(ii)} \\E_t \\brk*{\\norm*{g_t} \\prn*{1 - \\frac{\\gclip}{\\| g_t \\|} } \\indic{\\norm{g_t} > \\gclip}} \n       \\le \n        \\E_t \\brk*{\\norm{g_t}\\indic{\\norm{g_t} > \\gclip}}  \\\\ &\n        \\le \\frac{1}{\\gclip} \\E_t \\norm{g_t}^2 \\overle{(iii)} \\frac{L^2}{\\gclip},\n    \\end{flalign*}\n    where $(i)$ is from Jensen's inequality applied to the Euclidean norm, $(ii)$ from the definition of $\\gbar_t$ and $(iii)$ is from the definition of $\\Iclass[\\SMLip]^{L, R}$. \n    Therefore, using the triangle inequality, Cauchy-Schwarz, the bound $\\norm{x_t-\\xopt}\\le 2R$, and our setting of $\\gclip$, we get that\n    \\[\n    \\abs*{\\sum_{t=0}^{T-1} \\inner{\\E_t \\gbar_t - \\E_t g_t}{x_t - \\xopt} } \\le \\sum_{t=0}^{T-1} \\norm{\\E_t g_t - \\E_t \\gbar_t} \\norm{x_t - \\xopt}  \\le \\frac{2L^2R T}{\\gclip} = 2LR \\sqrt{T\\log\\frac{1}{\\delta}}.\n    \\numberthis \\label{eq:clipped-sgd-bound-2}\n    \\]\n\n    To bound the second sum, we note that $\\E_t \\brk*{\\inner{\\gbar_t - \\E_t \\gbar_t}{x_t - \\xopt}} = 0$ for all $t$. Moreover, again by Cauchy-Schwarz and $\\norm{x_t-\\xopt}\\le 2R$, we have that \n    \\[\n        \\E_t \\brk*{\\inner{\\gbar_t - \\E_t \\gbar_t}{x_t - \\xopt}}^2\n        \\le 4R^2 \\E_t \\norm{\\gbar_t - \\E_t \\gbar_t}^2 \n        \\le 4R^2 \\E_t \\norm{\\gbar_t}^2 \n        \\le 4R^2 \\E_t \\norm{g_t}^2 \n        \\le 4L^2 R^2,\n    \\]\n    as well as\n    \\[\n        \\abs*{\\inner{\\gbar_t - \\E_t \\gbar_t}{x_t - \\xopt}}\n        \\le \n        \\norm{\\gbar_t - \\E_t \\gbar_t} \\norm{x_t - \\xopt}\n        \\le 4\\gclip R,\n    \\]\n    due to the gradient clipping.\n    Therefore, we may apply \\Cref{fact:freedman} on $X_t = \\inner{\\E_t\\gbar_t -  \\gbar_t}{x_t - \\xopt}$ to conclude that, with probability at least $1-\\frac{\\delta}{2}$,\n    \\[\n        \\sum_{t=0}^{T-1} \\inner{\\E_t \\gbar_t - \\gbar_t}{x_t - \\xopt}\n        \\le LR \\sqrt{8T \\log\\frac{2}{\\delta}} + \\frac{4}{3} \\gclip R \\log\\frac{2}{\\delta} = O\\prn*{LR \\sqrt{T \\log\\frac{1}{\\delta}} }.\n        \\numberthis \\label{eq:clipped-sgd-bound-3}\n    \\]\n\n    For the third and final sum, we observe that $\\E_t \\norm{\\gbar_t}^4 \\le \\gclip^2 \\E_t \\norm{\\gbar_t}^2 \\le (\\gclip L)^2$, and that $\\norm{\\gbar_t}^2 \\le \\gclip^2$ with probability 1. We then apply \\Cref{fact:freedman} again, this time with $X_t = \\norm{\\gbar_t}^2$ to obtain that, with probability at least $1-\\frac{\\delta}{2}$,\n    \\begin{flalign*}\n        \\frac{\\eta}{2} \\sum_{t=0}^{T-1} \\norm{\\gbar_t}^2 \n        & \\le \\frac{\\eta}{2} \\sum_{t=0}^{T-1} \\E_t\\norm{\\gbar_t}^2 \n        + \\frac{\\eta}{2} \\gclip L \\sqrt{ 2T \\log \\frac{1}{\\delta}} \n        + \\frac{\\eta}{6} \\gclip^2 \\log \\frac{1}{\\delta}\n        \\\\ &\n        \\overle{(i)} \\frac{\\eta}{2} L^2 T \\cdot \\prn*{1 + \\sqrt{2} + \\frac{1}{3} } \\overle{(ii)} O\\prn*{LR \\sqrt{T }},\n        \\numberthis \\label{eq:clipped-sgd-bound-4}\n    \\end{flalign*}\n    due to $(i)$ the choice $\\gclip$ and $\\E_t\\norm{\\gbar_t}^2 \\le \\E_t\\norm{g_t}^2 \\le L^2$, and $(ii)$ the choice of $\\eta$. \n\n    Substituting \\cref{eq:clipped-sgd-bound-1,eq:clipped-sgd-bound-2,eq:clipped-sgd-bound-3,eq:clipped-sgd-bound-4} into \\cref{eq:clipped-sgd-regret-rearranged} and using again the choice of $\\eta$ and a union bound over the two events holding with probability at least $1-\\frac{\\delta}{2}$, we conclude that $T \\brk*{F(\\bar{x}_T) - F(\\xopt)} \\le O\\prn*{LR \\sqrt{T \\log\\frac{1}{\\delta}} }$ with probability at least $1-\\delta$, leading to the claimed rate of convergence.\n\\end{proof}","sha256_utf8":"85a0dd63527df459c956708407963192c1d21121f42d16e7caedee1dfc2674c5"},{"start":59729,"end":59994,"start_line":1152,"end_line":1157,"raw_tex":"\\begin{customprop}{1b}\\label{prop:high-prop-lip-lb}\n    For all $L,R>0$ and $\\delta\\in (0,\\frac{1}{2})$ we have $\\ErrHP{\\Aall}{\\Iclass[\\Lip]^{L,R}} \\ge \n    \\Omega\\prn*{  \n        LR \\min\\crl*{ \\sqrt{\\frac{\\log\\frac{1}{\\delta}}{T}}, 1 }\n        }$.\n\\end{customprop}","sha256_utf8":"67b8ef2eefb18ae2a892ff61b3d9890f7d8fc6293e6dfeb3a7a95a9f8c8e59e3"},{"start":59996,"end":63686,"start_line":1159,"end_line":1216,"raw_tex":"\\begin{proof}\n    We prove \\Cref{prop:high-prop-lip-lb} with a construction almost identical to the one in the proof of \\Cref{thm:log-PoA-lb}. \n    We let $\\xset=\\R$, $\\sset = \\{0,1\\}$ and $f:\\xset\\times\\sset\\to\\R$ be\n\t\\begin{equation*}\n\t\tf(x;s) = \\begin{cases}\n\t\t\tL|x| & s = 0 \\\\\n\t\t\tL|x-R| & s = 1,\\\\\n\t\t\\end{cases}\n\t\\end{equation*}\n\tand we set, assuming for now $\\delta \\le \\frac{1}{5}$,\n\t\\begin{equation*}\n\t\t\\eps \\defeq \\min\\crl*{\\sqrt{\\frac{\\log\\frac{1-2\\delta}{2\\delta}}{8T}}, \\frac{1}{2}}~~\\mbox{and}~~\\Pv \\defeq \\bernoulli\\prn*{\\frac{1+(2v-1)\\eps}{2}}.\n\t\\end{equation*}\n\tClearly, $f(\\cdot;s)$ is $L$-Lipschitz for both $s\\in\\{0,1\\}$ and moreover,\n    \\begin{flalign*}\n        F_{f,\\Pv}(x) - \\inf_{\\xopt\\in\\R} F_{f,\\Pv}(\\xopt) \\ge \n        \\begin{cases}\n        \\eps L |x+R| & v = 0 \\\\\n        \\eps L |x-R| & v = 1.\\\\\n        \\end{cases} \\numberthis \\label{eq:minimax-lb-error-expression}\n    \\end{flalign*} \n    Consequently,  $(f, \\Pv) \\in \\Iclass[\\Lip]^{L,R}$ for $v\\in \\{0,1\\}$.\n    \n    For any $\\alg\\in\\Aall$, let $x_T^v$ be the output of $\\alg$ when interacting with $(f,\\Pv)$ by observing $S_1,\\ldots,S_T\\simiid \\Pv$, and let $\\hat{v}(x) \\defeq \\indic{x>0}$. Substituting into~\\eqref{eq:minimax-lb-error-expression}, we have \n    \\[\n    \\ErrHP{\\alg}{(f,\\Pv)} \\ge \\eps LR \\indicb{ \\Pr(\\hat{v}\\prn*{x_T^v} \\ne v) \\ge \\delta }\n    ~~\n    \\mbox{for}\n    ~~\n    v\\in \\{0,1\\}.\n    \\]\n    Therefore, since $ \\max_{v\\in\\{0,1\\}}\\Pr(\\hat{v}\\prn*{x_T^v} \\ne v) \\ge \\E_{V\\sim \\bernoulli(2\\delta)}\\Pr(\\hat{v}\\prn*{x_T^V} \\ne V)$ we have\n    \\[\n    \\ErrHP{\\alg}{\\Iclass[\\Lip]^{L,R}} \\ge \\eps LR \\indicb{ \\max_{v\\in\\{0,1\\}}\\Pr(\\hat{v}\\prn*{x_T^v} \\ne v) \\ge \\delta } \n    \\ge \n    \\eps LR \\indicb{ \\E_{V\\sim \\bernoulli(2\\delta)}\\Pr(\\hat{v}\\prn*{x_T^V} \\ne V) \\ge \\delta }.\n    \\]\n    Let $\\hat{V} \\defeq \\hat{v}\\prn*{x_T^V}$ and note that it depends on $V$ only through $x_T^V$ and hence through $S_1,\\ldots,S_T\\simiid P_V$. By our choice of $\\eps$ and the current assumption that $2 \\delta < 2/5 \\le 1/2$, we may apply \\Cref{lem:skewed-binary-bound} with $p=2\\delta$ and conclude that $\\Pr(\\hat{V} \\ne V) \\ge \\delta$, implying\n    \\[\n        \\ErrHP{\\alg}{\\Iclass[\\Lip]^{L,R}} \\ge \\eps LR = \\Omega\\prn*{  \n            LR \\min\\crl*{ \\sqrt{\\frac{\\log\\frac{1}{\\delta}}{T}}, 1 }\n            }\n    \\]\n    for all $\\delta \\le \\frac{1}{5}$. \n\n    To extend the lower bound to $\\delta \\in (\\frac{1}{5}, \\frac{1}{2})$,\n    %\n    we only need to show that $\\ErrCustom{1/2}{\\alg}{\\Iclass[\\Lip]^{L,R}} = \\Omega(LR/\\sqrt{T})$. We do this with a simple ``anti-boosting'' argument. Let $k=4$ and consider $k$ copies of the construction above for $\\delta'=\\frac{1}{5}$, each operating on independent samples and independent coordinates, scaled so that the sum of the copies is in $\\Iclass[\\Lip]^{L,R}$. Concretely, the construction uses $\\xset=\\R^k$ and $\\sset=\\{0,1\\}^k$, and given $f:\\R\\times \\{0,1\\}\\to \\R$ described above, defines $\\tilde{f}: \\xset\\times\\sset\\to\\R$ such that\n    \\[\n        \\tilde{f}(x,s) = k^{-3/2}\\sum_{i \\in [k]} f(x_i \\sqrt{k};s_i).\n    \\]\n    For $v=(v_1,\\ldots,v_k)\\in\\{0,1\\}^k$ and $P=P_{v_1}\\times \\cdots \\times P_{v_k}$, this construction has optimality gap\n    \\[\n        F_{f,\\Pv}(x) - \\inf_{\\xopt\\in\\xset} F_{f,\\Pv}(\\xopt)\n        \\ge \\tfrac{1}{k^{3/2}}\\eps LR \\cdot  \\norm*{x \\sqrt{k} - (2v-1)R}_1.\n    \\]\n    Therefore, disagreeing with the sign of $\\xopt$ in even a single coordinate implies optimality gap of at least $k^{-3/2}\\eps LR = \\Omega(LR/\\sqrt{T})$ by our choice of $\\eps$. By the discussion above, the probability of mistaking at least one of the coordinates is at least $1-(1-\\delta')^k > \\half$ for $\\delta' = 1/5$ and $k=4$, completing the proof.\n\\end{proof}","sha256_utf8":"1f4751560b630c52c18c5a92b8781f0afcb13a0bca83b9e2ad30ca0b7b826f92"},{"start":68740,"end":72340,"start_line":1296,"end_line":1385,"raw_tex":"\\begin{proof}\n\t%\n\t%\n\t%\n\t%\n\t%\n\t %\n\tWe pick two elements from $\\Mclass[\\Lip]^{1,\\rho}$, namely $\\IclassMinus = \\Iclass[\\Lip]^{1, 1}$ and $\\IclassPlus = \\Iclass[\\Lip]^{1, \\rho}$. We then pick two instances $(f,\\Pminus)$ and $(f,\\Pplus)$ such that $(f, \\Pv) \\in \\IclassV$ for $v\\in \\{0,1\\}$. From \\cref{eq:PoA-lower-bounding-strategy}, the PoA is bounded from below by\n\t\\begin{equation*}\n\t\t\\PoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \n\t\t\\ge\n\t\t\\max_{v\\in\\{0,1\\}} \\frac{\\ErrE{\\alg}{(f,\\Pv)}}{\\ErrE{\\Aall}{\\IclassV}} \\ge \\half \\cdot \\frac{\\ErrE{\\alg}{(f,\\Pminus)}}{\\ErrE{\\Aall}{\\IclassMinus}}\n\t\t+  \\half \\cdot \\frac{\\ErrE{\\alg}{(f,\\Pplus)}}{\\ErrE{\\Aall}{\\IclassPlus}}\n\t\t.\n\t\\end{equation*}\n\tSubstituting $\\ErrE{\\Aall}{\\Iclass[\\Lip]^{L, R}} = O(L R / \\sqrt{T} )$ as per Lemma~\\ref{prop:standard-error-bounds}, we obtain\n\t\\begin{equation}\\label{eq:poa-as-expectation}\n\t\t\\PoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \\ge \\frac{\\sqrt{T}}{2} \\ErrE{\\alg}{(f,\\Pminus)} + \\frac{\\sqrt{T}}{2\\rho} \\ErrE{\\alg}{(f,\\Pplus)}.\n\t\\end{equation}\n\tIt remains to specify $f$, $\\Pminus$ and $\\Pplus$, argue that $(f, \\Pv) \\in \\IclassV$ for $v\\in \\{0,1\\}$ holds, and lower bound $\\ErrE{\\alg}{(f,\\Pv)}$. \n\tWe let $\\xset=\\R$, $\\sset = \\{0,1\\}$ and $f:\\xset\\times\\sset\\to\\R$ be\n\t\\begin{equation*}\n\t\tf(x;s) = \\begin{cases}\n\t\t\t|x| & s = 0 \\\\\n\t\t\t|x-\\rho| & s = 1.\\\\\n\t\t\\end{cases}\n\t\\end{equation*}\n\tFurthermore, we let \n\t\\begin{equation*}\n\t\t\\eps \\defeq \\min\\crl*{\\sqrt{\\frac{\\log\\rho}{8T}}, \\frac{1}{2}}~~\\mbox{and}~~\\Pv \\defeq \\bernoulli\\prn*{\\frac{1+(2v-1)\\eps}{2}}.\n\t\\end{equation*}\n\tClearly, $f(\\cdot;s)$ is 1-Lipschitz for both $s\\in\\{0,1\\}$ and moreover,\n%\n%\n%\n%\n%\n%\n%\n%\n\\begin{flalign}\n\tF_{f,\\Pv}(x) - \\inf_{\\xopt\\in\\R} F_{f,\\Pv}(\\xopt) \\ge \n\t\\begin{cases}\n\t\\eps |x| & v = 0 \\\\\n\t\\eps |x-\\rho| & v = 1.\\\\\n\t\\end{cases} \\label{eq:expectation-lb-error-expression}\n\t\\end{flalign} \n\tConsequently,  $(f, \\Pv) \\in \\IclassV$ for $v\\in \\{0,1\\}$ as required. \n\n\tLet $S_1, \\dots, S_T\\sim P_v$, yielding the  sample functions $f(\\cdot, S_1), \\dots, f(\\cdot, S_T)$.\n\tTo lower bound the error, let $x_T$ be the output of the algorithm after $T$ oracle queries. Using the expression~\\eqref{eq:expectation-lb-error-expression}, we have\n\t\\begin{equation*}\n\t\t\\ErrE{\\alg}{(f,\\Pminus)} \\ge \\frac{\\eps\\rho}{2} \\cdot \n\t\t%\n\t\t\\Pminus\\prn*{x_T \\ge \\frac{\\rho}{2}}\n\t\\quad \\text{and} \\quad \n\t\t\\ErrE{\\alg}{(f,\\Pplus)} \\ge \\frac{\\eps\\rho}{2} \\cdot \n\t\t%\n\t\t\\Pplus\\prn*{x_T < \\frac{\\rho}{2}}.\n\t\\end{equation*}\n\tTo translate these lower bounds into the language of hypothesis testing, let $\\hat{V} \\defeq \\indicb{x_T \\ge \\frac{\\rho}{2}}$\n\tand note that, for $v\\in\\{0,1\\}$,\n\t\\[\n\t\t%\n\t\t\\ErrE{\\alg}{(f,\\Pv)} \\ge \\frac{\\eps\\rho}{2} \\Pv(\\hat{V} \\ne v ).\n\t\t%\n\t\\]\n\tSubstituting into~\\eqref{eq:poa-as-expectation}, we have that\n\t%\n\t\t$\\bigPoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \\ge \\frac{\\eps\\sqrt{T}}{4}\\prn[\\big]{\n\t\t\t\\rho \\Pminus(\\hat{V} \\ne 0) + \\Pplus(\\hat{V} \\ne 1)\n\t\t}$.\n\t%\n\tWriting $p \\defeq \\frac{1}{1+\\rho}$ and letting $V\\sim \\bernoulli(p)$, we may rewrite the above PoA as\n\t\\[\n\t%\n\t\\PoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \\ge \\frac{\\eps\\sqrt{T}}{4p} \n\t\t\\prn[\\Big]{\n\t\t\t(1-p) \\Pminus(\\hat{V} \\ne 0) + p \\Pplus(\\hat{V} \\ne 1)\n\t\t}\n\t\t= \\frac{\\eps\\sqrt{T}}{4p} \\P(\\hat{V} \\ne V).\n\t%\n\t\\]\n\tNoting that $p\\le \\half$ since $\\rho \\ge 1$ and that $\\rho = \\frac{1-p}{p}$, we invoke \\Cref{lem:skewed-binary-bound} (in \\Cref{app:skewed-binary-bound}) to conclude that $\\P(\\hat{V} \\ne V) \\ge \\frac{p}{2}$ and therefore (recalling our choice of $\\veps$)\n\t$\\PoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \\ge \\frac{\\eps \\sqrt{T}}{8} \\ge \n\t\t\\frac{1}{32}\\min\\crl{\n\t\t\\sqrt{\\log\\rho}, \\sqrt{T}\n\t\t}$\n\tas required.\n\\end{proof}","sha256_utf8":"f44c7ff840a45ac93ce6dc74b008ed3950fbaccf63b76cedaf71946cf959691a"},{"start":102360,"end":102974,"start_line":2017,"end_line":2026,"raw_tex":"Let $\\sgd$ denote fixed-steps size SGD with step size $\\eta$ (outputting the iterate average). Then, for $(f,P) \\in \\Iclass[\\SMLip]^{L,R}$ it is well-known the iterates of SGD satisfy \\cite[see, e.g.,][Theorem 2.13]{orabona2021modern}:\n\\[\n\\frac{1}{T} \\sum_{t=0}^{T-1} \\Delta_t \\le \\frac{\\| x_0 - x_\\star \\|^2}{2 \\eta T} + \\frac{\\eta}{2 T} \\sum_{t=0}^{T-1} \\norm{g_t}^2\n\\] \nwhere $x_\\star$ is any optimal solution.\nBy online-to-batch conversion \\cite[Chapter 3]{orabona2021modern} we get\n\\[\n\\ErrE{\\sgd}{\\Iclass[\\SMLip]^{L,R}} \\le \\frac{R^2}{2\\eta T} + \\frac{\\eta L^2}{2}.\n\\numberthis \\label{eq:sgd-expected-err}\n\\]\n","sha256_utf8":"a9a7b3ef12705d2c6cd3953857b3d58c80a0e89a47551e11e61638dedc02b487"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"notation","kind":"notation","title":"范数与次梯度记号","source_units":[{"start":25149,"end":25942,"start_line":689,"end_line":695,"raw_tex":"\\paragraph{Notation.}\nThroughout the paper, $\\xset$ denotes a closed convex set (our lower bounds use $\\xset=\\R$) and $\\| \\cdot \\|$ denotes the Euclidean norm of a vector, and the infimum Euclidean norm in a set of vectors (or infinity if the set is empty).\n%\nFor convex function $h:\\xset \\to \\R$ we write (with minor abuse of notation) $\\grad h(x)$ for an arbitrary subgradient of $h$ at $x$, and let $\\del h(x)$ denote the set of all subgradients at $x$. \nWe say that $h$ is $L$-Lipschitz if $\\abs{ h(x) - h(y) } \\le L \\| x - y \\|$ for all $u, v \\in \\xset$ or, equivalently, $\\| \\grad h(x) \\| \\le L$ for almost-all $x\\in\\xset$. We use $[n]\\defeq\\{1,\\ldots,n\\}$ and $\\indic{\\cdot}$ for the indicator function.\n%\nThroughout, $\\log(\\cdot)$ denotes the natural logarithm and $\\N$ starts from 1.\n","sha256_utf8":"edc5e1a4660c0779f8cbaba70700fe5d204be49320c091e1ef87a01efdfc6f40"}]},{"id":"S002","semantic_type":"notation","kind":"notation","title":"互信息与二元熵","source_units":[{"start":38281,"end":38700,"start_line":772,"end_line":780,"raw_tex":"The developments in this section use the standard notation\n\\[\n    I(X;Y) = H(X)-H(X\\mid Y)=H(Y)-H(Y\\mid X)\n\\]\nfor mutual information $I(X;Y)$, entropy $H(X)=\\E_{x\\sim X}\\log \\frac{1}{\\Pr(X=x)}$ and conditional entropy $H(X\\mid Y) = \\E_{x,y\\sim X,Y}\\log \\frac{1}{\\Pr(X=x | Y=y)}$.  We also write\n\\[\n    h_2(q) \\defeq H(\\bernoulli(q)) = q \\log\\frac{1}{q} + (1-q) \\log\\frac{1}{1-q}    \n\\]\nfor the binary entropy function.\n","sha256_utf8":"f717d5a1baf3594e9eb58df14b5f0afcf3dcfe59c7351ba3f9ff8303b3481637"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"随机凸优化实例类","source_units":[{"start":46229,"end":47810,"start_line":944,"end_line":964,"raw_tex":"\\paragraph{Stochastic optimization (SO) problems.}\nA SO problem instance is a tuple $(f,P)$ containing a distribution $P$ over sample space $\\sset$ and a sample objective $f:\\xset\\times \\sset \\to \\R$. We write\n\\[\nF_{f,P}(x) \\defeq \\E_{S\\sim P} f(x;S) \n~~\\mbox{and}~~\nX^\\star_{f,P} \\defeq \\argmin_{x\\in\\xset} F_{f,P}(x)\n\\]\nfor the population objective (which we wish to minimize) and its set of minimizers, respectively. Let $\\Iclass$ denote a class of SO problem instances. We consider two fundamental classes of convex functions with a minimizer at most $R$ away from the origin.\\footnote{This implicitly, and without loss of generality, assumes that optimization methods are initialized at $x_0=0$.} The first class contains problems with bounded stochastic gradient norm (i.e., where each sample function is $L$-Lipschitz),\n\\begin{equation*}\n\t\\Iclass[\\Lip]^{L,R} \\defeq \\crl*{(f,P) \\mid f(\\cdot; s) \\text{ is convex and $L$-Lipschitz for all $s \\in \\sset$, and } \\norm*{X^\\star_{f,P}} \\le R }.\n\\end{equation*}\nThe second class contains problems with bounded stochastic gradient second moment, \n%\n%\n\\begin{equation*}\n\t\\Iclass[\\SMLip]^{L,R} \\defeq \n\t\\crl*{(f,P) \\mid f(\\cdot; s) \\text{ is convex $\\forall s\\in\\sset$ and }\n\t\\E_{S\\sim P}\\norm{\\grad f(\\cdot; S)}^2 \\le L^2,\\text{ and } \\norm*{X^\\star_{f,P}} \\le R }.\n\t%\n\\end{equation*}\nClearly, $\\Iclass[\\Lip]^{L,R} \\subset \\Iclass[\\SMLip]^{L,R}$, with $\\Iclass[\\SMLip]^{L,R}$ also including problems with heavy-tailed gradient noise.\\yair{consider defining $\\Iclass[*\\text{-}Lip]$ and $\\Mclass[*\\text{-}Lip]$ as a catch-both term.}\n","sha256_utf8":"40ffb65746bfc7eba381d05aa142ed2ba63dadbe23c94a82b39d3241a4261334"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"随机优化算法类别","source_units":[{"start":47811,"end":48810,"start_line":966,"end_line":983,"raw_tex":"\\paragraph*{Optimization algorithms.}\nGeneral SO algorithms have unrestricted access to the function $f(\\cdot;\\cdot)$ but observe $P$ only through samples $S_1,S_2,\\ldots\\simiid P$. The output of general SO algorithms is therefore of the form\n\\[\n%\nx_{t} = \\alg_t\\prn[\\big]{ f(\\cdot; S_1) , \\dots,  f(\\cdot; S_T) ; \\xi } \\in \\xset,\n%\n\\]\nwhere $\\alg_t$ is some measurable mapping and $\\xi \\sim \\uniform([0,1])\\perp S_1, \\ldots, S_T$ allows randomization.\n\nStochastic first-order (SFO) optimization algorithms are SO algorithms that depend on $f$ only through gradients observed at past iterates. That is, their output sequence takes the form \n\\[\n%\nx_{t} = \\alg_{t}\\prn*{\\grad f(x_0; S_1) , \\dots,  \\grad f(x_{t-1}; S_t); \\xi} \\in \\xset,\n%\n\\]\nwith $\\xi \\sim \\uniform([0,1])\\perp S_1, \\ldots, S_T$ as before.\n%\nWe write $\\Aall$ and $\\AFO$ for the sets of all SO and all SFO algorithms, respectively, so that $\\AFO\\subset \\Aall$. All the algorithms from \\Cref{tab:poa-upper-lower-bounds} are in $\\AFO$. \n","sha256_utf8":"5c5618ce8011a41902e33036585a4f92199d1f8f397a4a6f736b199af06d95cb"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"期望与分位数误差","source_units":[{"start":48812,"end":49550,"start_line":986,"end_line":997,"raw_tex":"\\paragraph{Error metrics.}\nFor an algorithm $\\alg$, SO instance $(f,P)$, and budget $T$, we let $x_T$ denote the algorithm's output given $T$ samples and define the expected error to be\n\\begin{equation*}\n\t\\ErrE{\\alg}{(f,P)} \\defeq \\E F_{f,P}(x_T) - \\inf_{\\xopt\\in\\xset} F_{f,P}(\\xopt),\n\\end{equation*}\nwhere the expectation is with respect to $x_T$ and hence $S_1,\\ldots,S_T$ and $\\xi$. \n%\nWe similarly define the high-probability error at confidence level $\\delta$ as\n\\begin{equation*}\n\t\\ErrHP{\\alg}{(f,P)} \\defeq Q_{1-\\delta}( F_{f,P}(x_T) ) - \\inf_{\\xopt\\in\\xset} F_{f,P}(\\xopt),\n\\end{equation*}\nwhere $Q_p(Y)=\\min\\{ y : \\Pr( Y \\le y ) \\ge p\\}$ denotes the $p$'th quantile of random variable $Y$ \\cite[Ch. 3]{borovkov1999probability}.\n","sha256_utf8":"9e80943e9f825e51754952abad8e881c89ea57dd1a288355e149fdaacf881a70"},{"start":49564,"end":49663,"start_line":1005,"end_line":1005,"raw_tex":"Throughout, we drop the superscripts $\\E$  and $\\delta$ when making statements that apply to both.\n","sha256_utf8":"302928dc5452e4fb64f85dfcd14400482ecd11da5bc39d59d61b9d32cdcee832"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"最坏误差与极小极大误差","source_units":[{"start":49664,"end":50243,"start_line":1007,"end_line":1018,"raw_tex":"\\paragraph{Minimax error.}\nGiven an instance class $\\Iclass$ and an algorithm $\\alg$, we overload the notation above to denote worst-case error over the class,\n\\begin{equation*}\n\t\\Err{\\alg}{\\Iclass} \\defeq \\sup_{(f,P)\\in\\Iclass} \\Err{\\alg}{(f,P)}.\n\\end{equation*}\n%\nGiven an algorithm class $\\Aclass$, we further overload our notation to express minimax optimal worst-case error\n\\begin{equation}\n\t\\Err{\\Aclass}{\\Iclass} \\defeq \\inf_{\\alg\\in\\Aclass} \\Err{\\alg}{\\Iclass}\n\t= \\inf_{\\alg\\in\\Aclass} \\sup_{(f,P)\\in\\Iclass} \\Err{\\alg}{(f,P)}.\n\t\\label{eq:minimax-err-def}\n\\end{equation}\n","sha256_utf8":"5cf2501793c548bcccbd0a8aa50c7fccb4ec1a465881481ce6dee09353af1f4f"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"参数不确定性元类","source_units":[{"start":63882,"end":64164,"start_line":1223,"end_line":1227,"raw_tex":"We begin with the notion of a \\emph{meta-class}, i.e., a class of classes, which we denote by $\\Mclass$. For any $\\ell,\\rho \\ge 1$, we define\n\\begin{equation*}\n\t\\Mclass[\\Lip]^{\\ell, \\rho} \\defeq \n\t\\crl*{\\Iclass[\\Lip]^{L,R} \\mid L\\in[1, \\ell]\\text{ and }R\\in[1,\\rho]}\n\\end{equation*}","sha256_utf8":"346e3f537c72bc34e2e92c6f6111148bcd7578e3494621274cd8ed06b6f3f468"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"自适应代价","source_units":[{"start":64603,"end":64834,"start_line":1230,"end_line":1234,"raw_tex":"The PoA of algorithm $\\alg$ competing against all SO algorithms on meta-class $\\Mclass$ is\n\\begin{equation*}\n\t\\PoA{\\alg}{\\Mclass} \\defeq \\sup_{\\Iclass \\in \\Mclass} \n\t\\frac{\\Err{\\alg}{\\Iclass}}{\\Err{\\Aall}{\\Iclass}}.\n\\end{equation*}","sha256_utf8":"ad357ee2c3629da69c4ed8ca2a0f798bd0e2328b461cb0d5ec99127164b0f03b"},{"start":64835,"end":65368,"start_line":1235,"end_line":1241,"raw_tex":"We use $\\PoAE{\\alg}{\\Mclass}$ and $\\PoAHP{\\alg}{\\Mclass}$ to denote the price of adaptivity w.r.t.\\ expected and quantile error, respectively (i.e., with $\\ErrBlank$ standing for $\\ErrBlank^{\\E}$ and $\\ErrBlank^{\\delta}$, respectively). \nWe may also consider the price of adaptivity in competing against algorithms in $\\Aclass\\subset\\Aall$, which we denote \n\\begin{equation}\n\t\\PoAGen{\\alg}{\\Mclass}{\\Aclass} \\defeq \\sup_{\\Iclass \\in \\Mclass} \n\t\\frac{\\Err{\\alg}{\\Iclass}}{\\Err{\\Aclass}{\\Iclass}}.\n\t\\label{eq:PoAGen-def}\n\\end{equation}","sha256_utf8":"3065a6195a4e9ae4330abee71e4d3d43449e96d1c07b08eb25251008dcff16ec"}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","title":"困难实例下界策略","source_units":[{"start":65369,"end":65897,"start_line":1242,"end_line":1247,"raw_tex":"Our strategy to lower bound the PoA in this paper is to carefully construct a collection of $n$ `hard' problem instances $(f_1, P_1), \\dots, (f_n, P_n)$ such that $(f_k, P_k) \\in {\\Iclass}_k$ with ${\\Iclass}_k \\in \\Mclass$  and then utilize the following observation\n\\begin{flalign}\\label{eq:PoA-lower-bounding-strategy}\n\\PoAGen{\\alg}{\\Mclass}{\\Aclass} \\ge \\max_{k \\in [n]} \n\\frac{\\Err{\\alg}{{\\Iclass}_k}}{\\Err{\\Aclass}{{\\Iclass}_k}} \\ge \\max_{k \\in [n]} \n\\frac{\\Err{\\alg}{(f_k, P_k)}}{\\Err{\\Aclass}{{\\Iclass}_k}}.\n\\end{flalign}","sha256_utf8":"6a1b569b2dffac1f75284cd88e85314440b9e00c7abeddba10161d0286fda546"}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","title":"投影裁剪随机梯度法","source_units":[{"start":53432,"end":53861,"start_line":1063,"end_line":1067,"raw_tex":"Given stochastic optimization instance ($f,P$), step size parameter $\\eta$, clipping parameter $\\gclip$ and initial point $x_0 \\in \\xset$, clipped SGD consists of the following recursion:\n\\begin{equation*}\n    x_{t+1} = \\Pi_{\\xset'}\\prn*{x_t - \\eta \\gbar_t},~~\\mbox{where}~~\\gbar_t \\defeq \\frac{g_t}{\\max\\{1, \\norm{g_t} / \\gclip\\}}~~\\mbox{and}~~g_t \\in \\del f(x_t; S_{t+1}).\n    \\numberthis \\label{eq:clipped-SGD}\n\\end{equation*}","sha256_utf8":"957330ec84fac7db1653d88431c7edfd9bef6dd54d795903f90e6673d9ed6519"},{"start":53862,"end":53969,"start_line":1068,"end_line":1068,"raw_tex":"Here $\\Pi_{\\xset'}$ is the Euclidean projection onto $\\xset' = \\xset \\cap \\{ x \\mid \\norm{x-x_0} \\le R\\}$.\n","sha256_utf8":"31804100500cea760894a7fca2bf70ea6d6588ecaf538e76e076a72deaf57375"}]},{"id":"S011","semantic_type":"notation","kind":"notation","title":"梯度与样本损失差","source_units":[{"start":102232,"end":102340,"start_line":2013,"end_line":2013,"raw_tex":"For brevity we define $g_t = \\grad f(x_t; S_{t+1})$ and $\\Delta_t = f(x_t; S_{t+1}) - f(x_\\star; S_{t+1})$.\n","sha256_utf8":"0af93e20974a0c3c25ca77d43ee12250880102e5a7e019baf7d9cd9eccc551c9"}]},{"id":"S012","semantic_type":"theorem","kind":"theorem","title":"期望自适应代价对数下界","source_units":[{"start":68508,"end":68734,"start_line":1283,"end_line":1291,"raw_tex":"\\begin{theorem}\\label{thm:log-PoA-lb}\nFor all $T\\in\\N$, $\\rho \\ge 1$, $\\alg\\in\\Aall$ we have \n%\n%\n$\\bigPoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \\ge \\Omega\\prn[\\big]{\\min\\crl[\\big]{\\sqrt{\\log (\\rho)}, \\sqrt{T}}}$.\n%\n%\n%\n\\end{theorem}","sha256_utf8":"8f81ebdec28583b8736ff6295e9fc86711f816f8712a716a21530b79618e9d0a"}]},{"id":"S013","semantic_type":"proof","kind":"proof","title":"对数下界完整证明","source_units":[{"start":68740,"end":72340,"start_line":1296,"end_line":1385,"raw_tex":"\\begin{proof}\n\t%\n\t%\n\t%\n\t%\n\t%\n\t %\n\tWe pick two elements from $\\Mclass[\\Lip]^{1,\\rho}$, namely $\\IclassMinus = \\Iclass[\\Lip]^{1, 1}$ and $\\IclassPlus = \\Iclass[\\Lip]^{1, \\rho}$. We then pick two instances $(f,\\Pminus)$ and $(f,\\Pplus)$ such that $(f, \\Pv) \\in \\IclassV$ for $v\\in \\{0,1\\}$. From \\cref{eq:PoA-lower-bounding-strategy}, the PoA is bounded from below by\n\t\\begin{equation*}\n\t\t\\PoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \n\t\t\\ge\n\t\t\\max_{v\\in\\{0,1\\}} \\frac{\\ErrE{\\alg}{(f,\\Pv)}}{\\ErrE{\\Aall}{\\IclassV}} \\ge \\half \\cdot \\frac{\\ErrE{\\alg}{(f,\\Pminus)}}{\\ErrE{\\Aall}{\\IclassMinus}}\n\t\t+  \\half \\cdot \\frac{\\ErrE{\\alg}{(f,\\Pplus)}}{\\ErrE{\\Aall}{\\IclassPlus}}\n\t\t.\n\t\\end{equation*}\n\tSubstituting $\\ErrE{\\Aall}{\\Iclass[\\Lip]^{L, R}} = O(L R / \\sqrt{T} )$ as per Lemma~\\ref{prop:standard-error-bounds}, we obtain\n\t\\begin{equation}\\label{eq:poa-as-expectation}\n\t\t\\PoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \\ge \\frac{\\sqrt{T}}{2} \\ErrE{\\alg}{(f,\\Pminus)} + \\frac{\\sqrt{T}}{2\\rho} \\ErrE{\\alg}{(f,\\Pplus)}.\n\t\\end{equation}\n\tIt remains to specify $f$, $\\Pminus$ and $\\Pplus$, argue that $(f, \\Pv) \\in \\IclassV$ for $v\\in \\{0,1\\}$ holds, and lower bound $\\ErrE{\\alg}{(f,\\Pv)}$. \n\tWe let $\\xset=\\R$, $\\sset = \\{0,1\\}$ and $f:\\xset\\times\\sset\\to\\R$ be\n\t\\begin{equation*}\n\t\tf(x;s) = \\begin{cases}\n\t\t\t|x| & s = 0 \\\\\n\t\t\t|x-\\rho| & s = 1.\\\\\n\t\t\\end{cases}\n\t\\end{equation*}\n\tFurthermore, we let \n\t\\begin{equation*}\n\t\t\\eps \\defeq \\min\\crl*{\\sqrt{\\frac{\\log\\rho}{8T}}, \\frac{1}{2}}~~\\mbox{and}~~\\Pv \\defeq \\bernoulli\\prn*{\\frac{1+(2v-1)\\eps}{2}}.\n\t\\end{equation*}\n\tClearly, $f(\\cdot;s)$ is 1-Lipschitz for both $s\\in\\{0,1\\}$ and moreover,\n%\n%\n%\n%\n%\n%\n%\n%\n\\begin{flalign}\n\tF_{f,\\Pv}(x) - \\inf_{\\xopt\\in\\R} F_{f,\\Pv}(\\xopt) \\ge \n\t\\begin{cases}\n\t\\eps |x| & v = 0 \\\\\n\t\\eps |x-\\rho| & v = 1.\\\\\n\t\\end{cases} \\label{eq:expectation-lb-error-expression}\n\t\\end{flalign} \n\tConsequently,  $(f, \\Pv) \\in \\IclassV$ for $v\\in \\{0,1\\}$ as required. \n\n\tLet $S_1, \\dots, S_T\\sim P_v$, yielding the  sample functions $f(\\cdot, S_1), \\dots, f(\\cdot, S_T)$.\n\tTo lower bound the error, let $x_T$ be the output of the algorithm after $T$ oracle queries. Using the expression~\\eqref{eq:expectation-lb-error-expression}, we have\n\t\\begin{equation*}\n\t\t\\ErrE{\\alg}{(f,\\Pminus)} \\ge \\frac{\\eps\\rho}{2} \\cdot \n\t\t%\n\t\t\\Pminus\\prn*{x_T \\ge \\frac{\\rho}{2}}\n\t\\quad \\text{and} \\quad \n\t\t\\ErrE{\\alg}{(f,\\Pplus)} \\ge \\frac{\\eps\\rho}{2} \\cdot \n\t\t%\n\t\t\\Pplus\\prn*{x_T < \\frac{\\rho}{2}}.\n\t\\end{equation*}\n\tTo translate these lower bounds into the language of hypothesis testing, let $\\hat{V} \\defeq \\indicb{x_T \\ge \\frac{\\rho}{2}}$\n\tand note that, for $v\\in\\{0,1\\}$,\n\t\\[\n\t\t%\n\t\t\\ErrE{\\alg}{(f,\\Pv)} \\ge \\frac{\\eps\\rho}{2} \\Pv(\\hat{V} \\ne v ).\n\t\t%\n\t\\]\n\tSubstituting into~\\eqref{eq:poa-as-expectation}, we have that\n\t%\n\t\t$\\bigPoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \\ge \\frac{\\eps\\sqrt{T}}{4}\\prn[\\big]{\n\t\t\t\\rho \\Pminus(\\hat{V} \\ne 0) + \\Pplus(\\hat{V} \\ne 1)\n\t\t}$.\n\t%\n\tWriting $p \\defeq \\frac{1}{1+\\rho}$ and letting $V\\sim \\bernoulli(p)$, we may rewrite the above PoA as\n\t\\[\n\t%\n\t\\PoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \\ge \\frac{\\eps\\sqrt{T}}{4p} \n\t\t\\prn[\\Big]{\n\t\t\t(1-p) \\Pminus(\\hat{V} \\ne 0) + p \\Pplus(\\hat{V} \\ne 1)\n\t\t}\n\t\t= \\frac{\\eps\\sqrt{T}}{4p} \\P(\\hat{V} \\ne V).\n\t%\n\t\\]\n\tNoting that $p\\le \\half$ since $\\rho \\ge 1$ and that $\\rho = \\frac{1-p}{p}$, we invoke \\Cref{lem:skewed-binary-bound} (in \\Cref{app:skewed-binary-bound}) to conclude that $\\P(\\hat{V} \\ne V) \\ge \\frac{p}{2}$ and therefore (recalling our choice of $\\veps$)\n\t$\\PoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \\ge \\frac{\\eps \\sqrt{T}}{8} \\ge \n\t\t\\frac{1}{32}\\min\\crl{\n\t\t\\sqrt{\\log\\rho}, \\sqrt{T}\n\t\t}$\n\tas required.\n\\end{proof}","sha256_utf8":"f44c7ff840a45ac93ce6dc74b008ed3950fbaccf63b76cedaf71946cf959691a"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"两类实例与误差归一化","source_units":[{"start":68740,"end":69896,"start_line":1296,"end_line":1315,"raw_tex":"\\begin{proof}\n\t%\n\t%\n\t%\n\t%\n\t%\n\t %\n\tWe pick two elements from $\\Mclass[\\Lip]^{1,\\rho}$, namely $\\IclassMinus = \\Iclass[\\Lip]^{1, 1}$ and $\\IclassPlus = \\Iclass[\\Lip]^{1, \\rho}$. We then pick two instances $(f,\\Pminus)$ and $(f,\\Pplus)$ such that $(f, \\Pv) \\in \\IclassV$ for $v\\in \\{0,1\\}$. From \\cref{eq:PoA-lower-bounding-strategy}, the PoA is bounded from below by\n\t\\begin{equation*}\n\t\t\\PoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \n\t\t\\ge\n\t\t\\max_{v\\in\\{0,1\\}} \\frac{\\ErrE{\\alg}{(f,\\Pv)}}{\\ErrE{\\Aall}{\\IclassV}} \\ge \\half \\cdot \\frac{\\ErrE{\\alg}{(f,\\Pminus)}}{\\ErrE{\\Aall}{\\IclassMinus}}\n\t\t+  \\half \\cdot \\frac{\\ErrE{\\alg}{(f,\\Pplus)}}{\\ErrE{\\Aall}{\\IclassPlus}}\n\t\t.\n\t\\end{equation*}\n\tSubstituting $\\ErrE{\\Aall}{\\Iclass[\\Lip]^{L, R}} = O(L R / \\sqrt{T} )$ as per Lemma~\\ref{prop:standard-error-bounds}, we obtain\n\t\\begin{equation}\\label{eq:poa-as-expectation}\n\t\t\\PoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \\ge \\frac{\\sqrt{T}}{2} \\ErrE{\\alg}{(f,\\Pminus)} + \\frac{\\sqrt{T}}{2\\rho} \\ErrE{\\alg}{(f,\\Pplus)}.\n\t\\end{equation}\n\tIt remains to specify $f$, $\\Pminus$ and $\\Pplus$, argue that $(f, \\Pv) \\in \\IclassV$ for $v\\in \\{0,1\\}$ holds, and lower bound $\\ErrE{\\alg}{(f,\\Pv)}$. \n","sha256_utf8":"75df327b1af6f4311887cc757ed7e04a0a52ae30bf96a4b7659f255e57352db8"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"构造偏置硬币优化实例","source_units":[{"start":69896,"end":70641,"start_line":1316,"end_line":1343,"raw_tex":"\tWe let $\\xset=\\R$, $\\sset = \\{0,1\\}$ and $f:\\xset\\times\\sset\\to\\R$ be\n\t\\begin{equation*}\n\t\tf(x;s) = \\begin{cases}\n\t\t\t|x| & s = 0 \\\\\n\t\t\t|x-\\rho| & s = 1.\\\\\n\t\t\\end{cases}\n\t\\end{equation*}\n\tFurthermore, we let \n\t\\begin{equation*}\n\t\t\\eps \\defeq \\min\\crl*{\\sqrt{\\frac{\\log\\rho}{8T}}, \\frac{1}{2}}~~\\mbox{and}~~\\Pv \\defeq \\bernoulli\\prn*{\\frac{1+(2v-1)\\eps}{2}}.\n\t\\end{equation*}\n\tClearly, $f(\\cdot;s)$ is 1-Lipschitz for both $s\\in\\{0,1\\}$ and moreover,\n%\n%\n%\n%\n%\n%\n%\n%\n\\begin{flalign}\n\tF_{f,\\Pv}(x) - \\inf_{\\xopt\\in\\R} F_{f,\\Pv}(\\xopt) \\ge \n\t\\begin{cases}\n\t\\eps |x| & v = 0 \\\\\n\t\\eps |x-\\rho| & v = 1.\\\\\n\t\\end{cases} \\label{eq:expectation-lb-error-expression}\n\t\\end{flalign} \n\tConsequently,  $(f, \\Pv) \\in \\IclassV$ for $v\\in \\{0,1\\}$ as required. \n","sha256_utf8":"d743251d4567f26b4c5c3266fd8c005cf19a4f9876e38e43994be28e1b82d9aa"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"优化误差归约为检验错误","source_units":[{"start":70642,"end":71414,"start_line":1345,"end_line":1362,"raw_tex":"\tLet $S_1, \\dots, S_T\\sim P_v$, yielding the  sample functions $f(\\cdot, S_1), \\dots, f(\\cdot, S_T)$.\n\tTo lower bound the error, let $x_T$ be the output of the algorithm after $T$ oracle queries. Using the expression~\\eqref{eq:expectation-lb-error-expression}, we have\n\t\\begin{equation*}\n\t\t\\ErrE{\\alg}{(f,\\Pminus)} \\ge \\frac{\\eps\\rho}{2} \\cdot \n\t\t%\n\t\t\\Pminus\\prn*{x_T \\ge \\frac{\\rho}{2}}\n\t\\quad \\text{and} \\quad \n\t\t\\ErrE{\\alg}{(f,\\Pplus)} \\ge \\frac{\\eps\\rho}{2} \\cdot \n\t\t%\n\t\t\\Pplus\\prn*{x_T < \\frac{\\rho}{2}}.\n\t\\end{equation*}\n\tTo translate these lower bounds into the language of hypothesis testing, let $\\hat{V} \\defeq \\indicb{x_T \\ge \\frac{\\rho}{2}}$\n\tand note that, for $v\\in\\{0,1\\}$,\n\t\\[\n\t\t%\n\t\t\\ErrE{\\alg}{(f,\\Pv)} \\ge \\frac{\\eps\\rho}{2} \\Pv(\\hat{V} \\ne v ).\n\t\t%\n\t\\]\n","sha256_utf8":"85d7cd35f5c3f1c10766a44da81186d8b42c1f5edf2edf0c850608014e63f4f6"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"加权错误转为偏置先验风险","source_units":[{"start":71414,"end":71931,"start_line":1363,"end_line":1378,"raw_tex":"\tSubstituting into~\\eqref{eq:poa-as-expectation}, we have that\n\t%\n\t\t$\\bigPoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \\ge \\frac{\\eps\\sqrt{T}}{4}\\prn[\\big]{\n\t\t\t\\rho \\Pminus(\\hat{V} \\ne 0) + \\Pplus(\\hat{V} \\ne 1)\n\t\t}$.\n\t%\n\tWriting $p \\defeq \\frac{1}{1+\\rho}$ and letting $V\\sim \\bernoulli(p)$, we may rewrite the above PoA as\n\t\\[\n\t%\n\t\\PoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \\ge \\frac{\\eps\\sqrt{T}}{4p} \n\t\t\\prn[\\Big]{\n\t\t\t(1-p) \\Pminus(\\hat{V} \\ne 0) + p \\Pplus(\\hat{V} \\ne 1)\n\t\t}\n\t\t= \\frac{\\eps\\sqrt{T}}{4p} \\P(\\hat{V} \\ne V).\n\t%\n\t\\]\n","sha256_utf8":"35b74e243ce1b8415c026c2c4fefa35369d60d7b3030efc2938d174727855dfe"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"应用检验困难性完成下界","source_units":[{"start":71931,"end":72340,"start_line":1379,"end_line":1385,"raw_tex":"\tNoting that $p\\le \\half$ since $\\rho \\ge 1$ and that $\\rho = \\frac{1-p}{p}$, we invoke \\Cref{lem:skewed-binary-bound} (in \\Cref{app:skewed-binary-bound}) to conclude that $\\P(\\hat{V} \\ne V) \\ge \\frac{p}{2}$ and therefore (recalling our choice of $\\veps$)\n\t$\\PoAE{\\alg}{\\Mclass[\\Lip]^{1,\\rho}} \\ge \\frac{\\eps \\sqrt{T}}{8} \\ge \n\t\t\\frac{1}{32}\\min\\crl{\n\t\t\\sqrt{\\log\\rho}, \\sqrt{T}\n\t\t}$\n\tas required.\n\\end{proof}","sha256_utf8":"dcfe082024f7650988561a5d6072e4872ebcf1d4e8f087f051362e1b6904cbc1"}]},{"id":"S019","semantic_type":"proposition","kind":"proposition","title":"已知参数极小极大误差","source_units":[{"start":50394,"end":50860,"start_line":1026,"end_line":1030,"raw_tex":"\\begin{restatable}{prop}{restatePropStandardErrorBounds}\\label{prop:standard-error-bounds}\nFor all $R, L > 0$, instance class $\\Iclass \\in \\{ \\Iclass[\\Lip]^{L,R}, \\Iclass[\\SMLip]^{L,R} \\}$ and \n$\\Aclass \\in \\{ \\Aall, \\AFO \\}$, we have \n$\\ErrE{\\Aclass}{\\Iclass}=\\Theta\\left( \\frac{LR}{\\sqrt{T} } \\right)$. Moreover, for any $\\delta\\in(0,\\half)$ we have $\\ErrHP{\\Aclass}{\\Iclass}=\\Theta\\left( \\frac{LR}{\\sqrt{1+ {T}/{\\log \\frac{1}{\\delta}}}} \\right)$.\n\\end{restatable}","sha256_utf8":"be047a1b983dbaa352a3df2ab77623308f9d814ebdb902d53ee768a0633025a9"}]},{"id":"S020","semantic_type":"proof","kind":"proof","title":"极小极大误差完整证明","source_units":[{"start":50862,"end":51660,"start_line":1032,"end_line":1038,"raw_tex":"\\begin{proof}\nSince $\\Iclass[\\Lip]^{L,R}\\subset \\Iclass[\\SMLip]^{L,R}$ and $\\AFO\\subset\\Aall$, we only need to prove minimax error lower bounds on $(\\Aall,\\Iclass[\\Lip]^{L,R})$ and minimax upper bounds on $(\\AFO, \\Iclass[\\SMLip]^{L,R})$. \n\nFor expected error these results are well-known. The upper bound on $\\ErrE{\\AFO}{ \\Iclass[\\SMLip]^{L,R}}$ follows, for example, by setting $\\eta = \\frac{R}{L\\sqrt{T}}$ in \\cref{eq:sgd-expected-err}. For the lower bound on $\\ErrE{\\Aall}{ \\Iclass[\\Lip]^{L,R}}$, originally due to~\\citet{nemirovski1983problem}, see, e.g., \\citet[Theorem 5.2.10.]{duchi2018introductory}.\n\nFor error quantiles, however, we could not find direct proof of the required upper and lower bounds, and we prove them in \\Cref{app:high-prob-ms-lip,app:high-prop-lip-lb} below.\n\\end{proof}","sha256_utf8":"8b1ec9c9354e24906767a152c691cd72d7e9e71c54d38d9d93bcb318cd18176c"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"利用实例类与算法类包含关系","source_units":[{"start":50862,"end":51101,"start_line":1032,"end_line":1033,"raw_tex":"\\begin{proof}\nSince $\\Iclass[\\Lip]^{L,R}\\subset \\Iclass[\\SMLip]^{L,R}$ and $\\AFO\\subset\\Aall$, we only need to prove minimax error lower bounds on $(\\Aall,\\Iclass[\\Lip]^{L,R})$ and minimax upper bounds on $(\\AFO, \\Iclass[\\SMLip]^{L,R})$. \n","sha256_utf8":"259387bae545285d75ab9b038b9c24558cfd83e684986a1dc9b7f98dc1f013a5"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"期望误差上下界","source_units":[{"start":51102,"end":51470,"start_line":1035,"end_line":1035,"raw_tex":"For expected error these results are well-known. The upper bound on $\\ErrE{\\AFO}{ \\Iclass[\\SMLip]^{L,R}}$ follows, for example, by setting $\\eta = \\frac{R}{L\\sqrt{T}}$ in \\cref{eq:sgd-expected-err}. For the lower bound on $\\ErrE{\\Aall}{ \\Iclass[\\Lip]^{L,R}}$, originally due to~\\citet{nemirovski1983problem}, see, e.g., \\citet[Theorem 5.2.10.]{duchi2018introductory}.\n","sha256_utf8":"d7ff1b9b52d84df7508278d3301fba0fa74353c372c61d2323335fbb82e9598a"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"分位数上下界的证明归约","source_units":[{"start":51471,"end":51660,"start_line":1037,"end_line":1038,"raw_tex":"For error quantiles, however, we could not find direct proof of the required upper and lower bounds, and we prove them in \\Cref{app:high-prob-ms-lip,app:high-prop-lip-lb} below.\n\\end{proof}","sha256_utf8":"5926f54ee5ec708270c1ac525fbc0f27c06043c963867fb66175db45a7b8a1bd"}]},{"id":"S024","semantic_type":"proposition","kind":"proposition","title":"裁剪随机梯度高概率界","source_units":[{"start":54056,"end":54593,"start_line":1071,"end_line":1073,"raw_tex":"\\begin{customprop}{1a}\\label{prop:ms-lip-hp-bound}\n    For any $\\delta \\in (0,\\half)$, any $L,R>0$, and any instance in $\\Iclass[\\SMLip]^{L, R}$, we have that $T$ steps of clipped SGD~\\eqref{eq:clipped-SGD} with step size $\\eta = \\frac{R}{L\\sqrt{T}}$ and $\\gclip = \\frac{L\\sqrt{T}}{\\sqrt{\\log (1/\\delta)}}$ produce iterates $x_0, \\ldots, x_{T-1}$ whose average $\\bar{x}_T = \\frac{1}{T} \\sum_{t=0}^{T-1} x_t$ has suboptimality $O\\prn*{\\frac{LR}{\\sqrt{T}}\\sqrt{\\log\\frac{1}{\\delta}}}$ with probability at least $1-\\delta$.\n\\end{customprop}","sha256_utf8":"f2254604f6b2c068b8e75a37c403c1642721ce6117993cf8dfb53fadeaddb515"}]},{"id":"S025","semantic_type":"proof","kind":"proof","title":"裁剪算法完整收敛证明","source_units":[{"start":54595,"end":59649,"start_line":1075,"end_line":1149,"raw_tex":"\\begin{proof}\n    Let $F(x) = \\E_{S\\sim P} f(x;S)$ be the objective function and let $\\xopt$ be its minimizer in $\\xset'$ (and hence, by assumption, also in $\\xset$). The standard SGD regret bound, e.g., \\cite[Lemma 2.12.]{orabona2021modern}, gives\n    \\[\n    \\sum_{t=0}^{T-1} \\inner{\\gbar_t}{x_t - \\xopt} \\le \\frac{R^2}{2\\eta} + \\frac{\\eta}{2} \\sum_{t=0}^{T-1} \\norm{\\gbar_t}^2\n    \\]\n    with probability 1. \n    Writing $\\E_t Z = \\Ex*{Z | x_0, \\ldots, x_t}$ to streamline notation, \n    and using $\\gbar_t = \\E_t g_t - (\\E_t g_t - \\E_t \\gbar_t) - (\\E_t \\gbar_t - \\gbar_t)$ we rearrange the above display to read,\n    \\[\n        \\sum_{t=0}^{T-1} \\inner{\\E_t g_t}{x_t - \\xopt} \n        \\le \\frac{R^2}{2\\eta} \n        + \\sum_{t=0}^{T-1} \\inner{\\E_t g_t - \\E_t \\gbar_t}{x_t - \\xopt} \n        + \\sum_{t=0}^{T-1} \\inner{\\E_t \\gbar_t - \\gbar_t}{x_t - \\xopt}\n        + \\frac{\\eta}{2} \\sum_{t=0}^{T-1} \\norm{\\gbar_t}^2. \n        \\numberthis \\label{eq:clipped-sgd-regret-rearranged}\n    \\]\n    \n    Since $\\E_t g_t\\in\\del F(x_t)$, we have $\\inner{\\E_t g_t}{x_t - \\xopt} \\ge F(x_t) - F(\\xopt)$ and therefore, by Jensen's inequality we obtain the following lower bound on the LHS of \\Cref{eq:clipped-sgd-regret-rearranged}\n    \\[\n        \\sum_{t=0}^{T-1} \\inner{\\E_t g_t}{x_t - \\xopt}  \\ge T \\brk*{F(\\bar{x}_T) - F(\\xopt)}.\n        \\numberthis \\label{eq:clipped-sgd-bound-1}\n    \\]\n    We have $R^2 / (2 \\eta) = O(LR\\sqrt{T})$ by our choice of $\\eta$, and it remains to bound the three sums in the RHS of \\Cref{eq:clipped-sgd-regret-rearranged}.\n\n    To bound the first sum, we observe that\n    \\begin{flalign*}\n        \\norm{\\E_t g_t - \\E_t \\gbar_t} &\\overle{(i)} \\E_t \\norm{g_t - \\gbar_t} \\overeq{(ii)} \\E_t \\brk*{\\norm*{g_t} \\prn*{1 - \\frac{\\gclip}{\\| g_t \\|} } \\indic{\\norm{g_t} > \\gclip}} \n       \\le \n        \\E_t \\brk*{\\norm{g_t}\\indic{\\norm{g_t} > \\gclip}}  \\\\ &\n        \\le \\frac{1}{\\gclip} \\E_t \\norm{g_t}^2 \\overle{(iii)} \\frac{L^2}{\\gclip},\n    \\end{flalign*}\n    where $(i)$ is from Jensen's inequality applied to the Euclidean norm, $(ii)$ from the definition of $\\gbar_t$ and $(iii)$ is from the definition of $\\Iclass[\\SMLip]^{L, R}$. \n    Therefore, using the triangle inequality, Cauchy-Schwarz, the bound $\\norm{x_t-\\xopt}\\le 2R$, and our setting of $\\gclip$, we get that\n    \\[\n    \\abs*{\\sum_{t=0}^{T-1} \\inner{\\E_t \\gbar_t - \\E_t g_t}{x_t - \\xopt} } \\le \\sum_{t=0}^{T-1} \\norm{\\E_t g_t - \\E_t \\gbar_t} \\norm{x_t - \\xopt}  \\le \\frac{2L^2R T}{\\gclip} = 2LR \\sqrt{T\\log\\frac{1}{\\delta}}.\n    \\numberthis \\label{eq:clipped-sgd-bound-2}\n    \\]\n\n    To bound the second sum, we note that $\\E_t \\brk*{\\inner{\\gbar_t - \\E_t \\gbar_t}{x_t - \\xopt}} = 0$ for all $t$. Moreover, again by Cauchy-Schwarz and $\\norm{x_t-\\xopt}\\le 2R$, we have that \n    \\[\n        \\E_t \\brk*{\\inner{\\gbar_t - \\E_t \\gbar_t}{x_t - \\xopt}}^2\n        \\le 4R^2 \\E_t \\norm{\\gbar_t - \\E_t \\gbar_t}^2 \n        \\le 4R^2 \\E_t \\norm{\\gbar_t}^2 \n        \\le 4R^2 \\E_t \\norm{g_t}^2 \n        \\le 4L^2 R^2,\n    \\]\n    as well as\n    \\[\n        \\abs*{\\inner{\\gbar_t - \\E_t \\gbar_t}{x_t - \\xopt}}\n        \\le \n        \\norm{\\gbar_t - \\E_t \\gbar_t} \\norm{x_t - \\xopt}\n        \\le 4\\gclip R,\n    \\]\n    due to the gradient clipping.\n    Therefore, we may apply \\Cref{fact:freedman} on $X_t = \\inner{\\E_t\\gbar_t -  \\gbar_t}{x_t - \\xopt}$ to conclude that, with probability at least $1-\\frac{\\delta}{2}$,\n    \\[\n        \\sum_{t=0}^{T-1} \\inner{\\E_t \\gbar_t - \\gbar_t}{x_t - \\xopt}\n        \\le LR \\sqrt{8T \\log\\frac{2}{\\delta}} + \\frac{4}{3} \\gclip R \\log\\frac{2}{\\delta} = O\\prn*{LR \\sqrt{T \\log\\frac{1}{\\delta}} }.\n        \\numberthis \\label{eq:clipped-sgd-bound-3}\n    \\]\n\n    For the third and final sum, we observe that $\\E_t \\norm{\\gbar_t}^4 \\le \\gclip^2 \\E_t \\norm{\\gbar_t}^2 \\le (\\gclip L)^2$, and that $\\norm{\\gbar_t}^2 \\le \\gclip^2$ with probability 1. We then apply \\Cref{fact:freedman} again, this time with $X_t = \\norm{\\gbar_t}^2$ to obtain that, with probability at least $1-\\frac{\\delta}{2}$,\n    \\begin{flalign*}\n        \\frac{\\eta}{2} \\sum_{t=0}^{T-1} \\norm{\\gbar_t}^2 \n        & \\le \\frac{\\eta}{2} \\sum_{t=0}^{T-1} \\E_t\\norm{\\gbar_t}^2 \n        + \\frac{\\eta}{2} \\gclip L \\sqrt{ 2T \\log \\frac{1}{\\delta}} \n        + \\frac{\\eta}{6} \\gclip^2 \\log \\frac{1}{\\delta}\n        \\\\ &\n        \\overle{(i)} \\frac{\\eta}{2} L^2 T \\cdot \\prn*{1 + \\sqrt{2} + \\frac{1}{3} } \\overle{(ii)} O\\prn*{LR \\sqrt{T }},\n        \\numberthis \\label{eq:clipped-sgd-bound-4}\n    \\end{flalign*}\n    due to $(i)$ the choice $\\gclip$ and $\\E_t\\norm{\\gbar_t}^2 \\le \\E_t\\norm{g_t}^2 \\le L^2$, and $(ii)$ the choice of $\\eta$. \n\n    Substituting \\cref{eq:clipped-sgd-bound-1,eq:clipped-sgd-bound-2,eq:clipped-sgd-bound-3,eq:clipped-sgd-bound-4} into \\cref{eq:clipped-sgd-regret-rearranged} and using again the choice of $\\eta$ and a union bound over the two events holding with probability at least $1-\\frac{\\delta}{2}$, we conclude that $T \\brk*{F(\\bar{x}_T) - F(\\xopt)} \\le O\\prn*{LR \\sqrt{T \\log\\frac{1}{\\delta}} }$ with probability at least $1-\\delta$, leading to the claimed rate of convergence.\n\\end{proof}","sha256_utf8":"85a0dd63527df459c956708407963192c1d21121f42d16e7caedee1dfc2674c5"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"遗憾界与随机误差分解","source_units":[{"start":54595,"end":55578,"start_line":1075,"end_line":1090,"raw_tex":"\\begin{proof}\n    Let $F(x) = \\E_{S\\sim P} f(x;S)$ be the objective function and let $\\xopt$ be its minimizer in $\\xset'$ (and hence, by assumption, also in $\\xset$). The standard SGD regret bound, e.g., \\cite[Lemma 2.12.]{orabona2021modern}, gives\n    \\[\n    \\sum_{t=0}^{T-1} \\inner{\\gbar_t}{x_t - \\xopt} \\le \\frac{R^2}{2\\eta} + \\frac{\\eta}{2} \\sum_{t=0}^{T-1} \\norm{\\gbar_t}^2\n    \\]\n    with probability 1. \n    Writing $\\E_t Z = \\Ex*{Z | x_0, \\ldots, x_t}$ to streamline notation, \n    and using $\\gbar_t = \\E_t g_t - (\\E_t g_t - \\E_t \\gbar_t) - (\\E_t \\gbar_t - \\gbar_t)$ we rearrange the above display to read,\n    \\[\n        \\sum_{t=0}^{T-1} \\inner{\\E_t g_t}{x_t - \\xopt} \n        \\le \\frac{R^2}{2\\eta} \n        + \\sum_{t=0}^{T-1} \\inner{\\E_t g_t - \\E_t \\gbar_t}{x_t - \\xopt} \n        + \\sum_{t=0}^{T-1} \\inner{\\E_t \\gbar_t - \\gbar_t}{x_t - \\xopt}\n        + \\frac{\\eta}{2} \\sum_{t=0}^{T-1} \\norm{\\gbar_t}^2. \n        \\numberthis \\label{eq:clipped-sgd-regret-rearranged}\n    \\]\n","sha256_utf8":"496f814dd2beaa1bb4794c916f042cdd482567ac5c2031f1d44a7384a4f9109d"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"凸性给出平均迭代误差","source_units":[{"start":55583,"end":56132,"start_line":1092,"end_line":1097,"raw_tex":"    Since $\\E_t g_t\\in\\del F(x_t)$, we have $\\inner{\\E_t g_t}{x_t - \\xopt} \\ge F(x_t) - F(\\xopt)$ and therefore, by Jensen's inequality we obtain the following lower bound on the LHS of \\Cref{eq:clipped-sgd-regret-rearranged}\n    \\[\n        \\sum_{t=0}^{T-1} \\inner{\\E_t g_t}{x_t - \\xopt}  \\ge T \\brk*{F(\\bar{x}_T) - F(\\xopt)}.\n        \\numberthis \\label{eq:clipped-sgd-bound-1}\n    \\]\n    We have $R^2 / (2 \\eta) = O(LR\\sqrt{T})$ by our choice of $\\eta$, and it remains to bound the three sums in the RHS of \\Cref{eq:clipped-sgd-regret-rearranged}.\n","sha256_utf8":"61b79ff8e060fa8b6446065d249cab961a8775dc58fa8270c20372d43c569388"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"控制裁剪偏差","source_units":[{"start":56133,"end":57147,"start_line":1099,"end_line":1111,"raw_tex":"    To bound the first sum, we observe that\n    \\begin{flalign*}\n        \\norm{\\E_t g_t - \\E_t \\gbar_t} &\\overle{(i)} \\E_t \\norm{g_t - \\gbar_t} \\overeq{(ii)} \\E_t \\brk*{\\norm*{g_t} \\prn*{1 - \\frac{\\gclip}{\\| g_t \\|} } \\indic{\\norm{g_t} > \\gclip}} \n       \\le \n        \\E_t \\brk*{\\norm{g_t}\\indic{\\norm{g_t} > \\gclip}}  \\\\ &\n        \\le \\frac{1}{\\gclip} \\E_t \\norm{g_t}^2 \\overle{(iii)} \\frac{L^2}{\\gclip},\n    \\end{flalign*}\n    where $(i)$ is from Jensen's inequality applied to the Euclidean norm, $(ii)$ from the definition of $\\gbar_t$ and $(iii)$ is from the definition of $\\Iclass[\\SMLip]^{L, R}$. \n    Therefore, using the triangle inequality, Cauchy-Schwarz, the bound $\\norm{x_t-\\xopt}\\le 2R$, and our setting of $\\gclip$, we get that\n    \\[\n    \\abs*{\\sum_{t=0}^{T-1} \\inner{\\E_t \\gbar_t - \\E_t g_t}{x_t - \\xopt} } \\le \\sum_{t=0}^{T-1} \\norm{\\E_t g_t - \\E_t \\gbar_t} \\norm{x_t - \\xopt}  \\le \\frac{2L^2R T}{\\gclip} = 2LR \\sqrt{T\\log\\frac{1}{\\delta}}.\n    \\numberthis \\label{eq:clipped-sgd-bound-2}\n    \\]\n","sha256_utf8":"54968b67ed030373f4116024bd98b8c853dc1f4ac4a0d9c07200453dc8a62fe9"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"鞅差的二阶矩与幅度","source_units":[{"start":57148,"end":57791,"start_line":1113,"end_line":1128,"raw_tex":"    To bound the second sum, we note that $\\E_t \\brk*{\\inner{\\gbar_t - \\E_t \\gbar_t}{x_t - \\xopt}} = 0$ for all $t$. Moreover, again by Cauchy-Schwarz and $\\norm{x_t-\\xopt}\\le 2R$, we have that \n    \\[\n        \\E_t \\brk*{\\inner{\\gbar_t - \\E_t \\gbar_t}{x_t - \\xopt}}^2\n        \\le 4R^2 \\E_t \\norm{\\gbar_t - \\E_t \\gbar_t}^2 \n        \\le 4R^2 \\E_t \\norm{\\gbar_t}^2 \n        \\le 4R^2 \\E_t \\norm{g_t}^2 \n        \\le 4L^2 R^2,\n    \\]\n    as well as\n    \\[\n        \\abs*{\\inner{\\gbar_t - \\E_t \\gbar_t}{x_t - \\xopt}}\n        \\le \n        \\norm{\\gbar_t - \\E_t \\gbar_t} \\norm{x_t - \\xopt}\n        \\le 4\\gclip R,\n    \\]\n    due to the gradient clipping.\n","sha256_utf8":"6505e5670192a8a69011b13138fc0f592746d327bb98fa4a66c9658abda21646"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"控制梯度随机波动","source_units":[{"start":57791,"end":58230,"start_line":1129,"end_line":1134,"raw_tex":"    Therefore, we may apply \\Cref{fact:freedman} on $X_t = \\inner{\\E_t\\gbar_t -  \\gbar_t}{x_t - \\xopt}$ to conclude that, with probability at least $1-\\frac{\\delta}{2}$,\n    \\[\n        \\sum_{t=0}^{T-1} \\inner{\\E_t \\gbar_t - \\gbar_t}{x_t - \\xopt}\n        \\le LR \\sqrt{8T \\log\\frac{2}{\\delta}} + \\frac{4}{3} \\gclip R \\log\\frac{2}{\\delta} = O\\prn*{LR \\sqrt{T \\log\\frac{1}{\\delta}} }.\n        \\numberthis \\label{eq:clipped-sgd-bound-3}\n    \\]\n","sha256_utf8":"136a6dec8db7043300fae417b7155ee2ab749d6d778c409d18917a490cdecc8b"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"控制裁剪梯度平方和","source_units":[{"start":58231,"end":59165,"start_line":1136,"end_line":1146,"raw_tex":"    For the third and final sum, we observe that $\\E_t \\norm{\\gbar_t}^4 \\le \\gclip^2 \\E_t \\norm{\\gbar_t}^2 \\le (\\gclip L)^2$, and that $\\norm{\\gbar_t}^2 \\le \\gclip^2$ with probability 1. We then apply \\Cref{fact:freedman} again, this time with $X_t = \\norm{\\gbar_t}^2$ to obtain that, with probability at least $1-\\frac{\\delta}{2}$,\n    \\begin{flalign*}\n        \\frac{\\eta}{2} \\sum_{t=0}^{T-1} \\norm{\\gbar_t}^2 \n        & \\le \\frac{\\eta}{2} \\sum_{t=0}^{T-1} \\E_t\\norm{\\gbar_t}^2 \n        + \\frac{\\eta}{2} \\gclip L \\sqrt{ 2T \\log \\frac{1}{\\delta}} \n        + \\frac{\\eta}{6} \\gclip^2 \\log \\frac{1}{\\delta}\n        \\\\ &\n        \\overle{(i)} \\frac{\\eta}{2} L^2 T \\cdot \\prn*{1 + \\sqrt{2} + \\frac{1}{3} } \\overle{(ii)} O\\prn*{LR \\sqrt{T }},\n        \\numberthis \\label{eq:clipped-sgd-bound-4}\n    \\end{flalign*}\n    due to $(i)$ the choice $\\gclip$ and $\\E_t\\norm{\\gbar_t}^2 \\le \\E_t\\norm{g_t}^2 \\le L^2$, and $(ii)$ the choice of $\\eta$. \n","sha256_utf8":"58abecb56e18077b6ae8f62f398e03212c1021efe4485a37a14a39bf0d55a835"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"合并事件得到收敛率","source_units":[{"start":59166,"end":59649,"start_line":1148,"end_line":1149,"raw_tex":"    Substituting \\cref{eq:clipped-sgd-bound-1,eq:clipped-sgd-bound-2,eq:clipped-sgd-bound-3,eq:clipped-sgd-bound-4} into \\cref{eq:clipped-sgd-regret-rearranged} and using again the choice of $\\eta$ and a union bound over the two events holding with probability at least $1-\\frac{\\delta}{2}$, we conclude that $T \\brk*{F(\\bar{x}_T) - F(\\xopt)} \\le O\\prn*{LR \\sqrt{T \\log\\frac{1}{\\delta}} }$ with probability at least $1-\\delta$, leading to the claimed rate of convergence.\n\\end{proof}","sha256_utf8":"db1000eacf0c9b0a016d4ce730a61146ff67d171ea52b939df250a60d1d29fb8"}]},{"id":"S033","semantic_type":"proposition","kind":"proposition","title":"误差分位数极小极大下界","source_units":[{"start":59729,"end":59994,"start_line":1152,"end_line":1157,"raw_tex":"\\begin{customprop}{1b}\\label{prop:high-prop-lip-lb}\n    For all $L,R>0$ and $\\delta\\in (0,\\frac{1}{2})$ we have $\\ErrHP{\\Aall}{\\Iclass[\\Lip]^{L,R}} \\ge \n    \\Omega\\prn*{  \n        LR \\min\\crl*{ \\sqrt{\\frac{\\log\\frac{1}{\\delta}}{T}}, 1 }\n        }$.\n\\end{customprop}","sha256_utf8":"67b8ef2eefb18ae2a892ff61b3d9890f7d8fc6293e6dfeb3a7a95a9f8c8e59e3"}]},{"id":"S034","semantic_type":"proof","kind":"proof","title":"分位数下界完整证明","source_units":[{"start":59996,"end":63686,"start_line":1159,"end_line":1216,"raw_tex":"\\begin{proof}\n    We prove \\Cref{prop:high-prop-lip-lb} with a construction almost identical to the one in the proof of \\Cref{thm:log-PoA-lb}. \n    We let $\\xset=\\R$, $\\sset = \\{0,1\\}$ and $f:\\xset\\times\\sset\\to\\R$ be\n\t\\begin{equation*}\n\t\tf(x;s) = \\begin{cases}\n\t\t\tL|x| & s = 0 \\\\\n\t\t\tL|x-R| & s = 1,\\\\\n\t\t\\end{cases}\n\t\\end{equation*}\n\tand we set, assuming for now $\\delta \\le \\frac{1}{5}$,\n\t\\begin{equation*}\n\t\t\\eps \\defeq \\min\\crl*{\\sqrt{\\frac{\\log\\frac{1-2\\delta}{2\\delta}}{8T}}, \\frac{1}{2}}~~\\mbox{and}~~\\Pv \\defeq \\bernoulli\\prn*{\\frac{1+(2v-1)\\eps}{2}}.\n\t\\end{equation*}\n\tClearly, $f(\\cdot;s)$ is $L$-Lipschitz for both $s\\in\\{0,1\\}$ and moreover,\n    \\begin{flalign*}\n        F_{f,\\Pv}(x) - \\inf_{\\xopt\\in\\R} F_{f,\\Pv}(\\xopt) \\ge \n        \\begin{cases}\n        \\eps L |x+R| & v = 0 \\\\\n        \\eps L |x-R| & v = 1.\\\\\n        \\end{cases} \\numberthis \\label{eq:minimax-lb-error-expression}\n    \\end{flalign*} \n    Consequently,  $(f, \\Pv) \\in \\Iclass[\\Lip]^{L,R}$ for $v\\in \\{0,1\\}$.\n    \n    For any $\\alg\\in\\Aall$, let $x_T^v$ be the output of $\\alg$ when interacting with $(f,\\Pv)$ by observing $S_1,\\ldots,S_T\\simiid \\Pv$, and let $\\hat{v}(x) \\defeq \\indic{x>0}$. Substituting into~\\eqref{eq:minimax-lb-error-expression}, we have \n    \\[\n    \\ErrHP{\\alg}{(f,\\Pv)} \\ge \\eps LR \\indicb{ \\Pr(\\hat{v}\\prn*{x_T^v} \\ne v) \\ge \\delta }\n    ~~\n    \\mbox{for}\n    ~~\n    v\\in \\{0,1\\}.\n    \\]\n    Therefore, since $ \\max_{v\\in\\{0,1\\}}\\Pr(\\hat{v}\\prn*{x_T^v} \\ne v) \\ge \\E_{V\\sim \\bernoulli(2\\delta)}\\Pr(\\hat{v}\\prn*{x_T^V} \\ne V)$ we have\n    \\[\n    \\ErrHP{\\alg}{\\Iclass[\\Lip]^{L,R}} \\ge \\eps LR \\indicb{ \\max_{v\\in\\{0,1\\}}\\Pr(\\hat{v}\\prn*{x_T^v} \\ne v) \\ge \\delta } \n    \\ge \n    \\eps LR \\indicb{ \\E_{V\\sim \\bernoulli(2\\delta)}\\Pr(\\hat{v}\\prn*{x_T^V} \\ne V) \\ge \\delta }.\n    \\]\n    Let $\\hat{V} \\defeq \\hat{v}\\prn*{x_T^V}$ and note that it depends on $V$ only through $x_T^V$ and hence through $S_1,\\ldots,S_T\\simiid P_V$. By our choice of $\\eps$ and the current assumption that $2 \\delta < 2/5 \\le 1/2$, we may apply \\Cref{lem:skewed-binary-bound} with $p=2\\delta$ and conclude that $\\Pr(\\hat{V} \\ne V) \\ge \\delta$, implying\n    \\[\n        \\ErrHP{\\alg}{\\Iclass[\\Lip]^{L,R}} \\ge \\eps LR = \\Omega\\prn*{  \n            LR \\min\\crl*{ \\sqrt{\\frac{\\log\\frac{1}{\\delta}}{T}}, 1 }\n            }\n    \\]\n    for all $\\delta \\le \\frac{1}{5}$. \n\n    To extend the lower bound to $\\delta \\in (\\frac{1}{5}, \\frac{1}{2})$,\n    %\n    we only need to show that $\\ErrCustom{1/2}{\\alg}{\\Iclass[\\Lip]^{L,R}} = \\Omega(LR/\\sqrt{T})$. We do this with a simple ``anti-boosting'' argument. Let $k=4$ and consider $k$ copies of the construction above for $\\delta'=\\frac{1}{5}$, each operating on independent samples and independent coordinates, scaled so that the sum of the copies is in $\\Iclass[\\Lip]^{L,R}$. Concretely, the construction uses $\\xset=\\R^k$ and $\\sset=\\{0,1\\}^k$, and given $f:\\R\\times \\{0,1\\}\\to \\R$ described above, defines $\\tilde{f}: \\xset\\times\\sset\\to\\R$ such that\n    \\[\n        \\tilde{f}(x,s) = k^{-3/2}\\sum_{i \\in [k]} f(x_i \\sqrt{k};s_i).\n    \\]\n    For $v=(v_1,\\ldots,v_k)\\in\\{0,1\\}^k$ and $P=P_{v_1}\\times \\cdots \\times P_{v_k}$, this construction has optimality gap\n    \\[\n        F_{f,\\Pv}(x) - \\inf_{\\xopt\\in\\xset} F_{f,\\Pv}(\\xopt)\n        \\ge \\tfrac{1}{k^{3/2}}\\eps LR \\cdot  \\norm*{x \\sqrt{k} - (2v-1)R}_1.\n    \\]\n    Therefore, disagreeing with the sign of $\\xopt$ in even a single coordinate implies optimality gap of at least $k^{-3/2}\\eps LR = \\Omega(LR/\\sqrt{T})$ by our choice of $\\eps$. By the discussion above, the probability of mistaking at least one of the coordinates is at least $1-(1-\\delta')^k > \\half$ for $\\delta' = 1/5$ and $k=4$, completing the proof.\n\\end{proof}","sha256_utf8":"1f4751560b630c52c18c5a92b8781f0afcb13a0bca83b9e2ad30ca0b7b826f92"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"构造分位数下界实例","source_units":[{"start":59996,"end":60984,"start_line":1159,"end_line":1180,"raw_tex":"\\begin{proof}\n    We prove \\Cref{prop:high-prop-lip-lb} with a construction almost identical to the one in the proof of \\Cref{thm:log-PoA-lb}. \n    We let $\\xset=\\R$, $\\sset = \\{0,1\\}$ and $f:\\xset\\times\\sset\\to\\R$ be\n\t\\begin{equation*}\n\t\tf(x;s) = \\begin{cases}\n\t\t\tL|x| & s = 0 \\\\\n\t\t\tL|x-R| & s = 1,\\\\\n\t\t\\end{cases}\n\t\\end{equation*}\n\tand we set, assuming for now $\\delta \\le \\frac{1}{5}$,\n\t\\begin{equation*}\n\t\t\\eps \\defeq \\min\\crl*{\\sqrt{\\frac{\\log\\frac{1-2\\delta}{2\\delta}}{8T}}, \\frac{1}{2}}~~\\mbox{and}~~\\Pv \\defeq \\bernoulli\\prn*{\\frac{1+(2v-1)\\eps}{2}}.\n\t\\end{equation*}\n\tClearly, $f(\\cdot;s)$ is $L$-Lipschitz for both $s\\in\\{0,1\\}$ and moreover,\n    \\begin{flalign*}\n        F_{f,\\Pv}(x) - \\inf_{\\xopt\\in\\R} F_{f,\\Pv}(\\xopt) \\ge \n        \\begin{cases}\n        \\eps L |x+R| & v = 0 \\\\\n        \\eps L |x-R| & v = 1.\\\\\n        \\end{cases} \\numberthis \\label{eq:minimax-lb-error-expression}\n    \\end{flalign*} \n    Consequently,  $(f, \\Pv) \\in \\Iclass[\\Lip]^{L,R}$ for $v\\in \\{0,1\\}$.\n","sha256_utf8":"c7bddcfea3d2b577d39c18988e9067f4083f2b5c4802e2385578811e6d6b5ab0"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"分位数误差转为检验风险","source_units":[{"start":60989,"end":61774,"start_line":1182,"end_line":1195,"raw_tex":"    For any $\\alg\\in\\Aall$, let $x_T^v$ be the output of $\\alg$ when interacting with $(f,\\Pv)$ by observing $S_1,\\ldots,S_T\\simiid \\Pv$, and let $\\hat{v}(x) \\defeq \\indic{x>0}$. Substituting into~\\eqref{eq:minimax-lb-error-expression}, we have \n    \\[\n    \\ErrHP{\\alg}{(f,\\Pv)} \\ge \\eps LR \\indicb{ \\Pr(\\hat{v}\\prn*{x_T^v} \\ne v) \\ge \\delta }\n    ~~\n    \\mbox{for}\n    ~~\n    v\\in \\{0,1\\}.\n    \\]\n    Therefore, since $ \\max_{v\\in\\{0,1\\}}\\Pr(\\hat{v}\\prn*{x_T^v} \\ne v) \\ge \\E_{V\\sim \\bernoulli(2\\delta)}\\Pr(\\hat{v}\\prn*{x_T^V} \\ne V)$ we have\n    \\[\n    \\ErrHP{\\alg}{\\Iclass[\\Lip]^{L,R}} \\ge \\eps LR \\indicb{ \\max_{v\\in\\{0,1\\}}\\Pr(\\hat{v}\\prn*{x_T^v} \\ne v) \\ge \\delta } \n    \\ge \n    \\eps LR \\indicb{ \\E_{V\\sim \\bernoulli(2\\delta)}\\Pr(\\hat{v}\\prn*{x_T^V} \\ne V) \\ge \\delta }.\n    \\]\n","sha256_utf8":"f0b7c298d6bcd55598e934e94bfe0922cc385305f4312e5a5e513d11a30d5151"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"小置信参数下的检验下界","source_units":[{"start":61774,"end":62329,"start_line":1196,"end_line":1202,"raw_tex":"    Let $\\hat{V} \\defeq \\hat{v}\\prn*{x_T^V}$ and note that it depends on $V$ only through $x_T^V$ and hence through $S_1,\\ldots,S_T\\simiid P_V$. By our choice of $\\eps$ and the current assumption that $2 \\delta < 2/5 \\le 1/2$, we may apply \\Cref{lem:skewed-binary-bound} with $p=2\\delta$ and conclude that $\\Pr(\\hat{V} \\ne V) \\ge \\delta$, implying\n    \\[\n        \\ErrHP{\\alg}{\\Iclass[\\Lip]^{L,R}} \\ge \\eps LR = \\Omega\\prn*{  \n            LR \\min\\crl*{ \\sqrt{\\frac{\\log\\frac{1}{\\delta}}{T}}, 1 }\n            }\n    \\]\n    for all $\\delta \\le \\frac{1}{5}$. \n","sha256_utf8":"a7375841452ca3f31c48bf143baf10d38f9823b1d2bab8b401c99e52b86ec727"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"独立坐标乘积构造","source_units":[{"start":62330,"end":63043,"start_line":1204,"end_line":1209,"raw_tex":"    To extend the lower bound to $\\delta \\in (\\frac{1}{5}, \\frac{1}{2})$,\n    %\n    we only need to show that $\\ErrCustom{1/2}{\\alg}{\\Iclass[\\Lip]^{L,R}} = \\Omega(LR/\\sqrt{T})$. We do this with a simple ``anti-boosting'' argument. Let $k=4$ and consider $k$ copies of the construction above for $\\delta'=\\frac{1}{5}$, each operating on independent samples and independent coordinates, scaled so that the sum of the copies is in $\\Iclass[\\Lip]^{L,R}$. Concretely, the construction uses $\\xset=\\R^k$ and $\\sset=\\{0,1\\}^k$, and given $f:\\R\\times \\{0,1\\}\\to \\R$ described above, defines $\\tilde{f}: \\xset\\times\\sset\\to\\R$ such that\n    \\[\n        \\tilde{f}(x,s) = k^{-3/2}\\sum_{i \\in [k]} f(x_i \\sqrt{k};s_i).\n    \\]\n","sha256_utf8":"395e5e0fa541159ce7f6a4360f71d8a8cc1a800e50d9e18a9670bb655bd7da90"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"反放大论证扩展置信范围","source_units":[{"start":63043,"end":63686,"start_line":1210,"end_line":1216,"raw_tex":"    For $v=(v_1,\\ldots,v_k)\\in\\{0,1\\}^k$ and $P=P_{v_1}\\times \\cdots \\times P_{v_k}$, this construction has optimality gap\n    \\[\n        F_{f,\\Pv}(x) - \\inf_{\\xopt\\in\\xset} F_{f,\\Pv}(\\xopt)\n        \\ge \\tfrac{1}{k^{3/2}}\\eps LR \\cdot  \\norm*{x \\sqrt{k} - (2v-1)R}_1.\n    \\]\n    Therefore, disagreeing with the sign of $\\xopt$ in even a single coordinate implies optimality gap of at least $k^{-3/2}\\eps LR = \\Omega(LR/\\sqrt{T})$ by our choice of $\\eps$. By the discussion above, the probability of mistaking at least one of the coordinates is at least $1-(1-\\delta')^k > \\half$ for $\\delta' = 1/5$ and $k=4$, completing the proof.\n\\end{proof}","sha256_utf8":"1fcfd5a203e49dbcb81b00900f1e1d2e5c6a53f42bbfdf2213d83ffe888e40cd"}]},{"id":"S040","semantic_type":"lemma","kind":"lemma","title":"偏置伯努利先验估计困难性","source_units":[{"start":43002,"end":43495,"start_line":866,"end_line":878,"raw_tex":"\\begin{lem}\\label{lem:skewed-binary-bound}\n\tFor any $p\\in(0,\\frac{1}{2}]$ and $T\\in \\N$, let $V\\sim \\bernoulli(p)$ and\n\t%\n\t%\n\t%\n\t\\[\n\t\\eps = \\min\\crl*{\\sqrt{\\frac{\\log \\frac{1-p}{p}}{8T}}, \\half}.\t\n\t\\]\n\tLet $P_0 = \\bernoulli(\\frac{1-\\eps}{2})$ and $P_1 = \\bernoulli(\\frac{1+\\eps}{2})$. Then, for $S_1, \\ldots, S_T \\simiid P_V$ and (potentially randomized) estimator $\\hat{V}$ that depends on $V$ only through $S_1, \\ldots, S_T$, we have %\n\t\\[\n\t\t\\Pr(\\hat{V} \\ne V) \\ge \\frac{p}{2}.\n\t\\]\n\\end{lem}","sha256_utf8":"937913e8d3aae8ca814564552748eebe478e13538dfd8800338f18db5058678b"}]},{"id":"S041","semantic_type":"proof","kind":"proof","title":"偏置先验困难性完整证明","source_units":[{"start":43496,"end":44330,"start_line":879,"end_line":897,"raw_tex":"\\begin{proof}\n\tTo streamline notation, we write \n\t\\[\n\t\t%\n\t\t%\n\t\tq \\defeq \\Pr(\\hat{V} \\ne V). \n\t\\] \n\tBy Fano's inequality (\\Cref{fact:fano}) with $n=2$,\n\t\\[\n\t\th_2(q) \\ge H(V\\mid \\hat{V}) =  H(V) - I(V;\\hat{V}).\t\n\t\\numberthis \\label{eq:skewed-fano}\n\t\\]\n\tby \\cref{eq:general-mi-bound-2} in \\Cref{lem:general-mi-bound}, \n    \\[\n\t\tI(\\hat{V}; V) \\le 4p\\eps^2 T \\overle{(i)} \\frac{p}{2} \\log\\frac{1-p}{p} \\overeq{(ii)} \\prn*{p - \\frac{p}{2}}h_2'(p) \\overle{(iii)} h_2(p) - h_2(p/2), \n\t\t%\n\t\\]\n\tdue to $(i)$ our choice of $\\eps \\le \\sqrt{\\frac{\\log \\frac{1-p}{p}}{8T}}$, (ii) the fact that $h_2'(x) = \\log\\frac{1-x}{x}$ and $(iii)$ concavity of entropy. Noting that $H(V) = h_2(p)$ and substituting back to~\\eqref{eq:skewed-fano}, we get that $h_2(q) \\ge h_2(p/2)$, which implies $q\\ge p/2$ since $h_2$ is monotone in $[0, \\half]$. \n\\end{proof}","sha256_utf8":"5a4fd246e85e2a50897f6b3d179168662d6964cb3806522d1bd92a5fa83bbac4"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"用法诺不等式控制错误熵","source_units":[{"start":43496,"end":43746,"start_line":879,"end_line":890,"raw_tex":"\\begin{proof}\n\tTo streamline notation, we write \n\t\\[\n\t\t%\n\t\t%\n\t\tq \\defeq \\Pr(\\hat{V} \\ne V). \n\t\\] \n\tBy Fano's inequality (\\Cref{fact:fano}) with $n=2$,\n\t\\[\n\t\th_2(q) \\ge H(V\\mid \\hat{V}) =  H(V) - I(V;\\hat{V}).\t\n\t\\numberthis \\label{eq:skewed-fano}\n\t\\]\n","sha256_utf8":"3de8aa805c33e406d11efd82065d6c434ad3d60bf924bcaabdce25e821b0c8d7"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"互信息与熵凹性给出错误下界","source_units":[{"start":43746,"end":44330,"start_line":891,"end_line":897,"raw_tex":"\tby \\cref{eq:general-mi-bound-2} in \\Cref{lem:general-mi-bound}, \n    \\[\n\t\tI(\\hat{V}; V) \\le 4p\\eps^2 T \\overle{(i)} \\frac{p}{2} \\log\\frac{1-p}{p} \\overeq{(ii)} \\prn*{p - \\frac{p}{2}}h_2'(p) \\overle{(iii)} h_2(p) - h_2(p/2), \n\t\t%\n\t\\]\n\tdue to $(i)$ our choice of $\\eps \\le \\sqrt{\\frac{\\log \\frac{1-p}{p}}{8T}}$, (ii) the fact that $h_2'(x) = \\log\\frac{1-x}{x}$ and $(iii)$ concavity of entropy. Noting that $H(V) = h_2(p)$ and substituting back to~\\eqref{eq:skewed-fano}, we get that $h_2(q) \\ge h_2(p/2)$, which implies $q\\ge p/2$ since $h_2$ is monotone in $[0, \\half]$. \n\\end{proof}","sha256_utf8":"f0780af187e3933d5b2e516f6e872b765af3570a4fc477441221f68ea7be6a10"}]},{"id":"S044","semantic_type":"lemma","kind":"lemma","title":"二元观测互信息上界","source_units":[{"start":39460,"end":40368,"start_line":798,"end_line":816,"raw_tex":"\\begin{lem}\\label{lem:general-mi-bound}\n    For any $T\\in \\N$ and $\\eps \\in [0,1]$ and any random variable $V$, let $S_1,\\ldots, S_T$ be a sequence of binary random variables such that, for all $t\\le T$ \\yair{could also write it as a TV distance bound}\n    %\n    \\[\n        \\abs*{\\Pr(S_t = 0| S_1, \\ldots, S_{t-1}, V) - \\half} \\le \\frac{\\eps}{2}\n    \\]\n     with probability 1 w.r.t.\\ $S_1,\\ldots, S_{t-1}$ and $V$. Then, for every randomized estimator $\\hat{V}$ that depends on $V$ only through $S_1, \\ldots, S_T$, we have\n    \\[\n        I(V;\\hat{V}) \\le \\eps^2 T.\\numberthis \\label{eq:general-mi-bound-1}\n    \\]\n    If in addition $\\eps \\le \\half$ and, for some $p\\in [0,\\half]$, we have that\n    \\[\n        \\abs*{\\Pr(S_t = 0) - \\half} \\ge \\eps \\prn*{\\frac{1}{2} - p}\n    \\]\n    for every $t\\le T$, then\n    \\[\n        I(V;\\hat{V}) \\le 4p \\eps^2 T.\\numberthis \\label{eq:general-mi-bound-2}\n    \\]\n\\end{lem}","sha256_utf8":"c870125d66ba52735ad04a246b1e548ac3bec6887bc24ddf23d4af44ed5d6a14"}]},{"id":"S045","semantic_type":"proof","kind":"proof","title":"互信息上界完整证明","source_units":[{"start":40370,"end":42907,"start_line":818,"end_line":862,"raw_tex":"\\begin{proof}\n    By the data processing inequality \\citep[Theorem 2.8.1]{cover1991elements} and chain rule for mutual information \\cite[Theorem 2.5.2]{cover1991elements}, \n    %\n\t\\begin{flalign*}\n\t\tI(\\hat{V}; V) & \\le I(S_1, \\ldots, S_T; V) \n\t\t= \\sum_{t=1}^{T} \\brk[\\Big]{ H(S_t \\mid S_1, \\ldots, S_{t-1}) - H(S_t \\mid S_1, \\ldots, S_{t-1}, V) }. \\numberthis \\label{eq:log-mi-expression}\n\t\\end{flalign*}\n\tUsing the facts conditioning decreases entropy and that binary random variables have entropy at most $\\log 2$, we get\n    \\[\n\t\tH(S_t \\mid S_1, \\ldots, S_{t-1}) \\le H(S_t) \\le \\log 2.\n\t\\]\n    Next, we have \n    \\[\n\t\tH(S_t \\mid S_1, \\ldots, S_{t-1}, V) = \\E_{S_1, \\ldots, S_{t-1}, V} \\brk*{h_2 \\prn[\\Big]{\\Pr(S_t = 0| S_1, \\ldots, S_{t-1}, V) }}\n         \\ge  h_2\\prn*{\\frac{1-\\eps}{2}},\n    \\]\n    by the assumption that $\\abs*{\\Pr(S_t = 0| S_1, \\ldots, S_{t-1}, V) - \\half} \\le \\frac{\\eps}{2}$ with probability 1 and the fact that $h_2(x)$ has a single maximum at $x=\\half$.\n\n    Combining the last two displays and using $\\log(1+x)\\le x$, we obtain\n    \\begin{flalign*}\n        H(S_t \\mid S_1, \\ldots, S_{t-1}) &- H(S_t \\mid S_1, \\ldots, S_{t-1}, V) \\\\& \n        \\le \\log 2 - h_2\\prn*{\\frac{1-\\eps}{2}}\n        \\\\ &=\n\t\t\\frac{1+\\veps}{2} \\log(1+\\veps) + \\frac{1-\\veps}{2} \\log(1-\\veps) \n        \\le \n\t\t\\veps^2,\n    \\end{flalign*}\n    where the last inequality is due to $\\log(1+x)\\le x$.\n    Substituting back into~\\cref{eq:log-mi-expression} yields the first claimed bound, $I(\\hat{V}; V) \\le \\eps^2 T$. \n\n    To obtain the second claimed bound we use the assumption that $\\abs*{\\Pr(S_t = 0) - \\half} \\ge \\eps \\prn*{\\frac{1}{2} - p}$ to write\n    \\[\n\t\tH(S_t \\mid S_1, \\ldots, S_{t-1}) \\le H(S_t) \n        \\le h_2\\prn*{\\frac{1-(1-2p)\\eps}{2}},\n    \\]\n    improving the bound on $H(S_t \\mid S_1, \\ldots, S_{t-1}) - H(S_t \\mid S_1, \\ldots, S_{t-1}, V)$ to \n    \\begin{flalign*}\n        h_2\\prn*{\\frac{1-(1-2p)\\eps}{2}} - h_2\\prn*{\\frac{1-\\eps}{2}}\n         & \\overle{(i)} \n        \\prn*{\\frac{1-(1-2p)\\eps}{2} - \\frac{1-\\eps}{2}} h_2'\\prn*{\\frac{1-\\eps}{2}}\n        \\\\ & \\overeq{(ii)} \n        p \\eps \\log\\prn*{1+\\frac{2\\eps}{1-\\eps}} \\overle{(iii)} \\frac{2p\\eps^2}{1-\\eps} \\overle{(iv)} 4p\\eps^2,\n    \\end{flalign*}\n    due to $(i)$ concavity of binary entropy, $(ii)$ the fact that $h_2'(x) = \\log\\prn*{\\frac{1-x}{x}}$, $(iii)$ the inequality $\\log(1+x)\\le x$ and $(iv)$ our assumption that $\\veps \\le \\half$. Substituting back into~\\eqref{eq:log-mi-expression}, we get that $I(\\hat{V}; V) \\le 4p\\eps^2 T$, as required.\n\\end{proof}","sha256_utf8":"0a404dbb60a6dc759d96d65c11c8d5eec29a0104d5c1fa1fc9cb391d649e38cb"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"链式展开与条件熵界","source_units":[{"start":40370,"end":41351,"start_line":818,"end_line":834,"raw_tex":"\\begin{proof}\n    By the data processing inequality \\citep[Theorem 2.8.1]{cover1991elements} and chain rule for mutual information \\cite[Theorem 2.5.2]{cover1991elements}, \n    %\n\t\\begin{flalign*}\n\t\tI(\\hat{V}; V) & \\le I(S_1, \\ldots, S_T; V) \n\t\t= \\sum_{t=1}^{T} \\brk[\\Big]{ H(S_t \\mid S_1, \\ldots, S_{t-1}) - H(S_t \\mid S_1, \\ldots, S_{t-1}, V) }. \\numberthis \\label{eq:log-mi-expression}\n\t\\end{flalign*}\n\tUsing the facts conditioning decreases entropy and that binary random variables have entropy at most $\\log 2$, we get\n    \\[\n\t\tH(S_t \\mid S_1, \\ldots, S_{t-1}) \\le H(S_t) \\le \\log 2.\n\t\\]\n    Next, we have \n    \\[\n\t\tH(S_t \\mid S_1, \\ldots, S_{t-1}, V) = \\E_{S_1, \\ldots, S_{t-1}, V} \\brk*{h_2 \\prn[\\Big]{\\Pr(S_t = 0| S_1, \\ldots, S_{t-1}, V) }}\n         \\ge  h_2\\prn*{\\frac{1-\\eps}{2}},\n    \\]\n    by the assumption that $\\abs*{\\Pr(S_t = 0| S_1, \\ldots, S_{t-1}, V) - \\half} \\le \\frac{\\eps}{2}$ with probability 1 and the fact that $h_2(x)$ has a single maximum at $x=\\half$.\n","sha256_utf8":"73fd7b2d605a81d55d3ec2a6e56766223344ce573546e06e6f976dce988721f3"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"推导一般互信息界","source_units":[{"start":41352,"end":41882,"start_line":836,"end_line":846,"raw_tex":"    Combining the last two displays and using $\\log(1+x)\\le x$, we obtain\n    \\begin{flalign*}\n        H(S_t \\mid S_1, \\ldots, S_{t-1}) &- H(S_t \\mid S_1, \\ldots, S_{t-1}, V) \\\\& \n        \\le \\log 2 - h_2\\prn*{\\frac{1-\\eps}{2}}\n        \\\\ &=\n\t\t\\frac{1+\\veps}{2} \\log(1+\\veps) + \\frac{1-\\veps}{2} \\log(1-\\veps) \n        \\le \n\t\t\\veps^2,\n    \\end{flalign*}\n    where the last inequality is due to $\\log(1+x)\\le x$.\n    Substituting back into~\\cref{eq:log-mi-expression} yields the first claimed bound, $I(\\hat{V}; V) \\le \\eps^2 T$. \n","sha256_utf8":"9bf655d14004defb446efe94d323abe9bdac6e57fb280f168e51938203db6dc7"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"利用边缘偏置改进互信息界","source_units":[{"start":41883,"end":42907,"start_line":848,"end_line":862,"raw_tex":"    To obtain the second claimed bound we use the assumption that $\\abs*{\\Pr(S_t = 0) - \\half} \\ge \\eps \\prn*{\\frac{1}{2} - p}$ to write\n    \\[\n\t\tH(S_t \\mid S_1, \\ldots, S_{t-1}) \\le H(S_t) \n        \\le h_2\\prn*{\\frac{1-(1-2p)\\eps}{2}},\n    \\]\n    improving the bound on $H(S_t \\mid S_1, \\ldots, S_{t-1}) - H(S_t \\mid S_1, \\ldots, S_{t-1}, V)$ to \n    \\begin{flalign*}\n        h_2\\prn*{\\frac{1-(1-2p)\\eps}{2}} - h_2\\prn*{\\frac{1-\\eps}{2}}\n         & \\overle{(i)} \n        \\prn*{\\frac{1-(1-2p)\\eps}{2} - \\frac{1-\\eps}{2}} h_2'\\prn*{\\frac{1-\\eps}{2}}\n        \\\\ & \\overeq{(ii)} \n        p \\eps \\log\\prn*{1+\\frac{2\\eps}{1-\\eps}} \\overle{(iii)} \\frac{2p\\eps^2}{1-\\eps} \\overle{(iv)} 4p\\eps^2,\n    \\end{flalign*}\n    due to $(i)$ concavity of binary entropy, $(ii)$ the fact that $h_2'(x) = \\log\\prn*{\\frac{1-x}{x}}$, $(iii)$ the inequality $\\log(1+x)\\le x$ and $(iv)$ our assumption that $\\veps \\le \\half$. Substituting back into~\\eqref{eq:log-mi-expression}, we get that $I(\\hat{V}; V) \\le 4p\\eps^2 T$, as required.\n\\end{proof}","sha256_utf8":"b19fb61b5fcaf63e57a7eb7705c0ca1bf15a6f22d0720f22022c9a795e85bb20"}]},{"id":"S049","semantic_type":"proof","kind":"proof","title":"随机梯度期望界推导","source_units":[{"start":102360,"end":102974,"start_line":2017,"end_line":2026,"raw_tex":"Let $\\sgd$ denote fixed-steps size SGD with step size $\\eta$ (outputting the iterate average). Then, for $(f,P) \\in \\Iclass[\\SMLip]^{L,R}$ it is well-known the iterates of SGD satisfy \\cite[see, e.g.,][Theorem 2.13]{orabona2021modern}:\n\\[\n\\frac{1}{T} \\sum_{t=0}^{T-1} \\Delta_t \\le \\frac{\\| x_0 - x_\\star \\|^2}{2 \\eta T} + \\frac{\\eta}{2 T} \\sum_{t=0}^{T-1} \\norm{g_t}^2\n\\] \nwhere $x_\\star$ is any optimal solution.\nBy online-to-batch conversion \\cite[Chapter 3]{orabona2021modern} we get\n\\[\n\\ErrE{\\sgd}{\\Iclass[\\SMLip]^{L,R}} \\le \\frac{R^2}{2\\eta T} + \\frac{\\eta L^2}{2}.\n\\numberthis \\label{eq:sgd-expected-err}\n\\]\n","sha256_utf8":"a9a7b3ef12705d2c6cd3953857b3d58c80a0e89a47551e11e61638dedc02b487"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"固定步长随机梯度遗憾界","source_units":[{"start":102360,"end":102774,"start_line":2017,"end_line":2021,"raw_tex":"Let $\\sgd$ denote fixed-steps size SGD with step size $\\eta$ (outputting the iterate average). Then, for $(f,P) \\in \\Iclass[\\SMLip]^{L,R}$ it is well-known the iterates of SGD satisfy \\cite[see, e.g.,][Theorem 2.13]{orabona2021modern}:\n\\[\n\\frac{1}{T} \\sum_{t=0}^{T-1} \\Delta_t \\le \\frac{\\| x_0 - x_\\star \\|^2}{2 \\eta T} + \\frac{\\eta}{2 T} \\sum_{t=0}^{T-1} \\norm{g_t}^2\n\\] \nwhere $x_\\star$ is any optimal solution.\n","sha256_utf8":"5277bf9a1f34eb18816737da2afd515f358881d94e4a20f17fe87141788c670b"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"在线转批量得到期望误差","source_units":[{"start":102774,"end":102974,"start_line":2022,"end_line":2026,"raw_tex":"By online-to-batch conversion \\cite[Chapter 3]{orabona2021modern} we get\n\\[\n\\ErrE{\\sgd}{\\Iclass[\\SMLip]^{L,R}} \\le \\frac{R^2}{2\\eta T} + \\frac{\\eta L^2}{2}.\n\\numberthis \\label{eq:sgd-expected-err}\n\\]\n","sha256_utf8":"cb0024a2e2376fdd4dc110d9599330081e8d19ec9db5c78236b92274c169af22"}]},{"id":"S052","semantic_type":"external_fact","kind":"external_fact","title":"法诺不等式","source_units":[],"declared_external":true,"declaration_text":["If V takes n values and Vhat is an estimator, then h_2(P(Vhat != V)) + P(Vhat != V) log(n-1) >= H(V | Vhat). This is the standard cited boundary fact fact:fano."]},{"id":"S053","semantic_type":"external_fact","kind":"external_fact","title":"弗里德曼集中不等式","source_units":[],"declared_external":true,"declaration_text":["The simplified Freedman inequality used in fact:freedman: if E[X_t^2 | X_1,...,X_{t-1}] <= A^2 and |X_t| <= B almost surely, then P(sum_t X_t > sum_t E[X_t | X_1,...,X_{t-1}] + A sqrt(2T log(1/delta)) + (B/3) log(1/delta)) <= delta."]},{"id":"S054","semantic_type":"external_fact","kind":"external_fact","title":"信息论基本恒等式","source_units":[],"declared_external":true,"declaration_text":["Data processing bounds I(V; Vhat) by I(V; S_1,...,S_T) when the estimator depends on V only through the observations. The mutual-information chain rule expresses the latter as the sum of conditional entropy differences. Conditioning cannot increase entropy. These are the standard facts cited from Cover and Thomas, Theorems 2.8.1 and 2.5.2."]},{"id":"S055","semantic_type":"external_fact","kind":"external_fact","title":"投影梯度遗憾界","source_units":[],"declared_external":true,"declaration_text":["For Euclidean projected gradient updates with step size eta and feasible comparator xstar, sum_t <g_t, x_t-xstar> <= ||x_0-xstar||^2/(2 eta) + (eta/2) sum_t ||g_t||^2. For convex sample losses, their loss differences are bounded by these inner products. This is the standard regret fact cited from Orabona, Lemma 2.12 and Theorem 2.13."]},{"id":"S056","semantic_type":"external_fact","kind":"external_fact","title":"在线转批量原理","source_units":[],"declared_external":true,"declaration_text":["For convex stochastic objectives with fresh independent samples, the expected population suboptimality of the iterate average is bounded by the expected average sample regret against a fixed population minimizer. This is the online-to-batch conversion cited from Orabona, Chapter 3."]},{"id":"S057","semantic_type":"external_fact","kind":"external_fact","title":"经典期望极小极大下界","source_units":[],"declared_external":true,"declaration_text":["The classical known-parameter lower bound cited in B0065 is Err_T^E(A_SO, I_Lip^{L,R}) >= c LR/sqrt(T), for a universal positive constant c; the author cites Nemirovski and Yudin and Duchi, Theorem 5.2.10."]},{"id":"S058","semantic_type":"external_fact","kind":"external_fact","title":"基础凸性概率与熵性质","source_units":[],"declared_external":true,"declaration_text":["The selected arguments use the convex subgradient inequality, Jensen's inequality, Cauchy-Schwarz, the triangle inequality, the union bound, standard quantile properties, and elementary binary-entropy calculus: h_2 is concave, symmetric about 1/2, increasing on [0,1/2], bounded above by log 2, and h_2'(p)=log((1-p)/p). They also use log(1+x)<=x on its domain and elementary product-probability calculations for independent coordinates."]},{"id":"S059","semantic_type":"assumption","kind":"assumption","title":"预算距离与算法假设","source_units":[{"start":68546,"end":68602,"start_line":1284,"end_line":1284,"raw_tex":"For all $T\\in\\N$, $\\rho \\ge 1$, $\\alg\\in\\Aall$ we have \n","sha256_utf8":"f85f3ecb0d51985b22e4b0c99428e6e3de71f84ef7313c50924ee36aec15b3c9"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S012","edges":[{"source_id":"S003","target_id":"S006","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S003","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S006","target_id":"S009","kind":"supports"},{"source_id":"S007","target_id":"S014","kind":"supports"},{"source_id":"S008","target_id":"S014","kind":"supports"},{"source_id":"S009","target_id":"S014","kind":"supports"},{"source_id":"S019","target_id":"S014","kind":"supports"},{"source_id":"S059","target_id":"S014","kind":"supports"},{"source_id":"S001","target_id":"S015","kind":"supports"},{"source_id":"S003","target_id":"S015","kind":"supports"},{"source_id":"S059","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S004","target_id":"S016","kind":"supports"},{"source_id":"S005","target_id":"S016","kind":"supports"},{"source_id":"S014","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S040","target_id":"S018","kind":"supports"},{"source_id":"S059","target_id":"S018","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S015","target_id":"S013","kind":"supports"},{"source_id":"S016","target_id":"S013","kind":"supports"},{"source_id":"S017","target_id":"S013","kind":"supports"},{"source_id":"S018","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S003","target_id":"S021","kind":"supports"},{"source_id":"S004","target_id":"S021","kind":"supports"},{"source_id":"S006","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S023","kind":"supports"},{"source_id":"S049","target_id":"S022","kind":"supports"},{"source_id":"S057","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S033","target_id":"S023","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S020","kind":"supports"},{"source_id":"S023","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S003","target_id":"S010","kind":"supports"},{"source_id":"S001","target_id":"S010","kind":"supports"},{"source_id":"S010","target_id":"S026","kind":"supports"},{"source_id":"S003","target_id":"S026","kind":"supports"},{"source_id":"S055","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S058","target_id":"S027","kind":"supports"},{"source_id":"S010","target_id":"S028","kind":"supports"},{"source_id":"S003","target_id":"S028","kind":"supports"},{"source_id":"S058","target_id":"S028","kind":"supports"},{"source_id":"S010","target_id":"S029","kind":"supports"},{"source_id":"S003","target_id":"S029","kind":"supports"},{"source_id":"S058","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S053","target_id":"S030","kind":"supports"},{"source_id":"S010","target_id":"S031","kind":"supports"},{"source_id":"S003","target_id":"S031","kind":"supports"},{"source_id":"S053","target_id":"S031","kind":"supports"},{"source_id":"S026","target_id":"S032","kind":"supports"},{"source_id":"S027","target_id":"S032","kind":"supports"},{"source_id":"S028","target_id":"S032","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S058","target_id":"S032","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S027","target_id":"S025","kind":"supports"},{"source_id":"S028","target_id":"S025","kind":"supports"},{"source_id":"S029","target_id":"S025","kind":"supports"},{"source_id":"S030","target_id":"S025","kind":"supports"},{"source_id":"S031","target_id":"S025","kind":"supports"},{"source_id":"S032","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S003","target_id":"S035","kind":"supports"},{"source_id":"S001","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S004","target_id":"S036","kind":"supports"},{"source_id":"S005","target_id":"S036","kind":"supports"},{"source_id":"S006","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S040","target_id":"S037","kind":"supports"},{"source_id":"S035","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S003","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S058","target_id":"S039","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S034","kind":"supports"},{"source_id":"S037","target_id":"S034","kind":"supports"},{"source_id":"S038","target_id":"S034","kind":"supports"},{"source_id":"S039","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S002","target_id":"S042","kind":"supports"},{"source_id":"S052","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S058","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S043","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S002","target_id":"S046","kind":"supports"},{"source_id":"S054","target_id":"S046","kind":"supports"},{"source_id":"S058","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S058","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S058","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S048","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S011","target_id":"S050","kind":"supports"},{"source_id":"S055","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S056","target_id":"S051","kind":"supports"},{"source_id":"S003","target_id":"S051","kind":"supports"},{"source_id":"S005","target_id":"S051","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S051","target_id":"S049","kind":"supports"}]}}
{"record_id":"2402.15646v2","paper":{"title":"Stochastic ISTA/FISTA Adaptive Step Search Algorithms for Convex Composite Optimization","authors":["Lam M. Nguyen","Katya Scheinberg","Trang H. Tran"],"arxiv_id":"2402.15646","version":"v2","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-025-02621-8","abstract_url":"https://arxiv.org/abs/2402.15646v2","pdf_url":"https://arxiv.org/pdf/2402.15646v2"},"context":{"raw_tex":"\\begin{align*}\n\\min_{x \\in \\R^n} F(x) = f(x) + h(x),  %\\label{eq:main_prob}\n\\end{align*}\n\n\\begin{assumption}\\label{assumpt:convex_smooth}\n Function  $f$ is convex and $L$-smooth, function $h$ is proper, closed and convex.\n\\end{assumption}\n\n\\begin{assumption}\\label{assumpt:bounded_below}\nThere exists a global minimizer of $F(x)$, $x^*$,  such that $F(x^*)=F^* > -\\infty$. \n  \\end{assumption}\n\n\\begin{align}\n\\text{prox}_h (y) = {\\arg \\min}_{x \\in \\R^n} \\left\\{h(x) +\\frac{1}{2}\\|x-y\\|^2\\right\\},  \\label{eq:prox_oper}\n\\end{align}\n\nGiven a  point $y\\in\\R^n$ and a step-size parameter $\\alpha$, we define the following approximation $Q_\\alpha(x, y)$ of the objective function $F$: \n\\begin{align*}\nQ_\\alpha(x, y) = f(y) +\\nabla f(y)^\\top (x -y)  + \\frac{1}{2\\alpha}\\|x-y\\|^2 + h(x).\n\\end{align*}\n\nThe minimizer of this function with respect to $x$, which we denote by $p_\\alpha (y)$ is given by the prox-operator: \n\\begin{align*}\n    p_\\alpha (y) := \\arg\\min_x Q_\\alpha(x, y) = \\text{prox}_{\\alpha h} (y - \\alpha \\nabla f(y)).\n\\end{align*}\n\nWe let $D_{\\alpha}(y)$ denote the gradient mapping of $F$ \\cite{Nesterov}:\n\\begin{align}\\label{eq:gradient_mapping}\nD_{\\alpha}(y) := \\frac{1}{\\alpha}\\Big(y - \\text{prox}_{\\alpha h}(y - \\alpha\\nabla{f}(y))\\Big)= \\frac{1}{\\alpha}\\Big(y - p_\\alpha (y)\\Big), \\text{ for } \\alpha > 0.\n\\end{align}\n\nWe also define the following approximating function based on an estimate  $g$ instead of  $\\nabla f(y)$:\n\\begin{align*}\n\\tilde{Q}_\\alpha(x, y) = f(y) + g^\\top (x -y)  + \\frac{1}{2\\alpha}\\|x-y\\|^2 + h(x).\n\\end{align*}\n\nSimilarly we define\n\\begin{align*}\n    \\tilde{p}_\\alpha (y) := \\arg\\min_x \\tilde{Q}_\\alpha(x, y) = \\text{prox}_{\\alpha h} (y - \\alpha g).\n\\end{align*}\n\n{\\bf Stochastic first-order oracle SFO$_{\\constG, p}(y,\\alpha)$.}\t \n{\\em Given a vector $y\\in \\R^n$ and a step-size parameter $\\alpha > 0$, the oracle computes $G(y,\\xi)$, a (random) estimate of the gradient $\\nabla f (y)$, such that\n\\begin{align}\\label{eq:first_order1}\n\\mathbb P_{\\xi} \\left\\{\\|G(y,\\xi)-\\nabla f(y)\\|\\leq \\constG \\cdot  \\|D_{\\alpha}(y)\\|   \\right\\}\n\\geq p, \n\\end{align}\n\nwhere $\\xi$ is a random variable whose distribution may depend on $y$ and $D_{\\alpha}(y)$ is defined in \\eqref{eq:gradient_mapping}. The values of the constants  $\\constG$ and $p$ are intrinsic to the oracle, and satisfy  {$\\constG \\leq \\frac{1}{3}$} and $p > \\frac{1}{2}$.\n\n\n\\begin{algorithm}[ht]\n\\caption{Stochastic ISTA Step Search Algorithm}\n\\label{alg:ista_bktr_line_search}\n\\begin{algorithmic}[1]\n\\STATE {Initialization:} \nChoose an initial point $x_0\\in \\mathbb{R}^d$. Set $0<\\gamma<1$,  with $\\alpha_1>0$. Let oracle SFO$_{\\constG, p}$ be given with  {$0 \\leq \\constG \\leq \\frac{1}{3}$} and $\\frac{1}{2} < p \\leq 1 $.\n\\FOR{$k=1,2,\\cdots,$}\n    \\STATE Set  $y_k = x_{k-1}$.\n    \\STATE Compute the gradient estimate $g_k=G(y_k,\\xi_k)$ using the stochastic first-order oracle SFO$_{\\constG, p}(y_k,\\alpha_k)$.\n    Compute the reference point $\\tilde{p}_{\\alpha_k} (y_k) =\\text{prox}_{\\alpha_k h} (y_k- \\alpha_k g_k)$.\n    \\STATE Check the sufficient decrease condition $F(\\tilde{p}_{\\alpha_k} (y_k)) \\leq \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (y_k), y_k)$.\n        \\STATE \\hspace{1.25em}If unsuccessful: \n            \\STATE \\hspace{2.5em}Set $x_{k} = x_{k-1}$ and  $\\alpha_{k+1} = \\gamma\\alpha_k$.\n        \\STATE \\hspace{1.25em}If successful: \n            \\STATE \\hspace{2.5em}Set $x_k = \\tilde{p}_{\\alpha_k} (y_k)$ and $\\alpha_{k+1} = \\tfrac{\\alpha_k}{\\gamma}$.\n\\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}\n\n\\textit{Filtration.}\nAlgorithm \\ref{alg:ista_bktr_line_search} generates a stochastic process $\\{Y_k, X_k, G_k, \\Stepsize{k}\\}$, where $G_k$ denotes the random gradient estimate computed by the oracle SFO$_{\\constG, p}(Y_k,\\Stepsize{k})$ at iteration $k$, $Y_k$ and $X_k$ are the random iterates and  $\\Stepsize{k}$ is the random step-size  parameter. \nWe use $g_k$, $y_k$, $x_k$ and $\\alpha_k$  to denotes realizations of these quantities for a particular sample path. We omit explicitly stating the dependence on the sample, for brevity. \nLet $\\pastone$ denote the $\\sigma$-algebra generated by the random variables $G_0, \\hdots, G_{k-1}$. We note that by the nature of the algorithm, distribution of $G_k$ is determined by $Y_k$, while  $X_k$,  $\\Stepsize{k+1}$, and, consequently,  $Y_{k+1}$ are  deterministic conditioned on $G_k$. Therefore  process $\\{G_k\\}$ captures the randomness of the stochastic process $\\{Y_k, X_k, G_k,  \\Stepsize{k}\\}$, in other words,  $\\{\\pastone\\}_{k\\ge 1}$ is a filtration. \n\n\n\\textit{The hitting time $N_\\epsilon$ to reach $\\epsilon$ accurate solution.} \nGiven a target  accuracy $\\epsilon>0$, we aim to derive a bound on the expected number of iterations after which\nthe algorithm generates the iterate that satisfies $\\{F(x_k)-F^*\\leq \\epsilon\\}$. We denote the optimality gap $v_k = F(x_k)-F^*$ for every iteration $k$.\nLet  $N_\\epsilon=\\min\\{k=1,2, \\ldots\\ : F(X_k)-F^*\\leq \\epsilon\\}$, that is $N_\\epsilon$ is the first iteration for which the event $\\{F(x_k)-F^*\\leq \\epsilon\\}$ occurs.  $N_\\epsilon$ is a stopping time (hitting time) with respect to the filtration $\\{\\mathcal{F}_k\\}_{k\\ge 1}$   as it is defined by the realization of $X_k$.\n\n\n\\begin{definition}[Successful Iteration]\\label{def:success} \nLet $\\iterSucc=\\mathbbm{1}\\{{\\rm Iteration\\ } k\\ {\\rm is\\ successful } \\}$.\n\\end{definition}\n\n\\begin{definition}[True Iteration]\\label{def:good_model}\nWe say iteration $k$ is {\\em true} if \n\\begin{align*}\n    \\|g_k-\\nabla f(y_k)\\|\\leq \\constG  \\|D_{\\alpha_k}(y_k)\\|, \\  {\\constG\\leq \\frac{1}{3}}.\n\\end{align*}\n\nIn other words, the first-order oracle delivers a sufficiently accurate estimate. \nWe use an indicator variable $\\iterTrue$ to indicate that iteration $k$ is true. \n\\end{definition}\n\n\\begin{assumption}[Probability of True Iteration]\\label{assumpt:key}\nWe assume that the first-order oracle  SFO$_{\\constG, p}$ is implemented in Algorithm \\ref{alg:ista_bktr_line_search} with  {$\\constG \\leq \\frac{1}{3}$} and $p > \\frac{1}{2}$ for every input $(y_k,\\alpha_k)$. \nThen, by the properties of  the oracle and the filtration $\\pastone$ we have, \n\\begin{align*} \n\\Prob(\\iterTrue| \\pastone) \\ge p \\text{ for all } k \\geq 0.\n\\end{align*}\n\\end{assumption}\n\nLet us define\n$N_G = \\sum_{k=1}^{N_\\epsilon-1} \\mathbbm{1}\\{ \\Alpha_k \\ge \\stepsizebar \\} \\cdot  \\mathbbm{1}\\{{\\rm Iteration\\ } k\\ {\\rm is\\ true\\ and \\ successful} \\} $ be the number of {\\em good} iterations prior to the stopping time $N_\\epsilon$.\n\n\n\\begin{assumption}\\label{assumpt:ista_noise}\nAt every iteration $k$,  $g_k=g(y_k,\\xi_k)$ computed using the stochastic first-order oracle SFO$_{\\constG, p}(y_k,\\alpha_k)$ satisfies  \n\\begin{align}\\label{eq:ista_noise}\n    &\\EE_{\\xi_k} \\left[ \\| (\\nabla f(y_{k} )- G_{k} )\\|^2 | \\pastone\\right] \\leq \\frac{1}{\\alpha_{k}^2\\cdot k^{2+ \\beta}},\n\\end{align}\nfor some fixed $\\beta>0$. \n%\\trang{I thought conditioning on the filtration is essentially the same as the randomness in the oracle? since that was the only random element of the algo.} \\katya{Yes, this just makes it more explicit, I think}. \n\\end{assumption}\n\nWe first present that deterministic algorithm and then show how it is modified for the stochastic first-order oracles. Compared to  FISTA  in \\cite{Beck-Teboulle}, FISTA-BKTR in \\cite{Scheinberg-Goldfarb-Bai} uses an additional  parameter $\\theta_k$, which keeps record of the changes in  $\\alpha_k$  and \nutilizes the following update  \n\\begin{align}\\label{eq:fista_step}\n   (t_{new}, y) = \\text{FISTAStep}(x, x^{prev}, t, \\theta).\n\\end{align}\n\nwhere FISTAStep computes \n\\begin{align*}\n    t_{new} &=  \\frac{1 + \\sqrt{1+4\\theta t^2}}{2}, \\text{ and }\n    y = x + \\frac{t-1}{t_{new}} ( x - x^{prev} ).\n\\end{align*}\n\n Here we will use the main FISTA-BKTR  scheme, however, as in the case of stochastic ISTA, the gradient estimate needs to be recomputed each time a step is attempted, even when the iteration is unsuccessful. Therefore the iteration counter gets increased on unsuccessful steps as well as successful ones and the algorithm notation needs to be updated accordingly. In particular \nwe use  $x^{prev}_k$ (instead of $x_{k-1}$) to denote the iterate of the last successful iteration, that is the last iterate different from $x_k$. \nSimilarly, $t_k^{next}$ denotes the value of $t_{k+1}$ that is assigned if iteration $k$ is successful, otherwise $t_{k+1}=t_k$. \nWe also need to keep track of the $\\alpha_k^{succ}$ - the step-size  parameter of the last successful iteration up to iteration $k$. \nOn the other hand, $\\alpha_k^0$ value is not used since the dynamics of the step-size  parameter are simplified - it is decreased by the factor of $\\gamma$ on each unsuccessful iteration and increased proportionally on each successful one. \n\n\n\\begin{algorithm}[ht]\n\\caption{Stochastic FISTA Step Search Algorithm}\n\\label{alg:fista_bktr_line_search}\n\\begin{algorithmic}[1]\n\\STATE {Initialization:} Choose an initial point $x_0\\in \\mathbb{R}^d$. Set $t_0 = 0$, let $x^{prev}_0 = x_0$. \\nonumber\\\\\nSet $0<\\gamma<1$, $\\theta_0 = \\gamma$ with $\\alpha_1>0$.  Let oracle SFO$_{\\constG, p}$ be given with  {$0 \\leq \\constG \\leq \\frac{1}{3}$} and $\\frac{1}{2} < p \\leq 1 $.\n\\FOR{$k=1,2,\\cdots,$}\n    \\STATE Compute $(t_k^{next}, y_k) = \\text{FISTAStep}(x_{k-1}, x^{prev}_{k-1}, t_{k-1}, \\theta_{k-1}) $\n    \\STATE Compute the gradient estimate $g_k=G(y_k,\\xi_k)$ using the stochastic first-order oracle SFO$_{\\constG, p}(y_k,\\alpha_k)$.\n    Compute the reference point $\\tilde{p}_{\\alpha_k} (y_k) =\\text{prox}_{\\alpha_k h} (y_k- \\alpha_k g_k)$.\n    \\STATE Check sufficient decrease condition $F(\\tilde{p}_{\\alpha_k} (y_k)) \\leq \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (y_k), y_k)$\n        \\STATE \\hspace{1.25em}If unsuccessful: \n            \\STATE \\hspace{2.5em}Set $\\left(x_{k},x^{prev}_{k},t_k\\right) = \\left(x_{k-1}, x^{prev}_{k-1}, t_{k-1}\\right)$, then set $\\alpha_{k+1} = \\gamma\\alpha_k$, %$\\alpha^{succ}_{k}=\\alpha^{succ}_{k-1}$ \n            and $\\theta_{k} = \\tfrac{\\theta_{k-1}}{\\gamma}$.\n        \\STATE \\hspace{1.25em}If successful: \n            \\STATE \\hspace{2.5em}Set $\\left(x_k, x^{prev}_{k},t_k\\right) = \\left(\\tilde{p}_{\\alpha_k} (y_k), x_{k-1}, t_k^{next}\\right)$,\n    then set $\\alpha_{k+1} = \\tfrac{\\alpha_k}{\\gamma},$ %$\\alpha^{succ}_{k}=\\alpha_k$ \n    and $\\theta_{k} = \\gamma$.\n\\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}\n\nSimilar to Algorithm \\ref{alg:ista_bktr_line_search}, \nAlgorithm \\ref{alg:fista_bktr_line_search} generates a stochastic process $\\{Y_k, X_k, G_k, \\Stepsize{k}, T_k\\}$ with realizations $\\{y_k, x_k, g_k, \\alpha_k, t_k\\}$. \n\n\n\\begin{align}\\label{eq:define_m}\n    m_k = 2\\alpha^{succ}_{k} t_{k}^2v_{k} +\\norm{u_k }^2,\n\\end{align}\n\nwhere $v_k =  F(x_k) - F^* $ and $u_k = (t_{k}-1)x^{prev}_{k} + x^* - t_k x_k $. \n\n\n\\begin{assumption}\\label{assumpt:fista_noise}\nAt every iteration $k$,  $g_k=g(y_k,\\xi_k)$ computed using the stochastic first-order oracle SFO$_{\\constG, p}(y_k,\\alpha_k)$ satisfies \n\\begin{align}\\label{eq:fista_noise}\n&\\EE_{\\xi_k} \\left[ \\| (\\nabla f(y_{k} )- G_{k} )\\|^2 | \\pastone\\right] \\leq \\frac{1}{\\alpha_{k}^2t_{k}^2\\cdot k^{2+ \\beta}},\n\\end{align}\nfor some fixed $\\beta>0$.\n\\end{assumption}","units":[{"start":4789,"end":4877,"start_line":135,"end_line":137,"raw_tex":"\\begin{align*}\n\\min_{x \\in \\R^n} F(x) = f(x) + h(x),  %\\label{eq:main_prob}\n\\end{align*}","sha256_utf8":"73e2b5ba69857b4986d4b9798ecb8c704ff257f312acb4bc534771d118339f4c"},{"start":4912,"end":5060,"start_line":139,"end_line":141,"raw_tex":"\\begin{assumption}\\label{assumpt:convex_smooth}\n Function  $f$ is convex and $L$-smooth, function $h$ is proper, closed and convex.\n\\end{assumption}","sha256_utf8":"73a24eebe5d017f22222d86baf04942a24fbe40d2d4b4fd8f82be10b0ed179ee"},{"start":5061,"end":5213,"start_line":142,"end_line":144,"raw_tex":"\\begin{assumption}\\label{assumpt:bounded_below}\nThere exists a global minimizer of $F(x)$, $x^*$,  such that $F(x^*)=F^* > -\\infty$. \n  \\end{assumption}","sha256_utf8":"072b84631f3db1b54bd43bca0501c40f4d41db9bcadc4255bee426a9a03f357d"},{"start":5639,"end":5774,"start_line":150,"end_line":152,"raw_tex":"\\begin{align}\n\\text{prox}_h (y) = {\\arg \\min}_{x \\in \\R^n} \\left\\{h(x) +\\frac{1}{2}\\|x-y\\|^2\\right\\},  \\label{eq:prox_oper}\n\\end{align}","sha256_utf8":"6ab2a36d00258db509f67d11e441d0a8675b0655710b15f34bc69d28c1e3e3d7"},{"start":27301,"end":27562,"start_line":254,"end_line":257,"raw_tex":"Given a  point $y\\in\\R^n$ and a step-size parameter $\\alpha$, we define the following approximation $Q_\\alpha(x, y)$ of the objective function $F$: \n\\begin{align*}\nQ_\\alpha(x, y) = f(y) +\\nabla f(y)^\\top (x -y)  + \\frac{1}{2\\alpha}\\|x-y\\|^2 + h(x).\n\\end{align*}","sha256_utf8":"36ec851677677f3b92b04ac9778a2d07cbe0448933970898ebfc5cab54647b4f"},{"start":27563,"end":27805,"start_line":258,"end_line":261,"raw_tex":"The minimizer of this function with respect to $x$, which we denote by $p_\\alpha (y)$ is given by the prox-operator: \n\\begin{align*}\n    p_\\alpha (y) := \\arg\\min_x Q_\\alpha(x, y) = \\text{prox}_{\\alpha h} (y - \\alpha \\nabla f(y)).\n\\end{align*}","sha256_utf8":"8840292fd54bc02254ed886220ef2c29d10b9c688800a71f68babfea42a4ba77"},{"start":27806,"end":28097,"start_line":262,"end_line":265,"raw_tex":"We let $D_{\\alpha}(y)$ denote the gradient mapping of $F$ \\cite{Nesterov}:\n\\begin{align}\\label{eq:gradient_mapping}\nD_{\\alpha}(y) := \\frac{1}{\\alpha}\\Big(y - \\text{prox}_{\\alpha h}(y - \\alpha\\nabla{f}(y))\\Big)= \\frac{1}{\\alpha}\\Big(y - p_\\alpha (y)\\Big), \\text{ for } \\alpha > 0.\n\\end{align}","sha256_utf8":"3ef38b8771e68ebcf300ab34172c2c87882bd834e556e98792b23f68105fdb5c"},{"start":28169,"end":28385,"start_line":267,"end_line":270,"raw_tex":"We also define the following approximating function based on an estimate  $g$ instead of  $\\nabla f(y)$:\n\\begin{align*}\n\\tilde{Q}_\\alpha(x, y) = f(y) + g^\\top (x -y)  + \\frac{1}{2\\alpha}\\|x-y\\|^2 + h(x).\n\\end{align*}","sha256_utf8":"96da3a31082e7232994fef1e0ecc1307f9a0196e2a0d3a7c0ff1dab34b0c69f3"},{"start":28386,"end":28536,"start_line":271,"end_line":274,"raw_tex":"Similarly we define\n\\begin{align*}\n    \\tilde{p}_\\alpha (y) := \\arg\\min_x \\tilde{Q}_\\alpha(x, y) = \\text{prox}_{\\alpha h} (y - \\alpha g).\n\\end{align*}","sha256_utf8":"95bbaaa6b57fffc45add795135a5c9645dbde4d6c5d784d32be40c8dfe0fdfe2"},{"start":29902,"end":30289,"start_line":290,"end_line":295,"raw_tex":"{\\bf Stochastic first-order oracle SFO$_{\\constG, p}(y,\\alpha)$.}\t \n{\\em Given a vector $y\\in \\R^n$ and a step-size parameter $\\alpha > 0$, the oracle computes $G(y,\\xi)$, a (random) estimate of the gradient $\\nabla f (y)$, such that\n\\begin{align}\\label{eq:first_order1}\n\\mathbb P_{\\xi} \\left\\{\\|G(y,\\xi)-\\nabla f(y)\\|\\leq \\constG \\cdot  \\|D_{\\alpha}(y)\\|   \\right\\}\n\\geq p, \n\\end{align}","sha256_utf8":"0d1b68006dffb9c80f21ff3ce855dbd9d507c5c78682758b8b0d1521b21dbab5"},{"start":30290,"end":30564,"start_line":296,"end_line":296,"raw_tex":"where $\\xi$ is a random variable whose distribution may depend on $y$ and $D_{\\alpha}(y)$ is defined in \\eqref{eq:gradient_mapping}. The values of the constants  $\\constG$ and $p$ are intrinsic to the oracle, and satisfy  {$\\constG \\leq \\frac{1}{3}$} and $p > \\frac{1}{2}$.\n","sha256_utf8":"f22c0832d9b9da324ee02ad712c70a7d3196570397002e753d3c67d20bd04381"},{"start":36430,"end":37567,"start_line":328,"end_line":345,"raw_tex":"\\begin{algorithm}[ht]\n\\caption{Stochastic ISTA Step Search Algorithm}\n\\label{alg:ista_bktr_line_search}\n\\begin{algorithmic}[1]\n\\STATE {Initialization:} \nChoose an initial point $x_0\\in \\mathbb{R}^d$. Set $0<\\gamma<1$,  with $\\alpha_1>0$. Let oracle SFO$_{\\constG, p}$ be given with  {$0 \\leq \\constG \\leq \\frac{1}{3}$} and $\\frac{1}{2} < p \\leq 1 $.\n\\FOR{$k=1,2,\\cdots,$}\n    \\STATE Set  $y_k = x_{k-1}$.\n    \\STATE Compute the gradient estimate $g_k=G(y_k,\\xi_k)$ using the stochastic first-order oracle SFO$_{\\constG, p}(y_k,\\alpha_k)$.\n    Compute the reference point $\\tilde{p}_{\\alpha_k} (y_k) =\\text{prox}_{\\alpha_k h} (y_k- \\alpha_k g_k)$.\n    \\STATE Check the sufficient decrease condition $F(\\tilde{p}_{\\alpha_k} (y_k)) \\leq \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (y_k), y_k)$.\n        \\STATE \\hspace{1.25em}If unsuccessful: \n            \\STATE \\hspace{2.5em}Set $x_{k} = x_{k-1}$ and  $\\alpha_{k+1} = \\gamma\\alpha_k$.\n        \\STATE \\hspace{1.25em}If successful: \n            \\STATE \\hspace{2.5em}Set $x_k = \\tilde{p}_{\\alpha_k} (y_k)$ and $\\alpha_{k+1} = \\tfrac{\\alpha_k}{\\gamma}$.\n\\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"1b3c1b6dc2906e4eaf807995986d319b7bb1da16d36c6ecc23f3e6277f1360f1"},{"start":37656,"end":38667,"start_line":349,"end_line":352,"raw_tex":"\\textit{Filtration.}\nAlgorithm \\ref{alg:ista_bktr_line_search} generates a stochastic process $\\{Y_k, X_k, G_k, \\Stepsize{k}\\}$, where $G_k$ denotes the random gradient estimate computed by the oracle SFO$_{\\constG, p}(Y_k,\\Stepsize{k})$ at iteration $k$, $Y_k$ and $X_k$ are the random iterates and  $\\Stepsize{k}$ is the random step-size  parameter. \nWe use $g_k$, $y_k$, $x_k$ and $\\alpha_k$  to denotes realizations of these quantities for a particular sample path. We omit explicitly stating the dependence on the sample, for brevity. \nLet $\\pastone$ denote the $\\sigma$-algebra generated by the random variables $G_0, \\hdots, G_{k-1}$. We note that by the nature of the algorithm, distribution of $G_k$ is determined by $Y_k$, while  $X_k$,  $\\Stepsize{k+1}$, and, consequently,  $Y_{k+1}$ are  deterministic conditioned on $G_k$. Therefore  process $\\{G_k\\}$ captures the randomness of the stochastic process $\\{Y_k, X_k, G_k,  \\Stepsize{k}\\}$, in other words,  $\\{\\pastone\\}_{k\\ge 1}$ is a filtration. \n","sha256_utf8":"9ba92c5f9eb260c6c72f7765f5708aec950e64c6653aa3e63a6c9b0457b72b2b"},{"start":38754,"end":39427,"start_line":357,"end_line":360,"raw_tex":"\\textit{The hitting time $N_\\epsilon$ to reach $\\epsilon$ accurate solution.} \nGiven a target  accuracy $\\epsilon>0$, we aim to derive a bound on the expected number of iterations after which\nthe algorithm generates the iterate that satisfies $\\{F(x_k)-F^*\\leq \\epsilon\\}$. We denote the optimality gap $v_k = F(x_k)-F^*$ for every iteration $k$.\nLet  $N_\\epsilon=\\min\\{k=1,2, \\ldots\\ : F(X_k)-F^*\\leq \\epsilon\\}$, that is $N_\\epsilon$ is the first iteration for which the event $\\{F(x_k)-F^*\\leq \\epsilon\\}$ occurs.  $N_\\epsilon$ is a stopping time (hitting time) with respect to the filtration $\\{\\mathcal{F}_k\\}_{k\\ge 1}$   as it is defined by the realization of $X_k$.\n","sha256_utf8":"7438ec9560a93266337b155302e48926d64d5b20e6966aa5c9181c4922bcf1f3"},{"start":39480,"end":39633,"start_line":364,"end_line":366,"raw_tex":"\\begin{definition}[Successful Iteration]\\label{def:success} \nLet $\\iterSucc=\\mathbbm{1}\\{{\\rm Iteration\\ } k\\ {\\rm is\\ successful } \\}$.\n\\end{definition}","sha256_utf8":"56dc8e30265fa27d299be25f5ce16fda2f3b7c6b5a10928aa520e51d63daabe6"},{"start":39635,"end":40033,"start_line":368,"end_line":376,"raw_tex":"\\begin{definition}[True Iteration]\\label{def:good_model}\nWe say iteration $k$ is {\\em true} if \n\\begin{align*}\n    \\|g_k-\\nabla f(y_k)\\|\\leq \\constG  \\|D_{\\alpha_k}(y_k)\\|, \\  {\\constG\\leq \\frac{1}{3}}.\n\\end{align*}\n\nIn other words, the first-order oracle delivers a sufficiently accurate estimate. \nWe use an indicator variable $\\iterTrue$ to indicate that iteration $k$ is true. \n\\end{definition}","sha256_utf8":"0c47ec6d7249a232c08e76878b43c51dd4184a1dcdf8254aba017be831f0df00"},{"start":40305,"end":40768,"start_line":381,"end_line":387,"raw_tex":"\\begin{assumption}[Probability of True Iteration]\\label{assumpt:key}\nWe assume that the first-order oracle  SFO$_{\\constG, p}$ is implemented in Algorithm \\ref{alg:ista_bktr_line_search} with  {$\\constG \\leq \\frac{1}{3}$} and $p > \\frac{1}{2}$ for every input $(y_k,\\alpha_k)$. \nThen, by the properties of  the oracle and the filtration $\\pastone$ we have, \n\\begin{align*} \n\\Prob(\\iterTrue| \\pastone) \\ge p \\text{ for all } k \\geq 0.\n\\end{align*}\n\\end{assumption}","sha256_utf8":"327e0e4aa1ba80008001056f398c1dd543aadd73b32be197947b11a83ec54956"},{"start":45883,"end":46132,"start_line":467,"end_line":468,"raw_tex":"Let us define\n$N_G = \\sum_{k=1}^{N_\\epsilon-1} \\mathbbm{1}\\{ \\Alpha_k \\ge \\stepsizebar \\} \\cdot  \\mathbbm{1}\\{{\\rm Iteration\\ } k\\ {\\rm is\\ true\\ and \\ successful} \\} $ be the number of {\\em good} iterations prior to the stopping time $N_\\epsilon$.\n","sha256_utf8":"d4efb30447b04fe26bf7e5f60acbcf61c4e928115e2cc48fdbec4eb2a79c62bf"},{"start":58478,"end":59090,"start_line":669,"end_line":676,"raw_tex":"\\begin{assumption}\\label{assumpt:ista_noise}\nAt every iteration $k$,  $g_k=g(y_k,\\xi_k)$ computed using the stochastic first-order oracle SFO$_{\\constG, p}(y_k,\\alpha_k)$ satisfies  \n\\begin{align}\\label{eq:ista_noise}\n    &\\EE_{\\xi_k} \\left[ \\| (\\nabla f(y_{k} )- G_{k} )\\|^2 | \\pastone\\right] \\leq \\frac{1}{\\alpha_{k}^2\\cdot k^{2+ \\beta}},\n\\end{align}\nfor some fixed $\\beta>0$. \n%\\trang{I thought conditioning on the filtration is essentially the same as the randomness in the oracle? since that was the only random element of the algo.} \\katya{Yes, this just makes it more explicit, I think}. \n\\end{assumption}","sha256_utf8":"0b44195bfc09bf89afee421d6f49bb85a209547fe7d6f94998e654a566439bf8"},{"start":73063,"end":73507,"start_line":821,"end_line":825,"raw_tex":"We first present that deterministic algorithm and then show how it is modified for the stochastic first-order oracles. Compared to  FISTA  in \\cite{Beck-Teboulle}, FISTA-BKTR in \\cite{Scheinberg-Goldfarb-Bai} uses an additional  parameter $\\theta_k$, which keeps record of the changes in  $\\alpha_k$  and \nutilizes the following update  \n\\begin{align}\\label{eq:fista_step}\n   (t_{new}, y) = \\text{FISTAStep}(x, x^{prev}, t, \\theta).\n\\end{align}","sha256_utf8":"c7678fc1881b79b950daf3e3bfef5affb5e3f3b828ea24b3983e88fc54339313"},{"start":73508,"end":73676,"start_line":826,"end_line":830,"raw_tex":"where FISTAStep computes \n\\begin{align*}\n    t_{new} &=  \\frac{1 + \\sqrt{1+4\\theta t^2}}{2}, \\text{ and }\n    y = x + \\frac{t-1}{t_{new}} ( x - x^{prev} ).\n\\end{align*}","sha256_utf8":"154e985a6b71883da34d42af7cafea3e0651681a89c030eaaa1ef9d41c6e52ba"},{"start":75686,"end":76718,"start_line":863,"end_line":867,"raw_tex":" Here we will use the main FISTA-BKTR  scheme, however, as in the case of stochastic ISTA, the gradient estimate needs to be recomputed each time a step is attempted, even when the iteration is unsuccessful. Therefore the iteration counter gets increased on unsuccessful steps as well as successful ones and the algorithm notation needs to be updated accordingly. In particular \nwe use  $x^{prev}_k$ (instead of $x_{k-1}$) to denote the iterate of the last successful iteration, that is the last iterate different from $x_k$. \nSimilarly, $t_k^{next}$ denotes the value of $t_{k+1}$ that is assigned if iteration $k$ is successful, otherwise $t_{k+1}=t_k$. \nWe also need to keep track of the $\\alpha_k^{succ}$ - the step-size  parameter of the last successful iteration up to iteration $k$. \nOn the other hand, $\\alpha_k^0$ value is not used since the dynamics of the step-size  parameter are simplified - it is decreased by the factor of $\\gamma$ on each unsuccessful iteration and increased proportionally on each successful one. \n","sha256_utf8":"e11cbf33afe3b4bbe65666579f06b7cebadc1e9e6cf567c8147527ad96d6e806"},{"start":78475,"end":80066,"start_line":882,"end_line":902,"raw_tex":"\\begin{algorithm}[ht]\n\\caption{Stochastic FISTA Step Search Algorithm}\n\\label{alg:fista_bktr_line_search}\n\\begin{algorithmic}[1]\n\\STATE {Initialization:} Choose an initial point $x_0\\in \\mathbb{R}^d$. Set $t_0 = 0$, let $x^{prev}_0 = x_0$. \\nonumber\\\\\nSet $0<\\gamma<1$, $\\theta_0 = \\gamma$ with $\\alpha_1>0$.  Let oracle SFO$_{\\constG, p}$ be given with  {$0 \\leq \\constG \\leq \\frac{1}{3}$} and $\\frac{1}{2} < p \\leq 1 $.\n\\FOR{$k=1,2,\\cdots,$}\n    \\STATE Compute $(t_k^{next}, y_k) = \\text{FISTAStep}(x_{k-1}, x^{prev}_{k-1}, t_{k-1}, \\theta_{k-1}) $\n    \\STATE Compute the gradient estimate $g_k=G(y_k,\\xi_k)$ using the stochastic first-order oracle SFO$_{\\constG, p}(y_k,\\alpha_k)$.\n    Compute the reference point $\\tilde{p}_{\\alpha_k} (y_k) =\\text{prox}_{\\alpha_k h} (y_k- \\alpha_k g_k)$.\n    \\STATE Check sufficient decrease condition $F(\\tilde{p}_{\\alpha_k} (y_k)) \\leq \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (y_k), y_k)$\n        \\STATE \\hspace{1.25em}If unsuccessful: \n            \\STATE \\hspace{2.5em}Set $\\left(x_{k},x^{prev}_{k},t_k\\right) = \\left(x_{k-1}, x^{prev}_{k-1}, t_{k-1}\\right)$, then set $\\alpha_{k+1} = \\gamma\\alpha_k$, %$\\alpha^{succ}_{k}=\\alpha^{succ}_{k-1}$ \n            and $\\theta_{k} = \\tfrac{\\theta_{k-1}}{\\gamma}$.\n        \\STATE \\hspace{1.25em}If successful: \n            \\STATE \\hspace{2.5em}Set $\\left(x_k, x^{prev}_{k},t_k\\right) = \\left(\\tilde{p}_{\\alpha_k} (y_k), x_{k-1}, t_k^{next}\\right)$,\n    then set $\\alpha_{k+1} = \\tfrac{\\alpha_k}{\\gamma},$ %$\\alpha^{succ}_{k}=\\alpha_k$ \n    and $\\theta_{k} = \\gamma$.\n\\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"8f0b0b0c1c212966ce0e01ae645afd9b165ffedb027399ce3bd126c1821e0ab1"},{"start":80533,"end":80756,"start_line":907,"end_line":908,"raw_tex":"Similar to Algorithm \\ref{alg:ista_bktr_line_search}, \nAlgorithm \\ref{alg:fista_bktr_line_search} generates a stochastic process $\\{Y_k, X_k, G_k, \\Stepsize{k}, T_k\\}$ with realizations $\\{y_k, x_k, g_k, \\alpha_k, t_k\\}$. \n","sha256_utf8":"933182186d4a7878b68b03aa1a47597ca7bfa3b48ad0635c3732784baf8fbb39"},{"start":88051,"end":88153,"start_line":1011,"end_line":1013,"raw_tex":"\\begin{align}\\label{eq:define_m}\n    m_k = 2\\alpha^{succ}_{k} t_{k}^2v_{k} +\\norm{u_k }^2,\n\\end{align}","sha256_utf8":"85a09a0fa48d5bef11836766dcf483bb9218c2d5adf725f6944dee194ec5a653"},{"start":88154,"end":88236,"start_line":1014,"end_line":1014,"raw_tex":"where $v_k =  F(x_k) - F^* $ and $u_k = (t_{k}-1)x^{prev}_{k} + x^* - t_k x_k $. \n","sha256_utf8":"86e33995aeaa15c3677b657a04fbb32b82e1b953ea24e6968fcf70a70c928fa6"},{"start":102376,"end":102775,"start_line":1245,"end_line":1251,"raw_tex":"\\begin{assumption}\\label{assumpt:fista_noise}\nAt every iteration $k$,  $g_k=g(y_k,\\xi_k)$ computed using the stochastic first-order oracle SFO$_{\\constG, p}(y_k,\\alpha_k)$ satisfies \n\\begin{align}\\label{eq:fista_noise}\n&\\EE_{\\xi_k} \\left[ \\| (\\nabla f(y_{k} )- G_{k} )\\|^2 | \\pastone\\right] \\leq \\frac{1}{\\alpha_{k}^2t_{k}^2\\cdot k^{2+ \\beta}},\n\\end{align}\nfor some fixed $\\beta>0$.\n\\end{assumption}","sha256_utf8":"0f325bfa699fc76740e6eae9da907da20816982df68880483737fd8082437fe7"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{th:final_results}\nUnder Assumptions \\ref{assumpt:convex_smooth}, \\ref{assumpt:bounded_below}, \\ref{assumpt:key} and Assumption \\ref{assumpt:fista_noise} the number of iterations $N_\\epsilon$ to reach an $\\epsilon$-accurate solution satisfies the following bound: \n\\begin{align*}\n\\EE (N_\\epsilon) \\leq \\frac{2p}{(2p-1)^2}\\left ( \\left(\n \\frac{8 \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\frac{4(\\beta + 2)^2}{\\beta^2} + \\frac{8(\\beta+2)}{\\beta+1}  \n \\right]}{\\stepsizebar\\epsilon} \\right)^{\\frac{1}{2}}+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right) +1,\n\\end{align*}\nwhere  $\\stepsizebar $ is defined in Lemma \\ref{lem:good_est_good_model_successful}.\n\\end{theorem}","units":[{"start":107325,"end":108041,"start_line":1319,"end_line":1327,"raw_tex":"\\begin{theorem}\\label{th:final_results}\nUnder Assumptions \\ref{assumpt:convex_smooth}, \\ref{assumpt:bounded_below}, \\ref{assumpt:key} and Assumption \\ref{assumpt:fista_noise} the number of iterations $N_\\epsilon$ to reach an $\\epsilon$-accurate solution satisfies the following bound: \n\\begin{align*}\n\\EE (N_\\epsilon) \\leq \\frac{2p}{(2p-1)^2}\\left ( \\left(\n \\frac{8 \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\frac{4(\\beta + 2)^2}{\\beta^2} + \\frac{8(\\beta+2)}{\\beta+1}  \n \\right]}{\\stepsizebar\\epsilon} \\right)^{\\frac{1}{2}}+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right) +1,\n\\end{align*}\nwhere  $\\stepsizebar $ is defined in Lemma \\ref{lem:good_est_good_model_successful}.\n\\end{theorem}","sha256_utf8":"53d23ec759a6fe3d517b74ce4a37f93b71ed925c769195c517aeb6df4e43d0f1"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\n\\label{lem:good_est_good_model_successful} \nSuppose that iteration $k$ is true, i.e. \n\\begin{align*}\n    \\|g_k-\\nabla f(y_k)\\|\\leq \\constG  \\|D_{\\alpha_k}(y_k)\\|,\n\\end{align*}\nwith  {$\\constG \\leq \\frac{1}{3}$} and $D_{\\alpha_k}(y_k)$ is the gradient mapping of $F$ at $y_k$. If \n\\begin{align*}\n\\stepsize{k}{} \\le \\stepsizebar := \\frac{1}{L}\\left( 1 - \\frac{2\\constG}{1-\\constG}\\right) ,\n\\end{align*}\nthen the $k$-th step is successful. \n\n\\end{lemma}\n\n\\begin{proof}\n\n\nFrom the $L$-smoothness of $f$ we have\n\\begin{align*}\n    f(\\tilde{p}_{\\alpha_k} (y_k))  &\\le f(y_k) + \\nabla f(y_k)^\\top(\\tilde{p}_{\\alpha_k} (y_k) - y_k) + \\frac{L}{2}\\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 \\nonumber\\\\\n     &= f(y_k) +  g_k^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k) + [\\nabla f(y_k)-g_k]^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k)  \\nonumber\\\\\n     &\\quad+ \\frac{1}{2\\alpha_k} \\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2  + \\left(\\frac{L}{2}- \\frac{1}{2\\alpha_k}\\right)\\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 .\n\\end{align*}\nAdding the non-smooth term $ h(\\tilde{p}_{\\alpha_k} (y_k))$ we have\n\\begin{align*}\n    &F (\\tilde{p}_{\\alpha_k} (y_k))  \\nonumber\\\\\n     &\\le f(y_k) +  g_k^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k) + \\frac{1}{2\\alpha_k} \\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 + h(\\tilde{p}_{\\alpha_k} (y_k))  \\nonumber\\\\\n     &\\quad + [\\nabla f(y_k)-g_k]^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k) + \\left(\\frac{L}{2}- \\frac{1}{2\\alpha_k}\\right)\\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 \n     \\nonumber\\\\\n     &= \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (y_k), y_k) + [\\nabla f(y_k)-g_k]^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k) + \\left(\\frac{L}{2}- \\frac{1}{2\\alpha_k}\\right)\\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 .\n\\end{align*}\nTo show that the iteration is successful i.e. $F(\\tilde{p}_{\\alpha_k} (y_k)) \\leq \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (y_k), y_k)$\nwe need to prove that \n\\begin{align}\\label{eq:02}\n    [\\nabla f(y_k)-g_k]^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k) \\leq \\left(\\frac{1}{2\\alpha_k} - \\frac{L}{2}\\right)\\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 .\n\\end{align}\nNote that the right hand side is non-negative since $\\alpha_k \\leq \\frac{1}{L}$. \nOur definition of the true iteration leads to \n\\begin{align}\\label{eq:01}\n    \\|g_k-\\nabla f(y_k)\\|\n    &\\leq \\constG \\left \\|\\frac{1}{\\alpha_k}\\Big(y_k - p_{\\alpha_k} (y_k)\\Big)\\right\\|\\nonumber  \\\\\n    &\\leq  \\frac{\\constG}{\\alpha_k} \\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\| + \\frac{\\constG}{\\alpha_k} \\left \\|\\tilde{p}_{\\alpha_k} (y_k) - p_{\\alpha_k} (y_k)\\right\\|\\nonumber  \\\\\n    &=  \\frac{\\constG}{\\alpha_k} \\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\| + \\frac{\\constG}{\\alpha_k} \\norm{\\alpha_k\\nabla f(y_k)-\\alpha_k g_k },\n\\end{align}\nwhere the last inequality follows  from the nonexpansiveness of the prox-operator $\\text{prox}_{\\alpha h}$ for  proper, closed, and convex $h(x)$, i.e.,\n\\begin{align*}\n    \\|\\text{prox}_{\\alpha h}(z_1) - \\text{prox}_{\\alpha h}(z_2)\\| \\leq \\|z_1 - z_2\\| \\text{ for all } z_1, z_2\\in\\R^d.\n\\end{align*}\nThe bound \\eqref{eq:01} is equivalent to\n\\begin{align*}\n    (1-\\constG)\\norm{\\nabla f(y_k)-g_k} \\leq  \\frac{\\constG}{\\alpha_k} \\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\|.\n\\end{align*}\nTherefore we show \\eqref{eq:02} by\n\\begin{align*}\n    \\norm{\\nabla f(y_k)-g_k} &\\leq  \\frac{\\constG}{1-\\constG} \\frac{1}{\\alpha_k} \\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\|  \\nonumber\\\\\n    &\\leq \\left( \\frac{1- \\alpha_k  L}{2}\\right)\\frac{1}{\\alpha_k} \\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\| = \\left( \\frac{1}{2\\alpha_k} - \\frac{L}{2}\\right)\\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\|,\n\\end{align*}\nwhere we use the condition that \n\\begin{align*}\n    \\frac{\\constG}{1-\\constG} \\leq \\frac{1- \\alpha_k  L}{2},\n\\end{align*}\nwhich is the consequence of the small step-size parameter: \n\\begin{align*}\n    \\alpha_k  \\leq \\frac{1}{L}\\left( 1 - \\frac{2\\constG}{1-\\constG}\\right).\n\\end{align*}\n\n~\\qed \\end{proof}\n\n\\begin{theorem}[Bounding $\\EE (N_\\epsilon)$ based on $\\EE(N_G)$]\\label{th:framework}\nLet $\\stepsizebar$ be the value stated in Lemma \\ref{lem:good_est_good_model_successful}.\n Under Assumption \\ref{assumpt:key} we have \n\\begin{align*}\n\\EE (N_\\epsilon -1) \\leq \\frac{2p}{(2p-1)^2}\\left (2\\EE(N_G)+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right).\n\\end{align*}\n\\end{theorem}\n\n\\begin{proof}\nBy the update rule of $\\alpha_k$ in Algorithm \\ref{alg:ista_bktr_line_search}, the stochastic process $\\{\\Alpha_k\\}_{k\\geq 1}$ follows: \n\\begin{align*}\n\\Alpha_{1}&= \\alpha_1, \\quad \n\\Alpha_{k+1}=\\left \\{ \\begin{array}{ll} \n\\gamma^{-1} \\Alpha_k & {\\rm if\\  }  \\iterSucc=1,  \\\\\n \\gamma \\Alpha_k &   {\\rm if\\  }  \\iterSucc=0 .\n\\end{array}\\right . \n\\end{align*}\n\nLet us now define the following indicator random variables: %two indicator random variables, in addition to $\\iterTrue$ defined earlier,\n\\begin{align*}\n\\iterLarge=\\mathbbm{1}\\{ \\Alpha_k >\\stepsizebar\\}, \\\\\n\\iterLargebar= \\mathbbm{1}\\{ \\Alpha_k \\geq \\stepsizebar\\}.\n\\end{align*}\nClearly \n\\begin{align*}\n\\EE (N_\\epsilon-1) = \\EE\\left (\\sum_{k=1}^{N_\\epsilon-1} \\iterLarge\\right ) + \\EE \\left (\\sum_{k=1}^{N_\\epsilon-1} (1-\\iterLarge)\\right ).\n\\end{align*}\n\nFrom Lemma 2.3 in \\cite{Cartis-Scheinberg}, under Assumption \\ref{assumpt:key} and using the fact that all true iterations are successful when $\\alpha_k \\le \\stepsizebar$, we have: \n\\begin{align*}\n\\EE\\left (\\sum_{k=1}^{N_\\epsilon-1} (1-\\iterLarge)\\right)\\leq \\frac{1}{2p}\\EE(N_\\epsilon -1 ).\n\\end{align*}\n\n\nWe define \n\\begin{itemize}\n\\item $N_{FS}=\\sum_{k=1}^{N_\\epsilon-1} \\iterLargebar (1-\\iterTrue)\\iterSucc$, which is the number of false successful iterations with $\\Alpha_k\\geq \\stepsizebar$.\n\\item $N_F=\\sum_{k=1}^{N_\\epsilon-1} \\iterLargebar (1-\\iterTrue)$, which is the number of false  iterations with $\\Alpha_k\\geq \\stepsizebar$.\n\\item $N_G=\\sum_{k=1}^{N_\\epsilon-1} \\iterLargebar \\iterTrue \\iterSucc$, which is the number of true successful iterations with $\\Alpha_k\\geq \\stepsizebar$.\n\\item $N_T=\\sum_{k=1}^{N_\\epsilon-1} \\iterLargebar \\iterTrue$, which is the number of true iterations with $\\Alpha_k\\geq \\stepsizebar$.\n% \\item $N_{TU}=\\sum_{k=1}^{N_\\epsilon-1} \\iterLarge \\iterTrue (1-\\iterSucc)$, which is the number of true unsuccessful iterations with $\\Alpha_k> \\stepsizebar$.\n\\item $N_U=\\sum_{k=1}^{N_\\epsilon-1} \\iterLarge  (1-\\iterSucc)$, which is the number of unsuccessful iterations with $\\Alpha_k> \\stepsizebar$.\n\\end{itemize}\n\nFrom equations (11) and (12) of Lemma 2.7 in \\cite{Cartis-Scheinberg} we have\n\\begin{align}\\label{eq:binom}\n\\EE(N_{FS})\\leq \\EE(N_{F})\\leq\\frac {1-p}{p}[\\EE(N_G)+\\EE(N_U)],\n\\end{align}\nand\n\\begin{align}\\label{eq:boundM3}\n\\EE(N_U)\\leq \\EE(N_{FS})+\\EE(N_G) + \\log_\\gamma(\\stepsizebar/\\alpha_1).\n\\end{align}\nPlugging \\eqref{eq:boundM3} into  \\eqref{eq:binom}\nwe obtain\n\\begin{align*}\n\\EE(N_{FS})\\leq \\frac {1-p}{p}\\left[\\EE(N_{FS})+2\\EE(N_G) + \\log_\\gamma(\\stepsizebar/\\alpha_1)\\right],\n\\end{align*}\nand, hence,\n\\begin{align*}\n\\frac {2p-1}{p}\\EE(N_{FS})\\leq \\frac {1-p}{p}\\left[ 2\\EE(N_G) + \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right].\n\\end{align*}\nThis finally implies \n\\begin{align}\\label{eq:boundN1}\n\\EE(N_{FS})\\leq \\frac {1-p}{2p-1}\\left [2\\EE(N_G)+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right ].\n\\end{align}\nNow we can bound the expected total  number of  iterations when $\\alpha_k>\\stepsizebar$, using %\\eqref{eq:N2}, \n\\eqref{eq:boundM3} and \\eqref{eq:boundN1}  \nand adding the terms to obtain the result of the lemma, namely,\n\\begin{align*}\n\\EE(N_F+N_T)&\\leq \\EE(N_F+N_U+N_G)\\leq \\frac{1}{p}\\EE(N_U+N_G)\\nonumber\\\\\n&\\leq \\frac{1}{2p-1}\\left (2\\EE(N_G)+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right).\n\\end{align*}\nNext, from Lemma 2.7 in \\cite{Cartis-Scheinberg},\\footnote{Final statement of the Lemma 2.7 in \\cite{Cartis-Scheinberg}  replaces $\\EE(N_G)$ with its bound, but here we\nare yet to derive such a  bound, therefore we explicitly use $\\EE(N_G)$.}\n\\begin{align*}\n\\EE\\left (\\sum_{k=1}^{N_\\epsilon-1}\\iterLarge \\right )\\leq \\frac{1}{2p-1}\\left (2\\EE(N_G)+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right). \n\\end{align*}\nHence, using the two previous results we have \n\\begin{align*}\n\\EE (N_\\epsilon -1)\\leq \\frac{1}{2p} \\EE (N_\\epsilon -1) +   \\frac{1}{2p-1} \\left (2\\EE(N_G)+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right).\n\\end{align*}\nThe result of the theorem follows. \n~\\qed \\end{proof}\n\n\\begin{lemma}[Bounds for a successful step]\\label{lem:successful_step_bound1}\nIf $k$ is a successful iteration then we have:\n\\begin{align*}\n\\|x_{k} - x^*\\|^2 - \\|x_{k-1} - x^* \\|^2 \\leq  - 2\\alpha_k (F(x_k) - F(x^*) )  + \\lambda_k \\cdot \\|x_{k-1} -x^*\\|,\n\\end{align*}\nwhere $\\lambda_k = \\| 2\\alpha_k (\\nabla f(x_{k-1})-g_k) \\| $.\n\\end{lemma}\n\n\\begin{proof}\nSince iteration $k$ is successful,  the sufficient decrease condition holds: \n\\begin{align*}\n    F(\\tilde{p}_{\\alpha_k} (x_{k-1})) &\\leq \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (x_{k-1}), x_{k-1}) \\nonumber\\\\\n    &= f(x_{k-1}) +(x_k -x_{k-1})^\\top g_k + \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2 + h(x_k).\n\\end{align*}\nBy the  convexity of $f$ and $h$, applied at $x_{k-1}$ and $x_k$ respectively, for any $x$,  \n \\begin{align*}\nf(x) &\\geq f(x_{k-1}) + \\nabla f(x_{k-1})^\\top (x-x_{k-1}),\n\\nonumber\\\\\nh(x) &\\geq h(x_k) + \\gamma(x_{k-1})^\\top (x-x_k),\n\\end{align*}\nwhere the second bound holds for every subgradient $\\gamma(x_{k-1})$ of $h$ at $x_{k}$. \nSumming those inequality we have \n\\begin{align*}\nf(x) +h(x) \\geq f(x_{k-1}) + \\nabla f(x_{k-1})^\\top (x-x_{k-1})+ h(x_k) + \\gamma(x_{k-1})^\\top (x-x_k).\n\\end{align*}\nCombining this with the sufficient decrease condition we get: \n\\begin{align*}\n    &F(x) - F(x_k) \\nonumber\\\\&\\geq \\nabla f(x_{k-1})^\\top (x-x_{k-1}) + \\gamma(x_{k-1})^\\top (x-x_k) - (x_k -x_{k-1})^\\top g_k - \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2\\nonumber\\\\\n    &= (\\nabla f(x_{k-1})-g_k) ^\\top (x-x_{k-1}) + (\\gamma(x_{k-1})+g_k)^\\top (x-x_k) - \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2.\n\\end{align*}\nUsing the fact that $x_k = \\tilde{p}_{\\alpha_k} (x_{k-1}) =\\arg\\min_x \\tilde{Q}_{\\alpha_k}(x, x_{k-1})$, we get that $ x_{k-1} - x_k= \\alpha_k[\\gamma(x_{k-1})+g_k]$ for some subgradient $\\gamma(x_{k-1})$ of $h$ at $x_{k}$. Therefore\n\\begin{align*}\n    &F(x) - F(x_k) \\nonumber\\\\\n    &\\geq (\\nabla f(x_{k-1})-g_k) ^\\top (x-x_{k-1}) + \\frac{1}{\\alpha_k}(x_{k-1} - x_k)^\\top (x-x_k) - \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2\\nonumber\\\\\n    &= (\\nabla f(x_{k-1})-g_k) ^\\top (x-x_{k-1}) + \\frac{1}{\\alpha_k}(x_{k-1} - x_k)^\\top (x-x_{k-1} + x_{k-1} -x_k) \\nonumber\\\\\n    & \\quad - \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2\\nonumber\\\\\n    &= (\\nabla f(x_{k-1})-g_k) ^\\top (x-x_{k-1}) + \\frac{1}{\\alpha_k}(x_{k-1} - x_k)^\\top (x-x_{k-1}) + \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2\\nonumber\\\\\n    &= (\\nabla f(x_{k-1})-g_k) ^\\top (x-x_{k-1}) + \\frac{1}{2\\alpha_k} \\left( \\|x-x_k \\|^2-\\|x-x_{k-1} \\|^2\\right),\n\\end{align*}\nand we get the bound for $x=x^*$: \n\\begin{align*}\n    &2\\alpha_k (F(x_k) - F(x^*) ) - 2\\alpha_k (\\nabla f(x_{k-1})-g_k) ^\\top (x_{k-1} -x^*)\n    \\leq \\|x_{k-1} - x^*\\|^2 - \\|x_k - x^* \\|^2.\n\\end{align*}\nRearranging the terms, we get the statement of Lemma \\ref{lem:successful_step_bound1}:\n\\begin{align*}\n    \\|x_k - x^* \\|^2 - \\|x_{k-1} - x^*\\|^2 &\\leq -2\\alpha_k (F(x_k) - F(x^*) ) + 2\\alpha_k (\\nabla f(x_{k-1})-g_k) ^\\top (x_{k-1} -x^*)\\nonumber\\\\\n    &\\leq -2\\alpha_k (F(x_k) - F(x^*) ) + \\lambda_k \\cdot\\|x_{k-1} -x^*\\|,\n\\end{align*}\nwhere $\\lambda_k = \\|2\\alpha_k (\\nabla f(x_{k-1})-g_k)\\|$.\n\n~\\qed \\end{proof}\n\n\\begin{lemma}\\label{lem:after_K1}\n    After $K$ iterations of the algorithm, the following bound holds: \n\\begin{align*}\n\\sum_{k\\in S, k \\leq K} 2\\alpha_k (F(x_k) - F(x^*) )\n    &\\leq 2\\|x_0 - x^* \\|^2 + \\sum_{k=1}^K \\lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\lambda_i \\right)+ 2\\sum_{k=1}^K \\lambda_k^2,\n\\end{align*}\nwhere $S$ is the set of successful iterations.\n\\end{lemma}\n\n\\begin{proof}\nFor every $K$, from Lemma \\ref{lem:successful_step_bound1} we have\n\\begin{align*}\n    \\|x_K - x^* \\|^2 - \\|x_0 - x^* \\|^2 \n    &= \\sum_{k=1}^{K} \\left(\\|x_{k} - x^* \\|^2 -\\|x_{k-1} - x^* \\|^2 \\right) \\nonumber\\\\\n    &\\leq \\sum_{k\\in S, k \\leq K} \\left[\\lambda_k \\cdot \\|x_{k-1} -x^*\\| - 2\\alpha_k (F(x_k) - F(x^*) ) \\right],\n\\end{align*}\nwhere $S$ is the set of successful iterations. \nHence we arrive at our key bound\n\\begin{align}\\label{eq:thm_key1}\n    &\\|x_K - x^* \\|^2 - \\|x_0 - x^* \\|^2 + \\sum_{k\\in S, k \\leq K} 2\\alpha_k (F(x_k) - F(x^*) )\\leq  \\sum_{k=1}^K \\lambda_k  \\cdot \\| x_{k-1} -x^* \\|.\n\\end{align}\nNow we follow a similar argument as \\cite{Schmidt-Roux-Bach} with two steps. First, we use the key bound \\eqref{eq:thm_key1} to bound the distance to optimal point $u_{k-1} = \\| x_{k-1} -x^* \\|$ in terms of the initial distance $u_{0} = \\| x_{0} -x^* \\|$ and the noise terms $\\lambda_k$. Then we use this bound on $\\| x_{k-1} -x^* \\|$ to substitute into the last sum of the key bound \\eqref{eq:thm_key1} and obtain the final derivation in terms of the noise $\\lambda_k$.\\\\\nDropping the positive term $\\sum_{k\\in S, k \\leq K} 2\\alpha_k (F(x_k) - F(x^*) )$ we get\n\\begin{align*}\n    \\|x_K - x^* \\|^2 \n    &\\leq \\|x_0 - x^* \\|^2 + \\sum_{k=1}^K \\lambda_k \\cdot \\| x_{k-1} -x^* \\| \\nonumber\\\\\n    &= \\|x_0 - x^* \\|^2 + \\lambda_1 \\cdot \\| x_{0} -x^* \\| + \\sum_{k=1}^K \\lambda_{k+1} \\cdot\\| x_{k} -x^* \\|.\n\\end{align*}\nLetting $u_k = \\| x_{k} -x^* \\|$, this is equivalent to \n\\begin{align*}\n    u_K^2 = \\|x_K - x^* \\|^2  \\leq Z + \\sum_{k=1}^K\\lambda_{k+1} u_k.\n\\end{align*}\nwhere $Z = \\|x_0 - x^* \\|^2 + \\lambda_1 \\cdot \\| x_{0} -x^* \\| = u_0^2 + \\lambda_1 u_0$. %\\katya{there seems to be incorrect switching between $k$ and $K$ in a few places here} \nUsing this inequality and applying Lemma 1 in \\cite{Schmidt-Roux-Bach} for the non-negative sequence $\\{u_K\\}$ have\n\\begin{align*}\n     u_{k} \\leq \\frac{1}{2} \\sum_{i=1}^{k} \\lambda_{i+1} + \\left(Z + \\left(\\frac{1}{2} \\sum_{i=1}^{k} \\lambda_{i+1} \\right)^2\\right)^{1/2}\n    \\leq  \\sum_{i=1}^{k} \\lambda_{i+1} + Z^{1/2}  \\leq  \\sum_{i=1}^{k+1} \\lambda_{i} + u_0 ,\n\\end{align*}\nwhere the last line follows from the fact that $Z^{1/2} = \\left(u_0^2 + \\lambda_1  \\cdot u_0 \\right)^{1/2} \\leq u_0 + \\lambda_1.$\n\\\\\nPutting the bound for $u_k$ into the key bound \\eqref{eq:thm_key1} we have\n\\allowdisplaybreaks\n\\begin{align*}\n    &\\|x_K - x^* \\|^2 - \\|x_0 - x^* \\|^2 + \\sum_{k\\in S, k \\leq K} 2\\alpha_k (F(x_k) - F(x^*) )\\nonumber\\\\\n    &\\leq \\sum_{k=1}^K \\lambda_k \\cdot \\| x_{k-1} -x^* \\|\\leq \\sum_{k=1}^K \\lambda_k \\cdot \\left( \\sum_{i=1}^{k} \\lambda_i + \\|x_0 - x^* \\|\\right)\\nonumber\\\\\n    &= \\sum_{k=1}^K \\lambda_k \\cdot \\left( \\sum_{i=1}^{k} \\lambda_i \\right) + \\|x_0 - x^* \\| \\cdot \\sum_{k=1}^K \\lambda_k\\nonumber\\\\\n    &\\leq  \\sum_{k=1}^K \\lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\lambda_i \\right)+ \\sum_{k=1}^K \\lambda_k^2 + \\|x_0 - x^* \\|^2 + \\sum_{k=1}^K \\lambda_k^2 \\nonumber\\\\\n    &= \\|x_0 - x^* \\|^2 + \\sum_{k=1}^K \\lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\lambda_i \\right)+ 2\\sum_{k=1}^K \\lambda_k^2,\n\\end{align*}\nand the statement of Lemma \\ref{lem:after_K1} follows. \n~\\qed\n\\end{proof}\n\n\\begin{theorem}[Upper bound of the number of good steps]\\label{th:bound_expectation_N_TS1} Under Assumption \\ref{assumpt:convex_smooth}, \\ref{assumpt:bounded_below}  ,\nand  Assumption \\ref{assumpt:ista_noise}, we have \n \\begin{align*}\n\\EE [N_G] \n\\leq \\frac{1}{2\\stepsizebar\\epsilon} \\left [2\\|x_0 - x^* \\|^2 + \\frac{4(\\beta + 2)^2}{\\beta^2} + \\frac{8(\\beta+2)}{\\beta+1} \n \\right]. \n\\end{align*}\n\n\\end{theorem}\n\n\\begin{proof}\nWe recall the definition of the noise term $\\lambda_k =  \\|2\\alpha_{k} (\\nabla f(y_{k} )- g_{k} )\\| $. Let $\\Lambda_k =  \\|2\\Alpha_{k} (\\nabla f(Y_{k} )- G_{k} )\\| $, thus $\\Lambda_k$ is a random variable with realizations  $\\lambda_k$. \nSince the statement of Lemma \\ref{lem:after_K1} holds for every realizations of Algorithm \\ref{alg:ista_bktr_line_search}, for $K = N_\\epsilon -1$ we have:  \n\\begin{align*}\n\\sum_{k\\in S, k \\leq N_\\epsilon -1} 2\\Alpha_k (F(X_k) - F(x^*) )\n    &\\leq 2\\|x_0 - x^* \\|^2 + \\sum_{k=1}^{N_\\epsilon -1} \\Lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\Lambda_i \\right)+ 2\\sum_{k=1}^{N_\\epsilon -1} \\Lambda_k^2,\n\\end{align*}\nwhere $S$ is the set of successful iterations.\nRecall that the stopping time $N_\\epsilon = \\min\\{k = 1,2, \\ldots\\,:  F(X_k) - F^*  \\leq \\epsilon\\}$. Hence for every $k \\leq N_\\epsilon-1$, we have that $ F(X_k) - F^*> \\epsilon$ and\n\\begin{align*}\n    \\epsilon \\cdot \\sum_{k\\in S, k \\leq N_\\epsilon -1} 2\\Alpha_k  &<  \\sum_{k\\in S, k \\leq N_\\epsilon -1} 2\\Alpha_k (F(X_k) - F(x^*) )\\nonumber\\\\\n    &\\leq 2\\|x_0 - x^* \\|^2 + \\sum_{k=1}^{N_\\epsilon} \\Lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\Lambda_i \\right)+ 2\\sum_{k=1}^{N_\\epsilon} \\Lambda_k^2.\n\\end{align*}\nLet $\\mathcal{N}_G$ be the set of \\textit{good} iterations  i.e. true and successful iterations $k$ with $\\alpha_k\\geq \\stepsizebar, k \\leq N_\\epsilon -1 $, then $N_G$ is the cardinality of that set. We have \n\\begin{align*} \n\\sum_{k\\in S, k \\leq N_\\epsilon -1} 2\\Alpha_k \\geq\\sum_{k\\in \\mathcal{N}_G, k \\le N_\\epsilon -1 }  2\\Alpha_k \\geq N_G \\cdot 2\\stepsizebar = 2 N_G \\stepsizebar.\n\\end{align*}\nTherefore \n\\begin{align*}\n    2 N_G \\stepsizebar \\cdot \\epsilon  \n    &\\leq 2\\|x_0 - x^* \\|^2 + \\sum_{k=1}^{N_\\epsilon} \\Lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\Lambda_i \\right)+ 2\\sum_{k=1}^{N_\\epsilon} \\Lambda_k^2,\n\\end{align*}\nand \n\\begin{align*}\n    N_G   \n    &\\leq \\frac{2\\|x_0 - x^* \\|^2 + \\sum_{k=1}^{N_\\epsilon} \\Lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\Lambda_i \\right)+ 2\\sum_{k=1}^{N_\\epsilon} \\Lambda_k^2}{2 \\stepsizebar \\epsilon}.\n\\end{align*}\nTaking expectation we have %\\trang{notations here}\n\\begin{align}\\label{eq:bound1}\n\\EE [N_G ] \n &\\leq \\frac{1}{2\\stepsizebar\\epsilon} \\left [2\\|x_0 - x^* \\|^2 \n+ \\EE \\left[ \\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} \\cdot \\left(\\sum_{i=1}^{k-1} \\Lambda_{i} \\right)  \\right] + 2\\EE \\left[\\sum_{k=1}^{N_\\epsilon}  \\Lambda_{k} ^2  \\right] \n \\right]  .\n\\end{align}\nIn order to bound \\eqref{eq:bound1}, we derive the expected values of $\\Lambda_k^2$, $\\Lambda_k$ and $\\Lambda_k \\Lambda_i$ for $1\\leq i, k \\leq N_\\epsilon$\nusing Assumption \\ref{assumpt:ista_noise}. From \\eqref{eq:ista_noise} we have: \n\\begin{align}\\label{eq:mu_k_squared1}\n    \\EE \\left[\\Lambda_{k}^2| \\pastone\\right] = \\EE \\left[4\\alpha_{k}^2 \\| (\\nabla f(y_{k} )- G_{k} )\\|^2 | \\pastone\\right] \\leq \\frac{4}{k^{2+ \\beta}},\n\\end{align}\nwhich leads to \n\\begin{align*}\n    \\left(\\EE \\left[\\Lambda_{k}| \\pastone\\right]\\right)^2 \\leq \\EE \\left[\\Lambda_{k}^2| \\pastone\\right] \\leq \\frac{4}{k^{2+ \\beta}}.\n\\end{align*}\nTherefore we have the bound for $\\EE \\left[\\Lambda_{k}| \\pastone\\right]$: \n\\begin{align}\\label{eq:mu_k1}\n    \\EE \\left[\\Lambda_{k}| \\pastone\\right] \\leq \\frac{2}{k^{1+ 0.5\\beta}}.\n\\end{align} \nTaking the total expectation of \\eqref{eq:mu_k_squared1} and \\eqref{eq:mu_k1} we have \n\\begin{align*}\n    \\EE \\left[\\Lambda_{k}^2\\right] \\leq \\frac{4}{k^{2+ \\beta}} \\text{ and } \\EE \\left[\\Lambda_{k}\\right] \\leq \\frac{2}{k^{1+ 0.5\\beta}}.\n\\end{align*} \nNow we consider the quantity $\\EE \\left[\\Lambda_{k}\\Lambda_{i}| \\pastone\\right] $ where $i< k $. \nFrom the definition of  the filtration $\\pastone$ and $\\Lambda_{i}$,  {we have that} $\\Lambda_i \\in \\pastone$ for all $i < k$. Therefore by the rules of conditional probability\n\\begin{align*}\n    \\EE \\left[\\Lambda_{k}\\Lambda_{i}| \\pastone\\right] =  \\EE \\left[\\Lambda_{k}| \\pastone\\right] \\Lambda_i .\n\\end{align*}\nHence we can bound the expect value of $\\Lambda_{k}\\Lambda_{i}$ as follows: \n\\begin{align*}\n    \\EE \\left[\\Lambda_{k}\\Lambda_{i}  \\right] \n    &=\\EE \\left[\\EE \\left[\\Lambda_{k}\\Lambda_{i}| \\pastone \\right] \\right]  =\\EE \\left[\\EE \\left[\\Lambda_{k}| \\pastone\\right] \\Lambda_i  \\right] \n    \\nonumber\\\\ \\nonumber\n    &  {\\leq} \\EE \\left[\\EE \\left[\\frac{2}{k^{1+ 0.5\\beta}} \\Lambda_i  \\right]  \\right]   = \\frac{2}{k^{1+ 0.5\\beta}} \\EE \\left[\\EE \\left[ \\Lambda_i | \\pastonei \\right]  \\right]\n    \\nonumber\\\\ \\nonumber\n    &  {\\leq} \\frac{2}{k^{1+ 0.5\\beta}} \\frac{2}{i^{1+ 0.5\\beta}} .\n\\end{align*}\nSince $N_\\epsilon$ is a hitting time and the random variables $\\Lambda_k^2$, $\\Lambda_{k}\\Lambda_{i}$ have finite means and are summable (over $\\{k = 1,2, \\ldots\\}$ and over $\\{i, k = 1,2,\\ldots\\}$ respectively), from the general Wald's identity  {\\cite{Wald}} we have: \n\\begin{align*}\n\\EE \\left[\\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} ^2  \\right] &= \\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\EE \\left[  \\Lambda_{k} ^2  \\right]  \\right)\\nonumber\\\\\n\\EE \\left[ \\sum_{k=1}^{N_\\epsilon}\\Lambda_{k} \\cdot \\left(\\sum_{i=1}^{k-1} \\Lambda_{i} \\right)  \\right] &= \\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\sum_{i=1}^{k-1}\\EE \\left[  \\Lambda_{k} \\Lambda_{i}  \\right] \\right) .\n\\end{align*}\nApplying the previous total expected value bounds in these inequality we have: \n \\begin{align*}\n\\EE \\left[\\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} ^2  \\right] & = \\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\EE \\left[  \\Lambda_{k} ^2  \\right]  \\right)\n\\leq 4\\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\frac{1}{k^{2+ \\beta}}\\right) \\leq  \\frac{4(\\beta +2)}{\\beta+1}\\nonumber\\\\\n \\EE \\left[ \\sum_{k=1}^{N_\\epsilon}\\Lambda_{k} \\left(\\sum_{i=1}^{k-1} \\Lambda_{i} \\right)  \\right] &= \\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\sum_{i=1}^{k-1}\\EE \\left[  \\Lambda_{k} \\Lambda_{i}  \\right] \\right) \\leq 4\\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\frac{1}{k^{1+ 0.5\\beta}} \\sum_{i=1}^{k-1} \\frac{1}{i^{1+ 0.5\\beta}}\\right) \\leq \\frac{4(\\beta + 2)}{\\beta},\n\\end{align*}\nwhere the last computations follows from \n\\begin{align*}\n    &\\sum_{k=1}^{N_\\epsilon}\\frac{1}{k^{2+ \\beta}} < 1 + \\sum_{k=2}^{\\infty} \\frac{1}{k^{2+ \\beta}} < 1+  \\int_{1}^{\\infty} \\frac{1}{x^{2+ \\beta}} dx =  \\frac{\\beta +2}{\\beta+1},\\nonumber\\\\\n    &\\sum_{k=1}^{N_\\epsilon}\\frac{1}{k^{1+ 0.5\\beta}} < 1 + \\sum_{k=2}^{\\infty} \\frac{1}{k^{1+ 0.5\\beta}} < 1+  \\int_{1}^{\\infty} \\frac{1}{x^{1+ 0.5\\beta}} dx = \\frac{\\beta + 2}{\\beta}.\n\\end{align*}\nSubstituting these bounds into the previous inequality \\eqref{eq:bound1}, we have\n\\begin{align*}\n\\EE [N_G ] \n &\\leq \\frac{1}{2\\stepsizebar\\epsilon} \\left [2\\|x_0 - x^* \\|^2 + \\frac{4(\\beta + 2)^2}{\\beta^2} + \\frac{8(\\beta+2)}{\\beta+1} \n \\right]. \n\\end{align*}\nwhich is the statement of Theorem \\ref{th:bound_expectation_N_TS1}.\n~\\qed \\end{proof}\n\nWe note that the steps 4 and 5 of Algorithm \\ref{alg:fista_bktr_line_search} are exactly as Algorithm \\ref{alg:ista_bktr_line_search}.\nSince the dynamics of the step-size parameter is based on the reference points $y_k$, $\\tilde{p}_{\\alpha_k} (y_k)$ in Step 4 and the sufficient decrease condition in Step 5, \nthe results in Sections \\ref{sec:stoc_proc} and \\ref{subsec:successful}  hold  for Algorithm \\ref{alg:fista_bktr_line_search} under the same assumptions as for \nAlgorithm \\ref{alg:ista_bktr_line_search}. \n\n\n\\begin{lemma}\\label{lem:fista_linesearch}\n We denote $\\alpha^{succ}_k$ be the step-size parameter  at the successful iteration up to iteration $k$: \n\\begin{align*}\n    \\alpha^{succ}_k = \\alpha_{k'}, \\text{ where } k' = \\max \\{k'' \\text{ successful }, k'' \\leq k\\} .\n\\end{align*}\nWe also adopt the convention that $\\alpha^{succ}_k = \\alpha_0 = \\gamma \\alpha_1$ if there is no successful iteration up to iteration $k$. Then we have the following properties:\n\n1. $\\alpha^{succ}_{k-1} t_{k-1}^2\\geq \\alpha_k t_k(t_{k}-1)$ for every successful iteration $k$.\n\n2. $\\alpha^{succ}_{K} t_{K}^2 \\geq \\left(\\sum_{k \\in S, k \\le K } \\sqrt{\\alpha_k} /2\\right)^2$ for any number of iterations $K$.\n\n\\end{lemma}\n\n\\begin{proof}\n1. From the update of the algorithm and the fact that $k$ is successful: \n\\begin{align*}\n    t_k = t_k^{next} &=  \\frac{1 + \\sqrt{1+4\\theta_{k-1} t_{k-1}^2}}{2},\n\\end{align*}\nwe have $ \\theta_{k-1} t_{k-1}^2=  t_k(t_{k}-1)$.\nThe statement of the lemma $\\alpha^{succ}_{k-1} t_{k-1}^2\\geq \\alpha_k t_k(t_{k}-1)$ is equivalent to  \n\\begin{align*}\n\\alpha^{succ}_{k-1} \\geq \\alpha_k \\theta_{k-1}.\n\\end{align*}\nLet $k'$ be the previous successful iteration prior to $k$ ($k' < k$), i.e. $\\alpha^{succ}_{k-1} = \\alpha_{k'}$ and $\\alpha^{succ}_{k-1} = \\alpha_{0}$ if there is no successful iteration prior to $k$. By the definition of $k'$ we have \n$\\alpha_{k'+1} = \\tfrac{\\alpha_{k'}}{\\gamma},$ and $\\theta_{k'} = \\gamma $.\nSince the iterations from $k' + 1$ to $k-1$ are unsuccessful (if $k' <k-1$): \n\\begin{align*}\n    \\alpha_{k} &= \\alpha_{k'+1} \\cdot  \\gamma^{k-k'-1}\\nonumber\\\\\n    \\theta_{k-1} &= \\theta_{k'} \\gamma^{k'+1 -k}.\n\\end{align*}\nTherefore \n\\begin{align*}\n    \\alpha_{k} &= \\alpha_{k'+1} \\cdot  \\gamma^{k-k'-1}= \\alpha_{k'} \\cdot  \\gamma^{k-k'-2} \\nonumber\\\\\n    \\theta_{k-1} &= \\theta_{k'} \\gamma^{k'+1 -k} = \\gamma^{k'+2 -k},\n\\end{align*}\nand it proves the statement: \n    \\begin{align}\\label{eq:theta}\n    \\alpha_k \\theta_{k-1} = \\alpha_{k'} \\cdot  \\gamma^{k-k'-2} \\cdot \\gamma^{k'+2 -k} = \\alpha_{k'}. \n    \\end{align}\n    \n\n\n% \\katya{This proof is critical. Is it actually true for all iterations, including unsuccessful ones?}\n\n% \\trang{Actually we need $ \\alpha^{succ}_{K} t_{K}^2 \\geq \\left(\\sum_{k \\in S, k \\le K } \\sqrt{\\alpha_k} /2\\right)^2$ for every $K \\leq  N_\\epsilon-1$. (I forgot the $^{succ}$ before). Maybe the proof only need $K = N_\\epsilon-1$ but this lemma needs to be true for every iteration. We can prove this for all successful iterations, then for unsuccessful one, it still holds since it holds for the prior successful one i.e. no term changes (algo do not update $t_k$). But looking at this, technically $t_k$ do not has at least order $k$ directly, it has the order of the number of good iterations before $k$}\n\n\n2. First, we prove the statement for all iterations $K$ where $K$ is successful. We prove this by induction. \n\\begin{itemize}\n\\item Let $k_1 \\geq 1$ be the first successful iteration of the algorithm. By definition we have \n$\\alpha^{succ}_{k_1} = \\alpha_{k_1}, \\text{ since } k_1 = \\max \\{k'' \\text{ successful }, k'' \\leq k_1\\} $.  \nSince $k_1$ is successful:\n\\begin{align*}\n    t_{k_1} = t_{k_1}^{next} = \\frac{1 + \\sqrt{1+4\\theta_{(k_1-1)} t_{(k_1-1)}^2}}{2}= \\frac{1 + \\sqrt{1+4\\theta_{(k_1-1)} \\cdot 0}}{2} = 1.\n\\end{align*}\nTherefore the statement holds for $k_1$: \n\\begin{align*}\n    \\alpha^{succ}_{k_1} t_{k_1}^2 =  \\alpha_{k_1} \\cdot 1 \\geq \\left( \\sqrt{\\alpha_{k_1}} /2\\right)^2 = \\left(\\sum_{k \\in S, k \\le k_1 } \\sqrt{\\alpha_k} /2\\right)^2.\n\\end{align*}\n\\item Now for an iteration $K$ where $K$ is successful, we assume the statement holds for the successful iteration $k'$ prior to $K$ ($k'$ is the largest index of a successful iteration satisfying $k' < K$). We have \n$ \\alpha^{succ}_{k'} t_{k'}^2 \\geq \\left(\\sum_{k \\in S, k \\le k' } \\sqrt{\\alpha_k} /2\\right)^2$ and  \n\\begin{align*}\n    \\sqrt{\\alpha^{succ}_{k'}} t_{k'} \\geq \\sum_{k \\in S, k \\le k' } \\sqrt{\\alpha_k} /2.\n\\end{align*}\n\nSince both $K$ and $k'$ are successful, $ \\alpha^{succ}_{K} =  \\alpha_{K} $ and $ \\alpha^{succ}_{k'} =  \\alpha_{k'} $.\nUsing the fact that $K$ is successful: \n\\begin{align*}\n    t_{K} &= t_{K}^{next} = \\frac{1 + \\sqrt{1+4\\theta_{(K-1)} t_{(K-1)}^2}}{2} \\geq \\frac{1}{2} +  \\sqrt{\\theta_{(K-1)}}  t_{(K-1)}\\\\\n    &\\geq \\frac{1}{2} +  \\sqrt{\\theta_{(K-1)} } \\cdot t_{k'} \\geq \\frac{1}{2} +  \\sqrt{\\frac{\\alpha_{k'}}{\\alpha_{K}} } \\cdot t_{k'}.\n\\end{align*}\nwhere the last inequality follows from previous argument \\eqref{eq:theta}.\nHence we have \n\\begin{align*}\n    \\sqrt{\\alpha^{succ}_{K}} t_{K} = \\sqrt{\\alpha_{K}} t_{K} \\geq \\sqrt{\\alpha_{k'}} t_{k'} + \\sqrt{\\alpha_K} /2 = \\sqrt{\\alpha^{succ}_{k'}} t_{k'} + \\sqrt{\\alpha_K} /2.\n\\end{align*}\nTherefore \n\\begin{align*}\n    \\sqrt{\\alpha^{succ}_{K}} t_{K} \\geq \\sum_{k \\in S, k \\le k' } \\sqrt{\\alpha_k} /2 + \\sqrt{\\alpha_K} /2 = \\sum_{k \\in S, k \\le K } \\sqrt{\\alpha_k} /2,\n\\end{align*}\nand the statement holds for iteration $K$:\n$ \\alpha^{succ}_{K} t_{K}^2 \\geq \\left(\\sum_{k \\in S, k \\le K } \\sqrt{\\alpha_k} /2\\right)^2$.\n\\item By induction, the statement holds for all the successful iterations. For an arbitrary unsuccessful iteration $K$, the statement holds for the successful iteration $k'$ prior to $K$ and neither right hand side  {nor} left hand side term changes at  unsuccessful iteration. The statement is also true in the case when there is no successful iteration prior to $K$ as both sides equal to 0 ($t_0 = 0$). Therefore it holds for every iteration. \n\\end{itemize}\n% \\katya{what is there is no prior successful iteration? Show that is holds for $K=1$}\n~\\qed \\end{proof}\n\n\\begin{lemma}[$m_{k}$ on unsuccsessful steps]\\label{lem:unsuccessful_step_bound}\nLet $m_k$ be defined as in \\eqref{eq:define_m}. \nIf iteration $k$ of Algorithm \\ref{alg:fista_bktr_line_search}  is unsuccessful, then we have \n\\begin{align*}m_{k} = m_{k-1}. \\end{align*}\n\\end{lemma}\n\n\\begin{lemma}[$m_k$ on  successful step]\\label{lem:successful_step_bound}\nUnder Assumption \\ref{assumpt:convex_smooth}, if $k$ is a successful iteration of Algorithm \\ref{alg:fista_bktr_line_search}, then we have:\n %\\katya{I think here and in ISTA analysis it is best to use $b_k$ instead of $b_k$ for realizations and switch to $b_k$ when taking expectation. }\n\\begin{align*}\n&2\\alpha^{succ}_{k}t_k^2 v_k -2\\alpha^{succ}_{k-1} t_{k-1}^2v_{k-1} \\leq \\norm{u_{k-1}}^2 -  \\norm{u_k }^2 + \\lambda_k \\cdot \\| u_{k-1}\\|,\n\\end{align*}\nwhere \n\\begin{align*}\n\\lambda_k =  \\|2\\alpha_kt_k (\\nabla f(y_k )- g_k )\\|.\n\\end{align*}\nEquivalently, \n\\begin{align*}\nm_{k} -m_{k-1} \\leq \\lambda_k \\cdot \\|u_{k-1}\\|.\n\\end{align*}\n\\end{lemma}\n\n\\begin{proof}\nSince $k$ is a successful iteration, we have $F(\\tilde{p}_{\\alpha_k} (y_k)) \\leq \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (y_k), y_k)$. \nAlso $x_k = \\tilde{p}_{\\alpha_k} (y_k) $. %; t_k = t_k^{next}$.\nHence \n\\begin{align*}\n    F(x_k) = f(x_k) + h(x_k) \\leq f(y_k) +(x_k -y_k)^\\top g_k + \\frac{1}{2\\alpha_k}\\|x_k-y_k\\|^2 + h(x_k) .\n\\end{align*}\nBy the  convexity of $f$ at $x$ and $y_k$ and the convexity of $h$ at $x$ and $x_k$:\n\\begin{align*}\nf(x) \\geq f(y_k) + \\nabla f(y_k )^\\top (x-y_k),\n\\nonumber\\\\\nh(x) \\geq h(x_k) + \\gamma(y_k)^\\top (x-x_k),\n\\end{align*}\nwhere the second bound holds for every subgradient $\\gamma(y_k)$ of $h$ at $x_k$. \nWe have \n\\begin{align*}\nF(x) &= f(x) + h(x)\\geq f(y_k)+ h(x_k) + \\nabla f(y_k )^\\top (x-y_k)  + \\gamma(y_k)^\\top (x-x_k)\n\\nonumber\\\\\n&= f(x) + h(x)\\geq f(y_k)+ h(x_k) + [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x-y_k)  + \\gamma(y_k)^\\top ( y_k -x_k) \n\\end{align*}\nCombine this with the previous bound of $F(x_k)$ we have \n\\begin{align*}\n&F(x) - F(x_k) \\nonumber\\\\&\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x-y_k)  + \\gamma(y_k)^\\top ( y_k -x_k) - (x_k -y_k)^\\top g_k - \\frac{1}{2\\alpha_k}\\|x_k-y_k\\|^2\\nonumber\\\\\n&=  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x-y_k)  + (\\gamma(y_k)+g_k)^\\top ( y_k -x_k)  - \\frac{1}{2\\alpha_k}\\|x_k-y_k\\|^2.\n\\end{align*}\nUsing the fact that $x_k = \\tilde{p}_{\\alpha_k} (y_k) =\\arg\\min_y \\tilde{Q}_{\\alpha_k}(y, y_k)$, we get that $ y_k - x_k= \\alpha_k[\\gamma(y_k)+g_k]$ for some subgradient $\\gamma(y_k)$ of $h$ at $x_k$. Therefore\n\\begin{align*}\nF(x) - F(x_k)\n&\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x-y_k)  + \\alpha_k\\|\\gamma(y_k)+g_k) \\|^2 - \\frac{1}{2} \\alpha_k \\|\\gamma(y_k)+g_k)\\|^2\\nonumber\\\\\n&=  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x-y_k)  + \\frac{\\alpha_k}{2}\\|\\gamma(y_k)+g_k) \\|^2 .\n\\end{align*}\nApply this equation for $x = x_{k-1}$ and $x=x^*$ we have \n\\begin{align*}\nF(x_{k-1}) - F(x_k) &\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x_{k-1}-y_k)  + \\frac{\\alpha_k}{2}\\|\\gamma(y_k)+g_k) \\|^2 ,\n\\nonumber\\\\\nF(x^*) - F(x_k) &\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x^*-y_k)   + \\frac{\\alpha_k}{2}\\|\\gamma(y_k)+g_k) \\|^2 .\n\\end{align*}\nLet $v_k =  F(x_k) - F(x^*) $ then \n\\begin{align*}\nv_{k-1} - v_k &\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x_{k-1}-y_k)  + \\frac{\\alpha_k}{2}\\|\\gamma(y_k)+g_k) \\|^2  ,\n\\nonumber\\\\\n-v_k &\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x^*-y_k)   + \\frac{\\alpha_k}{2}\\|\\gamma(y_k)+g_k) \\|^2  .\n\\end{align*}\nMultiplying the first equality by $t_{k}-1$ and adding to the second equality, we get\n\\begin{align*}\n(t_{k}-1)v_{k-1} - t_k v_k &\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k )   + \\frac{\\alpha_k}{2} t_k \\|\\gamma(y_k)+g_k) \\|^2 .\n\\end{align*}\nMultiplying the previous inequality by $t_k$ we have\n \\begin{align*}\n &t_k(t_{k}-1)v_{k-1} - t_k^2 v_k \\nonumber\\\\\n &\\geq t_k [\\nabla f(y_k )+ \\gamma(y_k)]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k )   + \\frac{\\alpha_k}{2}t_k^2\\|\\gamma(y_k)+g_k) \\|^2 \\nonumber\\\\\n&=  t_k [\\nabla f(y_k )- g_k +g_k + \\gamma(y_k)]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k )  + \\frac{\\alpha_k}{2}t_k^2\\|\\gamma(y_k)+g_k) \\|^2\n\\nonumber\\\\\n&=  t_k [g_k + \\gamma(y_k)]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k )  + \\frac{\\alpha_k}{2}t_k^2\\|\\gamma(y_k)+g_k) \\|^2\n \\nonumber\\\\&\n+ t_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) \n\\nonumber\\\\\n&=  t_k \\frac{1}{\\alpha_k}(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k )  + \\frac{\\alpha_k}{2}t_k^2\\|\\gamma(y_k)+g_k) \\|^2\n \\nonumber\\\\&\n+ t_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) \n\\nonumber\\\\\n&=  t_k \n\\frac{1}{\\alpha_k}(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   + t_k \n\\frac{1}{\\alpha_k}(y_k - x_k)^\\top t_k(x_k -y_k)  \\nonumber\\\\&\n+ \\frac{1}{2\\alpha_k}t_k^2\\|x_k -y_k \\|^2\n+ t_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) \n\\nonumber\\\\\n&=  t_k \n\\frac{1}{\\alpha_k}(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n\\frac{1}{2\\alpha_k}\\norm{t_k (x_k -y_k)}^2 \\nonumber\\\\&\n+ t_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) .\n\\end{align*}\nMultiplying the final statement by $2\\alpha_k $ we get \n \\begin{align}\\label{eq_bound_mu_t}\n&~2\\alpha_k t_k(t_{k}-1)v_{k-1} - 2\\alpha_kt_k^2 v_k \\nonumber\\\\\n&\\geq 2t_k \n(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n\\norm{t_k (x_k -y_k)}^2  \\nonumber\\\\\n&  + 2\\alpha_kt_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) .\n\\end{align}\nNow we consider the following term: $2t_k \n(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n\\norm{t_k (x_k -y_k)}^2$.\n% \\begin{align*}\n% 2t_k \n% (y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n% \\norm{t_k (x_k -y_k)}^2.\n% \\end{align*}\nLet $a  = t_k \n(y_k - x_k), b = (t_{k}-1)x_{k-1} + x^* - t_k x_k$ we have \n\\begin{align}\\label{eq:03}\n    & \\quad \\ 2t_k \n(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n\\norm{t_k (x_k -y_k)}^2\\nonumber\\\\ &=  2ab - a^2 \\nonumber\\\\\n&= b^2 -b^2 +2ab - a^2= b^2 -(a-b)^2\\nonumber\\\\\n&=\\norm{(t_{k}-1)x_{k-1} + x^* - t_k x_k }^2 -\\norm{(t_{k}-1)x_{k-1} + x^* - t_k x_k  -t_k (y_k - x_k)}^2\\nonumber\\\\\n&=\\norm{(t_{k}-1)x_{k-1} + x^* - t_k x_k }^2 -\\norm{(t_{k}-1)x_{k-1} + x^* - t_k y_k  }^2.\n\\end{align}\nSince $k$ is a successful iteration, we have that $x^{prev}_{k} = x_{k-1}$. From the definition of $u_k$\n\\begin{align*}u_k = (t_{k}-1)x^{prev}_{k} + x^* - t_k x_k, \\end{align*} \nwe have that \n\\begin{align*}u_k = (t_{k}-1)x_{k-1} + x^* - t_k x_k. \\end{align*} \nNow from the definition of FISTAStep \\eqref{eq:fista_step}\n\\begin{align*}y_k = x_{k-1} + \\frac{t_{k-1}-1}{t_{k}} ( x_{k-1} - x^{prev}_{k-1} ),\\end{align*}\nwe have \n\\begin{align*}t_k y_k = t_kx_{k-1} + (t_{k-1}-1) ( x_{k-1} - x^{prev}_{k-1} ).\\end{align*}\nCombining these equations with \\eqref{eq:03} we get: \n\\begin{align*}\n& \\quad \\ 2t_k \n(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n\\norm{t_k (x_k -y_k)}^2\\nonumber\\\\\n&=\\norm{(t_{k}-1)x_{k-1} + x^* - t_k x_k }^2 -\\norm{(t_{k}-1)x_{k-1} + x^* - t_k y_k  }^2\\nonumber\\\\\n&=\\norm{u_k }^2 -\\norm{(t_{k}-1)x_{k-1} + x^* - t_kx_{k-1} - (t_{k-1}-1) ( x_{k-1} -  x^{prev}_{k-1} )  }^2\\nonumber\\\\\n&=\\norm{u_k }^2 -\\norm{-t_{k-1} x_{k-1} + x^*  + (t_{k-1}-1)  x^{prev}_{k-1}  }^2\\nonumber\\\\\n&=\\norm{u_k }^2 -\\norm{-u_{k-1}}^2.\n\\end{align*}\nSubstituting back to previous equality \\eqref{eq_bound_mu_t} we get \n \\begin{align*}\n&2\\alpha_k t_k(t_{k}-1)v_{k-1} - 2\\alpha_kt_k^2 v_k \\nonumber\\\\\n&\\geq \\norm{u_k }^2 -\\norm{u_{k-1}}^2  + 2\\alpha_kt_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) .\n\\end{align*}\nTherefore  \n  \\begin{align*}\n&2\\alpha_kt_k^2 v_k -2\\alpha_k t_k(t_{k}-1)v_{k-1}\\nonumber\\\\ % &\\leq \\norm{u_{k-1}}^2 -  \\norm{u_k }^2 -2\\alpha_kt_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) \\nonumber\\\\\n&\\leq \\norm{u_{k-1}}^2 -  \\norm{u_k }^2 + \\|2\\alpha_kt_k (\\nabla f(y_k )- g_k )\\| \\cdot \\| u_{k-1}\\|.\n\\end{align*}\nFrom Lemma \\ref{lem:fista_linesearch} we have $\\alpha^{succ}_{k-1} t_{k-1}^2\\geq \\alpha_k t_k(t_{k}-1)$\nfor every successful iteration $k$. Also since $k$ is successful, we have  $\\alpha_k = \\alpha^{succ}_{k}$.\nTherefore we have the final bound of Lemma \\ref{lem:successful_step_bound}.\n~\\qed \\end{proof}\n\n\\begin{lemma}\\label{lem:after_K}\nGiven any integer $K$, after $K$ iterations of the algorithm, the following bound holds: \n\\begin{align*}\n    2\\alpha^{succ}_{K} t_{K}^2v_{K} \\leq 3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\lambda_{i} \\right]  + 2\\sum_{k=1}^{K} \\lambda_{k} ^2  .\n\\end{align*}\n\\end{lemma}\n\n\\begin{proof}\nFor every $K$, from Lemma \\ref{lem:unsuccessful_step_bound}  we have\n\\begin{align*}\n    m_K - m_0 = \\sum_{k \\in S, k\\leq K} (m_{k} -m_{k-1}),\n\\end{align*}\nwhere $S$ is the set of successful iterations.\nWe apply Lemma \\ref{lem:successful_step_bound} for successful iterations and then \n\\begin{align*}\n    m_K - m_0 &= \\sum_{k \\in S, k\\leq K} (m_{k} -m_{k-1})\\leq \\sum_{k \\in S, k\\leq K} \\lambda_k \\cdot \\|u_{k-1}\\| \\leq \\sum_{k=1}^{K} \\lambda_k \\cdot \\| u_{k-1}  \\|  .\n\\end{align*}\nPlugging this and the definition of $m_k$ back to the previous inequality, we have our key bound: \n\\begin{align}\\label{eq:thm_key}\n    2\\alpha^{succ}_{K} t_{K}^2v_{K} +\\norm{u_K }^2 \n    &\\leq \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2\\right) + \\sum_{k=1}^{K} \\lambda_k \\cdot \\| u_{k-1}\\| .\n\\end{align}\nSimilar to the proof of ISTA, \nwe follow the argument in \\cite{Schmidt-Roux-Bach} with two steps. First, we use the key bound \\eqref{eq:thm_key} to bound $\\| u_{k-1}\\|$ in terms of the noise terms $\\lambda_k$. Then we use this bound on $\\| u_{k-1}\\|$ to substitute into the last sum of the key bound \\eqref{eq:thm_key} and obtain the final derivation in terms of the noise $\\lambda_k$.\nIgnoring the positive term $2\\alpha^{succ}_{K} t_{K}^2v_{K}$, we have \n\\begin{align*}\n    \\norm{u_K }^2 \n    &\\leq \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2\\right) + \\sum_{k=1}^K \\lambda_k \\cdot \\| u_{k-1}\\| \\nonumber\\\\\n    &= 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\lambda_1 \\cdot \\| u_{0}\\| + \\sum_{k=1}^{K} \\lambda_{k+1} \\cdot \\| u_{k}\\|.\n\\end{align*}\nThis is equivalent to \n\\begin{align*}\n    \\norm{u_K }^2 \n    \\leq Z + \\sum_{k=1}^{K} \\lambda_{k+1} \\| u_{k}\\|,\n\\end{align*}\nwhere $Z = 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\lambda_1 \\| u_{0}\\|$.\nUsing this inequality and applying Lemma 1 in \\cite{Schmidt-Roux-Bach} for the non-negative sequence $\\{\\|u_K\\|\\}$, we have\n\\allowdisplaybreaks\n\\begin{align*}\n    \\| u_{k}\\| &\\leq \\frac{1}{2} \\sum_{i=1}^{k} \\lambda_{i+1} + \\left( Z + \\left(\\frac{1}{2} \\sum_{i=1}^{k} \\lambda_{i+1} \\right)^2\\right)^{1/2}\n    \\leq  \\sum_{i=1}^{k} \\lambda_{i+1} +  Z^{1/2} \\nonumber\\\\\n    &\\leq  \\sum_{i=1}^{k+1} \\lambda_{i} + \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  \\right)^{1/2},\n\\end{align*}\nwhere the last line follows from the fact that\n\\begin{align*}\n    Z^{1/2} &= \\sqrt{2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\lambda_1 \\| u_{0}\\|  } \\leq  \\sqrt{2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\lambda_1^2 +\\| u_{0}\\|^2  } \\nonumber\\\\\n    &\\leq \\sqrt{2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\lambda_1^2  } \\leq  \\lambda_1 + \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  \\right)^{1/2}.\n\\end{align*}\nDropping the term $\\norm{u_K }^2 $ in \nthe inequality \\eqref{eq:thm_key} and using the bound on $\\| u_{k}\\|$ we have: \n\\allowdisplaybreaks\n\\begin{align*}\n    &2\\alpha^{succ}_{K} t_{K}^2v_{K} \\nonumber\\\\\n    &\\leq 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\| u_{k-1}\\| \\nonumber\\\\\n    &\\leq 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\left[\\sum_{i=1}^{k} \\lambda_{i} + \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  \\right)^{1/2} \\right] \\nonumber\\\\\n    &= 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\left[\\sum_{i=1}^{k} \\lambda_{i} \\right] + \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  \\right)^{1/2} \\sum_{k=1}^{K} \\lambda_{k}   \\nonumber\\\\\n    &\\leq 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\lambda_{i} \\right]  + \\sum_{k=1}^{K} \\lambda_{k} ^2+  \\frac{1}{2} \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  \\right)  + \\sum_{k=1}^{K} \\lambda_{k} ^2\\nonumber\\\\\n    &= 3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\lambda_{i} \\right]  + 2\\sum_{k=1}^{K} \\lambda_{k} ^2,  \n\\end{align*}\nwhich is our desired bound. \n~\\qed \n\\end{proof}\n\n\\begin{theorem}[Upper bound of the number of large true successful steps]\\label{th:bound_expectation_N_TS}\nUnder Assumptions \\ref{assumpt:convex_smooth}, \\ref{assumpt:bounded_below}\nand  \\ref{assumpt:fista_noise}, we have  \n \\begin{align*}\n \\EE [N_G] \\leq \\sqrt{\n \\frac{2}{\\stepsizebar\\epsilon} \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\frac{4(\\beta + 2)^2}{\\beta^2} + \\frac{8(\\beta+2)}{\\beta+1}  \n \\right] }. \n\\end{align*}\n\n\\end{theorem}\n\n\\begin{proof} \nSince the statement of Lemma \\ref{lem:after_K} holds for every realizations of Algorithm \\ref{alg:fista_bktr_line_search}, for $K = N_\\epsilon -1$ we have: \n\\begin{align*}\n    2\\Alpha^{succ}_{N_\\epsilon -1} T_{N_\\epsilon -1}^2 (F(X_{N_\\epsilon -1}) - F^*)\\leq 3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{N_\\epsilon -1} \\Lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\Lambda_{i} \\right]  + 2\\sum_{k=1}^{N_\\epsilon -1} \\Lambda_{k} ^2.\n\\end{align*}\nRecall that $N_\\epsilon$ be the stopping time that $N_\\epsilon = \\inf\\{k: F(X_k) - F^* \\leq \\epsilon\\}$. Hence for $K = N_\\epsilon -1$, we have that $F(X_K) - F^* > \\epsilon$ and \n\\begin{align*}\n    2\\Alpha^{succ}_{N_\\epsilon -1} T_{N_\\epsilon -1}^2 &\\leq \\frac{3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{N_\\epsilon -1} \\Lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\Lambda_{i} \\right]  + 2\\sum_{k=1}^{N_\\epsilon -1} \\Lambda_{k}^2}{F(X_{N_\\epsilon -1}) - F^*} \\nonumber\\\\\n    &\\leq \\frac{3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\Lambda_{i} \\right]  + 2\\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} ^2}{\\epsilon}.\n\\end{align*}\nLet $\\mathcal{N}_G$ be the set of \\textit{good} iterations  (i.e. true and successful iterations $k$ with $\\alpha_k\\geq \\stepsizebar $), with $ k \\leq N_\\epsilon -1$, then $N_G$ is the cardinality of that set. \nUsing the inequality $ \\alpha^{succ}_{K} t_{K}^2 \\geq \\left(\\sum_{k \\in S, k \\le K } \\sqrt{\\alpha_k} /2\\right)^2$ for $K =  N_\\epsilon-1$ from Lemma \\ref{lem:fista_linesearch} we have \n\\begin{align*} \n\\Alpha^{succ}_{N_\\epsilon -1} T_{N_\\epsilon -1}^2 \n&\\geq \\left(\\sum_{k \\in S, k \\le N_\\epsilon -1} \\sqrt{\\Alpha_k} /2\\right)^2 \\nonumber\\\\\n&\\geq \\left(\\sum_{k\\in \\mathcal{N}_G, k \\le N_\\epsilon -1 } \\sqrt{\\Alpha_k} /2\\right)^2\n\\geq \\left(N_G \\sqrt{\\stepsizebar} /2\\right)^2 \n= \\frac{1}{4} (N_G)^2 \\stepsizebar.\n\\end{align*}\nSubstituting this to the previous inequality we get \n \\begin{align*}\n(N_G)^2  &\\leq  \\frac{2}{\\stepsizebar\\epsilon} \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\Lambda_{i} \\right]  + 2\\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} ^2   \\right].\n\\end{align*}\nTaking total expectation we have: \n\\begin{align*}\n\\EE [(N_G)^2 ]   &\\leq \\frac{2}{\\stepsizebar\\epsilon} \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 \n+ \\EE \\left[ \\sum_{k=1}^{N_\\epsilon } \\Lambda_{k} \\cdot \\left(\\sum_{i=1}^{k-1} \\Lambda_{i} \\right)  \\right] + 2\\EE \\left[\\sum_{k=1}^{N_\\epsilon }  \\Lambda_{k} ^2  \\right] \n \\right].\n\\end{align*}\nFrom \\eqref{eq:fista_noise} we have: \n\\begin{align*}\n    \\EE \\left[\\Lambda_{k}^2| \\pastone\\right] = \\EE \\left[4\\alpha_{k}^2t_k^2 \\| (\\nabla f(y_{k} )- G_{k} )\\|^2 | \\pastone\\right] \\leq \\frac{4}{k^{2+ \\beta}},\n\\end{align*}\nHence using a similar argument as Theorem \\ref{th:bound_expectation_N_TS1} we get \n\\begin{align*}\n\\EE [(N_G)^2 ]\n &\\leq \\frac{2}{\\stepsizebar\\epsilon} \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  + \\frac{4(\\beta + 2)^2}{\\beta^2} + \\frac{8(\\beta+2)}{\\beta+1} \n \\right].\n\\end{align*}\nBy the expectation inequality: \n \\begin{align*}\n\\EE [N_G] \\leq \\sqrt{\n \\frac{2}{\\stepsizebar\\epsilon} \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\frac{4(\\beta + 2)^2}{\\beta^2} + \\frac{8(\\beta+2)}{\\beta+1}  \n \\right] }.\n\\end{align*}\nwe get the final results of Theorem \\ref{th:bound_expectation_N_TS}. \n~\\qed \\end{proof}\n\nOur final result is a straightforward corollary of Theorem \\ref{th:framework} and \\ref{th:bound_expectation_N_TS}. \n","units":[{"start":41250,"end":41714,"start_line":393,"end_line":405,"raw_tex":"\\begin{lemma}\n\\label{lem:good_est_good_model_successful} \nSuppose that iteration $k$ is true, i.e. \n\\begin{align*}\n    \\|g_k-\\nabla f(y_k)\\|\\leq \\constG  \\|D_{\\alpha_k}(y_k)\\|,\n\\end{align*}\nwith  {$\\constG \\leq \\frac{1}{3}$} and $D_{\\alpha_k}(y_k)$ is the gradient mapping of $F$ at $y_k$. If \n\\begin{align*}\n\\stepsize{k}{} \\le \\stepsizebar := \\frac{1}{L}\\left( 1 - \\frac{2\\constG}{1-\\constG}\\right) ,\n\\end{align*}\nthen the $k$-th step is successful. \n\n\\end{lemma}","sha256_utf8":"fcf99e1ec4baa893a6cc12197390889eb086a34350840081260e99bedb2e77fd"},{"start":41717,"end":45196,"start_line":407,"end_line":459,"raw_tex":"\\begin{proof}\n\n\nFrom the $L$-smoothness of $f$ we have\n\\begin{align*}\n    f(\\tilde{p}_{\\alpha_k} (y_k))  &\\le f(y_k) + \\nabla f(y_k)^\\top(\\tilde{p}_{\\alpha_k} (y_k) - y_k) + \\frac{L}{2}\\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 \\nonumber\\\\\n     &= f(y_k) +  g_k^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k) + [\\nabla f(y_k)-g_k]^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k)  \\nonumber\\\\\n     &\\quad+ \\frac{1}{2\\alpha_k} \\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2  + \\left(\\frac{L}{2}- \\frac{1}{2\\alpha_k}\\right)\\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 .\n\\end{align*}\nAdding the non-smooth term $ h(\\tilde{p}_{\\alpha_k} (y_k))$ we have\n\\begin{align*}\n    &F (\\tilde{p}_{\\alpha_k} (y_k))  \\nonumber\\\\\n     &\\le f(y_k) +  g_k^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k) + \\frac{1}{2\\alpha_k} \\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 + h(\\tilde{p}_{\\alpha_k} (y_k))  \\nonumber\\\\\n     &\\quad + [\\nabla f(y_k)-g_k]^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k) + \\left(\\frac{L}{2}- \\frac{1}{2\\alpha_k}\\right)\\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 \n     \\nonumber\\\\\n     &= \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (y_k), y_k) + [\\nabla f(y_k)-g_k]^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k) + \\left(\\frac{L}{2}- \\frac{1}{2\\alpha_k}\\right)\\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 .\n\\end{align*}\nTo show that the iteration is successful i.e. $F(\\tilde{p}_{\\alpha_k} (y_k)) \\leq \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (y_k), y_k)$\nwe need to prove that \n\\begin{align}\\label{eq:02}\n    [\\nabla f(y_k)-g_k]^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k) \\leq \\left(\\frac{1}{2\\alpha_k} - \\frac{L}{2}\\right)\\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 .\n\\end{align}\nNote that the right hand side is non-negative since $\\alpha_k \\leq \\frac{1}{L}$. \nOur definition of the true iteration leads to \n\\begin{align}\\label{eq:01}\n    \\|g_k-\\nabla f(y_k)\\|\n    &\\leq \\constG \\left \\|\\frac{1}{\\alpha_k}\\Big(y_k - p_{\\alpha_k} (y_k)\\Big)\\right\\|\\nonumber  \\\\\n    &\\leq  \\frac{\\constG}{\\alpha_k} \\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\| + \\frac{\\constG}{\\alpha_k} \\left \\|\\tilde{p}_{\\alpha_k} (y_k) - p_{\\alpha_k} (y_k)\\right\\|\\nonumber  \\\\\n    &=  \\frac{\\constG}{\\alpha_k} \\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\| + \\frac{\\constG}{\\alpha_k} \\norm{\\alpha_k\\nabla f(y_k)-\\alpha_k g_k },\n\\end{align}\nwhere the last inequality follows  from the nonexpansiveness of the prox-operator $\\text{prox}_{\\alpha h}$ for  proper, closed, and convex $h(x)$, i.e.,\n\\begin{align*}\n    \\|\\text{prox}_{\\alpha h}(z_1) - \\text{prox}_{\\alpha h}(z_2)\\| \\leq \\|z_1 - z_2\\| \\text{ for all } z_1, z_2\\in\\R^d.\n\\end{align*}\nThe bound \\eqref{eq:01} is equivalent to\n\\begin{align*}\n    (1-\\constG)\\norm{\\nabla f(y_k)-g_k} \\leq  \\frac{\\constG}{\\alpha_k} \\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\|.\n\\end{align*}\nTherefore we show \\eqref{eq:02} by\n\\begin{align*}\n    \\norm{\\nabla f(y_k)-g_k} &\\leq  \\frac{\\constG}{1-\\constG} \\frac{1}{\\alpha_k} \\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\|  \\nonumber\\\\\n    &\\leq \\left( \\frac{1- \\alpha_k  L}{2}\\right)\\frac{1}{\\alpha_k} \\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\| = \\left( \\frac{1}{2\\alpha_k} - \\frac{L}{2}\\right)\\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\|,\n\\end{align*}\nwhere we use the condition that \n\\begin{align*}\n    \\frac{\\constG}{1-\\constG} \\leq \\frac{1- \\alpha_k  L}{2},\n\\end{align*}\nwhich is the consequence of the small step-size parameter: \n\\begin{align*}\n    \\alpha_k  \\leq \\frac{1}{L}\\left( 1 - \\frac{2\\constG}{1-\\constG}\\right).\n\\end{align*}\n\n~\\qed \\end{proof}","sha256_utf8":"ba0640886b91c5eb45d2dc7d72877e91929302d19d4dea5914d59cc85283a2f3"},{"start":46134,"end":46519,"start_line":470,"end_line":476,"raw_tex":"\\begin{theorem}[Bounding $\\EE (N_\\epsilon)$ based on $\\EE(N_G)$]\\label{th:framework}\nLet $\\stepsizebar$ be the value stated in Lemma \\ref{lem:good_est_good_model_successful}.\n Under Assumption \\ref{assumpt:key} we have \n\\begin{align*}\n\\EE (N_\\epsilon -1) \\leq \\frac{2p}{(2p-1)^2}\\left (2\\EE(N_G)+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right).\n\\end{align*}\n\\end{theorem}","sha256_utf8":"c59465760237095e2a01fcd4da684375bf36221ddb3a9023f6f34447d4ec4b49"},{"start":46520,"end":50575,"start_line":477,"end_line":551,"raw_tex":"\\begin{proof}\nBy the update rule of $\\alpha_k$ in Algorithm \\ref{alg:ista_bktr_line_search}, the stochastic process $\\{\\Alpha_k\\}_{k\\geq 1}$ follows: \n\\begin{align*}\n\\Alpha_{1}&= \\alpha_1, \\quad \n\\Alpha_{k+1}=\\left \\{ \\begin{array}{ll} \n\\gamma^{-1} \\Alpha_k & {\\rm if\\  }  \\iterSucc=1,  \\\\\n \\gamma \\Alpha_k &   {\\rm if\\  }  \\iterSucc=0 .\n\\end{array}\\right . \n\\end{align*}\n\nLet us now define the following indicator random variables: %two indicator random variables, in addition to $\\iterTrue$ defined earlier,\n\\begin{align*}\n\\iterLarge=\\mathbbm{1}\\{ \\Alpha_k >\\stepsizebar\\}, \\\\\n\\iterLargebar= \\mathbbm{1}\\{ \\Alpha_k \\geq \\stepsizebar\\}.\n\\end{align*}\nClearly \n\\begin{align*}\n\\EE (N_\\epsilon-1) = \\EE\\left (\\sum_{k=1}^{N_\\epsilon-1} \\iterLarge\\right ) + \\EE \\left (\\sum_{k=1}^{N_\\epsilon-1} (1-\\iterLarge)\\right ).\n\\end{align*}\n\nFrom Lemma 2.3 in \\cite{Cartis-Scheinberg}, under Assumption \\ref{assumpt:key} and using the fact that all true iterations are successful when $\\alpha_k \\le \\stepsizebar$, we have: \n\\begin{align*}\n\\EE\\left (\\sum_{k=1}^{N_\\epsilon-1} (1-\\iterLarge)\\right)\\leq \\frac{1}{2p}\\EE(N_\\epsilon -1 ).\n\\end{align*}\n\n\nWe define \n\\begin{itemize}\n\\item $N_{FS}=\\sum_{k=1}^{N_\\epsilon-1} \\iterLargebar (1-\\iterTrue)\\iterSucc$, which is the number of false successful iterations with $\\Alpha_k\\geq \\stepsizebar$.\n\\item $N_F=\\sum_{k=1}^{N_\\epsilon-1} \\iterLargebar (1-\\iterTrue)$, which is the number of false  iterations with $\\Alpha_k\\geq \\stepsizebar$.\n\\item $N_G=\\sum_{k=1}^{N_\\epsilon-1} \\iterLargebar \\iterTrue \\iterSucc$, which is the number of true successful iterations with $\\Alpha_k\\geq \\stepsizebar$.\n\\item $N_T=\\sum_{k=1}^{N_\\epsilon-1} \\iterLargebar \\iterTrue$, which is the number of true iterations with $\\Alpha_k\\geq \\stepsizebar$.\n% \\item $N_{TU}=\\sum_{k=1}^{N_\\epsilon-1} \\iterLarge \\iterTrue (1-\\iterSucc)$, which is the number of true unsuccessful iterations with $\\Alpha_k> \\stepsizebar$.\n\\item $N_U=\\sum_{k=1}^{N_\\epsilon-1} \\iterLarge  (1-\\iterSucc)$, which is the number of unsuccessful iterations with $\\Alpha_k> \\stepsizebar$.\n\\end{itemize}\n\nFrom equations (11) and (12) of Lemma 2.7 in \\cite{Cartis-Scheinberg} we have\n\\begin{align}\\label{eq:binom}\n\\EE(N_{FS})\\leq \\EE(N_{F})\\leq\\frac {1-p}{p}[\\EE(N_G)+\\EE(N_U)],\n\\end{align}\nand\n\\begin{align}\\label{eq:boundM3}\n\\EE(N_U)\\leq \\EE(N_{FS})+\\EE(N_G) + \\log_\\gamma(\\stepsizebar/\\alpha_1).\n\\end{align}\nPlugging \\eqref{eq:boundM3} into  \\eqref{eq:binom}\nwe obtain\n\\begin{align*}\n\\EE(N_{FS})\\leq \\frac {1-p}{p}\\left[\\EE(N_{FS})+2\\EE(N_G) + \\log_\\gamma(\\stepsizebar/\\alpha_1)\\right],\n\\end{align*}\nand, hence,\n\\begin{align*}\n\\frac {2p-1}{p}\\EE(N_{FS})\\leq \\frac {1-p}{p}\\left[ 2\\EE(N_G) + \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right].\n\\end{align*}\nThis finally implies \n\\begin{align}\\label{eq:boundN1}\n\\EE(N_{FS})\\leq \\frac {1-p}{2p-1}\\left [2\\EE(N_G)+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right ].\n\\end{align}\nNow we can bound the expected total  number of  iterations when $\\alpha_k>\\stepsizebar$, using %\\eqref{eq:N2}, \n\\eqref{eq:boundM3} and \\eqref{eq:boundN1}  \nand adding the terms to obtain the result of the lemma, namely,\n\\begin{align*}\n\\EE(N_F+N_T)&\\leq \\EE(N_F+N_U+N_G)\\leq \\frac{1}{p}\\EE(N_U+N_G)\\nonumber\\\\\n&\\leq \\frac{1}{2p-1}\\left (2\\EE(N_G)+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right).\n\\end{align*}\nNext, from Lemma 2.7 in \\cite{Cartis-Scheinberg},\\footnote{Final statement of the Lemma 2.7 in \\cite{Cartis-Scheinberg}  replaces $\\EE(N_G)$ with its bound, but here we\nare yet to derive such a  bound, therefore we explicitly use $\\EE(N_G)$.}\n\\begin{align*}\n\\EE\\left (\\sum_{k=1}^{N_\\epsilon-1}\\iterLarge \\right )\\leq \\frac{1}{2p-1}\\left (2\\EE(N_G)+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right). \n\\end{align*}\nHence, using the two previous results we have \n\\begin{align*}\n\\EE (N_\\epsilon -1)\\leq \\frac{1}{2p} \\EE (N_\\epsilon -1) +   \\frac{1}{2p-1} \\left (2\\EE(N_G)+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right).\n\\end{align*}\nThe result of the theorem follows. \n~\\qed \\end{proof}","sha256_utf8":"b4127294bec3b5123ca6ead8e3b368ee730194465cd1a348f55cde096ecea1a2"},{"start":51085,"end":51426,"start_line":561,"end_line":567,"raw_tex":"\\begin{lemma}[Bounds for a successful step]\\label{lem:successful_step_bound1}\nIf $k$ is a successful iteration then we have:\n\\begin{align*}\n\\|x_{k} - x^*\\|^2 - \\|x_{k-1} - x^* \\|^2 \\leq  - 2\\alpha_k (F(x_k) - F(x^*) )  + \\lambda_k \\cdot \\|x_{k-1} -x^*\\|,\n\\end{align*}\nwhere $\\lambda_k = \\| 2\\alpha_k (\\nabla f(x_{k-1})-g_k) \\| $.\n\\end{lemma}","sha256_utf8":"1a7cc48416c21241fd96fef760bdd1d8b8ac7629d23a357d29a6ed3661f34ebb"},{"start":51428,"end":54172,"start_line":569,"end_line":612,"raw_tex":"\\begin{proof}\nSince iteration $k$ is successful,  the sufficient decrease condition holds: \n\\begin{align*}\n    F(\\tilde{p}_{\\alpha_k} (x_{k-1})) &\\leq \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (x_{k-1}), x_{k-1}) \\nonumber\\\\\n    &= f(x_{k-1}) +(x_k -x_{k-1})^\\top g_k + \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2 + h(x_k).\n\\end{align*}\nBy the  convexity of $f$ and $h$, applied at $x_{k-1}$ and $x_k$ respectively, for any $x$,  \n \\begin{align*}\nf(x) &\\geq f(x_{k-1}) + \\nabla f(x_{k-1})^\\top (x-x_{k-1}),\n\\nonumber\\\\\nh(x) &\\geq h(x_k) + \\gamma(x_{k-1})^\\top (x-x_k),\n\\end{align*}\nwhere the second bound holds for every subgradient $\\gamma(x_{k-1})$ of $h$ at $x_{k}$. \nSumming those inequality we have \n\\begin{align*}\nf(x) +h(x) \\geq f(x_{k-1}) + \\nabla f(x_{k-1})^\\top (x-x_{k-1})+ h(x_k) + \\gamma(x_{k-1})^\\top (x-x_k).\n\\end{align*}\nCombining this with the sufficient decrease condition we get: \n\\begin{align*}\n    &F(x) - F(x_k) \\nonumber\\\\&\\geq \\nabla f(x_{k-1})^\\top (x-x_{k-1}) + \\gamma(x_{k-1})^\\top (x-x_k) - (x_k -x_{k-1})^\\top g_k - \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2\\nonumber\\\\\n    &= (\\nabla f(x_{k-1})-g_k) ^\\top (x-x_{k-1}) + (\\gamma(x_{k-1})+g_k)^\\top (x-x_k) - \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2.\n\\end{align*}\nUsing the fact that $x_k = \\tilde{p}_{\\alpha_k} (x_{k-1}) =\\arg\\min_x \\tilde{Q}_{\\alpha_k}(x, x_{k-1})$, we get that $ x_{k-1} - x_k= \\alpha_k[\\gamma(x_{k-1})+g_k]$ for some subgradient $\\gamma(x_{k-1})$ of $h$ at $x_{k}$. Therefore\n\\begin{align*}\n    &F(x) - F(x_k) \\nonumber\\\\\n    &\\geq (\\nabla f(x_{k-1})-g_k) ^\\top (x-x_{k-1}) + \\frac{1}{\\alpha_k}(x_{k-1} - x_k)^\\top (x-x_k) - \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2\\nonumber\\\\\n    &= (\\nabla f(x_{k-1})-g_k) ^\\top (x-x_{k-1}) + \\frac{1}{\\alpha_k}(x_{k-1} - x_k)^\\top (x-x_{k-1} + x_{k-1} -x_k) \\nonumber\\\\\n    & \\quad - \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2\\nonumber\\\\\n    &= (\\nabla f(x_{k-1})-g_k) ^\\top (x-x_{k-1}) + \\frac{1}{\\alpha_k}(x_{k-1} - x_k)^\\top (x-x_{k-1}) + \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2\\nonumber\\\\\n    &= (\\nabla f(x_{k-1})-g_k) ^\\top (x-x_{k-1}) + \\frac{1}{2\\alpha_k} \\left( \\|x-x_k \\|^2-\\|x-x_{k-1} \\|^2\\right),\n\\end{align*}\nand we get the bound for $x=x^*$: \n\\begin{align*}\n    &2\\alpha_k (F(x_k) - F(x^*) ) - 2\\alpha_k (\\nabla f(x_{k-1})-g_k) ^\\top (x_{k-1} -x^*)\n    \\leq \\|x_{k-1} - x^*\\|^2 - \\|x_k - x^* \\|^2.\n\\end{align*}\nRearranging the terms, we get the statement of Lemma \\ref{lem:successful_step_bound1}:\n\\begin{align*}\n    \\|x_k - x^* \\|^2 - \\|x_{k-1} - x^*\\|^2 &\\leq -2\\alpha_k (F(x_k) - F(x^*) ) + 2\\alpha_k (\\nabla f(x_{k-1})-g_k) ^\\top (x_{k-1} -x^*)\\nonumber\\\\\n    &\\leq -2\\alpha_k (F(x_k) - F(x^*) ) + \\lambda_k \\cdot\\|x_{k-1} -x^*\\|,\n\\end{align*}\nwhere $\\lambda_k = \\|2\\alpha_k (\\nabla f(x_{k-1})-g_k)\\|$.\n\n~\\qed \\end{proof}","sha256_utf8":"a447eb588a2fb26032018a0710ef8f024ca64155de7763710298f8732c01933e"},{"start":54460,"end":54833,"start_line":615,"end_line":622,"raw_tex":"\\begin{lemma}\\label{lem:after_K1}\n    After $K$ iterations of the algorithm, the following bound holds: \n\\begin{align*}\n\\sum_{k\\in S, k \\leq K} 2\\alpha_k (F(x_k) - F(x^*) )\n    &\\leq 2\\|x_0 - x^* \\|^2 + \\sum_{k=1}^K \\lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\lambda_i \\right)+ 2\\sum_{k=1}^K \\lambda_k^2,\n\\end{align*}\nwhere $S$ is the set of successful iterations.\n\\end{lemma}","sha256_utf8":"2b17f03dc722745e5efcccf6af7971ad95468ca7f2a58ab0340141f62f4f35d0"},{"start":54834,"end":58002,"start_line":623,"end_line":665,"raw_tex":"\\begin{proof}\nFor every $K$, from Lemma \\ref{lem:successful_step_bound1} we have\n\\begin{align*}\n    \\|x_K - x^* \\|^2 - \\|x_0 - x^* \\|^2 \n    &= \\sum_{k=1}^{K} \\left(\\|x_{k} - x^* \\|^2 -\\|x_{k-1} - x^* \\|^2 \\right) \\nonumber\\\\\n    &\\leq \\sum_{k\\in S, k \\leq K} \\left[\\lambda_k \\cdot \\|x_{k-1} -x^*\\| - 2\\alpha_k (F(x_k) - F(x^*) ) \\right],\n\\end{align*}\nwhere $S$ is the set of successful iterations. \nHence we arrive at our key bound\n\\begin{align}\\label{eq:thm_key1}\n    &\\|x_K - x^* \\|^2 - \\|x_0 - x^* \\|^2 + \\sum_{k\\in S, k \\leq K} 2\\alpha_k (F(x_k) - F(x^*) )\\leq  \\sum_{k=1}^K \\lambda_k  \\cdot \\| x_{k-1} -x^* \\|.\n\\end{align}\nNow we follow a similar argument as \\cite{Schmidt-Roux-Bach} with two steps. First, we use the key bound \\eqref{eq:thm_key1} to bound the distance to optimal point $u_{k-1} = \\| x_{k-1} -x^* \\|$ in terms of the initial distance $u_{0} = \\| x_{0} -x^* \\|$ and the noise terms $\\lambda_k$. Then we use this bound on $\\| x_{k-1} -x^* \\|$ to substitute into the last sum of the key bound \\eqref{eq:thm_key1} and obtain the final derivation in terms of the noise $\\lambda_k$.\\\\\nDropping the positive term $\\sum_{k\\in S, k \\leq K} 2\\alpha_k (F(x_k) - F(x^*) )$ we get\n\\begin{align*}\n    \\|x_K - x^* \\|^2 \n    &\\leq \\|x_0 - x^* \\|^2 + \\sum_{k=1}^K \\lambda_k \\cdot \\| x_{k-1} -x^* \\| \\nonumber\\\\\n    &= \\|x_0 - x^* \\|^2 + \\lambda_1 \\cdot \\| x_{0} -x^* \\| + \\sum_{k=1}^K \\lambda_{k+1} \\cdot\\| x_{k} -x^* \\|.\n\\end{align*}\nLetting $u_k = \\| x_{k} -x^* \\|$, this is equivalent to \n\\begin{align*}\n    u_K^2 = \\|x_K - x^* \\|^2  \\leq Z + \\sum_{k=1}^K\\lambda_{k+1} u_k.\n\\end{align*}\nwhere $Z = \\|x_0 - x^* \\|^2 + \\lambda_1 \\cdot \\| x_{0} -x^* \\| = u_0^2 + \\lambda_1 u_0$. %\\katya{there seems to be incorrect switching between $k$ and $K$ in a few places here} \nUsing this inequality and applying Lemma 1 in \\cite{Schmidt-Roux-Bach} for the non-negative sequence $\\{u_K\\}$ have\n\\begin{align*}\n     u_{k} \\leq \\frac{1}{2} \\sum_{i=1}^{k} \\lambda_{i+1} + \\left(Z + \\left(\\frac{1}{2} \\sum_{i=1}^{k} \\lambda_{i+1} \\right)^2\\right)^{1/2}\n    \\leq  \\sum_{i=1}^{k} \\lambda_{i+1} + Z^{1/2}  \\leq  \\sum_{i=1}^{k+1} \\lambda_{i} + u_0 ,\n\\end{align*}\nwhere the last line follows from the fact that $Z^{1/2} = \\left(u_0^2 + \\lambda_1  \\cdot u_0 \\right)^{1/2} \\leq u_0 + \\lambda_1.$\n\\\\\nPutting the bound for $u_k$ into the key bound \\eqref{eq:thm_key1} we have\n\\allowdisplaybreaks\n\\begin{align*}\n    &\\|x_K - x^* \\|^2 - \\|x_0 - x^* \\|^2 + \\sum_{k\\in S, k \\leq K} 2\\alpha_k (F(x_k) - F(x^*) )\\nonumber\\\\\n    &\\leq \\sum_{k=1}^K \\lambda_k \\cdot \\| x_{k-1} -x^* \\|\\leq \\sum_{k=1}^K \\lambda_k \\cdot \\left( \\sum_{i=1}^{k} \\lambda_i + \\|x_0 - x^* \\|\\right)\\nonumber\\\\\n    &= \\sum_{k=1}^K \\lambda_k \\cdot \\left( \\sum_{i=1}^{k} \\lambda_i \\right) + \\|x_0 - x^* \\| \\cdot \\sum_{k=1}^K \\lambda_k\\nonumber\\\\\n    &\\leq  \\sum_{k=1}^K \\lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\lambda_i \\right)+ \\sum_{k=1}^K \\lambda_k^2 + \\|x_0 - x^* \\|^2 + \\sum_{k=1}^K \\lambda_k^2 \\nonumber\\\\\n    &= \\|x_0 - x^* \\|^2 + \\sum_{k=1}^K \\lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\lambda_i \\right)+ 2\\sum_{k=1}^K \\lambda_k^2,\n\\end{align*}\nand the statement of Lemma \\ref{lem:after_K1} follows. \n~\\qed\n\\end{proof}","sha256_utf8":"f6a9bb609db4fa7876347a3fe0a85c096c71e62eb0e10892ae84beb48b88318c"},{"start":59385,"end":59794,"start_line":680,"end_line":688,"raw_tex":"\\begin{theorem}[Upper bound of the number of good steps]\\label{th:bound_expectation_N_TS1} Under Assumption \\ref{assumpt:convex_smooth}, \\ref{assumpt:bounded_below}  ,\nand  Assumption \\ref{assumpt:ista_noise}, we have \n \\begin{align*}\n\\EE [N_G] \n\\leq \\frac{1}{2\\stepsizebar\\epsilon} \\left [2\\|x_0 - x^* \\|^2 + \\frac{4(\\beta + 2)^2}{\\beta^2} + \\frac{8(\\beta+2)}{\\beta+1} \n \\right]. \n\\end{align*}\n\n\\end{theorem}","sha256_utf8":"03847b98b3a1b611da9ca4338f3813a6f6296e80a2a614514f2a81e7a9a16dfd"},{"start":59795,"end":66453,"start_line":689,"end_line":777,"raw_tex":"\\begin{proof}\nWe recall the definition of the noise term $\\lambda_k =  \\|2\\alpha_{k} (\\nabla f(y_{k} )- g_{k} )\\| $. Let $\\Lambda_k =  \\|2\\Alpha_{k} (\\nabla f(Y_{k} )- G_{k} )\\| $, thus $\\Lambda_k$ is a random variable with realizations  $\\lambda_k$. \nSince the statement of Lemma \\ref{lem:after_K1} holds for every realizations of Algorithm \\ref{alg:ista_bktr_line_search}, for $K = N_\\epsilon -1$ we have:  \n\\begin{align*}\n\\sum_{k\\in S, k \\leq N_\\epsilon -1} 2\\Alpha_k (F(X_k) - F(x^*) )\n    &\\leq 2\\|x_0 - x^* \\|^2 + \\sum_{k=1}^{N_\\epsilon -1} \\Lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\Lambda_i \\right)+ 2\\sum_{k=1}^{N_\\epsilon -1} \\Lambda_k^2,\n\\end{align*}\nwhere $S$ is the set of successful iterations.\nRecall that the stopping time $N_\\epsilon = \\min\\{k = 1,2, \\ldots\\,:  F(X_k) - F^*  \\leq \\epsilon\\}$. Hence for every $k \\leq N_\\epsilon-1$, we have that $ F(X_k) - F^*> \\epsilon$ and\n\\begin{align*}\n    \\epsilon \\cdot \\sum_{k\\in S, k \\leq N_\\epsilon -1} 2\\Alpha_k  &<  \\sum_{k\\in S, k \\leq N_\\epsilon -1} 2\\Alpha_k (F(X_k) - F(x^*) )\\nonumber\\\\\n    &\\leq 2\\|x_0 - x^* \\|^2 + \\sum_{k=1}^{N_\\epsilon} \\Lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\Lambda_i \\right)+ 2\\sum_{k=1}^{N_\\epsilon} \\Lambda_k^2.\n\\end{align*}\nLet $\\mathcal{N}_G$ be the set of \\textit{good} iterations  i.e. true and successful iterations $k$ with $\\alpha_k\\geq \\stepsizebar, k \\leq N_\\epsilon -1 $, then $N_G$ is the cardinality of that set. We have \n\\begin{align*} \n\\sum_{k\\in S, k \\leq N_\\epsilon -1} 2\\Alpha_k \\geq\\sum_{k\\in \\mathcal{N}_G, k \\le N_\\epsilon -1 }  2\\Alpha_k \\geq N_G \\cdot 2\\stepsizebar = 2 N_G \\stepsizebar.\n\\end{align*}\nTherefore \n\\begin{align*}\n    2 N_G \\stepsizebar \\cdot \\epsilon  \n    &\\leq 2\\|x_0 - x^* \\|^2 + \\sum_{k=1}^{N_\\epsilon} \\Lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\Lambda_i \\right)+ 2\\sum_{k=1}^{N_\\epsilon} \\Lambda_k^2,\n\\end{align*}\nand \n\\begin{align*}\n    N_G   \n    &\\leq \\frac{2\\|x_0 - x^* \\|^2 + \\sum_{k=1}^{N_\\epsilon} \\Lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\Lambda_i \\right)+ 2\\sum_{k=1}^{N_\\epsilon} \\Lambda_k^2}{2 \\stepsizebar \\epsilon}.\n\\end{align*}\nTaking expectation we have %\\trang{notations here}\n\\begin{align}\\label{eq:bound1}\n\\EE [N_G ] \n &\\leq \\frac{1}{2\\stepsizebar\\epsilon} \\left [2\\|x_0 - x^* \\|^2 \n+ \\EE \\left[ \\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} \\cdot \\left(\\sum_{i=1}^{k-1} \\Lambda_{i} \\right)  \\right] + 2\\EE \\left[\\sum_{k=1}^{N_\\epsilon}  \\Lambda_{k} ^2  \\right] \n \\right]  .\n\\end{align}\nIn order to bound \\eqref{eq:bound1}, we derive the expected values of $\\Lambda_k^2$, $\\Lambda_k$ and $\\Lambda_k \\Lambda_i$ for $1\\leq i, k \\leq N_\\epsilon$\nusing Assumption \\ref{assumpt:ista_noise}. From \\eqref{eq:ista_noise} we have: \n\\begin{align}\\label{eq:mu_k_squared1}\n    \\EE \\left[\\Lambda_{k}^2| \\pastone\\right] = \\EE \\left[4\\alpha_{k}^2 \\| (\\nabla f(y_{k} )- G_{k} )\\|^2 | \\pastone\\right] \\leq \\frac{4}{k^{2+ \\beta}},\n\\end{align}\nwhich leads to \n\\begin{align*}\n    \\left(\\EE \\left[\\Lambda_{k}| \\pastone\\right]\\right)^2 \\leq \\EE \\left[\\Lambda_{k}^2| \\pastone\\right] \\leq \\frac{4}{k^{2+ \\beta}}.\n\\end{align*}\nTherefore we have the bound for $\\EE \\left[\\Lambda_{k}| \\pastone\\right]$: \n\\begin{align}\\label{eq:mu_k1}\n    \\EE \\left[\\Lambda_{k}| \\pastone\\right] \\leq \\frac{2}{k^{1+ 0.5\\beta}}.\n\\end{align} \nTaking the total expectation of \\eqref{eq:mu_k_squared1} and \\eqref{eq:mu_k1} we have \n\\begin{align*}\n    \\EE \\left[\\Lambda_{k}^2\\right] \\leq \\frac{4}{k^{2+ \\beta}} \\text{ and } \\EE \\left[\\Lambda_{k}\\right] \\leq \\frac{2}{k^{1+ 0.5\\beta}}.\n\\end{align*} \nNow we consider the quantity $\\EE \\left[\\Lambda_{k}\\Lambda_{i}| \\pastone\\right] $ where $i< k $. \nFrom the definition of  the filtration $\\pastone$ and $\\Lambda_{i}$,  {we have that} $\\Lambda_i \\in \\pastone$ for all $i < k$. Therefore by the rules of conditional probability\n\\begin{align*}\n    \\EE \\left[\\Lambda_{k}\\Lambda_{i}| \\pastone\\right] =  \\EE \\left[\\Lambda_{k}| \\pastone\\right] \\Lambda_i .\n\\end{align*}\nHence we can bound the expect value of $\\Lambda_{k}\\Lambda_{i}$ as follows: \n\\begin{align*}\n    \\EE \\left[\\Lambda_{k}\\Lambda_{i}  \\right] \n    &=\\EE \\left[\\EE \\left[\\Lambda_{k}\\Lambda_{i}| \\pastone \\right] \\right]  =\\EE \\left[\\EE \\left[\\Lambda_{k}| \\pastone\\right] \\Lambda_i  \\right] \n    \\nonumber\\\\ \\nonumber\n    &  {\\leq} \\EE \\left[\\EE \\left[\\frac{2}{k^{1+ 0.5\\beta}} \\Lambda_i  \\right]  \\right]   = \\frac{2}{k^{1+ 0.5\\beta}} \\EE \\left[\\EE \\left[ \\Lambda_i | \\pastonei \\right]  \\right]\n    \\nonumber\\\\ \\nonumber\n    &  {\\leq} \\frac{2}{k^{1+ 0.5\\beta}} \\frac{2}{i^{1+ 0.5\\beta}} .\n\\end{align*}\nSince $N_\\epsilon$ is a hitting time and the random variables $\\Lambda_k^2$, $\\Lambda_{k}\\Lambda_{i}$ have finite means and are summable (over $\\{k = 1,2, \\ldots\\}$ and over $\\{i, k = 1,2,\\ldots\\}$ respectively), from the general Wald's identity  {\\cite{Wald}} we have: \n\\begin{align*}\n\\EE \\left[\\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} ^2  \\right] &= \\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\EE \\left[  \\Lambda_{k} ^2  \\right]  \\right)\\nonumber\\\\\n\\EE \\left[ \\sum_{k=1}^{N_\\epsilon}\\Lambda_{k} \\cdot \\left(\\sum_{i=1}^{k-1} \\Lambda_{i} \\right)  \\right] &= \\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\sum_{i=1}^{k-1}\\EE \\left[  \\Lambda_{k} \\Lambda_{i}  \\right] \\right) .\n\\end{align*}\nApplying the previous total expected value bounds in these inequality we have: \n \\begin{align*}\n\\EE \\left[\\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} ^2  \\right] & = \\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\EE \\left[  \\Lambda_{k} ^2  \\right]  \\right)\n\\leq 4\\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\frac{1}{k^{2+ \\beta}}\\right) \\leq  \\frac{4(\\beta +2)}{\\beta+1}\\nonumber\\\\\n \\EE \\left[ \\sum_{k=1}^{N_\\epsilon}\\Lambda_{k} \\left(\\sum_{i=1}^{k-1} \\Lambda_{i} \\right)  \\right] &= \\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\sum_{i=1}^{k-1}\\EE \\left[  \\Lambda_{k} \\Lambda_{i}  \\right] \\right) \\leq 4\\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\frac{1}{k^{1+ 0.5\\beta}} \\sum_{i=1}^{k-1} \\frac{1}{i^{1+ 0.5\\beta}}\\right) \\leq \\frac{4(\\beta + 2)}{\\beta},\n\\end{align*}\nwhere the last computations follows from \n\\begin{align*}\n    &\\sum_{k=1}^{N_\\epsilon}\\frac{1}{k^{2+ \\beta}} < 1 + \\sum_{k=2}^{\\infty} \\frac{1}{k^{2+ \\beta}} < 1+  \\int_{1}^{\\infty} \\frac{1}{x^{2+ \\beta}} dx =  \\frac{\\beta +2}{\\beta+1},\\nonumber\\\\\n    &\\sum_{k=1}^{N_\\epsilon}\\frac{1}{k^{1+ 0.5\\beta}} < 1 + \\sum_{k=2}^{\\infty} \\frac{1}{k^{1+ 0.5\\beta}} < 1+  \\int_{1}^{\\infty} \\frac{1}{x^{1+ 0.5\\beta}} dx = \\frac{\\beta + 2}{\\beta}.\n\\end{align*}\nSubstituting these bounds into the previous inequality \\eqref{eq:bound1}, we have\n\\begin{align*}\n\\EE [N_G ] \n &\\leq \\frac{1}{2\\stepsizebar\\epsilon} \\left [2\\|x_0 - x^* \\|^2 + \\frac{4(\\beta + 2)^2}{\\beta^2} + \\frac{8(\\beta+2)}{\\beta+1} \n \\right]. \n\\end{align*}\nwhich is the statement of Theorem \\ref{th:bound_expectation_N_TS1}.\n~\\qed \\end{proof}","sha256_utf8":"9a922d39e216f693abcfe6e24eb28ddaea43e52133d4cb693d8d242f04267d6b"},{"start":80756,"end":81271,"start_line":909,"end_line":912,"raw_tex":"We note that the steps 4 and 5 of Algorithm \\ref{alg:fista_bktr_line_search} are exactly as Algorithm \\ref{alg:ista_bktr_line_search}.\nSince the dynamics of the step-size parameter is based on the reference points $y_k$, $\\tilde{p}_{\\alpha_k} (y_k)$ in Step 4 and the sufficient decrease condition in Step 5, \nthe results in Sections \\ref{sec:stoc_proc} and \\ref{subsec:successful}  hold  for Algorithm \\ref{alg:fista_bktr_line_search} under the same assumptions as for \nAlgorithm \\ref{alg:ista_bktr_line_search}. \n","sha256_utf8":"7443ff318e9afabe7c78f3b8888baf07963d511a0d7ef023b69d788b287192ca"},{"start":81483,"end":82179,"start_line":916,"end_line":927,"raw_tex":"\\begin{lemma}\\label{lem:fista_linesearch}\n We denote $\\alpha^{succ}_k$ be the step-size parameter  at the successful iteration up to iteration $k$: \n\\begin{align*}\n    \\alpha^{succ}_k = \\alpha_{k'}, \\text{ where } k' = \\max \\{k'' \\text{ successful }, k'' \\leq k\\} .\n\\end{align*}\nWe also adopt the convention that $\\alpha^{succ}_k = \\alpha_0 = \\gamma \\alpha_1$ if there is no successful iteration up to iteration $k$. Then we have the following properties:\n\n1. $\\alpha^{succ}_{k-1} t_{k-1}^2\\geq \\alpha_k t_k(t_{k}-1)$ for every successful iteration $k$.\n\n2. $\\alpha^{succ}_{K} t_{K}^2 \\geq \\left(\\sum_{k \\in S, k \\le K } \\sqrt{\\alpha_k} /2\\right)^2$ for any number of iterations $K$.\n\n\\end{lemma}","sha256_utf8":"4ccaa3e8ecb5a51011daa5f83380a886f5416b22da360ff06dbbbad0cc44dbff"},{"start":82182,"end":87089,"start_line":930,"end_line":1002,"raw_tex":"\\begin{proof}\n1. From the update of the algorithm and the fact that $k$ is successful: \n\\begin{align*}\n    t_k = t_k^{next} &=  \\frac{1 + \\sqrt{1+4\\theta_{k-1} t_{k-1}^2}}{2},\n\\end{align*}\nwe have $ \\theta_{k-1} t_{k-1}^2=  t_k(t_{k}-1)$.\nThe statement of the lemma $\\alpha^{succ}_{k-1} t_{k-1}^2\\geq \\alpha_k t_k(t_{k}-1)$ is equivalent to  \n\\begin{align*}\n\\alpha^{succ}_{k-1} \\geq \\alpha_k \\theta_{k-1}.\n\\end{align*}\nLet $k'$ be the previous successful iteration prior to $k$ ($k' < k$), i.e. $\\alpha^{succ}_{k-1} = \\alpha_{k'}$ and $\\alpha^{succ}_{k-1} = \\alpha_{0}$ if there is no successful iteration prior to $k$. By the definition of $k'$ we have \n$\\alpha_{k'+1} = \\tfrac{\\alpha_{k'}}{\\gamma},$ and $\\theta_{k'} = \\gamma $.\nSince the iterations from $k' + 1$ to $k-1$ are unsuccessful (if $k' <k-1$): \n\\begin{align*}\n    \\alpha_{k} &= \\alpha_{k'+1} \\cdot  \\gamma^{k-k'-1}\\nonumber\\\\\n    \\theta_{k-1} &= \\theta_{k'} \\gamma^{k'+1 -k}.\n\\end{align*}\nTherefore \n\\begin{align*}\n    \\alpha_{k} &= \\alpha_{k'+1} \\cdot  \\gamma^{k-k'-1}= \\alpha_{k'} \\cdot  \\gamma^{k-k'-2} \\nonumber\\\\\n    \\theta_{k-1} &= \\theta_{k'} \\gamma^{k'+1 -k} = \\gamma^{k'+2 -k},\n\\end{align*}\nand it proves the statement: \n    \\begin{align}\\label{eq:theta}\n    \\alpha_k \\theta_{k-1} = \\alpha_{k'} \\cdot  \\gamma^{k-k'-2} \\cdot \\gamma^{k'+2 -k} = \\alpha_{k'}. \n    \\end{align}\n    \n\n\n% \\katya{This proof is critical. Is it actually true for all iterations, including unsuccessful ones?}\n\n% \\trang{Actually we need $ \\alpha^{succ}_{K} t_{K}^2 \\geq \\left(\\sum_{k \\in S, k \\le K } \\sqrt{\\alpha_k} /2\\right)^2$ for every $K \\leq  N_\\epsilon-1$. (I forgot the $^{succ}$ before). Maybe the proof only need $K = N_\\epsilon-1$ but this lemma needs to be true for every iteration. We can prove this for all successful iterations, then for unsuccessful one, it still holds since it holds for the prior successful one i.e. no term changes (algo do not update $t_k$). But looking at this, technically $t_k$ do not has at least order $k$ directly, it has the order of the number of good iterations before $k$}\n\n\n2. First, we prove the statement for all iterations $K$ where $K$ is successful. We prove this by induction. \n\\begin{itemize}\n\\item Let $k_1 \\geq 1$ be the first successful iteration of the algorithm. By definition we have \n$\\alpha^{succ}_{k_1} = \\alpha_{k_1}, \\text{ since } k_1 = \\max \\{k'' \\text{ successful }, k'' \\leq k_1\\} $.  \nSince $k_1$ is successful:\n\\begin{align*}\n    t_{k_1} = t_{k_1}^{next} = \\frac{1 + \\sqrt{1+4\\theta_{(k_1-1)} t_{(k_1-1)}^2}}{2}= \\frac{1 + \\sqrt{1+4\\theta_{(k_1-1)} \\cdot 0}}{2} = 1.\n\\end{align*}\nTherefore the statement holds for $k_1$: \n\\begin{align*}\n    \\alpha^{succ}_{k_1} t_{k_1}^2 =  \\alpha_{k_1} \\cdot 1 \\geq \\left( \\sqrt{\\alpha_{k_1}} /2\\right)^2 = \\left(\\sum_{k \\in S, k \\le k_1 } \\sqrt{\\alpha_k} /2\\right)^2.\n\\end{align*}\n\\item Now for an iteration $K$ where $K$ is successful, we assume the statement holds for the successful iteration $k'$ prior to $K$ ($k'$ is the largest index of a successful iteration satisfying $k' < K$). We have \n$ \\alpha^{succ}_{k'} t_{k'}^2 \\geq \\left(\\sum_{k \\in S, k \\le k' } \\sqrt{\\alpha_k} /2\\right)^2$ and  \n\\begin{align*}\n    \\sqrt{\\alpha^{succ}_{k'}} t_{k'} \\geq \\sum_{k \\in S, k \\le k' } \\sqrt{\\alpha_k} /2.\n\\end{align*}\n\nSince both $K$ and $k'$ are successful, $ \\alpha^{succ}_{K} =  \\alpha_{K} $ and $ \\alpha^{succ}_{k'} =  \\alpha_{k'} $.\nUsing the fact that $K$ is successful: \n\\begin{align*}\n    t_{K} &= t_{K}^{next} = \\frac{1 + \\sqrt{1+4\\theta_{(K-1)} t_{(K-1)}^2}}{2} \\geq \\frac{1}{2} +  \\sqrt{\\theta_{(K-1)}}  t_{(K-1)}\\\\\n    &\\geq \\frac{1}{2} +  \\sqrt{\\theta_{(K-1)} } \\cdot t_{k'} \\geq \\frac{1}{2} +  \\sqrt{\\frac{\\alpha_{k'}}{\\alpha_{K}} } \\cdot t_{k'}.\n\\end{align*}\nwhere the last inequality follows from previous argument \\eqref{eq:theta}.\nHence we have \n\\begin{align*}\n    \\sqrt{\\alpha^{succ}_{K}} t_{K} = \\sqrt{\\alpha_{K}} t_{K} \\geq \\sqrt{\\alpha_{k'}} t_{k'} + \\sqrt{\\alpha_K} /2 = \\sqrt{\\alpha^{succ}_{k'}} t_{k'} + \\sqrt{\\alpha_K} /2.\n\\end{align*}\nTherefore \n\\begin{align*}\n    \\sqrt{\\alpha^{succ}_{K}} t_{K} \\geq \\sum_{k \\in S, k \\le k' } \\sqrt{\\alpha_k} /2 + \\sqrt{\\alpha_K} /2 = \\sum_{k \\in S, k \\le K } \\sqrt{\\alpha_k} /2,\n\\end{align*}\nand the statement holds for iteration $K$:\n$ \\alpha^{succ}_{K} t_{K}^2 \\geq \\left(\\sum_{k \\in S, k \\le K } \\sqrt{\\alpha_k} /2\\right)^2$.\n\\item By induction, the statement holds for all the successful iterations. For an arbitrary unsuccessful iteration $K$, the statement holds for the successful iteration $k'$ prior to $K$ and neither right hand side  {nor} left hand side term changes at  unsuccessful iteration. The statement is also true in the case when there is no successful iteration prior to $K$ as both sides equal to 0 ($t_0 = 0$). Therefore it holds for every iteration. \n\\end{itemize}\n% \\katya{what is there is no prior successful iteration? Show that is holds for $K=1$}\n~\\qed \\end{proof}","sha256_utf8":"8c6080ca4704e7f89f0afa40f10560ad689188f4021bb057073aa3d3a6d1a549"},{"start":88484,"end":88764,"start_line":1017,"end_line":1021,"raw_tex":"\\begin{lemma}[$m_{k}$ on unsuccsessful steps]\\label{lem:unsuccessful_step_bound}\nLet $m_k$ be defined as in \\eqref{eq:define_m}. \nIf iteration $k$ of Algorithm \\ref{alg:fista_bktr_line_search}  is unsuccessful, then we have \n\\begin{align*}m_{k} = m_{k-1}. \\end{align*}\n\\end{lemma}","sha256_utf8":"ccd4257d2d411167887a2059ed558360e1425ca9084107deb31a9a74d8c8b10e"},{"start":89388,"end":90109,"start_line":1029,"end_line":1043,"raw_tex":"\\begin{lemma}[$m_k$ on  successful step]\\label{lem:successful_step_bound}\nUnder Assumption \\ref{assumpt:convex_smooth}, if $k$ is a successful iteration of Algorithm \\ref{alg:fista_bktr_line_search}, then we have:\n %\\katya{I think here and in ISTA analysis it is best to use $b_k$ instead of $b_k$ for realizations and switch to $b_k$ when taking expectation. }\n\\begin{align*}\n&2\\alpha^{succ}_{k}t_k^2 v_k -2\\alpha^{succ}_{k-1} t_{k-1}^2v_{k-1} \\leq \\norm{u_{k-1}}^2 -  \\norm{u_k }^2 + \\lambda_k \\cdot \\| u_{k-1}\\|,\n\\end{align*}\nwhere \n\\begin{align*}\n\\lambda_k =  \\|2\\alpha_kt_k (\\nabla f(y_k )- g_k )\\|.\n\\end{align*}\nEquivalently, \n\\begin{align*}\nm_{k} -m_{k-1} \\leq \\lambda_k \\cdot \\|u_{k-1}\\|.\n\\end{align*}\n\\end{lemma}","sha256_utf8":"95f2e3284a26485c85591785bf5f15d71c5daed0912ca06fa80d5bbdf1687f27"},{"start":90111,"end":97277,"start_line":1045,"end_line":1174,"raw_tex":"\\begin{proof}\nSince $k$ is a successful iteration, we have $F(\\tilde{p}_{\\alpha_k} (y_k)) \\leq \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (y_k), y_k)$. \nAlso $x_k = \\tilde{p}_{\\alpha_k} (y_k) $. %; t_k = t_k^{next}$.\nHence \n\\begin{align*}\n    F(x_k) = f(x_k) + h(x_k) \\leq f(y_k) +(x_k -y_k)^\\top g_k + \\frac{1}{2\\alpha_k}\\|x_k-y_k\\|^2 + h(x_k) .\n\\end{align*}\nBy the  convexity of $f$ at $x$ and $y_k$ and the convexity of $h$ at $x$ and $x_k$:\n\\begin{align*}\nf(x) \\geq f(y_k) + \\nabla f(y_k )^\\top (x-y_k),\n\\nonumber\\\\\nh(x) \\geq h(x_k) + \\gamma(y_k)^\\top (x-x_k),\n\\end{align*}\nwhere the second bound holds for every subgradient $\\gamma(y_k)$ of $h$ at $x_k$. \nWe have \n\\begin{align*}\nF(x) &= f(x) + h(x)\\geq f(y_k)+ h(x_k) + \\nabla f(y_k )^\\top (x-y_k)  + \\gamma(y_k)^\\top (x-x_k)\n\\nonumber\\\\\n&= f(x) + h(x)\\geq f(y_k)+ h(x_k) + [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x-y_k)  + \\gamma(y_k)^\\top ( y_k -x_k) \n\\end{align*}\nCombine this with the previous bound of $F(x_k)$ we have \n\\begin{align*}\n&F(x) - F(x_k) \\nonumber\\\\&\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x-y_k)  + \\gamma(y_k)^\\top ( y_k -x_k) - (x_k -y_k)^\\top g_k - \\frac{1}{2\\alpha_k}\\|x_k-y_k\\|^2\\nonumber\\\\\n&=  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x-y_k)  + (\\gamma(y_k)+g_k)^\\top ( y_k -x_k)  - \\frac{1}{2\\alpha_k}\\|x_k-y_k\\|^2.\n\\end{align*}\nUsing the fact that $x_k = \\tilde{p}_{\\alpha_k} (y_k) =\\arg\\min_y \\tilde{Q}_{\\alpha_k}(y, y_k)$, we get that $ y_k - x_k= \\alpha_k[\\gamma(y_k)+g_k]$ for some subgradient $\\gamma(y_k)$ of $h$ at $x_k$. Therefore\n\\begin{align*}\nF(x) - F(x_k)\n&\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x-y_k)  + \\alpha_k\\|\\gamma(y_k)+g_k) \\|^2 - \\frac{1}{2} \\alpha_k \\|\\gamma(y_k)+g_k)\\|^2\\nonumber\\\\\n&=  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x-y_k)  + \\frac{\\alpha_k}{2}\\|\\gamma(y_k)+g_k) \\|^2 .\n\\end{align*}\nApply this equation for $x = x_{k-1}$ and $x=x^*$ we have \n\\begin{align*}\nF(x_{k-1}) - F(x_k) &\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x_{k-1}-y_k)  + \\frac{\\alpha_k}{2}\\|\\gamma(y_k)+g_k) \\|^2 ,\n\\nonumber\\\\\nF(x^*) - F(x_k) &\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x^*-y_k)   + \\frac{\\alpha_k}{2}\\|\\gamma(y_k)+g_k) \\|^2 .\n\\end{align*}\nLet $v_k =  F(x_k) - F(x^*) $ then \n\\begin{align*}\nv_{k-1} - v_k &\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x_{k-1}-y_k)  + \\frac{\\alpha_k}{2}\\|\\gamma(y_k)+g_k) \\|^2  ,\n\\nonumber\\\\\n-v_k &\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x^*-y_k)   + \\frac{\\alpha_k}{2}\\|\\gamma(y_k)+g_k) \\|^2  .\n\\end{align*}\nMultiplying the first equality by $t_{k}-1$ and adding to the second equality, we get\n\\begin{align*}\n(t_{k}-1)v_{k-1} - t_k v_k &\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k )   + \\frac{\\alpha_k}{2} t_k \\|\\gamma(y_k)+g_k) \\|^2 .\n\\end{align*}\nMultiplying the previous inequality by $t_k$ we have\n \\begin{align*}\n &t_k(t_{k}-1)v_{k-1} - t_k^2 v_k \\nonumber\\\\\n &\\geq t_k [\\nabla f(y_k )+ \\gamma(y_k)]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k )   + \\frac{\\alpha_k}{2}t_k^2\\|\\gamma(y_k)+g_k) \\|^2 \\nonumber\\\\\n&=  t_k [\\nabla f(y_k )- g_k +g_k + \\gamma(y_k)]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k )  + \\frac{\\alpha_k}{2}t_k^2\\|\\gamma(y_k)+g_k) \\|^2\n\\nonumber\\\\\n&=  t_k [g_k + \\gamma(y_k)]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k )  + \\frac{\\alpha_k}{2}t_k^2\\|\\gamma(y_k)+g_k) \\|^2\n \\nonumber\\\\&\n+ t_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) \n\\nonumber\\\\\n&=  t_k \\frac{1}{\\alpha_k}(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k )  + \\frac{\\alpha_k}{2}t_k^2\\|\\gamma(y_k)+g_k) \\|^2\n \\nonumber\\\\&\n+ t_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) \n\\nonumber\\\\\n&=  t_k \n\\frac{1}{\\alpha_k}(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   + t_k \n\\frac{1}{\\alpha_k}(y_k - x_k)^\\top t_k(x_k -y_k)  \\nonumber\\\\&\n+ \\frac{1}{2\\alpha_k}t_k^2\\|x_k -y_k \\|^2\n+ t_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) \n\\nonumber\\\\\n&=  t_k \n\\frac{1}{\\alpha_k}(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n\\frac{1}{2\\alpha_k}\\norm{t_k (x_k -y_k)}^2 \\nonumber\\\\&\n+ t_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) .\n\\end{align*}\nMultiplying the final statement by $2\\alpha_k $ we get \n \\begin{align}\\label{eq_bound_mu_t}\n&~2\\alpha_k t_k(t_{k}-1)v_{k-1} - 2\\alpha_kt_k^2 v_k \\nonumber\\\\\n&\\geq 2t_k \n(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n\\norm{t_k (x_k -y_k)}^2  \\nonumber\\\\\n&  + 2\\alpha_kt_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) .\n\\end{align}\nNow we consider the following term: $2t_k \n(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n\\norm{t_k (x_k -y_k)}^2$.\n% \\begin{align*}\n% 2t_k \n% (y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n% \\norm{t_k (x_k -y_k)}^2.\n% \\end{align*}\nLet $a  = t_k \n(y_k - x_k), b = (t_{k}-1)x_{k-1} + x^* - t_k x_k$ we have \n\\begin{align}\\label{eq:03}\n    & \\quad \\ 2t_k \n(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n\\norm{t_k (x_k -y_k)}^2\\nonumber\\\\ &=  2ab - a^2 \\nonumber\\\\\n&= b^2 -b^2 +2ab - a^2= b^2 -(a-b)^2\\nonumber\\\\\n&=\\norm{(t_{k}-1)x_{k-1} + x^* - t_k x_k }^2 -\\norm{(t_{k}-1)x_{k-1} + x^* - t_k x_k  -t_k (y_k - x_k)}^2\\nonumber\\\\\n&=\\norm{(t_{k}-1)x_{k-1} + x^* - t_k x_k }^2 -\\norm{(t_{k}-1)x_{k-1} + x^* - t_k y_k  }^2.\n\\end{align}\nSince $k$ is a successful iteration, we have that $x^{prev}_{k} = x_{k-1}$. From the definition of $u_k$\n\\begin{align*}u_k = (t_{k}-1)x^{prev}_{k} + x^* - t_k x_k, \\end{align*} \nwe have that \n\\begin{align*}u_k = (t_{k}-1)x_{k-1} + x^* - t_k x_k. \\end{align*} \nNow from the definition of FISTAStep \\eqref{eq:fista_step}\n\\begin{align*}y_k = x_{k-1} + \\frac{t_{k-1}-1}{t_{k}} ( x_{k-1} - x^{prev}_{k-1} ),\\end{align*}\nwe have \n\\begin{align*}t_k y_k = t_kx_{k-1} + (t_{k-1}-1) ( x_{k-1} - x^{prev}_{k-1} ).\\end{align*}\nCombining these equations with \\eqref{eq:03} we get: \n\\begin{align*}\n& \\quad \\ 2t_k \n(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n\\norm{t_k (x_k -y_k)}^2\\nonumber\\\\\n&=\\norm{(t_{k}-1)x_{k-1} + x^* - t_k x_k }^2 -\\norm{(t_{k}-1)x_{k-1} + x^* - t_k y_k  }^2\\nonumber\\\\\n&=\\norm{u_k }^2 -\\norm{(t_{k}-1)x_{k-1} + x^* - t_kx_{k-1} - (t_{k-1}-1) ( x_{k-1} -  x^{prev}_{k-1} )  }^2\\nonumber\\\\\n&=\\norm{u_k }^2 -\\norm{-t_{k-1} x_{k-1} + x^*  + (t_{k-1}-1)  x^{prev}_{k-1}  }^2\\nonumber\\\\\n&=\\norm{u_k }^2 -\\norm{-u_{k-1}}^2.\n\\end{align*}\nSubstituting back to previous equality \\eqref{eq_bound_mu_t} we get \n \\begin{align*}\n&2\\alpha_k t_k(t_{k}-1)v_{k-1} - 2\\alpha_kt_k^2 v_k \\nonumber\\\\\n&\\geq \\norm{u_k }^2 -\\norm{u_{k-1}}^2  + 2\\alpha_kt_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) .\n\\end{align*}\nTherefore  \n  \\begin{align*}\n&2\\alpha_kt_k^2 v_k -2\\alpha_k t_k(t_{k}-1)v_{k-1}\\nonumber\\\\ % &\\leq \\norm{u_{k-1}}^2 -  \\norm{u_k }^2 -2\\alpha_kt_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) \\nonumber\\\\\n&\\leq \\norm{u_{k-1}}^2 -  \\norm{u_k }^2 + \\|2\\alpha_kt_k (\\nabla f(y_k )- g_k )\\| \\cdot \\| u_{k-1}\\|.\n\\end{align*}\nFrom Lemma \\ref{lem:fista_linesearch} we have $\\alpha^{succ}_{k-1} t_{k-1}^2\\geq \\alpha_k t_k(t_{k}-1)$\nfor every successful iteration $k$. Also since $k$ is successful, we have  $\\alpha_k = \\alpha^{succ}_{k}$.\nTherefore we have the final bound of Lemma \\ref{lem:successful_step_bound}.\n~\\qed \\end{proof}","sha256_utf8":"ae7e744e86768b9d54259b1226051dc5da1761d3b601e023a5ff81aeb673b83c"},{"start":97927,"end":98293,"start_line":1181,"end_line":1186,"raw_tex":"\\begin{lemma}\\label{lem:after_K}\nGiven any integer $K$, after $K$ iterations of the algorithm, the following bound holds: \n\\begin{align*}\n    2\\alpha^{succ}_{K} t_{K}^2v_{K} \\leq 3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\lambda_{i} \\right]  + 2\\sum_{k=1}^{K} \\lambda_{k} ^2  .\n\\end{align*}\n\\end{lemma}","sha256_utf8":"ea1f6f54ffd9207ba5d86885f2e0307003e0d7375418d88191379475ef38555e"},{"start":98295,"end":102300,"start_line":1188,"end_line":1242,"raw_tex":"\\begin{proof}\nFor every $K$, from Lemma \\ref{lem:unsuccessful_step_bound}  we have\n\\begin{align*}\n    m_K - m_0 = \\sum_{k \\in S, k\\leq K} (m_{k} -m_{k-1}),\n\\end{align*}\nwhere $S$ is the set of successful iterations.\nWe apply Lemma \\ref{lem:successful_step_bound} for successful iterations and then \n\\begin{align*}\n    m_K - m_0 &= \\sum_{k \\in S, k\\leq K} (m_{k} -m_{k-1})\\leq \\sum_{k \\in S, k\\leq K} \\lambda_k \\cdot \\|u_{k-1}\\| \\leq \\sum_{k=1}^{K} \\lambda_k \\cdot \\| u_{k-1}  \\|  .\n\\end{align*}\nPlugging this and the definition of $m_k$ back to the previous inequality, we have our key bound: \n\\begin{align}\\label{eq:thm_key}\n    2\\alpha^{succ}_{K} t_{K}^2v_{K} +\\norm{u_K }^2 \n    &\\leq \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2\\right) + \\sum_{k=1}^{K} \\lambda_k \\cdot \\| u_{k-1}\\| .\n\\end{align}\nSimilar to the proof of ISTA, \nwe follow the argument in \\cite{Schmidt-Roux-Bach} with two steps. First, we use the key bound \\eqref{eq:thm_key} to bound $\\| u_{k-1}\\|$ in terms of the noise terms $\\lambda_k$. Then we use this bound on $\\| u_{k-1}\\|$ to substitute into the last sum of the key bound \\eqref{eq:thm_key} and obtain the final derivation in terms of the noise $\\lambda_k$.\nIgnoring the positive term $2\\alpha^{succ}_{K} t_{K}^2v_{K}$, we have \n\\begin{align*}\n    \\norm{u_K }^2 \n    &\\leq \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2\\right) + \\sum_{k=1}^K \\lambda_k \\cdot \\| u_{k-1}\\| \\nonumber\\\\\n    &= 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\lambda_1 \\cdot \\| u_{0}\\| + \\sum_{k=1}^{K} \\lambda_{k+1} \\cdot \\| u_{k}\\|.\n\\end{align*}\nThis is equivalent to \n\\begin{align*}\n    \\norm{u_K }^2 \n    \\leq Z + \\sum_{k=1}^{K} \\lambda_{k+1} \\| u_{k}\\|,\n\\end{align*}\nwhere $Z = 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\lambda_1 \\| u_{0}\\|$.\nUsing this inequality and applying Lemma 1 in \\cite{Schmidt-Roux-Bach} for the non-negative sequence $\\{\\|u_K\\|\\}$, we have\n\\allowdisplaybreaks\n\\begin{align*}\n    \\| u_{k}\\| &\\leq \\frac{1}{2} \\sum_{i=1}^{k} \\lambda_{i+1} + \\left( Z + \\left(\\frac{1}{2} \\sum_{i=1}^{k} \\lambda_{i+1} \\right)^2\\right)^{1/2}\n    \\leq  \\sum_{i=1}^{k} \\lambda_{i+1} +  Z^{1/2} \\nonumber\\\\\n    &\\leq  \\sum_{i=1}^{k+1} \\lambda_{i} + \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  \\right)^{1/2},\n\\end{align*}\nwhere the last line follows from the fact that\n\\begin{align*}\n    Z^{1/2} &= \\sqrt{2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\lambda_1 \\| u_{0}\\|  } \\leq  \\sqrt{2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\lambda_1^2 +\\| u_{0}\\|^2  } \\nonumber\\\\\n    &\\leq \\sqrt{2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\lambda_1^2  } \\leq  \\lambda_1 + \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  \\right)^{1/2}.\n\\end{align*}\nDropping the term $\\norm{u_K }^2 $ in \nthe inequality \\eqref{eq:thm_key} and using the bound on $\\| u_{k}\\|$ we have: \n\\allowdisplaybreaks\n\\begin{align*}\n    &2\\alpha^{succ}_{K} t_{K}^2v_{K} \\nonumber\\\\\n    &\\leq 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\| u_{k-1}\\| \\nonumber\\\\\n    &\\leq 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\left[\\sum_{i=1}^{k} \\lambda_{i} + \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  \\right)^{1/2} \\right] \\nonumber\\\\\n    &= 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\left[\\sum_{i=1}^{k} \\lambda_{i} \\right] + \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  \\right)^{1/2} \\sum_{k=1}^{K} \\lambda_{k}   \\nonumber\\\\\n    &\\leq 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\lambda_{i} \\right]  + \\sum_{k=1}^{K} \\lambda_{k} ^2+  \\frac{1}{2} \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  \\right)  + \\sum_{k=1}^{K} \\lambda_{k} ^2\\nonumber\\\\\n    &= 3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\lambda_{i} \\right]  + 2\\sum_{k=1}^{K} \\lambda_{k} ^2,  \n\\end{align*}\nwhich is our desired bound. \n~\\qed \n\\end{proof}","sha256_utf8":"d563232a7b948fd70177d1025cd2b587dd95ce803dac8dd9c98249b656234e50"},{"start":103307,"end":103761,"start_line":1258,"end_line":1267,"raw_tex":"\\begin{theorem}[Upper bound of the number of large true successful steps]\\label{th:bound_expectation_N_TS}\nUnder Assumptions \\ref{assumpt:convex_smooth}, \\ref{assumpt:bounded_below}\nand  \\ref{assumpt:fista_noise}, we have  \n \\begin{align*}\n \\EE [N_G] \\leq \\sqrt{\n \\frac{2}{\\stepsizebar\\epsilon} \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\frac{4(\\beta + 2)^2}{\\beta^2} + \\frac{8(\\beta+2)}{\\beta+1}  \n \\right] }. \n\\end{align*}\n\n\\end{theorem}","sha256_utf8":"b5ea9fba31bdc2c780b9190e826b4a2d940eff72ffec6dac5fb8682d52b678f9"},{"start":103764,"end":107207,"start_line":1270,"end_line":1316,"raw_tex":"\\begin{proof} \nSince the statement of Lemma \\ref{lem:after_K} holds for every realizations of Algorithm \\ref{alg:fista_bktr_line_search}, for $K = N_\\epsilon -1$ we have: \n\\begin{align*}\n    2\\Alpha^{succ}_{N_\\epsilon -1} T_{N_\\epsilon -1}^2 (F(X_{N_\\epsilon -1}) - F^*)\\leq 3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{N_\\epsilon -1} \\Lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\Lambda_{i} \\right]  + 2\\sum_{k=1}^{N_\\epsilon -1} \\Lambda_{k} ^2.\n\\end{align*}\nRecall that $N_\\epsilon$ be the stopping time that $N_\\epsilon = \\inf\\{k: F(X_k) - F^* \\leq \\epsilon\\}$. Hence for $K = N_\\epsilon -1$, we have that $F(X_K) - F^* > \\epsilon$ and \n\\begin{align*}\n    2\\Alpha^{succ}_{N_\\epsilon -1} T_{N_\\epsilon -1}^2 &\\leq \\frac{3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{N_\\epsilon -1} \\Lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\Lambda_{i} \\right]  + 2\\sum_{k=1}^{N_\\epsilon -1} \\Lambda_{k}^2}{F(X_{N_\\epsilon -1}) - F^*} \\nonumber\\\\\n    &\\leq \\frac{3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\Lambda_{i} \\right]  + 2\\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} ^2}{\\epsilon}.\n\\end{align*}\nLet $\\mathcal{N}_G$ be the set of \\textit{good} iterations  (i.e. true and successful iterations $k$ with $\\alpha_k\\geq \\stepsizebar $), with $ k \\leq N_\\epsilon -1$, then $N_G$ is the cardinality of that set. \nUsing the inequality $ \\alpha^{succ}_{K} t_{K}^2 \\geq \\left(\\sum_{k \\in S, k \\le K } \\sqrt{\\alpha_k} /2\\right)^2$ for $K =  N_\\epsilon-1$ from Lemma \\ref{lem:fista_linesearch} we have \n\\begin{align*} \n\\Alpha^{succ}_{N_\\epsilon -1} T_{N_\\epsilon -1}^2 \n&\\geq \\left(\\sum_{k \\in S, k \\le N_\\epsilon -1} \\sqrt{\\Alpha_k} /2\\right)^2 \\nonumber\\\\\n&\\geq \\left(\\sum_{k\\in \\mathcal{N}_G, k \\le N_\\epsilon -1 } \\sqrt{\\Alpha_k} /2\\right)^2\n\\geq \\left(N_G \\sqrt{\\stepsizebar} /2\\right)^2 \n= \\frac{1}{4} (N_G)^2 \\stepsizebar.\n\\end{align*}\nSubstituting this to the previous inequality we get \n \\begin{align*}\n(N_G)^2  &\\leq  \\frac{2}{\\stepsizebar\\epsilon} \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\Lambda_{i} \\right]  + 2\\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} ^2   \\right].\n\\end{align*}\nTaking total expectation we have: \n\\begin{align*}\n\\EE [(N_G)^2 ]   &\\leq \\frac{2}{\\stepsizebar\\epsilon} \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 \n+ \\EE \\left[ \\sum_{k=1}^{N_\\epsilon } \\Lambda_{k} \\cdot \\left(\\sum_{i=1}^{k-1} \\Lambda_{i} \\right)  \\right] + 2\\EE \\left[\\sum_{k=1}^{N_\\epsilon }  \\Lambda_{k} ^2  \\right] \n \\right].\n\\end{align*}\nFrom \\eqref{eq:fista_noise} we have: \n\\begin{align*}\n    \\EE \\left[\\Lambda_{k}^2| \\pastone\\right] = \\EE \\left[4\\alpha_{k}^2t_k^2 \\| (\\nabla f(y_{k} )- G_{k} )\\|^2 | \\pastone\\right] \\leq \\frac{4}{k^{2+ \\beta}},\n\\end{align*}\nHence using a similar argument as Theorem \\ref{th:bound_expectation_N_TS1} we get \n\\begin{align*}\n\\EE [(N_G)^2 ]\n &\\leq \\frac{2}{\\stepsizebar\\epsilon} \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  + \\frac{4(\\beta + 2)^2}{\\beta^2} + \\frac{8(\\beta+2)}{\\beta+1} \n \\right].\n\\end{align*}\nBy the expectation inequality: \n \\begin{align*}\n\\EE [N_G] \\leq \\sqrt{\n \\frac{2}{\\stepsizebar\\epsilon} \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\frac{4(\\beta + 2)^2}{\\beta^2} + \\frac{8(\\beta+2)}{\\beta+1}  \n \\right] }.\n\\end{align*}\nwe get the final results of Theorem \\ref{th:bound_expectation_N_TS}. \n~\\qed \\end{proof}","sha256_utf8":"70c830bc0d99863d287fcabe025ea083cea697b289edbfc8c5eea3aae5028835"},{"start":107209,"end":107325,"start_line":1318,"end_line":1318,"raw_tex":"Our final result is a straightforward corollary of Theorem \\ref{th:framework} and \\ref{th:bound_expectation_N_TS}. \n","sha256_utf8":"49e15b67fda00f4026598e1275163bbf7908b905b86d9f26513bb40f11f8fd9c"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"随机FISTA期望复杂度","source_units":[{"start":107325,"end":108041,"start_line":1319,"end_line":1327,"raw_tex":"\\begin{theorem}\\label{th:final_results}\nUnder Assumptions \\ref{assumpt:convex_smooth}, \\ref{assumpt:bounded_below}, \\ref{assumpt:key} and Assumption \\ref{assumpt:fista_noise} the number of iterations $N_\\epsilon$ to reach an $\\epsilon$-accurate solution satisfies the following bound: \n\\begin{align*}\n\\EE (N_\\epsilon) \\leq \\frac{2p}{(2p-1)^2}\\left ( \\left(\n \\frac{8 \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\frac{4(\\beta + 2)^2}{\\beta^2} + \\frac{8(\\beta+2)}{\\beta+1}  \n \\right]}{\\stepsizebar\\epsilon} \\right)^{\\frac{1}{2}}+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right) +1,\n\\end{align*}\nwhere  $\\stepsizebar $ is defined in Lemma \\ref{lem:good_est_good_model_successful}.\n\\end{theorem}","sha256_utf8":"53d23ec759a6fe3d517b74ce4a37f93b71ed925c769195c517aeb6df4e43d0f1"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"最终复杂度证明","source_units":[{"start":107209,"end":107325,"start_line":1318,"end_line":1318,"raw_tex":"Our final result is a straightforward corollary of Theorem \\ref{th:framework} and \\ref{th:bound_expectation_N_TS}. \n","sha256_utf8":"49e15b67fda00f4026598e1275163bbf7908b905b86d9f26513bb40f11f8fd9c"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"合并计数与好步界","source_units":[{"start":107209,"end":107325,"start_line":1318,"end_line":1318,"raw_tex":"Our final result is a straightforward corollary of Theorem \\ref{th:framework} and \\ref{th:bound_expectation_N_TS}. \n","sha256_utf8":"49e15b67fda00f4026598e1275163bbf7908b905b86d9f26513bb40f11f8fd9c"}]},{"id":"S004","semantic_type":"problem_setting","kind":"problem_setting","title":"凸复合优化问题","source_units":[{"start":4789,"end":4877,"start_line":135,"end_line":137,"raw_tex":"\\begin{align*}\n\\min_{x \\in \\R^n} F(x) = f(x) + h(x),  %\\label{eq:main_prob}\n\\end{align*}","sha256_utf8":"73e2b5ba69857b4986d4b9798ecb8c704ff257f312acb4bc534771d118339f4c"}]},{"id":"S005","semantic_type":"assumption","kind":"assumption","title":"凸性与光滑性","source_units":[{"start":4912,"end":5060,"start_line":139,"end_line":141,"raw_tex":"\\begin{assumption}\\label{assumpt:convex_smooth}\n Function  $f$ is convex and $L$-smooth, function $h$ is proper, closed and convex.\n\\end{assumption}","sha256_utf8":"73a24eebe5d017f22222d86baf04942a24fbe40d2d4b4fd8f82be10b0ed179ee"}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","title":"全局最优解存在","source_units":[{"start":5061,"end":5213,"start_line":142,"end_line":144,"raw_tex":"\\begin{assumption}\\label{assumpt:bounded_below}\nThere exists a global minimizer of $F(x)$, $x^*$,  such that $F(x^*)=F^* > -\\infty$. \n  \\end{assumption}","sha256_utf8":"072b84631f3db1b54bd43bca0501c40f4d41db9bcadc4255bee426a9a03f357d"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"近端算子","source_units":[{"start":5639,"end":5774,"start_line":150,"end_line":152,"raw_tex":"\\begin{align}\n\\text{prox}_h (y) = {\\arg \\min}_{x \\in \\R^n} \\left\\{h(x) +\\frac{1}{2}\\|x-y\\|^2\\right\\},  \\label{eq:prox_oper}\n\\end{align}","sha256_utf8":"6ab2a36d00258db509f67d11e441d0a8675b0655710b15f34bc69d28c1e3e3d7"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"精确模型与近端步","source_units":[{"start":27301,"end":27562,"start_line":254,"end_line":257,"raw_tex":"Given a  point $y\\in\\R^n$ and a step-size parameter $\\alpha$, we define the following approximation $Q_\\alpha(x, y)$ of the objective function $F$: \n\\begin{align*}\nQ_\\alpha(x, y) = f(y) +\\nabla f(y)^\\top (x -y)  + \\frac{1}{2\\alpha}\\|x-y\\|^2 + h(x).\n\\end{align*}","sha256_utf8":"36ec851677677f3b92b04ac9778a2d07cbe0448933970898ebfc5cab54647b4f"},{"start":27563,"end":27805,"start_line":258,"end_line":261,"raw_tex":"The minimizer of this function with respect to $x$, which we denote by $p_\\alpha (y)$ is given by the prox-operator: \n\\begin{align*}\n    p_\\alpha (y) := \\arg\\min_x Q_\\alpha(x, y) = \\text{prox}_{\\alpha h} (y - \\alpha \\nabla f(y)).\n\\end{align*}","sha256_utf8":"8840292fd54bc02254ed886220ef2c29d10b9c688800a71f68babfea42a4ba77"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"梯度映射","source_units":[{"start":27806,"end":28097,"start_line":262,"end_line":265,"raw_tex":"We let $D_{\\alpha}(y)$ denote the gradient mapping of $F$ \\cite{Nesterov}:\n\\begin{align}\\label{eq:gradient_mapping}\nD_{\\alpha}(y) := \\frac{1}{\\alpha}\\Big(y - \\text{prox}_{\\alpha h}(y - \\alpha\\nabla{f}(y))\\Big)= \\frac{1}{\\alpha}\\Big(y - p_\\alpha (y)\\Big), \\text{ for } \\alpha > 0.\n\\end{align}","sha256_utf8":"3ef38b8771e68ebcf300ab34172c2c87882bd834e556e98792b23f68105fdb5c"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"随机模型与近端步","source_units":[{"start":28169,"end":28385,"start_line":267,"end_line":270,"raw_tex":"We also define the following approximating function based on an estimate  $g$ instead of  $\\nabla f(y)$:\n\\begin{align*}\n\\tilde{Q}_\\alpha(x, y) = f(y) + g^\\top (x -y)  + \\frac{1}{2\\alpha}\\|x-y\\|^2 + h(x).\n\\end{align*}","sha256_utf8":"96da3a31082e7232994fef1e0ecc1307f9a0196e2a0d3a7c0ff1dab34b0c69f3"},{"start":28386,"end":28536,"start_line":271,"end_line":274,"raw_tex":"Similarly we define\n\\begin{align*}\n    \\tilde{p}_\\alpha (y) := \\arg\\min_x \\tilde{Q}_\\alpha(x, y) = \\text{prox}_{\\alpha h} (y - \\alpha g).\n\\end{align*}","sha256_utf8":"95bbaaa6b57fffc45add795135a5c9645dbde4d6c5d784d32be40c8dfe0fdfe2"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"随机一阶预言机","source_units":[{"start":29902,"end":30289,"start_line":290,"end_line":295,"raw_tex":"{\\bf Stochastic first-order oracle SFO$_{\\constG, p}(y,\\alpha)$.}\t \n{\\em Given a vector $y\\in \\R^n$ and a step-size parameter $\\alpha > 0$, the oracle computes $G(y,\\xi)$, a (random) estimate of the gradient $\\nabla f (y)$, such that\n\\begin{align}\\label{eq:first_order1}\n\\mathbb P_{\\xi} \\left\\{\\|G(y,\\xi)-\\nabla f(y)\\|\\leq \\constG \\cdot  \\|D_{\\alpha}(y)\\|   \\right\\}\n\\geq p, \n\\end{align}","sha256_utf8":"0d1b68006dffb9c80f21ff3ce855dbd9d507c5c78682758b8b0d1521b21dbab5"},{"start":30290,"end":30564,"start_line":296,"end_line":296,"raw_tex":"where $\\xi$ is a random variable whose distribution may depend on $y$ and $D_{\\alpha}(y)$ is defined in \\eqref{eq:gradient_mapping}. The values of the constants  $\\constG$ and $p$ are intrinsic to the oracle, and satisfy  {$\\constG \\leq \\frac{1}{3}$} and $p > \\frac{1}{2}$.\n","sha256_utf8":"f22c0832d9b9da324ee02ad712c70a7d3196570397002e753d3c67d20bd04381"}]},{"id":"S012","semantic_type":"algorithm","kind":"algorithm","title":"随机ISTA步长搜索","source_units":[{"start":36430,"end":37567,"start_line":328,"end_line":345,"raw_tex":"\\begin{algorithm}[ht]\n\\caption{Stochastic ISTA Step Search Algorithm}\n\\label{alg:ista_bktr_line_search}\n\\begin{algorithmic}[1]\n\\STATE {Initialization:} \nChoose an initial point $x_0\\in \\mathbb{R}^d$. Set $0<\\gamma<1$,  with $\\alpha_1>0$. Let oracle SFO$_{\\constG, p}$ be given with  {$0 \\leq \\constG \\leq \\frac{1}{3}$} and $\\frac{1}{2} < p \\leq 1 $.\n\\FOR{$k=1,2,\\cdots,$}\n    \\STATE Set  $y_k = x_{k-1}$.\n    \\STATE Compute the gradient estimate $g_k=G(y_k,\\xi_k)$ using the stochastic first-order oracle SFO$_{\\constG, p}(y_k,\\alpha_k)$.\n    Compute the reference point $\\tilde{p}_{\\alpha_k} (y_k) =\\text{prox}_{\\alpha_k h} (y_k- \\alpha_k g_k)$.\n    \\STATE Check the sufficient decrease condition $F(\\tilde{p}_{\\alpha_k} (y_k)) \\leq \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (y_k), y_k)$.\n        \\STATE \\hspace{1.25em}If unsuccessful: \n            \\STATE \\hspace{2.5em}Set $x_{k} = x_{k-1}$ and  $\\alpha_{k+1} = \\gamma\\alpha_k$.\n        \\STATE \\hspace{1.25em}If successful: \n            \\STATE \\hspace{2.5em}Set $x_k = \\tilde{p}_{\\alpha_k} (y_k)$ and $\\alpha_{k+1} = \\tfrac{\\alpha_k}{\\gamma}$.\n\\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"1b3c1b6dc2906e4eaf807995986d319b7bb1da16d36c6ecc23f3e6277f1360f1"}]},{"id":"S013","semantic_type":"notation","kind":"notation","title":"随机过程与过滤","source_units":[{"start":37656,"end":38667,"start_line":349,"end_line":352,"raw_tex":"\\textit{Filtration.}\nAlgorithm \\ref{alg:ista_bktr_line_search} generates a stochastic process $\\{Y_k, X_k, G_k, \\Stepsize{k}\\}$, where $G_k$ denotes the random gradient estimate computed by the oracle SFO$_{\\constG, p}(Y_k,\\Stepsize{k})$ at iteration $k$, $Y_k$ and $X_k$ are the random iterates and  $\\Stepsize{k}$ is the random step-size  parameter. \nWe use $g_k$, $y_k$, $x_k$ and $\\alpha_k$  to denotes realizations of these quantities for a particular sample path. We omit explicitly stating the dependence on the sample, for brevity. \nLet $\\pastone$ denote the $\\sigma$-algebra generated by the random variables $G_0, \\hdots, G_{k-1}$. We note that by the nature of the algorithm, distribution of $G_k$ is determined by $Y_k$, while  $X_k$,  $\\Stepsize{k+1}$, and, consequently,  $Y_{k+1}$ are  deterministic conditioned on $G_k$. Therefore  process $\\{G_k\\}$ captures the randomness of the stochastic process $\\{Y_k, X_k, G_k,  \\Stepsize{k}\\}$, in other words,  $\\{\\pastone\\}_{k\\ge 1}$ is a filtration. \n","sha256_utf8":"9ba92c5f9eb260c6c72f7765f5708aec950e64c6653aa3e63a6c9b0457b72b2b"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"误差与首次到达时间","source_units":[{"start":38754,"end":39427,"start_line":357,"end_line":360,"raw_tex":"\\textit{The hitting time $N_\\epsilon$ to reach $\\epsilon$ accurate solution.} \nGiven a target  accuracy $\\epsilon>0$, we aim to derive a bound on the expected number of iterations after which\nthe algorithm generates the iterate that satisfies $\\{F(x_k)-F^*\\leq \\epsilon\\}$. We denote the optimality gap $v_k = F(x_k)-F^*$ for every iteration $k$.\nLet  $N_\\epsilon=\\min\\{k=1,2, \\ldots\\ : F(X_k)-F^*\\leq \\epsilon\\}$, that is $N_\\epsilon$ is the first iteration for which the event $\\{F(x_k)-F^*\\leq \\epsilon\\}$ occurs.  $N_\\epsilon$ is a stopping time (hitting time) with respect to the filtration $\\{\\mathcal{F}_k\\}_{k\\ge 1}$   as it is defined by the realization of $X_k$.\n","sha256_utf8":"7438ec9560a93266337b155302e48926d64d5b20e6966aa5c9181c4922bcf1f3"}]},{"id":"S015","semantic_type":"definition","kind":"definition","title":"成功迭代指标","source_units":[{"start":39480,"end":39633,"start_line":364,"end_line":366,"raw_tex":"\\begin{definition}[Successful Iteration]\\label{def:success} \nLet $\\iterSucc=\\mathbbm{1}\\{{\\rm Iteration\\ } k\\ {\\rm is\\ successful } \\}$.\n\\end{definition}","sha256_utf8":"56dc8e30265fa27d299be25f5ce16fda2f3b7c6b5a10928aa520e51d63daabe6"}]},{"id":"S016","semantic_type":"definition","kind":"definition","title":"真迭代及其指标","source_units":[{"start":39635,"end":40033,"start_line":368,"end_line":376,"raw_tex":"\\begin{definition}[True Iteration]\\label{def:good_model}\nWe say iteration $k$ is {\\em true} if \n\\begin{align*}\n    \\|g_k-\\nabla f(y_k)\\|\\leq \\constG  \\|D_{\\alpha_k}(y_k)\\|, \\  {\\constG\\leq \\frac{1}{3}}.\n\\end{align*}\n\nIn other words, the first-order oracle delivers a sufficiently accurate estimate. \nWe use an indicator variable $\\iterTrue$ to indicate that iteration $k$ is true. \n\\end{definition}","sha256_utf8":"0c47ec6d7249a232c08e76878b43c51dd4184a1dcdf8254aba017be831f0df00"}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","title":"真迭代条件概率","source_units":[{"start":40305,"end":40768,"start_line":381,"end_line":387,"raw_tex":"\\begin{assumption}[Probability of True Iteration]\\label{assumpt:key}\nWe assume that the first-order oracle  SFO$_{\\constG, p}$ is implemented in Algorithm \\ref{alg:ista_bktr_line_search} with  {$\\constG \\leq \\frac{1}{3}$} and $p > \\frac{1}{2}$ for every input $(y_k,\\alpha_k)$. \nThen, by the properties of  the oracle and the filtration $\\pastone$ we have, \n\\begin{align*} \n\\Prob(\\iterTrue| \\pastone) \\ge p \\text{ for all } k \\geq 0.\n\\end{align*}\n\\end{assumption}","sha256_utf8":"327e0e4aa1ba80008001056f398c1dd543aadd73b32be197947b11a83ec54956"}]},{"id":"S018","semantic_type":"definition","kind":"definition","title":"好迭代计数","source_units":[{"start":45883,"end":46132,"start_line":467,"end_line":468,"raw_tex":"Let us define\n$N_G = \\sum_{k=1}^{N_\\epsilon-1} \\mathbbm{1}\\{ \\Alpha_k \\ge \\stepsizebar \\} \\cdot  \\mathbbm{1}\\{{\\rm Iteration\\ } k\\ {\\rm is\\ true\\ and \\ successful} \\} $ be the number of {\\em good} iterations prior to the stopping time $N_\\epsilon$.\n","sha256_utf8":"d4efb30447b04fe26bf7e5f60acbcf61c4e928115e2cc48fdbec4eb2a79c62bf"}]},{"id":"S019","semantic_type":"assumption","kind":"assumption","title":"ISTA噪声衰减","source_units":[{"start":58478,"end":59090,"start_line":669,"end_line":676,"raw_tex":"\\begin{assumption}\\label{assumpt:ista_noise}\nAt every iteration $k$,  $g_k=g(y_k,\\xi_k)$ computed using the stochastic first-order oracle SFO$_{\\constG, p}(y_k,\\alpha_k)$ satisfies  \n\\begin{align}\\label{eq:ista_noise}\n    &\\EE_{\\xi_k} \\left[ \\| (\\nabla f(y_{k} )- G_{k} )\\|^2 | \\pastone\\right] \\leq \\frac{1}{\\alpha_{k}^2\\cdot k^{2+ \\beta}},\n\\end{align}\nfor some fixed $\\beta>0$. \n%\\trang{I thought conditioning on the filtration is essentially the same as the randomness in the oracle? since that was the only random element of the algo.} \\katya{Yes, this just makes it more explicit, I think}. \n\\end{assumption}","sha256_utf8":"0b44195bfc09bf89afee421d6f49bb85a209547fe7d6f94998e654a566439bf8"}]},{"id":"S020","semantic_type":"definition","kind":"definition","title":"FISTA外推更新","source_units":[{"start":73063,"end":73507,"start_line":821,"end_line":825,"raw_tex":"We first present that deterministic algorithm and then show how it is modified for the stochastic first-order oracles. Compared to  FISTA  in \\cite{Beck-Teboulle}, FISTA-BKTR in \\cite{Scheinberg-Goldfarb-Bai} uses an additional  parameter $\\theta_k$, which keeps record of the changes in  $\\alpha_k$  and \nutilizes the following update  \n\\begin{align}\\label{eq:fista_step}\n   (t_{new}, y) = \\text{FISTAStep}(x, x^{prev}, t, \\theta).\n\\end{align}","sha256_utf8":"c7678fc1881b79b950daf3e3bfef5affb5e3f3b828ea24b3983e88fc54339313"},{"start":73508,"end":73676,"start_line":826,"end_line":830,"raw_tex":"where FISTAStep computes \n\\begin{align*}\n    t_{new} &=  \\frac{1 + \\sqrt{1+4\\theta t^2}}{2}, \\text{ and }\n    y = x + \\frac{t-1}{t_{new}} ( x - x^{prev} ).\n\\end{align*}","sha256_utf8":"154e985a6b71883da34d42af7cafea3e0651681a89c030eaaa1ef9d41c6e52ba"}]},{"id":"S021","semantic_type":"notation","kind":"notation","title":"成功步记忆变量","source_units":[{"start":75686,"end":76718,"start_line":863,"end_line":867,"raw_tex":" Here we will use the main FISTA-BKTR  scheme, however, as in the case of stochastic ISTA, the gradient estimate needs to be recomputed each time a step is attempted, even when the iteration is unsuccessful. Therefore the iteration counter gets increased on unsuccessful steps as well as successful ones and the algorithm notation needs to be updated accordingly. In particular \nwe use  $x^{prev}_k$ (instead of $x_{k-1}$) to denote the iterate of the last successful iteration, that is the last iterate different from $x_k$. \nSimilarly, $t_k^{next}$ denotes the value of $t_{k+1}$ that is assigned if iteration $k$ is successful, otherwise $t_{k+1}=t_k$. \nWe also need to keep track of the $\\alpha_k^{succ}$ - the step-size  parameter of the last successful iteration up to iteration $k$. \nOn the other hand, $\\alpha_k^0$ value is not used since the dynamics of the step-size  parameter are simplified - it is decreased by the factor of $\\gamma$ on each unsuccessful iteration and increased proportionally on each successful one. \n","sha256_utf8":"e11cbf33afe3b4bbe65666579f06b7cebadc1e9e6cf567c8147527ad96d6e806"}]},{"id":"S022","semantic_type":"algorithm","kind":"algorithm","title":"随机FISTA步长搜索","source_units":[{"start":78475,"end":80066,"start_line":882,"end_line":902,"raw_tex":"\\begin{algorithm}[ht]\n\\caption{Stochastic FISTA Step Search Algorithm}\n\\label{alg:fista_bktr_line_search}\n\\begin{algorithmic}[1]\n\\STATE {Initialization:} Choose an initial point $x_0\\in \\mathbb{R}^d$. Set $t_0 = 0$, let $x^{prev}_0 = x_0$. \\nonumber\\\\\nSet $0<\\gamma<1$, $\\theta_0 = \\gamma$ with $\\alpha_1>0$.  Let oracle SFO$_{\\constG, p}$ be given with  {$0 \\leq \\constG \\leq \\frac{1}{3}$} and $\\frac{1}{2} < p \\leq 1 $.\n\\FOR{$k=1,2,\\cdots,$}\n    \\STATE Compute $(t_k^{next}, y_k) = \\text{FISTAStep}(x_{k-1}, x^{prev}_{k-1}, t_{k-1}, \\theta_{k-1}) $\n    \\STATE Compute the gradient estimate $g_k=G(y_k,\\xi_k)$ using the stochastic first-order oracle SFO$_{\\constG, p}(y_k,\\alpha_k)$.\n    Compute the reference point $\\tilde{p}_{\\alpha_k} (y_k) =\\text{prox}_{\\alpha_k h} (y_k- \\alpha_k g_k)$.\n    \\STATE Check sufficient decrease condition $F(\\tilde{p}_{\\alpha_k} (y_k)) \\leq \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (y_k), y_k)$\n        \\STATE \\hspace{1.25em}If unsuccessful: \n            \\STATE \\hspace{2.5em}Set $\\left(x_{k},x^{prev}_{k},t_k\\right) = \\left(x_{k-1}, x^{prev}_{k-1}, t_{k-1}\\right)$, then set $\\alpha_{k+1} = \\gamma\\alpha_k$, %$\\alpha^{succ}_{k}=\\alpha^{succ}_{k-1}$ \n            and $\\theta_{k} = \\tfrac{\\theta_{k-1}}{\\gamma}$.\n        \\STATE \\hspace{1.25em}If successful: \n            \\STATE \\hspace{2.5em}Set $\\left(x_k, x^{prev}_{k},t_k\\right) = \\left(\\tilde{p}_{\\alpha_k} (y_k), x_{k-1}, t_k^{next}\\right)$,\n    then set $\\alpha_{k+1} = \\tfrac{\\alpha_k}{\\gamma},$ %$\\alpha^{succ}_{k}=\\alpha_k$ \n    and $\\theta_{k} = \\gamma$.\n\\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"8f0b0b0c1c212966ce0e01ae645afd9b165ffedb027399ce3bd126c1821e0ab1"}]},{"id":"S023","semantic_type":"notation","kind":"notation","title":"FISTA随机过程","source_units":[{"start":80533,"end":80756,"start_line":907,"end_line":908,"raw_tex":"Similar to Algorithm \\ref{alg:ista_bktr_line_search}, \nAlgorithm \\ref{alg:fista_bktr_line_search} generates a stochastic process $\\{Y_k, X_k, G_k, \\Stepsize{k}, T_k\\}$ with realizations $\\{y_k, x_k, g_k, \\alpha_k, t_k\\}$. \n","sha256_utf8":"933182186d4a7878b68b03aa1a47597ca7bfa3b48ad0635c3732784baf8fbb39"}]},{"id":"S024","semantic_type":"definition","kind":"definition","title":"能量与外推误差","source_units":[{"start":88051,"end":88153,"start_line":1011,"end_line":1013,"raw_tex":"\\begin{align}\\label{eq:define_m}\n    m_k = 2\\alpha^{succ}_{k} t_{k}^2v_{k} +\\norm{u_k }^2,\n\\end{align}","sha256_utf8":"85a09a0fa48d5bef11836766dcf483bb9218c2d5adf725f6944dee194ec5a653"},{"start":88154,"end":88236,"start_line":1014,"end_line":1014,"raw_tex":"where $v_k =  F(x_k) - F^* $ and $u_k = (t_{k}-1)x^{prev}_{k} + x^* - t_k x_k $. \n","sha256_utf8":"86e33995aeaa15c3677b657a04fbb32b82e1b953ea24e6968fcf70a70c928fa6"}]},{"id":"S025","semantic_type":"assumption","kind":"assumption","title":"FISTA噪声衰减","source_units":[{"start":102376,"end":102775,"start_line":1245,"end_line":1251,"raw_tex":"\\begin{assumption}\\label{assumpt:fista_noise}\nAt every iteration $k$,  $g_k=g(y_k,\\xi_k)$ computed using the stochastic first-order oracle SFO$_{\\constG, p}(y_k,\\alpha_k)$ satisfies \n\\begin{align}\\label{eq:fista_noise}\n&\\EE_{\\xi_k} \\left[ \\| (\\nabla f(y_{k} )- G_{k} )\\|^2 | \\pastone\\right] \\leq \\frac{1}{\\alpha_{k}^2t_{k}^2\\cdot k^{2+ \\beta}},\n\\end{align}\nfor some fixed $\\beta>0$.\n\\end{assumption}","sha256_utf8":"0f325bfa699fc76740e6eae9da907da20816982df68880483737fd8082437fe7"}]},{"id":"S026","semantic_type":"proof","kind":"proof","title":"公共计数框架迁移","source_units":[{"start":80756,"end":81271,"start_line":909,"end_line":912,"raw_tex":"We note that the steps 4 and 5 of Algorithm \\ref{alg:fista_bktr_line_search} are exactly as Algorithm \\ref{alg:ista_bktr_line_search}.\nSince the dynamics of the step-size parameter is based on the reference points $y_k$, $\\tilde{p}_{\\alpha_k} (y_k)$ in Step 4 and the sufficient decrease condition in Step 5, \nthe results in Sections \\ref{sec:stoc_proc} and \\ref{subsec:successful}  hold  for Algorithm \\ref{alg:fista_bktr_line_search} under the same assumptions as for \nAlgorithm \\ref{alg:ista_bktr_line_search}. \n","sha256_utf8":"7443ff318e9afabe7c78f3b8888baf07963d511a0d7ef023b69d788b287192ca"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"核对相同步长机制","source_units":[{"start":80756,"end":81271,"start_line":909,"end_line":912,"raw_tex":"We note that the steps 4 and 5 of Algorithm \\ref{alg:fista_bktr_line_search} are exactly as Algorithm \\ref{alg:ista_bktr_line_search}.\nSince the dynamics of the step-size parameter is based on the reference points $y_k$, $\\tilde{p}_{\\alpha_k} (y_k)$ in Step 4 and the sufficient decrease condition in Step 5, \nthe results in Sections \\ref{sec:stoc_proc} and \\ref{subsec:successful}  hold  for Algorithm \\ref{alg:fista_bktr_line_search} under the same assumptions as for \nAlgorithm \\ref{alg:ista_bktr_line_search}. \n","sha256_utf8":"7443ff318e9afabe7c78f3b8888baf07963d511a0d7ef023b69d788b287192ca"}]},{"id":"S028","semantic_type":"lemma","kind":"lemma","title":"小步长真迭代成功","source_units":[{"start":41250,"end":41714,"start_line":393,"end_line":405,"raw_tex":"\\begin{lemma}\n\\label{lem:good_est_good_model_successful} \nSuppose that iteration $k$ is true, i.e. \n\\begin{align*}\n    \\|g_k-\\nabla f(y_k)\\|\\leq \\constG  \\|D_{\\alpha_k}(y_k)\\|,\n\\end{align*}\nwith  {$\\constG \\leq \\frac{1}{3}$} and $D_{\\alpha_k}(y_k)$ is the gradient mapping of $F$ at $y_k$. If \n\\begin{align*}\n\\stepsize{k}{} \\le \\stepsizebar := \\frac{1}{L}\\left( 1 - \\frac{2\\constG}{1-\\constG}\\right) ,\n\\end{align*}\nthen the $k$-th step is successful. \n\n\\end{lemma}","sha256_utf8":"fcf99e1ec4baa893a6cc12197390889eb086a34350840081260e99bedb2e77fd"}]},{"id":"S029","semantic_type":"proof","kind":"proof","title":"小步长成功性证明","source_units":[{"start":41717,"end":45196,"start_line":407,"end_line":459,"raw_tex":"\\begin{proof}\n\n\nFrom the $L$-smoothness of $f$ we have\n\\begin{align*}\n    f(\\tilde{p}_{\\alpha_k} (y_k))  &\\le f(y_k) + \\nabla f(y_k)^\\top(\\tilde{p}_{\\alpha_k} (y_k) - y_k) + \\frac{L}{2}\\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 \\nonumber\\\\\n     &= f(y_k) +  g_k^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k) + [\\nabla f(y_k)-g_k]^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k)  \\nonumber\\\\\n     &\\quad+ \\frac{1}{2\\alpha_k} \\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2  + \\left(\\frac{L}{2}- \\frac{1}{2\\alpha_k}\\right)\\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 .\n\\end{align*}\nAdding the non-smooth term $ h(\\tilde{p}_{\\alpha_k} (y_k))$ we have\n\\begin{align*}\n    &F (\\tilde{p}_{\\alpha_k} (y_k))  \\nonumber\\\\\n     &\\le f(y_k) +  g_k^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k) + \\frac{1}{2\\alpha_k} \\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 + h(\\tilde{p}_{\\alpha_k} (y_k))  \\nonumber\\\\\n     &\\quad + [\\nabla f(y_k)-g_k]^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k) + \\left(\\frac{L}{2}- \\frac{1}{2\\alpha_k}\\right)\\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 \n     \\nonumber\\\\\n     &= \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (y_k), y_k) + [\\nabla f(y_k)-g_k]^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k) + \\left(\\frac{L}{2}- \\frac{1}{2\\alpha_k}\\right)\\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 .\n\\end{align*}\nTo show that the iteration is successful i.e. $F(\\tilde{p}_{\\alpha_k} (y_k)) \\leq \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (y_k), y_k)$\nwe need to prove that \n\\begin{align}\\label{eq:02}\n    [\\nabla f(y_k)-g_k]^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k) \\leq \\left(\\frac{1}{2\\alpha_k} - \\frac{L}{2}\\right)\\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 .\n\\end{align}\nNote that the right hand side is non-negative since $\\alpha_k \\leq \\frac{1}{L}$. \nOur definition of the true iteration leads to \n\\begin{align}\\label{eq:01}\n    \\|g_k-\\nabla f(y_k)\\|\n    &\\leq \\constG \\left \\|\\frac{1}{\\alpha_k}\\Big(y_k - p_{\\alpha_k} (y_k)\\Big)\\right\\|\\nonumber  \\\\\n    &\\leq  \\frac{\\constG}{\\alpha_k} \\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\| + \\frac{\\constG}{\\alpha_k} \\left \\|\\tilde{p}_{\\alpha_k} (y_k) - p_{\\alpha_k} (y_k)\\right\\|\\nonumber  \\\\\n    &=  \\frac{\\constG}{\\alpha_k} \\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\| + \\frac{\\constG}{\\alpha_k} \\norm{\\alpha_k\\nabla f(y_k)-\\alpha_k g_k },\n\\end{align}\nwhere the last inequality follows  from the nonexpansiveness of the prox-operator $\\text{prox}_{\\alpha h}$ for  proper, closed, and convex $h(x)$, i.e.,\n\\begin{align*}\n    \\|\\text{prox}_{\\alpha h}(z_1) - \\text{prox}_{\\alpha h}(z_2)\\| \\leq \\|z_1 - z_2\\| \\text{ for all } z_1, z_2\\in\\R^d.\n\\end{align*}\nThe bound \\eqref{eq:01} is equivalent to\n\\begin{align*}\n    (1-\\constG)\\norm{\\nabla f(y_k)-g_k} \\leq  \\frac{\\constG}{\\alpha_k} \\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\|.\n\\end{align*}\nTherefore we show \\eqref{eq:02} by\n\\begin{align*}\n    \\norm{\\nabla f(y_k)-g_k} &\\leq  \\frac{\\constG}{1-\\constG} \\frac{1}{\\alpha_k} \\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\|  \\nonumber\\\\\n    &\\leq \\left( \\frac{1- \\alpha_k  L}{2}\\right)\\frac{1}{\\alpha_k} \\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\| = \\left( \\frac{1}{2\\alpha_k} - \\frac{L}{2}\\right)\\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\|,\n\\end{align*}\nwhere we use the condition that \n\\begin{align*}\n    \\frac{\\constG}{1-\\constG} \\leq \\frac{1- \\alpha_k  L}{2},\n\\end{align*}\nwhich is the consequence of the small step-size parameter: \n\\begin{align*}\n    \\alpha_k  \\leq \\frac{1}{L}\\left( 1 - \\frac{2\\constG}{1-\\constG}\\right).\n\\end{align*}\n\n~\\qed \\end{proof}","sha256_utf8":"ba0640886b91c5eb45d2dc7d72877e91929302d19d4dea5914d59cc85283a2f3"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"光滑性展开模型误差","source_units":[{"start":41733,"end":42987,"start_line":410,"end_line":423,"raw_tex":"From the $L$-smoothness of $f$ we have\n\\begin{align*}\n    f(\\tilde{p}_{\\alpha_k} (y_k))  &\\le f(y_k) + \\nabla f(y_k)^\\top(\\tilde{p}_{\\alpha_k} (y_k) - y_k) + \\frac{L}{2}\\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 \\nonumber\\\\\n     &= f(y_k) +  g_k^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k) + [\\nabla f(y_k)-g_k]^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k)  \\nonumber\\\\\n     &\\quad+ \\frac{1}{2\\alpha_k} \\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2  + \\left(\\frac{L}{2}- \\frac{1}{2\\alpha_k}\\right)\\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 .\n\\end{align*}\nAdding the non-smooth term $ h(\\tilde{p}_{\\alpha_k} (y_k))$ we have\n\\begin{align*}\n    &F (\\tilde{p}_{\\alpha_k} (y_k))  \\nonumber\\\\\n     &\\le f(y_k) +  g_k^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k) + \\frac{1}{2\\alpha_k} \\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 + h(\\tilde{p}_{\\alpha_k} (y_k))  \\nonumber\\\\\n     &\\quad + [\\nabla f(y_k)-g_k]^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k) + \\left(\\frac{L}{2}- \\frac{1}{2\\alpha_k}\\right)\\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 \n     \\nonumber\\\\\n     &= \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (y_k), y_k) + [\\nabla f(y_k)-g_k]^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k) + \\left(\\frac{L}{2}- \\frac{1}{2\\alpha_k}\\right)\\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 .\n\\end{align*}\n","sha256_utf8":"ac14fc5fc4eb5247e74339ae343fbabb39fe2601f581644a30a8e1a9f410f128"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"归约充分下降条件","source_units":[{"start":42987,"end":43427,"start_line":424,"end_line":429,"raw_tex":"To show that the iteration is successful i.e. $F(\\tilde{p}_{\\alpha_k} (y_k)) \\leq \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (y_k), y_k)$\nwe need to prove that \n\\begin{align}\\label{eq:02}\n    [\\nabla f(y_k)-g_k]^\\top (\\tilde{p}_{\\alpha_k} (y_k) - y_k) \\leq \\left(\\frac{1}{2\\alpha_k} - \\frac{L}{2}\\right)\\norm{\\tilde{p}_{\\alpha_k} (y_k) - y_k}^2 .\n\\end{align}\nNote that the right hand side is non-negative since $\\alpha_k \\leq \\frac{1}{L}$. \n","sha256_utf8":"d03ed8a4260beefb4be24ec5f7e6723291a38bd90d1d6e487eba2e637c6ac5b4"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"非扩张性控制误差","source_units":[{"start":43427,"end":44471,"start_line":430,"end_line":444,"raw_tex":"Our definition of the true iteration leads to \n\\begin{align}\\label{eq:01}\n    \\|g_k-\\nabla f(y_k)\\|\n    &\\leq \\constG \\left \\|\\frac{1}{\\alpha_k}\\Big(y_k - p_{\\alpha_k} (y_k)\\Big)\\right\\|\\nonumber  \\\\\n    &\\leq  \\frac{\\constG}{\\alpha_k} \\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\| + \\frac{\\constG}{\\alpha_k} \\left \\|\\tilde{p}_{\\alpha_k} (y_k) - p_{\\alpha_k} (y_k)\\right\\|\\nonumber  \\\\\n    &=  \\frac{\\constG}{\\alpha_k} \\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\| + \\frac{\\constG}{\\alpha_k} \\norm{\\alpha_k\\nabla f(y_k)-\\alpha_k g_k },\n\\end{align}\nwhere the last inequality follows  from the nonexpansiveness of the prox-operator $\\text{prox}_{\\alpha h}$ for  proper, closed, and convex $h(x)$, i.e.,\n\\begin{align*}\n    \\|\\text{prox}_{\\alpha h}(z_1) - \\text{prox}_{\\alpha h}(z_2)\\| \\leq \\|z_1 - z_2\\| \\text{ for all } z_1, z_2\\in\\R^d.\n\\end{align*}\nThe bound \\eqref{eq:01} is equivalent to\n\\begin{align*}\n    (1-\\constG)\\norm{\\nabla f(y_k)-g_k} \\leq  \\frac{\\constG}{\\alpha_k} \\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\|.\n\\end{align*}\n","sha256_utf8":"5aec08fe88af0463ede1dcb2fe1c42b916e52aa2c08968a4e0da1ff6935a1985"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"步长阈值保证下降","source_units":[{"start":44471,"end":45196,"start_line":445,"end_line":459,"raw_tex":"Therefore we show \\eqref{eq:02} by\n\\begin{align*}\n    \\norm{\\nabla f(y_k)-g_k} &\\leq  \\frac{\\constG}{1-\\constG} \\frac{1}{\\alpha_k} \\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\|  \\nonumber\\\\\n    &\\leq \\left( \\frac{1- \\alpha_k  L}{2}\\right)\\frac{1}{\\alpha_k} \\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\| = \\left( \\frac{1}{2\\alpha_k} - \\frac{L}{2}\\right)\\left \\|y_k - \\tilde{p}_{\\alpha_k} (y_k)\\right\\|,\n\\end{align*}\nwhere we use the condition that \n\\begin{align*}\n    \\frac{\\constG}{1-\\constG} \\leq \\frac{1- \\alpha_k  L}{2},\n\\end{align*}\nwhich is the consequence of the small step-size parameter: \n\\begin{align*}\n    \\alpha_k  \\leq \\frac{1}{L}\\left( 1 - \\frac{2\\constG}{1-\\constG}\\right).\n\\end{align*}\n\n~\\qed \\end{proof}","sha256_utf8":"81d5c7aa29769053257b879f890b5099e2550a7ab2d0bf81fe83477109d9a01a"}]},{"id":"S034","semantic_type":"theorem","kind":"theorem","title":"总迭代与好迭代关系","source_units":[{"start":46134,"end":46519,"start_line":470,"end_line":476,"raw_tex":"\\begin{theorem}[Bounding $\\EE (N_\\epsilon)$ based on $\\EE(N_G)$]\\label{th:framework}\nLet $\\stepsizebar$ be the value stated in Lemma \\ref{lem:good_est_good_model_successful}.\n Under Assumption \\ref{assumpt:key} we have \n\\begin{align*}\n\\EE (N_\\epsilon -1) \\leq \\frac{2p}{(2p-1)^2}\\left (2\\EE(N_G)+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right).\n\\end{align*}\n\\end{theorem}","sha256_utf8":"c59465760237095e2a01fcd4da684375bf36221ddb3a9023f6f34447d4ec4b49"}]},{"id":"S035","semantic_type":"proof","kind":"proof","title":"随机步长计数证明","source_units":[{"start":46520,"end":50575,"start_line":477,"end_line":551,"raw_tex":"\\begin{proof}\nBy the update rule of $\\alpha_k$ in Algorithm \\ref{alg:ista_bktr_line_search}, the stochastic process $\\{\\Alpha_k\\}_{k\\geq 1}$ follows: \n\\begin{align*}\n\\Alpha_{1}&= \\alpha_1, \\quad \n\\Alpha_{k+1}=\\left \\{ \\begin{array}{ll} \n\\gamma^{-1} \\Alpha_k & {\\rm if\\  }  \\iterSucc=1,  \\\\\n \\gamma \\Alpha_k &   {\\rm if\\  }  \\iterSucc=0 .\n\\end{array}\\right . \n\\end{align*}\n\nLet us now define the following indicator random variables: %two indicator random variables, in addition to $\\iterTrue$ defined earlier,\n\\begin{align*}\n\\iterLarge=\\mathbbm{1}\\{ \\Alpha_k >\\stepsizebar\\}, \\\\\n\\iterLargebar= \\mathbbm{1}\\{ \\Alpha_k \\geq \\stepsizebar\\}.\n\\end{align*}\nClearly \n\\begin{align*}\n\\EE (N_\\epsilon-1) = \\EE\\left (\\sum_{k=1}^{N_\\epsilon-1} \\iterLarge\\right ) + \\EE \\left (\\sum_{k=1}^{N_\\epsilon-1} (1-\\iterLarge)\\right ).\n\\end{align*}\n\nFrom Lemma 2.3 in \\cite{Cartis-Scheinberg}, under Assumption \\ref{assumpt:key} and using the fact that all true iterations are successful when $\\alpha_k \\le \\stepsizebar$, we have: \n\\begin{align*}\n\\EE\\left (\\sum_{k=1}^{N_\\epsilon-1} (1-\\iterLarge)\\right)\\leq \\frac{1}{2p}\\EE(N_\\epsilon -1 ).\n\\end{align*}\n\n\nWe define \n\\begin{itemize}\n\\item $N_{FS}=\\sum_{k=1}^{N_\\epsilon-1} \\iterLargebar (1-\\iterTrue)\\iterSucc$, which is the number of false successful iterations with $\\Alpha_k\\geq \\stepsizebar$.\n\\item $N_F=\\sum_{k=1}^{N_\\epsilon-1} \\iterLargebar (1-\\iterTrue)$, which is the number of false  iterations with $\\Alpha_k\\geq \\stepsizebar$.\n\\item $N_G=\\sum_{k=1}^{N_\\epsilon-1} \\iterLargebar \\iterTrue \\iterSucc$, which is the number of true successful iterations with $\\Alpha_k\\geq \\stepsizebar$.\n\\item $N_T=\\sum_{k=1}^{N_\\epsilon-1} \\iterLargebar \\iterTrue$, which is the number of true iterations with $\\Alpha_k\\geq \\stepsizebar$.\n% \\item $N_{TU}=\\sum_{k=1}^{N_\\epsilon-1} \\iterLarge \\iterTrue (1-\\iterSucc)$, which is the number of true unsuccessful iterations with $\\Alpha_k> \\stepsizebar$.\n\\item $N_U=\\sum_{k=1}^{N_\\epsilon-1} \\iterLarge  (1-\\iterSucc)$, which is the number of unsuccessful iterations with $\\Alpha_k> \\stepsizebar$.\n\\end{itemize}\n\nFrom equations (11) and (12) of Lemma 2.7 in \\cite{Cartis-Scheinberg} we have\n\\begin{align}\\label{eq:binom}\n\\EE(N_{FS})\\leq \\EE(N_{F})\\leq\\frac {1-p}{p}[\\EE(N_G)+\\EE(N_U)],\n\\end{align}\nand\n\\begin{align}\\label{eq:boundM3}\n\\EE(N_U)\\leq \\EE(N_{FS})+\\EE(N_G) + \\log_\\gamma(\\stepsizebar/\\alpha_1).\n\\end{align}\nPlugging \\eqref{eq:boundM3} into  \\eqref{eq:binom}\nwe obtain\n\\begin{align*}\n\\EE(N_{FS})\\leq \\frac {1-p}{p}\\left[\\EE(N_{FS})+2\\EE(N_G) + \\log_\\gamma(\\stepsizebar/\\alpha_1)\\right],\n\\end{align*}\nand, hence,\n\\begin{align*}\n\\frac {2p-1}{p}\\EE(N_{FS})\\leq \\frac {1-p}{p}\\left[ 2\\EE(N_G) + \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right].\n\\end{align*}\nThis finally implies \n\\begin{align}\\label{eq:boundN1}\n\\EE(N_{FS})\\leq \\frac {1-p}{2p-1}\\left [2\\EE(N_G)+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right ].\n\\end{align}\nNow we can bound the expected total  number of  iterations when $\\alpha_k>\\stepsizebar$, using %\\eqref{eq:N2}, \n\\eqref{eq:boundM3} and \\eqref{eq:boundN1}  \nand adding the terms to obtain the result of the lemma, namely,\n\\begin{align*}\n\\EE(N_F+N_T)&\\leq \\EE(N_F+N_U+N_G)\\leq \\frac{1}{p}\\EE(N_U+N_G)\\nonumber\\\\\n&\\leq \\frac{1}{2p-1}\\left (2\\EE(N_G)+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right).\n\\end{align*}\nNext, from Lemma 2.7 in \\cite{Cartis-Scheinberg},\\footnote{Final statement of the Lemma 2.7 in \\cite{Cartis-Scheinberg}  replaces $\\EE(N_G)$ with its bound, but here we\nare yet to derive such a  bound, therefore we explicitly use $\\EE(N_G)$.}\n\\begin{align*}\n\\EE\\left (\\sum_{k=1}^{N_\\epsilon-1}\\iterLarge \\right )\\leq \\frac{1}{2p-1}\\left (2\\EE(N_G)+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right). \n\\end{align*}\nHence, using the two previous results we have \n\\begin{align*}\n\\EE (N_\\epsilon -1)\\leq \\frac{1}{2p} \\EE (N_\\epsilon -1) +   \\frac{1}{2p-1} \\left (2\\EE(N_G)+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right).\n\\end{align*}\nThe result of the theorem follows. \n~\\qed \\end{proof}","sha256_utf8":"b4127294bec3b5123ca6ead8e3b368ee730194465cd1a348f55cde096ecea1a2"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"步长递推及迭代分解","source_units":[{"start":46520,"end":47347,"start_line":477,"end_line":495,"raw_tex":"\\begin{proof}\nBy the update rule of $\\alpha_k$ in Algorithm \\ref{alg:ista_bktr_line_search}, the stochastic process $\\{\\Alpha_k\\}_{k\\geq 1}$ follows: \n\\begin{align*}\n\\Alpha_{1}&= \\alpha_1, \\quad \n\\Alpha_{k+1}=\\left \\{ \\begin{array}{ll} \n\\gamma^{-1} \\Alpha_k & {\\rm if\\  }  \\iterSucc=1,  \\\\\n \\gamma \\Alpha_k &   {\\rm if\\  }  \\iterSucc=0 .\n\\end{array}\\right . \n\\end{align*}\n\nLet us now define the following indicator random variables: %two indicator random variables, in addition to $\\iterTrue$ defined earlier,\n\\begin{align*}\n\\iterLarge=\\mathbbm{1}\\{ \\Alpha_k >\\stepsizebar\\}, \\\\\n\\iterLargebar= \\mathbbm{1}\\{ \\Alpha_k \\geq \\stepsizebar\\}.\n\\end{align*}\nClearly \n\\begin{align*}\n\\EE (N_\\epsilon-1) = \\EE\\left (\\sum_{k=1}^{N_\\epsilon-1} \\iterLarge\\right ) + \\EE \\left (\\sum_{k=1}^{N_\\epsilon-1} (1-\\iterLarge)\\right ).\n\\end{align*}\n","sha256_utf8":"9759e51a2c70ab2a33270a5dcd9309b18aed46635592d8d6ff2486452e62f844"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"控制小步长迭代数","source_units":[{"start":47348,"end":47653,"start_line":497,"end_line":500,"raw_tex":"From Lemma 2.3 in \\cite{Cartis-Scheinberg}, under Assumption \\ref{assumpt:key} and using the fact that all true iterations are successful when $\\alpha_k \\le \\stepsizebar$, we have: \n\\begin{align*}\n\\EE\\left (\\sum_{k=1}^{N_\\epsilon-1} (1-\\iterLarge)\\right)\\leq \\frac{1}{2p}\\EE(N_\\epsilon -1 ).\n\\end{align*}\n","sha256_utf8":"0565bace79d5ffa5c1027369839da522c216041800d5adb093daa03ce5407956"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"定义真假成功计数","source_units":[{"start":47655,"end":48600,"start_line":503,"end_line":511,"raw_tex":"We define \n\\begin{itemize}\n\\item $N_{FS}=\\sum_{k=1}^{N_\\epsilon-1} \\iterLargebar (1-\\iterTrue)\\iterSucc$, which is the number of false successful iterations with $\\Alpha_k\\geq \\stepsizebar$.\n\\item $N_F=\\sum_{k=1}^{N_\\epsilon-1} \\iterLargebar (1-\\iterTrue)$, which is the number of false  iterations with $\\Alpha_k\\geq \\stepsizebar$.\n\\item $N_G=\\sum_{k=1}^{N_\\epsilon-1} \\iterLargebar \\iterTrue \\iterSucc$, which is the number of true successful iterations with $\\Alpha_k\\geq \\stepsizebar$.\n\\item $N_T=\\sum_{k=1}^{N_\\epsilon-1} \\iterLargebar \\iterTrue$, which is the number of true iterations with $\\Alpha_k\\geq \\stepsizebar$.\n% \\item $N_{TU}=\\sum_{k=1}^{N_\\epsilon-1} \\iterLarge \\iterTrue (1-\\iterSucc)$, which is the number of true unsuccessful iterations with $\\Alpha_k> \\stepsizebar$.\n\\item $N_U=\\sum_{k=1}^{N_\\epsilon-1} \\iterLarge  (1-\\iterSucc)$, which is the number of unsuccessful iterations with $\\Alpha_k> \\stepsizebar$.\n\\end{itemize}\n","sha256_utf8":"30ac7d271456f0d6f59c9d67abc664c56dd07017e1153e4313a5eb1297336f90"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"调用搜索计数不等式","source_units":[{"start":48601,"end":48906,"start_line":513,"end_line":520,"raw_tex":"From equations (11) and (12) of Lemma 2.7 in \\cite{Cartis-Scheinberg} we have\n\\begin{align}\\label{eq:binom}\n\\EE(N_{FS})\\leq \\EE(N_{F})\\leq\\frac {1-p}{p}[\\EE(N_G)+\\EE(N_U)],\n\\end{align}\nand\n\\begin{align}\\label{eq:boundM3}\n\\EE(N_U)\\leq \\EE(N_{FS})+\\EE(N_G) + \\log_\\gamma(\\stepsizebar/\\alpha_1).\n\\end{align}\n","sha256_utf8":"b67b1d4ceeb630d84f75960c4ec1b9e06e868701ea50bb3c596bf0a655acc8d3"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"消去失败成功计数","source_units":[{"start":48906,"end":49444,"start_line":521,"end_line":533,"raw_tex":"Plugging \\eqref{eq:boundM3} into  \\eqref{eq:binom}\nwe obtain\n\\begin{align*}\n\\EE(N_{FS})\\leq \\frac {1-p}{p}\\left[\\EE(N_{FS})+2\\EE(N_G) + \\log_\\gamma(\\stepsizebar/\\alpha_1)\\right],\n\\end{align*}\nand, hence,\n\\begin{align*}\n\\frac {2p-1}{p}\\EE(N_{FS})\\leq \\frac {1-p}{p}\\left[ 2\\EE(N_G) + \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right].\n\\end{align*}\nThis finally implies \n\\begin{align}\\label{eq:boundN1}\n\\EE(N_{FS})\\leq \\frac {1-p}{2p-1}\\left [2\\EE(N_G)+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right ].\n\\end{align}\n","sha256_utf8":"ff530a13f02b57b069030b58027e98500ba6a2207e69b836b42626639a5b4afb"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"界定大步长迭代数","source_units":[{"start":49444,"end":50291,"start_line":534,"end_line":545,"raw_tex":"Now we can bound the expected total  number of  iterations when $\\alpha_k>\\stepsizebar$, using %\\eqref{eq:N2}, \n\\eqref{eq:boundM3} and \\eqref{eq:boundN1}  \nand adding the terms to obtain the result of the lemma, namely,\n\\begin{align*}\n\\EE(N_F+N_T)&\\leq \\EE(N_F+N_U+N_G)\\leq \\frac{1}{p}\\EE(N_U+N_G)\\nonumber\\\\\n&\\leq \\frac{1}{2p-1}\\left (2\\EE(N_G)+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right).\n\\end{align*}\nNext, from Lemma 2.7 in \\cite{Cartis-Scheinberg},\\footnote{Final statement of the Lemma 2.7 in \\cite{Cartis-Scheinberg}  replaces $\\EE(N_G)$ with its bound, but here we\nare yet to derive such a  bound, therefore we explicitly use $\\EE(N_G)$.}\n\\begin{align*}\n\\EE\\left (\\sum_{k=1}^{N_\\epsilon-1}\\iterLarge \\right )\\leq \\frac{1}{2p-1}\\left (2\\EE(N_G)+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right). \n\\end{align*}\n","sha256_utf8":"534d1308fb28bda598e8567d2ba8d8a0c371edf15a33d5ffd62b5cd6a712493a"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"合并两类迭代计数","source_units":[{"start":50291,"end":50575,"start_line":546,"end_line":551,"raw_tex":"Hence, using the two previous results we have \n\\begin{align*}\n\\EE (N_\\epsilon -1)\\leq \\frac{1}{2p} \\EE (N_\\epsilon -1) +   \\frac{1}{2p-1} \\left (2\\EE(N_G)+ \\log_\\gamma\\left(\\frac{\\stepsizebar}{\\alpha_1}\\right)\\right).\n\\end{align*}\nThe result of the theorem follows. \n~\\qed \\end{proof}","sha256_utf8":"a1f711a8c8d95ce481f44884270b89d2e6381bc52acac523cf26818a02a3009b"}]},{"id":"S043","semantic_type":"lemma","kind":"lemma","title":"ISTA成功步距离界","source_units":[{"start":51085,"end":51426,"start_line":561,"end_line":567,"raw_tex":"\\begin{lemma}[Bounds for a successful step]\\label{lem:successful_step_bound1}\nIf $k$ is a successful iteration then we have:\n\\begin{align*}\n\\|x_{k} - x^*\\|^2 - \\|x_{k-1} - x^* \\|^2 \\leq  - 2\\alpha_k (F(x_k) - F(x^*) )  + \\lambda_k \\cdot \\|x_{k-1} -x^*\\|,\n\\end{align*}\nwhere $\\lambda_k = \\| 2\\alpha_k (\\nabla f(x_{k-1})-g_k) \\| $.\n\\end{lemma}","sha256_utf8":"1a7cc48416c21241fd96fef760bdd1d8b8ac7629d23a357d29a6ed3661f34ebb"}]},{"id":"S044","semantic_type":"proof","kind":"proof","title":"ISTA单步估计证明","source_units":[{"start":51428,"end":54172,"start_line":569,"end_line":612,"raw_tex":"\\begin{proof}\nSince iteration $k$ is successful,  the sufficient decrease condition holds: \n\\begin{align*}\n    F(\\tilde{p}_{\\alpha_k} (x_{k-1})) &\\leq \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (x_{k-1}), x_{k-1}) \\nonumber\\\\\n    &= f(x_{k-1}) +(x_k -x_{k-1})^\\top g_k + \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2 + h(x_k).\n\\end{align*}\nBy the  convexity of $f$ and $h$, applied at $x_{k-1}$ and $x_k$ respectively, for any $x$,  \n \\begin{align*}\nf(x) &\\geq f(x_{k-1}) + \\nabla f(x_{k-1})^\\top (x-x_{k-1}),\n\\nonumber\\\\\nh(x) &\\geq h(x_k) + \\gamma(x_{k-1})^\\top (x-x_k),\n\\end{align*}\nwhere the second bound holds for every subgradient $\\gamma(x_{k-1})$ of $h$ at $x_{k}$. \nSumming those inequality we have \n\\begin{align*}\nf(x) +h(x) \\geq f(x_{k-1}) + \\nabla f(x_{k-1})^\\top (x-x_{k-1})+ h(x_k) + \\gamma(x_{k-1})^\\top (x-x_k).\n\\end{align*}\nCombining this with the sufficient decrease condition we get: \n\\begin{align*}\n    &F(x) - F(x_k) \\nonumber\\\\&\\geq \\nabla f(x_{k-1})^\\top (x-x_{k-1}) + \\gamma(x_{k-1})^\\top (x-x_k) - (x_k -x_{k-1})^\\top g_k - \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2\\nonumber\\\\\n    &= (\\nabla f(x_{k-1})-g_k) ^\\top (x-x_{k-1}) + (\\gamma(x_{k-1})+g_k)^\\top (x-x_k) - \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2.\n\\end{align*}\nUsing the fact that $x_k = \\tilde{p}_{\\alpha_k} (x_{k-1}) =\\arg\\min_x \\tilde{Q}_{\\alpha_k}(x, x_{k-1})$, we get that $ x_{k-1} - x_k= \\alpha_k[\\gamma(x_{k-1})+g_k]$ for some subgradient $\\gamma(x_{k-1})$ of $h$ at $x_{k}$. Therefore\n\\begin{align*}\n    &F(x) - F(x_k) \\nonumber\\\\\n    &\\geq (\\nabla f(x_{k-1})-g_k) ^\\top (x-x_{k-1}) + \\frac{1}{\\alpha_k}(x_{k-1} - x_k)^\\top (x-x_k) - \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2\\nonumber\\\\\n    &= (\\nabla f(x_{k-1})-g_k) ^\\top (x-x_{k-1}) + \\frac{1}{\\alpha_k}(x_{k-1} - x_k)^\\top (x-x_{k-1} + x_{k-1} -x_k) \\nonumber\\\\\n    & \\quad - \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2\\nonumber\\\\\n    &= (\\nabla f(x_{k-1})-g_k) ^\\top (x-x_{k-1}) + \\frac{1}{\\alpha_k}(x_{k-1} - x_k)^\\top (x-x_{k-1}) + \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2\\nonumber\\\\\n    &= (\\nabla f(x_{k-1})-g_k) ^\\top (x-x_{k-1}) + \\frac{1}{2\\alpha_k} \\left( \\|x-x_k \\|^2-\\|x-x_{k-1} \\|^2\\right),\n\\end{align*}\nand we get the bound for $x=x^*$: \n\\begin{align*}\n    &2\\alpha_k (F(x_k) - F(x^*) ) - 2\\alpha_k (\\nabla f(x_{k-1})-g_k) ^\\top (x_{k-1} -x^*)\n    \\leq \\|x_{k-1} - x^*\\|^2 - \\|x_k - x^* \\|^2.\n\\end{align*}\nRearranging the terms, we get the statement of Lemma \\ref{lem:successful_step_bound1}:\n\\begin{align*}\n    \\|x_k - x^* \\|^2 - \\|x_{k-1} - x^*\\|^2 &\\leq -2\\alpha_k (F(x_k) - F(x^*) ) + 2\\alpha_k (\\nabla f(x_{k-1})-g_k) ^\\top (x_{k-1} -x^*)\\nonumber\\\\\n    &\\leq -2\\alpha_k (F(x_k) - F(x^*) ) + \\lambda_k \\cdot\\|x_{k-1} -x^*\\|,\n\\end{align*}\nwhere $\\lambda_k = \\|2\\alpha_k (\\nabla f(x_{k-1})-g_k)\\|$.\n\n~\\qed \\end{proof}","sha256_utf8":"a447eb588a2fb26032018a0710ef8f024ca64155de7763710298f8732c01933e"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"写出成功步下降式","source_units":[{"start":51428,"end":51758,"start_line":569,"end_line":574,"raw_tex":"\\begin{proof}\nSince iteration $k$ is successful,  the sufficient decrease condition holds: \n\\begin{align*}\n    F(\\tilde{p}_{\\alpha_k} (x_{k-1})) &\\leq \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (x_{k-1}), x_{k-1}) \\nonumber\\\\\n    &= f(x_{k-1}) +(x_k -x_{k-1})^\\top g_k + \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2 + h(x_k).\n\\end{align*}\n","sha256_utf8":"aeb22f5a6e0db8b41e7612958b0c866a2181ee4508e8eaf7522a282ec555ed6b"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"相加凸性支撑不等式","source_units":[{"start":51758,"end":52258,"start_line":575,"end_line":585,"raw_tex":"By the  convexity of $f$ and $h$, applied at $x_{k-1}$ and $x_k$ respectively, for any $x$,  \n \\begin{align*}\nf(x) &\\geq f(x_{k-1}) + \\nabla f(x_{k-1})^\\top (x-x_{k-1}),\n\\nonumber\\\\\nh(x) &\\geq h(x_k) + \\gamma(x_{k-1})^\\top (x-x_k),\n\\end{align*}\nwhere the second bound holds for every subgradient $\\gamma(x_{k-1})$ of $h$ at $x_{k}$. \nSumming those inequality we have \n\\begin{align*}\nf(x) +h(x) \\geq f(x_{k-1}) + \\nabla f(x_{k-1})^\\top (x-x_{k-1})+ h(x_k) + \\gamma(x_{k-1})^\\top (x-x_k).\n\\end{align*}\n","sha256_utf8":"0663f4f880fa23f3f78c5af596fefb042ea2cc53151a854e1017dba5b2b9bbba"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"合并目标差与噪声","source_units":[{"start":52258,"end":52653,"start_line":586,"end_line":590,"raw_tex":"Combining this with the sufficient decrease condition we get: \n\\begin{align*}\n    &F(x) - F(x_k) \\nonumber\\\\&\\geq \\nabla f(x_{k-1})^\\top (x-x_{k-1}) + \\gamma(x_{k-1})^\\top (x-x_k) - (x_k -x_{k-1})^\\top g_k - \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2\\nonumber\\\\\n    &= (\\nabla f(x_{k-1})-g_k) ^\\top (x-x_{k-1}) + (\\gamma(x_{k-1})+g_k)^\\top (x-x_k) - \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2.\n\\end{align*}\n","sha256_utf8":"98d4787ec4a3ccc259920fad40ddd0e92488f523d63d3c148d95d120fcf55c89"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"近端最优条件化距离差","source_units":[{"start":52653,"end":53555,"start_line":591,"end_line":599,"raw_tex":"Using the fact that $x_k = \\tilde{p}_{\\alpha_k} (x_{k-1}) =\\arg\\min_x \\tilde{Q}_{\\alpha_k}(x, x_{k-1})$, we get that $ x_{k-1} - x_k= \\alpha_k[\\gamma(x_{k-1})+g_k]$ for some subgradient $\\gamma(x_{k-1})$ of $h$ at $x_{k}$. Therefore\n\\begin{align*}\n    &F(x) - F(x_k) \\nonumber\\\\\n    &\\geq (\\nabla f(x_{k-1})-g_k) ^\\top (x-x_{k-1}) + \\frac{1}{\\alpha_k}(x_{k-1} - x_k)^\\top (x-x_k) - \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2\\nonumber\\\\\n    &= (\\nabla f(x_{k-1})-g_k) ^\\top (x-x_{k-1}) + \\frac{1}{\\alpha_k}(x_{k-1} - x_k)^\\top (x-x_{k-1} + x_{k-1} -x_k) \\nonumber\\\\\n    & \\quad - \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2\\nonumber\\\\\n    &= (\\nabla f(x_{k-1})-g_k) ^\\top (x-x_{k-1}) + \\frac{1}{\\alpha_k}(x_{k-1} - x_k)^\\top (x-x_{k-1}) + \\frac{1}{2\\alpha_k}\\|x_k-x_{k-1}\\|^2\\nonumber\\\\\n    &= (\\nabla f(x_{k-1})-g_k) ^\\top (x-x_{k-1}) + \\frac{1}{2\\alpha_k} \\left( \\|x-x_k \\|^2-\\|x-x_{k-1} \\|^2\\right),\n\\end{align*}\n","sha256_utf8":"ab91b75807405e1d8b235e32e61d4a1c6db2a99a2ef7e85fc64e1b715494eb7a"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"代入最优点并界定噪声","source_units":[{"start":53555,"end":54172,"start_line":600,"end_line":612,"raw_tex":"and we get the bound for $x=x^*$: \n\\begin{align*}\n    &2\\alpha_k (F(x_k) - F(x^*) ) - 2\\alpha_k (\\nabla f(x_{k-1})-g_k) ^\\top (x_{k-1} -x^*)\n    \\leq \\|x_{k-1} - x^*\\|^2 - \\|x_k - x^* \\|^2.\n\\end{align*}\nRearranging the terms, we get the statement of Lemma \\ref{lem:successful_step_bound1}:\n\\begin{align*}\n    \\|x_k - x^* \\|^2 - \\|x_{k-1} - x^*\\|^2 &\\leq -2\\alpha_k (F(x_k) - F(x^*) ) + 2\\alpha_k (\\nabla f(x_{k-1})-g_k) ^\\top (x_{k-1} -x^*)\\nonumber\\\\\n    &\\leq -2\\alpha_k (F(x_k) - F(x^*) ) + \\lambda_k \\cdot\\|x_{k-1} -x^*\\|,\n\\end{align*}\nwhere $\\lambda_k = \\|2\\alpha_k (\\nabla f(x_{k-1})-g_k)\\|$.\n\n~\\qed \\end{proof}","sha256_utf8":"80b99d1c5e2b92dda46b497e8fae3e845f329eaf4df715ea51c806fa823a6e90"}]},{"id":"S050","semantic_type":"lemma","kind":"lemma","title":"ISTA累计误差界","source_units":[{"start":54460,"end":54833,"start_line":615,"end_line":622,"raw_tex":"\\begin{lemma}\\label{lem:after_K1}\n    After $K$ iterations of the algorithm, the following bound holds: \n\\begin{align*}\n\\sum_{k\\in S, k \\leq K} 2\\alpha_k (F(x_k) - F(x^*) )\n    &\\leq 2\\|x_0 - x^* \\|^2 + \\sum_{k=1}^K \\lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\lambda_i \\right)+ 2\\sum_{k=1}^K \\lambda_k^2,\n\\end{align*}\nwhere $S$ is the set of successful iterations.\n\\end{lemma}","sha256_utf8":"2b17f03dc722745e5efcccf6af7971ad95468ca7f2a58ab0340141f62f4f35d0"}]},{"id":"S051","semantic_type":"proof","kind":"proof","title":"ISTA误差累积证明","source_units":[{"start":54834,"end":58002,"start_line":623,"end_line":665,"raw_tex":"\\begin{proof}\nFor every $K$, from Lemma \\ref{lem:successful_step_bound1} we have\n\\begin{align*}\n    \\|x_K - x^* \\|^2 - \\|x_0 - x^* \\|^2 \n    &= \\sum_{k=1}^{K} \\left(\\|x_{k} - x^* \\|^2 -\\|x_{k-1} - x^* \\|^2 \\right) \\nonumber\\\\\n    &\\leq \\sum_{k\\in S, k \\leq K} \\left[\\lambda_k \\cdot \\|x_{k-1} -x^*\\| - 2\\alpha_k (F(x_k) - F(x^*) ) \\right],\n\\end{align*}\nwhere $S$ is the set of successful iterations. \nHence we arrive at our key bound\n\\begin{align}\\label{eq:thm_key1}\n    &\\|x_K - x^* \\|^2 - \\|x_0 - x^* \\|^2 + \\sum_{k\\in S, k \\leq K} 2\\alpha_k (F(x_k) - F(x^*) )\\leq  \\sum_{k=1}^K \\lambda_k  \\cdot \\| x_{k-1} -x^* \\|.\n\\end{align}\nNow we follow a similar argument as \\cite{Schmidt-Roux-Bach} with two steps. First, we use the key bound \\eqref{eq:thm_key1} to bound the distance to optimal point $u_{k-1} = \\| x_{k-1} -x^* \\|$ in terms of the initial distance $u_{0} = \\| x_{0} -x^* \\|$ and the noise terms $\\lambda_k$. Then we use this bound on $\\| x_{k-1} -x^* \\|$ to substitute into the last sum of the key bound \\eqref{eq:thm_key1} and obtain the final derivation in terms of the noise $\\lambda_k$.\\\\\nDropping the positive term $\\sum_{k\\in S, k \\leq K} 2\\alpha_k (F(x_k) - F(x^*) )$ we get\n\\begin{align*}\n    \\|x_K - x^* \\|^2 \n    &\\leq \\|x_0 - x^* \\|^2 + \\sum_{k=1}^K \\lambda_k \\cdot \\| x_{k-1} -x^* \\| \\nonumber\\\\\n    &= \\|x_0 - x^* \\|^2 + \\lambda_1 \\cdot \\| x_{0} -x^* \\| + \\sum_{k=1}^K \\lambda_{k+1} \\cdot\\| x_{k} -x^* \\|.\n\\end{align*}\nLetting $u_k = \\| x_{k} -x^* \\|$, this is equivalent to \n\\begin{align*}\n    u_K^2 = \\|x_K - x^* \\|^2  \\leq Z + \\sum_{k=1}^K\\lambda_{k+1} u_k.\n\\end{align*}\nwhere $Z = \\|x_0 - x^* \\|^2 + \\lambda_1 \\cdot \\| x_{0} -x^* \\| = u_0^2 + \\lambda_1 u_0$. %\\katya{there seems to be incorrect switching between $k$ and $K$ in a few places here} \nUsing this inequality and applying Lemma 1 in \\cite{Schmidt-Roux-Bach} for the non-negative sequence $\\{u_K\\}$ have\n\\begin{align*}\n     u_{k} \\leq \\frac{1}{2} \\sum_{i=1}^{k} \\lambda_{i+1} + \\left(Z + \\left(\\frac{1}{2} \\sum_{i=1}^{k} \\lambda_{i+1} \\right)^2\\right)^{1/2}\n    \\leq  \\sum_{i=1}^{k} \\lambda_{i+1} + Z^{1/2}  \\leq  \\sum_{i=1}^{k+1} \\lambda_{i} + u_0 ,\n\\end{align*}\nwhere the last line follows from the fact that $Z^{1/2} = \\left(u_0^2 + \\lambda_1  \\cdot u_0 \\right)^{1/2} \\leq u_0 + \\lambda_1.$\n\\\\\nPutting the bound for $u_k$ into the key bound \\eqref{eq:thm_key1} we have\n\\allowdisplaybreaks\n\\begin{align*}\n    &\\|x_K - x^* \\|^2 - \\|x_0 - x^* \\|^2 + \\sum_{k\\in S, k \\leq K} 2\\alpha_k (F(x_k) - F(x^*) )\\nonumber\\\\\n    &\\leq \\sum_{k=1}^K \\lambda_k \\cdot \\| x_{k-1} -x^* \\|\\leq \\sum_{k=1}^K \\lambda_k \\cdot \\left( \\sum_{i=1}^{k} \\lambda_i + \\|x_0 - x^* \\|\\right)\\nonumber\\\\\n    &= \\sum_{k=1}^K \\lambda_k \\cdot \\left( \\sum_{i=1}^{k} \\lambda_i \\right) + \\|x_0 - x^* \\| \\cdot \\sum_{k=1}^K \\lambda_k\\nonumber\\\\\n    &\\leq  \\sum_{k=1}^K \\lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\lambda_i \\right)+ \\sum_{k=1}^K \\lambda_k^2 + \\|x_0 - x^* \\|^2 + \\sum_{k=1}^K \\lambda_k^2 \\nonumber\\\\\n    &= \\|x_0 - x^* \\|^2 + \\sum_{k=1}^K \\lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\lambda_i \\right)+ 2\\sum_{k=1}^K \\lambda_k^2,\n\\end{align*}\nand the statement of Lemma \\ref{lem:after_K1} follows. \n~\\qed\n\\end{proof}","sha256_utf8":"f6a9bb609db4fa7876347a3fe0a85c096c71e62eb0e10892ae84beb48b88318c"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"望远镜求和得关键界","source_units":[{"start":54834,"end":55463,"start_line":623,"end_line":634,"raw_tex":"\\begin{proof}\nFor every $K$, from Lemma \\ref{lem:successful_step_bound1} we have\n\\begin{align*}\n    \\|x_K - x^* \\|^2 - \\|x_0 - x^* \\|^2 \n    &= \\sum_{k=1}^{K} \\left(\\|x_{k} - x^* \\|^2 -\\|x_{k-1} - x^* \\|^2 \\right) \\nonumber\\\\\n    &\\leq \\sum_{k\\in S, k \\leq K} \\left[\\lambda_k \\cdot \\|x_{k-1} -x^*\\| - 2\\alpha_k (F(x_k) - F(x^*) ) \\right],\n\\end{align*}\nwhere $S$ is the set of successful iterations. \nHence we arrive at our key bound\n\\begin{align}\\label{eq:thm_key1}\n    &\\|x_K - x^* \\|^2 - \\|x_0 - x^* \\|^2 + \\sum_{k\\in S, k \\leq K} 2\\alpha_k (F(x_k) - F(x^*) )\\leq  \\sum_{k=1}^K \\lambda_k  \\cdot \\| x_{k-1} -x^* \\|.\n\\end{align}\n","sha256_utf8":"e4e1f447dcb67807d49db84a1d011118e083011455bc0568208b0a5a49bc4433"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"建立距离标量递推","source_units":[{"start":55463,"end":56608,"start_line":635,"end_line":646,"raw_tex":"Now we follow a similar argument as \\cite{Schmidt-Roux-Bach} with two steps. First, we use the key bound \\eqref{eq:thm_key1} to bound the distance to optimal point $u_{k-1} = \\| x_{k-1} -x^* \\|$ in terms of the initial distance $u_{0} = \\| x_{0} -x^* \\|$ and the noise terms $\\lambda_k$. Then we use this bound on $\\| x_{k-1} -x^* \\|$ to substitute into the last sum of the key bound \\eqref{eq:thm_key1} and obtain the final derivation in terms of the noise $\\lambda_k$.\\\\\nDropping the positive term $\\sum_{k\\in S, k \\leq K} 2\\alpha_k (F(x_k) - F(x^*) )$ we get\n\\begin{align*}\n    \\|x_K - x^* \\|^2 \n    &\\leq \\|x_0 - x^* \\|^2 + \\sum_{k=1}^K \\lambda_k \\cdot \\| x_{k-1} -x^* \\| \\nonumber\\\\\n    &= \\|x_0 - x^* \\|^2 + \\lambda_1 \\cdot \\| x_{0} -x^* \\| + \\sum_{k=1}^K \\lambda_{k+1} \\cdot\\| x_{k} -x^* \\|.\n\\end{align*}\nLetting $u_k = \\| x_{k} -x^* \\|$, this is equivalent to \n\\begin{align*}\n    u_K^2 = \\|x_K - x^* \\|^2  \\leq Z + \\sum_{k=1}^K\\lambda_{k+1} u_k.\n\\end{align*}\nwhere $Z = \\|x_0 - x^* \\|^2 + \\lambda_1 \\cdot \\| x_{0} -x^* \\| = u_0^2 + \\lambda_1 u_0$. %\\katya{there seems to be incorrect switching between $k$ and $K$ in a few places here} \n","sha256_utf8":"17d6a3fffbe9ee5a2799fbd436aae3f4e71a73c7478cb0129c3650cbac127abd"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"调用非负序列界","source_units":[{"start":56608,"end":57114,"start_line":647,"end_line":652,"raw_tex":"Using this inequality and applying Lemma 1 in \\cite{Schmidt-Roux-Bach} for the non-negative sequence $\\{u_K\\}$ have\n\\begin{align*}\n     u_{k} \\leq \\frac{1}{2} \\sum_{i=1}^{k} \\lambda_{i+1} + \\left(Z + \\left(\\frac{1}{2} \\sum_{i=1}^{k} \\lambda_{i+1} \\right)^2\\right)^{1/2}\n    \\leq  \\sum_{i=1}^{k} \\lambda_{i+1} + Z^{1/2}  \\leq  \\sum_{i=1}^{k+1} \\lambda_{i} + u_0 ,\n\\end{align*}\nwhere the last line follows from the fact that $Z^{1/2} = \\left(u_0^2 + \\lambda_1  \\cdot u_0 \\right)^{1/2} \\leq u_0 + \\lambda_1.$\n","sha256_utf8":"eb8e24304dd97b04dd5db8f3bab0bb3e517a8d66c28c15b4e414201047d1b9fb"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"回代距离界并整理","source_units":[{"start":57114,"end":58002,"start_line":653,"end_line":665,"raw_tex":"\\\\\nPutting the bound for $u_k$ into the key bound \\eqref{eq:thm_key1} we have\n\\allowdisplaybreaks\n\\begin{align*}\n    &\\|x_K - x^* \\|^2 - \\|x_0 - x^* \\|^2 + \\sum_{k\\in S, k \\leq K} 2\\alpha_k (F(x_k) - F(x^*) )\\nonumber\\\\\n    &\\leq \\sum_{k=1}^K \\lambda_k \\cdot \\| x_{k-1} -x^* \\|\\leq \\sum_{k=1}^K \\lambda_k \\cdot \\left( \\sum_{i=1}^{k} \\lambda_i + \\|x_0 - x^* \\|\\right)\\nonumber\\\\\n    &= \\sum_{k=1}^K \\lambda_k \\cdot \\left( \\sum_{i=1}^{k} \\lambda_i \\right) + \\|x_0 - x^* \\| \\cdot \\sum_{k=1}^K \\lambda_k\\nonumber\\\\\n    &\\leq  \\sum_{k=1}^K \\lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\lambda_i \\right)+ \\sum_{k=1}^K \\lambda_k^2 + \\|x_0 - x^* \\|^2 + \\sum_{k=1}^K \\lambda_k^2 \\nonumber\\\\\n    &= \\|x_0 - x^* \\|^2 + \\sum_{k=1}^K \\lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\lambda_i \\right)+ 2\\sum_{k=1}^K \\lambda_k^2,\n\\end{align*}\nand the statement of Lemma \\ref{lem:after_K1} follows. \n~\\qed\n\\end{proof}","sha256_utf8":"4eedf23518c20e6279b8bb41d55f5b293f18910b66a60aaef3c53115c855b133"}]},{"id":"S056","semantic_type":"theorem","kind":"theorem","title":"ISTA好迭代期望界","source_units":[{"start":59385,"end":59794,"start_line":680,"end_line":688,"raw_tex":"\\begin{theorem}[Upper bound of the number of good steps]\\label{th:bound_expectation_N_TS1} Under Assumption \\ref{assumpt:convex_smooth}, \\ref{assumpt:bounded_below}  ,\nand  Assumption \\ref{assumpt:ista_noise}, we have \n \\begin{align*}\n\\EE [N_G] \n\\leq \\frac{1}{2\\stepsizebar\\epsilon} \\left [2\\|x_0 - x^* \\|^2 + \\frac{4(\\beta + 2)^2}{\\beta^2} + \\frac{8(\\beta+2)}{\\beta+1} \n \\right]. \n\\end{align*}\n\n\\end{theorem}","sha256_utf8":"03847b98b3a1b611da9ca4338f3813a6f6296e80a2a614514f2a81e7a9a16dfd"}]},{"id":"S057","semantic_type":"proof","kind":"proof","title":"ISTA停止噪声估计证明","source_units":[{"start":59795,"end":66453,"start_line":689,"end_line":777,"raw_tex":"\\begin{proof}\nWe recall the definition of the noise term $\\lambda_k =  \\|2\\alpha_{k} (\\nabla f(y_{k} )- g_{k} )\\| $. Let $\\Lambda_k =  \\|2\\Alpha_{k} (\\nabla f(Y_{k} )- G_{k} )\\| $, thus $\\Lambda_k$ is a random variable with realizations  $\\lambda_k$. \nSince the statement of Lemma \\ref{lem:after_K1} holds for every realizations of Algorithm \\ref{alg:ista_bktr_line_search}, for $K = N_\\epsilon -1$ we have:  \n\\begin{align*}\n\\sum_{k\\in S, k \\leq N_\\epsilon -1} 2\\Alpha_k (F(X_k) - F(x^*) )\n    &\\leq 2\\|x_0 - x^* \\|^2 + \\sum_{k=1}^{N_\\epsilon -1} \\Lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\Lambda_i \\right)+ 2\\sum_{k=1}^{N_\\epsilon -1} \\Lambda_k^2,\n\\end{align*}\nwhere $S$ is the set of successful iterations.\nRecall that the stopping time $N_\\epsilon = \\min\\{k = 1,2, \\ldots\\,:  F(X_k) - F^*  \\leq \\epsilon\\}$. Hence for every $k \\leq N_\\epsilon-1$, we have that $ F(X_k) - F^*> \\epsilon$ and\n\\begin{align*}\n    \\epsilon \\cdot \\sum_{k\\in S, k \\leq N_\\epsilon -1} 2\\Alpha_k  &<  \\sum_{k\\in S, k \\leq N_\\epsilon -1} 2\\Alpha_k (F(X_k) - F(x^*) )\\nonumber\\\\\n    &\\leq 2\\|x_0 - x^* \\|^2 + \\sum_{k=1}^{N_\\epsilon} \\Lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\Lambda_i \\right)+ 2\\sum_{k=1}^{N_\\epsilon} \\Lambda_k^2.\n\\end{align*}\nLet $\\mathcal{N}_G$ be the set of \\textit{good} iterations  i.e. true and successful iterations $k$ with $\\alpha_k\\geq \\stepsizebar, k \\leq N_\\epsilon -1 $, then $N_G$ is the cardinality of that set. We have \n\\begin{align*} \n\\sum_{k\\in S, k \\leq N_\\epsilon -1} 2\\Alpha_k \\geq\\sum_{k\\in \\mathcal{N}_G, k \\le N_\\epsilon -1 }  2\\Alpha_k \\geq N_G \\cdot 2\\stepsizebar = 2 N_G \\stepsizebar.\n\\end{align*}\nTherefore \n\\begin{align*}\n    2 N_G \\stepsizebar \\cdot \\epsilon  \n    &\\leq 2\\|x_0 - x^* \\|^2 + \\sum_{k=1}^{N_\\epsilon} \\Lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\Lambda_i \\right)+ 2\\sum_{k=1}^{N_\\epsilon} \\Lambda_k^2,\n\\end{align*}\nand \n\\begin{align*}\n    N_G   \n    &\\leq \\frac{2\\|x_0 - x^* \\|^2 + \\sum_{k=1}^{N_\\epsilon} \\Lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\Lambda_i \\right)+ 2\\sum_{k=1}^{N_\\epsilon} \\Lambda_k^2}{2 \\stepsizebar \\epsilon}.\n\\end{align*}\nTaking expectation we have %\\trang{notations here}\n\\begin{align}\\label{eq:bound1}\n\\EE [N_G ] \n &\\leq \\frac{1}{2\\stepsizebar\\epsilon} \\left [2\\|x_0 - x^* \\|^2 \n+ \\EE \\left[ \\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} \\cdot \\left(\\sum_{i=1}^{k-1} \\Lambda_{i} \\right)  \\right] + 2\\EE \\left[\\sum_{k=1}^{N_\\epsilon}  \\Lambda_{k} ^2  \\right] \n \\right]  .\n\\end{align}\nIn order to bound \\eqref{eq:bound1}, we derive the expected values of $\\Lambda_k^2$, $\\Lambda_k$ and $\\Lambda_k \\Lambda_i$ for $1\\leq i, k \\leq N_\\epsilon$\nusing Assumption \\ref{assumpt:ista_noise}. From \\eqref{eq:ista_noise} we have: \n\\begin{align}\\label{eq:mu_k_squared1}\n    \\EE \\left[\\Lambda_{k}^2| \\pastone\\right] = \\EE \\left[4\\alpha_{k}^2 \\| (\\nabla f(y_{k} )- G_{k} )\\|^2 | \\pastone\\right] \\leq \\frac{4}{k^{2+ \\beta}},\n\\end{align}\nwhich leads to \n\\begin{align*}\n    \\left(\\EE \\left[\\Lambda_{k}| \\pastone\\right]\\right)^2 \\leq \\EE \\left[\\Lambda_{k}^2| \\pastone\\right] \\leq \\frac{4}{k^{2+ \\beta}}.\n\\end{align*}\nTherefore we have the bound for $\\EE \\left[\\Lambda_{k}| \\pastone\\right]$: \n\\begin{align}\\label{eq:mu_k1}\n    \\EE \\left[\\Lambda_{k}| \\pastone\\right] \\leq \\frac{2}{k^{1+ 0.5\\beta}}.\n\\end{align} \nTaking the total expectation of \\eqref{eq:mu_k_squared1} and \\eqref{eq:mu_k1} we have \n\\begin{align*}\n    \\EE \\left[\\Lambda_{k}^2\\right] \\leq \\frac{4}{k^{2+ \\beta}} \\text{ and } \\EE \\left[\\Lambda_{k}\\right] \\leq \\frac{2}{k^{1+ 0.5\\beta}}.\n\\end{align*} \nNow we consider the quantity $\\EE \\left[\\Lambda_{k}\\Lambda_{i}| \\pastone\\right] $ where $i< k $. \nFrom the definition of  the filtration $\\pastone$ and $\\Lambda_{i}$,  {we have that} $\\Lambda_i \\in \\pastone$ for all $i < k$. Therefore by the rules of conditional probability\n\\begin{align*}\n    \\EE \\left[\\Lambda_{k}\\Lambda_{i}| \\pastone\\right] =  \\EE \\left[\\Lambda_{k}| \\pastone\\right] \\Lambda_i .\n\\end{align*}\nHence we can bound the expect value of $\\Lambda_{k}\\Lambda_{i}$ as follows: \n\\begin{align*}\n    \\EE \\left[\\Lambda_{k}\\Lambda_{i}  \\right] \n    &=\\EE \\left[\\EE \\left[\\Lambda_{k}\\Lambda_{i}| \\pastone \\right] \\right]  =\\EE \\left[\\EE \\left[\\Lambda_{k}| \\pastone\\right] \\Lambda_i  \\right] \n    \\nonumber\\\\ \\nonumber\n    &  {\\leq} \\EE \\left[\\EE \\left[\\frac{2}{k^{1+ 0.5\\beta}} \\Lambda_i  \\right]  \\right]   = \\frac{2}{k^{1+ 0.5\\beta}} \\EE \\left[\\EE \\left[ \\Lambda_i | \\pastonei \\right]  \\right]\n    \\nonumber\\\\ \\nonumber\n    &  {\\leq} \\frac{2}{k^{1+ 0.5\\beta}} \\frac{2}{i^{1+ 0.5\\beta}} .\n\\end{align*}\nSince $N_\\epsilon$ is a hitting time and the random variables $\\Lambda_k^2$, $\\Lambda_{k}\\Lambda_{i}$ have finite means and are summable (over $\\{k = 1,2, \\ldots\\}$ and over $\\{i, k = 1,2,\\ldots\\}$ respectively), from the general Wald's identity  {\\cite{Wald}} we have: \n\\begin{align*}\n\\EE \\left[\\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} ^2  \\right] &= \\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\EE \\left[  \\Lambda_{k} ^2  \\right]  \\right)\\nonumber\\\\\n\\EE \\left[ \\sum_{k=1}^{N_\\epsilon}\\Lambda_{k} \\cdot \\left(\\sum_{i=1}^{k-1} \\Lambda_{i} \\right)  \\right] &= \\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\sum_{i=1}^{k-1}\\EE \\left[  \\Lambda_{k} \\Lambda_{i}  \\right] \\right) .\n\\end{align*}\nApplying the previous total expected value bounds in these inequality we have: \n \\begin{align*}\n\\EE \\left[\\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} ^2  \\right] & = \\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\EE \\left[  \\Lambda_{k} ^2  \\right]  \\right)\n\\leq 4\\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\frac{1}{k^{2+ \\beta}}\\right) \\leq  \\frac{4(\\beta +2)}{\\beta+1}\\nonumber\\\\\n \\EE \\left[ \\sum_{k=1}^{N_\\epsilon}\\Lambda_{k} \\left(\\sum_{i=1}^{k-1} \\Lambda_{i} \\right)  \\right] &= \\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\sum_{i=1}^{k-1}\\EE \\left[  \\Lambda_{k} \\Lambda_{i}  \\right] \\right) \\leq 4\\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\frac{1}{k^{1+ 0.5\\beta}} \\sum_{i=1}^{k-1} \\frac{1}{i^{1+ 0.5\\beta}}\\right) \\leq \\frac{4(\\beta + 2)}{\\beta},\n\\end{align*}\nwhere the last computations follows from \n\\begin{align*}\n    &\\sum_{k=1}^{N_\\epsilon}\\frac{1}{k^{2+ \\beta}} < 1 + \\sum_{k=2}^{\\infty} \\frac{1}{k^{2+ \\beta}} < 1+  \\int_{1}^{\\infty} \\frac{1}{x^{2+ \\beta}} dx =  \\frac{\\beta +2}{\\beta+1},\\nonumber\\\\\n    &\\sum_{k=1}^{N_\\epsilon}\\frac{1}{k^{1+ 0.5\\beta}} < 1 + \\sum_{k=2}^{\\infty} \\frac{1}{k^{1+ 0.5\\beta}} < 1+  \\int_{1}^{\\infty} \\frac{1}{x^{1+ 0.5\\beta}} dx = \\frac{\\beta + 2}{\\beta}.\n\\end{align*}\nSubstituting these bounds into the previous inequality \\eqref{eq:bound1}, we have\n\\begin{align*}\n\\EE [N_G ] \n &\\leq \\frac{1}{2\\stepsizebar\\epsilon} \\left [2\\|x_0 - x^* \\|^2 + \\frac{4(\\beta + 2)^2}{\\beta^2} + \\frac{8(\\beta+2)}{\\beta+1} \n \\right]. \n\\end{align*}\nwhich is the statement of Theorem \\ref{th:bound_expectation_N_TS1}.\n~\\qed \\end{proof}","sha256_utf8":"9a922d39e216f693abcfe6e24eb28ddaea43e52133d4cb693d8d242f04267d6b"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"停止累计界与误差阈值","source_units":[{"start":59795,"end":61011,"start_line":689,"end_line":701,"raw_tex":"\\begin{proof}\nWe recall the definition of the noise term $\\lambda_k =  \\|2\\alpha_{k} (\\nabla f(y_{k} )- g_{k} )\\| $. Let $\\Lambda_k =  \\|2\\Alpha_{k} (\\nabla f(Y_{k} )- G_{k} )\\| $, thus $\\Lambda_k$ is a random variable with realizations  $\\lambda_k$. \nSince the statement of Lemma \\ref{lem:after_K1} holds for every realizations of Algorithm \\ref{alg:ista_bktr_line_search}, for $K = N_\\epsilon -1$ we have:  \n\\begin{align*}\n\\sum_{k\\in S, k \\leq N_\\epsilon -1} 2\\Alpha_k (F(X_k) - F(x^*) )\n    &\\leq 2\\|x_0 - x^* \\|^2 + \\sum_{k=1}^{N_\\epsilon -1} \\Lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\Lambda_i \\right)+ 2\\sum_{k=1}^{N_\\epsilon -1} \\Lambda_k^2,\n\\end{align*}\nwhere $S$ is the set of successful iterations.\nRecall that the stopping time $N_\\epsilon = \\min\\{k = 1,2, \\ldots\\,:  F(X_k) - F^*  \\leq \\epsilon\\}$. Hence for every $k \\leq N_\\epsilon-1$, we have that $ F(X_k) - F^*> \\epsilon$ and\n\\begin{align*}\n    \\epsilon \\cdot \\sum_{k\\in S, k \\leq N_\\epsilon -1} 2\\Alpha_k  &<  \\sum_{k\\in S, k \\leq N_\\epsilon -1} 2\\Alpha_k (F(X_k) - F(x^*) )\\nonumber\\\\\n    &\\leq 2\\|x_0 - x^* \\|^2 + \\sum_{k=1}^{N_\\epsilon} \\Lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\Lambda_i \\right)+ 2\\sum_{k=1}^{N_\\epsilon} \\Lambda_k^2.\n\\end{align*}\n","sha256_utf8":"98a6de8bcfcf9768facafd4383ce7f397095762c27696e1842aa9643b254f3ea"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"用好迭代下界步长和","source_units":[{"start":61011,"end":61866,"start_line":702,"end_line":715,"raw_tex":"Let $\\mathcal{N}_G$ be the set of \\textit{good} iterations  i.e. true and successful iterations $k$ with $\\alpha_k\\geq \\stepsizebar, k \\leq N_\\epsilon -1 $, then $N_G$ is the cardinality of that set. We have \n\\begin{align*} \n\\sum_{k\\in S, k \\leq N_\\epsilon -1} 2\\Alpha_k \\geq\\sum_{k\\in \\mathcal{N}_G, k \\le N_\\epsilon -1 }  2\\Alpha_k \\geq N_G \\cdot 2\\stepsizebar = 2 N_G \\stepsizebar.\n\\end{align*}\nTherefore \n\\begin{align*}\n    2 N_G \\stepsizebar \\cdot \\epsilon  \n    &\\leq 2\\|x_0 - x^* \\|^2 + \\sum_{k=1}^{N_\\epsilon} \\Lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\Lambda_i \\right)+ 2\\sum_{k=1}^{N_\\epsilon} \\Lambda_k^2,\n\\end{align*}\nand \n\\begin{align*}\n    N_G   \n    &\\leq \\frac{2\\|x_0 - x^* \\|^2 + \\sum_{k=1}^{N_\\epsilon} \\Lambda_k \\cdot \\left( \\sum_{i=1}^{k-1} \\Lambda_i \\right)+ 2\\sum_{k=1}^{N_\\epsilon} \\Lambda_k^2}{2 \\stepsizebar \\epsilon}.\n\\end{align*}\n","sha256_utf8":"357a47020f9c8834c74400fbf653ea5c2cd6f812e8f42a76ded64c66b0652c58"}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","title":"对好迭代界取期望","source_units":[{"start":61866,"end":62219,"start_line":716,"end_line":722,"raw_tex":"Taking expectation we have %\\trang{notations here}\n\\begin{align}\\label{eq:bound1}\n\\EE [N_G ] \n &\\leq \\frac{1}{2\\stepsizebar\\epsilon} \\left [2\\|x_0 - x^* \\|^2 \n+ \\EE \\left[ \\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} \\cdot \\left(\\sum_{i=1}^{k-1} \\Lambda_{i} \\right)  \\right] + 2\\EE \\left[\\sum_{k=1}^{N_\\epsilon}  \\Lambda_{k} ^2  \\right] \n \\right]  .\n\\end{align}\n","sha256_utf8":"f57d7bcb84fdd672f4228f028590b154a30b0bb0bf8d19dc5aeb0f6ba1fc7283"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"控制噪声一二阶矩","source_units":[{"start":62219,"end":63280,"start_line":723,"end_line":739,"raw_tex":"In order to bound \\eqref{eq:bound1}, we derive the expected values of $\\Lambda_k^2$, $\\Lambda_k$ and $\\Lambda_k \\Lambda_i$ for $1\\leq i, k \\leq N_\\epsilon$\nusing Assumption \\ref{assumpt:ista_noise}. From \\eqref{eq:ista_noise} we have: \n\\begin{align}\\label{eq:mu_k_squared1}\n    \\EE \\left[\\Lambda_{k}^2| \\pastone\\right] = \\EE \\left[4\\alpha_{k}^2 \\| (\\nabla f(y_{k} )- G_{k} )\\|^2 | \\pastone\\right] \\leq \\frac{4}{k^{2+ \\beta}},\n\\end{align}\nwhich leads to \n\\begin{align*}\n    \\left(\\EE \\left[\\Lambda_{k}| \\pastone\\right]\\right)^2 \\leq \\EE \\left[\\Lambda_{k}^2| \\pastone\\right] \\leq \\frac{4}{k^{2+ \\beta}}.\n\\end{align*}\nTherefore we have the bound for $\\EE \\left[\\Lambda_{k}| \\pastone\\right]$: \n\\begin{align}\\label{eq:mu_k1}\n    \\EE \\left[\\Lambda_{k}| \\pastone\\right] \\leq \\frac{2}{k^{1+ 0.5\\beta}}.\n\\end{align} \nTaking the total expectation of \\eqref{eq:mu_k_squared1} and \\eqref{eq:mu_k1} we have \n\\begin{align*}\n    \\EE \\left[\\Lambda_{k}^2\\right] \\leq \\frac{4}{k^{2+ \\beta}} \\text{ and } \\EE \\left[\\Lambda_{k}\\right] \\leq \\frac{2}{k^{1+ 0.5\\beta}}.\n\\end{align*} \n","sha256_utf8":"320ffc460c0b10c00f438d552236d31001e2c94679542350345b45eefe1a4ddc"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"条件期望控制交叉矩","source_units":[{"start":63280,"end":64287,"start_line":740,"end_line":753,"raw_tex":"Now we consider the quantity $\\EE \\left[\\Lambda_{k}\\Lambda_{i}| \\pastone\\right] $ where $i< k $. \nFrom the definition of  the filtration $\\pastone$ and $\\Lambda_{i}$,  {we have that} $\\Lambda_i \\in \\pastone$ for all $i < k$. Therefore by the rules of conditional probability\n\\begin{align*}\n    \\EE \\left[\\Lambda_{k}\\Lambda_{i}| \\pastone\\right] =  \\EE \\left[\\Lambda_{k}| \\pastone\\right] \\Lambda_i .\n\\end{align*}\nHence we can bound the expect value of $\\Lambda_{k}\\Lambda_{i}$ as follows: \n\\begin{align*}\n    \\EE \\left[\\Lambda_{k}\\Lambda_{i}  \\right] \n    &=\\EE \\left[\\EE \\left[\\Lambda_{k}\\Lambda_{i}| \\pastone \\right] \\right]  =\\EE \\left[\\EE \\left[\\Lambda_{k}| \\pastone\\right] \\Lambda_i  \\right] \n    \\nonumber\\\\ \\nonumber\n    &  {\\leq} \\EE \\left[\\EE \\left[\\frac{2}{k^{1+ 0.5\\beta}} \\Lambda_i  \\right]  \\right]   = \\frac{2}{k^{1+ 0.5\\beta}} \\EE \\left[\\EE \\left[ \\Lambda_i | \\pastonei \\right]  \\right]\n    \\nonumber\\\\ \\nonumber\n    &  {\\leq} \\frac{2}{k^{1+ 0.5\\beta}} \\frac{2}{i^{1+ 0.5\\beta}} .\n\\end{align*}\n","sha256_utf8":"9ffa2ef496b1644a933caf79302cd416dad2ca64381ac4c5dba7cb5b124dca20"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"应用停止和恒等式","source_units":[{"start":64287,"end":64947,"start_line":754,"end_line":758,"raw_tex":"Since $N_\\epsilon$ is a hitting time and the random variables $\\Lambda_k^2$, $\\Lambda_{k}\\Lambda_{i}$ have finite means and are summable (over $\\{k = 1,2, \\ldots\\}$ and over $\\{i, k = 1,2,\\ldots\\}$ respectively), from the general Wald's identity  {\\cite{Wald}} we have: \n\\begin{align*}\n\\EE \\left[\\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} ^2  \\right] &= \\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\EE \\left[  \\Lambda_{k} ^2  \\right]  \\right)\\nonumber\\\\\n\\EE \\left[ \\sum_{k=1}^{N_\\epsilon}\\Lambda_{k} \\cdot \\left(\\sum_{i=1}^{k-1} \\Lambda_{i} \\right)  \\right] &= \\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\sum_{i=1}^{k-1}\\EE \\left[  \\Lambda_{k} \\Lambda_{i}  \\right] \\right) .\n\\end{align*}\n","sha256_utf8":"205625c46ae4e23cdb67160681f4f763cc0843f2e4165bd495820659fd256644"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"可和矩界与积分估计","source_units":[{"start":64947,"end":66108,"start_line":759,"end_line":769,"raw_tex":"Applying the previous total expected value bounds in these inequality we have: \n \\begin{align*}\n\\EE \\left[\\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} ^2  \\right] & = \\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\EE \\left[  \\Lambda_{k} ^2  \\right]  \\right)\n\\leq 4\\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\frac{1}{k^{2+ \\beta}}\\right) \\leq  \\frac{4(\\beta +2)}{\\beta+1}\\nonumber\\\\\n \\EE \\left[ \\sum_{k=1}^{N_\\epsilon}\\Lambda_{k} \\left(\\sum_{i=1}^{k-1} \\Lambda_{i} \\right)  \\right] &= \\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\sum_{i=1}^{k-1}\\EE \\left[  \\Lambda_{k} \\Lambda_{i}  \\right] \\right) \\leq 4\\EE \\left(\\sum_{k=1}^{N_\\epsilon}\\frac{1}{k^{1+ 0.5\\beta}} \\sum_{i=1}^{k-1} \\frac{1}{i^{1+ 0.5\\beta}}\\right) \\leq \\frac{4(\\beta + 2)}{\\beta},\n\\end{align*}\nwhere the last computations follows from \n\\begin{align*}\n    &\\sum_{k=1}^{N_\\epsilon}\\frac{1}{k^{2+ \\beta}} < 1 + \\sum_{k=2}^{\\infty} \\frac{1}{k^{2+ \\beta}} < 1+  \\int_{1}^{\\infty} \\frac{1}{x^{2+ \\beta}} dx =  \\frac{\\beta +2}{\\beta+1},\\nonumber\\\\\n    &\\sum_{k=1}^{N_\\epsilon}\\frac{1}{k^{1+ 0.5\\beta}} < 1 + \\sum_{k=2}^{\\infty} \\frac{1}{k^{1+ 0.5\\beta}} < 1+  \\int_{1}^{\\infty} \\frac{1}{x^{1+ 0.5\\beta}} dx = \\frac{\\beta + 2}{\\beta}.\n\\end{align*}\n","sha256_utf8":"7c142b171e1ee569ffc89d1ac7a813b9cc4faf1d86c06e5ddfefbf1a0f53fde1"}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","title":"代入常数完成ISTA界","source_units":[{"start":66108,"end":66453,"start_line":770,"end_line":777,"raw_tex":"Substituting these bounds into the previous inequality \\eqref{eq:bound1}, we have\n\\begin{align*}\n\\EE [N_G ] \n &\\leq \\frac{1}{2\\stepsizebar\\epsilon} \\left [2\\|x_0 - x^* \\|^2 + \\frac{4(\\beta + 2)^2}{\\beta^2} + \\frac{8(\\beta+2)}{\\beta+1} \n \\right]. \n\\end{align*}\nwhich is the statement of Theorem \\ref{th:bound_expectation_N_TS1}.\n~\\qed \\end{proof}","sha256_utf8":"d83d2a1f883f9691ac4fed0140c646dd4b6cd6f3319161cb5d28319fb8d66c45"}]},{"id":"S066","semantic_type":"lemma","kind":"lemma","title":"FISTA步长动量关系","source_units":[{"start":81483,"end":82179,"start_line":916,"end_line":927,"raw_tex":"\\begin{lemma}\\label{lem:fista_linesearch}\n We denote $\\alpha^{succ}_k$ be the step-size parameter  at the successful iteration up to iteration $k$: \n\\begin{align*}\n    \\alpha^{succ}_k = \\alpha_{k'}, \\text{ where } k' = \\max \\{k'' \\text{ successful }, k'' \\leq k\\} .\n\\end{align*}\nWe also adopt the convention that $\\alpha^{succ}_k = \\alpha_0 = \\gamma \\alpha_1$ if there is no successful iteration up to iteration $k$. Then we have the following properties:\n\n1. $\\alpha^{succ}_{k-1} t_{k-1}^2\\geq \\alpha_k t_k(t_{k}-1)$ for every successful iteration $k$.\n\n2. $\\alpha^{succ}_{K} t_{K}^2 \\geq \\left(\\sum_{k \\in S, k \\le K } \\sqrt{\\alpha_k} /2\\right)^2$ for any number of iterations $K$.\n\n\\end{lemma}","sha256_utf8":"4ccaa3e8ecb5a51011daa5f83380a886f5416b22da360ff06dbbbad0cc44dbff"}]},{"id":"S067","semantic_type":"proof","kind":"proof","title":"步长动量关系证明","source_units":[{"start":82182,"end":87089,"start_line":930,"end_line":1002,"raw_tex":"\\begin{proof}\n1. From the update of the algorithm and the fact that $k$ is successful: \n\\begin{align*}\n    t_k = t_k^{next} &=  \\frac{1 + \\sqrt{1+4\\theta_{k-1} t_{k-1}^2}}{2},\n\\end{align*}\nwe have $ \\theta_{k-1} t_{k-1}^2=  t_k(t_{k}-1)$.\nThe statement of the lemma $\\alpha^{succ}_{k-1} t_{k-1}^2\\geq \\alpha_k t_k(t_{k}-1)$ is equivalent to  \n\\begin{align*}\n\\alpha^{succ}_{k-1} \\geq \\alpha_k \\theta_{k-1}.\n\\end{align*}\nLet $k'$ be the previous successful iteration prior to $k$ ($k' < k$), i.e. $\\alpha^{succ}_{k-1} = \\alpha_{k'}$ and $\\alpha^{succ}_{k-1} = \\alpha_{0}$ if there is no successful iteration prior to $k$. By the definition of $k'$ we have \n$\\alpha_{k'+1} = \\tfrac{\\alpha_{k'}}{\\gamma},$ and $\\theta_{k'} = \\gamma $.\nSince the iterations from $k' + 1$ to $k-1$ are unsuccessful (if $k' <k-1$): \n\\begin{align*}\n    \\alpha_{k} &= \\alpha_{k'+1} \\cdot  \\gamma^{k-k'-1}\\nonumber\\\\\n    \\theta_{k-1} &= \\theta_{k'} \\gamma^{k'+1 -k}.\n\\end{align*}\nTherefore \n\\begin{align*}\n    \\alpha_{k} &= \\alpha_{k'+1} \\cdot  \\gamma^{k-k'-1}= \\alpha_{k'} \\cdot  \\gamma^{k-k'-2} \\nonumber\\\\\n    \\theta_{k-1} &= \\theta_{k'} \\gamma^{k'+1 -k} = \\gamma^{k'+2 -k},\n\\end{align*}\nand it proves the statement: \n    \\begin{align}\\label{eq:theta}\n    \\alpha_k \\theta_{k-1} = \\alpha_{k'} \\cdot  \\gamma^{k-k'-2} \\cdot \\gamma^{k'+2 -k} = \\alpha_{k'}. \n    \\end{align}\n    \n\n\n% \\katya{This proof is critical. Is it actually true for all iterations, including unsuccessful ones?}\n\n% \\trang{Actually we need $ \\alpha^{succ}_{K} t_{K}^2 \\geq \\left(\\sum_{k \\in S, k \\le K } \\sqrt{\\alpha_k} /2\\right)^2$ for every $K \\leq  N_\\epsilon-1$. (I forgot the $^{succ}$ before). Maybe the proof only need $K = N_\\epsilon-1$ but this lemma needs to be true for every iteration. We can prove this for all successful iterations, then for unsuccessful one, it still holds since it holds for the prior successful one i.e. no term changes (algo do not update $t_k$). But looking at this, technically $t_k$ do not has at least order $k$ directly, it has the order of the number of good iterations before $k$}\n\n\n2. First, we prove the statement for all iterations $K$ where $K$ is successful. We prove this by induction. \n\\begin{itemize}\n\\item Let $k_1 \\geq 1$ be the first successful iteration of the algorithm. By definition we have \n$\\alpha^{succ}_{k_1} = \\alpha_{k_1}, \\text{ since } k_1 = \\max \\{k'' \\text{ successful }, k'' \\leq k_1\\} $.  \nSince $k_1$ is successful:\n\\begin{align*}\n    t_{k_1} = t_{k_1}^{next} = \\frac{1 + \\sqrt{1+4\\theta_{(k_1-1)} t_{(k_1-1)}^2}}{2}= \\frac{1 + \\sqrt{1+4\\theta_{(k_1-1)} \\cdot 0}}{2} = 1.\n\\end{align*}\nTherefore the statement holds for $k_1$: \n\\begin{align*}\n    \\alpha^{succ}_{k_1} t_{k_1}^2 =  \\alpha_{k_1} \\cdot 1 \\geq \\left( \\sqrt{\\alpha_{k_1}} /2\\right)^2 = \\left(\\sum_{k \\in S, k \\le k_1 } \\sqrt{\\alpha_k} /2\\right)^2.\n\\end{align*}\n\\item Now for an iteration $K$ where $K$ is successful, we assume the statement holds for the successful iteration $k'$ prior to $K$ ($k'$ is the largest index of a successful iteration satisfying $k' < K$). We have \n$ \\alpha^{succ}_{k'} t_{k'}^2 \\geq \\left(\\sum_{k \\in S, k \\le k' } \\sqrt{\\alpha_k} /2\\right)^2$ and  \n\\begin{align*}\n    \\sqrt{\\alpha^{succ}_{k'}} t_{k'} \\geq \\sum_{k \\in S, k \\le k' } \\sqrt{\\alpha_k} /2.\n\\end{align*}\n\nSince both $K$ and $k'$ are successful, $ \\alpha^{succ}_{K} =  \\alpha_{K} $ and $ \\alpha^{succ}_{k'} =  \\alpha_{k'} $.\nUsing the fact that $K$ is successful: \n\\begin{align*}\n    t_{K} &= t_{K}^{next} = \\frac{1 + \\sqrt{1+4\\theta_{(K-1)} t_{(K-1)}^2}}{2} \\geq \\frac{1}{2} +  \\sqrt{\\theta_{(K-1)}}  t_{(K-1)}\\\\\n    &\\geq \\frac{1}{2} +  \\sqrt{\\theta_{(K-1)} } \\cdot t_{k'} \\geq \\frac{1}{2} +  \\sqrt{\\frac{\\alpha_{k'}}{\\alpha_{K}} } \\cdot t_{k'}.\n\\end{align*}\nwhere the last inequality follows from previous argument \\eqref{eq:theta}.\nHence we have \n\\begin{align*}\n    \\sqrt{\\alpha^{succ}_{K}} t_{K} = \\sqrt{\\alpha_{K}} t_{K} \\geq \\sqrt{\\alpha_{k'}} t_{k'} + \\sqrt{\\alpha_K} /2 = \\sqrt{\\alpha^{succ}_{k'}} t_{k'} + \\sqrt{\\alpha_K} /2.\n\\end{align*}\nTherefore \n\\begin{align*}\n    \\sqrt{\\alpha^{succ}_{K}} t_{K} \\geq \\sum_{k \\in S, k \\le k' } \\sqrt{\\alpha_k} /2 + \\sqrt{\\alpha_K} /2 = \\sum_{k \\in S, k \\le K } \\sqrt{\\alpha_k} /2,\n\\end{align*}\nand the statement holds for iteration $K$:\n$ \\alpha^{succ}_{K} t_{K}^2 \\geq \\left(\\sum_{k \\in S, k \\le K } \\sqrt{\\alpha_k} /2\\right)^2$.\n\\item By induction, the statement holds for all the successful iterations. For an arbitrary unsuccessful iteration $K$, the statement holds for the successful iteration $k'$ prior to $K$ and neither right hand side  {nor} left hand side term changes at  unsuccessful iteration. The statement is also true in the case when there is no successful iteration prior to $K$ as both sides equal to 0 ($t_0 = 0$). Therefore it holds for every iteration. \n\\end{itemize}\n% \\katya{what is there is no prior successful iteration? Show that is holds for $K=1$}\n~\\qed \\end{proof}","sha256_utf8":"8c6080ca4704e7f89f0afa40f10560ad689188f4021bb057073aa3d3a6d1a549"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"动量关系归约步长关系","source_units":[{"start":82182,"end":82601,"start_line":930,"end_line":939,"raw_tex":"\\begin{proof}\n1. From the update of the algorithm and the fact that $k$ is successful: \n\\begin{align*}\n    t_k = t_k^{next} &=  \\frac{1 + \\sqrt{1+4\\theta_{k-1} t_{k-1}^2}}{2},\n\\end{align*}\nwe have $ \\theta_{k-1} t_{k-1}^2=  t_k(t_{k}-1)$.\nThe statement of the lemma $\\alpha^{succ}_{k-1} t_{k-1}^2\\geq \\alpha_k t_k(t_{k}-1)$ is equivalent to  \n\\begin{align*}\n\\alpha^{succ}_{k-1} \\geq \\alpha_k \\theta_{k-1}.\n\\end{align*}\n","sha256_utf8":"82ba9e0b2a26e04e830baaef6fc3dbbdc8cdbbcac0ebd84c7b13ec08d3193cb4"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"追踪连续失败后的参数","source_units":[{"start":82601,"end":83528,"start_line":940,"end_line":955,"raw_tex":"Let $k'$ be the previous successful iteration prior to $k$ ($k' < k$), i.e. $\\alpha^{succ}_{k-1} = \\alpha_{k'}$ and $\\alpha^{succ}_{k-1} = \\alpha_{0}$ if there is no successful iteration prior to $k$. By the definition of $k'$ we have \n$\\alpha_{k'+1} = \\tfrac{\\alpha_{k'}}{\\gamma},$ and $\\theta_{k'} = \\gamma $.\nSince the iterations from $k' + 1$ to $k-1$ are unsuccessful (if $k' <k-1$): \n\\begin{align*}\n    \\alpha_{k} &= \\alpha_{k'+1} \\cdot  \\gamma^{k-k'-1}\\nonumber\\\\\n    \\theta_{k-1} &= \\theta_{k'} \\gamma^{k'+1 -k}.\n\\end{align*}\nTherefore \n\\begin{align*}\n    \\alpha_{k} &= \\alpha_{k'+1} \\cdot  \\gamma^{k-k'-1}= \\alpha_{k'} \\cdot  \\gamma^{k-k'-2} \\nonumber\\\\\n    \\theta_{k-1} &= \\theta_{k'} \\gamma^{k'+1 -k} = \\gamma^{k'+2 -k},\n\\end{align*}\nand it proves the statement: \n    \\begin{align}\\label{eq:theta}\n    \\alpha_k \\theta_{k-1} = \\alpha_{k'} \\cdot  \\gamma^{k-k'-2} \\cdot \\gamma^{k'+2 -k} = \\alpha_{k'}. \n    \\end{align}\n","sha256_utf8":"a5c18445b83bdd00b68b5c1410f0fd8bf202ec07e6cf4708dc17a63cfbd861bf"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"验证首个成功步基例","source_units":[{"start":84250,"end":85016,"start_line":964,"end_line":975,"raw_tex":"2. First, we prove the statement for all iterations $K$ where $K$ is successful. We prove this by induction. \n\\begin{itemize}\n\\item Let $k_1 \\geq 1$ be the first successful iteration of the algorithm. By definition we have \n$\\alpha^{succ}_{k_1} = \\alpha_{k_1}, \\text{ since } k_1 = \\max \\{k'' \\text{ successful }, k'' \\leq k_1\\} $.  \nSince $k_1$ is successful:\n\\begin{align*}\n    t_{k_1} = t_{k_1}^{next} = \\frac{1 + \\sqrt{1+4\\theta_{(k_1-1)} t_{(k_1-1)}^2}}{2}= \\frac{1 + \\sqrt{1+4\\theta_{(k_1-1)} \\cdot 0}}{2} = 1.\n\\end{align*}\nTherefore the statement holds for $k_1$: \n\\begin{align*}\n    \\alpha^{succ}_{k_1} t_{k_1}^2 =  \\alpha_{k_1} \\cdot 1 \\geq \\left( \\sqrt{\\alpha_{k_1}} /2\\right)^2 = \\left(\\sum_{k \\in S, k \\le k_1 } \\sqrt{\\alpha_k} /2\\right)^2.\n\\end{align*}\n","sha256_utf8":"1966d79ff2a51f2b71d826ec566797ca1e8a691eb90c279199172dcb6d77a383"}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","title":"建立成功步动量增量","source_units":[{"start":85016,"end":85982,"start_line":976,"end_line":988,"raw_tex":"\\item Now for an iteration $K$ where $K$ is successful, we assume the statement holds for the successful iteration $k'$ prior to $K$ ($k'$ is the largest index of a successful iteration satisfying $k' < K$). We have \n$ \\alpha^{succ}_{k'} t_{k'}^2 \\geq \\left(\\sum_{k \\in S, k \\le k' } \\sqrt{\\alpha_k} /2\\right)^2$ and  \n\\begin{align*}\n    \\sqrt{\\alpha^{succ}_{k'}} t_{k'} \\geq \\sum_{k \\in S, k \\le k' } \\sqrt{\\alpha_k} /2.\n\\end{align*}\n\nSince both $K$ and $k'$ are successful, $ \\alpha^{succ}_{K} =  \\alpha_{K} $ and $ \\alpha^{succ}_{k'} =  \\alpha_{k'} $.\nUsing the fact that $K$ is successful: \n\\begin{align*}\n    t_{K} &= t_{K}^{next} = \\frac{1 + \\sqrt{1+4\\theta_{(K-1)} t_{(K-1)}^2}}{2} \\geq \\frac{1}{2} +  \\sqrt{\\theta_{(K-1)}}  t_{(K-1)}\\\\\n    &\\geq \\frac{1}{2} +  \\sqrt{\\theta_{(K-1)} } \\cdot t_{k'} \\geq \\frac{1}{2} +  \\sqrt{\\frac{\\alpha_{k'}}{\\alpha_{K}} } \\cdot t_{k'}.\n\\end{align*}\nwhere the last inequality follows from previous argument \\eqref{eq:theta}.\n","sha256_utf8":"f78ac5a20d1becedf2c666f288e550cb2997bedab66533375b06bd429fb3fc88"}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","title":"完成成功步归纳","source_units":[{"start":85982,"end":86524,"start_line":989,"end_line":998,"raw_tex":"Hence we have \n\\begin{align*}\n    \\sqrt{\\alpha^{succ}_{K}} t_{K} = \\sqrt{\\alpha_{K}} t_{K} \\geq \\sqrt{\\alpha_{k'}} t_{k'} + \\sqrt{\\alpha_K} /2 = \\sqrt{\\alpha^{succ}_{k'}} t_{k'} + \\sqrt{\\alpha_K} /2.\n\\end{align*}\nTherefore \n\\begin{align*}\n    \\sqrt{\\alpha^{succ}_{K}} t_{K} \\geq \\sum_{k \\in S, k \\le k' } \\sqrt{\\alpha_k} /2 + \\sqrt{\\alpha_K} /2 = \\sum_{k \\in S, k \\le K } \\sqrt{\\alpha_k} /2,\n\\end{align*}\nand the statement holds for iteration $K$:\n$ \\alpha^{succ}_{K} t_{K}^2 \\geq \\left(\\sum_{k \\in S, k \\le K } \\sqrt{\\alpha_k} /2\\right)^2$.\n","sha256_utf8":"f53fa51c53840e966030a1325037eb5ad2b33fcffd50b2a6d9c5012fcac55d87"}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","title":"扩展至失败及空成功情形","source_units":[{"start":86524,"end":87089,"start_line":999,"end_line":1002,"raw_tex":"\\item By induction, the statement holds for all the successful iterations. For an arbitrary unsuccessful iteration $K$, the statement holds for the successful iteration $k'$ prior to $K$ and neither right hand side  {nor} left hand side term changes at  unsuccessful iteration. The statement is also true in the case when there is no successful iteration prior to $K$ as both sides equal to 0 ($t_0 = 0$). Therefore it holds for every iteration. \n\\end{itemize}\n% \\katya{what is there is no prior successful iteration? Show that is holds for $K=1$}\n~\\qed \\end{proof}","sha256_utf8":"4614c2ce959e3a179e90483b1efd6c05af34c9332d3372bac0fab903d0dc52a7"}]},{"id":"S074","semantic_type":"lemma","kind":"lemma","title":"失败步能量不变","source_units":[{"start":88484,"end":88764,"start_line":1017,"end_line":1021,"raw_tex":"\\begin{lemma}[$m_{k}$ on unsuccsessful steps]\\label{lem:unsuccessful_step_bound}\nLet $m_k$ be defined as in \\eqref{eq:define_m}. \nIf iteration $k$ of Algorithm \\ref{alg:fista_bktr_line_search}  is unsuccessful, then we have \n\\begin{align*}m_{k} = m_{k-1}. \\end{align*}\n\\end{lemma}","sha256_utf8":"ccd4257d2d411167887a2059ed558360e1425ca9084107deb31a9a74d8c8b10e"}]},{"id":"S075","semantic_type":"lemma","kind":"lemma","title":"成功步能量递推","source_units":[{"start":89388,"end":90109,"start_line":1029,"end_line":1043,"raw_tex":"\\begin{lemma}[$m_k$ on  successful step]\\label{lem:successful_step_bound}\nUnder Assumption \\ref{assumpt:convex_smooth}, if $k$ is a successful iteration of Algorithm \\ref{alg:fista_bktr_line_search}, then we have:\n %\\katya{I think here and in ISTA analysis it is best to use $b_k$ instead of $b_k$ for realizations and switch to $b_k$ when taking expectation. }\n\\begin{align*}\n&2\\alpha^{succ}_{k}t_k^2 v_k -2\\alpha^{succ}_{k-1} t_{k-1}^2v_{k-1} \\leq \\norm{u_{k-1}}^2 -  \\norm{u_k }^2 + \\lambda_k \\cdot \\| u_{k-1}\\|,\n\\end{align*}\nwhere \n\\begin{align*}\n\\lambda_k =  \\|2\\alpha_kt_k (\\nabla f(y_k )- g_k )\\|.\n\\end{align*}\nEquivalently, \n\\begin{align*}\nm_{k} -m_{k-1} \\leq \\lambda_k \\cdot \\|u_{k-1}\\|.\n\\end{align*}\n\\end{lemma}","sha256_utf8":"95f2e3284a26485c85591785bf5f15d71c5daed0912ca06fa80d5bbdf1687f27"}]},{"id":"S076","semantic_type":"proof","kind":"proof","title":"成功步能量估计证明","source_units":[{"start":90111,"end":97277,"start_line":1045,"end_line":1174,"raw_tex":"\\begin{proof}\nSince $k$ is a successful iteration, we have $F(\\tilde{p}_{\\alpha_k} (y_k)) \\leq \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (y_k), y_k)$. \nAlso $x_k = \\tilde{p}_{\\alpha_k} (y_k) $. %; t_k = t_k^{next}$.\nHence \n\\begin{align*}\n    F(x_k) = f(x_k) + h(x_k) \\leq f(y_k) +(x_k -y_k)^\\top g_k + \\frac{1}{2\\alpha_k}\\|x_k-y_k\\|^2 + h(x_k) .\n\\end{align*}\nBy the  convexity of $f$ at $x$ and $y_k$ and the convexity of $h$ at $x$ and $x_k$:\n\\begin{align*}\nf(x) \\geq f(y_k) + \\nabla f(y_k )^\\top (x-y_k),\n\\nonumber\\\\\nh(x) \\geq h(x_k) + \\gamma(y_k)^\\top (x-x_k),\n\\end{align*}\nwhere the second bound holds for every subgradient $\\gamma(y_k)$ of $h$ at $x_k$. \nWe have \n\\begin{align*}\nF(x) &= f(x) + h(x)\\geq f(y_k)+ h(x_k) + \\nabla f(y_k )^\\top (x-y_k)  + \\gamma(y_k)^\\top (x-x_k)\n\\nonumber\\\\\n&= f(x) + h(x)\\geq f(y_k)+ h(x_k) + [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x-y_k)  + \\gamma(y_k)^\\top ( y_k -x_k) \n\\end{align*}\nCombine this with the previous bound of $F(x_k)$ we have \n\\begin{align*}\n&F(x) - F(x_k) \\nonumber\\\\&\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x-y_k)  + \\gamma(y_k)^\\top ( y_k -x_k) - (x_k -y_k)^\\top g_k - \\frac{1}{2\\alpha_k}\\|x_k-y_k\\|^2\\nonumber\\\\\n&=  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x-y_k)  + (\\gamma(y_k)+g_k)^\\top ( y_k -x_k)  - \\frac{1}{2\\alpha_k}\\|x_k-y_k\\|^2.\n\\end{align*}\nUsing the fact that $x_k = \\tilde{p}_{\\alpha_k} (y_k) =\\arg\\min_y \\tilde{Q}_{\\alpha_k}(y, y_k)$, we get that $ y_k - x_k= \\alpha_k[\\gamma(y_k)+g_k]$ for some subgradient $\\gamma(y_k)$ of $h$ at $x_k$. Therefore\n\\begin{align*}\nF(x) - F(x_k)\n&\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x-y_k)  + \\alpha_k\\|\\gamma(y_k)+g_k) \\|^2 - \\frac{1}{2} \\alpha_k \\|\\gamma(y_k)+g_k)\\|^2\\nonumber\\\\\n&=  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x-y_k)  + \\frac{\\alpha_k}{2}\\|\\gamma(y_k)+g_k) \\|^2 .\n\\end{align*}\nApply this equation for $x = x_{k-1}$ and $x=x^*$ we have \n\\begin{align*}\nF(x_{k-1}) - F(x_k) &\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x_{k-1}-y_k)  + \\frac{\\alpha_k}{2}\\|\\gamma(y_k)+g_k) \\|^2 ,\n\\nonumber\\\\\nF(x^*) - F(x_k) &\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x^*-y_k)   + \\frac{\\alpha_k}{2}\\|\\gamma(y_k)+g_k) \\|^2 .\n\\end{align*}\nLet $v_k =  F(x_k) - F(x^*) $ then \n\\begin{align*}\nv_{k-1} - v_k &\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x_{k-1}-y_k)  + \\frac{\\alpha_k}{2}\\|\\gamma(y_k)+g_k) \\|^2  ,\n\\nonumber\\\\\n-v_k &\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x^*-y_k)   + \\frac{\\alpha_k}{2}\\|\\gamma(y_k)+g_k) \\|^2  .\n\\end{align*}\nMultiplying the first equality by $t_{k}-1$ and adding to the second equality, we get\n\\begin{align*}\n(t_{k}-1)v_{k-1} - t_k v_k &\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k )   + \\frac{\\alpha_k}{2} t_k \\|\\gamma(y_k)+g_k) \\|^2 .\n\\end{align*}\nMultiplying the previous inequality by $t_k$ we have\n \\begin{align*}\n &t_k(t_{k}-1)v_{k-1} - t_k^2 v_k \\nonumber\\\\\n &\\geq t_k [\\nabla f(y_k )+ \\gamma(y_k)]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k )   + \\frac{\\alpha_k}{2}t_k^2\\|\\gamma(y_k)+g_k) \\|^2 \\nonumber\\\\\n&=  t_k [\\nabla f(y_k )- g_k +g_k + \\gamma(y_k)]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k )  + \\frac{\\alpha_k}{2}t_k^2\\|\\gamma(y_k)+g_k) \\|^2\n\\nonumber\\\\\n&=  t_k [g_k + \\gamma(y_k)]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k )  + \\frac{\\alpha_k}{2}t_k^2\\|\\gamma(y_k)+g_k) \\|^2\n \\nonumber\\\\&\n+ t_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) \n\\nonumber\\\\\n&=  t_k \\frac{1}{\\alpha_k}(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k )  + \\frac{\\alpha_k}{2}t_k^2\\|\\gamma(y_k)+g_k) \\|^2\n \\nonumber\\\\&\n+ t_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) \n\\nonumber\\\\\n&=  t_k \n\\frac{1}{\\alpha_k}(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   + t_k \n\\frac{1}{\\alpha_k}(y_k - x_k)^\\top t_k(x_k -y_k)  \\nonumber\\\\&\n+ \\frac{1}{2\\alpha_k}t_k^2\\|x_k -y_k \\|^2\n+ t_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) \n\\nonumber\\\\\n&=  t_k \n\\frac{1}{\\alpha_k}(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n\\frac{1}{2\\alpha_k}\\norm{t_k (x_k -y_k)}^2 \\nonumber\\\\&\n+ t_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) .\n\\end{align*}\nMultiplying the final statement by $2\\alpha_k $ we get \n \\begin{align}\\label{eq_bound_mu_t}\n&~2\\alpha_k t_k(t_{k}-1)v_{k-1} - 2\\alpha_kt_k^2 v_k \\nonumber\\\\\n&\\geq 2t_k \n(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n\\norm{t_k (x_k -y_k)}^2  \\nonumber\\\\\n&  + 2\\alpha_kt_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) .\n\\end{align}\nNow we consider the following term: $2t_k \n(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n\\norm{t_k (x_k -y_k)}^2$.\n% \\begin{align*}\n% 2t_k \n% (y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n% \\norm{t_k (x_k -y_k)}^2.\n% \\end{align*}\nLet $a  = t_k \n(y_k - x_k), b = (t_{k}-1)x_{k-1} + x^* - t_k x_k$ we have \n\\begin{align}\\label{eq:03}\n    & \\quad \\ 2t_k \n(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n\\norm{t_k (x_k -y_k)}^2\\nonumber\\\\ &=  2ab - a^2 \\nonumber\\\\\n&= b^2 -b^2 +2ab - a^2= b^2 -(a-b)^2\\nonumber\\\\\n&=\\norm{(t_{k}-1)x_{k-1} + x^* - t_k x_k }^2 -\\norm{(t_{k}-1)x_{k-1} + x^* - t_k x_k  -t_k (y_k - x_k)}^2\\nonumber\\\\\n&=\\norm{(t_{k}-1)x_{k-1} + x^* - t_k x_k }^2 -\\norm{(t_{k}-1)x_{k-1} + x^* - t_k y_k  }^2.\n\\end{align}\nSince $k$ is a successful iteration, we have that $x^{prev}_{k} = x_{k-1}$. From the definition of $u_k$\n\\begin{align*}u_k = (t_{k}-1)x^{prev}_{k} + x^* - t_k x_k, \\end{align*} \nwe have that \n\\begin{align*}u_k = (t_{k}-1)x_{k-1} + x^* - t_k x_k. \\end{align*} \nNow from the definition of FISTAStep \\eqref{eq:fista_step}\n\\begin{align*}y_k = x_{k-1} + \\frac{t_{k-1}-1}{t_{k}} ( x_{k-1} - x^{prev}_{k-1} ),\\end{align*}\nwe have \n\\begin{align*}t_k y_k = t_kx_{k-1} + (t_{k-1}-1) ( x_{k-1} - x^{prev}_{k-1} ).\\end{align*}\nCombining these equations with \\eqref{eq:03} we get: \n\\begin{align*}\n& \\quad \\ 2t_k \n(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n\\norm{t_k (x_k -y_k)}^2\\nonumber\\\\\n&=\\norm{(t_{k}-1)x_{k-1} + x^* - t_k x_k }^2 -\\norm{(t_{k}-1)x_{k-1} + x^* - t_k y_k  }^2\\nonumber\\\\\n&=\\norm{u_k }^2 -\\norm{(t_{k}-1)x_{k-1} + x^* - t_kx_{k-1} - (t_{k-1}-1) ( x_{k-1} -  x^{prev}_{k-1} )  }^2\\nonumber\\\\\n&=\\norm{u_k }^2 -\\norm{-t_{k-1} x_{k-1} + x^*  + (t_{k-1}-1)  x^{prev}_{k-1}  }^2\\nonumber\\\\\n&=\\norm{u_k }^2 -\\norm{-u_{k-1}}^2.\n\\end{align*}\nSubstituting back to previous equality \\eqref{eq_bound_mu_t} we get \n \\begin{align*}\n&2\\alpha_k t_k(t_{k}-1)v_{k-1} - 2\\alpha_kt_k^2 v_k \\nonumber\\\\\n&\\geq \\norm{u_k }^2 -\\norm{u_{k-1}}^2  + 2\\alpha_kt_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) .\n\\end{align*}\nTherefore  \n  \\begin{align*}\n&2\\alpha_kt_k^2 v_k -2\\alpha_k t_k(t_{k}-1)v_{k-1}\\nonumber\\\\ % &\\leq \\norm{u_{k-1}}^2 -  \\norm{u_k }^2 -2\\alpha_kt_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) \\nonumber\\\\\n&\\leq \\norm{u_{k-1}}^2 -  \\norm{u_k }^2 + \\|2\\alpha_kt_k (\\nabla f(y_k )- g_k )\\| \\cdot \\| u_{k-1}\\|.\n\\end{align*}\nFrom Lemma \\ref{lem:fista_linesearch} we have $\\alpha^{succ}_{k-1} t_{k-1}^2\\geq \\alpha_k t_k(t_{k}-1)$\nfor every successful iteration $k$. Also since $k$ is successful, we have  $\\alpha_k = \\alpha^{succ}_{k}$.\nTherefore we have the final bound of Lemma \\ref{lem:successful_step_bound}.\n~\\qed \\end{proof}","sha256_utf8":"ae7e744e86768b9d54259b1226051dc5da1761d3b601e023a5ff81aeb673b83c"}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","title":"成功条件给出目标上界","source_units":[{"start":90111,"end":90470,"start_line":1045,"end_line":1051,"raw_tex":"\\begin{proof}\nSince $k$ is a successful iteration, we have $F(\\tilde{p}_{\\alpha_k} (y_k)) \\leq \\tilde{Q}_{\\alpha_k}(\\tilde{p}_{\\alpha_k} (y_k), y_k)$. \nAlso $x_k = \\tilde{p}_{\\alpha_k} (y_k) $. %; t_k = t_k^{next}$.\nHence \n\\begin{align*}\n    F(x_k) = f(x_k) + h(x_k) \\leq f(y_k) +(x_k -y_k)^\\top g_k + \\frac{1}{2\\alpha_k}\\|x_k-y_k\\|^2 + h(x_k) .\n\\end{align*}\n","sha256_utf8":"f02d18cd945494a51e371f6a21455a4b10c5dd4c976714caa1d7b7abfa7e3a0a"}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","title":"凸性给出比较点下界","source_units":[{"start":90470,"end":91029,"start_line":1052,"end_line":1064,"raw_tex":"By the  convexity of $f$ at $x$ and $y_k$ and the convexity of $h$ at $x$ and $x_k$:\n\\begin{align*}\nf(x) \\geq f(y_k) + \\nabla f(y_k )^\\top (x-y_k),\n\\nonumber\\\\\nh(x) \\geq h(x_k) + \\gamma(y_k)^\\top (x-x_k),\n\\end{align*}\nwhere the second bound holds for every subgradient $\\gamma(y_k)$ of $h$ at $x_k$. \nWe have \n\\begin{align*}\nF(x) &= f(x) + h(x)\\geq f(y_k)+ h(x_k) + \\nabla f(y_k )^\\top (x-y_k)  + \\gamma(y_k)^\\top (x-x_k)\n\\nonumber\\\\\n&= f(x) + h(x)\\geq f(y_k)+ h(x_k) + [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x-y_k)  + \\gamma(y_k)^\\top ( y_k -x_k) \n\\end{align*}\n","sha256_utf8":"86a64dc48d2debd337449d924bb285fa46f1fa0f38e93a3a3a3b44c5ebe70d9d"}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","title":"近端最优性化简目标差","source_units":[{"start":91029,"end":91902,"start_line":1065,"end_line":1075,"raw_tex":"Combine this with the previous bound of $F(x_k)$ we have \n\\begin{align*}\n&F(x) - F(x_k) \\nonumber\\\\&\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x-y_k)  + \\gamma(y_k)^\\top ( y_k -x_k) - (x_k -y_k)^\\top g_k - \\frac{1}{2\\alpha_k}\\|x_k-y_k\\|^2\\nonumber\\\\\n&=  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x-y_k)  + (\\gamma(y_k)+g_k)^\\top ( y_k -x_k)  - \\frac{1}{2\\alpha_k}\\|x_k-y_k\\|^2.\n\\end{align*}\nUsing the fact that $x_k = \\tilde{p}_{\\alpha_k} (y_k) =\\arg\\min_y \\tilde{Q}_{\\alpha_k}(y, y_k)$, we get that $ y_k - x_k= \\alpha_k[\\gamma(y_k)+g_k]$ for some subgradient $\\gamma(y_k)$ of $h$ at $x_k$. Therefore\n\\begin{align*}\nF(x) - F(x_k)\n&\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x-y_k)  + \\alpha_k\\|\\gamma(y_k)+g_k) \\|^2 - \\frac{1}{2} \\alpha_k \\|\\gamma(y_k)+g_k)\\|^2\\nonumber\\\\\n&=  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x-y_k)  + \\frac{\\alpha_k}{2}\\|\\gamma(y_k)+g_k) \\|^2 .\n\\end{align*}\n","sha256_utf8":"fea2dbf3844eddfdc1202705cddf456f11b7c8f814328c7f440219ae5b1f051a"}]},{"id":"S080","semantic_type":"proof_step","kind":"proof_step","title":"选取前次迭代与最优点","source_units":[{"start":91902,"end":92540,"start_line":1076,"end_line":1087,"raw_tex":"Apply this equation for $x = x_{k-1}$ and $x=x^*$ we have \n\\begin{align*}\nF(x_{k-1}) - F(x_k) &\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x_{k-1}-y_k)  + \\frac{\\alpha_k}{2}\\|\\gamma(y_k)+g_k) \\|^2 ,\n\\nonumber\\\\\nF(x^*) - F(x_k) &\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x^*-y_k)   + \\frac{\\alpha_k}{2}\\|\\gamma(y_k)+g_k) \\|^2 .\n\\end{align*}\nLet $v_k =  F(x_k) - F(x^*) $ then \n\\begin{align*}\nv_{k-1} - v_k &\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x_{k-1}-y_k)  + \\frac{\\alpha_k}{2}\\|\\gamma(y_k)+g_k) \\|^2  ,\n\\nonumber\\\\\n-v_k &\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top (x^*-y_k)   + \\frac{\\alpha_k}{2}\\|\\gamma(y_k)+g_k) \\|^2  .\n\\end{align*}\n","sha256_utf8":"1f7042d97505cc2368ef269b52ccec25497bf8aaf6aef5a2ce7e3dd8a79163aa"}]},{"id":"S081","semantic_type":"proof_step","kind":"proof_step","title":"动量加权并分离噪声","source_units":[{"start":92540,"end":94170,"start_line":1088,"end_line":1116,"raw_tex":"Multiplying the first equality by $t_{k}-1$ and adding to the second equality, we get\n\\begin{align*}\n(t_{k}-1)v_{k-1} - t_k v_k &\\geq  [\\nabla f(y_k )+ \\gamma(y_k)]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k )   + \\frac{\\alpha_k}{2} t_k \\|\\gamma(y_k)+g_k) \\|^2 .\n\\end{align*}\nMultiplying the previous inequality by $t_k$ we have\n \\begin{align*}\n &t_k(t_{k}-1)v_{k-1} - t_k^2 v_k \\nonumber\\\\\n &\\geq t_k [\\nabla f(y_k )+ \\gamma(y_k)]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k )   + \\frac{\\alpha_k}{2}t_k^2\\|\\gamma(y_k)+g_k) \\|^2 \\nonumber\\\\\n&=  t_k [\\nabla f(y_k )- g_k +g_k + \\gamma(y_k)]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k )  + \\frac{\\alpha_k}{2}t_k^2\\|\\gamma(y_k)+g_k) \\|^2\n\\nonumber\\\\\n&=  t_k [g_k + \\gamma(y_k)]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k )  + \\frac{\\alpha_k}{2}t_k^2\\|\\gamma(y_k)+g_k) \\|^2\n \\nonumber\\\\&\n+ t_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) \n\\nonumber\\\\\n&=  t_k \\frac{1}{\\alpha_k}(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k )  + \\frac{\\alpha_k}{2}t_k^2\\|\\gamma(y_k)+g_k) \\|^2\n \\nonumber\\\\&\n+ t_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) \n\\nonumber\\\\\n&=  t_k \n\\frac{1}{\\alpha_k}(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   + t_k \n\\frac{1}{\\alpha_k}(y_k - x_k)^\\top t_k(x_k -y_k)  \\nonumber\\\\&\n+ \\frac{1}{2\\alpha_k}t_k^2\\|x_k -y_k \\|^2\n+ t_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) \n\\nonumber\\\\\n&=  t_k \n\\frac{1}{\\alpha_k}(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n\\frac{1}{2\\alpha_k}\\norm{t_k (x_k -y_k)}^2 \\nonumber\\\\&\n+ t_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) .\n\\end{align*}\n","sha256_utf8":"d7a4c22d6fbc8af3db7f75d9b5b5ff6f6603d67380c75bf2b8a3f3cee0a896d7"}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","title":"缩放得到能量比较式","source_units":[{"start":94170,"end":94531,"start_line":1117,"end_line":1124,"raw_tex":"Multiplying the final statement by $2\\alpha_k $ we get \n \\begin{align}\\label{eq_bound_mu_t}\n&~2\\alpha_k t_k(t_{k}-1)v_{k-1} - 2\\alpha_kt_k^2 v_k \\nonumber\\\\\n&\\geq 2t_k \n(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n\\norm{t_k (x_k -y_k)}^2  \\nonumber\\\\\n&  + 2\\alpha_kt_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) .\n\\end{align}\n","sha256_utf8":"0b3ad7666b5a4493e2c9b42eb0407421c5d0bc923487c02852d3d01054331df2"}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","title":"内积恒等式化平方差","source_units":[{"start":94531,"end":94659,"start_line":1125,"end_line":1127,"raw_tex":"Now we consider the following term: $2t_k \n(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n\\norm{t_k (x_k -y_k)}^2$.\n","sha256_utf8":"2aae4c89b34247d1e03d22aaeca7a692de42aea73a8c1635dcf5174675e7aa4d"},{"start":94787,"end":95297,"start_line":1133,"end_line":1142,"raw_tex":"Let $a  = t_k \n(y_k - x_k), b = (t_{k}-1)x_{k-1} + x^* - t_k x_k$ we have \n\\begin{align}\\label{eq:03}\n    & \\quad \\ 2t_k \n(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n\\norm{t_k (x_k -y_k)}^2\\nonumber\\\\ &=  2ab - a^2 \\nonumber\\\\\n&= b^2 -b^2 +2ab - a^2= b^2 -(a-b)^2\\nonumber\\\\\n&=\\norm{(t_{k}-1)x_{k-1} + x^* - t_k x_k }^2 -\\norm{(t_{k}-1)x_{k-1} + x^* - t_k x_k  -t_k (y_k - x_k)}^2\\nonumber\\\\\n&=\\norm{(t_{k}-1)x_{k-1} + x^* - t_k x_k }^2 -\\norm{(t_{k}-1)x_{k-1} + x^* - t_k y_k  }^2.\n\\end{align}\n","sha256_utf8":"025af88010f5e0a86268e2b884345708d73dfae89a9ef9711fc9927948c4d871"}]},{"id":"S084","semantic_type":"proof_step","kind":"proof_step","title":"外推公式识别误差向量","source_units":[{"start":95297,"end":96353,"start_line":1143,"end_line":1160,"raw_tex":"Since $k$ is a successful iteration, we have that $x^{prev}_{k} = x_{k-1}$. From the definition of $u_k$\n\\begin{align*}u_k = (t_{k}-1)x^{prev}_{k} + x^* - t_k x_k, \\end{align*} \nwe have that \n\\begin{align*}u_k = (t_{k}-1)x_{k-1} + x^* - t_k x_k. \\end{align*} \nNow from the definition of FISTAStep \\eqref{eq:fista_step}\n\\begin{align*}y_k = x_{k-1} + \\frac{t_{k-1}-1}{t_{k}} ( x_{k-1} - x^{prev}_{k-1} ),\\end{align*}\nwe have \n\\begin{align*}t_k y_k = t_kx_{k-1} + (t_{k-1}-1) ( x_{k-1} - x^{prev}_{k-1} ).\\end{align*}\nCombining these equations with \\eqref{eq:03} we get: \n\\begin{align*}\n& \\quad \\ 2t_k \n(y_k - x_k)^\\top ((t_{k}-1)x_{k-1} + x^* - t_k x_k )   -  \n\\norm{t_k (x_k -y_k)}^2\\nonumber\\\\\n&=\\norm{(t_{k}-1)x_{k-1} + x^* - t_k x_k }^2 -\\norm{(t_{k}-1)x_{k-1} + x^* - t_k y_k  }^2\\nonumber\\\\\n&=\\norm{u_k }^2 -\\norm{(t_{k}-1)x_{k-1} + x^* - t_kx_{k-1} - (t_{k-1}-1) ( x_{k-1} -  x^{prev}_{k-1} )  }^2\\nonumber\\\\\n&=\\norm{u_k }^2 -\\norm{-t_{k-1} x_{k-1} + x^*  + (t_{k-1}-1)  x^{prev}_{k-1}  }^2\\nonumber\\\\\n&=\\norm{u_k }^2 -\\norm{-u_{k-1}}^2.\n\\end{align*}\n","sha256_utf8":"863ba567c160a359978e29687be0adc214c65e0b5a1235f48d9330c65446ae00"}]},{"id":"S085","semantic_type":"proof_step","kind":"proof_step","title":"噪声范数与动量界收束","source_units":[{"start":96353,"end":97277,"start_line":1161,"end_line":1174,"raw_tex":"Substituting back to previous equality \\eqref{eq_bound_mu_t} we get \n \\begin{align*}\n&2\\alpha_k t_k(t_{k}-1)v_{k-1} - 2\\alpha_kt_k^2 v_k \\nonumber\\\\\n&\\geq \\norm{u_k }^2 -\\norm{u_{k-1}}^2  + 2\\alpha_kt_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) .\n\\end{align*}\nTherefore  \n  \\begin{align*}\n&2\\alpha_kt_k^2 v_k -2\\alpha_k t_k(t_{k}-1)v_{k-1}\\nonumber\\\\ % &\\leq \\norm{u_{k-1}}^2 -  \\norm{u_k }^2 -2\\alpha_kt_k [\\nabla f(y_k )- g_k ]^\\top ((t_{k}-1)x_{k-1} + x^* - t_k y_k ) \\nonumber\\\\\n&\\leq \\norm{u_{k-1}}^2 -  \\norm{u_k }^2 + \\|2\\alpha_kt_k (\\nabla f(y_k )- g_k )\\| \\cdot \\| u_{k-1}\\|.\n\\end{align*}\nFrom Lemma \\ref{lem:fista_linesearch} we have $\\alpha^{succ}_{k-1} t_{k-1}^2\\geq \\alpha_k t_k(t_{k}-1)$\nfor every successful iteration $k$. Also since $k$ is successful, we have  $\\alpha_k = \\alpha^{succ}_{k}$.\nTherefore we have the final bound of Lemma \\ref{lem:successful_step_bound}.\n~\\qed \\end{proof}","sha256_utf8":"a95dad3d1776df02d1266068d43bca31edb3af6eaab77eca43100e7a791b910e"}]},{"id":"S086","semantic_type":"lemma","kind":"lemma","title":"FISTA累计误差界","source_units":[{"start":97927,"end":98293,"start_line":1181,"end_line":1186,"raw_tex":"\\begin{lemma}\\label{lem:after_K}\nGiven any integer $K$, after $K$ iterations of the algorithm, the following bound holds: \n\\begin{align*}\n    2\\alpha^{succ}_{K} t_{K}^2v_{K} \\leq 3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\lambda_{i} \\right]  + 2\\sum_{k=1}^{K} \\lambda_{k} ^2  .\n\\end{align*}\n\\end{lemma}","sha256_utf8":"ea1f6f54ffd9207ba5d86885f2e0307003e0d7375418d88191379475ef38555e"}]},{"id":"S087","semantic_type":"proof","kind":"proof","title":"FISTA误差累积证明","source_units":[{"start":98295,"end":102300,"start_line":1188,"end_line":1242,"raw_tex":"\\begin{proof}\nFor every $K$, from Lemma \\ref{lem:unsuccessful_step_bound}  we have\n\\begin{align*}\n    m_K - m_0 = \\sum_{k \\in S, k\\leq K} (m_{k} -m_{k-1}),\n\\end{align*}\nwhere $S$ is the set of successful iterations.\nWe apply Lemma \\ref{lem:successful_step_bound} for successful iterations and then \n\\begin{align*}\n    m_K - m_0 &= \\sum_{k \\in S, k\\leq K} (m_{k} -m_{k-1})\\leq \\sum_{k \\in S, k\\leq K} \\lambda_k \\cdot \\|u_{k-1}\\| \\leq \\sum_{k=1}^{K} \\lambda_k \\cdot \\| u_{k-1}  \\|  .\n\\end{align*}\nPlugging this and the definition of $m_k$ back to the previous inequality, we have our key bound: \n\\begin{align}\\label{eq:thm_key}\n    2\\alpha^{succ}_{K} t_{K}^2v_{K} +\\norm{u_K }^2 \n    &\\leq \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2\\right) + \\sum_{k=1}^{K} \\lambda_k \\cdot \\| u_{k-1}\\| .\n\\end{align}\nSimilar to the proof of ISTA, \nwe follow the argument in \\cite{Schmidt-Roux-Bach} with two steps. First, we use the key bound \\eqref{eq:thm_key} to bound $\\| u_{k-1}\\|$ in terms of the noise terms $\\lambda_k$. Then we use this bound on $\\| u_{k-1}\\|$ to substitute into the last sum of the key bound \\eqref{eq:thm_key} and obtain the final derivation in terms of the noise $\\lambda_k$.\nIgnoring the positive term $2\\alpha^{succ}_{K} t_{K}^2v_{K}$, we have \n\\begin{align*}\n    \\norm{u_K }^2 \n    &\\leq \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2\\right) + \\sum_{k=1}^K \\lambda_k \\cdot \\| u_{k-1}\\| \\nonumber\\\\\n    &= 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\lambda_1 \\cdot \\| u_{0}\\| + \\sum_{k=1}^{K} \\lambda_{k+1} \\cdot \\| u_{k}\\|.\n\\end{align*}\nThis is equivalent to \n\\begin{align*}\n    \\norm{u_K }^2 \n    \\leq Z + \\sum_{k=1}^{K} \\lambda_{k+1} \\| u_{k}\\|,\n\\end{align*}\nwhere $Z = 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\lambda_1 \\| u_{0}\\|$.\nUsing this inequality and applying Lemma 1 in \\cite{Schmidt-Roux-Bach} for the non-negative sequence $\\{\\|u_K\\|\\}$, we have\n\\allowdisplaybreaks\n\\begin{align*}\n    \\| u_{k}\\| &\\leq \\frac{1}{2} \\sum_{i=1}^{k} \\lambda_{i+1} + \\left( Z + \\left(\\frac{1}{2} \\sum_{i=1}^{k} \\lambda_{i+1} \\right)^2\\right)^{1/2}\n    \\leq  \\sum_{i=1}^{k} \\lambda_{i+1} +  Z^{1/2} \\nonumber\\\\\n    &\\leq  \\sum_{i=1}^{k+1} \\lambda_{i} + \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  \\right)^{1/2},\n\\end{align*}\nwhere the last line follows from the fact that\n\\begin{align*}\n    Z^{1/2} &= \\sqrt{2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\lambda_1 \\| u_{0}\\|  } \\leq  \\sqrt{2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\lambda_1^2 +\\| u_{0}\\|^2  } \\nonumber\\\\\n    &\\leq \\sqrt{2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\lambda_1^2  } \\leq  \\lambda_1 + \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  \\right)^{1/2}.\n\\end{align*}\nDropping the term $\\norm{u_K }^2 $ in \nthe inequality \\eqref{eq:thm_key} and using the bound on $\\| u_{k}\\|$ we have: \n\\allowdisplaybreaks\n\\begin{align*}\n    &2\\alpha^{succ}_{K} t_{K}^2v_{K} \\nonumber\\\\\n    &\\leq 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\| u_{k-1}\\| \\nonumber\\\\\n    &\\leq 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\left[\\sum_{i=1}^{k} \\lambda_{i} + \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  \\right)^{1/2} \\right] \\nonumber\\\\\n    &= 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\left[\\sum_{i=1}^{k} \\lambda_{i} \\right] + \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  \\right)^{1/2} \\sum_{k=1}^{K} \\lambda_{k}   \\nonumber\\\\\n    &\\leq 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\lambda_{i} \\right]  + \\sum_{k=1}^{K} \\lambda_{k} ^2+  \\frac{1}{2} \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  \\right)  + \\sum_{k=1}^{K} \\lambda_{k} ^2\\nonumber\\\\\n    &= 3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\lambda_{i} \\right]  + 2\\sum_{k=1}^{K} \\lambda_{k} ^2,  \n\\end{align*}\nwhich is our desired bound. \n~\\qed \n\\end{proof}","sha256_utf8":"d563232a7b948fd70177d1025cd2b587dd95ce803dac8dd9c98249b656234e50"}]},{"id":"S088","semantic_type":"proof_step","kind":"proof_step","title":"两类迭代望远镜求和","source_units":[{"start":98295,"end":99103,"start_line":1188,"end_line":1202,"raw_tex":"\\begin{proof}\nFor every $K$, from Lemma \\ref{lem:unsuccessful_step_bound}  we have\n\\begin{align*}\n    m_K - m_0 = \\sum_{k \\in S, k\\leq K} (m_{k} -m_{k-1}),\n\\end{align*}\nwhere $S$ is the set of successful iterations.\nWe apply Lemma \\ref{lem:successful_step_bound} for successful iterations and then \n\\begin{align*}\n    m_K - m_0 &= \\sum_{k \\in S, k\\leq K} (m_{k} -m_{k-1})\\leq \\sum_{k \\in S, k\\leq K} \\lambda_k \\cdot \\|u_{k-1}\\| \\leq \\sum_{k=1}^{K} \\lambda_k \\cdot \\| u_{k-1}  \\|  .\n\\end{align*}\nPlugging this and the definition of $m_k$ back to the previous inequality, we have our key bound: \n\\begin{align}\\label{eq:thm_key}\n    2\\alpha^{succ}_{K} t_{K}^2v_{K} +\\norm{u_K }^2 \n    &\\leq \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2\\right) + \\sum_{k=1}^{K} \\lambda_k \\cdot \\| u_{k-1}\\| .\n\\end{align}\n","sha256_utf8":"d9248225e68204905dabbe5d5b9223d0920bd261638a315f937352e572f1f749"}]},{"id":"S089","semantic_type":"proof_step","kind":"proof_step","title":"归约误差范数递推","source_units":[{"start":99103,"end":100072,"start_line":1203,"end_line":1216,"raw_tex":"Similar to the proof of ISTA, \nwe follow the argument in \\cite{Schmidt-Roux-Bach} with two steps. First, we use the key bound \\eqref{eq:thm_key} to bound $\\| u_{k-1}\\|$ in terms of the noise terms $\\lambda_k$. Then we use this bound on $\\| u_{k-1}\\|$ to substitute into the last sum of the key bound \\eqref{eq:thm_key} and obtain the final derivation in terms of the noise $\\lambda_k$.\nIgnoring the positive term $2\\alpha^{succ}_{K} t_{K}^2v_{K}$, we have \n\\begin{align*}\n    \\norm{u_K }^2 \n    &\\leq \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2\\right) + \\sum_{k=1}^K \\lambda_k \\cdot \\| u_{k-1}\\| \\nonumber\\\\\n    &= 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\lambda_1 \\cdot \\| u_{0}\\| + \\sum_{k=1}^{K} \\lambda_{k+1} \\cdot \\| u_{k}\\|.\n\\end{align*}\nThis is equivalent to \n\\begin{align*}\n    \\norm{u_K }^2 \n    \\leq Z + \\sum_{k=1}^{K} \\lambda_{k+1} \\| u_{k}\\|,\n\\end{align*}\nwhere $Z = 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\lambda_1 \\| u_{0}\\|$.\n","sha256_utf8":"69b461d810cceaca66176e327499d6e0e5a75010121108c2c2fd0bb29e7aa1f9"}]},{"id":"S090","semantic_type":"proof_step","kind":"proof_step","title":"标量引理控制误差范数","source_units":[{"start":100072,"end":101003,"start_line":1217,"end_line":1228,"raw_tex":"Using this inequality and applying Lemma 1 in \\cite{Schmidt-Roux-Bach} for the non-negative sequence $\\{\\|u_K\\|\\}$, we have\n\\allowdisplaybreaks\n\\begin{align*}\n    \\| u_{k}\\| &\\leq \\frac{1}{2} \\sum_{i=1}^{k} \\lambda_{i+1} + \\left( Z + \\left(\\frac{1}{2} \\sum_{i=1}^{k} \\lambda_{i+1} \\right)^2\\right)^{1/2}\n    \\leq  \\sum_{i=1}^{k} \\lambda_{i+1} +  Z^{1/2} \\nonumber\\\\\n    &\\leq  \\sum_{i=1}^{k+1} \\lambda_{i} + \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  \\right)^{1/2},\n\\end{align*}\nwhere the last line follows from the fact that\n\\begin{align*}\n    Z^{1/2} &= \\sqrt{2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\lambda_1 \\| u_{0}\\|  } \\leq  \\sqrt{2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\lambda_1^2 +\\| u_{0}\\|^2  } \\nonumber\\\\\n    &\\leq \\sqrt{2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\lambda_1^2  } \\leq  \\lambda_1 + \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  \\right)^{1/2}.\n\\end{align*}\n","sha256_utf8":"9443417cdd3c5fb43fee56e7d51223e4944b23ddac10361eb80abb43595f2d50"}]},{"id":"S091","semantic_type":"proof_step","kind":"proof_step","title":"回代并整理累计噪声","source_units":[{"start":101003,"end":102300,"start_line":1229,"end_line":1242,"raw_tex":"Dropping the term $\\norm{u_K }^2 $ in \nthe inequality \\eqref{eq:thm_key} and using the bound on $\\| u_{k}\\|$ we have: \n\\allowdisplaybreaks\n\\begin{align*}\n    &2\\alpha^{succ}_{K} t_{K}^2v_{K} \\nonumber\\\\\n    &\\leq 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\| u_{k-1}\\| \\nonumber\\\\\n    &\\leq 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\left[\\sum_{i=1}^{k} \\lambda_{i} + \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  \\right)^{1/2} \\right] \\nonumber\\\\\n    &= 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\left[\\sum_{i=1}^{k} \\lambda_{i} \\right] + \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  \\right)^{1/2} \\sum_{k=1}^{K} \\lambda_{k}   \\nonumber\\\\\n    &\\leq 2\\alpha^{succ}_{0} t_{0}^2v_{0} +\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\lambda_{i} \\right]  + \\sum_{k=1}^{K} \\lambda_{k} ^2+  \\frac{1}{2} \\left(2\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  \\right)  + \\sum_{k=1}^{K} \\lambda_{k} ^2\\nonumber\\\\\n    &= 3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{K} \\lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\lambda_{i} \\right]  + 2\\sum_{k=1}^{K} \\lambda_{k} ^2,  \n\\end{align*}\nwhich is our desired bound. \n~\\qed \n\\end{proof}","sha256_utf8":"7eab27f4ff283c46fdaf8b47df98a5e231de52b6aee7d3340c8c8822964463e5"}]},{"id":"S092","semantic_type":"theorem","kind":"theorem","title":"FISTA好迭代期望界","source_units":[{"start":103307,"end":103761,"start_line":1258,"end_line":1267,"raw_tex":"\\begin{theorem}[Upper bound of the number of large true successful steps]\\label{th:bound_expectation_N_TS}\nUnder Assumptions \\ref{assumpt:convex_smooth}, \\ref{assumpt:bounded_below}\nand  \\ref{assumpt:fista_noise}, we have  \n \\begin{align*}\n \\EE [N_G] \\leq \\sqrt{\n \\frac{2}{\\stepsizebar\\epsilon} \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\frac{4(\\beta + 2)^2}{\\beta^2} + \\frac{8(\\beta+2)}{\\beta+1}  \n \\right] }. \n\\end{align*}\n\n\\end{theorem}","sha256_utf8":"b5ea9fba31bdc2c780b9190e826b4a2d940eff72ffec6dac5fb8682d52b678f9"}]},{"id":"S093","semantic_type":"proof","kind":"proof","title":"FISTA好迭代计数证明","source_units":[{"start":103764,"end":107207,"start_line":1270,"end_line":1316,"raw_tex":"\\begin{proof} \nSince the statement of Lemma \\ref{lem:after_K} holds for every realizations of Algorithm \\ref{alg:fista_bktr_line_search}, for $K = N_\\epsilon -1$ we have: \n\\begin{align*}\n    2\\Alpha^{succ}_{N_\\epsilon -1} T_{N_\\epsilon -1}^2 (F(X_{N_\\epsilon -1}) - F^*)\\leq 3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{N_\\epsilon -1} \\Lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\Lambda_{i} \\right]  + 2\\sum_{k=1}^{N_\\epsilon -1} \\Lambda_{k} ^2.\n\\end{align*}\nRecall that $N_\\epsilon$ be the stopping time that $N_\\epsilon = \\inf\\{k: F(X_k) - F^* \\leq \\epsilon\\}$. Hence for $K = N_\\epsilon -1$, we have that $F(X_K) - F^* > \\epsilon$ and \n\\begin{align*}\n    2\\Alpha^{succ}_{N_\\epsilon -1} T_{N_\\epsilon -1}^2 &\\leq \\frac{3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{N_\\epsilon -1} \\Lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\Lambda_{i} \\right]  + 2\\sum_{k=1}^{N_\\epsilon -1} \\Lambda_{k}^2}{F(X_{N_\\epsilon -1}) - F^*} \\nonumber\\\\\n    &\\leq \\frac{3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\Lambda_{i} \\right]  + 2\\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} ^2}{\\epsilon}.\n\\end{align*}\nLet $\\mathcal{N}_G$ be the set of \\textit{good} iterations  (i.e. true and successful iterations $k$ with $\\alpha_k\\geq \\stepsizebar $), with $ k \\leq N_\\epsilon -1$, then $N_G$ is the cardinality of that set. \nUsing the inequality $ \\alpha^{succ}_{K} t_{K}^2 \\geq \\left(\\sum_{k \\in S, k \\le K } \\sqrt{\\alpha_k} /2\\right)^2$ for $K =  N_\\epsilon-1$ from Lemma \\ref{lem:fista_linesearch} we have \n\\begin{align*} \n\\Alpha^{succ}_{N_\\epsilon -1} T_{N_\\epsilon -1}^2 \n&\\geq \\left(\\sum_{k \\in S, k \\le N_\\epsilon -1} \\sqrt{\\Alpha_k} /2\\right)^2 \\nonumber\\\\\n&\\geq \\left(\\sum_{k\\in \\mathcal{N}_G, k \\le N_\\epsilon -1 } \\sqrt{\\Alpha_k} /2\\right)^2\n\\geq \\left(N_G \\sqrt{\\stepsizebar} /2\\right)^2 \n= \\frac{1}{4} (N_G)^2 \\stepsizebar.\n\\end{align*}\nSubstituting this to the previous inequality we get \n \\begin{align*}\n(N_G)^2  &\\leq  \\frac{2}{\\stepsizebar\\epsilon} \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\Lambda_{i} \\right]  + 2\\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} ^2   \\right].\n\\end{align*}\nTaking total expectation we have: \n\\begin{align*}\n\\EE [(N_G)^2 ]   &\\leq \\frac{2}{\\stepsizebar\\epsilon} \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 \n+ \\EE \\left[ \\sum_{k=1}^{N_\\epsilon } \\Lambda_{k} \\cdot \\left(\\sum_{i=1}^{k-1} \\Lambda_{i} \\right)  \\right] + 2\\EE \\left[\\sum_{k=1}^{N_\\epsilon }  \\Lambda_{k} ^2  \\right] \n \\right].\n\\end{align*}\nFrom \\eqref{eq:fista_noise} we have: \n\\begin{align*}\n    \\EE \\left[\\Lambda_{k}^2| \\pastone\\right] = \\EE \\left[4\\alpha_{k}^2t_k^2 \\| (\\nabla f(y_{k} )- G_{k} )\\|^2 | \\pastone\\right] \\leq \\frac{4}{k^{2+ \\beta}},\n\\end{align*}\nHence using a similar argument as Theorem \\ref{th:bound_expectation_N_TS1} we get \n\\begin{align*}\n\\EE [(N_G)^2 ]\n &\\leq \\frac{2}{\\stepsizebar\\epsilon} \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  + \\frac{4(\\beta + 2)^2}{\\beta^2} + \\frac{8(\\beta+2)}{\\beta+1} \n \\right].\n\\end{align*}\nBy the expectation inequality: \n \\begin{align*}\n\\EE [N_G] \\leq \\sqrt{\n \\frac{2}{\\stepsizebar\\epsilon} \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\frac{4(\\beta + 2)^2}{\\beta^2} + \\frac{8(\\beta+2)}{\\beta+1}  \n \\right] }.\n\\end{align*}\nwe get the final results of Theorem \\ref{th:bound_expectation_N_TS}. \n~\\qed \\end{proof}","sha256_utf8":"70c830bc0d99863d287fcabe025ea083cea697b289edbfc8c5eea3aae5028835"}]},{"id":"S094","semantic_type":"proof_step","kind":"proof_step","title":"停止时刻能量上界","source_units":[{"start":103764,"end":104942,"start_line":1270,"end_line":1279,"raw_tex":"\\begin{proof} \nSince the statement of Lemma \\ref{lem:after_K} holds for every realizations of Algorithm \\ref{alg:fista_bktr_line_search}, for $K = N_\\epsilon -1$ we have: \n\\begin{align*}\n    2\\Alpha^{succ}_{N_\\epsilon -1} T_{N_\\epsilon -1}^2 (F(X_{N_\\epsilon -1}) - F^*)\\leq 3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{N_\\epsilon -1} \\Lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\Lambda_{i} \\right]  + 2\\sum_{k=1}^{N_\\epsilon -1} \\Lambda_{k} ^2.\n\\end{align*}\nRecall that $N_\\epsilon$ be the stopping time that $N_\\epsilon = \\inf\\{k: F(X_k) - F^* \\leq \\epsilon\\}$. Hence for $K = N_\\epsilon -1$, we have that $F(X_K) - F^* > \\epsilon$ and \n\\begin{align*}\n    2\\Alpha^{succ}_{N_\\epsilon -1} T_{N_\\epsilon -1}^2 &\\leq \\frac{3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{N_\\epsilon -1} \\Lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\Lambda_{i} \\right]  + 2\\sum_{k=1}^{N_\\epsilon -1} \\Lambda_{k}^2}{F(X_{N_\\epsilon -1}) - F^*} \\nonumber\\\\\n    &\\leq \\frac{3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\Lambda_{i} \\right]  + 2\\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} ^2}{\\epsilon}.\n\\end{align*}\n","sha256_utf8":"f3c3bee064e45824d1983aa75367f6fb26a16a5ef256b4dc2499c997e74743a7"}]},{"id":"S095","semantic_type":"proof_step","kind":"proof_step","title":"动量下界控制好步平方","source_units":[{"start":104942,"end":106003,"start_line":1280,"end_line":1292,"raw_tex":"Let $\\mathcal{N}_G$ be the set of \\textit{good} iterations  (i.e. true and successful iterations $k$ with $\\alpha_k\\geq \\stepsizebar $), with $ k \\leq N_\\epsilon -1$, then $N_G$ is the cardinality of that set. \nUsing the inequality $ \\alpha^{succ}_{K} t_{K}^2 \\geq \\left(\\sum_{k \\in S, k \\le K } \\sqrt{\\alpha_k} /2\\right)^2$ for $K =  N_\\epsilon-1$ from Lemma \\ref{lem:fista_linesearch} we have \n\\begin{align*} \n\\Alpha^{succ}_{N_\\epsilon -1} T_{N_\\epsilon -1}^2 \n&\\geq \\left(\\sum_{k \\in S, k \\le N_\\epsilon -1} \\sqrt{\\Alpha_k} /2\\right)^2 \\nonumber\\\\\n&\\geq \\left(\\sum_{k\\in \\mathcal{N}_G, k \\le N_\\epsilon -1 } \\sqrt{\\Alpha_k} /2\\right)^2\n\\geq \\left(N_G \\sqrt{\\stepsizebar} /2\\right)^2 \n= \\frac{1}{4} (N_G)^2 \\stepsizebar.\n\\end{align*}\nSubstituting this to the previous inequality we get \n \\begin{align*}\n(N_G)^2  &\\leq  \\frac{2}{\\stepsizebar\\epsilon} \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} \\cdot \\left[\\sum_{i=1}^{k-1} \\Lambda_{i} \\right]  + 2\\sum_{k=1}^{N_\\epsilon} \\Lambda_{k} ^2   \\right].\n\\end{align*}\n","sha256_utf8":"dc80e22d0f0489cf78aad33500dc8db476f4516e04e08fbd6a892ef170b58843"}]},{"id":"S096","semantic_type":"proof_step","kind":"proof_step","title":"好步平方界取期望","source_units":[{"start":106003,"end":106358,"start_line":1293,"end_line":1298,"raw_tex":"Taking total expectation we have: \n\\begin{align*}\n\\EE [(N_G)^2 ]   &\\leq \\frac{2}{\\stepsizebar\\epsilon} \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 \n+ \\EE \\left[ \\sum_{k=1}^{N_\\epsilon } \\Lambda_{k} \\cdot \\left(\\sum_{i=1}^{k-1} \\Lambda_{i} \\right)  \\right] + 2\\EE \\left[\\sum_{k=1}^{N_\\epsilon }  \\Lambda_{k} ^2  \\right] \n \\right].\n\\end{align*}\n","sha256_utf8":"b8010a507f07bed5a48d2cb1667ee9041ee2f517d9b482ab8eee25044b991f5e"}]},{"id":"S097","semantic_type":"proof_step","kind":"proof_step","title":"沿用ISTA噪声矩求和","source_units":[{"start":106358,"end":106874,"start_line":1299,"end_line":1308,"raw_tex":"From \\eqref{eq:fista_noise} we have: \n\\begin{align*}\n    \\EE \\left[\\Lambda_{k}^2| \\pastone\\right] = \\EE \\left[4\\alpha_{k}^2t_k^2 \\| (\\nabla f(y_{k} )- G_{k} )\\|^2 | \\pastone\\right] \\leq \\frac{4}{k^{2+ \\beta}},\n\\end{align*}\nHence using a similar argument as Theorem \\ref{th:bound_expectation_N_TS1} we get \n\\begin{align*}\n\\EE [(N_G)^2 ]\n &\\leq \\frac{2}{\\stepsizebar\\epsilon} \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2  + \\frac{4(\\beta + 2)^2}{\\beta^2} + \\frac{8(\\beta+2)}{\\beta+1} \n \\right].\n\\end{align*}\n","sha256_utf8":"755d7e1b92430f27b08032b6fd9f7efe64d88308ef6814c47667a62d1e90eafb"}]},{"id":"S098","semantic_type":"proof_step","kind":"proof_step","title":"二阶矩转为一阶矩界","source_units":[{"start":106874,"end":107207,"start_line":1309,"end_line":1316,"raw_tex":"By the expectation inequality: \n \\begin{align*}\n\\EE [N_G] \\leq \\sqrt{\n \\frac{2}{\\stepsizebar\\epsilon} \\left [3\\alpha^{succ}_{0} t_{0}^2v_{0} +2\\norm{u_0 }^2 + \\frac{4(\\beta + 2)^2}{\\beta^2} + \\frac{8(\\beta+2)}{\\beta+1}  \n \\right] }.\n\\end{align*}\nwe get the final results of Theorem \\ref{th:bound_expectation_N_TS}. \n~\\qed \\end{proof}","sha256_utf8":"6d6fceaee165ca6362ce85e31d841db382e54d9c6db995b10357d2a03597b0a2"}]},{"id":"S099","semantic_type":"external_fact","kind":"external_fact","title":"近端非扩张性","source_units":[],"declared_external":true,"declaration_text":["对 proper、closed、convex 的 h 及 α>0，||prox_{αh}(z_1)-prox_{αh}(z_2)||≤||z_1-z_2||。"]},{"id":"S100","semantic_type":"external_fact","kind":"external_fact","title":"小步长停留计数界","source_units":[],"declared_external":true,"declaration_text":["Cartis-Scheinberg，Lemma 2.3：在所引用的乘性步长搜索框架、条件真迭代概率至少 p 及小步长真迭代必成功的条件下，E[Σ_{k<N_ε}(1-I_k^{large})]≤E[N_ε-1]/(2p)。"]},{"id":"S101","semantic_type":"external_fact","kind":"external_fact","title":"大步长搜索计数界","source_units":[],"declared_external":true,"declaration_text":["Cartis-Scheinberg，Lemma 2.7 及 equations (11)、(12)：对证明中定义的计数，E[N_FS]≤E[N_F]≤((1-p)/p)(E[N_G]+E[N_U])，E[N_U]≤E[N_FS]+E[N_G]+log_γ(ᾱ/α_1)，且 E[Σ_{k<N_ε}I_k^{large}]≤(2E[N_G]+log_γ(ᾱ/α_1))/(2p-1)。"]},{"id":"S102","semantic_type":"external_fact","kind":"external_fact","title":"非负序列递推引理","source_units":[],"declared_external":true,"declaration_text":["Schmidt-Roux-Bach，Lemma 1：若非负序列对每个前缀满足 a_k²≤C+Σ_{i=1}^k b_i a_i，其中 C≥0、b_i≥0，则 a_k≤(Σ_{i=1}^k b_i)/2+sqrt(C+((Σ_{i=1}^k b_i)/2)²)。"]},{"id":"S103","semantic_type":"external_fact","kind":"external_fact","title":"停止和与条件期望","source_units":[],"declared_external":true,"declaration_text":["Wald 停止和恒等式在其适用条件下给出 E[Σ_{k=1}^N Z_k]=E[Σ_{k=1}^N E Z_k]；另使用 E[E(Z|F)]=E Z、F-可测因子可提出条件期望，以及非负可和级数与期望的交换。"]},{"id":"S104","semantic_type":"external_fact","kind":"external_fact","title":"凸分析与基本不等式","source_units":[],"declared_external":true,"declaration_text":["凸函数满足次梯度支撑不等式；近端最小化满足相应次梯度一阶最优条件；L-smooth 函数满足下降引理。另使用 Cauchy-Schwarz、Jensen（特别是 E Z≤sqrt(E Z²)）及正项级数的积分比较。"]},{"id":"S105","semantic_type":"assumption","kind":"assumption","title":"成功性引理局部条件","source_units":[{"start":41308,"end":41665,"start_line":395,"end_line":402,"raw_tex":"Suppose that iteration $k$ is true, i.e. \n\\begin{align*}\n    \\|g_k-\\nabla f(y_k)\\|\\leq \\constG  \\|D_{\\alpha_k}(y_k)\\|,\n\\end{align*}\nwith  {$\\constG \\leq \\frac{1}{3}$} and $D_{\\alpha_k}(y_k)$ is the gradient mapping of $F$ at $y_k$. If \n\\begin{align*}\n\\stepsize{k}{} \\le \\stepsizebar := \\frac{1}{L}\\left( 1 - \\frac{2\\constG}{1-\\constG}\\right) ,\n\\end{align*}\n","sha256_utf8":"9b7f71a61e5ff56d6e3f7f139b01399de4f5ab13672c5056198a9240df337465"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S034","target_id":"S003","kind":"supports"},{"source_id":"S092","target_id":"S003","kind":"supports"},{"source_id":"S026","target_id":"S003","kind":"supports"},{"source_id":"S028","target_id":"S003","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S012","target_id":"S027","kind":"supports"},{"source_id":"S022","target_id":"S027","kind":"supports"},{"source_id":"S023","target_id":"S027","kind":"supports"},{"source_id":"S034","target_id":"S027","kind":"supports"},{"source_id":"S017","target_id":"S027","kind":"supports"},{"source_id":"S004","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S004","target_id":"S010","kind":"supports"},{"source_id":"S007","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S011","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S006","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S009","target_id":"S016","kind":"supports"},{"source_id":"S011","target_id":"S017","kind":"supports"},{"source_id":"S013","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S014","target_id":"S018","kind":"supports"},{"source_id":"S015","target_id":"S018","kind":"supports"},{"source_id":"S016","target_id":"S018","kind":"supports"},{"source_id":"S028","target_id":"S018","kind":"supports"},{"source_id":"S020","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S010","target_id":"S022","kind":"supports"},{"source_id":"S011","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S006","target_id":"S024","kind":"supports"},{"source_id":"S021","target_id":"S024","kind":"supports"},{"source_id":"S066","target_id":"S024","kind":"supports"},{"source_id":"S013","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S032","target_id":"S029","kind":"supports"},{"source_id":"S033","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S005","target_id":"S030","kind":"supports"},{"source_id":"S010","target_id":"S030","kind":"supports"},{"source_id":"S104","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S009","target_id":"S032","kind":"supports"},{"source_id":"S008","target_id":"S032","kind":"supports"},{"source_id":"S010","target_id":"S032","kind":"supports"},{"source_id":"S016","target_id":"S032","kind":"supports"},{"source_id":"S005","target_id":"S032","kind":"supports"},{"source_id":"S099","target_id":"S032","kind":"supports"},{"source_id":"S105","target_id":"S032","kind":"supports"},{"source_id":"S105","target_id":"S033","kind":"supports"},{"source_id":"S031","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S104","target_id":"S033","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S035","kind":"supports"},{"source_id":"S039","target_id":"S035","kind":"supports"},{"source_id":"S040","target_id":"S035","kind":"supports"},{"source_id":"S041","target_id":"S035","kind":"supports"},{"source_id":"S042","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S012","target_id":"S036","kind":"supports"},{"source_id":"S014","target_id":"S036","kind":"supports"},{"source_id":"S015","target_id":"S036","kind":"supports"},{"source_id":"S028","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S028","target_id":"S037","kind":"supports"},{"source_id":"S017","target_id":"S037","kind":"supports"},{"source_id":"S100","target_id":"S037","kind":"supports"},{"source_id":"S015","target_id":"S038","kind":"supports"},{"source_id":"S016","target_id":"S038","kind":"supports"},{"source_id":"S018","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S036","target_id":"S039","kind":"supports"},{"source_id":"S017","target_id":"S039","kind":"supports"},{"source_id":"S101","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S038","target_id":"S041","kind":"supports"},{"source_id":"S101","target_id":"S041","kind":"supports"},{"source_id":"S036","target_id":"S042","kind":"supports"},{"source_id":"S037","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S044","kind":"supports"},{"source_id":"S049","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S012","target_id":"S045","kind":"supports"},{"source_id":"S010","target_id":"S045","kind":"supports"},{"source_id":"S005","target_id":"S046","kind":"supports"},{"source_id":"S104","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S010","target_id":"S048","kind":"supports"},{"source_id":"S012","target_id":"S048","kind":"supports"},{"source_id":"S104","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S006","target_id":"S049","kind":"supports"},{"source_id":"S104","target_id":"S049","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S051","kind":"supports"},{"source_id":"S055","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S043","target_id":"S052","kind":"supports"},{"source_id":"S012","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S006","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S102","target_id":"S054","kind":"supports"},{"source_id":"S052","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S104","target_id":"S055","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S059","target_id":"S057","kind":"supports"},{"source_id":"S060","target_id":"S057","kind":"supports"},{"source_id":"S061","target_id":"S057","kind":"supports"},{"source_id":"S062","target_id":"S057","kind":"supports"},{"source_id":"S063","target_id":"S057","kind":"supports"},{"source_id":"S064","target_id":"S057","kind":"supports"},{"source_id":"S065","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S050","target_id":"S058","kind":"supports"},{"source_id":"S014","target_id":"S058","kind":"supports"},{"source_id":"S013","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S018","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S019","target_id":"S061","kind":"supports"},{"source_id":"S058","target_id":"S061","kind":"supports"},{"source_id":"S013","target_id":"S061","kind":"supports"},{"source_id":"S104","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S013","target_id":"S062","kind":"supports"},{"source_id":"S103","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S063","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S014","target_id":"S063","kind":"supports"},{"source_id":"S103","target_id":"S063","kind":"supports"},{"source_id":"S061","target_id":"S064","kind":"supports"},{"source_id":"S062","target_id":"S064","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S104","target_id":"S064","kind":"supports"},{"source_id":"S060","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S068","target_id":"S067","kind":"supports"},{"source_id":"S069","target_id":"S067","kind":"supports"},{"source_id":"S070","target_id":"S067","kind":"supports"},{"source_id":"S071","target_id":"S067","kind":"supports"},{"source_id":"S072","target_id":"S067","kind":"supports"},{"source_id":"S073","target_id":"S067","kind":"supports"},{"source_id":"S067","target_id":"S066","kind":"supports"},{"source_id":"S020","target_id":"S068","kind":"supports"},{"source_id":"S022","target_id":"S068","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S022","target_id":"S069","kind":"supports"},{"source_id":"S021","target_id":"S069","kind":"supports"},{"source_id":"S022","target_id":"S070","kind":"supports"},{"source_id":"S020","target_id":"S070","kind":"supports"},{"source_id":"S069","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S020","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S022","target_id":"S073","kind":"supports"},{"source_id":"S022","target_id":"S074","kind":"supports"},{"source_id":"S024","target_id":"S074","kind":"supports"},{"source_id":"S077","target_id":"S076","kind":"supports"},{"source_id":"S078","target_id":"S076","kind":"supports"},{"source_id":"S079","target_id":"S076","kind":"supports"},{"source_id":"S080","target_id":"S076","kind":"supports"},{"source_id":"S081","target_id":"S076","kind":"supports"},{"source_id":"S082","target_id":"S076","kind":"supports"},{"source_id":"S083","target_id":"S076","kind":"supports"},{"source_id":"S084","target_id":"S076","kind":"supports"},{"source_id":"S085","target_id":"S076","kind":"supports"},{"source_id":"S076","target_id":"S075","kind":"supports"},{"source_id":"S022","target_id":"S077","kind":"supports"},{"source_id":"S010","target_id":"S077","kind":"supports"},{"source_id":"S005","target_id":"S078","kind":"supports"},{"source_id":"S104","target_id":"S078","kind":"supports"},{"source_id":"S077","target_id":"S079","kind":"supports"},{"source_id":"S078","target_id":"S079","kind":"supports"},{"source_id":"S010","target_id":"S079","kind":"supports"},{"source_id":"S104","target_id":"S079","kind":"supports"},{"source_id":"S079","target_id":"S080","kind":"supports"},{"source_id":"S006","target_id":"S080","kind":"supports"},{"source_id":"S024","target_id":"S080","kind":"supports"},{"source_id":"S080","target_id":"S081","kind":"supports"},{"source_id":"S079","target_id":"S081","kind":"supports"},{"source_id":"S081","target_id":"S082","kind":"supports"},{"source_id":"S082","target_id":"S083","kind":"supports"},{"source_id":"S083","target_id":"S084","kind":"supports"},{"source_id":"S020","target_id":"S084","kind":"supports"},{"source_id":"S021","target_id":"S084","kind":"supports"},{"source_id":"S022","target_id":"S084","kind":"supports"},{"source_id":"S024","target_id":"S084","kind":"supports"},{"source_id":"S082","target_id":"S085","kind":"supports"},{"source_id":"S084","target_id":"S085","kind":"supports"},{"source_id":"S066","target_id":"S085","kind":"supports"},{"source_id":"S006","target_id":"S085","kind":"supports"},{"source_id":"S104","target_id":"S085","kind":"supports"},{"source_id":"S088","target_id":"S087","kind":"supports"},{"source_id":"S089","target_id":"S087","kind":"supports"},{"source_id":"S090","target_id":"S087","kind":"supports"},{"source_id":"S091","target_id":"S087","kind":"supports"},{"source_id":"S087","target_id":"S086","kind":"supports"},{"source_id":"S074","target_id":"S088","kind":"supports"},{"source_id":"S075","target_id":"S088","kind":"supports"},{"source_id":"S024","target_id":"S088","kind":"supports"},{"source_id":"S088","target_id":"S089","kind":"supports"},{"source_id":"S006","target_id":"S089","kind":"supports"},{"source_id":"S089","target_id":"S090","kind":"supports"},{"source_id":"S102","target_id":"S090","kind":"supports"},{"source_id":"S104","target_id":"S090","kind":"supports"},{"source_id":"S088","target_id":"S091","kind":"supports"},{"source_id":"S090","target_id":"S091","kind":"supports"},{"source_id":"S104","target_id":"S091","kind":"supports"},{"source_id":"S094","target_id":"S093","kind":"supports"},{"source_id":"S095","target_id":"S093","kind":"supports"},{"source_id":"S096","target_id":"S093","kind":"supports"},{"source_id":"S097","target_id":"S093","kind":"supports"},{"source_id":"S098","target_id":"S093","kind":"supports"},{"source_id":"S093","target_id":"S092","kind":"supports"},{"source_id":"S086","target_id":"S094","kind":"supports"},{"source_id":"S014","target_id":"S094","kind":"supports"},{"source_id":"S023","target_id":"S094","kind":"supports"},{"source_id":"S094","target_id":"S095","kind":"supports"},{"source_id":"S066","target_id":"S095","kind":"supports"},{"source_id":"S018","target_id":"S095","kind":"supports"},{"source_id":"S095","target_id":"S096","kind":"supports"},{"source_id":"S096","target_id":"S097","kind":"supports"},{"source_id":"S025","target_id":"S097","kind":"supports"},{"source_id":"S057","target_id":"S097","kind":"supports"},{"source_id":"S061","target_id":"S097","kind":"supports"},{"source_id":"S062","target_id":"S097","kind":"supports"},{"source_id":"S063","target_id":"S097","kind":"supports"},{"source_id":"S064","target_id":"S097","kind":"supports"},{"source_id":"S097","target_id":"S098","kind":"supports"},{"source_id":"S104","target_id":"S098","kind":"supports"}]}}
{"record_id":"2403.12338v4","paper":{"title":"Stochastic Halpern iteration in normed spaces and applications to reinforcement learning","authors":["Mario Bravo","Juan Pablo Contreras"],"arxiv_id":"2403.12338","version":"v4","journal":"Mathematical Programming","doi":"10.1007/s10107-026-02345-1","abstract_url":"https://arxiv.org/abs/2403.12338v4","pdf_url":"https://arxiv.org/pdf/2403.12338v4"},"context":{"raw_tex":"We consider $\\RR^d$ endowed with a general norm $\\norm{\\,\\cdot\\,}$, and denote by $\\norm{x}_2$ the standard Euclidean norm, {\\em i.e.} $\\norm{x}_2^2 = \\sum_{i=1}^d x_i^2$. In what follows,  $T:\\RR^d \\to \\RR^d$ is such that\n\\[\n\\norm{Tx-Ty} \\le \\gamma\\norm{x-y}, \\qquad \\forall x,y\\in \\RR^d.\n\\]\nWe say that $T$ is a contraction if $\\gamma \\in (0, 1)$, and that $T$ is nonexpansive if $\\gamma = 1$. We assume that in the nonexpansive case $\\Fix \\, T \\neq \\emptyset$, where $\\Fix\\, T$ stands for the set of fixed-points of $T$. We are interested in computing an approximate solution of the fixed-point equation $x=Tx$.\n\n\nwhere $(\\beta_n)_n$ is a nondecreasing stepsize sequence satisfying $0< \\beta_n <1$, $\\lim_{n \\to \\infty}\\beta_n=1$, and $x^0 \\in \\RR^d$ is an arbitrary initial condition.  It is worth mentioning that the original Halpern iteration uses an arbitrary $u$ called {\\em the anchoring point} that can be different from the initial point $x^0$. In that case, the iteration reads $x^{n} = (1-\\beta_{n})u + \\beta_{n}Tx^{n-1}$. We assume here, without loss of generality, that the anchoring point is exactly $x^0$.\n\n\n\\begin{assumption} \\label{assumption1} \n    The access to the operator $T$ occurs through a stochastic oracle $\\tilde{T}: \\RR^d\\times \\Xi \\to \\RR^d$ which is an unbiased approximation of $T$ in the sense that \n\\begin{equation*}\n    \\EE(\\tilde{T}(x,\\xi)) = Tx \\qquad \\mbox{for all $x\\in \\RR^d$},\n\\end{equation*}\nwith $\\xi$ drawn from some distribution $\\mathcal D_x$ over $\\Xi$. Additionally, we assume that $\\EE \\left(||\\tilde{T}(x,\\xi)-Tx||^2_2\\right)$ is finite for every $x \\in \\RR^d$.\n\\end{assumption}\n\nWe say that the oracle $\\tilde T$ has {\\em uniformly bounded variance} (with bound $\\sigma^2)$ if there exists a positive constant $\\sigma^2$ such that\n\\begin{equation*}\n \\sup_{x \\in \\RR^d}\\EE \\left(\\norm{\\tilde{T}(x,\\xi)-Tx}^2_2\\right)\\leq \\sigma^2.\n\\end{equation*}\n\n\n\\begin{equation}\n    x^{n} = (1-\\beta_{n})x^0 + \\beta_{n}(Tx^{n-1} + U_{n}), \\quad \\forall \\, n \\geq 1,\n    \\label{halpern}\n    \\tag{$\\textsc{iH}$}\n\\end{equation}\n\nwhere $U_n$ is such that $\\EE(U_n)=0$ and the term $\\EE(\\norm{U_n})$ can be controlled appropriately. It is worth noting that we do not assume further properties on the stochastic oracle in order to maintain our results as general as possible. See Section~\\ref{sec:method} for a precise definition and related discussions. \n\n\nWe are interested in the problem of estimating the oracle complexity to find an approximate fixed-point of $T$, which is measured as the number of calls to the vector stochastic oracle $\\tilde T$. In the case where $T$ is nonexpansive, we estimate the oracle complexity to find a last-iterate approximate solution $x^n$ in the sense that the expected fixed-point residual $\\EE (\\norm{x^n - Tx^n})$ is smaller than a given tolerance. For a contracting operator $T$, we aim to find oracle complexity bounds for the expected distance of the last iterate $x^n$ to the unique fixed-point $x^*$, {\\em i.e.} the quantity $\\EE (\\norm{x^n - x^*})$.\n\n\n\\noindent {\\bf Notation.} Let us set some basic notation that we use in this paper.  Given that all norms in $\\RR^d$ are equivalent, let $\\mu > 0$ be such that $\\norm{x} \\le \\mu \\norm{x}_2$ for all $x\\in \\RR^d$ and we denote by $\\norm{x}_{\\text{sp}}=\\max_{i}x_i - \\min_{i}x_i$ the so-called span seminorm of $x$.\n\nWe adopt the standard big-$O$ notation, that is, for two real functions $f,g:$ we say $f(x)=O(g(x))$ if there are two constants $C$ and $c$ such that $f(x)\\leq C g(x)$ for all $x\\geq c$. We use $\\tilde O$ if we ignore logarithmic factors. Analogously, $f(x)=\\Omega(g(x))$ if $f(x)\\geq C g(x)$ for all $x\\geq c$.\n\n\nAlgorithm~\\ref{alg_stochastic_halpern} outlines the stochastic Halpern iteration with minibatches. At each iteration $n \\leq N$, a minibatch estimator is constructed with $k_n$ queries to $\\tilde T$ at $x^{n-1}$ using $k_n$ independent fresh samples from $\\mathcal D_{x^{n-1}}$. Using this estimate, the next iterate is calculated via the Halpern iteration. \n\n\n\\begin{algorithm}[H]\n\\caption{Stochastic Halpern method}\n\\label{alg_stochastic_halpern}\n\\begin{algorithmic}[1]\n\\Require Stochastic oracle $\\tilde{T}$, $(k_n)_n$ minibatch size sequence, number of iterations $N$, initial point $x^0$    \n\\For{$n=1,\\dots, N$}\n\\State Set the size of the minibatch $k_n$ \n\\State Sample $k_n$ independent queries at $x^{n-1}$: $\\tilde{T}(x^{n-1}, \\xi^n_{1}), \\dots, \\tilde{T}(x^{n-1}, \\xi^n_{k_n})$.    \n\\State Set\n$x^n \\leftarrow (1-\\beta_n)x^0 + \\beta_n\\left(\\frac{1}{k_n}\\displaystyle{\\sum_{j=1}^{k_n}} \\tilde{T}(x^{n-1}, \\xi^n_{j})\\right)$\n\\EndFor\n\\State Return $x^N$\n\\end{algorithmic}\n\\end{algorithm}\n\nNotice that if we define the error term at the iteration $n \\geq 1$  by \n\\begin{equation*}\nU_n := \\frac{1}{k_n}\\displaystyle{\\sum_{j=1}^{k_n}} \\tilde{T}(x^{n-1}, \\xi^n_{j}) - Tx^{n-1},\n\\end{equation*}\nthen Algorithm~\\ref{alg_stochastic_halpern} can be cast in the form \\eqref{halpern}. Indeed, we will refer to both procedures indistinctly when there is no confusion.\n\nBy construction, $Tx^{n-1} + U_n$ is an unbiased estimator of $Tx^{n-1}$ and may have a reduced variance compared to the error computed with a single query. \n\n\n\\begin{equation}    \\label{notation_beta_sigma}\n\\left \\{\n\\begin{aligned} \n    &\\sigma_n:=\\EE(\\norm{U_n})\\,\\,\\, \\text{and}\\,\\,\\,\\sigma_0=\\beta_0=0;\\\\\n    & B_i^n:= \\prod_{j=i}^n \\beta_j, \\,\\,\\,\\text{with the convention}\\,\\,\\,  B_{i}^n=1 \\,\\,\\, \\text{if}\\,\\,\\,  0\\leq n<i.\n    \\end{aligned}\n    \\right.\n\\end{equation}\n\n\\begin{equation}\\tag{$\\textsc{H}_1$} \\label{H1}\n\\text{There exists } \\overline \\kappa <+\\infty \\, \\text{ such that }\\, \\sup \\nolimits_{n \\geq 0}\\EE (\\norm{T x^n-x^0}) \\leq \\overline \\kappa < +\\infty.\n\\end{equation}\n\nThe following corollary provides an upper bound on the number of queries for the stochastic Halpern iteration with minibatching when the stochastic oracle has uniformly bounded variance. Due to Lemma~\\ref{lem:sigma_explicito}, the constants involved can be explicitly estimated in this case. For the sake of simplicity, we assume in what follows that $\\varepsilon>0$ is sufficiently small.\n","units":[{"start":6046,"end":6661,"start_line":141,"end_line":145,"raw_tex":"We consider $\\RR^d$ endowed with a general norm $\\norm{\\,\\cdot\\,}$, and denote by $\\norm{x}_2$ the standard Euclidean norm, {\\em i.e.} $\\norm{x}_2^2 = \\sum_{i=1}^d x_i^2$. In what follows,  $T:\\RR^d \\to \\RR^d$ is such that\n\\[\n\\norm{Tx-Ty} \\le \\gamma\\norm{x-y}, \\qquad \\forall x,y\\in \\RR^d.\n\\]\nWe say that $T$ is a contraction if $\\gamma \\in (0, 1)$, and that $T$ is nonexpansive if $\\gamma = 1$. We assume that in the nonexpansive case $\\Fix \\, T \\neq \\emptyset$, where $\\Fix\\, T$ stands for the set of fixed-points of $T$. We are interested in computing an approximate solution of the fixed-point equation $x=Tx$.\n","sha256_utf8":"1ee7e7d3e02c1089238b504c1fe7944b06c6907624de96b61faeed20ae43f5df"},{"start":6900,"end":7406,"start_line":151,"end_line":151,"raw_tex":"where $(\\beta_n)_n$ is a nondecreasing stepsize sequence satisfying $0< \\beta_n <1$, $\\lim_{n \\to \\infty}\\beta_n=1$, and $x^0 \\in \\RR^d$ is an arbitrary initial condition.  It is worth mentioning that the original Halpern iteration uses an arbitrary $u$ called {\\em the anchoring point} that can be different from the initial point $x^0$. In that case, the iteration reads $x^{n} = (1-\\beta_{n})u + \\beta_{n}Tx^{n-1}$. We assume here, without loss of generality, that the anchoring point is exactly $x^0$.\n","sha256_utf8":"9df54456d9a8d9b42808aeda372a61be01f9b310bbb953e50e10d1c6df37b67e"},{"start":7694,"end":8199,"start_line":154,"end_line":160,"raw_tex":"\\begin{assumption} \\label{assumption1} \n    The access to the operator $T$ occurs through a stochastic oracle $\\tilde{T}: \\RR^d\\times \\Xi \\to \\RR^d$ which is an unbiased approximation of $T$ in the sense that \n\\begin{equation*}\n    \\EE(\\tilde{T}(x,\\xi)) = Tx \\qquad \\mbox{for all $x\\in \\RR^d$},\n\\end{equation*}\nwith $\\xi$ drawn from some distribution $\\mathcal D_x$ over $\\Xi$. Additionally, we assume that $\\EE \\left(||\\tilde{T}(x,\\xi)-Tx||^2_2\\right)$ is finite for every $x \\in \\RR^d$.\n\\end{assumption}","sha256_utf8":"7ffa1948e38dc13902e9515e6254e4a51bf456c2d0263e23d46cbb9a16e2f5b1"},{"start":8200,"end":8467,"start_line":161,"end_line":164,"raw_tex":"We say that the oracle $\\tilde T$ has {\\em uniformly bounded variance} (with bound $\\sigma^2)$ if there exists a positive constant $\\sigma^2$ such that\n\\begin{equation*}\n \\sup_{x \\in \\RR^d}\\EE \\left(\\norm{\\tilde{T}(x,\\xi)-Tx}^2_2\\right)\\leq \\sigma^2.\n\\end{equation*}\n","sha256_utf8":"180adbf16186854d666ca04af06a1e62fd771372470e0b0c2cfcd26583689fd5"},{"start":9371,"end":9533,"start_line":170,"end_line":174,"raw_tex":"\\begin{equation}\n    x^{n} = (1-\\beta_{n})x^0 + \\beta_{n}(Tx^{n-1} + U_{n}), \\quad \\forall \\, n \\geq 1,\n    \\label{halpern}\n    \\tag{$\\textsc{iH}$}\n\\end{equation}","sha256_utf8":"9c5c9939207b2bcf7e40035d898eb9bd77bd21bb8af006da880a52925a69dd86"},{"start":9534,"end":9858,"start_line":175,"end_line":175,"raw_tex":"where $U_n$ is such that $\\EE(U_n)=0$ and the term $\\EE(\\norm{U_n})$ can be controlled appropriately. It is worth noting that we do not assume further properties on the stochastic oracle in order to maintain our results as general as possible. See Section~\\ref{sec:method} for a precise definition and related discussions. \n","sha256_utf8":"edc04ea90b620d57c109b87212eba0a8bd27da5635d64be432caa70c1375afbd"},{"start":9859,"end":10499,"start_line":177,"end_line":177,"raw_tex":"We are interested in the problem of estimating the oracle complexity to find an approximate fixed-point of $T$, which is measured as the number of calls to the vector stochastic oracle $\\tilde T$. In the case where $T$ is nonexpansive, we estimate the oracle complexity to find a last-iterate approximate solution $x^n$ in the sense that the expected fixed-point residual $\\EE (\\norm{x^n - Tx^n})$ is smaller than a given tolerance. For a contracting operator $T$, we aim to find oracle complexity bounds for the expected distance of the last iterate $x^n$ to the unique fixed-point $x^*$, {\\em i.e.} the quantity $\\EE (\\norm{x^n - x^*})$.\n","sha256_utf8":"b47b9c76220d3810c9c80d9e2781b4224504fc86baa3094cbb2bafb068060428"},{"start":10513,"end":11139,"start_line":180,"end_line":182,"raw_tex":"\\noindent {\\bf Notation.} Let us set some basic notation that we use in this paper.  Given that all norms in $\\RR^d$ are equivalent, let $\\mu > 0$ be such that $\\norm{x} \\le \\mu \\norm{x}_2$ for all $x\\in \\RR^d$ and we denote by $\\norm{x}_{\\text{sp}}=\\max_{i}x_i - \\min_{i}x_i$ the so-called span seminorm of $x$.\n\nWe adopt the standard big-$O$ notation, that is, for two real functions $f,g:$ we say $f(x)=O(g(x))$ if there are two constants $C$ and $c$ such that $f(x)\\leq C g(x)$ for all $x\\geq c$. We use $\\tilde O$ if we ignore logarithmic factors. Analogously, $f(x)=\\Omega(g(x))$ if $f(x)\\geq C g(x)$ for all $x\\geq c$.\n","sha256_utf8":"577a4515641fd0a5b950647c3a4f407b555808a189769b35ba2c181ab809a8fc"},{"start":28515,"end":28874,"start_line":265,"end_line":265,"raw_tex":"Algorithm~\\ref{alg_stochastic_halpern} outlines the stochastic Halpern iteration with minibatches. At each iteration $n \\leq N$, a minibatch estimator is constructed with $k_n$ queries to $\\tilde T$ at $x^{n-1}$ using $k_n$ independent fresh samples from $\\mathcal D_{x^{n-1}}$. Using this estimate, the next iterate is calculated via the Halpern iteration. \n","sha256_utf8":"b9cbda52957644269cab21180ea9469b6e68c8dc46b5399f8e53c74f61422381"},{"start":28876,"end":29509,"start_line":268,"end_line":281,"raw_tex":"\\begin{algorithm}[H]\n\\caption{Stochastic Halpern method}\n\\label{alg_stochastic_halpern}\n\\begin{algorithmic}[1]\n\\Require Stochastic oracle $\\tilde{T}$, $(k_n)_n$ minibatch size sequence, number of iterations $N$, initial point $x^0$    \n\\For{$n=1,\\dots, N$}\n\\State Set the size of the minibatch $k_n$ \n\\State Sample $k_n$ independent queries at $x^{n-1}$: $\\tilde{T}(x^{n-1}, \\xi^n_{1}), \\dots, \\tilde{T}(x^{n-1}, \\xi^n_{k_n})$.    \n\\State Set\n$x^n \\leftarrow (1-\\beta_n)x^0 + \\beta_n\\left(\\frac{1}{k_n}\\displaystyle{\\sum_{j=1}^{k_n}} \\tilde{T}(x^{n-1}, \\xi^n_{j})\\right)$\n\\EndFor\n\\State Return $x^N$\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"94a768329d9756d1b4760a1f76075f8b932de1e7eb47f97f96ee811c71c37642"},{"start":29510,"end":30037,"start_line":282,"end_line":288,"raw_tex":"Notice that if we define the error term at the iteration $n \\geq 1$  by \n\\begin{equation*}\nU_n := \\frac{1}{k_n}\\displaystyle{\\sum_{j=1}^{k_n}} \\tilde{T}(x^{n-1}, \\xi^n_{j}) - Tx^{n-1},\n\\end{equation*}\nthen Algorithm~\\ref{alg_stochastic_halpern} can be cast in the form \\eqref{halpern}. Indeed, we will refer to both procedures indistinctly when there is no confusion.\n\nBy construction, $Tx^{n-1} + U_n$ is an unbiased estimator of $Tx^{n-1}$ and may have a reduced variance compared to the error computed with a single query. \n","sha256_utf8":"255b468fd5b9a441e79d25731cb4d93c71539c3d78278cabc7254c976ec75c4b"},{"start":30111,"end":30426,"start_line":290,"end_line":297,"raw_tex":"\\begin{equation}    \\label{notation_beta_sigma}\n\\left \\{\n\\begin{aligned} \n    &\\sigma_n:=\\EE(\\norm{U_n})\\,\\,\\, \\text{and}\\,\\,\\,\\sigma_0=\\beta_0=0;\\\\\n    & B_i^n:= \\prod_{j=i}^n \\beta_j, \\,\\,\\,\\text{with the convention}\\,\\,\\,  B_{i}^n=1 \\,\\,\\, \\text{if}\\,\\,\\,  0\\leq n<i.\n    \\end{aligned}\n    \\right.\n\\end{equation}","sha256_utf8":"c550329fbd4044ccfe5fb68ce8dc6078ce125dad8b65e9507075745ee9c8a393"},{"start":32025,"end":32239,"start_line":312,"end_line":314,"raw_tex":"\\begin{equation}\\tag{$\\textsc{H}_1$} \\label{H1}\n\\text{There exists } \\overline \\kappa <+\\infty \\, \\text{ such that }\\, \\sup \\nolimits_{n \\geq 0}\\EE (\\norm{T x^n-x^0}) \\leq \\overline \\kappa < +\\infty.\n\\end{equation}","sha256_utf8":"0eeab37ea2c483d82a2c6156c5bc4313c6e055a32130575d167bf0c67c28b372"},{"start":42646,"end":43036,"start_line":450,"end_line":450,"raw_tex":"The following corollary provides an upper bound on the number of queries for the stochastic Halpern iteration with minibatching when the stochastic oracle has uniformly bounded variance. Due to Lemma~\\ref{lem:sigma_explicito}, the constants involved can be explicitly estimated in this case. For the sake of simplicity, we assume in what follows that $\\varepsilon>0$ is sufficiently small.\n","sha256_utf8":"7f4510e86b798a652dac9dca337e60adf95091d81b48386a2a5329ddf562c49d"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{corollary} \\label{cor:bounded_variance}\n Assume that $\\tilde{T}$ has uniformly bounded variance $\\sigma^2$. Let $(x^n)_n$ be the sequence generated by the Halpern iteration~\\eqref{halpern} with the stepsize $\\beta_n=\\frac{n}{n+1}$ and batch size $k_n = n^{4}$. Then, given $\\varepsilon>0$, the Algorithm~\\ref{alg_stochastic_halpern} finds $x^N$ such that $\\EE(\\norm{x^N-Tx^N})\\le \\varepsilon$ using at most $\\tilde{O}(L^5\\varepsilon^{-5})$ queries to the stochastic oracle, where $L=\\max\\{\\mu \\sigma, \\dist(x^0, \\Fix T)\\}$.\n\\end{corollary}","units":[{"start":43051,"end":43597,"start_line":453,"end_line":455,"raw_tex":"\\begin{corollary} \\label{cor:bounded_variance}\n Assume that $\\tilde{T}$ has uniformly bounded variance $\\sigma^2$. Let $(x^n)_n$ be the sequence generated by the Halpern iteration~\\eqref{halpern} with the stepsize $\\beta_n=\\frac{n}{n+1}$ and batch size $k_n = n^{4}$. Then, given $\\varepsilon>0$, the Algorithm~\\ref{alg_stochastic_halpern} finds $x^N$ such that $\\EE(\\norm{x^N-Tx^N})\\le \\varepsilon$ using at most $\\tilde{O}(L^5\\varepsilon^{-5})$ queries to the stochastic oracle, where $L=\\max\\{\\mu \\sigma, \\dist(x^0, \\Fix T)\\}$.\n\\end{corollary}","sha256_utf8":"8209e00569908a70353b64656272441380efc14425b7a8f4408721c5ba552d28"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"For the sake of completeness, we recall how a standard argument allows to transfer the variance reduction into a bound on $\\sigma_n$. \n\nFor a given iteration $n$, let $\\xi^n_{1},\\ldots,\\xi^n_{k_n}$ be $k_n$ independent samples of the distribution $\\mathcal D_{x^{n-1}}$ and let $e_n^2=\\EE (||\\tilde{T}(x^{n-1}, \\xi^n_{1}) - T x^{n-1}||_2^2)$ be the variance of the error term of one query. Then\n\\begin{equation}\n\\sigma_n^2=\\EE(\\norm{U_n})^2 \\le \\frac{\\mu^2}{k_n^2} \\EE\\left(\\norm{\\sum \\nolimits_{j=1}^{k_n} \\tilde{T}(x^{n-1}, \\xi^n_{j}) - Tx^{n-1} }^2_2\\right) \\le \\frac{\\mu^2}{k_n}e_n^2,\n\\label{minibatch}\n\\end{equation}\nwhere we have use Jensen's inequality, sample independence, and the equivalence between the norms. We observe that $\\EE(\\norm{U_n})$ scales by a factor $\\frac{\\mu}{\\sqrt{k_n}}$ compared to $e_n$.  \n\n\n\\begin{proposition} \\label{prop:residual-nonexpansive}\nLet $(x^n)_{n\\ge 1}$ the sequence generated by the Halpern iteration \\eqref{halpern}. Assume that \\eqref{H1} holds for some $\\overline \\kappa \\geq 0$. Then, for all $n\\ge 1$\n\\[\n\\EE(\\norm{x^n-Tx^n}) \\le \\overline\\kappa(1-\\beta_n) + \\sum_{i=1}^{n} B_i^n \\left(\\overline\\kappa(\\beta_i -\\beta_{i-1}) + \\beta_i\\sigma_i+\\beta_{i-1}\\sigma_{i-1} \\right )+ \\beta_n\\sigma_n,\n\\]\nwhere $B^n_i$ and $\\sigma_i$ are defined in \\eqref{notation_beta_sigma}.\n\\end{proposition}\n\n\\begin{proof}\nRecall that $\\sigma_0=\\beta_0=0$ and let $\\kappa_n= \\norm{Tx^n-x^0}$. We start by observing that the (random) sequence $(P_n)_n$ defined inductively as $P_0=0$ and\n\\[ P_n=\n\\kappa_{n-1} (\\beta_n-\\beta_{n-1})+\\beta_{n-1} P_{n-1}+\\beta_n\\norm{U_n}+\\beta_{n-1}\\norm{U_{n-1}},\\,\\,  n\\geq 1,\n\\]\nis such that $\\norm{x^n-x^{n-1}}\\leq P_n$ for all $n \\in \\NN$. Indeed, for the first term we write\n\\[\n\\norm{x^1-x^0}= \\beta_1\\norm{x^0 -Tx^0 + U_1} \\leq \\kappa_0 (\\beta_1-\\beta_0) + \\beta_1 \\norm{U_1}=P_1.\n\\]\nFrom the definition of \\eqref{halpern}, using again the triangular inequality, the nonexpansivity of $T$, and the induction hypothesis, we have that for $n \\geq 2$,\n\\begin{equation*}\n\\begin{aligned}\n\\!\\!\\norm{x^n\\!-\\!x^{n-1}} &=\\norm{(1\\!-\\! \\beta_{n}) x^0 + \\beta_n(Tx^{n-1} +U_n)-(1\\!-\\!\\beta_{n-1})x^0 \\!-\\! \\beta_{n-1}(Tx^{n-2} +U_{n\\!-\\!1})}\\\\\n&=\\norm{(\\beta_{n}\\!-\\!\\beta_{n-1})(T x^{n-1}\\!-\\!x^0)\\! + \\!\\beta_{n-1}(Tx^{n-1}\\!\\!-Tx^{n-2}) \\!+\\!\\beta_{n} U_n\\!-\\!\\beta_{n-1}U_{n-1}} \\\\\n&\\leq (\\beta_n-\\beta_{n-1})\\norm{T x^{n-1}\\!-\\!x^0}\\!+\\! \\beta_{n-1}\\norm{x^{n-1}\\!-\\!x^{n-2}}\\!+\\!\\beta_n\\norm{U_{n}}\\!+\\!\\beta_{n-1}\\norm{U_{n-1}} \\\\\n&\\le \\kappa_{n-1} (\\beta_n-\\beta_{n-1})+\\beta_{n-1} P_{n-1}+\\beta_n\\norm{U_{n}}\\!+\\!\\beta_{n-1}\\norm{U_{n-1}}=P_n.\n\\end{aligned}\n\\end{equation*}\n\nThen, it follows that $P_n$ can be computed explicitly as\n\\[\nP_n=\\sum_{i=1}^{n}B_i^{n-1} \\left ( \\kappa_{i-1}(\\beta_i -\\beta_{i-1}) + \\beta_i\\norm{U_{i}}+\\beta_{i-1}\\norm{U_{i-1}} \\right ).\n\\]\nwhere we used the convention $B^{n-1}_n = 1$. Now, casting \\eqref{halpern} as\n   \\[\n    x^{n} - Tx^n = (1-\\beta_{n})(x^0-Tx^{n}) + \\beta_{n} (Tx^{n-1}-Tx^n + U_{n}), \\quad \\forall \\, n \\geq 1,\n   \\]\nwe get that \n\\begin{equation}\\label{eq:residual}\n\\begin{aligned}\n\\!\\!\\!\\norm{x^n-\\!Tx^n} & \\!\\!=\\! \\!\\norm{(1\\!-\\!\\beta_n)(x^0-Tx^{n})+\\beta_n(Tx^{n-1}-Tx^n+U_n )} \\\\\n&\\le \\kappa_n (1-\\beta_n) +  \\beta_n  \\norm{x^n-x^{n-1}}+ \\beta_n\\norm{U_n} \\\\\n&\\leq \\kappa_n(1-\\beta_n) +  \\beta_n  P_n+ \\beta_n\\norm{U_n},\\\\\n& = \\kappa_n(1-\\!\\beta_n)\\! +\\! \\\\\n&+\\sum_{i=1}^{n}B_i^{n}\\left ( \\!\\kappa_{i-1}(\\beta_i -\\beta_{i-1})\\! + \\!\\beta_i\\norm{U_i}\\!\\!+\\!\\!\\beta_{i-1}\\norm{U_{i-1}} \\!\\right )+ \\beta_n\\norm{U_n}\n\\end{aligned}\n\\end{equation}\nwhere we used that $B^n_i = \\beta_nB^{n-1}_i$ for every $i=1,...,n$ and that $B^{n-1}_{n} = 1$. Taking expectations in the expression above, and using that $\\sigma_i = \\EE(\\norm{U_i})$, we obtain\n\\begin{equation*}\n\\begin{aligned}\n\\EE(\\norm{x^n\\!\\!-\\!\\!Tx^n}) &\\leq \\EE(\\kappa_n)(1-\\!\\beta_n)+ \\sum_{i=1}^{n}B_i^{n}\\left ( \\EE(\\kappa_{i-1})(\\beta_i -\\beta_{i-1})\\! + \\!\\beta_i\\sigma_i +\\beta_{i\\!-\\!1}\\sigma_{i-1} \\!\\right )\\!+\\! \\beta_n\\sigma_n. \n\\end{aligned}\n\\end{equation*}\nThe conclusion follows from the fact that $\\sup_{n\\geq 0} \\EE(\\kappa_n) \\leq \\overline \\kappa$.\n\\end{proof}\n\n\\begin{lemma} \\label{lem:kappa}\n{\\em Let $T$ be nonexpansive and $(x^n)_n$ be the sequence generated by \\eqref{halpern}. Recalling the notation \\eqref{notation_beta_sigma}, it holds:\n\\begin{itemize}\n    \\item[$(i)$] If $T$ has bounded range i.e. there exist a constant $M>0$ such that $\\norm{Tx} \\le M$ for all $x\\in \\RR^d$, then the condition \\eqref{H1} holds with $\\bar \\kappa = M + \\norm{x^0}$.\n    \\item[$(ii)$] If the sequences $(\\beta_n)_n$ and $(\\sigma_n)_n$ satisfies\n    \\begin{equation}\\label{H2}\\tag{$\\textsc{H}_2$}\n    \\sup_{n\\ge 1}\\sum_{i=1}^n B_i^n\\sigma_i\\leq S < +\\infty,,\n    \\end{equation}\n    for some $S\\ge 0$, then condition \\eqref{H1} holds with $\\bar \\kappa = 2\\dist(x^0, \\Fix(T)) + {S}$. \n    \n    In particular, if $\\sum_n \\sigma_n <+\\infty$ then  \\eqref{H2} holds. \n\\end{itemize}\n}\n\\end{lemma}\n\n\\begin{proof} \nPart $(i)$ follows from the triangular inequality. For part $(ii)$, let us fix $x^*\\in \\Fix(T)$. Using the fact that $T$ is nonexpansive we have\n\\begin{align}\n    \\norm{x^n-x^*} & = \\norm{(1-\\beta_n)(x^0-x^*) + \\beta_n(Tx^{n-1}-x^*) + \\beta_nU_n} \\nonumber \\\\\n    & \\le (1-\\beta_n)\\norm{x^0-x^*} + \\beta_n \\norm{x^{n-1}-x^*} + \\beta_n\\norm{U_n} \\nonumber \\\\\n    &=\\sum_{i=0}^n B^n_{i+1}( \\norm{x^0-x^*}(1-\\beta_i)+\\beta_i\\norm{U_n})\\nonumber  \\\\\n    &=  \\norm{x^0-x^*} + \\sum_{i=1}^n B^n_i \\norm{U_i}, \\label{eq:bounded}\n\\end{align}\n\nwhere we used $\\sum_{i=0}^n B^n_{i+1}(1-\\beta_i)=1$ and the convention $B^n_{n+1} = 1$. Taking expectations, we have\n\\begin{equation*}\n\\begin{aligned}\n\\EE(\\norm{Tx^n-x^0})&\\le \\EE(\\norm{Tx^n-Tx^*})+\\norm{x^*-x^0}\\\\\n&\\le 2 \\norm{x^*-x^0} + \\sum_{i=1}^n B_i^n\\sigma_i \\leq  2 \\norm{x^*-x^0}+ S.\n\\end{aligned}\n\\end{equation*}\nFinally, $\\sup_{n \\geq 0}\\EE(\\norm{Tx^n-x^0})\\leq 2\\inf_{x^* \\in \\Fix T}\\norm{x^*-x^0} +S.$\n\\end{proof}\n\n\\begin{theorem} \\label{thm:rate-nonexpansive}\n Let $(x^n)_n$ be the sequence generated by the Halpern iteration~\\eqref{halpern} with the stepsize $\\beta_n=\\frac{n}{n+1}$. Assume that \\eqref{H1} holds for some $\\overline \\kappa \\geq 0$ and that $\\sigma_n= \\EE(\\norm{U_n})$ is such that $\\sum_{n=1}^N n\\sigma_n =\\tilde O(1)$. Then, given $\\varepsilon>0$,  $\\EE(\\norm{x^N-Tx^N})\\le \\varepsilon$ for all $N \\geq \\bar N= \\tilde{O}(\\varepsilon^{-1})$ iterations.\n\\end{theorem}\n\n\\begin{proof}\nFor this particular choice of $\\beta_n$, we have that $B_i^n=\\frac{i}{n+1}$ for all $1 \\leq i\\leq n$. Then \n\\[\nS = \\sup_{N\\geq 1}\\sum_{n=1}^N B_n^N  \\sigma_n= \\sup_{N\\geq 1}\\frac{1}{N+1}\\sum_{n=1}^N n \\sigma_n< \\infty, \n\\]\ngiven that $\\sum_{n= 1}^N n \\sigma_n = \\tilde{O}(1)$ and we conclude that \\eqref{H2} holds and \\eqref{H1} is satisfied with $\\bar \\kappa=S+2\\dist(x^0, \\Fix T)$. Now, invoking the estimates given by Proposition~\\ref{prop:residual-nonexpansive} \n\\begin{equation} \\label{bound-expected-nonexpansive} \n\\begin{aligned}\n \\!\\!\\! \\EE(\\norm{ x^N-Tx^N }) & \\le \\! \\frac{\\bar{\\kappa}}{N\\!+\\!1} \\!+\\!  \\frac{1}{N\\!+\\!1}\\sum_{n=1}^N \\left( \\frac{\\bar{\\kappa}}{n\\!+\\!1} \\!+\\! \\frac{n^2}{n\\!+\\!1}\\sigma_n \\!+\\!(n\\!-\\!1)\\sigma_{n-1} \\right ) \\!+\\!\\frac{N}{N\\!+\\!1}\\sigma_N\\\\\n&=\\frac{1}{N\\!+\\!1}\\left(\\bar{\\kappa}\\sum_{n=0}^{N}\\frac{1}{n\\!+\\!1}  \\!+\\!\\sum_{n=1}^N\\frac{n^2}{n\\!+\\!1}\\sigma_n +\\sum_{n=2}^N(n\\!-\\!1)\\sigma_{n-1}+ N \\sigma_N\\right )\\\\\n  &\\le\\frac{\\bar{\\kappa}}{N\\!+\\!1}\\sum_{n=0}^{N}\\frac{1}{n\\!+\\!1}+\\frac{2}{N\\!+\\!1}\\sum_{n=1}^N n\\sigma_n - \\frac{1}{N+1}\\sum_{n=1}^N \\sigma_n,\n\\end{aligned}\n\\end{equation}\nwhere we have used in the second line that $\\sigma_0=0$. We observe that the first term $\\tilde{O}(1/N)$ and the second also is by hypothesis. Therefore, $\\EE(\\norm{x^N-Tx^N}) = \\tilde{O}(1/N)$, and $\\EE(\\norm{ x^N-Tx^N }) \\le \\varepsilon$, for all $N \\geq \\bar N= \\tilde{O}(\\varepsilon^{-1})$ iterations as claimed.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:sigma_explicito}\nUnder the assumptions of Theorem~\\ref{thm:rate-nonexpansive}, let us suppose additionally that there exists $C\\geq 0$ such that $\\sum_{n=1}^N n\\sigma_n \\leq C \\ln(N+1)$ for all $N \\geq 1$. Let $\\rho \\geq 0$ such that $3\\bar{\\kappa} + 2C\\leq \\rho$. Then, for all $\\varepsilon>0$, if $\\varepsilon/\\rho < e^{-1}$, then \n\\[\n\\EE(\\norm{x^N-Tx^N})\\le \\varepsilon \\quad \\text{for all }\\,\\, N \\geq \\mbox{$\\left \\lceil 2\\frac{\\rho}{\\varepsilon} \\ln\\left (\\frac{\\rho}{\\varepsilon}\\right )\\right \\rceil$} .\n\\]\nWhen $\\varepsilon/\\rho \\geq e^{-1}$, then $N=1$ iteration is sufficient.\n\\end{lemma}\n\n\\noindent {\\em Proof.} Equation~\\eqref{minibatch} guarantees that \n $\\sigma_n\\le \\mu\\sigma{n^{-2}},$ \n which readily implies that\n $$\\sum_{n=1}^N n\\sigma_n\\leq \\mu\\sigma\\sum_{n=1}^N \\frac{1}{n}\\leq \\frac{3}{2}\\mu\\sigma\\ln(N+1).$$ \n Recalling that for this particular choice of $\\beta_n$, $B_i^n= \\frac{i}{n+1}$, $i=1,\\ldots,n$, we have that\n \\[\n\\sum_{i=1}^n B_i^n\\sigma_i=\\frac{1}{n+1} \\sum_{i=1}^n i \\sigma_i\\leq \\mu\\sigma\\frac{1}{n+1}\\sum_{i=1}^n \\frac{1}{i} \\leq \\frac{\\mu\\sigma}{2}.\n \\]\n Therefore, condition~\\eqref{H2} holds with $S= \\mu\\sigma/2$ and, from Lemma~\\ref{lem:kappa}$(ii)$,  \\eqref{H1} is verified with $\\bar \\kappa= \\mu\\sigma/2 + 2 \\dist(x^0, \\Fix T)$.  Then, using Lemma~\\ref{lem:sigma_explicito} with $C=\\frac{3}{2}\\mu\\sigma$ implies that $\\EE(\\norm{x^N-Tx^N})\\le \\varepsilon$ for all $N$ larger that $ \\bar N= \\left \\lceil 2\\frac{\\rho}{\\varepsilon} \\ln\\left (\\frac{\\rho}{\\varepsilon}\\right )\\right \\rceil $, with \\vspace{-1ex}\n $$\\frac{9}{2}\\mu\\sigma+6 \\dist(x^0, \\Fix T)\\leq 12 \\max \\{\\mu\\sigma,\\dist(x^0, \\Fix T)\\}:=\\rho.$$ Finally, as each iteration $n=1,...,\\bar N$ requires $n^4$ queries to the stochastic oracle, the overall oracle complexity to achieve precision $\\varepsilon>0$ is \n $$ \\qquad \\qquad \\qquad \\qquad \\qquad \\qquad  \\sum_{n=1}^{\\bar N} k_n = \\sum_{n=1}^{\\bar N} n^{4} \\le \\frac{(\\bar N+1)^5}{5}\\leq \\frac{32}{5} \\left (\\frac{\\rho}{\\varepsilon}\\ln\\left (\\frac{\\rho}{\\varepsilon}\\right ) +1\\right )^5. \\qquad \\qquad \\qquad \\qquad \\qquad  \\,\\, \\QEDA$$\n\n\n\\subsection{Proof of Lemma~\\ref{lem:sigma_explicito}} \\label{apx:explicito}\nAssuming that $\\sum_{n=1}^N n\\sigma_n \\leq C \\ln(N+1)$, the bound \\eqref{bound-expected-nonexpansive} implies that\n\\begin{equation*} \n\\EE(\\norm{ x^N-Tx^N}) \\le \\frac{1}{N+1}\\left( 3\\bar{\\kappa}+2C\\right)\\ln(N+1) \\le \\underbrace{(3\\bar{\\kappa} + 2C)}_{\\leq\\rho}\\frac{\\ln(N+1)}{N+1}, \\end{equation*}\nwhere we have used the rough bound $3\\ln(N+1)\\geq \\sum_{n=1}^{N+1}\\frac{1}{n}$ for all $N \\geq 1$, in order to keep the estimates simple at the expense of some small factor. \n\n\n\nFor $\\varepsilon>0$, let us call $\\tau:=\\frac{\\varepsilon}{\\rho}>0$. It is sufficient to find some $\\bar N\\in \\NN$ such that, for any $N \\geq \\overline{N}$,\n\\[\n\\frac{\\ln(N+1)}{N+1} \\leq \\tau, \\quad \\text{or, equivalently,}\\,\\, -\\tau (N+1)e^{-\\tau(N+1)}\\geq -\\tau.\n\\]\nFirst, note that if $\\tau \\geq e^{-1}$ then the inequality above holds for all $N \\geq 0$. We assume in what follows that $\\tau < e^{-1}$. \n\n\n\nIt is well-known that if we want to solve the equation $-\\tau (x+1)e^{-\\tau(x+1)}= -\\tau$, the solution is given by the relation $-\\tau(x+1) = W(-\\tau)$, where $W$ is the Lambert function. Given that $\\tau >0$, there is two solutions of this equation, provided by the upper and lower branches of $W$, $W_0$ and $W_{-1}$, respectively. We recall that both functions are continuous, defined in the interval $[-e^{-1},0]$, and such that $W_0(-e^{-1})=W_{-1}(-e^{-1})=-1$, $W_0$ increases to cero and $W_{-1}$ decreases to $-\\infty$. This implies \nthat for all $\\tau \\in (0,e^{-1})$, $W_0(-\\tau) \\in (-1,0)$ and that $W_{-1}(-\\tau) \\in (-\\infty, -1)$. Let us pick the solution given by the lower branch, i.e. the one such that $x+1\\geq \\tau^{-1}$. In that case, \n$$x= -\\frac{1}{\\tau}W_{-1}(-\\tau)-1.$$\nTherefore if we set $m=\\lceil -\\frac{1}{\\tau}W_{-1}(-\\tau) -1 \\rceil \\in \\NN$ we obtain that \n$$\\frac{\\ln(N+1)}{N+1} \\leq \\tau, \\quad \\text{for all } N\\in \\NN, N\\geq m.$$\nLet us show that  $m \\leq \\lceil 2\\frac{\\rho}{\\varepsilon} \\ln\\left (\\frac{\\rho}{\\varepsilon}\\right ) \\rceil= \\lceil \\frac{2}{\\tau} \\ln\\left (\\frac{1}{\\tau}\\right ) \\rceil$. For that purpose, we use following lower bound for the function $W_{-1}$ (see \\cite[Theorem 1]{chatzigeorgiou2013bounds}):\n\n$$W_{-1}(-e^{-u-1}) \\geq -1-\\sqrt{2u}-u, \\quad \\text{for all} \\, u>0.$$ \n\nTaking $u=-\\ln(\\tau)-1>0$, we get that $-e^{-u-1}=-\\tau$ and hence\n\\[\n\\begin{aligned}\n \\left \\lceil -\\frac{1}{\\tau}W_{-1}(-\\tau) -1 \\right \\rceil&\\leq \n\\left \\lceil\\frac{1}{\\tau}\\left(1+\\sqrt{2(-\\ln(\\tau)-1)}+(-\\ln(\\tau)-1)\\right )-1\\right\\rceil,\\\\\n & = \n\\left \\lceil\\frac{1}{\\tau}\\left(\\sqrt{-2(\\ln(\\tau)+1)}-\\ln(\\tau)\\right )-1\\right\\rceil\\leq \\left \\lceil 2\\frac{1}{\\tau} \\ln\\left (\\frac{1}{\\tau}\\right )-1\\right \\rceil,\n \\end{aligned}\n\\]\nwhere the last inequality follows from the trivial bound $\\sqrt{-2(u+1)}\\leq -u$ for any $u\\leq -1$. Observe that the last quantity is at least $1$ for all $\\tau \\in (0,e^{-1})$.\n\n\n\n","units":[{"start":30427,"end":31247,"start_line":298,"end_line":305,"raw_tex":"For the sake of completeness, we recall how a standard argument allows to transfer the variance reduction into a bound on $\\sigma_n$. \n\nFor a given iteration $n$, let $\\xi^n_{1},\\ldots,\\xi^n_{k_n}$ be $k_n$ independent samples of the distribution $\\mathcal D_{x^{n-1}}$ and let $e_n^2=\\EE (||\\tilde{T}(x^{n-1}, \\xi^n_{1}) - T x^{n-1}||_2^2)$ be the variance of the error term of one query. Then\n\\begin{equation}\n\\sigma_n^2=\\EE(\\norm{U_n})^2 \\le \\frac{\\mu^2}{k_n^2} \\EE\\left(\\norm{\\sum \\nolimits_{j=1}^{k_n} \\tilde{T}(x^{n-1}, \\xi^n_{j}) - Tx^{n-1} }^2_2\\right) \\le \\frac{\\mu^2}{k_n}e_n^2,\n\\label{minibatch}\n\\end{equation}\nwhere we have use Jensen's inequality, sample independence, and the equivalence between the norms. We observe that $\\EE(\\norm{U_n})$ scales by a factor $\\frac{\\mu}{\\sqrt{k_n}}$ compared to $e_n$.  \n","sha256_utf8":"da05f46f390db7ca01a36c9d0c09edcd100889ff1ba0208f5bcf3d76e47aa4d8"},{"start":32459,"end":32972,"start_line":321,"end_line":327,"raw_tex":"\\begin{proposition} \\label{prop:residual-nonexpansive}\nLet $(x^n)_{n\\ge 1}$ the sequence generated by the Halpern iteration \\eqref{halpern}. Assume that \\eqref{H1} holds for some $\\overline \\kappa \\geq 0$. Then, for all $n\\ge 1$\n\\[\n\\EE(\\norm{x^n-Tx^n}) \\le \\overline\\kappa(1-\\beta_n) + \\sum_{i=1}^{n} B_i^n \\left(\\overline\\kappa(\\beta_i -\\beta_{i-1}) + \\beta_i\\sigma_i+\\beta_{i-1}\\sigma_{i-1} \\right )+ \\beta_n\\sigma_n,\n\\]\nwhere $B^n_i$ and $\\sigma_i$ are defined in \\eqref{notation_beta_sigma}.\n\\end{proposition}","sha256_utf8":"82321ffd9f4cb4a13daa21aa1e6f89786a8937b5cddefec08d2859c5824721c5"},{"start":32974,"end":35781,"start_line":329,"end_line":373,"raw_tex":"\\begin{proof}\nRecall that $\\sigma_0=\\beta_0=0$ and let $\\kappa_n= \\norm{Tx^n-x^0}$. We start by observing that the (random) sequence $(P_n)_n$ defined inductively as $P_0=0$ and\n\\[ P_n=\n\\kappa_{n-1} (\\beta_n-\\beta_{n-1})+\\beta_{n-1} P_{n-1}+\\beta_n\\norm{U_n}+\\beta_{n-1}\\norm{U_{n-1}},\\,\\,  n\\geq 1,\n\\]\nis such that $\\norm{x^n-x^{n-1}}\\leq P_n$ for all $n \\in \\NN$. Indeed, for the first term we write\n\\[\n\\norm{x^1-x^0}= \\beta_1\\norm{x^0 -Tx^0 + U_1} \\leq \\kappa_0 (\\beta_1-\\beta_0) + \\beta_1 \\norm{U_1}=P_1.\n\\]\nFrom the definition of \\eqref{halpern}, using again the triangular inequality, the nonexpansivity of $T$, and the induction hypothesis, we have that for $n \\geq 2$,\n\\begin{equation*}\n\\begin{aligned}\n\\!\\!\\norm{x^n\\!-\\!x^{n-1}} &=\\norm{(1\\!-\\! \\beta_{n}) x^0 + \\beta_n(Tx^{n-1} +U_n)-(1\\!-\\!\\beta_{n-1})x^0 \\!-\\! \\beta_{n-1}(Tx^{n-2} +U_{n\\!-\\!1})}\\\\\n&=\\norm{(\\beta_{n}\\!-\\!\\beta_{n-1})(T x^{n-1}\\!-\\!x^0)\\! + \\!\\beta_{n-1}(Tx^{n-1}\\!\\!-Tx^{n-2}) \\!+\\!\\beta_{n} U_n\\!-\\!\\beta_{n-1}U_{n-1}} \\\\\n&\\leq (\\beta_n-\\beta_{n-1})\\norm{T x^{n-1}\\!-\\!x^0}\\!+\\! \\beta_{n-1}\\norm{x^{n-1}\\!-\\!x^{n-2}}\\!+\\!\\beta_n\\norm{U_{n}}\\!+\\!\\beta_{n-1}\\norm{U_{n-1}} \\\\\n&\\le \\kappa_{n-1} (\\beta_n-\\beta_{n-1})+\\beta_{n-1} P_{n-1}+\\beta_n\\norm{U_{n}}\\!+\\!\\beta_{n-1}\\norm{U_{n-1}}=P_n.\n\\end{aligned}\n\\end{equation*}\n\nThen, it follows that $P_n$ can be computed explicitly as\n\\[\nP_n=\\sum_{i=1}^{n}B_i^{n-1} \\left ( \\kappa_{i-1}(\\beta_i -\\beta_{i-1}) + \\beta_i\\norm{U_{i}}+\\beta_{i-1}\\norm{U_{i-1}} \\right ).\n\\]\nwhere we used the convention $B^{n-1}_n = 1$. Now, casting \\eqref{halpern} as\n   \\[\n    x^{n} - Tx^n = (1-\\beta_{n})(x^0-Tx^{n}) + \\beta_{n} (Tx^{n-1}-Tx^n + U_{n}), \\quad \\forall \\, n \\geq 1,\n   \\]\nwe get that \n\\begin{equation}\\label{eq:residual}\n\\begin{aligned}\n\\!\\!\\!\\norm{x^n-\\!Tx^n} & \\!\\!=\\! \\!\\norm{(1\\!-\\!\\beta_n)(x^0-Tx^{n})+\\beta_n(Tx^{n-1}-Tx^n+U_n )} \\\\\n&\\le \\kappa_n (1-\\beta_n) +  \\beta_n  \\norm{x^n-x^{n-1}}+ \\beta_n\\norm{U_n} \\\\\n&\\leq \\kappa_n(1-\\beta_n) +  \\beta_n  P_n+ \\beta_n\\norm{U_n},\\\\\n& = \\kappa_n(1-\\!\\beta_n)\\! +\\! \\\\\n&+\\sum_{i=1}^{n}B_i^{n}\\left ( \\!\\kappa_{i-1}(\\beta_i -\\beta_{i-1})\\! + \\!\\beta_i\\norm{U_i}\\!\\!+\\!\\!\\beta_{i-1}\\norm{U_{i-1}} \\!\\right )+ \\beta_n\\norm{U_n}\n\\end{aligned}\n\\end{equation}\nwhere we used that $B^n_i = \\beta_nB^{n-1}_i$ for every $i=1,...,n$ and that $B^{n-1}_{n} = 1$. Taking expectations in the expression above, and using that $\\sigma_i = \\EE(\\norm{U_i})$, we obtain\n\\begin{equation*}\n\\begin{aligned}\n\\EE(\\norm{x^n\\!\\!-\\!\\!Tx^n}) &\\leq \\EE(\\kappa_n)(1-\\!\\beta_n)+ \\sum_{i=1}^{n}B_i^{n}\\left ( \\EE(\\kappa_{i-1})(\\beta_i -\\beta_{i-1})\\! + \\!\\beta_i\\sigma_i +\\beta_{i\\!-\\!1}\\sigma_{i-1} \\!\\right )\\!+\\! \\beta_n\\sigma_n. \n\\end{aligned}\n\\end{equation*}\nThe conclusion follows from the fact that $\\sup_{n\\geq 0} \\EE(\\kappa_n) \\leq \\overline \\kappa$.\n\\end{proof}","sha256_utf8":"445e547c9bf4646c857c09075ed740562d5b567fe848afdb7d4652b5090bff6c"},{"start":35985,"end":36804,"start_line":379,"end_line":392,"raw_tex":"\\begin{lemma} \\label{lem:kappa}\n{\\em Let $T$ be nonexpansive and $(x^n)_n$ be the sequence generated by \\eqref{halpern}. Recalling the notation \\eqref{notation_beta_sigma}, it holds:\n\\begin{itemize}\n    \\item[$(i)$] If $T$ has bounded range i.e. there exist a constant $M>0$ such that $\\norm{Tx} \\le M$ for all $x\\in \\RR^d$, then the condition \\eqref{H1} holds with $\\bar \\kappa = M + \\norm{x^0}$.\n    \\item[$(ii)$] If the sequences $(\\beta_n)_n$ and $(\\sigma_n)_n$ satisfies\n    \\begin{equation}\\label{H2}\\tag{$\\textsc{H}_2$}\n    \\sup_{n\\ge 1}\\sum_{i=1}^n B_i^n\\sigma_i\\leq S < +\\infty,,\n    \\end{equation}\n    for some $S\\ge 0$, then condition \\eqref{H1} holds with $\\bar \\kappa = 2\\dist(x^0, \\Fix(T)) + {S}$. \n    \n    In particular, if $\\sum_n \\sigma_n <+\\infty$ then  \\eqref{H2} holds. \n\\end{itemize}\n}\n\\end{lemma}","sha256_utf8":"80d0d54d127092a39d8f7a81f36e0ba2f17f10d4f68414970d20323d96d3d49e"},{"start":36806,"end":37781,"start_line":394,"end_line":411,"raw_tex":"\\begin{proof} \nPart $(i)$ follows from the triangular inequality. For part $(ii)$, let us fix $x^*\\in \\Fix(T)$. Using the fact that $T$ is nonexpansive we have\n\\begin{align}\n    \\norm{x^n-x^*} & = \\norm{(1-\\beta_n)(x^0-x^*) + \\beta_n(Tx^{n-1}-x^*) + \\beta_nU_n} \\nonumber \\\\\n    & \\le (1-\\beta_n)\\norm{x^0-x^*} + \\beta_n \\norm{x^{n-1}-x^*} + \\beta_n\\norm{U_n} \\nonumber \\\\\n    &=\\sum_{i=0}^n B^n_{i+1}( \\norm{x^0-x^*}(1-\\beta_i)+\\beta_i\\norm{U_n})\\nonumber  \\\\\n    &=  \\norm{x^0-x^*} + \\sum_{i=1}^n B^n_i \\norm{U_i}, \\label{eq:bounded}\n\\end{align}\n\nwhere we used $\\sum_{i=0}^n B^n_{i+1}(1-\\beta_i)=1$ and the convention $B^n_{n+1} = 1$. Taking expectations, we have\n\\begin{equation*}\n\\begin{aligned}\n\\EE(\\norm{Tx^n-x^0})&\\le \\EE(\\norm{Tx^n-Tx^*})+\\norm{x^*-x^0}\\\\\n&\\le 2 \\norm{x^*-x^0} + \\sum_{i=1}^n B_i^n\\sigma_i \\leq  2 \\norm{x^*-x^0}+ S.\n\\end{aligned}\n\\end{equation*}\nFinally, $\\sup_{n \\geq 0}\\EE(\\norm{Tx^n-x^0})\\leq 2\\inf_{x^* \\in \\Fix T}\\norm{x^*-x^0} +S.$\n\\end{proof}","sha256_utf8":"01425f9a024f2bfef5b756f752e357375e106f164ac9f5187e12ceb042d1a166"},{"start":38146,"end":38616,"start_line":416,"end_line":418,"raw_tex":"\\begin{theorem} \\label{thm:rate-nonexpansive}\n Let $(x^n)_n$ be the sequence generated by the Halpern iteration~\\eqref{halpern} with the stepsize $\\beta_n=\\frac{n}{n+1}$. Assume that \\eqref{H1} holds for some $\\overline \\kappa \\geq 0$ and that $\\sigma_n= \\EE(\\norm{U_n})$ is such that $\\sum_{n=1}^N n\\sigma_n =\\tilde O(1)$. Then, given $\\varepsilon>0$,  $\\EE(\\norm{x^N-Tx^N})\\le \\varepsilon$ for all $N \\geq \\bar N= \\tilde{O}(\\varepsilon^{-1})$ iterations.\n\\end{theorem}","sha256_utf8":"8aba77c8b9a14ca859a7a7663f50801358574cd8ae4715ae65d05759dc08f135"},{"start":38618,"end":40087,"start_line":420,"end_line":434,"raw_tex":"\\begin{proof}\nFor this particular choice of $\\beta_n$, we have that $B_i^n=\\frac{i}{n+1}$ for all $1 \\leq i\\leq n$. Then \n\\[\nS = \\sup_{N\\geq 1}\\sum_{n=1}^N B_n^N  \\sigma_n= \\sup_{N\\geq 1}\\frac{1}{N+1}\\sum_{n=1}^N n \\sigma_n< \\infty, \n\\]\ngiven that $\\sum_{n= 1}^N n \\sigma_n = \\tilde{O}(1)$ and we conclude that \\eqref{H2} holds and \\eqref{H1} is satisfied with $\\bar \\kappa=S+2\\dist(x^0, \\Fix T)$. Now, invoking the estimates given by Proposition~\\ref{prop:residual-nonexpansive} \n\\begin{equation} \\label{bound-expected-nonexpansive} \n\\begin{aligned}\n \\!\\!\\! \\EE(\\norm{ x^N-Tx^N }) & \\le \\! \\frac{\\bar{\\kappa}}{N\\!+\\!1} \\!+\\!  \\frac{1}{N\\!+\\!1}\\sum_{n=1}^N \\left( \\frac{\\bar{\\kappa}}{n\\!+\\!1} \\!+\\! \\frac{n^2}{n\\!+\\!1}\\sigma_n \\!+\\!(n\\!-\\!1)\\sigma_{n-1} \\right ) \\!+\\!\\frac{N}{N\\!+\\!1}\\sigma_N\\\\\n&=\\frac{1}{N\\!+\\!1}\\left(\\bar{\\kappa}\\sum_{n=0}^{N}\\frac{1}{n\\!+\\!1}  \\!+\\!\\sum_{n=1}^N\\frac{n^2}{n\\!+\\!1}\\sigma_n +\\sum_{n=2}^N(n\\!-\\!1)\\sigma_{n-1}+ N \\sigma_N\\right )\\\\\n  &\\le\\frac{\\bar{\\kappa}}{N\\!+\\!1}\\sum_{n=0}^{N}\\frac{1}{n\\!+\\!1}+\\frac{2}{N\\!+\\!1}\\sum_{n=1}^N n\\sigma_n - \\frac{1}{N+1}\\sum_{n=1}^N \\sigma_n,\n\\end{aligned}\n\\end{equation}\nwhere we have used in the second line that $\\sigma_0=0$. We observe that the first term $\\tilde{O}(1/N)$ and the second also is by hypothesis. Therefore, $\\EE(\\norm{x^N-Tx^N}) = \\tilde{O}(1/N)$, and $\\EE(\\norm{ x^N-Tx^N }) \\le \\varepsilon$, for all $N \\geq \\bar N= \\tilde{O}(\\varepsilon^{-1})$ iterations as claimed.\n\\end{proof}","sha256_utf8":"1717fc089dfe7a20f91caec9f535980ba7e791f82e5f4c1a6552e5b62278e8f9"},{"start":42021,"end":42644,"start_line":442,"end_line":448,"raw_tex":"\\begin{lemma}\\label{lem:sigma_explicito}\nUnder the assumptions of Theorem~\\ref{thm:rate-nonexpansive}, let us suppose additionally that there exists $C\\geq 0$ such that $\\sum_{n=1}^N n\\sigma_n \\leq C \\ln(N+1)$ for all $N \\geq 1$. Let $\\rho \\geq 0$ such that $3\\bar{\\kappa} + 2C\\leq \\rho$. Then, for all $\\varepsilon>0$, if $\\varepsilon/\\rho < e^{-1}$, then \n\\[\n\\EE(\\norm{x^N-Tx^N})\\le \\varepsilon \\quad \\text{for all }\\,\\, N \\geq \\mbox{$\\left \\lceil 2\\frac{\\rho}{\\varepsilon} \\ln\\left (\\frac{\\rho}{\\varepsilon}\\right )\\right \\rceil$} .\n\\]\nWhen $\\varepsilon/\\rho \\geq e^{-1}$, then $N=1$ iteration is sufficient.\n\\end{lemma}","sha256_utf8":"d2e41d48bbfc72f126b0690f48116d416d8266ea3079c5a04a9bc2b5f6f3efc4"},{"start":43612,"end":45103,"start_line":457,"end_line":467,"raw_tex":"\\noindent {\\em Proof.} Equation~\\eqref{minibatch} guarantees that \n $\\sigma_n\\le \\mu\\sigma{n^{-2}},$ \n which readily implies that\n $$\\sum_{n=1}^N n\\sigma_n\\leq \\mu\\sigma\\sum_{n=1}^N \\frac{1}{n}\\leq \\frac{3}{2}\\mu\\sigma\\ln(N+1).$$ \n Recalling that for this particular choice of $\\beta_n$, $B_i^n= \\frac{i}{n+1}$, $i=1,\\ldots,n$, we have that\n \\[\n\\sum_{i=1}^n B_i^n\\sigma_i=\\frac{1}{n+1} \\sum_{i=1}^n i \\sigma_i\\leq \\mu\\sigma\\frac{1}{n+1}\\sum_{i=1}^n \\frac{1}{i} \\leq \\frac{\\mu\\sigma}{2}.\n \\]\n Therefore, condition~\\eqref{H2} holds with $S= \\mu\\sigma/2$ and, from Lemma~\\ref{lem:kappa}$(ii)$,  \\eqref{H1} is verified with $\\bar \\kappa= \\mu\\sigma/2 + 2 \\dist(x^0, \\Fix T)$.  Then, using Lemma~\\ref{lem:sigma_explicito} with $C=\\frac{3}{2}\\mu\\sigma$ implies that $\\EE(\\norm{x^N-Tx^N})\\le \\varepsilon$ for all $N$ larger that $ \\bar N= \\left \\lceil 2\\frac{\\rho}{\\varepsilon} \\ln\\left (\\frac{\\rho}{\\varepsilon}\\right )\\right \\rceil $, with \\vspace{-1ex}\n $$\\frac{9}{2}\\mu\\sigma+6 \\dist(x^0, \\Fix T)\\leq 12 \\max \\{\\mu\\sigma,\\dist(x^0, \\Fix T)\\}:=\\rho.$$ Finally, as each iteration $n=1,...,\\bar N$ requires $n^4$ queries to the stochastic oracle, the overall oracle complexity to achieve precision $\\varepsilon>0$ is \n $$ \\qquad \\qquad \\qquad \\qquad \\qquad \\qquad  \\sum_{n=1}^{\\bar N} k_n = \\sum_{n=1}^{\\bar N} n^{4} \\le \\frac{(\\bar N+1)^5}{5}\\leq \\frac{32}{5} \\left (\\frac{\\rho}{\\varepsilon}\\ln\\left (\\frac{\\rho}{\\varepsilon}\\right ) +1\\right )^5. \\qquad \\qquad \\qquad \\qquad \\qquad  \\,\\, \\QEDA$$\n","sha256_utf8":"887c2e6a2327811850b2824ab221956045a845242910edc37bbe7e0c2ac73142"},{"start":97034,"end":99961,"start_line":1148,"end_line":1185,"raw_tex":"\\subsection{Proof of Lemma~\\ref{lem:sigma_explicito}} \\label{apx:explicito}\nAssuming that $\\sum_{n=1}^N n\\sigma_n \\leq C \\ln(N+1)$, the bound \\eqref{bound-expected-nonexpansive} implies that\n\\begin{equation*} \n\\EE(\\norm{ x^N-Tx^N}) \\le \\frac{1}{N+1}\\left( 3\\bar{\\kappa}+2C\\right)\\ln(N+1) \\le \\underbrace{(3\\bar{\\kappa} + 2C)}_{\\leq\\rho}\\frac{\\ln(N+1)}{N+1}, \\end{equation*}\nwhere we have used the rough bound $3\\ln(N+1)\\geq \\sum_{n=1}^{N+1}\\frac{1}{n}$ for all $N \\geq 1$, in order to keep the estimates simple at the expense of some small factor. \n\n\n\nFor $\\varepsilon>0$, let us call $\\tau:=\\frac{\\varepsilon}{\\rho}>0$. It is sufficient to find some $\\bar N\\in \\NN$ such that, for any $N \\geq \\overline{N}$,\n\\[\n\\frac{\\ln(N+1)}{N+1} \\leq \\tau, \\quad \\text{or, equivalently,}\\,\\, -\\tau (N+1)e^{-\\tau(N+1)}\\geq -\\tau.\n\\]\nFirst, note that if $\\tau \\geq e^{-1}$ then the inequality above holds for all $N \\geq 0$. We assume in what follows that $\\tau < e^{-1}$. \n\n\n\nIt is well-known that if we want to solve the equation $-\\tau (x+1)e^{-\\tau(x+1)}= -\\tau$, the solution is given by the relation $-\\tau(x+1) = W(-\\tau)$, where $W$ is the Lambert function. Given that $\\tau >0$, there is two solutions of this equation, provided by the upper and lower branches of $W$, $W_0$ and $W_{-1}$, respectively. We recall that both functions are continuous, defined in the interval $[-e^{-1},0]$, and such that $W_0(-e^{-1})=W_{-1}(-e^{-1})=-1$, $W_0$ increases to cero and $W_{-1}$ decreases to $-\\infty$. This implies \nthat for all $\\tau \\in (0,e^{-1})$, $W_0(-\\tau) \\in (-1,0)$ and that $W_{-1}(-\\tau) \\in (-\\infty, -1)$. Let us pick the solution given by the lower branch, i.e. the one such that $x+1\\geq \\tau^{-1}$. In that case, \n$$x= -\\frac{1}{\\tau}W_{-1}(-\\tau)-1.$$\nTherefore if we set $m=\\lceil -\\frac{1}{\\tau}W_{-1}(-\\tau) -1 \\rceil \\in \\NN$ we obtain that \n$$\\frac{\\ln(N+1)}{N+1} \\leq \\tau, \\quad \\text{for all } N\\in \\NN, N\\geq m.$$\nLet us show that  $m \\leq \\lceil 2\\frac{\\rho}{\\varepsilon} \\ln\\left (\\frac{\\rho}{\\varepsilon}\\right ) \\rceil= \\lceil \\frac{2}{\\tau} \\ln\\left (\\frac{1}{\\tau}\\right ) \\rceil$. For that purpose, we use following lower bound for the function $W_{-1}$ (see \\cite[Theorem 1]{chatzigeorgiou2013bounds}):\n\n$$W_{-1}(-e^{-u-1}) \\geq -1-\\sqrt{2u}-u, \\quad \\text{for all} \\, u>0.$$ \n\nTaking $u=-\\ln(\\tau)-1>0$, we get that $-e^{-u-1}=-\\tau$ and hence\n\\[\n\\begin{aligned}\n \\left \\lceil -\\frac{1}{\\tau}W_{-1}(-\\tau) -1 \\right \\rceil&\\leq \n\\left \\lceil\\frac{1}{\\tau}\\left(1+\\sqrt{2(-\\ln(\\tau)-1)}+(-\\ln(\\tau)-1)\\right )-1\\right\\rceil,\\\\\n & = \n\\left \\lceil\\frac{1}{\\tau}\\left(\\sqrt{-2(\\ln(\\tau)+1)}-\\ln(\\tau)\\right )-1\\right\\rceil\\leq \\left \\lceil 2\\frac{1}{\\tau} \\ln\\left (\\frac{1}{\\tau}\\right )-1\\right \\rceil,\n \\end{aligned}\n\\]\nwhere the last inequality follows from the trivial bound $\\sqrt{-2(u+1)}\\leq -u$ for any $u\\leq -1$. Observe that the last quantity is at least $1$ for all $\\tau \\in (0,e^{-1})$.\n\n\n\n","sha256_utf8":"5bc779a6bbc7b3061f21818709a89ba80a2c97fe66680666434b164cb27b33e1"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"范数空间与非扩张映射","source_units":[{"start":6046,"end":6661,"start_line":141,"end_line":145,"raw_tex":"We consider $\\RR^d$ endowed with a general norm $\\norm{\\,\\cdot\\,}$, and denote by $\\norm{x}_2$ the standard Euclidean norm, {\\em i.e.} $\\norm{x}_2^2 = \\sum_{i=1}^d x_i^2$. In what follows,  $T:\\RR^d \\to \\RR^d$ is such that\n\\[\n\\norm{Tx-Ty} \\le \\gamma\\norm{x-y}, \\qquad \\forall x,y\\in \\RR^d.\n\\]\nWe say that $T$ is a contraction if $\\gamma \\in (0, 1)$, and that $T$ is nonexpansive if $\\gamma = 1$. We assume that in the nonexpansive case $\\Fix \\, T \\neq \\emptyset$, where $\\Fix\\, T$ stands for the set of fixed-points of $T$. We are interested in computing an approximate solution of the fixed-point equation $x=Tx$.\n","sha256_utf8":"1ee7e7d3e02c1089238b504c1fe7944b06c6907624de96b61faeed20ae43f5df"}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","title":"步长与锚点约定","source_units":[{"start":6900,"end":7406,"start_line":151,"end_line":151,"raw_tex":"where $(\\beta_n)_n$ is a nondecreasing stepsize sequence satisfying $0< \\beta_n <1$, $\\lim_{n \\to \\infty}\\beta_n=1$, and $x^0 \\in \\RR^d$ is an arbitrary initial condition.  It is worth mentioning that the original Halpern iteration uses an arbitrary $u$ called {\\em the anchoring point} that can be different from the initial point $x^0$. In that case, the iteration reads $x^{n} = (1-\\beta_{n})u + \\beta_{n}Tx^{n-1}$. We assume here, without loss of generality, that the anchoring point is exactly $x^0$.\n","sha256_utf8":"9df54456d9a8d9b42808aeda372a61be01f9b310bbb953e50e10d1c6df37b67e"}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","title":"无偏随机预言机","source_units":[{"start":7694,"end":8199,"start_line":154,"end_line":160,"raw_tex":"\\begin{assumption} \\label{assumption1} \n    The access to the operator $T$ occurs through a stochastic oracle $\\tilde{T}: \\RR^d\\times \\Xi \\to \\RR^d$ which is an unbiased approximation of $T$ in the sense that \n\\begin{equation*}\n    \\EE(\\tilde{T}(x,\\xi)) = Tx \\qquad \\mbox{for all $x\\in \\RR^d$},\n\\end{equation*}\nwith $\\xi$ drawn from some distribution $\\mathcal D_x$ over $\\Xi$. Additionally, we assume that $\\EE \\left(||\\tilde{T}(x,\\xi)-Tx||^2_2\\right)$ is finite for every $x \\in \\RR^d$.\n\\end{assumption}","sha256_utf8":"7ffa1948e38dc13902e9515e6254e4a51bf456c2d0263e23d46cbb9a16e2f5b1"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"一致有界方差","source_units":[{"start":8200,"end":8467,"start_line":161,"end_line":164,"raw_tex":"We say that the oracle $\\tilde T$ has {\\em uniformly bounded variance} (with bound $\\sigma^2)$ if there exists a positive constant $\\sigma^2$ such that\n\\begin{equation*}\n \\sup_{x \\in \\RR^d}\\EE \\left(\\norm{\\tilde{T}(x,\\xi)-Tx}^2_2\\right)\\leq \\sigma^2.\n\\end{equation*}\n","sha256_utf8":"180adbf16186854d666ca04af06a1e62fd771372470e0b0c2cfcd26583689fd5"}]},{"id":"S005","semantic_type":"algorithm","kind":"algorithm","title":"随机小批量迭代","source_units":[{"start":28515,"end":28874,"start_line":265,"end_line":265,"raw_tex":"Algorithm~\\ref{alg_stochastic_halpern} outlines the stochastic Halpern iteration with minibatches. At each iteration $n \\leq N$, a minibatch estimator is constructed with $k_n$ queries to $\\tilde T$ at $x^{n-1}$ using $k_n$ independent fresh samples from $\\mathcal D_{x^{n-1}}$. Using this estimate, the next iterate is calculated via the Halpern iteration. \n","sha256_utf8":"b9cbda52957644269cab21180ea9469b6e68c8dc46b5399f8e53c74f61422381"},{"start":28876,"end":29509,"start_line":268,"end_line":281,"raw_tex":"\\begin{algorithm}[H]\n\\caption{Stochastic Halpern method}\n\\label{alg_stochastic_halpern}\n\\begin{algorithmic}[1]\n\\Require Stochastic oracle $\\tilde{T}$, $(k_n)_n$ minibatch size sequence, number of iterations $N$, initial point $x^0$    \n\\For{$n=1,\\dots, N$}\n\\State Set the size of the minibatch $k_n$ \n\\State Sample $k_n$ independent queries at $x^{n-1}$: $\\tilde{T}(x^{n-1}, \\xi^n_{1}), \\dots, \\tilde{T}(x^{n-1}, \\xi^n_{k_n})$.    \n\\State Set\n$x^n \\leftarrow (1-\\beta_n)x^0 + \\beta_n\\left(\\frac{1}{k_n}\\displaystyle{\\sum_{j=1}^{k_n}} \\tilde{T}(x^{n-1}, \\xi^n_{j})\\right)$\n\\EndFor\n\\State Return $x^N$\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"94a768329d9756d1b4760a1f76075f8b932de1e7eb47f97f96ee811c71c37642"}]},{"id":"S006","semantic_type":"algorithm","kind":"algorithm","title":"含噪Halpern递推","source_units":[{"start":9371,"end":9533,"start_line":170,"end_line":174,"raw_tex":"\\begin{equation}\n    x^{n} = (1-\\beta_{n})x^0 + \\beta_{n}(Tx^{n-1} + U_{n}), \\quad \\forall \\, n \\geq 1,\n    \\label{halpern}\n    \\tag{$\\textsc{iH}$}\n\\end{equation}","sha256_utf8":"9c5c9939207b2bcf7e40035d898eb9bd77bd21bb8af006da880a52925a69dd86"},{"start":9534,"end":9858,"start_line":175,"end_line":175,"raw_tex":"where $U_n$ is such that $\\EE(U_n)=0$ and the term $\\EE(\\norm{U_n})$ can be controlled appropriately. It is worth noting that we do not assume further properties on the stochastic oracle in order to maintain our results as general as possible. See Section~\\ref{sec:method} for a precise definition and related discussions. \n","sha256_utf8":"edc04ea90b620d57c109b87212eba0a8bd27da5635d64be432caa70c1375afbd"}]},{"id":"S007","semantic_type":"notation","kind":"notation","title":"范数比较常数","source_units":[{"start":10513,"end":10826,"start_line":180,"end_line":180,"raw_tex":"\\noindent {\\bf Notation.} Let us set some basic notation that we use in this paper.  Given that all norms in $\\RR^d$ are equivalent, let $\\mu > 0$ be such that $\\norm{x} \\le \\mu \\norm{x}_2$ for all $x\\in \\RR^d$ and we denote by $\\norm{x}_{\\text{sp}}=\\max_{i}x_i - \\min_{i}x_i$ the so-called span seminorm of $x$.\n","sha256_utf8":"6a3b71e7a88e4a59ac63a10a2b4c0e46a38effd64c63ab6f79e758c095e022ed"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"复杂度与渐近记号","source_units":[{"start":9859,"end":10499,"start_line":177,"end_line":177,"raw_tex":"We are interested in the problem of estimating the oracle complexity to find an approximate fixed-point of $T$, which is measured as the number of calls to the vector stochastic oracle $\\tilde T$. In the case where $T$ is nonexpansive, we estimate the oracle complexity to find a last-iterate approximate solution $x^n$ in the sense that the expected fixed-point residual $\\EE (\\norm{x^n - Tx^n})$ is smaller than a given tolerance. For a contracting operator $T$, we aim to find oracle complexity bounds for the expected distance of the last iterate $x^n$ to the unique fixed-point $x^*$, {\\em i.e.} the quantity $\\EE (\\norm{x^n - x^*})$.\n","sha256_utf8":"b47b9c76220d3810c9c80d9e2781b4224504fc86baa3094cbb2bafb068060428"},{"start":10826,"end":11139,"start_line":181,"end_line":182,"raw_tex":"\nWe adopt the standard big-$O$ notation, that is, for two real functions $f,g:$ we say $f(x)=O(g(x))$ if there are two constants $C$ and $c$ such that $f(x)\\leq C g(x)$ for all $x\\geq c$. We use $\\tilde O$ if we ignore logarithmic factors. Analogously, $f(x)=\\Omega(g(x))$ if $f(x)\\geq C g(x)$ for all $x\\geq c$.\n","sha256_utf8":"eb3c4d3158364839ebd001dbacce135c97b989a77ade47aff24167248ac30737"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"小批量误差定义","source_units":[{"start":29510,"end":30037,"start_line":282,"end_line":288,"raw_tex":"Notice that if we define the error term at the iteration $n \\geq 1$  by \n\\begin{equation*}\nU_n := \\frac{1}{k_n}\\displaystyle{\\sum_{j=1}^{k_n}} \\tilde{T}(x^{n-1}, \\xi^n_{j}) - Tx^{n-1},\n\\end{equation*}\nthen Algorithm~\\ref{alg_stochastic_halpern} can be cast in the form \\eqref{halpern}. Indeed, we will refer to both procedures indistinctly when there is no confusion.\n\nBy construction, $Tx^{n-1} + U_n$ is an unbiased estimator of $Tx^{n-1}$ and may have a reduced variance compared to the error computed with a single query. \n","sha256_utf8":"255b468fd5b9a441e79d25731cb4d93c71539c3d78278cabc7254c976ec75c4b"}]},{"id":"S010","semantic_type":"notation","kind":"notation","title":"误差均值与步长乘积","source_units":[{"start":30111,"end":30426,"start_line":290,"end_line":297,"raw_tex":"\\begin{equation}    \\label{notation_beta_sigma}\n\\left \\{\n\\begin{aligned} \n    &\\sigma_n:=\\EE(\\norm{U_n})\\,\\,\\, \\text{and}\\,\\,\\,\\sigma_0=\\beta_0=0;\\\\\n    & B_i^n:= \\prod_{j=i}^n \\beta_j, \\,\\,\\,\\text{with the convention}\\,\\,\\,  B_{i}^n=1 \\,\\,\\, \\text{if}\\,\\,\\,  0\\leq n<i.\n    \\end{aligned}\n    \\right.\n\\end{equation}","sha256_utf8":"c550329fbd4044ccfe5fb68ce8dc6078ce125dad8b65e9507075745ee9c8a393"}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","title":"映像期望有界条件","source_units":[{"start":32025,"end":32239,"start_line":312,"end_line":314,"raw_tex":"\\begin{equation}\\tag{$\\textsc{H}_1$} \\label{H1}\n\\text{There exists } \\overline \\kappa <+\\infty \\, \\text{ such that }\\, \\sup \\nolimits_{n \\geq 0}\\EE (\\norm{T x^n-x^0}) \\leq \\overline \\kappa < +\\infty.\n\\end{equation}","sha256_utf8":"0eeab37ea2c483d82a2c6156c5bc4313c6e055a32130575d167bf0c67c28b372"}]},{"id":"S012","semantic_type":"proof","kind":"proof","title":"小批量误差界证明","source_units":[{"start":30427,"end":31247,"start_line":298,"end_line":305,"raw_tex":"For the sake of completeness, we recall how a standard argument allows to transfer the variance reduction into a bound on $\\sigma_n$. \n\nFor a given iteration $n$, let $\\xi^n_{1},\\ldots,\\xi^n_{k_n}$ be $k_n$ independent samples of the distribution $\\mathcal D_{x^{n-1}}$ and let $e_n^2=\\EE (||\\tilde{T}(x^{n-1}, \\xi^n_{1}) - T x^{n-1}||_2^2)$ be the variance of the error term of one query. Then\n\\begin{equation}\n\\sigma_n^2=\\EE(\\norm{U_n})^2 \\le \\frac{\\mu^2}{k_n^2} \\EE\\left(\\norm{\\sum \\nolimits_{j=1}^{k_n} \\tilde{T}(x^{n-1}, \\xi^n_{j}) - Tx^{n-1} }^2_2\\right) \\le \\frac{\\mu^2}{k_n}e_n^2,\n\\label{minibatch}\n\\end{equation}\nwhere we have use Jensen's inequality, sample independence, and the equivalence between the norms. We observe that $\\EE(\\norm{U_n})$ scales by a factor $\\frac{\\mu}{\\sqrt{k_n}}$ compared to $e_n$.  \n","sha256_utf8":"da05f46f390db7ca01a36c9d0c09edcd100889ff1ba0208f5bcf3d76e47aa4d8"}]},{"id":"S013","semantic_type":"proof_step","kind":"proof_step","title":"独立样本与单次方差","source_units":[{"start":30427,"end":30822,"start_line":298,"end_line":300,"raw_tex":"For the sake of completeness, we recall how a standard argument allows to transfer the variance reduction into a bound on $\\sigma_n$. \n\nFor a given iteration $n$, let $\\xi^n_{1},\\ldots,\\xi^n_{k_n}$ be $k_n$ independent samples of the distribution $\\mathcal D_{x^{n-1}}$ and let $e_n^2=\\EE (||\\tilde{T}(x^{n-1}, \\xi^n_{1}) - T x^{n-1}||_2^2)$ be the variance of the error term of one query. Then\n","sha256_utf8":"4b9a4ab0d1b0c454d72ad24d2c7d50e624f5dba19335eb021cf730a7d9adde4f"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"方差缩减推出误差界","source_units":[{"start":30822,"end":31247,"start_line":301,"end_line":305,"raw_tex":"\\begin{equation}\n\\sigma_n^2=\\EE(\\norm{U_n})^2 \\le \\frac{\\mu^2}{k_n^2} \\EE\\left(\\norm{\\sum \\nolimits_{j=1}^{k_n} \\tilde{T}(x^{n-1}, \\xi^n_{j}) - Tx^{n-1} }^2_2\\right) \\le \\frac{\\mu^2}{k_n}e_n^2,\n\\label{minibatch}\n\\end{equation}\nwhere we have use Jensen's inequality, sample independence, and the equivalence between the norms. We observe that $\\EE(\\norm{U_n})$ scales by a factor $\\frac{\\mu}{\\sqrt{k_n}}$ compared to $e_n$.  \n","sha256_utf8":"8d3cc75f77c12d004a2f5bcc4e0bc5031112c786c109e55c5dca921e1ce8ed4c"}]},{"id":"S015","semantic_type":"corollary","kind":"corollary","title":"非扩张情形预言机复杂度","source_units":[{"start":43051,"end":43597,"start_line":453,"end_line":455,"raw_tex":"\\begin{corollary} \\label{cor:bounded_variance}\n Assume that $\\tilde{T}$ has uniformly bounded variance $\\sigma^2$. Let $(x^n)_n$ be the sequence generated by the Halpern iteration~\\eqref{halpern} with the stepsize $\\beta_n=\\frac{n}{n+1}$ and batch size $k_n = n^{4}$. Then, given $\\varepsilon>0$, the Algorithm~\\ref{alg_stochastic_halpern} finds $x^N$ such that $\\EE(\\norm{x^N-Tx^N})\\le \\varepsilon$ using at most $\\tilde{O}(L^5\\varepsilon^{-5})$ queries to the stochastic oracle, where $L=\\max\\{\\mu \\sigma, \\dist(x^0, \\Fix T)\\}$.\n\\end{corollary}","sha256_utf8":"8209e00569908a70353b64656272441380efc14425b7a8f4408721c5ba552d28"}]},{"id":"S016","semantic_type":"proof","kind":"proof","title":"复杂度推论完整证明","source_units":[{"start":43612,"end":45103,"start_line":457,"end_line":467,"raw_tex":"\\noindent {\\em Proof.} Equation~\\eqref{minibatch} guarantees that \n $\\sigma_n\\le \\mu\\sigma{n^{-2}},$ \n which readily implies that\n $$\\sum_{n=1}^N n\\sigma_n\\leq \\mu\\sigma\\sum_{n=1}^N \\frac{1}{n}\\leq \\frac{3}{2}\\mu\\sigma\\ln(N+1).$$ \n Recalling that for this particular choice of $\\beta_n$, $B_i^n= \\frac{i}{n+1}$, $i=1,\\ldots,n$, we have that\n \\[\n\\sum_{i=1}^n B_i^n\\sigma_i=\\frac{1}{n+1} \\sum_{i=1}^n i \\sigma_i\\leq \\mu\\sigma\\frac{1}{n+1}\\sum_{i=1}^n \\frac{1}{i} \\leq \\frac{\\mu\\sigma}{2}.\n \\]\n Therefore, condition~\\eqref{H2} holds with $S= \\mu\\sigma/2$ and, from Lemma~\\ref{lem:kappa}$(ii)$,  \\eqref{H1} is verified with $\\bar \\kappa= \\mu\\sigma/2 + 2 \\dist(x^0, \\Fix T)$.  Then, using Lemma~\\ref{lem:sigma_explicito} with $C=\\frac{3}{2}\\mu\\sigma$ implies that $\\EE(\\norm{x^N-Tx^N})\\le \\varepsilon$ for all $N$ larger that $ \\bar N= \\left \\lceil 2\\frac{\\rho}{\\varepsilon} \\ln\\left (\\frac{\\rho}{\\varepsilon}\\right )\\right \\rceil $, with \\vspace{-1ex}\n $$\\frac{9}{2}\\mu\\sigma+6 \\dist(x^0, \\Fix T)\\leq 12 \\max \\{\\mu\\sigma,\\dist(x^0, \\Fix T)\\}:=\\rho.$$ Finally, as each iteration $n=1,...,\\bar N$ requires $n^4$ queries to the stochastic oracle, the overall oracle complexity to achieve precision $\\varepsilon>0$ is \n $$ \\qquad \\qquad \\qquad \\qquad \\qquad \\qquad  \\sum_{n=1}^{\\bar N} k_n = \\sum_{n=1}^{\\bar N} n^{4} \\le \\frac{(\\bar N+1)^5}{5}\\leq \\frac{32}{5} \\left (\\frac{\\rho}{\\varepsilon}\\ln\\left (\\frac{\\rho}{\\varepsilon}\\right ) +1\\right )^5. \\qquad \\qquad \\qquad \\qquad \\qquad  \\,\\, \\QEDA$$\n","sha256_utf8":"887c2e6a2327811850b2824ab221956045a845242910edc37bbe7e0c2ac73142"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"四次批量控制加权误差","source_units":[{"start":43612,"end":43843,"start_line":457,"end_line":460,"raw_tex":"\\noindent {\\em Proof.} Equation~\\eqref{minibatch} guarantees that \n $\\sigma_n\\le \\mu\\sigma{n^{-2}},$ \n which readily implies that\n $$\\sum_{n=1}^N n\\sigma_n\\leq \\mu\\sigma\\sum_{n=1}^N \\frac{1}{n}\\leq \\frac{3}{2}\\mu\\sigma\\ln(N+1).$$ \n","sha256_utf8":"182a5b4b10728c4ae695c3f5e83fcdf37aa25c4f1920b8b0ad4f649a15526691"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"验证加权噪声有界","source_units":[{"start":43843,"end":44103,"start_line":461,"end_line":464,"raw_tex":" Recalling that for this particular choice of $\\beta_n$, $B_i^n= \\frac{i}{n+1}$, $i=1,\\ldots,n$, we have that\n \\[\n\\sum_{i=1}^n B_i^n\\sigma_i=\\frac{1}{n+1} \\sum_{i=1}^n i \\sigma_i\\leq \\mu\\sigma\\frac{1}{n+1}\\sum_{i=1}^n \\frac{1}{i} \\leq \\frac{\\mu\\sigma}{2}.\n \\]\n","sha256_utf8":"d71f4665fb37a60c5c00599138f7bfbe1188902a4fd897a120b4aad15dd7d3d7"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"确定迭代数并累计查询","source_units":[{"start":44103,"end":45103,"start_line":465,"end_line":467,"raw_tex":" Therefore, condition~\\eqref{H2} holds with $S= \\mu\\sigma/2$ and, from Lemma~\\ref{lem:kappa}$(ii)$,  \\eqref{H1} is verified with $\\bar \\kappa= \\mu\\sigma/2 + 2 \\dist(x^0, \\Fix T)$.  Then, using Lemma~\\ref{lem:sigma_explicito} with $C=\\frac{3}{2}\\mu\\sigma$ implies that $\\EE(\\norm{x^N-Tx^N})\\le \\varepsilon$ for all $N$ larger that $ \\bar N= \\left \\lceil 2\\frac{\\rho}{\\varepsilon} \\ln\\left (\\frac{\\rho}{\\varepsilon}\\right )\\right \\rceil $, with \\vspace{-1ex}\n $$\\frac{9}{2}\\mu\\sigma+6 \\dist(x^0, \\Fix T)\\leq 12 \\max \\{\\mu\\sigma,\\dist(x^0, \\Fix T)\\}:=\\rho.$$ Finally, as each iteration $n=1,...,\\bar N$ requires $n^4$ queries to the stochastic oracle, the overall oracle complexity to achieve precision $\\varepsilon>0$ is \n $$ \\qquad \\qquad \\qquad \\qquad \\qquad \\qquad  \\sum_{n=1}^{\\bar N} k_n = \\sum_{n=1}^{\\bar N} n^{4} \\le \\frac{(\\bar N+1)^5}{5}\\leq \\frac{32}{5} \\left (\\frac{\\rho}{\\varepsilon}\\ln\\left (\\frac{\\rho}{\\varepsilon}\\right ) +1\\right )^5. \\qquad \\qquad \\qquad \\qquad \\qquad  \\,\\, \\QEDA$$\n","sha256_utf8":"6963d3d33e1be670520b86ea96cb3b36c812887029d2334f6c50bbf61723cdf6"}]},{"id":"S020","semantic_type":"proposition","kind":"proposition","title":"一般步长残差上界","source_units":[{"start":32459,"end":32972,"start_line":321,"end_line":327,"raw_tex":"\\begin{proposition} \\label{prop:residual-nonexpansive}\nLet $(x^n)_{n\\ge 1}$ the sequence generated by the Halpern iteration \\eqref{halpern}. Assume that \\eqref{H1} holds for some $\\overline \\kappa \\geq 0$. Then, for all $n\\ge 1$\n\\[\n\\EE(\\norm{x^n-Tx^n}) \\le \\overline\\kappa(1-\\beta_n) + \\sum_{i=1}^{n} B_i^n \\left(\\overline\\kappa(\\beta_i -\\beta_{i-1}) + \\beta_i\\sigma_i+\\beta_{i-1}\\sigma_{i-1} \\right )+ \\beta_n\\sigma_n,\n\\]\nwhere $B^n_i$ and $\\sigma_i$ are defined in \\eqref{notation_beta_sigma}.\n\\end{proposition}","sha256_utf8":"82321ffd9f4cb4a13daa21aa1e6f89786a8937b5cddefec08d2859c5824721c5"}]},{"id":"S021","semantic_type":"proof","kind":"proof","title":"残差上界完整证明","source_units":[{"start":32974,"end":35781,"start_line":329,"end_line":373,"raw_tex":"\\begin{proof}\nRecall that $\\sigma_0=\\beta_0=0$ and let $\\kappa_n= \\norm{Tx^n-x^0}$. We start by observing that the (random) sequence $(P_n)_n$ defined inductively as $P_0=0$ and\n\\[ P_n=\n\\kappa_{n-1} (\\beta_n-\\beta_{n-1})+\\beta_{n-1} P_{n-1}+\\beta_n\\norm{U_n}+\\beta_{n-1}\\norm{U_{n-1}},\\,\\,  n\\geq 1,\n\\]\nis such that $\\norm{x^n-x^{n-1}}\\leq P_n$ for all $n \\in \\NN$. Indeed, for the first term we write\n\\[\n\\norm{x^1-x^0}= \\beta_1\\norm{x^0 -Tx^0 + U_1} \\leq \\kappa_0 (\\beta_1-\\beta_0) + \\beta_1 \\norm{U_1}=P_1.\n\\]\nFrom the definition of \\eqref{halpern}, using again the triangular inequality, the nonexpansivity of $T$, and the induction hypothesis, we have that for $n \\geq 2$,\n\\begin{equation*}\n\\begin{aligned}\n\\!\\!\\norm{x^n\\!-\\!x^{n-1}} &=\\norm{(1\\!-\\! \\beta_{n}) x^0 + \\beta_n(Tx^{n-1} +U_n)-(1\\!-\\!\\beta_{n-1})x^0 \\!-\\! \\beta_{n-1}(Tx^{n-2} +U_{n\\!-\\!1})}\\\\\n&=\\norm{(\\beta_{n}\\!-\\!\\beta_{n-1})(T x^{n-1}\\!-\\!x^0)\\! + \\!\\beta_{n-1}(Tx^{n-1}\\!\\!-Tx^{n-2}) \\!+\\!\\beta_{n} U_n\\!-\\!\\beta_{n-1}U_{n-1}} \\\\\n&\\leq (\\beta_n-\\beta_{n-1})\\norm{T x^{n-1}\\!-\\!x^0}\\!+\\! \\beta_{n-1}\\norm{x^{n-1}\\!-\\!x^{n-2}}\\!+\\!\\beta_n\\norm{U_{n}}\\!+\\!\\beta_{n-1}\\norm{U_{n-1}} \\\\\n&\\le \\kappa_{n-1} (\\beta_n-\\beta_{n-1})+\\beta_{n-1} P_{n-1}+\\beta_n\\norm{U_{n}}\\!+\\!\\beta_{n-1}\\norm{U_{n-1}}=P_n.\n\\end{aligned}\n\\end{equation*}\n\nThen, it follows that $P_n$ can be computed explicitly as\n\\[\nP_n=\\sum_{i=1}^{n}B_i^{n-1} \\left ( \\kappa_{i-1}(\\beta_i -\\beta_{i-1}) + \\beta_i\\norm{U_{i}}+\\beta_{i-1}\\norm{U_{i-1}} \\right ).\n\\]\nwhere we used the convention $B^{n-1}_n = 1$. Now, casting \\eqref{halpern} as\n   \\[\n    x^{n} - Tx^n = (1-\\beta_{n})(x^0-Tx^{n}) + \\beta_{n} (Tx^{n-1}-Tx^n + U_{n}), \\quad \\forall \\, n \\geq 1,\n   \\]\nwe get that \n\\begin{equation}\\label{eq:residual}\n\\begin{aligned}\n\\!\\!\\!\\norm{x^n-\\!Tx^n} & \\!\\!=\\! \\!\\norm{(1\\!-\\!\\beta_n)(x^0-Tx^{n})+\\beta_n(Tx^{n-1}-Tx^n+U_n )} \\\\\n&\\le \\kappa_n (1-\\beta_n) +  \\beta_n  \\norm{x^n-x^{n-1}}+ \\beta_n\\norm{U_n} \\\\\n&\\leq \\kappa_n(1-\\beta_n) +  \\beta_n  P_n+ \\beta_n\\norm{U_n},\\\\\n& = \\kappa_n(1-\\!\\beta_n)\\! +\\! \\\\\n&+\\sum_{i=1}^{n}B_i^{n}\\left ( \\!\\kappa_{i-1}(\\beta_i -\\beta_{i-1})\\! + \\!\\beta_i\\norm{U_i}\\!\\!+\\!\\!\\beta_{i-1}\\norm{U_{i-1}} \\!\\right )+ \\beta_n\\norm{U_n}\n\\end{aligned}\n\\end{equation}\nwhere we used that $B^n_i = \\beta_nB^{n-1}_i$ for every $i=1,...,n$ and that $B^{n-1}_{n} = 1$. Taking expectations in the expression above, and using that $\\sigma_i = \\EE(\\norm{U_i})$, we obtain\n\\begin{equation*}\n\\begin{aligned}\n\\EE(\\norm{x^n\\!\\!-\\!\\!Tx^n}) &\\leq \\EE(\\kappa_n)(1-\\!\\beta_n)+ \\sum_{i=1}^{n}B_i^{n}\\left ( \\EE(\\kappa_{i-1})(\\beta_i -\\beta_{i-1})\\! + \\!\\beta_i\\sigma_i +\\beta_{i\\!-\\!1}\\sigma_{i-1} \\!\\right )\\!+\\! \\beta_n\\sigma_n. \n\\end{aligned}\n\\end{equation*}\nThe conclusion follows from the fact that $\\sup_{n\\geq 0} \\EE(\\kappa_n) \\leq \\overline \\kappa$.\n\\end{proof}","sha256_utf8":"445e547c9bf4646c857c09075ed740562d5b567fe848afdb7d4652b5090bff6c"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"构造增量界并验证初项","source_units":[{"start":32974,"end":33486,"start_line":329,"end_line":337,"raw_tex":"\\begin{proof}\nRecall that $\\sigma_0=\\beta_0=0$ and let $\\kappa_n= \\norm{Tx^n-x^0}$. We start by observing that the (random) sequence $(P_n)_n$ defined inductively as $P_0=0$ and\n\\[ P_n=\n\\kappa_{n-1} (\\beta_n-\\beta_{n-1})+\\beta_{n-1} P_{n-1}+\\beta_n\\norm{U_n}+\\beta_{n-1}\\norm{U_{n-1}},\\,\\,  n\\geq 1,\n\\]\nis such that $\\norm{x^n-x^{n-1}}\\leq P_n$ for all $n \\in \\NN$. Indeed, for the first term we write\n\\[\n\\norm{x^1-x^0}= \\beta_1\\norm{x^0 -Tx^0 + U_1} \\leq \\kappa_0 (\\beta_1-\\beta_0) + \\beta_1 \\norm{U_1}=P_1.\n\\]\n","sha256_utf8":"7e2796b125a95363de5451319af95657fbecc41d9f20f0c7fa6ea11e7c8dcc25"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"归纳控制相邻迭代差","source_units":[{"start":33486,"end":34274,"start_line":338,"end_line":346,"raw_tex":"From the definition of \\eqref{halpern}, using again the triangular inequality, the nonexpansivity of $T$, and the induction hypothesis, we have that for $n \\geq 2$,\n\\begin{equation*}\n\\begin{aligned}\n\\!\\!\\norm{x^n\\!-\\!x^{n-1}} &=\\norm{(1\\!-\\! \\beta_{n}) x^0 + \\beta_n(Tx^{n-1} +U_n)-(1\\!-\\!\\beta_{n-1})x^0 \\!-\\! \\beta_{n-1}(Tx^{n-2} +U_{n\\!-\\!1})}\\\\\n&=\\norm{(\\beta_{n}\\!-\\!\\beta_{n-1})(T x^{n-1}\\!-\\!x^0)\\! + \\!\\beta_{n-1}(Tx^{n-1}\\!\\!-Tx^{n-2}) \\!+\\!\\beta_{n} U_n\\!-\\!\\beta_{n-1}U_{n-1}} \\\\\n&\\leq (\\beta_n-\\beta_{n-1})\\norm{T x^{n-1}\\!-\\!x^0}\\!+\\! \\beta_{n-1}\\norm{x^{n-1}\\!-\\!x^{n-2}}\\!+\\!\\beta_n\\norm{U_{n}}\\!+\\!\\beta_{n-1}\\norm{U_{n-1}} \\\\\n&\\le \\kappa_{n-1} (\\beta_n-\\beta_{n-1})+\\beta_{n-1} P_{n-1}+\\beta_n\\norm{U_{n}}\\!+\\!\\beta_{n-1}\\norm{U_{n-1}}=P_n.\n\\end{aligned}\n\\end{equation*}\n","sha256_utf8":"20967f089503819530550fa27d606cce6a820a0d9390e2fe04761ca1eddda6f1"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"展开增量界与残差恒等式","source_units":[{"start":34275,"end":34667,"start_line":348,"end_line":355,"raw_tex":"Then, it follows that $P_n$ can be computed explicitly as\n\\[\nP_n=\\sum_{i=1}^{n}B_i^{n-1} \\left ( \\kappa_{i-1}(\\beta_i -\\beta_{i-1}) + \\beta_i\\norm{U_{i}}+\\beta_{i-1}\\norm{U_{i-1}} \\right ).\n\\]\nwhere we used the convention $B^{n-1}_n = 1$. Now, casting \\eqref{halpern} as\n   \\[\n    x^{n} - Tx^n = (1-\\beta_{n})(x^0-Tx^{n}) + \\beta_{n} (Tx^{n-1}-Tx^n + U_{n}), \\quad \\forall \\, n \\geq 1,\n   \\]\n","sha256_utf8":"9ebe37c85011cb75278c063ed307a39ea658a9d606f10c5ec639cce5a1184c17"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"获得逐路径残差估计","source_units":[{"start":34667,"end":35197,"start_line":356,"end_line":365,"raw_tex":"we get that \n\\begin{equation}\\label{eq:residual}\n\\begin{aligned}\n\\!\\!\\!\\norm{x^n-\\!Tx^n} & \\!\\!=\\! \\!\\norm{(1\\!-\\!\\beta_n)(x^0-Tx^{n})+\\beta_n(Tx^{n-1}-Tx^n+U_n )} \\\\\n&\\le \\kappa_n (1-\\beta_n) +  \\beta_n  \\norm{x^n-x^{n-1}}+ \\beta_n\\norm{U_n} \\\\\n&\\leq \\kappa_n(1-\\beta_n) +  \\beta_n  P_n+ \\beta_n\\norm{U_n},\\\\\n& = \\kappa_n(1-\\!\\beta_n)\\! +\\! \\\\\n&+\\sum_{i=1}^{n}B_i^{n}\\left ( \\!\\kappa_{i-1}(\\beta_i -\\beta_{i-1})\\! + \\!\\beta_i\\norm{U_i}\\!\\!+\\!\\!\\beta_{i-1}\\norm{U_{i-1}} \\!\\right )+ \\beta_n\\norm{U_n}\n\\end{aligned}\n\\end{equation}\n","sha256_utf8":"43382c1c3147c3a25cb967e62dbd8157991ab64fa8948284c6ffe400cf1db9f3"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"取期望并应用有界条件","source_units":[{"start":35197,"end":35781,"start_line":366,"end_line":373,"raw_tex":"where we used that $B^n_i = \\beta_nB^{n-1}_i$ for every $i=1,...,n$ and that $B^{n-1}_{n} = 1$. Taking expectations in the expression above, and using that $\\sigma_i = \\EE(\\norm{U_i})$, we obtain\n\\begin{equation*}\n\\begin{aligned}\n\\EE(\\norm{x^n\\!\\!-\\!\\!Tx^n}) &\\leq \\EE(\\kappa_n)(1-\\!\\beta_n)+ \\sum_{i=1}^{n}B_i^{n}\\left ( \\EE(\\kappa_{i-1})(\\beta_i -\\beta_{i-1})\\! + \\!\\beta_i\\sigma_i +\\beta_{i\\!-\\!1}\\sigma_{i-1} \\!\\right )\\!+\\! \\beta_n\\sigma_n. \n\\end{aligned}\n\\end{equation*}\nThe conclusion follows from the fact that $\\sup_{n\\geq 0} \\EE(\\kappa_n) \\leq \\overline \\kappa$.\n\\end{proof}","sha256_utf8":"03842a5ca65af5d370dfe1206010d6a89d660215d5fadcb4c8f63a32ca5b7032"}]},{"id":"S027","semantic_type":"lemma","kind":"lemma","title":"映像有界的充分条件","source_units":[{"start":35985,"end":36804,"start_line":379,"end_line":392,"raw_tex":"\\begin{lemma} \\label{lem:kappa}\n{\\em Let $T$ be nonexpansive and $(x^n)_n$ be the sequence generated by \\eqref{halpern}. Recalling the notation \\eqref{notation_beta_sigma}, it holds:\n\\begin{itemize}\n    \\item[$(i)$] If $T$ has bounded range i.e. there exist a constant $M>0$ such that $\\norm{Tx} \\le M$ for all $x\\in \\RR^d$, then the condition \\eqref{H1} holds with $\\bar \\kappa = M + \\norm{x^0}$.\n    \\item[$(ii)$] If the sequences $(\\beta_n)_n$ and $(\\sigma_n)_n$ satisfies\n    \\begin{equation}\\label{H2}\\tag{$\\textsc{H}_2$}\n    \\sup_{n\\ge 1}\\sum_{i=1}^n B_i^n\\sigma_i\\leq S < +\\infty,,\n    \\end{equation}\n    for some $S\\ge 0$, then condition \\eqref{H1} holds with $\\bar \\kappa = 2\\dist(x^0, \\Fix(T)) + {S}$. \n    \n    In particular, if $\\sum_n \\sigma_n <+\\infty$ then  \\eqref{H2} holds. \n\\end{itemize}\n}\n\\end{lemma}","sha256_utf8":"80d0d54d127092a39d8f7a81f36e0ba2f17f10d4f68414970d20323d96d3d49e"}]},{"id":"S028","semantic_type":"proof","kind":"proof","title":"有界性引理完整证明","source_units":[{"start":36806,"end":37781,"start_line":394,"end_line":411,"raw_tex":"\\begin{proof} \nPart $(i)$ follows from the triangular inequality. For part $(ii)$, let us fix $x^*\\in \\Fix(T)$. Using the fact that $T$ is nonexpansive we have\n\\begin{align}\n    \\norm{x^n-x^*} & = \\norm{(1-\\beta_n)(x^0-x^*) + \\beta_n(Tx^{n-1}-x^*) + \\beta_nU_n} \\nonumber \\\\\n    & \\le (1-\\beta_n)\\norm{x^0-x^*} + \\beta_n \\norm{x^{n-1}-x^*} + \\beta_n\\norm{U_n} \\nonumber \\\\\n    &=\\sum_{i=0}^n B^n_{i+1}( \\norm{x^0-x^*}(1-\\beta_i)+\\beta_i\\norm{U_n})\\nonumber  \\\\\n    &=  \\norm{x^0-x^*} + \\sum_{i=1}^n B^n_i \\norm{U_i}, \\label{eq:bounded}\n\\end{align}\n\nwhere we used $\\sum_{i=0}^n B^n_{i+1}(1-\\beta_i)=1$ and the convention $B^n_{n+1} = 1$. Taking expectations, we have\n\\begin{equation*}\n\\begin{aligned}\n\\EE(\\norm{Tx^n-x^0})&\\le \\EE(\\norm{Tx^n-Tx^*})+\\norm{x^*-x^0}\\\\\n&\\le 2 \\norm{x^*-x^0} + \\sum_{i=1}^n B_i^n\\sigma_i \\leq  2 \\norm{x^*-x^0}+ S.\n\\end{aligned}\n\\end{equation*}\nFinally, $\\sup_{n \\geq 0}\\EE(\\norm{Tx^n-x^0})\\leq 2\\inf_{x^* \\in \\Fix T}\\norm{x^*-x^0} +S.$\n\\end{proof}","sha256_utf8":"01425f9a024f2bfef5b756f752e357375e106f164ac9f5187e12ceb042d1a166"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"有界映像与固定点距离递推","source_units":[{"start":36806,"end":37354,"start_line":394,"end_line":401,"raw_tex":"\\begin{proof} \nPart $(i)$ follows from the triangular inequality. For part $(ii)$, let us fix $x^*\\in \\Fix(T)$. Using the fact that $T$ is nonexpansive we have\n\\begin{align}\n    \\norm{x^n-x^*} & = \\norm{(1-\\beta_n)(x^0-x^*) + \\beta_n(Tx^{n-1}-x^*) + \\beta_nU_n} \\nonumber \\\\\n    & \\le (1-\\beta_n)\\norm{x^0-x^*} + \\beta_n \\norm{x^{n-1}-x^*} + \\beta_n\\norm{U_n} \\nonumber \\\\\n    &=\\sum_{i=0}^n B^n_{i+1}( \\norm{x^0-x^*}(1-\\beta_i)+\\beta_i\\norm{U_n})\\nonumber  \\\\\n    &=  \\norm{x^0-x^*} + \\sum_{i=1}^n B^n_i \\norm{U_i}, \\label{eq:bounded}\n\\end{align}\n","sha256_utf8":"330ddc6f0d25efe18f3b11d2e5fa5c2663f6536c9c305fab406b5308370bf57d"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"取期望并最小化初始距离","source_units":[{"start":37355,"end":37781,"start_line":403,"end_line":411,"raw_tex":"where we used $\\sum_{i=0}^n B^n_{i+1}(1-\\beta_i)=1$ and the convention $B^n_{n+1} = 1$. Taking expectations, we have\n\\begin{equation*}\n\\begin{aligned}\n\\EE(\\norm{Tx^n-x^0})&\\le \\EE(\\norm{Tx^n-Tx^*})+\\norm{x^*-x^0}\\\\\n&\\le 2 \\norm{x^*-x^0} + \\sum_{i=1}^n B_i^n\\sigma_i \\leq  2 \\norm{x^*-x^0}+ S.\n\\end{aligned}\n\\end{equation*}\nFinally, $\\sup_{n \\geq 0}\\EE(\\norm{Tx^n-x^0})\\leq 2\\inf_{x^* \\in \\Fix T}\\norm{x^*-x^0} +S.$\n\\end{proof}","sha256_utf8":"bb90f22a357896a55382ee7d5aabeaf6b5c2339bf9e812d1796d1123941cefb5"}]},{"id":"S031","semantic_type":"theorem","kind":"theorem","title":"特殊步长期望残差速率","source_units":[{"start":38146,"end":38616,"start_line":416,"end_line":418,"raw_tex":"\\begin{theorem} \\label{thm:rate-nonexpansive}\n Let $(x^n)_n$ be the sequence generated by the Halpern iteration~\\eqref{halpern} with the stepsize $\\beta_n=\\frac{n}{n+1}$. Assume that \\eqref{H1} holds for some $\\overline \\kappa \\geq 0$ and that $\\sigma_n= \\EE(\\norm{U_n})$ is such that $\\sum_{n=1}^N n\\sigma_n =\\tilde O(1)$. Then, given $\\varepsilon>0$,  $\\EE(\\norm{x^N-Tx^N})\\le \\varepsilon$ for all $N \\geq \\bar N= \\tilde{O}(\\varepsilon^{-1})$ iterations.\n\\end{theorem}","sha256_utf8":"8aba77c8b9a14ca859a7a7663f50801358574cd8ae4715ae65d05759dc08f135"}]},{"id":"S032","semantic_type":"proof","kind":"proof","title":"残差速率完整证明","source_units":[{"start":38618,"end":40087,"start_line":420,"end_line":434,"raw_tex":"\\begin{proof}\nFor this particular choice of $\\beta_n$, we have that $B_i^n=\\frac{i}{n+1}$ for all $1 \\leq i\\leq n$. Then \n\\[\nS = \\sup_{N\\geq 1}\\sum_{n=1}^N B_n^N  \\sigma_n= \\sup_{N\\geq 1}\\frac{1}{N+1}\\sum_{n=1}^N n \\sigma_n< \\infty, \n\\]\ngiven that $\\sum_{n= 1}^N n \\sigma_n = \\tilde{O}(1)$ and we conclude that \\eqref{H2} holds and \\eqref{H1} is satisfied with $\\bar \\kappa=S+2\\dist(x^0, \\Fix T)$. Now, invoking the estimates given by Proposition~\\ref{prop:residual-nonexpansive} \n\\begin{equation} \\label{bound-expected-nonexpansive} \n\\begin{aligned}\n \\!\\!\\! \\EE(\\norm{ x^N-Tx^N }) & \\le \\! \\frac{\\bar{\\kappa}}{N\\!+\\!1} \\!+\\!  \\frac{1}{N\\!+\\!1}\\sum_{n=1}^N \\left( \\frac{\\bar{\\kappa}}{n\\!+\\!1} \\!+\\! \\frac{n^2}{n\\!+\\!1}\\sigma_n \\!+\\!(n\\!-\\!1)\\sigma_{n-1} \\right ) \\!+\\!\\frac{N}{N\\!+\\!1}\\sigma_N\\\\\n&=\\frac{1}{N\\!+\\!1}\\left(\\bar{\\kappa}\\sum_{n=0}^{N}\\frac{1}{n\\!+\\!1}  \\!+\\!\\sum_{n=1}^N\\frac{n^2}{n\\!+\\!1}\\sigma_n +\\sum_{n=2}^N(n\\!-\\!1)\\sigma_{n-1}+ N \\sigma_N\\right )\\\\\n  &\\le\\frac{\\bar{\\kappa}}{N\\!+\\!1}\\sum_{n=0}^{N}\\frac{1}{n\\!+\\!1}+\\frac{2}{N\\!+\\!1}\\sum_{n=1}^N n\\sigma_n - \\frac{1}{N+1}\\sum_{n=1}^N \\sigma_n,\n\\end{aligned}\n\\end{equation}\nwhere we have used in the second line that $\\sigma_0=0$. We observe that the first term $\\tilde{O}(1/N)$ and the second also is by hypothesis. Therefore, $\\EE(\\norm{x^N-Tx^N}) = \\tilde{O}(1/N)$, and $\\EE(\\norm{ x^N-Tx^N }) \\le \\varepsilon$, for all $N \\geq \\bar N= \\tilde{O}(\\varepsilon^{-1})$ iterations as claimed.\n\\end{proof}","sha256_utf8":"1717fc089dfe7a20f91caec9f535980ba7e791f82e5f4c1a6552e5b62278e8f9"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"计算乘积并验证有界性","source_units":[{"start":38618,"end":39099,"start_line":420,"end_line":425,"raw_tex":"\\begin{proof}\nFor this particular choice of $\\beta_n$, we have that $B_i^n=\\frac{i}{n+1}$ for all $1 \\leq i\\leq n$. Then \n\\[\nS = \\sup_{N\\geq 1}\\sum_{n=1}^N B_n^N  \\sigma_n= \\sup_{N\\geq 1}\\frac{1}{N+1}\\sum_{n=1}^N n \\sigma_n< \\infty, \n\\]\ngiven that $\\sum_{n= 1}^N n \\sigma_n = \\tilde{O}(1)$ and we conclude that \\eqref{H2} holds and \\eqref{H1} is satisfied with $\\bar \\kappa=S+2\\dist(x^0, \\Fix T)$. Now, invoking the estimates given by Proposition~\\ref{prop:residual-nonexpansive} \n","sha256_utf8":"0ade12a360a48255fdcab5f0459ce93e2bb404e93a51b668eae8ea5ec80c7e99"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"显式残差界推出迭代速率","source_units":[{"start":39099,"end":40087,"start_line":426,"end_line":434,"raw_tex":"\\begin{equation} \\label{bound-expected-nonexpansive} \n\\begin{aligned}\n \\!\\!\\! \\EE(\\norm{ x^N-Tx^N }) & \\le \\! \\frac{\\bar{\\kappa}}{N\\!+\\!1} \\!+\\!  \\frac{1}{N\\!+\\!1}\\sum_{n=1}^N \\left( \\frac{\\bar{\\kappa}}{n\\!+\\!1} \\!+\\! \\frac{n^2}{n\\!+\\!1}\\sigma_n \\!+\\!(n\\!-\\!1)\\sigma_{n-1} \\right ) \\!+\\!\\frac{N}{N\\!+\\!1}\\sigma_N\\\\\n&=\\frac{1}{N\\!+\\!1}\\left(\\bar{\\kappa}\\sum_{n=0}^{N}\\frac{1}{n\\!+\\!1}  \\!+\\!\\sum_{n=1}^N\\frac{n^2}{n\\!+\\!1}\\sigma_n +\\sum_{n=2}^N(n\\!-\\!1)\\sigma_{n-1}+ N \\sigma_N\\right )\\\\\n  &\\le\\frac{\\bar{\\kappa}}{N\\!+\\!1}\\sum_{n=0}^{N}\\frac{1}{n\\!+\\!1}+\\frac{2}{N\\!+\\!1}\\sum_{n=1}^N n\\sigma_n - \\frac{1}{N+1}\\sum_{n=1}^N \\sigma_n,\n\\end{aligned}\n\\end{equation}\nwhere we have used in the second line that $\\sigma_0=0$. We observe that the first term $\\tilde{O}(1/N)$ and the second also is by hypothesis. Therefore, $\\EE(\\norm{x^N-Tx^N}) = \\tilde{O}(1/N)$, and $\\EE(\\norm{ x^N-Tx^N }) \\le \\varepsilon$, for all $N \\geq \\bar N= \\tilde{O}(\\varepsilon^{-1})$ iterations as claimed.\n\\end{proof}","sha256_utf8":"ac6a5526e0b91772751f82250d70f996e2f025b55631e76e9499344b03127870"}]},{"id":"S035","semantic_type":"lemma","kind":"lemma","title":"显式精度迭代阈值","source_units":[{"start":42021,"end":42644,"start_line":442,"end_line":448,"raw_tex":"\\begin{lemma}\\label{lem:sigma_explicito}\nUnder the assumptions of Theorem~\\ref{thm:rate-nonexpansive}, let us suppose additionally that there exists $C\\geq 0$ such that $\\sum_{n=1}^N n\\sigma_n \\leq C \\ln(N+1)$ for all $N \\geq 1$. Let $\\rho \\geq 0$ such that $3\\bar{\\kappa} + 2C\\leq \\rho$. Then, for all $\\varepsilon>0$, if $\\varepsilon/\\rho < e^{-1}$, then \n\\[\n\\EE(\\norm{x^N-Tx^N})\\le \\varepsilon \\quad \\text{for all }\\,\\, N \\geq \\mbox{$\\left \\lceil 2\\frac{\\rho}{\\varepsilon} \\ln\\left (\\frac{\\rho}{\\varepsilon}\\right )\\right \\rceil$} .\n\\]\nWhen $\\varepsilon/\\rho \\geq e^{-1}$, then $N=1$ iteration is sufficient.\n\\end{lemma}","sha256_utf8":"d2e41d48bbfc72f126b0690f48116d416d8266ea3079c5a04a9bc2b5f6f3efc4"}]},{"id":"S036","semantic_type":"proof","kind":"proof","title":"显式阈值完整证明","source_units":[{"start":97034,"end":99961,"start_line":1148,"end_line":1185,"raw_tex":"\\subsection{Proof of Lemma~\\ref{lem:sigma_explicito}} \\label{apx:explicito}\nAssuming that $\\sum_{n=1}^N n\\sigma_n \\leq C \\ln(N+1)$, the bound \\eqref{bound-expected-nonexpansive} implies that\n\\begin{equation*} \n\\EE(\\norm{ x^N-Tx^N}) \\le \\frac{1}{N+1}\\left( 3\\bar{\\kappa}+2C\\right)\\ln(N+1) \\le \\underbrace{(3\\bar{\\kappa} + 2C)}_{\\leq\\rho}\\frac{\\ln(N+1)}{N+1}, \\end{equation*}\nwhere we have used the rough bound $3\\ln(N+1)\\geq \\sum_{n=1}^{N+1}\\frac{1}{n}$ for all $N \\geq 1$, in order to keep the estimates simple at the expense of some small factor. \n\n\n\nFor $\\varepsilon>0$, let us call $\\tau:=\\frac{\\varepsilon}{\\rho}>0$. It is sufficient to find some $\\bar N\\in \\NN$ such that, for any $N \\geq \\overline{N}$,\n\\[\n\\frac{\\ln(N+1)}{N+1} \\leq \\tau, \\quad \\text{or, equivalently,}\\,\\, -\\tau (N+1)e^{-\\tau(N+1)}\\geq -\\tau.\n\\]\nFirst, note that if $\\tau \\geq e^{-1}$ then the inequality above holds for all $N \\geq 0$. We assume in what follows that $\\tau < e^{-1}$. \n\n\n\nIt is well-known that if we want to solve the equation $-\\tau (x+1)e^{-\\tau(x+1)}= -\\tau$, the solution is given by the relation $-\\tau(x+1) = W(-\\tau)$, where $W$ is the Lambert function. Given that $\\tau >0$, there is two solutions of this equation, provided by the upper and lower branches of $W$, $W_0$ and $W_{-1}$, respectively. We recall that both functions are continuous, defined in the interval $[-e^{-1},0]$, and such that $W_0(-e^{-1})=W_{-1}(-e^{-1})=-1$, $W_0$ increases to cero and $W_{-1}$ decreases to $-\\infty$. This implies \nthat for all $\\tau \\in (0,e^{-1})$, $W_0(-\\tau) \\in (-1,0)$ and that $W_{-1}(-\\tau) \\in (-\\infty, -1)$. Let us pick the solution given by the lower branch, i.e. the one such that $x+1\\geq \\tau^{-1}$. In that case, \n$$x= -\\frac{1}{\\tau}W_{-1}(-\\tau)-1.$$\nTherefore if we set $m=\\lceil -\\frac{1}{\\tau}W_{-1}(-\\tau) -1 \\rceil \\in \\NN$ we obtain that \n$$\\frac{\\ln(N+1)}{N+1} \\leq \\tau, \\quad \\text{for all } N\\in \\NN, N\\geq m.$$\nLet us show that  $m \\leq \\lceil 2\\frac{\\rho}{\\varepsilon} \\ln\\left (\\frac{\\rho}{\\varepsilon}\\right ) \\rceil= \\lceil \\frac{2}{\\tau} \\ln\\left (\\frac{1}{\\tau}\\right ) \\rceil$. For that purpose, we use following lower bound for the function $W_{-1}$ (see \\cite[Theorem 1]{chatzigeorgiou2013bounds}):\n\n$$W_{-1}(-e^{-u-1}) \\geq -1-\\sqrt{2u}-u, \\quad \\text{for all} \\, u>0.$$ \n\nTaking $u=-\\ln(\\tau)-1>0$, we get that $-e^{-u-1}=-\\tau$ and hence\n\\[\n\\begin{aligned}\n \\left \\lceil -\\frac{1}{\\tau}W_{-1}(-\\tau) -1 \\right \\rceil&\\leq \n\\left \\lceil\\frac{1}{\\tau}\\left(1+\\sqrt{2(-\\ln(\\tau)-1)}+(-\\ln(\\tau)-1)\\right )-1\\right\\rceil,\\\\\n & = \n\\left \\lceil\\frac{1}{\\tau}\\left(\\sqrt{-2(\\ln(\\tau)+1)}-\\ln(\\tau)\\right )-1\\right\\rceil\\leq \\left \\lceil 2\\frac{1}{\\tau} \\ln\\left (\\frac{1}{\\tau}\\right )-1\\right \\rceil,\n \\end{aligned}\n\\]\nwhere the last inequality follows from the trivial bound $\\sqrt{-2(u+1)}\\leq -u$ for any $u\\leq -1$. Observe that the last quantity is at least $1$ for all $\\tau \\in (0,e^{-1})$.\n\n\n\n","sha256_utf8":"5bc779a6bbc7b3061f21818709a89ba80a2c97fe66680666434b164cb27b33e1"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"将残差化为对数比值","source_units":[{"start":97034,"end":97583,"start_line":1148,"end_line":1152,"raw_tex":"\\subsection{Proof of Lemma~\\ref{lem:sigma_explicito}} \\label{apx:explicito}\nAssuming that $\\sum_{n=1}^N n\\sigma_n \\leq C \\ln(N+1)$, the bound \\eqref{bound-expected-nonexpansive} implies that\n\\begin{equation*} \n\\EE(\\norm{ x^N-Tx^N}) \\le \\frac{1}{N+1}\\left( 3\\bar{\\kappa}+2C\\right)\\ln(N+1) \\le \\underbrace{(3\\bar{\\kappa} + 2C)}_{\\leq\\rho}\\frac{\\ln(N+1)}{N+1}, \\end{equation*}\nwhere we have used the rough bound $3\\ln(N+1)\\geq \\sum_{n=1}^{N+1}\\frac{1}{n}$ for all $N \\geq 1$, in order to keep the estimates simple at the expense of some small factor. \n","sha256_utf8":"52cd1c07e3ec34105c455db34091fff231db9c2de5aa390cdba6f93178a0495a"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"归一化精度并区分阈值","source_units":[{"start":97586,"end":97993,"start_line":1156,"end_line":1160,"raw_tex":"For $\\varepsilon>0$, let us call $\\tau:=\\frac{\\varepsilon}{\\rho}>0$. It is sufficient to find some $\\bar N\\in \\NN$ such that, for any $N \\geq \\overline{N}$,\n\\[\n\\frac{\\ln(N+1)}{N+1} \\leq \\tau, \\quad \\text{or, equivalently,}\\,\\, -\\tau (N+1)e^{-\\tau(N+1)}\\geq -\\tau.\n\\]\nFirst, note that if $\\tau \\geq e^{-1}$ then the inequality above holds for all $N \\geq 0$. We assume in what follows that $\\tau < e^{-1}$. \n","sha256_utf8":"4a91900e4a6b46c6c7212ba9527aa6cf419b5ec1edbec44f79cc075220585540"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"利用Lambert下支求逆","source_units":[{"start":97996,"end":98965,"start_line":1164,"end_line":1168,"raw_tex":"It is well-known that if we want to solve the equation $-\\tau (x+1)e^{-\\tau(x+1)}= -\\tau$, the solution is given by the relation $-\\tau(x+1) = W(-\\tau)$, where $W$ is the Lambert function. Given that $\\tau >0$, there is two solutions of this equation, provided by the upper and lower branches of $W$, $W_0$ and $W_{-1}$, respectively. We recall that both functions are continuous, defined in the interval $[-e^{-1},0]$, and such that $W_0(-e^{-1})=W_{-1}(-e^{-1})=-1$, $W_0$ increases to cero and $W_{-1}$ decreases to $-\\infty$. This implies \nthat for all $\\tau \\in (0,e^{-1})$, $W_0(-\\tau) \\in (-1,0)$ and that $W_{-1}(-\\tau) \\in (-\\infty, -1)$. Let us pick the solution given by the lower branch, i.e. the one such that $x+1\\geq \\tau^{-1}$. In that case, \n$$x= -\\frac{1}{\\tau}W_{-1}(-\\tau)-1.$$\nTherefore if we set $m=\\lceil -\\frac{1}{\\tau}W_{-1}(-\\tau) -1 \\rceil \\in \\NN$ we obtain that \n$$\\frac{\\ln(N+1)}{N+1} \\leq \\tau, \\quad \\text{for all } N\\in \\NN, N\\geq m.$$\n","sha256_utf8":"b62cd2d09e4dbf218e4806dbe920c5da05ab24209b1023dd2acd607fdeec78e1"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"引用Lambert下支估计","source_units":[{"start":98965,"end":99336,"start_line":1169,"end_line":1171,"raw_tex":"Let us show that  $m \\leq \\lceil 2\\frac{\\rho}{\\varepsilon} \\ln\\left (\\frac{\\rho}{\\varepsilon}\\right ) \\rceil= \\lceil \\frac{2}{\\tau} \\ln\\left (\\frac{1}{\\tau}\\right ) \\rceil$. For that purpose, we use following lower bound for the function $W_{-1}$ (see \\cite[Theorem 1]{chatzigeorgiou2013bounds}):\n\n$$W_{-1}(-e^{-u-1}) \\geq -1-\\sqrt{2u}-u, \\quad \\text{for all} \\, u>0.$$ \n","sha256_utf8":"a1a4699fdd9fedc687770aea9ef2c7b03bf837ec63ccc5524da8c5583b4b4166"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"得到显式整数迭代阈值","source_units":[{"start":99337,"end":99961,"start_line":1173,"end_line":1185,"raw_tex":"Taking $u=-\\ln(\\tau)-1>0$, we get that $-e^{-u-1}=-\\tau$ and hence\n\\[\n\\begin{aligned}\n \\left \\lceil -\\frac{1}{\\tau}W_{-1}(-\\tau) -1 \\right \\rceil&\\leq \n\\left \\lceil\\frac{1}{\\tau}\\left(1+\\sqrt{2(-\\ln(\\tau)-1)}+(-\\ln(\\tau)-1)\\right )-1\\right\\rceil,\\\\\n & = \n\\left \\lceil\\frac{1}{\\tau}\\left(\\sqrt{-2(\\ln(\\tau)+1)}-\\ln(\\tau)\\right )-1\\right\\rceil\\leq \\left \\lceil 2\\frac{1}{\\tau} \\ln\\left (\\frac{1}{\\tau}\\right )-1\\right \\rceil,\n \\end{aligned}\n\\]\nwhere the last inequality follows from the trivial bound $\\sqrt{-2(u+1)}\\leq -u$ for any $u\\leq -1$. Observe that the last quantity is at least $1$ for all $\\tau \\in (0,e^{-1})$.\n\n\n\n","sha256_utf8":"d137ac4261316845e75c5b12aeb724253bccf0b9681fb28d60828e5862b95322"}]},{"id":"S042","semantic_type":"external_fact","kind":"external_fact","title":"独立均值方差恒等式","source_units":[],"declared_external":true,"declaration_text":["Jensen's inequality gives (E||V||)^2 <= E||V||^2. For square-integrable errors E_j that are independent and zero-mean conditional on the sampling point, E(||sum_j E_j||_2^2 | sampling point) = sum_j E(||E_j||_2^2 | sampling point)."]},{"id":"S043","semantic_type":"external_fact","kind":"external_fact","title":"Lambert实分支及下界","source_units":[],"declared_external":true,"declaration_text":["The real Lambert W branches satisfy W(z)exp(W(z))=z. On [-exp(-1),0), W_0 increases from -1 toward 0, whereas W_{-1} decreases from -1 toward negative infinity. Theorem 1 of chatzigeorgiou2013bounds gives W_{-1}(-exp(-u-1)) >= -1-sqrt(2u)-u for u>0."]},{"id":"S044","semantic_type":"assumption","kind":"assumption","title":"小精度约定","source_units":[{"start":42646,"end":43036,"start_line":450,"end_line":450,"raw_tex":"The following corollary provides an upper bound on the number of queries for the stochastic Halpern iteration with minibatching when the stochastic oracle has uniformly bounded variance. Due to Lemma~\\ref{lem:sigma_explicito}, the constants involved can be explicitly estimated in this case. For the sake of simplicity, we assume in what follows that $\\varepsilon>0$ is sufficiently small.\n","sha256_utf8":"7f4510e86b798a652dac9dca337e60adf95091d81b48386a2a5329ddf562c49d"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S015","edges":[{"source_id":"S001","target_id":"S023","kind":"supports"},{"source_id":"S001","target_id":"S025","kind":"supports"},{"source_id":"S001","target_id":"S029","kind":"supports"},{"source_id":"S001","target_id":"S030","kind":"supports"},{"source_id":"S002","target_id":"S023","kind":"supports"},{"source_id":"S002","target_id":"S029","kind":"supports"},{"source_id":"S003","target_id":"S014","kind":"supports"},{"source_id":"S004","target_id":"S017","kind":"supports"},{"source_id":"S005","target_id":"S013","kind":"supports"},{"source_id":"S005","target_id":"S019","kind":"supports"},{"source_id":"S006","target_id":"S022","kind":"supports"},{"source_id":"S006","target_id":"S023","kind":"supports"},{"source_id":"S006","target_id":"S024","kind":"supports"},{"source_id":"S006","target_id":"S029","kind":"supports"},{"source_id":"S007","target_id":"S014","kind":"supports"},{"source_id":"S008","target_id":"S019","kind":"supports"},{"source_id":"S008","target_id":"S034","kind":"supports"},{"source_id":"S009","target_id":"S013","kind":"supports"},{"source_id":"S010","target_id":"S014","kind":"supports"},{"source_id":"S010","target_id":"S017","kind":"supports"},{"source_id":"S010","target_id":"S022","kind":"supports"},{"source_id":"S010","target_id":"S024","kind":"supports"},{"source_id":"S010","target_id":"S026","kind":"supports"},{"source_id":"S010","target_id":"S029","kind":"supports"},{"source_id":"S010","target_id":"S030","kind":"supports"},{"source_id":"S010","target_id":"S033","kind":"supports"},{"source_id":"S011","target_id":"S026","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S014","target_id":"S012","kind":"supports"},{"source_id":"S042","target_id":"S014","kind":"supports"},{"source_id":"S012","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S027","target_id":"S019","kind":"supports"},{"source_id":"S035","target_id":"S019","kind":"supports"},{"source_id":"S044","target_id":"S019","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S016","kind":"supports"},{"source_id":"S019","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S023","target_id":"S021","kind":"supports"},{"source_id":"S024","target_id":"S021","kind":"supports"},{"source_id":"S025","target_id":"S021","kind":"supports"},{"source_id":"S026","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S034","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S032","target_id":"S037","kind":"supports"},{"source_id":"S034","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S043","target_id":"S039","kind":"supports"},{"source_id":"S043","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S039","target_id":"S036","kind":"supports"},{"source_id":"S040","target_id":"S036","kind":"supports"},{"source_id":"S041","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"}]}}
{"record_id":"2403.13320v2","paper":{"title":"Direct search for stochastic optimization in random subspaces with zeroth-, first-, and second-order convergence and expected complexity","authors":["K. J. Dzahini","S. M. Wild"],"arxiv_id":"2403.13320","version":"v2","journal":"Mathematical Programming","doi":"10.1007/s10107-026-02404-7","abstract_url":"https://arxiv.org/abs/2403.13320v2","pdf_url":"https://arxiv.org/pdf/2403.13320v2"},"context":{"raw_tex":"\\begin{equation}\\label{probl1}\n\\underset{\\x\\in\\bcX}{\\min} f(\\x), \\qquad\\mbox{with}\\quad f(\\x)=\\Esp_{\\ubar{\\theta}}\\left[f_{\\ubar{\\theta}}(\\x)\\right],\n\\end{equation}\n\nwhere {\\bl the nonempty set} $\\bcX \\subseteq\\rn$ {\\bl of nonzero measure} is a feasible region; the values of the function $f:\\rn\\to\\R$ are available only through $f_{\\ubar{\\theta}}$, a stochastically noisy version of $f$;  ${\\ubar{\\theta}}$ is a random variable whose probability distribution is possibly unknown; and $\\Esp_{\\ubar{\\theta}}$ denotes the {\\it expectation} with respect to ${\\ubar{\\theta}}$. Problem~\\eqref{probl1} arises, for example, in statistical machine learning~\\cite{bottou2018optimization} where $\\ubar{\\theta}$ is a random variable describing a given sample set while $f_{\\ubar{\\theta}}$ represents the composition of a loss function and a prediction function. {\\bl We particularly target settings where the problem dimension $n>100$, which is large in the context of derivative-free methods for stochastic optimization.}\n\n\nThroughout the manuscript, vectors are written in lowercase boldface (e.g., $\\vb\\in\\rn, n\\geq 2$), and matrices are written in uppercase boldface (e.g., $\\Ub\\in\\rnp$). Unless otherwise stated, all random quantities are underlined (e.g., $\\ubar{z}$, $\\vrandom$, $\\Urandom$) and are defined on the same probability space $(\\Omega,\\F,\\pr)$. The nonempty set~$\\Omega$ referred to as the {\\it sample space} is a collection of elements~$\\omega$ called {\\it sample points}, while subsets of~$\\Omega$ are called {\\it events}. The collection, $\\F$, of these events is called a $\\sigma$-algebra; and the finite measure~$\\pr$ defined on the {\\it measurable space} $(\\Omega,\\F)$, with $\\prob{\\Omega}=1$, is a {\\it probability measure}. A {\\it filtration} is an increasing subsequence $\\accolade{\\mathcal{H}_k}_k$ of $\\sigma$-algebras of~$\\F$. By denoting by~$\\mathscr{B}(\\rn)$ the Borel $\\sigma$-algebra generated by open sets of~$\\rn$, a real-valued random variable~$\\ubar{z}$ is a measurable map $\\ubar{z}:(\\Omega,\\F,\\pr)\\to (\\R,\\mathscr{B}(\\R))$, where measurability means that $\\accolade{\\ubar{z}\\in \\mathscr{E}}:=\\accolade{\\omega\\in\\Omega:\\ubar{z}(\\omega)\\in\\mathscr{E}}=:\\ubar{z}^{-1}(\\mathscr{E})\\in\\F$ for all $\\mathscr{E}\\in \\mathscr{B}(\\R)$~\\cite{bhattacharya2007basic}. A real-valued random matrix is a matrix-valued random variable, that is, a matrix whose entries are real-valued random variables. $\\normii{\\ub}:=\\left(\\sum_{i=1}^n u_i^2\\right)^{1/2}$ denotes the Euclidean norm of a vector $\\ub:=(u_1,\\dots,u_n)^{\\top}\\in\\rn$.  $\\norme{\\cdot}$ is used for the norm of a matrix and is assumed \n%GAIL - bu \"supposed to be\" do you mean that it really is not? or do you mean it is assumed to be?\nto be {\\it consistent} with the Euclidean norm, that is, $\\normii{\\Q\\ub}\\leq \\norme{\\Q}\\normii{\\ub}$, which holds for both spectral and Frobenius matrix norms. $\\ub^{\\top}\\vb=\\scal{\\ub}{\\vb}:=\\sum_{i=1}^n u_i v_i$ denotes the {\\it scalar product} or {\\it dot product} of the vectors $\\ub$ and $\\vb:=(v_1,\\dots,v_n)^{\\top}\\in\\rn$. $K_{\\infty}$ denotes the random set of all {\\it unsuccessful iterations} of a given algorithm. A realization of a random set~$K$ is denoted by~$\\mathcal{K}$; that is, $K(\\omega)=\\mathcal{K}$ for some~$\\omega\\in\\Omega$. {\\bl For $a\\leq b$, $\\intbracket{a,b}:=[a,b]\\cap\\Z$ is the set of integers in the interval $[a,b]$.}\n\n\nSDDS algorithms were introduced in~\\cite{dzahini2020expected} to solve the stochastic optimization problem~\\eqref{probl1} in the particular case $\\bcX=\\rn$. These algorithms use a framework inspired by that of StoMADS~\\cite{audet2019stomads} and the broad class of methods presented in~\\cite{Vicente2013} using a {\\it generating set search} method~\\cite{KoLeTo03a}, such as pattern search and generalized pattern search~\\cite{AuDe03a}. Each iteration of SDDS algorithms is composed of two main steps: an optional {\\it search}, not shown in~\\cite[Algorithm~1]{dzahini2020expected}, and not detailed in Algorithm~\\ref{vagueDirAlgo} for simplicity, and the ``more disciplined''~\\cite{LaMeWi2019} {\\it poll} on which  the theoretical analyses rely. Broadly speaking, each poll step of an SDDS method generates a finite set of {\\it poll points} by taking an {\\it incumbent solution}~$\\xk\\in\\rn$ and adding terms of the form~$\\dk\\di^n$, where $\\dk>0$ is the {\\it stepsize} and $\\di^n\\in\\rn$ is a nonzero vector from a finite set $\\mathbb{D}^n_k\\subset\\rn$ of {\\it poll directions}. More precisely, $\\mathbb{D}^n_k$ is a PSS, meaning that any vector in~$\\rn$ can be written as a nonnegative linear combination of elements in~$\\mathbb{D}^n_k$~\\cite[Definition~6.1]{AuHa2017} or, equivalently, the cosine measure $\\kappa(\\mathbb{D}^n_k)>0$, where \n%for a given $\\mathbb{D}\\subset\\rn$,\n\\begin{equation}\\label{cosineMeasureDef}\n\\kappa(\\mathbb{D}^q):=\\underset{\\wb\\in \\R^q,\\wb\\neq \\bm{0}}{\\min}\\ \\underset{\\di^q\\in \\mathbb{D}^q}{\\max}\\frac{\\scal{\\wb}{\\di^q}}{\\normii{\\wb}\\normii{\\di^q}}\\quad\\mbox{with}\\quad \\mathbb{D}^q\\subset\\R^q.\n\\end{equation}\n\n\\begin{equation}\\label{cosineDescentDkn}\n\\mbox{for any}\\ \\vi\\in\\rn,\\quad\\exists \\di_{\\star}^n(\\vi)\\in \\mathbb{D}^n\\subset\\rn\\quad\\mbox{such that}\\quad \\kappa(\\mathbb{D}^n)\\normii{\\vi}\\normii{\\di^n_{\\star}(\\vi)}\\leq \\scal{\\vi}{\\di^n_{\\star}(\\vi)}.\n\\end{equation}\n\n\\begin{definition}\\label{wellAlignment}\nFor fixed $\\alpha_Q,\\beta_Q\\in (0,1/2)$, \n%a sequence $\\accoladekinN{\\QkRandom}$ of independent \na random matrix $\\Qrandom\\in\\rnp$ is called $(1-\\beta_Q)$-probabilistically $\\alpha_Q$-well aligned if, for all $\\vb\\in\\rn$, $\\prob{\\normii{\\Qrandom^{\\top}\\vb}\\geq \\alpha_Q\\normii{\\vb}}\\geq 1-\\beta_Q$. Such matrices are referred to as $WAM(\\alpha_Q,\\beta_Q)$.\n\\end{definition}\n\n\\begin{assumption}\\label{JltAss1}{\\bl (Inspired by~\\cite[Assumptions~2 and~3]{DzaWildSub2022} and~\\cite{CRsubspace2021})}\nThe random matrix $\\Qrandom\\in\\rnp$ is a $WAM(\\alpha_Q,\\beta_Q)$ for fixed $\\alpha_Q,\\beta_Q\\in (0,1/2)$ and satisfies $\\norme{\\Qrandom}\\leq\\Qmax$ almost surely for some $\\Qmax>0$. \n%and $\\prob{\\normii{\\QrandomTranspose\\bm{v}}\\geq \\alpha_Q\\normii{\\bm{v}}}\\geq 1-\\beta_Q$ for all $\\bm{v}\\in\\rn$, for some $\\alpha_Q\\in (0,1)$ and $\\beta_Q\\in (0,1/2)$.\n\\end{assumption}\n\nThe poll directions used by Algorithm~\\ref{algoStoDARS} are generated by using Haar-distributed orthogonal random matrices. Recall that a matrix $\\bm{O}\\in \\rnn$ is {\\it orthogonal} if and only if $\\bm{O}^{\\top}\\bm{O}=\\bm{O}\\bm{O}^{\\top}=\\bm{I}_n$ or equivalently if its columns form an orthonormal basis of $\\rn$~\\cite[Section~1.1]{MeckesElizHaar2019}. Throughout the manuscript, the compact Lie group of $n\\times n$ orthogonal matrices over~$\\R$ will be denoted by $\\mathbb{O}(n)$. \n%We also recall that $\\mathbb{O}(n)$ is a group under multiplication, and that this group admits a compact Lie group structure, that is, it is also a differentiable manifold such that the multiplication operation $(\\bm{A}, \\bm{B})\\mapsto \\bm{A}\\bm{B}$ and the inversion operation $\\bm{A}\\mapsto \\bm{A}^{-1}$ are smooth maps. \nA {\\it uniform random element} of $\\mathbb{O}(n)$ is a random $\\Urandom\\in \\mathbb{O}(n)$ whose distribution is translation invariant; that is, for any fixed $\\bm{O}\\in \\mathbb{O}(n)$, $\\bm{O}\\Urandom\\equald \\Urandom\\bm{O}\\equald \\Urandom$~\\cite[Section~1.2]{MeckesElizHaar2019}, where ``$\\equald$'' is used for the equality in distribution. In particular, there exists a unique translation-invariant probability measure on $\\mathbb{O}(n)$ called Haar measure on~$\\mathbb{O}(n)$~\\cite[Theorem~1.4]{MeckesElizHaar2019}. Any random matrix whose distribution is Haar measure will be called Haar-{\\it distributed matrix}. \n\n\n\\begin{definition}\\label{epsAccDef0}\nGiven $\\ef>0$, $\\xk\\in\\rn$, and $\\ub:=\\Ukp\\di^p=\\sqrt{\\frac{p}{n}}\\Qk\\di^p\\in\\rn$ with $\\di^p\\in\\pollD_k$, $\\fok$ and $\\fuk$ are called $\\ef$-accurate estimates of $f(\\xk)$ and $f(\\xk+\\dk\\ub)$, respectively, for a given $\\dk$ if\n\\begin{equation}\\label{epsAccDef1}\n\\abs{\\fok-f(\\xk)}\\leq \\ef\\dk^2\\qquad \\mbox{and}\\qquad \\abs{\\fuk-f(\\xk+\\dk\\ub)}\\leq \\ef\\dk^2.\n\\end{equation}\n\\end{definition}\n\n\\begin{algorithm}[H]\n\\caption{StoDARS algorithm}\n\\label{algoStoDARS} \n%\\nonl\n% $\\accolade{\\Urandomtilde_t}_{t=0}^\\infty$, where the $\\Urandomtilde_t:=\\left[\\Urandomtilde_{t,\\ntp}\\ \\ \\Urandomtilde_{t,\\ntnmp}\\right]\\in\\mathbb{O}(n)$ are Haar-distributed, is a sequence of independent matrices, with $p\\in\\intbracket{1,n}$ chosen so that $\\sqrt{\\frac{n}{p}}\\Urandomtilde_{t,\\ntp}\\in\\rnp$ is $WAM(\\alpha_Q,\\beta_Q)$ for some $\\alpha_Q, \\beta_Q\\in(0,1/2)$. \\\\\n% $\\accolade{\\BrandomtildeSet_t^+}_{t=0}^\\infty$ is a sequence of random PSSs such that the matrices $\\Brandomtilde_t^+\\in\\R^{p\\times m}$ with unit columns in $\\BrandomtildeSet_t^+\\subset\\mathbb{S}^{p-1}$ are independent.    \\\\\n% $ $\\\\\n\\textbf{[0] Initialization}\\\\\n\\hspace*{10mm}Choose $\\x_0\\in\\bcX\\subseteq\\rn$, $\\delta_0>0$, $\\ef>0$, $\\gamma>2$, $0<\\tau<\\left(\\frac{\\gamma-2}{\\gamma+2}\\right)^{1/2}$, $j_{\\max}\\in\\N$, $\\dmax=\\tau^{-j_{\\max}}\\delta_0$ \\\\ \n\\hspace*{10mm}$\\ell_0=0$ and $t_0=0$. Generate $\\tilde{\\Ub}_{0,\\ntp}$, where $\\Urandomtilde_0:=\\left[\\Urandomtilde_{0,\\ntp}\\ \\ \\Urandomtilde_{0,\\ntnmp}\\right]\\in\\mathbb{O}(n)$ is Haar-distributed, \\\\\n\\hspace*{10mm}with $p\\in\\intbracket{1,n}$ chosen so that $\\sqrt{\\frac{n}{p}}\\Urandomtilde_{0,\\ntp}\\in\\rnp$ is $WAM(\\alpha_Q,\\beta_Q)$ for some $\\alpha_Q, \\beta_Q\\in(0,1/2)$.\\\\\n\\hspace*{10mm}Construct a PSS $\\tilde{\\mathbb{D}}^p_0\\subset\\spOne$ using {\\bl any matrix} $\\Drandomtildep_0\\in\\R^{p\\times m}$, with $m\\in {\\bl \\intbracket{p+1,2p}}$, {\\bl generated}\\\\ \n\\hspace*{10mm}{\\bl independently of $\\Urandomtilde_{0}$, whose columns form a PSS of~$\\rp$}.\\\\\n\\hspace*{10mm}Set the iteration counter $k \\gets 0$.\\\\\n% {\\bl\\textbf{[1] Matrix index update}\\\\\n% \\hspace*{10mm}If $\\ell_{k} \\geq  \\max_{i\\leq k-1} \\ell_i$ or $k=0$, set $t_{k}\\gets \\ell_{k}$.   \\\\\n% \\hspace*{10mm}Otherwise $t_{k}\\gets 1+\\max_{i\\leq k-1} t_i$.\\\\}\n\\textbf{[{1}] Matrix and PSS for subspace polling}\\\\\n% \\hspace*{10mm}If $k\\geq 1$ and $t_k=t_{i_0}$ for some $i_0\\in \\intbracket{0,k-1}$   \\\\\n% \\hspace*{16mm}Generate $\\tilde{\\Ub}_{t_k,\\ntp}$ using an independent copy $\\Urandomtilde_{t_k,\\ntp}$ of $\\Urandomtilde_{t_{i_0},\\ntp}$ such that  \\\\\n% \\hspace*{16mm}$\\{\\Urandomtilde_{t_i,\\ntp}\\}_{i=0}^{k-1}$ and $\\Urandomtilde_{t_k,\\ntp}$ are i.i.d. \\\\\n% \\hspace*{16mm}Construct a PSS $\\tilde{\\mathbb{D}}^p_{t_k}\\subset\\spOne$ using the columns of an independent copy $\\Drandomtildep_{t_k}$ of $\\Drandomtildep_{t_{i_0}}$ \\\\\n% \\hspace*{16mm}such that $\\{\\Drandomtildep_{t_i}\\}_{i=0}^{k-1}$ and $\\Drandomtildep_{t_k}$ are i.i.d. \\\\\n% \\hspace*{10mm}If $k\\geq 1$ and $t_k\\not\\in\\accolade{t_0,\\dots,t_{k-1}}$ \\\\\n\\hspace*{10mm}Generate $\\tilde{\\Ub}_{t_k,\\ntp}$ as a realization of $\\Urandomtilde_{t_k,\\ntp}$, with $\\{\\Urandomtilde_{t_i,\\ntp}\\}_{i=0}^{k}$ {\\bl obtained from the i.i.d. $\\{\\Urandomtilde_{t_i}\\}_{i=0}^{k}$}.\\\\\n\\hspace*{10mm}Construct a PSS $\\tilde{\\mathbb{D}}^p_{t_k}\\subset\\spOne$ using the columns of $\\Drandomtildep_{t_k}$, where the independent $\\{\\Drandomtildep_{t_i}\\}_{i=0}^{k}$ {\\bl are generated}\\\\ \n\\hspace*{10mm}{\\bl independently of $\\{\\Urandomtilde_{t_i}\\}_{i=0}^{k}$}.  \\\\\n% \\hspace*{16mm}such that $\\accolade{\\Urandom_{t_i,\\ntp}}_{i=0}^k$ and $\\{\\Urandomtilde_{t_i,\\ntp}\\}_{i=0}^k$ are independent.   \\\\\n% \\hspace*{16mm}Select $\\BrandomSet_{t_k}^+$ as the columns of an independent copy $\\Brandom_{t_k}^+$ of $\\Brandomtilde_{t_k}^+$, where $\\Brandom_{t_k}^+$ is independent of  \\\\\n% % \\hspace*{16mm}$\\Brandom_{t_i}^+,\\ i\\leq k-1$, and $\\Brandomtilde_t^+, t\\geq 0$. \\\\\n% \\hspace*{16mm}$\\Brandom_{t_i}^+,\\ i\\leq k-1$, and $\\accolade{\\Brandom_{t_i}^+}_{i=0}^k$ and $\\{\\Brandomtilde_{t_i}^+\\}_{i=0}^k$ are independent. \\\\\n% \\hspace*{10mm}Otherwise $\\Urandom_{t_k,\\ntp}:=\\Urandomtilde_{t_k,\\ntp}$ and $\\BrandomSet_{t_k}^+:=\\BrandomtildeSet_{t_k}^+$. \\\\\n\\hspace*{10mm}Define $\\Ukp:=\\tilde{\\Ub}_{t_k,\\ntp}$ and $\\pollD_k:=\\tilde{\\mathbb{D}}^p_{t_k}$.\\\\\n\\textbf{[{2}] Poll}\\\\\n%\\hspace*{10mm}Select a PSS $\\pollD_k\\subset\\rp$ such that $\\pollDrandom_k$ is independent of $\\pollDrandom_0,\\dots,\\pollDrandom_{k-1}$ when $k\\geq 1$. \\\\\n\\hspace*{10mm}Generate a set of poll points $\\mathcal{P}_k:=\\{\\xk+\\dk\\ub: \\ub\\in\\mathbb{U}_k^n \\}$; where $\\mathbb{U}_k^n:=\\{\\Ukp\\di^p:\\di^p\\in \\pollD_k\\}$, {\\bbl and}\\\\\n\\hspace*{10mm}{\\bbbl $\\pollD_k$ is generated independently of $\\Ukp$}.\\\\\n%\\hspace*{8mm}{}\\\\ \n\\hspace*{10mm}Obtain estimates\\footnotemark $\\fuk$ and $\\fok$ of $f$, respectively, at ${\\xk+\\dk\\ub}\\in\\mathcal{P}_k$ and $\\xk$.\\\\\n%\\hspace*{10mm}function evaluations.\\\\\n%\\hspace*{10mm}\\textbf{{Success}}\\\\\n\\hspace*{10mm}If $\\Pkfeas:=\\Pk\\cap\\bcX\\neq\\emptyset$ and $\\fuk -\\fok \\leq -\\gamma \\ef\\dk^2$ for some ${\\xk+\\dk\\ub}\\in\\Pkfeas$ (\\textbf{successful})\\\\ \n\\hspace*{16mm}set $\\xkun\\gets \\xk+\\dk\\ub$,  $\\dkun\\gets \\min\\{\\tau^{-1}\\dk,\\dmax\\}$, and $\\ell_{k+1}\\gets \\ell_k-1$. \\\\\n%\\hspace*{10mm}\\textbf{{Failure}}\\\\\n\\hspace*{10mm}Otherwise (\\textbf{unsuccessful}) set $\\xkun\\gets \\xk$, $\\delta_{k+1}\\gets\\tau\\dk$, and $\\ell_{k+1}\\gets \\ell_k+1$. \\\\\n\\textbf{[3] Matrix index update}\\\\\n\\hspace*{10mm}If $\\ell_{k+1} \\geq  \\max_{i\\leq k} \\ell_i$, set $t_{k+1}\\gets \\ell_{k+1}$.   \\\\\n\\hspace*{10mm}Otherwise $t_{k+1}\\gets 1+\\max_{i\\leq k} t_i$.\\\\\n\\textbf{[4] Termination}\\\\\n\\hspace*{10mm}If no termination criterion is met, set $k\\gets k+1$ and go to \\textbf{[1]}.\\\\\n%\\hspace*{10mm}\\\\\n\\hspace*{10mm}Otherwise stop.\n\\end{algorithm}\n\n\\begin{assumption}\\label{QkHaarAssumpt}{\\bl (Motivated by Theorem~\\ref{JLforHaarTheor})}\nFor all $k\\geq 0$, the independent random matrices  $\\Drandom_k\\in\\R^{p\\times m}$, with $m\\in {\\bl \\intbracket{p+1,2p}}$, and $\\Ukprandom\\in\\rnp$ used for subspace polling in Algorithm~\\ref{algoStoDARS} are such that $\\pollDrandom_k\\subset\\spOne$, $\\Urandom_k:=\\left[\\Urandom_{k,\\ntp}\\ \\, \\Urandom_{k,\\ntnmp}\\right]\\in \\mathbb{O}(n)$ is Haar-distributed, and $\\QkRandom:=\\sqrt{\\frac{n}{p}}\\Ukprandom$ is $WAM(\\alpha_Q,\\beta_Q)$ for some $\\alpha_Q, \\beta_Q\\in(0,1/2)$.\n\\end{assumption}\n\nNext, $(\\QkRandom)_{ij}$, with $(i,j)\\in\\mathbb{I}_{\\ntp}:=\\intbracket{1,n}\\times\\intbracket{1,p}$, denote the entries of $\\QkRandom$. Likewise, $(\\Drandom_k)_{i'j'}$, with $(i',j')\\in\\mathbb{I}_{p,m}$, denote the entries of $\\pollDrandom_k$. The conditioning on the past is formalized by using the filtrations $\\accoladekinN{\\falgebra}$ and $\\accoladekinN{\\fqdalgebra}$ defined by\n\\begin{eqnarray*}\n\\falgebra&:=&\\sigma\\left({\\ubar{f}}_{\\ell}({\\bm{0}}), {\\ubar{f}}_{\\ell}(\\urandom),(\\Qrandom_\\ell)_{ij}, (\\Drandom_\\ell)_{i'j'}\\ \\mbox{with}\\ \\ell\\in\\intbracket{0,k-1}, (i,j)\\in\\mathbb{I}_{\\ntp}\\ \\mbox{and}\\ (i',j')\\in\\mathbb{I}_{p,m}\\right)\\\\\n\\fqdalgebra&:=&\\sigma\\left({\\ubar{f}}_{\\ell}({\\bm{0}}), {\\ubar{f}}_{\\ell}(\\urandom),(\\Qrandom_\\ell)_{ij}, (\\Drandom_\\ell)_{i'j'},(\\QkRandom)_{ij}, (\\Drandom_k)_{i'j'} \\ \\mbox{with}\\ \\ell\\in\\intbracket{0,k-1}, (i,j)\\in\\mathbb{I}_{\\ntp}\\ \\mbox{and}\\ (i',j')\\in\\mathbb{I}_{p,m}\\right),\n\\end{eqnarray*}\nwhere for {\\it completeness}~\\cite{billingsley1995ThirdEdition}, $\\F_{-1}:=\\sigma(\\x_0)$. The idea behind the above definitions is explained in~\\cite[Remark~2.1]{DzaWildSub2022}. By construction, $\\QkRandom$ and $\\Drandom_k$ are $\\fqdalgebra$-measurable so that $\\E{\\QkRandom|\\fqdalgebra}=\\QkRandom$ almost surely and $\\E{\\Drandom_k|\\fqdalgebra}=\\Drandom_k$ almost surely. The sufficient accuracy of the random estimates is formalized as follows.\n\n\n\\begin{definition}\nA sequence $\\accolade{\\Fok,\\Fuk}$ of random estimates is called $\\beta_f$-probabilistically $\\ef$-accurate with respect to the corresponding sequence $\\accolade{\\Xk,\\Dk,\\QkRandom,\\Drandom_k}$ if the events\n\\begin{eqnarray*}\nJ_k&:=&\\left\\lbrace \\mbox{Given}\\ \\QkRandom, \\Drandom_k\\ \\mbox{and}\\ \\Xk, \\Fok\\ \\mbox{and}\\ \\Fuk\\ \\mbox{are}\\ \\ef\\mbox{-accurate}\\ \\mbox{estimates of}\\ f(\\Xk)\\ \\mbox{and} \\right.\\\\\n& & \\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\left. f(\\Xk+\\Dk\\urandom),\\ \\mbox{respectively, for}\\ \\Dk \\right\\rbrace\n\\end{eqnarray*}\nsatisfy the condition $\\prob{J_k|\\fqdalgebra}=\\E{\\ijk|\\fqdalgebra}\\geq \\beta_f$ almost surely. The estimates $\\Fok$ and $\\Fuk$ are called ``good'' if $\\ijk=1$. Otherwise, they are called ``bad.''\n\\end{definition}\n\n\\begin{assumption}\\label{efkfAssump}\nFor fixed $\\ef>0$, {\\bl $\\gamma>2$, $\\nu,\\tau\\in (0,1)$ satisfying $\\frac{\\nu}{1-\\nu}\\geq \\frac{2(\\tau^{-2}-1)}{\\gamma-2}$} and $\\beta_f\\in ({\\bl 1/2},1)$ {\\bl satisfying  $\\frac{\\beta_f}{1-\\beta_f}\\geq \\frac{4\\nu}{(1-\\nu)(1-\\tau^2)}$}, the following hold for the random quantities generated by Algorithm~\\ref{algoStoDARS}:\n\\begin{itemize}\n\\item[(i)] The sequence $\\accolade{\\Fok,\\Fuk}$ is $\\beta_f$-probabilistically $\\ef$-accurate.\n\\item[(ii)] The sequence $\\accolade{\\Fok,\\Fuk}$ satisfies the following $\\kappa_f$-variance conditions, each of which holds almost surely:\n\\[\\E{\\abs{\\Fok-f(\\Xk)}^2|\\fqdalgebra}\\leq \\ef^2(1-\\beta_f)\\Dk^4\\quad\\mbox{and}\\quad \\E{\\abs{\\Fuk-f(\\Xk+\\Dk\\urandom)}^2|\\fqdalgebra}\\leq \\ef^2(1-\\beta_f)\\Dk^4.\\]\n\\end{itemize}\n\\end{assumption}\n\n\\begin{assumption}\\label{boundedFassumpt}{\\bl (Inspired by~\\cite[Assumption~10.5]{CoScVibook},~\\cite[Theorems~3.4 and Section~3.8]{KoLeTo03a})}\nThe function $f$ is bounded from below; that is, $-\\infty<f_{\\min}\\leq f(\\x)$ for some $f_{\\min}\\in\\R$ and all $\\x\\in{\\bl \\mathcal{X}}$.\n\\end{assumption}\n\nBy using an existing supermartingale-based framework proposed by Blanchet et al.~\\cite{blanchet2016convergence}, this section derives in Theorem~\\ref{complexResult} an upper bound on the expected number of iterations required by Algorithm~\\ref{algoStoDARS} to drive~$\\normii{\\nabla f(\\Xk)}$ below a given threshold $\\epsilon\\in (0,1)$, when applied to Problem~\\eqref{probl1} in the particular case~$\\bcX:=\\rn$. Following~\\cite{dzahini2020expected,DzaWildSub2022,paquette2018stochastic}, Section~\\ref{sec51} introduces a general renewal-reward martingale process and its associated {\\it stopping time} along with related assumptions. Then Section~\\ref{sec52} demonstrates that these assumptions are satisfied by the stochastic process generated by Algorithm~\\ref{algoStoDARS}, leading to the desired complexity result.\n\n\n\\begin{definition}\\cite[Definition~1]{blanchet2016convergence}\n$\\Tepsilon$ is called a \\emph{stopping time} with respect to the discrete time process $\\accolade{\\X_k}_{k\\in\\N}$ if, for all $k\\in\\N$, $\\accolade{\\Tepsilon=k}\\in \\sigma\\left(\\X_1,\\dots,\\X_k \\right)$. \n\\end{definition}\n\nConsider the discrete time process $\\accolade{\\left(\\Phik,\\Dk\\right)}_{k\\in\\N}$, with $\\Phik, \\Dk\\in [0,\\infty)$. On the same probability space as $\\accolade{\\left(\\Phik,\\Dk\\right)}_{k\\in\\N}$, let us introduce the biased random walk process $\\accolade{\\Wtildek}_{k\\in\\N}$ satisfying\n\\begin{equation}\\label{dynamicsWk}\n\\prob{\\Wtildekun=1|\\mathcal{H}_k}=q\\qquad \\mbox{ and }\\qquad \\prob{\\Wtildekun=-1|\\mathcal{H}_k}=1-q,\n\\end{equation}\nwhere $\\mathcal{H}_k=\\sigma\\left( \\left(\\PhiZero,\\Dzero,\\WtildeZero\\right),\\dots,\\left(\\Phik,\\Dk,\\Wtildek\\right) \\right)$ with ${\\WtildeZero=1}$, and  $q\\in (1/2, 1)$.\n\nLet $\\accolade{\\Tepsilon}_{\\epsilon>0}$ be a family of stopping times with respect to $\\accoladekinN{\\mathcal{H}_k}$, parameterized by $\\epsilon$. In order to derive in~\\cite{blanchet2016convergence,dzahini2020expected,DzaWildSub2022,paquette2018stochastic} an upper bound on $\\E{\\Tepsilon}$,  the following are assumed.\n\n\n\\begin{assumption}{\\bl \\cite{blanchet2016convergence}}\\label{assumptionRenewal}\nThe discrete time process $\\accoladekinN{\\left(\\Phik,\\Dk,\\Wtildek\\right)}$ satisfies the following:\n\\begin{itemize}\n\\item[(i)] There exist $\\lambda>0$ and $\\dmax=\\delta_0 e^{\\lambda j_{\\max}}$ such that $\\Dk\\leq \\dmax$ for all $k$, where $j_{\\max}\\in\\Z$.\n\\item[(ii)] For some $j_{\\epsilon}\\in\\Z\\cap (-\\infty, 0]$, there exists $\\depsilon=\\delta_0 e^{\\lambda j_{\\epsilon}}$ such that for all $k$\n\\begin{equation}\\label{dynamicsRenewal} \n\\itksup\\Dkun\\geq \\itksup\\min\\accolade{\\Dk e^{\\lambda\\Wtildekun}, \\depsilon}, \n\\end{equation}\nwhere $\\Wtildekun$ satisfies~\\eqref{dynamicsWk}. \n\\item[(iii)] There exist a nondecreasing function $h:(0,\\infty)\\to (0,\\infty)$ and a constant $\\varrho>0$ such that for all $k$\n\\[\\E{\\Phikun-\\Phik|\\mathcal{H}_k}\\itksup\\leq-\\varrho h\\left(\\Dk\\right)\\itksup\\quad\\mbox{almost surely.}\\]\n\\end{itemize}\n\\end{assumption}\n\nConsider the discrete time process $\\accoladekinN{\\left(\\Phik,\\Dk,\\Wtildek\\right)}$ defined by the random function~$\\Phik$ introduced in Theorem~\\ref{zerothOrdTheor}, the stepsize~$\\Dk$, and the random variables $\\Wtildek:=2\\left(\\iak\\ijk-\\frac{1}{2}\\right)$, where \n\\begin{equation}\\label{AkEventDef}\nA_k:=\\accolade{\\alpha_Q\\kappa(\\pollDrandom_k)\\normii{\\nabla f(\\Xk)}\\leq -\\sqrt{\\frac{n}{p}}\\nabla f(\\Xk)^{\\top}\\left(\\Ukprandom{\\ubar{\\hat{\\di}}^p_k}\\right)}.\n\\end{equation}\n\nFor an arbitrary fixed $\\epsilon\\in (0,1)$, consider the random number $\\Tepsilon:=\\inf\\accolade{k\\in\\N:\\normii{\\nabla f(\\Xk)}\\leq \\eps}$ of iterations required by Algorithm~\\ref{algoStoDARS} to drive the norm of the gradient of~$f$ below~$\\epsilon$.\n\n\n\\begin{assumption}\\label{kapminAssump}{\\bl (\\cite[Assumption~(3.12)]{KoLeTo03a}  and~\\cite[Assumption~1]{Vicente2013})}\n%To SW: above it's really Assumption~\"(3.12)\"; it's actually an equation, not a standalone assumption\nThere exists $\\kappa_{\\min}>0$ such that the realizations of the PSSs used by Algorithm~\\ref{algoStoDARS} for subspace polling satisfy $\\kappa(\\pollD_k)\\geq \\kappa_{\\min}$.\n\\end{assumption}\n\n\\begin{assumption}\\label{gradLipAssump}{\\bl (Inspired by~\\cite[Theorem~2.8]{CoScVibook}, \\cite[Theorem~3.3]{KoLeTo03a}, and~\\cite[Theorem~1]{Vicente2013}).}\nThe gradient $\\nabla f$ of the objective function is $L_g$-Lipschitz continuous {\\bl in an open set containing all the points in the poll sets~$\\mathcal{P}_k$}.\n%, with $L_g>\\kappa_{\\min}$.\n\\end{assumption}\n\nIn the following result inspired by~\\cite[Lemma~2]{dzahini2020expected}, $\\depsilon$ is a constant defined by \n\\begin{equation}\\label{depsEq}\n\\depsilon:=\\frac{\\epsilon}{\\xi}\\qquad\\mbox{with}\\qquad\\xi>\\kappa_{\\min}^{-1}\\alpha_Q^{-1}\\left(L_g\\sqrt{\\frac{n}{p}}+(\\gamma+2)\\ef\\right)=:\\hat{\\xi}.\n\\end{equation}\n\nThus, as in~\\cite{dzahini2020expected} and~\\cite{blanchet2016convergence}, it can be assumed without any loss of generality that $\\depsilon=\\tau^{-i}\\delta_0$ for some integer $i\\leq 0$, so that $\\Dk=\\tau^{-i_k}\\depsilon$ for some integer $i_k\\leq 0$.\n","units":[{"start":37999,"end":38163,"start_line":841,"end_line":843,"raw_tex":"\\begin{equation}\\label{probl1}\n\\underset{\\x\\in\\bcX}{\\min} f(\\x), \\qquad\\mbox{with}\\quad f(\\x)=\\Esp_{\\ubar{\\theta}}\\left[f_{\\ubar{\\theta}}(\\x)\\right],\n\\end{equation}","sha256_utf8":"71c0aa734bf9b6aea2a49b91c8dfbb3c97bb51d091eef36b1864e2c548c55ce1"},{"start":38164,"end":39010,"start_line":844,"end_line":844,"raw_tex":"where {\\bl the nonempty set} $\\bcX \\subseteq\\rn$ {\\bl of nonzero measure} is a feasible region; the values of the function $f:\\rn\\to\\R$ are available only through $f_{\\ubar{\\theta}}$, a stochastically noisy version of $f$;  ${\\ubar{\\theta}}$ is a random variable whose probability distribution is possibly unknown; and $\\Esp_{\\ubar{\\theta}}$ denotes the {\\it expectation} with respect to ${\\ubar{\\theta}}$. Problem~\\eqref{probl1} arises, for example, in statistical machine learning~\\cite{bottou2018optimization} where $\\ubar{\\theta}$ is a random variable describing a given sample set while $f_{\\ubar{\\theta}}$ represents the composition of a loss function and a prediction function. {\\bl We particularly target settings where the problem dimension $n>100$, which is large in the context of derivative-free methods for stochastic optimization.}\n","sha256_utf8":"69414cd02e3c2263736fb3e6f43f64fe667a84b90d0c026cb44d53e0817036cb"},{"start":50665,"end":53009,"start_line":862,"end_line":864,"raw_tex":"Throughout the manuscript, vectors are written in lowercase boldface (e.g., $\\vb\\in\\rn, n\\geq 2$), and matrices are written in uppercase boldface (e.g., $\\Ub\\in\\rnp$). Unless otherwise stated, all random quantities are underlined (e.g., $\\ubar{z}$, $\\vrandom$, $\\Urandom$) and are defined on the same probability space $(\\Omega,\\F,\\pr)$. The nonempty set~$\\Omega$ referred to as the {\\it sample space} is a collection of elements~$\\omega$ called {\\it sample points}, while subsets of~$\\Omega$ are called {\\it events}. The collection, $\\F$, of these events is called a $\\sigma$-algebra; and the finite measure~$\\pr$ defined on the {\\it measurable space} $(\\Omega,\\F)$, with $\\prob{\\Omega}=1$, is a {\\it probability measure}. A {\\it filtration} is an increasing subsequence $\\accolade{\\mathcal{H}_k}_k$ of $\\sigma$-algebras of~$\\F$. By denoting by~$\\mathscr{B}(\\rn)$ the Borel $\\sigma$-algebra generated by open sets of~$\\rn$, a real-valued random variable~$\\ubar{z}$ is a measurable map $\\ubar{z}:(\\Omega,\\F,\\pr)\\to (\\R,\\mathscr{B}(\\R))$, where measurability means that $\\accolade{\\ubar{z}\\in \\mathscr{E}}:=\\accolade{\\omega\\in\\Omega:\\ubar{z}(\\omega)\\in\\mathscr{E}}=:\\ubar{z}^{-1}(\\mathscr{E})\\in\\F$ for all $\\mathscr{E}\\in \\mathscr{B}(\\R)$~\\cite{bhattacharya2007basic}. A real-valued random matrix is a matrix-valued random variable, that is, a matrix whose entries are real-valued random variables. $\\normii{\\ub}:=\\left(\\sum_{i=1}^n u_i^2\\right)^{1/2}$ denotes the Euclidean norm of a vector $\\ub:=(u_1,\\dots,u_n)^{\\top}\\in\\rn$.  $\\norme{\\cdot}$ is used for the norm of a matrix and is assumed \n%GAIL - bu \"supposed to be\" do you mean that it really is not? or do you mean it is assumed to be?\nto be {\\it consistent} with the Euclidean norm, that is, $\\normii{\\Q\\ub}\\leq \\norme{\\Q}\\normii{\\ub}$, which holds for both spectral and Frobenius matrix norms. $\\ub^{\\top}\\vb=\\scal{\\ub}{\\vb}:=\\sum_{i=1}^n u_i v_i$ denotes the {\\it scalar product} or {\\it dot product} of the vectors $\\ub$ and $\\vb:=(v_1,\\dots,v_n)^{\\top}\\in\\rn$. $K_{\\infty}$ denotes the random set of all {\\it unsuccessful iterations} of a given algorithm. A realization of a random set~$K$ is denoted by~$\\mathcal{K}$; that is, $K(\\omega)=\\mathcal{K}$ for some~$\\omega\\in\\Omega$. {\\bl For $a\\leq b$, $\\intbracket{a,b}:=[a,b]\\cap\\Z$ is the set of integers in the interval $[a,b]$.}\n","sha256_utf8":"fc7346a3f5ca51209e227b7d5ce0ceb010a20e1da22665621428963acd8c0fdf"},{"start":54018,"end":55653,"start_line":876,"end_line":880,"raw_tex":"SDDS algorithms were introduced in~\\cite{dzahini2020expected} to solve the stochastic optimization problem~\\eqref{probl1} in the particular case $\\bcX=\\rn$. These algorithms use a framework inspired by that of StoMADS~\\cite{audet2019stomads} and the broad class of methods presented in~\\cite{Vicente2013} using a {\\it generating set search} method~\\cite{KoLeTo03a}, such as pattern search and generalized pattern search~\\cite{AuDe03a}. Each iteration of SDDS algorithms is composed of two main steps: an optional {\\it search}, not shown in~\\cite[Algorithm~1]{dzahini2020expected}, and not detailed in Algorithm~\\ref{vagueDirAlgo} for simplicity, and the ``more disciplined''~\\cite{LaMeWi2019} {\\it poll} on which  the theoretical analyses rely. Broadly speaking, each poll step of an SDDS method generates a finite set of {\\it poll points} by taking an {\\it incumbent solution}~$\\xk\\in\\rn$ and adding terms of the form~$\\dk\\di^n$, where $\\dk>0$ is the {\\it stepsize} and $\\di^n\\in\\rn$ is a nonzero vector from a finite set $\\mathbb{D}^n_k\\subset\\rn$ of {\\it poll directions}. More precisely, $\\mathbb{D}^n_k$ is a PSS, meaning that any vector in~$\\rn$ can be written as a nonnegative linear combination of elements in~$\\mathbb{D}^n_k$~\\cite[Definition~6.1]{AuHa2017} or, equivalently, the cosine measure $\\kappa(\\mathbb{D}^n_k)>0$, where \n%for a given $\\mathbb{D}\\subset\\rn$,\n\\begin{equation}\\label{cosineMeasureDef}\n\\kappa(\\mathbb{D}^q):=\\underset{\\wb\\in \\R^q,\\wb\\neq \\bm{0}}{\\min}\\ \\underset{\\di^q\\in \\mathbb{D}^q}{\\max}\\frac{\\scal{\\wb}{\\di^q}}{\\normii{\\wb}\\normii{\\di^q}}\\quad\\mbox{with}\\quad \\mathbb{D}^q\\subset\\R^q.\n\\end{equation}","sha256_utf8":"5f5a26e111e775294892565eb37bb353597441faec539a427bbd4e4d21883234"},{"start":58959,"end":59220,"start_line":916,"end_line":918,"raw_tex":"\\begin{equation}\\label{cosineDescentDkn}\n\\mbox{for any}\\ \\vi\\in\\rn,\\quad\\exists \\di_{\\star}^n(\\vi)\\in \\mathbb{D}^n\\subset\\rn\\quad\\mbox{such that}\\quad \\kappa(\\mathbb{D}^n)\\normii{\\vi}\\normii{\\di^n_{\\star}(\\vi)}\\leq \\scal{\\vi}{\\di^n_{\\star}(\\vi)}.\n\\end{equation}","sha256_utf8":"9cbf677077ba72668e46acfcc4719522cf0f303ac79579ae78adba33bf98984a"},{"start":61684,"end":62097,"start_line":931,"end_line":935,"raw_tex":"\\begin{definition}\\label{wellAlignment}\nFor fixed $\\alpha_Q,\\beta_Q\\in (0,1/2)$, \n%a sequence $\\accoladekinN{\\QkRandom}$ of independent \na random matrix $\\Qrandom\\in\\rnp$ is called $(1-\\beta_Q)$-probabilistically $\\alpha_Q$-well aligned if, for all $\\vb\\in\\rn$, $\\prob{\\normii{\\Qrandom^{\\top}\\vb}\\geq \\alpha_Q\\normii{\\vb}}\\geq 1-\\beta_Q$. Such matrices are referred to as $WAM(\\alpha_Q,\\beta_Q)$.\n\\end{definition}","sha256_utf8":"ff2c7df51dbfc2dd1d68709c72d53cb734cdd26566075337d033d1e25b537db8"},{"start":63322,"end":63810,"start_line":944,"end_line":947,"raw_tex":"\\begin{assumption}\\label{JltAss1}{\\bl (Inspired by~\\cite[Assumptions~2 and~3]{DzaWildSub2022} and~\\cite{CRsubspace2021})}\nThe random matrix $\\Qrandom\\in\\rnp$ is a $WAM(\\alpha_Q,\\beta_Q)$ for fixed $\\alpha_Q,\\beta_Q\\in (0,1/2)$ and satisfies $\\norme{\\Qrandom}\\leq\\Qmax$ almost surely for some $\\Qmax>0$. \n%and $\\prob{\\normii{\\QrandomTranspose\\bm{v}}\\geq \\alpha_Q\\normii{\\bm{v}}}\\geq 1-\\beta_Q$ for all $\\bm{v}\\in\\rn$, for some $\\alpha_Q\\in (0,1)$ and $\\beta_Q\\in (0,1/2)$.\n\\end{assumption}","sha256_utf8":"a2d594800ab920e3a5be97f31d6055474a7ae3cc134ff45b4e245075b07e171b"},{"start":66855,"end":68285,"start_line":973,"end_line":975,"raw_tex":"The poll directions used by Algorithm~\\ref{algoStoDARS} are generated by using Haar-distributed orthogonal random matrices. Recall that a matrix $\\bm{O}\\in \\rnn$ is {\\it orthogonal} if and only if $\\bm{O}^{\\top}\\bm{O}=\\bm{O}\\bm{O}^{\\top}=\\bm{I}_n$ or equivalently if its columns form an orthonormal basis of $\\rn$~\\cite[Section~1.1]{MeckesElizHaar2019}. Throughout the manuscript, the compact Lie group of $n\\times n$ orthogonal matrices over~$\\R$ will be denoted by $\\mathbb{O}(n)$. \n%We also recall that $\\mathbb{O}(n)$ is a group under multiplication, and that this group admits a compact Lie group structure, that is, it is also a differentiable manifold such that the multiplication operation $(\\bm{A}, \\bm{B})\\mapsto \\bm{A}\\bm{B}$ and the inversion operation $\\bm{A}\\mapsto \\bm{A}^{-1}$ are smooth maps. \nA {\\it uniform random element} of $\\mathbb{O}(n)$ is a random $\\Urandom\\in \\mathbb{O}(n)$ whose distribution is translation invariant; that is, for any fixed $\\bm{O}\\in \\mathbb{O}(n)$, $\\bm{O}\\Urandom\\equald \\Urandom\\bm{O}\\equald \\Urandom$~\\cite[Section~1.2]{MeckesElizHaar2019}, where ``$\\equald$'' is used for the equality in distribution. In particular, there exists a unique translation-invariant probability measure on $\\mathbb{O}(n)$ called Haar measure on~$\\mathbb{O}(n)$~\\cite[Theorem~1.4]{MeckesElizHaar2019}. Any random matrix whose distribution is Haar measure will be called Haar-{\\it distributed matrix}. \n","sha256_utf8":"9c65241b07a4bcf26a7657114892d697f13054aecce303a6d036818584d4ae70"},{"start":97149,"end":97574,"start_line":1192,"end_line":1197,"raw_tex":"\\begin{definition}\\label{epsAccDef0}\nGiven $\\ef>0$, $\\xk\\in\\rn$, and $\\ub:=\\Ukp\\di^p=\\sqrt{\\frac{p}{n}}\\Qk\\di^p\\in\\rn$ with $\\di^p\\in\\pollD_k$, $\\fok$ and $\\fuk$ are called $\\ef$-accurate estimates of $f(\\xk)$ and $f(\\xk+\\dk\\ub)$, respectively, for a given $\\dk$ if\n\\begin{equation}\\label{epsAccDef1}\n\\abs{\\fok-f(\\xk)}\\leq \\ef\\dk^2\\qquad \\mbox{and}\\qquad \\abs{\\fuk-f(\\xk+\\dk\\ub)}\\leq \\ef\\dk^2.\n\\end{equation}\n\\end{definition}","sha256_utf8":"137d73020f9a645ff98acb39b939944ba643ad5055b504bf5928759389647dc2"},{"start":98212,"end":103522,"start_line":1218,"end_line":1270,"raw_tex":"\\begin{algorithm}[H]\n\\caption{StoDARS algorithm}\n\\label{algoStoDARS} \n%\\nonl\n% $\\accolade{\\Urandomtilde_t}_{t=0}^\\infty$, where the $\\Urandomtilde_t:=\\left[\\Urandomtilde_{t,\\ntp}\\ \\ \\Urandomtilde_{t,\\ntnmp}\\right]\\in\\mathbb{O}(n)$ are Haar-distributed, is a sequence of independent matrices, with $p\\in\\intbracket{1,n}$ chosen so that $\\sqrt{\\frac{n}{p}}\\Urandomtilde_{t,\\ntp}\\in\\rnp$ is $WAM(\\alpha_Q,\\beta_Q)$ for some $\\alpha_Q, \\beta_Q\\in(0,1/2)$. \\\\\n% $\\accolade{\\BrandomtildeSet_t^+}_{t=0}^\\infty$ is a sequence of random PSSs such that the matrices $\\Brandomtilde_t^+\\in\\R^{p\\times m}$ with unit columns in $\\BrandomtildeSet_t^+\\subset\\mathbb{S}^{p-1}$ are independent.    \\\\\n% $ $\\\\\n\\textbf{[0] Initialization}\\\\\n\\hspace*{10mm}Choose $\\x_0\\in\\bcX\\subseteq\\rn$, $\\delta_0>0$, $\\ef>0$, $\\gamma>2$, $0<\\tau<\\left(\\frac{\\gamma-2}{\\gamma+2}\\right)^{1/2}$, $j_{\\max}\\in\\N$, $\\dmax=\\tau^{-j_{\\max}}\\delta_0$ \\\\ \n\\hspace*{10mm}$\\ell_0=0$ and $t_0=0$. Generate $\\tilde{\\Ub}_{0,\\ntp}$, where $\\Urandomtilde_0:=\\left[\\Urandomtilde_{0,\\ntp}\\ \\ \\Urandomtilde_{0,\\ntnmp}\\right]\\in\\mathbb{O}(n)$ is Haar-distributed, \\\\\n\\hspace*{10mm}with $p\\in\\intbracket{1,n}$ chosen so that $\\sqrt{\\frac{n}{p}}\\Urandomtilde_{0,\\ntp}\\in\\rnp$ is $WAM(\\alpha_Q,\\beta_Q)$ for some $\\alpha_Q, \\beta_Q\\in(0,1/2)$.\\\\\n\\hspace*{10mm}Construct a PSS $\\tilde{\\mathbb{D}}^p_0\\subset\\spOne$ using {\\bl any matrix} $\\Drandomtildep_0\\in\\R^{p\\times m}$, with $m\\in {\\bl \\intbracket{p+1,2p}}$, {\\bl generated}\\\\ \n\\hspace*{10mm}{\\bl independently of $\\Urandomtilde_{0}$, whose columns form a PSS of~$\\rp$}.\\\\\n\\hspace*{10mm}Set the iteration counter $k \\gets 0$.\\\\\n% {\\bl\\textbf{[1] Matrix index update}\\\\\n% \\hspace*{10mm}If $\\ell_{k} \\geq  \\max_{i\\leq k-1} \\ell_i$ or $k=0$, set $t_{k}\\gets \\ell_{k}$.   \\\\\n% \\hspace*{10mm}Otherwise $t_{k}\\gets 1+\\max_{i\\leq k-1} t_i$.\\\\}\n\\textbf{[{1}] Matrix and PSS for subspace polling}\\\\\n% \\hspace*{10mm}If $k\\geq 1$ and $t_k=t_{i_0}$ for some $i_0\\in \\intbracket{0,k-1}$   \\\\\n% \\hspace*{16mm}Generate $\\tilde{\\Ub}_{t_k,\\ntp}$ using an independent copy $\\Urandomtilde_{t_k,\\ntp}$ of $\\Urandomtilde_{t_{i_0},\\ntp}$ such that  \\\\\n% \\hspace*{16mm}$\\{\\Urandomtilde_{t_i,\\ntp}\\}_{i=0}^{k-1}$ and $\\Urandomtilde_{t_k,\\ntp}$ are i.i.d. \\\\\n% \\hspace*{16mm}Construct a PSS $\\tilde{\\mathbb{D}}^p_{t_k}\\subset\\spOne$ using the columns of an independent copy $\\Drandomtildep_{t_k}$ of $\\Drandomtildep_{t_{i_0}}$ \\\\\n% \\hspace*{16mm}such that $\\{\\Drandomtildep_{t_i}\\}_{i=0}^{k-1}$ and $\\Drandomtildep_{t_k}$ are i.i.d. \\\\\n% \\hspace*{10mm}If $k\\geq 1$ and $t_k\\not\\in\\accolade{t_0,\\dots,t_{k-1}}$ \\\\\n\\hspace*{10mm}Generate $\\tilde{\\Ub}_{t_k,\\ntp}$ as a realization of $\\Urandomtilde_{t_k,\\ntp}$, with $\\{\\Urandomtilde_{t_i,\\ntp}\\}_{i=0}^{k}$ {\\bl obtained from the i.i.d. $\\{\\Urandomtilde_{t_i}\\}_{i=0}^{k}$}.\\\\\n\\hspace*{10mm}Construct a PSS $\\tilde{\\mathbb{D}}^p_{t_k}\\subset\\spOne$ using the columns of $\\Drandomtildep_{t_k}$, where the independent $\\{\\Drandomtildep_{t_i}\\}_{i=0}^{k}$ {\\bl are generated}\\\\ \n\\hspace*{10mm}{\\bl independently of $\\{\\Urandomtilde_{t_i}\\}_{i=0}^{k}$}.  \\\\\n% \\hspace*{16mm}such that $\\accolade{\\Urandom_{t_i,\\ntp}}_{i=0}^k$ and $\\{\\Urandomtilde_{t_i,\\ntp}\\}_{i=0}^k$ are independent.   \\\\\n% \\hspace*{16mm}Select $\\BrandomSet_{t_k}^+$ as the columns of an independent copy $\\Brandom_{t_k}^+$ of $\\Brandomtilde_{t_k}^+$, where $\\Brandom_{t_k}^+$ is independent of  \\\\\n% % \\hspace*{16mm}$\\Brandom_{t_i}^+,\\ i\\leq k-1$, and $\\Brandomtilde_t^+, t\\geq 0$. \\\\\n% \\hspace*{16mm}$\\Brandom_{t_i}^+,\\ i\\leq k-1$, and $\\accolade{\\Brandom_{t_i}^+}_{i=0}^k$ and $\\{\\Brandomtilde_{t_i}^+\\}_{i=0}^k$ are independent. \\\\\n% \\hspace*{10mm}Otherwise $\\Urandom_{t_k,\\ntp}:=\\Urandomtilde_{t_k,\\ntp}$ and $\\BrandomSet_{t_k}^+:=\\BrandomtildeSet_{t_k}^+$. \\\\\n\\hspace*{10mm}Define $\\Ukp:=\\tilde{\\Ub}_{t_k,\\ntp}$ and $\\pollD_k:=\\tilde{\\mathbb{D}}^p_{t_k}$.\\\\\n\\textbf{[{2}] Poll}\\\\\n%\\hspace*{10mm}Select a PSS $\\pollD_k\\subset\\rp$ such that $\\pollDrandom_k$ is independent of $\\pollDrandom_0,\\dots,\\pollDrandom_{k-1}$ when $k\\geq 1$. \\\\\n\\hspace*{10mm}Generate a set of poll points $\\mathcal{P}_k:=\\{\\xk+\\dk\\ub: \\ub\\in\\mathbb{U}_k^n \\}$; where $\\mathbb{U}_k^n:=\\{\\Ukp\\di^p:\\di^p\\in \\pollD_k\\}$, {\\bbl and}\\\\\n\\hspace*{10mm}{\\bbbl $\\pollD_k$ is generated independently of $\\Ukp$}.\\\\\n%\\hspace*{8mm}{}\\\\ \n\\hspace*{10mm}Obtain estimates\\footnotemark $\\fuk$ and $\\fok$ of $f$, respectively, at ${\\xk+\\dk\\ub}\\in\\mathcal{P}_k$ and $\\xk$.\\\\\n%\\hspace*{10mm}function evaluations.\\\\\n%\\hspace*{10mm}\\textbf{{Success}}\\\\\n\\hspace*{10mm}If $\\Pkfeas:=\\Pk\\cap\\bcX\\neq\\emptyset$ and $\\fuk -\\fok \\leq -\\gamma \\ef\\dk^2$ for some ${\\xk+\\dk\\ub}\\in\\Pkfeas$ (\\textbf{successful})\\\\ \n\\hspace*{16mm}set $\\xkun\\gets \\xk+\\dk\\ub$,  $\\dkun\\gets \\min\\{\\tau^{-1}\\dk,\\dmax\\}$, and $\\ell_{k+1}\\gets \\ell_k-1$. \\\\\n%\\hspace*{10mm}\\textbf{{Failure}}\\\\\n\\hspace*{10mm}Otherwise (\\textbf{unsuccessful}) set $\\xkun\\gets \\xk$, $\\delta_{k+1}\\gets\\tau\\dk$, and $\\ell_{k+1}\\gets \\ell_k+1$. \\\\\n\\textbf{[3] Matrix index update}\\\\\n\\hspace*{10mm}If $\\ell_{k+1} \\geq  \\max_{i\\leq k} \\ell_i$, set $t_{k+1}\\gets \\ell_{k+1}$.   \\\\\n\\hspace*{10mm}Otherwise $t_{k+1}\\gets 1+\\max_{i\\leq k} t_i$.\\\\\n\\textbf{[4] Termination}\\\\\n\\hspace*{10mm}If no termination criterion is met, set $k\\gets k+1$ and go to \\textbf{[1]}.\\\\\n%\\hspace*{10mm}\\\\\n\\hspace*{10mm}Otherwise stop.\n\\end{algorithm}","sha256_utf8":"0c634a324393ba942c705323d86d4b6f04e7465b1c6d2d5cc73e3b87384e551f"},{"start":119750,"end":120323,"start_line":1353,"end_line":1355,"raw_tex":"\\begin{assumption}\\label{QkHaarAssumpt}{\\bl (Motivated by Theorem~\\ref{JLforHaarTheor})}\nFor all $k\\geq 0$, the independent random matrices  $\\Drandom_k\\in\\R^{p\\times m}$, with $m\\in {\\bl \\intbracket{p+1,2p}}$, and $\\Ukprandom\\in\\rnp$ used for subspace polling in Algorithm~\\ref{algoStoDARS} are such that $\\pollDrandom_k\\subset\\spOne$, $\\Urandom_k:=\\left[\\Urandom_{k,\\ntp}\\ \\, \\Urandom_{k,\\ntnmp}\\right]\\in \\mathbb{O}(n)$ is Haar-distributed, and $\\QkRandom:=\\sqrt{\\frac{n}{p}}\\Ukprandom$ is $WAM(\\alpha_Q,\\beta_Q)$ for some $\\alpha_Q, \\beta_Q\\in(0,1/2)$.\n\\end{assumption}","sha256_utf8":"774db7e8979e9250a2bbdf42784ba4710539d68d4007d85aecbf8012c846a6c5"},{"start":120327,"end":121716,"start_line":1359,"end_line":1364,"raw_tex":"Next, $(\\QkRandom)_{ij}$, with $(i,j)\\in\\mathbb{I}_{\\ntp}:=\\intbracket{1,n}\\times\\intbracket{1,p}$, denote the entries of $\\QkRandom$. Likewise, $(\\Drandom_k)_{i'j'}$, with $(i',j')\\in\\mathbb{I}_{p,m}$, denote the entries of $\\pollDrandom_k$. The conditioning on the past is formalized by using the filtrations $\\accoladekinN{\\falgebra}$ and $\\accoladekinN{\\fqdalgebra}$ defined by\n\\begin{eqnarray*}\n\\falgebra&:=&\\sigma\\left({\\ubar{f}}_{\\ell}({\\bm{0}}), {\\ubar{f}}_{\\ell}(\\urandom),(\\Qrandom_\\ell)_{ij}, (\\Drandom_\\ell)_{i'j'}\\ \\mbox{with}\\ \\ell\\in\\intbracket{0,k-1}, (i,j)\\in\\mathbb{I}_{\\ntp}\\ \\mbox{and}\\ (i',j')\\in\\mathbb{I}_{p,m}\\right)\\\\\n\\fqdalgebra&:=&\\sigma\\left({\\ubar{f}}_{\\ell}({\\bm{0}}), {\\ubar{f}}_{\\ell}(\\urandom),(\\Qrandom_\\ell)_{ij}, (\\Drandom_\\ell)_{i'j'},(\\QkRandom)_{ij}, (\\Drandom_k)_{i'j'} \\ \\mbox{with}\\ \\ell\\in\\intbracket{0,k-1}, (i,j)\\in\\mathbb{I}_{\\ntp}\\ \\mbox{and}\\ (i',j')\\in\\mathbb{I}_{p,m}\\right),\n\\end{eqnarray*}\nwhere for {\\it completeness}~\\cite{billingsley1995ThirdEdition}, $\\F_{-1}:=\\sigma(\\x_0)$. The idea behind the above definitions is explained in~\\cite[Remark~2.1]{DzaWildSub2022}. By construction, $\\QkRandom$ and $\\Drandom_k$ are $\\fqdalgebra$-measurable so that $\\E{\\QkRandom|\\fqdalgebra}=\\QkRandom$ almost surely and $\\E{\\Drandom_k|\\fqdalgebra}=\\Drandom_k$ almost surely. The sufficient accuracy of the random estimates is formalized as follows.\n","sha256_utf8":"dba1c2093fdbc20a0950edca4c3bf41019b67c46bef2a6f086e3252806fedbd6"},{"start":121717,"end":122505,"start_line":1366,"end_line":1373,"raw_tex":"\\begin{definition}\nA sequence $\\accolade{\\Fok,\\Fuk}$ of random estimates is called $\\beta_f$-probabilistically $\\ef$-accurate with respect to the corresponding sequence $\\accolade{\\Xk,\\Dk,\\QkRandom,\\Drandom_k}$ if the events\n\\begin{eqnarray*}\nJ_k&:=&\\left\\lbrace \\mbox{Given}\\ \\QkRandom, \\Drandom_k\\ \\mbox{and}\\ \\Xk, \\Fok\\ \\mbox{and}\\ \\Fuk\\ \\mbox{are}\\ \\ef\\mbox{-accurate}\\ \\mbox{estimates of}\\ f(\\Xk)\\ \\mbox{and} \\right.\\\\\n& & \\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\left. f(\\Xk+\\Dk\\urandom),\\ \\mbox{respectively, for}\\ \\Dk \\right\\rbrace\n\\end{eqnarray*}\nsatisfy the condition $\\prob{J_k|\\fqdalgebra}=\\E{\\ijk|\\fqdalgebra}\\geq \\beta_f$ almost surely. The estimates $\\Fok$ and $\\Fuk$ are called ``good'' if $\\ijk=1$. Otherwise, they are called ``bad.''\n\\end{definition}","sha256_utf8":"f0605b0601971da79d04af66212e7bcda6b3dd054203abb241192106cea752ed"},{"start":124024,"end":124825,"start_line":1381,"end_line":1388,"raw_tex":"\\begin{assumption}\\label{efkfAssump}\nFor fixed $\\ef>0$, {\\bl $\\gamma>2$, $\\nu,\\tau\\in (0,1)$ satisfying $\\frac{\\nu}{1-\\nu}\\geq \\frac{2(\\tau^{-2}-1)}{\\gamma-2}$} and $\\beta_f\\in ({\\bl 1/2},1)$ {\\bl satisfying  $\\frac{\\beta_f}{1-\\beta_f}\\geq \\frac{4\\nu}{(1-\\nu)(1-\\tau^2)}$}, the following hold for the random quantities generated by Algorithm~\\ref{algoStoDARS}:\n\\begin{itemize}\n\\item[(i)] The sequence $\\accolade{\\Fok,\\Fuk}$ is $\\beta_f$-probabilistically $\\ef$-accurate.\n\\item[(ii)] The sequence $\\accolade{\\Fok,\\Fuk}$ satisfies the following $\\kappa_f$-variance conditions, each of which holds almost surely:\n\\[\\E{\\abs{\\Fok-f(\\Xk)}^2|\\fqdalgebra}\\leq \\ef^2(1-\\beta_f)\\Dk^4\\quad\\mbox{and}\\quad \\E{\\abs{\\Fuk-f(\\Xk+\\Dk\\urandom)}^2|\\fqdalgebra}\\leq \\ef^2(1-\\beta_f)\\Dk^4.\\]\n\\end{itemize}\n\\end{assumption}","sha256_utf8":"9a8538145af9dc4991079e3c4b94480e178f4e4717514fdcb60210e4c00a7c3e"},{"start":127130,"end":127427,"start_line":1408,"end_line":1410,"raw_tex":"\\begin{assumption}\\label{boundedFassumpt}{\\bl (Inspired by~\\cite[Assumption~10.5]{CoScVibook},~\\cite[Theorems~3.4 and Section~3.8]{KoLeTo03a})}\nThe function $f$ is bounded from below; that is, $-\\infty<f_{\\min}\\leq f(\\x)$ for some $f_{\\min}\\in\\R$ and all $\\x\\in{\\bl \\mathcal{X}}$.\n\\end{assumption}","sha256_utf8":"63395adb11214de3a2ea79d73f88ce4104b6234dd0a285fa286eb4672fe4f8cc"},{"start":128472,"end":129290,"start_line":1437,"end_line":1437,"raw_tex":"By using an existing supermartingale-based framework proposed by Blanchet et al.~\\cite{blanchet2016convergence}, this section derives in Theorem~\\ref{complexResult} an upper bound on the expected number of iterations required by Algorithm~\\ref{algoStoDARS} to drive~$\\normii{\\nabla f(\\Xk)}$ below a given threshold $\\epsilon\\in (0,1)$, when applied to Problem~\\eqref{probl1} in the particular case~$\\bcX:=\\rn$. Following~\\cite{dzahini2020expected,DzaWildSub2022,paquette2018stochastic}, Section~\\ref{sec51} introduces a general renewal-reward martingale process and its associated {\\it stopping time} along with related assumptions. Then Section~\\ref{sec52} demonstrates that these assumptions are satisfied by the stochastic process generated by Algorithm~\\ref{algoStoDARS}, leading to the desired complexity result.\n","sha256_utf8":"e905b8ca9e690cb4d9736f8b327007601386070e5a74c3270e65c5515bf94efd"},{"start":129495,"end":129776,"start_line":1442,"end_line":1444,"raw_tex":"\\begin{definition}\\cite[Definition~1]{blanchet2016convergence}\n$\\Tepsilon$ is called a \\emph{stopping time} with respect to the discrete time process $\\accolade{\\X_k}_{k\\in\\N}$ if, for all $k\\in\\N$, $\\accolade{\\Tepsilon=k}\\in \\sigma\\left(\\X_1,\\dots,\\X_k \\right)$. \n\\end{definition}","sha256_utf8":"cb0daab4ea231180dbb371001d7fdb0d2758a4f632726c59c99669a2da37907c"},{"start":129778,"end":130702,"start_line":1446,"end_line":1452,"raw_tex":"Consider the discrete time process $\\accolade{\\left(\\Phik,\\Dk\\right)}_{k\\in\\N}$, with $\\Phik, \\Dk\\in [0,\\infty)$. On the same probability space as $\\accolade{\\left(\\Phik,\\Dk\\right)}_{k\\in\\N}$, let us introduce the biased random walk process $\\accolade{\\Wtildek}_{k\\in\\N}$ satisfying\n\\begin{equation}\\label{dynamicsWk}\n\\prob{\\Wtildekun=1|\\mathcal{H}_k}=q\\qquad \\mbox{ and }\\qquad \\prob{\\Wtildekun=-1|\\mathcal{H}_k}=1-q,\n\\end{equation}\nwhere $\\mathcal{H}_k=\\sigma\\left( \\left(\\PhiZero,\\Dzero,\\WtildeZero\\right),\\dots,\\left(\\Phik,\\Dk,\\Wtildek\\right) \\right)$ with ${\\WtildeZero=1}$, and  $q\\in (1/2, 1)$.\n\nLet $\\accolade{\\Tepsilon}_{\\epsilon>0}$ be a family of stopping times with respect to $\\accoladekinN{\\mathcal{H}_k}$, parameterized by $\\epsilon$. In order to derive in~\\cite{blanchet2016convergence,dzahini2020expected,DzaWildSub2022,paquette2018stochastic} an upper bound on $\\E{\\Tepsilon}$,  the following are assumed.\n","sha256_utf8":"47ef16cf5de039893d73a3625f8b99ee5fda2dfd17c0b58b62b40e284f7e34ce"},{"start":130703,"end":131626,"start_line":1454,"end_line":1466,"raw_tex":"\\begin{assumption}{\\bl \\cite{blanchet2016convergence}}\\label{assumptionRenewal}\nThe discrete time process $\\accoladekinN{\\left(\\Phik,\\Dk,\\Wtildek\\right)}$ satisfies the following:\n\\begin{itemize}\n\\item[(i)] There exist $\\lambda>0$ and $\\dmax=\\delta_0 e^{\\lambda j_{\\max}}$ such that $\\Dk\\leq \\dmax$ for all $k$, where $j_{\\max}\\in\\Z$.\n\\item[(ii)] For some $j_{\\epsilon}\\in\\Z\\cap (-\\infty, 0]$, there exists $\\depsilon=\\delta_0 e^{\\lambda j_{\\epsilon}}$ such that for all $k$\n\\begin{equation}\\label{dynamicsRenewal} \n\\itksup\\Dkun\\geq \\itksup\\min\\accolade{\\Dk e^{\\lambda\\Wtildekun}, \\depsilon}, \n\\end{equation}\nwhere $\\Wtildekun$ satisfies~\\eqref{dynamicsWk}. \n\\item[(iii)] There exist a nondecreasing function $h:(0,\\infty)\\to (0,\\infty)$ and a constant $\\varrho>0$ such that for all $k$\n\\[\\E{\\Phikun-\\Phik|\\mathcal{H}_k}\\itksup\\leq-\\varrho h\\left(\\Dk\\right)\\itksup\\quad\\mbox{almost surely.}\\]\n\\end{itemize}\n\\end{assumption}","sha256_utf8":"2047d97bf9540e5739045f85492a2942b54975f841a164330783cde02299bf9c"},{"start":132251,"end":132726,"start_line":1476,"end_line":1479,"raw_tex":"Consider the discrete time process $\\accoladekinN{\\left(\\Phik,\\Dk,\\Wtildek\\right)}$ defined by the random function~$\\Phik$ introduced in Theorem~\\ref{zerothOrdTheor}, the stepsize~$\\Dk$, and the random variables $\\Wtildek:=2\\left(\\iak\\ijk-\\frac{1}{2}\\right)$, where \n\\begin{equation}\\label{AkEventDef}\nA_k:=\\accolade{\\alpha_Q\\kappa(\\pollDrandom_k)\\normii{\\nabla f(\\Xk)}\\leq -\\sqrt{\\frac{n}{p}}\\nabla f(\\Xk)^{\\top}\\left(\\Ukprandom{\\ubar{\\hat{\\di}}^p_k}\\right)}.\n\\end{equation}","sha256_utf8":"671f4d6eb243652b308f47850b39d2c278815429e5a0a4fb9a3fce1f33b6d7af"},{"start":132727,"end":132978,"start_line":1480,"end_line":1480,"raw_tex":"For an arbitrary fixed $\\epsilon\\in (0,1)$, consider the random number $\\Tepsilon:=\\inf\\accolade{k\\in\\N:\\normii{\\nabla f(\\Xk)}\\leq \\eps}$ of iterations required by Algorithm~\\ref{algoStoDARS} to drive the norm of the gradient of~$f$ below~$\\epsilon$.\n","sha256_utf8":"020710503630b880b4bb1ae363c136595a45fca9fa14d0ba2862afff0a01bc67"},{"start":136642,"end":137053,"start_line":1504,"end_line":1507,"raw_tex":"\\begin{assumption}\\label{kapminAssump}{\\bl (\\cite[Assumption~(3.12)]{KoLeTo03a}  and~\\cite[Assumption~1]{Vicente2013})}\n%To SW: above it's really Assumption~\"(3.12)\"; it's actually an equation, not a standalone assumption\nThere exists $\\kappa_{\\min}>0$ such that the realizations of the PSSs used by Algorithm~\\ref{algoStoDARS} for subspace polling satisfy $\\kappa(\\pollD_k)\\geq \\kappa_{\\min}$.\n\\end{assumption}","sha256_utf8":"5e1269be066cd3ba9719c5de695fa88c6d56484cb8528f69fcb5b73af7ce9d88"},{"start":137054,"end":137417,"start_line":1508,"end_line":1511,"raw_tex":"\\begin{assumption}\\label{gradLipAssump}{\\bl (Inspired by~\\cite[Theorem~2.8]{CoScVibook}, \\cite[Theorem~3.3]{KoLeTo03a}, and~\\cite[Theorem~1]{Vicente2013}).}\nThe gradient $\\nabla f$ of the objective function is $L_g$-Lipschitz continuous {\\bl in an open set containing all the points in the poll sets~$\\mathcal{P}_k$}.\n%, with $L_g>\\kappa_{\\min}$.\n\\end{assumption}","sha256_utf8":"7d2e93dbea55788eea6301f2ea93e03d8709dcfc688c9d92f0e93fc8f7f33761"},{"start":137419,"end":137725,"start_line":1513,"end_line":1516,"raw_tex":"In the following result inspired by~\\cite[Lemma~2]{dzahini2020expected}, $\\depsilon$ is a constant defined by \n\\begin{equation}\\label{depsEq}\n\\depsilon:=\\frac{\\epsilon}{\\xi}\\qquad\\mbox{with}\\qquad\\xi>\\kappa_{\\min}^{-1}\\alpha_Q^{-1}\\left(L_g\\sqrt{\\frac{n}{p}}+(\\gamma+2)\\ef\\right)=:\\hat{\\xi}.\n\\end{equation}","sha256_utf8":"0e520beb76d7184c654bdddd452210880e6f8b5a6529130e377c77c0f526bacd"},{"start":137726,"end":137978,"start_line":1517,"end_line":1517,"raw_tex":"Thus, as in~\\cite{dzahini2020expected} and~\\cite{blanchet2016convergence}, it can be assumed without any loss of generality that $\\depsilon=\\tau^{-i}\\delta_0$ for some integer $i\\leq 0$, so that $\\Dk=\\tau^{-i_k}\\depsilon$ for some integer $i_k\\leq 0$.\n","sha256_utf8":"f0fb65cb12e9f6f2ec0fa205f6d5425132ae50bcf70bd1b36ac1a36e2a3f93a0"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{complexResult}\nLet all assumptions made in Lemma~\\ref{dynamicsLemma} hold. Then $\\E{\\Tepsilon}\\leq \\frac{\\tilde{\\beta}}{2\\tilde{\\beta}-1}\\cdot\\frac{\\PhiZero{\\tilde{\\xi}}^2}{\\varrho \\eps^2}+1$, for some $\\tilde{\\xi}\\geq\\hat{\\xi}$, where~{\\bl $\\varrho=\\frac{1}{2}\\beta_f(1-\\nu)(1-\\tau^2)$}, \n%is the constant of Theorem~\\ref{zerothOrdTheor}, \n$\\tilde{\\beta}$ is the same parameter of Lemma~\\ref{dynamicsLemma}, and $\\hat{\\xi}$ is given by~\\eqref{depsEq}.\n\\end{theorem}","units":[{"start":140676,"end":141164,"start_line":1540,"end_line":1544,"raw_tex":"\\begin{theorem}\\label{complexResult}\nLet all assumptions made in Lemma~\\ref{dynamicsLemma} hold. Then $\\E{\\Tepsilon}\\leq \\frac{\\tilde{\\beta}}{2\\tilde{\\beta}-1}\\cdot\\frac{\\PhiZero{\\tilde{\\xi}}^2}{\\varrho \\eps^2}+1$, for some $\\tilde{\\xi}\\geq\\hat{\\xi}$, where~{\\bl $\\varrho=\\frac{1}{2}\\beta_f(1-\\nu)(1-\\tau^2)$}, \n%is the constant of Theorem~\\ref{zerothOrdTheor}, \n$\\tilde{\\beta}$ is the same parameter of Lemma~\\ref{dynamicsLemma}, and $\\hat{\\xi}$ is given by~\\eqref{depsEq}.\n\\end{theorem}","sha256_utf8":"af6daff8ead198b3f863b7912d4a116910c9ed4e52e10f3c00275f417023f3af"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{proposition}\\label{propKappaD1}\nAssume that $\\Qrandom\\in\\rnp$ satisfies Assumption~\\ref{JltAss1}, and let $\\pollD\\subset \\rp$ be a PSS. For any $\\bm{v}\\in \\rn$, \n%there exists a direction $\\di_{\\star}\\in\\pollD$ such that \n\\begin{equation}\\label{kdprob1}\n\\prob{\\accolade{\\exists\\ubar{\\di}_{\\star}^p\\in\\pollD:\\frac{\\alpha_Q}{\\Qmax}\\kappa(\\pollD)\\normii{\\bm{v}}\\normii{\\Qrandom\\ubar{\\di}_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom\\ubar{\\di}_{\\star}^p}}}\\geq 1-\\beta_Q.\n\\end{equation}\nIn other words, with high probability, there always exists a random direction $\\Qrandom\\ubar{\\di}_{\\star}^p\\in\\accolade{\\Qrandom\\di^p:\\di^p\\in\\pollD}\\subset\\rn$ that makes an acute angle with $\\bm{v}$. The same result holds in the case of a random PSS $\\pollDrandom\\subset\\rp$; that is,\n\\begin{equation}\\label{kdprob1prime}\n\\prob{\\accolade{\\exists\\ubar{\\di}_{\\star}^p\\in\\pollDrandom:\\frac{\\alpha_Q}{\\Qmax}\\kappa(\\pollDrandom)\\normii{\\bm{v}}\\normii{\\Qrandom\\ubar{\\di}_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom\\ubar{\\di}_{\\star}^p}}}\\geq 1-\\beta_Q.\n\\end{equation}\n\\end{proposition}\n\n\\begin{proof}\nLet $\\bm{v}\\in \\rn$ be given. Assume that the event $\\mathscr{E}_1:=\\accolade{\\normii{\\QrandomTranspose\\bm{v}}\\geq \\alpha_Q\\normii{\\bm{v}}}$ occurs, which happens with a probability of at least $1-\\beta_Q$, and let $\\omega\\in\\mathscr{E}_1$ be arbitrary. Since $\\pollD$ is a PSS, it contains a direction $\\di_{\\star}^p=\\ubar{\\di}_{\\star}^p(\\omega)$ that makes an acute angle with the deterministic vector $\\Qrandom(\\omega)^\\top \\bm{v}\\in \\rp$. Formally thanks to~\\eqref{cosineDescentDkn}, this means that \n\\begin{equation}\\label{kdeq1}\n\\kappa(\\pollD)\\normii{\\Qrandom(\\omega)^\\top \\bm{v}}\\normii{\\di_{\\star}^p}\\leq \\scal{\\Qrandom(\\omega)^\\top \\bm{v}}{\\di_{\\star}^p}=\\scal{\\bm{v}}{\\Qrandom(\\omega)\\di_{\\star}^p}.\n\\end{equation}\nIt follows from~\\eqref{kdeq1} and the inequality $\\normii{\\Qrandom(\\omega)^\\top\\bm{v}}\\geq \\alpha_Q\\normii{\\bm{v}}$ that $\\alpha_Q\\kappa(\\pollD)\\normii{\\bm{v}}\\normii{\\di_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom(\\omega)\\di_{\\star}^p}$, which leads to $\\frac{\\alpha_Q}{\\Qmax}\\kappa(\\pollD)\\normii{\\bm{v}}\\normii{\\Qrandom(\\omega)\\di_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom(\\omega)\\di_{\\star}^p}$, given that $\\normii{\\Qrandom(\\omega)\\di_{\\star}^p}\\leq \\Qmax\\normii{\\di_{\\star}^p}$. We have therefore proved that $\\mathscr{E}_1\\subseteq\\mathscr{E}_2:=\\accolade{\\exists\\ubar{\\di}_{\\star}^p\\in\\pollD:\\frac{\\alpha_Q}{\\Qmax}\\kappa(\\pollD)\\normii{\\bm{v}}\\normii{\\Qrandom\\ubar{\\di}_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom\\ubar{\\di}_{\\star}^p}}$, whence $\\prob{\\mathscr{E}_2}\\geq \\prob{\\mathscr{E}_1}\\geq 1-\\beta_Q$, leading to~\\eqref{kdprob1}. \n\nIn the case of a random PSS $\\pollDrandom$, the same reasoning  applies with $\\pollD$ and $\\di_{\\star}^p$ replaced with $\\pollDrandom(\\omega)$ and $\\di_{\\star}^p(\\omega)$ (which are deterministic), respectively, thus leading to~\\eqref{kdprob1prime}, and the proof is completed.\n\\end{proof}\n\n\\begin{lemma}\\label{HaarLemma}\\cite[Section~2.1]{MeckesElizHaar2019}\nLet $\\Urandom:=(\\UrandomScal_{ij})_{1\\leq i,j\\leq n}\\in \\mathbb{O}(n)$ be \\emph{Haar}-distributed. Then, the entries of $\\Urandom$ are identically distributed. Moreover, $\\E{\\UrandomScal_{11}}=0$ and $\\E{\\UrandomScal_{11}^2}=\\frac{1}{n}$.\n\\end{lemma}\n\n\\begin{theorem}\\label{LeviTheorem}(\\cite[Theorem~5.1.4 and Exercise~5.1.12]{vershynin2018HDP})\nLet $F$ be an $L_F$-Lipschitz function on the unit sphere $\\mathbb{S}^{n-1}$, and let $\\xiRandomVec\\sim \\mathcal{U}(\\mathbb{S}^{n-1})$. Then there exists an absolute constant $\\eta>0$ such that for every $t\\geq 0$\n\\[\\prob{\\abs{F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}}\\geq t}\\leq 2\\exp{\\left(-\\frac{\\eta n t^2}{L_F^2}\\right)}. \\]\n\\end{theorem}\n\n\\begin{theorem}\\label{JLforHaarTheor}(Inspired by~\\cite[Lemma~6.1]{MeckesElizHaar2019})\nLet $\\varepsilon_Q, \\beta_Q\\in (0,1)$ and $\\Urandom:=(\\UrandomScal_{ij})_{1\\leq i,j\\leq n}\\in \\mathbb{O}(n)$ be a \\emph{Haar}-distributed random matrix. Let $\\Urandom_{p\\times n}\\in \\rpn$ be the matrix consisting of the first $p$ rows of $\\Urandom$, and define $\\QrandomTranspose:=\\sqrt{\\frac{n}{p}}\\Urandom_{p\\times n}$. There exists an absolute constant $\\eta>0$ such that if $p\\geq  \\frac{4}{\\eta}\\varepsilon_Q^{-2}\\log\\left(\\frac{2}{\\beta_Q}\\right)$, then\n\\begin{equation}\\label{JLTEquat}\n\\prob{(1-\\varepsilon_Q)\\normii{\\x}\\leq \\normii{\\QrandomTranspose\\x}\\leq(1+\\varepsilon_Q)\\normii{\\x}}\\geq 1-\\beta_Q\\quad\\mbox{for all}\\ \\x\\in \\rn.\n\\end{equation} \n\\end{theorem}\n\n\\begin{proof}\n{\\bbl See Appendix~\\ref{theor22}.}\n\\end{proof}\n\n\\begin{lemma}\\label{lemKappaD3}\nLet $\\Qrandom\\in\\rnp$ be given by Theorem~\\ref{JLforHaarTheor}, and let $\\pollD\\subset \\rp$ be a PSS. For any $\\bm{v}\\in \\rn$, the random vector $\\hat{\\di}^p(\\vb,\\qrandom)\\in\\argmax\\accolade{\\scal{\\vb}{\\Qrandom\\di^p}:\\di^p\\in\\pollD}$ satisfies\n%there exists a direction $\\di^p_{\\star}\\in\\pollD$ such that \n\\begin{equation}\\label{kdpr1}\n\\prob{\\alpha_Q\\kappa(\\pollD)\\normii{\\bm{v}}\\leq \\langle\\vb, \\Qrandom\\hat{\\di}^p(\\vb,\\qrandom)\\rangle}\\geq 1-\\beta_Q.\n\\end{equation}\nA similar result holds for any $\\bm{v}\\in \\rn$ with ${\\hat{\\di}^p}(\\vb,\\qrandom,\\drandom)\\in\\argmax\\accolade{\\scal{\\vb}{\\Qrandom\\ubar{\\di}^p}:\\ubar{\\di}^p\\in\\pollDrandom}$ in the case of a random PSS $\\pollDrandom\\subset\\rp$; that is,\n\\begin{equation}\\label{kdpr2}\n\\prob{\\alpha_Q\\kappa(\\pollDrandom)\\normii{\\bm{v}}\\leq \\langle\\vb, \\Qrandom{\\hat{\\di}^p}(\\vb,\\qrandom,\\drandom)\\rangle}\\geq 1-\\beta_Q.\n\\end{equation}\n\\end{lemma}\n\n\\begin{proof}\nBy construction, $\\Q=\\sqrt{\\frac{n}{p}}\\Ub_{\\ntp}$, where $\\ubar{\\Ub}_{\\ntp}$ represents the first $p$ columns of a Haar-distributed orthogonal matrix and $\\normii{\\di^p}=1$ for all $\\di^p\\in\\pollD$, whence $\\norme{\\Q}=\\sqrt{\\frac{n}{p}}=:\\Qmax =\\normii{\\Q\\di^p}$ since $\\normii{\\Ub_{\\ntp}\\di^p}=1$. Consequently, $\\accolade{\\exists\\ubar{\\di}_{\\star}^p\\in\\pollD:\\frac{\\alpha_Q}{\\Qmax}\\kappa(\\pollD)\\normii{\\bm{v}}\\normii{\\Qrandom\\ubar{\\di}_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom\\ubar{\\di}_{\\star}^p}}=\\accolade{\\exists\\ubar{\\di}_{\\star}^p\\in\\pollD:\\alpha_Q\\kappa(\\pollD)\\normii{\\bm{v}}\\leq \\scal{\\bm{v}}{\\Qrandom\\ubar{\\di}_{\\star}^p}}$, which is included in the event $\\accolade{\\alpha_Q\\kappa(\\pollD)\\normii{\\bm{v}}\\leq \\langle\\vb, \\Qrandom{\\hat{\\di}^p}(\\vb,\\qrandom)\\rangle}$, leading to~\\eqref{kdpr1} thanks to~\\eqref{kdprob1}. The result in~\\eqref{kdpr2} follows from~\\eqref{kdprob1prime} using similar arguments by replacing $\\pollD$ with~$\\pollDrandom$ and ${\\hat{\\di}^p}(\\vb,\\qrandom)$ with ${\\hat{\\di}^p}(\\vb,\\qrandom,\\drandom)$, and the proof is completed.\n\\end{proof}\n\n\\begin{lemma}\\cite[Theorem~2.7-$(\\ell)$]{bhattacharya2007basic}\\label{substLemma}\nLet $\\ubar{\\vb}$ and $\\Mrandom$ be random maps into $(\\mathcal{S}_1, \\mathscr{S}_1)$ and $(\\mathcal{S}_2, \\mathscr{S}_2)$, respectively. Let $\\mathcal{F}$ be a $\\sigma$-algebra and $\\psi$ be a measurable real-valued function on $\\left(\\mathcal{S}_1\\times \\mathcal{S}_2, \\mathscr{S}_1\\bigotimes\\mathscr{S}_2\\right)$. If $\\ubar{\\vb}$ is $\\mathcal{F}$-measurable, $\\sigma(\\Mrandom)$ and $\\mathcal{F}$ are independent, and $\\E{\\abs{\\psi(\\ubar{\\vb},\\Mrandom)}}<\\infty$, then $\\E{\\psi(\\ubar{\\vb},\\Mrandom)|\\mathcal{F}}=\\wp(\\ubar{\\vb})$ almost surely, where $\\wp(\\vb):=\\E{\\psi(\\vb,\\Mrandom)}$ for all $\\vb\\in\\mathcal{S}_1$.\n\\end{lemma}\n\n\\begin{proposition}\\label{propKappaD2}\nLet $\\mathcal{F}$ be a $\\sigma$-algebra independent of a random matrix $\\Qrandom\\in\\rnp$ given by Theorem~\\ref{JLforHaarTheor}, and let $\\ubar{\\vb}\\in \\rn$ be an $\\mathcal{F}$-measurable random vector. Let $\\pollD\\subset \\rp$ be a PSS, and consider the random vector defined by $\\hat{\\di}^p(\\ubar{\\vb},\\qrandom)\\in\\argmax\\accolade{\\scal{\\ubar{\\vb}}{\\Qrandom\\di^p}:\\di^p\\in\\pollD}$. Then \n%There exists a direction $\\di_{\\star}^p\\in\\pollD$ such that \n\\begin{equation}\\label{kdprob2}\n\\prob{\\left.\\alpha_Q\\kappa(\\pollD)\\normii{\\ubar{\\vb}} \\leq \\langle\\ubar{\\vb}, \\Qrandom\\hat{\\di}^p(\\ubar{\\vb},\\qrandom)\\rangle \\right|\\F}\\geq 1-\\beta_Q\\quad\\mbox{almost surely.}\n\\end{equation}\nA similar result holds for any $\\ubar{\\vb}\\in \\rn$ with $\\hat{\\di}^p(\\ubar{\\vb},\\qrandom,\\drandom)\\in\\argmax\\accolade{\\scal{\\ubar{\\vb}}{\\Qrandom\\ubar{\\di}^p}:\\ubar{\\di}^p\\in\\pollDrandom}$ in the case of a random PSS $\\pollDrandom\\subset\\rp$ with $\\Drandom$ independent of $\\mathcal{F}$, \n%and for a direction ${\\ubar{\\di}^p}_{\\star}\\in \\pollDrandom$; \nthat is,\n\\begin{equation}\\label{kdprob2prime}\n\\prob{\\left.\\alpha_Q\\kappa(\\pollDrandom)\\normii{\\ubar{\\vb}} \\leq \\scal{\\ubar{\\vb} }{\\Qrandom{\\hat{\\di}^p}(\\ubar{\\vb},\\qrandom,\\drandom)} \\right|\\F}\\geq 1-\\beta_Q\\quad\\mbox{almost surely.}\n\\end{equation}\n\\end{proposition}\n\n\\begin{proof}\nLet $\\psi$ defined on $(\\rn\\times\\rnp, \\mathscr{B}(\\rn)\\bigotimes\\mathscr{B}(\\rnp))$ by $\\psi(\\ubar{\\vb},\\Qrandom)=\\mathds{1}_{\\accolade{\\alpha_Q\\kappa(\\pollD)\\normii{\\ubar{\\vb}} \\leq \\scal{\\ubar{\\vb}}{\\Qrandom\\hat{\\di}^p(\\ubar{\\vb},\\qrandom)}}}$ be a real-valued measurable function. Since $\\E{\\abs{\\psi(\\ubar{\\vb},\\Qrandom)}}<\\infty$ trivially, it follows from Lemma~\\ref{substLemma} that $\\E{\\psi(\\ubar{\\vb},\\Qrandom)|\\mathcal{F}}=\\wp(\\ubar{\\vb})$ almost surely, where $\\wp(\\vb):=\\E{\\mathds{1}_{\\accolade{\\alpha_Q\\kappa(\\pollD)\\normii{\\vb} \\leq \\scal{\\vb}{\\qrandom\\hat{\\di}^p(\\vb,\\qrandom)}}}}=\\prob{\\alpha_Q\\kappa(\\pollD)\\normii{\\vb} \\leq \\langle\\vb,\\Qrandom\\hat{\\di}^p(\\vb,\\qrandom) \\rangle}$ for all $\\vb\\in\\rn$. It follows from~\\eqref{kdpr1} that $\\wp(\\vb)\\geq 1-\\beta_Q$ for all $\\vb\\in\\rn$ whence $\\wp(\\ubar{\\vb})\\geq 1-\\beta_Q$ almost surely. This means that $\\wp(\\ubar{\\vb}):=\\E{\\psi(\\ubar{\\vb},\\Qrandom)|\\mathcal{F}}=\\prob{\\left.\\alpha_Q\\kappa(\\pollD)\\normii{\\ubar{\\vb}} \\leq \\langle\\ubar{\\vb}, \\Qrandom\\hat{\\di}^p(\\ubar{\\vb},\\qrandom)\\rangle \\right|\\F}\\geq 1-\\beta_Q$ almost surely, proving~\\eqref{kdprob2}. To prove~\\eqref{kdprob2prime}, consider the measurable real-valued function $\\chi$ defined on $(\\rn\\times\\R^{p\\times (n+m)}, \\mathscr{B}(\\rn)\\bigotimes\\mathscr{B}(\\R^{p\\times (n+m)}))$ by $\\chi\\left(\\ubar{\\vb},\\left[\\Qrandom^{\\top}\\ \\Drandom\\right]\\right)=\\mathds{1}_{\\accolade{\\alpha_Q\\kappa(\\pollDrandom)\\normii{\\ubar{\\vb}} \\leq \\scal{\\ubar{\\vb}}{\\qrandom\\hat{\\di}^p(\\ubar{\\vb},\\qrandom,\\drandom)}}}$, satisfying $\\E{\\abs{\\chi\\left(\\ubar{\\vb},\\left[\\Qrandom^{\\top}\\ \\Drandom\\right]\\right)}}<\\infty$. $\\sigma\\left(\\left[\\Qrandom^{\\top}\\ \\Drandom\\right]\\right)$ is independent of~$\\F$ whence $\\E{\\left. \\chi\\left(\\ubar{\\vb},\\left[\\Qrandom^{\\top}\\ \\Drandom\\right]\\right) \\right|\\F}=\\tilde{\\wp}(\\ubar{\\vb})$, where $\\tilde{\\wp}(\\vb):=\\prob{\\alpha_Q\\kappa(\\pollDrandom)\\normii{{\\vb}} \\leq \\scal{{\\vb}}{\\Qrandom\\hat{\\di}^p({\\vb},\\qrandom,\\drandom)}}$, thanks to Lemma~\\ref{substLemma}. It follows from~\\eqref{kdpr2} that $\\tilde{\\wp}(\\vb)\\geq 1-\\beta_Q$ $\\forall\\vb\\in\\rn$, whence $\\tilde{\\wp}(\\ubar{\\vb}):=\\E{\\left. \\chi\\left(\\ubar{\\vb},\\left[\\Qrandom^{\\top}\\ \\Drandom\\right]\\right) \\right|\\F}\\geq 1-\\beta_Q$ almost surely, showing~\\eqref{kdprob2prime}, and the proof is completed.\n\\end{proof}\n\n\\begin{proposition}\\label{decreaseProp1}(Inspired by~\\cite[Proposition~1]{dzahini2020expected})\nLet $\\gamma>2$ be a constant, and let $\\fok$ and $\\fuk$ be $\\ef$-accurate estimates of $f(\\xk)$ and $f(\\xk+\\dk\\ub)$, respectively: \n\\begin{equation}\\label{decreaseEqu}\n\\mbox{If}\\quad \\fuk-\\fok\\leq -\\gamma\\ef\\dk^2,\\qquad\\mbox{then}\\qquad f(\\xk+\\dk\\ub)-f(\\xk)\\leq -(\\gamma-2)\\ef\\dk^2.\n\\end{equation}\n\\end{proposition}\n\n\\begin{proof}\nThe proof trivially follows that of~\\cite[Proposition~1]{dzahini2020expected} with minor changes.\n\\end{proof}\n\n\\begin{lemma}\\label{SDDSlemma1}\\cite[Lemma~1]{dzahini2020expected}\nUnder Assumption~\\ref{efkfAssump}, the following hold for all $k\\geq 0$ for the random process generated by Algorithm~\\ref{algoStoDARS}:\n\\begin{eqnarray*}\n\\E{\\ijkc\\abs{\\Fok-f(\\Xk)}|\\fqdalgebra}&\\leq& \\ef(1-\\beta_f)\\Dk^2\\quad\\mbox{almost surely, and}\\\\\n\\E{\\ijkc\\abs{\\Fuk-f(\\Xk+\\Dk\\urandom)}|\\fqdalgebra}&\\leq& \\ef(1-\\beta_f)\\Dk^2\\quad\\mbox{almost surely.}\n\\end{eqnarray*} \n\\end{lemma}\n\n\\begin{proof}\nThe proof is almost identical to that of~\\cite[Lemma~1]{dzahini2020expected}.\n\\end{proof}\n\n\\begin{theorem}\\label{zerothOrdTheor}\nLet Assumptions~\\ref{QkHaarAssumpt},~\\ref{efkfAssump} and~\\ref{boundedFassumpt} hold. \n%Let $\\gamma>2$ and $\\nu,\\tau\\in (0,1)$ satisfying $\\frac{\\nu}{1-\\nu}\\geq \\frac{2(\\tau^{-2}-1)}{\\gamma-2}$. Assume that Assumption~\\ref{efkfAssump} holds with $\\beta_f\\in (0,1)$ satisfying  $\\frac{\\beta_f}{1-\\beta_f}\\geq \\frac{4\\nu}{(1-\\nu)(1-\\tau^2)}$. \nDefine $\\varrho=\\frac{1}{2}\\beta_f(1-\\nu)(1-\\tau^2)$. Then, the random function $\\Phik:=\\frac{\\nu}{\\ef}\\left(f(\\Xk)-f_{\\min}\\right)+(1-\\nu)\\Dk^2$ satisfies \n\\begin{equation}\\label{PhikDynamics}\n\\E{\\Phikun-\\Phik|\\fqdalgebra}\\leq -\\varrho\\Dk^2 \\quad\\mbox{almost surely, for all}\\ k\\in \\N, \n\\end{equation}\nwhich implies that \n\\begin{equation}\\label{finiteSeries}\n\\sum_{k=0}^{\\infty}\\Dk^2<\\infty\\quad\\mbox{almost surely, whence}\\qquad \\underset{k\\to\\infty}{\\lim}\\Dk=0\\quad\\mbox{almost surely.}\n\\end{equation}\n\\end{theorem}\n\n\\begin{proof}\n{\\bbl See Appendix~\\ref{th41}.}\n\\end{proof}\n\n\\begin{theorem}\\cite[Theorem~2]{blanchet2016convergence}\\label{theorComplexity1}\nUnder Assumption~\\ref{assumptionRenewal}, \n$\\E{\\Tepsilon}\\leq \\frac{q}{2q-1}\\cdot\\frac{\\PhiZero}{\\varrho h\\left(\\depsilon\\right)}+1.$\n\\end{theorem}\n\nAs in~\\cite{blanchet2016convergence,dzahini2020expected,DzaWildSub2022,paquette2018stochastic}, the occurrence of the event $\\accolade{\\Tepsilon=k}$ is determined by observing $\\left(\\PhiZero,\\Dzero,\\WtildeZero\\right),\\dots,\\left(\\Phikmun,\\Dkmun,\\Wtildekmun\\right)$, which means that~$\\Tepsilon$ is a stopping time for the stochastic process generated by Algorithm~\\ref{algoStoDARS}, and hence with respect to the filtration $\\accoladekinN{\\mathcal{H}_k}:=\\accoladekinN{\\falgebra}$. Bounding $\\E{\\Tepsilon}$ by applying Theorem~\\ref{theorComplexity1} requires showing next that Assumption~\\ref{assumptionRenewal} is satisfied for Algorithm~\\ref{algoStoDARS}.\n\nRegarding Assumption~\\ref{assumptionRenewal} {\\it (i)} and~{\\it (iii)}, the arguments are basically those in~\\cite{dzahini2020expected} and~\\cite{DzaWildSub2022}, respectively. More precisely,~{\\it (i)} is trivially satisfied for $\\lambda:=-\\log(\\tau)$ while, on the other hand, it follows from~\\eqref{PhikDynamics} and the inclusion $\\falgebra\\subset\\fqdalgebra$ that almost surely\n\\begin{equation}\\label{tower1}\n\\E{\\Phikun-\\Phik|\\falgebra}=\\E{\\E{\\Phikun-\\Phik|\\fqdalgebra}|\\falgebra}\\leq \\E{-\\varrho\\Dk^2|\\falgebra}=-\\varrho\\Dk^2.\n\\end{equation}\nThis shows that~{\\it (iii)} is satisfied with the function $h:\\scalebox{0.75}{$\\vartheta$}\\mapsto\\scalebox{0.75}{$\\vartheta$}^2$ since~\\eqref{tower1} holds for any realization of the random variable $\\E{\\Phikun-\\Phik|\\falgebra}$, and hence on the event $\\accolade{\\Tepsilon>k}$ in particular.\n\n\n\\begin{lemma}\\label{AkLemma1}\nLet Assumption~\\ref{QkHaarAssumpt} hold, and consider ${\\ubar{\\hat{\\di}}^p_k}:=\\hat{\\di}^p_k({\\footnotesize{\\X}},\\qrandom,\\drandom)\\in\\argmax\\accolade{-\\nabla f(\\Xk)^{\\top}\\left(\\QkRandom\\diRandom\\right):\\diRandom\\in\\pollDrandom_k}$. Then the event $A_k$ defined in~\\eqref{AkEventDef} satisfies $\\prob{A_k|\\falgebra}=\\E{\\iak|\\falgebra}\\geq 1-\\beta_Q$ almost surely.\n\\end{lemma}\n\n\\begin{proof}\nBy construction, $\\nabla f(\\Xk)$ is $\\falgebra$-measurable while $\\QkRandom$ and $\\pollDrandom_k$ are independent of $\\falgebra$. The conditions of Proposition~\\ref{propKappaD2} are therefore satisfied, and the result straightforwardly follows from~\\eqref{kdprob2prime}.\n\\end{proof}\n\n\\begin{lemma}\\label{smallLemmaHaar}\nLet Assumptions~\\ref{QkHaarAssumpt},~\\ref{kapminAssump} and~\\ref{gradLipAssump} hold, and assume that $\\dk\\leq \\depsilon$. Let $\\fok$ and $\\fuk$ be $\\ef$-accurate estimates of $f(\\xk)$ and  $f(\\xk+\\dk\\ub)$, respectively. Assume that the event $A_k$ occurs. If $\\normii{\\nabla f(\\xk)}>\\epsilon$, then there exists $\\ub\\in \\mathbb{U}_k^n:=\\{\\Ukp\\di^p:\\di^p\\in \\pollD_k\\}$ such that $\\fuk-\\fok\\leq -\\gamma\\ef\\dk^2$; that is, the iteration $k$ is successful. \n\\end{lemma}\n\n\\begin{proof}\n{\\bbl See Appendix~\\ref{lem52}.}\n\\end{proof}\n\n\\begin{lemma}\\label{dynamicsLemma}\n{\\bl Let Assumption~\\ref{QkHaarAssumpt} hold with $0<\\beta_Q<1-\\frac{1}{2\\beta_f}$ and let Assumptions~\\ref{QkHaarAssumpt},~\\ref{efkfAssump},~\\ref{boundedFassumpt},~\\ref{kapminAssump}, and~\\ref{gradLipAssump} hold.}\n%and all assumptions made in Theorem~\\ref{zerothOrdTheor} hold. \nThen Assumption~\\ref{assumptionRenewal}-$(ii)$ is satisfied for $\\Wtildek:=2\\left(\\iak\\ijk-\\frac{1}{2}\\right)$, with $\\lambda=-\\log(\\tau)$ and some fixed $q\\in (\\tilde{\\beta}, 1)$, where $\\tilde{\\beta}=\\beta_f(1-\\beta_Q)$.\n\\end{lemma}\n\n\\begin{proof}\nThe result is proved by suitably adapting the proof of~\\cite[Lemma~7]{blanchet2016convergence}, as was also done in~\\cite[Lemma~3]{dzahini2020expected}, \\cite[Lemma~5.3]{DzaWildSub2022} and~\\cite[Lemma~4.8]{paquette2018stochastic}. First, observe that~\\eqref{dynamicsRenewal} trivially holds when $\\itksup=0$. It is shown next that when $\\itksup=1$ (which is assumed in the remainder of the proof), then \n\\[\\Dkun\\geq \\min\\accolade{\\depsilon, \\min\\accolade{\\dmax,\\tau^{-1}\\Dk}\\iak\\ijk+\\tau\\Dk\\left(1-\\iak\\ijk\\right)}.\\] \nRecall that $\\dk=\\tau^{i_k}\\depsilon$ for some integer $i_k$ so that if $\\dk>\\depsilon$, then $\\dk\\geq \\tau^{-1}\\depsilon$, whence $\\dkun\\geq \\tau\\dk\\geq \\depsilon$. Next, assume that $\\dk\\leq \\depsilon$, and observe that $\\normii{\\nabla f(\\xk)}>\\eps$ since $\\Tepsilon>k$. If $\\iak\\ijk=1$, it follows from Lemma~\\ref{smallLemmaHaar} that the iteration is successful. Consequently, $\\dkun=\\min\\accolade{\\tau^{-1}\\dk,\\dmax}$. But if $\\iak\\ijk=0$, then it always holds that $\\dkun\\geq \\tau\\dk$. The proof is completed by observing that $\\prob{\\iak\\ijk=1|\\falgebra}=q$ for some $q\\geq\\tilde{\\beta}$.\n\\end{proof}\n\n\\begin{proof}\nThe proof, which trivially follows from Theorem~\\ref{theorComplexity1}, is identical to that of~\\cite[Theorem~5.4]{DzaWildSub2022}.\n\\end{proof}\n\n\\begin{proof}\nWithout any loss of generality, it is sufficient to prove the result for $\\x\\in \\mathbb{S}^{n-1}$, which is assumed throughout. Let $\\bm{e_1}$ be the first standard basis vector of $\\rn$. \n%By the translation invariance of Haar measure, \nFor any $\\x\\in \\mathbb{S}^{n-1}$ arbitrary, $\\Urandom_{p\\times n}\\x \\equald \\Urandom_{p\\times n} \\bm{e_1}=(\\UrandomScal_{11},\\UrandomScal_{21},\\dots, \\UrandomScal_{p1})^{\\top}$,\n%. On the other hand, it follows from the definition of Haar measure that the first column of $\\Urandom$ satisfies \nwith $(\\UrandomScal_{11}, \\UrandomScal_{21},\\dots, \\UrandomScal_{n1})^{\\top}\\sim \\mathcal{U}(\\mathbb{S}^{n-1})$, hence $\\Urandom_{p\\times n}\\x \\equald \\left(\\xiRandom_1,\\xiRandom_2,\\dots, \\xiRandom_p\\right)^{\\top}$, where $\\xiRandomVec:=\\left(\\xiRandom_1,\\xiRandom_2,\\dots, \\xiRandom_n\\right)^{\\top}\\sim \\mathcal{U}(\\mathbb{S}^{n-1})$. \n%Consider the function $F:\\mathbb{S}^{n-1}\\to \\R$ defined by \nLet \\scalebox{0.88}{$F(x_1,x_2,\\dots,x_n):=\\sqrt{\\frac{n}{p}}\\normii{(x_1,x_2,\\dots,x_p)}$}, so that \\scalebox{0.88}{$F\\left(\\xiRandomVec\\right)=\\sqrt{\\frac{n}{p}}\\normii{\\left(\\xiRandom_1,\\xiRandom_2,\\dots, \\xiRandom_p\\right)}\\equald \\sqrt{\\frac{n}{p}}\\normii{\\Urandom_{p\\times n}\\x}=\\normii{\\QrandomTranspose\\x}$}. \nThen, for all $\\x,\\bm{y}\\in \\mathbb{S}^{n-1}$, $\\abs{F(\\x)-F(\\bm{y})}\\leq \\sqrt{\\frac{n}{p}}\\normii{\\x-\\bm{y}}$,\n% \\begin{equation*}\n% \\begin{split}\n% %\\abs{F(\\x)-F(\\bm{y})}&=\\sqrt{\\frac{n}{p}}\\abs{\\normii{(x_1,x_2,\\dots,x_p)}-\\normii{(y_1,y_2,\\dots,y_p)}}\\\\\n% \\abs{F(\\x)-F(\\bm{y})}&\\leq \\sqrt{\\frac{n}{p}}\\normii{(x_1-y_1),(x_2-y_2),\\dots,(x_p-y_p)}\\leq \\sqrt{\\frac{n}{p}}\\normii{\\x-\\bm{y}},\n% \\end{split}\n% \\end{equation*}\nwhich shows that $F$ is $\\sqrt{\\frac{n}{p}}$-Lipschitz on $\\mathbb{S}^{n-1}$. Thus, it follows from Theorem~\\ref{LeviTheorem} that there exists $\\eta>0$ such that for all $\\varepsilon_Q\\in (0,1)$, \n{\\footnotesize{\n\\begin{equation}\\label{LeviEq1}\n\\prob{\\abs{F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}}\\geq \\frac{\\varepsilon_Q}{2}}\\leq 2\\exp{\\left(-\\frac{\\eta n (\\varepsilon_Q/2)^2}{(\\sqrt{n/p})^2}\\right)}=2e^{-\\frac{\\eta p\\varepsilon_Q^2}{4}}.\n\\end{equation}\n}}\n%On the other hand, it follows from Lemma~\\ref{HaarLemma} that $\\E{(F\\left(\\xiRandomVec\\right))^2}=1$, which implies $\\var{F\\left(\\xiRandomVec\\right)}+(\\E{F\\left(\\xiRandomVec\\right)})^2=1$. \nMoreover, it follows from the integral identity formula that \n{\\footnotesize{\n\\begin{equation}\\label{levEq2}\n\\var{F\\left(\\xiRandomVec\\right)}=\\E{\\left(F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}\\right)^2}=\\int_0^{+\\infty}\\prob{\\abs{F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}}^2\\geq t} dt\\leq\\int_0^{+\\infty} 2e^{-\\eta p t} dt=\\frac{2}{\\eta p},\n\\end{equation}\n}}\nwhere the inequality follows from Theorem~\\ref{LeviTheorem}. \n%It follows from~\\eqref{levEq2} and the equality $\\E{(F\\left(\\xiRandomVec\\right))^2}=\\var{F\\left(\\xiRandomVec\\right)}+\\left(\\E{F\\left(\\xiRandomVec\\right)}\\right)^2=1$ (thanks to Lemma~\\ref{HaarLemma}) that $\\sqrt{1-\\frac{2}{\\eta p}}\\leq \\E{F\\left(\\xiRandomVec\\right)}\\leq 1$. \nFrom~\\eqref{levEq2} and Lemma~\\ref{HaarLemma}, we obtain $\\sqrt{1-\\frac{2}{\\eta p}}\\leq \\E{F\\left(\\xiRandomVec\\right)}\\leq 1$.\nAssume that $p$ is chosen so that $1-\\frac{\\varepsilon_Q}{2}\\leq\\sqrt{1-\\frac{2}{\\eta p}}$ and $2e^{-\\frac{\\eta p\\varepsilon_Q^2}{4}}\\leq \\beta_Q$. Then $\\abs{F\\left(\\xiRandomVec\\right)-1}<\\varepsilon_Q$ since $1-\\frac{\\varepsilon_Q}{2}\\leq \\E{F\\left(\\xiRandomVec\\right)}\\leq 1$.\n%Since $1-\\frac{\\varepsilon_Q}{2}\\leq \\E{F\\left(\\xiRandomVec\\right)}\\leq 1$, then the inequality $\\abs{F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}}<\\frac{\\varepsilon_Q}{2}$ implies that $\\abs{F\\left(\\xiRandomVec\\right)-1}<\\varepsilon_Q$. \nConsequently, for all $\\x\\in \\mathbb{S}^{n-1}$, it follows from~\\eqref{LeviEq1} that\n\\[\\prob{\\abs{\\normii{\\QrandomTranspose\\x}-1}\\geq \\varepsilon_Q}=\\prob{\\abs{F\\left(\\xiRandomVec\\right)-1}\\geq \\varepsilon_Q}\\leq\\prob{\\abs{F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}}\\geq\\frac{\\varepsilon_Q}{2}}\\leq2e^{-\\frac{\\eta p\\varepsilon_Q^2}{4}}\\leq \\beta_Q,\\]\nwhere the equality is due to the equality in distribution $F\\left(\\xiRandomVec\\right)\\equald \\normii{\\QrandomTranspose\\x}$ as explained above. Equivalently, it holds that $\\prob{1-\\varepsilon_Q\\leq \\normii{\\QrandomTranspose\\x}\\leq 1+\\varepsilon_Q}\\geq 1-\\beta_Q$ for all $\\x\\in \\mathbb{S}^{n-1}$. It remains to prove that choosing $p\\geq \\max\\accolade{\\frac{8}{3\\eta}\\varepsilon_Q^{-2}, \\frac{4}{\\eta}\\varepsilon_Q^{-2}\\log\\left(\\frac{2}{\\beta_Q}\\right)}=\\frac{4}{\\eta}\\varepsilon_Q^{-2}\\log\\left(\\frac{2}{\\beta_Q}\\right)$ ensures $1-\\frac{\\varepsilon_Q}{2}\\leq\\sqrt{1-\\frac{2}{\\eta p}}$ and $2e^{-\\frac{\\eta p\\varepsilon_Q^2}{4}}\\leq \\beta_Q$ as assumed above, where the latter equality trivially holds for any $\\varepsilon_Q,\\beta_Q\\in (0,1)$. We first observe that $2e^{-\\frac{\\eta p\\varepsilon_Q^2}{4}}\\leq \\beta_Q\\iff p\\geq \\frac{4}{\\eta}\\varepsilon_Q^{-2}\\log\\left(\\frac{2}{\\beta_Q}\\right)$. On the other hand, $\\left(1-\\frac{\\varepsilon_Q}{2}\\right)^2\\leq 1-\\frac{2}{\\eta p}$ if $p\\geq \\frac{2}{\\eta\\left(\\varepsilon_Q-\\frac{\\varepsilon_Q^2}{4}\\right)}$.\n%, which completes the proof. \nSince $\\varepsilon_Q\\in (0,1)$, then $\\varepsilon_Q-\\frac{\\varepsilon_Q^2}{4}>\\frac{3}{4}\\varepsilon_Q^2$, which implies that $\\frac{8}{3\\eta}\\varepsilon_Q^{-2}\\geq \\frac{2}{\\eta\\left(\\varepsilon_Q-\\frac{\\varepsilon_Q^2}{4}\\right)}$. Hence,  choosing $p\\geq \\frac{8}{3\\eta}\\varepsilon_Q^{-2}$ ensures $p\\geq \\frac{2}{\\eta\\left(\\varepsilon_Q-\\frac{\\varepsilon_Q^2}{4}\\right)}$.\n%, and the proof is completed.\n\\end{proof}\n\n\\begin{proof}\nThe proof is similar to that of~\\cite[Theorem~2]{dzahini2020expected}, using ideas derived in~\\cite{blanchet2016convergence,chen2018stochastic,DzaWildSub2022,paquette2018stochastic}. It is presented here for the sake of clarity and self-containedness. The overall goal is to prove~\\eqref{PhikDynamics} since taking the expectation on both its sides leads to~\\eqref{finiteSeries} as detailed in~\\cite[Theorem~3]{dzahini2020expected}, given that $\\Phik>0$ for all~$k\\in\\N$. The proof considers two cases, {\\it good estimates} and {\\it bad estimates}, each of which is broken into whether the iteration is successful or not. Throughout, \n\\[\\mathfrak{S}_k:=\\accolade{\\mbox{The}\\ k\\mbox{th iteration is successful}}.\\]\n{\\bf Case~1:} {\\it (Good estimates, $\\ijk=1$).} The overall goal is to prove that almost surely, \n\\begin{equation}\\label{treizeEq}\n\\E{\\ijk\\left(\\Phikun-\\Phik\\right)|\\fqdalgebra}\\leq -\\beta_f(1-\\nu)(1-\\tau^2)\\Dk^2.\n\\end{equation}\n{{Subcase~$1i$}:} {\\it (Successful iteration, $\\isk=1$).} A decrease occurs in~$f$ according to~\\eqref{decreaseEqu}, and $\\Dk$ is updated according to~$\\Dkun=\\min\\accolade{\\tau^{-1}\\Dk,\\dmax}$, whence $\\ijk\\isk\\frac{\\nu}{\\ef}(f(\\Xkun)-f(\\Xk))\\leq -\\ijk\\isk\\nu(\\gamma-2)\\Dk^2$ and $\\ijk\\isk(1-\\nu)(\\Dkun^2-\\Dk^2)\\leq \\ijk\\isk(1-\\nu)(\\tau^{-2}-1)\\Dk^2$, leading to \n\\begin{equation}\\label{dixseptEq}\n\\ijk\\isk\\left(\\Phikun-\\Phik\\right)\\leq\\ijk\\isk\\left(-\\nu(\\gamma-2)+(1-\\nu)(\\tau^{-2}-1)\\right)\\Dk^2\\leq -\\ijk\\isk(1-\\nu)(1-\\tau^2)\\Dk^2,\n\\end{equation}\nwhere the last inequality follows from the above choice of~$\\nu$ along with the fact that $1-\\tau^2<\\tau^{-2}-1$. \\\\\n{{Subcase~$1ii$}:} {\\it (Unsuccessful iteration, $\\iskc=1$).} The parameter~$\\Dk$ is decreased according to $\\Dkun=\\tau\\Dk$, and $f(\\Xkun)=f(\\Xk)$, whence\n\\begin{equation}\\label{dixhuitEq}\n\\ijk\\iskc\\left(\\Phikun-\\Phik\\right)=-\\ijk\\iskc(1-\\nu)(1-\\tau^2)\\Dk^2.\n\\end{equation}\nEquations~\\eqref{dixseptEq} and~\\eqref{dixhuitEq} lead to\n\\begin{equation}\\label{vingtEq}\n\\ijk\\left(\\Phikun-\\Phik\\right)\\leq -\\ijk(1-\\nu)(1-\\tau^2)\\Dk^2.\n\\end{equation}\nTaking the conditional expectation with respect to $\\fqdalgebra$ on both sides of~\\eqref{vingtEq} and using Assumption~\\ref{efkfAssump}-{\\it (i)} lead to~\\eqref{treizeEq}.\n{\\bf Case~2:} {\\it (Bad estimates, $\\ijkc=1$).} Because of bad estimates, an iterate leading to an increase in~$f$ and~$\\Dk$, and hence in~$\\Phik$, can be accepted by Algorithm~\\ref{algoStoDARS}. Such an increase is controlled by using Assumption~\\ref{efkfAssump}-{\\it (ii)}. The overall goal is to show that almost surely,\n\\begin{equation}\\label{vingtEtUnEq}\n\\E{\\ijkc\\left(\\Phikun-\\Phik\\right)|\\fqdalgebra}\\leq 2\\nu(1-\\beta_f)\\Dk^2.\n\\end{equation}\n{{Subcase~$2i$}:} {\\it (Successful iteration, $\\isk=1$).} Since $\\Fuk-\\Fok\\leq-\\gamma\\ef\\Dk^2$, then \n\\[\\ijkc\\isk\\frac{\\nu}{\\ef}(f(\\Xkun)-f(\\Xk))\\leq\\ijkc\\isk\\nu\\left(-\\gamma\\Dk^2+\\frac{1}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right)\\right).\\]\nRecalling that $\\Dkun=\\min\\accolade{\\tau^{-1}\\Dk,\\dmax}$, and by noticing that $-\\nu\\gamma+(1-\\nu)(\\tau^{-2}-1)\\leq 0$ due to the above choice of $\\nu$, it holds that\n% \\begin{equation}\\label{vingtcinqEq}\n% \\begin{split}\n% \\ijkc\\isk\\left(\\Phikun-\\Phik\\right)&\\leq \\ijkc\\isk\\left(\\left(-\\nu\\gamma+(1-\\nu)(\\tau^{-2}-1) \\right)\\Dk^2+\\frac{\\nu}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right) \\right)\\\\\n% &\\leq \\ijkc\\isk\\left(\\frac{\\nu}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right) \\right)\n% \\end{split}\n% \\end{equation}\n\\begin{equation}\\label{vingtcinqEq}\n\\ijkc\\isk\\left(\\Phikun-\\Phik\\right)\\leq \\ijkc\\isk\\frac{\\nu}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right).\n\\end{equation}\n{{Subcase~$2ii$}:} {\\it (Unsuccessful iteration, $\\iskc=1$).} The upper bound on the change in~$\\Phik$ follows from~\\eqref{dixhuitEq} by replacing~$\\ijk$ with~$\\ijkc$. Thus, $\\ijkc\\iskc\\left(\\Phikun-\\Phik\\right)\\leq \\ijkc\\iskc\\frac{\\nu}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right)$, which, combined with~\\eqref{vingtcinqEq}, yields\n\\begin{equation}\\label{vingtseptEq}\n\\ijkc\\left(\\Phikun-\\Phik\\right)\\leq \\ijkc\\frac{\\nu}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right).\n\\end{equation}\nTaking the conditional expectation with respect to $\\fqdalgebra$ on both sides of~\\eqref{vingtseptEq} and applying Lemma~\\ref{SDDSlemma1} lead to~\\eqref{vingtEtUnEq}. Then, combining~\\eqref{treizeEq} and~\\eqref{vingtEtUnEq} yields\n\\[\\E{\\left(\\Phikun-\\Phik\\right)|\\fqdalgebra}\\leq (-\\beta_f(1-\\nu)(1-\\tau^2)+2\\nu(1-\\beta_f))\\Dk^2\\leq -\\varrho\\Dk^2, \\]\nwhere the last inequality follows from the above choice of $\\beta_f$, and the proof is completed.\n\\end{proof}\n\n\\begin{proof}\nThe proof is inspired by that of~\\cite[Lemma~2]{dzahini2020expected} using elements derived in~\\cite{KoLeTo03a}. Suppose that $\\dk\\leq \\depsilon$, that the event $A_k$ occurs, and that the estimates $\\fok$ and $\\fuk$ are $\\ef$-accurate. Then, assume by way of contradiction that $\\fuk-\\fok> -\\gamma\\ef\\dk^2$ for all $\\ub\\in \\mathbb{U}_k^n$ of the form $\\ub:=\\Ukp\\di^p$, with $\\di^p\\in \\pollD_k$. Since $A_k$ occurs, then the latter inequality holds in particular for $\\ub=\\Ukp\\hat{\\di}^p_k$, with $\\hat{\\di}^p_k\\in\\argmax\\accolade{-\\nabla f(\\xk)^{\\top}\\left(\\Qk\\di^p\\right):\\di^p\\in\\pollD_k}$ satisfying \n\\begin{equation}\\label{AkRealization}\n\\alpha_Q\\kappa(\\pollD_k)\\normii{\\nabla f(\\xk)}\\leq -\\sqrt{\\frac{n}{p}}\\nabla f(\\xk)^{\\top}\\left(\\Ukp{\\hat{\\di}^p_k}\\right).\n\\end{equation}\nIt follows from the $\\ef$-accuracy assumption, along with the equality \n\\[f(\\xk+\\dk\\ub)-f(\\xk)=f(\\xk+\\dk\\ub)-\\fuk+(\\fuk-\\fok)+\\fok-f(\\xk),\\]\nthat \n\\begin{equation}\\label{eq39Acc}\nf\\left(\\xk+\\dk\\Ukp\\hat{\\di}^p_k\\right)-f(\\xk)+(\\gamma+2)\\ef\\dk^2\\geq 0.\n\\end{equation}\nIt follows from the mean value theorem and~\\eqref{eq39Acc} that there exists $\\mu_k\\in[0,1]$ such that \n\\begin{equation}\\label{eq40MeanVal}\n0\\leq \\dk\\nabla f\\left(\\xk+\\mu_k\\dk\\Ukp\\hat{\\di}^p_k\\right)^{\\top}\\left(\\Ukp{\\hat{\\di}^p_k}\\right)+(\\gamma+2)\\ef\\dk^2.\n\\end{equation}\nMultiplying both sides of~\\eqref{eq40MeanVal} by $\\frac{1}{\\dk}\\sqrt{\\frac{n}{p}}$ and subtracting $\\sqrt{\\frac{n}{p}}\\nabla f(\\xk)^{\\top}\\left(\\Ukp{\\hat{\\di}^p_k}\\right)$ yield\n\\begin{equation}\\label{eq41yield}\n-\\sqrt{\\frac{n}{p}}\\nabla f(\\xk)^{\\top}\\left(\\Ukp{\\hat{\\di}^p_k}\\right)\\leq\\sqrt{\\frac{n}{p}}\\left(\\nabla f\\left(\\xk+\\mu_k\\dk\\Ukp\\hat{\\di}^p_k\\right)- \\nabla f(\\xk)\\right)^{\\top}\\left(\\Ukp{\\hat{\\di}^p_k}\\right)+(\\gamma+2)\\ef\\dk,\n\\end{equation}\nwhich, combined with~\\eqref{AkRealization}, leads to\n\\begin{equation}\\label{eq42Norm}\n\\begin{split}\n\\alpha_Q\\kappa(\\pollD_k)\\normii{\\nabla f(\\xk)}&\\leq L_g\\sqrt{\\frac{n}{p}}\\normii{\\mu_k\\dk\\Ukp{\\hat{\\di}^p_k}}\\normii{\\Ukp{\\hat{\\di}^p_k}}+(\\gamma+2)\\ef\\dk \\leq \\left(L_g\\sqrt{\\frac{n}{p}}+(\\gamma+2)\\ef\\right)\\depsilon,%\\\\\n%&\\leq L_g\\sqrt{\\frac{n}{p}}\\dk+(\\gamma+2)\\ef\\dk\n\\end{split}\n\\end{equation}\nwhere the last inequality follows from the $L_g$-Lipschitz continuity of $\\nabla f$, the fact that $\\normii{\\Ukp{\\hat{\\di}^p_k}}=1$ and $\\dk\\leq\\depsilon$. It follows from~\\eqref{eq42Norm} that \n\\[\\normii{\\nabla f(\\xk)}\\leq \\alpha_Q^{-1}\\kappa_{\\min}^{-1}\\left(L_g\\sqrt{\\frac{n}{p}}+(\\gamma+2)\\ef\\right)\\times \\frac{\\epsilon}{\\xi}< \\epsilon,\\]\nwhere the last inequality follows from~\\eqref{depsEq}, and the proof is completed.\n\\end{proof}","units":[{"start":63812,"end":64870,"start_line":949,"end_line":959,"raw_tex":"\\begin{proposition}\\label{propKappaD1}\nAssume that $\\Qrandom\\in\\rnp$ satisfies Assumption~\\ref{JltAss1}, and let $\\pollD\\subset \\rp$ be a PSS. For any $\\bm{v}\\in \\rn$, \n%there exists a direction $\\di_{\\star}\\in\\pollD$ such that \n\\begin{equation}\\label{kdprob1}\n\\prob{\\accolade{\\exists\\ubar{\\di}_{\\star}^p\\in\\pollD:\\frac{\\alpha_Q}{\\Qmax}\\kappa(\\pollD)\\normii{\\bm{v}}\\normii{\\Qrandom\\ubar{\\di}_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom\\ubar{\\di}_{\\star}^p}}}\\geq 1-\\beta_Q.\n\\end{equation}\nIn other words, with high probability, there always exists a random direction $\\Qrandom\\ubar{\\di}_{\\star}^p\\in\\accolade{\\Qrandom\\di^p:\\di^p\\in\\pollD}\\subset\\rn$ that makes an acute angle with $\\bm{v}$. The same result holds in the case of a random PSS $\\pollDrandom\\subset\\rp$; that is,\n\\begin{equation}\\label{kdprob1prime}\n\\prob{\\accolade{\\exists\\ubar{\\di}_{\\star}^p\\in\\pollDrandom:\\frac{\\alpha_Q}{\\Qmax}\\kappa(\\pollDrandom)\\normii{\\bm{v}}\\normii{\\Qrandom\\ubar{\\di}_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom\\ubar{\\di}_{\\star}^p}}}\\geq 1-\\beta_Q.\n\\end{equation}\n\\end{proposition}","sha256_utf8":"54cbd1b62b8b2e0d5ee9cd6322dfd6ae57b2fa2416fd3efbed82b309e4ecd041"},{"start":64872,"end":66730,"start_line":961,"end_line":969,"raw_tex":"\\begin{proof}\nLet $\\bm{v}\\in \\rn$ be given. Assume that the event $\\mathscr{E}_1:=\\accolade{\\normii{\\QrandomTranspose\\bm{v}}\\geq \\alpha_Q\\normii{\\bm{v}}}$ occurs, which happens with a probability of at least $1-\\beta_Q$, and let $\\omega\\in\\mathscr{E}_1$ be arbitrary. Since $\\pollD$ is a PSS, it contains a direction $\\di_{\\star}^p=\\ubar{\\di}_{\\star}^p(\\omega)$ that makes an acute angle with the deterministic vector $\\Qrandom(\\omega)^\\top \\bm{v}\\in \\rp$. Formally thanks to~\\eqref{cosineDescentDkn}, this means that \n\\begin{equation}\\label{kdeq1}\n\\kappa(\\pollD)\\normii{\\Qrandom(\\omega)^\\top \\bm{v}}\\normii{\\di_{\\star}^p}\\leq \\scal{\\Qrandom(\\omega)^\\top \\bm{v}}{\\di_{\\star}^p}=\\scal{\\bm{v}}{\\Qrandom(\\omega)\\di_{\\star}^p}.\n\\end{equation}\nIt follows from~\\eqref{kdeq1} and the inequality $\\normii{\\Qrandom(\\omega)^\\top\\bm{v}}\\geq \\alpha_Q\\normii{\\bm{v}}$ that $\\alpha_Q\\kappa(\\pollD)\\normii{\\bm{v}}\\normii{\\di_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom(\\omega)\\di_{\\star}^p}$, which leads to $\\frac{\\alpha_Q}{\\Qmax}\\kappa(\\pollD)\\normii{\\bm{v}}\\normii{\\Qrandom(\\omega)\\di_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom(\\omega)\\di_{\\star}^p}$, given that $\\normii{\\Qrandom(\\omega)\\di_{\\star}^p}\\leq \\Qmax\\normii{\\di_{\\star}^p}$. We have therefore proved that $\\mathscr{E}_1\\subseteq\\mathscr{E}_2:=\\accolade{\\exists\\ubar{\\di}_{\\star}^p\\in\\pollD:\\frac{\\alpha_Q}{\\Qmax}\\kappa(\\pollD)\\normii{\\bm{v}}\\normii{\\Qrandom\\ubar{\\di}_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom\\ubar{\\di}_{\\star}^p}}$, whence $\\prob{\\mathscr{E}_2}\\geq \\prob{\\mathscr{E}_1}\\geq 1-\\beta_Q$, leading to~\\eqref{kdprob1}. \n\nIn the case of a random PSS $\\pollDrandom$, the same reasoning  applies with $\\pollD$ and $\\di_{\\star}^p$ replaced with $\\pollDrandom(\\omega)$ and $\\di_{\\star}^p(\\omega)$ (which are deterministic), respectively, thus leading to~\\eqref{kdprob1prime}, and the proof is completed.\n\\end{proof}","sha256_utf8":"6a8b5b5fc58948b4658b22d00d5833d793a9b4238ef05285130cd7392213fc11"},{"start":69950,"end":70269,"start_line":983,"end_line":985,"raw_tex":"\\begin{lemma}\\label{HaarLemma}\\cite[Section~2.1]{MeckesElizHaar2019}\nLet $\\Urandom:=(\\UrandomScal_{ij})_{1\\leq i,j\\leq n}\\in \\mathbb{O}(n)$ be \\emph{Haar}-distributed. Then, the entries of $\\Urandom$ are identically distributed. Moreover, $\\E{\\UrandomScal_{11}}=0$ and $\\E{\\UrandomScal_{11}^2}=\\frac{1}{n}$.\n\\end{lemma}","sha256_utf8":"d443dc1dca0129ea29031271f66fb3f1c3e8670ba041c1413ba20f760f60760a"},{"start":70731,"end":71186,"start_line":989,"end_line":992,"raw_tex":"\\begin{theorem}\\label{LeviTheorem}(\\cite[Theorem~5.1.4 and Exercise~5.1.12]{vershynin2018HDP})\nLet $F$ be an $L_F$-Lipschitz function on the unit sphere $\\mathbb{S}^{n-1}$, and let $\\xiRandomVec\\sim \\mathcal{U}(\\mathbb{S}^{n-1})$. Then there exists an absolute constant $\\eta>0$ such that for every $t\\geq 0$\n\\[\\prob{\\abs{F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}}\\geq t}\\leq 2\\exp{\\left(-\\frac{\\eta n t^2}{L_F^2}\\right)}. \\]\n\\end{theorem}","sha256_utf8":"49a5f5a30091bef61d4f6dffb7203765faa0c135516f3e4fef13210434698d36"},{"start":73159,"end":73915,"start_line":1003,"end_line":1008,"raw_tex":"\\begin{theorem}\\label{JLforHaarTheor}(Inspired by~\\cite[Lemma~6.1]{MeckesElizHaar2019})\nLet $\\varepsilon_Q, \\beta_Q\\in (0,1)$ and $\\Urandom:=(\\UrandomScal_{ij})_{1\\leq i,j\\leq n}\\in \\mathbb{O}(n)$ be a \\emph{Haar}-distributed random matrix. Let $\\Urandom_{p\\times n}\\in \\rpn$ be the matrix consisting of the first $p$ rows of $\\Urandom$, and define $\\QrandomTranspose:=\\sqrt{\\frac{n}{p}}\\Urandom_{p\\times n}$. There exists an absolute constant $\\eta>0$ such that if $p\\geq  \\frac{4}{\\eta}\\varepsilon_Q^{-2}\\log\\left(\\frac{2}{\\beta_Q}\\right)$, then\n\\begin{equation}\\label{JLTEquat}\n\\prob{(1-\\varepsilon_Q)\\normii{\\x}\\leq \\normii{\\QrandomTranspose\\x}\\leq(1+\\varepsilon_Q)\\normii{\\x}}\\geq 1-\\beta_Q\\quad\\mbox{for all}\\ \\x\\in \\rn.\n\\end{equation} \n\\end{theorem}","sha256_utf8":"de3aba5dd48bdca96662316cfa5d8f0f56a52262bee582e611c4efef58388cb5"},{"start":73916,"end":73976,"start_line":1009,"end_line":1011,"raw_tex":"\\begin{proof}\n{\\bbl See Appendix~\\ref{theor22}.}\n\\end{proof}","sha256_utf8":"8538626f0bbc3bbea976bcc748e6d429255bad3f1531346294fcbbee65185536"},{"start":75955,"end":76880,"start_line":1019,"end_line":1029,"raw_tex":"\\begin{lemma}\\label{lemKappaD3}\nLet $\\Qrandom\\in\\rnp$ be given by Theorem~\\ref{JLforHaarTheor}, and let $\\pollD\\subset \\rp$ be a PSS. For any $\\bm{v}\\in \\rn$, the random vector $\\hat{\\di}^p(\\vb,\\qrandom)\\in\\argmax\\accolade{\\scal{\\vb}{\\Qrandom\\di^p}:\\di^p\\in\\pollD}$ satisfies\n%there exists a direction $\\di^p_{\\star}\\in\\pollD$ such that \n\\begin{equation}\\label{kdpr1}\n\\prob{\\alpha_Q\\kappa(\\pollD)\\normii{\\bm{v}}\\leq \\langle\\vb, \\Qrandom\\hat{\\di}^p(\\vb,\\qrandom)\\rangle}\\geq 1-\\beta_Q.\n\\end{equation}\nA similar result holds for any $\\bm{v}\\in \\rn$ with ${\\hat{\\di}^p}(\\vb,\\qrandom,\\drandom)\\in\\argmax\\accolade{\\scal{\\vb}{\\Qrandom\\ubar{\\di}^p}:\\ubar{\\di}^p\\in\\pollDrandom}$ in the case of a random PSS $\\pollDrandom\\subset\\rp$; that is,\n\\begin{equation}\\label{kdpr2}\n\\prob{\\alpha_Q\\kappa(\\pollDrandom)\\normii{\\bm{v}}\\leq \\langle\\vb, \\Qrandom{\\hat{\\di}^p}(\\vb,\\qrandom,\\drandom)\\rangle}\\geq 1-\\beta_Q.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"90e75cd9b16f46f0b0ce034bd1af211fd58f4f5fc2afc00870243c31d49cd73d"},{"start":76882,"end":77972,"start_line":1031,"end_line":1033,"raw_tex":"\\begin{proof}\nBy construction, $\\Q=\\sqrt{\\frac{n}{p}}\\Ub_{\\ntp}$, where $\\ubar{\\Ub}_{\\ntp}$ represents the first $p$ columns of a Haar-distributed orthogonal matrix and $\\normii{\\di^p}=1$ for all $\\di^p\\in\\pollD$, whence $\\norme{\\Q}=\\sqrt{\\frac{n}{p}}=:\\Qmax =\\normii{\\Q\\di^p}$ since $\\normii{\\Ub_{\\ntp}\\di^p}=1$. Consequently, $\\accolade{\\exists\\ubar{\\di}_{\\star}^p\\in\\pollD:\\frac{\\alpha_Q}{\\Qmax}\\kappa(\\pollD)\\normii{\\bm{v}}\\normii{\\Qrandom\\ubar{\\di}_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom\\ubar{\\di}_{\\star}^p}}=\\accolade{\\exists\\ubar{\\di}_{\\star}^p\\in\\pollD:\\alpha_Q\\kappa(\\pollD)\\normii{\\bm{v}}\\leq \\scal{\\bm{v}}{\\Qrandom\\ubar{\\di}_{\\star}^p}}$, which is included in the event $\\accolade{\\alpha_Q\\kappa(\\pollD)\\normii{\\bm{v}}\\leq \\langle\\vb, \\Qrandom{\\hat{\\di}^p}(\\vb,\\qrandom)\\rangle}$, leading to~\\eqref{kdpr1} thanks to~\\eqref{kdprob1}. The result in~\\eqref{kdpr2} follows from~\\eqref{kdprob1prime} using similar arguments by replacing $\\pollD$ with~$\\pollDrandom$ and ${\\hat{\\di}^p}(\\vb,\\qrandom)$ with ${\\hat{\\di}^p}(\\vb,\\qrandom,\\drandom)$, and the proof is completed.\n\\end{proof}","sha256_utf8":"fab0dae8f9b1b909cab50584c50a021a35c96aa37f1d2d9044d70b61071c92fb"},{"start":77975,"end":78685,"start_line":1036,"end_line":1038,"raw_tex":"\\begin{lemma}\\cite[Theorem~2.7-$(\\ell)$]{bhattacharya2007basic}\\label{substLemma}\nLet $\\ubar{\\vb}$ and $\\Mrandom$ be random maps into $(\\mathcal{S}_1, \\mathscr{S}_1)$ and $(\\mathcal{S}_2, \\mathscr{S}_2)$, respectively. Let $\\mathcal{F}$ be a $\\sigma$-algebra and $\\psi$ be a measurable real-valued function on $\\left(\\mathcal{S}_1\\times \\mathcal{S}_2, \\mathscr{S}_1\\bigotimes\\mathscr{S}_2\\right)$. If $\\ubar{\\vb}$ is $\\mathcal{F}$-measurable, $\\sigma(\\Mrandom)$ and $\\mathcal{F}$ are independent, and $\\E{\\abs{\\psi(\\ubar{\\vb},\\Mrandom)}}<\\infty$, then $\\E{\\psi(\\ubar{\\vb},\\Mrandom)|\\mathcal{F}}=\\wp(\\ubar{\\vb})$ almost surely, where $\\wp(\\vb):=\\E{\\psi(\\vb,\\Mrandom)}$ for all $\\vb\\in\\mathcal{S}_1$.\n\\end{lemma}","sha256_utf8":"ff82e662cf259fda53eb55aa797e7bf3509ab569d27ed8a853d1d7a121f182a9"},{"start":79185,"end":80516,"start_line":1041,"end_line":1053,"raw_tex":"\\begin{proposition}\\label{propKappaD2}\nLet $\\mathcal{F}$ be a $\\sigma$-algebra independent of a random matrix $\\Qrandom\\in\\rnp$ given by Theorem~\\ref{JLforHaarTheor}, and let $\\ubar{\\vb}\\in \\rn$ be an $\\mathcal{F}$-measurable random vector. Let $\\pollD\\subset \\rp$ be a PSS, and consider the random vector defined by $\\hat{\\di}^p(\\ubar{\\vb},\\qrandom)\\in\\argmax\\accolade{\\scal{\\ubar{\\vb}}{\\Qrandom\\di^p}:\\di^p\\in\\pollD}$. Then \n%There exists a direction $\\di_{\\star}^p\\in\\pollD$ such that \n\\begin{equation}\\label{kdprob2}\n\\prob{\\left.\\alpha_Q\\kappa(\\pollD)\\normii{\\ubar{\\vb}} \\leq \\langle\\ubar{\\vb}, \\Qrandom\\hat{\\di}^p(\\ubar{\\vb},\\qrandom)\\rangle \\right|\\F}\\geq 1-\\beta_Q\\quad\\mbox{almost surely.}\n\\end{equation}\nA similar result holds for any $\\ubar{\\vb}\\in \\rn$ with $\\hat{\\di}^p(\\ubar{\\vb},\\qrandom,\\drandom)\\in\\argmax\\accolade{\\scal{\\ubar{\\vb}}{\\Qrandom\\ubar{\\di}^p}:\\ubar{\\di}^p\\in\\pollDrandom}$ in the case of a random PSS $\\pollDrandom\\subset\\rp$ with $\\Drandom$ independent of $\\mathcal{F}$, \n%and for a direction ${\\ubar{\\di}^p}_{\\star}\\in \\pollDrandom$; \nthat is,\n\\begin{equation}\\label{kdprob2prime}\n\\prob{\\left.\\alpha_Q\\kappa(\\pollDrandom)\\normii{\\ubar{\\vb}} \\leq \\scal{\\ubar{\\vb} }{\\Qrandom{\\hat{\\di}^p}(\\ubar{\\vb},\\qrandom,\\drandom)} \\right|\\F}\\geq 1-\\beta_Q\\quad\\mbox{almost surely.}\n\\end{equation}\n\\end{proposition}","sha256_utf8":"383a41578513f3608c421ddcc8fc28edd1a0501737f28dcf75c2b5e70ff3bdad"},{"start":80555,"end":82880,"start_line":1056,"end_line":1058,"raw_tex":"\\begin{proof}\nLet $\\psi$ defined on $(\\rn\\times\\rnp, \\mathscr{B}(\\rn)\\bigotimes\\mathscr{B}(\\rnp))$ by $\\psi(\\ubar{\\vb},\\Qrandom)=\\mathds{1}_{\\accolade{\\alpha_Q\\kappa(\\pollD)\\normii{\\ubar{\\vb}} \\leq \\scal{\\ubar{\\vb}}{\\Qrandom\\hat{\\di}^p(\\ubar{\\vb},\\qrandom)}}}$ be a real-valued measurable function. Since $\\E{\\abs{\\psi(\\ubar{\\vb},\\Qrandom)}}<\\infty$ trivially, it follows from Lemma~\\ref{substLemma} that $\\E{\\psi(\\ubar{\\vb},\\Qrandom)|\\mathcal{F}}=\\wp(\\ubar{\\vb})$ almost surely, where $\\wp(\\vb):=\\E{\\mathds{1}_{\\accolade{\\alpha_Q\\kappa(\\pollD)\\normii{\\vb} \\leq \\scal{\\vb}{\\qrandom\\hat{\\di}^p(\\vb,\\qrandom)}}}}=\\prob{\\alpha_Q\\kappa(\\pollD)\\normii{\\vb} \\leq \\langle\\vb,\\Qrandom\\hat{\\di}^p(\\vb,\\qrandom) \\rangle}$ for all $\\vb\\in\\rn$. It follows from~\\eqref{kdpr1} that $\\wp(\\vb)\\geq 1-\\beta_Q$ for all $\\vb\\in\\rn$ whence $\\wp(\\ubar{\\vb})\\geq 1-\\beta_Q$ almost surely. This means that $\\wp(\\ubar{\\vb}):=\\E{\\psi(\\ubar{\\vb},\\Qrandom)|\\mathcal{F}}=\\prob{\\left.\\alpha_Q\\kappa(\\pollD)\\normii{\\ubar{\\vb}} \\leq \\langle\\ubar{\\vb}, \\Qrandom\\hat{\\di}^p(\\ubar{\\vb},\\qrandom)\\rangle \\right|\\F}\\geq 1-\\beta_Q$ almost surely, proving~\\eqref{kdprob2}. To prove~\\eqref{kdprob2prime}, consider the measurable real-valued function $\\chi$ defined on $(\\rn\\times\\R^{p\\times (n+m)}, \\mathscr{B}(\\rn)\\bigotimes\\mathscr{B}(\\R^{p\\times (n+m)}))$ by $\\chi\\left(\\ubar{\\vb},\\left[\\Qrandom^{\\top}\\ \\Drandom\\right]\\right)=\\mathds{1}_{\\accolade{\\alpha_Q\\kappa(\\pollDrandom)\\normii{\\ubar{\\vb}} \\leq \\scal{\\ubar{\\vb}}{\\qrandom\\hat{\\di}^p(\\ubar{\\vb},\\qrandom,\\drandom)}}}$, satisfying $\\E{\\abs{\\chi\\left(\\ubar{\\vb},\\left[\\Qrandom^{\\top}\\ \\Drandom\\right]\\right)}}<\\infty$. $\\sigma\\left(\\left[\\Qrandom^{\\top}\\ \\Drandom\\right]\\right)$ is independent of~$\\F$ whence $\\E{\\left. \\chi\\left(\\ubar{\\vb},\\left[\\Qrandom^{\\top}\\ \\Drandom\\right]\\right) \\right|\\F}=\\tilde{\\wp}(\\ubar{\\vb})$, where $\\tilde{\\wp}(\\vb):=\\prob{\\alpha_Q\\kappa(\\pollDrandom)\\normii{{\\vb}} \\leq \\scal{{\\vb}}{\\Qrandom\\hat{\\di}^p({\\vb},\\qrandom,\\drandom)}}$, thanks to Lemma~\\ref{substLemma}. It follows from~\\eqref{kdpr2} that $\\tilde{\\wp}(\\vb)\\geq 1-\\beta_Q$ $\\forall\\vb\\in\\rn$, whence $\\tilde{\\wp}(\\ubar{\\vb}):=\\E{\\left. \\chi\\left(\\ubar{\\vb},\\left[\\Qrandom^{\\top}\\ \\Drandom\\right]\\right) \\right|\\F}\\geq 1-\\beta_Q$ almost surely, showing~\\eqref{kdprob2prime}, and the proof is completed.\n\\end{proof}","sha256_utf8":"802bf26b1b4f4fd9fa509fe513db7bfb49aa9d8d4dfb26078edcc5aec325777f"},{"start":97575,"end":97986,"start_line":1198,"end_line":1203,"raw_tex":"\\begin{proposition}\\label{decreaseProp1}(Inspired by~\\cite[Proposition~1]{dzahini2020expected})\nLet $\\gamma>2$ be a constant, and let $\\fok$ and $\\fuk$ be $\\ef$-accurate estimates of $f(\\xk)$ and $f(\\xk+\\dk\\ub)$, respectively: \n\\begin{equation}\\label{decreaseEqu}\n\\mbox{If}\\quad \\fuk-\\fok\\leq -\\gamma\\ef\\dk^2,\\qquad\\mbox{then}\\qquad f(\\xk+\\dk\\ub)-f(\\xk)\\leq -(\\gamma-2)\\ef\\dk^2.\n\\end{equation}\n\\end{proposition}","sha256_utf8":"a27ea7eb583bc22b198f143f7aab0a251beb2a132ed7a7f4e97f22e512dc781e"},{"start":97990,"end":98113,"start_line":1205,"end_line":1207,"raw_tex":"\\begin{proof}\nThe proof trivially follows that of~\\cite[Proposition~1]{dzahini2020expected} with minor changes.\n\\end{proof}","sha256_utf8":"1da5f7cad731610a25f92bfa2d10e7d377c8ae39ea72a7bf8a13afaa98a0df3c"},{"start":126480,"end":126930,"start_line":1396,"end_line":1402,"raw_tex":"\\begin{lemma}\\label{SDDSlemma1}\\cite[Lemma~1]{dzahini2020expected}\nUnder Assumption~\\ref{efkfAssump}, the following hold for all $k\\geq 0$ for the random process generated by Algorithm~\\ref{algoStoDARS}:\n\\begin{eqnarray*}\n\\E{\\ijkc\\abs{\\Fok-f(\\Xk)}|\\fqdalgebra}&\\leq& \\ef(1-\\beta_f)\\Dk^2\\quad\\mbox{almost surely, and}\\\\\n\\E{\\ijkc\\abs{\\Fuk-f(\\Xk+\\Dk\\urandom)}|\\fqdalgebra}&\\leq& \\ef(1-\\beta_f)\\Dk^2\\quad\\mbox{almost surely.}\n\\end{eqnarray*} \n\\end{lemma}","sha256_utf8":"708e01ccb37c8337bc3bab041f8c476bc5a6dc8adf44d374e39c07a84a608e3b"},{"start":126932,"end":127035,"start_line":1404,"end_line":1406,"raw_tex":"\\begin{proof}\nThe proof is almost identical to that of~\\cite[Lemma~1]{dzahini2020expected}.\n\\end{proof}","sha256_utf8":"92143ad4adca929b5444f8a0c022cb3dfbc870b4e3efe59d34696526f907b2ef"},{"start":127431,"end":128329,"start_line":1414,"end_line":1425,"raw_tex":"\\begin{theorem}\\label{zerothOrdTheor}\nLet Assumptions~\\ref{QkHaarAssumpt},~\\ref{efkfAssump} and~\\ref{boundedFassumpt} hold. \n%Let $\\gamma>2$ and $\\nu,\\tau\\in (0,1)$ satisfying $\\frac{\\nu}{1-\\nu}\\geq \\frac{2(\\tau^{-2}-1)}{\\gamma-2}$. Assume that Assumption~\\ref{efkfAssump} holds with $\\beta_f\\in (0,1)$ satisfying  $\\frac{\\beta_f}{1-\\beta_f}\\geq \\frac{4\\nu}{(1-\\nu)(1-\\tau^2)}$. \nDefine $\\varrho=\\frac{1}{2}\\beta_f(1-\\nu)(1-\\tau^2)$. Then, the random function $\\Phik:=\\frac{\\nu}{\\ef}\\left(f(\\Xk)-f_{\\min}\\right)+(1-\\nu)\\Dk^2$ satisfies \n\\begin{equation}\\label{PhikDynamics}\n\\E{\\Phikun-\\Phik|\\fqdalgebra}\\leq -\\varrho\\Dk^2 \\quad\\mbox{almost surely, for all}\\ k\\in \\N, \n\\end{equation}\nwhich implies that \n\\begin{equation}\\label{finiteSeries}\n\\sum_{k=0}^{\\infty}\\Dk^2<\\infty\\quad\\mbox{almost surely, whence}\\qquad \\underset{k\\to\\infty}{\\lim}\\Dk=0\\quad\\mbox{almost surely.}\n\\end{equation}\n\\end{theorem}","sha256_utf8":"73dd67321cb0cea82b25eeb529136ee5168267387fd6908d84f90989b2c75181"},{"start":128331,"end":128388,"start_line":1427,"end_line":1429,"raw_tex":"\\begin{proof}\n{\\bbl See Appendix~\\ref{th41}.}\n\\end{proof}","sha256_utf8":"e448408b1916f96290b0e7ea75e78acfeb652937249a4224341b172ea7208dec"},{"start":131966,"end":132194,"start_line":1469,"end_line":1472,"raw_tex":"\\begin{theorem}\\cite[Theorem~2]{blanchet2016convergence}\\label{theorComplexity1}\nUnder Assumption~\\ref{assumptionRenewal}, \n$\\E{\\Tepsilon}\\leq \\frac{q}{2q-1}\\cdot\\frac{\\PhiZero}{\\varrho h\\left(\\depsilon\\right)}+1.$\n\\end{theorem}","sha256_utf8":"8b7d6d2893789a90283a2aeba1676b9467c59dd542aeb70fe31854db14c54ca0"},{"start":133100,"end":134601,"start_line":1484,"end_line":1490,"raw_tex":"As in~\\cite{blanchet2016convergence,dzahini2020expected,DzaWildSub2022,paquette2018stochastic}, the occurrence of the event $\\accolade{\\Tepsilon=k}$ is determined by observing $\\left(\\PhiZero,\\Dzero,\\WtildeZero\\right),\\dots,\\left(\\Phikmun,\\Dkmun,\\Wtildekmun\\right)$, which means that~$\\Tepsilon$ is a stopping time for the stochastic process generated by Algorithm~\\ref{algoStoDARS}, and hence with respect to the filtration $\\accoladekinN{\\mathcal{H}_k}:=\\accoladekinN{\\falgebra}$. Bounding $\\E{\\Tepsilon}$ by applying Theorem~\\ref{theorComplexity1} requires showing next that Assumption~\\ref{assumptionRenewal} is satisfied for Algorithm~\\ref{algoStoDARS}.\n\nRegarding Assumption~\\ref{assumptionRenewal} {\\it (i)} and~{\\it (iii)}, the arguments are basically those in~\\cite{dzahini2020expected} and~\\cite{DzaWildSub2022}, respectively. More precisely,~{\\it (i)} is trivially satisfied for $\\lambda:=-\\log(\\tau)$ while, on the other hand, it follows from~\\eqref{PhikDynamics} and the inclusion $\\falgebra\\subset\\fqdalgebra$ that almost surely\n\\begin{equation}\\label{tower1}\n\\E{\\Phikun-\\Phik|\\falgebra}=\\E{\\E{\\Phikun-\\Phik|\\fqdalgebra}|\\falgebra}\\leq \\E{-\\varrho\\Dk^2|\\falgebra}=-\\varrho\\Dk^2.\n\\end{equation}\nThis shows that~{\\it (iii)} is satisfied with the function $h:\\scalebox{0.75}{$\\vartheta$}\\mapsto\\scalebox{0.75}{$\\vartheta$}^2$ since~\\eqref{tower1} holds for any realization of the random variable $\\E{\\Phikun-\\Phik|\\falgebra}$, and hence on the event $\\accolade{\\Tepsilon>k}$ in particular.\n","sha256_utf8":"1a3ebb804bb4abfe9d97a634cb0e9c9630daf46e07cbb2aec8b1ce51ee961d94"},{"start":135452,"end":135859,"start_line":1495,"end_line":1497,"raw_tex":"\\begin{lemma}\\label{AkLemma1}\nLet Assumption~\\ref{QkHaarAssumpt} hold, and consider ${\\ubar{\\hat{\\di}}^p_k}:=\\hat{\\di}^p_k({\\footnotesize{\\X}},\\qrandom,\\drandom)\\in\\argmax\\accolade{-\\nabla f(\\Xk)^{\\top}\\left(\\QkRandom\\diRandom\\right):\\diRandom\\in\\pollDrandom_k}$. Then the event $A_k$ defined in~\\eqref{AkEventDef} satisfies $\\prob{A_k|\\falgebra}=\\E{\\iak|\\falgebra}\\geq 1-\\beta_Q$ almost surely.\n\\end{lemma}","sha256_utf8":"a78a0d4021931603039d6f9c6cdef3f912165c87bfb5f4a920ffe2445acf04eb"},{"start":135860,"end":136156,"start_line":1498,"end_line":1500,"raw_tex":"\\begin{proof}\nBy construction, $\\nabla f(\\Xk)$ is $\\falgebra$-measurable while $\\QkRandom$ and $\\pollDrandom_k$ are independent of $\\falgebra$. The conditions of Proposition~\\ref{propKappaD2} are therefore satisfied, and the result straightforwardly follows from~\\eqref{kdprob2prime}.\n\\end{proof}","sha256_utf8":"d5d2ea89b1f9ed0ec76ebb59eafa127522003745dc2b2c16966d562ea5948daa"},{"start":137978,"end":138481,"start_line":1518,"end_line":1520,"raw_tex":"\\begin{lemma}\\label{smallLemmaHaar}\nLet Assumptions~\\ref{QkHaarAssumpt},~\\ref{kapminAssump} and~\\ref{gradLipAssump} hold, and assume that $\\dk\\leq \\depsilon$. Let $\\fok$ and $\\fuk$ be $\\ef$-accurate estimates of $f(\\xk)$ and  $f(\\xk+\\dk\\ub)$, respectively. Assume that the event $A_k$ occurs. If $\\normii{\\nabla f(\\xk)}>\\epsilon$, then there exists $\\ub\\in \\mathbb{U}_k^n:=\\{\\Ukp\\di^p:\\di^p\\in \\pollD_k\\}$ such that $\\fuk-\\fok\\leq -\\gamma\\ef\\dk^2$; that is, the iteration $k$ is successful. \n\\end{lemma}","sha256_utf8":"a49321d1d6915e95aede27b26ad8cebc3a43bf6ac78d82883a2c86686ebfdd1e"},{"start":138483,"end":138541,"start_line":1522,"end_line":1524,"raw_tex":"\\begin{proof}\n{\\bbl See Appendix~\\ref{lem52}.}\n\\end{proof}","sha256_utf8":"10fd28fe133c9a9d58fa030acc44ca529accc59e0805e527be9329e50658b1ec"},{"start":138742,"end":139292,"start_line":1527,"end_line":1531,"raw_tex":"\\begin{lemma}\\label{dynamicsLemma}\n{\\bl Let Assumption~\\ref{QkHaarAssumpt} hold with $0<\\beta_Q<1-\\frac{1}{2\\beta_f}$ and let Assumptions~\\ref{QkHaarAssumpt},~\\ref{efkfAssump},~\\ref{boundedFassumpt},~\\ref{kapminAssump}, and~\\ref{gradLipAssump} hold.}\n%and all assumptions made in Theorem~\\ref{zerothOrdTheor} hold. \nThen Assumption~\\ref{assumptionRenewal}-$(ii)$ is satisfied for $\\Wtildek:=2\\left(\\iak\\ijk-\\frac{1}{2}\\right)$, with $\\lambda=-\\log(\\tau)$ and some fixed $q\\in (\\tilde{\\beta}, 1)$, where $\\tilde{\\beta}=\\beta_f(1-\\beta_Q)$.\n\\end{lemma}","sha256_utf8":"545d01b9f9098b65254d58f6268e3c7fd01c58c7cf8d70d2c2891c72ce67c767"},{"start":139294,"end":140435,"start_line":1533,"end_line":1537,"raw_tex":"\\begin{proof}\nThe result is proved by suitably adapting the proof of~\\cite[Lemma~7]{blanchet2016convergence}, as was also done in~\\cite[Lemma~3]{dzahini2020expected}, \\cite[Lemma~5.3]{DzaWildSub2022} and~\\cite[Lemma~4.8]{paquette2018stochastic}. First, observe that~\\eqref{dynamicsRenewal} trivially holds when $\\itksup=0$. It is shown next that when $\\itksup=1$ (which is assumed in the remainder of the proof), then \n\\[\\Dkun\\geq \\min\\accolade{\\depsilon, \\min\\accolade{\\dmax,\\tau^{-1}\\Dk}\\iak\\ijk+\\tau\\Dk\\left(1-\\iak\\ijk\\right)}.\\] \nRecall that $\\dk=\\tau^{i_k}\\depsilon$ for some integer $i_k$ so that if $\\dk>\\depsilon$, then $\\dk\\geq \\tau^{-1}\\depsilon$, whence $\\dkun\\geq \\tau\\dk\\geq \\depsilon$. Next, assume that $\\dk\\leq \\depsilon$, and observe that $\\normii{\\nabla f(\\xk)}>\\eps$ since $\\Tepsilon>k$. If $\\iak\\ijk=1$, it follows from Lemma~\\ref{smallLemmaHaar} that the iteration is successful. Consequently, $\\dkun=\\min\\accolade{\\tau^{-1}\\dk,\\dmax}$. But if $\\iak\\ijk=0$, then it always holds that $\\dkun\\geq \\tau\\dk$. The proof is completed by observing that $\\prob{\\iak\\ijk=1|\\falgebra}=q$ for some $q\\geq\\tilde{\\beta}$.\n\\end{proof}","sha256_utf8":"355c69567df7ac58194dad7d72b183ec95850f4eff951cabd1d18bd3e35b21ba"},{"start":141166,"end":141323,"start_line":1546,"end_line":1548,"raw_tex":"\\begin{proof}\nThe proof, which trivially follows from Theorem~\\ref{theorComplexity1}, is identical to that of~\\cite[Theorem~5.4]{DzaWildSub2022}.\n\\end{proof}","sha256_utf8":"f9a1434688e7e732ea8e5b8c7a1c68ea64fb7acb80520daa5a11ad1af06eb480"},{"start":183112,"end":188736,"start_line":1838,"end_line":1877,"raw_tex":"\\begin{proof}\nWithout any loss of generality, it is sufficient to prove the result for $\\x\\in \\mathbb{S}^{n-1}$, which is assumed throughout. Let $\\bm{e_1}$ be the first standard basis vector of $\\rn$. \n%By the translation invariance of Haar measure, \nFor any $\\x\\in \\mathbb{S}^{n-1}$ arbitrary, $\\Urandom_{p\\times n}\\x \\equald \\Urandom_{p\\times n} \\bm{e_1}=(\\UrandomScal_{11},\\UrandomScal_{21},\\dots, \\UrandomScal_{p1})^{\\top}$,\n%. On the other hand, it follows from the definition of Haar measure that the first column of $\\Urandom$ satisfies \nwith $(\\UrandomScal_{11}, \\UrandomScal_{21},\\dots, \\UrandomScal_{n1})^{\\top}\\sim \\mathcal{U}(\\mathbb{S}^{n-1})$, hence $\\Urandom_{p\\times n}\\x \\equald \\left(\\xiRandom_1,\\xiRandom_2,\\dots, \\xiRandom_p\\right)^{\\top}$, where $\\xiRandomVec:=\\left(\\xiRandom_1,\\xiRandom_2,\\dots, \\xiRandom_n\\right)^{\\top}\\sim \\mathcal{U}(\\mathbb{S}^{n-1})$. \n%Consider the function $F:\\mathbb{S}^{n-1}\\to \\R$ defined by \nLet \\scalebox{0.88}{$F(x_1,x_2,\\dots,x_n):=\\sqrt{\\frac{n}{p}}\\normii{(x_1,x_2,\\dots,x_p)}$}, so that \\scalebox{0.88}{$F\\left(\\xiRandomVec\\right)=\\sqrt{\\frac{n}{p}}\\normii{\\left(\\xiRandom_1,\\xiRandom_2,\\dots, \\xiRandom_p\\right)}\\equald \\sqrt{\\frac{n}{p}}\\normii{\\Urandom_{p\\times n}\\x}=\\normii{\\QrandomTranspose\\x}$}. \nThen, for all $\\x,\\bm{y}\\in \\mathbb{S}^{n-1}$, $\\abs{F(\\x)-F(\\bm{y})}\\leq \\sqrt{\\frac{n}{p}}\\normii{\\x-\\bm{y}}$,\n% \\begin{equation*}\n% \\begin{split}\n% %\\abs{F(\\x)-F(\\bm{y})}&=\\sqrt{\\frac{n}{p}}\\abs{\\normii{(x_1,x_2,\\dots,x_p)}-\\normii{(y_1,y_2,\\dots,y_p)}}\\\\\n% \\abs{F(\\x)-F(\\bm{y})}&\\leq \\sqrt{\\frac{n}{p}}\\normii{(x_1-y_1),(x_2-y_2),\\dots,(x_p-y_p)}\\leq \\sqrt{\\frac{n}{p}}\\normii{\\x-\\bm{y}},\n% \\end{split}\n% \\end{equation*}\nwhich shows that $F$ is $\\sqrt{\\frac{n}{p}}$-Lipschitz on $\\mathbb{S}^{n-1}$. Thus, it follows from Theorem~\\ref{LeviTheorem} that there exists $\\eta>0$ such that for all $\\varepsilon_Q\\in (0,1)$, \n{\\footnotesize{\n\\begin{equation}\\label{LeviEq1}\n\\prob{\\abs{F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}}\\geq \\frac{\\varepsilon_Q}{2}}\\leq 2\\exp{\\left(-\\frac{\\eta n (\\varepsilon_Q/2)^2}{(\\sqrt{n/p})^2}\\right)}=2e^{-\\frac{\\eta p\\varepsilon_Q^2}{4}}.\n\\end{equation}\n}}\n%On the other hand, it follows from Lemma~\\ref{HaarLemma} that $\\E{(F\\left(\\xiRandomVec\\right))^2}=1$, which implies $\\var{F\\left(\\xiRandomVec\\right)}+(\\E{F\\left(\\xiRandomVec\\right)})^2=1$. \nMoreover, it follows from the integral identity formula that \n{\\footnotesize{\n\\begin{equation}\\label{levEq2}\n\\var{F\\left(\\xiRandomVec\\right)}=\\E{\\left(F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}\\right)^2}=\\int_0^{+\\infty}\\prob{\\abs{F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}}^2\\geq t} dt\\leq\\int_0^{+\\infty} 2e^{-\\eta p t} dt=\\frac{2}{\\eta p},\n\\end{equation}\n}}\nwhere the inequality follows from Theorem~\\ref{LeviTheorem}. \n%It follows from~\\eqref{levEq2} and the equality $\\E{(F\\left(\\xiRandomVec\\right))^2}=\\var{F\\left(\\xiRandomVec\\right)}+\\left(\\E{F\\left(\\xiRandomVec\\right)}\\right)^2=1$ (thanks to Lemma~\\ref{HaarLemma}) that $\\sqrt{1-\\frac{2}{\\eta p}}\\leq \\E{F\\left(\\xiRandomVec\\right)}\\leq 1$. \nFrom~\\eqref{levEq2} and Lemma~\\ref{HaarLemma}, we obtain $\\sqrt{1-\\frac{2}{\\eta p}}\\leq \\E{F\\left(\\xiRandomVec\\right)}\\leq 1$.\nAssume that $p$ is chosen so that $1-\\frac{\\varepsilon_Q}{2}\\leq\\sqrt{1-\\frac{2}{\\eta p}}$ and $2e^{-\\frac{\\eta p\\varepsilon_Q^2}{4}}\\leq \\beta_Q$. Then $\\abs{F\\left(\\xiRandomVec\\right)-1}<\\varepsilon_Q$ since $1-\\frac{\\varepsilon_Q}{2}\\leq \\E{F\\left(\\xiRandomVec\\right)}\\leq 1$.\n%Since $1-\\frac{\\varepsilon_Q}{2}\\leq \\E{F\\left(\\xiRandomVec\\right)}\\leq 1$, then the inequality $\\abs{F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}}<\\frac{\\varepsilon_Q}{2}$ implies that $\\abs{F\\left(\\xiRandomVec\\right)-1}<\\varepsilon_Q$. \nConsequently, for all $\\x\\in \\mathbb{S}^{n-1}$, it follows from~\\eqref{LeviEq1} that\n\\[\\prob{\\abs{\\normii{\\QrandomTranspose\\x}-1}\\geq \\varepsilon_Q}=\\prob{\\abs{F\\left(\\xiRandomVec\\right)-1}\\geq \\varepsilon_Q}\\leq\\prob{\\abs{F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}}\\geq\\frac{\\varepsilon_Q}{2}}\\leq2e^{-\\frac{\\eta p\\varepsilon_Q^2}{4}}\\leq \\beta_Q,\\]\nwhere the equality is due to the equality in distribution $F\\left(\\xiRandomVec\\right)\\equald \\normii{\\QrandomTranspose\\x}$ as explained above. Equivalently, it holds that $\\prob{1-\\varepsilon_Q\\leq \\normii{\\QrandomTranspose\\x}\\leq 1+\\varepsilon_Q}\\geq 1-\\beta_Q$ for all $\\x\\in \\mathbb{S}^{n-1}$. It remains to prove that choosing $p\\geq \\max\\accolade{\\frac{8}{3\\eta}\\varepsilon_Q^{-2}, \\frac{4}{\\eta}\\varepsilon_Q^{-2}\\log\\left(\\frac{2}{\\beta_Q}\\right)}=\\frac{4}{\\eta}\\varepsilon_Q^{-2}\\log\\left(\\frac{2}{\\beta_Q}\\right)$ ensures $1-\\frac{\\varepsilon_Q}{2}\\leq\\sqrt{1-\\frac{2}{\\eta p}}$ and $2e^{-\\frac{\\eta p\\varepsilon_Q^2}{4}}\\leq \\beta_Q$ as assumed above, where the latter equality trivially holds for any $\\varepsilon_Q,\\beta_Q\\in (0,1)$. We first observe that $2e^{-\\frac{\\eta p\\varepsilon_Q^2}{4}}\\leq \\beta_Q\\iff p\\geq \\frac{4}{\\eta}\\varepsilon_Q^{-2}\\log\\left(\\frac{2}{\\beta_Q}\\right)$. On the other hand, $\\left(1-\\frac{\\varepsilon_Q}{2}\\right)^2\\leq 1-\\frac{2}{\\eta p}$ if $p\\geq \\frac{2}{\\eta\\left(\\varepsilon_Q-\\frac{\\varepsilon_Q^2}{4}\\right)}$.\n%, which completes the proof. \nSince $\\varepsilon_Q\\in (0,1)$, then $\\varepsilon_Q-\\frac{\\varepsilon_Q^2}{4}>\\frac{3}{4}\\varepsilon_Q^2$, which implies that $\\frac{8}{3\\eta}\\varepsilon_Q^{-2}\\geq \\frac{2}{\\eta\\left(\\varepsilon_Q-\\frac{\\varepsilon_Q^2}{4}\\right)}$. Hence,  choosing $p\\geq \\frac{8}{3\\eta}\\varepsilon_Q^{-2}$ ensures $p\\geq \\frac{2}{\\eta\\left(\\varepsilon_Q-\\frac{\\varepsilon_Q^2}{4}\\right)}$.\n%, and the proof is completed.\n\\end{proof}","sha256_utf8":"9352b4119b509fa643f17357d85d7f1e449bf4407cc52f8b780348d7b141baf6"},{"start":192773,"end":197390,"start_line":1897,"end_line":1941,"raw_tex":"\\begin{proof}\nThe proof is similar to that of~\\cite[Theorem~2]{dzahini2020expected}, using ideas derived in~\\cite{blanchet2016convergence,chen2018stochastic,DzaWildSub2022,paquette2018stochastic}. It is presented here for the sake of clarity and self-containedness. The overall goal is to prove~\\eqref{PhikDynamics} since taking the expectation on both its sides leads to~\\eqref{finiteSeries} as detailed in~\\cite[Theorem~3]{dzahini2020expected}, given that $\\Phik>0$ for all~$k\\in\\N$. The proof considers two cases, {\\it good estimates} and {\\it bad estimates}, each of which is broken into whether the iteration is successful or not. Throughout, \n\\[\\mathfrak{S}_k:=\\accolade{\\mbox{The}\\ k\\mbox{th iteration is successful}}.\\]\n{\\bf Case~1:} {\\it (Good estimates, $\\ijk=1$).} The overall goal is to prove that almost surely, \n\\begin{equation}\\label{treizeEq}\n\\E{\\ijk\\left(\\Phikun-\\Phik\\right)|\\fqdalgebra}\\leq -\\beta_f(1-\\nu)(1-\\tau^2)\\Dk^2.\n\\end{equation}\n{{Subcase~$1i$}:} {\\it (Successful iteration, $\\isk=1$).} A decrease occurs in~$f$ according to~\\eqref{decreaseEqu}, and $\\Dk$ is updated according to~$\\Dkun=\\min\\accolade{\\tau^{-1}\\Dk,\\dmax}$, whence $\\ijk\\isk\\frac{\\nu}{\\ef}(f(\\Xkun)-f(\\Xk))\\leq -\\ijk\\isk\\nu(\\gamma-2)\\Dk^2$ and $\\ijk\\isk(1-\\nu)(\\Dkun^2-\\Dk^2)\\leq \\ijk\\isk(1-\\nu)(\\tau^{-2}-1)\\Dk^2$, leading to \n\\begin{equation}\\label{dixseptEq}\n\\ijk\\isk\\left(\\Phikun-\\Phik\\right)\\leq\\ijk\\isk\\left(-\\nu(\\gamma-2)+(1-\\nu)(\\tau^{-2}-1)\\right)\\Dk^2\\leq -\\ijk\\isk(1-\\nu)(1-\\tau^2)\\Dk^2,\n\\end{equation}\nwhere the last inequality follows from the above choice of~$\\nu$ along with the fact that $1-\\tau^2<\\tau^{-2}-1$. \\\\\n{{Subcase~$1ii$}:} {\\it (Unsuccessful iteration, $\\iskc=1$).} The parameter~$\\Dk$ is decreased according to $\\Dkun=\\tau\\Dk$, and $f(\\Xkun)=f(\\Xk)$, whence\n\\begin{equation}\\label{dixhuitEq}\n\\ijk\\iskc\\left(\\Phikun-\\Phik\\right)=-\\ijk\\iskc(1-\\nu)(1-\\tau^2)\\Dk^2.\n\\end{equation}\nEquations~\\eqref{dixseptEq} and~\\eqref{dixhuitEq} lead to\n\\begin{equation}\\label{vingtEq}\n\\ijk\\left(\\Phikun-\\Phik\\right)\\leq -\\ijk(1-\\nu)(1-\\tau^2)\\Dk^2.\n\\end{equation}\nTaking the conditional expectation with respect to $\\fqdalgebra$ on both sides of~\\eqref{vingtEq} and using Assumption~\\ref{efkfAssump}-{\\it (i)} lead to~\\eqref{treizeEq}.\n{\\bf Case~2:} {\\it (Bad estimates, $\\ijkc=1$).} Because of bad estimates, an iterate leading to an increase in~$f$ and~$\\Dk$, and hence in~$\\Phik$, can be accepted by Algorithm~\\ref{algoStoDARS}. Such an increase is controlled by using Assumption~\\ref{efkfAssump}-{\\it (ii)}. The overall goal is to show that almost surely,\n\\begin{equation}\\label{vingtEtUnEq}\n\\E{\\ijkc\\left(\\Phikun-\\Phik\\right)|\\fqdalgebra}\\leq 2\\nu(1-\\beta_f)\\Dk^2.\n\\end{equation}\n{{Subcase~$2i$}:} {\\it (Successful iteration, $\\isk=1$).} Since $\\Fuk-\\Fok\\leq-\\gamma\\ef\\Dk^2$, then \n\\[\\ijkc\\isk\\frac{\\nu}{\\ef}(f(\\Xkun)-f(\\Xk))\\leq\\ijkc\\isk\\nu\\left(-\\gamma\\Dk^2+\\frac{1}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right)\\right).\\]\nRecalling that $\\Dkun=\\min\\accolade{\\tau^{-1}\\Dk,\\dmax}$, and by noticing that $-\\nu\\gamma+(1-\\nu)(\\tau^{-2}-1)\\leq 0$ due to the above choice of $\\nu$, it holds that\n% \\begin{equation}\\label{vingtcinqEq}\n% \\begin{split}\n% \\ijkc\\isk\\left(\\Phikun-\\Phik\\right)&\\leq \\ijkc\\isk\\left(\\left(-\\nu\\gamma+(1-\\nu)(\\tau^{-2}-1) \\right)\\Dk^2+\\frac{\\nu}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right) \\right)\\\\\n% &\\leq \\ijkc\\isk\\left(\\frac{\\nu}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right) \\right)\n% \\end{split}\n% \\end{equation}\n\\begin{equation}\\label{vingtcinqEq}\n\\ijkc\\isk\\left(\\Phikun-\\Phik\\right)\\leq \\ijkc\\isk\\frac{\\nu}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right).\n\\end{equation}\n{{Subcase~$2ii$}:} {\\it (Unsuccessful iteration, $\\iskc=1$).} The upper bound on the change in~$\\Phik$ follows from~\\eqref{dixhuitEq} by replacing~$\\ijk$ with~$\\ijkc$. Thus, $\\ijkc\\iskc\\left(\\Phikun-\\Phik\\right)\\leq \\ijkc\\iskc\\frac{\\nu}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right)$, which, combined with~\\eqref{vingtcinqEq}, yields\n\\begin{equation}\\label{vingtseptEq}\n\\ijkc\\left(\\Phikun-\\Phik\\right)\\leq \\ijkc\\frac{\\nu}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right).\n\\end{equation}\nTaking the conditional expectation with respect to $\\fqdalgebra$ on both sides of~\\eqref{vingtseptEq} and applying Lemma~\\ref{SDDSlemma1} lead to~\\eqref{vingtEtUnEq}. Then, combining~\\eqref{treizeEq} and~\\eqref{vingtEtUnEq} yields\n\\[\\E{\\left(\\Phikun-\\Phik\\right)|\\fqdalgebra}\\leq (-\\beta_f(1-\\nu)(1-\\tau^2)+2\\nu(1-\\beta_f))\\Dk^2\\leq -\\varrho\\Dk^2, \\]\nwhere the last inequality follows from the above choice of $\\beta_f$, and the proof is completed.\n\\end{proof}","sha256_utf8":"d912cf34af0f1639405ce55e9955bd3ca45468190c0e0e5946fbf32c86c814c6"},{"start":197453,"end":200081,"start_line":1946,"end_line":1975,"raw_tex":"\\begin{proof}\nThe proof is inspired by that of~\\cite[Lemma~2]{dzahini2020expected} using elements derived in~\\cite{KoLeTo03a}. Suppose that $\\dk\\leq \\depsilon$, that the event $A_k$ occurs, and that the estimates $\\fok$ and $\\fuk$ are $\\ef$-accurate. Then, assume by way of contradiction that $\\fuk-\\fok> -\\gamma\\ef\\dk^2$ for all $\\ub\\in \\mathbb{U}_k^n$ of the form $\\ub:=\\Ukp\\di^p$, with $\\di^p\\in \\pollD_k$. Since $A_k$ occurs, then the latter inequality holds in particular for $\\ub=\\Ukp\\hat{\\di}^p_k$, with $\\hat{\\di}^p_k\\in\\argmax\\accolade{-\\nabla f(\\xk)^{\\top}\\left(\\Qk\\di^p\\right):\\di^p\\in\\pollD_k}$ satisfying \n\\begin{equation}\\label{AkRealization}\n\\alpha_Q\\kappa(\\pollD_k)\\normii{\\nabla f(\\xk)}\\leq -\\sqrt{\\frac{n}{p}}\\nabla f(\\xk)^{\\top}\\left(\\Ukp{\\hat{\\di}^p_k}\\right).\n\\end{equation}\nIt follows from the $\\ef$-accuracy assumption, along with the equality \n\\[f(\\xk+\\dk\\ub)-f(\\xk)=f(\\xk+\\dk\\ub)-\\fuk+(\\fuk-\\fok)+\\fok-f(\\xk),\\]\nthat \n\\begin{equation}\\label{eq39Acc}\nf\\left(\\xk+\\dk\\Ukp\\hat{\\di}^p_k\\right)-f(\\xk)+(\\gamma+2)\\ef\\dk^2\\geq 0.\n\\end{equation}\nIt follows from the mean value theorem and~\\eqref{eq39Acc} that there exists $\\mu_k\\in[0,1]$ such that \n\\begin{equation}\\label{eq40MeanVal}\n0\\leq \\dk\\nabla f\\left(\\xk+\\mu_k\\dk\\Ukp\\hat{\\di}^p_k\\right)^{\\top}\\left(\\Ukp{\\hat{\\di}^p_k}\\right)+(\\gamma+2)\\ef\\dk^2.\n\\end{equation}\nMultiplying both sides of~\\eqref{eq40MeanVal} by $\\frac{1}{\\dk}\\sqrt{\\frac{n}{p}}$ and subtracting $\\sqrt{\\frac{n}{p}}\\nabla f(\\xk)^{\\top}\\left(\\Ukp{\\hat{\\di}^p_k}\\right)$ yield\n\\begin{equation}\\label{eq41yield}\n-\\sqrt{\\frac{n}{p}}\\nabla f(\\xk)^{\\top}\\left(\\Ukp{\\hat{\\di}^p_k}\\right)\\leq\\sqrt{\\frac{n}{p}}\\left(\\nabla f\\left(\\xk+\\mu_k\\dk\\Ukp\\hat{\\di}^p_k\\right)- \\nabla f(\\xk)\\right)^{\\top}\\left(\\Ukp{\\hat{\\di}^p_k}\\right)+(\\gamma+2)\\ef\\dk,\n\\end{equation}\nwhich, combined with~\\eqref{AkRealization}, leads to\n\\begin{equation}\\label{eq42Norm}\n\\begin{split}\n\\alpha_Q\\kappa(\\pollD_k)\\normii{\\nabla f(\\xk)}&\\leq L_g\\sqrt{\\frac{n}{p}}\\normii{\\mu_k\\dk\\Ukp{\\hat{\\di}^p_k}}\\normii{\\Ukp{\\hat{\\di}^p_k}}+(\\gamma+2)\\ef\\dk \\leq \\left(L_g\\sqrt{\\frac{n}{p}}+(\\gamma+2)\\ef\\right)\\depsilon,%\\\\\n%&\\leq L_g\\sqrt{\\frac{n}{p}}\\dk+(\\gamma+2)\\ef\\dk\n\\end{split}\n\\end{equation}\nwhere the last inequality follows from the $L_g$-Lipschitz continuity of $\\nabla f$, the fact that $\\normii{\\Ukp{\\hat{\\di}^p_k}}=1$ and $\\dk\\leq\\depsilon$. It follows from~\\eqref{eq42Norm} that \n\\[\\normii{\\nabla f(\\xk)}\\leq \\alpha_Q^{-1}\\kappa_{\\min}^{-1}\\left(L_g\\sqrt{\\frac{n}{p}}+(\\gamma+2)\\ef\\right)\\times \\frac{\\epsilon}{\\xi}< \\epsilon,\\]\nwhere the last inequality follows from~\\eqref{depsEq}, and the proof is completed.\n\\end{proof}","sha256_utf8":"1b3882609dfc2ac05920b1ee7e7941e52312ad99e1a5fbe1c6fbb467687b4e4e"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"StoDARS期望复杂度","source_units":[{"start":140676,"end":141164,"start_line":1540,"end_line":1544,"raw_tex":"\\begin{theorem}\\label{complexResult}\nLet all assumptions made in Lemma~\\ref{dynamicsLemma} hold. Then $\\E{\\Tepsilon}\\leq \\frac{\\tilde{\\beta}}{2\\tilde{\\beta}-1}\\cdot\\frac{\\PhiZero{\\tilde{\\xi}}^2}{\\varrho \\eps^2}+1$, for some $\\tilde{\\xi}\\geq\\hat{\\xi}$, where~{\\bl $\\varrho=\\frac{1}{2}\\beta_f(1-\\nu)(1-\\tau^2)$}, \n%is the constant of Theorem~\\ref{zerothOrdTheor}, \n$\\tilde{\\beta}$ is the same parameter of Lemma~\\ref{dynamicsLemma}, and $\\hat{\\xi}$ is given by~\\eqref{depsEq}.\n\\end{theorem}","sha256_utf8":"af6daff8ead198b3f863b7912d4a116910c9ed4e52e10f3c00275f417023f3af"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"复杂度完整证明","source_units":[{"start":133100,"end":134601,"start_line":1484,"end_line":1490,"raw_tex":"As in~\\cite{blanchet2016convergence,dzahini2020expected,DzaWildSub2022,paquette2018stochastic}, the occurrence of the event $\\accolade{\\Tepsilon=k}$ is determined by observing $\\left(\\PhiZero,\\Dzero,\\WtildeZero\\right),\\dots,\\left(\\Phikmun,\\Dkmun,\\Wtildekmun\\right)$, which means that~$\\Tepsilon$ is a stopping time for the stochastic process generated by Algorithm~\\ref{algoStoDARS}, and hence with respect to the filtration $\\accoladekinN{\\mathcal{H}_k}:=\\accoladekinN{\\falgebra}$. Bounding $\\E{\\Tepsilon}$ by applying Theorem~\\ref{theorComplexity1} requires showing next that Assumption~\\ref{assumptionRenewal} is satisfied for Algorithm~\\ref{algoStoDARS}.\n\nRegarding Assumption~\\ref{assumptionRenewal} {\\it (i)} and~{\\it (iii)}, the arguments are basically those in~\\cite{dzahini2020expected} and~\\cite{DzaWildSub2022}, respectively. More precisely,~{\\it (i)} is trivially satisfied for $\\lambda:=-\\log(\\tau)$ while, on the other hand, it follows from~\\eqref{PhikDynamics} and the inclusion $\\falgebra\\subset\\fqdalgebra$ that almost surely\n\\begin{equation}\\label{tower1}\n\\E{\\Phikun-\\Phik|\\falgebra}=\\E{\\E{\\Phikun-\\Phik|\\fqdalgebra}|\\falgebra}\\leq \\E{-\\varrho\\Dk^2|\\falgebra}=-\\varrho\\Dk^2.\n\\end{equation}\nThis shows that~{\\it (iii)} is satisfied with the function $h:\\scalebox{0.75}{$\\vartheta$}\\mapsto\\scalebox{0.75}{$\\vartheta$}^2$ since~\\eqref{tower1} holds for any realization of the random variable $\\E{\\Phikun-\\Phik|\\falgebra}$, and hence on the event $\\accolade{\\Tepsilon>k}$ in particular.\n","sha256_utf8":"1a3ebb804bb4abfe9d97a634cb0e9c9630daf46e07cbb2aec8b1ce51ee961d94"},{"start":141166,"end":141323,"start_line":1546,"end_line":1548,"raw_tex":"\\begin{proof}\nThe proof, which trivially follows from Theorem~\\ref{theorComplexity1}, is identical to that of~\\cite[Theorem~5.4]{DzaWildSub2022}.\n\\end{proof}","sha256_utf8":"f9a1434688e7e732ea8e5b8c7a1c68ea64fb7acb80520daa5a11ad1af06eb480"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"确认梯度阈值停止时间","source_units":[{"start":133100,"end":133759,"start_line":1484,"end_line":1484,"raw_tex":"As in~\\cite{blanchet2016convergence,dzahini2020expected,DzaWildSub2022,paquette2018stochastic}, the occurrence of the event $\\accolade{\\Tepsilon=k}$ is determined by observing $\\left(\\PhiZero,\\Dzero,\\WtildeZero\\right),\\dots,\\left(\\Phikmun,\\Dkmun,\\Wtildekmun\\right)$, which means that~$\\Tepsilon$ is a stopping time for the stochastic process generated by Algorithm~\\ref{algoStoDARS}, and hence with respect to the filtration $\\accoladekinN{\\mathcal{H}_k}:=\\accoladekinN{\\falgebra}$. Bounding $\\E{\\Tepsilon}$ by applying Theorem~\\ref{theorComplexity1} requires showing next that Assumption~\\ref{assumptionRenewal} is satisfied for Algorithm~\\ref{algoStoDARS}.\n","sha256_utf8":"9da6378c3c48958763a5b4c54167edbbe0a4b7020c4ea1c03361d9eeb0b7b7ef"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"验证步长上界与势函数漂移","source_units":[{"start":133760,"end":134601,"start_line":1486,"end_line":1490,"raw_tex":"Regarding Assumption~\\ref{assumptionRenewal} {\\it (i)} and~{\\it (iii)}, the arguments are basically those in~\\cite{dzahini2020expected} and~\\cite{DzaWildSub2022}, respectively. More precisely,~{\\it (i)} is trivially satisfied for $\\lambda:=-\\log(\\tau)$ while, on the other hand, it follows from~\\eqref{PhikDynamics} and the inclusion $\\falgebra\\subset\\fqdalgebra$ that almost surely\n\\begin{equation}\\label{tower1}\n\\E{\\Phikun-\\Phik|\\falgebra}=\\E{\\E{\\Phikun-\\Phik|\\fqdalgebra}|\\falgebra}\\leq \\E{-\\varrho\\Dk^2|\\falgebra}=-\\varrho\\Dk^2.\n\\end{equation}\nThis shows that~{\\it (iii)} is satisfied with the function $h:\\scalebox{0.75}{$\\vartheta$}\\mapsto\\scalebox{0.75}{$\\vartheta$}^2$ since~\\eqref{tower1} holds for any realization of the random variable $\\E{\\Phikun-\\Phik|\\falgebra}$, and hence on the event $\\accolade{\\Tepsilon>k}$ in particular.\n","sha256_utf8":"1536a2aeb9f66eaf943875249442d39254cd7a278eee34e8fdb7029f56754a50"}]},{"id":"S005","semantic_type":"proof_step","kind":"proof_step","title":"应用更新报酬停止时间界","source_units":[{"start":141166,"end":141323,"start_line":1546,"end_line":1548,"raw_tex":"\\begin{proof}\nThe proof, which trivially follows from Theorem~\\ref{theorComplexity1}, is identical to that of~\\cite[Theorem~5.4]{DzaWildSub2022}.\n\\end{proof}","sha256_utf8":"f9a1434688e7e732ea8e5b8c7a1c68ea64fb7acb80520daa5a11ad1af06eb480"}]},{"id":"S006","semantic_type":"problem_setting","kind":"problem_setting","title":"随机黑箱优化问题","source_units":[{"start":37999,"end":38163,"start_line":841,"end_line":843,"raw_tex":"\\begin{equation}\\label{probl1}\n\\underset{\\x\\in\\bcX}{\\min} f(\\x), \\qquad\\mbox{with}\\quad f(\\x)=\\Esp_{\\ubar{\\theta}}\\left[f_{\\ubar{\\theta}}(\\x)\\right],\n\\end{equation}","sha256_utf8":"71c0aa734bf9b6aea2a49b91c8dfbb3c97bb51d091eef36b1864e2c548c55ce1"},{"start":38164,"end":39010,"start_line":844,"end_line":844,"raw_tex":"where {\\bl the nonempty set} $\\bcX \\subseteq\\rn$ {\\bl of nonzero measure} is a feasible region; the values of the function $f:\\rn\\to\\R$ are available only through $f_{\\ubar{\\theta}}$, a stochastically noisy version of $f$;  ${\\ubar{\\theta}}$ is a random variable whose probability distribution is possibly unknown; and $\\Esp_{\\ubar{\\theta}}$ denotes the {\\it expectation} with respect to ${\\ubar{\\theta}}$. Problem~\\eqref{probl1} arises, for example, in statistical machine learning~\\cite{bottou2018optimization} where $\\ubar{\\theta}$ is a random variable describing a given sample set while $f_{\\ubar{\\theta}}$ represents the composition of a loss function and a prediction function. {\\bl We particularly target settings where the problem dimension $n>100$, which is large in the context of derivative-free methods for stochastic optimization.}\n","sha256_utf8":"69414cd02e3c2263736fb3e6f43f64fe667a84b90d0c026cb44d53e0817036cb"}]},{"id":"S007","semantic_type":"notation","kind":"notation","title":"随机量与范数记号","source_units":[{"start":50665,"end":53009,"start_line":862,"end_line":864,"raw_tex":"Throughout the manuscript, vectors are written in lowercase boldface (e.g., $\\vb\\in\\rn, n\\geq 2$), and matrices are written in uppercase boldface (e.g., $\\Ub\\in\\rnp$). Unless otherwise stated, all random quantities are underlined (e.g., $\\ubar{z}$, $\\vrandom$, $\\Urandom$) and are defined on the same probability space $(\\Omega,\\F,\\pr)$. The nonempty set~$\\Omega$ referred to as the {\\it sample space} is a collection of elements~$\\omega$ called {\\it sample points}, while subsets of~$\\Omega$ are called {\\it events}. The collection, $\\F$, of these events is called a $\\sigma$-algebra; and the finite measure~$\\pr$ defined on the {\\it measurable space} $(\\Omega,\\F)$, with $\\prob{\\Omega}=1$, is a {\\it probability measure}. A {\\it filtration} is an increasing subsequence $\\accolade{\\mathcal{H}_k}_k$ of $\\sigma$-algebras of~$\\F$. By denoting by~$\\mathscr{B}(\\rn)$ the Borel $\\sigma$-algebra generated by open sets of~$\\rn$, a real-valued random variable~$\\ubar{z}$ is a measurable map $\\ubar{z}:(\\Omega,\\F,\\pr)\\to (\\R,\\mathscr{B}(\\R))$, where measurability means that $\\accolade{\\ubar{z}\\in \\mathscr{E}}:=\\accolade{\\omega\\in\\Omega:\\ubar{z}(\\omega)\\in\\mathscr{E}}=:\\ubar{z}^{-1}(\\mathscr{E})\\in\\F$ for all $\\mathscr{E}\\in \\mathscr{B}(\\R)$~\\cite{bhattacharya2007basic}. A real-valued random matrix is a matrix-valued random variable, that is, a matrix whose entries are real-valued random variables. $\\normii{\\ub}:=\\left(\\sum_{i=1}^n u_i^2\\right)^{1/2}$ denotes the Euclidean norm of a vector $\\ub:=(u_1,\\dots,u_n)^{\\top}\\in\\rn$.  $\\norme{\\cdot}$ is used for the norm of a matrix and is assumed \n%GAIL - bu \"supposed to be\" do you mean that it really is not? or do you mean it is assumed to be?\nto be {\\it consistent} with the Euclidean norm, that is, $\\normii{\\Q\\ub}\\leq \\norme{\\Q}\\normii{\\ub}$, which holds for both spectral and Frobenius matrix norms. $\\ub^{\\top}\\vb=\\scal{\\ub}{\\vb}:=\\sum_{i=1}^n u_i v_i$ denotes the {\\it scalar product} or {\\it dot product} of the vectors $\\ub$ and $\\vb:=(v_1,\\dots,v_n)^{\\top}\\in\\rn$. $K_{\\infty}$ denotes the random set of all {\\it unsuccessful iterations} of a given algorithm. A realization of a random set~$K$ is denoted by~$\\mathcal{K}$; that is, $K(\\omega)=\\mathcal{K}$ for some~$\\omega\\in\\Omega$. {\\bl For $a\\leq b$, $\\intbracket{a,b}:=[a,b]\\cap\\Z$ is the set of integers in the interval $[a,b]$.}\n","sha256_utf8":"fc7346a3f5ca51209e227b7d5ce0ceb010a20e1da22665621428963acd8c0fdf"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"正生成集与余弦测度","source_units":[{"start":54018,"end":55653,"start_line":876,"end_line":880,"raw_tex":"SDDS algorithms were introduced in~\\cite{dzahini2020expected} to solve the stochastic optimization problem~\\eqref{probl1} in the particular case $\\bcX=\\rn$. These algorithms use a framework inspired by that of StoMADS~\\cite{audet2019stomads} and the broad class of methods presented in~\\cite{Vicente2013} using a {\\it generating set search} method~\\cite{KoLeTo03a}, such as pattern search and generalized pattern search~\\cite{AuDe03a}. Each iteration of SDDS algorithms is composed of two main steps: an optional {\\it search}, not shown in~\\cite[Algorithm~1]{dzahini2020expected}, and not detailed in Algorithm~\\ref{vagueDirAlgo} for simplicity, and the ``more disciplined''~\\cite{LaMeWi2019} {\\it poll} on which  the theoretical analyses rely. Broadly speaking, each poll step of an SDDS method generates a finite set of {\\it poll points} by taking an {\\it incumbent solution}~$\\xk\\in\\rn$ and adding terms of the form~$\\dk\\di^n$, where $\\dk>0$ is the {\\it stepsize} and $\\di^n\\in\\rn$ is a nonzero vector from a finite set $\\mathbb{D}^n_k\\subset\\rn$ of {\\it poll directions}. More precisely, $\\mathbb{D}^n_k$ is a PSS, meaning that any vector in~$\\rn$ can be written as a nonnegative linear combination of elements in~$\\mathbb{D}^n_k$~\\cite[Definition~6.1]{AuHa2017} or, equivalently, the cosine measure $\\kappa(\\mathbb{D}^n_k)>0$, where \n%for a given $\\mathbb{D}\\subset\\rn$,\n\\begin{equation}\\label{cosineMeasureDef}\n\\kappa(\\mathbb{D}^q):=\\underset{\\wb\\in \\R^q,\\wb\\neq \\bm{0}}{\\min}\\ \\underset{\\di^q\\in \\mathbb{D}^q}{\\max}\\frac{\\scal{\\wb}{\\di^q}}{\\normii{\\wb}\\normii{\\di^q}}\\quad\\mbox{with}\\quad \\mathbb{D}^q\\subset\\R^q.\n\\end{equation}","sha256_utf8":"5f5a26e111e775294892565eb37bb353597441faec539a427bbd4e4d21883234"}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","title":"正生成集方向内积界","source_units":[{"start":58959,"end":59220,"start_line":916,"end_line":918,"raw_tex":"\\begin{equation}\\label{cosineDescentDkn}\n\\mbox{for any}\\ \\vi\\in\\rn,\\quad\\exists \\di_{\\star}^n(\\vi)\\in \\mathbb{D}^n\\subset\\rn\\quad\\mbox{such that}\\quad \\kappa(\\mathbb{D}^n)\\normii{\\vi}\\normii{\\di^n_{\\star}(\\vi)}\\leq \\scal{\\vi}{\\di^n_{\\star}(\\vi)}.\n\\end{equation}","sha256_utf8":"9cbf677077ba72668e46acfcc4719522cf0f303ac79579ae78adba33bf98984a"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"概率良好对齐","source_units":[{"start":61684,"end":62097,"start_line":931,"end_line":935,"raw_tex":"\\begin{definition}\\label{wellAlignment}\nFor fixed $\\alpha_Q,\\beta_Q\\in (0,1/2)$, \n%a sequence $\\accoladekinN{\\QkRandom}$ of independent \na random matrix $\\Qrandom\\in\\rnp$ is called $(1-\\beta_Q)$-probabilistically $\\alpha_Q$-well aligned if, for all $\\vb\\in\\rn$, $\\prob{\\normii{\\Qrandom^{\\top}\\vb}\\geq \\alpha_Q\\normii{\\vb}}\\geq 1-\\beta_Q$. Such matrices are referred to as $WAM(\\alpha_Q,\\beta_Q)$.\n\\end{definition}","sha256_utf8":"ff2c7df51dbfc2dd1d68709c72d53cb734cdd26566075337d033d1e25b537db8"}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","title":"对齐概率与矩阵范数界","source_units":[{"start":63322,"end":63810,"start_line":944,"end_line":947,"raw_tex":"\\begin{assumption}\\label{JltAss1}{\\bl (Inspired by~\\cite[Assumptions~2 and~3]{DzaWildSub2022} and~\\cite{CRsubspace2021})}\nThe random matrix $\\Qrandom\\in\\rnp$ is a $WAM(\\alpha_Q,\\beta_Q)$ for fixed $\\alpha_Q,\\beta_Q\\in (0,1/2)$ and satisfies $\\norme{\\Qrandom}\\leq\\Qmax$ almost surely for some $\\Qmax>0$. \n%and $\\prob{\\normii{\\QrandomTranspose\\bm{v}}\\geq \\alpha_Q\\normii{\\bm{v}}}\\geq 1-\\beta_Q$ for all $\\bm{v}\\in\\rn$, for some $\\alpha_Q\\in (0,1)$ and $\\beta_Q\\in (0,1/2)$.\n\\end{assumption}","sha256_utf8":"a2d594800ab920e3a5be97f31d6055474a7ae3cc134ff45b4e245075b07e171b"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"正交群与Haar分布","source_units":[{"start":66855,"end":68285,"start_line":973,"end_line":975,"raw_tex":"The poll directions used by Algorithm~\\ref{algoStoDARS} are generated by using Haar-distributed orthogonal random matrices. Recall that a matrix $\\bm{O}\\in \\rnn$ is {\\it orthogonal} if and only if $\\bm{O}^{\\top}\\bm{O}=\\bm{O}\\bm{O}^{\\top}=\\bm{I}_n$ or equivalently if its columns form an orthonormal basis of $\\rn$~\\cite[Section~1.1]{MeckesElizHaar2019}. Throughout the manuscript, the compact Lie group of $n\\times n$ orthogonal matrices over~$\\R$ will be denoted by $\\mathbb{O}(n)$. \n%We also recall that $\\mathbb{O}(n)$ is a group under multiplication, and that this group admits a compact Lie group structure, that is, it is also a differentiable manifold such that the multiplication operation $(\\bm{A}, \\bm{B})\\mapsto \\bm{A}\\bm{B}$ and the inversion operation $\\bm{A}\\mapsto \\bm{A}^{-1}$ are smooth maps. \nA {\\it uniform random element} of $\\mathbb{O}(n)$ is a random $\\Urandom\\in \\mathbb{O}(n)$ whose distribution is translation invariant; that is, for any fixed $\\bm{O}\\in \\mathbb{O}(n)$, $\\bm{O}\\Urandom\\equald \\Urandom\\bm{O}\\equald \\Urandom$~\\cite[Section~1.2]{MeckesElizHaar2019}, where ``$\\equald$'' is used for the equality in distribution. In particular, there exists a unique translation-invariant probability measure on $\\mathbb{O}(n)$ called Haar measure on~$\\mathbb{O}(n)$~\\cite[Theorem~1.4]{MeckesElizHaar2019}. Any random matrix whose distribution is Haar measure will be called Haar-{\\it distributed matrix}. \n","sha256_utf8":"9c65241b07a4bcf26a7657114892d697f13054aecce303a6d036818584d4ae70"}]},{"id":"S013","semantic_type":"algorithm","kind":"algorithm","title":"StoDARS算法","source_units":[{"start":98212,"end":103522,"start_line":1218,"end_line":1270,"raw_tex":"\\begin{algorithm}[H]\n\\caption{StoDARS algorithm}\n\\label{algoStoDARS} \n%\\nonl\n% $\\accolade{\\Urandomtilde_t}_{t=0}^\\infty$, where the $\\Urandomtilde_t:=\\left[\\Urandomtilde_{t,\\ntp}\\ \\ \\Urandomtilde_{t,\\ntnmp}\\right]\\in\\mathbb{O}(n)$ are Haar-distributed, is a sequence of independent matrices, with $p\\in\\intbracket{1,n}$ chosen so that $\\sqrt{\\frac{n}{p}}\\Urandomtilde_{t,\\ntp}\\in\\rnp$ is $WAM(\\alpha_Q,\\beta_Q)$ for some $\\alpha_Q, \\beta_Q\\in(0,1/2)$. \\\\\n% $\\accolade{\\BrandomtildeSet_t^+}_{t=0}^\\infty$ is a sequence of random PSSs such that the matrices $\\Brandomtilde_t^+\\in\\R^{p\\times m}$ with unit columns in $\\BrandomtildeSet_t^+\\subset\\mathbb{S}^{p-1}$ are independent.    \\\\\n% $ $\\\\\n\\textbf{[0] Initialization}\\\\\n\\hspace*{10mm}Choose $\\x_0\\in\\bcX\\subseteq\\rn$, $\\delta_0>0$, $\\ef>0$, $\\gamma>2$, $0<\\tau<\\left(\\frac{\\gamma-2}{\\gamma+2}\\right)^{1/2}$, $j_{\\max}\\in\\N$, $\\dmax=\\tau^{-j_{\\max}}\\delta_0$ \\\\ \n\\hspace*{10mm}$\\ell_0=0$ and $t_0=0$. Generate $\\tilde{\\Ub}_{0,\\ntp}$, where $\\Urandomtilde_0:=\\left[\\Urandomtilde_{0,\\ntp}\\ \\ \\Urandomtilde_{0,\\ntnmp}\\right]\\in\\mathbb{O}(n)$ is Haar-distributed, \\\\\n\\hspace*{10mm}with $p\\in\\intbracket{1,n}$ chosen so that $\\sqrt{\\frac{n}{p}}\\Urandomtilde_{0,\\ntp}\\in\\rnp$ is $WAM(\\alpha_Q,\\beta_Q)$ for some $\\alpha_Q, \\beta_Q\\in(0,1/2)$.\\\\\n\\hspace*{10mm}Construct a PSS $\\tilde{\\mathbb{D}}^p_0\\subset\\spOne$ using {\\bl any matrix} $\\Drandomtildep_0\\in\\R^{p\\times m}$, with $m\\in {\\bl \\intbracket{p+1,2p}}$, {\\bl generated}\\\\ \n\\hspace*{10mm}{\\bl independently of $\\Urandomtilde_{0}$, whose columns form a PSS of~$\\rp$}.\\\\\n\\hspace*{10mm}Set the iteration counter $k \\gets 0$.\\\\\n% {\\bl\\textbf{[1] Matrix index update}\\\\\n% \\hspace*{10mm}If $\\ell_{k} \\geq  \\max_{i\\leq k-1} \\ell_i$ or $k=0$, set $t_{k}\\gets \\ell_{k}$.   \\\\\n% \\hspace*{10mm}Otherwise $t_{k}\\gets 1+\\max_{i\\leq k-1} t_i$.\\\\}\n\\textbf{[{1}] Matrix and PSS for subspace polling}\\\\\n% \\hspace*{10mm}If $k\\geq 1$ and $t_k=t_{i_0}$ for some $i_0\\in \\intbracket{0,k-1}$   \\\\\n% \\hspace*{16mm}Generate $\\tilde{\\Ub}_{t_k,\\ntp}$ using an independent copy $\\Urandomtilde_{t_k,\\ntp}$ of $\\Urandomtilde_{t_{i_0},\\ntp}$ such that  \\\\\n% \\hspace*{16mm}$\\{\\Urandomtilde_{t_i,\\ntp}\\}_{i=0}^{k-1}$ and $\\Urandomtilde_{t_k,\\ntp}$ are i.i.d. \\\\\n% \\hspace*{16mm}Construct a PSS $\\tilde{\\mathbb{D}}^p_{t_k}\\subset\\spOne$ using the columns of an independent copy $\\Drandomtildep_{t_k}$ of $\\Drandomtildep_{t_{i_0}}$ \\\\\n% \\hspace*{16mm}such that $\\{\\Drandomtildep_{t_i}\\}_{i=0}^{k-1}$ and $\\Drandomtildep_{t_k}$ are i.i.d. \\\\\n% \\hspace*{10mm}If $k\\geq 1$ and $t_k\\not\\in\\accolade{t_0,\\dots,t_{k-1}}$ \\\\\n\\hspace*{10mm}Generate $\\tilde{\\Ub}_{t_k,\\ntp}$ as a realization of $\\Urandomtilde_{t_k,\\ntp}$, with $\\{\\Urandomtilde_{t_i,\\ntp}\\}_{i=0}^{k}$ {\\bl obtained from the i.i.d. $\\{\\Urandomtilde_{t_i}\\}_{i=0}^{k}$}.\\\\\n\\hspace*{10mm}Construct a PSS $\\tilde{\\mathbb{D}}^p_{t_k}\\subset\\spOne$ using the columns of $\\Drandomtildep_{t_k}$, where the independent $\\{\\Drandomtildep_{t_i}\\}_{i=0}^{k}$ {\\bl are generated}\\\\ \n\\hspace*{10mm}{\\bl independently of $\\{\\Urandomtilde_{t_i}\\}_{i=0}^{k}$}.  \\\\\n% \\hspace*{16mm}such that $\\accolade{\\Urandom_{t_i,\\ntp}}_{i=0}^k$ and $\\{\\Urandomtilde_{t_i,\\ntp}\\}_{i=0}^k$ are independent.   \\\\\n% \\hspace*{16mm}Select $\\BrandomSet_{t_k}^+$ as the columns of an independent copy $\\Brandom_{t_k}^+$ of $\\Brandomtilde_{t_k}^+$, where $\\Brandom_{t_k}^+$ is independent of  \\\\\n% % \\hspace*{16mm}$\\Brandom_{t_i}^+,\\ i\\leq k-1$, and $\\Brandomtilde_t^+, t\\geq 0$. \\\\\n% \\hspace*{16mm}$\\Brandom_{t_i}^+,\\ i\\leq k-1$, and $\\accolade{\\Brandom_{t_i}^+}_{i=0}^k$ and $\\{\\Brandomtilde_{t_i}^+\\}_{i=0}^k$ are independent. \\\\\n% \\hspace*{10mm}Otherwise $\\Urandom_{t_k,\\ntp}:=\\Urandomtilde_{t_k,\\ntp}$ and $\\BrandomSet_{t_k}^+:=\\BrandomtildeSet_{t_k}^+$. \\\\\n\\hspace*{10mm}Define $\\Ukp:=\\tilde{\\Ub}_{t_k,\\ntp}$ and $\\pollD_k:=\\tilde{\\mathbb{D}}^p_{t_k}$.\\\\\n\\textbf{[{2}] Poll}\\\\\n%\\hspace*{10mm}Select a PSS $\\pollD_k\\subset\\rp$ such that $\\pollDrandom_k$ is independent of $\\pollDrandom_0,\\dots,\\pollDrandom_{k-1}$ when $k\\geq 1$. \\\\\n\\hspace*{10mm}Generate a set of poll points $\\mathcal{P}_k:=\\{\\xk+\\dk\\ub: \\ub\\in\\mathbb{U}_k^n \\}$; where $\\mathbb{U}_k^n:=\\{\\Ukp\\di^p:\\di^p\\in \\pollD_k\\}$, {\\bbl and}\\\\\n\\hspace*{10mm}{\\bbbl $\\pollD_k$ is generated independently of $\\Ukp$}.\\\\\n%\\hspace*{8mm}{}\\\\ \n\\hspace*{10mm}Obtain estimates\\footnotemark $\\fuk$ and $\\fok$ of $f$, respectively, at ${\\xk+\\dk\\ub}\\in\\mathcal{P}_k$ and $\\xk$.\\\\\n%\\hspace*{10mm}function evaluations.\\\\\n%\\hspace*{10mm}\\textbf{{Success}}\\\\\n\\hspace*{10mm}If $\\Pkfeas:=\\Pk\\cap\\bcX\\neq\\emptyset$ and $\\fuk -\\fok \\leq -\\gamma \\ef\\dk^2$ for some ${\\xk+\\dk\\ub}\\in\\Pkfeas$ (\\textbf{successful})\\\\ \n\\hspace*{16mm}set $\\xkun\\gets \\xk+\\dk\\ub$,  $\\dkun\\gets \\min\\{\\tau^{-1}\\dk,\\dmax\\}$, and $\\ell_{k+1}\\gets \\ell_k-1$. \\\\\n%\\hspace*{10mm}\\textbf{{Failure}}\\\\\n\\hspace*{10mm}Otherwise (\\textbf{unsuccessful}) set $\\xkun\\gets \\xk$, $\\delta_{k+1}\\gets\\tau\\dk$, and $\\ell_{k+1}\\gets \\ell_k+1$. \\\\\n\\textbf{[3] Matrix index update}\\\\\n\\hspace*{10mm}If $\\ell_{k+1} \\geq  \\max_{i\\leq k} \\ell_i$, set $t_{k+1}\\gets \\ell_{k+1}$.   \\\\\n\\hspace*{10mm}Otherwise $t_{k+1}\\gets 1+\\max_{i\\leq k} t_i$.\\\\\n\\textbf{[4] Termination}\\\\\n\\hspace*{10mm}If no termination criterion is met, set $k\\gets k+1$ and go to \\textbf{[1]}.\\\\\n%\\hspace*{10mm}\\\\\n\\hspace*{10mm}Otherwise stop.\n\\end{algorithm}","sha256_utf8":"0c634a324393ba942c705323d86d4b6f04e7465b1c6d2d5cc73e3b87384e551f"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"函数估计精度","source_units":[{"start":97149,"end":97574,"start_line":1192,"end_line":1197,"raw_tex":"\\begin{definition}\\label{epsAccDef0}\nGiven $\\ef>0$, $\\xk\\in\\rn$, and $\\ub:=\\Ukp\\di^p=\\sqrt{\\frac{p}{n}}\\Qk\\di^p\\in\\rn$ with $\\di^p\\in\\pollD_k$, $\\fok$ and $\\fuk$ are called $\\ef$-accurate estimates of $f(\\xk)$ and $f(\\xk+\\dk\\ub)$, respectively, for a given $\\dk$ if\n\\begin{equation}\\label{epsAccDef1}\n\\abs{\\fok-f(\\xk)}\\leq \\ef\\dk^2\\qquad \\mbox{and}\\qquad \\abs{\\fuk-f(\\xk+\\dk\\ub)}\\leq \\ef\\dk^2.\n\\end{equation}\n\\end{definition}","sha256_utf8":"137d73020f9a645ff98acb39b939944ba643ad5055b504bf5928759389647dc2"}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","title":"Haar子空间与独立轮询集","source_units":[{"start":119750,"end":120323,"start_line":1353,"end_line":1355,"raw_tex":"\\begin{assumption}\\label{QkHaarAssumpt}{\\bl (Motivated by Theorem~\\ref{JLforHaarTheor})}\nFor all $k\\geq 0$, the independent random matrices  $\\Drandom_k\\in\\R^{p\\times m}$, with $m\\in {\\bl \\intbracket{p+1,2p}}$, and $\\Ukprandom\\in\\rnp$ used for subspace polling in Algorithm~\\ref{algoStoDARS} are such that $\\pollDrandom_k\\subset\\spOne$, $\\Urandom_k:=\\left[\\Urandom_{k,\\ntp}\\ \\, \\Urandom_{k,\\ntnmp}\\right]\\in \\mathbb{O}(n)$ is Haar-distributed, and $\\QkRandom:=\\sqrt{\\frac{n}{p}}\\Ukprandom$ is $WAM(\\alpha_Q,\\beta_Q)$ for some $\\alpha_Q, \\beta_Q\\in(0,1/2)$.\n\\end{assumption}","sha256_utf8":"774db7e8979e9250a2bbdf42784ba4710539d68d4007d85aecbf8012c846a6c5"}]},{"id":"S016","semantic_type":"notation","kind":"notation","title":"历史与当前轮询信息","source_units":[{"start":120327,"end":121716,"start_line":1359,"end_line":1364,"raw_tex":"Next, $(\\QkRandom)_{ij}$, with $(i,j)\\in\\mathbb{I}_{\\ntp}:=\\intbracket{1,n}\\times\\intbracket{1,p}$, denote the entries of $\\QkRandom$. Likewise, $(\\Drandom_k)_{i'j'}$, with $(i',j')\\in\\mathbb{I}_{p,m}$, denote the entries of $\\pollDrandom_k$. The conditioning on the past is formalized by using the filtrations $\\accoladekinN{\\falgebra}$ and $\\accoladekinN{\\fqdalgebra}$ defined by\n\\begin{eqnarray*}\n\\falgebra&:=&\\sigma\\left({\\ubar{f}}_{\\ell}({\\bm{0}}), {\\ubar{f}}_{\\ell}(\\urandom),(\\Qrandom_\\ell)_{ij}, (\\Drandom_\\ell)_{i'j'}\\ \\mbox{with}\\ \\ell\\in\\intbracket{0,k-1}, (i,j)\\in\\mathbb{I}_{\\ntp}\\ \\mbox{and}\\ (i',j')\\in\\mathbb{I}_{p,m}\\right)\\\\\n\\fqdalgebra&:=&\\sigma\\left({\\ubar{f}}_{\\ell}({\\bm{0}}), {\\ubar{f}}_{\\ell}(\\urandom),(\\Qrandom_\\ell)_{ij}, (\\Drandom_\\ell)_{i'j'},(\\QkRandom)_{ij}, (\\Drandom_k)_{i'j'} \\ \\mbox{with}\\ \\ell\\in\\intbracket{0,k-1}, (i,j)\\in\\mathbb{I}_{\\ntp}\\ \\mbox{and}\\ (i',j')\\in\\mathbb{I}_{p,m}\\right),\n\\end{eqnarray*}\nwhere for {\\it completeness}~\\cite{billingsley1995ThirdEdition}, $\\F_{-1}:=\\sigma(\\x_0)$. The idea behind the above definitions is explained in~\\cite[Remark~2.1]{DzaWildSub2022}. By construction, $\\QkRandom$ and $\\Drandom_k$ are $\\fqdalgebra$-measurable so that $\\E{\\QkRandom|\\fqdalgebra}=\\QkRandom$ almost surely and $\\E{\\Drandom_k|\\fqdalgebra}=\\Drandom_k$ almost surely. The sufficient accuracy of the random estimates is formalized as follows.\n","sha256_utf8":"dba1c2093fdbc20a0950edca4c3bf41019b67c46bef2a6f086e3252806fedbd6"}]},{"id":"S017","semantic_type":"definition","kind":"definition","title":"条件概率准确估计","source_units":[{"start":121717,"end":122505,"start_line":1366,"end_line":1373,"raw_tex":"\\begin{definition}\nA sequence $\\accolade{\\Fok,\\Fuk}$ of random estimates is called $\\beta_f$-probabilistically $\\ef$-accurate with respect to the corresponding sequence $\\accolade{\\Xk,\\Dk,\\QkRandom,\\Drandom_k}$ if the events\n\\begin{eqnarray*}\nJ_k&:=&\\left\\lbrace \\mbox{Given}\\ \\QkRandom, \\Drandom_k\\ \\mbox{and}\\ \\Xk, \\Fok\\ \\mbox{and}\\ \\Fuk\\ \\mbox{are}\\ \\ef\\mbox{-accurate}\\ \\mbox{estimates of}\\ f(\\Xk)\\ \\mbox{and} \\right.\\\\\n& & \\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\left. f(\\Xk+\\Dk\\urandom),\\ \\mbox{respectively, for}\\ \\Dk \\right\\rbrace\n\\end{eqnarray*}\nsatisfy the condition $\\prob{J_k|\\fqdalgebra}=\\E{\\ijk|\\fqdalgebra}\\geq \\beta_f$ almost surely. The estimates $\\Fok$ and $\\Fuk$ are called ``good'' if $\\ijk=1$. Otherwise, they are called ``bad.''\n\\end{definition}","sha256_utf8":"f0605b0601971da79d04af66212e7bcda6b3dd054203abb241192106cea752ed"}]},{"id":"S018","semantic_type":"assumption","kind":"assumption","title":"估计概率与方差控制","source_units":[{"start":124024,"end":124825,"start_line":1381,"end_line":1388,"raw_tex":"\\begin{assumption}\\label{efkfAssump}\nFor fixed $\\ef>0$, {\\bl $\\gamma>2$, $\\nu,\\tau\\in (0,1)$ satisfying $\\frac{\\nu}{1-\\nu}\\geq \\frac{2(\\tau^{-2}-1)}{\\gamma-2}$} and $\\beta_f\\in ({\\bl 1/2},1)$ {\\bl satisfying  $\\frac{\\beta_f}{1-\\beta_f}\\geq \\frac{4\\nu}{(1-\\nu)(1-\\tau^2)}$}, the following hold for the random quantities generated by Algorithm~\\ref{algoStoDARS}:\n\\begin{itemize}\n\\item[(i)] The sequence $\\accolade{\\Fok,\\Fuk}$ is $\\beta_f$-probabilistically $\\ef$-accurate.\n\\item[(ii)] The sequence $\\accolade{\\Fok,\\Fuk}$ satisfies the following $\\kappa_f$-variance conditions, each of which holds almost surely:\n\\[\\E{\\abs{\\Fok-f(\\Xk)}^2|\\fqdalgebra}\\leq \\ef^2(1-\\beta_f)\\Dk^4\\quad\\mbox{and}\\quad \\E{\\abs{\\Fuk-f(\\Xk+\\Dk\\urandom)}^2|\\fqdalgebra}\\leq \\ef^2(1-\\beta_f)\\Dk^4.\\]\n\\end{itemize}\n\\end{assumption}","sha256_utf8":"9a8538145af9dc4991079e3c4b94480e178f4e4717514fdcb60210e4c00a7c3e"}]},{"id":"S019","semantic_type":"assumption","kind":"assumption","title":"目标函数下界","source_units":[{"start":127130,"end":127427,"start_line":1408,"end_line":1410,"raw_tex":"\\begin{assumption}\\label{boundedFassumpt}{\\bl (Inspired by~\\cite[Assumption~10.5]{CoScVibook},~\\cite[Theorems~3.4 and Section~3.8]{KoLeTo03a})}\nThe function $f$ is bounded from below; that is, $-\\infty<f_{\\min}\\leq f(\\x)$ for some $f_{\\min}\\in\\R$ and all $\\x\\in{\\bl \\mathcal{X}}$.\n\\end{assumption}","sha256_utf8":"63395adb11214de3a2ea79d73f88ce4104b6234dd0a285fa286eb4672fe4f8cc"}]},{"id":"S020","semantic_type":"definition","kind":"definition","title":"离散停止时间","source_units":[{"start":129495,"end":129776,"start_line":1442,"end_line":1444,"raw_tex":"\\begin{definition}\\cite[Definition~1]{blanchet2016convergence}\n$\\Tepsilon$ is called a \\emph{stopping time} with respect to the discrete time process $\\accolade{\\X_k}_{k\\in\\N}$ if, for all $k\\in\\N$, $\\accolade{\\Tepsilon=k}\\in \\sigma\\left(\\X_1,\\dots,\\X_k \\right)$. \n\\end{definition}","sha256_utf8":"cb0daab4ea231180dbb371001d7fdb0d2758a4f632726c59c99669a2da37907c"}]},{"id":"S021","semantic_type":"problem_setting","kind":"problem_setting","title":"更新报酬过程与偏置随机游走","source_units":[{"start":129778,"end":130702,"start_line":1446,"end_line":1452,"raw_tex":"Consider the discrete time process $\\accolade{\\left(\\Phik,\\Dk\\right)}_{k\\in\\N}$, with $\\Phik, \\Dk\\in [0,\\infty)$. On the same probability space as $\\accolade{\\left(\\Phik,\\Dk\\right)}_{k\\in\\N}$, let us introduce the biased random walk process $\\accolade{\\Wtildek}_{k\\in\\N}$ satisfying\n\\begin{equation}\\label{dynamicsWk}\n\\prob{\\Wtildekun=1|\\mathcal{H}_k}=q\\qquad \\mbox{ and }\\qquad \\prob{\\Wtildekun=-1|\\mathcal{H}_k}=1-q,\n\\end{equation}\nwhere $\\mathcal{H}_k=\\sigma\\left( \\left(\\PhiZero,\\Dzero,\\WtildeZero\\right),\\dots,\\left(\\Phik,\\Dk,\\Wtildek\\right) \\right)$ with ${\\WtildeZero=1}$, and  $q\\in (1/2, 1)$.\n\nLet $\\accolade{\\Tepsilon}_{\\epsilon>0}$ be a family of stopping times with respect to $\\accoladekinN{\\mathcal{H}_k}$, parameterized by $\\epsilon$. In order to derive in~\\cite{blanchet2016convergence,dzahini2020expected,DzaWildSub2022,paquette2018stochastic} an upper bound on $\\E{\\Tepsilon}$,  the following are assumed.\n","sha256_utf8":"47ef16cf5de039893d73a3625f8b99ee5fda2dfd17c0b58b62b40e284f7e34ce"}]},{"id":"S022","semantic_type":"assumption","kind":"assumption","title":"更新报酬过程三项条件","source_units":[{"start":130703,"end":131626,"start_line":1454,"end_line":1466,"raw_tex":"\\begin{assumption}{\\bl \\cite{blanchet2016convergence}}\\label{assumptionRenewal}\nThe discrete time process $\\accoladekinN{\\left(\\Phik,\\Dk,\\Wtildek\\right)}$ satisfies the following:\n\\begin{itemize}\n\\item[(i)] There exist $\\lambda>0$ and $\\dmax=\\delta_0 e^{\\lambda j_{\\max}}$ such that $\\Dk\\leq \\dmax$ for all $k$, where $j_{\\max}\\in\\Z$.\n\\item[(ii)] For some $j_{\\epsilon}\\in\\Z\\cap (-\\infty, 0]$, there exists $\\depsilon=\\delta_0 e^{\\lambda j_{\\epsilon}}$ such that for all $k$\n\\begin{equation}\\label{dynamicsRenewal} \n\\itksup\\Dkun\\geq \\itksup\\min\\accolade{\\Dk e^{\\lambda\\Wtildekun}, \\depsilon}, \n\\end{equation}\nwhere $\\Wtildekun$ satisfies~\\eqref{dynamicsWk}. \n\\item[(iii)] There exist a nondecreasing function $h:(0,\\infty)\\to (0,\\infty)$ and a constant $\\varrho>0$ such that for all $k$\n\\[\\E{\\Phikun-\\Phik|\\mathcal{H}_k}\\itksup\\leq-\\varrho h\\left(\\Dk\\right)\\itksup\\quad\\mbox{almost surely.}\\]\n\\end{itemize}\n\\end{assumption}","sha256_utf8":"2047d97bf9540e5739045f85492a2942b54975f841a164330783cde02299bf9c"}]},{"id":"S023","semantic_type":"definition","kind":"definition","title":"下降方向事件与梯度停止时间","source_units":[{"start":132251,"end":132726,"start_line":1476,"end_line":1479,"raw_tex":"Consider the discrete time process $\\accoladekinN{\\left(\\Phik,\\Dk,\\Wtildek\\right)}$ defined by the random function~$\\Phik$ introduced in Theorem~\\ref{zerothOrdTheor}, the stepsize~$\\Dk$, and the random variables $\\Wtildek:=2\\left(\\iak\\ijk-\\frac{1}{2}\\right)$, where \n\\begin{equation}\\label{AkEventDef}\nA_k:=\\accolade{\\alpha_Q\\kappa(\\pollDrandom_k)\\normii{\\nabla f(\\Xk)}\\leq -\\sqrt{\\frac{n}{p}}\\nabla f(\\Xk)^{\\top}\\left(\\Ukprandom{\\ubar{\\hat{\\di}}^p_k}\\right)}.\n\\end{equation}","sha256_utf8":"671f4d6eb243652b308f47850b39d2c278815429e5a0a4fb9a3fce1f33b6d7af"},{"start":132727,"end":132978,"start_line":1480,"end_line":1480,"raw_tex":"For an arbitrary fixed $\\epsilon\\in (0,1)$, consider the random number $\\Tepsilon:=\\inf\\accolade{k\\in\\N:\\normii{\\nabla f(\\Xk)}\\leq \\eps}$ of iterations required by Algorithm~\\ref{algoStoDARS} to drive the norm of the gradient of~$f$ below~$\\epsilon$.\n","sha256_utf8":"020710503630b880b4bb1ae363c136595a45fca9fa14d0ba2862afff0a01bc67"}]},{"id":"S024","semantic_type":"assumption","kind":"assumption","title":"余弦测度统一下界","source_units":[{"start":136642,"end":137053,"start_line":1504,"end_line":1507,"raw_tex":"\\begin{assumption}\\label{kapminAssump}{\\bl (\\cite[Assumption~(3.12)]{KoLeTo03a}  and~\\cite[Assumption~1]{Vicente2013})}\n%To SW: above it's really Assumption~\"(3.12)\"; it's actually an equation, not a standalone assumption\nThere exists $\\kappa_{\\min}>0$ such that the realizations of the PSSs used by Algorithm~\\ref{algoStoDARS} for subspace polling satisfy $\\kappa(\\pollD_k)\\geq \\kappa_{\\min}$.\n\\end{assumption}","sha256_utf8":"5e1269be066cd3ba9719c5de695fa88c6d56484cb8528f69fcb5b73af7ce9d88"}]},{"id":"S025","semantic_type":"assumption","kind":"assumption","title":"梯度Lipschitz连续","source_units":[{"start":137054,"end":137417,"start_line":1508,"end_line":1511,"raw_tex":"\\begin{assumption}\\label{gradLipAssump}{\\bl (Inspired by~\\cite[Theorem~2.8]{CoScVibook}, \\cite[Theorem~3.3]{KoLeTo03a}, and~\\cite[Theorem~1]{Vicente2013}).}\nThe gradient $\\nabla f$ of the objective function is $L_g$-Lipschitz continuous {\\bl in an open set containing all the points in the poll sets~$\\mathcal{P}_k$}.\n%, with $L_g>\\kappa_{\\min}$.\n\\end{assumption}","sha256_utf8":"7d2e93dbea55788eea6301f2ea93e03d8709dcfc688c9d92f0e93fc8f7f33761"}]},{"id":"S026","semantic_type":"assumption","kind":"assumption","title":"临界步长与离散尺度","source_units":[{"start":137419,"end":137725,"start_line":1513,"end_line":1516,"raw_tex":"In the following result inspired by~\\cite[Lemma~2]{dzahini2020expected}, $\\depsilon$ is a constant defined by \n\\begin{equation}\\label{depsEq}\n\\depsilon:=\\frac{\\epsilon}{\\xi}\\qquad\\mbox{with}\\qquad\\xi>\\kappa_{\\min}^{-1}\\alpha_Q^{-1}\\left(L_g\\sqrt{\\frac{n}{p}}+(\\gamma+2)\\ef\\right)=:\\hat{\\xi}.\n\\end{equation}","sha256_utf8":"0e520beb76d7184c654bdddd452210880e6f8b5a6529130e377c77c0f526bacd"},{"start":137726,"end":137978,"start_line":1517,"end_line":1517,"raw_tex":"Thus, as in~\\cite{dzahini2020expected} and~\\cite{blanchet2016convergence}, it can be assumed without any loss of generality that $\\depsilon=\\tau^{-i}\\delta_0$ for some integer $i\\leq 0$, so that $\\Dk=\\tau^{-i_k}\\depsilon$ for some integer $i_k\\leq 0$.\n","sha256_utf8":"f0fb65cb12e9f6f2ec0fa205f6d5425132ae50bcf70bd1b36ac1a36e2a3f93a0"}]},{"id":"S027","semantic_type":"lemma","kind":"lemma","title":"Haar矩阵元素矩","source_units":[{"start":69950,"end":70269,"start_line":983,"end_line":985,"raw_tex":"\\begin{lemma}\\label{HaarLemma}\\cite[Section~2.1]{MeckesElizHaar2019}\nLet $\\Urandom:=(\\UrandomScal_{ij})_{1\\leq i,j\\leq n}\\in \\mathbb{O}(n)$ be \\emph{Haar}-distributed. Then, the entries of $\\Urandom$ are identically distributed. Moreover, $\\E{\\UrandomScal_{11}}=0$ and $\\E{\\UrandomScal_{11}^2}=\\frac{1}{n}$.\n\\end{lemma}","sha256_utf8":"d443dc1dca0129ea29031271f66fb3f1c3e8670ba041c1413ba20f760f60760a"}]},{"id":"S028","semantic_type":"theorem","kind":"theorem","title":"球面Lévy集中界","source_units":[{"start":70731,"end":71186,"start_line":989,"end_line":992,"raw_tex":"\\begin{theorem}\\label{LeviTheorem}(\\cite[Theorem~5.1.4 and Exercise~5.1.12]{vershynin2018HDP})\nLet $F$ be an $L_F$-Lipschitz function on the unit sphere $\\mathbb{S}^{n-1}$, and let $\\xiRandomVec\\sim \\mathcal{U}(\\mathbb{S}^{n-1})$. Then there exists an absolute constant $\\eta>0$ such that for every $t\\geq 0$\n\\[\\prob{\\abs{F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}}\\geq t}\\leq 2\\exp{\\left(-\\frac{\\eta n t^2}{L_F^2}\\right)}. \\]\n\\end{theorem}","sha256_utf8":"49a5f5a30091bef61d4f6dffb7203765faa0c135516f3e4fef13210434698d36"}]},{"id":"S029","semantic_type":"theorem","kind":"theorem","title":"Haar投影的近等距概率","source_units":[{"start":73159,"end":73915,"start_line":1003,"end_line":1008,"raw_tex":"\\begin{theorem}\\label{JLforHaarTheor}(Inspired by~\\cite[Lemma~6.1]{MeckesElizHaar2019})\nLet $\\varepsilon_Q, \\beta_Q\\in (0,1)$ and $\\Urandom:=(\\UrandomScal_{ij})_{1\\leq i,j\\leq n}\\in \\mathbb{O}(n)$ be a \\emph{Haar}-distributed random matrix. Let $\\Urandom_{p\\times n}\\in \\rpn$ be the matrix consisting of the first $p$ rows of $\\Urandom$, and define $\\QrandomTranspose:=\\sqrt{\\frac{n}{p}}\\Urandom_{p\\times n}$. There exists an absolute constant $\\eta>0$ such that if $p\\geq  \\frac{4}{\\eta}\\varepsilon_Q^{-2}\\log\\left(\\frac{2}{\\beta_Q}\\right)$, then\n\\begin{equation}\\label{JLTEquat}\n\\prob{(1-\\varepsilon_Q)\\normii{\\x}\\leq \\normii{\\QrandomTranspose\\x}\\leq(1+\\varepsilon_Q)\\normii{\\x}}\\geq 1-\\beta_Q\\quad\\mbox{for all}\\ \\x\\in \\rn.\n\\end{equation} \n\\end{theorem}","sha256_utf8":"de3aba5dd48bdca96662316cfa5d8f0f56a52262bee582e611c4efef58388cb5"}]},{"id":"S030","semantic_type":"proof","kind":"proof","title":"Haar近等距完整证明","source_units":[{"start":73916,"end":73976,"start_line":1009,"end_line":1011,"raw_tex":"\\begin{proof}\n{\\bbl See Appendix~\\ref{theor22}.}\n\\end{proof}","sha256_utf8":"8538626f0bbc3bbea976bcc748e6d429255bad3f1531346294fcbbee65185536"},{"start":183112,"end":188736,"start_line":1838,"end_line":1877,"raw_tex":"\\begin{proof}\nWithout any loss of generality, it is sufficient to prove the result for $\\x\\in \\mathbb{S}^{n-1}$, which is assumed throughout. Let $\\bm{e_1}$ be the first standard basis vector of $\\rn$. \n%By the translation invariance of Haar measure, \nFor any $\\x\\in \\mathbb{S}^{n-1}$ arbitrary, $\\Urandom_{p\\times n}\\x \\equald \\Urandom_{p\\times n} \\bm{e_1}=(\\UrandomScal_{11},\\UrandomScal_{21},\\dots, \\UrandomScal_{p1})^{\\top}$,\n%. On the other hand, it follows from the definition of Haar measure that the first column of $\\Urandom$ satisfies \nwith $(\\UrandomScal_{11}, \\UrandomScal_{21},\\dots, \\UrandomScal_{n1})^{\\top}\\sim \\mathcal{U}(\\mathbb{S}^{n-1})$, hence $\\Urandom_{p\\times n}\\x \\equald \\left(\\xiRandom_1,\\xiRandom_2,\\dots, \\xiRandom_p\\right)^{\\top}$, where $\\xiRandomVec:=\\left(\\xiRandom_1,\\xiRandom_2,\\dots, \\xiRandom_n\\right)^{\\top}\\sim \\mathcal{U}(\\mathbb{S}^{n-1})$. \n%Consider the function $F:\\mathbb{S}^{n-1}\\to \\R$ defined by \nLet \\scalebox{0.88}{$F(x_1,x_2,\\dots,x_n):=\\sqrt{\\frac{n}{p}}\\normii{(x_1,x_2,\\dots,x_p)}$}, so that \\scalebox{0.88}{$F\\left(\\xiRandomVec\\right)=\\sqrt{\\frac{n}{p}}\\normii{\\left(\\xiRandom_1,\\xiRandom_2,\\dots, \\xiRandom_p\\right)}\\equald \\sqrt{\\frac{n}{p}}\\normii{\\Urandom_{p\\times n}\\x}=\\normii{\\QrandomTranspose\\x}$}. \nThen, for all $\\x,\\bm{y}\\in \\mathbb{S}^{n-1}$, $\\abs{F(\\x)-F(\\bm{y})}\\leq \\sqrt{\\frac{n}{p}}\\normii{\\x-\\bm{y}}$,\n% \\begin{equation*}\n% \\begin{split}\n% %\\abs{F(\\x)-F(\\bm{y})}&=\\sqrt{\\frac{n}{p}}\\abs{\\normii{(x_1,x_2,\\dots,x_p)}-\\normii{(y_1,y_2,\\dots,y_p)}}\\\\\n% \\abs{F(\\x)-F(\\bm{y})}&\\leq \\sqrt{\\frac{n}{p}}\\normii{(x_1-y_1),(x_2-y_2),\\dots,(x_p-y_p)}\\leq \\sqrt{\\frac{n}{p}}\\normii{\\x-\\bm{y}},\n% \\end{split}\n% \\end{equation*}\nwhich shows that $F$ is $\\sqrt{\\frac{n}{p}}$-Lipschitz on $\\mathbb{S}^{n-1}$. Thus, it follows from Theorem~\\ref{LeviTheorem} that there exists $\\eta>0$ such that for all $\\varepsilon_Q\\in (0,1)$, \n{\\footnotesize{\n\\begin{equation}\\label{LeviEq1}\n\\prob{\\abs{F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}}\\geq \\frac{\\varepsilon_Q}{2}}\\leq 2\\exp{\\left(-\\frac{\\eta n (\\varepsilon_Q/2)^2}{(\\sqrt{n/p})^2}\\right)}=2e^{-\\frac{\\eta p\\varepsilon_Q^2}{4}}.\n\\end{equation}\n}}\n%On the other hand, it follows from Lemma~\\ref{HaarLemma} that $\\E{(F\\left(\\xiRandomVec\\right))^2}=1$, which implies $\\var{F\\left(\\xiRandomVec\\right)}+(\\E{F\\left(\\xiRandomVec\\right)})^2=1$. \nMoreover, it follows from the integral identity formula that \n{\\footnotesize{\n\\begin{equation}\\label{levEq2}\n\\var{F\\left(\\xiRandomVec\\right)}=\\E{\\left(F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}\\right)^2}=\\int_0^{+\\infty}\\prob{\\abs{F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}}^2\\geq t} dt\\leq\\int_0^{+\\infty} 2e^{-\\eta p t} dt=\\frac{2}{\\eta p},\n\\end{equation}\n}}\nwhere the inequality follows from Theorem~\\ref{LeviTheorem}. \n%It follows from~\\eqref{levEq2} and the equality $\\E{(F\\left(\\xiRandomVec\\right))^2}=\\var{F\\left(\\xiRandomVec\\right)}+\\left(\\E{F\\left(\\xiRandomVec\\right)}\\right)^2=1$ (thanks to Lemma~\\ref{HaarLemma}) that $\\sqrt{1-\\frac{2}{\\eta p}}\\leq \\E{F\\left(\\xiRandomVec\\right)}\\leq 1$. \nFrom~\\eqref{levEq2} and Lemma~\\ref{HaarLemma}, we obtain $\\sqrt{1-\\frac{2}{\\eta p}}\\leq \\E{F\\left(\\xiRandomVec\\right)}\\leq 1$.\nAssume that $p$ is chosen so that $1-\\frac{\\varepsilon_Q}{2}\\leq\\sqrt{1-\\frac{2}{\\eta p}}$ and $2e^{-\\frac{\\eta p\\varepsilon_Q^2}{4}}\\leq \\beta_Q$. Then $\\abs{F\\left(\\xiRandomVec\\right)-1}<\\varepsilon_Q$ since $1-\\frac{\\varepsilon_Q}{2}\\leq \\E{F\\left(\\xiRandomVec\\right)}\\leq 1$.\n%Since $1-\\frac{\\varepsilon_Q}{2}\\leq \\E{F\\left(\\xiRandomVec\\right)}\\leq 1$, then the inequality $\\abs{F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}}<\\frac{\\varepsilon_Q}{2}$ implies that $\\abs{F\\left(\\xiRandomVec\\right)-1}<\\varepsilon_Q$. \nConsequently, for all $\\x\\in \\mathbb{S}^{n-1}$, it follows from~\\eqref{LeviEq1} that\n\\[\\prob{\\abs{\\normii{\\QrandomTranspose\\x}-1}\\geq \\varepsilon_Q}=\\prob{\\abs{F\\left(\\xiRandomVec\\right)-1}\\geq \\varepsilon_Q}\\leq\\prob{\\abs{F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}}\\geq\\frac{\\varepsilon_Q}{2}}\\leq2e^{-\\frac{\\eta p\\varepsilon_Q^2}{4}}\\leq \\beta_Q,\\]\nwhere the equality is due to the equality in distribution $F\\left(\\xiRandomVec\\right)\\equald \\normii{\\QrandomTranspose\\x}$ as explained above. Equivalently, it holds that $\\prob{1-\\varepsilon_Q\\leq \\normii{\\QrandomTranspose\\x}\\leq 1+\\varepsilon_Q}\\geq 1-\\beta_Q$ for all $\\x\\in \\mathbb{S}^{n-1}$. It remains to prove that choosing $p\\geq \\max\\accolade{\\frac{8}{3\\eta}\\varepsilon_Q^{-2}, \\frac{4}{\\eta}\\varepsilon_Q^{-2}\\log\\left(\\frac{2}{\\beta_Q}\\right)}=\\frac{4}{\\eta}\\varepsilon_Q^{-2}\\log\\left(\\frac{2}{\\beta_Q}\\right)$ ensures $1-\\frac{\\varepsilon_Q}{2}\\leq\\sqrt{1-\\frac{2}{\\eta p}}$ and $2e^{-\\frac{\\eta p\\varepsilon_Q^2}{4}}\\leq \\beta_Q$ as assumed above, where the latter equality trivially holds for any $\\varepsilon_Q,\\beta_Q\\in (0,1)$. We first observe that $2e^{-\\frac{\\eta p\\varepsilon_Q^2}{4}}\\leq \\beta_Q\\iff p\\geq \\frac{4}{\\eta}\\varepsilon_Q^{-2}\\log\\left(\\frac{2}{\\beta_Q}\\right)$. On the other hand, $\\left(1-\\frac{\\varepsilon_Q}{2}\\right)^2\\leq 1-\\frac{2}{\\eta p}$ if $p\\geq \\frac{2}{\\eta\\left(\\varepsilon_Q-\\frac{\\varepsilon_Q^2}{4}\\right)}$.\n%, which completes the proof. \nSince $\\varepsilon_Q\\in (0,1)$, then $\\varepsilon_Q-\\frac{\\varepsilon_Q^2}{4}>\\frac{3}{4}\\varepsilon_Q^2$, which implies that $\\frac{8}{3\\eta}\\varepsilon_Q^{-2}\\geq \\frac{2}{\\eta\\left(\\varepsilon_Q-\\frac{\\varepsilon_Q^2}{4}\\right)}$. Hence,  choosing $p\\geq \\frac{8}{3\\eta}\\varepsilon_Q^{-2}$ ensures $p\\geq \\frac{2}{\\eta\\left(\\varepsilon_Q-\\frac{\\varepsilon_Q^2}{4}\\right)}$.\n%, and the proof is completed.\n\\end{proof}","sha256_utf8":"9352b4119b509fa643f17357d85d7f1e449bf4407cc52f8b780348d7b141baf6"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"转至附录完整证明","source_units":[{"start":73916,"end":73976,"start_line":1009,"end_line":1011,"raw_tex":"\\begin{proof}\n{\\bbl See Appendix~\\ref{theor22}.}\n\\end{proof}","sha256_utf8":"8538626f0bbc3bbea976bcc748e6d429255bad3f1531346294fcbbee65185536"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"归约到球面均匀向量","source_units":[{"start":183126,"end":183995,"start_line":1839,"end_line":1843,"raw_tex":"Without any loss of generality, it is sufficient to prove the result for $\\x\\in \\mathbb{S}^{n-1}$, which is assumed throughout. Let $\\bm{e_1}$ be the first standard basis vector of $\\rn$. \n%By the translation invariance of Haar measure, \nFor any $\\x\\in \\mathbb{S}^{n-1}$ arbitrary, $\\Urandom_{p\\times n}\\x \\equald \\Urandom_{p\\times n} \\bm{e_1}=(\\UrandomScal_{11},\\UrandomScal_{21},\\dots, \\UrandomScal_{p1})^{\\top}$,\n%. On the other hand, it follows from the definition of Haar measure that the first column of $\\Urandom$ satisfies \nwith $(\\UrandomScal_{11}, \\UrandomScal_{21},\\dots, \\UrandomScal_{n1})^{\\top}\\sim \\mathcal{U}(\\mathbb{S}^{n-1})$, hence $\\Urandom_{p\\times n}\\x \\equald \\left(\\xiRandom_1,\\xiRandom_2,\\dots, \\xiRandom_p\\right)^{\\top}$, where $\\xiRandomVec:=\\left(\\xiRandom_1,\\xiRandom_2,\\dots, \\xiRandom_n\\right)^{\\top}\\sim \\mathcal{U}(\\mathbb{S}^{n-1})$. \n","sha256_utf8":"a0e20aab7622c8b8cd7b7f8e0246f53ef841f43cca73d5e2310f63239a8e91d7"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"构造Lipschitz范数并集中","source_units":[{"start":184057,"end":185277,"start_line":1845,"end_line":1858,"raw_tex":"Let \\scalebox{0.88}{$F(x_1,x_2,\\dots,x_n):=\\sqrt{\\frac{n}{p}}\\normii{(x_1,x_2,\\dots,x_p)}$}, so that \\scalebox{0.88}{$F\\left(\\xiRandomVec\\right)=\\sqrt{\\frac{n}{p}}\\normii{\\left(\\xiRandom_1,\\xiRandom_2,\\dots, \\xiRandom_p\\right)}\\equald \\sqrt{\\frac{n}{p}}\\normii{\\Urandom_{p\\times n}\\x}=\\normii{\\QrandomTranspose\\x}$}. \nThen, for all $\\x,\\bm{y}\\in \\mathbb{S}^{n-1}$, $\\abs{F(\\x)-F(\\bm{y})}\\leq \\sqrt{\\frac{n}{p}}\\normii{\\x-\\bm{y}}$,\n% \\begin{equation*}\n% \\begin{split}\n% %\\abs{F(\\x)-F(\\bm{y})}&=\\sqrt{\\frac{n}{p}}\\abs{\\normii{(x_1,x_2,\\dots,x_p)}-\\normii{(y_1,y_2,\\dots,y_p)}}\\\\\n% \\abs{F(\\x)-F(\\bm{y})}&\\leq \\sqrt{\\frac{n}{p}}\\normii{(x_1-y_1),(x_2-y_2),\\dots,(x_p-y_p)}\\leq \\sqrt{\\frac{n}{p}}\\normii{\\x-\\bm{y}},\n% \\end{split}\n% \\end{equation*}\nwhich shows that $F$ is $\\sqrt{\\frac{n}{p}}$-Lipschitz on $\\mathbb{S}^{n-1}$. Thus, it follows from Theorem~\\ref{LeviTheorem} that there exists $\\eta>0$ such that for all $\\varepsilon_Q\\in (0,1)$, \n{\\footnotesize{\n\\begin{equation}\\label{LeviEq1}\n\\prob{\\abs{F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}}\\geq \\frac{\\varepsilon_Q}{2}}\\leq 2\\exp{\\left(-\\frac{\\eta n (\\varepsilon_Q/2)^2}{(\\sqrt{n/p})^2}\\right)}=2e^{-\\frac{\\eta p\\varepsilon_Q^2}{4}}.\n\\end{equation}\n}}\n","sha256_utf8":"133f42f3bea66de26e0e50e3978f61d4a2fb5224aee9150095b55834ba777f67"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"尾积分控制方差与均值","source_units":[{"start":185468,"end":186325,"start_line":1860,"end_line":1868,"raw_tex":"Moreover, it follows from the integral identity formula that \n{\\footnotesize{\n\\begin{equation}\\label{levEq2}\n\\var{F\\left(\\xiRandomVec\\right)}=\\E{\\left(F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}\\right)^2}=\\int_0^{+\\infty}\\prob{\\abs{F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}}^2\\geq t} dt\\leq\\int_0^{+\\infty} 2e^{-\\eta p t} dt=\\frac{2}{\\eta p},\n\\end{equation}\n}}\nwhere the inequality follows from Theorem~\\ref{LeviTheorem}. \n%It follows from~\\eqref{levEq2} and the equality $\\E{(F\\left(\\xiRandomVec\\right))^2}=\\var{F\\left(\\xiRandomVec\\right)}+\\left(\\E{F\\left(\\xiRandomVec\\right)}\\right)^2=1$ (thanks to Lemma~\\ref{HaarLemma}) that $\\sqrt{1-\\frac{2}{\\eta p}}\\leq \\E{F\\left(\\xiRandomVec\\right)}\\leq 1$. \nFrom~\\eqref{levEq2} and Lemma~\\ref{HaarLemma}, we obtain $\\sqrt{1-\\frac{2}{\\eta p}}\\leq \\E{F\\left(\\xiRandomVec\\right)}\\leq 1$.\n","sha256_utf8":"7ec5d4fad34e4e8360a6bdc20d2d91aa2e863488d022e64e619f3147d84cf203"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"得到近等距事件并确定维数条件","source_units":[{"start":186325,"end":188286,"start_line":1869,"end_line":1873,"raw_tex":"Assume that $p$ is chosen so that $1-\\frac{\\varepsilon_Q}{2}\\leq\\sqrt{1-\\frac{2}{\\eta p}}$ and $2e^{-\\frac{\\eta p\\varepsilon_Q^2}{4}}\\leq \\beta_Q$. Then $\\abs{F\\left(\\xiRandomVec\\right)-1}<\\varepsilon_Q$ since $1-\\frac{\\varepsilon_Q}{2}\\leq \\E{F\\left(\\xiRandomVec\\right)}\\leq 1$.\n%Since $1-\\frac{\\varepsilon_Q}{2}\\leq \\E{F\\left(\\xiRandomVec\\right)}\\leq 1$, then the inequality $\\abs{F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}}<\\frac{\\varepsilon_Q}{2}$ implies that $\\abs{F\\left(\\xiRandomVec\\right)-1}<\\varepsilon_Q$. \nConsequently, for all $\\x\\in \\mathbb{S}^{n-1}$, it follows from~\\eqref{LeviEq1} that\n\\[\\prob{\\abs{\\normii{\\QrandomTranspose\\x}-1}\\geq \\varepsilon_Q}=\\prob{\\abs{F\\left(\\xiRandomVec\\right)-1}\\geq \\varepsilon_Q}\\leq\\prob{\\abs{F\\left(\\xiRandomVec\\right)-\\E{F\\left(\\xiRandomVec\\right)}}\\geq\\frac{\\varepsilon_Q}{2}}\\leq2e^{-\\frac{\\eta p\\varepsilon_Q^2}{4}}\\leq \\beta_Q,\\]\nwhere the equality is due to the equality in distribution $F\\left(\\xiRandomVec\\right)\\equald \\normii{\\QrandomTranspose\\x}$ as explained above. Equivalently, it holds that $\\prob{1-\\varepsilon_Q\\leq \\normii{\\QrandomTranspose\\x}\\leq 1+\\varepsilon_Q}\\geq 1-\\beta_Q$ for all $\\x\\in \\mathbb{S}^{n-1}$. It remains to prove that choosing $p\\geq \\max\\accolade{\\frac{8}{3\\eta}\\varepsilon_Q^{-2}, \\frac{4}{\\eta}\\varepsilon_Q^{-2}\\log\\left(\\frac{2}{\\beta_Q}\\right)}=\\frac{4}{\\eta}\\varepsilon_Q^{-2}\\log\\left(\\frac{2}{\\beta_Q}\\right)$ ensures $1-\\frac{\\varepsilon_Q}{2}\\leq\\sqrt{1-\\frac{2}{\\eta p}}$ and $2e^{-\\frac{\\eta p\\varepsilon_Q^2}{4}}\\leq \\beta_Q$ as assumed above, where the latter equality trivially holds for any $\\varepsilon_Q,\\beta_Q\\in (0,1)$. We first observe that $2e^{-\\frac{\\eta p\\varepsilon_Q^2}{4}}\\leq \\beta_Q\\iff p\\geq \\frac{4}{\\eta}\\varepsilon_Q^{-2}\\log\\left(\\frac{2}{\\beta_Q}\\right)$. On the other hand, $\\left(1-\\frac{\\varepsilon_Q}{2}\\right)^2\\leq 1-\\frac{2}{\\eta p}$ if $p\\geq \\frac{2}{\\eta\\left(\\varepsilon_Q-\\frac{\\varepsilon_Q^2}{4}\\right)}$.\n","sha256_utf8":"b2a7bc834893003d42a2f7b9f7be9511e0a999ef1f6ecba84d5bf5eadfbbcc54"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"验证剩余维数不等式","source_units":[{"start":188317,"end":188694,"start_line":1875,"end_line":1875,"raw_tex":"Since $\\varepsilon_Q\\in (0,1)$, then $\\varepsilon_Q-\\frac{\\varepsilon_Q^2}{4}>\\frac{3}{4}\\varepsilon_Q^2$, which implies that $\\frac{8}{3\\eta}\\varepsilon_Q^{-2}\\geq \\frac{2}{\\eta\\left(\\varepsilon_Q-\\frac{\\varepsilon_Q^2}{4}\\right)}$. Hence,  choosing $p\\geq \\frac{8}{3\\eta}\\varepsilon_Q^{-2}$ ensures $p\\geq \\frac{2}{\\eta\\left(\\varepsilon_Q-\\frac{\\varepsilon_Q^2}{4}\\right)}$.\n","sha256_utf8":"78a6a09b26dde0954adaa9175b0f14dbee3d162c9434a5ded4b95dc255c7bb9c"}]},{"id":"S037","semantic_type":"proposition","kind":"proposition","title":"随机子空间的锐角方向","source_units":[{"start":63812,"end":64870,"start_line":949,"end_line":959,"raw_tex":"\\begin{proposition}\\label{propKappaD1}\nAssume that $\\Qrandom\\in\\rnp$ satisfies Assumption~\\ref{JltAss1}, and let $\\pollD\\subset \\rp$ be a PSS. For any $\\bm{v}\\in \\rn$, \n%there exists a direction $\\di_{\\star}\\in\\pollD$ such that \n\\begin{equation}\\label{kdprob1}\n\\prob{\\accolade{\\exists\\ubar{\\di}_{\\star}^p\\in\\pollD:\\frac{\\alpha_Q}{\\Qmax}\\kappa(\\pollD)\\normii{\\bm{v}}\\normii{\\Qrandom\\ubar{\\di}_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom\\ubar{\\di}_{\\star}^p}}}\\geq 1-\\beta_Q.\n\\end{equation}\nIn other words, with high probability, there always exists a random direction $\\Qrandom\\ubar{\\di}_{\\star}^p\\in\\accolade{\\Qrandom\\di^p:\\di^p\\in\\pollD}\\subset\\rn$ that makes an acute angle with $\\bm{v}$. The same result holds in the case of a random PSS $\\pollDrandom\\subset\\rp$; that is,\n\\begin{equation}\\label{kdprob1prime}\n\\prob{\\accolade{\\exists\\ubar{\\di}_{\\star}^p\\in\\pollDrandom:\\frac{\\alpha_Q}{\\Qmax}\\kappa(\\pollDrandom)\\normii{\\bm{v}}\\normii{\\Qrandom\\ubar{\\di}_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom\\ubar{\\di}_{\\star}^p}}}\\geq 1-\\beta_Q.\n\\end{equation}\n\\end{proposition}","sha256_utf8":"54cbd1b62b8b2e0d5ee9cd6322dfd6ae57b2fa2416fd3efbed82b309e4ecd041"}]},{"id":"S038","semantic_type":"proof","kind":"proof","title":"锐角方向完整证明","source_units":[{"start":64872,"end":66730,"start_line":961,"end_line":969,"raw_tex":"\\begin{proof}\nLet $\\bm{v}\\in \\rn$ be given. Assume that the event $\\mathscr{E}_1:=\\accolade{\\normii{\\QrandomTranspose\\bm{v}}\\geq \\alpha_Q\\normii{\\bm{v}}}$ occurs, which happens with a probability of at least $1-\\beta_Q$, and let $\\omega\\in\\mathscr{E}_1$ be arbitrary. Since $\\pollD$ is a PSS, it contains a direction $\\di_{\\star}^p=\\ubar{\\di}_{\\star}^p(\\omega)$ that makes an acute angle with the deterministic vector $\\Qrandom(\\omega)^\\top \\bm{v}\\in \\rp$. Formally thanks to~\\eqref{cosineDescentDkn}, this means that \n\\begin{equation}\\label{kdeq1}\n\\kappa(\\pollD)\\normii{\\Qrandom(\\omega)^\\top \\bm{v}}\\normii{\\di_{\\star}^p}\\leq \\scal{\\Qrandom(\\omega)^\\top \\bm{v}}{\\di_{\\star}^p}=\\scal{\\bm{v}}{\\Qrandom(\\omega)\\di_{\\star}^p}.\n\\end{equation}\nIt follows from~\\eqref{kdeq1} and the inequality $\\normii{\\Qrandom(\\omega)^\\top\\bm{v}}\\geq \\alpha_Q\\normii{\\bm{v}}$ that $\\alpha_Q\\kappa(\\pollD)\\normii{\\bm{v}}\\normii{\\di_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom(\\omega)\\di_{\\star}^p}$, which leads to $\\frac{\\alpha_Q}{\\Qmax}\\kappa(\\pollD)\\normii{\\bm{v}}\\normii{\\Qrandom(\\omega)\\di_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom(\\omega)\\di_{\\star}^p}$, given that $\\normii{\\Qrandom(\\omega)\\di_{\\star}^p}\\leq \\Qmax\\normii{\\di_{\\star}^p}$. We have therefore proved that $\\mathscr{E}_1\\subseteq\\mathscr{E}_2:=\\accolade{\\exists\\ubar{\\di}_{\\star}^p\\in\\pollD:\\frac{\\alpha_Q}{\\Qmax}\\kappa(\\pollD)\\normii{\\bm{v}}\\normii{\\Qrandom\\ubar{\\di}_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom\\ubar{\\di}_{\\star}^p}}$, whence $\\prob{\\mathscr{E}_2}\\geq \\prob{\\mathscr{E}_1}\\geq 1-\\beta_Q$, leading to~\\eqref{kdprob1}. \n\nIn the case of a random PSS $\\pollDrandom$, the same reasoning  applies with $\\pollD$ and $\\di_{\\star}^p$ replaced with $\\pollDrandom(\\omega)$ and $\\di_{\\star}^p(\\omega)$ (which are deterministic), respectively, thus leading to~\\eqref{kdprob1prime}, and the proof is completed.\n\\end{proof}","sha256_utf8":"6a8b5b5fc58948b4658b22d00d5833d793a9b4238ef05285130cd7392213fc11"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"对齐事件上选取正生成方向","source_units":[{"start":64886,"end":65611,"start_line":962,"end_line":965,"raw_tex":"Let $\\bm{v}\\in \\rn$ be given. Assume that the event $\\mathscr{E}_1:=\\accolade{\\normii{\\QrandomTranspose\\bm{v}}\\geq \\alpha_Q\\normii{\\bm{v}}}$ occurs, which happens with a probability of at least $1-\\beta_Q$, and let $\\omega\\in\\mathscr{E}_1$ be arbitrary. Since $\\pollD$ is a PSS, it contains a direction $\\di_{\\star}^p=\\ubar{\\di}_{\\star}^p(\\omega)$ that makes an acute angle with the deterministic vector $\\Qrandom(\\omega)^\\top \\bm{v}\\in \\rp$. Formally thanks to~\\eqref{cosineDescentDkn}, this means that \n\\begin{equation}\\label{kdeq1}\n\\kappa(\\pollD)\\normii{\\Qrandom(\\omega)^\\top \\bm{v}}\\normii{\\di_{\\star}^p}\\leq \\scal{\\Qrandom(\\omega)^\\top \\bm{v}}{\\di_{\\star}^p}=\\scal{\\bm{v}}{\\Qrandom(\\omega)\\di_{\\star}^p}.\n\\end{equation}\n","sha256_utf8":"66276abbd65cb71d1963095599a5ad8f94343b6f53c8d8d40b512c6fcd21f27c"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"利用范数界比较事件概率","source_units":[{"start":65611,"end":66440,"start_line":966,"end_line":966,"raw_tex":"It follows from~\\eqref{kdeq1} and the inequality $\\normii{\\Qrandom(\\omega)^\\top\\bm{v}}\\geq \\alpha_Q\\normii{\\bm{v}}$ that $\\alpha_Q\\kappa(\\pollD)\\normii{\\bm{v}}\\normii{\\di_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom(\\omega)\\di_{\\star}^p}$, which leads to $\\frac{\\alpha_Q}{\\Qmax}\\kappa(\\pollD)\\normii{\\bm{v}}\\normii{\\Qrandom(\\omega)\\di_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom(\\omega)\\di_{\\star}^p}$, given that $\\normii{\\Qrandom(\\omega)\\di_{\\star}^p}\\leq \\Qmax\\normii{\\di_{\\star}^p}$. We have therefore proved that $\\mathscr{E}_1\\subseteq\\mathscr{E}_2:=\\accolade{\\exists\\ubar{\\di}_{\\star}^p\\in\\pollD:\\frac{\\alpha_Q}{\\Qmax}\\kappa(\\pollD)\\normii{\\bm{v}}\\normii{\\Qrandom\\ubar{\\di}_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom\\ubar{\\di}_{\\star}^p}}$, whence $\\prob{\\mathscr{E}_2}\\geq \\prob{\\mathscr{E}_1}\\geq 1-\\beta_Q$, leading to~\\eqref{kdprob1}. \n","sha256_utf8":"a3e30795692a553da6999d5c1d626189db1df7435964ef6ccbd3bfe4182b89c9"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"推广到随机正生成集","source_units":[{"start":66441,"end":66719,"start_line":968,"end_line":968,"raw_tex":"In the case of a random PSS $\\pollDrandom$, the same reasoning  applies with $\\pollD$ and $\\di_{\\star}^p$ replaced with $\\pollDrandom(\\omega)$ and $\\di_{\\star}^p(\\omega)$ (which are deterministic), respectively, thus leading to~\\eqref{kdprob1prime}, and the proof is completed.\n","sha256_utf8":"303fad0e6fc3955cc65a4b78a9457ad8bea2bcf2f7f66735b3c2d1b40247dbcc"}]},{"id":"S042","semantic_type":"lemma","kind":"lemma","title":"最大内积方向概率界","source_units":[{"start":75955,"end":76880,"start_line":1019,"end_line":1029,"raw_tex":"\\begin{lemma}\\label{lemKappaD3}\nLet $\\Qrandom\\in\\rnp$ be given by Theorem~\\ref{JLforHaarTheor}, and let $\\pollD\\subset \\rp$ be a PSS. For any $\\bm{v}\\in \\rn$, the random vector $\\hat{\\di}^p(\\vb,\\qrandom)\\in\\argmax\\accolade{\\scal{\\vb}{\\Qrandom\\di^p}:\\di^p\\in\\pollD}$ satisfies\n%there exists a direction $\\di^p_{\\star}\\in\\pollD$ such that \n\\begin{equation}\\label{kdpr1}\n\\prob{\\alpha_Q\\kappa(\\pollD)\\normii{\\bm{v}}\\leq \\langle\\vb, \\Qrandom\\hat{\\di}^p(\\vb,\\qrandom)\\rangle}\\geq 1-\\beta_Q.\n\\end{equation}\nA similar result holds for any $\\bm{v}\\in \\rn$ with ${\\hat{\\di}^p}(\\vb,\\qrandom,\\drandom)\\in\\argmax\\accolade{\\scal{\\vb}{\\Qrandom\\ubar{\\di}^p}:\\ubar{\\di}^p\\in\\pollDrandom}$ in the case of a random PSS $\\pollDrandom\\subset\\rp$; that is,\n\\begin{equation}\\label{kdpr2}\n\\prob{\\alpha_Q\\kappa(\\pollDrandom)\\normii{\\bm{v}}\\leq \\langle\\vb, \\Qrandom{\\hat{\\di}^p}(\\vb,\\qrandom,\\drandom)\\rangle}\\geq 1-\\beta_Q.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"90e75cd9b16f46f0b0ce034bd1af211fd58f4f5fc2afc00870243c31d49cd73d"}]},{"id":"S043","semantic_type":"proof","kind":"proof","title":"最大方向概率界证明","source_units":[{"start":76882,"end":77972,"start_line":1031,"end_line":1033,"raw_tex":"\\begin{proof}\nBy construction, $\\Q=\\sqrt{\\frac{n}{p}}\\Ub_{\\ntp}$, where $\\ubar{\\Ub}_{\\ntp}$ represents the first $p$ columns of a Haar-distributed orthogonal matrix and $\\normii{\\di^p}=1$ for all $\\di^p\\in\\pollD$, whence $\\norme{\\Q}=\\sqrt{\\frac{n}{p}}=:\\Qmax =\\normii{\\Q\\di^p}$ since $\\normii{\\Ub_{\\ntp}\\di^p}=1$. Consequently, $\\accolade{\\exists\\ubar{\\di}_{\\star}^p\\in\\pollD:\\frac{\\alpha_Q}{\\Qmax}\\kappa(\\pollD)\\normii{\\bm{v}}\\normii{\\Qrandom\\ubar{\\di}_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom\\ubar{\\di}_{\\star}^p}}=\\accolade{\\exists\\ubar{\\di}_{\\star}^p\\in\\pollD:\\alpha_Q\\kappa(\\pollD)\\normii{\\bm{v}}\\leq \\scal{\\bm{v}}{\\Qrandom\\ubar{\\di}_{\\star}^p}}$, which is included in the event $\\accolade{\\alpha_Q\\kappa(\\pollD)\\normii{\\bm{v}}\\leq \\langle\\vb, \\Qrandom{\\hat{\\di}^p}(\\vb,\\qrandom)\\rangle}$, leading to~\\eqref{kdpr1} thanks to~\\eqref{kdprob1}. The result in~\\eqref{kdpr2} follows from~\\eqref{kdprob1prime} using similar arguments by replacing $\\pollD$ with~$\\pollDrandom$ and ${\\hat{\\di}^p}(\\vb,\\qrandom)$ with ${\\hat{\\di}^p}(\\vb,\\qrandom,\\drandom)$, and the proof is completed.\n\\end{proof}","sha256_utf8":"fab0dae8f9b1b909cab50584c50a021a35c96aa37f1d2d9044d70b61071c92fb"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"正交归一性与最大化事件包含","source_units":[{"start":76882,"end":77972,"start_line":1031,"end_line":1033,"raw_tex":"\\begin{proof}\nBy construction, $\\Q=\\sqrt{\\frac{n}{p}}\\Ub_{\\ntp}$, where $\\ubar{\\Ub}_{\\ntp}$ represents the first $p$ columns of a Haar-distributed orthogonal matrix and $\\normii{\\di^p}=1$ for all $\\di^p\\in\\pollD$, whence $\\norme{\\Q}=\\sqrt{\\frac{n}{p}}=:\\Qmax =\\normii{\\Q\\di^p}$ since $\\normii{\\Ub_{\\ntp}\\di^p}=1$. Consequently, $\\accolade{\\exists\\ubar{\\di}_{\\star}^p\\in\\pollD:\\frac{\\alpha_Q}{\\Qmax}\\kappa(\\pollD)\\normii{\\bm{v}}\\normii{\\Qrandom\\ubar{\\di}_{\\star}^p}\\leq \\scal{\\bm{v}}{\\Qrandom\\ubar{\\di}_{\\star}^p}}=\\accolade{\\exists\\ubar{\\di}_{\\star}^p\\in\\pollD:\\alpha_Q\\kappa(\\pollD)\\normii{\\bm{v}}\\leq \\scal{\\bm{v}}{\\Qrandom\\ubar{\\di}_{\\star}^p}}$, which is included in the event $\\accolade{\\alpha_Q\\kappa(\\pollD)\\normii{\\bm{v}}\\leq \\langle\\vb, \\Qrandom{\\hat{\\di}^p}(\\vb,\\qrandom)\\rangle}$, leading to~\\eqref{kdpr1} thanks to~\\eqref{kdprob1}. The result in~\\eqref{kdpr2} follows from~\\eqref{kdprob1prime} using similar arguments by replacing $\\pollD$ with~$\\pollDrandom$ and ${\\hat{\\di}^p}(\\vb,\\qrandom)$ with ${\\hat{\\di}^p}(\\vb,\\qrandom,\\drandom)$, and the proof is completed.\n\\end{proof}","sha256_utf8":"fab0dae8f9b1b909cab50584c50a021a35c96aa37f1d2d9044d70b61071c92fb"}]},{"id":"S045","semantic_type":"lemma","kind":"lemma","title":"条件期望代入公式","source_units":[{"start":77975,"end":78685,"start_line":1036,"end_line":1038,"raw_tex":"\\begin{lemma}\\cite[Theorem~2.7-$(\\ell)$]{bhattacharya2007basic}\\label{substLemma}\nLet $\\ubar{\\vb}$ and $\\Mrandom$ be random maps into $(\\mathcal{S}_1, \\mathscr{S}_1)$ and $(\\mathcal{S}_2, \\mathscr{S}_2)$, respectively. Let $\\mathcal{F}$ be a $\\sigma$-algebra and $\\psi$ be a measurable real-valued function on $\\left(\\mathcal{S}_1\\times \\mathcal{S}_2, \\mathscr{S}_1\\bigotimes\\mathscr{S}_2\\right)$. If $\\ubar{\\vb}$ is $\\mathcal{F}$-measurable, $\\sigma(\\Mrandom)$ and $\\mathcal{F}$ are independent, and $\\E{\\abs{\\psi(\\ubar{\\vb},\\Mrandom)}}<\\infty$, then $\\E{\\psi(\\ubar{\\vb},\\Mrandom)|\\mathcal{F}}=\\wp(\\ubar{\\vb})$ almost surely, where $\\wp(\\vb):=\\E{\\psi(\\vb,\\Mrandom)}$ for all $\\vb\\in\\mathcal{S}_1$.\n\\end{lemma}","sha256_utf8":"ff82e662cf259fda53eb55aa797e7bf3509ab569d27ed8a853d1d7a121f182a9"}]},{"id":"S046","semantic_type":"proposition","kind":"proposition","title":"历史条件下的方向概率界","source_units":[{"start":79185,"end":80516,"start_line":1041,"end_line":1053,"raw_tex":"\\begin{proposition}\\label{propKappaD2}\nLet $\\mathcal{F}$ be a $\\sigma$-algebra independent of a random matrix $\\Qrandom\\in\\rnp$ given by Theorem~\\ref{JLforHaarTheor}, and let $\\ubar{\\vb}\\in \\rn$ be an $\\mathcal{F}$-measurable random vector. Let $\\pollD\\subset \\rp$ be a PSS, and consider the random vector defined by $\\hat{\\di}^p(\\ubar{\\vb},\\qrandom)\\in\\argmax\\accolade{\\scal{\\ubar{\\vb}}{\\Qrandom\\di^p}:\\di^p\\in\\pollD}$. Then \n%There exists a direction $\\di_{\\star}^p\\in\\pollD$ such that \n\\begin{equation}\\label{kdprob2}\n\\prob{\\left.\\alpha_Q\\kappa(\\pollD)\\normii{\\ubar{\\vb}} \\leq \\langle\\ubar{\\vb}, \\Qrandom\\hat{\\di}^p(\\ubar{\\vb},\\qrandom)\\rangle \\right|\\F}\\geq 1-\\beta_Q\\quad\\mbox{almost surely.}\n\\end{equation}\nA similar result holds for any $\\ubar{\\vb}\\in \\rn$ with $\\hat{\\di}^p(\\ubar{\\vb},\\qrandom,\\drandom)\\in\\argmax\\accolade{\\scal{\\ubar{\\vb}}{\\Qrandom\\ubar{\\di}^p}:\\ubar{\\di}^p\\in\\pollDrandom}$ in the case of a random PSS $\\pollDrandom\\subset\\rp$ with $\\Drandom$ independent of $\\mathcal{F}$, \n%and for a direction ${\\ubar{\\di}^p}_{\\star}\\in \\pollDrandom$; \nthat is,\n\\begin{equation}\\label{kdprob2prime}\n\\prob{\\left.\\alpha_Q\\kappa(\\pollDrandom)\\normii{\\ubar{\\vb}} \\leq \\scal{\\ubar{\\vb} }{\\Qrandom{\\hat{\\di}^p}(\\ubar{\\vb},\\qrandom,\\drandom)} \\right|\\F}\\geq 1-\\beta_Q\\quad\\mbox{almost surely.}\n\\end{equation}\n\\end{proposition}","sha256_utf8":"383a41578513f3608c421ddcc8fc28edd1a0501737f28dcf75c2b5e70ff3bdad"}]},{"id":"S047","semantic_type":"proof","kind":"proof","title":"条件方向概率完整证明","source_units":[{"start":80555,"end":82880,"start_line":1056,"end_line":1058,"raw_tex":"\\begin{proof}\nLet $\\psi$ defined on $(\\rn\\times\\rnp, \\mathscr{B}(\\rn)\\bigotimes\\mathscr{B}(\\rnp))$ by $\\psi(\\ubar{\\vb},\\Qrandom)=\\mathds{1}_{\\accolade{\\alpha_Q\\kappa(\\pollD)\\normii{\\ubar{\\vb}} \\leq \\scal{\\ubar{\\vb}}{\\Qrandom\\hat{\\di}^p(\\ubar{\\vb},\\qrandom)}}}$ be a real-valued measurable function. Since $\\E{\\abs{\\psi(\\ubar{\\vb},\\Qrandom)}}<\\infty$ trivially, it follows from Lemma~\\ref{substLemma} that $\\E{\\psi(\\ubar{\\vb},\\Qrandom)|\\mathcal{F}}=\\wp(\\ubar{\\vb})$ almost surely, where $\\wp(\\vb):=\\E{\\mathds{1}_{\\accolade{\\alpha_Q\\kappa(\\pollD)\\normii{\\vb} \\leq \\scal{\\vb}{\\qrandom\\hat{\\di}^p(\\vb,\\qrandom)}}}}=\\prob{\\alpha_Q\\kappa(\\pollD)\\normii{\\vb} \\leq \\langle\\vb,\\Qrandom\\hat{\\di}^p(\\vb,\\qrandom) \\rangle}$ for all $\\vb\\in\\rn$. It follows from~\\eqref{kdpr1} that $\\wp(\\vb)\\geq 1-\\beta_Q$ for all $\\vb\\in\\rn$ whence $\\wp(\\ubar{\\vb})\\geq 1-\\beta_Q$ almost surely. This means that $\\wp(\\ubar{\\vb}):=\\E{\\psi(\\ubar{\\vb},\\Qrandom)|\\mathcal{F}}=\\prob{\\left.\\alpha_Q\\kappa(\\pollD)\\normii{\\ubar{\\vb}} \\leq \\langle\\ubar{\\vb}, \\Qrandom\\hat{\\di}^p(\\ubar{\\vb},\\qrandom)\\rangle \\right|\\F}\\geq 1-\\beta_Q$ almost surely, proving~\\eqref{kdprob2}. To prove~\\eqref{kdprob2prime}, consider the measurable real-valued function $\\chi$ defined on $(\\rn\\times\\R^{p\\times (n+m)}, \\mathscr{B}(\\rn)\\bigotimes\\mathscr{B}(\\R^{p\\times (n+m)}))$ by $\\chi\\left(\\ubar{\\vb},\\left[\\Qrandom^{\\top}\\ \\Drandom\\right]\\right)=\\mathds{1}_{\\accolade{\\alpha_Q\\kappa(\\pollDrandom)\\normii{\\ubar{\\vb}} \\leq \\scal{\\ubar{\\vb}}{\\qrandom\\hat{\\di}^p(\\ubar{\\vb},\\qrandom,\\drandom)}}}$, satisfying $\\E{\\abs{\\chi\\left(\\ubar{\\vb},\\left[\\Qrandom^{\\top}\\ \\Drandom\\right]\\right)}}<\\infty$. $\\sigma\\left(\\left[\\Qrandom^{\\top}\\ \\Drandom\\right]\\right)$ is independent of~$\\F$ whence $\\E{\\left. \\chi\\left(\\ubar{\\vb},\\left[\\Qrandom^{\\top}\\ \\Drandom\\right]\\right) \\right|\\F}=\\tilde{\\wp}(\\ubar{\\vb})$, where $\\tilde{\\wp}(\\vb):=\\prob{\\alpha_Q\\kappa(\\pollDrandom)\\normii{{\\vb}} \\leq \\scal{{\\vb}}{\\Qrandom\\hat{\\di}^p({\\vb},\\qrandom,\\drandom)}}$, thanks to Lemma~\\ref{substLemma}. It follows from~\\eqref{kdpr2} that $\\tilde{\\wp}(\\vb)\\geq 1-\\beta_Q$ $\\forall\\vb\\in\\rn$, whence $\\tilde{\\wp}(\\ubar{\\vb}):=\\E{\\left. \\chi\\left(\\ubar{\\vb},\\left[\\Qrandom^{\\top}\\ \\Drandom\\right]\\right) \\right|\\F}\\geq 1-\\beta_Q$ almost surely, showing~\\eqref{kdprob2prime}, and the proof is completed.\n\\end{proof}","sha256_utf8":"802bf26b1b4f4fd9fa509fe513db7bfb49aa9d8d4dfb26078edcc5aec325777f"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"对事件指示函数应用代入公式","source_units":[{"start":80555,"end":82880,"start_line":1056,"end_line":1058,"raw_tex":"\\begin{proof}\nLet $\\psi$ defined on $(\\rn\\times\\rnp, \\mathscr{B}(\\rn)\\bigotimes\\mathscr{B}(\\rnp))$ by $\\psi(\\ubar{\\vb},\\Qrandom)=\\mathds{1}_{\\accolade{\\alpha_Q\\kappa(\\pollD)\\normii{\\ubar{\\vb}} \\leq \\scal{\\ubar{\\vb}}{\\Qrandom\\hat{\\di}^p(\\ubar{\\vb},\\qrandom)}}}$ be a real-valued measurable function. Since $\\E{\\abs{\\psi(\\ubar{\\vb},\\Qrandom)}}<\\infty$ trivially, it follows from Lemma~\\ref{substLemma} that $\\E{\\psi(\\ubar{\\vb},\\Qrandom)|\\mathcal{F}}=\\wp(\\ubar{\\vb})$ almost surely, where $\\wp(\\vb):=\\E{\\mathds{1}_{\\accolade{\\alpha_Q\\kappa(\\pollD)\\normii{\\vb} \\leq \\scal{\\vb}{\\qrandom\\hat{\\di}^p(\\vb,\\qrandom)}}}}=\\prob{\\alpha_Q\\kappa(\\pollD)\\normii{\\vb} \\leq \\langle\\vb,\\Qrandom\\hat{\\di}^p(\\vb,\\qrandom) \\rangle}$ for all $\\vb\\in\\rn$. It follows from~\\eqref{kdpr1} that $\\wp(\\vb)\\geq 1-\\beta_Q$ for all $\\vb\\in\\rn$ whence $\\wp(\\ubar{\\vb})\\geq 1-\\beta_Q$ almost surely. This means that $\\wp(\\ubar{\\vb}):=\\E{\\psi(\\ubar{\\vb},\\Qrandom)|\\mathcal{F}}=\\prob{\\left.\\alpha_Q\\kappa(\\pollD)\\normii{\\ubar{\\vb}} \\leq \\langle\\ubar{\\vb}, \\Qrandom\\hat{\\di}^p(\\ubar{\\vb},\\qrandom)\\rangle \\right|\\F}\\geq 1-\\beta_Q$ almost surely, proving~\\eqref{kdprob2}. To prove~\\eqref{kdprob2prime}, consider the measurable real-valued function $\\chi$ defined on $(\\rn\\times\\R^{p\\times (n+m)}, \\mathscr{B}(\\rn)\\bigotimes\\mathscr{B}(\\R^{p\\times (n+m)}))$ by $\\chi\\left(\\ubar{\\vb},\\left[\\Qrandom^{\\top}\\ \\Drandom\\right]\\right)=\\mathds{1}_{\\accolade{\\alpha_Q\\kappa(\\pollDrandom)\\normii{\\ubar{\\vb}} \\leq \\scal{\\ubar{\\vb}}{\\qrandom\\hat{\\di}^p(\\ubar{\\vb},\\qrandom,\\drandom)}}}$, satisfying $\\E{\\abs{\\chi\\left(\\ubar{\\vb},\\left[\\Qrandom^{\\top}\\ \\Drandom\\right]\\right)}}<\\infty$. $\\sigma\\left(\\left[\\Qrandom^{\\top}\\ \\Drandom\\right]\\right)$ is independent of~$\\F$ whence $\\E{\\left. \\chi\\left(\\ubar{\\vb},\\left[\\Qrandom^{\\top}\\ \\Drandom\\right]\\right) \\right|\\F}=\\tilde{\\wp}(\\ubar{\\vb})$, where $\\tilde{\\wp}(\\vb):=\\prob{\\alpha_Q\\kappa(\\pollDrandom)\\normii{{\\vb}} \\leq \\scal{{\\vb}}{\\Qrandom\\hat{\\di}^p({\\vb},\\qrandom,\\drandom)}}$, thanks to Lemma~\\ref{substLemma}. It follows from~\\eqref{kdpr2} that $\\tilde{\\wp}(\\vb)\\geq 1-\\beta_Q$ $\\forall\\vb\\in\\rn$, whence $\\tilde{\\wp}(\\ubar{\\vb}):=\\E{\\left. \\chi\\left(\\ubar{\\vb},\\left[\\Qrandom^{\\top}\\ \\Drandom\\right]\\right) \\right|\\F}\\geq 1-\\beta_Q$ almost surely, showing~\\eqref{kdprob2prime}, and the proof is completed.\n\\end{proof}","sha256_utf8":"802bf26b1b4f4fd9fa509fe513db7bfb49aa9d8d4dfb26078edcc5aec325777f"}]},{"id":"S049","semantic_type":"lemma","kind":"lemma","title":"下降方向事件条件概率","source_units":[{"start":135452,"end":135859,"start_line":1495,"end_line":1497,"raw_tex":"\\begin{lemma}\\label{AkLemma1}\nLet Assumption~\\ref{QkHaarAssumpt} hold, and consider ${\\ubar{\\hat{\\di}}^p_k}:=\\hat{\\di}^p_k({\\footnotesize{\\X}},\\qrandom,\\drandom)\\in\\argmax\\accolade{-\\nabla f(\\Xk)^{\\top}\\left(\\QkRandom\\diRandom\\right):\\diRandom\\in\\pollDrandom_k}$. Then the event $A_k$ defined in~\\eqref{AkEventDef} satisfies $\\prob{A_k|\\falgebra}=\\E{\\iak|\\falgebra}\\geq 1-\\beta_Q$ almost surely.\n\\end{lemma}","sha256_utf8":"a78a0d4021931603039d6f9c6cdef3f912165c87bfb5f4a920ffe2445acf04eb"}]},{"id":"S050","semantic_type":"proof","kind":"proof","title":"下降事件概率证明","source_units":[{"start":135860,"end":136156,"start_line":1498,"end_line":1500,"raw_tex":"\\begin{proof}\nBy construction, $\\nabla f(\\Xk)$ is $\\falgebra$-measurable while $\\QkRandom$ and $\\pollDrandom_k$ are independent of $\\falgebra$. The conditions of Proposition~\\ref{propKappaD2} are therefore satisfied, and the result straightforwardly follows from~\\eqref{kdprob2prime}.\n\\end{proof}","sha256_utf8":"d5d2ea89b1f9ed0ec76ebb59eafa127522003745dc2b2c16966d562ea5948daa"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"对历史可测负梯度应用条件界","source_units":[{"start":135860,"end":136156,"start_line":1498,"end_line":1500,"raw_tex":"\\begin{proof}\nBy construction, $\\nabla f(\\Xk)$ is $\\falgebra$-measurable while $\\QkRandom$ and $\\pollDrandom_k$ are independent of $\\falgebra$. The conditions of Proposition~\\ref{propKappaD2} are therefore satisfied, and the result straightforwardly follows from~\\eqref{kdprob2prime}.\n\\end{proof}","sha256_utf8":"d5d2ea89b1f9ed0ec76ebb59eafa127522003745dc2b2c16966d562ea5948daa"}]},{"id":"S052","semantic_type":"lemma","kind":"lemma","title":"小步长保证成功","source_units":[{"start":137978,"end":138481,"start_line":1518,"end_line":1520,"raw_tex":"\\begin{lemma}\\label{smallLemmaHaar}\nLet Assumptions~\\ref{QkHaarAssumpt},~\\ref{kapminAssump} and~\\ref{gradLipAssump} hold, and assume that $\\dk\\leq \\depsilon$. Let $\\fok$ and $\\fuk$ be $\\ef$-accurate estimates of $f(\\xk)$ and  $f(\\xk+\\dk\\ub)$, respectively. Assume that the event $A_k$ occurs. If $\\normii{\\nabla f(\\xk)}>\\epsilon$, then there exists $\\ub\\in \\mathbb{U}_k^n:=\\{\\Ukp\\di^p:\\di^p\\in \\pollD_k\\}$ such that $\\fuk-\\fok\\leq -\\gamma\\ef\\dk^2$; that is, the iteration $k$ is successful. \n\\end{lemma}","sha256_utf8":"a49321d1d6915e95aede27b26ad8cebc3a43bf6ac78d82883a2c86686ebfdd1e"}]},{"id":"S053","semantic_type":"proof","kind":"proof","title":"小步长成功完整证明","source_units":[{"start":138483,"end":138541,"start_line":1522,"end_line":1524,"raw_tex":"\\begin{proof}\n{\\bbl See Appendix~\\ref{lem52}.}\n\\end{proof}","sha256_utf8":"10fd28fe133c9a9d58fa030acc44ca529accc59e0805e527be9329e50658b1ec"},{"start":197453,"end":200081,"start_line":1946,"end_line":1975,"raw_tex":"\\begin{proof}\nThe proof is inspired by that of~\\cite[Lemma~2]{dzahini2020expected} using elements derived in~\\cite{KoLeTo03a}. Suppose that $\\dk\\leq \\depsilon$, that the event $A_k$ occurs, and that the estimates $\\fok$ and $\\fuk$ are $\\ef$-accurate. Then, assume by way of contradiction that $\\fuk-\\fok> -\\gamma\\ef\\dk^2$ for all $\\ub\\in \\mathbb{U}_k^n$ of the form $\\ub:=\\Ukp\\di^p$, with $\\di^p\\in \\pollD_k$. Since $A_k$ occurs, then the latter inequality holds in particular for $\\ub=\\Ukp\\hat{\\di}^p_k$, with $\\hat{\\di}^p_k\\in\\argmax\\accolade{-\\nabla f(\\xk)^{\\top}\\left(\\Qk\\di^p\\right):\\di^p\\in\\pollD_k}$ satisfying \n\\begin{equation}\\label{AkRealization}\n\\alpha_Q\\kappa(\\pollD_k)\\normii{\\nabla f(\\xk)}\\leq -\\sqrt{\\frac{n}{p}}\\nabla f(\\xk)^{\\top}\\left(\\Ukp{\\hat{\\di}^p_k}\\right).\n\\end{equation}\nIt follows from the $\\ef$-accuracy assumption, along with the equality \n\\[f(\\xk+\\dk\\ub)-f(\\xk)=f(\\xk+\\dk\\ub)-\\fuk+(\\fuk-\\fok)+\\fok-f(\\xk),\\]\nthat \n\\begin{equation}\\label{eq39Acc}\nf\\left(\\xk+\\dk\\Ukp\\hat{\\di}^p_k\\right)-f(\\xk)+(\\gamma+2)\\ef\\dk^2\\geq 0.\n\\end{equation}\nIt follows from the mean value theorem and~\\eqref{eq39Acc} that there exists $\\mu_k\\in[0,1]$ such that \n\\begin{equation}\\label{eq40MeanVal}\n0\\leq \\dk\\nabla f\\left(\\xk+\\mu_k\\dk\\Ukp\\hat{\\di}^p_k\\right)^{\\top}\\left(\\Ukp{\\hat{\\di}^p_k}\\right)+(\\gamma+2)\\ef\\dk^2.\n\\end{equation}\nMultiplying both sides of~\\eqref{eq40MeanVal} by $\\frac{1}{\\dk}\\sqrt{\\frac{n}{p}}$ and subtracting $\\sqrt{\\frac{n}{p}}\\nabla f(\\xk)^{\\top}\\left(\\Ukp{\\hat{\\di}^p_k}\\right)$ yield\n\\begin{equation}\\label{eq41yield}\n-\\sqrt{\\frac{n}{p}}\\nabla f(\\xk)^{\\top}\\left(\\Ukp{\\hat{\\di}^p_k}\\right)\\leq\\sqrt{\\frac{n}{p}}\\left(\\nabla f\\left(\\xk+\\mu_k\\dk\\Ukp\\hat{\\di}^p_k\\right)- \\nabla f(\\xk)\\right)^{\\top}\\left(\\Ukp{\\hat{\\di}^p_k}\\right)+(\\gamma+2)\\ef\\dk,\n\\end{equation}\nwhich, combined with~\\eqref{AkRealization}, leads to\n\\begin{equation}\\label{eq42Norm}\n\\begin{split}\n\\alpha_Q\\kappa(\\pollD_k)\\normii{\\nabla f(\\xk)}&\\leq L_g\\sqrt{\\frac{n}{p}}\\normii{\\mu_k\\dk\\Ukp{\\hat{\\di}^p_k}}\\normii{\\Ukp{\\hat{\\di}^p_k}}+(\\gamma+2)\\ef\\dk \\leq \\left(L_g\\sqrt{\\frac{n}{p}}+(\\gamma+2)\\ef\\right)\\depsilon,%\\\\\n%&\\leq L_g\\sqrt{\\frac{n}{p}}\\dk+(\\gamma+2)\\ef\\dk\n\\end{split}\n\\end{equation}\nwhere the last inequality follows from the $L_g$-Lipschitz continuity of $\\nabla f$, the fact that $\\normii{\\Ukp{\\hat{\\di}^p_k}}=1$ and $\\dk\\leq\\depsilon$. It follows from~\\eqref{eq42Norm} that \n\\[\\normii{\\nabla f(\\xk)}\\leq \\alpha_Q^{-1}\\kappa_{\\min}^{-1}\\left(L_g\\sqrt{\\frac{n}{p}}+(\\gamma+2)\\ef\\right)\\times \\frac{\\epsilon}{\\xi}< \\epsilon,\\]\nwhere the last inequality follows from~\\eqref{depsEq}, and the proof is completed.\n\\end{proof}","sha256_utf8":"1b3882609dfc2ac05920b1ee7e7941e52312ad99e1a5fbe1c6fbb467687b4e4e"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"转至小步长引理附录","source_units":[{"start":138483,"end":138541,"start_line":1522,"end_line":1524,"raw_tex":"\\begin{proof}\n{\\bbl See Appendix~\\ref{lem52}.}\n\\end{proof}","sha256_utf8":"10fd28fe133c9a9d58fa030acc44ca529accc59e0805e527be9329e50658b1ec"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"反设失败并选取最佳方向","source_units":[{"start":197467,"end":198249,"start_line":1947,"end_line":1950,"raw_tex":"The proof is inspired by that of~\\cite[Lemma~2]{dzahini2020expected} using elements derived in~\\cite{KoLeTo03a}. Suppose that $\\dk\\leq \\depsilon$, that the event $A_k$ occurs, and that the estimates $\\fok$ and $\\fuk$ are $\\ef$-accurate. Then, assume by way of contradiction that $\\fuk-\\fok> -\\gamma\\ef\\dk^2$ for all $\\ub\\in \\mathbb{U}_k^n$ of the form $\\ub:=\\Ukp\\di^p$, with $\\di^p\\in \\pollD_k$. Since $A_k$ occurs, then the latter inequality holds in particular for $\\ub=\\Ukp\\hat{\\di}^p_k$, with $\\hat{\\di}^p_k\\in\\argmax\\accolade{-\\nabla f(\\xk)^{\\top}\\left(\\Qk\\di^p\\right):\\di^p\\in\\pollD_k}$ satisfying \n\\begin{equation}\\label{AkRealization}\n\\alpha_Q\\kappa(\\pollD_k)\\normii{\\nabla f(\\xk)}\\leq -\\sqrt{\\frac{n}{p}}\\nabla f(\\xk)^{\\top}\\left(\\Ukp{\\hat{\\di}^p_k}\\right).\n\\end{equation}\n","sha256_utf8":"ae62f0268f75e82fcb376af089039575ca85bf01462a92bf484e230a14a86ddd"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"估计精度给出真实差值界","source_units":[{"start":198249,"end":198515,"start_line":1951,"end_line":1956,"raw_tex":"It follows from the $\\ef$-accuracy assumption, along with the equality \n\\[f(\\xk+\\dk\\ub)-f(\\xk)=f(\\xk+\\dk\\ub)-\\fuk+(\\fuk-\\fok)+\\fok-f(\\xk),\\]\nthat \n\\begin{equation}\\label{eq39Acc}\nf\\left(\\xk+\\dk\\Ukp\\hat{\\di}^p_k\\right)-f(\\xk)+(\\gamma+2)\\ef\\dk^2\\geq 0.\n\\end{equation}\n","sha256_utf8":"efa83ea9c4e354c378063babca295192b6954f4b8c73a86074046328faf98dac"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"中值定理展开函数差","source_units":[{"start":198515,"end":198789,"start_line":1957,"end_line":1960,"raw_tex":"It follows from the mean value theorem and~\\eqref{eq39Acc} that there exists $\\mu_k\\in[0,1]$ such that \n\\begin{equation}\\label{eq40MeanVal}\n0\\leq \\dk\\nabla f\\left(\\xk+\\mu_k\\dk\\Ukp\\hat{\\di}^p_k\\right)^{\\top}\\left(\\Ukp{\\hat{\\di}^p_k}\\right)+(\\gamma+2)\\ef\\dk^2.\n\\end{equation}\n","sha256_utf8":"1ea2f5c45c560546ddc1d53aff31d30dc593f3d579b51b98e1ba2528f0efa7e2"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"分离当前点负梯度内积","source_units":[{"start":198789,"end":199245,"start_line":1961,"end_line":1964,"raw_tex":"Multiplying both sides of~\\eqref{eq40MeanVal} by $\\frac{1}{\\dk}\\sqrt{\\frac{n}{p}}$ and subtracting $\\sqrt{\\frac{n}{p}}\\nabla f(\\xk)^{\\top}\\left(\\Ukp{\\hat{\\di}^p_k}\\right)$ yield\n\\begin{equation}\\label{eq41yield}\n-\\sqrt{\\frac{n}{p}}\\nabla f(\\xk)^{\\top}\\left(\\Ukp{\\hat{\\di}^p_k}\\right)\\leq\\sqrt{\\frac{n}{p}}\\left(\\nabla f\\left(\\xk+\\mu_k\\dk\\Ukp\\hat{\\di}^p_k\\right)- \\nabla f(\\xk)\\right)^{\\top}\\left(\\Ukp{\\hat{\\di}^p_k}\\right)+(\\gamma+2)\\ef\\dk,\n\\end{equation}\n","sha256_utf8":"3a836a906fd600a084f0d80ed527caa2ecb08e1d9fbbf7d369a9906dc1677123"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"Lipschitz界控制梯度范数","source_units":[{"start":199245,"end":199643,"start_line":1965,"end_line":1971,"raw_tex":"which, combined with~\\eqref{AkRealization}, leads to\n\\begin{equation}\\label{eq42Norm}\n\\begin{split}\n\\alpha_Q\\kappa(\\pollD_k)\\normii{\\nabla f(\\xk)}&\\leq L_g\\sqrt{\\frac{n}{p}}\\normii{\\mu_k\\dk\\Ukp{\\hat{\\di}^p_k}}\\normii{\\Ukp{\\hat{\\di}^p_k}}+(\\gamma+2)\\ef\\dk \\leq \\left(L_g\\sqrt{\\frac{n}{p}}+(\\gamma+2)\\ef\\right)\\depsilon,%\\\\\n%&\\leq L_g\\sqrt{\\frac{n}{p}}\\dk+(\\gamma+2)\\ef\\dk\n\\end{split}\n\\end{equation}\n","sha256_utf8":"6ce88aea8b574424ff59f38b0d9a65347ec08c92766114fcedcbf3a52d5007fd"}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","title":"临界步长导出阈值矛盾","source_units":[{"start":199643,"end":200070,"start_line":1972,"end_line":1974,"raw_tex":"where the last inequality follows from the $L_g$-Lipschitz continuity of $\\nabla f$, the fact that $\\normii{\\Ukp{\\hat{\\di}^p_k}}=1$ and $\\dk\\leq\\depsilon$. It follows from~\\eqref{eq42Norm} that \n\\[\\normii{\\nabla f(\\xk)}\\leq \\alpha_Q^{-1}\\kappa_{\\min}^{-1}\\left(L_g\\sqrt{\\frac{n}{p}}+(\\gamma+2)\\ef\\right)\\times \\frac{\\epsilon}{\\xi}< \\epsilon,\\]\nwhere the last inequality follows from~\\eqref{depsEq}, and the proof is completed.\n","sha256_utf8":"ede8f88b43d6feb109d550ee70110246a58e54d4243c7f9585abe98abeae4bd7"}]},{"id":"S061","semantic_type":"lemma","kind":"lemma","title":"步长更新比较","source_units":[{"start":138742,"end":139292,"start_line":1527,"end_line":1531,"raw_tex":"\\begin{lemma}\\label{dynamicsLemma}\n{\\bl Let Assumption~\\ref{QkHaarAssumpt} hold with $0<\\beta_Q<1-\\frac{1}{2\\beta_f}$ and let Assumptions~\\ref{QkHaarAssumpt},~\\ref{efkfAssump},~\\ref{boundedFassumpt},~\\ref{kapminAssump}, and~\\ref{gradLipAssump} hold.}\n%and all assumptions made in Theorem~\\ref{zerothOrdTheor} hold. \nThen Assumption~\\ref{assumptionRenewal}-$(ii)$ is satisfied for $\\Wtildek:=2\\left(\\iak\\ijk-\\frac{1}{2}\\right)$, with $\\lambda=-\\log(\\tau)$ and some fixed $q\\in (\\tilde{\\beta}, 1)$, where $\\tilde{\\beta}=\\beta_f(1-\\beta_Q)$.\n\\end{lemma}","sha256_utf8":"545d01b9f9098b65254d58f6268e3c7fd01c58c7cf8d70d2c2891c72ce67c767"}]},{"id":"S062","semantic_type":"proof","kind":"proof","title":"步长更新比较完整证明","source_units":[{"start":139294,"end":140435,"start_line":1533,"end_line":1537,"raw_tex":"\\begin{proof}\nThe result is proved by suitably adapting the proof of~\\cite[Lemma~7]{blanchet2016convergence}, as was also done in~\\cite[Lemma~3]{dzahini2020expected}, \\cite[Lemma~5.3]{DzaWildSub2022} and~\\cite[Lemma~4.8]{paquette2018stochastic}. First, observe that~\\eqref{dynamicsRenewal} trivially holds when $\\itksup=0$. It is shown next that when $\\itksup=1$ (which is assumed in the remainder of the proof), then \n\\[\\Dkun\\geq \\min\\accolade{\\depsilon, \\min\\accolade{\\dmax,\\tau^{-1}\\Dk}\\iak\\ijk+\\tau\\Dk\\left(1-\\iak\\ijk\\right)}.\\] \nRecall that $\\dk=\\tau^{i_k}\\depsilon$ for some integer $i_k$ so that if $\\dk>\\depsilon$, then $\\dk\\geq \\tau^{-1}\\depsilon$, whence $\\dkun\\geq \\tau\\dk\\geq \\depsilon$. Next, assume that $\\dk\\leq \\depsilon$, and observe that $\\normii{\\nabla f(\\xk)}>\\eps$ since $\\Tepsilon>k$. If $\\iak\\ijk=1$, it follows from Lemma~\\ref{smallLemmaHaar} that the iteration is successful. Consequently, $\\dkun=\\min\\accolade{\\tau^{-1}\\dk,\\dmax}$. But if $\\iak\\ijk=0$, then it always holds that $\\dkun\\geq \\tau\\dk$. The proof is completed by observing that $\\prob{\\iak\\ijk=1|\\falgebra}=q$ for some $q\\geq\\tilde{\\beta}$.\n\\end{proof}","sha256_utf8":"355c69567df7ac58194dad7d72b183ec95850f4eff951cabd1d18bd3e35b21ba"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"区分停止事件并提出更新下界","source_units":[{"start":139308,"end":139828,"start_line":1534,"end_line":1535,"raw_tex":"The result is proved by suitably adapting the proof of~\\cite[Lemma~7]{blanchet2016convergence}, as was also done in~\\cite[Lemma~3]{dzahini2020expected}, \\cite[Lemma~5.3]{DzaWildSub2022} and~\\cite[Lemma~4.8]{paquette2018stochastic}. First, observe that~\\eqref{dynamicsRenewal} trivially holds when $\\itksup=0$. It is shown next that when $\\itksup=1$ (which is assumed in the remainder of the proof), then \n\\[\\Dkun\\geq \\min\\accolade{\\depsilon, \\min\\accolade{\\dmax,\\tau^{-1}\\Dk}\\iak\\ijk+\\tau\\Dk\\left(1-\\iak\\ijk\\right)}.\\] \n","sha256_utf8":"0841a8eddf8aecc55e844b828418d804031f88798787e6d576d446608a0a05d8"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"分步长尺度验证更新并控制联合概率","source_units":[{"start":139828,"end":140424,"start_line":1536,"end_line":1536,"raw_tex":"Recall that $\\dk=\\tau^{i_k}\\depsilon$ for some integer $i_k$ so that if $\\dk>\\depsilon$, then $\\dk\\geq \\tau^{-1}\\depsilon$, whence $\\dkun\\geq \\tau\\dk\\geq \\depsilon$. Next, assume that $\\dk\\leq \\depsilon$, and observe that $\\normii{\\nabla f(\\xk)}>\\eps$ since $\\Tepsilon>k$. If $\\iak\\ijk=1$, it follows from Lemma~\\ref{smallLemmaHaar} that the iteration is successful. Consequently, $\\dkun=\\min\\accolade{\\tau^{-1}\\dk,\\dmax}$. But if $\\iak\\ijk=0$, then it always holds that $\\dkun\\geq \\tau\\dk$. The proof is completed by observing that $\\prob{\\iak\\ijk=1|\\falgebra}=q$ for some $q\\geq\\tilde{\\beta}$.\n","sha256_utf8":"7d5108bb3efc4a280c2c7ff877945c20a6f5602f8163daf7593e486a8fbbae06"}]},{"id":"S065","semantic_type":"theorem","kind":"theorem","title":"势函数负漂移与零阶收敛","source_units":[{"start":127431,"end":128329,"start_line":1414,"end_line":1425,"raw_tex":"\\begin{theorem}\\label{zerothOrdTheor}\nLet Assumptions~\\ref{QkHaarAssumpt},~\\ref{efkfAssump} and~\\ref{boundedFassumpt} hold. \n%Let $\\gamma>2$ and $\\nu,\\tau\\in (0,1)$ satisfying $\\frac{\\nu}{1-\\nu}\\geq \\frac{2(\\tau^{-2}-1)}{\\gamma-2}$. Assume that Assumption~\\ref{efkfAssump} holds with $\\beta_f\\in (0,1)$ satisfying  $\\frac{\\beta_f}{1-\\beta_f}\\geq \\frac{4\\nu}{(1-\\nu)(1-\\tau^2)}$. \nDefine $\\varrho=\\frac{1}{2}\\beta_f(1-\\nu)(1-\\tau^2)$. Then, the random function $\\Phik:=\\frac{\\nu}{\\ef}\\left(f(\\Xk)-f_{\\min}\\right)+(1-\\nu)\\Dk^2$ satisfies \n\\begin{equation}\\label{PhikDynamics}\n\\E{\\Phikun-\\Phik|\\fqdalgebra}\\leq -\\varrho\\Dk^2 \\quad\\mbox{almost surely, for all}\\ k\\in \\N, \n\\end{equation}\nwhich implies that \n\\begin{equation}\\label{finiteSeries}\n\\sum_{k=0}^{\\infty}\\Dk^2<\\infty\\quad\\mbox{almost surely, whence}\\qquad \\underset{k\\to\\infty}{\\lim}\\Dk=0\\quad\\mbox{almost surely.}\n\\end{equation}\n\\end{theorem}","sha256_utf8":"73dd67321cb0cea82b25eeb529136ee5168267387fd6908d84f90989b2c75181"}]},{"id":"S066","semantic_type":"proof","kind":"proof","title":"零阶收敛完整证明","source_units":[{"start":128331,"end":128388,"start_line":1427,"end_line":1429,"raw_tex":"\\begin{proof}\n{\\bbl See Appendix~\\ref{th41}.}\n\\end{proof}","sha256_utf8":"e448408b1916f96290b0e7ea75e78acfeb652937249a4224341b172ea7208dec"},{"start":192773,"end":197390,"start_line":1897,"end_line":1941,"raw_tex":"\\begin{proof}\nThe proof is similar to that of~\\cite[Theorem~2]{dzahini2020expected}, using ideas derived in~\\cite{blanchet2016convergence,chen2018stochastic,DzaWildSub2022,paquette2018stochastic}. It is presented here for the sake of clarity and self-containedness. The overall goal is to prove~\\eqref{PhikDynamics} since taking the expectation on both its sides leads to~\\eqref{finiteSeries} as detailed in~\\cite[Theorem~3]{dzahini2020expected}, given that $\\Phik>0$ for all~$k\\in\\N$. The proof considers two cases, {\\it good estimates} and {\\it bad estimates}, each of which is broken into whether the iteration is successful or not. Throughout, \n\\[\\mathfrak{S}_k:=\\accolade{\\mbox{The}\\ k\\mbox{th iteration is successful}}.\\]\n{\\bf Case~1:} {\\it (Good estimates, $\\ijk=1$).} The overall goal is to prove that almost surely, \n\\begin{equation}\\label{treizeEq}\n\\E{\\ijk\\left(\\Phikun-\\Phik\\right)|\\fqdalgebra}\\leq -\\beta_f(1-\\nu)(1-\\tau^2)\\Dk^2.\n\\end{equation}\n{{Subcase~$1i$}:} {\\it (Successful iteration, $\\isk=1$).} A decrease occurs in~$f$ according to~\\eqref{decreaseEqu}, and $\\Dk$ is updated according to~$\\Dkun=\\min\\accolade{\\tau^{-1}\\Dk,\\dmax}$, whence $\\ijk\\isk\\frac{\\nu}{\\ef}(f(\\Xkun)-f(\\Xk))\\leq -\\ijk\\isk\\nu(\\gamma-2)\\Dk^2$ and $\\ijk\\isk(1-\\nu)(\\Dkun^2-\\Dk^2)\\leq \\ijk\\isk(1-\\nu)(\\tau^{-2}-1)\\Dk^2$, leading to \n\\begin{equation}\\label{dixseptEq}\n\\ijk\\isk\\left(\\Phikun-\\Phik\\right)\\leq\\ijk\\isk\\left(-\\nu(\\gamma-2)+(1-\\nu)(\\tau^{-2}-1)\\right)\\Dk^2\\leq -\\ijk\\isk(1-\\nu)(1-\\tau^2)\\Dk^2,\n\\end{equation}\nwhere the last inequality follows from the above choice of~$\\nu$ along with the fact that $1-\\tau^2<\\tau^{-2}-1$. \\\\\n{{Subcase~$1ii$}:} {\\it (Unsuccessful iteration, $\\iskc=1$).} The parameter~$\\Dk$ is decreased according to $\\Dkun=\\tau\\Dk$, and $f(\\Xkun)=f(\\Xk)$, whence\n\\begin{equation}\\label{dixhuitEq}\n\\ijk\\iskc\\left(\\Phikun-\\Phik\\right)=-\\ijk\\iskc(1-\\nu)(1-\\tau^2)\\Dk^2.\n\\end{equation}\nEquations~\\eqref{dixseptEq} and~\\eqref{dixhuitEq} lead to\n\\begin{equation}\\label{vingtEq}\n\\ijk\\left(\\Phikun-\\Phik\\right)\\leq -\\ijk(1-\\nu)(1-\\tau^2)\\Dk^2.\n\\end{equation}\nTaking the conditional expectation with respect to $\\fqdalgebra$ on both sides of~\\eqref{vingtEq} and using Assumption~\\ref{efkfAssump}-{\\it (i)} lead to~\\eqref{treizeEq}.\n{\\bf Case~2:} {\\it (Bad estimates, $\\ijkc=1$).} Because of bad estimates, an iterate leading to an increase in~$f$ and~$\\Dk$, and hence in~$\\Phik$, can be accepted by Algorithm~\\ref{algoStoDARS}. Such an increase is controlled by using Assumption~\\ref{efkfAssump}-{\\it (ii)}. The overall goal is to show that almost surely,\n\\begin{equation}\\label{vingtEtUnEq}\n\\E{\\ijkc\\left(\\Phikun-\\Phik\\right)|\\fqdalgebra}\\leq 2\\nu(1-\\beta_f)\\Dk^2.\n\\end{equation}\n{{Subcase~$2i$}:} {\\it (Successful iteration, $\\isk=1$).} Since $\\Fuk-\\Fok\\leq-\\gamma\\ef\\Dk^2$, then \n\\[\\ijkc\\isk\\frac{\\nu}{\\ef}(f(\\Xkun)-f(\\Xk))\\leq\\ijkc\\isk\\nu\\left(-\\gamma\\Dk^2+\\frac{1}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right)\\right).\\]\nRecalling that $\\Dkun=\\min\\accolade{\\tau^{-1}\\Dk,\\dmax}$, and by noticing that $-\\nu\\gamma+(1-\\nu)(\\tau^{-2}-1)\\leq 0$ due to the above choice of $\\nu$, it holds that\n% \\begin{equation}\\label{vingtcinqEq}\n% \\begin{split}\n% \\ijkc\\isk\\left(\\Phikun-\\Phik\\right)&\\leq \\ijkc\\isk\\left(\\left(-\\nu\\gamma+(1-\\nu)(\\tau^{-2}-1) \\right)\\Dk^2+\\frac{\\nu}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right) \\right)\\\\\n% &\\leq \\ijkc\\isk\\left(\\frac{\\nu}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right) \\right)\n% \\end{split}\n% \\end{equation}\n\\begin{equation}\\label{vingtcinqEq}\n\\ijkc\\isk\\left(\\Phikun-\\Phik\\right)\\leq \\ijkc\\isk\\frac{\\nu}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right).\n\\end{equation}\n{{Subcase~$2ii$}:} {\\it (Unsuccessful iteration, $\\iskc=1$).} The upper bound on the change in~$\\Phik$ follows from~\\eqref{dixhuitEq} by replacing~$\\ijk$ with~$\\ijkc$. Thus, $\\ijkc\\iskc\\left(\\Phikun-\\Phik\\right)\\leq \\ijkc\\iskc\\frac{\\nu}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right)$, which, combined with~\\eqref{vingtcinqEq}, yields\n\\begin{equation}\\label{vingtseptEq}\n\\ijkc\\left(\\Phikun-\\Phik\\right)\\leq \\ijkc\\frac{\\nu}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right).\n\\end{equation}\nTaking the conditional expectation with respect to $\\fqdalgebra$ on both sides of~\\eqref{vingtseptEq} and applying Lemma~\\ref{SDDSlemma1} lead to~\\eqref{vingtEtUnEq}. Then, combining~\\eqref{treizeEq} and~\\eqref{vingtEtUnEq} yields\n\\[\\E{\\left(\\Phikun-\\Phik\\right)|\\fqdalgebra}\\leq (-\\beta_f(1-\\nu)(1-\\tau^2)+2\\nu(1-\\beta_f))\\Dk^2\\leq -\\varrho\\Dk^2, \\]\nwhere the last inequality follows from the above choice of $\\beta_f$, and the proof is completed.\n\\end{proof}","sha256_utf8":"d912cf34af0f1639405ce55e9955bd3ca45468190c0e0e5946fbf32c86c814c6"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"转至零阶收敛附录","source_units":[{"start":128331,"end":128388,"start_line":1427,"end_line":1429,"raw_tex":"\\begin{proof}\n{\\bbl See Appendix~\\ref{th41}.}\n\\end{proof}","sha256_utf8":"e448408b1916f96290b0e7ea75e78acfeb652937249a4224341b172ea7208dec"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"负漂移归约与成功事件定义","source_units":[{"start":192787,"end":193501,"start_line":1898,"end_line":1899,"raw_tex":"The proof is similar to that of~\\cite[Theorem~2]{dzahini2020expected}, using ideas derived in~\\cite{blanchet2016convergence,chen2018stochastic,DzaWildSub2022,paquette2018stochastic}. It is presented here for the sake of clarity and self-containedness. The overall goal is to prove~\\eqref{PhikDynamics} since taking the expectation on both its sides leads to~\\eqref{finiteSeries} as detailed in~\\cite[Theorem~3]{dzahini2020expected}, given that $\\Phik>0$ for all~$k\\in\\N$. The proof considers two cases, {\\it good estimates} and {\\it bad estimates}, each of which is broken into whether the iteration is successful or not. Throughout, \n\\[\\mathfrak{S}_k:=\\accolade{\\mbox{The}\\ k\\mbox{th iteration is successful}}.\\]\n","sha256_utf8":"2ee6299298b8bbc62ca192c07dba953d0757975ea724254298ffb34cfefaf4e1"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"好估计情形的条件漂移目标","source_units":[{"start":193501,"end":193730,"start_line":1900,"end_line":1903,"raw_tex":"{\\bf Case~1:} {\\it (Good estimates, $\\ijk=1$).} The overall goal is to prove that almost surely, \n\\begin{equation}\\label{treizeEq}\n\\E{\\ijk\\left(\\Phikun-\\Phik\\right)|\\fqdalgebra}\\leq -\\beta_f(1-\\nu)(1-\\tau^2)\\Dk^2.\n\\end{equation}\n","sha256_utf8":"251daa624c62902822d98580f2985015a70ac2b3d6a15ed0d2624c9e7fc30c9f"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"好估计且成功时的势函数下降","source_units":[{"start":193730,"end":194397,"start_line":1904,"end_line":1908,"raw_tex":"{{Subcase~$1i$}:} {\\it (Successful iteration, $\\isk=1$).} A decrease occurs in~$f$ according to~\\eqref{decreaseEqu}, and $\\Dk$ is updated according to~$\\Dkun=\\min\\accolade{\\tau^{-1}\\Dk,\\dmax}$, whence $\\ijk\\isk\\frac{\\nu}{\\ef}(f(\\Xkun)-f(\\Xk))\\leq -\\ijk\\isk\\nu(\\gamma-2)\\Dk^2$ and $\\ijk\\isk(1-\\nu)(\\Dkun^2-\\Dk^2)\\leq \\ijk\\isk(1-\\nu)(\\tau^{-2}-1)\\Dk^2$, leading to \n\\begin{equation}\\label{dixseptEq}\n\\ijk\\isk\\left(\\Phikun-\\Phik\\right)\\leq\\ijk\\isk\\left(-\\nu(\\gamma-2)+(1-\\nu)(\\tau^{-2}-1)\\right)\\Dk^2\\leq -\\ijk\\isk(1-\\nu)(1-\\tau^2)\\Dk^2,\n\\end{equation}\nwhere the last inequality follows from the above choice of~$\\nu$ along with the fact that $1-\\tau^2<\\tau^{-2}-1$. \\\\\n","sha256_utf8":"11c36ac5ba6de0445974d41f929ee8ee68649dce29979afd87098761baeea9e4"}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","title":"好估计且失败时的势函数下降","source_units":[{"start":194397,"end":194671,"start_line":1909,"end_line":1912,"raw_tex":"{{Subcase~$1ii$}:} {\\it (Unsuccessful iteration, $\\iskc=1$).} The parameter~$\\Dk$ is decreased according to $\\Dkun=\\tau\\Dk$, and $f(\\Xkun)=f(\\Xk)$, whence\n\\begin{equation}\\label{dixhuitEq}\n\\ijk\\iskc\\left(\\Phikun-\\Phik\\right)=-\\ijk\\iskc(1-\\nu)(1-\\tau^2)\\Dk^2.\n\\end{equation}\n","sha256_utf8":"8e88a31702915c8b93ea291205265cd9985f32f55cab5306d574e53ce10471c5"}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","title":"合并好估计情形并取条件期望","source_units":[{"start":194671,"end":195012,"start_line":1913,"end_line":1917,"raw_tex":"Equations~\\eqref{dixseptEq} and~\\eqref{dixhuitEq} lead to\n\\begin{equation}\\label{vingtEq}\n\\ijk\\left(\\Phikun-\\Phik\\right)\\leq -\\ijk(1-\\nu)(1-\\tau^2)\\Dk^2.\n\\end{equation}\nTaking the conditional expectation with respect to $\\fqdalgebra$ on both sides of~\\eqref{vingtEq} and using Assumption~\\ref{efkfAssump}-{\\it (i)} lead to~\\eqref{treizeEq}.\n","sha256_utf8":"0a7d9564e304e8a942ec872e9d7608260a3dd5275613f17e891e75c1694ede59"}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","title":"坏估计情形的漂移控制目标","source_units":[{"start":195012,"end":195461,"start_line":1918,"end_line":1921,"raw_tex":"{\\bf Case~2:} {\\it (Bad estimates, $\\ijkc=1$).} Because of bad estimates, an iterate leading to an increase in~$f$ and~$\\Dk$, and hence in~$\\Phik$, can be accepted by Algorithm~\\ref{algoStoDARS}. Such an increase is controlled by using Assumption~\\ref{efkfAssump}-{\\it (ii)}. The overall goal is to show that almost surely,\n\\begin{equation}\\label{vingtEtUnEq}\n\\E{\\ijkc\\left(\\Phikun-\\Phik\\right)|\\fqdalgebra}\\leq 2\\nu(1-\\beta_f)\\Dk^2.\n\\end{equation}\n","sha256_utf8":"6652c9c403aefe816d022117fd8d2cb58d66102c8aa7218badb5b89fdf8cdafe"}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","title":"坏估计且成功时以误差控制增长","source_units":[{"start":195461,"end":196424,"start_line":1922,"end_line":1933,"raw_tex":"{{Subcase~$2i$}:} {\\it (Successful iteration, $\\isk=1$).} Since $\\Fuk-\\Fok\\leq-\\gamma\\ef\\Dk^2$, then \n\\[\\ijkc\\isk\\frac{\\nu}{\\ef}(f(\\Xkun)-f(\\Xk))\\leq\\ijkc\\isk\\nu\\left(-\\gamma\\Dk^2+\\frac{1}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right)\\right).\\]\nRecalling that $\\Dkun=\\min\\accolade{\\tau^{-1}\\Dk,\\dmax}$, and by noticing that $-\\nu\\gamma+(1-\\nu)(\\tau^{-2}-1)\\leq 0$ due to the above choice of $\\nu$, it holds that\n% \\begin{equation}\\label{vingtcinqEq}\n% \\begin{split}\n% \\ijkc\\isk\\left(\\Phikun-\\Phik\\right)&\\leq \\ijkc\\isk\\left(\\left(-\\nu\\gamma+(1-\\nu)(\\tau^{-2}-1) \\right)\\Dk^2+\\frac{\\nu}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right) \\right)\\\\\n% &\\leq \\ijkc\\isk\\left(\\frac{\\nu}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right) \\right)\n% \\end{split}\n% \\end{equation}\n\\begin{equation}\\label{vingtcinqEq}\n\\ijkc\\isk\\left(\\Phikun-\\Phik\\right)\\leq \\ijkc\\isk\\frac{\\nu}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right).\n\\end{equation}\n","sha256_utf8":"fdde04d031d058510efff0a17dabbf28431027d5896eeef758cd53abb93322dc"}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","title":"合并坏估计的成功与失败情形","source_units":[{"start":196424,"end":196930,"start_line":1934,"end_line":1937,"raw_tex":"{{Subcase~$2ii$}:} {\\it (Unsuccessful iteration, $\\iskc=1$).} The upper bound on the change in~$\\Phik$ follows from~\\eqref{dixhuitEq} by replacing~$\\ijk$ with~$\\ijkc$. Thus, $\\ijkc\\iskc\\left(\\Phikun-\\Phik\\right)\\leq \\ijkc\\iskc\\frac{\\nu}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right)$, which, combined with~\\eqref{vingtcinqEq}, yields\n\\begin{equation}\\label{vingtseptEq}\n\\ijkc\\left(\\Phikun-\\Phik\\right)\\leq \\ijkc\\frac{\\nu}{\\ef}\\left(\\abs{f(\\Xkun)-\\Fuk} + \\abs{\\Fok-f(\\Xk)}\\right).\n\\end{equation}\n","sha256_utf8":"6580d6241c9471c294c73891896497a2fa1e7687fe396a28e9816c321a1abeed"}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","title":"估计误差期望界推出总负漂移","source_units":[{"start":196930,"end":197379,"start_line":1938,"end_line":1940,"raw_tex":"Taking the conditional expectation with respect to $\\fqdalgebra$ on both sides of~\\eqref{vingtseptEq} and applying Lemma~\\ref{SDDSlemma1} lead to~\\eqref{vingtEtUnEq}. Then, combining~\\eqref{treizeEq} and~\\eqref{vingtEtUnEq} yields\n\\[\\E{\\left(\\Phikun-\\Phik\\right)|\\fqdalgebra}\\leq (-\\beta_f(1-\\nu)(1-\\tau^2)+2\\nu(1-\\beta_f))\\Dk^2\\leq -\\varrho\\Dk^2, \\]\nwhere the last inequality follows from the above choice of $\\beta_f$, and the proof is completed.\n","sha256_utf8":"49c1008d07c1e52f47e5e7daad10fe3b9866e4541f29e75a82b24d92eac825ac"}]},{"id":"S077","semantic_type":"proposition","kind":"proposition","title":"估计下降蕴含真实下降","source_units":[{"start":97575,"end":97986,"start_line":1198,"end_line":1203,"raw_tex":"\\begin{proposition}\\label{decreaseProp1}(Inspired by~\\cite[Proposition~1]{dzahini2020expected})\nLet $\\gamma>2$ be a constant, and let $\\fok$ and $\\fuk$ be $\\ef$-accurate estimates of $f(\\xk)$ and $f(\\xk+\\dk\\ub)$, respectively: \n\\begin{equation}\\label{decreaseEqu}\n\\mbox{If}\\quad \\fuk-\\fok\\leq -\\gamma\\ef\\dk^2,\\qquad\\mbox{then}\\qquad f(\\xk+\\dk\\ub)-f(\\xk)\\leq -(\\gamma-2)\\ef\\dk^2.\n\\end{equation}\n\\end{proposition}","sha256_utf8":"a27ea7eb583bc22b198f143f7aab0a251beb2a132ed7a7f4e97f22e512dc781e"}]},{"id":"S078","semantic_type":"proof","kind":"proof","title":"真实下降命题的作者证明","source_units":[{"start":97990,"end":98113,"start_line":1205,"end_line":1207,"raw_tex":"\\begin{proof}\nThe proof trivially follows that of~\\cite[Proposition~1]{dzahini2020expected} with minor changes.\n\\end{proof}","sha256_utf8":"1da5f7cad731610a25f92bfa2d10e7d377c8ae39ea72a7bf8a13afaa98a0df3c"}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","title":"调用既有估计下降论证","source_units":[{"start":97990,"end":98113,"start_line":1205,"end_line":1207,"raw_tex":"\\begin{proof}\nThe proof trivially follows that of~\\cite[Proposition~1]{dzahini2020expected} with minor changes.\n\\end{proof}","sha256_utf8":"1da5f7cad731610a25f92bfa2d10e7d377c8ae39ea72a7bf8a13afaa98a0df3c"}]},{"id":"S080","semantic_type":"lemma","kind":"lemma","title":"坏估计误差的条件一阶矩界","source_units":[{"start":126480,"end":126930,"start_line":1396,"end_line":1402,"raw_tex":"\\begin{lemma}\\label{SDDSlemma1}\\cite[Lemma~1]{dzahini2020expected}\nUnder Assumption~\\ref{efkfAssump}, the following hold for all $k\\geq 0$ for the random process generated by Algorithm~\\ref{algoStoDARS}:\n\\begin{eqnarray*}\n\\E{\\ijkc\\abs{\\Fok-f(\\Xk)}|\\fqdalgebra}&\\leq& \\ef(1-\\beta_f)\\Dk^2\\quad\\mbox{almost surely, and}\\\\\n\\E{\\ijkc\\abs{\\Fuk-f(\\Xk+\\Dk\\urandom)}|\\fqdalgebra}&\\leq& \\ef(1-\\beta_f)\\Dk^2\\quad\\mbox{almost surely.}\n\\end{eqnarray*} \n\\end{lemma}","sha256_utf8":"708e01ccb37c8337bc3bab041f8c476bc5a6dc8adf44d374e39c07a84a608e3b"}]},{"id":"S081","semantic_type":"proof","kind":"proof","title":"坏估计误差界的作者证明","source_units":[{"start":126932,"end":127035,"start_line":1404,"end_line":1406,"raw_tex":"\\begin{proof}\nThe proof is almost identical to that of~\\cite[Lemma~1]{dzahini2020expected}.\n\\end{proof}","sha256_utf8":"92143ad4adca929b5444f8a0c022cb3dfbc870b4e3efe59d34696526f907b2ef"}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","title":"调用既有条件误差矩论证","source_units":[{"start":126932,"end":127035,"start_line":1404,"end_line":1406,"raw_tex":"\\begin{proof}\nThe proof is almost identical to that of~\\cite[Lemma~1]{dzahini2020expected}.\n\\end{proof}","sha256_utf8":"92143ad4adca929b5444f8a0c022cb3dfbc870b4e3efe59d34696526f907b2ef"}]},{"id":"S083","semantic_type":"theorem","kind":"theorem","title":"更新报酬期望停止时间界","source_units":[{"start":131966,"end":132194,"start_line":1469,"end_line":1472,"raw_tex":"\\begin{theorem}\\cite[Theorem~2]{blanchet2016convergence}\\label{theorComplexity1}\nUnder Assumption~\\ref{assumptionRenewal}, \n$\\E{\\Tepsilon}\\leq \\frac{q}{2q-1}\\cdot\\frac{\\PhiZero}{\\varrho h\\left(\\depsilon\\right)}+1.$\n\\end{theorem}","sha256_utf8":"8b7d6d2893789a90283a2aeba1676b9467c59dd542aeb70fe31854db14c54ca0"}]},{"id":"S084","semantic_type":"external_fact","kind":"external_fact","title":"Haar分布标准性质","source_units":[],"declared_external":true,"declaration_text":["Haar 正交矩阵的分布在固定正交矩阵的左右乘下不变，其列向量均匀分布于单位球面；矩阵元素同分布，均值为零且二阶矩为 1/n。采用 MeckesElizHaar2019 的标准性质及 B0018 所述矩公式。"]},{"id":"S085","semantic_type":"external_fact","kind":"external_fact","title":"球面集中标准定理","source_units":[],"declared_external":true,"declaration_text":["对单位球面上的 L-Lipschitz 实值函数 F 和球面均匀随机向量 ξ，存在绝对常数 η>0，使 P(|F(ξ)-E F(ξ)|≥t)≤2 exp(-η n t²/L²)。采用 vershynin2018HDP 中 B0019 引述的形式。"]},{"id":"S086","semantic_type":"external_fact","kind":"external_fact","title":"独立输入条件代入","source_units":[],"declared_external":true,"declaration_text":["若 V 对 σ-代数 G 可测，M 独立于 G，且 ψ(V,M) 可积，则 E[ψ(V,M)|G]=g(V)，其中 g(v)=E[ψ(v,M)]。采用 bhattacharya2007basic 中 B0030 引述的条件期望代入公式。"]},{"id":"S087","semantic_type":"external_fact","kind":"external_fact","title":"更新报酬停止时间定理","source_units":[],"declared_external":true,"declaration_text":["对满足 B0076 的非负离散过程，若偏置参数 q>1/2，则 E[Tε]≤q/(2q-1)·Φ0/(ρ h(δε))+1。该边界定理为 blanchet2016convergence, Theorem 2；B0094 引用的 DzaWildSub2022, Theorem 5.4 使用同一停止时间界进行复杂度代入。"]},{"id":"S088","semantic_type":"external_fact","kind":"external_fact","title":"准确估计的下降传递","source_units":[],"declared_external":true,"declaration_text":["若两个函数估计的绝对误差分别不超过 ε_f δ²，而其估计差不超过 -γ ε_f δ²，则真实函数差不超过 -(γ-2)ε_f δ²。这是三角不等式的直接应用，也是 B0048 委托给 dzahini2020expected, Proposition 1 的论证。"]},{"id":"S089","semantic_type":"external_fact","kind":"external_fact","title":"坏事件上的条件矩控制","source_units":[],"declared_external":true,"declaration_text":["条件 Cauchy–Schwarz 给出 E[1_B|e||G]≤P(B|G)^(1/2) E[|e|²|G]^(1/2)。结合 B0062 的坏事件概率和二阶矩界得到 B0064。B0066 将此标准论证委托给 dzahini2020expected, Lemma 1。"]},{"id":"S090","semantic_type":"external_fact","kind":"external_fact","title":"非负漂移的求和推论","source_units":[],"declared_external":true,"declaration_text":["若非负可积势函数满足 E[Φ(k+1)-Φ(k)|Gk]≤-ρ δk²，ρ>0，则对有限时段取全期望并求和，再用单调收敛，得到 E[Σ δk²]≤E[Φ0]/ρ，因而 Σ δk²<∞ 几乎处处且 δk→0。B0140 引用 dzahini2020expected, Theorem 3 说明该标准推论。"]},{"id":"S091","semantic_type":"external_fact","kind":"external_fact","title":"基础范数与积分工具","source_units":[],"declared_external":true,"declaration_text":["使用 Euclidean Cauchy–Schwarz 不等式、线段上的实值函数中值定理，以及非负随机变量 Y 的尾积分恒等式 E[Y]=∫₀∞P(Y≥t)dt。正生成集的余弦测度还保证 B0009 的方向内积界。"]},{"id":"S092","semantic_type":"problem_setting","kind":"problem_setting","title":"无约束复杂度适用域","source_units":[{"start":128472,"end":129290,"start_line":1437,"end_line":1437,"raw_tex":"By using an existing supermartingale-based framework proposed by Blanchet et al.~\\cite{blanchet2016convergence}, this section derives in Theorem~\\ref{complexResult} an upper bound on the expected number of iterations required by Algorithm~\\ref{algoStoDARS} to drive~$\\normii{\\nabla f(\\Xk)}$ below a given threshold $\\epsilon\\in (0,1)$, when applied to Problem~\\eqref{probl1} in the particular case~$\\bcX:=\\rn$. Following~\\cite{dzahini2020expected,DzaWildSub2022,paquette2018stochastic}, Section~\\ref{sec51} introduces a general renewal-reward martingale process and its associated {\\it stopping time} along with related assumptions. Then Section~\\ref{sec52} demonstrates that these assumptions are satisfied by the stochastic process generated by Algorithm~\\ref{algoStoDARS}, leading to the desired complexity result.\n","sha256_utf8":"e905b8ca9e690cb4d9736f8b327007601386070e5a74c3270e65c5515bf94efd"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S003","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S061","target_id":"S005","kind":"supports"},{"source_id":"S083","target_id":"S005","kind":"supports"},{"source_id":"S026","target_id":"S005","kind":"supports"},{"source_id":"S092","target_id":"S005","kind":"supports"},{"source_id":"S006","target_id":"S092","kind":"supports"},{"source_id":"S007","target_id":"S016","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S091","target_id":"S009","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S013","target_id":"S003","kind":"supports"},{"source_id":"S016","target_id":"S003","kind":"supports"},{"source_id":"S020","target_id":"S003","kind":"supports"},{"source_id":"S023","target_id":"S003","kind":"supports"},{"source_id":"S013","target_id":"S004","kind":"supports"},{"source_id":"S016","target_id":"S004","kind":"supports"},{"source_id":"S022","target_id":"S004","kind":"supports"},{"source_id":"S065","target_id":"S004","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S005","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S030","kind":"supports"},{"source_id":"S033","target_id":"S030","kind":"supports"},{"source_id":"S034","target_id":"S030","kind":"supports"},{"source_id":"S035","target_id":"S030","kind":"supports"},{"source_id":"S036","target_id":"S030","kind":"supports"},{"source_id":"S012","target_id":"S032","kind":"supports"},{"source_id":"S084","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S028","target_id":"S033","kind":"supports"},{"source_id":"S028","target_id":"S034","kind":"supports"},{"source_id":"S027","target_id":"S034","kind":"supports"},{"source_id":"S091","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S084","target_id":"S027","kind":"supports"},{"source_id":"S085","target_id":"S028","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S041","target_id":"S038","kind":"supports"},{"source_id":"S009","target_id":"S039","kind":"supports"},{"source_id":"S011","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S011","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S037","target_id":"S044","kind":"supports"},{"source_id":"S029","target_id":"S044","kind":"supports"},{"source_id":"S012","target_id":"S044","kind":"supports"},{"source_id":"S008","target_id":"S044","kind":"supports"},{"source_id":"S086","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S045","target_id":"S048","kind":"supports"},{"source_id":"S042","target_id":"S048","kind":"supports"},{"source_id":"S029","target_id":"S048","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S046","target_id":"S051","kind":"supports"},{"source_id":"S015","target_id":"S051","kind":"supports"},{"source_id":"S016","target_id":"S051","kind":"supports"},{"source_id":"S023","target_id":"S051","kind":"supports"},{"source_id":"S013","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S055","target_id":"S053","kind":"supports"},{"source_id":"S056","target_id":"S053","kind":"supports"},{"source_id":"S057","target_id":"S053","kind":"supports"},{"source_id":"S058","target_id":"S053","kind":"supports"},{"source_id":"S059","target_id":"S053","kind":"supports"},{"source_id":"S060","target_id":"S053","kind":"supports"},{"source_id":"S013","target_id":"S055","kind":"supports"},{"source_id":"S023","target_id":"S055","kind":"supports"},{"source_id":"S026","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S014","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S025","target_id":"S057","kind":"supports"},{"source_id":"S091","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S055","target_id":"S059","kind":"supports"},{"source_id":"S025","target_id":"S059","kind":"supports"},{"source_id":"S015","target_id":"S059","kind":"supports"},{"source_id":"S026","target_id":"S059","kind":"supports"},{"source_id":"S091","target_id":"S059","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S024","target_id":"S060","kind":"supports"},{"source_id":"S026","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S063","target_id":"S062","kind":"supports"},{"source_id":"S064","target_id":"S062","kind":"supports"},{"source_id":"S022","target_id":"S063","kind":"supports"},{"source_id":"S023","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S026","target_id":"S064","kind":"supports"},{"source_id":"S013","target_id":"S064","kind":"supports"},{"source_id":"S052","target_id":"S064","kind":"supports"},{"source_id":"S049","target_id":"S064","kind":"supports"},{"source_id":"S017","target_id":"S064","kind":"supports"},{"source_id":"S018","target_id":"S064","kind":"supports"},{"source_id":"S016","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"},{"source_id":"S067","target_id":"S066","kind":"supports"},{"source_id":"S068","target_id":"S066","kind":"supports"},{"source_id":"S069","target_id":"S066","kind":"supports"},{"source_id":"S070","target_id":"S066","kind":"supports"},{"source_id":"S071","target_id":"S066","kind":"supports"},{"source_id":"S072","target_id":"S066","kind":"supports"},{"source_id":"S073","target_id":"S066","kind":"supports"},{"source_id":"S074","target_id":"S066","kind":"supports"},{"source_id":"S075","target_id":"S066","kind":"supports"},{"source_id":"S076","target_id":"S066","kind":"supports"},{"source_id":"S019","target_id":"S068","kind":"supports"},{"source_id":"S090","target_id":"S068","kind":"supports"},{"source_id":"S076","target_id":"S068","kind":"supports"},{"source_id":"S013","target_id":"S068","kind":"supports"},{"source_id":"S017","target_id":"S069","kind":"supports"},{"source_id":"S077","target_id":"S070","kind":"supports"},{"source_id":"S013","target_id":"S070","kind":"supports"},{"source_id":"S018","target_id":"S070","kind":"supports"},{"source_id":"S013","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S072","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S018","target_id":"S072","kind":"supports"},{"source_id":"S016","target_id":"S072","kind":"supports"},{"source_id":"S018","target_id":"S073","kind":"supports"},{"source_id":"S013","target_id":"S074","kind":"supports"},{"source_id":"S018","target_id":"S074","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S071","target_id":"S075","kind":"supports"},{"source_id":"S075","target_id":"S076","kind":"supports"},{"source_id":"S080","target_id":"S076","kind":"supports"},{"source_id":"S072","target_id":"S076","kind":"supports"},{"source_id":"S018","target_id":"S076","kind":"supports"},{"source_id":"S078","target_id":"S077","kind":"supports"},{"source_id":"S079","target_id":"S078","kind":"supports"},{"source_id":"S014","target_id":"S079","kind":"supports"},{"source_id":"S088","target_id":"S079","kind":"supports"},{"source_id":"S081","target_id":"S080","kind":"supports"},{"source_id":"S082","target_id":"S081","kind":"supports"},{"source_id":"S018","target_id":"S082","kind":"supports"},{"source_id":"S017","target_id":"S082","kind":"supports"},{"source_id":"S089","target_id":"S082","kind":"supports"},{"source_id":"S087","target_id":"S083","kind":"supports"},{"source_id":"S022","target_id":"S083","kind":"supports"}]}}
{"record_id":"2404.17386v3","paper":{"title":"Stochastic Bregman Subgradient Methods for Nonsmooth Nonconvex Optimization Problems","authors":["Kuangyu Ding","Kim-Chuan Toh"],"arxiv_id":"2404.17386","version":"v3","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-025-02749-7","abstract_url":"https://arxiv.org/abs/2404.17386v3","pdf_url":"https://arxiv.org/pdf/2404.17386v3"},"context":{"raw_tex":"Given a proper and lower semicontinuous function $f:\\mathbb{R}^n\\rightarrow\\overline{\\mathbb{R}} := (-\\infty,\\infty]$, we denote its domain as $\\dom\\,f=\\{x:f(x)<\\infty\\}$. The Fenchel conjugate function of $f$ is defined as $f^*(y):=\\sup\\{\\langle x,y\\rangle - f(x):x\\in\\mathbb{R}^n\\}$. For a set $\\mathcal{S}\\subset\\mathbb{R}^n$, we use ${\\rm cl\\,\\mathcal{S}}$ to denote its closure, ${\\rm int}\\,\\mathcal{S}$ to denote the set of its interior points, and ${\\rm conv}(\\cS)$ to denote its convex hull. A function $f:\\cS\\rightarrow\\R$ is said to be of class ${\\cC}^k(\\mathcal{S})$ if it is $k$ times differentiable and the $k$-th derivative is continuous on $\\mathcal{S}$. When there is no ambiguity regarding the domain, we simply use the notation ${\\cC}^k$. We let $\\mathcal{C}(A,B)$ be the set of continuous mappings from set $A$ to set $B$. We use $\\|\\cdot\\|$ to denote the Euclidean norm for vectors and the Frobenius matrix norm for matrices. The $d$-dimensional unit ball is denoted by $\\mathbb{B}^d$. The distance between a point $w$ and a set $A$ is denoted by $\\dist(w,A):=\\inf\\{\\|w-u\\|:\\;u\\in A\\}$. We use the convention $\\dist(w,\\emptyset):=\\infty$. {The Minkowski sum of two sets $A$ and $B$ is denoted as \n$A+B:=\\{u+v:\\;u\\in A,\\;v\\in B\\}$. We use $\\alpha A:=\\{\\alpha u:\\;u\\in A\\}$ to denote the scaled set of $A$ by a given scalar $\\alpha$.} For a positive sequence $\\{\\eta_k\\}$, we define $\\lambda_\\eta(0):=0$, $\\lambda_\\eta(k):=\\sum_{i=0}^{k-1}\\eta_i$ for $k\\geq1$, and $\\Lambda_\\eta(t):=\\sup\\{k:\\lambda_\\eta(k)\\leq t\\}$. In other word, $\\Lambda_\\eta(t)=k$ if and only if $\\lambda_\\eta(k)\\leq t<\\lambda_\\eta(k+1)$. {Given a convex function $g$, we say $x$ is a $\\nu$-optimal solution of the problem $\\min_{x\\in\\R^n}\\;g(x)$ if ${\\rm dist}(0,\\partial g(x))\\leq\\nu$.}\n\n\n\\begin{definition}\\label{graph-def}\nLet $\\mathcal{X}\\subset\\mathbb{R}^n$. A set-valued mapping $S:\\mathcal{X}\\rightrightarrows\\mathbb{R}^m$ is said to be closed if its graph, defined by\n\\[\n{\\rm graph}(S):=\\{(x,y)\\in\\R^n\\times\\R^m:y\\in S(x)\\}\n\\]\nis a closed set in $\\R^n\\times\\R^m$. Give a nonnegative $\\delta$, the $\\delta$-perturbed set of $S$ is defined by \n\\[\nS^\\delta(x):=\\cup_{\\{y\\in\\R^n:\\norm{y-x}\\leq\\delta\\}}\\left(S(y)+\\delta\\mathbb{B}^m\\right).\n\\]\n$S:\\mathcal{X}\\rightrightarrows\\mathbb{R}^m$ is called outer semicontinuous at $\\bar{x}\\in\\mathcal{X}$ if for any sequence $x_i\\overset{\\mathcal{X}}{\\rightarrow}\\bar x$ and $v_i\\in S(x_i)$ converging to some $\\bar{v}\\in\\mathbb{R}^m$, we have $\\bar{v}\\in S(\\bar{x})$. $S$ is said to be outer semicontinuous if it is outer semicontinuous everywhere over $\\mathcal{X}$.\n\\end{definition}\n\n\\begin{definition}\\label{Bregman-distance-def}\n(Kernel function and Bregman distance over $\\R^n$).  A function $\\phi:\\R^n\\rightarrow\\R$ is called a kernel function over $\\R^n$, if $\\phi$ is convex and $\\phi\\in\\cC^1(\\R^n)$. The Bregman distance \\cite{bregman1967relaxation} generated by $\\phi$ is denoted as $\\cD_\\phi(x,y):\\R^n\\times\\R^n\\rightarrow[0,+\\infty)$, where\n\\[\n\\cD_\\phi(x,y)=\\phi(x)-\\phi(y)-\\inprod{\\nabla\\phi(y)}{x-y}.\n\\]\n\\end{definition}\n\n\\begin{definition}\\label{Legendre}\n(Legendre kernel over $\\R^n$). Let $\\phi$ be a kernel function over $\\R^n$, such that $\\lim_{k\\rightarrow\\infty}\\|\\nabla\\phi({x_k})\\|=\\infty$ whenever $\\{{x_k}\\}_{k\\in\\mathbb N}$ satisfies $\\lim_{k\\rightarrow\\infty}\\norm{x_k}=\\infty$. The function $\\phi$ is called a Legendre function over $\\R^n$ if it is also {strictly} convex on $\\R^n$.\n\\end{definition}\n\nWe end this subsection by introducing the concept of functional convergence over any compact set, as discussed in literature such as \\cite{benaim2005stochastic,duchi2018stochastic,davis2020stochastic}. Given a sequence of mappings $f_n\\in\\cC(\\mathbb{R}_+,\\mathbb{R}^n)$, we say that $f_n$ converges to $f$ in $\\cC(\\R_+,\\R^n)$ if, for any $T>0$, it holds that\n\\[\n\\lim_{n\\rightarrow\\infty}\\sup_{t\\in[0,T]}\\|f_n(t)-f(t)\\|=0.\n\\]\n\n\n\\begin{equation}\n    \\label{Eq:Breg_DI}\n    \\frac{d}{dt}\\nabla\\phi(x(t))\\in-\\mathcal{H}(x(t)), \\text{ for almost all $t\\geq0$.}\n\\end{equation}\n\nAny absolutely continuous solution to \\eqref{Eq:Breg_DI} is termed a trajectory of \\eqref{Eq:Breg_DI}. The stable set of \\eqref{Eq:Breg_DI} is defined as \n\\begin{equation}\n\\label{eq:def_stable_set}\n\\cH^{-1}(0):=\\{x\\in\\R^n:0\\in\\cH(x)\\}.\n\\end{equation}\n\n\\begin{equation}\n   \\label{Eq:Breg_general_iterative}  \\nabla\\phi(x_{k+1})=\\nabla\\phi(x_k)-\\eta_k(d_k+\\xi_k),\n\\end{equation}\n\nwhere $d_k$ is an evaluation of $\\cH(x_k)$ with possible inexactness, and $\\xi_k$ is the stochastic noise. This formulation, referred to as the general Bregman-type method, is notable for the versatile choices of $\\phi$ and $\\cH$. For the Bregman counterpart of the interpolated process \\eqref{Eq:x(t)_def} in the Euclidean setting, we introduce the linear interpolation for the dual sequence $\\{\\nabla\\phi(x_k)\\}$:\n\\begin{equation}\n \\label{Eq:interpolation}\n x(t):=\\nabla\\phi^*\\left(\\nabla\\phi(x_k)+\\frac{t-\\lambda_\\eta(k)}{\\eta_k}(\\nabla\\phi(x_{k+1})-\\nabla\\phi(x_k))\\right),\\;t\\in[\\lambda_\\eta(k),\\lambda_\\eta(k+1)). \n\\end{equation}\n\nIf $\\phi^*\\in\\cC^1(\\R^n)$, then \\eqref{Eq:interpolation} is well defined. Let $x^t(\\cdot)$ denote the time-shifted curve of the interpolated process, i.e. $x^t(\\cdot)=x(t+\\cdot)$. We make the following assumptions on \\eqref{Eq:Breg_DI} and \\eqref{Eq:Breg_general_iterative} to ensure that the iterative sequence generated by \\eqref{Eq:Breg_general_iterative} tracks a trajectory of \\eqref{Eq:Breg_DI} asymptotically.\n\\begin{assumption}\n\\label{Assumption: DI}\n    \\begin{enumerate}\n        \\item {$\\phi$ is a supercoecive Legendre kernel function over $\\R^n$}, and $\\nabla\\phi$ is differentiable almost everywhere.\n        \\item The sequences $\\{x_k\\}$, $\\{\\nabla\\phi(x_k)\\}$ and $\\{d_k\\}$ are uniformly bounded.\n        \\item The stepsize $\\{\\eta_k\\}$ satisfies $\\sum_{k=0}^\\infty\\eta_k=\\infty$ and $\\lim_{k\\rightarrow\\infty}\\eta_k=0$.\n        \\item For any $T>0$, the noise sequence $\\{\\xi_k\\}$ satisfies \n        \\begin{equation}\n            \\lim_{s\\rightarrow\\infty}\\sup_{s\\leq i\\leq\\Lambda_\\eta(\\lambda_\\eta(s)+T)}\\norm{\\sum_{k=s}^i\\eta_k\\xi_k}=0.\n            \\label{Eq:noise_cond}\n        \\end{equation}       \n        \\item The set-valued mapping $\\mathcal{H}$ has a closed graph. Additionally, for any unbounded increasing sequence $\\{k_j\\}$ such that $\\{x_{k_j}\\}$ converges to $\\bar x$, it holds that \n        \\begin{equation}\n            \\lim_{N\\rightarrow\\infty}{\\rm dist}\\left(\\frac{1}{N}\\sum_{j=1}^Nd_{k_j},\\mathcal{H}(\\bar x)\\right)=0.\n        \\end{equation}\n    \\end{enumerate}\n\\end{assumption}\n\n\\begin{remark}\n    \\label{rmk:assumption_DI}\n    \\begin{enumerate}\n        \\item {If the Legendre function $\\phi$ is supercoecive, i.e. $\\lim_{\\|u\\|\\rightarrow\\infty}\\frac{\\phi(u)}{\\|u\\|}=\\infty$, then by \\cite[Theorem 26.5,\\,Corollary 13.3.1]{rockafellar1997convex}, $\\phi^*\\in\\cC^1(\\R^n)$ is strictly convex, and $(\\nabla\\phi)^{-1}=\\nabla\\phi^*$.}\n        \\item Uniform boundedness of either $\\{x_k\\}$ or $\\{\\nabla\\phi(x_k)\\}$ may suffice under mild conditions. For example, if $\\phi$ is locally strongly convex, then the uniform boundedness of $\\{\\nabla\\phi(x_k)\\}$ implies the uniform boundedness of $\\{x_k\\}$. Conversely, if $\\nabla\\phi$ is locally Lipschitz continuous, then the uniform boundedness of $\\{x_k\\}$ leads to the uniform boundedness of $\\{\\nabla\\phi(x_k)\\}$.\n        \\item For a martingale difference noise sequence $\\{\\xi_k\\}$,  i.e. $\\E[\\xi_{k+1}|\\cF_k]=0$ holds  almost surely,  as shown in \\cite{benaim2005stochastic}, \\eqref{Eq:noise_cond} can be ensured almost surely under one of the following conditions:\n        \\begin{enumerate}\n            \\item Uniform boundedness of $\\{\\xi_k\\}$ with $\\eta_k=o\\left(\\frac{1}{\\log k}\\right)$ as utilized in \\cite{castera2021inertial,xiao2023adam,xiao2023convergence}.\n            \\item Variance-bounded $\\{\\xi_k\\}$, i.e. $\\E[\\norm{\\xi_{k+1}}^2|\\cF_k]\\leq\\sigma^2$ for some $\\sigma>0$, and square-summable $\\{\\eta_k\\}$, i.e. $\\sum_{k=0}^\\infty\\eta_k^2<\\infty$, as employed in \\cite{davis2020stochastic}.\n        \\end{enumerate}\n        \\item Assumption \\ref{Assumption: DI}(5) describes how $d_k$ approximates $\\cH(x_k)$. This assumption is quite mild. A sufficient condition for it hold is that $d_k\\in\\cH^{\\delta_k}(x_k)$ with $\\lim_{k\\to\\infty}\\delta_k=0$. \n    \\end{enumerate}\n\\end{remark}\n\n\\begin{assumption}\\label{assumption_Sard_Lyapunov}\n There exists a continuous function $\\Psi:\\R^n\\to\\R$, such that the following conditions hold:\n \\begin{enumerate}\n     \\item (Weak Morse-Sard). The set \n     {$\\{\\Psi(x): \\mbox{for $x$ such that $0\\in\\cH(x)$} \\}$}  has empty interior in $\\R$.\n     \\item (Lyapunov function). $\\Psi$ is lower bounded, i.e. $\\liminf_{x\\in\\R^n}\\Psi(x)>-\\infty$. For any trajectory $z(t)$ of the differential inclusion \\eqref{Eq:Breg_DI} with $z(0)\\notin\\cH^{-1}(0)$,  there exists $T > 0$ such that\n     \\[\n     \\Psi(z(T))<\\sup_{t\\in[0,T]}\\Psi(z(t))\\leq\\Psi(z(0)).\n     \\]\n \\end{enumerate}\n\\end{assumption}\n\nTo establish under Assumption \\ref{Assumption: DI} that, the discrete sequence generated by the general Bregman-type method \\eqref{Eq:Breg_general_iterative} tracks a trajectory of the differential inclusion \\eqref{Eq:Breg_DI}, we introduce a piecewise constant mapping $d(\\cdot)$ defined by $d(s)=d_k$ for any $s\\in[\\lambda_\\eta(k),\\lambda_\\eta(k+1))$. We also define a time-shifted solution $x_t(\\cdot)$ to the following ordinary differential equation:\n\\[\n\\frac{d}{ds}\\nabla\\phi({x_t(s)})=-d(s)\\;for\\;all\\;s\\geq t,\\;\\mbox{with initial condition}\\;\\nabla\\phi(x_t(t))=\\nabla\\phi(x(t)).\n\\]\nThe following lemma suggests that the interpolated process $x(t)$ defined by \\eqref{Eq:interpolation} asymptotically approximates this time-shifted solution.\nWe remind the reader to note the difference between $x_t(\\cdot)$ and $x^t(\\cdot)$.\n","units":[{"start":17325,"end":19103,"start_line":221,"end_line":222,"raw_tex":"Given a proper and lower semicontinuous function $f:\\mathbb{R}^n\\rightarrow\\overline{\\mathbb{R}} := (-\\infty,\\infty]$, we denote its domain as $\\dom\\,f=\\{x:f(x)<\\infty\\}$. The Fenchel conjugate function of $f$ is defined as $f^*(y):=\\sup\\{\\langle x,y\\rangle - f(x):x\\in\\mathbb{R}^n\\}$. For a set $\\mathcal{S}\\subset\\mathbb{R}^n$, we use ${\\rm cl\\,\\mathcal{S}}$ to denote its closure, ${\\rm int}\\,\\mathcal{S}$ to denote the set of its interior points, and ${\\rm conv}(\\cS)$ to denote its convex hull. A function $f:\\cS\\rightarrow\\R$ is said to be of class ${\\cC}^k(\\mathcal{S})$ if it is $k$ times differentiable and the $k$-th derivative is continuous on $\\mathcal{S}$. When there is no ambiguity regarding the domain, we simply use the notation ${\\cC}^k$. We let $\\mathcal{C}(A,B)$ be the set of continuous mappings from set $A$ to set $B$. We use $\\|\\cdot\\|$ to denote the Euclidean norm for vectors and the Frobenius matrix norm for matrices. The $d$-dimensional unit ball is denoted by $\\mathbb{B}^d$. The distance between a point $w$ and a set $A$ is denoted by $\\dist(w,A):=\\inf\\{\\|w-u\\|:\\;u\\in A\\}$. We use the convention $\\dist(w,\\emptyset):=\\infty$. {The Minkowski sum of two sets $A$ and $B$ is denoted as \n$A+B:=\\{u+v:\\;u\\in A,\\;v\\in B\\}$. We use $\\alpha A:=\\{\\alpha u:\\;u\\in A\\}$ to denote the scaled set of $A$ by a given scalar $\\alpha$.} For a positive sequence $\\{\\eta_k\\}$, we define $\\lambda_\\eta(0):=0$, $\\lambda_\\eta(k):=\\sum_{i=0}^{k-1}\\eta_i$ for $k\\geq1$, and $\\Lambda_\\eta(t):=\\sup\\{k:\\lambda_\\eta(k)\\leq t\\}$. In other word, $\\Lambda_\\eta(t)=k$ if and only if $\\lambda_\\eta(k)\\leq t<\\lambda_\\eta(k+1)$. {Given a convex function $g$, we say $x$ is a $\\nu$-optimal solution of the problem $\\min_{x\\in\\R^n}\\;g(x)$ if ${\\rm dist}(0,\\partial g(x))\\leq\\nu$.}\n","sha256_utf8":"53b99c244878fe3edf32853e13fd4a720b738c23fc922190446c0a8df569ea17"},{"start":19625,"end":20465,"start_line":231,"end_line":241,"raw_tex":"\\begin{definition}\\label{graph-def}\nLet $\\mathcal{X}\\subset\\mathbb{R}^n$. A set-valued mapping $S:\\mathcal{X}\\rightrightarrows\\mathbb{R}^m$ is said to be closed if its graph, defined by\n\\[\n{\\rm graph}(S):=\\{(x,y)\\in\\R^n\\times\\R^m:y\\in S(x)\\}\n\\]\nis a closed set in $\\R^n\\times\\R^m$. Give a nonnegative $\\delta$, the $\\delta$-perturbed set of $S$ is defined by \n\\[\nS^\\delta(x):=\\cup_{\\{y\\in\\R^n:\\norm{y-x}\\leq\\delta\\}}\\left(S(y)+\\delta\\mathbb{B}^m\\right).\n\\]\n$S:\\mathcal{X}\\rightrightarrows\\mathbb{R}^m$ is called outer semicontinuous at $\\bar{x}\\in\\mathcal{X}$ if for any sequence $x_i\\overset{\\mathcal{X}}{\\rightarrow}\\bar x$ and $v_i\\in S(x_i)$ converging to some $\\bar{v}\\in\\mathbb{R}^m$, we have $\\bar{v}\\in S(\\bar{x})$. $S$ is said to be outer semicontinuous if it is outer semicontinuous everywhere over $\\mathcal{X}$.\n\\end{definition}","sha256_utf8":"453d67bbc115eb27c914f0d077496daca089dfe936892d8b0337f2dce872911c"},{"start":32718,"end":33166,"start_line":352,"end_line":357,"raw_tex":"\\begin{definition}\\label{Bregman-distance-def}\n(Kernel function and Bregman distance over $\\R^n$).  A function $\\phi:\\R^n\\rightarrow\\R$ is called a kernel function over $\\R^n$, if $\\phi$ is convex and $\\phi\\in\\cC^1(\\R^n)$. The Bregman distance \\cite{bregman1967relaxation} generated by $\\phi$ is denoted as $\\cD_\\phi(x,y):\\R^n\\times\\R^n\\rightarrow[0,+\\infty)$, where\n\\[\n\\cD_\\phi(x,y)=\\phi(x)-\\phi(y)-\\inprod{\\nabla\\phi(y)}{x-y}.\n\\]\n\\end{definition}","sha256_utf8":"43b4d1c96f097f790180e71614547f72429dca594ce0b6ed4d3f69af56fcf8b1"},{"start":33168,"end":33559,"start_line":359,"end_line":361,"raw_tex":"\\begin{definition}\\label{Legendre}\n(Legendre kernel over $\\R^n$). Let $\\phi$ be a kernel function over $\\R^n$, such that $\\lim_{k\\rightarrow\\infty}\\|\\nabla\\phi({x_k})\\|=\\infty$ whenever $\\{{x_k}\\}_{k\\in\\mathbb N}$ satisfies $\\lim_{k\\rightarrow\\infty}\\norm{x_k}=\\infty$. The function $\\phi$ is called a Legendre function over $\\R^n$ if it is also {strictly} convex on $\\R^n$.\n\\end{definition}","sha256_utf8":"406aa752bc0cd75f6c20810ee5200b7612697ad13fb525bdd02973cb68dc67e8"},{"start":35114,"end":35539,"start_line":381,"end_line":384,"raw_tex":"We end this subsection by introducing the concept of functional convergence over any compact set, as discussed in literature such as \\cite{benaim2005stochastic,duchi2018stochastic,davis2020stochastic}. Given a sequence of mappings $f_n\\in\\cC(\\mathbb{R}_+,\\mathbb{R}^n)$, we say that $f_n$ converges to $f$ in $\\cC(\\R_+,\\R^n)$ if, for any $T>0$, it holds that\n\\[\n\\lim_{n\\rightarrow\\infty}\\sup_{t\\in[0,T]}\\|f_n(t)-f(t)\\|=0.\n\\]\n","sha256_utf8":"d82adb80a08d75e8b3d6d3be9df772e7f1acdc62eb0254056d46758021f64504"},{"start":36949,"end":37091,"start_line":397,"end_line":400,"raw_tex":"\\begin{equation}\n    \\label{Eq:Breg_DI}\n    \\frac{d}{dt}\\nabla\\phi(x(t))\\in-\\mathcal{H}(x(t)), \\text{ for almost all $t\\geq0$.}\n\\end{equation}","sha256_utf8":"bafa07b0138ad289fa8b6dfc7caa658f465c888c594c05edc87f6bd686887d36"},{"start":37092,"end":37342,"start_line":401,"end_line":405,"raw_tex":"Any absolutely continuous solution to \\eqref{Eq:Breg_DI} is termed a trajectory of \\eqref{Eq:Breg_DI}. The stable set of \\eqref{Eq:Breg_DI} is defined as \n\\begin{equation}\n\\label{eq:def_stable_set}\n\\cH^{-1}(0):=\\{x\\in\\R^n:0\\in\\cH(x)\\}.\n\\end{equation}","sha256_utf8":"281c1eaf84e372f5fecdf9a1a00894df7caa4328356a074632dcb03c9b6470a0"},{"start":37702,"end":37826,"start_line":407,"end_line":409,"raw_tex":"\\begin{equation}\n   \\label{Eq:Breg_general_iterative}  \\nabla\\phi(x_{k+1})=\\nabla\\phi(x_k)-\\eta_k(d_k+\\xi_k),\n\\end{equation}","sha256_utf8":"576e1be34ae8bae7ad9d5fa7d9feab2862abe7a490e36a07dff50e5fb6850f6c"},{"start":37827,"end":38462,"start_line":410,"end_line":414,"raw_tex":"where $d_k$ is an evaluation of $\\cH(x_k)$ with possible inexactness, and $\\xi_k$ is the stochastic noise. This formulation, referred to as the general Bregman-type method, is notable for the versatile choices of $\\phi$ and $\\cH$. For the Bregman counterpart of the interpolated process \\eqref{Eq:x(t)_def} in the Euclidean setting, we introduce the linear interpolation for the dual sequence $\\{\\nabla\\phi(x_k)\\}$:\n\\begin{equation}\n \\label{Eq:interpolation}\n x(t):=\\nabla\\phi^*\\left(\\nabla\\phi(x_k)+\\frac{t-\\lambda_\\eta(k)}{\\eta_k}(\\nabla\\phi(x_{k+1})-\\nabla\\phi(x_k))\\right),\\;t\\in[\\lambda_\\eta(k),\\lambda_\\eta(k+1)). \n\\end{equation}","sha256_utf8":"9926c7b1445a2c05ab845a4b298b989a8005ee8227fdf38420dadc232cf394ab"},{"start":38463,"end":39972,"start_line":415,"end_line":432,"raw_tex":"If $\\phi^*\\in\\cC^1(\\R^n)$, then \\eqref{Eq:interpolation} is well defined. Let $x^t(\\cdot)$ denote the time-shifted curve of the interpolated process, i.e. $x^t(\\cdot)=x(t+\\cdot)$. We make the following assumptions on \\eqref{Eq:Breg_DI} and \\eqref{Eq:Breg_general_iterative} to ensure that the iterative sequence generated by \\eqref{Eq:Breg_general_iterative} tracks a trajectory of \\eqref{Eq:Breg_DI} asymptotically.\n\\begin{assumption}\n\\label{Assumption: DI}\n    \\begin{enumerate}\n        \\item {$\\phi$ is a supercoecive Legendre kernel function over $\\R^n$}, and $\\nabla\\phi$ is differentiable almost everywhere.\n        \\item The sequences $\\{x_k\\}$, $\\{\\nabla\\phi(x_k)\\}$ and $\\{d_k\\}$ are uniformly bounded.\n        \\item The stepsize $\\{\\eta_k\\}$ satisfies $\\sum_{k=0}^\\infty\\eta_k=\\infty$ and $\\lim_{k\\rightarrow\\infty}\\eta_k=0$.\n        \\item For any $T>0$, the noise sequence $\\{\\xi_k\\}$ satisfies \n        \\begin{equation}\n            \\lim_{s\\rightarrow\\infty}\\sup_{s\\leq i\\leq\\Lambda_\\eta(\\lambda_\\eta(s)+T)}\\norm{\\sum_{k=s}^i\\eta_k\\xi_k}=0.\n            \\label{Eq:noise_cond}\n        \\end{equation}       \n        \\item The set-valued mapping $\\mathcal{H}$ has a closed graph. Additionally, for any unbounded increasing sequence $\\{k_j\\}$ such that $\\{x_{k_j}\\}$ converges to $\\bar x$, it holds that \n        \\begin{equation}\n            \\lim_{N\\rightarrow\\infty}{\\rm dist}\\left(\\frac{1}{N}\\sum_{j=1}^Nd_{k_j},\\mathcal{H}(\\bar x)\\right)=0.\n        \\end{equation}\n    \\end{enumerate}\n\\end{assumption}","sha256_utf8":"266ab241a7210e6caa0f3044c796693e80083727d78ada4ab7423fdb17140f5e"},{"start":40032,"end":41787,"start_line":434,"end_line":446,"raw_tex":"\\begin{remark}\n    \\label{rmk:assumption_DI}\n    \\begin{enumerate}\n        \\item {If the Legendre function $\\phi$ is supercoecive, i.e. $\\lim_{\\|u\\|\\rightarrow\\infty}\\frac{\\phi(u)}{\\|u\\|}=\\infty$, then by \\cite[Theorem 26.5,\\,Corollary 13.3.1]{rockafellar1997convex}, $\\phi^*\\in\\cC^1(\\R^n)$ is strictly convex, and $(\\nabla\\phi)^{-1}=\\nabla\\phi^*$.}\n        \\item Uniform boundedness of either $\\{x_k\\}$ or $\\{\\nabla\\phi(x_k)\\}$ may suffice under mild conditions. For example, if $\\phi$ is locally strongly convex, then the uniform boundedness of $\\{\\nabla\\phi(x_k)\\}$ implies the uniform boundedness of $\\{x_k\\}$. Conversely, if $\\nabla\\phi$ is locally Lipschitz continuous, then the uniform boundedness of $\\{x_k\\}$ leads to the uniform boundedness of $\\{\\nabla\\phi(x_k)\\}$.\n        \\item For a martingale difference noise sequence $\\{\\xi_k\\}$,  i.e. $\\E[\\xi_{k+1}|\\cF_k]=0$ holds  almost surely,  as shown in \\cite{benaim2005stochastic}, \\eqref{Eq:noise_cond} can be ensured almost surely under one of the following conditions:\n        \\begin{enumerate}\n            \\item Uniform boundedness of $\\{\\xi_k\\}$ with $\\eta_k=o\\left(\\frac{1}{\\log k}\\right)$ as utilized in \\cite{castera2021inertial,xiao2023adam,xiao2023convergence}.\n            \\item Variance-bounded $\\{\\xi_k\\}$, i.e. $\\E[\\norm{\\xi_{k+1}}^2|\\cF_k]\\leq\\sigma^2$ for some $\\sigma>0$, and square-summable $\\{\\eta_k\\}$, i.e. $\\sum_{k=0}^\\infty\\eta_k^2<\\infty$, as employed in \\cite{davis2020stochastic}.\n        \\end{enumerate}\n        \\item Assumption \\ref{Assumption: DI}(5) describes how $d_k$ approximates $\\cH(x_k)$. This assumption is quite mild. A sufficient condition for it hold is that $d_k\\in\\cH^{\\delta_k}(x_k)$ with $\\lim_{k\\to\\infty}\\delta_k=0$. \n    \\end{enumerate}\n\\end{remark}","sha256_utf8":"d9d1445aa35dd14eccedc223efcea4a057cafd51e1f38e4cfd1c394e5f0f8eaa"},{"start":44621,"end":45258,"start_line":472,"end_line":482,"raw_tex":"\\begin{assumption}\\label{assumption_Sard_Lyapunov}\n There exists a continuous function $\\Psi:\\R^n\\to\\R$, such that the following conditions hold:\n \\begin{enumerate}\n     \\item (Weak Morse-Sard). The set \n     {$\\{\\Psi(x): \\mbox{for $x$ such that $0\\in\\cH(x)$} \\}$}  has empty interior in $\\R$.\n     \\item (Lyapunov function). $\\Psi$ is lower bounded, i.e. $\\liminf_{x\\in\\R^n}\\Psi(x)>-\\infty$. For any trajectory $z(t)$ of the differential inclusion \\eqref{Eq:Breg_DI} with $z(0)\\notin\\cH^{-1}(0)$,  there exists $T > 0$ such that\n     \\[\n     \\Psi(z(T))<\\sup_{t\\in[0,T]}\\Psi(z(t))\\leq\\Psi(z(0)).\n     \\]\n \\end{enumerate}\n\\end{assumption}","sha256_utf8":"b556a5ff7da80358a81ece9f3e0b8e0ca6bc289563938e11d82230807be75024"},{"start":97049,"end":97879,"start_line":1140,"end_line":1145,"raw_tex":"To establish under Assumption \\ref{Assumption: DI} that, the discrete sequence generated by the general Bregman-type method \\eqref{Eq:Breg_general_iterative} tracks a trajectory of the differential inclusion \\eqref{Eq:Breg_DI}, we introduce a piecewise constant mapping $d(\\cdot)$ defined by $d(s)=d_k$ for any $s\\in[\\lambda_\\eta(k),\\lambda_\\eta(k+1))$. We also define a time-shifted solution $x_t(\\cdot)$ to the following ordinary differential equation:\n\\[\n\\frac{d}{ds}\\nabla\\phi({x_t(s)})=-d(s)\\;for\\;all\\;s\\geq t,\\;\\mbox{with initial condition}\\;\\nabla\\phi(x_t(t))=\\nabla\\phi(x(t)).\n\\]\nThe following lemma suggests that the interpolated process $x(t)$ defined by \\eqref{Eq:interpolation} asymptotically approximates this time-shifted solution.\nWe remind the reader to note the difference between $x_t(\\cdot)$ and $x^t(\\cdot)$.\n","sha256_utf8":"9ecb7e6aaedc227656866a5d6b93441131f52d80763427883d15f084e7fe2b40"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{convergence-thm-func-val}\nSuppose Assumptions \\ref{Assumption: DI} and \\ref{assumption_Sard_Lyapunov} hold. Then any cluster point of $\\{x_k\\}$ lies in $\\mathcal{H}^{-1}(0)$ and the function values $\\{\\Psi(x_k)\\}_{k\\geq1}$ converge.\n\\end{theorem}","units":[{"start":45532,"end":45800,"start_line":487,"end_line":489,"raw_tex":"\\begin{theorem}\\label{convergence-thm-func-val}\nSuppose Assumptions \\ref{Assumption: DI} and \\ref{assumption_Sard_Lyapunov} hold. Then any cluster point of $\\{x_k\\}$ lies in $\\mathcal{H}^{-1}(0)$ and the function values $\\{\\Psi(x_k)\\}_{k\\geq1}$ converge.\n\\end{theorem}","sha256_utf8":"712fec31e4129bf8cd787d14ac19b894e48e8f3cb669584b1f09cc71b7e88517"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{theorem}\\label{subsequential-thm}\nSuppose Assumption \\ref{Assumption: DI} holds. Then, for any sequence $\\{\\tau_k\\}_{k=1}^\\infty\\subset\\R_+$, the set of shifted sequence $\\{\\nabla\\phi(x^{\\tau_k}(\\cdot))\\}_{k=1}^\\infty$ is relatively compact in $\\mathcal{C}(\\R_+,\\R^n)$. If $\\lim_{k\\rightarrow\\infty}\\tau_k=\\infty$, then any cluster point $\\bar y(\\cdot)$ of $\\{\\nabla\\phi(x^{\\tau_k}(\\cdot))\\}_{k=1}^\\infty$ belongs to $\\mathcal{C}(\\R+,\\R^n)$. Define $\\bar x(\\cdot):=\\nabla\\phi^*(\\bar y(\\cdot))$. Then there exists a measurable $\\bar d(t)\\in\\mathcal{H}(\\bar x(t))$ satisfying \n\\begin{equation}\n\\nabla\\phi(\\bar x(t))=\\nabla\\phi(\\bar x(0))-\\int_{0}^t\\bar d(\\tau)d\\tau\\quad for\\;all\\;t\\geq0.\n\\label{Eq:DI_int_form}\n\\end{equation}\nEquivalently, we have \n\\[\n\\frac{d}{dt}\\nabla\\phi(\\bar x(t))\\in-\\mathcal{H}(\\bar x(t)),\\;for\\;almost\\;all\\;t\\geq0.\n\\]\n\\end{theorem}\n\n\\begin{proof}\n{Since $\\{x_k\\}$ is bounded, let $x^*$ be any cluster point of $\\{x_k\\}$, and ${\\lim_{i\\rightarrow\\infty}}{x_{k_i}}=x^*$. By Theorem \\ref{subsequential-thm}, up to a subsequence, $\\nabla\\phi(x^{\\lambda_\\eta(k_i)}(\\cdot))\\rightarrow\\nabla\\phi(\\bar x(\\cdot))$ for some $\\bar x(\\cdot)\\in\\cC(\\R_+,\\R^n)$. Note that $x_{{k_i}}=x^{\\lambda_\\eta(k_i)}(0)$, so we have \n\\[\n\\begin{aligned}\n\\norm{\\bar x(0)-x^*}\\leq&\\limsup_{i\\rightarrow\\infty}\\left(\\norm{\\bar x(0)-x^{\\lambda_\\eta(k_i)}(0)}+\\norm{x^{\\lambda_\\eta(k_i)}(0)-x_{k_i}}+\\norm{x_{k_i}-x^*}\\right)\\\\\n=&\\limsup_{i\\rightarrow\\infty}\\left(\\norm{\\nabla\\phi^*(\\nabla\\phi(\\bar x(0)))-\\nabla\\phi^*(\\nabla\\phi(x^{\\lambda_\\eta(k_i)}(0)))}+\\norm{x^{\\lambda_\\eta(k_i)}(0)-x_{k_i}}+\\norm{x_{k_i}-x^*}\\right)\\\\\n=&0,\n\\end{aligned}\n\\]\nwhere the last equality comes from the continuity of $\\nabla\\phi^*$.} Hence $\\bar x(0)=x^*$. Suppose $x^*\\notin \\mathcal{H}^{-1}(0)$, then by Assumption \\ref{assumption_Sard_Lyapunov}, there exists $T>0$, such that \n\\[\n\\Psi(\\bar x(T))<\\sup_{t\\in[0,T]}\\Psi(\\bar x(t))\\leq\\Psi(x^*).\n\\]\nOn the other hand, by Lemma \\ref{prop:nonescape}, $\\Psi(x(t))$ converges {as $t\\to\\infty$}. Therefore, we obtain\n\\[\n\\Psi(\\bar x(T))={\\lim_{i\\rightarrow\\infty}}\\Psi(x^{\\lambda_\\eta(k_i)}(T))=\n{\\lim_{i\\rightarrow\\infty}}\n\\Psi(x({\\lambda_\\eta(k_i)}+T))=\\lim_{t\\rightarrow\\infty}\\Psi(x(t))=\\Psi(x^*), \n\\]\nwhich leads to a contradiction. Therefore, $0\\in \\mathcal{H}(x^*)$. This completes the proof.\n\\end{proof}\n\n\\begin{lemma}\n\\label{le:shift convergence}\nSuppose Assumption \\ref{Assumption: DI} holds, then for any $T>0$, it holds that\n\\begin{equation}\n\\lim_{t\\rightarrow\\infty}\\sup_{s\\in[t,t+T]}\\|\\nabla\\phi(x(s))-\\nabla\\phi(x_t(s))\\|=0.\n\\label{asymptotic-lemma}\n\\end{equation}\n\\end{lemma}\n\n\\begin{proof}\nFix an arbitrary $s\\in[t,t+T]$, let $\\tau_t=\\Lambda_\\eta(t)$, $\\tau_s=\\Lambda_\\eta(s)$. By the definition of $x_t(\\cdot)$, we have\n\\[\n\\begin{aligned}\n&\\nabla\\phi(x_t(s))=\\nabla\\phi(x(t))-\\int_{t}^{\\lambda_\\eta(\\tau_t)}d(u)du-\\int_{\\lambda_\\eta(\\tau_t)}^{\\lambda_\\eta(\\tau_s)}d(u)du-\\int_{\\lambda_\\eta(\\tau_s)}^s d(u)du\\\\\n=&\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))-\\sum_{i=\\tau_t}^{\\tau_s-1}\\eta_id_i+\\left(\\nabla\\phi(x(t))-\\int_t^{\\lambda_\\eta(\\tau_t)}d(u)du-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))\\right)-\\int_{\\lambda_\\eta(\\tau_s)}^s d(u)du\\\\\n=&\\nabla\\phi(x({\\lambda_\\eta(\\tau_s})))+\\sum_{i=\\tau_t}^{\\tau_s-1}\\eta_i\\xi_i+\\left(\\nabla\\phi(x(t))-\\int_t^{\\lambda_\\eta(\\tau_t)}d(u)du-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))\\right)-\\int_{\\lambda_\\eta(\\tau_s)}^s d(u)du\\\\\n=&\\nabla\\phi(x(s))+\\sum_{i=\\tau_t}^{\\tau_s-1}\\eta_i\\xi_i+\\left(\\nabla\\phi(x(t))-\\int_t^{\\lambda_\\eta(\\tau_t)}d(u)du-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))\\right)\\\\\n&+\\left(\\nabla\\phi(x(\\lambda_\\eta(\\tau_s)))-\\int_{\\lambda_\\eta(\\tau_s)}^sd(u)du-\\nabla\\phi(x(s))\\right).\n\\end{aligned}\n\\]\nNote that\n\\[\n\\begin{aligned}\n&\\norm{\\nabla\\phi(x(t))-\\int_t^{\\lambda_\\eta(\\tau_t)}d(u)du-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))}\\\\\n\\leq&{\\|\\nabla\\phi(x(t))-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))\\|}\n+\\int_t^{\\lambda_\\eta(\\tau_t)}\\|d(u)\\|du\n\\\\\n \\leq&{\\|\\nabla\\phi(x(\\lambda_\\eta(\\tau_t+1)))-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))\\|}\n+\\int_t^{\\lambda_\\eta(\\tau_t)}\\|d(u)\\|du\n\\\\\n \\leq&\\eta_{\\tau_t}(\\|\\xi_{\\tau_t}\\|+2\\|d_{\\tau_t}\\|),\n \\end{aligned}\n\\]\nand similarly \n$\n\\big\\|\\nabla\\phi(x(\\lambda_\\eta(\\tau_s)))-\\int_{\\lambda_\\eta(\\tau_s)}^sd(u)du-\\nabla\\phi(x(s))\\big\\|\\leq\\eta_{\\tau_s}(\\|\\xi_{\\tau_s}\\|+2\\|d_{\\tau_s}\\|).\n$\nBy Assumption \\ref{Assumption: DI}, we \nhave that $\\lim\\sup\\limits_{t\\rightarrow\\infty}\\eta_{\\tau_t}(\\|\\xi_{\\tau_t}\\|+2\\|d_{\\tau_t}\\|)=0$, $\\lim\\sup\\limits_{s\\rightarrow\\infty}\\eta_{\\tau_s}(\\|\\xi_{\\tau_s}\\|+2\\|d_{\\tau_s}\\|)=0$, and $\\lim\\limits_{t\\rightarrow\\infty}\\sup\\limits_{s\\in[t,t+T]}\\sum_{i=\\tau_t}^{\\tau_s-1}\\eta_i\\xi_i=0$. Therefore, it holds that $\\lim\\limits_{t\\rightarrow\\infty}\\sup\\limits_{s\\in[t,t+T]}\\|\\nabla\\phi(x(s))-\\nabla\\phi(x_t(s))\\|=0$, which completes the proof.\n\\end{proof}\n\n\\noindent{\\textit{Proof of Theorem \\ref{subsequential-thm}\n.}} By the definition of $x_t(\\cdot)$, it follows that $\\nabla\\phi(x_t(s))=\\nabla\\phi(x(t))-\\int_t^s y(u)du$, for all $s\\geq t$. By the boundedness of $y(s)$, Arzel$\\grave{a}$-Ascoli's theorem \\cite{rudin1953principles} ensures that $\\{\\nabla\\phi(x_t(t+\\cdot))\\}_{t\\in\\R_+}$ is relatively compact in $\\cC(\\R_+,\\R^n)$. For any subset $\\{\\tau_k\\}\\subset\\R_+$, we consider the sequence $\\{\\nabla\\phi(x^{\\tau_k}(\\cdot))\\}_{k\\in\\mathbb{N}_+}$. There are two cases to consider. Case (i): the sequence $\\{\\tau_k\\}$ has a cluster point $t$. Without loss of generality, assume that $\\lim_{k\\rightarrow\\infty}\\tau_k=t$. By the definition of $x(\\cdot)$ {in \\eqref{Eq:interpolation}} and uniform boundedness assumption, it holds that  $\\nabla\\phi(x(\\cdot))$ is Lipschitz continuous. Thus, $\\nabla\\phi(x^{\\tau_k}(\\cdot))$ converges to $\\nabla\\phi(x^t(\\cdot))$ in $\\cC(\\R_+,\\R^n)$. Case (ii): $\\lim_{k\\rightarrow\\infty}\\tau_k=\\infty$. Suppose that $\\nabla\\phi(x^{\\tau_k}(\\cdot))$ does not has any cluster point in $\\cC(\\R_+,\\R^n)$. Since $\\{\\nabla\\phi(x_{\\tau_k}(\\tau_k+\\cdot))\\}$ is relatively compact in $\\cC(\\R_+,\\R^n)$, without loss of generality, we assume that $\\lim_{k\\rightarrow\\infty}\\nabla\\phi(x_{\\tau_k}(\\tau_k+\\cdot))=\\bar y(\\cdot)$. Then, for any compact set $C\\subset\\R_+$, it follows from Lemma \\ref{le:shift convergence} that\n\\[\n\\begin{aligned}\n&\\lim_{k\\rightarrow\\infty,s\\in C}\\norm{\\nabla\\phi(x^{\\tau_k}(s))-\\bar y(s)}\\\\\n\\leq&\\lim_{k\\rightarrow\\infty,s\\in C}\\norm{\\nabla\\phi(x^{\\tau_k}(s))-\\nabla\\phi(x_{\\tau_k}(\\tau_k+s))}+\\lim_{k\\rightarrow\\infty,s\\in C}\\norm{\\nabla\\phi(x_{\\tau_k}(\\tau_k+s))-\\bar y(s)}=0,\n\\end{aligned}\n\\]\nwhich contradicts that $\\nabla\\phi(x^{\\tau_k}(\\cdot))$ does not has any cluster point in $\\cC(\\R_+,\\R^n)$. Thus, in both cases, $\\{\\nabla\\phi(x^{\\tau_k}(\\cdot))\\}$ is relatively compact in $\\cC(\\R_+,\\R^n)$. Because $\\{\\tau_k\\}$ is an arbitrary subset in $\\R_+$, we have that $\\{\\nabla\\phi(x^t(\\cdot))\\}$ is relatively compact in $\\cC(\\R_+,\\R^n)$. \n\nNext, we aim to construct a trajectory of the differential inclusion. Define the shifts $d^t(\\cdot)=d(t+\\,\\cdot)$. Consider $\\{\\tau_k\\}$ satisfying $\\tau_k\\rightarrow\\infty$, and fix $T>0$. Without loss of generality, we assume that $\\nabla\\phi(x^{\\tau_k}(\\cdot))$ converges to $\\bar y(\\cdot)$ in $\\cC(\\R_+,\\R^n)$, otherwise, we choose its convergent subsequence. The set $\\mathcal{Y}_T:=\\{d^{\\tau_k}(s),s\\in[0,T]\\}_{k\\in\\mathbb{N}}\\subset L^2([0,T])$ is bounded. Therefore, it follows from the Banach-Alaoglu theorem \\cite{rudin1991functional} that $\\cY_T$ is weakly sequentially compact, i.e. there exists a subsequence $\\{\\tau_{k_j}\\}$ and $\\bar y(\\cdot)\\in L^2([0,T])$ such that $d^{\\tau_{k_j}}(\\cdot)\\rightarrow\\bar d(\\cdot)$ weakly in $L^2([0,T])$. On the other hand, by Lemma \\ref{le:shift convergence}, we have $\\nabla\\phi(x_{\\tau_{k_j}}(\\tau_{k_j}+\\cdot))$ \nconverges to $\\bar y(\\cdot)$ in $\\cC(\\R_+,\\R^n)$. For any $\\tau\\in[0,T]$, by the definition of $x_t(\\cdot)$, we have\n\\[\n\\nabla\\phi(x_t(t+\\tau))=\\nabla\\phi(x_t(t))-\\int_0^\\tau d^t(s)ds.\n\\]\nSetting $t=\\tau_{k_j}$ and taking the limit as $k_j\\rightarrow\\infty$, we deduce that\n\\[\n\\bar y(\\tau)=\\bar y(0)-\\int_0^\\tau\\bar d(s)ds.\n\\]\nLet $\\bar x(\\cdot)=\\nabla\\phi^*(\\bar y(\\cdot))$, since $T>0$ is arbitrary, we get \\eqref{Eq:DI_int_form}. \n\nThe remaining step is to verify that $\\bar d(s)\\in\\mathcal{H}(\\bar x(s))$ for almost all $s\\geq0$. We again fix an arbitrary $T > 0$. Given that $\\mathcal{Y}_T\\subset L^2([0,T])$ is bounded, the Banach-Saks theorem \\cite{rudin1991functional} implies that for ${\\tau_k}$ (choosing a subsequence if necessary), $\\frac{1}{N}\\sum_{k=1}^Nd^{\\tau_k}(s)$ strongly converges to $\\bar d(s)$ in $L^2([0,T])$ . By the definition of $d(\\cdot)$, we have $d^{\\tau_k}(s)=d_{\\Lambda_\\eta(\\tau_k+s)}$. Now for any $s\\in[0,T]$, we have\n\\[\n\\begin{aligned}\n    &\\|\\nabla\\phi(x(\\lambda_\\eta(\\Lambda_\\eta(\\tau_k+s))))-\\nabla\\phi(\\bar x(s))\\|\\\\\n    \\leq&\\|\\nabla\\phi(x(\\lambda_\\eta(\\Lambda_\\eta(\\tau_k+s))))-\\nabla\\phi(x(\\tau_k+s))\\|+\\|\\nabla\\phi(x^{\\tau_k}(s))-\\nabla\\phi(\\bar x(s))\\|\\\\\n    \\leq&\\|\\nabla\\phi(x(\\lambda_\\eta(\\Lambda_\\eta(\\tau_k+s)+1)))-\\nabla\\phi(x(\\lambda_\\eta(\\Lambda_\\eta(\\tau_k+s))))\\|+\\|\\nabla\\phi(x^{\\tau_k}(s))-\\nabla\\phi(\\bar x(s))\\|\\\\\n    \\leq&\\eta_{\\Lambda_\\eta(\\tau_k+s)}(\\|\\xi_{\\Lambda_\\eta(\\tau_k+s)}\\|+\\|d_{\\Lambda_\\eta(\\tau_k+s)}\\|)+\\|\\nabla\\phi(x^{\\tau_k}(s))-\\nabla\\phi(\\bar x(s))\\|,\n\\end{aligned}\n\\]\nwhich converges to zero as $k\\rightarrow\\infty$. By the continuity of $\\nabla\\phi^*$, we have that $x(\\lambda_\\eta(\\Lambda_\\eta(\\tau_k+s)))$ converges to $\\bar x(s)$ in $\\cC(\\R_+,\\R^n)$. By Assumption \\ref{Assumption: DI}, for almost any $s\\in[0,T]$, we have\n\\[\n\\begin{aligned}\n{\\rm dist}(\\bar d(s),\\mathcal{H}(\\bar x(s)))&\\leq\\bigg\\|\\frac{1}{N}\\sum_{k=1}^Nd^{\\tau_k}(s)-\\bar d(s)\\bigg\\|+{\\rm dist}\\left(\\frac{1}{N}\\sum_{k=1}^Nd^{\\tau_k}(s),\\mathcal{H}(\\bar x(s))\\right)\\\\\n=&\\bigg\\|\\frac{1}{N}\\sum_{k=1}^Nd^{\\tau_k}(s)-\\bar d(s)\\bigg\\|+\n{\\rm dist}\\left(\\frac{1}{N}\\sum_{k=1}^Nd_{\\Lambda_\\eta(\\lambda_\\eta(\\tau_k+s))},\\mathcal{H}(\\bar x(s))\\right)\\rightarrow0.\n\\end{aligned}\n\\]\nSince $T$ is arbitrary and $\\mathcal{H}(\\bar x(s))$ is a closed set, we conclude that $\\bar d(s)\\in\\mathcal{H}(\\bar x(s))$ for almost all $s\\geq0$. This completes the proof.\\qed\n\n\n\\begin{lemma}\n\\label{le:liminf_limsup}\nSuppose Assumption \\ref{Assumption: DI} is satisfied. Then, it holds that\n\\begin{equation}\n    \\label{Eq:liminf_limsup}\n    \\liminf_{t\\rightarrow\\infty}\\Psi(x(t))=\\liminf_{k\\rightarrow\\infty}\\Psi(x_k),\\quad \\limsup_{t\\rightarrow\\infty}\\Psi(x(t))=\\limsup_{k\\rightarrow\\infty}\\Psi(x_k).\n\\end{equation}\n\\end{lemma}\n\n\\begin{proof}\nFor simplicity, we only prove the case for $\\liminf$, the argument for $\\limsup$ follows similarly. By Assumption \\ref{Assumption: DI}, we have that \n\\begin{equation}\n\\lim_{k\\rightarrow\\infty}\\|\\nabla\\phi(x_{k+1})-\\nabla\\phi(x_k)\\|=0.\n\\label{Eq:succ_gap_convergence}\n\\end{equation}\nLet $\\tau_i\\rightarrow\\infty$ be an arbitrary sequence with $x(\\tau_i)\\rightarrow x^*$. By the definition of $\\lambda_\\eta$ and $\\Lambda_\\eta$, it follows that\n\\[\n\\begin{aligned}\n\\|\\nabla\\phi(x_{\\Lambda_\\eta(\\tau_i)})-\\nabla\\phi(x^*)\\|\\leq&\\|\\nabla\\phi(x_{\\Lambda_\\eta(\\tau_i)})-\\nabla\\phi(x(\\tau_i))\\|+\\|\\nabla\\phi(x(\\tau_i))-\\nabla\\phi(x^*)\\|\\\\\n\\leq&\\|\\nabla\\phi(x_{\\Lambda_\\eta(\\tau_i)})-\\nabla\\phi(x_{\\Lambda_\\eta(\\tau_i)+1})\\|+\\|\\nabla\\phi(x(\\tau_i))-\\nabla\\phi(x^*)\\|.\n\\end{aligned}\n\\]\nThe right-hand side converges to zero. By Remark \\ref{rmk:assumption_DI}.1, we have that $\\Psi\\circ\\nabla\\phi^*$ is continuous, so $\\lim_{i\\rightarrow\\infty}\\Psi(x_{\\Lambda_\\eta(\\tau_i)})=\\lim_{i\\rightarrow\\infty}\\Psi\\circ\\nabla\\phi^*\\circ\\nabla\\phi(x_{\\Lambda_\\eta(\\tau_i)})=\\Psi\\circ\\nabla\\phi^*\\circ\\nabla\\phi(x^*)=\\Psi(x^*)$. By choosing $\\tau_i\\rightarrow\\infty$ as the sequence realizing $\\liminf_{t\\rightarrow\\infty}\\Psi(x(t))$, and assuming without loss of generality that $x(\\tau_i)\\rightarrow x^*$, we get\n\\[   \\liminf_{k\\rightarrow\\infty}\\Psi(x_k)\\leq\n\\lim_{{i\\rightarrow\\infty}}\\Psi(x_{\\Lambda_\\eta(\\tau_i)})=\\Psi(x^*)\n=\\liminf_{t\\rightarrow\\infty}\\Psi(x(t)).\n\\]\nThis completes the proof.\n\\end{proof}\n\n\\begin{lemma}\n    \\label{prop:nonescape}\n    Suppose Assumption \\ref{Assumption: DI} and \\ref{assumption_Sard_Lyapunov} hold, then function value $\\Psi(x(t))$ converges {as $t\\to\\infty$.}\n\\end{lemma}\n\n\\begin{proof}\nAssuming $\\liminf_{t\\rightarrow\\infty}\\Psi(x(t))=0$, we define the level set $\\cL_r:=\\{x\\in\\R^n:\\;\\Psi(x)\\leq r\\}$. Choose any $\\epsilon>0$ such that $\\epsilon\\notin\\Psi(\\mathcal{H}^{-1}(0))$. The weak Morse-Sard condition in Assumption \\ref{Assumption: DI} implies that $\\epsilon$ can be chosen arbitrarily small, and Lemma \\ref{le:liminf_limsup} implies that there are infinitely many $k$ such that $x_k\\in\\cL_\\epsilon$. For any $x_k\\in\\cL_\\epsilon$, by the continuity of $\\Psi$, we have that there exists $\\alpha>0$ such that ${\\rm dist}(x_k,\\R^n\\setminus\\cL_{2\\epsilon})>\\alpha$. By \\eqref{Eq:succ_gap_convergence}, for sufficiently large $k$, we have that $\\norm{x_{k+1}-x_k}<\\alpha$. Therefore, for all large $k$, $x_k\\in\\cL_\\epsilon$ implies that $x_{k+1}\\in\\cL_{2\\epsilon}$.\nNow, we define the last entrance and the first subsequent exit times,\n\\begin{equation}\n\\begin{aligned}\n    &k_i=\\max\\{m\\geq j_{i-1}:x_m\\in\\cL_\\epsilon\\}, \\quad\n    &j_i=\\min\\{m\\geq k_i:x_m\\in\\R^n\\setminus\\cL_{2\\epsilon}\\}.\n\\end{aligned}\n\\label{Eq:upcross_def}\n\\end{equation}\nWe prove that such upcrossing occurs for finite times. Otherwise, if there exists $\\{k_i\\}$ such that $\\lim_{i\\rightarrow\\infty}k_i=\\infty$, then Theorem \\ref{subsequential-thm} indicates that, up to a subsequence, $\\nabla\\phi(x^{\\lambda_\\eta(k_i)}(\\cdot))$ converges to $\\nabla\\phi(\\bar x(\\cdot))$, where $\\bar x(\\cdot)$ is a trajectory of \\eqref{Eq:Breg_DI}. By the definition of $k_i$, we have $\\Psi(x_{k_i})\\leq\\epsilon$, $\\Psi(x_{k_i+1})>\\epsilon$. By \\eqref{Eq:succ_gap_convergence} and the continuity of $\\Psi\\circ\\nabla\\phi^*$, we have that $\\lim_{i\\rightarrow\\infty}\\Psi(x_{k_i})=\\lim_{i\\rightarrow\\infty}\\Psi(x_{k_i+1})=\\epsilon$. Recall that $x^{\\lambda_\\eta(k_i)}(0)=x_{k_i}$, therefore, $\\Psi(\\bar x(0))=\\lim_{i\\rightarrow\\infty}\\Psi(x_{k_i})=\\epsilon$. Since $\\bar x(0)$ is not in the stable set, there exists $T>0$, such that  \n\\[\n\\Psi(\\bar x(T))<\\sup_{s\\in[0,T]}\\Psi(\\bar x(s))\\leq\\Psi(\\bar x(0))=\\epsilon.\n\\]\nThen, there exists $\\delta>0$, such that $\\Psi(\\bar x(T))\\leq\\epsilon-2\\delta$. Moreover, for sufficiently large $i$, we have\n\\[\n\\sup_{s\\in[0,T]}\\Psi(x^{\\lambda_\\eta(k_i)}(s))\\leq\\sup_{s\\in[0,T]}\\Psi(\\bar x(s))+\\sup_{s\\in[0,T]}|\\Psi(x^{\\lambda_\\eta(k_i)}(s))-\\Psi(\\bar x(s))|\\leq2\\epsilon.\n\\]\nThe last inequality comes from the uniform convergence of $\\{\\nabla\\phi(x^{\\lambda_\\eta(k_i)}(\\cdot))\\}$ in $\\cC(\\R_+,\\R^n)$. This implies that for all large $i$, $\\{x(\\lambda_\\eta(k_i)+s):s\\in[0,T]\\}\\subset\\cL_{2\\epsilon}$. Thus it holds that $\\lambda_\\eta(j_i)>\\lambda_\\eta(k_i)+T$. Let $l_i=\\max\\{m:\\lambda_\\eta(k_i)\\leq \\lambda_\\eta(m)\\leq \\lambda_\\eta(k_i)+T\\}$. Then $\\|\\nabla\\phi(x_{l_i})-\\nabla\\phi(x^{\\lambda_\\eta(k_i)}(T))\\|\\leq\\|\\nabla\\phi(x_{l_i})-\\nabla\\phi(x_{l_i+1})\\|\\rightarrow0$, and hence $\\norm{\\nabla\\phi(x_{l_i})-\\nabla\\phi(\\bar x(T))}\\rightarrow0$ {as $i\\to\\infty$}. By the continuity of $\\Psi\\circ\\nabla\\phi^*$, we have that $\\Psi(x_{l_i})\\leq\\epsilon-\\delta$ for all large $i$. By the definition of $k_i$ and $j_i$, we have that $\\lambda_\\eta(j_i)<\\lambda_\\eta(l_i)\\leq\\lambda_\\eta(k_i)+T$, which leads to a contradiction. Therefore, for all large $k$, $\\Psi(x_k)\\leq2\\epsilon$. Since $\\epsilon$ can be chosen arbitrarily small, it holds that $\\lim_{k\\rightarrow\\infty}\\Psi(x_k)=0$. This completes the proof.\n\\end{proof}","units":[{"start":43593,"end":44455,"start_line":458,"end_line":468,"raw_tex":"\\begin{theorem}\\label{subsequential-thm}\nSuppose Assumption \\ref{Assumption: DI} holds. Then, for any sequence $\\{\\tau_k\\}_{k=1}^\\infty\\subset\\R_+$, the set of shifted sequence $\\{\\nabla\\phi(x^{\\tau_k}(\\cdot))\\}_{k=1}^\\infty$ is relatively compact in $\\mathcal{C}(\\R_+,\\R^n)$. If $\\lim_{k\\rightarrow\\infty}\\tau_k=\\infty$, then any cluster point $\\bar y(\\cdot)$ of $\\{\\nabla\\phi(x^{\\tau_k}(\\cdot))\\}_{k=1}^\\infty$ belongs to $\\mathcal{C}(\\R+,\\R^n)$. Define $\\bar x(\\cdot):=\\nabla\\phi^*(\\bar y(\\cdot))$. Then there exists a measurable $\\bar d(t)\\in\\mathcal{H}(\\bar x(t))$ satisfying \n\\begin{equation}\n\\nabla\\phi(\\bar x(t))=\\nabla\\phi(\\bar x(0))-\\int_{0}^t\\bar d(\\tau)d\\tau\\quad for\\;all\\;t\\geq0.\n\\label{Eq:DI_int_form}\n\\end{equation}\nEquivalently, we have \n\\[\n\\frac{d}{dt}\\nabla\\phi(\\bar x(t))\\in-\\mathcal{H}(\\bar x(t)),\\;for\\;almost\\;all\\;t\\geq0.\n\\]\n\\end{theorem}","sha256_utf8":"9795a9d4c932336a2a8d1c22a3338b26a90b952b4dfafe64190434981335a0fe"},{"start":45801,"end":47274,"start_line":490,"end_line":510,"raw_tex":"\\begin{proof}\n{Since $\\{x_k\\}$ is bounded, let $x^*$ be any cluster point of $\\{x_k\\}$, and ${\\lim_{i\\rightarrow\\infty}}{x_{k_i}}=x^*$. By Theorem \\ref{subsequential-thm}, up to a subsequence, $\\nabla\\phi(x^{\\lambda_\\eta(k_i)}(\\cdot))\\rightarrow\\nabla\\phi(\\bar x(\\cdot))$ for some $\\bar x(\\cdot)\\in\\cC(\\R_+,\\R^n)$. Note that $x_{{k_i}}=x^{\\lambda_\\eta(k_i)}(0)$, so we have \n\\[\n\\begin{aligned}\n\\norm{\\bar x(0)-x^*}\\leq&\\limsup_{i\\rightarrow\\infty}\\left(\\norm{\\bar x(0)-x^{\\lambda_\\eta(k_i)}(0)}+\\norm{x^{\\lambda_\\eta(k_i)}(0)-x_{k_i}}+\\norm{x_{k_i}-x^*}\\right)\\\\\n=&\\limsup_{i\\rightarrow\\infty}\\left(\\norm{\\nabla\\phi^*(\\nabla\\phi(\\bar x(0)))-\\nabla\\phi^*(\\nabla\\phi(x^{\\lambda_\\eta(k_i)}(0)))}+\\norm{x^{\\lambda_\\eta(k_i)}(0)-x_{k_i}}+\\norm{x_{k_i}-x^*}\\right)\\\\\n=&0,\n\\end{aligned}\n\\]\nwhere the last equality comes from the continuity of $\\nabla\\phi^*$.} Hence $\\bar x(0)=x^*$. Suppose $x^*\\notin \\mathcal{H}^{-1}(0)$, then by Assumption \\ref{assumption_Sard_Lyapunov}, there exists $T>0$, such that \n\\[\n\\Psi(\\bar x(T))<\\sup_{t\\in[0,T]}\\Psi(\\bar x(t))\\leq\\Psi(x^*).\n\\]\nOn the other hand, by Lemma \\ref{prop:nonescape}, $\\Psi(x(t))$ converges {as $t\\to\\infty$}. Therefore, we obtain\n\\[\n\\Psi(\\bar x(T))={\\lim_{i\\rightarrow\\infty}}\\Psi(x^{\\lambda_\\eta(k_i)}(T))=\n{\\lim_{i\\rightarrow\\infty}}\n\\Psi(x({\\lambda_\\eta(k_i)}+T))=\\lim_{t\\rightarrow\\infty}\\Psi(x(t))=\\Psi(x^*), \n\\]\nwhich leads to a contradiction. Therefore, $0\\in \\mathcal{H}(x^*)$. This completes the proof.\n\\end{proof}","sha256_utf8":"15f283cb20edea1fffe198cbb7c5275d05683a4748187377f6801a72fd200eb9"},{"start":97879,"end":98157,"start_line":1146,"end_line":1153,"raw_tex":"\\begin{lemma}\n\\label{le:shift convergence}\nSuppose Assumption \\ref{Assumption: DI} holds, then for any $T>0$, it holds that\n\\begin{equation}\n\\lim_{t\\rightarrow\\infty}\\sup_{s\\in[t,t+T]}\\|\\nabla\\phi(x(s))-\\nabla\\phi(x_t(s))\\|=0.\n\\label{asymptotic-lemma}\n\\end{equation}\n\\end{lemma}","sha256_utf8":"cbf34445120780485211b0f408b600167b1bf6ff432745006a6068d191a8d36f"},{"start":98158,"end":100318,"start_line":1154,"end_line":1184,"raw_tex":"\\begin{proof}\nFix an arbitrary $s\\in[t,t+T]$, let $\\tau_t=\\Lambda_\\eta(t)$, $\\tau_s=\\Lambda_\\eta(s)$. By the definition of $x_t(\\cdot)$, we have\n\\[\n\\begin{aligned}\n&\\nabla\\phi(x_t(s))=\\nabla\\phi(x(t))-\\int_{t}^{\\lambda_\\eta(\\tau_t)}d(u)du-\\int_{\\lambda_\\eta(\\tau_t)}^{\\lambda_\\eta(\\tau_s)}d(u)du-\\int_{\\lambda_\\eta(\\tau_s)}^s d(u)du\\\\\n=&\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))-\\sum_{i=\\tau_t}^{\\tau_s-1}\\eta_id_i+\\left(\\nabla\\phi(x(t))-\\int_t^{\\lambda_\\eta(\\tau_t)}d(u)du-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))\\right)-\\int_{\\lambda_\\eta(\\tau_s)}^s d(u)du\\\\\n=&\\nabla\\phi(x({\\lambda_\\eta(\\tau_s})))+\\sum_{i=\\tau_t}^{\\tau_s-1}\\eta_i\\xi_i+\\left(\\nabla\\phi(x(t))-\\int_t^{\\lambda_\\eta(\\tau_t)}d(u)du-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))\\right)-\\int_{\\lambda_\\eta(\\tau_s)}^s d(u)du\\\\\n=&\\nabla\\phi(x(s))+\\sum_{i=\\tau_t}^{\\tau_s-1}\\eta_i\\xi_i+\\left(\\nabla\\phi(x(t))-\\int_t^{\\lambda_\\eta(\\tau_t)}d(u)du-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))\\right)\\\\\n&+\\left(\\nabla\\phi(x(\\lambda_\\eta(\\tau_s)))-\\int_{\\lambda_\\eta(\\tau_s)}^sd(u)du-\\nabla\\phi(x(s))\\right).\n\\end{aligned}\n\\]\nNote that\n\\[\n\\begin{aligned}\n&\\norm{\\nabla\\phi(x(t))-\\int_t^{\\lambda_\\eta(\\tau_t)}d(u)du-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))}\\\\\n\\leq&{\\|\\nabla\\phi(x(t))-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))\\|}\n+\\int_t^{\\lambda_\\eta(\\tau_t)}\\|d(u)\\|du\n\\\\\n \\leq&{\\|\\nabla\\phi(x(\\lambda_\\eta(\\tau_t+1)))-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))\\|}\n+\\int_t^{\\lambda_\\eta(\\tau_t)}\\|d(u)\\|du\n\\\\\n \\leq&\\eta_{\\tau_t}(\\|\\xi_{\\tau_t}\\|+2\\|d_{\\tau_t}\\|),\n \\end{aligned}\n\\]\nand similarly \n$\n\\big\\|\\nabla\\phi(x(\\lambda_\\eta(\\tau_s)))-\\int_{\\lambda_\\eta(\\tau_s)}^sd(u)du-\\nabla\\phi(x(s))\\big\\|\\leq\\eta_{\\tau_s}(\\|\\xi_{\\tau_s}\\|+2\\|d_{\\tau_s}\\|).\n$\nBy Assumption \\ref{Assumption: DI}, we \nhave that $\\lim\\sup\\limits_{t\\rightarrow\\infty}\\eta_{\\tau_t}(\\|\\xi_{\\tau_t}\\|+2\\|d_{\\tau_t}\\|)=0$, $\\lim\\sup\\limits_{s\\rightarrow\\infty}\\eta_{\\tau_s}(\\|\\xi_{\\tau_s}\\|+2\\|d_{\\tau_s}\\|)=0$, and $\\lim\\limits_{t\\rightarrow\\infty}\\sup\\limits_{s\\in[t,t+T]}\\sum_{i=\\tau_t}^{\\tau_s-1}\\eta_i\\xi_i=0$. Therefore, it holds that $\\lim\\limits_{t\\rightarrow\\infty}\\sup\\limits_{s\\in[t,t+T]}\\|\\nabla\\phi(x(s))-\\nabla\\phi(x_t(s))\\|=0$, which completes the proof.\n\\end{proof}","sha256_utf8":"ee60c15b7dd00de44c1223a849269313d16a75c584af8fb0bcc4e6abea2ef1b9"},{"start":100319,"end":105626,"start_line":1185,"end_line":1223,"raw_tex":"\\noindent{\\textit{Proof of Theorem \\ref{subsequential-thm}\n.}} By the definition of $x_t(\\cdot)$, it follows that $\\nabla\\phi(x_t(s))=\\nabla\\phi(x(t))-\\int_t^s y(u)du$, for all $s\\geq t$. By the boundedness of $y(s)$, Arzel$\\grave{a}$-Ascoli's theorem \\cite{rudin1953principles} ensures that $\\{\\nabla\\phi(x_t(t+\\cdot))\\}_{t\\in\\R_+}$ is relatively compact in $\\cC(\\R_+,\\R^n)$. For any subset $\\{\\tau_k\\}\\subset\\R_+$, we consider the sequence $\\{\\nabla\\phi(x^{\\tau_k}(\\cdot))\\}_{k\\in\\mathbb{N}_+}$. There are two cases to consider. Case (i): the sequence $\\{\\tau_k\\}$ has a cluster point $t$. Without loss of generality, assume that $\\lim_{k\\rightarrow\\infty}\\tau_k=t$. By the definition of $x(\\cdot)$ {in \\eqref{Eq:interpolation}} and uniform boundedness assumption, it holds that  $\\nabla\\phi(x(\\cdot))$ is Lipschitz continuous. Thus, $\\nabla\\phi(x^{\\tau_k}(\\cdot))$ converges to $\\nabla\\phi(x^t(\\cdot))$ in $\\cC(\\R_+,\\R^n)$. Case (ii): $\\lim_{k\\rightarrow\\infty}\\tau_k=\\infty$. Suppose that $\\nabla\\phi(x^{\\tau_k}(\\cdot))$ does not has any cluster point in $\\cC(\\R_+,\\R^n)$. Since $\\{\\nabla\\phi(x_{\\tau_k}(\\tau_k+\\cdot))\\}$ is relatively compact in $\\cC(\\R_+,\\R^n)$, without loss of generality, we assume that $\\lim_{k\\rightarrow\\infty}\\nabla\\phi(x_{\\tau_k}(\\tau_k+\\cdot))=\\bar y(\\cdot)$. Then, for any compact set $C\\subset\\R_+$, it follows from Lemma \\ref{le:shift convergence} that\n\\[\n\\begin{aligned}\n&\\lim_{k\\rightarrow\\infty,s\\in C}\\norm{\\nabla\\phi(x^{\\tau_k}(s))-\\bar y(s)}\\\\\n\\leq&\\lim_{k\\rightarrow\\infty,s\\in C}\\norm{\\nabla\\phi(x^{\\tau_k}(s))-\\nabla\\phi(x_{\\tau_k}(\\tau_k+s))}+\\lim_{k\\rightarrow\\infty,s\\in C}\\norm{\\nabla\\phi(x_{\\tau_k}(\\tau_k+s))-\\bar y(s)}=0,\n\\end{aligned}\n\\]\nwhich contradicts that $\\nabla\\phi(x^{\\tau_k}(\\cdot))$ does not has any cluster point in $\\cC(\\R_+,\\R^n)$. Thus, in both cases, $\\{\\nabla\\phi(x^{\\tau_k}(\\cdot))\\}$ is relatively compact in $\\cC(\\R_+,\\R^n)$. Because $\\{\\tau_k\\}$ is an arbitrary subset in $\\R_+$, we have that $\\{\\nabla\\phi(x^t(\\cdot))\\}$ is relatively compact in $\\cC(\\R_+,\\R^n)$. \n\nNext, we aim to construct a trajectory of the differential inclusion. Define the shifts $d^t(\\cdot)=d(t+\\,\\cdot)$. Consider $\\{\\tau_k\\}$ satisfying $\\tau_k\\rightarrow\\infty$, and fix $T>0$. Without loss of generality, we assume that $\\nabla\\phi(x^{\\tau_k}(\\cdot))$ converges to $\\bar y(\\cdot)$ in $\\cC(\\R_+,\\R^n)$, otherwise, we choose its convergent subsequence. The set $\\mathcal{Y}_T:=\\{d^{\\tau_k}(s),s\\in[0,T]\\}_{k\\in\\mathbb{N}}\\subset L^2([0,T])$ is bounded. Therefore, it follows from the Banach-Alaoglu theorem \\cite{rudin1991functional} that $\\cY_T$ is weakly sequentially compact, i.e. there exists a subsequence $\\{\\tau_{k_j}\\}$ and $\\bar y(\\cdot)\\in L^2([0,T])$ such that $d^{\\tau_{k_j}}(\\cdot)\\rightarrow\\bar d(\\cdot)$ weakly in $L^2([0,T])$. On the other hand, by Lemma \\ref{le:shift convergence}, we have $\\nabla\\phi(x_{\\tau_{k_j}}(\\tau_{k_j}+\\cdot))$ \nconverges to $\\bar y(\\cdot)$ in $\\cC(\\R_+,\\R^n)$. For any $\\tau\\in[0,T]$, by the definition of $x_t(\\cdot)$, we have\n\\[\n\\nabla\\phi(x_t(t+\\tau))=\\nabla\\phi(x_t(t))-\\int_0^\\tau d^t(s)ds.\n\\]\nSetting $t=\\tau_{k_j}$ and taking the limit as $k_j\\rightarrow\\infty$, we deduce that\n\\[\n\\bar y(\\tau)=\\bar y(0)-\\int_0^\\tau\\bar d(s)ds.\n\\]\nLet $\\bar x(\\cdot)=\\nabla\\phi^*(\\bar y(\\cdot))$, since $T>0$ is arbitrary, we get \\eqref{Eq:DI_int_form}. \n\nThe remaining step is to verify that $\\bar d(s)\\in\\mathcal{H}(\\bar x(s))$ for almost all $s\\geq0$. We again fix an arbitrary $T > 0$. Given that $\\mathcal{Y}_T\\subset L^2([0,T])$ is bounded, the Banach-Saks theorem \\cite{rudin1991functional} implies that for ${\\tau_k}$ (choosing a subsequence if necessary), $\\frac{1}{N}\\sum_{k=1}^Nd^{\\tau_k}(s)$ strongly converges to $\\bar d(s)$ in $L^2([0,T])$ . By the definition of $d(\\cdot)$, we have $d^{\\tau_k}(s)=d_{\\Lambda_\\eta(\\tau_k+s)}$. Now for any $s\\in[0,T]$, we have\n\\[\n\\begin{aligned}\n    &\\|\\nabla\\phi(x(\\lambda_\\eta(\\Lambda_\\eta(\\tau_k+s))))-\\nabla\\phi(\\bar x(s))\\|\\\\\n    \\leq&\\|\\nabla\\phi(x(\\lambda_\\eta(\\Lambda_\\eta(\\tau_k+s))))-\\nabla\\phi(x(\\tau_k+s))\\|+\\|\\nabla\\phi(x^{\\tau_k}(s))-\\nabla\\phi(\\bar x(s))\\|\\\\\n    \\leq&\\|\\nabla\\phi(x(\\lambda_\\eta(\\Lambda_\\eta(\\tau_k+s)+1)))-\\nabla\\phi(x(\\lambda_\\eta(\\Lambda_\\eta(\\tau_k+s))))\\|+\\|\\nabla\\phi(x^{\\tau_k}(s))-\\nabla\\phi(\\bar x(s))\\|\\\\\n    \\leq&\\eta_{\\Lambda_\\eta(\\tau_k+s)}(\\|\\xi_{\\Lambda_\\eta(\\tau_k+s)}\\|+\\|d_{\\Lambda_\\eta(\\tau_k+s)}\\|)+\\|\\nabla\\phi(x^{\\tau_k}(s))-\\nabla\\phi(\\bar x(s))\\|,\n\\end{aligned}\n\\]\nwhich converges to zero as $k\\rightarrow\\infty$. By the continuity of $\\nabla\\phi^*$, we have that $x(\\lambda_\\eta(\\Lambda_\\eta(\\tau_k+s)))$ converges to $\\bar x(s)$ in $\\cC(\\R_+,\\R^n)$. By Assumption \\ref{Assumption: DI}, for almost any $s\\in[0,T]$, we have\n\\[\n\\begin{aligned}\n{\\rm dist}(\\bar d(s),\\mathcal{H}(\\bar x(s)))&\\leq\\bigg\\|\\frac{1}{N}\\sum_{k=1}^Nd^{\\tau_k}(s)-\\bar d(s)\\bigg\\|+{\\rm dist}\\left(\\frac{1}{N}\\sum_{k=1}^Nd^{\\tau_k}(s),\\mathcal{H}(\\bar x(s))\\right)\\\\\n=&\\bigg\\|\\frac{1}{N}\\sum_{k=1}^Nd^{\\tau_k}(s)-\\bar d(s)\\bigg\\|+\n{\\rm dist}\\left(\\frac{1}{N}\\sum_{k=1}^Nd_{\\Lambda_\\eta(\\lambda_\\eta(\\tau_k+s))},\\mathcal{H}(\\bar x(s))\\right)\\rightarrow0.\n\\end{aligned}\n\\]\nSince $T$ is arbitrary and $\\mathcal{H}(\\bar x(s))$ is a closed set, we conclude that $\\bar d(s)\\in\\mathcal{H}(\\bar x(s))$ for almost all $s\\geq0$. This completes the proof.\\qed\n","sha256_utf8":"e0507c19c79e7b657729f322476592ce526f3c4ab32f451187497e077ae5e8d1"},{"start":105718,"end":106068,"start_line":1229,"end_line":1236,"raw_tex":"\\begin{lemma}\n\\label{le:liminf_limsup}\nSuppose Assumption \\ref{Assumption: DI} is satisfied. Then, it holds that\n\\begin{equation}\n    \\label{Eq:liminf_limsup}\n    \\liminf_{t\\rightarrow\\infty}\\Psi(x(t))=\\liminf_{k\\rightarrow\\infty}\\Psi(x_k),\\quad \\limsup_{t\\rightarrow\\infty}\\Psi(x(t))=\\limsup_{k\\rightarrow\\infty}\\Psi(x_k).\n\\end{equation}\n\\end{lemma}","sha256_utf8":"4a62fb2df20ecaaa731ee41a507198b72dea6d9ca5b68ed78c70d759c5c15509"},{"start":106069,"end":107569,"start_line":1237,"end_line":1256,"raw_tex":"\\begin{proof}\nFor simplicity, we only prove the case for $\\liminf$, the argument for $\\limsup$ follows similarly. By Assumption \\ref{Assumption: DI}, we have that \n\\begin{equation}\n\\lim_{k\\rightarrow\\infty}\\|\\nabla\\phi(x_{k+1})-\\nabla\\phi(x_k)\\|=0.\n\\label{Eq:succ_gap_convergence}\n\\end{equation}\nLet $\\tau_i\\rightarrow\\infty$ be an arbitrary sequence with $x(\\tau_i)\\rightarrow x^*$. By the definition of $\\lambda_\\eta$ and $\\Lambda_\\eta$, it follows that\n\\[\n\\begin{aligned}\n\\|\\nabla\\phi(x_{\\Lambda_\\eta(\\tau_i)})-\\nabla\\phi(x^*)\\|\\leq&\\|\\nabla\\phi(x_{\\Lambda_\\eta(\\tau_i)})-\\nabla\\phi(x(\\tau_i))\\|+\\|\\nabla\\phi(x(\\tau_i))-\\nabla\\phi(x^*)\\|\\\\\n\\leq&\\|\\nabla\\phi(x_{\\Lambda_\\eta(\\tau_i)})-\\nabla\\phi(x_{\\Lambda_\\eta(\\tau_i)+1})\\|+\\|\\nabla\\phi(x(\\tau_i))-\\nabla\\phi(x^*)\\|.\n\\end{aligned}\n\\]\nThe right-hand side converges to zero. By Remark \\ref{rmk:assumption_DI}.1, we have that $\\Psi\\circ\\nabla\\phi^*$ is continuous, so $\\lim_{i\\rightarrow\\infty}\\Psi(x_{\\Lambda_\\eta(\\tau_i)})=\\lim_{i\\rightarrow\\infty}\\Psi\\circ\\nabla\\phi^*\\circ\\nabla\\phi(x_{\\Lambda_\\eta(\\tau_i)})=\\Psi\\circ\\nabla\\phi^*\\circ\\nabla\\phi(x^*)=\\Psi(x^*)$. By choosing $\\tau_i\\rightarrow\\infty$ as the sequence realizing $\\liminf_{t\\rightarrow\\infty}\\Psi(x(t))$, and assuming without loss of generality that $x(\\tau_i)\\rightarrow x^*$, we get\n\\[   \\liminf_{k\\rightarrow\\infty}\\Psi(x_k)\\leq\n\\lim_{{i\\rightarrow\\infty}}\\Psi(x_{\\Lambda_\\eta(\\tau_i)})=\\Psi(x^*)\n=\\liminf_{t\\rightarrow\\infty}\\Psi(x(t)).\n\\]\nThis completes the proof.\n\\end{proof}","sha256_utf8":"d3f2bf2f4a02c8af92865695f7b0b140050a090527cd3ab6b7ded9b07284421e"},{"start":107952,"end":108151,"start_line":1260,"end_line":1263,"raw_tex":"\\begin{lemma}\n    \\label{prop:nonescape}\n    Suppose Assumption \\ref{Assumption: DI} and \\ref{assumption_Sard_Lyapunov} hold, then function value $\\Psi(x(t))$ converges {as $t\\to\\infty$.}\n\\end{lemma}","sha256_utf8":"465c801cf775f7359484d2a7e6827989eb891765d960573fad8cc3eeb78fb255"},{"start":108152,"end":111488,"start_line":1264,"end_line":1283,"raw_tex":"\\begin{proof}\nAssuming $\\liminf_{t\\rightarrow\\infty}\\Psi(x(t))=0$, we define the level set $\\cL_r:=\\{x\\in\\R^n:\\;\\Psi(x)\\leq r\\}$. Choose any $\\epsilon>0$ such that $\\epsilon\\notin\\Psi(\\mathcal{H}^{-1}(0))$. The weak Morse-Sard condition in Assumption \\ref{Assumption: DI} implies that $\\epsilon$ can be chosen arbitrarily small, and Lemma \\ref{le:liminf_limsup} implies that there are infinitely many $k$ such that $x_k\\in\\cL_\\epsilon$. For any $x_k\\in\\cL_\\epsilon$, by the continuity of $\\Psi$, we have that there exists $\\alpha>0$ such that ${\\rm dist}(x_k,\\R^n\\setminus\\cL_{2\\epsilon})>\\alpha$. By \\eqref{Eq:succ_gap_convergence}, for sufficiently large $k$, we have that $\\norm{x_{k+1}-x_k}<\\alpha$. Therefore, for all large $k$, $x_k\\in\\cL_\\epsilon$ implies that $x_{k+1}\\in\\cL_{2\\epsilon}$.\nNow, we define the last entrance and the first subsequent exit times,\n\\begin{equation}\n\\begin{aligned}\n    &k_i=\\max\\{m\\geq j_{i-1}:x_m\\in\\cL_\\epsilon\\}, \\quad\n    &j_i=\\min\\{m\\geq k_i:x_m\\in\\R^n\\setminus\\cL_{2\\epsilon}\\}.\n\\end{aligned}\n\\label{Eq:upcross_def}\n\\end{equation}\nWe prove that such upcrossing occurs for finite times. Otherwise, if there exists $\\{k_i\\}$ such that $\\lim_{i\\rightarrow\\infty}k_i=\\infty$, then Theorem \\ref{subsequential-thm} indicates that, up to a subsequence, $\\nabla\\phi(x^{\\lambda_\\eta(k_i)}(\\cdot))$ converges to $\\nabla\\phi(\\bar x(\\cdot))$, where $\\bar x(\\cdot)$ is a trajectory of \\eqref{Eq:Breg_DI}. By the definition of $k_i$, we have $\\Psi(x_{k_i})\\leq\\epsilon$, $\\Psi(x_{k_i+1})>\\epsilon$. By \\eqref{Eq:succ_gap_convergence} and the continuity of $\\Psi\\circ\\nabla\\phi^*$, we have that $\\lim_{i\\rightarrow\\infty}\\Psi(x_{k_i})=\\lim_{i\\rightarrow\\infty}\\Psi(x_{k_i+1})=\\epsilon$. Recall that $x^{\\lambda_\\eta(k_i)}(0)=x_{k_i}$, therefore, $\\Psi(\\bar x(0))=\\lim_{i\\rightarrow\\infty}\\Psi(x_{k_i})=\\epsilon$. Since $\\bar x(0)$ is not in the stable set, there exists $T>0$, such that  \n\\[\n\\Psi(\\bar x(T))<\\sup_{s\\in[0,T]}\\Psi(\\bar x(s))\\leq\\Psi(\\bar x(0))=\\epsilon.\n\\]\nThen, there exists $\\delta>0$, such that $\\Psi(\\bar x(T))\\leq\\epsilon-2\\delta$. Moreover, for sufficiently large $i$, we have\n\\[\n\\sup_{s\\in[0,T]}\\Psi(x^{\\lambda_\\eta(k_i)}(s))\\leq\\sup_{s\\in[0,T]}\\Psi(\\bar x(s))+\\sup_{s\\in[0,T]}|\\Psi(x^{\\lambda_\\eta(k_i)}(s))-\\Psi(\\bar x(s))|\\leq2\\epsilon.\n\\]\nThe last inequality comes from the uniform convergence of $\\{\\nabla\\phi(x^{\\lambda_\\eta(k_i)}(\\cdot))\\}$ in $\\cC(\\R_+,\\R^n)$. This implies that for all large $i$, $\\{x(\\lambda_\\eta(k_i)+s):s\\in[0,T]\\}\\subset\\cL_{2\\epsilon}$. Thus it holds that $\\lambda_\\eta(j_i)>\\lambda_\\eta(k_i)+T$. Let $l_i=\\max\\{m:\\lambda_\\eta(k_i)\\leq \\lambda_\\eta(m)\\leq \\lambda_\\eta(k_i)+T\\}$. Then $\\|\\nabla\\phi(x_{l_i})-\\nabla\\phi(x^{\\lambda_\\eta(k_i)}(T))\\|\\leq\\|\\nabla\\phi(x_{l_i})-\\nabla\\phi(x_{l_i+1})\\|\\rightarrow0$, and hence $\\norm{\\nabla\\phi(x_{l_i})-\\nabla\\phi(\\bar x(T))}\\rightarrow0$ {as $i\\to\\infty$}. By the continuity of $\\Psi\\circ\\nabla\\phi^*$, we have that $\\Psi(x_{l_i})\\leq\\epsilon-\\delta$ for all large $i$. By the definition of $k_i$ and $j_i$, we have that $\\lambda_\\eta(j_i)<\\lambda_\\eta(l_i)\\leq\\lambda_\\eta(k_i)+T$, which leads to a contradiction. Therefore, for all large $k$, $\\Psi(x_k)\\leq2\\epsilon$. Since $\\epsilon$ can be chosen arbitrarily small, it holds that $\\lim_{k\\rightarrow\\infty}\\Psi(x_k)=0$. This completes the proof.\n\\end{proof}","sha256_utf8":"eaeb18f8081a64300a81e5200af85bef0be049d8b8e20c7753ac112413f74269"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"随机Bregman迭代收敛","source_units":[{"start":45532,"end":45800,"start_line":487,"end_line":489,"raw_tex":"\\begin{theorem}\\label{convergence-thm-func-val}\nSuppose Assumptions \\ref{Assumption: DI} and \\ref{assumption_Sard_Lyapunov} hold. Then any cluster point of $\\{x_k\\}$ lies in $\\mathcal{H}^{-1}(0)$ and the function values $\\{\\Psi(x_k)\\}_{k\\geq1}$ converge.\n\\end{theorem}","sha256_utf8":"712fec31e4129bf8cd787d14ac19b894e48e8f3cb669584b1f09cc71b7e88517"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"主定理完整证明","source_units":[{"start":45801,"end":47274,"start_line":490,"end_line":510,"raw_tex":"\\begin{proof}\n{Since $\\{x_k\\}$ is bounded, let $x^*$ be any cluster point of $\\{x_k\\}$, and ${\\lim_{i\\rightarrow\\infty}}{x_{k_i}}=x^*$. By Theorem \\ref{subsequential-thm}, up to a subsequence, $\\nabla\\phi(x^{\\lambda_\\eta(k_i)}(\\cdot))\\rightarrow\\nabla\\phi(\\bar x(\\cdot))$ for some $\\bar x(\\cdot)\\in\\cC(\\R_+,\\R^n)$. Note that $x_{{k_i}}=x^{\\lambda_\\eta(k_i)}(0)$, so we have \n\\[\n\\begin{aligned}\n\\norm{\\bar x(0)-x^*}\\leq&\\limsup_{i\\rightarrow\\infty}\\left(\\norm{\\bar x(0)-x^{\\lambda_\\eta(k_i)}(0)}+\\norm{x^{\\lambda_\\eta(k_i)}(0)-x_{k_i}}+\\norm{x_{k_i}-x^*}\\right)\\\\\n=&\\limsup_{i\\rightarrow\\infty}\\left(\\norm{\\nabla\\phi^*(\\nabla\\phi(\\bar x(0)))-\\nabla\\phi^*(\\nabla\\phi(x^{\\lambda_\\eta(k_i)}(0)))}+\\norm{x^{\\lambda_\\eta(k_i)}(0)-x_{k_i}}+\\norm{x_{k_i}-x^*}\\right)\\\\\n=&0,\n\\end{aligned}\n\\]\nwhere the last equality comes from the continuity of $\\nabla\\phi^*$.} Hence $\\bar x(0)=x^*$. Suppose $x^*\\notin \\mathcal{H}^{-1}(0)$, then by Assumption \\ref{assumption_Sard_Lyapunov}, there exists $T>0$, such that \n\\[\n\\Psi(\\bar x(T))<\\sup_{t\\in[0,T]}\\Psi(\\bar x(t))\\leq\\Psi(x^*).\n\\]\nOn the other hand, by Lemma \\ref{prop:nonescape}, $\\Psi(x(t))$ converges {as $t\\to\\infty$}. Therefore, we obtain\n\\[\n\\Psi(\\bar x(T))={\\lim_{i\\rightarrow\\infty}}\\Psi(x^{\\lambda_\\eta(k_i)}(T))=\n{\\lim_{i\\rightarrow\\infty}}\n\\Psi(x({\\lambda_\\eta(k_i)}+T))=\\lim_{t\\rightarrow\\infty}\\Psi(x(t))=\\Psi(x^*), \n\\]\nwhich leads to a contradiction. Therefore, $0\\in \\mathcal{H}(x^*)$. This completes the proof.\n\\end{proof}","sha256_utf8":"15f283cb20edea1fffe198cbb7c5275d05683a4748187377f6801a72fd200eb9"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"极限轨迹与严格下降","source_units":[{"start":45815,"end":46868,"start_line":491,"end_line":502,"raw_tex":"{Since $\\{x_k\\}$ is bounded, let $x^*$ be any cluster point of $\\{x_k\\}$, and ${\\lim_{i\\rightarrow\\infty}}{x_{k_i}}=x^*$. By Theorem \\ref{subsequential-thm}, up to a subsequence, $\\nabla\\phi(x^{\\lambda_\\eta(k_i)}(\\cdot))\\rightarrow\\nabla\\phi(\\bar x(\\cdot))$ for some $\\bar x(\\cdot)\\in\\cC(\\R_+,\\R^n)$. Note that $x_{{k_i}}=x^{\\lambda_\\eta(k_i)}(0)$, so we have \n\\[\n\\begin{aligned}\n\\norm{\\bar x(0)-x^*}\\leq&\\limsup_{i\\rightarrow\\infty}\\left(\\norm{\\bar x(0)-x^{\\lambda_\\eta(k_i)}(0)}+\\norm{x^{\\lambda_\\eta(k_i)}(0)-x_{k_i}}+\\norm{x_{k_i}-x^*}\\right)\\\\\n=&\\limsup_{i\\rightarrow\\infty}\\left(\\norm{\\nabla\\phi^*(\\nabla\\phi(\\bar x(0)))-\\nabla\\phi^*(\\nabla\\phi(x^{\\lambda_\\eta(k_i)}(0)))}+\\norm{x^{\\lambda_\\eta(k_i)}(0)-x_{k_i}}+\\norm{x_{k_i}-x^*}\\right)\\\\\n=&0,\n\\end{aligned}\n\\]\nwhere the last equality comes from the continuity of $\\nabla\\phi^*$.} Hence $\\bar x(0)=x^*$. Suppose $x^*\\notin \\mathcal{H}^{-1}(0)$, then by Assumption \\ref{assumption_Sard_Lyapunov}, there exists $T>0$, such that \n\\[\n\\Psi(\\bar x(T))<\\sup_{t\\in[0,T]}\\Psi(\\bar x(t))\\leq\\Psi(x^*).\n\\]\n","sha256_utf8":"791378a8974a0589ee3695412aa14a336d4345644a352dac48153bffe14ff820"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"函数值极限排除非稳态","source_units":[{"start":46868,"end":47263,"start_line":503,"end_line":509,"raw_tex":"On the other hand, by Lemma \\ref{prop:nonescape}, $\\Psi(x(t))$ converges {as $t\\to\\infty$}. Therefore, we obtain\n\\[\n\\Psi(\\bar x(T))={\\lim_{i\\rightarrow\\infty}}\\Psi(x^{\\lambda_\\eta(k_i)}(T))=\n{\\lim_{i\\rightarrow\\infty}}\n\\Psi(x({\\lambda_\\eta(k_i)}+T))=\\lim_{t\\rightarrow\\infty}\\Psi(x(t))=\\Psi(x^*), \n\\]\nwhich leads to a contradiction. Therefore, $0\\in \\mathcal{H}(x^*)$. This completes the proof.\n","sha256_utf8":"cd2ac4d797b09839330d70af06396ce052d9b9e6b69bd0a135a3288d3b7cbddc"}]},{"id":"S005","semantic_type":"theorem","kind":"theorem","title":"对偶插值渐近跟踪","source_units":[{"start":43593,"end":44455,"start_line":458,"end_line":468,"raw_tex":"\\begin{theorem}\\label{subsequential-thm}\nSuppose Assumption \\ref{Assumption: DI} holds. Then, for any sequence $\\{\\tau_k\\}_{k=1}^\\infty\\subset\\R_+$, the set of shifted sequence $\\{\\nabla\\phi(x^{\\tau_k}(\\cdot))\\}_{k=1}^\\infty$ is relatively compact in $\\mathcal{C}(\\R_+,\\R^n)$. If $\\lim_{k\\rightarrow\\infty}\\tau_k=\\infty$, then any cluster point $\\bar y(\\cdot)$ of $\\{\\nabla\\phi(x^{\\tau_k}(\\cdot))\\}_{k=1}^\\infty$ belongs to $\\mathcal{C}(\\R+,\\R^n)$. Define $\\bar x(\\cdot):=\\nabla\\phi^*(\\bar y(\\cdot))$. Then there exists a measurable $\\bar d(t)\\in\\mathcal{H}(\\bar x(t))$ satisfying \n\\begin{equation}\n\\nabla\\phi(\\bar x(t))=\\nabla\\phi(\\bar x(0))-\\int_{0}^t\\bar d(\\tau)d\\tau\\quad for\\;all\\;t\\geq0.\n\\label{Eq:DI_int_form}\n\\end{equation}\nEquivalently, we have \n\\[\n\\frac{d}{dt}\\nabla\\phi(\\bar x(t))\\in-\\mathcal{H}(\\bar x(t)),\\;for\\;almost\\;all\\;t\\geq0.\n\\]\n\\end{theorem}","sha256_utf8":"9795a9d4c932336a2a8d1c22a3338b26a90b952b4dfafe64190434981335a0fe"}]},{"id":"S006","semantic_type":"proof","kind":"proof","title":"渐近跟踪完整证明","source_units":[{"start":100319,"end":105626,"start_line":1185,"end_line":1223,"raw_tex":"\\noindent{\\textit{Proof of Theorem \\ref{subsequential-thm}\n.}} By the definition of $x_t(\\cdot)$, it follows that $\\nabla\\phi(x_t(s))=\\nabla\\phi(x(t))-\\int_t^s y(u)du$, for all $s\\geq t$. By the boundedness of $y(s)$, Arzel$\\grave{a}$-Ascoli's theorem \\cite{rudin1953principles} ensures that $\\{\\nabla\\phi(x_t(t+\\cdot))\\}_{t\\in\\R_+}$ is relatively compact in $\\cC(\\R_+,\\R^n)$. For any subset $\\{\\tau_k\\}\\subset\\R_+$, we consider the sequence $\\{\\nabla\\phi(x^{\\tau_k}(\\cdot))\\}_{k\\in\\mathbb{N}_+}$. There are two cases to consider. Case (i): the sequence $\\{\\tau_k\\}$ has a cluster point $t$. Without loss of generality, assume that $\\lim_{k\\rightarrow\\infty}\\tau_k=t$. By the definition of $x(\\cdot)$ {in \\eqref{Eq:interpolation}} and uniform boundedness assumption, it holds that  $\\nabla\\phi(x(\\cdot))$ is Lipschitz continuous. Thus, $\\nabla\\phi(x^{\\tau_k}(\\cdot))$ converges to $\\nabla\\phi(x^t(\\cdot))$ in $\\cC(\\R_+,\\R^n)$. Case (ii): $\\lim_{k\\rightarrow\\infty}\\tau_k=\\infty$. Suppose that $\\nabla\\phi(x^{\\tau_k}(\\cdot))$ does not has any cluster point in $\\cC(\\R_+,\\R^n)$. Since $\\{\\nabla\\phi(x_{\\tau_k}(\\tau_k+\\cdot))\\}$ is relatively compact in $\\cC(\\R_+,\\R^n)$, without loss of generality, we assume that $\\lim_{k\\rightarrow\\infty}\\nabla\\phi(x_{\\tau_k}(\\tau_k+\\cdot))=\\bar y(\\cdot)$. Then, for any compact set $C\\subset\\R_+$, it follows from Lemma \\ref{le:shift convergence} that\n\\[\n\\begin{aligned}\n&\\lim_{k\\rightarrow\\infty,s\\in C}\\norm{\\nabla\\phi(x^{\\tau_k}(s))-\\bar y(s)}\\\\\n\\leq&\\lim_{k\\rightarrow\\infty,s\\in C}\\norm{\\nabla\\phi(x^{\\tau_k}(s))-\\nabla\\phi(x_{\\tau_k}(\\tau_k+s))}+\\lim_{k\\rightarrow\\infty,s\\in C}\\norm{\\nabla\\phi(x_{\\tau_k}(\\tau_k+s))-\\bar y(s)}=0,\n\\end{aligned}\n\\]\nwhich contradicts that $\\nabla\\phi(x^{\\tau_k}(\\cdot))$ does not has any cluster point in $\\cC(\\R_+,\\R^n)$. Thus, in both cases, $\\{\\nabla\\phi(x^{\\tau_k}(\\cdot))\\}$ is relatively compact in $\\cC(\\R_+,\\R^n)$. Because $\\{\\tau_k\\}$ is an arbitrary subset in $\\R_+$, we have that $\\{\\nabla\\phi(x^t(\\cdot))\\}$ is relatively compact in $\\cC(\\R_+,\\R^n)$. \n\nNext, we aim to construct a trajectory of the differential inclusion. Define the shifts $d^t(\\cdot)=d(t+\\,\\cdot)$. Consider $\\{\\tau_k\\}$ satisfying $\\tau_k\\rightarrow\\infty$, and fix $T>0$. Without loss of generality, we assume that $\\nabla\\phi(x^{\\tau_k}(\\cdot))$ converges to $\\bar y(\\cdot)$ in $\\cC(\\R_+,\\R^n)$, otherwise, we choose its convergent subsequence. The set $\\mathcal{Y}_T:=\\{d^{\\tau_k}(s),s\\in[0,T]\\}_{k\\in\\mathbb{N}}\\subset L^2([0,T])$ is bounded. Therefore, it follows from the Banach-Alaoglu theorem \\cite{rudin1991functional} that $\\cY_T$ is weakly sequentially compact, i.e. there exists a subsequence $\\{\\tau_{k_j}\\}$ and $\\bar y(\\cdot)\\in L^2([0,T])$ such that $d^{\\tau_{k_j}}(\\cdot)\\rightarrow\\bar d(\\cdot)$ weakly in $L^2([0,T])$. On the other hand, by Lemma \\ref{le:shift convergence}, we have $\\nabla\\phi(x_{\\tau_{k_j}}(\\tau_{k_j}+\\cdot))$ \nconverges to $\\bar y(\\cdot)$ in $\\cC(\\R_+,\\R^n)$. For any $\\tau\\in[0,T]$, by the definition of $x_t(\\cdot)$, we have\n\\[\n\\nabla\\phi(x_t(t+\\tau))=\\nabla\\phi(x_t(t))-\\int_0^\\tau d^t(s)ds.\n\\]\nSetting $t=\\tau_{k_j}$ and taking the limit as $k_j\\rightarrow\\infty$, we deduce that\n\\[\n\\bar y(\\tau)=\\bar y(0)-\\int_0^\\tau\\bar d(s)ds.\n\\]\nLet $\\bar x(\\cdot)=\\nabla\\phi^*(\\bar y(\\cdot))$, since $T>0$ is arbitrary, we get \\eqref{Eq:DI_int_form}. \n\nThe remaining step is to verify that $\\bar d(s)\\in\\mathcal{H}(\\bar x(s))$ for almost all $s\\geq0$. We again fix an arbitrary $T > 0$. Given that $\\mathcal{Y}_T\\subset L^2([0,T])$ is bounded, the Banach-Saks theorem \\cite{rudin1991functional} implies that for ${\\tau_k}$ (choosing a subsequence if necessary), $\\frac{1}{N}\\sum_{k=1}^Nd^{\\tau_k}(s)$ strongly converges to $\\bar d(s)$ in $L^2([0,T])$ . By the definition of $d(\\cdot)$, we have $d^{\\tau_k}(s)=d_{\\Lambda_\\eta(\\tau_k+s)}$. Now for any $s\\in[0,T]$, we have\n\\[\n\\begin{aligned}\n    &\\|\\nabla\\phi(x(\\lambda_\\eta(\\Lambda_\\eta(\\tau_k+s))))-\\nabla\\phi(\\bar x(s))\\|\\\\\n    \\leq&\\|\\nabla\\phi(x(\\lambda_\\eta(\\Lambda_\\eta(\\tau_k+s))))-\\nabla\\phi(x(\\tau_k+s))\\|+\\|\\nabla\\phi(x^{\\tau_k}(s))-\\nabla\\phi(\\bar x(s))\\|\\\\\n    \\leq&\\|\\nabla\\phi(x(\\lambda_\\eta(\\Lambda_\\eta(\\tau_k+s)+1)))-\\nabla\\phi(x(\\lambda_\\eta(\\Lambda_\\eta(\\tau_k+s))))\\|+\\|\\nabla\\phi(x^{\\tau_k}(s))-\\nabla\\phi(\\bar x(s))\\|\\\\\n    \\leq&\\eta_{\\Lambda_\\eta(\\tau_k+s)}(\\|\\xi_{\\Lambda_\\eta(\\tau_k+s)}\\|+\\|d_{\\Lambda_\\eta(\\tau_k+s)}\\|)+\\|\\nabla\\phi(x^{\\tau_k}(s))-\\nabla\\phi(\\bar x(s))\\|,\n\\end{aligned}\n\\]\nwhich converges to zero as $k\\rightarrow\\infty$. By the continuity of $\\nabla\\phi^*$, we have that $x(\\lambda_\\eta(\\Lambda_\\eta(\\tau_k+s)))$ converges to $\\bar x(s)$ in $\\cC(\\R_+,\\R^n)$. By Assumption \\ref{Assumption: DI}, for almost any $s\\in[0,T]$, we have\n\\[\n\\begin{aligned}\n{\\rm dist}(\\bar d(s),\\mathcal{H}(\\bar x(s)))&\\leq\\bigg\\|\\frac{1}{N}\\sum_{k=1}^Nd^{\\tau_k}(s)-\\bar d(s)\\bigg\\|+{\\rm dist}\\left(\\frac{1}{N}\\sum_{k=1}^Nd^{\\tau_k}(s),\\mathcal{H}(\\bar x(s))\\right)\\\\\n=&\\bigg\\|\\frac{1}{N}\\sum_{k=1}^Nd^{\\tau_k}(s)-\\bar d(s)\\bigg\\|+\n{\\rm dist}\\left(\\frac{1}{N}\\sum_{k=1}^Nd_{\\Lambda_\\eta(\\lambda_\\eta(\\tau_k+s))},\\mathcal{H}(\\bar x(s))\\right)\\rightarrow0.\n\\end{aligned}\n\\]\nSince $T$ is arbitrary and $\\mathcal{H}(\\bar x(s))$ is a closed set, we conclude that $\\bar d(s)\\in\\mathcal{H}(\\bar x(s))$ for almost all $s\\geq0$. This completes the proof.\\qed\n","sha256_utf8":"e0507c19c79e7b657729f322476592ce526f3c4ab32f451187497e077ae5e8d1"}]},{"id":"S007","semantic_type":"proof_step","kind":"proof_step","title":"平移曲线相对紧性","source_units":[{"start":100319,"end":102356,"start_line":1185,"end_line":1193,"raw_tex":"\\noindent{\\textit{Proof of Theorem \\ref{subsequential-thm}\n.}} By the definition of $x_t(\\cdot)$, it follows that $\\nabla\\phi(x_t(s))=\\nabla\\phi(x(t))-\\int_t^s y(u)du$, for all $s\\geq t$. By the boundedness of $y(s)$, Arzel$\\grave{a}$-Ascoli's theorem \\cite{rudin1953principles} ensures that $\\{\\nabla\\phi(x_t(t+\\cdot))\\}_{t\\in\\R_+}$ is relatively compact in $\\cC(\\R_+,\\R^n)$. For any subset $\\{\\tau_k\\}\\subset\\R_+$, we consider the sequence $\\{\\nabla\\phi(x^{\\tau_k}(\\cdot))\\}_{k\\in\\mathbb{N}_+}$. There are two cases to consider. Case (i): the sequence $\\{\\tau_k\\}$ has a cluster point $t$. Without loss of generality, assume that $\\lim_{k\\rightarrow\\infty}\\tau_k=t$. By the definition of $x(\\cdot)$ {in \\eqref{Eq:interpolation}} and uniform boundedness assumption, it holds that  $\\nabla\\phi(x(\\cdot))$ is Lipschitz continuous. Thus, $\\nabla\\phi(x^{\\tau_k}(\\cdot))$ converges to $\\nabla\\phi(x^t(\\cdot))$ in $\\cC(\\R_+,\\R^n)$. Case (ii): $\\lim_{k\\rightarrow\\infty}\\tau_k=\\infty$. Suppose that $\\nabla\\phi(x^{\\tau_k}(\\cdot))$ does not has any cluster point in $\\cC(\\R_+,\\R^n)$. Since $\\{\\nabla\\phi(x_{\\tau_k}(\\tau_k+\\cdot))\\}$ is relatively compact in $\\cC(\\R_+,\\R^n)$, without loss of generality, we assume that $\\lim_{k\\rightarrow\\infty}\\nabla\\phi(x_{\\tau_k}(\\tau_k+\\cdot))=\\bar y(\\cdot)$. Then, for any compact set $C\\subset\\R_+$, it follows from Lemma \\ref{le:shift convergence} that\n\\[\n\\begin{aligned}\n&\\lim_{k\\rightarrow\\infty,s\\in C}\\norm{\\nabla\\phi(x^{\\tau_k}(s))-\\bar y(s)}\\\\\n\\leq&\\lim_{k\\rightarrow\\infty,s\\in C}\\norm{\\nabla\\phi(x^{\\tau_k}(s))-\\nabla\\phi(x_{\\tau_k}(\\tau_k+s))}+\\lim_{k\\rightarrow\\infty,s\\in C}\\norm{\\nabla\\phi(x_{\\tau_k}(\\tau_k+s))-\\bar y(s)}=0,\n\\end{aligned}\n\\]\nwhich contradicts that $\\nabla\\phi(x^{\\tau_k}(\\cdot))$ does not has any cluster point in $\\cC(\\R_+,\\R^n)$. Thus, in both cases, $\\{\\nabla\\phi(x^{\\tau_k}(\\cdot))\\}$ is relatively compact in $\\cC(\\R_+,\\R^n)$. Because $\\{\\tau_k\\}$ is an arbitrary subset in $\\R_+$, we have that $\\{\\nabla\\phi(x^t(\\cdot))\\}$ is relatively compact in $\\cC(\\R_+,\\R^n)$. \n","sha256_utf8":"16640bf7fc868df940ae28e60b58a11d63f1015fa04f3cf210a81257a31627f6"}]},{"id":"S008","semantic_type":"proof_step","kind":"proof_step","title":"弱极限满足积分关系","source_units":[{"start":102357,"end":103658,"start_line":1195,"end_line":1204,"raw_tex":"Next, we aim to construct a trajectory of the differential inclusion. Define the shifts $d^t(\\cdot)=d(t+\\,\\cdot)$. Consider $\\{\\tau_k\\}$ satisfying $\\tau_k\\rightarrow\\infty$, and fix $T>0$. Without loss of generality, we assume that $\\nabla\\phi(x^{\\tau_k}(\\cdot))$ converges to $\\bar y(\\cdot)$ in $\\cC(\\R_+,\\R^n)$, otherwise, we choose its convergent subsequence. The set $\\mathcal{Y}_T:=\\{d^{\\tau_k}(s),s\\in[0,T]\\}_{k\\in\\mathbb{N}}\\subset L^2([0,T])$ is bounded. Therefore, it follows from the Banach-Alaoglu theorem \\cite{rudin1991functional} that $\\cY_T$ is weakly sequentially compact, i.e. there exists a subsequence $\\{\\tau_{k_j}\\}$ and $\\bar y(\\cdot)\\in L^2([0,T])$ such that $d^{\\tau_{k_j}}(\\cdot)\\rightarrow\\bar d(\\cdot)$ weakly in $L^2([0,T])$. On the other hand, by Lemma \\ref{le:shift convergence}, we have $\\nabla\\phi(x_{\\tau_{k_j}}(\\tau_{k_j}+\\cdot))$ \nconverges to $\\bar y(\\cdot)$ in $\\cC(\\R_+,\\R^n)$. For any $\\tau\\in[0,T]$, by the definition of $x_t(\\cdot)$, we have\n\\[\n\\nabla\\phi(x_t(t+\\tau))=\\nabla\\phi(x_t(t))-\\int_0^\\tau d^t(s)ds.\n\\]\nSetting $t=\\tau_{k_j}$ and taking the limit as $k_j\\rightarrow\\infty$, we deduce that\n\\[\n\\bar y(\\tau)=\\bar y(0)-\\int_0^\\tau\\bar d(s)ds.\n\\]\nLet $\\bar x(\\cdot)=\\nabla\\phi^*(\\bar y(\\cdot))$, since $T>0$ is arbitrary, we get \\eqref{Eq:DI_int_form}. \n","sha256_utf8":"843209487d19bec8b808d3d7cd28a5a3dcac9f89cc010fb991b26dd517e46fbb"}]},{"id":"S009","semantic_type":"proof_step","kind":"proof_step","title":"平均强收敛与采样点逼近","source_units":[{"start":103659,"end":105030,"start_line":1206,"end_line":1215,"raw_tex":"The remaining step is to verify that $\\bar d(s)\\in\\mathcal{H}(\\bar x(s))$ for almost all $s\\geq0$. We again fix an arbitrary $T > 0$. Given that $\\mathcal{Y}_T\\subset L^2([0,T])$ is bounded, the Banach-Saks theorem \\cite{rudin1991functional} implies that for ${\\tau_k}$ (choosing a subsequence if necessary), $\\frac{1}{N}\\sum_{k=1}^Nd^{\\tau_k}(s)$ strongly converges to $\\bar d(s)$ in $L^2([0,T])$ . By the definition of $d(\\cdot)$, we have $d^{\\tau_k}(s)=d_{\\Lambda_\\eta(\\tau_k+s)}$. Now for any $s\\in[0,T]$, we have\n\\[\n\\begin{aligned}\n    &\\|\\nabla\\phi(x(\\lambda_\\eta(\\Lambda_\\eta(\\tau_k+s))))-\\nabla\\phi(\\bar x(s))\\|\\\\\n    \\leq&\\|\\nabla\\phi(x(\\lambda_\\eta(\\Lambda_\\eta(\\tau_k+s))))-\\nabla\\phi(x(\\tau_k+s))\\|+\\|\\nabla\\phi(x^{\\tau_k}(s))-\\nabla\\phi(\\bar x(s))\\|\\\\\n    \\leq&\\|\\nabla\\phi(x(\\lambda_\\eta(\\Lambda_\\eta(\\tau_k+s)+1)))-\\nabla\\phi(x(\\lambda_\\eta(\\Lambda_\\eta(\\tau_k+s))))\\|+\\|\\nabla\\phi(x^{\\tau_k}(s))-\\nabla\\phi(\\bar x(s))\\|\\\\\n    \\leq&\\eta_{\\Lambda_\\eta(\\tau_k+s)}(\\|\\xi_{\\Lambda_\\eta(\\tau_k+s)}\\|+\\|d_{\\Lambda_\\eta(\\tau_k+s)}\\|)+\\|\\nabla\\phi(x^{\\tau_k}(s))-\\nabla\\phi(\\bar x(s))\\|,\n\\end{aligned}\n\\]\nwhich converges to zero as $k\\rightarrow\\infty$. By the continuity of $\\nabla\\phi^*$, we have that $x(\\lambda_\\eta(\\Lambda_\\eta(\\tau_k+s)))$ converges to $\\bar x(s)$ in $\\cC(\\R_+,\\R^n)$. By Assumption \\ref{Assumption: DI}, for almost any $s\\in[0,T]$, we have\n","sha256_utf8":"a75db9320abd54dc63cc07dd6495086ed0c585d5b051fa5fffffbaf41e5d1c0e"}]},{"id":"S010","semantic_type":"proof_step","kind":"proof_step","title":"极限方向属于闭值映射","source_units":[{"start":105030,"end":105626,"start_line":1216,"end_line":1223,"raw_tex":"\\[\n\\begin{aligned}\n{\\rm dist}(\\bar d(s),\\mathcal{H}(\\bar x(s)))&\\leq\\bigg\\|\\frac{1}{N}\\sum_{k=1}^Nd^{\\tau_k}(s)-\\bar d(s)\\bigg\\|+{\\rm dist}\\left(\\frac{1}{N}\\sum_{k=1}^Nd^{\\tau_k}(s),\\mathcal{H}(\\bar x(s))\\right)\\\\\n=&\\bigg\\|\\frac{1}{N}\\sum_{k=1}^Nd^{\\tau_k}(s)-\\bar d(s)\\bigg\\|+\n{\\rm dist}\\left(\\frac{1}{N}\\sum_{k=1}^Nd_{\\Lambda_\\eta(\\lambda_\\eta(\\tau_k+s))},\\mathcal{H}(\\bar x(s))\\right)\\rightarrow0.\n\\end{aligned}\n\\]\nSince $T$ is arbitrary and $\\mathcal{H}(\\bar x(s))$ is a closed set, we conclude that $\\bar d(s)\\in\\mathcal{H}(\\bar x(s))$ for almost all $s\\geq0$. This completes the proof.\\qed\n","sha256_utf8":"700a65109de97f02e74f5098ad075b9ca30d0b8f3c502f54a9d83896b05d49b9"}]},{"id":"S011","semantic_type":"lemma","kind":"lemma","title":"插值与辅助流渐近一致","source_units":[{"start":97879,"end":98157,"start_line":1146,"end_line":1153,"raw_tex":"\\begin{lemma}\n\\label{le:shift convergence}\nSuppose Assumption \\ref{Assumption: DI} holds, then for any $T>0$, it holds that\n\\begin{equation}\n\\lim_{t\\rightarrow\\infty}\\sup_{s\\in[t,t+T]}\\|\\nabla\\phi(x(s))-\\nabla\\phi(x_t(s))\\|=0.\n\\label{asymptotic-lemma}\n\\end{equation}\n\\end{lemma}","sha256_utf8":"cbf34445120780485211b0f408b600167b1bf6ff432745006a6068d191a8d36f"}]},{"id":"S012","semantic_type":"proof","kind":"proof","title":"辅助流逼近完整证明","source_units":[{"start":98158,"end":100318,"start_line":1154,"end_line":1184,"raw_tex":"\\begin{proof}\nFix an arbitrary $s\\in[t,t+T]$, let $\\tau_t=\\Lambda_\\eta(t)$, $\\tau_s=\\Lambda_\\eta(s)$. By the definition of $x_t(\\cdot)$, we have\n\\[\n\\begin{aligned}\n&\\nabla\\phi(x_t(s))=\\nabla\\phi(x(t))-\\int_{t}^{\\lambda_\\eta(\\tau_t)}d(u)du-\\int_{\\lambda_\\eta(\\tau_t)}^{\\lambda_\\eta(\\tau_s)}d(u)du-\\int_{\\lambda_\\eta(\\tau_s)}^s d(u)du\\\\\n=&\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))-\\sum_{i=\\tau_t}^{\\tau_s-1}\\eta_id_i+\\left(\\nabla\\phi(x(t))-\\int_t^{\\lambda_\\eta(\\tau_t)}d(u)du-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))\\right)-\\int_{\\lambda_\\eta(\\tau_s)}^s d(u)du\\\\\n=&\\nabla\\phi(x({\\lambda_\\eta(\\tau_s})))+\\sum_{i=\\tau_t}^{\\tau_s-1}\\eta_i\\xi_i+\\left(\\nabla\\phi(x(t))-\\int_t^{\\lambda_\\eta(\\tau_t)}d(u)du-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))\\right)-\\int_{\\lambda_\\eta(\\tau_s)}^s d(u)du\\\\\n=&\\nabla\\phi(x(s))+\\sum_{i=\\tau_t}^{\\tau_s-1}\\eta_i\\xi_i+\\left(\\nabla\\phi(x(t))-\\int_t^{\\lambda_\\eta(\\tau_t)}d(u)du-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))\\right)\\\\\n&+\\left(\\nabla\\phi(x(\\lambda_\\eta(\\tau_s)))-\\int_{\\lambda_\\eta(\\tau_s)}^sd(u)du-\\nabla\\phi(x(s))\\right).\n\\end{aligned}\n\\]\nNote that\n\\[\n\\begin{aligned}\n&\\norm{\\nabla\\phi(x(t))-\\int_t^{\\lambda_\\eta(\\tau_t)}d(u)du-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))}\\\\\n\\leq&{\\|\\nabla\\phi(x(t))-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))\\|}\n+\\int_t^{\\lambda_\\eta(\\tau_t)}\\|d(u)\\|du\n\\\\\n \\leq&{\\|\\nabla\\phi(x(\\lambda_\\eta(\\tau_t+1)))-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))\\|}\n+\\int_t^{\\lambda_\\eta(\\tau_t)}\\|d(u)\\|du\n\\\\\n \\leq&\\eta_{\\tau_t}(\\|\\xi_{\\tau_t}\\|+2\\|d_{\\tau_t}\\|),\n \\end{aligned}\n\\]\nand similarly \n$\n\\big\\|\\nabla\\phi(x(\\lambda_\\eta(\\tau_s)))-\\int_{\\lambda_\\eta(\\tau_s)}^sd(u)du-\\nabla\\phi(x(s))\\big\\|\\leq\\eta_{\\tau_s}(\\|\\xi_{\\tau_s}\\|+2\\|d_{\\tau_s}\\|).\n$\nBy Assumption \\ref{Assumption: DI}, we \nhave that $\\lim\\sup\\limits_{t\\rightarrow\\infty}\\eta_{\\tau_t}(\\|\\xi_{\\tau_t}\\|+2\\|d_{\\tau_t}\\|)=0$, $\\lim\\sup\\limits_{s\\rightarrow\\infty}\\eta_{\\tau_s}(\\|\\xi_{\\tau_s}\\|+2\\|d_{\\tau_s}\\|)=0$, and $\\lim\\limits_{t\\rightarrow\\infty}\\sup\\limits_{s\\in[t,t+T]}\\sum_{i=\\tau_t}^{\\tau_s-1}\\eta_i\\xi_i=0$. Therefore, it holds that $\\lim\\limits_{t\\rightarrow\\infty}\\sup\\limits_{s\\in[t,t+T]}\\|\\nabla\\phi(x(s))-\\nabla\\phi(x_t(s))\\|=0$, which completes the proof.\n\\end{proof}","sha256_utf8":"ee60c15b7dd00de44c1223a849269313d16a75c584af8fb0bcc4e6abea2ef1b9"}]},{"id":"S013","semantic_type":"proof_step","kind":"proof_step","title":"展开流与插值之差","source_units":[{"start":98172,"end":99210,"start_line":1155,"end_line":1164,"raw_tex":"Fix an arbitrary $s\\in[t,t+T]$, let $\\tau_t=\\Lambda_\\eta(t)$, $\\tau_s=\\Lambda_\\eta(s)$. By the definition of $x_t(\\cdot)$, we have\n\\[\n\\begin{aligned}\n&\\nabla\\phi(x_t(s))=\\nabla\\phi(x(t))-\\int_{t}^{\\lambda_\\eta(\\tau_t)}d(u)du-\\int_{\\lambda_\\eta(\\tau_t)}^{\\lambda_\\eta(\\tau_s)}d(u)du-\\int_{\\lambda_\\eta(\\tau_s)}^s d(u)du\\\\\n=&\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))-\\sum_{i=\\tau_t}^{\\tau_s-1}\\eta_id_i+\\left(\\nabla\\phi(x(t))-\\int_t^{\\lambda_\\eta(\\tau_t)}d(u)du-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))\\right)-\\int_{\\lambda_\\eta(\\tau_s)}^s d(u)du\\\\\n=&\\nabla\\phi(x({\\lambda_\\eta(\\tau_s})))+\\sum_{i=\\tau_t}^{\\tau_s-1}\\eta_i\\xi_i+\\left(\\nabla\\phi(x(t))-\\int_t^{\\lambda_\\eta(\\tau_t)}d(u)du-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))\\right)-\\int_{\\lambda_\\eta(\\tau_s)}^s d(u)du\\\\\n=&\\nabla\\phi(x(s))+\\sum_{i=\\tau_t}^{\\tau_s-1}\\eta_i\\xi_i+\\left(\\nabla\\phi(x(t))-\\int_t^{\\lambda_\\eta(\\tau_t)}d(u)du-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))\\right)\\\\\n&+\\left(\\nabla\\phi(x(\\lambda_\\eta(\\tau_s)))-\\int_{\\lambda_\\eta(\\tau_s)}^sd(u)du-\\nabla\\phi(x(s))\\right).\n\\end{aligned}\n\\]\n","sha256_utf8":"ecefb0f747cdde675ab798a5592e5c58420e719db42b2667658e0fccfe17f74d"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"控制两端插值误差","source_units":[{"start":99210,"end":99821,"start_line":1165,"end_line":1181,"raw_tex":"Note that\n\\[\n\\begin{aligned}\n&\\norm{\\nabla\\phi(x(t))-\\int_t^{\\lambda_\\eta(\\tau_t)}d(u)du-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))}\\\\\n\\leq&{\\|\\nabla\\phi(x(t))-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))\\|}\n+\\int_t^{\\lambda_\\eta(\\tau_t)}\\|d(u)\\|du\n\\\\\n \\leq&{\\|\\nabla\\phi(x(\\lambda_\\eta(\\tau_t+1)))-\\nabla\\phi(x(\\lambda_\\eta(\\tau_t)))\\|}\n+\\int_t^{\\lambda_\\eta(\\tau_t)}\\|d(u)\\|du\n\\\\\n \\leq&\\eta_{\\tau_t}(\\|\\xi_{\\tau_t}\\|+2\\|d_{\\tau_t}\\|),\n \\end{aligned}\n\\]\nand similarly \n$\n\\big\\|\\nabla\\phi(x(\\lambda_\\eta(\\tau_s)))-\\int_{\\lambda_\\eta(\\tau_s)}^sd(u)du-\\nabla\\phi(x(s))\\big\\|\\leq\\eta_{\\tau_s}(\\|\\xi_{\\tau_s}\\|+2\\|d_{\\tau_s}\\|).\n$\n","sha256_utf8":"1304f7709f7fc62a087eb84005ebe138e50d321460fb1e5389d61fc26eefcb1a"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"噪声与端点误差消失","source_units":[{"start":99821,"end":100307,"start_line":1182,"end_line":1183,"raw_tex":"By Assumption \\ref{Assumption: DI}, we \nhave that $\\lim\\sup\\limits_{t\\rightarrow\\infty}\\eta_{\\tau_t}(\\|\\xi_{\\tau_t}\\|+2\\|d_{\\tau_t}\\|)=0$, $\\lim\\sup\\limits_{s\\rightarrow\\infty}\\eta_{\\tau_s}(\\|\\xi_{\\tau_s}\\|+2\\|d_{\\tau_s}\\|)=0$, and $\\lim\\limits_{t\\rightarrow\\infty}\\sup\\limits_{s\\in[t,t+T]}\\sum_{i=\\tau_t}^{\\tau_s-1}\\eta_i\\xi_i=0$. Therefore, it holds that $\\lim\\limits_{t\\rightarrow\\infty}\\sup\\limits_{s\\in[t,t+T]}\\|\\nabla\\phi(x(s))-\\nabla\\phi(x_t(s))\\|=0$, which completes the proof.\n","sha256_utf8":"3ceaa09527d292ba87cf40f62b6b7f6ea8d841f3bac2d7aba8c4a1c8b50e3a4f"}]},{"id":"S016","semantic_type":"lemma","kind":"lemma","title":"连续与离散函数值极限一致","source_units":[{"start":105718,"end":106068,"start_line":1229,"end_line":1236,"raw_tex":"\\begin{lemma}\n\\label{le:liminf_limsup}\nSuppose Assumption \\ref{Assumption: DI} is satisfied. Then, it holds that\n\\begin{equation}\n    \\label{Eq:liminf_limsup}\n    \\liminf_{t\\rightarrow\\infty}\\Psi(x(t))=\\liminf_{k\\rightarrow\\infty}\\Psi(x_k),\\quad \\limsup_{t\\rightarrow\\infty}\\Psi(x(t))=\\limsup_{k\\rightarrow\\infty}\\Psi(x_k).\n\\end{equation}\n\\end{lemma}","sha256_utf8":"4a62fb2df20ecaaa731ee41a507198b72dea6d9ca5b68ed78c70d759c5c15509"}]},{"id":"S017","semantic_type":"proof","kind":"proof","title":"上下极限一致完整证明","source_units":[{"start":106069,"end":107569,"start_line":1237,"end_line":1256,"raw_tex":"\\begin{proof}\nFor simplicity, we only prove the case for $\\liminf$, the argument for $\\limsup$ follows similarly. By Assumption \\ref{Assumption: DI}, we have that \n\\begin{equation}\n\\lim_{k\\rightarrow\\infty}\\|\\nabla\\phi(x_{k+1})-\\nabla\\phi(x_k)\\|=0.\n\\label{Eq:succ_gap_convergence}\n\\end{equation}\nLet $\\tau_i\\rightarrow\\infty$ be an arbitrary sequence with $x(\\tau_i)\\rightarrow x^*$. By the definition of $\\lambda_\\eta$ and $\\Lambda_\\eta$, it follows that\n\\[\n\\begin{aligned}\n\\|\\nabla\\phi(x_{\\Lambda_\\eta(\\tau_i)})-\\nabla\\phi(x^*)\\|\\leq&\\|\\nabla\\phi(x_{\\Lambda_\\eta(\\tau_i)})-\\nabla\\phi(x(\\tau_i))\\|+\\|\\nabla\\phi(x(\\tau_i))-\\nabla\\phi(x^*)\\|\\\\\n\\leq&\\|\\nabla\\phi(x_{\\Lambda_\\eta(\\tau_i)})-\\nabla\\phi(x_{\\Lambda_\\eta(\\tau_i)+1})\\|+\\|\\nabla\\phi(x(\\tau_i))-\\nabla\\phi(x^*)\\|.\n\\end{aligned}\n\\]\nThe right-hand side converges to zero. By Remark \\ref{rmk:assumption_DI}.1, we have that $\\Psi\\circ\\nabla\\phi^*$ is continuous, so $\\lim_{i\\rightarrow\\infty}\\Psi(x_{\\Lambda_\\eta(\\tau_i)})=\\lim_{i\\rightarrow\\infty}\\Psi\\circ\\nabla\\phi^*\\circ\\nabla\\phi(x_{\\Lambda_\\eta(\\tau_i)})=\\Psi\\circ\\nabla\\phi^*\\circ\\nabla\\phi(x^*)=\\Psi(x^*)$. By choosing $\\tau_i\\rightarrow\\infty$ as the sequence realizing $\\liminf_{t\\rightarrow\\infty}\\Psi(x(t))$, and assuming without loss of generality that $x(\\tau_i)\\rightarrow x^*$, we get\n\\[   \\liminf_{k\\rightarrow\\infty}\\Psi(x_k)\\leq\n\\lim_{{i\\rightarrow\\infty}}\\Psi(x_{\\Lambda_\\eta(\\tau_i)})=\\Psi(x^*)\n=\\liminf_{t\\rightarrow\\infty}\\Psi(x(t)).\n\\]\nThis completes the proof.\n\\end{proof}","sha256_utf8":"d3f2bf2f4a02c8af92865695f7b0b140050a090527cd3ab6b7ded9b07284421e"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"相邻对偶增量消失","source_units":[{"start":106083,"end":106365,"start_line":1238,"end_line":1242,"raw_tex":"For simplicity, we only prove the case for $\\liminf$, the argument for $\\limsup$ follows similarly. By Assumption \\ref{Assumption: DI}, we have that \n\\begin{equation}\n\\lim_{k\\rightarrow\\infty}\\|\\nabla\\phi(x_{k+1})-\\nabla\\phi(x_k)\\|=0.\n\\label{Eq:succ_gap_convergence}\n\\end{equation}\n","sha256_utf8":"d2bb7a71a082d2a794cd42c44d9d407ed10b7e1a1cb44a0877cca1f0cfd1d5da"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"插值聚点对应离散聚点","source_units":[{"start":106365,"end":107373,"start_line":1243,"end_line":1250,"raw_tex":"Let $\\tau_i\\rightarrow\\infty$ be an arbitrary sequence with $x(\\tau_i)\\rightarrow x^*$. By the definition of $\\lambda_\\eta$ and $\\Lambda_\\eta$, it follows that\n\\[\n\\begin{aligned}\n\\|\\nabla\\phi(x_{\\Lambda_\\eta(\\tau_i)})-\\nabla\\phi(x^*)\\|\\leq&\\|\\nabla\\phi(x_{\\Lambda_\\eta(\\tau_i)})-\\nabla\\phi(x(\\tau_i))\\|+\\|\\nabla\\phi(x(\\tau_i))-\\nabla\\phi(x^*)\\|\\\\\n\\leq&\\|\\nabla\\phi(x_{\\Lambda_\\eta(\\tau_i)})-\\nabla\\phi(x_{\\Lambda_\\eta(\\tau_i)+1})\\|+\\|\\nabla\\phi(x(\\tau_i))-\\nabla\\phi(x^*)\\|.\n\\end{aligned}\n\\]\nThe right-hand side converges to zero. By Remark \\ref{rmk:assumption_DI}.1, we have that $\\Psi\\circ\\nabla\\phi^*$ is continuous, so $\\lim_{i\\rightarrow\\infty}\\Psi(x_{\\Lambda_\\eta(\\tau_i)})=\\lim_{i\\rightarrow\\infty}\\Psi\\circ\\nabla\\phi^*\\circ\\nabla\\phi(x_{\\Lambda_\\eta(\\tau_i)})=\\Psi\\circ\\nabla\\phi^*\\circ\\nabla\\phi(x^*)=\\Psi(x^*)$. By choosing $\\tau_i\\rightarrow\\infty$ as the sequence realizing $\\liminf_{t\\rightarrow\\infty}\\Psi(x(t))$, and assuming without loss of generality that $x(\\tau_i)\\rightarrow x^*$, we get\n","sha256_utf8":"2c46b9828972917d66297323e798d6b7ba3620aab008f7bb867df6084abd3e99"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"实现下极限并结束证明","source_units":[{"start":107373,"end":107558,"start_line":1251,"end_line":1255,"raw_tex":"\\[   \\liminf_{k\\rightarrow\\infty}\\Psi(x_k)\\leq\n\\lim_{{i\\rightarrow\\infty}}\\Psi(x_{\\Lambda_\\eta(\\tau_i)})=\\Psi(x^*)\n=\\liminf_{t\\rightarrow\\infty}\\Psi(x(t)).\n\\]\nThis completes the proof.\n","sha256_utf8":"5fbc86b13f6ed50406e0cf33915e6cf59dbb96b411f56e02c5f77180e2243ec4"}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","title":"插值函数值收敛","source_units":[{"start":107952,"end":108151,"start_line":1260,"end_line":1263,"raw_tex":"\\begin{lemma}\n    \\label{prop:nonescape}\n    Suppose Assumption \\ref{Assumption: DI} and \\ref{assumption_Sard_Lyapunov} hold, then function value $\\Psi(x(t))$ converges {as $t\\to\\infty$.}\n\\end{lemma}","sha256_utf8":"465c801cf775f7359484d2a7e6827989eb891765d960573fad8cc3eeb78fb255"}]},{"id":"S022","semantic_type":"proof","kind":"proof","title":"不逃逸论证完整证明","source_units":[{"start":108152,"end":111488,"start_line":1264,"end_line":1283,"raw_tex":"\\begin{proof}\nAssuming $\\liminf_{t\\rightarrow\\infty}\\Psi(x(t))=0$, we define the level set $\\cL_r:=\\{x\\in\\R^n:\\;\\Psi(x)\\leq r\\}$. Choose any $\\epsilon>0$ such that $\\epsilon\\notin\\Psi(\\mathcal{H}^{-1}(0))$. The weak Morse-Sard condition in Assumption \\ref{Assumption: DI} implies that $\\epsilon$ can be chosen arbitrarily small, and Lemma \\ref{le:liminf_limsup} implies that there are infinitely many $k$ such that $x_k\\in\\cL_\\epsilon$. For any $x_k\\in\\cL_\\epsilon$, by the continuity of $\\Psi$, we have that there exists $\\alpha>0$ such that ${\\rm dist}(x_k,\\R^n\\setminus\\cL_{2\\epsilon})>\\alpha$. By \\eqref{Eq:succ_gap_convergence}, for sufficiently large $k$, we have that $\\norm{x_{k+1}-x_k}<\\alpha$. Therefore, for all large $k$, $x_k\\in\\cL_\\epsilon$ implies that $x_{k+1}\\in\\cL_{2\\epsilon}$.\nNow, we define the last entrance and the first subsequent exit times,\n\\begin{equation}\n\\begin{aligned}\n    &k_i=\\max\\{m\\geq j_{i-1}:x_m\\in\\cL_\\epsilon\\}, \\quad\n    &j_i=\\min\\{m\\geq k_i:x_m\\in\\R^n\\setminus\\cL_{2\\epsilon}\\}.\n\\end{aligned}\n\\label{Eq:upcross_def}\n\\end{equation}\nWe prove that such upcrossing occurs for finite times. Otherwise, if there exists $\\{k_i\\}$ such that $\\lim_{i\\rightarrow\\infty}k_i=\\infty$, then Theorem \\ref{subsequential-thm} indicates that, up to a subsequence, $\\nabla\\phi(x^{\\lambda_\\eta(k_i)}(\\cdot))$ converges to $\\nabla\\phi(\\bar x(\\cdot))$, where $\\bar x(\\cdot)$ is a trajectory of \\eqref{Eq:Breg_DI}. By the definition of $k_i$, we have $\\Psi(x_{k_i})\\leq\\epsilon$, $\\Psi(x_{k_i+1})>\\epsilon$. By \\eqref{Eq:succ_gap_convergence} and the continuity of $\\Psi\\circ\\nabla\\phi^*$, we have that $\\lim_{i\\rightarrow\\infty}\\Psi(x_{k_i})=\\lim_{i\\rightarrow\\infty}\\Psi(x_{k_i+1})=\\epsilon$. Recall that $x^{\\lambda_\\eta(k_i)}(0)=x_{k_i}$, therefore, $\\Psi(\\bar x(0))=\\lim_{i\\rightarrow\\infty}\\Psi(x_{k_i})=\\epsilon$. Since $\\bar x(0)$ is not in the stable set, there exists $T>0$, such that  \n\\[\n\\Psi(\\bar x(T))<\\sup_{s\\in[0,T]}\\Psi(\\bar x(s))\\leq\\Psi(\\bar x(0))=\\epsilon.\n\\]\nThen, there exists $\\delta>0$, such that $\\Psi(\\bar x(T))\\leq\\epsilon-2\\delta$. Moreover, for sufficiently large $i$, we have\n\\[\n\\sup_{s\\in[0,T]}\\Psi(x^{\\lambda_\\eta(k_i)}(s))\\leq\\sup_{s\\in[0,T]}\\Psi(\\bar x(s))+\\sup_{s\\in[0,T]}|\\Psi(x^{\\lambda_\\eta(k_i)}(s))-\\Psi(\\bar x(s))|\\leq2\\epsilon.\n\\]\nThe last inequality comes from the uniform convergence of $\\{\\nabla\\phi(x^{\\lambda_\\eta(k_i)}(\\cdot))\\}$ in $\\cC(\\R_+,\\R^n)$. This implies that for all large $i$, $\\{x(\\lambda_\\eta(k_i)+s):s\\in[0,T]\\}\\subset\\cL_{2\\epsilon}$. Thus it holds that $\\lambda_\\eta(j_i)>\\lambda_\\eta(k_i)+T$. Let $l_i=\\max\\{m:\\lambda_\\eta(k_i)\\leq \\lambda_\\eta(m)\\leq \\lambda_\\eta(k_i)+T\\}$. Then $\\|\\nabla\\phi(x_{l_i})-\\nabla\\phi(x^{\\lambda_\\eta(k_i)}(T))\\|\\leq\\|\\nabla\\phi(x_{l_i})-\\nabla\\phi(x_{l_i+1})\\|\\rightarrow0$, and hence $\\norm{\\nabla\\phi(x_{l_i})-\\nabla\\phi(\\bar x(T))}\\rightarrow0$ {as $i\\to\\infty$}. By the continuity of $\\Psi\\circ\\nabla\\phi^*$, we have that $\\Psi(x_{l_i})\\leq\\epsilon-\\delta$ for all large $i$. By the definition of $k_i$ and $j_i$, we have that $\\lambda_\\eta(j_i)<\\lambda_\\eta(l_i)\\leq\\lambda_\\eta(k_i)+T$, which leads to a contradiction. Therefore, for all large $k$, $\\Psi(x_k)\\leq2\\epsilon$. Since $\\epsilon$ can be chosen arbitrarily small, it holds that $\\lim_{k\\rightarrow\\infty}\\Psi(x_k)=0$. This completes the proof.\n\\end{proof}","sha256_utf8":"eaeb18f8081a64300a81e5200af85bef0be049d8b8e20c7753ac112413f74269"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"选取非临界水平及上穿时刻","source_units":[{"start":108166,"end":109224,"start_line":1265,"end_line":1273,"raw_tex":"Assuming $\\liminf_{t\\rightarrow\\infty}\\Psi(x(t))=0$, we define the level set $\\cL_r:=\\{x\\in\\R^n:\\;\\Psi(x)\\leq r\\}$. Choose any $\\epsilon>0$ such that $\\epsilon\\notin\\Psi(\\mathcal{H}^{-1}(0))$. The weak Morse-Sard condition in Assumption \\ref{Assumption: DI} implies that $\\epsilon$ can be chosen arbitrarily small, and Lemma \\ref{le:liminf_limsup} implies that there are infinitely many $k$ such that $x_k\\in\\cL_\\epsilon$. For any $x_k\\in\\cL_\\epsilon$, by the continuity of $\\Psi$, we have that there exists $\\alpha>0$ such that ${\\rm dist}(x_k,\\R^n\\setminus\\cL_{2\\epsilon})>\\alpha$. By \\eqref{Eq:succ_gap_convergence}, for sufficiently large $k$, we have that $\\norm{x_{k+1}-x_k}<\\alpha$. Therefore, for all large $k$, $x_k\\in\\cL_\\epsilon$ implies that $x_{k+1}\\in\\cL_{2\\epsilon}$.\nNow, we define the last entrance and the first subsequent exit times,\n\\begin{equation}\n\\begin{aligned}\n    &k_i=\\max\\{m\\geq j_{i-1}:x_m\\in\\cL_\\epsilon\\}, \\quad\n    &j_i=\\min\\{m\\geq k_i:x_m\\in\\R^n\\setminus\\cL_{2\\epsilon}\\}.\n\\end{aligned}\n\\label{Eq:upcross_def}\n\\end{equation}\n","sha256_utf8":"5c40d7adf8e54df8b3b6e30a57642fe40bd516909d9e1c54c1783c73da4ca026"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"无限上穿产生下降轨迹","source_units":[{"start":109224,"end":110150,"start_line":1274,"end_line":1277,"raw_tex":"We prove that such upcrossing occurs for finite times. Otherwise, if there exists $\\{k_i\\}$ such that $\\lim_{i\\rightarrow\\infty}k_i=\\infty$, then Theorem \\ref{subsequential-thm} indicates that, up to a subsequence, $\\nabla\\phi(x^{\\lambda_\\eta(k_i)}(\\cdot))$ converges to $\\nabla\\phi(\\bar x(\\cdot))$, where $\\bar x(\\cdot)$ is a trajectory of \\eqref{Eq:Breg_DI}. By the definition of $k_i$, we have $\\Psi(x_{k_i})\\leq\\epsilon$, $\\Psi(x_{k_i+1})>\\epsilon$. By \\eqref{Eq:succ_gap_convergence} and the continuity of $\\Psi\\circ\\nabla\\phi^*$, we have that $\\lim_{i\\rightarrow\\infty}\\Psi(x_{k_i})=\\lim_{i\\rightarrow\\infty}\\Psi(x_{k_i+1})=\\epsilon$. Recall that $x^{\\lambda_\\eta(k_i)}(0)=x_{k_i}$, therefore, $\\Psi(\\bar x(0))=\\lim_{i\\rightarrow\\infty}\\Psi(x_{k_i})=\\epsilon$. Since $\\bar x(0)$ is not in the stable set, there exists $T>0$, such that  \n\\[\n\\Psi(\\bar x(T))<\\sup_{s\\in[0,T]}\\Psi(\\bar x(s))\\leq\\Psi(\\bar x(0))=\\epsilon.\n\\]\n","sha256_utf8":"5560a292ee5f4a2a1c01b6bde4bf7be0eeb9861a7fd99a40e1a801b98e8b6e41"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"上穿矛盾推出函数值收敛","source_units":[{"start":110150,"end":111477,"start_line":1278,"end_line":1282,"raw_tex":"Then, there exists $\\delta>0$, such that $\\Psi(\\bar x(T))\\leq\\epsilon-2\\delta$. Moreover, for sufficiently large $i$, we have\n\\[\n\\sup_{s\\in[0,T]}\\Psi(x^{\\lambda_\\eta(k_i)}(s))\\leq\\sup_{s\\in[0,T]}\\Psi(\\bar x(s))+\\sup_{s\\in[0,T]}|\\Psi(x^{\\lambda_\\eta(k_i)}(s))-\\Psi(\\bar x(s))|\\leq2\\epsilon.\n\\]\nThe last inequality comes from the uniform convergence of $\\{\\nabla\\phi(x^{\\lambda_\\eta(k_i)}(\\cdot))\\}$ in $\\cC(\\R_+,\\R^n)$. This implies that for all large $i$, $\\{x(\\lambda_\\eta(k_i)+s):s\\in[0,T]\\}\\subset\\cL_{2\\epsilon}$. Thus it holds that $\\lambda_\\eta(j_i)>\\lambda_\\eta(k_i)+T$. Let $l_i=\\max\\{m:\\lambda_\\eta(k_i)\\leq \\lambda_\\eta(m)\\leq \\lambda_\\eta(k_i)+T\\}$. Then $\\|\\nabla\\phi(x_{l_i})-\\nabla\\phi(x^{\\lambda_\\eta(k_i)}(T))\\|\\leq\\|\\nabla\\phi(x_{l_i})-\\nabla\\phi(x_{l_i+1})\\|\\rightarrow0$, and hence $\\norm{\\nabla\\phi(x_{l_i})-\\nabla\\phi(\\bar x(T))}\\rightarrow0$ {as $i\\to\\infty$}. By the continuity of $\\Psi\\circ\\nabla\\phi^*$, we have that $\\Psi(x_{l_i})\\leq\\epsilon-\\delta$ for all large $i$. By the definition of $k_i$ and $j_i$, we have that $\\lambda_\\eta(j_i)<\\lambda_\\eta(l_i)\\leq\\lambda_\\eta(k_i)+T$, which leads to a contradiction. Therefore, for all large $k$, $\\Psi(x_k)\\leq2\\epsilon$. Since $\\epsilon$ can be chosen arbitrarily small, it holds that $\\lim_{k\\rightarrow\\infty}\\Psi(x_k)=0$. This completes the proof.\n","sha256_utf8":"c3427da27ea2ba3ce4ed674377d22551d3a31d3f3c700f7b488adf1fac6b31c6"}]},{"id":"S026","semantic_type":"notation","kind":"notation","title":"共轭距离与时间索引记号","source_units":[{"start":17325,"end":19103,"start_line":221,"end_line":222,"raw_tex":"Given a proper and lower semicontinuous function $f:\\mathbb{R}^n\\rightarrow\\overline{\\mathbb{R}} := (-\\infty,\\infty]$, we denote its domain as $\\dom\\,f=\\{x:f(x)<\\infty\\}$. The Fenchel conjugate function of $f$ is defined as $f^*(y):=\\sup\\{\\langle x,y\\rangle - f(x):x\\in\\mathbb{R}^n\\}$. For a set $\\mathcal{S}\\subset\\mathbb{R}^n$, we use ${\\rm cl\\,\\mathcal{S}}$ to denote its closure, ${\\rm int}\\,\\mathcal{S}$ to denote the set of its interior points, and ${\\rm conv}(\\cS)$ to denote its convex hull. A function $f:\\cS\\rightarrow\\R$ is said to be of class ${\\cC}^k(\\mathcal{S})$ if it is $k$ times differentiable and the $k$-th derivative is continuous on $\\mathcal{S}$. When there is no ambiguity regarding the domain, we simply use the notation ${\\cC}^k$. We let $\\mathcal{C}(A,B)$ be the set of continuous mappings from set $A$ to set $B$. We use $\\|\\cdot\\|$ to denote the Euclidean norm for vectors and the Frobenius matrix norm for matrices. The $d$-dimensional unit ball is denoted by $\\mathbb{B}^d$. The distance between a point $w$ and a set $A$ is denoted by $\\dist(w,A):=\\inf\\{\\|w-u\\|:\\;u\\in A\\}$. We use the convention $\\dist(w,\\emptyset):=\\infty$. {The Minkowski sum of two sets $A$ and $B$ is denoted as \n$A+B:=\\{u+v:\\;u\\in A,\\;v\\in B\\}$. We use $\\alpha A:=\\{\\alpha u:\\;u\\in A\\}$ to denote the scaled set of $A$ by a given scalar $\\alpha$.} For a positive sequence $\\{\\eta_k\\}$, we define $\\lambda_\\eta(0):=0$, $\\lambda_\\eta(k):=\\sum_{i=0}^{k-1}\\eta_i$ for $k\\geq1$, and $\\Lambda_\\eta(t):=\\sup\\{k:\\lambda_\\eta(k)\\leq t\\}$. In other word, $\\Lambda_\\eta(t)=k$ if and only if $\\lambda_\\eta(k)\\leq t<\\lambda_\\eta(k+1)$. {Given a convex function $g$, we say $x$ is a $\\nu$-optimal solution of the problem $\\min_{x\\in\\R^n}\\;g(x)$ if ${\\rm dist}(0,\\partial g(x))\\leq\\nu$.}\n","sha256_utf8":"53b99c244878fe3edf32853e13fd4a720b738c23fc922190446c0a8df569ea17"}]},{"id":"S027","semantic_type":"definition","kind":"definition","title":"闭图与外半连续映射","source_units":[{"start":19625,"end":20465,"start_line":231,"end_line":241,"raw_tex":"\\begin{definition}\\label{graph-def}\nLet $\\mathcal{X}\\subset\\mathbb{R}^n$. A set-valued mapping $S:\\mathcal{X}\\rightrightarrows\\mathbb{R}^m$ is said to be closed if its graph, defined by\n\\[\n{\\rm graph}(S):=\\{(x,y)\\in\\R^n\\times\\R^m:y\\in S(x)\\}\n\\]\nis a closed set in $\\R^n\\times\\R^m$. Give a nonnegative $\\delta$, the $\\delta$-perturbed set of $S$ is defined by \n\\[\nS^\\delta(x):=\\cup_{\\{y\\in\\R^n:\\norm{y-x}\\leq\\delta\\}}\\left(S(y)+\\delta\\mathbb{B}^m\\right).\n\\]\n$S:\\mathcal{X}\\rightrightarrows\\mathbb{R}^m$ is called outer semicontinuous at $\\bar{x}\\in\\mathcal{X}$ if for any sequence $x_i\\overset{\\mathcal{X}}{\\rightarrow}\\bar x$ and $v_i\\in S(x_i)$ converging to some $\\bar{v}\\in\\mathbb{R}^m$, we have $\\bar{v}\\in S(\\bar{x})$. $S$ is said to be outer semicontinuous if it is outer semicontinuous everywhere over $\\mathcal{X}$.\n\\end{definition}","sha256_utf8":"453d67bbc115eb27c914f0d077496daca089dfe936892d8b0337f2dce872911c"}]},{"id":"S028","semantic_type":"definition","kind":"definition","title":"核函数与Bregman距离","source_units":[{"start":32718,"end":33166,"start_line":352,"end_line":357,"raw_tex":"\\begin{definition}\\label{Bregman-distance-def}\n(Kernel function and Bregman distance over $\\R^n$).  A function $\\phi:\\R^n\\rightarrow\\R$ is called a kernel function over $\\R^n$, if $\\phi$ is convex and $\\phi\\in\\cC^1(\\R^n)$. The Bregman distance \\cite{bregman1967relaxation} generated by $\\phi$ is denoted as $\\cD_\\phi(x,y):\\R^n\\times\\R^n\\rightarrow[0,+\\infty)$, where\n\\[\n\\cD_\\phi(x,y)=\\phi(x)-\\phi(y)-\\inprod{\\nabla\\phi(y)}{x-y}.\n\\]\n\\end{definition}","sha256_utf8":"43b4d1c96f097f790180e71614547f72429dca594ce0b6ed4d3f69af56fcf8b1"}]},{"id":"S029","semantic_type":"definition","kind":"definition","title":"Legendre核定义","source_units":[{"start":33168,"end":33559,"start_line":359,"end_line":361,"raw_tex":"\\begin{definition}\\label{Legendre}\n(Legendre kernel over $\\R^n$). Let $\\phi$ be a kernel function over $\\R^n$, such that $\\lim_{k\\rightarrow\\infty}\\|\\nabla\\phi({x_k})\\|=\\infty$ whenever $\\{{x_k}\\}_{k\\in\\mathbb N}$ satisfies $\\lim_{k\\rightarrow\\infty}\\norm{x_k}=\\infty$. The function $\\phi$ is called a Legendre function over $\\R^n$ if it is also {strictly} convex on $\\R^n$.\n\\end{definition}","sha256_utf8":"406aa752bc0cd75f6c20810ee5200b7612697ad13fb525bdd02973cb68dc67e8"}]},{"id":"S030","semantic_type":"definition","kind":"definition","title":"紧时间区间上一致收敛","source_units":[{"start":35114,"end":35539,"start_line":381,"end_line":384,"raw_tex":"We end this subsection by introducing the concept of functional convergence over any compact set, as discussed in literature such as \\cite{benaim2005stochastic,duchi2018stochastic,davis2020stochastic}. Given a sequence of mappings $f_n\\in\\cC(\\mathbb{R}_+,\\mathbb{R}^n)$, we say that $f_n$ converges to $f$ in $\\cC(\\R_+,\\R^n)$ if, for any $T>0$, it holds that\n\\[\n\\lim_{n\\rightarrow\\infty}\\sup_{t\\in[0,T]}\\|f_n(t)-f(t)\\|=0.\n\\]\n","sha256_utf8":"d82adb80a08d75e8b3d6d3be9df772e7f1acdc62eb0254056d46758021f64504"}]},{"id":"S031","semantic_type":"problem_setting","kind":"problem_setting","title":"Bregman微分包含及稳定集","source_units":[{"start":36949,"end":37091,"start_line":397,"end_line":400,"raw_tex":"\\begin{equation}\n    \\label{Eq:Breg_DI}\n    \\frac{d}{dt}\\nabla\\phi(x(t))\\in-\\mathcal{H}(x(t)), \\text{ for almost all $t\\geq0$.}\n\\end{equation}","sha256_utf8":"bafa07b0138ad289fa8b6dfc7caa658f465c888c594c05edc87f6bd686887d36"},{"start":37092,"end":37342,"start_line":401,"end_line":405,"raw_tex":"Any absolutely continuous solution to \\eqref{Eq:Breg_DI} is termed a trajectory of \\eqref{Eq:Breg_DI}. The stable set of \\eqref{Eq:Breg_DI} is defined as \n\\begin{equation}\n\\label{eq:def_stable_set}\n\\cH^{-1}(0):=\\{x\\in\\R^n:0\\in\\cH(x)\\}.\n\\end{equation}","sha256_utf8":"281c1eaf84e372f5fecdf9a1a00894df7caa4328356a074632dcb03c9b6470a0"}]},{"id":"S032","semantic_type":"algorithm","kind":"algorithm","title":"含随机噪声的一般迭代","source_units":[{"start":37702,"end":37826,"start_line":407,"end_line":409,"raw_tex":"\\begin{equation}\n   \\label{Eq:Breg_general_iterative}  \\nabla\\phi(x_{k+1})=\\nabla\\phi(x_k)-\\eta_k(d_k+\\xi_k),\n\\end{equation}","sha256_utf8":"576e1be34ae8bae7ad9d5fa7d9feab2862abe7a490e36a07dff50e5fb6850f6c"},{"start":37827,"end":38243,"start_line":410,"end_line":410,"raw_tex":"where $d_k$ is an evaluation of $\\cH(x_k)$ with possible inexactness, and $\\xi_k$ is the stochastic noise. This formulation, referred to as the general Bregman-type method, is notable for the versatile choices of $\\phi$ and $\\cH$. For the Bregman counterpart of the interpolated process \\eqref{Eq:x(t)_def} in the Euclidean setting, we introduce the linear interpolation for the dual sequence $\\{\\nabla\\phi(x_k)\\}$:\n","sha256_utf8":"20cbd5a83e80f12a3fd5376d7d017c2e6c4bd1fe9103154258da93d028451a03"}]},{"id":"S033","semantic_type":"definition","kind":"definition","title":"对偶线性插值与时间平移","source_units":[{"start":38243,"end":38462,"start_line":411,"end_line":414,"raw_tex":"\\begin{equation}\n \\label{Eq:interpolation}\n x(t):=\\nabla\\phi^*\\left(\\nabla\\phi(x_k)+\\frac{t-\\lambda_\\eta(k)}{\\eta_k}(\\nabla\\phi(x_{k+1})-\\nabla\\phi(x_k))\\right),\\;t\\in[\\lambda_\\eta(k),\\lambda_\\eta(k+1)). \n\\end{equation}","sha256_utf8":"5990c260c909893407cc63a9f2d0d669f32a8ae8bfc4656cb204e6938a9b51c3"},{"start":38463,"end":38880,"start_line":415,"end_line":415,"raw_tex":"If $\\phi^*\\in\\cC^1(\\R^n)$, then \\eqref{Eq:interpolation} is well defined. Let $x^t(\\cdot)$ denote the time-shifted curve of the interpolated process, i.e. $x^t(\\cdot)=x(t+\\cdot)$. We make the following assumptions on \\eqref{Eq:Breg_DI} and \\eqref{Eq:Breg_general_iterative} to ensure that the iterative sequence generated by \\eqref{Eq:Breg_general_iterative} tracks a trajectory of \\eqref{Eq:Breg_DI} asymptotically.\n","sha256_utf8":"9951496337d5586bb395ea6ce8dcaf423500f53d610fdaf6a955177e9edaa537"}]},{"id":"S034","semantic_type":"assumption","kind":"assumption","title":"核有界性步长噪声与一致性","source_units":[{"start":38880,"end":39972,"start_line":416,"end_line":432,"raw_tex":"\\begin{assumption}\n\\label{Assumption: DI}\n    \\begin{enumerate}\n        \\item {$\\phi$ is a supercoecive Legendre kernel function over $\\R^n$}, and $\\nabla\\phi$ is differentiable almost everywhere.\n        \\item The sequences $\\{x_k\\}$, $\\{\\nabla\\phi(x_k)\\}$ and $\\{d_k\\}$ are uniformly bounded.\n        \\item The stepsize $\\{\\eta_k\\}$ satisfies $\\sum_{k=0}^\\infty\\eta_k=\\infty$ and $\\lim_{k\\rightarrow\\infty}\\eta_k=0$.\n        \\item For any $T>0$, the noise sequence $\\{\\xi_k\\}$ satisfies \n        \\begin{equation}\n            \\lim_{s\\rightarrow\\infty}\\sup_{s\\leq i\\leq\\Lambda_\\eta(\\lambda_\\eta(s)+T)}\\norm{\\sum_{k=s}^i\\eta_k\\xi_k}=0.\n            \\label{Eq:noise_cond}\n        \\end{equation}       \n        \\item The set-valued mapping $\\mathcal{H}$ has a closed graph. Additionally, for any unbounded increasing sequence $\\{k_j\\}$ such that $\\{x_{k_j}\\}$ converges to $\\bar x$, it holds that \n        \\begin{equation}\n            \\lim_{N\\rightarrow\\infty}{\\rm dist}\\left(\\frac{1}{N}\\sum_{j=1}^Nd_{k_j},\\mathcal{H}(\\bar x)\\right)=0.\n        \\end{equation}\n    \\end{enumerate}\n\\end{assumption}","sha256_utf8":"1718a1c1ba24aa67b6afc661bf2d0fea71f80920c8af61d019b888f2eebb0229"}]},{"id":"S035","semantic_type":"assumption","kind":"assumption","title":"弱Morse–Sard与Lyapunov条件","source_units":[{"start":44621,"end":45258,"start_line":472,"end_line":482,"raw_tex":"\\begin{assumption}\\label{assumption_Sard_Lyapunov}\n There exists a continuous function $\\Psi:\\R^n\\to\\R$, such that the following conditions hold:\n \\begin{enumerate}\n     \\item (Weak Morse-Sard). The set \n     {$\\{\\Psi(x): \\mbox{for $x$ such that $0\\in\\cH(x)$} \\}$}  has empty interior in $\\R$.\n     \\item (Lyapunov function). $\\Psi$ is lower bounded, i.e. $\\liminf_{x\\in\\R^n}\\Psi(x)>-\\infty$. For any trajectory $z(t)$ of the differential inclusion \\eqref{Eq:Breg_DI} with $z(0)\\notin\\cH^{-1}(0)$,  there exists $T > 0$ such that\n     \\[\n     \\Psi(z(T))<\\sup_{t\\in[0,T]}\\Psi(z(t))\\leq\\Psi(z(0)).\n     \\]\n \\end{enumerate}\n\\end{assumption}","sha256_utf8":"b556a5ff7da80358a81ece9f3e0b8e0ca6bc289563938e11d82230807be75024"}]},{"id":"S036","semantic_type":"derived_fact","kind":"derived_fact","title":"共轭梯度性质与框架假设说明","source_units":[{"start":40032,"end":41787,"start_line":434,"end_line":446,"raw_tex":"\\begin{remark}\n    \\label{rmk:assumption_DI}\n    \\begin{enumerate}\n        \\item {If the Legendre function $\\phi$ is supercoecive, i.e. $\\lim_{\\|u\\|\\rightarrow\\infty}\\frac{\\phi(u)}{\\|u\\|}=\\infty$, then by \\cite[Theorem 26.5,\\,Corollary 13.3.1]{rockafellar1997convex}, $\\phi^*\\in\\cC^1(\\R^n)$ is strictly convex, and $(\\nabla\\phi)^{-1}=\\nabla\\phi^*$.}\n        \\item Uniform boundedness of either $\\{x_k\\}$ or $\\{\\nabla\\phi(x_k)\\}$ may suffice under mild conditions. For example, if $\\phi$ is locally strongly convex, then the uniform boundedness of $\\{\\nabla\\phi(x_k)\\}$ implies the uniform boundedness of $\\{x_k\\}$. Conversely, if $\\nabla\\phi$ is locally Lipschitz continuous, then the uniform boundedness of $\\{x_k\\}$ leads to the uniform boundedness of $\\{\\nabla\\phi(x_k)\\}$.\n        \\item For a martingale difference noise sequence $\\{\\xi_k\\}$,  i.e. $\\E[\\xi_{k+1}|\\cF_k]=0$ holds  almost surely,  as shown in \\cite{benaim2005stochastic}, \\eqref{Eq:noise_cond} can be ensured almost surely under one of the following conditions:\n        \\begin{enumerate}\n            \\item Uniform boundedness of $\\{\\xi_k\\}$ with $\\eta_k=o\\left(\\frac{1}{\\log k}\\right)$ as utilized in \\cite{castera2021inertial,xiao2023adam,xiao2023convergence}.\n            \\item Variance-bounded $\\{\\xi_k\\}$, i.e. $\\E[\\norm{\\xi_{k+1}}^2|\\cF_k]\\leq\\sigma^2$ for some $\\sigma>0$, and square-summable $\\{\\eta_k\\}$, i.e. $\\sum_{k=0}^\\infty\\eta_k^2<\\infty$, as employed in \\cite{davis2020stochastic}.\n        \\end{enumerate}\n        \\item Assumption \\ref{Assumption: DI}(5) describes how $d_k$ approximates $\\cH(x_k)$. This assumption is quite mild. A sufficient condition for it hold is that $d_k\\in\\cH^{\\delta_k}(x_k)$ with $\\lim_{k\\to\\infty}\\delta_k=0$. \n    \\end{enumerate}\n\\end{remark}","sha256_utf8":"d9d1445aa35dd14eccedc223efcea4a057cafd51e1f38e4cfd1c394e5f0f8eaa"}]},{"id":"S037","semantic_type":"definition","kind":"definition","title":"分段常值方向与辅助流","source_units":[{"start":97049,"end":97879,"start_line":1140,"end_line":1145,"raw_tex":"To establish under Assumption \\ref{Assumption: DI} that, the discrete sequence generated by the general Bregman-type method \\eqref{Eq:Breg_general_iterative} tracks a trajectory of the differential inclusion \\eqref{Eq:Breg_DI}, we introduce a piecewise constant mapping $d(\\cdot)$ defined by $d(s)=d_k$ for any $s\\in[\\lambda_\\eta(k),\\lambda_\\eta(k+1))$. We also define a time-shifted solution $x_t(\\cdot)$ to the following ordinary differential equation:\n\\[\n\\frac{d}{ds}\\nabla\\phi({x_t(s)})=-d(s)\\;for\\;all\\;s\\geq t,\\;\\mbox{with initial condition}\\;\\nabla\\phi(x_t(t))=\\nabla\\phi(x(t)).\n\\]\nThe following lemma suggests that the interpolated process $x(t)$ defined by \\eqref{Eq:interpolation} asymptotically approximates this time-shifted solution.\nWe remind the reader to note the difference between $x_t(\\cdot)$ and $x^t(\\cdot)$.\n","sha256_utf8":"9ecb7e6aaedc227656866a5d6b93441131f52d80763427883d15f084e7fe2b40"}]},{"id":"S038","semantic_type":"external_fact","kind":"external_fact","title":"Legendre共轭定理","source_units":[],"declared_external":true,"declaration_text":["For the supercoercive Legendre kernel on Rⁿ used here, its Fenchel conjugate is C¹ and strictly convex, and ∇φ* = (∇φ)⁻¹; see Rockafellar, Convex Analysis, Theorem 26.5 and Corollary 13.3.1."]},{"id":"S039","semantic_type":"external_fact","kind":"external_fact","title":"Arzelà–Ascoli紧性定理","source_units":[],"declared_external":true,"declaration_text":["A uniformly bounded, equicontinuous family of finite-dimensional continuous functions on a compact interval is relatively compact in the uniform topology. A diagonal subsequence gives convergence on every compact time interval."]},{"id":"S040","semantic_type":"external_fact","kind":"external_fact","title":"L²弱紧性与积分连续性","source_units":[],"declared_external":true,"declaration_text":["Every bounded sequence in L²([0,T]; Rⁿ) has a weakly convergent subsequence. For each fixed τ in [0,T], the map d ↦ ∫₀^τ d(s) ds is a continuous linear functional and therefore preserves weak limits."]},{"id":"S041","semantic_type":"external_fact","kind":"external_fact","title":"Banach–Saks平均收敛定理","source_units":[],"declared_external":true,"declaration_text":["In L², a weakly convergent sequence admits a subsequence whose Cesàro means converge strongly to its weak limit. Strong L² convergence admits an almost-everywhere convergent subsequence."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S005","target_id":"S003","kind":"supports"},{"source_id":"S035","target_id":"S003","kind":"supports"},{"source_id":"S036","target_id":"S003","kind":"supports"},{"source_id":"S021","target_id":"S004","kind":"supports"},{"source_id":"S006","target_id":"S005","kind":"supports"},{"source_id":"S007","target_id":"S006","kind":"supports"},{"source_id":"S008","target_id":"S006","kind":"supports"},{"source_id":"S009","target_id":"S006","kind":"supports"},{"source_id":"S010","target_id":"S006","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S011","target_id":"S007","kind":"supports"},{"source_id":"S011","target_id":"S008","kind":"supports"},{"source_id":"S039","target_id":"S007","kind":"supports"},{"source_id":"S040","target_id":"S008","kind":"supports"},{"source_id":"S041","target_id":"S009","kind":"supports"},{"source_id":"S034","target_id":"S007","kind":"supports"},{"source_id":"S034","target_id":"S008","kind":"supports"},{"source_id":"S034","target_id":"S009","kind":"supports"},{"source_id":"S034","target_id":"S010","kind":"supports"},{"source_id":"S033","target_id":"S007","kind":"supports"},{"source_id":"S033","target_id":"S009","kind":"supports"},{"source_id":"S036","target_id":"S008","kind":"supports"},{"source_id":"S036","target_id":"S009","kind":"supports"},{"source_id":"S037","target_id":"S007","kind":"supports"},{"source_id":"S037","target_id":"S008","kind":"supports"},{"source_id":"S026","target_id":"S009","kind":"supports"},{"source_id":"S031","target_id":"S010","kind":"supports"},{"source_id":"S012","target_id":"S011","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S014","target_id":"S012","kind":"supports"},{"source_id":"S015","target_id":"S012","kind":"supports"},{"source_id":"S013","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S032","target_id":"S013","kind":"supports"},{"source_id":"S033","target_id":"S013","kind":"supports"},{"source_id":"S033","target_id":"S014","kind":"supports"},{"source_id":"S037","target_id":"S013","kind":"supports"},{"source_id":"S034","target_id":"S015","kind":"supports"},{"source_id":"S026","target_id":"S013","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S017","kind":"supports"},{"source_id":"S020","target_id":"S017","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S033","target_id":"S019","kind":"supports"},{"source_id":"S034","target_id":"S018","kind":"supports"},{"source_id":"S036","target_id":"S019","kind":"supports"},{"source_id":"S035","target_id":"S019","kind":"supports"},{"source_id":"S026","target_id":"S019","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S022","kind":"supports"},{"source_id":"S016","target_id":"S023","kind":"supports"},{"source_id":"S016","target_id":"S025","kind":"supports"},{"source_id":"S018","target_id":"S023","kind":"supports"},{"source_id":"S018","target_id":"S024","kind":"supports"},{"source_id":"S018","target_id":"S025","kind":"supports"},{"source_id":"S034","target_id":"S023","kind":"supports"},{"source_id":"S035","target_id":"S023","kind":"supports"},{"source_id":"S035","target_id":"S024","kind":"supports"},{"source_id":"S005","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S036","target_id":"S024","kind":"supports"},{"source_id":"S036","target_id":"S025","kind":"supports"},{"source_id":"S033","target_id":"S024","kind":"supports"},{"source_id":"S033","target_id":"S025","kind":"supports"},{"source_id":"S030","target_id":"S007","kind":"supports"},{"source_id":"S030","target_id":"S025","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S034","kind":"supports"},{"source_id":"S027","target_id":"S034","kind":"supports"},{"source_id":"S031","target_id":"S034","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S026","target_id":"S032","kind":"supports"},{"source_id":"S026","target_id":"S033","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"}]}}
{"record_id":"2405.10815v1","paper":{"title":"A Functional Model Method for Nonconvex Nonsmooth Conditional Stochastic Optimization","authors":["Andrzej Ruszczyński","Shangzhe Yang"],"arxiv_id":"2405.10815","version":"v1","journal":"SIAM Journal on Optimization","doi":"10.1137/23m1617965","abstract_url":"https://arxiv.org/abs/2405.10815v1","pdf_url":"https://arxiv.org/pdf/2405.10815v1"},"context":{"raw_tex":"The \\emph{Conditional Stochastic Optimization} (CSO) problem is formulated as follows:\n\\begin{equation}\n\\min_{\\beta \\in {\\Bb}} G(\\beta) \\triangleq \\Eb \\Big\\{g\\big(\\Eb [f(X,Y,\\beta) \\,|\\, X]\\big)\\Big\\}\n\\label{cso}\n\\end{equation}\n\nwhere $\\Bb \\subset \\Rb^{n_\\beta}$, $f: \\Rb^{n_X} \\times \\Rb^{n_Y} \\times \\Rb^{n_\\beta} \\to \\Rb^{n_f}$ , and $g: \\Rb^{n_f} \\to \\Rb$. In formula \\eqref{cso}, $X$ and $Y$ are random variables, and the symbols $\\Eb[\\Cdot]$ and $\\Eb[\\Cdot|X]$ represent the expected value and the conditional expected value, respectively.\n\n\n\\begin{equation}\n    \\label{def:F}\nF(X, \\beta) \\triangleq \\Eb[f(X, Y, \\beta) | X].\n\\end{equation}\n\nWe assume that a sufficiently rich class of functions $\\varPsi: \\Rb^{n_x} \\times \\Rb^{n_\\theta} \\to \\Rb^{n_f}$ has been established,\nsuch that for every\n$\\beta\\in {\\Bb}$ an (unknown) value of the parameters $\\bar{\\theta}(\\beta) \\in \\Rb^{n_\\theta}$ exists for which\n\\begin{equation}\n\\label{tracking-exact}\nF(X,\\beta) = \\varPsi\\big(X,\\bar{\\theta}(\\beta)\\big)  \\quad \\text{a.s.}\n\\end{equation}\n\n\\begin{equation}\nQ(\\beta, \\theta) \\triangleq \\frac{1}{2}\n\\Eb\\big[\\|F(X, \\beta) - \\varPsi(X, \\theta)\\|^2\\big],\n\\label{te:Q}\n\\end{equation}\n\nBelow we review some notation and conventions used throughout the paper. For simplicity, we ignore all the arguments of the functions,\nif they are obvious from the context. We also indicate generalized derivatives in the subscript of a function symbol,\nthat is, $F_x(\\Cdot) \\in \\partial F(\\Cdot)$.  We only use the $l_2$-norm in this paper, which is denoted as $\\|\\Cdot\\|$. A dot\nover a symbol, for example, $\\dt{F}$, represents the generalized derivative of $F(\\beta(t))$ with respect to the\ntime variable, where $\\beta(t)$ is an absolutely continuous path (usually, evident from the context).\nA function $g: \\Rb^n \\to \\Rb^m$ is \\emph{$L$-Lipschitz continuous} if $L > 0$ exists  such that $\\|g(x) - g(y)\\| \\le L \\|x-y\\|$ for all $x, y \\in \\Rb^n$. A differentiable function $g: \\Rb^n \\to \\Rb^m$ is called \\emph{$L$-Lipschitz smooth} if the vector function $\\nabla g(\\Cdot)$ is \\emph{$L$-Lipschitz continuous}. This implies that $\\|g(x) - g(y) - \\nabla g(y)^T (x - y)\\| \\le \\frac{L}{2} \\|x-y\\|^2$ for all $x, y \\in \\Xc$. A differentiable function $g: \\Xc \\to \\Rb$ is\n\\emph{strongly convex with modulus $L>0$} if $g(x) - g(y) - \\nabla g(y)^T (x - y) \\ge \\frac{L}{2} \\|x-y\\|^2$ for all $x, y \\in \\Xc$. It is \\emph{weakly convex with modulus $L$} if\n$g(x) - g(y) - \\nabla g(y)^T (x - y) \\ge -\\frac{L}{2} \\|x-y\\|^2$ for all $x, y \\in \\Xc$.\nAn $L$-Lipschitz smooth function is $L$-weakly convex.\n\n\n\\begin{description}\n\\item[(A1)] The set $\\Bb$ is convex and compact.\n\\item[(A2)] The function $g(\\Cdot)$ is continuously differentiable, its gradient is differentiable in the generalized sense, and constants $L_g$ and ${L_{\\nabla g}}$ exist,\nsuch that $\\| \\nabla g(u)\\| \\le L_g$ and $\\| \\nabla g(u) - \\nabla g(v)\\| \\le {L_{\\nabla g}}\\|u -v\\|$ for all $u,v \\in \\Rb^{n_f}$.\n\\item[(A3)] For every $x \\in \\Rb^{n_x}$ and every $y\\in \\Rb^{n_y}$ the function $f(x,y,\\Cdot)$ is differentiable in the generalized sense\nwith the generalized subdifferential $\\hat{\\partial} f(x,y,\\Cdot)$. Furthermore,  a function ${L_f}(x,y)$ and  constants $\\widebar{L}_f>0$ and $C_f>0$  exist, such that\nfor all $\\beta\\in \\Bb$ and all $(x,y)$\n\\begin{gather*}\n \\sup_{f_\\beta \\in \\hat{\\partial} f(x,y,\\beta)}\\|f_\\beta\\| \\le {L_f}(x,y),\\\\\n\\Eb [ {L_f}(X,Y)^4 ] \\le \\widebar{L}_f^4,\\\\\n\\intertext{and}\n\\Eb\\big[\\|f(X,Y,\\beta)\\|^4 \\big] \\le C_f^4.\n\\end{gather*}\n\n\n\\item[(A4)] For every $x \\in \\Rb^{n_x}$ the function $\\varPsi(x,\\Cdot)$ is differentiable in the generalized sense\nwith the generalized subdifferential $\\hat{\\partial} \\varPsi(x,\\Cdot)$. Furthermore, a function ${L_\\psi}(x)$ and  constants\n$\\widebar{L}_\\varPsi>0$ and  $C_{\\varPsi}>0$ exist, such that for all $x$\n\\begin{gather*}\n\\sup_{\\theta \\in \\Rb^{n_\\theta}}\\  \\sup_{\\varPsi_\\theta \\in \\hat{\\partial} \\varPsi(x,\\theta)}\\|\\varPsi_\\theta\\| \\le {L_\\psi}(x),\\\\\n\\Eb [ {L_\\psi}(X)^4 ] \\le \\widebar{L}_\\varPsi^4,\\\\\n\\intertext{and for some $\\theta^0$,}\n\\Eb [ \\|\\varPsi(X,\\theta^0)\\|^4 ] \\le C_\\varPsi^4.\n\\end{gather*}\n\\end{description}\n\n\nWe can compactly write the assertion of the above lemma as\\footnote{Here and elsewhere, the subdifferentiation is with respect to the parameters $\\beta$ or $\\theta$; we omit the corresponding subscripts of $\\hat{\\partial}$ to alleviate the notation.}\n\\begin{equation}\n\\label{dFbeta}\n\\hat{\\partial}_{  \\beta} F(X,\\beta) = \\Eb \\big[ \\hat{\\partial}_{  \\beta} f(X,Y,\\beta)\\,\\big|\\, X \\big] \\quad \\text{a.s.,}\n\\end{equation}\n\nwhere the conditional expectation on the right-hand side is understood as the collection of conditional expected values of all measurable selections of the multifunction\n$(X,Y) \\mapsto \\hat{\\partial}_{  \\beta} f(X,Y,\\beta)$.\n\n\nAs mentioned before, for every $\\beta \\in {\\Bb}$, a parameter value $\\bar{\\theta}(\\beta)\\in \\Rb^{n_\\theta}$ exists,\nsuch that $Q\\big(\\beta,\\bar{\\theta}(\\beta)\\big)=0$. In addition to that, we assume the following\ngrowth condition.\n\\begin{description}\n\\item[(A5)] A constant $M>0$ exists such that for all $\\beta\\in {\\Bb}$ and all $\\theta \\in \\Rb^{n_\\theta}$\n\\[\n Q(\\beta,\\theta)  \\le M \\big[\\dist\\big(0,\\hat{\\partial}_{\\theta} Q(\\beta,\\theta)\\big)\\big]^2.\n\\]\n\\end{description}\n\n\nFinally, we need an assumption about the behavior of the tracking error for very large $\\theta$.\n\n\\begin{description}\n\\item[(A6)] A constant $\\underline{R}>0$ exists such that if $\\beta\\in {\\Bb}$ and  $\\|\\theta\\|\\ge \\underline{R}$, then for all $Q_\\theta \\in\\hat{\\partial}_{\\theta} Q(\\beta,\\theta)  $,\n\\[\n\\langle \\theta, Q_\\theta \\rangle >0.\n\\]\n\\end{description}\n\n\nThe necessary optimality condition  for problem \\eqref{cso} has the form\n\\begin{equation}\n\\label{optimality}\n0 \\in \\hat{\\partial}_{ \\beta} G(\\beta) + \\Nc_{\\Bb}(\\beta),\n\\end{equation}\n\nwhere $\\hat{\\partial}_{ \\beta} G(\\Cdot)$ is the generalized subdifferential \\eqref{G-sub} and $\\Nc_{\\Bb}(\\Cdot)$ is the normal cone\nto the set $\\Bb$. We define the sets\n\\begin{align*}\n    {\\Bb}^* &= \\big\\{ \\beta\\in {\\Bb} : \\ \\text{\\eqref{optimality} is satisfied} \\big\\},\\\\\n    \\Zb^* &= \\big\\{ (\\beta,\\theta)\\in {\\Bb}^* \\times \\Rb^{n_\\theta}:  Q(\\beta,\\theta) = 0\\big\\}.\n\\end{align*}\n\nWe propose a stochastic subgradient method for finding an element of $\\Zb^*$.\n\n\nTo simplify the development and analysis\nof the method, we assume that $R \\ge \\underline{R}$ is known,\nso that $\\Zb^* \\subset \\Bb \\times \\Theta_R$,\nwhere $\\Theta_R = \\{ \\theta\\in \\Rb^{n_\\theta}: \\|\\theta \\| \\le R\\}$.\nDue to Assumption (A6), any $R\\ge \\underline{R}$ has this property.\n\nAt each iteration $k=0,1,\\dots$ of the method,\ngiven the current approximate solution $(\\beta^k,\\theta^k)\\in {\\Bb} \\times \\Theta_R$,\nwe use an observation $(X^k,Y^k)$ from the joint distribution of $(X,Y)$, to construct random directions\n$\\tilde{d}_\\beta^k$ and $\\tilde{d}_\\theta^k$, and to update the current point:\n\\begin{equation}\n\\label{method}\n\\begin{aligned}\n\\beta^{k+1} &= \\varPi_{\\Bb}(\\beta^k + \\tau_k \\tilde{d}_\\beta^k),\\\\\n\\theta^{k+1} &= \\varPi_{\\Theta_R}(\\theta^k + \\tau_k \\tilde{d}_\\theta^k).\n\\end{aligned}\n\\end{equation}\n\nHere, $\\tau_k>0$ is a stepsize of the method, $\\varPi_{\\Bb}(\\Cdot)$  denotes the orthogonal projection on $\\Bb$,\nand $\\varPi_{\\Theta_R}(\\Cdot)$ is the orthogonal projection on the ball $\\Theta_R$.\n\nTo construct the directions $\\tilde{d}_\\beta^k$ and $\\tilde{d}_\\theta^k$,\nwe use the values and  generalized gradients of the functions $f(X^k,Y^k,\\Cdot)$ and $\\varPsi(X^k,\\Cdot)$, which\ncan be observed at the current point $(\\beta^k,\\theta^k)$. We also use the derivatives of the function $g(\\Cdot)$ at the\npoints $\\varPsi(X^k,\\theta^k)$, which can be observed as well.\n\nIn our presentation of the method and its analysis, we denote by $(\\varOmega,\\Fc,P)$ the algorithmic probability space on which\nthe random sequences generated by the method are defined, and by\n$\\Fc_k$ the $\\sigma$-subalgebra defined by the history $(\\beta^0,\\theta^0,X^0,Y^0,\\dots,X^{k-1},Y^{k-1},\\beta^k,\\theta^k)$.\nWe assume that at iteration $k=0,1,\\dots$ we observe the pair $(X^k,Y^k)$, independently of $\\Fc_k$, and we calculate the following quantitites:\n\\begin{gather*}\nf^k  = f(X^k,Y^k,\\beta^k),\\\\\nf_\\beta^k  \\in \\hat{\\partial}_{ \\beta} f(X^k,Y^k,\\beta^k),\\\\\n\\varPsi^k  = \\varPsi(X^k,\\theta^k),\\\\\n\\varPsi_\\theta^k  \\in \\hat{\\partial}_{ \\theta} \\varPsi(X^k,\\theta^k),\\\\\n\\nabla g^k = \\nabla g(\\varPsi^k).\n%\\\\\n%{  \\nabla^2 g^k \\in \\hat{\\partial} \\nabla g(\\varPsi^k).}\n\\end{gather*}\n\nThen we calculate the directions in \\eqref{method}:\n\\begin{subequations}\n\\label{dzk}\n\\begin{align}\n    \\tilde{d}_\\beta^k &= - f_\\beta^k \\,\\nabla g^k,  \\label{dbetak}\\\\\n    \\tilde{d}_\\theta^k &=\\gamma \\,\\varPsi_\\theta^k (f^k - \\varPsi^k). \\label{dthetak}\n\\end{align}\n\n\\end{subequations}\nHere, $\\gamma>0$ is a parameter of the method.\n\n\n\\begin{description}\n\\item[(A7)]\nThe stepsizes $\\tau_k$ are $\\Fc_k$-measurable, and such that\\vspace{0.5ex}\n\\begin{tightlist}{iii}\n\\item  $\\tau_k >0, \\quad k=0,1,2,\\dots$;\n\\item   $ \\displaystyle{\\sum_{k=0}^\\infty} \\tau_k = \\infty \\quad \\text{a.s.}$;\n\\item $\\Eb \\displaystyle{\\sum_{k=0}^\\infty} \\tau_k^2 < \\infty$.\n\\end{tightlist}\n\\end{description}\n\n\nConsider the multifunction\n$\\varGamma: \\Bb \\times \\Theta_R \\rightrightarrows \\Rb^{n_f}$ defined as follows:\n\\[\n\\varGamma(\\beta,\\theta) = \\Eb\n\\left\\{\n\\begin{bmatrix}\n -\\hat{\\partial}_\\beta F(X,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big) \\\\\n\\gamma \\,\\hat{\\partial}_\\theta \\varPsi(X,\\theta)\\big(F(X,\\beta) -\\varPsi(X,\\theta)\\big)\n\\end{bmatrix}\n\\right\\}.\n\\]\n\nWe need to establish the properties of the multifunction $\\varGamma(\\Cdot,\\Cdot)$ that are relevant for the convergence analysis.\n\n\nDefine the average directions,\n\\begin{equation}\n\\label{def:avg_dir}\nd^k = \\Eb [\\tilde{d}^k \\,| \\, \\Fc_k],\\quad k=0,1,\\dots,\n\\end{equation}\n\nand let\n\\begin{equation}\n\\notag\n%\\label{def:e}\ne^k = \\tilde{d^k} - d^k.\n\\end{equation}\n\nFor brevity, denote $z^k = (\\beta^k,\\theta^k) $, $k=0,1,\\dots$, and $\\Zb=\\Bb \\times\\Theta_R$.\nWe consider a particular trajectory $z^k(\\omega) = (\\beta^k(\\omega),\\theta^k(\\omega)) $, $k=0,1,\\dots$; in our notation, we omit\nthe elementary event $\\omega\\in \\Omega$, which may be arbitrary, except for a zero measure set.\n\nFirst, we construct continuous-time trajectories by linear interpolation. We introduce the accumulated stepsizes\n\\[\nt_k = \\sum_{j=0}^{k-1}\\tau_j,\\quad k=0,1,\\dots,\n\\]\n\nand we construct the interpolated trajectory\n\\[\nZ_0(t) = z^k + \\frac{t-t_{k}}{\\tau_k}(z^{k+1}-z^k),\\quad t_{k}\\le t \\le t_{k+1},\\quad k=0,1,\\dots.\n\\]\n\nThen, for an increasing sequence of positive numbers $\\{s_m\\}$ diverging to $\\infty$, we define the shifted trajectories\n\\begin{equation}\n\\label{Z-shifted}\nZ_m(t)=Z_0(t+s_m), \\quad m=0,1,\\dots,\\quad t \\ge 0.\n\\end{equation}\n\nIn particular, if we set $s_m = t_m$, we obtain $Z_m(0) = z^m$.\n\n\nTo this end, we construct a specially tailored Lyapunov function with parameters $\\alpha > 0$ and $\\lambda \\ge 0$:\n\\begin{equation}\n\\label{Lyapunov}\n    w(\\beta,\\theta) \\triangleq G(\\beta) + \\alpha \\Delta^\\lambda(\\beta, \\theta),\n\\end{equation}\n\nwhere $G(\\beta)$ is the CSO loss function in \\eqref{cso} and\n\\begin{equation}\n\\Delta^{\\lambda}(\\beta,\\theta) \\triangleq \\Eb\\Big[g(F) - g(\\varPsi) - \\langle \\nabla g(\\varPsi), F - \\varPsi \\rangle + \\frac{\\lambda}{2} \\|F-\\varPsi\\|^2\\Big].\n\\label{te:g}\n\\end{equation}\n\nIn order to prove the convergence of the discrete-time algorithm, we need an additional technical assumption (known as the \\emph{Sard condition}).\n\\begin{description}\n\\item[(A8)] The set $\\{G(\\beta): \\beta\\in {\\Bb}^*\\}$ does not contain an interval of nonzero length.\n\\end{description}\n\n\n\\begin{definition}\n\\label{d:Norkin}\nA function $f:\\Rb^n\\to\\Rb$ is  \\emph{differentiable in a generalized sense at a point} $x\\in \\Rb^n$,\nif an open set $U\\subset \\Rb^n$ containing $x$,\nand a nonempty, convex, compact valued, and upper semicontinuous multifunction\n$\\hat{\\partial}f: U \\rightrightarrows \\Rb^n$ exist, such that for all $y\\in U$ and all $g \\in \\hat{\\partial}f(y)$ the following equation is true:\n\\[\nf(y) = f(x) + \\langle g(y), y-x \\rangle + o(x,y,g),\n\\]\nwith\n\\[\n\\lim_{y\\to x} \\sup_{g\\in \\hat{\\partial}f(y)} \\frac{o(x,y,g)}{\\|y-x\\|}=0.\n\\]\nThe set $\\hat{\\partial}f(y)$ is  the \\emph{generalized subdifferential} of $f$ at $y$.\nIf a function is differentiable in a generalized sense at every  $x \\in \\Rb^n$ with the same generalized subdifferential mapping $\\hat{\\partial}f:\\Rb^n\\rightrightarrows \\Rb^n$, we call it \\emph{differentiable in a generalized sense}.\n\nA function $f:\\Rb^n\\to\\Rb^m$ is  differentiable in a generalized sense, if each of its component functions, $f_i:\\Rb^n\\to\\Rb$, $i=1,\\dots,m$, has this property.\n\\end{definition}","units":[{"start":7231,"end":7458,"start_line":203,"end_line":207,"raw_tex":"The \\emph{Conditional Stochastic Optimization} (CSO) problem is formulated as follows:\n\\begin{equation}\n\\min_{\\beta \\in {\\Bb}} G(\\beta) \\triangleq \\Eb \\Big\\{g\\big(\\Eb [f(X,Y,\\beta) \\,|\\, X]\\big)\\Big\\}\n\\label{cso}\n\\end{equation}","sha256_utf8":"c0bfb8638afdf480c204ec0a3607ae073b545b67093cabe8ce4dbf3a348eec1c"},{"start":7459,"end":7776,"start_line":208,"end_line":208,"raw_tex":"where $\\Bb \\subset \\Rb^{n_\\beta}$, $f: \\Rb^{n_X} \\times \\Rb^{n_Y} \\times \\Rb^{n_\\beta} \\to \\Rb^{n_f}$ , and $g: \\Rb^{n_f} \\to \\Rb$. In formula \\eqref{cso}, $X$ and $Y$ are random variables, and the symbols $\\Eb[\\Cdot]$ and $\\Eb[\\Cdot|X]$ represent the expected value and the conditional expected value, respectively.\n","sha256_utf8":"8714a44553757ce9c75010d6483d4c22bf8d8a3732b1cc5e6ae7ed5d51c013e1"},{"start":18317,"end":18414,"start_line":358,"end_line":361,"raw_tex":"\\begin{equation}\n    \\label{def:F}\nF(X, \\beta) \\triangleq \\Eb[f(X, Y, \\beta) | X].\n\\end{equation}","sha256_utf8":"decfbf1d675e65708be8df2453f9746e83e77cf97f822eb3bf96f91ee9afc3a7"},{"start":18415,"end":18805,"start_line":362,"end_line":368,"raw_tex":"We assume that a sufficiently rich class of functions $\\varPsi: \\Rb^{n_x} \\times \\Rb^{n_\\theta} \\to \\Rb^{n_f}$ has been established,\nsuch that for every\n$\\beta\\in {\\Bb}$ an (unknown) value of the parameters $\\bar{\\theta}(\\beta) \\in \\Rb^{n_\\theta}$ exists for which\n\\begin{equation}\n\\label{tracking-exact}\nF(X,\\beta) = \\varPsi\\big(X,\\bar{\\theta}(\\beta)\\big)  \\quad \\text{a.s.}\n\\end{equation}","sha256_utf8":"8bf63ee8d77daea463adfbfdb391b0986bcc13a81c82d4d5b9f86d94d2ecec55"},{"start":20203,"end":20340,"start_line":390,"end_line":394,"raw_tex":"\\begin{equation}\nQ(\\beta, \\theta) \\triangleq \\frac{1}{2}\n\\Eb\\big[\\|F(X, \\beta) - \\varPsi(X, \\theta)\\|^2\\big],\n\\label{te:Q}\n\\end{equation}","sha256_utf8":"2b2b8c8ec44c914cf392a385d19060cdfa799021637ff14a025c4b4473c3cd24"},{"start":23456,"end":24847,"start_line":440,"end_line":448,"raw_tex":"Below we review some notation and conventions used throughout the paper. For simplicity, we ignore all the arguments of the functions,\nif they are obvious from the context. We also indicate generalized derivatives in the subscript of a function symbol,\nthat is, $F_x(\\Cdot) \\in \\partial F(\\Cdot)$.  We only use the $l_2$-norm in this paper, which is denoted as $\\|\\Cdot\\|$. A dot\nover a symbol, for example, $\\dt{F}$, represents the generalized derivative of $F(\\beta(t))$ with respect to the\ntime variable, where $\\beta(t)$ is an absolutely continuous path (usually, evident from the context).\nA function $g: \\Rb^n \\to \\Rb^m$ is \\emph{$L$-Lipschitz continuous} if $L > 0$ exists  such that $\\|g(x) - g(y)\\| \\le L \\|x-y\\|$ for all $x, y \\in \\Rb^n$. A differentiable function $g: \\Rb^n \\to \\Rb^m$ is called \\emph{$L$-Lipschitz smooth} if the vector function $\\nabla g(\\Cdot)$ is \\emph{$L$-Lipschitz continuous}. This implies that $\\|g(x) - g(y) - \\nabla g(y)^T (x - y)\\| \\le \\frac{L}{2} \\|x-y\\|^2$ for all $x, y \\in \\Xc$. A differentiable function $g: \\Xc \\to \\Rb$ is\n\\emph{strongly convex with modulus $L>0$} if $g(x) - g(y) - \\nabla g(y)^T (x - y) \\ge \\frac{L}{2} \\|x-y\\|^2$ for all $x, y \\in \\Xc$. It is \\emph{weakly convex with modulus $L$} if\n$g(x) - g(y) - \\nabla g(y)^T (x - y) \\ge -\\frac{L}{2} \\|x-y\\|^2$ for all $x, y \\in \\Xc$.\nAn $L$-Lipschitz smooth function is $L$-weakly convex.\n","sha256_utf8":"e21205f924d6fb8fd9802fe665752f5c670cd29d951fdb7dfecf6d743f78f9cc"},{"start":24990,"end":26553,"start_line":455,"end_line":479,"raw_tex":"\\begin{description}\n\\item[(A1)] The set $\\Bb$ is convex and compact.\n\\item[(A2)] The function $g(\\Cdot)$ is continuously differentiable, its gradient is differentiable in the generalized sense, and constants $L_g$ and ${L_{\\nabla g}}$ exist,\nsuch that $\\| \\nabla g(u)\\| \\le L_g$ and $\\| \\nabla g(u) - \\nabla g(v)\\| \\le {L_{\\nabla g}}\\|u -v\\|$ for all $u,v \\in \\Rb^{n_f}$.\n\\item[(A3)] For every $x \\in \\Rb^{n_x}$ and every $y\\in \\Rb^{n_y}$ the function $f(x,y,\\Cdot)$ is differentiable in the generalized sense\nwith the generalized subdifferential $\\hat{\\partial} f(x,y,\\Cdot)$. Furthermore,  a function ${L_f}(x,y)$ and  constants $\\widebar{L}_f>0$ and $C_f>0$  exist, such that\nfor all $\\beta\\in \\Bb$ and all $(x,y)$\n\\begin{gather*}\n \\sup_{f_\\beta \\in \\hat{\\partial} f(x,y,\\beta)}\\|f_\\beta\\| \\le {L_f}(x,y),\\\\\n\\Eb [ {L_f}(X,Y)^4 ] \\le \\widebar{L}_f^4,\\\\\n\\intertext{and}\n\\Eb\\big[\\|f(X,Y,\\beta)\\|^4 \\big] \\le C_f^4.\n\\end{gather*}\n\n\n\\item[(A4)] For every $x \\in \\Rb^{n_x}$ the function $\\varPsi(x,\\Cdot)$ is differentiable in the generalized sense\nwith the generalized subdifferential $\\hat{\\partial} \\varPsi(x,\\Cdot)$. Furthermore, a function ${L_\\psi}(x)$ and  constants\n$\\widebar{L}_\\varPsi>0$ and  $C_{\\varPsi}>0$ exist, such that for all $x$\n\\begin{gather*}\n\\sup_{\\theta \\in \\Rb^{n_\\theta}}\\  \\sup_{\\varPsi_\\theta \\in \\hat{\\partial} \\varPsi(x,\\theta)}\\|\\varPsi_\\theta\\| \\le {L_\\psi}(x),\\\\\n\\Eb [ {L_\\psi}(X)^4 ] \\le \\widebar{L}_\\varPsi^4,\\\\\n\\intertext{and for some $\\theta^0$,}\n\\Eb [ \\|\\varPsi(X,\\theta^0)\\|^4 ] \\le C_\\varPsi^4.\n\\end{gather*}\n\\end{description}\n","sha256_utf8":"17f5ec9105771cfc22308728c651ee19caba7c45bc6aaa6dae34cc87c27f97ec"},{"start":28030,"end":28449,"start_line":510,"end_line":514,"raw_tex":"We can compactly write the assertion of the above lemma as\\footnote{Here and elsewhere, the subdifferentiation is with respect to the parameters $\\beta$ or $\\theta$; we omit the corresponding subscripts of $\\hat{\\partial}$ to alleviate the notation.}\n\\begin{equation}\n\\label{dFbeta}\n\\hat{\\partial}_{  \\beta} F(X,\\beta) = \\Eb \\big[ \\hat{\\partial}_{  \\beta} f(X,Y,\\beta)\\,\\big|\\, X \\big] \\quad \\text{a.s.,}\n\\end{equation}","sha256_utf8":"cb7a190b64848ac077d35c23a826676bf5be4c62e0e268a89094acfb20e47e58"},{"start":28450,"end":28675,"start_line":515,"end_line":516,"raw_tex":"where the conditional expectation on the right-hand side is understood as the collection of conditional expected values of all measurable selections of the multifunction\n$(X,Y) \\mapsto \\hat{\\partial}_{  \\beta} f(X,Y,\\beta)$.\n","sha256_utf8":"23349834b987b5365ff506775553182ab0602b45ac3a81d94c4954197b13e21f"},{"start":31468,"end":31944,"start_line":582,"end_line":590,"raw_tex":"As mentioned before, for every $\\beta \\in {\\Bb}$, a parameter value $\\bar{\\theta}(\\beta)\\in \\Rb^{n_\\theta}$ exists,\nsuch that $Q\\big(\\beta,\\bar{\\theta}(\\beta)\\big)=0$. In addition to that, we assume the following\ngrowth condition.\n\\begin{description}\n\\item[(A5)] A constant $M>0$ exists such that for all $\\beta\\in {\\Bb}$ and all $\\theta \\in \\Rb^{n_\\theta}$\n\\[\n Q(\\beta,\\theta)  \\le M \\big[\\dist\\big(0,\\hat{\\partial}_{\\theta} Q(\\beta,\\theta)\\big)\\big]^2.\n\\]\n\\end{description}\n","sha256_utf8":"09de087378369525005a9653f4e60005cd7d5fe1b139b0c2ce9ddc0e337c5363"},{"start":35845,"end":36208,"start_line":654,"end_line":661,"raw_tex":"Finally, we need an assumption about the behavior of the tracking error for very large $\\theta$.\n\n\\begin{description}\n\\item[(A6)] A constant $\\underline{R}>0$ exists such that if $\\beta\\in {\\Bb}$ and  $\\|\\theta\\|\\ge \\underline{R}$, then for all $Q_\\theta \\in\\hat{\\partial}_{\\theta} Q(\\beta,\\theta)  $,\n\\[\n\\langle \\theta, Q_\\theta \\rangle >0.\n\\]\n\\end{description}\n","sha256_utf8":"4380819d1cc06dde8f6ba6c6e6d8caab5e8ee9ab8f16e66761d995cefe1b7d5a"},{"start":38690,"end":38872,"start_line":714,"end_line":718,"raw_tex":"The necessary optimality condition  for problem \\eqref{cso} has the form\n\\begin{equation}\n\\label{optimality}\n0 \\in \\hat{\\partial}_{ \\beta} G(\\beta) + \\Nc_{\\Bb}(\\beta),\n\\end{equation}","sha256_utf8":"fd5cf719808ecd2fa4799522328b4e3d545bd2386eb2f0db9d68f8194a4b6556"},{"start":38873,"end":39256,"start_line":719,"end_line":724,"raw_tex":"where $\\hat{\\partial}_{ \\beta} G(\\Cdot)$ is the generalized subdifferential \\eqref{G-sub} and $\\Nc_{\\Bb}(\\Cdot)$ is the normal cone\nto the set $\\Bb$. We define the sets\n\\begin{align*}\n    {\\Bb}^* &= \\big\\{ \\beta\\in {\\Bb} : \\ \\text{\\eqref{optimality} is satisfied} \\big\\},\\\\\n    \\Zb^* &= \\big\\{ (\\beta,\\theta)\\in {\\Bb}^* \\times \\Rb^{n_\\theta}:  Q(\\beta,\\theta) = 0\\big\\}.\n\\end{align*}","sha256_utf8":"81ba38b512633a0e766eb6595468c5d5376abf4a9a5965625e9ab3b06346e769"},{"start":39257,"end":39335,"start_line":725,"end_line":725,"raw_tex":"We propose a stochastic subgradient method for finding an element of $\\Zb^*$.\n","sha256_utf8":"308109fc8fdc5f7a08da6887fed11e9be2a65ff76ab9307b4970778414db1f61"},{"start":39476,"end":40295,"start_line":731,"end_line":747,"raw_tex":"To simplify the development and analysis\nof the method, we assume that $R \\ge \\underline{R}$ is known,\nso that $\\Zb^* \\subset \\Bb \\times \\Theta_R$,\nwhere $\\Theta_R = \\{ \\theta\\in \\Rb^{n_\\theta}: \\|\\theta \\| \\le R\\}$.\nDue to Assumption (A6), any $R\\ge \\underline{R}$ has this property.\n\nAt each iteration $k=0,1,\\dots$ of the method,\ngiven the current approximate solution $(\\beta^k,\\theta^k)\\in {\\Bb} \\times \\Theta_R$,\nwe use an observation $(X^k,Y^k)$ from the joint distribution of $(X,Y)$, to construct random directions\n$\\tilde{d}_\\beta^k$ and $\\tilde{d}_\\theta^k$, and to update the current point:\n\\begin{equation}\n\\label{method}\n\\begin{aligned}\n\\beta^{k+1} &= \\varPi_{\\Bb}(\\beta^k + \\tau_k \\tilde{d}_\\beta^k),\\\\\n\\theta^{k+1} &= \\varPi_{\\Theta_R}(\\theta^k + \\tau_k \\tilde{d}_\\theta^k).\n\\end{aligned}\n\\end{equation}","sha256_utf8":"4f17cf09d3f59a3aa86ed9e26637d903b190e78952af1e9b5f436f05216e83fe"},{"start":40296,"end":41653,"start_line":748,"end_line":768,"raw_tex":"Here, $\\tau_k>0$ is a stepsize of the method, $\\varPi_{\\Bb}(\\Cdot)$  denotes the orthogonal projection on $\\Bb$,\nand $\\varPi_{\\Theta_R}(\\Cdot)$ is the orthogonal projection on the ball $\\Theta_R$.\n\nTo construct the directions $\\tilde{d}_\\beta^k$ and $\\tilde{d}_\\theta^k$,\nwe use the values and  generalized gradients of the functions $f(X^k,Y^k,\\Cdot)$ and $\\varPsi(X^k,\\Cdot)$, which\ncan be observed at the current point $(\\beta^k,\\theta^k)$. We also use the derivatives of the function $g(\\Cdot)$ at the\npoints $\\varPsi(X^k,\\theta^k)$, which can be observed as well.\n\nIn our presentation of the method and its analysis, we denote by $(\\varOmega,\\Fc,P)$ the algorithmic probability space on which\nthe random sequences generated by the method are defined, and by\n$\\Fc_k$ the $\\sigma$-subalgebra defined by the history $(\\beta^0,\\theta^0,X^0,Y^0,\\dots,X^{k-1},Y^{k-1},\\beta^k,\\theta^k)$.\nWe assume that at iteration $k=0,1,\\dots$ we observe the pair $(X^k,Y^k)$, independently of $\\Fc_k$, and we calculate the following quantitites:\n\\begin{gather*}\nf^k  = f(X^k,Y^k,\\beta^k),\\\\\nf_\\beta^k  \\in \\hat{\\partial}_{ \\beta} f(X^k,Y^k,\\beta^k),\\\\\n\\varPsi^k  = \\varPsi(X^k,\\theta^k),\\\\\n\\varPsi_\\theta^k  \\in \\hat{\\partial}_{ \\theta} \\varPsi(X^k,\\theta^k),\\\\\n\\nabla g^k = \\nabla g(\\varPsi^k).\n%\\\\\n%{  \\nabla^2 g^k \\in \\hat{\\partial} \\nabla g(\\varPsi^k).}\n\\end{gather*}","sha256_utf8":"7ef370e8837d0d4788256fdbb90bbf5ba0087cfa8631cda395bba71348f40216"},{"start":41654,"end":41919,"start_line":769,"end_line":775,"raw_tex":"Then we calculate the directions in \\eqref{method}:\n\\begin{subequations}\n\\label{dzk}\n\\begin{align}\n    \\tilde{d}_\\beta^k &= - f_\\beta^k \\,\\nabla g^k,  \\label{dbetak}\\\\\n    \\tilde{d}_\\theta^k &=\\gamma \\,\\varPsi_\\theta^k (f^k - \\varPsi^k). \\label{dthetak}\n\\end{align}","sha256_utf8":"d1262598b2d3a80cba89687b90efa685c767bc2b25090240844335147d294e11"},{"start":41920,"end":41986,"start_line":776,"end_line":777,"raw_tex":"\\end{subequations}\nHere, $\\gamma>0$ is a parameter of the method.\n","sha256_utf8":"f63b25a16ac64b7956ef6f7381900297db37c7d239bf51b11092f956f8ccc0f9"},{"start":42950,"end":43298,"start_line":794,"end_line":802,"raw_tex":"\\begin{description}\n\\item[(A7)]\nThe stepsizes $\\tau_k$ are $\\Fc_k$-measurable, and such that\\vspace{0.5ex}\n\\begin{tightlist}{iii}\n\\item  $\\tau_k >0, \\quad k=0,1,2,\\dots$;\n\\item   $ \\displaystyle{\\sum_{k=0}^\\infty} \\tau_k = \\infty \\quad \\text{a.s.}$;\n\\item $\\Eb \\displaystyle{\\sum_{k=0}^\\infty} \\tau_k^2 < \\infty$.\n\\end{tightlist}\n\\end{description}\n","sha256_utf8":"d737359b2e0c24650e349211bd82a3653679ab19f17729225d78b669a1b8dc84"},{"start":43342,"end":43694,"start_line":806,"end_line":816,"raw_tex":"Consider the multifunction\n$\\varGamma: \\Bb \\times \\Theta_R \\rightrightarrows \\Rb^{n_f}$ defined as follows:\n\\[\n\\varGamma(\\beta,\\theta) = \\Eb\n\\left\\{\n\\begin{bmatrix}\n -\\hat{\\partial}_\\beta F(X,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big) \\\\\n\\gamma \\,\\hat{\\partial}_\\theta \\varPsi(X,\\theta)\\big(F(X,\\beta) -\\varPsi(X,\\theta)\\big)\n\\end{bmatrix}\n\\right\\}.\n\\]","sha256_utf8":"c92637d97843bfb1e762ebf53e2da270fca92c6f57d4ffdfcc6606ed96c9c250"},{"start":43695,"end":43825,"start_line":817,"end_line":817,"raw_tex":"We need to establish the properties of the multifunction $\\varGamma(\\Cdot,\\Cdot)$ that are relevant for the convergence analysis.\n","sha256_utf8":"ce23307c3c1a0817d153251cc4357c99c0e8e25ed1afaec05ebb84d39e5c7bea"},{"start":48077,"end":48215,"start_line":902,"end_line":906,"raw_tex":"Define the average directions,\n\\begin{equation}\n\\label{def:avg_dir}\nd^k = \\Eb [\\tilde{d}^k \\,| \\, \\Fc_k],\\quad k=0,1,\\dots,\n\\end{equation}","sha256_utf8":"b150d02fdea2c47bb7f3b9bbb58a96adea9c4c13be9393b2f65a530d65eedbd6"},{"start":48216,"end":48302,"start_line":907,"end_line":912,"raw_tex":"and let\n\\begin{equation}\n\\notag\n%\\label{def:e}\ne^k = \\tilde{d^k} - d^k.\n\\end{equation}","sha256_utf8":"fdfbcd7875cedc77863db0007e38bd9f965bf50f4fb454cc36deeaa67f1d4e4b"},{"start":52455,"end":52941,"start_line":988,"end_line":995,"raw_tex":"For brevity, denote $z^k = (\\beta^k,\\theta^k) $, $k=0,1,\\dots$, and $\\Zb=\\Bb \\times\\Theta_R$.\nWe consider a particular trajectory $z^k(\\omega) = (\\beta^k(\\omega),\\theta^k(\\omega)) $, $k=0,1,\\dots$; in our notation, we omit\nthe elementary event $\\omega\\in \\Omega$, which may be arbitrary, except for a zero measure set.\n\nFirst, we construct continuous-time trajectories by linear interpolation. We introduce the accumulated stepsizes\n\\[\nt_k = \\sum_{j=0}^{k-1}\\tau_j,\\quad k=0,1,\\dots,\n\\]","sha256_utf8":"da9c890c3021086ca70e4bd2195022783dfe58b5dfbb68766981d67c154b9389"},{"start":52942,"end":53091,"start_line":996,"end_line":999,"raw_tex":"and we construct the interpolated trajectory\n\\[\nZ_0(t) = z^k + \\frac{t-t_{k}}{\\tau_k}(z^{k+1}-z^k),\\quad t_{k}\\le t \\le t_{k+1},\\quad k=0,1,\\dots.\n\\]","sha256_utf8":"4e13e06543678104266a125646401091aee4ff0f9af7092ab9a0fbbbaecb34f7"},{"start":53092,"end":53314,"start_line":1000,"end_line":1004,"raw_tex":"Then, for an increasing sequence of positive numbers $\\{s_m\\}$ diverging to $\\infty$, we define the shifted trajectories\n\\begin{equation}\n\\label{Z-shifted}\nZ_m(t)=Z_0(t+s_m), \\quad m=0,1,\\dots,\\quad t \\ge 0.\n\\end{equation}","sha256_utf8":"6d55c0b251c564edf27c1126c6a95eb8aaf56dc072465754281de286c23e2d9c"},{"start":53315,"end":53379,"start_line":1005,"end_line":1005,"raw_tex":"In particular, if we set $s_m = t_m$, we obtain $Z_m(0) = z^m$.\n","sha256_utf8":"94e93a4bc82b29a43b49d545d3c5287e1ffd3c8aaf0f745f7b927542edb93a66"},{"start":54601,"end":54844,"start_line":1034,"end_line":1038,"raw_tex":"To this end, we construct a specially tailored Lyapunov function with parameters $\\alpha > 0$ and $\\lambda \\ge 0$:\n\\begin{equation}\n\\label{Lyapunov}\n    w(\\beta,\\theta) \\triangleq G(\\beta) + \\alpha \\Delta^\\lambda(\\beta, \\theta),\n\\end{equation}","sha256_utf8":"a71d7d381ee559c98464993071ed3239fb8ff4710729541523c8d3ebfd38a78b"},{"start":54845,"end":55109,"start_line":1039,"end_line":1043,"raw_tex":"where $G(\\beta)$ is the CSO loss function in \\eqref{cso} and\n\\begin{equation}\n\\Delta^{\\lambda}(\\beta,\\theta) \\triangleq \\Eb\\Big[g(F) - g(\\varPsi) - \\langle \\nabla g(\\varPsi), F - \\varPsi \\rangle + \\frac{\\lambda}{2} \\|F-\\varPsi\\|^2\\Big].\n\\label{te:g}\n\\end{equation}","sha256_utf8":"8ecc52bdf5eda2bf71a9e2e4285c348c4d9eff84a4d6122c9da637c7296f09e5"},{"start":72262,"end":72548,"start_line":1349,"end_line":1352,"raw_tex":"In order to prove the convergence of the discrete-time algorithm, we need an additional technical assumption (known as the \\emph{Sard condition}).\n\\begin{description}\n\\item[(A8)] The set $\\{G(\\beta): \\beta\\in {\\Bb}^*\\}$ does not contain an interval of nonzero length.\n\\end{description}\n","sha256_utf8":"ddfbfeba0827d66fc34c8385a2d701fc099043e80e94a8ceebd756ea38a39ec5"},{"start":102338,"end":103389,"start_line":1926,"end_line":1943,"raw_tex":"\\begin{definition}\n\\label{d:Norkin}\nA function $f:\\Rb^n\\to\\Rb$ is  \\emph{differentiable in a generalized sense at a point} $x\\in \\Rb^n$,\nif an open set $U\\subset \\Rb^n$ containing $x$,\nand a nonempty, convex, compact valued, and upper semicontinuous multifunction\n$\\hat{\\partial}f: U \\rightrightarrows \\Rb^n$ exist, such that for all $y\\in U$ and all $g \\in \\hat{\\partial}f(y)$ the following equation is true:\n\\[\nf(y) = f(x) + \\langle g(y), y-x \\rangle + o(x,y,g),\n\\]\nwith\n\\[\n\\lim_{y\\to x} \\sup_{g\\in \\hat{\\partial}f(y)} \\frac{o(x,y,g)}{\\|y-x\\|}=0.\n\\]\nThe set $\\hat{\\partial}f(y)$ is  the \\emph{generalized subdifferential} of $f$ at $y$.\nIf a function is differentiable in a generalized sense at every  $x \\in \\Rb^n$ with the same generalized subdifferential mapping $\\hat{\\partial}f:\\Rb^n\\rightrightarrows \\Rb^n$, we call it \\emph{differentiable in a generalized sense}.\n\nA function $f:\\Rb^n\\to\\Rb^m$ is  differentiable in a generalized sense, if each of its component functions, $f_i:\\Rb^n\\to\\Rb$, $i=1,\\dots,m$, has this property.\n\\end{definition}","sha256_utf8":"046f405c3b8919d5bda34c48a3f886e66730139ce2c71b11efc911b9ce3f8758"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\n    \\label{t:full-convergence}\n    Assume conditions {\\rm (A1)--(A8)} and \\eqref{negative-der}. Then, except for a zero measure set $\\varOmega_0\\subset \\varOmega$,  the sequence $\\big\\{w(z^k(\\omega))\\big\\}_{k\\in \\Nb}$ is convergent, and every accumulation point of the sequence\n$\\big\\{z^k(\\omega)\\big\\}_{k\\in \\Nb}$ is an element of $\\Zb^*$.\n\\end{theorem}","units":[{"start":72548,"end":72918,"start_line":1353,"end_line":1357,"raw_tex":"\\begin{theorem}\n    \\label{t:full-convergence}\n    Assume conditions {\\rm (A1)--(A8)} and \\eqref{negative-der}. Then, except for a zero measure set $\\varOmega_0\\subset \\varOmega$,  the sequence $\\big\\{w(z^k(\\omega))\\big\\}_{k\\in \\Nb}$ is convergent, and every accumulation point of the sequence\n$\\big\\{z^k(\\omega)\\big\\}_{k\\in \\Nb}$ is an element of $\\Zb^*$.\n\\end{theorem}","sha256_utf8":"9991c21c97ac42485dc6daf9c69663c552fb593644eebbb6d260229ecff35506"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\n\\label{l:conditional_subgradient}\nSuppose the function $f(X,Y,\\Cdot)$ is differentiable in the generalized sense and its subgradients are bounded\nby an integrable function in any bounded neighborhood of any point $\\beta$. Then for almost all $X$ the function\n$\\beta \\mapsto \\Eb\\big[ f(X,Y,\\beta) \\big| X\\big]$ is differentiable in the generalized sense, and\nfor every measurable function $s:\\Rb^{n_x}\\times \\Rb^{n_y} \\to \\Rb^{n_f}$ such that $s(X,Y)  \\in \\hat{\\partial}_\\beta f(X,Y,\\beta)$ a.s., we have\n\\[\n\\Eb\\big[s(X,Y) \\big| X\\big] \\in \\hat{\\partial} \\Eb\\big[ f(X,Y,\\beta) \\big| X\\big] \\quad  a.s.\n\\]\n\\end{lemma}\n\n\\begin{proof}\nThe probability measure $P$ on $\\Rb^{n_x}\\times \\Rb^{n_y}$\ncan be disintegrated into a marginal measure $P_X$ on $\\Rb^{n_x}$ and a kernel $T:\\Rb^{n_x} \\to \\Pc(\\Rb^{n_y})$,\n where $\\Pc(\\Rb^{n_y})$ is the space of probability measures\n on $\\Rb^{n_y}$. Then\n\\[\n\\Eb\\big[ f(X,Y,\\beta) \\big| X=x\\big] = \\int f(x,y,\\beta)\\;T(\\D y|x).\n\\]\nFor every bounded neighborhood $K$ of $\\beta$, the Lipschitz constant $L_K(x,y)$ of the function $f(x,y,\\Cdot)$\nis integrable, and thus, for almost\nall $x$, its conditional expectation\n$\\int L_K(x,y)\\,T(dy|x)$ exists as well.\nThe assertion then follows from Theorem \\ref{t:expected-generalized}.\n\\end{proof}\n\nFurthermore, by Theorem \\ref{t:composition} and Assumption (A2), the composition\n$g( F(x,\\Cdot))$ is differentiable in the generalized sense as well, and\n\\[\n\\hat{\\partial}_{  \\beta} [g\\circ F](x,\\beta)  = \\big\\{    F_\\beta(x,\\beta) \\nabla g(F(x,\\beta)) : F_\\beta(x,\\beta) \\in \\hat{\\partial}_{  \\beta} F (x,\\beta)\\big\\}\n\\]\nis its generalized subdifferential (we adopt the convention that the generalized Jacobian $F_\\beta$ is an $n_\\beta \\times n_f$ matrix).\nThanks to the integrability condition in (A3) and Theorem \\ref{t:expected-generalized}, the function $G(\\Cdot)$ in \\eqref{cso}\nis differentiable in the generalized sense as well, and a realization of its generalized subdifferential is the multifunction\n\\begin{equation}\n    \\label{G-sub}\n\\begin{aligned}\n\\hat{\\partial}_{  \\beta} G(\\beta) &= \\Eb \\big\\{ \\hat{\\partial}_{  \\beta} [g\\circ F](X,\\beta) \\big\\}\n=\\Eb \\big\\{\n\\hat{\\partial}_{  \\beta} F (X,\\beta) \\nabla g(F(X,\\beta))\n\\big\\}\n\\\\ &= \\Eb \\big\\{\n\\hat{\\partial}f (X,Y,\\beta) \\nabla g(F(X,\\beta))\n\\big\\}.\n\\end{aligned}\n\\end{equation}\nThe last equation follows by conditioning.\n\n\nDue to Assumptions (A3) and (A4), by virtue of Theorem \\ref{t:composition},\nthe error function \\eqref{te:Q} is differentiable in the generalized sense as well, and\nthe multifunction\n\\begin{equation}\n\\label{dQ}\n\\hat{\\partial}_{(\\beta,\\theta)} Q(\\beta,\\theta) =\n\\Eb\\left\\{ \\begin{bmatrix}\n\\hat{\\partial}_{  \\beta} f(X,Y,\\beta) \\\\\n-\\hat{\\partial}_{  \\theta} \\varPsi(X,\\theta)\n\\end{bmatrix}\n \\big(   F(X, \\beta) - \\varPsi(X, \\theta)\\big) \\right\\}\n\\end{equation}\nis its generalized subdifferential.\n%Evidently, we cannot observe unbiased estimates of the subgradients of $Q(\\Cdot,\\Cdot)$, because we do not see $F(X,\\beta)$.\nSince the set \\eqref{dQ} is a Cartesian product, its $\\theta$-part,\n\\[\n\\hat{\\partial}_{\\theta} Q(\\beta,\\theta) =\n- \\Eb\\left\\{\n\\hat{\\partial}_{  \\theta} \\varPsi(X,\\theta)\n \\big(   F(X, \\beta) - \\varPsi(X, \\theta)\\big) \\right\\},\n\\]\nis a subdifferential of $Q(\\beta,\\theta)$ with respect to $\\theta$.\n\n\n\\begin{lemma}\n\\label{l:Gamma}\nThe multifunction $\\varGamma(\\Cdot,\\Cdot)$ is nonempty-, convex-, and compact-valued and upper-semicontinuous.\n\\end{lemma}\n\n\\begin{proof}\nBy virtue of Lemma \\ref{l:conditional_subgradient}, the $\\beta$-component of $\\varGamma(\\beta,\\theta)$ equals\n\\begin{multline*}\n -\\Eb \\big[ \\hat{\\partial}_\\beta F(X,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big)\\big] =\n -\\Eb\\Big\\{ \\Eb\\big[ \\hat{\\partial}_\\beta f(X,Y,\\beta)\\big|X\\big] \\nabla g\\big(\\varPsi(X,\\theta)\\big)\\Big\\}\\\\\n% - \\hat{\\partial}_\\beta  \\Eb \\big[ F(X,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big)\\big]\\\\\n% =\n% - \\hat{\\partial}_\\beta  \\Eb \\big[ f(X,Y,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big)\\big]\n  =\n - \\Eb \\big[ \\hat{\\partial}_\\beta  f(X,Y,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big)\\big].\n\\end{multline*}\n\\begin{comment}\nThe interchange of the subdifferentiation and the expected value is justified because\nAssumptions (A2) and (A3) imply that:\n\\begin{tightlist}{ii}\n\\item the function\n$\\beta \\mapsto f(X,Y,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big)$ is\nNorkin differentiable; and\n\\item its Lipschitz constant $L_f(X,Y) L_g$ is integrable.\n\\end{tightlist}\n\\vspace{0.5ex}\n\\end{comment}\n\nThe $\\theta$-component of $\\varGamma(\\beta,\\theta)$ can be written as follows:\n\\[\n\\gamma \\,\\Eb \\big[ \\hat{\\partial}_\\theta \\varPsi(X,\\theta)\\big(F(X,\\beta) -\\varPsi(X,\\theta)\\big)\\big]\n= \\gamma \\,\\Eb \\big[ \\hat{\\partial}_\\theta \\varPsi(X,\\theta)\\big(f(X,Y,\\beta) -\\varPsi(X,\\theta)\\big)\\big].\n\\]\nDue to Assumptions (A2)--(A4), for all $(X,Y)$, the multifunction\n\\[\n(\\beta,\\theta) \\mapsto \\widetilde{\\varGamma}(\\beta,\\theta;X,Y) \\triangleq\n\\begin{bmatrix}\n -\\hat{\\partial}_{ \\beta} f(X,Y,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big) \\\\\n\\gamma \\,\\hat{\\partial}_{ \\theta} \\varPsi(X,\\theta)\\big(f(X,Y,\\beta) -\\varPsi(X,\\theta)\\big)\n\\end{bmatrix}\n\\]\nis upper-semicontinuous, and convex- and compact-valued. Furthermore, its $\\beta$-values are\nbounded by and integrable function $L_f(X,Y)L_g$, and its $\\theta$-values are bounded\nby an integrable function $  \\gamma L_{\\varPsi}(X)( C_f +  C_{\\varPsi} + 4  L_{\\varPsi}(X) R^2)$.\nThese two conditions imply that its integral -- the multifunction $\\varGamma(\\Cdot,\\Cdot)$ -- is upper-semicontinuous, and convex- and compact-valued as well. This implication is part of\nthe proof of Theorem \\ref{t:expected-generalized}; see \\cite[Lem. 21.7]{mikhalevich1987nonconvex}.\n\\end{proof}\n\n\\begin{lemma}\n\\label{l:dk-in-Gamma}\nFor all $k=0,1,2,\\dots$ we have $d^k \\in \\varGamma(\\beta^k,\\theta^k)$.\n\\end{lemma}\n\n\\begin{proof}\nSince the observation $(X^k,Y^k)$ is independent of $\\Fc_k$, Lemma \\ref{l:conditional_subgradient}  yields the relations\n\\begin{align*}\n\\Eb[\\tilde{d}_\\beta^k|\\Fc_k]  &\\in -\\Eb \\big[ \\hat{\\partial}_{ \\beta} f(X^k,Y^k,\\beta^k) \\nabla g(\\varPsi(X^k,\\theta^k)) ~\\big|~ \\Fc_k\\big] \\\\\n&= -\\Eb \\Big\\{ \\Eb \\big[ \\hat{\\partial}_{ \\beta} f(X^k,Y^k,\\beta^k)  ~\\big|~ \\beta^k,\\theta^k, X^k \\big] \\nabla g(\\varPsi(X^k,\\theta^k)) ~\\Big|~ \\beta^k,\\theta^k \\Big\\}\\\\\n&= -\\Eb \\big[ \\hat{\\partial}_{ \\beta} F(X^k,\\beta^k) \\nabla g(\\varPsi(X^k,\\theta^k)) ~\\big|~ \\beta^k,\\theta^k \\big]\n= \\varGamma_\\beta (\\beta^k,\\theta^k).\n\\end{align*}\nThe $\\theta$-component of the direction can also be treated by conditioning:\n\\begin{align*}\n\\Eb[\\tilde{d}_\\theta^k|\\Fc_k] &\\in \\gamma \\Eb \\big[ \\hat{\\partial}_{ \\theta} \\varPsi(X^k,\\theta^k)\\big(f(X^k,Y^k,\\beta^k) -\\varPsi(X^k,\\theta^k)\\big)\\big| \\Fc_k\\big]\\\\\n&= \\gamma \\Eb \\Big\\{\\hat{\\partial}_{ \\theta} \\varPsi(X^k,\\theta^k)\\Eb\\big[f(X^k,Y^k,\\beta^k) -\\varPsi(X^k,\\theta^k)\\big| X^k,\\beta^k,\\theta^k\\big]\\Big| \\beta^k,\\theta^k\\Big\\}\\\\\n&= \\gamma \\Eb \\big\\{\\hat{\\partial}_{ \\theta} \\varPsi(X^k,\\theta^k)\\big(F(X^k,\\beta^k) -\\varPsi(X^k,\\theta^k)\\big)\\big| \\beta^k,\\theta^k\\big\\} = \\varGamma_\\theta(\\beta^k,\\theta^k).\n\\end{align*}\n\\end{proof}\n\n\\begin{lemma}\n    \\label{l:e2-bounded}\nUnder Assumptions \\textup{(A1)--(A4)}, the directions $\\{d^k\\}_{k\\ge 0}$ are well-defined and a constant $\\sigma > 0$ exists, such that\n$\\Eb\\big[\\|e^k\\|^2 \\,\\big|\\,\\Fc_k\\big] \\le \\sigma^2$, for all $k \\ge 0$.\n\\end{lemma}\n\n\\begin{proof}\nFor the $\\beta$-component $e_\\beta^k$ of $e^k$,  we have\n\\[\n\\tilde{d}_\\beta^k = -f_\\beta(X^k,Y^k,\\beta^k)\\nabla g(\\varPsi(X^k,\\theta^k)),\n\\]\nand thus, by Assumptions (A2) and (A3),\n\\[\n\\Eb\\big[ \\|{e}_\\beta^k\\|^2\\big| \\Fc_k ] \\le \\Eb\\big[ \\|\\tilde{d}_\\beta^k\\|^2\\big| \\Fc_k ]\n\\le L_g^2 \\Eb \\big[ \\| f_\\beta(X^k,Y^k,\\beta^k)\\|^2 \\big| \\Fc_k \\big]\\le\n L_g^2 \\widebar{L}_f^2.\n\\]\nFor the $\\theta$-component,\n\\begin{align*}\n\\Eb\\big[ \\| \\tilde{d}_\\theta^k\\|^2\\big| \\Fc_k\\big]\n&\\le \\gamma^2 \\Eb\\big[ \\|\\varPsi_\\theta(X^k,\\theta^k)\\|^2 \\cdot \\| f(X^k,Y^k,\\beta^k)  - \\varPsi(X^k,\\theta^k)\\|^2\\big| \\Fc_k\\big]\\\\\n&\\le \\gamma^2 \\Eb\\big[{L_\\psi}(X^k)^2\\| f(X^k,Y^k,\\beta^k)  - \\varPsi(X^k,\\theta^k)\\|^2\\big| \\Fc_k\\big]\\\\\n&\\le 2\\gamma^2 \\Eb\\big[{L_\\psi}(X^k)^2\\| f(X^k,Y^k,\\beta^k)\\|^2\\big| \\Fc_k\\big]\\\\\n&{\\quad} +2 \\gamma^2 \\Eb\\big[ {L_\\psi}(X^k)^2\\|\\varPsi(X^k,\\theta^k)\\|^2\\big| \\Fc_k\\big].\n\\end{align*}\nUsing the Cauchy--Schwartz inequality and Assumptions (A3) and (A4) we obtain\n\\[\n\\Eb\\big[{L_\\psi}(X^k)^2\\| f(X^k,Y^k,\\beta^k)\\|^2\\big| \\Fc_k\\big]\n\\le  \\widebar{L}_\\varPsi^2 C_f^2.\n\\]\nIn a similar way, by (A4),\n\\begin{align*}\n\\lefteqn{\\Eb\\big[ {L_\\psi}(X^k)^2\\|\\varPsi(X^k,\\theta^k)\\|^2\\big| \\Fc_k\\big]}\\\\\n&\\le 2 \\Eb\\big[{L_\\psi}(X^k)^2\\big( \\|\\varPsi(X^k,\\theta^0)\\|^2 + {L_\\psi}(X^k)^2 \\|\\theta^k-\\theta^0\\|^2\\big)\\big| \\Fc_k\\big]\n\\le 2 \\widebar{L}_\\varPsi^2 C_\\varPsi^2 + 8R^2 \\widebar{L}_\\varPsi^4.\n\\end{align*}\nCombining the last three inequalities, we conclude that\n$\\Eb\\big[ \\| e_\\theta^k\\|^2\\big| \\Fc_k\\big] $, $k\\ge 0$, are uniformly bounded as well.\n\\end{proof}\n\nThe following theorem follows directly from \\cite[Thm. 4.1]{majewski2018analysis}.\n\\begin{theorem}\n\\label{t:diff-inc-beta}\nUnder Assumptions \\textup{(A1)--(A4)} and \\textup{(A7)},\nthe sequence $\\big\\{Z_m(\\Cdot)\\big\\}_{m\\ge 0}$ has a subsequence convergent a.s. to an absolutely continuous function $Z_\\infty:[0,\\infty) \\to \\Xc$ which is a solution of the differential inclusion:\n\\begin{equation}\n\\label{diff-inc-beta}\n\\dt{Z}_\\infty(t) \\in  \\varGamma(Z_{\\infty}(t))-\\Nc_{\\Zb}(Z_\\infty(t)), \\quad t \\ge 0.\n\\end{equation}\nMoreover, for any $t\\ge 0$, the point $Z_\\infty(t)$\nis an accumulation point of the sequence $\\{z^k\\}_{k\\ge 0}$.\n\\end{theorem}\n\nSince any solution of \\eqref{diff-inc-beta} must lie in $\\Zb$,\nwe can  write the differential inclusion \\eqref{diff-inc-beta} in an equivalent way \\cite{aubin2012differential}:\n\\begin{equation}\n\\label{diff-inc-beta-T}\n\\dt{Z}_\\infty(t) \\in \\varPi_{\\Tc_{\\Zb}(Z_\\infty(t))}\\big( \\varGamma(Z_\\infty(t))\\big), \\quad t \\ge 0,\n\\end{equation}\nwhich underlines the nature of the limiting subgradient process. Here, $\\Tc_\\Zb(z)$ is the tangent cone to the convex set $\\Zb$ at $z$.\n\n\n\\begin{lemma}\n\\label{l:w-gendif}\nUnder Assumptions \\textup{(A2)--(A4)} the Lyapunov function $w(\\Cdot,\\Cdot)$ is differentiable in the generalized sense.\n\\end{lemma}\n\n\\begin{proof}\nIn \\S \\ref{s:2} we established the generalized differentiability of the functions $G(\\Cdot)$ and $Q(\\Cdot,\\Cdot)$.  It remains to consider the function\n\\[\n\\Delta^{0}(\\beta,\\theta) = \\Eb\\Big[g(F) - g(\\varPsi) - \\langle \\nabla g(\\varPsi), F - \\varPsi \\rangle \\Big],\n\\]\nand the expression under the expected value:\n\\begin{equation}\n    \\label{delta0-beta}\n\\delta^{0}(\\beta,\\theta) = g(F) - g(\\varPsi) - \\langle \\nabla g(\\varPsi), F - \\varPsi \\rangle.\n\\end{equation}\nBy Assumptions (A2)--(A4) and Theorem \\ref{t:composition},\n$\\delta^{0}(\\Cdot,\\Cdot)$ is Norkin differentiable. Its generalized subgradients with respect to $\\beta$ have the form\n\\[\n\\delta^{0}_\\beta = F_\\beta\\big( \\nabla g(F) - \\nabla g(\\Psi)\\big),\n\\]\nand are bounded by $2L_g\\Eb[\\widebar{L}_f(X,Y)|X]$. It is an integrable function,\nthanks to (A2).\nIn a similar way, by Assumptions (A2) and (A4) and Theorem \\ref{t:composition}, the generalized subgradients\nwith respect to $\\theta$ have the form\n\\begin{equation}\n    \\label{delta0-theta}\n\\delta^{0}_\\theta = \\varPsi_\\theta H(\\varPsi) ( F - \\Psi),\n\\end{equation}\n or are convex combinations of elements of this structure,\n where we denote by $H(\\varPsi)$\nan arbitrary element of $\\hat{\\partial}(\\nabla g)(\\varPsi)$. Thus, their norm is\nbounded by $L_{\\nabla g}L_\\varPsi(X)\\|F-\\varPsi\\|$. As $L_\\varPsi$\nand $\\|F-\\varPsi\\|$ are square-integrable, the bound is integrable, by virtue\nof the Cauchy--Schwartz inequality.  Applying Theorem \\ref{t:expected-generalized},\nwe conclude that the expected value function $\\Delta^0$ is Norkin differentiable as well.\n\\end{proof}\n\nFollowing the differential inclusion method, consider the function $W(t) = w(\\beta(t),\\theta(t))$, $t \\ge 0$, where $(\\beta(\\Cdot),\\theta(\\Cdot)) \\triangleq Z(\\Cdot)$ is a solution of the\ninclusion \\eqref{diff-inc-beta}.\n\nAs the Lyapunov function $w(\\Cdot,\\Cdot)$ is Norkin differentiable, by virtue of Theorem \\ref{t:composition}, its\nweak derivative can be expressed by the chain rule:\n\\begin{equation}\n\\dt{W}\n= \\langle w_\\beta, \\dt{\\beta} \\rangle + \\langle w_\\theta, \\dt{\\theta} \\rangle\n= \\langle G_\\beta + \\alpha \\Delta^\\lambda_\\beta, \\dt{\\beta} \\rangle + \\alpha \\langle \\Delta^\\lambda_\\theta, \\dt{\\theta} \\rangle.\n\\label{grad:P}\n\\end{equation}\nDue to Theorem \\ref{t:chain}, the integral  $\\int_0^T \\dt{W}(t)\\,\\D t$ on any interval of time does not depend on the particular selections of the generalized subgradients used in this calculation.\n\n\n\\begin{lemma}\nUnder Assumptions \\textup{(A2)} and \\textup{(A4)},  $\\big|\\langle \\Delta^0_\\theta, Q_\\theta \\rangle \\big| \\le 2 {L_{\\nabla g}} \\widebar{L}_\\varPsi^2 Q$.\n\\label{l:QG_delta_theta}\n\\end{lemma}\n\n\\begin{proof}\nIn view of \\eqref{delta0-theta},  $\\Delta^0_\\theta = \\Eb\\big[ \\varPsi_\\theta H(\\varPsi)(F-\\varPsi)\\big]$. By virtue of Assumption (A2), $\\|H(\\varPsi)\\| \\le L_{\\nabla g}$.\nAlso,\n$Q_\\theta = \\Eb\\big[ \\varPsi_\\theta(F-\\varPsi)\\big]$.\nUsing the Cauchy--Schwartz inequality,\nwe obtain the following chain of estimates:\n\\begin{align*}\n\\big|\\langle \\Delta^0_\\theta, Q_\\theta \\rangle\\big|  &\\le \\| \\Delta^0_\\theta\\| \\| Q_\\theta \\|\n\\le {L_{\\nabla g}} \\Eb[\\|\\varPsi_\\theta\\|^2] \\Eb[\\|F - \\varPsi\\|^2] \\le 2 {L_{\\nabla g}} \\widebar{L}_\\varPsi^2 Q.\n\\end{align*}\nThe last inequality uses \\eqref{te:Q}.\n\\end{proof}\n\n\\begin{lemma}\nUnder Assumption \\textup{(A3)},  $\\|Q_\\beta\\|^2 \\le 2 \\widebar{L}_f^2 Q$.\n\\label{l:QG_Q_beta}\n\\end{lemma}\n\n\\begin{proof}\nBy the  Cauchy--Schwartz inequality,\n\n\\[\n\\|Q_\\beta\\| = \\big\\|\\Eb[ F_\\beta(F-\\varPsi)]\\big\\| \\le \\sqrt{\\Eb[\\|F_\\beta\\|^2]}\\sqrt{ \\Eb[\\|F-\\varPsi\\|^2]} \\le  \\widebar{L}_f \\sqrt{2 Q}.\n\\]\n\\end{proof}\n\n\\begin{lemma}\nUnder Assumptions \\textup{(A2)} and \\textup{(A3)}, $\\|\\Delta^0_\\beta\\|^2 \\le 2 L_{\\nabla g}^2 \\widebar{L}_f^2 Q$.\n\\label{l:QG_delta_beta}\n\\end{lemma}\n\n\\begin{proof}\nBy \\eqref{delta0-beta}, $\\Delta^0_\\beta = \\Eb[F_\\beta(\\nabla g(F) - \\nabla g(\\varPsi))$.\nSimilar to the proof of Lemma \\ref{l:QG_Q_beta}, by the Cauchy--Schwartz inequality and Assumption (A2),\n\\begin{align*}\n\\|\\Delta^0_\\beta\\|^2 &= \\big\\|\\Eb[F_\\beta(\\nabla g(F) - \\nabla g(\\varPsi)) ]\\big\\|^2\n\\le \\Eb[\\|F_\\beta\\|^2]\\, \\Eb[\\|\\nabla g(F) - \\nabla g(\\varPsi)\\|^2]  \\\\\n&\\le L_{\\nabla g}^2 \\Eb[\\|F_\\beta\\|^2]\\, \\Eb[\\|F - \\varPsi\\|^2]\n\\le 2L_{\\nabla g}^2 \\widebar{L}_f^2 Q.\n\\end{align*}\nThe last inequality uses Assumption (A3), as in Lemma \\ref{l:QG_Q_beta}.\n\\end{proof}\n\n\\begin{proposition}\nUnder assumptions \\textup{(A2)} and \\textup{(A3)},\n\\[\n\\langle w_\\beta, \\dt{\\beta} \\rangle \\le \\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 Q.\n\\]\nFurthermore, if $Q=0$, then\n\\[\n\\langle w_\\beta, \\dt{\\beta} \\rangle = - \\big\\| \\varPi_{\\Tc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big)\\big\\|^2.\n\\]\n\\label{p:QG_L_delta_Q_beta}\n\\end{proposition}\n\n\\begin{proof}\nThe first coordinate of the differential inclusion \\eqref{diff-inc-beta} reads:\n\\begin{equation}\n\\dt{\\beta} \\in \\varPi_{\\Tc_{\\Bb}(\\beta)}\\big(-\\Eb[F_\\beta \\nabla g(\\varPsi) ] \\big) = \\varPi_{\\Tc_{\\Bb}(\\beta)}\\big( -G_\\beta + \\Delta^{0}_\\beta \\big),\n\\label{grad:beta}\n\\end{equation}\nFor any solution $(\\beta(t), \\theta(t))$ of the inclusion \\eqref{t:diff-inc-beta}, with the corresponding element $N(t) \\in \\Nc_{\\Bb}(\\beta(t))$, we define the projection matrix\n\\[\nP(t)  = I - \\frac{N(t) N(t)^T}{\\|N(t)\\|^2}.\n\\]\nThen\n\\begin{equation}\n\\dt{\\beta} = P (-G_\\beta + \\Delta^{0}_\\beta).\n\\label{grad:beta_proj}\n\\end{equation}\nUsing the facts that $P$ is nonexpansive, $P^T=P$, and $P^2=P$,\nby elementary manipulations, we obtain an upper bound:\n\\begin{equation}\n\\label{w-dot-chain}\n\\begin{aligned}\n\\lefteqn{\\langle w_\\beta, \\dt{\\beta} \\rangle = -\\langle G_\\beta + \\alpha \\Delta^{\\lambda}_\\beta, P(G_\\beta - \\Delta^{0}_\\beta) \\rangle} \\\\\n&= -\\Big\\|P\\big(G_\\beta + \\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta - \\frac{1}{2} \\Delta^0_\\beta\\big)\\Big\\|^2 + \\Big\\|P\\big(\\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta + \\frac{1}{2} \\Delta^0_\\beta\\big)\\Big\\|^2 \\\\\n&\\le \\Big\\|P\\big(\\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta + \\frac{1}{2} \\Delta^0_\\beta\\big) \\Big\\|^2 =\n\\Big\\|P\\big(\\frac{\\alpha+1}{2} \\Delta^0_\\beta + \\frac{\\alpha \\lambda}{2} Q_\\beta\\big)\\Big\\|^2 \\\\\n&= \\Big\\|\\frac{\\alpha+1}{2} P \\Delta^0_\\beta\\Big\\|^2 + \\Big\\|\\frac{\\alpha \\lambda}{2} P Q_\\beta\\Big\\|^2 + 2 \\Big\\langle \\rho \\frac{\\alpha+1}{2} P \\Delta^0_\\beta, \\varrho^{-1} \\frac{\\alpha \\lambda}{2} P Q_\\beta \\Big\\rangle\\\\\n&\\le \\frac{(\\alpha+1)^2}{4} (1+\\rho^2)\\|P \\Delta^0_\\beta\\|^2 + \\frac{\\alpha^2 \\lambda^2}{4}(1+\\rho^{-2})\\|P Q_\\beta\\|^2,\n\\end{aligned}\n\\end{equation}\nfor any $\\rho>0$.\nLemmas \\ref{l:QG_Q_beta} and \\ref{l:QG_delta_beta} allow us to continue the last estimate as follows:\n\\begin{align*}\n\\langle w_\\beta, \\dt{\\beta} \\rangle &\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{4} (1+\\rho^2)\\|P \\Delta^0_\\beta\\|^2 + \\frac{\\alpha^2 \\lambda^2}{4}(1+\\rho^{-2})\\|P Q_\\beta\\|^2 \\Big] \\\\\n&\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{4} (1+\\rho^2)\\|\\Delta^0_\\beta\\|^2 + \\frac{\\alpha^2 \\lambda^2}{4}(1+\\rho^{-2})\\|Q_\\beta\\|^2 \\Big] \\\\\n&\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{2} (1+\\rho^2) L_{\\nabla g}^2 + \\frac{\\alpha^2 \\lambda^2}{2}(1+\\rho^{-2}) \\Big]   \\widebar{L}_f^2 Q \\\\\n&= \\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 Q.\n\\end{align*}\nThis proves the first assertion.\n\\begin{comment}\n\\begin{align*}\n\\langle w_\\beta, \\dt{\\beta} \\rangle &= -\\langle G_\\beta + \\alpha \\Delta^{\\lambda}_\\beta, P(G_\\beta - \\Delta^{0}_\\beta) \\rangle \\\\\n&= -\\Big\\|G_\\beta + \\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta - \\frac{1}{2} \\Delta^0_\\beta\\Big\\|_P^2 + \\Big\\|\\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta + \\frac{1}{2} \\Delta^0_\\beta\\Big\\|_P^2 \\\\\n&\\le \\Big\\|\\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta + \\frac{1}{2} \\Delta^0_\\beta\\Big\\|_P^2 =\n\\Big\\|\\frac{\\alpha+1}{2} \\Delta^0_\\beta + \\frac{\\alpha \\lambda}{2} Q_\\beta\\Big\\|_P^2 \\\\\n&= \\Big\\|\\frac{\\alpha+1}{2} \\Delta^0_\\beta\\Big\\|_P^2 + \\Big\\|\\frac{\\alpha \\lambda}{2} Q_\\beta\\Big\\|_P^2 + 2 \\Big\\langle \\rho \\frac{\\alpha+1}{2} \\Delta^0_\\beta, \\varrho^{-1} \\frac{\\alpha \\lambda}{2} P Q_\\beta \\Big\\rangle \\\\\n&\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{4} (1+\\rho^2)\\|\\Delta^0_\\beta\\|_P^2 + \\frac{\\alpha^2 \\lambda^2}{4}(1+\\rho^{-2})\\|Q_\\beta\\|_P^2 \\Big] \\\\\n&\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{4} (1+\\rho^2)\\|\\Delta^0_\\beta\\|^2 + \\frac{\\alpha^2 \\lambda^2}{4}(1+\\rho^{-2})\\|Q_\\beta\\|^2 \\Big] \\\\\n&\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{2} (1+\\rho^2) {L_{\\nabla g}}^2 + \\frac{\\alpha^2 \\lambda^2}{2}(1+\\rho^{-2}) \\Big]   \\widebar{L}_f^2 M \\|Q_\\theta\\|^2 \\\\\n&= \\frac{\\widebar{L}_f^2 M}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 \\|Q_\\theta\\|^2.\n\\end{align*}\n\\end{comment}\nNow, suppose $Q=0$. This implies that $\\varDelta^\\lambda=0$ for all $\\lambda \\ge 0$.\nThe first line of \\eqref{w-dot-chain} yields\n\\[\n\\langle w_\\beta, \\dt{\\beta} \\rangle = - \\| P G_\\beta \\|^2 = - \\big\\| \\varPi_{\\Tc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big)\\big\\|^2,\n\\]\nas claimed.\n\\end{proof}\n\nThe second coordinate of the differential inclusion \\eqref{diff-inc-beta} reads:\n\\begin{equation}\n\\dt{\\theta} =  \\varPi_{\\Tc_{\\Theta_R}(\\theta)}\\big(\\gamma\\, \\Eb[(F - \\varPsi)^T \\varPsi_\\theta] \\big)\n= \\varPi_{\\Tc_{\\Theta_R}(\\theta)}\\big( -\\gamma Q_\\theta\\big) = -\\gamma Q_\\theta,\n\\label{grad:simp}\n\\end{equation}\nbecause, by virtue of  Assumption (A6), when $\\|\\theta\\|=R$, then $ -\\gamma Q_\\theta \\in \\Tc_{\\Theta_R}(\\theta)$.\n\n\n\\begin{theorem}\n\\label{t:negative-der}\nUnder Assumptions \\textup{(A1)--(A6)}, if\n\\begin{equation}\n\\label{negative-der}\n\\lambda > 2 \\widebar{L}_\\varPsi^2 M {L_{\\nabla g}} \\quad \\text{and} \\quad\n\\gamma > \\frac{\\widebar{L}_f^2 M\\big( (\\alpha + 1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2}{2 \\alpha \\big(\\lambda - 2 \\widebar{L}_\\varPsi^2 M {L_{\\nabla g}} \\big)} ,\n\\end{equation}\nthen $\\dt{W} \\le 0$. Furthermore, $\\dt{W} < 0$ if $(\\beta,\\theta)\\notin \\Zb^*$.\n\\end{theorem}\n\n\\begin{proof}\nFollowing \\eqref{grad:P}, \\eqref{grad:simp}, Lemma \\ref{l:QG_delta_theta}, and Proposition \\ref{p:QG_L_delta_Q_beta},\n\\begin{align*}\n\\dt{W}\n&\\le \\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 Q - \\alpha \\gamma \\langle \\Delta^{\\lambda}_\\theta, Q_\\theta \\rangle \\\\\n&= \\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 Q - \\alpha \\gamma \\big[\\langle \\Delta^0_\\theta, Q_\\theta \\rangle + \\lambda \\|Q_\\theta\\|^2 \\big] \\\\\n&\\le \\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 Q + \\alpha \\gamma |\\langle \\Delta^0_\\theta, Q_\\theta \\rangle| - \\alpha \\gamma \\lambda \\|Q_\\theta\\|^2 \\\\\n&\\le \\Big[\\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 + \\alpha \\gamma \\cdot 2{L_{\\nabla g}} \\widebar{L}_\\varPsi^2 - \\frac{\\alpha \\gamma \\lambda}{M} \\Big] Q \\\\\n&= \\frac{1}{M} \\Big[\\frac{\\widebar{L}_f^2 M}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 - \\alpha \\gamma \\big(\\lambda  - 2 \\widebar{L}_\\varPsi^2 M{L_{\\nabla g}} \\big) \\Big] Q \\le 0.\n\\end{align*}\nThis verifies the first assertion.\nNow, suppose $\\dt{W}=0$. It follows from the last inequality that $Q=0$. Then, Proposition \\ref{p:QG_L_delta_Q_beta} implies that\n\\[\n\\varPi_{\\Tc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big) = 0\n\\]\nfor some $G_\\beta \\in \\hat{\\partial}{G}(\\beta)$. Consequently, using the decomposition into the projections on polar cones,\nwe obtain\n\\[\n-G_\\beta = \\varPi_{\\Tc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big) + \\varPi_{\\Nc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big) =\n\\varPi_{\\Nc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big).\n\\]\nThis means that the optimality condition \\eqref{optimality} is satisfied.\n\\end{proof}\n\n\\begin{proof}\nThe proof follows the steps of the proofs of \\cite[Thm. 3.20]{duchi2018stochastic} or \\cite[Thm. 3.5]{majewski2018analysis} {  or \\cite{nurminskii1972convergence}}.\nWe provide a short outline here, for the convenience of the Reader. For brevity, we omit the\nargument $\\omega$, but we stress that we consider a particular path of the method.\n\nDefine\n\\[\n\\underline{w} = \\liminf_{k \\to \\infty^{\\phantom{|}}} w(z^k), \\qquad \\overline{w} = \\limsup_{k \\to \\infty}w(z^k).\n\\]\nSuppose $\\underline{w} < \\overline{w}$. By Assumption (A7), a value ${v}\\in\n(\\underline{w},\\overline{w})$ exists,\nsuch that ${v} \\notin G(\\Bb^*)$. Take any $u \\in ({v},\\overline{w})$.\nDefine $L_1=\\min\\big\\{k : w(z^k) \\le {v} \\big\\}$ and for\n$j=1,2,\\dots$\n\\begin{align*}\n\\ell_j &= \\min\\big\\{k\\ge L_j : w(z^k) > u \\big\\},\\\\\nL_{j+1} &= \\min\\big\\{k\\ge \\ell_j : w(z^k) \\le {v} \\big\\}.\\\\\n\\intertext{Then we define}\nk_j &= \\max\\big\\{ k\\in [L_j,\\ell_j]: w(z^k) \\le {v} \\big\\}, \\quad j=1,2,\\dots.\n\\end{align*}\nBoth $k_j\\to \\infty$ and $\\ell_j\\to \\infty$, as $j\\to \\infty$.\n\nConsider the sums\n$T_j = \\sum_{k=k_j}^{\\ell_j-1} \\tau_k$. Due to the equicontinuity of the functions\n $w(Z_j(\\Cdot))$, a constant $T >0$ exists, such that $T_j\\ge T$ for all sufficiently\nlarge $j$. Set $s_j=t_{k_j}$ in \\eqref{Z-shifted} and consider the interval $[0,T]$.\nBy Theorem \\ref{t:diff-inc-beta}, a subsequence of the sequence $\\{Z_j(\\Cdot)\\}$ is convergent to a solution $Z_\\infty(\\Cdot)$ of the differential\ninclusion \\eqref{diff-inc-beta}. For simplicity, we still denote it by $\\{Z_j(\\Cdot)\\}$. As $W(\\Cdot)$ is absolutely continuous,\n\\begin{equation}\n\\label{gradient-desc-cont}\nw(Z_\\infty(T)) = w(Z_\\infty(0)) + \\int_0^T  \\dt{W}(t)  \\;\\D t.\n\\end{equation}\nSince ${v} \\notin w(\\Zb^*)$, then  $Z_\\infty(0) = \\lim_{j\\to \\infty} z^{k_j} \\notin \\Zb^*$.\nTheorem \\ref{t:negative-der} then implies that $\\dt{W}(0)<0$.\nThe generalized subdifferential $\\hat{\\partial} G(\\Cdot)$ and the normal cone $\\Nc_\\Bb(\\Cdot)$ are upper semicontinuous and closed-valued. The function $Q(\\Cdot,\\Cdot)$ is continuous.\nTherefore, $\\dt{W}(t)<0$\n in some interval $t\\in [0,\\bar{t}\\,]$ of positive length.\n\n It follows from \\eqref{gradient-desc-cont}\n that $w(Z_\\infty(T)) < w(Z_\\infty(0)) =v$.\n However, $Z_\\infty(T)$ is the limit of the sequence $\\{z^{m_j}\\}$, where\n\\begin{equation}\n\\label{mj}\nm_j = \\min\\bigg\\{m \\ge k_j: \\sum_{k=k_j}^{m_j-1}\\tau_k \\ge T\\bigg\\}, \\quad j=1,2,\\dots.\n\\end{equation}\nFor all sufficiently large $j$ we  have $k_j \\le m_j \\le \\ell_j$ and $w(z^{m_j}) > v$, which is a contradiction. Therefore,\n$\\underline{w} = \\overline{w}$ and the sequence $\\big\\{w(z^k)\\big\\}$ has a limit.\n\nSuppose a subsequence $\\{z^{k_j}\\}$ is convergent to a point $\\bar{z}\\notin \\Zb^*$. Again, we\nset $s_j = t_{k_j}$ in \\eqref{Z-shifted}, consider the interval $[0,T]$ with $T>0$, and conclude that\na  sub-subsequence of $\\{Z_j(\\Cdot)\\}$ exists, which is uniformly convergent to a solution $Z_\\infty(\\Cdot)$ of the\ndifferential inclusion \\eqref{diff-inc-beta}. Then the points $\\{z^{m_j}\\}$ are convergent to $Z_\\infty(T)$ on this sub-subsequence,\nwhere $m_j$ are defined as in \\eqref{mj}.\nAs $Z_\\infty(0)=\\bar{z}\\notin \\Zb^*$, we obtain $Z_\\infty(T)< Z_\\infty(0)$. This would mean that a sub-subsequence\nof $\\big\\{w(z^{m_j})\\big\\}$ has the limit $w(Z_\\infty(T))< w(\\bar{z})$, which contradicts the convergence of the sequence\n$\\big\\{w(z^{k})\\big\\}$.\n\\end{proof}","units":[{"start":26745,"end":27374,"start_line":483,"end_line":492,"raw_tex":"\\begin{lemma}\n\\label{l:conditional_subgradient}\nSuppose the function $f(X,Y,\\Cdot)$ is differentiable in the generalized sense and its subgradients are bounded\nby an integrable function in any bounded neighborhood of any point $\\beta$. Then for almost all $X$ the function\n$\\beta \\mapsto \\Eb\\big[ f(X,Y,\\beta) \\big| X\\big]$ is differentiable in the generalized sense, and\nfor every measurable function $s:\\Rb^{n_x}\\times \\Rb^{n_y} \\to \\Rb^{n_f}$ such that $s(X,Y)  \\in \\hat{\\partial}_\\beta f(X,Y,\\beta)$ a.s., we have\n\\[\n\\Eb\\big[s(X,Y) \\big| X\\big] \\in \\hat{\\partial} \\Eb\\big[ f(X,Y,\\beta) \\big| X\\big] \\quad  a.s.\n\\]\n\\end{lemma}","sha256_utf8":"842557bd97bd839458311003f8422616879ad2c85aaac6ea26b0df0af27a8686"},{"start":27376,"end":28027,"start_line":494,"end_line":507,"raw_tex":"\\begin{proof}\nThe probability measure $P$ on $\\Rb^{n_x}\\times \\Rb^{n_y}$\ncan be disintegrated into a marginal measure $P_X$ on $\\Rb^{n_x}$ and a kernel $T:\\Rb^{n_x} \\to \\Pc(\\Rb^{n_y})$,\n where $\\Pc(\\Rb^{n_y})$ is the space of probability measures\n on $\\Rb^{n_y}$. Then\n\\[\n\\Eb\\big[ f(X,Y,\\beta) \\big| X=x\\big] = \\int f(x,y,\\beta)\\;T(\\D y|x).\n\\]\nFor every bounded neighborhood $K$ of $\\beta$, the Lipschitz constant $L_K(x,y)$ of the function $f(x,y,\\Cdot)$\nis integrable, and thus, for almost\nall $x$, its conditional expectation\n$\\int L_K(x,y)\\,T(dy|x)$ exists as well.\nThe assertion then follows from Theorem \\ref{t:expected-generalized}.\n\\end{proof}","sha256_utf8":"dd806deef08b28da5a98fd48e9ddcef9c649ed7c555c4323f722ca10ec1d0f84"},{"start":28676,"end":29761,"start_line":518,"end_line":538,"raw_tex":"Furthermore, by Theorem \\ref{t:composition} and Assumption (A2), the composition\n$g( F(x,\\Cdot))$ is differentiable in the generalized sense as well, and\n\\[\n\\hat{\\partial}_{  \\beta} [g\\circ F](x,\\beta)  = \\big\\{    F_\\beta(x,\\beta) \\nabla g(F(x,\\beta)) : F_\\beta(x,\\beta) \\in \\hat{\\partial}_{  \\beta} F (x,\\beta)\\big\\}\n\\]\nis its generalized subdifferential (we adopt the convention that the generalized Jacobian $F_\\beta$ is an $n_\\beta \\times n_f$ matrix).\nThanks to the integrability condition in (A3) and Theorem \\ref{t:expected-generalized}, the function $G(\\Cdot)$ in \\eqref{cso}\nis differentiable in the generalized sense as well, and a realization of its generalized subdifferential is the multifunction\n\\begin{equation}\n    \\label{G-sub}\n\\begin{aligned}\n\\hat{\\partial}_{  \\beta} G(\\beta) &= \\Eb \\big\\{ \\hat{\\partial}_{  \\beta} [g\\circ F](X,\\beta) \\big\\}\n=\\Eb \\big\\{\n\\hat{\\partial}_{  \\beta} F (X,\\beta) \\nabla g(F(X,\\beta))\n\\big\\}\n\\\\ &= \\Eb \\big\\{\n\\hat{\\partial}f (X,Y,\\beta) \\nabla g(F(X,\\beta))\n\\big\\}.\n\\end{aligned}\n\\end{equation}\nThe last equation follows by conditioning.\n","sha256_utf8":"c499ce3429eab15c75ebfb1618dd6659d126144e0f1829e962de1c9484a6f8bf"},{"start":30549,"end":31467,"start_line":559,"end_line":580,"raw_tex":"Due to Assumptions (A3) and (A4), by virtue of Theorem \\ref{t:composition},\nthe error function \\eqref{te:Q} is differentiable in the generalized sense as well, and\nthe multifunction\n\\begin{equation}\n\\label{dQ}\n\\hat{\\partial}_{(\\beta,\\theta)} Q(\\beta,\\theta) =\n\\Eb\\left\\{ \\begin{bmatrix}\n\\hat{\\partial}_{  \\beta} f(X,Y,\\beta) \\\\\n-\\hat{\\partial}_{  \\theta} \\varPsi(X,\\theta)\n\\end{bmatrix}\n \\big(   F(X, \\beta) - \\varPsi(X, \\theta)\\big) \\right\\}\n\\end{equation}\nis its generalized subdifferential.\n%Evidently, we cannot observe unbiased estimates of the subgradients of $Q(\\Cdot,\\Cdot)$, because we do not see $F(X,\\beta)$.\nSince the set \\eqref{dQ} is a Cartesian product, its $\\theta$-part,\n\\[\n\\hat{\\partial}_{\\theta} Q(\\beta,\\theta) =\n- \\Eb\\left\\{\n\\hat{\\partial}_{  \\theta} \\varPsi(X,\\theta)\n \\big(   F(X, \\beta) - \\varPsi(X, \\theta)\\big) \\right\\},\n\\]\nis a subdifferential of $Q(\\beta,\\theta)$ with respect to $\\theta$.\n","sha256_utf8":"8f872fcad806fe63d63b39d09d33abf38b61e3ac88101b0eb0b3e18e273c5909"},{"start":43825,"end":43977,"start_line":818,"end_line":821,"raw_tex":"\\begin{lemma}\n\\label{l:Gamma}\nThe multifunction $\\varGamma(\\Cdot,\\Cdot)$ is nonempty-, convex-, and compact-valued and upper-semicontinuous.\n\\end{lemma}","sha256_utf8":"e4ee475adf57c1e547da9b2c5dac004a762c0e4a40b8d98a132f67f370da7445"},{"start":43978,"end":46209,"start_line":822,"end_line":863,"raw_tex":"\\begin{proof}\nBy virtue of Lemma \\ref{l:conditional_subgradient}, the $\\beta$-component of $\\varGamma(\\beta,\\theta)$ equals\n\\begin{multline*}\n -\\Eb \\big[ \\hat{\\partial}_\\beta F(X,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big)\\big] =\n -\\Eb\\Big\\{ \\Eb\\big[ \\hat{\\partial}_\\beta f(X,Y,\\beta)\\big|X\\big] \\nabla g\\big(\\varPsi(X,\\theta)\\big)\\Big\\}\\\\\n% - \\hat{\\partial}_\\beta  \\Eb \\big[ F(X,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big)\\big]\\\\\n% =\n% - \\hat{\\partial}_\\beta  \\Eb \\big[ f(X,Y,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big)\\big]\n  =\n - \\Eb \\big[ \\hat{\\partial}_\\beta  f(X,Y,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big)\\big].\n\\end{multline*}\n\\begin{comment}\nThe interchange of the subdifferentiation and the expected value is justified because\nAssumptions (A2) and (A3) imply that:\n\\begin{tightlist}{ii}\n\\item the function\n$\\beta \\mapsto f(X,Y,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big)$ is\nNorkin differentiable; and\n\\item its Lipschitz constant $L_f(X,Y) L_g$ is integrable.\n\\end{tightlist}\n\\vspace{0.5ex}\n\\end{comment}\n\nThe $\\theta$-component of $\\varGamma(\\beta,\\theta)$ can be written as follows:\n\\[\n\\gamma \\,\\Eb \\big[ \\hat{\\partial}_\\theta \\varPsi(X,\\theta)\\big(F(X,\\beta) -\\varPsi(X,\\theta)\\big)\\big]\n= \\gamma \\,\\Eb \\big[ \\hat{\\partial}_\\theta \\varPsi(X,\\theta)\\big(f(X,Y,\\beta) -\\varPsi(X,\\theta)\\big)\\big].\n\\]\nDue to Assumptions (A2)--(A4), for all $(X,Y)$, the multifunction\n\\[\n(\\beta,\\theta) \\mapsto \\widetilde{\\varGamma}(\\beta,\\theta;X,Y) \\triangleq\n\\begin{bmatrix}\n -\\hat{\\partial}_{ \\beta} f(X,Y,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big) \\\\\n\\gamma \\,\\hat{\\partial}_{ \\theta} \\varPsi(X,\\theta)\\big(f(X,Y,\\beta) -\\varPsi(X,\\theta)\\big)\n\\end{bmatrix}\n\\]\nis upper-semicontinuous, and convex- and compact-valued. Furthermore, its $\\beta$-values are\nbounded by and integrable function $L_f(X,Y)L_g$, and its $\\theta$-values are bounded\nby an integrable function $  \\gamma L_{\\varPsi}(X)( C_f +  C_{\\varPsi} + 4  L_{\\varPsi}(X) R^2)$.\nThese two conditions imply that its integral -- the multifunction $\\varGamma(\\Cdot,\\Cdot)$ -- is upper-semicontinuous, and convex- and compact-valued as well. This implication is part of\nthe proof of Theorem \\ref{t:expected-generalized}; see \\cite[Lem. 21.7]{mikhalevich1987nonconvex}.\n\\end{proof}","sha256_utf8":"2ab35f0909ce106ae4158c6ae88d314d8e1bf6df8a632a1648aad7ccb6ee97f3"},{"start":48451,"end":48569,"start_line":916,"end_line":919,"raw_tex":"\\begin{lemma}\n\\label{l:dk-in-Gamma}\nFor all $k=0,1,2,\\dots$ we have $d^k \\in \\varGamma(\\beta^k,\\theta^k)$.\n\\end{lemma}","sha256_utf8":"aeb9f8fa5022ba588a6625894cb5d9dcec6549b1740bb57e2b3ad6781ed66e76"},{"start":48570,"end":49842,"start_line":920,"end_line":934,"raw_tex":"\\begin{proof}\nSince the observation $(X^k,Y^k)$ is independent of $\\Fc_k$, Lemma \\ref{l:conditional_subgradient}  yields the relations\n\\begin{align*}\n\\Eb[\\tilde{d}_\\beta^k|\\Fc_k]  &\\in -\\Eb \\big[ \\hat{\\partial}_{ \\beta} f(X^k,Y^k,\\beta^k) \\nabla g(\\varPsi(X^k,\\theta^k)) ~\\big|~ \\Fc_k\\big] \\\\\n&= -\\Eb \\Big\\{ \\Eb \\big[ \\hat{\\partial}_{ \\beta} f(X^k,Y^k,\\beta^k)  ~\\big|~ \\beta^k,\\theta^k, X^k \\big] \\nabla g(\\varPsi(X^k,\\theta^k)) ~\\Big|~ \\beta^k,\\theta^k \\Big\\}\\\\\n&= -\\Eb \\big[ \\hat{\\partial}_{ \\beta} F(X^k,\\beta^k) \\nabla g(\\varPsi(X^k,\\theta^k)) ~\\big|~ \\beta^k,\\theta^k \\big]\n= \\varGamma_\\beta (\\beta^k,\\theta^k).\n\\end{align*}\nThe $\\theta$-component of the direction can also be treated by conditioning:\n\\begin{align*}\n\\Eb[\\tilde{d}_\\theta^k|\\Fc_k] &\\in \\gamma \\Eb \\big[ \\hat{\\partial}_{ \\theta} \\varPsi(X^k,\\theta^k)\\big(f(X^k,Y^k,\\beta^k) -\\varPsi(X^k,\\theta^k)\\big)\\big| \\Fc_k\\big]\\\\\n&= \\gamma \\Eb \\Big\\{\\hat{\\partial}_{ \\theta} \\varPsi(X^k,\\theta^k)\\Eb\\big[f(X^k,Y^k,\\beta^k) -\\varPsi(X^k,\\theta^k)\\big| X^k,\\beta^k,\\theta^k\\big]\\Big| \\beta^k,\\theta^k\\Big\\}\\\\\n&= \\gamma \\Eb \\big\\{\\hat{\\partial}_{ \\theta} \\varPsi(X^k,\\theta^k)\\big(F(X^k,\\beta^k) -\\varPsi(X^k,\\theta^k)\\big)\\big| \\beta^k,\\theta^k\\big\\} = \\varGamma_\\theta(\\beta^k,\\theta^k).\n\\end{align*}\n\\end{proof}","sha256_utf8":"8ffd87c30a78e3f8fbf016f9dcb0b288c4adcf20bd2a9e77cd497576d3d66683"},{"start":49925,"end":50184,"start_line":938,"end_line":942,"raw_tex":"\\begin{lemma}\n    \\label{l:e2-bounded}\nUnder Assumptions \\textup{(A1)--(A4)}, the directions $\\{d^k\\}_{k\\ge 0}$ are well-defined and a constant $\\sigma > 0$ exists, such that\n$\\Eb\\big[\\|e^k\\|^2 \\,\\big|\\,\\Fc_k\\big] \\le \\sigma^2$, for all $k \\ge 0$.\n\\end{lemma}","sha256_utf8":"00be6aefc5ef045dc68e577be16ea0b4a79c3ebbb3db09cae6414b2753d6ed29"},{"start":50185,"end":51761,"start_line":943,"end_line":975,"raw_tex":"\\begin{proof}\nFor the $\\beta$-component $e_\\beta^k$ of $e^k$,  we have\n\\[\n\\tilde{d}_\\beta^k = -f_\\beta(X^k,Y^k,\\beta^k)\\nabla g(\\varPsi(X^k,\\theta^k)),\n\\]\nand thus, by Assumptions (A2) and (A3),\n\\[\n\\Eb\\big[ \\|{e}_\\beta^k\\|^2\\big| \\Fc_k ] \\le \\Eb\\big[ \\|\\tilde{d}_\\beta^k\\|^2\\big| \\Fc_k ]\n\\le L_g^2 \\Eb \\big[ \\| f_\\beta(X^k,Y^k,\\beta^k)\\|^2 \\big| \\Fc_k \\big]\\le\n L_g^2 \\widebar{L}_f^2.\n\\]\nFor the $\\theta$-component,\n\\begin{align*}\n\\Eb\\big[ \\| \\tilde{d}_\\theta^k\\|^2\\big| \\Fc_k\\big]\n&\\le \\gamma^2 \\Eb\\big[ \\|\\varPsi_\\theta(X^k,\\theta^k)\\|^2 \\cdot \\| f(X^k,Y^k,\\beta^k)  - \\varPsi(X^k,\\theta^k)\\|^2\\big| \\Fc_k\\big]\\\\\n&\\le \\gamma^2 \\Eb\\big[{L_\\psi}(X^k)^2\\| f(X^k,Y^k,\\beta^k)  - \\varPsi(X^k,\\theta^k)\\|^2\\big| \\Fc_k\\big]\\\\\n&\\le 2\\gamma^2 \\Eb\\big[{L_\\psi}(X^k)^2\\| f(X^k,Y^k,\\beta^k)\\|^2\\big| \\Fc_k\\big]\\\\\n&{\\quad} +2 \\gamma^2 \\Eb\\big[ {L_\\psi}(X^k)^2\\|\\varPsi(X^k,\\theta^k)\\|^2\\big| \\Fc_k\\big].\n\\end{align*}\nUsing the Cauchy--Schwartz inequality and Assumptions (A3) and (A4) we obtain\n\\[\n\\Eb\\big[{L_\\psi}(X^k)^2\\| f(X^k,Y^k,\\beta^k)\\|^2\\big| \\Fc_k\\big]\n\\le  \\widebar{L}_\\varPsi^2 C_f^2.\n\\]\nIn a similar way, by (A4),\n\\begin{align*}\n\\lefteqn{\\Eb\\big[ {L_\\psi}(X^k)^2\\|\\varPsi(X^k,\\theta^k)\\|^2\\big| \\Fc_k\\big]}\\\\\n&\\le 2 \\Eb\\big[{L_\\psi}(X^k)^2\\big( \\|\\varPsi(X^k,\\theta^0)\\|^2 + {L_\\psi}(X^k)^2 \\|\\theta^k-\\theta^0\\|^2\\big)\\big| \\Fc_k\\big]\n\\le 2 \\widebar{L}_\\varPsi^2 C_\\varPsi^2 + 8R^2 \\widebar{L}_\\varPsi^4.\n\\end{align*}\nCombining the last three inequalities, we conclude that\n$\\Eb\\big[ \\| e_\\theta^k\\|^2\\big| \\Fc_k\\big] $, $k\\ge 0$, are uniformly bounded as well.\n\\end{proof}","sha256_utf8":"24b05329aed11de37cb65a22a76ca9c06618336ef518ff27ac6f79db2bbb50c3"},{"start":53380,"end":54025,"start_line":1007,"end_line":1018,"raw_tex":"The following theorem follows directly from \\cite[Thm. 4.1]{majewski2018analysis}.\n\\begin{theorem}\n\\label{t:diff-inc-beta}\nUnder Assumptions \\textup{(A1)--(A4)} and \\textup{(A7)},\nthe sequence $\\big\\{Z_m(\\Cdot)\\big\\}_{m\\ge 0}$ has a subsequence convergent a.s. to an absolutely continuous function $Z_\\infty:[0,\\infty) \\to \\Xc$ which is a solution of the differential inclusion:\n\\begin{equation}\n\\label{diff-inc-beta}\n\\dt{Z}_\\infty(t) \\in  \\varGamma(Z_{\\infty}(t))-\\Nc_{\\Zb}(Z_\\infty(t)), \\quad t \\ge 0.\n\\end{equation}\nMoreover, for any $t\\ge 0$, the point $Z_\\infty(t)$\nis an accumulation point of the sequence $\\{z^k\\}_{k\\ge 0}$.\n\\end{theorem}","sha256_utf8":"7d8f636d906c4955fcc7fb64e3cf39a25b8622c3450c8318e1e56720609b5ecd"},{"start":54026,"end":54497,"start_line":1019,"end_line":1025,"raw_tex":"Since any solution of \\eqref{diff-inc-beta} must lie in $\\Zb$,\nwe can  write the differential inclusion \\eqref{diff-inc-beta} in an equivalent way \\cite{aubin2012differential}:\n\\begin{equation}\n\\label{diff-inc-beta-T}\n\\dt{Z}_\\infty(t) \\in \\varPi_{\\Tc_{\\Zb}(Z_\\infty(t))}\\big( \\varGamma(Z_\\infty(t))\\big), \\quad t \\ge 0,\n\\end{equation}\nwhich underlines the nature of the limiting subgradient process. Here, $\\Tc_\\Zb(z)$ is the tangent cone to the convex set $\\Zb$ at $z$.\n","sha256_utf8":"73ff38aa47c9a615318045982c24b635990b7c0f37437d5e026810c6741f2be2"},{"start":55647,"end":55812,"start_line":1052,"end_line":1055,"raw_tex":"\\begin{lemma}\n\\label{l:w-gendif}\nUnder Assumptions \\textup{(A2)--(A4)} the Lyapunov function $w(\\Cdot,\\Cdot)$ is differentiable in the generalized sense.\n\\end{lemma}","sha256_utf8":"08ef27d2ab1d55ac7ddd785c7b543abac0d4bb5e69e5e5b6b9d5e49d95a6d89f"},{"start":55813,"end":57406,"start_line":1056,"end_line":1086,"raw_tex":"\\begin{proof}\nIn \\S \\ref{s:2} we established the generalized differentiability of the functions $G(\\Cdot)$ and $Q(\\Cdot,\\Cdot)$.  It remains to consider the function\n\\[\n\\Delta^{0}(\\beta,\\theta) = \\Eb\\Big[g(F) - g(\\varPsi) - \\langle \\nabla g(\\varPsi), F - \\varPsi \\rangle \\Big],\n\\]\nand the expression under the expected value:\n\\begin{equation}\n    \\label{delta0-beta}\n\\delta^{0}(\\beta,\\theta) = g(F) - g(\\varPsi) - \\langle \\nabla g(\\varPsi), F - \\varPsi \\rangle.\n\\end{equation}\nBy Assumptions (A2)--(A4) and Theorem \\ref{t:composition},\n$\\delta^{0}(\\Cdot,\\Cdot)$ is Norkin differentiable. Its generalized subgradients with respect to $\\beta$ have the form\n\\[\n\\delta^{0}_\\beta = F_\\beta\\big( \\nabla g(F) - \\nabla g(\\Psi)\\big),\n\\]\nand are bounded by $2L_g\\Eb[\\widebar{L}_f(X,Y)|X]$. It is an integrable function,\nthanks to (A2).\nIn a similar way, by Assumptions (A2) and (A4) and Theorem \\ref{t:composition}, the generalized subgradients\nwith respect to $\\theta$ have the form\n\\begin{equation}\n    \\label{delta0-theta}\n\\delta^{0}_\\theta = \\varPsi_\\theta H(\\varPsi) ( F - \\Psi),\n\\end{equation}\n or are convex combinations of elements of this structure,\n where we denote by $H(\\varPsi)$\nan arbitrary element of $\\hat{\\partial}(\\nabla g)(\\varPsi)$. Thus, their norm is\nbounded by $L_{\\nabla g}L_\\varPsi(X)\\|F-\\varPsi\\|$. As $L_\\varPsi$\nand $\\|F-\\varPsi\\|$ are square-integrable, the bound is integrable, by virtue\nof the Cauchy--Schwartz inequality.  Applying Theorem \\ref{t:expected-generalized},\nwe conclude that the expected value function $\\Delta^0$ is Norkin differentiable as well.\n\\end{proof}","sha256_utf8":"078b9a7322db8444b37522db21ca725a20b1b345e8086cf86e0c9b1bab029060"},{"start":57820,"end":58667,"start_line":1103,"end_line":1114,"raw_tex":"Following the differential inclusion method, consider the function $W(t) = w(\\beta(t),\\theta(t))$, $t \\ge 0$, where $(\\beta(\\Cdot),\\theta(\\Cdot)) \\triangleq Z(\\Cdot)$ is a solution of the\ninclusion \\eqref{diff-inc-beta}.\n\nAs the Lyapunov function $w(\\Cdot,\\Cdot)$ is Norkin differentiable, by virtue of Theorem \\ref{t:composition}, its\nweak derivative can be expressed by the chain rule:\n\\begin{equation}\n\\dt{W}\n= \\langle w_\\beta, \\dt{\\beta} \\rangle + \\langle w_\\theta, \\dt{\\theta} \\rangle\n= \\langle G_\\beta + \\alpha \\Delta^\\lambda_\\beta, \\dt{\\beta} \\rangle + \\alpha \\langle \\Delta^\\lambda_\\theta, \\dt{\\theta} \\rangle.\n\\label{grad:P}\n\\end{equation}\nDue to Theorem \\ref{t:chain}, the integral  $\\int_0^T \\dt{W}(t)\\,\\D t$ on any interval of time does not depend on the particular selections of the generalized subgradients used in this calculation.\n","sha256_utf8":"df23a6431335e7f7b81d37bbd73305aafbe89030ecbe8aef18a7952440b6578d"},{"start":59479,"end":59682,"start_line":1134,"end_line":1137,"raw_tex":"\\begin{lemma}\nUnder Assumptions \\textup{(A2)} and \\textup{(A4)},  $\\big|\\langle \\Delta^0_\\theta, Q_\\theta \\rangle \\big| \\le 2 {L_{\\nabla g}} \\widebar{L}_\\varPsi^2 Q$.\n\\label{l:QG_delta_theta}\n\\end{lemma}","sha256_utf8":"671f7aff5a19a2b2685abfdb88508533df881a13ac4adbf6fce806a342244014"},{"start":59683,"end":60297,"start_line":1138,"end_line":1149,"raw_tex":"\\begin{proof}\nIn view of \\eqref{delta0-theta},  $\\Delta^0_\\theta = \\Eb\\big[ \\varPsi_\\theta H(\\varPsi)(F-\\varPsi)\\big]$. By virtue of Assumption (A2), $\\|H(\\varPsi)\\| \\le L_{\\nabla g}$.\nAlso,\n$Q_\\theta = \\Eb\\big[ \\varPsi_\\theta(F-\\varPsi)\\big]$.\nUsing the Cauchy--Schwartz inequality,\nwe obtain the following chain of estimates:\n\\begin{align*}\n\\big|\\langle \\Delta^0_\\theta, Q_\\theta \\rangle\\big|  &\\le \\| \\Delta^0_\\theta\\| \\| Q_\\theta \\|\n\\le {L_{\\nabla g}} \\Eb[\\|\\varPsi_\\theta\\|^2] \\Eb[\\|F - \\varPsi\\|^2] \\le 2 {L_{\\nabla g}} \\widebar{L}_\\varPsi^2 Q.\n\\end{align*}\nThe last inequality uses \\eqref{te:Q}.\n\\end{proof}","sha256_utf8":"ccc68250f0e505c53e6765feeff2e7441855ab16f47ba5cb72eb0f5bfa03191c"},{"start":60299,"end":60418,"start_line":1151,"end_line":1154,"raw_tex":"\\begin{lemma}\nUnder Assumption \\textup{(A3)},  $\\|Q_\\beta\\|^2 \\le 2 \\widebar{L}_f^2 Q$.\n\\label{l:QG_Q_beta}\n\\end{lemma}","sha256_utf8":"88a4ad109470538c67693c19df381be449463c7a398eff556f528806f597182e"},{"start":60419,"end":60628,"start_line":1155,"end_line":1161,"raw_tex":"\\begin{proof}\nBy the  Cauchy--Schwartz inequality,\n\n\\[\n\\|Q_\\beta\\| = \\big\\|\\Eb[ F_\\beta(F-\\varPsi)]\\big\\| \\le \\sqrt{\\Eb[\\|F_\\beta\\|^2]}\\sqrt{ \\Eb[\\|F-\\varPsi\\|^2]} \\le  \\widebar{L}_f \\sqrt{2 Q}.\n\\]\n\\end{proof}","sha256_utf8":"008f851cdcbca36d54653c002f77877784a81e869342695fdc607432a93e541b"},{"start":60629,"end":60792,"start_line":1162,"end_line":1165,"raw_tex":"\\begin{lemma}\nUnder Assumptions \\textup{(A2)} and \\textup{(A3)}, $\\|\\Delta^0_\\beta\\|^2 \\le 2 L_{\\nabla g}^2 \\widebar{L}_f^2 Q$.\n\\label{l:QG_delta_beta}\n\\end{lemma}","sha256_utf8":"bebb85206d956449996e73249a408c73802fdd11af2c240226cfc52422db87d6"},{"start":60793,"end":61373,"start_line":1166,"end_line":1176,"raw_tex":"\\begin{proof}\nBy \\eqref{delta0-beta}, $\\Delta^0_\\beta = \\Eb[F_\\beta(\\nabla g(F) - \\nabla g(\\varPsi))$.\nSimilar to the proof of Lemma \\ref{l:QG_Q_beta}, by the Cauchy--Schwartz inequality and Assumption (A2),\n\\begin{align*}\n\\|\\Delta^0_\\beta\\|^2 &= \\big\\|\\Eb[F_\\beta(\\nabla g(F) - \\nabla g(\\varPsi)) ]\\big\\|^2\n\\le \\Eb[\\|F_\\beta\\|^2]\\, \\Eb[\\|\\nabla g(F) - \\nabla g(\\varPsi)\\|^2]  \\\\\n&\\le L_{\\nabla g}^2 \\Eb[\\|F_\\beta\\|^2]\\, \\Eb[\\|F - \\varPsi\\|^2]\n\\le 2L_{\\nabla g}^2 \\widebar{L}_f^2 Q.\n\\end{align*}\nThe last inequality uses Assumption (A3), as in Lemma \\ref{l:QG_Q_beta}.\n\\end{proof}","sha256_utf8":"3b5c852d77f7dfea84c9829ab778702de1f9b23e1a148cf3d21e143f0522a44d"},{"start":61796,"end":62178,"start_line":1188,"end_line":1198,"raw_tex":"\\begin{proposition}\nUnder assumptions \\textup{(A2)} and \\textup{(A3)},\n\\[\n\\langle w_\\beta, \\dt{\\beta} \\rangle \\le \\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 Q.\n\\]\nFurthermore, if $Q=0$, then\n\\[\n\\langle w_\\beta, \\dt{\\beta} \\rangle = - \\big\\| \\varPi_{\\Tc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big)\\big\\|^2.\n\\]\n\\label{p:QG_L_delta_Q_beta}\n\\end{proposition}","sha256_utf8":"1c3f429179ccf1800241b2573cbc3e40b20a08280ca8e110c3a08dcb5daa6b5f"},{"start":62179,"end":66281,"start_line":1199,"end_line":1255,"raw_tex":"\\begin{proof}\nThe first coordinate of the differential inclusion \\eqref{diff-inc-beta} reads:\n\\begin{equation}\n\\dt{\\beta} \\in \\varPi_{\\Tc_{\\Bb}(\\beta)}\\big(-\\Eb[F_\\beta \\nabla g(\\varPsi) ] \\big) = \\varPi_{\\Tc_{\\Bb}(\\beta)}\\big( -G_\\beta + \\Delta^{0}_\\beta \\big),\n\\label{grad:beta}\n\\end{equation}\nFor any solution $(\\beta(t), \\theta(t))$ of the inclusion \\eqref{t:diff-inc-beta}, with the corresponding element $N(t) \\in \\Nc_{\\Bb}(\\beta(t))$, we define the projection matrix\n\\[\nP(t)  = I - \\frac{N(t) N(t)^T}{\\|N(t)\\|^2}.\n\\]\nThen\n\\begin{equation}\n\\dt{\\beta} = P (-G_\\beta + \\Delta^{0}_\\beta).\n\\label{grad:beta_proj}\n\\end{equation}\nUsing the facts that $P$ is nonexpansive, $P^T=P$, and $P^2=P$,\nby elementary manipulations, we obtain an upper bound:\n\\begin{equation}\n\\label{w-dot-chain}\n\\begin{aligned}\n\\lefteqn{\\langle w_\\beta, \\dt{\\beta} \\rangle = -\\langle G_\\beta + \\alpha \\Delta^{\\lambda}_\\beta, P(G_\\beta - \\Delta^{0}_\\beta) \\rangle} \\\\\n&= -\\Big\\|P\\big(G_\\beta + \\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta - \\frac{1}{2} \\Delta^0_\\beta\\big)\\Big\\|^2 + \\Big\\|P\\big(\\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta + \\frac{1}{2} \\Delta^0_\\beta\\big)\\Big\\|^2 \\\\\n&\\le \\Big\\|P\\big(\\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta + \\frac{1}{2} \\Delta^0_\\beta\\big) \\Big\\|^2 =\n\\Big\\|P\\big(\\frac{\\alpha+1}{2} \\Delta^0_\\beta + \\frac{\\alpha \\lambda}{2} Q_\\beta\\big)\\Big\\|^2 \\\\\n&= \\Big\\|\\frac{\\alpha+1}{2} P \\Delta^0_\\beta\\Big\\|^2 + \\Big\\|\\frac{\\alpha \\lambda}{2} P Q_\\beta\\Big\\|^2 + 2 \\Big\\langle \\rho \\frac{\\alpha+1}{2} P \\Delta^0_\\beta, \\varrho^{-1} \\frac{\\alpha \\lambda}{2} P Q_\\beta \\Big\\rangle\\\\\n&\\le \\frac{(\\alpha+1)^2}{4} (1+\\rho^2)\\|P \\Delta^0_\\beta\\|^2 + \\frac{\\alpha^2 \\lambda^2}{4}(1+\\rho^{-2})\\|P Q_\\beta\\|^2,\n\\end{aligned}\n\\end{equation}\nfor any $\\rho>0$.\nLemmas \\ref{l:QG_Q_beta} and \\ref{l:QG_delta_beta} allow us to continue the last estimate as follows:\n\\begin{align*}\n\\langle w_\\beta, \\dt{\\beta} \\rangle &\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{4} (1+\\rho^2)\\|P \\Delta^0_\\beta\\|^2 + \\frac{\\alpha^2 \\lambda^2}{4}(1+\\rho^{-2})\\|P Q_\\beta\\|^2 \\Big] \\\\\n&\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{4} (1+\\rho^2)\\|\\Delta^0_\\beta\\|^2 + \\frac{\\alpha^2 \\lambda^2}{4}(1+\\rho^{-2})\\|Q_\\beta\\|^2 \\Big] \\\\\n&\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{2} (1+\\rho^2) L_{\\nabla g}^2 + \\frac{\\alpha^2 \\lambda^2}{2}(1+\\rho^{-2}) \\Big]   \\widebar{L}_f^2 Q \\\\\n&= \\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 Q.\n\\end{align*}\nThis proves the first assertion.\n\\begin{comment}\n\\begin{align*}\n\\langle w_\\beta, \\dt{\\beta} \\rangle &= -\\langle G_\\beta + \\alpha \\Delta^{\\lambda}_\\beta, P(G_\\beta - \\Delta^{0}_\\beta) \\rangle \\\\\n&= -\\Big\\|G_\\beta + \\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta - \\frac{1}{2} \\Delta^0_\\beta\\Big\\|_P^2 + \\Big\\|\\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta + \\frac{1}{2} \\Delta^0_\\beta\\Big\\|_P^2 \\\\\n&\\le \\Big\\|\\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta + \\frac{1}{2} \\Delta^0_\\beta\\Big\\|_P^2 =\n\\Big\\|\\frac{\\alpha+1}{2} \\Delta^0_\\beta + \\frac{\\alpha \\lambda}{2} Q_\\beta\\Big\\|_P^2 \\\\\n&= \\Big\\|\\frac{\\alpha+1}{2} \\Delta^0_\\beta\\Big\\|_P^2 + \\Big\\|\\frac{\\alpha \\lambda}{2} Q_\\beta\\Big\\|_P^2 + 2 \\Big\\langle \\rho \\frac{\\alpha+1}{2} \\Delta^0_\\beta, \\varrho^{-1} \\frac{\\alpha \\lambda}{2} P Q_\\beta \\Big\\rangle \\\\\n&\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{4} (1+\\rho^2)\\|\\Delta^0_\\beta\\|_P^2 + \\frac{\\alpha^2 \\lambda^2}{4}(1+\\rho^{-2})\\|Q_\\beta\\|_P^2 \\Big] \\\\\n&\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{4} (1+\\rho^2)\\|\\Delta^0_\\beta\\|^2 + \\frac{\\alpha^2 \\lambda^2}{4}(1+\\rho^{-2})\\|Q_\\beta\\|^2 \\Big] \\\\\n&\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{2} (1+\\rho^2) {L_{\\nabla g}}^2 + \\frac{\\alpha^2 \\lambda^2}{2}(1+\\rho^{-2}) \\Big]   \\widebar{L}_f^2 M \\|Q_\\theta\\|^2 \\\\\n&= \\frac{\\widebar{L}_f^2 M}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 \\|Q_\\theta\\|^2.\n\\end{align*}\n\\end{comment}\nNow, suppose $Q=0$. This implies that $\\varDelta^\\lambda=0$ for all $\\lambda \\ge 0$.\nThe first line of \\eqref{w-dot-chain} yields\n\\[\n\\langle w_\\beta, \\dt{\\beta} \\rangle = - \\| P G_\\beta \\|^2 = - \\big\\| \\varPi_{\\Tc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big)\\big\\|^2,\n\\]\nas claimed.\n\\end{proof}","sha256_utf8":"b9002b79c181a96ac70f19ad25513cb942852163824b4c4edc6887533baac816"},{"start":67462,"end":67890,"start_line":1273,"end_line":1279,"raw_tex":"The second coordinate of the differential inclusion \\eqref{diff-inc-beta} reads:\n\\begin{equation}\n\\dt{\\theta} =  \\varPi_{\\Tc_{\\Theta_R}(\\theta)}\\big(\\gamma\\, \\Eb[(F - \\varPsi)^T \\varPsi_\\theta] \\big)\n= \\varPi_{\\Tc_{\\Theta_R}(\\theta)}\\big( -\\gamma Q_\\theta\\big) = -\\gamma Q_\\theta,\n\\label{grad:simp}\n\\end{equation}\nbecause, by virtue of  Assumption (A6), when $\\|\\theta\\|=R$, then $ -\\gamma Q_\\theta \\in \\Tc_{\\Theta_R}(\\theta)$.\n","sha256_utf8":"b842566d0f073afb76e2721c12a6ca0160ab1356332357270e426b1af0ad734b"},{"start":67892,"end":68358,"start_line":1282,"end_line":1291,"raw_tex":"\\begin{theorem}\n\\label{t:negative-der}\nUnder Assumptions \\textup{(A1)--(A6)}, if\n\\begin{equation}\n\\label{negative-der}\n\\lambda > 2 \\widebar{L}_\\varPsi^2 M {L_{\\nabla g}} \\quad \\text{and} \\quad\n\\gamma > \\frac{\\widebar{L}_f^2 M\\big( (\\alpha + 1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2}{2 \\alpha \\big(\\lambda - 2 \\widebar{L}_\\varPsi^2 M {L_{\\nabla g}} \\big)} ,\n\\end{equation}\nthen $\\dt{W} \\le 0$. Furthermore, $\\dt{W} < 0$ if $(\\beta,\\theta)\\notin \\Zb^*$.\n\\end{theorem}","sha256_utf8":"bd928770065033eb15d84a5377fdc52bc5401a701046a2317402e508c5e60e20"},{"start":68359,"end":70049,"start_line":1292,"end_line":1314,"raw_tex":"\\begin{proof}\nFollowing \\eqref{grad:P}, \\eqref{grad:simp}, Lemma \\ref{l:QG_delta_theta}, and Proposition \\ref{p:QG_L_delta_Q_beta},\n\\begin{align*}\n\\dt{W}\n&\\le \\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 Q - \\alpha \\gamma \\langle \\Delta^{\\lambda}_\\theta, Q_\\theta \\rangle \\\\\n&= \\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 Q - \\alpha \\gamma \\big[\\langle \\Delta^0_\\theta, Q_\\theta \\rangle + \\lambda \\|Q_\\theta\\|^2 \\big] \\\\\n&\\le \\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 Q + \\alpha \\gamma |\\langle \\Delta^0_\\theta, Q_\\theta \\rangle| - \\alpha \\gamma \\lambda \\|Q_\\theta\\|^2 \\\\\n&\\le \\Big[\\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 + \\alpha \\gamma \\cdot 2{L_{\\nabla g}} \\widebar{L}_\\varPsi^2 - \\frac{\\alpha \\gamma \\lambda}{M} \\Big] Q \\\\\n&= \\frac{1}{M} \\Big[\\frac{\\widebar{L}_f^2 M}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 - \\alpha \\gamma \\big(\\lambda  - 2 \\widebar{L}_\\varPsi^2 M{L_{\\nabla g}} \\big) \\Big] Q \\le 0.\n\\end{align*}\nThis verifies the first assertion.\nNow, suppose $\\dt{W}=0$. It follows from the last inequality that $Q=0$. Then, Proposition \\ref{p:QG_L_delta_Q_beta} implies that\n\\[\n\\varPi_{\\Tc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big) = 0\n\\]\nfor some $G_\\beta \\in \\hat{\\partial}{G}(\\beta)$. Consequently, using the decomposition into the projections on polar cones,\nwe obtain\n\\[\n-G_\\beta = \\varPi_{\\Tc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big) + \\varPi_{\\Nc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big) =\n\\varPi_{\\Nc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big).\n\\]\nThis means that the optimality condition \\eqref{optimality} is satisfied.\n\\end{proof}","sha256_utf8":"db93443522da7f4802de6ee8e0598df18c0c149631f1db142bd17c5454a99442"},{"start":72919,"end":76318,"start_line":1358,"end_line":1414,"raw_tex":"\\begin{proof}\nThe proof follows the steps of the proofs of \\cite[Thm. 3.20]{duchi2018stochastic} or \\cite[Thm. 3.5]{majewski2018analysis} {  or \\cite{nurminskii1972convergence}}.\nWe provide a short outline here, for the convenience of the Reader. For brevity, we omit the\nargument $\\omega$, but we stress that we consider a particular path of the method.\n\nDefine\n\\[\n\\underline{w} = \\liminf_{k \\to \\infty^{\\phantom{|}}} w(z^k), \\qquad \\overline{w} = \\limsup_{k \\to \\infty}w(z^k).\n\\]\nSuppose $\\underline{w} < \\overline{w}$. By Assumption (A7), a value ${v}\\in\n(\\underline{w},\\overline{w})$ exists,\nsuch that ${v} \\notin G(\\Bb^*)$. Take any $u \\in ({v},\\overline{w})$.\nDefine $L_1=\\min\\big\\{k : w(z^k) \\le {v} \\big\\}$ and for\n$j=1,2,\\dots$\n\\begin{align*}\n\\ell_j &= \\min\\big\\{k\\ge L_j : w(z^k) > u \\big\\},\\\\\nL_{j+1} &= \\min\\big\\{k\\ge \\ell_j : w(z^k) \\le {v} \\big\\}.\\\\\n\\intertext{Then we define}\nk_j &= \\max\\big\\{ k\\in [L_j,\\ell_j]: w(z^k) \\le {v} \\big\\}, \\quad j=1,2,\\dots.\n\\end{align*}\nBoth $k_j\\to \\infty$ and $\\ell_j\\to \\infty$, as $j\\to \\infty$.\n\nConsider the sums\n$T_j = \\sum_{k=k_j}^{\\ell_j-1} \\tau_k$. Due to the equicontinuity of the functions\n $w(Z_j(\\Cdot))$, a constant $T >0$ exists, such that $T_j\\ge T$ for all sufficiently\nlarge $j$. Set $s_j=t_{k_j}$ in \\eqref{Z-shifted} and consider the interval $[0,T]$.\nBy Theorem \\ref{t:diff-inc-beta}, a subsequence of the sequence $\\{Z_j(\\Cdot)\\}$ is convergent to a solution $Z_\\infty(\\Cdot)$ of the differential\ninclusion \\eqref{diff-inc-beta}. For simplicity, we still denote it by $\\{Z_j(\\Cdot)\\}$. As $W(\\Cdot)$ is absolutely continuous,\n\\begin{equation}\n\\label{gradient-desc-cont}\nw(Z_\\infty(T)) = w(Z_\\infty(0)) + \\int_0^T  \\dt{W}(t)  \\;\\D t.\n\\end{equation}\nSince ${v} \\notin w(\\Zb^*)$, then  $Z_\\infty(0) = \\lim_{j\\to \\infty} z^{k_j} \\notin \\Zb^*$.\nTheorem \\ref{t:negative-der} then implies that $\\dt{W}(0)<0$.\nThe generalized subdifferential $\\hat{\\partial} G(\\Cdot)$ and the normal cone $\\Nc_\\Bb(\\Cdot)$ are upper semicontinuous and closed-valued. The function $Q(\\Cdot,\\Cdot)$ is continuous.\nTherefore, $\\dt{W}(t)<0$\n in some interval $t\\in [0,\\bar{t}\\,]$ of positive length.\n\n It follows from \\eqref{gradient-desc-cont}\n that $w(Z_\\infty(T)) < w(Z_\\infty(0)) =v$.\n However, $Z_\\infty(T)$ is the limit of the sequence $\\{z^{m_j}\\}$, where\n\\begin{equation}\n\\label{mj}\nm_j = \\min\\bigg\\{m \\ge k_j: \\sum_{k=k_j}^{m_j-1}\\tau_k \\ge T\\bigg\\}, \\quad j=1,2,\\dots.\n\\end{equation}\nFor all sufficiently large $j$ we  have $k_j \\le m_j \\le \\ell_j$ and $w(z^{m_j}) > v$, which is a contradiction. Therefore,\n$\\underline{w} = \\overline{w}$ and the sequence $\\big\\{w(z^k)\\big\\}$ has a limit.\n\nSuppose a subsequence $\\{z^{k_j}\\}$ is convergent to a point $\\bar{z}\\notin \\Zb^*$. Again, we\nset $s_j = t_{k_j}$ in \\eqref{Z-shifted}, consider the interval $[0,T]$ with $T>0$, and conclude that\na  sub-subsequence of $\\{Z_j(\\Cdot)\\}$ exists, which is uniformly convergent to a solution $Z_\\infty(\\Cdot)$ of the\ndifferential inclusion \\eqref{diff-inc-beta}. Then the points $\\{z^{m_j}\\}$ are convergent to $Z_\\infty(T)$ on this sub-subsequence,\nwhere $m_j$ are defined as in \\eqref{mj}.\nAs $Z_\\infty(0)=\\bar{z}\\notin \\Zb^*$, we obtain $Z_\\infty(T)< Z_\\infty(0)$. This would mean that a sub-subsequence\nof $\\big\\{w(z^{m_j})\\big\\}$ has the limit $w(Z_\\infty(T))< w(\\bar{z})$, which contradicts the convergence of the sequence\n$\\big\\{w(z^{k})\\big\\}$.\n\\end{proof}","sha256_utf8":"a2e1b1d2b41c974824392330d46559882c366c3316cc7c3b34243b64fd4e912e"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"随机算法几乎必然收敛","source_units":[{"start":72548,"end":72918,"start_line":1353,"end_line":1357,"raw_tex":"\\begin{theorem}\n    \\label{t:full-convergence}\n    Assume conditions {\\rm (A1)--(A8)} and \\eqref{negative-der}. Then, except for a zero measure set $\\varOmega_0\\subset \\varOmega$,  the sequence $\\big\\{w(z^k(\\omega))\\big\\}_{k\\in \\Nb}$ is convergent, and every accumulation point of the sequence\n$\\big\\{z^k(\\omega)\\big\\}_{k\\in \\Nb}$ is an element of $\\Zb^*$.\n\\end{theorem}","sha256_utf8":"9991c21c97ac42485dc6daf9c69663c552fb593644eebbb6d260229ecff35506"}]},{"id":"S002","semantic_type":"problem_setting","kind":"problem_setting","title":"条件随机优化问题","source_units":[{"start":7231,"end":7458,"start_line":203,"end_line":207,"raw_tex":"The \\emph{Conditional Stochastic Optimization} (CSO) problem is formulated as follows:\n\\begin{equation}\n\\min_{\\beta \\in {\\Bb}} G(\\beta) \\triangleq \\Eb \\Big\\{g\\big(\\Eb [f(X,Y,\\beta) \\,|\\, X]\\big)\\Big\\}\n\\label{cso}\n\\end{equation}","sha256_utf8":"c0bfb8638afdf480c204ec0a3607ae073b545b67093cabe8ce4dbf3a348eec1c"},{"start":7459,"end":7776,"start_line":208,"end_line":208,"raw_tex":"where $\\Bb \\subset \\Rb^{n_\\beta}$, $f: \\Rb^{n_X} \\times \\Rb^{n_Y} \\times \\Rb^{n_\\beta} \\to \\Rb^{n_f}$ , and $g: \\Rb^{n_f} \\to \\Rb$. In formula \\eqref{cso}, $X$ and $Y$ are random variables, and the symbols $\\Eb[\\Cdot]$ and $\\Eb[\\Cdot|X]$ represent the expected value and the conditional expected value, respectively.\n","sha256_utf8":"8714a44553757ce9c75010d6483d4c22bf8d8a3732b1cc5e6ae7ed5d51c013e1"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"条件期望与精确模型","source_units":[{"start":18317,"end":18414,"start_line":358,"end_line":361,"raw_tex":"\\begin{equation}\n    \\label{def:F}\nF(X, \\beta) \\triangleq \\Eb[f(X, Y, \\beta) | X].\n\\end{equation}","sha256_utf8":"decfbf1d675e65708be8df2453f9746e83e77cf97f822eb3bf96f91ee9afc3a7"},{"start":18415,"end":18805,"start_line":362,"end_line":368,"raw_tex":"We assume that a sufficiently rich class of functions $\\varPsi: \\Rb^{n_x} \\times \\Rb^{n_\\theta} \\to \\Rb^{n_f}$ has been established,\nsuch that for every\n$\\beta\\in {\\Bb}$ an (unknown) value of the parameters $\\bar{\\theta}(\\beta) \\in \\Rb^{n_\\theta}$ exists for which\n\\begin{equation}\n\\label{tracking-exact}\nF(X,\\beta) = \\varPsi\\big(X,\\bar{\\theta}(\\beta)\\big)  \\quad \\text{a.s.}\n\\end{equation}","sha256_utf8":"8bf63ee8d77daea463adfbfdb391b0986bcc13a81c82d4d5b9f86d94d2ecec55"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"均方跟踪误差","source_units":[{"start":20203,"end":20340,"start_line":390,"end_line":394,"raw_tex":"\\begin{equation}\nQ(\\beta, \\theta) \\triangleq \\frac{1}{2}\n\\Eb\\big[\\|F(X, \\beta) - \\varPsi(X, \\theta)\\|^2\\big],\n\\label{te:Q}\n\\end{equation}","sha256_utf8":"2b2b8c8ec44c914cf392a385d19060cdfa799021637ff14a025c4b4473c3cd24"}]},{"id":"S005","semantic_type":"notation","kind":"notation","title":"导数及光滑性记号","source_units":[{"start":23456,"end":24847,"start_line":440,"end_line":448,"raw_tex":"Below we review some notation and conventions used throughout the paper. For simplicity, we ignore all the arguments of the functions,\nif they are obvious from the context. We also indicate generalized derivatives in the subscript of a function symbol,\nthat is, $F_x(\\Cdot) \\in \\partial F(\\Cdot)$.  We only use the $l_2$-norm in this paper, which is denoted as $\\|\\Cdot\\|$. A dot\nover a symbol, for example, $\\dt{F}$, represents the generalized derivative of $F(\\beta(t))$ with respect to the\ntime variable, where $\\beta(t)$ is an absolutely continuous path (usually, evident from the context).\nA function $g: \\Rb^n \\to \\Rb^m$ is \\emph{$L$-Lipschitz continuous} if $L > 0$ exists  such that $\\|g(x) - g(y)\\| \\le L \\|x-y\\|$ for all $x, y \\in \\Rb^n$. A differentiable function $g: \\Rb^n \\to \\Rb^m$ is called \\emph{$L$-Lipschitz smooth} if the vector function $\\nabla g(\\Cdot)$ is \\emph{$L$-Lipschitz continuous}. This implies that $\\|g(x) - g(y) - \\nabla g(y)^T (x - y)\\| \\le \\frac{L}{2} \\|x-y\\|^2$ for all $x, y \\in \\Xc$. A differentiable function $g: \\Xc \\to \\Rb$ is\n\\emph{strongly convex with modulus $L>0$} if $g(x) - g(y) - \\nabla g(y)^T (x - y) \\ge \\frac{L}{2} \\|x-y\\|^2$ for all $x, y \\in \\Xc$. It is \\emph{weakly convex with modulus $L$} if\n$g(x) - g(y) - \\nabla g(y)^T (x - y) \\ge -\\frac{L}{2} \\|x-y\\|^2$ for all $x, y \\in \\Xc$.\nAn $L$-Lipschitz smooth function is $L$-weakly convex.\n","sha256_utf8":"e21205f924d6fb8fd9802fe665752f5c670cd29d951fdb7dfecf6d743f78f9cc"}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","title":"紧凸决策域","source_units":[{"start":24990,"end":25059,"start_line":455,"end_line":456,"raw_tex":"\\begin{description}\n\\item[(A1)] The set $\\Bb$ is convex and compact.\n","sha256_utf8":"ed411f000e0c693b2f952166179397198d7e47068feba66a4b337a84aaced588"}]},{"id":"S007","semantic_type":"assumption","kind":"assumption","title":"外层函数光滑性","source_units":[{"start":25059,"end":25362,"start_line":457,"end_line":458,"raw_tex":"\\item[(A2)] The function $g(\\Cdot)$ is continuously differentiable, its gradient is differentiable in the generalized sense, and constants $L_g$ and ${L_{\\nabla g}}$ exist,\nsuch that $\\| \\nabla g(u)\\| \\le L_g$ and $\\| \\nabla g(u) - \\nabla g(v)\\| \\le {L_{\\nabla g}}\\|u -v\\|$ for all $u,v \\in \\Rb^{n_f}$.\n","sha256_utf8":"abb580a47018dbfb2d797de09731142ba465c46137da78715e7a4f6100ef9e24"}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","title":"内层函数可微与矩界","source_units":[{"start":25362,"end":25921,"start_line":459,"end_line":469,"raw_tex":"\\item[(A3)] For every $x \\in \\Rb^{n_x}$ and every $y\\in \\Rb^{n_y}$ the function $f(x,y,\\Cdot)$ is differentiable in the generalized sense\nwith the generalized subdifferential $\\hat{\\partial} f(x,y,\\Cdot)$. Furthermore,  a function ${L_f}(x,y)$ and  constants $\\widebar{L}_f>0$ and $C_f>0$  exist, such that\nfor all $\\beta\\in \\Bb$ and all $(x,y)$\n\\begin{gather*}\n \\sup_{f_\\beta \\in \\hat{\\partial} f(x,y,\\beta)}\\|f_\\beta\\| \\le {L_f}(x,y),\\\\\n\\Eb [ {L_f}(X,Y)^4 ] \\le \\widebar{L}_f^4,\\\\\n\\intertext{and}\n\\Eb\\big[\\|f(X,Y,\\beta)\\|^4 \\big] \\le C_f^4.\n\\end{gather*}\n\n\n","sha256_utf8":"8f0e6d20b8c44a4552150aeab150ca35b1078508082cc4b7acb629c92232d5b7"}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","title":"参数模型可微与矩界","source_units":[{"start":25921,"end":26553,"start_line":470,"end_line":479,"raw_tex":"\\item[(A4)] For every $x \\in \\Rb^{n_x}$ the function $\\varPsi(x,\\Cdot)$ is differentiable in the generalized sense\nwith the generalized subdifferential $\\hat{\\partial} \\varPsi(x,\\Cdot)$. Furthermore, a function ${L_\\psi}(x)$ and  constants\n$\\widebar{L}_\\varPsi>0$ and  $C_{\\varPsi}>0$ exist, such that for all $x$\n\\begin{gather*}\n\\sup_{\\theta \\in \\Rb^{n_\\theta}}\\  \\sup_{\\varPsi_\\theta \\in \\hat{\\partial} \\varPsi(x,\\theta)}\\|\\varPsi_\\theta\\| \\le {L_\\psi}(x),\\\\\n\\Eb [ {L_\\psi}(X)^4 ] \\le \\widebar{L}_\\varPsi^4,\\\\\n\\intertext{and for some $\\theta^0$,}\n\\Eb [ \\|\\varPsi(X,\\theta^0)\\|^4 ] \\le C_\\varPsi^4.\n\\end{gather*}\n\\end{description}\n","sha256_utf8":"cf5dc0219c0c694d0f7c444717d5445d350093d478b5475a77cf779681f8d67d"}]},{"id":"S010","semantic_type":"notation","kind":"notation","title":"条件次微分集合记号","source_units":[{"start":28030,"end":28449,"start_line":510,"end_line":514,"raw_tex":"We can compactly write the assertion of the above lemma as\\footnote{Here and elsewhere, the subdifferentiation is with respect to the parameters $\\beta$ or $\\theta$; we omit the corresponding subscripts of $\\hat{\\partial}$ to alleviate the notation.}\n\\begin{equation}\n\\label{dFbeta}\n\\hat{\\partial}_{  \\beta} F(X,\\beta) = \\Eb \\big[ \\hat{\\partial}_{  \\beta} f(X,Y,\\beta)\\,\\big|\\, X \\big] \\quad \\text{a.s.,}\n\\end{equation}","sha256_utf8":"cb7a190b64848ac077d35c23a826676bf5be4c62e0e268a89094acfb20e47e58"},{"start":28450,"end":28675,"start_line":515,"end_line":516,"raw_tex":"where the conditional expectation on the right-hand side is understood as the collection of conditional expected values of all measurable selections of the multifunction\n$(X,Y) \\mapsto \\hat{\\partial}_{  \\beta} f(X,Y,\\beta)$.\n","sha256_utf8":"23349834b987b5365ff506775553182ab0602b45ac3a81d94c4954197b13e21f"}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","title":"统一误差增长条件","source_units":[{"start":31468,"end":31944,"start_line":582,"end_line":590,"raw_tex":"As mentioned before, for every $\\beta \\in {\\Bb}$, a parameter value $\\bar{\\theta}(\\beta)\\in \\Rb^{n_\\theta}$ exists,\nsuch that $Q\\big(\\beta,\\bar{\\theta}(\\beta)\\big)=0$. In addition to that, we assume the following\ngrowth condition.\n\\begin{description}\n\\item[(A5)] A constant $M>0$ exists such that for all $\\beta\\in {\\Bb}$ and all $\\theta \\in \\Rb^{n_\\theta}$\n\\[\n Q(\\beta,\\theta)  \\le M \\big[\\dist\\big(0,\\hat{\\partial}_{\\theta} Q(\\beta,\\theta)\\big)\\big]^2.\n\\]\n\\end{description}\n","sha256_utf8":"09de087378369525005a9653f4e60005cd7d5fe1b139b0c2ce9ddc0e337c5363"}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","title":"大参数处径向条件","source_units":[{"start":35845,"end":36208,"start_line":654,"end_line":661,"raw_tex":"Finally, we need an assumption about the behavior of the tracking error for very large $\\theta$.\n\n\\begin{description}\n\\item[(A6)] A constant $\\underline{R}>0$ exists such that if $\\beta\\in {\\Bb}$ and  $\\|\\theta\\|\\ge \\underline{R}$, then for all $Q_\\theta \\in\\hat{\\partial}_{\\theta} Q(\\beta,\\theta)  $,\n\\[\n\\langle \\theta, Q_\\theta \\rangle >0.\n\\]\n\\end{description}\n","sha256_utf8":"4380819d1cc06dde8f6ba6c6e6d8caab5e8ee9ab8f16e66761d995cefe1b7d5a"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"驻点与目标极限集合","source_units":[{"start":38690,"end":38872,"start_line":714,"end_line":718,"raw_tex":"The necessary optimality condition  for problem \\eqref{cso} has the form\n\\begin{equation}\n\\label{optimality}\n0 \\in \\hat{\\partial}_{ \\beta} G(\\beta) + \\Nc_{\\Bb}(\\beta),\n\\end{equation}","sha256_utf8":"fd5cf719808ecd2fa4799522328b4e3d545bd2386eb2f0db9d68f8194a4b6556"},{"start":38873,"end":39256,"start_line":719,"end_line":724,"raw_tex":"where $\\hat{\\partial}_{ \\beta} G(\\Cdot)$ is the generalized subdifferential \\eqref{G-sub} and $\\Nc_{\\Bb}(\\Cdot)$ is the normal cone\nto the set $\\Bb$. We define the sets\n\\begin{align*}\n    {\\Bb}^* &= \\big\\{ \\beta\\in {\\Bb} : \\ \\text{\\eqref{optimality} is satisfied} \\big\\},\\\\\n    \\Zb^* &= \\big\\{ (\\beta,\\theta)\\in {\\Bb}^* \\times \\Rb^{n_\\theta}:  Q(\\beta,\\theta) = 0\\big\\}.\n\\end{align*}","sha256_utf8":"81ba38b512633a0e766eb6595468c5d5376abf4a9a5965625e9ab3b06346e769"},{"start":39257,"end":39335,"start_line":725,"end_line":725,"raw_tex":"We propose a stochastic subgradient method for finding an element of $\\Zb^*$.\n","sha256_utf8":"308109fc8fdc5f7a08da6887fed11e9be2a65ff76ab9307b4970778414db1f61"}]},{"id":"S014","semantic_type":"algorithm","kind":"algorithm","title":"投影域与单次联合采样","source_units":[{"start":39476,"end":40079,"start_line":731,"end_line":740,"raw_tex":"To simplify the development and analysis\nof the method, we assume that $R \\ge \\underline{R}$ is known,\nso that $\\Zb^* \\subset \\Bb \\times \\Theta_R$,\nwhere $\\Theta_R = \\{ \\theta\\in \\Rb^{n_\\theta}: \\|\\theta \\| \\le R\\}$.\nDue to Assumption (A6), any $R\\ge \\underline{R}$ has this property.\n\nAt each iteration $k=0,1,\\dots$ of the method,\ngiven the current approximate solution $(\\beta^k,\\theta^k)\\in {\\Bb} \\times \\Theta_R$,\nwe use an observation $(X^k,Y^k)$ from the joint distribution of $(X,Y)$, to construct random directions\n$\\tilde{d}_\\beta^k$ and $\\tilde{d}_\\theta^k$, and to update the current point:\n","sha256_utf8":"24a4b598a5e8db54fb63e269148ad94511b6ed4a1ce4ae4bfa4f8884f5b1e575"}]},{"id":"S015","semantic_type":"algorithm","kind":"algorithm","title":"同步投影更新","source_units":[{"start":40079,"end":40295,"start_line":741,"end_line":747,"raw_tex":"\\begin{equation}\n\\label{method}\n\\begin{aligned}\n\\beta^{k+1} &= \\varPi_{\\Bb}(\\beta^k + \\tau_k \\tilde{d}_\\beta^k),\\\\\n\\theta^{k+1} &= \\varPi_{\\Theta_R}(\\theta^k + \\tau_k \\tilde{d}_\\theta^k).\n\\end{aligned}\n\\end{equation}","sha256_utf8":"140921c3d8301aa49f6aa6b9445a6640d156d270d2c917c2aa3a16cc41b8ade5"},{"start":40296,"end":40493,"start_line":748,"end_line":749,"raw_tex":"Here, $\\tau_k>0$ is a stepsize of the method, $\\varPi_{\\Bb}(\\Cdot)$  denotes the orthogonal projection on $\\Bb$,\nand $\\varPi_{\\Theta_R}(\\Cdot)$ is the orthogonal projection on the ball $\\Theta_R$.\n","sha256_utf8":"cb2f1224dc52b36829561fda12144e214bd2bf494460598f2a8b16586216b67b"}]},{"id":"S016","semantic_type":"algorithm","kind":"algorithm","title":"历史信息与独立观测","source_units":[{"start":40494,"end":41653,"start_line":751,"end_line":768,"raw_tex":"To construct the directions $\\tilde{d}_\\beta^k$ and $\\tilde{d}_\\theta^k$,\nwe use the values and  generalized gradients of the functions $f(X^k,Y^k,\\Cdot)$ and $\\varPsi(X^k,\\Cdot)$, which\ncan be observed at the current point $(\\beta^k,\\theta^k)$. We also use the derivatives of the function $g(\\Cdot)$ at the\npoints $\\varPsi(X^k,\\theta^k)$, which can be observed as well.\n\nIn our presentation of the method and its analysis, we denote by $(\\varOmega,\\Fc,P)$ the algorithmic probability space on which\nthe random sequences generated by the method are defined, and by\n$\\Fc_k$ the $\\sigma$-subalgebra defined by the history $(\\beta^0,\\theta^0,X^0,Y^0,\\dots,X^{k-1},Y^{k-1},\\beta^k,\\theta^k)$.\nWe assume that at iteration $k=0,1,\\dots$ we observe the pair $(X^k,Y^k)$, independently of $\\Fc_k$, and we calculate the following quantitites:\n\\begin{gather*}\nf^k  = f(X^k,Y^k,\\beta^k),\\\\\nf_\\beta^k  \\in \\hat{\\partial}_{ \\beta} f(X^k,Y^k,\\beta^k),\\\\\n\\varPsi^k  = \\varPsi(X^k,\\theta^k),\\\\\n\\varPsi_\\theta^k  \\in \\hat{\\partial}_{ \\theta} \\varPsi(X^k,\\theta^k),\\\\\n\\nabla g^k = \\nabla g(\\varPsi^k).\n%\\\\\n%{  \\nabla^2 g^k \\in \\hat{\\partial} \\nabla g(\\varPsi^k).}\n\\end{gather*}","sha256_utf8":"959606f5329d750921ad4dab0a93e908e8c6cafed007ed037f85afb64602dd23"}]},{"id":"S017","semantic_type":"algorithm","kind":"algorithm","title":"随机方向与比例参数","source_units":[{"start":41654,"end":41919,"start_line":769,"end_line":775,"raw_tex":"Then we calculate the directions in \\eqref{method}:\n\\begin{subequations}\n\\label{dzk}\n\\begin{align}\n    \\tilde{d}_\\beta^k &= - f_\\beta^k \\,\\nabla g^k,  \\label{dbetak}\\\\\n    \\tilde{d}_\\theta^k &=\\gamma \\,\\varPsi_\\theta^k (f^k - \\varPsi^k). \\label{dthetak}\n\\end{align}","sha256_utf8":"d1262598b2d3a80cba89687b90efa685c767bc2b25090240844335147d294e11"},{"start":41920,"end":41986,"start_line":776,"end_line":777,"raw_tex":"\\end{subequations}\nHere, $\\gamma>0$ is a parameter of the method.\n","sha256_utf8":"f63b25a16ac64b7956ef6f7381900297db37c7d239bf51b11092f956f8ccc0f9"}]},{"id":"S018","semantic_type":"assumption","kind":"assumption","title":"随机步长条件","source_units":[{"start":42950,"end":43298,"start_line":794,"end_line":802,"raw_tex":"\\begin{description}\n\\item[(A7)]\nThe stepsizes $\\tau_k$ are $\\Fc_k$-measurable, and such that\\vspace{0.5ex}\n\\begin{tightlist}{iii}\n\\item  $\\tau_k >0, \\quad k=0,1,2,\\dots$;\n\\item   $ \\displaystyle{\\sum_{k=0}^\\infty} \\tau_k = \\infty \\quad \\text{a.s.}$;\n\\item $\\Eb \\displaystyle{\\sum_{k=0}^\\infty} \\tau_k^2 < \\infty$.\n\\end{tightlist}\n\\end{description}\n","sha256_utf8":"d737359b2e0c24650e349211bd82a3653679ab19f17729225d78b669a1b8dc84"}]},{"id":"S019","semantic_type":"definition","kind":"definition","title":"平均漂移多函数","source_units":[{"start":43342,"end":43694,"start_line":806,"end_line":816,"raw_tex":"Consider the multifunction\n$\\varGamma: \\Bb \\times \\Theta_R \\rightrightarrows \\Rb^{n_f}$ defined as follows:\n\\[\n\\varGamma(\\beta,\\theta) = \\Eb\n\\left\\{\n\\begin{bmatrix}\n -\\hat{\\partial}_\\beta F(X,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big) \\\\\n\\gamma \\,\\hat{\\partial}_\\theta \\varPsi(X,\\theta)\\big(F(X,\\beta) -\\varPsi(X,\\theta)\\big)\n\\end{bmatrix}\n\\right\\}.\n\\]","sha256_utf8":"c92637d97843bfb1e762ebf53e2da270fca92c6f57d4ffdfcc6606ed96c9c250"},{"start":43695,"end":43825,"start_line":817,"end_line":817,"raw_tex":"We need to establish the properties of the multifunction $\\varGamma(\\Cdot,\\Cdot)$ that are relevant for the convergence analysis.\n","sha256_utf8":"ce23307c3c1a0817d153251cc4357c99c0e8e25ed1afaec05ebb84d39e5c7bea"}]},{"id":"S020","semantic_type":"definition","kind":"definition","title":"平均方向与噪声","source_units":[{"start":48077,"end":48215,"start_line":902,"end_line":906,"raw_tex":"Define the average directions,\n\\begin{equation}\n\\label{def:avg_dir}\nd^k = \\Eb [\\tilde{d}^k \\,| \\, \\Fc_k],\\quad k=0,1,\\dots,\n\\end{equation}","sha256_utf8":"b150d02fdea2c47bb7f3b9bbb58a96adea9c4c13be9393b2f65a530d65eedbd6"},{"start":48216,"end":48302,"start_line":907,"end_line":912,"raw_tex":"and let\n\\begin{equation}\n\\notag\n%\\label{def:e}\ne^k = \\tilde{d^k} - d^k.\n\\end{equation}","sha256_utf8":"fdfbcd7875cedc77863db0007e38bd9f965bf50f4fb454cc36deeaa67f1d4e4b"}]},{"id":"S021","semantic_type":"definition","kind":"definition","title":"插值轨迹及时间平移","source_units":[{"start":52455,"end":52941,"start_line":988,"end_line":995,"raw_tex":"For brevity, denote $z^k = (\\beta^k,\\theta^k) $, $k=0,1,\\dots$, and $\\Zb=\\Bb \\times\\Theta_R$.\nWe consider a particular trajectory $z^k(\\omega) = (\\beta^k(\\omega),\\theta^k(\\omega)) $, $k=0,1,\\dots$; in our notation, we omit\nthe elementary event $\\omega\\in \\Omega$, which may be arbitrary, except for a zero measure set.\n\nFirst, we construct continuous-time trajectories by linear interpolation. We introduce the accumulated stepsizes\n\\[\nt_k = \\sum_{j=0}^{k-1}\\tau_j,\\quad k=0,1,\\dots,\n\\]","sha256_utf8":"da9c890c3021086ca70e4bd2195022783dfe58b5dfbb68766981d67c154b9389"},{"start":52942,"end":53091,"start_line":996,"end_line":999,"raw_tex":"and we construct the interpolated trajectory\n\\[\nZ_0(t) = z^k + \\frac{t-t_{k}}{\\tau_k}(z^{k+1}-z^k),\\quad t_{k}\\le t \\le t_{k+1},\\quad k=0,1,\\dots.\n\\]","sha256_utf8":"4e13e06543678104266a125646401091aee4ff0f9af7092ab9a0fbbbaecb34f7"},{"start":53092,"end":53314,"start_line":1000,"end_line":1004,"raw_tex":"Then, for an increasing sequence of positive numbers $\\{s_m\\}$ diverging to $\\infty$, we define the shifted trajectories\n\\begin{equation}\n\\label{Z-shifted}\nZ_m(t)=Z_0(t+s_m), \\quad m=0,1,\\dots,\\quad t \\ge 0.\n\\end{equation}","sha256_utf8":"6d55c0b251c564edf27c1126c6a95eb8aaf56dc072465754281de286c23e2d9c"},{"start":53315,"end":53379,"start_line":1005,"end_line":1005,"raw_tex":"In particular, if we set $s_m = t_m$, we obtain $Z_m(0) = z^m$.\n","sha256_utf8":"94e93a4bc82b29a43b49d545d3c5287e1ffd3c8aaf0f745f7b927542edb93a66"}]},{"id":"S022","semantic_type":"definition","kind":"definition","title":"正则化跟踪与势函数","source_units":[{"start":54601,"end":54844,"start_line":1034,"end_line":1038,"raw_tex":"To this end, we construct a specially tailored Lyapunov function with parameters $\\alpha > 0$ and $\\lambda \\ge 0$:\n\\begin{equation}\n\\label{Lyapunov}\n    w(\\beta,\\theta) \\triangleq G(\\beta) + \\alpha \\Delta^\\lambda(\\beta, \\theta),\n\\end{equation}","sha256_utf8":"a71d7d381ee559c98464993071ed3239fb8ff4710729541523c8d3ebfd38a78b"},{"start":54845,"end":55109,"start_line":1039,"end_line":1043,"raw_tex":"where $G(\\beta)$ is the CSO loss function in \\eqref{cso} and\n\\begin{equation}\n\\Delta^{\\lambda}(\\beta,\\theta) \\triangleq \\Eb\\Big[g(F) - g(\\varPsi) - \\langle \\nabla g(\\varPsi), F - \\varPsi \\rangle + \\frac{\\lambda}{2} \\|F-\\varPsi\\|^2\\Big].\n\\label{te:g}\n\\end{equation}","sha256_utf8":"8ecc52bdf5eda2bf71a9e2e4285c348c4d9eff84a4d6122c9da637c7296f09e5"}]},{"id":"S023","semantic_type":"assumption","kind":"assumption","title":"驻点值集的萨德条件","source_units":[{"start":72262,"end":72548,"start_line":1349,"end_line":1352,"raw_tex":"In order to prove the convergence of the discrete-time algorithm, we need an additional technical assumption (known as the \\emph{Sard condition}).\n\\begin{description}\n\\item[(A8)] The set $\\{G(\\beta): \\beta\\in {\\Bb}^*\\}$ does not contain an interval of nonzero length.\n\\end{description}\n","sha256_utf8":"ddfbfeba0827d66fc34c8385a2d701fc099043e80e94a8ceebd756ea38a39ec5"}]},{"id":"S024","semantic_type":"definition","kind":"definition","title":"诺尔金广义可微性","source_units":[{"start":102338,"end":103389,"start_line":1926,"end_line":1943,"raw_tex":"\\begin{definition}\n\\label{d:Norkin}\nA function $f:\\Rb^n\\to\\Rb$ is  \\emph{differentiable in a generalized sense at a point} $x\\in \\Rb^n$,\nif an open set $U\\subset \\Rb^n$ containing $x$,\nand a nonempty, convex, compact valued, and upper semicontinuous multifunction\n$\\hat{\\partial}f: U \\rightrightarrows \\Rb^n$ exist, such that for all $y\\in U$ and all $g \\in \\hat{\\partial}f(y)$ the following equation is true:\n\\[\nf(y) = f(x) + \\langle g(y), y-x \\rangle + o(x,y,g),\n\\]\nwith\n\\[\n\\lim_{y\\to x} \\sup_{g\\in \\hat{\\partial}f(y)} \\frac{o(x,y,g)}{\\|y-x\\|}=0.\n\\]\nThe set $\\hat{\\partial}f(y)$ is  the \\emph{generalized subdifferential} of $f$ at $y$.\nIf a function is differentiable in a generalized sense at every  $x \\in \\Rb^n$ with the same generalized subdifferential mapping $\\hat{\\partial}f:\\Rb^n\\rightrightarrows \\Rb^n$, we call it \\emph{differentiable in a generalized sense}.\n\nA function $f:\\Rb^n\\to\\Rb^m$ is  differentiable in a generalized sense, if each of its component functions, $f_i:\\Rb^n\\to\\Rb$, $i=1,\\dots,m$, has this property.\n\\end{definition}","sha256_utf8":"046f405c3b8919d5bda34c48a3f886e66730139ce2c71b11efc911b9ce3f8758"}]},{"id":"S025","semantic_type":"lemma","kind":"lemma","title":"条件期望次微分交换","source_units":[{"start":26745,"end":27374,"start_line":483,"end_line":492,"raw_tex":"\\begin{lemma}\n\\label{l:conditional_subgradient}\nSuppose the function $f(X,Y,\\Cdot)$ is differentiable in the generalized sense and its subgradients are bounded\nby an integrable function in any bounded neighborhood of any point $\\beta$. Then for almost all $X$ the function\n$\\beta \\mapsto \\Eb\\big[ f(X,Y,\\beta) \\big| X\\big]$ is differentiable in the generalized sense, and\nfor every measurable function $s:\\Rb^{n_x}\\times \\Rb^{n_y} \\to \\Rb^{n_f}$ such that $s(X,Y)  \\in \\hat{\\partial}_\\beta f(X,Y,\\beta)$ a.s., we have\n\\[\n\\Eb\\big[s(X,Y) \\big| X\\big] \\in \\hat{\\partial} \\Eb\\big[ f(X,Y,\\beta) \\big| X\\big] \\quad  a.s.\n\\]\n\\end{lemma}","sha256_utf8":"842557bd97bd839458311003f8422616879ad2c85aaac6ea26b0df0af27a8686"}]},{"id":"S026","semantic_type":"proof","kind":"proof","title":"条件次微分交换证明","source_units":[{"start":27376,"end":28027,"start_line":494,"end_line":507,"raw_tex":"\\begin{proof}\nThe probability measure $P$ on $\\Rb^{n_x}\\times \\Rb^{n_y}$\ncan be disintegrated into a marginal measure $P_X$ on $\\Rb^{n_x}$ and a kernel $T:\\Rb^{n_x} \\to \\Pc(\\Rb^{n_y})$,\n where $\\Pc(\\Rb^{n_y})$ is the space of probability measures\n on $\\Rb^{n_y}$. Then\n\\[\n\\Eb\\big[ f(X,Y,\\beta) \\big| X=x\\big] = \\int f(x,y,\\beta)\\;T(\\D y|x).\n\\]\nFor every bounded neighborhood $K$ of $\\beta$, the Lipschitz constant $L_K(x,y)$ of the function $f(x,y,\\Cdot)$\nis integrable, and thus, for almost\nall $x$, its conditional expectation\n$\\int L_K(x,y)\\,T(dy|x)$ exists as well.\nThe assertion then follows from Theorem \\ref{t:expected-generalized}.\n\\end{proof}","sha256_utf8":"dd806deef08b28da5a98fd48e9ddcef9c649ed7c555c4323f722ca10ec1d0f84"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"概率分解为条件核积分","source_units":[{"start":27376,"end":27720,"start_line":494,"end_line":501,"raw_tex":"\\begin{proof}\nThe probability measure $P$ on $\\Rb^{n_x}\\times \\Rb^{n_y}$\ncan be disintegrated into a marginal measure $P_X$ on $\\Rb^{n_x}$ and a kernel $T:\\Rb^{n_x} \\to \\Pc(\\Rb^{n_y})$,\n where $\\Pc(\\Rb^{n_y})$ is the space of probability measures\n on $\\Rb^{n_y}$. Then\n\\[\n\\Eb\\big[ f(X,Y,\\beta) \\big| X=x\\big] = \\int f(x,y,\\beta)\\;T(\\D y|x).\n\\]\n","sha256_utf8":"53f332657ae7a28cef4fe8addd32134af46d2f81bc62f82481091d5c9fa1077a"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"条件可积界与期望演算","source_units":[{"start":27720,"end":28027,"start_line":502,"end_line":507,"raw_tex":"For every bounded neighborhood $K$ of $\\beta$, the Lipschitz constant $L_K(x,y)$ of the function $f(x,y,\\Cdot)$\nis integrable, and thus, for almost\nall $x$, its conditional expectation\n$\\int L_K(x,y)\\,T(dy|x)$ exists as well.\nThe assertion then follows from Theorem \\ref{t:expected-generalized}.\n\\end{proof}","sha256_utf8":"bab8d727967f28c31b82f1173157df14aa741952496c03f5d6ad28d4a6c93f2d"}]},{"id":"S029","semantic_type":"proof","kind":"proof","title":"目标广义次微分推导","source_units":[{"start":28676,"end":29761,"start_line":518,"end_line":538,"raw_tex":"Furthermore, by Theorem \\ref{t:composition} and Assumption (A2), the composition\n$g( F(x,\\Cdot))$ is differentiable in the generalized sense as well, and\n\\[\n\\hat{\\partial}_{  \\beta} [g\\circ F](x,\\beta)  = \\big\\{    F_\\beta(x,\\beta) \\nabla g(F(x,\\beta)) : F_\\beta(x,\\beta) \\in \\hat{\\partial}_{  \\beta} F (x,\\beta)\\big\\}\n\\]\nis its generalized subdifferential (we adopt the convention that the generalized Jacobian $F_\\beta$ is an $n_\\beta \\times n_f$ matrix).\nThanks to the integrability condition in (A3) and Theorem \\ref{t:expected-generalized}, the function $G(\\Cdot)$ in \\eqref{cso}\nis differentiable in the generalized sense as well, and a realization of its generalized subdifferential is the multifunction\n\\begin{equation}\n    \\label{G-sub}\n\\begin{aligned}\n\\hat{\\partial}_{  \\beta} G(\\beta) &= \\Eb \\big\\{ \\hat{\\partial}_{  \\beta} [g\\circ F](X,\\beta) \\big\\}\n=\\Eb \\big\\{\n\\hat{\\partial}_{  \\beta} F (X,\\beta) \\nabla g(F(X,\\beta))\n\\big\\}\n\\\\ &= \\Eb \\big\\{\n\\hat{\\partial}f (X,Y,\\beta) \\nabla g(F(X,\\beta))\n\\big\\}.\n\\end{aligned}\n\\end{equation}\nThe last equation follows by conditioning.\n","sha256_utf8":"c499ce3429eab15c75ebfb1618dd6659d126144e0f1829e962de1c9484a6f8bf"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"条件目标的复合次微分","source_units":[{"start":28676,"end":29134,"start_line":518,"end_line":523,"raw_tex":"Furthermore, by Theorem \\ref{t:composition} and Assumption (A2), the composition\n$g( F(x,\\Cdot))$ is differentiable in the generalized sense as well, and\n\\[\n\\hat{\\partial}_{  \\beta} [g\\circ F](x,\\beta)  = \\big\\{    F_\\beta(x,\\beta) \\nabla g(F(x,\\beta)) : F_\\beta(x,\\beta) \\in \\hat{\\partial}_{  \\beta} F (x,\\beta)\\big\\}\n\\]\nis its generalized subdifferential (we adopt the convention that the generalized Jacobian $F_\\beta$ is an $n_\\beta \\times n_f$ matrix).\n","sha256_utf8":"100758d4cded036a00c16e6281ce28e54cf97a6f857bae518f371946e40457b1"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"期望及条件化得到目标导数","source_units":[{"start":29134,"end":29761,"start_line":524,"end_line":538,"raw_tex":"Thanks to the integrability condition in (A3) and Theorem \\ref{t:expected-generalized}, the function $G(\\Cdot)$ in \\eqref{cso}\nis differentiable in the generalized sense as well, and a realization of its generalized subdifferential is the multifunction\n\\begin{equation}\n    \\label{G-sub}\n\\begin{aligned}\n\\hat{\\partial}_{  \\beta} G(\\beta) &= \\Eb \\big\\{ \\hat{\\partial}_{  \\beta} [g\\circ F](X,\\beta) \\big\\}\n=\\Eb \\big\\{\n\\hat{\\partial}_{  \\beta} F (X,\\beta) \\nabla g(F(X,\\beta))\n\\big\\}\n\\\\ &= \\Eb \\big\\{\n\\hat{\\partial}f (X,Y,\\beta) \\nabla g(F(X,\\beta))\n\\big\\}.\n\\end{aligned}\n\\end{equation}\nThe last equation follows by conditioning.\n","sha256_utf8":"43587b0072cae4f32d170c88badd181d055cacf38970ce8e132bb02840f27450"}]},{"id":"S032","semantic_type":"proof","kind":"proof","title":"均方误差次微分推导","source_units":[{"start":30549,"end":31467,"start_line":559,"end_line":580,"raw_tex":"Due to Assumptions (A3) and (A4), by virtue of Theorem \\ref{t:composition},\nthe error function \\eqref{te:Q} is differentiable in the generalized sense as well, and\nthe multifunction\n\\begin{equation}\n\\label{dQ}\n\\hat{\\partial}_{(\\beta,\\theta)} Q(\\beta,\\theta) =\n\\Eb\\left\\{ \\begin{bmatrix}\n\\hat{\\partial}_{  \\beta} f(X,Y,\\beta) \\\\\n-\\hat{\\partial}_{  \\theta} \\varPsi(X,\\theta)\n\\end{bmatrix}\n \\big(   F(X, \\beta) - \\varPsi(X, \\theta)\\big) \\right\\}\n\\end{equation}\nis its generalized subdifferential.\n%Evidently, we cannot observe unbiased estimates of the subgradients of $Q(\\Cdot,\\Cdot)$, because we do not see $F(X,\\beta)$.\nSince the set \\eqref{dQ} is a Cartesian product, its $\\theta$-part,\n\\[\n\\hat{\\partial}_{\\theta} Q(\\beta,\\theta) =\n- \\Eb\\left\\{\n\\hat{\\partial}_{  \\theta} \\varPsi(X,\\theta)\n \\big(   F(X, \\beta) - \\varPsi(X, \\theta)\\big) \\right\\},\n\\]\nis a subdifferential of $Q(\\beta,\\theta)$ with respect to $\\theta$.\n","sha256_utf8":"8f872fcad806fe63d63b39d09d33abf38b61e3ac88101b0eb0b3e18e273c5909"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"联合参数误差次微分","source_units":[{"start":30549,"end":31043,"start_line":559,"end_line":571,"raw_tex":"Due to Assumptions (A3) and (A4), by virtue of Theorem \\ref{t:composition},\nthe error function \\eqref{te:Q} is differentiable in the generalized sense as well, and\nthe multifunction\n\\begin{equation}\n\\label{dQ}\n\\hat{\\partial}_{(\\beta,\\theta)} Q(\\beta,\\theta) =\n\\Eb\\left\\{ \\begin{bmatrix}\n\\hat{\\partial}_{  \\beta} f(X,Y,\\beta) \\\\\n-\\hat{\\partial}_{  \\theta} \\varPsi(X,\\theta)\n\\end{bmatrix}\n \\big(   F(X, \\beta) - \\varPsi(X, \\theta)\\big) \\right\\}\n\\end{equation}\nis its generalized subdifferential.\n","sha256_utf8":"a1c997a0cf508f9e6b054f4cfd9d00094396a413a97347d4f4d831ecd174e794"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"提取模型参数次微分","source_units":[{"start":31043,"end":31467,"start_line":572,"end_line":580,"raw_tex":"%Evidently, we cannot observe unbiased estimates of the subgradients of $Q(\\Cdot,\\Cdot)$, because we do not see $F(X,\\beta)$.\nSince the set \\eqref{dQ} is a Cartesian product, its $\\theta$-part,\n\\[\n\\hat{\\partial}_{\\theta} Q(\\beta,\\theta) =\n- \\Eb\\left\\{\n\\hat{\\partial}_{  \\theta} \\varPsi(X,\\theta)\n \\big(   F(X, \\beta) - \\varPsi(X, \\theta)\\big) \\right\\},\n\\]\nis a subdifferential of $Q(\\beta,\\theta)$ with respect to $\\theta$.\n","sha256_utf8":"ee9c23fc8cb3f9e4d5dfe41ded53fe92c497a1869493f8ceca2e295f766cab4f"}]},{"id":"S035","semantic_type":"lemma","kind":"lemma","title":"漂移的紧凸性与上半连续性","source_units":[{"start":43825,"end":43977,"start_line":818,"end_line":821,"raw_tex":"\\begin{lemma}\n\\label{l:Gamma}\nThe multifunction $\\varGamma(\\Cdot,\\Cdot)$ is nonempty-, convex-, and compact-valued and upper-semicontinuous.\n\\end{lemma}","sha256_utf8":"e4ee475adf57c1e547da9b2c5dac004a762c0e4a40b8d98a132f67f370da7445"}]},{"id":"S036","semantic_type":"proof","kind":"proof","title":"平均漂移正则性证明","source_units":[{"start":43978,"end":46209,"start_line":822,"end_line":863,"raw_tex":"\\begin{proof}\nBy virtue of Lemma \\ref{l:conditional_subgradient}, the $\\beta$-component of $\\varGamma(\\beta,\\theta)$ equals\n\\begin{multline*}\n -\\Eb \\big[ \\hat{\\partial}_\\beta F(X,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big)\\big] =\n -\\Eb\\Big\\{ \\Eb\\big[ \\hat{\\partial}_\\beta f(X,Y,\\beta)\\big|X\\big] \\nabla g\\big(\\varPsi(X,\\theta)\\big)\\Big\\}\\\\\n% - \\hat{\\partial}_\\beta  \\Eb \\big[ F(X,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big)\\big]\\\\\n% =\n% - \\hat{\\partial}_\\beta  \\Eb \\big[ f(X,Y,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big)\\big]\n  =\n - \\Eb \\big[ \\hat{\\partial}_\\beta  f(X,Y,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big)\\big].\n\\end{multline*}\n\\begin{comment}\nThe interchange of the subdifferentiation and the expected value is justified because\nAssumptions (A2) and (A3) imply that:\n\\begin{tightlist}{ii}\n\\item the function\n$\\beta \\mapsto f(X,Y,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big)$ is\nNorkin differentiable; and\n\\item its Lipschitz constant $L_f(X,Y) L_g$ is integrable.\n\\end{tightlist}\n\\vspace{0.5ex}\n\\end{comment}\n\nThe $\\theta$-component of $\\varGamma(\\beta,\\theta)$ can be written as follows:\n\\[\n\\gamma \\,\\Eb \\big[ \\hat{\\partial}_\\theta \\varPsi(X,\\theta)\\big(F(X,\\beta) -\\varPsi(X,\\theta)\\big)\\big]\n= \\gamma \\,\\Eb \\big[ \\hat{\\partial}_\\theta \\varPsi(X,\\theta)\\big(f(X,Y,\\beta) -\\varPsi(X,\\theta)\\big)\\big].\n\\]\nDue to Assumptions (A2)--(A4), for all $(X,Y)$, the multifunction\n\\[\n(\\beta,\\theta) \\mapsto \\widetilde{\\varGamma}(\\beta,\\theta;X,Y) \\triangleq\n\\begin{bmatrix}\n -\\hat{\\partial}_{ \\beta} f(X,Y,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big) \\\\\n\\gamma \\,\\hat{\\partial}_{ \\theta} \\varPsi(X,\\theta)\\big(f(X,Y,\\beta) -\\varPsi(X,\\theta)\\big)\n\\end{bmatrix}\n\\]\nis upper-semicontinuous, and convex- and compact-valued. Furthermore, its $\\beta$-values are\nbounded by and integrable function $L_f(X,Y)L_g$, and its $\\theta$-values are bounded\nby an integrable function $  \\gamma L_{\\varPsi}(X)( C_f +  C_{\\varPsi} + 4  L_{\\varPsi}(X) R^2)$.\nThese two conditions imply that its integral -- the multifunction $\\varGamma(\\Cdot,\\Cdot)$ -- is upper-semicontinuous, and convex- and compact-valued as well. This implication is part of\nthe proof of Theorem \\ref{t:expected-generalized}; see \\cite[Lem. 21.7]{mikhalevich1987nonconvex}.\n\\end{proof}","sha256_utf8":"2ab35f0909ce106ae4158c6ae88d314d8e1bf6df8a632a1648aad7ccb6ee97f3"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"决策分量改写为联合期望","source_units":[{"start":43978,"end":44992,"start_line":822,"end_line":844,"raw_tex":"\\begin{proof}\nBy virtue of Lemma \\ref{l:conditional_subgradient}, the $\\beta$-component of $\\varGamma(\\beta,\\theta)$ equals\n\\begin{multline*}\n -\\Eb \\big[ \\hat{\\partial}_\\beta F(X,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big)\\big] =\n -\\Eb\\Big\\{ \\Eb\\big[ \\hat{\\partial}_\\beta f(X,Y,\\beta)\\big|X\\big] \\nabla g\\big(\\varPsi(X,\\theta)\\big)\\Big\\}\\\\\n% - \\hat{\\partial}_\\beta  \\Eb \\big[ F(X,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big)\\big]\\\\\n% =\n% - \\hat{\\partial}_\\beta  \\Eb \\big[ f(X,Y,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big)\\big]\n  =\n - \\Eb \\big[ \\hat{\\partial}_\\beta  f(X,Y,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big)\\big].\n\\end{multline*}\n\\begin{comment}\nThe interchange of the subdifferentiation and the expected value is justified because\nAssumptions (A2) and (A3) imply that:\n\\begin{tightlist}{ii}\n\\item the function\n$\\beta \\mapsto f(X,Y,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big)$ is\nNorkin differentiable; and\n\\item its Lipschitz constant $L_f(X,Y) L_g$ is integrable.\n\\end{tightlist}\n\\vspace{0.5ex}\n\\end{comment}\n\n","sha256_utf8":"187214debf81f26e7c468a70237132b4ef883ca0098c30f1e87088a4c812b2b9"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"模型分量改写为联合期望","source_units":[{"start":44992,"end":45288,"start_line":845,"end_line":849,"raw_tex":"The $\\theta$-component of $\\varGamma(\\beta,\\theta)$ can be written as follows:\n\\[\n\\gamma \\,\\Eb \\big[ \\hat{\\partial}_\\theta \\varPsi(X,\\theta)\\big(F(X,\\beta) -\\varPsi(X,\\theta)\\big)\\big]\n= \\gamma \\,\\Eb \\big[ \\hat{\\partial}_\\theta \\varPsi(X,\\theta)\\big(f(X,Y,\\beta) -\\varPsi(X,\\theta)\\big)\\big].\n\\]\n","sha256_utf8":"e8167679e58eb17be66cd463ab0f958c75f8af2512366b58a7cdb0bd85a26d7c"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"随机漂移可积界与集合积分","source_units":[{"start":45288,"end":46209,"start_line":850,"end_line":863,"raw_tex":"Due to Assumptions (A2)--(A4), for all $(X,Y)$, the multifunction\n\\[\n(\\beta,\\theta) \\mapsto \\widetilde{\\varGamma}(\\beta,\\theta;X,Y) \\triangleq\n\\begin{bmatrix}\n -\\hat{\\partial}_{ \\beta} f(X,Y,\\beta) \\nabla g\\big(\\varPsi(X,\\theta)\\big) \\\\\n\\gamma \\,\\hat{\\partial}_{ \\theta} \\varPsi(X,\\theta)\\big(f(X,Y,\\beta) -\\varPsi(X,\\theta)\\big)\n\\end{bmatrix}\n\\]\nis upper-semicontinuous, and convex- and compact-valued. Furthermore, its $\\beta$-values are\nbounded by and integrable function $L_f(X,Y)L_g$, and its $\\theta$-values are bounded\nby an integrable function $  \\gamma L_{\\varPsi}(X)( C_f +  C_{\\varPsi} + 4  L_{\\varPsi}(X) R^2)$.\nThese two conditions imply that its integral -- the multifunction $\\varGamma(\\Cdot,\\Cdot)$ -- is upper-semicontinuous, and convex- and compact-valued as well. This implication is part of\nthe proof of Theorem \\ref{t:expected-generalized}; see \\cite[Lem. 21.7]{mikhalevich1987nonconvex}.\n\\end{proof}","sha256_utf8":"8b98dbc07d0e19dd230b7a8f289b78ce3bcef4b9dcd0bdde08f03c1b5f4dc2f9"}]},{"id":"S040","semantic_type":"lemma","kind":"lemma","title":"平均方向属于漂移集合","source_units":[{"start":48451,"end":48569,"start_line":916,"end_line":919,"raw_tex":"\\begin{lemma}\n\\label{l:dk-in-Gamma}\nFor all $k=0,1,2,\\dots$ we have $d^k \\in \\varGamma(\\beta^k,\\theta^k)$.\n\\end{lemma}","sha256_utf8":"aeb9f8fa5022ba588a6625894cb5d9dcec6549b1740bb57e2b3ad6781ed66e76"}]},{"id":"S041","semantic_type":"proof","kind":"proof","title":"平均方向包含关系证明","source_units":[{"start":48570,"end":49842,"start_line":920,"end_line":934,"raw_tex":"\\begin{proof}\nSince the observation $(X^k,Y^k)$ is independent of $\\Fc_k$, Lemma \\ref{l:conditional_subgradient}  yields the relations\n\\begin{align*}\n\\Eb[\\tilde{d}_\\beta^k|\\Fc_k]  &\\in -\\Eb \\big[ \\hat{\\partial}_{ \\beta} f(X^k,Y^k,\\beta^k) \\nabla g(\\varPsi(X^k,\\theta^k)) ~\\big|~ \\Fc_k\\big] \\\\\n&= -\\Eb \\Big\\{ \\Eb \\big[ \\hat{\\partial}_{ \\beta} f(X^k,Y^k,\\beta^k)  ~\\big|~ \\beta^k,\\theta^k, X^k \\big] \\nabla g(\\varPsi(X^k,\\theta^k)) ~\\Big|~ \\beta^k,\\theta^k \\Big\\}\\\\\n&= -\\Eb \\big[ \\hat{\\partial}_{ \\beta} F(X^k,\\beta^k) \\nabla g(\\varPsi(X^k,\\theta^k)) ~\\big|~ \\beta^k,\\theta^k \\big]\n= \\varGamma_\\beta (\\beta^k,\\theta^k).\n\\end{align*}\nThe $\\theta$-component of the direction can also be treated by conditioning:\n\\begin{align*}\n\\Eb[\\tilde{d}_\\theta^k|\\Fc_k] &\\in \\gamma \\Eb \\big[ \\hat{\\partial}_{ \\theta} \\varPsi(X^k,\\theta^k)\\big(f(X^k,Y^k,\\beta^k) -\\varPsi(X^k,\\theta^k)\\big)\\big| \\Fc_k\\big]\\\\\n&= \\gamma \\Eb \\Big\\{\\hat{\\partial}_{ \\theta} \\varPsi(X^k,\\theta^k)\\Eb\\big[f(X^k,Y^k,\\beta^k) -\\varPsi(X^k,\\theta^k)\\big| X^k,\\beta^k,\\theta^k\\big]\\Big| \\beta^k,\\theta^k\\Big\\}\\\\\n&= \\gamma \\Eb \\big\\{\\hat{\\partial}_{ \\theta} \\varPsi(X^k,\\theta^k)\\big(F(X^k,\\beta^k) -\\varPsi(X^k,\\theta^k)\\big)\\big| \\beta^k,\\theta^k\\big\\} = \\varGamma_\\theta(\\beta^k,\\theta^k).\n\\end{align*}\n\\end{proof}","sha256_utf8":"8ffd87c30a78e3f8fbf016f9dcb0b288c4adcf20bd2a9e77cd497576d3d66683"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"独立采样下决策方向条件化","source_units":[{"start":48570,"end":49201,"start_line":920,"end_line":927,"raw_tex":"\\begin{proof}\nSince the observation $(X^k,Y^k)$ is independent of $\\Fc_k$, Lemma \\ref{l:conditional_subgradient}  yields the relations\n\\begin{align*}\n\\Eb[\\tilde{d}_\\beta^k|\\Fc_k]  &\\in -\\Eb \\big[ \\hat{\\partial}_{ \\beta} f(X^k,Y^k,\\beta^k) \\nabla g(\\varPsi(X^k,\\theta^k)) ~\\big|~ \\Fc_k\\big] \\\\\n&= -\\Eb \\Big\\{ \\Eb \\big[ \\hat{\\partial}_{ \\beta} f(X^k,Y^k,\\beta^k)  ~\\big|~ \\beta^k,\\theta^k, X^k \\big] \\nabla g(\\varPsi(X^k,\\theta^k)) ~\\Big|~ \\beta^k,\\theta^k \\Big\\}\\\\\n&= -\\Eb \\big[ \\hat{\\partial}_{ \\beta} F(X^k,\\beta^k) \\nabla g(\\varPsi(X^k,\\theta^k)) ~\\big|~ \\beta^k,\\theta^k \\big]\n= \\varGamma_\\beta (\\beta^k,\\theta^k).\n\\end{align*}\n","sha256_utf8":"286f5a91edba4c6b177feee778cbf25831f9663373931fb15f5342f1fa2b93b5"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"模型方向条件化","source_units":[{"start":49201,"end":49842,"start_line":928,"end_line":934,"raw_tex":"The $\\theta$-component of the direction can also be treated by conditioning:\n\\begin{align*}\n\\Eb[\\tilde{d}_\\theta^k|\\Fc_k] &\\in \\gamma \\Eb \\big[ \\hat{\\partial}_{ \\theta} \\varPsi(X^k,\\theta^k)\\big(f(X^k,Y^k,\\beta^k) -\\varPsi(X^k,\\theta^k)\\big)\\big| \\Fc_k\\big]\\\\\n&= \\gamma \\Eb \\Big\\{\\hat{\\partial}_{ \\theta} \\varPsi(X^k,\\theta^k)\\Eb\\big[f(X^k,Y^k,\\beta^k) -\\varPsi(X^k,\\theta^k)\\big| X^k,\\beta^k,\\theta^k\\big]\\Big| \\beta^k,\\theta^k\\Big\\}\\\\\n&= \\gamma \\Eb \\big\\{\\hat{\\partial}_{ \\theta} \\varPsi(X^k,\\theta^k)\\big(F(X^k,\\beta^k) -\\varPsi(X^k,\\theta^k)\\big)\\big| \\beta^k,\\theta^k\\big\\} = \\varGamma_\\theta(\\beta^k,\\theta^k).\n\\end{align*}\n\\end{proof}","sha256_utf8":"0aa7a41e6945541a8ad851797357edb4151ba8bee40a4160594a585bdfe9b7c0"}]},{"id":"S044","semantic_type":"lemma","kind":"lemma","title":"随机噪声条件方差有界","source_units":[{"start":49925,"end":50184,"start_line":938,"end_line":942,"raw_tex":"\\begin{lemma}\n    \\label{l:e2-bounded}\nUnder Assumptions \\textup{(A1)--(A4)}, the directions $\\{d^k\\}_{k\\ge 0}$ are well-defined and a constant $\\sigma > 0$ exists, such that\n$\\Eb\\big[\\|e^k\\|^2 \\,\\big|\\,\\Fc_k\\big] \\le \\sigma^2$, for all $k \\ge 0$.\n\\end{lemma}","sha256_utf8":"00be6aefc5ef045dc68e577be16ea0b4a79c3ebbb3db09cae6414b2753d6ed29"}]},{"id":"S045","semantic_type":"proof","kind":"proof","title":"噪声二阶矩界证明","source_units":[{"start":50185,"end":51761,"start_line":943,"end_line":975,"raw_tex":"\\begin{proof}\nFor the $\\beta$-component $e_\\beta^k$ of $e^k$,  we have\n\\[\n\\tilde{d}_\\beta^k = -f_\\beta(X^k,Y^k,\\beta^k)\\nabla g(\\varPsi(X^k,\\theta^k)),\n\\]\nand thus, by Assumptions (A2) and (A3),\n\\[\n\\Eb\\big[ \\|{e}_\\beta^k\\|^2\\big| \\Fc_k ] \\le \\Eb\\big[ \\|\\tilde{d}_\\beta^k\\|^2\\big| \\Fc_k ]\n\\le L_g^2 \\Eb \\big[ \\| f_\\beta(X^k,Y^k,\\beta^k)\\|^2 \\big| \\Fc_k \\big]\\le\n L_g^2 \\widebar{L}_f^2.\n\\]\nFor the $\\theta$-component,\n\\begin{align*}\n\\Eb\\big[ \\| \\tilde{d}_\\theta^k\\|^2\\big| \\Fc_k\\big]\n&\\le \\gamma^2 \\Eb\\big[ \\|\\varPsi_\\theta(X^k,\\theta^k)\\|^2 \\cdot \\| f(X^k,Y^k,\\beta^k)  - \\varPsi(X^k,\\theta^k)\\|^2\\big| \\Fc_k\\big]\\\\\n&\\le \\gamma^2 \\Eb\\big[{L_\\psi}(X^k)^2\\| f(X^k,Y^k,\\beta^k)  - \\varPsi(X^k,\\theta^k)\\|^2\\big| \\Fc_k\\big]\\\\\n&\\le 2\\gamma^2 \\Eb\\big[{L_\\psi}(X^k)^2\\| f(X^k,Y^k,\\beta^k)\\|^2\\big| \\Fc_k\\big]\\\\\n&{\\quad} +2 \\gamma^2 \\Eb\\big[ {L_\\psi}(X^k)^2\\|\\varPsi(X^k,\\theta^k)\\|^2\\big| \\Fc_k\\big].\n\\end{align*}\nUsing the Cauchy--Schwartz inequality and Assumptions (A3) and (A4) we obtain\n\\[\n\\Eb\\big[{L_\\psi}(X^k)^2\\| f(X^k,Y^k,\\beta^k)\\|^2\\big| \\Fc_k\\big]\n\\le  \\widebar{L}_\\varPsi^2 C_f^2.\n\\]\nIn a similar way, by (A4),\n\\begin{align*}\n\\lefteqn{\\Eb\\big[ {L_\\psi}(X^k)^2\\|\\varPsi(X^k,\\theta^k)\\|^2\\big| \\Fc_k\\big]}\\\\\n&\\le 2 \\Eb\\big[{L_\\psi}(X^k)^2\\big( \\|\\varPsi(X^k,\\theta^0)\\|^2 + {L_\\psi}(X^k)^2 \\|\\theta^k-\\theta^0\\|^2\\big)\\big| \\Fc_k\\big]\n\\le 2 \\widebar{L}_\\varPsi^2 C_\\varPsi^2 + 8R^2 \\widebar{L}_\\varPsi^4.\n\\end{align*}\nCombining the last three inequalities, we conclude that\n$\\Eb\\big[ \\| e_\\theta^k\\|^2\\big| \\Fc_k\\big] $, $k\\ge 0$, are uniformly bounded as well.\n\\end{proof}","sha256_utf8":"24b05329aed11de37cb65a22a76ca9c06618336ef518ff27ac6f79db2bbb50c3"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"决策方向噪声二阶矩界","source_units":[{"start":50185,"end":50573,"start_line":943,"end_line":953,"raw_tex":"\\begin{proof}\nFor the $\\beta$-component $e_\\beta^k$ of $e^k$,  we have\n\\[\n\\tilde{d}_\\beta^k = -f_\\beta(X^k,Y^k,\\beta^k)\\nabla g(\\varPsi(X^k,\\theta^k)),\n\\]\nand thus, by Assumptions (A2) and (A3),\n\\[\n\\Eb\\big[ \\|{e}_\\beta^k\\|^2\\big| \\Fc_k ] \\le \\Eb\\big[ \\|\\tilde{d}_\\beta^k\\|^2\\big| \\Fc_k ]\n\\le L_g^2 \\Eb \\big[ \\| f_\\beta(X^k,Y^k,\\beta^k)\\|^2 \\big| \\Fc_k \\big]\\le\n L_g^2 \\widebar{L}_f^2.\n\\]\n","sha256_utf8":"d120db0867ac08f0c0228fea91dcc4a1140eff356a18fa333c4880edc10adf88"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"分解模型方向的二阶矩","source_units":[{"start":50573,"end":51091,"start_line":954,"end_line":961,"raw_tex":"For the $\\theta$-component,\n\\begin{align*}\n\\Eb\\big[ \\| \\tilde{d}_\\theta^k\\|^2\\big| \\Fc_k\\big]\n&\\le \\gamma^2 \\Eb\\big[ \\|\\varPsi_\\theta(X^k,\\theta^k)\\|^2 \\cdot \\| f(X^k,Y^k,\\beta^k)  - \\varPsi(X^k,\\theta^k)\\|^2\\big| \\Fc_k\\big]\\\\\n&\\le \\gamma^2 \\Eb\\big[{L_\\psi}(X^k)^2\\| f(X^k,Y^k,\\beta^k)  - \\varPsi(X^k,\\theta^k)\\|^2\\big| \\Fc_k\\big]\\\\\n&\\le 2\\gamma^2 \\Eb\\big[{L_\\psi}(X^k)^2\\| f(X^k,Y^k,\\beta^k)\\|^2\\big| \\Fc_k\\big]\\\\\n&{\\quad} +2 \\gamma^2 \\Eb\\big[ {L_\\psi}(X^k)^2\\|\\varPsi(X^k,\\theta^k)\\|^2\\big| \\Fc_k\\big].\n\\end{align*}\n","sha256_utf8":"17afe887d2136ddea1a5dc4ca441224c3d4023ce7e2ac2a519413a1de255c4e9"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"控制观测函数乘积矩","source_units":[{"start":51091,"end":51274,"start_line":962,"end_line":966,"raw_tex":"Using the Cauchy--Schwartz inequality and Assumptions (A3) and (A4) we obtain\n\\[\n\\Eb\\big[{L_\\psi}(X^k)^2\\| f(X^k,Y^k,\\beta^k)\\|^2\\big| \\Fc_k\\big]\n\\le  \\widebar{L}_\\varPsi^2 C_f^2.\n\\]\n","sha256_utf8":"40245a82c0ab7f47a2f333793a8f79f380be398ff121f0a20f9e88691111ac74"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"控制模型乘积矩并合并","source_units":[{"start":51274,"end":51761,"start_line":967,"end_line":975,"raw_tex":"In a similar way, by (A4),\n\\begin{align*}\n\\lefteqn{\\Eb\\big[ {L_\\psi}(X^k)^2\\|\\varPsi(X^k,\\theta^k)\\|^2\\big| \\Fc_k\\big]}\\\\\n&\\le 2 \\Eb\\big[{L_\\psi}(X^k)^2\\big( \\|\\varPsi(X^k,\\theta^0)\\|^2 + {L_\\psi}(X^k)^2 \\|\\theta^k-\\theta^0\\|^2\\big)\\big| \\Fc_k\\big]\n\\le 2 \\widebar{L}_\\varPsi^2 C_\\varPsi^2 + 8R^2 \\widebar{L}_\\varPsi^4.\n\\end{align*}\nCombining the last three inequalities, we conclude that\n$\\Eb\\big[ \\| e_\\theta^k\\|^2\\big| \\Fc_k\\big] $, $k\\ge 0$, are uniformly bounded as well.\n\\end{proof}","sha256_utf8":"d487ed6d71918883f0141c8d63c44cd4af417a2c37d6dc66767f3b2a3df36b97"}]},{"id":"S050","semantic_type":"theorem","kind":"theorem","title":"随机轨迹的微分包含极限","source_units":[{"start":53463,"end":54025,"start_line":1008,"end_line":1018,"raw_tex":"\\begin{theorem}\n\\label{t:diff-inc-beta}\nUnder Assumptions \\textup{(A1)--(A4)} and \\textup{(A7)},\nthe sequence $\\big\\{Z_m(\\Cdot)\\big\\}_{m\\ge 0}$ has a subsequence convergent a.s. to an absolutely continuous function $Z_\\infty:[0,\\infty) \\to \\Xc$ which is a solution of the differential inclusion:\n\\begin{equation}\n\\label{diff-inc-beta}\n\\dt{Z}_\\infty(t) \\in  \\varGamma(Z_{\\infty}(t))-\\Nc_{\\Zb}(Z_\\infty(t)), \\quad t \\ge 0.\n\\end{equation}\nMoreover, for any $t\\ge 0$, the point $Z_\\infty(t)$\nis an accumulation point of the sequence $\\{z^k\\}_{k\\ge 0}$.\n\\end{theorem}","sha256_utf8":"3d69d37b2eb87cb5c735c1c0e2be957f13c0f2d2512422189ccb514980d38b1a"}]},{"id":"S051","semantic_type":"proof","kind":"proof","title":"应用随机逼近极限定理","source_units":[{"start":53380,"end":53463,"start_line":1007,"end_line":1007,"raw_tex":"The following theorem follows directly from \\cite[Thm. 4.1]{majewski2018analysis}.\n","sha256_utf8":"d62dbe6fdf76df769063c2541be91a3293400abcebe9bfa2358917883305f9a1"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"由外部极限定理直接推出","source_units":[{"start":53380,"end":53463,"start_line":1007,"end_line":1007,"raw_tex":"The following theorem follows directly from \\cite[Thm. 4.1]{majewski2018analysis}.\n","sha256_utf8":"d62dbe6fdf76df769063c2541be91a3293400abcebe9bfa2358917883305f9a1"}]},{"id":"S053","semantic_type":"proof","kind":"proof","title":"微分包含的投影改写","source_units":[{"start":54026,"end":54497,"start_line":1019,"end_line":1025,"raw_tex":"Since any solution of \\eqref{diff-inc-beta} must lie in $\\Zb$,\nwe can  write the differential inclusion \\eqref{diff-inc-beta} in an equivalent way \\cite{aubin2012differential}:\n\\begin{equation}\n\\label{diff-inc-beta-T}\n\\dt{Z}_\\infty(t) \\in \\varPi_{\\Tc_{\\Zb}(Z_\\infty(t))}\\big( \\varGamma(Z_\\infty(t))\\big), \\quad t \\ge 0,\n\\end{equation}\nwhich underlines the nature of the limiting subgradient process. Here, $\\Tc_\\Zb(z)$ is the tangent cone to the convex set $\\Zb$ at $z$.\n","sha256_utf8":"73ff38aa47c9a615318045982c24b635990b7c0f37437d5e026810c6741f2be2"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"法锥包含转为切锥投影","source_units":[{"start":54026,"end":54497,"start_line":1019,"end_line":1025,"raw_tex":"Since any solution of \\eqref{diff-inc-beta} must lie in $\\Zb$,\nwe can  write the differential inclusion \\eqref{diff-inc-beta} in an equivalent way \\cite{aubin2012differential}:\n\\begin{equation}\n\\label{diff-inc-beta-T}\n\\dt{Z}_\\infty(t) \\in \\varPi_{\\Tc_{\\Zb}(Z_\\infty(t))}\\big( \\varGamma(Z_\\infty(t))\\big), \\quad t \\ge 0,\n\\end{equation}\nwhich underlines the nature of the limiting subgradient process. Here, $\\Tc_\\Zb(z)$ is the tangent cone to the convex set $\\Zb$ at $z$.\n","sha256_utf8":"73ff38aa47c9a615318045982c24b635990b7c0f37437d5e026810c6741f2be2"}]},{"id":"S055","semantic_type":"lemma","kind":"lemma","title":"势函数广义可微","source_units":[{"start":55647,"end":55812,"start_line":1052,"end_line":1055,"raw_tex":"\\begin{lemma}\n\\label{l:w-gendif}\nUnder Assumptions \\textup{(A2)--(A4)} the Lyapunov function $w(\\Cdot,\\Cdot)$ is differentiable in the generalized sense.\n\\end{lemma}","sha256_utf8":"08ef27d2ab1d55ac7ddd785c7b543abac0d4bb5e69e5e5b6b9d5e49d95a6d89f"}]},{"id":"S056","semantic_type":"proof","kind":"proof","title":"势函数广义可微性证明","source_units":[{"start":55813,"end":57406,"start_line":1056,"end_line":1086,"raw_tex":"\\begin{proof}\nIn \\S \\ref{s:2} we established the generalized differentiability of the functions $G(\\Cdot)$ and $Q(\\Cdot,\\Cdot)$.  It remains to consider the function\n\\[\n\\Delta^{0}(\\beta,\\theta) = \\Eb\\Big[g(F) - g(\\varPsi) - \\langle \\nabla g(\\varPsi), F - \\varPsi \\rangle \\Big],\n\\]\nand the expression under the expected value:\n\\begin{equation}\n    \\label{delta0-beta}\n\\delta^{0}(\\beta,\\theta) = g(F) - g(\\varPsi) - \\langle \\nabla g(\\varPsi), F - \\varPsi \\rangle.\n\\end{equation}\nBy Assumptions (A2)--(A4) and Theorem \\ref{t:composition},\n$\\delta^{0}(\\Cdot,\\Cdot)$ is Norkin differentiable. Its generalized subgradients with respect to $\\beta$ have the form\n\\[\n\\delta^{0}_\\beta = F_\\beta\\big( \\nabla g(F) - \\nabla g(\\Psi)\\big),\n\\]\nand are bounded by $2L_g\\Eb[\\widebar{L}_f(X,Y)|X]$. It is an integrable function,\nthanks to (A2).\nIn a similar way, by Assumptions (A2) and (A4) and Theorem \\ref{t:composition}, the generalized subgradients\nwith respect to $\\theta$ have the form\n\\begin{equation}\n    \\label{delta0-theta}\n\\delta^{0}_\\theta = \\varPsi_\\theta H(\\varPsi) ( F - \\Psi),\n\\end{equation}\n or are convex combinations of elements of this structure,\n where we denote by $H(\\varPsi)$\nan arbitrary element of $\\hat{\\partial}(\\nabla g)(\\varPsi)$. Thus, their norm is\nbounded by $L_{\\nabla g}L_\\varPsi(X)\\|F-\\varPsi\\|$. As $L_\\varPsi$\nand $\\|F-\\varPsi\\|$ are square-integrable, the bound is integrable, by virtue\nof the Cauchy--Schwartz inequality.  Applying Theorem \\ref{t:expected-generalized},\nwe conclude that the expected value function $\\Delta^0$ is Norkin differentiable as well.\n\\end{proof}","sha256_utf8":"078b9a7322db8444b37522db21ca725a20b1b345e8086cf86e0c9b1bab029060"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"归约到随机散度被积函数","source_units":[{"start":55813,"end":56290,"start_line":1056,"end_line":1065,"raw_tex":"\\begin{proof}\nIn \\S \\ref{s:2} we established the generalized differentiability of the functions $G(\\Cdot)$ and $Q(\\Cdot,\\Cdot)$.  It remains to consider the function\n\\[\n\\Delta^{0}(\\beta,\\theta) = \\Eb\\Big[g(F) - g(\\varPsi) - \\langle \\nabla g(\\varPsi), F - \\varPsi \\rangle \\Big],\n\\]\nand the expression under the expected value:\n\\begin{equation}\n    \\label{delta0-beta}\n\\delta^{0}(\\beta,\\theta) = g(F) - g(\\varPsi) - \\langle \\nabla g(\\varPsi), F - \\varPsi \\rangle.\n\\end{equation}\n","sha256_utf8":"35b6237f4c41e237e83634239d7cac5843597b863b63b4a0ec79847718055f87"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"决策参数导数及可积界","source_units":[{"start":56290,"end":56639,"start_line":1066,"end_line":1072,"raw_tex":"By Assumptions (A2)--(A4) and Theorem \\ref{t:composition},\n$\\delta^{0}(\\Cdot,\\Cdot)$ is Norkin differentiable. Its generalized subgradients with respect to $\\beta$ have the form\n\\[\n\\delta^{0}_\\beta = F_\\beta\\big( \\nabla g(F) - \\nabla g(\\Psi)\\big),\n\\]\nand are bounded by $2L_g\\Eb[\\widebar{L}_f(X,Y)|X]$. It is an integrable function,\nthanks to (A2).\n","sha256_utf8":"17a3845e21990cc7ba1373fe2078e820a664a4e753a7208ddb4560940114ca77"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"模型参数导数与期望交换","source_units":[{"start":56639,"end":57406,"start_line":1073,"end_line":1086,"raw_tex":"In a similar way, by Assumptions (A2) and (A4) and Theorem \\ref{t:composition}, the generalized subgradients\nwith respect to $\\theta$ have the form\n\\begin{equation}\n    \\label{delta0-theta}\n\\delta^{0}_\\theta = \\varPsi_\\theta H(\\varPsi) ( F - \\Psi),\n\\end{equation}\n or are convex combinations of elements of this structure,\n where we denote by $H(\\varPsi)$\nan arbitrary element of $\\hat{\\partial}(\\nabla g)(\\varPsi)$. Thus, their norm is\nbounded by $L_{\\nabla g}L_\\varPsi(X)\\|F-\\varPsi\\|$. As $L_\\varPsi$\nand $\\|F-\\varPsi\\|$ are square-integrable, the bound is integrable, by virtue\nof the Cauchy--Schwartz inequality.  Applying Theorem \\ref{t:expected-generalized},\nwe conclude that the expected value function $\\Delta^0$ is Norkin differentiable as well.\n\\end{proof}","sha256_utf8":"dffca383c4061cc46b07a1ff1a8f18c5a00d7b36bc2ac3d6b307946e082987a2"}]},{"id":"S060","semantic_type":"proof","kind":"proof","title":"势函数沿轨迹的链式推导","source_units":[{"start":57820,"end":58667,"start_line":1103,"end_line":1114,"raw_tex":"Following the differential inclusion method, consider the function $W(t) = w(\\beta(t),\\theta(t))$, $t \\ge 0$, where $(\\beta(\\Cdot),\\theta(\\Cdot)) \\triangleq Z(\\Cdot)$ is a solution of the\ninclusion \\eqref{diff-inc-beta}.\n\nAs the Lyapunov function $w(\\Cdot,\\Cdot)$ is Norkin differentiable, by virtue of Theorem \\ref{t:composition}, its\nweak derivative can be expressed by the chain rule:\n\\begin{equation}\n\\dt{W}\n= \\langle w_\\beta, \\dt{\\beta} \\rangle + \\langle w_\\theta, \\dt{\\theta} \\rangle\n= \\langle G_\\beta + \\alpha \\Delta^\\lambda_\\beta, \\dt{\\beta} \\rangle + \\alpha \\langle \\Delta^\\lambda_\\theta, \\dt{\\theta} \\rangle.\n\\label{grad:P}\n\\end{equation}\nDue to Theorem \\ref{t:chain}, the integral  $\\int_0^T \\dt{W}(t)\\,\\D t$ on any interval of time does not depend on the particular selections of the generalized subgradients used in this calculation.\n","sha256_utf8":"df23a6431335e7f7b81d37bbd73305aafbe89030ecbe8aef18a7952440b6578d"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"分解势函数弱导数","source_units":[{"start":57820,"end":58469,"start_line":1103,"end_line":1113,"raw_tex":"Following the differential inclusion method, consider the function $W(t) = w(\\beta(t),\\theta(t))$, $t \\ge 0$, where $(\\beta(\\Cdot),\\theta(\\Cdot)) \\triangleq Z(\\Cdot)$ is a solution of the\ninclusion \\eqref{diff-inc-beta}.\n\nAs the Lyapunov function $w(\\Cdot,\\Cdot)$ is Norkin differentiable, by virtue of Theorem \\ref{t:composition}, its\nweak derivative can be expressed by the chain rule:\n\\begin{equation}\n\\dt{W}\n= \\langle w_\\beta, \\dt{\\beta} \\rangle + \\langle w_\\theta, \\dt{\\theta} \\rangle\n= \\langle G_\\beta + \\alpha \\Delta^\\lambda_\\beta, \\dt{\\beta} \\rangle + \\alpha \\langle \\Delta^\\lambda_\\theta, \\dt{\\theta} \\rangle.\n\\label{grad:P}\n\\end{equation}\n","sha256_utf8":"8f6a7491e516b964db675ff4fcb04346d1b7393bde34dd962915732fab9489ba"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"积分不依赖次梯度选择","source_units":[{"start":58469,"end":58667,"start_line":1114,"end_line":1114,"raw_tex":"Due to Theorem \\ref{t:chain}, the integral  $\\int_0^T \\dt{W}(t)\\,\\D t$ on any interval of time does not depend on the particular selections of the generalized subgradients used in this calculation.\n","sha256_utf8":"ed4414da2cd808f900fb1196da9067ab746872f85895a282cafada11cedb5390"}]},{"id":"S063","semantic_type":"lemma","kind":"lemma","title":"模型导数交叉项界","source_units":[{"start":59479,"end":59682,"start_line":1134,"end_line":1137,"raw_tex":"\\begin{lemma}\nUnder Assumptions \\textup{(A2)} and \\textup{(A4)},  $\\big|\\langle \\Delta^0_\\theta, Q_\\theta \\rangle \\big| \\le 2 {L_{\\nabla g}} \\widebar{L}_\\varPsi^2 Q$.\n\\label{l:QG_delta_theta}\n\\end{lemma}","sha256_utf8":"671f7aff5a19a2b2685abfdb88508533df881a13ac4adbf6fce806a342244014"}]},{"id":"S064","semantic_type":"proof","kind":"proof","title":"模型导数交叉项估计证明","source_units":[{"start":59683,"end":60297,"start_line":1138,"end_line":1149,"raw_tex":"\\begin{proof}\nIn view of \\eqref{delta0-theta},  $\\Delta^0_\\theta = \\Eb\\big[ \\varPsi_\\theta H(\\varPsi)(F-\\varPsi)\\big]$. By virtue of Assumption (A2), $\\|H(\\varPsi)\\| \\le L_{\\nabla g}$.\nAlso,\n$Q_\\theta = \\Eb\\big[ \\varPsi_\\theta(F-\\varPsi)\\big]$.\nUsing the Cauchy--Schwartz inequality,\nwe obtain the following chain of estimates:\n\\begin{align*}\n\\big|\\langle \\Delta^0_\\theta, Q_\\theta \\rangle\\big|  &\\le \\| \\Delta^0_\\theta\\| \\| Q_\\theta \\|\n\\le {L_{\\nabla g}} \\Eb[\\|\\varPsi_\\theta\\|^2] \\Eb[\\|F - \\varPsi\\|^2] \\le 2 {L_{\\nabla g}} \\widebar{L}_\\varPsi^2 Q.\n\\end{align*}\nThe last inequality uses \\eqref{te:Q}.\n\\end{proof}","sha256_utf8":"ccc68250f0e505c53e6765feeff2e7441855ab16f47ba5cb72eb0f5bfa03191c"}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","title":"写出散度及误差模型导数","source_units":[{"start":59683,"end":60011,"start_line":1138,"end_line":1143,"raw_tex":"\\begin{proof}\nIn view of \\eqref{delta0-theta},  $\\Delta^0_\\theta = \\Eb\\big[ \\varPsi_\\theta H(\\varPsi)(F-\\varPsi)\\big]$. By virtue of Assumption (A2), $\\|H(\\varPsi)\\| \\le L_{\\nabla g}$.\nAlso,\n$Q_\\theta = \\Eb\\big[ \\varPsi_\\theta(F-\\varPsi)\\big]$.\nUsing the Cauchy--Schwartz inequality,\nwe obtain the following chain of estimates:\n","sha256_utf8":"a3a27e71aac5a34be6fc0fb3f98d10351b0806c7a59151a50b6c28cb142cdaac"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"柯西不等式给出交叉项界","source_units":[{"start":60011,"end":60297,"start_line":1144,"end_line":1149,"raw_tex":"\\begin{align*}\n\\big|\\langle \\Delta^0_\\theta, Q_\\theta \\rangle\\big|  &\\le \\| \\Delta^0_\\theta\\| \\| Q_\\theta \\|\n\\le {L_{\\nabla g}} \\Eb[\\|\\varPsi_\\theta\\|^2] \\Eb[\\|F - \\varPsi\\|^2] \\le 2 {L_{\\nabla g}} \\widebar{L}_\\varPsi^2 Q.\n\\end{align*}\nThe last inequality uses \\eqref{te:Q}.\n\\end{proof}","sha256_utf8":"d5e4097d6d8c0794ef8254aa7bc1a4a33138461692bb7c81b7b0f20f61b7e12d"}]},{"id":"S067","semantic_type":"lemma","kind":"lemma","title":"误差决策导数范数界","source_units":[{"start":60299,"end":60418,"start_line":1151,"end_line":1154,"raw_tex":"\\begin{lemma}\nUnder Assumption \\textup{(A3)},  $\\|Q_\\beta\\|^2 \\le 2 \\widebar{L}_f^2 Q$.\n\\label{l:QG_Q_beta}\n\\end{lemma}","sha256_utf8":"88a4ad109470538c67693c19df381be449463c7a398eff556f528806f597182e"}]},{"id":"S068","semantic_type":"proof","kind":"proof","title":"误差决策导数界证明","source_units":[{"start":60419,"end":60628,"start_line":1155,"end_line":1161,"raw_tex":"\\begin{proof}\nBy the  Cauchy--Schwartz inequality,\n\n\\[\n\\|Q_\\beta\\| = \\big\\|\\Eb[ F_\\beta(F-\\varPsi)]\\big\\| \\le \\sqrt{\\Eb[\\|F_\\beta\\|^2]}\\sqrt{ \\Eb[\\|F-\\varPsi\\|^2]} \\le  \\widebar{L}_f \\sqrt{2 Q}.\n\\]\n\\end{proof}","sha256_utf8":"008f851cdcbca36d54653c002f77877784a81e869342695fdc607432a93e541b"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"由矩界控制误差导数","source_units":[{"start":60419,"end":60628,"start_line":1155,"end_line":1161,"raw_tex":"\\begin{proof}\nBy the  Cauchy--Schwartz inequality,\n\n\\[\n\\|Q_\\beta\\| = \\big\\|\\Eb[ F_\\beta(F-\\varPsi)]\\big\\| \\le \\sqrt{\\Eb[\\|F_\\beta\\|^2]}\\sqrt{ \\Eb[\\|F-\\varPsi\\|^2]} \\le  \\widebar{L}_f \\sqrt{2 Q}.\n\\]\n\\end{proof}","sha256_utf8":"008f851cdcbca36d54653c002f77877784a81e869342695fdc607432a93e541b"}]},{"id":"S070","semantic_type":"lemma","kind":"lemma","title":"散度决策导数范数界","source_units":[{"start":60629,"end":60792,"start_line":1162,"end_line":1165,"raw_tex":"\\begin{lemma}\nUnder Assumptions \\textup{(A2)} and \\textup{(A3)}, $\\|\\Delta^0_\\beta\\|^2 \\le 2 L_{\\nabla g}^2 \\widebar{L}_f^2 Q$.\n\\label{l:QG_delta_beta}\n\\end{lemma}","sha256_utf8":"bebb85206d956449996e73249a408c73802fdd11af2c240226cfc52422db87d6"}]},{"id":"S071","semantic_type":"proof","kind":"proof","title":"散度决策导数界证明","source_units":[{"start":60793,"end":61373,"start_line":1166,"end_line":1176,"raw_tex":"\\begin{proof}\nBy \\eqref{delta0-beta}, $\\Delta^0_\\beta = \\Eb[F_\\beta(\\nabla g(F) - \\nabla g(\\varPsi))$.\nSimilar to the proof of Lemma \\ref{l:QG_Q_beta}, by the Cauchy--Schwartz inequality and Assumption (A2),\n\\begin{align*}\n\\|\\Delta^0_\\beta\\|^2 &= \\big\\|\\Eb[F_\\beta(\\nabla g(F) - \\nabla g(\\varPsi)) ]\\big\\|^2\n\\le \\Eb[\\|F_\\beta\\|^2]\\, \\Eb[\\|\\nabla g(F) - \\nabla g(\\varPsi)\\|^2]  \\\\\n&\\le L_{\\nabla g}^2 \\Eb[\\|F_\\beta\\|^2]\\, \\Eb[\\|F - \\varPsi\\|^2]\n\\le 2L_{\\nabla g}^2 \\widebar{L}_f^2 Q.\n\\end{align*}\nThe last inequality uses Assumption (A3), as in Lemma \\ref{l:QG_Q_beta}.\n\\end{proof}","sha256_utf8":"3b5c852d77f7dfea84c9829ab778702de1f9b23e1a148cf3d21e143f0522a44d"}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","title":"散度导数写为梯度差期望","source_units":[{"start":60793,"end":61001,"start_line":1166,"end_line":1168,"raw_tex":"\\begin{proof}\nBy \\eqref{delta0-beta}, $\\Delta^0_\\beta = \\Eb[F_\\beta(\\nabla g(F) - \\nabla g(\\varPsi))$.\nSimilar to the proof of Lemma \\ref{l:QG_Q_beta}, by the Cauchy--Schwartz inequality and Assumption (A2),\n","sha256_utf8":"27e59a3fc99619843b24f44de7e17192e532f24c7fdc3240bebee53e4e26865b"}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","title":"光滑性与矩界控制梯度差","source_units":[{"start":61001,"end":61373,"start_line":1169,"end_line":1176,"raw_tex":"\\begin{align*}\n\\|\\Delta^0_\\beta\\|^2 &= \\big\\|\\Eb[F_\\beta(\\nabla g(F) - \\nabla g(\\varPsi)) ]\\big\\|^2\n\\le \\Eb[\\|F_\\beta\\|^2]\\, \\Eb[\\|\\nabla g(F) - \\nabla g(\\varPsi)\\|^2]  \\\\\n&\\le L_{\\nabla g}^2 \\Eb[\\|F_\\beta\\|^2]\\, \\Eb[\\|F - \\varPsi\\|^2]\n\\le 2L_{\\nabla g}^2 \\widebar{L}_f^2 Q.\n\\end{align*}\nThe last inequality uses Assumption (A3), as in Lemma \\ref{l:QG_Q_beta}.\n\\end{proof}","sha256_utf8":"0a3e3ebfe889a463b5951492ef6934e06d6347321fccde71f9c6fcf60d2c9fa6"}]},{"id":"S074","semantic_type":"proposition","kind":"proposition","title":"势函数决策分量下降界","source_units":[{"start":61796,"end":62178,"start_line":1188,"end_line":1198,"raw_tex":"\\begin{proposition}\nUnder assumptions \\textup{(A2)} and \\textup{(A3)},\n\\[\n\\langle w_\\beta, \\dt{\\beta} \\rangle \\le \\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 Q.\n\\]\nFurthermore, if $Q=0$, then\n\\[\n\\langle w_\\beta, \\dt{\\beta} \\rangle = - \\big\\| \\varPi_{\\Tc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big)\\big\\|^2.\n\\]\n\\label{p:QG_L_delta_Q_beta}\n\\end{proposition}","sha256_utf8":"1c3f429179ccf1800241b2573cbc3e40b20a08280ca8e110c3a08dcb5daa6b5f"}]},{"id":"S075","semantic_type":"proof","kind":"proof","title":"决策分量下降估计证明","source_units":[{"start":62179,"end":66281,"start_line":1199,"end_line":1255,"raw_tex":"\\begin{proof}\nThe first coordinate of the differential inclusion \\eqref{diff-inc-beta} reads:\n\\begin{equation}\n\\dt{\\beta} \\in \\varPi_{\\Tc_{\\Bb}(\\beta)}\\big(-\\Eb[F_\\beta \\nabla g(\\varPsi) ] \\big) = \\varPi_{\\Tc_{\\Bb}(\\beta)}\\big( -G_\\beta + \\Delta^{0}_\\beta \\big),\n\\label{grad:beta}\n\\end{equation}\nFor any solution $(\\beta(t), \\theta(t))$ of the inclusion \\eqref{t:diff-inc-beta}, with the corresponding element $N(t) \\in \\Nc_{\\Bb}(\\beta(t))$, we define the projection matrix\n\\[\nP(t)  = I - \\frac{N(t) N(t)^T}{\\|N(t)\\|^2}.\n\\]\nThen\n\\begin{equation}\n\\dt{\\beta} = P (-G_\\beta + \\Delta^{0}_\\beta).\n\\label{grad:beta_proj}\n\\end{equation}\nUsing the facts that $P$ is nonexpansive, $P^T=P$, and $P^2=P$,\nby elementary manipulations, we obtain an upper bound:\n\\begin{equation}\n\\label{w-dot-chain}\n\\begin{aligned}\n\\lefteqn{\\langle w_\\beta, \\dt{\\beta} \\rangle = -\\langle G_\\beta + \\alpha \\Delta^{\\lambda}_\\beta, P(G_\\beta - \\Delta^{0}_\\beta) \\rangle} \\\\\n&= -\\Big\\|P\\big(G_\\beta + \\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta - \\frac{1}{2} \\Delta^0_\\beta\\big)\\Big\\|^2 + \\Big\\|P\\big(\\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta + \\frac{1}{2} \\Delta^0_\\beta\\big)\\Big\\|^2 \\\\\n&\\le \\Big\\|P\\big(\\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta + \\frac{1}{2} \\Delta^0_\\beta\\big) \\Big\\|^2 =\n\\Big\\|P\\big(\\frac{\\alpha+1}{2} \\Delta^0_\\beta + \\frac{\\alpha \\lambda}{2} Q_\\beta\\big)\\Big\\|^2 \\\\\n&= \\Big\\|\\frac{\\alpha+1}{2} P \\Delta^0_\\beta\\Big\\|^2 + \\Big\\|\\frac{\\alpha \\lambda}{2} P Q_\\beta\\Big\\|^2 + 2 \\Big\\langle \\rho \\frac{\\alpha+1}{2} P \\Delta^0_\\beta, \\varrho^{-1} \\frac{\\alpha \\lambda}{2} P Q_\\beta \\Big\\rangle\\\\\n&\\le \\frac{(\\alpha+1)^2}{4} (1+\\rho^2)\\|P \\Delta^0_\\beta\\|^2 + \\frac{\\alpha^2 \\lambda^2}{4}(1+\\rho^{-2})\\|P Q_\\beta\\|^2,\n\\end{aligned}\n\\end{equation}\nfor any $\\rho>0$.\nLemmas \\ref{l:QG_Q_beta} and \\ref{l:QG_delta_beta} allow us to continue the last estimate as follows:\n\\begin{align*}\n\\langle w_\\beta, \\dt{\\beta} \\rangle &\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{4} (1+\\rho^2)\\|P \\Delta^0_\\beta\\|^2 + \\frac{\\alpha^2 \\lambda^2}{4}(1+\\rho^{-2})\\|P Q_\\beta\\|^2 \\Big] \\\\\n&\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{4} (1+\\rho^2)\\|\\Delta^0_\\beta\\|^2 + \\frac{\\alpha^2 \\lambda^2}{4}(1+\\rho^{-2})\\|Q_\\beta\\|^2 \\Big] \\\\\n&\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{2} (1+\\rho^2) L_{\\nabla g}^2 + \\frac{\\alpha^2 \\lambda^2}{2}(1+\\rho^{-2}) \\Big]   \\widebar{L}_f^2 Q \\\\\n&= \\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 Q.\n\\end{align*}\nThis proves the first assertion.\n\\begin{comment}\n\\begin{align*}\n\\langle w_\\beta, \\dt{\\beta} \\rangle &= -\\langle G_\\beta + \\alpha \\Delta^{\\lambda}_\\beta, P(G_\\beta - \\Delta^{0}_\\beta) \\rangle \\\\\n&= -\\Big\\|G_\\beta + \\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta - \\frac{1}{2} \\Delta^0_\\beta\\Big\\|_P^2 + \\Big\\|\\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta + \\frac{1}{2} \\Delta^0_\\beta\\Big\\|_P^2 \\\\\n&\\le \\Big\\|\\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta + \\frac{1}{2} \\Delta^0_\\beta\\Big\\|_P^2 =\n\\Big\\|\\frac{\\alpha+1}{2} \\Delta^0_\\beta + \\frac{\\alpha \\lambda}{2} Q_\\beta\\Big\\|_P^2 \\\\\n&= \\Big\\|\\frac{\\alpha+1}{2} \\Delta^0_\\beta\\Big\\|_P^2 + \\Big\\|\\frac{\\alpha \\lambda}{2} Q_\\beta\\Big\\|_P^2 + 2 \\Big\\langle \\rho \\frac{\\alpha+1}{2} \\Delta^0_\\beta, \\varrho^{-1} \\frac{\\alpha \\lambda}{2} P Q_\\beta \\Big\\rangle \\\\\n&\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{4} (1+\\rho^2)\\|\\Delta^0_\\beta\\|_P^2 + \\frac{\\alpha^2 \\lambda^2}{4}(1+\\rho^{-2})\\|Q_\\beta\\|_P^2 \\Big] \\\\\n&\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{4} (1+\\rho^2)\\|\\Delta^0_\\beta\\|^2 + \\frac{\\alpha^2 \\lambda^2}{4}(1+\\rho^{-2})\\|Q_\\beta\\|^2 \\Big] \\\\\n&\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{2} (1+\\rho^2) {L_{\\nabla g}}^2 + \\frac{\\alpha^2 \\lambda^2}{2}(1+\\rho^{-2}) \\Big]   \\widebar{L}_f^2 M \\|Q_\\theta\\|^2 \\\\\n&= \\frac{\\widebar{L}_f^2 M}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 \\|Q_\\theta\\|^2.\n\\end{align*}\n\\end{comment}\nNow, suppose $Q=0$. This implies that $\\varDelta^\\lambda=0$ for all $\\lambda \\ge 0$.\nThe first line of \\eqref{w-dot-chain} yields\n\\[\n\\langle w_\\beta, \\dt{\\beta} \\rangle = - \\| P G_\\beta \\|^2 = - \\big\\| \\varPi_{\\Tc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big)\\big\\|^2,\n\\]\nas claimed.\n\\end{proof}","sha256_utf8":"b9002b79c181a96ac70f19ad25513cb942852163824b4c4edc6887533baac816"}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","title":"投影决策动力学的矩阵表示","source_units":[{"start":62179,"end":62809,"start_line":1199,"end_line":1213,"raw_tex":"\\begin{proof}\nThe first coordinate of the differential inclusion \\eqref{diff-inc-beta} reads:\n\\begin{equation}\n\\dt{\\beta} \\in \\varPi_{\\Tc_{\\Bb}(\\beta)}\\big(-\\Eb[F_\\beta \\nabla g(\\varPsi) ] \\big) = \\varPi_{\\Tc_{\\Bb}(\\beta)}\\big( -G_\\beta + \\Delta^{0}_\\beta \\big),\n\\label{grad:beta}\n\\end{equation}\nFor any solution $(\\beta(t), \\theta(t))$ of the inclusion \\eqref{t:diff-inc-beta}, with the corresponding element $N(t) \\in \\Nc_{\\Bb}(\\beta(t))$, we define the projection matrix\n\\[\nP(t)  = I - \\frac{N(t) N(t)^T}{\\|N(t)\\|^2}.\n\\]\nThen\n\\begin{equation}\n\\dt{\\beta} = P (-G_\\beta + \\Delta^{0}_\\beta).\n\\label{grad:beta_proj}\n\\end{equation}\n","sha256_utf8":"1c344c50af488945e039a457a60e40bc020d594903600c11d66f925105e59ab0"}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","title":"配方及交叉项参数估计","source_units":[{"start":62809,"end":63918,"start_line":1214,"end_line":1227,"raw_tex":"Using the facts that $P$ is nonexpansive, $P^T=P$, and $P^2=P$,\nby elementary manipulations, we obtain an upper bound:\n\\begin{equation}\n\\label{w-dot-chain}\n\\begin{aligned}\n\\lefteqn{\\langle w_\\beta, \\dt{\\beta} \\rangle = -\\langle G_\\beta + \\alpha \\Delta^{\\lambda}_\\beta, P(G_\\beta - \\Delta^{0}_\\beta) \\rangle} \\\\\n&= -\\Big\\|P\\big(G_\\beta + \\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta - \\frac{1}{2} \\Delta^0_\\beta\\big)\\Big\\|^2 + \\Big\\|P\\big(\\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta + \\frac{1}{2} \\Delta^0_\\beta\\big)\\Big\\|^2 \\\\\n&\\le \\Big\\|P\\big(\\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta + \\frac{1}{2} \\Delta^0_\\beta\\big) \\Big\\|^2 =\n\\Big\\|P\\big(\\frac{\\alpha+1}{2} \\Delta^0_\\beta + \\frac{\\alpha \\lambda}{2} Q_\\beta\\big)\\Big\\|^2 \\\\\n&= \\Big\\|\\frac{\\alpha+1}{2} P \\Delta^0_\\beta\\Big\\|^2 + \\Big\\|\\frac{\\alpha \\lambda}{2} P Q_\\beta\\Big\\|^2 + 2 \\Big\\langle \\rho \\frac{\\alpha+1}{2} P \\Delta^0_\\beta, \\varrho^{-1} \\frac{\\alpha \\lambda}{2} P Q_\\beta \\Big\\rangle\\\\\n&\\le \\frac{(\\alpha+1)^2}{4} (1+\\rho^2)\\|P \\Delta^0_\\beta\\|^2 + \\frac{\\alpha^2 \\lambda^2}{4}(1+\\rho^{-2})\\|P Q_\\beta\\|^2,\n\\end{aligned}\n\\end{equation}\nfor any $\\rho>0$.\n","sha256_utf8":"794f63bbaec02af790e42f6cc44ed62e1d7a14d43b0de0aa75ab110b4b3f6176"}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","title":"代入导数界并优化参数","source_units":[{"start":63918,"end":65999,"start_line":1228,"end_line":1248,"raw_tex":"Lemmas \\ref{l:QG_Q_beta} and \\ref{l:QG_delta_beta} allow us to continue the last estimate as follows:\n\\begin{align*}\n\\langle w_\\beta, \\dt{\\beta} \\rangle &\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{4} (1+\\rho^2)\\|P \\Delta^0_\\beta\\|^2 + \\frac{\\alpha^2 \\lambda^2}{4}(1+\\rho^{-2})\\|P Q_\\beta\\|^2 \\Big] \\\\\n&\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{4} (1+\\rho^2)\\|\\Delta^0_\\beta\\|^2 + \\frac{\\alpha^2 \\lambda^2}{4}(1+\\rho^{-2})\\|Q_\\beta\\|^2 \\Big] \\\\\n&\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{2} (1+\\rho^2) L_{\\nabla g}^2 + \\frac{\\alpha^2 \\lambda^2}{2}(1+\\rho^{-2}) \\Big]   \\widebar{L}_f^2 Q \\\\\n&= \\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 Q.\n\\end{align*}\nThis proves the first assertion.\n\\begin{comment}\n\\begin{align*}\n\\langle w_\\beta, \\dt{\\beta} \\rangle &= -\\langle G_\\beta + \\alpha \\Delta^{\\lambda}_\\beta, P(G_\\beta - \\Delta^{0}_\\beta) \\rangle \\\\\n&= -\\Big\\|G_\\beta + \\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta - \\frac{1}{2} \\Delta^0_\\beta\\Big\\|_P^2 + \\Big\\|\\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta + \\frac{1}{2} \\Delta^0_\\beta\\Big\\|_P^2 \\\\\n&\\le \\Big\\|\\frac{\\alpha}{2} \\Delta^{\\lambda}_\\beta + \\frac{1}{2} \\Delta^0_\\beta\\Big\\|_P^2 =\n\\Big\\|\\frac{\\alpha+1}{2} \\Delta^0_\\beta + \\frac{\\alpha \\lambda}{2} Q_\\beta\\Big\\|_P^2 \\\\\n&= \\Big\\|\\frac{\\alpha+1}{2} \\Delta^0_\\beta\\Big\\|_P^2 + \\Big\\|\\frac{\\alpha \\lambda}{2} Q_\\beta\\Big\\|_P^2 + 2 \\Big\\langle \\rho \\frac{\\alpha+1}{2} \\Delta^0_\\beta, \\varrho^{-1} \\frac{\\alpha \\lambda}{2} P Q_\\beta \\Big\\rangle \\\\\n&\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{4} (1+\\rho^2)\\|\\Delta^0_\\beta\\|_P^2 + \\frac{\\alpha^2 \\lambda^2}{4}(1+\\rho^{-2})\\|Q_\\beta\\|_P^2 \\Big] \\\\\n&\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{4} (1+\\rho^2)\\|\\Delta^0_\\beta\\|^2 + \\frac{\\alpha^2 \\lambda^2}{4}(1+\\rho^{-2})\\|Q_\\beta\\|^2 \\Big] \\\\\n&\\le \\min_{\\rho > 0} \\Big[ \\frac{(\\alpha+1)^2}{2} (1+\\rho^2) {L_{\\nabla g}}^2 + \\frac{\\alpha^2 \\lambda^2}{2}(1+\\rho^{-2}) \\Big]   \\widebar{L}_f^2 M \\|Q_\\theta\\|^2 \\\\\n&= \\frac{\\widebar{L}_f^2 M}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 \\|Q_\\theta\\|^2.\n\\end{align*}\n\\end{comment}\n","sha256_utf8":"572610db8d4acc4e1a1ecbb4d2ae3f3d2fb562c9322d51bba62c640144f9d7ff"}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","title":"零误差时恢复投影梯度下降","source_units":[{"start":65999,"end":66281,"start_line":1249,"end_line":1255,"raw_tex":"Now, suppose $Q=0$. This implies that $\\varDelta^\\lambda=0$ for all $\\lambda \\ge 0$.\nThe first line of \\eqref{w-dot-chain} yields\n\\[\n\\langle w_\\beta, \\dt{\\beta} \\rangle = - \\| P G_\\beta \\|^2 = - \\big\\| \\varPi_{\\Tc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big)\\big\\|^2,\n\\]\nas claimed.\n\\end{proof}","sha256_utf8":"dedb509f78ed9535383348ee510e5b1b7abcbe3f6524ca338c06615a28c13988"}]},{"id":"S080","semantic_type":"proof","kind":"proof","title":"模型动力学去除边界投影","source_units":[{"start":67462,"end":67890,"start_line":1273,"end_line":1279,"raw_tex":"The second coordinate of the differential inclusion \\eqref{diff-inc-beta} reads:\n\\begin{equation}\n\\dt{\\theta} =  \\varPi_{\\Tc_{\\Theta_R}(\\theta)}\\big(\\gamma\\, \\Eb[(F - \\varPsi)^T \\varPsi_\\theta] \\big)\n= \\varPi_{\\Tc_{\\Theta_R}(\\theta)}\\big( -\\gamma Q_\\theta\\big) = -\\gamma Q_\\theta,\n\\label{grad:simp}\n\\end{equation}\nbecause, by virtue of  Assumption (A6), when $\\|\\theta\\|=R$, then $ -\\gamma Q_\\theta \\in \\Tc_{\\Theta_R}(\\theta)$.\n","sha256_utf8":"b842566d0f073afb76e2721c12a6ca0160ab1356332357270e426b1af0ad734b"}]},{"id":"S081","semantic_type":"proof_step","kind":"proof_step","title":"径向条件保证模型方向内向","source_units":[{"start":67462,"end":67890,"start_line":1273,"end_line":1279,"raw_tex":"The second coordinate of the differential inclusion \\eqref{diff-inc-beta} reads:\n\\begin{equation}\n\\dt{\\theta} =  \\varPi_{\\Tc_{\\Theta_R}(\\theta)}\\big(\\gamma\\, \\Eb[(F - \\varPsi)^T \\varPsi_\\theta] \\big)\n= \\varPi_{\\Tc_{\\Theta_R}(\\theta)}\\big( -\\gamma Q_\\theta\\big) = -\\gamma Q_\\theta,\n\\label{grad:simp}\n\\end{equation}\nbecause, by virtue of  Assumption (A6), when $\\|\\theta\\|=R$, then $ -\\gamma Q_\\theta \\in \\Tc_{\\Theta_R}(\\theta)$.\n","sha256_utf8":"b842566d0f073afb76e2721c12a6ca0160ab1356332357270e426b1af0ad734b"}]},{"id":"S082","semantic_type":"theorem","kind":"theorem","title":"目标集合外势函数严格下降","source_units":[{"start":67892,"end":68358,"start_line":1282,"end_line":1291,"raw_tex":"\\begin{theorem}\n\\label{t:negative-der}\nUnder Assumptions \\textup{(A1)--(A6)}, if\n\\begin{equation}\n\\label{negative-der}\n\\lambda > 2 \\widebar{L}_\\varPsi^2 M {L_{\\nabla g}} \\quad \\text{and} \\quad\n\\gamma > \\frac{\\widebar{L}_f^2 M\\big( (\\alpha + 1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2}{2 \\alpha \\big(\\lambda - 2 \\widebar{L}_\\varPsi^2 M {L_{\\nabla g}} \\big)} ,\n\\end{equation}\nthen $\\dt{W} \\le 0$. Furthermore, $\\dt{W} < 0$ if $(\\beta,\\theta)\\notin \\Zb^*$.\n\\end{theorem}","sha256_utf8":"bd928770065033eb15d84a5377fdc52bc5401a701046a2317402e508c5e60e20"}]},{"id":"S083","semantic_type":"proof","kind":"proof","title":"严格下降定理证明","source_units":[{"start":68359,"end":70049,"start_line":1292,"end_line":1314,"raw_tex":"\\begin{proof}\nFollowing \\eqref{grad:P}, \\eqref{grad:simp}, Lemma \\ref{l:QG_delta_theta}, and Proposition \\ref{p:QG_L_delta_Q_beta},\n\\begin{align*}\n\\dt{W}\n&\\le \\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 Q - \\alpha \\gamma \\langle \\Delta^{\\lambda}_\\theta, Q_\\theta \\rangle \\\\\n&= \\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 Q - \\alpha \\gamma \\big[\\langle \\Delta^0_\\theta, Q_\\theta \\rangle + \\lambda \\|Q_\\theta\\|^2 \\big] \\\\\n&\\le \\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 Q + \\alpha \\gamma |\\langle \\Delta^0_\\theta, Q_\\theta \\rangle| - \\alpha \\gamma \\lambda \\|Q_\\theta\\|^2 \\\\\n&\\le \\Big[\\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 + \\alpha \\gamma \\cdot 2{L_{\\nabla g}} \\widebar{L}_\\varPsi^2 - \\frac{\\alpha \\gamma \\lambda}{M} \\Big] Q \\\\\n&= \\frac{1}{M} \\Big[\\frac{\\widebar{L}_f^2 M}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 - \\alpha \\gamma \\big(\\lambda  - 2 \\widebar{L}_\\varPsi^2 M{L_{\\nabla g}} \\big) \\Big] Q \\le 0.\n\\end{align*}\nThis verifies the first assertion.\nNow, suppose $\\dt{W}=0$. It follows from the last inequality that $Q=0$. Then, Proposition \\ref{p:QG_L_delta_Q_beta} implies that\n\\[\n\\varPi_{\\Tc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big) = 0\n\\]\nfor some $G_\\beta \\in \\hat{\\partial}{G}(\\beta)$. Consequently, using the decomposition into the projections on polar cones,\nwe obtain\n\\[\n-G_\\beta = \\varPi_{\\Tc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big) + \\varPi_{\\Nc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big) =\n\\varPi_{\\Nc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big).\n\\]\nThis means that the optimality condition \\eqref{optimality} is satisfied.\n\\end{proof}","sha256_utf8":"db93443522da7f4802de6ee8e0598df18c0c149631f1db142bd17c5454a99442"}]},{"id":"S084","semantic_type":"proof_step","kind":"proof_step","title":"合并分量估计与增长条件","source_units":[{"start":68359,"end":69488,"start_line":1292,"end_line":1302,"raw_tex":"\\begin{proof}\nFollowing \\eqref{grad:P}, \\eqref{grad:simp}, Lemma \\ref{l:QG_delta_theta}, and Proposition \\ref{p:QG_L_delta_Q_beta},\n\\begin{align*}\n\\dt{W}\n&\\le \\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 Q - \\alpha \\gamma \\langle \\Delta^{\\lambda}_\\theta, Q_\\theta \\rangle \\\\\n&= \\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 Q - \\alpha \\gamma \\big[\\langle \\Delta^0_\\theta, Q_\\theta \\rangle + \\lambda \\|Q_\\theta\\|^2 \\big] \\\\\n&\\le \\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 Q + \\alpha \\gamma |\\langle \\Delta^0_\\theta, Q_\\theta \\rangle| - \\alpha \\gamma \\lambda \\|Q_\\theta\\|^2 \\\\\n&\\le \\Big[\\frac{\\widebar{L}_f^2}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 + \\alpha \\gamma \\cdot 2{L_{\\nabla g}} \\widebar{L}_\\varPsi^2 - \\frac{\\alpha \\gamma \\lambda}{M} \\Big] Q \\\\\n&= \\frac{1}{M} \\Big[\\frac{\\widebar{L}_f^2 M}{2} \\big( (\\alpha+1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2 - \\alpha \\gamma \\big(\\lambda  - 2 \\widebar{L}_\\varPsi^2 M{L_{\\nabla g}} \\big) \\Big] Q \\le 0.\n\\end{align*}\nThis verifies the first assertion.\n","sha256_utf8":"7b2b46861340f5fca9e928aae3f6e4d80665fb49473a9808c0d0f760ce4961f7"}]},{"id":"S085","semantic_type":"proof_step","kind":"proof_step","title":"零导数推出精确跟踪及驻点","source_units":[{"start":69488,"end":70049,"start_line":1303,"end_line":1314,"raw_tex":"Now, suppose $\\dt{W}=0$. It follows from the last inequality that $Q=0$. Then, Proposition \\ref{p:QG_L_delta_Q_beta} implies that\n\\[\n\\varPi_{\\Tc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big) = 0\n\\]\nfor some $G_\\beta \\in \\hat{\\partial}{G}(\\beta)$. Consequently, using the decomposition into the projections on polar cones,\nwe obtain\n\\[\n-G_\\beta = \\varPi_{\\Tc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big) + \\varPi_{\\Nc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big) =\n\\varPi_{\\Nc_{\\Bb}(\\beta)}\\big(-G_\\beta \\big).\n\\]\nThis means that the optimality condition \\eqref{optimality} is satisfied.\n\\end{proof}","sha256_utf8":"ed2f9ad9e058eebc1fe4b089ad19a1059e7482d441dc5a12c1be0b516dba7e19"}]},{"id":"S086","semantic_type":"assumption","kind":"assumption","title":"严格下降的参数条件","source_units":[{"start":67931,"end":68265,"start_line":1284,"end_line":1289,"raw_tex":"Under Assumptions \\textup{(A1)--(A6)}, if\n\\begin{equation}\n\\label{negative-der}\n\\lambda > 2 \\widebar{L}_\\varPsi^2 M {L_{\\nabla g}} \\quad \\text{and} \\quad\n\\gamma > \\frac{\\widebar{L}_f^2 M\\big( (\\alpha + 1){L_{\\nabla g}} + \\alpha \\lambda \\big)^2}{2 \\alpha \\big(\\lambda - 2 \\widebar{L}_\\varPsi^2 M {L_{\\nabla g}} \\big)} ,\n\\end{equation}\n","sha256_utf8":"6d684c28a641be0918a34554292b3ed934db09b4c9529720ddd104c05531a5b1"}]},{"id":"S087","semantic_type":"proof","kind":"proof","title":"随机算法收敛的完整证明","source_units":[{"start":72919,"end":76318,"start_line":1358,"end_line":1414,"raw_tex":"\\begin{proof}\nThe proof follows the steps of the proofs of \\cite[Thm. 3.20]{duchi2018stochastic} or \\cite[Thm. 3.5]{majewski2018analysis} {  or \\cite{nurminskii1972convergence}}.\nWe provide a short outline here, for the convenience of the Reader. For brevity, we omit the\nargument $\\omega$, but we stress that we consider a particular path of the method.\n\nDefine\n\\[\n\\underline{w} = \\liminf_{k \\to \\infty^{\\phantom{|}}} w(z^k), \\qquad \\overline{w} = \\limsup_{k \\to \\infty}w(z^k).\n\\]\nSuppose $\\underline{w} < \\overline{w}$. By Assumption (A7), a value ${v}\\in\n(\\underline{w},\\overline{w})$ exists,\nsuch that ${v} \\notin G(\\Bb^*)$. Take any $u \\in ({v},\\overline{w})$.\nDefine $L_1=\\min\\big\\{k : w(z^k) \\le {v} \\big\\}$ and for\n$j=1,2,\\dots$\n\\begin{align*}\n\\ell_j &= \\min\\big\\{k\\ge L_j : w(z^k) > u \\big\\},\\\\\nL_{j+1} &= \\min\\big\\{k\\ge \\ell_j : w(z^k) \\le {v} \\big\\}.\\\\\n\\intertext{Then we define}\nk_j &= \\max\\big\\{ k\\in [L_j,\\ell_j]: w(z^k) \\le {v} \\big\\}, \\quad j=1,2,\\dots.\n\\end{align*}\nBoth $k_j\\to \\infty$ and $\\ell_j\\to \\infty$, as $j\\to \\infty$.\n\nConsider the sums\n$T_j = \\sum_{k=k_j}^{\\ell_j-1} \\tau_k$. Due to the equicontinuity of the functions\n $w(Z_j(\\Cdot))$, a constant $T >0$ exists, such that $T_j\\ge T$ for all sufficiently\nlarge $j$. Set $s_j=t_{k_j}$ in \\eqref{Z-shifted} and consider the interval $[0,T]$.\nBy Theorem \\ref{t:diff-inc-beta}, a subsequence of the sequence $\\{Z_j(\\Cdot)\\}$ is convergent to a solution $Z_\\infty(\\Cdot)$ of the differential\ninclusion \\eqref{diff-inc-beta}. For simplicity, we still denote it by $\\{Z_j(\\Cdot)\\}$. As $W(\\Cdot)$ is absolutely continuous,\n\\begin{equation}\n\\label{gradient-desc-cont}\nw(Z_\\infty(T)) = w(Z_\\infty(0)) + \\int_0^T  \\dt{W}(t)  \\;\\D t.\n\\end{equation}\nSince ${v} \\notin w(\\Zb^*)$, then  $Z_\\infty(0) = \\lim_{j\\to \\infty} z^{k_j} \\notin \\Zb^*$.\nTheorem \\ref{t:negative-der} then implies that $\\dt{W}(0)<0$.\nThe generalized subdifferential $\\hat{\\partial} G(\\Cdot)$ and the normal cone $\\Nc_\\Bb(\\Cdot)$ are upper semicontinuous and closed-valued. The function $Q(\\Cdot,\\Cdot)$ is continuous.\nTherefore, $\\dt{W}(t)<0$\n in some interval $t\\in [0,\\bar{t}\\,]$ of positive length.\n\n It follows from \\eqref{gradient-desc-cont}\n that $w(Z_\\infty(T)) < w(Z_\\infty(0)) =v$.\n However, $Z_\\infty(T)$ is the limit of the sequence $\\{z^{m_j}\\}$, where\n\\begin{equation}\n\\label{mj}\nm_j = \\min\\bigg\\{m \\ge k_j: \\sum_{k=k_j}^{m_j-1}\\tau_k \\ge T\\bigg\\}, \\quad j=1,2,\\dots.\n\\end{equation}\nFor all sufficiently large $j$ we  have $k_j \\le m_j \\le \\ell_j$ and $w(z^{m_j}) > v$, which is a contradiction. Therefore,\n$\\underline{w} = \\overline{w}$ and the sequence $\\big\\{w(z^k)\\big\\}$ has a limit.\n\nSuppose a subsequence $\\{z^{k_j}\\}$ is convergent to a point $\\bar{z}\\notin \\Zb^*$. Again, we\nset $s_j = t_{k_j}$ in \\eqref{Z-shifted}, consider the interval $[0,T]$ with $T>0$, and conclude that\na  sub-subsequence of $\\{Z_j(\\Cdot)\\}$ exists, which is uniformly convergent to a solution $Z_\\infty(\\Cdot)$ of the\ndifferential inclusion \\eqref{diff-inc-beta}. Then the points $\\{z^{m_j}\\}$ are convergent to $Z_\\infty(T)$ on this sub-subsequence,\nwhere $m_j$ are defined as in \\eqref{mj}.\nAs $Z_\\infty(0)=\\bar{z}\\notin \\Zb^*$, we obtain $Z_\\infty(T)< Z_\\infty(0)$. This would mean that a sub-subsequence\nof $\\big\\{w(z^{m_j})\\big\\}$ has the limit $w(Z_\\infty(T))< w(\\bar{z})$, which contradicts the convergence of the sequence\n$\\big\\{w(z^{k})\\big\\}$.\n\\end{proof}","sha256_utf8":"a2e1b1d2b41c974824392330d46559882c366c3316cc7c3b34243b64fd4e912e"}]},{"id":"S088","semantic_type":"proof_step","kind":"proof_step","title":"固定样本路径并引入证明框架","source_units":[{"start":72919,"end":73275,"start_line":1358,"end_line":1362,"raw_tex":"\\begin{proof}\nThe proof follows the steps of the proofs of \\cite[Thm. 3.20]{duchi2018stochastic} or \\cite[Thm. 3.5]{majewski2018analysis} {  or \\cite{nurminskii1972convergence}}.\nWe provide a short outline here, for the convenience of the Reader. For brevity, we omit the\nargument $\\omega$, but we stress that we consider a particular path of the method.\n\n","sha256_utf8":"dcb980b61d53bbed82b4f87c1659541435ab137f00baa38249dd6ad182211954"}]},{"id":"S089","semantic_type":"proof_step","kind":"proof_step","title":"假设势值振荡并构造越界指标","source_units":[{"start":73275,"end":73965,"start_line":1363,"end_line":1378,"raw_tex":"Define\n\\[\n\\underline{w} = \\liminf_{k \\to \\infty^{\\phantom{|}}} w(z^k), \\qquad \\overline{w} = \\limsup_{k \\to \\infty}w(z^k).\n\\]\nSuppose $\\underline{w} < \\overline{w}$. By Assumption (A7), a value ${v}\\in\n(\\underline{w},\\overline{w})$ exists,\nsuch that ${v} \\notin G(\\Bb^*)$. Take any $u \\in ({v},\\overline{w})$.\nDefine $L_1=\\min\\big\\{k : w(z^k) \\le {v} \\big\\}$ and for\n$j=1,2,\\dots$\n\\begin{align*}\n\\ell_j &= \\min\\big\\{k\\ge L_j : w(z^k) > u \\big\\},\\\\\nL_{j+1} &= \\min\\big\\{k\\ge \\ell_j : w(z^k) \\le {v} \\big\\}.\\\\\n\\intertext{Then we define}\nk_j &= \\max\\big\\{ k\\in [L_j,\\ell_j]: w(z^k) \\le {v} \\big\\}, \\quad j=1,2,\\dots.\n\\end{align*}\nBoth $k_j\\to \\infty$ and $\\ell_j\\to \\infty$, as $j\\to \\infty$.\n","sha256_utf8":"b0a4dc536534db6b20d89167eb7c1b653a4a31a1b8a19fe0ddba0aa8d67e992d"}]},{"id":"S090","semantic_type":"proof_step","kind":"proof_step","title":"提取越界轨迹的微分包含极限","source_units":[{"start":73965,"end":74636,"start_line":1379,"end_line":1389,"raw_tex":"\nConsider the sums\n$T_j = \\sum_{k=k_j}^{\\ell_j-1} \\tau_k$. Due to the equicontinuity of the functions\n $w(Z_j(\\Cdot))$, a constant $T >0$ exists, such that $T_j\\ge T$ for all sufficiently\nlarge $j$. Set $s_j=t_{k_j}$ in \\eqref{Z-shifted} and consider the interval $[0,T]$.\nBy Theorem \\ref{t:diff-inc-beta}, a subsequence of the sequence $\\{Z_j(\\Cdot)\\}$ is convergent to a solution $Z_\\infty(\\Cdot)$ of the differential\ninclusion \\eqref{diff-inc-beta}. For simplicity, we still denote it by $\\{Z_j(\\Cdot)\\}$. As $W(\\Cdot)$ is absolutely continuous,\n\\begin{equation}\n\\label{gradient-desc-cont}\nw(Z_\\infty(T)) = w(Z_\\infty(0)) + \\int_0^T  \\dt{W}(t)  \\;\\D t.\n\\end{equation}\n","sha256_utf8":"a824db9a39ffd80eb25ab463b82af8a7ed0725a7c1b6c61fec0c67fc52aa64d5"}]},{"id":"S091","semantic_type":"proof_step","kind":"proof_step","title":"非临界初值产生正时段严格下降","source_units":[{"start":74636,"end":75059,"start_line":1390,"end_line":1395,"raw_tex":"Since ${v} \\notin w(\\Zb^*)$, then  $Z_\\infty(0) = \\lim_{j\\to \\infty} z^{k_j} \\notin \\Zb^*$.\nTheorem \\ref{t:negative-der} then implies that $\\dt{W}(0)<0$.\nThe generalized subdifferential $\\hat{\\partial} G(\\Cdot)$ and the normal cone $\\Nc_\\Bb(\\Cdot)$ are upper semicontinuous and closed-valued. The function $Q(\\Cdot,\\Cdot)$ is continuous.\nTherefore, $\\dt{W}(t)<0$\n in some interval $t\\in [0,\\bar{t}\\,]$ of positive length.\n\n","sha256_utf8":"878d51e0a9bee8cfe6397ae803744f03db1a6c3ef25d737327451ad1cdc16cc9"}]},{"id":"S092","semantic_type":"proof_step","kind":"proof_step","title":"离散越界与连续下降矛盾","source_units":[{"start":75059,"end":75559,"start_line":1396,"end_line":1405,"raw_tex":" It follows from \\eqref{gradient-desc-cont}\n that $w(Z_\\infty(T)) < w(Z_\\infty(0)) =v$.\n However, $Z_\\infty(T)$ is the limit of the sequence $\\{z^{m_j}\\}$, where\n\\begin{equation}\n\\label{mj}\nm_j = \\min\\bigg\\{m \\ge k_j: \\sum_{k=k_j}^{m_j-1}\\tau_k \\ge T\\bigg\\}, \\quad j=1,2,\\dots.\n\\end{equation}\nFor all sufficiently large $j$ we  have $k_j \\le m_j \\le \\ell_j$ and $w(z^{m_j}) > v$, which is a contradiction. Therefore,\n$\\underline{w} = \\overline{w}$ and the sequence $\\big\\{w(z^k)\\big\\}$ has a limit.\n\n","sha256_utf8":"082e98dc3b24c288489cdcdf6fa952c1b19ef26f123d35845a19764e61c0a6cb"}]},{"id":"S093","semantic_type":"proof_step","kind":"proof_step","title":"排除目标集合外的聚点","source_units":[{"start":75559,"end":76318,"start_line":1406,"end_line":1414,"raw_tex":"Suppose a subsequence $\\{z^{k_j}\\}$ is convergent to a point $\\bar{z}\\notin \\Zb^*$. Again, we\nset $s_j = t_{k_j}$ in \\eqref{Z-shifted}, consider the interval $[0,T]$ with $T>0$, and conclude that\na  sub-subsequence of $\\{Z_j(\\Cdot)\\}$ exists, which is uniformly convergent to a solution $Z_\\infty(\\Cdot)$ of the\ndifferential inclusion \\eqref{diff-inc-beta}. Then the points $\\{z^{m_j}\\}$ are convergent to $Z_\\infty(T)$ on this sub-subsequence,\nwhere $m_j$ are defined as in \\eqref{mj}.\nAs $Z_\\infty(0)=\\bar{z}\\notin \\Zb^*$, we obtain $Z_\\infty(T)< Z_\\infty(0)$. This would mean that a sub-subsequence\nof $\\big\\{w(z^{m_j})\\big\\}$ has the limit $w(Z_\\infty(T))< w(\\bar{z})$, which contradicts the convergence of the sequence\n$\\big\\{w(z^{k})\\big\\}$.\n\\end{proof}","sha256_utf8":"96c87614c1dfcc4c6b3e405233dadf3f6a02dbf55a9d43c8b584af0b64e55a55"}]},{"id":"S094","semantic_type":"external_fact","kind":"external_fact","title":"广义可微复合演算","source_units":[],"declared_external":true,"declaration_text":["广义可微函数的复合仍广义可微，其广义次微分可取各层广义导数乘积的凸包；对应附录定理 B0150。"]},{"id":"S095","semantic_type":"external_fact","kind":"external_fact","title":"期望与广义次微分交换","source_units":[],"declared_external":true,"declaration_text":["可测广义次微分在每个紧集上具有可积包络时，期望函数广义可微，其广义次微分可取随机广义次微分的集合值积分；对应附录定理 B0152。"]},{"id":"S096","semantic_type":"external_fact","kind":"external_fact","title":"绝对连续路径链式法则","source_units":[],"declared_external":true,"declaration_text":["广义可微函数沿绝对连续路径满足对所有广义次梯度选择成立的积分链式法则；对应附录定理 B0156。"]},{"id":"S097","semantic_type":"external_fact","kind":"external_fact","title":"概率分解与条件期望法则","source_units":[],"declared_external":true,"declaration_text":["欧氏空间上的联合概率测度可分解为边缘测度与正则条件概率核；条件期望满足塔式法则及可测因子提出法则。"]},{"id":"S098","semantic_type":"external_fact","kind":"external_fact","title":"集合值积分保持正则性","source_units":[],"declared_external":true,"declaration_text":["具有可积包络的可测、非空紧凸值、上半连续多函数，其集合值期望仍具有非空紧凸值及上半连续性；作者引用 mikhalevich1987nonconvex，Lemma 21.7。"]},{"id":"S099","semantic_type":"external_fact","kind":"external_fact","title":"投影随机逼近极限定理","source_units":[],"declared_external":true,"declaration_text":["在紧凸约束、标准步长条件、上半连续紧凸值平均漂移和一致有界条件二阶矩鞅差噪声下，时间平移插值轨迹具有局部一致收敛子列，其极限是相应法锥微分包含的绝对连续解，且极限轨迹上的点是离散迭代的聚点；作者引用 majewski2018analysis，Theorem 4.1。"]},{"id":"S100","semantic_type":"external_fact","kind":"external_fact","title":"凸投影与极锥分解","source_units":[],"declared_external":true,"declaration_text":["闭凸集的切锥与法锥互为极锥并满足正交投影分解；受约束微分包含可写为切锥投影动力系统，正交投影非扩张。"]},{"id":"S101","semantic_type":"external_fact","kind":"external_fact","title":"标准矩估计不等式","source_units":[],"declared_external":true,"declaration_text":["Cauchy–Schwarz、Jensen、Young 不等式成立，条件方差不超过相应条件二阶矩。"]},{"id":"S102","semantic_type":"external_fact","kind":"external_fact","title":"随机逼近的势值越界框架","source_units":[],"declared_external":true,"declaration_text":["随机逼近中的标准 Lyapunov 越界论证结合微分包含轨迹极限、严格下降及临界值集无非退化区间，排除势值持续越界和非平衡聚点；作者引用 duchi2018stochastic，Theorem 3.20，majewski2018analysis，Theorem 3.5，及 nurminskii1972convergence，并在主证明中展开该论证。"]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S030","kind":"supports"},{"source_id":"S003","target_id":"S027","kind":"supports"},{"source_id":"S003","target_id":"S030","kind":"supports"},{"source_id":"S003","target_id":"S033","kind":"supports"},{"source_id":"S004","target_id":"S033","kind":"supports"},{"source_id":"S004","target_id":"S066","kind":"supports"},{"source_id":"S004","target_id":"S069","kind":"supports"},{"source_id":"S004","target_id":"S073","kind":"supports"},{"source_id":"S005","target_id":"S061","kind":"supports"},{"source_id":"S006","target_id":"S052","kind":"supports"},{"source_id":"S007","target_id":"S030","kind":"supports"},{"source_id":"S007","target_id":"S039","kind":"supports"},{"source_id":"S007","target_id":"S046","kind":"supports"},{"source_id":"S007","target_id":"S058","kind":"supports"},{"source_id":"S007","target_id":"S059","kind":"supports"},{"source_id":"S007","target_id":"S065","kind":"supports"},{"source_id":"S007","target_id":"S073","kind":"supports"},{"source_id":"S008","target_id":"S028","kind":"supports"},{"source_id":"S008","target_id":"S031","kind":"supports"},{"source_id":"S008","target_id":"S033","kind":"supports"},{"source_id":"S008","target_id":"S039","kind":"supports"},{"source_id":"S008","target_id":"S046","kind":"supports"},{"source_id":"S008","target_id":"S048","kind":"supports"},{"source_id":"S008","target_id":"S058","kind":"supports"},{"source_id":"S008","target_id":"S069","kind":"supports"},{"source_id":"S008","target_id":"S073","kind":"supports"},{"source_id":"S009","target_id":"S033","kind":"supports"},{"source_id":"S009","target_id":"S039","kind":"supports"},{"source_id":"S009","target_id":"S047","kind":"supports"},{"source_id":"S009","target_id":"S048","kind":"supports"},{"source_id":"S009","target_id":"S049","kind":"supports"},{"source_id":"S009","target_id":"S059","kind":"supports"},{"source_id":"S009","target_id":"S066","kind":"supports"},{"source_id":"S010","target_id":"S031","kind":"supports"},{"source_id":"S010","target_id":"S069","kind":"supports"},{"source_id":"S011","target_id":"S084","kind":"supports"},{"source_id":"S012","target_id":"S081","kind":"supports"},{"source_id":"S013","target_id":"S085","kind":"supports"},{"source_id":"S013","target_id":"S089","kind":"supports"},{"source_id":"S014","target_id":"S049","kind":"supports"},{"source_id":"S014","target_id":"S052","kind":"supports"},{"source_id":"S014","target_id":"S081","kind":"supports"},{"source_id":"S015","target_id":"S052","kind":"supports"},{"source_id":"S016","target_id":"S042","kind":"supports"},{"source_id":"S016","target_id":"S043","kind":"supports"},{"source_id":"S016","target_id":"S046","kind":"supports"},{"source_id":"S017","target_id":"S042","kind":"supports"},{"source_id":"S017","target_id":"S043","kind":"supports"},{"source_id":"S017","target_id":"S046","kind":"supports"},{"source_id":"S017","target_id":"S047","kind":"supports"},{"source_id":"S018","target_id":"S052","kind":"supports"},{"source_id":"S018","target_id":"S090","kind":"supports"},{"source_id":"S018","target_id":"S092","kind":"supports"},{"source_id":"S019","target_id":"S037","kind":"supports"},{"source_id":"S019","target_id":"S038","kind":"supports"},{"source_id":"S019","target_id":"S042","kind":"supports"},{"source_id":"S019","target_id":"S043","kind":"supports"},{"source_id":"S020","target_id":"S042","kind":"supports"},{"source_id":"S020","target_id":"S043","kind":"supports"},{"source_id":"S020","target_id":"S046","kind":"supports"},{"source_id":"S020","target_id":"S049","kind":"supports"},{"source_id":"S020","target_id":"S052","kind":"supports"},{"source_id":"S021","target_id":"S052","kind":"supports"},{"source_id":"S021","target_id":"S090","kind":"supports"},{"source_id":"S021","target_id":"S092","kind":"supports"},{"source_id":"S021","target_id":"S093","kind":"supports"},{"source_id":"S022","target_id":"S057","kind":"supports"},{"source_id":"S022","target_id":"S061","kind":"supports"},{"source_id":"S022","target_id":"S077","kind":"supports"},{"source_id":"S022","target_id":"S079","kind":"supports"},{"source_id":"S022","target_id":"S084","kind":"supports"},{"source_id":"S022","target_id":"S089","kind":"supports"},{"source_id":"S023","target_id":"S089","kind":"supports"},{"source_id":"S024","target_id":"S028","kind":"supports"},{"source_id":"S024","target_id":"S039","kind":"supports"},{"source_id":"S024","target_id":"S091","kind":"supports"},{"source_id":"S025","target_id":"S010","kind":"supports"},{"source_id":"S025","target_id":"S037","kind":"supports"},{"source_id":"S025","target_id":"S042","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S029","target_id":"S057","kind":"supports"},{"source_id":"S029","target_id":"S076","kind":"supports"},{"source_id":"S029","target_id":"S085","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S032","target_id":"S057","kind":"supports"},{"source_id":"S032","target_id":"S065","kind":"supports"},{"source_id":"S032","target_id":"S069","kind":"supports"},{"source_id":"S032","target_id":"S081","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S032","kind":"supports"},{"source_id":"S035","target_id":"S052","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S039","target_id":"S036","kind":"supports"},{"source_id":"S040","target_id":"S052","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S043","target_id":"S041","kind":"supports"},{"source_id":"S044","target_id":"S052","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S045","kind":"supports"},{"source_id":"S049","target_id":"S045","kind":"supports"},{"source_id":"S050","target_id":"S054","kind":"supports"},{"source_id":"S050","target_id":"S061","kind":"supports"},{"source_id":"S050","target_id":"S090","kind":"supports"},{"source_id":"S050","target_id":"S093","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S076","kind":"supports"},{"source_id":"S053","target_id":"S081","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S055","target_id":"S061","kind":"supports"},{"source_id":"S055","target_id":"S062","kind":"supports"},{"source_id":"S056","target_id":"S055","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S059","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S056","kind":"supports"},{"source_id":"S058","target_id":"S072","kind":"supports"},{"source_id":"S059","target_id":"S056","kind":"supports"},{"source_id":"S059","target_id":"S065","kind":"supports"},{"source_id":"S060","target_id":"S084","kind":"supports"},{"source_id":"S060","target_id":"S090","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S060","kind":"supports"},{"source_id":"S063","target_id":"S084","kind":"supports"},{"source_id":"S064","target_id":"S063","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S064","kind":"supports"},{"source_id":"S067","target_id":"S072","kind":"supports"},{"source_id":"S067","target_id":"S073","kind":"supports"},{"source_id":"S067","target_id":"S078","kind":"supports"},{"source_id":"S068","target_id":"S067","kind":"supports"},{"source_id":"S069","target_id":"S068","kind":"supports"},{"source_id":"S070","target_id":"S078","kind":"supports"},{"source_id":"S071","target_id":"S070","kind":"supports"},{"source_id":"S072","target_id":"S073","kind":"supports"},{"source_id":"S072","target_id":"S071","kind":"supports"},{"source_id":"S073","target_id":"S071","kind":"supports"},{"source_id":"S074","target_id":"S084","kind":"supports"},{"source_id":"S074","target_id":"S085","kind":"supports"},{"source_id":"S075","target_id":"S074","kind":"supports"},{"source_id":"S076","target_id":"S077","kind":"supports"},{"source_id":"S076","target_id":"S075","kind":"supports"},{"source_id":"S077","target_id":"S078","kind":"supports"},{"source_id":"S077","target_id":"S079","kind":"supports"},{"source_id":"S077","target_id":"S075","kind":"supports"},{"source_id":"S078","target_id":"S075","kind":"supports"},{"source_id":"S079","target_id":"S075","kind":"supports"},{"source_id":"S080","target_id":"S084","kind":"supports"},{"source_id":"S081","target_id":"S080","kind":"supports"},{"source_id":"S082","target_id":"S091","kind":"supports"},{"source_id":"S082","target_id":"S092","kind":"supports"},{"source_id":"S082","target_id":"S093","kind":"supports"},{"source_id":"S083","target_id":"S082","kind":"supports"},{"source_id":"S084","target_id":"S085","kind":"supports"},{"source_id":"S084","target_id":"S083","kind":"supports"},{"source_id":"S085","target_id":"S083","kind":"supports"},{"source_id":"S086","target_id":"S084","kind":"supports"},{"source_id":"S086","target_id":"S085","kind":"supports"},{"source_id":"S087","target_id":"S001","kind":"supports"},{"source_id":"S088","target_id":"S087","kind":"supports"},{"source_id":"S089","target_id":"S090","kind":"supports"},{"source_id":"S089","target_id":"S091","kind":"supports"},{"source_id":"S089","target_id":"S092","kind":"supports"},{"source_id":"S089","target_id":"S087","kind":"supports"},{"source_id":"S090","target_id":"S091","kind":"supports"},{"source_id":"S090","target_id":"S092","kind":"supports"},{"source_id":"S090","target_id":"S087","kind":"supports"},{"source_id":"S091","target_id":"S092","kind":"supports"},{"source_id":"S091","target_id":"S087","kind":"supports"},{"source_id":"S092","target_id":"S093","kind":"supports"},{"source_id":"S092","target_id":"S087","kind":"supports"},{"source_id":"S093","target_id":"S087","kind":"supports"},{"source_id":"S094","target_id":"S030","kind":"supports"},{"source_id":"S094","target_id":"S033","kind":"supports"},{"source_id":"S094","target_id":"S058","kind":"supports"},{"source_id":"S094","target_id":"S059","kind":"supports"},{"source_id":"S094","target_id":"S061","kind":"supports"},{"source_id":"S095","target_id":"S028","kind":"supports"},{"source_id":"S095","target_id":"S031","kind":"supports"},{"source_id":"S095","target_id":"S033","kind":"supports"},{"source_id":"S095","target_id":"S059","kind":"supports"},{"source_id":"S096","target_id":"S062","kind":"supports"},{"source_id":"S097","target_id":"S027","kind":"supports"},{"source_id":"S097","target_id":"S028","kind":"supports"},{"source_id":"S097","target_id":"S031","kind":"supports"},{"source_id":"S097","target_id":"S037","kind":"supports"},{"source_id":"S097","target_id":"S038","kind":"supports"},{"source_id":"S097","target_id":"S042","kind":"supports"},{"source_id":"S097","target_id":"S043","kind":"supports"},{"source_id":"S098","target_id":"S039","kind":"supports"},{"source_id":"S099","target_id":"S052","kind":"supports"},{"source_id":"S100","target_id":"S054","kind":"supports"},{"source_id":"S100","target_id":"S076","kind":"supports"},{"source_id":"S100","target_id":"S077","kind":"supports"},{"source_id":"S100","target_id":"S081","kind":"supports"},{"source_id":"S100","target_id":"S085","kind":"supports"},{"source_id":"S101","target_id":"S039","kind":"supports"},{"source_id":"S101","target_id":"S046","kind":"supports"},{"source_id":"S101","target_id":"S047","kind":"supports"},{"source_id":"S101","target_id":"S048","kind":"supports"},{"source_id":"S101","target_id":"S049","kind":"supports"},{"source_id":"S101","target_id":"S059","kind":"supports"},{"source_id":"S101","target_id":"S066","kind":"supports"},{"source_id":"S101","target_id":"S069","kind":"supports"},{"source_id":"S101","target_id":"S073","kind":"supports"},{"source_id":"S101","target_id":"S077","kind":"supports"},{"source_id":"S102","target_id":"S088","kind":"supports"}]}}
{"record_id":"2406.00648v3","paper":{"title":"Level proximal subdifferential, variational convexity, and pointwise quadratic approximation","authors":["Honglin Luo","Xianfu Wang","Ziyuan Wang","Xinmin Yang"],"arxiv_id":"2406.00648","version":"v3","journal":"Mathematical Programming","doi":"10.1007/s10107-026-02355-z","abstract_url":"https://arxiv.org/abs/2406.00648v3","pdf_url":"https://arxiv.org/pdf/2406.00648v3"},"context":{"raw_tex":"The Euclidean space $\\Rn$ is equipped with the usual scalar product\n$\\ip{x}{y}:=\\sum_{i=1}^{n}\\xi_{i}\\eta_{i}$ for $x=(\\xi_{1},\\ldots, \\xi_{n})$ and $y=(\\eta_{1},\\ldots, \\eta_{n})$, and induced norm\n$\\norm{x}:=\\sqrt{\\ip{x}{x}}$.\nThe set of extended real numbers is $\\overline{\\R} :=(-\\infty, +\\infty]$.\nBy a proper lsc function $f:\\Rn\\rightarrow\\bR$ we mean that $f$ is\nlower semicontinuous on $\\Rn$ and\n$\\dom f:=\\{x\\in\\Rn| \\ f(x)<+\\infty\\}\\neq \\varnothing$. $\\conv f$ and $\\clconv f$ are \\emph{convex hull} and\n\\emph{lsc convex hull} of function $f$, respectively; see \\cite[pages 56-57]{rockafellar_variational_1998}.\nWe let $j :=\\|\\cdot\\|^2/2.$\nFor parameter value $\\lambda>0$, the \\emph{Moreau envelope} function $e_{\\lambda}f$ and \\emph{proximal mapping}\n$P_{\\lambda}f$ of $f$ are defined by\n\\begin{equation}\ne_{\\lambda}f(x) :=\\inf_{y\\in\\Rn}\\left\\{f(y)+\\frac{1}{2\\lambda}\\|y-x\\|^2\\right\\},\\text{ and }\nP_{\\lambda}f(x) :=\\argmin_{y\\in\\Rn}\\left\\{f(y)+\\frac{1}{2\\lambda}\\|y-x\\|^2\\right\\},\n\\end{equation}\nrespectively.\n%The $\\lambda$-proximal hull of $f$ is $h_{\\lambda}f=-e_{\\lambda}[-e_{\\lambda}f]$.\nA function $f:\\Rn\\rightarrow\\bR$ is \\emph{prox-bounded} if there exists $\\lambda>0$ such that\n$e_{\\lambda}f(x)>-\\infty$ for some $x\\in\\Rn$. The supremum of the set of\nall such $\\lambda$ is the threshold $\\lambda_{f}$ of prox-boundedness for\n$f$. The positivity of $\\lambda_{f}$ plays an important role in the paper.\nIf $\\lambda_{f}$ is positive, then $(\\forall \\lambda\\in (0,\\lambda_{f}))$ the proximal mapping\n$P_{\\lambda}f$ is nonempty, compact valued, and outer semicontinuous, and\nthe Moreau envelope\n$e_{\\lambda}f$ is locally Lipschitz; see, e.g., \\cite[Theorem 1.25, Example 10.32]{rockafellar_variational_1998}.\n\n\nGeneralized subdifferential constructions from variational analysis\nwill be widely used in the paper. Let $f:\\Rn\\to\\overline{\\R}$ be proper, $x\\in\\dom f$ and let $\\lambda>0$. In \\cite{rock2021} Rockafellar defined level proximal subdifferential.\nA vector $v\\in\\Rn$ is a \\emph{$\\lambda$-level proximal subgradient} of $f$ at $x$, denoted by $v\\in\\levp f(x)$, if\n\t\\begin{equation}\\label{lep ineq}\n\t\t(\\forall y\\in\\Rn)~f(y)\\geq f(x)+\\ip{v}{y-x}-\\frac{1}{2\\lambda}\\norm{y-x}^2.\n\t\\end{equation}\nIt is a special subclass of proximal subgradients. A vector $v\\in\\Rn$ is a \\emph{proximal subgradient} of $f$ at $x$, denoted by $v\\in\\partial_{p}f(x)$, if\n$(\\exists \\lambda>0)(\\exists \\delta>0)(\\forall \\|y-x\\|<\\delta) \\ f(y)\\geq f(x)+\\ip{v}{y-x}-\\frac{1}{2\\lambda}\\|y-x\\|^2.$\nNext is the classical Fenchel subdifferential.\nA vector $v\\in\\Rn$ is a \\emph{Fenchel subgradient} of $f$ at $x$, denoted by $v\\in\\partial_{F}f(x)$, if\n$(\\forall y\\in\\Rn)~f(y)\\geq f(x)+\\ip{v}{y-x}.\n$\nIt is a special class of regular subgradients. A vector $v\\in\\Rn$ is a \\emph{regular subgradient}\nof $f$ at $x$, denoted by\n$v\\in\\hat{\\partial}f(x)$, if $(\\forall y\\in\\Rn)\\ f(y)\\geq f(x)+\\ip{v}{y-x}+o(\\|y-x\\|).$\nIt is a \\emph{Mordukhovich limiting (or general) subgradient}, denoted by $v\\in\\partial f(x)$,\nif $\\exists v^{\\nu}\\in\\hat{\\partial}f(x^{\\nu})$ such that $v^{\\nu}\\rightarrow v$,\n$x^{\\nu}\\rightarrow x$, and $f(x^{\\nu})\\rightarrow f(x)$.\nWhen $x\\not\\in\\dom f$, all subdiffernetials are set to be empty.\n%$\\partial_{F}f(x)=\\varnothing$.\nSee the landmark books \\cite{mordukhovich2006, rockafellar_variational_1998} for\nfurther details and references on subdifferentials.\nWe say that $f$ is \\emph{prox-regular} at $\\bar x$\nfor $\\bar v\\in\\partial f(\\bar x)$ if $(\\exists \\varepsilon>0, \\rho\\geq 0)(\\forall y\\in\\Ball_{\\varepsilon}(\\bar x)) \\\nf(y)\\geq f(x)+\\ip{v}{y-x}-\\dfrac{\\rho}{2}\\norm{y-x}^2$ for\n$v\\in\\partial f(x), v\\in\\Ball_{\\varepsilon}(\\bar v), x\\in\\Ball_{\\varepsilon}(\\bar x),\nf(x)<f(\\bar x)+\\varepsilon$, where $\\Ball_{\\varepsilon}(a)$ is the open ball centered at $a$ with\nradius $\\varepsilon$. If this holds for all $\\bar v\\in\\partial f(\\bar x)$,\n$f$ is said to be prox-regular at\n$\\bar x$; see \\cite{poliquin1996,rockafellar_variational_1998,thibault}.\n\n\nNext we recall some notions for set-valued mappings.\nLet $A:\\Rn\\rightrightarrows\\Rn$ be a set-valued mapping and let $W$ be a nonempty subset of $\\Rn\\times\\Rn$.\nThen $A$ is \\emph{monotone relative to $W$} if\n\\(\n(\\forall (x_1,v_1), (x_2,v_2)\\in\\gph A\\cap W)~\n\\ip{x_1-x_2}{v_1-v_2}\\geq0;\n\\)\nand $A$ is \\emph{$\\lambda$-strongly monotone relative to $W$} if\n\\(\n(\\forall (x_1,v_1), (x_2,v_2)\\in\\gph A\\cap W)~\n\\ip{x_1-x_2}{v_1-v_2}\\geq\\lambda\\norm{x_1-x_2}^2,\n\\)\nwhere $\\lambda >0$. $A$ is \\emph{maximally monotone relative to $W$} if there does not exist\n $A':\\Rn\\rightrightarrows\\Rn$ such that $\\gph A'\\cap W\\supseteq\\gph A\\cap W$ but\n$\\gph A'\\cap W \\neq \\gph A\\cap W$ and $A'$ is monotone\nrelative to $W$. See \\cite{nghia2016,rock-vietnam,poliquin1996}\nfor more discussions. The \\emph{resolvent}\n of $A$ is\n$J_{A} :=(\\Id+A)^{-1}$ with $\\Id:\\Rn\\rightarrow\\Rn: x\\mapsto x$ being the identity mapping.\nLet $U$ be a nonempty subset of $\\Rn$ and let $T:U \\to\\Rn$. Then $T$  is \\emph{firmly nonexpansive}\n if %$(\\forall x,y\\in U)$ $\\norm{Tx-Ty}^2+\\norm{\\left(\\Id-T\\right)x-\\left(\\Id-T %\\right)y}^2\\leq\\norm{x-y}^2$;\n$(\\forall x,y\\in U)\\ \\norm{Tx-Ty}^2\\leq \\ip{x-y}{Tx-Ty}$;\n$T$ is \\emph{$\\lambda$-cocoercive} with $\\lambda>0$ if $\\lambda T$ is firmly nonexpansive;\n$T$ is \\emph{nonexpansive} if $(\\forall x,y\\in U)\\ \\norm{Tx-Ty}\\leq\\norm{x-y}$; and $T$ is \\emph{averaged} if\nthere exist nonexpansive $N:U\\to\\Rn$ and $0\\leq \\lambda<1$ such that $T:=(1-\\lambda)\\Id+\\lambda N$.\nWe use $\\Fix T$ for the set of fixed point of $T$.\nSee \\cite[Chapter 4]{BC} for abundant results and applications of nonexpansive operators.\n\n\nThe sets\n$\\dom A :=\\{x\\in\\Rn|\\ A(x)\\neq \\varnothing\\},$ $\\ran A :=\\bigcup_{x\\in\\Rn}Ax=A(\\Rn)$, and $\\gph A\n:=\\{(x,v)\\in\\Rn\\times \\Rn|\\ v\\in Ax\\}$ stand for\nthe domain, range, and graph of $A$, respectively.\n\n\n\\begin{defn}\\label{def:vc}\\emph{\\cite[Definition 2]{rock-vietnam}}\nA lsc function $f:\\Rn\\rightarrow\\overline{\\R}$ is called variationally convex at $\\bar x$ for $\\bar v\\in\\partial f(\\bar x)$,\nif for some convex neighborhood $U\\times V$ of $(\\bar x, \\bar v)$ there exist a lsc convex function $\\varphi\\leq f$ on $U$ and a number $\\varepsilon>0$ such that\n$[U_{\\varepsilon}\\times V]\\cap \\gph\\partial f=[U\\times V]\\cap \\gph \\partial \\varphi\\text { and } f(x)=\\varphi(x) \\text{ at the common element $(x,v)$,} $\nwhere $U_{\\varepsilon}:=\\{x\\in U|\\ f(x)< f(\\bar x)+\\varepsilon\\}$.\n\\end{defn}","units":[{"start":25120,"end":26842,"start_line":496,"end_line":520,"raw_tex":"The Euclidean space $\\Rn$ is equipped with the usual scalar product\n$\\ip{x}{y}:=\\sum_{i=1}^{n}\\xi_{i}\\eta_{i}$ for $x=(\\xi_{1},\\ldots, \\xi_{n})$ and $y=(\\eta_{1},\\ldots, \\eta_{n})$, and induced norm\n$\\norm{x}:=\\sqrt{\\ip{x}{x}}$.\nThe set of extended real numbers is $\\overline{\\R} :=(-\\infty, +\\infty]$.\nBy a proper lsc function $f:\\Rn\\rightarrow\\bR$ we mean that $f$ is\nlower semicontinuous on $\\Rn$ and\n$\\dom f:=\\{x\\in\\Rn| \\ f(x)<+\\infty\\}\\neq \\varnothing$. $\\conv f$ and $\\clconv f$ are \\emph{convex hull} and\n\\emph{lsc convex hull} of function $f$, respectively; see \\cite[pages 56-57]{rockafellar_variational_1998}.\nWe let $j :=\\|\\cdot\\|^2/2.$\nFor parameter value $\\lambda>0$, the \\emph{Moreau envelope} function $e_{\\lambda}f$ and \\emph{proximal mapping}\n$P_{\\lambda}f$ of $f$ are defined by\n\\begin{equation}\ne_{\\lambda}f(x) :=\\inf_{y\\in\\Rn}\\left\\{f(y)+\\frac{1}{2\\lambda}\\|y-x\\|^2\\right\\},\\text{ and }\nP_{\\lambda}f(x) :=\\argmin_{y\\in\\Rn}\\left\\{f(y)+\\frac{1}{2\\lambda}\\|y-x\\|^2\\right\\},\n\\end{equation}\nrespectively.\n%The $\\lambda$-proximal hull of $f$ is $h_{\\lambda}f=-e_{\\lambda}[-e_{\\lambda}f]$.\nA function $f:\\Rn\\rightarrow\\bR$ is \\emph{prox-bounded} if there exists $\\lambda>0$ such that\n$e_{\\lambda}f(x)>-\\infty$ for some $x\\in\\Rn$. The supremum of the set of\nall such $\\lambda$ is the threshold $\\lambda_{f}$ of prox-boundedness for\n$f$. The positivity of $\\lambda_{f}$ plays an important role in the paper.\nIf $\\lambda_{f}$ is positive, then $(\\forall \\lambda\\in (0,\\lambda_{f}))$ the proximal mapping\n$P_{\\lambda}f$ is nonempty, compact valued, and outer semicontinuous, and\nthe Moreau envelope\n$e_{\\lambda}f$ is locally Lipschitz; see, e.g., \\cite[Theorem 1.25, Example 10.32]{rockafellar_variational_1998}.\n","sha256_utf8":"877ef15801c59068fab6289a8cab42cbb93bbaa1958506643ede1e4226db4f1b"},{"start":27395,"end":29631,"start_line":529,"end_line":558,"raw_tex":"Generalized subdifferential constructions from variational analysis\nwill be widely used in the paper. Let $f:\\Rn\\to\\overline{\\R}$ be proper, $x\\in\\dom f$ and let $\\lambda>0$. In \\cite{rock2021} Rockafellar defined level proximal subdifferential.\nA vector $v\\in\\Rn$ is a \\emph{$\\lambda$-level proximal subgradient} of $f$ at $x$, denoted by $v\\in\\levp f(x)$, if\n\t\\begin{equation}\\label{lep ineq}\n\t\t(\\forall y\\in\\Rn)~f(y)\\geq f(x)+\\ip{v}{y-x}-\\frac{1}{2\\lambda}\\norm{y-x}^2.\n\t\\end{equation}\nIt is a special subclass of proximal subgradients. A vector $v\\in\\Rn$ is a \\emph{proximal subgradient} of $f$ at $x$, denoted by $v\\in\\partial_{p}f(x)$, if\n$(\\exists \\lambda>0)(\\exists \\delta>0)(\\forall \\|y-x\\|<\\delta) \\ f(y)\\geq f(x)+\\ip{v}{y-x}-\\frac{1}{2\\lambda}\\|y-x\\|^2.$\nNext is the classical Fenchel subdifferential.\nA vector $v\\in\\Rn$ is a \\emph{Fenchel subgradient} of $f$ at $x$, denoted by $v\\in\\partial_{F}f(x)$, if\n$(\\forall y\\in\\Rn)~f(y)\\geq f(x)+\\ip{v}{y-x}.\n$\nIt is a special class of regular subgradients. A vector $v\\in\\Rn$ is a \\emph{regular subgradient}\nof $f$ at $x$, denoted by\n$v\\in\\hat{\\partial}f(x)$, if $(\\forall y\\in\\Rn)\\ f(y)\\geq f(x)+\\ip{v}{y-x}+o(\\|y-x\\|).$\nIt is a \\emph{Mordukhovich limiting (or general) subgradient}, denoted by $v\\in\\partial f(x)$,\nif $\\exists v^{\\nu}\\in\\hat{\\partial}f(x^{\\nu})$ such that $v^{\\nu}\\rightarrow v$,\n$x^{\\nu}\\rightarrow x$, and $f(x^{\\nu})\\rightarrow f(x)$.\nWhen $x\\not\\in\\dom f$, all subdiffernetials are set to be empty.\n%$\\partial_{F}f(x)=\\varnothing$.\nSee the landmark books \\cite{mordukhovich2006, rockafellar_variational_1998} for\nfurther details and references on subdifferentials.\nWe say that $f$ is \\emph{prox-regular} at $\\bar x$\nfor $\\bar v\\in\\partial f(\\bar x)$ if $(\\exists \\varepsilon>0, \\rho\\geq 0)(\\forall y\\in\\Ball_{\\varepsilon}(\\bar x)) \\\nf(y)\\geq f(x)+\\ip{v}{y-x}-\\dfrac{\\rho}{2}\\norm{y-x}^2$ for\n$v\\in\\partial f(x), v\\in\\Ball_{\\varepsilon}(\\bar v), x\\in\\Ball_{\\varepsilon}(\\bar x),\nf(x)<f(\\bar x)+\\varepsilon$, where $\\Ball_{\\varepsilon}(a)$ is the open ball centered at $a$ with\nradius $\\varepsilon$. If this holds for all $\\bar v\\in\\partial f(\\bar x)$,\n$f$ is said to be prox-regular at\n$\\bar x$; see \\cite{poliquin1996,rockafellar_variational_1998,thibault}.\n","sha256_utf8":"492923efd91a308f0706baa7716d73034411c71bd950fa15534e7e62dddc763b"},{"start":29632,"end":31239,"start_line":560,"end_line":586,"raw_tex":"Next we recall some notions for set-valued mappings.\nLet $A:\\Rn\\rightrightarrows\\Rn$ be a set-valued mapping and let $W$ be a nonempty subset of $\\Rn\\times\\Rn$.\nThen $A$ is \\emph{monotone relative to $W$} if\n\\(\n(\\forall (x_1,v_1), (x_2,v_2)\\in\\gph A\\cap W)~\n\\ip{x_1-x_2}{v_1-v_2}\\geq0;\n\\)\nand $A$ is \\emph{$\\lambda$-strongly monotone relative to $W$} if\n\\(\n(\\forall (x_1,v_1), (x_2,v_2)\\in\\gph A\\cap W)~\n\\ip{x_1-x_2}{v_1-v_2}\\geq\\lambda\\norm{x_1-x_2}^2,\n\\)\nwhere $\\lambda >0$. $A$ is \\emph{maximally monotone relative to $W$} if there does not exist\n $A':\\Rn\\rightrightarrows\\Rn$ such that $\\gph A'\\cap W\\supseteq\\gph A\\cap W$ but\n$\\gph A'\\cap W \\neq \\gph A\\cap W$ and $A'$ is monotone\nrelative to $W$. See \\cite{nghia2016,rock-vietnam,poliquin1996}\nfor more discussions. The \\emph{resolvent}\n of $A$ is\n$J_{A} :=(\\Id+A)^{-1}$ with $\\Id:\\Rn\\rightarrow\\Rn: x\\mapsto x$ being the identity mapping.\nLet $U$ be a nonempty subset of $\\Rn$ and let $T:U \\to\\Rn$. Then $T$  is \\emph{firmly nonexpansive}\n if %$(\\forall x,y\\in U)$ $\\norm{Tx-Ty}^2+\\norm{\\left(\\Id-T\\right)x-\\left(\\Id-T %\\right)y}^2\\leq\\norm{x-y}^2$;\n$(\\forall x,y\\in U)\\ \\norm{Tx-Ty}^2\\leq \\ip{x-y}{Tx-Ty}$;\n$T$ is \\emph{$\\lambda$-cocoercive} with $\\lambda>0$ if $\\lambda T$ is firmly nonexpansive;\n$T$ is \\emph{nonexpansive} if $(\\forall x,y\\in U)\\ \\norm{Tx-Ty}\\leq\\norm{x-y}$; and $T$ is \\emph{averaged} if\nthere exist nonexpansive $N:U\\to\\Rn$ and $0\\leq \\lambda<1$ such that $T:=(1-\\lambda)\\Id+\\lambda N$.\nWe use $\\Fix T$ for the set of fixed point of $T$.\nSee \\cite[Chapter 4]{BC} for abundant results and applications of nonexpansive operators.\n","sha256_utf8":"970b2e6892dd6a953214046ebc74946686ebdf8fefd2b1d3487b31c1b12cb68e"},{"start":31469,"end":31676,"start_line":590,"end_line":593,"raw_tex":"The sets\n$\\dom A :=\\{x\\in\\Rn|\\ A(x)\\neq \\varnothing\\},$ $\\ran A :=\\bigcup_{x\\in\\Rn}Ax=A(\\Rn)$, and $\\gph A\n:=\\{(x,v)\\in\\Rn\\times \\Rn|\\ v\\in Ax\\}$ stand for\nthe domain, range, and graph of $A$, respectively.\n","sha256_utf8":"a2100462c5572391e7014260608a5a605e9669e4496d1c707dc853fbe868d1e6"},{"start":78505,"end":79090,"start_line":1586,"end_line":1591,"raw_tex":"\\begin{defn}\\label{def:vc}\\emph{\\cite[Definition 2]{rock-vietnam}}\nA lsc function $f:\\Rn\\rightarrow\\overline{\\R}$ is called variationally convex at $\\bar x$ for $\\bar v\\in\\partial f(\\bar x)$,\nif for some convex neighborhood $U\\times V$ of $(\\bar x, \\bar v)$ there exist a lsc convex function $\\varphi\\leq f$ on $U$ and a number $\\varepsilon>0$ such that\n$[U_{\\varepsilon}\\times V]\\cap \\gph\\partial f=[U\\times V]\\cap \\gph \\partial \\varphi\\text { and } f(x)=\\varphi(x) \\text{ at the common element $(x,v)$,} $\nwhere $U_{\\varepsilon}:=\\{x\\in U|\\ f(x)< f(\\bar x)+\\varepsilon\\}$.\n\\end{defn}","sha256_utf8":"fd235c00e7dd7301be2496b6651bdceab36d1e8cb90335f19bb93a6de8908c59"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}[characterizations of variational convexity]\\label{thm:coco env grad}\nSuppose that $f:\\mathbb{R}^n\\to\\overline{\\mathbb{R}}$ is proper, lsc and prox-bounded\nwith threshold $\\lambda_{f}>0$,\nand that $f$ is prox-regular at $\\bar x$ for $\\bar v\\in\\partial f(\\bar x)$.\n%{\\color{gray}\n%Then for all $\\lambda>0$ sufficiently small there exists an open convex neighborhood $U_{\\lambda}$\n%of $\\bar{x}+\\lambda \\bar{v}$ on which the following are equivalent:\n%}%\nThen there exists \\(0<\\bar\\lambda\\leq \\lambda_{f}\\) such that $(\\forall 0<\\lambda<\\bar{\\lambda})\\ P_{\\lambda}f(\\barx+\\lambda\\barv)=\\barx$, and $\\exists U_{\\lambda}$\nan open convex neighborhood of \\(\\bar x+\\lambda\\bar v\\) for which the following equivalent properties hold:\n\t\\begin{enumerate}\n\t\t\\item\\label{thm:coco env grad:: v cvx}\n\t\t$f$ is variationally convex at $\\bar x$ for $\\bar v$;\n\t\t\\item\\label{thm:coco env grad:: mono}\n\t\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\)\n\t\t$\\partial_p^\\lambda f$ is monotone relative to $W_\\lambda$;\n% and $\\bar v\\in\\partial_{p}^{\\lambda}f(\\bar x)$;\n\t\t\\item\\label{thmn:coco env grad:: max mono}\n\t\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\)\n\t\t$\\partial_p^\\lambda f$ is maximally monotone relative to $W_\\lambda$;\n% and $\\bar v\\in\\partial_{p}^{\\lambda}f(\\bar x)$;\n\t\t\\item\\label{thm:coco env grad:: firmly nonexap prox}\n\t\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\)\n\t\t$P_\\lambda f$ is firmly nonexpansive on $U_\\lambda$;\n\t\t\\item\\label{thm:coco env grad:: averaged}\n\t\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\) \\((\\exists0<\\alpha<1)\\)\n\t\t$P_\\lambda f$ is $\\alpha$-averaged on $U_\\lambda$;\n% and $P_{\\lambda}f(\\barx+\\lambda\\barv)=\\barx$;\n\t\t\\item\\label{thm:coco env grad:: ne}\n\t\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\)\n\t\t$P_\\lambda f$ is nonexpansive on $U_\\lambda$;\n% and $P_{\\lambda}f(\\barx+\\lambda\\barv)=\\barx$;\n\t%\t\\item\\label{thm:coco env grad:: comono}\n\t%\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\) \\((\\exists0<\\alpha<1)\\)\n\t%\t$\\partial_p^\\lambda f$ is $\\rho$-monotone relative to \\(W_\\lambda\\) with \\(\\rho=((2\\alpha)^{-1}-1)\\lambda^{-1}\\);\n\t\t\\item\\label{envel:convex}\n\t\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\)\n\t\t$e_\\lambda f$ is convex on $U_\\lambda$;\n\t\t\\item\\label{envel:convex spec}\n\t\t\\((\\exists0<\\lambda<\\bar \\lambda)\\)\n\t\t\\(e_\\lambda f\\) is convex on $U_\\lambda$;\n\t\t%($\\partial_F\\left(\\lambda^{-1}j+f\\right)$ is $1/\\lambda$-strongly monotone\n\t\t%relative to $\\Rn\\times\\left[ \\lambda^{-1}U_\\lambda\\right]$.)\n\t\t\\item\\label{thm:coco env grad:: coco}\n\t\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\)\n\t\t$\\nabla e_\\lambda f$ is $\\lambda$-cocoercive on $U_\\lambda$;\n\t\\end{enumerate}\n\t\\vspace*{-2mm}\n\t\twhere $W_\\lambda :=\\{(x,v)\\in\\Rn\\times\\Rn|\\ x+\\lambda v\\in U_\\lambda\\}$.\n\\end{theorem}","units":[{"start":91711,"end":94339,"start_line":1888,"end_line":1937,"raw_tex":"\\begin{theorem}[characterizations of variational convexity]\\label{thm:coco env grad}\nSuppose that $f:\\mathbb{R}^n\\to\\overline{\\mathbb{R}}$ is proper, lsc and prox-bounded\nwith threshold $\\lambda_{f}>0$,\nand that $f$ is prox-regular at $\\bar x$ for $\\bar v\\in\\partial f(\\bar x)$.\n%{\\color{gray}\n%Then for all $\\lambda>0$ sufficiently small there exists an open convex neighborhood $U_{\\lambda}$\n%of $\\bar{x}+\\lambda \\bar{v}$ on which the following are equivalent:\n%}%\nThen there exists \\(0<\\bar\\lambda\\leq \\lambda_{f}\\) such that $(\\forall 0<\\lambda<\\bar{\\lambda})\\ P_{\\lambda}f(\\barx+\\lambda\\barv)=\\barx$, and $\\exists U_{\\lambda}$\nan open convex neighborhood of \\(\\bar x+\\lambda\\bar v\\) for which the following equivalent properties hold:\n\t\\begin{enumerate}\n\t\t\\item\\label{thm:coco env grad:: v cvx}\n\t\t$f$ is variationally convex at $\\bar x$ for $\\bar v$;\n\t\t\\item\\label{thm:coco env grad:: mono}\n\t\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\)\n\t\t$\\partial_p^\\lambda f$ is monotone relative to $W_\\lambda$;\n% and $\\bar v\\in\\partial_{p}^{\\lambda}f(\\bar x)$;\n\t\t\\item\\label{thmn:coco env grad:: max mono}\n\t\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\)\n\t\t$\\partial_p^\\lambda f$ is maximally monotone relative to $W_\\lambda$;\n% and $\\bar v\\in\\partial_{p}^{\\lambda}f(\\bar x)$;\n\t\t\\item\\label{thm:coco env grad:: firmly nonexap prox}\n\t\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\)\n\t\t$P_\\lambda f$ is firmly nonexpansive on $U_\\lambda$;\n\t\t\\item\\label{thm:coco env grad:: averaged}\n\t\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\) \\((\\exists0<\\alpha<1)\\)\n\t\t$P_\\lambda f$ is $\\alpha$-averaged on $U_\\lambda$;\n% and $P_{\\lambda}f(\\barx+\\lambda\\barv)=\\barx$;\n\t\t\\item\\label{thm:coco env grad:: ne}\n\t\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\)\n\t\t$P_\\lambda f$ is nonexpansive on $U_\\lambda$;\n% and $P_{\\lambda}f(\\barx+\\lambda\\barv)=\\barx$;\n\t%\t\\item\\label{thm:coco env grad:: comono}\n\t%\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\) \\((\\exists0<\\alpha<1)\\)\n\t%\t$\\partial_p^\\lambda f$ is $\\rho$-monotone relative to \\(W_\\lambda\\) with \\(\\rho=((2\\alpha)^{-1}-1)\\lambda^{-1}\\);\n\t\t\\item\\label{envel:convex}\n\t\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\)\n\t\t$e_\\lambda f$ is convex on $U_\\lambda$;\n\t\t\\item\\label{envel:convex spec}\n\t\t\\((\\exists0<\\lambda<\\bar \\lambda)\\)\n\t\t\\(e_\\lambda f\\) is convex on $U_\\lambda$;\n\t\t%($\\partial_F\\left(\\lambda^{-1}j+f\\right)$ is $1/\\lambda$-strongly monotone\n\t\t%relative to $\\Rn\\times\\left[ \\lambda^{-1}U_\\lambda\\right]$.)\n\t\t\\item\\label{thm:coco env grad:: coco}\n\t\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\)\n\t\t$\\nabla e_\\lambda f$ is $\\lambda$-cocoercive on $U_\\lambda$;\n\t\\end{enumerate}\n\t\\vspace*{-2mm}\n\t\twhere $W_\\lambda :=\\{(x,v)\\in\\Rn\\times\\Rn|\\ x+\\lambda v\\in U_\\lambda\\}$.\n\\end{theorem}","sha256_utf8":"4884f147c60e7a41968973bf1fa9a5c29715a3eefb5b8764b8a6b3dbf07c4eb4"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{fact}\n%[resolvent representation of proximal mapping]\n\\label{fact: prox identity}\n\t\\emph{\\cite[Theorem 3.7]{wang23level}}\n\tLet $f:\\Rn\\to\\overline{\\R}$ be proper, lsc and prox-bounded with threshold\n$\\lambda_f>0$. Then\n\t\\begin{equation}\\label{prox and prox sub}\n\t\t(\\forall 0<\\lambda<\\lambda_f)~P_\\lambda  f =\\big(\\Id+\\lambda\\levp f\\big)^{-1}.\n\t\\end{equation}\nEquivalently,\n$ (\\forall x\\in\\Rn)\\ u\\in P_{\\lambda}f(x)\\Leftrightarrow (u, \\lambda^{-1}(x-u))\\in\\gph \\partial_{p}^{\\lambda}f$.\n\\begin{comment}\n\t\tConsequently, operator $\\left(\\Id+\\lambda\\levp f\\right)^{-1}:\\Rn\\to2^{\\Rn}$ has full domain, is compact-valued, and for every $x\\in\\Rn$\n\t\\begin{align*}\n\t\t\\left(\\Id+\\lambda\\partial_p^\\lambda f\\right)^{-1}(x)=\\big\\{v\\in\\Rn:v_k\\to v,\\text{where }(\\exists x_k\\to x)(\\exists \\lambda_k\\to\\lambda)v_k\\in\\left(\\Id+\\lambda_k\\partial_p^{\\lambda_k} f\\right)^{-1}(x_k)\\big\\}.\n\t\\end{align*}\n\tFurthermore $\\gph\\levp f=T\\left(\\gph P_\\lambda f\\right)$, where\n\t$T=\\begin{bmatrix}\n\t\t0&\\Id\\\\\\Id/\\lambda&-\\Id/\\lambda\n\t\\end{bmatrix}$ is an invertible linear transformation.\n\\end{comment}\n\\end{fact}\n\n\\begin{lemma}\\label{l:minty} Let $U$ be a nonempty set in $\\Rn$, let $\\lambda>0$, and let\n$W_{\\lambda} :=\\{(u,v)\\in\\Rn\\times\\Rn| \\ u+\\lambda v\\in U\\}.$\n\\begin{enumerate}\n\\item \\label{i:operator}\nSuppose that $A:\\Rn\\rightrightarrows\\Rn$ and that $J_{\\lambda A}$ is single-valued on $U$.\nThen\n\\begin{equation}\\label{e:mintylocal}\n\\gph A \\cap W_{\\lambda}=\\{(J_{\\lambda A}(x), \\lambda^{-1}[x-J_{\\lambda A}(x)])|\\ x\\in U\\}.\n\\end{equation}\n\\item\\label{i:function}\n Suppose that $f:\\Rn\\rightarrow\\bR$ is proper, lsc and prox-bounded with threshold $\\lambda_{f}$, and $0<\\lambda<\\lambda_{f}$. If $P_{\\lambda}f$ is single-valued on $U$, then\n \\begin{align}\n \\gph \\partial_{p}^{\\lambda}f\\cap W_{\\lambda} &=\\{(P_{\\lambda}f(x), \\lambda^{-1}[x-P_{\\lambda}f(x)])|\\ x\\in U\\}\\label{e:proxf}\\\\\n & =\\{(x-\\lambda \\nabla e_{\\lambda}f(x),\n    \\nabla e_{\\lambda}f(x))|\\ x\\in U\\}.\\label{e:gradf}\n \\end{align}\n\\end{enumerate}\n\\end{lemma}\n\n\\proof\n\\ref{i:operator}:\nDenote the right-hand side of \\eqref{e:mintylocal} by $E_{\\lambda}$. We have\n\\begin{align*}\n(u,v)\\in E_{\\lambda} & \\Leftrightarrow (\\exists x\\in U)\\ u=J_{\\lambda A}(x), v=\\lambda^{-1}[x-J_{\\lambda A}(x)]\\\\\n& \\Leftrightarrow (\\exists x\\in U)\\ x\\in u+\\lambda Au, x=u+\\lambda v\\\\\n& \\Leftrightarrow (\\exists x\\in U)\\ v\\in Au, x=x+\\lambda v\n\\Leftrightarrow (u,v)\\in \\gph A \\cap W_{\\lambda}.\n\\end{align*}\n\\ref{i:function}:  Apply \\ref{i:operator} with $A=\\partial_{p}^{\\lambda}f$ to obtain\n\\eqref{e:proxf}.\nWhen $0<\\lambda<\\lambda_{f}$ and $P_{\\lambda}f$ is single-valued on $U$,\n\\cite[Example 10.32]{rockafellar_variational_1998} implies\n$(\\forall x\\in U)\\ \\nabla e_{\\lambda}f(x)=\\lambda^{-1}[x-P_{\\lambda}f(x)]$.\nHence \\eqref{e:gradf} holds,\n\\endproof\n\n\\begin{lemma}\\label{lem:local coco Minty}\n\tLet $A:\\Rn\\rightrightarrows\\Rn$ be a set-valued mapping, let $\\lambda>0$, and\n\tlet $\\varnothing\\neq U\\subseteq\\dom J_{\\lambda A}$.\n\tDefine $W_\\lambda \\coloneqq \\{(u,v)\\in\\Rn\\times\\Rn|\\ u+\\lambda v\\in U\\}$.\n\tThen for \\(\\gamma>0\\) the following are equivalent:\n\\begin{enumerate}\n\\item\\label{i:resolvent:firm}\n\t$J_{\\lambda A}$ is \\(\\gamma\\)-cocoercive on $U$;\n\\item\\label{i:monotone:hypo}\n$A$ is \\(\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\)-monotone relative to $W_\\lambda$.\n\\end{enumerate}\n%\t\\red{\\textbf{(This should be comono because \\(\\gamma-1\\) may be negative.)}}\n%\t\\begin{enumerate}\n%\t\t\\item\\label{thm:local Minty::coco}\n%\t\t$J_{\\lambda A}$ is \\(\\gamma\\)-cocoercive on $U$;\n%\t\t\\item\\label{thm:local Minty::str mono}\n%\t\t$A$ is \\(\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\)-comonotone relative to $W_\\lambda$.\n%\t\\end{enumerate}\n\\end{lemma}\n\n\\proof\n``\\ref{i:resolvent:firm}$\\Rightarrow$\\ref{i:monotone:hypo}'':\n\tTake $(u_{i},v_{i})\\in \\gph A\n\t\\cap W_{\\lambda}$ and set $x_{i} \\coloneqq u_{i}+\\lambda v_{i}$ for $i=1, 2$. Then\n\t$x_{i}\\in U$ and\n\t$(u_{i},v_{i})=(J_{\\lambda A}(x_{i}), \\lambda^{-1}[x_{i}-J_{\\lambda A}(x_{i})])$\n\tby Lemma~\\ref{l:minty}\\ref{i:operator}.\n\tThen the assumption implies that\n\t\\begin{align*}\n\t\t\\ip{u_1-u_2}{v_1-v_2}\n\t\t&=\n\t\t\\tfrac{1}{\\lambda}\\ip{J_{\\lambda A}(x_1)-J_{\\lambda A}(x_2)}{(x_1-x_2)-(J_{\\lambda A}(x_1)-J_{\\lambda A}(x_2))}\\\\\n\t\t&\\geq\n\t\t\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\norm{J_{\\lambda A}(x_1)-J_{\\lambda A}(x_2)}^2\n\t\t=\n\t\t\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\norm{u_1-u_2}^2,\n\t\\end{align*}\n\tas claimed.\n\n``\\ref{i:monotone:hypo}$\\Rightarrow$\\ref{i:resolvent:firm}'':\n\tSuppose that $x\\in U\\subseteq \\dom J_{\\lambda A}$.\n\tThen $x=u+\\lambda v$ for some $(u,v)\\in\\gph A$, consequently $(u,\\lambda^{-1}(x-u))\\in\\gph A\\cap W_\\lambda$ and $u\\in J_{\\lambda A}(x)$.\n\tWe claim that $J_{\\lambda A}$ is single-valued.\n\tIndeed, let $u'\\in J_{\\lambda A}(x)$.\n\tThen the same process as above gives $(u',\\lambda^{-1}(x-u'))\\in\\gph A\\cap W_\\lambda$, and the\n\\(\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\)-monotonicity of\n$\\gph A\\cap W_\\lambda$ implies\n\t\\[\n\t\\ip{u-u'}{\\tfrac{1}{\\lambda}(x-u)-\\tfrac{1}{\\lambda}(x-u')}\n\t\\geq\n\t\\tfrac{1}{\\lambda}(\\gamma-1)\\norm{u-u'}^2\n\t\\Leftrightarrow\n\t\\gamma\\norm{u-u'}^2\\leq 0 \\Leftrightarrow u=u'.\n\t\\]\n\tNext, let $u_i \\coloneqq J_{\\lambda A}(x_i)$ with $x_i\\in U$ for $i=1,2$.\n\tThen there exists \\(v_i\\in A(u_i)\\) such that \\(x_i=u_i+\\lambda v_i\\) and\n\tthe \\(\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\)-monotonicity of $\\gph A\\cap W_\\lambda$ gives\n\t$\n\t\\ip{u_1-u_2}{v_1-v_2}\\geq\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\norm{u_1-u_2}^2\n\t$, i.e.,\n$\\ip{u_1-u_2}{(\\lambda v_1+u_{1})-(\\lambda v_2+u_{2})}\\geq \\gamma\\norm{u_1-u_2}^2$. Thus,\n$\\ip{J_{\\lambda A}(x_{1})-J_{\\lambda A}(x_{2})}{x_{1}-x_{2}}\\geq \\gamma\\norm{J_{\\lambda A}(x_{1})-J_{\\lambda A}(x_{2})}^2$\n and this is precisely \\ref{i:resolvent:firm}.\n\\endproof\n\n\\begin{lemma}\\label{thm:local Minty}\nLet $A:\\Rn\\rightrightarrows\\Rn$ be a set-valued mapping, let $\\lambda>0$, and\nlet $\\varnothing\\neq U\\subseteq\\dom J_{\\lambda A}$.\nDefine $W_\\lambda :=\\{(u,v)\\in\\Rn\\times\\Rn|\\ u+\\lambda v\\in U\\}$.\nThen the following are equivalent:\n\\begin{enumerate}\n\t\\item\\label{thm:local Minty::fnexp}\n\t $J_{\\lambda A}$ is firmly nonexpansive on $U$;\n\t\\item\\label{thm:local Minty::mono}\n\t$A$ is monotone relative to $W_\\lambda$;\n\t\\item\\label{thm:local Minty::max mono}\n\t $A$ is maximally monotone relative to $W_\\lambda$.\n\\end{enumerate}\n\\end{lemma}\n\n\\proof\nObserve that\n$\n\t[(x,u)\\in\\gph J_{\\lambda A} \\text{ and } x \\in U]\n\t\\Leftrightarrow\n\t\\left(u,\\lambda^{-1}(x-u)\\right)\\in\\gph A\\cap W_\\lambda.\n$\n%Indeed\n%$\n%\t(z,x)\\in\\gph J_{\\lambda A}, z\\in U\n%\t\\Leftrightarrow\n%\tx\\in J_{\\lambda A}(z),z\\in U\n%\t\\Leftrightarrow\n%\t(z-x)/\\lambda\\in Ax,z\\in U\n%$.\n\n``\\ref{thm:local Minty::fnexp}$\\Leftrightarrow$\n\\ref{thm:local Minty::mono}'': Apply Lemma~\\ref{lem:local coco Minty} with $\\gamma=1$.\n %Take $(u_{i},v_{i})\\in \\gph A\n%\\cap W_{\\lambda}$ and set $x_{i} :=u_{i}+\\lambda v_{i}$ for $i=1, 2$. Then\n%$x_{i}\\in U$ and\n%$(u_{i},v_{i})=(J_{\\lambda A}(x_{i}), \\lambda^{-1}[x_{i}-J_{\\lambda A}(x_{i})])$\n%by Lemma~\\ref{l:minty}\\ref{i:operator}.\n%Using \\ref{thm:local Minty::fnexp}, we derive\n%\\begin{align*}\n%\\ip{u_{1}-u_{2}}{v_{1}-v_{2}} &= \\ip{J_{\\lambda A}(x_{1})-J_{\\lambda A}(x_{2})}{\\lambda^{-1}[x_{1}-J_{\\lambda A}(x_{1})]-\\lambda^{-1}[x_{2}-J_{\\lambda A}(x_{2})]}\\\\\n%&=\\lambda^{-1}\\bigg(\\ip{J_{\\lambda A}(x_{1})-J_{\\lambda A}(x_{2})}{x_{1}-x_{2}}-\\norm{J_{\\lambda A}(x_{1})-J_{\\lambda A}(x_{2})}^2\\bigg)\\geq 0.\n%\\end{align*}\n%%Denote by $x_i=J_{\\lambda A}(z_i)$ for $z_i\\in U$ and $i=1,2$.\n%%We have\n%%\\[\n%%\\ip{x_1-x_2}{z_1-z_2}\\geq\\|x_1-x_2\\|^2.\n%%\\]\n%%Dividing by $\\lambda>0$ gives\n%%\\[\n%%\\ip{x_1-x_2}\n%%{\n%%\t\\frac{z_1-x_1}{\\lambda}-\\frac{z_2-x_2}{\\lambda}\n%%}\n%%\\geq0.\n%%\\]\n%Hence $A$ is monotone relative to $W_\\lambda$.\n\n``\\ref{thm:local Minty::mono}$\\Rightarrow$\\ref{thm:local Minty::max mono}\":\nTake $(u,v)\\in W_\\lambda$ that is monotonically related to $\\gph A\\cap W_\\lambda$.\nWe claim that $(u,v)\\in\\gph A$, which entails the desired maximal monotonicity.\nIndeed, $(u,v)\\in W_\\lambda$ implies $u+\\lambda v\\in U\\subseteq \\dom J_{\\lambda A}$.\nLet $y\\in J_{\\lambda A}(u+\\lambda v)$.\nThen\n$\n\\left(y,\\lambda^{-1}(u+\\lambda v-y)\\right)\\in\\gph A\\cap W_\\lambda.\n$\nBy the assumption that $(u,v)$ is monotonically related to $\\gph A\\cap W_\\lambda$, we obtain\n\\begin{align*}\n\\ip{u-y}{\\lambda^{-1}(y-u)}\n=\n\\ip{u-y}{v-\\lambda^{-1}(u+\\lambda v-y)}\\geq 0\n\\Rightarrow\nu=y,\n\\end{align*}\nenforcing that $u\\in J_{\\lambda A}(u+\\lambda v)$.\nIn turn, $u+\\lambda v\\in u+\\lambda Au \\Leftrightarrow v\\in Au$.\n\n``\\ref{thm:local Minty::max mono}$\\Rightarrow$\\ref{thm:local Minty::mono}\":\nClear.\n%Suppose that $x\\in U\\subseteq \\dom J_{\\lambda A}$.\n%Then $x=u+\\lambda v$ for some $(u,v)\\in\\gph A$, consequently $(u,\\lambda^{-1}(x-u))\\in\\gph A\\cap W_\\lambda$ and $u\\in J_{\\lambda A}(x)$.\n%We claim that $J_{\\lambda A}$ is single-valued.\n%Indeed, let $u'\\in J_{\\lambda A}(x)$.\n%Then the same process as above implies $(u',\\lambda^{-1}(x-u'))\\in\\gph A\\cap W_\\lambda$.\n%Appealing to the monotonicity of $\\gph A\\cap W_\\lambda$ yields\n%\\[\n%\\ip\n%{\n%u-u'\n%}\n%{\n%\\lambda^{-1}(x-u)-\\lambda^{-1}(x-u')\n%}\n%\\geq 0\n%\\Leftrightarrow\n%\\ip{u-u'}{-\\lambda^{-1}(u-u')}\\geq 0 \\Leftrightarrow u=u'.\n%\\]\n%Next, let $u_i :=J_{\\lambda A}(x_i)$ with $x_i\\in U$ for $i=1,2$.\n%Then the monotonicity of $\\gph A\\cap W_\\lambda$ again gives\n%$$\n%\\ip{u_1-u_2}{\\lambda^{-1}(x_1-u_1)-\\lambda^{-1}(x_2-u_2)}\\geq0\n%%\\Leftrightarrow\\ip{x_1-x_2}{(z_1-z_2)-(x_1-x_2)}\\geq0\n%\\Leftrightarrow\n%\\ip{u_1-u_2}{x_1-x_2}\\geq\\|u_1-u_2\\|^2,\n%$$\n%justifying \\ref{thm:local Minty::fnexp}.\n\\endproof\n\n\\proof\nBy\n\\cite[Proposition 5.3]{thibault},\nthere exists \\(0<\\bar\\lambda\\leq \\lambda_{f}\\) such that to every \\(0<\\lambda<\\bar\\lambda\\) corresponds an open convex neighborhood $U_\\lambda$ of $\\bar x+\\lambda\\bar v$ on which:\n\\begin{equation}\\label{e:grad1}\n\\text{$P_{\\lambda}f$ is monotone, single-valued and Lipschitz continuous; $P_{\\lambda}f(\\bar x+\\lambda{\\bar v})=\\bar x$;}\n\\end{equation}\nand $e_\\lambda f$ is differentiable with\n\\begin{equation}\\label{e:grad2}\n\\nabla e_{\\lambda}f=\\lambda^{-1}[\\Id-P_\\lambda f].\n\\end{equation}\nSee also \\cite[Proposition 13.37]{rockafellar_variational_1998} (for $\\bar v=0$).\nIn particular, \\eqref{e:grad1} implies $\\bar{v}\\in\\partial_{p}^{\\lambda}f(\\bar{x})\\subseteq\\hat{\\partial}f(\\bar x)$;\nsee Fact \\ref{fact: prox identity}.\n%on (see \\cite[Proposition 2.2]{Khanh2023variational}).\n\n\\begin{comment}\n\t``\\ref{thm:coco env grad:: v cvx}$\\Leftrightarrow$\\ref{envel:convex}\n$\\Leftrightarrow$\\ref{thm:coco env grad:: coco}\":\nAlthough only\n$\\ref{thm:coco env grad:: v cvx}\\Leftrightarrow\\ref{envel:convex}$ stated\nin \\cite[Theorem 3.2]{Khanh2023variational},\nthe proof there\n actually gives $\\ref{thm:coco env grad:: v cvx}\n\\Rightarrow \\ref{thm:coco env grad:: coco}\\Rightarrow\\ref{envel:convex}\n\\Rightarrow \\ref{thm:coco env grad:: v cvx}$.\n\\end{comment}\n\n``\\ref{thm:coco env grad:: v cvx}$\\Rightarrow$\\ref{thm:coco env grad:: mono}'':\nSince $f$ is variationally convex at $\\bar x$ for $\\bar v\\in\\hat{\\partial}f(\\bar{x})$,\nthere exist a convex neighborhood \\(U\\times V\\) of \\((\\bar x,\\bar v)\\) and \\(\\varepsilon>0\\) such that\n\\begin{equation}\\label{e:subf:monotone}\n\t\\gph\\partial f\\cap\\left[U_\\varepsilon\\times V\\right]\n\t\\text{ is monotone},\n\\end{equation}\nwhere \\(U_\\varepsilon :=\\{x\\in U|\\ f(x)<f(\\bar x)+\\varepsilon\\}\\); see \\cite[Theorem 1]{rock-vietnam}.\nFor every \\(0<\\lambda<\\bar\\lambda\\), \\eqref{e:grad1} and \\eqref{e:grad2} entail that\n$(\\forall u\\in U_{\\lambda})\\ u-\\lambda\\nabla e_\\lambda f(u)=P_{\\lambda}f(u)$,\n$P_{\\lambda}f$ is Lipschitz, and\n\\(\\nabla e_\\lambda f(\\bar x+\\lambda \\bar v)=\\bar v\\).\nHence,\n\\begin{equation}\\label{eq:limits}\n\\lim_{u\\to\\bar x+\\lambda\\bar v}\\nabla e_\\lambda f(u)=\\bar v,\n\\lim_{u\\to\\bar x+\\lambda\\bar v}(\\Id-\\lambda\\nabla e_\\lambda f)(u)=\\bar x,\n\\text{ and }\\lim_{u\\to\\bar x+\\lambda\\bar v}f(P_{\\lambda}f(u))=f(\\bar x),\n\\end{equation}\nwhere the last limit holds because\n\\begin{align}\n\\lim_{u\\to\\bar x+\\lambda\\bar v}\tf(P_{\\lambda}f(u)) & =\\lim_{u\\to\\bar x+\\lambda\\bar v }\\Big(e_\\lambda f(u)-\\tfrac{1}{2\\lambda}\\norm{u-P_{\\lambda}f(u)}^2\\Big)\\\\\n&=e_\\lambda f(\\bar x+\\lambda\\bar v)-\\tfrac{1}{2\\lambda}\\norm{\\bar x+\\lambda\\bar v-\\bar x}^2=f(\\bar x).\n\\end{align}\nOwing to \\eqref{eq:limits}, by using a smaller $U_{\\lambda}$ if necessary, we can assume\nthat for \\(u\\in U_\\lambda\\)\n\\begin{equation}\\label{e:f:attentative}\n\tu-\\lambda\\nabla e_\\lambda f(u)\\in U, \\nabla e_\\lambda f(u)\\in V,\n\\text{ and } f(u-\\lambda \\nabla e_\\lambda f(u))<f(\\bar x)+\\varepsilon.\n\\end{equation}\nMoreover,\n\\(u-\\lambda\\nabla e_\\lambda f(u)=P_\\lambda f(u)\\) implies, by Fact~\\ref{fact: prox identity}, that\n\\begin{equation}\\label{e:in:graph}\n(u-\\lambda\\nabla e_\\lambda f(u),\\nabla e_\\lambda f(u))\\in\n\\gra \\partial_{p}^{\\lambda}f\\subseteq \\gra\\partial f.\n\\end{equation}\n%\\(\\nabla e_\\lambda f(u)\\in\\partial f(u-\\lambda\\nabla e_\\lambda f(u))\\).\nCombining \\eqref{e:f:attentative}, \\eqref{e:in:graph} and \\eqref{e:subf:monotone} we obtain\n\\[\n\t\\{(u-\\lambda\\nabla e_\\lambda f(u), \\nabla e_\\lambda f(u))|\\ u\\in U_\\lambda\\}\\subseteq\\gph\\partial f\\cap\\left[U_\\varepsilon\\times V\\right]\n\t\\text{is monotone},\n\\]\nand so is \\(\\gph\\partial_p^\\lambda f\\cap W_\\lambda\\) by Lemma~\\ref{l:minty}.\n\n``\\ref{thm:coco env grad:: mono}$\\Leftrightarrow$\\ref{thmn:coco env grad:: max mono}$\\Leftrightarrow$\\ref{thm:coco env grad:: firmly nonexap prox}'':\nNote that \\(P_\\lambda f=(\\Id+\\lambda\\partial_p^\\lambda f)^{-1}\\).\nThen applying Lemma \\ref{thm:local Minty} completes the proof.\n\\begin{comment}\n\n``\\ref{thm:coco env grad:: coco}$\\Rightarrow$\\ref{thm:coco env grad:: mono}\":\nTake $(u_i,v_i)\\in\\operatorname{gph}\\partial_p^\\lambda f\\cap W_\\lambda$ and set\n$x_i \\coloneqq u_i+\\lambda v_i$ for $i=1,2$.\nThen $x_i\\in U_\\lambda$ by the definition of $W_\\lambda$. Furthermore, $u_i=P_\\lambda f(x_i)$\nby \\eqref{e:grad1} and the identity $P_\\lambda f=\\left(\\operatorname{Id}+\\lambda\\partial_p^\\lambda f\\right)^{-1}$, and $\\nabla e_\\lambda f(x_i)=\\lambda^{-1}\\left(x_i-P_\\lambda f(x_i)\\right)=\\lambda^{-1}\\left(u_i+\\lambda v_i-u_i\\right)=v_i$ by \\eqref{e:grad2}.\nAs such, the $\\lambda$-cocoercity of\n$\\nabla e_\\lambda f$ implies that\n\\begin{align*}\n\t\\left\\langle u_1-u_2,v_1-v_2 \\right\\rangle\n\t&=\n\t\\left\\langle x_1-x_2,v_1-v_2 \\right\\rangle-\\lambda\\left\\|v_1-v_2\\right\\|^2\\\\\n\t&=\\left\\langle x_1-x_2,\\nabla e_\\lambda f(x_1)-\\nabla e_\\lambda f(x_2) \\right\\rangle-\\lambda\\left\\|\\nabla e_\\lambda f(x_1)-\\nabla e_\\lambda f(x_2)\\right\\|^2\\geq0.\n\\end{align*}\nSince $(u_i,v_i)\\in\\operatorname{gph}\\partial_p^\\lambda f\\cap W_\\lambda$ for $i=1, 2$ are arbitrary, $\\partial_p^\\lambda f$ is monotone relative to $W_\\lambda$.\n\n``\\ref{thm:coco env grad:: mono}$\\Rightarrow$\\ref{thm:coco env grad:: coco}\":\nTake  $x_1,x_2\\in U_\\lambda$. Applying Lemma~\\ref{l:minty}\\ref{i:function} on $U_{\\lambda}$,\nwe have\n\\begin{equation}\\label{eq:mono claim}\n\t\\gph\\partial_p^\\lambda f\\cap\tW_\\lambda\n=\\menge{(x-\\lambda\\nabla e_\\lambda f(x),\\nabla e_\\lambda f(x))}{x\\in U_\\lambda}.\n\\end{equation}\nThe assumption entails\n\\begin{align*}\n\t0&\\leq\\left\\langle(x_1-\\lambda\\nabla e_\\lambda f(x_1))\n-(x_2-\\lambda\\nabla e_\\lambda f(x_2)),\\nabla e_\\lambda f(x_1)-\\nabla e_\\lambda f(x_2)\\right\\rangle\\\\\n\t&=\n\t\\left\\langle x_1-x_2,\\nabla e_\\lambda f(x_1)-\\nabla e_\\lambda f(x_2)\\right\\rangle-\\lambda\\left\\|\\nabla e_\\lambda f(x_1)-\\nabla e_\\lambda f(x_2)\\right\\|^2,\n\\end{align*}\nimplying \\ref{thm:coco env grad:: coco}.\n\\end{comment}\n\n%To justify the claim (\\ref{eq:mono claim}), we\n%use \\eqref{e:grad1} and \\eqref{e:grad2} to obtain\n%\\begin{align*}\n%\t(u,v)\\in E_\\lambda\n%\t&\\Leftrightarrow (\\exists x\\in U_\\lambda)~u=P_\\lambda f(x), v=\\nabla e_\\lambda f(x)=\\lambda^{-1}(x-P_\\lambda f(x))\\\\\n%\t&\\Leftrightarrow\n%\t(\\exists x\\in U_\\lambda)~ x\\in u+\\lambda\\partial_p^\\lambda f(u)\\text{ and }x=u+\\lambda v\\\\\n%\t&\\Leftrightarrow(\\exists x\\in U_\\lambda)~ v\\in\\partial_p^\\lambda f(u) \\text{ and }x=u+\\lambda v\\Leftrightarrow (u,v)\\in\\gph\\partial_p^\\lambda f\\cap W_\\lambda.\n%\\end{align*}\n\n\n\\begin{comment}\n{\\color{gray}\n\t``\\ref{thm:coco env grad:: mono}$\\Rightarrow$\\ref{thmn:coco env grad:: max mono}'':\n\tTake $(x,v)\\in W_\\lambda$.\n\tBy the definition of $W_\\lambda$, we have $x+\\lambda v\\in U_\\lambda$, on which the proximal\nmapping is single-valued.\n\tSuppose that\n\t\\begin{equation}\\label{eq: mono rel}\n\t(\\forall (y,u)\\in\\gph\\partial_p^\\lambda f\\cap W_\\lambda)~\n\t\\ip{x-y}{v-u}\\geq0.\n\t\\end{equation}\n\tWe have to prove $(x,v)\\in\\gph\\partial_p^\\lambda f$.\n\tTo this end, let $w \\coloneqq P_\\lambda f(x+\\lambda v)$. Fact~\\ref{fact: prox identity} yields\n\t\\[\n\tw=\n\t\\left(\n\t\\Id+\\lambda \\partial_p^\\lambda f\n\t\\right)^{-1}\n\t(x+\\lambda v)\n\t\\Leftrightarrow\n\t\\lambda^{-1}(x-w)\n\t+v\\in\\partial_p^\\lambda f(w),\n\t\\]\n\tand\n\t$\n\tw+\\lambda\\left(\\lambda^{-1}(x-w)\n\t+v\\right)\n\t=\n\tx+\\lambda v\n\t\\in U_\\lambda.\n\t$\n\tTherefore\n\t$\n\t(w,\\lambda^{-1}(x-w)+v)\\in\\gph \\partial_p^\\lambda f\\cap W_\\lambda\n\t$.\n\tConsequently (\\ref{eq: mono rel}) entails\n\t\\[\n\t\\ip\n\t{x-w}\n\t{v-\\lambda^{-1}(x-w)-v}\\geq0\n\t\\Leftrightarrow\n\t-\\lambda^{-1}\\norm{x-w}^2\\geq0\\Leftrightarrow x=w=P_\\lambda f(x+\\lambda v).\n\t\\]\n\tAppealing to Fact \\ref{fact: prox identity}, we obtain\n\t$\n\tx=P_\\lambda f\\left(x+\\lambda v\\right)=\\left(\\Id+\\lambda\\partial_p^\\lambda f\\right)^{-1}(x+\\lambda v)\\Leftrightarrow x+\\lambda v\\in \\left(\\Id+\\lambda\\partial_p^\\lambda f\\right)(x)\\Leftrightarrow v\\in\\partial_p^\\lambda f(x),\n\t$\n\tas desired.\n}\n\n\t{\\color{gray}\n\t``\\ref{thmn:coco env grad:: max mono}$\\Rightarrow$\\ref{thm:coco env grad:: firmly nonexap prox}'':\n\tFor every $x\\in U_\\lambda$, set $u\\coloneqq P_\\lambda f(x)$. Lemma~\\ref{l:minty}\\ref{i:function} implies\n%Fact \\ref{fact: prox identity} that\n%$(x-u)/\\lambda\\in\\partial_p^\\lambda f(u)$.\n%\tFurthermore $u+\\lambda[(x-u)/\\lambda]=x\\in U_\\lambda\\Leftrightarrow (u,(x-u)/\\lambda)\\in W_\\lambda$.\n%\tHence\n$(u,\\lambda^{-1}(x-u))\\in\\gph\\partial_p^\\lambda f\\cap W_\\lambda$.\n\tNow let $x_i\\in U_\\lambda$ and $u_i \\coloneqq  P_\\lambda f(x_i)$ for\n$x_i\\in U_\\lambda$ and $i=1,2$.\n\tThen the monotonicity of $\\gph\\partial_p^\\lambda f\\cap W_\\lambda$ entails\n\t$$\n\t\\ip{u_1-u_2}{\\lambda^{-1}(x_1-u_1)-\\lambda^{-1}(x_2-u_2)}\\geq 0\n\t\\Leftrightarrow\n\t\\ip{u_1-u_2}{x_1-x_2}\\geq\\norm{u_1-u_2}^2,\n\t$$\n\twhich means that $P_\\lambda f$ is firmly nonexpansive on $U_\\lambda$.\n\t}\n\\end{comment}\n\n``\\ref{thm:coco env grad:: firmly nonexap prox}$\\Rightarrow$\\ref{thm:coco env grad:: averaged}$\\Rightarrow$\\ref{thm:coco env grad:: ne}\": Clear.\n\n``\\ref{thm:coco env grad:: ne}$\\Rightarrow$\\ref{envel:convex}\": The assumption implies that $\\nabla e_{\\lambda}f$ is monotone on $U_{\\lambda}$, so $e_{\\lambda}f$ is convex on $U_{\\lambda}$\n{in view of \\cite[B Theorem 4.1.4]{hiriart2004fundamentals}}.\nIndeed, by the nonexpansiveness of $P_\\lambda f$ on $U_\\lambda$ and the Cauchy-Schwarz inequality,\n\\begin{equation}\\label{e:nonexp:grad}\n(\\forall x_{1}, x_{2}\\in U_\\lambda)~\n\\ip{P_\\lambda f(x_{1})-P_\\lambda f(x_{2})}{x_{1}-x_{2}}\\leq\\norm{x_{1}-x_{2}}^2.\n\\end{equation}\nSince\n$\\nabla e_{\\lambda}f=\\lambda^{-1}[\\Id-P_{\\lambda}f] \\text{ on $U_{\\lambda}$},$\nit follows from \\eqref{e:nonexp:grad} that\n$$(\\forall x_{1},x_{2}\\in U_\\lambda)~\\ip{x_{1}-x_{2}}{\\nabla e_\\lambda f(x_{1})-\\nabla e_\\lambda f(x_{2})}\\geq 0,$$\nhence $\\nabla e_{\\lambda}f$ is monotone on $U_{\\lambda}$.\n\n``\\ref{envel:convex}$\\Rightarrow$\\ref{envel:convex spec}$\\Rightarrow$\\ref{thm:coco env grad:: v cvx}'':\nThe first implication is trivial.\nWe prove the second following the same line of proof in \\cite[Theorem 3.2]{Khanh2023variational} for the sake of completeness.\nBy the prox-regularity of $f$ at $\\bar{x}$ for $\\bar{v}\\in\\partial f(\\bar{x})$,\nthere exist \\(\\gamma>0\\) and \\(T:\\Rn\\rightrightarrows\\Rn\\) defined by\n\\begin{align*}\n\tT(x):=\n\t\\begin{cases}\n\t\t\\{v\\in\\partial f(x)|\\norm{v-\\bar v}<\\gamma\\},&\\text{if }\\norm{x-\\bar x}<\\gamma, f(x)<f(\\bar x)+\\gamma;\\\\\n\t\t\\varnothing, &\\text{otherwise,}\n\t\\end{cases}\n\\end{align*}\nsuch that \\(\\nabla e_\\lambda f=\\lambda^{-1}[\\Id-P_{\\lambda}f]=\n\\left(\\lambda\\Id+T^{-1}\\right)^{-1}\\) on an open convex neighborhood \\(U_\\lambda\\) of $\\bar{x}+\\lambda\\bar{v}$; see\n\\cite[Proposition 13.37]{rockafellar_variational_1998} or \\cite[Proposition 4.4]{thibault}.\nBy taking a smaller $U_{\\lambda}$ if necessary, assumption \\ref{envel:convex spec} implies that\n\\begin{equation}\\label{eq:cvx env}\n\t(\\forall x,y\\in U_\\lambda)~\n\te_\\lambda f(y)\\geq e_\\lambda f(x)+\\ip{\\nabla e_\\lambda f(x)}{y-x}.\n\\end{equation}\nTake \\(\\varepsilon>0\\) sufficiently small so that \\(\\varepsilon<\\gamma\\) and\n\\begin{equation}\\label{e:uv}\n(\\forall x\\in U)(\\forall v\\in V)\\ x+\\lambda v\\in U_\\lambda,\n\\end{equation}\nwhere \\(U:=\\Ball_{\\varepsilon}(\\bar x)\\) and \\(V:=\\Ball_{\\varepsilon}(\\bar v)\\).\nThen \\(v\\in T(x)\\) for \\((x,v)\\in\\gph\\partial f\\cap[U_\\varepsilon\\times V]\\) with\n\\(U_\\varepsilon:=\\{x\\in U|\\ f(x)<f(\\bar x)+\\varepsilon\\}\\), implying that\n\\begin{align}\n\t\\nabla e_\\lambda f(x+\\lambda v) &=\\left(\\lambda\\Id+T^{-1}\\right)^{-1}(x+\\lambda v)=v, \\text{ and }\n\t\\label{eq:env and prox}\\\\\n\tP_\\lambda f(x+\\lambda v) &=(x+\\lambda v)-\\lambda\\nabla e_{\\lambda}f(x+\\lambda v)=x.\n\\label{eq:env and prox1}\n\\end{align}\nTherefore, for every \\(y\\in U\\) and \\((x,v)\\in\\gph\\partial f\\cap[U_\\varepsilon\\times V]\\), we have \\(y+\\lambda v, x+\\lambda v\\in U_\\lambda\\) by \\eqref{e:uv} so that\n\\begin{align*}\n\tf(y)\n&\\geq\n\te_\\lambda f(y+\\lambda v)-\\tfrac{1}{2\\lambda}\\norm{y+\\lambda v-y}^2\\\\\n&\\geq\n\te_\\lambda f(x+\\lambda v)+\\ip{\\nabla e_\\lambda f(x+\\lambda v)}{y-x}-\\tfrac{1}{2\\lambda}\\norm{x+\\lambda v-x}^2\\\\\n&=e_\\lambda f(x+\\lambda v)-\\tfrac{1}{2\\lambda}\\norm{x+\\lambda v-x}^2+\\ip{v}{y-x}\\\\\n&=\n\tf(x)+\\ip{v}{y-x},\n\\end{align*}\nwhere the second inequality owes to \\eqref{eq:cvx env}, the first equality by \\eqref{eq:env and prox}, and the second\nequality by \\eqref{eq:env and prox1}.\nAppealing to \\cite[Theorem 1(b)$\\Leftrightarrow$(c)]{rock-vietnam} completes the proof.\n\n``\\ref{thm:coco env grad:: firmly nonexap prox}$\\Leftrightarrow$\\ref{thm:coco env grad:: coco}'':\nObserve that \\ref{thm:coco env grad:: coco} amounts to \\(\\Id-P_\\lambda f=\\lambda\\nabla e_\\lambda f\\) is firmly nonexpansive on \\(U_\\lambda\\), which is equivalent to \\ref{thm:coco env grad:: firmly nonexap prox}.\n\\begin{comment}\n``\\ref{thm:coco env grad:: mono}$\\Leftrightarrow$\\ref{thm:coco env grad:: strly mono}'':\nDenote by $g \\coloneqq \\lambda^{-1}j+f$ for convenience.  Fact~\\ref{fact: level and Fenchel sub}\nentails that\n\\begin{equation}\\label{eq:sub graphs }\n\t(u,v)\\in\\gph\\partial_p^\\lambda f\\Leftrightarrow (u,\\lambda^{-1}u+v)\\in\\gph\\partial_F g.\n\\end{equation}\nSuppose that \\ref{thm:coco env grad:: mono} holds and take $(u_i,u_i^*)\\in\\gph\\partial_F g$ with $u_i^*\\in\\lambda^{-1}U_\\lambda$ for $i=1,2$. Set $v_i \\coloneqq u_i^*-\\lambda^{-1}u_i$.\nThen~(\\ref{eq:sub graphs }) implies $v_i\n\\in\\partial_p^\\lambda f(u_i)$ and $u_i+\\lambda v_i=u_i+\\lambda(u_i^*-\\lambda^{-1}u_i)=\\lambda u_i^*\\in U_\\lambda$ by the assumption that $u_i^*\\in\\lambda^{-1}U_\\lambda$.\nTherefore $(u_i,v_i)\\in\\gph\\partial_p^\\lambda f\\cap W_\\lambda$, to which the assumption \\ref{thm:coco env grad:: mono} is applicable and yields\n$\n\\ip{u_1-u_2}{u_1^*-u_2^*}-\\lambda^{-1}\\norm{u_1-u_2}^2=\\ip{u_1-u_2}{v_1-v_2}\\geq0,\n$\njustifying \\ref{thm:coco env grad:: strly mono}.\n\nConversely, suppose \\ref{thm:coco env grad:: strly mono} holds.\nLet $(u_{i},v_{i})\\in \\gph\\partial_p^\\lambda f\\cap W_\\lambda$\nand $x_i \\coloneqq u_{i}+\\lambda v_{i}$ for $i=1,2$. Then\n$v_{i}\\in \\partial_p^\\lambda f(u_{i}), x_i\\in U_{\\lambda},$\nwhich by \\eqref{eq:sub graphs } implies\n$$\\lambda^{-1}x_{i}=\\lambda^{-1}u_{i}+v_{i}\\in\\partial_{F} g(u_{i}), \\quad \\lambda^{-1}x_{i}\\in\\lambda^{-1} U_{\\lambda},$$\nthus\n$(u_{i},\\lambda^{-1}x_{i})\\in\\gph\\partial_{F}g\\cap (\\Rn\\times [\\lambda^{-1}U_{\\lambda}])$.\nBy \\ref{thm:coco env grad:: strly mono}, we have\n$$\n\\ip{u_{1}-u_{2}}{\\lambda^{-1} u_{1}+v_{1}-(\\lambda^{-1}u_{2}+v_{2})}\\geq \\lambda^{-1}\\norm{u_{1}-u_{2}}^2,\n$$\ni.e., $\\ip{u_{1}-u_{2}}{v_{1}-v_{2}}\\geq 0$,\nwhich establishes \\ref{thm:coco env grad:: mono}.\n\\end{comment}\n%The converse direction follows from an identical argument that appeals to (\\ref{eq:sub graphs }), hence is omitted for simplicity.\n\\begin{comment}\n\t{\\color{blue}\n``\\ref{thm:coco env grad:: averaged}$\\Leftrightarrow$\\ref{thm:coco env grad:: comono}'':\nFor \\(0<\\alpha<1\\), define \\(B=2\\alpha\\left[(\\lambda\\partial_p^\\lambda f)^{-1}-\\left((2\\alpha)^{-1}-1\\right)\\Id\\right]\\), \\(N=2J_B-\\Id\\), and \\(R=-(2\\alpha)^{-1}N(2\\alpha\\Id)\\).\nNote that \\(P_\\lambda f=(\\Id+\\lambda\\partial_p^\\lambda f)^{-1}\\).\nThen appealing to \\cite[Lemma 2.4]{luo2024various} yields \\(P_\\lambda f=(1-\\alpha)\\Id+\\alpha R\\).\nAs such, \\ref{thm:coco env grad:: averaged} \\(\\Leftrightarrow\\) \\(R\\) is nonexpansive on \\(U_\\lambda\\)  \\(\\Leftrightarrow\\) \\(N\\) is nonexpansive on \\(U_\\lambda\\)  \\(\\Leftrightarrow\\) \\(J_B\\) is firmly nonexpansive on \\(U_\\lambda\\), which is equivalent to \\(B\\) being (maximally) monotone relative to \\(W_\\lambda\\) in view of Lemma \\ref{thm:local Minty}.\nIn turn for every \\((x_i,u_i)\\in\\gph\\lambda\\partial_p^\\lambda f\\), \\(x_i\\in(\\lambda\\partial_p^\\lambda f)^{-1}(\\lambda^{-1}u_i)\\) and\n\\begin{align*}\n\t2\\alpha\\ip{\\lambda^{-1}(u_1-u_2)}{(x_1-x_2)-\\left((2\\alpha)^{-1}-1\\right)\\lambda^{-1}(u_1-u_2)}\\geq0,\n%\t\\Leftrightarrow\n%\t&\\ip{u_1-u_2}{x_1-x_2}\\geq\\left((2\\alpha)^{-1}-1\\right)\\lambda^{-1}\\norm{u_1-u_2}^2.\n\\end{align*}\nwhich amounts to \\ref{thm:coco env grad:: comono}.\n}\n\\end{comment}\n\\endproof","units":[{"start":34281,"end":35368,"start_line":647,"end_line":668,"raw_tex":"\\begin{fact}\n%[resolvent representation of proximal mapping]\n\\label{fact: prox identity}\n\t\\emph{\\cite[Theorem 3.7]{wang23level}}\n\tLet $f:\\Rn\\to\\overline{\\R}$ be proper, lsc and prox-bounded with threshold\n$\\lambda_f>0$. Then\n\t\\begin{equation}\\label{prox and prox sub}\n\t\t(\\forall 0<\\lambda<\\lambda_f)~P_\\lambda  f =\\big(\\Id+\\lambda\\levp f\\big)^{-1}.\n\t\\end{equation}\nEquivalently,\n$ (\\forall x\\in\\Rn)\\ u\\in P_{\\lambda}f(x)\\Leftrightarrow (u, \\lambda^{-1}(x-u))\\in\\gph \\partial_{p}^{\\lambda}f$.\n\\begin{comment}\n\t\tConsequently, operator $\\left(\\Id+\\lambda\\levp f\\right)^{-1}:\\Rn\\to2^{\\Rn}$ has full domain, is compact-valued, and for every $x\\in\\Rn$\n\t\\begin{align*}\n\t\t\\left(\\Id+\\lambda\\partial_p^\\lambda f\\right)^{-1}(x)=\\big\\{v\\in\\Rn:v_k\\to v,\\text{where }(\\exists x_k\\to x)(\\exists \\lambda_k\\to\\lambda)v_k\\in\\left(\\Id+\\lambda_k\\partial_p^{\\lambda_k} f\\right)^{-1}(x_k)\\big\\}.\n\t\\end{align*}\n\tFurthermore $\\gph\\levp f=T\\left(\\gph P_\\lambda f\\right)$, where\n\t$T=\\begin{bmatrix}\n\t\t0&\\Id\\\\\\Id/\\lambda&-\\Id/\\lambda\n\t\\end{bmatrix}$ is an invertible linear transformation.\n\\end{comment}\n\\end{fact}","sha256_utf8":"1cba6efc7e5bd36ce3473214103b2c25b24ec4ddc27acca061f1b7ee9a538e91"},{"start":82845,"end":83758,"start_line":1670,"end_line":1687,"raw_tex":"\\begin{lemma}\\label{l:minty} Let $U$ be a nonempty set in $\\Rn$, let $\\lambda>0$, and let\n$W_{\\lambda} :=\\{(u,v)\\in\\Rn\\times\\Rn| \\ u+\\lambda v\\in U\\}.$\n\\begin{enumerate}\n\\item \\label{i:operator}\nSuppose that $A:\\Rn\\rightrightarrows\\Rn$ and that $J_{\\lambda A}$ is single-valued on $U$.\nThen\n\\begin{equation}\\label{e:mintylocal}\n\\gph A \\cap W_{\\lambda}=\\{(J_{\\lambda A}(x), \\lambda^{-1}[x-J_{\\lambda A}(x)])|\\ x\\in U\\}.\n\\end{equation}\n\\item\\label{i:function}\n Suppose that $f:\\Rn\\rightarrow\\bR$ is proper, lsc and prox-bounded with threshold $\\lambda_{f}$, and $0<\\lambda<\\lambda_{f}$. If $P_{\\lambda}f$ is single-valued on $U$, then\n \\begin{align}\n \\gph \\partial_{p}^{\\lambda}f\\cap W_{\\lambda} &=\\{(P_{\\lambda}f(x), \\lambda^{-1}[x-P_{\\lambda}f(x)])|\\ x\\in U\\}\\label{e:proxf}\\\\\n & =\\{(x-\\lambda \\nabla e_{\\lambda}f(x),\n    \\nabla e_{\\lambda}f(x))|\\ x\\in U\\}.\\label{e:gradf}\n \\end{align}\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"92fdefe40daecf8dd810d6f90275fee08bcd2825211f6ea1014aa6d9a66f4413"},{"start":83759,"end":84531,"start_line":1688,"end_line":1703,"raw_tex":"\\proof\n\\ref{i:operator}:\nDenote the right-hand side of \\eqref{e:mintylocal} by $E_{\\lambda}$. We have\n\\begin{align*}\n(u,v)\\in E_{\\lambda} & \\Leftrightarrow (\\exists x\\in U)\\ u=J_{\\lambda A}(x), v=\\lambda^{-1}[x-J_{\\lambda A}(x)]\\\\\n& \\Leftrightarrow (\\exists x\\in U)\\ x\\in u+\\lambda Au, x=u+\\lambda v\\\\\n& \\Leftrightarrow (\\exists x\\in U)\\ v\\in Au, x=x+\\lambda v\n\\Leftrightarrow (u,v)\\in \\gph A \\cap W_{\\lambda}.\n\\end{align*}\n\\ref{i:function}:  Apply \\ref{i:operator} with $A=\\partial_{p}^{\\lambda}f$ to obtain\n\\eqref{e:proxf}.\nWhen $0<\\lambda<\\lambda_{f}$ and $P_{\\lambda}f$ is single-valued on $U$,\n\\cite[Example 10.32]{rockafellar_variational_1998} implies\n$(\\forall x\\in U)\\ \\nabla e_{\\lambda}f(x)=\\lambda^{-1}[x-P_{\\lambda}f(x)]$.\nHence \\eqref{e:gradf} holds,\n\\endproof","sha256_utf8":"12944021a4502ba36745a541207faf321b7e51dfbc5e8ead0b92156c447f672c"},{"start":84647,"end":85527,"start_line":1707,"end_line":1725,"raw_tex":"\\begin{lemma}\\label{lem:local coco Minty}\n\tLet $A:\\Rn\\rightrightarrows\\Rn$ be a set-valued mapping, let $\\lambda>0$, and\n\tlet $\\varnothing\\neq U\\subseteq\\dom J_{\\lambda A}$.\n\tDefine $W_\\lambda \\coloneqq \\{(u,v)\\in\\Rn\\times\\Rn|\\ u+\\lambda v\\in U\\}$.\n\tThen for \\(\\gamma>0\\) the following are equivalent:\n\\begin{enumerate}\n\\item\\label{i:resolvent:firm}\n\t$J_{\\lambda A}$ is \\(\\gamma\\)-cocoercive on $U$;\n\\item\\label{i:monotone:hypo}\n$A$ is \\(\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\)-monotone relative to $W_\\lambda$.\n\\end{enumerate}\n%\t\\red{\\textbf{(This should be comono because \\(\\gamma-1\\) may be negative.)}}\n%\t\\begin{enumerate}\n%\t\t\\item\\label{thm:local Minty::coco}\n%\t\t$J_{\\lambda A}$ is \\(\\gamma\\)-cocoercive on $U$;\n%\t\t\\item\\label{thm:local Minty::str mono}\n%\t\t$A$ is \\(\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\)-comonotone relative to $W_\\lambda$.\n%\t\\end{enumerate}\n\\end{lemma}","sha256_utf8":"31b3badea210d6929cc49b4f8fd49eac7485288cbff494b79461cf6871cea32d"},{"start":85528,"end":87554,"start_line":1726,"end_line":1769,"raw_tex":"\\proof\n``\\ref{i:resolvent:firm}$\\Rightarrow$\\ref{i:monotone:hypo}'':\n\tTake $(u_{i},v_{i})\\in \\gph A\n\t\\cap W_{\\lambda}$ and set $x_{i} \\coloneqq u_{i}+\\lambda v_{i}$ for $i=1, 2$. Then\n\t$x_{i}\\in U$ and\n\t$(u_{i},v_{i})=(J_{\\lambda A}(x_{i}), \\lambda^{-1}[x_{i}-J_{\\lambda A}(x_{i})])$\n\tby Lemma~\\ref{l:minty}\\ref{i:operator}.\n\tThen the assumption implies that\n\t\\begin{align*}\n\t\t\\ip{u_1-u_2}{v_1-v_2}\n\t\t&=\n\t\t\\tfrac{1}{\\lambda}\\ip{J_{\\lambda A}(x_1)-J_{\\lambda A}(x_2)}{(x_1-x_2)-(J_{\\lambda A}(x_1)-J_{\\lambda A}(x_2))}\\\\\n\t\t&\\geq\n\t\t\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\norm{J_{\\lambda A}(x_1)-J_{\\lambda A}(x_2)}^2\n\t\t=\n\t\t\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\norm{u_1-u_2}^2,\n\t\\end{align*}\n\tas claimed.\n\n``\\ref{i:monotone:hypo}$\\Rightarrow$\\ref{i:resolvent:firm}'':\n\tSuppose that $x\\in U\\subseteq \\dom J_{\\lambda A}$.\n\tThen $x=u+\\lambda v$ for some $(u,v)\\in\\gph A$, consequently $(u,\\lambda^{-1}(x-u))\\in\\gph A\\cap W_\\lambda$ and $u\\in J_{\\lambda A}(x)$.\n\tWe claim that $J_{\\lambda A}$ is single-valued.\n\tIndeed, let $u'\\in J_{\\lambda A}(x)$.\n\tThen the same process as above gives $(u',\\lambda^{-1}(x-u'))\\in\\gph A\\cap W_\\lambda$, and the\n\\(\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\)-monotonicity of\n$\\gph A\\cap W_\\lambda$ implies\n\t\\[\n\t\\ip{u-u'}{\\tfrac{1}{\\lambda}(x-u)-\\tfrac{1}{\\lambda}(x-u')}\n\t\\geq\n\t\\tfrac{1}{\\lambda}(\\gamma-1)\\norm{u-u'}^2\n\t\\Leftrightarrow\n\t\\gamma\\norm{u-u'}^2\\leq 0 \\Leftrightarrow u=u'.\n\t\\]\n\tNext, let $u_i \\coloneqq J_{\\lambda A}(x_i)$ with $x_i\\in U$ for $i=1,2$.\n\tThen there exists \\(v_i\\in A(u_i)\\) such that \\(x_i=u_i+\\lambda v_i\\) and\n\tthe \\(\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\)-monotonicity of $\\gph A\\cap W_\\lambda$ gives\n\t$\n\t\\ip{u_1-u_2}{v_1-v_2}\\geq\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\norm{u_1-u_2}^2\n\t$, i.e.,\n$\\ip{u_1-u_2}{(\\lambda v_1+u_{1})-(\\lambda v_2+u_{2})}\\geq \\gamma\\norm{u_1-u_2}^2$. Thus,\n$\\ip{J_{\\lambda A}(x_{1})-J_{\\lambda A}(x_{2})}{x_{1}-x_{2}}\\geq \\gamma\\norm{J_{\\lambda A}(x_{1})-J_{\\lambda A}(x_{2})}^2$\n and this is precisely \\ref{i:resolvent:firm}.\n\\endproof","sha256_utf8":"33763399a96ca3f6eaadf757bcc92cbaf78f534d3821040e690282f8c8a0319a"},{"start":87702,"end":88272,"start_line":1774,"end_line":1787,"raw_tex":"\\begin{lemma}\\label{thm:local Minty}\nLet $A:\\Rn\\rightrightarrows\\Rn$ be a set-valued mapping, let $\\lambda>0$, and\nlet $\\varnothing\\neq U\\subseteq\\dom J_{\\lambda A}$.\nDefine $W_\\lambda :=\\{(u,v)\\in\\Rn\\times\\Rn|\\ u+\\lambda v\\in U\\}$.\nThen the following are equivalent:\n\\begin{enumerate}\n\t\\item\\label{thm:local Minty::fnexp}\n\t $J_{\\lambda A}$ is firmly nonexpansive on $U$;\n\t\\item\\label{thm:local Minty::mono}\n\t$A$ is monotone relative to $W_\\lambda$;\n\t\\item\\label{thm:local Minty::max mono}\n\t $A$ is maximally monotone relative to $W_\\lambda$.\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"28b73a277042a721482b711610b471948aff3f3eba6f5c11697fc2419bde04b9"},{"start":88273,"end":91447,"start_line":1788,"end_line":1880,"raw_tex":"\\proof\nObserve that\n$\n\t[(x,u)\\in\\gph J_{\\lambda A} \\text{ and } x \\in U]\n\t\\Leftrightarrow\n\t\\left(u,\\lambda^{-1}(x-u)\\right)\\in\\gph A\\cap W_\\lambda.\n$\n%Indeed\n%$\n%\t(z,x)\\in\\gph J_{\\lambda A}, z\\in U\n%\t\\Leftrightarrow\n%\tx\\in J_{\\lambda A}(z),z\\in U\n%\t\\Leftrightarrow\n%\t(z-x)/\\lambda\\in Ax,z\\in U\n%$.\n\n``\\ref{thm:local Minty::fnexp}$\\Leftrightarrow$\n\\ref{thm:local Minty::mono}'': Apply Lemma~\\ref{lem:local coco Minty} with $\\gamma=1$.\n %Take $(u_{i},v_{i})\\in \\gph A\n%\\cap W_{\\lambda}$ and set $x_{i} :=u_{i}+\\lambda v_{i}$ for $i=1, 2$. Then\n%$x_{i}\\in U$ and\n%$(u_{i},v_{i})=(J_{\\lambda A}(x_{i}), \\lambda^{-1}[x_{i}-J_{\\lambda A}(x_{i})])$\n%by Lemma~\\ref{l:minty}\\ref{i:operator}.\n%Using \\ref{thm:local Minty::fnexp}, we derive\n%\\begin{align*}\n%\\ip{u_{1}-u_{2}}{v_{1}-v_{2}} &= \\ip{J_{\\lambda A}(x_{1})-J_{\\lambda A}(x_{2})}{\\lambda^{-1}[x_{1}-J_{\\lambda A}(x_{1})]-\\lambda^{-1}[x_{2}-J_{\\lambda A}(x_{2})]}\\\\\n%&=\\lambda^{-1}\\bigg(\\ip{J_{\\lambda A}(x_{1})-J_{\\lambda A}(x_{2})}{x_{1}-x_{2}}-\\norm{J_{\\lambda A}(x_{1})-J_{\\lambda A}(x_{2})}^2\\bigg)\\geq 0.\n%\\end{align*}\n%%Denote by $x_i=J_{\\lambda A}(z_i)$ for $z_i\\in U$ and $i=1,2$.\n%%We have\n%%\\[\n%%\\ip{x_1-x_2}{z_1-z_2}\\geq\\|x_1-x_2\\|^2.\n%%\\]\n%%Dividing by $\\lambda>0$ gives\n%%\\[\n%%\\ip{x_1-x_2}\n%%{\n%%\t\\frac{z_1-x_1}{\\lambda}-\\frac{z_2-x_2}{\\lambda}\n%%}\n%%\\geq0.\n%%\\]\n%Hence $A$ is monotone relative to $W_\\lambda$.\n\n``\\ref{thm:local Minty::mono}$\\Rightarrow$\\ref{thm:local Minty::max mono}\":\nTake $(u,v)\\in W_\\lambda$ that is monotonically related to $\\gph A\\cap W_\\lambda$.\nWe claim that $(u,v)\\in\\gph A$, which entails the desired maximal monotonicity.\nIndeed, $(u,v)\\in W_\\lambda$ implies $u+\\lambda v\\in U\\subseteq \\dom J_{\\lambda A}$.\nLet $y\\in J_{\\lambda A}(u+\\lambda v)$.\nThen\n$\n\\left(y,\\lambda^{-1}(u+\\lambda v-y)\\right)\\in\\gph A\\cap W_\\lambda.\n$\nBy the assumption that $(u,v)$ is monotonically related to $\\gph A\\cap W_\\lambda$, we obtain\n\\begin{align*}\n\\ip{u-y}{\\lambda^{-1}(y-u)}\n=\n\\ip{u-y}{v-\\lambda^{-1}(u+\\lambda v-y)}\\geq 0\n\\Rightarrow\nu=y,\n\\end{align*}\nenforcing that $u\\in J_{\\lambda A}(u+\\lambda v)$.\nIn turn, $u+\\lambda v\\in u+\\lambda Au \\Leftrightarrow v\\in Au$.\n\n``\\ref{thm:local Minty::max mono}$\\Rightarrow$\\ref{thm:local Minty::mono}\":\nClear.\n%Suppose that $x\\in U\\subseteq \\dom J_{\\lambda A}$.\n%Then $x=u+\\lambda v$ for some $(u,v)\\in\\gph A$, consequently $(u,\\lambda^{-1}(x-u))\\in\\gph A\\cap W_\\lambda$ and $u\\in J_{\\lambda A}(x)$.\n%We claim that $J_{\\lambda A}$ is single-valued.\n%Indeed, let $u'\\in J_{\\lambda A}(x)$.\n%Then the same process as above implies $(u',\\lambda^{-1}(x-u'))\\in\\gph A\\cap W_\\lambda$.\n%Appealing to the monotonicity of $\\gph A\\cap W_\\lambda$ yields\n%\\[\n%\\ip\n%{\n%u-u'\n%}\n%{\n%\\lambda^{-1}(x-u)-\\lambda^{-1}(x-u')\n%}\n%\\geq 0\n%\\Leftrightarrow\n%\\ip{u-u'}{-\\lambda^{-1}(u-u')}\\geq 0 \\Leftrightarrow u=u'.\n%\\]\n%Next, let $u_i :=J_{\\lambda A}(x_i)$ with $x_i\\in U$ for $i=1,2$.\n%Then the monotonicity of $\\gph A\\cap W_\\lambda$ again gives\n%$$\n%\\ip{u_1-u_2}{\\lambda^{-1}(x_1-u_1)-\\lambda^{-1}(x_2-u_2)}\\geq0\n%%\\Leftrightarrow\\ip{x_1-x_2}{(z_1-z_2)-(x_1-x_2)}\\geq0\n%\\Leftrightarrow\n%\\ip{u_1-u_2}{x_1-x_2}\\geq\\|u_1-u_2\\|^2,\n%$$\n%justifying \\ref{thm:local Minty::fnexp}.\n\\endproof","sha256_utf8":"7cf0e77e74861078be8e2bc1f8c303f922beba14ba4a2705a2968fe7a74f2440"},{"start":94341,"end":109979,"start_line":1939,"end_line":2241,"raw_tex":"\\proof\nBy\n\\cite[Proposition 5.3]{thibault},\nthere exists \\(0<\\bar\\lambda\\leq \\lambda_{f}\\) such that to every \\(0<\\lambda<\\bar\\lambda\\) corresponds an open convex neighborhood $U_\\lambda$ of $\\bar x+\\lambda\\bar v$ on which:\n\\begin{equation}\\label{e:grad1}\n\\text{$P_{\\lambda}f$ is monotone, single-valued and Lipschitz continuous; $P_{\\lambda}f(\\bar x+\\lambda{\\bar v})=\\bar x$;}\n\\end{equation}\nand $e_\\lambda f$ is differentiable with\n\\begin{equation}\\label{e:grad2}\n\\nabla e_{\\lambda}f=\\lambda^{-1}[\\Id-P_\\lambda f].\n\\end{equation}\nSee also \\cite[Proposition 13.37]{rockafellar_variational_1998} (for $\\bar v=0$).\nIn particular, \\eqref{e:grad1} implies $\\bar{v}\\in\\partial_{p}^{\\lambda}f(\\bar{x})\\subseteq\\hat{\\partial}f(\\bar x)$;\nsee Fact \\ref{fact: prox identity}.\n%on (see \\cite[Proposition 2.2]{Khanh2023variational}).\n\n\\begin{comment}\n\t``\\ref{thm:coco env grad:: v cvx}$\\Leftrightarrow$\\ref{envel:convex}\n$\\Leftrightarrow$\\ref{thm:coco env grad:: coco}\":\nAlthough only\n$\\ref{thm:coco env grad:: v cvx}\\Leftrightarrow\\ref{envel:convex}$ stated\nin \\cite[Theorem 3.2]{Khanh2023variational},\nthe proof there\n actually gives $\\ref{thm:coco env grad:: v cvx}\n\\Rightarrow \\ref{thm:coco env grad:: coco}\\Rightarrow\\ref{envel:convex}\n\\Rightarrow \\ref{thm:coco env grad:: v cvx}$.\n\\end{comment}\n\n``\\ref{thm:coco env grad:: v cvx}$\\Rightarrow$\\ref{thm:coco env grad:: mono}'':\nSince $f$ is variationally convex at $\\bar x$ for $\\bar v\\in\\hat{\\partial}f(\\bar{x})$,\nthere exist a convex neighborhood \\(U\\times V\\) of \\((\\bar x,\\bar v)\\) and \\(\\varepsilon>0\\) such that\n\\begin{equation}\\label{e:subf:monotone}\n\t\\gph\\partial f\\cap\\left[U_\\varepsilon\\times V\\right]\n\t\\text{ is monotone},\n\\end{equation}\nwhere \\(U_\\varepsilon :=\\{x\\in U|\\ f(x)<f(\\bar x)+\\varepsilon\\}\\); see \\cite[Theorem 1]{rock-vietnam}.\nFor every \\(0<\\lambda<\\bar\\lambda\\), \\eqref{e:grad1} and \\eqref{e:grad2} entail that\n$(\\forall u\\in U_{\\lambda})\\ u-\\lambda\\nabla e_\\lambda f(u)=P_{\\lambda}f(u)$,\n$P_{\\lambda}f$ is Lipschitz, and\n\\(\\nabla e_\\lambda f(\\bar x+\\lambda \\bar v)=\\bar v\\).\nHence,\n\\begin{equation}\\label{eq:limits}\n\\lim_{u\\to\\bar x+\\lambda\\bar v}\\nabla e_\\lambda f(u)=\\bar v,\n\\lim_{u\\to\\bar x+\\lambda\\bar v}(\\Id-\\lambda\\nabla e_\\lambda f)(u)=\\bar x,\n\\text{ and }\\lim_{u\\to\\bar x+\\lambda\\bar v}f(P_{\\lambda}f(u))=f(\\bar x),\n\\end{equation}\nwhere the last limit holds because\n\\begin{align}\n\\lim_{u\\to\\bar x+\\lambda\\bar v}\tf(P_{\\lambda}f(u)) & =\\lim_{u\\to\\bar x+\\lambda\\bar v }\\Big(e_\\lambda f(u)-\\tfrac{1}{2\\lambda}\\norm{u-P_{\\lambda}f(u)}^2\\Big)\\\\\n&=e_\\lambda f(\\bar x+\\lambda\\bar v)-\\tfrac{1}{2\\lambda}\\norm{\\bar x+\\lambda\\bar v-\\bar x}^2=f(\\bar x).\n\\end{align}\nOwing to \\eqref{eq:limits}, by using a smaller $U_{\\lambda}$ if necessary, we can assume\nthat for \\(u\\in U_\\lambda\\)\n\\begin{equation}\\label{e:f:attentative}\n\tu-\\lambda\\nabla e_\\lambda f(u)\\in U, \\nabla e_\\lambda f(u)\\in V,\n\\text{ and } f(u-\\lambda \\nabla e_\\lambda f(u))<f(\\bar x)+\\varepsilon.\n\\end{equation}\nMoreover,\n\\(u-\\lambda\\nabla e_\\lambda f(u)=P_\\lambda f(u)\\) implies, by Fact~\\ref{fact: prox identity}, that\n\\begin{equation}\\label{e:in:graph}\n(u-\\lambda\\nabla e_\\lambda f(u),\\nabla e_\\lambda f(u))\\in\n\\gra \\partial_{p}^{\\lambda}f\\subseteq \\gra\\partial f.\n\\end{equation}\n%\\(\\nabla e_\\lambda f(u)\\in\\partial f(u-\\lambda\\nabla e_\\lambda f(u))\\).\nCombining \\eqref{e:f:attentative}, \\eqref{e:in:graph} and \\eqref{e:subf:monotone} we obtain\n\\[\n\t\\{(u-\\lambda\\nabla e_\\lambda f(u), \\nabla e_\\lambda f(u))|\\ u\\in U_\\lambda\\}\\subseteq\\gph\\partial f\\cap\\left[U_\\varepsilon\\times V\\right]\n\t\\text{is monotone},\n\\]\nand so is \\(\\gph\\partial_p^\\lambda f\\cap W_\\lambda\\) by Lemma~\\ref{l:minty}.\n\n``\\ref{thm:coco env grad:: mono}$\\Leftrightarrow$\\ref{thmn:coco env grad:: max mono}$\\Leftrightarrow$\\ref{thm:coco env grad:: firmly nonexap prox}'':\nNote that \\(P_\\lambda f=(\\Id+\\lambda\\partial_p^\\lambda f)^{-1}\\).\nThen applying Lemma \\ref{thm:local Minty} completes the proof.\n\\begin{comment}\n\n``\\ref{thm:coco env grad:: coco}$\\Rightarrow$\\ref{thm:coco env grad:: mono}\":\nTake $(u_i,v_i)\\in\\operatorname{gph}\\partial_p^\\lambda f\\cap W_\\lambda$ and set\n$x_i \\coloneqq u_i+\\lambda v_i$ for $i=1,2$.\nThen $x_i\\in U_\\lambda$ by the definition of $W_\\lambda$. Furthermore, $u_i=P_\\lambda f(x_i)$\nby \\eqref{e:grad1} and the identity $P_\\lambda f=\\left(\\operatorname{Id}+\\lambda\\partial_p^\\lambda f\\right)^{-1}$, and $\\nabla e_\\lambda f(x_i)=\\lambda^{-1}\\left(x_i-P_\\lambda f(x_i)\\right)=\\lambda^{-1}\\left(u_i+\\lambda v_i-u_i\\right)=v_i$ by \\eqref{e:grad2}.\nAs such, the $\\lambda$-cocoercity of\n$\\nabla e_\\lambda f$ implies that\n\\begin{align*}\n\t\\left\\langle u_1-u_2,v_1-v_2 \\right\\rangle\n\t&=\n\t\\left\\langle x_1-x_2,v_1-v_2 \\right\\rangle-\\lambda\\left\\|v_1-v_2\\right\\|^2\\\\\n\t&=\\left\\langle x_1-x_2,\\nabla e_\\lambda f(x_1)-\\nabla e_\\lambda f(x_2) \\right\\rangle-\\lambda\\left\\|\\nabla e_\\lambda f(x_1)-\\nabla e_\\lambda f(x_2)\\right\\|^2\\geq0.\n\\end{align*}\nSince $(u_i,v_i)\\in\\operatorname{gph}\\partial_p^\\lambda f\\cap W_\\lambda$ for $i=1, 2$ are arbitrary, $\\partial_p^\\lambda f$ is monotone relative to $W_\\lambda$.\n\n``\\ref{thm:coco env grad:: mono}$\\Rightarrow$\\ref{thm:coco env grad:: coco}\":\nTake  $x_1,x_2\\in U_\\lambda$. Applying Lemma~\\ref{l:minty}\\ref{i:function} on $U_{\\lambda}$,\nwe have\n\\begin{equation}\\label{eq:mono claim}\n\t\\gph\\partial_p^\\lambda f\\cap\tW_\\lambda\n=\\menge{(x-\\lambda\\nabla e_\\lambda f(x),\\nabla e_\\lambda f(x))}{x\\in U_\\lambda}.\n\\end{equation}\nThe assumption entails\n\\begin{align*}\n\t0&\\leq\\left\\langle(x_1-\\lambda\\nabla e_\\lambda f(x_1))\n-(x_2-\\lambda\\nabla e_\\lambda f(x_2)),\\nabla e_\\lambda f(x_1)-\\nabla e_\\lambda f(x_2)\\right\\rangle\\\\\n\t&=\n\t\\left\\langle x_1-x_2,\\nabla e_\\lambda f(x_1)-\\nabla e_\\lambda f(x_2)\\right\\rangle-\\lambda\\left\\|\\nabla e_\\lambda f(x_1)-\\nabla e_\\lambda f(x_2)\\right\\|^2,\n\\end{align*}\nimplying \\ref{thm:coco env grad:: coco}.\n\\end{comment}\n\n%To justify the claim (\\ref{eq:mono claim}), we\n%use \\eqref{e:grad1} and \\eqref{e:grad2} to obtain\n%\\begin{align*}\n%\t(u,v)\\in E_\\lambda\n%\t&\\Leftrightarrow (\\exists x\\in U_\\lambda)~u=P_\\lambda f(x), v=\\nabla e_\\lambda f(x)=\\lambda^{-1}(x-P_\\lambda f(x))\\\\\n%\t&\\Leftrightarrow\n%\t(\\exists x\\in U_\\lambda)~ x\\in u+\\lambda\\partial_p^\\lambda f(u)\\text{ and }x=u+\\lambda v\\\\\n%\t&\\Leftrightarrow(\\exists x\\in U_\\lambda)~ v\\in\\partial_p^\\lambda f(u) \\text{ and }x=u+\\lambda v\\Leftrightarrow (u,v)\\in\\gph\\partial_p^\\lambda f\\cap W_\\lambda.\n%\\end{align*}\n\n\n\\begin{comment}\n{\\color{gray}\n\t``\\ref{thm:coco env grad:: mono}$\\Rightarrow$\\ref{thmn:coco env grad:: max mono}'':\n\tTake $(x,v)\\in W_\\lambda$.\n\tBy the definition of $W_\\lambda$, we have $x+\\lambda v\\in U_\\lambda$, on which the proximal\nmapping is single-valued.\n\tSuppose that\n\t\\begin{equation}\\label{eq: mono rel}\n\t(\\forall (y,u)\\in\\gph\\partial_p^\\lambda f\\cap W_\\lambda)~\n\t\\ip{x-y}{v-u}\\geq0.\n\t\\end{equation}\n\tWe have to prove $(x,v)\\in\\gph\\partial_p^\\lambda f$.\n\tTo this end, let $w \\coloneqq P_\\lambda f(x+\\lambda v)$. Fact~\\ref{fact: prox identity} yields\n\t\\[\n\tw=\n\t\\left(\n\t\\Id+\\lambda \\partial_p^\\lambda f\n\t\\right)^{-1}\n\t(x+\\lambda v)\n\t\\Leftrightarrow\n\t\\lambda^{-1}(x-w)\n\t+v\\in\\partial_p^\\lambda f(w),\n\t\\]\n\tand\n\t$\n\tw+\\lambda\\left(\\lambda^{-1}(x-w)\n\t+v\\right)\n\t=\n\tx+\\lambda v\n\t\\in U_\\lambda.\n\t$\n\tTherefore\n\t$\n\t(w,\\lambda^{-1}(x-w)+v)\\in\\gph \\partial_p^\\lambda f\\cap W_\\lambda\n\t$.\n\tConsequently (\\ref{eq: mono rel}) entails\n\t\\[\n\t\\ip\n\t{x-w}\n\t{v-\\lambda^{-1}(x-w)-v}\\geq0\n\t\\Leftrightarrow\n\t-\\lambda^{-1}\\norm{x-w}^2\\geq0\\Leftrightarrow x=w=P_\\lambda f(x+\\lambda v).\n\t\\]\n\tAppealing to Fact \\ref{fact: prox identity}, we obtain\n\t$\n\tx=P_\\lambda f\\left(x+\\lambda v\\right)=\\left(\\Id+\\lambda\\partial_p^\\lambda f\\right)^{-1}(x+\\lambda v)\\Leftrightarrow x+\\lambda v\\in \\left(\\Id+\\lambda\\partial_p^\\lambda f\\right)(x)\\Leftrightarrow v\\in\\partial_p^\\lambda f(x),\n\t$\n\tas desired.\n}\n\n\t{\\color{gray}\n\t``\\ref{thmn:coco env grad:: max mono}$\\Rightarrow$\\ref{thm:coco env grad:: firmly nonexap prox}'':\n\tFor every $x\\in U_\\lambda$, set $u\\coloneqq P_\\lambda f(x)$. Lemma~\\ref{l:minty}\\ref{i:function} implies\n%Fact \\ref{fact: prox identity} that\n%$(x-u)/\\lambda\\in\\partial_p^\\lambda f(u)$.\n%\tFurthermore $u+\\lambda[(x-u)/\\lambda]=x\\in U_\\lambda\\Leftrightarrow (u,(x-u)/\\lambda)\\in W_\\lambda$.\n%\tHence\n$(u,\\lambda^{-1}(x-u))\\in\\gph\\partial_p^\\lambda f\\cap W_\\lambda$.\n\tNow let $x_i\\in U_\\lambda$ and $u_i \\coloneqq  P_\\lambda f(x_i)$ for\n$x_i\\in U_\\lambda$ and $i=1,2$.\n\tThen the monotonicity of $\\gph\\partial_p^\\lambda f\\cap W_\\lambda$ entails\n\t$$\n\t\\ip{u_1-u_2}{\\lambda^{-1}(x_1-u_1)-\\lambda^{-1}(x_2-u_2)}\\geq 0\n\t\\Leftrightarrow\n\t\\ip{u_1-u_2}{x_1-x_2}\\geq\\norm{u_1-u_2}^2,\n\t$$\n\twhich means that $P_\\lambda f$ is firmly nonexpansive on $U_\\lambda$.\n\t}\n\\end{comment}\n\n``\\ref{thm:coco env grad:: firmly nonexap prox}$\\Rightarrow$\\ref{thm:coco env grad:: averaged}$\\Rightarrow$\\ref{thm:coco env grad:: ne}\": Clear.\n\n``\\ref{thm:coco env grad:: ne}$\\Rightarrow$\\ref{envel:convex}\": The assumption implies that $\\nabla e_{\\lambda}f$ is monotone on $U_{\\lambda}$, so $e_{\\lambda}f$ is convex on $U_{\\lambda}$\n{in view of \\cite[B Theorem 4.1.4]{hiriart2004fundamentals}}.\nIndeed, by the nonexpansiveness of $P_\\lambda f$ on $U_\\lambda$ and the Cauchy-Schwarz inequality,\n\\begin{equation}\\label{e:nonexp:grad}\n(\\forall x_{1}, x_{2}\\in U_\\lambda)~\n\\ip{P_\\lambda f(x_{1})-P_\\lambda f(x_{2})}{x_{1}-x_{2}}\\leq\\norm{x_{1}-x_{2}}^2.\n\\end{equation}\nSince\n$\\nabla e_{\\lambda}f=\\lambda^{-1}[\\Id-P_{\\lambda}f] \\text{ on $U_{\\lambda}$},$\nit follows from \\eqref{e:nonexp:grad} that\n$$(\\forall x_{1},x_{2}\\in U_\\lambda)~\\ip{x_{1}-x_{2}}{\\nabla e_\\lambda f(x_{1})-\\nabla e_\\lambda f(x_{2})}\\geq 0,$$\nhence $\\nabla e_{\\lambda}f$ is monotone on $U_{\\lambda}$.\n\n``\\ref{envel:convex}$\\Rightarrow$\\ref{envel:convex spec}$\\Rightarrow$\\ref{thm:coco env grad:: v cvx}'':\nThe first implication is trivial.\nWe prove the second following the same line of proof in \\cite[Theorem 3.2]{Khanh2023variational} for the sake of completeness.\nBy the prox-regularity of $f$ at $\\bar{x}$ for $\\bar{v}\\in\\partial f(\\bar{x})$,\nthere exist \\(\\gamma>0\\) and \\(T:\\Rn\\rightrightarrows\\Rn\\) defined by\n\\begin{align*}\n\tT(x):=\n\t\\begin{cases}\n\t\t\\{v\\in\\partial f(x)|\\norm{v-\\bar v}<\\gamma\\},&\\text{if }\\norm{x-\\bar x}<\\gamma, f(x)<f(\\bar x)+\\gamma;\\\\\n\t\t\\varnothing, &\\text{otherwise,}\n\t\\end{cases}\n\\end{align*}\nsuch that \\(\\nabla e_\\lambda f=\\lambda^{-1}[\\Id-P_{\\lambda}f]=\n\\left(\\lambda\\Id+T^{-1}\\right)^{-1}\\) on an open convex neighborhood \\(U_\\lambda\\) of $\\bar{x}+\\lambda\\bar{v}$; see\n\\cite[Proposition 13.37]{rockafellar_variational_1998} or \\cite[Proposition 4.4]{thibault}.\nBy taking a smaller $U_{\\lambda}$ if necessary, assumption \\ref{envel:convex spec} implies that\n\\begin{equation}\\label{eq:cvx env}\n\t(\\forall x,y\\in U_\\lambda)~\n\te_\\lambda f(y)\\geq e_\\lambda f(x)+\\ip{\\nabla e_\\lambda f(x)}{y-x}.\n\\end{equation}\nTake \\(\\varepsilon>0\\) sufficiently small so that \\(\\varepsilon<\\gamma\\) and\n\\begin{equation}\\label{e:uv}\n(\\forall x\\in U)(\\forall v\\in V)\\ x+\\lambda v\\in U_\\lambda,\n\\end{equation}\nwhere \\(U:=\\Ball_{\\varepsilon}(\\bar x)\\) and \\(V:=\\Ball_{\\varepsilon}(\\bar v)\\).\nThen \\(v\\in T(x)\\) for \\((x,v)\\in\\gph\\partial f\\cap[U_\\varepsilon\\times V]\\) with\n\\(U_\\varepsilon:=\\{x\\in U|\\ f(x)<f(\\bar x)+\\varepsilon\\}\\), implying that\n\\begin{align}\n\t\\nabla e_\\lambda f(x+\\lambda v) &=\\left(\\lambda\\Id+T^{-1}\\right)^{-1}(x+\\lambda v)=v, \\text{ and }\n\t\\label{eq:env and prox}\\\\\n\tP_\\lambda f(x+\\lambda v) &=(x+\\lambda v)-\\lambda\\nabla e_{\\lambda}f(x+\\lambda v)=x.\n\\label{eq:env and prox1}\n\\end{align}\nTherefore, for every \\(y\\in U\\) and \\((x,v)\\in\\gph\\partial f\\cap[U_\\varepsilon\\times V]\\), we have \\(y+\\lambda v, x+\\lambda v\\in U_\\lambda\\) by \\eqref{e:uv} so that\n\\begin{align*}\n\tf(y)\n&\\geq\n\te_\\lambda f(y+\\lambda v)-\\tfrac{1}{2\\lambda}\\norm{y+\\lambda v-y}^2\\\\\n&\\geq\n\te_\\lambda f(x+\\lambda v)+\\ip{\\nabla e_\\lambda f(x+\\lambda v)}{y-x}-\\tfrac{1}{2\\lambda}\\norm{x+\\lambda v-x}^2\\\\\n&=e_\\lambda f(x+\\lambda v)-\\tfrac{1}{2\\lambda}\\norm{x+\\lambda v-x}^2+\\ip{v}{y-x}\\\\\n&=\n\tf(x)+\\ip{v}{y-x},\n\\end{align*}\nwhere the second inequality owes to \\eqref{eq:cvx env}, the first equality by \\eqref{eq:env and prox}, and the second\nequality by \\eqref{eq:env and prox1}.\nAppealing to \\cite[Theorem 1(b)$\\Leftrightarrow$(c)]{rock-vietnam} completes the proof.\n\n``\\ref{thm:coco env grad:: firmly nonexap prox}$\\Leftrightarrow$\\ref{thm:coco env grad:: coco}'':\nObserve that \\ref{thm:coco env grad:: coco} amounts to \\(\\Id-P_\\lambda f=\\lambda\\nabla e_\\lambda f\\) is firmly nonexpansive on \\(U_\\lambda\\), which is equivalent to \\ref{thm:coco env grad:: firmly nonexap prox}.\n\\begin{comment}\n``\\ref{thm:coco env grad:: mono}$\\Leftrightarrow$\\ref{thm:coco env grad:: strly mono}'':\nDenote by $g \\coloneqq \\lambda^{-1}j+f$ for convenience.  Fact~\\ref{fact: level and Fenchel sub}\nentails that\n\\begin{equation}\\label{eq:sub graphs }\n\t(u,v)\\in\\gph\\partial_p^\\lambda f\\Leftrightarrow (u,\\lambda^{-1}u+v)\\in\\gph\\partial_F g.\n\\end{equation}\nSuppose that \\ref{thm:coco env grad:: mono} holds and take $(u_i,u_i^*)\\in\\gph\\partial_F g$ with $u_i^*\\in\\lambda^{-1}U_\\lambda$ for $i=1,2$. Set $v_i \\coloneqq u_i^*-\\lambda^{-1}u_i$.\nThen~(\\ref{eq:sub graphs }) implies $v_i\n\\in\\partial_p^\\lambda f(u_i)$ and $u_i+\\lambda v_i=u_i+\\lambda(u_i^*-\\lambda^{-1}u_i)=\\lambda u_i^*\\in U_\\lambda$ by the assumption that $u_i^*\\in\\lambda^{-1}U_\\lambda$.\nTherefore $(u_i,v_i)\\in\\gph\\partial_p^\\lambda f\\cap W_\\lambda$, to which the assumption \\ref{thm:coco env grad:: mono} is applicable and yields\n$\n\\ip{u_1-u_2}{u_1^*-u_2^*}-\\lambda^{-1}\\norm{u_1-u_2}^2=\\ip{u_1-u_2}{v_1-v_2}\\geq0,\n$\njustifying \\ref{thm:coco env grad:: strly mono}.\n\nConversely, suppose \\ref{thm:coco env grad:: strly mono} holds.\nLet $(u_{i},v_{i})\\in \\gph\\partial_p^\\lambda f\\cap W_\\lambda$\nand $x_i \\coloneqq u_{i}+\\lambda v_{i}$ for $i=1,2$. Then\n$v_{i}\\in \\partial_p^\\lambda f(u_{i}), x_i\\in U_{\\lambda},$\nwhich by \\eqref{eq:sub graphs } implies\n$$\\lambda^{-1}x_{i}=\\lambda^{-1}u_{i}+v_{i}\\in\\partial_{F} g(u_{i}), \\quad \\lambda^{-1}x_{i}\\in\\lambda^{-1} U_{\\lambda},$$\nthus\n$(u_{i},\\lambda^{-1}x_{i})\\in\\gph\\partial_{F}g\\cap (\\Rn\\times [\\lambda^{-1}U_{\\lambda}])$.\nBy \\ref{thm:coco env grad:: strly mono}, we have\n$$\n\\ip{u_{1}-u_{2}}{\\lambda^{-1} u_{1}+v_{1}-(\\lambda^{-1}u_{2}+v_{2})}\\geq \\lambda^{-1}\\norm{u_{1}-u_{2}}^2,\n$$\ni.e., $\\ip{u_{1}-u_{2}}{v_{1}-v_{2}}\\geq 0$,\nwhich establishes \\ref{thm:coco env grad:: mono}.\n\\end{comment}\n%The converse direction follows from an identical argument that appeals to (\\ref{eq:sub graphs }), hence is omitted for simplicity.\n\\begin{comment}\n\t{\\color{blue}\n``\\ref{thm:coco env grad:: averaged}$\\Leftrightarrow$\\ref{thm:coco env grad:: comono}'':\nFor \\(0<\\alpha<1\\), define \\(B=2\\alpha\\left[(\\lambda\\partial_p^\\lambda f)^{-1}-\\left((2\\alpha)^{-1}-1\\right)\\Id\\right]\\), \\(N=2J_B-\\Id\\), and \\(R=-(2\\alpha)^{-1}N(2\\alpha\\Id)\\).\nNote that \\(P_\\lambda f=(\\Id+\\lambda\\partial_p^\\lambda f)^{-1}\\).\nThen appealing to \\cite[Lemma 2.4]{luo2024various} yields \\(P_\\lambda f=(1-\\alpha)\\Id+\\alpha R\\).\nAs such, \\ref{thm:coco env grad:: averaged} \\(\\Leftrightarrow\\) \\(R\\) is nonexpansive on \\(U_\\lambda\\)  \\(\\Leftrightarrow\\) \\(N\\) is nonexpansive on \\(U_\\lambda\\)  \\(\\Leftrightarrow\\) \\(J_B\\) is firmly nonexpansive on \\(U_\\lambda\\), which is equivalent to \\(B\\) being (maximally) monotone relative to \\(W_\\lambda\\) in view of Lemma \\ref{thm:local Minty}.\nIn turn for every \\((x_i,u_i)\\in\\gph\\lambda\\partial_p^\\lambda f\\), \\(x_i\\in(\\lambda\\partial_p^\\lambda f)^{-1}(\\lambda^{-1}u_i)\\) and\n\\begin{align*}\n\t2\\alpha\\ip{\\lambda^{-1}(u_1-u_2)}{(x_1-x_2)-\\left((2\\alpha)^{-1}-1\\right)\\lambda^{-1}(u_1-u_2)}\\geq0,\n%\t\\Leftrightarrow\n%\t&\\ip{u_1-u_2}{x_1-x_2}\\geq\\left((2\\alpha)^{-1}-1\\right)\\lambda^{-1}\\norm{u_1-u_2}^2.\n\\end{align*}\nwhich amounts to \\ref{thm:coco env grad:: comono}.\n}\n\\end{comment}\n\\endproof","sha256_utf8":"be824fb146bf19e39c0f15c2e8743df580540b837c2fda9d48bf22c53ab10e99"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"变分凸性的等价刻画","source_units":[{"start":91711,"end":94339,"start_line":1888,"end_line":1937,"raw_tex":"\\begin{theorem}[characterizations of variational convexity]\\label{thm:coco env grad}\nSuppose that $f:\\mathbb{R}^n\\to\\overline{\\mathbb{R}}$ is proper, lsc and prox-bounded\nwith threshold $\\lambda_{f}>0$,\nand that $f$ is prox-regular at $\\bar x$ for $\\bar v\\in\\partial f(\\bar x)$.\n%{\\color{gray}\n%Then for all $\\lambda>0$ sufficiently small there exists an open convex neighborhood $U_{\\lambda}$\n%of $\\bar{x}+\\lambda \\bar{v}$ on which the following are equivalent:\n%}%\nThen there exists \\(0<\\bar\\lambda\\leq \\lambda_{f}\\) such that $(\\forall 0<\\lambda<\\bar{\\lambda})\\ P_{\\lambda}f(\\barx+\\lambda\\barv)=\\barx$, and $\\exists U_{\\lambda}$\nan open convex neighborhood of \\(\\bar x+\\lambda\\bar v\\) for which the following equivalent properties hold:\n\t\\begin{enumerate}\n\t\t\\item\\label{thm:coco env grad:: v cvx}\n\t\t$f$ is variationally convex at $\\bar x$ for $\\bar v$;\n\t\t\\item\\label{thm:coco env grad:: mono}\n\t\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\)\n\t\t$\\partial_p^\\lambda f$ is monotone relative to $W_\\lambda$;\n% and $\\bar v\\in\\partial_{p}^{\\lambda}f(\\bar x)$;\n\t\t\\item\\label{thmn:coco env grad:: max mono}\n\t\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\)\n\t\t$\\partial_p^\\lambda f$ is maximally monotone relative to $W_\\lambda$;\n% and $\\bar v\\in\\partial_{p}^{\\lambda}f(\\bar x)$;\n\t\t\\item\\label{thm:coco env grad:: firmly nonexap prox}\n\t\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\)\n\t\t$P_\\lambda f$ is firmly nonexpansive on $U_\\lambda$;\n\t\t\\item\\label{thm:coco env grad:: averaged}\n\t\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\) \\((\\exists0<\\alpha<1)\\)\n\t\t$P_\\lambda f$ is $\\alpha$-averaged on $U_\\lambda$;\n% and $P_{\\lambda}f(\\barx+\\lambda\\barv)=\\barx$;\n\t\t\\item\\label{thm:coco env grad:: ne}\n\t\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\)\n\t\t$P_\\lambda f$ is nonexpansive on $U_\\lambda$;\n% and $P_{\\lambda}f(\\barx+\\lambda\\barv)=\\barx$;\n\t%\t\\item\\label{thm:coco env grad:: comono}\n\t%\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\) \\((\\exists0<\\alpha<1)\\)\n\t%\t$\\partial_p^\\lambda f$ is $\\rho$-monotone relative to \\(W_\\lambda\\) with \\(\\rho=((2\\alpha)^{-1}-1)\\lambda^{-1}\\);\n\t\t\\item\\label{envel:convex}\n\t\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\)\n\t\t$e_\\lambda f$ is convex on $U_\\lambda$;\n\t\t\\item\\label{envel:convex spec}\n\t\t\\((\\exists0<\\lambda<\\bar \\lambda)\\)\n\t\t\\(e_\\lambda f\\) is convex on $U_\\lambda$;\n\t\t%($\\partial_F\\left(\\lambda^{-1}j+f\\right)$ is $1/\\lambda$-strongly monotone\n\t\t%relative to $\\Rn\\times\\left[ \\lambda^{-1}U_\\lambda\\right]$.)\n\t\t\\item\\label{thm:coco env grad:: coco}\n\t\t\\((\\forall 0<\\lambda<\\bar\\lambda)\\)\n\t\t$\\nabla e_\\lambda f$ is $\\lambda$-cocoercive on $U_\\lambda$;\n\t\\end{enumerate}\n\t\\vspace*{-2mm}\n\t\twhere $W_\\lambda :=\\{(x,v)\\in\\Rn\\times\\Rn|\\ x+\\lambda v\\in U_\\lambda\\}$.\n\\end{theorem}","sha256_utf8":"4884f147c60e7a41968973bf1fa9a5c29715a3eefb5b8764b8a6b3dbf07c4eb4"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"等价刻画的完整证明","source_units":[{"start":94341,"end":109979,"start_line":1939,"end_line":2241,"raw_tex":"\\proof\nBy\n\\cite[Proposition 5.3]{thibault},\nthere exists \\(0<\\bar\\lambda\\leq \\lambda_{f}\\) such that to every \\(0<\\lambda<\\bar\\lambda\\) corresponds an open convex neighborhood $U_\\lambda$ of $\\bar x+\\lambda\\bar v$ on which:\n\\begin{equation}\\label{e:grad1}\n\\text{$P_{\\lambda}f$ is monotone, single-valued and Lipschitz continuous; $P_{\\lambda}f(\\bar x+\\lambda{\\bar v})=\\bar x$;}\n\\end{equation}\nand $e_\\lambda f$ is differentiable with\n\\begin{equation}\\label{e:grad2}\n\\nabla e_{\\lambda}f=\\lambda^{-1}[\\Id-P_\\lambda f].\n\\end{equation}\nSee also \\cite[Proposition 13.37]{rockafellar_variational_1998} (for $\\bar v=0$).\nIn particular, \\eqref{e:grad1} implies $\\bar{v}\\in\\partial_{p}^{\\lambda}f(\\bar{x})\\subseteq\\hat{\\partial}f(\\bar x)$;\nsee Fact \\ref{fact: prox identity}.\n%on (see \\cite[Proposition 2.2]{Khanh2023variational}).\n\n\\begin{comment}\n\t``\\ref{thm:coco env grad:: v cvx}$\\Leftrightarrow$\\ref{envel:convex}\n$\\Leftrightarrow$\\ref{thm:coco env grad:: coco}\":\nAlthough only\n$\\ref{thm:coco env grad:: v cvx}\\Leftrightarrow\\ref{envel:convex}$ stated\nin \\cite[Theorem 3.2]{Khanh2023variational},\nthe proof there\n actually gives $\\ref{thm:coco env grad:: v cvx}\n\\Rightarrow \\ref{thm:coco env grad:: coco}\\Rightarrow\\ref{envel:convex}\n\\Rightarrow \\ref{thm:coco env grad:: v cvx}$.\n\\end{comment}\n\n``\\ref{thm:coco env grad:: v cvx}$\\Rightarrow$\\ref{thm:coco env grad:: mono}'':\nSince $f$ is variationally convex at $\\bar x$ for $\\bar v\\in\\hat{\\partial}f(\\bar{x})$,\nthere exist a convex neighborhood \\(U\\times V\\) of \\((\\bar x,\\bar v)\\) and \\(\\varepsilon>0\\) such that\n\\begin{equation}\\label{e:subf:monotone}\n\t\\gph\\partial f\\cap\\left[U_\\varepsilon\\times V\\right]\n\t\\text{ is monotone},\n\\end{equation}\nwhere \\(U_\\varepsilon :=\\{x\\in U|\\ f(x)<f(\\bar x)+\\varepsilon\\}\\); see \\cite[Theorem 1]{rock-vietnam}.\nFor every \\(0<\\lambda<\\bar\\lambda\\), \\eqref{e:grad1} and \\eqref{e:grad2} entail that\n$(\\forall u\\in U_{\\lambda})\\ u-\\lambda\\nabla e_\\lambda f(u)=P_{\\lambda}f(u)$,\n$P_{\\lambda}f$ is Lipschitz, and\n\\(\\nabla e_\\lambda f(\\bar x+\\lambda \\bar v)=\\bar v\\).\nHence,\n\\begin{equation}\\label{eq:limits}\n\\lim_{u\\to\\bar x+\\lambda\\bar v}\\nabla e_\\lambda f(u)=\\bar v,\n\\lim_{u\\to\\bar x+\\lambda\\bar v}(\\Id-\\lambda\\nabla e_\\lambda f)(u)=\\bar x,\n\\text{ and }\\lim_{u\\to\\bar x+\\lambda\\bar v}f(P_{\\lambda}f(u))=f(\\bar x),\n\\end{equation}\nwhere the last limit holds because\n\\begin{align}\n\\lim_{u\\to\\bar x+\\lambda\\bar v}\tf(P_{\\lambda}f(u)) & =\\lim_{u\\to\\bar x+\\lambda\\bar v }\\Big(e_\\lambda f(u)-\\tfrac{1}{2\\lambda}\\norm{u-P_{\\lambda}f(u)}^2\\Big)\\\\\n&=e_\\lambda f(\\bar x+\\lambda\\bar v)-\\tfrac{1}{2\\lambda}\\norm{\\bar x+\\lambda\\bar v-\\bar x}^2=f(\\bar x).\n\\end{align}\nOwing to \\eqref{eq:limits}, by using a smaller $U_{\\lambda}$ if necessary, we can assume\nthat for \\(u\\in U_\\lambda\\)\n\\begin{equation}\\label{e:f:attentative}\n\tu-\\lambda\\nabla e_\\lambda f(u)\\in U, \\nabla e_\\lambda f(u)\\in V,\n\\text{ and } f(u-\\lambda \\nabla e_\\lambda f(u))<f(\\bar x)+\\varepsilon.\n\\end{equation}\nMoreover,\n\\(u-\\lambda\\nabla e_\\lambda f(u)=P_\\lambda f(u)\\) implies, by Fact~\\ref{fact: prox identity}, that\n\\begin{equation}\\label{e:in:graph}\n(u-\\lambda\\nabla e_\\lambda f(u),\\nabla e_\\lambda f(u))\\in\n\\gra \\partial_{p}^{\\lambda}f\\subseteq \\gra\\partial f.\n\\end{equation}\n%\\(\\nabla e_\\lambda f(u)\\in\\partial f(u-\\lambda\\nabla e_\\lambda f(u))\\).\nCombining \\eqref{e:f:attentative}, \\eqref{e:in:graph} and \\eqref{e:subf:monotone} we obtain\n\\[\n\t\\{(u-\\lambda\\nabla e_\\lambda f(u), \\nabla e_\\lambda f(u))|\\ u\\in U_\\lambda\\}\\subseteq\\gph\\partial f\\cap\\left[U_\\varepsilon\\times V\\right]\n\t\\text{is monotone},\n\\]\nand so is \\(\\gph\\partial_p^\\lambda f\\cap W_\\lambda\\) by Lemma~\\ref{l:minty}.\n\n``\\ref{thm:coco env grad:: mono}$\\Leftrightarrow$\\ref{thmn:coco env grad:: max mono}$\\Leftrightarrow$\\ref{thm:coco env grad:: firmly nonexap prox}'':\nNote that \\(P_\\lambda f=(\\Id+\\lambda\\partial_p^\\lambda f)^{-1}\\).\nThen applying Lemma \\ref{thm:local Minty} completes the proof.\n\\begin{comment}\n\n``\\ref{thm:coco env grad:: coco}$\\Rightarrow$\\ref{thm:coco env grad:: mono}\":\nTake $(u_i,v_i)\\in\\operatorname{gph}\\partial_p^\\lambda f\\cap W_\\lambda$ and set\n$x_i \\coloneqq u_i+\\lambda v_i$ for $i=1,2$.\nThen $x_i\\in U_\\lambda$ by the definition of $W_\\lambda$. Furthermore, $u_i=P_\\lambda f(x_i)$\nby \\eqref{e:grad1} and the identity $P_\\lambda f=\\left(\\operatorname{Id}+\\lambda\\partial_p^\\lambda f\\right)^{-1}$, and $\\nabla e_\\lambda f(x_i)=\\lambda^{-1}\\left(x_i-P_\\lambda f(x_i)\\right)=\\lambda^{-1}\\left(u_i+\\lambda v_i-u_i\\right)=v_i$ by \\eqref{e:grad2}.\nAs such, the $\\lambda$-cocoercity of\n$\\nabla e_\\lambda f$ implies that\n\\begin{align*}\n\t\\left\\langle u_1-u_2,v_1-v_2 \\right\\rangle\n\t&=\n\t\\left\\langle x_1-x_2,v_1-v_2 \\right\\rangle-\\lambda\\left\\|v_1-v_2\\right\\|^2\\\\\n\t&=\\left\\langle x_1-x_2,\\nabla e_\\lambda f(x_1)-\\nabla e_\\lambda f(x_2) \\right\\rangle-\\lambda\\left\\|\\nabla e_\\lambda f(x_1)-\\nabla e_\\lambda f(x_2)\\right\\|^2\\geq0.\n\\end{align*}\nSince $(u_i,v_i)\\in\\operatorname{gph}\\partial_p^\\lambda f\\cap W_\\lambda$ for $i=1, 2$ are arbitrary, $\\partial_p^\\lambda f$ is monotone relative to $W_\\lambda$.\n\n``\\ref{thm:coco env grad:: mono}$\\Rightarrow$\\ref{thm:coco env grad:: coco}\":\nTake  $x_1,x_2\\in U_\\lambda$. Applying Lemma~\\ref{l:minty}\\ref{i:function} on $U_{\\lambda}$,\nwe have\n\\begin{equation}\\label{eq:mono claim}\n\t\\gph\\partial_p^\\lambda f\\cap\tW_\\lambda\n=\\menge{(x-\\lambda\\nabla e_\\lambda f(x),\\nabla e_\\lambda f(x))}{x\\in U_\\lambda}.\n\\end{equation}\nThe assumption entails\n\\begin{align*}\n\t0&\\leq\\left\\langle(x_1-\\lambda\\nabla e_\\lambda f(x_1))\n-(x_2-\\lambda\\nabla e_\\lambda f(x_2)),\\nabla e_\\lambda f(x_1)-\\nabla e_\\lambda f(x_2)\\right\\rangle\\\\\n\t&=\n\t\\left\\langle x_1-x_2,\\nabla e_\\lambda f(x_1)-\\nabla e_\\lambda f(x_2)\\right\\rangle-\\lambda\\left\\|\\nabla e_\\lambda f(x_1)-\\nabla e_\\lambda f(x_2)\\right\\|^2,\n\\end{align*}\nimplying \\ref{thm:coco env grad:: coco}.\n\\end{comment}\n\n%To justify the claim (\\ref{eq:mono claim}), we\n%use \\eqref{e:grad1} and \\eqref{e:grad2} to obtain\n%\\begin{align*}\n%\t(u,v)\\in E_\\lambda\n%\t&\\Leftrightarrow (\\exists x\\in U_\\lambda)~u=P_\\lambda f(x), v=\\nabla e_\\lambda f(x)=\\lambda^{-1}(x-P_\\lambda f(x))\\\\\n%\t&\\Leftrightarrow\n%\t(\\exists x\\in U_\\lambda)~ x\\in u+\\lambda\\partial_p^\\lambda f(u)\\text{ and }x=u+\\lambda v\\\\\n%\t&\\Leftrightarrow(\\exists x\\in U_\\lambda)~ v\\in\\partial_p^\\lambda f(u) \\text{ and }x=u+\\lambda v\\Leftrightarrow (u,v)\\in\\gph\\partial_p^\\lambda f\\cap W_\\lambda.\n%\\end{align*}\n\n\n\\begin{comment}\n{\\color{gray}\n\t``\\ref{thm:coco env grad:: mono}$\\Rightarrow$\\ref{thmn:coco env grad:: max mono}'':\n\tTake $(x,v)\\in W_\\lambda$.\n\tBy the definition of $W_\\lambda$, we have $x+\\lambda v\\in U_\\lambda$, on which the proximal\nmapping is single-valued.\n\tSuppose that\n\t\\begin{equation}\\label{eq: mono rel}\n\t(\\forall (y,u)\\in\\gph\\partial_p^\\lambda f\\cap W_\\lambda)~\n\t\\ip{x-y}{v-u}\\geq0.\n\t\\end{equation}\n\tWe have to prove $(x,v)\\in\\gph\\partial_p^\\lambda f$.\n\tTo this end, let $w \\coloneqq P_\\lambda f(x+\\lambda v)$. Fact~\\ref{fact: prox identity} yields\n\t\\[\n\tw=\n\t\\left(\n\t\\Id+\\lambda \\partial_p^\\lambda f\n\t\\right)^{-1}\n\t(x+\\lambda v)\n\t\\Leftrightarrow\n\t\\lambda^{-1}(x-w)\n\t+v\\in\\partial_p^\\lambda f(w),\n\t\\]\n\tand\n\t$\n\tw+\\lambda\\left(\\lambda^{-1}(x-w)\n\t+v\\right)\n\t=\n\tx+\\lambda v\n\t\\in U_\\lambda.\n\t$\n\tTherefore\n\t$\n\t(w,\\lambda^{-1}(x-w)+v)\\in\\gph \\partial_p^\\lambda f\\cap W_\\lambda\n\t$.\n\tConsequently (\\ref{eq: mono rel}) entails\n\t\\[\n\t\\ip\n\t{x-w}\n\t{v-\\lambda^{-1}(x-w)-v}\\geq0\n\t\\Leftrightarrow\n\t-\\lambda^{-1}\\norm{x-w}^2\\geq0\\Leftrightarrow x=w=P_\\lambda f(x+\\lambda v).\n\t\\]\n\tAppealing to Fact \\ref{fact: prox identity}, we obtain\n\t$\n\tx=P_\\lambda f\\left(x+\\lambda v\\right)=\\left(\\Id+\\lambda\\partial_p^\\lambda f\\right)^{-1}(x+\\lambda v)\\Leftrightarrow x+\\lambda v\\in \\left(\\Id+\\lambda\\partial_p^\\lambda f\\right)(x)\\Leftrightarrow v\\in\\partial_p^\\lambda f(x),\n\t$\n\tas desired.\n}\n\n\t{\\color{gray}\n\t``\\ref{thmn:coco env grad:: max mono}$\\Rightarrow$\\ref{thm:coco env grad:: firmly nonexap prox}'':\n\tFor every $x\\in U_\\lambda$, set $u\\coloneqq P_\\lambda f(x)$. Lemma~\\ref{l:minty}\\ref{i:function} implies\n%Fact \\ref{fact: prox identity} that\n%$(x-u)/\\lambda\\in\\partial_p^\\lambda f(u)$.\n%\tFurthermore $u+\\lambda[(x-u)/\\lambda]=x\\in U_\\lambda\\Leftrightarrow (u,(x-u)/\\lambda)\\in W_\\lambda$.\n%\tHence\n$(u,\\lambda^{-1}(x-u))\\in\\gph\\partial_p^\\lambda f\\cap W_\\lambda$.\n\tNow let $x_i\\in U_\\lambda$ and $u_i \\coloneqq  P_\\lambda f(x_i)$ for\n$x_i\\in U_\\lambda$ and $i=1,2$.\n\tThen the monotonicity of $\\gph\\partial_p^\\lambda f\\cap W_\\lambda$ entails\n\t$$\n\t\\ip{u_1-u_2}{\\lambda^{-1}(x_1-u_1)-\\lambda^{-1}(x_2-u_2)}\\geq 0\n\t\\Leftrightarrow\n\t\\ip{u_1-u_2}{x_1-x_2}\\geq\\norm{u_1-u_2}^2,\n\t$$\n\twhich means that $P_\\lambda f$ is firmly nonexpansive on $U_\\lambda$.\n\t}\n\\end{comment}\n\n``\\ref{thm:coco env grad:: firmly nonexap prox}$\\Rightarrow$\\ref{thm:coco env grad:: averaged}$\\Rightarrow$\\ref{thm:coco env grad:: ne}\": Clear.\n\n``\\ref{thm:coco env grad:: ne}$\\Rightarrow$\\ref{envel:convex}\": The assumption implies that $\\nabla e_{\\lambda}f$ is monotone on $U_{\\lambda}$, so $e_{\\lambda}f$ is convex on $U_{\\lambda}$\n{in view of \\cite[B Theorem 4.1.4]{hiriart2004fundamentals}}.\nIndeed, by the nonexpansiveness of $P_\\lambda f$ on $U_\\lambda$ and the Cauchy-Schwarz inequality,\n\\begin{equation}\\label{e:nonexp:grad}\n(\\forall x_{1}, x_{2}\\in U_\\lambda)~\n\\ip{P_\\lambda f(x_{1})-P_\\lambda f(x_{2})}{x_{1}-x_{2}}\\leq\\norm{x_{1}-x_{2}}^2.\n\\end{equation}\nSince\n$\\nabla e_{\\lambda}f=\\lambda^{-1}[\\Id-P_{\\lambda}f] \\text{ on $U_{\\lambda}$},$\nit follows from \\eqref{e:nonexp:grad} that\n$$(\\forall x_{1},x_{2}\\in U_\\lambda)~\\ip{x_{1}-x_{2}}{\\nabla e_\\lambda f(x_{1})-\\nabla e_\\lambda f(x_{2})}\\geq 0,$$\nhence $\\nabla e_{\\lambda}f$ is monotone on $U_{\\lambda}$.\n\n``\\ref{envel:convex}$\\Rightarrow$\\ref{envel:convex spec}$\\Rightarrow$\\ref{thm:coco env grad:: v cvx}'':\nThe first implication is trivial.\nWe prove the second following the same line of proof in \\cite[Theorem 3.2]{Khanh2023variational} for the sake of completeness.\nBy the prox-regularity of $f$ at $\\bar{x}$ for $\\bar{v}\\in\\partial f(\\bar{x})$,\nthere exist \\(\\gamma>0\\) and \\(T:\\Rn\\rightrightarrows\\Rn\\) defined by\n\\begin{align*}\n\tT(x):=\n\t\\begin{cases}\n\t\t\\{v\\in\\partial f(x)|\\norm{v-\\bar v}<\\gamma\\},&\\text{if }\\norm{x-\\bar x}<\\gamma, f(x)<f(\\bar x)+\\gamma;\\\\\n\t\t\\varnothing, &\\text{otherwise,}\n\t\\end{cases}\n\\end{align*}\nsuch that \\(\\nabla e_\\lambda f=\\lambda^{-1}[\\Id-P_{\\lambda}f]=\n\\left(\\lambda\\Id+T^{-1}\\right)^{-1}\\) on an open convex neighborhood \\(U_\\lambda\\) of $\\bar{x}+\\lambda\\bar{v}$; see\n\\cite[Proposition 13.37]{rockafellar_variational_1998} or \\cite[Proposition 4.4]{thibault}.\nBy taking a smaller $U_{\\lambda}$ if necessary, assumption \\ref{envel:convex spec} implies that\n\\begin{equation}\\label{eq:cvx env}\n\t(\\forall x,y\\in U_\\lambda)~\n\te_\\lambda f(y)\\geq e_\\lambda f(x)+\\ip{\\nabla e_\\lambda f(x)}{y-x}.\n\\end{equation}\nTake \\(\\varepsilon>0\\) sufficiently small so that \\(\\varepsilon<\\gamma\\) and\n\\begin{equation}\\label{e:uv}\n(\\forall x\\in U)(\\forall v\\in V)\\ x+\\lambda v\\in U_\\lambda,\n\\end{equation}\nwhere \\(U:=\\Ball_{\\varepsilon}(\\bar x)\\) and \\(V:=\\Ball_{\\varepsilon}(\\bar v)\\).\nThen \\(v\\in T(x)\\) for \\((x,v)\\in\\gph\\partial f\\cap[U_\\varepsilon\\times V]\\) with\n\\(U_\\varepsilon:=\\{x\\in U|\\ f(x)<f(\\bar x)+\\varepsilon\\}\\), implying that\n\\begin{align}\n\t\\nabla e_\\lambda f(x+\\lambda v) &=\\left(\\lambda\\Id+T^{-1}\\right)^{-1}(x+\\lambda v)=v, \\text{ and }\n\t\\label{eq:env and prox}\\\\\n\tP_\\lambda f(x+\\lambda v) &=(x+\\lambda v)-\\lambda\\nabla e_{\\lambda}f(x+\\lambda v)=x.\n\\label{eq:env and prox1}\n\\end{align}\nTherefore, for every \\(y\\in U\\) and \\((x,v)\\in\\gph\\partial f\\cap[U_\\varepsilon\\times V]\\), we have \\(y+\\lambda v, x+\\lambda v\\in U_\\lambda\\) by \\eqref{e:uv} so that\n\\begin{align*}\n\tf(y)\n&\\geq\n\te_\\lambda f(y+\\lambda v)-\\tfrac{1}{2\\lambda}\\norm{y+\\lambda v-y}^2\\\\\n&\\geq\n\te_\\lambda f(x+\\lambda v)+\\ip{\\nabla e_\\lambda f(x+\\lambda v)}{y-x}-\\tfrac{1}{2\\lambda}\\norm{x+\\lambda v-x}^2\\\\\n&=e_\\lambda f(x+\\lambda v)-\\tfrac{1}{2\\lambda}\\norm{x+\\lambda v-x}^2+\\ip{v}{y-x}\\\\\n&=\n\tf(x)+\\ip{v}{y-x},\n\\end{align*}\nwhere the second inequality owes to \\eqref{eq:cvx env}, the first equality by \\eqref{eq:env and prox}, and the second\nequality by \\eqref{eq:env and prox1}.\nAppealing to \\cite[Theorem 1(b)$\\Leftrightarrow$(c)]{rock-vietnam} completes the proof.\n\n``\\ref{thm:coco env grad:: firmly nonexap prox}$\\Leftrightarrow$\\ref{thm:coco env grad:: coco}'':\nObserve that \\ref{thm:coco env grad:: coco} amounts to \\(\\Id-P_\\lambda f=\\lambda\\nabla e_\\lambda f\\) is firmly nonexpansive on \\(U_\\lambda\\), which is equivalent to \\ref{thm:coco env grad:: firmly nonexap prox}.\n\\begin{comment}\n``\\ref{thm:coco env grad:: mono}$\\Leftrightarrow$\\ref{thm:coco env grad:: strly mono}'':\nDenote by $g \\coloneqq \\lambda^{-1}j+f$ for convenience.  Fact~\\ref{fact: level and Fenchel sub}\nentails that\n\\begin{equation}\\label{eq:sub graphs }\n\t(u,v)\\in\\gph\\partial_p^\\lambda f\\Leftrightarrow (u,\\lambda^{-1}u+v)\\in\\gph\\partial_F g.\n\\end{equation}\nSuppose that \\ref{thm:coco env grad:: mono} holds and take $(u_i,u_i^*)\\in\\gph\\partial_F g$ with $u_i^*\\in\\lambda^{-1}U_\\lambda$ for $i=1,2$. Set $v_i \\coloneqq u_i^*-\\lambda^{-1}u_i$.\nThen~(\\ref{eq:sub graphs }) implies $v_i\n\\in\\partial_p^\\lambda f(u_i)$ and $u_i+\\lambda v_i=u_i+\\lambda(u_i^*-\\lambda^{-1}u_i)=\\lambda u_i^*\\in U_\\lambda$ by the assumption that $u_i^*\\in\\lambda^{-1}U_\\lambda$.\nTherefore $(u_i,v_i)\\in\\gph\\partial_p^\\lambda f\\cap W_\\lambda$, to which the assumption \\ref{thm:coco env grad:: mono} is applicable and yields\n$\n\\ip{u_1-u_2}{u_1^*-u_2^*}-\\lambda^{-1}\\norm{u_1-u_2}^2=\\ip{u_1-u_2}{v_1-v_2}\\geq0,\n$\njustifying \\ref{thm:coco env grad:: strly mono}.\n\nConversely, suppose \\ref{thm:coco env grad:: strly mono} holds.\nLet $(u_{i},v_{i})\\in \\gph\\partial_p^\\lambda f\\cap W_\\lambda$\nand $x_i \\coloneqq u_{i}+\\lambda v_{i}$ for $i=1,2$. Then\n$v_{i}\\in \\partial_p^\\lambda f(u_{i}), x_i\\in U_{\\lambda},$\nwhich by \\eqref{eq:sub graphs } implies\n$$\\lambda^{-1}x_{i}=\\lambda^{-1}u_{i}+v_{i}\\in\\partial_{F} g(u_{i}), \\quad \\lambda^{-1}x_{i}\\in\\lambda^{-1} U_{\\lambda},$$\nthus\n$(u_{i},\\lambda^{-1}x_{i})\\in\\gph\\partial_{F}g\\cap (\\Rn\\times [\\lambda^{-1}U_{\\lambda}])$.\nBy \\ref{thm:coco env grad:: strly mono}, we have\n$$\n\\ip{u_{1}-u_{2}}{\\lambda^{-1} u_{1}+v_{1}-(\\lambda^{-1}u_{2}+v_{2})}\\geq \\lambda^{-1}\\norm{u_{1}-u_{2}}^2,\n$$\ni.e., $\\ip{u_{1}-u_{2}}{v_{1}-v_{2}}\\geq 0$,\nwhich establishes \\ref{thm:coco env grad:: mono}.\n\\end{comment}\n%The converse direction follows from an identical argument that appeals to (\\ref{eq:sub graphs }), hence is omitted for simplicity.\n\\begin{comment}\n\t{\\color{blue}\n``\\ref{thm:coco env grad:: averaged}$\\Leftrightarrow$\\ref{thm:coco env grad:: comono}'':\nFor \\(0<\\alpha<1\\), define \\(B=2\\alpha\\left[(\\lambda\\partial_p^\\lambda f)^{-1}-\\left((2\\alpha)^{-1}-1\\right)\\Id\\right]\\), \\(N=2J_B-\\Id\\), and \\(R=-(2\\alpha)^{-1}N(2\\alpha\\Id)\\).\nNote that \\(P_\\lambda f=(\\Id+\\lambda\\partial_p^\\lambda f)^{-1}\\).\nThen appealing to \\cite[Lemma 2.4]{luo2024various} yields \\(P_\\lambda f=(1-\\alpha)\\Id+\\alpha R\\).\nAs such, \\ref{thm:coco env grad:: averaged} \\(\\Leftrightarrow\\) \\(R\\) is nonexpansive on \\(U_\\lambda\\)  \\(\\Leftrightarrow\\) \\(N\\) is nonexpansive on \\(U_\\lambda\\)  \\(\\Leftrightarrow\\) \\(J_B\\) is firmly nonexpansive on \\(U_\\lambda\\), which is equivalent to \\(B\\) being (maximally) monotone relative to \\(W_\\lambda\\) in view of Lemma \\ref{thm:local Minty}.\nIn turn for every \\((x_i,u_i)\\in\\gph\\lambda\\partial_p^\\lambda f\\), \\(x_i\\in(\\lambda\\partial_p^\\lambda f)^{-1}(\\lambda^{-1}u_i)\\) and\n\\begin{align*}\n\t2\\alpha\\ip{\\lambda^{-1}(u_1-u_2)}{(x_1-x_2)-\\left((2\\alpha)^{-1}-1\\right)\\lambda^{-1}(u_1-u_2)}\\geq0,\n%\t\\Leftrightarrow\n%\t&\\ip{u_1-u_2}{x_1-x_2}\\geq\\left((2\\alpha)^{-1}-1\\right)\\lambda^{-1}\\norm{u_1-u_2}^2.\n\\end{align*}\nwhich amounts to \\ref{thm:coco env grad:: comono}.\n}\n\\end{comment}\n\\endproof","sha256_utf8":"be824fb146bf19e39c0f15c2e8743df580540b837c2fda9d48bf22c53ab10e99"}]},{"id":"S003","semantic_type":"notation","kind":"notation","title":"欧氏空间与函数记号","source_units":[{"start":25120,"end":25768,"start_line":496,"end_line":504,"raw_tex":"The Euclidean space $\\Rn$ is equipped with the usual scalar product\n$\\ip{x}{y}:=\\sum_{i=1}^{n}\\xi_{i}\\eta_{i}$ for $x=(\\xi_{1},\\ldots, \\xi_{n})$ and $y=(\\eta_{1},\\ldots, \\eta_{n})$, and induced norm\n$\\norm{x}:=\\sqrt{\\ip{x}{x}}$.\nThe set of extended real numbers is $\\overline{\\R} :=(-\\infty, +\\infty]$.\nBy a proper lsc function $f:\\Rn\\rightarrow\\bR$ we mean that $f$ is\nlower semicontinuous on $\\Rn$ and\n$\\dom f:=\\{x\\in\\Rn| \\ f(x)<+\\infty\\}\\neq \\varnothing$. $\\conv f$ and $\\clconv f$ are \\emph{convex hull} and\n\\emph{lsc convex hull} of function $f$, respectively; see \\cite[pages 56-57]{rockafellar_variational_1998}.\nWe let $j :=\\|\\cdot\\|^2/2.$\n","sha256_utf8":"1dbcc9486cdc8b7bd82050021825239d7d4a42853f9f181f29258d36cfbb4546"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"近端映射与包络","source_units":[{"start":25768,"end":26842,"start_line":505,"end_line":520,"raw_tex":"For parameter value $\\lambda>0$, the \\emph{Moreau envelope} function $e_{\\lambda}f$ and \\emph{proximal mapping}\n$P_{\\lambda}f$ of $f$ are defined by\n\\begin{equation}\ne_{\\lambda}f(x) :=\\inf_{y\\in\\Rn}\\left\\{f(y)+\\frac{1}{2\\lambda}\\|y-x\\|^2\\right\\},\\text{ and }\nP_{\\lambda}f(x) :=\\argmin_{y\\in\\Rn}\\left\\{f(y)+\\frac{1}{2\\lambda}\\|y-x\\|^2\\right\\},\n\\end{equation}\nrespectively.\n%The $\\lambda$-proximal hull of $f$ is $h_{\\lambda}f=-e_{\\lambda}[-e_{\\lambda}f]$.\nA function $f:\\Rn\\rightarrow\\bR$ is \\emph{prox-bounded} if there exists $\\lambda>0$ such that\n$e_{\\lambda}f(x)>-\\infty$ for some $x\\in\\Rn$. The supremum of the set of\nall such $\\lambda$ is the threshold $\\lambda_{f}$ of prox-boundedness for\n$f$. The positivity of $\\lambda_{f}$ plays an important role in the paper.\nIf $\\lambda_{f}$ is positive, then $(\\forall \\lambda\\in (0,\\lambda_{f}))$ the proximal mapping\n$P_{\\lambda}f$ is nonempty, compact valued, and outer semicontinuous, and\nthe Moreau envelope\n$e_{\\lambda}f$ is locally Lipschitz; see, e.g., \\cite[Theorem 1.25, Example 10.32]{rockafellar_variational_1998}.\n","sha256_utf8":"40972adee6cb556d7a0c37744aaab7a7feac29c4086cd7369cabb8f85bbf4fbd"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"水平近端次梯度","source_units":[{"start":27395,"end":28161,"start_line":529,"end_line":536,"raw_tex":"Generalized subdifferential constructions from variational analysis\nwill be widely used in the paper. Let $f:\\Rn\\to\\overline{\\R}$ be proper, $x\\in\\dom f$ and let $\\lambda>0$. In \\cite{rock2021} Rockafellar defined level proximal subdifferential.\nA vector $v\\in\\Rn$ is a \\emph{$\\lambda$-level proximal subgradient} of $f$ at $x$, denoted by $v\\in\\levp f(x)$, if\n\t\\begin{equation}\\label{lep ineq}\n\t\t(\\forall y\\in\\Rn)~f(y)\\geq f(x)+\\ip{v}{y-x}-\\frac{1}{2\\lambda}\\norm{y-x}^2.\n\t\\end{equation}\nIt is a special subclass of proximal subgradients. A vector $v\\in\\Rn$ is a \\emph{proximal subgradient} of $f$ at $x$, denoted by $v\\in\\partial_{p}f(x)$, if\n$(\\exists \\lambda>0)(\\exists \\delta>0)(\\forall \\|y-x\\|<\\delta) \\ f(y)\\geq f(x)+\\ip{v}{y-x}-\\frac{1}{2\\lambda}\\|y-x\\|^2.$\n","sha256_utf8":"ff54eac4104619362e7481d480241f18eed2d5fa24f738a4fa73d35df3caed4b"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"广义次微分","source_units":[{"start":28161,"end":29038,"start_line":537,"end_line":550,"raw_tex":"Next is the classical Fenchel subdifferential.\nA vector $v\\in\\Rn$ is a \\emph{Fenchel subgradient} of $f$ at $x$, denoted by $v\\in\\partial_{F}f(x)$, if\n$(\\forall y\\in\\Rn)~f(y)\\geq f(x)+\\ip{v}{y-x}.\n$\nIt is a special class of regular subgradients. A vector $v\\in\\Rn$ is a \\emph{regular subgradient}\nof $f$ at $x$, denoted by\n$v\\in\\hat{\\partial}f(x)$, if $(\\forall y\\in\\Rn)\\ f(y)\\geq f(x)+\\ip{v}{y-x}+o(\\|y-x\\|).$\nIt is a \\emph{Mordukhovich limiting (or general) subgradient}, denoted by $v\\in\\partial f(x)$,\nif $\\exists v^{\\nu}\\in\\hat{\\partial}f(x^{\\nu})$ such that $v^{\\nu}\\rightarrow v$,\n$x^{\\nu}\\rightarrow x$, and $f(x^{\\nu})\\rightarrow f(x)$.\nWhen $x\\not\\in\\dom f$, all subdiffernetials are set to be empty.\n%$\\partial_{F}f(x)=\\varnothing$.\nSee the landmark books \\cite{mordukhovich2006, rockafellar_variational_1998} for\nfurther details and references on subdifferentials.\n","sha256_utf8":"74c8cedba5abab51af1a8093a240411255844a5add9a34ee2954342302a88795"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"近端正则性","source_units":[{"start":29038,"end":29631,"start_line":551,"end_line":558,"raw_tex":"We say that $f$ is \\emph{prox-regular} at $\\bar x$\nfor $\\bar v\\in\\partial f(\\bar x)$ if $(\\exists \\varepsilon>0, \\rho\\geq 0)(\\forall y\\in\\Ball_{\\varepsilon}(\\bar x)) \\\nf(y)\\geq f(x)+\\ip{v}{y-x}-\\dfrac{\\rho}{2}\\norm{y-x}^2$ for\n$v\\in\\partial f(x), v\\in\\Ball_{\\varepsilon}(\\bar v), x\\in\\Ball_{\\varepsilon}(\\bar x),\nf(x)<f(\\bar x)+\\varepsilon$, where $\\Ball_{\\varepsilon}(a)$ is the open ball centered at $a$ with\nradius $\\varepsilon$. If this holds for all $\\bar v\\in\\partial f(\\bar x)$,\n$f$ is said to be prox-regular at\n$\\bar x$; see \\cite{poliquin1996,rockafellar_variational_1998,thibault}.\n","sha256_utf8":"94d580f4bffb3558d33d0b1b3fcaa885b05b5187d4b57130877b9d1307ae8f4b"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"相对单调性与预解式","source_units":[{"start":29632,"end":30528,"start_line":560,"end_line":578,"raw_tex":"Next we recall some notions for set-valued mappings.\nLet $A:\\Rn\\rightrightarrows\\Rn$ be a set-valued mapping and let $W$ be a nonempty subset of $\\Rn\\times\\Rn$.\nThen $A$ is \\emph{monotone relative to $W$} if\n\\(\n(\\forall (x_1,v_1), (x_2,v_2)\\in\\gph A\\cap W)~\n\\ip{x_1-x_2}{v_1-v_2}\\geq0;\n\\)\nand $A$ is \\emph{$\\lambda$-strongly monotone relative to $W$} if\n\\(\n(\\forall (x_1,v_1), (x_2,v_2)\\in\\gph A\\cap W)~\n\\ip{x_1-x_2}{v_1-v_2}\\geq\\lambda\\norm{x_1-x_2}^2,\n\\)\nwhere $\\lambda >0$. $A$ is \\emph{maximally monotone relative to $W$} if there does not exist\n $A':\\Rn\\rightrightarrows\\Rn$ such that $\\gph A'\\cap W\\supseteq\\gph A\\cap W$ but\n$\\gph A'\\cap W \\neq \\gph A\\cap W$ and $A'$ is monotone\nrelative to $W$. See \\cite{nghia2016,rock-vietnam,poliquin1996}\nfor more discussions. The \\emph{resolvent}\n of $A$ is\n$J_{A} :=(\\Id+A)^{-1}$ with $\\Id:\\Rn\\rightarrow\\Rn: x\\mapsto x$ being the identity mapping.\n","sha256_utf8":"3d59c60a6badfd6b602397ad3af9fc600175aca6eb4b6c18ccd05237c467c41a"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"非扩张与余强单调","source_units":[{"start":30528,"end":31239,"start_line":579,"end_line":586,"raw_tex":"Let $U$ be a nonempty subset of $\\Rn$ and let $T:U \\to\\Rn$. Then $T$  is \\emph{firmly nonexpansive}\n if %$(\\forall x,y\\in U)$ $\\norm{Tx-Ty}^2+\\norm{\\left(\\Id-T\\right)x-\\left(\\Id-T %\\right)y}^2\\leq\\norm{x-y}^2$;\n$(\\forall x,y\\in U)\\ \\norm{Tx-Ty}^2\\leq \\ip{x-y}{Tx-Ty}$;\n$T$ is \\emph{$\\lambda$-cocoercive} with $\\lambda>0$ if $\\lambda T$ is firmly nonexpansive;\n$T$ is \\emph{nonexpansive} if $(\\forall x,y\\in U)\\ \\norm{Tx-Ty}\\leq\\norm{x-y}$; and $T$ is \\emph{averaged} if\nthere exist nonexpansive $N:U\\to\\Rn$ and $0\\leq \\lambda<1$ such that $T:=(1-\\lambda)\\Id+\\lambda N$.\nWe use $\\Fix T$ for the set of fixed point of $T$.\nSee \\cite[Chapter 4]{BC} for abundant results and applications of nonexpansive operators.\n","sha256_utf8":"3e0e7bf12e54d712ac3ff69d51ccb98ed5b9735cced7e74fd4a975d2bac8b11a"}]},{"id":"S010","semantic_type":"notation","kind":"notation","title":"映射的图与定义域","source_units":[{"start":31469,"end":31676,"start_line":590,"end_line":593,"raw_tex":"The sets\n$\\dom A :=\\{x\\in\\Rn|\\ A(x)\\neq \\varnothing\\},$ $\\ran A :=\\bigcup_{x\\in\\Rn}Ax=A(\\Rn)$, and $\\gph A\n:=\\{(x,v)\\in\\Rn\\times \\Rn|\\ v\\in Ax\\}$ stand for\nthe domain, range, and graph of $A$, respectively.\n","sha256_utf8":"a2100462c5572391e7014260608a5a605e9669e4496d1c707dc853fbe868d1e6"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"变分凸性","source_units":[{"start":78505,"end":79090,"start_line":1586,"end_line":1591,"raw_tex":"\\begin{defn}\\label{def:vc}\\emph{\\cite[Definition 2]{rock-vietnam}}\nA lsc function $f:\\Rn\\rightarrow\\overline{\\R}$ is called variationally convex at $\\bar x$ for $\\bar v\\in\\partial f(\\bar x)$,\nif for some convex neighborhood $U\\times V$ of $(\\bar x, \\bar v)$ there exist a lsc convex function $\\varphi\\leq f$ on $U$ and a number $\\varepsilon>0$ such that\n$[U_{\\varepsilon}\\times V]\\cap \\gph\\partial f=[U\\times V]\\cap \\gph \\partial \\varphi\\text { and } f(x)=\\varphi(x) \\text{ at the common element $(x,v)$,} $\nwhere $U_{\\varepsilon}:=\\{x\\in U|\\ f(x)< f(\\bar x)+\\varepsilon\\}$.\n\\end{defn}","sha256_utf8":"fd235c00e7dd7301be2496b6651bdceab36d1e8cb90335f19bb93a6de8908c59"}]},{"id":"S012","semantic_type":"derived_fact","kind":"derived_fact","title":"近端映射的预解式表示","source_units":[{"start":34281,"end":35368,"start_line":647,"end_line":668,"raw_tex":"\\begin{fact}\n%[resolvent representation of proximal mapping]\n\\label{fact: prox identity}\n\t\\emph{\\cite[Theorem 3.7]{wang23level}}\n\tLet $f:\\Rn\\to\\overline{\\R}$ be proper, lsc and prox-bounded with threshold\n$\\lambda_f>0$. Then\n\t\\begin{equation}\\label{prox and prox sub}\n\t\t(\\forall 0<\\lambda<\\lambda_f)~P_\\lambda  f =\\big(\\Id+\\lambda\\levp f\\big)^{-1}.\n\t\\end{equation}\nEquivalently,\n$ (\\forall x\\in\\Rn)\\ u\\in P_{\\lambda}f(x)\\Leftrightarrow (u, \\lambda^{-1}(x-u))\\in\\gph \\partial_{p}^{\\lambda}f$.\n\\begin{comment}\n\t\tConsequently, operator $\\left(\\Id+\\lambda\\levp f\\right)^{-1}:\\Rn\\to2^{\\Rn}$ has full domain, is compact-valued, and for every $x\\in\\Rn$\n\t\\begin{align*}\n\t\t\\left(\\Id+\\lambda\\partial_p^\\lambda f\\right)^{-1}(x)=\\big\\{v\\in\\Rn:v_k\\to v,\\text{where }(\\exists x_k\\to x)(\\exists \\lambda_k\\to\\lambda)v_k\\in\\left(\\Id+\\lambda_k\\partial_p^{\\lambda_k} f\\right)^{-1}(x_k)\\big\\}.\n\t\\end{align*}\n\tFurthermore $\\gph\\levp f=T\\left(\\gph P_\\lambda f\\right)$, where\n\t$T=\\begin{bmatrix}\n\t\t0&\\Id\\\\\\Id/\\lambda&-\\Id/\\lambda\n\t\\end{bmatrix}$ is an invertible linear transformation.\n\\end{comment}\n\\end{fact}","sha256_utf8":"1cba6efc7e5bd36ce3473214103b2c25b24ec4ddc27acca061f1b7ee9a538e91"}]},{"id":"S013","semantic_type":"lemma","kind":"lemma","title":"局部明蒂参数化","source_units":[{"start":82845,"end":83758,"start_line":1670,"end_line":1687,"raw_tex":"\\begin{lemma}\\label{l:minty} Let $U$ be a nonempty set in $\\Rn$, let $\\lambda>0$, and let\n$W_{\\lambda} :=\\{(u,v)\\in\\Rn\\times\\Rn| \\ u+\\lambda v\\in U\\}.$\n\\begin{enumerate}\n\\item \\label{i:operator}\nSuppose that $A:\\Rn\\rightrightarrows\\Rn$ and that $J_{\\lambda A}$ is single-valued on $U$.\nThen\n\\begin{equation}\\label{e:mintylocal}\n\\gph A \\cap W_{\\lambda}=\\{(J_{\\lambda A}(x), \\lambda^{-1}[x-J_{\\lambda A}(x)])|\\ x\\in U\\}.\n\\end{equation}\n\\item\\label{i:function}\n Suppose that $f:\\Rn\\rightarrow\\bR$ is proper, lsc and prox-bounded with threshold $\\lambda_{f}$, and $0<\\lambda<\\lambda_{f}$. If $P_{\\lambda}f$ is single-valued on $U$, then\n \\begin{align}\n \\gph \\partial_{p}^{\\lambda}f\\cap W_{\\lambda} &=\\{(P_{\\lambda}f(x), \\lambda^{-1}[x-P_{\\lambda}f(x)])|\\ x\\in U\\}\\label{e:proxf}\\\\\n & =\\{(x-\\lambda \\nabla e_{\\lambda}f(x),\n    \\nabla e_{\\lambda}f(x))|\\ x\\in U\\}.\\label{e:gradf}\n \\end{align}\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"92fdefe40daecf8dd810d6f90275fee08bcd2825211f6ea1014aa6d9a66f4413"}]},{"id":"S014","semantic_type":"proof","kind":"proof","title":"明蒂参数化的证明","source_units":[{"start":83759,"end":84531,"start_line":1688,"end_line":1703,"raw_tex":"\\proof\n\\ref{i:operator}:\nDenote the right-hand side of \\eqref{e:mintylocal} by $E_{\\lambda}$. We have\n\\begin{align*}\n(u,v)\\in E_{\\lambda} & \\Leftrightarrow (\\exists x\\in U)\\ u=J_{\\lambda A}(x), v=\\lambda^{-1}[x-J_{\\lambda A}(x)]\\\\\n& \\Leftrightarrow (\\exists x\\in U)\\ x\\in u+\\lambda Au, x=u+\\lambda v\\\\\n& \\Leftrightarrow (\\exists x\\in U)\\ v\\in Au, x=x+\\lambda v\n\\Leftrightarrow (u,v)\\in \\gph A \\cap W_{\\lambda}.\n\\end{align*}\n\\ref{i:function}:  Apply \\ref{i:operator} with $A=\\partial_{p}^{\\lambda}f$ to obtain\n\\eqref{e:proxf}.\nWhen $0<\\lambda<\\lambda_{f}$ and $P_{\\lambda}f$ is single-valued on $U$,\n\\cite[Example 10.32]{rockafellar_variational_1998} implies\n$(\\forall x\\in U)\\ \\nabla e_{\\lambda}f(x)=\\lambda^{-1}[x-P_{\\lambda}f(x)]$.\nHence \\eqref{e:gradf} holds,\n\\endproof","sha256_utf8":"12944021a4502ba36745a541207faf321b7e51dfbc5e8ead0b92156c447f672c"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"算子图的参数表示","source_units":[{"start":83759,"end":84183,"start_line":1688,"end_line":1696,"raw_tex":"\\proof\n\\ref{i:operator}:\nDenote the right-hand side of \\eqref{e:mintylocal} by $E_{\\lambda}$. We have\n\\begin{align*}\n(u,v)\\in E_{\\lambda} & \\Leftrightarrow (\\exists x\\in U)\\ u=J_{\\lambda A}(x), v=\\lambda^{-1}[x-J_{\\lambda A}(x)]\\\\\n& \\Leftrightarrow (\\exists x\\in U)\\ x\\in u+\\lambda Au, x=u+\\lambda v\\\\\n& \\Leftrightarrow (\\exists x\\in U)\\ v\\in Au, x=x+\\lambda v\n\\Leftrightarrow (u,v)\\in \\gph A \\cap W_{\\lambda}.\n\\end{align*}\n","sha256_utf8":"f175d63ba11071b01b3c89030389cb9ac07690b13ddbe9f77f24d483ea25f4d1"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"函数情形与包络梯度","source_units":[{"start":84183,"end":84531,"start_line":1697,"end_line":1703,"raw_tex":"\\ref{i:function}:  Apply \\ref{i:operator} with $A=\\partial_{p}^{\\lambda}f$ to obtain\n\\eqref{e:proxf}.\nWhen $0<\\lambda<\\lambda_{f}$ and $P_{\\lambda}f$ is single-valued on $U$,\n\\cite[Example 10.32]{rockafellar_variational_1998} implies\n$(\\forall x\\in U)\\ \\nabla e_{\\lambda}f(x)=\\lambda^{-1}[x-P_{\\lambda}f(x)]$.\nHence \\eqref{e:gradf} holds,\n\\endproof","sha256_utf8":"2e3c0e7f949e8146972c20b86215e229afc043d8447816432fa046fe53b3ff83"}]},{"id":"S017","semantic_type":"lemma","kind":"lemma","title":"余强单调与相对单调","source_units":[{"start":84647,"end":85527,"start_line":1707,"end_line":1725,"raw_tex":"\\begin{lemma}\\label{lem:local coco Minty}\n\tLet $A:\\Rn\\rightrightarrows\\Rn$ be a set-valued mapping, let $\\lambda>0$, and\n\tlet $\\varnothing\\neq U\\subseteq\\dom J_{\\lambda A}$.\n\tDefine $W_\\lambda \\coloneqq \\{(u,v)\\in\\Rn\\times\\Rn|\\ u+\\lambda v\\in U\\}$.\n\tThen for \\(\\gamma>0\\) the following are equivalent:\n\\begin{enumerate}\n\\item\\label{i:resolvent:firm}\n\t$J_{\\lambda A}$ is \\(\\gamma\\)-cocoercive on $U$;\n\\item\\label{i:monotone:hypo}\n$A$ is \\(\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\)-monotone relative to $W_\\lambda$.\n\\end{enumerate}\n%\t\\red{\\textbf{(This should be comono because \\(\\gamma-1\\) may be negative.)}}\n%\t\\begin{enumerate}\n%\t\t\\item\\label{thm:local Minty::coco}\n%\t\t$J_{\\lambda A}$ is \\(\\gamma\\)-cocoercive on $U$;\n%\t\t\\item\\label{thm:local Minty::str mono}\n%\t\t$A$ is \\(\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\)-comonotone relative to $W_\\lambda$.\n%\t\\end{enumerate}\n\\end{lemma}","sha256_utf8":"31b3badea210d6929cc49b4f8fd49eac7485288cbff494b79461cf6871cea32d"}]},{"id":"S018","semantic_type":"proof","kind":"proof","title":"余强单调对应的证明","source_units":[{"start":85528,"end":87554,"start_line":1726,"end_line":1769,"raw_tex":"\\proof\n``\\ref{i:resolvent:firm}$\\Rightarrow$\\ref{i:monotone:hypo}'':\n\tTake $(u_{i},v_{i})\\in \\gph A\n\t\\cap W_{\\lambda}$ and set $x_{i} \\coloneqq u_{i}+\\lambda v_{i}$ for $i=1, 2$. Then\n\t$x_{i}\\in U$ and\n\t$(u_{i},v_{i})=(J_{\\lambda A}(x_{i}), \\lambda^{-1}[x_{i}-J_{\\lambda A}(x_{i})])$\n\tby Lemma~\\ref{l:minty}\\ref{i:operator}.\n\tThen the assumption implies that\n\t\\begin{align*}\n\t\t\\ip{u_1-u_2}{v_1-v_2}\n\t\t&=\n\t\t\\tfrac{1}{\\lambda}\\ip{J_{\\lambda A}(x_1)-J_{\\lambda A}(x_2)}{(x_1-x_2)-(J_{\\lambda A}(x_1)-J_{\\lambda A}(x_2))}\\\\\n\t\t&\\geq\n\t\t\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\norm{J_{\\lambda A}(x_1)-J_{\\lambda A}(x_2)}^2\n\t\t=\n\t\t\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\norm{u_1-u_2}^2,\n\t\\end{align*}\n\tas claimed.\n\n``\\ref{i:monotone:hypo}$\\Rightarrow$\\ref{i:resolvent:firm}'':\n\tSuppose that $x\\in U\\subseteq \\dom J_{\\lambda A}$.\n\tThen $x=u+\\lambda v$ for some $(u,v)\\in\\gph A$, consequently $(u,\\lambda^{-1}(x-u))\\in\\gph A\\cap W_\\lambda$ and $u\\in J_{\\lambda A}(x)$.\n\tWe claim that $J_{\\lambda A}$ is single-valued.\n\tIndeed, let $u'\\in J_{\\lambda A}(x)$.\n\tThen the same process as above gives $(u',\\lambda^{-1}(x-u'))\\in\\gph A\\cap W_\\lambda$, and the\n\\(\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\)-monotonicity of\n$\\gph A\\cap W_\\lambda$ implies\n\t\\[\n\t\\ip{u-u'}{\\tfrac{1}{\\lambda}(x-u)-\\tfrac{1}{\\lambda}(x-u')}\n\t\\geq\n\t\\tfrac{1}{\\lambda}(\\gamma-1)\\norm{u-u'}^2\n\t\\Leftrightarrow\n\t\\gamma\\norm{u-u'}^2\\leq 0 \\Leftrightarrow u=u'.\n\t\\]\n\tNext, let $u_i \\coloneqq J_{\\lambda A}(x_i)$ with $x_i\\in U$ for $i=1,2$.\n\tThen there exists \\(v_i\\in A(u_i)\\) such that \\(x_i=u_i+\\lambda v_i\\) and\n\tthe \\(\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\)-monotonicity of $\\gph A\\cap W_\\lambda$ gives\n\t$\n\t\\ip{u_1-u_2}{v_1-v_2}\\geq\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\norm{u_1-u_2}^2\n\t$, i.e.,\n$\\ip{u_1-u_2}{(\\lambda v_1+u_{1})-(\\lambda v_2+u_{2})}\\geq \\gamma\\norm{u_1-u_2}^2$. Thus,\n$\\ip{J_{\\lambda A}(x_{1})-J_{\\lambda A}(x_{2})}{x_{1}-x_{2}}\\geq \\gamma\\norm{J_{\\lambda A}(x_{1})-J_{\\lambda A}(x_{2})}^2$\n and this is precisely \\ref{i:resolvent:firm}.\n\\endproof","sha256_utf8":"33763399a96ca3f6eaadf757bcc92cbaf78f534d3821040e690282f8c8a0319a"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"参数化推出相对单调","source_units":[{"start":85528,"end":86234,"start_line":1726,"end_line":1743,"raw_tex":"\\proof\n``\\ref{i:resolvent:firm}$\\Rightarrow$\\ref{i:monotone:hypo}'':\n\tTake $(u_{i},v_{i})\\in \\gph A\n\t\\cap W_{\\lambda}$ and set $x_{i} \\coloneqq u_{i}+\\lambda v_{i}$ for $i=1, 2$. Then\n\t$x_{i}\\in U$ and\n\t$(u_{i},v_{i})=(J_{\\lambda A}(x_{i}), \\lambda^{-1}[x_{i}-J_{\\lambda A}(x_{i})])$\n\tby Lemma~\\ref{l:minty}\\ref{i:operator}.\n\tThen the assumption implies that\n\t\\begin{align*}\n\t\t\\ip{u_1-u_2}{v_1-v_2}\n\t\t&=\n\t\t\\tfrac{1}{\\lambda}\\ip{J_{\\lambda A}(x_1)-J_{\\lambda A}(x_2)}{(x_1-x_2)-(J_{\\lambda A}(x_1)-J_{\\lambda A}(x_2))}\\\\\n\t\t&\\geq\n\t\t\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\norm{J_{\\lambda A}(x_1)-J_{\\lambda A}(x_2)}^2\n\t\t=\n\t\t\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\norm{u_1-u_2}^2,\n\t\\end{align*}\n\tas claimed.\n","sha256_utf8":"6d72de245ad6b855e4cd3a85681a342265dee982651802e84141f466921f7aea"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"反向证明预解式单值","source_units":[{"start":86235,"end":86946,"start_line":1745,"end_line":1759,"raw_tex":"``\\ref{i:monotone:hypo}$\\Rightarrow$\\ref{i:resolvent:firm}'':\n\tSuppose that $x\\in U\\subseteq \\dom J_{\\lambda A}$.\n\tThen $x=u+\\lambda v$ for some $(u,v)\\in\\gph A$, consequently $(u,\\lambda^{-1}(x-u))\\in\\gph A\\cap W_\\lambda$ and $u\\in J_{\\lambda A}(x)$.\n\tWe claim that $J_{\\lambda A}$ is single-valued.\n\tIndeed, let $u'\\in J_{\\lambda A}(x)$.\n\tThen the same process as above gives $(u',\\lambda^{-1}(x-u'))\\in\\gph A\\cap W_\\lambda$, and the\n\\(\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\)-monotonicity of\n$\\gph A\\cap W_\\lambda$ implies\n\t\\[\n\t\\ip{u-u'}{\\tfrac{1}{\\lambda}(x-u)-\\tfrac{1}{\\lambda}(x-u')}\n\t\\geq\n\t\\tfrac{1}{\\lambda}(\\gamma-1)\\norm{u-u'}^2\n\t\\Leftrightarrow\n\t\\gamma\\norm{u-u'}^2\\leq 0 \\Leftrightarrow u=u'.\n\t\\]\n","sha256_utf8":"12bacf4501b1889956ac86692c314b8c544de834eb212a50096044ad973c6f79"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"反向推出余强单调","source_units":[{"start":86946,"end":87554,"start_line":1760,"end_line":1769,"raw_tex":"\tNext, let $u_i \\coloneqq J_{\\lambda A}(x_i)$ with $x_i\\in U$ for $i=1,2$.\n\tThen there exists \\(v_i\\in A(u_i)\\) such that \\(x_i=u_i+\\lambda v_i\\) and\n\tthe \\(\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\)-monotonicity of $\\gph A\\cap W_\\lambda$ gives\n\t$\n\t\\ip{u_1-u_2}{v_1-v_2}\\geq\\tfrac{1}{\\lambda}\\left(\\gamma-1\\right)\\norm{u_1-u_2}^2\n\t$, i.e.,\n$\\ip{u_1-u_2}{(\\lambda v_1+u_{1})-(\\lambda v_2+u_{2})}\\geq \\gamma\\norm{u_1-u_2}^2$. Thus,\n$\\ip{J_{\\lambda A}(x_{1})-J_{\\lambda A}(x_{2})}{x_{1}-x_{2}}\\geq \\gamma\\norm{J_{\\lambda A}(x_{1})-J_{\\lambda A}(x_{2})}^2$\n and this is precisely \\ref{i:resolvent:firm}.\n\\endproof","sha256_utf8":"19be6a6edd1135ce34c87f908c48090330f06142d5e9cea7aabbd411d5734f16"}]},{"id":"S022","semantic_type":"lemma","kind":"lemma","title":"局部明蒂等价关系","source_units":[{"start":87702,"end":88272,"start_line":1774,"end_line":1787,"raw_tex":"\\begin{lemma}\\label{thm:local Minty}\nLet $A:\\Rn\\rightrightarrows\\Rn$ be a set-valued mapping, let $\\lambda>0$, and\nlet $\\varnothing\\neq U\\subseteq\\dom J_{\\lambda A}$.\nDefine $W_\\lambda :=\\{(u,v)\\in\\Rn\\times\\Rn|\\ u+\\lambda v\\in U\\}$.\nThen the following are equivalent:\n\\begin{enumerate}\n\t\\item\\label{thm:local Minty::fnexp}\n\t $J_{\\lambda A}$ is firmly nonexpansive on $U$;\n\t\\item\\label{thm:local Minty::mono}\n\t$A$ is monotone relative to $W_\\lambda$;\n\t\\item\\label{thm:local Minty::max mono}\n\t $A$ is maximally monotone relative to $W_\\lambda$.\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"28b73a277042a721482b711610b471948aff3f3eba6f5c11697fc2419bde04b9"}]},{"id":"S023","semantic_type":"proof","kind":"proof","title":"局部明蒂等价的证明","source_units":[{"start":88273,"end":91447,"start_line":1788,"end_line":1880,"raw_tex":"\\proof\nObserve that\n$\n\t[(x,u)\\in\\gph J_{\\lambda A} \\text{ and } x \\in U]\n\t\\Leftrightarrow\n\t\\left(u,\\lambda^{-1}(x-u)\\right)\\in\\gph A\\cap W_\\lambda.\n$\n%Indeed\n%$\n%\t(z,x)\\in\\gph J_{\\lambda A}, z\\in U\n%\t\\Leftrightarrow\n%\tx\\in J_{\\lambda A}(z),z\\in U\n%\t\\Leftrightarrow\n%\t(z-x)/\\lambda\\in Ax,z\\in U\n%$.\n\n``\\ref{thm:local Minty::fnexp}$\\Leftrightarrow$\n\\ref{thm:local Minty::mono}'': Apply Lemma~\\ref{lem:local coco Minty} with $\\gamma=1$.\n %Take $(u_{i},v_{i})\\in \\gph A\n%\\cap W_{\\lambda}$ and set $x_{i} :=u_{i}+\\lambda v_{i}$ for $i=1, 2$. Then\n%$x_{i}\\in U$ and\n%$(u_{i},v_{i})=(J_{\\lambda A}(x_{i}), \\lambda^{-1}[x_{i}-J_{\\lambda A}(x_{i})])$\n%by Lemma~\\ref{l:minty}\\ref{i:operator}.\n%Using \\ref{thm:local Minty::fnexp}, we derive\n%\\begin{align*}\n%\\ip{u_{1}-u_{2}}{v_{1}-v_{2}} &= \\ip{J_{\\lambda A}(x_{1})-J_{\\lambda A}(x_{2})}{\\lambda^{-1}[x_{1}-J_{\\lambda A}(x_{1})]-\\lambda^{-1}[x_{2}-J_{\\lambda A}(x_{2})]}\\\\\n%&=\\lambda^{-1}\\bigg(\\ip{J_{\\lambda A}(x_{1})-J_{\\lambda A}(x_{2})}{x_{1}-x_{2}}-\\norm{J_{\\lambda A}(x_{1})-J_{\\lambda A}(x_{2})}^2\\bigg)\\geq 0.\n%\\end{align*}\n%%Denote by $x_i=J_{\\lambda A}(z_i)$ for $z_i\\in U$ and $i=1,2$.\n%%We have\n%%\\[\n%%\\ip{x_1-x_2}{z_1-z_2}\\geq\\|x_1-x_2\\|^2.\n%%\\]\n%%Dividing by $\\lambda>0$ gives\n%%\\[\n%%\\ip{x_1-x_2}\n%%{\n%%\t\\frac{z_1-x_1}{\\lambda}-\\frac{z_2-x_2}{\\lambda}\n%%}\n%%\\geq0.\n%%\\]\n%Hence $A$ is monotone relative to $W_\\lambda$.\n\n``\\ref{thm:local Minty::mono}$\\Rightarrow$\\ref{thm:local Minty::max mono}\":\nTake $(u,v)\\in W_\\lambda$ that is monotonically related to $\\gph A\\cap W_\\lambda$.\nWe claim that $(u,v)\\in\\gph A$, which entails the desired maximal monotonicity.\nIndeed, $(u,v)\\in W_\\lambda$ implies $u+\\lambda v\\in U\\subseteq \\dom J_{\\lambda A}$.\nLet $y\\in J_{\\lambda A}(u+\\lambda v)$.\nThen\n$\n\\left(y,\\lambda^{-1}(u+\\lambda v-y)\\right)\\in\\gph A\\cap W_\\lambda.\n$\nBy the assumption that $(u,v)$ is monotonically related to $\\gph A\\cap W_\\lambda$, we obtain\n\\begin{align*}\n\\ip{u-y}{\\lambda^{-1}(y-u)}\n=\n\\ip{u-y}{v-\\lambda^{-1}(u+\\lambda v-y)}\\geq 0\n\\Rightarrow\nu=y,\n\\end{align*}\nenforcing that $u\\in J_{\\lambda A}(u+\\lambda v)$.\nIn turn, $u+\\lambda v\\in u+\\lambda Au \\Leftrightarrow v\\in Au$.\n\n``\\ref{thm:local Minty::max mono}$\\Rightarrow$\\ref{thm:local Minty::mono}\":\nClear.\n%Suppose that $x\\in U\\subseteq \\dom J_{\\lambda A}$.\n%Then $x=u+\\lambda v$ for some $(u,v)\\in\\gph A$, consequently $(u,\\lambda^{-1}(x-u))\\in\\gph A\\cap W_\\lambda$ and $u\\in J_{\\lambda A}(x)$.\n%We claim that $J_{\\lambda A}$ is single-valued.\n%Indeed, let $u'\\in J_{\\lambda A}(x)$.\n%Then the same process as above implies $(u',\\lambda^{-1}(x-u'))\\in\\gph A\\cap W_\\lambda$.\n%Appealing to the monotonicity of $\\gph A\\cap W_\\lambda$ yields\n%\\[\n%\\ip\n%{\n%u-u'\n%}\n%{\n%\\lambda^{-1}(x-u)-\\lambda^{-1}(x-u')\n%}\n%\\geq 0\n%\\Leftrightarrow\n%\\ip{u-u'}{-\\lambda^{-1}(u-u')}\\geq 0 \\Leftrightarrow u=u'.\n%\\]\n%Next, let $u_i :=J_{\\lambda A}(x_i)$ with $x_i\\in U$ for $i=1,2$.\n%Then the monotonicity of $\\gph A\\cap W_\\lambda$ again gives\n%$$\n%\\ip{u_1-u_2}{\\lambda^{-1}(x_1-u_1)-\\lambda^{-1}(x_2-u_2)}\\geq0\n%%\\Leftrightarrow\\ip{x_1-x_2}{(z_1-z_2)-(x_1-x_2)}\\geq0\n%\\Leftrightarrow\n%\\ip{u_1-u_2}{x_1-x_2}\\geq\\|u_1-u_2\\|^2,\n%$$\n%justifying \\ref{thm:local Minty::fnexp}.\n\\endproof","sha256_utf8":"7cf0e77e74861078be8e2bc1f8c303f922beba14ba4a2705a2968fe7a74f2440"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"预解式图的对应","source_units":[{"start":88273,"end":88423,"start_line":1788,"end_line":1794,"raw_tex":"\\proof\nObserve that\n$\n\t[(x,u)\\in\\gph J_{\\lambda A} \\text{ and } x \\in U]\n\t\\Leftrightarrow\n\t\\left(u,\\lambda^{-1}(x-u)\\right)\\in\\gph A\\cap W_\\lambda.\n$\n","sha256_utf8":"cd378833f442680ee645b3abbf1ca1f6c601f835025bb3e8c3e3e34c7b47fd42"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"坚固非扩张等价单调","source_units":[{"start":88572,"end":88707,"start_line":1804,"end_line":1805,"raw_tex":"``\\ref{thm:local Minty::fnexp}$\\Leftrightarrow$\n\\ref{thm:local Minty::mono}'': Apply Lemma~\\ref{lem:local coco Minty} with $\\gamma=1$.\n","sha256_utf8":"66e3b80c06a9970dbb42a2e28e7ef4d78c39b4ad52181ccef5c50b15dcbb8011"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"单调性推出极大性","source_units":[{"start":89645,"end":90412,"start_line":1831,"end_line":1849,"raw_tex":"``\\ref{thm:local Minty::mono}$\\Rightarrow$\\ref{thm:local Minty::max mono}\":\nTake $(u,v)\\in W_\\lambda$ that is monotonically related to $\\gph A\\cap W_\\lambda$.\nWe claim that $(u,v)\\in\\gph A$, which entails the desired maximal monotonicity.\nIndeed, $(u,v)\\in W_\\lambda$ implies $u+\\lambda v\\in U\\subseteq \\dom J_{\\lambda A}$.\nLet $y\\in J_{\\lambda A}(u+\\lambda v)$.\nThen\n$\n\\left(y,\\lambda^{-1}(u+\\lambda v-y)\\right)\\in\\gph A\\cap W_\\lambda.\n$\nBy the assumption that $(u,v)$ is monotonically related to $\\gph A\\cap W_\\lambda$, we obtain\n\\begin{align*}\n\\ip{u-y}{\\lambda^{-1}(y-u)}\n=\n\\ip{u-y}{v-\\lambda^{-1}(u+\\lambda v-y)}\\geq 0\n\\Rightarrow\nu=y,\n\\end{align*}\nenforcing that $u\\in J_{\\lambda A}(u+\\lambda v)$.\nIn turn, $u+\\lambda v\\in u+\\lambda Au \\Leftrightarrow v\\in Au$.\n","sha256_utf8":"fb943c78178548de02b00c3e4d6890cd4355d751bdf95d1d77c522304f7a7156"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"极大单调蕴含单调","source_units":[{"start":90413,"end":90496,"start_line":1851,"end_line":1852,"raw_tex":"``\\ref{thm:local Minty::max mono}$\\Rightarrow$\\ref{thm:local Minty::mono}\":\nClear.\n","sha256_utf8":"ee5093b4a98421c5b868ba103d98ac9c9ae5beac8156e64986e98cedc2d08cf6"},{"start":91438,"end":91447,"start_line":1880,"end_line":1880,"raw_tex":"\\endproof","sha256_utf8":"5215cdf1ce8da117cd22d563c101d4e1791a2afc866951dba5dbecc236b99578"}]},{"id":"S028","semantic_type":"assumption","kind":"assumption","title":"主定理的函数假设","source_units":[{"start":91796,"end":91990,"start_line":1889,"end_line":1891,"raw_tex":"Suppose that $f:\\mathbb{R}^n\\to\\overline{\\mathbb{R}}$ is proper, lsc and prox-bounded\nwith threshold $\\lambda_{f}>0$,\nand that $f$ is prox-regular at $\\bar x$ for $\\bar v\\in\\partial f(\\bar x)$.\n","sha256_utf8":"8fc6730c35428afe4a7b7fa25f2e8b6898aa70012c33a47941c58a485be09efe"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"构造局部近端映射","source_units":[{"start":94341,"end":95108,"start_line":1939,"end_line":1952,"raw_tex":"\\proof\nBy\n\\cite[Proposition 5.3]{thibault},\nthere exists \\(0<\\bar\\lambda\\leq \\lambda_{f}\\) such that to every \\(0<\\lambda<\\bar\\lambda\\) corresponds an open convex neighborhood $U_\\lambda$ of $\\bar x+\\lambda\\bar v$ on which:\n\\begin{equation}\\label{e:grad1}\n\\text{$P_{\\lambda}f$ is monotone, single-valued and Lipschitz continuous; $P_{\\lambda}f(\\bar x+\\lambda{\\bar v})=\\bar x$;}\n\\end{equation}\nand $e_\\lambda f$ is differentiable with\n\\begin{equation}\\label{e:grad2}\n\\nabla e_{\\lambda}f=\\lambda^{-1}[\\Id-P_\\lambda f].\n\\end{equation}\nSee also \\cite[Proposition 13.37]{rockafellar_variational_1998} (for $\\bar v=0$).\nIn particular, \\eqref{e:grad1} implies $\\bar{v}\\in\\partial_{p}^{\\lambda}f(\\bar{x})\\subseteq\\hat{\\partial}f(\\bar x)$;\nsee Fact \\ref{fact: prox identity}.\n","sha256_utf8":"a1898ff0f8ff3e4b53795d828c434e3340da50ea2a4da5dd7c6785796ff0c3ff"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"变分凸给出局部单调","source_units":[{"start":95632,"end":96136,"start_line":1967,"end_line":1974,"raw_tex":"``\\ref{thm:coco env grad:: v cvx}$\\Rightarrow$\\ref{thm:coco env grad:: mono}'':\nSince $f$ is variationally convex at $\\bar x$ for $\\bar v\\in\\hat{\\partial}f(\\bar{x})$,\nthere exist a convex neighborhood \\(U\\times V\\) of \\((\\bar x,\\bar v)\\) and \\(\\varepsilon>0\\) such that\n\\begin{equation}\\label{e:subf:monotone}\n\t\\gph\\partial f\\cap\\left[U_\\varepsilon\\times V\\right]\n\t\\text{ is monotone},\n\\end{equation}\nwhere \\(U_\\varepsilon :=\\{x\\in U|\\ f(x)<f(\\bar x)+\\varepsilon\\}\\); see \\cite[Theorem 1]{rock-vietnam}.\n","sha256_utf8":"09abcc5ffc08e05e9674aa2a771ad88a0429957b08351a52857acdfe41f0e921"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"验证图点与函数值极限","source_units":[{"start":96136,"end":96973,"start_line":1975,"end_line":1989,"raw_tex":"For every \\(0<\\lambda<\\bar\\lambda\\), \\eqref{e:grad1} and \\eqref{e:grad2} entail that\n$(\\forall u\\in U_{\\lambda})\\ u-\\lambda\\nabla e_\\lambda f(u)=P_{\\lambda}f(u)$,\n$P_{\\lambda}f$ is Lipschitz, and\n\\(\\nabla e_\\lambda f(\\bar x+\\lambda \\bar v)=\\bar v\\).\nHence,\n\\begin{equation}\\label{eq:limits}\n\\lim_{u\\to\\bar x+\\lambda\\bar v}\\nabla e_\\lambda f(u)=\\bar v,\n\\lim_{u\\to\\bar x+\\lambda\\bar v}(\\Id-\\lambda\\nabla e_\\lambda f)(u)=\\bar x,\n\\text{ and }\\lim_{u\\to\\bar x+\\lambda\\bar v}f(P_{\\lambda}f(u))=f(\\bar x),\n\\end{equation}\nwhere the last limit holds because\n\\begin{align}\n\\lim_{u\\to\\bar x+\\lambda\\bar v}\tf(P_{\\lambda}f(u)) & =\\lim_{u\\to\\bar x+\\lambda\\bar v }\\Big(e_\\lambda f(u)-\\tfrac{1}{2\\lambda}\\norm{u-P_{\\lambda}f(u)}^2\\Big)\\\\\n&=e_\\lambda f(\\bar x+\\lambda\\bar v)-\\tfrac{1}{2\\lambda}\\norm{\\bar x+\\lambda\\bar v-\\bar x}^2=f(\\bar x).\n\\end{align}\n","sha256_utf8":"5d73a0ceff7efa8810fc9c636f452d47ff940745ee02651fb08951318546849f"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"转移到水平次微分图","source_units":[{"start":96973,"end":97961,"start_line":1990,"end_line":2008,"raw_tex":"Owing to \\eqref{eq:limits}, by using a smaller $U_{\\lambda}$ if necessary, we can assume\nthat for \\(u\\in U_\\lambda\\)\n\\begin{equation}\\label{e:f:attentative}\n\tu-\\lambda\\nabla e_\\lambda f(u)\\in U, \\nabla e_\\lambda f(u)\\in V,\n\\text{ and } f(u-\\lambda \\nabla e_\\lambda f(u))<f(\\bar x)+\\varepsilon.\n\\end{equation}\nMoreover,\n\\(u-\\lambda\\nabla e_\\lambda f(u)=P_\\lambda f(u)\\) implies, by Fact~\\ref{fact: prox identity}, that\n\\begin{equation}\\label{e:in:graph}\n(u-\\lambda\\nabla e_\\lambda f(u),\\nabla e_\\lambda f(u))\\in\n\\gra \\partial_{p}^{\\lambda}f\\subseteq \\gra\\partial f.\n\\end{equation}\n%\\(\\nabla e_\\lambda f(u)\\in\\partial f(u-\\lambda\\nabla e_\\lambda f(u))\\).\nCombining \\eqref{e:f:attentative}, \\eqref{e:in:graph} and \\eqref{e:subf:monotone} we obtain\n\\[\n\t\\{(u-\\lambda\\nabla e_\\lambda f(u), \\nabla e_\\lambda f(u))|\\ u\\in U_\\lambda\\}\\subseteq\\gph\\partial f\\cap\\left[U_\\varepsilon\\times V\\right]\n\t\\text{is monotone},\n\\]\nand so is \\(\\gph\\partial_p^\\lambda f\\cap W_\\lambda\\) by Lemma~\\ref{l:minty}.\n","sha256_utf8":"4af0c57a7189b2fdcfaf0c6d433f8b25c374e0d9dff7163cd3723cdc34c6c3d5"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"应用局部明蒂等价","source_units":[{"start":97962,"end":98241,"start_line":2010,"end_line":2012,"raw_tex":"``\\ref{thm:coco env grad:: mono}$\\Leftrightarrow$\\ref{thmn:coco env grad:: max mono}$\\Leftrightarrow$\\ref{thm:coco env grad:: firmly nonexap prox}'':\nNote that \\(P_\\lambda f=(\\Id+\\lambda\\partial_p^\\lambda f)^{-1}\\).\nThen applying Lemma \\ref{thm:local Minty} completes the proof.\n","sha256_utf8":"189c028cbc2335644ce18dc80af5a93bb0a84d68909f8fff72fbfeafcae277bb"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"推出平均性与非扩张性","source_units":[{"start":102938,"end":103083,"start_line":2128,"end_line":2128,"raw_tex":"``\\ref{thm:coco env grad:: firmly nonexap prox}$\\Rightarrow$\\ref{thm:coco env grad:: averaged}$\\Rightarrow$\\ref{thm:coco env grad:: ne}\": Clear.\n","sha256_utf8":"8e5cfdf9b7171479cf8a9685c220a0881a93acbf064b368c75a45cc7503d2f99"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"非扩张性推出包络凸性","source_units":[{"start":103084,"end":103907,"start_line":2130,"end_line":2141,"raw_tex":"``\\ref{thm:coco env grad:: ne}$\\Rightarrow$\\ref{envel:convex}\": The assumption implies that $\\nabla e_{\\lambda}f$ is monotone on $U_{\\lambda}$, so $e_{\\lambda}f$ is convex on $U_{\\lambda}$\n{in view of \\cite[B Theorem 4.1.4]{hiriart2004fundamentals}}.\nIndeed, by the nonexpansiveness of $P_\\lambda f$ on $U_\\lambda$ and the Cauchy-Schwarz inequality,\n\\begin{equation}\\label{e:nonexp:grad}\n(\\forall x_{1}, x_{2}\\in U_\\lambda)~\n\\ip{P_\\lambda f(x_{1})-P_\\lambda f(x_{2})}{x_{1}-x_{2}}\\leq\\norm{x_{1}-x_{2}}^2.\n\\end{equation}\nSince\n$\\nabla e_{\\lambda}f=\\lambda^{-1}[\\Id-P_{\\lambda}f] \\text{ on $U_{\\lambda}$},$\nit follows from \\eqref{e:nonexp:grad} that\n$$(\\forall x_{1},x_{2}\\in U_\\lambda)~\\ip{x_{1}-x_{2}}{\\nabla e_\\lambda f(x_{1})-\\nabla e_\\lambda f(x_{2})}\\geq 0,$$\nhence $\\nabla e_{\\lambda}f$ is monotone on $U_{\\lambda}$.\n","sha256_utf8":"06189b92aad19b6c52a9b6bb318100cd9c69a5666e28605333561c44bf98b2f7"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"构造局部次微分算子","source_units":[{"start":103908,"end":104799,"start_line":2143,"end_line":2157,"raw_tex":"``\\ref{envel:convex}$\\Rightarrow$\\ref{envel:convex spec}$\\Rightarrow$\\ref{thm:coco env grad:: v cvx}'':\nThe first implication is trivial.\nWe prove the second following the same line of proof in \\cite[Theorem 3.2]{Khanh2023variational} for the sake of completeness.\nBy the prox-regularity of $f$ at $\\bar{x}$ for $\\bar{v}\\in\\partial f(\\bar{x})$,\nthere exist \\(\\gamma>0\\) and \\(T:\\Rn\\rightrightarrows\\Rn\\) defined by\n\\begin{align*}\n\tT(x):=\n\t\\begin{cases}\n\t\t\\{v\\in\\partial f(x)|\\norm{v-\\bar v}<\\gamma\\},&\\text{if }\\norm{x-\\bar x}<\\gamma, f(x)<f(\\bar x)+\\gamma;\\\\\n\t\t\\varnothing, &\\text{otherwise,}\n\t\\end{cases}\n\\end{align*}\nsuch that \\(\\nabla e_\\lambda f=\\lambda^{-1}[\\Id-P_{\\lambda}f]=\n\\left(\\lambda\\Id+T^{-1}\\right)^{-1}\\) on an open convex neighborhood \\(U_\\lambda\\) of $\\bar{x}+\\lambda\\bar{v}$; see\n\\cite[Proposition 13.37]{rockafellar_variational_1998} or \\cite[Proposition 4.4]{thibault}.\n","sha256_utf8":"0e1629bebed776fbef987968f2f5be3dbdb158dcd42e589f850d8860a2fd7649"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"选择邻域并识别梯度","source_units":[{"start":104799,"end":105723,"start_line":2158,"end_line":2175,"raw_tex":"By taking a smaller $U_{\\lambda}$ if necessary, assumption \\ref{envel:convex spec} implies that\n\\begin{equation}\\label{eq:cvx env}\n\t(\\forall x,y\\in U_\\lambda)~\n\te_\\lambda f(y)\\geq e_\\lambda f(x)+\\ip{\\nabla e_\\lambda f(x)}{y-x}.\n\\end{equation}\nTake \\(\\varepsilon>0\\) sufficiently small so that \\(\\varepsilon<\\gamma\\) and\n\\begin{equation}\\label{e:uv}\n(\\forall x\\in U)(\\forall v\\in V)\\ x+\\lambda v\\in U_\\lambda,\n\\end{equation}\nwhere \\(U:=\\Ball_{\\varepsilon}(\\bar x)\\) and \\(V:=\\Ball_{\\varepsilon}(\\bar v)\\).\nThen \\(v\\in T(x)\\) for \\((x,v)\\in\\gph\\partial f\\cap[U_\\varepsilon\\times V]\\) with\n\\(U_\\varepsilon:=\\{x\\in U|\\ f(x)<f(\\bar x)+\\varepsilon\\}\\), implying that\n\\begin{align}\n\t\\nabla e_\\lambda f(x+\\lambda v) &=\\left(\\lambda\\Id+T^{-1}\\right)^{-1}(x+\\lambda v)=v, \\text{ and }\n\t\\label{eq:env and prox}\\\\\n\tP_\\lambda f(x+\\lambda v) &=(x+\\lambda v)-\\lambda\\nabla e_{\\lambda}f(x+\\lambda v)=x.\n\\label{eq:env and prox1}\n\\end{align}\n","sha256_utf8":"96ca39e94243d8d39db6563e7d8c465c6f0829f6130ca441c945a7352f98bc73"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"包络凸性推出变分凸性","source_units":[{"start":105723,"end":106465,"start_line":2176,"end_line":2189,"raw_tex":"Therefore, for every \\(y\\in U\\) and \\((x,v)\\in\\gph\\partial f\\cap[U_\\varepsilon\\times V]\\), we have \\(y+\\lambda v, x+\\lambda v\\in U_\\lambda\\) by \\eqref{e:uv} so that\n\\begin{align*}\n\tf(y)\n&\\geq\n\te_\\lambda f(y+\\lambda v)-\\tfrac{1}{2\\lambda}\\norm{y+\\lambda v-y}^2\\\\\n&\\geq\n\te_\\lambda f(x+\\lambda v)+\\ip{\\nabla e_\\lambda f(x+\\lambda v)}{y-x}-\\tfrac{1}{2\\lambda}\\norm{x+\\lambda v-x}^2\\\\\n&=e_\\lambda f(x+\\lambda v)-\\tfrac{1}{2\\lambda}\\norm{x+\\lambda v-x}^2+\\ip{v}{y-x}\\\\\n&=\n\tf(x)+\\ip{v}{y-x},\n\\end{align*}\nwhere the second inequality owes to \\eqref{eq:cvx env}, the first equality by \\eqref{eq:env and prox}, and the second\nequality by \\eqref{eq:env and prox1}.\nAppealing to \\cite[Theorem 1(b)$\\Leftrightarrow$(c)]{rock-vietnam} completes the proof.\n","sha256_utf8":"92d2cbd22b7324190ab55d53982357896118393a5070ecf8a05c488af53e7d81"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"包络梯度的余强单调","source_units":[{"start":106466,"end":106776,"start_line":2191,"end_line":2192,"raw_tex":"``\\ref{thm:coco env grad:: firmly nonexap prox}$\\Leftrightarrow$\\ref{thm:coco env grad:: coco}'':\nObserve that \\ref{thm:coco env grad:: coco} amounts to \\(\\Id-P_\\lambda f=\\lambda\\nabla e_\\lambda f\\) is firmly nonexpansive on \\(U_\\lambda\\), which is equivalent to \\ref{thm:coco env grad:: firmly nonexap prox}.\n","sha256_utf8":"85b9722d4c9541b3e54b7ffdf6ab107b430305eedb93092b52e634068780bbc7"},{"start":109970,"end":109979,"start_line":2241,"end_line":2241,"raw_tex":"\\endproof","sha256_utf8":"5215cdf1ce8da117cd22d563c101d4e1791a2afc866951dba5dbecc236b99578"}]},{"id":"S040","semantic_type":"external_fact","kind":"external_fact","title":"已有预解式恒等式","source_units":[],"declared_external":true,"declaration_text":["wang23level, Theorem 3.7: for proper lsc prox-bounded f and 0<lambda<lambda_f, P_lambda f=(Id+lambda partial_p^lambda f)^{-1}; equivalently, u belongs to P_lambda f(x) exactly when (u,(x-u)/lambda) belongs to the graph of partial_p^lambda f."]},{"id":"S041","semantic_type":"external_fact","kind":"external_fact","title":"近端正则的局部表示","source_units":[],"declared_external":true,"declaration_text":["thibault, Propositions 5.3 and 4.4, and rockafellar_variational_1998, Proposition 13.37: under the stated prox-regularity and prox-boundedness assumptions, sufficiently small lambda admit neighborhoods of bar x+lambda bar v where P_lambda f is monotone, single-valued and Lipschitz, P_lambda f(bar x+lambda bar v)=bar x, and the differentiable envelope satisfies grad e_lambda f=lambda^{-1}(Id-P_lambda f)=(lambda Id+T^{-1})^{-1}, with T an f-attentive localization of partial f."]},{"id":"S042","semantic_type":"external_fact","kind":"external_fact","title":"包络梯度公式","source_units":[],"declared_external":true,"declaration_text":["rockafellar_variational_1998, Example 10.32: under the prox-boundedness and parameter conditions used here, single-valuedness of P_lambda f gives grad e_lambda f(x)=lambda^{-1}(x-P_lambda f(x)) at the points in question."]},{"id":"S043","semantic_type":"external_fact","kind":"external_fact","title":"变分凸性的已有判据","source_units":[],"declared_external":true,"declaration_text":["rock-vietnam, Theorem 1: for the regular subgradient base point used in the proof, variational convexity is equivalent to monotonicity of partial f restricted to a suitable f-attentive neighborhood U_epsilon times V, and equivalently to f(y)>=f(x)+<v,y-x> for y in U and (x,v) in gra partial f intersected with U_epsilon times V."]},{"id":"S044","semantic_type":"external_fact","kind":"external_fact","title":"单调梯度与凸性","source_units":[],"declared_external":true,"declaration_text":["hiriart2004fundamentals, B Theorem 4.1.4: a differentiable real-valued function on an open convex set is convex if and only if its gradient is monotone there."]},{"id":"S045","semantic_type":"external_fact","kind":"external_fact","title":"坚固非扩张的基本性质","source_units":[],"declared_external":true,"declaration_text":["On a common domain, T is firmly nonexpansive if and only if Id-T is firmly nonexpansive. A firmly nonexpansive mapping is one-half averaged, and an averaged mapping is nonexpansive."]},{"id":"S046","semantic_type":"external_fact","kind":"external_fact","title":"柯西施瓦茨不等式","source_units":[],"declared_external":true,"declaration_text":["For Euclidean vectors a and b, <a,b><=||a|| ||b||."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S031","kind":"supports"},{"source_id":"S004","target_id":"S012","kind":"supports"},{"source_id":"S004","target_id":"S031","kind":"supports"},{"source_id":"S004","target_id":"S038","kind":"supports"},{"source_id":"S005","target_id":"S012","kind":"supports"},{"source_id":"S005","target_id":"S032","kind":"supports"},{"source_id":"S006","target_id":"S029","kind":"supports"},{"source_id":"S006","target_id":"S032","kind":"supports"},{"source_id":"S006","target_id":"S036","kind":"supports"},{"source_id":"S007","target_id":"S029","kind":"supports"},{"source_id":"S007","target_id":"S036","kind":"supports"},{"source_id":"S008","target_id":"S015","kind":"supports"},{"source_id":"S008","target_id":"S019","kind":"supports"},{"source_id":"S008","target_id":"S020","kind":"supports"},{"source_id":"S008","target_id":"S021","kind":"supports"},{"source_id":"S008","target_id":"S024","kind":"supports"},{"source_id":"S008","target_id":"S026","kind":"supports"},{"source_id":"S008","target_id":"S027","kind":"supports"},{"source_id":"S009","target_id":"S019","kind":"supports"},{"source_id":"S009","target_id":"S021","kind":"supports"},{"source_id":"S009","target_id":"S034","kind":"supports"},{"source_id":"S009","target_id":"S035","kind":"supports"},{"source_id":"S009","target_id":"S039","kind":"supports"},{"source_id":"S010","target_id":"S015","kind":"supports"},{"source_id":"S010","target_id":"S032","kind":"supports"},{"source_id":"S011","target_id":"S030","kind":"supports"},{"source_id":"S011","target_id":"S038","kind":"supports"},{"source_id":"S012","target_id":"S016","kind":"supports"},{"source_id":"S012","target_id":"S029","kind":"supports"},{"source_id":"S012","target_id":"S032","kind":"supports"},{"source_id":"S012","target_id":"S033","kind":"supports"},{"source_id":"S013","target_id":"S019","kind":"supports"},{"source_id":"S013","target_id":"S032","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S014","kind":"supports"},{"source_id":"S017","target_id":"S025","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S020","target_id":"S018","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S018","kind":"supports"},{"source_id":"S022","target_id":"S033","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S024","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S023","kind":"supports"},{"source_id":"S026","target_id":"S023","kind":"supports"},{"source_id":"S027","target_id":"S023","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S036","kind":"supports"},{"source_id":"S029","target_id":"S002","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S029","target_id":"S033","kind":"supports"},{"source_id":"S029","target_id":"S035","kind":"supports"},{"source_id":"S029","target_id":"S039","kind":"supports"},{"source_id":"S030","target_id":"S002","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S002","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S002","kind":"supports"},{"source_id":"S033","target_id":"S002","kind":"supports"},{"source_id":"S034","target_id":"S002","kind":"supports"},{"source_id":"S035","target_id":"S002","kind":"supports"},{"source_id":"S036","target_id":"S002","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S002","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S038","target_id":"S002","kind":"supports"},{"source_id":"S039","target_id":"S002","kind":"supports"},{"source_id":"S040","target_id":"S012","kind":"supports"},{"source_id":"S041","target_id":"S029","kind":"supports"},{"source_id":"S041","target_id":"S036","kind":"supports"},{"source_id":"S042","target_id":"S016","kind":"supports"},{"source_id":"S043","target_id":"S030","kind":"supports"},{"source_id":"S043","target_id":"S038","kind":"supports"},{"source_id":"S044","target_id":"S035","kind":"supports"},{"source_id":"S045","target_id":"S034","kind":"supports"},{"source_id":"S045","target_id":"S039","kind":"supports"},{"source_id":"S046","target_id":"S035","kind":"supports"}]}}
{"record_id":"2406.17506v2","paper":{"title":"Exact worst-case convergence rates of gradient descent: a complete analysis for all constant stepsizes over nonconvex and convex functions","authors":["Teodor Rotaru","François Glineur","Panagiotis Patrinos"],"arxiv_id":"2406.17506","version":"v2","journal":"Mathematical Programming","doi":"10.1007/s10107-025-02313-1","abstract_url":"https://arxiv.org/abs/2406.17506v2","pdf_url":"https://arxiv.org/pdf/2406.17506v2"},"context":{"raw_tex":"We consider the unconstrained optimization problem %\n$$\\minimize_{x \\in \\mathbb{R}^d} f(x),$$\n\nwhere $f$ belongs to the class of functions $\\mathcal{F}_{\\mu, L}$, as defined below.%\n\n\n\\begin{definition}\\label{def:upper_lower_curvature} \nLet $L \\in (0, \\infty)$ and $\\mu\\in (-\\infty,L)$. Then a function $f\\colon\\mathbb{R}^d\\rightarrow \\mathbb{R}$ belongs to the class of functions $\\mathcal{F}_{\\mu,L}$ if it has:%\n\\begin{enumerate}[label=(\\roman*)]%\n    \\item upper curvature $L$, i.e., $ \\tfrac{L}{2} \\|\\cdot\\|^2 - f$ is convex;%\n    \\item lower curvature $\\mu$, i.e., $ f - \\tfrac{\\mu}{2} \\|\\cdot\\|^2$ is convex. %\n\\end{enumerate}%\n\\end{definition}\n\nWe sometimes represent the curvature ratio $\\kappa{}\\coloneqq{}\\frac{\\mu}{L} \\in (-\\infty, 1)$, which, if $\\mu \\geq 0$, signifies the inverse condition number.%\n\n\n\\noindent {\\textbf{Gradient descent.}} \nStarting from some initial point $x_0 \\in \\mathbb{R}^d$, we consider $N$ iterations of gradient descent with fixed stepsizes $\\gamma_i \\in (0, \\tfrac{2}{L})$, defined as:\n\\begin{align}\\label{eq:GM_it_fixed_steps}\\tag{GD}\n    x_{i+1} = x_i - \\gamma_i \\nabla f(x_i), \\quad \\forall i = 0,\\dots,N-1.\n\\end{align}\n\nThe constraint on the stepsize values ensures a decrease in the function value after a single iteration. We assume access to a first-order oracle denoted by the set of triplets $\\mathcal{T}{}\\coloneqq{}\\{(x_i,g_i,f_i)\\}_{i \\in \\mathcal{I}} \\subseteq \\mathbb{R}^d\\times\\mathbb{R}^d\\times\\mathbb{R}$, where $\\mathcal{I}$ is an index set, providing information about the iterates, their gradients and the corresponding function values. We assume the function $f$ to be bounded from below and denote $f_* {}\\coloneqq{} \\inf_x f(x) > -\\infty$. %\n\n\nWe denote $[x]_{+} {}\\coloneqq{} \\max\\{x, 0\\}$ for any $x \\in \\mathbb{R}$.\n\n\n\\begin{definition}\\label{def:def_TN}% \nLet $k$ be a positive integer and define function $E_k:\\mathbb{R}\\backslash\\{0\\} \\rightarrow \\mathbb{R}$:%\n\\begin{align}\n    \\Ei{}{}\\coloneqq{}{}\\sum\\limits_{j=1}^{2k} x^{-j} \n        {}={} \n     \\left\\{\n    \\arraycolsep=6pt\n    \\def\\arraystretch{1.25}\n    \\begin{array}{cc}\n      \\frac{-1+x^{-2k}}{1-x}, & x \\neq 1; \\\\\n      2k, & x = 1.\n    \\end{array}\n    \\right. \\label{eq:def_Ei} \\tag{$E_k$}\n\\end{align}%\nLet $V \\coloneqq \\{(\\ell, \\kappa): \\, \\kappa \\in (-\\infty, 1), \\,\\ell \\in (1, \\frac{2}{1+[\\kappa]_{+}})\\}$ and define $T_k: V \\rightarrow \\mathbb{R}$ as %\n\\begin{align}\n    T_k(\\ell,\\kappa)\n        {}{}\\coloneqq{}{}&\n    \\Ei[k][1-\\kappa \\ell] - \\Ei[k][1-\\ell] \\label{eq:def_TN} \\tag{$T_k$}.\n\\end{align}\n\\end{definition}\n\n\\begin{definition}[Stepsize thresholds]\\label{def:stepsize_thresholds}%\nLet $\\gbari[\\infty]{}\\coloneqq{} \\frac{2}{1+[\\kappa]_{+}}$. The unique roots of $\\TigLmu$ in the interval $(1,\\gbari[\\infty])$ are referred to as stepsize thresholds and denoted by $\\gbari[k]$, where $\\gbari[0] {}{}\\coloneqq{}{} 1$ and:%\n\\begin{align}\\label{def:gamma_bar} \\tag{$\\xoverline{\\gamma L}_k$}\n    \\big\\{\\gbari[k]\\big\\} \n        {}{}\\coloneqq{}{} \n    \\big\\{ \\gamma L \\in (1,\\gbari[\\infty]) \\,|\\, \\TigLmu[k] = 0 \\big\\} {}\\quad{} \\forall k = 1,2,{\\dots}\\,\\,.%\n\\end{align}%\n\\end{definition}\n\n\\begin{definition}[{\\cite[Definition 2]{taylor_smooth_2017}}] The set $\\mathcal{T}{}\\coloneqq{}\\{(x_i, g_i, f_i)\\}_{i \\in \\mathcal{I}}$ is called $\\mathcal{F}_{\\mu, L}$-interpolable if and only if there exists a function $f \\in \\mathcal{F}_{\\mu, L}$ such that $\\nabla f(x_i)=g_i$ and $f(x_i)=f_i$ for all $i\\in \\mathcal{I}$.\n\\end{definition}\n\nInterpolation conditions are used in conjunction with the  \\eqref{eq:GM_it_fixed_steps} iterations by substituting $x_{j}=x_{i} - \\sum_{k=i}^{j-1} \\gamma_k \\nabla f(x_k)$, as detailed in \\cref{def:dist-N_interp_cond}. Since the proofs are limited to constant stepsizes, we constrain $\\gamma_k$ to $\\gamma$.%\n\\begin{definition}\\label{def:dist-N_interp_cond}\nInequalities \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} are necessary and sufficient conditions connecting gradient descent iterations separated by $(j-i)$ steps:\n% %%%%%%%%%%%%%%%%%%%%%%% Improved decent version %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n\\begin{align}\\label{eq:distance-N_interpolation_conditions_I_ij}\n    & \\hspace{-.75em}\n    \\begin{aligned}\n    f(x_{i})-f(x_{j})\n        {}\\geq {}  \n    \\gamma \\big \\langle \\nabla f(x_{j}), \\sSum_{k=i}^{j-1} \\nabla f(x_{k}) \\big \\rangle +\n    \\frac{1}{2L} \\left\\|\\nabla f(x_{i}) - \\nabla f(x_{j})\\right\\|^2 + \n    \\\\% {}\\qquad{}\n    \\frac{\\mu}{2L(L{-}\\mu)} \\big\\|\\nabla f(x_{i}) - \\nabla f(x_{j}) - \\gamma L \\sSum_{k=i}^{j-1} \\nabla f(x_{k})\\big\\|^2; \n    \\end{aligned} \\tag{$\\Iij$} \\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    & \\hspace{-.75em}\n    \\begin{aligned} \n    f(x_{j})-f(x_{i}) \n        {}\\geq {}\n    {-}\\gamma \\big \\langle \\nabla f(x_{i}), \\sSum_{k=i}^{j-1} \\nabla f(x_{k}) \\big \\rangle +\n    \\frac{1}{2L} \\left\\|\\nabla f(x_{i}) - \\nabla f(x_{j})\\right\\|^2 + \n    \\\\% {}\\qquad{}\n    \\frac{\\mu}{2L(L{-}\\mu)} \\big\\|\\nabla f(x_{i}) - \\nabla f(x_{j}) - \\gamma L \\sSum_{k=i}^{j-1} \\nabla f(x_{k})\\big\\|^2.\n    \\end{aligned} \\tag{$\\Iji$}\n    \\label{eq:distance-N_interpolation_conditions_I_ji}\n\\end{align}%\nWe refer to inequalities \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} as \\textit{distance-1} or \\textit{distance-2} inequalities if the distances between their indices satisfy $|i-j|\\leq 1$ or $|i-j|\\leq 2$, respectively.%\n\\end{definition}\n\nInequalities \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} contribute via nonnegative weights \n$\\alphaij[i][j]$ and $\\alphaij[j][i]$, respectively, to form a generalized inequality linking $N$ iterations\\footnote{For linear combination of inequalities, we adopt the convention of writing the summed expression on the left-hand side, while stating the corresponding inequalities explicitly on the right-hand side.} such that there exists nonnegative weights $\\sigma_i \\geq 0$ satisfying:%\n\\begin{align}\\label{eq:I^k_N_general_ineq}\\tag{\\ENQ}\n\\sum_{0\\leq i < j \\leq N} \\big(\\alphaij[i][j] \\Iij[i][j] + \\alphaij[j][i] \\Iij[j][i]\\big)\\colon \\,\\,\n    f(x_0) - f(x_N) {}\\geq{}  \\sum_{i=0}^N \\sigma_i \\frac{\\|\\nabla f(x_i)\\|^2}{2L}.%\n\\end{align}\n\n\\noindent{\\textbf{Notation.}}\\, For readability reasons, we sometimes denote $f_i {=} f(x_i)$ and $g_i {=} \\nabla f(x_i)$. Recall: $\\kappa {=} \\frac{\\mu}{L}$ is the curvature ratio; $\\gamma L \\in (0,2)$ is the (normalized) stepsize and $\\gbari[\\infty] = \\tfrac{2}{1+[\\kappa]_{+}}$.%\n\n\nFor $|j-i|=1$, inequalities \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} simplify to:%\n\\begin{align} \n% \\hspace{-.75em}\n    f_{i}{-}f_{i+1}\n        {}\\geq {} &\n    \\frac{\\gamma \\mu \\, \\gamma L {-} 2 \\gamma \\mu {+} 1}{L{-}\\mu} \\frac{\\|g_{i}\\|^2}{2} +\n    \\frac{1}{L{-}\\mu} \\frac{\\|g_{i+1}\\|^{2}}{2} +     \n    \\frac{\\gamma L{-}1}{L{-}\\mu} \\langle g_{i}, g_{i+1}\\rangle; \n\\label{eq:hypo_interp_(0,1)} \\tag{\\Iij[i][i+1]}\\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n% \\hspace{-1em}\n    f_{i+1}{-}f_{i}\n        {}\\geq {} &\n    \\frac{\\gamma \\mu \\gamma L {-} 2 \\gamma L {+} 1}{L-\\mu} \\frac{\\|g_{i}\\|^2}{2} +\n    \\frac{1}{L{-}\\mu} \\frac{\\|g_{i+1}\\|^{2}}{2} +     \n    \\frac{\\gamma \\mu {-} 1}{L{-}\\mu} \\langle g_{i}, g_{i+1}\\rangle.\n\\label{eq:hypo_interp_(1,0)} \\tag{\\Iji[i][i+1]}\n\\end{align}\n\n\\paragraph*{\\textbf{Additional notation. }}\\, For brevity, we sometimes use $\\n{=}\\n(\\gamma L){}{\\coloneqq}{}1{-}\\gamma L \\in (-1, 0)$, $\\m{=}\\m(\\gamma \\mu){}{\\coloneqq}{}1{-}\\gamma \\mu$. Then $T_k(\\gamma L, \\kappa) = E_k(1{-}\\kappa \\gamma L) {-} E_k(1{-}\\gamma L)$ (see \\cref{def:def_TN}), where\n$E_k(x) = \\sum_{j=1}^{2k} x^{-j}$, can be equivalently written as $\\Tinm{}\\coloneqq{}\\TigLmu$, with $\\Tinm=\\Ei[k][\\m]-\\Ei[k][\\n]$. Recall $\\gbari[\\infty] = \\frac{2}{1+[\\kappa]_{+}}$.%\n\n\n\\begin{definition}\\label{def:N_bar}\nLet $\\Nbar\n        {}{}\\coloneqq{}{}\n    \\max \\{ k \\in \\mathbb{N} \\,|\\, \\TigLmu[k] {}\\geq{} 0\\}$.\n\\end{definition}\n\n\\begin{definition}\n    We use the writing $\\TigLmu[k][\\gamma L] {}={} \\sum_{i=1}^k \\delta_i(\\gamma L, \\kappa)$, where\n    $$\n        \\delta_i(\\gamma L,\\kappa) \n            {}{}\\coloneqq{}{} \n        \\frac{2-\\kappa \\gamma L}{(1-\\kappa \\gamma L)^{2i}} - \n        \\frac{2-\\gamma L}{(1-\\gamma L)^{2i}}.\n    $$ \n\\end{definition}","units":[{"start":15890,"end":15983,"start_line":329,"end_line":330,"raw_tex":"We consider the unconstrained optimization problem %\n$$\\minimize_{x \\in \\mathbb{R}^d} f(x),$$","sha256_utf8":"cf2813890a35cd3cda8b1b026a9ad0623b789920c09c04963c35e27e6a5bf750"},{"start":15985,"end":16072,"start_line":331,"end_line":331,"raw_tex":"where $f$ belongs to the class of functions $\\mathcal{F}_{\\mu, L}$, as defined below.%\n","sha256_utf8":"f6724200c232a933991134addeb8c6b30d5ffea8867dbf8ff3e92702d93b3cf9"},{"start":16074,"end":16541,"start_line":333,"end_line":339,"raw_tex":"\\begin{definition}\\label{def:upper_lower_curvature} \nLet $L \\in (0, \\infty)$ and $\\mu\\in (-\\infty,L)$. Then a function $f\\colon\\mathbb{R}^d\\rightarrow \\mathbb{R}$ belongs to the class of functions $\\mathcal{F}_{\\mu,L}$ if it has:%\n\\begin{enumerate}[label=(\\roman*)]%\n    \\item upper curvature $L$, i.e., $ \\tfrac{L}{2} \\|\\cdot\\|^2 - f$ is convex;%\n    \\item lower curvature $\\mu$, i.e., $ f - \\tfrac{\\mu}{2} \\|\\cdot\\|^2$ is convex. %\n\\end{enumerate}%\n\\end{definition}","sha256_utf8":"a0c68f257c30eaaf5dfe19770e75782c2cdb61f1b650824a69d597d937d03c3d"},{"start":17328,"end":17489,"start_line":345,"end_line":345,"raw_tex":"We sometimes represent the curvature ratio $\\kappa{}\\coloneqq{}\\frac{\\mu}{L} \\in (-\\infty, 1)$, which, if $\\mu \\geq 0$, signifies the inverse condition number.%\n","sha256_utf8":"c0df0527c53f77be8cf1ffc1defd02851aabff80f4f853026038dfad8a9487bf"},{"start":17490,"end":17837,"start_line":347,"end_line":351,"raw_tex":"\\noindent {\\textbf{Gradient descent.}} \nStarting from some initial point $x_0 \\in \\mathbb{R}^d$, we consider $N$ iterations of gradient descent with fixed stepsizes $\\gamma_i \\in (0, \\tfrac{2}{L})$, defined as:\n\\begin{align}\\label{eq:GM_it_fixed_steps}\\tag{GD}\n    x_{i+1} = x_i - \\gamma_i \\nabla f(x_i), \\quad \\forall i = 0,\\dots,N-1.\n\\end{align}","sha256_utf8":"b62a770d458a6d8eaf93c87c4590aa10f1ba1a0d63f11c599aaa841becd21702"},{"start":17841,"end":18382,"start_line":353,"end_line":353,"raw_tex":"The constraint on the stepsize values ensures a decrease in the function value after a single iteration. We assume access to a first-order oracle denoted by the set of triplets $\\mathcal{T}{}\\coloneqq{}\\{(x_i,g_i,f_i)\\}_{i \\in \\mathcal{I}} \\subseteq \\mathbb{R}^d\\times\\mathbb{R}^d\\times\\mathbb{R}$, where $\\mathcal{I}$ is an index set, providing information about the iterates, their gradients and the corresponding function values. We assume the function $f$ to be bounded from below and denote $f_* {}\\coloneqq{} \\inf_x f(x) > -\\infty$. %\n","sha256_utf8":"a1bc84cd8061308b7e6d8383f34e36e09e4f114a2a54410b0efd8e203d6e6c9d"},{"start":18743,"end":18818,"start_line":356,"end_line":356,"raw_tex":"We denote $[x]_{+} {}\\coloneqq{} \\max\\{x, 0\\}$ for any $x \\in \\mathbb{R}$.\n","sha256_utf8":"b526ccbabd3c70600d4a28d7b23b355f69e6cbf5aa76bb4c3cebf7bd4f1e1552"},{"start":56454,"end":57222,"start_line":818,"end_line":838,"raw_tex":"\\begin{definition}\\label{def:def_TN}% \nLet $k$ be a positive integer and define function $E_k:\\mathbb{R}\\backslash\\{0\\} \\rightarrow \\mathbb{R}$:%\n\\begin{align}\n    \\Ei{}{}\\coloneqq{}{}\\sum\\limits_{j=1}^{2k} x^{-j} \n        {}={} \n     \\left\\{\n    \\arraycolsep=6pt\n    \\def\\arraystretch{1.25}\n    \\begin{array}{cc}\n      \\frac{-1+x^{-2k}}{1-x}, & x \\neq 1; \\\\\n      2k, & x = 1.\n    \\end{array}\n    \\right. \\label{eq:def_Ei} \\tag{$E_k$}\n\\end{align}%\nLet $V \\coloneqq \\{(\\ell, \\kappa): \\, \\kappa \\in (-\\infty, 1), \\,\\ell \\in (1, \\frac{2}{1+[\\kappa]_{+}})\\}$ and define $T_k: V \\rightarrow \\mathbb{R}$ as %\n\\begin{align}\n    T_k(\\ell,\\kappa)\n        {}{}\\coloneqq{}{}&\n    \\Ei[k][1-\\kappa \\ell] - \\Ei[k][1-\\ell] \\label{eq:def_TN} \\tag{$T_k$}.\n\\end{align}\n\\end{definition}","sha256_utf8":"cce081e27c981de5bdfc7d3c9f93e2b704b583f903027b0ef637014158ec2de3"},{"start":58002,"end":58572,"start_line":848,"end_line":855,"raw_tex":"\\begin{definition}[Stepsize thresholds]\\label{def:stepsize_thresholds}%\nLet $\\gbari[\\infty]{}\\coloneqq{} \\frac{2}{1+[\\kappa]_{+}}$. The unique roots of $\\TigLmu$ in the interval $(1,\\gbari[\\infty])$ are referred to as stepsize thresholds and denoted by $\\gbari[k]$, where $\\gbari[0] {}{}\\coloneqq{}{} 1$ and:%\n\\begin{align}\\label{def:gamma_bar} \\tag{$\\xoverline{\\gamma L}_k$}\n    \\big\\{\\gbari[k]\\big\\} \n        {}{}\\coloneqq{}{} \n    \\big\\{ \\gamma L \\in (1,\\gbari[\\infty]) \\,|\\, \\TigLmu[k] = 0 \\big\\} {}\\quad{} \\forall k = 1,2,{\\dots}\\,\\,.%\n\\end{align}%\n\\end{definition}","sha256_utf8":"76d4669edfdb11e1c93030fe57446a9c5c30db50c035fb8bb18ea7d6547a7430"},{"start":89517,"end":89858,"start_line":1206,"end_line":1207,"raw_tex":"\\begin{definition}[{\\cite[Definition 2]{taylor_smooth_2017}}] The set $\\mathcal{T}{}\\coloneqq{}\\{(x_i, g_i, f_i)\\}_{i \\in \\mathcal{I}}$ is called $\\mathcal{F}_{\\mu, L}$-interpolable if and only if there exists a function $f \\in \\mathcal{F}_{\\mu, L}$ such that $\\nabla f(x_i)=g_i$ and $f(x_i)=f_i$ for all $i\\in \\mathcal{I}$.\n\\end{definition}","sha256_utf8":"d5809c3891ab55ca627fa6bf1467e72c6c476dae0b4f99331402d338c88cd51f"},{"start":92345,"end":94363,"start_line":1234,"end_line":1261,"raw_tex":"Interpolation conditions are used in conjunction with the  \\eqref{eq:GM_it_fixed_steps} iterations by substituting $x_{j}=x_{i} - \\sum_{k=i}^{j-1} \\gamma_k \\nabla f(x_k)$, as detailed in \\cref{def:dist-N_interp_cond}. Since the proofs are limited to constant stepsizes, we constrain $\\gamma_k$ to $\\gamma$.%\n\\begin{definition}\\label{def:dist-N_interp_cond}\nInequalities \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} are necessary and sufficient conditions connecting gradient descent iterations separated by $(j-i)$ steps:\n% %%%%%%%%%%%%%%%%%%%%%%% Improved decent version %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n\\begin{align}\\label{eq:distance-N_interpolation_conditions_I_ij}\n    & \\hspace{-.75em}\n    \\begin{aligned}\n    f(x_{i})-f(x_{j})\n        {}\\geq {}  \n    \\gamma \\big \\langle \\nabla f(x_{j}), \\sSum_{k=i}^{j-1} \\nabla f(x_{k}) \\big \\rangle +\n    \\frac{1}{2L} \\left\\|\\nabla f(x_{i}) - \\nabla f(x_{j})\\right\\|^2 + \n    \\\\% {}\\qquad{}\n    \\frac{\\mu}{2L(L{-}\\mu)} \\big\\|\\nabla f(x_{i}) - \\nabla f(x_{j}) - \\gamma L \\sSum_{k=i}^{j-1} \\nabla f(x_{k})\\big\\|^2; \n    \\end{aligned} \\tag{$\\Iij$} \\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    & \\hspace{-.75em}\n    \\begin{aligned} \n    f(x_{j})-f(x_{i}) \n        {}\\geq {}\n    {-}\\gamma \\big \\langle \\nabla f(x_{i}), \\sSum_{k=i}^{j-1} \\nabla f(x_{k}) \\big \\rangle +\n    \\frac{1}{2L} \\left\\|\\nabla f(x_{i}) - \\nabla f(x_{j})\\right\\|^2 + \n    \\\\% {}\\qquad{}\n    \\frac{\\mu}{2L(L{-}\\mu)} \\big\\|\\nabla f(x_{i}) - \\nabla f(x_{j}) - \\gamma L \\sSum_{k=i}^{j-1} \\nabla f(x_{k})\\big\\|^2.\n    \\end{aligned} \\tag{$\\Iji$}\n    \\label{eq:distance-N_interpolation_conditions_I_ji}\n\\end{align}%\nWe refer to inequalities \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} as \\textit{distance-1} or \\textit{distance-2} inequalities if the distances between their indices satisfy $|i-j|\\leq 1$ or $|i-j|\\leq 2$, respectively.%\n\\end{definition}","sha256_utf8":"a1189a5d3fab1f89dd33d1b00d71aa646edb642e58ef788b60e0369095e6266a"},{"start":98944,"end":99744,"start_line":1295,"end_line":1300,"raw_tex":"Inequalities \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} contribute via nonnegative weights \n$\\alphaij[i][j]$ and $\\alphaij[j][i]$, respectively, to form a generalized inequality linking $N$ iterations\\footnote{For linear combination of inequalities, we adopt the convention of writing the summed expression on the left-hand side, while stating the corresponding inequalities explicitly on the right-hand side.} such that there exists nonnegative weights $\\sigma_i \\geq 0$ satisfying:%\n\\begin{align}\\label{eq:I^k_N_general_ineq}\\tag{\\ENQ}\n\\sum_{0\\leq i < j \\leq N} \\big(\\alphaij[i][j] \\Iij[i][j] + \\alphaij[j][i] \\Iij[j][i]\\big)\\colon \\,\\,\n    f(x_0) - f(x_N) {}\\geq{}  \\sum_{i=0}^N \\sigma_i \\frac{\\|\\nabla f(x_i)\\|^2}{2L}.%\n\\end{align}","sha256_utf8":"c4a49dc6189edf56cce83d573ecb823027a7f5e0f118c3d0ab6535053d27b3db"},{"start":101158,"end":101441,"start_line":1329,"end_line":1329,"raw_tex":"\\noindent{\\textbf{Notation.}}\\, For readability reasons, we sometimes denote $f_i {=} f(x_i)$ and $g_i {=} \\nabla f(x_i)$. Recall: $\\kappa {=} \\frac{\\mu}{L}$ is the curvature ratio; $\\gamma L \\in (0,2)$ is the (normalized) stepsize and $\\gbari[\\infty] = \\tfrac{2}{1+[\\kappa]_{+}}$.%\n","sha256_utf8":"6dda44fce39aac5f5821f75dae965c4bd10506b4e07e7aee3fe016d12c47abee"},{"start":101546,"end":102429,"start_line":1335,"end_line":1352,"raw_tex":"For $|j-i|=1$, inequalities \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} simplify to:%\n\\begin{align} \n% \\hspace{-.75em}\n    f_{i}{-}f_{i+1}\n        {}\\geq {} &\n    \\frac{\\gamma \\mu \\, \\gamma L {-} 2 \\gamma \\mu {+} 1}{L{-}\\mu} \\frac{\\|g_{i}\\|^2}{2} +\n    \\frac{1}{L{-}\\mu} \\frac{\\|g_{i+1}\\|^{2}}{2} +     \n    \\frac{\\gamma L{-}1}{L{-}\\mu} \\langle g_{i}, g_{i+1}\\rangle; \n\\label{eq:hypo_interp_(0,1)} \\tag{\\Iij[i][i+1]}\\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n% \\hspace{-1em}\n    f_{i+1}{-}f_{i}\n        {}\\geq {} &\n    \\frac{\\gamma \\mu \\gamma L {-} 2 \\gamma L {+} 1}{L-\\mu} \\frac{\\|g_{i}\\|^2}{2} +\n    \\frac{1}{L{-}\\mu} \\frac{\\|g_{i+1}\\|^{2}}{2} +     \n    \\frac{\\gamma \\mu {-} 1}{L{-}\\mu} \\langle g_{i}, g_{i+1}\\rangle.\n\\label{eq:hypo_interp_(1,0)} \\tag{\\Iji[i][i+1]}\n\\end{align}","sha256_utf8":"ff43dc7f322e5ee089bf9b10a6fc6c166a96433d0772fbd03693446efe935fb1"},{"start":109185,"end":109666,"start_line":1449,"end_line":1450,"raw_tex":"\\paragraph*{\\textbf{Additional notation. }}\\, For brevity, we sometimes use $\\n{=}\\n(\\gamma L){}{\\coloneqq}{}1{-}\\gamma L \\in (-1, 0)$, $\\m{=}\\m(\\gamma \\mu){}{\\coloneqq}{}1{-}\\gamma \\mu$. Then $T_k(\\gamma L, \\kappa) = E_k(1{-}\\kappa \\gamma L) {-} E_k(1{-}\\gamma L)$ (see \\cref{def:def_TN}), where\n$E_k(x) = \\sum_{j=1}^{2k} x^{-j}$, can be equivalently written as $\\Tinm{}\\coloneqq{}\\TigLmu$, with $\\Tinm=\\Ei[k][\\m]-\\Ei[k][\\n]$. Recall $\\gbari[\\infty] = \\frac{2}{1+[\\kappa]_{+}}$.%\n","sha256_utf8":"7ff7536607e0ce95236cd12472689b65e91a041676c62f90f5c3576e948989e4"},{"start":109668,"end":109818,"start_line":1452,"end_line":1456,"raw_tex":"\\begin{definition}\\label{def:N_bar}\nLet $\\Nbar\n        {}{}\\coloneqq{}{}\n    \\max \\{ k \\in \\mathbb{N} \\,|\\, \\TigLmu[k] {}\\geq{} 0\\}$.\n\\end{definition}","sha256_utf8":"1ad9dc04a7619a868025bfced50775cd5bdb52a5b981aa51be688d6241e1ec5e"},{"start":184332,"end":184655,"start_line":2758,"end_line":2766,"raw_tex":"\\begin{definition}\n    We use the writing $\\TigLmu[k][\\gamma L] {}={} \\sum_{i=1}^k \\delta_i(\\gamma L, \\kappa)$, where\n    $$\n        \\delta_i(\\gamma L,\\kappa) \n            {}{}\\coloneqq{}{} \n        \\frac{2-\\kappa \\gamma L}{(1-\\kappa \\gamma L)^{2i}} - \n        \\frac{2-\\gamma L}{(1-\\gamma L)^{2i}}.\n    $$ \n\\end{definition}","sha256_utf8":"68890f2b17c8487ea09a74d7626280d3b2dc2d38d882983133cd494ad6e99792"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{thm:strg_cvx_rate}%\nLet $f \\in \\mathcal{F}_{\\mu, L}$, with $\\mu \\in (0, L)$, and consider $N$ iterations of \\eqref{eq:GM_it_fixed_steps} with stepsizes $\\gamma L \\in (0, 2)$ starting from $x_0$. Then the following bound holds:%\n\\begin{align}\\label{eq:strg_cvx_full_ct}\n\\begin{aligned}\n    \\frac{1}{2L} \\|\\nabla f(x_N)\\|^2\n        {} \\leq {}  \n    \\frac{f(x_0)-f_*}\n    { 1 + \\gamma L \\min \\big\\{\n    \\frac{-1+(1-\\gamma \\mu)^{-2N}}{\\gamma \\mu}\n    \\,,\\,\n    \\frac{-1+(1-\\gamma L)^{-2N}}{\\gamma L}\n         \\big\\}}.\n\\end{aligned}\n\\end{align}%\n\\end{theorem}","units":[{"start":39285,"end":39861,"start_line":562,"end_line":576,"raw_tex":"\\begin{theorem}\\label{thm:strg_cvx_rate}%\nLet $f \\in \\mathcal{F}_{\\mu, L}$, with $\\mu \\in (0, L)$, and consider $N$ iterations of \\eqref{eq:GM_it_fixed_steps} with stepsizes $\\gamma L \\in (0, 2)$ starting from $x_0$. Then the following bound holds:%\n\\begin{align}\\label{eq:strg_cvx_full_ct}\n\\begin{aligned}\n    \\frac{1}{2L} \\|\\nabla f(x_N)\\|^2\n        {} \\leq {}  \n    \\frac{f(x_0)-f_*}\n    { 1 + \\gamma L \\min \\big\\{\n    \\frac{-1+(1-\\gamma \\mu)^{-2N}}{\\gamma \\mu}\n    \\,,\\,\n    \\frac{-1+(1-\\gamma L)^{-2N}}{\\gamma L}\n         \\big\\}}.\n\\end{aligned}\n\\end{align}%\n\\end{theorem}","sha256_utf8":"09f5d4ba7a1261848383af681852475f9855fa710a5f066300f2a71c3b491edb"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{proposition}\\label{prop:monotonicity_Tk}%\n    The function $\\TigLmu[k][\\cdot][\\kappa]$ is strictly increasing in the first argument.%\n\\end{proposition}\n\n\\begin{proof}%\nSee \\cref{appendix:proof_prop_monotonicity_Tk}.%\n\\qed\\end{proof}\n\n\\begin{proposition}\\label{prop:properties_step_size_thr}\n    The stepsize thresholds $\\gbari$ satisfy the following properties:%\n    \\begin{enumerate}[align=right,itemindent=1em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]\n        \\item $\\gbari[k] < \\gbari[k+1]$, for all integers $k \\geq 0$;\n        \\item $\\lim_{k\\rightarrow \\infty} \\gbari[k] = \\gbari[\\infty]$;\n        \\item \\shownewtextfinal{For any $\\gamma L \\in \\big[\\gbari[k],\\gbari[k+1]\\big)$: \n        $\\TigLmu[i][\\gamma L] \\geq 0$ $\\forall i \\leq k$, \n        and $\\TigLmu[i][\\gamma L] < 0$ $\\forall i \\geq k+1$.}\n     \\end{enumerate}%\n\\end{proposition}\n\n\\begin{proof}%\nSee \\cref{appendix:proof_properties_step_size_thr}.%\n\\qed\\end{proof}\n\n\\begin{theorem}[Interpolation conditions for smooth functions]\\label{thm:interp_hypo}\nSet $\\mathcal{T}=\\big\\{(x_i, g_i, f_i)\\big\\}_{i \\in \\mathcal{I}}$ is $\\mathcal{F}_{\\mu,L}$-interpolable, where $\\mu \\in (-\\infty, L)$, if and only if for every pair of indices $(i,j)$, with $i,j \\in \\mathcal{I}$, it holds:%\n\\begin{align}\\label{eq:Interp_hypoconvex}\nf_{i}{-}f_{j}{-}\\langle g_{j}, x_{i}-x_{j} \\rangle \n        {}\\geq{}\n    \\frac{1}{2L} \\|g_{i}{-}g_{j}\\|^{2} {}+{} \\frac{\\mu}{2L(L{-}\\mu)} \\|g_{i}{-}g_{j} {-} L (x_i {-}x_j)\\|^2.\n\\end{align}%\n\\end{theorem}\n\n\\begin{corollary} \\label{corr:direct_sum_interp_cond} \nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty, L)$. For all $x_i, x_j \\in \\mathbb{R}^d$ it holds:%\n\\begin{align}\\label{eq:direct_sum_interp_cond}\n    0 {}\\geq{} \\left\\langle \\nabla f(x_i) - \\nabla f(x_j) - L(x_i-x_j) \\,,\\, \n                           \\nabla f(x_i) - \\nabla f(x_j) - \\mu(x_i-x_j) \\right\\rangle.\n\\end{align}%\n\\end{corollary}\n\n\\begin{proof}\n    Inequality \\eqref{eq:direct_sum_interp_cond} results by summing up inequalities \\eqref{eq:Interp_hypoconvex} written for the pairs $(x_i,x_j)$ and $(x_j,x_i)$ and performing simplifications.\n\\qed\\end{proof}\n\n\\begin{lemma}\\label{lemma:sufficient_decrease_N_*}\nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty, L)$. One iteration of \\eqref{eq:GM_it_fixed_steps} with stepsize $\\gamma L \\in (0,2)$ decreases the function values and at each point $x_i$ it holds:\n\\begin{align}\\label{eq:sufficient_decrease_N_*}\n    f(x_i) - f_* {}\\geq{} \\frac{1}{2L} \\|\\nabla f(x_i)\\|^2.\n\\end{align}%\n\\end{lemma}\n\n\\begin{proposition}[General procedure]\\label{prop:general_procedure_min_grad_nrm}\n    Inequality \\eqref{eq:I^k_N_general_ineq} implies:\n\\[\n    \\frac{f(x_0) - f(x_N)}{\\sum_{i=0}^N \\sigma_i}\n        {}\\geq{}\n    \\frac{1}{2L} \\,\n    \\min_{0\\leq i \\leq N} \\big\\{\\|\\nabla f(x_i)\\|^2\\big\\},\n\\]\nwith equality only if all gradient norms $\\|\\nabla f(x_i)\\|^2$ with $\\sigma_i>0$ are equal. Moreover, \n\\[\n    \\frac{f(x_0) - f_*}{1 + \\sum_{i=0}^N \\sigma_i}\n        {}\\geq{}\n    \\frac{1}{2L} \\,\n    \\min_{0\\leq i \\leq N} \\big\\{\\|\\nabla f(x_i)\\|^2\\big\\}.\n\\]\n\\end{proposition}\n\n\\begin{proof}\n   The first inequality results after taking minimum gradient norm in \\eqref{eq:I^k_N_general_ineq}. %\nThe adjustment in the denominator of the second inequality results from combining \\eqref{eq:sufficient_decrease_N_*} with \\eqref{eq:I^k_N_general_ineq} and taking again the minimum gradient norm. \\qed\n\\end{proof}\n\n\\begin{lemma}\\label{lemma:sufficient_decrease_h_leq_1}\n    Let $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty, L)$, and consider one iteration of \\eqref{eq:GM_it_fixed_steps} with stepsize $\\gamma L \\in(0,1]$. Then the following inequality holds, with equality only if $\\nabla f(x_{i}) = \\nabla f(x_{i+1})$ or $\\gamma L = 1$:\n\\begin{align}\\label{eq:one_step_rate_h_leq_1} \\tag{N2SD}\n\\hspace{-.5em}\n    f(x_{i}) {-} f(x_{i+1})\n        {}\\geq {}\n    \\frac{\\gamma L \\, \\gamma \\mu {-} 2\\gamma \\mu {+} \\gamma L}{L{-}\\mu} \\frac{\\|\\nabla f(x_{i})\\|^2}{2} +\n    \\frac{\\gamma L}{L{-}\\mu} \\frac{\\|\\nabla f(x_{i+1})\\|^{2}}{2}.\n\\end{align}%\n    Furthermore, if $\\mu \\leq 0$, then the gradient norms from \\eqref{eq:one_step_rate_h_leq_1} are weighted by positive quantities.\n\\end{lemma}\n\n\\begin{proof}\\label{proof:lemma:sufficient_decrease_h_leq_1}\n    Inequality \\eqref{eq:hypo_interp_(0,1)} is reformulated by expressing the inner product in terms of gradient norms, i.e., $2 \\big\\langle g_{i}, g_{i+1}\\big\\rangle = \\|g_{i}\\|^2 + \\|g_{i+1}\\|^2 - \\|g_{i} - g_{i+1}\\|^2$, as:%\n\\begin{align} \\label{eq:tight_short_step_sizes}\n    f_{i}{-}f_{i+1}\n        {}\\geq {}\n    \\frac{\\gamma L \\, \\gamma \\mu {-} 2\\gamma \\mu {+} \\gamma L}{L{-}\\mu} \\frac{\\|g_{i}\\|^2}{2} +\n    \\frac{\\gamma L}{L{-}\\mu} \\frac{\\|g_{i+1}\\|^{2}}{2} +\n    \\frac{1 {-} \\gamma L}{L{-}\\mu} \\frac{\\|g_{i} {-} g_{i+1}\\|^{2}}{2}.\n\\end{align}\nSince $\\gamma L \\in (0,1]$, the mixed term can be neglected, resulting in inequality \\eqref{eq:one_step_rate_h_leq_1}, with equality only if $g_{i} = g_{i+1}$ or $\\gamma L = 1$. The numerator of first term rewrites as $-\\gamma \\mu(1-\\gamma L) + (\\gamma L - \\gamma \\mu)$, hence it is positive when $\\mu \\leq 0$.\n\\qed\\end{proof}\n\n\\begin{lemma}\\label{lemma:sufficient_decrease_h_geq_1}\n    Let $f \\in \\mathcal{F}_{\\mu,L}$, with, $\\mu \\in (-\\infty,L)$, and consider one iteration of \\eqref{eq:GM_it_fixed_steps} with stepsize $\\gamma L \\in [1,\\gbari[\\infty])$. Then the following inequality holds:%\n\\begin{align} \\label{eq:one_step_rate_h_geq_1} \\tag{N4SD}\n% \\hspace{-.25em}\n    f(x_{i}){-}f(x_{i+1})\n        {}\\geq {}\n    \\frac{\\gamma [(2{-}\\gamma L)(2{-}\\gamma \\mu){-}1]}{2{-}\\gamma L {- }\\gamma \\mu}\n    \\frac{\\|\\nabla f(x_{i})\\|^2}{2} +\n    \\frac{\\gamma}{2{-}\\gamma L {-} \\gamma \\mu} \\frac{\\|\\nabla f(x_{i+1})\\|^{2}}{2}.\n\\end{align}%\nWhen $\\gamma L > 1$, equality holds only if:%\n\\begin{align}\\label{eq:equality_case_h_geq_1}\n    {\\langle \\nabla f(x_{i})}, {\\nabla f(x_{i+1})\\rangle}\n        {}{=}{} \n    \\frac{(1{-}\\gamma \\mu)(1{-}\\gamma L)}{2{-}\\gamma L{-}\\gamma \\mu} \\|\\nabla f(x_{i})\\|^2 {}{+}{} \n    \\frac{1}{2{-}\\gamma L{-}\\gamma \\mu} \\|\\nabla f(x_{i+1})\\|^2.%\n\\end{align}%\nFurthermore, if $\\gamma L \\in [1,\\gbari[1]]$, then the gradient norms from \\eqref{eq:one_step_rate_h_geq_1} are weighted by nonnegative quantities.%\n\\end{lemma}\n\n\\begin{proof}\\label{proof:lemma:sufficient_decrease_h_geq_1}\nThe mixed term in inequality \\eqref{eq:tight_short_step_sizes} enters with a negative contribution for $\\gamma L > 1$. To counterbalance it, the complementary interpolation inequality \\eqref{eq:hypo_interp_(1,0)} is used, written with the equivalent expression:%\n\\begin{align*}\n   f_{i+1}-f_{i}\n        {}\\geq {}\n    \\frac{\\gamma L \\gamma \\mu {-} 2 \\gamma L {+} \\gamma \\mu}{L{-}\\mu} \\frac{\\|g_{i}\\|^2}{2} +\n    \\frac{\\gamma \\mu}{L{-}\\mu} \\frac{\\|g_{i+1}\\|^{2}}{2} +\n    \\frac{1 {-} \\gamma \\mu}{L{-}\\mu} \\frac{\\|g_{i} {-} g_{i+1}\\|^{2}}{2}.%\n\\end{align*}%\nConsider the sum of distance-1 inequalities $\\bar{I}_{i}^{1} {}\\coloneqq{} \\Iij[i][i+1] + \\Iji[i][i+1]$:%\n\\begin{align}\\label{eq:dist_1_blended_mixed_grads_diff} \\tag{$\\bar{I}_{i}^{1}$}\n\\hspace{-0.5em}\n0\n    {}\\geq{}\n\\frac{2 \\gamma L \\gamma \\mu {-} \\gamma L {-} \\gamma \\mu}{L{-}\\mu} \\frac{\\|g_{i}\\|^2}{2} {}+{}\n\\frac{\\gamma L {+} \\gamma \\mu}{L{-}\\mu} \\frac{\\|g_{i+1}\\|^2}{2} {}+{} \n\\frac{2{-}\\gamma L {-} \\gamma \\mu}{L{-}\\mu} \\frac{\\| g_{i} {-} g_{i+1}\\|^2}{2}.\n\\end{align}%\nBy weighting \\eqref{eq:dist_1_blended_mixed_grads_diff} with $\\beta(\\gamma L,\\gamma \\mu){}\\coloneqq{} \\tfrac{\\gamma L{-}1}{2{-}\\gamma L {-}\\gamma \\mu} \\geq 0$ and summing it with \\eqref{eq:tight_short_step_sizes}, the coefficient of $\\| g_{i} {-} g_{i+1}\\|^2$ gets canceled and we obtain \\eqref{eq:one_step_rate_h_geq_1}.\nThe equality condition \\eqref{eq:equality_case_h_geq_1} follows from the equality case of \\eqref{eq:dist_1_blended_mixed_grads_diff}. Nevertheless, the coefficient of $\\|g_{i}\\|^2$ is nonnegative for $\\gamma L \\in [1,\\gbari[1]]$, where $\\gbari[1]$ is the largest stepsize $\\gamma L \\in [1, 2)$ for which $(2-\\gamma L)(2-\\gamma \\mu) - 1 \\geq {} 0$, implying nonnegativity of the weight of $\\|\\nabla f(x_{i})\\|^2$ in \\eqref{eq:one_step_rate_h_geq_1}.\\qed\n\\end{proof}\n\n\\begin{proposition}\\label{prop:properties:m+n>0}%[Properties for distance-2]\n    Let $\\mu \\in (-\\infty, L)$ and $\\gamma L \\in \\big(1, \\gbari[\\infty] \\big)$. Then $\\m {}>{}  0$, $\\m + \\n {}>0$ and $\\n^{-2} {}>{} \\m^{-2}$.\n\\end{proposition}\n\n\\begin{proof}%\nCondition $\\gamma L < \\gbari[\\infty]$ implies $\\gamma L < \\frac{2}{1+[\\kappa]_{+}}$, hence $0 < 2 {-} \\gamma L {-} \\gamma L [\\kappa]_{+} {}\\leq{} 2 {-} \\gamma L {-} \\gamma L \\kappa$, implying $\\m + \\n > 0$. Since $\\n = 1-\\gamma L \\in (-1,0)$, we have $\\m=1-\\gamma \\mu > 0$. Condition $L>\\mu$ implies $\\m>\\n$ and together with $\\m+\\n>0$ we conclude that $\\m^2 > \\n^2$, therefore $\\n^{-2} > \\m^{-2}$.\\qed\n\\end{proof}\n\n\\begin{lemma}\\label{lemma:dist_1_L_1_step_decrease}\nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty,L)$, and consider one iteration of \\eqref{eq:GM_it_fixed_steps} from $x_i$ with stepsize $\\gamma L \\in (1, 2)$. Then it holds:\n\\begin{align} \\label{eq:dist_1_L_1_step_decrease} \n    \\begin{aligned}    \n    \\frac{f(x_{i}) - f(x_{i+1})}{\\gamma} \n                {}\\geq{}\n            {-}\\Ei[i][\\n] \\frac{\\|\\nabla f(x_{i})\\|^2}{2} {}+{}\n            \\Ei[i+1][\\n] \\frac{\\|\\nabla f(x_{i+1})\\|^2}{2} {}+{}            \n            \\\\ {}+{}\n            \\frac{ 1 - (\\m+\\n) \\Ei[i+1][\\n] }{\\m-\\n} \\frac{\\|\\nabla f(x_{i+1})-\\n \\nabla f(x_{i})\\|^2}{2}.\n    \\end{aligned}\n\\end{align}%\nFurthermore, if $\\mu \\in (0,L)$ and $\\gamma L \\in [\\frac{2}{1+\\kappa}, 2)$, then $1-(\\m+\\n) \\Ei[i][\\n] > 0$.\n\\end{lemma}\n\n\\begin{proof}\nLet $\\beta_{i+1}(\\n) {{}{}\\coloneqq{}{}} (-\\n) \\Ei[i+1][\\n] {}>{} 0$. Inequality \\eqref{eq:dist_1_L_1_step_decrease} results after performing the simplifications in the linear combination of interpolation inequalities:\n$$\n\\Iij[i][i+1] + \\beta_{i+1}(\\n) \\, (\\Iij[i][i+1] + \\Iji[i][i+1]),$$\nwhere \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} are equivalently rewritten for the pair $(i,i+1)$ as:%\n$$\n    \\begin{aligned}\n    (1+\\beta_{i+1}) {\\,} \\Iij[i][i+1] {}:{}\n        \\frac{f_{i} {-} f_{i+1}}{\\gamma}\n            {}\\geq{}&\n        (1+\\n) \\frac{\\|g_{i}\\|^2}{2} {+} \\frac{1}{\\m{-}\\n} \\frac{\\|g_{i+1}{-}\\n g_{i}\\|^2}{2}; \\\\\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\beta_{i+1} {\\,} \\Iji[i][i+1] {}:{}\n        \\frac{f_{i+1} {-} f_{i}}{\\gamma}\n            {}\\geq{}&\n        {-}\\frac{\\|g_{i}\\|^2}{2} {-} \\frac{1}{\\n} \\frac{\\|g_{i+1}\\|^2}{2} {+} \\frac{\\m}{\\n} \\frac{1}{\\m{-}\\n} \\frac{\\|g_{i+1}{-}\\n g_{i}\\|^2}{2}.\n    \\end{aligned}\n$$%\n\\noindent{}With $\\gamma L \\in {[}\\frac{2}{1+\\kappa}, 2)$, we have $\\m{+}\\n \\leq 0$ (see \\cref{prop:properties:m+n>0}) and therefore $1-(\\m+\\n)\\Ei[i+1][\\n] > 0$.%\n\\qed \\end{proof}\n\n\\begin{lemma}\\label{lemma:dist_1_mu_1_step_decrease}\nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty,L)$, and consider one iteration of \\eqref{eq:GM_it_fixed_steps} from $x_i$ with stepsize $\\gamma L \\in \\big(0, \\gbari[\\infty]\\big)$. Then it holds:%\n\\begin{align} \\label{eq:dist_1_mu_1_step_decrease}\n    \\begin{aligned}    \n    \\frac{f(x_{i}) - f(x_{i+1})}{\\gamma} \n                {}\\geq{} \n            -\\Ei[i][\\m] \\frac{\\|\\nabla f(x_{i})\\|^2}{2} {}+{}\n             \\Ei[i+1][\\m] \\frac{\\|\\nabla f(x_{i+1})\\|^2}{2} {}+{}            \n            \\\\ {}+{}\n            \\frac{{-1} +(\\m+\\n) \\Ei[i+1][\\m]}{\\m-\\n} \\frac{\\|\\nabla f(x_{i+1})-\\m \\nabla f(x_{i})\\|^2}{2}.\n    \\end{aligned}\n\\end{align}%\nFurthermore, if $\\mu \\in (0,L)$ and $\\gamma L \\in \\big(0, \\gbari[i+1]\\big)$, then ${-1} +(\\m+\\n) \\Ei[i+1][\\m] > 0$.%\n\\end{lemma}\n\n\\begin{proof}%\nFirstly, note that $\\m=1-\\gamma \\mu > 0$; the case $\\gamma L \\in \\big(1, \\frac{2}{1+[\\kappa]_{+}}\\big)$ is covered by \\cref{prop:properties:m+n>0}, while for $\\gamma L \\in (0,1]$ we have $1-\\gamma \\mu > 1-\\gamma L \\geq 0$. Let:%\n$$\n    \\beta_{i+1}(\\m) {}{}\\coloneqq{}{} {-1} + \\m \\Ei[i+1][\\m] {}={} \\frac{1 + \\m + \\dots + \\m^{2i}}{\\m^{2i+1}} > 0.\n$$%\nInequality \\eqref{eq:dist_1_mu_1_step_decrease} results after performing the simplifications in the linear combination of interpolation inequalities:%\n$$\n\\Iij[i][i+1] + \\beta_{i+1}(\\m) \\, (\\Iij[i][i+1] + \\Iji[i][i+1]),$$\nwhere \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} equivalently rewrite for the pair $(i,i+1)$ as:\n\\begin{align*}\n    (1+ \\beta_{i+1}) {\\,} \\Iij[i][i+1] {}:{}\\,{}{} \n        \\frac{f_{i} {-} f_{i+1}}{\\gamma}\n            {}\\geq{} &\n        \\frac{\\|g_{i}\\|^2}{2} {+} \\frac{1}{\\m} \\frac{\\|g_{i+1}\\|^2}{2} {+} \\frac{\\frac{\\n}{\\m}}{\\m{-}\\n} \\frac{\\|g_{i+1}{-}\\m g_{i}\\|^2}{2}; \\\\    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n     \\beta_{i+1} {\\,} \\Iji[i][i+1] {}:{}\\,{}{}\n        \\frac{f_{i+1} {-} f_{i}}{\\gamma}\n            {}\\geq{} &\n        {-}(1{+}\\m) \\frac{\\|g_{i}\\|^2}{2} {+} \\frac{1}{\\m{-}\\n} \\frac{\\|g_{i+1}{-}\\m g_{i}\\|^2}{2}.\n\\end{align*}%\nCondition $\\gamma L < \\gbari[i+1]$ implies $\\Tinm[i+1] < 0$, hence $i \\geq \\Nbar$ (recall \\cref{def:N_bar}: $\\Tinm[\\bar{N}] \\geq 0$ and $\\Tinm[\\bar{N}+1] < 0$, corresponding to $\\gamma L < \\gbari[\\bar{N}+1]$). Then the coefficient of the mixed term in \\eqref{eq:dist_1_mu_1_step_decrease} is positive:%\n$$\n    \\frac{{-1} {+} (\\m{+}\\n) E_{i+1}(\\m)}{\\m{-}\\n} \n        {}\\geq{} \n    \\frac{{-1} {+} (\\m{+}\\n) E_{\\bar{N}+1}(\\m)}{\\m{-}\\n}\n        {}={}\n    \\frac{\\Tinm[\\bar{N}]{-}\\n^2 \\Tinm[\\bar{N}+1]}{(\\m{-}\\n)^2} \n        {}>{} 0.\n$$\nThe first inequality results from the monotonic increase with index $i$ of $E_{i+1}(\\m)$, whereas the last one by the definition of $\\Nbar$.%\n\\qed \\end{proof}\n\n\\begin{lemma}[Central distance-2 result]\\label{lemma:telescoped_main_lemma}\nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty, L)$. Consider $k+1$ iterations of \\eqref{eq:GM_it_fixed_steps} with constant stepsize $\\gamma L \\in \\big[\\gbari[k], \\gbari[\\infty] \\big)$ starting from $x_0$, where $k \\geq 1$. Then it holds:%\n\\begin{align}\n        & \\frac{f(x_{0}) {-} f(x_{k})}{\\gamma} +\n        \\frac{(\\m{+}\\n)\\Tinm[k]}{2(\\m{-}\\n)} \\frac{f(x_{k}) {-} f(x_{k+1})}{\\gamma}\n            \\geq\n        % \\\\ & {}\\qquad{}\n        \\left(\\Ei[k][\\m] {+} \\Ei[k][\\n]\\right) \\frac{\\|\\nabla f(x_{k})\\|^2}{4} +\n        \\nonumber \\\\ & {}\\quad{}\\,\n        {}+{}\\frac{\\Tinm[k]}{2(\\m{-}\\n)} \\Big[ \\m \\n \\frac{\\|\\nabla f(x_{k})\\|^2}{2} + \n        \\langle \\nabla f(x_{k}) , \\nabla f(x_{k+1}) \\rangle + \n        \\frac{\\|\\nabla f(x_{k+1})\\|^2}{2} \\Big]. \\label{eq:telescoped_main_lemma} \\tag{D2}\n\\end{align}%\n\\end{lemma}\n\n\\begin{proof}\nConsider two iterations connecting $x_{i}$, $x_{i+1}$ and $x_{i+2}$, with $i \\geq 0$. The interpolation inequalities \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} written for distance-1 (\\Iij[i][i+1] and \\Iji[i][i+1]) and distance-2 (\\Iij[i][i+2] and \\Iji[i][i+2]), respectively:%\n\\begin{align*}\n    &\\begin{aligned}    \n    % & \\alpha_{[i-1>i]} \\times \n    \\Iij[i][i+1]\\colon \n        \\frac{f_{i} - f_{i+1}}{\\gamma}\n            {}\\geq{}\n        (1+\\n) \\frac{\\|g_{i}\\|^2}{2} + \\frac{1}{\\m-\\n} \\frac{\\|g_{i+1}-\\n g_{i}\\|^2}{2}; \n    \\end{aligned}\\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    &\\begin{aligned}    \n    % & \\alpha_{[i-1<i]}  \\times \n    \\Iji[i][i+1]\\colon \n        \\frac{f_{i+1} - f_{i}}{\\gamma}\n            {}\\geq{}\n        -\\frac{\\|g_{i}\\|^2}{2} - \\frac{1}{\\n} \\frac{\\|g_{i+1}\\|^2}{2} + \\frac{\\m}{\\n} \\frac{1}{\\m-\\n} \\frac{\\|g_{i+1}-\\n g_{i}\\|^2}{2}; \n    \\end{aligned}\\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    &\\begin{aligned}    \n    &% \\alpha_{[i-1>i+1]}  \\times \n    \\Iij[i][i+2]\\colon \n        \\frac{f_{i} - f_{i+2}}{\\gamma}\n            {}\\geq{}\n        (1+\\n+\\n^2) \\frac{\\|g_{i}\\|^2}{2} + \\n \\frac{\\|g_{i+1}\\|^2}{2} + \n        \\frac{1}{\\n(\\m-\\n)} \\cdot\n        \\\\ & {}\\qquad{}        \n        \\Big[\n        \\n(1-\\m) \\frac{\\|g_{i+1}-\\n g_{i}\\|^2}{2}\n        - (1-\\n) \\frac{\\|g_{i+2}-\\n g_{i+1}\\|^2}{2}\n        + \\frac{\\|g_{i+2}-\\n^2 g_{i}\\|^2}{2} \n        \\Big];\n    \\end{aligned}\\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    &\\begin{aligned}    \n    &% \\alpha_{[i-1<i+1]}  \\times \n    \\Iji[i][i+2]\\colon\n        \\frac{f_{i+2} - f_{i}}{\\gamma}\n            {}\\geq{} \n        - \\frac{\\|g_{i}\\|^2}{2} - \\frac{1}{\\n} \\frac{\\|g_{i+1}\\|^2}{2}\n        - \\frac{1+\\n}{\\n^2} \\frac{\\|g_{i+2}\\|^2}{2} + \\frac{1}{\\n(\\m-\\n)} \\cdot\n        \\\\ & {}\\qquad{} \n        \\Big[\n        \\m(1-\\n) \\frac{\\|g_{i+1}-\\n g_{i}\\|^2}{2}\n        - (1-\\m) \\frac{\\|g_{i+2}-\\n g_{i+1}\\|^2}{2}\n        + \\frac{\\m}{\\n} \\frac{\\|g_{i+2}-\\n^2 g_{i}\\|^2}{2} \n        \\Big],\n    \\end{aligned}\n\\end{align*}%\nare summed up in a linear combination with \\textit{nonnegative} corresponding weights:%\n$$\n        \\eqref{eq:dist_2_atom_ineq} \\equiv\n\\alphaij[i][i+1] \\Iij[i][i+1] +\n\\alphaij[i+1][i] \\Iji[i][i+1] +\n\\alphaij[i][i+2] \\Iij[i][i+2] +\n\\alphaij[i+2][i] \\Iji[i][i+2],\n$$%\nwhere:%\n$$\n\\begin{aligned}\n    \\alphaij[i][i+1] {}{}\\coloneqq{}{}& \n    1 - \\n \\Ei[i+1][\\n] + \\frac{(1-\\m) \\Tinm[i]}{2(\\m-\\n)} - \\frac{\\m(1+\\n) \\Tinm[i+1]}{2(\\m-\\n)};\n    \\\\\n   \\alphaij[i+1][i] {}{}\\coloneqq{}{}& \n    -\\n \\Ei[i+1][\\n] + \\frac{(1+\\n) \\Tinm[i]}{2(\\m-\\n)} + \\frac{\\n(1-\\m) \\Tinm[i+1]}{2(\\m-\\n)};\n    \\\\\n    \\alphaij[i][i+2] {}{}\\coloneqq{}{}&\n     \\frac{\\m \\Tinm[i+1]}{2(\\m-\\n)}; \n     \\\\\n    \\alphaij[i+2][i] {}{}\\coloneqq{}{}&\n    \\frac{-\\n \\Tinm[i+1]}{2(\\m-\\n)},\n\\end{aligned}%\n$$%\nand \\eqref{eq:dist_2_atom_ineq} is obtained after performing the simplifications:%\n\\begin{align}\\label{eq:dist_2_atom_ineq}    \n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\begin{aligned}\n        &\n        \\frac{f_{i} {-} f_{i+1}}{\\gamma} + \n        (\\m{+}\\n) \\Big[\n        {-}\\frac{\\Tinm[i]}{2(\\m{-}\\n)} \\frac{f_{i} {-} f_{i+1}}{\\gamma} +\n           \\frac{\\Tinm[i+1]}{2(\\m{-}\\n)} \\frac{f_{i+1} {-} f_{i+2}}{\\gamma}\n        \\Big]\n            {}\\geq{}\n        \\\\ & {}\\,{} \n        {}{-}{}\\frac{\\Tinm[i]}{2(\\m{-}\\n)} \\Big[ \\m \\n \\frac{\\|g_{i}\\|^2}{2} + \n        \\langle g_{i}, g_{i+1} \\rangle + \\frac{\\|g_{i+1}\\|^2}{2}  \\Big] \n        {-}\\left(\\Ei[i][\\m] {+} \\Ei[i][\\n]\\right) \\frac{\\|g_{i}\\|^2}{4}\n        \\\\ & {}\\,{} \n        {}{+}{}\\frac{\\Tinm[i+1]}{2(\\m{-}\\n)} \\Big[ \\m \\n \\frac{\\|g_{i+1}\\|^2}{2} \n        {+} {\\langle g_{i+1}, g_{i+2} \\rangle} + \\frac{\\|g_{i+2}\\|^2}{2}  \\Big] {}+{} \\left(\\Ei[i+1][\\m] {+} \\Ei[i+1][\\n]\\right) \\frac{\\|g_{i+1}\\|^2}{4}.\n    \\end{aligned}\n\\end{align}%\n\\noindent{}Telescoping it for indices $i=0,1,\\dots,k-1$ yields inequality \\eqref{eq:telescoped_main_lemma}. It remains to show the nonnegativity of multipliers. Since the stepsize thresholds increase with index $i$, condition $\\gamma L \\geq \\gbari[k]$ ensures $\\Tinm[i+1] \\geq 0$, for all $i=0,{\\dots},k-1$. This directly implies the positivity of $\\alphaij[i][i+2]$ and $\\alphaij[i+2][i]$. The positivity of distance-1 multipliers is evidenced by their equivalent representations as nonnegative sums (recall $\\n \\in (-1,0)$, $\\m \\in \\left(0,\\infty\\right)$; $\\m+\\n>0$ and $\\n^{-2} > \\m^{-2}$ from \\cref{prop:properties:m+n>0}):%\n$$\n\\begin{aligned}\n    &\\begin{aligned}    \n    2\\alphaij[i][i+1] {}={}& \n    1 + \\frac{\\m+\\m^{-2i}}{1+\\m} + \n    \\frac{(1+\\n)(\\m+\\n-\\n\\m)}{\\m \\n^2} \\frac{{-1} +\\n^{-2(i+1)}}{{-1} +\\n^{-2}} + \n    \\\\& {}\\qquad\\qquad{}\n    \\frac{(1+\\n)(1+\\m^2)}{\\m(\\m-\\n)} \n    % \\Big[ \\frac{1-\\n^{-2i}}{1-\\n^{-2}} - \\frac{1-\\m^{-2i}}{1-\\m^{-2}} \\Big] \n    \\sum_{j=0}^{i} \\big(\\n^{-2j} - \\m^{-2j}\\big)\n    {}>{} 0;  \n    \\end{aligned} \\\\%\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    &\\begin{aligned}  \n    2\\alphaij[i+1][i] {}={}&\n    \\frac{{-1} +\\n^{-2i}}{1-\\n} + \n    \\frac{\\m(1+\\n)}{1-\\n} \\frac{{-1} +\\m^{-2(i+1)}}{1-\\m}\n    + \\\\& {}\\qquad\\qquad{}\n     +\\frac{-\\n \\big[1-\\n+\\m(1+\\n)\\big]}{1-\\n} \\frac{\\n^{-2(i+1)} - \\m^{-2(i+1)}}{\\m-\\n} > 0. \n    % {}\\qquad\\qquad\\qquad\\qquad\\qquad{}\n    {}\\qquad{}\n    {}\\qed{}\n    \\end{aligned}%\n\\end{aligned}\n$$%\n\\end{proof}\n\n\\begin{lemma}[Multistep descent]\\label{lemma:hypo_GN4SD}\nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty, L)$, and consider $k+1 {}\\geq{} 1$ iterations of \\eqref{eq:GM_it_fixed_steps} with stepsize $\\gamma L \\in [\\gbari[k], \\gbari[\\infty])$, {\\shownewtext{satisfying $\\gamma L > 1$,}} starting from $x_0$. Then the following inequality holds:%\n\\begin{align}%\n% \\hspace{-.25em}\n    \\frac{f(x_0) {-} f(x_{k+1})}{\\gamma}\n        {}\\geq{}\n    \\frac{{-}\\m^2 \\n^2 \\Tinm[k+1]}{\\m^2{-}\\n^2} \\frac{\\|\\nabla f(x_{k})\\|^2}{2} {}+{}\n    \\frac{\\Tinm[k]{+}(\\m{-}\\n)}{\\m^2{-}\\n^2} \\frac{\\|\\nabla f(x_{k+1})\\|^2}{2}.\n\\label{eq:general_ineq_proof} \\tag{GN4SD}\n% \\tag{$I_{N+1}^{N}$}\n\\end{align}%\nFurthermore, if $\\mu \\in (-\\infty, 0]$ and $\\gamma L \\in [\\gbari[k], \\gbari[k+1])$, then the gradient norms from \\eqref{eq:general_ineq_proof} are weighted by nonnegative quantities.\n\\end{lemma}\n\n\\begin{proof}%\nFor $k=0$, the result follows from \\cref{lemma:sufficient_decrease_h_geq_1} and we obtain exactly \\eqref{eq:one_step_rate_h_geq_1}. \nFor $k\\geq 1$, inequality \\eqref{eq:general_ineq_proof} is derived by adjusting \\eqref{eq:telescoped_main_lemma} through the linear combination of interpolation inequalities $\\alphaij[k][k+1] \\Iij[k][k+1] + \\alphaij[k+1][k] \\Iji[k][k+1]$:%\n$$\n    \\begin{aligned}\n        &  \\Big[1 - \\frac{(\\m+\\n)\\,\\Tinm[k]}{2(\\m-\\n)} \\Big] \\frac{f_{k} - f_{k+1}}{\\gamma}\n        {}\\geq{}\n        \\\\ & {}\\qquad{}\n        \\Big(-\\Ei[k][\\m] - \\Ei[k][\\n] - \\frac{\\m \\n \\Tinm[k]}{\\m-\\n} {}-{} \\frac{2\\m^2\\n^2 \\Tinm[k+1]}{\\m^2-\\n^2} \\Big) \\frac{\\|g_{k}\\|^2}{4}\n        \\\\ & {}\\qquad{}        \n        % \\\\ & {}\\qquad{}\n        {}+{} \\Big(\\frac{\\m-\\n + \\Tinm[k]}{\\m^2-\\n^2} - \\frac{\\Tinm[k]}{2(\\m-\\n)} \\Big) \\frac{\\|g_{k+1}\\|^2}{2}\n        {}-{}\\frac{\\Tinm[k]}{2(\\m-\\n)} \\langle g_{k},g_{k+1} \\rangle,\n    \\end{aligned}\n$$%\nwhere the corresponding weights are defined as follows:%\n    $$\n    \\begin{aligned}\n        \\alphaij[k][k+1]\n             {}{}\\coloneqq{}{} &\n        1 + \\frac{-\\n}{\\m+\\n} + \\frac{(1-\\m)(\\m+\\n)-2\\n}{2(\\m^2-\\n^2)} \\Tinm[k];  \\\\\n        %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n        \\alphaij[k+1][k]\n            {}{}\\coloneqq{}{} &\n        \\frac{-\\n}{\\m+\\n} + \\frac{(1+\\n)(\\m+\\n)-2\\n}{2(\\m^2-\\n^2)} \\Tinm[k].\n    \\end{aligned}\n    $$ %\n    Multiplier $\\alphaij[k+1][k]$ is represented as a sum of positive quantities ($\\n < 0$).\n    Multiplier $\\alphaij[k][k+1]$ is directly written as sum of positive quantities if $(1-\\m)(\\m+\\n)-2\\n > 0$ (in particular, this covers the (strongly) convex case where $\\m \\leq 1$). Otherwise, $\\alphaij[k][k+1]$ is equivalently expressed as a sum of positive terms as follows, with $\\m>1$:%\n$$\n\\begin{aligned}\n    \\alphaij[k][k+1]\n        {}={}&\n    \\frac{\\m}{2(\\m+\\n)} + \\frac{1}{2(\\m^2-\\n^2)} \n        \\bigg[ \\frac{-\\n(\\m-\\n)(1+\\m^2)}{(\\m-1)(1-\\n)} {}+{}\n        \\\\ & {}\\quad{} \n        -\\big[(1-\\m)(\\m+\\n)-2\\n\\big] \\Big( \\frac{\\m^{-2k}}{\\m-1} + \\frac{\\n^{-2k}}{1-\\n} \\Big)  \\bigg]\n        {}>{} 0. {}\\quad{} \\qed\n\\end{aligned}\n$$\n\\end{proof}\n\n\\begin{lemma}\\label{lemma:general_lemma_ct_h_to_Linear} \nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty, L)$, and consider $k+1 \\geq 1$ iterations of \\eqref{eq:GM_it_fixed_steps} with stepsize $\\gamma L \\in [\\gbari[k], \\gbari[\\infty] \\big)$, satisfying $\\gamma L > 1$, starting from $x_0$. Then the following inequality holds:%\n\\begin{align}\\label{eq:general_ineq_proof_to_linear_compact}\n   \\frac{f(x_0) {-} f(x_{k+1})}{\\gamma}\n        {}\\geq{} \n    \\Ei[k+1][\\n] \\frac{\\|\\nabla f(x_{k+1})\\|^2}{2} {+} \n    \\frac{\\m^2 \\Tinm[k+1]}{(\\m{-}\\n)^2}\n    \\frac{\\| \\nabla f(x_{k+1}) {-} \\n \\nabla f(x_{k}) \\|^2}{2}.\n\\end{align}%\nFurthermore, if $\\gamma L \\in [\\gbari[k+1], \\gbari[\\infty] \\big)$, then $\\Tinm[k+1]\\geq 0 $.%\n\\end{lemma}\n\n\\begin{proof}%\nFor $k=0$, the result follows from \\cref{lemma:dist_1_L_1_step_decrease}. For $k \\geq 1$, inequality \\eqref{eq:general_ineq_proof_to_linear_compact} is derived by adjusting \\eqref{eq:telescoped_main_lemma} with the linear combination of interpolation inequalities $\\alphaij[k][k+1] \\Iij[k][k+1] + \\alphaij[k][k+1] \\Iji[k][k+1]$:%\n$$\n    % \\hspace{-1em}\n    \\begin{aligned}\n        &\\Big[1 - \\frac{(\\m+\\n)\\,\\Tinm[k]}{2(\\m-\\n)} \\Big] \\frac{f_{k} - f_{k+1}}{\\gamma}\n            {}\\geq{}\n        \\Big(-\\frac{\\Tinm[k]}{2(\\m-\\n)} - \\frac{\\m^2 \\n \\Tinm[k+1]}{(\\m-\\n)^2}\\Big) \\langle g_{k},g_{k+1} \\rangle {}+{}\n        \\\\ & {}\\quad\\qquad{}\n        {}+{} \\Big[ \\frac{2\\m^2 \\n^2 \\Tinm[k+1]}{(\\m-\\n)^2} -\\Ei[k][\\m] - \\Ei[k][\\n] - \\frac{\\m \\n \\Tinm[k]}{\\m-\\n} \\Big] \\frac{\\|g_{k}\\|^2}{4} {}+{}\n        \\\\ & {}\\quad\\qquad{}\n        {}+{} \\Big[ \\frac{\\m^2 \\, \\Tinm[k+1]}{(\\m-\\n)^2}  + \\Ei[k+1][\\n] -\\frac{\\Tinm[k]}{2(\\m-\\n)} \\Big] \\frac{\\|g_{k+1}\\|^2}{2},\n    \\end{aligned}\n$$%\nwhere the corresponding weights are defined as:\n$$\n\\begin{aligned}\n    \\alphaij[k][k+1] \n        {}{}\\coloneqq{}{} &\n    1+(-\\n)\\Ei[k+1][\\n] + \\frac{1-\\m}{2(\\m-\\n)} \\Tinm[k];\n    \\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\alphaij[k+1][k]\n        {}{}\\coloneqq{}{} &\n    (-\\n) \\Ei[k+1][\\n] + \n    \\frac{1+\\n}{2(\\m-\\n)} \\Tinm[k].\n\\end{aligned}%\n$$%\nMultiplier $\\alphaij[k+1][k]$ is represented as a sum of positive quantities, while $\\alphaij[k][k+1]$ is directly written as sum of positive terms if $\\m \\leq 1$ (the (strongly) convex case). Otherwise, $\\m > 1$ and $\\alphaij[k][k+1]$ is equivalently expressed as sum of positive terms as:%\n$$\n        \\alphaij[k][k+1]\n    {}={}\n        \\frac{1+(-\\n) \\n^{-2(k+1)}}{2(1-\\n)} + \\m \\frac{\\m \\m^{-2(k+1)} + (-\\n) \\n^{-2(k+1)}}{2(\\m-\\n)}. {}\\quad{} \\qed\n$$\n\\end{proof}\n\n\\begin{lemma}\\label{lemma:general_lemma_ct_h_to_Linear_mu} \nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty,L)$, and consider $k+1 \\geq 1$ iterations of \\eqref{eq:GM_it_fixed_steps} with stepsize $\\gamma L \\in [\\gbari[k], \\gbari[\\infty])$, satisfying $\\gamma L > 1$, starting from $x_0$. Then the following inequality holds:%\n\\begin{align}\\label{eq:general_ineq_proof_to_linear_mu_compact}\n% \\hspace{-.75em}\n    \\frac{f(x_0) {-} f(x_{k+1})}{\\gamma}\n        {\\geq}\n    \\Ei[k+1][\\m] \\frac{\\|\\nabla f(x_{k+1})\\|^2}{2} {+} \n    \\frac{-\\n^2 \\Tinm[k+1]}{(\\m{-}\\n)^2}\n    \\frac{\\| \\nabla f(x_{k+1}) {-} \\m \\nabla f(x_{k}) \\|^2}{2}.\n\\end{align}%\nFurthermore, if $\\gamma L \\in [\\gbari[k], \\gbari[k+1] \\big)$, then $\\Tinm[k+1]<0$.\n\\end{lemma}\n\n\\begin{proof}\nCase $k=0$ results from \\cref{lemma:dist_1_mu_1_step_decrease}; further on, $k\\geq 1$. Inequality \\eqref{eq:direct_sum_interp_cond} written for the pair $(i,j)=(k,k+1)$ reads:%\n   \\begin{align}\\label{eq:dist_1_correction_to_mu}\n       0\n        {}\\geq{}\n       \\frac{\\m \\n}{\\m-\\n} \\|g_{k}\\|^2 + \\frac{1}{\\m-\\n} \\|g_{k+1}\\|^2 - \\frac{\\m+\\n}{\\m-\\n} \\langle g_{k} \\,,\\, g_{k+1} \\rangle.\n   \\end{align}%\n   \\begin{enumerate}[align=right,itemindent=1em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]\n       \\item \\textbf{Case $\\gamma L \\in [\\gbari[k], \\gbari[k+1])$.} By using $\\m>0$, \\eqref{eq:dist_1_correction_to_mu} equivalently rewrites as:%\n   $$\n       0\n        {}\\geq{}\n       -\\m^2 \\frac{\\|g_{k}\\|^2}{2} + \\frac{\\|g_{k+1}\\|^2}{2} + \\frac{\\m+\\n}{\\m-\\n} \\frac{\\|g_{k+1}-\\m g_{k}\\|^2}{2}.\n   $$%\n   By multiplying it with $\\frac{-\\n^2}{\\m^2-\\n^2}\\Tinm[k+1]$ (positive because $\\Tinm[k+1] < 0$) and adding it to \\eqref{eq:general_ineq_proof}, the coefficient of $\\|g_{k}\\|^2$ is canceled and we get:%\n    $$\n    \\frac{f_0{-}f_{k+1}}{\\gamma}\n        {}{\\geq}{}\n    \\frac{\\Tinm[k]{+}(\\m{-}\\n){-}\\n^2 \\Tinm[k+1]}{\\m^2{-}\\n^2} \\frac{\\|g_{k+1}\\|^2}{2} {}+{}\n       \\frac{{-}\\n^2 \\Tinm[k+1]}{(\\m{-}\\n)^2}  \\frac{\\|g_{k+1}{-}\\m g_{k}\\|^2}{2},\n    $$\n    which after simplifications is exactly \\eqref{eq:general_ineq_proof_to_linear_mu_compact}.%\n    %\n    \\item \\textbf{Case $\\gamma L \\in \\big[\\gbari[k+1],\\gbari[\\infty] \\big)$.} After multiplying \\eqref{eq:dist_1_correction_to_mu} with $\\frac{-\\n \\m}{\\m-\\n} > 0$:\n    $$\n        0\n            {}\\geq{}\n        \\frac{\\|g_{k+1}\\|^2}{2} + \n        \\frac{-\\m^2}{(\\m-\\n)^2} \\frac{\\|g_{k+1}-\\n g_{k}\\|^2}{2} +\n        \\frac{-\\n^2}{(\\m-\\n)^2} \\frac{\\|g_{k+1}-\\m g_{k}\\|^2}{2}.\n    $$\n    By scaling it with $\\Tinm[k+1]>0$ and summing it with \\eqref{eq:general_ineq_proof_to_linear_compact}, the coefficient of $\\|g_{k+1} - \\n g_{k}\\|^2$ is canceled and after simplifications we get \\eqref{eq:general_ineq_proof_to_linear_mu_compact}.%\n    {}\\qed{}%\n    \\end{enumerate}%\n\\end{proof}\n\n\\begin{proof}[\\textbf{\\cref{thm:strg_cvx_rate}}]\\label{proof:thm:strg_cvx_rate}%\nWe split the proof over stepsize intervals:%\n\\[\n    (0,2) \n        {}={}\n    \\big(0, \\gbari[1]\\big] \\cup\n    \\Big( \\mathop{\\cup}\\limits_{k=1}^{N-1} \\big[\\gbari[k],\\gbari[k+1]\\big)\\Big)\n    {}\\cup{} \\big[\\gbari[N],\\tfrac{2}{1+\\kappa}\\big) \n    {}\\cup{} \\big[\\tfrac{2}{1+\\kappa}, 2\\big).\n\\]%\n%\n\\begin{enumerate}[align=right,itemindent=1em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]%\n\\item \\textbf{Case $\\gamma L \\in \\big(0, \\gbari[1]\\big]$}. By telescoping \\eqref{eq:dist_1_mu_1_step_decrease} for indices $i=0,\\dots,N-1$:%\n$$\n        \\frac{f_{0} - f_{N}}{\\gamma} \n            {}\\geq{}\n        \\Ei[N][\\m] \\frac{\\|g_{N}\\|^2}{2} {}+{}        \n            \\sum_{i=0}^{N-1}\n            \\frac{ {-1} + (\\m+\\n) \\Ei[i+1][\\m] }{\\m-\\n} \\frac{\\|g_{i+1}-\\m g_{i}\\|^2}{2},\n$$%\nwhere all mixed terms have nonnegative coefficients because $\\gamma L \\in \\big(0,  \\gbari[1]\\big]$ (see \\cref{lemma:dist_1_mu_1_step_decrease}). Hence they can be neglected to obtain the lower bound of type \\eqref{eq:I^k_N_general_ineq}:\n\\begin{align}\\label{eq:sufficient_decrease_N_its_mu_strongly_convex_towards_rate_no_dist_2}\n    \\frac{f_0 - f_N}{\\gamma}\n        {}\\geq{}\n    \\Ei[N][\\m] \\frac{\\|g_{N}\\|^2}{2}.\n\\end{align}%:\n\\item \\textbf{Case $\\gamma L \\in \\big[\\gbari[k],\\gbari[k+1]\\big)$}, with some $k \\in \\{1,\\dots,N-1\\}$. By telescoping \\eqref{eq:dist_1_mu_1_step_decrease} for indices $i=k+1,\\dots,N-1$:%\n$$\n\\begin{aligned}\n        \\frac{f_{k+1} {-} f_{N}}{\\gamma} \n            {}\\geq{} &\n        {-}\\Ei[k+1][\\m] \\frac{\\|g_{k+1}\\|^2}{2} {+}\n        \\Ei[N][\\m] \\frac{\\|g_{N}\\|^2}{2} {+} \\\\\n        & \\quad     \\sum_{i{=}k{+}1}^{N{-}1}\n            \\frac{ {-1} {+} (\\m{+}\\n) \\Ei[i{+}1][\\m] }{\\m{-}\\n} \\frac{\\|g_{i+1}{-}\\m g_{i}\\|^2}{2},\n\\end{aligned}\n$$%\nwith nonnegative weights of the mixed terms since $\\gamma L \\in \\big(0,  \\gbari[k+1]\\big)$ (see \\cref{lemma:dist_1_mu_1_step_decrease}). After appending it to \\eqref{eq:general_ineq_proof_to_linear_mu_compact} from \\cref{lemma:general_lemma_ct_h_to_Linear_mu}, we obtain:%\n$$\n\\begin{aligned}\n    \\frac{f_0 {-} f_N}{\\gamma}\n        {}\\geq{}\n    \\Ei[N][\\m] \\frac{\\|g_{N}\\|^2}{2} +\n    \\frac{-\\n^2 \\, \\Tinm[k+1]}{(\\m-\\n)^2} \\, \n    \\frac{\\| g_{k+1} - \\m g_{k} \\|^2}{2} +     \\\\\n    \\sum_{i=k+1}^{N-1}\n            \\frac{ {-1} + (\\m+\\n) \\Ei[i+1][\\m] }{\\m-\\n} \\frac{\\|g_{i+1}-\\m g_{i}\\|^2}{2}.\n\\end{aligned}\n$$\nSince $\\Tinm[k+1] < 0$, all mixed terms can be neglected to get a \\eqref{eq:I^k_N_general_ineq}-type inequality:%\n\\begin{align}\\label{eq:sufficient_decrease_N_its_mu_strongly_convex_towards_rate}\n    \\frac{f_0 - f_N}{\\gamma}\n        {}\\geq{}\n    \\Ei[N][\\m] \\frac{\\|g_{N}\\|^2}{2}.\n\\end{align}%\n%\n \\item \\textbf{Case $\\gamma L \\in \\big[\\gbari[N],\\frac{2}{1+\\kappa}\\big)$.} From \\cref{lemma:general_lemma_ct_h_to_Linear} with $k=N-1$ it holds:%\n$$\n   \\frac{f_0 - f_N}{\\gamma}\n        {}\\geq{}\n    \\Ei[N][\\n] \\frac{\\|g_N\\|^2}{2} + \n    \\frac{\\m^2 \\, \\Tinm[N]}{(\\m-\\n)^2} \\, \n    \\frac{\\| g_N - \\n g_{N-1} \\|^2}{2}.\n$$%\nSince $\\TigLmu[N] \\geq 0$, by neglecting the mixed term we obtain a \\eqref{eq:I^k_N_general_ineq}-type inequality:\n\\begin{align}\\label{eq:wc_linear_L}\n    \\frac{f_0 - f_N}{\\gamma}\n        {}\\geq{}\n    \\Ei[N][\\n] \\frac{\\|g_{N}\\|^2}{2}.\n\\end{align}%\n%\n\\item \\textbf{Case $\\gamma L \\in \\big[\\frac{2}{1+\\kappa},2\\big)$.} By telescoping \\cref{lemma:dist_1_L_1_step_decrease} for $i=0,1,\\dots,N-1$ it holds:%\n$$\n    \\frac{f_{0} - f_{N}}{\\gamma} \n                {}\\geq{}\n            \\Ei[N][\\n] \\frac{\\|g_{i}\\|^2}{2} +\n            \\sum_{i=0}^{N-1} \\frac{ 1 - (\\m+\\n) \\Ei[i+1][\\n] }{\\m-\\n} \\frac{\\|g_{i+1}-\\n g_{i}\\|^2}{2},\n$$%\nwhere the mixed terms can be neglected and then the inequality reduces to \\eqref{eq:wc_linear_L}.%\n\\end{enumerate}%\nSince the transition between the two linear regimes is given by the sign of $\\Tinm[N]=\\Ei[N][\\m]{-}\\Ei[N][\\n]$, we can glue together expressions \\eqref{eq:sufficient_decrease_N_its_mu_strongly_convex_towards_rate_no_dist_2}, \\eqref{eq:sufficient_decrease_N_its_mu_strongly_convex_towards_rate} and \\eqref{eq:wc_linear_L}:%\n$$\n    \\frac{f_0 - f_N}{\\gamma}\n        {}\\geq{}\n    \\min\\left\\{ \\Ei[N][\\m] \\,,\\, \\Ei[N][\\n]\\right\\} \\frac{\\|g_{N}\\|^2}{2},\n$$%\nequivalent to:\n$$\n    f_0 - f_N\n        {}\\geq{}\n    \\min\n    \\Big\\{ \\frac{{-1} +(1-\\gamma \\mu)^{-2N}}{\\mu} \\,,\\, \n    \\frac{{-1} +(1-\\gamma L)^{-2N}}{L}\n    \\Big\\} \\frac{\\|g_{N}\\|^2}{2},\n$$%\nwhich implies the bound \\eqref{eq:strg_cvx_full_ct} and the complementary one:\n$$\n\\begin{aligned}\n    \\frac{1}{2L} \\|\\nabla f(x_N)\\|^2\n        {} \\leq {}  \n    \\frac{f(x_0)-f(x_N)}\n    { \\min\\big\\{ \\frac{L}{\\mu} \\left[{-1}+(1-\\gamma \\mu)^{-2N}\\right] \\,,\\, {-1}+(1-\\gamma L)^{-2N} \\big\\} }. {}\\qquad{}\\qed \n\\end{aligned}%\n$$%\n\\end{proof}\n\n\\begin{proof}[\\textbf{\\cref{prop:monotonicity_Tk}}] \\label{appendix:proof_prop_monotonicity_Tk}\nWe show $\\frac{d\\TigLmu}{d (\\gamma L)} \\geq 0$ for $\\gamma L \\in (1, \\gbari[\\infty])$, which resumes to prove the positivity of\n  \\begin{align*}\n        \\frac{d \\delta_i(\\gamma L, \\kappa)}{d(\\gamma L)}\n            {}{}\\coloneqq{}{}\n      \\frac{-\\kappa \\left( 1 - 2i \\frac{2-\\kappa \\gamma L}{1-\\kappa \\gamma L} \\right) }{(1-\\kappa \\gamma L)^{2i}} + \\frac{ 1 + 2i \\frac{2-\\gamma L}{\\gamma L-1}}{(1-\\gamma L)^{2i}}.\n  \\end{align*}\n  \\begin{enumerate}[align=right,itemindent=1em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]\n      \\item \\textbf{Case $\\kappa \\geq 0$:} we lower bound $\\frac{d \\delta_i(\\gamma L, \\kappa)}{d(\\gamma L)}$ by neglecting the denominators:\n      \\begin{align*}\n          \\delta_i(\\gamma L, \\kappa) \n            {}\\geq{} \n          {-}\\kappa \\Big( 1 - 2i \\frac{2-\\kappa \\gamma L}{1-\\kappa \\gamma L} \\Big) + 1 + 2i \\frac{2-\\gamma L}{\\gamma L-1}\n            {}={}\n          (1-\\kappa) + 2i \\Big[ \\frac{\\kappa (2-\\kappa \\gamma L)}{1-\\kappa \\gamma L}  + \\frac{2-\\gamma L}{\\gamma L-1} \\Big] {}\\geq{} 0.\n      \\end{align*}%\n        %\n      \\item \\textbf{Case $\\kappa < 0$:} it is sufficient to show \n    \\begin{align*}\n        \\frac{1+2i \\frac{2-\\gamma L}{\\gamma L -1}}{(1-\\gamma L)^{2i}}\n            {}\\geq{} 1 {}\\geq{}\n        -\\kappa \\frac{-1+2i (1 + \\frac{1}{1-\\kappa \\gamma L})}{(1-\\kappa \\gamma L)^{2i}}.\n    \\end{align*}\n      The \\lhs holds because $\\gamma L \\in (1,2)$ and $(1-\\gamma L)^{2i}<1$. The \\rhs is equivalent to:%\n      \\begin{align*}\n          (1-\\kappa \\gamma L)^{2i} {}\\geq{} -\\kappa \\Big[-1+2i \\Big(1 + \\frac{1}{1-\\kappa \\gamma L}\\Big)\\Big].\n      \\end{align*}\nUsing $\\gamma L \\in (1,2)$ and the second order Taylor approximation for $(1-\\kappa)^{2i}$, we get the bounds:%\n      \\begin{align*}\n          (1-\\kappa \\gamma L)^{2i} {}\\geq{}& (1-\\kappa)^{2i} \n             {}\\geq{} \n         1 + 2i (-\\kappa) + i(2i-1) \\kappa^2 \\\\\n         %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n          -\\kappa \\Big[-1+2i \\Big(1 + \\frac{1}{1-\\kappa \\gamma L}\\Big)\\Big]\n            {}\\leq{}&\n          -\\kappa \\Big[-1+2i \\left(1 + \\frac{1}{1-\\kappa}\\right)\\Big].\n      \\end{align*}\nTherefore, it is sufficient to show:%\n$$\n    1 + 2i (-\\kappa) + i(2i-1) \\kappa^2 {}\\geq{} \n    -\\kappa \\Big[-1+2i \\Big(1 + \\frac{1}{1-\\kappa}\\Big)\\Big],%\n$$%\nwhich equivalently rewrites as the following inequality valid for all $i \\geq 1$:\n$$\n    \\kappa^2 i(i-1) + \\Big( \\kappa i + \\frac{1}{1-\\kappa} \\Big)^2 + \\frac{(1-\\kappa)^3-1}{(1-\\kappa)^2} \\geq 0. {}\\qquad \\qed{}%\n$$%\n\\end{enumerate}\n\\end{proof}\n\n\\begin{lemma} \\label{lemma:unique_delta_k_neg_thr}\nLet $\\kappa \\in (-\\infty, 1)$ and $\\gamma L \\in (1,\\gbari[\\infty])$. Then there exists an index $k$ such that $\\delta_k(\\gamma L, \\kappa) < 0$ and $\\delta_j(\\gamma L, \\kappa) < 0$ for all $j > k$.    \n\\end{lemma}\n\n\\begin{proof}\n    The sequence $\\delta_i$ is a difference of two monotone sequences: $\\frac{2-\\kappa \\gamma L}{(1-\\kappa \\gamma L)^{2i}}$ and $\\frac{2-\\kappa \\gamma L}{(1-\\kappa \\gamma L)^{2i}}$, respectively. The former is decreasing for $\\kappa < 0$ (weakly convex functions) and increasing for $\\kappa \\geq 0$ (convex functions), whereas the later is always decreasing. In particular, $\\delta_0(\\gamma L, \\kappa) > 0$. \n    Because $(1-\\gamma \\mu)^2 {}\\leq{} (1-\\gamma L)^2$, to the limit it holds $\\lim_{k \\rightarrow \\infty} \\delta_k(\\gamma L, \\kappa) = -\\infty$ and there exists some index $k$ with $\\delta_k(\\gamma L, \\kappa) < 0$. We prove by contradiction that $\\delta_j(\\gamma L, \\kappa) < 0$ for all $j > k$; assume there exists some index $l > k$ such that $\\delta_l(\\gamma L, \\kappa) \\geq 0$. The following inequalities hold:%\n\\begin{align*}\n    \\delta_{k} < 0 \\,:\\,& \n        \\frac{2-\\gamma \\mu}{(1-\\gamma \\mu)^{2k}} \n            {}<{} \n        \\frac{2-\\gamma L}{(1-\\gamma L)^{2k}} \\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\delta_{l} > 0 \\,:\\,& \n        \\frac{2-\\gamma \\mu}{(1-\\gamma \\mu)^{2l}} \n            {}>{} \n        \\frac{2-\\gamma L}{(1-\\gamma L)^{2l}}.\n\\end{align*}\nBy multiplying the inequalities and performing the simplifications we get:%\n\\begin{align*}\n    (1-\\gamma \\mu)^{2(k-l)} > (1-\\gamma L)^{2(k-l)},\n\\end{align*}\nwhich implies, due to $(1-\\gamma \\mu)^2 {}>{} (1-\\gamma L)^2$, that $k > l$, contradiction! \\qed\n\\end{proof}\n\n\\begin{proof}[\\textbf{\\cref{prop:properties_step_size_thr}}] \\label{appendix:proof_properties_step_size_thr} \n\\begin{enumerate}[align=right,itemindent=1em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]\n\\item Case $k=0$ holds by definition since $\\gbari[0]=1$. For any $k \\geq 1$, by using \\cref{lemma:unique_delta_k_neg_thr}, condition $\\TigLmu[k][\\gbari[k][]] = 0$ implies $\\delta_k(\\gamma L, \\kappa) \\leq 0$ and hence $\\delta_{k+1}(\\gamma L, \\kappa) < 0$. Furthermore, $\\TigLmu[k+1][\\gbari[k][]] {} \\leq {} \\TigLmu[k][\\gbari[k][]] = 0$. Then it holds:\n\\begin{align*}\n    \\TigLmu[k+1][\\gbari[k][]] {} \\leq {} \n    \\TigLmu[k][\\gbari[k][]] = 0 \n    {}={} \\TigLmu[k+1][\\gbari[k+1][]].\n\\end{align*}\nThe monotonicity of $\\TigLmu$ in stepsize $\\gamma L$ (see \\cref{prop:monotonicity_Tk}) implies $\\gbari[k] < \\gbari[k+1]$.\n\n\\item Because stepsize thresholds sequence is monotone, it has a limit.\n\\begin{enumerate}[align=right,itemindent=1.5em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]\n    %%%%%%%%%%%%%%%%%%%%%%%%%% 2. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\item \n    \\textbf{Case $\\kappa > 0$.} By definition, $\\gbari[k] < \\frac{2}{1+\\kappa} < 2$ for all integers $k\\geq 1$ and condition $\\TigLmu[k][\\gbari[k]]=0$ implies:%\n    \\begin{align*}\n        \\left(\\frac{1-\\gbari}{1-\\kappa \\gbari}\\right)^{2k}\n            {}={}& \n        \\kappa + (1-\\kappa) (1-\\gbari)^{2k}\n            {}\\Leftrightarrow{} \\\\\n            %%%%%%%%%%%%%%%%%%%%%%%%%%%%\n        \\left|\\frac{1-\\gbari}{1-\\kappa \\gbari}\\right|\n            {}={}&\n        \\exp \\left[ \\frac{\\ln \\left(\\kappa + (1-\\kappa) (1-\\gbari)^{2k}\\right)}{2k} \\right].\n    \\end{align*}%\n    To the limit $k \\rightarrow \\infty$, the \\rhs equals $1$ because $(1-\\gbari)^{2k}\\rightarrow 0$, hence $|1-\\gbari|=|1-\\kappa \\gbari|$, which implies the solution $\\gbari[\\infty]=\\frac{2}{1+\\kappa}$. \\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\item \\textbf{Case $\\kappa < 0$.} By definition, $\\gbari < 2$ for all integers $k\\geq 1$ and condition $\\TigLmu[k][\\gbari[k]]=0$ implies:\n    \\begin{align*}\n    (1-\\gbari)^{-2k} \n        {}={}&\n    1 + \\frac{1 - (1-\\kappa \\gbari)^{-2k}}{-\\kappa}\n        {}\\Leftrightarrow{} \\\\\n    % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\left|1-\\gbari\\right| \n        {}={}&\n    \\exp \\Bigg[ \\frac{ -\\ln \\Big(1 + \\frac{1 - (1-\\kappa \\gbari)^{-2k}}{-\\kappa}\\Big)}{2k} \\Bigg].\n\\end{align*}\n To the limit $k \\rightarrow \\infty$, the \\rhs equals $1$ because $(1-\\kappa \\gbari)^{-2k} \\rightarrow 0$ due to $\\kappa < 0$. Hence, $\\left|1-\\gbari\\right|=1$, with the solution $\\gbari[\\infty]=2$.%\n %\n\\item \\textbf{Case $\\kappa = 0$.} By definition, $\\gbari[k][(0)] < 2$ and, similarly to the case $\\kappa < 0$:%\n \\begin{align*}\n     \\left|1-\\gbari\\right|\n        {}={}&\n     \\left[ 1 + 2 k \\gbari \\right]^{\\frac{-1}{2k}}\n        {}={}\n    \\exp \\Big[ \\frac{-\\ln \\big( 1 + 2 k \\gbari \\big) }{2k} \\Big].\n \\end{align*}\n  To the limit $k \\rightarrow \\infty$, the \\rhs equals $1$, hence $\\left|1-\\gbari\\right|=1$, with the solution $\\gbari[\\infty]=2$.\n\\end{enumerate}%\n%\n\\item Condition $\\gamma L \\in [\\gbari[k], \\gbari[k+1])$ implies $\\TigLmu[k][\\gbari[k]] \\geq 0$ and $\\TigLmu[k+1][\\gbari[k]] < 0$. Because $\\delta_i(\\gamma L, \\kappa) < 0$ for all $i \\geq k$, we have (i) $\\TigLmu[i][\\gbari[k][]] {}>{} 0$ for $i=1,\\dots,k-1$ (since $k$ is uniquely defined) and (ii) $\\TigLmu[i][\\gbari[k][]] {}<{} 0$ for $i\\geq k+1$. {}\\qed{}%\n\\end{enumerate}\n\\end{proof}\n\n\\begin{proof}[\\textbf{\\cref{thm:interp_hypo}}]%\n    The proof follows the same steps outlined in \\cite[Theorem 4]{taylor_smooth_2017}, with a minimal extension to accommodate $\\mu < 0$. This extension impacts only the first step of the demonstration, known as minimal curvature subtraction, which involves shifting the lower curvature of the function to $0$, effectively converting it into a smooth convex function. This adjustment remains valid for $\\mu < 0$, as showed further on. By applying the \\textit{gradient inequality} for smooth convex functions: $h(x) {} \\geq {} h(y) + \\left \\langle \\nabla h(y), x-y \\right \\rangle, \\, \\forall x,y \\in \\mathbb{R}^d$ for $h = \\frac{L}{2}\\|\\cdot\\|^2-f$ and $h = f - \\frac{\\mu}{2}\\|\\cdot\\|^2$, we obtain:%\n    \\begin{align}\\label{eq:def_hypoconvex_smooth}\n    \\tfrac{\\mu}{2} \\|x-y\\|^2 \n        {} \\leq {} \n    f(x) - f(y) - \\langle \\nabla f(y), x-y \\rangle \n        {} \\leq {} \n    \\tfrac{L}{2} \\|x-y\\|^2 \\quad \\forall x, y \\in \\mathbb{R}^d.\n\\end{align}%        \n    Let $\\bar{f} \\in \\mathcal{F}_{0,L-\\mu}$, $\\bar{f}(x){}\\coloneqq{}f(x)-\\tfrac{\\mu}{2}\\|x\\|^2$, with $\\nabla \\bar{f}(x)=\\nabla f(x)-\\mu x$; by replacing $f$ with $\\bar{f}$ in the quadratic bounds inequality \\eqref{eq:def_hypoconvex_smooth} it holds:%\n$$\n    % \\frac{\\mu}{2} \\|x-y\\|^2 {}\\leq{}& f(x) - f(y) - \\langle \\nabla f(y), x-y \\rangle {}\\leq{} \\frac{L}{2} \\|x-y\\|^2 \\nonumber \\\\\n    % & \\frac{\\mu}{2} \\|x-y\\|^2 {}\\leq{} \\big(f(x)-\\frac{\\mu}{2} \\|x\\|^2\\big) - \\big(f(y)-\\frac{\\mu}{2} \\|y\\|^2\\big) - \\langle \\nabla \\big(f(y)-\\mu y\\big), x-y \\rangle {}\\leq{} \\frac{L}{2} \\|x-y\\|^2 \\nonumber \\\\\n    % \\frac{\\mu}{2} \\|x-y\\|^2 {}\\leq{}& \\bar{f}(x) - \\bar{f}(y)- \\langle \\nabla \\bar{f}(y), x-y \\rangle + \\frac{\\mu}{2} \\|x-y\\|^2 {}\\leq{} \\frac{L}{2} \\|x-y\\|^2 \\nonumber \\\\\n    0 {}\\leq{} \\bar{f}(x) - \\bar{f}(y)- \\langle \\nabla \\bar{f}(y), x-y \\rangle {}\\leq{} \\tfrac{L-\\mu}{2} \\|x-y\\|^2.\n$$\nHence, the set $\\{(x_i, g_i, f_i)\\}_{i \\in \\mathcal{I}}$ is $\\mathcal{F}_{\\mu,L}$-interpolable if and only if the set $\\{(x_i, g_i-\\mu x_i, f_i-\\tfrac{\\mu}{2}\\|x_i\\|^2)\\}_{i \\in \\mathcal{I}}$ is $\\mathcal{F}_{0, L-\\mu}$-interpolable. \nSubsequently, the proof continues identically to \\cite[Theorem 4]{taylor_smooth_2017}.\n\\qed\\end{proof}","units":[{"start":57603,"end":57761,"start_line":840,"end_line":842,"raw_tex":"\\begin{proposition}\\label{prop:monotonicity_Tk}%\n    The function $\\TigLmu[k][\\cdot][\\kappa]$ is strictly increasing in the first argument.%\n\\end{proposition}","sha256_utf8":"c2f686bffa0cd4b077911fba0f6ddcba0e0eabe1763afa5526f50a8d1e5c30dc"},{"start":57763,"end":57842,"start_line":843,"end_line":845,"raw_tex":"\\begin{proof}%\nSee \\cref{appendix:proof_prop_monotonicity_Tk}.%\n\\qed\\end{proof}","sha256_utf8":"85ccee2a3ffdba2f98ddc33791dc0b517329ac7c98662f1ac90e82cf203db23e"},{"start":58797,"end":59415,"start_line":858,"end_line":867,"raw_tex":"\\begin{proposition}\\label{prop:properties_step_size_thr}\n    The stepsize thresholds $\\gbari$ satisfy the following properties:%\n    \\begin{enumerate}[align=right,itemindent=1em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]\n        \\item $\\gbari[k] < \\gbari[k+1]$, for all integers $k \\geq 0$;\n        \\item $\\lim_{k\\rightarrow \\infty} \\gbari[k] = \\gbari[\\infty]$;\n        \\item \\shownewtextfinal{For any $\\gamma L \\in \\big[\\gbari[k],\\gbari[k+1]\\big)$: \n        $\\TigLmu[i][\\gamma L] \\geq 0$ $\\forall i \\leq k$, \n        and $\\TigLmu[i][\\gamma L] < 0$ $\\forall i \\geq k+1$.}\n     \\end{enumerate}%\n\\end{proposition}","sha256_utf8":"f4eca7a82cd391432dc312ea15071dcfeb152c10d96696eea4f1dffd167d9d15"},{"start":59417,"end":59500,"start_line":868,"end_line":870,"raw_tex":"\\begin{proof}%\nSee \\cref{appendix:proof_properties_step_size_thr}.%\n\\qed\\end{proof}","sha256_utf8":"3d4fdf0a2a39b158e647f8aba6d4f2c5b89e57c1f44edfe7f9ab30f08ff20e12"},{"start":90358,"end":90914,"start_line":1211,"end_line":1218,"raw_tex":"\\begin{theorem}[Interpolation conditions for smooth functions]\\label{thm:interp_hypo}\nSet $\\mathcal{T}=\\big\\{(x_i, g_i, f_i)\\big\\}_{i \\in \\mathcal{I}}$ is $\\mathcal{F}_{\\mu,L}$-interpolable, where $\\mu \\in (-\\infty, L)$, if and only if for every pair of indices $(i,j)$, with $i,j \\in \\mathcal{I}$, it holds:%\n\\begin{align}\\label{eq:Interp_hypoconvex}\nf_{i}{-}f_{j}{-}\\langle g_{j}, x_{i}-x_{j} \\rangle \n        {}\\geq{}\n    \\frac{1}{2L} \\|g_{i}{-}g_{j}\\|^{2} {}+{} \\frac{\\mu}{2L(L{-}\\mu)} \\|g_{i}{-}g_{j} {-} L (x_i {-}x_j)\\|^2.\n\\end{align}%\n\\end{theorem}","sha256_utf8":"7a6d21c95dc288917fe18f813a14253d201707309610c3dbbdbdbca61b29ae05"},{"start":91275,"end":91679,"start_line":1222,"end_line":1228,"raw_tex":"\\begin{corollary} \\label{corr:direct_sum_interp_cond} \nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty, L)$. For all $x_i, x_j \\in \\mathbb{R}^d$ it holds:%\n\\begin{align}\\label{eq:direct_sum_interp_cond}\n    0 {}\\geq{} \\left\\langle \\nabla f(x_i) - \\nabla f(x_j) - L(x_i-x_j) \\,,\\, \n                           \\nabla f(x_i) - \\nabla f(x_j) - \\mu(x_i-x_j) \\right\\rangle.\n\\end{align}%\n\\end{corollary}","sha256_utf8":"5c9cc15f9cc2a19b32c34e800e1129cf7b1cdbede483389c2bf36f546d1b73ed"},{"start":91681,"end":91905,"start_line":1229,"end_line":1231,"raw_tex":"\\begin{proof}\n    Inequality \\eqref{eq:direct_sum_interp_cond} results by summing up inequalities \\eqref{eq:Interp_hypoconvex} written for the pairs $(x_i,x_j)$ and $(x_j,x_i)$ and performing simplifications.\n\\qed\\end{proof}","sha256_utf8":"f97e843159ade9a6fdaab81167957d21e314df9aaf8f7a834b6cd90cb98415ff"},{"start":94817,"end":95207,"start_line":1266,"end_line":1271,"raw_tex":"\\begin{lemma}\\label{lemma:sufficient_decrease_N_*}\nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty, L)$. One iteration of \\eqref{eq:GM_it_fixed_steps} with stepsize $\\gamma L \\in (0,2)$ decreases the function values and at each point $x_i$ it holds:\n\\begin{align}\\label{eq:sufficient_decrease_N_*}\n    f(x_i) - f_* {}\\geq{} \\frac{1}{2L} \\|\\nabla f(x_i)\\|^2.\n\\end{align}%\n\\end{lemma}","sha256_utf8":"ce99cca578bea787ff99b477a2eefcb9e21c43a1d6c80d9db60a75b72531dbca"},{"start":99752,"end":100313,"start_line":1304,"end_line":1319,"raw_tex":"\\begin{proposition}[General procedure]\\label{prop:general_procedure_min_grad_nrm}\n    Inequality \\eqref{eq:I^k_N_general_ineq} implies:\n\\[\n    \\frac{f(x_0) - f(x_N)}{\\sum_{i=0}^N \\sigma_i}\n        {}\\geq{}\n    \\frac{1}{2L} \\,\n    \\min_{0\\leq i \\leq N} \\big\\{\\|\\nabla f(x_i)\\|^2\\big\\},\n\\]\nwith equality only if all gradient norms $\\|\\nabla f(x_i)\\|^2$ with $\\sigma_i>0$ are equal. Moreover, \n\\[\n    \\frac{f(x_0) - f_*}{1 + \\sum_{i=0}^N \\sigma_i}\n        {}\\geq{}\n    \\frac{1}{2L} \\,\n    \\min_{0\\leq i \\leq N} \\big\\{\\|\\nabla f(x_i)\\|^2\\big\\}.\n\\]\n\\end{proposition}","sha256_utf8":"cadefadf2103090fd76d30dd211abe4d5b4a4b03e732cd9c38fd39c76e88d0d7"},{"start":100317,"end":100646,"start_line":1321,"end_line":1324,"raw_tex":"\\begin{proof}\n   The first inequality results after taking minimum gradient norm in \\eqref{eq:I^k_N_general_ineq}. %\nThe adjustment in the denominator of the second inequality results from combining \\eqref{eq:sufficient_decrease_N_*} with \\eqref{eq:I^k_N_general_ineq} and taking again the minimum gradient norm. \\qed\n\\end{proof}","sha256_utf8":"71d42d4c41fddc071f89c1bbacf275b21bee0bcd6b74fc183a5b2476e9fd3123"},{"start":102606,"end":103384,"start_line":1355,"end_line":1365,"raw_tex":"\\begin{lemma}\\label{lemma:sufficient_decrease_h_leq_1}\n    Let $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty, L)$, and consider one iteration of \\eqref{eq:GM_it_fixed_steps} with stepsize $\\gamma L \\in(0,1]$. Then the following inequality holds, with equality only if $\\nabla f(x_{i}) = \\nabla f(x_{i+1})$ or $\\gamma L = 1$:\n\\begin{align}\\label{eq:one_step_rate_h_leq_1} \\tag{N2SD}\n\\hspace{-.5em}\n    f(x_{i}) {-} f(x_{i+1})\n        {}\\geq {}\n    \\frac{\\gamma L \\, \\gamma \\mu {-} 2\\gamma \\mu {+} \\gamma L}{L{-}\\mu} \\frac{\\|\\nabla f(x_{i})\\|^2}{2} +\n    \\frac{\\gamma L}{L{-}\\mu} \\frac{\\|\\nabla f(x_{i+1})\\|^{2}}{2}.\n\\end{align}%\n    Furthermore, if $\\mu \\leq 0$, then the gradient norms from \\eqref{eq:one_step_rate_h_leq_1} are weighted by positive quantities.\n\\end{lemma}","sha256_utf8":"f94359d6b23318bdea79bbbcdc70a6653adf09bd667171cefd2da0779ee8c77b"},{"start":103468,"end":104406,"start_line":1367,"end_line":1377,"raw_tex":"\\begin{proof}\\label{proof:lemma:sufficient_decrease_h_leq_1}\n    Inequality \\eqref{eq:hypo_interp_(0,1)} is reformulated by expressing the inner product in terms of gradient norms, i.e., $2 \\big\\langle g_{i}, g_{i+1}\\big\\rangle = \\|g_{i}\\|^2 + \\|g_{i+1}\\|^2 - \\|g_{i} - g_{i+1}\\|^2$, as:%\n\\begin{align} \\label{eq:tight_short_step_sizes}\n    f_{i}{-}f_{i+1}\n        {}\\geq {}\n    \\frac{\\gamma L \\, \\gamma \\mu {-} 2\\gamma \\mu {+} \\gamma L}{L{-}\\mu} \\frac{\\|g_{i}\\|^2}{2} +\n    \\frac{\\gamma L}{L{-}\\mu} \\frac{\\|g_{i+1}\\|^{2}}{2} +\n    \\frac{1 {-} \\gamma L}{L{-}\\mu} \\frac{\\|g_{i} {-} g_{i+1}\\|^{2}}{2}.\n\\end{align}\nSince $\\gamma L \\in (0,1]$, the mixed term can be neglected, resulting in inequality \\eqref{eq:one_step_rate_h_leq_1}, with equality only if $g_{i} = g_{i+1}$ or $\\gamma L = 1$. The numerator of first term rewrites as $-\\gamma \\mu(1-\\gamma L) + (\\gamma L - \\gamma \\mu)$, hence it is positive when $\\mu \\leq 0$.\n\\qed\\end{proof}","sha256_utf8":"dc726e1b6a8c1ae7c97c03b9da388a8c3264e186019f2c1571348d17ae7b27e7"},{"start":104512,"end":105624,"start_line":1380,"end_line":1398,"raw_tex":"\\begin{lemma}\\label{lemma:sufficient_decrease_h_geq_1}\n    Let $f \\in \\mathcal{F}_{\\mu,L}$, with, $\\mu \\in (-\\infty,L)$, and consider one iteration of \\eqref{eq:GM_it_fixed_steps} with stepsize $\\gamma L \\in [1,\\gbari[\\infty])$. Then the following inequality holds:%\n\\begin{align} \\label{eq:one_step_rate_h_geq_1} \\tag{N4SD}\n% \\hspace{-.25em}\n    f(x_{i}){-}f(x_{i+1})\n        {}\\geq {}\n    \\frac{\\gamma [(2{-}\\gamma L)(2{-}\\gamma \\mu){-}1]}{2{-}\\gamma L {- }\\gamma \\mu}\n    \\frac{\\|\\nabla f(x_{i})\\|^2}{2} +\n    \\frac{\\gamma}{2{-}\\gamma L {-} \\gamma \\mu} \\frac{\\|\\nabla f(x_{i+1})\\|^{2}}{2}.\n\\end{align}%\nWhen $\\gamma L > 1$, equality holds only if:%\n\\begin{align}\\label{eq:equality_case_h_geq_1}\n    {\\langle \\nabla f(x_{i})}, {\\nabla f(x_{i+1})\\rangle}\n        {}{=}{} \n    \\frac{(1{-}\\gamma \\mu)(1{-}\\gamma L)}{2{-}\\gamma L{-}\\gamma \\mu} \\|\\nabla f(x_{i})\\|^2 {}{+}{} \n    \\frac{1}{2{-}\\gamma L{-}\\gamma \\mu} \\|\\nabla f(x_{i+1})\\|^2.%\n\\end{align}%\nFurthermore, if $\\gamma L \\in [1,\\gbari[1]]$, then the gradient norms from \\eqref{eq:one_step_rate_h_geq_1} are weighted by nonnegative quantities.%\n\\end{lemma}","sha256_utf8":"6d927dd0dad2f284af3dcc5afd53a13f36eaa900332985bd282ebb7098e93f5b"},{"start":105629,"end":107506,"start_line":1400,"end_line":1420,"raw_tex":"\\begin{proof}\\label{proof:lemma:sufficient_decrease_h_geq_1}\nThe mixed term in inequality \\eqref{eq:tight_short_step_sizes} enters with a negative contribution for $\\gamma L > 1$. To counterbalance it, the complementary interpolation inequality \\eqref{eq:hypo_interp_(1,0)} is used, written with the equivalent expression:%\n\\begin{align*}\n   f_{i+1}-f_{i}\n        {}\\geq {}\n    \\frac{\\gamma L \\gamma \\mu {-} 2 \\gamma L {+} \\gamma \\mu}{L{-}\\mu} \\frac{\\|g_{i}\\|^2}{2} +\n    \\frac{\\gamma \\mu}{L{-}\\mu} \\frac{\\|g_{i+1}\\|^{2}}{2} +\n    \\frac{1 {-} \\gamma \\mu}{L{-}\\mu} \\frac{\\|g_{i} {-} g_{i+1}\\|^{2}}{2}.%\n\\end{align*}%\nConsider the sum of distance-1 inequalities $\\bar{I}_{i}^{1} {}\\coloneqq{} \\Iij[i][i+1] + \\Iji[i][i+1]$:%\n\\begin{align}\\label{eq:dist_1_blended_mixed_grads_diff} \\tag{$\\bar{I}_{i}^{1}$}\n\\hspace{-0.5em}\n0\n    {}\\geq{}\n\\frac{2 \\gamma L \\gamma \\mu {-} \\gamma L {-} \\gamma \\mu}{L{-}\\mu} \\frac{\\|g_{i}\\|^2}{2} {}+{}\n\\frac{\\gamma L {+} \\gamma \\mu}{L{-}\\mu} \\frac{\\|g_{i+1}\\|^2}{2} {}+{} \n\\frac{2{-}\\gamma L {-} \\gamma \\mu}{L{-}\\mu} \\frac{\\| g_{i} {-} g_{i+1}\\|^2}{2}.\n\\end{align}%\nBy weighting \\eqref{eq:dist_1_blended_mixed_grads_diff} with $\\beta(\\gamma L,\\gamma \\mu){}\\coloneqq{} \\tfrac{\\gamma L{-}1}{2{-}\\gamma L {-}\\gamma \\mu} \\geq 0$ and summing it with \\eqref{eq:tight_short_step_sizes}, the coefficient of $\\| g_{i} {-} g_{i+1}\\|^2$ gets canceled and we obtain \\eqref{eq:one_step_rate_h_geq_1}.\nThe equality condition \\eqref{eq:equality_case_h_geq_1} follows from the equality case of \\eqref{eq:dist_1_blended_mixed_grads_diff}. Nevertheless, the coefficient of $\\|g_{i}\\|^2$ is nonnegative for $\\gamma L \\in [1,\\gbari[1]]$, where $\\gbari[1]$ is the largest stepsize $\\gamma L \\in [1, 2)$ for which $(2-\\gamma L)(2-\\gamma \\mu) - 1 \\geq {} 0$, implying nonnegativity of the weight of $\\|\\nabla f(x_{i})\\|^2$ in \\eqref{eq:one_step_rate_h_geq_1}.\\qed\n\\end{proof}","sha256_utf8":"6d5429cbc85ca1921a9b82678940d77e46304cfb3fc0fb256daecd3b2aa6205e"},{"start":109822,"end":110060,"start_line":1458,"end_line":1460,"raw_tex":"\\begin{proposition}\\label{prop:properties:m+n>0}%[Properties for distance-2]\n    Let $\\mu \\in (-\\infty, L)$ and $\\gamma L \\in \\big(1, \\gbari[\\infty] \\big)$. Then $\\m {}>{}  0$, $\\m + \\n {}>0$ and $\\n^{-2} {}>{} \\m^{-2}$.\n\\end{proposition}","sha256_utf8":"0783371a0a3d71725a6939ad1c0b368212330714ff9ecf963f1d02a6004a3ade"},{"start":110062,"end":110491,"start_line":1461,"end_line":1463,"raw_tex":"\\begin{proof}%\nCondition $\\gamma L < \\gbari[\\infty]$ implies $\\gamma L < \\frac{2}{1+[\\kappa]_{+}}$, hence $0 < 2 {-} \\gamma L {-} \\gamma L [\\kappa]_{+} {}\\leq{} 2 {-} \\gamma L {-} \\gamma L \\kappa$, implying $\\m + \\n > 0$. Since $\\n = 1-\\gamma L \\in (-1,0)$, we have $\\m=1-\\gamma \\mu > 0$. Condition $L>\\mu$ implies $\\m>\\n$ and together with $\\m+\\n>0$ we conclude that $\\m^2 > \\n^2$, therefore $\\n^{-2} > \\m^{-2}$.\\qed\n\\end{proof}","sha256_utf8":"2b18a3dc9b529a65541f9785689d21b0a7df7e39f99bb0540a57d74087780ac1"},{"start":110494,"end":111291,"start_line":1465,"end_line":1478,"raw_tex":"\\begin{lemma}\\label{lemma:dist_1_L_1_step_decrease}\nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty,L)$, and consider one iteration of \\eqref{eq:GM_it_fixed_steps} from $x_i$ with stepsize $\\gamma L \\in (1, 2)$. Then it holds:\n\\begin{align} \\label{eq:dist_1_L_1_step_decrease} \n    \\begin{aligned}    \n    \\frac{f(x_{i}) - f(x_{i+1})}{\\gamma} \n                {}\\geq{}\n            {-}\\Ei[i][\\n] \\frac{\\|\\nabla f(x_{i})\\|^2}{2} {}+{}\n            \\Ei[i+1][\\n] \\frac{\\|\\nabla f(x_{i+1})\\|^2}{2} {}+{}            \n            \\\\ {}+{}\n            \\frac{ 1 - (\\m+\\n) \\Ei[i+1][\\n] }{\\m-\\n} \\frac{\\|\\nabla f(x_{i+1})-\\n \\nabla f(x_{i})\\|^2}{2}.\n    \\end{aligned}\n\\end{align}%\nFurthermore, if $\\mu \\in (0,L)$ and $\\gamma L \\in [\\frac{2}{1+\\kappa}, 2)$, then $1-(\\m+\\n) \\Ei[i][\\n] > 0$.\n\\end{lemma}","sha256_utf8":"82466954b224527d0f944db47ea7c4f0c70c9c848453ca8d1f17712bbfcf4d57"},{"start":111292,"end":112517,"start_line":1479,"end_line":1498,"raw_tex":"\\begin{proof}\nLet $\\beta_{i+1}(\\n) {{}{}\\coloneqq{}{}} (-\\n) \\Ei[i+1][\\n] {}>{} 0$. Inequality \\eqref{eq:dist_1_L_1_step_decrease} results after performing the simplifications in the linear combination of interpolation inequalities:\n$$\n\\Iij[i][i+1] + \\beta_{i+1}(\\n) \\, (\\Iij[i][i+1] + \\Iji[i][i+1]),$$\nwhere \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} are equivalently rewritten for the pair $(i,i+1)$ as:%\n$$\n    \\begin{aligned}\n    (1+\\beta_{i+1}) {\\,} \\Iij[i][i+1] {}:{}\n        \\frac{f_{i} {-} f_{i+1}}{\\gamma}\n            {}\\geq{}&\n        (1+\\n) \\frac{\\|g_{i}\\|^2}{2} {+} \\frac{1}{\\m{-}\\n} \\frac{\\|g_{i+1}{-}\\n g_{i}\\|^2}{2}; \\\\\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\beta_{i+1} {\\,} \\Iji[i][i+1] {}:{}\n        \\frac{f_{i+1} {-} f_{i}}{\\gamma}\n            {}\\geq{}&\n        {-}\\frac{\\|g_{i}\\|^2}{2} {-} \\frac{1}{\\n} \\frac{\\|g_{i+1}\\|^2}{2} {+} \\frac{\\m}{\\n} \\frac{1}{\\m{-}\\n} \\frac{\\|g_{i+1}{-}\\n g_{i}\\|^2}{2}.\n    \\end{aligned}\n$$%\n\\noindent{}With $\\gamma L \\in {[}\\frac{2}{1+\\kappa}, 2)$, we have $\\m{+}\\n \\leq 0$ (see \\cref{prop:properties:m+n>0}) and therefore $1-(\\m+\\n)\\Ei[i+1][\\n] > 0$.%\n\\qed \\end{proof}","sha256_utf8":"29b89c3117b02d0bc6ad35df97fa0581a925b836e01d79cd18098a34ef5f1d06"},{"start":112521,"end":113349,"start_line":1500,"end_line":1513,"raw_tex":"\\begin{lemma}\\label{lemma:dist_1_mu_1_step_decrease}\nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty,L)$, and consider one iteration of \\eqref{eq:GM_it_fixed_steps} from $x_i$ with stepsize $\\gamma L \\in \\big(0, \\gbari[\\infty]\\big)$. Then it holds:%\n\\begin{align} \\label{eq:dist_1_mu_1_step_decrease}\n    \\begin{aligned}    \n    \\frac{f(x_{i}) - f(x_{i+1})}{\\gamma} \n                {}\\geq{} \n            -\\Ei[i][\\m] \\frac{\\|\\nabla f(x_{i})\\|^2}{2} {}+{}\n             \\Ei[i+1][\\m] \\frac{\\|\\nabla f(x_{i+1})\\|^2}{2} {}+{}            \n            \\\\ {}+{}\n            \\frac{{-1} +(\\m+\\n) \\Ei[i+1][\\m]}{\\m-\\n} \\frac{\\|\\nabla f(x_{i+1})-\\m \\nabla f(x_{i})\\|^2}{2}.\n    \\end{aligned}\n\\end{align}%\nFurthermore, if $\\mu \\in (0,L)$ and $\\gamma L \\in \\big(0, \\gbari[i+1]\\big)$, then ${-1} +(\\m+\\n) \\Ei[i+1][\\m] > 0$.%\n\\end{lemma}","sha256_utf8":"151fdb4d95076ee9f25d7bb29dedaf37b1feb5624684c616f88776dbaa48e0cf"},{"start":113351,"end":115368,"start_line":1514,"end_line":1543,"raw_tex":"\\begin{proof}%\nFirstly, note that $\\m=1-\\gamma \\mu > 0$; the case $\\gamma L \\in \\big(1, \\frac{2}{1+[\\kappa]_{+}}\\big)$ is covered by \\cref{prop:properties:m+n>0}, while for $\\gamma L \\in (0,1]$ we have $1-\\gamma \\mu > 1-\\gamma L \\geq 0$. Let:%\n$$\n    \\beta_{i+1}(\\m) {}{}\\coloneqq{}{} {-1} + \\m \\Ei[i+1][\\m] {}={} \\frac{1 + \\m + \\dots + \\m^{2i}}{\\m^{2i+1}} > 0.\n$$%\nInequality \\eqref{eq:dist_1_mu_1_step_decrease} results after performing the simplifications in the linear combination of interpolation inequalities:%\n$$\n\\Iij[i][i+1] + \\beta_{i+1}(\\m) \\, (\\Iij[i][i+1] + \\Iji[i][i+1]),$$\nwhere \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} equivalently rewrite for the pair $(i,i+1)$ as:\n\\begin{align*}\n    (1+ \\beta_{i+1}) {\\,} \\Iij[i][i+1] {}:{}\\,{}{} \n        \\frac{f_{i} {-} f_{i+1}}{\\gamma}\n            {}\\geq{} &\n        \\frac{\\|g_{i}\\|^2}{2} {+} \\frac{1}{\\m} \\frac{\\|g_{i+1}\\|^2}{2} {+} \\frac{\\frac{\\n}{\\m}}{\\m{-}\\n} \\frac{\\|g_{i+1}{-}\\m g_{i}\\|^2}{2}; \\\\    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n     \\beta_{i+1} {\\,} \\Iji[i][i+1] {}:{}\\,{}{}\n        \\frac{f_{i+1} {-} f_{i}}{\\gamma}\n            {}\\geq{} &\n        {-}(1{+}\\m) \\frac{\\|g_{i}\\|^2}{2} {+} \\frac{1}{\\m{-}\\n} \\frac{\\|g_{i+1}{-}\\m g_{i}\\|^2}{2}.\n\\end{align*}%\nCondition $\\gamma L < \\gbari[i+1]$ implies $\\Tinm[i+1] < 0$, hence $i \\geq \\Nbar$ (recall \\cref{def:N_bar}: $\\Tinm[\\bar{N}] \\geq 0$ and $\\Tinm[\\bar{N}+1] < 0$, corresponding to $\\gamma L < \\gbari[\\bar{N}+1]$). Then the coefficient of the mixed term in \\eqref{eq:dist_1_mu_1_step_decrease} is positive:%\n$$\n    \\frac{{-1} {+} (\\m{+}\\n) E_{i+1}(\\m)}{\\m{-}\\n} \n        {}\\geq{} \n    \\frac{{-1} {+} (\\m{+}\\n) E_{\\bar{N}+1}(\\m)}{\\m{-}\\n}\n        {}={}\n    \\frac{\\Tinm[\\bar{N}]{-}\\n^2 \\Tinm[\\bar{N}+1]}{(\\m{-}\\n)^2} \n        {}>{} 0.\n$$\nThe first inequality results from the monotonic increase with index $i$ of $E_{i+1}(\\m)$, whereas the last one by the definition of $\\Nbar$.%\n\\qed \\end{proof}","sha256_utf8":"7cbf9733ad54dcdcc3e6f5dcd20667486339fd034f9140b46f90ac8ff5952f8a"},{"start":115606,"end":116504,"start_line":1549,"end_line":1562,"raw_tex":"\\begin{lemma}[Central distance-2 result]\\label{lemma:telescoped_main_lemma}\nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty, L)$. Consider $k+1$ iterations of \\eqref{eq:GM_it_fixed_steps} with constant stepsize $\\gamma L \\in \\big[\\gbari[k], \\gbari[\\infty] \\big)$ starting from $x_0$, where $k \\geq 1$. Then it holds:%\n\\begin{align}\n        & \\frac{f(x_{0}) {-} f(x_{k})}{\\gamma} +\n        \\frac{(\\m{+}\\n)\\Tinm[k]}{2(\\m{-}\\n)} \\frac{f(x_{k}) {-} f(x_{k+1})}{\\gamma}\n            \\geq\n        % \\\\ & {}\\qquad{}\n        \\left(\\Ei[k][\\m] {+} \\Ei[k][\\n]\\right) \\frac{\\|\\nabla f(x_{k})\\|^2}{4} +\n        \\nonumber \\\\ & {}\\quad{}\\,\n        {}+{}\\frac{\\Tinm[k]}{2(\\m{-}\\n)} \\Big[ \\m \\n \\frac{\\|\\nabla f(x_{k})\\|^2}{2} + \n        \\langle \\nabla f(x_{k}) , \\nabla f(x_{k+1}) \\rangle + \n        \\frac{\\|\\nabla f(x_{k+1})\\|^2}{2} \\Big]. \\label{eq:telescoped_main_lemma} \\tag{D2}\n\\end{align}%\n\\end{lemma}","sha256_utf8":"07d9d0b5ecc3b9fe7707662d51edc2216f008f8f01501eff63364353729e7f0e"},{"start":116506,"end":121932,"start_line":1563,"end_line":1682,"raw_tex":"\\begin{proof}\nConsider two iterations connecting $x_{i}$, $x_{i+1}$ and $x_{i+2}$, with $i \\geq 0$. The interpolation inequalities \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} written for distance-1 (\\Iij[i][i+1] and \\Iji[i][i+1]) and distance-2 (\\Iij[i][i+2] and \\Iji[i][i+2]), respectively:%\n\\begin{align*}\n    &\\begin{aligned}    \n    % & \\alpha_{[i-1>i]} \\times \n    \\Iij[i][i+1]\\colon \n        \\frac{f_{i} - f_{i+1}}{\\gamma}\n            {}\\geq{}\n        (1+\\n) \\frac{\\|g_{i}\\|^2}{2} + \\frac{1}{\\m-\\n} \\frac{\\|g_{i+1}-\\n g_{i}\\|^2}{2}; \n    \\end{aligned}\\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    &\\begin{aligned}    \n    % & \\alpha_{[i-1<i]}  \\times \n    \\Iji[i][i+1]\\colon \n        \\frac{f_{i+1} - f_{i}}{\\gamma}\n            {}\\geq{}\n        -\\frac{\\|g_{i}\\|^2}{2} - \\frac{1}{\\n} \\frac{\\|g_{i+1}\\|^2}{2} + \\frac{\\m}{\\n} \\frac{1}{\\m-\\n} \\frac{\\|g_{i+1}-\\n g_{i}\\|^2}{2}; \n    \\end{aligned}\\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    &\\begin{aligned}    \n    &% \\alpha_{[i-1>i+1]}  \\times \n    \\Iij[i][i+2]\\colon \n        \\frac{f_{i} - f_{i+2}}{\\gamma}\n            {}\\geq{}\n        (1+\\n+\\n^2) \\frac{\\|g_{i}\\|^2}{2} + \\n \\frac{\\|g_{i+1}\\|^2}{2} + \n        \\frac{1}{\\n(\\m-\\n)} \\cdot\n        \\\\ & {}\\qquad{}        \n        \\Big[\n        \\n(1-\\m) \\frac{\\|g_{i+1}-\\n g_{i}\\|^2}{2}\n        - (1-\\n) \\frac{\\|g_{i+2}-\\n g_{i+1}\\|^2}{2}\n        + \\frac{\\|g_{i+2}-\\n^2 g_{i}\\|^2}{2} \n        \\Big];\n    \\end{aligned}\\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    &\\begin{aligned}    \n    &% \\alpha_{[i-1<i+1]}  \\times \n    \\Iji[i][i+2]\\colon\n        \\frac{f_{i+2} - f_{i}}{\\gamma}\n            {}\\geq{} \n        - \\frac{\\|g_{i}\\|^2}{2} - \\frac{1}{\\n} \\frac{\\|g_{i+1}\\|^2}{2}\n        - \\frac{1+\\n}{\\n^2} \\frac{\\|g_{i+2}\\|^2}{2} + \\frac{1}{\\n(\\m-\\n)} \\cdot\n        \\\\ & {}\\qquad{} \n        \\Big[\n        \\m(1-\\n) \\frac{\\|g_{i+1}-\\n g_{i}\\|^2}{2}\n        - (1-\\m) \\frac{\\|g_{i+2}-\\n g_{i+1}\\|^2}{2}\n        + \\frac{\\m}{\\n} \\frac{\\|g_{i+2}-\\n^2 g_{i}\\|^2}{2} \n        \\Big],\n    \\end{aligned}\n\\end{align*}%\nare summed up in a linear combination with \\textit{nonnegative} corresponding weights:%\n$$\n        \\eqref{eq:dist_2_atom_ineq} \\equiv\n\\alphaij[i][i+1] \\Iij[i][i+1] +\n\\alphaij[i+1][i] \\Iji[i][i+1] +\n\\alphaij[i][i+2] \\Iij[i][i+2] +\n\\alphaij[i+2][i] \\Iji[i][i+2],\n$$%\nwhere:%\n$$\n\\begin{aligned}\n    \\alphaij[i][i+1] {}{}\\coloneqq{}{}& \n    1 - \\n \\Ei[i+1][\\n] + \\frac{(1-\\m) \\Tinm[i]}{2(\\m-\\n)} - \\frac{\\m(1+\\n) \\Tinm[i+1]}{2(\\m-\\n)};\n    \\\\\n   \\alphaij[i+1][i] {}{}\\coloneqq{}{}& \n    -\\n \\Ei[i+1][\\n] + \\frac{(1+\\n) \\Tinm[i]}{2(\\m-\\n)} + \\frac{\\n(1-\\m) \\Tinm[i+1]}{2(\\m-\\n)};\n    \\\\\n    \\alphaij[i][i+2] {}{}\\coloneqq{}{}&\n     \\frac{\\m \\Tinm[i+1]}{2(\\m-\\n)}; \n     \\\\\n    \\alphaij[i+2][i] {}{}\\coloneqq{}{}&\n    \\frac{-\\n \\Tinm[i+1]}{2(\\m-\\n)},\n\\end{aligned}%\n$$%\nand \\eqref{eq:dist_2_atom_ineq} is obtained after performing the simplifications:%\n\\begin{align}\\label{eq:dist_2_atom_ineq}    \n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\begin{aligned}\n        &\n        \\frac{f_{i} {-} f_{i+1}}{\\gamma} + \n        (\\m{+}\\n) \\Big[\n        {-}\\frac{\\Tinm[i]}{2(\\m{-}\\n)} \\frac{f_{i} {-} f_{i+1}}{\\gamma} +\n           \\frac{\\Tinm[i+1]}{2(\\m{-}\\n)} \\frac{f_{i+1} {-} f_{i+2}}{\\gamma}\n        \\Big]\n            {}\\geq{}\n        \\\\ & {}\\,{} \n        {}{-}{}\\frac{\\Tinm[i]}{2(\\m{-}\\n)} \\Big[ \\m \\n \\frac{\\|g_{i}\\|^2}{2} + \n        \\langle g_{i}, g_{i+1} \\rangle + \\frac{\\|g_{i+1}\\|^2}{2}  \\Big] \n        {-}\\left(\\Ei[i][\\m] {+} \\Ei[i][\\n]\\right) \\frac{\\|g_{i}\\|^2}{4}\n        \\\\ & {}\\,{} \n        {}{+}{}\\frac{\\Tinm[i+1]}{2(\\m{-}\\n)} \\Big[ \\m \\n \\frac{\\|g_{i+1}\\|^2}{2} \n        {+} {\\langle g_{i+1}, g_{i+2} \\rangle} + \\frac{\\|g_{i+2}\\|^2}{2}  \\Big] {}+{} \\left(\\Ei[i+1][\\m] {+} \\Ei[i+1][\\n]\\right) \\frac{\\|g_{i+1}\\|^2}{4}.\n    \\end{aligned}\n\\end{align}%\n\\noindent{}Telescoping it for indices $i=0,1,\\dots,k-1$ yields inequality \\eqref{eq:telescoped_main_lemma}. It remains to show the nonnegativity of multipliers. Since the stepsize thresholds increase with index $i$, condition $\\gamma L \\geq \\gbari[k]$ ensures $\\Tinm[i+1] \\geq 0$, for all $i=0,{\\dots},k-1$. This directly implies the positivity of $\\alphaij[i][i+2]$ and $\\alphaij[i+2][i]$. The positivity of distance-1 multipliers is evidenced by their equivalent representations as nonnegative sums (recall $\\n \\in (-1,0)$, $\\m \\in \\left(0,\\infty\\right)$; $\\m+\\n>0$ and $\\n^{-2} > \\m^{-2}$ from \\cref{prop:properties:m+n>0}):%\n$$\n\\begin{aligned}\n    &\\begin{aligned}    \n    2\\alphaij[i][i+1] {}={}& \n    1 + \\frac{\\m+\\m^{-2i}}{1+\\m} + \n    \\frac{(1+\\n)(\\m+\\n-\\n\\m)}{\\m \\n^2} \\frac{{-1} +\\n^{-2(i+1)}}{{-1} +\\n^{-2}} + \n    \\\\& {}\\qquad\\qquad{}\n    \\frac{(1+\\n)(1+\\m^2)}{\\m(\\m-\\n)} \n    % \\Big[ \\frac{1-\\n^{-2i}}{1-\\n^{-2}} - \\frac{1-\\m^{-2i}}{1-\\m^{-2}} \\Big] \n    \\sum_{j=0}^{i} \\big(\\n^{-2j} - \\m^{-2j}\\big)\n    {}>{} 0;  \n    \\end{aligned} \\\\%\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    &\\begin{aligned}  \n    2\\alphaij[i+1][i] {}={}&\n    \\frac{{-1} +\\n^{-2i}}{1-\\n} + \n    \\frac{\\m(1+\\n)}{1-\\n} \\frac{{-1} +\\m^{-2(i+1)}}{1-\\m}\n    + \\\\& {}\\qquad\\qquad{}\n     +\\frac{-\\n \\big[1-\\n+\\m(1+\\n)\\big]}{1-\\n} \\frac{\\n^{-2(i+1)} - \\m^{-2(i+1)}}{\\m-\\n} > 0. \n    % {}\\qquad\\qquad\\qquad\\qquad\\qquad{}\n    {}\\qquad{}\n    {}\\qed{}\n    \\end{aligned}%\n\\end{aligned}\n$$%\n\\end{proof}","sha256_utf8":"21a847512e29449a384eb973e62b74b0eb06a0d64fe761ffd91b50f13af1e10b"},{"start":123252,"end":124132,"start_line":1690,"end_line":1702,"raw_tex":"\\begin{lemma}[Multistep descent]\\label{lemma:hypo_GN4SD}\nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty, L)$, and consider $k+1 {}\\geq{} 1$ iterations of \\eqref{eq:GM_it_fixed_steps} with stepsize $\\gamma L \\in [\\gbari[k], \\gbari[\\infty])$, {\\shownewtext{satisfying $\\gamma L > 1$,}} starting from $x_0$. Then the following inequality holds:%\n\\begin{align}%\n% \\hspace{-.25em}\n    \\frac{f(x_0) {-} f(x_{k+1})}{\\gamma}\n        {}\\geq{}\n    \\frac{{-}\\m^2 \\n^2 \\Tinm[k+1]}{\\m^2{-}\\n^2} \\frac{\\|\\nabla f(x_{k})\\|^2}{2} {}+{}\n    \\frac{\\Tinm[k]{+}(\\m{-}\\n)}{\\m^2{-}\\n^2} \\frac{\\|\\nabla f(x_{k+1})\\|^2}{2}.\n\\label{eq:general_ineq_proof} \\tag{GN4SD}\n% \\tag{$I_{N+1}^{N}$}\n\\end{align}%\nFurthermore, if $\\mu \\in (-\\infty, 0]$ and $\\gamma L \\in [\\gbari[k], \\gbari[k+1])$, then the gradient norms from \\eqref{eq:general_ineq_proof} are weighted by nonnegative quantities.\n\\end{lemma}","sha256_utf8":"5e061aa9818d7789359b32ff4535cb695c0299079ef7ea64fc6a53022de8f83c"},{"start":124138,"end":126308,"start_line":1705,"end_line":1745,"raw_tex":"\\begin{proof}%\nFor $k=0$, the result follows from \\cref{lemma:sufficient_decrease_h_geq_1} and we obtain exactly \\eqref{eq:one_step_rate_h_geq_1}. \nFor $k\\geq 1$, inequality \\eqref{eq:general_ineq_proof} is derived by adjusting \\eqref{eq:telescoped_main_lemma} through the linear combination of interpolation inequalities $\\alphaij[k][k+1] \\Iij[k][k+1] + \\alphaij[k+1][k] \\Iji[k][k+1]$:%\n$$\n    \\begin{aligned}\n        &  \\Big[1 - \\frac{(\\m+\\n)\\,\\Tinm[k]}{2(\\m-\\n)} \\Big] \\frac{f_{k} - f_{k+1}}{\\gamma}\n        {}\\geq{}\n        \\\\ & {}\\qquad{}\n        \\Big(-\\Ei[k][\\m] - \\Ei[k][\\n] - \\frac{\\m \\n \\Tinm[k]}{\\m-\\n} {}-{} \\frac{2\\m^2\\n^2 \\Tinm[k+1]}{\\m^2-\\n^2} \\Big) \\frac{\\|g_{k}\\|^2}{4}\n        \\\\ & {}\\qquad{}        \n        % \\\\ & {}\\qquad{}\n        {}+{} \\Big(\\frac{\\m-\\n + \\Tinm[k]}{\\m^2-\\n^2} - \\frac{\\Tinm[k]}{2(\\m-\\n)} \\Big) \\frac{\\|g_{k+1}\\|^2}{2}\n        {}-{}\\frac{\\Tinm[k]}{2(\\m-\\n)} \\langle g_{k},g_{k+1} \\rangle,\n    \\end{aligned}\n$$%\nwhere the corresponding weights are defined as follows:%\n    $$\n    \\begin{aligned}\n        \\alphaij[k][k+1]\n             {}{}\\coloneqq{}{} &\n        1 + \\frac{-\\n}{\\m+\\n} + \\frac{(1-\\m)(\\m+\\n)-2\\n}{2(\\m^2-\\n^2)} \\Tinm[k];  \\\\\n        %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n        \\alphaij[k+1][k]\n            {}{}\\coloneqq{}{} &\n        \\frac{-\\n}{\\m+\\n} + \\frac{(1+\\n)(\\m+\\n)-2\\n}{2(\\m^2-\\n^2)} \\Tinm[k].\n    \\end{aligned}\n    $$ %\n    Multiplier $\\alphaij[k+1][k]$ is represented as a sum of positive quantities ($\\n < 0$).\n    Multiplier $\\alphaij[k][k+1]$ is directly written as sum of positive quantities if $(1-\\m)(\\m+\\n)-2\\n > 0$ (in particular, this covers the (strongly) convex case where $\\m \\leq 1$). Otherwise, $\\alphaij[k][k+1]$ is equivalently expressed as a sum of positive terms as follows, with $\\m>1$:%\n$$\n\\begin{aligned}\n    \\alphaij[k][k+1]\n        {}={}&\n    \\frac{\\m}{2(\\m+\\n)} + \\frac{1}{2(\\m^2-\\n^2)} \n        \\bigg[ \\frac{-\\n(\\m-\\n)(1+\\m^2)}{(\\m-1)(1-\\n)} {}+{}\n        \\\\ & {}\\quad{} \n        -\\big[(1-\\m)(\\m+\\n)-2\\n\\big] \\Big( \\frac{\\m^{-2k}}{\\m-1} + \\frac{\\n^{-2k}}{1-\\n} \\Big)  \\bigg]\n        {}>{} 0. {}\\quad{} \\qed\n\\end{aligned}\n$$\n\\end{proof}","sha256_utf8":"8f79ca0b050bc7379bae290176176454f8bb6116412c66cffb6868015490b133"},{"start":126312,"end":127046,"start_line":1747,"end_line":1757,"raw_tex":"\\begin{lemma}\\label{lemma:general_lemma_ct_h_to_Linear} \nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty, L)$, and consider $k+1 \\geq 1$ iterations of \\eqref{eq:GM_it_fixed_steps} with stepsize $\\gamma L \\in [\\gbari[k], \\gbari[\\infty] \\big)$, satisfying $\\gamma L > 1$, starting from $x_0$. Then the following inequality holds:%\n\\begin{align}\\label{eq:general_ineq_proof_to_linear_compact}\n   \\frac{f(x_0) {-} f(x_{k+1})}{\\gamma}\n        {}\\geq{} \n    \\Ei[k+1][\\n] \\frac{\\|\\nabla f(x_{k+1})\\|^2}{2} {+} \n    \\frac{\\m^2 \\Tinm[k+1]}{(\\m{-}\\n)^2}\n    \\frac{\\| \\nabla f(x_{k+1}) {-} \\n \\nabla f(x_{k}) \\|^2}{2}.\n\\end{align}%\nFurthermore, if $\\gamma L \\in [\\gbari[k+1], \\gbari[\\infty] \\big)$, then $\\Tinm[k+1]\\geq 0 $.%\n\\end{lemma}","sha256_utf8":"06deb370a91f2e1d20da341b6e11b404bf1cb2fcbe1d82cd61a94d5dd21e4c59"},{"start":127048,"end":128893,"start_line":1758,"end_line":1792,"raw_tex":"\\begin{proof}%\nFor $k=0$, the result follows from \\cref{lemma:dist_1_L_1_step_decrease}. For $k \\geq 1$, inequality \\eqref{eq:general_ineq_proof_to_linear_compact} is derived by adjusting \\eqref{eq:telescoped_main_lemma} with the linear combination of interpolation inequalities $\\alphaij[k][k+1] \\Iij[k][k+1] + \\alphaij[k][k+1] \\Iji[k][k+1]$:%\n$$\n    % \\hspace{-1em}\n    \\begin{aligned}\n        &\\Big[1 - \\frac{(\\m+\\n)\\,\\Tinm[k]}{2(\\m-\\n)} \\Big] \\frac{f_{k} - f_{k+1}}{\\gamma}\n            {}\\geq{}\n        \\Big(-\\frac{\\Tinm[k]}{2(\\m-\\n)} - \\frac{\\m^2 \\n \\Tinm[k+1]}{(\\m-\\n)^2}\\Big) \\langle g_{k},g_{k+1} \\rangle {}+{}\n        \\\\ & {}\\quad\\qquad{}\n        {}+{} \\Big[ \\frac{2\\m^2 \\n^2 \\Tinm[k+1]}{(\\m-\\n)^2} -\\Ei[k][\\m] - \\Ei[k][\\n] - \\frac{\\m \\n \\Tinm[k]}{\\m-\\n} \\Big] \\frac{\\|g_{k}\\|^2}{4} {}+{}\n        \\\\ & {}\\quad\\qquad{}\n        {}+{} \\Big[ \\frac{\\m^2 \\, \\Tinm[k+1]}{(\\m-\\n)^2}  + \\Ei[k+1][\\n] -\\frac{\\Tinm[k]}{2(\\m-\\n)} \\Big] \\frac{\\|g_{k+1}\\|^2}{2},\n    \\end{aligned}\n$$%\nwhere the corresponding weights are defined as:\n$$\n\\begin{aligned}\n    \\alphaij[k][k+1] \n        {}{}\\coloneqq{}{} &\n    1+(-\\n)\\Ei[k+1][\\n] + \\frac{1-\\m}{2(\\m-\\n)} \\Tinm[k];\n    \\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\alphaij[k+1][k]\n        {}{}\\coloneqq{}{} &\n    (-\\n) \\Ei[k+1][\\n] + \n    \\frac{1+\\n}{2(\\m-\\n)} \\Tinm[k].\n\\end{aligned}%\n$$%\nMultiplier $\\alphaij[k+1][k]$ is represented as a sum of positive quantities, while $\\alphaij[k][k+1]$ is directly written as sum of positive terms if $\\m \\leq 1$ (the (strongly) convex case). Otherwise, $\\m > 1$ and $\\alphaij[k][k+1]$ is equivalently expressed as sum of positive terms as:%\n$$\n        \\alphaij[k][k+1]\n    {}={}\n        \\frac{1+(-\\n) \\n^{-2(k+1)}}{2(1-\\n)} + \\m \\frac{\\m \\m^{-2(k+1)} + (-\\n) \\n^{-2(k+1)}}{2(\\m-\\n)}. {}\\quad{} \\qed\n$$\n\\end{proof}","sha256_utf8":"63daf2f8875c39b45f6a4aab97c70b2e7a7898387293277bfc695a973ceeed5b"},{"start":128897,"end":129637,"start_line":1794,"end_line":1805,"raw_tex":"\\begin{lemma}\\label{lemma:general_lemma_ct_h_to_Linear_mu} \nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty,L)$, and consider $k+1 \\geq 1$ iterations of \\eqref{eq:GM_it_fixed_steps} with stepsize $\\gamma L \\in [\\gbari[k], \\gbari[\\infty])$, satisfying $\\gamma L > 1$, starting from $x_0$. Then the following inequality holds:%\n\\begin{align}\\label{eq:general_ineq_proof_to_linear_mu_compact}\n% \\hspace{-.75em}\n    \\frac{f(x_0) {-} f(x_{k+1})}{\\gamma}\n        {\\geq}\n    \\Ei[k+1][\\m] \\frac{\\|\\nabla f(x_{k+1})\\|^2}{2} {+} \n    \\frac{-\\n^2 \\Tinm[k+1]}{(\\m{-}\\n)^2}\n    \\frac{\\| \\nabla f(x_{k+1}) {-} \\m \\nabla f(x_{k}) \\|^2}{2}.\n\\end{align}%\nFurthermore, if $\\gamma L \\in [\\gbari[k], \\gbari[k+1] \\big)$, then $\\Tinm[k+1]<0$.\n\\end{lemma}","sha256_utf8":"d0a796d2cb34eda0beaf1c1b1c987308a869ce0913d4683abc66f4ca96f3da32"},{"start":129643,"end":131680,"start_line":1808,"end_line":1842,"raw_tex":"\\begin{proof}\nCase $k=0$ results from \\cref{lemma:dist_1_mu_1_step_decrease}; further on, $k\\geq 1$. Inequality \\eqref{eq:direct_sum_interp_cond} written for the pair $(i,j)=(k,k+1)$ reads:%\n   \\begin{align}\\label{eq:dist_1_correction_to_mu}\n       0\n        {}\\geq{}\n       \\frac{\\m \\n}{\\m-\\n} \\|g_{k}\\|^2 + \\frac{1}{\\m-\\n} \\|g_{k+1}\\|^2 - \\frac{\\m+\\n}{\\m-\\n} \\langle g_{k} \\,,\\, g_{k+1} \\rangle.\n   \\end{align}%\n   \\begin{enumerate}[align=right,itemindent=1em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]\n       \\item \\textbf{Case $\\gamma L \\in [\\gbari[k], \\gbari[k+1])$.} By using $\\m>0$, \\eqref{eq:dist_1_correction_to_mu} equivalently rewrites as:%\n   $$\n       0\n        {}\\geq{}\n       -\\m^2 \\frac{\\|g_{k}\\|^2}{2} + \\frac{\\|g_{k+1}\\|^2}{2} + \\frac{\\m+\\n}{\\m-\\n} \\frac{\\|g_{k+1}-\\m g_{k}\\|^2}{2}.\n   $$%\n   By multiplying it with $\\frac{-\\n^2}{\\m^2-\\n^2}\\Tinm[k+1]$ (positive because $\\Tinm[k+1] < 0$) and adding it to \\eqref{eq:general_ineq_proof}, the coefficient of $\\|g_{k}\\|^2$ is canceled and we get:%\n    $$\n    \\frac{f_0{-}f_{k+1}}{\\gamma}\n        {}{\\geq}{}\n    \\frac{\\Tinm[k]{+}(\\m{-}\\n){-}\\n^2 \\Tinm[k+1]}{\\m^2{-}\\n^2} \\frac{\\|g_{k+1}\\|^2}{2} {}+{}\n       \\frac{{-}\\n^2 \\Tinm[k+1]}{(\\m{-}\\n)^2}  \\frac{\\|g_{k+1}{-}\\m g_{k}\\|^2}{2},\n    $$\n    which after simplifications is exactly \\eqref{eq:general_ineq_proof_to_linear_mu_compact}.%\n    %\n    \\item \\textbf{Case $\\gamma L \\in \\big[\\gbari[k+1],\\gbari[\\infty] \\big)$.} After multiplying \\eqref{eq:dist_1_correction_to_mu} with $\\frac{-\\n \\m}{\\m-\\n} > 0$:\n    $$\n        0\n            {}\\geq{}\n        \\frac{\\|g_{k+1}\\|^2}{2} + \n        \\frac{-\\m^2}{(\\m-\\n)^2} \\frac{\\|g_{k+1}-\\n g_{k}\\|^2}{2} +\n        \\frac{-\\n^2}{(\\m-\\n)^2} \\frac{\\|g_{k+1}-\\m g_{k}\\|^2}{2}.\n    $$\n    By scaling it with $\\Tinm[k+1]>0$ and summing it with \\eqref{eq:general_ineq_proof_to_linear_compact}, the coefficient of $\\|g_{k+1} - \\n g_{k}\\|^2$ is canceled and after simplifications we get \\eqref{eq:general_ineq_proof_to_linear_mu_compact}.%\n    {}\\qed{}%\n    \\end{enumerate}%\n\\end{proof}","sha256_utf8":"90749ca63c2b3c07f014d543621303911947adc199444c17bf22ea93710d28ad"},{"start":141130,"end":145878,"start_line":2038,"end_line":2142,"raw_tex":"\\begin{proof}[\\textbf{\\cref{thm:strg_cvx_rate}}]\\label{proof:thm:strg_cvx_rate}%\nWe split the proof over stepsize intervals:%\n\\[\n    (0,2) \n        {}={}\n    \\big(0, \\gbari[1]\\big] \\cup\n    \\Big( \\mathop{\\cup}\\limits_{k=1}^{N-1} \\big[\\gbari[k],\\gbari[k+1]\\big)\\Big)\n    {}\\cup{} \\big[\\gbari[N],\\tfrac{2}{1+\\kappa}\\big) \n    {}\\cup{} \\big[\\tfrac{2}{1+\\kappa}, 2\\big).\n\\]%\n%\n\\begin{enumerate}[align=right,itemindent=1em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]%\n\\item \\textbf{Case $\\gamma L \\in \\big(0, \\gbari[1]\\big]$}. By telescoping \\eqref{eq:dist_1_mu_1_step_decrease} for indices $i=0,\\dots,N-1$:%\n$$\n        \\frac{f_{0} - f_{N}}{\\gamma} \n            {}\\geq{}\n        \\Ei[N][\\m] \\frac{\\|g_{N}\\|^2}{2} {}+{}        \n            \\sum_{i=0}^{N-1}\n            \\frac{ {-1} + (\\m+\\n) \\Ei[i+1][\\m] }{\\m-\\n} \\frac{\\|g_{i+1}-\\m g_{i}\\|^2}{2},\n$$%\nwhere all mixed terms have nonnegative coefficients because $\\gamma L \\in \\big(0,  \\gbari[1]\\big]$ (see \\cref{lemma:dist_1_mu_1_step_decrease}). Hence they can be neglected to obtain the lower bound of type \\eqref{eq:I^k_N_general_ineq}:\n\\begin{align}\\label{eq:sufficient_decrease_N_its_mu_strongly_convex_towards_rate_no_dist_2}\n    \\frac{f_0 - f_N}{\\gamma}\n        {}\\geq{}\n    \\Ei[N][\\m] \\frac{\\|g_{N}\\|^2}{2}.\n\\end{align}%:\n\\item \\textbf{Case $\\gamma L \\in \\big[\\gbari[k],\\gbari[k+1]\\big)$}, with some $k \\in \\{1,\\dots,N-1\\}$. By telescoping \\eqref{eq:dist_1_mu_1_step_decrease} for indices $i=k+1,\\dots,N-1$:%\n$$\n\\begin{aligned}\n        \\frac{f_{k+1} {-} f_{N}}{\\gamma} \n            {}\\geq{} &\n        {-}\\Ei[k+1][\\m] \\frac{\\|g_{k+1}\\|^2}{2} {+}\n        \\Ei[N][\\m] \\frac{\\|g_{N}\\|^2}{2} {+} \\\\\n        & \\quad     \\sum_{i{=}k{+}1}^{N{-}1}\n            \\frac{ {-1} {+} (\\m{+}\\n) \\Ei[i{+}1][\\m] }{\\m{-}\\n} \\frac{\\|g_{i+1}{-}\\m g_{i}\\|^2}{2},\n\\end{aligned}\n$$%\nwith nonnegative weights of the mixed terms since $\\gamma L \\in \\big(0,  \\gbari[k+1]\\big)$ (see \\cref{lemma:dist_1_mu_1_step_decrease}). After appending it to \\eqref{eq:general_ineq_proof_to_linear_mu_compact} from \\cref{lemma:general_lemma_ct_h_to_Linear_mu}, we obtain:%\n$$\n\\begin{aligned}\n    \\frac{f_0 {-} f_N}{\\gamma}\n        {}\\geq{}\n    \\Ei[N][\\m] \\frac{\\|g_{N}\\|^2}{2} +\n    \\frac{-\\n^2 \\, \\Tinm[k+1]}{(\\m-\\n)^2} \\, \n    \\frac{\\| g_{k+1} - \\m g_{k} \\|^2}{2} +     \\\\\n    \\sum_{i=k+1}^{N-1}\n            \\frac{ {-1} + (\\m+\\n) \\Ei[i+1][\\m] }{\\m-\\n} \\frac{\\|g_{i+1}-\\m g_{i}\\|^2}{2}.\n\\end{aligned}\n$$\nSince $\\Tinm[k+1] < 0$, all mixed terms can be neglected to get a \\eqref{eq:I^k_N_general_ineq}-type inequality:%\n\\begin{align}\\label{eq:sufficient_decrease_N_its_mu_strongly_convex_towards_rate}\n    \\frac{f_0 - f_N}{\\gamma}\n        {}\\geq{}\n    \\Ei[N][\\m] \\frac{\\|g_{N}\\|^2}{2}.\n\\end{align}%\n%\n \\item \\textbf{Case $\\gamma L \\in \\big[\\gbari[N],\\frac{2}{1+\\kappa}\\big)$.} From \\cref{lemma:general_lemma_ct_h_to_Linear} with $k=N-1$ it holds:%\n$$\n   \\frac{f_0 - f_N}{\\gamma}\n        {}\\geq{}\n    \\Ei[N][\\n] \\frac{\\|g_N\\|^2}{2} + \n    \\frac{\\m^2 \\, \\Tinm[N]}{(\\m-\\n)^2} \\, \n    \\frac{\\| g_N - \\n g_{N-1} \\|^2}{2}.\n$$%\nSince $\\TigLmu[N] \\geq 0$, by neglecting the mixed term we obtain a \\eqref{eq:I^k_N_general_ineq}-type inequality:\n\\begin{align}\\label{eq:wc_linear_L}\n    \\frac{f_0 - f_N}{\\gamma}\n        {}\\geq{}\n    \\Ei[N][\\n] \\frac{\\|g_{N}\\|^2}{2}.\n\\end{align}%\n%\n\\item \\textbf{Case $\\gamma L \\in \\big[\\frac{2}{1+\\kappa},2\\big)$.} By telescoping \\cref{lemma:dist_1_L_1_step_decrease} for $i=0,1,\\dots,N-1$ it holds:%\n$$\n    \\frac{f_{0} - f_{N}}{\\gamma} \n                {}\\geq{}\n            \\Ei[N][\\n] \\frac{\\|g_{i}\\|^2}{2} +\n            \\sum_{i=0}^{N-1} \\frac{ 1 - (\\m+\\n) \\Ei[i+1][\\n] }{\\m-\\n} \\frac{\\|g_{i+1}-\\n g_{i}\\|^2}{2},\n$$%\nwhere the mixed terms can be neglected and then the inequality reduces to \\eqref{eq:wc_linear_L}.%\n\\end{enumerate}%\nSince the transition between the two linear regimes is given by the sign of $\\Tinm[N]=\\Ei[N][\\m]{-}\\Ei[N][\\n]$, we can glue together expressions \\eqref{eq:sufficient_decrease_N_its_mu_strongly_convex_towards_rate_no_dist_2}, \\eqref{eq:sufficient_decrease_N_its_mu_strongly_convex_towards_rate} and \\eqref{eq:wc_linear_L}:%\n$$\n    \\frac{f_0 - f_N}{\\gamma}\n        {}\\geq{}\n    \\min\\left\\{ \\Ei[N][\\m] \\,,\\, \\Ei[N][\\n]\\right\\} \\frac{\\|g_{N}\\|^2}{2},\n$$%\nequivalent to:\n$$\n    f_0 - f_N\n        {}\\geq{}\n    \\min\n    \\Big\\{ \\frac{{-1} +(1-\\gamma \\mu)^{-2N}}{\\mu} \\,,\\, \n    \\frac{{-1} +(1-\\gamma L)^{-2N}}{L}\n    \\Big\\} \\frac{\\|g_{N}\\|^2}{2},\n$$%\nwhich implies the bound \\eqref{eq:strg_cvx_full_ct} and the complementary one:\n$$\n\\begin{aligned}\n    \\frac{1}{2L} \\|\\nabla f(x_N)\\|^2\n        {} \\leq {}  \n    \\frac{f(x_0)-f(x_N)}\n    { \\min\\big\\{ \\frac{L}{\\mu} \\left[{-1}+(1-\\gamma \\mu)^{-2N}\\right] \\,,\\, {-1}+(1-\\gamma L)^{-2N} \\big\\} }. {}\\qquad{}\\qed \n\\end{aligned}%\n$$%\n\\end{proof}","sha256_utf8":"edf1d2965112e5b978e4384502750d0aa0b119d495010cccaafecb3f12fe87d0"},{"start":184657,"end":187264,"start_line":2767,"end_line":2814,"raw_tex":"\\begin{proof}[\\textbf{\\cref{prop:monotonicity_Tk}}] \\label{appendix:proof_prop_monotonicity_Tk}\nWe show $\\frac{d\\TigLmu}{d (\\gamma L)} \\geq 0$ for $\\gamma L \\in (1, \\gbari[\\infty])$, which resumes to prove the positivity of\n  \\begin{align*}\n        \\frac{d \\delta_i(\\gamma L, \\kappa)}{d(\\gamma L)}\n            {}{}\\coloneqq{}{}\n      \\frac{-\\kappa \\left( 1 - 2i \\frac{2-\\kappa \\gamma L}{1-\\kappa \\gamma L} \\right) }{(1-\\kappa \\gamma L)^{2i}} + \\frac{ 1 + 2i \\frac{2-\\gamma L}{\\gamma L-1}}{(1-\\gamma L)^{2i}}.\n  \\end{align*}\n  \\begin{enumerate}[align=right,itemindent=1em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]\n      \\item \\textbf{Case $\\kappa \\geq 0$:} we lower bound $\\frac{d \\delta_i(\\gamma L, \\kappa)}{d(\\gamma L)}$ by neglecting the denominators:\n      \\begin{align*}\n          \\delta_i(\\gamma L, \\kappa) \n            {}\\geq{} \n          {-}\\kappa \\Big( 1 - 2i \\frac{2-\\kappa \\gamma L}{1-\\kappa \\gamma L} \\Big) + 1 + 2i \\frac{2-\\gamma L}{\\gamma L-1}\n            {}={}\n          (1-\\kappa) + 2i \\Big[ \\frac{\\kappa (2-\\kappa \\gamma L)}{1-\\kappa \\gamma L}  + \\frac{2-\\gamma L}{\\gamma L-1} \\Big] {}\\geq{} 0.\n      \\end{align*}%\n        %\n      \\item \\textbf{Case $\\kappa < 0$:} it is sufficient to show \n    \\begin{align*}\n        \\frac{1+2i \\frac{2-\\gamma L}{\\gamma L -1}}{(1-\\gamma L)^{2i}}\n            {}\\geq{} 1 {}\\geq{}\n        -\\kappa \\frac{-1+2i (1 + \\frac{1}{1-\\kappa \\gamma L})}{(1-\\kappa \\gamma L)^{2i}}.\n    \\end{align*}\n      The \\lhs holds because $\\gamma L \\in (1,2)$ and $(1-\\gamma L)^{2i}<1$. The \\rhs is equivalent to:%\n      \\begin{align*}\n          (1-\\kappa \\gamma L)^{2i} {}\\geq{} -\\kappa \\Big[-1+2i \\Big(1 + \\frac{1}{1-\\kappa \\gamma L}\\Big)\\Big].\n      \\end{align*}\nUsing $\\gamma L \\in (1,2)$ and the second order Taylor approximation for $(1-\\kappa)^{2i}$, we get the bounds:%\n      \\begin{align*}\n          (1-\\kappa \\gamma L)^{2i} {}\\geq{}& (1-\\kappa)^{2i} \n             {}\\geq{} \n         1 + 2i (-\\kappa) + i(2i-1) \\kappa^2 \\\\\n         %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n          -\\kappa \\Big[-1+2i \\Big(1 + \\frac{1}{1-\\kappa \\gamma L}\\Big)\\Big]\n            {}\\leq{}&\n          -\\kappa \\Big[-1+2i \\left(1 + \\frac{1}{1-\\kappa}\\right)\\Big].\n      \\end{align*}\nTherefore, it is sufficient to show:%\n$$\n    1 + 2i (-\\kappa) + i(2i-1) \\kappa^2 {}\\geq{} \n    -\\kappa \\Big[-1+2i \\Big(1 + \\frac{1}{1-\\kappa}\\Big)\\Big],%\n$$%\nwhich equivalently rewrites as the following inequality valid for all $i \\geq 1$:\n$$\n    \\kappa^2 i(i-1) + \\Big( \\kappa i + \\frac{1}{1-\\kappa} \\Big)^2 + \\frac{(1-\\kappa)^3-1}{(1-\\kappa)^2} \\geq 0. {}\\qquad \\qed{}%\n$$%\n\\end{enumerate}\n\\end{proof}","sha256_utf8":"13017283f3ee0a4b3b7459b520ef457560e6a5262505bae7c464b86d593004cb"},{"start":187266,"end":187529,"start_line":2816,"end_line":2818,"raw_tex":"\\begin{lemma} \\label{lemma:unique_delta_k_neg_thr}\nLet $\\kappa \\in (-\\infty, 1)$ and $\\gamma L \\in (1,\\gbari[\\infty])$. Then there exists an index $k$ such that $\\delta_k(\\gamma L, \\kappa) < 0$ and $\\delta_j(\\gamma L, \\kappa) < 0$ for all $j > k$.    \n\\end{lemma}","sha256_utf8":"b6c6c135e33ebaf929b1cc6297e05348c755f79a6db1b4980834bfbaf962d4de"},{"start":187530,"end":188981,"start_line":2819,"end_line":2838,"raw_tex":"\\begin{proof}\n    The sequence $\\delta_i$ is a difference of two monotone sequences: $\\frac{2-\\kappa \\gamma L}{(1-\\kappa \\gamma L)^{2i}}$ and $\\frac{2-\\kappa \\gamma L}{(1-\\kappa \\gamma L)^{2i}}$, respectively. The former is decreasing for $\\kappa < 0$ (weakly convex functions) and increasing for $\\kappa \\geq 0$ (convex functions), whereas the later is always decreasing. In particular, $\\delta_0(\\gamma L, \\kappa) > 0$. \n    Because $(1-\\gamma \\mu)^2 {}\\leq{} (1-\\gamma L)^2$, to the limit it holds $\\lim_{k \\rightarrow \\infty} \\delta_k(\\gamma L, \\kappa) = -\\infty$ and there exists some index $k$ with $\\delta_k(\\gamma L, \\kappa) < 0$. We prove by contradiction that $\\delta_j(\\gamma L, \\kappa) < 0$ for all $j > k$; assume there exists some index $l > k$ such that $\\delta_l(\\gamma L, \\kappa) \\geq 0$. The following inequalities hold:%\n\\begin{align*}\n    \\delta_{k} < 0 \\,:\\,& \n        \\frac{2-\\gamma \\mu}{(1-\\gamma \\mu)^{2k}} \n            {}<{} \n        \\frac{2-\\gamma L}{(1-\\gamma L)^{2k}} \\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\delta_{l} > 0 \\,:\\,& \n        \\frac{2-\\gamma \\mu}{(1-\\gamma \\mu)^{2l}} \n            {}>{} \n        \\frac{2-\\gamma L}{(1-\\gamma L)^{2l}}.\n\\end{align*}\nBy multiplying the inequalities and performing the simplifications we get:%\n\\begin{align*}\n    (1-\\gamma \\mu)^{2(k-l)} > (1-\\gamma L)^{2(k-l)},\n\\end{align*}\nwhich implies, due to $(1-\\gamma \\mu)^2 {}>{} (1-\\gamma L)^2$, that $k > l$, contradiction! \\qed\n\\end{proof}","sha256_utf8":"ca6353a12ce44a740a47f4471f8195bc4bdf8cdb6075df7ce25881390f857ddc"},{"start":188985,"end":192418,"start_line":2840,"end_line":2893,"raw_tex":"\\begin{proof}[\\textbf{\\cref{prop:properties_step_size_thr}}] \\label{appendix:proof_properties_step_size_thr} \n\\begin{enumerate}[align=right,itemindent=1em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]\n\\item Case $k=0$ holds by definition since $\\gbari[0]=1$. For any $k \\geq 1$, by using \\cref{lemma:unique_delta_k_neg_thr}, condition $\\TigLmu[k][\\gbari[k][]] = 0$ implies $\\delta_k(\\gamma L, \\kappa) \\leq 0$ and hence $\\delta_{k+1}(\\gamma L, \\kappa) < 0$. Furthermore, $\\TigLmu[k+1][\\gbari[k][]] {} \\leq {} \\TigLmu[k][\\gbari[k][]] = 0$. Then it holds:\n\\begin{align*}\n    \\TigLmu[k+1][\\gbari[k][]] {} \\leq {} \n    \\TigLmu[k][\\gbari[k][]] = 0 \n    {}={} \\TigLmu[k+1][\\gbari[k+1][]].\n\\end{align*}\nThe monotonicity of $\\TigLmu$ in stepsize $\\gamma L$ (see \\cref{prop:monotonicity_Tk}) implies $\\gbari[k] < \\gbari[k+1]$.\n\n\\item Because stepsize thresholds sequence is monotone, it has a limit.\n\\begin{enumerate}[align=right,itemindent=1.5em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]\n    %%%%%%%%%%%%%%%%%%%%%%%%%% 2. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\item \n    \\textbf{Case $\\kappa > 0$.} By definition, $\\gbari[k] < \\frac{2}{1+\\kappa} < 2$ for all integers $k\\geq 1$ and condition $\\TigLmu[k][\\gbari[k]]=0$ implies:%\n    \\begin{align*}\n        \\left(\\frac{1-\\gbari}{1-\\kappa \\gbari}\\right)^{2k}\n            {}={}& \n        \\kappa + (1-\\kappa) (1-\\gbari)^{2k}\n            {}\\Leftrightarrow{} \\\\\n            %%%%%%%%%%%%%%%%%%%%%%%%%%%%\n        \\left|\\frac{1-\\gbari}{1-\\kappa \\gbari}\\right|\n            {}={}&\n        \\exp \\left[ \\frac{\\ln \\left(\\kappa + (1-\\kappa) (1-\\gbari)^{2k}\\right)}{2k} \\right].\n    \\end{align*}%\n    To the limit $k \\rightarrow \\infty$, the \\rhs equals $1$ because $(1-\\gbari)^{2k}\\rightarrow 0$, hence $|1-\\gbari|=|1-\\kappa \\gbari|$, which implies the solution $\\gbari[\\infty]=\\frac{2}{1+\\kappa}$. \\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\item \\textbf{Case $\\kappa < 0$.} By definition, $\\gbari < 2$ for all integers $k\\geq 1$ and condition $\\TigLmu[k][\\gbari[k]]=0$ implies:\n    \\begin{align*}\n    (1-\\gbari)^{-2k} \n        {}={}&\n    1 + \\frac{1 - (1-\\kappa \\gbari)^{-2k}}{-\\kappa}\n        {}\\Leftrightarrow{} \\\\\n    % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\left|1-\\gbari\\right| \n        {}={}&\n    \\exp \\Bigg[ \\frac{ -\\ln \\Big(1 + \\frac{1 - (1-\\kappa \\gbari)^{-2k}}{-\\kappa}\\Big)}{2k} \\Bigg].\n\\end{align*}\n To the limit $k \\rightarrow \\infty$, the \\rhs equals $1$ because $(1-\\kappa \\gbari)^{-2k} \\rightarrow 0$ due to $\\kappa < 0$. Hence, $\\left|1-\\gbari\\right|=1$, with the solution $\\gbari[\\infty]=2$.%\n %\n\\item \\textbf{Case $\\kappa = 0$.} By definition, $\\gbari[k][(0)] < 2$ and, similarly to the case $\\kappa < 0$:%\n \\begin{align*}\n     \\left|1-\\gbari\\right|\n        {}={}&\n     \\left[ 1 + 2 k \\gbari \\right]^{\\frac{-1}{2k}}\n        {}={}\n    \\exp \\Big[ \\frac{-\\ln \\big( 1 + 2 k \\gbari \\big) }{2k} \\Big].\n \\end{align*}\n  To the limit $k \\rightarrow \\infty$, the \\rhs equals $1$, hence $\\left|1-\\gbari\\right|=1$, with the solution $\\gbari[\\infty]=2$.\n\\end{enumerate}%\n%\n\\item Condition $\\gamma L \\in [\\gbari[k], \\gbari[k+1])$ implies $\\TigLmu[k][\\gbari[k]] \\geq 0$ and $\\TigLmu[k+1][\\gbari[k]] < 0$. Because $\\delta_i(\\gamma L, \\kappa) < 0$ for all $i \\geq k$, we have (i) $\\TigLmu[i][\\gbari[k][]] {}>{} 0$ for $i=1,\\dots,k-1$ (since $k$ is uniquely defined) and (ii) $\\TigLmu[i][\\gbari[k][]] {}<{} 0$ for $i\\geq k+1$. {}\\qed{}%\n\\end{enumerate}\n\\end{proof}","sha256_utf8":"d7f0e8a15454788ddbc603c8fcf1c88b1d39b6617f5a1035bcff06c7b9a446f7"},{"start":211060,"end":213294,"start_line":3231,"end_line":3249,"raw_tex":"\\begin{proof}[\\textbf{\\cref{thm:interp_hypo}}]%\n    The proof follows the same steps outlined in \\cite[Theorem 4]{taylor_smooth_2017}, with a minimal extension to accommodate $\\mu < 0$. This extension impacts only the first step of the demonstration, known as minimal curvature subtraction, which involves shifting the lower curvature of the function to $0$, effectively converting it into a smooth convex function. This adjustment remains valid for $\\mu < 0$, as showed further on. By applying the \\textit{gradient inequality} for smooth convex functions: $h(x) {} \\geq {} h(y) + \\left \\langle \\nabla h(y), x-y \\right \\rangle, \\, \\forall x,y \\in \\mathbb{R}^d$ for $h = \\frac{L}{2}\\|\\cdot\\|^2-f$ and $h = f - \\frac{\\mu}{2}\\|\\cdot\\|^2$, we obtain:%\n    \\begin{align}\\label{eq:def_hypoconvex_smooth}\n    \\tfrac{\\mu}{2} \\|x-y\\|^2 \n        {} \\leq {} \n    f(x) - f(y) - \\langle \\nabla f(y), x-y \\rangle \n        {} \\leq {} \n    \\tfrac{L}{2} \\|x-y\\|^2 \\quad \\forall x, y \\in \\mathbb{R}^d.\n\\end{align}%        \n    Let $\\bar{f} \\in \\mathcal{F}_{0,L-\\mu}$, $\\bar{f}(x){}\\coloneqq{}f(x)-\\tfrac{\\mu}{2}\\|x\\|^2$, with $\\nabla \\bar{f}(x)=\\nabla f(x)-\\mu x$; by replacing $f$ with $\\bar{f}$ in the quadratic bounds inequality \\eqref{eq:def_hypoconvex_smooth} it holds:%\n$$\n    % \\frac{\\mu}{2} \\|x-y\\|^2 {}\\leq{}& f(x) - f(y) - \\langle \\nabla f(y), x-y \\rangle {}\\leq{} \\frac{L}{2} \\|x-y\\|^2 \\nonumber \\\\\n    % & \\frac{\\mu}{2} \\|x-y\\|^2 {}\\leq{} \\big(f(x)-\\frac{\\mu}{2} \\|x\\|^2\\big) - \\big(f(y)-\\frac{\\mu}{2} \\|y\\|^2\\big) - \\langle \\nabla \\big(f(y)-\\mu y\\big), x-y \\rangle {}\\leq{} \\frac{L}{2} \\|x-y\\|^2 \\nonumber \\\\\n    % \\frac{\\mu}{2} \\|x-y\\|^2 {}\\leq{}& \\bar{f}(x) - \\bar{f}(y)- \\langle \\nabla \\bar{f}(y), x-y \\rangle + \\frac{\\mu}{2} \\|x-y\\|^2 {}\\leq{} \\frac{L}{2} \\|x-y\\|^2 \\nonumber \\\\\n    0 {}\\leq{} \\bar{f}(x) - \\bar{f}(y)- \\langle \\nabla \\bar{f}(y), x-y \\rangle {}\\leq{} \\tfrac{L-\\mu}{2} \\|x-y\\|^2.\n$$\nHence, the set $\\{(x_i, g_i, f_i)\\}_{i \\in \\mathcal{I}}$ is $\\mathcal{F}_{\\mu,L}$-interpolable if and only if the set $\\{(x_i, g_i-\\mu x_i, f_i-\\tfrac{\\mu}{2}\\|x_i\\|^2)\\}_{i \\in \\mathcal{I}}$ is $\\mathcal{F}_{0, L-\\mu}$-interpolable. \nSubsequently, the proof continues identically to \\cite[Theorem 4]{taylor_smooth_2017}.\n\\qed\\end{proof}","sha256_utf8":"a78d83ee393d5ab74c6f31878764fcf44b1391f13bd40f82bd5aa88ac65ea2c0"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"强凸情形全步长收敛界","source_units":[{"start":39285,"end":39861,"start_line":562,"end_line":576,"raw_tex":"\\begin{theorem}\\label{thm:strg_cvx_rate}%\nLet $f \\in \\mathcal{F}_{\\mu, L}$, with $\\mu \\in (0, L)$, and consider $N$ iterations of \\eqref{eq:GM_it_fixed_steps} with stepsizes $\\gamma L \\in (0, 2)$ starting from $x_0$. Then the following bound holds:%\n\\begin{align}\\label{eq:strg_cvx_full_ct}\n\\begin{aligned}\n    \\frac{1}{2L} \\|\\nabla f(x_N)\\|^2\n        {} \\leq {}  \n    \\frac{f(x_0)-f_*}\n    { 1 + \\gamma L \\min \\big\\{\n    \\frac{-1+(1-\\gamma \\mu)^{-2N}}{\\gamma \\mu}\n    \\,,\\,\n    \\frac{-1+(1-\\gamma L)^{-2N}}{\\gamma L}\n         \\big\\}}.\n\\end{aligned}\n\\end{align}%\n\\end{theorem}","sha256_utf8":"09f5d4ba7a1261848383af681852475f9855fa710a5f066300f2a71c3b491edb"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"强凸收敛界完整证明","source_units":[{"start":141130,"end":145878,"start_line":2038,"end_line":2142,"raw_tex":"\\begin{proof}[\\textbf{\\cref{thm:strg_cvx_rate}}]\\label{proof:thm:strg_cvx_rate}%\nWe split the proof over stepsize intervals:%\n\\[\n    (0,2) \n        {}={}\n    \\big(0, \\gbari[1]\\big] \\cup\n    \\Big( \\mathop{\\cup}\\limits_{k=1}^{N-1} \\big[\\gbari[k],\\gbari[k+1]\\big)\\Big)\n    {}\\cup{} \\big[\\gbari[N],\\tfrac{2}{1+\\kappa}\\big) \n    {}\\cup{} \\big[\\tfrac{2}{1+\\kappa}, 2\\big).\n\\]%\n%\n\\begin{enumerate}[align=right,itemindent=1em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]%\n\\item \\textbf{Case $\\gamma L \\in \\big(0, \\gbari[1]\\big]$}. By telescoping \\eqref{eq:dist_1_mu_1_step_decrease} for indices $i=0,\\dots,N-1$:%\n$$\n        \\frac{f_{0} - f_{N}}{\\gamma} \n            {}\\geq{}\n        \\Ei[N][\\m] \\frac{\\|g_{N}\\|^2}{2} {}+{}        \n            \\sum_{i=0}^{N-1}\n            \\frac{ {-1} + (\\m+\\n) \\Ei[i+1][\\m] }{\\m-\\n} \\frac{\\|g_{i+1}-\\m g_{i}\\|^2}{2},\n$$%\nwhere all mixed terms have nonnegative coefficients because $\\gamma L \\in \\big(0,  \\gbari[1]\\big]$ (see \\cref{lemma:dist_1_mu_1_step_decrease}). Hence they can be neglected to obtain the lower bound of type \\eqref{eq:I^k_N_general_ineq}:\n\\begin{align}\\label{eq:sufficient_decrease_N_its_mu_strongly_convex_towards_rate_no_dist_2}\n    \\frac{f_0 - f_N}{\\gamma}\n        {}\\geq{}\n    \\Ei[N][\\m] \\frac{\\|g_{N}\\|^2}{2}.\n\\end{align}%:\n\\item \\textbf{Case $\\gamma L \\in \\big[\\gbari[k],\\gbari[k+1]\\big)$}, with some $k \\in \\{1,\\dots,N-1\\}$. By telescoping \\eqref{eq:dist_1_mu_1_step_decrease} for indices $i=k+1,\\dots,N-1$:%\n$$\n\\begin{aligned}\n        \\frac{f_{k+1} {-} f_{N}}{\\gamma} \n            {}\\geq{} &\n        {-}\\Ei[k+1][\\m] \\frac{\\|g_{k+1}\\|^2}{2} {+}\n        \\Ei[N][\\m] \\frac{\\|g_{N}\\|^2}{2} {+} \\\\\n        & \\quad     \\sum_{i{=}k{+}1}^{N{-}1}\n            \\frac{ {-1} {+} (\\m{+}\\n) \\Ei[i{+}1][\\m] }{\\m{-}\\n} \\frac{\\|g_{i+1}{-}\\m g_{i}\\|^2}{2},\n\\end{aligned}\n$$%\nwith nonnegative weights of the mixed terms since $\\gamma L \\in \\big(0,  \\gbari[k+1]\\big)$ (see \\cref{lemma:dist_1_mu_1_step_decrease}). After appending it to \\eqref{eq:general_ineq_proof_to_linear_mu_compact} from \\cref{lemma:general_lemma_ct_h_to_Linear_mu}, we obtain:%\n$$\n\\begin{aligned}\n    \\frac{f_0 {-} f_N}{\\gamma}\n        {}\\geq{}\n    \\Ei[N][\\m] \\frac{\\|g_{N}\\|^2}{2} +\n    \\frac{-\\n^2 \\, \\Tinm[k+1]}{(\\m-\\n)^2} \\, \n    \\frac{\\| g_{k+1} - \\m g_{k} \\|^2}{2} +     \\\\\n    \\sum_{i=k+1}^{N-1}\n            \\frac{ {-1} + (\\m+\\n) \\Ei[i+1][\\m] }{\\m-\\n} \\frac{\\|g_{i+1}-\\m g_{i}\\|^2}{2}.\n\\end{aligned}\n$$\nSince $\\Tinm[k+1] < 0$, all mixed terms can be neglected to get a \\eqref{eq:I^k_N_general_ineq}-type inequality:%\n\\begin{align}\\label{eq:sufficient_decrease_N_its_mu_strongly_convex_towards_rate}\n    \\frac{f_0 - f_N}{\\gamma}\n        {}\\geq{}\n    \\Ei[N][\\m] \\frac{\\|g_{N}\\|^2}{2}.\n\\end{align}%\n%\n \\item \\textbf{Case $\\gamma L \\in \\big[\\gbari[N],\\frac{2}{1+\\kappa}\\big)$.} From \\cref{lemma:general_lemma_ct_h_to_Linear} with $k=N-1$ it holds:%\n$$\n   \\frac{f_0 - f_N}{\\gamma}\n        {}\\geq{}\n    \\Ei[N][\\n] \\frac{\\|g_N\\|^2}{2} + \n    \\frac{\\m^2 \\, \\Tinm[N]}{(\\m-\\n)^2} \\, \n    \\frac{\\| g_N - \\n g_{N-1} \\|^2}{2}.\n$$%\nSince $\\TigLmu[N] \\geq 0$, by neglecting the mixed term we obtain a \\eqref{eq:I^k_N_general_ineq}-type inequality:\n\\begin{align}\\label{eq:wc_linear_L}\n    \\frac{f_0 - f_N}{\\gamma}\n        {}\\geq{}\n    \\Ei[N][\\n] \\frac{\\|g_{N}\\|^2}{2}.\n\\end{align}%\n%\n\\item \\textbf{Case $\\gamma L \\in \\big[\\frac{2}{1+\\kappa},2\\big)$.} By telescoping \\cref{lemma:dist_1_L_1_step_decrease} for $i=0,1,\\dots,N-1$ it holds:%\n$$\n    \\frac{f_{0} - f_{N}}{\\gamma} \n                {}\\geq{}\n            \\Ei[N][\\n] \\frac{\\|g_{i}\\|^2}{2} +\n            \\sum_{i=0}^{N-1} \\frac{ 1 - (\\m+\\n) \\Ei[i+1][\\n] }{\\m-\\n} \\frac{\\|g_{i+1}-\\n g_{i}\\|^2}{2},\n$$%\nwhere the mixed terms can be neglected and then the inequality reduces to \\eqref{eq:wc_linear_L}.%\n\\end{enumerate}%\nSince the transition between the two linear regimes is given by the sign of $\\Tinm[N]=\\Ei[N][\\m]{-}\\Ei[N][\\n]$, we can glue together expressions \\eqref{eq:sufficient_decrease_N_its_mu_strongly_convex_towards_rate_no_dist_2}, \\eqref{eq:sufficient_decrease_N_its_mu_strongly_convex_towards_rate} and \\eqref{eq:wc_linear_L}:%\n$$\n    \\frac{f_0 - f_N}{\\gamma}\n        {}\\geq{}\n    \\min\\left\\{ \\Ei[N][\\m] \\,,\\, \\Ei[N][\\n]\\right\\} \\frac{\\|g_{N}\\|^2}{2},\n$$%\nequivalent to:\n$$\n    f_0 - f_N\n        {}\\geq{}\n    \\min\n    \\Big\\{ \\frac{{-1} +(1-\\gamma \\mu)^{-2N}}{\\mu} \\,,\\, \n    \\frac{{-1} +(1-\\gamma L)^{-2N}}{L}\n    \\Big\\} \\frac{\\|g_{N}\\|^2}{2},\n$$%\nwhich implies the bound \\eqref{eq:strg_cvx_full_ct} and the complementary one:\n$$\n\\begin{aligned}\n    \\frac{1}{2L} \\|\\nabla f(x_N)\\|^2\n        {} \\leq {}  \n    \\frac{f(x_0)-f(x_N)}\n    { \\min\\big\\{ \\frac{L}{\\mu} \\left[{-1}+(1-\\gamma \\mu)^{-2N}\\right] \\,,\\, {-1}+(1-\\gamma L)^{-2N} \\big\\} }. {}\\qquad{}\\qed \n\\end{aligned}%\n$$%\n\\end{proof}","sha256_utf8":"edf1d2965112e5b978e4384502750d0aa0b119d495010cccaafecb3f12fe87d0"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"划分四类步长区间","source_units":[{"start":141130,"end":141501,"start_line":2038,"end_line":2047,"raw_tex":"\\begin{proof}[\\textbf{\\cref{thm:strg_cvx_rate}}]\\label{proof:thm:strg_cvx_rate}%\nWe split the proof over stepsize intervals:%\n\\[\n    (0,2) \n        {}={}\n    \\big(0, \\gbari[1]\\big] \\cup\n    \\Big( \\mathop{\\cup}\\limits_{k=1}^{N-1} \\big[\\gbari[k],\\gbari[k+1]\\big)\\Big)\n    {}\\cup{} \\big[\\gbari[N],\\tfrac{2}{1+\\kappa}\\big) \n    {}\\cup{} \\big[\\tfrac{2}{1+\\kappa}, 2\\big).\n\\]%\n","sha256_utf8":"e9a4891bf1695bc94bbed6ee0c3004cb725caa279f5a984bc61dfe6f4637c09d"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"短步长望远镜求和","source_units":[{"start":141501,"end":142408,"start_line":2048,"end_line":2063,"raw_tex":"%\n\\begin{enumerate}[align=right,itemindent=1em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]%\n\\item \\textbf{Case $\\gamma L \\in \\big(0, \\gbari[1]\\big]$}. By telescoping \\eqref{eq:dist_1_mu_1_step_decrease} for indices $i=0,\\dots,N-1$:%\n$$\n        \\frac{f_{0} - f_{N}}{\\gamma} \n            {}\\geq{}\n        \\Ei[N][\\m] \\frac{\\|g_{N}\\|^2}{2} {}+{}        \n            \\sum_{i=0}^{N-1}\n            \\frac{ {-1} + (\\m+\\n) \\Ei[i+1][\\m] }{\\m-\\n} \\frac{\\|g_{i+1}-\\m g_{i}\\|^2}{2},\n$$%\nwhere all mixed terms have nonnegative coefficients because $\\gamma L \\in \\big(0,  \\gbari[1]\\big]$ (see \\cref{lemma:dist_1_mu_1_step_decrease}). Hence they can be neglected to obtain the lower bound of type \\eqref{eq:I^k_N_general_ineq}:\n\\begin{align}\\label{eq:sufficient_decrease_N_its_mu_strongly_convex_towards_rate_no_dist_2}\n    \\frac{f_0 - f_N}{\\gamma}\n        {}\\geq{}\n    \\Ei[N][\\m] \\frac{\\|g_{N}\\|^2}{2}.\n\\end{align}%:\n","sha256_utf8":"faed0e65638c0e4f765ae05a2e5921942f6067ce1d8540402de832ed072062bf"}]},{"id":"S005","semantic_type":"proof_step","kind":"proof_step","title":"中间区间拼接下降界","source_units":[{"start":142408,"end":143840,"start_line":2064,"end_line":2092,"raw_tex":"\\item \\textbf{Case $\\gamma L \\in \\big[\\gbari[k],\\gbari[k+1]\\big)$}, with some $k \\in \\{1,\\dots,N-1\\}$. By telescoping \\eqref{eq:dist_1_mu_1_step_decrease} for indices $i=k+1,\\dots,N-1$:%\n$$\n\\begin{aligned}\n        \\frac{f_{k+1} {-} f_{N}}{\\gamma} \n            {}\\geq{} &\n        {-}\\Ei[k+1][\\m] \\frac{\\|g_{k+1}\\|^2}{2} {+}\n        \\Ei[N][\\m] \\frac{\\|g_{N}\\|^2}{2} {+} \\\\\n        & \\quad     \\sum_{i{=}k{+}1}^{N{-}1}\n            \\frac{ {-1} {+} (\\m{+}\\n) \\Ei[i{+}1][\\m] }{\\m{-}\\n} \\frac{\\|g_{i+1}{-}\\m g_{i}\\|^2}{2},\n\\end{aligned}\n$$%\nwith nonnegative weights of the mixed terms since $\\gamma L \\in \\big(0,  \\gbari[k+1]\\big)$ (see \\cref{lemma:dist_1_mu_1_step_decrease}). After appending it to \\eqref{eq:general_ineq_proof_to_linear_mu_compact} from \\cref{lemma:general_lemma_ct_h_to_Linear_mu}, we obtain:%\n$$\n\\begin{aligned}\n    \\frac{f_0 {-} f_N}{\\gamma}\n        {}\\geq{}\n    \\Ei[N][\\m] \\frac{\\|g_{N}\\|^2}{2} +\n    \\frac{-\\n^2 \\, \\Tinm[k+1]}{(\\m-\\n)^2} \\, \n    \\frac{\\| g_{k+1} - \\m g_{k} \\|^2}{2} +     \\\\\n    \\sum_{i=k+1}^{N-1}\n            \\frac{ {-1} + (\\m+\\n) \\Ei[i+1][\\m] }{\\m-\\n} \\frac{\\|g_{i+1}-\\m g_{i}\\|^2}{2}.\n\\end{aligned}\n$$\nSince $\\Tinm[k+1] < 0$, all mixed terms can be neglected to get a \\eqref{eq:I^k_N_general_ineq}-type inequality:%\n\\begin{align}\\label{eq:sufficient_decrease_N_its_mu_strongly_convex_towards_rate}\n    \\frac{f_0 - f_N}{\\gamma}\n        {}\\geq{}\n    \\Ei[N][\\m] \\frac{\\|g_{N}\\|^2}{2}.\n\\end{align}%\n","sha256_utf8":"4358652f3b0b8642e32ace0a275d68d8b486cf3022cde4258df3e6e327a1411c"}]},{"id":"S006","semantic_type":"proof_step","kind":"proof_step","title":"阈值以上采用上曲率界","source_units":[{"start":143840,"end":144410,"start_line":2093,"end_line":2107,"raw_tex":"%\n \\item \\textbf{Case $\\gamma L \\in \\big[\\gbari[N],\\frac{2}{1+\\kappa}\\big)$.} From \\cref{lemma:general_lemma_ct_h_to_Linear} with $k=N-1$ it holds:%\n$$\n   \\frac{f_0 - f_N}{\\gamma}\n        {}\\geq{}\n    \\Ei[N][\\n] \\frac{\\|g_N\\|^2}{2} + \n    \\frac{\\m^2 \\, \\Tinm[N]}{(\\m-\\n)^2} \\, \n    \\frac{\\| g_N - \\n g_{N-1} \\|^2}{2}.\n$$%\nSince $\\TigLmu[N] \\geq 0$, by neglecting the mixed term we obtain a \\eqref{eq:I^k_N_general_ineq}-type inequality:\n\\begin{align}\\label{eq:wc_linear_L}\n    \\frac{f_0 - f_N}{\\gamma}\n        {}\\geq{}\n    \\Ei[N][\\n] \\frac{\\|g_{N}\\|^2}{2}.\n\\end{align}%\n","sha256_utf8":"884a7666ec1c43cb871371794247a5925856327b95be7f038e69c1935fc703e1"}]},{"id":"S007","semantic_type":"proof_step","kind":"proof_step","title":"最大步长区间求和","source_units":[{"start":144410,"end":144898,"start_line":2108,"end_line":2117,"raw_tex":"%\n\\item \\textbf{Case $\\gamma L \\in \\big[\\frac{2}{1+\\kappa},2\\big)$.} By telescoping \\cref{lemma:dist_1_L_1_step_decrease} for $i=0,1,\\dots,N-1$ it holds:%\n$$\n    \\frac{f_{0} - f_{N}}{\\gamma} \n                {}\\geq{}\n            \\Ei[N][\\n] \\frac{\\|g_{i}\\|^2}{2} +\n            \\sum_{i=0}^{N-1} \\frac{ 1 - (\\m+\\n) \\Ei[i+1][\\n] }{\\m-\\n} \\frac{\\|g_{i+1}-\\n g_{i}\\|^2}{2},\n$$%\nwhere the mixed terms can be neglected and then the inequality reduces to \\eqref{eq:wc_linear_L}.%\n\\end{enumerate}%\n","sha256_utf8":"b8ba24f4631e24b31e8a8183df7341ae2d332503963a1506fb434e16f35b698a"}]},{"id":"S008","semantic_type":"proof_step","kind":"proof_step","title":"按符号合并两种速率","source_units":[{"start":144898,"end":145541,"start_line":2118,"end_line":2132,"raw_tex":"Since the transition between the two linear regimes is given by the sign of $\\Tinm[N]=\\Ei[N][\\m]{-}\\Ei[N][\\n]$, we can glue together expressions \\eqref{eq:sufficient_decrease_N_its_mu_strongly_convex_towards_rate_no_dist_2}, \\eqref{eq:sufficient_decrease_N_its_mu_strongly_convex_towards_rate} and \\eqref{eq:wc_linear_L}:%\n$$\n    \\frac{f_0 - f_N}{\\gamma}\n        {}\\geq{}\n    \\min\\left\\{ \\Ei[N][\\m] \\,,\\, \\Ei[N][\\n]\\right\\} \\frac{\\|g_{N}\\|^2}{2},\n$$%\nequivalent to:\n$$\n    f_0 - f_N\n        {}\\geq{}\n    \\min\n    \\Big\\{ \\frac{{-1} +(1-\\gamma \\mu)^{-2N}}{\\mu} \\,,\\, \n    \\frac{{-1} +(1-\\gamma L)^{-2N}}{L}\n    \\Big\\} \\frac{\\|g_{N}\\|^2}{2},\n$$%\n","sha256_utf8":"d8ce7b99a3fc9cc4e2c79c32bfabc5d6a41eb68efc130c2c51508c391e9d629c"}]},{"id":"S009","semantic_type":"proof_step","kind":"proof_step","title":"转为最优值差收敛界","source_units":[{"start":145541,"end":145878,"start_line":2133,"end_line":2142,"raw_tex":"which implies the bound \\eqref{eq:strg_cvx_full_ct} and the complementary one:\n$$\n\\begin{aligned}\n    \\frac{1}{2L} \\|\\nabla f(x_N)\\|^2\n        {} \\leq {}  \n    \\frac{f(x_0)-f(x_N)}\n    { \\min\\big\\{ \\frac{L}{\\mu} \\left[{-1}+(1-\\gamma \\mu)^{-2N}\\right] \\,,\\, {-1}+(1-\\gamma L)^{-2N} \\big\\} }. {}\\qquad{}\\qed \n\\end{aligned}%\n$$%\n\\end{proof}","sha256_utf8":"5910a1347e4b6fba443075be2f7cf8266f9bd8994624c6516b2d3cc7975bdf2d"}]},{"id":"S010","semantic_type":"problem_setting","kind":"problem_setting","title":"无约束最小化问题","source_units":[{"start":15890,"end":15983,"start_line":329,"end_line":330,"raw_tex":"We consider the unconstrained optimization problem %\n$$\\minimize_{x \\in \\mathbb{R}^d} f(x),$$","sha256_utf8":"cf2813890a35cd3cda8b1b026a9ad0623b789920c09c04963c35e27e6a5bf750"},{"start":15985,"end":16072,"start_line":331,"end_line":331,"raw_tex":"where $f$ belongs to the class of functions $\\mathcal{F}_{\\mu, L}$, as defined below.%\n","sha256_utf8":"f6724200c232a933991134addeb8c6b30d5ffea8867dbf8ff3e92702d93b3cf9"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"上下曲率函数类","source_units":[{"start":16074,"end":16541,"start_line":333,"end_line":339,"raw_tex":"\\begin{definition}\\label{def:upper_lower_curvature} \nLet $L \\in (0, \\infty)$ and $\\mu\\in (-\\infty,L)$. Then a function $f\\colon\\mathbb{R}^d\\rightarrow \\mathbb{R}$ belongs to the class of functions $\\mathcal{F}_{\\mu,L}$ if it has:%\n\\begin{enumerate}[label=(\\roman*)]%\n    \\item upper curvature $L$, i.e., $ \\tfrac{L}{2} \\|\\cdot\\|^2 - f$ is convex;%\n    \\item lower curvature $\\mu$, i.e., $ f - \\tfrac{\\mu}{2} \\|\\cdot\\|^2$ is convex. %\n\\end{enumerate}%\n\\end{definition}","sha256_utf8":"a0c68f257c30eaaf5dfe19770e75782c2cdb61f1b650824a69d597d937d03c3d"}]},{"id":"S012","semantic_type":"notation","kind":"notation","title":"曲率比与正部记号","source_units":[{"start":17328,"end":17489,"start_line":345,"end_line":345,"raw_tex":"We sometimes represent the curvature ratio $\\kappa{}\\coloneqq{}\\frac{\\mu}{L} \\in (-\\infty, 1)$, which, if $\\mu \\geq 0$, signifies the inverse condition number.%\n","sha256_utf8":"c0df0527c53f77be8cf1ffc1defd02851aabff80f4f853026038dfad8a9487bf"},{"start":18743,"end":18818,"start_line":356,"end_line":356,"raw_tex":"We denote $[x]_{+} {}\\coloneqq{} \\max\\{x, 0\\}$ for any $x \\in \\mathbb{R}$.\n","sha256_utf8":"b526ccbabd3c70600d4a28d7b23b355f69e6cbf5aa76bb4c3cebf7bd4f1e1552"}]},{"id":"S013","semantic_type":"algorithm","kind":"algorithm","title":"梯度下降迭代","source_units":[{"start":17490,"end":17837,"start_line":347,"end_line":351,"raw_tex":"\\noindent {\\textbf{Gradient descent.}} \nStarting from some initial point $x_0 \\in \\mathbb{R}^d$, we consider $N$ iterations of gradient descent with fixed stepsizes $\\gamma_i \\in (0, \\tfrac{2}{L})$, defined as:\n\\begin{align}\\label{eq:GM_it_fixed_steps}\\tag{GD}\n    x_{i+1} = x_i - \\gamma_i \\nabla f(x_i), \\quad \\forall i = 0,\\dots,N-1.\n\\end{align}","sha256_utf8":"b62a770d458a6d8eaf93c87c4590aa10f1ba1a0d63f11c599aaa841becd21702"}]},{"id":"S014","semantic_type":"assumption","kind":"assumption","title":"一阶数据与有界下确界","source_units":[{"start":17841,"end":18382,"start_line":353,"end_line":353,"raw_tex":"The constraint on the stepsize values ensures a decrease in the function value after a single iteration. We assume access to a first-order oracle denoted by the set of triplets $\\mathcal{T}{}\\coloneqq{}\\{(x_i,g_i,f_i)\\}_{i \\in \\mathcal{I}} \\subseteq \\mathbb{R}^d\\times\\mathbb{R}^d\\times\\mathbb{R}$, where $\\mathcal{I}$ is an index set, providing information about the iterates, their gradients and the corresponding function values. We assume the function $f$ to be bounded from below and denote $f_* {}\\coloneqq{} \\inf_x f(x) > -\\infty$. %\n","sha256_utf8":"a1bc84cd8061308b7e6d8383f34e36e09e4f114a2a54410b0efd8e203d6e6c9d"}]},{"id":"S015","semantic_type":"notation","kind":"notation","title":"函数值与梯度简记","source_units":[{"start":101158,"end":101441,"start_line":1329,"end_line":1329,"raw_tex":"\\noindent{\\textbf{Notation.}}\\, For readability reasons, we sometimes denote $f_i {=} f(x_i)$ and $g_i {=} \\nabla f(x_i)$. Recall: $\\kappa {=} \\frac{\\mu}{L}$ is the curvature ratio; $\\gamma L \\in (0,2)$ is the (normalized) stepsize and $\\gbari[\\infty] = \\tfrac{2}{1+[\\kappa]_{+}}$.%\n","sha256_utf8":"6dda44fce39aac5f5821f75dae965c4bd10506b4e07e7aee3fe016d12c47abee"}]},{"id":"S016","semantic_type":"definition","kind":"definition","title":"有限和与阈值函数","source_units":[{"start":56454,"end":57222,"start_line":818,"end_line":838,"raw_tex":"\\begin{definition}\\label{def:def_TN}% \nLet $k$ be a positive integer and define function $E_k:\\mathbb{R}\\backslash\\{0\\} \\rightarrow \\mathbb{R}$:%\n\\begin{align}\n    \\Ei{}{}\\coloneqq{}{}\\sum\\limits_{j=1}^{2k} x^{-j} \n        {}={} \n     \\left\\{\n    \\arraycolsep=6pt\n    \\def\\arraystretch{1.25}\n    \\begin{array}{cc}\n      \\frac{-1+x^{-2k}}{1-x}, & x \\neq 1; \\\\\n      2k, & x = 1.\n    \\end{array}\n    \\right. \\label{eq:def_Ei} \\tag{$E_k$}\n\\end{align}%\nLet $V \\coloneqq \\{(\\ell, \\kappa): \\, \\kappa \\in (-\\infty, 1), \\,\\ell \\in (1, \\frac{2}{1+[\\kappa]_{+}})\\}$ and define $T_k: V \\rightarrow \\mathbb{R}$ as %\n\\begin{align}\n    T_k(\\ell,\\kappa)\n        {}{}\\coloneqq{}{}&\n    \\Ei[k][1-\\kappa \\ell] - \\Ei[k][1-\\ell] \\label{eq:def_TN} \\tag{$T_k$}.\n\\end{align}\n\\end{definition}","sha256_utf8":"cce081e27c981de5bdfc7d3c9f93e2b704b583f903027b0ef637014158ec2de3"}]},{"id":"S017","semantic_type":"definition","kind":"definition","title":"步长阈值定义","source_units":[{"start":58002,"end":58572,"start_line":848,"end_line":855,"raw_tex":"\\begin{definition}[Stepsize thresholds]\\label{def:stepsize_thresholds}%\nLet $\\gbari[\\infty]{}\\coloneqq{} \\frac{2}{1+[\\kappa]_{+}}$. The unique roots of $\\TigLmu$ in the interval $(1,\\gbari[\\infty])$ are referred to as stepsize thresholds and denoted by $\\gbari[k]$, where $\\gbari[0] {}{}\\coloneqq{}{} 1$ and:%\n\\begin{align}\\label{def:gamma_bar} \\tag{$\\xoverline{\\gamma L}_k$}\n    \\big\\{\\gbari[k]\\big\\} \n        {}{}\\coloneqq{}{} \n    \\big\\{ \\gamma L \\in (1,\\gbari[\\infty]) \\,|\\, \\TigLmu[k] = 0 \\big\\} {}\\quad{} \\forall k = 1,2,{\\dots}\\,\\,.%\n\\end{align}%\n\\end{definition}","sha256_utf8":"76d4669edfdb11e1c93030fe57446a9c5c30db50c035fb8bb18ea7d6547a7430"}]},{"id":"S018","semantic_type":"notation","kind":"notation","title":"两种收缩因子记号","source_units":[{"start":109185,"end":109666,"start_line":1449,"end_line":1450,"raw_tex":"\\paragraph*{\\textbf{Additional notation. }}\\, For brevity, we sometimes use $\\n{=}\\n(\\gamma L){}{\\coloneqq}{}1{-}\\gamma L \\in (-1, 0)$, $\\m{=}\\m(\\gamma \\mu){}{\\coloneqq}{}1{-}\\gamma \\mu$. Then $T_k(\\gamma L, \\kappa) = E_k(1{-}\\kappa \\gamma L) {-} E_k(1{-}\\gamma L)$ (see \\cref{def:def_TN}), where\n$E_k(x) = \\sum_{j=1}^{2k} x^{-j}$, can be equivalently written as $\\Tinm{}\\coloneqq{}\\TigLmu$, with $\\Tinm=\\Ei[k][\\m]-\\Ei[k][\\n]$. Recall $\\gbari[\\infty] = \\frac{2}{1+[\\kappa]_{+}}$.%\n","sha256_utf8":"7ff7536607e0ce95236cd12472689b65e91a041676c62f90f5c3576e948989e4"}]},{"id":"S019","semantic_type":"definition","kind":"definition","title":"非负阈值项最大指标","source_units":[{"start":109668,"end":109818,"start_line":1452,"end_line":1456,"raw_tex":"\\begin{definition}\\label{def:N_bar}\nLet $\\Nbar\n        {}{}\\coloneqq{}{}\n    \\max \\{ k \\in \\mathbb{N} \\,|\\, \\TigLmu[k] {}\\geq{} 0\\}$.\n\\end{definition}","sha256_utf8":"1ad9dc04a7619a868025bfced50775cd5bdb52a5b981aa51be688d6241e1ec5e"}]},{"id":"S020","semantic_type":"definition","kind":"definition","title":"数据可插值性","source_units":[{"start":89517,"end":89858,"start_line":1206,"end_line":1207,"raw_tex":"\\begin{definition}[{\\cite[Definition 2]{taylor_smooth_2017}}] The set $\\mathcal{T}{}\\coloneqq{}\\{(x_i, g_i, f_i)\\}_{i \\in \\mathcal{I}}$ is called $\\mathcal{F}_{\\mu, L}$-interpolable if and only if there exists a function $f \\in \\mathcal{F}_{\\mu, L}$ such that $\\nabla f(x_i)=g_i$ and $f(x_i)=f_i$ for all $i\\in \\mathcal{I}$.\n\\end{definition}","sha256_utf8":"d5809c3891ab55ca627fa6bf1467e72c6c476dae0b4f99331402d338c88cd51f"}]},{"id":"S021","semantic_type":"definition","kind":"definition","title":"迭代间插值不等式","source_units":[{"start":92345,"end":94363,"start_line":1234,"end_line":1261,"raw_tex":"Interpolation conditions are used in conjunction with the  \\eqref{eq:GM_it_fixed_steps} iterations by substituting $x_{j}=x_{i} - \\sum_{k=i}^{j-1} \\gamma_k \\nabla f(x_k)$, as detailed in \\cref{def:dist-N_interp_cond}. Since the proofs are limited to constant stepsizes, we constrain $\\gamma_k$ to $\\gamma$.%\n\\begin{definition}\\label{def:dist-N_interp_cond}\nInequalities \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} are necessary and sufficient conditions connecting gradient descent iterations separated by $(j-i)$ steps:\n% %%%%%%%%%%%%%%%%%%%%%%% Improved decent version %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n\\begin{align}\\label{eq:distance-N_interpolation_conditions_I_ij}\n    & \\hspace{-.75em}\n    \\begin{aligned}\n    f(x_{i})-f(x_{j})\n        {}\\geq {}  \n    \\gamma \\big \\langle \\nabla f(x_{j}), \\sSum_{k=i}^{j-1} \\nabla f(x_{k}) \\big \\rangle +\n    \\frac{1}{2L} \\left\\|\\nabla f(x_{i}) - \\nabla f(x_{j})\\right\\|^2 + \n    \\\\% {}\\qquad{}\n    \\frac{\\mu}{2L(L{-}\\mu)} \\big\\|\\nabla f(x_{i}) - \\nabla f(x_{j}) - \\gamma L \\sSum_{k=i}^{j-1} \\nabla f(x_{k})\\big\\|^2; \n    \\end{aligned} \\tag{$\\Iij$} \\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    & \\hspace{-.75em}\n    \\begin{aligned} \n    f(x_{j})-f(x_{i}) \n        {}\\geq {}\n    {-}\\gamma \\big \\langle \\nabla f(x_{i}), \\sSum_{k=i}^{j-1} \\nabla f(x_{k}) \\big \\rangle +\n    \\frac{1}{2L} \\left\\|\\nabla f(x_{i}) - \\nabla f(x_{j})\\right\\|^2 + \n    \\\\% {}\\qquad{}\n    \\frac{\\mu}{2L(L{-}\\mu)} \\big\\|\\nabla f(x_{i}) - \\nabla f(x_{j}) - \\gamma L \\sSum_{k=i}^{j-1} \\nabla f(x_{k})\\big\\|^2.\n    \\end{aligned} \\tag{$\\Iji$}\n    \\label{eq:distance-N_interpolation_conditions_I_ji}\n\\end{align}%\nWe refer to inequalities \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} as \\textit{distance-1} or \\textit{distance-2} inequalities if the distances between their indices satisfy $|i-j|\\leq 1$ or $|i-j|\\leq 2$, respectively.%\n\\end{definition}","sha256_utf8":"a1189a5d3fab1f89dd33d1b00d71aa646edb642e58ef788b60e0369095e6266a"}]},{"id":"S022","semantic_type":"notation","kind":"notation","title":"非负加权目标不等式","source_units":[{"start":98944,"end":99744,"start_line":1295,"end_line":1300,"raw_tex":"Inequalities \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} contribute via nonnegative weights \n$\\alphaij[i][j]$ and $\\alphaij[j][i]$, respectively, to form a generalized inequality linking $N$ iterations\\footnote{For linear combination of inequalities, we adopt the convention of writing the summed expression on the left-hand side, while stating the corresponding inequalities explicitly on the right-hand side.} such that there exists nonnegative weights $\\sigma_i \\geq 0$ satisfying:%\n\\begin{align}\\label{eq:I^k_N_general_ineq}\\tag{\\ENQ}\n\\sum_{0\\leq i < j \\leq N} \\big(\\alphaij[i][j] \\Iij[i][j] + \\alphaij[j][i] \\Iij[j][i]\\big)\\colon \\,\\,\n    f(x_0) - f(x_N) {}\\geq{}  \\sum_{i=0}^N \\sigma_i \\frac{\\|\\nabla f(x_i)\\|^2}{2L}.%\n\\end{align}","sha256_utf8":"c4a49dc6189edf56cce83d573ecb823027a7f5e0f118c3d0ab6535053d27b3db"}]},{"id":"S023","semantic_type":"derived_fact","kind":"derived_fact","title":"相邻迭代插值展开","source_units":[{"start":101546,"end":102429,"start_line":1335,"end_line":1352,"raw_tex":"For $|j-i|=1$, inequalities \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} simplify to:%\n\\begin{align} \n% \\hspace{-.75em}\n    f_{i}{-}f_{i+1}\n        {}\\geq {} &\n    \\frac{\\gamma \\mu \\, \\gamma L {-} 2 \\gamma \\mu {+} 1}{L{-}\\mu} \\frac{\\|g_{i}\\|^2}{2} +\n    \\frac{1}{L{-}\\mu} \\frac{\\|g_{i+1}\\|^{2}}{2} +     \n    \\frac{\\gamma L{-}1}{L{-}\\mu} \\langle g_{i}, g_{i+1}\\rangle; \n\\label{eq:hypo_interp_(0,1)} \\tag{\\Iij[i][i+1]}\\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n% \\hspace{-1em}\n    f_{i+1}{-}f_{i}\n        {}\\geq {} &\n    \\frac{\\gamma \\mu \\gamma L {-} 2 \\gamma L {+} 1}{L-\\mu} \\frac{\\|g_{i}\\|^2}{2} +\n    \\frac{1}{L{-}\\mu} \\frac{\\|g_{i+1}\\|^{2}}{2} +     \n    \\frac{\\gamma \\mu {-} 1}{L{-}\\mu} \\langle g_{i}, g_{i+1}\\rangle.\n\\label{eq:hypo_interp_(1,0)} \\tag{\\Iji[i][i+1]}\n\\end{align}","sha256_utf8":"ff43dc7f322e5ee089bf9b10a6fc6c166a96433d0772fbd03693446efe935fb1"}]},{"id":"S024","semantic_type":"definition","kind":"definition","title":"阈值函数增量","source_units":[{"start":184332,"end":184655,"start_line":2758,"end_line":2766,"raw_tex":"\\begin{definition}\n    We use the writing $\\TigLmu[k][\\gamma L] {}={} \\sum_{i=1}^k \\delta_i(\\gamma L, \\kappa)$, where\n    $$\n        \\delta_i(\\gamma L,\\kappa) \n            {}{}\\coloneqq{}{} \n        \\frac{2-\\kappa \\gamma L}{(1-\\kappa \\gamma L)^{2i}} - \n        \\frac{2-\\gamma L}{(1-\\gamma L)^{2i}}.\n    $$ \n\\end{definition}","sha256_utf8":"68890f2b17c8487ea09a74d7626280d3b2dc2d38d882983133cd494ad6e99792"}]},{"id":"S025","semantic_type":"proposition","kind":"proposition","title":"阈值函数严格递增","source_units":[{"start":57603,"end":57761,"start_line":840,"end_line":842,"raw_tex":"\\begin{proposition}\\label{prop:monotonicity_Tk}%\n    The function $\\TigLmu[k][\\cdot][\\kappa]$ is strictly increasing in the first argument.%\n\\end{proposition}","sha256_utf8":"c2f686bffa0cd4b077911fba0f6ddcba0e0eabe1763afa5526f50a8d1e5c30dc"}]},{"id":"S026","semantic_type":"proof","kind":"proof","title":"阈值函数单调性证明","source_units":[{"start":57763,"end":57842,"start_line":843,"end_line":845,"raw_tex":"\\begin{proof}%\nSee \\cref{appendix:proof_prop_monotonicity_Tk}.%\n\\qed\\end{proof}","sha256_utf8":"85ccee2a3ffdba2f98ddc33791dc0b517329ac7c98662f1ac90e82cf203db23e"},{"start":184657,"end":187264,"start_line":2767,"end_line":2814,"raw_tex":"\\begin{proof}[\\textbf{\\cref{prop:monotonicity_Tk}}] \\label{appendix:proof_prop_monotonicity_Tk}\nWe show $\\frac{d\\TigLmu}{d (\\gamma L)} \\geq 0$ for $\\gamma L \\in (1, \\gbari[\\infty])$, which resumes to prove the positivity of\n  \\begin{align*}\n        \\frac{d \\delta_i(\\gamma L, \\kappa)}{d(\\gamma L)}\n            {}{}\\coloneqq{}{}\n      \\frac{-\\kappa \\left( 1 - 2i \\frac{2-\\kappa \\gamma L}{1-\\kappa \\gamma L} \\right) }{(1-\\kappa \\gamma L)^{2i}} + \\frac{ 1 + 2i \\frac{2-\\gamma L}{\\gamma L-1}}{(1-\\gamma L)^{2i}}.\n  \\end{align*}\n  \\begin{enumerate}[align=right,itemindent=1em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]\n      \\item \\textbf{Case $\\kappa \\geq 0$:} we lower bound $\\frac{d \\delta_i(\\gamma L, \\kappa)}{d(\\gamma L)}$ by neglecting the denominators:\n      \\begin{align*}\n          \\delta_i(\\gamma L, \\kappa) \n            {}\\geq{} \n          {-}\\kappa \\Big( 1 - 2i \\frac{2-\\kappa \\gamma L}{1-\\kappa \\gamma L} \\Big) + 1 + 2i \\frac{2-\\gamma L}{\\gamma L-1}\n            {}={}\n          (1-\\kappa) + 2i \\Big[ \\frac{\\kappa (2-\\kappa \\gamma L)}{1-\\kappa \\gamma L}  + \\frac{2-\\gamma L}{\\gamma L-1} \\Big] {}\\geq{} 0.\n      \\end{align*}%\n        %\n      \\item \\textbf{Case $\\kappa < 0$:} it is sufficient to show \n    \\begin{align*}\n        \\frac{1+2i \\frac{2-\\gamma L}{\\gamma L -1}}{(1-\\gamma L)^{2i}}\n            {}\\geq{} 1 {}\\geq{}\n        -\\kappa \\frac{-1+2i (1 + \\frac{1}{1-\\kappa \\gamma L})}{(1-\\kappa \\gamma L)^{2i}}.\n    \\end{align*}\n      The \\lhs holds because $\\gamma L \\in (1,2)$ and $(1-\\gamma L)^{2i}<1$. The \\rhs is equivalent to:%\n      \\begin{align*}\n          (1-\\kappa \\gamma L)^{2i} {}\\geq{} -\\kappa \\Big[-1+2i \\Big(1 + \\frac{1}{1-\\kappa \\gamma L}\\Big)\\Big].\n      \\end{align*}\nUsing $\\gamma L \\in (1,2)$ and the second order Taylor approximation for $(1-\\kappa)^{2i}$, we get the bounds:%\n      \\begin{align*}\n          (1-\\kappa \\gamma L)^{2i} {}\\geq{}& (1-\\kappa)^{2i} \n             {}\\geq{} \n         1 + 2i (-\\kappa) + i(2i-1) \\kappa^2 \\\\\n         %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n          -\\kappa \\Big[-1+2i \\Big(1 + \\frac{1}{1-\\kappa \\gamma L}\\Big)\\Big]\n            {}\\leq{}&\n          -\\kappa \\Big[-1+2i \\left(1 + \\frac{1}{1-\\kappa}\\right)\\Big].\n      \\end{align*}\nTherefore, it is sufficient to show:%\n$$\n    1 + 2i (-\\kappa) + i(2i-1) \\kappa^2 {}\\geq{} \n    -\\kappa \\Big[-1+2i \\Big(1 + \\frac{1}{1-\\kappa}\\Big)\\Big],%\n$$%\nwhich equivalently rewrites as the following inequality valid for all $i \\geq 1$:\n$$\n    \\kappa^2 i(i-1) + \\Big( \\kappa i + \\frac{1}{1-\\kappa} \\Big)^2 + \\frac{(1-\\kappa)^3-1}{(1-\\kappa)^2} \\geq 0. {}\\qquad \\qed{}%\n$$%\n\\end{enumerate}\n\\end{proof}","sha256_utf8":"13017283f3ee0a4b3b7459b520ef457560e6a5262505bae7c464b86d593004cb"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"指向单调性附录证明","source_units":[{"start":57763,"end":57842,"start_line":843,"end_line":845,"raw_tex":"\\begin{proof}%\nSee \\cref{appendix:proof_prop_monotonicity_Tk}.%\n\\qed\\end{proof}","sha256_utf8":"85ccee2a3ffdba2f98ddc33791dc0b517329ac7c98662f1ac90e82cf203db23e"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"计算增量导数","source_units":[{"start":184657,"end":185181,"start_line":2767,"end_line":2773,"raw_tex":"\\begin{proof}[\\textbf{\\cref{prop:monotonicity_Tk}}] \\label{appendix:proof_prop_monotonicity_Tk}\nWe show $\\frac{d\\TigLmu}{d (\\gamma L)} \\geq 0$ for $\\gamma L \\in (1, \\gbari[\\infty])$, which resumes to prove the positivity of\n  \\begin{align*}\n        \\frac{d \\delta_i(\\gamma L, \\kappa)}{d(\\gamma L)}\n            {}{}\\coloneqq{}{}\n      \\frac{-\\kappa \\left( 1 - 2i \\frac{2-\\kappa \\gamma L}{1-\\kappa \\gamma L} \\right) }{(1-\\kappa \\gamma L)^{2i}} + \\frac{ 1 + 2i \\frac{2-\\gamma L}{\\gamma L-1}}{(1-\\gamma L)^{2i}}.\n  \\end{align*}\n","sha256_utf8":"cd840d6fb94ceac51a991c9a93b43334b7149bcac724ad4dc58f1be95baf8b79"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"非负曲率比导数界","source_units":[{"start":185181,"end":185796,"start_line":2774,"end_line":2782,"raw_tex":"  \\begin{enumerate}[align=right,itemindent=1em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]\n      \\item \\textbf{Case $\\kappa \\geq 0$:} we lower bound $\\frac{d \\delta_i(\\gamma L, \\kappa)}{d(\\gamma L)}$ by neglecting the denominators:\n      \\begin{align*}\n          \\delta_i(\\gamma L, \\kappa) \n            {}\\geq{} \n          {-}\\kappa \\Big( 1 - 2i \\frac{2-\\kappa \\gamma L}{1-\\kappa \\gamma L} \\Big) + 1 + 2i \\frac{2-\\gamma L}{\\gamma L-1}\n            {}={}\n          (1-\\kappa) + 2i \\Big[ \\frac{\\kappa (2-\\kappa \\gamma L)}{1-\\kappa \\gamma L}  + \\frac{2-\\gamma L}{\\gamma L-1} \\Big] {}\\geq{} 0.\n      \\end{align*}%\n","sha256_utf8":"85d22fa1454f2015e6429f290038b36e5ec63cbdf65f263c59143351156001a7"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"负曲率比归约","source_units":[{"start":185796,"end":186356,"start_line":2783,"end_line":2793,"raw_tex":"        %\n      \\item \\textbf{Case $\\kappa < 0$:} it is sufficient to show \n    \\begin{align*}\n        \\frac{1+2i \\frac{2-\\gamma L}{\\gamma L -1}}{(1-\\gamma L)^{2i}}\n            {}\\geq{} 1 {}\\geq{}\n        -\\kappa \\frac{-1+2i (1 + \\frac{1}{1-\\kappa \\gamma L})}{(1-\\kappa \\gamma L)^{2i}}.\n    \\end{align*}\n      The \\lhs holds because $\\gamma L \\in (1,2)$ and $(1-\\gamma L)^{2i}<1$. The \\rhs is equivalent to:%\n      \\begin{align*}\n          (1-\\kappa \\gamma L)^{2i} {}\\geq{} -\\kappa \\Big[-1+2i \\Big(1 + \\frac{1}{1-\\kappa \\gamma L}\\Big)\\Big].\n      \\end{align*}\n","sha256_utf8":"a1b13e7e5e74959218e27ec8940a65789351af656b875890dc3b0ee3f0ba5234"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"应用二阶展开下界","source_units":[{"start":186356,"end":186861,"start_line":2794,"end_line":2803,"raw_tex":"Using $\\gamma L \\in (1,2)$ and the second order Taylor approximation for $(1-\\kappa)^{2i}$, we get the bounds:%\n      \\begin{align*}\n          (1-\\kappa \\gamma L)^{2i} {}\\geq{}& (1-\\kappa)^{2i} \n             {}\\geq{} \n         1 + 2i (-\\kappa) + i(2i-1) \\kappa^2 \\\\\n         %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n          -\\kappa \\Big[-1+2i \\Big(1 + \\frac{1}{1-\\kappa \\gamma L}\\Big)\\Big]\n            {}\\leq{}&\n          -\\kappa \\Big[-1+2i \\left(1 + \\frac{1}{1-\\kappa}\\right)\\Big].\n      \\end{align*}\n","sha256_utf8":"cae16693b2bc90d289ca950194c737fc197a53713d1efd66fe218891c74f108d"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"平方和验证非负性","source_units":[{"start":186861,"end":187264,"start_line":2804,"end_line":2814,"raw_tex":"Therefore, it is sufficient to show:%\n$$\n    1 + 2i (-\\kappa) + i(2i-1) \\kappa^2 {}\\geq{} \n    -\\kappa \\Big[-1+2i \\Big(1 + \\frac{1}{1-\\kappa}\\Big)\\Big],%\n$$%\nwhich equivalently rewrites as the following inequality valid for all $i \\geq 1$:\n$$\n    \\kappa^2 i(i-1) + \\Big( \\kappa i + \\frac{1}{1-\\kappa} \\Big)^2 + \\frac{(1-\\kappa)^3-1}{(1-\\kappa)^2} \\geq 0. {}\\qquad \\qed{}%\n$$%\n\\end{enumerate}\n\\end{proof}","sha256_utf8":"e94151a19569260b4189aee5128c1f621ad9fa471ed54ae422d25c1769b99c00"}]},{"id":"S033","semantic_type":"lemma","kind":"lemma","title":"增量最终持续为负","source_units":[{"start":187266,"end":187529,"start_line":2816,"end_line":2818,"raw_tex":"\\begin{lemma} \\label{lemma:unique_delta_k_neg_thr}\nLet $\\kappa \\in (-\\infty, 1)$ and $\\gamma L \\in (1,\\gbari[\\infty])$. Then there exists an index $k$ such that $\\delta_k(\\gamma L, \\kappa) < 0$ and $\\delta_j(\\gamma L, \\kappa) < 0$ for all $j > k$.    \n\\end{lemma}","sha256_utf8":"b6c6c135e33ebaf929b1cc6297e05348c755f79a6db1b4980834bfbaf962d4de"}]},{"id":"S034","semantic_type":"proof","kind":"proof","title":"负增量持续性证明","source_units":[{"start":187530,"end":188981,"start_line":2819,"end_line":2838,"raw_tex":"\\begin{proof}\n    The sequence $\\delta_i$ is a difference of two monotone sequences: $\\frac{2-\\kappa \\gamma L}{(1-\\kappa \\gamma L)^{2i}}$ and $\\frac{2-\\kappa \\gamma L}{(1-\\kappa \\gamma L)^{2i}}$, respectively. The former is decreasing for $\\kappa < 0$ (weakly convex functions) and increasing for $\\kappa \\geq 0$ (convex functions), whereas the later is always decreasing. In particular, $\\delta_0(\\gamma L, \\kappa) > 0$. \n    Because $(1-\\gamma \\mu)^2 {}\\leq{} (1-\\gamma L)^2$, to the limit it holds $\\lim_{k \\rightarrow \\infty} \\delta_k(\\gamma L, \\kappa) = -\\infty$ and there exists some index $k$ with $\\delta_k(\\gamma L, \\kappa) < 0$. We prove by contradiction that $\\delta_j(\\gamma L, \\kappa) < 0$ for all $j > k$; assume there exists some index $l > k$ such that $\\delta_l(\\gamma L, \\kappa) \\geq 0$. The following inequalities hold:%\n\\begin{align*}\n    \\delta_{k} < 0 \\,:\\,& \n        \\frac{2-\\gamma \\mu}{(1-\\gamma \\mu)^{2k}} \n            {}<{} \n        \\frac{2-\\gamma L}{(1-\\gamma L)^{2k}} \\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\delta_{l} > 0 \\,:\\,& \n        \\frac{2-\\gamma \\mu}{(1-\\gamma \\mu)^{2l}} \n            {}>{} \n        \\frac{2-\\gamma L}{(1-\\gamma L)^{2l}}.\n\\end{align*}\nBy multiplying the inequalities and performing the simplifications we get:%\n\\begin{align*}\n    (1-\\gamma \\mu)^{2(k-l)} > (1-\\gamma L)^{2(k-l)},\n\\end{align*}\nwhich implies, due to $(1-\\gamma \\mu)^2 {}>{} (1-\\gamma L)^2$, that $k > l$, contradiction! \\qed\n\\end{proof}","sha256_utf8":"ca6353a12ce44a740a47f4471f8195bc4bdf8cdb6075df7ce25881390f857ddc"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"利用极限找到负增量","source_units":[{"start":187530,"end":188370,"start_line":2819,"end_line":2821,"raw_tex":"\\begin{proof}\n    The sequence $\\delta_i$ is a difference of two monotone sequences: $\\frac{2-\\kappa \\gamma L}{(1-\\kappa \\gamma L)^{2i}}$ and $\\frac{2-\\kappa \\gamma L}{(1-\\kappa \\gamma L)^{2i}}$, respectively. The former is decreasing for $\\kappa < 0$ (weakly convex functions) and increasing for $\\kappa \\geq 0$ (convex functions), whereas the later is always decreasing. In particular, $\\delta_0(\\gamma L, \\kappa) > 0$. \n    Because $(1-\\gamma \\mu)^2 {}\\leq{} (1-\\gamma L)^2$, to the limit it holds $\\lim_{k \\rightarrow \\infty} \\delta_k(\\gamma L, \\kappa) = -\\infty$ and there exists some index $k$ with $\\delta_k(\\gamma L, \\kappa) < 0$. We prove by contradiction that $\\delta_j(\\gamma L, \\kappa) < 0$ for all $j > k$; assume there exists some index $l > k$ such that $\\delta_l(\\gamma L, \\kappa) \\geq 0$. The following inequalities hold:%\n","sha256_utf8":"228119a01614d859da95fe7f91286ef1c8f74817c6bfde4cac4f0890cd581e4a"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"反证后续增量不能转正","source_units":[{"start":188370,"end":188981,"start_line":2822,"end_line":2838,"raw_tex":"\\begin{align*}\n    \\delta_{k} < 0 \\,:\\,& \n        \\frac{2-\\gamma \\mu}{(1-\\gamma \\mu)^{2k}} \n            {}<{} \n        \\frac{2-\\gamma L}{(1-\\gamma L)^{2k}} \\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\delta_{l} > 0 \\,:\\,& \n        \\frac{2-\\gamma \\mu}{(1-\\gamma \\mu)^{2l}} \n            {}>{} \n        \\frac{2-\\gamma L}{(1-\\gamma L)^{2l}}.\n\\end{align*}\nBy multiplying the inequalities and performing the simplifications we get:%\n\\begin{align*}\n    (1-\\gamma \\mu)^{2(k-l)} > (1-\\gamma L)^{2(k-l)},\n\\end{align*}\nwhich implies, due to $(1-\\gamma \\mu)^2 {}>{} (1-\\gamma L)^2$, that $k > l$, contradiction! \\qed\n\\end{proof}","sha256_utf8":"05f415b44722b784c2b4b34c8fc0707f594e238f54896ed65762e616630eff7b"}]},{"id":"S037","semantic_type":"proposition","kind":"proposition","title":"阈值序列及符号性质","source_units":[{"start":58797,"end":59415,"start_line":858,"end_line":867,"raw_tex":"\\begin{proposition}\\label{prop:properties_step_size_thr}\n    The stepsize thresholds $\\gbari$ satisfy the following properties:%\n    \\begin{enumerate}[align=right,itemindent=1em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]\n        \\item $\\gbari[k] < \\gbari[k+1]$, for all integers $k \\geq 0$;\n        \\item $\\lim_{k\\rightarrow \\infty} \\gbari[k] = \\gbari[\\infty]$;\n        \\item \\shownewtextfinal{For any $\\gamma L \\in \\big[\\gbari[k],\\gbari[k+1]\\big)$: \n        $\\TigLmu[i][\\gamma L] \\geq 0$ $\\forall i \\leq k$, \n        and $\\TigLmu[i][\\gamma L] < 0$ $\\forall i \\geq k+1$.}\n     \\end{enumerate}%\n\\end{proposition}","sha256_utf8":"f4eca7a82cd391432dc312ea15071dcfeb152c10d96696eea4f1dffd167d9d15"}]},{"id":"S038","semantic_type":"proof","kind":"proof","title":"阈值序列性质证明","source_units":[{"start":59417,"end":59500,"start_line":868,"end_line":870,"raw_tex":"\\begin{proof}%\nSee \\cref{appendix:proof_properties_step_size_thr}.%\n\\qed\\end{proof}","sha256_utf8":"3d4fdf0a2a39b158e647f8aba6d4f2c5b89e57c1f44edfe7f9ab30f08ff20e12"},{"start":188985,"end":192418,"start_line":2840,"end_line":2893,"raw_tex":"\\begin{proof}[\\textbf{\\cref{prop:properties_step_size_thr}}] \\label{appendix:proof_properties_step_size_thr} \n\\begin{enumerate}[align=right,itemindent=1em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]\n\\item Case $k=0$ holds by definition since $\\gbari[0]=1$. For any $k \\geq 1$, by using \\cref{lemma:unique_delta_k_neg_thr}, condition $\\TigLmu[k][\\gbari[k][]] = 0$ implies $\\delta_k(\\gamma L, \\kappa) \\leq 0$ and hence $\\delta_{k+1}(\\gamma L, \\kappa) < 0$. Furthermore, $\\TigLmu[k+1][\\gbari[k][]] {} \\leq {} \\TigLmu[k][\\gbari[k][]] = 0$. Then it holds:\n\\begin{align*}\n    \\TigLmu[k+1][\\gbari[k][]] {} \\leq {} \n    \\TigLmu[k][\\gbari[k][]] = 0 \n    {}={} \\TigLmu[k+1][\\gbari[k+1][]].\n\\end{align*}\nThe monotonicity of $\\TigLmu$ in stepsize $\\gamma L$ (see \\cref{prop:monotonicity_Tk}) implies $\\gbari[k] < \\gbari[k+1]$.\n\n\\item Because stepsize thresholds sequence is monotone, it has a limit.\n\\begin{enumerate}[align=right,itemindent=1.5em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]\n    %%%%%%%%%%%%%%%%%%%%%%%%%% 2. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\item \n    \\textbf{Case $\\kappa > 0$.} By definition, $\\gbari[k] < \\frac{2}{1+\\kappa} < 2$ for all integers $k\\geq 1$ and condition $\\TigLmu[k][\\gbari[k]]=0$ implies:%\n    \\begin{align*}\n        \\left(\\frac{1-\\gbari}{1-\\kappa \\gbari}\\right)^{2k}\n            {}={}& \n        \\kappa + (1-\\kappa) (1-\\gbari)^{2k}\n            {}\\Leftrightarrow{} \\\\\n            %%%%%%%%%%%%%%%%%%%%%%%%%%%%\n        \\left|\\frac{1-\\gbari}{1-\\kappa \\gbari}\\right|\n            {}={}&\n        \\exp \\left[ \\frac{\\ln \\left(\\kappa + (1-\\kappa) (1-\\gbari)^{2k}\\right)}{2k} \\right].\n    \\end{align*}%\n    To the limit $k \\rightarrow \\infty$, the \\rhs equals $1$ because $(1-\\gbari)^{2k}\\rightarrow 0$, hence $|1-\\gbari|=|1-\\kappa \\gbari|$, which implies the solution $\\gbari[\\infty]=\\frac{2}{1+\\kappa}$. \\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\item \\textbf{Case $\\kappa < 0$.} By definition, $\\gbari < 2$ for all integers $k\\geq 1$ and condition $\\TigLmu[k][\\gbari[k]]=0$ implies:\n    \\begin{align*}\n    (1-\\gbari)^{-2k} \n        {}={}&\n    1 + \\frac{1 - (1-\\kappa \\gbari)^{-2k}}{-\\kappa}\n        {}\\Leftrightarrow{} \\\\\n    % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\left|1-\\gbari\\right| \n        {}={}&\n    \\exp \\Bigg[ \\frac{ -\\ln \\Big(1 + \\frac{1 - (1-\\kappa \\gbari)^{-2k}}{-\\kappa}\\Big)}{2k} \\Bigg].\n\\end{align*}\n To the limit $k \\rightarrow \\infty$, the \\rhs equals $1$ because $(1-\\kappa \\gbari)^{-2k} \\rightarrow 0$ due to $\\kappa < 0$. Hence, $\\left|1-\\gbari\\right|=1$, with the solution $\\gbari[\\infty]=2$.%\n %\n\\item \\textbf{Case $\\kappa = 0$.} By definition, $\\gbari[k][(0)] < 2$ and, similarly to the case $\\kappa < 0$:%\n \\begin{align*}\n     \\left|1-\\gbari\\right|\n        {}={}&\n     \\left[ 1 + 2 k \\gbari \\right]^{\\frac{-1}{2k}}\n        {}={}\n    \\exp \\Big[ \\frac{-\\ln \\big( 1 + 2 k \\gbari \\big) }{2k} \\Big].\n \\end{align*}\n  To the limit $k \\rightarrow \\infty$, the \\rhs equals $1$, hence $\\left|1-\\gbari\\right|=1$, with the solution $\\gbari[\\infty]=2$.\n\\end{enumerate}%\n%\n\\item Condition $\\gamma L \\in [\\gbari[k], \\gbari[k+1])$ implies $\\TigLmu[k][\\gbari[k]] \\geq 0$ and $\\TigLmu[k+1][\\gbari[k]] < 0$. Because $\\delta_i(\\gamma L, \\kappa) < 0$ for all $i \\geq k$, we have (i) $\\TigLmu[i][\\gbari[k][]] {}>{} 0$ for $i=1,\\dots,k-1$ (since $k$ is uniquely defined) and (ii) $\\TigLmu[i][\\gbari[k][]] {}<{} 0$ for $i\\geq k+1$. {}\\qed{}%\n\\end{enumerate}\n\\end{proof}","sha256_utf8":"d7f0e8a15454788ddbc603c8fcf1c88b1d39b6617f5a1035bcff06c7b9a446f7"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"指向阈值性质附录证明","source_units":[{"start":59417,"end":59500,"start_line":868,"end_line":870,"raw_tex":"\\begin{proof}%\nSee \\cref{appendix:proof_properties_step_size_thr}.%\n\\qed\\end{proof}","sha256_utf8":"3d4fdf0a2a39b158e647f8aba6d4f2c5b89e57c1f44edfe7f9ab30f08ff20e12"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"证明阈值严格递增","source_units":[{"start":188985,"end":189806,"start_line":2840,"end_line":2848,"raw_tex":"\\begin{proof}[\\textbf{\\cref{prop:properties_step_size_thr}}] \\label{appendix:proof_properties_step_size_thr} \n\\begin{enumerate}[align=right,itemindent=1em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]\n\\item Case $k=0$ holds by definition since $\\gbari[0]=1$. For any $k \\geq 1$, by using \\cref{lemma:unique_delta_k_neg_thr}, condition $\\TigLmu[k][\\gbari[k][]] = 0$ implies $\\delta_k(\\gamma L, \\kappa) \\leq 0$ and hence $\\delta_{k+1}(\\gamma L, \\kappa) < 0$. Furthermore, $\\TigLmu[k+1][\\gbari[k][]] {} \\leq {} \\TigLmu[k][\\gbari[k][]] = 0$. Then it holds:\n\\begin{align*}\n    \\TigLmu[k+1][\\gbari[k][]] {} \\leq {} \n    \\TigLmu[k][\\gbari[k][]] = 0 \n    {}={} \\TigLmu[k+1][\\gbari[k+1][]].\n\\end{align*}\nThe monotonicity of $\\TigLmu$ in stepsize $\\gamma L$ (see \\cref{prop:monotonicity_Tk}) implies $\\gbari[k] < \\gbari[k+1]$.\n","sha256_utf8":"bb9993a6d0a761bf1df98acd3004e6c10022dcd55ef7ab80e86babf32462f2fa"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"正曲率比阈值极限","source_units":[{"start":189806,"end":190834,"start_line":2849,"end_line":2865,"raw_tex":"\n\\item Because stepsize thresholds sequence is monotone, it has a limit.\n\\begin{enumerate}[align=right,itemindent=1.5em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]\n    %%%%%%%%%%%%%%%%%%%%%%%%%% 2. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\item \n    \\textbf{Case $\\kappa > 0$.} By definition, $\\gbari[k] < \\frac{2}{1+\\kappa} < 2$ for all integers $k\\geq 1$ and condition $\\TigLmu[k][\\gbari[k]]=0$ implies:%\n    \\begin{align*}\n        \\left(\\frac{1-\\gbari}{1-\\kappa \\gbari}\\right)^{2k}\n            {}={}& \n        \\kappa + (1-\\kappa) (1-\\gbari)^{2k}\n            {}\\Leftrightarrow{} \\\\\n            %%%%%%%%%%%%%%%%%%%%%%%%%%%%\n        \\left|\\frac{1-\\gbari}{1-\\kappa \\gbari}\\right|\n            {}={}&\n        \\exp \\left[ \\frac{\\ln \\left(\\kappa + (1-\\kappa) (1-\\gbari)^{2k}\\right)}{2k} \\right].\n    \\end{align*}%\n    To the limit $k \\rightarrow \\infty$, the \\rhs equals $1$ because $(1-\\gbari)^{2k}\\rightarrow 0$, hence $|1-\\gbari|=|1-\\kappa \\gbari|$, which implies the solution $\\gbari[\\infty]=\\frac{2}{1+\\kappa}$. \\\\\n","sha256_utf8":"4e075c2d1e12b2505a06f5de6b635b4a1d7efb53c82905295ee8f45c48f8294d"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"负曲率比阈值极限","source_units":[{"start":190834,"end":191564,"start_line":2866,"end_line":2878,"raw_tex":"    %%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\item \\textbf{Case $\\kappa < 0$.} By definition, $\\gbari < 2$ for all integers $k\\geq 1$ and condition $\\TigLmu[k][\\gbari[k]]=0$ implies:\n    \\begin{align*}\n    (1-\\gbari)^{-2k} \n        {}={}&\n    1 + \\frac{1 - (1-\\kappa \\gbari)^{-2k}}{-\\kappa}\n        {}\\Leftrightarrow{} \\\\\n    % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\left|1-\\gbari\\right| \n        {}={}&\n    \\exp \\Bigg[ \\frac{ -\\ln \\Big(1 + \\frac{1 - (1-\\kappa \\gbari)^{-2k}}{-\\kappa}\\Big)}{2k} \\Bigg].\n\\end{align*}\n To the limit $k \\rightarrow \\infty$, the \\rhs equals $1$ because $(1-\\kappa \\gbari)^{-2k} \\rightarrow 0$ due to $\\kappa < 0$. Hence, $\\left|1-\\gbari\\right|=1$, with the solution $\\gbari[\\infty]=2$.%\n","sha256_utf8":"6756b872753aa8a282e80ff412e938a6ebeb4cc97ca8207698e0955f4bff155e"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"零曲率比阈值极限","source_units":[{"start":191564,"end":192030,"start_line":2879,"end_line":2889,"raw_tex":" %\n\\item \\textbf{Case $\\kappa = 0$.} By definition, $\\gbari[k][(0)] < 2$ and, similarly to the case $\\kappa < 0$:%\n \\begin{align*}\n     \\left|1-\\gbari\\right|\n        {}={}&\n     \\left[ 1 + 2 k \\gbari \\right]^{\\frac{-1}{2k}}\n        {}={}\n    \\exp \\Big[ \\frac{-\\ln \\big( 1 + 2 k \\gbari \\big) }{2k} \\Big].\n \\end{align*}\n  To the limit $k \\rightarrow \\infty$, the \\rhs equals $1$, hence $\\left|1-\\gbari\\right|=1$, with the solution $\\gbari[\\infty]=2$.\n\\end{enumerate}%\n","sha256_utf8":"a7b239dedd6a438eb4dc8989607f25a5146faae3922980c13c1ce4a60ce65d00"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"确定各指标的符号","source_units":[{"start":192030,"end":192418,"start_line":2890,"end_line":2893,"raw_tex":"%\n\\item Condition $\\gamma L \\in [\\gbari[k], \\gbari[k+1])$ implies $\\TigLmu[k][\\gbari[k]] \\geq 0$ and $\\TigLmu[k+1][\\gbari[k]] < 0$. Because $\\delta_i(\\gamma L, \\kappa) < 0$ for all $i \\geq k$, we have (i) $\\TigLmu[i][\\gbari[k][]] {}>{} 0$ for $i=1,\\dots,k-1$ (since $k$ is uniquely defined) and (ii) $\\TigLmu[i][\\gbari[k][]] {}<{} 0$ for $i\\geq k+1$. {}\\qed{}%\n\\end{enumerate}\n\\end{proof}","sha256_utf8":"48eb3340d2715136d5daf81b2b4f5b5b941e5bd7dd9d458c113868d8d5736139"}]},{"id":"S045","semantic_type":"theorem","kind":"theorem","title":"任意下曲率插值条件","source_units":[{"start":90358,"end":90914,"start_line":1211,"end_line":1218,"raw_tex":"\\begin{theorem}[Interpolation conditions for smooth functions]\\label{thm:interp_hypo}\nSet $\\mathcal{T}=\\big\\{(x_i, g_i, f_i)\\big\\}_{i \\in \\mathcal{I}}$ is $\\mathcal{F}_{\\mu,L}$-interpolable, where $\\mu \\in (-\\infty, L)$, if and only if for every pair of indices $(i,j)$, with $i,j \\in \\mathcal{I}$, it holds:%\n\\begin{align}\\label{eq:Interp_hypoconvex}\nf_{i}{-}f_{j}{-}\\langle g_{j}, x_{i}-x_{j} \\rangle \n        {}\\geq{}\n    \\frac{1}{2L} \\|g_{i}{-}g_{j}\\|^{2} {}+{} \\frac{\\mu}{2L(L{-}\\mu)} \\|g_{i}{-}g_{j} {-} L (x_i {-}x_j)\\|^2.\n\\end{align}%\n\\end{theorem}","sha256_utf8":"7a6d21c95dc288917fe18f813a14253d201707309610c3dbbdbdbca61b29ae05"}]},{"id":"S046","semantic_type":"proof","kind":"proof","title":"曲率平移插值证明","source_units":[{"start":211060,"end":213294,"start_line":3231,"end_line":3249,"raw_tex":"\\begin{proof}[\\textbf{\\cref{thm:interp_hypo}}]%\n    The proof follows the same steps outlined in \\cite[Theorem 4]{taylor_smooth_2017}, with a minimal extension to accommodate $\\mu < 0$. This extension impacts only the first step of the demonstration, known as minimal curvature subtraction, which involves shifting the lower curvature of the function to $0$, effectively converting it into a smooth convex function. This adjustment remains valid for $\\mu < 0$, as showed further on. By applying the \\textit{gradient inequality} for smooth convex functions: $h(x) {} \\geq {} h(y) + \\left \\langle \\nabla h(y), x-y \\right \\rangle, \\, \\forall x,y \\in \\mathbb{R}^d$ for $h = \\frac{L}{2}\\|\\cdot\\|^2-f$ and $h = f - \\frac{\\mu}{2}\\|\\cdot\\|^2$, we obtain:%\n    \\begin{align}\\label{eq:def_hypoconvex_smooth}\n    \\tfrac{\\mu}{2} \\|x-y\\|^2 \n        {} \\leq {} \n    f(x) - f(y) - \\langle \\nabla f(y), x-y \\rangle \n        {} \\leq {} \n    \\tfrac{L}{2} \\|x-y\\|^2 \\quad \\forall x, y \\in \\mathbb{R}^d.\n\\end{align}%        \n    Let $\\bar{f} \\in \\mathcal{F}_{0,L-\\mu}$, $\\bar{f}(x){}\\coloneqq{}f(x)-\\tfrac{\\mu}{2}\\|x\\|^2$, with $\\nabla \\bar{f}(x)=\\nabla f(x)-\\mu x$; by replacing $f$ with $\\bar{f}$ in the quadratic bounds inequality \\eqref{eq:def_hypoconvex_smooth} it holds:%\n$$\n    % \\frac{\\mu}{2} \\|x-y\\|^2 {}\\leq{}& f(x) - f(y) - \\langle \\nabla f(y), x-y \\rangle {}\\leq{} \\frac{L}{2} \\|x-y\\|^2 \\nonumber \\\\\n    % & \\frac{\\mu}{2} \\|x-y\\|^2 {}\\leq{} \\big(f(x)-\\frac{\\mu}{2} \\|x\\|^2\\big) - \\big(f(y)-\\frac{\\mu}{2} \\|y\\|^2\\big) - \\langle \\nabla \\big(f(y)-\\mu y\\big), x-y \\rangle {}\\leq{} \\frac{L}{2} \\|x-y\\|^2 \\nonumber \\\\\n    % \\frac{\\mu}{2} \\|x-y\\|^2 {}\\leq{}& \\bar{f}(x) - \\bar{f}(y)- \\langle \\nabla \\bar{f}(y), x-y \\rangle + \\frac{\\mu}{2} \\|x-y\\|^2 {}\\leq{} \\frac{L}{2} \\|x-y\\|^2 \\nonumber \\\\\n    0 {}\\leq{} \\bar{f}(x) - \\bar{f}(y)- \\langle \\nabla \\bar{f}(y), x-y \\rangle {}\\leq{} \\tfrac{L-\\mu}{2} \\|x-y\\|^2.\n$$\nHence, the set $\\{(x_i, g_i, f_i)\\}_{i \\in \\mathcal{I}}$ is $\\mathcal{F}_{\\mu,L}$-interpolable if and only if the set $\\{(x_i, g_i-\\mu x_i, f_i-\\tfrac{\\mu}{2}\\|x_i\\|^2)\\}_{i \\in \\mathcal{I}}$ is $\\mathcal{F}_{0, L-\\mu}$-interpolable. \nSubsequently, the proof continues identically to \\cite[Theorem 4]{taylor_smooth_2017}.\n\\qed\\end{proof}","sha256_utf8":"a78d83ee393d5ab74c6f31878764fcf44b1391f13bd40f82bd5aa88ac65ea2c0"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"由凸性取得双侧二次界","source_units":[{"start":211060,"end":212065,"start_line":3231,"end_line":3239,"raw_tex":"\\begin{proof}[\\textbf{\\cref{thm:interp_hypo}}]%\n    The proof follows the same steps outlined in \\cite[Theorem 4]{taylor_smooth_2017}, with a minimal extension to accommodate $\\mu < 0$. This extension impacts only the first step of the demonstration, known as minimal curvature subtraction, which involves shifting the lower curvature of the function to $0$, effectively converting it into a smooth convex function. This adjustment remains valid for $\\mu < 0$, as showed further on. By applying the \\textit{gradient inequality} for smooth convex functions: $h(x) {} \\geq {} h(y) + \\left \\langle \\nabla h(y), x-y \\right \\rangle, \\, \\forall x,y \\in \\mathbb{R}^d$ for $h = \\frac{L}{2}\\|\\cdot\\|^2-f$ and $h = f - \\frac{\\mu}{2}\\|\\cdot\\|^2$, we obtain:%\n    \\begin{align}\\label{eq:def_hypoconvex_smooth}\n    \\tfrac{\\mu}{2} \\|x-y\\|^2 \n        {} \\leq {} \n    f(x) - f(y) - \\langle \\nabla f(y), x-y \\rangle \n        {} \\leq {} \n    \\tfrac{L}{2} \\|x-y\\|^2 \\quad \\forall x, y \\in \\mathbb{R}^d.\n\\end{align}%        \n","sha256_utf8":"6c1354c2925cbadaa28126ad6636104afa4daaa4c9e8468c6471443e2c10776a"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"减去下曲率二次项","source_units":[{"start":212065,"end":212957,"start_line":3240,"end_line":3246,"raw_tex":"    Let $\\bar{f} \\in \\mathcal{F}_{0,L-\\mu}$, $\\bar{f}(x){}\\coloneqq{}f(x)-\\tfrac{\\mu}{2}\\|x\\|^2$, with $\\nabla \\bar{f}(x)=\\nabla f(x)-\\mu x$; by replacing $f$ with $\\bar{f}$ in the quadratic bounds inequality \\eqref{eq:def_hypoconvex_smooth} it holds:%\n$$\n    % \\frac{\\mu}{2} \\|x-y\\|^2 {}\\leq{}& f(x) - f(y) - \\langle \\nabla f(y), x-y \\rangle {}\\leq{} \\frac{L}{2} \\|x-y\\|^2 \\nonumber \\\\\n    % & \\frac{\\mu}{2} \\|x-y\\|^2 {}\\leq{} \\big(f(x)-\\frac{\\mu}{2} \\|x\\|^2\\big) - \\big(f(y)-\\frac{\\mu}{2} \\|y\\|^2\\big) - \\langle \\nabla \\big(f(y)-\\mu y\\big), x-y \\rangle {}\\leq{} \\frac{L}{2} \\|x-y\\|^2 \\nonumber \\\\\n    % \\frac{\\mu}{2} \\|x-y\\|^2 {}\\leq{}& \\bar{f}(x) - \\bar{f}(y)- \\langle \\nabla \\bar{f}(y), x-y \\rangle + \\frac{\\mu}{2} \\|x-y\\|^2 {}\\leq{} \\frac{L}{2} \\|x-y\\|^2 \\nonumber \\\\\n    0 {}\\leq{} \\bar{f}(x) - \\bar{f}(y)- \\langle \\nabla \\bar{f}(y), x-y \\rangle {}\\leq{} \\tfrac{L-\\mu}{2} \\|x-y\\|^2.\n$$\n","sha256_utf8":"da526b26b6d5cae04cbe6210a33efcfca3962d0c45f0ae1ec900079a6abad622"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"归约到凸函数插值定理","source_units":[{"start":212957,"end":213294,"start_line":3247,"end_line":3249,"raw_tex":"Hence, the set $\\{(x_i, g_i, f_i)\\}_{i \\in \\mathcal{I}}$ is $\\mathcal{F}_{\\mu,L}$-interpolable if and only if the set $\\{(x_i, g_i-\\mu x_i, f_i-\\tfrac{\\mu}{2}\\|x_i\\|^2)\\}_{i \\in \\mathcal{I}}$ is $\\mathcal{F}_{0, L-\\mu}$-interpolable. \nSubsequently, the proof continues identically to \\cite[Theorem 4]{taylor_smooth_2017}.\n\\qed\\end{proof}","sha256_utf8":"9a118f22b5ed3d4aafa8bc0a0d7e55491e05ea698c0a77347b60307a8f0cc44a"}]},{"id":"S050","semantic_type":"corollary","kind":"corollary","title":"双向插值求和条件","source_units":[{"start":91275,"end":91679,"start_line":1222,"end_line":1228,"raw_tex":"\\begin{corollary} \\label{corr:direct_sum_interp_cond} \nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty, L)$. For all $x_i, x_j \\in \\mathbb{R}^d$ it holds:%\n\\begin{align}\\label{eq:direct_sum_interp_cond}\n    0 {}\\geq{} \\left\\langle \\nabla f(x_i) - \\nabla f(x_j) - L(x_i-x_j) \\,,\\, \n                           \\nabla f(x_i) - \\nabla f(x_j) - \\mu(x_i-x_j) \\right\\rangle.\n\\end{align}%\n\\end{corollary}","sha256_utf8":"5c9cc15f9cc2a19b32c34e800e1129cf7b1cdbede483389c2bf36f546d1b73ed"}]},{"id":"S051","semantic_type":"proof","kind":"proof","title":"双向插值求和证明","source_units":[{"start":91681,"end":91905,"start_line":1229,"end_line":1231,"raw_tex":"\\begin{proof}\n    Inequality \\eqref{eq:direct_sum_interp_cond} results by summing up inequalities \\eqref{eq:Interp_hypoconvex} written for the pairs $(x_i,x_j)$ and $(x_j,x_i)$ and performing simplifications.\n\\qed\\end{proof}","sha256_utf8":"f97e843159ade9a6fdaab81167957d21e314df9aaf8f7a834b6cd90cb98415ff"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"交换指标后相加化简","source_units":[{"start":91681,"end":91905,"start_line":1229,"end_line":1231,"raw_tex":"\\begin{proof}\n    Inequality \\eqref{eq:direct_sum_interp_cond} results by summing up inequalities \\eqref{eq:Interp_hypoconvex} written for the pairs $(x_i,x_j)$ and $(x_j,x_i)$ and performing simplifications.\n\\qed\\end{proof}","sha256_utf8":"f97e843159ade9a6fdaab81167957d21e314df9aaf8f7a834b6cd90cb98415ff"}]},{"id":"S053","semantic_type":"lemma","kind":"lemma","title":"下降与最优值差下界","source_units":[{"start":94817,"end":95207,"start_line":1266,"end_line":1271,"raw_tex":"\\begin{lemma}\\label{lemma:sufficient_decrease_N_*}\nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty, L)$. One iteration of \\eqref{eq:GM_it_fixed_steps} with stepsize $\\gamma L \\in (0,2)$ decreases the function values and at each point $x_i$ it holds:\n\\begin{align}\\label{eq:sufficient_decrease_N_*}\n    f(x_i) - f_* {}\\geq{} \\frac{1}{2L} \\|\\nabla f(x_i)\\|^2.\n\\end{align}%\n\\end{lemma}","sha256_utf8":"ce99cca578bea787ff99b477a2eefcb9e21c43a1d6c80d9db60a75b72531dbca"}]},{"id":"S054","semantic_type":"proposition","kind":"proposition","title":"加权下降转收敛界","source_units":[{"start":99752,"end":100313,"start_line":1304,"end_line":1319,"raw_tex":"\\begin{proposition}[General procedure]\\label{prop:general_procedure_min_grad_nrm}\n    Inequality \\eqref{eq:I^k_N_general_ineq} implies:\n\\[\n    \\frac{f(x_0) - f(x_N)}{\\sum_{i=0}^N \\sigma_i}\n        {}\\geq{}\n    \\frac{1}{2L} \\,\n    \\min_{0\\leq i \\leq N} \\big\\{\\|\\nabla f(x_i)\\|^2\\big\\},\n\\]\nwith equality only if all gradient norms $\\|\\nabla f(x_i)\\|^2$ with $\\sigma_i>0$ are equal. Moreover, \n\\[\n    \\frac{f(x_0) - f_*}{1 + \\sum_{i=0}^N \\sigma_i}\n        {}\\geq{}\n    \\frac{1}{2L} \\,\n    \\min_{0\\leq i \\leq N} \\big\\{\\|\\nabla f(x_i)\\|^2\\big\\}.\n\\]\n\\end{proposition}","sha256_utf8":"cadefadf2103090fd76d30dd211abe4d5b4a4b03e732cd9c38fd39c76e88d0d7"}]},{"id":"S055","semantic_type":"proof","kind":"proof","title":"加权下降转换证明","source_units":[{"start":100317,"end":100646,"start_line":1321,"end_line":1324,"raw_tex":"\\begin{proof}\n   The first inequality results after taking minimum gradient norm in \\eqref{eq:I^k_N_general_ineq}. %\nThe adjustment in the denominator of the second inequality results from combining \\eqref{eq:sufficient_decrease_N_*} with \\eqref{eq:I^k_N_general_ineq} and taking again the minimum gradient norm. \\qed\n\\end{proof}","sha256_utf8":"71d42d4c41fddc071f89c1bbacf275b21bee0bcd6b74fc183a5b2476e9fd3123"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"取最小梯度范数","source_units":[{"start":100317,"end":100434,"start_line":1321,"end_line":1322,"raw_tex":"\\begin{proof}\n   The first inequality results after taking minimum gradient norm in \\eqref{eq:I^k_N_general_ineq}. %\n","sha256_utf8":"1f5b523987101fb74b9ac0c32ff51c6fe138f160048131690b7f4357977dc876"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"追加末点最优值差","source_units":[{"start":100434,"end":100646,"start_line":1323,"end_line":1324,"raw_tex":"The adjustment in the denominator of the second inequality results from combining \\eqref{eq:sufficient_decrease_N_*} with \\eqref{eq:I^k_N_general_ineq} and taking again the minimum gradient norm. \\qed\n\\end{proof}","sha256_utf8":"1cb5d9a0db704a0462f90fc71ac376c34cd33f633371f716ac4a997ec722dbf3"}]},{"id":"S058","semantic_type":"lemma","kind":"lemma","title":"短步长单步下降界","source_units":[{"start":102606,"end":103384,"start_line":1355,"end_line":1365,"raw_tex":"\\begin{lemma}\\label{lemma:sufficient_decrease_h_leq_1}\n    Let $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty, L)$, and consider one iteration of \\eqref{eq:GM_it_fixed_steps} with stepsize $\\gamma L \\in(0,1]$. Then the following inequality holds, with equality only if $\\nabla f(x_{i}) = \\nabla f(x_{i+1})$ or $\\gamma L = 1$:\n\\begin{align}\\label{eq:one_step_rate_h_leq_1} \\tag{N2SD}\n\\hspace{-.5em}\n    f(x_{i}) {-} f(x_{i+1})\n        {}\\geq {}\n    \\frac{\\gamma L \\, \\gamma \\mu {-} 2\\gamma \\mu {+} \\gamma L}{L{-}\\mu} \\frac{\\|\\nabla f(x_{i})\\|^2}{2} +\n    \\frac{\\gamma L}{L{-}\\mu} \\frac{\\|\\nabla f(x_{i+1})\\|^{2}}{2}.\n\\end{align}%\n    Furthermore, if $\\mu \\leq 0$, then the gradient norms from \\eqref{eq:one_step_rate_h_leq_1} are weighted by positive quantities.\n\\end{lemma}","sha256_utf8":"f94359d6b23318bdea79bbbcdc70a6653adf09bd667171cefd2da0779ee8c77b"}]},{"id":"S059","semantic_type":"proof","kind":"proof","title":"短步长下降界证明","source_units":[{"start":103468,"end":104406,"start_line":1367,"end_line":1377,"raw_tex":"\\begin{proof}\\label{proof:lemma:sufficient_decrease_h_leq_1}\n    Inequality \\eqref{eq:hypo_interp_(0,1)} is reformulated by expressing the inner product in terms of gradient norms, i.e., $2 \\big\\langle g_{i}, g_{i+1}\\big\\rangle = \\|g_{i}\\|^2 + \\|g_{i+1}\\|^2 - \\|g_{i} - g_{i+1}\\|^2$, as:%\n\\begin{align} \\label{eq:tight_short_step_sizes}\n    f_{i}{-}f_{i+1}\n        {}\\geq {}\n    \\frac{\\gamma L \\, \\gamma \\mu {-} 2\\gamma \\mu {+} \\gamma L}{L{-}\\mu} \\frac{\\|g_{i}\\|^2}{2} +\n    \\frac{\\gamma L}{L{-}\\mu} \\frac{\\|g_{i+1}\\|^{2}}{2} +\n    \\frac{1 {-} \\gamma L}{L{-}\\mu} \\frac{\\|g_{i} {-} g_{i+1}\\|^{2}}{2}.\n\\end{align}\nSince $\\gamma L \\in (0,1]$, the mixed term can be neglected, resulting in inequality \\eqref{eq:one_step_rate_h_leq_1}, with equality only if $g_{i} = g_{i+1}$ or $\\gamma L = 1$. The numerator of first term rewrites as $-\\gamma \\mu(1-\\gamma L) + (\\gamma L - \\gamma \\mu)$, hence it is positive when $\\mu \\leq 0$.\n\\qed\\end{proof}","sha256_utf8":"dc726e1b6a8c1ae7c97c03b9da388a8c3264e186019f2c1571348d17ae7b27e7"}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","title":"展开内积保留梯度差项","source_units":[{"start":103468,"end":104080,"start_line":1367,"end_line":1375,"raw_tex":"\\begin{proof}\\label{proof:lemma:sufficient_decrease_h_leq_1}\n    Inequality \\eqref{eq:hypo_interp_(0,1)} is reformulated by expressing the inner product in terms of gradient norms, i.e., $2 \\big\\langle g_{i}, g_{i+1}\\big\\rangle = \\|g_{i}\\|^2 + \\|g_{i+1}\\|^2 - \\|g_{i} - g_{i+1}\\|^2$, as:%\n\\begin{align} \\label{eq:tight_short_step_sizes}\n    f_{i}{-}f_{i+1}\n        {}\\geq {}\n    \\frac{\\gamma L \\, \\gamma \\mu {-} 2\\gamma \\mu {+} \\gamma L}{L{-}\\mu} \\frac{\\|g_{i}\\|^2}{2} +\n    \\frac{\\gamma L}{L{-}\\mu} \\frac{\\|g_{i+1}\\|^{2}}{2} +\n    \\frac{1 {-} \\gamma L}{L{-}\\mu} \\frac{\\|g_{i} {-} g_{i+1}\\|^{2}}{2}.\n\\end{align}\n","sha256_utf8":"0a41c42b1d5b6e437f335d9788520403788a5a2e5904259e06aa1d5281fd3c35"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"舍弃非负项并检查权重","source_units":[{"start":104080,"end":104406,"start_line":1376,"end_line":1377,"raw_tex":"Since $\\gamma L \\in (0,1]$, the mixed term can be neglected, resulting in inequality \\eqref{eq:one_step_rate_h_leq_1}, with equality only if $g_{i} = g_{i+1}$ or $\\gamma L = 1$. The numerator of first term rewrites as $-\\gamma \\mu(1-\\gamma L) + (\\gamma L - \\gamma \\mu)$, hence it is positive when $\\mu \\leq 0$.\n\\qed\\end{proof}","sha256_utf8":"c00b4fd63e18c748462bf9228ff45e4afdcc10369711dee001548cad595ae4fa"}]},{"id":"S062","semantic_type":"lemma","kind":"lemma","title":"长步长单步下降界","source_units":[{"start":104512,"end":105624,"start_line":1380,"end_line":1398,"raw_tex":"\\begin{lemma}\\label{lemma:sufficient_decrease_h_geq_1}\n    Let $f \\in \\mathcal{F}_{\\mu,L}$, with, $\\mu \\in (-\\infty,L)$, and consider one iteration of \\eqref{eq:GM_it_fixed_steps} with stepsize $\\gamma L \\in [1,\\gbari[\\infty])$. Then the following inequality holds:%\n\\begin{align} \\label{eq:one_step_rate_h_geq_1} \\tag{N4SD}\n% \\hspace{-.25em}\n    f(x_{i}){-}f(x_{i+1})\n        {}\\geq {}\n    \\frac{\\gamma [(2{-}\\gamma L)(2{-}\\gamma \\mu){-}1]}{2{-}\\gamma L {- }\\gamma \\mu}\n    \\frac{\\|\\nabla f(x_{i})\\|^2}{2} +\n    \\frac{\\gamma}{2{-}\\gamma L {-} \\gamma \\mu} \\frac{\\|\\nabla f(x_{i+1})\\|^{2}}{2}.\n\\end{align}%\nWhen $\\gamma L > 1$, equality holds only if:%\n\\begin{align}\\label{eq:equality_case_h_geq_1}\n    {\\langle \\nabla f(x_{i})}, {\\nabla f(x_{i+1})\\rangle}\n        {}{=}{} \n    \\frac{(1{-}\\gamma \\mu)(1{-}\\gamma L)}{2{-}\\gamma L{-}\\gamma \\mu} \\|\\nabla f(x_{i})\\|^2 {}{+}{} \n    \\frac{1}{2{-}\\gamma L{-}\\gamma \\mu} \\|\\nabla f(x_{i+1})\\|^2.%\n\\end{align}%\nFurthermore, if $\\gamma L \\in [1,\\gbari[1]]$, then the gradient norms from \\eqref{eq:one_step_rate_h_geq_1} are weighted by nonnegative quantities.%\n\\end{lemma}","sha256_utf8":"6d927dd0dad2f284af3dcc5afd53a13f36eaa900332985bd282ebb7098e93f5b"}]},{"id":"S063","semantic_type":"proof","kind":"proof","title":"长步长下降界证明","source_units":[{"start":105629,"end":107506,"start_line":1400,"end_line":1420,"raw_tex":"\\begin{proof}\\label{proof:lemma:sufficient_decrease_h_geq_1}\nThe mixed term in inequality \\eqref{eq:tight_short_step_sizes} enters with a negative contribution for $\\gamma L > 1$. To counterbalance it, the complementary interpolation inequality \\eqref{eq:hypo_interp_(1,0)} is used, written with the equivalent expression:%\n\\begin{align*}\n   f_{i+1}-f_{i}\n        {}\\geq {}\n    \\frac{\\gamma L \\gamma \\mu {-} 2 \\gamma L {+} \\gamma \\mu}{L{-}\\mu} \\frac{\\|g_{i}\\|^2}{2} +\n    \\frac{\\gamma \\mu}{L{-}\\mu} \\frac{\\|g_{i+1}\\|^{2}}{2} +\n    \\frac{1 {-} \\gamma \\mu}{L{-}\\mu} \\frac{\\|g_{i} {-} g_{i+1}\\|^{2}}{2}.%\n\\end{align*}%\nConsider the sum of distance-1 inequalities $\\bar{I}_{i}^{1} {}\\coloneqq{} \\Iij[i][i+1] + \\Iji[i][i+1]$:%\n\\begin{align}\\label{eq:dist_1_blended_mixed_grads_diff} \\tag{$\\bar{I}_{i}^{1}$}\n\\hspace{-0.5em}\n0\n    {}\\geq{}\n\\frac{2 \\gamma L \\gamma \\mu {-} \\gamma L {-} \\gamma \\mu}{L{-}\\mu} \\frac{\\|g_{i}\\|^2}{2} {}+{}\n\\frac{\\gamma L {+} \\gamma \\mu}{L{-}\\mu} \\frac{\\|g_{i+1}\\|^2}{2} {}+{} \n\\frac{2{-}\\gamma L {-} \\gamma \\mu}{L{-}\\mu} \\frac{\\| g_{i} {-} g_{i+1}\\|^2}{2}.\n\\end{align}%\nBy weighting \\eqref{eq:dist_1_blended_mixed_grads_diff} with $\\beta(\\gamma L,\\gamma \\mu){}\\coloneqq{} \\tfrac{\\gamma L{-}1}{2{-}\\gamma L {-}\\gamma \\mu} \\geq 0$ and summing it with \\eqref{eq:tight_short_step_sizes}, the coefficient of $\\| g_{i} {-} g_{i+1}\\|^2$ gets canceled and we obtain \\eqref{eq:one_step_rate_h_geq_1}.\nThe equality condition \\eqref{eq:equality_case_h_geq_1} follows from the equality case of \\eqref{eq:dist_1_blended_mixed_grads_diff}. Nevertheless, the coefficient of $\\|g_{i}\\|^2$ is nonnegative for $\\gamma L \\in [1,\\gbari[1]]$, where $\\gbari[1]$ is the largest stepsize $\\gamma L \\in [1, 2)$ for which $(2-\\gamma L)(2-\\gamma \\mu) - 1 \\geq {} 0$, implying nonnegativity of the weight of $\\|\\nabla f(x_{i})\\|^2$ in \\eqref{eq:one_step_rate_h_geq_1}.\\qed\n\\end{proof}","sha256_utf8":"6d5429cbc85ca1921a9b82678940d77e46304cfb3fc0fb256daecd3b2aa6205e"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"补入反向插值不等式","source_units":[{"start":105629,"end":106245,"start_line":1400,"end_line":1408,"raw_tex":"\\begin{proof}\\label{proof:lemma:sufficient_decrease_h_geq_1}\nThe mixed term in inequality \\eqref{eq:tight_short_step_sizes} enters with a negative contribution for $\\gamma L > 1$. To counterbalance it, the complementary interpolation inequality \\eqref{eq:hypo_interp_(1,0)} is used, written with the equivalent expression:%\n\\begin{align*}\n   f_{i+1}-f_{i}\n        {}\\geq {}\n    \\frac{\\gamma L \\gamma \\mu {-} 2 \\gamma L {+} \\gamma \\mu}{L{-}\\mu} \\frac{\\|g_{i}\\|^2}{2} +\n    \\frac{\\gamma \\mu}{L{-}\\mu} \\frac{\\|g_{i+1}\\|^{2}}{2} +\n    \\frac{1 {-} \\gamma \\mu}{L{-}\\mu} \\frac{\\|g_{i} {-} g_{i+1}\\|^{2}}{2}.%\n\\end{align*}%\n","sha256_utf8":"c94de6d5fb22bfbf148e625e226336cca5a90c6519cf13ca58919d75089c916a"}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","title":"形成相邻双向和","source_units":[{"start":106245,"end":106720,"start_line":1409,"end_line":1417,"raw_tex":"Consider the sum of distance-1 inequalities $\\bar{I}_{i}^{1} {}\\coloneqq{} \\Iij[i][i+1] + \\Iji[i][i+1]$:%\n\\begin{align}\\label{eq:dist_1_blended_mixed_grads_diff} \\tag{$\\bar{I}_{i}^{1}$}\n\\hspace{-0.5em}\n0\n    {}\\geq{}\n\\frac{2 \\gamma L \\gamma \\mu {-} \\gamma L {-} \\gamma \\mu}{L{-}\\mu} \\frac{\\|g_{i}\\|^2}{2} {}+{}\n\\frac{\\gamma L {+} \\gamma \\mu}{L{-}\\mu} \\frac{\\|g_{i+1}\\|^2}{2} {}+{} \n\\frac{2{-}\\gamma L {-} \\gamma \\mu}{L{-}\\mu} \\frac{\\| g_{i} {-} g_{i+1}\\|^2}{2}.\n\\end{align}%\n","sha256_utf8":"dbfac2519e5fad07febcebe73eb2e374ee866d3a87655ba5acc0ecdf18d1504a"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"加权消去梯度差项","source_units":[{"start":106720,"end":107042,"start_line":1418,"end_line":1418,"raw_tex":"By weighting \\eqref{eq:dist_1_blended_mixed_grads_diff} with $\\beta(\\gamma L,\\gamma \\mu){}\\coloneqq{} \\tfrac{\\gamma L{-}1}{2{-}\\gamma L {-}\\gamma \\mu} \\geq 0$ and summing it with \\eqref{eq:tight_short_step_sizes}, the coefficient of $\\| g_{i} {-} g_{i+1}\\|^2$ gets canceled and we obtain \\eqref{eq:one_step_rate_h_geq_1}.\n","sha256_utf8":"a441219c58a346cb297cf9a47aab9036971d158ce804e4bc5aca1ad460ca5a87"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"检查等号与系数符号","source_units":[{"start":107042,"end":107506,"start_line":1419,"end_line":1420,"raw_tex":"The equality condition \\eqref{eq:equality_case_h_geq_1} follows from the equality case of \\eqref{eq:dist_1_blended_mixed_grads_diff}. Nevertheless, the coefficient of $\\|g_{i}\\|^2$ is nonnegative for $\\gamma L \\in [1,\\gbari[1]]$, where $\\gbari[1]$ is the largest stepsize $\\gamma L \\in [1, 2)$ for which $(2-\\gamma L)(2-\\gamma \\mu) - 1 \\geq {} 0$, implying nonnegativity of the weight of $\\|\\nabla f(x_{i})\\|^2$ in \\eqref{eq:one_step_rate_h_geq_1}.\\qed\n\\end{proof}","sha256_utf8":"3e0fde6be2d6b41ada8c9a60d68d72827b3a7cb09d42a5387efebeb02b35d196"}]},{"id":"S068","semantic_type":"proposition","kind":"proposition","title":"收缩因子符号性质","source_units":[{"start":109822,"end":110060,"start_line":1458,"end_line":1460,"raw_tex":"\\begin{proposition}\\label{prop:properties:m+n>0}%[Properties for distance-2]\n    Let $\\mu \\in (-\\infty, L)$ and $\\gamma L \\in \\big(1, \\gbari[\\infty] \\big)$. Then $\\m {}>{}  0$, $\\m + \\n {}>0$ and $\\n^{-2} {}>{} \\m^{-2}$.\n\\end{proposition}","sha256_utf8":"0783371a0a3d71725a6939ad1c0b368212330714ff9ecf963f1d02a6004a3ade"}]},{"id":"S069","semantic_type":"proof","kind":"proof","title":"收缩因子符号证明","source_units":[{"start":110062,"end":110491,"start_line":1461,"end_line":1463,"raw_tex":"\\begin{proof}%\nCondition $\\gamma L < \\gbari[\\infty]$ implies $\\gamma L < \\frac{2}{1+[\\kappa]_{+}}$, hence $0 < 2 {-} \\gamma L {-} \\gamma L [\\kappa]_{+} {}\\leq{} 2 {-} \\gamma L {-} \\gamma L \\kappa$, implying $\\m + \\n > 0$. Since $\\n = 1-\\gamma L \\in (-1,0)$, we have $\\m=1-\\gamma \\mu > 0$. Condition $L>\\mu$ implies $\\m>\\n$ and together with $\\m+\\n>0$ we conclude that $\\m^2 > \\n^2$, therefore $\\n^{-2} > \\m^{-2}$.\\qed\n\\end{proof}","sha256_utf8":"2b18a3dc9b529a65541f9785689d21b0a7df7e39f99bb0540a57d74087780ac1"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"由步长上界比较平方","source_units":[{"start":110062,"end":110491,"start_line":1461,"end_line":1463,"raw_tex":"\\begin{proof}%\nCondition $\\gamma L < \\gbari[\\infty]$ implies $\\gamma L < \\frac{2}{1+[\\kappa]_{+}}$, hence $0 < 2 {-} \\gamma L {-} \\gamma L [\\kappa]_{+} {}\\leq{} 2 {-} \\gamma L {-} \\gamma L \\kappa$, implying $\\m + \\n > 0$. Since $\\n = 1-\\gamma L \\in (-1,0)$, we have $\\m=1-\\gamma \\mu > 0$. Condition $L>\\mu$ implies $\\m>\\n$ and together with $\\m+\\n>0$ we conclude that $\\m^2 > \\n^2$, therefore $\\n^{-2} > \\m^{-2}$.\\qed\n\\end{proof}","sha256_utf8":"2b18a3dc9b529a65541f9785689d21b0a7df7e39f99bb0540a57d74087780ac1"}]},{"id":"S071","semantic_type":"lemma","kind":"lemma","title":"上曲率单步望远镜界","source_units":[{"start":110494,"end":111291,"start_line":1465,"end_line":1478,"raw_tex":"\\begin{lemma}\\label{lemma:dist_1_L_1_step_decrease}\nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty,L)$, and consider one iteration of \\eqref{eq:GM_it_fixed_steps} from $x_i$ with stepsize $\\gamma L \\in (1, 2)$. Then it holds:\n\\begin{align} \\label{eq:dist_1_L_1_step_decrease} \n    \\begin{aligned}    \n    \\frac{f(x_{i}) - f(x_{i+1})}{\\gamma} \n                {}\\geq{}\n            {-}\\Ei[i][\\n] \\frac{\\|\\nabla f(x_{i})\\|^2}{2} {}+{}\n            \\Ei[i+1][\\n] \\frac{\\|\\nabla f(x_{i+1})\\|^2}{2} {}+{}            \n            \\\\ {}+{}\n            \\frac{ 1 - (\\m+\\n) \\Ei[i+1][\\n] }{\\m-\\n} \\frac{\\|\\nabla f(x_{i+1})-\\n \\nabla f(x_{i})\\|^2}{2}.\n    \\end{aligned}\n\\end{align}%\nFurthermore, if $\\mu \\in (0,L)$ and $\\gamma L \\in [\\frac{2}{1+\\kappa}, 2)$, then $1-(\\m+\\n) \\Ei[i][\\n] > 0$.\n\\end{lemma}","sha256_utf8":"82466954b224527d0f944db47ea7c4f0c70c9c848453ca8d1f17712bbfcf4d57"}]},{"id":"S072","semantic_type":"proof","kind":"proof","title":"上曲率单步界证明","source_units":[{"start":111292,"end":112517,"start_line":1479,"end_line":1498,"raw_tex":"\\begin{proof}\nLet $\\beta_{i+1}(\\n) {{}{}\\coloneqq{}{}} (-\\n) \\Ei[i+1][\\n] {}>{} 0$. Inequality \\eqref{eq:dist_1_L_1_step_decrease} results after performing the simplifications in the linear combination of interpolation inequalities:\n$$\n\\Iij[i][i+1] + \\beta_{i+1}(\\n) \\, (\\Iij[i][i+1] + \\Iji[i][i+1]),$$\nwhere \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} are equivalently rewritten for the pair $(i,i+1)$ as:%\n$$\n    \\begin{aligned}\n    (1+\\beta_{i+1}) {\\,} \\Iij[i][i+1] {}:{}\n        \\frac{f_{i} {-} f_{i+1}}{\\gamma}\n            {}\\geq{}&\n        (1+\\n) \\frac{\\|g_{i}\\|^2}{2} {+} \\frac{1}{\\m{-}\\n} \\frac{\\|g_{i+1}{-}\\n g_{i}\\|^2}{2}; \\\\\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\beta_{i+1} {\\,} \\Iji[i][i+1] {}:{}\n        \\frac{f_{i+1} {-} f_{i}}{\\gamma}\n            {}\\geq{}&\n        {-}\\frac{\\|g_{i}\\|^2}{2} {-} \\frac{1}{\\n} \\frac{\\|g_{i+1}\\|^2}{2} {+} \\frac{\\m}{\\n} \\frac{1}{\\m{-}\\n} \\frac{\\|g_{i+1}{-}\\n g_{i}\\|^2}{2}.\n    \\end{aligned}\n$$%\n\\noindent{}With $\\gamma L \\in {[}\\frac{2}{1+\\kappa}, 2)$, we have $\\m{+}\\n \\leq 0$ (see \\cref{prop:properties:m+n>0}) and therefore $1-(\\m+\\n)\\Ei[i+1][\\n] > 0$.%\n\\qed \\end{proof}","sha256_utf8":"29b89c3117b02d0bc6ad35df97fa0581a925b836e01d79cd18098a34ef5f1d06"}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","title":"指定上曲率组合权重","source_units":[{"start":111292,"end":111595,"start_line":1479,"end_line":1482,"raw_tex":"\\begin{proof}\nLet $\\beta_{i+1}(\\n) {{}{}\\coloneqq{}{}} (-\\n) \\Ei[i+1][\\n] {}>{} 0$. Inequality \\eqref{eq:dist_1_L_1_step_decrease} results after performing the simplifications in the linear combination of interpolation inequalities:\n$$\n\\Iij[i][i+1] + \\beta_{i+1}(\\n) \\, (\\Iij[i][i+1] + \\Iji[i][i+1]),$$\n","sha256_utf8":"4152b8a79a0396d2dd990d9fecc9555276166c14f65b37a9b5a708a56ffa6edf"}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","title":"重写双向相邻插值式","source_units":[{"start":111595,"end":112339,"start_line":1483,"end_line":1496,"raw_tex":"where \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} are equivalently rewritten for the pair $(i,i+1)$ as:%\n$$\n    \\begin{aligned}\n    (1+\\beta_{i+1}) {\\,} \\Iij[i][i+1] {}:{}\n        \\frac{f_{i} {-} f_{i+1}}{\\gamma}\n            {}\\geq{}&\n        (1+\\n) \\frac{\\|g_{i}\\|^2}{2} {+} \\frac{1}{\\m{-}\\n} \\frac{\\|g_{i+1}{-}\\n g_{i}\\|^2}{2}; \\\\\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\beta_{i+1} {\\,} \\Iji[i][i+1] {}:{}\n        \\frac{f_{i+1} {-} f_{i}}{\\gamma}\n            {}\\geq{}&\n        {-}\\frac{\\|g_{i}\\|^2}{2} {-} \\frac{1}{\\n} \\frac{\\|g_{i+1}\\|^2}{2} {+} \\frac{\\m}{\\n} \\frac{1}{\\m{-}\\n} \\frac{\\|g_{i+1}{-}\\n g_{i}\\|^2}{2}.\n    \\end{aligned}\n$$%\n","sha256_utf8":"5bed4e860fae8d3de8e266ac7124450c9651cd519e0848123c6aefe7c83426cd"}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","title":"检查大步长余项为正","source_units":[{"start":112339,"end":112517,"start_line":1497,"end_line":1498,"raw_tex":"\\noindent{}With $\\gamma L \\in {[}\\frac{2}{1+\\kappa}, 2)$, we have $\\m{+}\\n \\leq 0$ (see \\cref{prop:properties:m+n>0}) and therefore $1-(\\m+\\n)\\Ei[i+1][\\n] > 0$.%\n\\qed \\end{proof}","sha256_utf8":"009520a207de96007404ed1c74b2d74e77fc4dc65152d701f498fddea024ea69"}]},{"id":"S076","semantic_type":"lemma","kind":"lemma","title":"下曲率单步望远镜界","source_units":[{"start":112521,"end":113349,"start_line":1500,"end_line":1513,"raw_tex":"\\begin{lemma}\\label{lemma:dist_1_mu_1_step_decrease}\nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty,L)$, and consider one iteration of \\eqref{eq:GM_it_fixed_steps} from $x_i$ with stepsize $\\gamma L \\in \\big(0, \\gbari[\\infty]\\big)$. Then it holds:%\n\\begin{align} \\label{eq:dist_1_mu_1_step_decrease}\n    \\begin{aligned}    \n    \\frac{f(x_{i}) - f(x_{i+1})}{\\gamma} \n                {}\\geq{} \n            -\\Ei[i][\\m] \\frac{\\|\\nabla f(x_{i})\\|^2}{2} {}+{}\n             \\Ei[i+1][\\m] \\frac{\\|\\nabla f(x_{i+1})\\|^2}{2} {}+{}            \n            \\\\ {}+{}\n            \\frac{{-1} +(\\m+\\n) \\Ei[i+1][\\m]}{\\m-\\n} \\frac{\\|\\nabla f(x_{i+1})-\\m \\nabla f(x_{i})\\|^2}{2}.\n    \\end{aligned}\n\\end{align}%\nFurthermore, if $\\mu \\in (0,L)$ and $\\gamma L \\in \\big(0, \\gbari[i+1]\\big)$, then ${-1} +(\\m+\\n) \\Ei[i+1][\\m] > 0$.%\n\\end{lemma}","sha256_utf8":"151fdb4d95076ee9f25d7bb29dedaf37b1feb5624684c616f88776dbaa48e0cf"}]},{"id":"S077","semantic_type":"proof","kind":"proof","title":"下曲率单步界证明","source_units":[{"start":113351,"end":115368,"start_line":1514,"end_line":1543,"raw_tex":"\\begin{proof}%\nFirstly, note that $\\m=1-\\gamma \\mu > 0$; the case $\\gamma L \\in \\big(1, \\frac{2}{1+[\\kappa]_{+}}\\big)$ is covered by \\cref{prop:properties:m+n>0}, while for $\\gamma L \\in (0,1]$ we have $1-\\gamma \\mu > 1-\\gamma L \\geq 0$. Let:%\n$$\n    \\beta_{i+1}(\\m) {}{}\\coloneqq{}{} {-1} + \\m \\Ei[i+1][\\m] {}={} \\frac{1 + \\m + \\dots + \\m^{2i}}{\\m^{2i+1}} > 0.\n$$%\nInequality \\eqref{eq:dist_1_mu_1_step_decrease} results after performing the simplifications in the linear combination of interpolation inequalities:%\n$$\n\\Iij[i][i+1] + \\beta_{i+1}(\\m) \\, (\\Iij[i][i+1] + \\Iji[i][i+1]),$$\nwhere \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} equivalently rewrite for the pair $(i,i+1)$ as:\n\\begin{align*}\n    (1+ \\beta_{i+1}) {\\,} \\Iij[i][i+1] {}:{}\\,{}{} \n        \\frac{f_{i} {-} f_{i+1}}{\\gamma}\n            {}\\geq{} &\n        \\frac{\\|g_{i}\\|^2}{2} {+} \\frac{1}{\\m} \\frac{\\|g_{i+1}\\|^2}{2} {+} \\frac{\\frac{\\n}{\\m}}{\\m{-}\\n} \\frac{\\|g_{i+1}{-}\\m g_{i}\\|^2}{2}; \\\\    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n     \\beta_{i+1} {\\,} \\Iji[i][i+1] {}:{}\\,{}{}\n        \\frac{f_{i+1} {-} f_{i}}{\\gamma}\n            {}\\geq{} &\n        {-}(1{+}\\m) \\frac{\\|g_{i}\\|^2}{2} {+} \\frac{1}{\\m{-}\\n} \\frac{\\|g_{i+1}{-}\\m g_{i}\\|^2}{2}.\n\\end{align*}%\nCondition $\\gamma L < \\gbari[i+1]$ implies $\\Tinm[i+1] < 0$, hence $i \\geq \\Nbar$ (recall \\cref{def:N_bar}: $\\Tinm[\\bar{N}] \\geq 0$ and $\\Tinm[\\bar{N}+1] < 0$, corresponding to $\\gamma L < \\gbari[\\bar{N}+1]$). Then the coefficient of the mixed term in \\eqref{eq:dist_1_mu_1_step_decrease} is positive:%\n$$\n    \\frac{{-1} {+} (\\m{+}\\n) E_{i+1}(\\m)}{\\m{-}\\n} \n        {}\\geq{} \n    \\frac{{-1} {+} (\\m{+}\\n) E_{\\bar{N}+1}(\\m)}{\\m{-}\\n}\n        {}={}\n    \\frac{\\Tinm[\\bar{N}]{-}\\n^2 \\Tinm[\\bar{N}+1]}{(\\m{-}\\n)^2} \n        {}>{} 0.\n$$\nThe first inequality results from the monotonic increase with index $i$ of $E_{i+1}(\\m)$, whereas the last one by the definition of $\\Nbar$.%\n\\qed \\end{proof}","sha256_utf8":"7cbf9733ad54dcdcc3e6f5dcd20667486339fd034f9140b46f90ac8ff5952f8a"}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","title":"验证下曲率组合权重为正","source_units":[{"start":113351,"end":113717,"start_line":1514,"end_line":1518,"raw_tex":"\\begin{proof}%\nFirstly, note that $\\m=1-\\gamma \\mu > 0$; the case $\\gamma L \\in \\big(1, \\frac{2}{1+[\\kappa]_{+}}\\big)$ is covered by \\cref{prop:properties:m+n>0}, while for $\\gamma L \\in (0,1]$ we have $1-\\gamma \\mu > 1-\\gamma L \\geq 0$. Let:%\n$$\n    \\beta_{i+1}(\\m) {}{}\\coloneqq{}{} {-1} + \\m \\Ei[i+1][\\m] {}={} \\frac{1 + \\m + \\dots + \\m^{2i}}{\\m^{2i+1}} > 0.\n$$%\n","sha256_utf8":"5bdfe3c9662628690748378ba4f0213dfe1a4dbea762aa363f9543e89aeb9409"}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","title":"组合双向插值取得单步界","source_units":[{"start":113717,"end":114679,"start_line":1519,"end_line":1532,"raw_tex":"Inequality \\eqref{eq:dist_1_mu_1_step_decrease} results after performing the simplifications in the linear combination of interpolation inequalities:%\n$$\n\\Iij[i][i+1] + \\beta_{i+1}(\\m) \\, (\\Iij[i][i+1] + \\Iji[i][i+1]),$$\nwhere \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} equivalently rewrite for the pair $(i,i+1)$ as:\n\\begin{align*}\n    (1+ \\beta_{i+1}) {\\,} \\Iij[i][i+1] {}:{}\\,{}{} \n        \\frac{f_{i} {-} f_{i+1}}{\\gamma}\n            {}\\geq{} &\n        \\frac{\\|g_{i}\\|^2}{2} {+} \\frac{1}{\\m} \\frac{\\|g_{i+1}\\|^2}{2} {+} \\frac{\\frac{\\n}{\\m}}{\\m{-}\\n} \\frac{\\|g_{i+1}{-}\\m g_{i}\\|^2}{2}; \\\\    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n     \\beta_{i+1} {\\,} \\Iji[i][i+1] {}:{}\\,{}{}\n        \\frac{f_{i+1} {-} f_{i}}{\\gamma}\n            {}\\geq{} &\n        {-}(1{+}\\m) \\frac{\\|g_{i}\\|^2}{2} {+} \\frac{1}{\\m{-}\\n} \\frac{\\|g_{i+1}{-}\\m g_{i}\\|^2}{2}.\n\\end{align*}%\n","sha256_utf8":"9ab35e80894a9bdf403bc14af848e2b06500945a5c42c6e57fd5d476b064be0c"}]},{"id":"S080","semantic_type":"proof_step","kind":"proof_step","title":"借阈值符号验证余项","source_units":[{"start":114679,"end":115368,"start_line":1533,"end_line":1543,"raw_tex":"Condition $\\gamma L < \\gbari[i+1]$ implies $\\Tinm[i+1] < 0$, hence $i \\geq \\Nbar$ (recall \\cref{def:N_bar}: $\\Tinm[\\bar{N}] \\geq 0$ and $\\Tinm[\\bar{N}+1] < 0$, corresponding to $\\gamma L < \\gbari[\\bar{N}+1]$). Then the coefficient of the mixed term in \\eqref{eq:dist_1_mu_1_step_decrease} is positive:%\n$$\n    \\frac{{-1} {+} (\\m{+}\\n) E_{i+1}(\\m)}{\\m{-}\\n} \n        {}\\geq{} \n    \\frac{{-1} {+} (\\m{+}\\n) E_{\\bar{N}+1}(\\m)}{\\m{-}\\n}\n        {}={}\n    \\frac{\\Tinm[\\bar{N}]{-}\\n^2 \\Tinm[\\bar{N}+1]}{(\\m{-}\\n)^2} \n        {}>{} 0.\n$$\nThe first inequality results from the monotonic increase with index $i$ of $E_{i+1}(\\m)$, whereas the last one by the definition of $\\Nbar$.%\n\\qed \\end{proof}","sha256_utf8":"f7063efd232b6f2a1aac77c6ee4647827c035a2b416826d86ed65eb17f2d1e12"}]},{"id":"S081","semantic_type":"lemma","kind":"lemma","title":"核心距离二下降关系","source_units":[{"start":115606,"end":116504,"start_line":1549,"end_line":1562,"raw_tex":"\\begin{lemma}[Central distance-2 result]\\label{lemma:telescoped_main_lemma}\nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty, L)$. Consider $k+1$ iterations of \\eqref{eq:GM_it_fixed_steps} with constant stepsize $\\gamma L \\in \\big[\\gbari[k], \\gbari[\\infty] \\big)$ starting from $x_0$, where $k \\geq 1$. Then it holds:%\n\\begin{align}\n        & \\frac{f(x_{0}) {-} f(x_{k})}{\\gamma} +\n        \\frac{(\\m{+}\\n)\\Tinm[k]}{2(\\m{-}\\n)} \\frac{f(x_{k}) {-} f(x_{k+1})}{\\gamma}\n            \\geq\n        % \\\\ & {}\\qquad{}\n        \\left(\\Ei[k][\\m] {+} \\Ei[k][\\n]\\right) \\frac{\\|\\nabla f(x_{k})\\|^2}{4} +\n        \\nonumber \\\\ & {}\\quad{}\\,\n        {}+{}\\frac{\\Tinm[k]}{2(\\m{-}\\n)} \\Big[ \\m \\n \\frac{\\|\\nabla f(x_{k})\\|^2}{2} + \n        \\langle \\nabla f(x_{k}) , \\nabla f(x_{k+1}) \\rangle + \n        \\frac{\\|\\nabla f(x_{k+1})\\|^2}{2} \\Big]. \\label{eq:telescoped_main_lemma} \\tag{D2}\n\\end{align}%\n\\end{lemma}","sha256_utf8":"07d9d0b5ecc3b9fe7707662d51edc2216f008f8f01501eff63364353729e7f0e"}]},{"id":"S082","semantic_type":"proof","kind":"proof","title":"核心距离二关系证明","source_units":[{"start":116506,"end":121932,"start_line":1563,"end_line":1682,"raw_tex":"\\begin{proof}\nConsider two iterations connecting $x_{i}$, $x_{i+1}$ and $x_{i+2}$, with $i \\geq 0$. The interpolation inequalities \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} written for distance-1 (\\Iij[i][i+1] and \\Iji[i][i+1]) and distance-2 (\\Iij[i][i+2] and \\Iji[i][i+2]), respectively:%\n\\begin{align*}\n    &\\begin{aligned}    \n    % & \\alpha_{[i-1>i]} \\times \n    \\Iij[i][i+1]\\colon \n        \\frac{f_{i} - f_{i+1}}{\\gamma}\n            {}\\geq{}\n        (1+\\n) \\frac{\\|g_{i}\\|^2}{2} + \\frac{1}{\\m-\\n} \\frac{\\|g_{i+1}-\\n g_{i}\\|^2}{2}; \n    \\end{aligned}\\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    &\\begin{aligned}    \n    % & \\alpha_{[i-1<i]}  \\times \n    \\Iji[i][i+1]\\colon \n        \\frac{f_{i+1} - f_{i}}{\\gamma}\n            {}\\geq{}\n        -\\frac{\\|g_{i}\\|^2}{2} - \\frac{1}{\\n} \\frac{\\|g_{i+1}\\|^2}{2} + \\frac{\\m}{\\n} \\frac{1}{\\m-\\n} \\frac{\\|g_{i+1}-\\n g_{i}\\|^2}{2}; \n    \\end{aligned}\\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    &\\begin{aligned}    \n    &% \\alpha_{[i-1>i+1]}  \\times \n    \\Iij[i][i+2]\\colon \n        \\frac{f_{i} - f_{i+2}}{\\gamma}\n            {}\\geq{}\n        (1+\\n+\\n^2) \\frac{\\|g_{i}\\|^2}{2} + \\n \\frac{\\|g_{i+1}\\|^2}{2} + \n        \\frac{1}{\\n(\\m-\\n)} \\cdot\n        \\\\ & {}\\qquad{}        \n        \\Big[\n        \\n(1-\\m) \\frac{\\|g_{i+1}-\\n g_{i}\\|^2}{2}\n        - (1-\\n) \\frac{\\|g_{i+2}-\\n g_{i+1}\\|^2}{2}\n        + \\frac{\\|g_{i+2}-\\n^2 g_{i}\\|^2}{2} \n        \\Big];\n    \\end{aligned}\\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    &\\begin{aligned}    \n    &% \\alpha_{[i-1<i+1]}  \\times \n    \\Iji[i][i+2]\\colon\n        \\frac{f_{i+2} - f_{i}}{\\gamma}\n            {}\\geq{} \n        - \\frac{\\|g_{i}\\|^2}{2} - \\frac{1}{\\n} \\frac{\\|g_{i+1}\\|^2}{2}\n        - \\frac{1+\\n}{\\n^2} \\frac{\\|g_{i+2}\\|^2}{2} + \\frac{1}{\\n(\\m-\\n)} \\cdot\n        \\\\ & {}\\qquad{} \n        \\Big[\n        \\m(1-\\n) \\frac{\\|g_{i+1}-\\n g_{i}\\|^2}{2}\n        - (1-\\m) \\frac{\\|g_{i+2}-\\n g_{i+1}\\|^2}{2}\n        + \\frac{\\m}{\\n} \\frac{\\|g_{i+2}-\\n^2 g_{i}\\|^2}{2} \n        \\Big],\n    \\end{aligned}\n\\end{align*}%\nare summed up in a linear combination with \\textit{nonnegative} corresponding weights:%\n$$\n        \\eqref{eq:dist_2_atom_ineq} \\equiv\n\\alphaij[i][i+1] \\Iij[i][i+1] +\n\\alphaij[i+1][i] \\Iji[i][i+1] +\n\\alphaij[i][i+2] \\Iij[i][i+2] +\n\\alphaij[i+2][i] \\Iji[i][i+2],\n$$%\nwhere:%\n$$\n\\begin{aligned}\n    \\alphaij[i][i+1] {}{}\\coloneqq{}{}& \n    1 - \\n \\Ei[i+1][\\n] + \\frac{(1-\\m) \\Tinm[i]}{2(\\m-\\n)} - \\frac{\\m(1+\\n) \\Tinm[i+1]}{2(\\m-\\n)};\n    \\\\\n   \\alphaij[i+1][i] {}{}\\coloneqq{}{}& \n    -\\n \\Ei[i+1][\\n] + \\frac{(1+\\n) \\Tinm[i]}{2(\\m-\\n)} + \\frac{\\n(1-\\m) \\Tinm[i+1]}{2(\\m-\\n)};\n    \\\\\n    \\alphaij[i][i+2] {}{}\\coloneqq{}{}&\n     \\frac{\\m \\Tinm[i+1]}{2(\\m-\\n)}; \n     \\\\\n    \\alphaij[i+2][i] {}{}\\coloneqq{}{}&\n    \\frac{-\\n \\Tinm[i+1]}{2(\\m-\\n)},\n\\end{aligned}%\n$$%\nand \\eqref{eq:dist_2_atom_ineq} is obtained after performing the simplifications:%\n\\begin{align}\\label{eq:dist_2_atom_ineq}    \n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\begin{aligned}\n        &\n        \\frac{f_{i} {-} f_{i+1}}{\\gamma} + \n        (\\m{+}\\n) \\Big[\n        {-}\\frac{\\Tinm[i]}{2(\\m{-}\\n)} \\frac{f_{i} {-} f_{i+1}}{\\gamma} +\n           \\frac{\\Tinm[i+1]}{2(\\m{-}\\n)} \\frac{f_{i+1} {-} f_{i+2}}{\\gamma}\n        \\Big]\n            {}\\geq{}\n        \\\\ & {}\\,{} \n        {}{-}{}\\frac{\\Tinm[i]}{2(\\m{-}\\n)} \\Big[ \\m \\n \\frac{\\|g_{i}\\|^2}{2} + \n        \\langle g_{i}, g_{i+1} \\rangle + \\frac{\\|g_{i+1}\\|^2}{2}  \\Big] \n        {-}\\left(\\Ei[i][\\m] {+} \\Ei[i][\\n]\\right) \\frac{\\|g_{i}\\|^2}{4}\n        \\\\ & {}\\,{} \n        {}{+}{}\\frac{\\Tinm[i+1]}{2(\\m{-}\\n)} \\Big[ \\m \\n \\frac{\\|g_{i+1}\\|^2}{2} \n        {+} {\\langle g_{i+1}, g_{i+2} \\rangle} + \\frac{\\|g_{i+2}\\|^2}{2}  \\Big] {}+{} \\left(\\Ei[i+1][\\m] {+} \\Ei[i+1][\\n]\\right) \\frac{\\|g_{i+1}\\|^2}{4}.\n    \\end{aligned}\n\\end{align}%\n\\noindent{}Telescoping it for indices $i=0,1,\\dots,k-1$ yields inequality \\eqref{eq:telescoped_main_lemma}. It remains to show the nonnegativity of multipliers. Since the stepsize thresholds increase with index $i$, condition $\\gamma L \\geq \\gbari[k]$ ensures $\\Tinm[i+1] \\geq 0$, for all $i=0,{\\dots},k-1$. This directly implies the positivity of $\\alphaij[i][i+2]$ and $\\alphaij[i+2][i]$. The positivity of distance-1 multipliers is evidenced by their equivalent representations as nonnegative sums (recall $\\n \\in (-1,0)$, $\\m \\in \\left(0,\\infty\\right)$; $\\m+\\n>0$ and $\\n^{-2} > \\m^{-2}$ from \\cref{prop:properties:m+n>0}):%\n$$\n\\begin{aligned}\n    &\\begin{aligned}    \n    2\\alphaij[i][i+1] {}={}& \n    1 + \\frac{\\m+\\m^{-2i}}{1+\\m} + \n    \\frac{(1+\\n)(\\m+\\n-\\n\\m)}{\\m \\n^2} \\frac{{-1} +\\n^{-2(i+1)}}{{-1} +\\n^{-2}} + \n    \\\\& {}\\qquad\\qquad{}\n    \\frac{(1+\\n)(1+\\m^2)}{\\m(\\m-\\n)} \n    % \\Big[ \\frac{1-\\n^{-2i}}{1-\\n^{-2}} - \\frac{1-\\m^{-2i}}{1-\\m^{-2}} \\Big] \n    \\sum_{j=0}^{i} \\big(\\n^{-2j} - \\m^{-2j}\\big)\n    {}>{} 0;  \n    \\end{aligned} \\\\%\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    &\\begin{aligned}  \n    2\\alphaij[i+1][i] {}={}&\n    \\frac{{-1} +\\n^{-2i}}{1-\\n} + \n    \\frac{\\m(1+\\n)}{1-\\n} \\frac{{-1} +\\m^{-2(i+1)}}{1-\\m}\n    + \\\\& {}\\qquad\\qquad{}\n     +\\frac{-\\n \\big[1-\\n+\\m(1+\\n)\\big]}{1-\\n} \\frac{\\n^{-2(i+1)} - \\m^{-2(i+1)}}{\\m-\\n} > 0. \n    % {}\\qquad\\qquad\\qquad\\qquad\\qquad{}\n    {}\\qquad{}\n    {}\\qed{}\n    \\end{aligned}%\n\\end{aligned}\n$$%\n\\end{proof}","sha256_utf8":"21a847512e29449a384eb973e62b74b0eb06a0d64fe761ffd91b50f13af1e10b"}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","title":"展开四种距离插值式","source_units":[{"start":116506,"end":118694,"start_line":1563,"end_line":1611,"raw_tex":"\\begin{proof}\nConsider two iterations connecting $x_{i}$, $x_{i+1}$ and $x_{i+2}$, with $i \\geq 0$. The interpolation inequalities \\eqref{eq:distance-N_interpolation_conditions_I_ij} and \\eqref{eq:distance-N_interpolation_conditions_I_ji} written for distance-1 (\\Iij[i][i+1] and \\Iji[i][i+1]) and distance-2 (\\Iij[i][i+2] and \\Iji[i][i+2]), respectively:%\n\\begin{align*}\n    &\\begin{aligned}    \n    % & \\alpha_{[i-1>i]} \\times \n    \\Iij[i][i+1]\\colon \n        \\frac{f_{i} - f_{i+1}}{\\gamma}\n            {}\\geq{}\n        (1+\\n) \\frac{\\|g_{i}\\|^2}{2} + \\frac{1}{\\m-\\n} \\frac{\\|g_{i+1}-\\n g_{i}\\|^2}{2}; \n    \\end{aligned}\\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    &\\begin{aligned}    \n    % & \\alpha_{[i-1<i]}  \\times \n    \\Iji[i][i+1]\\colon \n        \\frac{f_{i+1} - f_{i}}{\\gamma}\n            {}\\geq{}\n        -\\frac{\\|g_{i}\\|^2}{2} - \\frac{1}{\\n} \\frac{\\|g_{i+1}\\|^2}{2} + \\frac{\\m}{\\n} \\frac{1}{\\m-\\n} \\frac{\\|g_{i+1}-\\n g_{i}\\|^2}{2}; \n    \\end{aligned}\\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    &\\begin{aligned}    \n    &% \\alpha_{[i-1>i+1]}  \\times \n    \\Iij[i][i+2]\\colon \n        \\frac{f_{i} - f_{i+2}}{\\gamma}\n            {}\\geq{}\n        (1+\\n+\\n^2) \\frac{\\|g_{i}\\|^2}{2} + \\n \\frac{\\|g_{i+1}\\|^2}{2} + \n        \\frac{1}{\\n(\\m-\\n)} \\cdot\n        \\\\ & {}\\qquad{}        \n        \\Big[\n        \\n(1-\\m) \\frac{\\|g_{i+1}-\\n g_{i}\\|^2}{2}\n        - (1-\\n) \\frac{\\|g_{i+2}-\\n g_{i+1}\\|^2}{2}\n        + \\frac{\\|g_{i+2}-\\n^2 g_{i}\\|^2}{2} \n        \\Big];\n    \\end{aligned}\\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    &\\begin{aligned}    \n    &% \\alpha_{[i-1<i+1]}  \\times \n    \\Iji[i][i+2]\\colon\n        \\frac{f_{i+2} - f_{i}}{\\gamma}\n            {}\\geq{} \n        - \\frac{\\|g_{i}\\|^2}{2} - \\frac{1}{\\n} \\frac{\\|g_{i+1}\\|^2}{2}\n        - \\frac{1+\\n}{\\n^2} \\frac{\\|g_{i+2}\\|^2}{2} + \\frac{1}{\\n(\\m-\\n)} \\cdot\n        \\\\ & {}\\qquad{} \n        \\Big[\n        \\m(1-\\n) \\frac{\\|g_{i+1}-\\n g_{i}\\|^2}{2}\n        - (1-\\m) \\frac{\\|g_{i+2}-\\n g_{i+1}\\|^2}{2}\n        + \\frac{\\m}{\\n} \\frac{\\|g_{i+2}-\\n^2 g_{i}\\|^2}{2} \n        \\Big],\n    \\end{aligned}\n\\end{align*}%\n","sha256_utf8":"265d80be8d91d05faff0f955343125b4ce90fde027ff4405d31c516f61453547"}]},{"id":"S084","semantic_type":"proof_step","kind":"proof_step","title":"构造四个组合乘子","source_units":[{"start":118694,"end":119458,"start_line":1612,"end_line":1635,"raw_tex":"are summed up in a linear combination with \\textit{nonnegative} corresponding weights:%\n$$\n        \\eqref{eq:dist_2_atom_ineq} \\equiv\n\\alphaij[i][i+1] \\Iij[i][i+1] +\n\\alphaij[i+1][i] \\Iji[i][i+1] +\n\\alphaij[i][i+2] \\Iij[i][i+2] +\n\\alphaij[i+2][i] \\Iji[i][i+2],\n$$%\nwhere:%\n$$\n\\begin{aligned}\n    \\alphaij[i][i+1] {}{}\\coloneqq{}{}& \n    1 - \\n \\Ei[i+1][\\n] + \\frac{(1-\\m) \\Tinm[i]}{2(\\m-\\n)} - \\frac{\\m(1+\\n) \\Tinm[i+1]}{2(\\m-\\n)};\n    \\\\\n   \\alphaij[i+1][i] {}{}\\coloneqq{}{}& \n    -\\n \\Ei[i+1][\\n] + \\frac{(1+\\n) \\Tinm[i]}{2(\\m-\\n)} + \\frac{\\n(1-\\m) \\Tinm[i+1]}{2(\\m-\\n)};\n    \\\\\n    \\alphaij[i][i+2] {}{}\\coloneqq{}{}&\n     \\frac{\\m \\Tinm[i+1]}{2(\\m-\\n)}; \n     \\\\\n    \\alphaij[i+2][i] {}{}\\coloneqq{}{}&\n    \\frac{-\\n \\Tinm[i+1]}{2(\\m-\\n)},\n\\end{aligned}%\n$$%\n","sha256_utf8":"eb51ec8d30a38abd3575395ac01611edf49fd2a6513e9dbb2c23e6fe4df986ee"}]},{"id":"S085","semantic_type":"proof_step","kind":"proof_step","title":"化简得到局部消去关系","source_units":[{"start":119458,"end":120451,"start_line":1636,"end_line":1655,"raw_tex":"and \\eqref{eq:dist_2_atom_ineq} is obtained after performing the simplifications:%\n\\begin{align}\\label{eq:dist_2_atom_ineq}    \n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\begin{aligned}\n        &\n        \\frac{f_{i} {-} f_{i+1}}{\\gamma} + \n        (\\m{+}\\n) \\Big[\n        {-}\\frac{\\Tinm[i]}{2(\\m{-}\\n)} \\frac{f_{i} {-} f_{i+1}}{\\gamma} +\n           \\frac{\\Tinm[i+1]}{2(\\m{-}\\n)} \\frac{f_{i+1} {-} f_{i+2}}{\\gamma}\n        \\Big]\n            {}\\geq{}\n        \\\\ & {}\\,{} \n        {}{-}{}\\frac{\\Tinm[i]}{2(\\m{-}\\n)} \\Big[ \\m \\n \\frac{\\|g_{i}\\|^2}{2} + \n        \\langle g_{i}, g_{i+1} \\rangle + \\frac{\\|g_{i+1}\\|^2}{2}  \\Big] \n        {-}\\left(\\Ei[i][\\m] {+} \\Ei[i][\\n]\\right) \\frac{\\|g_{i}\\|^2}{4}\n        \\\\ & {}\\,{} \n        {}{+}{}\\frac{\\Tinm[i+1]}{2(\\m{-}\\n)} \\Big[ \\m \\n \\frac{\\|g_{i+1}\\|^2}{2} \n        {+} {\\langle g_{i+1}, g_{i+2} \\rangle} + \\frac{\\|g_{i+2}\\|^2}{2}  \\Big] {}+{} \\left(\\Ei[i+1][\\m] {+} \\Ei[i+1][\\n]\\right) \\frac{\\|g_{i+1}\\|^2}{4}.\n    \\end{aligned}\n\\end{align}%\n","sha256_utf8":"895577883dca100abe5c3005af83ba6444706a2a2f54ac50696cc52856c99002"}]},{"id":"S086","semantic_type":"proof_step","kind":"proof_step","title":"望远镜求和与阈值检查","source_units":[{"start":120451,"end":121080,"start_line":1656,"end_line":1656,"raw_tex":"\\noindent{}Telescoping it for indices $i=0,1,\\dots,k-1$ yields inequality \\eqref{eq:telescoped_main_lemma}. It remains to show the nonnegativity of multipliers. Since the stepsize thresholds increase with index $i$, condition $\\gamma L \\geq \\gbari[k]$ ensures $\\Tinm[i+1] \\geq 0$, for all $i=0,{\\dots},k-1$. This directly implies the positivity of $\\alphaij[i][i+2]$ and $\\alphaij[i+2][i]$. The positivity of distance-1 multipliers is evidenced by their equivalent representations as nonnegative sums (recall $\\n \\in (-1,0)$, $\\m \\in \\left(0,\\infty\\right)$; $\\m+\\n>0$ and $\\n^{-2} > \\m^{-2}$ from \\cref{prop:properties:m+n>0}):%\n","sha256_utf8":"90396e9de5a7591af02e415078de84ac5d7fcfd157468f5cf0d79a516cb2cd0e"}]},{"id":"S087","semantic_type":"proof_step","kind":"proof_step","title":"将相邻乘子写成正项和","source_units":[{"start":121080,"end":121932,"start_line":1657,"end_line":1682,"raw_tex":"$$\n\\begin{aligned}\n    &\\begin{aligned}    \n    2\\alphaij[i][i+1] {}={}& \n    1 + \\frac{\\m+\\m^{-2i}}{1+\\m} + \n    \\frac{(1+\\n)(\\m+\\n-\\n\\m)}{\\m \\n^2} \\frac{{-1} +\\n^{-2(i+1)}}{{-1} +\\n^{-2}} + \n    \\\\& {}\\qquad\\qquad{}\n    \\frac{(1+\\n)(1+\\m^2)}{\\m(\\m-\\n)} \n    % \\Big[ \\frac{1-\\n^{-2i}}{1-\\n^{-2}} - \\frac{1-\\m^{-2i}}{1-\\m^{-2}} \\Big] \n    \\sum_{j=0}^{i} \\big(\\n^{-2j} - \\m^{-2j}\\big)\n    {}>{} 0;  \n    \\end{aligned} \\\\%\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    &\\begin{aligned}  \n    2\\alphaij[i+1][i] {}={}&\n    \\frac{{-1} +\\n^{-2i}}{1-\\n} + \n    \\frac{\\m(1+\\n)}{1-\\n} \\frac{{-1} +\\m^{-2(i+1)}}{1-\\m}\n    + \\\\& {}\\qquad\\qquad{}\n     +\\frac{-\\n \\big[1-\\n+\\m(1+\\n)\\big]}{1-\\n} \\frac{\\n^{-2(i+1)} - \\m^{-2(i+1)}}{\\m-\\n} > 0. \n    % {}\\qquad\\qquad\\qquad\\qquad\\qquad{}\n    {}\\qquad{}\n    {}\\qed{}\n    \\end{aligned}%\n\\end{aligned}\n$$%\n\\end{proof}","sha256_utf8":"a51de7f0c1384dc77a233ea27ea2773c9e76a738df783c786a4332bfed196c5d"}]},{"id":"S088","semantic_type":"lemma","kind":"lemma","title":"多步双梯度下降界","source_units":[{"start":123252,"end":124132,"start_line":1690,"end_line":1702,"raw_tex":"\\begin{lemma}[Multistep descent]\\label{lemma:hypo_GN4SD}\nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty, L)$, and consider $k+1 {}\\geq{} 1$ iterations of \\eqref{eq:GM_it_fixed_steps} with stepsize $\\gamma L \\in [\\gbari[k], \\gbari[\\infty])$, {\\shownewtext{satisfying $\\gamma L > 1$,}} starting from $x_0$. Then the following inequality holds:%\n\\begin{align}%\n% \\hspace{-.25em}\n    \\frac{f(x_0) {-} f(x_{k+1})}{\\gamma}\n        {}\\geq{}\n    \\frac{{-}\\m^2 \\n^2 \\Tinm[k+1]}{\\m^2{-}\\n^2} \\frac{\\|\\nabla f(x_{k})\\|^2}{2} {}+{}\n    \\frac{\\Tinm[k]{+}(\\m{-}\\n)}{\\m^2{-}\\n^2} \\frac{\\|\\nabla f(x_{k+1})\\|^2}{2}.\n\\label{eq:general_ineq_proof} \\tag{GN4SD}\n% \\tag{$I_{N+1}^{N}$}\n\\end{align}%\nFurthermore, if $\\mu \\in (-\\infty, 0]$ and $\\gamma L \\in [\\gbari[k], \\gbari[k+1])$, then the gradient norms from \\eqref{eq:general_ineq_proof} are weighted by nonnegative quantities.\n\\end{lemma}","sha256_utf8":"5e061aa9818d7789359b32ff4535cb695c0299079ef7ea64fc6a53022de8f83c"}]},{"id":"S089","semantic_type":"proof","kind":"proof","title":"多步双梯度界证明","source_units":[{"start":124138,"end":126308,"start_line":1705,"end_line":1745,"raw_tex":"\\begin{proof}%\nFor $k=0$, the result follows from \\cref{lemma:sufficient_decrease_h_geq_1} and we obtain exactly \\eqref{eq:one_step_rate_h_geq_1}. \nFor $k\\geq 1$, inequality \\eqref{eq:general_ineq_proof} is derived by adjusting \\eqref{eq:telescoped_main_lemma} through the linear combination of interpolation inequalities $\\alphaij[k][k+1] \\Iij[k][k+1] + \\alphaij[k+1][k] \\Iji[k][k+1]$:%\n$$\n    \\begin{aligned}\n        &  \\Big[1 - \\frac{(\\m+\\n)\\,\\Tinm[k]}{2(\\m-\\n)} \\Big] \\frac{f_{k} - f_{k+1}}{\\gamma}\n        {}\\geq{}\n        \\\\ & {}\\qquad{}\n        \\Big(-\\Ei[k][\\m] - \\Ei[k][\\n] - \\frac{\\m \\n \\Tinm[k]}{\\m-\\n} {}-{} \\frac{2\\m^2\\n^2 \\Tinm[k+1]}{\\m^2-\\n^2} \\Big) \\frac{\\|g_{k}\\|^2}{4}\n        \\\\ & {}\\qquad{}        \n        % \\\\ & {}\\qquad{}\n        {}+{} \\Big(\\frac{\\m-\\n + \\Tinm[k]}{\\m^2-\\n^2} - \\frac{\\Tinm[k]}{2(\\m-\\n)} \\Big) \\frac{\\|g_{k+1}\\|^2}{2}\n        {}-{}\\frac{\\Tinm[k]}{2(\\m-\\n)} \\langle g_{k},g_{k+1} \\rangle,\n    \\end{aligned}\n$$%\nwhere the corresponding weights are defined as follows:%\n    $$\n    \\begin{aligned}\n        \\alphaij[k][k+1]\n             {}{}\\coloneqq{}{} &\n        1 + \\frac{-\\n}{\\m+\\n} + \\frac{(1-\\m)(\\m+\\n)-2\\n}{2(\\m^2-\\n^2)} \\Tinm[k];  \\\\\n        %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n        \\alphaij[k+1][k]\n            {}{}\\coloneqq{}{} &\n        \\frac{-\\n}{\\m+\\n} + \\frac{(1+\\n)(\\m+\\n)-2\\n}{2(\\m^2-\\n^2)} \\Tinm[k].\n    \\end{aligned}\n    $$ %\n    Multiplier $\\alphaij[k+1][k]$ is represented as a sum of positive quantities ($\\n < 0$).\n    Multiplier $\\alphaij[k][k+1]$ is directly written as sum of positive quantities if $(1-\\m)(\\m+\\n)-2\\n > 0$ (in particular, this covers the (strongly) convex case where $\\m \\leq 1$). Otherwise, $\\alphaij[k][k+1]$ is equivalently expressed as a sum of positive terms as follows, with $\\m>1$:%\n$$\n\\begin{aligned}\n    \\alphaij[k][k+1]\n        {}={}&\n    \\frac{\\m}{2(\\m+\\n)} + \\frac{1}{2(\\m^2-\\n^2)} \n        \\bigg[ \\frac{-\\n(\\m-\\n)(1+\\m^2)}{(\\m-1)(1-\\n)} {}+{}\n        \\\\ & {}\\quad{} \n        -\\big[(1-\\m)(\\m+\\n)-2\\n\\big] \\Big( \\frac{\\m^{-2k}}{\\m-1} + \\frac{\\n^{-2k}}{1-\\n} \\Big)  \\bigg]\n        {}>{} 0. {}\\quad{} \\qed\n\\end{aligned}\n$$\n\\end{proof}","sha256_utf8":"8f79ca0b050bc7379bae290176176454f8bb6116412c66cffb6868015490b133"}]},{"id":"S090","semantic_type":"proof_step","kind":"proof_step","title":"处理基例并补入相邻式","source_units":[{"start":124138,"end":124526,"start_line":1705,"end_line":1707,"raw_tex":"\\begin{proof}%\nFor $k=0$, the result follows from \\cref{lemma:sufficient_decrease_h_geq_1} and we obtain exactly \\eqref{eq:one_step_rate_h_geq_1}. \nFor $k\\geq 1$, inequality \\eqref{eq:general_ineq_proof} is derived by adjusting \\eqref{eq:telescoped_main_lemma} through the linear combination of interpolation inequalities $\\alphaij[k][k+1] \\Iij[k][k+1] + \\alphaij[k+1][k] \\Iji[k][k+1]$:%\n","sha256_utf8":"e7a8c34a9d5d9754dfa73c9a1fada2e4f9e6620d8aafac4b858fd85593aef4b5"}]},{"id":"S091","semantic_type":"proof_step","kind":"proof_step","title":"写出末步修正关系","source_units":[{"start":124526,"end":125086,"start_line":1708,"end_line":1719,"raw_tex":"$$\n    \\begin{aligned}\n        &  \\Big[1 - \\frac{(\\m+\\n)\\,\\Tinm[k]}{2(\\m-\\n)} \\Big] \\frac{f_{k} - f_{k+1}}{\\gamma}\n        {}\\geq{}\n        \\\\ & {}\\qquad{}\n        \\Big(-\\Ei[k][\\m] - \\Ei[k][\\n] - \\frac{\\m \\n \\Tinm[k]}{\\m-\\n} {}-{} \\frac{2\\m^2\\n^2 \\Tinm[k+1]}{\\m^2-\\n^2} \\Big) \\frac{\\|g_{k}\\|^2}{4}\n        \\\\ & {}\\qquad{}        \n        % \\\\ & {}\\qquad{}\n        {}+{} \\Big(\\frac{\\m-\\n + \\Tinm[k]}{\\m^2-\\n^2} - \\frac{\\Tinm[k]}{2(\\m-\\n)} \\Big) \\frac{\\|g_{k+1}\\|^2}{2}\n        {}-{}\\frac{\\Tinm[k]}{2(\\m-\\n)} \\langle g_{k},g_{k+1} \\rangle,\n    \\end{aligned}\n$$%\n","sha256_utf8":"675fbda041e8646d2ab90571310b43a423969bdfb1841cfc7f8a19635d1719b1"}]},{"id":"S092","semantic_type":"proof_step","kind":"proof_step","title":"指定末步修正乘子","source_units":[{"start":125086,"end":125567,"start_line":1720,"end_line":1731,"raw_tex":"where the corresponding weights are defined as follows:%\n    $$\n    \\begin{aligned}\n        \\alphaij[k][k+1]\n             {}{}\\coloneqq{}{} &\n        1 + \\frac{-\\n}{\\m+\\n} + \\frac{(1-\\m)(\\m+\\n)-2\\n}{2(\\m^2-\\n^2)} \\Tinm[k];  \\\\\n        %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n        \\alphaij[k+1][k]\n            {}{}\\coloneqq{}{} &\n        \\frac{-\\n}{\\m+\\n} + \\frac{(1+\\n)(\\m+\\n)-2\\n}{2(\\m^2-\\n^2)} \\Tinm[k].\n    \\end{aligned}\n    $$ %\n","sha256_utf8":"b579fb4b2ca0a6c38352193ba22bdb5bc4c31b2cc29f10ead63afdcf9ad830ea"}]},{"id":"S093","semantic_type":"proof_step","kind":"proof_step","title":"验证末步乘子为正","source_units":[{"start":125567,"end":126308,"start_line":1732,"end_line":1745,"raw_tex":"    Multiplier $\\alphaij[k+1][k]$ is represented as a sum of positive quantities ($\\n < 0$).\n    Multiplier $\\alphaij[k][k+1]$ is directly written as sum of positive quantities if $(1-\\m)(\\m+\\n)-2\\n > 0$ (in particular, this covers the (strongly) convex case where $\\m \\leq 1$). Otherwise, $\\alphaij[k][k+1]$ is equivalently expressed as a sum of positive terms as follows, with $\\m>1$:%\n$$\n\\begin{aligned}\n    \\alphaij[k][k+1]\n        {}={}&\n    \\frac{\\m}{2(\\m+\\n)} + \\frac{1}{2(\\m^2-\\n^2)} \n        \\bigg[ \\frac{-\\n(\\m-\\n)(1+\\m^2)}{(\\m-1)(1-\\n)} {}+{}\n        \\\\ & {}\\quad{} \n        -\\big[(1-\\m)(\\m+\\n)-2\\n\\big] \\Big( \\frac{\\m^{-2k}}{\\m-1} + \\frac{\\n^{-2k}}{1-\\n} \\Big)  \\bigg]\n        {}>{} 0. {}\\quad{} \\qed\n\\end{aligned}\n$$\n\\end{proof}","sha256_utf8":"401354f17f0d39241b256acf349ade78758dc1513131600d6939696033ac487f"}]},{"id":"S094","semantic_type":"lemma","kind":"lemma","title":"多步上曲率线性界","source_units":[{"start":126312,"end":127046,"start_line":1747,"end_line":1757,"raw_tex":"\\begin{lemma}\\label{lemma:general_lemma_ct_h_to_Linear} \nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty, L)$, and consider $k+1 \\geq 1$ iterations of \\eqref{eq:GM_it_fixed_steps} with stepsize $\\gamma L \\in [\\gbari[k], \\gbari[\\infty] \\big)$, satisfying $\\gamma L > 1$, starting from $x_0$. Then the following inequality holds:%\n\\begin{align}\\label{eq:general_ineq_proof_to_linear_compact}\n   \\frac{f(x_0) {-} f(x_{k+1})}{\\gamma}\n        {}\\geq{} \n    \\Ei[k+1][\\n] \\frac{\\|\\nabla f(x_{k+1})\\|^2}{2} {+} \n    \\frac{\\m^2 \\Tinm[k+1]}{(\\m{-}\\n)^2}\n    \\frac{\\| \\nabla f(x_{k+1}) {-} \\n \\nabla f(x_{k}) \\|^2}{2}.\n\\end{align}%\nFurthermore, if $\\gamma L \\in [\\gbari[k+1], \\gbari[\\infty] \\big)$, then $\\Tinm[k+1]\\geq 0 $.%\n\\end{lemma}","sha256_utf8":"06deb370a91f2e1d20da341b6e11b404bf1cb2fcbe1d82cd61a94d5dd21e4c59"}]},{"id":"S095","semantic_type":"proof","kind":"proof","title":"多步上曲率界证明","source_units":[{"start":127048,"end":128893,"start_line":1758,"end_line":1792,"raw_tex":"\\begin{proof}%\nFor $k=0$, the result follows from \\cref{lemma:dist_1_L_1_step_decrease}. For $k \\geq 1$, inequality \\eqref{eq:general_ineq_proof_to_linear_compact} is derived by adjusting \\eqref{eq:telescoped_main_lemma} with the linear combination of interpolation inequalities $\\alphaij[k][k+1] \\Iij[k][k+1] + \\alphaij[k][k+1] \\Iji[k][k+1]$:%\n$$\n    % \\hspace{-1em}\n    \\begin{aligned}\n        &\\Big[1 - \\frac{(\\m+\\n)\\,\\Tinm[k]}{2(\\m-\\n)} \\Big] \\frac{f_{k} - f_{k+1}}{\\gamma}\n            {}\\geq{}\n        \\Big(-\\frac{\\Tinm[k]}{2(\\m-\\n)} - \\frac{\\m^2 \\n \\Tinm[k+1]}{(\\m-\\n)^2}\\Big) \\langle g_{k},g_{k+1} \\rangle {}+{}\n        \\\\ & {}\\quad\\qquad{}\n        {}+{} \\Big[ \\frac{2\\m^2 \\n^2 \\Tinm[k+1]}{(\\m-\\n)^2} -\\Ei[k][\\m] - \\Ei[k][\\n] - \\frac{\\m \\n \\Tinm[k]}{\\m-\\n} \\Big] \\frac{\\|g_{k}\\|^2}{4} {}+{}\n        \\\\ & {}\\quad\\qquad{}\n        {}+{} \\Big[ \\frac{\\m^2 \\, \\Tinm[k+1]}{(\\m-\\n)^2}  + \\Ei[k+1][\\n] -\\frac{\\Tinm[k]}{2(\\m-\\n)} \\Big] \\frac{\\|g_{k+1}\\|^2}{2},\n    \\end{aligned}\n$$%\nwhere the corresponding weights are defined as:\n$$\n\\begin{aligned}\n    \\alphaij[k][k+1] \n        {}{}\\coloneqq{}{} &\n    1+(-\\n)\\Ei[k+1][\\n] + \\frac{1-\\m}{2(\\m-\\n)} \\Tinm[k];\n    \\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\alphaij[k+1][k]\n        {}{}\\coloneqq{}{} &\n    (-\\n) \\Ei[k+1][\\n] + \n    \\frac{1+\\n}{2(\\m-\\n)} \\Tinm[k].\n\\end{aligned}%\n$$%\nMultiplier $\\alphaij[k+1][k]$ is represented as a sum of positive quantities, while $\\alphaij[k][k+1]$ is directly written as sum of positive terms if $\\m \\leq 1$ (the (strongly) convex case). Otherwise, $\\m > 1$ and $\\alphaij[k][k+1]$ is equivalently expressed as sum of positive terms as:%\n$$\n        \\alphaij[k][k+1]\n    {}={}\n        \\frac{1+(-\\n) \\n^{-2(k+1)}}{2(1-\\n)} + \\m \\frac{\\m \\m^{-2(k+1)} + (-\\n) \\n^{-2(k+1)}}{2(\\m-\\n)}. {}\\quad{} \\qed\n$$\n\\end{proof}","sha256_utf8":"63daf2f8875c39b45f6a4aab97c70b2e7a7898387293277bfc695a973ceeed5b"}]},{"id":"S096","semantic_type":"proof_step","kind":"proof_step","title":"归结基例及核心关系","source_units":[{"start":127048,"end":127393,"start_line":1758,"end_line":1759,"raw_tex":"\\begin{proof}%\nFor $k=0$, the result follows from \\cref{lemma:dist_1_L_1_step_decrease}. For $k \\geq 1$, inequality \\eqref{eq:general_ineq_proof_to_linear_compact} is derived by adjusting \\eqref{eq:telescoped_main_lemma} with the linear combination of interpolation inequalities $\\alphaij[k][k+1] \\Iij[k][k+1] + \\alphaij[k][k+1] \\Iji[k][k+1]$:%\n","sha256_utf8":"06ba2265536c09b6bf2169ff12c6605a5caf3ab2c20f702701e99ee77916624a"}]},{"id":"S097","semantic_type":"proof_step","kind":"proof_step","title":"构造上曲率末步修正","source_units":[{"start":127393,"end":128028,"start_line":1760,"end_line":1771,"raw_tex":"$$\n    % \\hspace{-1em}\n    \\begin{aligned}\n        &\\Big[1 - \\frac{(\\m+\\n)\\,\\Tinm[k]}{2(\\m-\\n)} \\Big] \\frac{f_{k} - f_{k+1}}{\\gamma}\n            {}\\geq{}\n        \\Big(-\\frac{\\Tinm[k]}{2(\\m-\\n)} - \\frac{\\m^2 \\n \\Tinm[k+1]}{(\\m-\\n)^2}\\Big) \\langle g_{k},g_{k+1} \\rangle {}+{}\n        \\\\ & {}\\quad\\qquad{}\n        {}+{} \\Big[ \\frac{2\\m^2 \\n^2 \\Tinm[k+1]}{(\\m-\\n)^2} -\\Ei[k][\\m] - \\Ei[k][\\n] - \\frac{\\m \\n \\Tinm[k]}{\\m-\\n} \\Big] \\frac{\\|g_{k}\\|^2}{4} {}+{}\n        \\\\ & {}\\quad\\qquad{}\n        {}+{} \\Big[ \\frac{\\m^2 \\, \\Tinm[k+1]}{(\\m-\\n)^2}  + \\Ei[k+1][\\n] -\\frac{\\Tinm[k]}{2(\\m-\\n)} \\Big] \\frac{\\|g_{k+1}\\|^2}{2},\n    \\end{aligned}\n$$%\n","sha256_utf8":"9067d557db8f0ab8b03df712a030e18f4dbe619051dd79bbf116b5058cc5e869"}]},{"id":"S098","semantic_type":"proof_step","kind":"proof_step","title":"指定上曲率修正乘子","source_units":[{"start":128028,"end":128429,"start_line":1772,"end_line":1785,"raw_tex":"where the corresponding weights are defined as:\n$$\n\\begin{aligned}\n    \\alphaij[k][k+1] \n        {}{}\\coloneqq{}{} &\n    1+(-\\n)\\Ei[k+1][\\n] + \\frac{1-\\m}{2(\\m-\\n)} \\Tinm[k];\n    \\\\\n    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n    \\alphaij[k+1][k]\n        {}{}\\coloneqq{}{} &\n    (-\\n) \\Ei[k+1][\\n] + \n    \\frac{1+\\n}{2(\\m-\\n)} \\Tinm[k].\n\\end{aligned}%\n$$%\n","sha256_utf8":"f665ea587f32e225fe4038e4dd9e95094b912fe510fd6365b59ba01aa4f49b59"}]},{"id":"S099","semantic_type":"proof_step","kind":"proof_step","title":"验证上曲率乘子为正","source_units":[{"start":128429,"end":128893,"start_line":1786,"end_line":1792,"raw_tex":"Multiplier $\\alphaij[k+1][k]$ is represented as a sum of positive quantities, while $\\alphaij[k][k+1]$ is directly written as sum of positive terms if $\\m \\leq 1$ (the (strongly) convex case). Otherwise, $\\m > 1$ and $\\alphaij[k][k+1]$ is equivalently expressed as sum of positive terms as:%\n$$\n        \\alphaij[k][k+1]\n    {}={}\n        \\frac{1+(-\\n) \\n^{-2(k+1)}}{2(1-\\n)} + \\m \\frac{\\m \\m^{-2(k+1)} + (-\\n) \\n^{-2(k+1)}}{2(\\m-\\n)}. {}\\quad{} \\qed\n$$\n\\end{proof}","sha256_utf8":"9a00b24e33b122fb467c6cde7e9e159fac372339b620aba9c6b999daedd110db"}]},{"id":"S100","semantic_type":"lemma","kind":"lemma","title":"多步下曲率线性界","source_units":[{"start":128897,"end":129637,"start_line":1794,"end_line":1805,"raw_tex":"\\begin{lemma}\\label{lemma:general_lemma_ct_h_to_Linear_mu} \nLet $f \\in \\mathcal{F}_{\\mu,L}$, with $\\mu \\in (-\\infty,L)$, and consider $k+1 \\geq 1$ iterations of \\eqref{eq:GM_it_fixed_steps} with stepsize $\\gamma L \\in [\\gbari[k], \\gbari[\\infty])$, satisfying $\\gamma L > 1$, starting from $x_0$. Then the following inequality holds:%\n\\begin{align}\\label{eq:general_ineq_proof_to_linear_mu_compact}\n% \\hspace{-.75em}\n    \\frac{f(x_0) {-} f(x_{k+1})}{\\gamma}\n        {\\geq}\n    \\Ei[k+1][\\m] \\frac{\\|\\nabla f(x_{k+1})\\|^2}{2} {+} \n    \\frac{-\\n^2 \\Tinm[k+1]}{(\\m{-}\\n)^2}\n    \\frac{\\| \\nabla f(x_{k+1}) {-} \\m \\nabla f(x_{k}) \\|^2}{2}.\n\\end{align}%\nFurthermore, if $\\gamma L \\in [\\gbari[k], \\gbari[k+1] \\big)$, then $\\Tinm[k+1]<0$.\n\\end{lemma}","sha256_utf8":"d0a796d2cb34eda0beaf1c1b1c987308a869ce0913d4683abc66f4ca96f3da32"}]},{"id":"S101","semantic_type":"proof","kind":"proof","title":"多步下曲率界完整证明","source_units":[{"start":129643,"end":131680,"start_line":1808,"end_line":1842,"raw_tex":"\\begin{proof}\nCase $k=0$ results from \\cref{lemma:dist_1_mu_1_step_decrease}; further on, $k\\geq 1$. Inequality \\eqref{eq:direct_sum_interp_cond} written for the pair $(i,j)=(k,k+1)$ reads:%\n   \\begin{align}\\label{eq:dist_1_correction_to_mu}\n       0\n        {}\\geq{}\n       \\frac{\\m \\n}{\\m-\\n} \\|g_{k}\\|^2 + \\frac{1}{\\m-\\n} \\|g_{k+1}\\|^2 - \\frac{\\m+\\n}{\\m-\\n} \\langle g_{k} \\,,\\, g_{k+1} \\rangle.\n   \\end{align}%\n   \\begin{enumerate}[align=right,itemindent=1em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]\n       \\item \\textbf{Case $\\gamma L \\in [\\gbari[k], \\gbari[k+1])$.} By using $\\m>0$, \\eqref{eq:dist_1_correction_to_mu} equivalently rewrites as:%\n   $$\n       0\n        {}\\geq{}\n       -\\m^2 \\frac{\\|g_{k}\\|^2}{2} + \\frac{\\|g_{k+1}\\|^2}{2} + \\frac{\\m+\\n}{\\m-\\n} \\frac{\\|g_{k+1}-\\m g_{k}\\|^2}{2}.\n   $$%\n   By multiplying it with $\\frac{-\\n^2}{\\m^2-\\n^2}\\Tinm[k+1]$ (positive because $\\Tinm[k+1] < 0$) and adding it to \\eqref{eq:general_ineq_proof}, the coefficient of $\\|g_{k}\\|^2$ is canceled and we get:%\n    $$\n    \\frac{f_0{-}f_{k+1}}{\\gamma}\n        {}{\\geq}{}\n    \\frac{\\Tinm[k]{+}(\\m{-}\\n){-}\\n^2 \\Tinm[k+1]}{\\m^2{-}\\n^2} \\frac{\\|g_{k+1}\\|^2}{2} {}+{}\n       \\frac{{-}\\n^2 \\Tinm[k+1]}{(\\m{-}\\n)^2}  \\frac{\\|g_{k+1}{-}\\m g_{k}\\|^2}{2},\n    $$\n    which after simplifications is exactly \\eqref{eq:general_ineq_proof_to_linear_mu_compact}.%\n    %\n    \\item \\textbf{Case $\\gamma L \\in \\big[\\gbari[k+1],\\gbari[\\infty] \\big)$.} After multiplying \\eqref{eq:dist_1_correction_to_mu} with $\\frac{-\\n \\m}{\\m-\\n} > 0$:\n    $$\n        0\n            {}\\geq{}\n        \\frac{\\|g_{k+1}\\|^2}{2} + \n        \\frac{-\\m^2}{(\\m-\\n)^2} \\frac{\\|g_{k+1}-\\n g_{k}\\|^2}{2} +\n        \\frac{-\\n^2}{(\\m-\\n)^2} \\frac{\\|g_{k+1}-\\m g_{k}\\|^2}{2}.\n    $$\n    By scaling it with $\\Tinm[k+1]>0$ and summing it with \\eqref{eq:general_ineq_proof_to_linear_compact}, the coefficient of $\\|g_{k+1} - \\n g_{k}\\|^2$ is canceled and after simplifications we get \\eqref{eq:general_ineq_proof_to_linear_mu_compact}.%\n    {}\\qed{}%\n    \\end{enumerate}%\n\\end{proof}","sha256_utf8":"90749ca63c2b3c07f014d543621303911947adc199444c17bf22ea93710d28ad"}]},{"id":"S102","semantic_type":"proof_step","kind":"proof_step","title":"基例及双向和修正式","source_units":[{"start":129643,"end":130057,"start_line":1808,"end_line":1814,"raw_tex":"\\begin{proof}\nCase $k=0$ results from \\cref{lemma:dist_1_mu_1_step_decrease}; further on, $k\\geq 1$. Inequality \\eqref{eq:direct_sum_interp_cond} written for the pair $(i,j)=(k,k+1)$ reads:%\n   \\begin{align}\\label{eq:dist_1_correction_to_mu}\n       0\n        {}\\geq{}\n       \\frac{\\m \\n}{\\m-\\n} \\|g_{k}\\|^2 + \\frac{1}{\\m-\\n} \\|g_{k+1}\\|^2 - \\frac{\\m+\\n}{\\m-\\n} \\langle g_{k} \\,,\\, g_{k+1} \\rangle.\n   \\end{align}%\n","sha256_utf8":"fded50b9c90bfb852dc274bba884ff21deb37234c1005f60b914b18769b3a961"}]},{"id":"S103","semantic_type":"proof_step","kind":"proof_step","title":"阈值区间内重写修正项","source_units":[{"start":130057,"end":130458,"start_line":1815,"end_line":1821,"raw_tex":"   \\begin{enumerate}[align=right,itemindent=1em,labelsep=2pt,labelwidth=1em,leftmargin=0pt,nosep]\n       \\item \\textbf{Case $\\gamma L \\in [\\gbari[k], \\gbari[k+1])$.} By using $\\m>0$, \\eqref{eq:dist_1_correction_to_mu} equivalently rewrites as:%\n   $$\n       0\n        {}\\geq{}\n       -\\m^2 \\frac{\\|g_{k}\\|^2}{2} + \\frac{\\|g_{k+1}\\|^2}{2} + \\frac{\\m+\\n}{\\m-\\n} \\frac{\\|g_{k+1}-\\m g_{k}\\|^2}{2}.\n   $$%\n","sha256_utf8":"fd99ad34b9cca886a0d1681122dccccec313f24687fd79ee52a4c05b186182f5"}]},{"id":"S104","semantic_type":"proof_step","kind":"proof_step","title":"消去前一梯度范数","source_units":[{"start":130458,"end":131000,"start_line":1822,"end_line":1829,"raw_tex":"   By multiplying it with $\\frac{-\\n^2}{\\m^2-\\n^2}\\Tinm[k+1]$ (positive because $\\Tinm[k+1] < 0$) and adding it to \\eqref{eq:general_ineq_proof}, the coefficient of $\\|g_{k}\\|^2$ is canceled and we get:%\n    $$\n    \\frac{f_0{-}f_{k+1}}{\\gamma}\n        {}{\\geq}{}\n    \\frac{\\Tinm[k]{+}(\\m{-}\\n){-}\\n^2 \\Tinm[k+1]}{\\m^2{-}\\n^2} \\frac{\\|g_{k+1}\\|^2}{2} {}+{}\n       \\frac{{-}\\n^2 \\Tinm[k+1]}{(\\m{-}\\n)^2}  \\frac{\\|g_{k+1}{-}\\m g_{k}\\|^2}{2},\n    $$\n    which after simplifications is exactly \\eqref{eq:general_ineq_proof_to_linear_mu_compact}.%\n","sha256_utf8":"226de07c3ac0e681be3b9eb816d9c3e027d449595de5eff203582f19df6524d6"}]},{"id":"S105","semantic_type":"proof_step","kind":"proof_step","title":"阈值以上转换两种余项","source_units":[{"start":131000,"end":131680,"start_line":1830,"end_line":1842,"raw_tex":"    %\n    \\item \\textbf{Case $\\gamma L \\in \\big[\\gbari[k+1],\\gbari[\\infty] \\big)$.} After multiplying \\eqref{eq:dist_1_correction_to_mu} with $\\frac{-\\n \\m}{\\m-\\n} > 0$:\n    $$\n        0\n            {}\\geq{}\n        \\frac{\\|g_{k+1}\\|^2}{2} + \n        \\frac{-\\m^2}{(\\m-\\n)^2} \\frac{\\|g_{k+1}-\\n g_{k}\\|^2}{2} +\n        \\frac{-\\n^2}{(\\m-\\n)^2} \\frac{\\|g_{k+1}-\\m g_{k}\\|^2}{2}.\n    $$\n    By scaling it with $\\Tinm[k+1]>0$ and summing it with \\eqref{eq:general_ineq_proof_to_linear_compact}, the coefficient of $\\|g_{k+1} - \\n g_{k}\\|^2$ is canceled and after simplifications we get \\eqref{eq:general_ineq_proof_to_linear_mu_compact}.%\n    {}\\qed{}%\n    \\end{enumerate}%\n\\end{proof}","sha256_utf8":"35aa9f96eb6067ecd234c84507446d9d0ce0a21da73871dc69e05e22285b930a"}]},{"id":"S106","semantic_type":"external_fact","kind":"external_fact","title":"标准光滑凸插值定理","source_units":[],"declared_external":true,"declaration_text":["For M>0, finite data (x_i,g_i,f_i) are interpolable by a convex M-smooth function if and only if f_i-f_j-<g_j,x_i-x_j> >= ||g_i-g_j||^2/(2M) for every ordered pair; this is the standard convex interpolation result invoked from taylor_smooth_2017, Theorem 4, in B0319."]},{"id":"S107","semantic_type":"external_fact","kind":"external_fact","title":"可微凸函数一阶不等式","source_units":[],"declared_external":true,"declaration_text":["Every differentiable convex function h satisfies h(x)>=h(y)+<∇h(y),x-y> for all x,y."]},{"id":"S108","semantic_type":"external_fact","kind":"external_fact","title":"标准上曲率下降引理","source_units":[],"declared_external":true,"declaration_text":["For a differentiable function with upper curvature L, f(x-γ∇f(x)) <= f(x)-γ(1-γL/2)||∇f(x)||^2. Thus 0<γ<2/L gives descent, and finite f_*=inf f implies f(x)-f_* >= ||∇f(x)||^2/(2L) by choosing γ=1/L. This is the cited standard prerequisite for B0095."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S006","target_id":"S002","kind":"supports"},{"source_id":"S007","target_id":"S002","kind":"supports"},{"source_id":"S008","target_id":"S002","kind":"supports"},{"source_id":"S009","target_id":"S002","kind":"supports"},{"source_id":"S003","target_id":"S008","kind":"supports"},{"source_id":"S004","target_id":"S008","kind":"supports"},{"source_id":"S005","target_id":"S008","kind":"supports"},{"source_id":"S006","target_id":"S008","kind":"supports"},{"source_id":"S007","target_id":"S008","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S037","target_id":"S003","kind":"supports"},{"source_id":"S076","target_id":"S004","kind":"supports"},{"source_id":"S076","target_id":"S005","kind":"supports"},{"source_id":"S100","target_id":"S005","kind":"supports"},{"source_id":"S094","target_id":"S006","kind":"supports"},{"source_id":"S071","target_id":"S007","kind":"supports"},{"source_id":"S016","target_id":"S008","kind":"supports"},{"source_id":"S037","target_id":"S008","kind":"supports"},{"source_id":"S053","target_id":"S009","kind":"supports"},{"source_id":"S054","target_id":"S009","kind":"supports"},{"source_id":"S014","target_id":"S009","kind":"supports"},{"source_id":"S022","target_id":"S004","kind":"supports"},{"source_id":"S022","target_id":"S005","kind":"supports"},{"source_id":"S022","target_id":"S006","kind":"supports"},{"source_id":"S022","target_id":"S007","kind":"supports"},{"source_id":"S015","target_id":"S004","kind":"supports"},{"source_id":"S018","target_id":"S008","kind":"supports"},{"source_id":"S017","target_id":"S003","kind":"supports"},{"source_id":"S011","target_id":"S001","kind":"supports"},{"source_id":"S013","target_id":"S001","kind":"supports"},{"source_id":"S010","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S016","kind":"supports"},{"source_id":"S012","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S018","kind":"supports"},{"source_id":"S016","target_id":"S019","kind":"supports"},{"source_id":"S016","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S029","target_id":"S026","kind":"supports"},{"source_id":"S030","target_id":"S026","kind":"supports"},{"source_id":"S031","target_id":"S026","kind":"supports"},{"source_id":"S032","target_id":"S026","kind":"supports"},{"source_id":"S024","target_id":"S028","kind":"supports"},{"source_id":"S017","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S034","kind":"supports"},{"source_id":"S024","target_id":"S035","kind":"supports"},{"source_id":"S017","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S041","target_id":"S038","kind":"supports"},{"source_id":"S042","target_id":"S038","kind":"supports"},{"source_id":"S043","target_id":"S038","kind":"supports"},{"source_id":"S044","target_id":"S038","kind":"supports"},{"source_id":"S025","target_id":"S040","kind":"supports"},{"source_id":"S033","target_id":"S040","kind":"supports"},{"source_id":"S017","target_id":"S040","kind":"supports"},{"source_id":"S024","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S040","target_id":"S043","kind":"supports"},{"source_id":"S016","target_id":"S041","kind":"supports"},{"source_id":"S016","target_id":"S042","kind":"supports"},{"source_id":"S016","target_id":"S043","kind":"supports"},{"source_id":"S025","target_id":"S044","kind":"supports"},{"source_id":"S033","target_id":"S044","kind":"supports"},{"source_id":"S040","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S046","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S049","target_id":"S046","kind":"supports"},{"source_id":"S011","target_id":"S047","kind":"supports"},{"source_id":"S107","target_id":"S047","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S020","target_id":"S049","kind":"supports"},{"source_id":"S106","target_id":"S049","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S045","target_id":"S052","kind":"supports"},{"source_id":"S045","target_id":"S021","kind":"supports"},{"source_id":"S013","target_id":"S021","kind":"supports"},{"source_id":"S014","target_id":"S020","kind":"supports"},{"source_id":"S021","target_id":"S023","kind":"supports"},{"source_id":"S015","target_id":"S023","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S108","target_id":"S053","kind":"supports"},{"source_id":"S011","target_id":"S053","kind":"supports"},{"source_id":"S013","target_id":"S053","kind":"supports"},{"source_id":"S014","target_id":"S053","kind":"supports"},{"source_id":"S055","target_id":"S054","kind":"supports"},{"source_id":"S056","target_id":"S055","kind":"supports"},{"source_id":"S057","target_id":"S055","kind":"supports"},{"source_id":"S022","target_id":"S056","kind":"supports"},{"source_id":"S022","target_id":"S057","kind":"supports"},{"source_id":"S053","target_id":"S057","kind":"supports"},{"source_id":"S014","target_id":"S057","kind":"supports"},{"source_id":"S059","target_id":"S058","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S061","target_id":"S059","kind":"supports"},{"source_id":"S023","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S063","target_id":"S062","kind":"supports"},{"source_id":"S064","target_id":"S063","kind":"supports"},{"source_id":"S065","target_id":"S063","kind":"supports"},{"source_id":"S066","target_id":"S063","kind":"supports"},{"source_id":"S067","target_id":"S063","kind":"supports"},{"source_id":"S059","target_id":"S064","kind":"supports"},{"source_id":"S060","target_id":"S064","kind":"supports"},{"source_id":"S023","target_id":"S064","kind":"supports"},{"source_id":"S060","target_id":"S065","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S060","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S066","kind":"supports"},{"source_id":"S065","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S017","target_id":"S067","kind":"supports"},{"source_id":"S069","target_id":"S068","kind":"supports"},{"source_id":"S070","target_id":"S069","kind":"supports"},{"source_id":"S017","target_id":"S070","kind":"supports"},{"source_id":"S018","target_id":"S070","kind":"supports"},{"source_id":"S072","target_id":"S071","kind":"supports"},{"source_id":"S073","target_id":"S072","kind":"supports"},{"source_id":"S074","target_id":"S072","kind":"supports"},{"source_id":"S075","target_id":"S072","kind":"supports"},{"source_id":"S016","target_id":"S073","kind":"supports"},{"source_id":"S018","target_id":"S073","kind":"supports"},{"source_id":"S021","target_id":"S074","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S068","target_id":"S075","kind":"supports"},{"source_id":"S018","target_id":"S075","kind":"supports"},{"source_id":"S077","target_id":"S076","kind":"supports"},{"source_id":"S078","target_id":"S077","kind":"supports"},{"source_id":"S079","target_id":"S077","kind":"supports"},{"source_id":"S080","target_id":"S077","kind":"supports"},{"source_id":"S068","target_id":"S078","kind":"supports"},{"source_id":"S016","target_id":"S078","kind":"supports"},{"source_id":"S018","target_id":"S078","kind":"supports"},{"source_id":"S078","target_id":"S079","kind":"supports"},{"source_id":"S021","target_id":"S079","kind":"supports"},{"source_id":"S019","target_id":"S080","kind":"supports"},{"source_id":"S037","target_id":"S080","kind":"supports"},{"source_id":"S016","target_id":"S080","kind":"supports"},{"source_id":"S082","target_id":"S081","kind":"supports"},{"source_id":"S083","target_id":"S082","kind":"supports"},{"source_id":"S084","target_id":"S082","kind":"supports"},{"source_id":"S085","target_id":"S082","kind":"supports"},{"source_id":"S086","target_id":"S082","kind":"supports"},{"source_id":"S087","target_id":"S082","kind":"supports"},{"source_id":"S021","target_id":"S083","kind":"supports"},{"source_id":"S018","target_id":"S083","kind":"supports"},{"source_id":"S016","target_id":"S084","kind":"supports"},{"source_id":"S083","target_id":"S085","kind":"supports"},{"source_id":"S084","target_id":"S085","kind":"supports"},{"source_id":"S085","target_id":"S086","kind":"supports"},{"source_id":"S037","target_id":"S086","kind":"supports"},{"source_id":"S068","target_id":"S086","kind":"supports"},{"source_id":"S084","target_id":"S087","kind":"supports"},{"source_id":"S068","target_id":"S087","kind":"supports"},{"source_id":"S089","target_id":"S088","kind":"supports"},{"source_id":"S090","target_id":"S089","kind":"supports"},{"source_id":"S091","target_id":"S089","kind":"supports"},{"source_id":"S092","target_id":"S089","kind":"supports"},{"source_id":"S093","target_id":"S089","kind":"supports"},{"source_id":"S062","target_id":"S090","kind":"supports"},{"source_id":"S081","target_id":"S090","kind":"supports"},{"source_id":"S021","target_id":"S091","kind":"supports"},{"source_id":"S092","target_id":"S091","kind":"supports"},{"source_id":"S091","target_id":"S090","kind":"supports"},{"source_id":"S016","target_id":"S092","kind":"supports"},{"source_id":"S092","target_id":"S093","kind":"supports"},{"source_id":"S068","target_id":"S093","kind":"supports"},{"source_id":"S037","target_id":"S093","kind":"supports"},{"source_id":"S095","target_id":"S094","kind":"supports"},{"source_id":"S096","target_id":"S095","kind":"supports"},{"source_id":"S097","target_id":"S095","kind":"supports"},{"source_id":"S098","target_id":"S095","kind":"supports"},{"source_id":"S099","target_id":"S095","kind":"supports"},{"source_id":"S071","target_id":"S096","kind":"supports"},{"source_id":"S081","target_id":"S096","kind":"supports"},{"source_id":"S021","target_id":"S097","kind":"supports"},{"source_id":"S098","target_id":"S097","kind":"supports"},{"source_id":"S097","target_id":"S096","kind":"supports"},{"source_id":"S016","target_id":"S098","kind":"supports"},{"source_id":"S098","target_id":"S099","kind":"supports"},{"source_id":"S068","target_id":"S099","kind":"supports"},{"source_id":"S037","target_id":"S099","kind":"supports"},{"source_id":"S101","target_id":"S100","kind":"supports"},{"source_id":"S102","target_id":"S101","kind":"supports"},{"source_id":"S103","target_id":"S101","kind":"supports"},{"source_id":"S104","target_id":"S101","kind":"supports"},{"source_id":"S105","target_id":"S101","kind":"supports"},{"source_id":"S076","target_id":"S102","kind":"supports"},{"source_id":"S050","target_id":"S102","kind":"supports"},{"source_id":"S018","target_id":"S102","kind":"supports"},{"source_id":"S102","target_id":"S103","kind":"supports"},{"source_id":"S068","target_id":"S103","kind":"supports"},{"source_id":"S103","target_id":"S104","kind":"supports"},{"source_id":"S088","target_id":"S104","kind":"supports"},{"source_id":"S037","target_id":"S104","kind":"supports"},{"source_id":"S102","target_id":"S105","kind":"supports"},{"source_id":"S094","target_id":"S105","kind":"supports"},{"source_id":"S037","target_id":"S105","kind":"supports"},{"source_id":"S068","target_id":"S105","kind":"supports"}]}}
{"record_id":"2407.03195v1","paper":{"title":"Incremental Gauss--Newton Methods with Superlinear Convergence Rates","authors":["Zhiling Zhou","Zhuanghua Liu","Chengchang Liu","Luo Luo"],"arxiv_id":"2407.03195","version":"v1","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-026-02990-8","abstract_url":"https://arxiv.org/abs/2407.03195v1","pdf_url":"https://arxiv.org/pdf/2407.03195v1"},"context":{"raw_tex":"We study the problem of solving the system of nonlinear equations\n\\begin{align}\n\\label{eq:ori_prob}\n    \\vf(\\vx) = \\vzero,\n\\end{align}\nwhere the nonlinear vector function $\\vf:\\BR^d\\to\\BR^n$ is Lipschitz continuous and its Jacobian is H\\\"older continuous.\n\n\nWe let $[n]=\\{1,\\dots,n\\}$ and use the notation $t\\%n$ to present the remainder of $t$ divided by $n$.\nWe denote $\\ve_i\\in\\BR^n$ as the $i$-th standard basis vector of the $n$-dimensional Euclidean space for all $i\\in [n]$. \nWe use $\\Norm{\\cdot}$ to represent the Euclidean norm for a given vector and the spectral norm for a given matrix. \nMoreover, we use the notation $\\sigma_{\\min}(\\cdot)$ to represent the smallest singular value for a given matrix.\n\nFor the system of nonlinear equations (\\ref{eq:ori_prob}), we partition the vector function $\\vf:\\BR^d\\to\\BR^n$ at $\\vx\\in\\BR^d$ as~$\\vf(\\vx)=[f_1(\\vx),\\dots, f_n(\\vx)]^\\top\\in\\BR^n$, where $f_i:\\BR^d\\to\\BR$.\nWe also denote the gradient of component $f_i(\\cdot)$ at $\\vx\\in\\BR^d$ as $\\vg_i(\\vx)=\\nabla f_i(\\vx)$, and we organize the corresponding Jacobian as~$\\mJ(\\vx)=[\\vg_1(\\vx), \\cdots, \\vg_n(\\vx)]^\\top\\in\\BR^{n\\times d}$.\n\n\n\\begin{asm}\n\\label{asm:L-f}\n    We suppose the vector function $\\vf:\\BR^d\\to\\BR^n$ is Lipschitz continuous, i.e., there exists constant $L_f>0$ such that\n    \\begin{align}\n        \\Norm{\\vf(\\vx)-\\vf(\\vy)} \\leq L_f\\Norm{\\vx - \\vy}\n    \\end{align}\n    for all $\\vx,\\vy\\in\\BR^d$.\n\\end{asm}\n\n\\begin{asm}\\label{asm:holder-g}\n    We suppose the Jacobian $\\mJ:\\BR^d\\to\\BR^{n\\times d}$  is $\\nu$-H\\\"older continuous for some $\\nu\\in(0,1]$, i.e., there exists constant $\\fH_\\nu>0$ such that\n    \\begin{align}\n        \\Norm{\\mJ(\\vx)-\\mJ(\\vy)} \\leq \\fH_\\nu\\Norm{\\vx - \\vy}^\\nu\n    \\end{align}\n    for all $\\vx,\\vy\\in\\BR^d$.\n\\end{asm}\n\n\\begin{asm}\n\\label{asm:b-J-sing}\n    The system of the nonlinear equations (\\ref{eq:ori_prob}) holds $n\\geq d$ and has a non-degenerate solution $\\vx^*\\in\\BR^d$, i.e., there exists some $\\mu>0$ such that\n    \\begin{align}\n    \\label{eq:b-J-sing}\n    \\mu = \\sigma_{\\min} (\\mJ(\\vx^*))> 0.\n    \\end{align}\n\\end{asm}\n\n\\begin{algorithm}[t]\n\\caption{Incremental Gauss–Newton Method (\\texttt{IGN})}\n\\label{alg:IGN-1}\n\\begin{algorithmic}[1]\n\\State \\textbf{Input:} $\\vx^0\\in\\BR^d$, $\\vu^0\\in\\BR^d$, $\\mH^0,\\mG^0\\in\\BR^{d\\times d}$  \\vskip0.1cm\n\\State \\textbf{for} $t=0,1,\\dots$ \\vskip0.12cm\n\\State \\quad  $\\vx^{t+1}=\\mG^t\\vu^t$ \\vskip0.12cm\n\\State \\quad  $i_t =t\\%n + 1$ \\vskip0.12cm\n\\State \\quad  $\\mU^t = \\big[\n    - \\vg_{i_t}(\\vz_{i_t}^t),~~ \n    \\vg_{i_t}(\\vx^{t+1})\n\\big]$ \\vskip0.12cm\n\\State \\quad\n$\\mV^t = \\big[\\vg_{i_t}(\\vz_{i_t}^t),~~ \n    \\vg_{i_t}(\\vx^{t+1})\\big]$ \\vskip0.12cm\n\\State \\quad $\\vu^{t+1} = \\vu^t - \\left(\\vg_{i_t}(\\vz_{i_t}^t)^\\top\\vz_{i_t}^{t} - f_{i_t}(\\vz_{i_t}^t)\\right)\\vg_{i_t}(\\vz_{i_t}^t) \n + \\left(\\vg_{i_t}(\\vx^{t+1})^\\top\\vx^{t+1} - f_{i_t}(\\vx^{t+1})\\right)\\vg_{i_t}(\\vx^{t+1})$ \\vskip0.12cm\n\\State \\quad $\\mH^{t+1} =\\mH^t - \\vg_{i_t}(\\vz_{i_t}^t)\\vg_{i_t}(\\vz_{i_t}^t)^\\top+ \\vg_{i_t}(\\vx^{t+1})\\vg_{i_t}(\\vx^{t+1})^\\top$ \\vskip0.12cm\n\\State\\label{line:update-G} \\quad  $\\mG^{t+1} =\\mG^{t} - \\mG^{t}\\mU^t(\\mI + (\\mV^t)^\\top\\mG^{t}\\mU^t)^{-1}(\\mV^t)^\\top\\mG^{t}$ \\vskip0.12cm\n\\State \\quad  $\\vz_i^{t+1} = \\begin{cases}\n        \\vx^{t+1}, & \\text{if~} i=i_t \\\\\n        \\vz_i^t, & \\text{otherwise}\n    \\end{cases}$ \\vskip0.12cm\n\\State \\textbf{end for}\n\\end{algorithmic}\n\\end{algorithm}\n\nwhere $\\vz_i^t\\in\\BR^d$ is some point related to component $f_i$ at the $t$-th iteration.\n\n\n\\begin{align}\\label{eq:ign-update-0}\n\\vx^{t+1} \n= \\left(\\sum_{i=1}^n \\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top\\right)^{-1}\\sum_{i=1}^n \\left(\\vg_i(\\vz_i^t)^\\top\\vz_i^t -  f_i(\\vz_i^t)\\right)\\vg_i(\\vz_i^t).\n\\end{align}\n\n\\begin{align}\n\\label{eq:update-1}\n    \\vz_i^{t+1} = \\begin{cases}\n        \\vx^{t+1}, & \\text{if~} i=i_t, \\\\\n        \\vz_i^t, & \\text{otherwise},\n    \\end{cases}   \n\\end{align}\n\n\\begin{align}\\label{eq:note-1}\n\\vu^t = \\sum_{i=1}^n \\left(\\vg_i(\\vz_i^t)^\\top\\vz_i^t -  f_i(\\vz_i^t)\\right)\\vg_i(\\vz_i^t), \\qquad  \n\\mH^t=\\sum_{i=1}^n \\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top\n\\qquad\\text{and}\\qquad \\mG^t = \\left(\\mH^t\\right)^{-1}.\n\\end{align}\n\n\\begin{dfn}\n\\label{def:series}\nWe define the following sequence $\\{a_t(n,\\nu)\\}_{t\\geq0}$ for given $n\\in\\BN^+$ and $\\nu\\in(0, 1]$:\n\\begin{align}\n\\label{eq:series}\na_t(n,\\nu)\\triangleq\n\\begin{cases}\n    1,~~~~&t=0,\\\\[0.15cm]\n    % \\frac{1}{2(1+\\nu)}~~~~&t=1\\\\\n    \\displaystyle{\\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=0}^{t-1}(a_j(n, \\nu))^{1+\\nu} + n - t\\right)},~~~~& 1\\leq t\\leq n,\\\\[0.6cm]\n     \\displaystyle{\\frac{1}{2(1+\\nu)n}\\sum_{j=t-n}^{t-1}(a_j(n, \\nu))^{1+\\nu}},~~~~&t> n.\n\\end{cases}\n\\end{align}\n\\end{dfn}","units":[{"start":11916,"end":12172,"start_line":418,"end_line":423,"raw_tex":"We study the problem of solving the system of nonlinear equations\n\\begin{align}\n\\label{eq:ori_prob}\n    \\vf(\\vx) = \\vzero,\n\\end{align}\nwhere the nonlinear vector function $\\vf:\\BR^d\\to\\BR^n$ is Lipschitz continuous and its Jacobian is H\\\"older continuous.\n","sha256_utf8":"38a92d0e0f14d2ce319e7148f3c6649d1ce9fce3be2798d58462fcf6664a6f4d"},{"start":17270,"end":18153,"start_line":471,"end_line":477,"raw_tex":"We let $[n]=\\{1,\\dots,n\\}$ and use the notation $t\\%n$ to present the remainder of $t$ divided by $n$.\nWe denote $\\ve_i\\in\\BR^n$ as the $i$-th standard basis vector of the $n$-dimensional Euclidean space for all $i\\in [n]$. \nWe use $\\Norm{\\cdot}$ to represent the Euclidean norm for a given vector and the spectral norm for a given matrix. \nMoreover, we use the notation $\\sigma_{\\min}(\\cdot)$ to represent the smallest singular value for a given matrix.\n\nFor the system of nonlinear equations (\\ref{eq:ori_prob}), we partition the vector function $\\vf:\\BR^d\\to\\BR^n$ at $\\vx\\in\\BR^d$ as~$\\vf(\\vx)=[f_1(\\vx),\\dots, f_n(\\vx)]^\\top\\in\\BR^n$, where $f_i:\\BR^d\\to\\BR$.\nWe also denote the gradient of component $f_i(\\cdot)$ at $\\vx\\in\\BR^d$ as $\\vg_i(\\vx)=\\nabla f_i(\\vx)$, and we organize the corresponding Jacobian as~$\\mJ(\\vx)=[\\vg_1(\\vx), \\cdots, \\vg_n(\\vx)]^\\top\\in\\BR^{n\\times d}$.\n","sha256_utf8":"36dc848d8ad0baf54753a12731412452d6328bc1d67006d999e00c9649b5040e"},{"start":18284,"end":18570,"start_line":482,"end_line":489,"raw_tex":"\\begin{asm}\n\\label{asm:L-f}\n    We suppose the vector function $\\vf:\\BR^d\\to\\BR^n$ is Lipschitz continuous, i.e., there exists constant $L_f>0$ such that\n    \\begin{align}\n        \\Norm{\\vf(\\vx)-\\vf(\\vy)} \\leq L_f\\Norm{\\vx - \\vy}\n    \\end{align}\n    for all $\\vx,\\vy\\in\\BR^d$.\n\\end{asm}","sha256_utf8":"b553951b911363ec89a3f82bac5eb97222071f15945df566b6897e21efa0aefe"},{"start":18571,"end":18905,"start_line":490,"end_line":496,"raw_tex":"\\begin{asm}\\label{asm:holder-g}\n    We suppose the Jacobian $\\mJ:\\BR^d\\to\\BR^{n\\times d}$  is $\\nu$-H\\\"older continuous for some $\\nu\\in(0,1]$, i.e., there exists constant $\\fH_\\nu>0$ such that\n    \\begin{align}\n        \\Norm{\\mJ(\\vx)-\\mJ(\\vy)} \\leq \\fH_\\nu\\Norm{\\vx - \\vy}^\\nu\n    \\end{align}\n    for all $\\vx,\\vy\\in\\BR^d$.\n\\end{asm}","sha256_utf8":"110834d58c33b0fa4bc093f9125891ec7fd54ff157c1c5c28ed6e1de35b3e010"},{"start":18906,"end":19218,"start_line":497,"end_line":504,"raw_tex":"\\begin{asm}\n\\label{asm:b-J-sing}\n    The system of the nonlinear equations (\\ref{eq:ori_prob}) holds $n\\geq d$ and has a non-degenerate solution $\\vx^*\\in\\BR^d$, i.e., there exists some $\\mu>0$ such that\n    \\begin{align}\n    \\label{eq:b-J-sing}\n    \\mu = \\sigma_{\\min} (\\mJ(\\vx^*))> 0.\n    \\end{align}\n\\end{asm}","sha256_utf8":"4d26ea3a5662013472ffa95fa892aeb86f96adde97f44be174f1f3973a2b6bb7"},{"start":19616,"end":20913,"start_line":510,"end_line":535,"raw_tex":"\\begin{algorithm}[t]\n\\caption{Incremental Gauss–Newton Method (\\texttt{IGN})}\n\\label{alg:IGN-1}\n\\begin{algorithmic}[1]\n\\State \\textbf{Input:} $\\vx^0\\in\\BR^d$, $\\vu^0\\in\\BR^d$, $\\mH^0,\\mG^0\\in\\BR^{d\\times d}$  \\vskip0.1cm\n\\State \\textbf{for} $t=0,1,\\dots$ \\vskip0.12cm\n\\State \\quad  $\\vx^{t+1}=\\mG^t\\vu^t$ \\vskip0.12cm\n\\State \\quad  $i_t =t\\%n + 1$ \\vskip0.12cm\n\\State \\quad  $\\mU^t = \\big[\n    - \\vg_{i_t}(\\vz_{i_t}^t),~~ \n    \\vg_{i_t}(\\vx^{t+1})\n\\big]$ \\vskip0.12cm\n\\State \\quad\n$\\mV^t = \\big[\\vg_{i_t}(\\vz_{i_t}^t),~~ \n    \\vg_{i_t}(\\vx^{t+1})\\big]$ \\vskip0.12cm\n\\State \\quad $\\vu^{t+1} = \\vu^t - \\left(\\vg_{i_t}(\\vz_{i_t}^t)^\\top\\vz_{i_t}^{t} - f_{i_t}(\\vz_{i_t}^t)\\right)\\vg_{i_t}(\\vz_{i_t}^t) \n + \\left(\\vg_{i_t}(\\vx^{t+1})^\\top\\vx^{t+1} - f_{i_t}(\\vx^{t+1})\\right)\\vg_{i_t}(\\vx^{t+1})$ \\vskip0.12cm\n\\State \\quad $\\mH^{t+1} =\\mH^t - \\vg_{i_t}(\\vz_{i_t}^t)\\vg_{i_t}(\\vz_{i_t}^t)^\\top+ \\vg_{i_t}(\\vx^{t+1})\\vg_{i_t}(\\vx^{t+1})^\\top$ \\vskip0.12cm\n\\State\\label{line:update-G} \\quad  $\\mG^{t+1} =\\mG^{t} - \\mG^{t}\\mU^t(\\mI + (\\mV^t)^\\top\\mG^{t}\\mU^t)^{-1}(\\mV^t)^\\top\\mG^{t}$ \\vskip0.12cm\n\\State \\quad  $\\vz_i^{t+1} = \\begin{cases}\n        \\vx^{t+1}, & \\text{if~} i=i_t \\\\\n        \\vz_i^t, & \\text{otherwise}\n    \\end{cases}$ \\vskip0.12cm\n\\State \\textbf{end for}\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"7527308eaa65b4bd7a0c771f91e09c64a69c6f2eda55e19090bad49fb0d5b4ef"},{"start":21710,"end":21800,"start_line":550,"end_line":550,"raw_tex":"where $\\vz_i^t\\in\\BR^d$ is some point related to component $f_i$ at the $t$-th iteration.\n","sha256_utf8":"d8fabe3f59b58a92088ea9c292f178691d2bbd92a84fb57fc95cb788a408a145"},{"start":22279,"end":22488,"start_line":557,"end_line":560,"raw_tex":"\\begin{align}\\label{eq:ign-update-0}\n\\vx^{t+1} \n= \\left(\\sum_{i=1}^n \\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top\\right)^{-1}\\sum_{i=1}^n \\left(\\vg_i(\\vz_i^t)^\\top\\vz_i^t -  f_i(\\vz_i^t)\\right)\\vg_i(\\vz_i^t).\n\\end{align}","sha256_utf8":"1ac3fd2349daedc4d174cd18a80cb6ba46797fc926f9018851ec62a30ff63427"},{"start":23047,"end":23222,"start_line":566,"end_line":572,"raw_tex":"\\begin{align}\n\\label{eq:update-1}\n    \\vz_i^{t+1} = \\begin{cases}\n        \\vx^{t+1}, & \\text{if~} i=i_t, \\\\\n        \\vz_i^t, & \\text{otherwise},\n    \\end{cases}   \n\\end{align}","sha256_utf8":"023ec74e8ddaec436ab8764703188d0c1198e615f5b297c68003494577ca5f65"},{"start":23438,"end":23690,"start_line":574,"end_line":578,"raw_tex":"\\begin{align}\\label{eq:note-1}\n\\vu^t = \\sum_{i=1}^n \\left(\\vg_i(\\vz_i^t)^\\top\\vz_i^t -  f_i(\\vz_i^t)\\right)\\vg_i(\\vz_i^t), \\qquad  \n\\mH^t=\\sum_{i=1}^n \\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top\n\\qquad\\text{and}\\qquad \\mG^t = \\left(\\mH^t\\right)^{-1}.\n\\end{align}","sha256_utf8":"489d1d2613f7f2c660596e148811cfc8b7d93b059918cb45250d172283e2d7a6"},{"start":56279,"end":56789,"start_line":1178,"end_line":1191,"raw_tex":"\\begin{dfn}\n\\label{def:series}\nWe define the following sequence $\\{a_t(n,\\nu)\\}_{t\\geq0}$ for given $n\\in\\BN^+$ and $\\nu\\in(0, 1]$:\n\\begin{align}\n\\label{eq:series}\na_t(n,\\nu)\\triangleq\n\\begin{cases}\n    1,~~~~&t=0,\\\\[0.15cm]\n    % \\frac{1}{2(1+\\nu)}~~~~&t=1\\\\\n    \\displaystyle{\\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=0}^{t-1}(a_j(n, \\nu))^{1+\\nu} + n - t\\right)},~~~~& 1\\leq t\\leq n,\\\\[0.6cm]\n     \\displaystyle{\\frac{1}{2(1+\\nu)n}\\sum_{j=t-n}^{t-1}(a_j(n, \\nu))^{1+\\nu}},~~~~&t> n.\n\\end{cases}\n\\end{align}\n\\end{dfn}","sha256_utf8":"11191c1488b81b075da98ae73805115328d624a09596590ac27ab23b2e1559dc"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{thm}\n\\label{thm:holder-IGN-1}\n{\n    Under Assumptions \\ref{asm:L-f}, \\ref{asm:holder-g} and \\ref{asm:b-J-sing}, running \\IGN~(Algorithm \\ref{alg:IGN-1}) with initialization $\\vx^0\\in\\BR^d$, $\\mH^0 = \\mJ(\\vx^0)^\\top\\mJ(\\vx^0)$ and $\\mG^0=(\\mH^0)^{-1}$ such that\n    \\begin{align*}\n    \\Norm{\\vx^0 - \\vx^*}\\leq \\left(\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\right)^{1/\\nu},\n    \\end{align*} \n    we have $\\mH^t \\succeq (\\mu^2/2)\\mI$ and $\\sigma_{\\min}(\\mI + (\\mV^t)^\\top\\mG^{t}\\mU^t)>0$ for all $t\\geq 0$.\n    Additionally, there exists sequence $\\{r^t\\}$ such that $\\Norm{\\vx^t-\\vx^*}\\leq r_t$ and it holds\n\n    \\begin{align*}\n     r_{t+1}\\leq  c^{(1+\\nu)^{\\left(\\floor{{t}/{n}}-1\\right)}}r_t \\qquad\\text{with}\\qquad c = 1- \\frac{1}{n}\\left(1-\\left(\\frac{1}{2 \n  (1+\\nu)}\\right)^{(1+\\nu)}\\right)\n    \\end{align*}\n    for all $t\\geq n$.\n}\n\\end{thm}","units":[{"start":28922,"end":29762,"start_line":675,"end_line":691,"raw_tex":"\\begin{thm}\n\\label{thm:holder-IGN-1}\n{\n    Under Assumptions \\ref{asm:L-f}, \\ref{asm:holder-g} and \\ref{asm:b-J-sing}, running \\IGN~(Algorithm \\ref{alg:IGN-1}) with initialization $\\vx^0\\in\\BR^d$, $\\mH^0 = \\mJ(\\vx^0)^\\top\\mJ(\\vx^0)$ and $\\mG^0=(\\mH^0)^{-1}$ such that\n    \\begin{align*}\n    \\Norm{\\vx^0 - \\vx^*}\\leq \\left(\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\right)^{1/\\nu},\n    \\end{align*} \n    we have $\\mH^t \\succeq (\\mu^2/2)\\mI$ and $\\sigma_{\\min}(\\mI + (\\mV^t)^\\top\\mG^{t}\\mU^t)>0$ for all $t\\geq 0$.\n    Additionally, there exists sequence $\\{r^t\\}$ such that $\\Norm{\\vx^t-\\vx^*}\\leq r_t$ and it holds\n\n    \\begin{align*}\n     r_{t+1}\\leq  c^{(1+\\nu)^{\\left(\\floor{{t}/{n}}-1\\right)}}r_t \\qquad\\text{with}\\qquad c = 1- \\frac{1}{n}\\left(1-\\left(\\frac{1}{2 \n  (1+\\nu)}\\right)^{(1+\\nu)}\\right)\n    \\end{align*}\n    for all $t\\geq n$.\n}\n\\end{thm}","sha256_utf8":"d5ad551c19a1068cbb94534d6ffa32d5896df14f3d212c46816a046a7de1da2e"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{prop}\\label{prop:b-JJ-sing}\nUnder Assumption \\ref{asm:b-J-sing}, it holds that\n\\begin{align}\\label{eq::b-JJ-sing}\n    \\sigma_{\\min} (\\mJ(\\vx^*)^\\top \\mJ(\\vx^*)) = \\mu^2 > 0.\n\\end{align}\n\\end{prop}\n\n\\begin{lem}\n\\label{le:L-JJ}\n{\n    Under Assumptions \\ref{asm:L-f} and \\ref{asm:holder-g}, we have \n    \\begin{align*}\n    \\Norm{\\mJ(\\vy)^\\top\\mJ(\\vy) - \\mJ(\\vx)^\\top\\mJ(\\vx)} & \\leq 2L_f\\fH_\\nu\\Norm{\\vy-\\vx}^\\nu\n    \\end{align*}\n    and\n    \\begin{align*}\n    \\Norm{\\vg_i(\\vy)\\vg_i(\\vy)^\\top - \\vg_i(\\vx)\\vg_i(\\vx)^\\top}   \\leq 2L_f\\fH_\\nu\\Norm{\\vy-\\vx}^\\nu\n    \\end{align*}\n    for all $\\vx, \\vy\\in\\BR^n$ and $i \\in [n]$.\n}\n\\end{lem}\n\n\\begin{lem}\n\\label{le:lb-Ht-new-1}\nUnder Assumptions \\ref{asm:L-f}, \\ref{asm:holder-g} and \\ref{asm:b-J-sing}, running \\IGN~(Algorithm \\ref{alg:IGN-1}) with $\\mH^0 = \\mJ(\\vx^0)^\\top\\mJ(\\vx^0)$ and~$\\mG^0=(\\mH^0)^{-1}$ holds that\n\\begin{align*}\n    \\sigma_{\\min}(\\mH^t) \\geq \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^t - \\vx^*}^\\nu\n\\end{align*}\nfor all $t\\geq 0$.\n\\end{lem}\n\n\\begin{lem}\n    \\label{le:H-con}\n    \\textnormal{(H\\\"older continuity of each gradient)} Under Assumption \\ref{asm:holder-g}, it satisfies that\n    \\begin{align}\n        \\Norm{\\vg_i(\\vy)-\\vg_i(\\vx)}\\leq \\fH_\\nu\\Norm{\\vy-\\vx}^\\nu,\n    \\end{align}\n    for any $\\vx, \\vy\\in\\BR^d$, and $i\\in[n]$.\n\\end{lem}\n\n\\begin{proof}\n    We denote\n    \\begin{align*}\n    \\Tilde{\\mJ} = \\begin{bmatrix}\n        (\\vg_1(\\vy)-\\vg_1(\\vx))^\\top \\\\\n        \\vdots \\\\\n        (\\vg_n(\\vy)-\\vg_n(\\vx))^\\top \n    \\end{bmatrix}\\in\\BR^{n\\times d} \\qquad\\text{with}\\qquad \\vg_i(\\vx)=\\nabla f_i(\\vx)\n    \\end{align*}\n    and let $\\ve_i\\in\\BR^{n}$ be the $i$-th standard basic vector in $n$-dimensional Euclidean space. \n    Then the facts~$\\Tilde{\\mJ}=\\mJ(\\vy)-\\mJ(\\vx)$ and $\\Tilde{\\mJ}^\\top \\ve_i=\\vg_i(\\vy)-\\vg_i(\\vx)$ imply we have\n    \\begin{align*}\n    \\Norm{\\vg_i(\\vy)-\\vg_i(\\vx)}\n    \\leq \\|\\Tilde{\\mJ}^\\top\\|\\Norm{\\ve_i} = \n    \\|\\Tilde{\\mJ}\\|=\\Norm{\\mJ(\\vy)-\\mJ(\\vx)}\\leq \\fH_\\nu\\Norm{\\vy-\\vx}^\\nu,\n    \\end{align*}\n    where the last step is based on the H\\\"older continuouity of $\\mJ(\\cdot)$.\n\\end{proof}\n\n\\begin{lem}\n\\label{le:power-bound}\n\\textnormal{(Bound for H\\\"older-continuous function)} Under Assumption \\ref{asm:holder-g}, we have\n    \\begin{align}\n        f_i(\\vy) - f_i(\\vx) - \\vg_i(\\vx)^\\top(\\vy-\\vx) \\leq \\frac{\\fH_\\nu}{1 + \\nu}\\Norm{\\vy-\\vx}^{1+\\nu},\n    \\end{align}\n    for any $\\vx, \\vy\\in\\BR^d$ and  $i\\in[n]$.\n\\end{lem}\n\n\\begin{proof}\n    Following the proof of \\cite{grapiglia2017regularized, grapiglia2019accelerated}, we have\n    \\begin{align*}\n        f_i(\\vy) - f_i(\\vx) - \\vg_i(\\vx)^\\top(\\vy - \\vx)\n        & = \\int_{t=0}^1 \\vg_i(\\vx + t(\\vy - \\vx))^\\top(\\vy - \\vx)\\text{d}t - \\vg_i(\\vx)^\\top(\\vy - \\vx)\\\\\n        & = \\int_{t=0}^1 \\left(\\vg_i(\\vx + t(\\vy - \\vx)) - \\vg_i(\\vx)\\right)^\\top(\\vy - \\vx)\\text{d}t\\\\\n        & \\leq \\int_{t=0}^1 \\Norm{\\vg_i(\\vx + t(\\vy - \\vx)) - \\vg_i(\\vx)}\\Norm{\\vy - \\vx}\\text{d}t\\\\\n        & \\leq \\int_{t=0}^1\\fH_\\nu t^\\nu\\Norm{\\vy - \\vx}^{1+\\nu}\\text{d}t\\\\\n        & = \\fH_\\nu \\Norm{\\vy - \\vx}^{1+\\nu}\\int_{t=0}^1 t^\\nu\\text{d}t\\\\\n        & = \\frac{\\fH_\\nu}{1 + \\nu} \\Norm{\\vy - \\vx}^{1+\\nu},\n    \\end{align*}\n    where the first inequality comes from Cauchy-Schwarz inequality, and the second one comes from Lemma \\ref{le:H-con} that each gradient is H\\\"older continuous.\n\\end{proof}\n\n\\begin{lem}\n\\label{le:b-grad}\n    \\textnormal{(Bound for Jacobian and gradient)} Under Assumption \\ref{asm:L-f}, we have\n    \\begin{align*}\n        \\Norm{\\vg_i(\\vx)}\\leq\\Norm{\\mJ(\\vx)}\\leq L_f\n    \\end{align*}\n    for all $ \\vx\\in\\BR^d$ and $i\\in [n]$.\n\\end{lem}\n\n\\begin{proof}\n\n    For all $\\vx, \\vv\\in\\BR^d$, we have\n    \\begin{align*}\n        \\mJ(\\vx)\\vv =\\lim_{h\\to0}\\frac{\\vf(\\vx+h\\vv)-\\vf(\\vx)}{h}.\n    \\end{align*}\n    Taking the spectral norm on both sides, we have\n    \\begin{align*}\n        \\Norm{\\mJ(\\vx)\\vv}\n        & = \\lim_{h\\to0}\\frac{\\Norm{\\vf(\\vx+h\\vv)-\\vf(\\vx)}}{\\vert h\\vert}\\\\\n        & \\leq \\lim_{h\\to0}\\frac{L_f\\Norm{\\vx+h\\vv-\\vx}}{\\vert h\\vert}\\\\\n        & = \\lim_{h\\to0}\\frac{L_f\\vert h\\vert\\Norm{\\vv}}{\\vert h\\vert}\\\\\n        & = L_f\\Norm{\\vv},\n    \\end{align*}\n    where the inequality comes from Assumption \\ref{asm:L-f}. \n    \n    Therefore, for all $\\vx\\in\\BR^d$ it holds\n    \\begin{align*}\n    \\Norm{\\mJ(\\vx)}=\\sup_{\\vv\\in\\BR^d}\\frac{\\Norm{\\mJ(\\vx)\\vv}}{\\Norm{\\vv}}\\leq L_f.\n    \\end{align*}\n    Let $\\ve_i\\in\\BR^{n}$ be the $i$-th standard basic vector in $n$-dimensional Euclid space, then we have \n\n    \\begin{align*}\n    \\Norm{\\vg_i(\\vx)} = \\Norm{\\mJ^\\top \\ve_i} \\leq \\Norm{\\mJ}\\Norm{\\ve_i}=\\Norm{\\mJ(\\vx)}\\leq L_f\n    \\end{align*}\n    for all $i\\in[n]$.\n\\end{proof}\n\n\\begin{lem}\n\\label{le:s-mono-000} The sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$ satisfies \n\\begin{align*}\n    a_t(n, \\nu) \\leq 1 \n\\end{align*}\nfor all $t\\geq 0$. \n\\end{lem}\n\n\\begin{proof}\n\\textbf{Part I:}\nWe first use induction to prove $a_t(n, \\nu) \\leq 1$ for all $t=0,1\\dots,n$.\nFor the induction base, we can verify that $a_0(n, \\nu) = 1 \\leq 1$.  For the induction step, we assume \n\\begin{align*}\na_j(n,\\nu) \\leq 1    \n\\end{align*}\nholds for all $j=1,\\dots,t-1$ such that $t\\leq n$.\nThen we have\n\\begin{align*}\n        a_t(n,\\nu) = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=0}^{t-1}(a_j(n,\\nu))^{1+\\nu} + n - t\\right) \\leq \\frac{1}{2(1+\\nu)n}\\left(t + n - t\\right) = \\frac{1}{2(1+\\nu)}\\leq 1,\n\\end{align*}\nwhere the first inequality is based on the induction hypothesis and the last inequality is based on the setting $\\nu\\in(0,1]$. This finishes the induction. \n\n\\textbf{Part II:} We then use induction to prove $a_t(n, \\nu) \\leq 1$ for all $t\\geq n+1$. For the induction base, we can verify that\n\\begin{align*}\n    a_{n+1}(n,\\nu) = \\frac{1}{2(1+\\nu)n}\\sum_{j=1}^{n}(a_j(n, \\nu))^{1+\\nu} \\leq \\frac{1}{2(1+\\nu)n}\\cdot n = \\frac{1}{2(1+\\nu)} \\leq 1,\n\\end{align*}\nwhere the first inequality is based on $a_t(n, \\nu) \\leq 1$ for all $t\\leq n$ (which have shown in Part I), and the last inequality is based on the setting $\\nu \\in (0, 1]$. For the induction step, we assume\n\\begin{align*}\n    a_{n+1}(n,\\nu) \\leq 1\n\\end{align*}\nholds for all $j=n+2,\\dots,t-1$ such that $t\\geq n+3$. Then we have\n\\begin{align*}\n    a_{t}(n,\\nu) = \\frac{1}{2(1+\\nu)n}\\sum_{j=t-n}^{t-1}(a_j(n, \\nu))^{1+\\nu} \\leq \\frac{1}{2(1+\\nu)n}\\cdot n = \\frac{1}{2(1+\\nu)} \\leq 1,\n\\end{align*}\n\nwhere the first inequality is based on the induction hypothesis and the last inequality is based on the setting $\\nu \\in (0, 1]$. This finishes the induction.\n\nCombining the results of above two parts, we finish the proof of this lemma.\n\\end{proof}\n\n\\begin{lem}\n\\label{le:s-mono}The sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$ satisfies \n\\begin{align*}\n    a_{t}(n, \\nu) \\geq a_{t+1}(n, \\nu) \n\\end{align*}\nfor all $t\\geq 0$. \n\n\\end{lem}\n\n\\begin{proof}\n\\textbf{Part I:} For $t=0$, the fact $\\nu\\in(0, 1]$ means\n\\begin{align*}\n    a_1(n, \\nu) = \\frac{1}{2(1+\\nu)} \\leq 1 = a_0(n, \\nu).\n\\end{align*}\n\\textbf{Part II:}\nFor all $t=1,\\dots,n-1$, we have\n\\begin{align*}\n        & a_{t+1}(n, \\nu) - a_{t}(n, \\nu) \\\\\n        & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=0}^{t}(a_j(n, \\nu))^{1+\\nu} + n - t - 1\\right) - \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=0}^{t-1}(a_j(n, \\nu))^{1+\\nu} + n - t\\right)\\\\\n        & = \\frac{1}{2(1+\\nu)n}\\left((a_{t}(n, \\nu))^{1+\\nu} - 1\\right)  \\leq 0,\n\\end{align*}     \nwhere the last inequality is based on Lemma \\ref{le:s-mono-000}. This indicates $a_{t+1}\\leq a_{t}$ for $t=1,\\dots,n-1$. \n\n\\textbf{Part III:} For all $t\\geq n$, we use induction to prove $a_{t+1}(n, \\nu)\\leq a_{t}(n, \\nu)$. \nFor the induction base, we can verify that\n    \\begin{align*}\n        & a_{n+1}(n, \\nu) - a_n(n, \\nu) \\\\\n        & = \\frac{1}{2(1+\\nu)n}\\sum_{j=1}^{n}(a_j(n, \\nu))^{1+\\nu} - \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=1}^{n-1}(a_j(n, \\nu))^{1+\\nu} + 1\\right)\\\\\n        & = \\frac{1}{2(1+\\nu)n}\\left((a_n(n, \\nu))^{1+\\nu} - 1\\right)\\\\\n        & \\leq 0,\n    \\end{align*}\n    where the last inequality is based on Lemma \\ref{le:s-mono-000}. \n    \n    For the induction step, we assume\n\\begin{align*}\n    a_{j+1}(n, \\nu) \\leq a_{j}(n, \\nu)\n\\end{align*}\nholds for all $j = n+1, \\cdots, t-1$ such that $t\\geq n+2$. Then we have\n    \\begin{align*}\n        & a_{t+1}(n, \\nu) - a_{t}(n, \\nu) \\\\\n         =&  \\frac{1}{2(1+\\nu)n}\\sum_{j=t-n + 1}^{t}(a_j(n, \\nu)) ^{1+\\nu} - \\frac{1}{2(1+\\nu)n}\\sum_{j=t-n}^{t-1}(a_j(n, \\nu))^{1+\\nu} \\leq 0,\n    \\end{align*}\nwhere the inequality is based on the induction hypothesis and the fact $a_{t+1}(n, \\nu) \\leq a_{t}(n, \\nu)$ for all~$t \\leq n - 1$ (which have shown in Part I).\n\nCombining the results of above three parts, we finish the proof of this lemma.\n\\end{proof}\n\n\\begin{lem}\n\\label{le:n-qua-con}\n     For the sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$, we have\n    \\begin{align*}\n    a_t(n, \\nu) \\leq \\frac{1}{2(1+\\nu)}(a_{t-n}(n, \\nu))^{1+\\nu}\n    \\end{align*}\n    for all $t\\geq n$.\n\\end{lem}\n\n\\begin{proof}\n    For all $t\\geq n$, the definition of $a_t(n, \\nu)$ implies\n    \\begin{align*}\n    & a_t(n, \\nu) = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=t-n}^{t-1}(a_j(n, \\nu))^{1+\\nu}\\right)  \\\\\n    & \\leq \\frac{1}{2(1+\\nu)} \\max\\{(a_{t-n}(n, \\nu))^{1+\\nu},\\cdots, (a_{t-1}(n, \\nu))^{1+\\nu}\\}.\n    \\end{align*}\n    Additionally, Lemma \\ref{le:s-mono} implies for all $t\\geq n$, we have\n    \\begin{align*}\n    \\max\\{(a_{t-n}(n, \\nu))^{1+\\nu},\\cdots, (a_{t-1}(n, \\nu))^{1+\\nu}\\} = (a_{t-n}(n, \\nu))^{1+\\nu}, \\quad t\\geq n.\n    \\end{align*}\n    Combining above results, we achieve\n    \\begin{align*}\n    a_t(n, \\nu) \\leq \\frac{1}{2(1+\\nu)}(a_{t-n}(n, \\nu))^{1+\\nu}\n    \\end{align*}\n    for all $t\\geq n$.\n\\end{proof}\n\n\\begin{lem}\n\\label{le:linear-con} For the sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$, we have \n\\begin{align*}\na_{t+1}(n, \\nu)\\leq c_0 a_t(n, \\nu)\n\\end{align*}\nfor all $t\\geq n$, where\n\\begin{align*}\n    c_0 = 1- \\frac{1}{n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right).\n\\end{align*}\n\\end{lem}\n\n\\begin{proof}\n    For all $t\\geq n$, we have\n    \\begin{align}\\label{eq:lemma9-ieq}\n    a_t(n, \\nu) \\leq \\frac{1}{2(1+\\nu)}(a_{t-n}(n, \\nu))^{1+\\nu} \\leq \\frac{1}{2(1+\\nu)}a_{t-n}(n, \\nu), \n    \\end{align}\n    where the first inequality is based on Lemma \\ref{le:n-qua-con} and the second one is based on Lemma \\ref{le:s-mono-000}.\n    \n    Then we also have \n    \\begin{align*}\\small\\begin{split}\n        a_{t+1} \n        & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=t-n+1}^{t}(a_j(n, \\nu))^{1+\\nu}\\right)\\\\\n        & \\leq \\frac{1}{2(1+\\nu)n}\\left(\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}(a_{t-n}(n, \\nu))^{1+\\nu} + \\sum_{j=t-n+1}^{t-1}(a_j(n, \\nu))^{1+\\nu}\\right)\\\\\n        & = \\frac{1}{2(1+\\nu)n}\\left(\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}(a_{t-n}(n, \\nu))^{1+\\nu} + \\sum_{j=t-n+1}^{t-1}(a_j(n, \\nu))^{1+\\nu} + (a_{t-n}(n, \\nu))^{1+\\nu} - (a_{t-n}(n, \\nu))^{1+\\nu} \\right)\\\\\n        & = a_{t}(n, \\nu) + \\frac{1}{2(1+\\nu)n}\\left(\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}(a_{t-n}(n, \\nu))^{1+\\nu} - (a_{t-n}(n, \\nu))^{1+\\nu} \\right)\\\\\n        & = a_{t}(n, \\nu) - \\frac{1}{2(1+\\nu)n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right)(a_{t-n}(n, \\nu))^{1+\\nu}\\\\\n        & \\leq a_{t}(n, \\nu) - \\frac{1}{n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right)a_t(n, \\nu)\\\\\n        & = \\left(1- \\frac{1}{n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right)\\right)a_t(n, \\nu),\n     \\end{split}\n    \\end{align*}\n    for all $t\\geq n$, where the first inequality is based on equation (\\ref{eq:lemma9-ieq}) and the last inequality is based on Lemma \\ref{le:n-qua-con}. This finish the proof.\n\\end{proof}\n\n\\begin{lem}\n\\label{le:imp-c} For the sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$, if there exists $c_1\\in(0,1)$ and $t_0\\geq 0$ such that\n\\begin{align}\\label{eq:ieq-t0n2n}\na_{t+1}(n, \\nu) \\leq c_1 a_t(n, \\nu)\n\\end{align}\nfor all $t\\geq t_0 + n$, \nthen we have\n\\begin{align*}\na_{t+1}(n, \\nu) \\leq c_1^{1+\\nu} a_t(n, \\nu)\n\\end{align*}\nfor all $t\\geq t_0 + 2n$.\n\\end{lem}\n\n\\begin{proof}\n    For all  $t\\geq t_0 + 2n$, we have\n    \\begin{align*}\n        & a_{t+1} (n, \\nu)\n         = \\frac{1}{2(1+\\nu)n}\\sum_{j=t-n+1}^{t}(a_j(n, \\nu))^{1+\\nu} \\\\\n        & \\leq \\frac{1}{2(1+\\nu)n}\\sum_{j=t-n}^{t-1}c_1^{1+\\nu}(a_j(n, \\nu))^{1+\\nu} = c_1^{1+\\nu} a_t(n, \\nu),\n    \\end{align*}\n    where the inequality is based on equation (\\ref{eq:ieq-t0n2n}).\n\\end{proof}\n\n\\begin{lem}\n\\label{le:super-con} For the sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$,\nwe have the superlinear convergence\n\\begin{align*}\na_{t+1}(n, \\nu) \\leq c^{(1+\\nu)^{\\left(\\floor{\\frac{t-1}{n}}-1\\right)}}a_t(n, \\nu) \n\\end{align*}\nfor all $t\\geq n$, where\n\\begin{align*}\n    c = 1- \\frac{1}{n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right).\n\\end{align*}\n\\begin{proof}\n    According to Lemma \\ref{le:linear-con}, we have\n    \\begin{align*}\n        a_{t+1}(n, \\nu)\\leq c a_t(n, \\nu) \n        \\quad \\text{for all}~~t \\geq n.\n    \\end{align*}\n    According to Lemma \\ref{le:imp-c}, we have\n    \\begin{align*}\n        a_{t+1}(n, \\nu) & \\leq c^{1+\\nu} a_t(n, \\nu) \\quad \\text{for all}~~t \\geq 2n,\\\\\n        a_{t+1}(n, \\nu) & \\leq c^{(1+\\nu)^2} a_t(n, \\nu) \\quad \\text{for all}~~t \\geq 3n,\\\\\n        a_{t+1}(n, \\nu) & \\leq c^{(1+\\nu)^3} a_t(n, \\nu) \\quad \\text{for all}~~t \\geq 4n,\\\\\n        \\cdots &\n    \\end{align*}\n    which implies\n    \\begin{align*}\n        a_{t+1}(n, \\nu) \\leq c^{(1+\\nu)^{\\left(\\lfloor {t}/{n}  \\rfloor-1\\right)}}a_t(n, \\nu)\n    \\end{align*}\n    for all $t \\geq n$.\n\n    The superlinear convergence of the sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$ can be verify by the fact\n    \\begin{align*}\n        \\lim_{t\\to\\infty}c^{(1+\\nu)^{\\left(\\lfloor {t}/{n}  \\rfloor-1\\right)}}=0.\n    \\end{align*}\n    Hence, we finish the proof.\n\\end{proof}\n\\end{lem}\n\n\\begin{proof}\nWe denote the singular value decomposition of $\\mJ(\\vx^*)$ as\n    \\begin{align*}\n        \\mJ(\\vx^*) = \\mP\\mD\\mQ^\\top,\n    \\end{align*}\n    where $\\mP\\in\\BR^{n\\times d}, \\mQ\\in\\BR^{d\\times d}$ are (column) orthogonal matrices and $\\mD\\in\\BR^{d\\times d}$ is diagonal matrix with the smallest diagonal entry of $\\mu>0$.\n    Therefore, we have\n    \\begin{align*}\n        \\mJ(\\vx^*)^\\top \\mJ(\\vx^*) = \\mQ\\mD^2\\mQ^\\top,\n    \\end{align*}\n    which means the smallest singular value of $\\mJ(\\vx^*)^\\top \\mJ(\\vx^*)$ is equal to the smallest value of $\\mD^2$, which is $\\mu^2$. Therefore, we have\n    \\begin{align*}\n        \\sigma_{\\min}(\\mJ(\\vx^*)^\\top\\mJ(\\vx^*)) \\geq \\mu^2.\n    \\end{align*}\n\\end{proof}\n\n\\begin{proof}\n    The Jacobian holds that\n    \\begin{align*}\n        \\Norm{\\mJ(\\vy)^\\top\\mJ(\\vy) - \\mJ(\\vx)^\\top\\mJ(\\vx)}\n        & = \\Norm{\\mJ(\\vy)^\\top\\mJ(\\vy) - \\mJ(\\vx)^\\top\\mJ(\\vy) + \\mJ(\\vx)^\\top\\mJ(\\vy) - \\mJ(\\vx)^\\top\\mJ(\\vx)}\\\\\n        & \\leq \\Norm{\\left(\\mJ(\\vy)-\\mJ(\\vx)\\right)^\\top\\mJ(\\vy)} + \\Norm{\\mJ(\\vx)^\\top\\left(\\mJ(\\vy) - \\mJ(\\vx)\\right)}\\\\\n        & \\leq \\Norm{\\mJ(\\vy)}\\Norm{\\mJ(\\vy)-\\mJ(\\vx)} + \\Norm{\\mJ(\\vx)}\\Norm{\\mJ(\\vy)-\\mJ(\\vx)}\\\\\n        & \\leq 2L_f\\fH_\\nu\\Norm{\\vy-\\vx}^\\nu,\n    \\end{align*}\n    where the first inequality comes from triangular inequality, the second inequality comes from property of norm, and the last inequality is based on Lemma \\ref{le:b-grad} and Assumption \\ref{asm:holder-g}. \n    \n    For all $j\\in[n]$, the gradient holds that\n    \\begin{align*}\n        \\Norm{\\vg_i(\\vy)\\vg_i(\\vy)^\\top\\!- \\vg_i(\\vx)\\vg_i(\\vx)^\\top}\n        & = \\Norm{\\vg_i(\\vy)\\vg_i(\\vy)^\\top\\!- \\vg_i(\\vx)\\vg_i(\\vy)^\\top + \\vg_i(\\vx)\\vg_i(\\vy)^\\top\\!- \\vg_i(\\vx)\\vg_i(\\vx)^\\top}\\\\\n        & \\leq \\Norm{\\left(\\vg_i(\\vy)-\\vg_i(\\vx)\\right)\\vg_i(\\vy)^\\top} + \\Norm{\\vg_i(\\vx)\\left(\\vg_i(\\vy)-\\vg_i(\\vx)\\right)^\\top}\\\\\n        & \\leq \\Norm{\\vg_i(\\vy)}\\Norm{\\vg_i(\\vy)-\\vg_i(\\vx)} + \\Norm{\\vg_i(\\vx)}\\Norm{\\vg_i(\\vy)-\\vg_i(\\vx)}\\\\\n        & \\leq 2L_f\\fH_\\nu\\Norm{\\vy-\\vx}^\\nu,\n    \\end{align*}\n    where the first inequality comes from triangular inequality, the second inequality comes from property of norm, and the last inequality is based on Lemma \\ref{le:H-con} and \\ref{le:b-grad}.\n\\end{proof}\n\n\\begin{proof}\n    We have\n    \\begin{align*}\n        \\Norm{\\mH^t - \\mJ(\\vx^*)^\\top\\mJ(\\vx^*)} &= \\Norm{\\sum_{i=1}^n\\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top - \\sum_{i=1}^n\\vg_i(\\vx^*)\\vg_i(\\vx^*)^\\top}\\\\\n        & \\leq \\sum_{i=1}^n\\Norm{\\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top - \\vg_i(\\vx^*)\\vg_i(\\vx^*)^\\top}\\\\\n        & \\leq \\sum_{i=1}^n 2L_f\\fH_\\nu\\Norm{\\vz_i^t - \\vx^*}^\\nu,\n    \\end{align*}\n    where the first inequality comes from the triangle inequality and the second inequality is based on Lemma \\ref{le:L-JJ}. \n    Thus, we have\n     \\begin{align*}\n         \\mH^{t}-\\mJ(\\vx^*)^{\\top}\\mJ(\\vx^*)\\succeq - \\sum_{i=1}^n 2L_f\\fH_\\nu\\Norm{\\vz_i^t - \\vx^*}^\\nu \\cdot\\mI, \n     \\end{align*}\n    which implies that\n    \\begin{align*}\n        \\sigma_{\\min}(\\mH^t) \\geq \\sigma_{\\min}(\\mJ(\\vx^*)^\\top\\mJ(\\vx^*)) - \\sum_{i=1}^n 2L_f\\fH_\\nu\\Norm{\\vz_i^t - \\vx^*}^\\nu = \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^t - \\vx^*}^\\nu,\n    \\end{align*}\n    where the last step is based on Proposition \\ref{prop:b-JJ-sing}.\n\\end{proof}\n\nWe first show the update\n\\begin{align*}\n\\mG^{t+1} =\\mG^{t} - \\mG^{t}\\mU^t(\\mI + (\\mV^t)^\\top\\mG^{t}\\mU^t)^{-1}(\\mV^t)^\\top\\mG^{t}    \n\\end{align*}\nin \\IGN~method (Line \\ref{line:update-G} of Algorithm \\ref{alg:IGN-1}) is well-defined if the matrices $\\mH^t$ and $\\mH^{t+1}$ are non-singular.\n\n\n\\begin{lem}\n\\label{le:via-update}\n    Following the setting of Theorem \\ref{thm:holder-IGN-1}, if the matrices $\\mH^t$ and $\\mH^{t+1}$ are non-singular, then the matrix $\\mI + {\\mV^t}^\\top\\mG^{t}\\mU^t$ is also non-singular, where\n    \\begin{align*}\n    \\mU^t = \\begin{bmatrix}\n        - \\vg_{i_t}(\\vz_{i_t}^t) \\!& \n        \\vg_{i_t}(\\vx^{t+1})\n    \\end{bmatrix}\\in\\BR^{d\\times 2},~~\n    \\mV^t = \\begin{bmatrix}\n        \\vg_{i_t}(\\vz_{i_t}^t) \\!& \n        \\vg_{i_t}(\\vx^{t+1})\n    \\end{bmatrix}\\in\\BR^{d\\times 2}~~\\text{and}~~i_t ={t\\%n} + 1.\n    \\end{align*}\n\\end{lem}\n\n\\begin{proof}\n    The recursion of $\\mH^t$ and the definition of $\\mU^t$ and $\\mV^t$ imply\n    \\begin{align*}\n        \\mH^{t+1} = \\mH^t - \\vg_{i_t}(\\vz_{i_t}^t)\\vg_{i_t}(\\vz_{i_t}^t)^\\top + \\vg_{i_t}(\\vx^{t+1})\\vg_{i_t}(\\vx^{t+1})^\\top = \\mH^t + \\mU^t{\\mV^t}^\\top.\n    \\end{align*}\n    Since we assume matrices $\\mH^t$ and $\\mH^{t+1}$ are non-singular, applying the matrix determinant lemma \\cite[Section 9.1.2]{petersen2008matrix} on above equation leads to\n    \\begin{align*}\n        \\det(\\mH^{t+1})=\\det(\\mH^t + \\mU^t{\\mV^t}^\\top) = \\det(\\mI + {\\mV^t}^\\top(\\mH^t)^{-1}\\mU^t)\\det(\\mH^t).\n    \\end{align*}\n    Then the definition $\\mG^t={\\mH^t}^{-1}$ implies\n    \\begin{align*}\n        \\det(\\mI + {\\mV^t}^\\top\\mG^t\\mU^t) = \\det(\\mI + {\\mV^t}^\\top{\\mH^t}^{-1}\\mU^t) \\neq 0\n    \\end{align*}\n    which finish the proofs.\n\\end{proof}\n\nThen we show the non-singular assumption on $\\{\\mH^j\\}_{j=0}^t$ can upper bound the distance $\\Norm{\\vx^{t+1} - \\vx^*}$.\n\n\n\\begin{lem}\\label{le:main-iter-1} \nUnder Assumptions \\ref{asm:L-f} and \\ref{asm:holder-g}, we\nassume matrices $\\{\\mH^j\\}_{j=0}^t$ are non-singular and run \\IGN~(Algorithm \\ref{alg:IGN-1}), then it holds\n\\begin{align*}\n    \\Norm{\\vx^{t+1} - \\vx^*}\n     \\leq \\frac{L_f\\fH_\\nu}{1 \n     +\\nu}\\Norm{\\mG^t}\\sum_{i=1}^n\\Norm{\\vz_i^t-\\vx^*}^{1+\\nu},\n    \\end{align*}\n    where $\\mG^t=\\left(\\mH^t\\right)^{-1}$.\n\\end{lem}\n\n\\begin{proof}\n    Subtracting the term $\\vx^*$ on both sides of equation (\\ref{eq:ign-update-0}), we have\n    \\begin{align*}\n    \\vx^{t+1} - \\vx^*\n    & = \\mG^t\\left(\\sum_{i=1}^n \\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top(\\vz_i^t-\\vx^*) - \\sum_{i=1}^n f_i(\\vz_i^t)\\vg_i(\\vz_i^t)\\right)\\\\\n    & = \\mG^t\\left(\\sum_{i=1}^n \\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top(\\vz_i^t-\\vx^*) - \\sum_{i=1}^n f_i(\\vz_i^t)\\vg_i(\\vz_i^t) + \\sum_{i=1}^n f_i(\\vx^*)\\vg_i(\\vz_i^t)\\right)\\\\\n    & = \\mG^t\\sum_{i=1}^n\\left( \\vg_i(\\vz_i^t)^\\top(\\vz_i^t-\\vx^*) - f_i(\\vz_i^t) + f_i(\\vx^*)\\right)\\vg_i(\\vz_i^t).\n    \\end{align*}\n\n    Taking the norm on the both sides of above results, we have\n    \\begin{align*}\n    \\Norm{\\vx^{t+1} - \\vx^*}\n    & = \\Norm{\\mG^t\\sum_{i=1}^n\\left( \\vg_i(\\vz_i^t)^\\top(\\vz_i^t-\\vx^*) - f_i(\\vz_i^t) + f_i(\\vx^*)\\right)\\vg_i(\\vz_i^t)}\\\\\n    & \\leq \\Norm{\\mG^t}\\Norm{\\sum_{i=1}^n\\left( \\vg_i(\\vz_i^t)^\\top(\\vz_i^t-\\vx^*) - f_i(\\vz_i^t) + f_i(\\vx^*)\\right)\\vg_i(\\vz_i^t)}\\\\\n    & \\leq \\frac{L_f\\fH_\\nu}{1 + \\nu}\\Norm{\\mG^t}\\sum_{i=1}^n\\Norm{\\vz_i^t-\\vx^*}^{1+\\nu}\n    \\end{align*}    \n    where the first inequality comes from the property of matrix norm, the second inequality is based on Lemma \\ref{le:power-bound} and \\ref{le:b-grad}.\n\\end{proof}\n\nWe split the results of Theorem \\ref{thm:holder-IGN-1} into two parts (i.e., Theorem \\ref{thm:holder-IGN-1-1} and \\ref{thm:holder-IGN-1-2}) and provide their proofs as follows.\nOur analysis is based on the properties of our the auxiliary sequence constructed in Section \\ref{sec:series}.\n\n\n\\begin{thm}\n\\label{thm:holder-IGN-1-1}\n    Under the Assumption \\ref{asm:L-f}, \\ref{asm:holder-g} and \\ref{asm:b-J-sing}, we run \\IGN~(Algorithm \\ref{alg:IGN-1}) with initialization $\\vx^0\\in\\BR^d$ and~$\\mH^0=\\mJ(\\vx^0)^\\top\\mJ(\\vx^0)$ such that\n    \\begin{align*}\n    \\Norm{\\vx^0 - \\vx^*}\\leq \\left(\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\right)^{{1}/{\\nu}},\n    \\end{align*} \n    then it holds \n    \\begin{align*}\n    \\sigma_{\\min}(\\mI + (\\mV^t)^\\top (\\mH^t)^{-1}\\mU^t) > 0,\\quad\n    \\mH^t \\succeq \\frac{\\mu^2}{2}\\mI\\quad\n    \\text{and}\\quad\n    \\Norm{\\vx^{t}-\\vx^*}\\leq a_{t+1}(n, \\nu)\\Norm{\\vx^0-\\vx^*}\n    \\end{align*}\n    for all $t \\geq 0$, where sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$ is defined in equation (\\ref{eq:series}). \n\\end{thm}\n\n\\begin{proof}\n\nWe first show\n\\begin{align}\\label{eq:Hxtxstar0}\n\\mH^t \\succeq \\frac{\\mu^2}{2}\\mI\\qquad\n\\text{and}\\qquad\n\\Norm{\\vx^{t}-\\vx^*}\\leq a_{t+1}(n, \\nu)\\Norm{\\vx^0-\\vx^*}\n\\end{align}\nholds for all $t\\geq 0$. We split the proof of results (\\ref{eq:Hxtxstar0}) into the following three parts.\n\n\\textbf{Part I:} For $t=0$, \nthe initialization and the fact $a_0=1$ leads to \n\\begin{align*}\n \\Norm{\\vx^0 - \\vx^*} = a_0(n, \\nu)\\Norm{\\vx^0 - \\vx^*}.\n\\end{align*}\n\n\n\\textbf{Part II:} For all $t=0,\\cdots, n-1$, we use induction to prove the results of \n\\begin{align}\\label{eq:Hxtxstar}\n\\mH^t \\succeq \\frac{\\mu^2}{2}\\mI\\qquad\n\\text{and}\\qquad\n\\Norm{\\vx^{t+1}-\\vx^*}\\leq a_{t+1}(n, \\nu)\\Norm{\\vx^0-\\vx^*}.\n\\end{align}\nFor the induction base, we can apply Lemma \\ref{le:lb-Ht-new-1} to verify\n\\begin{align*}\n    \\sigma_{\\min}(\\mH^0) &\\geq  \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^0 - \\vx^*}^\\nu\\\\\n    & = \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vx^0 - \\vx^*}^\\nu\\\\\n    & \\geq \\mu^2 - 2L_f\\fH_\\nu n \\frac{\\mu^2}{4L_f\\fH_\\nu n}\\\\\n    & = \\mu^2 - \\frac{\\mu^2}{2}\\\\\n    & = \\frac{\\mu^2}{2}.\n\\end{align*}\n\n\nThis implies\n\\begin{align}\\label{eq:boundG0}\n\\mH^0 \\succeq \\frac{\\mu^2}{2} \\qquad \\text{and}\\qquad\n\\Norm{\\mG^0} = \\Norm{(\\mH^0)^{-1}} \\leq \\frac{2}{\\mu^2}.\n\\end{align}\nAccording to Lemma \\ref{le:main-iter-1}, we have\n\\begin{align*}\n    \\Norm{\\vx^1 - \\vx^*} \n    &\\leq \\frac{L_f\\fH_\\nu}{1+\\nu}\\Norm{\\mG^0}\\sum_{i=1}^n\\Norm{\\vz_i^0-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{L_f\\fH_\\nu}{1+\\nu}\\cdot\\frac{2}{\\mu^2}\\cdot\\sum_{i=1}^n\\Norm{\\vz_i^0-\\vx^*}^{1+\\nu}\\\\\n    & = \\frac{L_f\\fH_\\nu}{1+\\nu}\\cdot\\frac{2}{\\mu^2}\\cdot n\\Norm{\\vx^0-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{nL_f\\fH_\\nu}{1+\\nu}\\cdot\\frac{2}{\\mu^2}\\cdot\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\Norm{\\vx^0-\\vx^*}\\\\\n    & = \\frac{1}{2(1+\\nu)}\\Norm{\\vx^0-\\vx^*}\\\\\n    & = a_1(n, \\nu)\\Norm{\\vx^0 - \\vx^*},\n\\end{align*}\nwhere the first inequality is based on equation (\\ref{eq:boundG0}) and the second inequality is based on initial condition. \nTherefore, the induction base holds\n\nFor the induction step, we assume\n\\begin{align*}\n    \\mH^j \\succeq \\frac{\\mu^2}{2} \\mI \\qquad\\text{and}\\qquad\\Norm{\\vx^{j+1} - \\vx^*} \\leq a_{j+1}(n, \\nu)\\Norm{\\vx^0 - \\vx^*}\n\\end{align*}\nhold for all $j = 2, \\cdots, t-1$ such that $t \\leq n - 1$. Therefore, the update (\\ref{eq:update-1}) means\n\\begin{align}\\label{eq:vziii}\n    \\vz_i^{t}=\n    \\begin{cases}\n        \\vx^i,~~~~&1 \\leq i\\leq t,\\\\\n        \\vx^0,~~~~&t < i \\leq n. \n    \\end{cases}\n\\end{align}\nThe induction hypothesis leads to\n\\begin{align*}\n    \\Norm{\\vx^j-\\vx^*}^{\\nu}\\leq (a_j(n, \\nu))^{\\nu}\\Norm{\\vx^0-\\vx^*}^{\\nu} \\leq  \\Norm{\\vx^0-\\vx^*}^{\\nu}\\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n}, \n\\end{align*}\nfor $j = 1, \\cdots, t-1$, where the second is based on Lemma \\ref{le:s-mono-000} and the third comes from the initial condition. \nCombining with the result of (\\ref{eq:vziii}), we achive\n\\begin{align*}\n    \\Norm{\\vz_i^{t} - \\vx^*}^{\\nu} \\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n}.\n\\end{align*}\n\nAccording to Lemma \\ref{le:lb-Ht-new-1}, we have\n\\begin{align*}\n    \\sigma_{\\min}(\\mH^{t}) &\\geq  \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^{t} - \\vx^*}^\\nu\\\\\n    & \\geq \\mu^2 - 2L_f\\fH_\\nu n \\frac{\\mu^2}{4L_f\\fH_\\nu n}\\\\\n    & = \\mu^2 - \\frac{\\mu^2}{2}\\\\\n    & = \\frac{\\mu^2}{2},\n\\end{align*}\n\nwhere the second inequality comes from the initial condition.\nTherefore, we have\n\\begin{align*}\n    \\mH^{t}  \\succeq \\frac{\\mu^2}{2}\\mI \\qquad \\text{and}\\qquad\n    \\Norm{\\mG^{t}}  = \\Norm{(\\mH^{t})^{-1}} \\leq \\frac{2}{\\mu^2}.\n\\end{align*}\nAccording to Lemma \\ref{le:main-iter-1}, we have\n\\begin{align*}\n    \\Norm{\\vx^{t+1} - \\vx^*} \n    & \\leq \\frac{L_f\\fH_\\nu }{1+\\nu}\\Norm{\\mG^{t}}\\sum_{i=1}^n\\Norm{\\vz_i^{t}-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{L_f\\fH_\\nu }{1+\\nu}\\frac{2}{\\mu^2}\\sum_{i=1}^n\\Norm{\\vz_i^{t}-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu}{(1+\\nu)\\mu^2}\\left(\\sum_{j=1}^{t}\\Norm{\\vx^{j}-\\vx^*}^{1+\\nu} + (n - t)\\Norm{\\vx^0 - \\vx^*}^{1+\\nu}\\right)\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\left(\\sum_{j=1}^{t} (a_j(n, \\nu))^{1+\\nu}\\Norm{\\vx^0 - \\vx^*}^{1+\\nu} + (n - t) \\Norm{\\vx^0 - \\vx^*}^{1+\\nu}\\right)\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\left(\\sum_{j=1}^{t}(a_j(n, \\nu))^{1+\\nu} + n - t\\right)\\Norm{\\vx^0 - \\vx^*}\\\\\n    & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=1}^{t}(a_j(n, \\nu))^{1+\\nu} + n - t\\right)\\Norm{\\vx^0 - \\vx^*}\\\\\n    & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=0}^{t}(a_j(n, \\nu))^{1+\\nu} + n - t - 1\\right)\\Norm{\\vx^0 - \\vx^*}\\\\\n    & = a_{t+1}(n, \\nu)\\Norm{\\vx^0 - \\vx^*},\n\\end{align*}\nwhere the last equality comes from the fact $a_0(n, \\nu)=1$. \nTherefore, we finish the induction.\n\n\n\\textbf{Part III:} For all $t \\geq n$, we use induction to prove \n\\begin{align*}\n\\mH^t\\succeq(\\mu^2/2)\\mI\\qquad\\text{and}\\qquad\\Norm{\\vx^{t+1} - \\vx^*}\\leq a_{t+1}(n, \\nu)\\Norm{\\vx^0 - \\vx^*}.    \n\\end{align*}\nFor the induction base, we can verify that it holds (from the result of Part II) \n\\begin{align*}\n\\mH^j \\succeq \\frac{\\mu^2}{2}\\mI \\qquad \\text{for all~~} j=0,\\dots,n - 1,\n\\end{align*}\nand\n\\begin{align*}\n\\Norm{\\vx^j-\\vx^*}\\leq a_j(n, \\nu)\\Norm{\\vx^0-\\vx^*} \\qquad \\text{for all~~} j=1,\\dots,n.\n\\end{align*}\nThen we have\n\\begin{align*}\n    \\Norm{\\vx^j-\\vx^*}^{\\nu}\\leq (a_j(n, \\nu))^{\\nu}\\Norm{\\vx^0-\\vx^*}^{\\nu} \\leq \\Norm{\\vx^0-\\vx^*}^{\\nu} \\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n}, \\quad \\text{for all~~} j=1,\\dots,n,\n\\end{align*}\nwhere the second inequality is based on Lemma \\ref{le:s-mono-000} and the third inequality is based on the initial condition.\n\nFrom Eq.~\\ref{eq:update-1}, we have\n\\begin{align*}\n    \\vz_i^{n}=\\vx^i \\qquad\\text{for all~~} i\\in[n].\n\\end{align*}\n\nTherefore, we have\n\\begin{align*}\n    \\Norm{\\vz_i^{n} - \\vx^*}^{\\nu} \\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n}, \\qquad\\text{for all~~} i\\in[n].\n\\end{align*}\nAccording to Lemma \\ref{le:lb-Ht-new-1}, we have\n\\begin{align*}\n    \\sigma_{\\min}(\\mH^{n}) &\\geq  \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^{n} - \\vx^*}^\\nu\\\\\n    & \\geq \\mu^2 - 2L_f\\fH_\\nu n \\frac{\\mu^2}{4L_f\\fH_\\nu n}\\\\\n    & \\geq \\mu^2 - \\frac{\\mu^2}{2} = \\frac{\\mu^2}{2},\n\\end{align*}\n\nwhich implies\n\\begin{align*}\n    \\mH^{n}  \\succeq \\frac{\\mu^2}{2}\\mI \\qquad\\text{and}\\qquad\n    \\Norm{\\mG^{n}}  = \\Norm{(\\mH^{n})^{-1}} \\leq \\frac{2}{\\mu^2}.\n\\end{align*}\nAccording to Lemma \\ref{le:main-iter-1}, we have\n\\begin{align*}\n    \\Norm{\\vx^{n+1} - \\vx^*}\n    & \\leq \\frac{L_f\\fH_\\nu }{1+\\nu}\\Norm{\\mG^n}\\sum_{i=1}^n\\Norm{\\vz_i^n-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\sum_{i=1}^n\\Norm{\\vz_i^n-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\left(\\sum_{j=1}^n (a_j(n, \\nu))^{1+\\nu}\\Norm{\\vx^0 - \\vx^*}^{1+\\nu}\\right)\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\left(\\sum_{j=1}^{n}(a_j(n, \\nu))^{1+\\nu}\\right)\\Norm{\\vx^0-\\vx^*}\\\\\n    & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=1}^{n}(a_j(n, \\nu))^{1+\\nu}\\right)\\Norm{\\vx^0-\\vx^*}\\\\\n    & = a_{n+1}(n, \\nu)\\Norm{\\vx^0-\\vx^*}.\n\\end{align*}\nHence, we have shown the induction base holds. \n\nFor the induction step, we assume\n\\begin{align*}\n    \\mH^j \\succeq \\frac{\\mu^2}{2} \\mI \\qquad\\text{and}\\qquad\\Norm{\\vx^{j+1} - \\vx^*} \\leq a_{j+1}(n, \\nu)\\Norm{\\vx^0 - \\vx^*}\n\\end{align*}\nholds for all $j = n+1, \\cdots, t-1$ such that $t \\geq n + 2$. Combining results of Part I and II, we have\n\n\\begin{align*}\n\\Norm{\\vx^j-\\vx^*}\\leq a_j(n, \\nu)\\Norm{\\vx^0-\\vx^*} \\qquad \\text{for all}~~ j=0,\\dots,t,\n\\end{align*}\nwhich implies\n\\begin{align*}\n    \\Norm{\\vx^j-\\vx^*}^{\\nu}\\leq (a_j(n, \\nu))^{\\nu}\\Norm{\\vx^0-\\vx^*}^{\\nu} \\leq \\Norm{\\vx^0-\\vx^*}^{\\nu} \\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n}, \\qquad\\text{for all}~~j=1,\\dots,t,\n\\end{align*}\nwhere the second inequality is based on Lemma \\ref{le:s-mono-000} and the last inequality is based on the condition condition.\n\n\nThe update~(\\ref{eq:update-1}) means the points $\\{\\vz_i^{t}\\}_{i=1}^n$ can be written as $\\{\\vx^{t+1-n}, \\cdots, \\vx^{t}\\}$, which implies\n\\begin{align*}\n    \\max\\{\\Norm{\\vz_1^{t}-\\vx^*}, \\cdots, \\Norm{\\vz_n^{t}-\\vx^*}\\}=\\max\\{\\Norm{\\vx^{t+1-n}-\\vx^*}, \\cdots, \\Norm{\\vx^{t}-\\vx^*}\\}.\n\\end{align*}\nTherefore, we have\n\\begin{align*}\n    \\Norm{\\vz_i^{n} - \\vx^*}^{\\nu} \\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n} \\qquad \\text{for all}~~i=1,\\dots,n.\n\\end{align*}\n\nCombing with Lemma \\ref{le:lb-Ht-new-1}, we have\n\\begin{align*}\n    \\sigma_{\\min}(\\mH^{t}) &\\geq  \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^{t} - \\vx^*}^\\nu\\\\\n    & \\geq \\mu^2 - 2L_f\\fH_\\nu n \\frac{\\mu^2}{4L_f\\fH_\\nu n}\\\\\n    & = \\mu^2 - \\frac{\\mu^2}{2} = \\frac{\\mu^2}{2}.\n\\end{align*}\nTherefore, we achieve\n\\begin{align*}\n    \\mH^{t}  \\succeq \\frac{\\mu^2}{2}\\mI \\qquad\\text{and}\\qquad\n    \\Norm{\\mG^{t}} & = \\Norm{(\\mH^{t})^{-1}} \\leq \\frac{2}{\\mu^2}.\n\\end{align*}\nAccording to Lemma \\ref{le:main-iter-1}, we have\n\\begin{align*}\n    \\Norm{\\vx^{t+1} - \\vx^*}\n    & \\leq \\frac{L_f\\fH_\\nu }{1+\\nu}\\Norm{\\mG^{t}}\\sum_{i=1}^n\\Norm{\\vz_i^{t}-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\sum_{i=1}^n\\Norm{\\vz_i^{t}-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\left(\\sum_{j=t-n+1}^{t} (a_j(n, \\nu))^{1+\\nu}\\Norm{\\vx^0 - \\vx^*}^{1+\\nu}\\right)\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\left(\\sum_{j=t-n+1}^{t} (a_j(n, \\nu))^{1+\\nu}\\right)\\Norm{\\vx^0 - \\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\left(\\sum_{j=t-n+1}^{t+1}a_j(n, \\nu)^{1+\\nu}\\right)\\Norm{\\vx^0 - \\vx^*}\\\\\n    & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=t-n+2}^{t+1}(a_j(n, \\nu))^{1+\\nu}\\right)\\Norm{\\vx^0 - \\vx^*}\\\\\n    & = a_{t+1}(n, \\nu)\\Norm{\\vx^0 - \\vx^*}.\n\\end{align*}\nHence, we finish the induction.\n\n\nCombining results of Part I, II and III completes the proof of (\\ref{eq:Hxtxstar0}).\n\nSince the non-singularity of $\\mH^t$ and $\\mH^{t+1}$ has been verified by result (\\ref{eq:Hxtxstar0}), we can apply Lemma \\ref{le:via-update} to achieve\n\\begin{align*}\n    \\sigma_{\\min}(\\mI + (\\mV^t)^\\top(\\mH^t)^{-1}\\mU^t) > 0.\n\\end{align*}\n\\end{proof}\n\n\\begin{thm}\n\\label{thm:holder-IGN-1-2}\nWe define the sequence $\\{r_t\\}_{t\\geq 0}$ such that\n\\begin{align*}\n    r_t\\triangleq\\begin{cases}\n    \\max\\{\\Norm{\\vx^0 - \\vx^*}, 1\\},~~~~&t=0,\\\\[0.2cm]\n    a_t(n, \\nu)r_0,~~~~& t\\geq 1,\\\\\n\\end{cases}\n\\end{align*}\nwhere the sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$ is defined by equation (\\ref{eq:series}). \nUnder the Assumptions \\ref{asm:L-f}, \\ref{asm:holder-g} and \\ref{asm:b-J-sing}, running \\IGN~(Algorithm \\ref{alg:IGN-1}) with initial condition shown in Theorem \\ref{thm:holder-IGN-1-1}, we have\n    \\begin{align}\n    \\label{eq:r_t_recursion}\n    \\Norm{\\vx^t - \\vx^*} \\leq r_t\n    \\qquad\\text{and}\\qquad\n    r_{t+1}\\leq  c^{(1+\\nu)^{\\left(\\floor{\\frac{t}{n}}-1\\right)}}r_t  \n    \\end{align}\n    for all $t\\geq n$, where\n    \\begin{align*}\n    c = 1- \\frac{1}{n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right).\n    \\end{align*}\n\\end{thm}\n\n\\begin{proof}\n    The definition of $\\{r_t\\}_{t\\geq 0}$ leads to \n    \\begin{align*}\n        r_0 = \\max\\{\\Norm{\\vx^0 - \\vx^*}, 1\\} \\geq \\Norm{\\vx^0 - \\vx^*}.\n    \\end{align*}\n    According to Theorem \\ref{thm:holder-IGN-1-1}, we have \n    \\begin{align*}\n    \\Norm{\\vx^t - \\vx^*} \\leq a_{t}(n, \\nu)\\Norm{\\vx^0 - \\vx^*} \\leq a_t(n, \\nu)r_0 = r_t.\n    \\end{align*}\n    According to Lemma \\ref{le:super-con}, we have\n    \\begin{align*}\n         a_{t+1}(n, \\nu)\\leq  c^{(1+\\nu)^{\\left(\\floor{\\frac{t}{n}}-1\\right)}}a_t(n, \\nu) \\qquad \\text{for all}~~t \\geq n.\n    \\end{align*}\n    Thus, achieve\n    \\begin{align*}\n        r_{t+1} = a_{t+1}(n, \\nu)r_0\\leq  c^{(1+\\nu)^{\\left(\\floor{\\frac{t}{n}}-1\\right)}}a_t(n, \\nu)r_0 =c^{(1+\\nu)^{\\left(\\floor{\\frac{t}{n}}-1\\right)}}r_t  \\qquad \\text{for all}~~t \\geq n,\n    \\end{align*}\n    where\n    \\begin{align*}\n        c = 1- \\frac{1}{n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right).\n    \\end{align*}\n\\end{proof}\n\nCombining the results of Theorem \\ref{thm:holder-IGN-1-1} and  \\ref{thm:holder-IGN-1-2}, we finish the proof of Theorem \\ref{thm:holder-IGN-1}.\n","units":[{"start":27035,"end":27238,"start_line":632,"end_line":637,"raw_tex":"\\begin{prop}\\label{prop:b-JJ-sing}\nUnder Assumption \\ref{asm:b-J-sing}, it holds that\n\\begin{align}\\label{eq::b-JJ-sing}\n    \\sigma_{\\min} (\\mJ(\\vx^*)^\\top \\mJ(\\vx^*)) = \\mu^2 > 0.\n\\end{align}\n\\end{prop}","sha256_utf8":"a4dd6c5faf5a14de441b1de6e01d0ad82ba5eccd36d175b8dfa89ff65839fee0"},{"start":27370,"end":27804,"start_line":642,"end_line":655,"raw_tex":"\\begin{lem}\n\\label{le:L-JJ}\n{\n    Under Assumptions \\ref{asm:L-f} and \\ref{asm:holder-g}, we have \n    \\begin{align*}\n    \\Norm{\\mJ(\\vy)^\\top\\mJ(\\vy) - \\mJ(\\vx)^\\top\\mJ(\\vx)} & \\leq 2L_f\\fH_\\nu\\Norm{\\vy-\\vx}^\\nu\n    \\end{align*}\n    and\n    \\begin{align*}\n    \\Norm{\\vg_i(\\vy)\\vg_i(\\vy)^\\top - \\vg_i(\\vx)\\vg_i(\\vx)^\\top}   \\leq 2L_f\\fH_\\nu\\Norm{\\vy-\\vx}^\\nu\n    \\end{align*}\n    for all $\\vx, \\vy\\in\\BR^n$ and $i \\in [n]$.\n}\n\\end{lem}","sha256_utf8":"b64c5f044320a1f55821bd8c3aa784d451195541e3b2a1e9e8d7ef6847823b7b"},{"start":28047,"end":28421,"start_line":661,"end_line":668,"raw_tex":"\\begin{lem}\n\\label{le:lb-Ht-new-1}\nUnder Assumptions \\ref{asm:L-f}, \\ref{asm:holder-g} and \\ref{asm:b-J-sing}, running \\IGN~(Algorithm \\ref{alg:IGN-1}) with $\\mH^0 = \\mJ(\\vx^0)^\\top\\mJ(\\vx^0)$ and~$\\mG^0=(\\mH^0)^{-1}$ holds that\n\\begin{align*}\n    \\sigma_{\\min}(\\mH^t) \\geq \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^t - \\vx^*}^\\nu\n\\end{align*}\nfor all $t\\geq 0$.\n\\end{lem}","sha256_utf8":"3771f77b2ae64d0497d93b06ea1f165a487e3384f487ecb6f15fd6ed9d019767"},{"start":52499,"end":52801,"start_line":1084,"end_line":1091,"raw_tex":"\\begin{lem}\n    \\label{le:H-con}\n    \\textnormal{(H\\\"older continuity of each gradient)} Under Assumption \\ref{asm:holder-g}, it satisfies that\n    \\begin{align}\n        \\Norm{\\vg_i(\\vy)-\\vg_i(\\vx)}\\leq \\fH_\\nu\\Norm{\\vy-\\vx}^\\nu,\n    \\end{align}\n    for any $\\vx, \\vy\\in\\BR^d$, and $i\\in[n]$.\n\\end{lem}","sha256_utf8":"bdce0cee8f1ac659190635fffb97a29f35d2049e4aa7896fdbb8a0a0d0f2b9a6"},{"start":52803,"end":53583,"start_line":1093,"end_line":1110,"raw_tex":"\\begin{proof}\n    We denote\n    \\begin{align*}\n    \\Tilde{\\mJ} = \\begin{bmatrix}\n        (\\vg_1(\\vy)-\\vg_1(\\vx))^\\top \\\\\n        \\vdots \\\\\n        (\\vg_n(\\vy)-\\vg_n(\\vx))^\\top \n    \\end{bmatrix}\\in\\BR^{n\\times d} \\qquad\\text{with}\\qquad \\vg_i(\\vx)=\\nabla f_i(\\vx)\n    \\end{align*}\n    and let $\\ve_i\\in\\BR^{n}$ be the $i$-th standard basic vector in $n$-dimensional Euclidean space. \n    Then the facts~$\\Tilde{\\mJ}=\\mJ(\\vy)-\\mJ(\\vx)$ and $\\Tilde{\\mJ}^\\top \\ve_i=\\vg_i(\\vy)-\\vg_i(\\vx)$ imply we have\n    \\begin{align*}\n    \\Norm{\\vg_i(\\vy)-\\vg_i(\\vx)}\n    \\leq \\|\\Tilde{\\mJ}^\\top\\|\\Norm{\\ve_i} = \n    \\|\\Tilde{\\mJ}\\|=\\Norm{\\mJ(\\vy)-\\mJ(\\vx)}\\leq \\fH_\\nu\\Norm{\\vy-\\vx}^\\nu,\n    \\end{align*}\n    where the last step is based on the H\\\"older continuouity of $\\mJ(\\cdot)$.\n\\end{proof}","sha256_utf8":"1954bd6ca68c9bf6d4d4252fcd97dbbb8bd03b29075b78073899aaff4d16ece9"},{"start":53585,"end":53916,"start_line":1112,"end_line":1119,"raw_tex":"\\begin{lem}\n\\label{le:power-bound}\n\\textnormal{(Bound for H\\\"older-continuous function)} Under Assumption \\ref{asm:holder-g}, we have\n    \\begin{align}\n        f_i(\\vy) - f_i(\\vx) - \\vg_i(\\vx)^\\top(\\vy-\\vx) \\leq \\frac{\\fH_\\nu}{1 + \\nu}\\Norm{\\vy-\\vx}^{1+\\nu},\n    \\end{align}\n    for any $\\vx, \\vy\\in\\BR^d$ and  $i\\in[n]$.\n\\end{lem}","sha256_utf8":"94f613c09c820ee6e5ca298b733808fd5ee3b7911152f2a4cf0a17c3f652520c"},{"start":53918,"end":54817,"start_line":1121,"end_line":1133,"raw_tex":"\\begin{proof}\n    Following the proof of \\cite{grapiglia2017regularized, grapiglia2019accelerated}, we have\n    \\begin{align*}\n        f_i(\\vy) - f_i(\\vx) - \\vg_i(\\vx)^\\top(\\vy - \\vx)\n        & = \\int_{t=0}^1 \\vg_i(\\vx + t(\\vy - \\vx))^\\top(\\vy - \\vx)\\text{d}t - \\vg_i(\\vx)^\\top(\\vy - \\vx)\\\\\n        & = \\int_{t=0}^1 \\left(\\vg_i(\\vx + t(\\vy - \\vx)) - \\vg_i(\\vx)\\right)^\\top(\\vy - \\vx)\\text{d}t\\\\\n        & \\leq \\int_{t=0}^1 \\Norm{\\vg_i(\\vx + t(\\vy - \\vx)) - \\vg_i(\\vx)}\\Norm{\\vy - \\vx}\\text{d}t\\\\\n        & \\leq \\int_{t=0}^1\\fH_\\nu t^\\nu\\Norm{\\vy - \\vx}^{1+\\nu}\\text{d}t\\\\\n        & = \\fH_\\nu \\Norm{\\vy - \\vx}^{1+\\nu}\\int_{t=0}^1 t^\\nu\\text{d}t\\\\\n        & = \\frac{\\fH_\\nu}{1 + \\nu} \\Norm{\\vy - \\vx}^{1+\\nu},\n    \\end{align*}\n    where the first inequality comes from Cauchy-Schwarz inequality, and the second one comes from Lemma \\ref{le:H-con} that each gradient is H\\\"older continuous.\n\\end{proof}","sha256_utf8":"7e06d66d9f733d8a3e976b28b2ee6b5bb7890721af13a505ca8591a068b68341"},{"start":54819,"end":55081,"start_line":1135,"end_line":1142,"raw_tex":"\\begin{lem}\n\\label{le:b-grad}\n    \\textnormal{(Bound for Jacobian and gradient)} Under Assumption \\ref{asm:L-f}, we have\n    \\begin{align*}\n        \\Norm{\\vg_i(\\vx)}\\leq\\Norm{\\mJ(\\vx)}\\leq L_f\n    \\end{align*}\n    for all $ \\vx\\in\\BR^d$ and $i\\in [n]$.\n\\end{lem}","sha256_utf8":"3396877002acc980f62febd820b1a07efd2e7ba2d2f35d988401a0aa6188ead0"},{"start":55083,"end":56119,"start_line":1144,"end_line":1170,"raw_tex":"\\begin{proof}\n\n    For all $\\vx, \\vv\\in\\BR^d$, we have\n    \\begin{align*}\n        \\mJ(\\vx)\\vv =\\lim_{h\\to0}\\frac{\\vf(\\vx+h\\vv)-\\vf(\\vx)}{h}.\n    \\end{align*}\n    Taking the spectral norm on both sides, we have\n    \\begin{align*}\n        \\Norm{\\mJ(\\vx)\\vv}\n        & = \\lim_{h\\to0}\\frac{\\Norm{\\vf(\\vx+h\\vv)-\\vf(\\vx)}}{\\vert h\\vert}\\\\\n        & \\leq \\lim_{h\\to0}\\frac{L_f\\Norm{\\vx+h\\vv-\\vx}}{\\vert h\\vert}\\\\\n        & = \\lim_{h\\to0}\\frac{L_f\\vert h\\vert\\Norm{\\vv}}{\\vert h\\vert}\\\\\n        & = L_f\\Norm{\\vv},\n    \\end{align*}\n    where the inequality comes from Assumption \\ref{asm:L-f}. \n    \n    Therefore, for all $\\vx\\in\\BR^d$ it holds\n    \\begin{align*}\n    \\Norm{\\mJ(\\vx)}=\\sup_{\\vv\\in\\BR^d}\\frac{\\Norm{\\mJ(\\vx)\\vv}}{\\Norm{\\vv}}\\leq L_f.\n    \\end{align*}\n    Let $\\ve_i\\in\\BR^{n}$ be the $i$-th standard basic vector in $n$-dimensional Euclid space, then we have \n\n    \\begin{align*}\n    \\Norm{\\vg_i(\\vx)} = \\Norm{\\mJ^\\top \\ve_i} \\leq \\Norm{\\mJ}\\Norm{\\ve_i}=\\Norm{\\mJ(\\vx)}\\leq L_f\n    \\end{align*}\n    for all $i\\in[n]$.\n\\end{proof}","sha256_utf8":"bd87440427c8b1ef9d43f24ab35851da14793b4ba36a0447fc3f3f6c42bbb509"},{"start":56884,"end":57051,"start_line":1195,"end_line":1201,"raw_tex":"\\begin{lem}\n\\label{le:s-mono-000} The sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$ satisfies \n\\begin{align*}\n    a_t(n, \\nu) \\leq 1 \n\\end{align*}\nfor all $t\\geq 0$. \n\\end{lem}","sha256_utf8":"03a7841da04ea2b4c22702e84c915ac35f6647f6a114979ac070c7342829c8f5"},{"start":57052,"end":58783,"start_line":1202,"end_line":1232,"raw_tex":"\\begin{proof}\n\\textbf{Part I:}\nWe first use induction to prove $a_t(n, \\nu) \\leq 1$ for all $t=0,1\\dots,n$.\nFor the induction base, we can verify that $a_0(n, \\nu) = 1 \\leq 1$.  For the induction step, we assume \n\\begin{align*}\na_j(n,\\nu) \\leq 1    \n\\end{align*}\nholds for all $j=1,\\dots,t-1$ such that $t\\leq n$.\nThen we have\n\\begin{align*}\n        a_t(n,\\nu) = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=0}^{t-1}(a_j(n,\\nu))^{1+\\nu} + n - t\\right) \\leq \\frac{1}{2(1+\\nu)n}\\left(t + n - t\\right) = \\frac{1}{2(1+\\nu)}\\leq 1,\n\\end{align*}\nwhere the first inequality is based on the induction hypothesis and the last inequality is based on the setting $\\nu\\in(0,1]$. This finishes the induction. \n\n\\textbf{Part II:} We then use induction to prove $a_t(n, \\nu) \\leq 1$ for all $t\\geq n+1$. For the induction base, we can verify that\n\\begin{align*}\n    a_{n+1}(n,\\nu) = \\frac{1}{2(1+\\nu)n}\\sum_{j=1}^{n}(a_j(n, \\nu))^{1+\\nu} \\leq \\frac{1}{2(1+\\nu)n}\\cdot n = \\frac{1}{2(1+\\nu)} \\leq 1,\n\\end{align*}\nwhere the first inequality is based on $a_t(n, \\nu) \\leq 1$ for all $t\\leq n$ (which have shown in Part I), and the last inequality is based on the setting $\\nu \\in (0, 1]$. For the induction step, we assume\n\\begin{align*}\n    a_{n+1}(n,\\nu) \\leq 1\n\\end{align*}\nholds for all $j=n+2,\\dots,t-1$ such that $t\\geq n+3$. Then we have\n\\begin{align*}\n    a_{t}(n,\\nu) = \\frac{1}{2(1+\\nu)n}\\sum_{j=t-n}^{t-1}(a_j(n, \\nu))^{1+\\nu} \\leq \\frac{1}{2(1+\\nu)n}\\cdot n = \\frac{1}{2(1+\\nu)} \\leq 1,\n\\end{align*}\n\nwhere the first inequality is based on the induction hypothesis and the last inequality is based on the setting $\\nu \\in (0, 1]$. This finishes the induction.\n\nCombining the results of above two parts, we finish the proof of this lemma.\n\\end{proof}","sha256_utf8":"d2ce043eab60f44d61e0f4278068fdcf3749d93a0b9b1e775ed34758beee3634"},{"start":58786,"end":58965,"start_line":1235,"end_line":1242,"raw_tex":"\\begin{lem}\n\\label{le:s-mono}The sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$ satisfies \n\\begin{align*}\n    a_{t}(n, \\nu) \\geq a_{t+1}(n, \\nu) \n\\end{align*}\nfor all $t\\geq 0$. \n\n\\end{lem}","sha256_utf8":"e9d132c2365b7882543f4fde35c226bd1b8a6d8c827912f52a2c84391d088cde"},{"start":58966,"end":60821,"start_line":1243,"end_line":1279,"raw_tex":"\\begin{proof}\n\\textbf{Part I:} For $t=0$, the fact $\\nu\\in(0, 1]$ means\n\\begin{align*}\n    a_1(n, \\nu) = \\frac{1}{2(1+\\nu)} \\leq 1 = a_0(n, \\nu).\n\\end{align*}\n\\textbf{Part II:}\nFor all $t=1,\\dots,n-1$, we have\n\\begin{align*}\n        & a_{t+1}(n, \\nu) - a_{t}(n, \\nu) \\\\\n        & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=0}^{t}(a_j(n, \\nu))^{1+\\nu} + n - t - 1\\right) - \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=0}^{t-1}(a_j(n, \\nu))^{1+\\nu} + n - t\\right)\\\\\n        & = \\frac{1}{2(1+\\nu)n}\\left((a_{t}(n, \\nu))^{1+\\nu} - 1\\right)  \\leq 0,\n\\end{align*}     \nwhere the last inequality is based on Lemma \\ref{le:s-mono-000}. This indicates $a_{t+1}\\leq a_{t}$ for $t=1,\\dots,n-1$. \n\n\\textbf{Part III:} For all $t\\geq n$, we use induction to prove $a_{t+1}(n, \\nu)\\leq a_{t}(n, \\nu)$. \nFor the induction base, we can verify that\n    \\begin{align*}\n        & a_{n+1}(n, \\nu) - a_n(n, \\nu) \\\\\n        & = \\frac{1}{2(1+\\nu)n}\\sum_{j=1}^{n}(a_j(n, \\nu))^{1+\\nu} - \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=1}^{n-1}(a_j(n, \\nu))^{1+\\nu} + 1\\right)\\\\\n        & = \\frac{1}{2(1+\\nu)n}\\left((a_n(n, \\nu))^{1+\\nu} - 1\\right)\\\\\n        & \\leq 0,\n    \\end{align*}\n    where the last inequality is based on Lemma \\ref{le:s-mono-000}. \n    \n    For the induction step, we assume\n\\begin{align*}\n    a_{j+1}(n, \\nu) \\leq a_{j}(n, \\nu)\n\\end{align*}\nholds for all $j = n+1, \\cdots, t-1$ such that $t\\geq n+2$. Then we have\n    \\begin{align*}\n        & a_{t+1}(n, \\nu) - a_{t}(n, \\nu) \\\\\n         =&  \\frac{1}{2(1+\\nu)n}\\sum_{j=t-n + 1}^{t}(a_j(n, \\nu)) ^{1+\\nu} - \\frac{1}{2(1+\\nu)n}\\sum_{j=t-n}^{t-1}(a_j(n, \\nu))^{1+\\nu} \\leq 0,\n    \\end{align*}\nwhere the inequality is based on the induction hypothesis and the fact $a_{t+1}(n, \\nu) \\leq a_{t}(n, \\nu)$ for all~$t \\leq n - 1$ (which have shown in Part I).\n\nCombining the results of above three parts, we finish the proof of this lemma.\n\\end{proof}","sha256_utf8":"f151a7e894f8de7748cfb78333b2654f26ca838942832a93c909b71e33874680"},{"start":60824,"end":61049,"start_line":1282,"end_line":1289,"raw_tex":"\\begin{lem}\n\\label{le:n-qua-con}\n     For the sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$, we have\n    \\begin{align*}\n    a_t(n, \\nu) \\leq \\frac{1}{2(1+\\nu)}(a_{t-n}(n, \\nu))^{1+\\nu}\n    \\end{align*}\n    for all $t\\geq n$.\n\\end{lem}","sha256_utf8":"1184dba5bc39e03ed93e3df72746aefd5a0eaf75c3a8c2ef2dfec5855f35abf2"},{"start":61050,"end":61760,"start_line":1290,"end_line":1305,"raw_tex":"\\begin{proof}\n    For all $t\\geq n$, the definition of $a_t(n, \\nu)$ implies\n    \\begin{align*}\n    & a_t(n, \\nu) = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=t-n}^{t-1}(a_j(n, \\nu))^{1+\\nu}\\right)  \\\\\n    & \\leq \\frac{1}{2(1+\\nu)} \\max\\{(a_{t-n}(n, \\nu))^{1+\\nu},\\cdots, (a_{t-1}(n, \\nu))^{1+\\nu}\\}.\n    \\end{align*}\n    Additionally, Lemma \\ref{le:s-mono} implies for all $t\\geq n$, we have\n    \\begin{align*}\n    \\max\\{(a_{t-n}(n, \\nu))^{1+\\nu},\\cdots, (a_{t-1}(n, \\nu))^{1+\\nu}\\} = (a_{t-n}(n, \\nu))^{1+\\nu}, \\quad t\\geq n.\n    \\end{align*}\n    Combining above results, we achieve\n    \\begin{align*}\n    a_t(n, \\nu) \\leq \\frac{1}{2(1+\\nu)}(a_{t-n}(n, \\nu))^{1+\\nu}\n    \\end{align*}\n    for all $t\\geq n$.\n\\end{proof}","sha256_utf8":"56c0a8589da70e1644c7dfad4df01b8c7302e1b90ebaa787b9bf0a7f287672ef"},{"start":61762,"end":62057,"start_line":1307,"end_line":1316,"raw_tex":"\\begin{lem}\n\\label{le:linear-con} For the sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$, we have \n\\begin{align*}\na_{t+1}(n, \\nu)\\leq c_0 a_t(n, \\nu)\n\\end{align*}\nfor all $t\\geq n$, where\n\\begin{align*}\n    c_0 = 1- \\frac{1}{n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right).\n\\end{align*}\n\\end{lem}","sha256_utf8":"dd69ece0890b3138db25407012ed29e49ef70912534793ae703d99c4b693e807"},{"start":62058,"end":63653,"start_line":1317,"end_line":1337,"raw_tex":"\\begin{proof}\n    For all $t\\geq n$, we have\n    \\begin{align}\\label{eq:lemma9-ieq}\n    a_t(n, \\nu) \\leq \\frac{1}{2(1+\\nu)}(a_{t-n}(n, \\nu))^{1+\\nu} \\leq \\frac{1}{2(1+\\nu)}a_{t-n}(n, \\nu), \n    \\end{align}\n    where the first inequality is based on Lemma \\ref{le:n-qua-con} and the second one is based on Lemma \\ref{le:s-mono-000}.\n    \n    Then we also have \n    \\begin{align*}\\small\\begin{split}\n        a_{t+1} \n        & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=t-n+1}^{t}(a_j(n, \\nu))^{1+\\nu}\\right)\\\\\n        & \\leq \\frac{1}{2(1+\\nu)n}\\left(\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}(a_{t-n}(n, \\nu))^{1+\\nu} + \\sum_{j=t-n+1}^{t-1}(a_j(n, \\nu))^{1+\\nu}\\right)\\\\\n        & = \\frac{1}{2(1+\\nu)n}\\left(\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}(a_{t-n}(n, \\nu))^{1+\\nu} + \\sum_{j=t-n+1}^{t-1}(a_j(n, \\nu))^{1+\\nu} + (a_{t-n}(n, \\nu))^{1+\\nu} - (a_{t-n}(n, \\nu))^{1+\\nu} \\right)\\\\\n        & = a_{t}(n, \\nu) + \\frac{1}{2(1+\\nu)n}\\left(\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}(a_{t-n}(n, \\nu))^{1+\\nu} - (a_{t-n}(n, \\nu))^{1+\\nu} \\right)\\\\\n        & = a_{t}(n, \\nu) - \\frac{1}{2(1+\\nu)n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right)(a_{t-n}(n, \\nu))^{1+\\nu}\\\\\n        & \\leq a_{t}(n, \\nu) - \\frac{1}{n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right)a_t(n, \\nu)\\\\\n        & = \\left(1- \\frac{1}{n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right)\\right)a_t(n, \\nu),\n     \\end{split}\n    \\end{align*}\n    for all $t\\geq n$, where the first inequality is based on equation (\\ref{eq:lemma9-ieq}) and the last inequality is based on Lemma \\ref{le:n-qua-con}. This finish the proof.\n\\end{proof}","sha256_utf8":"ab131d581e649a8b24e691f5d1524f1714afa572552323491dc3b94fb40ea15a"},{"start":63656,"end":64017,"start_line":1339,"end_line":1350,"raw_tex":"\\begin{lem}\n\\label{le:imp-c} For the sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$, if there exists $c_1\\in(0,1)$ and $t_0\\geq 0$ such that\n\\begin{align}\\label{eq:ieq-t0n2n}\na_{t+1}(n, \\nu) \\leq c_1 a_t(n, \\nu)\n\\end{align}\nfor all $t\\geq t_0 + n$, \nthen we have\n\\begin{align*}\na_{t+1}(n, \\nu) \\leq c_1^{1+\\nu} a_t(n, \\nu)\n\\end{align*}\nfor all $t\\geq t_0 + 2n$.\n\\end{lem}","sha256_utf8":"254345bdd7ca841e87ba467447efc119869455e36664fb8b737ef7ace3e4780c"},{"start":64018,"end":64398,"start_line":1351,"end_line":1359,"raw_tex":"\\begin{proof}\n    For all  $t\\geq t_0 + 2n$, we have\n    \\begin{align*}\n        & a_{t+1} (n, \\nu)\n         = \\frac{1}{2(1+\\nu)n}\\sum_{j=t-n+1}^{t}(a_j(n, \\nu))^{1+\\nu} \\\\\n        & \\leq \\frac{1}{2(1+\\nu)n}\\sum_{j=t-n}^{t-1}c_1^{1+\\nu}(a_j(n, \\nu))^{1+\\nu} = c_1^{1+\\nu} a_t(n, \\nu),\n    \\end{align*}\n    where the inequality is based on equation (\\ref{eq:ieq-t0n2n}).\n\\end{proof}","sha256_utf8":"edb9bf0c9090d87f4b0185c53af7c329b6239f411c5c9483f1510a822f16b163"},{"start":64400,"end":65761,"start_line":1361,"end_line":1396,"raw_tex":"\\begin{lem}\n\\label{le:super-con} For the sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$,\nwe have the superlinear convergence\n\\begin{align*}\na_{t+1}(n, \\nu) \\leq c^{(1+\\nu)^{\\left(\\floor{\\frac{t-1}{n}}-1\\right)}}a_t(n, \\nu) \n\\end{align*}\nfor all $t\\geq n$, where\n\\begin{align*}\n    c = 1- \\frac{1}{n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right).\n\\end{align*}\n\\begin{proof}\n    According to Lemma \\ref{le:linear-con}, we have\n    \\begin{align*}\n        a_{t+1}(n, \\nu)\\leq c a_t(n, \\nu) \n        \\quad \\text{for all}~~t \\geq n.\n    \\end{align*}\n    According to Lemma \\ref{le:imp-c}, we have\n    \\begin{align*}\n        a_{t+1}(n, \\nu) & \\leq c^{1+\\nu} a_t(n, \\nu) \\quad \\text{for all}~~t \\geq 2n,\\\\\n        a_{t+1}(n, \\nu) & \\leq c^{(1+\\nu)^2} a_t(n, \\nu) \\quad \\text{for all}~~t \\geq 3n,\\\\\n        a_{t+1}(n, \\nu) & \\leq c^{(1+\\nu)^3} a_t(n, \\nu) \\quad \\text{for all}~~t \\geq 4n,\\\\\n        \\cdots &\n    \\end{align*}\n    which implies\n    \\begin{align*}\n        a_{t+1}(n, \\nu) \\leq c^{(1+\\nu)^{\\left(\\lfloor {t}/{n}  \\rfloor-1\\right)}}a_t(n, \\nu)\n    \\end{align*}\n    for all $t \\geq n$.\n\n    The superlinear convergence of the sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$ can be verify by the fact\n    \\begin{align*}\n        \\lim_{t\\to\\infty}c^{(1+\\nu)^{\\left(\\lfloor {t}/{n}  \\rfloor-1\\right)}}=0.\n    \\end{align*}\n    Hence, we finish the proof.\n\\end{proof}\n\\end{lem}","sha256_utf8":"b2b2cbf471c1ea2d04aff75b22a7e8dce1092b7858fb6a19849045ab773da088"},{"start":65973,"end":66682,"start_line":1405,"end_line":1419,"raw_tex":"\\begin{proof}\nWe denote the singular value decomposition of $\\mJ(\\vx^*)$ as\n    \\begin{align*}\n        \\mJ(\\vx^*) = \\mP\\mD\\mQ^\\top,\n    \\end{align*}\n    where $\\mP\\in\\BR^{n\\times d}, \\mQ\\in\\BR^{d\\times d}$ are (column) orthogonal matrices and $\\mD\\in\\BR^{d\\times d}$ is diagonal matrix with the smallest diagonal entry of $\\mu>0$.\n    Therefore, we have\n    \\begin{align*}\n        \\mJ(\\vx^*)^\\top \\mJ(\\vx^*) = \\mQ\\mD^2\\mQ^\\top,\n    \\end{align*}\n    which means the smallest singular value of $\\mJ(\\vx^*)^\\top \\mJ(\\vx^*)$ is equal to the smallest value of $\\mD^2$, which is $\\mu^2$. Therefore, we have\n    \\begin{align*}\n        \\sigma_{\\min}(\\mJ(\\vx^*)^\\top\\mJ(\\vx^*)) \\geq \\mu^2.\n    \\end{align*}\n\\end{proof}","sha256_utf8":"605887f6ee65c82163e05653eb8ab9a77f17fe71139cc2d5225111567c082e8e"},{"start":66728,"end":68246,"start_line":1424,"end_line":1444,"raw_tex":"\\begin{proof}\n    The Jacobian holds that\n    \\begin{align*}\n        \\Norm{\\mJ(\\vy)^\\top\\mJ(\\vy) - \\mJ(\\vx)^\\top\\mJ(\\vx)}\n        & = \\Norm{\\mJ(\\vy)^\\top\\mJ(\\vy) - \\mJ(\\vx)^\\top\\mJ(\\vy) + \\mJ(\\vx)^\\top\\mJ(\\vy) - \\mJ(\\vx)^\\top\\mJ(\\vx)}\\\\\n        & \\leq \\Norm{\\left(\\mJ(\\vy)-\\mJ(\\vx)\\right)^\\top\\mJ(\\vy)} + \\Norm{\\mJ(\\vx)^\\top\\left(\\mJ(\\vy) - \\mJ(\\vx)\\right)}\\\\\n        & \\leq \\Norm{\\mJ(\\vy)}\\Norm{\\mJ(\\vy)-\\mJ(\\vx)} + \\Norm{\\mJ(\\vx)}\\Norm{\\mJ(\\vy)-\\mJ(\\vx)}\\\\\n        & \\leq 2L_f\\fH_\\nu\\Norm{\\vy-\\vx}^\\nu,\n    \\end{align*}\n    where the first inequality comes from triangular inequality, the second inequality comes from property of norm, and the last inequality is based on Lemma \\ref{le:b-grad} and Assumption \\ref{asm:holder-g}. \n    \n    For all $j\\in[n]$, the gradient holds that\n    \\begin{align*}\n        \\Norm{\\vg_i(\\vy)\\vg_i(\\vy)^\\top\\!- \\vg_i(\\vx)\\vg_i(\\vx)^\\top}\n        & = \\Norm{\\vg_i(\\vy)\\vg_i(\\vy)^\\top\\!- \\vg_i(\\vx)\\vg_i(\\vy)^\\top + \\vg_i(\\vx)\\vg_i(\\vy)^\\top\\!- \\vg_i(\\vx)\\vg_i(\\vx)^\\top}\\\\\n        & \\leq \\Norm{\\left(\\vg_i(\\vy)-\\vg_i(\\vx)\\right)\\vg_i(\\vy)^\\top} + \\Norm{\\vg_i(\\vx)\\left(\\vg_i(\\vy)-\\vg_i(\\vx)\\right)^\\top}\\\\\n        & \\leq \\Norm{\\vg_i(\\vy)}\\Norm{\\vg_i(\\vy)-\\vg_i(\\vx)} + \\Norm{\\vg_i(\\vx)}\\Norm{\\vg_i(\\vy)-\\vg_i(\\vx)}\\\\\n        & \\leq 2L_f\\fH_\\nu\\Norm{\\vy-\\vx}^\\nu,\n    \\end{align*}\n    where the first inequality comes from triangular inequality, the second inequality comes from property of norm, and the last inequality is based on Lemma \\ref{le:H-con} and \\ref{le:b-grad}.\n\\end{proof}","sha256_utf8":"df00ed31897057272c21246b36d15111bb334dd681f6ceed83f815153c10ec96"},{"start":68299,"end":69309,"start_line":1449,"end_line":1466,"raw_tex":"\\begin{proof}\n    We have\n    \\begin{align*}\n        \\Norm{\\mH^t - \\mJ(\\vx^*)^\\top\\mJ(\\vx^*)} &= \\Norm{\\sum_{i=1}^n\\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top - \\sum_{i=1}^n\\vg_i(\\vx^*)\\vg_i(\\vx^*)^\\top}\\\\\n        & \\leq \\sum_{i=1}^n\\Norm{\\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top - \\vg_i(\\vx^*)\\vg_i(\\vx^*)^\\top}\\\\\n        & \\leq \\sum_{i=1}^n 2L_f\\fH_\\nu\\Norm{\\vz_i^t - \\vx^*}^\\nu,\n    \\end{align*}\n    where the first inequality comes from the triangle inequality and the second inequality is based on Lemma \\ref{le:L-JJ}. \n    Thus, we have\n     \\begin{align*}\n         \\mH^{t}-\\mJ(\\vx^*)^{\\top}\\mJ(\\vx^*)\\succeq - \\sum_{i=1}^n 2L_f\\fH_\\nu\\Norm{\\vz_i^t - \\vx^*}^\\nu \\cdot\\mI, \n     \\end{align*}\n    which implies that\n    \\begin{align*}\n        \\sigma_{\\min}(\\mH^t) \\geq \\sigma_{\\min}(\\mJ(\\vx^*)^\\top\\mJ(\\vx^*)) - \\sum_{i=1}^n 2L_f\\fH_\\nu\\Norm{\\vz_i^t - \\vx^*}^\\nu = \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^t - \\vx^*}^\\nu,\n    \\end{align*}\n    where the last step is based on Proposition \\ref{prop:b-JJ-sing}.\n\\end{proof}","sha256_utf8":"046b042b56968b6cf2f1365a29dc51f3fa90bdca400e954fa5834e6084b72219"},{"start":69369,"end":69661,"start_line":1470,"end_line":1474,"raw_tex":"We first show the update\n\\begin{align*}\n\\mG^{t+1} =\\mG^{t} - \\mG^{t}\\mU^t(\\mI + (\\mV^t)^\\top\\mG^{t}\\mU^t)^{-1}(\\mV^t)^\\top\\mG^{t}    \n\\end{align*}\nin \\IGN~method (Line \\ref{line:update-G} of Algorithm \\ref{alg:IGN-1}) is well-defined if the matrices $\\mH^t$ and $\\mH^{t+1}$ are non-singular.\n","sha256_utf8":"db0d7929d42f55269076b6517ac47900fcfbf2f347716c6005f88eb31b9574fa"},{"start":69662,"end":70230,"start_line":1476,"end_line":1489,"raw_tex":"\\begin{lem}\n\\label{le:via-update}\n    Following the setting of Theorem \\ref{thm:holder-IGN-1}, if the matrices $\\mH^t$ and $\\mH^{t+1}$ are non-singular, then the matrix $\\mI + {\\mV^t}^\\top\\mG^{t}\\mU^t$ is also non-singular, where\n    \\begin{align*}\n    \\mU^t = \\begin{bmatrix}\n        - \\vg_{i_t}(\\vz_{i_t}^t) \\!& \n        \\vg_{i_t}(\\vx^{t+1})\n    \\end{bmatrix}\\in\\BR^{d\\times 2},~~\n    \\mV^t = \\begin{bmatrix}\n        \\vg_{i_t}(\\vz_{i_t}^t) \\!& \n        \\vg_{i_t}(\\vx^{t+1})\n    \\end{bmatrix}\\in\\BR^{d\\times 2}~~\\text{and}~~i_t ={t\\%n} + 1.\n    \\end{align*}\n\\end{lem}","sha256_utf8":"5da9f68b009678f4bedc639f6b4b40c28b38edc9cb33b7d2123678d7e5953a4b"},{"start":70231,"end":71061,"start_line":1490,"end_line":1504,"raw_tex":"\\begin{proof}\n    The recursion of $\\mH^t$ and the definition of $\\mU^t$ and $\\mV^t$ imply\n    \\begin{align*}\n        \\mH^{t+1} = \\mH^t - \\vg_{i_t}(\\vz_{i_t}^t)\\vg_{i_t}(\\vz_{i_t}^t)^\\top + \\vg_{i_t}(\\vx^{t+1})\\vg_{i_t}(\\vx^{t+1})^\\top = \\mH^t + \\mU^t{\\mV^t}^\\top.\n    \\end{align*}\n    Since we assume matrices $\\mH^t$ and $\\mH^{t+1}$ are non-singular, applying the matrix determinant lemma \\cite[Section 9.1.2]{petersen2008matrix} on above equation leads to\n    \\begin{align*}\n        \\det(\\mH^{t+1})=\\det(\\mH^t + \\mU^t{\\mV^t}^\\top) = \\det(\\mI + {\\mV^t}^\\top(\\mH^t)^{-1}\\mU^t)\\det(\\mH^t).\n    \\end{align*}\n    Then the definition $\\mG^t={\\mH^t}^{-1}$ implies\n    \\begin{align*}\n        \\det(\\mI + {\\mV^t}^\\top\\mG^t\\mU^t) = \\det(\\mI + {\\mV^t}^\\top{\\mH^t}^{-1}\\mU^t) \\neq 0\n    \\end{align*}\n    which finish the proofs.\n\\end{proof}","sha256_utf8":"1ce8784f90b882fe231fe7bab7304720c9a725e17b07a6c5b9e39e2985beac3f"},{"start":71063,"end":71184,"start_line":1506,"end_line":1506,"raw_tex":"Then we show the non-singular assumption on $\\{\\mH^j\\}_{j=0}^t$ can upper bound the distance $\\Norm{\\vx^{t+1} - \\vx^*}$.\n","sha256_utf8":"ba4dce1de9ae0a0680e6706f7361d04a1e45aa079bd366302cf5692cda23aca5"},{"start":71185,"end":71596,"start_line":1508,"end_line":1517,"raw_tex":"\\begin{lem}\\label{le:main-iter-1} \nUnder Assumptions \\ref{asm:L-f} and \\ref{asm:holder-g}, we\nassume matrices $\\{\\mH^j\\}_{j=0}^t$ are non-singular and run \\IGN~(Algorithm \\ref{alg:IGN-1}), then it holds\n\\begin{align*}\n    \\Norm{\\vx^{t+1} - \\vx^*}\n     \\leq \\frac{L_f\\fH_\\nu}{1 \n     +\\nu}\\Norm{\\mG^t}\\sum_{i=1}^n\\Norm{\\vz_i^t-\\vx^*}^{1+\\nu},\n    \\end{align*}\n    where $\\mG^t=\\left(\\mH^t\\right)^{-1}$.\n\\end{lem}","sha256_utf8":"2294eb786446e0fad888d9600fd8c3e53d45b3865276373e1b72f62ea5d8f17d"},{"start":71597,"end":72830,"start_line":1518,"end_line":1535,"raw_tex":"\\begin{proof}\n    Subtracting the term $\\vx^*$ on both sides of equation (\\ref{eq:ign-update-0}), we have\n    \\begin{align*}\n    \\vx^{t+1} - \\vx^*\n    & = \\mG^t\\left(\\sum_{i=1}^n \\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top(\\vz_i^t-\\vx^*) - \\sum_{i=1}^n f_i(\\vz_i^t)\\vg_i(\\vz_i^t)\\right)\\\\\n    & = \\mG^t\\left(\\sum_{i=1}^n \\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top(\\vz_i^t-\\vx^*) - \\sum_{i=1}^n f_i(\\vz_i^t)\\vg_i(\\vz_i^t) + \\sum_{i=1}^n f_i(\\vx^*)\\vg_i(\\vz_i^t)\\right)\\\\\n    & = \\mG^t\\sum_{i=1}^n\\left( \\vg_i(\\vz_i^t)^\\top(\\vz_i^t-\\vx^*) - f_i(\\vz_i^t) + f_i(\\vx^*)\\right)\\vg_i(\\vz_i^t).\n    \\end{align*}\n\n    Taking the norm on the both sides of above results, we have\n    \\begin{align*}\n    \\Norm{\\vx^{t+1} - \\vx^*}\n    & = \\Norm{\\mG^t\\sum_{i=1}^n\\left( \\vg_i(\\vz_i^t)^\\top(\\vz_i^t-\\vx^*) - f_i(\\vz_i^t) + f_i(\\vx^*)\\right)\\vg_i(\\vz_i^t)}\\\\\n    & \\leq \\Norm{\\mG^t}\\Norm{\\sum_{i=1}^n\\left( \\vg_i(\\vz_i^t)^\\top(\\vz_i^t-\\vx^*) - f_i(\\vz_i^t) + f_i(\\vx^*)\\right)\\vg_i(\\vz_i^t)}\\\\\n    & \\leq \\frac{L_f\\fH_\\nu}{1 + \\nu}\\Norm{\\mG^t}\\sum_{i=1}^n\\Norm{\\vz_i^t-\\vx^*}^{1+\\nu}\n    \\end{align*}    \n    where the first inequality comes from the property of matrix norm, the second inequality is based on Lemma \\ref{le:power-bound} and \\ref{le:b-grad}.\n\\end{proof}","sha256_utf8":"939bc550fb5c4065c4038401ee9fa3eec647c8f31d6d9e07997489ffa877712f"},{"start":72834,"end":73122,"start_line":1539,"end_line":1540,"raw_tex":"We split the results of Theorem \\ref{thm:holder-IGN-1} into two parts (i.e., Theorem \\ref{thm:holder-IGN-1-1} and \\ref{thm:holder-IGN-1-2}) and provide their proofs as follows.\nOur analysis is based on the properties of our the auxiliary sequence constructed in Section \\ref{sec:series}.\n","sha256_utf8":"e753af49daff86bc0b6d1b99ce67c178875c16159dd1310f0e775c904258ef22"},{"start":73124,"end":73855,"start_line":1543,"end_line":1557,"raw_tex":"\\begin{thm}\n\\label{thm:holder-IGN-1-1}\n    Under the Assumption \\ref{asm:L-f}, \\ref{asm:holder-g} and \\ref{asm:b-J-sing}, we run \\IGN~(Algorithm \\ref{alg:IGN-1}) with initialization $\\vx^0\\in\\BR^d$ and~$\\mH^0=\\mJ(\\vx^0)^\\top\\mJ(\\vx^0)$ such that\n    \\begin{align*}\n    \\Norm{\\vx^0 - \\vx^*}\\leq \\left(\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\right)^{{1}/{\\nu}},\n    \\end{align*} \n    then it holds \n    \\begin{align*}\n    \\sigma_{\\min}(\\mI + (\\mV^t)^\\top (\\mH^t)^{-1}\\mU^t) > 0,\\quad\n    \\mH^t \\succeq \\frac{\\mu^2}{2}\\mI\\quad\n    \\text{and}\\quad\n    \\Norm{\\vx^{t}-\\vx^*}\\leq a_{t+1}(n, \\nu)\\Norm{\\vx^0-\\vx^*}\n    \\end{align*}\n    for all $t \\geq 0$, where sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$ is defined in equation (\\ref{eq:series}). \n\\end{thm}","sha256_utf8":"3097733d3cea1d2d02f7ec6de0b4b75448b4fa8d99b9f8f70211cbf84e751f5d"},{"start":73856,"end":83677,"start_line":1558,"end_line":1768,"raw_tex":"\\begin{proof}\n\nWe first show\n\\begin{align}\\label{eq:Hxtxstar0}\n\\mH^t \\succeq \\frac{\\mu^2}{2}\\mI\\qquad\n\\text{and}\\qquad\n\\Norm{\\vx^{t}-\\vx^*}\\leq a_{t+1}(n, \\nu)\\Norm{\\vx^0-\\vx^*}\n\\end{align}\nholds for all $t\\geq 0$. We split the proof of results (\\ref{eq:Hxtxstar0}) into the following three parts.\n\n\\textbf{Part I:} For $t=0$, \nthe initialization and the fact $a_0=1$ leads to \n\\begin{align*}\n \\Norm{\\vx^0 - \\vx^*} = a_0(n, \\nu)\\Norm{\\vx^0 - \\vx^*}.\n\\end{align*}\n\n\n\\textbf{Part II:} For all $t=0,\\cdots, n-1$, we use induction to prove the results of \n\\begin{align}\\label{eq:Hxtxstar}\n\\mH^t \\succeq \\frac{\\mu^2}{2}\\mI\\qquad\n\\text{and}\\qquad\n\\Norm{\\vx^{t+1}-\\vx^*}\\leq a_{t+1}(n, \\nu)\\Norm{\\vx^0-\\vx^*}.\n\\end{align}\nFor the induction base, we can apply Lemma \\ref{le:lb-Ht-new-1} to verify\n\\begin{align*}\n    \\sigma_{\\min}(\\mH^0) &\\geq  \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^0 - \\vx^*}^\\nu\\\\\n    & = \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vx^0 - \\vx^*}^\\nu\\\\\n    & \\geq \\mu^2 - 2L_f\\fH_\\nu n \\frac{\\mu^2}{4L_f\\fH_\\nu n}\\\\\n    & = \\mu^2 - \\frac{\\mu^2}{2}\\\\\n    & = \\frac{\\mu^2}{2}.\n\\end{align*}\n\n\nThis implies\n\\begin{align}\\label{eq:boundG0}\n\\mH^0 \\succeq \\frac{\\mu^2}{2} \\qquad \\text{and}\\qquad\n\\Norm{\\mG^0} = \\Norm{(\\mH^0)^{-1}} \\leq \\frac{2}{\\mu^2}.\n\\end{align}\nAccording to Lemma \\ref{le:main-iter-1}, we have\n\\begin{align*}\n    \\Norm{\\vx^1 - \\vx^*} \n    &\\leq \\frac{L_f\\fH_\\nu}{1+\\nu}\\Norm{\\mG^0}\\sum_{i=1}^n\\Norm{\\vz_i^0-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{L_f\\fH_\\nu}{1+\\nu}\\cdot\\frac{2}{\\mu^2}\\cdot\\sum_{i=1}^n\\Norm{\\vz_i^0-\\vx^*}^{1+\\nu}\\\\\n    & = \\frac{L_f\\fH_\\nu}{1+\\nu}\\cdot\\frac{2}{\\mu^2}\\cdot n\\Norm{\\vx^0-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{nL_f\\fH_\\nu}{1+\\nu}\\cdot\\frac{2}{\\mu^2}\\cdot\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\Norm{\\vx^0-\\vx^*}\\\\\n    & = \\frac{1}{2(1+\\nu)}\\Norm{\\vx^0-\\vx^*}\\\\\n    & = a_1(n, \\nu)\\Norm{\\vx^0 - \\vx^*},\n\\end{align*}\nwhere the first inequality is based on equation (\\ref{eq:boundG0}) and the second inequality is based on initial condition. \nTherefore, the induction base holds\n\nFor the induction step, we assume\n\\begin{align*}\n    \\mH^j \\succeq \\frac{\\mu^2}{2} \\mI \\qquad\\text{and}\\qquad\\Norm{\\vx^{j+1} - \\vx^*} \\leq a_{j+1}(n, \\nu)\\Norm{\\vx^0 - \\vx^*}\n\\end{align*}\nhold for all $j = 2, \\cdots, t-1$ such that $t \\leq n - 1$. Therefore, the update (\\ref{eq:update-1}) means\n\\begin{align}\\label{eq:vziii}\n    \\vz_i^{t}=\n    \\begin{cases}\n        \\vx^i,~~~~&1 \\leq i\\leq t,\\\\\n        \\vx^0,~~~~&t < i \\leq n. \n    \\end{cases}\n\\end{align}\nThe induction hypothesis leads to\n\\begin{align*}\n    \\Norm{\\vx^j-\\vx^*}^{\\nu}\\leq (a_j(n, \\nu))^{\\nu}\\Norm{\\vx^0-\\vx^*}^{\\nu} \\leq  \\Norm{\\vx^0-\\vx^*}^{\\nu}\\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n}, \n\\end{align*}\nfor $j = 1, \\cdots, t-1$, where the second is based on Lemma \\ref{le:s-mono-000} and the third comes from the initial condition. \nCombining with the result of (\\ref{eq:vziii}), we achive\n\\begin{align*}\n    \\Norm{\\vz_i^{t} - \\vx^*}^{\\nu} \\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n}.\n\\end{align*}\n\nAccording to Lemma \\ref{le:lb-Ht-new-1}, we have\n\\begin{align*}\n    \\sigma_{\\min}(\\mH^{t}) &\\geq  \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^{t} - \\vx^*}^\\nu\\\\\n    & \\geq \\mu^2 - 2L_f\\fH_\\nu n \\frac{\\mu^2}{4L_f\\fH_\\nu n}\\\\\n    & = \\mu^2 - \\frac{\\mu^2}{2}\\\\\n    & = \\frac{\\mu^2}{2},\n\\end{align*}\n\nwhere the second inequality comes from the initial condition.\nTherefore, we have\n\\begin{align*}\n    \\mH^{t}  \\succeq \\frac{\\mu^2}{2}\\mI \\qquad \\text{and}\\qquad\n    \\Norm{\\mG^{t}}  = \\Norm{(\\mH^{t})^{-1}} \\leq \\frac{2}{\\mu^2}.\n\\end{align*}\nAccording to Lemma \\ref{le:main-iter-1}, we have\n\\begin{align*}\n    \\Norm{\\vx^{t+1} - \\vx^*} \n    & \\leq \\frac{L_f\\fH_\\nu }{1+\\nu}\\Norm{\\mG^{t}}\\sum_{i=1}^n\\Norm{\\vz_i^{t}-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{L_f\\fH_\\nu }{1+\\nu}\\frac{2}{\\mu^2}\\sum_{i=1}^n\\Norm{\\vz_i^{t}-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu}{(1+\\nu)\\mu^2}\\left(\\sum_{j=1}^{t}\\Norm{\\vx^{j}-\\vx^*}^{1+\\nu} + (n - t)\\Norm{\\vx^0 - \\vx^*}^{1+\\nu}\\right)\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\left(\\sum_{j=1}^{t} (a_j(n, \\nu))^{1+\\nu}\\Norm{\\vx^0 - \\vx^*}^{1+\\nu} + (n - t) \\Norm{\\vx^0 - \\vx^*}^{1+\\nu}\\right)\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\left(\\sum_{j=1}^{t}(a_j(n, \\nu))^{1+\\nu} + n - t\\right)\\Norm{\\vx^0 - \\vx^*}\\\\\n    & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=1}^{t}(a_j(n, \\nu))^{1+\\nu} + n - t\\right)\\Norm{\\vx^0 - \\vx^*}\\\\\n    & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=0}^{t}(a_j(n, \\nu))^{1+\\nu} + n - t - 1\\right)\\Norm{\\vx^0 - \\vx^*}\\\\\n    & = a_{t+1}(n, \\nu)\\Norm{\\vx^0 - \\vx^*},\n\\end{align*}\nwhere the last equality comes from the fact $a_0(n, \\nu)=1$. \nTherefore, we finish the induction.\n\n\n\\textbf{Part III:} For all $t \\geq n$, we use induction to prove \n\\begin{align*}\n\\mH^t\\succeq(\\mu^2/2)\\mI\\qquad\\text{and}\\qquad\\Norm{\\vx^{t+1} - \\vx^*}\\leq a_{t+1}(n, \\nu)\\Norm{\\vx^0 - \\vx^*}.    \n\\end{align*}\nFor the induction base, we can verify that it holds (from the result of Part II) \n\\begin{align*}\n\\mH^j \\succeq \\frac{\\mu^2}{2}\\mI \\qquad \\text{for all~~} j=0,\\dots,n - 1,\n\\end{align*}\nand\n\\begin{align*}\n\\Norm{\\vx^j-\\vx^*}\\leq a_j(n, \\nu)\\Norm{\\vx^0-\\vx^*} \\qquad \\text{for all~~} j=1,\\dots,n.\n\\end{align*}\nThen we have\n\\begin{align*}\n    \\Norm{\\vx^j-\\vx^*}^{\\nu}\\leq (a_j(n, \\nu))^{\\nu}\\Norm{\\vx^0-\\vx^*}^{\\nu} \\leq \\Norm{\\vx^0-\\vx^*}^{\\nu} \\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n}, \\quad \\text{for all~~} j=1,\\dots,n,\n\\end{align*}\nwhere the second inequality is based on Lemma \\ref{le:s-mono-000} and the third inequality is based on the initial condition.\n\nFrom Eq.~\\ref{eq:update-1}, we have\n\\begin{align*}\n    \\vz_i^{n}=\\vx^i \\qquad\\text{for all~~} i\\in[n].\n\\end{align*}\n\nTherefore, we have\n\\begin{align*}\n    \\Norm{\\vz_i^{n} - \\vx^*}^{\\nu} \\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n}, \\qquad\\text{for all~~} i\\in[n].\n\\end{align*}\nAccording to Lemma \\ref{le:lb-Ht-new-1}, we have\n\\begin{align*}\n    \\sigma_{\\min}(\\mH^{n}) &\\geq  \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^{n} - \\vx^*}^\\nu\\\\\n    & \\geq \\mu^2 - 2L_f\\fH_\\nu n \\frac{\\mu^2}{4L_f\\fH_\\nu n}\\\\\n    & \\geq \\mu^2 - \\frac{\\mu^2}{2} = \\frac{\\mu^2}{2},\n\\end{align*}\n\nwhich implies\n\\begin{align*}\n    \\mH^{n}  \\succeq \\frac{\\mu^2}{2}\\mI \\qquad\\text{and}\\qquad\n    \\Norm{\\mG^{n}}  = \\Norm{(\\mH^{n})^{-1}} \\leq \\frac{2}{\\mu^2}.\n\\end{align*}\nAccording to Lemma \\ref{le:main-iter-1}, we have\n\\begin{align*}\n    \\Norm{\\vx^{n+1} - \\vx^*}\n    & \\leq \\frac{L_f\\fH_\\nu }{1+\\nu}\\Norm{\\mG^n}\\sum_{i=1}^n\\Norm{\\vz_i^n-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\sum_{i=1}^n\\Norm{\\vz_i^n-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\left(\\sum_{j=1}^n (a_j(n, \\nu))^{1+\\nu}\\Norm{\\vx^0 - \\vx^*}^{1+\\nu}\\right)\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\left(\\sum_{j=1}^{n}(a_j(n, \\nu))^{1+\\nu}\\right)\\Norm{\\vx^0-\\vx^*}\\\\\n    & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=1}^{n}(a_j(n, \\nu))^{1+\\nu}\\right)\\Norm{\\vx^0-\\vx^*}\\\\\n    & = a_{n+1}(n, \\nu)\\Norm{\\vx^0-\\vx^*}.\n\\end{align*}\nHence, we have shown the induction base holds. \n\nFor the induction step, we assume\n\\begin{align*}\n    \\mH^j \\succeq \\frac{\\mu^2}{2} \\mI \\qquad\\text{and}\\qquad\\Norm{\\vx^{j+1} - \\vx^*} \\leq a_{j+1}(n, \\nu)\\Norm{\\vx^0 - \\vx^*}\n\\end{align*}\nholds for all $j = n+1, \\cdots, t-1$ such that $t \\geq n + 2$. Combining results of Part I and II, we have\n\n\\begin{align*}\n\\Norm{\\vx^j-\\vx^*}\\leq a_j(n, \\nu)\\Norm{\\vx^0-\\vx^*} \\qquad \\text{for all}~~ j=0,\\dots,t,\n\\end{align*}\nwhich implies\n\\begin{align*}\n    \\Norm{\\vx^j-\\vx^*}^{\\nu}\\leq (a_j(n, \\nu))^{\\nu}\\Norm{\\vx^0-\\vx^*}^{\\nu} \\leq \\Norm{\\vx^0-\\vx^*}^{\\nu} \\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n}, \\qquad\\text{for all}~~j=1,\\dots,t,\n\\end{align*}\nwhere the second inequality is based on Lemma \\ref{le:s-mono-000} and the last inequality is based on the condition condition.\n\n\nThe update~(\\ref{eq:update-1}) means the points $\\{\\vz_i^{t}\\}_{i=1}^n$ can be written as $\\{\\vx^{t+1-n}, \\cdots, \\vx^{t}\\}$, which implies\n\\begin{align*}\n    \\max\\{\\Norm{\\vz_1^{t}-\\vx^*}, \\cdots, \\Norm{\\vz_n^{t}-\\vx^*}\\}=\\max\\{\\Norm{\\vx^{t+1-n}-\\vx^*}, \\cdots, \\Norm{\\vx^{t}-\\vx^*}\\}.\n\\end{align*}\nTherefore, we have\n\\begin{align*}\n    \\Norm{\\vz_i^{n} - \\vx^*}^{\\nu} \\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n} \\qquad \\text{for all}~~i=1,\\dots,n.\n\\end{align*}\n\nCombing with Lemma \\ref{le:lb-Ht-new-1}, we have\n\\begin{align*}\n    \\sigma_{\\min}(\\mH^{t}) &\\geq  \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^{t} - \\vx^*}^\\nu\\\\\n    & \\geq \\mu^2 - 2L_f\\fH_\\nu n \\frac{\\mu^2}{4L_f\\fH_\\nu n}\\\\\n    & = \\mu^2 - \\frac{\\mu^2}{2} = \\frac{\\mu^2}{2}.\n\\end{align*}\nTherefore, we achieve\n\\begin{align*}\n    \\mH^{t}  \\succeq \\frac{\\mu^2}{2}\\mI \\qquad\\text{and}\\qquad\n    \\Norm{\\mG^{t}} & = \\Norm{(\\mH^{t})^{-1}} \\leq \\frac{2}{\\mu^2}.\n\\end{align*}\nAccording to Lemma \\ref{le:main-iter-1}, we have\n\\begin{align*}\n    \\Norm{\\vx^{t+1} - \\vx^*}\n    & \\leq \\frac{L_f\\fH_\\nu }{1+\\nu}\\Norm{\\mG^{t}}\\sum_{i=1}^n\\Norm{\\vz_i^{t}-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\sum_{i=1}^n\\Norm{\\vz_i^{t}-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\left(\\sum_{j=t-n+1}^{t} (a_j(n, \\nu))^{1+\\nu}\\Norm{\\vx^0 - \\vx^*}^{1+\\nu}\\right)\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\left(\\sum_{j=t-n+1}^{t} (a_j(n, \\nu))^{1+\\nu}\\right)\\Norm{\\vx^0 - \\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\left(\\sum_{j=t-n+1}^{t+1}a_j(n, \\nu)^{1+\\nu}\\right)\\Norm{\\vx^0 - \\vx^*}\\\\\n    & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=t-n+2}^{t+1}(a_j(n, \\nu))^{1+\\nu}\\right)\\Norm{\\vx^0 - \\vx^*}\\\\\n    & = a_{t+1}(n, \\nu)\\Norm{\\vx^0 - \\vx^*}.\n\\end{align*}\nHence, we finish the induction.\n\n\nCombining results of Part I, II and III completes the proof of (\\ref{eq:Hxtxstar0}).\n\nSince the non-singularity of $\\mH^t$ and $\\mH^{t+1}$ has been verified by result (\\ref{eq:Hxtxstar0}), we can apply Lemma \\ref{le:via-update} to achieve\n\\begin{align*}\n    \\sigma_{\\min}(\\mI + (\\mV^t)^\\top(\\mH^t)^{-1}\\mU^t) > 0.\n\\end{align*}\n\\end{proof}","sha256_utf8":"e95288a2352cd48a23d368c5000f7077d5acd3588dfd3d91095d8ca370cb8471"},{"start":83680,"end":84566,"start_line":1771,"end_line":1792,"raw_tex":"\\begin{thm}\n\\label{thm:holder-IGN-1-2}\nWe define the sequence $\\{r_t\\}_{t\\geq 0}$ such that\n\\begin{align*}\n    r_t\\triangleq\\begin{cases}\n    \\max\\{\\Norm{\\vx^0 - \\vx^*}, 1\\},~~~~&t=0,\\\\[0.2cm]\n    a_t(n, \\nu)r_0,~~~~& t\\geq 1,\\\\\n\\end{cases}\n\\end{align*}\nwhere the sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$ is defined by equation (\\ref{eq:series}). \nUnder the Assumptions \\ref{asm:L-f}, \\ref{asm:holder-g} and \\ref{asm:b-J-sing}, running \\IGN~(Algorithm \\ref{alg:IGN-1}) with initial condition shown in Theorem \\ref{thm:holder-IGN-1-1}, we have\n    \\begin{align}\n    \\label{eq:r_t_recursion}\n    \\Norm{\\vx^t - \\vx^*} \\leq r_t\n    \\qquad\\text{and}\\qquad\n    r_{t+1}\\leq  c^{(1+\\nu)^{\\left(\\floor{\\frac{t}{n}}-1\\right)}}r_t  \n    \\end{align}\n    for all $t\\geq n$, where\n    \\begin{align*}\n    c = 1- \\frac{1}{n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right).\n    \\end{align*}\n\\end{thm}","sha256_utf8":"1e083e3d42a67f1dbebbce58ff1e110c5a0fe200d276ae8c2c2b8bc458adb4b7"},{"start":84568,"end":85525,"start_line":1794,"end_line":1815,"raw_tex":"\\begin{proof}\n    The definition of $\\{r_t\\}_{t\\geq 0}$ leads to \n    \\begin{align*}\n        r_0 = \\max\\{\\Norm{\\vx^0 - \\vx^*}, 1\\} \\geq \\Norm{\\vx^0 - \\vx^*}.\n    \\end{align*}\n    According to Theorem \\ref{thm:holder-IGN-1-1}, we have \n    \\begin{align*}\n    \\Norm{\\vx^t - \\vx^*} \\leq a_{t}(n, \\nu)\\Norm{\\vx^0 - \\vx^*} \\leq a_t(n, \\nu)r_0 = r_t.\n    \\end{align*}\n    According to Lemma \\ref{le:super-con}, we have\n    \\begin{align*}\n         a_{t+1}(n, \\nu)\\leq  c^{(1+\\nu)^{\\left(\\floor{\\frac{t}{n}}-1\\right)}}a_t(n, \\nu) \\qquad \\text{for all}~~t \\geq n.\n    \\end{align*}\n    Thus, achieve\n    \\begin{align*}\n        r_{t+1} = a_{t+1}(n, \\nu)r_0\\leq  c^{(1+\\nu)^{\\left(\\floor{\\frac{t}{n}}-1\\right)}}a_t(n, \\nu)r_0 =c^{(1+\\nu)^{\\left(\\floor{\\frac{t}{n}}-1\\right)}}r_t  \\qquad \\text{for all}~~t \\geq n,\n    \\end{align*}\n    where\n    \\begin{align*}\n        c = 1- \\frac{1}{n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right).\n    \\end{align*}\n\\end{proof}","sha256_utf8":"20f875a8b291b6f948ddce9ab19c4adbb097419a792b6883b0fa7811aad686e5"},{"start":85527,"end":85671,"start_line":1817,"end_line":1817,"raw_tex":"Combining the results of Theorem \\ref{thm:holder-IGN-1-1} and  \\ref{thm:holder-IGN-1-2}, we finish the proof of Theorem \\ref{thm:holder-IGN-1}.\n","sha256_utf8":"99dc24f5965992069bda55d9811061bcf1554be69e9d51c25dd28f9020416214"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"IGN局部超线性收敛","source_units":[{"start":28922,"end":29762,"start_line":675,"end_line":691,"raw_tex":"\\begin{thm}\n\\label{thm:holder-IGN-1}\n{\n    Under Assumptions \\ref{asm:L-f}, \\ref{asm:holder-g} and \\ref{asm:b-J-sing}, running \\IGN~(Algorithm \\ref{alg:IGN-1}) with initialization $\\vx^0\\in\\BR^d$, $\\mH^0 = \\mJ(\\vx^0)^\\top\\mJ(\\vx^0)$ and $\\mG^0=(\\mH^0)^{-1}$ such that\n    \\begin{align*}\n    \\Norm{\\vx^0 - \\vx^*}\\leq \\left(\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\right)^{1/\\nu},\n    \\end{align*} \n    we have $\\mH^t \\succeq (\\mu^2/2)\\mI$ and $\\sigma_{\\min}(\\mI + (\\mV^t)^\\top\\mG^{t}\\mU^t)>0$ for all $t\\geq 0$.\n    Additionally, there exists sequence $\\{r^t\\}$ such that $\\Norm{\\vx^t-\\vx^*}\\leq r_t$ and it holds\n\n    \\begin{align*}\n     r_{t+1}\\leq  c^{(1+\\nu)^{\\left(\\floor{{t}/{n}}-1\\right)}}r_t \\qquad\\text{with}\\qquad c = 1- \\frac{1}{n}\\left(1-\\left(\\frac{1}{2 \n  (1+\\nu)}\\right)^{(1+\\nu)}\\right)\n    \\end{align*}\n    for all $t\\geq n$.\n}\n\\end{thm}","sha256_utf8":"d5ad551c19a1068cbb94534d6ffa32d5896df14f3d212c46816a046a7de1da2e"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"主定理完整组合证明","source_units":[{"start":69369,"end":69661,"start_line":1470,"end_line":1474,"raw_tex":"We first show the update\n\\begin{align*}\n\\mG^{t+1} =\\mG^{t} - \\mG^{t}\\mU^t(\\mI + (\\mV^t)^\\top\\mG^{t}\\mU^t)^{-1}(\\mV^t)^\\top\\mG^{t}    \n\\end{align*}\nin \\IGN~method (Line \\ref{line:update-G} of Algorithm \\ref{alg:IGN-1}) is well-defined if the matrices $\\mH^t$ and $\\mH^{t+1}$ are non-singular.\n","sha256_utf8":"db0d7929d42f55269076b6517ac47900fcfbf2f347716c6005f88eb31b9574fa"},{"start":71063,"end":71184,"start_line":1506,"end_line":1506,"raw_tex":"Then we show the non-singular assumption on $\\{\\mH^j\\}_{j=0}^t$ can upper bound the distance $\\Norm{\\vx^{t+1} - \\vx^*}$.\n","sha256_utf8":"ba4dce1de9ae0a0680e6706f7361d04a1e45aa079bd366302cf5692cda23aca5"},{"start":72834,"end":73122,"start_line":1539,"end_line":1540,"raw_tex":"We split the results of Theorem \\ref{thm:holder-IGN-1} into two parts (i.e., Theorem \\ref{thm:holder-IGN-1-1} and \\ref{thm:holder-IGN-1-2}) and provide their proofs as follows.\nOur analysis is based on the properties of our the auxiliary sequence constructed in Section \\ref{sec:series}.\n","sha256_utf8":"e753af49daff86bc0b6d1b99ce67c178875c16159dd1310f0e775c904258ef22"},{"start":85527,"end":85671,"start_line":1817,"end_line":1817,"raw_tex":"Combining the results of Theorem \\ref{thm:holder-IGN-1-1} and  \\ref{thm:holder-IGN-1-2}, we finish the proof of Theorem \\ref{thm:holder-IGN-1}.\n","sha256_utf8":"99dc24f5965992069bda55d9811061bcf1554be69e9d51c25dd28f9020416214"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"拆分稳定性与收敛率","source_units":[{"start":72834,"end":73122,"start_line":1539,"end_line":1540,"raw_tex":"We split the results of Theorem \\ref{thm:holder-IGN-1} into two parts (i.e., Theorem \\ref{thm:holder-IGN-1-1} and \\ref{thm:holder-IGN-1-2}) and provide their proofs as follows.\nOur analysis is based on the properties of our the auxiliary sequence constructed in Section \\ref{sec:series}.\n","sha256_utf8":"e753af49daff86bc0b6d1b99ce67c178875c16159dd1310f0e775c904258ef22"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"合并两个辅助定理","source_units":[{"start":85527,"end":85671,"start_line":1817,"end_line":1817,"raw_tex":"Combining the results of Theorem \\ref{thm:holder-IGN-1-1} and  \\ref{thm:holder-IGN-1-2}, we finish the proof of Theorem \\ref{thm:holder-IGN-1}.\n","sha256_utf8":"99dc24f5965992069bda55d9811061bcf1554be69e9d51c25dd28f9020416214"}]},{"id":"S005","semantic_type":"problem_setting","kind":"problem_setting","title":"非线性方程组","source_units":[{"start":11916,"end":12172,"start_line":418,"end_line":423,"raw_tex":"We study the problem of solving the system of nonlinear equations\n\\begin{align}\n\\label{eq:ori_prob}\n    \\vf(\\vx) = \\vzero,\n\\end{align}\nwhere the nonlinear vector function $\\vf:\\BR^d\\to\\BR^n$ is Lipschitz continuous and its Jacobian is H\\\"older continuous.\n","sha256_utf8":"38a92d0e0f14d2ce319e7148f3c6649d1ce9fce3be2798d58462fcf6664a6f4d"}]},{"id":"S006","semantic_type":"notation","kind":"notation","title":"分量梯度与矩阵记号","source_units":[{"start":17270,"end":18153,"start_line":471,"end_line":477,"raw_tex":"We let $[n]=\\{1,\\dots,n\\}$ and use the notation $t\\%n$ to present the remainder of $t$ divided by $n$.\nWe denote $\\ve_i\\in\\BR^n$ as the $i$-th standard basis vector of the $n$-dimensional Euclidean space for all $i\\in [n]$. \nWe use $\\Norm{\\cdot}$ to represent the Euclidean norm for a given vector and the spectral norm for a given matrix. \nMoreover, we use the notation $\\sigma_{\\min}(\\cdot)$ to represent the smallest singular value for a given matrix.\n\nFor the system of nonlinear equations (\\ref{eq:ori_prob}), we partition the vector function $\\vf:\\BR^d\\to\\BR^n$ at $\\vx\\in\\BR^d$ as~$\\vf(\\vx)=[f_1(\\vx),\\dots, f_n(\\vx)]^\\top\\in\\BR^n$, where $f_i:\\BR^d\\to\\BR$.\nWe also denote the gradient of component $f_i(\\cdot)$ at $\\vx\\in\\BR^d$ as $\\vg_i(\\vx)=\\nabla f_i(\\vx)$, and we organize the corresponding Jacobian as~$\\mJ(\\vx)=[\\vg_1(\\vx), \\cdots, \\vg_n(\\vx)]^\\top\\in\\BR^{n\\times d}$.\n","sha256_utf8":"36dc848d8ad0baf54753a12731412452d6328bc1d67006d999e00c9649b5040e"}]},{"id":"S007","semantic_type":"assumption","kind":"assumption","title":"函数Lipschitz连续","source_units":[{"start":18284,"end":18570,"start_line":482,"end_line":489,"raw_tex":"\\begin{asm}\n\\label{asm:L-f}\n    We suppose the vector function $\\vf:\\BR^d\\to\\BR^n$ is Lipschitz continuous, i.e., there exists constant $L_f>0$ such that\n    \\begin{align}\n        \\Norm{\\vf(\\vx)-\\vf(\\vy)} \\leq L_f\\Norm{\\vx - \\vy}\n    \\end{align}\n    for all $\\vx,\\vy\\in\\BR^d$.\n\\end{asm}","sha256_utf8":"b553951b911363ec89a3f82bac5eb97222071f15945df566b6897e21efa0aefe"}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","title":"雅可比Hölder连续","source_units":[{"start":18571,"end":18905,"start_line":490,"end_line":496,"raw_tex":"\\begin{asm}\\label{asm:holder-g}\n    We suppose the Jacobian $\\mJ:\\BR^d\\to\\BR^{n\\times d}$  is $\\nu$-H\\\"older continuous for some $\\nu\\in(0,1]$, i.e., there exists constant $\\fH_\\nu>0$ such that\n    \\begin{align}\n        \\Norm{\\mJ(\\vx)-\\mJ(\\vy)} \\leq \\fH_\\nu\\Norm{\\vx - \\vy}^\\nu\n    \\end{align}\n    for all $\\vx,\\vy\\in\\BR^d$.\n\\end{asm}","sha256_utf8":"110834d58c33b0fa4bc093f9125891ec7fd54ff157c1c5c28ed6e1de35b3e010"}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","title":"非退化方程解","source_units":[{"start":18906,"end":19218,"start_line":497,"end_line":504,"raw_tex":"\\begin{asm}\n\\label{asm:b-J-sing}\n    The system of the nonlinear equations (\\ref{eq:ori_prob}) holds $n\\geq d$ and has a non-degenerate solution $\\vx^*\\in\\BR^d$, i.e., there exists some $\\mu>0$ such that\n    \\begin{align}\n    \\label{eq:b-J-sing}\n    \\mu = \\sigma_{\\min} (\\mJ(\\vx^*))> 0.\n    \\end{align}\n\\end{asm}","sha256_utf8":"4d26ea3a5662013472ffa95fa892aeb86f96adde97f44be174f1f3973a2b6bb7"}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","title":"IGN循环增量算法","source_units":[{"start":19616,"end":20913,"start_line":510,"end_line":535,"raw_tex":"\\begin{algorithm}[t]\n\\caption{Incremental Gauss–Newton Method (\\texttt{IGN})}\n\\label{alg:IGN-1}\n\\begin{algorithmic}[1]\n\\State \\textbf{Input:} $\\vx^0\\in\\BR^d$, $\\vu^0\\in\\BR^d$, $\\mH^0,\\mG^0\\in\\BR^{d\\times d}$  \\vskip0.1cm\n\\State \\textbf{for} $t=0,1,\\dots$ \\vskip0.12cm\n\\State \\quad  $\\vx^{t+1}=\\mG^t\\vu^t$ \\vskip0.12cm\n\\State \\quad  $i_t =t\\%n + 1$ \\vskip0.12cm\n\\State \\quad  $\\mU^t = \\big[\n    - \\vg_{i_t}(\\vz_{i_t}^t),~~ \n    \\vg_{i_t}(\\vx^{t+1})\n\\big]$ \\vskip0.12cm\n\\State \\quad\n$\\mV^t = \\big[\\vg_{i_t}(\\vz_{i_t}^t),~~ \n    \\vg_{i_t}(\\vx^{t+1})\\big]$ \\vskip0.12cm\n\\State \\quad $\\vu^{t+1} = \\vu^t - \\left(\\vg_{i_t}(\\vz_{i_t}^t)^\\top\\vz_{i_t}^{t} - f_{i_t}(\\vz_{i_t}^t)\\right)\\vg_{i_t}(\\vz_{i_t}^t) \n + \\left(\\vg_{i_t}(\\vx^{t+1})^\\top\\vx^{t+1} - f_{i_t}(\\vx^{t+1})\\right)\\vg_{i_t}(\\vx^{t+1})$ \\vskip0.12cm\n\\State \\quad $\\mH^{t+1} =\\mH^t - \\vg_{i_t}(\\vz_{i_t}^t)\\vg_{i_t}(\\vz_{i_t}^t)^\\top+ \\vg_{i_t}(\\vx^{t+1})\\vg_{i_t}(\\vx^{t+1})^\\top$ \\vskip0.12cm\n\\State\\label{line:update-G} \\quad  $\\mG^{t+1} =\\mG^{t} - \\mG^{t}\\mU^t(\\mI + (\\mV^t)^\\top\\mG^{t}\\mU^t)^{-1}(\\mV^t)^\\top\\mG^{t}$ \\vskip0.12cm\n\\State \\quad  $\\vz_i^{t+1} = \\begin{cases}\n        \\vx^{t+1}, & \\text{if~} i=i_t \\\\\n        \\vz_i^t, & \\text{otherwise}\n    \\end{cases}$ \\vskip0.12cm\n\\State \\textbf{end for}\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"7527308eaa65b4bd7a0c771f91e09c64a69c6f2eda55e19090bad49fb0d5b4ef"}]},{"id":"S011","semantic_type":"derived_fact","kind":"derived_fact","title":"迭代点闭式更新","source_units":[{"start":22279,"end":22488,"start_line":557,"end_line":560,"raw_tex":"\\begin{align}\\label{eq:ign-update-0}\n\\vx^{t+1} \n= \\left(\\sum_{i=1}^n \\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top\\right)^{-1}\\sum_{i=1}^n \\left(\\vg_i(\\vz_i^t)^\\top\\vz_i^t -  f_i(\\vz_i^t)\\right)\\vg_i(\\vz_i^t).\n\\end{align}","sha256_utf8":"1ac3fd2349daedc4d174cd18a80cb6ba46797fc926f9018851ec62a30ff63427"}]},{"id":"S012","semantic_type":"algorithm","kind":"algorithm","title":"历史点循环替换","source_units":[{"start":23047,"end":23222,"start_line":566,"end_line":572,"raw_tex":"\\begin{align}\n\\label{eq:update-1}\n    \\vz_i^{t+1} = \\begin{cases}\n        \\vx^{t+1}, & \\text{if~} i=i_t, \\\\\n        \\vz_i^t, & \\text{otherwise},\n    \\end{cases}   \n\\end{align}","sha256_utf8":"023ec74e8ddaec436ab8764703188d0c1198e615f5b297c68003494577ca5f65"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"聚合变量及逆矩阵","source_units":[{"start":23438,"end":23690,"start_line":574,"end_line":578,"raw_tex":"\\begin{align}\\label{eq:note-1}\n\\vu^t = \\sum_{i=1}^n \\left(\\vg_i(\\vz_i^t)^\\top\\vz_i^t -  f_i(\\vz_i^t)\\right)\\vg_i(\\vz_i^t), \\qquad  \n\\mH^t=\\sum_{i=1}^n \\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top\n\\qquad\\text{and}\\qquad \\mG^t = \\left(\\mH^t\\right)^{-1}.\n\\end{align}","sha256_utf8":"489d1d2613f7f2c660596e148811cfc8b7d93b059918cb45250d172283e2d7a6"}]},{"id":"S014","semantic_type":"notation","kind":"notation","title":"分量历史点含义","source_units":[{"start":21710,"end":21800,"start_line":550,"end_line":550,"raw_tex":"where $\\vz_i^t\\in\\BR^d$ is some point related to component $f_i$ at the $t$-th iteration.\n","sha256_utf8":"d8fabe3f59b58a92088ea9c292f178691d2bbd92a84fb57fc95cb788a408a145"}]},{"id":"S015","semantic_type":"definition","kind":"definition","title":"辅助误差序列","source_units":[{"start":56279,"end":56789,"start_line":1178,"end_line":1191,"raw_tex":"\\begin{dfn}\n\\label{def:series}\nWe define the following sequence $\\{a_t(n,\\nu)\\}_{t\\geq0}$ for given $n\\in\\BN^+$ and $\\nu\\in(0, 1]$:\n\\begin{align}\n\\label{eq:series}\na_t(n,\\nu)\\triangleq\n\\begin{cases}\n    1,~~~~&t=0,\\\\[0.15cm]\n    % \\frac{1}{2(1+\\nu)}~~~~&t=1\\\\\n    \\displaystyle{\\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=0}^{t-1}(a_j(n, \\nu))^{1+\\nu} + n - t\\right)},~~~~& 1\\leq t\\leq n,\\\\[0.6cm]\n     \\displaystyle{\\frac{1}{2(1+\\nu)n}\\sum_{j=t-n}^{t-1}(a_j(n, \\nu))^{1+\\nu}},~~~~&t> n.\n\\end{cases}\n\\end{align}\n\\end{dfn}","sha256_utf8":"11191c1488b81b075da98ae73805115328d624a09596590ac27ab23b2e1559dc"}]},{"id":"S016","semantic_type":"lemma","kind":"lemma","title":"各分量梯度连续性","source_units":[{"start":52499,"end":52801,"start_line":1084,"end_line":1091,"raw_tex":"\\begin{lem}\n    \\label{le:H-con}\n    \\textnormal{(H\\\"older continuity of each gradient)} Under Assumption \\ref{asm:holder-g}, it satisfies that\n    \\begin{align}\n        \\Norm{\\vg_i(\\vy)-\\vg_i(\\vx)}\\leq \\fH_\\nu\\Norm{\\vy-\\vx}^\\nu,\n    \\end{align}\n    for any $\\vx, \\vy\\in\\BR^d$, and $i\\in[n]$.\n\\end{lem}","sha256_utf8":"bdce0cee8f1ac659190635fffb97a29f35d2049e4aa7896fdbb8a0a0d0f2b9a6"}]},{"id":"S017","semantic_type":"proof","kind":"proof","title":"分量梯度连续性证明","source_units":[{"start":52803,"end":53583,"start_line":1093,"end_line":1110,"raw_tex":"\\begin{proof}\n    We denote\n    \\begin{align*}\n    \\Tilde{\\mJ} = \\begin{bmatrix}\n        (\\vg_1(\\vy)-\\vg_1(\\vx))^\\top \\\\\n        \\vdots \\\\\n        (\\vg_n(\\vy)-\\vg_n(\\vx))^\\top \n    \\end{bmatrix}\\in\\BR^{n\\times d} \\qquad\\text{with}\\qquad \\vg_i(\\vx)=\\nabla f_i(\\vx)\n    \\end{align*}\n    and let $\\ve_i\\in\\BR^{n}$ be the $i$-th standard basic vector in $n$-dimensional Euclidean space. \n    Then the facts~$\\Tilde{\\mJ}=\\mJ(\\vy)-\\mJ(\\vx)$ and $\\Tilde{\\mJ}^\\top \\ve_i=\\vg_i(\\vy)-\\vg_i(\\vx)$ imply we have\n    \\begin{align*}\n    \\Norm{\\vg_i(\\vy)-\\vg_i(\\vx)}\n    \\leq \\|\\Tilde{\\mJ}^\\top\\|\\Norm{\\ve_i} = \n    \\|\\Tilde{\\mJ}\\|=\\Norm{\\mJ(\\vy)-\\mJ(\\vx)}\\leq \\fH_\\nu\\Norm{\\vy-\\vx}^\\nu,\n    \\end{align*}\n    where the last step is based on the H\\\"older continuouity of $\\mJ(\\cdot)$.\n\\end{proof}","sha256_utf8":"1954bd6ca68c9bf6d4d4252fcd97dbbb8bd03b29075b78073899aaff4d16ece9"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"构造雅可比差矩阵","source_units":[{"start":52803,"end":53187,"start_line":1093,"end_line":1102,"raw_tex":"\\begin{proof}\n    We denote\n    \\begin{align*}\n    \\Tilde{\\mJ} = \\begin{bmatrix}\n        (\\vg_1(\\vy)-\\vg_1(\\vx))^\\top \\\\\n        \\vdots \\\\\n        (\\vg_n(\\vy)-\\vg_n(\\vx))^\\top \n    \\end{bmatrix}\\in\\BR^{n\\times d} \\qquad\\text{with}\\qquad \\vg_i(\\vx)=\\nabla f_i(\\vx)\n    \\end{align*}\n    and let $\\ve_i\\in\\BR^{n}$ be the $i$-th standard basic vector in $n$-dimensional Euclidean space. \n","sha256_utf8":"2ed96accbdfa185d3303d1d170e90e52f4ed6922e464ddffe825308ad36d197a"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"由矩阵界控制行向量","source_units":[{"start":53187,"end":53583,"start_line":1103,"end_line":1110,"raw_tex":"    Then the facts~$\\Tilde{\\mJ}=\\mJ(\\vy)-\\mJ(\\vx)$ and $\\Tilde{\\mJ}^\\top \\ve_i=\\vg_i(\\vy)-\\vg_i(\\vx)$ imply we have\n    \\begin{align*}\n    \\Norm{\\vg_i(\\vy)-\\vg_i(\\vx)}\n    \\leq \\|\\Tilde{\\mJ}^\\top\\|\\Norm{\\ve_i} = \n    \\|\\Tilde{\\mJ}\\|=\\Norm{\\mJ(\\vy)-\\mJ(\\vx)}\\leq \\fH_\\nu\\Norm{\\vy-\\vx}^\\nu,\n    \\end{align*}\n    where the last step is based on the H\\\"older continuouity of $\\mJ(\\cdot)$.\n\\end{proof}","sha256_utf8":"0909fcf4eb1f51df5d234582b226d30bed67229ed209700d00e0ef1303f75732"}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","title":"Hölder线性化余项界","source_units":[{"start":53585,"end":53916,"start_line":1112,"end_line":1119,"raw_tex":"\\begin{lem}\n\\label{le:power-bound}\n\\textnormal{(Bound for H\\\"older-continuous function)} Under Assumption \\ref{asm:holder-g}, we have\n    \\begin{align}\n        f_i(\\vy) - f_i(\\vx) - \\vg_i(\\vx)^\\top(\\vy-\\vx) \\leq \\frac{\\fH_\\nu}{1 + \\nu}\\Norm{\\vy-\\vx}^{1+\\nu},\n    \\end{align}\n    for any $\\vx, \\vy\\in\\BR^d$ and  $i\\in[n]$.\n\\end{lem}","sha256_utf8":"94f613c09c820ee6e5ca298b733808fd5ee3b7911152f2a4cf0a17c3f652520c"}]},{"id":"S021","semantic_type":"proof","kind":"proof","title":"线性化余项界证明","source_units":[{"start":53918,"end":54817,"start_line":1121,"end_line":1133,"raw_tex":"\\begin{proof}\n    Following the proof of \\cite{grapiglia2017regularized, grapiglia2019accelerated}, we have\n    \\begin{align*}\n        f_i(\\vy) - f_i(\\vx) - \\vg_i(\\vx)^\\top(\\vy - \\vx)\n        & = \\int_{t=0}^1 \\vg_i(\\vx + t(\\vy - \\vx))^\\top(\\vy - \\vx)\\text{d}t - \\vg_i(\\vx)^\\top(\\vy - \\vx)\\\\\n        & = \\int_{t=0}^1 \\left(\\vg_i(\\vx + t(\\vy - \\vx)) - \\vg_i(\\vx)\\right)^\\top(\\vy - \\vx)\\text{d}t\\\\\n        & \\leq \\int_{t=0}^1 \\Norm{\\vg_i(\\vx + t(\\vy - \\vx)) - \\vg_i(\\vx)}\\Norm{\\vy - \\vx}\\text{d}t\\\\\n        & \\leq \\int_{t=0}^1\\fH_\\nu t^\\nu\\Norm{\\vy - \\vx}^{1+\\nu}\\text{d}t\\\\\n        & = \\fH_\\nu \\Norm{\\vy - \\vx}^{1+\\nu}\\int_{t=0}^1 t^\\nu\\text{d}t\\\\\n        & = \\frac{\\fH_\\nu}{1 + \\nu} \\Norm{\\vy - \\vx}^{1+\\nu},\n    \\end{align*}\n    where the first inequality comes from Cauchy-Schwarz inequality, and the second one comes from Lemma \\ref{le:H-con} that each gradient is H\\\"older continuous.\n\\end{proof}","sha256_utf8":"7e06d66d9f733d8a3e976b28b2ee6b5bb7890721af13a505ca8591a068b68341"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"沿线段积分估计余项","source_units":[{"start":53918,"end":54817,"start_line":1121,"end_line":1133,"raw_tex":"\\begin{proof}\n    Following the proof of \\cite{grapiglia2017regularized, grapiglia2019accelerated}, we have\n    \\begin{align*}\n        f_i(\\vy) - f_i(\\vx) - \\vg_i(\\vx)^\\top(\\vy - \\vx)\n        & = \\int_{t=0}^1 \\vg_i(\\vx + t(\\vy - \\vx))^\\top(\\vy - \\vx)\\text{d}t - \\vg_i(\\vx)^\\top(\\vy - \\vx)\\\\\n        & = \\int_{t=0}^1 \\left(\\vg_i(\\vx + t(\\vy - \\vx)) - \\vg_i(\\vx)\\right)^\\top(\\vy - \\vx)\\text{d}t\\\\\n        & \\leq \\int_{t=0}^1 \\Norm{\\vg_i(\\vx + t(\\vy - \\vx)) - \\vg_i(\\vx)}\\Norm{\\vy - \\vx}\\text{d}t\\\\\n        & \\leq \\int_{t=0}^1\\fH_\\nu t^\\nu\\Norm{\\vy - \\vx}^{1+\\nu}\\text{d}t\\\\\n        & = \\fH_\\nu \\Norm{\\vy - \\vx}^{1+\\nu}\\int_{t=0}^1 t^\\nu\\text{d}t\\\\\n        & = \\frac{\\fH_\\nu}{1 + \\nu} \\Norm{\\vy - \\vx}^{1+\\nu},\n    \\end{align*}\n    where the first inequality comes from Cauchy-Schwarz inequality, and the second one comes from Lemma \\ref{le:H-con} that each gradient is H\\\"older continuous.\n\\end{proof}","sha256_utf8":"7e06d66d9f733d8a3e976b28b2ee6b5bb7890721af13a505ca8591a068b68341"}]},{"id":"S023","semantic_type":"lemma","kind":"lemma","title":"雅可比与梯度有界","source_units":[{"start":54819,"end":55081,"start_line":1135,"end_line":1142,"raw_tex":"\\begin{lem}\n\\label{le:b-grad}\n    \\textnormal{(Bound for Jacobian and gradient)} Under Assumption \\ref{asm:L-f}, we have\n    \\begin{align*}\n        \\Norm{\\vg_i(\\vx)}\\leq\\Norm{\\mJ(\\vx)}\\leq L_f\n    \\end{align*}\n    for all $ \\vx\\in\\BR^d$ and $i\\in [n]$.\n\\end{lem}","sha256_utf8":"3396877002acc980f62febd820b1a07efd2e7ba2d2f35d988401a0aa6188ead0"}]},{"id":"S024","semantic_type":"proof","kind":"proof","title":"雅可比与梯度界证明","source_units":[{"start":55083,"end":56119,"start_line":1144,"end_line":1170,"raw_tex":"\\begin{proof}\n\n    For all $\\vx, \\vv\\in\\BR^d$, we have\n    \\begin{align*}\n        \\mJ(\\vx)\\vv =\\lim_{h\\to0}\\frac{\\vf(\\vx+h\\vv)-\\vf(\\vx)}{h}.\n    \\end{align*}\n    Taking the spectral norm on both sides, we have\n    \\begin{align*}\n        \\Norm{\\mJ(\\vx)\\vv}\n        & = \\lim_{h\\to0}\\frac{\\Norm{\\vf(\\vx+h\\vv)-\\vf(\\vx)}}{\\vert h\\vert}\\\\\n        & \\leq \\lim_{h\\to0}\\frac{L_f\\Norm{\\vx+h\\vv-\\vx}}{\\vert h\\vert}\\\\\n        & = \\lim_{h\\to0}\\frac{L_f\\vert h\\vert\\Norm{\\vv}}{\\vert h\\vert}\\\\\n        & = L_f\\Norm{\\vv},\n    \\end{align*}\n    where the inequality comes from Assumption \\ref{asm:L-f}. \n    \n    Therefore, for all $\\vx\\in\\BR^d$ it holds\n    \\begin{align*}\n    \\Norm{\\mJ(\\vx)}=\\sup_{\\vv\\in\\BR^d}\\frac{\\Norm{\\mJ(\\vx)\\vv}}{\\Norm{\\vv}}\\leq L_f.\n    \\end{align*}\n    Let $\\ve_i\\in\\BR^{n}$ be the $i$-th standard basic vector in $n$-dimensional Euclid space, then we have \n\n    \\begin{align*}\n    \\Norm{\\vg_i(\\vx)} = \\Norm{\\mJ^\\top \\ve_i} \\leq \\Norm{\\mJ}\\Norm{\\ve_i}=\\Norm{\\mJ(\\vx)}\\leq L_f\n    \\end{align*}\n    for all $i\\in[n]$.\n\\end{proof}","sha256_utf8":"bd87440427c8b1ef9d43f24ab35851da14793b4ba36a0447fc3f3f6c42bbb509"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"差商控制方向导数","source_units":[{"start":55083,"end":55669,"start_line":1144,"end_line":1158,"raw_tex":"\\begin{proof}\n\n    For all $\\vx, \\vv\\in\\BR^d$, we have\n    \\begin{align*}\n        \\mJ(\\vx)\\vv =\\lim_{h\\to0}\\frac{\\vf(\\vx+h\\vv)-\\vf(\\vx)}{h}.\n    \\end{align*}\n    Taking the spectral norm on both sides, we have\n    \\begin{align*}\n        \\Norm{\\mJ(\\vx)\\vv}\n        & = \\lim_{h\\to0}\\frac{\\Norm{\\vf(\\vx+h\\vv)-\\vf(\\vx)}}{\\vert h\\vert}\\\\\n        & \\leq \\lim_{h\\to0}\\frac{L_f\\Norm{\\vx+h\\vv-\\vx}}{\\vert h\\vert}\\\\\n        & = \\lim_{h\\to0}\\frac{L_f\\vert h\\vert\\Norm{\\vv}}{\\vert h\\vert}\\\\\n        & = L_f\\Norm{\\vv},\n    \\end{align*}\n    where the inequality comes from Assumption \\ref{asm:L-f}. \n","sha256_utf8":"aa49449054ab79509b2fb27e6ed763c7bc7d626d9d41b5f736f1eac1d8d4030a"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"取得谱范数与梯度界","source_units":[{"start":55674,"end":56119,"start_line":1160,"end_line":1170,"raw_tex":"    Therefore, for all $\\vx\\in\\BR^d$ it holds\n    \\begin{align*}\n    \\Norm{\\mJ(\\vx)}=\\sup_{\\vv\\in\\BR^d}\\frac{\\Norm{\\mJ(\\vx)\\vv}}{\\Norm{\\vv}}\\leq L_f.\n    \\end{align*}\n    Let $\\ve_i\\in\\BR^{n}$ be the $i$-th standard basic vector in $n$-dimensional Euclid space, then we have \n\n    \\begin{align*}\n    \\Norm{\\vg_i(\\vx)} = \\Norm{\\mJ^\\top \\ve_i} \\leq \\Norm{\\mJ}\\Norm{\\ve_i}=\\Norm{\\mJ(\\vx)}\\leq L_f\n    \\end{align*}\n    for all $i\\in[n]$.\n\\end{proof}","sha256_utf8":"391dcfdf27ef013abf06857e63b66de05bbb0a0d4f76d0f5433f0b92c44f35ae"}]},{"id":"S027","semantic_type":"proposition","kind":"proposition","title":"解处Gram矩阵非退化","source_units":[{"start":27035,"end":27238,"start_line":632,"end_line":637,"raw_tex":"\\begin{prop}\\label{prop:b-JJ-sing}\nUnder Assumption \\ref{asm:b-J-sing}, it holds that\n\\begin{align}\\label{eq::b-JJ-sing}\n    \\sigma_{\\min} (\\mJ(\\vx^*)^\\top \\mJ(\\vx^*)) = \\mu^2 > 0.\n\\end{align}\n\\end{prop}","sha256_utf8":"a4dd6c5faf5a14de441b1de6e01d0ad82ba5eccd36d175b8dfa89ff65839fee0"}]},{"id":"S028","semantic_type":"proof","kind":"proof","title":"解处Gram谱界证明","source_units":[{"start":65973,"end":66682,"start_line":1405,"end_line":1419,"raw_tex":"\\begin{proof}\nWe denote the singular value decomposition of $\\mJ(\\vx^*)$ as\n    \\begin{align*}\n        \\mJ(\\vx^*) = \\mP\\mD\\mQ^\\top,\n    \\end{align*}\n    where $\\mP\\in\\BR^{n\\times d}, \\mQ\\in\\BR^{d\\times d}$ are (column) orthogonal matrices and $\\mD\\in\\BR^{d\\times d}$ is diagonal matrix with the smallest diagonal entry of $\\mu>0$.\n    Therefore, we have\n    \\begin{align*}\n        \\mJ(\\vx^*)^\\top \\mJ(\\vx^*) = \\mQ\\mD^2\\mQ^\\top,\n    \\end{align*}\n    which means the smallest singular value of $\\mJ(\\vx^*)^\\top \\mJ(\\vx^*)$ is equal to the smallest value of $\\mD^2$, which is $\\mu^2$. Therefore, we have\n    \\begin{align*}\n        \\sigma_{\\min}(\\mJ(\\vx^*)^\\top\\mJ(\\vx^*)) \\geq \\mu^2.\n    \\end{align*}\n\\end{proof}","sha256_utf8":"605887f6ee65c82163e05653eb8ab9a77f17fe71139cc2d5225111567c082e8e"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"分解解处雅可比","source_units":[{"start":65973,"end":66304,"start_line":1405,"end_line":1410,"raw_tex":"\\begin{proof}\nWe denote the singular value decomposition of $\\mJ(\\vx^*)$ as\n    \\begin{align*}\n        \\mJ(\\vx^*) = \\mP\\mD\\mQ^\\top,\n    \\end{align*}\n    where $\\mP\\in\\BR^{n\\times d}, \\mQ\\in\\BR^{d\\times d}$ are (column) orthogonal matrices and $\\mD\\in\\BR^{d\\times d}$ is diagonal matrix with the smallest diagonal entry of $\\mu>0$.\n","sha256_utf8":"34c4359f145b9567accbcd3f400b1aeba7910e5241f5b232ff9a96e913b08ff8"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"计算Gram最小奇异值","source_units":[{"start":66304,"end":66682,"start_line":1411,"end_line":1419,"raw_tex":"    Therefore, we have\n    \\begin{align*}\n        \\mJ(\\vx^*)^\\top \\mJ(\\vx^*) = \\mQ\\mD^2\\mQ^\\top,\n    \\end{align*}\n    which means the smallest singular value of $\\mJ(\\vx^*)^\\top \\mJ(\\vx^*)$ is equal to the smallest value of $\\mD^2$, which is $\\mu^2$. Therefore, we have\n    \\begin{align*}\n        \\sigma_{\\min}(\\mJ(\\vx^*)^\\top\\mJ(\\vx^*)) \\geq \\mu^2.\n    \\end{align*}\n\\end{proof}","sha256_utf8":"c3adc8987ab831aa3655ef45ee32cb7837babe54765ae3c65d01308cc7a99705"}]},{"id":"S031","semantic_type":"lemma","kind":"lemma","title":"Gram矩阵Hölder界","source_units":[{"start":27370,"end":27804,"start_line":642,"end_line":655,"raw_tex":"\\begin{lem}\n\\label{le:L-JJ}\n{\n    Under Assumptions \\ref{asm:L-f} and \\ref{asm:holder-g}, we have \n    \\begin{align*}\n    \\Norm{\\mJ(\\vy)^\\top\\mJ(\\vy) - \\mJ(\\vx)^\\top\\mJ(\\vx)} & \\leq 2L_f\\fH_\\nu\\Norm{\\vy-\\vx}^\\nu\n    \\end{align*}\n    and\n    \\begin{align*}\n    \\Norm{\\vg_i(\\vy)\\vg_i(\\vy)^\\top - \\vg_i(\\vx)\\vg_i(\\vx)^\\top}   \\leq 2L_f\\fH_\\nu\\Norm{\\vy-\\vx}^\\nu\n    \\end{align*}\n    for all $\\vx, \\vy\\in\\BR^n$ and $i \\in [n]$.\n}\n\\end{lem}","sha256_utf8":"b64c5f044320a1f55821bd8c3aa784d451195541e3b2a1e9e8d7ef6847823b7b"}]},{"id":"S032","semantic_type":"proof","kind":"proof","title":"Gram连续性证明","source_units":[{"start":66728,"end":68246,"start_line":1424,"end_line":1444,"raw_tex":"\\begin{proof}\n    The Jacobian holds that\n    \\begin{align*}\n        \\Norm{\\mJ(\\vy)^\\top\\mJ(\\vy) - \\mJ(\\vx)^\\top\\mJ(\\vx)}\n        & = \\Norm{\\mJ(\\vy)^\\top\\mJ(\\vy) - \\mJ(\\vx)^\\top\\mJ(\\vy) + \\mJ(\\vx)^\\top\\mJ(\\vy) - \\mJ(\\vx)^\\top\\mJ(\\vx)}\\\\\n        & \\leq \\Norm{\\left(\\mJ(\\vy)-\\mJ(\\vx)\\right)^\\top\\mJ(\\vy)} + \\Norm{\\mJ(\\vx)^\\top\\left(\\mJ(\\vy) - \\mJ(\\vx)\\right)}\\\\\n        & \\leq \\Norm{\\mJ(\\vy)}\\Norm{\\mJ(\\vy)-\\mJ(\\vx)} + \\Norm{\\mJ(\\vx)}\\Norm{\\mJ(\\vy)-\\mJ(\\vx)}\\\\\n        & \\leq 2L_f\\fH_\\nu\\Norm{\\vy-\\vx}^\\nu,\n    \\end{align*}\n    where the first inequality comes from triangular inequality, the second inequality comes from property of norm, and the last inequality is based on Lemma \\ref{le:b-grad} and Assumption \\ref{asm:holder-g}. \n    \n    For all $j\\in[n]$, the gradient holds that\n    \\begin{align*}\n        \\Norm{\\vg_i(\\vy)\\vg_i(\\vy)^\\top\\!- \\vg_i(\\vx)\\vg_i(\\vx)^\\top}\n        & = \\Norm{\\vg_i(\\vy)\\vg_i(\\vy)^\\top\\!- \\vg_i(\\vx)\\vg_i(\\vy)^\\top + \\vg_i(\\vx)\\vg_i(\\vy)^\\top\\!- \\vg_i(\\vx)\\vg_i(\\vx)^\\top}\\\\\n        & \\leq \\Norm{\\left(\\vg_i(\\vy)-\\vg_i(\\vx)\\right)\\vg_i(\\vy)^\\top} + \\Norm{\\vg_i(\\vx)\\left(\\vg_i(\\vy)-\\vg_i(\\vx)\\right)^\\top}\\\\\n        & \\leq \\Norm{\\vg_i(\\vy)}\\Norm{\\vg_i(\\vy)-\\vg_i(\\vx)} + \\Norm{\\vg_i(\\vx)}\\Norm{\\vg_i(\\vy)-\\vg_i(\\vx)}\\\\\n        & \\leq 2L_f\\fH_\\nu\\Norm{\\vy-\\vx}^\\nu,\n    \\end{align*}\n    where the first inequality comes from triangular inequality, the second inequality comes from property of norm, and the last inequality is based on Lemma \\ref{le:H-con} and \\ref{le:b-grad}.\n\\end{proof}","sha256_utf8":"df00ed31897057272c21246b36d15111bb334dd681f6ceed83f815153c10ec96"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"分解整体Gram差","source_units":[{"start":66728,"end":67460,"start_line":1424,"end_line":1433,"raw_tex":"\\begin{proof}\n    The Jacobian holds that\n    \\begin{align*}\n        \\Norm{\\mJ(\\vy)^\\top\\mJ(\\vy) - \\mJ(\\vx)^\\top\\mJ(\\vx)}\n        & = \\Norm{\\mJ(\\vy)^\\top\\mJ(\\vy) - \\mJ(\\vx)^\\top\\mJ(\\vy) + \\mJ(\\vx)^\\top\\mJ(\\vy) - \\mJ(\\vx)^\\top\\mJ(\\vx)}\\\\\n        & \\leq \\Norm{\\left(\\mJ(\\vy)-\\mJ(\\vx)\\right)^\\top\\mJ(\\vy)} + \\Norm{\\mJ(\\vx)^\\top\\left(\\mJ(\\vy) - \\mJ(\\vx)\\right)}\\\\\n        & \\leq \\Norm{\\mJ(\\vy)}\\Norm{\\mJ(\\vy)-\\mJ(\\vx)} + \\Norm{\\mJ(\\vx)}\\Norm{\\mJ(\\vy)-\\mJ(\\vx)}\\\\\n        & \\leq 2L_f\\fH_\\nu\\Norm{\\vy-\\vx}^\\nu,\n    \\end{align*}\n    where the first inequality comes from triangular inequality, the second inequality comes from property of norm, and the last inequality is based on Lemma \\ref{le:b-grad} and Assumption \\ref{asm:holder-g}. \n","sha256_utf8":"cb427d0954b8009068f73ea9dadf8b8792b1df05a92e3cc26ac250db8bf14d7e"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"分解分量外积差","source_units":[{"start":67465,"end":68246,"start_line":1435,"end_line":1444,"raw_tex":"    For all $j\\in[n]$, the gradient holds that\n    \\begin{align*}\n        \\Norm{\\vg_i(\\vy)\\vg_i(\\vy)^\\top\\!- \\vg_i(\\vx)\\vg_i(\\vx)^\\top}\n        & = \\Norm{\\vg_i(\\vy)\\vg_i(\\vy)^\\top\\!- \\vg_i(\\vx)\\vg_i(\\vy)^\\top + \\vg_i(\\vx)\\vg_i(\\vy)^\\top\\!- \\vg_i(\\vx)\\vg_i(\\vx)^\\top}\\\\\n        & \\leq \\Norm{\\left(\\vg_i(\\vy)-\\vg_i(\\vx)\\right)\\vg_i(\\vy)^\\top} + \\Norm{\\vg_i(\\vx)\\left(\\vg_i(\\vy)-\\vg_i(\\vx)\\right)^\\top}\\\\\n        & \\leq \\Norm{\\vg_i(\\vy)}\\Norm{\\vg_i(\\vy)-\\vg_i(\\vx)} + \\Norm{\\vg_i(\\vx)}\\Norm{\\vg_i(\\vy)-\\vg_i(\\vx)}\\\\\n        & \\leq 2L_f\\fH_\\nu\\Norm{\\vy-\\vx}^\\nu,\n    \\end{align*}\n    where the first inequality comes from triangular inequality, the second inequality comes from property of norm, and the last inequality is based on Lemma \\ref{le:H-con} and \\ref{le:b-grad}.\n\\end{proof}","sha256_utf8":"1d834bb31095c8ea992bab61885774e091032d51af1bb47df01fe90dac58a3cd"}]},{"id":"S035","semantic_type":"lemma","kind":"lemma","title":"增量Gram谱下界","source_units":[{"start":28047,"end":28421,"start_line":661,"end_line":668,"raw_tex":"\\begin{lem}\n\\label{le:lb-Ht-new-1}\nUnder Assumptions \\ref{asm:L-f}, \\ref{asm:holder-g} and \\ref{asm:b-J-sing}, running \\IGN~(Algorithm \\ref{alg:IGN-1}) with $\\mH^0 = \\mJ(\\vx^0)^\\top\\mJ(\\vx^0)$ and~$\\mG^0=(\\mH^0)^{-1}$ holds that\n\\begin{align*}\n    \\sigma_{\\min}(\\mH^t) \\geq \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^t - \\vx^*}^\\nu\n\\end{align*}\nfor all $t\\geq 0$.\n\\end{lem}","sha256_utf8":"3771f77b2ae64d0497d93b06ea1f165a487e3384f487ecb6f15fd6ed9d019767"}]},{"id":"S036","semantic_type":"proof","kind":"proof","title":"增量Gram谱界证明","source_units":[{"start":68299,"end":69309,"start_line":1449,"end_line":1466,"raw_tex":"\\begin{proof}\n    We have\n    \\begin{align*}\n        \\Norm{\\mH^t - \\mJ(\\vx^*)^\\top\\mJ(\\vx^*)} &= \\Norm{\\sum_{i=1}^n\\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top - \\sum_{i=1}^n\\vg_i(\\vx^*)\\vg_i(\\vx^*)^\\top}\\\\\n        & \\leq \\sum_{i=1}^n\\Norm{\\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top - \\vg_i(\\vx^*)\\vg_i(\\vx^*)^\\top}\\\\\n        & \\leq \\sum_{i=1}^n 2L_f\\fH_\\nu\\Norm{\\vz_i^t - \\vx^*}^\\nu,\n    \\end{align*}\n    where the first inequality comes from the triangle inequality and the second inequality is based on Lemma \\ref{le:L-JJ}. \n    Thus, we have\n     \\begin{align*}\n         \\mH^{t}-\\mJ(\\vx^*)^{\\top}\\mJ(\\vx^*)\\succeq - \\sum_{i=1}^n 2L_f\\fH_\\nu\\Norm{\\vz_i^t - \\vx^*}^\\nu \\cdot\\mI, \n     \\end{align*}\n    which implies that\n    \\begin{align*}\n        \\sigma_{\\min}(\\mH^t) \\geq \\sigma_{\\min}(\\mJ(\\vx^*)^\\top\\mJ(\\vx^*)) - \\sum_{i=1}^n 2L_f\\fH_\\nu\\Norm{\\vz_i^t - \\vx^*}^\\nu = \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^t - \\vx^*}^\\nu,\n    \\end{align*}\n    where the last step is based on Proposition \\ref{prop:b-JJ-sing}.\n\\end{proof}","sha256_utf8":"046b042b56968b6cf2f1365a29dc51f3fa90bdca400e954fa5834e6084b72219"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"累计历史点矩阵扰动","source_units":[{"start":68299,"end":68807,"start_line":1449,"end_line":1456,"raw_tex":"\\begin{proof}\n    We have\n    \\begin{align*}\n        \\Norm{\\mH^t - \\mJ(\\vx^*)^\\top\\mJ(\\vx^*)} &= \\Norm{\\sum_{i=1}^n\\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top - \\sum_{i=1}^n\\vg_i(\\vx^*)\\vg_i(\\vx^*)^\\top}\\\\\n        & \\leq \\sum_{i=1}^n\\Norm{\\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top - \\vg_i(\\vx^*)\\vg_i(\\vx^*)^\\top}\\\\\n        & \\leq \\sum_{i=1}^n 2L_f\\fH_\\nu\\Norm{\\vz_i^t - \\vx^*}^\\nu,\n    \\end{align*}\n    where the first inequality comes from the triangle inequality and the second inequality is based on Lemma \\ref{le:L-JJ}. \n","sha256_utf8":"6befaa4d139bfa9c3235e2d8f50590e8e2b2a9b89c0f2cb0376666a3f99e8b80"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"扰动界转为谱下界","source_units":[{"start":68807,"end":69309,"start_line":1457,"end_line":1466,"raw_tex":"    Thus, we have\n     \\begin{align*}\n         \\mH^{t}-\\mJ(\\vx^*)^{\\top}\\mJ(\\vx^*)\\succeq - \\sum_{i=1}^n 2L_f\\fH_\\nu\\Norm{\\vz_i^t - \\vx^*}^\\nu \\cdot\\mI, \n     \\end{align*}\n    which implies that\n    \\begin{align*}\n        \\sigma_{\\min}(\\mH^t) \\geq \\sigma_{\\min}(\\mJ(\\vx^*)^\\top\\mJ(\\vx^*)) - \\sum_{i=1}^n 2L_f\\fH_\\nu\\Norm{\\vz_i^t - \\vx^*}^\\nu = \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^t - \\vx^*}^\\nu,\n    \\end{align*}\n    where the last step is based on Proposition \\ref{prop:b-JJ-sing}.\n\\end{proof}","sha256_utf8":"b63c36c7a9d4b685fc8264970fb9e7df46dad15b0379a2c21ccb674dfdca6d8c"}]},{"id":"S039","semantic_type":"lemma","kind":"lemma","title":"辅助序列不超过一","source_units":[{"start":56884,"end":57051,"start_line":1195,"end_line":1201,"raw_tex":"\\begin{lem}\n\\label{le:s-mono-000} The sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$ satisfies \n\\begin{align*}\n    a_t(n, \\nu) \\leq 1 \n\\end{align*}\nfor all $t\\geq 0$. \n\\end{lem}","sha256_utf8":"03a7841da04ea2b4c22702e84c915ac35f6647f6a114979ac070c7342829c8f5"}]},{"id":"S040","semantic_type":"proof","kind":"proof","title":"辅助序列有界证明","source_units":[{"start":57052,"end":58783,"start_line":1202,"end_line":1232,"raw_tex":"\\begin{proof}\n\\textbf{Part I:}\nWe first use induction to prove $a_t(n, \\nu) \\leq 1$ for all $t=0,1\\dots,n$.\nFor the induction base, we can verify that $a_0(n, \\nu) = 1 \\leq 1$.  For the induction step, we assume \n\\begin{align*}\na_j(n,\\nu) \\leq 1    \n\\end{align*}\nholds for all $j=1,\\dots,t-1$ such that $t\\leq n$.\nThen we have\n\\begin{align*}\n        a_t(n,\\nu) = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=0}^{t-1}(a_j(n,\\nu))^{1+\\nu} + n - t\\right) \\leq \\frac{1}{2(1+\\nu)n}\\left(t + n - t\\right) = \\frac{1}{2(1+\\nu)}\\leq 1,\n\\end{align*}\nwhere the first inequality is based on the induction hypothesis and the last inequality is based on the setting $\\nu\\in(0,1]$. This finishes the induction. \n\n\\textbf{Part II:} We then use induction to prove $a_t(n, \\nu) \\leq 1$ for all $t\\geq n+1$. For the induction base, we can verify that\n\\begin{align*}\n    a_{n+1}(n,\\nu) = \\frac{1}{2(1+\\nu)n}\\sum_{j=1}^{n}(a_j(n, \\nu))^{1+\\nu} \\leq \\frac{1}{2(1+\\nu)n}\\cdot n = \\frac{1}{2(1+\\nu)} \\leq 1,\n\\end{align*}\nwhere the first inequality is based on $a_t(n, \\nu) \\leq 1$ for all $t\\leq n$ (which have shown in Part I), and the last inequality is based on the setting $\\nu \\in (0, 1]$. For the induction step, we assume\n\\begin{align*}\n    a_{n+1}(n,\\nu) \\leq 1\n\\end{align*}\nholds for all $j=n+2,\\dots,t-1$ such that $t\\geq n+3$. Then we have\n\\begin{align*}\n    a_{t}(n,\\nu) = \\frac{1}{2(1+\\nu)n}\\sum_{j=t-n}^{t-1}(a_j(n, \\nu))^{1+\\nu} \\leq \\frac{1}{2(1+\\nu)n}\\cdot n = \\frac{1}{2(1+\\nu)} \\leq 1,\n\\end{align*}\n\nwhere the first inequality is based on the induction hypothesis and the last inequality is based on the setting $\\nu \\in (0, 1]$. This finishes the induction.\n\nCombining the results of above two parts, we finish the proof of this lemma.\n\\end{proof}","sha256_utf8":"d2ce043eab60f44d61e0f4278068fdcf3749d93a0b9b1e775ed34758beee3634"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"首周期有界归纳","source_units":[{"start":57052,"end":57737,"start_line":1202,"end_line":1214,"raw_tex":"\\begin{proof}\n\\textbf{Part I:}\nWe first use induction to prove $a_t(n, \\nu) \\leq 1$ for all $t=0,1\\dots,n$.\nFor the induction base, we can verify that $a_0(n, \\nu) = 1 \\leq 1$.  For the induction step, we assume \n\\begin{align*}\na_j(n,\\nu) \\leq 1    \n\\end{align*}\nholds for all $j=1,\\dots,t-1$ such that $t\\leq n$.\nThen we have\n\\begin{align*}\n        a_t(n,\\nu) = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=0}^{t-1}(a_j(n,\\nu))^{1+\\nu} + n - t\\right) \\leq \\frac{1}{2(1+\\nu)n}\\left(t + n - t\\right) = \\frac{1}{2(1+\\nu)}\\leq 1,\n\\end{align*}\nwhere the first inequality is based on the induction hypothesis and the last inequality is based on the setting $\\nu\\in(0,1]$. This finishes the induction. \n","sha256_utf8":"4bf61be257dcac2a9ea4376f17662dc090bedcc2453b5387fd36458183a77ffb"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"后续周期有界归纳","source_units":[{"start":57738,"end":58783,"start_line":1216,"end_line":1232,"raw_tex":"\\textbf{Part II:} We then use induction to prove $a_t(n, \\nu) \\leq 1$ for all $t\\geq n+1$. For the induction base, we can verify that\n\\begin{align*}\n    a_{n+1}(n,\\nu) = \\frac{1}{2(1+\\nu)n}\\sum_{j=1}^{n}(a_j(n, \\nu))^{1+\\nu} \\leq \\frac{1}{2(1+\\nu)n}\\cdot n = \\frac{1}{2(1+\\nu)} \\leq 1,\n\\end{align*}\nwhere the first inequality is based on $a_t(n, \\nu) \\leq 1$ for all $t\\leq n$ (which have shown in Part I), and the last inequality is based on the setting $\\nu \\in (0, 1]$. For the induction step, we assume\n\\begin{align*}\n    a_{n+1}(n,\\nu) \\leq 1\n\\end{align*}\nholds for all $j=n+2,\\dots,t-1$ such that $t\\geq n+3$. Then we have\n\\begin{align*}\n    a_{t}(n,\\nu) = \\frac{1}{2(1+\\nu)n}\\sum_{j=t-n}^{t-1}(a_j(n, \\nu))^{1+\\nu} \\leq \\frac{1}{2(1+\\nu)n}\\cdot n = \\frac{1}{2(1+\\nu)} \\leq 1,\n\\end{align*}\n\nwhere the first inequality is based on the induction hypothesis and the last inequality is based on the setting $\\nu \\in (0, 1]$. This finishes the induction.\n\nCombining the results of above two parts, we finish the proof of this lemma.\n\\end{proof}","sha256_utf8":"f6c38e512b5d40aecba8000a0b86980804c2fc8397f965f35e47dc265881500d"}]},{"id":"S043","semantic_type":"lemma","kind":"lemma","title":"辅助序列单调递减","source_units":[{"start":58786,"end":58965,"start_line":1235,"end_line":1242,"raw_tex":"\\begin{lem}\n\\label{le:s-mono}The sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$ satisfies \n\\begin{align*}\n    a_{t}(n, \\nu) \\geq a_{t+1}(n, \\nu) \n\\end{align*}\nfor all $t\\geq 0$. \n\n\\end{lem}","sha256_utf8":"e9d132c2365b7882543f4fde35c226bd1b8a6d8c827912f52a2c84391d088cde"}]},{"id":"S044","semantic_type":"proof","kind":"proof","title":"辅助序列单调性证明","source_units":[{"start":58966,"end":60821,"start_line":1243,"end_line":1279,"raw_tex":"\\begin{proof}\n\\textbf{Part I:} For $t=0$, the fact $\\nu\\in(0, 1]$ means\n\\begin{align*}\n    a_1(n, \\nu) = \\frac{1}{2(1+\\nu)} \\leq 1 = a_0(n, \\nu).\n\\end{align*}\n\\textbf{Part II:}\nFor all $t=1,\\dots,n-1$, we have\n\\begin{align*}\n        & a_{t+1}(n, \\nu) - a_{t}(n, \\nu) \\\\\n        & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=0}^{t}(a_j(n, \\nu))^{1+\\nu} + n - t - 1\\right) - \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=0}^{t-1}(a_j(n, \\nu))^{1+\\nu} + n - t\\right)\\\\\n        & = \\frac{1}{2(1+\\nu)n}\\left((a_{t}(n, \\nu))^{1+\\nu} - 1\\right)  \\leq 0,\n\\end{align*}     \nwhere the last inequality is based on Lemma \\ref{le:s-mono-000}. This indicates $a_{t+1}\\leq a_{t}$ for $t=1,\\dots,n-1$. \n\n\\textbf{Part III:} For all $t\\geq n$, we use induction to prove $a_{t+1}(n, \\nu)\\leq a_{t}(n, \\nu)$. \nFor the induction base, we can verify that\n    \\begin{align*}\n        & a_{n+1}(n, \\nu) - a_n(n, \\nu) \\\\\n        & = \\frac{1}{2(1+\\nu)n}\\sum_{j=1}^{n}(a_j(n, \\nu))^{1+\\nu} - \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=1}^{n-1}(a_j(n, \\nu))^{1+\\nu} + 1\\right)\\\\\n        & = \\frac{1}{2(1+\\nu)n}\\left((a_n(n, \\nu))^{1+\\nu} - 1\\right)\\\\\n        & \\leq 0,\n    \\end{align*}\n    where the last inequality is based on Lemma \\ref{le:s-mono-000}. \n    \n    For the induction step, we assume\n\\begin{align*}\n    a_{j+1}(n, \\nu) \\leq a_{j}(n, \\nu)\n\\end{align*}\nholds for all $j = n+1, \\cdots, t-1$ such that $t\\geq n+2$. Then we have\n    \\begin{align*}\n        & a_{t+1}(n, \\nu) - a_{t}(n, \\nu) \\\\\n         =&  \\frac{1}{2(1+\\nu)n}\\sum_{j=t-n + 1}^{t}(a_j(n, \\nu)) ^{1+\\nu} - \\frac{1}{2(1+\\nu)n}\\sum_{j=t-n}^{t-1}(a_j(n, \\nu))^{1+\\nu} \\leq 0,\n    \\end{align*}\nwhere the inequality is based on the induction hypothesis and the fact $a_{t+1}(n, \\nu) \\leq a_{t}(n, \\nu)$ for all~$t \\leq n - 1$ (which have shown in Part I).\n\nCombining the results of above three parts, we finish the proof of this lemma.\n\\end{proof}","sha256_utf8":"f151a7e894f8de7748cfb78333b2654f26ca838942832a93c909b71e33874680"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"验证初始单调性","source_units":[{"start":58966,"end":59125,"start_line":1243,"end_line":1247,"raw_tex":"\\begin{proof}\n\\textbf{Part I:} For $t=0$, the fact $\\nu\\in(0, 1]$ means\n\\begin{align*}\n    a_1(n, \\nu) = \\frac{1}{2(1+\\nu)} \\leq 1 = a_0(n, \\nu).\n\\end{align*}\n","sha256_utf8":"f52a0809ebe342fb0cf8986b2560b3365c49cc8ca186cfbdb96858d24f2de9df"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"首周期相邻项作差","source_units":[{"start":59125,"end":59631,"start_line":1248,"end_line":1255,"raw_tex":"\\textbf{Part II:}\nFor all $t=1,\\dots,n-1$, we have\n\\begin{align*}\n        & a_{t+1}(n, \\nu) - a_{t}(n, \\nu) \\\\\n        & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=0}^{t}(a_j(n, \\nu))^{1+\\nu} + n - t - 1\\right) - \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=0}^{t-1}(a_j(n, \\nu))^{1+\\nu} + n - t\\right)\\\\\n        & = \\frac{1}{2(1+\\nu)n}\\left((a_{t}(n, \\nu))^{1+\\nu} - 1\\right)  \\leq 0,\n\\end{align*}     \nwhere the last inequality is based on Lemma \\ref{le:s-mono-000}. This indicates $a_{t+1}\\leq a_{t}$ for $t=1,\\dots,n-1$. \n","sha256_utf8":"f8fa2e26c45ba3c3c71ccc75f427e9d750affc5dcffb40541ef199aaea71693d"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"验证跨周期归纳基","source_units":[{"start":59632,"end":60161,"start_line":1257,"end_line":1265,"raw_tex":"\\textbf{Part III:} For all $t\\geq n$, we use induction to prove $a_{t+1}(n, \\nu)\\leq a_{t}(n, \\nu)$. \nFor the induction base, we can verify that\n    \\begin{align*}\n        & a_{n+1}(n, \\nu) - a_n(n, \\nu) \\\\\n        & = \\frac{1}{2(1+\\nu)n}\\sum_{j=1}^{n}(a_j(n, \\nu))^{1+\\nu} - \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=1}^{n-1}(a_j(n, \\nu))^{1+\\nu} + 1\\right)\\\\\n        & = \\frac{1}{2(1+\\nu)n}\\left((a_n(n, \\nu))^{1+\\nu} - 1\\right)\\\\\n        & \\leq 0,\n    \\end{align*}\n    where the last inequality is based on Lemma \\ref{le:s-mono-000}. \n","sha256_utf8":"7bef279ceef8aee09b55c3946cf9b92fd3f811b4f9776d6055a18e4c7dbd20d2"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"滑动窗口单调归纳","source_units":[{"start":60166,"end":60821,"start_line":1267,"end_line":1279,"raw_tex":"    For the induction step, we assume\n\\begin{align*}\n    a_{j+1}(n, \\nu) \\leq a_{j}(n, \\nu)\n\\end{align*}\nholds for all $j = n+1, \\cdots, t-1$ such that $t\\geq n+2$. Then we have\n    \\begin{align*}\n        & a_{t+1}(n, \\nu) - a_{t}(n, \\nu) \\\\\n         =&  \\frac{1}{2(1+\\nu)n}\\sum_{j=t-n + 1}^{t}(a_j(n, \\nu)) ^{1+\\nu} - \\frac{1}{2(1+\\nu)n}\\sum_{j=t-n}^{t-1}(a_j(n, \\nu))^{1+\\nu} \\leq 0,\n    \\end{align*}\nwhere the inequality is based on the induction hypothesis and the fact $a_{t+1}(n, \\nu) \\leq a_{t}(n, \\nu)$ for all~$t \\leq n - 1$ (which have shown in Part I).\n\nCombining the results of above three parts, we finish the proof of this lemma.\n\\end{proof}","sha256_utf8":"3a5e1a977a11f34a58953254a33b778c9091160b6c1b1023f322a2d72f550570"}]},{"id":"S049","semantic_type":"lemma","kind":"lemma","title":"辅助序列周期幂次界","source_units":[{"start":60824,"end":61049,"start_line":1282,"end_line":1289,"raw_tex":"\\begin{lem}\n\\label{le:n-qua-con}\n     For the sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$, we have\n    \\begin{align*}\n    a_t(n, \\nu) \\leq \\frac{1}{2(1+\\nu)}(a_{t-n}(n, \\nu))^{1+\\nu}\n    \\end{align*}\n    for all $t\\geq n$.\n\\end{lem}","sha256_utf8":"1184dba5bc39e03ed93e3df72746aefd5a0eaf75c3a8c2ef2dfec5855f35abf2"}]},{"id":"S050","semantic_type":"proof","kind":"proof","title":"周期幂次界证明","source_units":[{"start":61050,"end":61760,"start_line":1290,"end_line":1305,"raw_tex":"\\begin{proof}\n    For all $t\\geq n$, the definition of $a_t(n, \\nu)$ implies\n    \\begin{align*}\n    & a_t(n, \\nu) = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=t-n}^{t-1}(a_j(n, \\nu))^{1+\\nu}\\right)  \\\\\n    & \\leq \\frac{1}{2(1+\\nu)} \\max\\{(a_{t-n}(n, \\nu))^{1+\\nu},\\cdots, (a_{t-1}(n, \\nu))^{1+\\nu}\\}.\n    \\end{align*}\n    Additionally, Lemma \\ref{le:s-mono} implies for all $t\\geq n$, we have\n    \\begin{align*}\n    \\max\\{(a_{t-n}(n, \\nu))^{1+\\nu},\\cdots, (a_{t-1}(n, \\nu))^{1+\\nu}\\} = (a_{t-n}(n, \\nu))^{1+\\nu}, \\quad t\\geq n.\n    \\end{align*}\n    Combining above results, we achieve\n    \\begin{align*}\n    a_t(n, \\nu) \\leq \\frac{1}{2(1+\\nu)}(a_{t-n}(n, \\nu))^{1+\\nu}\n    \\end{align*}\n    for all $t\\geq n$.\n\\end{proof}","sha256_utf8":"56c0a8589da70e1644c7dfad4df01b8c7302e1b90ebaa787b9bf0a7f287672ef"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"窗口平均由最大项控制","source_units":[{"start":61050,"end":61358,"start_line":1290,"end_line":1295,"raw_tex":"\\begin{proof}\n    For all $t\\geq n$, the definition of $a_t(n, \\nu)$ implies\n    \\begin{align*}\n    & a_t(n, \\nu) = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=t-n}^{t-1}(a_j(n, \\nu))^{1+\\nu}\\right)  \\\\\n    & \\leq \\frac{1}{2(1+\\nu)} \\max\\{(a_{t-n}(n, \\nu))^{1+\\nu},\\cdots, (a_{t-1}(n, \\nu))^{1+\\nu}\\}.\n    \\end{align*}\n","sha256_utf8":"8ed6ae86abe9e6a8fb9812a98813f2ec1251004f889128b0d4ea290f9620b6ea"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"单调性确定窗口最大项","source_units":[{"start":61358,"end":61760,"start_line":1296,"end_line":1305,"raw_tex":"    Additionally, Lemma \\ref{le:s-mono} implies for all $t\\geq n$, we have\n    \\begin{align*}\n    \\max\\{(a_{t-n}(n, \\nu))^{1+\\nu},\\cdots, (a_{t-1}(n, \\nu))^{1+\\nu}\\} = (a_{t-n}(n, \\nu))^{1+\\nu}, \\quad t\\geq n.\n    \\end{align*}\n    Combining above results, we achieve\n    \\begin{align*}\n    a_t(n, \\nu) \\leq \\frac{1}{2(1+\\nu)}(a_{t-n}(n, \\nu))^{1+\\nu}\n    \\end{align*}\n    for all $t\\geq n$.\n\\end{proof}","sha256_utf8":"fbc36879342afe2c435bb5985cd98bbf5593b3a149ac9fac9595670c5acced8a"}]},{"id":"S053","semantic_type":"lemma","kind":"lemma","title":"辅助序列线性收缩","source_units":[{"start":61762,"end":62057,"start_line":1307,"end_line":1316,"raw_tex":"\\begin{lem}\n\\label{le:linear-con} For the sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$, we have \n\\begin{align*}\na_{t+1}(n, \\nu)\\leq c_0 a_t(n, \\nu)\n\\end{align*}\nfor all $t\\geq n$, where\n\\begin{align*}\n    c_0 = 1- \\frac{1}{n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right).\n\\end{align*}\n\\end{lem}","sha256_utf8":"dd69ece0890b3138db25407012ed29e49ef70912534793ae703d99c4b693e807"}]},{"id":"S054","semantic_type":"proof","kind":"proof","title":"线性收缩证明","source_units":[{"start":62058,"end":63653,"start_line":1317,"end_line":1337,"raw_tex":"\\begin{proof}\n    For all $t\\geq n$, we have\n    \\begin{align}\\label{eq:lemma9-ieq}\n    a_t(n, \\nu) \\leq \\frac{1}{2(1+\\nu)}(a_{t-n}(n, \\nu))^{1+\\nu} \\leq \\frac{1}{2(1+\\nu)}a_{t-n}(n, \\nu), \n    \\end{align}\n    where the first inequality is based on Lemma \\ref{le:n-qua-con} and the second one is based on Lemma \\ref{le:s-mono-000}.\n    \n    Then we also have \n    \\begin{align*}\\small\\begin{split}\n        a_{t+1} \n        & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=t-n+1}^{t}(a_j(n, \\nu))^{1+\\nu}\\right)\\\\\n        & \\leq \\frac{1}{2(1+\\nu)n}\\left(\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}(a_{t-n}(n, \\nu))^{1+\\nu} + \\sum_{j=t-n+1}^{t-1}(a_j(n, \\nu))^{1+\\nu}\\right)\\\\\n        & = \\frac{1}{2(1+\\nu)n}\\left(\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}(a_{t-n}(n, \\nu))^{1+\\nu} + \\sum_{j=t-n+1}^{t-1}(a_j(n, \\nu))^{1+\\nu} + (a_{t-n}(n, \\nu))^{1+\\nu} - (a_{t-n}(n, \\nu))^{1+\\nu} \\right)\\\\\n        & = a_{t}(n, \\nu) + \\frac{1}{2(1+\\nu)n}\\left(\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}(a_{t-n}(n, \\nu))^{1+\\nu} - (a_{t-n}(n, \\nu))^{1+\\nu} \\right)\\\\\n        & = a_{t}(n, \\nu) - \\frac{1}{2(1+\\nu)n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right)(a_{t-n}(n, \\nu))^{1+\\nu}\\\\\n        & \\leq a_{t}(n, \\nu) - \\frac{1}{n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right)a_t(n, \\nu)\\\\\n        & = \\left(1- \\frac{1}{n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right)\\right)a_t(n, \\nu),\n     \\end{split}\n    \\end{align*}\n    for all $t\\geq n$, where the first inequality is based on equation (\\ref{eq:lemma9-ieq}) and the last inequality is based on Lemma \\ref{le:n-qua-con}. This finish the proof.\n\\end{proof}","sha256_utf8":"ab131d581e649a8b24e691f5d1524f1714afa572552323491dc3b94fb40ea15a"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"建立延迟项比例界","source_units":[{"start":62058,"end":62390,"start_line":1317,"end_line":1322,"raw_tex":"\\begin{proof}\n    For all $t\\geq n$, we have\n    \\begin{align}\\label{eq:lemma9-ieq}\n    a_t(n, \\nu) \\leq \\frac{1}{2(1+\\nu)}(a_{t-n}(n, \\nu))^{1+\\nu} \\leq \\frac{1}{2(1+\\nu)}a_{t-n}(n, \\nu), \n    \\end{align}\n    where the first inequality is based on Lemma \\ref{le:n-qua-con} and the second one is based on Lemma \\ref{le:s-mono-000}.\n","sha256_utf8":"2bc8df65597735fdf6cd44f4166c4c7b412c61fc029e94f23af0f60f160bae82"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"递推作差得到收缩因子","source_units":[{"start":62395,"end":63653,"start_line":1324,"end_line":1337,"raw_tex":"    Then we also have \n    \\begin{align*}\\small\\begin{split}\n        a_{t+1} \n        & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=t-n+1}^{t}(a_j(n, \\nu))^{1+\\nu}\\right)\\\\\n        & \\leq \\frac{1}{2(1+\\nu)n}\\left(\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}(a_{t-n}(n, \\nu))^{1+\\nu} + \\sum_{j=t-n+1}^{t-1}(a_j(n, \\nu))^{1+\\nu}\\right)\\\\\n        & = \\frac{1}{2(1+\\nu)n}\\left(\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}(a_{t-n}(n, \\nu))^{1+\\nu} + \\sum_{j=t-n+1}^{t-1}(a_j(n, \\nu))^{1+\\nu} + (a_{t-n}(n, \\nu))^{1+\\nu} - (a_{t-n}(n, \\nu))^{1+\\nu} \\right)\\\\\n        & = a_{t}(n, \\nu) + \\frac{1}{2(1+\\nu)n}\\left(\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}(a_{t-n}(n, \\nu))^{1+\\nu} - (a_{t-n}(n, \\nu))^{1+\\nu} \\right)\\\\\n        & = a_{t}(n, \\nu) - \\frac{1}{2(1+\\nu)n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right)(a_{t-n}(n, \\nu))^{1+\\nu}\\\\\n        & \\leq a_{t}(n, \\nu) - \\frac{1}{n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right)a_t(n, \\nu)\\\\\n        & = \\left(1- \\frac{1}{n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right)\\right)a_t(n, \\nu),\n     \\end{split}\n    \\end{align*}\n    for all $t\\geq n$, where the first inequality is based on equation (\\ref{eq:lemma9-ieq}) and the last inequality is based on Lemma \\ref{le:n-qua-con}. This finish the proof.\n\\end{proof}","sha256_utf8":"e4d06c506853bb4f90c0991eb678ac868a003886a0ab80a7d67ec790963612cd"}]},{"id":"S057","semantic_type":"lemma","kind":"lemma","title":"收缩因子的周期增强","source_units":[{"start":63656,"end":64017,"start_line":1339,"end_line":1350,"raw_tex":"\\begin{lem}\n\\label{le:imp-c} For the sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$, if there exists $c_1\\in(0,1)$ and $t_0\\geq 0$ such that\n\\begin{align}\\label{eq:ieq-t0n2n}\na_{t+1}(n, \\nu) \\leq c_1 a_t(n, \\nu)\n\\end{align}\nfor all $t\\geq t_0 + n$, \nthen we have\n\\begin{align*}\na_{t+1}(n, \\nu) \\leq c_1^{1+\\nu} a_t(n, \\nu)\n\\end{align*}\nfor all $t\\geq t_0 + 2n$.\n\\end{lem}","sha256_utf8":"254345bdd7ca841e87ba467447efc119869455e36664fb8b737ef7ace3e4780c"}]},{"id":"S058","semantic_type":"proof","kind":"proof","title":"收缩因子增强证明","source_units":[{"start":64018,"end":64398,"start_line":1351,"end_line":1359,"raw_tex":"\\begin{proof}\n    For all  $t\\geq t_0 + 2n$, we have\n    \\begin{align*}\n        & a_{t+1} (n, \\nu)\n         = \\frac{1}{2(1+\\nu)n}\\sum_{j=t-n+1}^{t}(a_j(n, \\nu))^{1+\\nu} \\\\\n        & \\leq \\frac{1}{2(1+\\nu)n}\\sum_{j=t-n}^{t-1}c_1^{1+\\nu}(a_j(n, \\nu))^{1+\\nu} = c_1^{1+\\nu} a_t(n, \\nu),\n    \\end{align*}\n    where the inequality is based on equation (\\ref{eq:ieq-t0n2n}).\n\\end{proof}","sha256_utf8":"edb9bf0c9090d87f4b0185c53af7c329b6239f411c5c9483f1510a822f16b163"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"移位求和提升收缩幂次","source_units":[{"start":64018,"end":64398,"start_line":1351,"end_line":1359,"raw_tex":"\\begin{proof}\n    For all  $t\\geq t_0 + 2n$, we have\n    \\begin{align*}\n        & a_{t+1} (n, \\nu)\n         = \\frac{1}{2(1+\\nu)n}\\sum_{j=t-n+1}^{t}(a_j(n, \\nu))^{1+\\nu} \\\\\n        & \\leq \\frac{1}{2(1+\\nu)n}\\sum_{j=t-n}^{t-1}c_1^{1+\\nu}(a_j(n, \\nu))^{1+\\nu} = c_1^{1+\\nu} a_t(n, \\nu),\n    \\end{align*}\n    where the inequality is based on equation (\\ref{eq:ieq-t0n2n}).\n\\end{proof}","sha256_utf8":"edb9bf0c9090d87f4b0185c53af7c329b6239f411c5c9483f1510a822f16b163"}]},{"id":"S060","semantic_type":"assumption","kind":"assumption","title":"已有统一线性收缩","source_units":[{"start":63668,"end":63896,"start_line":1340,"end_line":1344,"raw_tex":"\\label{le:imp-c} For the sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$, if there exists $c_1\\in(0,1)$ and $t_0\\geq 0$ such that\n\\begin{align}\\label{eq:ieq-t0n2n}\na_{t+1}(n, \\nu) \\leq c_1 a_t(n, \\nu)\n\\end{align}\nfor all $t\\geq t_0 + n$, \n","sha256_utf8":"b92fde10b8a46dc177c8d2ec6c73226be24497a1f3afa2899df79606bd12dc40"}]},{"id":"S061","semantic_type":"lemma","kind":"lemma","title":"辅助序列超线性收缩","source_units":[{"start":64400,"end":64758,"start_line":1361,"end_line":1370,"raw_tex":"\\begin{lem}\n\\label{le:super-con} For the sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$,\nwe have the superlinear convergence\n\\begin{align*}\na_{t+1}(n, \\nu) \\leq c^{(1+\\nu)^{\\left(\\floor{\\frac{t-1}{n}}-1\\right)}}a_t(n, \\nu) \n\\end{align*}\nfor all $t\\geq n$, where\n\\begin{align*}\n    c = 1- \\frac{1}{n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right).\n\\end{align*}\n","sha256_utf8":"c365f72b889542653978b07bc0abe2b40c0ab6f562f39f6cf800a4fa8998901f"},{"start":65751,"end":65761,"start_line":1395,"end_line":1396,"raw_tex":"\n\\end{lem}","sha256_utf8":"401658723a1e057444124cf77a6be177a3c9cb2efee9af44c2830fc4a15ddaba"}]},{"id":"S062","semantic_type":"proof","kind":"proof","title":"辅助序列超线性证明","source_units":[{"start":64758,"end":65751,"start_line":1371,"end_line":1395,"raw_tex":"\\begin{proof}\n    According to Lemma \\ref{le:linear-con}, we have\n    \\begin{align*}\n        a_{t+1}(n, \\nu)\\leq c a_t(n, \\nu) \n        \\quad \\text{for all}~~t \\geq n.\n    \\end{align*}\n    According to Lemma \\ref{le:imp-c}, we have\n    \\begin{align*}\n        a_{t+1}(n, \\nu) & \\leq c^{1+\\nu} a_t(n, \\nu) \\quad \\text{for all}~~t \\geq 2n,\\\\\n        a_{t+1}(n, \\nu) & \\leq c^{(1+\\nu)^2} a_t(n, \\nu) \\quad \\text{for all}~~t \\geq 3n,\\\\\n        a_{t+1}(n, \\nu) & \\leq c^{(1+\\nu)^3} a_t(n, \\nu) \\quad \\text{for all}~~t \\geq 4n,\\\\\n        \\cdots &\n    \\end{align*}\n    which implies\n    \\begin{align*}\n        a_{t+1}(n, \\nu) \\leq c^{(1+\\nu)^{\\left(\\lfloor {t}/{n}  \\rfloor-1\\right)}}a_t(n, \\nu)\n    \\end{align*}\n    for all $t \\geq n$.\n\n    The superlinear convergence of the sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$ can be verify by the fact\n    \\begin{align*}\n        \\lim_{t\\to\\infty}c^{(1+\\nu)^{\\left(\\lfloor {t}/{n}  \\rfloor-1\\right)}}=0.\n    \\end{align*}\n    Hence, we finish the proof.\n\\end{proof}","sha256_utf8":"f684422c801e569e5b8e69c3736b4907114b8b740e2625dab289acae7927f988"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"取得初始线性因子","source_units":[{"start":64758,"end":64943,"start_line":1371,"end_line":1376,"raw_tex":"\\begin{proof}\n    According to Lemma \\ref{le:linear-con}, we have\n    \\begin{align*}\n        a_{t+1}(n, \\nu)\\leq c a_t(n, \\nu) \n        \\quad \\text{for all}~~t \\geq n.\n    \\end{align*}\n","sha256_utf8":"2d594f71126a59594e81c320002ef0512e196ac5adad6277146db0f19a70ab51"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"逐周期增强为显式界","source_units":[{"start":64943,"end":65487,"start_line":1377,"end_line":1388,"raw_tex":"    According to Lemma \\ref{le:imp-c}, we have\n    \\begin{align*}\n        a_{t+1}(n, \\nu) & \\leq c^{1+\\nu} a_t(n, \\nu) \\quad \\text{for all}~~t \\geq 2n,\\\\\n        a_{t+1}(n, \\nu) & \\leq c^{(1+\\nu)^2} a_t(n, \\nu) \\quad \\text{for all}~~t \\geq 3n,\\\\\n        a_{t+1}(n, \\nu) & \\leq c^{(1+\\nu)^3} a_t(n, \\nu) \\quad \\text{for all}~~t \\geq 4n,\\\\\n        \\cdots &\n    \\end{align*}\n    which implies\n    \\begin{align*}\n        a_{t+1}(n, \\nu) \\leq c^{(1+\\nu)^{\\left(\\lfloor {t}/{n}  \\rfloor-1\\right)}}a_t(n, \\nu)\n    \\end{align*}\n    for all $t \\geq n$.\n","sha256_utf8":"d9ebdffaadd16270e990fa8c29bc74f51607795105349de6ad29cc1c99de9dc4"}]},{"id":"S065","semantic_type":"proof_step","kind":"proof_step","title":"收缩因子趋于零","source_units":[{"start":65488,"end":65751,"start_line":1390,"end_line":1395,"raw_tex":"    The superlinear convergence of the sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$ can be verify by the fact\n    \\begin{align*}\n        \\lim_{t\\to\\infty}c^{(1+\\nu)^{\\left(\\lfloor {t}/{n}  \\rfloor-1\\right)}}=0.\n    \\end{align*}\n    Hence, we finish the proof.\n\\end{proof}","sha256_utf8":"5c5b55b61bb3b27c7befc714da715b8f4dd85f54eb586ceb98636a71ca9a199d"}]},{"id":"S066","semantic_type":"lemma","kind":"lemma","title":"低秩更新内部矩阵可逆","source_units":[{"start":69662,"end":70230,"start_line":1476,"end_line":1489,"raw_tex":"\\begin{lem}\n\\label{le:via-update}\n    Following the setting of Theorem \\ref{thm:holder-IGN-1}, if the matrices $\\mH^t$ and $\\mH^{t+1}$ are non-singular, then the matrix $\\mI + {\\mV^t}^\\top\\mG^{t}\\mU^t$ is also non-singular, where\n    \\begin{align*}\n    \\mU^t = \\begin{bmatrix}\n        - \\vg_{i_t}(\\vz_{i_t}^t) \\!& \n        \\vg_{i_t}(\\vx^{t+1})\n    \\end{bmatrix}\\in\\BR^{d\\times 2},~~\n    \\mV^t = \\begin{bmatrix}\n        \\vg_{i_t}(\\vz_{i_t}^t) \\!& \n        \\vg_{i_t}(\\vx^{t+1})\n    \\end{bmatrix}\\in\\BR^{d\\times 2}~~\\text{and}~~i_t ={t\\%n} + 1.\n    \\end{align*}\n\\end{lem}","sha256_utf8":"5da9f68b009678f4bedc639f6b4b40c28b38edc9cb33b7d2123678d7e5953a4b"}]},{"id":"S067","semantic_type":"proof","kind":"proof","title":"内部矩阵可逆性证明","source_units":[{"start":70231,"end":71061,"start_line":1490,"end_line":1504,"raw_tex":"\\begin{proof}\n    The recursion of $\\mH^t$ and the definition of $\\mU^t$ and $\\mV^t$ imply\n    \\begin{align*}\n        \\mH^{t+1} = \\mH^t - \\vg_{i_t}(\\vz_{i_t}^t)\\vg_{i_t}(\\vz_{i_t}^t)^\\top + \\vg_{i_t}(\\vx^{t+1})\\vg_{i_t}(\\vx^{t+1})^\\top = \\mH^t + \\mU^t{\\mV^t}^\\top.\n    \\end{align*}\n    Since we assume matrices $\\mH^t$ and $\\mH^{t+1}$ are non-singular, applying the matrix determinant lemma \\cite[Section 9.1.2]{petersen2008matrix} on above equation leads to\n    \\begin{align*}\n        \\det(\\mH^{t+1})=\\det(\\mH^t + \\mU^t{\\mV^t}^\\top) = \\det(\\mI + {\\mV^t}^\\top(\\mH^t)^{-1}\\mU^t)\\det(\\mH^t).\n    \\end{align*}\n    Then the definition $\\mG^t={\\mH^t}^{-1}$ implies\n    \\begin{align*}\n        \\det(\\mI + {\\mV^t}^\\top\\mG^t\\mU^t) = \\det(\\mI + {\\mV^t}^\\top{\\mH^t}^{-1}\\mU^t) \\neq 0\n    \\end{align*}\n    which finish the proofs.\n\\end{proof}","sha256_utf8":"1ce8784f90b882fe231fe7bab7304720c9a725e17b07a6c5b9e39e2985beac3f"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"写成低秩矩阵更新","source_units":[{"start":70231,"end":70513,"start_line":1490,"end_line":1494,"raw_tex":"\\begin{proof}\n    The recursion of $\\mH^t$ and the definition of $\\mU^t$ and $\\mV^t$ imply\n    \\begin{align*}\n        \\mH^{t+1} = \\mH^t - \\vg_{i_t}(\\vz_{i_t}^t)\\vg_{i_t}(\\vz_{i_t}^t)^\\top + \\vg_{i_t}(\\vx^{t+1})\\vg_{i_t}(\\vx^{t+1})^\\top = \\mH^t + \\mU^t{\\mV^t}^\\top.\n    \\end{align*}\n","sha256_utf8":"167ee546d1ae41b6a95d046445ed7df11b6d3031e0935f9797c26e6a99c8a063"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"应用矩阵行列式引理","source_units":[{"start":70513,"end":70838,"start_line":1495,"end_line":1498,"raw_tex":"    Since we assume matrices $\\mH^t$ and $\\mH^{t+1}$ are non-singular, applying the matrix determinant lemma \\cite[Section 9.1.2]{petersen2008matrix} on above equation leads to\n    \\begin{align*}\n        \\det(\\mH^{t+1})=\\det(\\mH^t + \\mU^t{\\mV^t}^\\top) = \\det(\\mI + {\\mV^t}^\\top(\\mH^t)^{-1}\\mU^t)\\det(\\mH^t).\n    \\end{align*}\n","sha256_utf8":"6394d661de895fbfd59e7dd223e2d18536bce5b43608454c073e19e1cb2f97f1"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"内部行列式非零","source_units":[{"start":70838,"end":71061,"start_line":1499,"end_line":1504,"raw_tex":"    Then the definition $\\mG^t={\\mH^t}^{-1}$ implies\n    \\begin{align*}\n        \\det(\\mI + {\\mV^t}^\\top\\mG^t\\mU^t) = \\det(\\mI + {\\mV^t}^\\top{\\mH^t}^{-1}\\mU^t) \\neq 0\n    \\end{align*}\n    which finish the proofs.\n\\end{proof}","sha256_utf8":"e98edf499c9db5a740593a515e439e7cfc76b77af143a0394733f16812f625bf"}]},{"id":"S071","semantic_type":"lemma","kind":"lemma","title":"单步误差幂次估计","source_units":[{"start":71185,"end":71596,"start_line":1508,"end_line":1517,"raw_tex":"\\begin{lem}\\label{le:main-iter-1} \nUnder Assumptions \\ref{asm:L-f} and \\ref{asm:holder-g}, we\nassume matrices $\\{\\mH^j\\}_{j=0}^t$ are non-singular and run \\IGN~(Algorithm \\ref{alg:IGN-1}), then it holds\n\\begin{align*}\n    \\Norm{\\vx^{t+1} - \\vx^*}\n     \\leq \\frac{L_f\\fH_\\nu}{1 \n     +\\nu}\\Norm{\\mG^t}\\sum_{i=1}^n\\Norm{\\vz_i^t-\\vx^*}^{1+\\nu},\n    \\end{align*}\n    where $\\mG^t=\\left(\\mH^t\\right)^{-1}$.\n\\end{lem}","sha256_utf8":"2294eb786446e0fad888d9600fd8c3e53d45b3865276373e1b72f62ea5d8f17d"}]},{"id":"S072","semantic_type":"proof","kind":"proof","title":"单步误差估计证明","source_units":[{"start":71597,"end":72830,"start_line":1518,"end_line":1535,"raw_tex":"\\begin{proof}\n    Subtracting the term $\\vx^*$ on both sides of equation (\\ref{eq:ign-update-0}), we have\n    \\begin{align*}\n    \\vx^{t+1} - \\vx^*\n    & = \\mG^t\\left(\\sum_{i=1}^n \\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top(\\vz_i^t-\\vx^*) - \\sum_{i=1}^n f_i(\\vz_i^t)\\vg_i(\\vz_i^t)\\right)\\\\\n    & = \\mG^t\\left(\\sum_{i=1}^n \\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top(\\vz_i^t-\\vx^*) - \\sum_{i=1}^n f_i(\\vz_i^t)\\vg_i(\\vz_i^t) + \\sum_{i=1}^n f_i(\\vx^*)\\vg_i(\\vz_i^t)\\right)\\\\\n    & = \\mG^t\\sum_{i=1}^n\\left( \\vg_i(\\vz_i^t)^\\top(\\vz_i^t-\\vx^*) - f_i(\\vz_i^t) + f_i(\\vx^*)\\right)\\vg_i(\\vz_i^t).\n    \\end{align*}\n\n    Taking the norm on the both sides of above results, we have\n    \\begin{align*}\n    \\Norm{\\vx^{t+1} - \\vx^*}\n    & = \\Norm{\\mG^t\\sum_{i=1}^n\\left( \\vg_i(\\vz_i^t)^\\top(\\vz_i^t-\\vx^*) - f_i(\\vz_i^t) + f_i(\\vx^*)\\right)\\vg_i(\\vz_i^t)}\\\\\n    & \\leq \\Norm{\\mG^t}\\Norm{\\sum_{i=1}^n\\left( \\vg_i(\\vz_i^t)^\\top(\\vz_i^t-\\vx^*) - f_i(\\vz_i^t) + f_i(\\vx^*)\\right)\\vg_i(\\vz_i^t)}\\\\\n    & \\leq \\frac{L_f\\fH_\\nu}{1 + \\nu}\\Norm{\\mG^t}\\sum_{i=1}^n\\Norm{\\vz_i^t-\\vx^*}^{1+\\nu}\n    \\end{align*}    \n    where the first inequality comes from the property of matrix norm, the second inequality is based on Lemma \\ref{le:power-bound} and \\ref{le:b-grad}.\n\\end{proof}","sha256_utf8":"939bc550fb5c4065c4038401ee9fa3eec647c8f31d6d9e07997489ffa877712f"}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","title":"利用零残差重排误差","source_units":[{"start":71597,"end":72182,"start_line":1518,"end_line":1525,"raw_tex":"\\begin{proof}\n    Subtracting the term $\\vx^*$ on both sides of equation (\\ref{eq:ign-update-0}), we have\n    \\begin{align*}\n    \\vx^{t+1} - \\vx^*\n    & = \\mG^t\\left(\\sum_{i=1}^n \\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top(\\vz_i^t-\\vx^*) - \\sum_{i=1}^n f_i(\\vz_i^t)\\vg_i(\\vz_i^t)\\right)\\\\\n    & = \\mG^t\\left(\\sum_{i=1}^n \\vg_i(\\vz_i^t)\\vg_i(\\vz_i^t)^\\top(\\vz_i^t-\\vx^*) - \\sum_{i=1}^n f_i(\\vz_i^t)\\vg_i(\\vz_i^t) + \\sum_{i=1}^n f_i(\\vx^*)\\vg_i(\\vz_i^t)\\right)\\\\\n    & = \\mG^t\\sum_{i=1}^n\\left( \\vg_i(\\vz_i^t)^\\top(\\vz_i^t-\\vx^*) - f_i(\\vz_i^t) + f_i(\\vx^*)\\right)\\vg_i(\\vz_i^t).\n    \\end{align*}\n","sha256_utf8":"fc50c803a096d4dcbca2bec20475865a0b79367d288e064fec3334a7546f0723"}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","title":"余项与梯度界控制误差","source_units":[{"start":72183,"end":72830,"start_line":1527,"end_line":1535,"raw_tex":"    Taking the norm on the both sides of above results, we have\n    \\begin{align*}\n    \\Norm{\\vx^{t+1} - \\vx^*}\n    & = \\Norm{\\mG^t\\sum_{i=1}^n\\left( \\vg_i(\\vz_i^t)^\\top(\\vz_i^t-\\vx^*) - f_i(\\vz_i^t) + f_i(\\vx^*)\\right)\\vg_i(\\vz_i^t)}\\\\\n    & \\leq \\Norm{\\mG^t}\\Norm{\\sum_{i=1}^n\\left( \\vg_i(\\vz_i^t)^\\top(\\vz_i^t-\\vx^*) - f_i(\\vz_i^t) + f_i(\\vx^*)\\right)\\vg_i(\\vz_i^t)}\\\\\n    & \\leq \\frac{L_f\\fH_\\nu}{1 + \\nu}\\Norm{\\mG^t}\\sum_{i=1}^n\\Norm{\\vz_i^t-\\vx^*}^{1+\\nu}\n    \\end{align*}    \n    where the first inequality comes from the property of matrix norm, the second inequality is based on Lemma \\ref{le:power-bound} and \\ref{le:b-grad}.\n\\end{proof}","sha256_utf8":"522e0823970811c26fea19fbd740f0fdb5f82a0509fd9dc5d1337a67c7bc331b"}]},{"id":"S075","semantic_type":"theorem","kind":"theorem","title":"迭代稳定性与序列支配","source_units":[{"start":73124,"end":73855,"start_line":1543,"end_line":1557,"raw_tex":"\\begin{thm}\n\\label{thm:holder-IGN-1-1}\n    Under the Assumption \\ref{asm:L-f}, \\ref{asm:holder-g} and \\ref{asm:b-J-sing}, we run \\IGN~(Algorithm \\ref{alg:IGN-1}) with initialization $\\vx^0\\in\\BR^d$ and~$\\mH^0=\\mJ(\\vx^0)^\\top\\mJ(\\vx^0)$ such that\n    \\begin{align*}\n    \\Norm{\\vx^0 - \\vx^*}\\leq \\left(\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\right)^{{1}/{\\nu}},\n    \\end{align*} \n    then it holds \n    \\begin{align*}\n    \\sigma_{\\min}(\\mI + (\\mV^t)^\\top (\\mH^t)^{-1}\\mU^t) > 0,\\quad\n    \\mH^t \\succeq \\frac{\\mu^2}{2}\\mI\\quad\n    \\text{and}\\quad\n    \\Norm{\\vx^{t}-\\vx^*}\\leq a_{t+1}(n, \\nu)\\Norm{\\vx^0-\\vx^*}\n    \\end{align*}\n    for all $t \\geq 0$, where sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$ is defined in equation (\\ref{eq:series}). \n\\end{thm}","sha256_utf8":"3097733d3cea1d2d02f7ec6de0b4b75448b4fa8d99b9f8f70211cbf84e751f5d"}]},{"id":"S076","semantic_type":"proof","kind":"proof","title":"稳定性完整归纳证明","source_units":[{"start":73856,"end":83677,"start_line":1558,"end_line":1768,"raw_tex":"\\begin{proof}\n\nWe first show\n\\begin{align}\\label{eq:Hxtxstar0}\n\\mH^t \\succeq \\frac{\\mu^2}{2}\\mI\\qquad\n\\text{and}\\qquad\n\\Norm{\\vx^{t}-\\vx^*}\\leq a_{t+1}(n, \\nu)\\Norm{\\vx^0-\\vx^*}\n\\end{align}\nholds for all $t\\geq 0$. We split the proof of results (\\ref{eq:Hxtxstar0}) into the following three parts.\n\n\\textbf{Part I:} For $t=0$, \nthe initialization and the fact $a_0=1$ leads to \n\\begin{align*}\n \\Norm{\\vx^0 - \\vx^*} = a_0(n, \\nu)\\Norm{\\vx^0 - \\vx^*}.\n\\end{align*}\n\n\n\\textbf{Part II:} For all $t=0,\\cdots, n-1$, we use induction to prove the results of \n\\begin{align}\\label{eq:Hxtxstar}\n\\mH^t \\succeq \\frac{\\mu^2}{2}\\mI\\qquad\n\\text{and}\\qquad\n\\Norm{\\vx^{t+1}-\\vx^*}\\leq a_{t+1}(n, \\nu)\\Norm{\\vx^0-\\vx^*}.\n\\end{align}\nFor the induction base, we can apply Lemma \\ref{le:lb-Ht-new-1} to verify\n\\begin{align*}\n    \\sigma_{\\min}(\\mH^0) &\\geq  \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^0 - \\vx^*}^\\nu\\\\\n    & = \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vx^0 - \\vx^*}^\\nu\\\\\n    & \\geq \\mu^2 - 2L_f\\fH_\\nu n \\frac{\\mu^2}{4L_f\\fH_\\nu n}\\\\\n    & = \\mu^2 - \\frac{\\mu^2}{2}\\\\\n    & = \\frac{\\mu^2}{2}.\n\\end{align*}\n\n\nThis implies\n\\begin{align}\\label{eq:boundG0}\n\\mH^0 \\succeq \\frac{\\mu^2}{2} \\qquad \\text{and}\\qquad\n\\Norm{\\mG^0} = \\Norm{(\\mH^0)^{-1}} \\leq \\frac{2}{\\mu^2}.\n\\end{align}\nAccording to Lemma \\ref{le:main-iter-1}, we have\n\\begin{align*}\n    \\Norm{\\vx^1 - \\vx^*} \n    &\\leq \\frac{L_f\\fH_\\nu}{1+\\nu}\\Norm{\\mG^0}\\sum_{i=1}^n\\Norm{\\vz_i^0-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{L_f\\fH_\\nu}{1+\\nu}\\cdot\\frac{2}{\\mu^2}\\cdot\\sum_{i=1}^n\\Norm{\\vz_i^0-\\vx^*}^{1+\\nu}\\\\\n    & = \\frac{L_f\\fH_\\nu}{1+\\nu}\\cdot\\frac{2}{\\mu^2}\\cdot n\\Norm{\\vx^0-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{nL_f\\fH_\\nu}{1+\\nu}\\cdot\\frac{2}{\\mu^2}\\cdot\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\Norm{\\vx^0-\\vx^*}\\\\\n    & = \\frac{1}{2(1+\\nu)}\\Norm{\\vx^0-\\vx^*}\\\\\n    & = a_1(n, \\nu)\\Norm{\\vx^0 - \\vx^*},\n\\end{align*}\nwhere the first inequality is based on equation (\\ref{eq:boundG0}) and the second inequality is based on initial condition. \nTherefore, the induction base holds\n\nFor the induction step, we assume\n\\begin{align*}\n    \\mH^j \\succeq \\frac{\\mu^2}{2} \\mI \\qquad\\text{and}\\qquad\\Norm{\\vx^{j+1} - \\vx^*} \\leq a_{j+1}(n, \\nu)\\Norm{\\vx^0 - \\vx^*}\n\\end{align*}\nhold for all $j = 2, \\cdots, t-1$ such that $t \\leq n - 1$. Therefore, the update (\\ref{eq:update-1}) means\n\\begin{align}\\label{eq:vziii}\n    \\vz_i^{t}=\n    \\begin{cases}\n        \\vx^i,~~~~&1 \\leq i\\leq t,\\\\\n        \\vx^0,~~~~&t < i \\leq n. \n    \\end{cases}\n\\end{align}\nThe induction hypothesis leads to\n\\begin{align*}\n    \\Norm{\\vx^j-\\vx^*}^{\\nu}\\leq (a_j(n, \\nu))^{\\nu}\\Norm{\\vx^0-\\vx^*}^{\\nu} \\leq  \\Norm{\\vx^0-\\vx^*}^{\\nu}\\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n}, \n\\end{align*}\nfor $j = 1, \\cdots, t-1$, where the second is based on Lemma \\ref{le:s-mono-000} and the third comes from the initial condition. \nCombining with the result of (\\ref{eq:vziii}), we achive\n\\begin{align*}\n    \\Norm{\\vz_i^{t} - \\vx^*}^{\\nu} \\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n}.\n\\end{align*}\n\nAccording to Lemma \\ref{le:lb-Ht-new-1}, we have\n\\begin{align*}\n    \\sigma_{\\min}(\\mH^{t}) &\\geq  \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^{t} - \\vx^*}^\\nu\\\\\n    & \\geq \\mu^2 - 2L_f\\fH_\\nu n \\frac{\\mu^2}{4L_f\\fH_\\nu n}\\\\\n    & = \\mu^2 - \\frac{\\mu^2}{2}\\\\\n    & = \\frac{\\mu^2}{2},\n\\end{align*}\n\nwhere the second inequality comes from the initial condition.\nTherefore, we have\n\\begin{align*}\n    \\mH^{t}  \\succeq \\frac{\\mu^2}{2}\\mI \\qquad \\text{and}\\qquad\n    \\Norm{\\mG^{t}}  = \\Norm{(\\mH^{t})^{-1}} \\leq \\frac{2}{\\mu^2}.\n\\end{align*}\nAccording to Lemma \\ref{le:main-iter-1}, we have\n\\begin{align*}\n    \\Norm{\\vx^{t+1} - \\vx^*} \n    & \\leq \\frac{L_f\\fH_\\nu }{1+\\nu}\\Norm{\\mG^{t}}\\sum_{i=1}^n\\Norm{\\vz_i^{t}-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{L_f\\fH_\\nu }{1+\\nu}\\frac{2}{\\mu^2}\\sum_{i=1}^n\\Norm{\\vz_i^{t}-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu}{(1+\\nu)\\mu^2}\\left(\\sum_{j=1}^{t}\\Norm{\\vx^{j}-\\vx^*}^{1+\\nu} + (n - t)\\Norm{\\vx^0 - \\vx^*}^{1+\\nu}\\right)\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\left(\\sum_{j=1}^{t} (a_j(n, \\nu))^{1+\\nu}\\Norm{\\vx^0 - \\vx^*}^{1+\\nu} + (n - t) \\Norm{\\vx^0 - \\vx^*}^{1+\\nu}\\right)\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\left(\\sum_{j=1}^{t}(a_j(n, \\nu))^{1+\\nu} + n - t\\right)\\Norm{\\vx^0 - \\vx^*}\\\\\n    & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=1}^{t}(a_j(n, \\nu))^{1+\\nu} + n - t\\right)\\Norm{\\vx^0 - \\vx^*}\\\\\n    & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=0}^{t}(a_j(n, \\nu))^{1+\\nu} + n - t - 1\\right)\\Norm{\\vx^0 - \\vx^*}\\\\\n    & = a_{t+1}(n, \\nu)\\Norm{\\vx^0 - \\vx^*},\n\\end{align*}\nwhere the last equality comes from the fact $a_0(n, \\nu)=1$. \nTherefore, we finish the induction.\n\n\n\\textbf{Part III:} For all $t \\geq n$, we use induction to prove \n\\begin{align*}\n\\mH^t\\succeq(\\mu^2/2)\\mI\\qquad\\text{and}\\qquad\\Norm{\\vx^{t+1} - \\vx^*}\\leq a_{t+1}(n, \\nu)\\Norm{\\vx^0 - \\vx^*}.    \n\\end{align*}\nFor the induction base, we can verify that it holds (from the result of Part II) \n\\begin{align*}\n\\mH^j \\succeq \\frac{\\mu^2}{2}\\mI \\qquad \\text{for all~~} j=0,\\dots,n - 1,\n\\end{align*}\nand\n\\begin{align*}\n\\Norm{\\vx^j-\\vx^*}\\leq a_j(n, \\nu)\\Norm{\\vx^0-\\vx^*} \\qquad \\text{for all~~} j=1,\\dots,n.\n\\end{align*}\nThen we have\n\\begin{align*}\n    \\Norm{\\vx^j-\\vx^*}^{\\nu}\\leq (a_j(n, \\nu))^{\\nu}\\Norm{\\vx^0-\\vx^*}^{\\nu} \\leq \\Norm{\\vx^0-\\vx^*}^{\\nu} \\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n}, \\quad \\text{for all~~} j=1,\\dots,n,\n\\end{align*}\nwhere the second inequality is based on Lemma \\ref{le:s-mono-000} and the third inequality is based on the initial condition.\n\nFrom Eq.~\\ref{eq:update-1}, we have\n\\begin{align*}\n    \\vz_i^{n}=\\vx^i \\qquad\\text{for all~~} i\\in[n].\n\\end{align*}\n\nTherefore, we have\n\\begin{align*}\n    \\Norm{\\vz_i^{n} - \\vx^*}^{\\nu} \\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n}, \\qquad\\text{for all~~} i\\in[n].\n\\end{align*}\nAccording to Lemma \\ref{le:lb-Ht-new-1}, we have\n\\begin{align*}\n    \\sigma_{\\min}(\\mH^{n}) &\\geq  \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^{n} - \\vx^*}^\\nu\\\\\n    & \\geq \\mu^2 - 2L_f\\fH_\\nu n \\frac{\\mu^2}{4L_f\\fH_\\nu n}\\\\\n    & \\geq \\mu^2 - \\frac{\\mu^2}{2} = \\frac{\\mu^2}{2},\n\\end{align*}\n\nwhich implies\n\\begin{align*}\n    \\mH^{n}  \\succeq \\frac{\\mu^2}{2}\\mI \\qquad\\text{and}\\qquad\n    \\Norm{\\mG^{n}}  = \\Norm{(\\mH^{n})^{-1}} \\leq \\frac{2}{\\mu^2}.\n\\end{align*}\nAccording to Lemma \\ref{le:main-iter-1}, we have\n\\begin{align*}\n    \\Norm{\\vx^{n+1} - \\vx^*}\n    & \\leq \\frac{L_f\\fH_\\nu }{1+\\nu}\\Norm{\\mG^n}\\sum_{i=1}^n\\Norm{\\vz_i^n-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\sum_{i=1}^n\\Norm{\\vz_i^n-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\left(\\sum_{j=1}^n (a_j(n, \\nu))^{1+\\nu}\\Norm{\\vx^0 - \\vx^*}^{1+\\nu}\\right)\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\left(\\sum_{j=1}^{n}(a_j(n, \\nu))^{1+\\nu}\\right)\\Norm{\\vx^0-\\vx^*}\\\\\n    & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=1}^{n}(a_j(n, \\nu))^{1+\\nu}\\right)\\Norm{\\vx^0-\\vx^*}\\\\\n    & = a_{n+1}(n, \\nu)\\Norm{\\vx^0-\\vx^*}.\n\\end{align*}\nHence, we have shown the induction base holds. \n\nFor the induction step, we assume\n\\begin{align*}\n    \\mH^j \\succeq \\frac{\\mu^2}{2} \\mI \\qquad\\text{and}\\qquad\\Norm{\\vx^{j+1} - \\vx^*} \\leq a_{j+1}(n, \\nu)\\Norm{\\vx^0 - \\vx^*}\n\\end{align*}\nholds for all $j = n+1, \\cdots, t-1$ such that $t \\geq n + 2$. Combining results of Part I and II, we have\n\n\\begin{align*}\n\\Norm{\\vx^j-\\vx^*}\\leq a_j(n, \\nu)\\Norm{\\vx^0-\\vx^*} \\qquad \\text{for all}~~ j=0,\\dots,t,\n\\end{align*}\nwhich implies\n\\begin{align*}\n    \\Norm{\\vx^j-\\vx^*}^{\\nu}\\leq (a_j(n, \\nu))^{\\nu}\\Norm{\\vx^0-\\vx^*}^{\\nu} \\leq \\Norm{\\vx^0-\\vx^*}^{\\nu} \\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n}, \\qquad\\text{for all}~~j=1,\\dots,t,\n\\end{align*}\nwhere the second inequality is based on Lemma \\ref{le:s-mono-000} and the last inequality is based on the condition condition.\n\n\nThe update~(\\ref{eq:update-1}) means the points $\\{\\vz_i^{t}\\}_{i=1}^n$ can be written as $\\{\\vx^{t+1-n}, \\cdots, \\vx^{t}\\}$, which implies\n\\begin{align*}\n    \\max\\{\\Norm{\\vz_1^{t}-\\vx^*}, \\cdots, \\Norm{\\vz_n^{t}-\\vx^*}\\}=\\max\\{\\Norm{\\vx^{t+1-n}-\\vx^*}, \\cdots, \\Norm{\\vx^{t}-\\vx^*}\\}.\n\\end{align*}\nTherefore, we have\n\\begin{align*}\n    \\Norm{\\vz_i^{n} - \\vx^*}^{\\nu} \\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n} \\qquad \\text{for all}~~i=1,\\dots,n.\n\\end{align*}\n\nCombing with Lemma \\ref{le:lb-Ht-new-1}, we have\n\\begin{align*}\n    \\sigma_{\\min}(\\mH^{t}) &\\geq  \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^{t} - \\vx^*}^\\nu\\\\\n    & \\geq \\mu^2 - 2L_f\\fH_\\nu n \\frac{\\mu^2}{4L_f\\fH_\\nu n}\\\\\n    & = \\mu^2 - \\frac{\\mu^2}{2} = \\frac{\\mu^2}{2}.\n\\end{align*}\nTherefore, we achieve\n\\begin{align*}\n    \\mH^{t}  \\succeq \\frac{\\mu^2}{2}\\mI \\qquad\\text{and}\\qquad\n    \\Norm{\\mG^{t}} & = \\Norm{(\\mH^{t})^{-1}} \\leq \\frac{2}{\\mu^2}.\n\\end{align*}\nAccording to Lemma \\ref{le:main-iter-1}, we have\n\\begin{align*}\n    \\Norm{\\vx^{t+1} - \\vx^*}\n    & \\leq \\frac{L_f\\fH_\\nu }{1+\\nu}\\Norm{\\mG^{t}}\\sum_{i=1}^n\\Norm{\\vz_i^{t}-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\sum_{i=1}^n\\Norm{\\vz_i^{t}-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\left(\\sum_{j=t-n+1}^{t} (a_j(n, \\nu))^{1+\\nu}\\Norm{\\vx^0 - \\vx^*}^{1+\\nu}\\right)\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\left(\\sum_{j=t-n+1}^{t} (a_j(n, \\nu))^{1+\\nu}\\right)\\Norm{\\vx^0 - \\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\left(\\sum_{j=t-n+1}^{t+1}a_j(n, \\nu)^{1+\\nu}\\right)\\Norm{\\vx^0 - \\vx^*}\\\\\n    & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=t-n+2}^{t+1}(a_j(n, \\nu))^{1+\\nu}\\right)\\Norm{\\vx^0 - \\vx^*}\\\\\n    & = a_{t+1}(n, \\nu)\\Norm{\\vx^0 - \\vx^*}.\n\\end{align*}\nHence, we finish the induction.\n\n\nCombining results of Part I, II and III completes the proof of (\\ref{eq:Hxtxstar0}).\n\nSince the non-singularity of $\\mH^t$ and $\\mH^{t+1}$ has been verified by result (\\ref{eq:Hxtxstar0}), we can apply Lemma \\ref{le:via-update} to achieve\n\\begin{align*}\n    \\sigma_{\\min}(\\mI + (\\mV^t)^\\top(\\mH^t)^{-1}\\mU^t) > 0.\n\\end{align*}\n\\end{proof}","sha256_utf8":"e95288a2352cd48a23d368c5000f7077d5acd3588dfd3d91095d8ca370cb8471"}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","title":"确定联合归纳目标","source_units":[{"start":73856,"end":74154,"start_line":1558,"end_line":1566,"raw_tex":"\\begin{proof}\n\nWe first show\n\\begin{align}\\label{eq:Hxtxstar0}\n\\mH^t \\succeq \\frac{\\mu^2}{2}\\mI\\qquad\n\\text{and}\\qquad\n\\Norm{\\vx^{t}-\\vx^*}\\leq a_{t+1}(n, \\nu)\\Norm{\\vx^0-\\vx^*}\n\\end{align}\nholds for all $t\\geq 0$. We split the proof of results (\\ref{eq:Hxtxstar0}) into the following three parts.\n","sha256_utf8":"082f9aff7d2588f15e9f7257430dc3b49a776b2c7ace4eaa980afadfa49d2f2b"}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","title":"验证初始误差恒等式","source_units":[{"start":74155,"end":74319,"start_line":1568,"end_line":1572,"raw_tex":"\\textbf{Part I:} For $t=0$, \nthe initialization and the fact $a_0=1$ leads to \n\\begin{align*}\n \\Norm{\\vx^0 - \\vx^*} = a_0(n, \\nu)\\Norm{\\vx^0 - \\vx^*}.\n\\end{align*}\n","sha256_utf8":"ce29f614e9b25c243d873a0faa2bd7a142010a255d5b5ef6d00184567bcb8f9d"}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","title":"首周期初始Gram正定","source_units":[{"start":74321,"end":74955,"start_line":1575,"end_line":1588,"raw_tex":"\\textbf{Part II:} For all $t=0,\\cdots, n-1$, we use induction to prove the results of \n\\begin{align}\\label{eq:Hxtxstar}\n\\mH^t \\succeq \\frac{\\mu^2}{2}\\mI\\qquad\n\\text{and}\\qquad\n\\Norm{\\vx^{t+1}-\\vx^*}\\leq a_{t+1}(n, \\nu)\\Norm{\\vx^0-\\vx^*}.\n\\end{align}\nFor the induction base, we can apply Lemma \\ref{le:lb-Ht-new-1} to verify\n\\begin{align*}\n    \\sigma_{\\min}(\\mH^0) &\\geq  \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^0 - \\vx^*}^\\nu\\\\\n    & = \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vx^0 - \\vx^*}^\\nu\\\\\n    & \\geq \\mu^2 - 2L_f\\fH_\\nu n \\frac{\\mu^2}{4L_f\\fH_\\nu n}\\\\\n    & = \\mu^2 - \\frac{\\mu^2}{2}\\\\\n    & = \\frac{\\mu^2}{2}.\n\\end{align*}\n","sha256_utf8":"cad87a236c79c489962966b7c0d8986ce1dfedb1c65659d0105785ed3839a6fb"}]},{"id":"S080","semantic_type":"proof_step","kind":"proof_step","title":"初始逆矩阵界与首步误差","source_units":[{"start":74957,"end":75866,"start_line":1591,"end_line":1607,"raw_tex":"This implies\n\\begin{align}\\label{eq:boundG0}\n\\mH^0 \\succeq \\frac{\\mu^2}{2} \\qquad \\text{and}\\qquad\n\\Norm{\\mG^0} = \\Norm{(\\mH^0)^{-1}} \\leq \\frac{2}{\\mu^2}.\n\\end{align}\nAccording to Lemma \\ref{le:main-iter-1}, we have\n\\begin{align*}\n    \\Norm{\\vx^1 - \\vx^*} \n    &\\leq \\frac{L_f\\fH_\\nu}{1+\\nu}\\Norm{\\mG^0}\\sum_{i=1}^n\\Norm{\\vz_i^0-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{L_f\\fH_\\nu}{1+\\nu}\\cdot\\frac{2}{\\mu^2}\\cdot\\sum_{i=1}^n\\Norm{\\vz_i^0-\\vx^*}^{1+\\nu}\\\\\n    & = \\frac{L_f\\fH_\\nu}{1+\\nu}\\cdot\\frac{2}{\\mu^2}\\cdot n\\Norm{\\vx^0-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{nL_f\\fH_\\nu}{1+\\nu}\\cdot\\frac{2}{\\mu^2}\\cdot\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\Norm{\\vx^0-\\vx^*}\\\\\n    & = \\frac{1}{2(1+\\nu)}\\Norm{\\vx^0-\\vx^*}\\\\\n    & = a_1(n, \\nu)\\Norm{\\vx^0 - \\vx^*},\n\\end{align*}\nwhere the first inequality is based on equation (\\ref{eq:boundG0}) and the second inequality is based on initial condition. \nTherefore, the induction base holds\n","sha256_utf8":"86216639689471aaf32ad2d5a153dc5873969efee20e4a54a2cf3ad9db146ae3"}]},{"id":"S081","semantic_type":"proof_step","kind":"proof_step","title":"首周期历史点保持邻域","source_units":[{"start":75867,"end":76813,"start_line":1609,"end_line":1629,"raw_tex":"For the induction step, we assume\n\\begin{align*}\n    \\mH^j \\succeq \\frac{\\mu^2}{2} \\mI \\qquad\\text{and}\\qquad\\Norm{\\vx^{j+1} - \\vx^*} \\leq a_{j+1}(n, \\nu)\\Norm{\\vx^0 - \\vx^*}\n\\end{align*}\nhold for all $j = 2, \\cdots, t-1$ such that $t \\leq n - 1$. Therefore, the update (\\ref{eq:update-1}) means\n\\begin{align}\\label{eq:vziii}\n    \\vz_i^{t}=\n    \\begin{cases}\n        \\vx^i,~~~~&1 \\leq i\\leq t,\\\\\n        \\vx^0,~~~~&t < i \\leq n. \n    \\end{cases}\n\\end{align}\nThe induction hypothesis leads to\n\\begin{align*}\n    \\Norm{\\vx^j-\\vx^*}^{\\nu}\\leq (a_j(n, \\nu))^{\\nu}\\Norm{\\vx^0-\\vx^*}^{\\nu} \\leq  \\Norm{\\vx^0-\\vx^*}^{\\nu}\\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n}, \n\\end{align*}\nfor $j = 1, \\cdots, t-1$, where the second is based on Lemma \\ref{le:s-mono-000} and the third comes from the initial condition. \nCombining with the result of (\\ref{eq:vziii}), we achive\n\\begin{align*}\n    \\Norm{\\vz_i^{t} - \\vx^*}^{\\nu} \\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n}.\n\\end{align*}\n","sha256_utf8":"7a7285676a5b2e4a94ee81165ab5043c05e79ff0b6c609186ae6a012fe160bca"}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","title":"首周期维持Gram正定","source_units":[{"start":76814,"end":77350,"start_line":1631,"end_line":1644,"raw_tex":"According to Lemma \\ref{le:lb-Ht-new-1}, we have\n\\begin{align*}\n    \\sigma_{\\min}(\\mH^{t}) &\\geq  \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^{t} - \\vx^*}^\\nu\\\\\n    & \\geq \\mu^2 - 2L_f\\fH_\\nu n \\frac{\\mu^2}{4L_f\\fH_\\nu n}\\\\\n    & = \\mu^2 - \\frac{\\mu^2}{2}\\\\\n    & = \\frac{\\mu^2}{2},\n\\end{align*}\n\nwhere the second inequality comes from the initial condition.\nTherefore, we have\n\\begin{align*}\n    \\mH^{t}  \\succeq \\frac{\\mu^2}{2}\\mI \\qquad \\text{and}\\qquad\n    \\Norm{\\mG^{t}}  = \\Norm{(\\mH^{t})^{-1}} \\leq \\frac{2}{\\mu^2}.\n\\end{align*}\n","sha256_utf8":"f2d6739839b5c9298080277c4828feee6dcea83d9936a34f9f1c0ea174f806f0"}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","title":"完成首周期误差归纳","source_units":[{"start":77350,"end":78459,"start_line":1645,"end_line":1658,"raw_tex":"According to Lemma \\ref{le:main-iter-1}, we have\n\\begin{align*}\n    \\Norm{\\vx^{t+1} - \\vx^*} \n    & \\leq \\frac{L_f\\fH_\\nu }{1+\\nu}\\Norm{\\mG^{t}}\\sum_{i=1}^n\\Norm{\\vz_i^{t}-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{L_f\\fH_\\nu }{1+\\nu}\\frac{2}{\\mu^2}\\sum_{i=1}^n\\Norm{\\vz_i^{t}-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu}{(1+\\nu)\\mu^2}\\left(\\sum_{j=1}^{t}\\Norm{\\vx^{j}-\\vx^*}^{1+\\nu} + (n - t)\\Norm{\\vx^0 - \\vx^*}^{1+\\nu}\\right)\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\left(\\sum_{j=1}^{t} (a_j(n, \\nu))^{1+\\nu}\\Norm{\\vx^0 - \\vx^*}^{1+\\nu} + (n - t) \\Norm{\\vx^0 - \\vx^*}^{1+\\nu}\\right)\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\left(\\sum_{j=1}^{t}(a_j(n, \\nu))^{1+\\nu} + n - t\\right)\\Norm{\\vx^0 - \\vx^*}\\\\\n    & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=1}^{t}(a_j(n, \\nu))^{1+\\nu} + n - t\\right)\\Norm{\\vx^0 - \\vx^*}\\\\\n    & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=0}^{t}(a_j(n, \\nu))^{1+\\nu} + n - t - 1\\right)\\Norm{\\vx^0 - \\vx^*}\\\\\n    & = a_{t+1}(n, \\nu)\\Norm{\\vx^0 - \\vx^*},\n\\end{align*}\nwhere the last equality comes from the fact $a_0(n, \\nu)=1$. \nTherefore, we finish the induction.\n","sha256_utf8":"5eadce5e33165c7e65a3cea5fd152304cef84b86a902b469e61b76fe6795746a"}]},{"id":"S084","semantic_type":"proof_step","kind":"proof_step","title":"后续周期归纳基的邻域界","source_units":[{"start":78461,"end":79321,"start_line":1661,"end_line":1677,"raw_tex":"\\textbf{Part III:} For all $t \\geq n$, we use induction to prove \n\\begin{align*}\n\\mH^t\\succeq(\\mu^2/2)\\mI\\qquad\\text{and}\\qquad\\Norm{\\vx^{t+1} - \\vx^*}\\leq a_{t+1}(n, \\nu)\\Norm{\\vx^0 - \\vx^*}.    \n\\end{align*}\nFor the induction base, we can verify that it holds (from the result of Part II) \n\\begin{align*}\n\\mH^j \\succeq \\frac{\\mu^2}{2}\\mI \\qquad \\text{for all~~} j=0,\\dots,n - 1,\n\\end{align*}\nand\n\\begin{align*}\n\\Norm{\\vx^j-\\vx^*}\\leq a_j(n, \\nu)\\Norm{\\vx^0-\\vx^*} \\qquad \\text{for all~~} j=1,\\dots,n.\n\\end{align*}\nThen we have\n\\begin{align*}\n    \\Norm{\\vx^j-\\vx^*}^{\\nu}\\leq (a_j(n, \\nu))^{\\nu}\\Norm{\\vx^0-\\vx^*}^{\\nu} \\leq \\Norm{\\vx^0-\\vx^*}^{\\nu} \\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n}, \\quad \\text{for all~~} j=1,\\dots,n,\n\\end{align*}\nwhere the second inequality is based on Lemma \\ref{le:s-mono-000} and the third inequality is based on the initial condition.\n","sha256_utf8":"ffb6eec1e18430332f030dc3c3d1ec63498cb6400b2653150eb266421c35704c"}]},{"id":"S085","semantic_type":"proof_step","kind":"proof_step","title":"完整首轮后的Gram正定","source_units":[{"start":79322,"end":80050,"start_line":1679,"end_line":1699,"raw_tex":"From Eq.~\\ref{eq:update-1}, we have\n\\begin{align*}\n    \\vz_i^{n}=\\vx^i \\qquad\\text{for all~~} i\\in[n].\n\\end{align*}\n\nTherefore, we have\n\\begin{align*}\n    \\Norm{\\vz_i^{n} - \\vx^*}^{\\nu} \\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n}, \\qquad\\text{for all~~} i\\in[n].\n\\end{align*}\nAccording to Lemma \\ref{le:lb-Ht-new-1}, we have\n\\begin{align*}\n    \\sigma_{\\min}(\\mH^{n}) &\\geq  \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^{n} - \\vx^*}^\\nu\\\\\n    & \\geq \\mu^2 - 2L_f\\fH_\\nu n \\frac{\\mu^2}{4L_f\\fH_\\nu n}\\\\\n    & \\geq \\mu^2 - \\frac{\\mu^2}{2} = \\frac{\\mu^2}{2},\n\\end{align*}\n\nwhich implies\n\\begin{align*}\n    \\mH^{n}  \\succeq \\frac{\\mu^2}{2}\\mI \\qquad\\text{and}\\qquad\n    \\Norm{\\mG^{n}}  = \\Norm{(\\mH^{n})^{-1}} \\leq \\frac{2}{\\mu^2}.\n\\end{align*}\n","sha256_utf8":"8423c3f82b267ec00d3345c1fc912877fd8c8e596f659e524652313fca9b54d9"}]},{"id":"S086","semantic_type":"proof_step","kind":"proof_step","title":"验证后续周期误差归纳基","source_units":[{"start":80050,"end":80783,"start_line":1700,"end_line":1710,"raw_tex":"According to Lemma \\ref{le:main-iter-1}, we have\n\\begin{align*}\n    \\Norm{\\vx^{n+1} - \\vx^*}\n    & \\leq \\frac{L_f\\fH_\\nu }{1+\\nu}\\Norm{\\mG^n}\\sum_{i=1}^n\\Norm{\\vz_i^n-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\sum_{i=1}^n\\Norm{\\vz_i^n-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\left(\\sum_{j=1}^n (a_j(n, \\nu))^{1+\\nu}\\Norm{\\vx^0 - \\vx^*}^{1+\\nu}\\right)\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\left(\\sum_{j=1}^{n}(a_j(n, \\nu))^{1+\\nu}\\right)\\Norm{\\vx^0-\\vx^*}\\\\\n    & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=1}^{n}(a_j(n, \\nu))^{1+\\nu}\\right)\\Norm{\\vx^0-\\vx^*}\\\\\n    & = a_{n+1}(n, \\nu)\\Norm{\\vx^0-\\vx^*}.\n\\end{align*}\nHence, we have shown the induction base holds. \n","sha256_utf8":"664440856b3cd7bcf3088ca5166ef68b01bb2f0cd38ba5daab8e5c2174804ba1"}]},{"id":"S087","semantic_type":"proof_step","kind":"proof_step","title":"历史迭代误差保持邻域","source_units":[{"start":80784,"end":81543,"start_line":1712,"end_line":1725,"raw_tex":"For the induction step, we assume\n\\begin{align*}\n    \\mH^j \\succeq \\frac{\\mu^2}{2} \\mI \\qquad\\text{and}\\qquad\\Norm{\\vx^{j+1} - \\vx^*} \\leq a_{j+1}(n, \\nu)\\Norm{\\vx^0 - \\vx^*}\n\\end{align*}\nholds for all $j = n+1, \\cdots, t-1$ such that $t \\geq n + 2$. Combining results of Part I and II, we have\n\n\\begin{align*}\n\\Norm{\\vx^j-\\vx^*}\\leq a_j(n, \\nu)\\Norm{\\vx^0-\\vx^*} \\qquad \\text{for all}~~ j=0,\\dots,t,\n\\end{align*}\nwhich implies\n\\begin{align*}\n    \\Norm{\\vx^j-\\vx^*}^{\\nu}\\leq (a_j(n, \\nu))^{\\nu}\\Norm{\\vx^0-\\vx^*}^{\\nu} \\leq \\Norm{\\vx^0-\\vx^*}^{\\nu} \\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n}, \\qquad\\text{for all}~~j=1,\\dots,t,\n\\end{align*}\nwhere the second inequality is based on Lemma \\ref{le:s-mono-000} and the last inequality is based on the condition condition.\n","sha256_utf8":"5e7ee45ddf5beed7106ed2d99fc046bc409b64daacd21c5fb33018d7154b7796"}]},{"id":"S088","semantic_type":"proof_step","kind":"proof_step","title":"历史点构成最近迭代窗口","source_units":[{"start":81545,"end":81995,"start_line":1728,"end_line":1735,"raw_tex":"The update~(\\ref{eq:update-1}) means the points $\\{\\vz_i^{t}\\}_{i=1}^n$ can be written as $\\{\\vx^{t+1-n}, \\cdots, \\vx^{t}\\}$, which implies\n\\begin{align*}\n    \\max\\{\\Norm{\\vz_1^{t}-\\vx^*}, \\cdots, \\Norm{\\vz_n^{t}-\\vx^*}\\}=\\max\\{\\Norm{\\vx^{t+1-n}-\\vx^*}, \\cdots, \\Norm{\\vx^{t}-\\vx^*}\\}.\n\\end{align*}\nTherefore, we have\n\\begin{align*}\n    \\Norm{\\vz_i^{n} - \\vx^*}^{\\nu} \\leq \\frac{\\mu^2}{4L_f\\fH_\\nu n} \\qquad \\text{for all}~~i=1,\\dots,n.\n\\end{align*}\n","sha256_utf8":"adf2e90495ce46ad132780a71f1988839f20327d4aa36609e1214f1cbe9eb4c9"}]},{"id":"S089","semantic_type":"proof_step","kind":"proof_step","title":"一般迭代Gram及逆的界","source_units":[{"start":81996,"end":82464,"start_line":1737,"end_line":1747,"raw_tex":"Combing with Lemma \\ref{le:lb-Ht-new-1}, we have\n\\begin{align*}\n    \\sigma_{\\min}(\\mH^{t}) &\\geq  \\mu^2 - 2L_f\\fH_\\nu\\sum_{i=1}^n \\Norm{\\vz_i^{t} - \\vx^*}^\\nu\\\\\n    & \\geq \\mu^2 - 2L_f\\fH_\\nu n \\frac{\\mu^2}{4L_f\\fH_\\nu n}\\\\\n    & = \\mu^2 - \\frac{\\mu^2}{2} = \\frac{\\mu^2}{2}.\n\\end{align*}\nTherefore, we achieve\n\\begin{align*}\n    \\mH^{t}  \\succeq \\frac{\\mu^2}{2}\\mI \\qquad\\text{and}\\qquad\n    \\Norm{\\mG^{t}} & = \\Norm{(\\mH^{t})^{-1}} \\leq \\frac{2}{\\mu^2}.\n\\end{align*}\n","sha256_utf8":"f1e70dc06933f3f082ac0126666446b43eb675acecc2cf4046f74c83e8bf625e"}]},{"id":"S090","semantic_type":"proof_step","kind":"proof_step","title":"完成滑动窗口误差归纳","source_units":[{"start":82464,"end":83337,"start_line":1748,"end_line":1759,"raw_tex":"According to Lemma \\ref{le:main-iter-1}, we have\n\\begin{align*}\n    \\Norm{\\vx^{t+1} - \\vx^*}\n    & \\leq \\frac{L_f\\fH_\\nu }{1+\\nu}\\Norm{\\mG^{t}}\\sum_{i=1}^n\\Norm{\\vz_i^{t}-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\sum_{i=1}^n\\Norm{\\vz_i^{t}-\\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\left(\\sum_{j=t-n+1}^{t} (a_j(n, \\nu))^{1+\\nu}\\Norm{\\vx^0 - \\vx^*}^{1+\\nu}\\right)\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\left(\\sum_{j=t-n+1}^{t} (a_j(n, \\nu))^{1+\\nu}\\right)\\Norm{\\vx^0 - \\vx^*}^{1+\\nu}\\\\\n    & \\leq \\frac{2L_f\\fH_\\nu }{(1+\\nu)\\mu^2}\\frac{\\mu^2}{4L_f\\fH_\\nu n}\\left(\\sum_{j=t-n+1}^{t+1}a_j(n, \\nu)^{1+\\nu}\\right)\\Norm{\\vx^0 - \\vx^*}\\\\\n    & = \\frac{1}{2(1+\\nu)n}\\left(\\sum_{j=t-n+2}^{t+1}(a_j(n, \\nu))^{1+\\nu}\\right)\\Norm{\\vx^0 - \\vx^*}\\\\\n    & = a_{t+1}(n, \\nu)\\Norm{\\vx^0 - \\vx^*}.\n\\end{align*}\nHence, we finish the induction.\n","sha256_utf8":"294824bdd17e04e8b68c592d64d445390fa7cc25469fe5b03b9af41429672eeb"}]},{"id":"S091","semantic_type":"proof_step","kind":"proof_step","title":"合并归纳并验证更新可逆","source_units":[{"start":83339,"end":83677,"start_line":1762,"end_line":1768,"raw_tex":"Combining results of Part I, II and III completes the proof of (\\ref{eq:Hxtxstar0}).\n\nSince the non-singularity of $\\mH^t$ and $\\mH^{t+1}$ has been verified by result (\\ref{eq:Hxtxstar0}), we can apply Lemma \\ref{le:via-update} to achieve\n\\begin{align*}\n    \\sigma_{\\min}(\\mI + (\\mV^t)^\\top(\\mH^t)^{-1}\\mU^t) > 0.\n\\end{align*}\n\\end{proof}","sha256_utf8":"6fb373b341c7679ea600cea57b9fe562de304fdd4ae8cf3e226286a008351695"}]},{"id":"S092","semantic_type":"theorem","kind":"theorem","title":"误差包络的超线性收缩","source_units":[{"start":83680,"end":84566,"start_line":1771,"end_line":1792,"raw_tex":"\\begin{thm}\n\\label{thm:holder-IGN-1-2}\nWe define the sequence $\\{r_t\\}_{t\\geq 0}$ such that\n\\begin{align*}\n    r_t\\triangleq\\begin{cases}\n    \\max\\{\\Norm{\\vx^0 - \\vx^*}, 1\\},~~~~&t=0,\\\\[0.2cm]\n    a_t(n, \\nu)r_0,~~~~& t\\geq 1,\\\\\n\\end{cases}\n\\end{align*}\nwhere the sequence $\\{a_t(n, \\nu)\\}_{t\\geq 0}$ is defined by equation (\\ref{eq:series}). \nUnder the Assumptions \\ref{asm:L-f}, \\ref{asm:holder-g} and \\ref{asm:b-J-sing}, running \\IGN~(Algorithm \\ref{alg:IGN-1}) with initial condition shown in Theorem \\ref{thm:holder-IGN-1-1}, we have\n    \\begin{align}\n    \\label{eq:r_t_recursion}\n    \\Norm{\\vx^t - \\vx^*} \\leq r_t\n    \\qquad\\text{and}\\qquad\n    r_{t+1}\\leq  c^{(1+\\nu)^{\\left(\\floor{\\frac{t}{n}}-1\\right)}}r_t  \n    \\end{align}\n    for all $t\\geq n$, where\n    \\begin{align*}\n    c = 1- \\frac{1}{n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right).\n    \\end{align*}\n\\end{thm}","sha256_utf8":"1e083e3d42a67f1dbebbce58ff1e110c5a0fe200d276ae8c2c2b8bc458adb4b7"}]},{"id":"S093","semantic_type":"proof","kind":"proof","title":"误差包络收缩证明","source_units":[{"start":84568,"end":85525,"start_line":1794,"end_line":1815,"raw_tex":"\\begin{proof}\n    The definition of $\\{r_t\\}_{t\\geq 0}$ leads to \n    \\begin{align*}\n        r_0 = \\max\\{\\Norm{\\vx^0 - \\vx^*}, 1\\} \\geq \\Norm{\\vx^0 - \\vx^*}.\n    \\end{align*}\n    According to Theorem \\ref{thm:holder-IGN-1-1}, we have \n    \\begin{align*}\n    \\Norm{\\vx^t - \\vx^*} \\leq a_{t}(n, \\nu)\\Norm{\\vx^0 - \\vx^*} \\leq a_t(n, \\nu)r_0 = r_t.\n    \\end{align*}\n    According to Lemma \\ref{le:super-con}, we have\n    \\begin{align*}\n         a_{t+1}(n, \\nu)\\leq  c^{(1+\\nu)^{\\left(\\floor{\\frac{t}{n}}-1\\right)}}a_t(n, \\nu) \\qquad \\text{for all}~~t \\geq n.\n    \\end{align*}\n    Thus, achieve\n    \\begin{align*}\n        r_{t+1} = a_{t+1}(n, \\nu)r_0\\leq  c^{(1+\\nu)^{\\left(\\floor{\\frac{t}{n}}-1\\right)}}a_t(n, \\nu)r_0 =c^{(1+\\nu)^{\\left(\\floor{\\frac{t}{n}}-1\\right)}}r_t  \\qquad \\text{for all}~~t \\geq n,\n    \\end{align*}\n    where\n    \\begin{align*}\n        c = 1- \\frac{1}{n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right).\n    \\end{align*}\n\\end{proof}","sha256_utf8":"20f875a8b291b6f948ddce9ab19c4adbb097419a792b6883b0fa7811aad686e5"}]},{"id":"S094","semantic_type":"proof_step","kind":"proof_step","title":"包络覆盖初始误差","source_units":[{"start":84568,"end":84743,"start_line":1794,"end_line":1798,"raw_tex":"\\begin{proof}\n    The definition of $\\{r_t\\}_{t\\geq 0}$ leads to \n    \\begin{align*}\n        r_0 = \\max\\{\\Norm{\\vx^0 - \\vx^*}, 1\\} \\geq \\Norm{\\vx^0 - \\vx^*}.\n    \\end{align*}\n","sha256_utf8":"6de030b79827f322557d096b79167ef8d0563afec27c3728c24101b4cf619df2"}]},{"id":"S095","semantic_type":"proof_step","kind":"proof_step","title":"包络覆盖所有迭代误差","source_units":[{"start":84743,"end":84930,"start_line":1799,"end_line":1802,"raw_tex":"    According to Theorem \\ref{thm:holder-IGN-1-1}, we have \n    \\begin{align*}\n    \\Norm{\\vx^t - \\vx^*} \\leq a_{t}(n, \\nu)\\Norm{\\vx^0 - \\vx^*} \\leq a_t(n, \\nu)r_0 = r_t.\n    \\end{align*}\n","sha256_utf8":"41fda5473c88f26c7d781a3e472fbfc3b0ed0c58bf46b54b4c93107990c8ba75"}]},{"id":"S096","semantic_type":"proof_step","kind":"proof_step","title":"调用辅助序列显式收缩","source_units":[{"start":84930,"end":85140,"start_line":1803,"end_line":1806,"raw_tex":"    According to Lemma \\ref{le:super-con}, we have\n    \\begin{align*}\n         a_{t+1}(n, \\nu)\\leq  c^{(1+\\nu)^{\\left(\\floor{\\frac{t}{n}}-1\\right)}}a_t(n, \\nu) \\qquad \\text{for all}~~t \\geq n.\n    \\end{align*}\n","sha256_utf8":"873ce642e5996e6a4fd895fdae67a57bd0352fc90a4f3a4d4b4e930bcf0dbfdc"}]},{"id":"S097","semantic_type":"proof_step","kind":"proof_step","title":"缩放得到包络收敛率","source_units":[{"start":85140,"end":85525,"start_line":1807,"end_line":1815,"raw_tex":"    Thus, achieve\n    \\begin{align*}\n        r_{t+1} = a_{t+1}(n, \\nu)r_0\\leq  c^{(1+\\nu)^{\\left(\\floor{\\frac{t}{n}}-1\\right)}}a_t(n, \\nu)r_0 =c^{(1+\\nu)^{\\left(\\floor{\\frac{t}{n}}-1\\right)}}r_t  \\qquad \\text{for all}~~t \\geq n,\n    \\end{align*}\n    where\n    \\begin{align*}\n        c = 1- \\frac{1}{n}\\left(1-\\left(\\frac{1}{2(1+\\nu)}\\right)^{1+\\nu}\\right).\n    \\end{align*}\n\\end{proof}","sha256_utf8":"a3134b067eb78e5a8cd271fee028b570e87dfa55279c60ac41bca063b0f1a4de"}]},{"id":"S098","semantic_type":"external_fact","kind":"external_fact","title":"范数与谱的标准性质","source_units":[],"declared_external":true,"declaration_text":["欧氏范数和谱范数满足三角不等式、次乘性及转置不变性；矩阵行向量范数不超过谱范数。奇异值分解给出 J^T J 的特征值为 J 的奇异值平方。对称矩阵扰动满足 -||E||I ≼ E ≼ ||E||I；正定矩阵 H ≽ αI 蕴含 ||H^{-1}|| ≤ 1/α。"]},{"id":"S099","semantic_type":"external_fact","kind":"external_fact","title":"标准微积分与内积不等式","source_units":[],"declared_external":true,"declaration_text":["连续可微函数沿线段的函数差等于梯度与线段方向内积的积分；方向导数由差商极限给出；Cauchy–Schwarz 不等式控制内积。所选 Hölder 余项证明中的引用仅作为这一积分估计方法的标准出处，作者的完整推导已包含。"]},{"id":"S100","semantic_type":"external_fact","kind":"external_fact","title":"低秩行列式与求逆公式","source_units":[],"declared_external":true,"declaration_text":["若 A 可逆，则 det(A+UV^T)=det(A)det(I+V^T A^{-1}U)；当内部矩阵亦可逆时，(A+UV^T)^{-1}=A^{-1}-A^{-1}U(I+V^T A^{-1}U)^{-1}V^T A^{-1}。"]},{"id":"S101","semantic_type":"proof_step","kind":"proof_step","title":"提出低秩更新适定性任务","source_units":[{"start":69369,"end":69661,"start_line":1470,"end_line":1474,"raw_tex":"We first show the update\n\\begin{align*}\n\\mG^{t+1} =\\mG^{t} - \\mG^{t}\\mU^t(\\mI + (\\mV^t)^\\top\\mG^{t}\\mU^t)^{-1}(\\mV^t)^\\top\\mG^{t}    \n\\end{align*}\nin \\IGN~method (Line \\ref{line:update-G} of Algorithm \\ref{alg:IGN-1}) is well-defined if the matrices $\\mH^t$ and $\\mH^{t+1}$ are non-singular.\n","sha256_utf8":"db0d7929d42f55269076b6517ac47900fcfbf2f347716c6005f88eb31b9574fa"}]},{"id":"S102","semantic_type":"proof_step","kind":"proof_step","title":"提出单步误差估计任务","source_units":[{"start":71063,"end":71184,"start_line":1506,"end_line":1506,"raw_tex":"Then we show the non-singular assumption on $\\{\\mH^j\\}_{j=0}^t$ can upper bound the distance $\\Norm{\\vx^{t+1} - \\vx^*}$.\n","sha256_utf8":"ba4dce1de9ae0a0680e6706f7361d04a1e45aa079bd366302cf5692cda23aca5"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S101","target_id":"S002","kind":"supports"},{"source_id":"S102","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S075","target_id":"S004","kind":"supports"},{"source_id":"S092","target_id":"S004","kind":"supports"},{"source_id":"S015","target_id":"S003","kind":"supports"},{"source_id":"S066","target_id":"S101","kind":"supports"},{"source_id":"S071","target_id":"S102","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S006","target_id":"S018","kind":"supports"},{"source_id":"S008","target_id":"S019","kind":"supports"},{"source_id":"S098","target_id":"S019","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S016","target_id":"S022","kind":"supports"},{"source_id":"S006","target_id":"S022","kind":"supports"},{"source_id":"S099","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S007","target_id":"S025","kind":"supports"},{"source_id":"S006","target_id":"S025","kind":"supports"},{"source_id":"S099","target_id":"S025","kind":"supports"},{"source_id":"S098","target_id":"S026","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S009","target_id":"S029","kind":"supports"},{"source_id":"S098","target_id":"S029","kind":"supports"},{"source_id":"S098","target_id":"S030","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S023","target_id":"S033","kind":"supports"},{"source_id":"S008","target_id":"S033","kind":"supports"},{"source_id":"S098","target_id":"S033","kind":"supports"},{"source_id":"S016","target_id":"S034","kind":"supports"},{"source_id":"S023","target_id":"S034","kind":"supports"},{"source_id":"S098","target_id":"S034","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S013","target_id":"S037","kind":"supports"},{"source_id":"S031","target_id":"S037","kind":"supports"},{"source_id":"S098","target_id":"S037","kind":"supports"},{"source_id":"S027","target_id":"S038","kind":"supports"},{"source_id":"S098","target_id":"S038","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S042","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S015","target_id":"S041","kind":"supports"},{"source_id":"S015","target_id":"S042","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S048","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S015","target_id":"S045","kind":"supports"},{"source_id":"S015","target_id":"S046","kind":"supports"},{"source_id":"S015","target_id":"S047","kind":"supports"},{"source_id":"S015","target_id":"S048","kind":"supports"},{"source_id":"S039","target_id":"S046","kind":"supports"},{"source_id":"S039","target_id":"S047","kind":"supports"},{"source_id":"S045","target_id":"S048","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S015","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S043","target_id":"S052","kind":"supports"},{"source_id":"S055","target_id":"S054","kind":"supports"},{"source_id":"S056","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S049","target_id":"S055","kind":"supports"},{"source_id":"S039","target_id":"S055","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S015","target_id":"S056","kind":"supports"},{"source_id":"S049","target_id":"S056","kind":"supports"},{"source_id":"S059","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S060","target_id":"S059","kind":"supports"},{"source_id":"S015","target_id":"S059","kind":"supports"},{"source_id":"S063","target_id":"S062","kind":"supports"},{"source_id":"S064","target_id":"S062","kind":"supports"},{"source_id":"S065","target_id":"S062","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S053","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S057","target_id":"S064","kind":"supports"},{"source_id":"S064","target_id":"S065","kind":"supports"},{"source_id":"S068","target_id":"S067","kind":"supports"},{"source_id":"S069","target_id":"S067","kind":"supports"},{"source_id":"S070","target_id":"S067","kind":"supports"},{"source_id":"S067","target_id":"S066","kind":"supports"},{"source_id":"S010","target_id":"S068","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S100","target_id":"S069","kind":"supports"},{"source_id":"S069","target_id":"S070","kind":"supports"},{"source_id":"S013","target_id":"S070","kind":"supports"},{"source_id":"S073","target_id":"S072","kind":"supports"},{"source_id":"S074","target_id":"S072","kind":"supports"},{"source_id":"S072","target_id":"S071","kind":"supports"},{"source_id":"S011","target_id":"S073","kind":"supports"},{"source_id":"S013","target_id":"S073","kind":"supports"},{"source_id":"S005","target_id":"S073","kind":"supports"},{"source_id":"S009","target_id":"S073","kind":"supports"},{"source_id":"S073","target_id":"S074","kind":"supports"},{"source_id":"S020","target_id":"S074","kind":"supports"},{"source_id":"S023","target_id":"S074","kind":"supports"},{"source_id":"S098","target_id":"S074","kind":"supports"},{"source_id":"S077","target_id":"S076","kind":"supports"},{"source_id":"S078","target_id":"S076","kind":"supports"},{"source_id":"S079","target_id":"S076","kind":"supports"},{"source_id":"S080","target_id":"S076","kind":"supports"},{"source_id":"S081","target_id":"S076","kind":"supports"},{"source_id":"S082","target_id":"S076","kind":"supports"},{"source_id":"S083","target_id":"S076","kind":"supports"},{"source_id":"S084","target_id":"S076","kind":"supports"},{"source_id":"S085","target_id":"S076","kind":"supports"},{"source_id":"S086","target_id":"S076","kind":"supports"},{"source_id":"S087","target_id":"S076","kind":"supports"},{"source_id":"S088","target_id":"S076","kind":"supports"},{"source_id":"S089","target_id":"S076","kind":"supports"},{"source_id":"S090","target_id":"S076","kind":"supports"},{"source_id":"S091","target_id":"S076","kind":"supports"},{"source_id":"S076","target_id":"S075","kind":"supports"},{"source_id":"S015","target_id":"S078","kind":"supports"},{"source_id":"S035","target_id":"S079","kind":"supports"},{"source_id":"S009","target_id":"S079","kind":"supports"},{"source_id":"S078","target_id":"S080","kind":"supports"},{"source_id":"S079","target_id":"S080","kind":"supports"},{"source_id":"S071","target_id":"S080","kind":"supports"},{"source_id":"S015","target_id":"S080","kind":"supports"},{"source_id":"S098","target_id":"S080","kind":"supports"},{"source_id":"S100","target_id":"S080","kind":"supports"},{"source_id":"S080","target_id":"S081","kind":"supports"},{"source_id":"S012","target_id":"S081","kind":"supports"},{"source_id":"S014","target_id":"S081","kind":"supports"},{"source_id":"S039","target_id":"S081","kind":"supports"},{"source_id":"S081","target_id":"S082","kind":"supports"},{"source_id":"S035","target_id":"S082","kind":"supports"},{"source_id":"S098","target_id":"S082","kind":"supports"},{"source_id":"S082","target_id":"S083","kind":"supports"},{"source_id":"S081","target_id":"S083","kind":"supports"},{"source_id":"S071","target_id":"S083","kind":"supports"},{"source_id":"S015","target_id":"S083","kind":"supports"},{"source_id":"S083","target_id":"S084","kind":"supports"},{"source_id":"S039","target_id":"S084","kind":"supports"},{"source_id":"S084","target_id":"S085","kind":"supports"},{"source_id":"S012","target_id":"S085","kind":"supports"},{"source_id":"S035","target_id":"S085","kind":"supports"},{"source_id":"S098","target_id":"S085","kind":"supports"},{"source_id":"S085","target_id":"S086","kind":"supports"},{"source_id":"S084","target_id":"S086","kind":"supports"},{"source_id":"S071","target_id":"S086","kind":"supports"},{"source_id":"S015","target_id":"S086","kind":"supports"},{"source_id":"S086","target_id":"S087","kind":"supports"},{"source_id":"S083","target_id":"S087","kind":"supports"},{"source_id":"S039","target_id":"S087","kind":"supports"},{"source_id":"S087","target_id":"S088","kind":"supports"},{"source_id":"S012","target_id":"S088","kind":"supports"},{"source_id":"S088","target_id":"S089","kind":"supports"},{"source_id":"S035","target_id":"S089","kind":"supports"},{"source_id":"S098","target_id":"S089","kind":"supports"},{"source_id":"S089","target_id":"S090","kind":"supports"},{"source_id":"S087","target_id":"S090","kind":"supports"},{"source_id":"S088","target_id":"S090","kind":"supports"},{"source_id":"S071","target_id":"S090","kind":"supports"},{"source_id":"S015","target_id":"S090","kind":"supports"},{"source_id":"S079","target_id":"S091","kind":"supports"},{"source_id":"S082","target_id":"S091","kind":"supports"},{"source_id":"S085","target_id":"S091","kind":"supports"},{"source_id":"S089","target_id":"S091","kind":"supports"},{"source_id":"S090","target_id":"S091","kind":"supports"},{"source_id":"S066","target_id":"S091","kind":"supports"},{"source_id":"S094","target_id":"S093","kind":"supports"},{"source_id":"S095","target_id":"S093","kind":"supports"},{"source_id":"S096","target_id":"S093","kind":"supports"},{"source_id":"S097","target_id":"S093","kind":"supports"},{"source_id":"S093","target_id":"S092","kind":"supports"},{"source_id":"S094","target_id":"S095","kind":"supports"},{"source_id":"S075","target_id":"S095","kind":"supports"},{"source_id":"S015","target_id":"S095","kind":"supports"},{"source_id":"S061","target_id":"S096","kind":"supports"},{"source_id":"S096","target_id":"S097","kind":"supports"},{"source_id":"S094","target_id":"S097","kind":"supports"},{"source_id":"S015","target_id":"S097","kind":"supports"}]}}
{"record_id":"2407.10073v4","paper":{"title":"Universal subgradient and proximal bundle methods for convex and strongly convex hybrid composite optimization","authors":["Vincent Guigues","Jiaming Liang","Renato D. C. Monteiro"],"arxiv_id":"2407.10073","version":"v4","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-025-02927-7","abstract_url":"https://arxiv.org/abs/2407.10073v4","pdf_url":"https://arxiv.org/pdf/2407.10073v4"},"context":{"raw_tex":"\\begin{equation}\\label{eq:ProbIntro2}\n\t\\phi_{*}:=\\min \\left\\{\\phi(x):=f(x)+h(x): x \\in \\R^n\\right\\},\n\t\\end{equation}\n\nThe intrinsic convexity parameter\n$\\mu_\\psi$\nof a convex function $\\psi$ is defined as the largest scalar $\\mu$ such that $\\psi(\\cdot)-\\mu\\|\\cdot\\|^2/2$ is convex.\n\n\n    Let $\\R$ denote the set of real numbers.\n    Let $ \\R_+ $ (resp., $ \\R_{++} $) denote the set of non-negative real numbers (resp., the set of positive real numbers).\n\tLet $\\R^n$ denote the standard $n$-dimensional Euclidean space equipped with  inner product and norm denoted by $\\left\\langle \\cdot,\\cdot\\right\\rangle $\n\tand $\\|\\cdot\\|$, respectively. \n\n\n\tLet $\\log(\\cdot)$ denote the natural logarithm.\n\t\n\tFor given $\\Phi: \\R^n\\rightarrow (-\\infty,+\\infty]$, let $\\dom \\Phi:=\\{x \\in \\R^n: \\Phi (x) <\\infty\\}$ denote the effective domain of $\\Phi$ \n\tand $\\Phi$ is proper if $\\dom \\Phi \\ne \\emptyset$.\n\tA proper function $\\Phi: \\R^n\\rightarrow (-\\infty,+\\infty]$ is $\\mu$-convex for some $\\mu \\ge 0$ if\n\t\\[\n\t\\Phi(\\alpha x+(1-\\alpha) y)\\leq \\alpha \\Phi(x)+(1-\\alpha)\\Phi(y) - \\frac{\\alpha(1-\\alpha) \\mu}{2}\\|x-y\\|^2\n\t\\]\n\tfor every $x, y \\in \\dom \\Phi$ and $\\alpha \\in [0,1]$.\nLet $\\mConv{n}$ denote the set of all proper lower semicontinuous $\\mu$-convex functions.\n We simply denote $\\mConv{n}$ by $\\bConv{n}$ when $\\mu=0$.\n\tFor $\\varepsilon \\ge 0$, the \\emph{$\\varepsilon$-subdifferential} of $ \\Phi $ at $x \\in \\dom \\Phi$ is denoted by\n\t\\[\n        \\partial_\\varepsilon \\Phi (x):=\\left\\{ s \\in\\R^n: \\Phi(y)\\geq \\Phi(x)+\\left\\langle s,y-x\\right\\rangle -\\varepsilon, \\forall y\\in\\R^n\\right\\}.\n        \\]\n\tWe denote the subdifferential of $\\Phi$ at $x \\in \\dom \\Phi$ by $\\partial \\Phi (x)$, which is the set  $\\partial_0 \\Phi(x)$ by definition.\n    For a given subgradient\n$\\Phi'(x) \\in \\partial \\Phi(x)$, we denote the linearization of convex function $\\Phi$ at $x$ by $\\ell_\\Phi(\\cdot,x)$, which is defined as\n\\begin{equation}\\label{linfdef}\n\\ell_\\Phi(\\cdot,x)=\\Phi(x)+\\langle \\Phi'(x),\\cdot-x\\rangle.\n\\end{equation}\n\nWe assume that \n\\begin{itemize}\n\n\n\\item[(A1)] $f, h \\in \\bConv{n}$ are such that\n\t\t$\\dom h \\subset \\dom f$, and a subgradient oracle,\n\t\t i.e.,\n\t\ta function $f':\\dom h \\to \\R^n$\n\t\tsatisfying $f'(x) \\in \\partial f(x)$ for every $x \\in \\dom h$, is available;\n\t\t\\item[(A2)]\n\t\tthere exists $(M_f,L_f)\\in \\R^2_+$ such that for every $x,y \\in \\dom h$,\n\t\t\\[\n\t\t\\|f'(x)-f'(y)\\| \\le 2M_f + L_f \\|x-y\\|;\n\t\t\\]\n\\item[(A3)]\n\t\tthe set of optimal solutions $X_*$ of\n\t\tproblem \\eqref{eq:ProbIntro2} is nonempty.\n\\end{itemize}\n\n\nIt is known that that (A2) implies that for every $x,y \\in \\dom h$, \n\\begin{equation}\\label{ineq:est} \t\nf(x)-\\ell_f(x;y) \\le 2M_f \\|x-y\\| + \\frac{L_f}{2} \\| x-y \\|^2.\n\t\\end{equation}\n\nOur goal in this section is to describe two completely universal methods for solving \\eqref{eq:ProbIntro2}  and describe  their iteration-complexity guarantees for finding \na $\\bar \\varepsilon$-solution $\\bar x$ of \\eqref{eq:ProbIntro2}, i.e., \na point $\\bar  x$ such that $\\phi(\\bar x) - \\phi_* \\le \\bar \\varepsilon$, in terms of\n$(M_f,L_f,\\mu_\\phi )$, $\\bar \\varepsilon$, and $d_0$, where $d_0$\nis the distance of the initial iterate $\\hat x_0$ with respect to $X_*$, i.e.,\n\\begin{equation}\\label{defd0}\nd_0 := \\min \\{\\|x-\\hat x_0\\| : x \\in X_*\\}.\n\\end{equation}\n\n\tU-CS($\\hat x_0,\\chi,\\lam_0,\\bar \\varepsilon$)\n\t\n\t\\noindent\\rule[0.5ex]{1\\columnwidth}{1pt}\n\t\n    {\\bf Input:} $(\\hat x_0,\\chi,\\lam_0,\\bar \\varepsilon)\\in \\dom h \\times [0,1) \\times \\R_{++} \\times \\R_{++}$.\n        \\begin{itemize}\n\t\t\\item [0.]  Set $\\lam=\\lam_0$ and $k=1$;\n\t    \\item[1.] Compute \n\t    \\begin{equation}\\label{def:x}\n\t        x =\\underset{u\\in  \\R^n}\\argmin\n\t\t\\left\\lbrace  \\ell_f(u;\\hat x_{k-1})+h(u) +\\frac{1}{2\\lam} \\|u- \\hat x_{k-1} \\|^2 \\right\\rbrace;\n\t    \\end{equation}\n        if $\\phi(x) - \\phi_* \\le \\bar \\varepsilon$, then {\\bf stop};\n\t\t\\item[2.]  {\\bf If} the inequality $$f(x)-\\ell_f(x;\\hat x_{k-1}) - \\frac{1-\\chi}{2\\lam} \\|x-\\hat x_{k-1}\\|^2 \\le \\frac{(1-\\chi) \\bar \\varepsilon}2$$ does not hold, {\\bf then} set $\\lam = \\lam/2$ and go to step~1; {\\bf else}, \n        set $\\lam_k=\\lam$, $\\hat x_k=x$,  $k\\leftarrow k+1$, and go to step 1.\n\t\\end{itemize}\n\t\\rule[0.5ex]{1\\columnwidth}{1pt}\n\n\n\\begin{equation}\\label{eq:ProbIntro}\n\t\\phi_{*}:=\\min \\left\\{\\phi(x): x \\in \\R^n\\right\\}\n\t\\end{equation}\n\nwhere $\\phi \\in \\bConv{n}$.\n\n\n\t Black-box (BB)\n     $(x^-,\\chi,\\varepsilon)$\n\t\n\t\\noindent\\rule[0.5ex]{1\\columnwidth}{1pt}\n    {\\bf Input:}  $(x^-,\\chi,\\varepsilon) \\in \\dom \\phi \\times [0,1) \\times \\R_{++}$.\n    \n    \\vgap\n    \n    \\noindent\n    {\\bf Output:}\n    $(x,y,\\Gamma,\\lam) \\in \\dom \\phi \\times \\dom \\phi \\times \\bConv{n} \\times \\R_{++}$ satisfying $\\Gamma \\le \\phi$,\n\\begin{equation}\\label{propyb}\n    \\phi(y)+\\frac{\\chi}{2 \\lambda}\\|y-x^-\\|^2 - \\underset{u\\in\\R^n}\\min\\left\\{\\Gamma(u) + \\frac{1}{2\\lam}\\|u- x^- \\|^2\\right\\} \\le \\varepsilon,\n\\end{equation}\nand\n\\beq \n    x = \\underset{u\\in\\R^n} \\argmin \\left\\lbrace \\Gamma(u) + \\frac{1}{2\\lam}\\|u- x^- \\|^2 \\right\\rbrace. \\label{defxhatk}\n     \\eeq\n\n\nwhere \n\\begin{equation}\\label{eq:sigma}\n    \\sigma = \\sigma(\\mu,\\nu) := \\frac{\\lam[\\nu(1+\\mu \\lam)+\\chi(\\mu-\\nu)]}{1+\\mu  \\lambda+\\chi  \\lambda(\\nu-\\mu)}.\n\\end{equation} \n\n\n\t FSCO$(\\hat x_0, \\chi, \\bar \\varepsilon)$\n\t\n\t\\noindent\\rule[0.5ex]{1\\columnwidth}{1pt}\n    % {\\bf Input:} $(\\hat x_0, \\chi, \\bar \\varepsilon) \\in \\dom \\phi \\times [0,1) \\times \\R_+ $\n\t\\begin{itemize}\n\t\t\\item [0.] Let $(\\hat x_0, \\chi, \\bar \\varepsilon) \\in \\dom \\phi \\times [0,1) \\times \\R_+ $ be given and set $k=1$;\n\n\n    \\item[1.] Compute \n$(\\hat x_k, \\hat y_k, \\hat \\Gamma_k, \\lambda_k) = \\BB(\\hat x_{k-1},\\chi, (1-\\chi) \\bar \\varepsilon/2)$;\n    \n\t\t\\item[2.]\n  Check whether $\\phi(\\hat y_k) - \\phi_* \\le \\bar \\varepsilon$ and if so {\\bf stop}; else set $k\\leftarrow k+1$ and go to step 1.\n  % \\item[3.]\n\t\t% Set $k\\leftarrow k+1$ and go to step 1.\n\t\\end{itemize}\n\t\\rule[0.5ex]{1\\columnwidth}{1pt}\n\n\n The main complexity result for FSCO is relative to the  instances that satisfy the following condition:\n % In addition to $\\phi \\in \\bConv{n}$, this result requires the following mild assumption: \n\\begin{itemize}\n% \\item[(F1)] there exists $\\nu \\in [0,\\mu(\\phi)]$ such that\n% $\\hat \\Gamma_k \\in \\nConv{n}$;\n\\item[(F1)] there exists $\\underline \\lambda>0$ such that\n$\\lambda_k \\geq {\\underline \\lambda}$ for every $k \\ge 0$.\n\\end{itemize}\n\n\nDefine \n \\begin{align}\n\t s &:= \\frac{x^- - x}{\\lam}, \\label{defsb}\\\\\n\t \\eta &:= \\phi(y) - \\Gamma(x) - \\inner{s}{y- x}. \\label{defetab}\n\t\\end{align}\n","units":[{"start":11637,"end":11753,"start_line":301,"end_line":303,"raw_tex":"\\begin{equation}\\label{eq:ProbIntro2}\n\t\\phi_{*}:=\\min \\left\\{\\phi(x):=f(x)+h(x): x \\in \\R^n\\right\\},\n\t\\end{equation}","sha256_utf8":"df85d33c54c601ab5543164609373b6594e620d4d6909c4bdbf8f534ae904e90"},{"start":12206,"end":12370,"start_line":310,"end_line":312,"raw_tex":"The intrinsic convexity parameter\n$\\mu_\\psi$\nof a convex function $\\psi$ is defined as the largest scalar $\\mu$ such that $\\psi(\\cdot)-\\mu\\|\\cdot\\|^2/2$ is convex.\n","sha256_utf8":"b80e409c12c20d31edbaef5025c7957ea356466008484734836e54206ddc98a4"},{"start":33456,"end":33813,"start_line":692,"end_line":695,"raw_tex":"    Let $\\R$ denote the set of real numbers.\n    Let $ \\R_+ $ (resp., $ \\R_{++} $) denote the set of non-negative real numbers (resp., the set of positive real numbers).\n\tLet $\\R^n$ denote the standard $n$-dimensional Euclidean space equipped with  inner product and norm denoted by $\\left\\langle \\cdot,\\cdot\\right\\rangle $\n\tand $\\|\\cdot\\|$, respectively. \n","sha256_utf8":"9895973caf4aa6364c0f39ebe722efa3ade91534302ac28d92d2203edcf9cf21"},{"start":33943,"end":35303,"start_line":698,"end_line":718,"raw_tex":"\tLet $\\log(\\cdot)$ denote the natural logarithm.\n\t\n\tFor given $\\Phi: \\R^n\\rightarrow (-\\infty,+\\infty]$, let $\\dom \\Phi:=\\{x \\in \\R^n: \\Phi (x) <\\infty\\}$ denote the effective domain of $\\Phi$ \n\tand $\\Phi$ is proper if $\\dom \\Phi \\ne \\emptyset$.\n\tA proper function $\\Phi: \\R^n\\rightarrow (-\\infty,+\\infty]$ is $\\mu$-convex for some $\\mu \\ge 0$ if\n\t\\[\n\t\\Phi(\\alpha x+(1-\\alpha) y)\\leq \\alpha \\Phi(x)+(1-\\alpha)\\Phi(y) - \\frac{\\alpha(1-\\alpha) \\mu}{2}\\|x-y\\|^2\n\t\\]\n\tfor every $x, y \\in \\dom \\Phi$ and $\\alpha \\in [0,1]$.\nLet $\\mConv{n}$ denote the set of all proper lower semicontinuous $\\mu$-convex functions.\n We simply denote $\\mConv{n}$ by $\\bConv{n}$ when $\\mu=0$.\n\tFor $\\varepsilon \\ge 0$, the \\emph{$\\varepsilon$-subdifferential} of $ \\Phi $ at $x \\in \\dom \\Phi$ is denoted by\n\t\\[\n        \\partial_\\varepsilon \\Phi (x):=\\left\\{ s \\in\\R^n: \\Phi(y)\\geq \\Phi(x)+\\left\\langle s,y-x\\right\\rangle -\\varepsilon, \\forall y\\in\\R^n\\right\\}.\n        \\]\n\tWe denote the subdifferential of $\\Phi$ at $x \\in \\dom \\Phi$ by $\\partial \\Phi (x)$, which is the set  $\\partial_0 \\Phi(x)$ by definition.\n    For a given subgradient\n$\\Phi'(x) \\in \\partial \\Phi(x)$, we denote the linearization of convex function $\\Phi$ at $x$ by $\\ell_\\Phi(\\cdot,x)$, which is defined as\n\\begin{equation}\\label{linfdef}\n\\ell_\\Phi(\\cdot,x)=\\Phi(x)+\\langle \\Phi'(x),\\cdot-x\\rangle.\n\\end{equation}","sha256_utf8":"a30836d6c7452e589ee052ec0b3ee986d065177779369c2d1ca77dcb60e24e43"},{"start":37370,"end":37402,"start_line":755,"end_line":756,"raw_tex":"We assume that \n\\begin{itemize}\n","sha256_utf8":"fd92f1913ca2bbafe6cdd90ce63d9ff4af43897165b0809e7ee996f249cdd0a0"},{"start":37723,"end":38197,"start_line":761,"end_line":774,"raw_tex":"\\item[(A1)] $f, h \\in \\bConv{n}$ are such that\n\t\t$\\dom h \\subset \\dom f$, and a subgradient oracle,\n\t\t i.e.,\n\t\ta function $f':\\dom h \\to \\R^n$\n\t\tsatisfying $f'(x) \\in \\partial f(x)$ for every $x \\in \\dom h$, is available;\n\t\t\\item[(A2)]\n\t\tthere exists $(M_f,L_f)\\in \\R^2_+$ such that for every $x,y \\in \\dom h$,\n\t\t\\[\n\t\t\\|f'(x)-f'(y)\\| \\le 2M_f + L_f \\|x-y\\|;\n\t\t\\]\n\\item[(A3)]\n\t\tthe set of optimal solutions $X_*$ of\n\t\tproblem \\eqref{eq:ProbIntro2} is nonempty.\n\\end{itemize}\n","sha256_utf8":"d42001ebc0d84d34c51002916be3387cbe2a2474990bd73043683dc46eb3bed0"},{"start":38198,"end":38380,"start_line":776,"end_line":779,"raw_tex":"It is known that that (A2) implies that for every $x,y \\in \\dom h$, \n\\begin{equation}\\label{ineq:est} \t\nf(x)-\\ell_f(x;y) \\le 2M_f \\|x-y\\| + \\frac{L_f}{2} \\| x-y \\|^2.\n\t\\end{equation}","sha256_utf8":"1e059872c59c0c1968932dbb8a3880c06c51314eb28661536738eddb7f70fe69"},{"start":39239,"end":39803,"start_line":794,"end_line":801,"raw_tex":"Our goal in this section is to describe two completely universal methods for solving \\eqref{eq:ProbIntro2}  and describe  their iteration-complexity guarantees for finding \na $\\bar \\varepsilon$-solution $\\bar x$ of \\eqref{eq:ProbIntro2}, i.e., \na point $\\bar  x$ such that $\\phi(\\bar x) - \\phi_* \\le \\bar \\varepsilon$, in terms of\n$(M_f,L_f,\\mu_\\phi )$, $\\bar \\varepsilon$, and $d_0$, where $d_0$\nis the distance of the initial iterate $\\hat x_0$ with respect to $X_*$, i.e.,\n\\begin{equation}\\label{defd0}\nd_0 := \\min \\{\\|x-\\hat x_0\\| : x \\in X_*\\}.\n\\end{equation}","sha256_utf8":"97b83a2ec4596f333bf3aa343edff7c685fb9e0c84bbd5b885d39170c1315464"},{"start":45621,"end":46539,"start_line":915,"end_line":931,"raw_tex":"\tU-CS($\\hat x_0,\\chi,\\lam_0,\\bar \\varepsilon$)\n\t\n\t\\noindent\\rule[0.5ex]{1\\columnwidth}{1pt}\n\t\n    {\\bf Input:} $(\\hat x_0,\\chi,\\lam_0,\\bar \\varepsilon)\\in \\dom h \\times [0,1) \\times \\R_{++} \\times \\R_{++}$.\n        \\begin{itemize}\n\t\t\\item [0.]  Set $\\lam=\\lam_0$ and $k=1$;\n\t    \\item[1.] Compute \n\t    \\begin{equation}\\label{def:x}\n\t        x =\\underset{u\\in  \\R^n}\\argmin\n\t\t\\left\\lbrace  \\ell_f(u;\\hat x_{k-1})+h(u) +\\frac{1}{2\\lam} \\|u- \\hat x_{k-1} \\|^2 \\right\\rbrace;\n\t    \\end{equation}\n        if $\\phi(x) - \\phi_* \\le \\bar \\varepsilon$, then {\\bf stop};\n\t\t\\item[2.]  {\\bf If} the inequality $$f(x)-\\ell_f(x;\\hat x_{k-1}) - \\frac{1-\\chi}{2\\lam} \\|x-\\hat x_{k-1}\\|^2 \\le \\frac{(1-\\chi) \\bar \\varepsilon}2$$ does not hold, {\\bf then} set $\\lam = \\lam/2$ and go to step~1; {\\bf else}, \n        set $\\lam_k=\\lam$, $\\hat x_k=x$,  $k\\leftarrow k+1$, and go to step 1.\n\t\\end{itemize}\n\t\\rule[0.5ex]{1\\columnwidth}{1pt}\n","sha256_utf8":"74df4e30b23473165213342c87732bcb259b1fb9afea66b3a55f20d6b0fa4d61"},{"start":83232,"end":83335,"start_line":1690,"end_line":1692,"raw_tex":"\\begin{equation}\\label{eq:ProbIntro}\n\t\\phi_{*}:=\\min \\left\\{\\phi(x): x \\in \\R^n\\right\\}\n\t\\end{equation}","sha256_utf8":"5cc03961598fcbe5d0710b288bcdd567f5151b6f7ba306a7556742c5ed5abf90"},{"start":83336,"end":83364,"start_line":1693,"end_line":1693,"raw_tex":"where $\\phi \\in \\bConv{n}$.\n","sha256_utf8":"37c570c6d20764a7fd3a69bf33559744d4950a92b69e58f410834819e0bfc6fa"},{"start":85644,"end":86323,"start_line":1737,"end_line":1754,"raw_tex":"\t Black-box (BB)\n     $(x^-,\\chi,\\varepsilon)$\n\t\n\t\\noindent\\rule[0.5ex]{1\\columnwidth}{1pt}\n    {\\bf Input:}  $(x^-,\\chi,\\varepsilon) \\in \\dom \\phi \\times [0,1) \\times \\R_{++}$.\n    \n    \\vgap\n    \n    \\noindent\n    {\\bf Output:}\n    $(x,y,\\Gamma,\\lam) \\in \\dom \\phi \\times \\dom \\phi \\times \\bConv{n} \\times \\R_{++}$ satisfying $\\Gamma \\le \\phi$,\n\\begin{equation}\\label{propyb}\n    \\phi(y)+\\frac{\\chi}{2 \\lambda}\\|y-x^-\\|^2 - \\underset{u\\in\\R^n}\\min\\left\\{\\Gamma(u) + \\frac{1}{2\\lam}\\|u- x^- \\|^2\\right\\} \\le \\varepsilon,\n\\end{equation}\nand\n\\beq \n    x = \\underset{u\\in\\R^n} \\argmin \\left\\lbrace \\Gamma(u) + \\frac{1}{2\\lam}\\|u- x^- \\|^2 \\right\\rbrace. \\label{defxhatk}\n     \\eeq\n","sha256_utf8":"512a5d7b5e881e79beabf8442610befed8e484b875d9e8818d6b27b3ef80fec3"},{"start":88341,"end":88512,"start_line":1795,"end_line":1798,"raw_tex":"where \n\\begin{equation}\\label{eq:sigma}\n    \\sigma = \\sigma(\\mu,\\nu) := \\frac{\\lam[\\nu(1+\\mu \\lam)+\\chi(\\mu-\\nu)]}{1+\\mu  \\lambda+\\chi  \\lambda(\\nu-\\mu)}.\n\\end{equation} \n","sha256_utf8":"317a2abbe081a4b456151589acd4e65246e97c51af803b63a5eeae4e3bae33b6"},{"start":95560,"end":95879,"start_line":1920,"end_line":1925,"raw_tex":"\t FSCO$(\\hat x_0, \\chi, \\bar \\varepsilon)$\n\t\n\t\\noindent\\rule[0.5ex]{1\\columnwidth}{1pt}\n    % {\\bf Input:} $(\\hat x_0, \\chi, \\bar \\varepsilon) \\in \\dom \\phi \\times [0,1) \\times \\R_+ $\n\t\\begin{itemize}\n\t\t\\item [0.] Let $(\\hat x_0, \\chi, \\bar \\varepsilon) \\in \\dom \\phi \\times [0,1) \\times \\R_+ $ be given and set $k=1$;\n","sha256_utf8":"51d749be865463d26c5289911f563c0e0b2ef3ad8ba134ff0530f1b0b19eaaf1"},{"start":96450,"end":96831,"start_line":1936,"end_line":1944,"raw_tex":"    \\item[1.] Compute \n$(\\hat x_k, \\hat y_k, \\hat \\Gamma_k, \\lambda_k) = \\BB(\\hat x_{k-1},\\chi, (1-\\chi) \\bar \\varepsilon/2)$;\n    \n\t\t\\item[2.]\n  Check whether $\\phi(\\hat y_k) - \\phi_* \\le \\bar \\varepsilon$ and if so {\\bf stop}; else set $k\\leftarrow k+1$ and go to step 1.\n  % \\item[3.]\n\t\t% Set $k\\leftarrow k+1$ and go to step 1.\n\t\\end{itemize}\n\t\\rule[0.5ex]{1\\columnwidth}{1pt}\n","sha256_utf8":"66c329c796219e040cab5d92a80c3215bd04f4f70bf95dd38ee18425a35d3521"},{"start":100065,"end":100504,"start_line":2010,"end_line":2017,"raw_tex":" The main complexity result for FSCO is relative to the  instances that satisfy the following condition:\n % In addition to $\\phi \\in \\bConv{n}$, this result requires the following mild assumption: \n\\begin{itemize}\n% \\item[(F1)] there exists $\\nu \\in [0,\\mu(\\phi)]$ such that\n% $\\hat \\Gamma_k \\in \\nConv{n}$;\n\\item[(F1)] there exists $\\underline \\lambda>0$ such that\n$\\lambda_k \\geq {\\underline \\lambda}$ for every $k \\ge 0$.\n\\end{itemize}\n","sha256_utf8":"3f702c36bda57670d3d0889904682f2b4451cf24e1707061d179432a9273a3d6"},{"start":110300,"end":110448,"start_line":2208,"end_line":2212,"raw_tex":"Define \n \\begin{align}\n\t s &:= \\frac{x^- - x}{\\lam}, \\label{defsb}\\\\\n\t \\eta &:= \\phi(y) - \\Gamma(x) - \\inner{s}{y- x}. \\label{defetab}\n\t\\end{align}\n","sha256_utf8":"19f622c7782c229d4b0dc44f1aac37d31320bcdada3a9ad66f1bef258ac3e745"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{thoutercs}\n% Let $\\chi \\in [0,1)$ and tolerance $\\bar \\varepsilon>0$ be given, and\n% consider U-CS with $\\varepsilon = (1-\\chi)\\bar \\varepsilon/2$.\nU-CS terminates in at most\n\\begin{equation}\\label{complprimal-2}\n\\min\\left\\{\\min\\left[ \\frac{1}{\\chi}\\left(1+\\frac{Q({\\bar \\varepsilon})}{\\bar \\varepsilon \\mu_{\\phi}}\\right),  1+\\frac{Q({\\bar \\varepsilon})}{\\bar \\varepsilon \\mu_h}\\right]\n\\log \\left(1+\\frac{\\mu_\\phi d_0^2}{\\bar \\varepsilon}\\right)\\, , \\, \\frac{d_0^2 Q({\\bar \\varepsilon})}{{\\bar \\varepsilon}^2} \\right\\}+\n\\displaystyle \\left \\lceil \n2 \\log \\frac{\\lambda_0 Q({\\bar \\varepsilon})}{\\bar \\varepsilon} \\right \\rceil\n\\end{equation}\niterations and\n\\begin{equation}\\label{defQf}\nQ({\\bar \\varepsilon})= \\frac{8 M_f^2}{(1-\\chi)^2}+ {\\bar \\varepsilon} \\left( \\lam_0^{-1} + \n\\frac{2L_f}{(1-\\chi)^2} \\right).\n\\end{equation}\n\\end{theorem}","units":[{"start":50372,"end":51233,"start_line":999,"end_line":1014,"raw_tex":"\\begin{theorem}\\label{thoutercs}\n% Let $\\chi \\in [0,1)$ and tolerance $\\bar \\varepsilon>0$ be given, and\n% consider U-CS with $\\varepsilon = (1-\\chi)\\bar \\varepsilon/2$.\nU-CS terminates in at most\n\\begin{equation}\\label{complprimal-2}\n\\min\\left\\{\\min\\left[ \\frac{1}{\\chi}\\left(1+\\frac{Q({\\bar \\varepsilon})}{\\bar \\varepsilon \\mu_{\\phi}}\\right),  1+\\frac{Q({\\bar \\varepsilon})}{\\bar \\varepsilon \\mu_h}\\right]\n\\log \\left(1+\\frac{\\mu_\\phi d_0^2}{\\bar \\varepsilon}\\right)\\, , \\, \\frac{d_0^2 Q({\\bar \\varepsilon})}{{\\bar \\varepsilon}^2} \\right\\}+\n\\displaystyle \\left \\lceil \n2 \\log \\frac{\\lambda_0 Q({\\bar \\varepsilon})}{\\bar \\varepsilon} \\right \\rceil\n\\end{equation}\niterations and\n\\begin{equation}\\label{defQf}\nQ({\\bar \\varepsilon})= \\frac{8 M_f^2}{(1-\\chi)^2}+ {\\bar \\varepsilon} \\left( \\lam_0^{-1} + \n\\frac{2L_f}{(1-\\chi)^2} \\right).\n\\end{equation}\n\\end{theorem}","sha256_utf8":"840bf6b2b0075a06971bf16cdafcc8abcd63626cf61ab407b4eb30325bc73a0f"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\\label{lem:lam}\nIf $\\lam$ satisfies \n\\begin{equation}\\label{ineq:lam}\n    \\lam \\le \\frac{(1-\\chi)^2 \\bar \\varepsilon}{ 4{M}_f^2 + \\bar \\varepsilon {L}_f},\n\\end{equation}\nthen the point $x=x_k(\\lam)$ computed in \\eqref{def:x} satisfies the inequality in step 2. As a consequence, if the initial prox stepsize $\\lam_0$ is chosen to be the right hand side of \\eqref{ineq:lam}, then $\\lam$ remains constant throughout the method.\n\\end{lemma}\n\n\\begin{proof}\n    Assume that $\\lam$ satisfies \\eqref{ineq:lam}. Using \\eqref{ineq:est} with $(M_f, L_f, u,v)=(M_f, L_f, x,\\hat x_{k-1})$ and the inequality \n    $2M_f \\Delta - a \\Delta^2 \\leq M_f^2/a$ with\n    $\\Delta=\\|x-\\hat x_{k-1}\\|$ and $a=(1-\\chi-\\lam L_f)/2\\lam$, we have\n    \\begin{align*}\n    f(x)-\\ell_f(x;\\hat x_{k-1}) +\\frac{\\chi-1}{2 \\lambda}  \\|x-\\hat x_{k-1}\\|^2\n    &\\stackrel{\\eqref{ineq:est}}\\le 2M_f \\|x-\\hat x_{k-1}\\| + \\frac{L_f}2 \\|x-\\hat x_{k-1}\\|^2+\\frac{\\chi-1}{2 \\lambda  }\\|x-\\hat x_{k-1}\\|^2 \\\\\n    &= 2M_f \\|x-\\hat x_{k-1}\\| - \\frac{1-\\chi-\\lam L_f}{2\\lam} \\|x-\\hat x_{k-1}\\|^2 \\\\\n    &\\le \\frac{2\\lam M_f^2}{1-\\chi-\\lam L_f} \\stackrel{\\eqref{ineq:lam}}\\le \\frac{(1-\\chi)\\bar \\varepsilon}2, %\\label{ineq:error}\n    \\end{align*}\n    where the last inequality is due to\n    \\eqref{ineq:lam}.\n    Hence, the lemma is proved.\n\\end{proof}\n\n\\begin{proposition}\\label{propouter}\nAssume that  $\\phi \\in \\bConv{n}$ and let $\\mu=\\mu_\\phi$.\nAssume also that\n$(x,y,\\Gamma,\\lam) = \\BB(x^-,\\chi,\\varepsilon)$\nfor some $(x^-,\\chi,\\varepsilon) \\in \\dom \\phi \\times [0,1) \\times \\R^n_{++}$, and $\\Gamma \\in \\nConv{n}$ for some $\\nu \\in [0,\\mu]$.\nThen, for every $u \\in \\R^n$, we have\n    \n% and $\\zeta \\in [0,\\chi]$,\n\\begin{equation}\\label{ineq:contraction}\n    2 \\lambda\\left[\\phi(y)-\\phi(u)\\right]  \\le \\frac{2\\lam \\varepsilon}{1-\\chi} + \\left\\|x^- - u\\right\\|^2 - (1+\\sigma) \\|x-u\\|^2,\n\\end{equation}\nwhere \n\\begin{equation}\\label{eq:sigma}\n    \\sigma = \\sigma(\\mu,\\nu) := \\frac{\\lam[\\nu(1+\\mu \\lam)+\\chi(\\mu-\\nu)]}{1+\\mu  \\lambda+\\chi  \\lambda(\\nu-\\mu)}.\n\\end{equation} \n\\end{proposition}\n\n\\begin{theorem}\\label{thouter}\n% Assume that (F1) holds. Then,\nThe number of iterations performed by any instance of  FSCO that satisfies condition (F1) is bounded by\n\\begin{equation}\\label{complprimal}\n \\min\\left\\{\\min\\left[ \\frac{1}{\\chi}\\left(1+\\frac{1}{\\underline{\\lam} \\mu}\\right),  1+\\frac{1}{\\underline{\\lam} \\nu}\\right] \\log\\left(1+ \\frac{\\mu d_0^2}{\\bar \\varepsilon}\\right), \\frac{d_0^2}{{\\underline \\lambda}{\\bar \\varepsilon}} \\right\\},\n\\end{equation}\nwhere the scalars $\\chi$ and $\\bar \\varepsilon$ are input to FSCO, $d_0$ is as in \\eqref{defd0}, and\n\\begin{equation}\\label{eq:nu}\n    \\mu=\\mu_\\phi, \\quad \\nu := \\min \\left\\{ \\mu_\\phi \\, , \\, \\inf_k \\mu (\\hat \\Gamma_k) \\right\\}.\n\\end{equation}\n${\\rm (}$By convention, \\eqref{complprimal} \nshould be understood as being equal to $d_0^2/({\\underline{\\lambda}}{\\bar{\\varepsilon}})$ when $\\mu=0$.${\\rm )}$\n\\end{theorem}\n\n\\begin{proof}\nIt is straightforward to see from Lemma \\ref{lem:sigma}(b) that $\\sigma(\\lam) \\ge \\underline \\sigma := \\sigma(\\underline \\lam)$ for $\\lam \\ge \\underline \\lam$, where $\\sigma(\\lam)$ denotes the $\\sigma$ defined in \\eqref{eq:sigma}, considered as a function of $\\lam$.\nNote that we invoke the oracle $(\\hat x_k, \\hat y_k, \\hat \\Gamma_k, \\lambda_k) = \\BB(\\hat x_{k-1},\\chi, (1-\\chi) \\bar \\varepsilon/2)$ in step 1.\nUsing \\eqref{ineq:contraction}, assumption (F1), and the observation that $\\sigma(\\lam_k) \\ge \\underline \\sigma$, we have\n\\begin{equation}\\label{ohiustb}\n    2 \\lambda_k\\left[\\phi(\\hat y_k)-\\phi(u)\\right]  \\le \\lam_k \\bar \\varepsilon + \\left\\|\\hat x_{k-1}-u\\right\\|^2 - (1+\\underline \\sigma) \\|\\hat x_k-u\\|^2,\n\\end{equation}\n% ------------\n% \n% Assume that\n% \\[\n%     2 \\lambda_k\\left[\\phi(\\hat y_k)-\\Gamma_k(u)\\right]  \\le \\frac{2\\lam_k \\varepsilon}{1-\\chi} + \\left\\|\\hat x_{k-1}-u\\right\\|^2 - (1+\\underline \\sigma_1) \\|\\hat x_k-u\\|^2,\n% \\]\n% \\[\n% \\phi_* \\ge \\ell_K \\ge \\Gamma_K(u_K)  \\ge \\Gamma_k(u_K) \\quad \\forall k \\le K\n% \\]\n% \\[\n% \\|\\hat x_K - u_K\\| \n% \\le \\alpha_K \\|\\hat x_0-u_k\\| + \\tau = \n% \\]\n% \\[\n% \\ell_K = \\max \\{ \\Gamma_k(u_k) : k \\le K \\}\n% \\]\n% \\[\n% \\phi(y_K) - \\ell_K \\le \\bar \\varepsilon\n% \\]\n% \\[\n% u_k = \\min \\{ \\Gamma_k(u) : u \\in \\Omega \\}\n% \\]\n% \n% -------------\nwhere \n\\begin{equation}\\label{def:sigma}\n    \\underline \\sigma = \\frac{\\underline{\\lam}[\\nu(1+\\mu \\underline{\\lam})+\\chi(\\mu-\\nu)]}{1+\\mu {\\underline \\lambda}+\\chi {\\underline \\lambda}(\\nu-\\mu)},\n\\end{equation}\nand $\\mu$ and $\\nu$ are as in \\eqref{eq:nu}.\nIt is easy to see that \\eqref{ohiustb} with $u=x_*$, where $x_*$ denotes the closest solution of \\eqref{eq:ProbIntro} to the initial point $\\hat x_0$, satisfies \n$$\n\t\\gamma_k \\eta_k \\le \\alpha_{k-1} - (1+\\sigma)\\alpha_k + \\gamma_k \\delta\n$$\nwith\n\\begin{equation}\\label{eq:parameter}\n    \\gamma_k = 2\\lam_k, \\quad \\eta_k=\\phi(\\hat y_k) - \\phi_*, \\quad \\alpha_k = \\|\\hat x_k-x_*\\|^2, \\quad \\delta = \\frac{\\bar \\varepsilon}{2}, \\quad \\sigma = \\underline \\sigma, \\quad {\\underline \\gamma} = 2{\\underline \\lam},\n\\end{equation}\nwhere ${\\underline \\gamma}$\nsatisfies $\\gamma_k \\geq {\\underline \\gamma}$.\n\nUsing Lemma \\ref{lm:easyrecur1}(b) with the above parameters, Lemma \\ref{lem:sigma}(c), and the definition of $\\underline \\sigma$ in \\eqref{def:sigma} that the complexity to find a $\\bar \\varepsilon$-solution is\n\\begin{equation}\\label{eq:complexity}\n  \\min\\left\\{\\frac{1+\\underline \\sigma}{\\underline \\sigma} \\log\\left(1+ \\frac{\\underline\\sigma d_0^2}{{\\underline \\lambda}{\\bar \\varepsilon}}\\right), \\frac{d_0^2}{{\\underline \\lambda}{\\bar \\varepsilon}} \\right\\} \\le \\min\\left\\{\\min\\left[ \\frac{1}{\\chi}\\left(1+\\frac{1}{\\underline{\\lam} \\mu}\\right),  1+\\frac{1}{\\underline{\\lam} \\nu}\\right] \\log\\left(1+ \\frac{\\underline \\sigma d_0^2}{{\\underline \\lambda}{\\bar \\varepsilon}}\\right), \\frac{d_0^2}{{\\underline \\lambda}{\\bar \\varepsilon}} \\right\\}.  \n\\end{equation}\n% Moreover, using the definition of $\\underline \\sigma$ in \\eqref{def:sigma} and the assumptions that $\\chi\\in [0,1)$ and $\\nu \\le \\mu$, we have\n% \\[\n% \\frac{\\underline \\sigma}{\\underline{\\lam}} \\stackrel{\\eqref{def:sigma}}= \\frac{\\nu(1+\\mu \\underline{\\lam})+\\chi(\\mu-\\nu)}{1+\\mu {\\underline \\lambda}+\\chi {\\underline \\lambda}(\\nu-\\mu)} \\le \\frac{\\nu(1+\\mu \\underline{\\lam})+\\mu-\\nu}{1+\\mu {\\underline \\lambda}+ {\\underline \\lambda}(\\nu-\\mu)} = \\frac{\\mu(1+\\nu \\underline{\\lam})}{1+\\nu {\\underline \\lambda}} = \\mu.\n% \\]\nMoreover, we note that $\\underline \\sigma \\le \\underline \\lam \\mu$ in view of the second inequality in Lemma \\ref{lem:sigma}(a).\nFinally, this observation and \\eqref{eq:complexity} immediately imply the complexity in \\eqref{complprimal}.\n\\end{proof}\n\n\\begin{lemma}\\label{lemetab} \nDefine \n \\begin{align}\n\t s &:= \\frac{x^- - x}{\\lam}, \\label{defsb}\\\\\n\t \\eta &:= \\phi(y) - \\Gamma(x) - \\inner{s}{y- x}. \\label{defetab}\n\t\\end{align}\n\nThen, the following statements hold:\n\\begin{itemize}\n\\item[a)]\n  $s \\in \\partial \\Gamma (x)$, i.e., for every $u \\in \\R^n$,\n \\begin{equation}\\label{eq:subdifb}\n    \\Gamma(u) \\ge \\Gamma(x) + \\inner{s}{u-x}; \n \\end{equation}  \n  \\item[b)] $s \\in \\partial_{\\eta} \\phi(y)$ and\n  \\begin{equation}\\label{ineqetab}\n  0 \\le 2 \\lam \\eta \\le  2 \\lam \\varepsilon - \n\t    \\|y- x\\|^2 + (1-\\chi)\\|y- x^-\\|^2.\n     \\end{equation}\n\\end{itemize}\n\\end{lemma}\n\n\\begin{proof} \n (a) The optimality condition of \\eqref{defxhatk} yields $0 \\in \\partial \\Gamma_k(x) +(x-x^-)/\\lambda$, which together with the definition of $s$ in \\eqref{defsb} implies that the inclusion in a) holds.\n Relation \\eqref{eq:subdifb} immediately follows from the inclusion in a) and the definition of the subdifferential.\n \n(b) The relation $\\phi \\geq \\Gamma$ (see step 1 of FSCO), \\eqref{eq:subdifb}, the definition of $\\eta$ in \\eqref{defetab} imply that for every $u\\in \\dom \\phi$,\n\t\\[\n\t\\phi(u) \\ge \\Gamma(u) \\stackrel{\\eqref{eq:subdifb}}\\ge  \\Gamma(x) + \\inner{s}{u- x}\n      \\stackrel{\\eqref{defetab}}=  \\phi(y) + \\inner{s}{u- y} - \\eta,\n\t\\]\nwhich yields the inclusion in b). \nTaking $u=y$ in the above inequality gives $\\eta \\ge 0$, and hence the first inequality in \\eqref{ineqetab} holds.\nUsing the definitions of $s$ and $\\eta$ in \\eqref{defsb} and \\eqref{defetab}, respectively, and \\eqref{propyb} and \\eqref{defxhatk}, we have\n\\[\n\\begin{array}{lcl}\n    \\eta &\\stackrel{\\eqref{defetab}}=& \\displaystyle \\phi(y) - \\Gamma(x) - \\inner{s}{y- x} \\\\\n     & \\stackrel{\\eqref{propyb},\\eqref{defxhatk}}{\\le} &\\displaystyle \\left[ \\varepsilon - \\frac\\chi{2\\lam}\\|y- x^-\\|^2 + \\frac{1}{2\\lam}\\|x- x^- \\|^2 \\right] - \\inner{s}{y- x}\\\\\n\t    &\\stackrel{\\eqref{defsb}}{=} &  \\displaystyle \\varepsilon - \\frac\\chi{2\\lam}\\|y- x^-\\|^2  +  \\frac1{2\\lam}\\|x- x^-\\|^2 + \\Inner{\\frac{x- x^-}{\\lam}}{y- x} \\\\\n     &=&\\displaystyle  \\varepsilon + \\frac{1-\\chi}{2\\lam}\n\t    \\|y- x^- \\|^2 - \\frac1{2\\lam}\n\t    \\|y- x\\|^2.\n\t\\end{array}\n\\]\nHence, the second inequality in \\eqref{ineqetab} holds.\n% (c) Since $\\phi-\\mu\\|\\cdot\\|^2/2$ is convex, using the inclusion in part (b) and Lemma \\ref{lem:app} with $\\tau=(1-\\zeta)/\\zeta$ for $\\zeta \\in [0,1)$, we obtain\n% \\[\n% \\phi(u) \\ge \\phi(y) + \\inner{s}{u- y} + \\frac{\\zeta \\mu}{2} \\|u- y\\|^2 - \\frac{\\eta}{1-\\zeta}.\n% \\]\n% That is \\eqref{qphi-1} with $\\nu = 0$.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:zeta}\n    Let $\\mu=\\mu_\\phi$ and assume that $\\Gamma \\in \\nConv{n}$ for some $\\nu \\in [0,\\mu]$.\n    For any $\\zeta \\in [0,1)$ and every $u\\in \\dom \\phi$, we have\n     \\begin{equation}\\label{qphi-1}\n    \\phi(u) \\ge \\phi(y) + \\inner{s}{u- y} + \\frac{\\zeta (\\mu-\\nu)}{2} \\|u- y\\|^2 - \\frac{\\eta}{1-\\zeta} + \\frac{\\zeta}{1-\\zeta} \\frac{\\nu}{2}\\|y - x\\|^2 + \\frac{\\nu}{2}\\|u - x\\|^2,\n\t\\end{equation}\n    where $s$ and $\\eta$ are as in \\eqref{defsb} and \\eqref{defetab}, respectively.\n\\end{lemma}\n\n\\begin{proof}\n    Define\n\\begin{equation}\\label{eq:tilde}\n    \\tilde s = s + \\nu(x^- - x), \\quad \\tilde \\eta = \\eta - \\frac{\\nu}{2}\\|y - x\\|^2, \\quad \\tilde \\phi = \\phi - \\frac{\\nu}{2}\\|\\cdot - x^-\\|^2, \\quad \\tilde \\Gamma = \\Gamma - \\frac{\\nu}{2}\\|\\cdot - x^-\\|^2.\n\\end{equation}\nIt follows from Lemma \\ref{lemetab}(a) and the above definitions of $\\tilde s$ and $\\tilde \\Gamma$ that\n$\\tilde s \\in \\partial \\tilde \\Gamma(x)$.\nIn view of \\eqref{eq:tilde} and the assumption that $\\Gamma \\in \\nConv{n}$ for some $\\nu \\in [0,\\mu]$, we observe that $\\tilde \\Gamma$ is convex and $\\tilde \\phi \\ge \\tilde \\Gamma$.\nHence, following a similar argument as in Lemma~\\ref{lemetab}(b), we conclude that\n$\\tilde s \\in \\partial_{\\tilde \\eta} \\tilde \\phi(y)$.\nUsing this inclusion, the fact that $\\tilde \\phi$ is \n$(\\mu -\\nu)$-convex, and Lemma \\ref{lem:app} with $(\\psi,\\xi,y,v,\\eta,Q)=(\\tilde \\phi, \\mu-\\nu,y,\\tilde s, \\tilde \\eta,I)$, we have for any $\\tau\\in (0,\\infty]$ and every $u\\in \\dom \\phi$,\n\\[\n\\tilde \\phi(u) \\stackrel{\\eqref{ineq:app}}\\ge \\tilde \\phi(y) + \\inner{\\tilde s}{u- y} + \\frac{\\mu -\\nu}{2(1+\\tau)} \\|u- y\\|^2 -  (1+\\tau^{-1}) \\tilde \\eta.\n\\]\nTaking $\\tau=(1-\\zeta)/\\zeta$ for $\\zeta \\in [0,1)$ in the above inequality, we obtain\n\\[\n     \\tilde \\phi(u) \\ge \\tilde \\phi(y) + \\inner{\\tilde s}{u- y} + \\frac{\\zeta (\\mu -\\nu)}{2} \\|u- y\\|^2 -   \\frac{\\tilde \\eta}{1-\\zeta}.\n\t\\]\nFinally, \\eqref{qphi-1} follows from using the definitions of $\\tilde \\phi$, $\\tilde s$, and $\\tilde \\eta$ in \\eqref{eq:tilde}, and rearranging the terms in the above inequality.\n\\end{proof}\n\n{\\bf Proof of Proposition \\ref{propouter}.}\nIt follows from the definition of $s$ in \\eqref{defsb} and Lemma \\ref{lem:zeta} with $\\zeta =\\chi$,\n\\begin{align}\n&\\|x^--u \\|^2  - \\|x-u\\|^2 -\\left(\\|x^--y\\|^2 - \\|x-y\\|^2\\right) \n= -2\\lam \\inner{s}{u-y} \\nn \\\\\n\\stackrel{\\eqref{qphi-1}}\\ge & 2 \\lambda \\left[\\phi(y)- \\phi(u)\\right] + \\chi (\\mu -\\nu) \\lam \\|u-y\\|^2 -   \\frac{2\\lam \\eta}{1-\\chi} + \\frac{\\chi \\nu \\lam}{1-\\chi} \\|y - x\\|^2 + \\nu \\lam\\|u - x\\|^2. \\label{ineq:long}\n\\end{align}\nRearranging the terms in \\eqref{ineq:long} and using Lemma \\ref{lemetab}(b), we have\n\\begin{align}\n&\\left\\|x^--u \\right\\|^2  - (1+\\nu \\lam)\\|x-u\\|^2  -2 \\lambda \\left[\\phi(y)- \\phi(u)\\right] \\nn \\\\\n& \\stackrel{\\eqref{ineq:long}}\\ge \\|x^--y\\|^2 + \\left( \\frac{\\chi \\nu \\lam}{1-\\chi} - 1 \\right)\\|x-y\\|^2 + \\chi (\\mu -\\nu) \\lam \\|u-y\\|^2 -   \\frac{2\\lam \\eta}{1-\\chi} \\nn \\\\\n& \\stackrel{\\eqref{ineqetab}}\\ge  -\\frac{2 \\lam \\varepsilon}{1-\\chi} + \\frac{\\chi(1+\\nu \\lambda)}{1-\\chi} \\|x-y\\|^2 + \\chi (\\mu -\\nu) \\lam \\|u-y\\|^2. \\label{ineq:imp}\n\\end{align}\nRearranging the terms in the above inequality, we have\n\\begin{align}\n    2\\lambda \\left[\\phi(y)- \\phi(u)\\right] \\le& \n\\frac{2 \\lam \\varepsilon}{1-\\chi} + \\left\\|x^--u \\right\\|^2  - (1+\\nu \\lam)\\|x-u\\|^2 \\nn \\\\\n&- \\chi \\left(\\frac{1+\\nu \\lam}{1-\\chi} \\|x-y\\|^2 + (\\mu -\\nu) \\lam \\|u-y\\|^2\\right). \\label{telescopic1}\n\\end{align}\nUsing the triangle inequality and the fact that $(a_1+a_2)^2 \\le (b_1^{-1}+b_2^{-1}) (a_1^2 b_1 + a_2^2 b_2)$ with $(a_1,a_2) =(\\|x-y\\|,\\|u-y\\|)$ and $(b_1,b_2) =((1+\\nu \\lam)/(1-\\chi), (\\mu -\\nu) \\lam)$\nwe have\n\\begin{equation}\\label{ineqxku}\n    \\|x-u\\|^2 \\le \\left(\\frac{1-\\chi}{1+\\nu \\lam} + \\frac{1}{(\\mu -\\nu)\\lam}\\right) \\left(\\frac{1+\\nu \\lam}{1-\\chi} \\|x-y\\|^2 + (\\mu -\\nu) \\lam \\|u-y\\|^2\\right).\n\\end{equation}\nPlugging the above inequality into \\eqref{telescopic1}, we have\n\\[\n2\\lambda \\left[\\phi(y)-\\phi(u)\\right] \n\\le   \\frac{2 \\lam \\varepsilon}{1-\\chi} +  \n\\left\\|x^--u\\right\\|^2  - \n\\left[ 1 + \\nu \\lam + \\chi \\left(\\frac{1-\\chi}{1+\\nu \\lam} + \\frac{1}{(\\mu -\\nu)\\lam}\\right)^{-1} \\right]\n\\|x-u\\|^2,\n\\]\nwhich is the same as \\eqref{ineq:contraction} after simplification.\n\\QEDA\n\n\n\\begin{proposition}\\label{lem:ACSCS}\nThe following statements hold for U-CS:\n    \\begin{itemize}\n    \\item[a)] $\\{\\lam_k\\}$ is a non-increasing sequence;\n        \\item[b)] for every $k\\ge 1$, we have\n        \\begin{align}\n\t\t&\\hat x_k =\\underset{u\\in  \\R^n}\\argmin\n\t\t\\left\\lbrace  \\ell_f(u;\\hat x_{k-1})+h(u) +\\frac{1}{2\\lam_{k}} \\|u- \\hat x_{k-1} \\|^2 \\right\\rbrace, \\label{eq:sub1cs} \\\\\n\t\t&f(\\hat x_{k}) -\\ell_f(\\hat x_{k};\\hat x_{k-1})+\\frac{\\chi-1}{2 \\lambda_{k}}\\|\\hat x_{k}-\\hat x_{k-1}\\|^2 \\le \\frac{(1-\\chi)\\bar \\varepsilon}2,\n  \\label{ineq:iterationcs}\\\\\n   & \\lam_k \\ge \\min \\left \\lbrace \\frac{(1-\\chi)^2 \\bar \\varepsilon}{8 M_f^2 + 2 \\bar \\varepsilon L_f}, \\lam_0 \\right\\rbrace.\\label{ineq:lamj2}\n\t    \\end{align}\n    \n\n       \\item[c)]     U-CS is a special case of FSCO where:\n    \\begin{itemize}\n        \\item [i)]\n        $\\hat y_k=\\hat x_k$ and $\\hat \\Gamma_k(\\cdot)=\\ell_f(\\cdot;\\hat x_{k-1})+h(\\cdot)$\n    for every $k \\ge 1$;\n    \\item[ii)] assumption (F1) is satisfied with\n    $\\underline \\lam$ given by \\eqref{ineq:lamj2}.\n    \\end{itemize}\n\n    \\end{itemize}\n    \\end{proposition}\n\n\\begin{proof}\n    a) This statement directly follows from the description of U-CS.\n    \n    b) Relations \\eqref{eq:sub1cs} and \\eqref{ineq:iterationcs} directly follow from the description of U-CS.\nNext, we prove \\eqref{ineq:lamj2}.\nSupposing $\\lam_{k-1}\\le (1-\\chi)^2 \\bar \\varepsilon/(4 M_f^2 + \\bar \\varepsilon L_f)$,\nthen it follows from Lemma \\ref{lem:lam} that \\eqref{ineq:iterationcs} holds with $\\lam_k$ replaced by $\\lambda_{k-1}$. This indicates that if $\\lam$ is small enough, then it will remain unchanged. Therefore, following from the update scheme of $\\lam$ in step 2 of U-CS, there is a lower bound as in \\eqref{ineq:lamj2}.\n\nc) Relations\n  \\eqref{eq:sub1cs}  and\n     \\eqref{ineq:iterationcs}\n     are the analogues of\n     relations \\eqref{propyb} and  \\eqref{defxhatk}\n     of FSCO with \n$\\hat y_k=\\hat x_k$ and $\\hat \\Gamma_k$  as in (i).\nInequality \\eqref{ineq:lamj2}\nshows that Assumption (F1) is\nsatisfied. \n    \\end{proof}\n\n{\\bf Proof of Theorem \\ref{thoutercs}.}\nDefine \n\\begin{equation}\\label{defkbar}\n\\bar k=\\left \\lceil 2 \\log \\max\\left\\{\\frac{ \\lambda_0 (8M_f^2 + 2 \\bar \\varepsilon L_f) }{(1-\\chi)^2 \\bar \\varepsilon},1\\right\\} \\right \\rceil.\n\\end{equation}\nObserve that $\\lambda_0/2^{k} \\leq {\\underline \\lambda}$ for $k \\geq \\bar k$ where ${\\underline{\\lambda}}$ is given by the right-hand side of \\eqref{ineq:lamj2},\nand from Proposition \\ref{lem:ACSCS} that we cannot halve $\\lambda$ more than $\\bar k$ iterations. \nIt follows from \\eqref{defkbar} and the definition of $ Q(\\bar \\varepsilon)$ in \\eqref{defQf} that\n\\[\n\\bar k \\leq  \\displaystyle \\left \\lceil \n2 \\log \\frac{\\lambda_0 Q({\\bar \\varepsilon})}{\\bar \\varepsilon} \\right \\rceil.\n\\]\nTherefore, the second term on the right-hand side of \\eqref{complprimal-2}\ngives an upper bound on the number of iterations with backtracking of $\\lambda$.\nWe now provide a bound on the number of remaining iterations to obtain a $\\bar \\varepsilon$-solution with U-CS.\nIn the $k$-th iteration of U-CS, the model $\\hat \\Gamma_k$ for $\\phi=f+h$ is simply $\\ell_f(\\cdot,\\hat x_{k-1})+h(\\cdot)$ where $\\ell_f$ is the linearization of $f$ as in \\eqref{linfdef}, hence $\\mu(\\hat \\Gamma_k) = \\mu_h \\le \\mu_\\phi$ and $\\nu=\\mu_h$ in \\eqref{eq:nu}.\nSince Proposition \\ref{lem:ACSCS} shows that U-CS is a special case of FSCO, Theorem~\\ref{thouter} immediately gives for U-CS the upper bound \\eqref{complprimal} with ${\\underline{\\lambda}}$ given by \\eqref{ineq:lamj2}\non the number of the remaining iterations\n(where $\\lambda$ is not halved)\nrequired to find a $\\bar \\varepsilon$-solution \nof \\eqref{eq:ProbIntro2}.\nUsing the inequality\n\\begin{equation}\\label{upperlbar}\n\\frac{1}{\\underline{\\lam}} \\stackrel{\\eqref{ineq:lamj2}}=\\max \\left \\lbrace \\frac{8 M_f^2 + 2 \\bar \\varepsilon L_f}{(1-\\chi)^2 \\bar \\varepsilon}, \\frac{1}{\\lambda_0} \\right\\rbrace\n\\leq \\frac{8 M_f^2 + 2 \\bar \\varepsilon L_f}{(1-\\chi)^2 \\bar \\varepsilon} + \\frac{1}{\\lambda_0}\n\\end{equation}\nand the definition of $ Q(\\bar \\varepsilon)$ in \\eqref{defQf},\nwe conclude that\n$1/\\underline{\\lam} \\le \nQ(\\bar \\varepsilon)/\\bar \\varepsilon$.\nThis observation and \\eqref{complprimal} with ${\\underline{\\lambda}}$ given by \\eqref{ineq:lamj2} thus imply that the first term in \\eqref{complprimal-2} is an upper bound on the number of the remaining iterations\n(where $\\lambda$ is not halved).\nThis completes the proof.\n\\QEDA\n\n\n\\begin{lemma}\\label{lem:app}\nAssume that $\\xi>0, \\psi \\in \\bConv{n}$ and $Q \\in \\mathcal{S}_{++}^n$ are such that $\\psi-$ $(\\xi / 2)\\|\\cdot\\|_Q^2$ is convex and let $(y, v, \\eta) \\in \\R^n \\times \\R^n \\times \\R_{+}$be such that $v \\in \\partial_\\eta \\psi(y)$. Then, for any $\\tau>0$,\n    \\begin{equation}\\label{ineq:app}\n        \\psi(u) \\ge \\psi(y)+\\inner{v}{u-y}-\\left(1+\\tau^{-1}\\right) \\eta+\\frac{(1+\\tau)^{-1} \\xi}{2}\\|u-y\\|_Q^2 \\quad \\forall u \\in \\R^n .\n    \\end{equation}\n\\end{lemma}\n\n\\begin{proof}\n    The proof of this result can be found in Lemma A.1 in \\cite{melo2023proximal}.\n\\end{proof}\n\n\\begin{lemma} \\label{lm:easyrecur1}\n\t\tAssume that sequences $\\{\\gamma_j\\}$, $\\{\\eta_j\\}$, and  $\\{\\alpha_j\\}$ satisfy for every $j\\ge 1$, $\\gamma_j \\ge {\\underline \\gamma}$ and \n\t\t\\beq \\label{eq:easyrecur1}\n\t\t\\gamma_j \\eta_j \\le \\alpha_{j-1} - (1+\\sigma)\\alpha_j + \\gamma_j \\delta\n\t\t\\eeq\n  for some $\\sigma \\ge 0$, $\\delta \\ge 0$ and ${\\underline \\gamma} > 0$. Then, the following statements hold:\n\t\\begin{itemize}\n            \\item[a)] for every $k\\ge 1$,\n            \\begin{equation}\\label{ineq:eta}\n            \\min_{1\\le j \\le k}  \\eta_j \\le \\frac{\\alpha_0 - (1+\\sigma)^k \\alpha_k}{\\sum_{j=1}^k (1+\\sigma)^{j-1} \\gamma_j} + \\delta;\n        \\end{equation}\n  %           \\item[b)] if the sequence $\\{\\eta_j\\}$ is nonnegative, then for every $k\\ge 1$,\n  %           \\begin{equation}\\label{ineq:alpha}\n  %           \\alpha_k \\le \\frac{\\alpha_0}{(1+\\sigma)^k}  +\n\t\t% \\frac{\\sum_{j=1}^k (1+\\sigma)^{j-1}\\gamma_j \\delta}{(1+\\sigma)^k};\n  %       \\end{equation}\n\t\t\\item[b)]  if the sequence $\\{\\alpha_j\\}$ is nonnegative, then $ \\min_{1\\le j \\le k} \\eta_j \\le 2\\delta $ for every $ k\\ge 1$ such that\n\t\t\\[\n\t\tk \\ge \\min \\left\\lbrace  \\frac{1+\\sigma}{\\sigma} \\log\\left( \\frac{\\sigma \\alpha_0}{{\\underline \\gamma}\\delta} + 1 \\right), \\frac{\\alpha_0}{{\\underline \\gamma}\\delta} \\right\\rbrace\n\t\t\\]\n\t\twith the convention that the first term is equal to the second term\n\t\twhen $\\sigma=0$. (Note  that the first term converges to the second term\n\t\tas $\\sigma \\downarrow 0$.)\n\t\\end{itemize}\n\\end{lemma}\n\n\\begin{proof}\n\ta) Multiplying \\eqref{eq:easyrecur1} by $ (1+\\sigma)^{j-1} $ and summing the resulting inequality from $ j=1 $ to $ k $, we have\n\t\\begin{align}\n\t    \\sum_{j=1}^k (1+\\sigma)^{j-1} \\gamma_j \\left[ \\min_{1\\le j \\le k}  \\eta_j \\right] &\\le \\sum_{j=1}^k (1+\\sigma)^{j-1} \\gamma_j \\eta_j\n     \\le  \\sum_{j=1}^k (1+\\sigma)^{j-1} \\left(  \\alpha_{j-1} - (1+\\sigma) \\alpha_j + \\gamma_j \\delta \\right) \\nn \\\\\n\t\t &=  \\alpha_0 - (1+\\sigma)^k \\alpha_k +\n\t\t\\sum_{j=1}^k (1+\\sigma)^{j-1}\\gamma_j \\delta. \\label{ineq:sum}\n\t\\end{align}\n        Inequality \\eqref{ineq:eta} follows immediately from the above inequality.\n\n        % b) This statement follows immediately from \\eqref{ineq:sum} and the fact that $\\eta_j\\ge 0$.\n        \n        b) It follows from \\eqref{ineq:eta}, and the facts that $\\alpha_k\\ge 0$ and $\\gamma_j \\ge {\\underline \\gamma}$ that\n        \\begin{equation}\\label{ineq:eta1}\n            \\min_{1\\le j \\le k}  \\eta_j  \\le \\frac{\\alpha_0}{{\\underline \\gamma} \\sum_{j=1}^k (1+\\sigma)^{j-1}} + \\delta.\n        \\end{equation}\n\tUsing the fact that $ 1+\\sigma \\ge e^{\\sigma/(1+\\sigma)} $\n\tfor every $\\sigma \\ge 0$, \n\twe have\n\t\\begin{equation}\\label{ineq:theta}\n\t    \\sum_{j=1}^{k} (1+\\sigma)^{j-1} = \\max \\left\\lbrace \\frac{(1+\\sigma)^k-1}{\\sigma}, k \\right\\rbrace \\ge \\max\\left\\lbrace \\frac{e^{\\sigma k /(1+\\sigma)} - 1}{\\sigma}, k\\right\\rbrace.\n\t\\end{equation}\n\tPlugging the above inequality into \\eqref{ineq:eta1}, we have for every $ k\\ge 1 $,\n\t\\[\n\t\\min_{1\\le j \\le k}  \\eta_j \\le \\frac{\\alpha_0}{{\\underline \\gamma}}\\min \\left\\lbrace \\frac{\\sigma}{e^{\\sigma k /(1+\\sigma)} - 1}, \\frac 1k \\right\\rbrace + \\delta,\n\t\\]\n\twhich can be easily seen to imply (b).\n\\end{proof}\n\n\\begin{lemma}\\label{lem:sigma}\n  For every $\\chi \\in [0,1)$, $\\lam>0$, $\\mu>0$, and $\\nu \\in [0,\\mu]$, the \n  quantity $\\sigma=\\sigma(\\mu,\\nu;\\lambda)$ defined in \\eqref{eq:sigma} satisfies the following statements:\n  \\begin{itemize}\n      \\item[a)] $\\lam [ \\chi  \\mu + (1-\\chi)  \\nu] \\le \\sigma \\le \\lam \\mu$, and hence $\\sigma \\ge \\lambda \\nu$;\n      \\item[b)] the function $\\lam >0 \\mapsto \\sigma(\\mu,\\nu;\\lambda)$ is non-decreasing;\n      \\item[c)] there holds\n      \\[\n      \\frac{1+\\sigma}{\\sigma} \\le \\min\\left\\{ \\frac{1}{\\chi}\\left(1+\\frac{1}{\\lam \\mu}\\right),  1+\\frac{1}{\\lam \\nu}\\right\\}.\n      \\]\n  \\end{itemize}\n\\end{lemma}\n\n\\begin{proof}\n    a) It follows from \\eqref{eq:sigma} and the fact that $\\nu \\le \\mu$ that\n \\[\n    \\sigma - \\lam \\mu =\\frac{\\lam(\\mu-\\nu)(1+\\lam \\mu)(\\chi-1)}{1+\\mu  \\lambda+\\chi  \\lambda(\\nu-\\mu)}\n    \\ge \\frac{\\lam(\\mu-\\nu)(1+\\lam \\mu)(\\chi-1)}{1+\\mu  \\lambda} =\n\\lam(\\mu-\\nu)(\\chi-1),\n    \\]\n    and hence that $\\sigma \\ge \\lam [ \\chi  \\mu + (1-\\chi)  \\nu]$. Moreover, the second inequality of (a) immediately follows from the first identity above and the facts that $\\chi<1$ and $\\nu \\le \\mu$.\n\n    b) Viewing $\\sigma=\\sigma(\\lam)$ in \\eqref{eq:sigma} as a function of $\\lam$, one can verify that\n\\[\n\\sigma(\\lam) = \\frac{\\nu(1+\\mu \\lam)+\\chi(\\mu-\\nu)}{\\lam^{-1}+\\mu +\\chi (\\nu-\\mu)}\n\\]\nis a non-decreasing function of $\\lam$.\n\n    c) It follows from the definition of $\\sigma$ in \\eqref{eq:sigma} that \n    \\[\n    \\frac{1+\\sigma}{\\sigma} = \\frac{(1+\\nu {\\lam})(1+\\mu {\\lam})}{{\\lam}[\\nu(1+ \\mu {\\lam})+\\chi(\\mu-\\nu)]}.\n    \\]\n    Hence, it suffices to prove that\n\\begin{equation}\\label{ineq:combine}\n    \\frac{(1+\\nu {\\lam})(1+\\mu {\\lam})}{{\\lam}[\\nu(1+ \\mu {\\lam})+\\chi(\\mu-\\nu)]} \\le \\min\\left\\{ \\frac{1}{\\chi}\\left(1+\\frac{1}{{\\lam} \\mu}\\right),  1+\\frac{1}{{\\lam} \\nu}\\right\\}. \n\\end{equation}\nSince $\\chi \\in [0,1)$, it is easy to verify that\n\\[\n\\frac{1+\\nu {\\lam}}{\\nu(1+ \\mu {\\lam})+\\chi(\\mu-\\nu)} \\le \\frac{1}{\\chi \\mu},\n\\]\nand hence that\n\\[\n\\frac{(1+\\nu {\\lam})(1+\\mu {\\lam})}{{\\lam}[\\nu(1+ \\mu {\\lam})+\\chi(\\mu-\\nu)]} \\le \\frac{1+ \\mu {\\lam}}{\\chi {\\lam} \\mu} = \\frac{1}{\\chi}\\left(1+\\frac{1}{{\\lam} \\mu}\\right).\n\\]\nMoreover, noting that $\\chi \\in [0,1)$ and $\\mu\\ge \\nu$, we also have \n\\[\n\\frac{(1+\\nu {\\lam})(1+\\mu {\\lam})}{{\\lam}[\\nu(1+ \\mu {\\lam})+\\chi(\\mu-\\nu)]} \\le 1+\\frac{1}{{\\lam} \\nu}.\n\\]\nCombining the above two inequalities, we conclude that \\eqref{ineq:combine} holds.\n\\end{proof}","units":[{"start":48492,"end":48942,"start_line":963,"end_line":969,"raw_tex":"\\begin{lemma}\\label{lem:lam}\nIf $\\lam$ satisfies \n\\begin{equation}\\label{ineq:lam}\n    \\lam \\le \\frac{(1-\\chi)^2 \\bar \\varepsilon}{ 4{M}_f^2 + \\bar \\varepsilon {L}_f},\n\\end{equation}\nthen the point $x=x_k(\\lam)$ computed in \\eqref{def:x} satisfies the inequality in step 2. As a consequence, if the initial prox stepsize $\\lam_0$ is chosen to be the right hand side of \\eqref{ineq:lam}, then $\\lam$ remains constant throughout the method.\n\\end{lemma}","sha256_utf8":"ccc5cb70861ec652538e57a5ad53f14b75baace79b3d715a85950e188b320d58"},{"start":48945,"end":49808,"start_line":972,"end_line":985,"raw_tex":"\\begin{proof}\n    Assume that $\\lam$ satisfies \\eqref{ineq:lam}. Using \\eqref{ineq:est} with $(M_f, L_f, u,v)=(M_f, L_f, x,\\hat x_{k-1})$ and the inequality \n    $2M_f \\Delta - a \\Delta^2 \\leq M_f^2/a$ with\n    $\\Delta=\\|x-\\hat x_{k-1}\\|$ and $a=(1-\\chi-\\lam L_f)/2\\lam$, we have\n    \\begin{align*}\n    f(x)-\\ell_f(x;\\hat x_{k-1}) +\\frac{\\chi-1}{2 \\lambda}  \\|x-\\hat x_{k-1}\\|^2\n    &\\stackrel{\\eqref{ineq:est}}\\le 2M_f \\|x-\\hat x_{k-1}\\| + \\frac{L_f}2 \\|x-\\hat x_{k-1}\\|^2+\\frac{\\chi-1}{2 \\lambda  }\\|x-\\hat x_{k-1}\\|^2 \\\\\n    &= 2M_f \\|x-\\hat x_{k-1}\\| - \\frac{1-\\chi-\\lam L_f}{2\\lam} \\|x-\\hat x_{k-1}\\|^2 \\\\\n    &\\le \\frac{2\\lam M_f^2}{1-\\chi-\\lam L_f} \\stackrel{\\eqref{ineq:lam}}\\le \\frac{(1-\\chi)\\bar \\varepsilon}2, %\\label{ineq:error}\n    \\end{align*}\n    where the last inequality is due to\n    \\eqref{ineq:lam}.\n    Hence, the lemma is proved.\n\\end{proof}","sha256_utf8":"bc772fb8e637723eec011e30049d47061c81ef49276c92216a40f42e6076355b"},{"start":87789,"end":88529,"start_line":1784,"end_line":1799,"raw_tex":"\\begin{proposition}\\label{propouter}\nAssume that  $\\phi \\in \\bConv{n}$ and let $\\mu=\\mu_\\phi$.\nAssume also that\n$(x,y,\\Gamma,\\lam) = \\BB(x^-,\\chi,\\varepsilon)$\nfor some $(x^-,\\chi,\\varepsilon) \\in \\dom \\phi \\times [0,1) \\times \\R^n_{++}$, and $\\Gamma \\in \\nConv{n}$ for some $\\nu \\in [0,\\mu]$.\nThen, for every $u \\in \\R^n$, we have\n    \n% and $\\zeta \\in [0,\\chi]$,\n\\begin{equation}\\label{ineq:contraction}\n    2 \\lambda\\left[\\phi(y)-\\phi(u)\\right]  \\le \\frac{2\\lam \\varepsilon}{1-\\chi} + \\left\\|x^- - u\\right\\|^2 - (1+\\sigma) \\|x-u\\|^2,\n\\end{equation}\nwhere \n\\begin{equation}\\label{eq:sigma}\n    \\sigma = \\sigma(\\mu,\\nu) := \\frac{\\lam[\\nu(1+\\mu \\lam)+\\chi(\\mu-\\nu)]}{1+\\mu  \\lambda+\\chi  \\lambda(\\nu-\\mu)}.\n\\end{equation} \n\\end{proposition}","sha256_utf8":"4624b932533c9704aa5f405c9d744d628d6611f7191b3278e31d2afd273a4e3e"},{"start":100811,"end":101688,"start_line":2027,"end_line":2039,"raw_tex":"\\begin{theorem}\\label{thouter}\n% Assume that (F1) holds. Then,\nThe number of iterations performed by any instance of  FSCO that satisfies condition (F1) is bounded by\n\\begin{equation}\\label{complprimal}\n \\min\\left\\{\\min\\left[ \\frac{1}{\\chi}\\left(1+\\frac{1}{\\underline{\\lam} \\mu}\\right),  1+\\frac{1}{\\underline{\\lam} \\nu}\\right] \\log\\left(1+ \\frac{\\mu d_0^2}{\\bar \\varepsilon}\\right), \\frac{d_0^2}{{\\underline \\lambda}{\\bar \\varepsilon}} \\right\\},\n\\end{equation}\nwhere the scalars $\\chi$ and $\\bar \\varepsilon$ are input to FSCO, $d_0$ is as in \\eqref{defd0}, and\n\\begin{equation}\\label{eq:nu}\n    \\mu=\\mu_\\phi, \\quad \\nu := \\min \\left\\{ \\mu_\\phi \\, , \\, \\inf_k \\mu (\\hat \\Gamma_k) \\right\\}.\n\\end{equation}\n${\\rm (}$By convention, \\eqref{complprimal} \nshould be understood as being equal to $d_0^2/({\\underline{\\lambda}}{\\bar{\\varepsilon}})$ when $\\mu=0$.${\\rm )}$\n\\end{theorem}","sha256_utf8":"8ece7115244cf751d3dec9f87bf67e0dcda068762e428ee0a67b0b8893fb461b"},{"start":101691,"end":105390,"start_line":2042,"end_line":2099,"raw_tex":"\\begin{proof}\nIt is straightforward to see from Lemma \\ref{lem:sigma}(b) that $\\sigma(\\lam) \\ge \\underline \\sigma := \\sigma(\\underline \\lam)$ for $\\lam \\ge \\underline \\lam$, where $\\sigma(\\lam)$ denotes the $\\sigma$ defined in \\eqref{eq:sigma}, considered as a function of $\\lam$.\nNote that we invoke the oracle $(\\hat x_k, \\hat y_k, \\hat \\Gamma_k, \\lambda_k) = \\BB(\\hat x_{k-1},\\chi, (1-\\chi) \\bar \\varepsilon/2)$ in step 1.\nUsing \\eqref{ineq:contraction}, assumption (F1), and the observation that $\\sigma(\\lam_k) \\ge \\underline \\sigma$, we have\n\\begin{equation}\\label{ohiustb}\n    2 \\lambda_k\\left[\\phi(\\hat y_k)-\\phi(u)\\right]  \\le \\lam_k \\bar \\varepsilon + \\left\\|\\hat x_{k-1}-u\\right\\|^2 - (1+\\underline \\sigma) \\|\\hat x_k-u\\|^2,\n\\end{equation}\n% ------------\n% \n% Assume that\n% \\[\n%     2 \\lambda_k\\left[\\phi(\\hat y_k)-\\Gamma_k(u)\\right]  \\le \\frac{2\\lam_k \\varepsilon}{1-\\chi} + \\left\\|\\hat x_{k-1}-u\\right\\|^2 - (1+\\underline \\sigma_1) \\|\\hat x_k-u\\|^2,\n% \\]\n% \\[\n% \\phi_* \\ge \\ell_K \\ge \\Gamma_K(u_K)  \\ge \\Gamma_k(u_K) \\quad \\forall k \\le K\n% \\]\n% \\[\n% \\|\\hat x_K - u_K\\| \n% \\le \\alpha_K \\|\\hat x_0-u_k\\| + \\tau = \n% \\]\n% \\[\n% \\ell_K = \\max \\{ \\Gamma_k(u_k) : k \\le K \\}\n% \\]\n% \\[\n% \\phi(y_K) - \\ell_K \\le \\bar \\varepsilon\n% \\]\n% \\[\n% u_k = \\min \\{ \\Gamma_k(u) : u \\in \\Omega \\}\n% \\]\n% \n% -------------\nwhere \n\\begin{equation}\\label{def:sigma}\n    \\underline \\sigma = \\frac{\\underline{\\lam}[\\nu(1+\\mu \\underline{\\lam})+\\chi(\\mu-\\nu)]}{1+\\mu {\\underline \\lambda}+\\chi {\\underline \\lambda}(\\nu-\\mu)},\n\\end{equation}\nand $\\mu$ and $\\nu$ are as in \\eqref{eq:nu}.\nIt is easy to see that \\eqref{ohiustb} with $u=x_*$, where $x_*$ denotes the closest solution of \\eqref{eq:ProbIntro} to the initial point $\\hat x_0$, satisfies \n$$\n\t\\gamma_k \\eta_k \\le \\alpha_{k-1} - (1+\\sigma)\\alpha_k + \\gamma_k \\delta\n$$\nwith\n\\begin{equation}\\label{eq:parameter}\n    \\gamma_k = 2\\lam_k, \\quad \\eta_k=\\phi(\\hat y_k) - \\phi_*, \\quad \\alpha_k = \\|\\hat x_k-x_*\\|^2, \\quad \\delta = \\frac{\\bar \\varepsilon}{2}, \\quad \\sigma = \\underline \\sigma, \\quad {\\underline \\gamma} = 2{\\underline \\lam},\n\\end{equation}\nwhere ${\\underline \\gamma}$\nsatisfies $\\gamma_k \\geq {\\underline \\gamma}$.\n\nUsing Lemma \\ref{lm:easyrecur1}(b) with the above parameters, Lemma \\ref{lem:sigma}(c), and the definition of $\\underline \\sigma$ in \\eqref{def:sigma} that the complexity to find a $\\bar \\varepsilon$-solution is\n\\begin{equation}\\label{eq:complexity}\n  \\min\\left\\{\\frac{1+\\underline \\sigma}{\\underline \\sigma} \\log\\left(1+ \\frac{\\underline\\sigma d_0^2}{{\\underline \\lambda}{\\bar \\varepsilon}}\\right), \\frac{d_0^2}{{\\underline \\lambda}{\\bar \\varepsilon}} \\right\\} \\le \\min\\left\\{\\min\\left[ \\frac{1}{\\chi}\\left(1+\\frac{1}{\\underline{\\lam} \\mu}\\right),  1+\\frac{1}{\\underline{\\lam} \\nu}\\right] \\log\\left(1+ \\frac{\\underline \\sigma d_0^2}{{\\underline \\lambda}{\\bar \\varepsilon}}\\right), \\frac{d_0^2}{{\\underline \\lambda}{\\bar \\varepsilon}} \\right\\}.  \n\\end{equation}\n% Moreover, using the definition of $\\underline \\sigma$ in \\eqref{def:sigma} and the assumptions that $\\chi\\in [0,1)$ and $\\nu \\le \\mu$, we have\n% \\[\n% \\frac{\\underline \\sigma}{\\underline{\\lam}} \\stackrel{\\eqref{def:sigma}}= \\frac{\\nu(1+\\mu \\underline{\\lam})+\\chi(\\mu-\\nu)}{1+\\mu {\\underline \\lambda}+\\chi {\\underline \\lambda}(\\nu-\\mu)} \\le \\frac{\\nu(1+\\mu \\underline{\\lam})+\\mu-\\nu}{1+\\mu {\\underline \\lambda}+ {\\underline \\lambda}(\\nu-\\mu)} = \\frac{\\mu(1+\\nu \\underline{\\lam})}{1+\\nu {\\underline \\lambda}} = \\mu.\n% \\]\nMoreover, we note that $\\underline \\sigma \\le \\underline \\lam \\mu$ in view of the second inequality in Lemma \\ref{lem:sigma}(a).\nFinally, this observation and \\eqref{eq:complexity} immediately imply the complexity in \\eqref{complprimal}.\n\\end{proof}","sha256_utf8":"4327d1865678951064134aed0a128635f742ff5012c55710ad0be7da690f4c94"},{"start":110270,"end":110889,"start_line":2207,"end_line":2227,"raw_tex":"\\begin{lemma}\\label{lemetab} \nDefine \n \\begin{align}\n\t s &:= \\frac{x^- - x}{\\lam}, \\label{defsb}\\\\\n\t \\eta &:= \\phi(y) - \\Gamma(x) - \\inner{s}{y- x}. \\label{defetab}\n\t\\end{align}\n\nThen, the following statements hold:\n\\begin{itemize}\n\\item[a)]\n  $s \\in \\partial \\Gamma (x)$, i.e., for every $u \\in \\R^n$,\n \\begin{equation}\\label{eq:subdifb}\n    \\Gamma(u) \\ge \\Gamma(x) + \\inner{s}{u-x}; \n \\end{equation}  \n  \\item[b)] $s \\in \\partial_{\\eta} \\phi(y)$ and\n  \\begin{equation}\\label{ineqetab}\n  0 \\le 2 \\lam \\eta \\le  2 \\lam \\varepsilon - \n\t    \\|y- x\\|^2 + (1-\\chi)\\|y- x^-\\|^2.\n     \\end{equation}\n\\end{itemize}\n\\end{lemma}","sha256_utf8":"bdfb3a84253f1b63a565b1c660704b4bd045aa339139da90866585fa270db069"},{"start":110892,"end":112806,"start_line":2229,"end_line":2257,"raw_tex":"\\begin{proof} \n (a) The optimality condition of \\eqref{defxhatk} yields $0 \\in \\partial \\Gamma_k(x) +(x-x^-)/\\lambda$, which together with the definition of $s$ in \\eqref{defsb} implies that the inclusion in a) holds.\n Relation \\eqref{eq:subdifb} immediately follows from the inclusion in a) and the definition of the subdifferential.\n \n(b) The relation $\\phi \\geq \\Gamma$ (see step 1 of FSCO), \\eqref{eq:subdifb}, the definition of $\\eta$ in \\eqref{defetab} imply that for every $u\\in \\dom \\phi$,\n\t\\[\n\t\\phi(u) \\ge \\Gamma(u) \\stackrel{\\eqref{eq:subdifb}}\\ge  \\Gamma(x) + \\inner{s}{u- x}\n      \\stackrel{\\eqref{defetab}}=  \\phi(y) + \\inner{s}{u- y} - \\eta,\n\t\\]\nwhich yields the inclusion in b). \nTaking $u=y$ in the above inequality gives $\\eta \\ge 0$, and hence the first inequality in \\eqref{ineqetab} holds.\nUsing the definitions of $s$ and $\\eta$ in \\eqref{defsb} and \\eqref{defetab}, respectively, and \\eqref{propyb} and \\eqref{defxhatk}, we have\n\\[\n\\begin{array}{lcl}\n    \\eta &\\stackrel{\\eqref{defetab}}=& \\displaystyle \\phi(y) - \\Gamma(x) - \\inner{s}{y- x} \\\\\n     & \\stackrel{\\eqref{propyb},\\eqref{defxhatk}}{\\le} &\\displaystyle \\left[ \\varepsilon - \\frac\\chi{2\\lam}\\|y- x^-\\|^2 + \\frac{1}{2\\lam}\\|x- x^- \\|^2 \\right] - \\inner{s}{y- x}\\\\\n\t    &\\stackrel{\\eqref{defsb}}{=} &  \\displaystyle \\varepsilon - \\frac\\chi{2\\lam}\\|y- x^-\\|^2  +  \\frac1{2\\lam}\\|x- x^-\\|^2 + \\Inner{\\frac{x- x^-}{\\lam}}{y- x} \\\\\n     &=&\\displaystyle  \\varepsilon + \\frac{1-\\chi}{2\\lam}\n\t    \\|y- x^- \\|^2 - \\frac1{2\\lam}\n\t    \\|y- x\\|^2.\n\t\\end{array}\n\\]\nHence, the second inequality in \\eqref{ineqetab} holds.\n% (c) Since $\\phi-\\mu\\|\\cdot\\|^2/2$ is convex, using the inclusion in part (b) and Lemma \\ref{lem:app} with $\\tau=(1-\\zeta)/\\zeta$ for $\\zeta \\in [0,1)$, we obtain\n% \\[\n% \\phi(u) \\ge \\phi(y) + \\inner{s}{u- y} + \\frac{\\zeta \\mu}{2} \\|u- y\\|^2 - \\frac{\\eta}{1-\\zeta}.\n% \\]\n% That is \\eqref{qphi-1} with $\\nu = 0$.\n\\end{proof}","sha256_utf8":"49ce67dfe2022c4e83d07359b54563430cd97b6a9b259acd18cccf273b9bd85e"},{"start":113060,"end":113574,"start_line":2261,"end_line":2268,"raw_tex":"\\begin{lemma}\\label{lem:zeta}\n    Let $\\mu=\\mu_\\phi$ and assume that $\\Gamma \\in \\nConv{n}$ for some $\\nu \\in [0,\\mu]$.\n    For any $\\zeta \\in [0,1)$ and every $u\\in \\dom \\phi$, we have\n     \\begin{equation}\\label{qphi-1}\n    \\phi(u) \\ge \\phi(y) + \\inner{s}{u- y} + \\frac{\\zeta (\\mu-\\nu)}{2} \\|u- y\\|^2 - \\frac{\\eta}{1-\\zeta} + \\frac{\\zeta}{1-\\zeta} \\frac{\\nu}{2}\\|y - x\\|^2 + \\frac{\\nu}{2}\\|u - x\\|^2,\n\t\\end{equation}\n    where $s$ and $\\eta$ are as in \\eqref{defsb} and \\eqref{defetab}, respectively.\n\\end{lemma}","sha256_utf8":"f0755bdf3b5eadabdbbe4b6b16aee5ecb550a2b44ac066b0333daf312086eeac"},{"start":113576,"end":115148,"start_line":2270,"end_line":2290,"raw_tex":"\\begin{proof}\n    Define\n\\begin{equation}\\label{eq:tilde}\n    \\tilde s = s + \\nu(x^- - x), \\quad \\tilde \\eta = \\eta - \\frac{\\nu}{2}\\|y - x\\|^2, \\quad \\tilde \\phi = \\phi - \\frac{\\nu}{2}\\|\\cdot - x^-\\|^2, \\quad \\tilde \\Gamma = \\Gamma - \\frac{\\nu}{2}\\|\\cdot - x^-\\|^2.\n\\end{equation}\nIt follows from Lemma \\ref{lemetab}(a) and the above definitions of $\\tilde s$ and $\\tilde \\Gamma$ that\n$\\tilde s \\in \\partial \\tilde \\Gamma(x)$.\nIn view of \\eqref{eq:tilde} and the assumption that $\\Gamma \\in \\nConv{n}$ for some $\\nu \\in [0,\\mu]$, we observe that $\\tilde \\Gamma$ is convex and $\\tilde \\phi \\ge \\tilde \\Gamma$.\nHence, following a similar argument as in Lemma~\\ref{lemetab}(b), we conclude that\n$\\tilde s \\in \\partial_{\\tilde \\eta} \\tilde \\phi(y)$.\nUsing this inclusion, the fact that $\\tilde \\phi$ is \n$(\\mu -\\nu)$-convex, and Lemma \\ref{lem:app} with $(\\psi,\\xi,y,v,\\eta,Q)=(\\tilde \\phi, \\mu-\\nu,y,\\tilde s, \\tilde \\eta,I)$, we have for any $\\tau\\in (0,\\infty]$ and every $u\\in \\dom \\phi$,\n\\[\n\\tilde \\phi(u) \\stackrel{\\eqref{ineq:app}}\\ge \\tilde \\phi(y) + \\inner{\\tilde s}{u- y} + \\frac{\\mu -\\nu}{2(1+\\tau)} \\|u- y\\|^2 -  (1+\\tau^{-1}) \\tilde \\eta.\n\\]\nTaking $\\tau=(1-\\zeta)/\\zeta$ for $\\zeta \\in [0,1)$ in the above inequality, we obtain\n\\[\n     \\tilde \\phi(u) \\ge \\tilde \\phi(y) + \\inner{\\tilde s}{u- y} + \\frac{\\zeta (\\mu -\\nu)}{2} \\|u- y\\|^2 -   \\frac{\\tilde \\eta}{1-\\zeta}.\n\t\\]\nFinally, \\eqref{qphi-1} follows from using the definitions of $\\tilde \\phi$, $\\tilde s$, and $\\tilde \\eta$ in \\eqref{eq:tilde}, and rearranging the terms in the above inequality.\n\\end{proof}","sha256_utf8":"a1ca4c16c34b6590bfbcd812d2f56d1b6ed011ed4ea63ac19ac2b20dbd91628d"},{"start":117295,"end":119452,"start_line":2319,"end_line":2352,"raw_tex":"{\\bf Proof of Proposition \\ref{propouter}.}\nIt follows from the definition of $s$ in \\eqref{defsb} and Lemma \\ref{lem:zeta} with $\\zeta =\\chi$,\n\\begin{align}\n&\\|x^--u \\|^2  - \\|x-u\\|^2 -\\left(\\|x^--y\\|^2 - \\|x-y\\|^2\\right) \n= -2\\lam \\inner{s}{u-y} \\nn \\\\\n\\stackrel{\\eqref{qphi-1}}\\ge & 2 \\lambda \\left[\\phi(y)- \\phi(u)\\right] + \\chi (\\mu -\\nu) \\lam \\|u-y\\|^2 -   \\frac{2\\lam \\eta}{1-\\chi} + \\frac{\\chi \\nu \\lam}{1-\\chi} \\|y - x\\|^2 + \\nu \\lam\\|u - x\\|^2. \\label{ineq:long}\n\\end{align}\nRearranging the terms in \\eqref{ineq:long} and using Lemma \\ref{lemetab}(b), we have\n\\begin{align}\n&\\left\\|x^--u \\right\\|^2  - (1+\\nu \\lam)\\|x-u\\|^2  -2 \\lambda \\left[\\phi(y)- \\phi(u)\\right] \\nn \\\\\n& \\stackrel{\\eqref{ineq:long}}\\ge \\|x^--y\\|^2 + \\left( \\frac{\\chi \\nu \\lam}{1-\\chi} - 1 \\right)\\|x-y\\|^2 + \\chi (\\mu -\\nu) \\lam \\|u-y\\|^2 -   \\frac{2\\lam \\eta}{1-\\chi} \\nn \\\\\n& \\stackrel{\\eqref{ineqetab}}\\ge  -\\frac{2 \\lam \\varepsilon}{1-\\chi} + \\frac{\\chi(1+\\nu \\lambda)}{1-\\chi} \\|x-y\\|^2 + \\chi (\\mu -\\nu) \\lam \\|u-y\\|^2. \\label{ineq:imp}\n\\end{align}\nRearranging the terms in the above inequality, we have\n\\begin{align}\n    2\\lambda \\left[\\phi(y)- \\phi(u)\\right] \\le& \n\\frac{2 \\lam \\varepsilon}{1-\\chi} + \\left\\|x^--u \\right\\|^2  - (1+\\nu \\lam)\\|x-u\\|^2 \\nn \\\\\n&- \\chi \\left(\\frac{1+\\nu \\lam}{1-\\chi} \\|x-y\\|^2 + (\\mu -\\nu) \\lam \\|u-y\\|^2\\right). \\label{telescopic1}\n\\end{align}\nUsing the triangle inequality and the fact that $(a_1+a_2)^2 \\le (b_1^{-1}+b_2^{-1}) (a_1^2 b_1 + a_2^2 b_2)$ with $(a_1,a_2) =(\\|x-y\\|,\\|u-y\\|)$ and $(b_1,b_2) =((1+\\nu \\lam)/(1-\\chi), (\\mu -\\nu) \\lam)$\nwe have\n\\begin{equation}\\label{ineqxku}\n    \\|x-u\\|^2 \\le \\left(\\frac{1-\\chi}{1+\\nu \\lam} + \\frac{1}{(\\mu -\\nu)\\lam}\\right) \\left(\\frac{1+\\nu \\lam}{1-\\chi} \\|x-y\\|^2 + (\\mu -\\nu) \\lam \\|u-y\\|^2\\right).\n\\end{equation}\nPlugging the above inequality into \\eqref{telescopic1}, we have\n\\[\n2\\lambda \\left[\\phi(y)-\\phi(u)\\right] \n\\le   \\frac{2 \\lam \\varepsilon}{1-\\chi} +  \n\\left\\|x^--u\\right\\|^2  - \n\\left[ 1 + \\nu \\lam + \\chi \\left(\\frac{1-\\chi}{1+\\nu \\lam} + \\frac{1}{(\\mu -\\nu)\\lam}\\right)^{-1} \\right]\n\\|x-u\\|^2,\n\\]\nwhich is the same as \\eqref{ineq:contraction} after simplification.\n\\QEDA\n","sha256_utf8":"c1821820ea05ef94ec1782ee0f479edb88754fe977e12af50e18f11c90ccbce0"},{"start":119746,"end":120849,"start_line":2369,"end_line":2393,"raw_tex":"\\begin{proposition}\\label{lem:ACSCS}\nThe following statements hold for U-CS:\n    \\begin{itemize}\n    \\item[a)] $\\{\\lam_k\\}$ is a non-increasing sequence;\n        \\item[b)] for every $k\\ge 1$, we have\n        \\begin{align}\n\t\t&\\hat x_k =\\underset{u\\in  \\R^n}\\argmin\n\t\t\\left\\lbrace  \\ell_f(u;\\hat x_{k-1})+h(u) +\\frac{1}{2\\lam_{k}} \\|u- \\hat x_{k-1} \\|^2 \\right\\rbrace, \\label{eq:sub1cs} \\\\\n\t\t&f(\\hat x_{k}) -\\ell_f(\\hat x_{k};\\hat x_{k-1})+\\frac{\\chi-1}{2 \\lambda_{k}}\\|\\hat x_{k}-\\hat x_{k-1}\\|^2 \\le \\frac{(1-\\chi)\\bar \\varepsilon}2,\n  \\label{ineq:iterationcs}\\\\\n   & \\lam_k \\ge \\min \\left \\lbrace \\frac{(1-\\chi)^2 \\bar \\varepsilon}{8 M_f^2 + 2 \\bar \\varepsilon L_f}, \\lam_0 \\right\\rbrace.\\label{ineq:lamj2}\n\t    \\end{align}\n    \n\n       \\item[c)]     U-CS is a special case of FSCO where:\n    \\begin{itemize}\n        \\item [i)]\n        $\\hat y_k=\\hat x_k$ and $\\hat \\Gamma_k(\\cdot)=\\ell_f(\\cdot;\\hat x_{k-1})+h(\\cdot)$\n    for every $k \\ge 1$;\n    \\item[ii)] assumption (F1) is satisfied with\n    $\\underline \\lam$ given by \\eqref{ineq:lamj2}.\n    \\end{itemize}\n\n    \\end{itemize}\n    \\end{proposition}","sha256_utf8":"400c83acbe03e9c968fc9a6a023ed1547f815523a650125156a8bce5e9a387c7"},{"start":120858,"end":121804,"start_line":2394,"end_line":2412,"raw_tex":"\\begin{proof}\n    a) This statement directly follows from the description of U-CS.\n    \n    b) Relations \\eqref{eq:sub1cs} and \\eqref{ineq:iterationcs} directly follow from the description of U-CS.\nNext, we prove \\eqref{ineq:lamj2}.\nSupposing $\\lam_{k-1}\\le (1-\\chi)^2 \\bar \\varepsilon/(4 M_f^2 + \\bar \\varepsilon L_f)$,\nthen it follows from Lemma \\ref{lem:lam} that \\eqref{ineq:iterationcs} holds with $\\lam_k$ replaced by $\\lambda_{k-1}$. This indicates that if $\\lam$ is small enough, then it will remain unchanged. Therefore, following from the update scheme of $\\lam$ in step 2 of U-CS, there is a lower bound as in \\eqref{ineq:lamj2}.\n\nc) Relations\n  \\eqref{eq:sub1cs}  and\n     \\eqref{ineq:iterationcs}\n     are the analogues of\n     relations \\eqref{propyb} and  \\eqref{defxhatk}\n     of FSCO with \n$\\hat y_k=\\hat x_k$ and $\\hat \\Gamma_k$  as in (i).\nInequality \\eqref{ineq:lamj2}\nshows that Assumption (F1) is\nsatisfied. \n    \\end{proof}","sha256_utf8":"629cc21da9c534f9bc528b678f594ad117ff7a37fb96bcfda6053a90c8e293b4"},{"start":121823,"end":124224,"start_line":2417,"end_line":2450,"raw_tex":"{\\bf Proof of Theorem \\ref{thoutercs}.}\nDefine \n\\begin{equation}\\label{defkbar}\n\\bar k=\\left \\lceil 2 \\log \\max\\left\\{\\frac{ \\lambda_0 (8M_f^2 + 2 \\bar \\varepsilon L_f) }{(1-\\chi)^2 \\bar \\varepsilon},1\\right\\} \\right \\rceil.\n\\end{equation}\nObserve that $\\lambda_0/2^{k} \\leq {\\underline \\lambda}$ for $k \\geq \\bar k$ where ${\\underline{\\lambda}}$ is given by the right-hand side of \\eqref{ineq:lamj2},\nand from Proposition \\ref{lem:ACSCS} that we cannot halve $\\lambda$ more than $\\bar k$ iterations. \nIt follows from \\eqref{defkbar} and the definition of $ Q(\\bar \\varepsilon)$ in \\eqref{defQf} that\n\\[\n\\bar k \\leq  \\displaystyle \\left \\lceil \n2 \\log \\frac{\\lambda_0 Q({\\bar \\varepsilon})}{\\bar \\varepsilon} \\right \\rceil.\n\\]\nTherefore, the second term on the right-hand side of \\eqref{complprimal-2}\ngives an upper bound on the number of iterations with backtracking of $\\lambda$.\nWe now provide a bound on the number of remaining iterations to obtain a $\\bar \\varepsilon$-solution with U-CS.\nIn the $k$-th iteration of U-CS, the model $\\hat \\Gamma_k$ for $\\phi=f+h$ is simply $\\ell_f(\\cdot,\\hat x_{k-1})+h(\\cdot)$ where $\\ell_f$ is the linearization of $f$ as in \\eqref{linfdef}, hence $\\mu(\\hat \\Gamma_k) = \\mu_h \\le \\mu_\\phi$ and $\\nu=\\mu_h$ in \\eqref{eq:nu}.\nSince Proposition \\ref{lem:ACSCS} shows that U-CS is a special case of FSCO, Theorem~\\ref{thouter} immediately gives for U-CS the upper bound \\eqref{complprimal} with ${\\underline{\\lambda}}$ given by \\eqref{ineq:lamj2}\non the number of the remaining iterations\n(where $\\lambda$ is not halved)\nrequired to find a $\\bar \\varepsilon$-solution \nof \\eqref{eq:ProbIntro2}.\nUsing the inequality\n\\begin{equation}\\label{upperlbar}\n\\frac{1}{\\underline{\\lam}} \\stackrel{\\eqref{ineq:lamj2}}=\\max \\left \\lbrace \\frac{8 M_f^2 + 2 \\bar \\varepsilon L_f}{(1-\\chi)^2 \\bar \\varepsilon}, \\frac{1}{\\lambda_0} \\right\\rbrace\n\\leq \\frac{8 M_f^2 + 2 \\bar \\varepsilon L_f}{(1-\\chi)^2 \\bar \\varepsilon} + \\frac{1}{\\lambda_0}\n\\end{equation}\nand the definition of $ Q(\\bar \\varepsilon)$ in \\eqref{defQf},\nwe conclude that\n$1/\\underline{\\lam} \\le \nQ(\\bar \\varepsilon)/\\bar \\varepsilon$.\nThis observation and \\eqref{complprimal} with ${\\underline{\\lambda}}$ given by \\eqref{ineq:lamj2} thus imply that the first term in \\eqref{complprimal-2} is an upper bound on the number of the remaining iterations\n(where $\\lambda$ is not halved).\nThis completes the proof.\n\\QEDA\n","sha256_utf8":"c4b9344afe534b79e0d2f03d1cb0e077eebd59df963f0ba53eda50471fed9bc9"},{"start":145693,"end":146181,"start_line":2855,"end_line":2860,"raw_tex":"\\begin{lemma}\\label{lem:app}\nAssume that $\\xi>0, \\psi \\in \\bConv{n}$ and $Q \\in \\mathcal{S}_{++}^n$ are such that $\\psi-$ $(\\xi / 2)\\|\\cdot\\|_Q^2$ is convex and let $(y, v, \\eta) \\in \\R^n \\times \\R^n \\times \\R_{+}$be such that $v \\in \\partial_\\eta \\psi(y)$. Then, for any $\\tau>0$,\n    \\begin{equation}\\label{ineq:app}\n        \\psi(u) \\ge \\psi(y)+\\inner{v}{u-y}-\\left(1+\\tau^{-1}\\right) \\eta+\\frac{(1+\\tau)^{-1} \\xi}{2}\\|u-y\\|_Q^2 \\quad \\forall u \\in \\R^n .\n    \\end{equation}\n\\end{lemma}","sha256_utf8":"e7c408fcd9d77cd412eafe7aed75d2e93730069102dda01485f9fae3c78e283a"},{"start":146182,"end":146290,"start_line":2861,"end_line":2863,"raw_tex":"\\begin{proof}\n    The proof of this result can be found in Lemma A.1 in \\cite{melo2023proximal}.\n\\end{proof}","sha256_utf8":"c442a153f4435f5767d921c1fb66ff74451499ba65b2cc579c28b3ff1b0814f0"},{"start":146368,"end":147856,"start_line":2869,"end_line":2893,"raw_tex":"\\begin{lemma} \\label{lm:easyrecur1}\n\t\tAssume that sequences $\\{\\gamma_j\\}$, $\\{\\eta_j\\}$, and  $\\{\\alpha_j\\}$ satisfy for every $j\\ge 1$, $\\gamma_j \\ge {\\underline \\gamma}$ and \n\t\t\\beq \\label{eq:easyrecur1}\n\t\t\\gamma_j \\eta_j \\le \\alpha_{j-1} - (1+\\sigma)\\alpha_j + \\gamma_j \\delta\n\t\t\\eeq\n  for some $\\sigma \\ge 0$, $\\delta \\ge 0$ and ${\\underline \\gamma} > 0$. Then, the following statements hold:\n\t\\begin{itemize}\n            \\item[a)] for every $k\\ge 1$,\n            \\begin{equation}\\label{ineq:eta}\n            \\min_{1\\le j \\le k}  \\eta_j \\le \\frac{\\alpha_0 - (1+\\sigma)^k \\alpha_k}{\\sum_{j=1}^k (1+\\sigma)^{j-1} \\gamma_j} + \\delta;\n        \\end{equation}\n  %           \\item[b)] if the sequence $\\{\\eta_j\\}$ is nonnegative, then for every $k\\ge 1$,\n  %           \\begin{equation}\\label{ineq:alpha}\n  %           \\alpha_k \\le \\frac{\\alpha_0}{(1+\\sigma)^k}  +\n\t\t% \\frac{\\sum_{j=1}^k (1+\\sigma)^{j-1}\\gamma_j \\delta}{(1+\\sigma)^k};\n  %       \\end{equation}\n\t\t\\item[b)]  if the sequence $\\{\\alpha_j\\}$ is nonnegative, then $ \\min_{1\\le j \\le k} \\eta_j \\le 2\\delta $ for every $ k\\ge 1$ such that\n\t\t\\[\n\t\tk \\ge \\min \\left\\lbrace  \\frac{1+\\sigma}{\\sigma} \\log\\left( \\frac{\\sigma \\alpha_0}{{\\underline \\gamma}\\delta} + 1 \\right), \\frac{\\alpha_0}{{\\underline \\gamma}\\delta} \\right\\rbrace\n\t\t\\]\n\t\twith the convention that the first term is equal to the second term\n\t\twhen $\\sigma=0$. (Note  that the first term converges to the second term\n\t\tas $\\sigma \\downarrow 0$.)\n\t\\end{itemize}\n\\end{lemma}","sha256_utf8":"e2e8c7f251369fc476d8dec4050ec2511bd07e736224d75bf3b91f2f8ab6fb83"},{"start":147858,"end":149542,"start_line":2895,"end_line":2922,"raw_tex":"\\begin{proof}\n\ta) Multiplying \\eqref{eq:easyrecur1} by $ (1+\\sigma)^{j-1} $ and summing the resulting inequality from $ j=1 $ to $ k $, we have\n\t\\begin{align}\n\t    \\sum_{j=1}^k (1+\\sigma)^{j-1} \\gamma_j \\left[ \\min_{1\\le j \\le k}  \\eta_j \\right] &\\le \\sum_{j=1}^k (1+\\sigma)^{j-1} \\gamma_j \\eta_j\n     \\le  \\sum_{j=1}^k (1+\\sigma)^{j-1} \\left(  \\alpha_{j-1} - (1+\\sigma) \\alpha_j + \\gamma_j \\delta \\right) \\nn \\\\\n\t\t &=  \\alpha_0 - (1+\\sigma)^k \\alpha_k +\n\t\t\\sum_{j=1}^k (1+\\sigma)^{j-1}\\gamma_j \\delta. \\label{ineq:sum}\n\t\\end{align}\n        Inequality \\eqref{ineq:eta} follows immediately from the above inequality.\n\n        % b) This statement follows immediately from \\eqref{ineq:sum} and the fact that $\\eta_j\\ge 0$.\n        \n        b) It follows from \\eqref{ineq:eta}, and the facts that $\\alpha_k\\ge 0$ and $\\gamma_j \\ge {\\underline \\gamma}$ that\n        \\begin{equation}\\label{ineq:eta1}\n            \\min_{1\\le j \\le k}  \\eta_j  \\le \\frac{\\alpha_0}{{\\underline \\gamma} \\sum_{j=1}^k (1+\\sigma)^{j-1}} + \\delta.\n        \\end{equation}\n\tUsing the fact that $ 1+\\sigma \\ge e^{\\sigma/(1+\\sigma)} $\n\tfor every $\\sigma \\ge 0$, \n\twe have\n\t\\begin{equation}\\label{ineq:theta}\n\t    \\sum_{j=1}^{k} (1+\\sigma)^{j-1} = \\max \\left\\lbrace \\frac{(1+\\sigma)^k-1}{\\sigma}, k \\right\\rbrace \\ge \\max\\left\\lbrace \\frac{e^{\\sigma k /(1+\\sigma)} - 1}{\\sigma}, k\\right\\rbrace.\n\t\\end{equation}\n\tPlugging the above inequality into \\eqref{ineq:eta1}, we have for every $ k\\ge 1 $,\n\t\\[\n\t\\min_{1\\le j \\le k}  \\eta_j \\le \\frac{\\alpha_0}{{\\underline \\gamma}}\\min \\left\\lbrace \\frac{\\sigma}{e^{\\sigma k /(1+\\sigma)} - 1}, \\frac 1k \\right\\rbrace + \\delta,\n\t\\]\n\twhich can be easily seen to imply (b).\n\\end{proof}","sha256_utf8":"ec1489e5da204ebd0e27cb56bc2a8edd718e55e451bd79bf78a28f4a86f5d99b"},{"start":149627,"end":150263,"start_line":2926,"end_line":2937,"raw_tex":"\\begin{lemma}\\label{lem:sigma}\n  For every $\\chi \\in [0,1)$, $\\lam>0$, $\\mu>0$, and $\\nu \\in [0,\\mu]$, the \n  quantity $\\sigma=\\sigma(\\mu,\\nu;\\lambda)$ defined in \\eqref{eq:sigma} satisfies the following statements:\n  \\begin{itemize}\n      \\item[a)] $\\lam [ \\chi  \\mu + (1-\\chi)  \\nu] \\le \\sigma \\le \\lam \\mu$, and hence $\\sigma \\ge \\lambda \\nu$;\n      \\item[b)] the function $\\lam >0 \\mapsto \\sigma(\\mu,\\nu;\\lambda)$ is non-decreasing;\n      \\item[c)] there holds\n      \\[\n      \\frac{1+\\sigma}{\\sigma} \\le \\min\\left\\{ \\frac{1}{\\chi}\\left(1+\\frac{1}{\\lam \\mu}\\right),  1+\\frac{1}{\\lam \\nu}\\right\\}.\n      \\]\n  \\end{itemize}\n\\end{lemma}","sha256_utf8":"cbdc948ad431091379887072c434f2105e148e1107ab1b90c8d4354dfc85d851"},{"start":150265,"end":152072,"start_line":2939,"end_line":2975,"raw_tex":"\\begin{proof}\n    a) It follows from \\eqref{eq:sigma} and the fact that $\\nu \\le \\mu$ that\n \\[\n    \\sigma - \\lam \\mu =\\frac{\\lam(\\mu-\\nu)(1+\\lam \\mu)(\\chi-1)}{1+\\mu  \\lambda+\\chi  \\lambda(\\nu-\\mu)}\n    \\ge \\frac{\\lam(\\mu-\\nu)(1+\\lam \\mu)(\\chi-1)}{1+\\mu  \\lambda} =\n\\lam(\\mu-\\nu)(\\chi-1),\n    \\]\n    and hence that $\\sigma \\ge \\lam [ \\chi  \\mu + (1-\\chi)  \\nu]$. Moreover, the second inequality of (a) immediately follows from the first identity above and the facts that $\\chi<1$ and $\\nu \\le \\mu$.\n\n    b) Viewing $\\sigma=\\sigma(\\lam)$ in \\eqref{eq:sigma} as a function of $\\lam$, one can verify that\n\\[\n\\sigma(\\lam) = \\frac{\\nu(1+\\mu \\lam)+\\chi(\\mu-\\nu)}{\\lam^{-1}+\\mu +\\chi (\\nu-\\mu)}\n\\]\nis a non-decreasing function of $\\lam$.\n\n    c) It follows from the definition of $\\sigma$ in \\eqref{eq:sigma} that \n    \\[\n    \\frac{1+\\sigma}{\\sigma} = \\frac{(1+\\nu {\\lam})(1+\\mu {\\lam})}{{\\lam}[\\nu(1+ \\mu {\\lam})+\\chi(\\mu-\\nu)]}.\n    \\]\n    Hence, it suffices to prove that\n\\begin{equation}\\label{ineq:combine}\n    \\frac{(1+\\nu {\\lam})(1+\\mu {\\lam})}{{\\lam}[\\nu(1+ \\mu {\\lam})+\\chi(\\mu-\\nu)]} \\le \\min\\left\\{ \\frac{1}{\\chi}\\left(1+\\frac{1}{{\\lam} \\mu}\\right),  1+\\frac{1}{{\\lam} \\nu}\\right\\}. \n\\end{equation}\nSince $\\chi \\in [0,1)$, it is easy to verify that\n\\[\n\\frac{1+\\nu {\\lam}}{\\nu(1+ \\mu {\\lam})+\\chi(\\mu-\\nu)} \\le \\frac{1}{\\chi \\mu},\n\\]\nand hence that\n\\[\n\\frac{(1+\\nu {\\lam})(1+\\mu {\\lam})}{{\\lam}[\\nu(1+ \\mu {\\lam})+\\chi(\\mu-\\nu)]} \\le \\frac{1+ \\mu {\\lam}}{\\chi {\\lam} \\mu} = \\frac{1}{\\chi}\\left(1+\\frac{1}{{\\lam} \\mu}\\right).\n\\]\nMoreover, noting that $\\chi \\in [0,1)$ and $\\mu\\ge \\nu$, we also have \n\\[\n\\frac{(1+\\nu {\\lam})(1+\\mu {\\lam})}{{\\lam}[\\nu(1+ \\mu {\\lam})+\\chi(\\mu-\\nu)]} \\le 1+\\frac{1}{{\\lam} \\nu}.\n\\]\nCombining the above two inequalities, we conclude that \\eqref{ineq:combine} holds.\n\\end{proof}","sha256_utf8":"04615823d0a6e6217c855bf17a8d3fe06d5a42e1519394e86a1a93b713266914"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"U-CS通用迭代复杂度","source_units":[{"start":50372,"end":51233,"start_line":999,"end_line":1014,"raw_tex":"\\begin{theorem}\\label{thoutercs}\n% Let $\\chi \\in [0,1)$ and tolerance $\\bar \\varepsilon>0$ be given, and\n% consider U-CS with $\\varepsilon = (1-\\chi)\\bar \\varepsilon/2$.\nU-CS terminates in at most\n\\begin{equation}\\label{complprimal-2}\n\\min\\left\\{\\min\\left[ \\frac{1}{\\chi}\\left(1+\\frac{Q({\\bar \\varepsilon})}{\\bar \\varepsilon \\mu_{\\phi}}\\right),  1+\\frac{Q({\\bar \\varepsilon})}{\\bar \\varepsilon \\mu_h}\\right]\n\\log \\left(1+\\frac{\\mu_\\phi d_0^2}{\\bar \\varepsilon}\\right)\\, , \\, \\frac{d_0^2 Q({\\bar \\varepsilon})}{{\\bar \\varepsilon}^2} \\right\\}+\n\\displaystyle \\left \\lceil \n2 \\log \\frac{\\lambda_0 Q({\\bar \\varepsilon})}{\\bar \\varepsilon} \\right \\rceil\n\\end{equation}\niterations and\n\\begin{equation}\\label{defQf}\nQ({\\bar \\varepsilon})= \\frac{8 M_f^2}{(1-\\chi)^2}+ {\\bar \\varepsilon} \\left( \\lam_0^{-1} + \n\\frac{2L_f}{(1-\\chi)^2} \\right).\n\\end{equation}\n\\end{theorem}","sha256_utf8":"840bf6b2b0075a06971bf16cdafcc8abcd63626cf61ab407b4eb30325bc73a0f"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"U-CS复杂度完整证明","source_units":[{"start":121823,"end":124224,"start_line":2417,"end_line":2450,"raw_tex":"{\\bf Proof of Theorem \\ref{thoutercs}.}\nDefine \n\\begin{equation}\\label{defkbar}\n\\bar k=\\left \\lceil 2 \\log \\max\\left\\{\\frac{ \\lambda_0 (8M_f^2 + 2 \\bar \\varepsilon L_f) }{(1-\\chi)^2 \\bar \\varepsilon},1\\right\\} \\right \\rceil.\n\\end{equation}\nObserve that $\\lambda_0/2^{k} \\leq {\\underline \\lambda}$ for $k \\geq \\bar k$ where ${\\underline{\\lambda}}$ is given by the right-hand side of \\eqref{ineq:lamj2},\nand from Proposition \\ref{lem:ACSCS} that we cannot halve $\\lambda$ more than $\\bar k$ iterations. \nIt follows from \\eqref{defkbar} and the definition of $ Q(\\bar \\varepsilon)$ in \\eqref{defQf} that\n\\[\n\\bar k \\leq  \\displaystyle \\left \\lceil \n2 \\log \\frac{\\lambda_0 Q({\\bar \\varepsilon})}{\\bar \\varepsilon} \\right \\rceil.\n\\]\nTherefore, the second term on the right-hand side of \\eqref{complprimal-2}\ngives an upper bound on the number of iterations with backtracking of $\\lambda$.\nWe now provide a bound on the number of remaining iterations to obtain a $\\bar \\varepsilon$-solution with U-CS.\nIn the $k$-th iteration of U-CS, the model $\\hat \\Gamma_k$ for $\\phi=f+h$ is simply $\\ell_f(\\cdot,\\hat x_{k-1})+h(\\cdot)$ where $\\ell_f$ is the linearization of $f$ as in \\eqref{linfdef}, hence $\\mu(\\hat \\Gamma_k) = \\mu_h \\le \\mu_\\phi$ and $\\nu=\\mu_h$ in \\eqref{eq:nu}.\nSince Proposition \\ref{lem:ACSCS} shows that U-CS is a special case of FSCO, Theorem~\\ref{thouter} immediately gives for U-CS the upper bound \\eqref{complprimal} with ${\\underline{\\lambda}}$ given by \\eqref{ineq:lamj2}\non the number of the remaining iterations\n(where $\\lambda$ is not halved)\nrequired to find a $\\bar \\varepsilon$-solution \nof \\eqref{eq:ProbIntro2}.\nUsing the inequality\n\\begin{equation}\\label{upperlbar}\n\\frac{1}{\\underline{\\lam}} \\stackrel{\\eqref{ineq:lamj2}}=\\max \\left \\lbrace \\frac{8 M_f^2 + 2 \\bar \\varepsilon L_f}{(1-\\chi)^2 \\bar \\varepsilon}, \\frac{1}{\\lambda_0} \\right\\rbrace\n\\leq \\frac{8 M_f^2 + 2 \\bar \\varepsilon L_f}{(1-\\chi)^2 \\bar \\varepsilon} + \\frac{1}{\\lambda_0}\n\\end{equation}\nand the definition of $ Q(\\bar \\varepsilon)$ in \\eqref{defQf},\nwe conclude that\n$1/\\underline{\\lam} \\le \nQ(\\bar \\varepsilon)/\\bar \\varepsilon$.\nThis observation and \\eqref{complprimal} with ${\\underline{\\lambda}}$ given by \\eqref{ineq:lamj2} thus imply that the first term in \\eqref{complprimal-2} is an upper bound on the number of the remaining iterations\n(where $\\lambda$ is not halved).\nThis completes the proof.\n\\QEDA\n","sha256_utf8":"c4b9344afe534b79e0d2f03d1cb0e077eebd59df963f0ba53eda50471fed9bc9"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"控制步长回溯次数","source_units":[{"start":121823,"end":122706,"start_line":2417,"end_line":2430,"raw_tex":"{\\bf Proof of Theorem \\ref{thoutercs}.}\nDefine \n\\begin{equation}\\label{defkbar}\n\\bar k=\\left \\lceil 2 \\log \\max\\left\\{\\frac{ \\lambda_0 (8M_f^2 + 2 \\bar \\varepsilon L_f) }{(1-\\chi)^2 \\bar \\varepsilon},1\\right\\} \\right \\rceil.\n\\end{equation}\nObserve that $\\lambda_0/2^{k} \\leq {\\underline \\lambda}$ for $k \\geq \\bar k$ where ${\\underline{\\lambda}}$ is given by the right-hand side of \\eqref{ineq:lamj2},\nand from Proposition \\ref{lem:ACSCS} that we cannot halve $\\lambda$ more than $\\bar k$ iterations. \nIt follows from \\eqref{defkbar} and the definition of $ Q(\\bar \\varepsilon)$ in \\eqref{defQf} that\n\\[\n\\bar k \\leq  \\displaystyle \\left \\lceil \n2 \\log \\frac{\\lambda_0 Q({\\bar \\varepsilon})}{\\bar \\varepsilon} \\right \\rceil.\n\\]\nTherefore, the second term on the right-hand side of \\eqref{complprimal-2}\ngives an upper bound on the number of iterations with backtracking of $\\lambda$.\n","sha256_utf8":"14bbcbd4e418b8b56a4fa82f9fbe02c61ac0f2f20da68f3e903bc71001598768"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"应用框架复杂度","source_units":[{"start":122706,"end":123455,"start_line":2431,"end_line":2437,"raw_tex":"We now provide a bound on the number of remaining iterations to obtain a $\\bar \\varepsilon$-solution with U-CS.\nIn the $k$-th iteration of U-CS, the model $\\hat \\Gamma_k$ for $\\phi=f+h$ is simply $\\ell_f(\\cdot,\\hat x_{k-1})+h(\\cdot)$ where $\\ell_f$ is the linearization of $f$ as in \\eqref{linfdef}, hence $\\mu(\\hat \\Gamma_k) = \\mu_h \\le \\mu_\\phi$ and $\\nu=\\mu_h$ in \\eqref{eq:nu}.\nSince Proposition \\ref{lem:ACSCS} shows that U-CS is a special case of FSCO, Theorem~\\ref{thouter} immediately gives for U-CS the upper bound \\eqref{complprimal} with ${\\underline{\\lambda}}$ given by \\eqref{ineq:lamj2}\non the number of the remaining iterations\n(where $\\lambda$ is not halved)\nrequired to find a $\\bar \\varepsilon$-solution \nof \\eqref{eq:ProbIntro2}.\n","sha256_utf8":"6fa5b48e9b6b21f840b13985651ca61f90e1f768557d6158ec40de3159dddc4b"}]},{"id":"S005","semantic_type":"proof_step","kind":"proof_step","title":"代入步长倒数上界","source_units":[{"start":123455,"end":124224,"start_line":2438,"end_line":2450,"raw_tex":"Using the inequality\n\\begin{equation}\\label{upperlbar}\n\\frac{1}{\\underline{\\lam}} \\stackrel{\\eqref{ineq:lamj2}}=\\max \\left \\lbrace \\frac{8 M_f^2 + 2 \\bar \\varepsilon L_f}{(1-\\chi)^2 \\bar \\varepsilon}, \\frac{1}{\\lambda_0} \\right\\rbrace\n\\leq \\frac{8 M_f^2 + 2 \\bar \\varepsilon L_f}{(1-\\chi)^2 \\bar \\varepsilon} + \\frac{1}{\\lambda_0}\n\\end{equation}\nand the definition of $ Q(\\bar \\varepsilon)$ in \\eqref{defQf},\nwe conclude that\n$1/\\underline{\\lam} \\le \nQ(\\bar \\varepsilon)/\\bar \\varepsilon$.\nThis observation and \\eqref{complprimal} with ${\\underline{\\lambda}}$ given by \\eqref{ineq:lamj2} thus imply that the first term in \\eqref{complprimal-2} is an upper bound on the number of the remaining iterations\n(where $\\lambda$ is not halved).\nThis completes the proof.\n\\QEDA\n","sha256_utf8":"56a9eb49ae13f749cb92b321e335e74a8c57cd0d1a3c9ab348442f5b5b8f4370"}]},{"id":"S006","semantic_type":"problem_setting","kind":"problem_setting","title":"混合复合优化问题","source_units":[{"start":11637,"end":11753,"start_line":301,"end_line":303,"raw_tex":"\\begin{equation}\\label{eq:ProbIntro2}\n\t\\phi_{*}:=\\min \\left\\{\\phi(x):=f(x)+h(x): x \\in \\R^n\\right\\},\n\t\\end{equation}","sha256_utf8":"df85d33c54c601ab5543164609373b6594e620d4d6909c4bdbf8f534ae904e90"}]},{"id":"S007","semantic_type":"notation","kind":"notation","title":"欧氏空间与对数记号","source_units":[{"start":33456,"end":33813,"start_line":692,"end_line":695,"raw_tex":"    Let $\\R$ denote the set of real numbers.\n    Let $ \\R_+ $ (resp., $ \\R_{++} $) denote the set of non-negative real numbers (resp., the set of positive real numbers).\n\tLet $\\R^n$ denote the standard $n$-dimensional Euclidean space equipped with  inner product and norm denoted by $\\left\\langle \\cdot,\\cdot\\right\\rangle $\n\tand $\\|\\cdot\\|$, respectively. \n","sha256_utf8":"9895973caf4aa6364c0f39ebe722efa3ade91534302ac28d92d2203edcf9cf21"},{"start":33943,"end":33992,"start_line":698,"end_line":698,"raw_tex":"\tLet $\\log(\\cdot)$ denote the natural logarithm.\n","sha256_utf8":"fa3cdfd3306b1c99001f7f41776a259ab1ae9ac07ef028472c80b0ea5c6d7983"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"函数域与强凸函数类","source_units":[{"start":33994,"end":34611,"start_line":700,"end_line":708,"raw_tex":"\tFor given $\\Phi: \\R^n\\rightarrow (-\\infty,+\\infty]$, let $\\dom \\Phi:=\\{x \\in \\R^n: \\Phi (x) <\\infty\\}$ denote the effective domain of $\\Phi$ \n\tand $\\Phi$ is proper if $\\dom \\Phi \\ne \\emptyset$.\n\tA proper function $\\Phi: \\R^n\\rightarrow (-\\infty,+\\infty]$ is $\\mu$-convex for some $\\mu \\ge 0$ if\n\t\\[\n\t\\Phi(\\alpha x+(1-\\alpha) y)\\leq \\alpha \\Phi(x)+(1-\\alpha)\\Phi(y) - \\frac{\\alpha(1-\\alpha) \\mu}{2}\\|x-y\\|^2\n\t\\]\n\tfor every $x, y \\in \\dom \\Phi$ and $\\alpha \\in [0,1]$.\nLet $\\mConv{n}$ denote the set of all proper lower semicontinuous $\\mu$-convex functions.\n We simply denote $\\mConv{n}$ by $\\bConv{n}$ when $\\mu=0$.\n","sha256_utf8":"82c85d6a2df326c50d835a0f4f36365466865f57aa46dfb8dd52410038cf63d7"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"近似次微分定义","source_units":[{"start":34611,"end":35030,"start_line":709,"end_line":713,"raw_tex":"\tFor $\\varepsilon \\ge 0$, the \\emph{$\\varepsilon$-subdifferential} of $ \\Phi $ at $x \\in \\dom \\Phi$ is denoted by\n\t\\[\n        \\partial_\\varepsilon \\Phi (x):=\\left\\{ s \\in\\R^n: \\Phi(y)\\geq \\Phi(x)+\\left\\langle s,y-x\\right\\rangle -\\varepsilon, \\forall y\\in\\R^n\\right\\}.\n        \\]\n\tWe denote the subdifferential of $\\Phi$ at $x \\in \\dom \\Phi$ by $\\partial \\Phi (x)$, which is the set  $\\partial_0 \\Phi(x)$ by definition.\n","sha256_utf8":"52bd855790af6bf809874b7822f2b82e8173234ce4a91ac996c350aea3b1891d"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"凸函数线性化","source_units":[{"start":35030,"end":35303,"start_line":714,"end_line":718,"raw_tex":"    For a given subgradient\n$\\Phi'(x) \\in \\partial \\Phi(x)$, we denote the linearization of convex function $\\Phi$ at $x$ by $\\ell_\\Phi(\\cdot,x)$, which is defined as\n\\begin{equation}\\label{linfdef}\n\\ell_\\Phi(\\cdot,x)=\\Phi(x)+\\langle \\Phi'(x),\\cdot-x\\rangle.\n\\end{equation}","sha256_utf8":"4a21d7f84de21c56b6ce9e6b64474b9d1a8d327a329d29c1b2eba63b17738011"}]},{"id":"S011","semantic_type":"assumption","kind":"assumption","title":"复合问题基本假设","source_units":[{"start":37370,"end":37402,"start_line":755,"end_line":756,"raw_tex":"We assume that \n\\begin{itemize}\n","sha256_utf8":"fd92f1913ca2bbafe6cdd90ce63d9ff4af43897165b0809e7ee996f249cdd0a0"},{"start":37723,"end":38197,"start_line":761,"end_line":774,"raw_tex":"\\item[(A1)] $f, h \\in \\bConv{n}$ are such that\n\t\t$\\dom h \\subset \\dom f$, and a subgradient oracle,\n\t\t i.e.,\n\t\ta function $f':\\dom h \\to \\R^n$\n\t\tsatisfying $f'(x) \\in \\partial f(x)$ for every $x \\in \\dom h$, is available;\n\t\t\\item[(A2)]\n\t\tthere exists $(M_f,L_f)\\in \\R^2_+$ such that for every $x,y \\in \\dom h$,\n\t\t\\[\n\t\t\\|f'(x)-f'(y)\\| \\le 2M_f + L_f \\|x-y\\|;\n\t\t\\]\n\\item[(A3)]\n\t\tthe set of optimal solutions $X_*$ of\n\t\tproblem \\eqref{eq:ProbIntro2} is nonempty.\n\\end{itemize}\n","sha256_utf8":"d42001ebc0d84d34c51002916be3387cbe2a2474990bd73043683dc46eb3bed0"}]},{"id":"S012","semantic_type":"derived_fact","kind":"derived_fact","title":"混合模型误差估计","source_units":[{"start":38198,"end":38380,"start_line":776,"end_line":779,"raw_tex":"It is known that that (A2) implies that for every $x,y \\in \\dom h$, \n\\begin{equation}\\label{ineq:est} \t\nf(x)-\\ell_f(x;y) \\le 2M_f \\|x-y\\| + \\frac{L_f}{2} \\| x-y \\|^2.\n\t\\end{equation}","sha256_utf8":"1e059872c59c0c1968932dbb8a3880c06c51314eb28661536738eddb7f70fe69"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"内在强凸参数","source_units":[{"start":12206,"end":12370,"start_line":310,"end_line":312,"raw_tex":"The intrinsic convexity parameter\n$\\mu_\\psi$\nof a convex function $\\psi$ is defined as the largest scalar $\\mu$ such that $\\psi(\\cdot)-\\mu\\|\\cdot\\|^2/2$ is convex.\n","sha256_utf8":"b80e409c12c20d31edbaef5025c7957ea356466008484734836e54206ddc98a4"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"精度与初始距离","source_units":[{"start":39239,"end":39803,"start_line":794,"end_line":801,"raw_tex":"Our goal in this section is to describe two completely universal methods for solving \\eqref{eq:ProbIntro2}  and describe  their iteration-complexity guarantees for finding \na $\\bar \\varepsilon$-solution $\\bar x$ of \\eqref{eq:ProbIntro2}, i.e., \na point $\\bar  x$ such that $\\phi(\\bar x) - \\phi_* \\le \\bar \\varepsilon$, in terms of\n$(M_f,L_f,\\mu_\\phi )$, $\\bar \\varepsilon$, and $d_0$, where $d_0$\nis the distance of the initial iterate $\\hat x_0$ with respect to $X_*$, i.e.,\n\\begin{equation}\\label{defd0}\nd_0 := \\min \\{\\|x-\\hat x_0\\| : x \\in X_*\\}.\n\\end{equation}","sha256_utf8":"97b83a2ec4596f333bf3aa343edff7c685fb9e0c84bbd5b885d39170c1315464"}]},{"id":"S015","semantic_type":"algorithm","kind":"algorithm","title":"U-CS算法","source_units":[{"start":45621,"end":46539,"start_line":915,"end_line":931,"raw_tex":"\tU-CS($\\hat x_0,\\chi,\\lam_0,\\bar \\varepsilon$)\n\t\n\t\\noindent\\rule[0.5ex]{1\\columnwidth}{1pt}\n\t\n    {\\bf Input:} $(\\hat x_0,\\chi,\\lam_0,\\bar \\varepsilon)\\in \\dom h \\times [0,1) \\times \\R_{++} \\times \\R_{++}$.\n        \\begin{itemize}\n\t\t\\item [0.]  Set $\\lam=\\lam_0$ and $k=1$;\n\t    \\item[1.] Compute \n\t    \\begin{equation}\\label{def:x}\n\t        x =\\underset{u\\in  \\R^n}\\argmin\n\t\t\\left\\lbrace  \\ell_f(u;\\hat x_{k-1})+h(u) +\\frac{1}{2\\lam} \\|u- \\hat x_{k-1} \\|^2 \\right\\rbrace;\n\t    \\end{equation}\n        if $\\phi(x) - \\phi_* \\le \\bar \\varepsilon$, then {\\bf stop};\n\t\t\\item[2.]  {\\bf If} the inequality $$f(x)-\\ell_f(x;\\hat x_{k-1}) - \\frac{1-\\chi}{2\\lam} \\|x-\\hat x_{k-1}\\|^2 \\le \\frac{(1-\\chi) \\bar \\varepsilon}2$$ does not hold, {\\bf then} set $\\lam = \\lam/2$ and go to step~1; {\\bf else}, \n        set $\\lam_k=\\lam$, $\\hat x_k=x$,  $k\\leftarrow k+1$, and go to step 1.\n\t\\end{itemize}\n\t\\rule[0.5ex]{1\\columnwidth}{1pt}\n","sha256_utf8":"74df4e30b23473165213342c87732bcb259b1fb9afea66b3a55f20d6b0fa4d61"}]},{"id":"S016","semantic_type":"problem_setting","kind":"problem_setting","title":"一般凸优化问题","source_units":[{"start":83232,"end":83335,"start_line":1690,"end_line":1692,"raw_tex":"\\begin{equation}\\label{eq:ProbIntro}\n\t\\phi_{*}:=\\min \\left\\{\\phi(x): x \\in \\R^n\\right\\}\n\t\\end{equation}","sha256_utf8":"5cc03961598fcbe5d0710b288bcdd567f5151b6f7ba306a7556742c5ed5abf90"},{"start":83336,"end":83364,"start_line":1693,"end_line":1693,"raw_tex":"where $\\phi \\in \\bConv{n}$.\n","sha256_utf8":"37c570c6d20764a7fd3a69bf33559744d4950a92b69e58f410834819e0bfc6fa"}]},{"id":"S017","semantic_type":"algorithm","kind":"algorithm","title":"近端黑盒条件","source_units":[{"start":85644,"end":86323,"start_line":1737,"end_line":1754,"raw_tex":"\t Black-box (BB)\n     $(x^-,\\chi,\\varepsilon)$\n\t\n\t\\noindent\\rule[0.5ex]{1\\columnwidth}{1pt}\n    {\\bf Input:}  $(x^-,\\chi,\\varepsilon) \\in \\dom \\phi \\times [0,1) \\times \\R_{++}$.\n    \n    \\vgap\n    \n    \\noindent\n    {\\bf Output:}\n    $(x,y,\\Gamma,\\lam) \\in \\dom \\phi \\times \\dom \\phi \\times \\bConv{n} \\times \\R_{++}$ satisfying $\\Gamma \\le \\phi$,\n\\begin{equation}\\label{propyb}\n    \\phi(y)+\\frac{\\chi}{2 \\lambda}\\|y-x^-\\|^2 - \\underset{u\\in\\R^n}\\min\\left\\{\\Gamma(u) + \\frac{1}{2\\lam}\\|u- x^- \\|^2\\right\\} \\le \\varepsilon,\n\\end{equation}\nand\n\\beq \n    x = \\underset{u\\in\\R^n} \\argmin \\left\\lbrace \\Gamma(u) + \\frac{1}{2\\lam}\\|u- x^- \\|^2 \\right\\rbrace. \\label{defxhatk}\n     \\eeq\n","sha256_utf8":"512a5d7b5e881e79beabf8442610befed8e484b875d9e8818d6b27b3ef80fec3"}]},{"id":"S018","semantic_type":"algorithm","kind":"algorithm","title":"FSCO框架","source_units":[{"start":95560,"end":95879,"start_line":1920,"end_line":1925,"raw_tex":"\t FSCO$(\\hat x_0, \\chi, \\bar \\varepsilon)$\n\t\n\t\\noindent\\rule[0.5ex]{1\\columnwidth}{1pt}\n    % {\\bf Input:} $(\\hat x_0, \\chi, \\bar \\varepsilon) \\in \\dom \\phi \\times [0,1) \\times \\R_+ $\n\t\\begin{itemize}\n\t\t\\item [0.] Let $(\\hat x_0, \\chi, \\bar \\varepsilon) \\in \\dom \\phi \\times [0,1) \\times \\R_+ $ be given and set $k=1$;\n","sha256_utf8":"51d749be865463d26c5289911f563c0e0b2ef3ad8ba134ff0530f1b0b19eaaf1"},{"start":96450,"end":96831,"start_line":1936,"end_line":1944,"raw_tex":"    \\item[1.] Compute \n$(\\hat x_k, \\hat y_k, \\hat \\Gamma_k, \\lambda_k) = \\BB(\\hat x_{k-1},\\chi, (1-\\chi) \\bar \\varepsilon/2)$;\n    \n\t\t\\item[2.]\n  Check whether $\\phi(\\hat y_k) - \\phi_* \\le \\bar \\varepsilon$ and if so {\\bf stop}; else set $k\\leftarrow k+1$ and go to step 1.\n  % \\item[3.]\n\t\t% Set $k\\leftarrow k+1$ and go to step 1.\n\t\\end{itemize}\n\t\\rule[0.5ex]{1\\columnwidth}{1pt}\n","sha256_utf8":"66c329c796219e040cab5d92a80c3215bd04f4f70bf95dd38ee18425a35d3521"}]},{"id":"S019","semantic_type":"assumption","kind":"assumption","title":"步长正下界条件","source_units":[{"start":100065,"end":100504,"start_line":2010,"end_line":2017,"raw_tex":" The main complexity result for FSCO is relative to the  instances that satisfy the following condition:\n % In addition to $\\phi \\in \\bConv{n}$, this result requires the following mild assumption: \n\\begin{itemize}\n% \\item[(F1)] there exists $\\nu \\in [0,\\mu(\\phi)]$ such that\n% $\\hat \\Gamma_k \\in \\nConv{n}$;\n\\item[(F1)] there exists $\\underline \\lambda>0$ such that\n$\\lambda_k \\geq {\\underline \\lambda}$ for every $k \\ge 0$.\n\\end{itemize}\n","sha256_utf8":"3f702c36bda57670d3d0889904682f2b4451cf24e1707061d179432a9273a3d6"}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","title":"小步长保证接受","source_units":[{"start":48492,"end":48942,"start_line":963,"end_line":969,"raw_tex":"\\begin{lemma}\\label{lem:lam}\nIf $\\lam$ satisfies \n\\begin{equation}\\label{ineq:lam}\n    \\lam \\le \\frac{(1-\\chi)^2 \\bar \\varepsilon}{ 4{M}_f^2 + \\bar \\varepsilon {L}_f},\n\\end{equation}\nthen the point $x=x_k(\\lam)$ computed in \\eqref{def:x} satisfies the inequality in step 2. As a consequence, if the initial prox stepsize $\\lam_0$ is chosen to be the right hand side of \\eqref{ineq:lam}, then $\\lam$ remains constant throughout the method.\n\\end{lemma}","sha256_utf8":"ccc5cb70861ec652538e57a5ad53f14b75baace79b3d715a85950e188b320d58"}]},{"id":"S021","semantic_type":"proof","kind":"proof","title":"步长接受引理证明","source_units":[{"start":48945,"end":49808,"start_line":972,"end_line":985,"raw_tex":"\\begin{proof}\n    Assume that $\\lam$ satisfies \\eqref{ineq:lam}. Using \\eqref{ineq:est} with $(M_f, L_f, u,v)=(M_f, L_f, x,\\hat x_{k-1})$ and the inequality \n    $2M_f \\Delta - a \\Delta^2 \\leq M_f^2/a$ with\n    $\\Delta=\\|x-\\hat x_{k-1}\\|$ and $a=(1-\\chi-\\lam L_f)/2\\lam$, we have\n    \\begin{align*}\n    f(x)-\\ell_f(x;\\hat x_{k-1}) +\\frac{\\chi-1}{2 \\lambda}  \\|x-\\hat x_{k-1}\\|^2\n    &\\stackrel{\\eqref{ineq:est}}\\le 2M_f \\|x-\\hat x_{k-1}\\| + \\frac{L_f}2 \\|x-\\hat x_{k-1}\\|^2+\\frac{\\chi-1}{2 \\lambda  }\\|x-\\hat x_{k-1}\\|^2 \\\\\n    &= 2M_f \\|x-\\hat x_{k-1}\\| - \\frac{1-\\chi-\\lam L_f}{2\\lam} \\|x-\\hat x_{k-1}\\|^2 \\\\\n    &\\le \\frac{2\\lam M_f^2}{1-\\chi-\\lam L_f} \\stackrel{\\eqref{ineq:lam}}\\le \\frac{(1-\\chi)\\bar \\varepsilon}2, %\\label{ineq:error}\n    \\end{align*}\n    where the last inequality is due to\n    \\eqref{ineq:lam}.\n    Hence, the lemma is proved.\n\\end{proof}","sha256_utf8":"bc772fb8e637723eec011e30049d47061c81ef49276c92216a40f42e6076355b"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"配方控制线性化误差","source_units":[{"start":48959,"end":49797,"start_line":973,"end_line":984,"raw_tex":"    Assume that $\\lam$ satisfies \\eqref{ineq:lam}. Using \\eqref{ineq:est} with $(M_f, L_f, u,v)=(M_f, L_f, x,\\hat x_{k-1})$ and the inequality \n    $2M_f \\Delta - a \\Delta^2 \\leq M_f^2/a$ with\n    $\\Delta=\\|x-\\hat x_{k-1}\\|$ and $a=(1-\\chi-\\lam L_f)/2\\lam$, we have\n    \\begin{align*}\n    f(x)-\\ell_f(x;\\hat x_{k-1}) +\\frac{\\chi-1}{2 \\lambda}  \\|x-\\hat x_{k-1}\\|^2\n    &\\stackrel{\\eqref{ineq:est}}\\le 2M_f \\|x-\\hat x_{k-1}\\| + \\frac{L_f}2 \\|x-\\hat x_{k-1}\\|^2+\\frac{\\chi-1}{2 \\lambda  }\\|x-\\hat x_{k-1}\\|^2 \\\\\n    &= 2M_f \\|x-\\hat x_{k-1}\\| - \\frac{1-\\chi-\\lam L_f}{2\\lam} \\|x-\\hat x_{k-1}\\|^2 \\\\\n    &\\le \\frac{2\\lam M_f^2}{1-\\chi-\\lam L_f} \\stackrel{\\eqref{ineq:lam}}\\le \\frac{(1-\\chi)\\bar \\varepsilon}2, %\\label{ineq:error}\n    \\end{align*}\n    where the last inequality is due to\n    \\eqref{ineq:lam}.\n    Hence, the lemma is proved.\n","sha256_utf8":"81ed63268e66b01da6ff8984c877a77f81289513b2e7ff1ae5b9ea28aabd0073"}]},{"id":"S023","semantic_type":"proposition","kind":"proposition","title":"U-CS嵌入FSCO","source_units":[{"start":119746,"end":120849,"start_line":2369,"end_line":2393,"raw_tex":"\\begin{proposition}\\label{lem:ACSCS}\nThe following statements hold for U-CS:\n    \\begin{itemize}\n    \\item[a)] $\\{\\lam_k\\}$ is a non-increasing sequence;\n        \\item[b)] for every $k\\ge 1$, we have\n        \\begin{align}\n\t\t&\\hat x_k =\\underset{u\\in  \\R^n}\\argmin\n\t\t\\left\\lbrace  \\ell_f(u;\\hat x_{k-1})+h(u) +\\frac{1}{2\\lam_{k}} \\|u- \\hat x_{k-1} \\|^2 \\right\\rbrace, \\label{eq:sub1cs} \\\\\n\t\t&f(\\hat x_{k}) -\\ell_f(\\hat x_{k};\\hat x_{k-1})+\\frac{\\chi-1}{2 \\lambda_{k}}\\|\\hat x_{k}-\\hat x_{k-1}\\|^2 \\le \\frac{(1-\\chi)\\bar \\varepsilon}2,\n  \\label{ineq:iterationcs}\\\\\n   & \\lam_k \\ge \\min \\left \\lbrace \\frac{(1-\\chi)^2 \\bar \\varepsilon}{8 M_f^2 + 2 \\bar \\varepsilon L_f}, \\lam_0 \\right\\rbrace.\\label{ineq:lamj2}\n\t    \\end{align}\n    \n\n       \\item[c)]     U-CS is a special case of FSCO where:\n    \\begin{itemize}\n        \\item [i)]\n        $\\hat y_k=\\hat x_k$ and $\\hat \\Gamma_k(\\cdot)=\\ell_f(\\cdot;\\hat x_{k-1})+h(\\cdot)$\n    for every $k \\ge 1$;\n    \\item[ii)] assumption (F1) is satisfied with\n    $\\underline \\lam$ given by \\eqref{ineq:lamj2}.\n    \\end{itemize}\n\n    \\end{itemize}\n    \\end{proposition}","sha256_utf8":"400c83acbe03e9c968fc9a6a023ed1547f815523a650125156a8bce5e9a387c7"}]},{"id":"S024","semantic_type":"proof","kind":"proof","title":"框架嵌入命题证明","source_units":[{"start":120858,"end":121804,"start_line":2394,"end_line":2412,"raw_tex":"\\begin{proof}\n    a) This statement directly follows from the description of U-CS.\n    \n    b) Relations \\eqref{eq:sub1cs} and \\eqref{ineq:iterationcs} directly follow from the description of U-CS.\nNext, we prove \\eqref{ineq:lamj2}.\nSupposing $\\lam_{k-1}\\le (1-\\chi)^2 \\bar \\varepsilon/(4 M_f^2 + \\bar \\varepsilon L_f)$,\nthen it follows from Lemma \\ref{lem:lam} that \\eqref{ineq:iterationcs} holds with $\\lam_k$ replaced by $\\lambda_{k-1}$. This indicates that if $\\lam$ is small enough, then it will remain unchanged. Therefore, following from the update scheme of $\\lam$ in step 2 of U-CS, there is a lower bound as in \\eqref{ineq:lamj2}.\n\nc) Relations\n  \\eqref{eq:sub1cs}  and\n     \\eqref{ineq:iterationcs}\n     are the analogues of\n     relations \\eqref{propyb} and  \\eqref{defxhatk}\n     of FSCO with \n$\\hat y_k=\\hat x_k$ and $\\hat \\Gamma_k$  as in (i).\nInequality \\eqref{ineq:lamj2}\nshows that Assumption (F1) is\nsatisfied. \n    \\end{proof}","sha256_utf8":"629cc21da9c534f9bc528b678f594ad117ff7a37fb96bcfda6053a90c8e293b4"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"步长序列单调","source_units":[{"start":120872,"end":120941,"start_line":2395,"end_line":2395,"raw_tex":"    a) This statement directly follows from the description of U-CS.\n","sha256_utf8":"7e3cbc62e092b401db28095f0830e2bd68617e2ade8ccb40174d31a2ac7812a2"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"验证接受条件与步长下界","source_units":[{"start":120946,"end":121499,"start_line":2397,"end_line":2400,"raw_tex":"    b) Relations \\eqref{eq:sub1cs} and \\eqref{ineq:iterationcs} directly follow from the description of U-CS.\nNext, we prove \\eqref{ineq:lamj2}.\nSupposing $\\lam_{k-1}\\le (1-\\chi)^2 \\bar \\varepsilon/(4 M_f^2 + \\bar \\varepsilon L_f)$,\nthen it follows from Lemma \\ref{lem:lam} that \\eqref{ineq:iterationcs} holds with $\\lam_k$ replaced by $\\lambda_{k-1}$. This indicates that if $\\lam$ is small enough, then it will remain unchanged. Therefore, following from the update scheme of $\\lam$ in step 2 of U-CS, there is a lower bound as in \\eqref{ineq:lamj2}.\n","sha256_utf8":"e1fc79f75b8238a87ecee27269621c3274c08170fe26648acd2b0b5dc567159e"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"核对黑盒及框架条件","source_units":[{"start":121500,"end":121789,"start_line":2402,"end_line":2411,"raw_tex":"c) Relations\n  \\eqref{eq:sub1cs}  and\n     \\eqref{ineq:iterationcs}\n     are the analogues of\n     relations \\eqref{propyb} and  \\eqref{defxhatk}\n     of FSCO with \n$\\hat y_k=\\hat x_k$ and $\\hat \\Gamma_k$  as in (i).\nInequality \\eqref{ineq:lamj2}\nshows that Assumption (F1) is\nsatisfied. \n","sha256_utf8":"93c436a7fe225e9932f36eccd0d8df5072a3a6061f2554ebaf0fb38859ba00e3"}]},{"id":"S028","semantic_type":"theorem","kind":"theorem","title":"FSCO函数值复杂度","source_units":[{"start":100811,"end":101688,"start_line":2027,"end_line":2039,"raw_tex":"\\begin{theorem}\\label{thouter}\n% Assume that (F1) holds. Then,\nThe number of iterations performed by any instance of  FSCO that satisfies condition (F1) is bounded by\n\\begin{equation}\\label{complprimal}\n \\min\\left\\{\\min\\left[ \\frac{1}{\\chi}\\left(1+\\frac{1}{\\underline{\\lam} \\mu}\\right),  1+\\frac{1}{\\underline{\\lam} \\nu}\\right] \\log\\left(1+ \\frac{\\mu d_0^2}{\\bar \\varepsilon}\\right), \\frac{d_0^2}{{\\underline \\lambda}{\\bar \\varepsilon}} \\right\\},\n\\end{equation}\nwhere the scalars $\\chi$ and $\\bar \\varepsilon$ are input to FSCO, $d_0$ is as in \\eqref{defd0}, and\n\\begin{equation}\\label{eq:nu}\n    \\mu=\\mu_\\phi, \\quad \\nu := \\min \\left\\{ \\mu_\\phi \\, , \\, \\inf_k \\mu (\\hat \\Gamma_k) \\right\\}.\n\\end{equation}\n${\\rm (}$By convention, \\eqref{complprimal} \nshould be understood as being equal to $d_0^2/({\\underline{\\lambda}}{\\bar{\\varepsilon}})$ when $\\mu=0$.${\\rm )}$\n\\end{theorem}","sha256_utf8":"8ece7115244cf751d3dec9f87bf67e0dcda068762e428ee0a67b0b8893fb461b"}]},{"id":"S029","semantic_type":"proof","kind":"proof","title":"FSCO复杂度完整证明","source_units":[{"start":101691,"end":105390,"start_line":2042,"end_line":2099,"raw_tex":"\\begin{proof}\nIt is straightforward to see from Lemma \\ref{lem:sigma}(b) that $\\sigma(\\lam) \\ge \\underline \\sigma := \\sigma(\\underline \\lam)$ for $\\lam \\ge \\underline \\lam$, where $\\sigma(\\lam)$ denotes the $\\sigma$ defined in \\eqref{eq:sigma}, considered as a function of $\\lam$.\nNote that we invoke the oracle $(\\hat x_k, \\hat y_k, \\hat \\Gamma_k, \\lambda_k) = \\BB(\\hat x_{k-1},\\chi, (1-\\chi) \\bar \\varepsilon/2)$ in step 1.\nUsing \\eqref{ineq:contraction}, assumption (F1), and the observation that $\\sigma(\\lam_k) \\ge \\underline \\sigma$, we have\n\\begin{equation}\\label{ohiustb}\n    2 \\lambda_k\\left[\\phi(\\hat y_k)-\\phi(u)\\right]  \\le \\lam_k \\bar \\varepsilon + \\left\\|\\hat x_{k-1}-u\\right\\|^2 - (1+\\underline \\sigma) \\|\\hat x_k-u\\|^2,\n\\end{equation}\n% ------------\n% \n% Assume that\n% \\[\n%     2 \\lambda_k\\left[\\phi(\\hat y_k)-\\Gamma_k(u)\\right]  \\le \\frac{2\\lam_k \\varepsilon}{1-\\chi} + \\left\\|\\hat x_{k-1}-u\\right\\|^2 - (1+\\underline \\sigma_1) \\|\\hat x_k-u\\|^2,\n% \\]\n% \\[\n% \\phi_* \\ge \\ell_K \\ge \\Gamma_K(u_K)  \\ge \\Gamma_k(u_K) \\quad \\forall k \\le K\n% \\]\n% \\[\n% \\|\\hat x_K - u_K\\| \n% \\le \\alpha_K \\|\\hat x_0-u_k\\| + \\tau = \n% \\]\n% \\[\n% \\ell_K = \\max \\{ \\Gamma_k(u_k) : k \\le K \\}\n% \\]\n% \\[\n% \\phi(y_K) - \\ell_K \\le \\bar \\varepsilon\n% \\]\n% \\[\n% u_k = \\min \\{ \\Gamma_k(u) : u \\in \\Omega \\}\n% \\]\n% \n% -------------\nwhere \n\\begin{equation}\\label{def:sigma}\n    \\underline \\sigma = \\frac{\\underline{\\lam}[\\nu(1+\\mu \\underline{\\lam})+\\chi(\\mu-\\nu)]}{1+\\mu {\\underline \\lambda}+\\chi {\\underline \\lambda}(\\nu-\\mu)},\n\\end{equation}\nand $\\mu$ and $\\nu$ are as in \\eqref{eq:nu}.\nIt is easy to see that \\eqref{ohiustb} with $u=x_*$, where $x_*$ denotes the closest solution of \\eqref{eq:ProbIntro} to the initial point $\\hat x_0$, satisfies \n$$\n\t\\gamma_k \\eta_k \\le \\alpha_{k-1} - (1+\\sigma)\\alpha_k + \\gamma_k \\delta\n$$\nwith\n\\begin{equation}\\label{eq:parameter}\n    \\gamma_k = 2\\lam_k, \\quad \\eta_k=\\phi(\\hat y_k) - \\phi_*, \\quad \\alpha_k = \\|\\hat x_k-x_*\\|^2, \\quad \\delta = \\frac{\\bar \\varepsilon}{2}, \\quad \\sigma = \\underline \\sigma, \\quad {\\underline \\gamma} = 2{\\underline \\lam},\n\\end{equation}\nwhere ${\\underline \\gamma}$\nsatisfies $\\gamma_k \\geq {\\underline \\gamma}$.\n\nUsing Lemma \\ref{lm:easyrecur1}(b) with the above parameters, Lemma \\ref{lem:sigma}(c), and the definition of $\\underline \\sigma$ in \\eqref{def:sigma} that the complexity to find a $\\bar \\varepsilon$-solution is\n\\begin{equation}\\label{eq:complexity}\n  \\min\\left\\{\\frac{1+\\underline \\sigma}{\\underline \\sigma} \\log\\left(1+ \\frac{\\underline\\sigma d_0^2}{{\\underline \\lambda}{\\bar \\varepsilon}}\\right), \\frac{d_0^2}{{\\underline \\lambda}{\\bar \\varepsilon}} \\right\\} \\le \\min\\left\\{\\min\\left[ \\frac{1}{\\chi}\\left(1+\\frac{1}{\\underline{\\lam} \\mu}\\right),  1+\\frac{1}{\\underline{\\lam} \\nu}\\right] \\log\\left(1+ \\frac{\\underline \\sigma d_0^2}{{\\underline \\lambda}{\\bar \\varepsilon}}\\right), \\frac{d_0^2}{{\\underline \\lambda}{\\bar \\varepsilon}} \\right\\}.  \n\\end{equation}\n% Moreover, using the definition of $\\underline \\sigma$ in \\eqref{def:sigma} and the assumptions that $\\chi\\in [0,1)$ and $\\nu \\le \\mu$, we have\n% \\[\n% \\frac{\\underline \\sigma}{\\underline{\\lam}} \\stackrel{\\eqref{def:sigma}}= \\frac{\\nu(1+\\mu \\underline{\\lam})+\\chi(\\mu-\\nu)}{1+\\mu {\\underline \\lambda}+\\chi {\\underline \\lambda}(\\nu-\\mu)} \\le \\frac{\\nu(1+\\mu \\underline{\\lam})+\\mu-\\nu}{1+\\mu {\\underline \\lambda}+ {\\underline \\lambda}(\\nu-\\mu)} = \\frac{\\mu(1+\\nu \\underline{\\lam})}{1+\\nu {\\underline \\lambda}} = \\mu.\n% \\]\nMoreover, we note that $\\underline \\sigma \\le \\underline \\lam \\mu$ in view of the second inequality in Lemma \\ref{lem:sigma}(a).\nFinally, this observation and \\eqref{eq:complexity} immediately imply the complexity in \\eqref{complprimal}.\n\\end{proof}","sha256_utf8":"4327d1865678951064134aed0a128635f742ff5012c55710ad0be7da690f4c94"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"取得统一收缩系数","source_units":[{"start":101691,"end":103261,"start_line":2042,"end_line":2077,"raw_tex":"\\begin{proof}\nIt is straightforward to see from Lemma \\ref{lem:sigma}(b) that $\\sigma(\\lam) \\ge \\underline \\sigma := \\sigma(\\underline \\lam)$ for $\\lam \\ge \\underline \\lam$, where $\\sigma(\\lam)$ denotes the $\\sigma$ defined in \\eqref{eq:sigma}, considered as a function of $\\lam$.\nNote that we invoke the oracle $(\\hat x_k, \\hat y_k, \\hat \\Gamma_k, \\lambda_k) = \\BB(\\hat x_{k-1},\\chi, (1-\\chi) \\bar \\varepsilon/2)$ in step 1.\nUsing \\eqref{ineq:contraction}, assumption (F1), and the observation that $\\sigma(\\lam_k) \\ge \\underline \\sigma$, we have\n\\begin{equation}\\label{ohiustb}\n    2 \\lambda_k\\left[\\phi(\\hat y_k)-\\phi(u)\\right]  \\le \\lam_k \\bar \\varepsilon + \\left\\|\\hat x_{k-1}-u\\right\\|^2 - (1+\\underline \\sigma) \\|\\hat x_k-u\\|^2,\n\\end{equation}\n% ------------\n% \n% Assume that\n% \\[\n%     2 \\lambda_k\\left[\\phi(\\hat y_k)-\\Gamma_k(u)\\right]  \\le \\frac{2\\lam_k \\varepsilon}{1-\\chi} + \\left\\|\\hat x_{k-1}-u\\right\\|^2 - (1+\\underline \\sigma_1) \\|\\hat x_k-u\\|^2,\n% \\]\n% \\[\n% \\phi_* \\ge \\ell_K \\ge \\Gamma_K(u_K)  \\ge \\Gamma_k(u_K) \\quad \\forall k \\le K\n% \\]\n% \\[\n% \\|\\hat x_K - u_K\\| \n% \\le \\alpha_K \\|\\hat x_0-u_k\\| + \\tau = \n% \\]\n% \\[\n% \\ell_K = \\max \\{ \\Gamma_k(u_k) : k \\le K \\}\n% \\]\n% \\[\n% \\phi(y_K) - \\ell_K \\le \\bar \\varepsilon\n% \\]\n% \\[\n% u_k = \\min \\{ \\Gamma_k(u) : u \\in \\Omega \\}\n% \\]\n% \n% -------------\nwhere \n\\begin{equation}\\label{def:sigma}\n    \\underline \\sigma = \\frac{\\underline{\\lam}[\\nu(1+\\mu \\underline{\\lam})+\\chi(\\mu-\\nu)]}{1+\\mu {\\underline \\lambda}+\\chi {\\underline \\lambda}(\\nu-\\mu)},\n\\end{equation}\nand $\\mu$ and $\\nu$ are as in \\eqref{eq:nu}.\n","sha256_utf8":"bbc01089b79d31f3fbb5756843487a28c2feb3bb1b7a71a72a97f5807528e134"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"构造最优点距离递推","source_units":[{"start":103261,"end":103858,"start_line":2078,"end_line":2087,"raw_tex":"It is easy to see that \\eqref{ohiustb} with $u=x_*$, where $x_*$ denotes the closest solution of \\eqref{eq:ProbIntro} to the initial point $\\hat x_0$, satisfies \n$$\n\t\\gamma_k \\eta_k \\le \\alpha_{k-1} - (1+\\sigma)\\alpha_k + \\gamma_k \\delta\n$$\nwith\n\\begin{equation}\\label{eq:parameter}\n    \\gamma_k = 2\\lam_k, \\quad \\eta_k=\\phi(\\hat y_k) - \\phi_*, \\quad \\alpha_k = \\|\\hat x_k-x_*\\|^2, \\quad \\delta = \\frac{\\bar \\varepsilon}{2}, \\quad \\sigma = \\underline \\sigma, \\quad {\\underline \\gamma} = 2{\\underline \\lam},\n\\end{equation}\nwhere ${\\underline \\gamma}$\nsatisfies $\\gamma_k \\geq {\\underline \\gamma}$.\n","sha256_utf8":"73a6cecf77f1fb904441f06bda4799adc0bb6fbfc1d648480d25cfb18ae8713d"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"由递推推出复杂度界","source_units":[{"start":103858,"end":105390,"start_line":2088,"end_line":2099,"raw_tex":"\nUsing Lemma \\ref{lm:easyrecur1}(b) with the above parameters, Lemma \\ref{lem:sigma}(c), and the definition of $\\underline \\sigma$ in \\eqref{def:sigma} that the complexity to find a $\\bar \\varepsilon$-solution is\n\\begin{equation}\\label{eq:complexity}\n  \\min\\left\\{\\frac{1+\\underline \\sigma}{\\underline \\sigma} \\log\\left(1+ \\frac{\\underline\\sigma d_0^2}{{\\underline \\lambda}{\\bar \\varepsilon}}\\right), \\frac{d_0^2}{{\\underline \\lambda}{\\bar \\varepsilon}} \\right\\} \\le \\min\\left\\{\\min\\left[ \\frac{1}{\\chi}\\left(1+\\frac{1}{\\underline{\\lam} \\mu}\\right),  1+\\frac{1}{\\underline{\\lam} \\nu}\\right] \\log\\left(1+ \\frac{\\underline \\sigma d_0^2}{{\\underline \\lambda}{\\bar \\varepsilon}}\\right), \\frac{d_0^2}{{\\underline \\lambda}{\\bar \\varepsilon}} \\right\\}.  \n\\end{equation}\n% Moreover, using the definition of $\\underline \\sigma$ in \\eqref{def:sigma} and the assumptions that $\\chi\\in [0,1)$ and $\\nu \\le \\mu$, we have\n% \\[\n% \\frac{\\underline \\sigma}{\\underline{\\lam}} \\stackrel{\\eqref{def:sigma}}= \\frac{\\nu(1+\\mu \\underline{\\lam})+\\chi(\\mu-\\nu)}{1+\\mu {\\underline \\lambda}+\\chi {\\underline \\lambda}(\\nu-\\mu)} \\le \\frac{\\nu(1+\\mu \\underline{\\lam})+\\mu-\\nu}{1+\\mu {\\underline \\lambda}+ {\\underline \\lambda}(\\nu-\\mu)} = \\frac{\\mu(1+\\nu \\underline{\\lam})}{1+\\nu {\\underline \\lambda}} = \\mu.\n% \\]\nMoreover, we note that $\\underline \\sigma \\le \\underline \\lam \\mu$ in view of the second inequality in Lemma \\ref{lem:sigma}(a).\nFinally, this observation and \\eqref{eq:complexity} immediately imply the complexity in \\eqref{complprimal}.\n\\end{proof}","sha256_utf8":"d3561b1a4a84573d09725b581463146365404459df5298df151a5466a14e9fcb"}]},{"id":"S033","semantic_type":"proposition","kind":"proposition","title":"黑盒近端中心收缩","source_units":[{"start":87789,"end":88529,"start_line":1784,"end_line":1799,"raw_tex":"\\begin{proposition}\\label{propouter}\nAssume that  $\\phi \\in \\bConv{n}$ and let $\\mu=\\mu_\\phi$.\nAssume also that\n$(x,y,\\Gamma,\\lam) = \\BB(x^-,\\chi,\\varepsilon)$\nfor some $(x^-,\\chi,\\varepsilon) \\in \\dom \\phi \\times [0,1) \\times \\R^n_{++}$, and $\\Gamma \\in \\nConv{n}$ for some $\\nu \\in [0,\\mu]$.\nThen, for every $u \\in \\R^n$, we have\n    \n% and $\\zeta \\in [0,\\chi]$,\n\\begin{equation}\\label{ineq:contraction}\n    2 \\lambda\\left[\\phi(y)-\\phi(u)\\right]  \\le \\frac{2\\lam \\varepsilon}{1-\\chi} + \\left\\|x^- - u\\right\\|^2 - (1+\\sigma) \\|x-u\\|^2,\n\\end{equation}\nwhere \n\\begin{equation}\\label{eq:sigma}\n    \\sigma = \\sigma(\\mu,\\nu) := \\frac{\\lam[\\nu(1+\\mu \\lam)+\\chi(\\mu-\\nu)]}{1+\\mu  \\lambda+\\chi  \\lambda(\\nu-\\mu)}.\n\\end{equation} \n\\end{proposition}","sha256_utf8":"4624b932533c9704aa5f405c9d744d628d6611f7191b3278e31d2afd273a4e3e"}]},{"id":"S034","semantic_type":"proof","kind":"proof","title":"收缩命题完整证明","source_units":[{"start":117295,"end":119452,"start_line":2319,"end_line":2352,"raw_tex":"{\\bf Proof of Proposition \\ref{propouter}.}\nIt follows from the definition of $s$ in \\eqref{defsb} and Lemma \\ref{lem:zeta} with $\\zeta =\\chi$,\n\\begin{align}\n&\\|x^--u \\|^2  - \\|x-u\\|^2 -\\left(\\|x^--y\\|^2 - \\|x-y\\|^2\\right) \n= -2\\lam \\inner{s}{u-y} \\nn \\\\\n\\stackrel{\\eqref{qphi-1}}\\ge & 2 \\lambda \\left[\\phi(y)- \\phi(u)\\right] + \\chi (\\mu -\\nu) \\lam \\|u-y\\|^2 -   \\frac{2\\lam \\eta}{1-\\chi} + \\frac{\\chi \\nu \\lam}{1-\\chi} \\|y - x\\|^2 + \\nu \\lam\\|u - x\\|^2. \\label{ineq:long}\n\\end{align}\nRearranging the terms in \\eqref{ineq:long} and using Lemma \\ref{lemetab}(b), we have\n\\begin{align}\n&\\left\\|x^--u \\right\\|^2  - (1+\\nu \\lam)\\|x-u\\|^2  -2 \\lambda \\left[\\phi(y)- \\phi(u)\\right] \\nn \\\\\n& \\stackrel{\\eqref{ineq:long}}\\ge \\|x^--y\\|^2 + \\left( \\frac{\\chi \\nu \\lam}{1-\\chi} - 1 \\right)\\|x-y\\|^2 + \\chi (\\mu -\\nu) \\lam \\|u-y\\|^2 -   \\frac{2\\lam \\eta}{1-\\chi} \\nn \\\\\n& \\stackrel{\\eqref{ineqetab}}\\ge  -\\frac{2 \\lam \\varepsilon}{1-\\chi} + \\frac{\\chi(1+\\nu \\lambda)}{1-\\chi} \\|x-y\\|^2 + \\chi (\\mu -\\nu) \\lam \\|u-y\\|^2. \\label{ineq:imp}\n\\end{align}\nRearranging the terms in the above inequality, we have\n\\begin{align}\n    2\\lambda \\left[\\phi(y)- \\phi(u)\\right] \\le& \n\\frac{2 \\lam \\varepsilon}{1-\\chi} + \\left\\|x^--u \\right\\|^2  - (1+\\nu \\lam)\\|x-u\\|^2 \\nn \\\\\n&- \\chi \\left(\\frac{1+\\nu \\lam}{1-\\chi} \\|x-y\\|^2 + (\\mu -\\nu) \\lam \\|u-y\\|^2\\right). \\label{telescopic1}\n\\end{align}\nUsing the triangle inequality and the fact that $(a_1+a_2)^2 \\le (b_1^{-1}+b_2^{-1}) (a_1^2 b_1 + a_2^2 b_2)$ with $(a_1,a_2) =(\\|x-y\\|,\\|u-y\\|)$ and $(b_1,b_2) =((1+\\nu \\lam)/(1-\\chi), (\\mu -\\nu) \\lam)$\nwe have\n\\begin{equation}\\label{ineqxku}\n    \\|x-u\\|^2 \\le \\left(\\frac{1-\\chi}{1+\\nu \\lam} + \\frac{1}{(\\mu -\\nu)\\lam}\\right) \\left(\\frac{1+\\nu \\lam}{1-\\chi} \\|x-y\\|^2 + (\\mu -\\nu) \\lam \\|u-y\\|^2\\right).\n\\end{equation}\nPlugging the above inequality into \\eqref{telescopic1}, we have\n\\[\n2\\lambda \\left[\\phi(y)-\\phi(u)\\right] \n\\le   \\frac{2 \\lam \\varepsilon}{1-\\chi} +  \n\\left\\|x^--u\\right\\|^2  - \n\\left[ 1 + \\nu \\lam + \\chi \\left(\\frac{1-\\chi}{1+\\nu \\lam} + \\frac{1}{(\\mu -\\nu)\\lam}\\right)^{-1} \\right]\n\\|x-u\\|^2,\n\\]\nwhich is the same as \\eqref{ineq:contraction} after simplification.\n\\QEDA\n","sha256_utf8":"c1821820ea05ef94ec1782ee0f479edb88754fe977e12af50e18f11c90ccbce0"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"距离恒等式结合强凸下界","source_units":[{"start":117295,"end":117780,"start_line":2319,"end_line":2325,"raw_tex":"{\\bf Proof of Proposition \\ref{propouter}.}\nIt follows from the definition of $s$ in \\eqref{defsb} and Lemma \\ref{lem:zeta} with $\\zeta =\\chi$,\n\\begin{align}\n&\\|x^--u \\|^2  - \\|x-u\\|^2 -\\left(\\|x^--y\\|^2 - \\|x-y\\|^2\\right) \n= -2\\lam \\inner{s}{u-y} \\nn \\\\\n\\stackrel{\\eqref{qphi-1}}\\ge & 2 \\lambda \\left[\\phi(y)- \\phi(u)\\right] + \\chi (\\mu -\\nu) \\lam \\|u-y\\|^2 -   \\frac{2\\lam \\eta}{1-\\chi} + \\frac{\\chi \\nu \\lam}{1-\\chi} \\|y - x\\|^2 + \\nu \\lam\\|u - x\\|^2. \\label{ineq:long}\n\\end{align}\n","sha256_utf8":"300ec2d49ca4ed76671eefca021a0eff9a5721f3c92fd95ac3572245fbcfdb04"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"消去近似次梯度误差","source_units":[{"start":117780,"end":118660,"start_line":2326,"end_line":2337,"raw_tex":"Rearranging the terms in \\eqref{ineq:long} and using Lemma \\ref{lemetab}(b), we have\n\\begin{align}\n&\\left\\|x^--u \\right\\|^2  - (1+\\nu \\lam)\\|x-u\\|^2  -2 \\lambda \\left[\\phi(y)- \\phi(u)\\right] \\nn \\\\\n& \\stackrel{\\eqref{ineq:long}}\\ge \\|x^--y\\|^2 + \\left( \\frac{\\chi \\nu \\lam}{1-\\chi} - 1 \\right)\\|x-y\\|^2 + \\chi (\\mu -\\nu) \\lam \\|u-y\\|^2 -   \\frac{2\\lam \\eta}{1-\\chi} \\nn \\\\\n& \\stackrel{\\eqref{ineqetab}}\\ge  -\\frac{2 \\lam \\varepsilon}{1-\\chi} + \\frac{\\chi(1+\\nu \\lambda)}{1-\\chi} \\|x-y\\|^2 + \\chi (\\mu -\\nu) \\lam \\|u-y\\|^2. \\label{ineq:imp}\n\\end{align}\nRearranging the terms in the above inequality, we have\n\\begin{align}\n    2\\lambda \\left[\\phi(y)- \\phi(u)\\right] \\le& \n\\frac{2 \\lam \\varepsilon}{1-\\chi} + \\left\\|x^--u \\right\\|^2  - (1+\\nu \\lam)\\|x-u\\|^2 \\nn \\\\\n&- \\chi \\left(\\frac{1+\\nu \\lam}{1-\\chi} \\|x-y\\|^2 + (\\mu -\\nu) \\lam \\|u-y\\|^2\\right). \\label{telescopic1}\n\\end{align}\n","sha256_utf8":"7e05bc822a20749601b10619203426a456c2399eeba0cf0e3f30a92c2f737e9b"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"合并距离项得到收缩","source_units":[{"start":118660,"end":119452,"start_line":2338,"end_line":2352,"raw_tex":"Using the triangle inequality and the fact that $(a_1+a_2)^2 \\le (b_1^{-1}+b_2^{-1}) (a_1^2 b_1 + a_2^2 b_2)$ with $(a_1,a_2) =(\\|x-y\\|,\\|u-y\\|)$ and $(b_1,b_2) =((1+\\nu \\lam)/(1-\\chi), (\\mu -\\nu) \\lam)$\nwe have\n\\begin{equation}\\label{ineqxku}\n    \\|x-u\\|^2 \\le \\left(\\frac{1-\\chi}{1+\\nu \\lam} + \\frac{1}{(\\mu -\\nu)\\lam}\\right) \\left(\\frac{1+\\nu \\lam}{1-\\chi} \\|x-y\\|^2 + (\\mu -\\nu) \\lam \\|u-y\\|^2\\right).\n\\end{equation}\nPlugging the above inequality into \\eqref{telescopic1}, we have\n\\[\n2\\lambda \\left[\\phi(y)-\\phi(u)\\right] \n\\le   \\frac{2 \\lam \\varepsilon}{1-\\chi} +  \n\\left\\|x^--u\\right\\|^2  - \n\\left[ 1 + \\nu \\lam + \\chi \\left(\\frac{1-\\chi}{1+\\nu \\lam} + \\frac{1}{(\\mu -\\nu)\\lam}\\right)^{-1} \\right]\n\\|x-u\\|^2,\n\\]\nwhich is the same as \\eqref{ineq:contraction} after simplification.\n\\QEDA\n","sha256_utf8":"da212f6376fc963e733bf7103dc405b8e7b1e1334971bc0782cfa73f10d30c1e"}]},{"id":"S038","semantic_type":"lemma","kind":"lemma","title":"近似次梯度与误差界","source_units":[{"start":110270,"end":110889,"start_line":2207,"end_line":2227,"raw_tex":"\\begin{lemma}\\label{lemetab} \nDefine \n \\begin{align}\n\t s &:= \\frac{x^- - x}{\\lam}, \\label{defsb}\\\\\n\t \\eta &:= \\phi(y) - \\Gamma(x) - \\inner{s}{y- x}. \\label{defetab}\n\t\\end{align}\n\nThen, the following statements hold:\n\\begin{itemize}\n\\item[a)]\n  $s \\in \\partial \\Gamma (x)$, i.e., for every $u \\in \\R^n$,\n \\begin{equation}\\label{eq:subdifb}\n    \\Gamma(u) \\ge \\Gamma(x) + \\inner{s}{u-x}; \n \\end{equation}  \n  \\item[b)] $s \\in \\partial_{\\eta} \\phi(y)$ and\n  \\begin{equation}\\label{ineqetab}\n  0 \\le 2 \\lam \\eta \\le  2 \\lam \\varepsilon - \n\t    \\|y- x\\|^2 + (1-\\chi)\\|y- x^-\\|^2.\n     \\end{equation}\n\\end{itemize}\n\\end{lemma}","sha256_utf8":"bdfb3a84253f1b63a565b1c660704b4bd045aa339139da90866585fa270db069"}]},{"id":"S039","semantic_type":"proof","kind":"proof","title":"近似次梯度引理证明","source_units":[{"start":110892,"end":112806,"start_line":2229,"end_line":2257,"raw_tex":"\\begin{proof} \n (a) The optimality condition of \\eqref{defxhatk} yields $0 \\in \\partial \\Gamma_k(x) +(x-x^-)/\\lambda$, which together with the definition of $s$ in \\eqref{defsb} implies that the inclusion in a) holds.\n Relation \\eqref{eq:subdifb} immediately follows from the inclusion in a) and the definition of the subdifferential.\n \n(b) The relation $\\phi \\geq \\Gamma$ (see step 1 of FSCO), \\eqref{eq:subdifb}, the definition of $\\eta$ in \\eqref{defetab} imply that for every $u\\in \\dom \\phi$,\n\t\\[\n\t\\phi(u) \\ge \\Gamma(u) \\stackrel{\\eqref{eq:subdifb}}\\ge  \\Gamma(x) + \\inner{s}{u- x}\n      \\stackrel{\\eqref{defetab}}=  \\phi(y) + \\inner{s}{u- y} - \\eta,\n\t\\]\nwhich yields the inclusion in b). \nTaking $u=y$ in the above inequality gives $\\eta \\ge 0$, and hence the first inequality in \\eqref{ineqetab} holds.\nUsing the definitions of $s$ and $\\eta$ in \\eqref{defsb} and \\eqref{defetab}, respectively, and \\eqref{propyb} and \\eqref{defxhatk}, we have\n\\[\n\\begin{array}{lcl}\n    \\eta &\\stackrel{\\eqref{defetab}}=& \\displaystyle \\phi(y) - \\Gamma(x) - \\inner{s}{y- x} \\\\\n     & \\stackrel{\\eqref{propyb},\\eqref{defxhatk}}{\\le} &\\displaystyle \\left[ \\varepsilon - \\frac\\chi{2\\lam}\\|y- x^-\\|^2 + \\frac{1}{2\\lam}\\|x- x^- \\|^2 \\right] - \\inner{s}{y- x}\\\\\n\t    &\\stackrel{\\eqref{defsb}}{=} &  \\displaystyle \\varepsilon - \\frac\\chi{2\\lam}\\|y- x^-\\|^2  +  \\frac1{2\\lam}\\|x- x^-\\|^2 + \\Inner{\\frac{x- x^-}{\\lam}}{y- x} \\\\\n     &=&\\displaystyle  \\varepsilon + \\frac{1-\\chi}{2\\lam}\n\t    \\|y- x^- \\|^2 - \\frac1{2\\lam}\n\t    \\|y- x\\|^2.\n\t\\end{array}\n\\]\nHence, the second inequality in \\eqref{ineqetab} holds.\n% (c) Since $\\phi-\\mu\\|\\cdot\\|^2/2$ is convex, using the inclusion in part (b) and Lemma \\ref{lem:app} with $\\tau=(1-\\zeta)/\\zeta$ for $\\zeta \\in [0,1)$, we obtain\n% \\[\n% \\phi(u) \\ge \\phi(y) + \\inner{s}{u- y} + \\frac{\\zeta \\mu}{2} \\|u- y\\|^2 - \\frac{\\eta}{1-\\zeta}.\n% \\]\n% That is \\eqref{qphi-1} with $\\nu = 0$.\n\\end{proof}","sha256_utf8":"49ce67dfe2022c4e83d07359b54563430cd97b6a9b259acd18cccf273b9bd85e"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"近端最优性给出次梯度","source_units":[{"start":110892,"end":111227,"start_line":2229,"end_line":2231,"raw_tex":"\\begin{proof} \n (a) The optimality condition of \\eqref{defxhatk} yields $0 \\in \\partial \\Gamma_k(x) +(x-x^-)/\\lambda$, which together with the definition of $s$ in \\eqref{defsb} implies that the inclusion in a) holds.\n Relation \\eqref{eq:subdifb} immediately follows from the inclusion in a) and the definition of the subdifferential.\n","sha256_utf8":"de7e6ddfb1e02cf091f8a1ea5288d714bbf4007889af1d32183b62421d55ef23"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"建立近似次梯度包含","source_units":[{"start":111227,"end":111702,"start_line":2232,"end_line":2239,"raw_tex":" \n(b) The relation $\\phi \\geq \\Gamma$ (see step 1 of FSCO), \\eqref{eq:subdifb}, the definition of $\\eta$ in \\eqref{defetab} imply that for every $u\\in \\dom \\phi$,\n\t\\[\n\t\\phi(u) \\ge \\Gamma(u) \\stackrel{\\eqref{eq:subdifb}}\\ge  \\Gamma(x) + \\inner{s}{u- x}\n      \\stackrel{\\eqref{defetab}}=  \\phi(y) + \\inner{s}{u- y} - \\eta,\n\t\\]\nwhich yields the inclusion in b). \nTaking $u=y$ in the above inequality gives $\\eta \\ge 0$, and hence the first inequality in \\eqref{ineqetab} holds.\n","sha256_utf8":"14dbf48b2346b3a26033e07747bfb21d4296fce5ea21b8381d205b446c00a8b0"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"展开黑盒误差证书","source_units":[{"start":111702,"end":112806,"start_line":2240,"end_line":2257,"raw_tex":"Using the definitions of $s$ and $\\eta$ in \\eqref{defsb} and \\eqref{defetab}, respectively, and \\eqref{propyb} and \\eqref{defxhatk}, we have\n\\[\n\\begin{array}{lcl}\n    \\eta &\\stackrel{\\eqref{defetab}}=& \\displaystyle \\phi(y) - \\Gamma(x) - \\inner{s}{y- x} \\\\\n     & \\stackrel{\\eqref{propyb},\\eqref{defxhatk}}{\\le} &\\displaystyle \\left[ \\varepsilon - \\frac\\chi{2\\lam}\\|y- x^-\\|^2 + \\frac{1}{2\\lam}\\|x- x^- \\|^2 \\right] - \\inner{s}{y- x}\\\\\n\t    &\\stackrel{\\eqref{defsb}}{=} &  \\displaystyle \\varepsilon - \\frac\\chi{2\\lam}\\|y- x^-\\|^2  +  \\frac1{2\\lam}\\|x- x^-\\|^2 + \\Inner{\\frac{x- x^-}{\\lam}}{y- x} \\\\\n     &=&\\displaystyle  \\varepsilon + \\frac{1-\\chi}{2\\lam}\n\t    \\|y- x^- \\|^2 - \\frac1{2\\lam}\n\t    \\|y- x\\|^2.\n\t\\end{array}\n\\]\nHence, the second inequality in \\eqref{ineqetab} holds.\n% (c) Since $\\phi-\\mu\\|\\cdot\\|^2/2$ is convex, using the inclusion in part (b) and Lemma \\ref{lem:app} with $\\tau=(1-\\zeta)/\\zeta$ for $\\zeta \\in [0,1)$, we obtain\n% \\[\n% \\phi(u) \\ge \\phi(y) + \\inner{s}{u- y} + \\frac{\\zeta \\mu}{2} \\|u- y\\|^2 - \\frac{\\eta}{1-\\zeta}.\n% \\]\n% That is \\eqref{qphi-1} with $\\nu = 0$.\n\\end{proof}","sha256_utf8":"2e387fd7c12806c6ef2f347b45615b3c424ab0fa5e415202ce799be70198a55e"}]},{"id":"S043","semantic_type":"lemma","kind":"lemma","title":"恢复目标内在强凸性","source_units":[{"start":113060,"end":113574,"start_line":2261,"end_line":2268,"raw_tex":"\\begin{lemma}\\label{lem:zeta}\n    Let $\\mu=\\mu_\\phi$ and assume that $\\Gamma \\in \\nConv{n}$ for some $\\nu \\in [0,\\mu]$.\n    For any $\\zeta \\in [0,1)$ and every $u\\in \\dom \\phi$, we have\n     \\begin{equation}\\label{qphi-1}\n    \\phi(u) \\ge \\phi(y) + \\inner{s}{u- y} + \\frac{\\zeta (\\mu-\\nu)}{2} \\|u- y\\|^2 - \\frac{\\eta}{1-\\zeta} + \\frac{\\zeta}{1-\\zeta} \\frac{\\nu}{2}\\|y - x\\|^2 + \\frac{\\nu}{2}\\|u - x\\|^2,\n\t\\end{equation}\n    where $s$ and $\\eta$ are as in \\eqref{defsb} and \\eqref{defetab}, respectively.\n\\end{lemma}","sha256_utf8":"f0755bdf3b5eadabdbbe4b6b16aee5ecb550a2b44ac066b0333daf312086eeac"}]},{"id":"S044","semantic_type":"proof","kind":"proof","title":"强凸下界引理证明","source_units":[{"start":113576,"end":115148,"start_line":2270,"end_line":2290,"raw_tex":"\\begin{proof}\n    Define\n\\begin{equation}\\label{eq:tilde}\n    \\tilde s = s + \\nu(x^- - x), \\quad \\tilde \\eta = \\eta - \\frac{\\nu}{2}\\|y - x\\|^2, \\quad \\tilde \\phi = \\phi - \\frac{\\nu}{2}\\|\\cdot - x^-\\|^2, \\quad \\tilde \\Gamma = \\Gamma - \\frac{\\nu}{2}\\|\\cdot - x^-\\|^2.\n\\end{equation}\nIt follows from Lemma \\ref{lemetab}(a) and the above definitions of $\\tilde s$ and $\\tilde \\Gamma$ that\n$\\tilde s \\in \\partial \\tilde \\Gamma(x)$.\nIn view of \\eqref{eq:tilde} and the assumption that $\\Gamma \\in \\nConv{n}$ for some $\\nu \\in [0,\\mu]$, we observe that $\\tilde \\Gamma$ is convex and $\\tilde \\phi \\ge \\tilde \\Gamma$.\nHence, following a similar argument as in Lemma~\\ref{lemetab}(b), we conclude that\n$\\tilde s \\in \\partial_{\\tilde \\eta} \\tilde \\phi(y)$.\nUsing this inclusion, the fact that $\\tilde \\phi$ is \n$(\\mu -\\nu)$-convex, and Lemma \\ref{lem:app} with $(\\psi,\\xi,y,v,\\eta,Q)=(\\tilde \\phi, \\mu-\\nu,y,\\tilde s, \\tilde \\eta,I)$, we have for any $\\tau\\in (0,\\infty]$ and every $u\\in \\dom \\phi$,\n\\[\n\\tilde \\phi(u) \\stackrel{\\eqref{ineq:app}}\\ge \\tilde \\phi(y) + \\inner{\\tilde s}{u- y} + \\frac{\\mu -\\nu}{2(1+\\tau)} \\|u- y\\|^2 -  (1+\\tau^{-1}) \\tilde \\eta.\n\\]\nTaking $\\tau=(1-\\zeta)/\\zeta$ for $\\zeta \\in [0,1)$ in the above inequality, we obtain\n\\[\n     \\tilde \\phi(u) \\ge \\tilde \\phi(y) + \\inner{\\tilde s}{u- y} + \\frac{\\zeta (\\mu -\\nu)}{2} \\|u- y\\|^2 -   \\frac{\\tilde \\eta}{1-\\zeta}.\n\t\\]\nFinally, \\eqref{qphi-1} follows from using the definitions of $\\tilde \\phi$, $\\tilde s$, and $\\tilde \\eta$ in \\eqref{eq:tilde}, and rearranging the terms in the above inequality.\n\\end{proof}","sha256_utf8":"a1ca4c16c34b6590bfbcd812d2f56d1b6ed011ed4ea63ac19ac2b20dbd91628d"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"平移模型与近似次梯度","source_units":[{"start":113576,"end":114322,"start_line":2270,"end_line":2279,"raw_tex":"\\begin{proof}\n    Define\n\\begin{equation}\\label{eq:tilde}\n    \\tilde s = s + \\nu(x^- - x), \\quad \\tilde \\eta = \\eta - \\frac{\\nu}{2}\\|y - x\\|^2, \\quad \\tilde \\phi = \\phi - \\frac{\\nu}{2}\\|\\cdot - x^-\\|^2, \\quad \\tilde \\Gamma = \\Gamma - \\frac{\\nu}{2}\\|\\cdot - x^-\\|^2.\n\\end{equation}\nIt follows from Lemma \\ref{lemetab}(a) and the above definitions of $\\tilde s$ and $\\tilde \\Gamma$ that\n$\\tilde s \\in \\partial \\tilde \\Gamma(x)$.\nIn view of \\eqref{eq:tilde} and the assumption that $\\Gamma \\in \\nConv{n}$ for some $\\nu \\in [0,\\mu]$, we observe that $\\tilde \\Gamma$ is convex and $\\tilde \\phi \\ge \\tilde \\Gamma$.\nHence, following a similar argument as in Lemma~\\ref{lemetab}(b), we conclude that\n$\\tilde s \\in \\partial_{\\tilde \\eta} \\tilde \\phi(y)$.\n","sha256_utf8":"b02d40a51dc9425893c2d61559bf74246f259f3a2d11f5385d21524b4b333a04"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"应用强凸近似次梯度界","source_units":[{"start":114322,"end":114727,"start_line":2280,"end_line":2284,"raw_tex":"Using this inclusion, the fact that $\\tilde \\phi$ is \n$(\\mu -\\nu)$-convex, and Lemma \\ref{lem:app} with $(\\psi,\\xi,y,v,\\eta,Q)=(\\tilde \\phi, \\mu-\\nu,y,\\tilde s, \\tilde \\eta,I)$, we have for any $\\tau\\in (0,\\infty]$ and every $u\\in \\dom \\phi$,\n\\[\n\\tilde \\phi(u) \\stackrel{\\eqref{ineq:app}}\\ge \\tilde \\phi(y) + \\inner{\\tilde s}{u- y} + \\frac{\\mu -\\nu}{2(1+\\tau)} \\|u- y\\|^2 -  (1+\\tau^{-1}) \\tilde \\eta.\n\\]\n","sha256_utf8":"e76c1c42ce7d7be8a2c5169fac07260069c59daff63e32569b4142355d8a2e38"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"选择参数并还原变量","source_units":[{"start":114727,"end":115148,"start_line":2285,"end_line":2290,"raw_tex":"Taking $\\tau=(1-\\zeta)/\\zeta$ for $\\zeta \\in [0,1)$ in the above inequality, we obtain\n\\[\n     \\tilde \\phi(u) \\ge \\tilde \\phi(y) + \\inner{\\tilde s}{u- y} + \\frac{\\zeta (\\mu -\\nu)}{2} \\|u- y\\|^2 -   \\frac{\\tilde \\eta}{1-\\zeta}.\n\t\\]\nFinally, \\eqref{qphi-1} follows from using the definitions of $\\tilde \\phi$, $\\tilde s$, and $\\tilde \\eta$ in \\eqref{eq:tilde}, and rearranging the terms in the above inequality.\n\\end{proof}","sha256_utf8":"177810349e5e87b7f9a879de16c2c6d54d0c8f1ee6855a39d683d8d56eec1e60"}]},{"id":"S048","semantic_type":"lemma","kind":"lemma","title":"强凸近似次梯度标准界","source_units":[{"start":145693,"end":146181,"start_line":2855,"end_line":2860,"raw_tex":"\\begin{lemma}\\label{lem:app}\nAssume that $\\xi>0, \\psi \\in \\bConv{n}$ and $Q \\in \\mathcal{S}_{++}^n$ are such that $\\psi-$ $(\\xi / 2)\\|\\cdot\\|_Q^2$ is convex and let $(y, v, \\eta) \\in \\R^n \\times \\R^n \\times \\R_{+}$be such that $v \\in \\partial_\\eta \\psi(y)$. Then, for any $\\tau>0$,\n    \\begin{equation}\\label{ineq:app}\n        \\psi(u) \\ge \\psi(y)+\\inner{v}{u-y}-\\left(1+\\tau^{-1}\\right) \\eta+\\frac{(1+\\tau)^{-1} \\xi}{2}\\|u-y\\|_Q^2 \\quad \\forall u \\in \\R^n .\n    \\end{equation}\n\\end{lemma}","sha256_utf8":"e7c408fcd9d77cd412eafe7aed75d2e93730069102dda01485f9fae3c78e283a"}]},{"id":"S049","semantic_type":"proof","kind":"proof","title":"标准界的作者引文证明","source_units":[{"start":146182,"end":146290,"start_line":2861,"end_line":2863,"raw_tex":"\\begin{proof}\n    The proof of this result can be found in Lemma A.1 in \\cite{melo2023proximal}.\n\\end{proof}","sha256_utf8":"c442a153f4435f5767d921c1fb66ff74451499ba65b2cc579c28b3ff1b0814f0"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"援引已有引理","source_units":[{"start":146182,"end":146290,"start_line":2861,"end_line":2863,"raw_tex":"\\begin{proof}\n    The proof of this result can be found in Lemma A.1 in \\cite{melo2023proximal}.\n\\end{proof}","sha256_utf8":"c442a153f4435f5767d921c1fb66ff74451499ba65b2cc579c28b3ff1b0814f0"}]},{"id":"S051","semantic_type":"lemma","kind":"lemma","title":"误差递推复杂度引理","source_units":[{"start":146368,"end":147856,"start_line":2869,"end_line":2893,"raw_tex":"\\begin{lemma} \\label{lm:easyrecur1}\n\t\tAssume that sequences $\\{\\gamma_j\\}$, $\\{\\eta_j\\}$, and  $\\{\\alpha_j\\}$ satisfy for every $j\\ge 1$, $\\gamma_j \\ge {\\underline \\gamma}$ and \n\t\t\\beq \\label{eq:easyrecur1}\n\t\t\\gamma_j \\eta_j \\le \\alpha_{j-1} - (1+\\sigma)\\alpha_j + \\gamma_j \\delta\n\t\t\\eeq\n  for some $\\sigma \\ge 0$, $\\delta \\ge 0$ and ${\\underline \\gamma} > 0$. Then, the following statements hold:\n\t\\begin{itemize}\n            \\item[a)] for every $k\\ge 1$,\n            \\begin{equation}\\label{ineq:eta}\n            \\min_{1\\le j \\le k}  \\eta_j \\le \\frac{\\alpha_0 - (1+\\sigma)^k \\alpha_k}{\\sum_{j=1}^k (1+\\sigma)^{j-1} \\gamma_j} + \\delta;\n        \\end{equation}\n  %           \\item[b)] if the sequence $\\{\\eta_j\\}$ is nonnegative, then for every $k\\ge 1$,\n  %           \\begin{equation}\\label{ineq:alpha}\n  %           \\alpha_k \\le \\frac{\\alpha_0}{(1+\\sigma)^k}  +\n\t\t% \\frac{\\sum_{j=1}^k (1+\\sigma)^{j-1}\\gamma_j \\delta}{(1+\\sigma)^k};\n  %       \\end{equation}\n\t\t\\item[b)]  if the sequence $\\{\\alpha_j\\}$ is nonnegative, then $ \\min_{1\\le j \\le k} \\eta_j \\le 2\\delta $ for every $ k\\ge 1$ such that\n\t\t\\[\n\t\tk \\ge \\min \\left\\lbrace  \\frac{1+\\sigma}{\\sigma} \\log\\left( \\frac{\\sigma \\alpha_0}{{\\underline \\gamma}\\delta} + 1 \\right), \\frac{\\alpha_0}{{\\underline \\gamma}\\delta} \\right\\rbrace\n\t\t\\]\n\t\twith the convention that the first term is equal to the second term\n\t\twhen $\\sigma=0$. (Note  that the first term converges to the second term\n\t\tas $\\sigma \\downarrow 0$.)\n\t\\end{itemize}\n\\end{lemma}","sha256_utf8":"e2e8c7f251369fc476d8dec4050ec2511bd07e736224d75bf3b91f2f8ab6fb83"}]},{"id":"S052","semantic_type":"proof","kind":"proof","title":"递推复杂度引理证明","source_units":[{"start":147858,"end":149542,"start_line":2895,"end_line":2922,"raw_tex":"\\begin{proof}\n\ta) Multiplying \\eqref{eq:easyrecur1} by $ (1+\\sigma)^{j-1} $ and summing the resulting inequality from $ j=1 $ to $ k $, we have\n\t\\begin{align}\n\t    \\sum_{j=1}^k (1+\\sigma)^{j-1} \\gamma_j \\left[ \\min_{1\\le j \\le k}  \\eta_j \\right] &\\le \\sum_{j=1}^k (1+\\sigma)^{j-1} \\gamma_j \\eta_j\n     \\le  \\sum_{j=1}^k (1+\\sigma)^{j-1} \\left(  \\alpha_{j-1} - (1+\\sigma) \\alpha_j + \\gamma_j \\delta \\right) \\nn \\\\\n\t\t &=  \\alpha_0 - (1+\\sigma)^k \\alpha_k +\n\t\t\\sum_{j=1}^k (1+\\sigma)^{j-1}\\gamma_j \\delta. \\label{ineq:sum}\n\t\\end{align}\n        Inequality \\eqref{ineq:eta} follows immediately from the above inequality.\n\n        % b) This statement follows immediately from \\eqref{ineq:sum} and the fact that $\\eta_j\\ge 0$.\n        \n        b) It follows from \\eqref{ineq:eta}, and the facts that $\\alpha_k\\ge 0$ and $\\gamma_j \\ge {\\underline \\gamma}$ that\n        \\begin{equation}\\label{ineq:eta1}\n            \\min_{1\\le j \\le k}  \\eta_j  \\le \\frac{\\alpha_0}{{\\underline \\gamma} \\sum_{j=1}^k (1+\\sigma)^{j-1}} + \\delta.\n        \\end{equation}\n\tUsing the fact that $ 1+\\sigma \\ge e^{\\sigma/(1+\\sigma)} $\n\tfor every $\\sigma \\ge 0$, \n\twe have\n\t\\begin{equation}\\label{ineq:theta}\n\t    \\sum_{j=1}^{k} (1+\\sigma)^{j-1} = \\max \\left\\lbrace \\frac{(1+\\sigma)^k-1}{\\sigma}, k \\right\\rbrace \\ge \\max\\left\\lbrace \\frac{e^{\\sigma k /(1+\\sigma)} - 1}{\\sigma}, k\\right\\rbrace.\n\t\\end{equation}\n\tPlugging the above inequality into \\eqref{ineq:eta1}, we have for every $ k\\ge 1 $,\n\t\\[\n\t\\min_{1\\le j \\le k}  \\eta_j \\le \\frac{\\alpha_0}{{\\underline \\gamma}}\\min \\left\\lbrace \\frac{\\sigma}{e^{\\sigma k /(1+\\sigma)} - 1}, \\frac 1k \\right\\rbrace + \\delta,\n\t\\]\n\twhich can be easily seen to imply (b).\n\\end{proof}","sha256_utf8":"ec1489e5da204ebd0e27cb56bc2a8edd718e55e451bd79bf78a28f4a86f5d99b"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"加权求和得到望远镜界","source_units":[{"start":147858,"end":148474,"start_line":2895,"end_line":2903,"raw_tex":"\\begin{proof}\n\ta) Multiplying \\eqref{eq:easyrecur1} by $ (1+\\sigma)^{j-1} $ and summing the resulting inequality from $ j=1 $ to $ k $, we have\n\t\\begin{align}\n\t    \\sum_{j=1}^k (1+\\sigma)^{j-1} \\gamma_j \\left[ \\min_{1\\le j \\le k}  \\eta_j \\right] &\\le \\sum_{j=1}^k (1+\\sigma)^{j-1} \\gamma_j \\eta_j\n     \\le  \\sum_{j=1}^k (1+\\sigma)^{j-1} \\left(  \\alpha_{j-1} - (1+\\sigma) \\alpha_j + \\gamma_j \\delta \\right) \\nn \\\\\n\t\t &=  \\alpha_0 - (1+\\sigma)^k \\alpha_k +\n\t\t\\sum_{j=1}^k (1+\\sigma)^{j-1}\\gamma_j \\delta. \\label{ineq:sum}\n\t\\end{align}\n        Inequality \\eqref{ineq:eta} follows immediately from the above inequality.\n","sha256_utf8":"7a0620a90f8c4128bb7e8039525fa4a2c98121221b1e163f1c7fb84fa33fcb32"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"利用非负性简化误差界","source_units":[{"start":148474,"end":148898,"start_line":2904,"end_line":2910,"raw_tex":"\n        % b) This statement follows immediately from \\eqref{ineq:sum} and the fact that $\\eta_j\\ge 0$.\n        \n        b) It follows from \\eqref{ineq:eta}, and the facts that $\\alpha_k\\ge 0$ and $\\gamma_j \\ge {\\underline \\gamma}$ that\n        \\begin{equation}\\label{ineq:eta1}\n            \\min_{1\\le j \\le k}  \\eta_j  \\le \\frac{\\alpha_0}{{\\underline \\gamma} \\sum_{j=1}^k (1+\\sigma)^{j-1}} + \\delta.\n        \\end{equation}\n","sha256_utf8":"9b501470d554924e4d2e0df8b84731e6e6b2824500f600a4b9b9b82a6204b961"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"估计几何和并反解次数","source_units":[{"start":148898,"end":149542,"start_line":2911,"end_line":2922,"raw_tex":"\tUsing the fact that $ 1+\\sigma \\ge e^{\\sigma/(1+\\sigma)} $\n\tfor every $\\sigma \\ge 0$, \n\twe have\n\t\\begin{equation}\\label{ineq:theta}\n\t    \\sum_{j=1}^{k} (1+\\sigma)^{j-1} = \\max \\left\\lbrace \\frac{(1+\\sigma)^k-1}{\\sigma}, k \\right\\rbrace \\ge \\max\\left\\lbrace \\frac{e^{\\sigma k /(1+\\sigma)} - 1}{\\sigma}, k\\right\\rbrace.\n\t\\end{equation}\n\tPlugging the above inequality into \\eqref{ineq:eta1}, we have for every $ k\\ge 1 $,\n\t\\[\n\t\\min_{1\\le j \\le k}  \\eta_j \\le \\frac{\\alpha_0}{{\\underline \\gamma}}\\min \\left\\lbrace \\frac{\\sigma}{e^{\\sigma k /(1+\\sigma)} - 1}, \\frac 1k \\right\\rbrace + \\delta,\n\t\\]\n\twhich can be easily seen to imply (b).\n\\end{proof}","sha256_utf8":"d4625f33724a2e42bb2a97502753a2d48a5ed1c02b4142542e55515263670eec"}]},{"id":"S056","semantic_type":"lemma","kind":"lemma","title":"收缩系数性质","source_units":[{"start":149627,"end":150263,"start_line":2926,"end_line":2937,"raw_tex":"\\begin{lemma}\\label{lem:sigma}\n  For every $\\chi \\in [0,1)$, $\\lam>0$, $\\mu>0$, and $\\nu \\in [0,\\mu]$, the \n  quantity $\\sigma=\\sigma(\\mu,\\nu;\\lambda)$ defined in \\eqref{eq:sigma} satisfies the following statements:\n  \\begin{itemize}\n      \\item[a)] $\\lam [ \\chi  \\mu + (1-\\chi)  \\nu] \\le \\sigma \\le \\lam \\mu$, and hence $\\sigma \\ge \\lambda \\nu$;\n      \\item[b)] the function $\\lam >0 \\mapsto \\sigma(\\mu,\\nu;\\lambda)$ is non-decreasing;\n      \\item[c)] there holds\n      \\[\n      \\frac{1+\\sigma}{\\sigma} \\le \\min\\left\\{ \\frac{1}{\\chi}\\left(1+\\frac{1}{\\lam \\mu}\\right),  1+\\frac{1}{\\lam \\nu}\\right\\}.\n      \\]\n  \\end{itemize}\n\\end{lemma}","sha256_utf8":"cbdc948ad431091379887072c434f2105e148e1107ab1b90c8d4354dfc85d851"}]},{"id":"S057","semantic_type":"proof","kind":"proof","title":"收缩系数性质证明","source_units":[{"start":150265,"end":152072,"start_line":2939,"end_line":2975,"raw_tex":"\\begin{proof}\n    a) It follows from \\eqref{eq:sigma} and the fact that $\\nu \\le \\mu$ that\n \\[\n    \\sigma - \\lam \\mu =\\frac{\\lam(\\mu-\\nu)(1+\\lam \\mu)(\\chi-1)}{1+\\mu  \\lambda+\\chi  \\lambda(\\nu-\\mu)}\n    \\ge \\frac{\\lam(\\mu-\\nu)(1+\\lam \\mu)(\\chi-1)}{1+\\mu  \\lambda} =\n\\lam(\\mu-\\nu)(\\chi-1),\n    \\]\n    and hence that $\\sigma \\ge \\lam [ \\chi  \\mu + (1-\\chi)  \\nu]$. Moreover, the second inequality of (a) immediately follows from the first identity above and the facts that $\\chi<1$ and $\\nu \\le \\mu$.\n\n    b) Viewing $\\sigma=\\sigma(\\lam)$ in \\eqref{eq:sigma} as a function of $\\lam$, one can verify that\n\\[\n\\sigma(\\lam) = \\frac{\\nu(1+\\mu \\lam)+\\chi(\\mu-\\nu)}{\\lam^{-1}+\\mu +\\chi (\\nu-\\mu)}\n\\]\nis a non-decreasing function of $\\lam$.\n\n    c) It follows from the definition of $\\sigma$ in \\eqref{eq:sigma} that \n    \\[\n    \\frac{1+\\sigma}{\\sigma} = \\frac{(1+\\nu {\\lam})(1+\\mu {\\lam})}{{\\lam}[\\nu(1+ \\mu {\\lam})+\\chi(\\mu-\\nu)]}.\n    \\]\n    Hence, it suffices to prove that\n\\begin{equation}\\label{ineq:combine}\n    \\frac{(1+\\nu {\\lam})(1+\\mu {\\lam})}{{\\lam}[\\nu(1+ \\mu {\\lam})+\\chi(\\mu-\\nu)]} \\le \\min\\left\\{ \\frac{1}{\\chi}\\left(1+\\frac{1}{{\\lam} \\mu}\\right),  1+\\frac{1}{{\\lam} \\nu}\\right\\}. \n\\end{equation}\nSince $\\chi \\in [0,1)$, it is easy to verify that\n\\[\n\\frac{1+\\nu {\\lam}}{\\nu(1+ \\mu {\\lam})+\\chi(\\mu-\\nu)} \\le \\frac{1}{\\chi \\mu},\n\\]\nand hence that\n\\[\n\\frac{(1+\\nu {\\lam})(1+\\mu {\\lam})}{{\\lam}[\\nu(1+ \\mu {\\lam})+\\chi(\\mu-\\nu)]} \\le \\frac{1+ \\mu {\\lam}}{\\chi {\\lam} \\mu} = \\frac{1}{\\chi}\\left(1+\\frac{1}{{\\lam} \\mu}\\right).\n\\]\nMoreover, noting that $\\chi \\in [0,1)$ and $\\mu\\ge \\nu$, we also have \n\\[\n\\frac{(1+\\nu {\\lam})(1+\\mu {\\lam})}{{\\lam}[\\nu(1+ \\mu {\\lam})+\\chi(\\mu-\\nu)]} \\le 1+\\frac{1}{{\\lam} \\nu}.\n\\]\nCombining the above two inequalities, we conclude that \\eqref{ineq:combine} holds.\n\\end{proof}","sha256_utf8":"04615823d0a6e6217c855bf17a8d3fe06d5a42e1519394e86a1a93b713266914"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"证明收缩系数上下界","source_units":[{"start":150265,"end":150763,"start_line":2939,"end_line":2946,"raw_tex":"\\begin{proof}\n    a) It follows from \\eqref{eq:sigma} and the fact that $\\nu \\le \\mu$ that\n \\[\n    \\sigma - \\lam \\mu =\\frac{\\lam(\\mu-\\nu)(1+\\lam \\mu)(\\chi-1)}{1+\\mu  \\lambda+\\chi  \\lambda(\\nu-\\mu)}\n    \\ge \\frac{\\lam(\\mu-\\nu)(1+\\lam \\mu)(\\chi-1)}{1+\\mu  \\lambda} =\n\\lam(\\mu-\\nu)(\\chi-1),\n    \\]\n    and hence that $\\sigma \\ge \\lam [ \\chi  \\mu + (1-\\chi)  \\nu]$. Moreover, the second inequality of (a) immediately follows from the first identity above and the facts that $\\chi<1$ and $\\nu \\le \\mu$.\n","sha256_utf8":"d6b85a08b46ef037efc064a362785cf5e6c453ffa445cc68281d5b2b9cfcbf14"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"证明步长单调性","source_units":[{"start":150763,"end":150995,"start_line":2947,"end_line":2952,"raw_tex":"\n    b) Viewing $\\sigma=\\sigma(\\lam)$ in \\eqref{eq:sigma} as a function of $\\lam$, one can verify that\n\\[\n\\sigma(\\lam) = \\frac{\\nu(1+\\mu \\lam)+\\chi(\\mu-\\nu)}{\\lam^{-1}+\\mu +\\chi (\\nu-\\mu)}\n\\]\nis a non-decreasing function of $\\lam$.\n","sha256_utf8":"8b9e646b9b1e98c072dbe89ea352d485211bf8f879f90795bab5fc22f2ccfefe"}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","title":"展开收缩比率","source_units":[{"start":150995,"end":151467,"start_line":2953,"end_line":2961,"raw_tex":"\n    c) It follows from the definition of $\\sigma$ in \\eqref{eq:sigma} that \n    \\[\n    \\frac{1+\\sigma}{\\sigma} = \\frac{(1+\\nu {\\lam})(1+\\mu {\\lam})}{{\\lam}[\\nu(1+ \\mu {\\lam})+\\chi(\\mu-\\nu)]}.\n    \\]\n    Hence, it suffices to prove that\n\\begin{equation}\\label{ineq:combine}\n    \\frac{(1+\\nu {\\lam})(1+\\mu {\\lam})}{{\\lam}[\\nu(1+ \\mu {\\lam})+\\chi(\\mu-\\nu)]} \\le \\min\\left\\{ \\frac{1}{\\chi}\\left(1+\\frac{1}{{\\lam} \\mu}\\right),  1+\\frac{1}{{\\lam} \\nu}\\right\\}. \n\\end{equation}\n","sha256_utf8":"756921e3e5eba9ff9a9f7dd877045e438e36c08ab8efdc92a86c301d2c1fe1ed"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"建立目标强凸参数界","source_units":[{"start":151467,"end":151795,"start_line":2962,"end_line":2969,"raw_tex":"Since $\\chi \\in [0,1)$, it is easy to verify that\n\\[\n\\frac{1+\\nu {\\lam}}{\\nu(1+ \\mu {\\lam})+\\chi(\\mu-\\nu)} \\le \\frac{1}{\\chi \\mu},\n\\]\nand hence that\n\\[\n\\frac{(1+\\nu {\\lam})(1+\\mu {\\lam})}{{\\lam}[\\nu(1+ \\mu {\\lam})+\\chi(\\mu-\\nu)]} \\le \\frac{1+ \\mu {\\lam}}{\\chi {\\lam} \\mu} = \\frac{1}{\\chi}\\left(1+\\frac{1}{{\\lam} \\mu}\\right).\n\\]\n","sha256_utf8":"fdb551ff3a3a1ed5c99d7e263c0cfb27160cff01ca12e3a6bc5ac58b64958556"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"建立模型强凸界并合并","source_units":[{"start":151795,"end":152072,"start_line":2970,"end_line":2975,"raw_tex":"Moreover, noting that $\\chi \\in [0,1)$ and $\\mu\\ge \\nu$, we also have \n\\[\n\\frac{(1+\\nu {\\lam})(1+\\mu {\\lam})}{{\\lam}[\\nu(1+ \\mu {\\lam})+\\chi(\\mu-\\nu)]} \\le 1+\\frac{1}{{\\lam} \\nu}.\n\\]\nCombining the above two inequalities, we conclude that \\eqref{ineq:combine} holds.\n\\end{proof}","sha256_utf8":"f25c2adb6937ef39baf748492414683732e282010f801b278d4832a9392f347f"}]},{"id":"S063","semantic_type":"external_fact","kind":"external_fact","title":"引文中的强凸次梯度界","source_units":[],"declared_external":true,"declaration_text":["melo2023proximal 的 Lemma A.1：若 ψ−(ξ/2)||·||_Q² 凸且 v∈∂_ηψ(y)，则对 τ>0 有 ψ(u)≥ψ(y)+⟨v,u−y⟩−(1+τ⁻¹)η+ξ||u−y||_Q²/[2(1+τ)]。这里 Q 为对称正定矩阵，||z||_Q²=zᵀQz。"]},{"id":"S064","semantic_type":"external_fact","kind":"external_fact","title":"凸近端最优性与二次平移","source_units":[],"declared_external":true,"declaration_text":["若 x 最小化 Γ(u)+||u−a||²/(2λ)，其中 Γ 为正常闭凸函数且 λ>0，则 (a−x)/λ∈∂Γ(x)。若 Γ 为 ν-强凸且 s∈∂Γ(x)，则 s−ν(x−a)∈∂[Γ−ν||·−a||²/2](x)。"]},{"id":"S065","semantic_type":"external_fact","kind":"external_fact","title":"配方与加权距离不等式","source_units":[],"declared_external":true,"declaration_text":["a>0 时，2MΔ−aΔ²≤M²/a。正权 b₁,b₂ 下，(a₁+a₂)²≤(b₁⁻¹+b₂⁻¹)(b₁a₁²+b₂a₂²)；欧氏范数满足三角不等式。"]},{"id":"S066","semantic_type":"external_fact","kind":"external_fact","title":"几何和的指数下界","source_units":[],"declared_external":true,"declaration_text":["对 σ≥0，有 1+σ≥exp(σ/(1+σ))；对 σ>0，有 Σ_{j=1}^k(1+σ)^{j−1}=((1+σ)^k−1)/σ，并且该和至少为 k；σ=0 时该和等于 k。"]},{"id":"S067","semantic_type":"notation","kind":"notation","title":"收缩系数定义","source_units":[{"start":88341,"end":88512,"start_line":1795,"end_line":1798,"raw_tex":"where \n\\begin{equation}\\label{eq:sigma}\n    \\sigma = \\sigma(\\mu,\\nu) := \\frac{\\lam[\\nu(1+\\mu \\lam)+\\chi(\\mu-\\nu)]}{1+\\mu  \\lambda+\\chi  \\lambda(\\nu-\\mu)}.\n\\end{equation} \n","sha256_utf8":"317a2abbe081a4b456151589acd4e65246e97c51af803b63a5eeae4e3bae33b6"}]},{"id":"S068","semantic_type":"notation","kind":"notation","title":"残差向量与误差定义","source_units":[{"start":110300,"end":110448,"start_line":2208,"end_line":2212,"raw_tex":"Define \n \\begin{align}\n\t s &:= \\frac{x^- - x}{\\lam}, \\label{defsb}\\\\\n\t \\eta &:= \\phi(y) - \\Gamma(x) - \\inner{s}{y- x}. \\label{defetab}\n\t\\end{align}\n","sha256_utf8":"19f622c7782c229d4b0dc44f1aac37d31320bcdada3a9ad66f1bef258ac3e745"}]},{"id":"S069","semantic_type":"assumption","kind":"assumption","title":"递推引理局部假设","source_units":[{"start":146404,"end":146766,"start_line":2870,"end_line":2874,"raw_tex":"\t\tAssume that sequences $\\{\\gamma_j\\}$, $\\{\\eta_j\\}$, and  $\\{\\alpha_j\\}$ satisfy for every $j\\ge 1$, $\\gamma_j \\ge {\\underline \\gamma}$ and \n\t\t\\beq \\label{eq:easyrecur1}\n\t\t\\gamma_j \\eta_j \\le \\alpha_{j-1} - (1+\\sigma)\\alpha_j + \\gamma_j \\delta\n\t\t\\eeq\n  for some $\\sigma \\ge 0$, $\\delta \\ge 0$ and ${\\underline \\gamma} > 0$. Then, the following statements hold:\n","sha256_utf8":"9ec85dfc09dab1763a055167fe3f5a447d108385dd1c506b8ab0acea47e48dfc"}]},{"id":"S070","semantic_type":"assumption","kind":"assumption","title":"收缩系数参数范围","source_units":[{"start":149658,"end":149843,"start_line":2927,"end_line":2928,"raw_tex":"  For every $\\chi \\in [0,1)$, $\\lam>0$, $\\mu>0$, and $\\nu \\in [0,\\mu]$, the \n  quantity $\\sigma=\\sigma(\\mu,\\nu;\\lambda)$ defined in \\eqref{eq:sigma} satisfies the following statements:\n","sha256_utf8":"01b64be7de971996e12dc816a5151867169735378a55b5120daea6dad44bdd7a"}]},{"id":"S071","semantic_type":"external_fact","kind":"external_fact","title":"混合条件的标准误差界","source_units":[],"declared_external":true,"declaration_text":["凸函数的次梯度选择若满足 ||f′(x)−f′(y)||≤2M_f+L_f||x−y||，则在该凸域内 f(x)−f(y)−⟨f′(y),x−y⟩≤2M_f||x−y||+(L_f/2)||x−y||²；这是原文776–779作为已知事实使用的估计。"]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S006","target_id":"S014","kind":"supports"},{"source_id":"S006","target_id":"S015","kind":"supports"},{"source_id":"S006","target_id":"S004","kind":"supports"},{"source_id":"S007","target_id":"S022","kind":"supports"},{"source_id":"S007","target_id":"S037","kind":"supports"},{"source_id":"S007","target_id":"S055","kind":"supports"},{"source_id":"S008","target_id":"S040","kind":"supports"},{"source_id":"S008","target_id":"S045","kind":"supports"},{"source_id":"S008","target_id":"S004","kind":"supports"},{"source_id":"S009","target_id":"S040","kind":"supports"},{"source_id":"S009","target_id":"S041","kind":"supports"},{"source_id":"S009","target_id":"S045","kind":"supports"},{"source_id":"S009","target_id":"S046","kind":"supports"},{"source_id":"S010","target_id":"S022","kind":"supports"},{"source_id":"S010","target_id":"S027","kind":"supports"},{"source_id":"S010","target_id":"S004","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S027","kind":"supports"},{"source_id":"S011","target_id":"S031","kind":"supports"},{"source_id":"S012","target_id":"S022","kind":"supports"},{"source_id":"S013","target_id":"S004","kind":"supports"},{"source_id":"S013","target_id":"S030","kind":"supports"},{"source_id":"S013","target_id":"S045","kind":"supports"},{"source_id":"S013","target_id":"S058","kind":"supports"},{"source_id":"S014","target_id":"S031","kind":"supports"},{"source_id":"S014","target_id":"S005","kind":"supports"},{"source_id":"S015","target_id":"S025","kind":"supports"},{"source_id":"S015","target_id":"S026","kind":"supports"},{"source_id":"S015","target_id":"S027","kind":"supports"},{"source_id":"S015","target_id":"S003","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S031","kind":"supports"},{"source_id":"S017","target_id":"S040","kind":"supports"},{"source_id":"S017","target_id":"S041","kind":"supports"},{"source_id":"S017","target_id":"S042","kind":"supports"},{"source_id":"S017","target_id":"S027","kind":"supports"},{"source_id":"S017","target_id":"S030","kind":"supports"},{"source_id":"S018","target_id":"S027","kind":"supports"},{"source_id":"S018","target_id":"S030","kind":"supports"},{"source_id":"S018","target_id":"S032","kind":"supports"},{"source_id":"S019","target_id":"S027","kind":"supports"},{"source_id":"S019","target_id":"S030","kind":"supports"},{"source_id":"S019","target_id":"S031","kind":"supports"},{"source_id":"S020","target_id":"S026","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S023","target_id":"S003","kind":"supports"},{"source_id":"S023","target_id":"S004","kind":"supports"},{"source_id":"S023","target_id":"S005","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S027","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S028","target_id":"S004","kind":"supports"},{"source_id":"S028","target_id":"S005","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S032","target_id":"S029","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S033","target_id":"S030","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S034","kind":"supports"},{"source_id":"S037","target_id":"S034","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S038","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S045","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S041","target_id":"S039","kind":"supports"},{"source_id":"S042","target_id":"S039","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S043","target_id":"S035","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S044","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S048","target_id":"S046","kind":"supports"},{"source_id":"S049","target_id":"S048","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S051","target_id":"S032","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S055","target_id":"S052","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S056","target_id":"S030","kind":"supports"},{"source_id":"S056","target_id":"S032","kind":"supports"},{"source_id":"S057","target_id":"S056","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S059","target_id":"S057","kind":"supports"},{"source_id":"S060","target_id":"S057","kind":"supports"},{"source_id":"S061","target_id":"S057","kind":"supports"},{"source_id":"S062","target_id":"S057","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S060","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S063","target_id":"S050","kind":"supports"},{"source_id":"S064","target_id":"S040","kind":"supports"},{"source_id":"S064","target_id":"S045","kind":"supports"},{"source_id":"S065","target_id":"S022","kind":"supports"},{"source_id":"S065","target_id":"S037","kind":"supports"},{"source_id":"S066","target_id":"S055","kind":"supports"},{"source_id":"S067","target_id":"S030","kind":"supports"},{"source_id":"S067","target_id":"S037","kind":"supports"},{"source_id":"S067","target_id":"S058","kind":"supports"},{"source_id":"S067","target_id":"S059","kind":"supports"},{"source_id":"S067","target_id":"S060","kind":"supports"},{"source_id":"S068","target_id":"S040","kind":"supports"},{"source_id":"S068","target_id":"S041","kind":"supports"},{"source_id":"S068","target_id":"S042","kind":"supports"},{"source_id":"S068","target_id":"S045","kind":"supports"},{"source_id":"S068","target_id":"S035","kind":"supports"},{"source_id":"S069","target_id":"S053","kind":"supports"},{"source_id":"S069","target_id":"S054","kind":"supports"},{"source_id":"S070","target_id":"S058","kind":"supports"},{"source_id":"S070","target_id":"S059","kind":"supports"},{"source_id":"S070","target_id":"S060","kind":"supports"},{"source_id":"S070","target_id":"S061","kind":"supports"},{"source_id":"S070","target_id":"S062","kind":"supports"},{"source_id":"S071","target_id":"S012","kind":"supports"}]}}
{"record_id":"2408.05021v1","paper":{"title":"Two-norm discrepancy and convergence of the stochastic gradient method with application to shape optimization","authors":["Marc Dambrine","Caroline Geiersbach","Helmut Harbrecht"],"arxiv_id":"2408.05021","version":"v1","journal":"SIAM Journal on Optimization","doi":"10.1137/24m168372x","abstract_url":"https://arxiv.org/abs/2408.05021v1","pdf_url":"https://arxiv.org/pdf/2408.05021v1"},"context":{"raw_tex":"the free boundary problem \\eqref{bvp}--\\eqref{flux}. Let $X\\subset H$\nbe two Hilbert spaces, which are dense in $L^2$, %\\CG{for the convergence theory, I do not seem to need that these spaces dense in $L^2$. Is that unexpected? If it is not needed, it would appear to be more general to mention of $L^2$.} \n%\\HH{Die Stetigkeits- und Elliptizit\\\"atsabsch\\\"atzungen brauchen meines Erachtens den Gelfand-Dreier. Wir benutzen ja, dass der  Hessian eine Bilinear-Form auf $H^{1/2}\\times H^{1/2}$ definiert, also ein Operator $H^{1/2}\\to H^{-1/2}$ ist. Mir w\\\"are es also wohler, wenn wir uns hier einschr\\\"anken wegen der zugrundeliegenden PDE-Theorie.}\nendowed with the \ninner products $( \\cdot,\\cdot)_X$ and $(\\cdot, \\cdot)_H$, respectively, \nand corresponding norms $\\lVert \\cdot\\rVert_X$ and $\\lVert \\cdot \\rVert_H$. \nThe respective dual spaces are denoted by $X^*$ and $H^*$, which yields \nthe Gelfand chain $X \\subset H\\subset L^2 \\subset H^*\\subset X^*$.\nA ball centered at $r$ in a space $X'$ is denoted by $B_{\\delta}^{X'}(r) \n= \\{ h \\in X' \\mid \\lVert h-r\\rVert_{X'}< \\delta \\}$. We assume that \n$X \\subset H$ with continuous embedding and that $X_{ad} \\subset X$ \nis a bounded, closed, convex, and nonempty admissible set. \n\nLet $(\\Omega, \\mathcal{F}, \\pP)$ be a complete probability space and \n$\\xi \\colon \\Omega \\to\\Xi$ be a random function mapping to a (real) \ncomplete separable metric space $\\Xi$. Now, consider the problem \n\\begin{equation}\\label{eq:SAproblem}\n%======================================================\n\\underset{h \\in X_{ad}}{\\textup{minimize}} \\quad \\{j(h) = \\E[\\mathcal{J}(h,\\xi)] \\},\n\\end{equation}\n\nwhere %, $J \\colon X \\times \\Xi \\to \\R$, \nwe assume that $j \\colon O \\subset X \\rightarrow \\R$ is twice continuously differentiable on an open set $O \\supset X_{ad}$. A point $h^*$ is said to be a \\textit{local solution} of \\eqref{eq:SAproblem} in $X'$ if $j(h^*) \\leq j(h)$ for all $h \\in X_{ad} \\cap B_{\\delta}^{X'}(h^*)$ for some $\\delta >0$. A necessary condition for $h^*\\in X_{ad}$ to be a local solution (in $X$) to \n\\eqref{eq:SAproblem} is given by the variational inequality \n%See e.g. Troeltzsch, Lemma 2.21\n\\begin{equation}\\label{eq:OC}\n%======================================================\nD j(h^*)[h-h^*] \\geq 0 \\quad \\forall h \\in X_{ad}.\n\\end{equation}\n\nIn the event that $j$ is convex on $X_{ad}$, this condition is \nalso sufficient and $h^*$ is even a global solution. Notice that since $H$ is the weaker space, any local solution of \\eqref{eq:SAproblem} in $H$ is also a local solution in $X$; i.e., local solutions in $H$ also satisfy the condition \\eqref{eq:OC}. Our strategy of handling the two-norm discrepancy will be to show that our method converges in $H$ to a local solution.\n\n\n\\begin{assumption}\\label{assumption:j}\n%======================================================\nThe functional $j \\colon O\\to\\R$ is twice continuously \ndifferentiable. We assume that for every $h \\in X_{ad},$ %there exists $\\overline{r} \\in X$ \n%satisfying \\eqref{eq:OC} such that for all $r$ on a \n%neighborhood $\\overline{B_{\\delta}^X(\\overline{r})}$ of \n%$\\overline{r}$, \nthe second derivative $D^2 j(h)\\in \n\\mathcal{L}(X,X^*)$ extends continuously to a bilinear \nform on $H\\times H$, i.e., there exists $C_S >0$ such that \n\\begin{equation}\\label{eq:assumption-continuity-bilinear-form}\n%======================================================================\n    |D^2 j(h)[q_1,q_2]| \\leq C_S \\lVert q_1 \\rVert_H \\lVert q_2 \\rVert_H  \n    \\ \\forall q_1, q_2 \\in H.\n\\end{equation}\nAdditionally, we assume that for there exists $c_E >0$ such that \nthe following strong coercivity condition is satisfied for every $h \\in X_{ad}$:\n\\begin{equation}\n    \\label{eq:strong-coercivity} \n    D^2j(h)[q,q] \\ge c_E \\lVert q\\rVert_H^2 \n    \\quad \\forall q \\in H.\n\\end{equation}\n\\end{assumption}\n\nLet $\\pi_{X_{ad}} \\colon X \\rightarrow X_{ad}$ be a projection onto the set $X_{ad}$, defined by\n\\[\n\\pi_{X_{ad}}(h) = \\argmin_{w \\in X_{ad}} \\, \\lVert h-w\\rVert_X, \n\\]\n\nwhich is well-defined and single-valued since $X_{ad} \\subset X$ is assumed to be nonempty, closed, and convex.   We assume that it is possible to compute an approximation of the gradient in the form of a \\textit{stochastic gradient} $G\\colon X \\times \\Xi \\rightarrow X$, which is defined as the (parameterized) Riesz representative of the mapping $D\\mathcal{J}(\\cdot,\\hat{\\xi}) \\colon X \\rightarrow X^*$, i.e., we have for every $\\hat{\\xi} \\in \\Xi$ that\n\\begin{equation}\n\\label{eq:grad_projection}\n%==========================================\n(G(h,\\hat{\\xi}),q)_X = \\langle D\\mathcal{J}(h,\\hat{\\xi}),q\\rangle_{X^*,X} \\quad \\forall (h,q) \\in X_{ad}\\times X.\n\\end{equation}\n\nWe use the notation $\\nabla j$ for the  gradient of $j$ in $X$, i.e., $(\\nabla j(h),q)_{X} = \\langle Dj(h),q\\rangle_{X^*,X}$, where $h,q \\in X$. \nThe projected stochastic gradient method relies on a recursion of the form\n\\begin{equation}\n\\label{eq:SGM-recursion}\nh_{n+1} := \\pi_{X_{ad}}(h_n - t_n G(h_n,\\xi_n)),\n\\end{equation}\n\nwhere $h_1 \\in X_{ad}$ and $\\xi_n$ is randomly sampled from the law $\\mathbb{P}\\circ\\xi^{-1}$ independently of previous samples $\\xi_1, \\dots, \\xi_{n-1}.$ We require that the step sizes given in \\eqref{eq:SGM-recursion} satisfy the Robbins--Monro rule from the original paper \\cite{Robbins1951} on stochastic approximation:\n\n\n\\begin{equation}\\label{eq:RobbinsMonroStepSize}\nt_n \\geq 0, \\quad \\sum_{n=1}^\\infty t_n = \\infty, \\quad  \\sum_{n=1}^\\infty t_n^2 < \\infty.\n \\end{equation}\n\nFirst, we recall some concepts that will be of use in the proof. A filtration is a sequence $\\{ \\mathcal{F}_n\\}$ of sub-$\\sigma$-algebras of $\\mathcal{F}$ such that {$\\mathcal{F}_1 \\subset \\mathcal{F}_2 \\subset \\cdots \\subset \\mathcal{F}.$} \nGiven a Banach space $Y$, we define a discrete $Y$-valued stochastic process as a collection of $Y$-valued random variables indexed by $n$, in other words, the set $\\{ \\beta_n \\mid \\Omega \\rightarrow Y \\mid n \\in \\mathbb{N}\\}.$ \nThe stochastic process is said to be adapted to a filtration $\\{ \\mathcal{F}_n \\}$ if and only if $\\beta_n$ is $\\mathcal{F}_n$-measurable for all $n$. Suppose $\\mathcal{B}(Y)$ denotes the set of Borel sets of $Y$.\nThe natural filtration is the filtration generated by the sequence $\\{\\beta_n\\}$ and is given by $\\mathcal{F}_n = \\sigma(\\{\\beta_1, \\dots ,\\beta_n\\}) = \\{ \\beta_i^{-1}(B)\\mid B \\in \\mathcal{B}(Y), i = 1, \\dots, n\\}$.\nIf for an event $F \\in \\mathcal{F}$ we have that $\\mathbb{P}(F) = 1$, we say $F$ occurs almost surely (a.s.).\n%A sequence of random variables $\\{\\beta_n\\}$ is said to converge almost surely to a random variable $\\beta$ if and only if %see Gut2013, Def. 4.4 $P\\left(\\left\\lbrace \\omega \\in \\Omega: \\lim_{n \\rightarrow \\infty} \\beta_n(\\omega) = \\beta(\\omega) \\right\\rbrace\\right) = 1.$\nFor an integrable random variable $\\beta\\colon \\Omega \\rightarrow \\R$, the conditional expectation is denoted by $\\E[\\beta | \\mathcal{F}_n]$, which is itself a random variable that is $\\mathcal{F}_n$-measurable and which satisfies $\\int_A \\E[\\beta | \\mathcal{F}_n](\\omega) \\D \\pP(\\omega) = \\int_A \\beta(\\omega) \\D \\pP(\\omega)$ for all $A \\in \\mathcal{F}_n$.  \n\n\n\\begin{assumption}\\label{assumption:gradient}\nLet $\\{\\mathcal{F}_n \\}$ be an increasing sequence of $\\sigma$-algebras.  For each $n$, there exist $b_n$, $w_n$ with\n$$b_n = \\E[G(h_n,\\xi_n) | \\mathcal{F}_n] - \\nabla j(h_n), \\quad w_n = G(h_n, \\xi_n) - \\E[G(h_n, \\xi_n) | \\mathcal{F}_n],$$ \nwhich satisfy the following assumptions:\\\\ (i) $ h_n$ and $b_n $ are $\\mathcal{F}_n$-measurable; (ii) for $K_n:=\\esssup_{\\omega \\in \\Omega} \\lVert b_n(\\omega)\\rVert_X$ we have that \\mbox{$ \\sum_{n=1}^\\infty t_n K_n< \\infty$} and \\mbox{$\\sup_n K_n < \\infty$}; (iii) there exists $M\\geq 0$ such that $\\E[\\lVert G(h,\\xi)\\rVert_X^2] \\leq M$ for all $r \\in X_{ad}$.\n\\end{assumption}","units":[{"start":60425,"end":62054,"start_line":1258,"end_line":1277,"raw_tex":"the free boundary problem \\eqref{bvp}--\\eqref{flux}. Let $X\\subset H$\nbe two Hilbert spaces, which are dense in $L^2$, %\\CG{for the convergence theory, I do not seem to need that these spaces dense in $L^2$. Is that unexpected? If it is not needed, it would appear to be more general to mention of $L^2$.} \n%\\HH{Die Stetigkeits- und Elliptizit\\\"atsabsch\\\"atzungen brauchen meines Erachtens den Gelfand-Dreier. Wir benutzen ja, dass der  Hessian eine Bilinear-Form auf $H^{1/2}\\times H^{1/2}$ definiert, also ein Operator $H^{1/2}\\to H^{-1/2}$ ist. Mir w\\\"are es also wohler, wenn wir uns hier einschr\\\"anken wegen der zugrundeliegenden PDE-Theorie.}\nendowed with the \ninner products $( \\cdot,\\cdot)_X$ and $(\\cdot, \\cdot)_H$, respectively, \nand corresponding norms $\\lVert \\cdot\\rVert_X$ and $\\lVert \\cdot \\rVert_H$. \nThe respective dual spaces are denoted by $X^*$ and $H^*$, which yields \nthe Gelfand chain $X \\subset H\\subset L^2 \\subset H^*\\subset X^*$.\nA ball centered at $r$ in a space $X'$ is denoted by $B_{\\delta}^{X'}(r) \n= \\{ h \\in X' \\mid \\lVert h-r\\rVert_{X'}< \\delta \\}$. We assume that \n$X \\subset H$ with continuous embedding and that $X_{ad} \\subset X$ \nis a bounded, closed, convex, and nonempty admissible set. \n\nLet $(\\Omega, \\mathcal{F}, \\pP)$ be a complete probability space and \n$\\xi \\colon \\Omega \\to\\Xi$ be a random function mapping to a (real) \ncomplete separable metric space $\\Xi$. Now, consider the problem \n\\begin{equation}\\label{eq:SAproblem}\n%======================================================\n\\underset{h \\in X_{ad}}{\\textup{minimize}} \\quad \\{j(h) = \\E[\\mathcal{J}(h,\\xi)] \\},\n\\end{equation}","sha256_utf8":"967f6fb9c941fa06b5ed9585aff186bb4349ff1b0c946d649a0f6682f76a1b59"},{"start":62055,"end":62724,"start_line":1278,"end_line":1285,"raw_tex":"where %, $J \\colon X \\times \\Xi \\to \\R$, \nwe assume that $j \\colon O \\subset X \\rightarrow \\R$ is twice continuously differentiable on an open set $O \\supset X_{ad}$. A point $h^*$ is said to be a \\textit{local solution} of \\eqref{eq:SAproblem} in $X'$ if $j(h^*) \\leq j(h)$ for all $h \\in X_{ad} \\cap B_{\\delta}^{X'}(h^*)$ for some $\\delta >0$. A necessary condition for $h^*\\in X_{ad}$ to be a local solution (in $X$) to \n\\eqref{eq:SAproblem} is given by the variational inequality \n%See e.g. Troeltzsch, Lemma 2.21\n\\begin{equation}\\label{eq:OC}\n%======================================================\nD j(h^*)[h-h^*] \\geq 0 \\quad \\forall h \\in X_{ad}.\n\\end{equation}","sha256_utf8":"7ab11416914a0bd4cd5c623361846acfaaaf9d4a2800d2f0f62d9e9d95e21b1d"},{"start":62725,"end":63159,"start_line":1286,"end_line":1287,"raw_tex":"In the event that $j$ is convex on $X_{ad}$, this condition is \nalso sufficient and $h^*$ is even a global solution. Notice that since $H$ is the weaker space, any local solution of \\eqref{eq:SAproblem} in $H$ is also a local solution in $X$; i.e., local solutions in $H$ also satisfy the condition \\eqref{eq:OC}. Our strategy of handling the two-norm discrepancy will be to show that our method converges in $H$ to a local solution.\n","sha256_utf8":"3a6da420341d7dd061cde075b40e210bac854fe9555a1d61a023c37fa08ad7f7"},{"start":63395,"end":64477,"start_line":1293,"end_line":1315,"raw_tex":"\\begin{assumption}\\label{assumption:j}\n%======================================================\nThe functional $j \\colon O\\to\\R$ is twice continuously \ndifferentiable. We assume that for every $h \\in X_{ad},$ %there exists $\\overline{r} \\in X$ \n%satisfying \\eqref{eq:OC} such that for all $r$ on a \n%neighborhood $\\overline{B_{\\delta}^X(\\overline{r})}$ of \n%$\\overline{r}$, \nthe second derivative $D^2 j(h)\\in \n\\mathcal{L}(X,X^*)$ extends continuously to a bilinear \nform on $H\\times H$, i.e., there exists $C_S >0$ such that \n\\begin{equation}\\label{eq:assumption-continuity-bilinear-form}\n%======================================================================\n    |D^2 j(h)[q_1,q_2]| \\leq C_S \\lVert q_1 \\rVert_H \\lVert q_2 \\rVert_H  \n    \\ \\forall q_1, q_2 \\in H.\n\\end{equation}\nAdditionally, we assume that for there exists $c_E >0$ such that \nthe following strong coercivity condition is satisfied for every $h \\in X_{ad}$:\n\\begin{equation}\n    \\label{eq:strong-coercivity} \n    D^2j(h)[q,q] \\ge c_E \\lVert q\\rVert_H^2 \n    \\quad \\forall q \\in H.\n\\end{equation}\n\\end{assumption}","sha256_utf8":"433e39e9e48477150c16af35dd9426bd96ad1a742058d16d4c899125f9095b96"},{"start":70112,"end":70279,"start_line":1402,"end_line":1405,"raw_tex":"Let $\\pi_{X_{ad}} \\colon X \\rightarrow X_{ad}$ be a projection onto the set $X_{ad}$, defined by\n\\[\n\\pi_{X_{ad}}(h) = \\argmin_{w \\in X_{ad}} \\, \\lVert h-w\\rVert_X, \n\\]","sha256_utf8":"0256f2b89914414603af56bd9ae2b7b2b97d0beff63f93918748483bff313c63"},{"start":70280,"end":70951,"start_line":1406,"end_line":1411,"raw_tex":"which is well-defined and single-valued since $X_{ad} \\subset X$ is assumed to be nonempty, closed, and convex.   We assume that it is possible to compute an approximation of the gradient in the form of a \\textit{stochastic gradient} $G\\colon X \\times \\Xi \\rightarrow X$, which is defined as the (parameterized) Riesz representative of the mapping $D\\mathcal{J}(\\cdot,\\hat{\\xi}) \\colon X \\rightarrow X^*$, i.e., we have for every $\\hat{\\xi} \\in \\Xi$ that\n\\begin{equation}\n\\label{eq:grad_projection}\n%==========================================\n(G(h,\\hat{\\xi}),q)_X = \\langle D\\mathcal{J}(h,\\hat{\\xi}),q\\rangle_{X^*,X} \\quad \\forall (h,q) \\in X_{ad}\\times X.\n\\end{equation}","sha256_utf8":"460d4ebdbd33dd25eb5672ef813ab0f7ff9c86e0e77b449f68325a87787d9db7"},{"start":71060,"end":71386,"start_line":1413,"end_line":1418,"raw_tex":"We use the notation $\\nabla j$ for the  gradient of $j$ in $X$, i.e., $(\\nabla j(h),q)_{X} = \\langle Dj(h),q\\rangle_{X^*,X}$, where $h,q \\in X$. \nThe projected stochastic gradient method relies on a recursion of the form\n\\begin{equation}\n\\label{eq:SGM-recursion}\nh_{n+1} := \\pi_{X_{ad}}(h_n - t_n G(h_n,\\xi_n)),\n\\end{equation}","sha256_utf8":"2165fec43fe8e4808f6fb67f795996fc8a13250760040346feb4c7e7f1493668"},{"start":71387,"end":71711,"start_line":1419,"end_line":1419,"raw_tex":"where $h_1 \\in X_{ad}$ and $\\xi_n$ is randomly sampled from the law $\\mathbb{P}\\circ\\xi^{-1}$ independently of previous samples $\\xi_1, \\dots, \\xi_{n-1}.$ We require that the step sizes given in \\eqref{eq:SGM-recursion} satisfy the Robbins--Monro rule from the original paper \\cite{Robbins1951} on stochastic approximation:\n","sha256_utf8":"5d57999f39915d761962d06a8abd922f003c0c915ef26f6e19e326bdb079f078"},{"start":71712,"end":71866,"start_line":1420,"end_line":1422,"raw_tex":"\\begin{equation}\\label{eq:RobbinsMonroStepSize}\nt_n \\geq 0, \\quad \\sum_{n=1}^\\infty t_n = \\infty, \\quad  \\sum_{n=1}^\\infty t_n^2 < \\infty.\n \\end{equation}","sha256_utf8":"cf621fe1928a3ebf003921450fc240f4d124fca8cee538482ed4477144a0acf9"},{"start":72364,"end":74010,"start_line":1426,"end_line":1432,"raw_tex":"First, we recall some concepts that will be of use in the proof. A filtration is a sequence $\\{ \\mathcal{F}_n\\}$ of sub-$\\sigma$-algebras of $\\mathcal{F}$ such that {$\\mathcal{F}_1 \\subset \\mathcal{F}_2 \\subset \\cdots \\subset \\mathcal{F}.$} \nGiven a Banach space $Y$, we define a discrete $Y$-valued stochastic process as a collection of $Y$-valued random variables indexed by $n$, in other words, the set $\\{ \\beta_n \\mid \\Omega \\rightarrow Y \\mid n \\in \\mathbb{N}\\}.$ \nThe stochastic process is said to be adapted to a filtration $\\{ \\mathcal{F}_n \\}$ if and only if $\\beta_n$ is $\\mathcal{F}_n$-measurable for all $n$. Suppose $\\mathcal{B}(Y)$ denotes the set of Borel sets of $Y$.\nThe natural filtration is the filtration generated by the sequence $\\{\\beta_n\\}$ and is given by $\\mathcal{F}_n = \\sigma(\\{\\beta_1, \\dots ,\\beta_n\\}) = \\{ \\beta_i^{-1}(B)\\mid B \\in \\mathcal{B}(Y), i = 1, \\dots, n\\}$.\nIf for an event $F \\in \\mathcal{F}$ we have that $\\mathbb{P}(F) = 1$, we say $F$ occurs almost surely (a.s.).\n%A sequence of random variables $\\{\\beta_n\\}$ is said to converge almost surely to a random variable $\\beta$ if and only if %see Gut2013, Def. 4.4 $P\\left(\\left\\lbrace \\omega \\in \\Omega: \\lim_{n \\rightarrow \\infty} \\beta_n(\\omega) = \\beta(\\omega) \\right\\rbrace\\right) = 1.$\nFor an integrable random variable $\\beta\\colon \\Omega \\rightarrow \\R$, the conditional expectation is denoted by $\\E[\\beta | \\mathcal{F}_n]$, which is itself a random variable that is $\\mathcal{F}_n$-measurable and which satisfies $\\int_A \\E[\\beta | \\mathcal{F}_n](\\omega) \\D \\pP(\\omega) = \\int_A \\beta(\\omega) \\D \\pP(\\omega)$ for all $A \\in \\mathcal{F}_n$.  \n","sha256_utf8":"4534a52f10711c8704c8e48280416ffa85cb5a75f67a88ac170e5dab985fdbc9"},{"start":75594,"end":76259,"start_line":1454,"end_line":1458,"raw_tex":"\\begin{assumption}\\label{assumption:gradient}\nLet $\\{\\mathcal{F}_n \\}$ be an increasing sequence of $\\sigma$-algebras.  For each $n$, there exist $b_n$, $w_n$ with\n$$b_n = \\E[G(h_n,\\xi_n) | \\mathcal{F}_n] - \\nabla j(h_n), \\quad w_n = G(h_n, \\xi_n) - \\E[G(h_n, \\xi_n) | \\mathcal{F}_n],$$ \nwhich satisfy the following assumptions:\\\\ (i) $ h_n$ and $b_n $ are $\\mathcal{F}_n$-measurable; (ii) for $K_n:=\\esssup_{\\omega \\in \\Omega} \\lVert b_n(\\omega)\\rVert_X$ we have that \\mbox{$ \\sum_{n=1}^\\infty t_n K_n< \\infty$} and \\mbox{$\\sup_n K_n < \\infty$}; (iii) there exists $M\\geq 0$ such that $\\E[\\lVert G(h,\\xi)\\rVert_X^2] \\leq M$ for all $r \\in X_{ad}$.\n\\end{assumption}","sha256_utf8":"40375776907524ed9f9cc9ecce5c032d678c65da99e00cb32d9673a5457937f9"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{lemma:convergenceofPSG}\nSuppose that Assumption~\\ref{assumption:j} and Assumption~\\ref{assumption:gradient} hold. If the sequence of step sizes satisfy \\eqref{eq:RobbinsMonroStepSize}, then for iterates defined by the recursion \\eqref{eq:SGM-recursion}, we have\n\\begin{enumerate}\n \\item $\\{j(h_n) \\}$ converges a.s.\\,and $\\lim_{n \\rightarrow \\infty} j(h_n) = j(h^*)$,\n \\item $\\{h_n\\}$ almost surely converges weakly in $X$ to $h^*$, and\n \\item $\\{h_n \\}$ almost surely converges strongly in $H$ to $h^*$. \n\\end{enumerate}\n\\end{theorem}","units":[{"start":76698,"end":77255,"start_line":1462,"end_line":1469,"raw_tex":"\\begin{theorem}\\label{lemma:convergenceofPSG}\nSuppose that Assumption~\\ref{assumption:j} and Assumption~\\ref{assumption:gradient} hold. If the sequence of step sizes satisfy \\eqref{eq:RobbinsMonroStepSize}, then for iterates defined by the recursion \\eqref{eq:SGM-recursion}, we have\n\\begin{enumerate}\n \\item $\\{j(h_n) \\}$ converges a.s.\\,and $\\lim_{n \\rightarrow \\infty} j(h_n) = j(h^*)$,\n \\item $\\{h_n\\}$ almost surely converges weakly in $X$ to $h^*$, and\n \\item $\\{h_n \\}$ almost surely converges strongly in $H$ to $h^*$. \n\\end{enumerate}\n\\end{theorem}","sha256_utf8":"d54056ad66e210ec781678989690e9306fdc7cf31c8756b1506014b19a079dc8"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"The inequality \\eqref{eq:strong-coercivity} implies strong convexity with respect to $H$. Indeed, a Taylor expansion around any point $h' \\in X_{ad}$ gives \\[j(h'+q)=j(h')+Dj(h')[q]+\\tfrac{1}{2}D^2j(\\eta)[q,q]\\] with $\\eta$ being a point on the segment between $h'$ and $h'+q$. Therefore,\n \\begin{equation}\n \\label{eq:strong-convexity}\n j(h'+q) - j(h') \\geq Dj(h')[q] + \\frac{c_E}{2} \\lVert q \\rVert^2_H\n \\end{equation}\n provided that $h'+q \\in X_{ad}$. If $h'=h^*$ is a (local) optimum, we have using \\eqref{eq:OC} that\n \\begin{equation}\n  \\label{eq:strong-convexity-2}\n j(h)-j(h^*) \\geq \\frac{c_E}{2} \\lVert h-h^*\\rVert_H^2 \\quad \\forall h \\in X_{ad}.\n \\end{equation}\nIn particular, $h^*$ is a strict minimizer in $X$ since $j(h) > j(h^*)$ for $h\\neq h^*.$\n\n\n\\begin{lemma}[Robbins--Siegmund \\cite{Robbins1971}]\\label{lemma:convegence_nonnegative_supermartingales}\nAssume that $\\{\\mathcal{F}_n\\}$ is a filtration and $v_n$,\n$a_n$, $b_n$, $c_n$ nonnegative random variables adapted to $\\{\\mathcal{F}_n\\}$. If\n$$\\mathbb{E} [v_{n+1}| \\mathcal{F}_n ] \\leq v_n(1 + a_n) + b_n - c_n \\mbox{ a.s. }$$\nand $\\sum_{n=1}^{\\infty} a_n < \\infty, \\, \\sum_{n=1}^{\\infty} b_n < \\infty$ a.s.,\nthen with probability one, $\\lbrace v_n \\rbrace $ is\nconvergent and $\\sum_{n=1}^{\\infty} c_n < \\infty.$\n\\end{lemma}\n\n\\begin{proposition}[\\cite{Geiersbach2019a}]\n\\label{prop:technicalproposition}\nLet $\\{\\tau_n \\}$ be a nonnegative deterministic sequence and $\\{ \\beta_n\\}$ a nonnegative random sequence in $\\mathbb{R}$ adapted to $\\{\\mathcal{F}_n$\\}. Assume that\n$\\sum_{n=1}^\\infty \\tau_n = \\infty$ and  $\\mathbb{E} [\\sum_{n=1}^\\infty \\tau_n \\beta_n] < \\infty$. Moreover assume that \n$\\beta_{n} - \\mathbb{E}[\\beta_{n+1}|\\mathcal{F}_n] \\leq \\gamma \\tau_n$ a.s.~for all $n$ and some $\\gamma > 0$. Then\n$$\\beta_n  \\hbox{ converges to } 0  \\hbox{ a.s. }$$\n\\end{proposition}\n\n\\begin{proof}\nRecall that since $j$ is strongly convex with respect to $H$, a unique minimum $h^*\\in X_{ad}$ exists. We note that $\\pi_{X_{ad}}(h^*) = h^*$. Now, with $g_n:=G(h_n,\\xi_n)$, we use the nonexpansivity of the projection to obtain\n\\begin{equation}\n\\label{eq:first-steps-convergence}\n\\begin{aligned}\n    \\lVert h_{n+1} -h^* \\lVert_X^2 &= \\lVert \\pi_{X_{ad}} (h_n - t_n g_n) - \\pi_{X_{ad}}(h^*)\\rVert_X^2\\\\\n    & \\leq \\lVert h_n - t_n g_n - h^*\\lVert_X^2\\\\\n    &=\\lVert h_n - h^* \\lVert_X^2 - 2t_n ( g_n, h_n-h^*)_X + t_n^2 \\lVert  g_n\\rVert_X^2.\n   % &=\\lVert h_n - h^* \\lVert_X^2 - 2t_n  D\\mathcal{J}(h_n,\\xi_n)[ h_n-h^*] + t_n^2 \\lVert  g_n\\rVert_X^2.\n\\end{aligned}\n\\end{equation}\n\nDue to (strong) convexity and the fact that $h_n, h^* \\in X_{ad}$, we have \n\\[ j(h^*) - j(h_n) \\geq Dj(h_n)[h^*-h_n] + \\frac{c_E}{2}\\lVert h^*-h_n\\rVert_H^2 \\geq Dj(h_n)[h^*-h_n]\\]\nso that $-(j(h_n) - j(h^*)) \\geq -D j(h_n)[h_n-h^*]=- (\\nabla j(h_n), h_n-h^*)_X$. Moreover, optimality of $h^*$ gives \\eqref{eq:OC}.\nTaking the conditional expectation on both sides of \\eqref{eq:first-steps-convergence} and applying Cauchy--Schwarz for the bias term, we have\n\\begin{equation}\n\\label{eq:conditional-inequality}\n\\begin{aligned}\n\\E[\\lVert h_{n+1} -h^* \\lVert_X^2|\\mathcal{F}_n] &\\leq \\lVert h_n - h^* \\lVert_X^2 - 2t_n ( \\nabla j(h_n)+b_n, h_n-h^*)_X + t_n^2 \\E[\\lVert g_n\\rVert_X^2|\\mathcal{F}_n]\\\\\n&\\leq (1+2t_nK_n)\\lVert h_n - h^* \\lVert_X^2 -2t_n(j(h_n)-j(h^*)) + t_n^2 M.\n\\end{aligned}\n\\end{equation}\nLemma~\\ref{lemma:convegence_nonnegative_supermartingales} implies that $\\{\\lVert h_n - h^*\\rVert_X \\}$ is a.s.~convergent and $\\sum_{n=1}^\\infty t_n (j(h_n)-j(h^*)) < \\infty$ a.s., from which we can conclude that $\\liminf_{n \\rightarrow \\infty} j(h_n) = j(h^*)$ with probability one. To show that in fact $\\lim_{n\\rightarrow \\infty} j(h_n) = j(h^*)$, can use a simpler argument than in \\cite[Theorem 3.6]{Geiersbach2019a} since we assumed $X_{ad}$ to be bounded. Indeed, applying expectation \\eqref{eq:conditional-inequality} and again using Lemma~\\ref{lemma:convegence_nonnegative_supermartingales}, we obtain  that $\\sum_{n=1}^\\infty t_n \\E[j(h_n)-j(h^*)] < \\infty$ surely. Convexity of $j$ implies that \n\\begin{align*}\n    j(h_n)-j(h_{n+1}) &\\leq (\\nabla j(h_n),h_n-h_{n+1})_X\\\\\n    & \\leq \\lVert \\nabla j(h_n) \\rVert_X \\lVert h_n-h_{n+1}\\rVert_X\\\\\n    & =\\lVert \\nabla j(h_n) \\rVert_X \\lVert \\pi_{X_{ad}}(h_n)-\\pi_{X_{ad}}(h_{n}-t_n g_n)\\rVert_X\\\\\n    & \\leq \\lVert \\nabla j(h_n) \\rVert_X \\lVert t_n g_n\\rVert_X.\n\\end{align*}\nSince $X_{ad}$ is bounded, so is $\\{h_n\\}$, and so there exists a $\\tilde{M}>0$ such that $ \\lVert \\nabla j(h_n) \\rVert_X \\leq \\tilde{M}$ for all $n$. After applying the conditional expectation, we have that \n\\[\nj(h_n)-\\E[j(h_{n+1})|\\mathcal{F}_n] \\leq t_n \\tilde{M} \\E[\\lVert g_n\\rVert_X|\\mathcal{F}_n].\n\\] \nFrom Jensen's inequality, we see that \n\\[\n(\\E[\\lVert g_n\\rVert_X|\\mathcal{F}_n])^2 \\leq \\E[\\lVert g_n\\rVert_X^2|\\mathcal{F}_n] \\leq M,\n\\]\nfrom which we can conclude that $j(h_n)-\\E[j(h_{n+1})|\\mathcal{F}_n] \\leq t_n \\tilde{M} M.$\nNow, we can apply Proposition~\\ref{prop:technicalproposition} to conclude that  $\\lim_{n\\rightarrow \\infty} j(h_n) = j(h^*) = 0$, which was the first claim.\n\nFor the second claim, we observe an arbitrary trajectory of the random sequence $\\{h_n\\}$. Since $\\{\\lVert h_n -h^*\\rVert_X^2\\}$ is convergent, it is also bounded. In particular, there exists a weak accumulation point $\\bar{h} \\in X_{ad}$ of the sequence $\\{h_n\\}$. Let $\\{h_{n_k}\\}$ be a subsequence such that $h_{n_k}\\rightharpoonup_X \\bar{h}$. By weak lower semicontinuity of $j$ (which follows from the continuity and convexity of $j$ in $X$), we have\n\\[\nj(\\bar{h})\\leq \\lim_{k\\rightarrow \\infty} j(h_{n_k}) = j(h^*),\n\\]\nwhere equality follows by the first part of this proof. Since $h^*$ is the (unique) minimizer, it follows that $j(h^*) = j(\\bar{h})$, from which we can conclude that $h^* = \\bar{h}$. The accumulation point being unique gives in fact $h_{n} \\rightharpoonup_X h^*$ with probability one.\n\nThe third claim follows now directly from \\eqref{eq:strong-convexity-2}, namely that $\\lVert h_n - h^*\\rVert_H^2 \\rightarrow 0$ a.s.~as $n \\rightarrow \\infty$.\n\\end{proof}","units":[{"start":64982,"end":65741,"start_line":1325,"end_line":1335,"raw_tex":"The inequality \\eqref{eq:strong-coercivity} implies strong convexity with respect to $H$. Indeed, a Taylor expansion around any point $h' \\in X_{ad}$ gives \\[j(h'+q)=j(h')+Dj(h')[q]+\\tfrac{1}{2}D^2j(\\eta)[q,q]\\] with $\\eta$ being a point on the segment between $h'$ and $h'+q$. Therefore,\n \\begin{equation}\n \\label{eq:strong-convexity}\n j(h'+q) - j(h') \\geq Dj(h')[q] + \\frac{c_E}{2} \\lVert q \\rVert^2_H\n \\end{equation}\n provided that $h'+q \\in X_{ad}$. If $h'=h^*$ is a (local) optimum, we have using \\eqref{eq:OC} that\n \\begin{equation}\n  \\label{eq:strong-convexity-2}\n j(h)-j(h^*) \\geq \\frac{c_E}{2} \\lVert h-h^*\\rVert_H^2 \\quad \\forall h \\in X_{ad}.\n \\end{equation}\nIn particular, $h^*$ is a strict minimizer in $X$ since $j(h) > j(h^*)$ for $h\\neq h^*.$\n","sha256_utf8":"7e76038de6cbdbd40008a286e9c538d4a12581e0c467d56a10171a643f86b60b"},{"start":74075,"end":74605,"start_line":1436,"end_line":1443,"raw_tex":"\\begin{lemma}[Robbins--Siegmund \\cite{Robbins1971}]\\label{lemma:convegence_nonnegative_supermartingales}\nAssume that $\\{\\mathcal{F}_n\\}$ is a filtration and $v_n$,\n$a_n$, $b_n$, $c_n$ nonnegative random variables adapted to $\\{\\mathcal{F}_n\\}$. If\n$$\\mathbb{E} [v_{n+1}| \\mathcal{F}_n ] \\leq v_n(1 + a_n) + b_n - c_n \\mbox{ a.s. }$$\nand $\\sum_{n=1}^{\\infty} a_n < \\infty, \\, \\sum_{n=1}^{\\infty} b_n < \\infty$ a.s.,\nthen with probability one, $\\lbrace v_n \\rbrace $ is\nconvergent and $\\sum_{n=1}^{\\infty} c_n < \\infty.$\n\\end{lemma}","sha256_utf8":"5cc48ed9624c3246535ebe2f845316d4214bfc3d6622a52d84a8fcc86314b2f6"},{"start":74646,"end":75197,"start_line":1445,"end_line":1451,"raw_tex":"\\begin{proposition}[\\cite{Geiersbach2019a}]\n\\label{prop:technicalproposition}\nLet $\\{\\tau_n \\}$ be a nonnegative deterministic sequence and $\\{ \\beta_n\\}$ a nonnegative random sequence in $\\mathbb{R}$ adapted to $\\{\\mathcal{F}_n$\\}. Assume that\n$\\sum_{n=1}^\\infty \\tau_n = \\infty$ and  $\\mathbb{E} [\\sum_{n=1}^\\infty \\tau_n \\beta_n] < \\infty$. Moreover assume that \n$\\beta_{n} - \\mathbb{E}[\\beta_{n+1}|\\mathcal{F}_n] \\leq \\gamma \\tau_n$ a.s.~for all $n$ and some $\\gamma > 0$. Then\n$$\\beta_n  \\hbox{ converges to } 0  \\hbox{ a.s. }$$\n\\end{proposition}","sha256_utf8":"6fca79b580c587f09c658269bf43ca6ad36c819df0963ae4600025be462927c0"},{"start":77256,"end":81461,"start_line":1470,"end_line":1518,"raw_tex":"\\begin{proof}\nRecall that since $j$ is strongly convex with respect to $H$, a unique minimum $h^*\\in X_{ad}$ exists. We note that $\\pi_{X_{ad}}(h^*) = h^*$. Now, with $g_n:=G(h_n,\\xi_n)$, we use the nonexpansivity of the projection to obtain\n\\begin{equation}\n\\label{eq:first-steps-convergence}\n\\begin{aligned}\n    \\lVert h_{n+1} -h^* \\lVert_X^2 &= \\lVert \\pi_{X_{ad}} (h_n - t_n g_n) - \\pi_{X_{ad}}(h^*)\\rVert_X^2\\\\\n    & \\leq \\lVert h_n - t_n g_n - h^*\\lVert_X^2\\\\\n    &=\\lVert h_n - h^* \\lVert_X^2 - 2t_n ( g_n, h_n-h^*)_X + t_n^2 \\lVert  g_n\\rVert_X^2.\n   % &=\\lVert h_n - h^* \\lVert_X^2 - 2t_n  D\\mathcal{J}(h_n,\\xi_n)[ h_n-h^*] + t_n^2 \\lVert  g_n\\rVert_X^2.\n\\end{aligned}\n\\end{equation}\n\nDue to (strong) convexity and the fact that $h_n, h^* \\in X_{ad}$, we have \n\\[ j(h^*) - j(h_n) \\geq Dj(h_n)[h^*-h_n] + \\frac{c_E}{2}\\lVert h^*-h_n\\rVert_H^2 \\geq Dj(h_n)[h^*-h_n]\\]\nso that $-(j(h_n) - j(h^*)) \\geq -D j(h_n)[h_n-h^*]=- (\\nabla j(h_n), h_n-h^*)_X$. Moreover, optimality of $h^*$ gives \\eqref{eq:OC}.\nTaking the conditional expectation on both sides of \\eqref{eq:first-steps-convergence} and applying Cauchy--Schwarz for the bias term, we have\n\\begin{equation}\n\\label{eq:conditional-inequality}\n\\begin{aligned}\n\\E[\\lVert h_{n+1} -h^* \\lVert_X^2|\\mathcal{F}_n] &\\leq \\lVert h_n - h^* \\lVert_X^2 - 2t_n ( \\nabla j(h_n)+b_n, h_n-h^*)_X + t_n^2 \\E[\\lVert g_n\\rVert_X^2|\\mathcal{F}_n]\\\\\n&\\leq (1+2t_nK_n)\\lVert h_n - h^* \\lVert_X^2 -2t_n(j(h_n)-j(h^*)) + t_n^2 M.\n\\end{aligned}\n\\end{equation}\nLemma~\\ref{lemma:convegence_nonnegative_supermartingales} implies that $\\{\\lVert h_n - h^*\\rVert_X \\}$ is a.s.~convergent and $\\sum_{n=1}^\\infty t_n (j(h_n)-j(h^*)) < \\infty$ a.s., from which we can conclude that $\\liminf_{n \\rightarrow \\infty} j(h_n) = j(h^*)$ with probability one. To show that in fact $\\lim_{n\\rightarrow \\infty} j(h_n) = j(h^*)$, can use a simpler argument than in \\cite[Theorem 3.6]{Geiersbach2019a} since we assumed $X_{ad}$ to be bounded. Indeed, applying expectation \\eqref{eq:conditional-inequality} and again using Lemma~\\ref{lemma:convegence_nonnegative_supermartingales}, we obtain  that $\\sum_{n=1}^\\infty t_n \\E[j(h_n)-j(h^*)] < \\infty$ surely. Convexity of $j$ implies that \n\\begin{align*}\n    j(h_n)-j(h_{n+1}) &\\leq (\\nabla j(h_n),h_n-h_{n+1})_X\\\\\n    & \\leq \\lVert \\nabla j(h_n) \\rVert_X \\lVert h_n-h_{n+1}\\rVert_X\\\\\n    & =\\lVert \\nabla j(h_n) \\rVert_X \\lVert \\pi_{X_{ad}}(h_n)-\\pi_{X_{ad}}(h_{n}-t_n g_n)\\rVert_X\\\\\n    & \\leq \\lVert \\nabla j(h_n) \\rVert_X \\lVert t_n g_n\\rVert_X.\n\\end{align*}\nSince $X_{ad}$ is bounded, so is $\\{h_n\\}$, and so there exists a $\\tilde{M}>0$ such that $ \\lVert \\nabla j(h_n) \\rVert_X \\leq \\tilde{M}$ for all $n$. After applying the conditional expectation, we have that \n\\[\nj(h_n)-\\E[j(h_{n+1})|\\mathcal{F}_n] \\leq t_n \\tilde{M} \\E[\\lVert g_n\\rVert_X|\\mathcal{F}_n].\n\\] \nFrom Jensen's inequality, we see that \n\\[\n(\\E[\\lVert g_n\\rVert_X|\\mathcal{F}_n])^2 \\leq \\E[\\lVert g_n\\rVert_X^2|\\mathcal{F}_n] \\leq M,\n\\]\nfrom which we can conclude that $j(h_n)-\\E[j(h_{n+1})|\\mathcal{F}_n] \\leq t_n \\tilde{M} M.$\nNow, we can apply Proposition~\\ref{prop:technicalproposition} to conclude that  $\\lim_{n\\rightarrow \\infty} j(h_n) = j(h^*) = 0$, which was the first claim.\n\nFor the second claim, we observe an arbitrary trajectory of the random sequence $\\{h_n\\}$. Since $\\{\\lVert h_n -h^*\\rVert_X^2\\}$ is convergent, it is also bounded. In particular, there exists a weak accumulation point $\\bar{h} \\in X_{ad}$ of the sequence $\\{h_n\\}$. Let $\\{h_{n_k}\\}$ be a subsequence such that $h_{n_k}\\rightharpoonup_X \\bar{h}$. By weak lower semicontinuity of $j$ (which follows from the continuity and convexity of $j$ in $X$), we have\n\\[\nj(\\bar{h})\\leq \\lim_{k\\rightarrow \\infty} j(h_{n_k}) = j(h^*),\n\\]\nwhere equality follows by the first part of this proof. Since $h^*$ is the (unique) minimizer, it follows that $j(h^*) = j(\\bar{h})$, from which we can conclude that $h^* = \\bar{h}$. The accumulation point being unique gives in fact $h_{n} \\rightharpoonup_X h^*$ with probability one.\n\nThe third claim follows now directly from \\eqref{eq:strong-convexity-2}, namely that $\\lVert h_n - h^*\\rVert_H^2 \\rightarrow 0$ a.s.~as $n \\rightarrow \\infty$.\n\\end{proof}","sha256_utf8":"037e863a7f8d3d255c1724cddee929601fb956f49fb3dd0e6eaf87bb8f222d21"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"双空间与可行集","source_units":[{"start":60425,"end":61656,"start_line":1258,"end_line":1269,"raw_tex":"the free boundary problem \\eqref{bvp}--\\eqref{flux}. Let $X\\subset H$\nbe two Hilbert spaces, which are dense in $L^2$, %\\CG{for the convergence theory, I do not seem to need that these spaces dense in $L^2$. Is that unexpected? If it is not needed, it would appear to be more general to mention of $L^2$.} \n%\\HH{Die Stetigkeits- und Elliptizit\\\"atsabsch\\\"atzungen brauchen meines Erachtens den Gelfand-Dreier. Wir benutzen ja, dass der  Hessian eine Bilinear-Form auf $H^{1/2}\\times H^{1/2}$ definiert, also ein Operator $H^{1/2}\\to H^{-1/2}$ ist. Mir w\\\"are es also wohler, wenn wir uns hier einschr\\\"anken wegen der zugrundeliegenden PDE-Theorie.}\nendowed with the \ninner products $( \\cdot,\\cdot)_X$ and $(\\cdot, \\cdot)_H$, respectively, \nand corresponding norms $\\lVert \\cdot\\rVert_X$ and $\\lVert \\cdot \\rVert_H$. \nThe respective dual spaces are denoted by $X^*$ and $H^*$, which yields \nthe Gelfand chain $X \\subset H\\subset L^2 \\subset H^*\\subset X^*$.\nA ball centered at $r$ in a space $X'$ is denoted by $B_{\\delta}^{X'}(r) \n= \\{ h \\in X' \\mid \\lVert h-r\\rVert_{X'}< \\delta \\}$. We assume that \n$X \\subset H$ with continuous embedding and that $X_{ad} \\subset X$ \nis a bounded, closed, convex, and nonempty admissible set. \n","sha256_utf8":"69448e22ba50ee22fca1bd2de0c8701c91097d48ead7a2cf032c03548ff6f376"}]},{"id":"S002","semantic_type":"problem_setting","kind":"problem_setting","title":"期望最小化问题","source_units":[{"start":61657,"end":62054,"start_line":1271,"end_line":1277,"raw_tex":"Let $(\\Omega, \\mathcal{F}, \\pP)$ be a complete probability space and \n$\\xi \\colon \\Omega \\to\\Xi$ be a random function mapping to a (real) \ncomplete separable metric space $\\Xi$. Now, consider the problem \n\\begin{equation}\\label{eq:SAproblem}\n%======================================================\n\\underset{h \\in X_{ad}}{\\textup{minimize}} \\quad \\{j(h) = \\E[\\mathcal{J}(h,\\xi)] \\},\n\\end{equation}","sha256_utf8":"d877cad596a2c881d6c2cf15599e729b957d9a3e48a44b5a8c858535d5dc3576"},{"start":62055,"end":62479,"start_line":1278,"end_line":1279,"raw_tex":"where %, $J \\colon X \\times \\Xi \\to \\R$, \nwe assume that $j \\colon O \\subset X \\rightarrow \\R$ is twice continuously differentiable on an open set $O \\supset X_{ad}$. A point $h^*$ is said to be a \\textit{local solution} of \\eqref{eq:SAproblem} in $X'$ if $j(h^*) \\leq j(h)$ for all $h \\in X_{ad} \\cap B_{\\delta}^{X'}(h^*)$ for some $\\delta >0$. A necessary condition for $h^*\\in X_{ad}$ to be a local solution (in $X$) to \n","sha256_utf8":"b289032fe35a3740f983de780ef8b72d4839c5eff7fd4b21dda8993cc27f2a37"}]},{"id":"S003","semantic_type":"derived_fact","kind":"derived_fact","title":"变分最优性条件","source_units":[{"start":62479,"end":62724,"start_line":1280,"end_line":1285,"raw_tex":"\\eqref{eq:SAproblem} is given by the variational inequality \n%See e.g. Troeltzsch, Lemma 2.21\n\\begin{equation}\\label{eq:OC}\n%======================================================\nD j(h^*)[h-h^*] \\geq 0 \\quad \\forall h \\in X_{ad}.\n\\end{equation}","sha256_utf8":"803e6f8a35f9ed5547770c522383daa44bf3ce47c466382534eabccb382d7b8a"},{"start":62725,"end":63159,"start_line":1286,"end_line":1287,"raw_tex":"In the event that $j$ is convex on $X_{ad}$, this condition is \nalso sufficient and $h^*$ is even a global solution. Notice that since $H$ is the weaker space, any local solution of \\eqref{eq:SAproblem} in $H$ is also a local solution in $X$; i.e., local solutions in $H$ also satisfy the condition \\eqref{eq:OC}. Our strategy of handling the two-norm discrepancy will be to show that our method converges in $H$ to a local solution.\n","sha256_utf8":"3a6da420341d7dd061cde075b40e210bac854fe9555a1d61a023c37fa08ad7f7"}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","title":"弱范数下的海森条件","source_units":[{"start":63395,"end":64477,"start_line":1293,"end_line":1315,"raw_tex":"\\begin{assumption}\\label{assumption:j}\n%======================================================\nThe functional $j \\colon O\\to\\R$ is twice continuously \ndifferentiable. We assume that for every $h \\in X_{ad},$ %there exists $\\overline{r} \\in X$ \n%satisfying \\eqref{eq:OC} such that for all $r$ on a \n%neighborhood $\\overline{B_{\\delta}^X(\\overline{r})}$ of \n%$\\overline{r}$, \nthe second derivative $D^2 j(h)\\in \n\\mathcal{L}(X,X^*)$ extends continuously to a bilinear \nform on $H\\times H$, i.e., there exists $C_S >0$ such that \n\\begin{equation}\\label{eq:assumption-continuity-bilinear-form}\n%======================================================================\n    |D^2 j(h)[q_1,q_2]| \\leq C_S \\lVert q_1 \\rVert_H \\lVert q_2 \\rVert_H  \n    \\ \\forall q_1, q_2 \\in H.\n\\end{equation}\nAdditionally, we assume that for there exists $c_E >0$ such that \nthe following strong coercivity condition is satisfied for every $h \\in X_{ad}$:\n\\begin{equation}\n    \\label{eq:strong-coercivity} \n    D^2j(h)[q,q] \\ge c_E \\lVert q\\rVert_H^2 \n    \\quad \\forall q \\in H.\n\\end{equation}\n\\end{assumption}","sha256_utf8":"433e39e9e48477150c16af35dd9426bd96ad1a742058d16d4c899125f9095b96"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"投影与梯度定义","source_units":[{"start":70112,"end":70279,"start_line":1402,"end_line":1405,"raw_tex":"Let $\\pi_{X_{ad}} \\colon X \\rightarrow X_{ad}$ be a projection onto the set $X_{ad}$, defined by\n\\[\n\\pi_{X_{ad}}(h) = \\argmin_{w \\in X_{ad}} \\, \\lVert h-w\\rVert_X, \n\\]","sha256_utf8":"0256f2b89914414603af56bd9ae2b7b2b97d0beff63f93918748483bff313c63"},{"start":70280,"end":70951,"start_line":1406,"end_line":1411,"raw_tex":"which is well-defined and single-valued since $X_{ad} \\subset X$ is assumed to be nonempty, closed, and convex.   We assume that it is possible to compute an approximation of the gradient in the form of a \\textit{stochastic gradient} $G\\colon X \\times \\Xi \\rightarrow X$, which is defined as the (parameterized) Riesz representative of the mapping $D\\mathcal{J}(\\cdot,\\hat{\\xi}) \\colon X \\rightarrow X^*$, i.e., we have for every $\\hat{\\xi} \\in \\Xi$ that\n\\begin{equation}\n\\label{eq:grad_projection}\n%==========================================\n(G(h,\\hat{\\xi}),q)_X = \\langle D\\mathcal{J}(h,\\hat{\\xi}),q\\rangle_{X^*,X} \\quad \\forall (h,q) \\in X_{ad}\\times X.\n\\end{equation}","sha256_utf8":"460d4ebdbd33dd25eb5672ef813ab0f7ff9c86e0e77b449f68325a87787d9db7"},{"start":71060,"end":71206,"start_line":1413,"end_line":1413,"raw_tex":"We use the notation $\\nabla j$ for the  gradient of $j$ in $X$, i.e., $(\\nabla j(h),q)_{X} = \\langle Dj(h),q\\rangle_{X^*,X}$, where $h,q \\in X$. \n","sha256_utf8":"32493903eba7313284cce3371431dee2c34acd3a6776b2e17a14049ec7e5d2ec"}]},{"id":"S006","semantic_type":"algorithm","kind":"algorithm","title":"随机投影迭代与步长","source_units":[{"start":71206,"end":71386,"start_line":1414,"end_line":1418,"raw_tex":"The projected stochastic gradient method relies on a recursion of the form\n\\begin{equation}\n\\label{eq:SGM-recursion}\nh_{n+1} := \\pi_{X_{ad}}(h_n - t_n G(h_n,\\xi_n)),\n\\end{equation}","sha256_utf8":"5671ad2ccc0d87b1049eb18a4421d4612d45e1fd62e16fd61a4058adb208485c"},{"start":71387,"end":71711,"start_line":1419,"end_line":1419,"raw_tex":"where $h_1 \\in X_{ad}$ and $\\xi_n$ is randomly sampled from the law $\\mathbb{P}\\circ\\xi^{-1}$ independently of previous samples $\\xi_1, \\dots, \\xi_{n-1}.$ We require that the step sizes given in \\eqref{eq:SGM-recursion} satisfy the Robbins--Monro rule from the original paper \\cite{Robbins1951} on stochastic approximation:\n","sha256_utf8":"5d57999f39915d761962d06a8abd922f003c0c915ef26f6e19e326bdb079f078"},{"start":71712,"end":71866,"start_line":1420,"end_line":1422,"raw_tex":"\\begin{equation}\\label{eq:RobbinsMonroStepSize}\nt_n \\geq 0, \\quad \\sum_{n=1}^\\infty t_n = \\infty, \\quad  \\sum_{n=1}^\\infty t_n^2 < \\infty.\n \\end{equation}","sha256_utf8":"cf621fe1928a3ebf003921450fc240f4d124fca8cee538482ed4477144a0acf9"}]},{"id":"S007","semantic_type":"assumption","kind":"assumption","title":"偏差与二阶矩条件","source_units":[{"start":75594,"end":76259,"start_line":1454,"end_line":1458,"raw_tex":"\\begin{assumption}\\label{assumption:gradient}\nLet $\\{\\mathcal{F}_n \\}$ be an increasing sequence of $\\sigma$-algebras.  For each $n$, there exist $b_n$, $w_n$ with\n$$b_n = \\E[G(h_n,\\xi_n) | \\mathcal{F}_n] - \\nabla j(h_n), \\quad w_n = G(h_n, \\xi_n) - \\E[G(h_n, \\xi_n) | \\mathcal{F}_n],$$ \nwhich satisfy the following assumptions:\\\\ (i) $ h_n$ and $b_n $ are $\\mathcal{F}_n$-measurable; (ii) for $K_n:=\\esssup_{\\omega \\in \\Omega} \\lVert b_n(\\omega)\\rVert_X$ we have that \\mbox{$ \\sum_{n=1}^\\infty t_n K_n< \\infty$} and \\mbox{$\\sup_n K_n < \\infty$}; (iii) there exists $M\\geq 0$ such that $\\E[\\lVert G(h,\\xi)\\rVert_X^2] \\leq M$ for all $r \\in X_{ad}$.\n\\end{assumption}","sha256_utf8":"40375776907524ed9f9cc9ecce5c032d678c65da99e00cb32d9673a5457937f9"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"滤过与条件期望","source_units":[{"start":72364,"end":74010,"start_line":1426,"end_line":1432,"raw_tex":"First, we recall some concepts that will be of use in the proof. A filtration is a sequence $\\{ \\mathcal{F}_n\\}$ of sub-$\\sigma$-algebras of $\\mathcal{F}$ such that {$\\mathcal{F}_1 \\subset \\mathcal{F}_2 \\subset \\cdots \\subset \\mathcal{F}.$} \nGiven a Banach space $Y$, we define a discrete $Y$-valued stochastic process as a collection of $Y$-valued random variables indexed by $n$, in other words, the set $\\{ \\beta_n \\mid \\Omega \\rightarrow Y \\mid n \\in \\mathbb{N}\\}.$ \nThe stochastic process is said to be adapted to a filtration $\\{ \\mathcal{F}_n \\}$ if and only if $\\beta_n$ is $\\mathcal{F}_n$-measurable for all $n$. Suppose $\\mathcal{B}(Y)$ denotes the set of Borel sets of $Y$.\nThe natural filtration is the filtration generated by the sequence $\\{\\beta_n\\}$ and is given by $\\mathcal{F}_n = \\sigma(\\{\\beta_1, \\dots ,\\beta_n\\}) = \\{ \\beta_i^{-1}(B)\\mid B \\in \\mathcal{B}(Y), i = 1, \\dots, n\\}$.\nIf for an event $F \\in \\mathcal{F}$ we have that $\\mathbb{P}(F) = 1$, we say $F$ occurs almost surely (a.s.).\n%A sequence of random variables $\\{\\beta_n\\}$ is said to converge almost surely to a random variable $\\beta$ if and only if %see Gut2013, Def. 4.4 $P\\left(\\left\\lbrace \\omega \\in \\Omega: \\lim_{n \\rightarrow \\infty} \\beta_n(\\omega) = \\beta(\\omega) \\right\\rbrace\\right) = 1.$\nFor an integrable random variable $\\beta\\colon \\Omega \\rightarrow \\R$, the conditional expectation is denoted by $\\E[\\beta | \\mathcal{F}_n]$, which is itself a random variable that is $\\mathcal{F}_n$-measurable and which satisfies $\\int_A \\E[\\beta | \\mathcal{F}_n](\\omega) \\D \\pP(\\omega) = \\int_A \\beta(\\omega) \\D \\pP(\\omega)$ for all $A \\in \\mathcal{F}_n$.  \n","sha256_utf8":"4534a52f10711c8704c8e48280416ffa85cb5a75f67a88ac170e5dab985fdbc9"}]},{"id":"S009","semantic_type":"lemma","kind":"lemma","title":"几乎超鞅收敛引理","source_units":[{"start":74075,"end":74605,"start_line":1436,"end_line":1443,"raw_tex":"\\begin{lemma}[Robbins--Siegmund \\cite{Robbins1971}]\\label{lemma:convegence_nonnegative_supermartingales}\nAssume that $\\{\\mathcal{F}_n\\}$ is a filtration and $v_n$,\n$a_n$, $b_n$, $c_n$ nonnegative random variables adapted to $\\{\\mathcal{F}_n\\}$. If\n$$\\mathbb{E} [v_{n+1}| \\mathcal{F}_n ] \\leq v_n(1 + a_n) + b_n - c_n \\mbox{ a.s. }$$\nand $\\sum_{n=1}^{\\infty} a_n < \\infty, \\, \\sum_{n=1}^{\\infty} b_n < \\infty$ a.s.,\nthen with probability one, $\\lbrace v_n \\rbrace $ is\nconvergent and $\\sum_{n=1}^{\\infty} c_n < \\infty.$\n\\end{lemma}","sha256_utf8":"5cc48ed9624c3246535ebe2f845316d4214bfc3d6622a52d84a8fcc86314b2f6"}]},{"id":"S010","semantic_type":"proposition","kind":"proposition","title":"非负随机序列消失准则","source_units":[{"start":74646,"end":75197,"start_line":1445,"end_line":1451,"raw_tex":"\\begin{proposition}[\\cite{Geiersbach2019a}]\n\\label{prop:technicalproposition}\nLet $\\{\\tau_n \\}$ be a nonnegative deterministic sequence and $\\{ \\beta_n\\}$ a nonnegative random sequence in $\\mathbb{R}$ adapted to $\\{\\mathcal{F}_n$\\}. Assume that\n$\\sum_{n=1}^\\infty \\tau_n = \\infty$ and  $\\mathbb{E} [\\sum_{n=1}^\\infty \\tau_n \\beta_n] < \\infty$. Moreover assume that \n$\\beta_{n} - \\mathbb{E}[\\beta_{n+1}|\\mathcal{F}_n] \\leq \\gamma \\tau_n$ a.s.~for all $n$ and some $\\gamma > 0$. Then\n$$\\beta_n  \\hbox{ converges to } 0  \\hbox{ a.s. }$$\n\\end{proposition}","sha256_utf8":"6fca79b580c587f09c658269bf43ca6ad36c819df0963ae4600025be462927c0"}]},{"id":"S011","semantic_type":"proof","kind":"proof","title":"强凸性与增长界证明","source_units":[{"start":64982,"end":65741,"start_line":1325,"end_line":1335,"raw_tex":"The inequality \\eqref{eq:strong-coercivity} implies strong convexity with respect to $H$. Indeed, a Taylor expansion around any point $h' \\in X_{ad}$ gives \\[j(h'+q)=j(h')+Dj(h')[q]+\\tfrac{1}{2}D^2j(\\eta)[q,q]\\] with $\\eta$ being a point on the segment between $h'$ and $h'+q$. Therefore,\n \\begin{equation}\n \\label{eq:strong-convexity}\n j(h'+q) - j(h') \\geq Dj(h')[q] + \\frac{c_E}{2} \\lVert q \\rVert^2_H\n \\end{equation}\n provided that $h'+q \\in X_{ad}$. If $h'=h^*$ is a (local) optimum, we have using \\eqref{eq:OC} that\n \\begin{equation}\n  \\label{eq:strong-convexity-2}\n j(h)-j(h^*) \\geq \\frac{c_E}{2} \\lVert h-h^*\\rVert_H^2 \\quad \\forall h \\in X_{ad}.\n \\end{equation}\nIn particular, $h^*$ is a strict minimizer in $X$ since $j(h) > j(h^*)$ for $h\\neq h^*.$\n","sha256_utf8":"7e76038de6cbdbd40008a286e9c538d4a12581e0c467d56a10171a643f86b60b"}]},{"id":"S012","semantic_type":"proof_step","kind":"proof_step","title":"泰勒展开推出强凸性","source_units":[{"start":64982,"end":65402,"start_line":1325,"end_line":1329,"raw_tex":"The inequality \\eqref{eq:strong-coercivity} implies strong convexity with respect to $H$. Indeed, a Taylor expansion around any point $h' \\in X_{ad}$ gives \\[j(h'+q)=j(h')+Dj(h')[q]+\\tfrac{1}{2}D^2j(\\eta)[q,q]\\] with $\\eta$ being a point on the segment between $h'$ and $h'+q$. Therefore,\n \\begin{equation}\n \\label{eq:strong-convexity}\n j(h'+q) - j(h') \\geq Dj(h')[q] + \\frac{c_E}{2} \\lVert q \\rVert^2_H\n \\end{equation}\n","sha256_utf8":"16c3f299bc0caecc74c9785e0860934024242e37c22ba5f1a15aea5495ea08c9"}]},{"id":"S013","semantic_type":"proof_step","kind":"proof_step","title":"最优性推出二次增长","source_units":[{"start":65402,"end":65741,"start_line":1330,"end_line":1335,"raw_tex":" provided that $h'+q \\in X_{ad}$. If $h'=h^*$ is a (local) optimum, we have using \\eqref{eq:OC} that\n \\begin{equation}\n  \\label{eq:strong-convexity-2}\n j(h)-j(h^*) \\geq \\frac{c_E}{2} \\lVert h-h^*\\rVert_H^2 \\quad \\forall h \\in X_{ad}.\n \\end{equation}\nIn particular, $h^*$ is a strict minimizer in $X$ since $j(h) > j(h^*)$ for $h\\neq h^*.$\n","sha256_utf8":"db88e796eac11a4a85bc5a95e8a6218587c7e5886c668c8fe753a0a15646d12d"}]},{"id":"S014","semantic_type":"theorem","kind":"theorem","title":"双范数随机梯度收敛定理","source_units":[{"start":76698,"end":77255,"start_line":1462,"end_line":1469,"raw_tex":"\\begin{theorem}\\label{lemma:convergenceofPSG}\nSuppose that Assumption~\\ref{assumption:j} and Assumption~\\ref{assumption:gradient} hold. If the sequence of step sizes satisfy \\eqref{eq:RobbinsMonroStepSize}, then for iterates defined by the recursion \\eqref{eq:SGM-recursion}, we have\n\\begin{enumerate}\n \\item $\\{j(h_n) \\}$ converges a.s.\\,and $\\lim_{n \\rightarrow \\infty} j(h_n) = j(h^*)$,\n \\item $\\{h_n\\}$ almost surely converges weakly in $X$ to $h^*$, and\n \\item $\\{h_n \\}$ almost surely converges strongly in $H$ to $h^*$. \n\\end{enumerate}\n\\end{theorem}","sha256_utf8":"d54056ad66e210ec781678989690e9306fdc7cf31c8756b1506014b19a079dc8"}]},{"id":"S015","semantic_type":"proof","kind":"proof","title":"随机梯度收敛完整证明","source_units":[{"start":77256,"end":81461,"start_line":1470,"end_line":1518,"raw_tex":"\\begin{proof}\nRecall that since $j$ is strongly convex with respect to $H$, a unique minimum $h^*\\in X_{ad}$ exists. We note that $\\pi_{X_{ad}}(h^*) = h^*$. Now, with $g_n:=G(h_n,\\xi_n)$, we use the nonexpansivity of the projection to obtain\n\\begin{equation}\n\\label{eq:first-steps-convergence}\n\\begin{aligned}\n    \\lVert h_{n+1} -h^* \\lVert_X^2 &= \\lVert \\pi_{X_{ad}} (h_n - t_n g_n) - \\pi_{X_{ad}}(h^*)\\rVert_X^2\\\\\n    & \\leq \\lVert h_n - t_n g_n - h^*\\lVert_X^2\\\\\n    &=\\lVert h_n - h^* \\lVert_X^2 - 2t_n ( g_n, h_n-h^*)_X + t_n^2 \\lVert  g_n\\rVert_X^2.\n   % &=\\lVert h_n - h^* \\lVert_X^2 - 2t_n  D\\mathcal{J}(h_n,\\xi_n)[ h_n-h^*] + t_n^2 \\lVert  g_n\\rVert_X^2.\n\\end{aligned}\n\\end{equation}\n\nDue to (strong) convexity and the fact that $h_n, h^* \\in X_{ad}$, we have \n\\[ j(h^*) - j(h_n) \\geq Dj(h_n)[h^*-h_n] + \\frac{c_E}{2}\\lVert h^*-h_n\\rVert_H^2 \\geq Dj(h_n)[h^*-h_n]\\]\nso that $-(j(h_n) - j(h^*)) \\geq -D j(h_n)[h_n-h^*]=- (\\nabla j(h_n), h_n-h^*)_X$. Moreover, optimality of $h^*$ gives \\eqref{eq:OC}.\nTaking the conditional expectation on both sides of \\eqref{eq:first-steps-convergence} and applying Cauchy--Schwarz for the bias term, we have\n\\begin{equation}\n\\label{eq:conditional-inequality}\n\\begin{aligned}\n\\E[\\lVert h_{n+1} -h^* \\lVert_X^2|\\mathcal{F}_n] &\\leq \\lVert h_n - h^* \\lVert_X^2 - 2t_n ( \\nabla j(h_n)+b_n, h_n-h^*)_X + t_n^2 \\E[\\lVert g_n\\rVert_X^2|\\mathcal{F}_n]\\\\\n&\\leq (1+2t_nK_n)\\lVert h_n - h^* \\lVert_X^2 -2t_n(j(h_n)-j(h^*)) + t_n^2 M.\n\\end{aligned}\n\\end{equation}\nLemma~\\ref{lemma:convegence_nonnegative_supermartingales} implies that $\\{\\lVert h_n - h^*\\rVert_X \\}$ is a.s.~convergent and $\\sum_{n=1}^\\infty t_n (j(h_n)-j(h^*)) < \\infty$ a.s., from which we can conclude that $\\liminf_{n \\rightarrow \\infty} j(h_n) = j(h^*)$ with probability one. To show that in fact $\\lim_{n\\rightarrow \\infty} j(h_n) = j(h^*)$, can use a simpler argument than in \\cite[Theorem 3.6]{Geiersbach2019a} since we assumed $X_{ad}$ to be bounded. Indeed, applying expectation \\eqref{eq:conditional-inequality} and again using Lemma~\\ref{lemma:convegence_nonnegative_supermartingales}, we obtain  that $\\sum_{n=1}^\\infty t_n \\E[j(h_n)-j(h^*)] < \\infty$ surely. Convexity of $j$ implies that \n\\begin{align*}\n    j(h_n)-j(h_{n+1}) &\\leq (\\nabla j(h_n),h_n-h_{n+1})_X\\\\\n    & \\leq \\lVert \\nabla j(h_n) \\rVert_X \\lVert h_n-h_{n+1}\\rVert_X\\\\\n    & =\\lVert \\nabla j(h_n) \\rVert_X \\lVert \\pi_{X_{ad}}(h_n)-\\pi_{X_{ad}}(h_{n}-t_n g_n)\\rVert_X\\\\\n    & \\leq \\lVert \\nabla j(h_n) \\rVert_X \\lVert t_n g_n\\rVert_X.\n\\end{align*}\nSince $X_{ad}$ is bounded, so is $\\{h_n\\}$, and so there exists a $\\tilde{M}>0$ such that $ \\lVert \\nabla j(h_n) \\rVert_X \\leq \\tilde{M}$ for all $n$. After applying the conditional expectation, we have that \n\\[\nj(h_n)-\\E[j(h_{n+1})|\\mathcal{F}_n] \\leq t_n \\tilde{M} \\E[\\lVert g_n\\rVert_X|\\mathcal{F}_n].\n\\] \nFrom Jensen's inequality, we see that \n\\[\n(\\E[\\lVert g_n\\rVert_X|\\mathcal{F}_n])^2 \\leq \\E[\\lVert g_n\\rVert_X^2|\\mathcal{F}_n] \\leq M,\n\\]\nfrom which we can conclude that $j(h_n)-\\E[j(h_{n+1})|\\mathcal{F}_n] \\leq t_n \\tilde{M} M.$\nNow, we can apply Proposition~\\ref{prop:technicalproposition} to conclude that  $\\lim_{n\\rightarrow \\infty} j(h_n) = j(h^*) = 0$, which was the first claim.\n\nFor the second claim, we observe an arbitrary trajectory of the random sequence $\\{h_n\\}$. Since $\\{\\lVert h_n -h^*\\rVert_X^2\\}$ is convergent, it is also bounded. In particular, there exists a weak accumulation point $\\bar{h} \\in X_{ad}$ of the sequence $\\{h_n\\}$. Let $\\{h_{n_k}\\}$ be a subsequence such that $h_{n_k}\\rightharpoonup_X \\bar{h}$. By weak lower semicontinuity of $j$ (which follows from the continuity and convexity of $j$ in $X$), we have\n\\[\nj(\\bar{h})\\leq \\lim_{k\\rightarrow \\infty} j(h_{n_k}) = j(h^*),\n\\]\nwhere equality follows by the first part of this proof. Since $h^*$ is the (unique) minimizer, it follows that $j(h^*) = j(\\bar{h})$, from which we can conclude that $h^* = \\bar{h}$. The accumulation point being unique gives in fact $h_{n} \\rightharpoonup_X h^*$ with probability one.\n\nThe third claim follows now directly from \\eqref{eq:strong-convexity-2}, namely that $\\lVert h_n - h^*\\rVert_H^2 \\rightarrow 0$ a.s.~as $n \\rightarrow \\infty$.\n\\end{proof}","sha256_utf8":"037e863a7f8d3d255c1724cddee929601fb956f49fb3dd0e6eaf87bb8f222d21"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"唯一极小点与投影误差界","source_units":[{"start":77256,"end":77949,"start_line":1470,"end_line":1480,"raw_tex":"\\begin{proof}\nRecall that since $j$ is strongly convex with respect to $H$, a unique minimum $h^*\\in X_{ad}$ exists. We note that $\\pi_{X_{ad}}(h^*) = h^*$. Now, with $g_n:=G(h_n,\\xi_n)$, we use the nonexpansivity of the projection to obtain\n\\begin{equation}\n\\label{eq:first-steps-convergence}\n\\begin{aligned}\n    \\lVert h_{n+1} -h^* \\lVert_X^2 &= \\lVert \\pi_{X_{ad}} (h_n - t_n g_n) - \\pi_{X_{ad}}(h^*)\\rVert_X^2\\\\\n    & \\leq \\lVert h_n - t_n g_n - h^*\\lVert_X^2\\\\\n    &=\\lVert h_n - h^* \\lVert_X^2 - 2t_n ( g_n, h_n-h^*)_X + t_n^2 \\lVert  g_n\\rVert_X^2.\n   % &=\\lVert h_n - h^* \\lVert_X^2 - 2t_n  D\\mathcal{J}(h_n,\\xi_n)[ h_n-h^*] + t_n^2 \\lVert  g_n\\rVert_X^2.\n\\end{aligned}\n\\end{equation}\n","sha256_utf8":"9752904301cb1a0ca51471b55d35ff69840a2b48114a58c88ff0cd9cc54fbbaa"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"凸性控制梯度内积","source_units":[{"start":77950,"end":78265,"start_line":1482,"end_line":1484,"raw_tex":"Due to (strong) convexity and the fact that $h_n, h^* \\in X_{ad}$, we have \n\\[ j(h^*) - j(h_n) \\geq Dj(h_n)[h^*-h_n] + \\frac{c_E}{2}\\lVert h^*-h_n\\rVert_H^2 \\geq Dj(h_n)[h^*-h_n]\\]\nso that $-(j(h_n) - j(h^*)) \\geq -D j(h_n)[h_n-h^*]=- (\\nabla j(h_n), h_n-h^*)_X$. Moreover, optimality of $h^*$ gives \\eqref{eq:OC}.\n","sha256_utf8":"d35498160f6cb0c7f49b96eb9e2b61fe6ae1225f9e6bac0b1cee0a869955143e"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"条件期望误差递推","source_units":[{"start":78265,"end":78752,"start_line":1485,"end_line":1492,"raw_tex":"Taking the conditional expectation on both sides of \\eqref{eq:first-steps-convergence} and applying Cauchy--Schwarz for the bias term, we have\n\\begin{equation}\n\\label{eq:conditional-inequality}\n\\begin{aligned}\n\\E[\\lVert h_{n+1} -h^* \\lVert_X^2|\\mathcal{F}_n] &\\leq \\lVert h_n - h^* \\lVert_X^2 - 2t_n ( \\nabla j(h_n)+b_n, h_n-h^*)_X + t_n^2 \\E[\\lVert g_n\\rVert_X^2|\\mathcal{F}_n]\\\\\n&\\leq (1+2t_nK_n)\\lVert h_n - h^* \\lVert_X^2 -2t_n(j(h_n)-j(h^*)) + t_n^2 M.\n\\end{aligned}\n\\end{equation}\n","sha256_utf8":"640ab4c378d9c62a982f853ac2ea341089e21c7fe7eaa591ea8414b2baa8d7d1"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"超鞅给出收敛与可和性","source_units":[{"start":78752,"end":79459,"start_line":1493,"end_line":1493,"raw_tex":"Lemma~\\ref{lemma:convegence_nonnegative_supermartingales} implies that $\\{\\lVert h_n - h^*\\rVert_X \\}$ is a.s.~convergent and $\\sum_{n=1}^\\infty t_n (j(h_n)-j(h^*)) < \\infty$ a.s., from which we can conclude that $\\liminf_{n \\rightarrow \\infty} j(h_n) = j(h^*)$ with probability one. To show that in fact $\\lim_{n\\rightarrow \\infty} j(h_n) = j(h^*)$, can use a simpler argument than in \\cite[Theorem 3.6]{Geiersbach2019a} since we assumed $X_{ad}$ to be bounded. Indeed, applying expectation \\eqref{eq:conditional-inequality} and again using Lemma~\\ref{lemma:convegence_nonnegative_supermartingales}, we obtain  that $\\sum_{n=1}^\\infty t_n \\E[j(h_n)-j(h^*)] < \\infty$ surely. Convexity of $j$ implies that \n","sha256_utf8":"ae53e29ab75efc71c9ab15c50fedeba5f0f20e04efb316e723397fff3bae9aae"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"目标下降量的投影估计","source_units":[{"start":79459,"end":79782,"start_line":1494,"end_line":1499,"raw_tex":"\\begin{align*}\n    j(h_n)-j(h_{n+1}) &\\leq (\\nabla j(h_n),h_n-h_{n+1})_X\\\\\n    & \\leq \\lVert \\nabla j(h_n) \\rVert_X \\lVert h_n-h_{n+1}\\rVert_X\\\\\n    & =\\lVert \\nabla j(h_n) \\rVert_X \\lVert \\pi_{X_{ad}}(h_n)-\\pi_{X_{ad}}(h_{n}-t_n g_n)\\rVert_X\\\\\n    & \\leq \\lVert \\nabla j(h_n) \\rVert_X \\lVert t_n g_n\\rVert_X.\n\\end{align*}\n","sha256_utf8":"851655ac03929dbbbcec9b886ab30e2c439c6faf51020bd94b06223809b1a67b"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"梯度有界与条件下降界","source_units":[{"start":79782,"end":80091,"start_line":1500,"end_line":1503,"raw_tex":"Since $X_{ad}$ is bounded, so is $\\{h_n\\}$, and so there exists a $\\tilde{M}>0$ such that $ \\lVert \\nabla j(h_n) \\rVert_X \\leq \\tilde{M}$ for all $n$. After applying the conditional expectation, we have that \n\\[\nj(h_n)-\\E[j(h_{n+1})|\\mathcal{F}_n] \\leq t_n \\tilde{M} \\E[\\lVert g_n\\rVert_X|\\mathcal{F}_n].\n\\] \n","sha256_utf8":"cf882c96bf1ad981224e561382e435ef5676a6fd1888f485fab890f7a3168262"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"应用消失准则得到目标收敛","source_units":[{"start":80091,"end":80478,"start_line":1504,"end_line":1509,"raw_tex":"From Jensen's inequality, we see that \n\\[\n(\\E[\\lVert g_n\\rVert_X|\\mathcal{F}_n])^2 \\leq \\E[\\lVert g_n\\rVert_X^2|\\mathcal{F}_n] \\leq M,\n\\]\nfrom which we can conclude that $j(h_n)-\\E[j(h_{n+1})|\\mathcal{F}_n] \\leq t_n \\tilde{M} M.$\nNow, we can apply Proposition~\\ref{prop:technicalproposition} to conclude that  $\\lim_{n\\rightarrow \\infty} j(h_n) = j(h^*) = 0$, which was the first claim.\n","sha256_utf8":"1218ba2694875fdb2b1247d52819705abfb81d27ca8e8ad18b4e86932913644d"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"弱聚点唯一推出弱收敛","source_units":[{"start":80479,"end":81289,"start_line":1511,"end_line":1515,"raw_tex":"For the second claim, we observe an arbitrary trajectory of the random sequence $\\{h_n\\}$. Since $\\{\\lVert h_n -h^*\\rVert_X^2\\}$ is convergent, it is also bounded. In particular, there exists a weak accumulation point $\\bar{h} \\in X_{ad}$ of the sequence $\\{h_n\\}$. Let $\\{h_{n_k}\\}$ be a subsequence such that $h_{n_k}\\rightharpoonup_X \\bar{h}$. By weak lower semicontinuity of $j$ (which follows from the continuity and convexity of $j$ in $X$), we have\n\\[\nj(\\bar{h})\\leq \\lim_{k\\rightarrow \\infty} j(h_{n_k}) = j(h^*),\n\\]\nwhere equality follows by the first part of this proof. Since $h^*$ is the (unique) minimizer, it follows that $j(h^*) = j(\\bar{h})$, from which we can conclude that $h^* = \\bar{h}$. The accumulation point being unique gives in fact $h_{n} \\rightharpoonup_X h^*$ with probability one.\n","sha256_utf8":"f296852800918fe93ebcc4f3b5db167ec6d7454b84f2dc3681bf6baed44c812a"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"二次增长推出弱范数强收敛","source_units":[{"start":81290,"end":81461,"start_line":1517,"end_line":1518,"raw_tex":"The third claim follows now directly from \\eqref{eq:strong-convexity-2}, namely that $\\lVert h_n - h^*\\rVert_H^2 \\rightarrow 0$ a.s.~as $n \\rightarrow \\infty$.\n\\end{proof}","sha256_utf8":"e4f381accbf5884d99b5571994f6fed4c50e0f8591632b9889c1237585a60044"}]},{"id":"S025","semantic_type":"external_fact","kind":"external_fact","title":"超鞅收敛外部定理","source_units":[],"declared_external":true,"declaration_text":["Robbins–Siegmund theorem: for nonnegative adapted sequences satisfying E[v_{n+1}|F_n] ≤ (1+a_n)v_n+b_n−c_n, almost sure summability of a_n and b_n implies almost sure convergence of v_n and summability of c_n."]},{"id":"S026","semantic_type":"external_fact","kind":"external_fact","title":"随机序列消失外部命题","source_units":[],"declared_external":true,"declaration_text":["The result attributed to Geiersbach2019a in B0113: if nonnegative deterministic τ_n have divergent sum, nonnegative adapted β_n satisfy E[Σ τ_nβ_n]<∞, and β_n−E[β_{n+1}|F_n]≤γτ_n almost surely, then β_n→0 almost surely."]},{"id":"S027","semantic_type":"external_fact","kind":"external_fact","title":"希尔伯特空间弱紧性事实","source_units":[],"declared_external":true,"declaration_text":["In a Hilbert space, bounded sequences have weakly convergent subsequences and closed convex sets are weakly closed. Continuous convex functionals are weakly lower semicontinuous on their convex domains, hence attain their minimum on nonempty bounded closed convex sets. Strict convexity gives uniqueness; a bounded sequence with a unique weak cluster point converges weakly to it."]},{"id":"S028","semantic_type":"external_fact","kind":"external_fact","title":"凸集投影非扩张性","source_units":[],"declared_external":true,"declaration_text":["The metric projection onto a nonempty closed convex subset of a Hilbert space exists uniquely, fixes every point of that subset, and satisfies ||P(x)−P(y)||≤||x−y||."]},{"id":"S029","semantic_type":"external_fact","kind":"external_fact","title":"条件期望基本估计","source_units":[],"declared_external":true,"declaration_text":["Conditional expectation preserves inequalities and admits measurable factors and the tower property. Conditional Jensen gives (E[||Z|| |F])²≤E[||Z||²|F]. Cauchy–Schwarz bounds inner products, and nonnegative series may be interchanged with expectation."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S014","edges":[{"source_id":"S001","target_id":"S012","kind":"supports"},{"source_id":"S001","target_id":"S016","kind":"supports"},{"source_id":"S001","target_id":"S021","kind":"supports"},{"source_id":"S001","target_id":"S023","kind":"supports"},{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S016","kind":"supports"},{"source_id":"S003","target_id":"S013","kind":"supports"},{"source_id":"S003","target_id":"S017","kind":"supports"},{"source_id":"S004","target_id":"S012","kind":"supports"},{"source_id":"S004","target_id":"S021","kind":"supports"},{"source_id":"S005","target_id":"S016","kind":"supports"},{"source_id":"S005","target_id":"S017","kind":"supports"},{"source_id":"S005","target_id":"S020","kind":"supports"},{"source_id":"S006","target_id":"S016","kind":"supports"},{"source_id":"S006","target_id":"S019","kind":"supports"},{"source_id":"S006","target_id":"S022","kind":"supports"},{"source_id":"S007","target_id":"S018","kind":"supports"},{"source_id":"S007","target_id":"S019","kind":"supports"},{"source_id":"S007","target_id":"S022","kind":"supports"},{"source_id":"S008","target_id":"S018","kind":"supports"},{"source_id":"S008","target_id":"S019","kind":"supports"},{"source_id":"S008","target_id":"S022","kind":"supports"},{"source_id":"S009","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S022","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S011","kind":"supports"},{"source_id":"S013","target_id":"S011","kind":"supports"},{"source_id":"S011","target_id":"S016","kind":"supports"},{"source_id":"S011","target_id":"S017","kind":"supports"},{"source_id":"S011","target_id":"S024","kind":"supports"},{"source_id":"S016","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S020","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S022","kind":"supports"},{"source_id":"S019","target_id":"S023","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S024","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S015","kind":"supports"},{"source_id":"S018","target_id":"S015","kind":"supports"},{"source_id":"S019","target_id":"S015","kind":"supports"},{"source_id":"S020","target_id":"S015","kind":"supports"},{"source_id":"S021","target_id":"S015","kind":"supports"},{"source_id":"S022","target_id":"S015","kind":"supports"},{"source_id":"S023","target_id":"S015","kind":"supports"},{"source_id":"S024","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S025","target_id":"S009","kind":"supports"},{"source_id":"S026","target_id":"S010","kind":"supports"},{"source_id":"S027","target_id":"S016","kind":"supports"},{"source_id":"S027","target_id":"S023","kind":"supports"},{"source_id":"S028","target_id":"S016","kind":"supports"},{"source_id":"S028","target_id":"S020","kind":"supports"},{"source_id":"S029","target_id":"S018","kind":"supports"},{"source_id":"S029","target_id":"S019","kind":"supports"},{"source_id":"S029","target_id":"S022","kind":"supports"}]}}
{"record_id":"2409.07122v3","paper":{"title":"Decentralized Conjugate Gradient and Memoryless BFGS Methods","authors":["Liping Wang","Hao Wu","Hongchao Zhang"],"arxiv_id":"2409.07122","version":"v3","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-026-03112-0","abstract_url":"https://arxiv.org/abs/2409.07122v3","pdf_url":"https://arxiv.org/pdf/2409.07122v3"},"context":{"raw_tex":"In this paper, we consider the following decentralized optimization problem over an undirected and connected network containing $n$ nodes\n\\begin{equation}\\label{obj_fun1}\n\t\\mathop {\\min } \\limits_{\\m{z} \\in {\\R^p}} \\; F(\\m{z}):= \\frac{1}{n}\\sum\\limits_{i = 1}^n {{f_i}(\\m{z})},\n\\end{equation}\n\nwhere the local objective function $f_i:\\R^{p} \\rightarrow \\R$ is  continuously differentiable. \nConsider the underlying network $\\mathcal{G}=\\left(\\mathcal{V},\\mathcal{E}\\right)$, where $\\mathcal{V} = \\{1,\\ldots,n\\}$ is the set of nodes, and $\\mathcal{E}$ is the collection of unordered edges.\n\n\nWe use uppercase boldface letters, e.g.~$\\m{W}$, for matrices and lowercase boldface letters,  e.g.~$\\m{w}$, for vectors.\nFor any vectors $\\m{v}_i \\in \\R^p$, $i=1,\\ldots,n$, we define  $\\bar{\\m{v}}=\\frac{1}{n} \\sum_{i=1}^n \\m{v}_i$ and $\\m{v} = [\\m{v}_1; \\m{v}_2; \\ldots, \\m{v}_n] \\in \\R^{np}$.\n\n\nGiven an undirected network  $\\mathcal{G}=\\left(\\mathcal{V},\\mathcal{E}\\right)$,\n let $\\m{x}_i$ denote the local copy of the global variable $\\m{z}$ at node $i$ and $\\mathcal{N}_i$ denote the set consisting of the neighbors of node $i$ \n(for convenience, we treat node $i$ itself as one of its neighbors). \nWe define $f(\\m{x}) = \\sum_{i = 1}^n {{f_i}({\\m{x}_i})}$ and use $\\m{g}^t$, $\\m{g}_i^t$ to stand for $\\nabla f(\\m{x}^t)$, $\\nabla f_i(\\m{x}^t_{i})$ respectively,\nwhere, for clarification, the gradient of $f(\\m{x})$ is defined as\n$ \\nabla f(\\m{x}) =\\left[ \\nabla f_1(\\m{x}_{1}); \\nabla f_2(\\m{x}_{2}); \\ldots, \\nabla f_n(\\m{x}_{n}) \\right] \\in \\R^{np}$. \n\n\nIn addition,  we define $\\overline{\\nabla} f(\\m{x}^t)=\\frac{1}{n} \\sum_{i=1}^n \\nabla f_i(\\m{x}_i^t) \n\\in \\R^p$.\nWe say that $\\m{x}$ is consensual or gets consensus if ${\\m{x}_1}={\\m{x}_2}=\\ldots={\\m{x}_n}$. \n$\\m{I}_p$ denotes the $p \\times p$ identity matrix and $\\m{I}$ denote $\\m{I}_{np}$ for simplicity. Kronecker Product is denoted as $\\otimes$.\nGiven a vector $\\m{v}$ and a matrix $\\m{N}$, $\\operatorname{span}(\\m{v})$ stands for the linear subspace spanned by $\\m{v}$;  $\\left\\|\\m{v}\\right\\|_{\\m{N}}^2$ denotes $\\m{v}\\tr \\m{N} \\m{v}$;\n$\\mbox{Null}(\\m{N})$ and $\\m{N}\\tr$  denote the null space and transpose of $\\m{N}$, respectively;\n${\\lambda _{\\min }(\\m{N})}$,  ${\\lambda _{\\max }(\\m{N})}$, and $\\rho(\\m{N})$ denote smallest eigenvalue, largest eigenvalue, and  the spectral radius of $\\m{N}$,  respectively;\nFor matrices $\\m{N}_1$ and $\\m{N}_2$ with same dimension, $\\m{N}_1 \\succeq \\m{N}_2$ means $\\m{N}_1 - \\m{N}_2$ is positive semidefinite, while $\\m{N}_1 \\ge \\m{N}_2$ means $\\m{N}_1 - \\m{N}_2$ is component-wise nonnegative.\nWe denote $\\operatorname{log}_{10}(\\cdot)$ by $\\operatorname{log}(\\cdot)$  and define $\\m{M}=\\frac{1}{n} \\m{1}_n\\m{1}_n\\tr\\otimes \\m{I}_p$ where $\\m{1}_n \\in \\R^n$ denotes the\nvector with all components ones. \n\n\n\\begin{assum}\\label{as0}\nThe local objective functions $\\{f_i \\}_{i=1}^n$ are bounded below and Lipschitz continuously differentiable, that is, for any $\\m{z}, \\tilde{\\m{z}} \\in \\R^p$ and $i=1, \\ldots,n$, \nwe have  $f_i(z)>-\\infty$ and \n\t\\begin{equation}\\label{3.1}\n\t\t\\left\\| {\\nabla {f_i}(\\m{z}) - \\nabla {f_i}(\\tilde{\\m{z}})} \\right\\| \\le L \\left\\| {\\m{z} - \\tilde{\\m{z}}} \\right\\|,\n\t\\end{equation}\n\\end{assum}\n\nwhere $L >0$ is a Lipschitz constant.\n\n\n\\begin{defin}\\label{mix}\n\t(Mixing matrix $\\tilde{\\m{W}}$ for given network $\\mathcal{G} = \\left(\\mathcal{V},\\mathcal{E}\\right)$)\n\t\\begin{itemize}\n\t\t\\item [1.] $\\tilde{\\m{W}}$ is nonnegative, where each component $\\tilde{W}_{ij}$ characterizes the active link $(i,j)$, i.e., $\\tilde{{W}}_{i,j}>0$ if $j \\in \\mathcal{N}_i$;  $\\tilde{{W}}_{i,j}=0$, otherwise.\n\t\t\\item [2.] $\\tilde{\\m{W}}$ is symmetric and doubly stochastic, i.e., $\\tilde{\\m{W}}=\\tilde{\\m{W}}\\tr$ and $\\tilde{\\m{W}}\\m{1}_n=\\m{1}_n$.\\\\\n\t\\end{itemize}\n\\end{defin}\n\nLet $\\lambda_{i}(\\tilde{\\m{W}})$ denote the $i$-th largest eigenvalue of $\\tilde{\\m{W}}$ and $\\sigma$ be the second largest magnitude eigenvalue of $\\tilde{\\m{W}}$. \nThen, the following properties hold \\cite{xin2019distributed}.\n\\begin{lemma}\\label{property W}\nFor $\\tilde{\\m{W}}$ defined in Definition~\\ref{mix} and  $\\m{W} :=\\tilde{\\m{W}}\\otimes\\m{I}_p$, we have\n\t\\begin{itemize}\n\t\t\\item [1.] $1=\\lambda_{1}(\\tilde{\\m{W}})>\\lambda_{2}(\\tilde{\\m{W}})\\geq\\ldots\\geq\\lambda_{n}(\\tilde{\\m{W}})>-1$;\n\t\t\\item [2.] $\t0<\\rho(\\m{W}-\\m{M})=\\sigma=\\max \\left\\{|\\lambda_{2}(\\tilde{\\m{W}})|, |\\lambda_{n}(\\tilde{\\m{W}})|\\right\\}<1$;\n\t\t\\item [3.] $\\m{M}=\\m{M}\\m{W}=\\m{W}\\m{M}$;\n\t\t\\item [4.]  $\\|\\m{W}\\m{x}-\\m{M}\\m{x}\\|=\\|(\\m{W}-\\m{M})(\\m{x}-\\m{M}\\m{x})\\| \\leq \\sigma \\|\\m{x}-\\m{M}\\m{x}\\|$\n\t\tfor any $\\m{x} \\in \\R^{np}$. \n\t\\end{itemize}\n\\end{lemma}\n\n\\begin{lemma}\\label{young}\n\t(Young's inequality) For any two vectors $\\m{v}_1,\\m{v}_2\\in \\R^p$, $\\eta>0$, we have\n\\[\n2\\m{v}_1\\tr\\m{v}_2 \\leq \\eta \\|\\m{v}_1\\|^2+ \\frac{1}{\\eta}\\|\\m{v}_2\\|^2 \\quad \\mbox{and} \\quad\n\t\t\\|\\m{v}_1+\\m{v}_2\\|^2 \\leq (1+\\eta) \\|\\m{v}_1\\|^2+ \\left(1+\\frac{1}{\\eta}\\right)\\|\\m{v}_2\\|^2.\n\\]\n\\end{lemma}\n\n\\begin{lemma}\\label{important}\n\t(Jensen's inequality) For any set of vectors $\\{\\m{v}_i\\}_{i=1}^n \\subset \\R^p$, we have\n\t$$\\left\\|\\frac{1}{n}\\sum_{i=1}^n\\m{v}_i\\right\\|^2\\leq\\frac{1}{n}\\sum_{i=1}^n\\left\\|\\m{v}_i\\right\\|^2.$$ \n\\end{lemma}\n\nwhere\n\\begin{equation*}\n\t{\\bm{\\beta}^{t+1}} = \\left[ {\\begin{array}{*{20}{c}}\n\t\t\t{{ \\beta_1^{t+1}}\\m{I}_p\n\t\t\t}&{}&{}\\\\\n\t\t\t{}&\\ddots&{}\\\\\n\t\t\t{}&{}&{\\beta_n^{t+1}}\\m{I}_p\n\t\\end{array}} \\right] \\in \\R^{np \\times np}.\n\\end{equation*}\n\n\nSpecifically, we introduce a variable $\\m{v}_i$ on node $i$ to track the local gradient average by a dynamical average consensus technique \\cite{sayed2014diffusion}.\nInitializing  $\\m{v}_i^0=\\m{g}_i^0$, we use the ATC structure to update  $\\m{v}_i^{t+1}$ as\n\\begin{equation}\\label{tracking}\n\t\t\\m{v}_i^{t+1}=\\sum_{j \\in \\mathcal{N}_i}\\tilde{W}_{ij}(\\m{v}_j^{t}+\\m{g}_j^{t+1}-\\m{g}_j^{t}).\n\\end{equation}\n\n\\begin{algorithm}[htb]\n\t\\caption{NDCG with respect to node $i$}\n\t\\label{alg:Framwork+}\n\t\\begin{algorithmic}[1]\n\t\t\\Require\n\t\tInitial point $\\m{x}_i^0$,  Maximum iteration T, Stepsize $\\alpha>0$,\n\t\t Mixing matrix $\\tilde{\\m{W}}$.\n\t\t\\State Set $t=0$,  $\\m{v}_i^0=\\m{g}_i^0$ and $\\tilde{\\m{d}}_i^{0}=-\\tilde{\\m{v}}_i^{0}=-{\\m{g}}_i^{0}-\\frac{1}{\\alpha}(\\m{x}_i^{0}-\\sum_{j \\in \\mathcal{N}_i} \\tilde{W}_{i j} \\m{x}_j^{0})$.\n\t\t\\State If $t \\geq T$, stop.\n\t\t\\State $\\m{x}_i^{t+1}=\\m{x}_i^{t}+\\alpha \\tilde{\\m{d}}_i^{t}$.\n\t\t\\State \t$\t\\m{v}_i^{t+1}=\\sum_{j \\in \\mathcal{N}_i}\\tilde{W}_{ij}(\\m{v}_j^{t}+\\m{g}_j^{t+1}-\\m{g}_j^{t})$.\n\t\t\\State $\\tilde{\\m{v}}_i^{t+1} = \\m{v}_i^{t+1}+\\frac{1}{\\alpha}(\\m{x}_i^{t+1}-\\sum_{j \\in \\mathcal{N}_i} \\tilde{W}_{i j} \\m{x}_j^{t+1})$.\n\t\t\\State $\\beta_i^{t+1}=\\frac{(\\tilde{\\m{v}}^{t+1}_i)\\tr ({\\m{g}}^{t+1}_i-{\\m{g}}^t_i)}{\\Vert \\tilde{\\m{v}}^t_i \\Vert ^2}$.\n\t\t\\State $\\tilde{\\m{d}}_i^{t+1}=-\\tilde{\\m{v}}_i^{t+1}+\\beta_i^{t+1} \\tilde{\\m{d}}_i^{t}$. \n\t\t\\State Set $t=t+1$ and go to Step 2.\n\t\t\\Ensure\n\t\t$\\m{x}^T$.\n\t\\end{algorithmic}\n\\end{algorithm}\n\n\\begin{equation}\\label{Potential-P}\n\tP(\\m{x}^{t},\\m{v}^{t})=F(\\bar{\\m{x}}^t)+\\frac{1}{2\\alpha n} \\|\\m{x}^{t}\\|^2_{\\m{I}-\\m{W}}+\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2+\\|\\m{v}^{t}-\\m{M}\\m{v}^{t}\\|^2.\n\\end{equation}\n\n\\begin{lemma}\\label{lem2.5a}\n\t Assume $\\alpha$ is some constant in $(0,\\frac{1}{4L}]$. Define the sequence $\\left\\{\\xi^t\\right\\}$ as\n \\begin{equation}\\label{def-xi}\n\t\\xi^0=1 ; \\quad \\xi^{t+1}=1+L \\alpha (\\xi^t)^2, \\quad t \\geq 0 .\n \\end{equation}\n\tThen, we have $ 1 \\leq \\xi^t<c $ for all $ t \\geq 0$,\n\twhere $ c \\in (0,2] $ is a constant given as\n\\begin{equation}\\label{const-c}\n\tc=2\\left(1+\\sqrt{1-4L \\alpha}\\right)^{-1}.\n\\end{equation}\n\\end{lemma}","units":[{"start":5523,"end":5815,"start_line":134,"end_line":137,"raw_tex":"In this paper, we consider the following decentralized optimization problem over an undirected and connected network containing $n$ nodes\n\\begin{equation}\\label{obj_fun1}\n\t\\mathop {\\min } \\limits_{\\m{z} \\in {\\R^p}} \\; F(\\m{z}):= \\frac{1}{n}\\sum\\limits_{i = 1}^n {{f_i}(\\m{z})},\n\\end{equation}","sha256_utf8":"5259da680cb9d347af46c462e68394ac129f50087bacd303d49de305e0f435d9"},{"start":5816,"end":6111,"start_line":138,"end_line":139,"raw_tex":"where the local objective function $f_i:\\R^{p} \\rightarrow \\R$ is  continuously differentiable. \nConsider the underlying network $\\mathcal{G}=\\left(\\mathcal{V},\\mathcal{E}\\right)$, where $\\mathcal{V} = \\{1,\\ldots,n\\}$ is the set of nodes, and $\\mathcal{E}$ is the collection of unordered edges.\n","sha256_utf8":"8ec4eb935cd79aedd6d76d8d0315e42d00979c22a9283ff723ee47ea75c5d3e0"},{"start":15799,"end":16094,"start_line":213,"end_line":214,"raw_tex":"We use uppercase boldface letters, e.g.~$\\m{W}$, for matrices and lowercase boldface letters,  e.g.~$\\m{w}$, for vectors.\nFor any vectors $\\m{v}_i \\in \\R^p$, $i=1,\\ldots,n$, we define  $\\bar{\\m{v}}=\\frac{1}{n} \\sum_{i=1}^n \\m{v}_i$ and $\\m{v} = [\\m{v}_1; \\m{v}_2; \\ldots, \\m{v}_n] \\in \\R^{np}$.\n","sha256_utf8":"97feb68bd92c1eb0a59cb5a1c582a3c12781eaf862e808999846851435e4b813"},{"start":16277,"end":16938,"start_line":223,"end_line":228,"raw_tex":"Given an undirected network  $\\mathcal{G}=\\left(\\mathcal{V},\\mathcal{E}\\right)$,\n let $\\m{x}_i$ denote the local copy of the global variable $\\m{z}$ at node $i$ and $\\mathcal{N}_i$ denote the set consisting of the neighbors of node $i$ \n(for convenience, we treat node $i$ itself as one of its neighbors). \nWe define $f(\\m{x}) = \\sum_{i = 1}^n {{f_i}({\\m{x}_i})}$ and use $\\m{g}^t$, $\\m{g}_i^t$ to stand for $\\nabla f(\\m{x}^t)$, $\\nabla f_i(\\m{x}^t_{i})$ respectively,\nwhere, for clarification, the gradient of $f(\\m{x})$ is defined as\n$ \\nabla f(\\m{x}) =\\left[ \\nabla f_1(\\m{x}_{1}); \\nabla f_2(\\m{x}_{2}); \\ldots, \\nabla f_n(\\m{x}_{n}) \\right] \\in \\R^{np}$. \n","sha256_utf8":"609cd7ea155d89a06aba191d61ce566b29aa7b1ae1d7c0eb30be3f06a062f897"},{"start":17150,"end":18399,"start_line":237,"end_line":246,"raw_tex":"In addition,  we define $\\overline{\\nabla} f(\\m{x}^t)=\\frac{1}{n} \\sum_{i=1}^n \\nabla f_i(\\m{x}_i^t) \n\\in \\R^p$.\nWe say that $\\m{x}$ is consensual or gets consensus if ${\\m{x}_1}={\\m{x}_2}=\\ldots={\\m{x}_n}$. \n$\\m{I}_p$ denotes the $p \\times p$ identity matrix and $\\m{I}$ denote $\\m{I}_{np}$ for simplicity. Kronecker Product is denoted as $\\otimes$.\nGiven a vector $\\m{v}$ and a matrix $\\m{N}$, $\\operatorname{span}(\\m{v})$ stands for the linear subspace spanned by $\\m{v}$;  $\\left\\|\\m{v}\\right\\|_{\\m{N}}^2$ denotes $\\m{v}\\tr \\m{N} \\m{v}$;\n$\\mbox{Null}(\\m{N})$ and $\\m{N}\\tr$  denote the null space and transpose of $\\m{N}$, respectively;\n${\\lambda _{\\min }(\\m{N})}$,  ${\\lambda _{\\max }(\\m{N})}$, and $\\rho(\\m{N})$ denote smallest eigenvalue, largest eigenvalue, and  the spectral radius of $\\m{N}$,  respectively;\nFor matrices $\\m{N}_1$ and $\\m{N}_2$ with same dimension, $\\m{N}_1 \\succeq \\m{N}_2$ means $\\m{N}_1 - \\m{N}_2$ is positive semidefinite, while $\\m{N}_1 \\ge \\m{N}_2$ means $\\m{N}_1 - \\m{N}_2$ is component-wise nonnegative.\nWe denote $\\operatorname{log}_{10}(\\cdot)$ by $\\operatorname{log}(\\cdot)$  and define $\\m{M}=\\frac{1}{n} \\m{1}_n\\m{1}_n\\tr\\otimes \\m{I}_p$ where $\\m{1}_n \\in \\R^n$ denotes the\nvector with all components ones. \n","sha256_utf8":"c7e047bba7211f8c293232b24aba3da89264732f33d341cfacab8d3a0d332053"},{"start":19621,"end":20033,"start_line":256,"end_line":262,"raw_tex":"\\begin{assum}\\label{as0}\nThe local objective functions $\\{f_i \\}_{i=1}^n$ are bounded below and Lipschitz continuously differentiable, that is, for any $\\m{z}, \\tilde{\\m{z}} \\in \\R^p$ and $i=1, \\ldots,n$, \nwe have  $f_i(z)>-\\infty$ and \n\t\\begin{equation}\\label{3.1}\n\t\t\\left\\| {\\nabla {f_i}(\\m{z}) - \\nabla {f_i}(\\tilde{\\m{z}})} \\right\\| \\le L \\left\\| {\\m{z} - \\tilde{\\m{z}}} \\right\\|,\n\t\\end{equation}\n\\end{assum}","sha256_utf8":"5ac8ac2405944a96f093b7e53e6485acdca3ccf2cd5a1e72d5ae06851306d97c"},{"start":20034,"end":20072,"start_line":263,"end_line":263,"raw_tex":"where $L >0$ is a Lipschitz constant.\n","sha256_utf8":"3ab72b3458c989056443cf709fcf9438b39a4a6fd6b18af1cf4fe0f560442e87"},{"start":20255,"end":20780,"start_line":267,"end_line":273,"raw_tex":"\\begin{defin}\\label{mix}\n\t(Mixing matrix $\\tilde{\\m{W}}$ for given network $\\mathcal{G} = \\left(\\mathcal{V},\\mathcal{E}\\right)$)\n\t\\begin{itemize}\n\t\t\\item [1.] $\\tilde{\\m{W}}$ is nonnegative, where each component $\\tilde{W}_{ij}$ characterizes the active link $(i,j)$, i.e., $\\tilde{{W}}_{i,j}>0$ if $j \\in \\mathcal{N}_i$;  $\\tilde{{W}}_{i,j}=0$, otherwise.\n\t\t\\item [2.] $\\tilde{\\m{W}}$ is symmetric and doubly stochastic, i.e., $\\tilde{\\m{W}}=\\tilde{\\m{W}}\\tr$ and $\\tilde{\\m{W}}\\m{1}_n=\\m{1}_n$.\\\\\n\t\\end{itemize}\n\\end{defin}","sha256_utf8":"8dcfd408850b275092b337f2fadd7e9d0e85ea32995ab5ad42e8fe83527f8a9a"},{"start":21008,"end":21843,"start_line":276,"end_line":287,"raw_tex":"Let $\\lambda_{i}(\\tilde{\\m{W}})$ denote the $i$-th largest eigenvalue of $\\tilde{\\m{W}}$ and $\\sigma$ be the second largest magnitude eigenvalue of $\\tilde{\\m{W}}$. \nThen, the following properties hold \\cite{xin2019distributed}.\n\\begin{lemma}\\label{property W}\nFor $\\tilde{\\m{W}}$ defined in Definition~\\ref{mix} and  $\\m{W} :=\\tilde{\\m{W}}\\otimes\\m{I}_p$, we have\n\t\\begin{itemize}\n\t\t\\item [1.] $1=\\lambda_{1}(\\tilde{\\m{W}})>\\lambda_{2}(\\tilde{\\m{W}})\\geq\\ldots\\geq\\lambda_{n}(\\tilde{\\m{W}})>-1$;\n\t\t\\item [2.] $\t0<\\rho(\\m{W}-\\m{M})=\\sigma=\\max \\left\\{|\\lambda_{2}(\\tilde{\\m{W}})|, |\\lambda_{n}(\\tilde{\\m{W}})|\\right\\}<1$;\n\t\t\\item [3.] $\\m{M}=\\m{M}\\m{W}=\\m{W}\\m{M}$;\n\t\t\\item [4.]  $\\|\\m{W}\\m{x}-\\m{M}\\m{x}\\|=\\|(\\m{W}-\\m{M})(\\m{x}-\\m{M}\\m{x})\\| \\leq \\sigma \\|\\m{x}-\\m{M}\\m{x}\\|$\n\t\tfor any $\\m{x} \\in \\R^{np}$. \n\t\\end{itemize}\n\\end{lemma}","sha256_utf8":"0939f6b14e656a37ea72ca5db03c3d35484f37e146495899f3d5d954fd02d109"},{"start":21921,"end":22244,"start_line":290,"end_line":296,"raw_tex":"\\begin{lemma}\\label{young}\n\t(Young's inequality) For any two vectors $\\m{v}_1,\\m{v}_2\\in \\R^p$, $\\eta>0$, we have\n\\[\n2\\m{v}_1\\tr\\m{v}_2 \\leq \\eta \\|\\m{v}_1\\|^2+ \\frac{1}{\\eta}\\|\\m{v}_2\\|^2 \\quad \\mbox{and} \\quad\n\t\t\\|\\m{v}_1+\\m{v}_2\\|^2 \\leq (1+\\eta) \\|\\m{v}_1\\|^2+ \\left(1+\\frac{1}{\\eta}\\right)\\|\\m{v}_2\\|^2.\n\\]\n\\end{lemma}","sha256_utf8":"78966ba658b836386c3232093d9f7f16cc26865192baeee05cd26db5218f042f"},{"start":22245,"end":22483,"start_line":297,"end_line":300,"raw_tex":"\\begin{lemma}\\label{important}\n\t(Jensen's inequality) For any set of vectors $\\{\\m{v}_i\\}_{i=1}^n \\subset \\R^p$, we have\n\t$$\\left\\|\\frac{1}{n}\\sum_{i=1}^n\\m{v}_i\\right\\|^2\\leq\\frac{1}{n}\\sum_{i=1}^n\\left\\|\\m{v}_i\\right\\|^2.$$ \n\\end{lemma}","sha256_utf8":"a17fd65e54df2583d009b6431e4484a829c53b52f2c1836fe48c10182065ea1e"},{"start":25114,"end":25344,"start_line":337,"end_line":345,"raw_tex":"where\n\\begin{equation*}\n\t{\\bm{\\beta}^{t+1}} = \\left[ {\\begin{array}{*{20}{c}}\n\t\t\t{{ \\beta_1^{t+1}}\\m{I}_p\n\t\t\t}&{}&{}\\\\\n\t\t\t{}&\\ddots&{}\\\\\n\t\t\t{}&{}&{\\beta_n^{t+1}}\\m{I}_p\n\t\\end{array}} \\right] \\in \\R^{np \\times np}.\n\\end{equation*}\n","sha256_utf8":"eb0f95d2229c7e5324fd43e23c4a7709a128d0e68272c06291ab821ed6d882c0"},{"start":33131,"end":33533,"start_line":430,"end_line":434,"raw_tex":"Specifically, we introduce a variable $\\m{v}_i$ on node $i$ to track the local gradient average by a dynamical average consensus technique \\cite{sayed2014diffusion}.\nInitializing  $\\m{v}_i^0=\\m{g}_i^0$, we use the ATC structure to update  $\\m{v}_i^{t+1}$ as\n\\begin{equation}\\label{tracking}\n\t\t\\m{v}_i^{t+1}=\\sum_{j \\in \\mathcal{N}_i}\\tilde{W}_{ij}(\\m{v}_j^{t}+\\m{g}_j^{t+1}-\\m{g}_j^{t}).\n\\end{equation}","sha256_utf8":"73873f72d9fc7afa37b6403931487c02112ea0cb2c8f5823ac79a223e6674ff3"},{"start":34742,"end":35815,"start_line":453,"end_line":471,"raw_tex":"\\begin{algorithm}[htb]\n\t\\caption{NDCG with respect to node $i$}\n\t\\label{alg:Framwork+}\n\t\\begin{algorithmic}[1]\n\t\t\\Require\n\t\tInitial point $\\m{x}_i^0$,  Maximum iteration T, Stepsize $\\alpha>0$,\n\t\t Mixing matrix $\\tilde{\\m{W}}$.\n\t\t\\State Set $t=0$,  $\\m{v}_i^0=\\m{g}_i^0$ and $\\tilde{\\m{d}}_i^{0}=-\\tilde{\\m{v}}_i^{0}=-{\\m{g}}_i^{0}-\\frac{1}{\\alpha}(\\m{x}_i^{0}-\\sum_{j \\in \\mathcal{N}_i} \\tilde{W}_{i j} \\m{x}_j^{0})$.\n\t\t\\State If $t \\geq T$, stop.\n\t\t\\State $\\m{x}_i^{t+1}=\\m{x}_i^{t}+\\alpha \\tilde{\\m{d}}_i^{t}$.\n\t\t\\State \t$\t\\m{v}_i^{t+1}=\\sum_{j \\in \\mathcal{N}_i}\\tilde{W}_{ij}(\\m{v}_j^{t}+\\m{g}_j^{t+1}-\\m{g}_j^{t})$.\n\t\t\\State $\\tilde{\\m{v}}_i^{t+1} = \\m{v}_i^{t+1}+\\frac{1}{\\alpha}(\\m{x}_i^{t+1}-\\sum_{j \\in \\mathcal{N}_i} \\tilde{W}_{i j} \\m{x}_j^{t+1})$.\n\t\t\\State $\\beta_i^{t+1}=\\frac{(\\tilde{\\m{v}}^{t+1}_i)\\tr ({\\m{g}}^{t+1}_i-{\\m{g}}^t_i)}{\\Vert \\tilde{\\m{v}}^t_i \\Vert ^2}$.\n\t\t\\State $\\tilde{\\m{d}}_i^{t+1}=-\\tilde{\\m{v}}_i^{t+1}+\\beta_i^{t+1} \\tilde{\\m{d}}_i^{t}$. \n\t\t\\State Set $t=t+1$ and go to Step 2.\n\t\t\\Ensure\n\t\t$\\m{x}^T$.\n\t\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"886376e45c8f9083d14b890e9a2d34c16ffa1b5fc6bafaeee829c15bbbe7df83"},{"start":39011,"end":39215,"start_line":521,"end_line":523,"raw_tex":"\\begin{equation}\\label{Potential-P}\n\tP(\\m{x}^{t},\\m{v}^{t})=F(\\bar{\\m{x}}^t)+\\frac{1}{2\\alpha n} \\|\\m{x}^{t}\\|^2_{\\m{I}-\\m{W}}+\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2+\\|\\m{v}^{t}-\\m{M}\\m{v}^{t}\\|^2.\n\\end{equation}","sha256_utf8":"dbb590d7a4448451400e1b49ade5b868cdc9021fca140687273b820c69d2f5c8"},{"start":40035,"end":40485,"start_line":533,"end_line":543,"raw_tex":"\\begin{lemma}\\label{lem2.5a}\n\t Assume $\\alpha$ is some constant in $(0,\\frac{1}{4L}]$. Define the sequence $\\left\\{\\xi^t\\right\\}$ as\n \\begin{equation}\\label{def-xi}\n\t\\xi^0=1 ; \\quad \\xi^{t+1}=1+L \\alpha (\\xi^t)^2, \\quad t \\geq 0 .\n \\end{equation}\n\tThen, we have $ 1 \\leq \\xi^t<c $ for all $ t \\geq 0$,\n\twhere $ c \\in (0,2] $ is a constant given as\n\\begin{equation}\\label{const-c}\n\tc=2\\left(1+\\sqrt{1-4L \\alpha}\\right)^{-1}.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"100734a76fdb509dd23d461b334469bda2ea6040fa344c3ee0c7d84a87bbbe88"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{nonconvex_convergence}\n\tSuppose Assumption \\ref{as0} holds. If\n\t\\begin{align}\\label{alpha}\n\t0 <\t\\alpha < \\min \\left\\{\\frac{(1-{\\sigma}^2)n}{32L^2},\\frac{2(5-2\\sqrt{5})(1-{\\sigma}^2)}{4n(5\\sigma^2L^2+18-8\\sqrt{5})+5L},\\frac{2\\sqrt{5}-4}{5L}\\right\\},\n\t\\end{align}\n\tthen we have  \n\t\\begin{equation}\\label{cg-converge-rate}\n\t \\min_{0 \\le t \\le T} \\left\\{ \\|\\tilde{\\m{v}}^t\\|^2+\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2 \\right\\}\n\t\t\\leq \\frac{P(\\m{x}^{0};\\m{v}^{0})-P(\\m{x}^{T};\\m{v}^{T})}{\\gamma \\min\\{\\alpha,1\\}T},\n\t\\end{equation}\n\twhere $P$ is the potential function defined in \\eqref{Potential-P} and\n\t$\\gamma >0$ is a constant depending only on $n$, $L$, and $\\sigma$.\n\\end{theorem}","units":[{"start":49236,"end":49959,"start_line":661,"end_line":673,"raw_tex":"\\begin{theorem}\\label{nonconvex_convergence}\n\tSuppose Assumption \\ref{as0} holds. If\n\t\\begin{align}\\label{alpha}\n\t0 <\t\\alpha < \\min \\left\\{\\frac{(1-{\\sigma}^2)n}{32L^2},\\frac{2(5-2\\sqrt{5})(1-{\\sigma}^2)}{4n(5\\sigma^2L^2+18-8\\sqrt{5})+5L},\\frac{2\\sqrt{5}-4}{5L}\\right\\},\n\t\\end{align}\n\tthen we have  \n\t\\begin{equation}\\label{cg-converge-rate}\n\t \\min_{0 \\le t \\le T} \\left\\{ \\|\\tilde{\\m{v}}^t\\|^2+\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2 \\right\\}\n\t\t\\leq \\frac{P(\\m{x}^{0};\\m{v}^{0})-P(\\m{x}^{T};\\m{v}^{T})}{\\gamma \\min\\{\\alpha,1\\}T},\n\t\\end{equation}\n\twhere $P$ is the potential function defined in \\eqref{Potential-P} and\n\t$\\gamma >0$ is a constant depending only on $n$, $L$, and $\\sigma$.\n\\end{theorem}","sha256_utf8":"04fe87b5b00adec6efaeaad347cb8758863937fa35bc6b621b96f11d33a24bab"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"By the above formula \\eqref{tracking} and Lemma~\\ref{property W}, it follows from induction that\n\\begin{equation}\\label{vequalg}\n\\m{M}{\\m{v}}^t=\\m{M}{\\m{g}}^t \\quad \\Longleftrightarrow \\quad \\bar{\\m{v}}^t=\\bar{\\m{g}}^t,\n\\end{equation} \nwhich shows the average of $\\m{v}_i^t$ over $i$ is the same as the local gradient average.\n\n\n\\begin{lemma}\\label{lem2.5}\n\tSuppose Assumption \\ref{as0} holds. Consider the NDCG algorithm with  $ \\alpha \\in(0,\\frac{1}{4L}]$. Then, for all $t \\geq 0$ and any $i$, we have\n\t\\begin{align}\n\t\t&\\left(2-\\xi^t\\right)\\left\\|\\tilde{\\m{v}}_i^t\\right\\|^2  \\leq -(\\tilde{\\m{v}}_i^t)\\tr\\tilde{\\m{d}}_i^t \\leq \\xi^t\\left\\|\\tilde{\\m{v}}_i^t\\right\\|^2, \\label{descent1}\\\\\n\t\t&\\left(2-\\xi^t\\right)\\left\\|\\tilde{\\m{v}}_i^t\\right\\| \\leq\\left\\|\\tilde{\\m{d}}_i^t\\right\\| \\leq \\xi^t\\left\\|\\tilde{\\m{v}}_i^t\\right\\|\\label{descent2},\n\t\\end{align}\n\twhere $\\xi^t$ is the sequence defined in \\eqref{def-xi}.\n\\end{lemma}\n\n\\begin{proof}\n\tSince $\\tilde{\\m{d}}_i^0=-\\tilde{\\m{v}}_i^0$ and $\\xi^0=1$, \\eqref{descent1} and \\eqref{descent2} clearly hold for $t=0$. \nAssume that \\eqref{descent1} and \\eqref{descent2} hold for some $t$. Then by the Cauchy-Schwartz inequality,  $\\m{x}_i^{t+1}=\\m{x}_i^{t}+\\alpha \\tilde{\\m{d}}_i^{t}$ and \\eqref{3.1}, \nwe have\n\t\\begin{align}\\label{lem2.5_eq1}\n\t\t\\|\\tilde{\\m{d}}_i^{t+1}+\\tilde{\\m{v}}_i^{t+1}\\|=&\\|\\beta_i^{t+1} \\tilde{\\m{d}}_i^{t}\\|\n\t\t\\leq \\frac{\\|{\\m{g}}^{t+1}_i-{\\m{g}}^{t}_i\\| \\|\\tilde{\\m{v}}^{t+1}_i\\|}{\\|\\tilde{\\m{v}}^{t}_i\\|^2}\\|\\tilde{\\m{d}}_i^{t}\\| \\\\\n\t\t\\leq &\\frac{L\\alpha \\|\\tilde{\\m{d}}_i^{t}\\|^2}{\\|\\tilde{\\m{v}}^{t}_i\\|^2}\\|\\tilde{\\m{v}}^{t+1}_i\\|\n\t\t\\leq L\\alpha (\\xi^t)^2\\|\\tilde{\\m{v}}^{t+1}_i\\|.\\notag\n\t\\end{align}\n\tBy the triangular inequality and the above relation, we obtain\n\t\\begin{align*}\n\t\t\\|\\tilde{\\m{d}}_i^{t+1}\\|\\leq \\|\\tilde{\\m{v}}_i^{t+1}\\|+\\|\\tilde{\\m{d}}_i^{t+1}+\\tilde{\\m{v}}_i^{t+1}\\|\\leq \\left(1+L\\alpha (\\xi^t)^2\\right)\\|\\tilde{\\m{v}}^{t+1}_i\\|\n\t\\end{align*}\n\tand\n\t\\begin{align*}\n\t\t\\|\\tilde{\\m{d}}_i^{t+1}\\|\\geq \\|\\tilde{\\m{v}}_i^{t+1}\\|-\\|\\tilde{\\m{d}}_i^{t+1}+\\tilde{\\m{v}}_i^{t+1}\\|\n\t\t\\geq \\left(1-L\\alpha (\\xi^t)^2\\right)\\|\\tilde{\\m{v}}^{t+1}_i\\|.\n\t\\end{align*}\n\tThen, it follows from the recursion  \\eqref{def-xi} of $\\xi^t$ and the choice of $\\alpha$ that \\eqref{descent2} holds for $t+1$. \n       By \\eqref{lem2.5_eq1} and the Cauchy-Schwartz inequality, we have that\n\t\\begin{align*}\n\t\t\\vert (\\tilde{\\m{v}}_i^{t+1})\\tr\\tilde{\\m{d}}_i^{t+1}+\\|\\tilde{\\m{v}}_i^{t+1}\\|^2 \\vert \n\t\t\\leq \\|\\tilde{\\m{v}}_i^{t+1}\\| \\|\\tilde{\\m{d}}_i^{t+1}+\\tilde{\\m{v}}_i^{t+1}\\|\n\t\t\\leq  L\\alpha (\\xi^t)^2\\|\\tilde{\\m{v}}^{t+1}_i\\|^2.\n\t\\end{align*}\n\tSimilarly, by the recursion  \\eqref{def-xi} of $\\xi^t$ and the choice of $\\alpha$, \\eqref{descent1} holds for $t+1$.\n\\end{proof}\n\n\\begin{lemma}\n\tSuppose Assumption~\\ref{as0} holds. If $ \\alpha \\in(0,\\frac{2\\sqrt{5}-4}{5L}]$, then for all $t \\geq 0$ we have\n\t\\begin{align}\n\t\tF(\\bar{\\m{x}}^{t+1})+\\frac{1}{2\\alpha n} \\|\\m{x}^{t+1}\\|^2_{\\m{I}-\\m{W}}\\leq& F(\\bar{\\m{x}}^t)+\\frac{1}{2\\alpha n} \\|\\m{x}^{t}\\|^2_{\\m{I}-\\m{W}}-\\left(\\frac{\\alpha(5-2\\sqrt{5})}{4 n}-\\frac{5L\\alpha^2}{8n}\\right)\\|\\tilde{\\m{v}}^t\\|^2 \\notag \\\\\n&+\\frac{16L^2\\alpha}{n}\\|\\m{x}^t-\\m{M}\\m{x}^t\\|^2+\\frac{16\\alpha}{n}\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2. \\label{term1}\n\t\\end{align} \n\\end{lemma}\n\n\\begin{proof}\n\tBy Assumption \\ref{as0},  $F$ is Lipschitz continuously differentiable with Lipschitz constant $L$, which gives\n\t\\begin{align*}\n\t\tF(\\bar{\\m{x}}^{t+1})\\leq F(\\bar{\\m{x}}^t)+\\Big\\langle\\nabla F(\\bar{\\m{x}}^t),\\bar{\\m{x}}^{t+1}-\\bar{\\m{x}}^{t}\\Big\\rangle+\\frac{L}{2}\\|\\bar{\\m{x}}^{t+1}-\\bar{\\m{x}}^{t}\\|^{2}.\n\t\\end{align*}\n\tSince $\\rho(\\m{I}-\\m{W})<2$, we  have\n\t\\begin{align*}\n\t\t\\frac{1}{2\\alpha n} \\|\\m{x}^{t+1}\\|^2_{\\m{I}-\\m{W}} \\leq \\frac{1}{2\\alpha n} \\|\\m{x}^{t}\\|^2_{\\m{I}-\\m{W}}+\\frac{1}{n} \\Big\\langle \\frac{1}{\\alpha} (\\m{I}-\\m{W})\\m{x}^t, \\m{x}^{t+1}-\\m{x}^t\\Big\\rangle + \\frac{1}{2\\alpha n}\\|\\m{x}^{t+1}-\\m{x}^t\\|^2.\n\t\\end{align*}\nIn addition, we have\n\\begin{align*}\n\t&\\frac{L}{2}\\|\\bar{\\m{x}}^{t+1}-\\bar{\\m{x}}^{t}\\|^{2}=\\frac{L}{2n}\\|\\m{1}\\otimes\\bar{\\m{x}}^{t+1}-\\m{1}\\otimes\\bar{\\m{x}}^{t}\\|^{2}=\\frac{L}{2n}\\|\\m{M}{\\m{x}}^{t+1}-\\m{M}{\\m{x}}^{t}\\|^{2}\\leq\\frac{L}{2n}\\|{\\m{x}}^{t+1}-{\\m{x}}^{t}\\|^{2}.\n\\end{align*}\n\tAdding the above three inequalities together and using Lemma~\\ref{lem2.5} yields \n\t\\begin{eqnarray}\\label{add}\n\t   & & F(\\bar{\\m{x}}^{t+1})+\\frac{1}{2\\alpha n} \\|\\m{x}^{t+1}\\|^2_{\\m{I}-\\m{W}} \\\\\n  &\\leq& F(\\bar{\\m{x}}^t)+\\frac{1}{2\\alpha n} \\|\\m{x}^{t}\\|^2_{\\m{I}-\\m{W}}+\\left(\\frac{L\\alpha^2c^2}{2n}+\\frac{\\alpha c^2}{2 n}\\right) \\|\\tilde{\\m{v}}^t\\|^2 + \\mbox{RemainTerm} \\notag \n\t\\end{eqnarray}\n\twhere \n\t\\begin{align}\n\t\t&\\mbox{RemainTerm} = \\Big\\langle\\nabla F(\\bar{\\m{x}}^t),\\bar{\\m{x}}^{t+1}-\\bar{\\m{x}}^{t}\\Big\\rangle + \\frac{1}{n} \\Big\\langle \\frac{1}{\\alpha} (\\m{I}-\\m{W})\\m{x}^t, \\m{x}^{t+1}-\\m{x}^t\\Big\\rangle \\notag\\\\\n\t\t=&\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\nabla F(\\bar{\\m{x}}^t),{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle+\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle \\frac{1}{\\alpha} (\\m{x}_i^{t}-\\sum_{j \\in \\mathcal{N}_i} \\tilde{W}_{i j} \\m{x}_j^{t}), \\m{x}_i^{t+1}-\\m{x}_i^t\\Big\\rangle \\notag\\\\\n\t\t=&\\underbrace{\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\tilde{\\m{v}}_i^t,{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle}_{\\text {term (A)}}\n\t\t+\\underbrace{\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\nabla F(\\bar{\\m{x}}^t)-\\overline{\\nabla} f({\\m{x}}^t),{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle}_{\\text {term (B)}}\\notag\\\\\n\t\t&+\\underbrace{\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\overline{\\nabla} f({\\m{x}}^t)-{\\m{v}}_i^t,{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle}_{\\text {term (C)}}. \\label{terms}\n\t\\end{align}\n\tFor term (A), by Lemma~\\ref{lem2.5} and $\\m{x}_i^{t+1}=\\m{x}_i^{t}+\\alpha \\tilde{\\m{d}}_i^{t}$, we have\n\t\\begin{align}\\label{termA}\n\t\t\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\tilde{\\m{v}}_i^t,{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle \\leq -\\frac{\\alpha(2-c)}{ n} \\|\\tilde{\\m{v}}^t\\|^2.\n\t\\end{align}\n\tFor term (B), by Young's inequality with constant $b >0$ and  $\\m{1}\\otimes\\bar{\\m{x}}^t=\\m{M}{\\m{x}}^t$, we have \n\t\\begin{eqnarray}\\label{termB}\n\t  &&\t\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\nabla F(\\bar{\\m{x}}^t)-\\overline{\\nabla} f({\\m{x}}^t),{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle \\\\\n  &\\leq& \\frac{b}{2}\\|\\nabla F(\\bar{\\m{x}}^t)-\\overline{\\nabla} f({\\m{x}}^t)\\|^2+\\frac{1}{2nb}\\|{\\m{x}}^{t+1}-{\\m{x}}^{t}\\|^2 \\notag\\\\\n\t\t&\\leq & \\frac{L^2b}{2n}\\|\\m{1}\\otimes\\bar{\\m{x}}^t-\\m{x}^t \\|^2+\\frac{2\\alpha^2}{nb}\\|\\tilde{\\m{v}}^t\\|^2\n\t   =\\frac{L^2b}{2n}\\|\\m{M}\\m{x}^t-\\m{x}^t \\|^2+\\frac{2\\alpha^2}{nb}\\|\\tilde{\\m{v}}^t\\|^2,\\notag\n\t\\end{eqnarray}\nwhere the last inequality uses Lemma~\\ref{important}, Lemma~\\ref{lem2.5},  $L$-Lipschitz continuity of $\\nabla F$.\n\tFor term (C), by Young's inequality with constant $d>0$, we have\n\t\\begin{align}\\label{termC}\n\t\t\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\overline{\\nabla} f({\\m{x}}^t)-{\\m{v}}_i^t,{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle \\leq &\\frac{d}{2n}\\|\\m{1}\\otimes\\overline{\\nabla} f({\\m{x}}^t)-{\\m{v}}^t\\|^2+\\frac{1}{2nd}\\|{\\m{x}}^{t+1}-{\\m{x}}^{t}\\|^2 \\notag \\\\\n\t\t\\leq & \\frac{d}{2n}\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+\\frac{2\\alpha^2}{nd}\\|\\tilde{\\m{v}}^t\\|^2,\n\t\\end{align}\nwhere the last inequality is due to Lemma~\\ref{lem2.5} and the relation $\\m{1}\\otimes\\overline{\\nabla} f({\\m{x}}^t)=\\m{M}{\\m{v}}^t$ by \\eqref{vequalg}.\n\tSubstituting \\eqref{termA}, \\eqref{termB}, and \\eqref{termC} into \\eqref{terms}, we obtain\n\t\\begin{align*}\n\t\t&\\Big\\langle\\nabla F(\\bar{\\m{x}}^t),\\bar{\\m{x}}^{t+1}-\\bar{\\m{x}}^{t}\\Big\\rangle+\\frac{1}{n} \\Big\\langle \\frac{1}{\\alpha} (\\m{I}-\\m{W})\\m{x}^t, \\m{x}^{t+1}-\\m{x}^t\\Big\\rangle\\\\\n\t\t\\leq &-\\left(\\frac{\\alpha(2-c)}{ n}-\\frac{2\\alpha^2}{n}\\left(\\frac{1}{b}+\\frac{1}{d}\\right)\\right)\\|\\tilde{\\m{v}}^t\\|^2\n\t\t+\\frac{L^2b}{2n}\\|\\m{x}^t-\\m{M}\\m{x}^t\\|^2+\\frac{d}{2n}\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2.\n\t\\end{align*}\n\tBy substituting the above inequality into \\eqref{add}, it holds that\n\t\\begin{eqnarray*}\n\t&& F(\\bar{\\m{x}}^{t+1})+\\frac{1}{2\\alpha n} \\|\\m{x}^{t+1}\\|^2_{\\m{I}-\\m{W}} \\\\ \n      &\\leq& F(\\bar{\\m{x}}^t)+\\frac{1}{2\\alpha n} \\|\\m{x}^{t}\\|^2_{\\m{I}-\\m{W}}\n\t-\\left(\\frac{\\alpha(4-2c-c^2)}{ 2n}-\\frac{2\\alpha^2}{n}\\left(\\frac{1}{b}+\\frac{1}{d}\\right)-\\frac{L\\alpha^2c^2}{2n}\\right)\\|\\tilde{\\m{v}}^t\\|^2\\\\\n\t&&+\\frac{L^2b}{2n}\\|\\m{x}^t-\\m{M}\\m{x}^t\\|^2+\\frac{d}{2n}\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2.\n\\end{eqnarray*} \nSince $0<\\alpha \\leq \\frac{2\\sqrt{5}-4}{5L}$, we have from the definition of constant $c$ in \\eqref{const-c} that \n\\begin{equation}\\label{new-bd-c}\n1<c\\leq \\sqrt{5}/2, \n\\end{equation}\n which implies $\\frac{11}{4}-\\sqrt{5}\\leq 4-2c-c^2 \\leq 1$. Then, setting $b=d=32\\alpha$ yields \\eqref{term1}.\n%\t\\begin{align*}\n%\t\tF(\\bar{\\m{x}}^{t+1})+\\frac{1}{2\\alpha n} \\|\\m{x}^{t+1}\\|^2_{\\m{I}-\\m{W}}\\leq& F(\\bar{\\m{x}}^t)+\\frac{1}{2\\alpha n} \\|\\m{x}^{t}\\|^2_{\\m{I}-\\m{W}}-\\left(\\frac{\\alpha(5-2\\sqrt{5})}{4 n}-\\frac{5L\\alpha^2}{8n}\\right)\\| \\tilde{\\m{v}}^t\\|^2\\\\\n%\t\t&+\\frac{16L^2\\alpha}{n}\\|\\m{x}^t-\\m{M}\\m{x}^t\\|^2+\\frac{16\\alpha}{n}\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2.\n%\t\\end{align*} \n\\end{proof}\n\n\\begin{proof}\n\tFirst, by using Young's inequality with constant $\\eta >0$, we \n\t establish a recursion upper bound on $\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2$.\n\t\\begin{align}\\label{term2}\n\t\t\\|{\\m{v}}^{t+1}-\\m{M}{\\m{v}}^{t+1}\\|^2\n\t\t=&\\| \\m{W}{\\m{v}}^{t}+\\m{W}\\m{g}^{t+1}-\\m{W}\\m{g}^{t} -\\m{M}\\m{v}^t-\\m{M}\\m{g}^{t+1}+\\m{M}\\m{g}^t\\|^2 \\notag\\\\\n\t\t\\leq & (1+\\eta)\\|\\m{W}{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+(1+1/\\eta)\\|(\\m{W}-\\m{M})(\\m{g}^{t+1}-\\m{g}^t)\\|^2\\notag\\\\\n\t\t\\leq & (1+\\eta)\\sigma^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+(1+1/\\eta)\\sigma^2L^2\\|\\m{x}^{t+1}-\\m{x}^t\\|^2,\\notag\\\\\n\t\t\\leq & (1+\\eta)\\sigma^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+(1+1/\\eta)\\sigma^2L^2 c^2 \\alpha^2\\|\\tilde{\\m{v}}^t\\|^2,\\notag\\\\\n\t\t\\leq & (1+\\eta)\\sigma^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+(1+1/\\eta)(5/4)\\sigma^2L^2\\alpha^2\\|\\tilde{\\m{v}}^t\\|^2,\n\t\\end{align}\nwhere the second inequality uses Lemma~\\ref{property W} and $L$-Lipschitz continuity of $\\nabla f$,\nthe third inequality uses Lemma~\\ref{lem2.5},\nthe last inequality applies  $\\alpha \\leq \\frac{2\\sqrt{5}-4}{5L}$ and \\eqref{new-bd-c}.\n\n We now establish a recursion upper bound on $\\|{\\m{x}}^t-\\m{M}{\\m{x}}^t\\|^2$.\n  By rewriting $\\m{x}^{t+1}=\\m{W}\\m{x}^t-\\alpha\\m{v}^t+\\alpha \\bm{\\beta}^{t}\\tilde{\\m{d}}^{t-1}$\n  and applying Young's inequality with constant $\\tau >0$, we have\n\t\\begin{align}\\label{term3}\n\t\t\\|\\m{x}^{t+1}-\\m{M}\\m{x}^{t+1}\\|^2\n\t\t=&\\|\\m{W}\\m{x}^t-\\m{M}\\m{x}^t-\\alpha\\m{v}^t+\\alpha\\m{M}\\m{v}^t+\\alpha \\bm{\\beta}^{t}\\tilde{\\m{d}}^{t-1}-\\alpha\\m{M} \\bm{\\beta}^{t}\\tilde{\\m{d}}^{t-1}\\|^2 \\notag \\\\\n\t\t\\leq &(1+\\tau) \\|\\m{W}\\m{x}^t-\\m{M}\\m{x}^t\\|^2\\notag+2(1+1/\\tau)\\alpha^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2\\notag\\\\\n\t\t&+2(1+1/\\tau)\\alpha^2\\|(\\m{I}-\\m{M})\\bm{\\beta}^{t}\\tilde{\\m{d}}^{t-1}\\|^2\\notag\\\\\n\t\t\\leq&(1+\\tau){\\sigma}^2 \\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2+2(1+1/\\tau)\\alpha^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2 \\notag \\\\\n\t\t&+2(1+1/\\tau)\\alpha^2\\|\\bm{\\beta}^{t}\\tilde{\\m{d}}^{t-1}\\|^2,\\notag\\\\\n\t\t\\leq&(1+\\tau){\\sigma}^2 \\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2+2(1+1/\\tau)\\alpha^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2 \\notag \\\\\n\t\t&+2(1+1/\\tau)(9/4-\\sqrt{5})\\alpha^2\\|\\tilde{\\m{v}}^{t}\\|^2, \n\t\\end{align}\t\nwhere the second inequality uses Lemma \\ref{property W} and  $\\rho(\\m{I}-\\m{M}) \\leq 1$,\nand the last inequality uses\n$\\alpha \\leq \\frac{2\\sqrt{5}-4}{5L}$, Lemma~\\ref{lem2.5}, \\eqref{lem2.5_eq1} and \\eqref{new-bd-c}.\nAdding up \\eqref{term1}, \\eqref{term2}, and \\eqref{term3} yields\n\t\\begin{align*}\n\t\t&P(\\m{x}^{t+1},\\m{v}^{t+1})\\\\\n\t\t\\leq &\tP(\\m{x}^{t},\\m{v}^{t})-\\left(1-(1+\\tau){\\sigma}^2-\\frac{16L^2\\alpha}{n}\\right)\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2\n\t\t\\\\\n\t\t&-\\left(1-(1+\\eta)\\sigma^2-2\\left(1+\\frac{1}{\\tau}\\right)\\alpha^2-\\frac{16\\alpha}{n}\\right)\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2\\notag\\\\\n\t\t&-\\left(\\frac{\\alpha(5-2\\sqrt{5})}{4 n}-\\frac{5L\\alpha^2}{8n}-\\left(1+\\frac{1}{\\eta}\\right)\\frac{5}{4}\\sigma^2L^2\\alpha^2-2\\left(1+\\frac{1}{\\tau}\\right)\\left(\\frac{9}{4}-\\sqrt{5}\\right)\\alpha^2\\right)\\|\\tilde{\\m{v}}^{t}\\|^2.\n\t\\end{align*}\nThen, setting $\\eta=\\tau=\\frac{1-{\\sigma}^2}{2{\\sigma}^2}$, we have\n\t\\begin{align}\\label{P_descent}\n\t\t&P(\\m{x}^{t+1},\\m{v}^{t+1})\\\\\n\t\t\\leq \t&P(\\m{x}^{t},\\m{v}^{t})-\\left(\\frac{1-{\\sigma}^2}{2}-\\frac{16L^2\\alpha}{n}\\right)\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2\\notag\\\\\n\t\t&-\\left(\\frac{\\alpha(5-2\\sqrt{5})}{4 n}-\\frac{5L\\alpha^2}{8n}-\\frac{5(1+\\sigma^2)\\sigma^2L^2\\alpha^2}{4(1-\\sigma^2)}-\\frac{2(1+{\\sigma}^2)\\alpha^2}{1-{\\sigma}^2}\\left(\\frac{9}{4}-\\sqrt{5}\\right)\\right)\\|\\tilde{\\m{v}}^{t}\\|^2\\notag\\\\\n\t\t&-\\left(\\frac{1-\\sigma^2}{2}-\\frac{2(1+{\\sigma}^2)\\alpha^2}{1-{\\sigma}^2}-\\frac{16\\alpha}{n}\\right)\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2\n\t\t\\notag\\\\\n\t\t\\leq& P(,\\m{x}^{t},\\m{v}^{t})\n\t\t-\\left(\\frac{\\alpha(5-2\\sqrt{5})}{4 n}-\\frac{4n(5\\sigma^2L^2+18-8\\sqrt{5})+5L}{8n(1-{\\sigma}^2)}\\alpha^2\\right)\\|\\tilde{\\m{v}}^{t}\\|^2\\notag\\\\\n\t\t&-\\left(\\frac{1-\\sigma^2}{2}-\\frac{4\\alpha^2}{1-{\\sigma}^2}-\\frac{16\\alpha}{n}\\right)\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2\n\t\t-\\left(\\frac{1-{\\sigma}^2}{2}-\\frac{16L^2\\alpha}{n}\\right)\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2,\\notag\n\t\\end{align}\nwhere the last inequality sues $0<\\sigma<1$.\n\tThen, by the choice of $\\alpha$ in \\eqref{alpha}, we can derive from \\eqref{P_descent}\n\tand direct calculations\tthat there exist a constant $\\gamma >0$, depending only on \n\t$n$, $L$, $\\sigma$, such that\n\t\\begin{equation*}\n\t\tP(\\m{x}^{t+1},\\m{v}^{t+1}) \\leq P(\\m{x}^{t},\\m{v}^{t}) - \\gamma \\left( \n\t\t\\alpha \\|\\tilde{\\m{v}}^t\\|^2 +\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2 \\right). \n\t\\end{equation*}\n\tSumming the above relation over $t=1, \\ldots, T$ and dividing $\\gamma \\min\\{\\alpha,1\\}$ from \n\tboth sides gives \\eqref{cg-converge-rate}.\n\\end{proof}","units":[{"start":33534,"end":33861,"start_line":435,"end_line":439,"raw_tex":"By the above formula \\eqref{tracking} and Lemma~\\ref{property W}, it follows from induction that\n\\begin{equation}\\label{vequalg}\n\\m{M}{\\m{v}}^t=\\m{M}{\\m{g}}^t \\quad \\Longleftrightarrow \\quad \\bar{\\m{v}}^t=\\bar{\\m{g}}^t,\n\\end{equation} \nwhich shows the average of $\\m{v}_i^t$ over $i$ is the same as the local gradient average.\n","sha256_utf8":"4dc08bdb1a0847bb2697631a142db950d510f0fbdca3aa84ec03315f27481c28"},{"start":40487,"end":41083,"start_line":545,"end_line":552,"raw_tex":"\\begin{lemma}\\label{lem2.5}\n\tSuppose Assumption \\ref{as0} holds. Consider the NDCG algorithm with  $ \\alpha \\in(0,\\frac{1}{4L}]$. Then, for all $t \\geq 0$ and any $i$, we have\n\t\\begin{align}\n\t\t&\\left(2-\\xi^t\\right)\\left\\|\\tilde{\\m{v}}_i^t\\right\\|^2  \\leq -(\\tilde{\\m{v}}_i^t)\\tr\\tilde{\\m{d}}_i^t \\leq \\xi^t\\left\\|\\tilde{\\m{v}}_i^t\\right\\|^2, \\label{descent1}\\\\\n\t\t&\\left(2-\\xi^t\\right)\\left\\|\\tilde{\\m{v}}_i^t\\right\\| \\leq\\left\\|\\tilde{\\m{d}}_i^t\\right\\| \\leq \\xi^t\\left\\|\\tilde{\\m{v}}_i^t\\right\\|\\label{descent2},\n\t\\end{align}\n\twhere $\\xi^t$ is the sequence defined in \\eqref{def-xi}.\n\\end{lemma}","sha256_utf8":"b8e3ee3f425fd40df391974d15ce71694cdfd05cf9f89fc1ea5aea76722e05be"},{"start":41084,"end":42896,"start_line":553,"end_line":580,"raw_tex":"\\begin{proof}\n\tSince $\\tilde{\\m{d}}_i^0=-\\tilde{\\m{v}}_i^0$ and $\\xi^0=1$, \\eqref{descent1} and \\eqref{descent2} clearly hold for $t=0$. \nAssume that \\eqref{descent1} and \\eqref{descent2} hold for some $t$. Then by the Cauchy-Schwartz inequality,  $\\m{x}_i^{t+1}=\\m{x}_i^{t}+\\alpha \\tilde{\\m{d}}_i^{t}$ and \\eqref{3.1}, \nwe have\n\t\\begin{align}\\label{lem2.5_eq1}\n\t\t\\|\\tilde{\\m{d}}_i^{t+1}+\\tilde{\\m{v}}_i^{t+1}\\|=&\\|\\beta_i^{t+1} \\tilde{\\m{d}}_i^{t}\\|\n\t\t\\leq \\frac{\\|{\\m{g}}^{t+1}_i-{\\m{g}}^{t}_i\\| \\|\\tilde{\\m{v}}^{t+1}_i\\|}{\\|\\tilde{\\m{v}}^{t}_i\\|^2}\\|\\tilde{\\m{d}}_i^{t}\\| \\\\\n\t\t\\leq &\\frac{L\\alpha \\|\\tilde{\\m{d}}_i^{t}\\|^2}{\\|\\tilde{\\m{v}}^{t}_i\\|^2}\\|\\tilde{\\m{v}}^{t+1}_i\\|\n\t\t\\leq L\\alpha (\\xi^t)^2\\|\\tilde{\\m{v}}^{t+1}_i\\|.\\notag\n\t\\end{align}\n\tBy the triangular inequality and the above relation, we obtain\n\t\\begin{align*}\n\t\t\\|\\tilde{\\m{d}}_i^{t+1}\\|\\leq \\|\\tilde{\\m{v}}_i^{t+1}\\|+\\|\\tilde{\\m{d}}_i^{t+1}+\\tilde{\\m{v}}_i^{t+1}\\|\\leq \\left(1+L\\alpha (\\xi^t)^2\\right)\\|\\tilde{\\m{v}}^{t+1}_i\\|\n\t\\end{align*}\n\tand\n\t\\begin{align*}\n\t\t\\|\\tilde{\\m{d}}_i^{t+1}\\|\\geq \\|\\tilde{\\m{v}}_i^{t+1}\\|-\\|\\tilde{\\m{d}}_i^{t+1}+\\tilde{\\m{v}}_i^{t+1}\\|\n\t\t\\geq \\left(1-L\\alpha (\\xi^t)^2\\right)\\|\\tilde{\\m{v}}^{t+1}_i\\|.\n\t\\end{align*}\n\tThen, it follows from the recursion  \\eqref{def-xi} of $\\xi^t$ and the choice of $\\alpha$ that \\eqref{descent2} holds for $t+1$. \n       By \\eqref{lem2.5_eq1} and the Cauchy-Schwartz inequality, we have that\n\t\\begin{align*}\n\t\t\\vert (\\tilde{\\m{v}}_i^{t+1})\\tr\\tilde{\\m{d}}_i^{t+1}+\\|\\tilde{\\m{v}}_i^{t+1}\\|^2 \\vert \n\t\t\\leq \\|\\tilde{\\m{v}}_i^{t+1}\\| \\|\\tilde{\\m{d}}_i^{t+1}+\\tilde{\\m{v}}_i^{t+1}\\|\n\t\t\\leq  L\\alpha (\\xi^t)^2\\|\\tilde{\\m{v}}^{t+1}_i\\|^2.\n\t\\end{align*}\n\tSimilarly, by the recursion  \\eqref{def-xi} of $\\xi^t$ and the choice of $\\alpha$, \\eqref{descent1} holds for $t+1$.\n\\end{proof}","sha256_utf8":"3cb0185296e8e5e3c8096842a2adcd922bf558d0b962e061847439f16cb3969a"},{"start":43011,"end":43537,"start_line":584,"end_line":590,"raw_tex":"\\begin{lemma}\n\tSuppose Assumption~\\ref{as0} holds. If $ \\alpha \\in(0,\\frac{2\\sqrt{5}-4}{5L}]$, then for all $t \\geq 0$ we have\n\t\\begin{align}\n\t\tF(\\bar{\\m{x}}^{t+1})+\\frac{1}{2\\alpha n} \\|\\m{x}^{t+1}\\|^2_{\\m{I}-\\m{W}}\\leq& F(\\bar{\\m{x}}^t)+\\frac{1}{2\\alpha n} \\|\\m{x}^{t}\\|^2_{\\m{I}-\\m{W}}-\\left(\\frac{\\alpha(5-2\\sqrt{5})}{4 n}-\\frac{5L\\alpha^2}{8n}\\right)\\|\\tilde{\\m{v}}^t\\|^2 \\notag \\\\\n&+\\frac{16L^2\\alpha}{n}\\|\\m{x}^t-\\m{M}\\m{x}^t\\|^2+\\frac{16\\alpha}{n}\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2. \\label{term1}\n\t\\end{align} \n\\end{lemma}","sha256_utf8":"2d9406b95c62a3110d5c9daff0de4d3e3513d25a2a56eb8ad19cf60e349683f7"},{"start":43539,"end":49233,"start_line":592,"end_line":658,"raw_tex":"\\begin{proof}\n\tBy Assumption \\ref{as0},  $F$ is Lipschitz continuously differentiable with Lipschitz constant $L$, which gives\n\t\\begin{align*}\n\t\tF(\\bar{\\m{x}}^{t+1})\\leq F(\\bar{\\m{x}}^t)+\\Big\\langle\\nabla F(\\bar{\\m{x}}^t),\\bar{\\m{x}}^{t+1}-\\bar{\\m{x}}^{t}\\Big\\rangle+\\frac{L}{2}\\|\\bar{\\m{x}}^{t+1}-\\bar{\\m{x}}^{t}\\|^{2}.\n\t\\end{align*}\n\tSince $\\rho(\\m{I}-\\m{W})<2$, we  have\n\t\\begin{align*}\n\t\t\\frac{1}{2\\alpha n} \\|\\m{x}^{t+1}\\|^2_{\\m{I}-\\m{W}} \\leq \\frac{1}{2\\alpha n} \\|\\m{x}^{t}\\|^2_{\\m{I}-\\m{W}}+\\frac{1}{n} \\Big\\langle \\frac{1}{\\alpha} (\\m{I}-\\m{W})\\m{x}^t, \\m{x}^{t+1}-\\m{x}^t\\Big\\rangle + \\frac{1}{2\\alpha n}\\|\\m{x}^{t+1}-\\m{x}^t\\|^2.\n\t\\end{align*}\nIn addition, we have\n\\begin{align*}\n\t&\\frac{L}{2}\\|\\bar{\\m{x}}^{t+1}-\\bar{\\m{x}}^{t}\\|^{2}=\\frac{L}{2n}\\|\\m{1}\\otimes\\bar{\\m{x}}^{t+1}-\\m{1}\\otimes\\bar{\\m{x}}^{t}\\|^{2}=\\frac{L}{2n}\\|\\m{M}{\\m{x}}^{t+1}-\\m{M}{\\m{x}}^{t}\\|^{2}\\leq\\frac{L}{2n}\\|{\\m{x}}^{t+1}-{\\m{x}}^{t}\\|^{2}.\n\\end{align*}\n\tAdding the above three inequalities together and using Lemma~\\ref{lem2.5} yields \n\t\\begin{eqnarray}\\label{add}\n\t   & & F(\\bar{\\m{x}}^{t+1})+\\frac{1}{2\\alpha n} \\|\\m{x}^{t+1}\\|^2_{\\m{I}-\\m{W}} \\\\\n  &\\leq& F(\\bar{\\m{x}}^t)+\\frac{1}{2\\alpha n} \\|\\m{x}^{t}\\|^2_{\\m{I}-\\m{W}}+\\left(\\frac{L\\alpha^2c^2}{2n}+\\frac{\\alpha c^2}{2 n}\\right) \\|\\tilde{\\m{v}}^t\\|^2 + \\mbox{RemainTerm} \\notag \n\t\\end{eqnarray}\n\twhere \n\t\\begin{align}\n\t\t&\\mbox{RemainTerm} = \\Big\\langle\\nabla F(\\bar{\\m{x}}^t),\\bar{\\m{x}}^{t+1}-\\bar{\\m{x}}^{t}\\Big\\rangle + \\frac{1}{n} \\Big\\langle \\frac{1}{\\alpha} (\\m{I}-\\m{W})\\m{x}^t, \\m{x}^{t+1}-\\m{x}^t\\Big\\rangle \\notag\\\\\n\t\t=&\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\nabla F(\\bar{\\m{x}}^t),{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle+\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle \\frac{1}{\\alpha} (\\m{x}_i^{t}-\\sum_{j \\in \\mathcal{N}_i} \\tilde{W}_{i j} \\m{x}_j^{t}), \\m{x}_i^{t+1}-\\m{x}_i^t\\Big\\rangle \\notag\\\\\n\t\t=&\\underbrace{\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\tilde{\\m{v}}_i^t,{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle}_{\\text {term (A)}}\n\t\t+\\underbrace{\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\nabla F(\\bar{\\m{x}}^t)-\\overline{\\nabla} f({\\m{x}}^t),{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle}_{\\text {term (B)}}\\notag\\\\\n\t\t&+\\underbrace{\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\overline{\\nabla} f({\\m{x}}^t)-{\\m{v}}_i^t,{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle}_{\\text {term (C)}}. \\label{terms}\n\t\\end{align}\n\tFor term (A), by Lemma~\\ref{lem2.5} and $\\m{x}_i^{t+1}=\\m{x}_i^{t}+\\alpha \\tilde{\\m{d}}_i^{t}$, we have\n\t\\begin{align}\\label{termA}\n\t\t\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\tilde{\\m{v}}_i^t,{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle \\leq -\\frac{\\alpha(2-c)}{ n} \\|\\tilde{\\m{v}}^t\\|^2.\n\t\\end{align}\n\tFor term (B), by Young's inequality with constant $b >0$ and  $\\m{1}\\otimes\\bar{\\m{x}}^t=\\m{M}{\\m{x}}^t$, we have \n\t\\begin{eqnarray}\\label{termB}\n\t  &&\t\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\nabla F(\\bar{\\m{x}}^t)-\\overline{\\nabla} f({\\m{x}}^t),{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle \\\\\n  &\\leq& \\frac{b}{2}\\|\\nabla F(\\bar{\\m{x}}^t)-\\overline{\\nabla} f({\\m{x}}^t)\\|^2+\\frac{1}{2nb}\\|{\\m{x}}^{t+1}-{\\m{x}}^{t}\\|^2 \\notag\\\\\n\t\t&\\leq & \\frac{L^2b}{2n}\\|\\m{1}\\otimes\\bar{\\m{x}}^t-\\m{x}^t \\|^2+\\frac{2\\alpha^2}{nb}\\|\\tilde{\\m{v}}^t\\|^2\n\t   =\\frac{L^2b}{2n}\\|\\m{M}\\m{x}^t-\\m{x}^t \\|^2+\\frac{2\\alpha^2}{nb}\\|\\tilde{\\m{v}}^t\\|^2,\\notag\n\t\\end{eqnarray}\nwhere the last inequality uses Lemma~\\ref{important}, Lemma~\\ref{lem2.5},  $L$-Lipschitz continuity of $\\nabla F$.\n\tFor term (C), by Young's inequality with constant $d>0$, we have\n\t\\begin{align}\\label{termC}\n\t\t\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\overline{\\nabla} f({\\m{x}}^t)-{\\m{v}}_i^t,{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle \\leq &\\frac{d}{2n}\\|\\m{1}\\otimes\\overline{\\nabla} f({\\m{x}}^t)-{\\m{v}}^t\\|^2+\\frac{1}{2nd}\\|{\\m{x}}^{t+1}-{\\m{x}}^{t}\\|^2 \\notag \\\\\n\t\t\\leq & \\frac{d}{2n}\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+\\frac{2\\alpha^2}{nd}\\|\\tilde{\\m{v}}^t\\|^2,\n\t\\end{align}\nwhere the last inequality is due to Lemma~\\ref{lem2.5} and the relation $\\m{1}\\otimes\\overline{\\nabla} f({\\m{x}}^t)=\\m{M}{\\m{v}}^t$ by \\eqref{vequalg}.\n\tSubstituting \\eqref{termA}, \\eqref{termB}, and \\eqref{termC} into \\eqref{terms}, we obtain\n\t\\begin{align*}\n\t\t&\\Big\\langle\\nabla F(\\bar{\\m{x}}^t),\\bar{\\m{x}}^{t+1}-\\bar{\\m{x}}^{t}\\Big\\rangle+\\frac{1}{n} \\Big\\langle \\frac{1}{\\alpha} (\\m{I}-\\m{W})\\m{x}^t, \\m{x}^{t+1}-\\m{x}^t\\Big\\rangle\\\\\n\t\t\\leq &-\\left(\\frac{\\alpha(2-c)}{ n}-\\frac{2\\alpha^2}{n}\\left(\\frac{1}{b}+\\frac{1}{d}\\right)\\right)\\|\\tilde{\\m{v}}^t\\|^2\n\t\t+\\frac{L^2b}{2n}\\|\\m{x}^t-\\m{M}\\m{x}^t\\|^2+\\frac{d}{2n}\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2.\n\t\\end{align*}\n\tBy substituting the above inequality into \\eqref{add}, it holds that\n\t\\begin{eqnarray*}\n\t&& F(\\bar{\\m{x}}^{t+1})+\\frac{1}{2\\alpha n} \\|\\m{x}^{t+1}\\|^2_{\\m{I}-\\m{W}} \\\\ \n      &\\leq& F(\\bar{\\m{x}}^t)+\\frac{1}{2\\alpha n} \\|\\m{x}^{t}\\|^2_{\\m{I}-\\m{W}}\n\t-\\left(\\frac{\\alpha(4-2c-c^2)}{ 2n}-\\frac{2\\alpha^2}{n}\\left(\\frac{1}{b}+\\frac{1}{d}\\right)-\\frac{L\\alpha^2c^2}{2n}\\right)\\|\\tilde{\\m{v}}^t\\|^2\\\\\n\t&&+\\frac{L^2b}{2n}\\|\\m{x}^t-\\m{M}\\m{x}^t\\|^2+\\frac{d}{2n}\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2.\n\\end{eqnarray*} \nSince $0<\\alpha \\leq \\frac{2\\sqrt{5}-4}{5L}$, we have from the definition of constant $c$ in \\eqref{const-c} that \n\\begin{equation}\\label{new-bd-c}\n1<c\\leq \\sqrt{5}/2, \n\\end{equation}\n which implies $\\frac{11}{4}-\\sqrt{5}\\leq 4-2c-c^2 \\leq 1$. Then, setting $b=d=32\\alpha$ yields \\eqref{term1}.\n%\t\\begin{align*}\n%\t\tF(\\bar{\\m{x}}^{t+1})+\\frac{1}{2\\alpha n} \\|\\m{x}^{t+1}\\|^2_{\\m{I}-\\m{W}}\\leq& F(\\bar{\\m{x}}^t)+\\frac{1}{2\\alpha n} \\|\\m{x}^{t}\\|^2_{\\m{I}-\\m{W}}-\\left(\\frac{\\alpha(5-2\\sqrt{5})}{4 n}-\\frac{5L\\alpha^2}{8n}\\right)\\| \\tilde{\\m{v}}^t\\|^2\\\\\n%\t\t&+\\frac{16L^2\\alpha}{n}\\|\\m{x}^t-\\m{M}\\m{x}^t\\|^2+\\frac{16\\alpha}{n}\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2.\n%\t\\end{align*} \n\\end{proof}","sha256_utf8":"4fde0c3738d1470201fadc324f1a6a50b05e3356fbbe453342a63d51b919f33b"},{"start":49960,"end":54537,"start_line":674,"end_line":735,"raw_tex":"\\begin{proof}\n\tFirst, by using Young's inequality with constant $\\eta >0$, we \n\t establish a recursion upper bound on $\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2$.\n\t\\begin{align}\\label{term2}\n\t\t\\|{\\m{v}}^{t+1}-\\m{M}{\\m{v}}^{t+1}\\|^2\n\t\t=&\\| \\m{W}{\\m{v}}^{t}+\\m{W}\\m{g}^{t+1}-\\m{W}\\m{g}^{t} -\\m{M}\\m{v}^t-\\m{M}\\m{g}^{t+1}+\\m{M}\\m{g}^t\\|^2 \\notag\\\\\n\t\t\\leq & (1+\\eta)\\|\\m{W}{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+(1+1/\\eta)\\|(\\m{W}-\\m{M})(\\m{g}^{t+1}-\\m{g}^t)\\|^2\\notag\\\\\n\t\t\\leq & (1+\\eta)\\sigma^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+(1+1/\\eta)\\sigma^2L^2\\|\\m{x}^{t+1}-\\m{x}^t\\|^2,\\notag\\\\\n\t\t\\leq & (1+\\eta)\\sigma^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+(1+1/\\eta)\\sigma^2L^2 c^2 \\alpha^2\\|\\tilde{\\m{v}}^t\\|^2,\\notag\\\\\n\t\t\\leq & (1+\\eta)\\sigma^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+(1+1/\\eta)(5/4)\\sigma^2L^2\\alpha^2\\|\\tilde{\\m{v}}^t\\|^2,\n\t\\end{align}\nwhere the second inequality uses Lemma~\\ref{property W} and $L$-Lipschitz continuity of $\\nabla f$,\nthe third inequality uses Lemma~\\ref{lem2.5},\nthe last inequality applies  $\\alpha \\leq \\frac{2\\sqrt{5}-4}{5L}$ and \\eqref{new-bd-c}.\n\n We now establish a recursion upper bound on $\\|{\\m{x}}^t-\\m{M}{\\m{x}}^t\\|^2$.\n  By rewriting $\\m{x}^{t+1}=\\m{W}\\m{x}^t-\\alpha\\m{v}^t+\\alpha \\bm{\\beta}^{t}\\tilde{\\m{d}}^{t-1}$\n  and applying Young's inequality with constant $\\tau >0$, we have\n\t\\begin{align}\\label{term3}\n\t\t\\|\\m{x}^{t+1}-\\m{M}\\m{x}^{t+1}\\|^2\n\t\t=&\\|\\m{W}\\m{x}^t-\\m{M}\\m{x}^t-\\alpha\\m{v}^t+\\alpha\\m{M}\\m{v}^t+\\alpha \\bm{\\beta}^{t}\\tilde{\\m{d}}^{t-1}-\\alpha\\m{M} \\bm{\\beta}^{t}\\tilde{\\m{d}}^{t-1}\\|^2 \\notag \\\\\n\t\t\\leq &(1+\\tau) \\|\\m{W}\\m{x}^t-\\m{M}\\m{x}^t\\|^2\\notag+2(1+1/\\tau)\\alpha^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2\\notag\\\\\n\t\t&+2(1+1/\\tau)\\alpha^2\\|(\\m{I}-\\m{M})\\bm{\\beta}^{t}\\tilde{\\m{d}}^{t-1}\\|^2\\notag\\\\\n\t\t\\leq&(1+\\tau){\\sigma}^2 \\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2+2(1+1/\\tau)\\alpha^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2 \\notag \\\\\n\t\t&+2(1+1/\\tau)\\alpha^2\\|\\bm{\\beta}^{t}\\tilde{\\m{d}}^{t-1}\\|^2,\\notag\\\\\n\t\t\\leq&(1+\\tau){\\sigma}^2 \\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2+2(1+1/\\tau)\\alpha^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2 \\notag \\\\\n\t\t&+2(1+1/\\tau)(9/4-\\sqrt{5})\\alpha^2\\|\\tilde{\\m{v}}^{t}\\|^2, \n\t\\end{align}\t\nwhere the second inequality uses Lemma \\ref{property W} and  $\\rho(\\m{I}-\\m{M}) \\leq 1$,\nand the last inequality uses\n$\\alpha \\leq \\frac{2\\sqrt{5}-4}{5L}$, Lemma~\\ref{lem2.5}, \\eqref{lem2.5_eq1} and \\eqref{new-bd-c}.\nAdding up \\eqref{term1}, \\eqref{term2}, and \\eqref{term3} yields\n\t\\begin{align*}\n\t\t&P(\\m{x}^{t+1},\\m{v}^{t+1})\\\\\n\t\t\\leq &\tP(\\m{x}^{t},\\m{v}^{t})-\\left(1-(1+\\tau){\\sigma}^2-\\frac{16L^2\\alpha}{n}\\right)\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2\n\t\t\\\\\n\t\t&-\\left(1-(1+\\eta)\\sigma^2-2\\left(1+\\frac{1}{\\tau}\\right)\\alpha^2-\\frac{16\\alpha}{n}\\right)\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2\\notag\\\\\n\t\t&-\\left(\\frac{\\alpha(5-2\\sqrt{5})}{4 n}-\\frac{5L\\alpha^2}{8n}-\\left(1+\\frac{1}{\\eta}\\right)\\frac{5}{4}\\sigma^2L^2\\alpha^2-2\\left(1+\\frac{1}{\\tau}\\right)\\left(\\frac{9}{4}-\\sqrt{5}\\right)\\alpha^2\\right)\\|\\tilde{\\m{v}}^{t}\\|^2.\n\t\\end{align*}\nThen, setting $\\eta=\\tau=\\frac{1-{\\sigma}^2}{2{\\sigma}^2}$, we have\n\t\\begin{align}\\label{P_descent}\n\t\t&P(\\m{x}^{t+1},\\m{v}^{t+1})\\\\\n\t\t\\leq \t&P(\\m{x}^{t},\\m{v}^{t})-\\left(\\frac{1-{\\sigma}^2}{2}-\\frac{16L^2\\alpha}{n}\\right)\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2\\notag\\\\\n\t\t&-\\left(\\frac{\\alpha(5-2\\sqrt{5})}{4 n}-\\frac{5L\\alpha^2}{8n}-\\frac{5(1+\\sigma^2)\\sigma^2L^2\\alpha^2}{4(1-\\sigma^2)}-\\frac{2(1+{\\sigma}^2)\\alpha^2}{1-{\\sigma}^2}\\left(\\frac{9}{4}-\\sqrt{5}\\right)\\right)\\|\\tilde{\\m{v}}^{t}\\|^2\\notag\\\\\n\t\t&-\\left(\\frac{1-\\sigma^2}{2}-\\frac{2(1+{\\sigma}^2)\\alpha^2}{1-{\\sigma}^2}-\\frac{16\\alpha}{n}\\right)\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2\n\t\t\\notag\\\\\n\t\t\\leq& P(,\\m{x}^{t},\\m{v}^{t})\n\t\t-\\left(\\frac{\\alpha(5-2\\sqrt{5})}{4 n}-\\frac{4n(5\\sigma^2L^2+18-8\\sqrt{5})+5L}{8n(1-{\\sigma}^2)}\\alpha^2\\right)\\|\\tilde{\\m{v}}^{t}\\|^2\\notag\\\\\n\t\t&-\\left(\\frac{1-\\sigma^2}{2}-\\frac{4\\alpha^2}{1-{\\sigma}^2}-\\frac{16\\alpha}{n}\\right)\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2\n\t\t-\\left(\\frac{1-{\\sigma}^2}{2}-\\frac{16L^2\\alpha}{n}\\right)\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2,\\notag\n\t\\end{align}\nwhere the last inequality sues $0<\\sigma<1$.\n\tThen, by the choice of $\\alpha$ in \\eqref{alpha}, we can derive from \\eqref{P_descent}\n\tand direct calculations\tthat there exist a constant $\\gamma >0$, depending only on \n\t$n$, $L$, $\\sigma$, such that\n\t\\begin{equation*}\n\t\tP(\\m{x}^{t+1},\\m{v}^{t+1}) \\leq P(\\m{x}^{t},\\m{v}^{t}) - \\gamma \\left( \n\t\t\\alpha \\|\\tilde{\\m{v}}^t\\|^2 +\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2 \\right). \n\t\\end{equation*}\n\tSumming the above relation over $t=1, \\ldots, T$ and dividing $\\gamma \\min\\{\\alpha,1\\}$ from \n\tboth sides gives \\eqref{cg-converge-rate}.\n\\end{proof}","sha256_utf8":"b3d74282c7aa0a392e863dbb126fd2d0da4c832bfdadb2365d974a0777c5bd97"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"网络优化问题","source_units":[{"start":5523,"end":5815,"start_line":134,"end_line":137,"raw_tex":"In this paper, we consider the following decentralized optimization problem over an undirected and connected network containing $n$ nodes\n\\begin{equation}\\label{obj_fun1}\n\t\\mathop {\\min } \\limits_{\\m{z} \\in {\\R^p}} \\; F(\\m{z}):= \\frac{1}{n}\\sum\\limits_{i = 1}^n {{f_i}(\\m{z})},\n\\end{equation}","sha256_utf8":"5259da680cb9d347af46c462e68394ac129f50087bacd303d49de305e0f435d9"},{"start":5816,"end":6111,"start_line":138,"end_line":139,"raw_tex":"where the local objective function $f_i:\\R^{p} \\rightarrow \\R$ is  continuously differentiable. \nConsider the underlying network $\\mathcal{G}=\\left(\\mathcal{V},\\mathcal{E}\\right)$, where $\\mathcal{V} = \\{1,\\ldots,n\\}$ is the set of nodes, and $\\mathcal{E}$ is the collection of unordered edges.\n","sha256_utf8":"8ec4eb935cd79aedd6d76d8d0315e42d00979c22a9283ff723ee47ea75c5d3e0"}]},{"id":"S002","semantic_type":"notation","kind":"notation","title":"堆叠变量与平均投影","source_units":[{"start":15799,"end":16094,"start_line":213,"end_line":214,"raw_tex":"We use uppercase boldface letters, e.g.~$\\m{W}$, for matrices and lowercase boldface letters,  e.g.~$\\m{w}$, for vectors.\nFor any vectors $\\m{v}_i \\in \\R^p$, $i=1,\\ldots,n$, we define  $\\bar{\\m{v}}=\\frac{1}{n} \\sum_{i=1}^n \\m{v}_i$ and $\\m{v} = [\\m{v}_1; \\m{v}_2; \\ldots, \\m{v}_n] \\in \\R^{np}$.\n","sha256_utf8":"97feb68bd92c1eb0a59cb5a1c582a3c12781eaf862e808999846851435e4b813"},{"start":16277,"end":16938,"start_line":223,"end_line":228,"raw_tex":"Given an undirected network  $\\mathcal{G}=\\left(\\mathcal{V},\\mathcal{E}\\right)$,\n let $\\m{x}_i$ denote the local copy of the global variable $\\m{z}$ at node $i$ and $\\mathcal{N}_i$ denote the set consisting of the neighbors of node $i$ \n(for convenience, we treat node $i$ itself as one of its neighbors). \nWe define $f(\\m{x}) = \\sum_{i = 1}^n {{f_i}({\\m{x}_i})}$ and use $\\m{g}^t$, $\\m{g}_i^t$ to stand for $\\nabla f(\\m{x}^t)$, $\\nabla f_i(\\m{x}^t_{i})$ respectively,\nwhere, for clarification, the gradient of $f(\\m{x})$ is defined as\n$ \\nabla f(\\m{x}) =\\left[ \\nabla f_1(\\m{x}_{1}); \\nabla f_2(\\m{x}_{2}); \\ldots, \\nabla f_n(\\m{x}_{n}) \\right] \\in \\R^{np}$. \n","sha256_utf8":"609cd7ea155d89a06aba191d61ce566b29aa7b1ae1d7c0eb30be3f06a062f897"},{"start":17150,"end":18399,"start_line":237,"end_line":246,"raw_tex":"In addition,  we define $\\overline{\\nabla} f(\\m{x}^t)=\\frac{1}{n} \\sum_{i=1}^n \\nabla f_i(\\m{x}_i^t) \n\\in \\R^p$.\nWe say that $\\m{x}$ is consensual or gets consensus if ${\\m{x}_1}={\\m{x}_2}=\\ldots={\\m{x}_n}$. \n$\\m{I}_p$ denotes the $p \\times p$ identity matrix and $\\m{I}$ denote $\\m{I}_{np}$ for simplicity. Kronecker Product is denoted as $\\otimes$.\nGiven a vector $\\m{v}$ and a matrix $\\m{N}$, $\\operatorname{span}(\\m{v})$ stands for the linear subspace spanned by $\\m{v}$;  $\\left\\|\\m{v}\\right\\|_{\\m{N}}^2$ denotes $\\m{v}\\tr \\m{N} \\m{v}$;\n$\\mbox{Null}(\\m{N})$ and $\\m{N}\\tr$  denote the null space and transpose of $\\m{N}$, respectively;\n${\\lambda _{\\min }(\\m{N})}$,  ${\\lambda _{\\max }(\\m{N})}$, and $\\rho(\\m{N})$ denote smallest eigenvalue, largest eigenvalue, and  the spectral radius of $\\m{N}$,  respectively;\nFor matrices $\\m{N}_1$ and $\\m{N}_2$ with same dimension, $\\m{N}_1 \\succeq \\m{N}_2$ means $\\m{N}_1 - \\m{N}_2$ is positive semidefinite, while $\\m{N}_1 \\ge \\m{N}_2$ means $\\m{N}_1 - \\m{N}_2$ is component-wise nonnegative.\nWe denote $\\operatorname{log}_{10}(\\cdot)$ by $\\operatorname{log}(\\cdot)$  and define $\\m{M}=\\frac{1}{n} \\m{1}_n\\m{1}_n\\tr\\otimes \\m{I}_p$ where $\\m{1}_n \\in \\R^n$ denotes the\nvector with all components ones. \n","sha256_utf8":"c7e047bba7211f8c293232b24aba3da89264732f33d341cfacab8d3a0d332053"}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","title":"下界与梯度光滑性","source_units":[{"start":19621,"end":20033,"start_line":256,"end_line":262,"raw_tex":"\\begin{assum}\\label{as0}\nThe local objective functions $\\{f_i \\}_{i=1}^n$ are bounded below and Lipschitz continuously differentiable, that is, for any $\\m{z}, \\tilde{\\m{z}} \\in \\R^p$ and $i=1, \\ldots,n$, \nwe have  $f_i(z)>-\\infty$ and \n\t\\begin{equation}\\label{3.1}\n\t\t\\left\\| {\\nabla {f_i}(\\m{z}) - \\nabla {f_i}(\\tilde{\\m{z}})} \\right\\| \\le L \\left\\| {\\m{z} - \\tilde{\\m{z}}} \\right\\|,\n\t\\end{equation}\n\\end{assum}","sha256_utf8":"5ac8ac2405944a96f093b7e53e6485acdca3ccf2cd5a1e72d5ae06851306d97c"},{"start":20034,"end":20072,"start_line":263,"end_line":263,"raw_tex":"where $L >0$ is a Lipschitz constant.\n","sha256_utf8":"3ab72b3458c989056443cf709fcf9438b39a4a6fd6b18af1cf4fe0f560442e87"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"混合矩阵定义","source_units":[{"start":20255,"end":20780,"start_line":267,"end_line":273,"raw_tex":"\\begin{defin}\\label{mix}\n\t(Mixing matrix $\\tilde{\\m{W}}$ for given network $\\mathcal{G} = \\left(\\mathcal{V},\\mathcal{E}\\right)$)\n\t\\begin{itemize}\n\t\t\\item [1.] $\\tilde{\\m{W}}$ is nonnegative, where each component $\\tilde{W}_{ij}$ characterizes the active link $(i,j)$, i.e., $\\tilde{{W}}_{i,j}>0$ if $j \\in \\mathcal{N}_i$;  $\\tilde{{W}}_{i,j}=0$, otherwise.\n\t\t\\item [2.] $\\tilde{\\m{W}}$ is symmetric and doubly stochastic, i.e., $\\tilde{\\m{W}}=\\tilde{\\m{W}}\\tr$ and $\\tilde{\\m{W}}\\m{1}_n=\\m{1}_n$.\\\\\n\t\\end{itemize}\n\\end{defin}","sha256_utf8":"8dcfd408850b275092b337f2fadd7e9d0e85ea32995ab5ad42e8fe83527f8a9a"}]},{"id":"S005","semantic_type":"notation","kind":"notation","title":"网络谱参数","source_units":[{"start":21008,"end":21237,"start_line":276,"end_line":277,"raw_tex":"Let $\\lambda_{i}(\\tilde{\\m{W}})$ denote the $i$-th largest eigenvalue of $\\tilde{\\m{W}}$ and $\\sigma$ be the second largest magnitude eigenvalue of $\\tilde{\\m{W}}$. \nThen, the following properties hold \\cite{xin2019distributed}.\n","sha256_utf8":"29dcac9a5674c7be6b7bc47827427f27395619d4c104dc286732aee483d42cb6"}]},{"id":"S006","semantic_type":"lemma","kind":"lemma","title":"混合矩阵收缩性质","source_units":[{"start":21237,"end":21843,"start_line":278,"end_line":287,"raw_tex":"\\begin{lemma}\\label{property W}\nFor $\\tilde{\\m{W}}$ defined in Definition~\\ref{mix} and  $\\m{W} :=\\tilde{\\m{W}}\\otimes\\m{I}_p$, we have\n\t\\begin{itemize}\n\t\t\\item [1.] $1=\\lambda_{1}(\\tilde{\\m{W}})>\\lambda_{2}(\\tilde{\\m{W}})\\geq\\ldots\\geq\\lambda_{n}(\\tilde{\\m{W}})>-1$;\n\t\t\\item [2.] $\t0<\\rho(\\m{W}-\\m{M})=\\sigma=\\max \\left\\{|\\lambda_{2}(\\tilde{\\m{W}})|, |\\lambda_{n}(\\tilde{\\m{W}})|\\right\\}<1$;\n\t\t\\item [3.] $\\m{M}=\\m{M}\\m{W}=\\m{W}\\m{M}$;\n\t\t\\item [4.]  $\\|\\m{W}\\m{x}-\\m{M}\\m{x}\\|=\\|(\\m{W}-\\m{M})(\\m{x}-\\m{M}\\m{x})\\| \\leq \\sigma \\|\\m{x}-\\m{M}\\m{x}\\|$\n\t\tfor any $\\m{x} \\in \\R^{np}$. \n\t\\end{itemize}\n\\end{lemma}","sha256_utf8":"0c7f2f9bf4ee901b242e3762d55d63480864a63949c61d6d39e055973ebd8bae"}]},{"id":"S007","semantic_type":"lemma","kind":"lemma","title":"Young不等式","source_units":[{"start":21921,"end":22244,"start_line":290,"end_line":296,"raw_tex":"\\begin{lemma}\\label{young}\n\t(Young's inequality) For any two vectors $\\m{v}_1,\\m{v}_2\\in \\R^p$, $\\eta>0$, we have\n\\[\n2\\m{v}_1\\tr\\m{v}_2 \\leq \\eta \\|\\m{v}_1\\|^2+ \\frac{1}{\\eta}\\|\\m{v}_2\\|^2 \\quad \\mbox{and} \\quad\n\t\t\\|\\m{v}_1+\\m{v}_2\\|^2 \\leq (1+\\eta) \\|\\m{v}_1\\|^2+ \\left(1+\\frac{1}{\\eta}\\right)\\|\\m{v}_2\\|^2.\n\\]\n\\end{lemma}","sha256_utf8":"78966ba658b836386c3232093d9f7f16cc26865192baeee05cd26db5218f042f"}]},{"id":"S008","semantic_type":"lemma","kind":"lemma","title":"Jensen不等式","source_units":[{"start":22245,"end":22483,"start_line":297,"end_line":300,"raw_tex":"\\begin{lemma}\\label{important}\n\t(Jensen's inequality) For any set of vectors $\\{\\m{v}_i\\}_{i=1}^n \\subset \\R^p$, we have\n\t$$\\left\\|\\frac{1}{n}\\sum_{i=1}^n\\m{v}_i\\right\\|^2\\leq\\frac{1}{n}\\sum_{i=1}^n\\left\\|\\m{v}_i\\right\\|^2.$$ \n\\end{lemma}","sha256_utf8":"a17fd65e54df2583d009b6431e4484a829c53b52f2c1836fe48c10182065ea1e"}]},{"id":"S009","semantic_type":"lemma","kind":"lemma","title":"标量递推统一界","source_units":[{"start":40035,"end":40485,"start_line":533,"end_line":543,"raw_tex":"\\begin{lemma}\\label{lem2.5a}\n\t Assume $\\alpha$ is some constant in $(0,\\frac{1}{4L}]$. Define the sequence $\\left\\{\\xi^t\\right\\}$ as\n \\begin{equation}\\label{def-xi}\n\t\\xi^0=1 ; \\quad \\xi^{t+1}=1+L \\alpha (\\xi^t)^2, \\quad t \\geq 0 .\n \\end{equation}\n\tThen, we have $ 1 \\leq \\xi^t<c $ for all $ t \\geq 0$,\n\twhere $ c \\in (0,2] $ is a constant given as\n\\begin{equation}\\label{const-c}\n\tc=2\\left(1+\\sqrt{1-4L \\alpha}\\right)^{-1}.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"100734a76fdb509dd23d461b334469bda2ea6040fa344c3ee0c7d84a87bbbe88"}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","title":"NDCG完整算法","source_units":[{"start":34742,"end":35815,"start_line":453,"end_line":471,"raw_tex":"\\begin{algorithm}[htb]\n\t\\caption{NDCG with respect to node $i$}\n\t\\label{alg:Framwork+}\n\t\\begin{algorithmic}[1]\n\t\t\\Require\n\t\tInitial point $\\m{x}_i^0$,  Maximum iteration T, Stepsize $\\alpha>0$,\n\t\t Mixing matrix $\\tilde{\\m{W}}$.\n\t\t\\State Set $t=0$,  $\\m{v}_i^0=\\m{g}_i^0$ and $\\tilde{\\m{d}}_i^{0}=-\\tilde{\\m{v}}_i^{0}=-{\\m{g}}_i^{0}-\\frac{1}{\\alpha}(\\m{x}_i^{0}-\\sum_{j \\in \\mathcal{N}_i} \\tilde{W}_{i j} \\m{x}_j^{0})$.\n\t\t\\State If $t \\geq T$, stop.\n\t\t\\State $\\m{x}_i^{t+1}=\\m{x}_i^{t}+\\alpha \\tilde{\\m{d}}_i^{t}$.\n\t\t\\State \t$\t\\m{v}_i^{t+1}=\\sum_{j \\in \\mathcal{N}_i}\\tilde{W}_{ij}(\\m{v}_j^{t}+\\m{g}_j^{t+1}-\\m{g}_j^{t})$.\n\t\t\\State $\\tilde{\\m{v}}_i^{t+1} = \\m{v}_i^{t+1}+\\frac{1}{\\alpha}(\\m{x}_i^{t+1}-\\sum_{j \\in \\mathcal{N}_i} \\tilde{W}_{i j} \\m{x}_j^{t+1})$.\n\t\t\\State $\\beta_i^{t+1}=\\frac{(\\tilde{\\m{v}}^{t+1}_i)\\tr ({\\m{g}}^{t+1}_i-{\\m{g}}^t_i)}{\\Vert \\tilde{\\m{v}}^t_i \\Vert ^2}$.\n\t\t\\State $\\tilde{\\m{d}}_i^{t+1}=-\\tilde{\\m{v}}_i^{t+1}+\\beta_i^{t+1} \\tilde{\\m{d}}_i^{t}$. \n\t\t\\State Set $t=t+1$ and go to Step 2.\n\t\t\\Ensure\n\t\t$\\m{x}^T$.\n\t\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"886376e45c8f9083d14b890e9a2d34c16ffa1b5fc6bafaeee829c15bbbe7df83"}]},{"id":"S011","semantic_type":"algorithm","kind":"algorithm","title":"梯度跟踪初始化与更新","source_units":[{"start":33131,"end":33533,"start_line":430,"end_line":434,"raw_tex":"Specifically, we introduce a variable $\\m{v}_i$ on node $i$ to track the local gradient average by a dynamical average consensus technique \\cite{sayed2014diffusion}.\nInitializing  $\\m{v}_i^0=\\m{g}_i^0$, we use the ATC structure to update  $\\m{v}_i^{t+1}$ as\n\\begin{equation}\\label{tracking}\n\t\t\\m{v}_i^{t+1}=\\sum_{j \\in \\mathcal{N}_i}\\tilde{W}_{ij}(\\m{v}_j^{t}+\\m{g}_j^{t+1}-\\m{g}_j^{t}).\n\\end{equation}","sha256_utf8":"73873f72d9fc7afa37b6403931487c02112ea0cb2c8f5823ac79a223e6674ff3"}]},{"id":"S012","semantic_type":"notation","kind":"notation","title":"共轭参数对角矩阵","source_units":[{"start":25114,"end":25344,"start_line":337,"end_line":345,"raw_tex":"where\n\\begin{equation*}\n\t{\\bm{\\beta}^{t+1}} = \\left[ {\\begin{array}{*{20}{c}}\n\t\t\t{{ \\beta_1^{t+1}}\\m{I}_p\n\t\t\t}&{}&{}\\\\\n\t\t\t{}&\\ddots&{}\\\\\n\t\t\t{}&{}&{\\beta_n^{t+1}}\\m{I}_p\n\t\\end{array}} \\right] \\in \\R^{np \\times np}.\n\\end{equation*}\n","sha256_utf8":"eb0f95d2229c7e5324fd43e23c4a7709a128d0e68272c06291ab821ed6d882c0"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"势函数定义","source_units":[{"start":39011,"end":39215,"start_line":521,"end_line":523,"raw_tex":"\\begin{equation}\\label{Potential-P}\n\tP(\\m{x}^{t},\\m{v}^{t})=F(\\bar{\\m{x}}^t)+\\frac{1}{2\\alpha n} \\|\\m{x}^{t}\\|^2_{\\m{I}-\\m{W}}+\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2+\\|\\m{v}^{t}-\\m{M}\\m{v}^{t}\\|^2.\n\\end{equation}","sha256_utf8":"dbb590d7a4448451400e1b49ade5b868cdc9021fca140687273b820c69d2f5c8"}]},{"id":"S014","semantic_type":"derived_fact","kind":"derived_fact","title":"跟踪变量保持梯度平均","source_units":[{"start":33631,"end":33768,"start_line":436,"end_line":438,"raw_tex":"\\begin{equation}\\label{vequalg}\n\\m{M}{\\m{v}}^t=\\m{M}{\\m{g}}^t \\quad \\Longleftrightarrow \\quad \\bar{\\m{v}}^t=\\bar{\\m{g}}^t,\n\\end{equation}","sha256_utf8":"0e0dfe1cfef28841dd5ee850bbd22d318a83d95908cd71d3e3a6768e831054b9"},{"start":33770,"end":33861,"start_line":439,"end_line":439,"raw_tex":"which shows the average of $\\m{v}_i^t$ over $i$ is the same as the local gradient average.\n","sha256_utf8":"5cee08c74779a32a02f984a1b3f7cf7b7ffc1c37177af1b4a8829b4c9b4d58f3"}]},{"id":"S015","semantic_type":"proof","kind":"proof","title":"平均保持性质完整论证","source_units":[{"start":33534,"end":33861,"start_line":435,"end_line":439,"raw_tex":"By the above formula \\eqref{tracking} and Lemma~\\ref{property W}, it follows from induction that\n\\begin{equation}\\label{vequalg}\n\\m{M}{\\m{v}}^t=\\m{M}{\\m{g}}^t \\quad \\Longleftrightarrow \\quad \\bar{\\m{v}}^t=\\bar{\\m{g}}^t,\n\\end{equation} \nwhich shows the average of $\\m{v}_i^t$ over $i$ is the same as the local gradient average.\n","sha256_utf8":"4dc08bdb1a0847bb2697631a142db950d510f0fbdca3aa84ec03315f27481c28"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"由跟踪递推归纳","source_units":[{"start":33534,"end":33631,"start_line":435,"end_line":435,"raw_tex":"By the above formula \\eqref{tracking} and Lemma~\\ref{property W}, it follows from induction that\n","sha256_utf8":"f552008a1247d03ab8e9a136315cc296ef0d4cb041458eaeebcc45e4455a2257"}]},{"id":"S017","semantic_type":"lemma","kind":"lemma","title":"共轭方向下降与范数界","source_units":[{"start":40487,"end":41083,"start_line":545,"end_line":552,"raw_tex":"\\begin{lemma}\\label{lem2.5}\n\tSuppose Assumption \\ref{as0} holds. Consider the NDCG algorithm with  $ \\alpha \\in(0,\\frac{1}{4L}]$. Then, for all $t \\geq 0$ and any $i$, we have\n\t\\begin{align}\n\t\t&\\left(2-\\xi^t\\right)\\left\\|\\tilde{\\m{v}}_i^t\\right\\|^2  \\leq -(\\tilde{\\m{v}}_i^t)\\tr\\tilde{\\m{d}}_i^t \\leq \\xi^t\\left\\|\\tilde{\\m{v}}_i^t\\right\\|^2, \\label{descent1}\\\\\n\t\t&\\left(2-\\xi^t\\right)\\left\\|\\tilde{\\m{v}}_i^t\\right\\| \\leq\\left\\|\\tilde{\\m{d}}_i^t\\right\\| \\leq \\xi^t\\left\\|\\tilde{\\m{v}}_i^t\\right\\|\\label{descent2},\n\t\\end{align}\n\twhere $\\xi^t$ is the sequence defined in \\eqref{def-xi}.\n\\end{lemma}","sha256_utf8":"b8e3ee3f425fd40df391974d15ce71694cdfd05cf9f89fc1ea5aea76722e05be"}]},{"id":"S018","semantic_type":"proof","kind":"proof","title":"共轭方向界完整证明","source_units":[{"start":41084,"end":42896,"start_line":553,"end_line":580,"raw_tex":"\\begin{proof}\n\tSince $\\tilde{\\m{d}}_i^0=-\\tilde{\\m{v}}_i^0$ and $\\xi^0=1$, \\eqref{descent1} and \\eqref{descent2} clearly hold for $t=0$. \nAssume that \\eqref{descent1} and \\eqref{descent2} hold for some $t$. Then by the Cauchy-Schwartz inequality,  $\\m{x}_i^{t+1}=\\m{x}_i^{t}+\\alpha \\tilde{\\m{d}}_i^{t}$ and \\eqref{3.1}, \nwe have\n\t\\begin{align}\\label{lem2.5_eq1}\n\t\t\\|\\tilde{\\m{d}}_i^{t+1}+\\tilde{\\m{v}}_i^{t+1}\\|=&\\|\\beta_i^{t+1} \\tilde{\\m{d}}_i^{t}\\|\n\t\t\\leq \\frac{\\|{\\m{g}}^{t+1}_i-{\\m{g}}^{t}_i\\| \\|\\tilde{\\m{v}}^{t+1}_i\\|}{\\|\\tilde{\\m{v}}^{t}_i\\|^2}\\|\\tilde{\\m{d}}_i^{t}\\| \\\\\n\t\t\\leq &\\frac{L\\alpha \\|\\tilde{\\m{d}}_i^{t}\\|^2}{\\|\\tilde{\\m{v}}^{t}_i\\|^2}\\|\\tilde{\\m{v}}^{t+1}_i\\|\n\t\t\\leq L\\alpha (\\xi^t)^2\\|\\tilde{\\m{v}}^{t+1}_i\\|.\\notag\n\t\\end{align}\n\tBy the triangular inequality and the above relation, we obtain\n\t\\begin{align*}\n\t\t\\|\\tilde{\\m{d}}_i^{t+1}\\|\\leq \\|\\tilde{\\m{v}}_i^{t+1}\\|+\\|\\tilde{\\m{d}}_i^{t+1}+\\tilde{\\m{v}}_i^{t+1}\\|\\leq \\left(1+L\\alpha (\\xi^t)^2\\right)\\|\\tilde{\\m{v}}^{t+1}_i\\|\n\t\\end{align*}\n\tand\n\t\\begin{align*}\n\t\t\\|\\tilde{\\m{d}}_i^{t+1}\\|\\geq \\|\\tilde{\\m{v}}_i^{t+1}\\|-\\|\\tilde{\\m{d}}_i^{t+1}+\\tilde{\\m{v}}_i^{t+1}\\|\n\t\t\\geq \\left(1-L\\alpha (\\xi^t)^2\\right)\\|\\tilde{\\m{v}}^{t+1}_i\\|.\n\t\\end{align*}\n\tThen, it follows from the recursion  \\eqref{def-xi} of $\\xi^t$ and the choice of $\\alpha$ that \\eqref{descent2} holds for $t+1$. \n       By \\eqref{lem2.5_eq1} and the Cauchy-Schwartz inequality, we have that\n\t\\begin{align*}\n\t\t\\vert (\\tilde{\\m{v}}_i^{t+1})\\tr\\tilde{\\m{d}}_i^{t+1}+\\|\\tilde{\\m{v}}_i^{t+1}\\|^2 \\vert \n\t\t\\leq \\|\\tilde{\\m{v}}_i^{t+1}\\| \\|\\tilde{\\m{d}}_i^{t+1}+\\tilde{\\m{v}}_i^{t+1}\\|\n\t\t\\leq  L\\alpha (\\xi^t)^2\\|\\tilde{\\m{v}}^{t+1}_i\\|^2.\n\t\\end{align*}\n\tSimilarly, by the recursion  \\eqref{def-xi} of $\\xi^t$ and the choice of $\\alpha$, \\eqref{descent1} holds for $t+1$.\n\\end{proof}","sha256_utf8":"3cb0185296e8e5e3c8096842a2adcd922bf558d0b962e061847439f16cb3969a"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"归纳初值与方向偏差界","source_units":[{"start":41098,"end":41833,"start_line":554,"end_line":562,"raw_tex":"\tSince $\\tilde{\\m{d}}_i^0=-\\tilde{\\m{v}}_i^0$ and $\\xi^0=1$, \\eqref{descent1} and \\eqref{descent2} clearly hold for $t=0$. \nAssume that \\eqref{descent1} and \\eqref{descent2} hold for some $t$. Then by the Cauchy-Schwartz inequality,  $\\m{x}_i^{t+1}=\\m{x}_i^{t}+\\alpha \\tilde{\\m{d}}_i^{t}$ and \\eqref{3.1}, \nwe have\n\t\\begin{align}\\label{lem2.5_eq1}\n\t\t\\|\\tilde{\\m{d}}_i^{t+1}+\\tilde{\\m{v}}_i^{t+1}\\|=&\\|\\beta_i^{t+1} \\tilde{\\m{d}}_i^{t}\\|\n\t\t\\leq \\frac{\\|{\\m{g}}^{t+1}_i-{\\m{g}}^{t}_i\\| \\|\\tilde{\\m{v}}^{t+1}_i\\|}{\\|\\tilde{\\m{v}}^{t}_i\\|^2}\\|\\tilde{\\m{d}}_i^{t}\\| \\\\\n\t\t\\leq &\\frac{L\\alpha \\|\\tilde{\\m{d}}_i^{t}\\|^2}{\\|\\tilde{\\m{v}}^{t}_i\\|^2}\\|\\tilde{\\m{v}}^{t+1}_i\\|\n\t\t\\leq L\\alpha (\\xi^t)^2\\|\\tilde{\\m{v}}^{t+1}_i\\|.\\notag\n\t\\end{align}\n","sha256_utf8":"d636eaba2c7cb4c8927e503299f409d183e7ea352baf5e25dcd6dc79ef9742f0"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"递推得到方向范数界","source_units":[{"start":41833,"end":42433,"start_line":563,"end_line":572,"raw_tex":"\tBy the triangular inequality and the above relation, we obtain\n\t\\begin{align*}\n\t\t\\|\\tilde{\\m{d}}_i^{t+1}\\|\\leq \\|\\tilde{\\m{v}}_i^{t+1}\\|+\\|\\tilde{\\m{d}}_i^{t+1}+\\tilde{\\m{v}}_i^{t+1}\\|\\leq \\left(1+L\\alpha (\\xi^t)^2\\right)\\|\\tilde{\\m{v}}^{t+1}_i\\|\n\t\\end{align*}\n\tand\n\t\\begin{align*}\n\t\t\\|\\tilde{\\m{d}}_i^{t+1}\\|\\geq \\|\\tilde{\\m{v}}_i^{t+1}\\|-\\|\\tilde{\\m{d}}_i^{t+1}+\\tilde{\\m{v}}_i^{t+1}\\|\n\t\t\\geq \\left(1-L\\alpha (\\xi^t)^2\\right)\\|\\tilde{\\m{v}}^{t+1}_i\\|.\n\t\\end{align*}\n\tThen, it follows from the recursion  \\eqref{def-xi} of $\\xi^t$ and the choice of $\\alpha$ that \\eqref{descent2} holds for $t+1$. \n","sha256_utf8":"38516614c4814318e2235d3c2a2959c16d3a0e26ee57c99663674e17190fb43c"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"递推得到内积下降界","source_units":[{"start":42433,"end":42885,"start_line":573,"end_line":579,"raw_tex":"       By \\eqref{lem2.5_eq1} and the Cauchy-Schwartz inequality, we have that\n\t\\begin{align*}\n\t\t\\vert (\\tilde{\\m{v}}_i^{t+1})\\tr\\tilde{\\m{d}}_i^{t+1}+\\|\\tilde{\\m{v}}_i^{t+1}\\|^2 \\vert \n\t\t\\leq \\|\\tilde{\\m{v}}_i^{t+1}\\| \\|\\tilde{\\m{d}}_i^{t+1}+\\tilde{\\m{v}}_i^{t+1}\\|\n\t\t\\leq  L\\alpha (\\xi^t)^2\\|\\tilde{\\m{v}}^{t+1}_i\\|^2.\n\t\\end{align*}\n\tSimilarly, by the recursion  \\eqref{def-xi} of $\\xi^t$ and the choice of $\\alpha$, \\eqref{descent1} holds for $t+1$.\n","sha256_utf8":"c84c7aa9c17cf90a5b756c150b3bcffc5227f6e8c0fa6f85c86dee9d994fa8d4"}]},{"id":"S022","semantic_type":"lemma","kind":"lemma","title":"目标与网络能量递推","source_units":[{"start":43011,"end":43537,"start_line":584,"end_line":590,"raw_tex":"\\begin{lemma}\n\tSuppose Assumption~\\ref{as0} holds. If $ \\alpha \\in(0,\\frac{2\\sqrt{5}-4}{5L}]$, then for all $t \\geq 0$ we have\n\t\\begin{align}\n\t\tF(\\bar{\\m{x}}^{t+1})+\\frac{1}{2\\alpha n} \\|\\m{x}^{t+1}\\|^2_{\\m{I}-\\m{W}}\\leq& F(\\bar{\\m{x}}^t)+\\frac{1}{2\\alpha n} \\|\\m{x}^{t}\\|^2_{\\m{I}-\\m{W}}-\\left(\\frac{\\alpha(5-2\\sqrt{5})}{4 n}-\\frac{5L\\alpha^2}{8n}\\right)\\|\\tilde{\\m{v}}^t\\|^2 \\notag \\\\\n&+\\frac{16L^2\\alpha}{n}\\|\\m{x}^t-\\m{M}\\m{x}^t\\|^2+\\frac{16\\alpha}{n}\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2. \\label{term1}\n\t\\end{align} \n\\end{lemma}","sha256_utf8":"2d9406b95c62a3110d5c9daff0de4d3e3513d25a2a56eb8ad19cf60e349683f7"}]},{"id":"S023","semantic_type":"proof","kind":"proof","title":"能量递推完整证明","source_units":[{"start":43539,"end":49233,"start_line":592,"end_line":658,"raw_tex":"\\begin{proof}\n\tBy Assumption \\ref{as0},  $F$ is Lipschitz continuously differentiable with Lipschitz constant $L$, which gives\n\t\\begin{align*}\n\t\tF(\\bar{\\m{x}}^{t+1})\\leq F(\\bar{\\m{x}}^t)+\\Big\\langle\\nabla F(\\bar{\\m{x}}^t),\\bar{\\m{x}}^{t+1}-\\bar{\\m{x}}^{t}\\Big\\rangle+\\frac{L}{2}\\|\\bar{\\m{x}}^{t+1}-\\bar{\\m{x}}^{t}\\|^{2}.\n\t\\end{align*}\n\tSince $\\rho(\\m{I}-\\m{W})<2$, we  have\n\t\\begin{align*}\n\t\t\\frac{1}{2\\alpha n} \\|\\m{x}^{t+1}\\|^2_{\\m{I}-\\m{W}} \\leq \\frac{1}{2\\alpha n} \\|\\m{x}^{t}\\|^2_{\\m{I}-\\m{W}}+\\frac{1}{n} \\Big\\langle \\frac{1}{\\alpha} (\\m{I}-\\m{W})\\m{x}^t, \\m{x}^{t+1}-\\m{x}^t\\Big\\rangle + \\frac{1}{2\\alpha n}\\|\\m{x}^{t+1}-\\m{x}^t\\|^2.\n\t\\end{align*}\nIn addition, we have\n\\begin{align*}\n\t&\\frac{L}{2}\\|\\bar{\\m{x}}^{t+1}-\\bar{\\m{x}}^{t}\\|^{2}=\\frac{L}{2n}\\|\\m{1}\\otimes\\bar{\\m{x}}^{t+1}-\\m{1}\\otimes\\bar{\\m{x}}^{t}\\|^{2}=\\frac{L}{2n}\\|\\m{M}{\\m{x}}^{t+1}-\\m{M}{\\m{x}}^{t}\\|^{2}\\leq\\frac{L}{2n}\\|{\\m{x}}^{t+1}-{\\m{x}}^{t}\\|^{2}.\n\\end{align*}\n\tAdding the above three inequalities together and using Lemma~\\ref{lem2.5} yields \n\t\\begin{eqnarray}\\label{add}\n\t   & & F(\\bar{\\m{x}}^{t+1})+\\frac{1}{2\\alpha n} \\|\\m{x}^{t+1}\\|^2_{\\m{I}-\\m{W}} \\\\\n  &\\leq& F(\\bar{\\m{x}}^t)+\\frac{1}{2\\alpha n} \\|\\m{x}^{t}\\|^2_{\\m{I}-\\m{W}}+\\left(\\frac{L\\alpha^2c^2}{2n}+\\frac{\\alpha c^2}{2 n}\\right) \\|\\tilde{\\m{v}}^t\\|^2 + \\mbox{RemainTerm} \\notag \n\t\\end{eqnarray}\n\twhere \n\t\\begin{align}\n\t\t&\\mbox{RemainTerm} = \\Big\\langle\\nabla F(\\bar{\\m{x}}^t),\\bar{\\m{x}}^{t+1}-\\bar{\\m{x}}^{t}\\Big\\rangle + \\frac{1}{n} \\Big\\langle \\frac{1}{\\alpha} (\\m{I}-\\m{W})\\m{x}^t, \\m{x}^{t+1}-\\m{x}^t\\Big\\rangle \\notag\\\\\n\t\t=&\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\nabla F(\\bar{\\m{x}}^t),{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle+\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle \\frac{1}{\\alpha} (\\m{x}_i^{t}-\\sum_{j \\in \\mathcal{N}_i} \\tilde{W}_{i j} \\m{x}_j^{t}), \\m{x}_i^{t+1}-\\m{x}_i^t\\Big\\rangle \\notag\\\\\n\t\t=&\\underbrace{\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\tilde{\\m{v}}_i^t,{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle}_{\\text {term (A)}}\n\t\t+\\underbrace{\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\nabla F(\\bar{\\m{x}}^t)-\\overline{\\nabla} f({\\m{x}}^t),{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle}_{\\text {term (B)}}\\notag\\\\\n\t\t&+\\underbrace{\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\overline{\\nabla} f({\\m{x}}^t)-{\\m{v}}_i^t,{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle}_{\\text {term (C)}}. \\label{terms}\n\t\\end{align}\n\tFor term (A), by Lemma~\\ref{lem2.5} and $\\m{x}_i^{t+1}=\\m{x}_i^{t}+\\alpha \\tilde{\\m{d}}_i^{t}$, we have\n\t\\begin{align}\\label{termA}\n\t\t\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\tilde{\\m{v}}_i^t,{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle \\leq -\\frac{\\alpha(2-c)}{ n} \\|\\tilde{\\m{v}}^t\\|^2.\n\t\\end{align}\n\tFor term (B), by Young's inequality with constant $b >0$ and  $\\m{1}\\otimes\\bar{\\m{x}}^t=\\m{M}{\\m{x}}^t$, we have \n\t\\begin{eqnarray}\\label{termB}\n\t  &&\t\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\nabla F(\\bar{\\m{x}}^t)-\\overline{\\nabla} f({\\m{x}}^t),{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle \\\\\n  &\\leq& \\frac{b}{2}\\|\\nabla F(\\bar{\\m{x}}^t)-\\overline{\\nabla} f({\\m{x}}^t)\\|^2+\\frac{1}{2nb}\\|{\\m{x}}^{t+1}-{\\m{x}}^{t}\\|^2 \\notag\\\\\n\t\t&\\leq & \\frac{L^2b}{2n}\\|\\m{1}\\otimes\\bar{\\m{x}}^t-\\m{x}^t \\|^2+\\frac{2\\alpha^2}{nb}\\|\\tilde{\\m{v}}^t\\|^2\n\t   =\\frac{L^2b}{2n}\\|\\m{M}\\m{x}^t-\\m{x}^t \\|^2+\\frac{2\\alpha^2}{nb}\\|\\tilde{\\m{v}}^t\\|^2,\\notag\n\t\\end{eqnarray}\nwhere the last inequality uses Lemma~\\ref{important}, Lemma~\\ref{lem2.5},  $L$-Lipschitz continuity of $\\nabla F$.\n\tFor term (C), by Young's inequality with constant $d>0$, we have\n\t\\begin{align}\\label{termC}\n\t\t\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\overline{\\nabla} f({\\m{x}}^t)-{\\m{v}}_i^t,{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle \\leq &\\frac{d}{2n}\\|\\m{1}\\otimes\\overline{\\nabla} f({\\m{x}}^t)-{\\m{v}}^t\\|^2+\\frac{1}{2nd}\\|{\\m{x}}^{t+1}-{\\m{x}}^{t}\\|^2 \\notag \\\\\n\t\t\\leq & \\frac{d}{2n}\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+\\frac{2\\alpha^2}{nd}\\|\\tilde{\\m{v}}^t\\|^2,\n\t\\end{align}\nwhere the last inequality is due to Lemma~\\ref{lem2.5} and the relation $\\m{1}\\otimes\\overline{\\nabla} f({\\m{x}}^t)=\\m{M}{\\m{v}}^t$ by \\eqref{vequalg}.\n\tSubstituting \\eqref{termA}, \\eqref{termB}, and \\eqref{termC} into \\eqref{terms}, we obtain\n\t\\begin{align*}\n\t\t&\\Big\\langle\\nabla F(\\bar{\\m{x}}^t),\\bar{\\m{x}}^{t+1}-\\bar{\\m{x}}^{t}\\Big\\rangle+\\frac{1}{n} \\Big\\langle \\frac{1}{\\alpha} (\\m{I}-\\m{W})\\m{x}^t, \\m{x}^{t+1}-\\m{x}^t\\Big\\rangle\\\\\n\t\t\\leq &-\\left(\\frac{\\alpha(2-c)}{ n}-\\frac{2\\alpha^2}{n}\\left(\\frac{1}{b}+\\frac{1}{d}\\right)\\right)\\|\\tilde{\\m{v}}^t\\|^2\n\t\t+\\frac{L^2b}{2n}\\|\\m{x}^t-\\m{M}\\m{x}^t\\|^2+\\frac{d}{2n}\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2.\n\t\\end{align*}\n\tBy substituting the above inequality into \\eqref{add}, it holds that\n\t\\begin{eqnarray*}\n\t&& F(\\bar{\\m{x}}^{t+1})+\\frac{1}{2\\alpha n} \\|\\m{x}^{t+1}\\|^2_{\\m{I}-\\m{W}} \\\\ \n      &\\leq& F(\\bar{\\m{x}}^t)+\\frac{1}{2\\alpha n} \\|\\m{x}^{t}\\|^2_{\\m{I}-\\m{W}}\n\t-\\left(\\frac{\\alpha(4-2c-c^2)}{ 2n}-\\frac{2\\alpha^2}{n}\\left(\\frac{1}{b}+\\frac{1}{d}\\right)-\\frac{L\\alpha^2c^2}{2n}\\right)\\|\\tilde{\\m{v}}^t\\|^2\\\\\n\t&&+\\frac{L^2b}{2n}\\|\\m{x}^t-\\m{M}\\m{x}^t\\|^2+\\frac{d}{2n}\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2.\n\\end{eqnarray*} \nSince $0<\\alpha \\leq \\frac{2\\sqrt{5}-4}{5L}$, we have from the definition of constant $c$ in \\eqref{const-c} that \n\\begin{equation}\\label{new-bd-c}\n1<c\\leq \\sqrt{5}/2, \n\\end{equation}\n which implies $\\frac{11}{4}-\\sqrt{5}\\leq 4-2c-c^2 \\leq 1$. Then, setting $b=d=32\\alpha$ yields \\eqref{term1}.\n%\t\\begin{align*}\n%\t\tF(\\bar{\\m{x}}^{t+1})+\\frac{1}{2\\alpha n} \\|\\m{x}^{t+1}\\|^2_{\\m{I}-\\m{W}}\\leq& F(\\bar{\\m{x}}^t)+\\frac{1}{2\\alpha n} \\|\\m{x}^{t}\\|^2_{\\m{I}-\\m{W}}-\\left(\\frac{\\alpha(5-2\\sqrt{5})}{4 n}-\\frac{5L\\alpha^2}{8n}\\right)\\| \\tilde{\\m{v}}^t\\|^2\\\\\n%\t\t&+\\frac{16L^2\\alpha}{n}\\|\\m{x}^t-\\m{M}\\m{x}^t\\|^2+\\frac{16\\alpha}{n}\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2.\n%\t\\end{align*} \n\\end{proof}","sha256_utf8":"4fde0c3738d1470201fadc324f1a6a50b05e3356fbbe453342a63d51b919f33b"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"平均目标光滑上界","source_units":[{"start":43553,"end":43874,"start_line":593,"end_line":596,"raw_tex":"\tBy Assumption \\ref{as0},  $F$ is Lipschitz continuously differentiable with Lipschitz constant $L$, which gives\n\t\\begin{align*}\n\t\tF(\\bar{\\m{x}}^{t+1})\\leq F(\\bar{\\m{x}}^t)+\\Big\\langle\\nabla F(\\bar{\\m{x}}^t),\\bar{\\m{x}}^{t+1}-\\bar{\\m{x}}^{t}\\Big\\rangle+\\frac{L}{2}\\|\\bar{\\m{x}}^{t+1}-\\bar{\\m{x}}^{t}\\|^{2}.\n\t\\end{align*}\n","sha256_utf8":"93538d7297d109fbd80563cc5efe034e81762efe9bbdfddf5fabd719bdc7e5e3"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"网络二次能量上界","source_units":[{"start":43874,"end":44194,"start_line":597,"end_line":600,"raw_tex":"\tSince $\\rho(\\m{I}-\\m{W})<2$, we  have\n\t\\begin{align*}\n\t\t\\frac{1}{2\\alpha n} \\|\\m{x}^{t+1}\\|^2_{\\m{I}-\\m{W}} \\leq \\frac{1}{2\\alpha n} \\|\\m{x}^{t}\\|^2_{\\m{I}-\\m{W}}+\\frac{1}{n} \\Big\\langle \\frac{1}{\\alpha} (\\m{I}-\\m{W})\\m{x}^t, \\m{x}^{t+1}-\\m{x}^t\\Big\\rangle + \\frac{1}{2\\alpha n}\\|\\m{x}^{t+1}-\\m{x}^t\\|^2.\n\t\\end{align*}\n","sha256_utf8":"4696a143631d8470fd1f1d4336b7cb5a9b0a5f9a0080edc7e80dde71c71e035c"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"平均位移范数控制","source_units":[{"start":44194,"end":44482,"start_line":601,"end_line":604,"raw_tex":"In addition, we have\n\\begin{align*}\n\t&\\frac{L}{2}\\|\\bar{\\m{x}}^{t+1}-\\bar{\\m{x}}^{t}\\|^{2}=\\frac{L}{2n}\\|\\m{1}\\otimes\\bar{\\m{x}}^{t+1}-\\m{1}\\otimes\\bar{\\m{x}}^{t}\\|^{2}=\\frac{L}{2n}\\|\\m{M}{\\m{x}}^{t+1}-\\m{M}{\\m{x}}^{t}\\|^{2}\\leq\\frac{L}{2n}\\|{\\m{x}}^{t+1}-{\\m{x}}^{t}\\|^{2}.\n\\end{align*}\n","sha256_utf8":"3817988f6d5562fff95b4285fde5b0aeb09847f3e079786105c9c1ac7de4f06a"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"合并初步能量上界","source_units":[{"start":44482,"end":44880,"start_line":605,"end_line":609,"raw_tex":"\tAdding the above three inequalities together and using Lemma~\\ref{lem2.5} yields \n\t\\begin{eqnarray}\\label{add}\n\t   & & F(\\bar{\\m{x}}^{t+1})+\\frac{1}{2\\alpha n} \\|\\m{x}^{t+1}\\|^2_{\\m{I}-\\m{W}} \\\\\n  &\\leq& F(\\bar{\\m{x}}^t)+\\frac{1}{2\\alpha n} \\|\\m{x}^{t}\\|^2_{\\m{I}-\\m{W}}+\\left(\\frac{L\\alpha^2c^2}{2n}+\\frac{\\alpha c^2}{2 n}\\right) \\|\\tilde{\\m{v}}^t\\|^2 + \\mbox{RemainTerm} \\notag \n\t\\end{eqnarray}\n","sha256_utf8":"80a9f0a6adc634876935433884211b81b833feed53d11f8fe2ced77fadb3791a"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"余项分解为三项","source_units":[{"start":44880,"end":45874,"start_line":610,"end_line":617,"raw_tex":"\twhere \n\t\\begin{align}\n\t\t&\\mbox{RemainTerm} = \\Big\\langle\\nabla F(\\bar{\\m{x}}^t),\\bar{\\m{x}}^{t+1}-\\bar{\\m{x}}^{t}\\Big\\rangle + \\frac{1}{n} \\Big\\langle \\frac{1}{\\alpha} (\\m{I}-\\m{W})\\m{x}^t, \\m{x}^{t+1}-\\m{x}^t\\Big\\rangle \\notag\\\\\n\t\t=&\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\nabla F(\\bar{\\m{x}}^t),{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle+\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle \\frac{1}{\\alpha} (\\m{x}_i^{t}-\\sum_{j \\in \\mathcal{N}_i} \\tilde{W}_{i j} \\m{x}_j^{t}), \\m{x}_i^{t+1}-\\m{x}_i^t\\Big\\rangle \\notag\\\\\n\t\t=&\\underbrace{\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\tilde{\\m{v}}_i^t,{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle}_{\\text {term (A)}}\n\t\t+\\underbrace{\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\nabla F(\\bar{\\m{x}}^t)-\\overline{\\nabla} f({\\m{x}}^t),{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle}_{\\text {term (B)}}\\notag\\\\\n\t\t&+\\underbrace{\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\overline{\\nabla} f({\\m{x}}^t)-{\\m{v}}_i^t,{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle}_{\\text {term (C)}}. \\label{terms}\n\t\\end{align}\n","sha256_utf8":"9030db9e0358eed5c9647394ca575e2777e3572c3ef8b6a5dc910625f4c3e929"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"估计方向下降项","source_units":[{"start":45874,"end":46169,"start_line":618,"end_line":621,"raw_tex":"\tFor term (A), by Lemma~\\ref{lem2.5} and $\\m{x}_i^{t+1}=\\m{x}_i^{t}+\\alpha \\tilde{\\m{d}}_i^{t}$, we have\n\t\\begin{align}\\label{termA}\n\t\t\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\tilde{\\m{v}}_i^t,{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle \\leq -\\frac{\\alpha(2-c)}{ n} \\|\\tilde{\\m{v}}^t\\|^2.\n\t\\end{align}\n","sha256_utf8":"e3719377fa0b37302db3b31751290786ca037c796ca72791a9859fa6e259f01b"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"估计梯度不一致项","source_units":[{"start":46169,"end":46928,"start_line":622,"end_line":629,"raw_tex":"\tFor term (B), by Young's inequality with constant $b >0$ and  $\\m{1}\\otimes\\bar{\\m{x}}^t=\\m{M}{\\m{x}}^t$, we have \n\t\\begin{eqnarray}\\label{termB}\n\t  &&\t\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\nabla F(\\bar{\\m{x}}^t)-\\overline{\\nabla} f({\\m{x}}^t),{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle \\\\\n  &\\leq& \\frac{b}{2}\\|\\nabla F(\\bar{\\m{x}}^t)-\\overline{\\nabla} f({\\m{x}}^t)\\|^2+\\frac{1}{2nb}\\|{\\m{x}}^{t+1}-{\\m{x}}^{t}\\|^2 \\notag\\\\\n\t\t&\\leq & \\frac{L^2b}{2n}\\|\\m{1}\\otimes\\bar{\\m{x}}^t-\\m{x}^t \\|^2+\\frac{2\\alpha^2}{nb}\\|\\tilde{\\m{v}}^t\\|^2\n\t   =\\frac{L^2b}{2n}\\|\\m{M}\\m{x}^t-\\m{x}^t \\|^2+\\frac{2\\alpha^2}{nb}\\|\\tilde{\\m{v}}^t\\|^2,\\notag\n\t\\end{eqnarray}\nwhere the last inequality uses Lemma~\\ref{important}, Lemma~\\ref{lem2.5},  $L$-Lipschitz continuity of $\\nabla F$.\n","sha256_utf8":"9ce33e175c982bebed6e4e9237532c0b6f8508f7f44820170e173191e19d5420"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"估计梯度跟踪项","source_units":[{"start":46928,"end":47536,"start_line":630,"end_line":635,"raw_tex":"\tFor term (C), by Young's inequality with constant $d>0$, we have\n\t\\begin{align}\\label{termC}\n\t\t\\frac{1}{n} \\sum_{i=1}^n \\Big\\langle\\overline{\\nabla} f({\\m{x}}^t)-{\\m{v}}_i^t,{\\m{x}}_i^{t+1}-{\\m{x}}_i^{t}\\Big\\rangle \\leq &\\frac{d}{2n}\\|\\m{1}\\otimes\\overline{\\nabla} f({\\m{x}}^t)-{\\m{v}}^t\\|^2+\\frac{1}{2nd}\\|{\\m{x}}^{t+1}-{\\m{x}}^{t}\\|^2 \\notag \\\\\n\t\t\\leq & \\frac{d}{2n}\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+\\frac{2\\alpha^2}{nd}\\|\\tilde{\\m{v}}^t\\|^2,\n\t\\end{align}\nwhere the last inequality is due to Lemma~\\ref{lem2.5} and the relation $\\m{1}\\otimes\\overline{\\nabla} f({\\m{x}}^t)=\\m{M}{\\m{v}}^t$ by \\eqref{vequalg}.\n","sha256_utf8":"5eb37da34637caf739deb3ddbbcb09bca118c48ade73f19804015759085ceb12"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"合并三项余项界","source_units":[{"start":47536,"end":48048,"start_line":636,"end_line":641,"raw_tex":"\tSubstituting \\eqref{termA}, \\eqref{termB}, and \\eqref{termC} into \\eqref{terms}, we obtain\n\t\\begin{align*}\n\t\t&\\Big\\langle\\nabla F(\\bar{\\m{x}}^t),\\bar{\\m{x}}^{t+1}-\\bar{\\m{x}}^{t}\\Big\\rangle+\\frac{1}{n} \\Big\\langle \\frac{1}{\\alpha} (\\m{I}-\\m{W})\\m{x}^t, \\m{x}^{t+1}-\\m{x}^t\\Big\\rangle\\\\\n\t\t\\leq &-\\left(\\frac{\\alpha(2-c)}{ n}-\\frac{2\\alpha^2}{n}\\left(\\frac{1}{b}+\\frac{1}{d}\\right)\\right)\\|\\tilde{\\m{v}}^t\\|^2\n\t\t+\\frac{L^2b}{2n}\\|\\m{x}^t-\\m{M}\\m{x}^t\\|^2+\\frac{d}{2n}\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2.\n\t\\end{align*}\n","sha256_utf8":"3e8fc1940b53affbcfef6bfb1a715f6fa4597599e425bdae25a58fdd74663cf8"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"代回参数化能量界","source_units":[{"start":48048,"end":48552,"start_line":642,"end_line":648,"raw_tex":"\tBy substituting the above inequality into \\eqref{add}, it holds that\n\t\\begin{eqnarray*}\n\t&& F(\\bar{\\m{x}}^{t+1})+\\frac{1}{2\\alpha n} \\|\\m{x}^{t+1}\\|^2_{\\m{I}-\\m{W}} \\\\ \n      &\\leq& F(\\bar{\\m{x}}^t)+\\frac{1}{2\\alpha n} \\|\\m{x}^{t}\\|^2_{\\m{I}-\\m{W}}\n\t-\\left(\\frac{\\alpha(4-2c-c^2)}{ 2n}-\\frac{2\\alpha^2}{n}\\left(\\frac{1}{b}+\\frac{1}{d}\\right)-\\frac{L\\alpha^2c^2}{2n}\\right)\\|\\tilde{\\m{v}}^t\\|^2\\\\\n\t&&+\\frac{L^2b}{2n}\\|\\m{x}^t-\\m{M}\\m{x}^t\\|^2+\\frac{d}{2n}\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2.\n\\end{eqnarray*} \n","sha256_utf8":"7f58d8a47035c80daf443b01341323e1369213195a6cd70bf33e8b088ea11352"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"控制递推常数并选参","source_units":[{"start":48552,"end":48847,"start_line":649,"end_line":653,"raw_tex":"Since $0<\\alpha \\leq \\frac{2\\sqrt{5}-4}{5L}$, we have from the definition of constant $c$ in \\eqref{const-c} that \n\\begin{equation}\\label{new-bd-c}\n1<c\\leq \\sqrt{5}/2, \n\\end{equation}\n which implies $\\frac{11}{4}-\\sqrt{5}\\leq 4-2c-c^2 \\leq 1$. Then, setting $b=d=32\\alpha$ yields \\eqref{term1}.\n","sha256_utf8":"87a9a918d42dd2c6e012afc941537fe099a8c70c54229b34671672b940aaf155"}]},{"id":"S035","semantic_type":"theorem","kind":"theorem","title":"NDCG非凸收敛速率","source_units":[{"start":49236,"end":49959,"start_line":661,"end_line":673,"raw_tex":"\\begin{theorem}\\label{nonconvex_convergence}\n\tSuppose Assumption \\ref{as0} holds. If\n\t\\begin{align}\\label{alpha}\n\t0 <\t\\alpha < \\min \\left\\{\\frac{(1-{\\sigma}^2)n}{32L^2},\\frac{2(5-2\\sqrt{5})(1-{\\sigma}^2)}{4n(5\\sigma^2L^2+18-8\\sqrt{5})+5L},\\frac{2\\sqrt{5}-4}{5L}\\right\\},\n\t\\end{align}\n\tthen we have  \n\t\\begin{equation}\\label{cg-converge-rate}\n\t \\min_{0 \\le t \\le T} \\left\\{ \\|\\tilde{\\m{v}}^t\\|^2+\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2 \\right\\}\n\t\t\\leq \\frac{P(\\m{x}^{0};\\m{v}^{0})-P(\\m{x}^{T};\\m{v}^{T})}{\\gamma \\min\\{\\alpha,1\\}T},\n\t\\end{equation}\n\twhere $P$ is the potential function defined in \\eqref{Potential-P} and\n\t$\\gamma >0$ is a constant depending only on $n$, $L$, and $\\sigma$.\n\\end{theorem}","sha256_utf8":"04fe87b5b00adec6efaeaad347cb8758863937fa35bc6b621b96f11d33a24bab"}]},{"id":"S036","semantic_type":"assumption","kind":"assumption","title":"主定理步长条件","source_units":[{"start":49281,"end":49520,"start_line":662,"end_line":665,"raw_tex":"\tSuppose Assumption \\ref{as0} holds. If\n\t\\begin{align}\\label{alpha}\n\t0 <\t\\alpha < \\min \\left\\{\\frac{(1-{\\sigma}^2)n}{32L^2},\\frac{2(5-2\\sqrt{5})(1-{\\sigma}^2)}{4n(5\\sigma^2L^2+18-8\\sqrt{5})+5L},\\frac{2\\sqrt{5}-4}{5L}\\right\\},\n\t\\end{align}\n","sha256_utf8":"0d042fa00944a6d0e023c8769b2d1453e1063e3be848cc69eeea9edfb0b99581"}]},{"id":"S037","semantic_type":"proof","kind":"proof","title":"非凸收敛定理完整证明","source_units":[{"start":49960,"end":54537,"start_line":674,"end_line":735,"raw_tex":"\\begin{proof}\n\tFirst, by using Young's inequality with constant $\\eta >0$, we \n\t establish a recursion upper bound on $\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2$.\n\t\\begin{align}\\label{term2}\n\t\t\\|{\\m{v}}^{t+1}-\\m{M}{\\m{v}}^{t+1}\\|^2\n\t\t=&\\| \\m{W}{\\m{v}}^{t}+\\m{W}\\m{g}^{t+1}-\\m{W}\\m{g}^{t} -\\m{M}\\m{v}^t-\\m{M}\\m{g}^{t+1}+\\m{M}\\m{g}^t\\|^2 \\notag\\\\\n\t\t\\leq & (1+\\eta)\\|\\m{W}{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+(1+1/\\eta)\\|(\\m{W}-\\m{M})(\\m{g}^{t+1}-\\m{g}^t)\\|^2\\notag\\\\\n\t\t\\leq & (1+\\eta)\\sigma^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+(1+1/\\eta)\\sigma^2L^2\\|\\m{x}^{t+1}-\\m{x}^t\\|^2,\\notag\\\\\n\t\t\\leq & (1+\\eta)\\sigma^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+(1+1/\\eta)\\sigma^2L^2 c^2 \\alpha^2\\|\\tilde{\\m{v}}^t\\|^2,\\notag\\\\\n\t\t\\leq & (1+\\eta)\\sigma^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+(1+1/\\eta)(5/4)\\sigma^2L^2\\alpha^2\\|\\tilde{\\m{v}}^t\\|^2,\n\t\\end{align}\nwhere the second inequality uses Lemma~\\ref{property W} and $L$-Lipschitz continuity of $\\nabla f$,\nthe third inequality uses Lemma~\\ref{lem2.5},\nthe last inequality applies  $\\alpha \\leq \\frac{2\\sqrt{5}-4}{5L}$ and \\eqref{new-bd-c}.\n\n We now establish a recursion upper bound on $\\|{\\m{x}}^t-\\m{M}{\\m{x}}^t\\|^2$.\n  By rewriting $\\m{x}^{t+1}=\\m{W}\\m{x}^t-\\alpha\\m{v}^t+\\alpha \\bm{\\beta}^{t}\\tilde{\\m{d}}^{t-1}$\n  and applying Young's inequality with constant $\\tau >0$, we have\n\t\\begin{align}\\label{term3}\n\t\t\\|\\m{x}^{t+1}-\\m{M}\\m{x}^{t+1}\\|^2\n\t\t=&\\|\\m{W}\\m{x}^t-\\m{M}\\m{x}^t-\\alpha\\m{v}^t+\\alpha\\m{M}\\m{v}^t+\\alpha \\bm{\\beta}^{t}\\tilde{\\m{d}}^{t-1}-\\alpha\\m{M} \\bm{\\beta}^{t}\\tilde{\\m{d}}^{t-1}\\|^2 \\notag \\\\\n\t\t\\leq &(1+\\tau) \\|\\m{W}\\m{x}^t-\\m{M}\\m{x}^t\\|^2\\notag+2(1+1/\\tau)\\alpha^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2\\notag\\\\\n\t\t&+2(1+1/\\tau)\\alpha^2\\|(\\m{I}-\\m{M})\\bm{\\beta}^{t}\\tilde{\\m{d}}^{t-1}\\|^2\\notag\\\\\n\t\t\\leq&(1+\\tau){\\sigma}^2 \\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2+2(1+1/\\tau)\\alpha^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2 \\notag \\\\\n\t\t&+2(1+1/\\tau)\\alpha^2\\|\\bm{\\beta}^{t}\\tilde{\\m{d}}^{t-1}\\|^2,\\notag\\\\\n\t\t\\leq&(1+\\tau){\\sigma}^2 \\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2+2(1+1/\\tau)\\alpha^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2 \\notag \\\\\n\t\t&+2(1+1/\\tau)(9/4-\\sqrt{5})\\alpha^2\\|\\tilde{\\m{v}}^{t}\\|^2, \n\t\\end{align}\t\nwhere the second inequality uses Lemma \\ref{property W} and  $\\rho(\\m{I}-\\m{M}) \\leq 1$,\nand the last inequality uses\n$\\alpha \\leq \\frac{2\\sqrt{5}-4}{5L}$, Lemma~\\ref{lem2.5}, \\eqref{lem2.5_eq1} and \\eqref{new-bd-c}.\nAdding up \\eqref{term1}, \\eqref{term2}, and \\eqref{term3} yields\n\t\\begin{align*}\n\t\t&P(\\m{x}^{t+1},\\m{v}^{t+1})\\\\\n\t\t\\leq &\tP(\\m{x}^{t},\\m{v}^{t})-\\left(1-(1+\\tau){\\sigma}^2-\\frac{16L^2\\alpha}{n}\\right)\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2\n\t\t\\\\\n\t\t&-\\left(1-(1+\\eta)\\sigma^2-2\\left(1+\\frac{1}{\\tau}\\right)\\alpha^2-\\frac{16\\alpha}{n}\\right)\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2\\notag\\\\\n\t\t&-\\left(\\frac{\\alpha(5-2\\sqrt{5})}{4 n}-\\frac{5L\\alpha^2}{8n}-\\left(1+\\frac{1}{\\eta}\\right)\\frac{5}{4}\\sigma^2L^2\\alpha^2-2\\left(1+\\frac{1}{\\tau}\\right)\\left(\\frac{9}{4}-\\sqrt{5}\\right)\\alpha^2\\right)\\|\\tilde{\\m{v}}^{t}\\|^2.\n\t\\end{align*}\nThen, setting $\\eta=\\tau=\\frac{1-{\\sigma}^2}{2{\\sigma}^2}$, we have\n\t\\begin{align}\\label{P_descent}\n\t\t&P(\\m{x}^{t+1},\\m{v}^{t+1})\\\\\n\t\t\\leq \t&P(\\m{x}^{t},\\m{v}^{t})-\\left(\\frac{1-{\\sigma}^2}{2}-\\frac{16L^2\\alpha}{n}\\right)\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2\\notag\\\\\n\t\t&-\\left(\\frac{\\alpha(5-2\\sqrt{5})}{4 n}-\\frac{5L\\alpha^2}{8n}-\\frac{5(1+\\sigma^2)\\sigma^2L^2\\alpha^2}{4(1-\\sigma^2)}-\\frac{2(1+{\\sigma}^2)\\alpha^2}{1-{\\sigma}^2}\\left(\\frac{9}{4}-\\sqrt{5}\\right)\\right)\\|\\tilde{\\m{v}}^{t}\\|^2\\notag\\\\\n\t\t&-\\left(\\frac{1-\\sigma^2}{2}-\\frac{2(1+{\\sigma}^2)\\alpha^2}{1-{\\sigma}^2}-\\frac{16\\alpha}{n}\\right)\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2\n\t\t\\notag\\\\\n\t\t\\leq& P(,\\m{x}^{t},\\m{v}^{t})\n\t\t-\\left(\\frac{\\alpha(5-2\\sqrt{5})}{4 n}-\\frac{4n(5\\sigma^2L^2+18-8\\sqrt{5})+5L}{8n(1-{\\sigma}^2)}\\alpha^2\\right)\\|\\tilde{\\m{v}}^{t}\\|^2\\notag\\\\\n\t\t&-\\left(\\frac{1-\\sigma^2}{2}-\\frac{4\\alpha^2}{1-{\\sigma}^2}-\\frac{16\\alpha}{n}\\right)\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2\n\t\t-\\left(\\frac{1-{\\sigma}^2}{2}-\\frac{16L^2\\alpha}{n}\\right)\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2,\\notag\n\t\\end{align}\nwhere the last inequality sues $0<\\sigma<1$.\n\tThen, by the choice of $\\alpha$ in \\eqref{alpha}, we can derive from \\eqref{P_descent}\n\tand direct calculations\tthat there exist a constant $\\gamma >0$, depending only on \n\t$n$, $L$, $\\sigma$, such that\n\t\\begin{equation*}\n\t\tP(\\m{x}^{t+1},\\m{v}^{t+1}) \\leq P(\\m{x}^{t},\\m{v}^{t}) - \\gamma \\left( \n\t\t\\alpha \\|\\tilde{\\m{v}}^t\\|^2 +\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2 \\right). \n\t\\end{equation*}\n\tSumming the above relation over $t=1, \\ldots, T$ and dividing $\\gamma \\min\\{\\alpha,1\\}$ from \n\tboth sides gives \\eqref{cg-converge-rate}.\n\\end{proof}","sha256_utf8":"b3d74282c7aa0a392e863dbb126fd2d0da4c832bfdadb2365d974a0777c5bd97"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"梯度跟踪误差递推","source_units":[{"start":49974,"end":50999,"start_line":675,"end_line":687,"raw_tex":"\tFirst, by using Young's inequality with constant $\\eta >0$, we \n\t establish a recursion upper bound on $\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2$.\n\t\\begin{align}\\label{term2}\n\t\t\\|{\\m{v}}^{t+1}-\\m{M}{\\m{v}}^{t+1}\\|^2\n\t\t=&\\| \\m{W}{\\m{v}}^{t}+\\m{W}\\m{g}^{t+1}-\\m{W}\\m{g}^{t} -\\m{M}\\m{v}^t-\\m{M}\\m{g}^{t+1}+\\m{M}\\m{g}^t\\|^2 \\notag\\\\\n\t\t\\leq & (1+\\eta)\\|\\m{W}{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+(1+1/\\eta)\\|(\\m{W}-\\m{M})(\\m{g}^{t+1}-\\m{g}^t)\\|^2\\notag\\\\\n\t\t\\leq & (1+\\eta)\\sigma^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+(1+1/\\eta)\\sigma^2L^2\\|\\m{x}^{t+1}-\\m{x}^t\\|^2,\\notag\\\\\n\t\t\\leq & (1+\\eta)\\sigma^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+(1+1/\\eta)\\sigma^2L^2 c^2 \\alpha^2\\|\\tilde{\\m{v}}^t\\|^2,\\notag\\\\\n\t\t\\leq & (1+\\eta)\\sigma^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+(1+1/\\eta)(5/4)\\sigma^2L^2\\alpha^2\\|\\tilde{\\m{v}}^t\\|^2,\n\t\\end{align}\nwhere the second inequality uses Lemma~\\ref{property W} and $L$-Lipschitz continuity of $\\nabla f$,\nthe third inequality uses Lemma~\\ref{lem2.5},\nthe last inequality applies  $\\alpha \\leq \\frac{2\\sqrt{5}-4}{5L}$ and \\eqref{new-bd-c}.\n","sha256_utf8":"3a66deccce6ce8a4dc63cd55077d17c0da2da949831a23b42c133c8bd03b15a4"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"变量一致性误差递推","source_units":[{"start":51000,"end":52271,"start_line":689,"end_line":704,"raw_tex":" We now establish a recursion upper bound on $\\|{\\m{x}}^t-\\m{M}{\\m{x}}^t\\|^2$.\n  By rewriting $\\m{x}^{t+1}=\\m{W}\\m{x}^t-\\alpha\\m{v}^t+\\alpha \\bm{\\beta}^{t}\\tilde{\\m{d}}^{t-1}$\n  and applying Young's inequality with constant $\\tau >0$, we have\n\t\\begin{align}\\label{term3}\n\t\t\\|\\m{x}^{t+1}-\\m{M}\\m{x}^{t+1}\\|^2\n\t\t=&\\|\\m{W}\\m{x}^t-\\m{M}\\m{x}^t-\\alpha\\m{v}^t+\\alpha\\m{M}\\m{v}^t+\\alpha \\bm{\\beta}^{t}\\tilde{\\m{d}}^{t-1}-\\alpha\\m{M} \\bm{\\beta}^{t}\\tilde{\\m{d}}^{t-1}\\|^2 \\notag \\\\\n\t\t\\leq &(1+\\tau) \\|\\m{W}\\m{x}^t-\\m{M}\\m{x}^t\\|^2\\notag+2(1+1/\\tau)\\alpha^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2\\notag\\\\\n\t\t&+2(1+1/\\tau)\\alpha^2\\|(\\m{I}-\\m{M})\\bm{\\beta}^{t}\\tilde{\\m{d}}^{t-1}\\|^2\\notag\\\\\n\t\t\\leq&(1+\\tau){\\sigma}^2 \\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2+2(1+1/\\tau)\\alpha^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2 \\notag \\\\\n\t\t&+2(1+1/\\tau)\\alpha^2\\|\\bm{\\beta}^{t}\\tilde{\\m{d}}^{t-1}\\|^2,\\notag\\\\\n\t\t\\leq&(1+\\tau){\\sigma}^2 \\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2+2(1+1/\\tau)\\alpha^2\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2 \\notag \\\\\n\t\t&+2(1+1/\\tau)(9/4-\\sqrt{5})\\alpha^2\\|\\tilde{\\m{v}}^{t}\\|^2, \n\t\\end{align}\t\nwhere the second inequality uses Lemma \\ref{property W} and  $\\rho(\\m{I}-\\m{M}) \\leq 1$,\nand the last inequality uses\n$\\alpha \\leq \\frac{2\\sqrt{5}-4}{5L}$, Lemma~\\ref{lem2.5}, \\eqref{lem2.5_eq1} and \\eqref{new-bd-c}.\n","sha256_utf8":"e4a9892fc51df9157ad59c4f6aedce3d2b78764662dbbdc15b3165201d35e9f3"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"合并势函数三项递推","source_units":[{"start":52271,"end":52880,"start_line":705,"end_line":712,"raw_tex":"Adding up \\eqref{term1}, \\eqref{term2}, and \\eqref{term3} yields\n\t\\begin{align*}\n\t\t&P(\\m{x}^{t+1},\\m{v}^{t+1})\\\\\n\t\t\\leq &\tP(\\m{x}^{t},\\m{v}^{t})-\\left(1-(1+\\tau){\\sigma}^2-\\frac{16L^2\\alpha}{n}\\right)\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2\n\t\t\\\\\n\t\t&-\\left(1-(1+\\eta)\\sigma^2-2\\left(1+\\frac{1}{\\tau}\\right)\\alpha^2-\\frac{16\\alpha}{n}\\right)\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2\\notag\\\\\n\t\t&-\\left(\\frac{\\alpha(5-2\\sqrt{5})}{4 n}-\\frac{5L\\alpha^2}{8n}-\\left(1+\\frac{1}{\\eta}\\right)\\frac{5}{4}\\sigma^2L^2\\alpha^2-2\\left(1+\\frac{1}{\\tau}\\right)\\left(\\frac{9}{4}-\\sqrt{5}\\right)\\alpha^2\\right)\\|\\tilde{\\m{v}}^{t}\\|^2.\n\t\\end{align*}\n","sha256_utf8":"292d254842fba4dd6fcd92169f3d79b2c34fa2855566b80a242ea58db12329cf"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"选取Young参数并化简","source_units":[{"start":52880,"end":53969,"start_line":713,"end_line":725,"raw_tex":"Then, setting $\\eta=\\tau=\\frac{1-{\\sigma}^2}{2{\\sigma}^2}$, we have\n\t\\begin{align}\\label{P_descent}\n\t\t&P(\\m{x}^{t+1},\\m{v}^{t+1})\\\\\n\t\t\\leq \t&P(\\m{x}^{t},\\m{v}^{t})-\\left(\\frac{1-{\\sigma}^2}{2}-\\frac{16L^2\\alpha}{n}\\right)\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2\\notag\\\\\n\t\t&-\\left(\\frac{\\alpha(5-2\\sqrt{5})}{4 n}-\\frac{5L\\alpha^2}{8n}-\\frac{5(1+\\sigma^2)\\sigma^2L^2\\alpha^2}{4(1-\\sigma^2)}-\\frac{2(1+{\\sigma}^2)\\alpha^2}{1-{\\sigma}^2}\\left(\\frac{9}{4}-\\sqrt{5}\\right)\\right)\\|\\tilde{\\m{v}}^{t}\\|^2\\notag\\\\\n\t\t&-\\left(\\frac{1-\\sigma^2}{2}-\\frac{2(1+{\\sigma}^2)\\alpha^2}{1-{\\sigma}^2}-\\frac{16\\alpha}{n}\\right)\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2\n\t\t\\notag\\\\\n\t\t\\leq& P(,\\m{x}^{t},\\m{v}^{t})\n\t\t-\\left(\\frac{\\alpha(5-2\\sqrt{5})}{4 n}-\\frac{4n(5\\sigma^2L^2+18-8\\sqrt{5})+5L}{8n(1-{\\sigma}^2)}\\alpha^2\\right)\\|\\tilde{\\m{v}}^{t}\\|^2\\notag\\\\\n\t\t&-\\left(\\frac{1-\\sigma^2}{2}-\\frac{4\\alpha^2}{1-{\\sigma}^2}-\\frac{16\\alpha}{n}\\right)\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2\n\t\t-\\left(\\frac{1-{\\sigma}^2}{2}-\\frac{16L^2\\alpha}{n}\\right)\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2,\\notag\n\t\\end{align}\nwhere the last inequality sues $0<\\sigma<1$.\n","sha256_utf8":"fae93afb4b54eaa440a4e23429a79324e31db474647217f6882d171689eedf23"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"获得势函数充分下降","source_units":[{"start":53969,"end":54387,"start_line":726,"end_line":732,"raw_tex":"\tThen, by the choice of $\\alpha$ in \\eqref{alpha}, we can derive from \\eqref{P_descent}\n\tand direct calculations\tthat there exist a constant $\\gamma >0$, depending only on \n\t$n$, $L$, $\\sigma$, such that\n\t\\begin{equation*}\n\t\tP(\\m{x}^{t+1},\\m{v}^{t+1}) \\leq P(\\m{x}^{t},\\m{v}^{t}) - \\gamma \\left( \n\t\t\\alpha \\|\\tilde{\\m{v}}^t\\|^2 +\\|{\\m{v}}^t-\\m{M}{\\m{v}}^t\\|^2+\\|\\m{x}^{t}-\\m{M}\\m{x}^{t}\\|^2 \\right). \n\t\\end{equation*}\n","sha256_utf8":"f113b89d4128518667a96661c9d1192ddf1bc3c9e07931ef3870b2509f9cbb59"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"累加下降式得到速率","source_units":[{"start":54387,"end":54526,"start_line":733,"end_line":734,"raw_tex":"\tSumming the above relation over $t=1, \\ldots, T$ and dividing $\\gamma \\min\\{\\alpha,1\\}$ from \n\tboth sides gives \\eqref{cg-converge-rate}.\n","sha256_utf8":"a89087a44d73d0bead5ad229e792cdb11fa5ee050c1a1e08e915b356871ea7cb"}]},{"id":"S044","semantic_type":"external_fact","kind":"external_fact","title":"标准混合矩阵谱理论","source_units":[],"declared_external":true,"declaration_text":["引用 xin2019distributed 的标准边界事实：对于本文连通无向网络及混合矩阵，B0008 所列谱界、MW=WM=M，以及 ||(W-M)x||≤σ||(I-M)x|| 成立。"]},{"id":"S045","semantic_type":"external_fact","kind":"external_fact","title":"标准Young不等式","source_units":[],"declared_external":true,"declaration_text":["对欧氏向量 a,b 及 η>0，有 2aᵀb≤η||a||²+η⁻¹||b||²，且 ||a+b||²≤(1+η)||a||²+(1+η⁻¹)||b||²。"]},{"id":"S046","semantic_type":"external_fact","kind":"external_fact","title":"标准Jensen不等式","source_units":[],"declared_external":true,"declaration_text":["对有限组欧氏向量 a_i，有 ||n⁻¹Σ_i a_i||²≤n⁻¹Σ_i||a_i||²。"]},{"id":"S047","semantic_type":"external_fact","kind":"external_fact","title":"已引用的标量递推界","source_units":[],"declared_external":true,"declaration_text":["dai2011convergence 的 Lemma 2.2 提供 B0037 的边界事实：若 0<α≤1/(4L)，ξ⁰=1，ξᵗ⁺¹=1+Lα(ξᵗ)²，则所有 t≥0 满足 1≤ξᵗ<c，其中 c=2/(1+√(1-4Lα))≤2。"]},{"id":"S048","semantic_type":"external_fact","kind":"external_fact","title":"光滑函数下降引理","source_units":[],"declared_external":true,"declaration_text":["若 h 的梯度为 L-Lipschitz 连续，则任意 u,w 满足 h(w)≤h(u)+⟨∇h(u),w-u⟩+(L/2)||w-u||²。"]},{"id":"S049","semantic_type":"external_fact","kind":"external_fact","title":"欧氏范数与投影性质","source_units":[],"declared_external":true,"declaration_text":["欧氏向量满足 |aᵀb|≤||a||||b|| 及三角不等式；对正交投影 Q，有 ||Qa||≤||a||，其补投影 I-Q 亦非扩张。"]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S035","edges":[{"source_id":"S001","target_id":"S024","kind":"supports"},{"source_id":"S002","target_id":"S010","kind":"supports"},{"source_id":"S002","target_id":"S011","kind":"supports"},{"source_id":"S002","target_id":"S013","kind":"supports"},{"source_id":"S002","target_id":"S026","kind":"supports"},{"source_id":"S002","target_id":"S028","kind":"supports"},{"source_id":"S003","target_id":"S019","kind":"supports"},{"source_id":"S003","target_id":"S024","kind":"supports"},{"source_id":"S003","target_id":"S030","kind":"supports"},{"source_id":"S003","target_id":"S038","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S004","target_id":"S010","kind":"supports"},{"source_id":"S004","target_id":"S011","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S006","target_id":"S016","kind":"supports"},{"source_id":"S006","target_id":"S025","kind":"supports"},{"source_id":"S006","target_id":"S038","kind":"supports"},{"source_id":"S006","target_id":"S039","kind":"supports"},{"source_id":"S006","target_id":"S041","kind":"supports"},{"source_id":"S007","target_id":"S030","kind":"supports"},{"source_id":"S007","target_id":"S031","kind":"supports"},{"source_id":"S007","target_id":"S038","kind":"supports"},{"source_id":"S007","target_id":"S039","kind":"supports"},{"source_id":"S008","target_id":"S030","kind":"supports"},{"source_id":"S009","target_id":"S019","kind":"supports"},{"source_id":"S009","target_id":"S020","kind":"supports"},{"source_id":"S009","target_id":"S021","kind":"supports"},{"source_id":"S009","target_id":"S027","kind":"supports"},{"source_id":"S009","target_id":"S029","kind":"supports"},{"source_id":"S009","target_id":"S034","kind":"supports"},{"source_id":"S010","target_id":"S019","kind":"supports"},{"source_id":"S010","target_id":"S027","kind":"supports"},{"source_id":"S010","target_id":"S028","kind":"supports"},{"source_id":"S010","target_id":"S029","kind":"supports"},{"source_id":"S010","target_id":"S030","kind":"supports"},{"source_id":"S010","target_id":"S031","kind":"supports"},{"source_id":"S010","target_id":"S038","kind":"supports"},{"source_id":"S010","target_id":"S039","kind":"supports"},{"source_id":"S011","target_id":"S016","kind":"supports"},{"source_id":"S011","target_id":"S038","kind":"supports"},{"source_id":"S012","target_id":"S039","kind":"supports"},{"source_id":"S013","target_id":"S040","kind":"supports"},{"source_id":"S013","target_id":"S043","kind":"supports"},{"source_id":"S016","target_id":"S014","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S031","kind":"supports"},{"source_id":"S017","target_id":"S027","kind":"supports"},{"source_id":"S017","target_id":"S029","kind":"supports"},{"source_id":"S017","target_id":"S030","kind":"supports"},{"source_id":"S017","target_id":"S031","kind":"supports"},{"source_id":"S017","target_id":"S038","kind":"supports"},{"source_id":"S017","target_id":"S039","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S019","target_id":"S039","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S020","target_id":"S018","kind":"supports"},{"source_id":"S021","target_id":"S018","kind":"supports"},{"source_id":"S022","target_id":"S040","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S027","kind":"supports"},{"source_id":"S025","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S033","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S030","kind":"supports"},{"source_id":"S028","target_id":"S031","kind":"supports"},{"source_id":"S028","target_id":"S032","kind":"supports"},{"source_id":"S029","target_id":"S032","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S038","kind":"supports"},{"source_id":"S034","target_id":"S039","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S023","kind":"supports"},{"source_id":"S026","target_id":"S023","kind":"supports"},{"source_id":"S027","target_id":"S023","kind":"supports"},{"source_id":"S028","target_id":"S023","kind":"supports"},{"source_id":"S029","target_id":"S023","kind":"supports"},{"source_id":"S030","target_id":"S023","kind":"supports"},{"source_id":"S031","target_id":"S023","kind":"supports"},{"source_id":"S032","target_id":"S023","kind":"supports"},{"source_id":"S033","target_id":"S023","kind":"supports"},{"source_id":"S034","target_id":"S023","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S039","kind":"supports"},{"source_id":"S036","target_id":"S042","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S037","kind":"supports"},{"source_id":"S040","target_id":"S037","kind":"supports"},{"source_id":"S041","target_id":"S037","kind":"supports"},{"source_id":"S042","target_id":"S037","kind":"supports"},{"source_id":"S043","target_id":"S037","kind":"supports"},{"source_id":"S044","target_id":"S006","kind":"supports"},{"source_id":"S045","target_id":"S007","kind":"supports"},{"source_id":"S046","target_id":"S008","kind":"supports"},{"source_id":"S047","target_id":"S009","kind":"supports"},{"source_id":"S048","target_id":"S024","kind":"supports"},{"source_id":"S049","target_id":"S019","kind":"supports"},{"source_id":"S049","target_id":"S020","kind":"supports"},{"source_id":"S049","target_id":"S021","kind":"supports"},{"source_id":"S049","target_id":"S026","kind":"supports"},{"source_id":"S049","target_id":"S039","kind":"supports"}]}}
{"record_id":"2409.09906v4","paper":{"title":"Variance-reduced first-order methods for deterministically constrained stochastic nonconvex optimization with strong convergence guarantees","authors":["Zhaosong Lu","Sanyou Mei","Yifeng Xiao"],"arxiv_id":"2409.09906","version":"v4","journal":"SIAM Journal on Optimization","doi":"10.1137/24m1693933","abstract_url":"https://arxiv.org/abs/2409.09906v4","pdf_url":"https://arxiv.org/pdf/2409.09906v4"},"context":{"raw_tex":"\\beq\\label{prob}\n\\begin{aligned}\n\\min_{x\\in\\mcX}&\\quad f(x) :=\\bbE[\\tilde f(x,\\xi)]\\\\\n\\mbox{s.t.}&\\quad c(x)=0,\n\\end{aligned}\n\\eeq\n\nwhere $\\xi$ is a random variable with sample space $\\Xi$, $\\tf(\\cdot,\\xi)$ is continuously differentiable for each $\\xi\\in\\Xi$,  $c:\\rr^n\\to\\rr^m$ is a deterministic smooth mapping, and $\\mcX\\subseteq \\rr^n$ is a simple closed convex set whose projection operator can be evaluated exactly.\\footnote{For simplicity, we focus on problem \\eqref{prob} with equality constraints only. However, our results can be directly extended to problems involving both equality and inequality constraints; see the concluding remarks for further details.} Problem \\eqref{prob} arises in a variety of important areas, including energy systems \\cite{shabazbegian2020stochastic}, healthcare \\cite{shehadeh2022stochastic}, image processing \\cite{luke2020proximal}, machine learning \\cite{cuomo2022scientific,karniadakis2021physics},  network optimization \\cite{bertsekasnetwork}, optimal control \\cite{betts2010practical}, PDE-constrained optimization \\cite{rees2010optimal}, resource allocation \\cite{fontaine2020adaptive}, and transportation \\cite{maggioni2009stochastic}. More applications can be found, for example, in \\cite{bottou2018optimization,combettes2011proximal,jiang2018consensus,kushner2012stochastic,liu2015sparse}, and references therein.\n\n\n\\beq\\label{def-Q} \nQ_\\rho(x) := f(x) + \\frac{\\rho}{2} \\|c(x)\\|^2. \n\\eeq\n\nThe following notation will be used throughout this paper.  Let $\\rr_{>0}$ denote the set of positive real numbers, and  $\\rr^n$ denote the Euclidean space of dimension $n$. The standard inner product and Euclidean norm are denoted by $\\langle\\cdot,\\cdot\\rangle$ and $\\|\\cdot\\|$, respectively.  For any $r>0$, let $\\cB(r)$ represent the Euclidean ball centered at the origin with radius $r$, that is, $\\cB(r)=\\{x:\\|x\\|\\leq r\\}$.  For any $t\\in\\rr$, let $t_+=\\max\\{t,0\\}$ and $\\lceil t \\rceil$ denote the least integer greater than or equal to $t$.\n\nA mapping $\\phi$ is said to be \\emph{$L_{\\phi}$-Lipschitz continuous} on a set $\\Omega$ if $\\|\\phi(x)-\\phi(x')\\| \\leq L_{\\phi} \\|x-x'\\|$ for all $x,x'\\in \\Omega$. Also, it is said to be \\emph{$L_{\\nabla\\phi}$-smooth} on $\\Omega$ if $\\|\\nabla\\phi(x)-\\nabla\\phi(x')\\| \\leq L_{\\nabla\\phi} \\|x-x'\\|$ for all $x,x'\\in \\Omega$, where $\\nabla \\phi$ denotes the transpose of the Jacobian of $\\phi$.  Given a nonempty closed convex set $\\cC$, $\\dist(x,\\cC)$ denotes the Euclidean distance from $x$ to $\\cC$, and $\\Pi_\\cC(x)$ denotes the Euclidean projection of $x$ onto $\\Omega$.  In addition,  the normal cone of $\\cC$ at any $x\\in \\cC$ is denoted by $\\mcN_\\cC(x)$.\nFinally, we use $\\tmcO(\\cdot)$ to denote the asymptotic upper bound that ignores logarithmic factors.\n\n\n\\begin{assumption}\\label{a1}\n\\begin{enumerate}[label=(\\roman*)]\n\\item The optimal value $f^*$ of problem \\eqref{prob}  and $Q_1^*:= \\min_{x\\in X} \\{f(x)+\\|c(x)\\|^2/2\\}$ are finite.\n\\item $f$ is differentiable and $L_f$-Lipschitz continuous in an open neighborhood of $\\mcX$.\n\\item For each $\\xi\\in\\Xi$, $\\tf(\\cdot,\\xi)$ is differentiable on $\\mcX$ and satisfies the following conditions:\n\\[\n\\bbE[\\nabla\\tf(x,\\xi)]=\\nabla f(x),\\quad\\bbE[\\|\\nabla\\tf(x,\\xi)-\\nabla f(x)\\|^2]\\leq\\sigma^2 \\qquad \\forall x\\in\\mcX\n\\]\nfor some constant $\\sigma \\geq 0$. \n\\item The mapping $c$ is $L_c$-Lipschitz continuous and $L_{\\nabla c}$-smooth in an open neighborhood of $\\mcX$. Additionally, $\\|c(x)\\|\\leq C_c$ for all $x\\in\\mcX$, and there exist constants $\\gamma>0$ and $\\theta\\geq1$ such that\n\\beq \\label{err-cond}\n\\dist\\left(0,\\nabla c(x)c(x)+\\mcN_\\mcX(x)\\right)\\geq\\gamma\\|c(x)\\|^\\theta\\qquad \\forall x\\in\\mcX.\n\\eeq\n\\end{enumerate}\n\\end{assumption}\n\n\\begin{align}\nL:= L_c^2+C_cL_{\\nabla c}.\\label{def-L}\n\\end{align}\n\nIt follows from this and Assumption \\ref{a1} that $\\|c(x)\\|^2/2$ is $L$-smooth on $\\mcX$, and \n\\beq\\label{cf-bnd}\n\\|\\nabla f(x)\\|\\leq L_f,\\quad\\|\\nabla c(x)\\|\\leq L_c\\qquad\\forall x\\in\\mcX.\n\\eeq\n\n\\begin{algorithm}[H]\n\\caption{A stochastic first-order method with a truncated  recursive momentum for problem \\eqref{prob}}\\label{alg1}\n\\begin{algorithmic}[1]\n\\REQUIRE $x_1\\in\\mcX$, $\\{\\alpha_k\\} \\subset (0,1]$,  $\\{\\rho_k\\}, \\{\\eta_k\\} \\subset \\rr_{>0}$, and $L_f$ given in Assumption \\ref{a1}.\n\\STATE Sample $\\xi_1$ and set $g_1=\\Pi_{\\cB(L_f)}(\\nabla\\tf(x_1,\\xi_1))$.\n\\FOR{$k=1,2,\\dots$}\n\\STATE $G_k=g_k+\\rho_k\\nabla c(x_k)c(x_k)$.\n\\STATE $x_{k+1}=\\Pi_\\mcX(x_k-\\eta_kG_k)$.\n\\STATE Sample $\\xi_{k+1}$ and set $g_{k+1}=\\Pi_{\\cB(L_f)}\\big(\\nabla\\tf(x_{k+1},\\xi_{k+1})+(1-\\alpha_k)(g_k-\\nabla\\tf(x_k,\\xi_{k+1}))\\big)$.\n\\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}\n\n\\begin{assumption}\\label{a-storm}\nThe function $\\tf(x,\\xi)$ satisfies the average smoothness condition:\n\\[\n\\bbE[ \\|\\nabla\\tf(u,\\xi)-\\nabla\\tf(v,\\xi)\\|^2]\\leq \\bL_{\\nabla f}^2\\|u-v\\|^2\\qquad \\forall u,v\\in\\mcX.\n\\]\n\\end{assumption}\n\nIt can be observed that Assumption \\ref{a-storm} implies that $\\nabla f$ is $\\bL_{\\nabla f}$-smooth on $X$, that is,\n\\beq \\label{bL-smooth}\n \\|\\nabla f(u)-\\nabla f(v)\\| \\leq \\bL_{\\nabla f} \\|u-v\\| \\qquad \\forall u,v\\in\\mcX.\n\\eeq\n\n\\beq\\label{def-h}\nh(x) := \\frac{1}{2}\\|c(x)\\|^2.\n\\eeq\n\nOne can observe from Assumption \\ref{a1}(iv) that $h$ is $L$-smooth on $\\mcX$, where $L$ is given in  \\eqref{def-L}.\n","units":[{"start":7947,"end":8077,"start_line":212,"end_line":217,"raw_tex":"\\beq\\label{prob}\n\\begin{aligned}\n\\min_{x\\in\\mcX}&\\quad f(x) :=\\bbE[\\tilde f(x,\\xi)]\\\\\n\\mbox{s.t.}&\\quad c(x)=0,\n\\end{aligned}\n\\eeq","sha256_utf8":"af7d8f91179b89c4bd665ec101333cdcde087813ff0282f4708a65857de5f843"},{"start":8078,"end":9312,"start_line":218,"end_line":218,"raw_tex":"where $\\xi$ is a random variable with sample space $\\Xi$, $\\tf(\\cdot,\\xi)$ is continuously differentiable for each $\\xi\\in\\Xi$,  $c:\\rr^n\\to\\rr^m$ is a deterministic smooth mapping, and $\\mcX\\subseteq \\rr^n$ is a simple closed convex set whose projection operator can be evaluated exactly.\\footnote{For simplicity, we focus on problem \\eqref{prob} with equality constraints only. However, our results can be directly extended to problems involving both equality and inequality constraints; see the concluding remarks for further details.} Problem \\eqref{prob} arises in a variety of important areas, including energy systems \\cite{shabazbegian2020stochastic}, healthcare \\cite{shehadeh2022stochastic}, image processing \\cite{luke2020proximal}, machine learning \\cite{cuomo2022scientific,karniadakis2021physics},  network optimization \\cite{bertsekasnetwork}, optimal control \\cite{betts2010practical}, PDE-constrained optimization \\cite{rees2010optimal}, resource allocation \\cite{fontaine2020adaptive}, and transportation \\cite{maggioni2009stochastic}. More applications can be found, for example, in \\cite{bottou2018optimization,combettes2011proximal,jiang2018consensus,kushner2012stochastic,liu2015sparse}, and references therein.\n","sha256_utf8":"be30b6be18299b8c44d3a1d609f01275bcdfe127ea8c66dbda8212ccd5be3006"},{"start":12922,"end":12993,"start_line":234,"end_line":236,"raw_tex":"\\beq\\label{def-Q} \nQ_\\rho(x) := f(x) + \\frac{\\rho}{2} \\|c(x)\\|^2. \n\\eeq","sha256_utf8":"087a5eb1aaaa66d1c101fe40aa7cf3eb7dfc465dd5cf61ed14f24c7406c2605c"},{"start":20095,"end":21404,"start_line":266,"end_line":269,"raw_tex":"The following notation will be used throughout this paper.  Let $\\rr_{>0}$ denote the set of positive real numbers, and  $\\rr^n$ denote the Euclidean space of dimension $n$. The standard inner product and Euclidean norm are denoted by $\\langle\\cdot,\\cdot\\rangle$ and $\\|\\cdot\\|$, respectively.  For any $r>0$, let $\\cB(r)$ represent the Euclidean ball centered at the origin with radius $r$, that is, $\\cB(r)=\\{x:\\|x\\|\\leq r\\}$.  For any $t\\in\\rr$, let $t_+=\\max\\{t,0\\}$ and $\\lceil t \\rceil$ denote the least integer greater than or equal to $t$.\n\nA mapping $\\phi$ is said to be \\emph{$L_{\\phi}$-Lipschitz continuous} on a set $\\Omega$ if $\\|\\phi(x)-\\phi(x')\\| \\leq L_{\\phi} \\|x-x'\\|$ for all $x,x'\\in \\Omega$. Also, it is said to be \\emph{$L_{\\nabla\\phi}$-smooth} on $\\Omega$ if $\\|\\nabla\\phi(x)-\\nabla\\phi(x')\\| \\leq L_{\\nabla\\phi} \\|x-x'\\|$ for all $x,x'\\in \\Omega$, where $\\nabla \\phi$ denotes the transpose of the Jacobian of $\\phi$.  Given a nonempty closed convex set $\\cC$, $\\dist(x,\\cC)$ denotes the Euclidean distance from $x$ to $\\cC$, and $\\Pi_\\cC(x)$ denotes the Euclidean projection of $x$ onto $\\Omega$.  In addition,  the normal cone of $\\cC$ at any $x\\in \\cC$ is denoted by $\\mcN_\\cC(x)$.\nFinally, we use $\\tmcO(\\cdot)$ to denote the asymptotic upper bound that ignores logarithmic factors.\n","sha256_utf8":"d58b2059ef7321bca15a57b2027903b18a528a94d6cebee62e7947c2b1ce223f"},{"start":21489,"end":22424,"start_line":273,"end_line":287,"raw_tex":"\\begin{assumption}\\label{a1}\n\\begin{enumerate}[label=(\\roman*)]\n\\item The optimal value $f^*$ of problem \\eqref{prob}  and $Q_1^*:= \\min_{x\\in X} \\{f(x)+\\|c(x)\\|^2/2\\}$ are finite.\n\\item $f$ is differentiable and $L_f$-Lipschitz continuous in an open neighborhood of $\\mcX$.\n\\item For each $\\xi\\in\\Xi$, $\\tf(\\cdot,\\xi)$ is differentiable on $\\mcX$ and satisfies the following conditions:\n\\[\n\\bbE[\\nabla\\tf(x,\\xi)]=\\nabla f(x),\\quad\\bbE[\\|\\nabla\\tf(x,\\xi)-\\nabla f(x)\\|^2]\\leq\\sigma^2 \\qquad \\forall x\\in\\mcX\n\\]\nfor some constant $\\sigma \\geq 0$. \n\\item The mapping $c$ is $L_c$-Lipschitz continuous and $L_{\\nabla c}$-smooth in an open neighborhood of $\\mcX$. Additionally, $\\|c(x)\\|\\leq C_c$ for all $x\\in\\mcX$, and there exist constants $\\gamma>0$ and $\\theta\\geq1$ such that\n\\beq \\label{err-cond}\n\\dist\\left(0,\\nabla c(x)c(x)+\\mcN_\\mcX(x)\\right)\\geq\\gamma\\|c(x)\\|^\\theta\\qquad \\forall x\\in\\mcX.\n\\eeq\n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"30a6a3eb4c78ce235c3acc54b1483f5c584c5196a11e417f0cf4eda8edc95370"},{"start":22477,"end":22542,"start_line":290,"end_line":292,"raw_tex":"\\begin{align}\nL:= L_c^2+C_cL_{\\nabla c}.\\label{def-L}\n\\end{align}","sha256_utf8":"1bf2799092977746cdcf8c3b350c181760ac209b6f0dcaab20419685c01ca15c"},{"start":22543,"end":22737,"start_line":293,"end_line":296,"raw_tex":"It follows from this and Assumption \\ref{a1} that $\\|c(x)\\|^2/2$ is $L$-smooth on $\\mcX$, and \n\\beq\\label{cf-bnd}\n\\|\\nabla f(x)\\|\\leq L_f,\\quad\\|\\nabla c(x)\\|\\leq L_c\\qquad\\forall x\\in\\mcX.\n\\eeq","sha256_utf8":"cfbf5c3ce0d0cf9af78168b4e7eab3fa3542bb360d59352553494de93928f853"},{"start":26750,"end":27409,"start_line":329,"end_line":340,"raw_tex":"\\begin{algorithm}[H]\n\\caption{A stochastic first-order method with a truncated  recursive momentum for problem \\eqref{prob}}\\label{alg1}\n\\begin{algorithmic}[1]\n\\REQUIRE $x_1\\in\\mcX$, $\\{\\alpha_k\\} \\subset (0,1]$,  $\\{\\rho_k\\}, \\{\\eta_k\\} \\subset \\rr_{>0}$, and $L_f$ given in Assumption \\ref{a1}.\n\\STATE Sample $\\xi_1$ and set $g_1=\\Pi_{\\cB(L_f)}(\\nabla\\tf(x_1,\\xi_1))$.\n\\FOR{$k=1,2,\\dots$}\n\\STATE $G_k=g_k+\\rho_k\\nabla c(x_k)c(x_k)$.\n\\STATE $x_{k+1}=\\Pi_\\mcX(x_k-\\eta_kG_k)$.\n\\STATE Sample $\\xi_{k+1}$ and set $g_{k+1}=\\Pi_{\\cB(L_f)}\\big(\\nabla\\tf(x_{k+1},\\xi_{k+1})+(1-\\alpha_k)(g_k-\\nabla\\tf(x_k,\\xi_{k+1}))\\big)$.\n\\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"a0cac5052dd0f147dfba22ff78da346d0d4237674e46c08c8d3abf8567725ac6"},{"start":31890,"end":32119,"start_line":367,"end_line":372,"raw_tex":"\\begin{assumption}\\label{a-storm}\nThe function $\\tf(x,\\xi)$ satisfies the average smoothness condition:\n\\[\n\\bbE[ \\|\\nabla\\tf(u,\\xi)-\\nabla\\tf(v,\\xi)\\|^2]\\leq \\bL_{\\nabla f}^2\\|u-v\\|^2\\qquad \\forall u,v\\in\\mcX.\n\\]\n\\end{assumption}","sha256_utf8":"b7bfb42161432c2adc1b255d19a7777200da482055a5fc69637df313561fccf7"},{"start":32446,"end":32674,"start_line":375,"end_line":378,"raw_tex":"It can be observed that Assumption \\ref{a-storm} implies that $\\nabla f$ is $\\bL_{\\nabla f}$-smooth on $X$, that is,\n\\beq \\label{bL-smooth}\n \\|\\nabla f(u)-\\nabla f(v)\\| \\leq \\bL_{\\nabla f} \\|u-v\\| \\qquad \\forall u,v\\in\\mcX.\n\\eeq","sha256_utf8":"1ce96b6e2ab83e61b5ee3a147e8b8de9336f468442b39eb61015f819fa483b68"},{"start":50709,"end":50762,"start_line":553,"end_line":555,"raw_tex":"\\beq\\label{def-h}\nh(x) := \\frac{1}{2}\\|c(x)\\|^2.\n\\eeq","sha256_utf8":"a8e2100c73cabd22500a46e665ccf24fe9f8a3cc2b5f2f487eef9ae3febbf9b3"},{"start":50763,"end":50880,"start_line":556,"end_line":556,"raw_tex":"One can observe from Assumption \\ref{a1}(iv) that $h$ is $L$-smooth on $\\mcX$, where $L$ is given in  \\eqref{def-L}.\n","sha256_utf8":"be3046b8f295ed5246de1aff44b25d8a7a6e5957cc35edfda9cc81a27335dadc"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{thm}\\label{thm:storm}\nSuppose that Assumptions \\ref{a1} and \\ref{a-storm} hold, and $\\{x_k\\}$ is generated by Algorithm \\ref{alg1}. Let $L$ be defined in \\eqref{def-L}, $L_f$, $\\bL_{\\nabla f}$, $L_c$, $C_c$, $\\sigma$,  $\\gamma$, $\\theta$ and $Q_1^*$ be given in Assumptions \\ref{a1} and \\ref{a-storm}, $g_1$ be given in Algorithm \\ref{alg1}, and $\\iota_k$  be the random variable uniformly generated from $\\left\\{\\lceil k/2\\rceil+1,\\dots,k\\right\\}$ for $k\\geq 2$, and let $\\rho_k$, $\\eta_k$ and $\\alpha_k$ be chosen as\n\\beq\\label{def-para2}\n\\rho_k=k^\\nu,\\quad\\eta_k=k^{-\\nu}/(4\\log(k+2)), \\quad \\alpha_k=k^{-2\\nu},\\quad\\mbox{where}\\quad \\nu=\\min\\{\\hat\\theta/(\\hat\\theta+2),1/2\\}\n\\eeq\nfor some $\\hat\\theta\\geq1$. Then for all $k\\geq2\\tK_1$, we have\n\\begin{align*}\n&\\bbE\\left[\\dist^2\\left(0,\\nabla f(x_{\\iota_k})+\\rho_{\\iota_k-1}\\nabla c(x_{\\iota_k})c(x_{\\iota_k})+\\mcN_\\mcX(x_{\\iota_k})\\right)\\right]\\\\\n&\\leq\\frac{102\\log(k+1)}{(k-1)^{1-\\nu}}\\Bigg(Q_1(x_1)-Q_1^*+\\frac{\\|g_1-\\nabla f(x_1)\\|^2}{2\\log 3}+\\frac{1}{2}C_1\\max\\{1,k^{\\nu-\\frac{2\\nu}{\\theta}}\\}(1+\\log k)\\nn \\\\ \n&\\quad+\\frac{1}{2}C_c^2\\big(\\tK_1^{\\nu}-1\\big)\n+\\frac{3\\sqrt{2}\\sigma^2(1+\\log k)}{2\\log3}+\\frac{1}{16}(1+\\log\\tK_1)\\big(\\bL_{\\nabla f}+\\tK_1^{\\frac{1}{2}}L+6\\tK_1^{\\frac{1}{2}}\\bL_{\\nabla f}^2\\big)(L_f^2+C_c^2L_c^2\\tK_1)\\Bigg),\\\\\n&\\|c(x_{\\iota_k})\\|^2\\leq2C_1 (k/2)^{-2\\nu/\\theta},\n\\end{align*}\nwhere\n\\begin{align}\n&\\tK_1=\\left\\lceil\\max\\left\\{(2\\bL_{\\nabla f})^{\\frac{1}{\\nu}},e^{6\\times2^{\\nu/2}\\bL_{\\nabla f}^2},e^{2L},e^{2\\theta},\\left(e^{-1}\\gamma^{-2}2^{6-\\theta}\\log(e^{2\\theta}+2)\\right)^{2\\theta}\\right\\}\\right\\rceil,\\label{def-tK2}\\\\\n&C_1=\\max\\left\\{1,\\tK_1^{2\\nu/\\theta}C_c^2/2,2^{3-\\theta}L_f^2\\gamma^{-2}\\right\\}.\\label{def-nuC2}\n\\end{align}\n\n\\end{thm}","units":[{"start":33416,"end":35160,"start_line":388,"end_line":407,"raw_tex":"\\begin{thm}\\label{thm:storm}\nSuppose that Assumptions \\ref{a1} and \\ref{a-storm} hold, and $\\{x_k\\}$ is generated by Algorithm \\ref{alg1}. Let $L$ be defined in \\eqref{def-L}, $L_f$, $\\bL_{\\nabla f}$, $L_c$, $C_c$, $\\sigma$,  $\\gamma$, $\\theta$ and $Q_1^*$ be given in Assumptions \\ref{a1} and \\ref{a-storm}, $g_1$ be given in Algorithm \\ref{alg1}, and $\\iota_k$  be the random variable uniformly generated from $\\left\\{\\lceil k/2\\rceil+1,\\dots,k\\right\\}$ for $k\\geq 2$, and let $\\rho_k$, $\\eta_k$ and $\\alpha_k$ be chosen as\n\\beq\\label{def-para2}\n\\rho_k=k^\\nu,\\quad\\eta_k=k^{-\\nu}/(4\\log(k+2)), \\quad \\alpha_k=k^{-2\\nu},\\quad\\mbox{where}\\quad \\nu=\\min\\{\\hat\\theta/(\\hat\\theta+2),1/2\\}\n\\eeq\nfor some $\\hat\\theta\\geq1$. Then for all $k\\geq2\\tK_1$, we have\n\\begin{align*}\n&\\bbE\\left[\\dist^2\\left(0,\\nabla f(x_{\\iota_k})+\\rho_{\\iota_k-1}\\nabla c(x_{\\iota_k})c(x_{\\iota_k})+\\mcN_\\mcX(x_{\\iota_k})\\right)\\right]\\\\\n&\\leq\\frac{102\\log(k+1)}{(k-1)^{1-\\nu}}\\Bigg(Q_1(x_1)-Q_1^*+\\frac{\\|g_1-\\nabla f(x_1)\\|^2}{2\\log 3}+\\frac{1}{2}C_1\\max\\{1,k^{\\nu-\\frac{2\\nu}{\\theta}}\\}(1+\\log k)\\nn \\\\ \n&\\quad+\\frac{1}{2}C_c^2\\big(\\tK_1^{\\nu}-1\\big)\n+\\frac{3\\sqrt{2}\\sigma^2(1+\\log k)}{2\\log3}+\\frac{1}{16}(1+\\log\\tK_1)\\big(\\bL_{\\nabla f}+\\tK_1^{\\frac{1}{2}}L+6\\tK_1^{\\frac{1}{2}}\\bL_{\\nabla f}^2\\big)(L_f^2+C_c^2L_c^2\\tK_1)\\Bigg),\\\\\n&\\|c(x_{\\iota_k})\\|^2\\leq2C_1 (k/2)^{-2\\nu/\\theta},\n\\end{align*}\nwhere\n\\begin{align}\n&\\tK_1=\\left\\lceil\\max\\left\\{(2\\bL_{\\nabla f})^{\\frac{1}{\\nu}},e^{6\\times2^{\\nu/2}\\bL_{\\nabla f}^2},e^{2L},e^{2\\theta},\\left(e^{-1}\\gamma^{-2}2^{6-\\theta}\\log(e^{2\\theta}+2)\\right)^{2\\theta}\\right\\}\\right\\rceil,\\label{def-tK2}\\\\\n&C_1=\\max\\left\\{1,\\tK_1^{2\\nu/\\theta}C_c^2/2,2^{3-\\theta}L_f^2\\gamma^{-2}\\right\\}.\\label{def-nuC2}\n\\end{align}\n\n\\end{thm}","sha256_utf8":"b385a2df7be2e986fb84da87541de704d4f99c8fb71a2a4173fd7e3733d0093b"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\\label{l-rec} \nSuppose that Assumption \\ref{a1} holds, and $x_{k+1}$ is generated by Algorithm \\ref{alg1} for some $k\\geq1$ with $\\rho_k\\eta_k\\leq(\\sqrt{5}-1)/(2L)$.  Then we have\n\\[\nh(x_{k+1})+2^{\\theta-2}\\gamma^2\\rho_k\\eta_k[h(x_{k+1})]^{\\theta}\\leq h(x_k)+L_f^2\\rho_k^{-1}\\eta_k/2,\n\\]\nwhere $\\rho_k$ and $\\eta_k$ are given in Algorithm \\ref{alg1}, $L_f$, $\\gamma$ and $\\theta$ are given in Assumption \\ref{a1},  and $L$ and $h$ are defined in \\eqref{def-L} and \\eqref{def-h}, respectively.\n\\end{lemma}\n\n\\begin{proof}\nLet $G_k$ be given in Algorithm \\ref{alg1}. For convenience, we define\n\\beq\\label{l3-e1}\n\\widetilde G_k=\\rho_k^{-1}G_k,\\quad\\tilde\\eta_k=\\rho_k\\eta_k.\n\\eeq\n It then follows from these, \\eqref{def-h}, and the expression of $x_{k+1}$ in Algorithm \\ref{alg1} that\n\\beq\\label{l3-x}\nx_{k+1}=\\Pi_\\mcX(x_k-\\eta_kG_k)=\\Pi_\\mcX(x_k-\\tilde\\eta_k\\widetilde G_k),\n\\eeq\nwhich implies that\n\\beq \\label{subprob-opt1}\n0\\in x_{k+1}-x_k+\\tilde\\eta_k\\widetilde G_k+\\mcN_\\mcX(x_{k+1})\\quad\\Rightarrow\\quad \\nabla h(x_{k+1})+\\tilde\\eta_k^{-1}(x_k-x_{k+1})-\\widetilde G_k\\in\\nabla h(x_{k+1})+\\mcN_\\mcX(x_{k+1}).\n\\eeq\nUsing this, \\eqref{def-h} and Assumption \\ref{a1}(iv), we have\n\\begin{align}\n2^\\theta\\gamma^2[h(x_{k+1})]^\\theta&\\ =\\gamma^2\\|c(x_{k+1})\\|^{2\\theta}\\leq\\dist^2\\left(0,\\nabla c(x_{k+1})c(x_{k+1})+\\mcN_\\mcX(x_{k+1})\\right) \\nn \\\\\n&\\overset{\\eqref{def-h}}{=}\\dist^2\\left(0,\\nabla h(x_{k+1})+\\mcN_\\mcX(x_{k+1})\\right)\\overset{\\eqref{subprob-opt1}}\\leq\\|\\nabla h(x_{k+1})+\\tilde\\eta_k^{-1}(x_k-x_{k+1})-\\widetilde G_k\\|^2\\nn\\\\\n&\\ \\leq2\\|\\tilde\\eta_k^{-1}(x_k-x_{k+1})+\\nabla h(x_k)-\\widetilde G_k\\|^2+2\\|\\nabla h(x_{k+1})-\\nabla h(x_k)\\|^2\\nn\\\\\n&\\ =2\\tilde\\eta_k^{-2}\\|x_{k+1}-x_k\\|^2+4\\tilde\\eta_k^{-1}\\langle\\widetilde G_k-\\nabla h(x_k), x_{k+1}-x_k\\rangle+2\\|\\widetilde G_k-\\nabla h(x_k)\\|^2\\nn\\\\\n&\\ \\quad +2\\|\\nabla h(x_{k+1})-\\nabla h(x_k)\\|^2\\nn\\\\\n&\\ \\leq2(\\tilde\\eta_k^{-2}+L^2)\\|x_{k+1}-x_k\\|^2+4\\tilde\\eta_k^{-1}\\langle\\widetilde G_k-\\nabla h(x_k), x_{k+1}-x_k\\rangle+2\\|\\widetilde G_k-\\nabla h(x_k)\\|^2,\\label{l3-e2}\n\\end{align}\nwhere the first inequality follows from Assumption \\ref{a1}(iv), the second inequality is due to the convexity of $\\|\\cdot\\|^2$, and the last inequality follows from the $L$-smoothness of $h$. In addition,  by \\eqref{l3-x} and $x_k\\in\\mcX$, one has\n\\[\n\\langle x_{k+1}-x_k+\\tilde\\eta_k\\widetilde G_k,x_k-x_{k+1}\\rangle\\geq0\\quad\\Rightarrow\\quad \\langle \\widetilde G_k,x_{k+1}-x_k\\rangle\\leq-\\tilde\\eta_k^{-1}\\|x_{k+1}-x_k\\|^2.\n\\]\nThis together with the $L$-smoothness of $h$ yields\n\\begin{align*}\nh(x_{k+1})& \\leq h(x_k)+\\langle\\nabla h(x_k),x_{k+1}-x_k\\rangle +\\frac{L}{2}\\|x_{k+1}-x_k\\|^2\\nn\\\\\n&=h(x_k)+\\langle\\widetilde G_k,x_{k+1}-x_k\\rangle+\\langle\\nabla h(x_k)-\\widetilde G_k,x_{k+1}-x_k\\rangle +\\frac{L}{2}\\|x_{k+1}-x_k\\|^2\\nn\\\\\n& \\leq h(x_k)-\\tilde\\eta_k^{-1}\\|x_{k+1}-x_k\\|^2+\\langle\\nabla h(x_k)-\\widetilde G_k,x_{k+1}-x_k\\rangle +\\frac{L}{2}\\|x_{k+1}-x_k\\|^2.\n\\end{align*}\nUsing this and \\eqref{l3-e2}, we obtain that\n\\beq\\label{l3-e3}\nh(x_{k+1})+2^{\\theta-2}\\gamma^2\\tilde\\eta_k[h(x_{k+1})]^{\\theta}\\leq h(x_k)+\\frac{1}{2}(L^2\\tilde\\eta_k-\\tilde\\eta_k^{-1}+L)\\|x_{k+1}-x_k\\|^2+\\frac{\\tilde\\eta_k}{2}\\|\\widetilde G_k-\\nabla h(x_k)\\|^2.\n\\eeq\nObserve from \\eqref{l3-e1} and $\\rho_k\\eta_k\\leq(\\sqrt{5}-1)/(2L)$ that $L\\tilde\\eta_k=L\\rho_k\\eta_k\\leq(\\sqrt{5}-1)/2$, which implies that\n\\beq\\label{l3-e4}\nL^2\\tilde\\eta_k-\\tilde\\eta_k^{-1}+L=\\tilde\\eta_k^{-1}( L^2\\tilde\\eta_k^2+L\\tilde\\eta_k-1)\\leq0.\n\\eeq\nNotice from the expression of $g_k$ in Algorithm \\ref{alg1} that $g_k\\in\\cB(L_f)$ and hence $\\|g_k\\|\\leq L_f$. \nAlso, observe from Algorithm \\ref{alg1} and \\eqref{def-h} that $G_k=g_k+\\rho_k\\nabla h(x_k)$. Using these and \\eqref{l3-e1}, we have\n\\[\n\\|\\widetilde G_k-\\nabla h(x_k)\\|= \\|\\rho_k^{-1}G_k-\\nabla h(x_k)\\| =\\|\\rho_k^{-1}(g_k+\\rho_k\\nabla h(x_k))-\\nabla h(x_k)\\|= \\rho_k^{-1}\\|g_k\\|\\leq\\rho_k^{-1}L_f.\n\\]\nIt then follows from this, \\eqref{l3-e3} and \\eqref{l3-e4} that \n\\[\nh(x_{k+1})+2^{\\theta-2}\\gamma^2\\tilde\\eta_k[h(x_{k+1})]^\\theta\\leq h(x_k)+L_f^2\\tilde\\eta_k/(2\\rho_k^2).\n\\]\nThis and the definition of $\\tilde\\eta_k$ in \\eqref{l3-e1} imply that the conclusion of this lemma holds.\n\\end{proof}\n\n\\begin{lemma}\\label{l-cnst2}\nLet $\\nu$, $\\tK_1$, and $C_1$ be given in \\eqref{def-para2}, \\eqref{def-tK2} and \\eqref{def-nuC2}, respectively. Suppose that Assumption \\ref{a1} holds, and $\\{x_k\\}$ is generated by Algorithm \\ref{alg1} with $\\{\\rho_k\\}$, $\\{\\eta_k\\}$ and $\\{\\alpha_k\\}$ given in \\eqref{def-para2}. Then  we have $\\|c(x_k)\\|^2\\leq 2C_1k^{-2\\nu/\\theta}$ for all $k\\geq\\tK_1$.\n\\end{lemma}\n\n\\begin{proof}\nLet $h$ be defined in \\eqref{def-h}. To prove this lemma, it is equivalent to show that $h(x_k)\\leq C_1k^{-2\\nu/\\theta}$ for all $k\\geq\\tK_1$.  We now prove this by induction. Indeed, notice from Algorithm \\ref{alg1} that $x_{\\tK_1}\\in X$. It then follows from  \\eqref{def-nuC2}, \\eqref{def-h}, and Assumption \\ref{a1}(iv) that\n\\[\nh(x_{\\tK_1})\\overset{\\eqref{def-h}}{=}\\frac{1}{2}\\|c(x_{\\tK_1})\\|^2\\leq\\frac{1}{2}C_c^2\\overset{\\eqref{def-nuC2}}{\\leq} C_1\\tK_1^{-2\\nu/\\theta}.\n\\]\nHence,  the conclusion holds for $k=\\tK_1$. Now, suppose for induction that $h(x_k)\\leq C_1k^{-2\\nu/\\theta}$ holds for some $k\\geq\\tK_1$. Recall that $\\rho_k$, $\\eta_k$ and $\\tK_1$ are given in \\eqref{def-para2} and \\eqref{def-tK2}.  In view of these, one can observe that \n\\[\n\\rho_k\\eta_k\\overset{\\eqref{def-para2} }=\\frac{1}{4\\log(k+2)}\\leq\\frac{1}{4\\log(\\tK_1+2)}\\overset{\\eqref{def-tK2}}\\leq \\frac{1}{8L}\\leq\\frac{\\sqrt{5}-1}{2L},\n\\]\nand hence Lemma \\ref{l-rec} holds for such $k$.  Using Lemma \\ref{l-rec} with the choice of $\\rho_k$ and $\\eta_k$ given in \\eqref{def-para2}, we obtain that\n\\beq\\label{rec2}\nh(x_{k+1})+2^{\\theta-4}\\gamma^2[h(x_{k+1})]^{\\theta}/\\log(k+2)\\leq h(x_k)+L_f^2k^{-2\\nu}/(8\\log(k+2)).\n\\eeq\nFurther, let\n\\beq\\label{def-phi2}\n\\phi(t)=t+2^{\\theta-4}\\gamma^2t^\\theta/\\log(k+2).\n\\eeq\nNotice from \\eqref{def-para2} and \\eqref{def-nuC2} that $\\nu=\\min\\{\\hat\\theta/(\\hat\\theta+2),1/2\\}$ for some $\\hat\\theta\\geq1$ and $C_1\\geq1$. Using these and \\eqref{def-phi2},  we have\n\\begin{align}\n&\\ \\phi(C_1(k+1)^{-2\\nu/\\theta})-C_1k^{-2\\nu/\\theta}-L_f^2k^{-2\\nu}/(8\\log(k+2))\\nn\\\\\n&\\overset{\\eqref{def-phi2}}{=}C_1^\\theta 2^{\\theta-4}\\gamma^2(k+1)^{-2\\nu}/\\log(k+2)+C_1(k+1)^{-2\\nu/\\theta}-C_1k^{-2\\nu/\\theta}-L_f^2k^{-2\\nu}/(8\\log(k+2))\\nn\\\\\n&\\ \\geq C_1^\\theta 2^{\\theta-4}\\gamma^2(k+1)^{-2\\nu}/\\log(k+2)-2\\nu C_1 k^{-2\\nu/\\theta-1}/\\theta-L_f^2k^{-2\\nu}/(8\\log(k+2))\\nn\\\\\n&\\ = \\frac{k^{-2\\nu}}{\\log(k+2)}\\left(C_1^\\theta 2^{\\theta-4}\\gamma^2\\left(\\frac{k}{k+1}\\right)^{2\\nu}- 2\\nu C_1 k^{2\\nu(1-\\theta^{-1})-1}\\log(k+2)/\\theta-L_f^2/8\\right)\\nn\\\\\n&\\ \\geq \\frac{k^{-2\\nu}}{\\log(k+2)}\\left(C_1^\\theta 2^{\\theta-5}\\gamma^2-C_1 k^{-\\frac{1}{\\theta}}\\log(k+2)-L_f^2/8\\right)\\nn\\\\\n&\\ \\geq \\frac{k^{-2\\nu}}{\\log(k+2)}\\left(C_12^{\\theta-5}\\gamma^2-C_1 k^{-\\frac{1}{\\theta}}\\log(k+2)-L_f^2/8\\right),\\label{l4-e1g}\n\\end{align}\nwhere the first inequality follows from $(k+1)^{-2\\nu/\\theta}-k^{-2\\nu/\\theta}\\geq-2\\nu k^{-2\\nu/\\theta-1}/\\theta$ thanks to the convexity of $t^{-2\\nu/\\theta}$, the second inequality is due to $\\theta\\geq1$, $\\nu\\leq1/2$ and  $k/(k+1)\\geq1/2$, and the last inequality follows from $\\theta\\geq1$ and $C_1\\geq1$. In addition, one can verify that $t^{-\\frac{1}{2\\theta}}\\log(t+2)$ is decreasing on $[e^{2\\theta},\\infty)$. Using this, \\eqref{def-tK2}, and $k\\geq\\tK_1\\geq e^{2\\theta}$, we obtain that\n\\[\nk^{-\\frac{1}{2\\theta}}\\log(k+2)\\leq\\log(e^{2\\theta}+2)/e,\\quad k^{-\\frac{1}{2\\theta}}\\leq\\tK_1^{-\\frac{1}{2\\theta}}\\leq e\\gamma^2/(2^{6-\\theta}\\log(e^{2\\theta}+2)).\n\\]\nMultiplying both sides of these inequalities yields $k^{-1/\\theta}\\log(k+2)\\leq2^{\\theta-6}\\gamma^2$, which together with \\eqref{def-nuC2}  implies that\n\\begin{align*}\nC_12^{\\theta-5}\\gamma^2-C_1 k^{-\\frac{1}{\\theta}}\\log(k+2)-L_f^2/8\\geq C_12^{\\theta-6}\\gamma^2-L_f^2/8\\overset{\\eqref{def-nuC2}}{\\geq}0.\n\\end{align*}\nUsing this,  \\eqref{rec2}, \\eqref{def-phi2},  \\eqref{l4-e1g}, and the induction hypothesis that $h(x_k)\\leq C_1k^{-2\\nu/\\theta}$, we obtain that\n\\[\n\\phi(C_1(k+1)^{-2\\nu/\\theta})\\geq C_1k^{-2\\nu/\\theta}+L_f^2k^{-2\\nu}/(8\\log(k+2))\\geq h(x_k)+L_f^2k^{-2\\nu}/(8\\log(k+2))\\overset{\\eqref{rec2}\\eqref{def-phi2}}{\\geq}\\phi(h(x_{k+1})).\n\\]\nIt then follows from this inequality and the strict monotonicity of $\\phi$ on $[0,\\infty)$ that $h(x_{k+1})\\leq C_1(k+1)^{-2\\nu/\\theta}$. Hence, the induction is completed and the conclusion of this lemma holds. \n\\end{proof}\n\n\\begin{lemma}\\label{l-exp}\nSuppose that Assumptions \\ref{a1} and \\ref{a-storm} hold, and $\\{g_k\\}$ and $\\{x_k\\}$ are generated by Algorithm \\ref{alg1}. Then for all $k\\geq1$, we have\n\\[\n\\bbE\\left[\\|g_{k+1}-\\nabla f(x_{k+1})\\|^2\\right]\\leq(1-\\alpha_k)^2\\bbE\\left[\\|g_k-\\nabla f(x_k)\\|^2\\right]+6\\bL_{\\nabla f}^2\\bbE\\left[\\|x_{k+1}-x_k\\|^2\\right]+3\\sigma^2\\alpha_k^2,\n\\]\nwhere $\\{\\alpha_k\\}$ is given in Algorithm \\ref{alg1}, and $\\sigma$ and $\\bL_{\\nabla f}$ are given in Assumptions \\ref{a1} and \\ref{a-storm}, respectively. \n\\end{lemma}\n\n\\begin{proof}\nNotice from \\eqref{cf-bnd} that $\\nabla f(x_{k+1})\\in\\cB(L_f)$ and hence $\\nabla f(x_{k+1})=\\Pi_{\\cB(L_f)}(\\nabla f(x_{k+1}))$. By this, the expression of $g_{k+1}$, and the nonexpansiveness of the projection operator $\\Pi_{\\cB(L_f)}$, one has\n\\begin{align}\n&\\|g_{k+1}-\\nabla f(x_{k+1})\\|^2=\\|\\Pi_{\\cB(L_f)}(\\nabla \\tf(x_{k+1},\\xi_{k+1})+(1-\\alpha_k)(g_k-\\nabla \\tf(x_k,\\xi_{k+1})))-\\Pi_{\\cB(L_f)}(\\nabla f(x_{k+1}))\\|^2 \\nn \\\\\n&\\leq\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})+(1-\\alpha_k)(g_k-\\nabla \\tf(x_k,\\xi_{k+1}))-\\nabla f(x_{k+1})\\|^2 \\nn \\\\\n&=\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})+(1-\\alpha_k)(g_k-\\nabla f(x_k)+\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1}))\\|^2 \\nn \\\\\n&=\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})+(1-\\alpha_k)(\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1}))\\|^2+(1-\\alpha_k)^2\\|g_k-\\nabla f(x_k)\\|^2\\nn \\\\\n&\\quad+2(1-\\alpha_k)\\langle g_k-\\nabla f(x_k),\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})\\rangle\\nn \\\\\n&\\quad+2(1-\\alpha_k)^2\\langle g_k-\\nabla f(x_k),\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1})\\rangle. \\label{g-bnd}\n\\end{align}\nLet $\\Xi_k=\\{\\xi_1,\\ldots, \\xi_k\\}$ denote the collection of samples drawn up to iteration $k-1$ in Algorithm \n\\ref{alg1}.  It then follows from Assumption \\ref{a1}(iii) that \n\\[\n\\bbE[\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})|\\Xi_k]=0, \\quad \\bbE[\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1})|\\Xi_k]=0,\n\\]\nwhich imply that\n\\begin{align*}\n&\\bbE[\\langle g_k-\\nabla f(x_k),\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})\\rangle|\\Xi_k]=\\langle g_k-\\nabla f(x_k),\\bbE[\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})|\\Xi_k]\\rangle=0, \\\\\n&\\bbE[\\langle g_k-\\nabla f(x_k),\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1})\\rangle|\\Xi_k]=\\langle g_k-\\nabla f(x_k),\\bbE[\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1})|\\Xi_k]\\rangle=0.\n\\end{align*}\nUsing these and taking a conditional expectation on both sides of \\eqref{g-bnd}, we have\n\\begin{align}\n\\bbE[\\|g_{k+1}-\\nabla f(x_{k+1})\\|^2|\\Xi_k]\\leq&\\bbE[\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})+(1-\\alpha_k)(\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1}))\\|^2|\\Xi_k]\\nn\\\\\n&+(1-\\alpha_k)^2\\|g_k-\\nabla f(x_k)\\|^2.\\label{l1-e1}\n\\end{align}\nIn addition, it follows from Assumption \\ref{a1} that\n\\begin{align*}\n&\\bbE[\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})+(1-\\alpha_k)(\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1}))\\|^2|\\Xi_k]\\\\\n&=\\bbE[\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla \\tf(x_k,\\xi_{k+1})+\\nabla f(x_k)-\\nabla f(x_{k+1})-\\alpha_k(\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1}))\\|^2|\\Xi_k]\\\\\n&\\leq3\\bbE[\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla \\tf(x_k,\\xi_{k+1})\\|^2|\\Xi_k]+3\\|\\nabla f(x_{k+1})-\\nabla f(x_k)\\|^2\\\\\n&\\quad+3\\alpha_k^2\\bbE[\\|\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1})\\|^2|\\Xi_k]\\leq6\\bL_{\\nabla f}^2\\|x_{k+1}-x_k\\|^2+3\\sigma^2\\alpha_k^2,\n\\end{align*}\nwhere the first inequality follows from the convexity of $\\|\\cdot\\|^2$, and the last inequality is due to \\eqref{bL-smooth} and Assumptions \\ref{a1}(iii) and \\ref{a-storm}. By this and \\eqref{l1-e1}, one has\n\\[\n\\bbE[\\|g_{k+1}-\\nabla f(x_{k+1})\\|^2|\\Xi_k]\\leq (1-\\alpha_k)^2\\|g_k-\\nabla f(x_k)\\|^2+6\\bL_{\\nabla f}^2\\|x_{k+1}-x_k\\|^2+3\\sigma^2\\alpha_k^2.\n\\]\nThe conclusion of this lemma follows from taking expectation on both sides of this inequality.\n\\end{proof}\n\n\\begin{lemma}\\label{l-pfunc}\nSuppose that Assumptions \\ref{a1} and \\ref{a-storm} hold. Let $\\{g_k\\}$ and $\\{x_k\\}$ be generated by Algorithm \\ref{alg1}, and  let \n\\beq\\label{def-zeta}\n\\zeta_k=k^{\\nu}/(2\\log 3),\n\\eeq\nwhere $\\nu$ is defined in \\eqref{def-para2}. Then for all $k\\geq1$, we have\n\\begin{align}\n&\\bbE\\left[Q_{\\rho_k}(x_k)+\\zeta_k\\|g_k-\\nabla f(x_k)\\|^2\\right] \\leq Q_{\\rho_1}(x_1)+\\zeta_1\\|g_1-\\nabla f(x_1)\\|^2+\\frac{1}{2}\\sum_{i=1}^{k-1}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right] \\nn \\\\\n&+\\frac{1}{2}\\sum_{i=1}^{k-1}\\left(\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+3\\sigma^2\\sum_{i=1}^{k-1}\\zeta_{i+1}\\alpha_i^2, \\label{penalty-bnd}\n\\end{align}\nwhere $\\{\\alpha_k\\}$, $\\{\\rho_k\\}$ and $\\{\\eta_k\\}$ are given in Algorithm \\ref{alg1}, $Q_\\rho$  and $L$ are respectively defined in \\eqref{def-Q} and \\eqref{def-L},  and $\\sigma$ and $\\bL_{\\nabla f}$  are given in Assumptions \\ref{a1} and \\ref{a-storm}, respectively.\n\\end{lemma}\n\n\\begin{proof}\nObserve from Assumptions \\ref{a1} and \\ref{a-storm} and the definition of $Q_\\rho$ in \\eqref{def-Q} that $Q_{\\rho_k}$ is $(\\bL_{\\nabla f}+\\rho_kL)$-smooth on $\\mcX$, where $L$ is defined in \\eqref{def-L}. Notice from Algorithm \\ref{alg1} that $x_k\\in\\mcX$ and $x_{k+1}= \\Pi_X(x_k-\\eta_k G_k)$, which imply that\n\\beq \\label{subprob-opt2}\n\\langle x_{k+1}-x_k+\\eta_k G_k,x_k-x_{k+1}\\rangle\\geq0\\quad\\Rightarrow\\quad \\langle G_k,x_{k+1}-x_k\\rangle\\leq-\\eta_k^{-1}\\|x_{k+1}-x_k\\|^2.\n\\eeq\nAlso, notice from Algorithm \\ref{alg1} and \\eqref{def-Q} that \n\\[\nG_k=g_k+\\rho_k\\nabla c(x_k)c(x_k), \\quad \\nabla Q_{\\rho_k}(x_k)=\\nabla f(x_k)+\\rho_k\\nabla c(x_k)c(x_k),\n\\] \nand hence $\\nabla Q_{\\rho_k}(x_k)-G_k=g_k-\\nabla f(x_k)$.  In addition, by Young's inequality,  one has \n\\beq \\label{young-ineq}\n\\langle\\nabla Q_{\\rho_k}(x_k)-G_k,x_{k+1}-x_k\\rangle \\leq \\frac{1}{2\\eta_k}\\|x_{k+1}-x_k\\|^2+\\frac{\\eta_k}{2}\\|\\nabla Q_{\\rho_k}(x_k)-G_k\\|^2\n\\eeq\nUsing the last two relations,  \\eqref{subprob-opt2}, and the $(\\bL_{\\nabla f}+\\rho_kL)$-smoothness of $Q_{\\rho_k}$, we obtain that\n\\begin{align*}\nQ_{\\rho_k}(x_{k+1})&\\ \\leq  Q_{\\rho_k}(x_k)+\\langle\\nabla Q_{\\rho_k}(x_k),x_{k+1}-x_k\\rangle+\\frac{1}{2}(\\bL_{\\nabla f}+\\rho_kL)\\|x_{k+1}-x_k\\|^2\\\\\n&\\ = Q_{\\rho_k}(x_k)+\\langle G_k,x_{k+1}-x_k\\rangle+\\langle\\nabla Q_{\\rho_k}(x_k)-G_k,x_{k+1}-x_k\\rangle+\\frac{1}{2}(\\bL_{\\nabla f}+\\rho_kL)\\|x_{k+1}-x_k\\|^2\\\\\n& \\overset{\\eqref{young-ineq}}{\\leq} Q_{\\rho_k}(x_k)+\\langle G_k,x_{k+1}-x_k\\rangle+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL+\\eta_k^{-1}\\right)\\|x_{k+1}-x_k\\|^2+\\frac{\\eta_k}{2}\\|\\nabla Q_{\\rho_k}(x_k)-G_k\\|^2\\\\\n&\\ \\leq Q_{\\rho_k}(x_k)+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL-\\eta_k^{-1}\\right)\\|x_{k+1}-x_k\\|^2+\\frac{\\eta_k}{2}\\|g_k-\\nabla f(x_k)\\|^2,\n\\end{align*}\nwhere the first inequality is due to the $(\\bL_{\\nabla f}+\\rho_kL)$-smoothness of $Q_{\\rho_k}$, and the last inequality follows from \\eqref{subprob-opt2} and the relation $\\nabla Q_{\\rho_k}(x_k)-G_k=g_k-\\nabla f(x_k)$. By this and \\eqref{def-Q}, we further have\n\\begin{align}\n&Q_{\\rho_{k+1}}(x_{k+1})\\leq Q_{\\rho_k}(x_k)+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL-\\eta_k^{-1}\\right)\\|x_{k+1}-x_k\\|^2+\\frac{\\eta_k}{2}\\|g_k-\\nabla f(x_k)\\|^2\\nn\\\\\n&\\qquad\\qquad\\qquad\\,\\,+Q_{\\rho_{k+1}}(x_{k+1})-Q_{\\rho_k}(x_{k+1})\\nn\\\\\n&\\overset{\\eqref{def-Q}}{=} Q_{\\rho_k}(x_k)+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL-\\eta_k^{-1}\\right)\\|x_{k+1}-x_k\\|^2+\\frac{\\eta_k}{2}\\|g_k-\\nabla f(x_k)\\|^2+\\frac{1}{2}(\\rho_{k+1}-\\rho_k)\\|c(x_{k+1})\\|^2.\\label{l5-e1}\n\\end{align}\n\nBy the definitions of $\\eta_k$, $\\alpha_k$, $\\nu$ and $\\zeta_k$ in \\eqref{def-para2}, \\eqref{def-nuC2} and \\eqref{def-zeta}, one has\n\\begin{align*}\n&\\ \\zeta_k-\\zeta_{k+1}(1-\\alpha_k)^2-\\eta_k=\\zeta_{k+1}\\alpha_k(2-\\alpha_k)-\\zeta_{k+1}+\\zeta_k-\\eta_k\\geq \\zeta_{k+1}\\alpha_k-\\zeta_{k+1}+\\zeta_k-\\eta_k\\\\\n&\\overset{\\eqref{def-para2}\\eqref{def-zeta}}\\geq(k+1)^\\nu k^{-2\\nu}/(2\\log 3)-(k+1)^\\nu/(2\\log 3)+k^\\nu/(2\\log 3)-k^{-\\nu}/(4\\log(k+2))\\\\\n&\\ \\ \\ \\geq k^{-\\nu}/(2\\log 3)-(k+1)^\\nu/(2\\log 3)+k^\\nu/(2\\log 3)-k^{-\\nu}/(4\\log(k+2))\\\\\n&\\ \\ \\ \\geq k^{-\\nu}/(2\\log 3)-\\nu k^{\\nu-1}/(2\\log 3)-k^{-\\nu}/(4\\log(k+2))\\\\\n&\\ \\ \\ = \\frac{k^{-\\nu}}{4\\log 3}\\left(2-2\\nu k^{2\\nu-1}-\\log 3/\\log(k+2)\\right) \\geq\\frac{k^{-\\nu}}{4\\log 3}\\left(2-1-\\log 3/\\log(k+2)\\right)\\geq0,\n\\end{align*}\nwhere the first inequality is due to  $0<\\alpha_k\\leq1$, the third inequality follows from $(k+1)^\\nu>k^\\nu$, the fourth inequality is due to $(k+1)^\\nu-k^\\nu\\leq \\nu k^{\\nu-1}$ thanks to the concavity of $t^\\nu$, the fifth inequality follows from $\\nu\\leq1/2$, and the last inequality is due to $k\\geq1$. Hence, we obtain\n\\[\n\\zeta_{k+1}(1-\\alpha_k)^2+\\eta_k\\leq\\zeta_k\\qquad \\forall k\\geq1.\n\\]\nUsing this, taking expectation on both sides of \\eqref{l5-e1}, and summing the resulting inequality with the inequality in Lemma \\ref{l-exp}, we obtain that\n\\begin{align}\n&\\bbE\\left[Q_{\\rho_{k+1}}(x_{k+1})+\\zeta_{k+1}\\|g_{k+1}-\\nabla f(x_{k+1})\\|^2\\right]\\nn\\\\\n&\\leq \\bbE\\left[Q_{\\rho_k}(x_k)+\\left(\\zeta_{k+1}(1-\\alpha_k)^2+\\eta_k\\right)\\|g_k-\\nabla f(x_k)\\|^2\\right]+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL-\\eta_k^{-1}+12\\zeta_{k+1}\\bL_{\\nabla f}^2\\right) \\nn \\\\ \n&\\quad\\times \\bbE\\left[\\|x_{k+1}-x_k\\|^2\\right] -\\frac{\\eta_k}{2}\\bbE\\left[\\|g_k-\\nabla f(x_k)\\|^2\\right]+\\frac{1}{2}(\\rho_{k+1}-\\rho_k)\\bbE\\left[\\|c(x_{k+1})\\|^2\\right]+3\\sigma^2\\zeta_{k+1}\\alpha_k^2\\nn\\\\\n&\\leq \\bbE\\left[Q_{\\rho_k}(x_k)+\\zeta_k\\|g_k-\\nabla f(x_k)\\|^2\\right]+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL-\\eta_k^{-1}+12\\zeta_{k+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{k+1}-x_k\\|^2\\right]\\nn\\\\\n&\\quad-\\frac{\\eta_k}{2}\\bbE\\left[\\|g_k-\\nabla f(x_k)\\|^2\\right]+\\frac{1}{2}(\\rho_{k+1}-\\rho_k)\\bbE\\left[\\|c(x_{k+1})\\|^2\\right]+3\\sigma^2\\zeta_{k+1}\\alpha_k^2.\\label{l5-e2}\n\\end{align}\nThe conclusion of this lemma follows by replacing $k$ with $i$ in the above inequalities and summing them up for all $1\\leq i\\leq k-1$.\n\\end{proof}\n\n\\begin{lemma}\\label{l-dist}\nSuppose that Assumptions \\ref{a1} and \\ref{a-storm} hold, and $\\{g_k\\}$ and  $\\{x_k\\}$ are generated by Algorithm \\ref{alg1}. Then for all $k\\geq1$, we have\n\\beq \\label{residual-bnd}\n\\dist^2\\left(0,\\nabla Q_{\\rho_k}(x_{k+1})+\\mcN_\\mcX(x_{k+1})\\right)\\leq3\\left(\\eta_k^{-2}+(\\bL_{\\nabla f}+\\rho_kL)^2\\right)\\|x_{k+1}-x_k\\|^2+3\\|g_k-\\nabla f(x_k)\\|^2,\n\\eeq\nwhere  $\\{\\rho_k\\}$ and $\\{\\eta_k\\}$ are given in Algorithm \\ref{alg1}, $\\bL_{\\nabla f}$ is given in Assumption \\ref{a-storm},  and $L$ and $Q_\\rho$ are defined in \\eqref{def-L} and \\eqref{def-Q}, respectively.\n\\end{lemma}\n\n\\begin{proof}\nBy the expression of $x_{k+1}$ in Algorithm \\ref{alg1}, one has\n\\beq \\label{subprob_opt}\n0\\in x_{k+1}-x_k+\\eta_k G_k+\\mcN_\\mcX(x_{k+1})\\quad\\Rightarrow\\quad\\eta_k^{-1}(x_k-x_{k+1})-G_k\\in\\mcN_\\mcX(x_{k+1}).\n\\eeq\nNotice from the definition of $Q_\\rho$ in \\eqref{def-Q} that $\\nabla Q_{\\rho_k}(x)=\\nabla f(x)+\\rho_k\\nabla c(x)c(x)$, which together with \\eqref{def-L}, \\eqref{bL-smooth} and Assumption \\ref{a1}(iv) implies that $Q_{\\rho_k}$ is $(\\bL_{\\nabla f}+\\rho_kL)$-smooth on $\\mcX$. Using \\eqref{subprob_opt}, the expression of $\\nabla Q_{\\rho_k}$, and $G_k=g_k+\\rho_k\\nabla c(x_k)c(x_k)$ (see Algorithm \\ref{alg1}), we have\n\\[\n\\eta_k^{-1}(x_k-x_{k+1})+\\nabla f(x_k)-g_k-\\nabla Q_{\\rho_k}(x_k) = \\eta_k^{-1}(x_k-x_{k+1})-g_k-\\rho_k\\nabla c(x_k)c(x_k) \\in \\mcN_\\mcX(x_{k+1}).\n\\]\nBy this and the $(\\bL_{\\nabla f}+\\rho_kL)$-smoothness of $Q_{\\rho_k}$, one has\n\\begin{align*}\n&\\dist^2\\left(0,\\nabla Q_{\\rho_k}(x_{k+1})+\\mcN_\\mcX(x_{k+1})\\right) \\leq \\|\\nabla Q_{\\rho_k}(x_{k+1})\n+(\\eta_k^{-1}(x_k-x_{k+1})+\\nabla f(x_k)-g_k-\\nabla Q_{\\rho_k}(x_k))\\|^2\\\\\n&\\leq 3\\left(\\|\\nabla Q_{\\rho_k}(x_{k+1})-\\nabla Q_{\\rho_k}(x_k)\\|^2+\\eta_k^{-2}\\|x_{k+1}-x_k\\|^2+\\| g_k-\\nabla f(x_k)\\|^2\\right)\\\\\n&\\leq 3\\left(\\eta_k^{-2}+(\\bL_{\\nabla f}+\\rho_kL)^2\\right)\\|x_{k+1}-x_k\\|^2+3\\|g_k-\\nabla f(x_k)\\|^2,\n\\end{align*}\nwhere the second inequality follows from the convexity of $\\|\\cdot\\|^2$, and the last inequality is due to the $(\\bL_{\\nabla f}+\\rho_kL)$-smoothness of $Q_{\\rho_k}$. Hence, the conclusion of this lemma holds.\n\\end{proof}\n\n\\begin{proof}[\\textbf{Proof of Theorem \\ref{thm:storm}}]\nUsing \\eqref{def-para2}, \\eqref{def-tK2} and \\eqref{def-zeta}, we have that for all $i\\geq\\tK_1$,\n\\begin{align}\n\\bL_{\\nabla f}+\\rho_iL &\\overset{\\eqref{def-para2}}=\\bL_{\\nabla f}i^{-\\nu}\\eta_i^{-1}/(4\\log(i+2))+L\\eta_i^{-1}/(4\\log(i+2))\\nn\\\\\n&\\ \\leq\\bL_{\\nabla f}\\tK_1^{-\\nu}\\eta_i^{-1}/4+L\\eta_i^{-1}/(4\\log(\\tK_1+2))\\overset{\\eqref{def-tK2}}\\leq\\eta_i^{-1}/4,\\label{t1-e2}\\\\\n12\\zeta_{i+1}\\bL_{\\nabla f}^2& \\overset{\\eqref{def-para2}\\eqref{def-zeta}}=3\\bL_{\\nabla f}^2\\big((i+1)/i\\big)^\\nu\\eta_i^{-1}/\\big(2\\log(i+2)\\log 3\\big)\\nn\\\\\n&\\ \\ \\ \\leq3\\bL_{\\nabla f}^22^\\nu\\eta_i^{-1}/(2\\log(\\tK_1+2))\\overset{\\eqref{def-tK2}}\\leq\\eta_i^{-1}/4,\\nn\n\\end{align}\nwhich imply that \n\\beq\\label{t1-ineq2}\n\\bL_{\\nabla f}+\\rho_iL+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\leq\\eta_i^{-1}/2\\qquad\\forall i\\geq\\tK_1.\n\\eeq\nIt then follows from the proof of Lemma \\ref{l-pfunc} that \\eqref{l5-e2} holds. Using \\eqref{t1-ineq2} and rearranging the terms of \\eqref{l5-e2} with $k$ replaced by $i$, we obtain that for all $i\\geq\\tK_1$, \n\\begin{align}\n&\\frac{1}{4\\eta_i}\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\frac{\\eta_i}{2}\\bbE\\left[\\|g_i-\\nabla f(x_i)\\|^2\\right]\\nn\\\\\n&\\overset{\\eqref{l5-e2}}\\leq \\bbE\\left[Q_{\\rho_i}(x_i)+\\zeta_i\\|g_i-\\nabla f(x_i)\\|^2\\right]-\\bbE\\left[Q_{\\rho_{i+1}}(x_{i+1})+\\zeta_{i+1}\\|g_{i+1}-\\nabla f(x_{i+1})\\|^2\\right]\\nn\\\\\n&\\quad\\,\\,\\,+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_iL+12\\zeta_{i+1}\\bL_{\\nabla f}^2-\\eta_i^{-1}/2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\frac{1}{2}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]+3\\sigma^2\\zeta_{i+1}\\alpha_i^2\\nn\\\\\n&\\overset{\\eqref{t1-ineq2}}{\\leq}\\bbE\\left[Q_{\\rho_i}(x_i)+\\zeta_i\\|g_i-\\nabla f(x_i)\\|^2\\right]-\\bbE\\left[Q_{\\rho_{i+1}}(x_{i+1})+\\zeta_{i+1}\\|g_{i+1}-\\nabla f(x_{i+1})\\|^2\\right]\\nn\\\\\n&\\quad\\,\\,\\,+\\frac{1}{2}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]+3\\sigma^2\\zeta_{i+1}\\alpha_i^2.\\label{t1-p2}\n\\end{align}\nRecall that $\\iota_k$ is the random variable uniformly generated in $\\left\\{\\lceil k/2\\rceil+1,\\dots,k\\right\\}$. In addition, observe from \\eqref{def-para2} that $\\eta_i ^{-1}<\\eta_{k-1}^{-1}$ for all $\\lceil k/2\\rceil \\leq i \\leq k-1$. By these, \\eqref{def-Q}, \\eqref{def-para2}, \\eqref{penalty-bnd}, \\eqref{residual-bnd}, \\eqref{t1-e2} and \\eqref{t1-p2}, one has that for all $k\\geq2\\tK_1$,\n\\begin{align}\n&\\ \\bbE\\left[\\dist^2\\left(0,\\nabla f(x_{\\iota_k})+\\rho_{\\iota_k-1}\\nabla c(x_{\\iota_k})c(x_{\\iota_k})+\\mcN_\\mcX(x_{\\iota_k})\\right)\\right]=\\bbE\\left[\\dist^2\\left(0,\\nabla Q_{\\rho_{\\iota_k-1}}(x_{\\iota_k})+\\mcN_\\mcX(x_{\\iota_k})\\right)\\right]\\nn\\\\\n&\\ =\\frac{1}{k-\\lceil k/2\\rceil}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\bbE\\left[\\dist^2\\left(0,\\nabla Q_{\\rho_i}(x_{i+1})+\\mcN_\\mcX(x_{i+1})\\right)\\right]\\nn\\\\\n& \\overset{\\eqref{residual-bnd}}\\leq  \\frac{3}{k-\\lceil k/2\\rceil}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\left(\\left(\\eta_i^{-2}+(\\bL_{\\nabla f}+\\rho_iL)^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\bbE\\left[\\|g_i-\\nabla f(x_i)\\|^2\\right]\\right)\\nn\\\\\n& \\overset{\\eqref{t1-e2}}{\\leq} \\frac{3}{k-\\lceil k/2\\rceil}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\left((\\eta_i^{-2}+\\eta_i^{-2}/16)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\bbE\\left[\\|g_i-\\nabla f(x_i)\\|^2\\right]\\right)\\nn\\\\\n&\\ \\leq \\frac{51}{8(k-1)}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\left(\\eta_i^{-2}\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+2\\bbE\\left[\\|g_i-\\nabla f(x_i)\\|^2\\right]\\right)\\nn\\\\\n&\\ \\leq \\frac{51}{2(k-1)\\eta_{k-1}}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\left(\\frac{1}{4\\eta_i}\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\frac{\\eta_i}{2}\\bbE\\left[\\|g_i-\\nabla f(x_i)\\|^2\\right]\\right)\\nn\\\\\n& \\overset{\\eqref{t1-p2}}{\\leq}\\frac{51}{2(k-1)\\eta_{k-1}}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\Big(\\bbE\\left[Q_{\\rho_i}(x_i)+\\zeta_i\\|g_i-\\nabla f(x_i)\\|^2\\right]-\\bbE\\left[Q_{\\rho_{i+1}}(x_{i+1})+\\zeta_{i+1}\\|g_{i+1}-\\nabla f(x_{i+1})\\|^2\\right]\\nn\\\\\n&\\ \\quad\\,\\,\\,+\\frac{1}{2}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]+3\\sigma^2\\zeta_{i+1}\\alpha_i^2\\Big)\\nn\\\\\n&\\ = \\frac{51}{2(k-1)\\eta_{k-1}}\\Bigg(\\bbE\\left[Q_{\\rho_{\\lceil k/2\\rceil}}(x_{\\lceil k/2\\rceil})+\\zeta_{\\lceil k/2\\rceil}\\|g_{\\lceil k/2\\rceil}-\\nabla f(x_{\\lceil k/2\\rceil})\\|^2\\right]-\\bbE\\left[Q_{\\rho_k}(x_k)+\\zeta_k\\|g_k-\\nabla f(x_k)\\|^2\\right]\\nn\\\\\n&\\ \\quad\\,\\,\\,+\\frac{1}{2}\\sum_{i=\\lceil k/2\\rceil}^{k-1}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]+3\\sigma^2\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\zeta_{i+1}\\alpha_i^2\\Bigg)\\nn\\\\\n&\\ \\leq \\frac{51}{2(k-1)\\eta_{k-1}}\\Bigg(Q_1(x_1)-Q_1^*+\\zeta_1\\|g_1-\\nabla f(x_1)\\|^2+\\frac{1}{2}\\sum_{i=1}^{k-1}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]+3\\sigma^2\\sum_{i=1}^{k-1}\\zeta_{i+1}\\alpha_i^2\\nn\\\\\n&\\ \\quad\\,\\,\\,+\\frac{1}{2}\\sum_{i=1}^{\\lceil k/2\\rceil-1}\\left(\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]\\Bigg),\\label{t1-sum2}\n\\end{align}\nwhere the first equality is due to \\eqref{def-Q}, the first inequality follows from taking expectation on both sides of \\eqref{residual-bnd}, the third inequality is due to the fact that $\\lceil k/2\\rceil\\leq(k+1)/2$, the fourth inequality follows from the relation $\\eta_i ^{-1}<\\eta_{k-1}^{-1}$ for all $\\lceil k/2\\rceil \\leq i \\leq k-1$, and the last inequality follows from \\eqref{penalty-bnd} with $k$ replaced by $\\lceil k/2\\rceil$, $\\rho_1=1$, and the fact that $Q_{\\rho_k}(x_k)\\geq Q_1 (x_k) \\geq Q_1^*$.\n\nWe next bound each term in the summation \\eqref{t1-sum2}.  Indeed, it follows from \\eqref{cf-bnd}, Assumption \\ref{a1}(iv), the nonexpansiveness of $\\Pi_X$, and the expressions of $x_{k+1}$, $g_k$ and $G_k$ in Algorithm \\ref{alg1} that\n\\begin{align}\n\\|x_{k+1}-x_k\\|^2&=\\|\\Pi_\\mcX(x_k-\\eta_kG_k)-\\Pi_X(x_k)\\|^2\\leq\\eta_k^2\\|G_k\\|^2=\\eta_k^2\\|g_k+\\rho_k\\nabla c(x_k)c(x_k)\\|^2\\nn\\\\\n&\\leq2\\eta_k^2\\left(\\|g_k\\|^2+\\rho_k^2\\|\\nabla c(x_k)c(x_k)\\|^2\\right)\\leq2\\eta_k^2\\left(L_f^2+C_c^2L_c^2\\rho_k^2\\right).\\label{t1-x1}\n\\end{align}\nRecall that $\\nu=\\min\\{\\hat\\theta/(\\hat\\theta+2),1/2\\}$ for some $\\hat\\theta\\geq1$ and $\\|c(x_i)\\| \\leq C_c$ for all $i$. By this, Lemma \\ref{l-cnst2}, \\eqref{def-para2}, \\eqref{def-zeta},  \\eqref{t1-x1},  one has that for all $k\\geq2\\tK_1$, \n\\begin{align}\n&\\sum_{i=1}^{\\tK_1-1}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]\\leq C_c^2\\sum_{i=1}^{\\tK_1-1}\\big((i+1)^\\nu-i^\\nu\\big)=C_c^2\\big(\\tK_1^\\nu-1\\big),\\label{sum2-c1}\\\\\n&\\sum_{i=\\tK_1}^{k-1}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]\\leq2C_1\\sum_{i=\\tK_1}^{k-1}((i+1)^\\nu-i^\\nu)(i+1)^{-\\frac{2\\nu}{\\theta}}\\leq 2\\nu C_1\\sum_{i=\\tK_1}^{k-1}i^{\\nu-1}(i+1)^{-\\frac{2\\nu}{\\theta}} \\label{sum2-c2} \\\\\n&\\leq C_1\\sum_{i=\\tK_1}^{k-1}i^{-1}(i+1)^{\\nu-\\frac{2\\nu}{\\theta}}\\leq C_1\\max\\big\\{1,k^{\\nu-\\frac{2\\nu}{\\theta}}\\big\\}\\sum_{i=\\tK_1}^{k-1}i^{-1}\\leq C_1\\max\\big\\{1,k^{\\nu-\\frac{2\\nu}{\\theta}}\\big\\}(1+\\log k), \\label{sum2-c3} \\\\\n&\\sum_{i=1}^{k-1}\\zeta_{i+1}\\alpha_i^2\\overset{\\eqref{def-zeta}}=\\frac{1}{2\\log3}\\sum_{i=1}^{k-1}(i+1)^{\\nu}i^{-4\\nu}= \\frac{1}{2\\log3}\\sum_{i=1}^{k-1}i^{-3\\nu}((i+1)/i)^{\\nu} \\nn \\\\\n& \\leq\\frac{\\sqrt{2}}{2\\log3}\\sum_{i=1}^{k-1}i^{-3\\nu} \\leq\\frac{\\sqrt{2}}{2\\log3}\\sum_{i=1}^{k-1}i^{-1}\\leq\\frac{\\sqrt{2}(1+\\log k)}{2\\log3},\\label{sum2-c4a}\\\\\n&\\ \\sum_{i=1}^{\\lceil k/2\\rceil-1}\\left(\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]\\nn\\\\\n&\\ =\\sum_{i=1}^{\\tK_1-1}\\left(\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\sum_{i=\\tK_1}^{\\lceil k/2\\rceil-1}\\Big((\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2)\\nn \\\\ \n&\\ \\quad\\,\\,\\,\\times \\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]\\Big) \\nn \\\\\n&\\overset{\\eqref{t1-ineq2}}{\\leq}\\sum_{i=1}^{\\tK_1-1}\\left(\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right] \\nn\\\\\n&\\overset{\\eqref{t1-x1}}{\\leq}2\\sum_{i=1}^{\\tK_1-1}\\left(\\bL_{\\nabla f}+\\rho_iL+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\eta_i^2\\left(L_f^2+C_c^2L_c^2\\rho_i^2\\right)\\nn\\\\\n&\\overset{\\eqref{def-para2}}{\\leq}\\frac{1}{8}(1+\\log\\tK_1)\\big(\\bL_{\\nabla f}+\\tK_1^{\\frac{1}{2}}L+6\\tK_1^{\\frac{1}{2}}\\bL_{\\nabla f}^2\\big)(L_f^2+C_c^2L_c^2\\tK_1),\\label{sum2-c4}\n\\end{align}\nwhere the inequality in \\eqref{sum2-c1} follows from \\eqref{def-para2}, $\\|c(x_i)\\| \\leq C_c$ for all $i$, the inequalities in \\eqref{sum2-c2} are due to \\eqref{def-para2}, Lemma \\ref{l-cnst2},  and $(i+1)^\\nu-i^\\nu\\leq \\nu i^{\\nu-1}$ for all $i\\geq1$ thanks to the concavity of $t^\\nu$, the inequalities in \\eqref{sum2-c4a} are due to $((i+1)/i)^\\nu\\leq\\sqrt{2}$ for all $i\\geq1$ and $\\nu\\geq1/3$, and the last inequality in \\eqref{sum2-c4} is due to the relations $\\rho_i \\leq \\tK_1^{1/2}$ and $\\zeta_i \\leq \\tK_1^{1/2}/2$ for $1\\leq i\\leq \\tK_1-1$ and\n$\\sum_{i=1}^{\\tK_1-1}\\eta_i^2 \\leq \\sum_{i=1}^{\\tK_1-1}i^{-1}/16\\leq (1+\\log\\tK_1)/16$ thanks to $\\nu\\leq 1/2$ and the choice of $\\rho_i$, $\\eta_i$ and $\\zeta_i$ in \\eqref{def-para2} and \\eqref{def-zeta}.\n\nUsing \\eqref{def-para2}, \\eqref{def-zeta}, \\eqref{t1-sum2}, \\eqref{sum2-c1}, \\eqref{sum2-c3}, \\eqref{sum2-c4a} and \\eqref{sum2-c4}, we have\n\\begin{align*}\n&\\bbE\\left[\\dist^2\\left(0,\\nabla f(x_{\\iota_k})+\\rho_{\\iota_k-1}\\nabla c(x_{\\iota_k})c(x_{\\iota_k})+\\mcN_\\mcX(x_{\\iota_k})\\right)\\right]\\\\\n&\\leq \\frac{102\\log(k+1)}{(k-1)^{1-\\nu}}\\Bigg(Q_1(x_1)-Q_1^*+\\frac{\\|g_1-\\nabla f(x_1)\\|^2}{2\\log 3}+\\frac{1}{2}C_1\\max\\{1,k^{\\nu-\\frac{2\\nu}{\\theta}}\\}(1+\\log k)\\nn \\\\ \n&\\quad+\\frac{1}{2}C_c^2\\big(\\tK_1^{\\nu}-1\\big)\n+\\frac{3\\sqrt{2}\\sigma^2(1+\\log k)}{2\\log3}+\\frac{1}{16}(1+\\log\\tK_1)\\big(\\bL_{\\nabla f}+\\tK_1^{\\frac{1}{2}}L+6\\tK_1^{\\frac{1}{2}}\\bL_{\\nabla f}^2\\big)(L_f^2+C_c^2L_c^2\\tK_1)\\Bigg).\n\\end{align*}\nBy this, \\eqref{def-Q}, $\\iota_k>\\lceil k/2\\rceil\\geq\\tK_1$ for all $k\\geq2\\tK_1$, and Lemma \\ref{l-cnst2} with $k$ replaced by $\\iota_k$, one can see that Theorem \\ref{thm:storm} holds.\n\\end{proof}","units":[{"start":51021,"end":51538,"start_line":560,"end_line":566,"raw_tex":"\\begin{lemma}\\label{l-rec} \nSuppose that Assumption \\ref{a1} holds, and $x_{k+1}$ is generated by Algorithm \\ref{alg1} for some $k\\geq1$ with $\\rho_k\\eta_k\\leq(\\sqrt{5}-1)/(2L)$.  Then we have\n\\[\nh(x_{k+1})+2^{\\theta-2}\\gamma^2\\rho_k\\eta_k[h(x_{k+1})]^{\\theta}\\leq h(x_k)+L_f^2\\rho_k^{-1}\\eta_k/2,\n\\]\nwhere $\\rho_k$ and $\\eta_k$ are given in Algorithm \\ref{alg1}, $L_f$, $\\gamma$ and $\\theta$ are given in Assumption \\ref{a1},  and $L$ and $h$ are defined in \\eqref{def-L} and \\eqref{def-h}, respectively.\n\\end{lemma}","sha256_utf8":"5b155be6ba769d4c43bc6786448421bb202ddd1fc1f77014044055b2a8a5d3df"},{"start":51540,"end":55199,"start_line":568,"end_line":618,"raw_tex":"\\begin{proof}\nLet $G_k$ be given in Algorithm \\ref{alg1}. For convenience, we define\n\\beq\\label{l3-e1}\n\\widetilde G_k=\\rho_k^{-1}G_k,\\quad\\tilde\\eta_k=\\rho_k\\eta_k.\n\\eeq\n It then follows from these, \\eqref{def-h}, and the expression of $x_{k+1}$ in Algorithm \\ref{alg1} that\n\\beq\\label{l3-x}\nx_{k+1}=\\Pi_\\mcX(x_k-\\eta_kG_k)=\\Pi_\\mcX(x_k-\\tilde\\eta_k\\widetilde G_k),\n\\eeq\nwhich implies that\n\\beq \\label{subprob-opt1}\n0\\in x_{k+1}-x_k+\\tilde\\eta_k\\widetilde G_k+\\mcN_\\mcX(x_{k+1})\\quad\\Rightarrow\\quad \\nabla h(x_{k+1})+\\tilde\\eta_k^{-1}(x_k-x_{k+1})-\\widetilde G_k\\in\\nabla h(x_{k+1})+\\mcN_\\mcX(x_{k+1}).\n\\eeq\nUsing this, \\eqref{def-h} and Assumption \\ref{a1}(iv), we have\n\\begin{align}\n2^\\theta\\gamma^2[h(x_{k+1})]^\\theta&\\ =\\gamma^2\\|c(x_{k+1})\\|^{2\\theta}\\leq\\dist^2\\left(0,\\nabla c(x_{k+1})c(x_{k+1})+\\mcN_\\mcX(x_{k+1})\\right) \\nn \\\\\n&\\overset{\\eqref{def-h}}{=}\\dist^2\\left(0,\\nabla h(x_{k+1})+\\mcN_\\mcX(x_{k+1})\\right)\\overset{\\eqref{subprob-opt1}}\\leq\\|\\nabla h(x_{k+1})+\\tilde\\eta_k^{-1}(x_k-x_{k+1})-\\widetilde G_k\\|^2\\nn\\\\\n&\\ \\leq2\\|\\tilde\\eta_k^{-1}(x_k-x_{k+1})+\\nabla h(x_k)-\\widetilde G_k\\|^2+2\\|\\nabla h(x_{k+1})-\\nabla h(x_k)\\|^2\\nn\\\\\n&\\ =2\\tilde\\eta_k^{-2}\\|x_{k+1}-x_k\\|^2+4\\tilde\\eta_k^{-1}\\langle\\widetilde G_k-\\nabla h(x_k), x_{k+1}-x_k\\rangle+2\\|\\widetilde G_k-\\nabla h(x_k)\\|^2\\nn\\\\\n&\\ \\quad +2\\|\\nabla h(x_{k+1})-\\nabla h(x_k)\\|^2\\nn\\\\\n&\\ \\leq2(\\tilde\\eta_k^{-2}+L^2)\\|x_{k+1}-x_k\\|^2+4\\tilde\\eta_k^{-1}\\langle\\widetilde G_k-\\nabla h(x_k), x_{k+1}-x_k\\rangle+2\\|\\widetilde G_k-\\nabla h(x_k)\\|^2,\\label{l3-e2}\n\\end{align}\nwhere the first inequality follows from Assumption \\ref{a1}(iv), the second inequality is due to the convexity of $\\|\\cdot\\|^2$, and the last inequality follows from the $L$-smoothness of $h$. In addition,  by \\eqref{l3-x} and $x_k\\in\\mcX$, one has\n\\[\n\\langle x_{k+1}-x_k+\\tilde\\eta_k\\widetilde G_k,x_k-x_{k+1}\\rangle\\geq0\\quad\\Rightarrow\\quad \\langle \\widetilde G_k,x_{k+1}-x_k\\rangle\\leq-\\tilde\\eta_k^{-1}\\|x_{k+1}-x_k\\|^2.\n\\]\nThis together with the $L$-smoothness of $h$ yields\n\\begin{align*}\nh(x_{k+1})& \\leq h(x_k)+\\langle\\nabla h(x_k),x_{k+1}-x_k\\rangle +\\frac{L}{2}\\|x_{k+1}-x_k\\|^2\\nn\\\\\n&=h(x_k)+\\langle\\widetilde G_k,x_{k+1}-x_k\\rangle+\\langle\\nabla h(x_k)-\\widetilde G_k,x_{k+1}-x_k\\rangle +\\frac{L}{2}\\|x_{k+1}-x_k\\|^2\\nn\\\\\n& \\leq h(x_k)-\\tilde\\eta_k^{-1}\\|x_{k+1}-x_k\\|^2+\\langle\\nabla h(x_k)-\\widetilde G_k,x_{k+1}-x_k\\rangle +\\frac{L}{2}\\|x_{k+1}-x_k\\|^2.\n\\end{align*}\nUsing this and \\eqref{l3-e2}, we obtain that\n\\beq\\label{l3-e3}\nh(x_{k+1})+2^{\\theta-2}\\gamma^2\\tilde\\eta_k[h(x_{k+1})]^{\\theta}\\leq h(x_k)+\\frac{1}{2}(L^2\\tilde\\eta_k-\\tilde\\eta_k^{-1}+L)\\|x_{k+1}-x_k\\|^2+\\frac{\\tilde\\eta_k}{2}\\|\\widetilde G_k-\\nabla h(x_k)\\|^2.\n\\eeq\nObserve from \\eqref{l3-e1} and $\\rho_k\\eta_k\\leq(\\sqrt{5}-1)/(2L)$ that $L\\tilde\\eta_k=L\\rho_k\\eta_k\\leq(\\sqrt{5}-1)/2$, which implies that\n\\beq\\label{l3-e4}\nL^2\\tilde\\eta_k-\\tilde\\eta_k^{-1}+L=\\tilde\\eta_k^{-1}( L^2\\tilde\\eta_k^2+L\\tilde\\eta_k-1)\\leq0.\n\\eeq\nNotice from the expression of $g_k$ in Algorithm \\ref{alg1} that $g_k\\in\\cB(L_f)$ and hence $\\|g_k\\|\\leq L_f$. \nAlso, observe from Algorithm \\ref{alg1} and \\eqref{def-h} that $G_k=g_k+\\rho_k\\nabla h(x_k)$. Using these and \\eqref{l3-e1}, we have\n\\[\n\\|\\widetilde G_k-\\nabla h(x_k)\\|= \\|\\rho_k^{-1}G_k-\\nabla h(x_k)\\| =\\|\\rho_k^{-1}(g_k+\\rho_k\\nabla h(x_k))-\\nabla h(x_k)\\|= \\rho_k^{-1}\\|g_k\\|\\leq\\rho_k^{-1}L_f.\n\\]\nIt then follows from this, \\eqref{l3-e3} and \\eqref{l3-e4} that \n\\[\nh(x_{k+1})+2^{\\theta-2}\\gamma^2\\tilde\\eta_k[h(x_{k+1})]^\\theta\\leq h(x_k)+L_f^2\\tilde\\eta_k/(2\\rho_k^2).\n\\]\nThis and the definition of $\\tilde\\eta_k$ in \\eqref{l3-e1} imply that the conclusion of this lemma holds.\n\\end{proof}","sha256_utf8":"a57ebcca7735da14977af170510ce9ab15bdefb3cdd28ff09d780445f800d672"},{"start":55332,"end":55731,"start_line":622,"end_line":624,"raw_tex":"\\begin{lemma}\\label{l-cnst2}\nLet $\\nu$, $\\tK_1$, and $C_1$ be given in \\eqref{def-para2}, \\eqref{def-tK2} and \\eqref{def-nuC2}, respectively. Suppose that Assumption \\ref{a1} holds, and $\\{x_k\\}$ is generated by Algorithm \\ref{alg1} with $\\{\\rho_k\\}$, $\\{\\eta_k\\}$ and $\\{\\alpha_k\\}$ given in \\eqref{def-para2}. Then  we have $\\|c(x_k)\\|^2\\leq 2C_1k^{-2\\nu/\\theta}$ for all $k\\geq\\tK_1$.\n\\end{lemma}","sha256_utf8":"7f3a66c7941d9b969d378ae1500c9c30b67cff8d250ad92761678be46544992a"},{"start":55733,"end":59603,"start_line":626,"end_line":665,"raw_tex":"\\begin{proof}\nLet $h$ be defined in \\eqref{def-h}. To prove this lemma, it is equivalent to show that $h(x_k)\\leq C_1k^{-2\\nu/\\theta}$ for all $k\\geq\\tK_1$.  We now prove this by induction. Indeed, notice from Algorithm \\ref{alg1} that $x_{\\tK_1}\\in X$. It then follows from  \\eqref{def-nuC2}, \\eqref{def-h}, and Assumption \\ref{a1}(iv) that\n\\[\nh(x_{\\tK_1})\\overset{\\eqref{def-h}}{=}\\frac{1}{2}\\|c(x_{\\tK_1})\\|^2\\leq\\frac{1}{2}C_c^2\\overset{\\eqref{def-nuC2}}{\\leq} C_1\\tK_1^{-2\\nu/\\theta}.\n\\]\nHence,  the conclusion holds for $k=\\tK_1$. Now, suppose for induction that $h(x_k)\\leq C_1k^{-2\\nu/\\theta}$ holds for some $k\\geq\\tK_1$. Recall that $\\rho_k$, $\\eta_k$ and $\\tK_1$ are given in \\eqref{def-para2} and \\eqref{def-tK2}.  In view of these, one can observe that \n\\[\n\\rho_k\\eta_k\\overset{\\eqref{def-para2} }=\\frac{1}{4\\log(k+2)}\\leq\\frac{1}{4\\log(\\tK_1+2)}\\overset{\\eqref{def-tK2}}\\leq \\frac{1}{8L}\\leq\\frac{\\sqrt{5}-1}{2L},\n\\]\nand hence Lemma \\ref{l-rec} holds for such $k$.  Using Lemma \\ref{l-rec} with the choice of $\\rho_k$ and $\\eta_k$ given in \\eqref{def-para2}, we obtain that\n\\beq\\label{rec2}\nh(x_{k+1})+2^{\\theta-4}\\gamma^2[h(x_{k+1})]^{\\theta}/\\log(k+2)\\leq h(x_k)+L_f^2k^{-2\\nu}/(8\\log(k+2)).\n\\eeq\nFurther, let\n\\beq\\label{def-phi2}\n\\phi(t)=t+2^{\\theta-4}\\gamma^2t^\\theta/\\log(k+2).\n\\eeq\nNotice from \\eqref{def-para2} and \\eqref{def-nuC2} that $\\nu=\\min\\{\\hat\\theta/(\\hat\\theta+2),1/2\\}$ for some $\\hat\\theta\\geq1$ and $C_1\\geq1$. Using these and \\eqref{def-phi2},  we have\n\\begin{align}\n&\\ \\phi(C_1(k+1)^{-2\\nu/\\theta})-C_1k^{-2\\nu/\\theta}-L_f^2k^{-2\\nu}/(8\\log(k+2))\\nn\\\\\n&\\overset{\\eqref{def-phi2}}{=}C_1^\\theta 2^{\\theta-4}\\gamma^2(k+1)^{-2\\nu}/\\log(k+2)+C_1(k+1)^{-2\\nu/\\theta}-C_1k^{-2\\nu/\\theta}-L_f^2k^{-2\\nu}/(8\\log(k+2))\\nn\\\\\n&\\ \\geq C_1^\\theta 2^{\\theta-4}\\gamma^2(k+1)^{-2\\nu}/\\log(k+2)-2\\nu C_1 k^{-2\\nu/\\theta-1}/\\theta-L_f^2k^{-2\\nu}/(8\\log(k+2))\\nn\\\\\n&\\ = \\frac{k^{-2\\nu}}{\\log(k+2)}\\left(C_1^\\theta 2^{\\theta-4}\\gamma^2\\left(\\frac{k}{k+1}\\right)^{2\\nu}- 2\\nu C_1 k^{2\\nu(1-\\theta^{-1})-1}\\log(k+2)/\\theta-L_f^2/8\\right)\\nn\\\\\n&\\ \\geq \\frac{k^{-2\\nu}}{\\log(k+2)}\\left(C_1^\\theta 2^{\\theta-5}\\gamma^2-C_1 k^{-\\frac{1}{\\theta}}\\log(k+2)-L_f^2/8\\right)\\nn\\\\\n&\\ \\geq \\frac{k^{-2\\nu}}{\\log(k+2)}\\left(C_12^{\\theta-5}\\gamma^2-C_1 k^{-\\frac{1}{\\theta}}\\log(k+2)-L_f^2/8\\right),\\label{l4-e1g}\n\\end{align}\nwhere the first inequality follows from $(k+1)^{-2\\nu/\\theta}-k^{-2\\nu/\\theta}\\geq-2\\nu k^{-2\\nu/\\theta-1}/\\theta$ thanks to the convexity of $t^{-2\\nu/\\theta}$, the second inequality is due to $\\theta\\geq1$, $\\nu\\leq1/2$ and  $k/(k+1)\\geq1/2$, and the last inequality follows from $\\theta\\geq1$ and $C_1\\geq1$. In addition, one can verify that $t^{-\\frac{1}{2\\theta}}\\log(t+2)$ is decreasing on $[e^{2\\theta},\\infty)$. Using this, \\eqref{def-tK2}, and $k\\geq\\tK_1\\geq e^{2\\theta}$, we obtain that\n\\[\nk^{-\\frac{1}{2\\theta}}\\log(k+2)\\leq\\log(e^{2\\theta}+2)/e,\\quad k^{-\\frac{1}{2\\theta}}\\leq\\tK_1^{-\\frac{1}{2\\theta}}\\leq e\\gamma^2/(2^{6-\\theta}\\log(e^{2\\theta}+2)).\n\\]\nMultiplying both sides of these inequalities yields $k^{-1/\\theta}\\log(k+2)\\leq2^{\\theta-6}\\gamma^2$, which together with \\eqref{def-nuC2}  implies that\n\\begin{align*}\nC_12^{\\theta-5}\\gamma^2-C_1 k^{-\\frac{1}{\\theta}}\\log(k+2)-L_f^2/8\\geq C_12^{\\theta-6}\\gamma^2-L_f^2/8\\overset{\\eqref{def-nuC2}}{\\geq}0.\n\\end{align*}\nUsing this,  \\eqref{rec2}, \\eqref{def-phi2},  \\eqref{l4-e1g}, and the induction hypothesis that $h(x_k)\\leq C_1k^{-2\\nu/\\theta}$, we obtain that\n\\[\n\\phi(C_1(k+1)^{-2\\nu/\\theta})\\geq C_1k^{-2\\nu/\\theta}+L_f^2k^{-2\\nu}/(8\\log(k+2))\\geq h(x_k)+L_f^2k^{-2\\nu}/(8\\log(k+2))\\overset{\\eqref{rec2}\\eqref{def-phi2}}{\\geq}\\phi(h(x_{k+1})).\n\\]\nIt then follows from this inequality and the strict monotonicity of $\\phi$ on $[0,\\infty)$ that $h(x_{k+1})\\leq C_1(k+1)^{-2\\nu/\\theta}$. Hence, the induction is completed and the conclusion of this lemma holds. \n\\end{proof}","sha256_utf8":"5af095f81a62b69373379c398c2b21e00638d561c0e3e3da7e2f1ef6ea94ddc5"},{"start":59758,"end":60295,"start_line":670,"end_line":676,"raw_tex":"\\begin{lemma}\\label{l-exp}\nSuppose that Assumptions \\ref{a1} and \\ref{a-storm} hold, and $\\{g_k\\}$ and $\\{x_k\\}$ are generated by Algorithm \\ref{alg1}. Then for all $k\\geq1$, we have\n\\[\n\\bbE\\left[\\|g_{k+1}-\\nabla f(x_{k+1})\\|^2\\right]\\leq(1-\\alpha_k)^2\\bbE\\left[\\|g_k-\\nabla f(x_k)\\|^2\\right]+6\\bL_{\\nabla f}^2\\bbE\\left[\\|x_{k+1}-x_k\\|^2\\right]+3\\sigma^2\\alpha_k^2,\n\\]\nwhere $\\{\\alpha_k\\}$ is given in Algorithm \\ref{alg1}, and $\\sigma$ and $\\bL_{\\nabla f}$ are given in Assumptions \\ref{a1} and \\ref{a-storm}, respectively. \n\\end{lemma}","sha256_utf8":"051dee35e3d4972aa55cd5ddd39e6ac8c47d737086db42e41c72253a95bf62ed"},{"start":60297,"end":63510,"start_line":678,"end_line":715,"raw_tex":"\\begin{proof}\nNotice from \\eqref{cf-bnd} that $\\nabla f(x_{k+1})\\in\\cB(L_f)$ and hence $\\nabla f(x_{k+1})=\\Pi_{\\cB(L_f)}(\\nabla f(x_{k+1}))$. By this, the expression of $g_{k+1}$, and the nonexpansiveness of the projection operator $\\Pi_{\\cB(L_f)}$, one has\n\\begin{align}\n&\\|g_{k+1}-\\nabla f(x_{k+1})\\|^2=\\|\\Pi_{\\cB(L_f)}(\\nabla \\tf(x_{k+1},\\xi_{k+1})+(1-\\alpha_k)(g_k-\\nabla \\tf(x_k,\\xi_{k+1})))-\\Pi_{\\cB(L_f)}(\\nabla f(x_{k+1}))\\|^2 \\nn \\\\\n&\\leq\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})+(1-\\alpha_k)(g_k-\\nabla \\tf(x_k,\\xi_{k+1}))-\\nabla f(x_{k+1})\\|^2 \\nn \\\\\n&=\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})+(1-\\alpha_k)(g_k-\\nabla f(x_k)+\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1}))\\|^2 \\nn \\\\\n&=\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})+(1-\\alpha_k)(\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1}))\\|^2+(1-\\alpha_k)^2\\|g_k-\\nabla f(x_k)\\|^2\\nn \\\\\n&\\quad+2(1-\\alpha_k)\\langle g_k-\\nabla f(x_k),\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})\\rangle\\nn \\\\\n&\\quad+2(1-\\alpha_k)^2\\langle g_k-\\nabla f(x_k),\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1})\\rangle. \\label{g-bnd}\n\\end{align}\nLet $\\Xi_k=\\{\\xi_1,\\ldots, \\xi_k\\}$ denote the collection of samples drawn up to iteration $k-1$ in Algorithm \n\\ref{alg1}.  It then follows from Assumption \\ref{a1}(iii) that \n\\[\n\\bbE[\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})|\\Xi_k]=0, \\quad \\bbE[\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1})|\\Xi_k]=0,\n\\]\nwhich imply that\n\\begin{align*}\n&\\bbE[\\langle g_k-\\nabla f(x_k),\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})\\rangle|\\Xi_k]=\\langle g_k-\\nabla f(x_k),\\bbE[\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})|\\Xi_k]\\rangle=0, \\\\\n&\\bbE[\\langle g_k-\\nabla f(x_k),\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1})\\rangle|\\Xi_k]=\\langle g_k-\\nabla f(x_k),\\bbE[\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1})|\\Xi_k]\\rangle=0.\n\\end{align*}\nUsing these and taking a conditional expectation on both sides of \\eqref{g-bnd}, we have\n\\begin{align}\n\\bbE[\\|g_{k+1}-\\nabla f(x_{k+1})\\|^2|\\Xi_k]\\leq&\\bbE[\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})+(1-\\alpha_k)(\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1}))\\|^2|\\Xi_k]\\nn\\\\\n&+(1-\\alpha_k)^2\\|g_k-\\nabla f(x_k)\\|^2.\\label{l1-e1}\n\\end{align}\nIn addition, it follows from Assumption \\ref{a1} that\n\\begin{align*}\n&\\bbE[\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})+(1-\\alpha_k)(\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1}))\\|^2|\\Xi_k]\\\\\n&=\\bbE[\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla \\tf(x_k,\\xi_{k+1})+\\nabla f(x_k)-\\nabla f(x_{k+1})-\\alpha_k(\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1}))\\|^2|\\Xi_k]\\\\\n&\\leq3\\bbE[\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla \\tf(x_k,\\xi_{k+1})\\|^2|\\Xi_k]+3\\|\\nabla f(x_{k+1})-\\nabla f(x_k)\\|^2\\\\\n&\\quad+3\\alpha_k^2\\bbE[\\|\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1})\\|^2|\\Xi_k]\\leq6\\bL_{\\nabla f}^2\\|x_{k+1}-x_k\\|^2+3\\sigma^2\\alpha_k^2,\n\\end{align*}\nwhere the first inequality follows from the convexity of $\\|\\cdot\\|^2$, and the last inequality is due to \\eqref{bL-smooth} and Assumptions \\ref{a1}(iii) and \\ref{a-storm}. By this and \\eqref{l1-e1}, one has\n\\[\n\\bbE[\\|g_{k+1}-\\nabla f(x_{k+1})\\|^2|\\Xi_k]\\leq (1-\\alpha_k)^2\\|g_k-\\nabla f(x_k)\\|^2+6\\bL_{\\nabla f}^2\\|x_{k+1}-x_k\\|^2+3\\sigma^2\\alpha_k^2.\n\\]\nThe conclusion of this lemma follows from taking expectation on both sides of this inequality.\n\\end{proof}","sha256_utf8":"88f45a604b60d100b407ede719e7e2f2388c389a984bb77c9647abd6edecabb5"},{"start":63658,"end":64669,"start_line":719,"end_line":730,"raw_tex":"\\begin{lemma}\\label{l-pfunc}\nSuppose that Assumptions \\ref{a1} and \\ref{a-storm} hold. Let $\\{g_k\\}$ and $\\{x_k\\}$ be generated by Algorithm \\ref{alg1}, and  let \n\\beq\\label{def-zeta}\n\\zeta_k=k^{\\nu}/(2\\log 3),\n\\eeq\nwhere $\\nu$ is defined in \\eqref{def-para2}. Then for all $k\\geq1$, we have\n\\begin{align}\n&\\bbE\\left[Q_{\\rho_k}(x_k)+\\zeta_k\\|g_k-\\nabla f(x_k)\\|^2\\right] \\leq Q_{\\rho_1}(x_1)+\\zeta_1\\|g_1-\\nabla f(x_1)\\|^2+\\frac{1}{2}\\sum_{i=1}^{k-1}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right] \\nn \\\\\n&+\\frac{1}{2}\\sum_{i=1}^{k-1}\\left(\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+3\\sigma^2\\sum_{i=1}^{k-1}\\zeta_{i+1}\\alpha_i^2, \\label{penalty-bnd}\n\\end{align}\nwhere $\\{\\alpha_k\\}$, $\\{\\rho_k\\}$ and $\\{\\eta_k\\}$ are given in Algorithm \\ref{alg1}, $Q_\\rho$  and $L$ are respectively defined in \\eqref{def-Q} and \\eqref{def-L},  and $\\sigma$ and $\\bL_{\\nabla f}$  are given in Assumptions \\ref{a1} and \\ref{a-storm}, respectively.\n\\end{lemma}","sha256_utf8":"438305b7713a343ffed9dd2ca99cc07a8244abb0cfe99f8612cecbfcaa070ca2"},{"start":64671,"end":69558,"start_line":732,"end_line":780,"raw_tex":"\\begin{proof}\nObserve from Assumptions \\ref{a1} and \\ref{a-storm} and the definition of $Q_\\rho$ in \\eqref{def-Q} that $Q_{\\rho_k}$ is $(\\bL_{\\nabla f}+\\rho_kL)$-smooth on $\\mcX$, where $L$ is defined in \\eqref{def-L}. Notice from Algorithm \\ref{alg1} that $x_k\\in\\mcX$ and $x_{k+1}= \\Pi_X(x_k-\\eta_k G_k)$, which imply that\n\\beq \\label{subprob-opt2}\n\\langle x_{k+1}-x_k+\\eta_k G_k,x_k-x_{k+1}\\rangle\\geq0\\quad\\Rightarrow\\quad \\langle G_k,x_{k+1}-x_k\\rangle\\leq-\\eta_k^{-1}\\|x_{k+1}-x_k\\|^2.\n\\eeq\nAlso, notice from Algorithm \\ref{alg1} and \\eqref{def-Q} that \n\\[\nG_k=g_k+\\rho_k\\nabla c(x_k)c(x_k), \\quad \\nabla Q_{\\rho_k}(x_k)=\\nabla f(x_k)+\\rho_k\\nabla c(x_k)c(x_k),\n\\] \nand hence $\\nabla Q_{\\rho_k}(x_k)-G_k=g_k-\\nabla f(x_k)$.  In addition, by Young's inequality,  one has \n\\beq \\label{young-ineq}\n\\langle\\nabla Q_{\\rho_k}(x_k)-G_k,x_{k+1}-x_k\\rangle \\leq \\frac{1}{2\\eta_k}\\|x_{k+1}-x_k\\|^2+\\frac{\\eta_k}{2}\\|\\nabla Q_{\\rho_k}(x_k)-G_k\\|^2\n\\eeq\nUsing the last two relations,  \\eqref{subprob-opt2}, and the $(\\bL_{\\nabla f}+\\rho_kL)$-smoothness of $Q_{\\rho_k}$, we obtain that\n\\begin{align*}\nQ_{\\rho_k}(x_{k+1})&\\ \\leq  Q_{\\rho_k}(x_k)+\\langle\\nabla Q_{\\rho_k}(x_k),x_{k+1}-x_k\\rangle+\\frac{1}{2}(\\bL_{\\nabla f}+\\rho_kL)\\|x_{k+1}-x_k\\|^2\\\\\n&\\ = Q_{\\rho_k}(x_k)+\\langle G_k,x_{k+1}-x_k\\rangle+\\langle\\nabla Q_{\\rho_k}(x_k)-G_k,x_{k+1}-x_k\\rangle+\\frac{1}{2}(\\bL_{\\nabla f}+\\rho_kL)\\|x_{k+1}-x_k\\|^2\\\\\n& \\overset{\\eqref{young-ineq}}{\\leq} Q_{\\rho_k}(x_k)+\\langle G_k,x_{k+1}-x_k\\rangle+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL+\\eta_k^{-1}\\right)\\|x_{k+1}-x_k\\|^2+\\frac{\\eta_k}{2}\\|\\nabla Q_{\\rho_k}(x_k)-G_k\\|^2\\\\\n&\\ \\leq Q_{\\rho_k}(x_k)+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL-\\eta_k^{-1}\\right)\\|x_{k+1}-x_k\\|^2+\\frac{\\eta_k}{2}\\|g_k-\\nabla f(x_k)\\|^2,\n\\end{align*}\nwhere the first inequality is due to the $(\\bL_{\\nabla f}+\\rho_kL)$-smoothness of $Q_{\\rho_k}$, and the last inequality follows from \\eqref{subprob-opt2} and the relation $\\nabla Q_{\\rho_k}(x_k)-G_k=g_k-\\nabla f(x_k)$. By this and \\eqref{def-Q}, we further have\n\\begin{align}\n&Q_{\\rho_{k+1}}(x_{k+1})\\leq Q_{\\rho_k}(x_k)+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL-\\eta_k^{-1}\\right)\\|x_{k+1}-x_k\\|^2+\\frac{\\eta_k}{2}\\|g_k-\\nabla f(x_k)\\|^2\\nn\\\\\n&\\qquad\\qquad\\qquad\\,\\,+Q_{\\rho_{k+1}}(x_{k+1})-Q_{\\rho_k}(x_{k+1})\\nn\\\\\n&\\overset{\\eqref{def-Q}}{=} Q_{\\rho_k}(x_k)+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL-\\eta_k^{-1}\\right)\\|x_{k+1}-x_k\\|^2+\\frac{\\eta_k}{2}\\|g_k-\\nabla f(x_k)\\|^2+\\frac{1}{2}(\\rho_{k+1}-\\rho_k)\\|c(x_{k+1})\\|^2.\\label{l5-e1}\n\\end{align}\n\nBy the definitions of $\\eta_k$, $\\alpha_k$, $\\nu$ and $\\zeta_k$ in \\eqref{def-para2}, \\eqref{def-nuC2} and \\eqref{def-zeta}, one has\n\\begin{align*}\n&\\ \\zeta_k-\\zeta_{k+1}(1-\\alpha_k)^2-\\eta_k=\\zeta_{k+1}\\alpha_k(2-\\alpha_k)-\\zeta_{k+1}+\\zeta_k-\\eta_k\\geq \\zeta_{k+1}\\alpha_k-\\zeta_{k+1}+\\zeta_k-\\eta_k\\\\\n&\\overset{\\eqref{def-para2}\\eqref{def-zeta}}\\geq(k+1)^\\nu k^{-2\\nu}/(2\\log 3)-(k+1)^\\nu/(2\\log 3)+k^\\nu/(2\\log 3)-k^{-\\nu}/(4\\log(k+2))\\\\\n&\\ \\ \\ \\geq k^{-\\nu}/(2\\log 3)-(k+1)^\\nu/(2\\log 3)+k^\\nu/(2\\log 3)-k^{-\\nu}/(4\\log(k+2))\\\\\n&\\ \\ \\ \\geq k^{-\\nu}/(2\\log 3)-\\nu k^{\\nu-1}/(2\\log 3)-k^{-\\nu}/(4\\log(k+2))\\\\\n&\\ \\ \\ = \\frac{k^{-\\nu}}{4\\log 3}\\left(2-2\\nu k^{2\\nu-1}-\\log 3/\\log(k+2)\\right) \\geq\\frac{k^{-\\nu}}{4\\log 3}\\left(2-1-\\log 3/\\log(k+2)\\right)\\geq0,\n\\end{align*}\nwhere the first inequality is due to  $0<\\alpha_k\\leq1$, the third inequality follows from $(k+1)^\\nu>k^\\nu$, the fourth inequality is due to $(k+1)^\\nu-k^\\nu\\leq \\nu k^{\\nu-1}$ thanks to the concavity of $t^\\nu$, the fifth inequality follows from $\\nu\\leq1/2$, and the last inequality is due to $k\\geq1$. Hence, we obtain\n\\[\n\\zeta_{k+1}(1-\\alpha_k)^2+\\eta_k\\leq\\zeta_k\\qquad \\forall k\\geq1.\n\\]\nUsing this, taking expectation on both sides of \\eqref{l5-e1}, and summing the resulting inequality with the inequality in Lemma \\ref{l-exp}, we obtain that\n\\begin{align}\n&\\bbE\\left[Q_{\\rho_{k+1}}(x_{k+1})+\\zeta_{k+1}\\|g_{k+1}-\\nabla f(x_{k+1})\\|^2\\right]\\nn\\\\\n&\\leq \\bbE\\left[Q_{\\rho_k}(x_k)+\\left(\\zeta_{k+1}(1-\\alpha_k)^2+\\eta_k\\right)\\|g_k-\\nabla f(x_k)\\|^2\\right]+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL-\\eta_k^{-1}+12\\zeta_{k+1}\\bL_{\\nabla f}^2\\right) \\nn \\\\ \n&\\quad\\times \\bbE\\left[\\|x_{k+1}-x_k\\|^2\\right] -\\frac{\\eta_k}{2}\\bbE\\left[\\|g_k-\\nabla f(x_k)\\|^2\\right]+\\frac{1}{2}(\\rho_{k+1}-\\rho_k)\\bbE\\left[\\|c(x_{k+1})\\|^2\\right]+3\\sigma^2\\zeta_{k+1}\\alpha_k^2\\nn\\\\\n&\\leq \\bbE\\left[Q_{\\rho_k}(x_k)+\\zeta_k\\|g_k-\\nabla f(x_k)\\|^2\\right]+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL-\\eta_k^{-1}+12\\zeta_{k+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{k+1}-x_k\\|^2\\right]\\nn\\\\\n&\\quad-\\frac{\\eta_k}{2}\\bbE\\left[\\|g_k-\\nabla f(x_k)\\|^2\\right]+\\frac{1}{2}(\\rho_{k+1}-\\rho_k)\\bbE\\left[\\|c(x_{k+1})\\|^2\\right]+3\\sigma^2\\zeta_{k+1}\\alpha_k^2.\\label{l5-e2}\n\\end{align}\nThe conclusion of this lemma follows by replacing $k$ with $i$ in the above inequalities and summing them up for all $1\\leq i\\leq k-1$.\n\\end{proof}","sha256_utf8":"91afd2bb7faa89b11307622828575e64ce950522a068ec81841e81af4786e460"},{"start":69679,"end":70284,"start_line":784,"end_line":790,"raw_tex":"\\begin{lemma}\\label{l-dist}\nSuppose that Assumptions \\ref{a1} and \\ref{a-storm} hold, and $\\{g_k\\}$ and  $\\{x_k\\}$ are generated by Algorithm \\ref{alg1}. Then for all $k\\geq1$, we have\n\\beq \\label{residual-bnd}\n\\dist^2\\left(0,\\nabla Q_{\\rho_k}(x_{k+1})+\\mcN_\\mcX(x_{k+1})\\right)\\leq3\\left(\\eta_k^{-2}+(\\bL_{\\nabla f}+\\rho_kL)^2\\right)\\|x_{k+1}-x_k\\|^2+3\\|g_k-\\nabla f(x_k)\\|^2,\n\\eeq\nwhere  $\\{\\rho_k\\}$ and $\\{\\eta_k\\}$ are given in Algorithm \\ref{alg1}, $\\bL_{\\nabla f}$ is given in Assumption \\ref{a-storm},  and $L$ and $Q_\\rho$ are defined in \\eqref{def-L} and \\eqref{def-Q}, respectively.\n\\end{lemma}","sha256_utf8":"b0e689ae921747d342a43acc0312ffd1d0cf44f9bb288d14819d125321628032"},{"start":70286,"end":71820,"start_line":792,"end_line":809,"raw_tex":"\\begin{proof}\nBy the expression of $x_{k+1}$ in Algorithm \\ref{alg1}, one has\n\\beq \\label{subprob_opt}\n0\\in x_{k+1}-x_k+\\eta_k G_k+\\mcN_\\mcX(x_{k+1})\\quad\\Rightarrow\\quad\\eta_k^{-1}(x_k-x_{k+1})-G_k\\in\\mcN_\\mcX(x_{k+1}).\n\\eeq\nNotice from the definition of $Q_\\rho$ in \\eqref{def-Q} that $\\nabla Q_{\\rho_k}(x)=\\nabla f(x)+\\rho_k\\nabla c(x)c(x)$, which together with \\eqref{def-L}, \\eqref{bL-smooth} and Assumption \\ref{a1}(iv) implies that $Q_{\\rho_k}$ is $(\\bL_{\\nabla f}+\\rho_kL)$-smooth on $\\mcX$. Using \\eqref{subprob_opt}, the expression of $\\nabla Q_{\\rho_k}$, and $G_k=g_k+\\rho_k\\nabla c(x_k)c(x_k)$ (see Algorithm \\ref{alg1}), we have\n\\[\n\\eta_k^{-1}(x_k-x_{k+1})+\\nabla f(x_k)-g_k-\\nabla Q_{\\rho_k}(x_k) = \\eta_k^{-1}(x_k-x_{k+1})-g_k-\\rho_k\\nabla c(x_k)c(x_k) \\in \\mcN_\\mcX(x_{k+1}).\n\\]\nBy this and the $(\\bL_{\\nabla f}+\\rho_kL)$-smoothness of $Q_{\\rho_k}$, one has\n\\begin{align*}\n&\\dist^2\\left(0,\\nabla Q_{\\rho_k}(x_{k+1})+\\mcN_\\mcX(x_{k+1})\\right) \\leq \\|\\nabla Q_{\\rho_k}(x_{k+1})\n+(\\eta_k^{-1}(x_k-x_{k+1})+\\nabla f(x_k)-g_k-\\nabla Q_{\\rho_k}(x_k))\\|^2\\\\\n&\\leq 3\\left(\\|\\nabla Q_{\\rho_k}(x_{k+1})-\\nabla Q_{\\rho_k}(x_k)\\|^2+\\eta_k^{-2}\\|x_{k+1}-x_k\\|^2+\\| g_k-\\nabla f(x_k)\\|^2\\right)\\\\\n&\\leq 3\\left(\\eta_k^{-2}+(\\bL_{\\nabla f}+\\rho_kL)^2\\right)\\|x_{k+1}-x_k\\|^2+3\\|g_k-\\nabla f(x_k)\\|^2,\n\\end{align*}\nwhere the second inequality follows from the convexity of $\\|\\cdot\\|^2$, and the last inequality is due to the $(\\bL_{\\nabla f}+\\rho_kL)$-smoothness of $Q_{\\rho_k}$. Hence, the conclusion of this lemma holds.\n\\end{proof}","sha256_utf8":"c0973958c1ec9a03d07cbbade6f0f664c1a532c387d0b801e600f9bbae4094f8"},{"start":71940,"end":81659,"start_line":813,"end_line":880,"raw_tex":"\\begin{proof}[\\textbf{Proof of Theorem \\ref{thm:storm}}]\nUsing \\eqref{def-para2}, \\eqref{def-tK2} and \\eqref{def-zeta}, we have that for all $i\\geq\\tK_1$,\n\\begin{align}\n\\bL_{\\nabla f}+\\rho_iL &\\overset{\\eqref{def-para2}}=\\bL_{\\nabla f}i^{-\\nu}\\eta_i^{-1}/(4\\log(i+2))+L\\eta_i^{-1}/(4\\log(i+2))\\nn\\\\\n&\\ \\leq\\bL_{\\nabla f}\\tK_1^{-\\nu}\\eta_i^{-1}/4+L\\eta_i^{-1}/(4\\log(\\tK_1+2))\\overset{\\eqref{def-tK2}}\\leq\\eta_i^{-1}/4,\\label{t1-e2}\\\\\n12\\zeta_{i+1}\\bL_{\\nabla f}^2& \\overset{\\eqref{def-para2}\\eqref{def-zeta}}=3\\bL_{\\nabla f}^2\\big((i+1)/i\\big)^\\nu\\eta_i^{-1}/\\big(2\\log(i+2)\\log 3\\big)\\nn\\\\\n&\\ \\ \\ \\leq3\\bL_{\\nabla f}^22^\\nu\\eta_i^{-1}/(2\\log(\\tK_1+2))\\overset{\\eqref{def-tK2}}\\leq\\eta_i^{-1}/4,\\nn\n\\end{align}\nwhich imply that \n\\beq\\label{t1-ineq2}\n\\bL_{\\nabla f}+\\rho_iL+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\leq\\eta_i^{-1}/2\\qquad\\forall i\\geq\\tK_1.\n\\eeq\nIt then follows from the proof of Lemma \\ref{l-pfunc} that \\eqref{l5-e2} holds. Using \\eqref{t1-ineq2} and rearranging the terms of \\eqref{l5-e2} with $k$ replaced by $i$, we obtain that for all $i\\geq\\tK_1$, \n\\begin{align}\n&\\frac{1}{4\\eta_i}\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\frac{\\eta_i}{2}\\bbE\\left[\\|g_i-\\nabla f(x_i)\\|^2\\right]\\nn\\\\\n&\\overset{\\eqref{l5-e2}}\\leq \\bbE\\left[Q_{\\rho_i}(x_i)+\\zeta_i\\|g_i-\\nabla f(x_i)\\|^2\\right]-\\bbE\\left[Q_{\\rho_{i+1}}(x_{i+1})+\\zeta_{i+1}\\|g_{i+1}-\\nabla f(x_{i+1})\\|^2\\right]\\nn\\\\\n&\\quad\\,\\,\\,+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_iL+12\\zeta_{i+1}\\bL_{\\nabla f}^2-\\eta_i^{-1}/2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\frac{1}{2}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]+3\\sigma^2\\zeta_{i+1}\\alpha_i^2\\nn\\\\\n&\\overset{\\eqref{t1-ineq2}}{\\leq}\\bbE\\left[Q_{\\rho_i}(x_i)+\\zeta_i\\|g_i-\\nabla f(x_i)\\|^2\\right]-\\bbE\\left[Q_{\\rho_{i+1}}(x_{i+1})+\\zeta_{i+1}\\|g_{i+1}-\\nabla f(x_{i+1})\\|^2\\right]\\nn\\\\\n&\\quad\\,\\,\\,+\\frac{1}{2}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]+3\\sigma^2\\zeta_{i+1}\\alpha_i^2.\\label{t1-p2}\n\\end{align}\nRecall that $\\iota_k$ is the random variable uniformly generated in $\\left\\{\\lceil k/2\\rceil+1,\\dots,k\\right\\}$. In addition, observe from \\eqref{def-para2} that $\\eta_i ^{-1}<\\eta_{k-1}^{-1}$ for all $\\lceil k/2\\rceil \\leq i \\leq k-1$. By these, \\eqref{def-Q}, \\eqref{def-para2}, \\eqref{penalty-bnd}, \\eqref{residual-bnd}, \\eqref{t1-e2} and \\eqref{t1-p2}, one has that for all $k\\geq2\\tK_1$,\n\\begin{align}\n&\\ \\bbE\\left[\\dist^2\\left(0,\\nabla f(x_{\\iota_k})+\\rho_{\\iota_k-1}\\nabla c(x_{\\iota_k})c(x_{\\iota_k})+\\mcN_\\mcX(x_{\\iota_k})\\right)\\right]=\\bbE\\left[\\dist^2\\left(0,\\nabla Q_{\\rho_{\\iota_k-1}}(x_{\\iota_k})+\\mcN_\\mcX(x_{\\iota_k})\\right)\\right]\\nn\\\\\n&\\ =\\frac{1}{k-\\lceil k/2\\rceil}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\bbE\\left[\\dist^2\\left(0,\\nabla Q_{\\rho_i}(x_{i+1})+\\mcN_\\mcX(x_{i+1})\\right)\\right]\\nn\\\\\n& \\overset{\\eqref{residual-bnd}}\\leq  \\frac{3}{k-\\lceil k/2\\rceil}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\left(\\left(\\eta_i^{-2}+(\\bL_{\\nabla f}+\\rho_iL)^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\bbE\\left[\\|g_i-\\nabla f(x_i)\\|^2\\right]\\right)\\nn\\\\\n& \\overset{\\eqref{t1-e2}}{\\leq} \\frac{3}{k-\\lceil k/2\\rceil}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\left((\\eta_i^{-2}+\\eta_i^{-2}/16)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\bbE\\left[\\|g_i-\\nabla f(x_i)\\|^2\\right]\\right)\\nn\\\\\n&\\ \\leq \\frac{51}{8(k-1)}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\left(\\eta_i^{-2}\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+2\\bbE\\left[\\|g_i-\\nabla f(x_i)\\|^2\\right]\\right)\\nn\\\\\n&\\ \\leq \\frac{51}{2(k-1)\\eta_{k-1}}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\left(\\frac{1}{4\\eta_i}\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\frac{\\eta_i}{2}\\bbE\\left[\\|g_i-\\nabla f(x_i)\\|^2\\right]\\right)\\nn\\\\\n& \\overset{\\eqref{t1-p2}}{\\leq}\\frac{51}{2(k-1)\\eta_{k-1}}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\Big(\\bbE\\left[Q_{\\rho_i}(x_i)+\\zeta_i\\|g_i-\\nabla f(x_i)\\|^2\\right]-\\bbE\\left[Q_{\\rho_{i+1}}(x_{i+1})+\\zeta_{i+1}\\|g_{i+1}-\\nabla f(x_{i+1})\\|^2\\right]\\nn\\\\\n&\\ \\quad\\,\\,\\,+\\frac{1}{2}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]+3\\sigma^2\\zeta_{i+1}\\alpha_i^2\\Big)\\nn\\\\\n&\\ = \\frac{51}{2(k-1)\\eta_{k-1}}\\Bigg(\\bbE\\left[Q_{\\rho_{\\lceil k/2\\rceil}}(x_{\\lceil k/2\\rceil})+\\zeta_{\\lceil k/2\\rceil}\\|g_{\\lceil k/2\\rceil}-\\nabla f(x_{\\lceil k/2\\rceil})\\|^2\\right]-\\bbE\\left[Q_{\\rho_k}(x_k)+\\zeta_k\\|g_k-\\nabla f(x_k)\\|^2\\right]\\nn\\\\\n&\\ \\quad\\,\\,\\,+\\frac{1}{2}\\sum_{i=\\lceil k/2\\rceil}^{k-1}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]+3\\sigma^2\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\zeta_{i+1}\\alpha_i^2\\Bigg)\\nn\\\\\n&\\ \\leq \\frac{51}{2(k-1)\\eta_{k-1}}\\Bigg(Q_1(x_1)-Q_1^*+\\zeta_1\\|g_1-\\nabla f(x_1)\\|^2+\\frac{1}{2}\\sum_{i=1}^{k-1}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]+3\\sigma^2\\sum_{i=1}^{k-1}\\zeta_{i+1}\\alpha_i^2\\nn\\\\\n&\\ \\quad\\,\\,\\,+\\frac{1}{2}\\sum_{i=1}^{\\lceil k/2\\rceil-1}\\left(\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]\\Bigg),\\label{t1-sum2}\n\\end{align}\nwhere the first equality is due to \\eqref{def-Q}, the first inequality follows from taking expectation on both sides of \\eqref{residual-bnd}, the third inequality is due to the fact that $\\lceil k/2\\rceil\\leq(k+1)/2$, the fourth inequality follows from the relation $\\eta_i ^{-1}<\\eta_{k-1}^{-1}$ for all $\\lceil k/2\\rceil \\leq i \\leq k-1$, and the last inequality follows from \\eqref{penalty-bnd} with $k$ replaced by $\\lceil k/2\\rceil$, $\\rho_1=1$, and the fact that $Q_{\\rho_k}(x_k)\\geq Q_1 (x_k) \\geq Q_1^*$.\n\nWe next bound each term in the summation \\eqref{t1-sum2}.  Indeed, it follows from \\eqref{cf-bnd}, Assumption \\ref{a1}(iv), the nonexpansiveness of $\\Pi_X$, and the expressions of $x_{k+1}$, $g_k$ and $G_k$ in Algorithm \\ref{alg1} that\n\\begin{align}\n\\|x_{k+1}-x_k\\|^2&=\\|\\Pi_\\mcX(x_k-\\eta_kG_k)-\\Pi_X(x_k)\\|^2\\leq\\eta_k^2\\|G_k\\|^2=\\eta_k^2\\|g_k+\\rho_k\\nabla c(x_k)c(x_k)\\|^2\\nn\\\\\n&\\leq2\\eta_k^2\\left(\\|g_k\\|^2+\\rho_k^2\\|\\nabla c(x_k)c(x_k)\\|^2\\right)\\leq2\\eta_k^2\\left(L_f^2+C_c^2L_c^2\\rho_k^2\\right).\\label{t1-x1}\n\\end{align}\nRecall that $\\nu=\\min\\{\\hat\\theta/(\\hat\\theta+2),1/2\\}$ for some $\\hat\\theta\\geq1$ and $\\|c(x_i)\\| \\leq C_c$ for all $i$. By this, Lemma \\ref{l-cnst2}, \\eqref{def-para2}, \\eqref{def-zeta},  \\eqref{t1-x1},  one has that for all $k\\geq2\\tK_1$, \n\\begin{align}\n&\\sum_{i=1}^{\\tK_1-1}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]\\leq C_c^2\\sum_{i=1}^{\\tK_1-1}\\big((i+1)^\\nu-i^\\nu\\big)=C_c^2\\big(\\tK_1^\\nu-1\\big),\\label{sum2-c1}\\\\\n&\\sum_{i=\\tK_1}^{k-1}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]\\leq2C_1\\sum_{i=\\tK_1}^{k-1}((i+1)^\\nu-i^\\nu)(i+1)^{-\\frac{2\\nu}{\\theta}}\\leq 2\\nu C_1\\sum_{i=\\tK_1}^{k-1}i^{\\nu-1}(i+1)^{-\\frac{2\\nu}{\\theta}} \\label{sum2-c2} \\\\\n&\\leq C_1\\sum_{i=\\tK_1}^{k-1}i^{-1}(i+1)^{\\nu-\\frac{2\\nu}{\\theta}}\\leq C_1\\max\\big\\{1,k^{\\nu-\\frac{2\\nu}{\\theta}}\\big\\}\\sum_{i=\\tK_1}^{k-1}i^{-1}\\leq C_1\\max\\big\\{1,k^{\\nu-\\frac{2\\nu}{\\theta}}\\big\\}(1+\\log k), \\label{sum2-c3} \\\\\n&\\sum_{i=1}^{k-1}\\zeta_{i+1}\\alpha_i^2\\overset{\\eqref{def-zeta}}=\\frac{1}{2\\log3}\\sum_{i=1}^{k-1}(i+1)^{\\nu}i^{-4\\nu}= \\frac{1}{2\\log3}\\sum_{i=1}^{k-1}i^{-3\\nu}((i+1)/i)^{\\nu} \\nn \\\\\n& \\leq\\frac{\\sqrt{2}}{2\\log3}\\sum_{i=1}^{k-1}i^{-3\\nu} \\leq\\frac{\\sqrt{2}}{2\\log3}\\sum_{i=1}^{k-1}i^{-1}\\leq\\frac{\\sqrt{2}(1+\\log k)}{2\\log3},\\label{sum2-c4a}\\\\\n&\\ \\sum_{i=1}^{\\lceil k/2\\rceil-1}\\left(\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]\\nn\\\\\n&\\ =\\sum_{i=1}^{\\tK_1-1}\\left(\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\sum_{i=\\tK_1}^{\\lceil k/2\\rceil-1}\\Big((\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2)\\nn \\\\ \n&\\ \\quad\\,\\,\\,\\times \\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]\\Big) \\nn \\\\\n&\\overset{\\eqref{t1-ineq2}}{\\leq}\\sum_{i=1}^{\\tK_1-1}\\left(\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right] \\nn\\\\\n&\\overset{\\eqref{t1-x1}}{\\leq}2\\sum_{i=1}^{\\tK_1-1}\\left(\\bL_{\\nabla f}+\\rho_iL+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\eta_i^2\\left(L_f^2+C_c^2L_c^2\\rho_i^2\\right)\\nn\\\\\n&\\overset{\\eqref{def-para2}}{\\leq}\\frac{1}{8}(1+\\log\\tK_1)\\big(\\bL_{\\nabla f}+\\tK_1^{\\frac{1}{2}}L+6\\tK_1^{\\frac{1}{2}}\\bL_{\\nabla f}^2\\big)(L_f^2+C_c^2L_c^2\\tK_1),\\label{sum2-c4}\n\\end{align}\nwhere the inequality in \\eqref{sum2-c1} follows from \\eqref{def-para2}, $\\|c(x_i)\\| \\leq C_c$ for all $i$, the inequalities in \\eqref{sum2-c2} are due to \\eqref{def-para2}, Lemma \\ref{l-cnst2},  and $(i+1)^\\nu-i^\\nu\\leq \\nu i^{\\nu-1}$ for all $i\\geq1$ thanks to the concavity of $t^\\nu$, the inequalities in \\eqref{sum2-c4a} are due to $((i+1)/i)^\\nu\\leq\\sqrt{2}$ for all $i\\geq1$ and $\\nu\\geq1/3$, and the last inequality in \\eqref{sum2-c4} is due to the relations $\\rho_i \\leq \\tK_1^{1/2}$ and $\\zeta_i \\leq \\tK_1^{1/2}/2$ for $1\\leq i\\leq \\tK_1-1$ and\n$\\sum_{i=1}^{\\tK_1-1}\\eta_i^2 \\leq \\sum_{i=1}^{\\tK_1-1}i^{-1}/16\\leq (1+\\log\\tK_1)/16$ thanks to $\\nu\\leq 1/2$ and the choice of $\\rho_i$, $\\eta_i$ and $\\zeta_i$ in \\eqref{def-para2} and \\eqref{def-zeta}.\n\nUsing \\eqref{def-para2}, \\eqref{def-zeta}, \\eqref{t1-sum2}, \\eqref{sum2-c1}, \\eqref{sum2-c3}, \\eqref{sum2-c4a} and \\eqref{sum2-c4}, we have\n\\begin{align*}\n&\\bbE\\left[\\dist^2\\left(0,\\nabla f(x_{\\iota_k})+\\rho_{\\iota_k-1}\\nabla c(x_{\\iota_k})c(x_{\\iota_k})+\\mcN_\\mcX(x_{\\iota_k})\\right)\\right]\\\\\n&\\leq \\frac{102\\log(k+1)}{(k-1)^{1-\\nu}}\\Bigg(Q_1(x_1)-Q_1^*+\\frac{\\|g_1-\\nabla f(x_1)\\|^2}{2\\log 3}+\\frac{1}{2}C_1\\max\\{1,k^{\\nu-\\frac{2\\nu}{\\theta}}\\}(1+\\log k)\\nn \\\\ \n&\\quad+\\frac{1}{2}C_c^2\\big(\\tK_1^{\\nu}-1\\big)\n+\\frac{3\\sqrt{2}\\sigma^2(1+\\log k)}{2\\log3}+\\frac{1}{16}(1+\\log\\tK_1)\\big(\\bL_{\\nabla f}+\\tK_1^{\\frac{1}{2}}L+6\\tK_1^{\\frac{1}{2}}\\bL_{\\nabla f}^2\\big)(L_f^2+C_c^2L_c^2\\tK_1)\\Bigg).\n\\end{align*}\nBy this, \\eqref{def-Q}, $\\iota_k>\\lceil k/2\\rceil\\geq\\tK_1$ for all $k\\geq2\\tK_1$, and Lemma \\ref{l-cnst2} with $k$ replaced by $\\iota_k$, one can see that Theorem \\ref{thm:storm} holds.\n\\end{proof}","sha256_utf8":"6c46f811a924e2f2c3aa41449d426732e11193027a676382c53f1c7f9a1596c4"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"递归动量法的可行性与驻点收敛率","source_units":[{"start":33416,"end":35160,"start_line":388,"end_line":407,"raw_tex":"\\begin{thm}\\label{thm:storm}\nSuppose that Assumptions \\ref{a1} and \\ref{a-storm} hold, and $\\{x_k\\}$ is generated by Algorithm \\ref{alg1}. Let $L$ be defined in \\eqref{def-L}, $L_f$, $\\bL_{\\nabla f}$, $L_c$, $C_c$, $\\sigma$,  $\\gamma$, $\\theta$ and $Q_1^*$ be given in Assumptions \\ref{a1} and \\ref{a-storm}, $g_1$ be given in Algorithm \\ref{alg1}, and $\\iota_k$  be the random variable uniformly generated from $\\left\\{\\lceil k/2\\rceil+1,\\dots,k\\right\\}$ for $k\\geq 2$, and let $\\rho_k$, $\\eta_k$ and $\\alpha_k$ be chosen as\n\\beq\\label{def-para2}\n\\rho_k=k^\\nu,\\quad\\eta_k=k^{-\\nu}/(4\\log(k+2)), \\quad \\alpha_k=k^{-2\\nu},\\quad\\mbox{where}\\quad \\nu=\\min\\{\\hat\\theta/(\\hat\\theta+2),1/2\\}\n\\eeq\nfor some $\\hat\\theta\\geq1$. Then for all $k\\geq2\\tK_1$, we have\n\\begin{align*}\n&\\bbE\\left[\\dist^2\\left(0,\\nabla f(x_{\\iota_k})+\\rho_{\\iota_k-1}\\nabla c(x_{\\iota_k})c(x_{\\iota_k})+\\mcN_\\mcX(x_{\\iota_k})\\right)\\right]\\\\\n&\\leq\\frac{102\\log(k+1)}{(k-1)^{1-\\nu}}\\Bigg(Q_1(x_1)-Q_1^*+\\frac{\\|g_1-\\nabla f(x_1)\\|^2}{2\\log 3}+\\frac{1}{2}C_1\\max\\{1,k^{\\nu-\\frac{2\\nu}{\\theta}}\\}(1+\\log k)\\nn \\\\ \n&\\quad+\\frac{1}{2}C_c^2\\big(\\tK_1^{\\nu}-1\\big)\n+\\frac{3\\sqrt{2}\\sigma^2(1+\\log k)}{2\\log3}+\\frac{1}{16}(1+\\log\\tK_1)\\big(\\bL_{\\nabla f}+\\tK_1^{\\frac{1}{2}}L+6\\tK_1^{\\frac{1}{2}}\\bL_{\\nabla f}^2\\big)(L_f^2+C_c^2L_c^2\\tK_1)\\Bigg),\\\\\n&\\|c(x_{\\iota_k})\\|^2\\leq2C_1 (k/2)^{-2\\nu/\\theta},\n\\end{align*}\nwhere\n\\begin{align}\n&\\tK_1=\\left\\lceil\\max\\left\\{(2\\bL_{\\nabla f})^{\\frac{1}{\\nu}},e^{6\\times2^{\\nu/2}\\bL_{\\nabla f}^2},e^{2L},e^{2\\theta},\\left(e^{-1}\\gamma^{-2}2^{6-\\theta}\\log(e^{2\\theta}+2)\\right)^{2\\theta}\\right\\}\\right\\rceil,\\label{def-tK2}\\\\\n&C_1=\\max\\left\\{1,\\tK_1^{2\\nu/\\theta}C_c^2/2,2^{3-\\theta}L_f^2\\gamma^{-2}\\right\\}.\\label{def-nuC2}\n\\end{align}\n\n\\end{thm}","sha256_utf8":"b385a2df7be2e986fb84da87541de704d4f99c8fb71a2a4173fd7e3733d0093b"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"主收敛定理的完整证明","source_units":[{"start":71940,"end":81659,"start_line":813,"end_line":880,"raw_tex":"\\begin{proof}[\\textbf{Proof of Theorem \\ref{thm:storm}}]\nUsing \\eqref{def-para2}, \\eqref{def-tK2} and \\eqref{def-zeta}, we have that for all $i\\geq\\tK_1$,\n\\begin{align}\n\\bL_{\\nabla f}+\\rho_iL &\\overset{\\eqref{def-para2}}=\\bL_{\\nabla f}i^{-\\nu}\\eta_i^{-1}/(4\\log(i+2))+L\\eta_i^{-1}/(4\\log(i+2))\\nn\\\\\n&\\ \\leq\\bL_{\\nabla f}\\tK_1^{-\\nu}\\eta_i^{-1}/4+L\\eta_i^{-1}/(4\\log(\\tK_1+2))\\overset{\\eqref{def-tK2}}\\leq\\eta_i^{-1}/4,\\label{t1-e2}\\\\\n12\\zeta_{i+1}\\bL_{\\nabla f}^2& \\overset{\\eqref{def-para2}\\eqref{def-zeta}}=3\\bL_{\\nabla f}^2\\big((i+1)/i\\big)^\\nu\\eta_i^{-1}/\\big(2\\log(i+2)\\log 3\\big)\\nn\\\\\n&\\ \\ \\ \\leq3\\bL_{\\nabla f}^22^\\nu\\eta_i^{-1}/(2\\log(\\tK_1+2))\\overset{\\eqref{def-tK2}}\\leq\\eta_i^{-1}/4,\\nn\n\\end{align}\nwhich imply that \n\\beq\\label{t1-ineq2}\n\\bL_{\\nabla f}+\\rho_iL+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\leq\\eta_i^{-1}/2\\qquad\\forall i\\geq\\tK_1.\n\\eeq\nIt then follows from the proof of Lemma \\ref{l-pfunc} that \\eqref{l5-e2} holds. Using \\eqref{t1-ineq2} and rearranging the terms of \\eqref{l5-e2} with $k$ replaced by $i$, we obtain that for all $i\\geq\\tK_1$, \n\\begin{align}\n&\\frac{1}{4\\eta_i}\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\frac{\\eta_i}{2}\\bbE\\left[\\|g_i-\\nabla f(x_i)\\|^2\\right]\\nn\\\\\n&\\overset{\\eqref{l5-e2}}\\leq \\bbE\\left[Q_{\\rho_i}(x_i)+\\zeta_i\\|g_i-\\nabla f(x_i)\\|^2\\right]-\\bbE\\left[Q_{\\rho_{i+1}}(x_{i+1})+\\zeta_{i+1}\\|g_{i+1}-\\nabla f(x_{i+1})\\|^2\\right]\\nn\\\\\n&\\quad\\,\\,\\,+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_iL+12\\zeta_{i+1}\\bL_{\\nabla f}^2-\\eta_i^{-1}/2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\frac{1}{2}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]+3\\sigma^2\\zeta_{i+1}\\alpha_i^2\\nn\\\\\n&\\overset{\\eqref{t1-ineq2}}{\\leq}\\bbE\\left[Q_{\\rho_i}(x_i)+\\zeta_i\\|g_i-\\nabla f(x_i)\\|^2\\right]-\\bbE\\left[Q_{\\rho_{i+1}}(x_{i+1})+\\zeta_{i+1}\\|g_{i+1}-\\nabla f(x_{i+1})\\|^2\\right]\\nn\\\\\n&\\quad\\,\\,\\,+\\frac{1}{2}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]+3\\sigma^2\\zeta_{i+1}\\alpha_i^2.\\label{t1-p2}\n\\end{align}\nRecall that $\\iota_k$ is the random variable uniformly generated in $\\left\\{\\lceil k/2\\rceil+1,\\dots,k\\right\\}$. In addition, observe from \\eqref{def-para2} that $\\eta_i ^{-1}<\\eta_{k-1}^{-1}$ for all $\\lceil k/2\\rceil \\leq i \\leq k-1$. By these, \\eqref{def-Q}, \\eqref{def-para2}, \\eqref{penalty-bnd}, \\eqref{residual-bnd}, \\eqref{t1-e2} and \\eqref{t1-p2}, one has that for all $k\\geq2\\tK_1$,\n\\begin{align}\n&\\ \\bbE\\left[\\dist^2\\left(0,\\nabla f(x_{\\iota_k})+\\rho_{\\iota_k-1}\\nabla c(x_{\\iota_k})c(x_{\\iota_k})+\\mcN_\\mcX(x_{\\iota_k})\\right)\\right]=\\bbE\\left[\\dist^2\\left(0,\\nabla Q_{\\rho_{\\iota_k-1}}(x_{\\iota_k})+\\mcN_\\mcX(x_{\\iota_k})\\right)\\right]\\nn\\\\\n&\\ =\\frac{1}{k-\\lceil k/2\\rceil}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\bbE\\left[\\dist^2\\left(0,\\nabla Q_{\\rho_i}(x_{i+1})+\\mcN_\\mcX(x_{i+1})\\right)\\right]\\nn\\\\\n& \\overset{\\eqref{residual-bnd}}\\leq  \\frac{3}{k-\\lceil k/2\\rceil}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\left(\\left(\\eta_i^{-2}+(\\bL_{\\nabla f}+\\rho_iL)^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\bbE\\left[\\|g_i-\\nabla f(x_i)\\|^2\\right]\\right)\\nn\\\\\n& \\overset{\\eqref{t1-e2}}{\\leq} \\frac{3}{k-\\lceil k/2\\rceil}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\left((\\eta_i^{-2}+\\eta_i^{-2}/16)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\bbE\\left[\\|g_i-\\nabla f(x_i)\\|^2\\right]\\right)\\nn\\\\\n&\\ \\leq \\frac{51}{8(k-1)}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\left(\\eta_i^{-2}\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+2\\bbE\\left[\\|g_i-\\nabla f(x_i)\\|^2\\right]\\right)\\nn\\\\\n&\\ \\leq \\frac{51}{2(k-1)\\eta_{k-1}}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\left(\\frac{1}{4\\eta_i}\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\frac{\\eta_i}{2}\\bbE\\left[\\|g_i-\\nabla f(x_i)\\|^2\\right]\\right)\\nn\\\\\n& \\overset{\\eqref{t1-p2}}{\\leq}\\frac{51}{2(k-1)\\eta_{k-1}}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\Big(\\bbE\\left[Q_{\\rho_i}(x_i)+\\zeta_i\\|g_i-\\nabla f(x_i)\\|^2\\right]-\\bbE\\left[Q_{\\rho_{i+1}}(x_{i+1})+\\zeta_{i+1}\\|g_{i+1}-\\nabla f(x_{i+1})\\|^2\\right]\\nn\\\\\n&\\ \\quad\\,\\,\\,+\\frac{1}{2}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]+3\\sigma^2\\zeta_{i+1}\\alpha_i^2\\Big)\\nn\\\\\n&\\ = \\frac{51}{2(k-1)\\eta_{k-1}}\\Bigg(\\bbE\\left[Q_{\\rho_{\\lceil k/2\\rceil}}(x_{\\lceil k/2\\rceil})+\\zeta_{\\lceil k/2\\rceil}\\|g_{\\lceil k/2\\rceil}-\\nabla f(x_{\\lceil k/2\\rceil})\\|^2\\right]-\\bbE\\left[Q_{\\rho_k}(x_k)+\\zeta_k\\|g_k-\\nabla f(x_k)\\|^2\\right]\\nn\\\\\n&\\ \\quad\\,\\,\\,+\\frac{1}{2}\\sum_{i=\\lceil k/2\\rceil}^{k-1}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]+3\\sigma^2\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\zeta_{i+1}\\alpha_i^2\\Bigg)\\nn\\\\\n&\\ \\leq \\frac{51}{2(k-1)\\eta_{k-1}}\\Bigg(Q_1(x_1)-Q_1^*+\\zeta_1\\|g_1-\\nabla f(x_1)\\|^2+\\frac{1}{2}\\sum_{i=1}^{k-1}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]+3\\sigma^2\\sum_{i=1}^{k-1}\\zeta_{i+1}\\alpha_i^2\\nn\\\\\n&\\ \\quad\\,\\,\\,+\\frac{1}{2}\\sum_{i=1}^{\\lceil k/2\\rceil-1}\\left(\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]\\Bigg),\\label{t1-sum2}\n\\end{align}\nwhere the first equality is due to \\eqref{def-Q}, the first inequality follows from taking expectation on both sides of \\eqref{residual-bnd}, the third inequality is due to the fact that $\\lceil k/2\\rceil\\leq(k+1)/2$, the fourth inequality follows from the relation $\\eta_i ^{-1}<\\eta_{k-1}^{-1}$ for all $\\lceil k/2\\rceil \\leq i \\leq k-1$, and the last inequality follows from \\eqref{penalty-bnd} with $k$ replaced by $\\lceil k/2\\rceil$, $\\rho_1=1$, and the fact that $Q_{\\rho_k}(x_k)\\geq Q_1 (x_k) \\geq Q_1^*$.\n\nWe next bound each term in the summation \\eqref{t1-sum2}.  Indeed, it follows from \\eqref{cf-bnd}, Assumption \\ref{a1}(iv), the nonexpansiveness of $\\Pi_X$, and the expressions of $x_{k+1}$, $g_k$ and $G_k$ in Algorithm \\ref{alg1} that\n\\begin{align}\n\\|x_{k+1}-x_k\\|^2&=\\|\\Pi_\\mcX(x_k-\\eta_kG_k)-\\Pi_X(x_k)\\|^2\\leq\\eta_k^2\\|G_k\\|^2=\\eta_k^2\\|g_k+\\rho_k\\nabla c(x_k)c(x_k)\\|^2\\nn\\\\\n&\\leq2\\eta_k^2\\left(\\|g_k\\|^2+\\rho_k^2\\|\\nabla c(x_k)c(x_k)\\|^2\\right)\\leq2\\eta_k^2\\left(L_f^2+C_c^2L_c^2\\rho_k^2\\right).\\label{t1-x1}\n\\end{align}\nRecall that $\\nu=\\min\\{\\hat\\theta/(\\hat\\theta+2),1/2\\}$ for some $\\hat\\theta\\geq1$ and $\\|c(x_i)\\| \\leq C_c$ for all $i$. By this, Lemma \\ref{l-cnst2}, \\eqref{def-para2}, \\eqref{def-zeta},  \\eqref{t1-x1},  one has that for all $k\\geq2\\tK_1$, \n\\begin{align}\n&\\sum_{i=1}^{\\tK_1-1}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]\\leq C_c^2\\sum_{i=1}^{\\tK_1-1}\\big((i+1)^\\nu-i^\\nu\\big)=C_c^2\\big(\\tK_1^\\nu-1\\big),\\label{sum2-c1}\\\\\n&\\sum_{i=\\tK_1}^{k-1}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]\\leq2C_1\\sum_{i=\\tK_1}^{k-1}((i+1)^\\nu-i^\\nu)(i+1)^{-\\frac{2\\nu}{\\theta}}\\leq 2\\nu C_1\\sum_{i=\\tK_1}^{k-1}i^{\\nu-1}(i+1)^{-\\frac{2\\nu}{\\theta}} \\label{sum2-c2} \\\\\n&\\leq C_1\\sum_{i=\\tK_1}^{k-1}i^{-1}(i+1)^{\\nu-\\frac{2\\nu}{\\theta}}\\leq C_1\\max\\big\\{1,k^{\\nu-\\frac{2\\nu}{\\theta}}\\big\\}\\sum_{i=\\tK_1}^{k-1}i^{-1}\\leq C_1\\max\\big\\{1,k^{\\nu-\\frac{2\\nu}{\\theta}}\\big\\}(1+\\log k), \\label{sum2-c3} \\\\\n&\\sum_{i=1}^{k-1}\\zeta_{i+1}\\alpha_i^2\\overset{\\eqref{def-zeta}}=\\frac{1}{2\\log3}\\sum_{i=1}^{k-1}(i+1)^{\\nu}i^{-4\\nu}= \\frac{1}{2\\log3}\\sum_{i=1}^{k-1}i^{-3\\nu}((i+1)/i)^{\\nu} \\nn \\\\\n& \\leq\\frac{\\sqrt{2}}{2\\log3}\\sum_{i=1}^{k-1}i^{-3\\nu} \\leq\\frac{\\sqrt{2}}{2\\log3}\\sum_{i=1}^{k-1}i^{-1}\\leq\\frac{\\sqrt{2}(1+\\log k)}{2\\log3},\\label{sum2-c4a}\\\\\n&\\ \\sum_{i=1}^{\\lceil k/2\\rceil-1}\\left(\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]\\nn\\\\\n&\\ =\\sum_{i=1}^{\\tK_1-1}\\left(\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\sum_{i=\\tK_1}^{\\lceil k/2\\rceil-1}\\Big((\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2)\\nn \\\\ \n&\\ \\quad\\,\\,\\,\\times \\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]\\Big) \\nn \\\\\n&\\overset{\\eqref{t1-ineq2}}{\\leq}\\sum_{i=1}^{\\tK_1-1}\\left(\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right] \\nn\\\\\n&\\overset{\\eqref{t1-x1}}{\\leq}2\\sum_{i=1}^{\\tK_1-1}\\left(\\bL_{\\nabla f}+\\rho_iL+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\eta_i^2\\left(L_f^2+C_c^2L_c^2\\rho_i^2\\right)\\nn\\\\\n&\\overset{\\eqref{def-para2}}{\\leq}\\frac{1}{8}(1+\\log\\tK_1)\\big(\\bL_{\\nabla f}+\\tK_1^{\\frac{1}{2}}L+6\\tK_1^{\\frac{1}{2}}\\bL_{\\nabla f}^2\\big)(L_f^2+C_c^2L_c^2\\tK_1),\\label{sum2-c4}\n\\end{align}\nwhere the inequality in \\eqref{sum2-c1} follows from \\eqref{def-para2}, $\\|c(x_i)\\| \\leq C_c$ for all $i$, the inequalities in \\eqref{sum2-c2} are due to \\eqref{def-para2}, Lemma \\ref{l-cnst2},  and $(i+1)^\\nu-i^\\nu\\leq \\nu i^{\\nu-1}$ for all $i\\geq1$ thanks to the concavity of $t^\\nu$, the inequalities in \\eqref{sum2-c4a} are due to $((i+1)/i)^\\nu\\leq\\sqrt{2}$ for all $i\\geq1$ and $\\nu\\geq1/3$, and the last inequality in \\eqref{sum2-c4} is due to the relations $\\rho_i \\leq \\tK_1^{1/2}$ and $\\zeta_i \\leq \\tK_1^{1/2}/2$ for $1\\leq i\\leq \\tK_1-1$ and\n$\\sum_{i=1}^{\\tK_1-1}\\eta_i^2 \\leq \\sum_{i=1}^{\\tK_1-1}i^{-1}/16\\leq (1+\\log\\tK_1)/16$ thanks to $\\nu\\leq 1/2$ and the choice of $\\rho_i$, $\\eta_i$ and $\\zeta_i$ in \\eqref{def-para2} and \\eqref{def-zeta}.\n\nUsing \\eqref{def-para2}, \\eqref{def-zeta}, \\eqref{t1-sum2}, \\eqref{sum2-c1}, \\eqref{sum2-c3}, \\eqref{sum2-c4a} and \\eqref{sum2-c4}, we have\n\\begin{align*}\n&\\bbE\\left[\\dist^2\\left(0,\\nabla f(x_{\\iota_k})+\\rho_{\\iota_k-1}\\nabla c(x_{\\iota_k})c(x_{\\iota_k})+\\mcN_\\mcX(x_{\\iota_k})\\right)\\right]\\\\\n&\\leq \\frac{102\\log(k+1)}{(k-1)^{1-\\nu}}\\Bigg(Q_1(x_1)-Q_1^*+\\frac{\\|g_1-\\nabla f(x_1)\\|^2}{2\\log 3}+\\frac{1}{2}C_1\\max\\{1,k^{\\nu-\\frac{2\\nu}{\\theta}}\\}(1+\\log k)\\nn \\\\ \n&\\quad+\\frac{1}{2}C_c^2\\big(\\tK_1^{\\nu}-1\\big)\n+\\frac{3\\sqrt{2}\\sigma^2(1+\\log k)}{2\\log3}+\\frac{1}{16}(1+\\log\\tK_1)\\big(\\bL_{\\nabla f}+\\tK_1^{\\frac{1}{2}}L+6\\tK_1^{\\frac{1}{2}}\\bL_{\\nabla f}^2\\big)(L_f^2+C_c^2L_c^2\\tK_1)\\Bigg).\n\\end{align*}\nBy this, \\eqref{def-Q}, $\\iota_k>\\lceil k/2\\rceil\\geq\\tK_1$ for all $k\\geq2\\tK_1$, and Lemma \\ref{l-cnst2} with $k$ replaced by $\\iota_k$, one can see that Theorem \\ref{thm:storm} holds.\n\\end{proof}","sha256_utf8":"6c46f811a924e2f2c3aa41449d426732e11193027a676382c53f1c7f9a1596c4"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"确定约束随机优化问题","source_units":[{"start":7947,"end":8077,"start_line":212,"end_line":217,"raw_tex":"\\beq\\label{prob}\n\\begin{aligned}\n\\min_{x\\in\\mcX}&\\quad f(x) :=\\bbE[\\tilde f(x,\\xi)]\\\\\n\\mbox{s.t.}&\\quad c(x)=0,\n\\end{aligned}\n\\eeq","sha256_utf8":"af7d8f91179b89c4bd665ec101333cdcde087813ff0282f4708a65857de5f843"},{"start":8078,"end":9312,"start_line":218,"end_line":218,"raw_tex":"where $\\xi$ is a random variable with sample space $\\Xi$, $\\tf(\\cdot,\\xi)$ is continuously differentiable for each $\\xi\\in\\Xi$,  $c:\\rr^n\\to\\rr^m$ is a deterministic smooth mapping, and $\\mcX\\subseteq \\rr^n$ is a simple closed convex set whose projection operator can be evaluated exactly.\\footnote{For simplicity, we focus on problem \\eqref{prob} with equality constraints only. However, our results can be directly extended to problems involving both equality and inequality constraints; see the concluding remarks for further details.} Problem \\eqref{prob} arises in a variety of important areas, including energy systems \\cite{shabazbegian2020stochastic}, healthcare \\cite{shehadeh2022stochastic}, image processing \\cite{luke2020proximal}, machine learning \\cite{cuomo2022scientific,karniadakis2021physics},  network optimization \\cite{bertsekasnetwork}, optimal control \\cite{betts2010practical}, PDE-constrained optimization \\cite{rees2010optimal}, resource allocation \\cite{fontaine2020adaptive}, and transportation \\cite{maggioni2009stochastic}. More applications can be found, for example, in \\cite{bottou2018optimization,combettes2011proximal,jiang2018consensus,kushner2012stochastic,liu2015sparse}, and references therein.\n","sha256_utf8":"be30b6be18299b8c44d3a1d609f01275bcdfe127ea8c66dbda8212ccd5be3006"}]},{"id":"S004","semantic_type":"notation","kind":"notation","title":"范数、投影与法锥记号","source_units":[{"start":20095,"end":21404,"start_line":266,"end_line":269,"raw_tex":"The following notation will be used throughout this paper.  Let $\\rr_{>0}$ denote the set of positive real numbers, and  $\\rr^n$ denote the Euclidean space of dimension $n$. The standard inner product and Euclidean norm are denoted by $\\langle\\cdot,\\cdot\\rangle$ and $\\|\\cdot\\|$, respectively.  For any $r>0$, let $\\cB(r)$ represent the Euclidean ball centered at the origin with radius $r$, that is, $\\cB(r)=\\{x:\\|x\\|\\leq r\\}$.  For any $t\\in\\rr$, let $t_+=\\max\\{t,0\\}$ and $\\lceil t \\rceil$ denote the least integer greater than or equal to $t$.\n\nA mapping $\\phi$ is said to be \\emph{$L_{\\phi}$-Lipschitz continuous} on a set $\\Omega$ if $\\|\\phi(x)-\\phi(x')\\| \\leq L_{\\phi} \\|x-x'\\|$ for all $x,x'\\in \\Omega$. Also, it is said to be \\emph{$L_{\\nabla\\phi}$-smooth} on $\\Omega$ if $\\|\\nabla\\phi(x)-\\nabla\\phi(x')\\| \\leq L_{\\nabla\\phi} \\|x-x'\\|$ for all $x,x'\\in \\Omega$, where $\\nabla \\phi$ denotes the transpose of the Jacobian of $\\phi$.  Given a nonempty closed convex set $\\cC$, $\\dist(x,\\cC)$ denotes the Euclidean distance from $x$ to $\\cC$, and $\\Pi_\\cC(x)$ denotes the Euclidean projection of $x$ onto $\\Omega$.  In addition,  the normal cone of $\\cC$ at any $x\\in \\cC$ is denoted by $\\mcN_\\cC(x)$.\nFinally, we use $\\tmcO(\\cdot)$ to denote the asymptotic upper bound that ignores logarithmic factors.\n","sha256_utf8":"d58b2059ef7321bca15a57b2027903b18a528a94d6cebee62e7947c2b1ce223f"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"二次罚函数","source_units":[{"start":12922,"end":12993,"start_line":234,"end_line":236,"raw_tex":"\\beq\\label{def-Q} \nQ_\\rho(x) := f(x) + \\frac{\\rho}{2} \\|c(x)\\|^2. \n\\eeq","sha256_utf8":"087a5eb1aaaa66d1c101fe40aa7cf3eb7dfc465dd5cf61ed14f24c7406c2605c"}]},{"id":"S006","semantic_type":"assumption","kind":"assumption","title":"有界性、随机梯度与误差界假设","source_units":[{"start":21489,"end":22424,"start_line":273,"end_line":287,"raw_tex":"\\begin{assumption}\\label{a1}\n\\begin{enumerate}[label=(\\roman*)]\n\\item The optimal value $f^*$ of problem \\eqref{prob}  and $Q_1^*:= \\min_{x\\in X} \\{f(x)+\\|c(x)\\|^2/2\\}$ are finite.\n\\item $f$ is differentiable and $L_f$-Lipschitz continuous in an open neighborhood of $\\mcX$.\n\\item For each $\\xi\\in\\Xi$, $\\tf(\\cdot,\\xi)$ is differentiable on $\\mcX$ and satisfies the following conditions:\n\\[\n\\bbE[\\nabla\\tf(x,\\xi)]=\\nabla f(x),\\quad\\bbE[\\|\\nabla\\tf(x,\\xi)-\\nabla f(x)\\|^2]\\leq\\sigma^2 \\qquad \\forall x\\in\\mcX\n\\]\nfor some constant $\\sigma \\geq 0$. \n\\item The mapping $c$ is $L_c$-Lipschitz continuous and $L_{\\nabla c}$-smooth in an open neighborhood of $\\mcX$. Additionally, $\\|c(x)\\|\\leq C_c$ for all $x\\in\\mcX$, and there exist constants $\\gamma>0$ and $\\theta\\geq1$ such that\n\\beq \\label{err-cond}\n\\dist\\left(0,\\nabla c(x)c(x)+\\mcN_\\mcX(x)\\right)\\geq\\gamma\\|c(x)\\|^\\theta\\qquad \\forall x\\in\\mcX.\n\\eeq\n\\end{enumerate}\n\\end{assumption}","sha256_utf8":"30a6a3eb4c78ce235c3acc54b1483f5c584c5196a11e417f0cf4eda8edc95370"}]},{"id":"S007","semantic_type":"derived_fact","kind":"derived_fact","title":"罚项平滑常数与梯度界","source_units":[{"start":22477,"end":22542,"start_line":290,"end_line":292,"raw_tex":"\\begin{align}\nL:= L_c^2+C_cL_{\\nabla c}.\\label{def-L}\n\\end{align}","sha256_utf8":"1bf2799092977746cdcf8c3b350c181760ac209b6f0dcaab20419685c01ca15c"},{"start":22543,"end":22737,"start_line":293,"end_line":296,"raw_tex":"It follows from this and Assumption \\ref{a1} that $\\|c(x)\\|^2/2$ is $L$-smooth on $\\mcX$, and \n\\beq\\label{cf-bnd}\n\\|\\nabla f(x)\\|\\leq L_f,\\quad\\|\\nabla c(x)\\|\\leq L_c\\qquad\\forall x\\in\\mcX.\n\\eeq","sha256_utf8":"cfbf5c3ce0d0cf9af78168b4e7eab3fa3542bb360d59352553494de93928f853"}]},{"id":"S008","semantic_type":"algorithm","kind":"algorithm","title":"截断递归动量算法","source_units":[{"start":26750,"end":27409,"start_line":329,"end_line":340,"raw_tex":"\\begin{algorithm}[H]\n\\caption{A stochastic first-order method with a truncated  recursive momentum for problem \\eqref{prob}}\\label{alg1}\n\\begin{algorithmic}[1]\n\\REQUIRE $x_1\\in\\mcX$, $\\{\\alpha_k\\} \\subset (0,1]$,  $\\{\\rho_k\\}, \\{\\eta_k\\} \\subset \\rr_{>0}$, and $L_f$ given in Assumption \\ref{a1}.\n\\STATE Sample $\\xi_1$ and set $g_1=\\Pi_{\\cB(L_f)}(\\nabla\\tf(x_1,\\xi_1))$.\n\\FOR{$k=1,2,\\dots$}\n\\STATE $G_k=g_k+\\rho_k\\nabla c(x_k)c(x_k)$.\n\\STATE $x_{k+1}=\\Pi_\\mcX(x_k-\\eta_kG_k)$.\n\\STATE Sample $\\xi_{k+1}$ and set $g_{k+1}=\\Pi_{\\cB(L_f)}\\big(\\nabla\\tf(x_{k+1},\\xi_{k+1})+(1-\\alpha_k)(g_k-\\nabla\\tf(x_k,\\xi_{k+1}))\\big)$.\n\\ENDFOR\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"a0cac5052dd0f147dfba22ff78da346d0d4237674e46c08c8d3abf8567725ac6"}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","title":"随机梯度平均平滑假设","source_units":[{"start":31890,"end":32119,"start_line":367,"end_line":372,"raw_tex":"\\begin{assumption}\\label{a-storm}\nThe function $\\tf(x,\\xi)$ satisfies the average smoothness condition:\n\\[\n\\bbE[ \\|\\nabla\\tf(u,\\xi)-\\nabla\\tf(v,\\xi)\\|^2]\\leq \\bL_{\\nabla f}^2\\|u-v\\|^2\\qquad \\forall u,v\\in\\mcX.\n\\]\n\\end{assumption}","sha256_utf8":"b7bfb42161432c2adc1b255d19a7777200da482055a5fc69637df313561fccf7"}]},{"id":"S010","semantic_type":"derived_fact","kind":"derived_fact","title":"目标梯度的确定平滑性","source_units":[{"start":32446,"end":32674,"start_line":375,"end_line":378,"raw_tex":"It can be observed that Assumption \\ref{a-storm} implies that $\\nabla f$ is $\\bL_{\\nabla f}$-smooth on $X$, that is,\n\\beq \\label{bL-smooth}\n \\|\\nabla f(u)-\\nabla f(v)\\| \\leq \\bL_{\\nabla f} \\|u-v\\| \\qquad \\forall u,v\\in\\mcX.\n\\eeq","sha256_utf8":"1ce96b6e2ab83e61b5ee3a147e8b8de9336f468442b39eb61015f819fa483b68"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"可行性函数及其平滑性","source_units":[{"start":50709,"end":50762,"start_line":553,"end_line":555,"raw_tex":"\\beq\\label{def-h}\nh(x) := \\frac{1}{2}\\|c(x)\\|^2.\n\\eeq","sha256_utf8":"a8e2100c73cabd22500a46e665ccf24fe9f8a3cc2b5f2f487eef9ae3febbf9b3"},{"start":50763,"end":50880,"start_line":556,"end_line":556,"raw_tex":"One can observe from Assumption \\ref{a1}(iv) that $h$ is $L$-smooth on $\\mcX$, where $L$ is given in  \\eqref{def-L}.\n","sha256_utf8":"be3046b8f295ed5246de1aff44b25d8a7a6e5957cc35edfda9cc81a27335dadc"}]},{"id":"S012","semantic_type":"lemma","kind":"lemma","title":"可行性函数的单步递推","source_units":[{"start":51021,"end":51538,"start_line":560,"end_line":566,"raw_tex":"\\begin{lemma}\\label{l-rec} \nSuppose that Assumption \\ref{a1} holds, and $x_{k+1}$ is generated by Algorithm \\ref{alg1} for some $k\\geq1$ with $\\rho_k\\eta_k\\leq(\\sqrt{5}-1)/(2L)$.  Then we have\n\\[\nh(x_{k+1})+2^{\\theta-2}\\gamma^2\\rho_k\\eta_k[h(x_{k+1})]^{\\theta}\\leq h(x_k)+L_f^2\\rho_k^{-1}\\eta_k/2,\n\\]\nwhere $\\rho_k$ and $\\eta_k$ are given in Algorithm \\ref{alg1}, $L_f$, $\\gamma$ and $\\theta$ are given in Assumption \\ref{a1},  and $L$ and $h$ are defined in \\eqref{def-L} and \\eqref{def-h}, respectively.\n\\end{lemma}","sha256_utf8":"5b155be6ba769d4c43bc6786448421bb202ddd1fc1f77014044055b2a8a5d3df"}]},{"id":"S013","semantic_type":"proof","kind":"proof","title":"单步可行性递推的证明","source_units":[{"start":51540,"end":55199,"start_line":568,"end_line":618,"raw_tex":"\\begin{proof}\nLet $G_k$ be given in Algorithm \\ref{alg1}. For convenience, we define\n\\beq\\label{l3-e1}\n\\widetilde G_k=\\rho_k^{-1}G_k,\\quad\\tilde\\eta_k=\\rho_k\\eta_k.\n\\eeq\n It then follows from these, \\eqref{def-h}, and the expression of $x_{k+1}$ in Algorithm \\ref{alg1} that\n\\beq\\label{l3-x}\nx_{k+1}=\\Pi_\\mcX(x_k-\\eta_kG_k)=\\Pi_\\mcX(x_k-\\tilde\\eta_k\\widetilde G_k),\n\\eeq\nwhich implies that\n\\beq \\label{subprob-opt1}\n0\\in x_{k+1}-x_k+\\tilde\\eta_k\\widetilde G_k+\\mcN_\\mcX(x_{k+1})\\quad\\Rightarrow\\quad \\nabla h(x_{k+1})+\\tilde\\eta_k^{-1}(x_k-x_{k+1})-\\widetilde G_k\\in\\nabla h(x_{k+1})+\\mcN_\\mcX(x_{k+1}).\n\\eeq\nUsing this, \\eqref{def-h} and Assumption \\ref{a1}(iv), we have\n\\begin{align}\n2^\\theta\\gamma^2[h(x_{k+1})]^\\theta&\\ =\\gamma^2\\|c(x_{k+1})\\|^{2\\theta}\\leq\\dist^2\\left(0,\\nabla c(x_{k+1})c(x_{k+1})+\\mcN_\\mcX(x_{k+1})\\right) \\nn \\\\\n&\\overset{\\eqref{def-h}}{=}\\dist^2\\left(0,\\nabla h(x_{k+1})+\\mcN_\\mcX(x_{k+1})\\right)\\overset{\\eqref{subprob-opt1}}\\leq\\|\\nabla h(x_{k+1})+\\tilde\\eta_k^{-1}(x_k-x_{k+1})-\\widetilde G_k\\|^2\\nn\\\\\n&\\ \\leq2\\|\\tilde\\eta_k^{-1}(x_k-x_{k+1})+\\nabla h(x_k)-\\widetilde G_k\\|^2+2\\|\\nabla h(x_{k+1})-\\nabla h(x_k)\\|^2\\nn\\\\\n&\\ =2\\tilde\\eta_k^{-2}\\|x_{k+1}-x_k\\|^2+4\\tilde\\eta_k^{-1}\\langle\\widetilde G_k-\\nabla h(x_k), x_{k+1}-x_k\\rangle+2\\|\\widetilde G_k-\\nabla h(x_k)\\|^2\\nn\\\\\n&\\ \\quad +2\\|\\nabla h(x_{k+1})-\\nabla h(x_k)\\|^2\\nn\\\\\n&\\ \\leq2(\\tilde\\eta_k^{-2}+L^2)\\|x_{k+1}-x_k\\|^2+4\\tilde\\eta_k^{-1}\\langle\\widetilde G_k-\\nabla h(x_k), x_{k+1}-x_k\\rangle+2\\|\\widetilde G_k-\\nabla h(x_k)\\|^2,\\label{l3-e2}\n\\end{align}\nwhere the first inequality follows from Assumption \\ref{a1}(iv), the second inequality is due to the convexity of $\\|\\cdot\\|^2$, and the last inequality follows from the $L$-smoothness of $h$. In addition,  by \\eqref{l3-x} and $x_k\\in\\mcX$, one has\n\\[\n\\langle x_{k+1}-x_k+\\tilde\\eta_k\\widetilde G_k,x_k-x_{k+1}\\rangle\\geq0\\quad\\Rightarrow\\quad \\langle \\widetilde G_k,x_{k+1}-x_k\\rangle\\leq-\\tilde\\eta_k^{-1}\\|x_{k+1}-x_k\\|^2.\n\\]\nThis together with the $L$-smoothness of $h$ yields\n\\begin{align*}\nh(x_{k+1})& \\leq h(x_k)+\\langle\\nabla h(x_k),x_{k+1}-x_k\\rangle +\\frac{L}{2}\\|x_{k+1}-x_k\\|^2\\nn\\\\\n&=h(x_k)+\\langle\\widetilde G_k,x_{k+1}-x_k\\rangle+\\langle\\nabla h(x_k)-\\widetilde G_k,x_{k+1}-x_k\\rangle +\\frac{L}{2}\\|x_{k+1}-x_k\\|^2\\nn\\\\\n& \\leq h(x_k)-\\tilde\\eta_k^{-1}\\|x_{k+1}-x_k\\|^2+\\langle\\nabla h(x_k)-\\widetilde G_k,x_{k+1}-x_k\\rangle +\\frac{L}{2}\\|x_{k+1}-x_k\\|^2.\n\\end{align*}\nUsing this and \\eqref{l3-e2}, we obtain that\n\\beq\\label{l3-e3}\nh(x_{k+1})+2^{\\theta-2}\\gamma^2\\tilde\\eta_k[h(x_{k+1})]^{\\theta}\\leq h(x_k)+\\frac{1}{2}(L^2\\tilde\\eta_k-\\tilde\\eta_k^{-1}+L)\\|x_{k+1}-x_k\\|^2+\\frac{\\tilde\\eta_k}{2}\\|\\widetilde G_k-\\nabla h(x_k)\\|^2.\n\\eeq\nObserve from \\eqref{l3-e1} and $\\rho_k\\eta_k\\leq(\\sqrt{5}-1)/(2L)$ that $L\\tilde\\eta_k=L\\rho_k\\eta_k\\leq(\\sqrt{5}-1)/2$, which implies that\n\\beq\\label{l3-e4}\nL^2\\tilde\\eta_k-\\tilde\\eta_k^{-1}+L=\\tilde\\eta_k^{-1}( L^2\\tilde\\eta_k^2+L\\tilde\\eta_k-1)\\leq0.\n\\eeq\nNotice from the expression of $g_k$ in Algorithm \\ref{alg1} that $g_k\\in\\cB(L_f)$ and hence $\\|g_k\\|\\leq L_f$. \nAlso, observe from Algorithm \\ref{alg1} and \\eqref{def-h} that $G_k=g_k+\\rho_k\\nabla h(x_k)$. Using these and \\eqref{l3-e1}, we have\n\\[\n\\|\\widetilde G_k-\\nabla h(x_k)\\|= \\|\\rho_k^{-1}G_k-\\nabla h(x_k)\\| =\\|\\rho_k^{-1}(g_k+\\rho_k\\nabla h(x_k))-\\nabla h(x_k)\\|= \\rho_k^{-1}\\|g_k\\|\\leq\\rho_k^{-1}L_f.\n\\]\nIt then follows from this, \\eqref{l3-e3} and \\eqref{l3-e4} that \n\\[\nh(x_{k+1})+2^{\\theta-2}\\gamma^2\\tilde\\eta_k[h(x_{k+1})]^\\theta\\leq h(x_k)+L_f^2\\tilde\\eta_k/(2\\rho_k^2).\n\\]\nThis and the definition of $\\tilde\\eta_k$ in \\eqref{l3-e1} imply that the conclusion of this lemma holds.\n\\end{proof}","sha256_utf8":"a57ebcca7735da14977af170510ce9ab15bdefb3cdd28ff09d780445f800d672"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"缩放更新并写出投影最优性","source_units":[{"start":51540,"end":52149,"start_line":568,"end_line":580,"raw_tex":"\\begin{proof}\nLet $G_k$ be given in Algorithm \\ref{alg1}. For convenience, we define\n\\beq\\label{l3-e1}\n\\widetilde G_k=\\rho_k^{-1}G_k,\\quad\\tilde\\eta_k=\\rho_k\\eta_k.\n\\eeq\n It then follows from these, \\eqref{def-h}, and the expression of $x_{k+1}$ in Algorithm \\ref{alg1} that\n\\beq\\label{l3-x}\nx_{k+1}=\\Pi_\\mcX(x_k-\\eta_kG_k)=\\Pi_\\mcX(x_k-\\tilde\\eta_k\\widetilde G_k),\n\\eeq\nwhich implies that\n\\beq \\label{subprob-opt1}\n0\\in x_{k+1}-x_k+\\tilde\\eta_k\\widetilde G_k+\\mcN_\\mcX(x_{k+1})\\quad\\Rightarrow\\quad \\nabla h(x_{k+1})+\\tilde\\eta_k^{-1}(x_k-x_{k+1})-\\widetilde G_k\\in\\nabla h(x_{k+1})+\\mcN_\\mcX(x_{k+1}).\n\\eeq\n","sha256_utf8":"2e279180174a590d6b3133e6a7720ffb0b0418ee08a06f0a7c8a1e8580e5a767"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"误差界控制残差并约束内积","source_units":[{"start":52149,"end":53512,"start_line":581,"end_line":593,"raw_tex":"Using this, \\eqref{def-h} and Assumption \\ref{a1}(iv), we have\n\\begin{align}\n2^\\theta\\gamma^2[h(x_{k+1})]^\\theta&\\ =\\gamma^2\\|c(x_{k+1})\\|^{2\\theta}\\leq\\dist^2\\left(0,\\nabla c(x_{k+1})c(x_{k+1})+\\mcN_\\mcX(x_{k+1})\\right) \\nn \\\\\n&\\overset{\\eqref{def-h}}{=}\\dist^2\\left(0,\\nabla h(x_{k+1})+\\mcN_\\mcX(x_{k+1})\\right)\\overset{\\eqref{subprob-opt1}}\\leq\\|\\nabla h(x_{k+1})+\\tilde\\eta_k^{-1}(x_k-x_{k+1})-\\widetilde G_k\\|^2\\nn\\\\\n&\\ \\leq2\\|\\tilde\\eta_k^{-1}(x_k-x_{k+1})+\\nabla h(x_k)-\\widetilde G_k\\|^2+2\\|\\nabla h(x_{k+1})-\\nabla h(x_k)\\|^2\\nn\\\\\n&\\ =2\\tilde\\eta_k^{-2}\\|x_{k+1}-x_k\\|^2+4\\tilde\\eta_k^{-1}\\langle\\widetilde G_k-\\nabla h(x_k), x_{k+1}-x_k\\rangle+2\\|\\widetilde G_k-\\nabla h(x_k)\\|^2\\nn\\\\\n&\\ \\quad +2\\|\\nabla h(x_{k+1})-\\nabla h(x_k)\\|^2\\nn\\\\\n&\\ \\leq2(\\tilde\\eta_k^{-2}+L^2)\\|x_{k+1}-x_k\\|^2+4\\tilde\\eta_k^{-1}\\langle\\widetilde G_k-\\nabla h(x_k), x_{k+1}-x_k\\rangle+2\\|\\widetilde G_k-\\nabla h(x_k)\\|^2,\\label{l3-e2}\n\\end{align}\nwhere the first inequality follows from Assumption \\ref{a1}(iv), the second inequality is due to the convexity of $\\|\\cdot\\|^2$, and the last inequality follows from the $L$-smoothness of $h$. In addition,  by \\eqref{l3-x} and $x_k\\in\\mcX$, one has\n\\[\n\\langle x_{k+1}-x_k+\\tilde\\eta_k\\widetilde G_k,x_k-x_{k+1}\\rangle\\geq0\\quad\\Rightarrow\\quad \\langle \\widetilde G_k,x_{k+1}-x_k\\rangle\\leq-\\tilde\\eta_k^{-1}\\|x_{k+1}-x_k\\|^2.\n\\]\n","sha256_utf8":"c3edafefb8c6a018c021c949b7b16912f1f4ce789964c0fb200e6c064240d11d"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"合并平滑下降与残差估计","source_units":[{"start":53512,"end":54234,"start_line":594,"end_line":603,"raw_tex":"This together with the $L$-smoothness of $h$ yields\n\\begin{align*}\nh(x_{k+1})& \\leq h(x_k)+\\langle\\nabla h(x_k),x_{k+1}-x_k\\rangle +\\frac{L}{2}\\|x_{k+1}-x_k\\|^2\\nn\\\\\n&=h(x_k)+\\langle\\widetilde G_k,x_{k+1}-x_k\\rangle+\\langle\\nabla h(x_k)-\\widetilde G_k,x_{k+1}-x_k\\rangle +\\frac{L}{2}\\|x_{k+1}-x_k\\|^2\\nn\\\\\n& \\leq h(x_k)-\\tilde\\eta_k^{-1}\\|x_{k+1}-x_k\\|^2+\\langle\\nabla h(x_k)-\\widetilde G_k,x_{k+1}-x_k\\rangle +\\frac{L}{2}\\|x_{k+1}-x_k\\|^2.\n\\end{align*}\nUsing this and \\eqref{l3-e2}, we obtain that\n\\beq\\label{l3-e3}\nh(x_{k+1})+2^{\\theta-2}\\gamma^2\\tilde\\eta_k[h(x_{k+1})]^{\\theta}\\leq h(x_k)+\\frac{1}{2}(L^2\\tilde\\eta_k-\\tilde\\eta_k^{-1}+L)\\|x_{k+1}-x_k\\|^2+\\frac{\\tilde\\eta_k}{2}\\|\\widetilde G_k-\\nabla h(x_k)\\|^2.\n\\eeq\n","sha256_utf8":"73e22014cf302a819104d0c0d3e7b7dc346d4fcc82d79bfa5d2f802d78b1bc2d"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"步长与截断界给出最终递推","source_units":[{"start":54234,"end":55199,"start_line":604,"end_line":618,"raw_tex":"Observe from \\eqref{l3-e1} and $\\rho_k\\eta_k\\leq(\\sqrt{5}-1)/(2L)$ that $L\\tilde\\eta_k=L\\rho_k\\eta_k\\leq(\\sqrt{5}-1)/2$, which implies that\n\\beq\\label{l3-e4}\nL^2\\tilde\\eta_k-\\tilde\\eta_k^{-1}+L=\\tilde\\eta_k^{-1}( L^2\\tilde\\eta_k^2+L\\tilde\\eta_k-1)\\leq0.\n\\eeq\nNotice from the expression of $g_k$ in Algorithm \\ref{alg1} that $g_k\\in\\cB(L_f)$ and hence $\\|g_k\\|\\leq L_f$. \nAlso, observe from Algorithm \\ref{alg1} and \\eqref{def-h} that $G_k=g_k+\\rho_k\\nabla h(x_k)$. Using these and \\eqref{l3-e1}, we have\n\\[\n\\|\\widetilde G_k-\\nabla h(x_k)\\|= \\|\\rho_k^{-1}G_k-\\nabla h(x_k)\\| =\\|\\rho_k^{-1}(g_k+\\rho_k\\nabla h(x_k))-\\nabla h(x_k)\\|= \\rho_k^{-1}\\|g_k\\|\\leq\\rho_k^{-1}L_f.\n\\]\nIt then follows from this, \\eqref{l3-e3} and \\eqref{l3-e4} that \n\\[\nh(x_{k+1})+2^{\\theta-2}\\gamma^2\\tilde\\eta_k[h(x_{k+1})]^\\theta\\leq h(x_k)+L_f^2\\tilde\\eta_k/(2\\rho_k^2).\n\\]\nThis and the definition of $\\tilde\\eta_k$ in \\eqref{l3-e1} imply that the conclusion of this lemma holds.\n\\end{proof}","sha256_utf8":"5102cd0bb5940dade6b15350954aa4f4155c2dc13e9016bf74fb5c093cd90d1f"}]},{"id":"S018","semantic_type":"lemma","kind":"lemma","title":"确定可行性违反的衰减界","source_units":[{"start":55332,"end":55731,"start_line":622,"end_line":624,"raw_tex":"\\begin{lemma}\\label{l-cnst2}\nLet $\\nu$, $\\tK_1$, and $C_1$ be given in \\eqref{def-para2}, \\eqref{def-tK2} and \\eqref{def-nuC2}, respectively. Suppose that Assumption \\ref{a1} holds, and $\\{x_k\\}$ is generated by Algorithm \\ref{alg1} with $\\{\\rho_k\\}$, $\\{\\eta_k\\}$ and $\\{\\alpha_k\\}$ given in \\eqref{def-para2}. Then  we have $\\|c(x_k)\\|^2\\leq 2C_1k^{-2\\nu/\\theta}$ for all $k\\geq\\tK_1$.\n\\end{lemma}","sha256_utf8":"7f3a66c7941d9b969d378ae1500c9c30b67cff8d250ad92761678be46544992a"}]},{"id":"S019","semantic_type":"proof","kind":"proof","title":"可行性衰减界的归纳证明","source_units":[{"start":55733,"end":59603,"start_line":626,"end_line":665,"raw_tex":"\\begin{proof}\nLet $h$ be defined in \\eqref{def-h}. To prove this lemma, it is equivalent to show that $h(x_k)\\leq C_1k^{-2\\nu/\\theta}$ for all $k\\geq\\tK_1$.  We now prove this by induction. Indeed, notice from Algorithm \\ref{alg1} that $x_{\\tK_1}\\in X$. It then follows from  \\eqref{def-nuC2}, \\eqref{def-h}, and Assumption \\ref{a1}(iv) that\n\\[\nh(x_{\\tK_1})\\overset{\\eqref{def-h}}{=}\\frac{1}{2}\\|c(x_{\\tK_1})\\|^2\\leq\\frac{1}{2}C_c^2\\overset{\\eqref{def-nuC2}}{\\leq} C_1\\tK_1^{-2\\nu/\\theta}.\n\\]\nHence,  the conclusion holds for $k=\\tK_1$. Now, suppose for induction that $h(x_k)\\leq C_1k^{-2\\nu/\\theta}$ holds for some $k\\geq\\tK_1$. Recall that $\\rho_k$, $\\eta_k$ and $\\tK_1$ are given in \\eqref{def-para2} and \\eqref{def-tK2}.  In view of these, one can observe that \n\\[\n\\rho_k\\eta_k\\overset{\\eqref{def-para2} }=\\frac{1}{4\\log(k+2)}\\leq\\frac{1}{4\\log(\\tK_1+2)}\\overset{\\eqref{def-tK2}}\\leq \\frac{1}{8L}\\leq\\frac{\\sqrt{5}-1}{2L},\n\\]\nand hence Lemma \\ref{l-rec} holds for such $k$.  Using Lemma \\ref{l-rec} with the choice of $\\rho_k$ and $\\eta_k$ given in \\eqref{def-para2}, we obtain that\n\\beq\\label{rec2}\nh(x_{k+1})+2^{\\theta-4}\\gamma^2[h(x_{k+1})]^{\\theta}/\\log(k+2)\\leq h(x_k)+L_f^2k^{-2\\nu}/(8\\log(k+2)).\n\\eeq\nFurther, let\n\\beq\\label{def-phi2}\n\\phi(t)=t+2^{\\theta-4}\\gamma^2t^\\theta/\\log(k+2).\n\\eeq\nNotice from \\eqref{def-para2} and \\eqref{def-nuC2} that $\\nu=\\min\\{\\hat\\theta/(\\hat\\theta+2),1/2\\}$ for some $\\hat\\theta\\geq1$ and $C_1\\geq1$. Using these and \\eqref{def-phi2},  we have\n\\begin{align}\n&\\ \\phi(C_1(k+1)^{-2\\nu/\\theta})-C_1k^{-2\\nu/\\theta}-L_f^2k^{-2\\nu}/(8\\log(k+2))\\nn\\\\\n&\\overset{\\eqref{def-phi2}}{=}C_1^\\theta 2^{\\theta-4}\\gamma^2(k+1)^{-2\\nu}/\\log(k+2)+C_1(k+1)^{-2\\nu/\\theta}-C_1k^{-2\\nu/\\theta}-L_f^2k^{-2\\nu}/(8\\log(k+2))\\nn\\\\\n&\\ \\geq C_1^\\theta 2^{\\theta-4}\\gamma^2(k+1)^{-2\\nu}/\\log(k+2)-2\\nu C_1 k^{-2\\nu/\\theta-1}/\\theta-L_f^2k^{-2\\nu}/(8\\log(k+2))\\nn\\\\\n&\\ = \\frac{k^{-2\\nu}}{\\log(k+2)}\\left(C_1^\\theta 2^{\\theta-4}\\gamma^2\\left(\\frac{k}{k+1}\\right)^{2\\nu}- 2\\nu C_1 k^{2\\nu(1-\\theta^{-1})-1}\\log(k+2)/\\theta-L_f^2/8\\right)\\nn\\\\\n&\\ \\geq \\frac{k^{-2\\nu}}{\\log(k+2)}\\left(C_1^\\theta 2^{\\theta-5}\\gamma^2-C_1 k^{-\\frac{1}{\\theta}}\\log(k+2)-L_f^2/8\\right)\\nn\\\\\n&\\ \\geq \\frac{k^{-2\\nu}}{\\log(k+2)}\\left(C_12^{\\theta-5}\\gamma^2-C_1 k^{-\\frac{1}{\\theta}}\\log(k+2)-L_f^2/8\\right),\\label{l4-e1g}\n\\end{align}\nwhere the first inequality follows from $(k+1)^{-2\\nu/\\theta}-k^{-2\\nu/\\theta}\\geq-2\\nu k^{-2\\nu/\\theta-1}/\\theta$ thanks to the convexity of $t^{-2\\nu/\\theta}$, the second inequality is due to $\\theta\\geq1$, $\\nu\\leq1/2$ and  $k/(k+1)\\geq1/2$, and the last inequality follows from $\\theta\\geq1$ and $C_1\\geq1$. In addition, one can verify that $t^{-\\frac{1}{2\\theta}}\\log(t+2)$ is decreasing on $[e^{2\\theta},\\infty)$. Using this, \\eqref{def-tK2}, and $k\\geq\\tK_1\\geq e^{2\\theta}$, we obtain that\n\\[\nk^{-\\frac{1}{2\\theta}}\\log(k+2)\\leq\\log(e^{2\\theta}+2)/e,\\quad k^{-\\frac{1}{2\\theta}}\\leq\\tK_1^{-\\frac{1}{2\\theta}}\\leq e\\gamma^2/(2^{6-\\theta}\\log(e^{2\\theta}+2)).\n\\]\nMultiplying both sides of these inequalities yields $k^{-1/\\theta}\\log(k+2)\\leq2^{\\theta-6}\\gamma^2$, which together with \\eqref{def-nuC2}  implies that\n\\begin{align*}\nC_12^{\\theta-5}\\gamma^2-C_1 k^{-\\frac{1}{\\theta}}\\log(k+2)-L_f^2/8\\geq C_12^{\\theta-6}\\gamma^2-L_f^2/8\\overset{\\eqref{def-nuC2}}{\\geq}0.\n\\end{align*}\nUsing this,  \\eqref{rec2}, \\eqref{def-phi2},  \\eqref{l4-e1g}, and the induction hypothesis that $h(x_k)\\leq C_1k^{-2\\nu/\\theta}$, we obtain that\n\\[\n\\phi(C_1(k+1)^{-2\\nu/\\theta})\\geq C_1k^{-2\\nu/\\theta}+L_f^2k^{-2\\nu}/(8\\log(k+2))\\geq h(x_k)+L_f^2k^{-2\\nu}/(8\\log(k+2))\\overset{\\eqref{rec2}\\eqref{def-phi2}}{\\geq}\\phi(h(x_{k+1})).\n\\]\nIt then follows from this inequality and the strict monotonicity of $\\phi$ on $[0,\\infty)$ that $h(x_{k+1})\\leq C_1(k+1)^{-2\\nu/\\theta}$. Hence, the induction is completed and the conclusion of this lemma holds. \n\\end{proof}","sha256_utf8":"5af095f81a62b69373379c398c2b21e00638d561c0e3e3da7e2f1ef6ea94ddc5"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"建立归纳初值与有效递推","source_units":[{"start":55733,"end":56946,"start_line":626,"end_line":638,"raw_tex":"\\begin{proof}\nLet $h$ be defined in \\eqref{def-h}. To prove this lemma, it is equivalent to show that $h(x_k)\\leq C_1k^{-2\\nu/\\theta}$ for all $k\\geq\\tK_1$.  We now prove this by induction. Indeed, notice from Algorithm \\ref{alg1} that $x_{\\tK_1}\\in X$. It then follows from  \\eqref{def-nuC2}, \\eqref{def-h}, and Assumption \\ref{a1}(iv) that\n\\[\nh(x_{\\tK_1})\\overset{\\eqref{def-h}}{=}\\frac{1}{2}\\|c(x_{\\tK_1})\\|^2\\leq\\frac{1}{2}C_c^2\\overset{\\eqref{def-nuC2}}{\\leq} C_1\\tK_1^{-2\\nu/\\theta}.\n\\]\nHence,  the conclusion holds for $k=\\tK_1$. Now, suppose for induction that $h(x_k)\\leq C_1k^{-2\\nu/\\theta}$ holds for some $k\\geq\\tK_1$. Recall that $\\rho_k$, $\\eta_k$ and $\\tK_1$ are given in \\eqref{def-para2} and \\eqref{def-tK2}.  In view of these, one can observe that \n\\[\n\\rho_k\\eta_k\\overset{\\eqref{def-para2} }=\\frac{1}{4\\log(k+2)}\\leq\\frac{1}{4\\log(\\tK_1+2)}\\overset{\\eqref{def-tK2}}\\leq \\frac{1}{8L}\\leq\\frac{\\sqrt{5}-1}{2L},\n\\]\nand hence Lemma \\ref{l-rec} holds for such $k$.  Using Lemma \\ref{l-rec} with the choice of $\\rho_k$ and $\\eta_k$ given in \\eqref{def-para2}, we obtain that\n\\beq\\label{rec2}\nh(x_{k+1})+2^{\\theta-4}\\gamma^2[h(x_{k+1})]^{\\theta}/\\log(k+2)\\leq h(x_k)+L_f^2k^{-2\\nu}/(8\\log(k+2)).\n\\eeq\n","sha256_utf8":"da99a1d992eeae4109d4b3cf0cee6aa85fa52f5b6b60dac56e88ad0f1f03bd47"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"构造比较函数并验证屏障界","source_units":[{"start":56946,"end":59046,"start_line":639,"end_line":659,"raw_tex":"Further, let\n\\beq\\label{def-phi2}\n\\phi(t)=t+2^{\\theta-4}\\gamma^2t^\\theta/\\log(k+2).\n\\eeq\nNotice from \\eqref{def-para2} and \\eqref{def-nuC2} that $\\nu=\\min\\{\\hat\\theta/(\\hat\\theta+2),1/2\\}$ for some $\\hat\\theta\\geq1$ and $C_1\\geq1$. Using these and \\eqref{def-phi2},  we have\n\\begin{align}\n&\\ \\phi(C_1(k+1)^{-2\\nu/\\theta})-C_1k^{-2\\nu/\\theta}-L_f^2k^{-2\\nu}/(8\\log(k+2))\\nn\\\\\n&\\overset{\\eqref{def-phi2}}{=}C_1^\\theta 2^{\\theta-4}\\gamma^2(k+1)^{-2\\nu}/\\log(k+2)+C_1(k+1)^{-2\\nu/\\theta}-C_1k^{-2\\nu/\\theta}-L_f^2k^{-2\\nu}/(8\\log(k+2))\\nn\\\\\n&\\ \\geq C_1^\\theta 2^{\\theta-4}\\gamma^2(k+1)^{-2\\nu}/\\log(k+2)-2\\nu C_1 k^{-2\\nu/\\theta-1}/\\theta-L_f^2k^{-2\\nu}/(8\\log(k+2))\\nn\\\\\n&\\ = \\frac{k^{-2\\nu}}{\\log(k+2)}\\left(C_1^\\theta 2^{\\theta-4}\\gamma^2\\left(\\frac{k}{k+1}\\right)^{2\\nu}- 2\\nu C_1 k^{2\\nu(1-\\theta^{-1})-1}\\log(k+2)/\\theta-L_f^2/8\\right)\\nn\\\\\n&\\ \\geq \\frac{k^{-2\\nu}}{\\log(k+2)}\\left(C_1^\\theta 2^{\\theta-5}\\gamma^2-C_1 k^{-\\frac{1}{\\theta}}\\log(k+2)-L_f^2/8\\right)\\nn\\\\\n&\\ \\geq \\frac{k^{-2\\nu}}{\\log(k+2)}\\left(C_12^{\\theta-5}\\gamma^2-C_1 k^{-\\frac{1}{\\theta}}\\log(k+2)-L_f^2/8\\right),\\label{l4-e1g}\n\\end{align}\nwhere the first inequality follows from $(k+1)^{-2\\nu/\\theta}-k^{-2\\nu/\\theta}\\geq-2\\nu k^{-2\\nu/\\theta-1}/\\theta$ thanks to the convexity of $t^{-2\\nu/\\theta}$, the second inequality is due to $\\theta\\geq1$, $\\nu\\leq1/2$ and  $k/(k+1)\\geq1/2$, and the last inequality follows from $\\theta\\geq1$ and $C_1\\geq1$. In addition, one can verify that $t^{-\\frac{1}{2\\theta}}\\log(t+2)$ is decreasing on $[e^{2\\theta},\\infty)$. Using this, \\eqref{def-tK2}, and $k\\geq\\tK_1\\geq e^{2\\theta}$, we obtain that\n\\[\nk^{-\\frac{1}{2\\theta}}\\log(k+2)\\leq\\log(e^{2\\theta}+2)/e,\\quad k^{-\\frac{1}{2\\theta}}\\leq\\tK_1^{-\\frac{1}{2\\theta}}\\leq e\\gamma^2/(2^{6-\\theta}\\log(e^{2\\theta}+2)).\n\\]\nMultiplying both sides of these inequalities yields $k^{-1/\\theta}\\log(k+2)\\leq2^{\\theta-6}\\gamma^2$, which together with \\eqref{def-nuC2}  implies that\n\\begin{align*}\nC_12^{\\theta-5}\\gamma^2-C_1 k^{-\\frac{1}{\\theta}}\\log(k+2)-L_f^2/8\\geq C_12^{\\theta-6}\\gamma^2-L_f^2/8\\overset{\\eqref{def-nuC2}}{\\geq}0.\n\\end{align*}\n","sha256_utf8":"9eb0c3c09ebcba889525e30b76b30eb635f00d0ef7439b23eccc5f359389cdc8"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"利用严格单调性闭合归纳","source_units":[{"start":59046,"end":59603,"start_line":660,"end_line":665,"raw_tex":"Using this,  \\eqref{rec2}, \\eqref{def-phi2},  \\eqref{l4-e1g}, and the induction hypothesis that $h(x_k)\\leq C_1k^{-2\\nu/\\theta}$, we obtain that\n\\[\n\\phi(C_1(k+1)^{-2\\nu/\\theta})\\geq C_1k^{-2\\nu/\\theta}+L_f^2k^{-2\\nu}/(8\\log(k+2))\\geq h(x_k)+L_f^2k^{-2\\nu}/(8\\log(k+2))\\overset{\\eqref{rec2}\\eqref{def-phi2}}{\\geq}\\phi(h(x_{k+1})).\n\\]\nIt then follows from this inequality and the strict monotonicity of $\\phi$ on $[0,\\infty)$ that $h(x_{k+1})\\leq C_1(k+1)^{-2\\nu/\\theta}$. Hence, the induction is completed and the conclusion of this lemma holds. \n\\end{proof}","sha256_utf8":"aefc331ba552b952796011723289c6ca268cd7f8ec2280a9bd2e8a6f0957c07b"}]},{"id":"S023","semantic_type":"lemma","kind":"lemma","title":"递归梯度估计的方差递推","source_units":[{"start":59758,"end":60295,"start_line":670,"end_line":676,"raw_tex":"\\begin{lemma}\\label{l-exp}\nSuppose that Assumptions \\ref{a1} and \\ref{a-storm} hold, and $\\{g_k\\}$ and $\\{x_k\\}$ are generated by Algorithm \\ref{alg1}. Then for all $k\\geq1$, we have\n\\[\n\\bbE\\left[\\|g_{k+1}-\\nabla f(x_{k+1})\\|^2\\right]\\leq(1-\\alpha_k)^2\\bbE\\left[\\|g_k-\\nabla f(x_k)\\|^2\\right]+6\\bL_{\\nabla f}^2\\bbE\\left[\\|x_{k+1}-x_k\\|^2\\right]+3\\sigma^2\\alpha_k^2,\n\\]\nwhere $\\{\\alpha_k\\}$ is given in Algorithm \\ref{alg1}, and $\\sigma$ and $\\bL_{\\nabla f}$ are given in Assumptions \\ref{a1} and \\ref{a-storm}, respectively. \n\\end{lemma}","sha256_utf8":"051dee35e3d4972aa55cd5ddd39e6ac8c47d737086db42e41c72253a95bf62ed"}]},{"id":"S024","semantic_type":"proof","kind":"proof","title":"截断估计方差递推的证明","source_units":[{"start":60297,"end":63510,"start_line":678,"end_line":715,"raw_tex":"\\begin{proof}\nNotice from \\eqref{cf-bnd} that $\\nabla f(x_{k+1})\\in\\cB(L_f)$ and hence $\\nabla f(x_{k+1})=\\Pi_{\\cB(L_f)}(\\nabla f(x_{k+1}))$. By this, the expression of $g_{k+1}$, and the nonexpansiveness of the projection operator $\\Pi_{\\cB(L_f)}$, one has\n\\begin{align}\n&\\|g_{k+1}-\\nabla f(x_{k+1})\\|^2=\\|\\Pi_{\\cB(L_f)}(\\nabla \\tf(x_{k+1},\\xi_{k+1})+(1-\\alpha_k)(g_k-\\nabla \\tf(x_k,\\xi_{k+1})))-\\Pi_{\\cB(L_f)}(\\nabla f(x_{k+1}))\\|^2 \\nn \\\\\n&\\leq\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})+(1-\\alpha_k)(g_k-\\nabla \\tf(x_k,\\xi_{k+1}))-\\nabla f(x_{k+1})\\|^2 \\nn \\\\\n&=\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})+(1-\\alpha_k)(g_k-\\nabla f(x_k)+\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1}))\\|^2 \\nn \\\\\n&=\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})+(1-\\alpha_k)(\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1}))\\|^2+(1-\\alpha_k)^2\\|g_k-\\nabla f(x_k)\\|^2\\nn \\\\\n&\\quad+2(1-\\alpha_k)\\langle g_k-\\nabla f(x_k),\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})\\rangle\\nn \\\\\n&\\quad+2(1-\\alpha_k)^2\\langle g_k-\\nabla f(x_k),\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1})\\rangle. \\label{g-bnd}\n\\end{align}\nLet $\\Xi_k=\\{\\xi_1,\\ldots, \\xi_k\\}$ denote the collection of samples drawn up to iteration $k-1$ in Algorithm \n\\ref{alg1}.  It then follows from Assumption \\ref{a1}(iii) that \n\\[\n\\bbE[\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})|\\Xi_k]=0, \\quad \\bbE[\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1})|\\Xi_k]=0,\n\\]\nwhich imply that\n\\begin{align*}\n&\\bbE[\\langle g_k-\\nabla f(x_k),\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})\\rangle|\\Xi_k]=\\langle g_k-\\nabla f(x_k),\\bbE[\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})|\\Xi_k]\\rangle=0, \\\\\n&\\bbE[\\langle g_k-\\nabla f(x_k),\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1})\\rangle|\\Xi_k]=\\langle g_k-\\nabla f(x_k),\\bbE[\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1})|\\Xi_k]\\rangle=0.\n\\end{align*}\nUsing these and taking a conditional expectation on both sides of \\eqref{g-bnd}, we have\n\\begin{align}\n\\bbE[\\|g_{k+1}-\\nabla f(x_{k+1})\\|^2|\\Xi_k]\\leq&\\bbE[\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})+(1-\\alpha_k)(\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1}))\\|^2|\\Xi_k]\\nn\\\\\n&+(1-\\alpha_k)^2\\|g_k-\\nabla f(x_k)\\|^2.\\label{l1-e1}\n\\end{align}\nIn addition, it follows from Assumption \\ref{a1} that\n\\begin{align*}\n&\\bbE[\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})+(1-\\alpha_k)(\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1}))\\|^2|\\Xi_k]\\\\\n&=\\bbE[\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla \\tf(x_k,\\xi_{k+1})+\\nabla f(x_k)-\\nabla f(x_{k+1})-\\alpha_k(\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1}))\\|^2|\\Xi_k]\\\\\n&\\leq3\\bbE[\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla \\tf(x_k,\\xi_{k+1})\\|^2|\\Xi_k]+3\\|\\nabla f(x_{k+1})-\\nabla f(x_k)\\|^2\\\\\n&\\quad+3\\alpha_k^2\\bbE[\\|\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1})\\|^2|\\Xi_k]\\leq6\\bL_{\\nabla f}^2\\|x_{k+1}-x_k\\|^2+3\\sigma^2\\alpha_k^2,\n\\end{align*}\nwhere the first inequality follows from the convexity of $\\|\\cdot\\|^2$, and the last inequality is due to \\eqref{bL-smooth} and Assumptions \\ref{a1}(iii) and \\ref{a-storm}. By this and \\eqref{l1-e1}, one has\n\\[\n\\bbE[\\|g_{k+1}-\\nabla f(x_{k+1})\\|^2|\\Xi_k]\\leq (1-\\alpha_k)^2\\|g_k-\\nabla f(x_k)\\|^2+6\\bL_{\\nabla f}^2\\|x_{k+1}-x_k\\|^2+3\\sigma^2\\alpha_k^2.\n\\]\nThe conclusion of this lemma follows from taking expectation on both sides of this inequality.\n\\end{proof}","sha256_utf8":"88f45a604b60d100b407ede719e7e2f2388c389a984bb77c9647abd6edecabb5"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"用投影非扩张性展开估计误差","source_units":[{"start":60297,"end":61367,"start_line":678,"end_line":687,"raw_tex":"\\begin{proof}\nNotice from \\eqref{cf-bnd} that $\\nabla f(x_{k+1})\\in\\cB(L_f)$ and hence $\\nabla f(x_{k+1})=\\Pi_{\\cB(L_f)}(\\nabla f(x_{k+1}))$. By this, the expression of $g_{k+1}$, and the nonexpansiveness of the projection operator $\\Pi_{\\cB(L_f)}$, one has\n\\begin{align}\n&\\|g_{k+1}-\\nabla f(x_{k+1})\\|^2=\\|\\Pi_{\\cB(L_f)}(\\nabla \\tf(x_{k+1},\\xi_{k+1})+(1-\\alpha_k)(g_k-\\nabla \\tf(x_k,\\xi_{k+1})))-\\Pi_{\\cB(L_f)}(\\nabla f(x_{k+1}))\\|^2 \\nn \\\\\n&\\leq\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})+(1-\\alpha_k)(g_k-\\nabla \\tf(x_k,\\xi_{k+1}))-\\nabla f(x_{k+1})\\|^2 \\nn \\\\\n&=\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})+(1-\\alpha_k)(g_k-\\nabla f(x_k)+\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1}))\\|^2 \\nn \\\\\n&=\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})+(1-\\alpha_k)(\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1}))\\|^2+(1-\\alpha_k)^2\\|g_k-\\nabla f(x_k)\\|^2\\nn \\\\\n&\\quad+2(1-\\alpha_k)\\langle g_k-\\nabla f(x_k),\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})\\rangle\\nn \\\\\n&\\quad+2(1-\\alpha_k)^2\\langle g_k-\\nabla f(x_k),\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1})\\rangle. \\label{g-bnd}\n\\end{align}\n","sha256_utf8":"534a7aaa2b6844894702b19abf233aff4dc2704c0412c71bdf15e43985083bb7"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"条件无偏性消去交叉项","source_units":[{"start":61367,"end":62427,"start_line":688,"end_line":702,"raw_tex":"Let $\\Xi_k=\\{\\xi_1,\\ldots, \\xi_k\\}$ denote the collection of samples drawn up to iteration $k-1$ in Algorithm \n\\ref{alg1}.  It then follows from Assumption \\ref{a1}(iii) that \n\\[\n\\bbE[\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})|\\Xi_k]=0, \\quad \\bbE[\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1})|\\Xi_k]=0,\n\\]\nwhich imply that\n\\begin{align*}\n&\\bbE[\\langle g_k-\\nabla f(x_k),\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})\\rangle|\\Xi_k]=\\langle g_k-\\nabla f(x_k),\\bbE[\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})|\\Xi_k]\\rangle=0, \\\\\n&\\bbE[\\langle g_k-\\nabla f(x_k),\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1})\\rangle|\\Xi_k]=\\langle g_k-\\nabla f(x_k),\\bbE[\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1})|\\Xi_k]\\rangle=0.\n\\end{align*}\nUsing these and taking a conditional expectation on both sides of \\eqref{g-bnd}, we have\n\\begin{align}\n\\bbE[\\|g_{k+1}-\\nabla f(x_{k+1})\\|^2|\\Xi_k]\\leq&\\bbE[\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})+(1-\\alpha_k)(\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1}))\\|^2|\\Xi_k]\\nn\\\\\n&+(1-\\alpha_k)^2\\|g_k-\\nabla f(x_k)\\|^2.\\label{l1-e1}\n\\end{align}\n","sha256_utf8":"1e43b5f119c17c7d5f8514343b2299d81dcd7f49fa83c42fbbe4d3b73ea3938c"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"平均平滑性与方差界完成估计","source_units":[{"start":62427,"end":63510,"start_line":703,"end_line":715,"raw_tex":"In addition, it follows from Assumption \\ref{a1} that\n\\begin{align*}\n&\\bbE[\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla f(x_{k+1})+(1-\\alpha_k)(\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1}))\\|^2|\\Xi_k]\\\\\n&=\\bbE[\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla \\tf(x_k,\\xi_{k+1})+\\nabla f(x_k)-\\nabla f(x_{k+1})-\\alpha_k(\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1}))\\|^2|\\Xi_k]\\\\\n&\\leq3\\bbE[\\|\\nabla \\tf(x_{k+1},\\xi_{k+1})-\\nabla \\tf(x_k,\\xi_{k+1})\\|^2|\\Xi_k]+3\\|\\nabla f(x_{k+1})-\\nabla f(x_k)\\|^2\\\\\n&\\quad+3\\alpha_k^2\\bbE[\\|\\nabla f(x_k)-\\nabla \\tf(x_k,\\xi_{k+1})\\|^2|\\Xi_k]\\leq6\\bL_{\\nabla f}^2\\|x_{k+1}-x_k\\|^2+3\\sigma^2\\alpha_k^2,\n\\end{align*}\nwhere the first inequality follows from the convexity of $\\|\\cdot\\|^2$, and the last inequality is due to \\eqref{bL-smooth} and Assumptions \\ref{a1}(iii) and \\ref{a-storm}. By this and \\eqref{l1-e1}, one has\n\\[\n\\bbE[\\|g_{k+1}-\\nabla f(x_{k+1})\\|^2|\\Xi_k]\\leq (1-\\alpha_k)^2\\|g_k-\\nabla f(x_k)\\|^2+6\\bL_{\\nabla f}^2\\|x_{k+1}-x_k\\|^2+3\\sigma^2\\alpha_k^2.\n\\]\nThe conclusion of this lemma follows from taking expectation on both sides of this inequality.\n\\end{proof}","sha256_utf8":"ad971022ab8a4cf4db963d723a9b77a29068dd4c7cdc9e060f47165143aa4f5a"}]},{"id":"S028","semantic_type":"lemma","kind":"lemma","title":"加权罚函数势能的累计上界","source_units":[{"start":63658,"end":64669,"start_line":719,"end_line":730,"raw_tex":"\\begin{lemma}\\label{l-pfunc}\nSuppose that Assumptions \\ref{a1} and \\ref{a-storm} hold. Let $\\{g_k\\}$ and $\\{x_k\\}$ be generated by Algorithm \\ref{alg1}, and  let \n\\beq\\label{def-zeta}\n\\zeta_k=k^{\\nu}/(2\\log 3),\n\\eeq\nwhere $\\nu$ is defined in \\eqref{def-para2}. Then for all $k\\geq1$, we have\n\\begin{align}\n&\\bbE\\left[Q_{\\rho_k}(x_k)+\\zeta_k\\|g_k-\\nabla f(x_k)\\|^2\\right] \\leq Q_{\\rho_1}(x_1)+\\zeta_1\\|g_1-\\nabla f(x_1)\\|^2+\\frac{1}{2}\\sum_{i=1}^{k-1}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right] \\nn \\\\\n&+\\frac{1}{2}\\sum_{i=1}^{k-1}\\left(\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+3\\sigma^2\\sum_{i=1}^{k-1}\\zeta_{i+1}\\alpha_i^2, \\label{penalty-bnd}\n\\end{align}\nwhere $\\{\\alpha_k\\}$, $\\{\\rho_k\\}$ and $\\{\\eta_k\\}$ are given in Algorithm \\ref{alg1}, $Q_\\rho$  and $L$ are respectively defined in \\eqref{def-Q} and \\eqref{def-L},  and $\\sigma$ and $\\bL_{\\nabla f}$  are given in Assumptions \\ref{a1} and \\ref{a-storm}, respectively.\n\\end{lemma}","sha256_utf8":"438305b7713a343ffed9dd2ca99cc07a8244abb0cfe99f8612cecbfcaa070ca2"}]},{"id":"S029","semantic_type":"proof","kind":"proof","title":"加权势能上界的完整证明","source_units":[{"start":64671,"end":69558,"start_line":732,"end_line":780,"raw_tex":"\\begin{proof}\nObserve from Assumptions \\ref{a1} and \\ref{a-storm} and the definition of $Q_\\rho$ in \\eqref{def-Q} that $Q_{\\rho_k}$ is $(\\bL_{\\nabla f}+\\rho_kL)$-smooth on $\\mcX$, where $L$ is defined in \\eqref{def-L}. Notice from Algorithm \\ref{alg1} that $x_k\\in\\mcX$ and $x_{k+1}= \\Pi_X(x_k-\\eta_k G_k)$, which imply that\n\\beq \\label{subprob-opt2}\n\\langle x_{k+1}-x_k+\\eta_k G_k,x_k-x_{k+1}\\rangle\\geq0\\quad\\Rightarrow\\quad \\langle G_k,x_{k+1}-x_k\\rangle\\leq-\\eta_k^{-1}\\|x_{k+1}-x_k\\|^2.\n\\eeq\nAlso, notice from Algorithm \\ref{alg1} and \\eqref{def-Q} that \n\\[\nG_k=g_k+\\rho_k\\nabla c(x_k)c(x_k), \\quad \\nabla Q_{\\rho_k}(x_k)=\\nabla f(x_k)+\\rho_k\\nabla c(x_k)c(x_k),\n\\] \nand hence $\\nabla Q_{\\rho_k}(x_k)-G_k=g_k-\\nabla f(x_k)$.  In addition, by Young's inequality,  one has \n\\beq \\label{young-ineq}\n\\langle\\nabla Q_{\\rho_k}(x_k)-G_k,x_{k+1}-x_k\\rangle \\leq \\frac{1}{2\\eta_k}\\|x_{k+1}-x_k\\|^2+\\frac{\\eta_k}{2}\\|\\nabla Q_{\\rho_k}(x_k)-G_k\\|^2\n\\eeq\nUsing the last two relations,  \\eqref{subprob-opt2}, and the $(\\bL_{\\nabla f}+\\rho_kL)$-smoothness of $Q_{\\rho_k}$, we obtain that\n\\begin{align*}\nQ_{\\rho_k}(x_{k+1})&\\ \\leq  Q_{\\rho_k}(x_k)+\\langle\\nabla Q_{\\rho_k}(x_k),x_{k+1}-x_k\\rangle+\\frac{1}{2}(\\bL_{\\nabla f}+\\rho_kL)\\|x_{k+1}-x_k\\|^2\\\\\n&\\ = Q_{\\rho_k}(x_k)+\\langle G_k,x_{k+1}-x_k\\rangle+\\langle\\nabla Q_{\\rho_k}(x_k)-G_k,x_{k+1}-x_k\\rangle+\\frac{1}{2}(\\bL_{\\nabla f}+\\rho_kL)\\|x_{k+1}-x_k\\|^2\\\\\n& \\overset{\\eqref{young-ineq}}{\\leq} Q_{\\rho_k}(x_k)+\\langle G_k,x_{k+1}-x_k\\rangle+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL+\\eta_k^{-1}\\right)\\|x_{k+1}-x_k\\|^2+\\frac{\\eta_k}{2}\\|\\nabla Q_{\\rho_k}(x_k)-G_k\\|^2\\\\\n&\\ \\leq Q_{\\rho_k}(x_k)+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL-\\eta_k^{-1}\\right)\\|x_{k+1}-x_k\\|^2+\\frac{\\eta_k}{2}\\|g_k-\\nabla f(x_k)\\|^2,\n\\end{align*}\nwhere the first inequality is due to the $(\\bL_{\\nabla f}+\\rho_kL)$-smoothness of $Q_{\\rho_k}$, and the last inequality follows from \\eqref{subprob-opt2} and the relation $\\nabla Q_{\\rho_k}(x_k)-G_k=g_k-\\nabla f(x_k)$. By this and \\eqref{def-Q}, we further have\n\\begin{align}\n&Q_{\\rho_{k+1}}(x_{k+1})\\leq Q_{\\rho_k}(x_k)+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL-\\eta_k^{-1}\\right)\\|x_{k+1}-x_k\\|^2+\\frac{\\eta_k}{2}\\|g_k-\\nabla f(x_k)\\|^2\\nn\\\\\n&\\qquad\\qquad\\qquad\\,\\,+Q_{\\rho_{k+1}}(x_{k+1})-Q_{\\rho_k}(x_{k+1})\\nn\\\\\n&\\overset{\\eqref{def-Q}}{=} Q_{\\rho_k}(x_k)+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL-\\eta_k^{-1}\\right)\\|x_{k+1}-x_k\\|^2+\\frac{\\eta_k}{2}\\|g_k-\\nabla f(x_k)\\|^2+\\frac{1}{2}(\\rho_{k+1}-\\rho_k)\\|c(x_{k+1})\\|^2.\\label{l5-e1}\n\\end{align}\n\nBy the definitions of $\\eta_k$, $\\alpha_k$, $\\nu$ and $\\zeta_k$ in \\eqref{def-para2}, \\eqref{def-nuC2} and \\eqref{def-zeta}, one has\n\\begin{align*}\n&\\ \\zeta_k-\\zeta_{k+1}(1-\\alpha_k)^2-\\eta_k=\\zeta_{k+1}\\alpha_k(2-\\alpha_k)-\\zeta_{k+1}+\\zeta_k-\\eta_k\\geq \\zeta_{k+1}\\alpha_k-\\zeta_{k+1}+\\zeta_k-\\eta_k\\\\\n&\\overset{\\eqref{def-para2}\\eqref{def-zeta}}\\geq(k+1)^\\nu k^{-2\\nu}/(2\\log 3)-(k+1)^\\nu/(2\\log 3)+k^\\nu/(2\\log 3)-k^{-\\nu}/(4\\log(k+2))\\\\\n&\\ \\ \\ \\geq k^{-\\nu}/(2\\log 3)-(k+1)^\\nu/(2\\log 3)+k^\\nu/(2\\log 3)-k^{-\\nu}/(4\\log(k+2))\\\\\n&\\ \\ \\ \\geq k^{-\\nu}/(2\\log 3)-\\nu k^{\\nu-1}/(2\\log 3)-k^{-\\nu}/(4\\log(k+2))\\\\\n&\\ \\ \\ = \\frac{k^{-\\nu}}{4\\log 3}\\left(2-2\\nu k^{2\\nu-1}-\\log 3/\\log(k+2)\\right) \\geq\\frac{k^{-\\nu}}{4\\log 3}\\left(2-1-\\log 3/\\log(k+2)\\right)\\geq0,\n\\end{align*}\nwhere the first inequality is due to  $0<\\alpha_k\\leq1$, the third inequality follows from $(k+1)^\\nu>k^\\nu$, the fourth inequality is due to $(k+1)^\\nu-k^\\nu\\leq \\nu k^{\\nu-1}$ thanks to the concavity of $t^\\nu$, the fifth inequality follows from $\\nu\\leq1/2$, and the last inequality is due to $k\\geq1$. Hence, we obtain\n\\[\n\\zeta_{k+1}(1-\\alpha_k)^2+\\eta_k\\leq\\zeta_k\\qquad \\forall k\\geq1.\n\\]\nUsing this, taking expectation on both sides of \\eqref{l5-e1}, and summing the resulting inequality with the inequality in Lemma \\ref{l-exp}, we obtain that\n\\begin{align}\n&\\bbE\\left[Q_{\\rho_{k+1}}(x_{k+1})+\\zeta_{k+1}\\|g_{k+1}-\\nabla f(x_{k+1})\\|^2\\right]\\nn\\\\\n&\\leq \\bbE\\left[Q_{\\rho_k}(x_k)+\\left(\\zeta_{k+1}(1-\\alpha_k)^2+\\eta_k\\right)\\|g_k-\\nabla f(x_k)\\|^2\\right]+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL-\\eta_k^{-1}+12\\zeta_{k+1}\\bL_{\\nabla f}^2\\right) \\nn \\\\ \n&\\quad\\times \\bbE\\left[\\|x_{k+1}-x_k\\|^2\\right] -\\frac{\\eta_k}{2}\\bbE\\left[\\|g_k-\\nabla f(x_k)\\|^2\\right]+\\frac{1}{2}(\\rho_{k+1}-\\rho_k)\\bbE\\left[\\|c(x_{k+1})\\|^2\\right]+3\\sigma^2\\zeta_{k+1}\\alpha_k^2\\nn\\\\\n&\\leq \\bbE\\left[Q_{\\rho_k}(x_k)+\\zeta_k\\|g_k-\\nabla f(x_k)\\|^2\\right]+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL-\\eta_k^{-1}+12\\zeta_{k+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{k+1}-x_k\\|^2\\right]\\nn\\\\\n&\\quad-\\frac{\\eta_k}{2}\\bbE\\left[\\|g_k-\\nabla f(x_k)\\|^2\\right]+\\frac{1}{2}(\\rho_{k+1}-\\rho_k)\\bbE\\left[\\|c(x_{k+1})\\|^2\\right]+3\\sigma^2\\zeta_{k+1}\\alpha_k^2.\\label{l5-e2}\n\\end{align}\nThe conclusion of this lemma follows by replacing $k$ with $i$ in the above inequalities and summing them up for all $1\\leq i\\leq k-1$.\n\\end{proof}","sha256_utf8":"91afd2bb7faa89b11307622828575e64ce950522a068ec81841e81af4786e460"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"确认罚函数平滑性与投影下降","source_units":[{"start":64671,"end":65168,"start_line":732,"end_line":736,"raw_tex":"\\begin{proof}\nObserve from Assumptions \\ref{a1} and \\ref{a-storm} and the definition of $Q_\\rho$ in \\eqref{def-Q} that $Q_{\\rho_k}$ is $(\\bL_{\\nabla f}+\\rho_kL)$-smooth on $\\mcX$, where $L$ is defined in \\eqref{def-L}. Notice from Algorithm \\ref{alg1} that $x_k\\in\\mcX$ and $x_{k+1}= \\Pi_X(x_k-\\eta_k G_k)$, which imply that\n\\beq \\label{subprob-opt2}\n\\langle x_{k+1}-x_k+\\eta_k G_k,x_k-x_{k+1}\\rangle\\geq0\\quad\\Rightarrow\\quad \\langle G_k,x_{k+1}-x_k\\rangle\\leq-\\eta_k^{-1}\\|x_{k+1}-x_k\\|^2.\n\\eeq\n","sha256_utf8":"1316b18e24f808303a189137c15339567a26e03fe00524de2db46e4fa7a1e882"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"导出含罚参数变化的单步下降","source_units":[{"start":65168,"end":67186,"start_line":737,"end_line":757,"raw_tex":"Also, notice from Algorithm \\ref{alg1} and \\eqref{def-Q} that \n\\[\nG_k=g_k+\\rho_k\\nabla c(x_k)c(x_k), \\quad \\nabla Q_{\\rho_k}(x_k)=\\nabla f(x_k)+\\rho_k\\nabla c(x_k)c(x_k),\n\\] \nand hence $\\nabla Q_{\\rho_k}(x_k)-G_k=g_k-\\nabla f(x_k)$.  In addition, by Young's inequality,  one has \n\\beq \\label{young-ineq}\n\\langle\\nabla Q_{\\rho_k}(x_k)-G_k,x_{k+1}-x_k\\rangle \\leq \\frac{1}{2\\eta_k}\\|x_{k+1}-x_k\\|^2+\\frac{\\eta_k}{2}\\|\\nabla Q_{\\rho_k}(x_k)-G_k\\|^2\n\\eeq\nUsing the last two relations,  \\eqref{subprob-opt2}, and the $(\\bL_{\\nabla f}+\\rho_kL)$-smoothness of $Q_{\\rho_k}$, we obtain that\n\\begin{align*}\nQ_{\\rho_k}(x_{k+1})&\\ \\leq  Q_{\\rho_k}(x_k)+\\langle\\nabla Q_{\\rho_k}(x_k),x_{k+1}-x_k\\rangle+\\frac{1}{2}(\\bL_{\\nabla f}+\\rho_kL)\\|x_{k+1}-x_k\\|^2\\\\\n&\\ = Q_{\\rho_k}(x_k)+\\langle G_k,x_{k+1}-x_k\\rangle+\\langle\\nabla Q_{\\rho_k}(x_k)-G_k,x_{k+1}-x_k\\rangle+\\frac{1}{2}(\\bL_{\\nabla f}+\\rho_kL)\\|x_{k+1}-x_k\\|^2\\\\\n& \\overset{\\eqref{young-ineq}}{\\leq} Q_{\\rho_k}(x_k)+\\langle G_k,x_{k+1}-x_k\\rangle+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL+\\eta_k^{-1}\\right)\\|x_{k+1}-x_k\\|^2+\\frac{\\eta_k}{2}\\|\\nabla Q_{\\rho_k}(x_k)-G_k\\|^2\\\\\n&\\ \\leq Q_{\\rho_k}(x_k)+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL-\\eta_k^{-1}\\right)\\|x_{k+1}-x_k\\|^2+\\frac{\\eta_k}{2}\\|g_k-\\nabla f(x_k)\\|^2,\n\\end{align*}\nwhere the first inequality is due to the $(\\bL_{\\nabla f}+\\rho_kL)$-smoothness of $Q_{\\rho_k}$, and the last inequality follows from \\eqref{subprob-opt2} and the relation $\\nabla Q_{\\rho_k}(x_k)-G_k=g_k-\\nabla f(x_k)$. By this and \\eqref{def-Q}, we further have\n\\begin{align}\n&Q_{\\rho_{k+1}}(x_{k+1})\\leq Q_{\\rho_k}(x_k)+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL-\\eta_k^{-1}\\right)\\|x_{k+1}-x_k\\|^2+\\frac{\\eta_k}{2}\\|g_k-\\nabla f(x_k)\\|^2\\nn\\\\\n&\\qquad\\qquad\\qquad\\,\\,+Q_{\\rho_{k+1}}(x_{k+1})-Q_{\\rho_k}(x_{k+1})\\nn\\\\\n&\\overset{\\eqref{def-Q}}{=} Q_{\\rho_k}(x_k)+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL-\\eta_k^{-1}\\right)\\|x_{k+1}-x_k\\|^2+\\frac{\\eta_k}{2}\\|g_k-\\nabla f(x_k)\\|^2+\\frac{1}{2}(\\rho_{k+1}-\\rho_k)\\|c(x_{k+1})\\|^2.\\label{l5-e1}\n\\end{align}\n","sha256_utf8":"bacf1db0f1b648a97159e98d07d2cc6091fffd6c4182172c53a9d16729e6cf7d"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"验证势能权重的吸收关系","source_units":[{"start":67187,"end":68356,"start_line":759,"end_line":770,"raw_tex":"By the definitions of $\\eta_k$, $\\alpha_k$, $\\nu$ and $\\zeta_k$ in \\eqref{def-para2}, \\eqref{def-nuC2} and \\eqref{def-zeta}, one has\n\\begin{align*}\n&\\ \\zeta_k-\\zeta_{k+1}(1-\\alpha_k)^2-\\eta_k=\\zeta_{k+1}\\alpha_k(2-\\alpha_k)-\\zeta_{k+1}+\\zeta_k-\\eta_k\\geq \\zeta_{k+1}\\alpha_k-\\zeta_{k+1}+\\zeta_k-\\eta_k\\\\\n&\\overset{\\eqref{def-para2}\\eqref{def-zeta}}\\geq(k+1)^\\nu k^{-2\\nu}/(2\\log 3)-(k+1)^\\nu/(2\\log 3)+k^\\nu/(2\\log 3)-k^{-\\nu}/(4\\log(k+2))\\\\\n&\\ \\ \\ \\geq k^{-\\nu}/(2\\log 3)-(k+1)^\\nu/(2\\log 3)+k^\\nu/(2\\log 3)-k^{-\\nu}/(4\\log(k+2))\\\\\n&\\ \\ \\ \\geq k^{-\\nu}/(2\\log 3)-\\nu k^{\\nu-1}/(2\\log 3)-k^{-\\nu}/(4\\log(k+2))\\\\\n&\\ \\ \\ = \\frac{k^{-\\nu}}{4\\log 3}\\left(2-2\\nu k^{2\\nu-1}-\\log 3/\\log(k+2)\\right) \\geq\\frac{k^{-\\nu}}{4\\log 3}\\left(2-1-\\log 3/\\log(k+2)\\right)\\geq0,\n\\end{align*}\nwhere the first inequality is due to  $0<\\alpha_k\\leq1$, the third inequality follows from $(k+1)^\\nu>k^\\nu$, the fourth inequality is due to $(k+1)^\\nu-k^\\nu\\leq \\nu k^{\\nu-1}$ thanks to the concavity of $t^\\nu$, the fifth inequality follows from $\\nu\\leq1/2$, and the last inequality is due to $k\\geq1$. Hence, we obtain\n\\[\n\\zeta_{k+1}(1-\\alpha_k)^2+\\eta_k\\leq\\zeta_k\\qquad \\forall k\\geq1.\n\\]\n","sha256_utf8":"edcf205e243aaee88f5fccc8975772c96a0f1dfc250a396d0801f934266658bd"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"合并方差递推并累加势能差","source_units":[{"start":68356,"end":69558,"start_line":771,"end_line":780,"raw_tex":"Using this, taking expectation on both sides of \\eqref{l5-e1}, and summing the resulting inequality with the inequality in Lemma \\ref{l-exp}, we obtain that\n\\begin{align}\n&\\bbE\\left[Q_{\\rho_{k+1}}(x_{k+1})+\\zeta_{k+1}\\|g_{k+1}-\\nabla f(x_{k+1})\\|^2\\right]\\nn\\\\\n&\\leq \\bbE\\left[Q_{\\rho_k}(x_k)+\\left(\\zeta_{k+1}(1-\\alpha_k)^2+\\eta_k\\right)\\|g_k-\\nabla f(x_k)\\|^2\\right]+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL-\\eta_k^{-1}+12\\zeta_{k+1}\\bL_{\\nabla f}^2\\right) \\nn \\\\ \n&\\quad\\times \\bbE\\left[\\|x_{k+1}-x_k\\|^2\\right] -\\frac{\\eta_k}{2}\\bbE\\left[\\|g_k-\\nabla f(x_k)\\|^2\\right]+\\frac{1}{2}(\\rho_{k+1}-\\rho_k)\\bbE\\left[\\|c(x_{k+1})\\|^2\\right]+3\\sigma^2\\zeta_{k+1}\\alpha_k^2\\nn\\\\\n&\\leq \\bbE\\left[Q_{\\rho_k}(x_k)+\\zeta_k\\|g_k-\\nabla f(x_k)\\|^2\\right]+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_kL-\\eta_k^{-1}+12\\zeta_{k+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{k+1}-x_k\\|^2\\right]\\nn\\\\\n&\\quad-\\frac{\\eta_k}{2}\\bbE\\left[\\|g_k-\\nabla f(x_k)\\|^2\\right]+\\frac{1}{2}(\\rho_{k+1}-\\rho_k)\\bbE\\left[\\|c(x_{k+1})\\|^2\\right]+3\\sigma^2\\zeta_{k+1}\\alpha_k^2.\\label{l5-e2}\n\\end{align}\nThe conclusion of this lemma follows by replacing $k$ with $i$ in the above inequalities and summing them up for all $1\\leq i\\leq k-1$.\n\\end{proof}","sha256_utf8":"56dac6335d22685ae7141e4b7f80372e2c1cc5c511a53598cf82eeba5e6cd2ef"}]},{"id":"S034","semantic_type":"lemma","kind":"lemma","title":"驻点残差的步长与估计误差界","source_units":[{"start":69679,"end":70284,"start_line":784,"end_line":790,"raw_tex":"\\begin{lemma}\\label{l-dist}\nSuppose that Assumptions \\ref{a1} and \\ref{a-storm} hold, and $\\{g_k\\}$ and  $\\{x_k\\}$ are generated by Algorithm \\ref{alg1}. Then for all $k\\geq1$, we have\n\\beq \\label{residual-bnd}\n\\dist^2\\left(0,\\nabla Q_{\\rho_k}(x_{k+1})+\\mcN_\\mcX(x_{k+1})\\right)\\leq3\\left(\\eta_k^{-2}+(\\bL_{\\nabla f}+\\rho_kL)^2\\right)\\|x_{k+1}-x_k\\|^2+3\\|g_k-\\nabla f(x_k)\\|^2,\n\\eeq\nwhere  $\\{\\rho_k\\}$ and $\\{\\eta_k\\}$ are given in Algorithm \\ref{alg1}, $\\bL_{\\nabla f}$ is given in Assumption \\ref{a-storm},  and $L$ and $Q_\\rho$ are defined in \\eqref{def-L} and \\eqref{def-Q}, respectively.\n\\end{lemma}","sha256_utf8":"b0e689ae921747d342a43acc0312ffd1d0cf44f9bb288d14819d125321628032"}]},{"id":"S035","semantic_type":"proof","kind":"proof","title":"驻点残差上界的证明","source_units":[{"start":70286,"end":71820,"start_line":792,"end_line":809,"raw_tex":"\\begin{proof}\nBy the expression of $x_{k+1}$ in Algorithm \\ref{alg1}, one has\n\\beq \\label{subprob_opt}\n0\\in x_{k+1}-x_k+\\eta_k G_k+\\mcN_\\mcX(x_{k+1})\\quad\\Rightarrow\\quad\\eta_k^{-1}(x_k-x_{k+1})-G_k\\in\\mcN_\\mcX(x_{k+1}).\n\\eeq\nNotice from the definition of $Q_\\rho$ in \\eqref{def-Q} that $\\nabla Q_{\\rho_k}(x)=\\nabla f(x)+\\rho_k\\nabla c(x)c(x)$, which together with \\eqref{def-L}, \\eqref{bL-smooth} and Assumption \\ref{a1}(iv) implies that $Q_{\\rho_k}$ is $(\\bL_{\\nabla f}+\\rho_kL)$-smooth on $\\mcX$. Using \\eqref{subprob_opt}, the expression of $\\nabla Q_{\\rho_k}$, and $G_k=g_k+\\rho_k\\nabla c(x_k)c(x_k)$ (see Algorithm \\ref{alg1}), we have\n\\[\n\\eta_k^{-1}(x_k-x_{k+1})+\\nabla f(x_k)-g_k-\\nabla Q_{\\rho_k}(x_k) = \\eta_k^{-1}(x_k-x_{k+1})-g_k-\\rho_k\\nabla c(x_k)c(x_k) \\in \\mcN_\\mcX(x_{k+1}).\n\\]\nBy this and the $(\\bL_{\\nabla f}+\\rho_kL)$-smoothness of $Q_{\\rho_k}$, one has\n\\begin{align*}\n&\\dist^2\\left(0,\\nabla Q_{\\rho_k}(x_{k+1})+\\mcN_\\mcX(x_{k+1})\\right) \\leq \\|\\nabla Q_{\\rho_k}(x_{k+1})\n+(\\eta_k^{-1}(x_k-x_{k+1})+\\nabla f(x_k)-g_k-\\nabla Q_{\\rho_k}(x_k))\\|^2\\\\\n&\\leq 3\\left(\\|\\nabla Q_{\\rho_k}(x_{k+1})-\\nabla Q_{\\rho_k}(x_k)\\|^2+\\eta_k^{-2}\\|x_{k+1}-x_k\\|^2+\\| g_k-\\nabla f(x_k)\\|^2\\right)\\\\\n&\\leq 3\\left(\\eta_k^{-2}+(\\bL_{\\nabla f}+\\rho_kL)^2\\right)\\|x_{k+1}-x_k\\|^2+3\\|g_k-\\nabla f(x_k)\\|^2,\n\\end{align*}\nwhere the second inequality follows from the convexity of $\\|\\cdot\\|^2$, and the last inequality is due to the $(\\bL_{\\nabla f}+\\rho_kL)$-smoothness of $Q_{\\rho_k}$. Hence, the conclusion of this lemma holds.\n\\end{proof}","sha256_utf8":"c0973958c1ec9a03d07cbbade6f0f664c1a532c387d0b801e600f9bbae4094f8"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"构造法锥元素并确定平滑常数","source_units":[{"start":70286,"end":71081,"start_line":792,"end_line":800,"raw_tex":"\\begin{proof}\nBy the expression of $x_{k+1}$ in Algorithm \\ref{alg1}, one has\n\\beq \\label{subprob_opt}\n0\\in x_{k+1}-x_k+\\eta_k G_k+\\mcN_\\mcX(x_{k+1})\\quad\\Rightarrow\\quad\\eta_k^{-1}(x_k-x_{k+1})-G_k\\in\\mcN_\\mcX(x_{k+1}).\n\\eeq\nNotice from the definition of $Q_\\rho$ in \\eqref{def-Q} that $\\nabla Q_{\\rho_k}(x)=\\nabla f(x)+\\rho_k\\nabla c(x)c(x)$, which together with \\eqref{def-L}, \\eqref{bL-smooth} and Assumption \\ref{a1}(iv) implies that $Q_{\\rho_k}$ is $(\\bL_{\\nabla f}+\\rho_kL)$-smooth on $\\mcX$. Using \\eqref{subprob_opt}, the expression of $\\nabla Q_{\\rho_k}$, and $G_k=g_k+\\rho_k\\nabla c(x_k)c(x_k)$ (see Algorithm \\ref{alg1}), we have\n\\[\n\\eta_k^{-1}(x_k-x_{k+1})+\\nabla f(x_k)-g_k-\\nabla Q_{\\rho_k}(x_k) = \\eta_k^{-1}(x_k-x_{k+1})-g_k-\\rho_k\\nabla c(x_k)c(x_k) \\in \\mcN_\\mcX(x_{k+1}).\n\\]\n","sha256_utf8":"169e2a0e67c8e154d9400727964e9929b93a372f993eee075181eaebdcf94d17"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"平方范数分解得到残差上界","source_units":[{"start":71081,"end":71820,"start_line":801,"end_line":809,"raw_tex":"By this and the $(\\bL_{\\nabla f}+\\rho_kL)$-smoothness of $Q_{\\rho_k}$, one has\n\\begin{align*}\n&\\dist^2\\left(0,\\nabla Q_{\\rho_k}(x_{k+1})+\\mcN_\\mcX(x_{k+1})\\right) \\leq \\|\\nabla Q_{\\rho_k}(x_{k+1})\n+(\\eta_k^{-1}(x_k-x_{k+1})+\\nabla f(x_k)-g_k-\\nabla Q_{\\rho_k}(x_k))\\|^2\\\\\n&\\leq 3\\left(\\|\\nabla Q_{\\rho_k}(x_{k+1})-\\nabla Q_{\\rho_k}(x_k)\\|^2+\\eta_k^{-2}\\|x_{k+1}-x_k\\|^2+\\| g_k-\\nabla f(x_k)\\|^2\\right)\\\\\n&\\leq 3\\left(\\eta_k^{-2}+(\\bL_{\\nabla f}+\\rho_kL)^2\\right)\\|x_{k+1}-x_k\\|^2+3\\|g_k-\\nabla f(x_k)\\|^2,\n\\end{align*}\nwhere the second inequality follows from the convexity of $\\|\\cdot\\|^2$, and the last inequality is due to the $(\\bL_{\\nabla f}+\\rho_kL)$-smoothness of $Q_{\\rho_k}$. Hence, the conclusion of this lemma holds.\n\\end{proof}","sha256_utf8":"bd43a520dc8a2efe9749587cca40f313ab5a3fd2eb8e2823627c987c96d5f600"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"验证充分大迭代后的系数控制","source_units":[{"start":71940,"end":72790,"start_line":813,"end_line":824,"raw_tex":"\\begin{proof}[\\textbf{Proof of Theorem \\ref{thm:storm}}]\nUsing \\eqref{def-para2}, \\eqref{def-tK2} and \\eqref{def-zeta}, we have that for all $i\\geq\\tK_1$,\n\\begin{align}\n\\bL_{\\nabla f}+\\rho_iL &\\overset{\\eqref{def-para2}}=\\bL_{\\nabla f}i^{-\\nu}\\eta_i^{-1}/(4\\log(i+2))+L\\eta_i^{-1}/(4\\log(i+2))\\nn\\\\\n&\\ \\leq\\bL_{\\nabla f}\\tK_1^{-\\nu}\\eta_i^{-1}/4+L\\eta_i^{-1}/(4\\log(\\tK_1+2))\\overset{\\eqref{def-tK2}}\\leq\\eta_i^{-1}/4,\\label{t1-e2}\\\\\n12\\zeta_{i+1}\\bL_{\\nabla f}^2& \\overset{\\eqref{def-para2}\\eqref{def-zeta}}=3\\bL_{\\nabla f}^2\\big((i+1)/i\\big)^\\nu\\eta_i^{-1}/\\big(2\\log(i+2)\\log 3\\big)\\nn\\\\\n&\\ \\ \\ \\leq3\\bL_{\\nabla f}^22^\\nu\\eta_i^{-1}/(2\\log(\\tK_1+2))\\overset{\\eqref{def-tK2}}\\leq\\eta_i^{-1}/4,\\nn\n\\end{align}\nwhich imply that \n\\beq\\label{t1-ineq2}\n\\bL_{\\nabla f}+\\rho_iL+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\leq\\eta_i^{-1}/2\\qquad\\forall i\\geq\\tK_1.\n\\eeq\n","sha256_utf8":"a1235ac0076bcba4759a773d69edc1ba624526b35cddea9e1d8c9931a723d939"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"势能下降控制步差与估计误差","source_units":[{"start":72790,"end":73869,"start_line":825,"end_line":832,"raw_tex":"It then follows from the proof of Lemma \\ref{l-pfunc} that \\eqref{l5-e2} holds. Using \\eqref{t1-ineq2} and rearranging the terms of \\eqref{l5-e2} with $k$ replaced by $i$, we obtain that for all $i\\geq\\tK_1$, \n\\begin{align}\n&\\frac{1}{4\\eta_i}\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\frac{\\eta_i}{2}\\bbE\\left[\\|g_i-\\nabla f(x_i)\\|^2\\right]\\nn\\\\\n&\\overset{\\eqref{l5-e2}}\\leq \\bbE\\left[Q_{\\rho_i}(x_i)+\\zeta_i\\|g_i-\\nabla f(x_i)\\|^2\\right]-\\bbE\\left[Q_{\\rho_{i+1}}(x_{i+1})+\\zeta_{i+1}\\|g_{i+1}-\\nabla f(x_{i+1})\\|^2\\right]\\nn\\\\\n&\\quad\\,\\,\\,+\\frac{1}{2}\\left(\\bL_{\\nabla f}+\\rho_iL+12\\zeta_{i+1}\\bL_{\\nabla f}^2-\\eta_i^{-1}/2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\frac{1}{2}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]+3\\sigma^2\\zeta_{i+1}\\alpha_i^2\\nn\\\\\n&\\overset{\\eqref{t1-ineq2}}{\\leq}\\bbE\\left[Q_{\\rho_i}(x_i)+\\zeta_i\\|g_i-\\nabla f(x_i)\\|^2\\right]-\\bbE\\left[Q_{\\rho_{i+1}}(x_{i+1})+\\zeta_{i+1}\\|g_{i+1}-\\nabla f(x_{i+1})\\|^2\\right]\\nn\\\\\n&\\quad\\,\\,\\,+\\frac{1}{2}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]+3\\sigma^2\\zeta_{i+1}\\alpha_i^2.\\label{t1-p2}\n\\end{align}\n","sha256_utf8":"1e5c5588e501718db0dd15e97dc42c3db2491d46a3c9382b903a95eca19244d1"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"均匀抽样与望远镜求和控制残差","source_units":[{"start":73869,"end":77227,"start_line":833,"end_line":848,"raw_tex":"Recall that $\\iota_k$ is the random variable uniformly generated in $\\left\\{\\lceil k/2\\rceil+1,\\dots,k\\right\\}$. In addition, observe from \\eqref{def-para2} that $\\eta_i ^{-1}<\\eta_{k-1}^{-1}$ for all $\\lceil k/2\\rceil \\leq i \\leq k-1$. By these, \\eqref{def-Q}, \\eqref{def-para2}, \\eqref{penalty-bnd}, \\eqref{residual-bnd}, \\eqref{t1-e2} and \\eqref{t1-p2}, one has that for all $k\\geq2\\tK_1$,\n\\begin{align}\n&\\ \\bbE\\left[\\dist^2\\left(0,\\nabla f(x_{\\iota_k})+\\rho_{\\iota_k-1}\\nabla c(x_{\\iota_k})c(x_{\\iota_k})+\\mcN_\\mcX(x_{\\iota_k})\\right)\\right]=\\bbE\\left[\\dist^2\\left(0,\\nabla Q_{\\rho_{\\iota_k-1}}(x_{\\iota_k})+\\mcN_\\mcX(x_{\\iota_k})\\right)\\right]\\nn\\\\\n&\\ =\\frac{1}{k-\\lceil k/2\\rceil}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\bbE\\left[\\dist^2\\left(0,\\nabla Q_{\\rho_i}(x_{i+1})+\\mcN_\\mcX(x_{i+1})\\right)\\right]\\nn\\\\\n& \\overset{\\eqref{residual-bnd}}\\leq  \\frac{3}{k-\\lceil k/2\\rceil}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\left(\\left(\\eta_i^{-2}+(\\bL_{\\nabla f}+\\rho_iL)^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\bbE\\left[\\|g_i-\\nabla f(x_i)\\|^2\\right]\\right)\\nn\\\\\n& \\overset{\\eqref{t1-e2}}{\\leq} \\frac{3}{k-\\lceil k/2\\rceil}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\left((\\eta_i^{-2}+\\eta_i^{-2}/16)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\bbE\\left[\\|g_i-\\nabla f(x_i)\\|^2\\right]\\right)\\nn\\\\\n&\\ \\leq \\frac{51}{8(k-1)}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\left(\\eta_i^{-2}\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+2\\bbE\\left[\\|g_i-\\nabla f(x_i)\\|^2\\right]\\right)\\nn\\\\\n&\\ \\leq \\frac{51}{2(k-1)\\eta_{k-1}}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\left(\\frac{1}{4\\eta_i}\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\frac{\\eta_i}{2}\\bbE\\left[\\|g_i-\\nabla f(x_i)\\|^2\\right]\\right)\\nn\\\\\n& \\overset{\\eqref{t1-p2}}{\\leq}\\frac{51}{2(k-1)\\eta_{k-1}}\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\Big(\\bbE\\left[Q_{\\rho_i}(x_i)+\\zeta_i\\|g_i-\\nabla f(x_i)\\|^2\\right]-\\bbE\\left[Q_{\\rho_{i+1}}(x_{i+1})+\\zeta_{i+1}\\|g_{i+1}-\\nabla f(x_{i+1})\\|^2\\right]\\nn\\\\\n&\\ \\quad\\,\\,\\,+\\frac{1}{2}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]+3\\sigma^2\\zeta_{i+1}\\alpha_i^2\\Big)\\nn\\\\\n&\\ = \\frac{51}{2(k-1)\\eta_{k-1}}\\Bigg(\\bbE\\left[Q_{\\rho_{\\lceil k/2\\rceil}}(x_{\\lceil k/2\\rceil})+\\zeta_{\\lceil k/2\\rceil}\\|g_{\\lceil k/2\\rceil}-\\nabla f(x_{\\lceil k/2\\rceil})\\|^2\\right]-\\bbE\\left[Q_{\\rho_k}(x_k)+\\zeta_k\\|g_k-\\nabla f(x_k)\\|^2\\right]\\nn\\\\\n&\\ \\quad\\,\\,\\,+\\frac{1}{2}\\sum_{i=\\lceil k/2\\rceil}^{k-1}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]+3\\sigma^2\\sum_{i=\\lceil k/2\\rceil}^{k-1}\\zeta_{i+1}\\alpha_i^2\\Bigg)\\nn\\\\\n&\\ \\leq \\frac{51}{2(k-1)\\eta_{k-1}}\\Bigg(Q_1(x_1)-Q_1^*+\\zeta_1\\|g_1-\\nabla f(x_1)\\|^2+\\frac{1}{2}\\sum_{i=1}^{k-1}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]+3\\sigma^2\\sum_{i=1}^{k-1}\\zeta_{i+1}\\alpha_i^2\\nn\\\\\n&\\ \\quad\\,\\,\\,+\\frac{1}{2}\\sum_{i=1}^{\\lceil k/2\\rceil-1}\\left(\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]\\Bigg),\\label{t1-sum2}\n\\end{align}\nwhere the first equality is due to \\eqref{def-Q}, the first inequality follows from taking expectation on both sides of \\eqref{residual-bnd}, the third inequality is due to the fact that $\\lceil k/2\\rceil\\leq(k+1)/2$, the fourth inequality follows from the relation $\\eta_i ^{-1}<\\eta_{k-1}^{-1}$ for all $\\lceil k/2\\rceil \\leq i \\leq k-1$, and the last inequality follows from \\eqref{penalty-bnd} with $k$ replaced by $\\lceil k/2\\rceil$, $\\rho_1=1$, and the fact that $Q_{\\rho_k}(x_k)\\geq Q_1 (x_k) \\geq Q_1^*$.\n","sha256_utf8":"78345037cd955201258937eda8ec52ad3f5fd0c2b289e56429b176572280966d"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"利用截断与非扩张性界定步差","source_units":[{"start":77228,"end":77755,"start_line":850,"end_line":854,"raw_tex":"We next bound each term in the summation \\eqref{t1-sum2}.  Indeed, it follows from \\eqref{cf-bnd}, Assumption \\ref{a1}(iv), the nonexpansiveness of $\\Pi_X$, and the expressions of $x_{k+1}$, $g_k$ and $G_k$ in Algorithm \\ref{alg1} that\n\\begin{align}\n\\|x_{k+1}-x_k\\|^2&=\\|\\Pi_\\mcX(x_k-\\eta_kG_k)-\\Pi_X(x_k)\\|^2\\leq\\eta_k^2\\|G_k\\|^2=\\eta_k^2\\|g_k+\\rho_k\\nabla c(x_k)c(x_k)\\|^2\\nn\\\\\n&\\leq2\\eta_k^2\\left(\\|g_k\\|^2+\\rho_k^2\\|\\nabla c(x_k)c(x_k)\\|^2\\right)\\leq2\\eta_k^2\\left(L_f^2+C_c^2L_c^2\\rho_k^2\\right).\\label{t1-x1}\n\\end{align}\n","sha256_utf8":"27b6f5a17e277b2ebda71fded1ad18f34789745988f802ab314bb68376a7ee62"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"估计罚参数增量、噪声与初始项","source_units":[{"start":77755,"end":80754,"start_line":855,"end_line":870,"raw_tex":"Recall that $\\nu=\\min\\{\\hat\\theta/(\\hat\\theta+2),1/2\\}$ for some $\\hat\\theta\\geq1$ and $\\|c(x_i)\\| \\leq C_c$ for all $i$. By this, Lemma \\ref{l-cnst2}, \\eqref{def-para2}, \\eqref{def-zeta},  \\eqref{t1-x1},  one has that for all $k\\geq2\\tK_1$, \n\\begin{align}\n&\\sum_{i=1}^{\\tK_1-1}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]\\leq C_c^2\\sum_{i=1}^{\\tK_1-1}\\big((i+1)^\\nu-i^\\nu\\big)=C_c^2\\big(\\tK_1^\\nu-1\\big),\\label{sum2-c1}\\\\\n&\\sum_{i=\\tK_1}^{k-1}(\\rho_{i+1}-\\rho_i)\\bbE\\left[\\|c(x_{i+1})\\|^2\\right]\\leq2C_1\\sum_{i=\\tK_1}^{k-1}((i+1)^\\nu-i^\\nu)(i+1)^{-\\frac{2\\nu}{\\theta}}\\leq 2\\nu C_1\\sum_{i=\\tK_1}^{k-1}i^{\\nu-1}(i+1)^{-\\frac{2\\nu}{\\theta}} \\label{sum2-c2} \\\\\n&\\leq C_1\\sum_{i=\\tK_1}^{k-1}i^{-1}(i+1)^{\\nu-\\frac{2\\nu}{\\theta}}\\leq C_1\\max\\big\\{1,k^{\\nu-\\frac{2\\nu}{\\theta}}\\big\\}\\sum_{i=\\tK_1}^{k-1}i^{-1}\\leq C_1\\max\\big\\{1,k^{\\nu-\\frac{2\\nu}{\\theta}}\\big\\}(1+\\log k), \\label{sum2-c3} \\\\\n&\\sum_{i=1}^{k-1}\\zeta_{i+1}\\alpha_i^2\\overset{\\eqref{def-zeta}}=\\frac{1}{2\\log3}\\sum_{i=1}^{k-1}(i+1)^{\\nu}i^{-4\\nu}= \\frac{1}{2\\log3}\\sum_{i=1}^{k-1}i^{-3\\nu}((i+1)/i)^{\\nu} \\nn \\\\\n& \\leq\\frac{\\sqrt{2}}{2\\log3}\\sum_{i=1}^{k-1}i^{-3\\nu} \\leq\\frac{\\sqrt{2}}{2\\log3}\\sum_{i=1}^{k-1}i^{-1}\\leq\\frac{\\sqrt{2}(1+\\log k)}{2\\log3},\\label{sum2-c4a}\\\\\n&\\ \\sum_{i=1}^{\\lceil k/2\\rceil-1}\\left(\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]\\nn\\\\\n&\\ =\\sum_{i=1}^{\\tK_1-1}\\left(\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]+\\sum_{i=\\tK_1}^{\\lceil k/2\\rceil-1}\\Big((\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2)\\nn \\\\ \n&\\ \\quad\\,\\,\\,\\times \\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right]\\Big) \\nn \\\\\n&\\overset{\\eqref{t1-ineq2}}{\\leq}\\sum_{i=1}^{\\tK_1-1}\\left(\\bL_{\\nabla f}+\\rho_iL-\\eta_i^{-1}+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\bbE\\left[\\|x_{i+1}-x_i\\|^2\\right] \\nn\\\\\n&\\overset{\\eqref{t1-x1}}{\\leq}2\\sum_{i=1}^{\\tK_1-1}\\left(\\bL_{\\nabla f}+\\rho_iL+12\\zeta_{i+1}\\bL_{\\nabla f}^2\\right)\\eta_i^2\\left(L_f^2+C_c^2L_c^2\\rho_i^2\\right)\\nn\\\\\n&\\overset{\\eqref{def-para2}}{\\leq}\\frac{1}{8}(1+\\log\\tK_1)\\big(\\bL_{\\nabla f}+\\tK_1^{\\frac{1}{2}}L+6\\tK_1^{\\frac{1}{2}}\\bL_{\\nabla f}^2\\big)(L_f^2+C_c^2L_c^2\\tK_1),\\label{sum2-c4}\n\\end{align}\nwhere the inequality in \\eqref{sum2-c1} follows from \\eqref{def-para2}, $\\|c(x_i)\\| \\leq C_c$ for all $i$, the inequalities in \\eqref{sum2-c2} are due to \\eqref{def-para2}, Lemma \\ref{l-cnst2},  and $(i+1)^\\nu-i^\\nu\\leq \\nu i^{\\nu-1}$ for all $i\\geq1$ thanks to the concavity of $t^\\nu$, the inequalities in \\eqref{sum2-c4a} are due to $((i+1)/i)^\\nu\\leq\\sqrt{2}$ for all $i\\geq1$ and $\\nu\\geq1/3$, and the last inequality in \\eqref{sum2-c4} is due to the relations $\\rho_i \\leq \\tK_1^{1/2}$ and $\\zeta_i \\leq \\tK_1^{1/2}/2$ for $1\\leq i\\leq \\tK_1-1$ and\n$\\sum_{i=1}^{\\tK_1-1}\\eta_i^2 \\leq \\sum_{i=1}^{\\tK_1-1}i^{-1}/16\\leq (1+\\log\\tK_1)/16$ thanks to $\\nu\\leq 1/2$ and the choice of $\\rho_i$, $\\eta_i$ and $\\zeta_i$ in \\eqref{def-para2} and \\eqref{def-zeta}.\n","sha256_utf8":"3db125e2aca137055daa01b9685f111bb07de375805a373795a48b4b989d82e1"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"代入各项界获得期望驻点收敛率","source_units":[{"start":80755,"end":81461,"start_line":872,"end_line":878,"raw_tex":"Using \\eqref{def-para2}, \\eqref{def-zeta}, \\eqref{t1-sum2}, \\eqref{sum2-c1}, \\eqref{sum2-c3}, \\eqref{sum2-c4a} and \\eqref{sum2-c4}, we have\n\\begin{align*}\n&\\bbE\\left[\\dist^2\\left(0,\\nabla f(x_{\\iota_k})+\\rho_{\\iota_k-1}\\nabla c(x_{\\iota_k})c(x_{\\iota_k})+\\mcN_\\mcX(x_{\\iota_k})\\right)\\right]\\\\\n&\\leq \\frac{102\\log(k+1)}{(k-1)^{1-\\nu}}\\Bigg(Q_1(x_1)-Q_1^*+\\frac{\\|g_1-\\nabla f(x_1)\\|^2}{2\\log 3}+\\frac{1}{2}C_1\\max\\{1,k^{\\nu-\\frac{2\\nu}{\\theta}}\\}(1+\\log k)\\nn \\\\ \n&\\quad+\\frac{1}{2}C_c^2\\big(\\tK_1^{\\nu}-1\\big)\n+\\frac{3\\sqrt{2}\\sigma^2(1+\\log k)}{2\\log3}+\\frac{1}{16}(1+\\log\\tK_1)\\big(\\bL_{\\nabla f}+\\tK_1^{\\frac{1}{2}}L+6\\tK_1^{\\frac{1}{2}}\\bL_{\\nabla f}^2\\big)(L_f^2+C_c^2L_c^2\\tK_1)\\Bigg).\n\\end{align*}\n","sha256_utf8":"56f253cf99eb724cf799c973bd84537256bfc0bc67de6a8bbffc130e71135080"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"结合抽样位置得到确定可行性界","source_units":[{"start":81461,"end":81659,"start_line":879,"end_line":880,"raw_tex":"By this, \\eqref{def-Q}, $\\iota_k>\\lceil k/2\\rceil\\geq\\tK_1$ for all $k\\geq2\\tK_1$, and Lemma \\ref{l-cnst2} with $k$ replaced by $\\iota_k$, one can see that Theorem \\ref{thm:storm} holds.\n\\end{proof}","sha256_utf8":"640f0942073781f6a212afbcbdddd90423ec92665e368cfa03a974f9d0374968"}]},{"id":"S045","semantic_type":"external_fact","kind":"external_fact","title":"闭凸投影的标准性质","source_units":[],"declared_external":true,"declaration_text":["对非空闭凸集 C，p=Π_C(z) 当且仅当 z-p∈N_C(p)，等价地 ⟨p-z,y-p⟩≥0 对所有 y∈C 成立；并且 ‖Π_C(u)-Π_C(v)‖≤‖u-v‖。"]},{"id":"S046","semantic_type":"external_fact","kind":"external_fact","title":"平滑函数的下降引理","source_units":[],"declared_external":true,"declaration_text":["若可微函数 φ 的梯度在凸集上为 L-Lipschitz，则 φ(y)≤φ(x)+⟨∇φ(x),y-x⟩+(L/2)‖y-x‖²。"]},{"id":"S047","semantic_type":"external_fact","kind":"external_fact","title":"平方范数与Young不等式","source_units":[],"declared_external":true,"declaration_text":["平方范数是凸函数；对 r 个向量有 ‖Σ_j v_j‖²≤rΣ_j‖v_j‖²；对任意 τ>0，有 ⟨u,v⟩≤(τ/2)‖u‖²+(1/(2τ))‖v‖²。"]},{"id":"S048","semantic_type":"external_fact","kind":"external_fact","title":"条件期望与Jensen法则","source_units":[],"declared_external":true,"declaration_text":["条件期望满足塔式法则和可测因子提出法则，因此条件均值为零的随机向量与条件可测向量的内积期望为零；Jensen 不等式给出 ‖E[Y]‖²≤E[‖Y‖²]。独立均匀抽取有限指标时，相应期望等于各指标期望的算术平均。"]},{"id":"S049","semantic_type":"external_fact","kind":"external_fact","title":"幂函数切线与积分估计","source_units":[],"declared_external":true,"declaration_text":["可微凸函数位于其切线之上，可微凹函数位于其切线之下；初等求导可检验所用幂函数与对数函数组合的单调性；正递减函数的离散和可用积分比较，特别地 Σ_{i=1}^N i^{-1}≤1+log N。"]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S003","target_id":"S008","kind":"supports"},{"source_id":"S004","target_id":"S008","kind":"supports"},{"source_id":"S004","target_id":"S015","kind":"supports"},{"source_id":"S005","target_id":"S030","kind":"supports"},{"source_id":"S005","target_id":"S031","kind":"supports"},{"source_id":"S005","target_id":"S036","kind":"supports"},{"source_id":"S005","target_id":"S040","kind":"supports"},{"source_id":"S005","target_id":"S044","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S015","kind":"supports"},{"source_id":"S006","target_id":"S020","kind":"supports"},{"source_id":"S006","target_id":"S026","kind":"supports"},{"source_id":"S006","target_id":"S027","kind":"supports"},{"source_id":"S006","target_id":"S040","kind":"supports"},{"source_id":"S006","target_id":"S041","kind":"supports"},{"source_id":"S006","target_id":"S042","kind":"supports"},{"source_id":"S007","target_id":"S011","kind":"supports"},{"source_id":"S007","target_id":"S015","kind":"supports"},{"source_id":"S007","target_id":"S016","kind":"supports"},{"source_id":"S007","target_id":"S025","kind":"supports"},{"source_id":"S007","target_id":"S030","kind":"supports"},{"source_id":"S007","target_id":"S036","kind":"supports"},{"source_id":"S007","target_id":"S041","kind":"supports"},{"source_id":"S008","target_id":"S014","kind":"supports"},{"source_id":"S008","target_id":"S017","kind":"supports"},{"source_id":"S008","target_id":"S020","kind":"supports"},{"source_id":"S008","target_id":"S025","kind":"supports"},{"source_id":"S008","target_id":"S026","kind":"supports"},{"source_id":"S008","target_id":"S030","kind":"supports"},{"source_id":"S008","target_id":"S031","kind":"supports"},{"source_id":"S008","target_id":"S036","kind":"supports"},{"source_id":"S008","target_id":"S041","kind":"supports"},{"source_id":"S009","target_id":"S010","kind":"supports"},{"source_id":"S009","target_id":"S027","kind":"supports"},{"source_id":"S010","target_id":"S027","kind":"supports"},{"source_id":"S010","target_id":"S030","kind":"supports"},{"source_id":"S010","target_id":"S036","kind":"supports"},{"source_id":"S011","target_id":"S014","kind":"supports"},{"source_id":"S011","target_id":"S015","kind":"supports"},{"source_id":"S011","target_id":"S016","kind":"supports"},{"source_id":"S011","target_id":"S017","kind":"supports"},{"source_id":"S011","target_id":"S020","kind":"supports"},{"source_id":"S014","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S015","target_id":"S013","kind":"supports"},{"source_id":"S016","target_id":"S013","kind":"supports"},{"source_id":"S017","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S019","kind":"supports"},{"source_id":"S022","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S027","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S023","target_id":"S033","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S032","target_id":"S029","kind":"supports"},{"source_id":"S033","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S033","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S028","target_id":"S040","kind":"supports"},{"source_id":"S034","target_id":"S040","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S018","target_id":"S042","kind":"supports"},{"source_id":"S038","target_id":"S042","kind":"supports"},{"source_id":"S040","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S018","target_id":"S044","kind":"supports"},{"source_id":"S038","target_id":"S002","kind":"supports"},{"source_id":"S039","target_id":"S002","kind":"supports"},{"source_id":"S040","target_id":"S002","kind":"supports"},{"source_id":"S041","target_id":"S002","kind":"supports"},{"source_id":"S042","target_id":"S002","kind":"supports"},{"source_id":"S043","target_id":"S002","kind":"supports"},{"source_id":"S044","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S045","target_id":"S014","kind":"supports"},{"source_id":"S045","target_id":"S015","kind":"supports"},{"source_id":"S045","target_id":"S025","kind":"supports"},{"source_id":"S045","target_id":"S030","kind":"supports"},{"source_id":"S045","target_id":"S036","kind":"supports"},{"source_id":"S045","target_id":"S041","kind":"supports"},{"source_id":"S046","target_id":"S016","kind":"supports"},{"source_id":"S046","target_id":"S031","kind":"supports"},{"source_id":"S047","target_id":"S015","kind":"supports"},{"source_id":"S047","target_id":"S027","kind":"supports"},{"source_id":"S047","target_id":"S031","kind":"supports"},{"source_id":"S047","target_id":"S037","kind":"supports"},{"source_id":"S047","target_id":"S041","kind":"supports"},{"source_id":"S048","target_id":"S010","kind":"supports"},{"source_id":"S048","target_id":"S026","kind":"supports"},{"source_id":"S048","target_id":"S027","kind":"supports"},{"source_id":"S048","target_id":"S040","kind":"supports"},{"source_id":"S049","target_id":"S021","kind":"supports"},{"source_id":"S049","target_id":"S022","kind":"supports"},{"source_id":"S049","target_id":"S032","kind":"supports"},{"source_id":"S049","target_id":"S042","kind":"supports"}]}}
{"record_id":"2410.01382v2","paper":{"title":"Analyzing the speed of convergence in nonsmooth optimization via the Goldstein subdifferential with application to descent methods","authors":["Bennet Gebken"],"arxiv_id":"2410.01382","version":"v2","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-025-02748-8","abstract_url":"https://arxiv.org/abs/2410.01382v2","pdf_url":"https://arxiv.org/pdf/2410.01382v2"},"context":{"raw_tex":"    In this section, we briefly introduce the basics of nonsmooth analysis. For a more thorough introduction, we refer to \\cite{C1990}. To this end, let $f : \\R^n \\rightarrow \\R$ be locally Lipschitz continuous and let $\\Omega$ be the set of points in which $f$ is not differentiable. By Rademacher's Theorem, $\\Omega$ is a null set. The \\emph{Clarke subdifferential of} $f$ \\emph{at} $x \\in \\R^n$ is defined as\n\n\n\\begin{equation*}\n        \\begin{aligned}\n            \\partial f(x) := \\conv &\\left( \\left\\{ \\xi \\in \\R^n : \\exists (x^j)_j \\in \\R^n \\setminus \\Omega \\text{ with } \\lim_{j \\rightarrow \\infty} x^j = x \\text{ and } \\right. \\right. \\\\\n            &\\left. \\left. \\lim_{j \\rightarrow \\infty} \\nabla f(x^j) = \\xi \\right\\} \\right),\n        \\end{aligned}\n    \\end{equation*}\n\n    To circumvent some of the practical issues of the Clarke subdifferential (cf. \\cite{BGLS2006}, Section 9.1), the \\emph{Goldstein} $\\eps$\\emph{-subdifferential} may be used instead. For $x \\in \\R^n$ and $\\eps \\geq 0$, it is defined as the compact set\n\n\n\\begin{align*}\n        \\partial_\\eps f(x) := \\conv \\left( \\bigcup_{y \\in \\Bcl_\\eps(x)} \\partial f(y) \\right),\n    \\end{align*}\n\n    where $\\Bcl_\\eps(x) := \\{ y \\in \\R^n : \\| y - x \\| \\leq \\eps \\}$ and $\\| \\cdot \\|$ is the Euclidean norm. For $\\eps, \\delta \\geq 0$ we say that $x$ is $(\\eps,\\delta)$\\emph{-critical}, if $\\min(\\| \\partial_\\eps f(x) \\|) \\leq \\delta$, i.e., if there is some $\\xi \\in \\partial_\\eps f(x)$ with $\\|\\xi \\| \\leq \\delta$. Clearly, $(\\eps,\\delta)$-criticality is a weaker optimality condition than criticality. However, upper semicontinuity of $\\partial f$ implies that if there are sequences $(x^j)_j \\in \\R^n$, $(\\eps_j)_j, (\\delta_j)_j \\in \\R^{\\geq 0}$ with $x^j \\rightarrow x^*$, $\\eps_j \\rightarrow 0$ and $\\delta_j \\rightarrow 0$ such that $x^j$ is $(\\eps_j,\\delta_j)$-critical for all $j \\in \\N$, then $x^*$ is critical (cf. Lemma 4.4.4 in \\cite{G2022}). In addition to subdifferentials, we also require directional derivatives of $f$. To assure that they exist and are well-behaved, we occasionally assume that $f$ is \\emph{semismooth} \\cite{M1977}, which means that $f$ is locally Lipschitz continuous and for any $x \\in \\R^n$ and $d \\in \\R^n$, the limit\n    \\begin{align} \\label{eq:def_semismooth}\n        \\lim_{\\xi \\in \\partial f(x + t d'), t \\searrow 0, d' \\rightarrow d} \\langle \\xi, d' \\rangle\n    \\end{align}\n    exists. If $f$ is semismooth, then for any $d \\in \\R^n$, the \\emph{directional derivative}\n    \\begin{align*}\n        f'(x,d) = \\lim_{t \\searrow 0} \\frac{f(x + t d) - f(x)}{t}\n    \\end{align*}\n    exists and equals the limit \\eqref{eq:def_semismooth}. A large class of semismooth functions are the \\emph{piecewise $p$-times differentiable} functions \\cite{S2012}, which are continuous functions $f : \\R^n \\rightarrow \\R$ for which there are $p$-times continuously differentiable functions $f_1, \\dots, f_m : \\R^n \\rightarrow \\R$, called \\emph{selection functions}, such that $f(x) \\in \\{ f_1(x), \\dots, f_m(x) \\}$ for all $x \\in \\R^n$. For such a function, the set $A(x) := \\{ i \\in \\{1,\\dots,m\\} : f(x) = f_i(x) \\}$ is called the \\emph{active set}, and the Clarke subdifferential satisfies $\\partial f(x) \\subseteq \\conv(\\{ \\nabla f_i(x) : i \\in A(x) \\})$.\n\n\n\\begin{definition} \\label{def:order_minimum}\n        A point $x^* \\in \\R^n$ is called a \\emph{minimum of order} $p \\in \\N$ \\emph{with constant} $\\beta > 0$, if there is some open set $U \\subseteq\\R^n$ with $x^* \\in U$ such that\n        \\begin{align*}\n            f(x) \\geq f(x^*) + \\beta \\| x - x^* \\|^p \\quad \\forall x \\in U.\n        \\end{align*}\n    \\end{definition}\n\n\\begin{align} \\label{eq:p_order_min_semismooth}\n        \\liminf_{\\xi \\in \\partial f(x^* + t d'), t \\searrow 0, d' \\rightarrow d} \\frac{\\langle \\xi,  d' \\rangle}{t^{p-1}} \\geq \\beta \\| d \\|^p \\quad \\forall d \\in \\R^n \\setminus \\{ 0 \\}.\n    \\end{align}","units":[{"start":11305,"end":11717,"start_line":105,"end_line":105,"raw_tex":"    In this section, we briefly introduce the basics of nonsmooth analysis. For a more thorough introduction, we refer to \\cite{C1990}. To this end, let $f : \\R^n \\rightarrow \\R$ be locally Lipschitz continuous and let $\\Omega$ be the set of points in which $f$ is not differentiable. By Rademacher's Theorem, $\\Omega$ is a null set. The \\emph{Clarke subdifferential of} $f$ \\emph{at} $x \\in \\R^n$ is defined as\n","sha256_utf8":"dd14d41f0adade5cd5689aa1b04f782e9d823db6f0557f67b0fa47d18d3b1829"},{"start":11721,"end":12087,"start_line":106,"end_line":111,"raw_tex":"\\begin{equation*}\n        \\begin{aligned}\n            \\partial f(x) := \\conv &\\left( \\left\\{ \\xi \\in \\R^n : \\exists (x^j)_j \\in \\R^n \\setminus \\Omega \\text{ with } \\lim_{j \\rightarrow \\infty} x^j = x \\text{ and } \\right. \\right. \\\\\n            &\\left. \\left. \\lim_{j \\rightarrow \\infty} \\nabla f(x^j) = \\xi \\right\\} \\right),\n        \\end{aligned}\n    \\end{equation*}","sha256_utf8":"4c412d74eb127763db68040f184e48e733d6b4ac5978492c416be829b4593a20"},{"start":12556,"end":12810,"start_line":116,"end_line":116,"raw_tex":"    To circumvent some of the practical issues of the Clarke subdifferential (cf. \\cite{BGLS2006}, Section 9.1), the \\emph{Goldstein} $\\eps$\\emph{-subdifferential} may be used instead. For $x \\in \\R^n$ and $\\eps \\geq 0$, it is defined as the compact set\n","sha256_utf8":"06ba24ea63d4bb3d36424521b813db57590ee13bac3c016947552d34e36fed64"},{"start":12814,"end":12940,"start_line":117,"end_line":119,"raw_tex":"\\begin{align*}\n        \\partial_\\eps f(x) := \\conv \\left( \\bigcup_{y \\in \\Bcl_\\eps(x)} \\partial f(y) \\right),\n    \\end{align*}","sha256_utf8":"ce37f8687ad927df6360eb6dfaaf0a7a986468bc58fd5308c74023924e0913c3"},{"start":12941,"end":15022,"start_line":120,"end_line":128,"raw_tex":"    where $\\Bcl_\\eps(x) := \\{ y \\in \\R^n : \\| y - x \\| \\leq \\eps \\}$ and $\\| \\cdot \\|$ is the Euclidean norm. For $\\eps, \\delta \\geq 0$ we say that $x$ is $(\\eps,\\delta)$\\emph{-critical}, if $\\min(\\| \\partial_\\eps f(x) \\|) \\leq \\delta$, i.e., if there is some $\\xi \\in \\partial_\\eps f(x)$ with $\\|\\xi \\| \\leq \\delta$. Clearly, $(\\eps,\\delta)$-criticality is a weaker optimality condition than criticality. However, upper semicontinuity of $\\partial f$ implies that if there are sequences $(x^j)_j \\in \\R^n$, $(\\eps_j)_j, (\\delta_j)_j \\in \\R^{\\geq 0}$ with $x^j \\rightarrow x^*$, $\\eps_j \\rightarrow 0$ and $\\delta_j \\rightarrow 0$ such that $x^j$ is $(\\eps_j,\\delta_j)$-critical for all $j \\in \\N$, then $x^*$ is critical (cf. Lemma 4.4.4 in \\cite{G2022}). In addition to subdifferentials, we also require directional derivatives of $f$. To assure that they exist and are well-behaved, we occasionally assume that $f$ is \\emph{semismooth} \\cite{M1977}, which means that $f$ is locally Lipschitz continuous and for any $x \\in \\R^n$ and $d \\in \\R^n$, the limit\n    \\begin{align} \\label{eq:def_semismooth}\n        \\lim_{\\xi \\in \\partial f(x + t d'), t \\searrow 0, d' \\rightarrow d} \\langle \\xi, d' \\rangle\n    \\end{align}\n    exists. If $f$ is semismooth, then for any $d \\in \\R^n$, the \\emph{directional derivative}\n    \\begin{align*}\n        f'(x,d) = \\lim_{t \\searrow 0} \\frac{f(x + t d) - f(x)}{t}\n    \\end{align*}\n    exists and equals the limit \\eqref{eq:def_semismooth}. A large class of semismooth functions are the \\emph{piecewise $p$-times differentiable} functions \\cite{S2012}, which are continuous functions $f : \\R^n \\rightarrow \\R$ for which there are $p$-times continuously differentiable functions $f_1, \\dots, f_m : \\R^n \\rightarrow \\R$, called \\emph{selection functions}, such that $f(x) \\in \\{ f_1(x), \\dots, f_m(x) \\}$ for all $x \\in \\R^n$. For such a function, the set $A(x) := \\{ i \\in \\{1,\\dots,m\\} : f(x) = f_i(x) \\}$ is called the \\emph{active set}, and the Clarke subdifferential satisfies $\\partial f(x) \\subseteq \\conv(\\{ \\nabla f_i(x) : i \\in A(x) \\})$.\n","sha256_utf8":"1c5efa01f89f18d81cfb1ea8c1fa90bdb4600b55b520f28baf58ffc58776e9d8"},{"start":22935,"end":23303,"start_line":198,"end_line":203,"raw_tex":"\\begin{definition} \\label{def:order_minimum}\n        A point $x^* \\in \\R^n$ is called a \\emph{minimum of order} $p \\in \\N$ \\emph{with constant} $\\beta > 0$, if there is some open set $U \\subseteq\\R^n$ with $x^* \\in U$ such that\n        \\begin{align*}\n            f(x) \\geq f(x^*) + \\beta \\| x - x^* \\|^p \\quad \\forall x \\in U.\n        \\end{align*}\n    \\end{definition}","sha256_utf8":"6befd426a5f318058de4c27cf79db8e7c8b5e6db3071d87a6759dbb58e85726e"},{"start":32050,"end":32300,"start_line":322,"end_line":324,"raw_tex":"\\begin{align} \\label{eq:p_order_min_semismooth}\n        \\liminf_{\\xi \\in \\partial f(x^* + t d'), t \\searrow 0, d' \\rightarrow d} \\frac{\\langle \\xi,  d' \\rangle}{t^{p-1}} \\geq \\beta \\| d \\|^p \\quad \\forall d \\in \\R^n \\setminus \\{ 0 \\}.\n    \\end{align}","sha256_utf8":"d15e54a743923413c103ff67edd4380ccdfb6f7b86a634a5e1f63cec7a51dc8f"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem} \\label{thm:xj_speed_higher_order}\n        Let $f : \\R^n \\rightarrow \\R$ be locally Lipschitz continuous. Let $(x^j)_j \\in \\R^n$, $(\\eps_j)_j \\in \\R^{\\geq 0}$ and $(\\delta_j)_j \\in \\R^{\\geq 0}$ such that $x^j \\rightarrow x^* \\in \\R^n$, $\\eps_j \\rightarrow 0$ and $\\min(\\| \\partial_{\\eps_j} f(x^j) \\|) \\leq \\delta_j$ for all $j \\in \\N$.\n        Assume that $x^*$ is a minimum of order $p \\in \\N$ with constant $\\beta > 0$ and that \\eqref{eq:p_order_min_semismooth} holds in $x^*$.\n        \\begin{enumerate}\n            \\item[(i)] If $p = 1$ and $\\delta_j \\rightarrow \\Bar{\\delta} < \\beta$, then there are $M > 0$ and $N \\in \\N$ such that\n            \\begin{align} \\label{eq:xj_speed_higher_order_1}\n                \\| x^j - x^* \\| \\leq M \\eps_j \\quad \\forall j > N.\n            \\end{align}\n            \\item[(ii)] If $p \\geq 2$ then there are $M > 0$ and $N \\in \\N$ such that\n            \\begin{align} \\label{eq:xj_speed_higher_order_2}\n                \\| x^j - x^* \\| \\leq M \\max(\\eps_j^{\\frac{1}{p}}, \\delta_j^{\\frac{1}{p-1}}) \\quad \\forall j > N.\n            \\end{align}\n        \\end{enumerate}\n    \\end{theorem}","units":[{"start":52996,"end":54125,"start_line":555,"end_line":568,"raw_tex":"\\begin{theorem} \\label{thm:xj_speed_higher_order}\n        Let $f : \\R^n \\rightarrow \\R$ be locally Lipschitz continuous. Let $(x^j)_j \\in \\R^n$, $(\\eps_j)_j \\in \\R^{\\geq 0}$ and $(\\delta_j)_j \\in \\R^{\\geq 0}$ such that $x^j \\rightarrow x^* \\in \\R^n$, $\\eps_j \\rightarrow 0$ and $\\min(\\| \\partial_{\\eps_j} f(x^j) \\|) \\leq \\delta_j$ for all $j \\in \\N$.\n        Assume that $x^*$ is a minimum of order $p \\in \\N$ with constant $\\beta > 0$ and that \\eqref{eq:p_order_min_semismooth} holds in $x^*$.\n        \\begin{enumerate}\n            \\item[(i)] If $p = 1$ and $\\delta_j \\rightarrow \\Bar{\\delta} < \\beta$, then there are $M > 0$ and $N \\in \\N$ such that\n            \\begin{align} \\label{eq:xj_speed_higher_order_1}\n                \\| x^j - x^* \\| \\leq M \\eps_j \\quad \\forall j > N.\n            \\end{align}\n            \\item[(ii)] If $p \\geq 2$ then there are $M > 0$ and $N \\in \\N$ such that\n            \\begin{align} \\label{eq:xj_speed_higher_order_2}\n                \\| x^j - x^* \\| \\leq M \\max(\\eps_j^{\\frac{1}{p}}, \\delta_j^{\\frac{1}{p-1}}) \\quad \\forall j > N.\n            \\end{align}\n        \\end{enumerate}\n    \\end{theorem}","sha256_utf8":"eb03bb0248052a6a68c219c47b00d752b1098bd8e05965b1d90651fc428fd31c"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{proof}\n        Assume that (i) or (ii) do not hold. Then in both cases there is a subsequence $(j_l)_l \\in \\N$ with $x^{j_l} \\neq x^*$ for all $l \\in \\N$,\n        \\begin{align} \\label{eq:proof_thm_xj_speed_higher_order_1}\n            \\lim_{l \\rightarrow \\infty} \\frac{\\eps_{j_l}}{\\| x^{j_l} - x^* \\|^{p}} = 0\n            \\quad \\text{and} \\quad\n             \\lim_{l \\rightarrow \\infty} \\frac{\\delta_{j_l}}{\\| x^{j_l} - x^* \\|^{p-1}} < \\beta.\n        \\end{align}\n        (In case (ii) does not hold, the second limit actually vanishes.) \\\\\n        \\textbf{Step 1:} For $l \\in \\N$ let $g^l \\in \\partial_{\\eps_{j_l}} f(x^{j_l})$ with $\\| g^l \\| \\leq \\delta_{j_l}$. By Carathéodory's theorem, for $i \\in \\{1,\\dots,n+1\\}$, there are $y_i^l \\in \\Bcl_{\\eps_{j_l}}(x^{j_l})$, $\\xi_i^l \\in \\partial f(y_i^l) \\subseteq \\partial_{\\eps_{j_l}} f(x^{j_l})$ and $\\alpha_i^l \\geq 0$ with $\\sum_{i = 1}^{n+1} \\alpha_i^l = 1$ and\n        \\begin{align*}\n            g^l = \\sum_{i = 1}^{n+1} \\alpha_i^l \\xi_i^l.\n        \\end{align*}\n        Since $\\| y_i^l - x^{j_l} \\| \\leq \\eps_{j_l}$ it holds $\\lim_{l \\rightarrow \\infty} y_i^l = x^*$ for all $i \\in \\{1,\\dots,n+1\\}$. By compactness of the $\\eps$-subdifferential at $x^*$ (cf. \\cite{G1977}, Proposition 2.3), we can assume w.l.o.g. that $\\lim_{l \\rightarrow \\infty} \\xi_i^l = \\Bar{\\xi}_i \\in \\R^n$ for all $i \\in \\{1,\\dots,n+1\\}$. By \\cite{C1990}, Proposition 2.1.5, it follows that $\\Bar{\\xi}_i \\in \\partial f(x^*)$. Furthermore, by compactness of the set of convex coefficients, we can assume w.l.o.g. that $\\lim_{l \\rightarrow \\infty} \\alpha_i^l = \\Bar{\\alpha}_i$ for all $i \\in \\{1,\\dots,n+1\\}$, such that\n        \\begin{align*}\n            \\Bar{g} := \\sum_{i = 1}^{n+1} \\Bar{\\alpha}_i \\Bar{\\xi}_i \\in \\partial f(x^*)\n        \\end{align*}\n        is the limit of $(g^l)_l$. (In case (ii) we have $\\Bar{g} = 0$.) \\\\\n        \\textbf{Step 2:} Consider the sequences $(d^l)_l$ and $(d_i^l)_l$ given by\n        \\begin{align*}\n            d^l := \\frac{x^{j_l} - x^*}{\\| x^{j_l} - x^* \\|} \\quad \\text{and} \\quad d_i^l := \\frac{y_i^l - x^*}{\\| x^{j_l} - x^* \\|}, \\ i \\in \\{1,\\dots,n+1\\}.\n        \\end{align*}\n        By compactness, $(d^l)_l$ must have an accumulation point $\\Bar{d} \\in \\R^n$ with $\\| \\Bar{d} \\| = 1$. Assume w.l.o.g. that $\\Bar{d}$ is the limit of $(d^l)_l$. \n        For all $i \\in \\{1,\\dots,n+1\\}$, it holds\n        \\begin{align*}\n            \\| d_i^l - d^l \\| \n            = \\frac{\\| y_i^l - x^* - (x^{j_l} - x^*) \\|}{\\| x^{j_l} - x^* \\|}\n            = \\frac{\\| y_i^l - x^{j_l} \\|}{\\| x^{j_l} - x^* \\|}\n            \\leq \\frac{\\eps_{j_l}}{\\| x^{j_l} - x^* \\|},\n        \\end{align*}\n        and combined with \\eqref{eq:proof_thm_xj_speed_higher_order_1}, we obtain\n        \\begin{align} \\label{eq:proof_thm_xj_speed_higher_order_2}\n            \\frac{\\| d_i^l - d^l \\|}{\\| x^{j_l} - x^* \\|^{p-1}} \\leq \\frac{\\eps_{j_l}}{\\| x^{j_l} - x^* \\|^{p}} \\xrightarrow[]{l \\rightarrow \\infty} 0.\n        \\end{align}\n        In particular, $\\lim_{l \\rightarrow \\infty} d_i^l = \\bar{d}$ for all $i \\in \\{1,\\dots,n+1\\}$.\\\\ \n        \\textbf{Step 3:} By construction we have \n        \\begin{align*}\n            \\xi_i^l \\in \\partial f(y_i^l) \n            = \\partial f \\left( x^* + \\| x^{j_l} - x^* \\| \\frac{y_i^l - x^*}{\\| x^{j_l} - x^* \\|} \\right) \n            = \\partial f(x^* + \\| x^{j_l} - x^* \\| d_i^l).\n        \\end{align*}\n        Assume w.l.o.g. that $x^{j_l} \\in U$ with $U$ as in Definition \\ref{def:order_minimum}. Combination of \\eqref{eq:p_order_min_semismooth} and \\eqref{eq:proof_thm_xj_speed_higher_order_2} (and boundedness of $(\\xi_i^l)_l$) yields\n        \\begin{equation}\n            \\begin{aligned} \\label{eq:proof_thm_xj_speed_higher_order_3}\n                \\liminf_{l \\rightarrow \\infty} \\frac{\\langle \\xi_i^l, d^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}}\n                &= \\liminf_{l \\rightarrow \\infty} \\left( \\frac{\\langle \\xi_i^l, d_i^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}} + \\frac{\\langle \\xi_i^l, d^l - d_i^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}} \\right)\\\\\n                &\\stackrel{\\eqref{eq:proof_thm_xj_speed_higher_order_2}}{=} \\liminf_{l \\rightarrow \\infty} \\frac{\\langle \\xi_i^l, d_i^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}} \n                \\stackrel{\\eqref{eq:p_order_min_semismooth}}{\\geq} \\beta \\| \\Bar{d} \\|^p = \\beta\n            \\end{aligned}\n        \\end{equation}\n        for all $i \\in \\{1,\\dots,n+1\\}$. Furthermore note that\n        \\begin{equation} \\label{eq:proof_thm_xj_speed_higher_order_5}\n            \\begin{aligned}\n                \\delta_{j_l}\n                &\\geq \\| g^l \\| \n                = \\left\\| \\sum_{i = 1}^{n+1} \\alpha_i^l \\xi_i^l \\right\\|\n                = \\left\\| \\sum_{i = 1}^{n+1} \\alpha_i^l \\xi_i^l \\right\\| \\| d^l \\| \\\\\n                &\\geq \\langle \\sum_{i = 1}^{n+1} \\alpha_i^l \\xi_i^l, d^l \\rangle\n                = \\sum_{i = 1}^{n+1} \\alpha_i^l \\langle \\xi_i^l, d^l \\rangle \\quad \\forall l \\in \\N.\n            \\end{aligned}\n        \\end{equation}\n        Division of \\eqref{eq:proof_thm_xj_speed_higher_order_5} by $\\| x^{j_l} - x^* \\|^{p-1}$ and combination with \\eqref{eq:proof_thm_xj_speed_higher_order_1} and \\eqref{eq:proof_thm_xj_speed_higher_order_3} yields\n        \\begin{equation*}\n            \\begin{aligned}\n                \\beta \n                &\\stackrel{\\eqref{eq:proof_thm_xj_speed_higher_order_1}}{>} \\liminf_{l \\rightarrow \\infty} \\frac{\\delta_{j_l}}{\\| x^{j_l} - x^* \\|^{p-1}}\n                \\stackrel{\\eqref{eq:proof_thm_xj_speed_higher_order_5}}{\\geq} \\liminf_{l \\rightarrow \\infty} \\frac{ \\sum_{i = 1}^{n+1} \\alpha_i^l \\langle \\xi_i^l, d^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}} \\\\\n                &= \\liminf_{l \\rightarrow \\infty} \\sum_{i = 1}^{n+1} \\alpha_i^l \\frac{\\langle \\xi_i^l, d^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}}\n                \\geq \\sum_{i = 1}^{n+1} \\liminf_{l \\rightarrow \\infty} \\alpha_i^l \\frac{\\langle \\xi_i^l, d^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}} \\\\\n                &\\stackrel{\\eqref{eq:proof_thm_xj_speed_higher_order_3}}{\\geq} \\sum_{i = 1}^{n+1} \\Bar{\\alpha}_i \\beta = \\beta,\n            \\end{aligned}\n        \\end{equation*}\n        which is a contradiction.\n    \\end{proof}","units":[{"start":54130,"end":60251,"start_line":569,"end_line":641,"raw_tex":"\\begin{proof}\n        Assume that (i) or (ii) do not hold. Then in both cases there is a subsequence $(j_l)_l \\in \\N$ with $x^{j_l} \\neq x^*$ for all $l \\in \\N$,\n        \\begin{align} \\label{eq:proof_thm_xj_speed_higher_order_1}\n            \\lim_{l \\rightarrow \\infty} \\frac{\\eps_{j_l}}{\\| x^{j_l} - x^* \\|^{p}} = 0\n            \\quad \\text{and} \\quad\n             \\lim_{l \\rightarrow \\infty} \\frac{\\delta_{j_l}}{\\| x^{j_l} - x^* \\|^{p-1}} < \\beta.\n        \\end{align}\n        (In case (ii) does not hold, the second limit actually vanishes.) \\\\\n        \\textbf{Step 1:} For $l \\in \\N$ let $g^l \\in \\partial_{\\eps_{j_l}} f(x^{j_l})$ with $\\| g^l \\| \\leq \\delta_{j_l}$. By Carathéodory's theorem, for $i \\in \\{1,\\dots,n+1\\}$, there are $y_i^l \\in \\Bcl_{\\eps_{j_l}}(x^{j_l})$, $\\xi_i^l \\in \\partial f(y_i^l) \\subseteq \\partial_{\\eps_{j_l}} f(x^{j_l})$ and $\\alpha_i^l \\geq 0$ with $\\sum_{i = 1}^{n+1} \\alpha_i^l = 1$ and\n        \\begin{align*}\n            g^l = \\sum_{i = 1}^{n+1} \\alpha_i^l \\xi_i^l.\n        \\end{align*}\n        Since $\\| y_i^l - x^{j_l} \\| \\leq \\eps_{j_l}$ it holds $\\lim_{l \\rightarrow \\infty} y_i^l = x^*$ for all $i \\in \\{1,\\dots,n+1\\}$. By compactness of the $\\eps$-subdifferential at $x^*$ (cf. \\cite{G1977}, Proposition 2.3), we can assume w.l.o.g. that $\\lim_{l \\rightarrow \\infty} \\xi_i^l = \\Bar{\\xi}_i \\in \\R^n$ for all $i \\in \\{1,\\dots,n+1\\}$. By \\cite{C1990}, Proposition 2.1.5, it follows that $\\Bar{\\xi}_i \\in \\partial f(x^*)$. Furthermore, by compactness of the set of convex coefficients, we can assume w.l.o.g. that $\\lim_{l \\rightarrow \\infty} \\alpha_i^l = \\Bar{\\alpha}_i$ for all $i \\in \\{1,\\dots,n+1\\}$, such that\n        \\begin{align*}\n            \\Bar{g} := \\sum_{i = 1}^{n+1} \\Bar{\\alpha}_i \\Bar{\\xi}_i \\in \\partial f(x^*)\n        \\end{align*}\n        is the limit of $(g^l)_l$. (In case (ii) we have $\\Bar{g} = 0$.) \\\\\n        \\textbf{Step 2:} Consider the sequences $(d^l)_l$ and $(d_i^l)_l$ given by\n        \\begin{align*}\n            d^l := \\frac{x^{j_l} - x^*}{\\| x^{j_l} - x^* \\|} \\quad \\text{and} \\quad d_i^l := \\frac{y_i^l - x^*}{\\| x^{j_l} - x^* \\|}, \\ i \\in \\{1,\\dots,n+1\\}.\n        \\end{align*}\n        By compactness, $(d^l)_l$ must have an accumulation point $\\Bar{d} \\in \\R^n$ with $\\| \\Bar{d} \\| = 1$. Assume w.l.o.g. that $\\Bar{d}$ is the limit of $(d^l)_l$. \n        For all $i \\in \\{1,\\dots,n+1\\}$, it holds\n        \\begin{align*}\n            \\| d_i^l - d^l \\| \n            = \\frac{\\| y_i^l - x^* - (x^{j_l} - x^*) \\|}{\\| x^{j_l} - x^* \\|}\n            = \\frac{\\| y_i^l - x^{j_l} \\|}{\\| x^{j_l} - x^* \\|}\n            \\leq \\frac{\\eps_{j_l}}{\\| x^{j_l} - x^* \\|},\n        \\end{align*}\n        and combined with \\eqref{eq:proof_thm_xj_speed_higher_order_1}, we obtain\n        \\begin{align} \\label{eq:proof_thm_xj_speed_higher_order_2}\n            \\frac{\\| d_i^l - d^l \\|}{\\| x^{j_l} - x^* \\|^{p-1}} \\leq \\frac{\\eps_{j_l}}{\\| x^{j_l} - x^* \\|^{p}} \\xrightarrow[]{l \\rightarrow \\infty} 0.\n        \\end{align}\n        In particular, $\\lim_{l \\rightarrow \\infty} d_i^l = \\bar{d}$ for all $i \\in \\{1,\\dots,n+1\\}$.\\\\ \n        \\textbf{Step 3:} By construction we have \n        \\begin{align*}\n            \\xi_i^l \\in \\partial f(y_i^l) \n            = \\partial f \\left( x^* + \\| x^{j_l} - x^* \\| \\frac{y_i^l - x^*}{\\| x^{j_l} - x^* \\|} \\right) \n            = \\partial f(x^* + \\| x^{j_l} - x^* \\| d_i^l).\n        \\end{align*}\n        Assume w.l.o.g. that $x^{j_l} \\in U$ with $U$ as in Definition \\ref{def:order_minimum}. Combination of \\eqref{eq:p_order_min_semismooth} and \\eqref{eq:proof_thm_xj_speed_higher_order_2} (and boundedness of $(\\xi_i^l)_l$) yields\n        \\begin{equation}\n            \\begin{aligned} \\label{eq:proof_thm_xj_speed_higher_order_3}\n                \\liminf_{l \\rightarrow \\infty} \\frac{\\langle \\xi_i^l, d^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}}\n                &= \\liminf_{l \\rightarrow \\infty} \\left( \\frac{\\langle \\xi_i^l, d_i^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}} + \\frac{\\langle \\xi_i^l, d^l - d_i^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}} \\right)\\\\\n                &\\stackrel{\\eqref{eq:proof_thm_xj_speed_higher_order_2}}{=} \\liminf_{l \\rightarrow \\infty} \\frac{\\langle \\xi_i^l, d_i^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}} \n                \\stackrel{\\eqref{eq:p_order_min_semismooth}}{\\geq} \\beta \\| \\Bar{d} \\|^p = \\beta\n            \\end{aligned}\n        \\end{equation}\n        for all $i \\in \\{1,\\dots,n+1\\}$. Furthermore note that\n        \\begin{equation} \\label{eq:proof_thm_xj_speed_higher_order_5}\n            \\begin{aligned}\n                \\delta_{j_l}\n                &\\geq \\| g^l \\| \n                = \\left\\| \\sum_{i = 1}^{n+1} \\alpha_i^l \\xi_i^l \\right\\|\n                = \\left\\| \\sum_{i = 1}^{n+1} \\alpha_i^l \\xi_i^l \\right\\| \\| d^l \\| \\\\\n                &\\geq \\langle \\sum_{i = 1}^{n+1} \\alpha_i^l \\xi_i^l, d^l \\rangle\n                = \\sum_{i = 1}^{n+1} \\alpha_i^l \\langle \\xi_i^l, d^l \\rangle \\quad \\forall l \\in \\N.\n            \\end{aligned}\n        \\end{equation}\n        Division of \\eqref{eq:proof_thm_xj_speed_higher_order_5} by $\\| x^{j_l} - x^* \\|^{p-1}$ and combination with \\eqref{eq:proof_thm_xj_speed_higher_order_1} and \\eqref{eq:proof_thm_xj_speed_higher_order_3} yields\n        \\begin{equation*}\n            \\begin{aligned}\n                \\beta \n                &\\stackrel{\\eqref{eq:proof_thm_xj_speed_higher_order_1}}{>} \\liminf_{l \\rightarrow \\infty} \\frac{\\delta_{j_l}}{\\| x^{j_l} - x^* \\|^{p-1}}\n                \\stackrel{\\eqref{eq:proof_thm_xj_speed_higher_order_5}}{\\geq} \\liminf_{l \\rightarrow \\infty} \\frac{ \\sum_{i = 1}^{n+1} \\alpha_i^l \\langle \\xi_i^l, d^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}} \\\\\n                &= \\liminf_{l \\rightarrow \\infty} \\sum_{i = 1}^{n+1} \\alpha_i^l \\frac{\\langle \\xi_i^l, d^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}}\n                \\geq \\sum_{i = 1}^{n+1} \\liminf_{l \\rightarrow \\infty} \\alpha_i^l \\frac{\\langle \\xi_i^l, d^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}} \\\\\n                &\\stackrel{\\eqref{eq:proof_thm_xj_speed_higher_order_3}}{\\geq} \\sum_{i = 1}^{n+1} \\Bar{\\alpha}_i \\beta = \\beta,\n            \\end{aligned}\n        \\end{equation*}\n        which is a contradiction.\n    \\end{proof}","sha256_utf8":"0b3abe98db64078549c45e90e439fc8da6d8e9c9e15387eb65fea6b813be6ff8"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"近似临界点的收敛速度","source_units":[{"start":52996,"end":54125,"start_line":555,"end_line":568,"raw_tex":"\\begin{theorem} \\label{thm:xj_speed_higher_order}\n        Let $f : \\R^n \\rightarrow \\R$ be locally Lipschitz continuous. Let $(x^j)_j \\in \\R^n$, $(\\eps_j)_j \\in \\R^{\\geq 0}$ and $(\\delta_j)_j \\in \\R^{\\geq 0}$ such that $x^j \\rightarrow x^* \\in \\R^n$, $\\eps_j \\rightarrow 0$ and $\\min(\\| \\partial_{\\eps_j} f(x^j) \\|) \\leq \\delta_j$ for all $j \\in \\N$.\n        Assume that $x^*$ is a minimum of order $p \\in \\N$ with constant $\\beta > 0$ and that \\eqref{eq:p_order_min_semismooth} holds in $x^*$.\n        \\begin{enumerate}\n            \\item[(i)] If $p = 1$ and $\\delta_j \\rightarrow \\Bar{\\delta} < \\beta$, then there are $M > 0$ and $N \\in \\N$ such that\n            \\begin{align} \\label{eq:xj_speed_higher_order_1}\n                \\| x^j - x^* \\| \\leq M \\eps_j \\quad \\forall j > N.\n            \\end{align}\n            \\item[(ii)] If $p \\geq 2$ then there are $M > 0$ and $N \\in \\N$ such that\n            \\begin{align} \\label{eq:xj_speed_higher_order_2}\n                \\| x^j - x^* \\| \\leq M \\max(\\eps_j^{\\frac{1}{p}}, \\delta_j^{\\frac{1}{p-1}}) \\quad \\forall j > N.\n            \\end{align}\n        \\end{enumerate}\n    \\end{theorem}","sha256_utf8":"eb03bb0248052a6a68c219c47b00d752b1098bd8e05965b1d90651fc428fd31c"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"收敛速度定理完整证明","source_units":[{"start":54130,"end":60251,"start_line":569,"end_line":641,"raw_tex":"\\begin{proof}\n        Assume that (i) or (ii) do not hold. Then in both cases there is a subsequence $(j_l)_l \\in \\N$ with $x^{j_l} \\neq x^*$ for all $l \\in \\N$,\n        \\begin{align} \\label{eq:proof_thm_xj_speed_higher_order_1}\n            \\lim_{l \\rightarrow \\infty} \\frac{\\eps_{j_l}}{\\| x^{j_l} - x^* \\|^{p}} = 0\n            \\quad \\text{and} \\quad\n             \\lim_{l \\rightarrow \\infty} \\frac{\\delta_{j_l}}{\\| x^{j_l} - x^* \\|^{p-1}} < \\beta.\n        \\end{align}\n        (In case (ii) does not hold, the second limit actually vanishes.) \\\\\n        \\textbf{Step 1:} For $l \\in \\N$ let $g^l \\in \\partial_{\\eps_{j_l}} f(x^{j_l})$ with $\\| g^l \\| \\leq \\delta_{j_l}$. By Carathéodory's theorem, for $i \\in \\{1,\\dots,n+1\\}$, there are $y_i^l \\in \\Bcl_{\\eps_{j_l}}(x^{j_l})$, $\\xi_i^l \\in \\partial f(y_i^l) \\subseteq \\partial_{\\eps_{j_l}} f(x^{j_l})$ and $\\alpha_i^l \\geq 0$ with $\\sum_{i = 1}^{n+1} \\alpha_i^l = 1$ and\n        \\begin{align*}\n            g^l = \\sum_{i = 1}^{n+1} \\alpha_i^l \\xi_i^l.\n        \\end{align*}\n        Since $\\| y_i^l - x^{j_l} \\| \\leq \\eps_{j_l}$ it holds $\\lim_{l \\rightarrow \\infty} y_i^l = x^*$ for all $i \\in \\{1,\\dots,n+1\\}$. By compactness of the $\\eps$-subdifferential at $x^*$ (cf. \\cite{G1977}, Proposition 2.3), we can assume w.l.o.g. that $\\lim_{l \\rightarrow \\infty} \\xi_i^l = \\Bar{\\xi}_i \\in \\R^n$ for all $i \\in \\{1,\\dots,n+1\\}$. By \\cite{C1990}, Proposition 2.1.5, it follows that $\\Bar{\\xi}_i \\in \\partial f(x^*)$. Furthermore, by compactness of the set of convex coefficients, we can assume w.l.o.g. that $\\lim_{l \\rightarrow \\infty} \\alpha_i^l = \\Bar{\\alpha}_i$ for all $i \\in \\{1,\\dots,n+1\\}$, such that\n        \\begin{align*}\n            \\Bar{g} := \\sum_{i = 1}^{n+1} \\Bar{\\alpha}_i \\Bar{\\xi}_i \\in \\partial f(x^*)\n        \\end{align*}\n        is the limit of $(g^l)_l$. (In case (ii) we have $\\Bar{g} = 0$.) \\\\\n        \\textbf{Step 2:} Consider the sequences $(d^l)_l$ and $(d_i^l)_l$ given by\n        \\begin{align*}\n            d^l := \\frac{x^{j_l} - x^*}{\\| x^{j_l} - x^* \\|} \\quad \\text{and} \\quad d_i^l := \\frac{y_i^l - x^*}{\\| x^{j_l} - x^* \\|}, \\ i \\in \\{1,\\dots,n+1\\}.\n        \\end{align*}\n        By compactness, $(d^l)_l$ must have an accumulation point $\\Bar{d} \\in \\R^n$ with $\\| \\Bar{d} \\| = 1$. Assume w.l.o.g. that $\\Bar{d}$ is the limit of $(d^l)_l$. \n        For all $i \\in \\{1,\\dots,n+1\\}$, it holds\n        \\begin{align*}\n            \\| d_i^l - d^l \\| \n            = \\frac{\\| y_i^l - x^* - (x^{j_l} - x^*) \\|}{\\| x^{j_l} - x^* \\|}\n            = \\frac{\\| y_i^l - x^{j_l} \\|}{\\| x^{j_l} - x^* \\|}\n            \\leq \\frac{\\eps_{j_l}}{\\| x^{j_l} - x^* \\|},\n        \\end{align*}\n        and combined with \\eqref{eq:proof_thm_xj_speed_higher_order_1}, we obtain\n        \\begin{align} \\label{eq:proof_thm_xj_speed_higher_order_2}\n            \\frac{\\| d_i^l - d^l \\|}{\\| x^{j_l} - x^* \\|^{p-1}} \\leq \\frac{\\eps_{j_l}}{\\| x^{j_l} - x^* \\|^{p}} \\xrightarrow[]{l \\rightarrow \\infty} 0.\n        \\end{align}\n        In particular, $\\lim_{l \\rightarrow \\infty} d_i^l = \\bar{d}$ for all $i \\in \\{1,\\dots,n+1\\}$.\\\\ \n        \\textbf{Step 3:} By construction we have \n        \\begin{align*}\n            \\xi_i^l \\in \\partial f(y_i^l) \n            = \\partial f \\left( x^* + \\| x^{j_l} - x^* \\| \\frac{y_i^l - x^*}{\\| x^{j_l} - x^* \\|} \\right) \n            = \\partial f(x^* + \\| x^{j_l} - x^* \\| d_i^l).\n        \\end{align*}\n        Assume w.l.o.g. that $x^{j_l} \\in U$ with $U$ as in Definition \\ref{def:order_minimum}. Combination of \\eqref{eq:p_order_min_semismooth} and \\eqref{eq:proof_thm_xj_speed_higher_order_2} (and boundedness of $(\\xi_i^l)_l$) yields\n        \\begin{equation}\n            \\begin{aligned} \\label{eq:proof_thm_xj_speed_higher_order_3}\n                \\liminf_{l \\rightarrow \\infty} \\frac{\\langle \\xi_i^l, d^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}}\n                &= \\liminf_{l \\rightarrow \\infty} \\left( \\frac{\\langle \\xi_i^l, d_i^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}} + \\frac{\\langle \\xi_i^l, d^l - d_i^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}} \\right)\\\\\n                &\\stackrel{\\eqref{eq:proof_thm_xj_speed_higher_order_2}}{=} \\liminf_{l \\rightarrow \\infty} \\frac{\\langle \\xi_i^l, d_i^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}} \n                \\stackrel{\\eqref{eq:p_order_min_semismooth}}{\\geq} \\beta \\| \\Bar{d} \\|^p = \\beta\n            \\end{aligned}\n        \\end{equation}\n        for all $i \\in \\{1,\\dots,n+1\\}$. Furthermore note that\n        \\begin{equation} \\label{eq:proof_thm_xj_speed_higher_order_5}\n            \\begin{aligned}\n                \\delta_{j_l}\n                &\\geq \\| g^l \\| \n                = \\left\\| \\sum_{i = 1}^{n+1} \\alpha_i^l \\xi_i^l \\right\\|\n                = \\left\\| \\sum_{i = 1}^{n+1} \\alpha_i^l \\xi_i^l \\right\\| \\| d^l \\| \\\\\n                &\\geq \\langle \\sum_{i = 1}^{n+1} \\alpha_i^l \\xi_i^l, d^l \\rangle\n                = \\sum_{i = 1}^{n+1} \\alpha_i^l \\langle \\xi_i^l, d^l \\rangle \\quad \\forall l \\in \\N.\n            \\end{aligned}\n        \\end{equation}\n        Division of \\eqref{eq:proof_thm_xj_speed_higher_order_5} by $\\| x^{j_l} - x^* \\|^{p-1}$ and combination with \\eqref{eq:proof_thm_xj_speed_higher_order_1} and \\eqref{eq:proof_thm_xj_speed_higher_order_3} yields\n        \\begin{equation*}\n            \\begin{aligned}\n                \\beta \n                &\\stackrel{\\eqref{eq:proof_thm_xj_speed_higher_order_1}}{>} \\liminf_{l \\rightarrow \\infty} \\frac{\\delta_{j_l}}{\\| x^{j_l} - x^* \\|^{p-1}}\n                \\stackrel{\\eqref{eq:proof_thm_xj_speed_higher_order_5}}{\\geq} \\liminf_{l \\rightarrow \\infty} \\frac{ \\sum_{i = 1}^{n+1} \\alpha_i^l \\langle \\xi_i^l, d^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}} \\\\\n                &= \\liminf_{l \\rightarrow \\infty} \\sum_{i = 1}^{n+1} \\alpha_i^l \\frac{\\langle \\xi_i^l, d^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}}\n                \\geq \\sum_{i = 1}^{n+1} \\liminf_{l \\rightarrow \\infty} \\alpha_i^l \\frac{\\langle \\xi_i^l, d^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}} \\\\\n                &\\stackrel{\\eqref{eq:proof_thm_xj_speed_higher_order_3}}{\\geq} \\sum_{i = 1}^{n+1} \\Bar{\\alpha}_i \\beta = \\beta,\n            \\end{aligned}\n        \\end{equation*}\n        which is a contradiction.\n    \\end{proof}","sha256_utf8":"0b3abe98db64078549c45e90e439fc8da6d8e9c9e15387eb65fea6b813be6ff8"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"Clarke次微分","source_units":[{"start":11305,"end":11717,"start_line":105,"end_line":105,"raw_tex":"    In this section, we briefly introduce the basics of nonsmooth analysis. For a more thorough introduction, we refer to \\cite{C1990}. To this end, let $f : \\R^n \\rightarrow \\R$ be locally Lipschitz continuous and let $\\Omega$ be the set of points in which $f$ is not differentiable. By Rademacher's Theorem, $\\Omega$ is a null set. The \\emph{Clarke subdifferential of} $f$ \\emph{at} $x \\in \\R^n$ is defined as\n","sha256_utf8":"dd14d41f0adade5cd5689aa1b04f782e9d823db6f0557f67b0fa47d18d3b1829"},{"start":11721,"end":12087,"start_line":106,"end_line":111,"raw_tex":"\\begin{equation*}\n        \\begin{aligned}\n            \\partial f(x) := \\conv &\\left( \\left\\{ \\xi \\in \\R^n : \\exists (x^j)_j \\in \\R^n \\setminus \\Omega \\text{ with } \\lim_{j \\rightarrow \\infty} x^j = x \\text{ and } \\right. \\right. \\\\\n            &\\left. \\left. \\lim_{j \\rightarrow \\infty} \\nabla f(x^j) = \\xi \\right\\} \\right),\n        \\end{aligned}\n    \\end{equation*}","sha256_utf8":"4c412d74eb127763db68040f184e48e733d6b4ac5978492c416be829b4593a20"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"Goldstein次微分","source_units":[{"start":12556,"end":12810,"start_line":116,"end_line":116,"raw_tex":"    To circumvent some of the practical issues of the Clarke subdifferential (cf. \\cite{BGLS2006}, Section 9.1), the \\emph{Goldstein} $\\eps$\\emph{-subdifferential} may be used instead. For $x \\in \\R^n$ and $\\eps \\geq 0$, it is defined as the compact set\n","sha256_utf8":"06ba24ea63d4bb3d36424521b813db57590ee13bac3c016947552d34e36fed64"},{"start":12814,"end":12940,"start_line":117,"end_line":119,"raw_tex":"\\begin{align*}\n        \\partial_\\eps f(x) := \\conv \\left( \\bigcup_{y \\in \\Bcl_\\eps(x)} \\partial f(y) \\right),\n    \\end{align*}","sha256_utf8":"ce37f8687ad927df6360eb6dfaaf0a7a986468bc58fd5308c74023924e0913c3"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"球、临界性与正则性记号","source_units":[{"start":12941,"end":15022,"start_line":120,"end_line":128,"raw_tex":"    where $\\Bcl_\\eps(x) := \\{ y \\in \\R^n : \\| y - x \\| \\leq \\eps \\}$ and $\\| \\cdot \\|$ is the Euclidean norm. For $\\eps, \\delta \\geq 0$ we say that $x$ is $(\\eps,\\delta)$\\emph{-critical}, if $\\min(\\| \\partial_\\eps f(x) \\|) \\leq \\delta$, i.e., if there is some $\\xi \\in \\partial_\\eps f(x)$ with $\\|\\xi \\| \\leq \\delta$. Clearly, $(\\eps,\\delta)$-criticality is a weaker optimality condition than criticality. However, upper semicontinuity of $\\partial f$ implies that if there are sequences $(x^j)_j \\in \\R^n$, $(\\eps_j)_j, (\\delta_j)_j \\in \\R^{\\geq 0}$ with $x^j \\rightarrow x^*$, $\\eps_j \\rightarrow 0$ and $\\delta_j \\rightarrow 0$ such that $x^j$ is $(\\eps_j,\\delta_j)$-critical for all $j \\in \\N$, then $x^*$ is critical (cf. Lemma 4.4.4 in \\cite{G2022}). In addition to subdifferentials, we also require directional derivatives of $f$. To assure that they exist and are well-behaved, we occasionally assume that $f$ is \\emph{semismooth} \\cite{M1977}, which means that $f$ is locally Lipschitz continuous and for any $x \\in \\R^n$ and $d \\in \\R^n$, the limit\n    \\begin{align} \\label{eq:def_semismooth}\n        \\lim_{\\xi \\in \\partial f(x + t d'), t \\searrow 0, d' \\rightarrow d} \\langle \\xi, d' \\rangle\n    \\end{align}\n    exists. If $f$ is semismooth, then for any $d \\in \\R^n$, the \\emph{directional derivative}\n    \\begin{align*}\n        f'(x,d) = \\lim_{t \\searrow 0} \\frac{f(x + t d) - f(x)}{t}\n    \\end{align*}\n    exists and equals the limit \\eqref{eq:def_semismooth}. A large class of semismooth functions are the \\emph{piecewise $p$-times differentiable} functions \\cite{S2012}, which are continuous functions $f : \\R^n \\rightarrow \\R$ for which there are $p$-times continuously differentiable functions $f_1, \\dots, f_m : \\R^n \\rightarrow \\R$, called \\emph{selection functions}, such that $f(x) \\in \\{ f_1(x), \\dots, f_m(x) \\}$ for all $x \\in \\R^n$. For such a function, the set $A(x) := \\{ i \\in \\{1,\\dots,m\\} : f(x) = f_i(x) \\}$ is called the \\emph{active set}, and the Clarke subdifferential satisfies $\\partial f(x) \\subseteq \\conv(\\{ \\nabla f_i(x) : i \\in A(x) \\})$.\n","sha256_utf8":"1c5efa01f89f18d81cfb1ea8c1fa90bdb4600b55b520f28baf58ffc58776e9d8"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"高阶极小点与增长常数","source_units":[{"start":22935,"end":23303,"start_line":198,"end_line":203,"raw_tex":"\\begin{definition} \\label{def:order_minimum}\n        A point $x^* \\in \\R^n$ is called a \\emph{minimum of order} $p \\in \\N$ \\emph{with constant} $\\beta > 0$, if there is some open set $U \\subseteq\\R^n$ with $x^* \\in U$ such that\n        \\begin{align*}\n            f(x) \\geq f(x^*) + \\beta \\| x - x^* \\|^p \\quad \\forall x \\in U.\n        \\end{align*}\n    \\end{definition}","sha256_utf8":"6befd426a5f318058de4c27cf79db8e7c8b5e6db3071d87a6759dbb58e85726e"}]},{"id":"S007","semantic_type":"assumption","kind":"assumption","title":"高阶次梯度下界条件","source_units":[{"start":32050,"end":32300,"start_line":322,"end_line":324,"raw_tex":"\\begin{align} \\label{eq:p_order_min_semismooth}\n        \\liminf_{\\xi \\in \\partial f(x^* + t d'), t \\searrow 0, d' \\rightarrow d} \\frac{\\langle \\xi,  d' \\rangle}{t^{p-1}} \\geq \\beta \\| d \\|^p \\quad \\forall d \\in \\R^n \\setminus \\{ 0 \\}.\n    \\end{align}","sha256_utf8":"d15e54a743923413c103ff67edd4380ccdfb6f7b86a634a5e1f63cec7a51dc8f"}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","title":"序列与极小点假设","source_units":[{"start":53046,"end":53491,"start_line":556,"end_line":557,"raw_tex":"        Let $f : \\R^n \\rightarrow \\R$ be locally Lipschitz continuous. Let $(x^j)_j \\in \\R^n$, $(\\eps_j)_j \\in \\R^{\\geq 0}$ and $(\\delta_j)_j \\in \\R^{\\geq 0}$ such that $x^j \\rightarrow x^* \\in \\R^n$, $\\eps_j \\rightarrow 0$ and $\\min(\\| \\partial_{\\eps_j} f(x^j) \\|) \\leq \\delta_j$ for all $j \\in \\N$.\n        Assume that $x^*$ is a minimum of order $p \\in \\N$ with constant $\\beta > 0$ and that \\eqref{eq:p_order_min_semismooth} holds in $x^*$.\n","sha256_utf8":"743a6fff3acb97d42bf46db6c2d53882ab06012e51ad15fe8bd6ceadf84b7d53"}]},{"id":"S009","semantic_type":"proof_step","kind":"proof_step","title":"反设并抽取违背估计的子列","source_units":[{"start":54130,"end":54675,"start_line":569,"end_line":576,"raw_tex":"\\begin{proof}\n        Assume that (i) or (ii) do not hold. Then in both cases there is a subsequence $(j_l)_l \\in \\N$ with $x^{j_l} \\neq x^*$ for all $l \\in \\N$,\n        \\begin{align} \\label{eq:proof_thm_xj_speed_higher_order_1}\n            \\lim_{l \\rightarrow \\infty} \\frac{\\eps_{j_l}}{\\| x^{j_l} - x^* \\|^{p}} = 0\n            \\quad \\text{and} \\quad\n             \\lim_{l \\rightarrow \\infty} \\frac{\\delta_{j_l}}{\\| x^{j_l} - x^* \\|^{p-1}} < \\beta.\n        \\end{align}\n        (In case (ii) does not hold, the second limit actually vanishes.) \\\\\n","sha256_utf8":"135bb6443c7cde7c56c6e9cbb802e1def400ac0c62ce37de575da96345fb22f1"}]},{"id":"S010","semantic_type":"proof_step","kind":"proof_step","title":"分解近似次梯度并抽取极限","source_units":[{"start":54675,"end":55988,"start_line":577,"end_line":585,"raw_tex":"        \\textbf{Step 1:} For $l \\in \\N$ let $g^l \\in \\partial_{\\eps_{j_l}} f(x^{j_l})$ with $\\| g^l \\| \\leq \\delta_{j_l}$. By Carathéodory's theorem, for $i \\in \\{1,\\dots,n+1\\}$, there are $y_i^l \\in \\Bcl_{\\eps_{j_l}}(x^{j_l})$, $\\xi_i^l \\in \\partial f(y_i^l) \\subseteq \\partial_{\\eps_{j_l}} f(x^{j_l})$ and $\\alpha_i^l \\geq 0$ with $\\sum_{i = 1}^{n+1} \\alpha_i^l = 1$ and\n        \\begin{align*}\n            g^l = \\sum_{i = 1}^{n+1} \\alpha_i^l \\xi_i^l.\n        \\end{align*}\n        Since $\\| y_i^l - x^{j_l} \\| \\leq \\eps_{j_l}$ it holds $\\lim_{l \\rightarrow \\infty} y_i^l = x^*$ for all $i \\in \\{1,\\dots,n+1\\}$. By compactness of the $\\eps$-subdifferential at $x^*$ (cf. \\cite{G1977}, Proposition 2.3), we can assume w.l.o.g. that $\\lim_{l \\rightarrow \\infty} \\xi_i^l = \\Bar{\\xi}_i \\in \\R^n$ for all $i \\in \\{1,\\dots,n+1\\}$. By \\cite{C1990}, Proposition 2.1.5, it follows that $\\Bar{\\xi}_i \\in \\partial f(x^*)$. Furthermore, by compactness of the set of convex coefficients, we can assume w.l.o.g. that $\\lim_{l \\rightarrow \\infty} \\alpha_i^l = \\Bar{\\alpha}_i$ for all $i \\in \\{1,\\dots,n+1\\}$, such that\n        \\begin{align*}\n            \\Bar{g} := \\sum_{i = 1}^{n+1} \\Bar{\\alpha}_i \\Bar{\\xi}_i \\in \\partial f(x^*)\n        \\end{align*}\n        is the limit of $(g^l)_l$. (In case (ii) we have $\\Bar{g} = 0$.) \\\\\n","sha256_utf8":"34b5f174fe9d4611223b939413800d3f87d19e7102a7133c2a7717b976be333d"}]},{"id":"S011","semantic_type":"proof_step","kind":"proof_step","title":"归一化方向与扰动误差","source_units":[{"start":55988,"end":57194,"start_line":586,"end_line":602,"raw_tex":"        \\textbf{Step 2:} Consider the sequences $(d^l)_l$ and $(d_i^l)_l$ given by\n        \\begin{align*}\n            d^l := \\frac{x^{j_l} - x^*}{\\| x^{j_l} - x^* \\|} \\quad \\text{and} \\quad d_i^l := \\frac{y_i^l - x^*}{\\| x^{j_l} - x^* \\|}, \\ i \\in \\{1,\\dots,n+1\\}.\n        \\end{align*}\n        By compactness, $(d^l)_l$ must have an accumulation point $\\Bar{d} \\in \\R^n$ with $\\| \\Bar{d} \\| = 1$. Assume w.l.o.g. that $\\Bar{d}$ is the limit of $(d^l)_l$. \n        For all $i \\in \\{1,\\dots,n+1\\}$, it holds\n        \\begin{align*}\n            \\| d_i^l - d^l \\| \n            = \\frac{\\| y_i^l - x^* - (x^{j_l} - x^*) \\|}{\\| x^{j_l} - x^* \\|}\n            = \\frac{\\| y_i^l - x^{j_l} \\|}{\\| x^{j_l} - x^* \\|}\n            \\leq \\frac{\\eps_{j_l}}{\\| x^{j_l} - x^* \\|},\n        \\end{align*}\n        and combined with \\eqref{eq:proof_thm_xj_speed_higher_order_1}, we obtain\n        \\begin{align} \\label{eq:proof_thm_xj_speed_higher_order_2}\n            \\frac{\\| d_i^l - d^l \\|}{\\| x^{j_l} - x^* \\|^{p-1}} \\leq \\frac{\\eps_{j_l}}{\\| x^{j_l} - x^* \\|^{p}} \\xrightarrow[]{l \\rightarrow \\infty} 0.\n        \\end{align}\n        In particular, $\\lim_{l \\rightarrow \\infty} d_i^l = \\bar{d}$ for all $i \\in \\{1,\\dots,n+1\\}$.\\\\ \n","sha256_utf8":"e122d4fbacb11c6eef466a576cbb5cf827b4d38b41a87c6d27e51aa7f36415cb"}]},{"id":"S012","semantic_type":"proof_step","kind":"proof_step","title":"将高阶下界转移到共同方向","source_units":[{"start":57194,"end":58465,"start_line":603,"end_line":617,"raw_tex":"        \\textbf{Step 3:} By construction we have \n        \\begin{align*}\n            \\xi_i^l \\in \\partial f(y_i^l) \n            = \\partial f \\left( x^* + \\| x^{j_l} - x^* \\| \\frac{y_i^l - x^*}{\\| x^{j_l} - x^* \\|} \\right) \n            = \\partial f(x^* + \\| x^{j_l} - x^* \\| d_i^l).\n        \\end{align*}\n        Assume w.l.o.g. that $x^{j_l} \\in U$ with $U$ as in Definition \\ref{def:order_minimum}. Combination of \\eqref{eq:p_order_min_semismooth} and \\eqref{eq:proof_thm_xj_speed_higher_order_2} (and boundedness of $(\\xi_i^l)_l$) yields\n        \\begin{equation}\n            \\begin{aligned} \\label{eq:proof_thm_xj_speed_higher_order_3}\n                \\liminf_{l \\rightarrow \\infty} \\frac{\\langle \\xi_i^l, d^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}}\n                &= \\liminf_{l \\rightarrow \\infty} \\left( \\frac{\\langle \\xi_i^l, d_i^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}} + \\frac{\\langle \\xi_i^l, d^l - d_i^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}} \\right)\\\\\n                &\\stackrel{\\eqref{eq:proof_thm_xj_speed_higher_order_2}}{=} \\liminf_{l \\rightarrow \\infty} \\frac{\\langle \\xi_i^l, d_i^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}} \n                \\stackrel{\\eqref{eq:p_order_min_semismooth}}{\\geq} \\beta \\| \\Bar{d} \\|^p = \\beta\n            \\end{aligned}\n        \\end{equation}\n","sha256_utf8":"668202892dfe9899b730d0ec72fd6c232e4d6daf899272284d786c53d6e6ab4b"}]},{"id":"S013","semantic_type":"proof_step","kind":"proof_step","title":"以方向投影估计临界容差","source_units":[{"start":58465,"end":59078,"start_line":618,"end_line":628,"raw_tex":"        for all $i \\in \\{1,\\dots,n+1\\}$. Furthermore note that\n        \\begin{equation} \\label{eq:proof_thm_xj_speed_higher_order_5}\n            \\begin{aligned}\n                \\delta_{j_l}\n                &\\geq \\| g^l \\| \n                = \\left\\| \\sum_{i = 1}^{n+1} \\alpha_i^l \\xi_i^l \\right\\|\n                = \\left\\| \\sum_{i = 1}^{n+1} \\alpha_i^l \\xi_i^l \\right\\| \\| d^l \\| \\\\\n                &\\geq \\langle \\sum_{i = 1}^{n+1} \\alpha_i^l \\xi_i^l, d^l \\rangle\n                = \\sum_{i = 1}^{n+1} \\alpha_i^l \\langle \\xi_i^l, d^l \\rangle \\quad \\forall l \\in \\N.\n            \\end{aligned}\n        \\end{equation}\n","sha256_utf8":"ac7fa459a78831871c113030e4bf4cb6bf37b89b7aa36611ce08a51e146170bb"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"合并极限下界得到矛盾","source_units":[{"start":59078,"end":60251,"start_line":629,"end_line":641,"raw_tex":"        Division of \\eqref{eq:proof_thm_xj_speed_higher_order_5} by $\\| x^{j_l} - x^* \\|^{p-1}$ and combination with \\eqref{eq:proof_thm_xj_speed_higher_order_1} and \\eqref{eq:proof_thm_xj_speed_higher_order_3} yields\n        \\begin{equation*}\n            \\begin{aligned}\n                \\beta \n                &\\stackrel{\\eqref{eq:proof_thm_xj_speed_higher_order_1}}{>} \\liminf_{l \\rightarrow \\infty} \\frac{\\delta_{j_l}}{\\| x^{j_l} - x^* \\|^{p-1}}\n                \\stackrel{\\eqref{eq:proof_thm_xj_speed_higher_order_5}}{\\geq} \\liminf_{l \\rightarrow \\infty} \\frac{ \\sum_{i = 1}^{n+1} \\alpha_i^l \\langle \\xi_i^l, d^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}} \\\\\n                &= \\liminf_{l \\rightarrow \\infty} \\sum_{i = 1}^{n+1} \\alpha_i^l \\frac{\\langle \\xi_i^l, d^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}}\n                \\geq \\sum_{i = 1}^{n+1} \\liminf_{l \\rightarrow \\infty} \\alpha_i^l \\frac{\\langle \\xi_i^l, d^l \\rangle}{\\| x^{j_l} - x^* \\|^{p-1}} \\\\\n                &\\stackrel{\\eqref{eq:proof_thm_xj_speed_higher_order_3}}{\\geq} \\sum_{i = 1}^{n+1} \\Bar{\\alpha}_i \\beta = \\beta,\n            \\end{aligned}\n        \\end{equation*}\n        which is a contradiction.\n    \\end{proof}","sha256_utf8":"56ae71b97c0252d71d92427cb930cbf9ae45bbcbeaa15e61d7d04d4a83003ba2"}]},{"id":"S015","semantic_type":"external_fact","kind":"external_fact","title":"有限凸组合表示","source_units":[],"declared_external":true,"declaration_text":["Carathéodory's theorem: every element of the convex hull of a subset of R^n can be represented as a convex combination of at most n+1 elements of that subset."]},{"id":"S016","semantic_type":"external_fact","kind":"external_fact","title":"次微分紧性与闭图性质","source_units":[],"declared_external":true,"declaration_text":["For a locally Lipschitz function on R^n, the Clarke subdifferential has nonempty compact convex values, is locally bounded, and has a closed graph. Its Goldstein subdifferential over a fixed finite closed ball is compact. Thus subgradients at points approaching x* admit convergent subsequences, and their limits belong to the Clarke subdifferential at x*."]},{"id":"S017","semantic_type":"external_fact","kind":"external_fact","title":"欧氏紧性与极限运算","source_units":[],"declared_external":true,"declaration_text":["In finite-dimensional Euclidean space, the unit sphere and the simplex of convex coefficients are compact, and the Cauchy–Schwarz inequality holds. If finitely many scalar sequences each have limit inferior at least beta, then their convex combinations with nonnegative coefficients summing to one also have limit inferior at least beta; convergent coefficient subsequences and finite-sum limit-inferior inequalities justify the corresponding calculation."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S003","target_id":"S010","kind":"supports"},{"source_id":"S004","target_id":"S010","kind":"supports"},{"source_id":"S005","target_id":"S010","kind":"supports"},{"source_id":"S005","target_id":"S011","kind":"supports"},{"source_id":"S006","target_id":"S012","kind":"supports"},{"source_id":"S007","target_id":"S012","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S010","kind":"supports"},{"source_id":"S008","target_id":"S011","kind":"supports"},{"source_id":"S009","target_id":"S011","kind":"supports"},{"source_id":"S009","target_id":"S014","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S010","target_id":"S013","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S015","target_id":"S010","kind":"supports"},{"source_id":"S016","target_id":"S010","kind":"supports"},{"source_id":"S017","target_id":"S010","kind":"supports"},{"source_id":"S017","target_id":"S011","kind":"supports"},{"source_id":"S017","target_id":"S012","kind":"supports"},{"source_id":"S017","target_id":"S013","kind":"supports"},{"source_id":"S017","target_id":"S014","kind":"supports"},{"source_id":"S009","target_id":"S002","kind":"supports"},{"source_id":"S010","target_id":"S002","kind":"supports"},{"source_id":"S011","target_id":"S002","kind":"supports"},{"source_id":"S012","target_id":"S002","kind":"supports"},{"source_id":"S013","target_id":"S002","kind":"supports"},{"source_id":"S014","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S001","kind":"supports"}]}}
{"record_id":"2410.02626v1","paper":{"title":"Online Learning Guided Quasi-Newton Methods with Global Non-Asymptotic Convergence","authors":["Ruichen Jiang","Aryan Mokhtari"],"arxiv_id":"2410.02626","version":"v1","journal":"Mathematical Programming","doi":"10.1007/s10107-026-02396-4","abstract_url":"https://arxiv.org/abs/2410.02626v1","pdf_url":"https://arxiv.org/pdf/2410.02626v1"},"context":{"raw_tex":"In this paper, we consider the problem of solving a system of nonlinear equations:\n%\n\\begin{equation}\\label{eq:monotone}\n  \\vF(\\vz) = 0,\n  %\n\\end{equation}\n\nwhere $\\vF : \\reals^d \\rightarrow \\reals^d$ is a continuously differentiable operator.\n\n\n\\begin{assumption}\\label{assum:strong_monotone}  \nThe operator $\\vF$ is $\\mu$-strongly monotone, i.e., $\\langle \\vF(\\vz)-\\vF(\\vz'), \\vz-\\vz' \\rangle \\geq \\mu \\|\\vz-\\vz'\\|^2$,  $\\forall \\vz,\\vz' \\in \\reals^d$. \n   %\n\\end{assumption}\n\n\\begin{assumption}\\label{assum:monotone}  \n  $\\vF$ is monotone, i.e., $\\langle \\vF(\\vz)-\\vF(\\vz'), \\vz-\\vz' \\rangle \\geq 0$,  $\\forall \\vz,\\vz' \\in \\reals^d$. \n  %\n\\end{assumption}\n\nNote that under Assumption~\\ref{assum:strong_monotone}, Problem \\eqref{eq:monotone} has a unique solution, which we denote by $\\vz^*$ throughout the paper.\nMoreover, Assumption~\\ref{assum:monotone} can be regarded as a special case of Assumption~\\ref{assum:strong_monotone} by setting $\\mu=0$. Therefore, we will sometimes use this observation to present our results in both settings in a united manner, avoiding unnecessary repetition. In addition, we make the following two assumptions that the operator $\\vF$ and its Jacobian are Lipschitz. \n\n\n\\begin{assumption}\\label{assum:operator_lips}\nThe operator $\\vF$ is $L_1$-Lipschitz, i.e., $\\|\\vF(\\vz)-\\vF(\\vz')\\| \\leq L_1\\|\\vz-\\vz'\\|$, $\\forall \\vz,\\vz' \\in \\reals^d$.  \n %\n\\end{assumption}\n\n\\begin{assumption}\\label{assum:jacobian_lips}\n  The Jacobian of $\\vF$ is $L_2$-Lipschitz, i.e., $\\|\\nabla \\vF(\\vz)-\\nabla \\vF(\\vz')\\|_{\\op} \\leq L_2\\|\\vz-\\vz'\\|$, $\\forall \\vz,\\vz'\\in \\reals^d$, where $\\|\\mA\\|_{\\op} \\mydef \\sup_{\\|\\vz\\| = 1} \\|\\mA \\vz\\|$. \n  %\n\\end{assumption}\n\n\\myalert{{Minimization.}} As a special instance of Problem \\eqref{eq:monotone}, consider the unconstrained minimization problem \n\\begin{equation}\\label{eq:minimization}\n  \\min_{\\vz \\in \\reals^d} f(\\vz),\n\\end{equation}\n\n%\nwhere $f\\!:\\! \\reals^d\\! \\rightarrow \\!\\reals$ is a (strongly) convex function. By the first-order optimality condition, the optimal solution of \\eqref{eq:minimization} satisfies the nonlinear equation $\\nabla f(\\vz) = 0$. Thus, in this case, $\\vF$ is  the gradient operator, i.e., $\\vF(\\vz) = \\nabla f(\\vz)$, and its Jacobian is the Hessian matrix of $f$, i.e., $\\nabla \\vF(\\vz) = \\nabla^2 f(\\vz)$. In particular, we observe that the Jacobian is a symmetric matrix, which allows us to simplify the conditions in Lemma~\\ref{lem:jacobian}. Specifically, under Assumptions~\\ref{assum:strong_monotone} and~\\ref{assum:operator_lips},  $\\nabla \\vF$ belongs to the set $\\{\\mA \\in \\mathbb{S}^d\\!: \\mu \\mI \\preceq\\! \\mA\\! \\preceq L\\mI\\}$. Meanwhile, under Assumptions~\\ref{assum:monotone} and~\\ref{assum:operator_lips}, it belongs to the set $\\{\\mA \\in \\mathbb{S}^d\\!: 0 \\preceq\\! \\mA\\! \\preceq L\\mI\\}$. As discussed in Section~\\ref{subsec:nonlinear_eqs}, in our method, we ensure that the Jacobian approximation matrix stays symmetric and positive semidefinite, which is a key property for the efficient implementation of the subroutines.\n\n\n\\myalert{Minimax optimization.} Another important instance of Problem \\eqref{eq:monotone} is the unconstrained minimax optimization problem \n\\begin{equation}\\label{eq:minimax}\n  \\min_{\\vx \\in \\reals^m} \\max_{\\vy \\in \\reals^n} f(\\vx,\\vy),\n\\end{equation}\n\n%\nwhere $f$ is (strongly) convex with respect to $\\vx$ and (strongly) concave with respect to $\\vy$. To reformulate this problem as \\eqref{eq:monotone}, we define the saddle point operator as $\\vF(\\vz) = (\\nabla f_{\\vx}(\\vx,\\vy),-\\nabla f_{\\vy}(\\vx,\\vy)) \\in \\reals^{m+n}$, where $\\vz = (\\vx,\\vy) \\in \\reals^{m+n}$. Indeed, finding a solution of $\\vF(\\vz)$ is equivalent to finding a stationary point of the minimax problem, which is the optimal solution. In this case, the Jacobian is   \n$\n    \\nabla \\vF(\\vz) = \n    \\begin{bmatrix}\n        \\nabla^2 f_{\\vx\\vx}(\\vx,\\vy) & \\nabla^2 f_{\\vx \\vy}(\\vx,\\vy) \\\\\n        -\\nabla^2 f_{\\vx \\vy}(\\vx,\\vy)^\\top & -\\nabla^2 f_{\\vy\\vy}(\\vx,\\vy)\n    \\end{bmatrix}$.\nAs noted in~\\cite{asl2023j}, $\\nabla \\vF(\\vz)$ exhibits $\\mJ$-symmetric structure~\\cite{mackey2003structured}, where it is symmetric on the main diagonal blocks and anti-symmetric on the off-diagonal blocks. \nSpecifically, for  $\n    \\mJ = \\begin{bmatrix}\n        \\mI_{m\\times m} & 0 \\\\\n        0 & - \\mI_{n\\times n}\n    \\end{bmatrix}$,  matrix $\\mM$ is $\\mJ$-symmetric if $ \\mJ\\mM$ is symmetric, i.e., $ \\mJ \\mM = \\mM^\\top \\mJ$. \nSimilarly, as discussed in Section~\\ref{subsec:nonlinear_eqs}, we can ensure that our Jacobian approximation matrix in our method respects the $\\mJ$-symmetry property, simplifying the implementation of some subroutines.  \n\n\nSpecifically, assume that we know the sparsity pattern of the Jacobian matrices, defined as the set\n$\n  \\Omega = \\{(i,j):\\,\\exists\\, \\vz \\in \\reals^{d}\\,\\mathrm{s.t.}\\,[\\nabla \\vF(\\vz)]_{ij} \\neq 0, \\;1\\leq i \\leq d,\\; 1\\leq j \\leq d\\}$,\nwhere $[\\nabla \\vF(\\vz)]_{ij}$ denotes the $(i,j)$-the entry of the matrix $\\nabla \\vF(\\vz)$. \nMoreover, we say a matrix $\\mM\\in \\reals^{d \\times d}$ has the sparsity pattern $\\Omega$ if $[\\mM]_{ij} = 0$ for all $(i,j) \\notin \\Omega$ and $i \\neq j$. \n\n\nIn the strongly monotone setting (under Assumption~\\ref{assum:strong_monotone}), \nrecall that there is a unique solution $\\vz^*$ to Problem~\\eqref{eq:monotone}. Hence, we measure the suboptimality in terms of distance to $\\vz^*$. \n\n\n\\begin{equation}\n  \\gap_{\\text{w}}(\\vz; \\mathcal{D}) = \\max_{\\vz' \\in \\mathcal{D}} \\langle \\vF(\\vz'),   \\vz - \\vz' \\rangle,\n\\end{equation}\n\n\\begin{equation}\n  \\gap_f(\\vz; \\vz^*) = f(\\vz) - f(\\vz^*). \n\\end{equation}\n\n\\begin{equation}\n  \\gap_{\\text{pd}}(\\vz;\\calX\\times \\calY) = \\max_{\\vy' \\in \\calY} f(\\vx,\\vy') - \\min_{\\vx'\\in \\calX} f(\\vx',\\vy),\n\\end{equation}\n\nwhere $\\vz = (\\vx,\\vy)$ and $\\calX$ and $\\calY$ are given compact sets in $\\reals^m$ and $\\reals^n$, respectively. \n\n\nIn Lemma~\\ref{lem:averaging}, we observe that the gap functions in the three cases can all be upper bounded by a similar quantity. Thus, to unify these different notions, we define  \n\\begin{equation}\\label{eq:gap}\n  \\gap(\\vz; \\mathcal{D}) = \\begin{cases}\n    \\gap_{\\mathrm{w}}(\\vz; \\mathcal{D}), & \\text{for Problem~\\eqref{eq:monotone};}\\\\\n    \\max_{\\vz' \\in \\mathcal{D}}\\{f(\\vz) - f(\\vz')\\}, & \\text{for Problem~\\eqref{eq:minimization}};\\\\\n    \\max_{(\\vx',\\vy') \\in \\calD} \\{f(\\vx,\\vy') - f(\\vx',\\vy)\\}, & \\text{for Problem~\\eqref{eq:minimax}}.\n  \\end{cases}\n\\end{equation}\n\nSpecifically, note that $\\gap(\\vz; \\{\\vz^*\\})$ reduces to $\\gap_f(\\vz;\\vz^*)$ for Problem~\\eqref{eq:minimization} and $\\gap(\\vz; \\calX\\times \\calY)$ reduces to $\\gap_{\\text{pd}}(\\vz;\\calX\\times \\calY)$ for Problem~\\eqref{eq:minimax}. \n\n\n\\begin{algorithm}[t!]\\small\n  \\caption{Quasi-Newton Proximal Extragradient (QNPE) Method (informal)}\\label{alg:Full_Equasi-Newton}\n    \\begin{algorithmic}[1]\n        \\STATE \\textbf{Input:} strong monotonicity parameter $\\mu \\geq 0$, Jacobian feasible set $\\mathcal{Z}$, line search parameters $\\alpha_1\\geq 0$ and $\\alpha_2>0$ such that $\\alpha_1+\\alpha_2 < 1$,  and initial trial step size $\\sigma_0>0$ \n        \\STATE \\textbf{Initialization:} initial point $\\vz_0\\in \\mathbb{R}^d$ and initial Jacobian approximation $\\mB_0\\in \\mathcal{Z}$ %\n        %\n        \\FOR{iteration $k=0,\\ldots,N-1$}\n        %\n        %\n        \\STATE Let $\\eta_k$ be the largest possible step size in $\\{\\sigma_k\\beta^i:i\\geq 0\\}$ such that  \\tikzmark{top} \n        %\n        \\begin{gather*}\n          \\hspace{-8em}\\|{\\hat{\\vz}_k}-\\vz_k+{\\eta_k}(\\vF(\\vz_k)+\\mB_k({\\hat{\\vz}_k}-\\vz_k))\\| \\leq \\alpha_1 \\sqrt{1+\\eta_k \\mu} \\|\\hat{\\vz}_k-\\vz_k\\|, \\tikzmark{right} %\n          \\\\\n          \\hspace{-8em}\\|\\hat{\\vz}_k - \\vz_k + \\eta_k \\vF(\\hat{\\vz}_k)\\| \\leq { (\\alpha_1+\\alpha_2)\\sqrt{1+\\eta_k \\mu} \\|{\\hat{\\vz}_k}-\\vz_k\\|}\n           %\n      \\end{gather*}%\n\n%\n        \\STATE Set $\\sigma_{k+1} \\leftarrow \\eta_{k}/\\beta$ \\tikzmark{bottom}\n        \\STATE Update \n        $\n          \\vz_{k+1} \\leftarrow \\theta_k (\\vz_k - \\eta_k \\vF(\\hat{\\vz}_k))+(1-\\theta_k) \\hat{\\vz}_k\n        $, where $\\theta_k = \\frac{1}{1+2\\eta_k\\mu}$\n        %\n        %\n        \\IF[Line search accepted the initial trial step size]{$\\eta_k = \\sigma_k$ \\tikzmark{top2}}\n        \\STATE Set $\\mB_{k+1} \\leftarrow \\mB_k$ \\label{line:Hessian_approx_unchanged}\n        \\ELSE[Line search bactracked]\n        \\STATE Let $\\tilde{\\vz}_k$ be the last rejected iterate in the line search  %\n        \\STATE Set $\\vu_k \\leftarrow \\vF(\\tilde{\\vz}_k) -\\vF({\\vz_k})$, $\\vs_k \\leftarrow \\tilde{\\vz}_k-\\vz_k$\n        \\STATE Define the loss function $\\ell_k(\\mB) = \\frac{\\|\\vu_k-\\mB\\vs_k\\|^2}{\\|\\vs_k\\|^2}$ \n        \\STATE Feed $\\ell_k(\\mB)$ to an online learning algorithm $\\mathcal{A}$ and obtain $\\mB_{k+1}$ \\quad \\tikzmark{right2}\n        \\ENDIF \\tikzmark{bottom2}\n        \\ENDFOR\n                 \n    \\end{algorithmic}\n    \\AddNote{top}{bottom}{right}{\\hspace{-.5em}\\color{comment}\\textit{\\quad Line search subroutine;\\\\\n             \\hspace{.2em} see Section~\\ref{subsec:ls}}}\n    \\AddNote{top2}{bottom2}{right2}{\\hspace{-.5em}\\color{comment}\\textit{\\quad Jacobian approximation\\\\ \\hspace{.2em} update subroutine; see \\\\\n    \\hspace{.2em} Section~\\ref{sec:jacobian_online_learning}}}\n    %\n  \\end{algorithm}\n\n\\begin{align}\n    \\|{\\hat{\\vz}_k}-\\vz_k+{\\eta_k}(\\vF(\\vz_k)+\\mB_k({\\hat{\\vz}_k}-\\vz_k))\\| &\\leq {\\alpha_1} \\sqrt{1+\\eta_k \\mu} \\|\\hat{\\vz}_k-\\vz_k\\|, \\label{eq:inexact_linear_solver} \\\\\n    \\|\\hat{\\vz}_k - \\vz_k + \\eta_k \\vF(\\hat{\\vz}_k)\\| &\\leq {(\\alpha_1+\\alpha_2) \\sqrt{1+\\eta_k \\mu} \\|{\\hat{\\vz}_k}-\\vz_k\\|} \\label{eq:inexact_proximal_point}, \n%\n\\end{align}\n\nwhere $\\alpha_1\\in [0,1)$ and $\\alpha_2 \\in (0,1)$ are user-specified parameters with $\\alpha_1+\\alpha_2<1$.  \n\n\n\\begin{equation}\\label{eq:extragradient_mixing}\n%\n  \\vz_{k+1} = \\theta_k(\\vz_k - \\eta_k \\vF(\\hat{\\vz}_k))+(1-\\theta_k)\\hat{\\vz}_k,\n\\end{equation}\n\nwhere $\\theta_k \\in [0,1]$ is chosen based on our convergence analysis. Specifically, in the monotone setting (under Assumption~\\ref{assum:monotone}), we choose $\\theta_k = 1$ and \\eqref{eq:extragradient_mixing} reduces to the extragradient step in \\eqref{eq:extragradient}. Moreover, in the strongly monotone setting (under Assumption~\\ref{assum:strong_monotone}), we choose $\\theta_k  = \\frac{1}{1+2\\eta_k\\mu}$.\n\n\n\\begin{algorithmic}[1]\n      \\STATE \\textbf{Input:} iterate $\\vz \\in \\mathbb{R}^d$,  operator $\\vg\\in \\reals^d$,  Jacobian approximation $\\mB$, initial trial step size $\\sigma>0$\n      \\STATE \\textbf{Parameters:} line search parameters $\\beta\\in (0,1)$, $\\alpha_1\\geq 0$ and $\\alpha_2>0$ such that $\\alpha_1+\\alpha_2<1$\n      %\n      \\STATE Set ${\\eta}_{+} \\leftarrow \\sigma$, {$\\vs_{+} \\leftarrow \\mathsf{LinearSolver}(\\mI+\\eta_{+}\\mB, -\\eta_{+}\\vg; {\\alpha_1}\\sqrt{1+\\eta_+ \\mu})$ and $\\hat{\\vz}_{+} \\leftarrow \\vz+\\vs_{+}$}\n      \\WHILE{$\\|\\hat{\\vz}_+ - \\vz + \\eta_+ \\vF(\\hat{\\vz}_+)\\| \\geq (\\alpha_1+\\alpha_2)\\sqrt{1+\\eta_+ \\mu} \\|{\\hat{\\vz}_{+}}-\\vz\\|$\\label{line:check_condition}}\n        \\STATE Set $\\tilde{\\vz} \\leftarrow \\hat{\\vz}_{+}$ %\n          and $\\eta_{+} \\leftarrow \\beta\\eta_+ $ \\label{line:decreasing_stepsize}\n      \\STATE Compute $\\vs_{+} \\leftarrow \\mathsf{LinearSolver}(\\mI+\\eta_{+}\\mB, -\\eta_{+}\\vg; \\alpha_1\\sqrt{1+\\eta_+ \\mu})$ and $\\hat{\\vz}_{+} \\leftarrow \\vz+\\vs_{+}$\n      \\ENDWHILE\n      \\IF{$\\eta_+ = \\sigma$}\n      \\STATE \\textbf{Return} $\\eta_{+}$ and  $\\hat{\\vz}_{+}$ \\label{line:return_1}\n      \\ELSE \n      \\STATE \\textbf{Return} $\\eta_{+}$, $\\hat{\\vz}_{+}$ and $\\tilde{\\vz}$ \\label{line:return_2}\n      \\ENDIF\n  \\end{algorithmic}\n\nNext, we present the backtracking line search scheme for selecting $\\eta_k$ and the iterate $\\hat{\\vz}_k$ in the first stage of QNPE. For brevity, we denote $\\vF(\\vz_k)$ by $\\vg$ and omit the iteration subscript $k$ from both $\\vz_k$ and $\\mB_k$. %\nIn light of \\eqref{eq:inexact_linear_solver} and \\eqref{eq:inexact_proximal_point}, our goal at the $k$-th iteration is to identify a pair $(\\eta_{+}, \\hat{\\vz}_{+})$ satisfying  \n%\n\\begin{align}\n  %\n  \\|\\hat{\\vz}_{+}-\\vz+{\\eta_{+}}(\\vg+\\mB({\\hat{\\vz}_{+}}-\\vz))\\| &\\leq  {\\alpha_1}\\sqrt{1+\\eta_+ \\mu} \\|{\\hat{\\vz}_{+}}-\\vz\\|,\n  \\label{eq:x_plus_update} \\\\\n  \\|\\hat{\\vz}_+ - \\vz + \\eta_+ \\vF(\\hat{\\vz}_+)\\| &\\leq (\\alpha_1+\\alpha_2)\\sqrt{1+\\eta_+ \\mu} \\|{\\hat{\\vz}_{+}}-\\vz\\|  \\label{eq:step size_condition}. \n\\end{align}\n\nAs previously discussed, with $\\eta_{+}$ fixed, the first condition in \\eqref{eq:x_plus_update} can be met by solving the linear system $(\\mI+\\eta_{+}\\mB)(\\hat{\\vz}_+-\\vz) = -\\eta_+\\vg$ to a desired accuracy.\n\n\n\\begin{equation}\\label{eq:linear_solver_update}\n  \\vs_{+} = \\mathsf{LinearSolver}(\\mI+\\eta_{+}\\mB, -\\eta_{+}\\vg; {\\alpha_1}\\sqrt{1+\\eta_+ \\mu}) \\quad \\text{and} \\quad \\hat{\\vz}_{+} = \\vz+\\vs_{+},\n  %\n\\end{equation}\n\nwhere the $\\mathsf{LinearSolver}$ oracle is defined as follows. %\n\n\n\\begin{definition}\\label{def:linear_solver}\n  The oracle $\\mathsf{LinearSolver}(\\mA,\\vb; \\rho)$ takes  a matrix $\\mA \\in \\reals^{d\\times d}$, a vector $\\vb\\in \\reals^d$ and $\\rho>0$ as input, and returns an approximate solution $\\vs_{+}$ \n  %\n  satisfying $\\|\\mA\\vs_{+}-\\vb\\| \\leq \\rho \\|\\vs_{+}\\|$. \n  %\n  %\n  %\n  %\n  %\n  %\n\\end{definition}\n\nRecall that $\\calB$ is the set of indices where the line search subroutine backtracks, i.e., $\\calB = \\{k: \\eta_k < \\sigma_k\\}$. \n\n\nFormally, define the loss function $\\ell_k : \\reals^{d \\times d} \\rightarrow \\reals$ at iteration $k$ as\n\n\n\\begin{equation}\\label{eq:loss_of_jacobian}\n  \\ell_k(\\mB) \\triangleq \n  \\begin{cases}\n    0, & \\text{if } k\\notin \\mathcal{B}, \\\\\n    \\frac{\\|\\vu_k-\\mB\\vs_k\\|^2}{\\|\\vs_k\\|^2}, & \\text{otherwise}.\n  \\end{cases}\n  %\n\\end{equation}\n\nGiven these points and the discussions in Section~\\ref{subsec:structures}, we choose the feasible set~$\\calZ$ as follows: \n\\begin{enumerate}[(a)]\n  \\item For the minimization problem in \\eqref{eq:minimization}, we choose \n\n\n\\begin{equation}\\label{eq:feasible_set_min}\n    \\calZ = \\{\\mB \\in \\sS^d: \\; \\mu \\mI \\preceq \\mB \\preceq L_1 \\mI,\\; \\mB \\in \\calL\\},\n  \\end{equation}\n\n  where $\\calL \\subset \\sS^d$ is a linear subspace. in the general case, we set $\\calL = \\sS^d$. However, when the Hessians of $f$ exhibit the sparsity pattern $\\Omega$, we define $\\calL$ as $\\calL=\\{\\mB\\in \\sS^{d}: [\\mB]_{ij} = 0 \\text{ for all } (i,j)\\notin \\Omega \\text{ and }i\\neq j\\}$.\n  \\item For the minimax problem in \\eqref{eq:minimax} and the nonlinear equation in \\eqref{eq:monotone}, we choose \n\n\n\\begin{equation}\\label{eq:feasible_set}\n    \\mathcal{Z} = \\left\\{ \\mB \\in \\reals^{d\\times d}:\\mu \\mI \\preceq \\frac{1}{2}(\\mB + \\mB^\\top )\\preceq L_1\\mI , \\;\\|\\mB\\|_{\\op} \\leq L_1, \\mB \\in \\mathcal{L} \\right\\},\n    %\n  \\end{equation}\n\n  where $\\mathcal{L} \\subset \\reals^{d\\times d}$ is a linear subspace. For general nonlinear equations $\\mathcal{L} = \\reals^{d \\times d}$, for minimax problems $\\mathcal{L} = \\{\\mB\\in \\reals^{d\\times d}: \\mB = \\mJ \\mB^\\top \\mJ\\}$, and for sparse nonlinear equations $\\calL=\\{\\mB\\in \\reals^{d\\times d}: [\\mB]_{ij} = 0 \\text{ for all } (i,j)\\notin \\Omega \\text{ and }i\\neq j\\}$.\n\\end{enumerate}\n\n\n\\begin{algorithm}[!t]\\small\n  \\caption{Projection-Free Online Learning}\\label{alg:projection_free_online_learning}\n  \\begin{algorithmic}[1]\n      \\STATE \\textbf{Input:} Initial point $\\vw_0 \\in  \\vspan(\\calC)$, the orthogonal projection matrix $\\mP$ associated with the subspace $\\vspan(\\calC)$, step size $\\rho>0$, $\\delta>0$, radius $R>0$\n      %\n      %\n      \\FOR{$t=0,1,\\dots T-1$}\n      \\STATE Query the oracle $(\\gamma_t,\\vs_t) \\leftarrow \\mathsf{SEP}_{\\calC}(\\vw_t;\\delta)$\n      \\IF[Case I: we have $\\vw_t\\in (1+\\delta)\\mathcal{C}$]{$\\gamma_t \\leq 1$}\n      %\n      %\n        \\STATE Set $\\vx_t \\leftarrow \\begin{cases} \\vw_t & (\\textbf{Option I}) \\\\ \\frac{\\vw_t}{1+\\delta} & (\\textbf{Option II}) \\end{cases}$ and play the action $\\vx_t$ \n        \\STATE Receive the loss $\\ell_t(\\vx_t)$ and the subspace gradient $\\vg_t = \\mP\\nabla \\ell_t(\\vx_t)$\n        \\STATE Set $\\tilde{\\vg}_t \\leftarrow \\vg_t$\n      %\n      %\n      \\ELSE[Case II: we have $\\vw_t/\\gamma_t\\in (1+\\delta)\\mathcal{C}$] \n      %\n        \\STATE Set $\\vx_t \\leftarrow \\begin{cases} \\frac{\\vw_t}{\\gamma_t} & (\\textbf{Option I}) \\\\ \\frac{\\vw_t}{(1+\\delta)\\gamma_t} & (\\textbf{Option II}) \\end{cases}$ and play the action $\\vx_t$ \n      \\STATE Receive the loss $\\ell_t(\\vx_t)$ and the subspace gradient $\\vg_t = \\mP\\nabla \\ell_t(\\vx_t)$\n      \\STATE Set $\\tilde{\\vg}_t \\leftarrow \\vg_t+\\max\\left\\{0, - \\frac{1}{\\gamma_t}\\langle \\vg_t, \\vw_t \\rangle\\right\\} \\vs_t$\n      %\n      \\ENDIF\n      %\n      \\STATE Update $\\vw_{t+1} \\leftarrow \\frac{R(\\vw_t-\\rho \\tilde{\\vg}_t)}{\\max\\{\\|\\vw_t-\\rho \\tilde{\\vg}_t\\|_2,R\\}} $ \n      \\COMMENT{Online projected gradient descent}\n      \\ENDFOR\n  \\end{algorithmic}\n\\end{algorithm}\n\nTo set the stage for our Jacobian approximation update algorithm, we take a detour and consider a general online learning problem over a compact feasible set $\\calC \\subset \\reals^D$. For $T$ consecutive rounds $t=0,\\dots,T-1$, a learner chooses an action $\\vx_t \\in \\reals^D$ and then observes a convex loss function $\\ell_t:\\reals^D \\rightarrow \\reals$. \n\\blue{The goal is to minimize either the \\emph{static regret} defined by $\\mathrm{Reg}_T(\\vu; \\calC) \\triangleq \\sum_{t=0}^{T-1} \\ell_t(\\vx_t) - \\sum_{t=0}^{T-1} \\ell_t(\\vu)$ with a fixed competitor $\\vx \\in \\calC$, or the \\emph{dynamic regret} defined by $\\dreg_T(\\vu_0,\\dots,\\vu_{T-1}; \\calC) \\triangleq \\sum_{t=0}^{T-1} \\ell_t(\\vx_t) - \\sum_{t=0}^{T-1} \\ell_t(\\vu_t)$ with a sequence of competitors $\\vu_t \\in \\calC$ for any $t \\geq 0$.}\nWe assume $0\\in \\mathcal{C}$ without loss of generality. Moreover, we assume that the set $\\calC$ is contained in the Euclidean ball $B_R(0):= \\{\\vw\\in \\reals^D: \\|\\vw\\|\\leq R\\}$ for some radius $R>0$.\n\n\nTo start, we will define the approximate separation oracle $\\SEP_{\\cal{C}}$ for a compact set $\\calC$. We use $\\vspan(\\mathcal{C})$ to denote the linear span of $\\mathcal{C}$, i.e., $\\vspan(\\calC) = \\left\\{\\sum_{i=1}^{k}  \\alpha_i \\vx_i : k \\in \\mathbb{Z}_+,\\;\\vx_i \\in \\calC,\\; \\alpha_i \\in \\reals  \\right\\}$. Intuitively, $\\vspan(\\calC)$ is the smallest linear subspace that contains the set $\\calC$. %\n\n\n\\begin{definition}\\label{def:gauge}\n  Let $\\calC$ be a compact convex set in $\\reals^D$ containing the origin. \n  The oracle $\\mathsf{SEP}_{\\mathcal{C}}(\\vw; \\delta)$ takes $\\vw\\!\\in\\! \\vspan(\\calC)$ and $\\delta>0$ and returns $\\gamma>0$ and a vector $\\vs\\in \\vspan(\\calC)$ with one of these outcomes:\n   %\n   %\n    \\begin{itemize}\n      \\item Case I: $\\gamma \\leq 1$, which implies that $\\vw \\in (1+\\delta)\\mathcal{C}$;\n       %\n      \\item Case II: $\\gamma>1$, which implies that $\\vw/\\gamma \\in (1+\\delta)\\mathcal{C} \\ $ and $\\ \\langle \\vs, \\vw-\\vx \\rangle \\geq {\\gamma-1},$  $ \\forall\\vx\\in \\mathcal{C}$. \n    \\end{itemize}\n    %\n    %\n    %\n    %\n    %\n    \\end{definition}\n\n\\begin{equation}\\label{eq:w_update}\n  \\vw_{t+1}  = \\mathrm{\\Pi}_{\\mathcal{B}_R(0)}\\bigl(\\vw_t-\\rho \\nabla \\tilde{\\ell}_t(\\vw_t)\\bigr) = \\frac{R}{\\max\\{\\|\\vw_t-\\rho \\tilde{\\vg}_t\\|_2,R\\}} (\\vw_t-\\rho \\tilde{\\vg}_t).\n\\end{equation}\n\n\\begin{algorithmic}[1]\n      \\STATE \\textbf{Input:} Initial matrix $\\mB_0\\in \\mathcal{Z}$, the orthogonal projection matrix $\\mP$, step size $\\rho>0$, $\\delta>0$, failure probabilities $\\{q_t\\}_{t=1}^{T-1}$, Lipschitz constant $L_1>0$ and strong monotonicity parameter $\\mu \\geq 0$\n      \\STATE \\textbf{Initialize:} set $\\mW_0 \\leftarrow \\frac{1}{L_1}(\\mB-(L_1+\\mu)\\mI)$, $\\mG_0 \\leftarrow \\frac{1}{L_1} \\mP\\nabla \\ell_0(\\mB_0)$ and $\\tilde{\\mG}_0 \\leftarrow \\mG_0$\n      \\STATE {Update $\\mW_{1} \\leftarrow \\frac{\\sqrt{d}(\\mW_0 - \\rho \\tilde{\\mG}_0)}{\\max\\{\\sqrt{d},\\|\\mW_0 - \\rho \\tilde{\\mG}_0\\|_F\\}}$ \\COMMENT{Projection onto $\\mathcal{B}_{\\sqrt{d}}(0)$}\\vspace{-.5em}\n      \\FOR{$t=1,\\dots,T-1$}\n      \\STATE Query $(\\gamma^{(1)}_t,\\mS^{(1)}_t) \\leftarrow \\mathsf{ExtEvec}(\\frac{\\mW_t+\\mW_t^\\top}{2};\\delta, \\frac{q_t}{2} )$ and $(\\gamma^{(2)}_t,\\mS^{(2)}_t) \\leftarrow \\mathsf{MaxSvec}(\\mW_t;\\delta, \\frac{q_t}{2} )$ %\n      \\STATE Let $i = \\argmax_{i\\in \\{1,2\\}} \\gamma_t^{(i)}$, and set $\\gamma_t \\leftarrow \\gamma_t^{(i)}$ and $\\mS_t \\leftarrow \\mP \\mS_t^{(i)}$\n      \\IF[Case I]{$\\gamma_t \\leq 1$}\n      %\n        \\STATE Set $\\hat{\\mB}_t \\leftarrow \\begin{cases} \\mW_t & \\text{if }\\mu>0\\;(\\textbf{Option I})\\\\ \\frac{\\mW_t}{1+\\delta} & \\text{if }\\mu=0\\; (\\textbf{Option II})\\end{cases}$ and $\\mB_t \\leftarrow {L_1}\\hat{\\mB}_t+{(L_1+\\mu)}\\mI$\n        \\STATE Set $\\mG_t \\leftarrow \\frac{1}{L_1}\\mP\\nabla \\ell_t(\\mB_t)$ and $\\tilde{\\mG}_t \\leftarrow \\mG_t$\n        %\n      \\ELSE[Case II]\n        \\STATE Set $\\hat{\\mB}_t \\leftarrow \\begin{cases} \\frac{\\mW_t}{\\gamma_t} & \\text{if }\\mu>0\\;(\\textbf{Option I})\\\\ \\frac{\\mW_t}{(1+\\delta)\\gamma_t} & \\text{if }\\mu=0\\; (\\textbf{Option II})\\end{cases}$ and $\\mB_t \\leftarrow {L_1}\\hat{\\mB}_t+{(L_1+\\mu)}\\mI$ \n        \\STATE Set $\\mG_t \\leftarrow \\frac{1}{L_1}\\mP\\nabla \\ell_t(\\mB_t)$ and $\\tilde{\\mG}_t \\leftarrow \\mG_t+\\max\\{0,-\\frac{1}{\\gamma_t}\\langle \\mG_t, \\mW_t \\rangle\\} \\mS_t$\n        %\n      \\ENDIF\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      \\STATE %\n      %\n      Update \n      $\\mW_{t+1} \\leftarrow \\frac{\\sqrt{d}(\\mW_t - \\rho \\tilde{\\mG}_t)}{\\max\\{\\sqrt{d},\\|\\mW_t - \\rho \\tilde{\\mG}_t\\|_F\\}}$ \\COMMENT{Projection onto $\\mathcal{B}_{\\sqrt{d}}(0)$}\n        %\n        %\n        %\n        %\n      %\n      \\ENDFOR}\n  \\end{algorithmic}\n\nNext, we present our online learning algorithm for updating $\\{\\mB_k\\}_{k \\geq 0}$. We follow the projection-free online learning algorithm outlined in Section~\\ref{subsec:projection_free_online_learning} and apply it to the online learning problem in Section~\\ref{subsec:convergence}.\n\\blue{Moreover, recall that $\\calB$ denotes the set of indices where the line search subroutine backtracks, and we have $\\ell_k(\\mB)=0$ when $k\\notin \\mathcal{B}$. Thus, we can simply keep $\\mB_{k+1}$ unchanged for these iterations (cf. Line~\\ref{line:Hessian_approx_unchanged} in Algorithm~\\ref{alg:Full_Equasi-Newton}). Suppose $\\calB = \\{k_0,k_1,\\dots,k_{T-1}\\}$, where $T \\leq N$. With a slight abuse of notation, in the following, we relabel the indices in $\\mathcal{B}$ as $t=0,\\dots,T-1$.}\n\nThe remaining question is how to construct the approximate separation oracle, required by the projection-free scheme and defined in Definition~\\ref{def:gauge}, for our specific set $\\calZ$.\nTo start, we consider the nonlinear monotone equation in \\eqref{eq:monotone}; we will further discuss the special cases in the next section.  \nRecall that the feasible set~$\\calZ$ for our online learning problem in Section~\\ref{subsec:convergence} is given in \\eqref{eq:feasible_set} with a linear subspace $\\calL$. In the following, we assume that the orthogonal projection matrix $\\mP$ associated with the linear subspace $\\calL$ is given. Specifically, for a general nonlinear monotone equation, we have $\\calL = \\reals^d$ and the matrix $\\mP$ is simply the identity matrix. Moreover, for those special cases discussed in Section~\\ref{subsec:structures}, the linear subspace is specified at the end of Section~\\ref{subsec:convergence} and we will discuss the corresponding projection matrix in Section~\\ref{subsec:special_cases}. \n\n\nHence, we will work with the new set $\\calC$ in our online learning algorithm. Moreover, note that we can write $\\calC = \\calC_1 \\cap \\calC_2 \\cap \\calL$, where we define \n%\n  $\\mathcal{C}_1 \\mydef \\{\\hat{\\mB} \\in \\mathbb{R}^{d\\times d}:  -\\mI \\preceq \\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top )\\preceq \\mI \\}$ and $\\mathcal{C}_2 \\mydef \\{\\hat{\\mB} \\in \\mathbb{R}^{d\\times d}: \\|\\hat{\\mB}\\|_{\\op} \\leq 3 \\}$.  \n\n\n\\begin{definition}\\label{def:extevec}\n  The oracle $\\mathsf{ExtEvec}(\\mW;\\delta,q)$ takes $\\mW \\in \\reals^{d\\times d}$, $\\delta>0$, and $q\\in (0,1)$ as input and returns a scalar $\\gamma>0$ and a matrix $\\mS\\in \\mathbb{S}^d$. With probability at least $1-q$, the returned $\\gamma$ and $\\mS$ satisfy one of the following properties::\n  %\n  \\begin{itemize}\n    \\item Case I: $\\gamma \\leq 1$, which implies that\n    %\n    $-(1+\\delta)\\mI \\preceq \\frac{1}{2}(\\mW+\\mW^\\top) \\preceq (1+\\delta)\\mI$;\n    %\n    \\item Case II: $\\gamma>1$, which implies that %\n    %\n    $-(1+\\delta)\\mI \\preceq \\frac{1}{2\\gamma}(\\mW+\\mW^\\top) \\preceq (1+\\delta)\\mI$, \n    $\\|\\mS\\|_F \\leq 1$ and $\\langle \\mS,\\mW-\\hat{\\mB}\\rangle \\geq \\gamma -1$ for any $\\hat{\\mB}$ such that $-\\mI \\preceq \\frac{1}{2}(\\hat{\\mB}+\\hat{\\mB}^\\top) \\preceq \\mI$.\n  \\end{itemize}\n%\n%\n%\n%\n%\n%\n\\end{definition}\n\n\\begin{definition}\\label{def:MaxSvec}\n  The oracle $\\mathsf{MaxSvec}(\\mW;\\delta,q)$ takes $\\mW \\in \\mathbb{R}^{d\\times d}$, $\\delta>0$, and $q\\in (0,1)$ as input and returns a scalar $\\gamma>0$ and a matrix $\\mS\\in \\mathbb{R}^{d\\times d}$. With probability at least $1-q$, the returned $\\gamma$ and $\\mS$ satisfy one of the following properties::\n  %\n  \\begin{itemize}\n    \\item Case I: $\\gamma \\leq 1$, which implies that $ \\|\\mW\\|_{\\op} \\leq 3(1+\\delta) $;\n    %\n    \\item Case II: $\\gamma>1$, which implies that $ \\|\\mW/\\gamma \\|_{\\op} \\leq 3(1+\\delta)$, $\\|\\mS\\|_F \\leq 1$ and $\\langle \\mS,\\mW-\\hat{\\mB}\\rangle \\geq \\gamma -1$ for any $\\hat{\\mB} \\in \\reals^{d\\times d}$ such that $ \\|\\hat{\\mB}\\|_{\\op}  \\leq 3 $.\n  \\end{itemize}\n\\end{definition}\n\n\\begin{algorithmic}[1]\n      \\STATE \\textbf{Input:} $\\mW \\in \\reals^{d\\times d}$, $\\delta>0$, $q\\in (0,1)$\n      \\STATE \\textbf{Initialize:} sample $\\vv_1\\in \\reals^d$ uniformly from the unit sphere, $\\beta_1 \\leftarrow 0$, $\\vv_0\\leftarrow 0$\n      \\STATE Set $\\bar{\\mW} \\leftarrow \\frac{1}{2}(\\mW + \\mW^\\top)$ and set the number of iterations \n      $N \\leftarrow \\Bigl\\lceil\\frac{1}{4}\\sqrt{2(1+\\frac{1}{\\delta})}\\log(\\frac{11d}{q^2})+\\frac{1}{2} \\rceil$\n      %\n      %\n      %\n      %\n      \\FOR{$k=1,\\dots,N$ \\tikzmark{top}}\n      \\STATE Set $\\vw_k \\leftarrow \\bar{\\mW} \\vv_k-\\beta_k \\vv_{k-1}$ \n      \\STATE Set $\\alpha_k \\leftarrow \\langle \\vw_k,\\vv_k \\rangle $ and $\\vw_k \\leftarrow \\vw_k-\\alpha_k\\vv_k$\n      \\STATE Set $\\beta_{k+1} \\leftarrow \\|\\vw_k\\|$ and $\\vv_{k+1}\\leftarrow \\vw_k/\\beta_{k+1}$\n      \\ENDFOR\n    \\STATE Form a tridiagonal matrix $\\mT \\leftarrow \\mathsf{tridiag}(\\beta_{2:N},\\alpha_{1:N},\\beta_{2:N})$\n    \\STATE \\hspace{-1em}\\COMMENT{Use the tridiagonal structure to compute eigenvectors of $\\mT$}\n    \\STATE Compute $(\\hat{\\lambda}_1,\\vz^{(1)}) \\leftarrow \\mathsf{MaxEvec}(\\mT)$ and $(\\hat{\\lambda}_d, \\vz^{(d)}) \\leftarrow \\mathsf{MinEvec}(\\mT)$ \\tikzmark{right}\n    \\STATE Set $\\vu^{(1)} \\leftarrow \\sum_{k=1}^N z^{(1)}_k\\vv_k$ and $\\vu^{(d)} \\leftarrow \\sum_{k=1}^N z^{(d)}_k\\vv_k$ \\tikzmark{bottom}\n    \\STATE Set $\\gamma \\leftarrow \\max\\{\\hat{\\lambda}_1,-\\hat{\\lambda}_d\\}$ \n    \\IF{$\\gamma \\leq 1$}\n    \\STATE Return $\\gamma$ and $\\mS = 0$ \\COMMENT{Case I: $\\gamma\\leq 1$, which implies $\\|\\mW\\|_{\\op} \\leq 1+\\delta$}\n    \\ELSIF{$\\hat{\\lambda}_1 \\geq -\\hat{\\lambda}_d$}\n      \\STATE Return $\\gamma$ and $\\mS = \\vu^{(1)}(\\vu^{(1)})^\\top$ \\hspace{.75em}\\COMMENT{Case II: $\\gamma> 1$ and $\\mS$ defines a separating hyperplane}\n    \\ELSE\n      \\STATE Return $\\gamma$ and $\\mS = -\\vu^{(d)}(\\vu^{(d)})^\\top$ \\COMMENT{Case II: $\\gamma> 1$ and $\\mS$ defines a separating hyperplane}\n    \\ENDIF\n  \\end{algorithmic}\n\n\\begin{algorithmic}[1]\n    \\STATE \\textbf{Input:} $\\mW \\in \\mathbb{R}^{d\\times d}$, $\\delta>0$, $q\\in (0,1)$\n    %\n    \\STATE Set $\\tilde{\\mW} \\leftarrow \\begin{bmatrix} 0 & \\mW \\\\ \\mW^\\top & 0\n    \\end{bmatrix} \\in \\mathbb{S}^{2d}$ and the number of iterations $N  \\leftarrow \n    %\n    \\Bigl\\lceil \\frac{1}{4}\\sqrt{2(1+\\frac{1}{\\delta})}\\log\\frac{22d}{q^2}+\\frac{1}{2}\\Bigr\n    \\rceil\n    %\n    $\n    \n    \\STATE Compute approximate largest eigenvalue $\\tilde{\\lambda}_1$ and its eigenvector $\\tilde{\\vv} \\in \\reals^{2d}$, by running the Lanczos method on $\\tilde{\\mW}$ for $N$ iterations \\COMMENT{See Subroutine~\\ref{alg:lanczos}}\n    \\STATE Partition $\\tilde{\\vv}$ as $\\tilde{\\vv} = [\\vv,\\vv']$ where $\\vv, \\vv' \\in \\reals^d$ \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    \\STATE Set $\\gamma \\leftarrow \\tilde{\\lambda}_1/3$ \n    \\IF{$\\gamma \\leq 1$}\n    \\STATE Return $\\gamma$ and $\\mS = 0$ \\COMMENT{Case I: $\\gamma\\leq 1$, which implies $\\|\\mW\\|_{\\op} \\leq 3(1+\\delta)$}\n    \\ELSE\n      \\STATE Return $\\gamma$ and $\\mS = \\frac{2}{3}\\vv'\\vv^\\top \\in \\reals^{d\\times d}$ \\hspace{.75em}\\COMMENT{Case II: $\\gamma> 1$ and $\\mS$ defines a separating hyperplane}\n    %\n    %\n    \\ENDIF\n  \\end{algorithmic}","units":[{"start":18753,"end":18908,"start_line":684,"end_line":689,"raw_tex":"In this paper, we consider the problem of solving a system of nonlinear equations:\n%\n\\begin{equation}\\label{eq:monotone}\n  \\vF(\\vz) = 0,\n  %\n\\end{equation}","sha256_utf8":"daf1a1184722314aa1224af98de70c5ff325fae57d5f0a2c3a2671fd8d3efccc"},{"start":18909,"end":18996,"start_line":690,"end_line":690,"raw_tex":"where $\\vF : \\reals^d \\rightarrow \\reals^d$ is a continuously differentiable operator.\n","sha256_utf8":"6f2b7816885eaaf550349fa39cfe7d92b3899c362a2a7e3a2ad815f7663bfbea"},{"start":35042,"end":35273,"start_line":946,"end_line":949,"raw_tex":"\\begin{assumption}\\label{assum:strong_monotone}  \nThe operator $\\vF$ is $\\mu$-strongly monotone, i.e., $\\langle \\vF(\\vz)-\\vF(\\vz'), \\vz-\\vz' \\rangle \\geq \\mu \\|\\vz-\\vz'\\|^2$,  $\\forall \\vz,\\vz' \\in \\reals^d$. \n   %\n\\end{assumption}","sha256_utf8":"9cde0702e46789b444eb0089a787f31a9584a70a8c42012f5dd103b55905de6d"},{"start":35275,"end":35455,"start_line":951,"end_line":954,"raw_tex":"\\begin{assumption}\\label{assum:monotone}  \n  $\\vF$ is monotone, i.e., $\\langle \\vF(\\vz)-\\vF(\\vz'), \\vz-\\vz' \\rangle \\geq 0$,  $\\forall \\vz,\\vz' \\in \\reals^d$. \n  %\n\\end{assumption}","sha256_utf8":"5c43752a1c1ac1e2729d97cbaeea71f0a040f4e8dced29374f4b622cdfd2c375"},{"start":35524,"end":36069,"start_line":958,"end_line":959,"raw_tex":"Note that under Assumption~\\ref{assum:strong_monotone}, Problem \\eqref{eq:monotone} has a unique solution, which we denote by $\\vz^*$ throughout the paper.\nMoreover, Assumption~\\ref{assum:monotone} can be regarded as a special case of Assumption~\\ref{assum:strong_monotone} by setting $\\mu=0$. Therefore, we will sometimes use this observation to present our results in both settings in a united manner, avoiding unnecessary repetition. In addition, we make the following two assumptions that the operator $\\vF$ and its Jacobian are Lipschitz. \n","sha256_utf8":"db085d32777cc9f0f8a97f31ed63c69e555cf7c00959c67c521e3783bb97e500"},{"start":36070,"end":36262,"start_line":961,"end_line":964,"raw_tex":"\\begin{assumption}\\label{assum:operator_lips}\nThe operator $\\vF$ is $L_1$-Lipschitz, i.e., $\\|\\vF(\\vz)-\\vF(\\vz')\\| \\leq L_1\\|\\vz-\\vz'\\|$, $\\forall \\vz,\\vz' \\in \\reals^d$.  \n %\n\\end{assumption}","sha256_utf8":"657178a9c67b0fa6178e0f0a573269dee27ac684a9b847a158dcc97c13a77c68"},{"start":36264,"end":36541,"start_line":966,"end_line":969,"raw_tex":"\\begin{assumption}\\label{assum:jacobian_lips}\n  The Jacobian of $\\vF$ is $L_2$-Lipschitz, i.e., $\\|\\nabla \\vF(\\vz)-\\nabla \\vF(\\vz')\\|_{\\op} \\leq L_2\\|\\vz-\\vz'\\|$, $\\forall \\vz,\\vz'\\in \\reals^d$, where $\\|\\mA\\|_{\\op} \\mydef \\sup_{\\|\\vz\\| = 1} \\|\\mA \\vz\\|$. \n  %\n\\end{assumption}","sha256_utf8":"808362cdb6abe2379b3f1deb854c685a0552b1d2131dcc40625bdd89355ff65a"},{"start":38834,"end":39051,"start_line":1008,"end_line":1011,"raw_tex":"\\myalert{{Minimization.}} As a special instance of Problem \\eqref{eq:monotone}, consider the unconstrained minimization problem \n\\begin{equation}\\label{eq:minimization}\n  \\min_{\\vz \\in \\reals^d} f(\\vz),\n\\end{equation}","sha256_utf8":"509fe7789c34cd4926cb09dff0ddfb894784f86d7520bac2f7f379b08812a7be"},{"start":39052,"end":40186,"start_line":1012,"end_line":1013,"raw_tex":"%\nwhere $f\\!:\\! \\reals^d\\! \\rightarrow \\!\\reals$ is a (strongly) convex function. By the first-order optimality condition, the optimal solution of \\eqref{eq:minimization} satisfies the nonlinear equation $\\nabla f(\\vz) = 0$. Thus, in this case, $\\vF$ is  the gradient operator, i.e., $\\vF(\\vz) = \\nabla f(\\vz)$, and its Jacobian is the Hessian matrix of $f$, i.e., $\\nabla \\vF(\\vz) = \\nabla^2 f(\\vz)$. In particular, we observe that the Jacobian is a symmetric matrix, which allows us to simplify the conditions in Lemma~\\ref{lem:jacobian}. Specifically, under Assumptions~\\ref{assum:strong_monotone} and~\\ref{assum:operator_lips},  $\\nabla \\vF$ belongs to the set $\\{\\mA \\in \\mathbb{S}^d\\!: \\mu \\mI \\preceq\\! \\mA\\! \\preceq L\\mI\\}$. Meanwhile, under Assumptions~\\ref{assum:monotone} and~\\ref{assum:operator_lips}, it belongs to the set $\\{\\mA \\in \\mathbb{S}^d\\!: 0 \\preceq\\! \\mA\\! \\preceq L\\mI\\}$. As discussed in Section~\\ref{subsec:nonlinear_eqs}, in our method, we ensure that the Jacobian approximation matrix stays symmetric and positive semidefinite, which is a key property for the efficient implementation of the subroutines.\n","sha256_utf8":"a72f16538da8be2c1ab426fe4823f616866d57e071172d37f21dab9ebd784ba8"},{"start":40200,"end":40452,"start_line":1016,"end_line":1019,"raw_tex":"\\myalert{Minimax optimization.} Another important instance of Problem \\eqref{eq:monotone} is the unconstrained minimax optimization problem \n\\begin{equation}\\label{eq:minimax}\n  \\min_{\\vx \\in \\reals^m} \\max_{\\vy \\in \\reals^n} f(\\vx,\\vy),\n\\end{equation}","sha256_utf8":"6d2857885db78116cd31e2798a1b65f319331bf03ac11704aae29203ad58fb7f"},{"start":40453,"end":41808,"start_line":1020,"end_line":1034,"raw_tex":"%\nwhere $f$ is (strongly) convex with respect to $\\vx$ and (strongly) concave with respect to $\\vy$. To reformulate this problem as \\eqref{eq:monotone}, we define the saddle point operator as $\\vF(\\vz) = (\\nabla f_{\\vx}(\\vx,\\vy),-\\nabla f_{\\vy}(\\vx,\\vy)) \\in \\reals^{m+n}$, where $\\vz = (\\vx,\\vy) \\in \\reals^{m+n}$. Indeed, finding a solution of $\\vF(\\vz)$ is equivalent to finding a stationary point of the minimax problem, which is the optimal solution. In this case, the Jacobian is   \n$\n    \\nabla \\vF(\\vz) = \n    \\begin{bmatrix}\n        \\nabla^2 f_{\\vx\\vx}(\\vx,\\vy) & \\nabla^2 f_{\\vx \\vy}(\\vx,\\vy) \\\\\n        -\\nabla^2 f_{\\vx \\vy}(\\vx,\\vy)^\\top & -\\nabla^2 f_{\\vy\\vy}(\\vx,\\vy)\n    \\end{bmatrix}$.\nAs noted in~\\cite{asl2023j}, $\\nabla \\vF(\\vz)$ exhibits $\\mJ$-symmetric structure~\\cite{mackey2003structured}, where it is symmetric on the main diagonal blocks and anti-symmetric on the off-diagonal blocks. \nSpecifically, for  $\n    \\mJ = \\begin{bmatrix}\n        \\mI_{m\\times m} & 0 \\\\\n        0 & - \\mI_{n\\times n}\n    \\end{bmatrix}$,  matrix $\\mM$ is $\\mJ$-symmetric if $ \\mJ\\mM$ is symmetric, i.e., $ \\mJ \\mM = \\mM^\\top \\mJ$. \nSimilarly, as discussed in Section~\\ref{subsec:nonlinear_eqs}, we can ensure that our Jacobian approximation matrix in our method respects the $\\mJ$-symmetry property, simplifying the implementation of some subroutines.  \n","sha256_utf8":"75f3422cab9a8c3c7cb89a7d9fe392d74d27d584ec7a5872b7798b95e5ea6fe5"},{"start":42198,"end":42687,"start_line":1041,"end_line":1045,"raw_tex":"Specifically, assume that we know the sparsity pattern of the Jacobian matrices, defined as the set\n$\n  \\Omega = \\{(i,j):\\,\\exists\\, \\vz \\in \\reals^{d}\\,\\mathrm{s.t.}\\,[\\nabla \\vF(\\vz)]_{ij} \\neq 0, \\;1\\leq i \\leq d,\\; 1\\leq j \\leq d\\}$,\nwhere $[\\nabla \\vF(\\vz)]_{ij}$ denotes the $(i,j)$-the entry of the matrix $\\nabla \\vF(\\vz)$. \nMoreover, we say a matrix $\\mM\\in \\reals^{d \\times d}$ has the sparsity pattern $\\Omega$ if $[\\mM]_{ij} = 0$ for all $(i,j) \\notin \\Omega$ and $i \\neq j$. \n","sha256_utf8":"749ae7cedec0d39aff3ca14a07826cb27ad1bb9e477e58bbaf7e30307351b31b"},{"start":43154,"end":43385,"start_line":1054,"end_line":1055,"raw_tex":"In the strongly monotone setting (under Assumption~\\ref{assum:strong_monotone}), \nrecall that there is a unique solution $\\vz^*$ to Problem~\\eqref{eq:monotone}. Hence, we measure the suboptimality in terms of distance to $\\vz^*$. \n","sha256_utf8":"1617f1e1aca3bcec10a3b61dc2f5f190b3c897c859151ded6d997b535e374b36"},{"start":44061,"end":44199,"start_line":1058,"end_line":1060,"raw_tex":"\\begin{equation}\n  \\gap_{\\text{w}}(\\vz; \\mathcal{D}) = \\max_{\\vz' \\in \\mathcal{D}} \\langle \\vF(\\vz'),   \\vz - \\vz' \\rangle,\n\\end{equation}","sha256_utf8":"dbf4c5202bb4be6bf8f99c4a1035671ba7d48a4ef628ff01202a098005da34f4"},{"start":44996,"end":45070,"start_line":1064,"end_line":1066,"raw_tex":"\\begin{equation}\n  \\gap_f(\\vz; \\vz^*) = f(\\vz) - f(\\vz^*). \n\\end{equation}","sha256_utf8":"a6d6d77a3f06068a5c1d60e7dd52a7238bda6d6cee932a704667c330b7b42851"},{"start":45164,"end":45309,"start_line":1068,"end_line":1070,"raw_tex":"\\begin{equation}\n  \\gap_{\\text{pd}}(\\vz;\\calX\\times \\calY) = \\max_{\\vy' \\in \\calY} f(\\vx,\\vy') - \\min_{\\vx'\\in \\calX} f(\\vx',\\vy),\n\\end{equation}","sha256_utf8":"ab4efe42efcaa9c624d08262eaa92c4b8d201e3dac52483a71bc6d246326d7d5"},{"start":45310,"end":45426,"start_line":1071,"end_line":1071,"raw_tex":"where $\\vz = (\\vx,\\vy)$ and $\\calX$ and $\\calY$ are given compact sets in $\\reals^m$ and $\\reals^n$, respectively. \n","sha256_utf8":"f6f367b3be238d27e95dbcc6241af6e756228e60edf7165d83fec5688c03afa9"},{"start":46435,"end":47009,"start_line":1083,"end_line":1090,"raw_tex":"In Lemma~\\ref{lem:averaging}, we observe that the gap functions in the three cases can all be upper bounded by a similar quantity. Thus, to unify these different notions, we define  \n\\begin{equation}\\label{eq:gap}\n  \\gap(\\vz; \\mathcal{D}) = \\begin{cases}\n    \\gap_{\\mathrm{w}}(\\vz; \\mathcal{D}), & \\text{for Problem~\\eqref{eq:monotone};}\\\\\n    \\max_{\\vz' \\in \\mathcal{D}}\\{f(\\vz) - f(\\vz')\\}, & \\text{for Problem~\\eqref{eq:minimization}};\\\\\n    \\max_{(\\vx',\\vy') \\in \\calD} \\{f(\\vx,\\vy') - f(\\vx',\\vy)\\}, & \\text{for Problem~\\eqref{eq:minimax}}.\n  \\end{cases}\n\\end{equation}","sha256_utf8":"7527f8e5ce23798431eae4f36fc2359590f810d1a43cc4a2f13b58b6787f0148"},{"start":47010,"end":47245,"start_line":1091,"end_line":1091,"raw_tex":"Specifically, note that $\\gap(\\vz; \\{\\vz^*\\})$ reduces to $\\gap_f(\\vz;\\vz^*)$ for Problem~\\eqref{eq:minimization} and $\\gap(\\vz; \\calX\\times \\calY)$ reduces to $\\gap_{\\text{pd}}(\\vz;\\calX\\times \\calY)$ for Problem~\\eqref{eq:minimax}. \n","sha256_utf8":"32ee5175992efe4b2c153e4b3699e14a6f74089bdd220d7ec7e60ecc72d63888"},{"start":51153,"end":53695,"start_line":1142,"end_line":1184,"raw_tex":"\\begin{algorithm}[t!]\\small\n  \\caption{Quasi-Newton Proximal Extragradient (QNPE) Method (informal)}\\label{alg:Full_Equasi-Newton}\n    \\begin{algorithmic}[1]\n        \\STATE \\textbf{Input:} strong monotonicity parameter $\\mu \\geq 0$, Jacobian feasible set $\\mathcal{Z}$, line search parameters $\\alpha_1\\geq 0$ and $\\alpha_2>0$ such that $\\alpha_1+\\alpha_2 < 1$,  and initial trial step size $\\sigma_0>0$ \n        \\STATE \\textbf{Initialization:} initial point $\\vz_0\\in \\mathbb{R}^d$ and initial Jacobian approximation $\\mB_0\\in \\mathcal{Z}$ %\n        %\n        \\FOR{iteration $k=0,\\ldots,N-1$}\n        %\n        %\n        \\STATE Let $\\eta_k$ be the largest possible step size in $\\{\\sigma_k\\beta^i:i\\geq 0\\}$ such that  \\tikzmark{top} \n        %\n        \\begin{gather*}\n          \\hspace{-8em}\\|{\\hat{\\vz}_k}-\\vz_k+{\\eta_k}(\\vF(\\vz_k)+\\mB_k({\\hat{\\vz}_k}-\\vz_k))\\| \\leq \\alpha_1 \\sqrt{1+\\eta_k \\mu} \\|\\hat{\\vz}_k-\\vz_k\\|, \\tikzmark{right} %\n          \\\\\n          \\hspace{-8em}\\|\\hat{\\vz}_k - \\vz_k + \\eta_k \\vF(\\hat{\\vz}_k)\\| \\leq { (\\alpha_1+\\alpha_2)\\sqrt{1+\\eta_k \\mu} \\|{\\hat{\\vz}_k}-\\vz_k\\|}\n           %\n      \\end{gather*}%\n\n%\n        \\STATE Set $\\sigma_{k+1} \\leftarrow \\eta_{k}/\\beta$ \\tikzmark{bottom}\n        \\STATE Update \n        $\n          \\vz_{k+1} \\leftarrow \\theta_k (\\vz_k - \\eta_k \\vF(\\hat{\\vz}_k))+(1-\\theta_k) \\hat{\\vz}_k\n        $, where $\\theta_k = \\frac{1}{1+2\\eta_k\\mu}$\n        %\n        %\n        \\IF[Line search accepted the initial trial step size]{$\\eta_k = \\sigma_k$ \\tikzmark{top2}}\n        \\STATE Set $\\mB_{k+1} \\leftarrow \\mB_k$ \\label{line:Hessian_approx_unchanged}\n        \\ELSE[Line search bactracked]\n        \\STATE Let $\\tilde{\\vz}_k$ be the last rejected iterate in the line search  %\n        \\STATE Set $\\vu_k \\leftarrow \\vF(\\tilde{\\vz}_k) -\\vF({\\vz_k})$, $\\vs_k \\leftarrow \\tilde{\\vz}_k-\\vz_k$\n        \\STATE Define the loss function $\\ell_k(\\mB) = \\frac{\\|\\vu_k-\\mB\\vs_k\\|^2}{\\|\\vs_k\\|^2}$ \n        \\STATE Feed $\\ell_k(\\mB)$ to an online learning algorithm $\\mathcal{A}$ and obtain $\\mB_{k+1}$ \\quad \\tikzmark{right2}\n        \\ENDIF \\tikzmark{bottom2}\n        \\ENDFOR\n                 \n    \\end{algorithmic}\n    \\AddNote{top}{bottom}{right}{\\hspace{-.5em}\\color{comment}\\textit{\\quad Line search subroutine;\\\\\n             \\hspace{.2em} see Section~\\ref{subsec:ls}}}\n    \\AddNote{top2}{bottom2}{right2}{\\hspace{-.5em}\\color{comment}\\textit{\\quad Jacobian approximation\\\\ \\hspace{.2em} update subroutine; see \\\\\n    \\hspace{.2em} Section~\\ref{sec:jacobian_online_learning}}}\n    %\n  \\end{algorithm}","sha256_utf8":"19ee4be9420cc8edbad5dec6ace028193cc015eec22d51bcae60830abbcdadcc"},{"start":55418,"end":55779,"start_line":1201,"end_line":1205,"raw_tex":"\\begin{align}\n    \\|{\\hat{\\vz}_k}-\\vz_k+{\\eta_k}(\\vF(\\vz_k)+\\mB_k({\\hat{\\vz}_k}-\\vz_k))\\| &\\leq {\\alpha_1} \\sqrt{1+\\eta_k \\mu} \\|\\hat{\\vz}_k-\\vz_k\\|, \\label{eq:inexact_linear_solver} \\\\\n    \\|\\hat{\\vz}_k - \\vz_k + \\eta_k \\vF(\\hat{\\vz}_k)\\| &\\leq {(\\alpha_1+\\alpha_2) \\sqrt{1+\\eta_k \\mu} \\|{\\hat{\\vz}_k}-\\vz_k\\|} \\label{eq:inexact_proximal_point}, \n%\n\\end{align}","sha256_utf8":"8bf0213f4df432f3cb4f25863edafaf966164c5ffc426563138063849b686e57"},{"start":55780,"end":55891,"start_line":1206,"end_line":1206,"raw_tex":"where $\\alpha_1\\in [0,1)$ and $\\alpha_2 \\in (0,1)$ are user-specified parameters with $\\alpha_1+\\alpha_2<1$.  \n","sha256_utf8":"3ee73c0f130270baaa7e3aa8121066811e5f4bf6f38443797a4270697ddd1d66"},{"start":57402,"end":57547,"start_line":1216,"end_line":1219,"raw_tex":"\\begin{equation}\\label{eq:extragradient_mixing}\n%\n  \\vz_{k+1} = \\theta_k(\\vz_k - \\eta_k \\vF(\\hat{\\vz}_k))+(1-\\theta_k)\\hat{\\vz}_k,\n\\end{equation}","sha256_utf8":"b722d49886605a1fc7aba479ee78ea1dea024bcff0ae6fe3838fa206b92241ef"},{"start":57548,"end":57962,"start_line":1220,"end_line":1220,"raw_tex":"where $\\theta_k \\in [0,1]$ is chosen based on our convergence analysis. Specifically, in the monotone setting (under Assumption~\\ref{assum:monotone}), we choose $\\theta_k = 1$ and \\eqref{eq:extragradient_mixing} reduces to the extragradient step in \\eqref{eq:extragradient}. Moreover, in the strongly monotone setting (under Assumption~\\ref{assum:strong_monotone}), we choose $\\theta_k  = \\frac{1}{1+2\\eta_k\\mu}$.\n","sha256_utf8":"272e0e3aaa81a78fed9ebbd990cc500428e6d42bd243a7eb6657895e2caaec5e"},{"start":59742,"end":61008,"start_line":1244,"end_line":1259,"raw_tex":"\\begin{algorithmic}[1]\n      \\STATE \\textbf{Input:} iterate $\\vz \\in \\mathbb{R}^d$,  operator $\\vg\\in \\reals^d$,  Jacobian approximation $\\mB$, initial trial step size $\\sigma>0$\n      \\STATE \\textbf{Parameters:} line search parameters $\\beta\\in (0,1)$, $\\alpha_1\\geq 0$ and $\\alpha_2>0$ such that $\\alpha_1+\\alpha_2<1$\n      %\n      \\STATE Set ${\\eta}_{+} \\leftarrow \\sigma$, {$\\vs_{+} \\leftarrow \\mathsf{LinearSolver}(\\mI+\\eta_{+}\\mB, -\\eta_{+}\\vg; {\\alpha_1}\\sqrt{1+\\eta_+ \\mu})$ and $\\hat{\\vz}_{+} \\leftarrow \\vz+\\vs_{+}$}\n      \\WHILE{$\\|\\hat{\\vz}_+ - \\vz + \\eta_+ \\vF(\\hat{\\vz}_+)\\| \\geq (\\alpha_1+\\alpha_2)\\sqrt{1+\\eta_+ \\mu} \\|{\\hat{\\vz}_{+}}-\\vz\\|$\\label{line:check_condition}}\n        \\STATE Set $\\tilde{\\vz} \\leftarrow \\hat{\\vz}_{+}$ %\n          and $\\eta_{+} \\leftarrow \\beta\\eta_+ $ \\label{line:decreasing_stepsize}\n      \\STATE Compute $\\vs_{+} \\leftarrow \\mathsf{LinearSolver}(\\mI+\\eta_{+}\\mB, -\\eta_{+}\\vg; \\alpha_1\\sqrt{1+\\eta_+ \\mu})$ and $\\hat{\\vz}_{+} \\leftarrow \\vz+\\vs_{+}$\n      \\ENDWHILE\n      \\IF{$\\eta_+ = \\sigma$}\n      \\STATE \\textbf{Return} $\\eta_{+}$ and  $\\hat{\\vz}_{+}$ \\label{line:return_1}\n      \\ELSE \n      \\STATE \\textbf{Return} $\\eta_{+}$, $\\hat{\\vz}_{+}$ and $\\tilde{\\vz}$ \\label{line:return_2}\n      \\ENDIF\n  \\end{algorithmic}","sha256_utf8":"ad485b8be3aefeba0506cbbe2dcbca10de2e4af693aab892c30cff2f622356d2"},{"start":61027,"end":61797,"start_line":1262,"end_line":1270,"raw_tex":"Next, we present the backtracking line search scheme for selecting $\\eta_k$ and the iterate $\\hat{\\vz}_k$ in the first stage of QNPE. For brevity, we denote $\\vF(\\vz_k)$ by $\\vg$ and omit the iteration subscript $k$ from both $\\vz_k$ and $\\mB_k$. %\nIn light of \\eqref{eq:inexact_linear_solver} and \\eqref{eq:inexact_proximal_point}, our goal at the $k$-th iteration is to identify a pair $(\\eta_{+}, \\hat{\\vz}_{+})$ satisfying  \n%\n\\begin{align}\n  %\n  \\|\\hat{\\vz}_{+}-\\vz+{\\eta_{+}}(\\vg+\\mB({\\hat{\\vz}_{+}}-\\vz))\\| &\\leq  {\\alpha_1}\\sqrt{1+\\eta_+ \\mu} \\|{\\hat{\\vz}_{+}}-\\vz\\|,\n  \\label{eq:x_plus_update} \\\\\n  \\|\\hat{\\vz}_+ - \\vz + \\eta_+ \\vF(\\hat{\\vz}_+)\\| &\\leq (\\alpha_1+\\alpha_2)\\sqrt{1+\\eta_+ \\mu} \\|{\\hat{\\vz}_{+}}-\\vz\\|  \\label{eq:step size_condition}. \n\\end{align}","sha256_utf8":"42ff624ab576ef76e8032c30f4a403d27581526d6dfa0ad66759fcb9769c3976"},{"start":61798,"end":62007,"start_line":1271,"end_line":1271,"raw_tex":"As previously discussed, with $\\eta_{+}$ fixed, the first condition in \\eqref{eq:x_plus_update} can be met by solving the linear system $(\\mI+\\eta_{+}\\mB)(\\hat{\\vz}_+-\\vz) = -\\eta_+\\vg$ to a desired accuracy.\n","sha256_utf8":"a76cf956036650002274442ee3ec8497cfd4dec305888f59a7db9e7f8c6fd9ef"},{"start":62036,"end":62250,"start_line":1276,"end_line":1279,"raw_tex":"\\begin{equation}\\label{eq:linear_solver_update}\n  \\vs_{+} = \\mathsf{LinearSolver}(\\mI+\\eta_{+}\\mB, -\\eta_{+}\\vg; {\\alpha_1}\\sqrt{1+\\eta_+ \\mu}) \\quad \\text{and} \\quad \\hat{\\vz}_{+} = \\vz+\\vs_{+},\n  %\n\\end{equation}","sha256_utf8":"c850ccd4d186c4a64114664219bb184ac7047431030aab70024e147896d9abc2"},{"start":62251,"end":62317,"start_line":1280,"end_line":1280,"raw_tex":"where the $\\mathsf{LinearSolver}$ oracle is defined as follows. %\n","sha256_utf8":"38920ab11ac75d3afb53e6ff4937c8ee825ce2c6249f936c4c483dcfc7566d78"},{"start":62319,"end":62660,"start_line":1283,"end_line":1293,"raw_tex":"\\begin{definition}\\label{def:linear_solver}\n  The oracle $\\mathsf{LinearSolver}(\\mA,\\vb; \\rho)$ takes  a matrix $\\mA \\in \\reals^{d\\times d}$, a vector $\\vb\\in \\reals^d$ and $\\rho>0$ as input, and returns an approximate solution $\\vs_{+}$ \n  %\n  satisfying $\\|\\mA\\vs_{+}-\\vb\\| \\leq \\rho \\|\\vs_{+}\\|$. \n  %\n  %\n  %\n  %\n  %\n  %\n\\end{definition}","sha256_utf8":"cc80dca79490851acf0f6091c231e39ebc0b45ddf359524fec1971383df50ff8"},{"start":75808,"end":75938,"start_line":1446,"end_line":1446,"raw_tex":"Recall that $\\calB$ is the set of indices where the line search subroutine backtracks, i.e., $\\calB = \\{k: \\eta_k < \\sigma_k\\}$. \n","sha256_utf8":"2b41e200b5f235eff60d5f1cad78868fa020143f41ecb547417ef7890b04d443"},{"start":90071,"end":90176,"start_line":1700,"end_line":1700,"raw_tex":"Formally, define the loss function $\\ell_k : \\reals^{d \\times d} \\rightarrow \\reals$ at iteration $k$ as\n","sha256_utf8":"efe574605b474ad0f35986bbb49712219315391c1b9228c387d67e0584a1dc93"},{"start":90178,"end":90406,"start_line":1702,"end_line":1709,"raw_tex":"\\begin{equation}\\label{eq:loss_of_jacobian}\n  \\ell_k(\\mB) \\triangleq \n  \\begin{cases}\n    0, & \\text{if } k\\notin \\mathcal{B}, \\\\\n    \\frac{\\|\\vu_k-\\mB\\vs_k\\|^2}{\\|\\vs_k\\|^2}, & \\text{otherwise}.\n  \\end{cases}\n  %\n\\end{equation}","sha256_utf8":"8270f5a03c70c54a33f580344b2b77c6f1c609bc2461b97e63b5c63d5ab293f1"},{"start":94569,"end":94791,"start_line":1739,"end_line":1741,"raw_tex":"Given these points and the discussions in Section~\\ref{subsec:structures}, we choose the feasible set~$\\calZ$ as follows: \n\\begin{enumerate}[(a)]\n  \\item For the minimization problem in \\eqref{eq:minimization}, we choose \n","sha256_utf8":"f50cb4d526ff2dd0ad9d3b565e2d5de1e9b2685e05920a79cca1751a4935a709"},{"start":94793,"end":94941,"start_line":1742,"end_line":1744,"raw_tex":"\\begin{equation}\\label{eq:feasible_set_min}\n    \\calZ = \\{\\mB \\in \\sS^d: \\; \\mu \\mI \\preceq \\mB \\preceq L_1 \\mI,\\; \\mB \\in \\calL\\},\n  \\end{equation}","sha256_utf8":"0dcb3297008d2a242861c66e09ab623017932657422c2725860e3b760128172c"},{"start":94942,"end":95349,"start_line":1745,"end_line":1746,"raw_tex":"  where $\\calL \\subset \\sS^d$ is a linear subspace. in the general case, we set $\\calL = \\sS^d$. However, when the Hessians of $f$ exhibit the sparsity pattern $\\Omega$, we define $\\calL$ as $\\calL=\\{\\mB\\in \\sS^{d}: [\\mB]_{ij} = 0 \\text{ for all } (i,j)\\notin \\Omega \\text{ and }i\\neq j\\}$.\n  \\item For the minimax problem in \\eqref{eq:minimax} and the nonlinear equation in \\eqref{eq:monotone}, we choose \n","sha256_utf8":"051949f810115eb6fb6c34c1089fcbed42660fac5b6e4697d36334df0353f67c"},{"start":95351,"end":95583,"start_line":1747,"end_line":1750,"raw_tex":"\\begin{equation}\\label{eq:feasible_set}\n    \\mathcal{Z} = \\left\\{ \\mB \\in \\reals^{d\\times d}:\\mu \\mI \\preceq \\frac{1}{2}(\\mB + \\mB^\\top )\\preceq L_1\\mI , \\;\\|\\mB\\|_{\\op} \\leq L_1, \\mB \\in \\mathcal{L} \\right\\},\n    %\n  \\end{equation}","sha256_utf8":"36bbf1824d2d3b204cd9a5edc17a61391ed8aae60259a858fe872dc27b629e95"},{"start":95584,"end":95978,"start_line":1751,"end_line":1752,"raw_tex":"  where $\\mathcal{L} \\subset \\reals^{d\\times d}$ is a linear subspace. For general nonlinear equations $\\mathcal{L} = \\reals^{d \\times d}$, for minimax problems $\\mathcal{L} = \\{\\mB\\in \\reals^{d\\times d}: \\mB = \\mJ \\mB^\\top \\mJ\\}$, and for sparse nonlinear equations $\\calL=\\{\\mB\\in \\reals^{d\\times d}: [\\mB]_{ij} = 0 \\text{ for all } (i,j)\\notin \\Omega \\text{ and }i\\neq j\\}$.\n\\end{enumerate}\n","sha256_utf8":"7d3008c6941a1d10b92662356417784e5e15b31fcbc02773a24e83c93c65a91b"},{"start":96824,"end":98508,"start_line":1793,"end_line":1821,"raw_tex":"\\begin{algorithm}[!t]\\small\n  \\caption{Projection-Free Online Learning}\\label{alg:projection_free_online_learning}\n  \\begin{algorithmic}[1]\n      \\STATE \\textbf{Input:} Initial point $\\vw_0 \\in  \\vspan(\\calC)$, the orthogonal projection matrix $\\mP$ associated with the subspace $\\vspan(\\calC)$, step size $\\rho>0$, $\\delta>0$, radius $R>0$\n      %\n      %\n      \\FOR{$t=0,1,\\dots T-1$}\n      \\STATE Query the oracle $(\\gamma_t,\\vs_t) \\leftarrow \\mathsf{SEP}_{\\calC}(\\vw_t;\\delta)$\n      \\IF[Case I: we have $\\vw_t\\in (1+\\delta)\\mathcal{C}$]{$\\gamma_t \\leq 1$}\n      %\n      %\n        \\STATE Set $\\vx_t \\leftarrow \\begin{cases} \\vw_t & (\\textbf{Option I}) \\\\ \\frac{\\vw_t}{1+\\delta} & (\\textbf{Option II}) \\end{cases}$ and play the action $\\vx_t$ \n        \\STATE Receive the loss $\\ell_t(\\vx_t)$ and the subspace gradient $\\vg_t = \\mP\\nabla \\ell_t(\\vx_t)$\n        \\STATE Set $\\tilde{\\vg}_t \\leftarrow \\vg_t$\n      %\n      %\n      \\ELSE[Case II: we have $\\vw_t/\\gamma_t\\in (1+\\delta)\\mathcal{C}$] \n      %\n        \\STATE Set $\\vx_t \\leftarrow \\begin{cases} \\frac{\\vw_t}{\\gamma_t} & (\\textbf{Option I}) \\\\ \\frac{\\vw_t}{(1+\\delta)\\gamma_t} & (\\textbf{Option II}) \\end{cases}$ and play the action $\\vx_t$ \n      \\STATE Receive the loss $\\ell_t(\\vx_t)$ and the subspace gradient $\\vg_t = \\mP\\nabla \\ell_t(\\vx_t)$\n      \\STATE Set $\\tilde{\\vg}_t \\leftarrow \\vg_t+\\max\\left\\{0, - \\frac{1}{\\gamma_t}\\langle \\vg_t, \\vw_t \\rangle\\right\\} \\vs_t$\n      %\n      \\ENDIF\n      %\n      \\STATE Update $\\vw_{t+1} \\leftarrow \\frac{R(\\vw_t-\\rho \\tilde{\\vg}_t)}{\\max\\{\\|\\vw_t-\\rho \\tilde{\\vg}_t\\|_2,R\\}} $ \n      \\COMMENT{Online projected gradient descent}\n      \\ENDFOR\n  \\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"c89799edb9105168a6deb4dbfa4fa3a659cafd3f443ad2a3d68f7ff6bcf90779"},{"start":98510,"end":99510,"start_line":1823,"end_line":1825,"raw_tex":"To set the stage for our Jacobian approximation update algorithm, we take a detour and consider a general online learning problem over a compact feasible set $\\calC \\subset \\reals^D$. For $T$ consecutive rounds $t=0,\\dots,T-1$, a learner chooses an action $\\vx_t \\in \\reals^D$ and then observes a convex loss function $\\ell_t:\\reals^D \\rightarrow \\reals$. \n\\blue{The goal is to minimize either the \\emph{static regret} defined by $\\mathrm{Reg}_T(\\vu; \\calC) \\triangleq \\sum_{t=0}^{T-1} \\ell_t(\\vx_t) - \\sum_{t=0}^{T-1} \\ell_t(\\vu)$ with a fixed competitor $\\vx \\in \\calC$, or the \\emph{dynamic regret} defined by $\\dreg_T(\\vu_0,\\dots,\\vu_{T-1}; \\calC) \\triangleq \\sum_{t=0}^{T-1} \\ell_t(\\vx_t) - \\sum_{t=0}^{T-1} \\ell_t(\\vu_t)$ with a sequence of competitors $\\vu_t \\in \\calC$ for any $t \\geq 0$.}\nWe assume $0\\in \\mathcal{C}$ without loss of generality. Moreover, we assume that the set $\\calC$ is contained in the Euclidean ball $B_R(0):= \\{\\vw\\in \\reals^D: \\|\\vw\\|\\leq R\\}$ for some radius $R>0$.\n","sha256_utf8":"2c031611f3a1e292c21c8f5ba1c5217495d6336f5fecb076a56ddb2a10742310"},{"start":99829,"end":100234,"start_line":1830,"end_line":1830,"raw_tex":"To start, we will define the approximate separation oracle $\\SEP_{\\cal{C}}$ for a compact set $\\calC$. We use $\\vspan(\\mathcal{C})$ to denote the linear span of $\\mathcal{C}$, i.e., $\\vspan(\\calC) = \\left\\{\\sum_{i=1}^{k}  \\alpha_i \\vx_i : k \\in \\mathbb{Z}_+,\\;\\vx_i \\in \\calC,\\; \\alpha_i \\in \\reals  \\right\\}$. Intuitively, $\\vspan(\\calC)$ is the smallest linear subspace that contains the set $\\calC$. %\n","sha256_utf8":"a14e07dcb3e5b88f8468a798dc7bb49b15b42a7f20536e90ce7d01cbd2a3febc"},{"start":100236,"end":100914,"start_line":1833,"end_line":1848,"raw_tex":"\\begin{definition}\\label{def:gauge}\n  Let $\\calC$ be a compact convex set in $\\reals^D$ containing the origin. \n  The oracle $\\mathsf{SEP}_{\\mathcal{C}}(\\vw; \\delta)$ takes $\\vw\\!\\in\\! \\vspan(\\calC)$ and $\\delta>0$ and returns $\\gamma>0$ and a vector $\\vs\\in \\vspan(\\calC)$ with one of these outcomes:\n   %\n   %\n    \\begin{itemize}\n      \\item Case I: $\\gamma \\leq 1$, which implies that $\\vw \\in (1+\\delta)\\mathcal{C}$;\n       %\n      \\item Case II: $\\gamma>1$, which implies that $\\vw/\\gamma \\in (1+\\delta)\\mathcal{C} \\ $ and $\\ \\langle \\vs, \\vw-\\vx \\rangle \\geq {\\gamma-1},$  $ \\forall\\vx\\in \\mathcal{C}$. \n    \\end{itemize}\n    %\n    %\n    %\n    %\n    %\n    \\end{definition}","sha256_utf8":"a3d4087636b44d0500ad61daacbf36e44d2b3e9ae86271c8e27af902c50e399d"},{"start":105818,"end":106047,"start_line":1879,"end_line":1881,"raw_tex":"\\begin{equation}\\label{eq:w_update}\n  \\vw_{t+1}  = \\mathrm{\\Pi}_{\\mathcal{B}_R(0)}\\bigl(\\vw_t-\\rho \\nabla \\tilde{\\ell}_t(\\vw_t)\\bigr) = \\frac{R}{\\max\\{\\|\\vw_t-\\rho \\tilde{\\vg}_t\\|_2,R\\}} (\\vw_t-\\rho \\tilde{\\vg}_t).\n\\end{equation}","sha256_utf8":"143712ad471dde04a35a8607f3ccd4ad911559f0c2aa6b65410458f591d071d8"},{"start":120265,"end":122644,"start_line":2155,"end_line":2198,"raw_tex":"\\begin{algorithmic}[1]\n      \\STATE \\textbf{Input:} Initial matrix $\\mB_0\\in \\mathcal{Z}$, the orthogonal projection matrix $\\mP$, step size $\\rho>0$, $\\delta>0$, failure probabilities $\\{q_t\\}_{t=1}^{T-1}$, Lipschitz constant $L_1>0$ and strong monotonicity parameter $\\mu \\geq 0$\n      \\STATE \\textbf{Initialize:} set $\\mW_0 \\leftarrow \\frac{1}{L_1}(\\mB-(L_1+\\mu)\\mI)$, $\\mG_0 \\leftarrow \\frac{1}{L_1} \\mP\\nabla \\ell_0(\\mB_0)$ and $\\tilde{\\mG}_0 \\leftarrow \\mG_0$\n      \\STATE {Update $\\mW_{1} \\leftarrow \\frac{\\sqrt{d}(\\mW_0 - \\rho \\tilde{\\mG}_0)}{\\max\\{\\sqrt{d},\\|\\mW_0 - \\rho \\tilde{\\mG}_0\\|_F\\}}$ \\COMMENT{Projection onto $\\mathcal{B}_{\\sqrt{d}}(0)$}\\vspace{-.5em}\n      \\FOR{$t=1,\\dots,T-1$}\n      \\STATE Query $(\\gamma^{(1)}_t,\\mS^{(1)}_t) \\leftarrow \\mathsf{ExtEvec}(\\frac{\\mW_t+\\mW_t^\\top}{2};\\delta, \\frac{q_t}{2} )$ and $(\\gamma^{(2)}_t,\\mS^{(2)}_t) \\leftarrow \\mathsf{MaxSvec}(\\mW_t;\\delta, \\frac{q_t}{2} )$ %\n      \\STATE Let $i = \\argmax_{i\\in \\{1,2\\}} \\gamma_t^{(i)}$, and set $\\gamma_t \\leftarrow \\gamma_t^{(i)}$ and $\\mS_t \\leftarrow \\mP \\mS_t^{(i)}$\n      \\IF[Case I]{$\\gamma_t \\leq 1$}\n      %\n        \\STATE Set $\\hat{\\mB}_t \\leftarrow \\begin{cases} \\mW_t & \\text{if }\\mu>0\\;(\\textbf{Option I})\\\\ \\frac{\\mW_t}{1+\\delta} & \\text{if }\\mu=0\\; (\\textbf{Option II})\\end{cases}$ and $\\mB_t \\leftarrow {L_1}\\hat{\\mB}_t+{(L_1+\\mu)}\\mI$\n        \\STATE Set $\\mG_t \\leftarrow \\frac{1}{L_1}\\mP\\nabla \\ell_t(\\mB_t)$ and $\\tilde{\\mG}_t \\leftarrow \\mG_t$\n        %\n      \\ELSE[Case II]\n        \\STATE Set $\\hat{\\mB}_t \\leftarrow \\begin{cases} \\frac{\\mW_t}{\\gamma_t} & \\text{if }\\mu>0\\;(\\textbf{Option I})\\\\ \\frac{\\mW_t}{(1+\\delta)\\gamma_t} & \\text{if }\\mu=0\\; (\\textbf{Option II})\\end{cases}$ and $\\mB_t \\leftarrow {L_1}\\hat{\\mB}_t+{(L_1+\\mu)}\\mI$ \n        \\STATE Set $\\mG_t \\leftarrow \\frac{1}{L_1}\\mP\\nabla \\ell_t(\\mB_t)$ and $\\tilde{\\mG}_t \\leftarrow \\mG_t+\\max\\{0,-\\frac{1}{\\gamma_t}\\langle \\mG_t, \\mW_t \\rangle\\} \\mS_t$\n        %\n      \\ENDIF\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      \\STATE %\n      %\n      Update \n      $\\mW_{t+1} \\leftarrow \\frac{\\sqrt{d}(\\mW_t - \\rho \\tilde{\\mG}_t)}{\\max\\{\\sqrt{d},\\|\\mW_t - \\rho \\tilde{\\mG}_t\\|_F\\}}$ \\COMMENT{Projection onto $\\mathcal{B}_{\\sqrt{d}}(0)$}\n        %\n        %\n        %\n        %\n      %\n      \\ENDFOR}\n  \\end{algorithmic}","sha256_utf8":"ed907b4d314da5147448e9ddf6bebaf5a2b07f16db15b500c93e2f149deffe2b"},{"start":122663,"end":124471,"start_line":2201,"end_line":2206,"raw_tex":"Next, we present our online learning algorithm for updating $\\{\\mB_k\\}_{k \\geq 0}$. We follow the projection-free online learning algorithm outlined in Section~\\ref{subsec:projection_free_online_learning} and apply it to the online learning problem in Section~\\ref{subsec:convergence}.\n\\blue{Moreover, recall that $\\calB$ denotes the set of indices where the line search subroutine backtracks, and we have $\\ell_k(\\mB)=0$ when $k\\notin \\mathcal{B}$. Thus, we can simply keep $\\mB_{k+1}$ unchanged for these iterations (cf. Line~\\ref{line:Hessian_approx_unchanged} in Algorithm~\\ref{alg:Full_Equasi-Newton}). Suppose $\\calB = \\{k_0,k_1,\\dots,k_{T-1}\\}$, where $T \\leq N$. With a slight abuse of notation, in the following, we relabel the indices in $\\mathcal{B}$ as $t=0,\\dots,T-1$.}\n\nThe remaining question is how to construct the approximate separation oracle, required by the projection-free scheme and defined in Definition~\\ref{def:gauge}, for our specific set $\\calZ$.\nTo start, we consider the nonlinear monotone equation in \\eqref{eq:monotone}; we will further discuss the special cases in the next section.  \nRecall that the feasible set~$\\calZ$ for our online learning problem in Section~\\ref{subsec:convergence} is given in \\eqref{eq:feasible_set} with a linear subspace $\\calL$. In the following, we assume that the orthogonal projection matrix $\\mP$ associated with the linear subspace $\\calL$ is given. Specifically, for a general nonlinear monotone equation, we have $\\calL = \\reals^d$ and the matrix $\\mP$ is simply the identity matrix. Moreover, for those special cases discussed in Section~\\ref{subsec:structures}, the linear subspace is specified at the end of Section~\\ref{subsec:convergence} and we will discuss the corresponding projection matrix in Section~\\ref{subsec:special_cases}. \n","sha256_utf8":"a5b84d1dd7908e76ef591b68b200b56598a61e241625c81c220071459ee27e5e"},{"start":127167,"end":127575,"start_line":2255,"end_line":2257,"raw_tex":"Hence, we will work with the new set $\\calC$ in our online learning algorithm. Moreover, note that we can write $\\calC = \\calC_1 \\cap \\calC_2 \\cap \\calL$, where we define \n%\n  $\\mathcal{C}_1 \\mydef \\{\\hat{\\mB} \\in \\mathbb{R}^{d\\times d}:  -\\mI \\preceq \\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top )\\preceq \\mI \\}$ and $\\mathcal{C}_2 \\mydef \\{\\hat{\\mB} \\in \\mathbb{R}^{d\\times d}: \\|\\hat{\\mB}\\|_{\\op} \\leq 3 \\}$.  \n","sha256_utf8":"9ca3e0174df69f07fa317f8b9cc7e58e8759e76ff394c229730c471be16e024c"},{"start":129770,"end":130629,"start_line":2275,"end_line":2294,"raw_tex":"\\begin{definition}\\label{def:extevec}\n  The oracle $\\mathsf{ExtEvec}(\\mW;\\delta,q)$ takes $\\mW \\in \\reals^{d\\times d}$, $\\delta>0$, and $q\\in (0,1)$ as input and returns a scalar $\\gamma>0$ and a matrix $\\mS\\in \\mathbb{S}^d$. With probability at least $1-q$, the returned $\\gamma$ and $\\mS$ satisfy one of the following properties::\n  %\n  \\begin{itemize}\n    \\item Case I: $\\gamma \\leq 1$, which implies that\n    %\n    $-(1+\\delta)\\mI \\preceq \\frac{1}{2}(\\mW+\\mW^\\top) \\preceq (1+\\delta)\\mI$;\n    %\n    \\item Case II: $\\gamma>1$, which implies that %\n    %\n    $-(1+\\delta)\\mI \\preceq \\frac{1}{2\\gamma}(\\mW+\\mW^\\top) \\preceq (1+\\delta)\\mI$, \n    $\\|\\mS\\|_F \\leq 1$ and $\\langle \\mS,\\mW-\\hat{\\mB}\\rangle \\geq \\gamma -1$ for any $\\hat{\\mB}$ such that $-\\mI \\preceq \\frac{1}{2}(\\hat{\\mB}+\\hat{\\mB}^\\top) \\preceq \\mI$.\n  \\end{itemize}\n%\n%\n%\n%\n%\n%\n\\end{definition}","sha256_utf8":"4472affc88894f00a9a0304669b7f605934042fedc5ed16a4091fff71ab65593"},{"start":132399,"end":133148,"start_line":2302,"end_line":2310,"raw_tex":"\\begin{definition}\\label{def:MaxSvec}\n  The oracle $\\mathsf{MaxSvec}(\\mW;\\delta,q)$ takes $\\mW \\in \\mathbb{R}^{d\\times d}$, $\\delta>0$, and $q\\in (0,1)$ as input and returns a scalar $\\gamma>0$ and a matrix $\\mS\\in \\mathbb{R}^{d\\times d}$. With probability at least $1-q$, the returned $\\gamma$ and $\\mS$ satisfy one of the following properties::\n  %\n  \\begin{itemize}\n    \\item Case I: $\\gamma \\leq 1$, which implies that $ \\|\\mW\\|_{\\op} \\leq 3(1+\\delta) $;\n    %\n    \\item Case II: $\\gamma>1$, which implies that $ \\|\\mW/\\gamma \\|_{\\op} \\leq 3(1+\\delta)$, $\\|\\mS\\|_F \\leq 1$ and $\\langle \\mS,\\mW-\\hat{\\mB}\\rangle \\geq \\gamma -1$ for any $\\hat{\\mB} \\in \\reals^{d\\times d}$ such that $ \\|\\hat{\\mB}\\|_{\\op}  \\leq 3 $.\n  \\end{itemize}\n\\end{definition}","sha256_utf8":"8732120fb2ec7f35f54d591add6397ec16060bf8200dc7b882c8f9d3c785944a"},{"start":144946,"end":146889,"start_line":2515,"end_line":2541,"raw_tex":"\\begin{algorithmic}[1]\n      \\STATE \\textbf{Input:} $\\mW \\in \\reals^{d\\times d}$, $\\delta>0$, $q\\in (0,1)$\n      \\STATE \\textbf{Initialize:} sample $\\vv_1\\in \\reals^d$ uniformly from the unit sphere, $\\beta_1 \\leftarrow 0$, $\\vv_0\\leftarrow 0$\n      \\STATE Set $\\bar{\\mW} \\leftarrow \\frac{1}{2}(\\mW + \\mW^\\top)$ and set the number of iterations \n      $N \\leftarrow \\Bigl\\lceil\\frac{1}{4}\\sqrt{2(1+\\frac{1}{\\delta})}\\log(\\frac{11d}{q^2})+\\frac{1}{2} \\rceil$\n      %\n      %\n      %\n      %\n      \\FOR{$k=1,\\dots,N$ \\tikzmark{top}}\n      \\STATE Set $\\vw_k \\leftarrow \\bar{\\mW} \\vv_k-\\beta_k \\vv_{k-1}$ \n      \\STATE Set $\\alpha_k \\leftarrow \\langle \\vw_k,\\vv_k \\rangle $ and $\\vw_k \\leftarrow \\vw_k-\\alpha_k\\vv_k$\n      \\STATE Set $\\beta_{k+1} \\leftarrow \\|\\vw_k\\|$ and $\\vv_{k+1}\\leftarrow \\vw_k/\\beta_{k+1}$\n      \\ENDFOR\n    \\STATE Form a tridiagonal matrix $\\mT \\leftarrow \\mathsf{tridiag}(\\beta_{2:N},\\alpha_{1:N},\\beta_{2:N})$\n    \\STATE \\hspace{-1em}\\COMMENT{Use the tridiagonal structure to compute eigenvectors of $\\mT$}\n    \\STATE Compute $(\\hat{\\lambda}_1,\\vz^{(1)}) \\leftarrow \\mathsf{MaxEvec}(\\mT)$ and $(\\hat{\\lambda}_d, \\vz^{(d)}) \\leftarrow \\mathsf{MinEvec}(\\mT)$ \\tikzmark{right}\n    \\STATE Set $\\vu^{(1)} \\leftarrow \\sum_{k=1}^N z^{(1)}_k\\vv_k$ and $\\vu^{(d)} \\leftarrow \\sum_{k=1}^N z^{(d)}_k\\vv_k$ \\tikzmark{bottom}\n    \\STATE Set $\\gamma \\leftarrow \\max\\{\\hat{\\lambda}_1,-\\hat{\\lambda}_d\\}$ \n    \\IF{$\\gamma \\leq 1$}\n    \\STATE Return $\\gamma$ and $\\mS = 0$ \\COMMENT{Case I: $\\gamma\\leq 1$, which implies $\\|\\mW\\|_{\\op} \\leq 1+\\delta$}\n    \\ELSIF{$\\hat{\\lambda}_1 \\geq -\\hat{\\lambda}_d$}\n      \\STATE Return $\\gamma$ and $\\mS = \\vu^{(1)}(\\vu^{(1)})^\\top$ \\hspace{.75em}\\COMMENT{Case II: $\\gamma> 1$ and $\\mS$ defines a separating hyperplane}\n    \\ELSE\n      \\STATE Return $\\gamma$ and $\\mS = -\\vu^{(d)}(\\vu^{(d)})^\\top$ \\COMMENT{Case II: $\\gamma> 1$ and $\\mS$ defines a separating hyperplane}\n    \\ENDIF\n  \\end{algorithmic}","sha256_utf8":"d877a217bde80ca0e7ed667edf5116d837ab2d393420e904cc0c2542b283b527"},{"start":153362,"end":154589,"start_line":2637,"end_line":2669,"raw_tex":"\\begin{algorithmic}[1]\n    \\STATE \\textbf{Input:} $\\mW \\in \\mathbb{R}^{d\\times d}$, $\\delta>0$, $q\\in (0,1)$\n    %\n    \\STATE Set $\\tilde{\\mW} \\leftarrow \\begin{bmatrix} 0 & \\mW \\\\ \\mW^\\top & 0\n    \\end{bmatrix} \\in \\mathbb{S}^{2d}$ and the number of iterations $N  \\leftarrow \n    %\n    \\Bigl\\lceil \\frac{1}{4}\\sqrt{2(1+\\frac{1}{\\delta})}\\log\\frac{22d}{q^2}+\\frac{1}{2}\\Bigr\n    \\rceil\n    %\n    $\n    \n    \\STATE Compute approximate largest eigenvalue $\\tilde{\\lambda}_1$ and its eigenvector $\\tilde{\\vv} \\in \\reals^{2d}$, by running the Lanczos method on $\\tilde{\\mW}$ for $N$ iterations \\COMMENT{See Subroutine~\\ref{alg:lanczos}}\n    \\STATE Partition $\\tilde{\\vv}$ as $\\tilde{\\vv} = [\\vv,\\vv']$ where $\\vv, \\vv' \\in \\reals^d$ \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    \\STATE Set $\\gamma \\leftarrow \\tilde{\\lambda}_1/3$ \n    \\IF{$\\gamma \\leq 1$}\n    \\STATE Return $\\gamma$ and $\\mS = 0$ \\COMMENT{Case I: $\\gamma\\leq 1$, which implies $\\|\\mW\\|_{\\op} \\leq 3(1+\\delta)$}\n    \\ELSE\n      \\STATE Return $\\gamma$ and $\\mS = \\frac{2}{3}\\vv'\\vv^\\top \\in \\reals^{d\\times d}$ \\hspace{.75em}\\COMMENT{Case II: $\\gamma> 1$ and $\\mS$ defines a separating hyperplane}\n    %\n    %\n    \\ENDIF\n  \\end{algorithmic}","sha256_utf8":"478467f444bb80a4a3d4ac943543b8191c8f50854bbff199e2fc406e52070986"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\label{thm:main}\nSuppose Assumptions~\\ref{assum:strong_monotone},~\\ref{assum:operator_lips} and~\\ref{assum:jacobian_lips} hold. Let $\\{\\vz_k\\}$ and $\\{\\hat{\\vz}_k\\}$ be the iterates generated by Algorithm~\\ref{alg:Full_Equasi-Newton} using the line search scheme in Subroutine~\\ref{alg:ls}, where \\blue{$\\alpha_1, \\alpha_2 \\in (0,\\frac{1}{2})$, $\\beta\\in(0,1)$,}\n%\nand $\\sigma_0 \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$. In addition, the Jacobian approximation matrices are updated in Subroutine~\\ref{alg:jacobian_approx} with \\textbf{Option I}, where {$\\rho = \\frac{1}{121}$}, $\\delta_t = {\\frac{\\mu}{2L_1}}$, and $q_t = \\frac{p}{2.5(t+1)\\log^2(t+1)}$ for $t \\geq 1$. With probability at least $1-p$, the following holds:  \n  \\begin{enumerate}[(a)]\n  %\n    \\item (Linear convergence) For any $k\\geq 0$, we have\n    %\n        $ \\frac{\\|\\vz_{k+1}-\\vz^*\\|^2}{\\|\\vz_k-\\vz^*\\|^2 }\\leq  \\left(1+\\frac{4\\alpha_2\\beta\\mu}{15L_1}\\right)^{-1}$.\n        %\n    \\item (Superlinear convergence) We have $\\lim_{k\\rightarrow \\infty}\\frac{\\|\\vz_{k+1}-\\vz^*\\|^2}{\\|\\vz_k-\\vz^*\\|^2 } = 0$. Furthermore, define the absolute constant $C$ and the quantity $M$ by  \n    \\begin{align}\\label{eq:def_c1}\n      C &= \\frac{(1+\\alpha_1)^2}{2(1-\\alpha_1)^2\\beta^2 (1-(\\alpha_1+\\alpha_2))}, \\\\\n       M &= \\frac{121\\|\\mB_0-\\nabla^2f(\\vz^*)\\|_F^2}{L_1^2}+ \\left(C+2+\\frac{15L_1}{2\\alpha_2 \\beta \\mu}\\right)\\frac{L_2^2\\|\\vz_0-\\vz^*\\|^2}{L_1^2}. \\label{eq:def_M}\n    \\end{align}\n    Then for any $k\\geq 0$, we have \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    \\begin{equation*}\n      {%\n      \\frac{\\|\\vz_k-\\vz^*\\|^2}{\\|\\vz_0-\\vz^*\\|^2} \\leq   \\Biggl(1+ \n       \\frac{(1-\\beta)\\alpha_2\\beta\\mu}{L_1} \\min\\left\\{ \\frac{2}{15}k, \\sqrt{\\frac{k}{M}}\\right\\}\n      %\n      \\Biggr)^{-k}%\n      .}\n      %\n    \\end{equation*}\n    %\n  \\end{enumerate}\n\\end{theorem}","units":[{"start":158832,"end":160678,"start_line":2788,"end_line":2824,"raw_tex":"\\begin{theorem}\\label{thm:main}\nSuppose Assumptions~\\ref{assum:strong_monotone},~\\ref{assum:operator_lips} and~\\ref{assum:jacobian_lips} hold. Let $\\{\\vz_k\\}$ and $\\{\\hat{\\vz}_k\\}$ be the iterates generated by Algorithm~\\ref{alg:Full_Equasi-Newton} using the line search scheme in Subroutine~\\ref{alg:ls}, where \\blue{$\\alpha_1, \\alpha_2 \\in (0,\\frac{1}{2})$, $\\beta\\in(0,1)$,}\n%\nand $\\sigma_0 \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$. In addition, the Jacobian approximation matrices are updated in Subroutine~\\ref{alg:jacobian_approx} with \\textbf{Option I}, where {$\\rho = \\frac{1}{121}$}, $\\delta_t = {\\frac{\\mu}{2L_1}}$, and $q_t = \\frac{p}{2.5(t+1)\\log^2(t+1)}$ for $t \\geq 1$. With probability at least $1-p$, the following holds:  \n  \\begin{enumerate}[(a)]\n  %\n    \\item (Linear convergence) For any $k\\geq 0$, we have\n    %\n        $ \\frac{\\|\\vz_{k+1}-\\vz^*\\|^2}{\\|\\vz_k-\\vz^*\\|^2 }\\leq  \\left(1+\\frac{4\\alpha_2\\beta\\mu}{15L_1}\\right)^{-1}$.\n        %\n    \\item (Superlinear convergence) We have $\\lim_{k\\rightarrow \\infty}\\frac{\\|\\vz_{k+1}-\\vz^*\\|^2}{\\|\\vz_k-\\vz^*\\|^2 } = 0$. Furthermore, define the absolute constant $C$ and the quantity $M$ by  \n    \\begin{align}\\label{eq:def_c1}\n      C &= \\frac{(1+\\alpha_1)^2}{2(1-\\alpha_1)^2\\beta^2 (1-(\\alpha_1+\\alpha_2))}, \\\\\n       M &= \\frac{121\\|\\mB_0-\\nabla^2f(\\vz^*)\\|_F^2}{L_1^2}+ \\left(C+2+\\frac{15L_1}{2\\alpha_2 \\beta \\mu}\\right)\\frac{L_2^2\\|\\vz_0-\\vz^*\\|^2}{L_1^2}. \\label{eq:def_M}\n    \\end{align}\n    Then for any $k\\geq 0$, we have \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    \\begin{equation*}\n      {%\n      \\frac{\\|\\vz_k-\\vz^*\\|^2}{\\|\\vz_0-\\vz^*\\|^2} \\leq   \\Biggl(1+ \n       \\frac{(1-\\beta)\\alpha_2\\beta\\mu}{L_1} \\min\\left\\{ \\frac{2}{15}k, \\sqrt{\\frac{k}{M}}\\right\\}\n      %\n      \\Biggr)^{-k}%\n      .}\n      %\n    \\end{equation*}\n    %\n  \\end{enumerate}\n\\end{theorem}","sha256_utf8":"ef3d7df826ffe0c96f6286b91f998f458b473f6896c20060d59e9a960b60d314"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\\label{lem:jacobian}\nUnder Assumption~\\ref{assum:strong_monotone}, we have $\\frac{1}{2}(\\nabla \\vF(\\vz) + \\nabla \\vF(\\vz)^\\top) \\succeq \\mu \\mI$ for any $\\vz\\in \\reals^d$, while under Assumption~\\ref{assum:monotone}, we have $\\frac{1}{2}(\\nabla \\vF(\\vz) + \\nabla \\vF(\\vz)^\\top) \\succeq 0$ for any $\\vz\\in \\reals^d$. Further, under Assumption~\\ref{assum:operator_lips}, we have $\\|\\nabla \\vF(\\vz)\\|_{\\op} \\leq L_1$ for any $\\vz\\in \\reals^d$. \n\\end{lemma}\n\n\\begin{lemma}\\label{lem:averaging}\n  Suppose Assumption~\\ref{assum:monotone} holds. \n    Let $\\theta_0, \\dots, \\theta_{T-1} \\geq 0$ with $\\sum_{t=0}^{T-1} \\theta_t = 1$ and let $\\vz_0,\\dots,\\vz_{T-1}\\in \\reals^d$. Define the averaged iterates as $\\bar{\\vz}_T = \\sum_{t=0}^{T-1} \\theta_t \\vz_t$. Then: \n    \\begin{enumerate}[(i)]\n      \\item For Problem~\\eqref{eq:monotone}, $\\gap_{\\mathrm{w}}(\\vz; \\mathcal{D}) \\leq \\max_{\\vz \\in \\calD}\\sum_{t=0}^{T-1} \\theta_t \\langle \\vF(\\vz_t), \\vz_t-\\vz\\rangle$. \n      \\item For Problem~\\eqref{eq:minimization}, $\\gap_{f}(\\vz; \\vz^*) \\leq \\sum_{t=0}^{T-1} \\theta_t \\langle \\vF(\\vz_t), \\vz_t-\\vz^*\\rangle$. \n      \\item For Problem~\\eqref{eq:minimax}, $\\gap_{\\mathrm{pd}}(\\vz;\\calX\\times \\calY) \\leq \\max_{\\vz \\in \\calX \\times \\calY}\\sum_{t=0}^{T-1} \\theta_t \\langle \\vF(\\vz_t), \\vz_t-\\vz\\rangle$. \n    \\end{enumerate} \n   %\n\\end{lemma}\n\n\\begin{lemma}\\label{lem:one_step}\n  Suppose that the operator $\\vF$ in \\eqref{eq:monotone} is $\\mu$-strongly monotone with $\\mu \\geq 0$.  \n  Let $\\{\\vz_k\\}_{k\\geq 0}$ and $\\{\\hat{\\vz}_k\\}_{k\\geq 0}$ be the iterates generated by Algorithm~\\ref{alg:Full_Equasi-Newton}, where $\\alpha_1\\in [0,1)$, $\\alpha_2 \\in (0,1)$ and $\\alpha_1+\\alpha_2 <1$. Then for any $\\vz \\in \\reals^d$ and $k \\geq 0$, it holds that\n  \\begin{equation}\\label{eq:one_step}\n    \\begin{aligned}\n      \\eta_k \\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz \\rangle &\\leq \\frac{\\|\\vz_k\\!-\\! \\vz\\|^2}{2} - \\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1} - \\vz\\|^2 + \\eta_k\\mu \\|\\hat{\\vz}_k - \\vz\\|^2 \\\\\n      & \\phantom{{}\\leq{}} \n      - \\frac{1-\\alpha_1-\\alpha_2}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2).\n    \\end{aligned}\n  \\end{equation} \n\n\\end{lemma}\n\n\\begin{proof}\n  To simplify the notation, let $\\alpha \\triangleq \\alpha_1+\\alpha_2\\in (0,1)$. For any $\\vz\\in \\reals^d$, we have \n\\begin{equation}\\label{eq:inexact_pp_decomp}\n  \\eta_k \\langle \\vF(\\hat{\\vz}_k), \\hat{\\vz}_k-\\vz \\rangle =  \\langle \\hat{\\vz}_k-\\vz_k+\\eta_k \\vF(\\hat{\\vz}_k), \\hat{\\vz}_k-\\vz \\rangle\n  +\\langle \\vz_k-\\hat{\\vz}_k, \\hat{\\vz}_k-\\vz\\rangle. %\n   %\n   %\n\\end{equation}\nFor the first term in \\eqref{eq:inexact_pp_decomp}, we can bound it by\n\\begin{align}\n  \\langle \\hat{\\vz}_k-\\vz_k+\\eta_k \\vF(\\hat{\\vz}_k), \\hat{\\vz}_k-\\vz \\rangle &\\leq \\|\\hat{\\vz}_k-\\vz_k+\\eta_k\\vF(\\hat{\\vz}_k)\\|\\|\\hat{\\vz}_k-\\vz\\| \\nonumber\\\\\n  &\\leq \\alpha\\sqrt{1+\\eta_k\\mu}\\|\\hat{\\vz}_k-\\vz_k\\|\\|\\hat{\\vz}_k-\\vz\\| \\nonumber\\\\\n  &\\leq {\\frac{\\alpha}{2}\\|\\hat{\\vz}_k-\\vz_k\\|^2+\\frac{\\alpha(1+\\eta_k \\mu)}{2}\\|\\hat{\\vz}_k-\\vz\\|^2}.\\label{eq:triangle_ipp}\n\\end{align}\nHere, the first inequality is due to the Cauchy-Schwarz inequality, the second inequality comes from \\eqref{eq:inexact_proximal_point}, and the last inequality is a result of Young's inequality. \nAdditionally, to handle the second term in \\eqref{eq:inexact_pp_decomp}, we apply the three-point equality, yielding:\n\\begin{equation}\\label{eq:three_point}\n  \\langle \\vz_k-\\hat{\\vz}_k, \\hat{\\vz}_k-\\vz\\rangle = \\frac{1}{2}\\|\\vz_k-\\vz\\|^2-\\frac{1}{2}\\|\\hat{\\vz}_k - \\vz_k\\|^2-\\frac{1}{2}\\|\\hat{\\vz}_k-\\vz\\|^2.\n\\end{equation}\n%\nBy combining \\eqref{eq:inexact_pp_decomp}, \\eqref{eq:triangle_ipp} and \\eqref{eq:three_point}, we obtain that    \n\\begin{equation}\\label{eq:intermediate_1}\n  \\eta_k \\langle \\vF(\\hat{\\vz}_k), \\hat{\\vz}_k-\\vz \\rangle \\leq \\frac{\\|\\vz_k-\\vz\\|^2 }{2}-\\frac{1-\\alpha}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz\\|^2) + \\frac{\\alpha\\eta_k \\mu}{2}\\|\\hat{\\vz}_k-\\vz\\|^2.\n\\end{equation}\nFurthermore, it follows from the update rule in \\eqref{eq:extragradient_mixing} that $\\eta_k\\vF(\\hat{\\vz}_k) = \\vz_k-\\vz_{k+1}+2\\eta_k\\mu (\\hat{\\vz}_k-\\vz_{k+1})$. Thus, for any $\\vz\\in \\reals^d$, it also holds that\n%\n\\begin{equation}\n  \\begin{aligned}\\label{eq:intermediate_2}\n    \\phantom{{}={}}\\eta_k \\langle \\vF(\\hat{\\vz}_k), \\vz_{k+1}-\\vz \\rangle &= \\langle \\vz_k-\\vz_{k+1}, \\vz_{k+1}-\\vz \\rangle+2\\eta_k\\mu \\langle \\hat{\\vz}_k-\\vz_{k+1}, \\vz_{k+1}-\\vz\\rangle %\n    \\\\\n    & = \\frac{\\|\\vz_{k}-\\vz\\|^2}{2}-\\frac{\\|\\vz_k-\\vz_{k+1}\\|^2}{2}-\\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1}-\\vz\\|^2\\\\\n    & \\phantom{{}={}}+ {\\eta_k\\mu}\\|\\hat{\\vz}_{k}-\\vz\\|^2-{\\eta_k\\mu}\\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2, \n  \\end{aligned}\n\\end{equation}\nwhere we applied the three-point equality twice in the last equality. Hence, by adding the inequality in \\eqref{eq:intermediate_1} with $\\vz = \\vz_{k+1}$ to the inequality in \\eqref{eq:intermediate_2}, we obtain \n\\begin{equation}\\label{eq:linearized_loss}\n  \\begin{aligned}\n    \\eta_k \\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz \\rangle \n    &= \\eta_k \\langle \\vF(\\hat{\\vz}_k),{\\vz}_{k+1} - \\vz \\rangle \n    +\\eta_k \\langle \\vF(\\hat{\\vz}_k), \\hat{\\vz}_{k} - \\vz_{k+1} \\rangle \\\\\n    &\\leq \\frac{\\|\\vz_{k}-\\vz\\|^2}{2}-\\bcancel{\\frac{\\|\\vz_k-\\vz_{k+1}\\|^2}{2}}-\\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1}-\\vz\\|^2 \\\\\n    &\\phantom{{}={}} + {\\eta_k\\mu}\\|\\hat{\\vz}_{k}-\\vz\\|^2-{{\\eta_k\\mu}\\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2} + \\bcancel{\\frac{\\|\\vz_k-\\vz_{k+1}\\|^2}{2}}\\\\\n    &\\phantom{{}={}} -\\frac{1\\!-\\!\\alpha}{2}(\\|\\hat{\\vz}_k \\!-\\! \\vz_k\\|^2 \\!+\\! \\|\\hat{\\vz}_k\\!-\\!\\vz_{k+1}\\|^2 )+{\\frac{\\alpha \\eta_k\\mu}{2}\\|\\hat{\\vz}_k\\!-\\!\\vz_{k+1}\\|^2} \\!. %\n    %\n  \\end{aligned}\n\\end{equation} \nMoreover, since $\\alpha <1$, we further have $-{{\\eta_k\\mu}\\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2} + \\frac{\\alpha \\eta_k \\mu}{2}\\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2 \\leq -\\frac{\\eta_k\\mu}{2}\\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2 \\leq 0$.  Combining this with \\eqref{eq:linearized_loss}  and rearranging the terms, we arrive at the desired result in~\\eqref{eq:one_step}. \n\\end{proof}\n\n\\begin{proposition}\\label{prop:HPE}\n  Let $\\{\\vz_k\\}_{k\\geq 0}$ and $\\{\\hat{\\vz}_k\\}_{k\\geq 0}$ be the iterates generated by Algorithm~\\ref{alg:Full_Equasi-Newton}, where $\\alpha_1\\in [0,1)$, $\\alpha_2 \\in (0,1)$ and $\\alpha_1+\\alpha_2 <1$. \n  %\n  \\begin{enumerate}[(a)]\n    \\item Under Assumption~\\ref{assum:strong_monotone}, we have $\\|\\vz_{k+1}-\\vz^*\\|^2 \\leq \\|\\vz_k-\\vz^*\\|^2 (1+2\\eta_k \\mu)^{-1}$ for any $k \\geq 0$. \n    \\item Under Assumption~\\ref{assum:monotone}, we have  $\\|\\vz_{k+1}-\\vz^*\\| \\leq \\|\\vz_k-\\vz^*\\|$ for any $k \\geq 0$. Moreover, define the averaged iterate $\\bar{\\vz}_N$ by $\\bar{\\vz}_N = \\frac{\\sum_{k=0}^{N-1} \\eta_k \\hat{\\vz}_k}{\\sum_{k=0}^{N-1} \\eta_k}$. Then for any compact set $\\calD \\subset \\reals^d$, we have \n    %\n    $\\gap(\\bar{\\vz}_N; \\calD) \\leq \\frac{\\max_{\\vz \\in \\calD}\\,\\|\\vz_0-\\vz\\|^2}{2 \\sum_{k=0}^{N-1} \\eta_k}$.  \n    %\n    %\n    %\n    %\n    %\n    %\n  \\end{enumerate}\n\\end{proposition}\n\n\\begin{proof}\nLet $\\alpha = \\alpha_1 + \\alpha_2$. First, we prove Part (a). Since $\\mF(\\vz^*) = 0$ and $\\vF$ is $\\mu$-strongly monotone by Assumption~\\ref{assum:strong_monotone}, it holds that  \n\\begin{equation}\\label{eq:strong_monotone}\n\\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz^* \\rangle = \\langle \\vF(\\hat{\\vz}_k)-\\vF({\\vz^*}),\\hat{\\vz}_k - \\vz^* \\rangle \\geq \\mu\\|\\hat{\\vz}_k - \\vz^*\\|^2.\n\\end{equation}\nBy setting $\\vz = \\vz^*$ in \\eqref{eq:one_step} and applying \\eqref{eq:strong_monotone}, it yields \n\\begin{equation}\\label{eq:one_step_strongly}\n    0 \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1} - \\vz^*\\|^2  - \\frac{1-\\alpha}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2).\n\\end{equation} \nSince $\\alpha = \\alpha_1 + \\alpha_2 <1$, we can drop the non-positive last term in \\eqref{eq:one_step_strongly} and obtain $0 \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1} - \\vz^*\\|^2$, which is equivalent to $\\|\\vz_{k+1} - \\vz^*\\|^2 \\leq \\|\\vz_k- \\vz^*\\|^2 (1+2\\eta_k\\mu)^{-1}$. This completes the proof of Part (a). \n\nNext, we prove Part (b). Since $\\vF$ is monotone by Assumption~\\ref{assum:monotone}, we have\n  \\begin{equation}\\label{eq:monotone_star}\n    \\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz^* \\rangle = \\langle \\vF(\\hat{\\vz}_k)-\\vF({\\vz^*}),\\hat{\\vz}_k - \\vz^* \\rangle \\geq 0.\n    \\end{equation}\n    By setting $\\vz = \\vz^*$ in \\eqref{eq:one_step} with $\\mu = 0$ and applying \\eqref{eq:monotone_star}, it yields \n    \\begin{equation}\\label{eq:one_step_monotone}\n      0 \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1}{2}\\|\\vz_{k+1} - \\vz^*\\|^2  - \\frac{1-\\alpha}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2).\n    \\end{equation}\n    Again, since the last term in \\eqref{eq:one_step_monotone} is non-positive, this immediately implies $\\frac{1}{2}\\|\\vz_{k+1} - \\vz^*\\|^2 \\leq \\frac{1}{2}\\|\\vz_{k} - \\vz^*\\|^2$, which is equivalent to $\\|\\vz_{k+1} - \\vz^*\\| \\leq \\|\\vz_{k} - \\vz^*\\|$. Additionally, by dropping the non-positive last term in \\eqref{eq:one_step} and noting that $\\mu = 0$, we also have \n    %\n    $\\eta_k \\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz \\rangle \\leq \\frac{1}{2}\\|\\vz_k- \\vz\\|^2 - \\frac{1}{2}\\|\\vz_{k+1} - \\vz\\|^2$.\n    %\n    By summing this inequality from $k=0$ to $k=N-1$, we further get \n    %\n    $\\sum_{k=0}^{N-1} \\eta_k \\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz \\rangle \\leq \\frac{1}{2}\\|\\vz_0- \\vz\\|^2 - \\frac{1}{2}\\|\\vz_{N} - \\vz\\|^2 \\leq \\frac{1}{2}\\|\\vz_0- \\vz\\|^2$. \n    %\n    Hence, it follows from Lemma~\\ref{lem:averaging} that $\\gap(\\bar{\\vz}_N; \\calD) \\leq \\frac{\\max_{\\vz \\in \\calD}\\,\\|\\vz_0-\\vz\\|^2}{2 \\sum_{k=0}^{N-1} \\eta_k}$, where $\\bar{\\vz}_N$ is the averaged iterate given by $\\bar{\\vz}_N = \\frac{\\sum_{k=0}^{N-1} \\eta_k \\hat{\\vz}_k}{\\sum_{k=0}^{N-1} \\eta_k}$.  \n\\end{proof}\n\n\\begin{lemma}\\label{lem:step size_lb}\n  %\n  %\n    %\n   %\n   %\n    Recall that ${\\tilde{\\vz}_k}$ is the auxiliary iterate returned by our line search scheme in Subroutine~\\ref{alg:ls}. For $k\\notin \\mathcal{B}$ we have $\\eta_k = \\sigma_k$, while for $k\\in \\mathcal{B}$ we have \n   %\n    %\n     $ \\eta_k \\!> \\!\n      %\n      {\\frac{{\\alpha_2} \\beta\\|{\\tilde{\\vz}_k}-\\vz_k\\|}{\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\|}}$.\n      %\n      %\n     %\n    %\n    Moreover, if $\\frac{1}{2}(\\mB_k + \\mB_k^\\top) \\succeq \\frac{1}{2}\\mu \\mI$, we have $\\|{\\tilde{\\vz}_k}-\\vz_k\\| \\leq \\frac{1+\\alpha_1}{\\beta(1-\\alpha_1)} \\|{\\hat{\\vz}_k}-\\vz_k\\|$. \n    %\n  %\n\\end{lemma}\n\n\\begin{proof}\n  If $k\\notin \\mathcal{B}$, as per the definition, the line search scheme adopts the initial trial step size at the $k$-th iteration and thus $\\eta_k = \\sigma_k$. On the other hand, if $k \\in \\mathcal{B}$, we go through the backtracking procedure in Subroutine~\\ref{alg:ls}.\n  %\n  Let $\\tilde{\\vz}_k$ denote the last rejected point in the line search scheme, which is calculated from \\eqref{eq:linear_solver_update} using the step size $\\tilde{\\eta}_k = \\eta_k/\\beta$.\n  %\n  This means that the pair $(\\tilde{\\vz}_k,\\tilde{\\eta}_k)$ satisfies \\eqref{eq:x_plus_update} but not  \\eqref{eq:step size_condition}, i.e.,$\\|\\tilde{\\vz}_k-\\vz_k+\\tilde{\\eta}_k(\\vF(\\vz_k)+\\mB_k(\\tilde{\\vz}_k-\\vz))\\| \\leq  {\\alpha_1}\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|\\tilde{\\vz}_k-\\vz_k\\|$ and $\\|\\tilde{\\vz}_k - \\vz_k + \\tilde{\\eta}_k \\vF(\\tilde{\\vz}_k)\\| > (\\alpha_1+\\alpha_2)\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|{\\tilde{\\vz}_{k}}-\\vz_k\\|$. Moreover, note that $\\tilde{\\eta}_k (\\vF(\\tilde{\\vz}_k) - \\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)) = \\left(\\tilde{\\vz}_k - \\vz_k + \\tilde{\\eta}_k \\vF(\\tilde{\\vz}_k)\\right) - ({\\tilde{\\vz}_k}-\\vz_k+\\tilde{\\eta}_k(\\vF(\\vz_k)+\\mB_k({\\tilde{\\vz}_k}-\\vz_k)))$.  \n  %\n  %\n  %\n  %\n  %\n  Thus, it follows from the triangle inequality that $\\tilde{\\eta}_k \\|\\vF(\\tilde{\\vz}_k) - \\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\| > (\\alpha_1+\\alpha_2)\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|{\\tilde{\\vz}_{k}}-\\vz_k\\| - \\alpha_1\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|{\\tilde{\\vz}_{k}}-\\vz_k\\| = \\alpha_2\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|{\\tilde{\\vz}_{k}}-\\vz_k\\|$.\n  %\n  %\n  %\n  %\n  %\n  This further implies that \n  %\n  $\\tilde{\\eta}_k > \\frac{\\alpha_2\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|{\\tilde{\\vz}_{k}}-\\vz_k\\|}{\\|\\vF(\\tilde{\\vz}_k) - \\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\|}$. \n  %\n  %\n  %\n  %\n  %\n    Since $1 + \\tilde{\\eta}_k \\mu \\geq 1$, \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    we obtain the first result in Lemma~\\ref{lem:step size_lb}.\n    To prove the second result in Lemma~\\ref{lem:step size_lb}, \n    recall from~\\eqref{eq:x_plus_update} that $\\hat{\\vz}_k$ and $\\tilde{\\vz}_k$ are inexact solutions of the linear system of equations:  \n    \\begin{equation*}\n      (\\mI+{\\eta_k}\\mB_k)({\\vz}-\\vz_k) = -\\eta_k\\vF(\\vz_k) \\quad \\text{and} \\quad (\\mI+\\tilde{\\eta}_k\\mB_k)({\\vz}-\\vz_k) = -\\tilde{\\eta}_k\\vF(\\vz_k),\n    \\end{equation*}\n    %\n    respectively. \n    Define $\\hat{\\vz}_k^* = \\vz_k - {\\eta}_k(\\mI+ {\\eta}_k\\mB_k)^{-1} \\vF(\\vz_k)$ and $\\tilde{\\vz}_k^* = \\vz_k - \\tilde{\\eta}_k(\\mI+ \\tilde{\\eta}_k\\mB_k)^{-1} \\vF(\\vz_k)$, i.e., the exact solutions of the above linear systems. Since $(\\hat{\\vz}_k,\\eta_k)$ and $(\\tilde{\\vz}_k,\\tilde{\\eta}_k)$ satisfy the condition in \\eqref{eq:x_plus_update}, we have \n    \\begin{align}\\label{eq:inexact_condition_linear}\n      \\|(\\mI+\\eta_k\\mB_k)(\\hat{\\vz}_k-\\hat{\\vz}_k^*)\\| &\\leq \\alpha_1 \\sqrt{1+\\eta_k \\mu} \\|\\hat{\\vz}_k - \\vz_k\\| \\\\\n      \\quad \\text{and} \\quad  \\|(\\mI+\\tilde{\\eta}_k\\mB_k)(\\tilde{\\vz}_k-\\tilde{\\vz}_k^*)\\| &\\leq \\alpha_1 \\sqrt{1+\\tilde{\\eta}_k \\mu}  \\|\\tilde{\\vz}_k - \\vz_k\\|.\n    \\end{align} \n    We divide the proof of the second result {in Lemma~\\ref{lem:step size_lb}} into the following three steps.\n    %\n      First, we  show that \n      %\n      %\n        \\begin{align}\n                    (1-\\alpha_1)\\|\\hat{\\vz}_k - \\vz_k\\| &\\leq \\|\\hat{\\vz}_k^*-{\\vz}_k\\| \\leq (1+\\alpha_1)\\|\\hat{\\vz}_k - \\vz_k\\|, \\label{eq:relation_exact_inexact1}\\\\ (1-\\alpha_1)\\|\\tilde{\\vz}_k - \\vz_k\\| &\\leq \\|\\tilde{\\vz}_k^*-{\\vz}_k\\| \\leq (1+\\alpha_1)\\|\\tilde{\\vz}_k - \\vz_k\\|.\\label{eq:relation_exact_inexact2}\n        \\end{align}\n      %\n      %\n      In the following, we will only prove \\eqref{eq:relation_exact_inexact1}, since the proof of \\eqref{eq:relation_exact_inexact2} follows similarly.\n      Since we assume $\\frac{1}{2}(\\mB_k + \\mB_k^\\top) \\succeq \\frac{\\mu}{2} \\mI$, we have $\\vv^\\top \\mB_k \\vv \\geq \\frac{\\mu}{2} \\|\\vv\\|^2$ for any $\\vv \\in \\reals^d$. Therefore, by using Cauchy-Schwarz inequality, we get   \n      %\n      $\\|\\hat{\\vz}_k-\\hat{\\vz}_k^*\\| \\|(\\mI+\\eta_k\\mB_k)(\\hat{\\vz}_k-\\hat{\\vz}_k^*)\\| \\geq (\\hat{\\vz}_k-\\hat{\\vz}_k^*)^\\top (\\mI+\\eta_k\\mB_k)(\\hat{\\vz}_k-\\hat{\\vz}_k^*) \\geq (1+ \\frac{\\eta_k \\mu}{2})\\|\\hat{\\vz}_k-\\hat{\\vz}_k^*\\|^2$,\n      %\n      which further implies that $\\|(\\mI+\\eta_k\\mB_k)(\\hat{\\vz}_k-\\hat{\\vz}_k^*)\\| \\geq (1+ \\frac{1}{2}\\eta_k \\mu)\\|\\hat{\\vz}_k-\\hat{\\vz}_k^*\\|$. \n      Moreover, since $ \\sqrt{1+\\eta_k\\mu} \\leq 1+ \\frac{1}{2} \\eta_k\\mu $, by~\\eqref{eq:inexact_condition_linear} we also have $\\|(\\mI+\\eta_k\\mB_k)(\\hat{\\vz}_k-\\hat{\\vz}_k^*)\\| \\leq \\alpha_1 {(1+ \\frac{1}{2}\\eta_k \\mu)} \\|\\hat{\\vz}_k - \\vz_k\\|$. \n      %\n      %\n      %\n      %\n      Hence, combining these two inequalities, we get $\\|\\hat{\\vz}_k-\\hat{\\vz}_k^*\\| \\leq \\alpha_1 \\|\\hat{\\vz}_k-\\vz_k\\|$. It then follows from the triangle inequality that\n      \\begin{align*}\n        \\|\\hat{\\vz}_k^*-\\vz_k\\| \\leq \\|\\hat{\\vz}_k - \\vz_k\\| + \\|\\hat{\\vz}_k^*-\\hat{\\vz}_k\\| \\leq (1+\\alpha_1)\\|\\hat{\\vz}_k - \\vz_k\\|, \\\\\n        \\|\\hat{\\vz}_k^*-\\vz_k\\| \\geq \\|\\hat{\\vz}_k - \\vz_k\\| -  \\|\\hat{\\vz}_k^*-\\hat{\\vz}_k\\| \\geq (1-\\alpha_1)\\|\\hat{\\vz}_k - \\vz_k\\|,\n      \\end{align*}\n      which proves \\eqref{eq:relation_exact_inexact1}.\n    %\n  Next, we show that \n     \\begin{equation}\\label{eq:relation_displacement}\n      \\|{\\tilde{\\vz}^*_k}-\\vz_k\\|\\leq \\frac{1}{\\beta}\\|{\\hat{\\vz}^*_k}-\\vz_k\\|.\n     \\end{equation}\n     This follows from \\cite[Lemma 7.8]{monteiro2010complexity}. For completeness, we present its proof below. Note that by definition, we have $(\\mI+{\\eta_k}\\mB_k)(\\hat{\\vz}_k^*-\\vz_k) = -\\eta_k\\vF(\\vz_k)$ and $(\\mI+\\tilde{\\eta}_k\\mB_k)(\\tilde{\\vz}_k^*-\\vz_k) = -\\tilde{\\eta}_k\\vF(\\vz_k)$.\n    %\n    %\n    %\n     Hence, we further have \n     \\begin{equation*}\n      \\mB_k(\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*) = \\frac{1}{\\tilde{\\eta}_k}(\\tilde{\\vz}_k^*-\\vz_k) - \\frac{1}{\\eta_k}(\\hat{\\vz}_k^*-\\vz_k) = \\frac{1}{\\tilde{\\eta}_k} (\\tilde{\\vz}_k^* - \\hat{\\vz}_k^*) + \\left(\\frac{1}{\\tilde{\\eta}_k}-\\frac{1}{\\eta_k}\\right) (\\hat{\\vz}_k^*-\\vz_k).\n     \\end{equation*}\n     Since $(\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*)^\\top \\mB_k(\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*) \\geq 0$, by taking the inner product with $\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*$ on both sides of the above inequality, we obtain\n     %\n    $\\left(\\frac{1}{\\tilde{\\eta}_k}-\\frac{1}{\\eta_k}\\right) (\\hat{\\vz}_k^*-\\vz_k)^\\top (\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*)  \\geq \\frac{1}{\\tilde{\\eta}_k} \\|\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*\\|^2$.\n     %\n     Since ${\\eta}_k = \\beta \\tilde{\\eta}_k$, using the Cauchy-Schwarz inequality, this further leads to \n     %\n    $\\|\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*\\|^2 \\leq \\left(\\frac{1}{\\beta}-1\\right)(\\vz_k - \\hat{\\vz}_k^*)^\\top (\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*) \\leq  \\left(\\frac{1}{\\beta}-1\\right) \\|\\hat{\\vz}_k^* - \\vz_k\\| \\|\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*\\|$.\n     %\n     Hence, we further have $\\|\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*\\| \\leq \\left(\\frac{1}{\\beta}-1\\right) \\|\\hat{\\vz}_k^* - \\vz_k\\|$, which implies that $\\|{\\tilde{\\vz}^*_k}-\\vz_k\\| \\leq \\|{\\tilde{\\vz}^*_k}-\\hat{\\vz}^*_k\\|+\\|\\hat{\\vz}^*_k-\\vz_k\\| \\leq \\frac{1}{\\beta} \\|\\hat{\\vz}^*_k-\\vz_k\\|$. \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    This completes the proof of \\eqref{eq:relation_displacement}. Finally, by combining \\eqref{eq:relation_exact_inexact1}, \\eqref{eq:relation_exact_inexact2}, and \\eqref{eq:relation_displacement}, it follows that $\\|{\\tilde{\\vz}_k}-\\vz_k\\| \\leq \\frac{1}{1-\\alpha_1}\\|\\tilde{\\vz}_k^*-{\\vz}_k\\| \\leq \\frac{1}{(1-\\alpha_1)\\beta}\\|{\\hat{\\vz}^*_k}-\\vz_k\\| \\leq \\frac{1+\\alpha_1}{1-\\alpha_1}\\|\\tilde{\\vz}_k^*-{\\vz}_k\\| \\leq \\frac{1+\\alpha_1}{(1-\\alpha_1)\\beta}\\|{\\hat{\\vz}_k}-\\vz_k\\|$. This proves the second result {in Lemma~\\ref{lem:step size_lb}}. \n  \\end{proof}\n\nTo begin with, \\blue{in the strongly monotone setting,} by repeatedly applying Proposition~\\ref{prop:HPE}(a), we obtain that \n%\n\\begin{equation}\\label{eq:after_jensen}\n   \\frac{\\|\\vz_{N}-\\vz^*\\|^2}{\\|\\vz_0-\\vz^*\\|^2} \\leq \\prod_{k=0}^{N-1} (1+2\\eta_k\\mu)^{-1} \\leq \\biggl(1+{\\frac{2\\mu N}{\\sum_{k=0}^{N-1} 1/\\eta_k}}\\biggr)^{-N}, \n   %\n\\end{equation}\nwhere the last inequality is obtained by applying Jensen's inequality to the function $\\log(1+\\frac{1}{t})$. \n\n\n\\begin{lemma}\\label{lem:stepsize_bnd}\nLet $\\{\\eta_k\\}_{k=0}^{N-1}$ be the step sizes in Algorithm~\\ref{alg:Full_Equasi-Newton} obtained by the line search in Subroutine~\\ref{alg:ls}.~Then,\n%\n\\begin{equation}\\label{eq:goal}\n     \\sum_{k=0}^{N-1}\\frac{1}{\\eta_k} \\leq \\frac{1}{(1-\\beta)\\sigma_0}+ \\frac{1}{(1-\\beta)\\alpha_2\\beta} \n    %\n     \\sqrt{N \\sum_{k\\in \\mathcal{B}} \\frac{\\|\\vu_k-\\mB_k\\vs_k\\|^2}{\\|\\vs_k\\|^2}}, \n     %\n      %\n\\end{equation}\nwhere $\\vu_k \\triangleq \\vF(\\tilde{\\vz}_k) -\\vF({\\vz_k})$, $\\vs_k \\triangleq \\tilde{\\vz}_k-\\vz_k$, and $\\tilde{\\vz}_k$ is the auxiliary iterate in Subroutine~\\ref{alg:ls}. \n\\end{lemma}\n\n\\begin{proof}\n  Let $\\vu_k \\triangleq \\vF(\\tilde{\\vz}_k) -\\vF({\\vz_k})$ and $\\vs_k \\triangleq \\tilde{\\vz}_k-\\vz_k$. \n  In Lemma~\\ref{lem:step size_lb}, we showed that $\\eta_k = \\sigma_k$ if $k\\notin \\mathcal{B}$ and $\\eta_k> \\frac{\\alpha_2 \\beta \\|\\vs_k\\|}{\\|\\vu_k-\\mB_k\\vs_k\\|}$ otherwise. %\n%\n%\nUsing the observations above, we can write \n\\begin{equation}\\label{eq:intermediate_bound}\n  %\n    \\sum_{k=0}^{N-1}\\frac{1}{\\eta_k} \n  = \\sum_{k\\notin \\mathcal{B}}\\frac{1}{\\eta_k} + \\sum_{k\\in \\mathcal{B}}\\frac{1}{\\eta_k} \\leq\n  \\sum_{k\\notin \\mathcal{B}}\\frac{1}{\\sigma_k} + \\sum_{k\\in \\mathcal{B}}\\frac{1}{\\eta_k}\n  \\leq \\frac{1}{\\sigma_0} + \n  \\beta\\sum_{k\\notin \\mathcal{B},k\\geq 1}\\frac{1}{\\eta_{k-1}}+\\sum_{k\\in \\mathcal{B}}\\frac{1}{\\eta_k},\n  %\n\\end{equation}\n%\n%\n%\n%\n%\n%\n%\n%\n%\nwhere we used $\\sigma_k = \\frac{\\eta_{k-1}}{\\beta}$ for $k\\geq 1$ in the last equality. \nSince\n%\n$\\sum_{k\\notin \\mathcal{B},k\\geq 1}\\frac{1}{\\eta_{k-1}} \n%\n\\leq \\sum_{k= 0}^{N-1}\\frac{1}{\\eta_{k}}$,\n%\nrearranging and simplifying the terms in \\eqref{eq:intermediate_bound}, we arrive at \n\\begin{equation}\\label{eq:all_vs_B}\n  \\sum_{k=0}^{N-1}\\frac{1}{\\eta_k} \\leq \\frac{1}{(1-\\beta)\\sigma_0} + \\frac{1}{1-\\beta} \\sum_{k\\in \\mathcal{B}}\\frac{1}{\\eta_k}. \n\\end{equation}\n%\nNow using $\\eta_k > \\frac{\\alpha_2 \\beta \\|\\vs_k\\|}{\\|\\vu_k-\\mB_k\\vs_k\\|}$ for $k \\in \\calB$, we further have \n\\begin{align}\n  \\sum_{k \\in \\calB} \\frac{1}{\\eta_k} \\leq \\sum_{k \\in \\calB} \\frac{\\|\\vu_k-\\mB_k\\vs_k\\|}{\\alpha_2 \\beta \\|\\vs_k\\|} &\\leq \\frac{1}{\\alpha_2\\beta} \\sqrt{|\\calB| \\sum_{k\\in \\mathcal{B}} \\frac{\\|\\vu_k-\\mB_k\\vs_k\\|^2}{\\|\\vs_k\\|^2}} \n  %\n  %\n  \\leq \\frac{1}{\\alpha_2\\beta} \n  \\sqrt{N \\sum_{k\\in \\mathcal{B}} \\frac{\\|\\vu_k-\\mB_k\\vs_k\\|^2}{\\|\\vs_k\\|^2}}, \\label{eq:B_bound1}\n\\end{align}\nwhere the second inequality is due to the generalized mean inequality. \n%\n%\n%\n%\nFinally, the inequality in \\eqref{eq:goal} follows from \\eqref{eq:all_vs_B} and \\eqref{eq:B_bound1}. \n%\nThis completes the proof.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:regret_reduction}\n Suppose the loss function $\\ell_t$ is convex for any $t\\geq 0$ and let $\\{\\vx_t\\}_{t=0}^{T-1}$ and $\\{\\vw_t\\}_{t=0}^{T-1}$ be the iterates generated by Algorithm~\\ref{alg:projection_free_online_learning}. Then we have $\\vw_t \\in \\vspan(\\calC)$ for any $t \\geq 0$. Moreover: \n  \\begin{enumerate}[(a)]\n    \\item In \\textbf{Option I}, we have $\\vx_t \\in (1+\\delta)\\calC$ for $t=0,1,\\dots,T-1$. Also, for any $\\vu\\in \\mathcal{C}$, it holds that \n    \\begin{equation}\n    %\n    \\ell_t(\\vx_t) - \\ell_t(\\vu) %\n    \\leq \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle %\n    \\leq \\frac{\\|\\vw_t-\\vu\\|^2_2}{2\\rho}-\\frac{\\|\\vw_{t+1}-\\vu\\|^2_2}{2\\rho}+ \\frac{\\rho}{2}\\|\\tilde{\\vg}_t\\|_2^2,\\label{eq:regret_reduction} %\n\\end{equation}\n%\n\\begin{equation}\\label{eq:surrogate_loss}\n   \\|\\tilde{\\vg}_t\\| \\leq \\|\\vg_t\\|+|\\langle \\vg_t, \\vx_t\\rangle|\\|\\vs_t\\|.\n\\end{equation}\n\\item In \\textbf{Option II}, we have $\\vx_t \\in \\calC$ for $t=0,1,\\dots,T-1$. Also, for any $\\vu\\in \\mathcal{C}$, it holds that \n  \\begin{align}\n  %\n  \\ell_t(\\vx_t) - \\ell_t(\\vu) \n  %\n  &\\leq \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle - \\delta \\langle \\vg_t, \\vx_t \\rangle \\label{eq:regret_reduction_II_1}\\\\\n  &\\leq \\frac{\\|\\vw_t-\\vu\\|^2_2}{2\\rho}-\\frac{\\|\\vw_{t+1}-\\vu\\|^2_2}{2\\rho}+ \\frac{\\rho}{2}\\|\\tilde{\\vg}_t\\|_2^2 - \\delta \\langle \\vg_t, \\vx_t \\rangle,\\label{eq:regret_reduction_II_2} %\n\\end{align}\n%\n\\begin{equation}\\label{eq:surrogate_loss_II}\n  \\|\\tilde{\\vg}_t\\| \\leq \\|\\vg_t\\|+(1+\\delta)|\\langle \\vg_t, \\vx_t\\rangle|\\|\\vs_t\\|.\n\\end{equation}\n  \\end{enumerate}\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n\\end{lemma}\n\n\\begin{proof}\n  %\n  %\n  %\n  %\n%\n%\nTo begin with, we prove by induction that $\\vw_t \\in \\vspan(\\calC)$. By initialization, we have $\\vw_0 \\in \\vspan(\\calC)$. Now suppose $\\vw_t \\in \\vspan(\\calC)$ for some $t \\geq 0$. Since $\\vg_t =  \\mP\\nabla \\ell_t(\\vx_t) \\in \\vspan(\\calC)$ and $\\vs_t \\in \\vspan(\\calC)$ by Definition~\\ref{def:gauge}, we obtain that $\\tilde{\\vg}_t \\in \\vspan(\\calC)$ in both \\textbf{Options I} and \\textbf{II}. Moreover, since $\\vw_{t+1}$ is expressed as a linear combination of $\\vw_t$ and $\\tilde{\\vg}_t$ according to \\eqref{eq:w_update}, we obtain that $\\vw_{t+1} \\in \\vspan(\\calC)$. Hence, we conclude by induction that $\\vw_t \\in \\vspan(\\calC)$ for any $t\\geq 0$.  \n\nNext, we consider the result in Part (a) for \\textbf{Option I}.  We distinguish two cases depending on the outcome of $\\mathsf{SEP}(\\vw_t;\\delta)$. \n\\begin{itemize}\n  \\item If $\\gamma_t \\leq 1$,  we have $\\vw_t \\in (1+\\delta) \\mathcal{C}$ by Definition~\\ref{def:gauge}. Thus, we have $\\vx_t=\\vw_t\\in (1+\\delta) \\mathcal{C}$ and $\\tilde{\\vg}_t = \\vg_t$, which immediately implies \\eqref{eq:surrogate_loss}. Moreover, since $\\ell_t$ is convex, we have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\nabla \\ell_t(\\vx_t), \\vx_t - \\vu \\rangle$. Note that both $\\vx_t$ and $\\vu$ are in $\\vspan(\\calC)$ and recall that $\\mP$ denotes the orthogonal projection matrix associated with $\\vspan(\\calC)$. Thus, we further have $\\vx_t - \\vu = \\mP(\\vx_t - \\vu)$, which implies that $\\langle \\nabla \\ell_t(\\vx_t), \\vx_t - \\vu \\rangle = \\langle \\nabla \\ell_t(\\vx_t), \\mP(\\vx_t - \\vu )\\rangle = \\langle \\mP\\nabla \\ell_t(\\vx_t), \\vx_t - \\vu \\rangle = \\langle \\vg_t, \\vx_t - \\vu \\rangle$. \n  Therefore, we have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle = \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle$, which proves the first inequality in \\eqref{eq:regret_reduction}. \n  \\item Otherwise, if $\\gamma_t>1$,  we have $\\frac{\\vw_t}{\\gamma_t} \\in (1+\\delta)\\mathcal{C}$ by Definition~\\ref{def:gauge} and $\\langle \\vs_t, \\vw_t-\\vx \\rangle \\geq {\\gamma_t-1}$,  $ \\forall\\vx\\in \\mathcal{C}$. In this case, Algorithm~\\ref{alg:projection_free_online_learning} chooses $\\vx_t  = \\frac{\\vw_t}{\\gamma_t} \\in (1+\\delta)\\mathcal{C}$ and $\\tilde{\\vg}_t = \\vg_t+\\max\\{0, -\\frac{1}{\\gamma_t}\\langle \\vg_t, \\vw_t \\rangle\\} \\vs_t = \\vg_t+\\max\\{0, -\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t$. To prove the first inequality in \\eqref{eq:regret_reduction}, \n  first note that we also have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle$ using similar arguments as above. Moreover, for any $\\vu\\in \\mathcal{C}$,\n  \\begin{align*}\n    \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle &= \\langle\\vg_t+\\max\\{0, -\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t, \\vw_t-\\vu \\rangle \\\\\n    & = \\langle \\vg_t, \\gamma_t\\vx_t-\\vu \\rangle + \\max\\{0, -\\langle \\vg_t, \\vx_t \\rangle\\} \\langle \\vs_t, \\vw_t-\\vu\\rangle \\\\\n    & \\geq \\langle \\vg_t, \\vx_t-\\vu\\rangle + (\\gamma_t-1)\\langle\\vg_t,\\vx_t\\rangle + (\\gamma_t-1)\\max\\{0, -\\langle \\vg_t, \\vx_t \\rangle\\}  \\\\\n    %\n    & \\geq \\langle \\vg_t, \\vx_t-\\vu\\rangle, \n  \\end{align*}\n  where we used $\\vw_t = \\gamma_t \\vx_t$ in the second equality and $\\langle \\vs_t, \\vw_t-\\vu\\rangle \\geq \\gamma_t -1$ in the first inequality. \n  Also, by the triangle inequality we obtain \n  \\begin{equation*}\n    \\|\\tilde{\\vg}_t\\|_2 =\\|\\vg_t+\\max\\{0, -\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t\\|_2 %\n    \\leq  \\|\\vg_t\\|_2+|\\langle \\vg_t, \\vx_t \\rangle|\\|\\vs_t\\|_2,\n  \\end{equation*}\n  which proves \\eqref{eq:surrogate_loss}. \n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n\\end{itemize}\nFinally, from the update rule of $\\vw_{t+1}$ in \\eqref{eq:w_update}, for any $\\vu\\in \\mathcal{C} \\subset \\mathcal{B}_R(0)$, we have \n$\n  \\langle\\vw_t-\\rho\\tilde{\\vg}_t-\\vw_{t+1}, \\vw_{t+1}-\\vu\\rangle \\geq 0\n$. This further implies that \n\\begin{align}\n  \\langle \\tilde{\\vg}_t, \\vw_{t}-\\vu \\rangle &\\leq  \\langle \\tilde{\\vg}_t, \\vw_{t}-\\vw_{t+1}\\rangle+ \\frac{1}{\\rho}\\langle \\vw_t-\\vw_{t+1}, \\vw_{t+1}-\\vu \\rangle \\nonumber\\\\\n  & =  \\langle \\tilde{\\vg}_t, \\vw_{t}-\\vw_{t+1}\\rangle+ \\frac{\\|\\vw_t-\\vu\\|_2^2}{2\\rho}-\\frac{\\|\\vw_{t+1}-\\vu\\|_2^2}{2\\rho}-\\frac{\\|\\vw_t-\\vw_{t+1}\\|_2^2}{2\\rho} \\nonumber\\\\\n  &\\leq \\frac{1}{2\\rho}\\|\\vw_t-\\vu\\|_2^2-\\frac{1}{2\\rho}\\|\\vw_{t+1}-\\vu\\|_2^2 +\\frac{\\rho}{2}\\|\\tilde{\\vg}_t\\|_2^2, \\label{eq:regret_wt}\n\\end{align}\nwhere we used the Young's inequality $\\langle \\tilde{\\vg}_t, \\vw_{t}-\\vw_{t+1}\\rangle \\leq \\frac{\\|\\vw_{t}-\\vw_{t+1}\\|^2}{2\\rho} + \\frac{\\rho}{2}\\|\\tilde{\\vg}_t\\|^2$ in the last inequality. \nThis proves the second inequality in \\eqref{eq:regret_reduction}. \n\n\\blue{Next, we consider Part (b) for \\textbf{Option II}. Similarly, we distinguish two cases depending on the outcome of $\\mathsf{SEP}(\\vw_t;\\delta)$. \n\\begin{itemize}\n  \\item If $\\gamma_t \\leq 1$,  we have $\\vw_t \\in (1+\\delta) \\mathcal{C}$ by Definition~\\ref{def:gauge}. According to \\textbf{Option II} in Algorithm~\\ref{alg:projection_free_online_learning}, we have $\\vx_t= \\frac{\\vw_t}{1+\\delta}\\in \\mathcal{C}$ and $\\tilde{\\vg}_t = \\vg_t$, which immediately implies~\\eqref{eq:surrogate_loss_II}. Using similar arguments as in \\textbf{Option I}, we have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle$. Moreover, since $\\vw_t = (1+\\delta)\\vx_t$, we have $\\langle {\\vg}_t, \\vw_t - \\vu \\rangle = \\langle {\\vg}_t, (1+\\delta)\\vx_t - \\vu \\rangle = \\langle {\\vg}_t, \\vx_t-\\vu \\rangle + \\delta \\langle {\\vg}_t, \\vx_t\\rangle$. Combining these two, we further have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle \\leq\\langle {\\vg}_t, \\vw_t - \\vu \\rangle- \\delta \\langle {\\vg}_t, \\vx_t\\rangle$. This proves the inequality in \\eqref{eq:regret_reduction_II_1}. \n  %\n  \\item Otherwise, if $\\gamma_t>1$, By Definition~\\ref{def:gauge} we have $ \\frac{\\vw_t}{\\gamma_t} \\in (1+\\delta)\\mathcal{C}$ and $\\langle \\vs_t, \\vw_t-\\vx \\rangle \\geq {\\gamma_t-1}$  $ \\forall\\vx\\in \\mathcal{C}$. According to \\textbf{Option II} in Algorithm~\\ref{alg:projection_free_online_learning}, we have $\\vx_t  = \\frac{\\vw_t}{\\gamma_t(1+\\delta)} \\in \\mathcal{C}$ and $\\tilde{\\vg}_t = \\vg_t+\\max\\{0, -\\frac{1}{\\gamma_t}\\langle \\vg_t, \\vw_t \\rangle\\} \\vs_t = \\vg_t+\\max\\{0, -(1+\\delta)\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t$. \n  %\n  Following similar arguments as in \\textbf{Option I}, we have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle$. Moreover, for any $\\vu\\in \\mathcal{C}$, \n  \\begin{align*}\n    \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle &= \\langle\\vg_t+\\max\\{0, -(1+\\delta)\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t, \\vw_t-\\vu \\rangle \\\\\n    & = \\langle \\vg_t, (1+\\delta)\\gamma_t\\vx_t-\\vu \\rangle + \\max\\{0, -(1+\\delta)\\langle \\vg_t, \\vx_t \\rangle\\} \\langle \\vs_t, \\vw_t-\\vu\\rangle \\\\\n    %\n    %\n    & \\geq \\langle \\vg_t, (1+\\delta)\\vx_t-\\vu\\rangle, \n  \\end{align*}\n  where we used $\\langle \\vs_t, \\vw_t-\\vu\\rangle \\geq \\gamma_t -1$ in the first inequality. By rearranging the above inequality, we obtain that $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle \\leq \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle - \\delta \\langle \\vg_t, \\vx_t \\rangle$, which proves~\\eqref{eq:regret_reduction_II_1}.  \n  Also, by the triangle inequality we obtain \n  \\begin{equation*}\n    \\|\\tilde{\\vg}_t\\|_2 =\\|\\vg_t+\\max\\{0, -(1+\\delta)\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t\\|_2 %\n    \\leq  \\|\\vg_t\\|_2+(1+\\delta)|\\langle \\vg_t, \\vx_t \\rangle|\\|\\vs_t\\|_2,\n  \\end{equation*}\n  which proves \\eqref{eq:surrogate_loss_II}. \n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n\\end{itemize}\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\nFinally, the second inequality in \\eqref{eq:regret_reduction_II_2} can be shown similarly as in \\textbf{Option I}. \n }\n\\end{proof}\n\n\\begin{lemma}\\label{lem:sep_intersect}\n  Assume that $\\mathcal{C} = \\mathcal{C}_1 \\cap \\mathcal{C}_2 \\cap \\mathcal{L}$, where $0 \\in \\calC_1 \\cap \\calC_2$ and $\\mathcal{L}= \\vspan(\\calC)$ is a linear subspace. Suppose we have access to $\\mathsf{SEP}_{\\mathcal{C}_1}(\\vw; \\delta)$, $\\mathsf{SEP}_{\\mathcal{C}_2}(\\vw; \\delta)$ and the orthogonal projection matrix $\\mP_{\\calL}$ associated with the subspace $\\calL$. Then given inputs $\\vw \\in \\calL$ and $\\delta>0$, the oracle $\\mathsf{SEP}_{\\mathcal{C}}(\\vw; \\delta)$ can be constructed in the following ways: \n  \\begin{enumerate}[(a)]\n  %\n    \\item  let $(\\gamma_1,\\vs_1) = \\mathsf{SEP}_{\\mathcal{C}_1}(\\vw; \\delta)$ and $(\\gamma_2,\\vs_2) = \\mathsf{SEP}_{\\mathcal{C}_2}(\\vw; \\delta)$.\n    %\n    \\item Let $i = \\argmax_{i \\in \\{1,2\\}} \\gamma_i$ and then output $(\\gamma,\\vs) = (\\gamma_i,\\mP_{\\calL}\\vs_i)$.\n  \\end{enumerate}\n  %\n\\end{lemma}\n\n\\begin{proof}\nWithout loss of generality, we assume $\\gamma_1 \\geq \\gamma_2$ and thus the output is given by $(\\gamma,\\vs) = (\\gamma_1,\\mP_{\\calL}\\vs_1)$. We consider two cases depending on whether $\\gamma_1 \\leq 1$ or not. \n\nIn the first case where $\\gamma = \\gamma_1 \\leq 1$, we have $\\gamma_2 \\leq \\gamma_1 \\leq 1$. According to the definitions of $\\SEP_{\\calC_1}(\\vw;\\delta)$ and $ \\SEP_{\\calC_2}(\\vw;\\delta)$, we have $ \\vw \\in (1+\\delta)\\calC_1$ and $\\vw \\in (1+\\delta)\\calC_2$. This implies that $\\vw \\in (1+\\delta) (\\calC_1 \\cap \\calC_2)$. Moreover, since we assume that $\\vw \\in \\calL$, this leads to $ \\vw \\in (1+\\delta) (\\calC_1 \\cap \\calC_2 \\cap \\calL) = (1+\\delta) \\calC$. \n\nIn the second case where $ \\gamma = \\gamma_1 > 1$, according to the definition of $\\SEP_{\\calC_1}(\\vw;\\delta)$ we have $\\vw/\\gamma_1 \\in (1+\\delta)\\calC_1$ and $ \\langle \\vs_1, \\vw-\\vx \\rangle \\geq \\gamma_1 - 1$ for all $\\vx \\in \\calC_1$. Moreover, by the definition of $\\SEP_{\\calC_2}(\\vw;\\delta)$, we have $\\vw/\\gamma_2 \\in (1+\\delta)\\calC_2$.\nFirst, we show that $\\vw/\\gamma \\in (1+\\delta)\\calC$. \nSince $\\gamma_2 \\leq \\gamma_1$, we can write $ \\vw/\\gamma_1$ as a convex combination of the origin and $\\vw/\\gamma_2$, and thus it follows from the convexity of $\\calC_2$ that $ \\vw/\\gamma_1 \\in (1+\\delta) \\calC_2$. Similar to the arguments in the first case, we obtain $\\vw/\\gamma =  \\vw/\\gamma_1 \\in (1+\\delta) \\calC$. \nNext, we will prove that $\\langle \\vs, \\vw-\\vx \\rangle \\geq \\gamma-1$.  Since $\\calC_1 \\subset \\calC$, we have $\\vx \\in \\calC_1$ for any $\\vx \\in \\calC$. Therefore,  it implies that $\\langle \\vs_1, \\vw - \\vx \\rangle \\geq \\gamma_1-1 = \\gamma-1$. Moreover, since $\\vw \\in \\calL$ and $\\vx \\in \\calL$, we have $\\mP_{\\calL}(\\vw-\\vx) = \\vw-\\vx$. Therefore, we conclude that $\\langle \\vs, \\vw-\\vx \\rangle = \\langle \\mP_{\\calL}\\vs_1, \\vw-\\vx \\rangle = \\langle \\vs_1, \\mP_{\\calL}(\\vw-\\vx) \\rangle = \\langle \\vs_1, \\vw-\\vx \\rangle \\geq \\gamma_1 -1 = \\gamma-1$. \n\nCombining both cases, we observe that the procedure described in Lemma~\\ref{lem:sep_intersect} indeed satisfies the definition in Definition~\\ref{def:gauge}. This completes the proof. \n\\end{proof}\n\n\\begin{lemma}\\label{lem:transform}\n   Recall the set $\\calZ$ defined in \\eqref{eq:feasible_set} and suppose that the linear subspace $\\calL$ satisfies $\\mI \\in \\calL$.  Define the set $\\calC$ as the following: \n   \\begin{equation}\\label{eq:translated_set}\n  {\\mathcal{C}} \\mydef \\left\\{\\hat{\\mB} \\in \\reals^{d\\times d}: -\\mI \\preceq \\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top ) \\preceq \\mI , \\ \\|\\hat{\\mB}\\|_{\\op} \\leq 3, \\ \\hat{\\mB} \\in \\calL \\right\\}. \n  \\end{equation}\n    Let $\\hat{\\mB} \\mydef \\frac{1}{L_1}(\\mB-(L_1+\\mu)\\mI)$. If $\\mB \\in \\calZ$, then $\\hat{\\mB} \\in \\mathcal{C}$. Conversely, if $\\hat{\\mB} \\in (1+\\delta)\\mathcal{C}$, then  $\\frac{1}{2}(\\mB + \\mB^\\top )\\succeq (\\mu -L_1\\delta) \\mI$ and $\\|\\mB\\|_{\\op} \\leq 4L_1 + \\mu+ 3\\delta L_1$. \n\\end{lemma}\n\n\\begin{proof}\n  First, we show that $\\mB\\in \\calZ$ implies $\\hat{\\mB} \\in \\calC$. By the definition of $\\hat{\\mB}$, we have  \n%\n$\\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top ) = \\frac{1}{2L_1}(\\mB + \\mB^\\top) - \\frac{L_1+\\mu}{L_1}\\mI \\succeq \\frac{\\mu}{L_1} \\mI - \\frac{L_1+\\mu}{L_1}\\mI = -\\mI$. \n%\nMoreover, we also have \n%\n$\\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top ) \\preceq  \\frac{1}{2L_1}(\\mB + \\mB^\\top) \\preceq \\mI$.\n%\nCombining these two, we obtain that $ -\\mI \\preceq \\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top ) \\preceq \\mI$. Additionally, we can bound $\\|\\hat{\\mB}\\|_{\\op} \\leq \\frac{1}{L_1} (\\|\\mB\\|_{\\op} + (L_1+\\mu)) \\leq 3$ since $\\|\\mB\\|_{\\op} \\leq L_1$ and $\\mu \\leq L_1$. Finally, note that $\\hat{\\mB}$ is a linear combination of $\\mB$ and $\\mI$. Since $\\mI \\in \\calL$ and $\\mB \\in \\calL$, we also have $\\hat{\\mB} \\in \\calL$. Hence, we conclude that $\\hat{\\mB} \\in \\calC$. \n\nFor the other direction, assume $\\hat{\\mB} \\in (1+\\delta)\\calC$ and note that $ \\mB = L_1 \\hat{\\mB} + (L_1+\\mu)\\mI$. Since $\\hat{\\mB} \\in (1+\\delta)\\calC$ implies that $\\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top) \\succeq -(1+\\delta)\\mI$, we have \n%\n$\\frac{1}{2}(\\mB + \\mB^\\top) \\succeq \\frac{L_1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top) + (L_1 + \\mu) \\mI \\succeq (\\mu-L_1 \\delta)\\mI$.\n%\nFinally, $\\hat{\\mB} \\in (1+\\delta)\\calC$ also implies that $\\|\\hat{\\mB}\\|_{\\op} \\leq 3(1+\\delta)$, and thus it holds that $\\|\\mB\\|_{\\op}  \\leq L_1\\|\\hat{\\mB}\\|_{\\op} + (L_1+\\mu) \\leq 4L_1 + \\mu+ 3\\delta L_1$. This completes the proof.  \n\\end{proof}\n\nThe oracles in Definitions~\\ref{def:extevec} and~\\ref{def:MaxSvec} provide the approximate separation oracles for $\\calC_1$ and $\\calC_2$, respectively. Using these two building blocks, we can construct the approximate separation oracle for $\\calC$ by following the procedure in Lemma~\\ref{lem:sep_intersect}. \nBy instantiating Algorithm~\\ref{alg:projection_free_online_learning}, we obtain the complete Jacobian approximation update given in Subroutine~\\ref{alg:jacobian_approx}. \n\n\n\\begin{proposition}[{\\cite[Theorem~4.2]{kuczynski1992estimating}}]\\label{prop:lanczos}\n  Consider a symmetric matrix $\\mW$ and let $\\lambda_1(\\mW)$ and $\\lambda_d(\\mW)$ denote its largest and smallest eigenvalues, respectively. Then after $k$ iterations of the Lanczos method with a random start, we find unit vectors $\\vu^{(1)}$ and $\\vu^{(d)}$  such that \n  \\begin{align*}\n    \\mathbb{P}(\\langle\\mW \\vu^{(1)}, \\vu^{(1)}\\rangle\\leq \\lambda_1(\\mW)-\\epsilon(\\lambda_1(\\mW)-\\lambda_d(\\mW))) \\leq 1.648\\sqrt{d}e^{-\\sqrt{\\epsilon}(2k-1)},\\\\\n    \\mathbb{P}(\\langle\\mW \\vu^{(d)}, \\vu^{(d)}\\rangle\\geq \\lambda_d(\\mW)+\\epsilon(\\lambda_1(\\mW)-\\lambda_d(\\mW))) \\leq 1.648\\sqrt{d}e^{-\\sqrt{\\epsilon}(2k-1)},\n  \\end{align*}\n  As a corollary, to ensure that, with probability at least $1-q$, $ \\langle\\mW \\vu^{(1)}, \\vu^{(1)}\\rangle> \\lambda_1(\\mW)-\\epsilon(\\lambda_1(\\mW)-\\lambda_d(\\mW))$ and $\\langle\\mW \\vu^{(d)}, \\vu^{(d)}\\rangle< \\lambda_n(\\mW)+\\epsilon(\\lambda_1(\\mW)-\\lambda_d(\\mW))$,\n  %\n  %\n  %\n  %\n  %\n  %\n  the number of iterations can be bounded by  $\\lceil\\frac{1}{4}\\epsilon^{-1/2}\\log(\\frac{11d}{q^2})+\\frac{1}{2} \\rceil$.\n\\end{proposition}\n\n\\begin{lemma}\\label{lem:extevec_bound}\n  Let $\\gamma$ and $\\mS$ be the output of $\\mathsf{ExtEvec}(\\mW;\\delta,q)$ in Subroutine~\\ref{alg:lanczos}. Then with probability at least $1-q$, they satisfy one of the properties given in Definition~\\ref{def:extevec}.  \n  %\n  %\n  %\n  %\n\\end{lemma}\n\n\\begin{proof}\n  Note that in Subroutine~\\ref{alg:lanczos}, we run the Lanczos method for $\\Bigl\\lceil \\frac{1}{4}\\epsilon^{-1/2}\\log\\frac{11d}{q^2}+\\frac{1}{2}\\Bigr\\rceil$ iterations, where $\\epsilon = \\frac{\\delta}{2(1+\\delta)}$. \n  Thus, by Proposition~\\ref{prop:lanczos}, with probability at least $1-q$ we have \n  \\begin{align}\n    \\hat{\\lambda}_1 \\triangleq \\langle\\bar{\\mW} \\vu^{(1)}, \\vu^{(1)}\\rangle \\geq  \\lambda_1(\\bar{\\mW})-\\frac{\\delta}{2(1+\\delta)}(\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})), \\label{eq:lambda_1}\\\\ \n    %\n    \\hat{\\lambda}_d \\triangleq \\langle\\bar{\\mW} \\vu^{(d)}, \\vu^{(d)}\\rangle \\leq  \\lambda_d(\\bar{\\mW})+\\frac{\\delta}{2(1+\\delta)}(\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})). \\label{eq:lambda_d}\n  \\end{align}\n  %\n  %\n  Combining \\eqref{eq:lambda_1} and \\eqref{eq:lambda_d}, we obtain $\\frac{\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})}{1+\\delta}\\leq \\hat{\\lambda}_1-\\hat{\\lambda}_d$, which further implies that $\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})\\leq (1+\\delta)(\\hat{\\lambda}_1-\\hat{\\lambda}_d)$. \n    %\n    %\n    %\n  By plugging the above inequality back into \\eqref{eq:lambda_1} and \\eqref{eq:lambda_d}, we further have \n  \\begin{align}\n    \\lambda_1(\\bar{\\mW}) \\leq \\hat{\\lambda}_1+\\frac{\\delta}{2(1+\\delta)}(\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})) \\leq \\hat{\\lambda}_1+\\frac{\\delta}{2}(\\hat{\\lambda}_1-\\hat{\\lambda}_d), \\label{eq:upper_bound_lambda_1} \\\\\n    \\lambda_d(\\bar{\\mW}) \\geq \\hat{\\lambda}_d-\\frac{\\delta}{2(1+\\delta)}(\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})) \\geq \\hat{\\lambda}_d-\\frac{\\delta}{2}(\\hat{\\lambda}_1-\\hat{\\lambda}_d). \\label{eq:lower_bound_lambda_d}\n  \\end{align}\n  Recall that $\\gamma = \\max\\{\\hat{\\lambda}_1,-\\hat{\\lambda}_d\\}$. By \\eqref{eq:upper_bound_lambda_1} and \\eqref{eq:lower_bound_lambda_d}, \n  we can further bound the eigenvalues of $\\bar{\\mW}$ by  \n  \\begin{equation}\n    \\lambda_1(\\bar{\\mW}) \\leq \\gamma +\\frac{\\delta}{2}\\cdot 2\\gamma ={(1+\\delta)\\gamma} \\quad \\text{and} \\quad \\lambda_d(\\bar{\\mW}) \\geq -\\gamma -\\frac{\\delta}{2}\\cdot 2\\gamma = -(1+\\delta){\\gamma}.\n  \\end{equation}\n  Hence, we can see that $-(1+\\delta)\\gamma \\mI\\preceq \\bar{\\mW} = \\frac{1}{2}(\\mW + \\mW^\\top) \\preceq (1+\\delta)\\gamma \\mI$. \n  %\n  Now we distinguish three cases. \n  \\begin{enumerate}[(a)]\n    \\item If $\\gamma \\leq 1$, then we are in \\textbf{Case I} and the $\\mathsf{ExtEvec}$ oracle outputs $\\gamma$ and $\\mS = 0$. In this case, we indeed have $-(1+\\delta) \\mI\\preceq \\frac{1}{2}(\\mW + \\mW^\\top) \\preceq (1+\\delta) \\mI$. \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    \\item If $\\gamma >1$ and $\\hat{\\lambda}_1 \\geq -\\hat{\\lambda}_d$, then we are in \\textbf{Case II} and the $\\mathsf{ExtEvec}$ oracle returns $\\gamma $ and $\\mS = \\vu^{(1)}(\\vu^{(1)})^\\top$. In this case, since $\\max\\{\\lambda_1(\\bar{\\mW}),-\\lambda_d(\\bar{\\mW})\\} \\leq \\gamma(1+\\delta)$, we have $-(1+\\delta) \\mI\\preceq \\frac{1}{2\\gamma}(\\mW + \\mW^\\top) \\preceq (1+\\delta) \\mI$. Also, since $\\vu^{(1)}$ is a unit vector, we have $\\|\\mS\\|_F = \\|\\vu^{(1)}\\|^2 = 1$. Finally, for any $\\hat{\\mB}$ such that $-\\mI \\preceq \\frac{1}{2}(\\hat{\\mB}+\\hat{\\mB}^\\top) \\preceq \\mI$, \n    %\n    we have \n    %\n    $\\langle \\mS,\\mW-\\hat{\\mB}\\rangle = \\vu_1^\\top{\\mW} \\vu_1-\\vu_1^\\top \\hat{\\mB} \\vu_1 = \\vu_1^\\top{\\bar{\\mW}} \\vu_1 -\\frac{1}{2}\\vu_1^\\top (\\hat{\\mB}+\\hat{\\mB}^\\top) \\vu_1 \\geq \\hat{\\lambda}_1-1 = \\gamma-1$. Thus, $\\gamma$ and $\\mS$ satisfy all the properties in \\textbf{Case II}. \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    \\item If $\\gamma >1$ and $-\\hat{\\lambda}_d \\geq \\hat{\\lambda}_1$, then we are also in \\textbf{Case II} and the $\\mathsf{ExtEvec}$ oracle returns $\\gamma$ and $\\mS = -\\vu^{(d)}(\\vu^{(d)})^\\top$. The rest follows similarly to the case above.\n    %\n    %\n    %\n    %\n    %\n  \\end{enumerate}\n  This completes the proof.\n\\end{proof}\n\n\\begin{lemma}[{\\cite[Section 8.6.1]{Golub2013Matrix}}]\\label{lem:singular_to_eigen}\n  Let $\\mW\\in \\reals^{d\\times d}$ be an arbitrary matrix. Define the augmented matrix $\\tilde{\\mW}$ by \n  \\begin{equation}\\label{eq:augmented}\n    \\tilde{\\mW} = \\begin{bmatrix} 0 & \\mW \\\\ \\mW^\\top & 0\n    \\end{bmatrix} \\in \\mathbb{S}^{2d}. \n  \\end{equation}\n  %\n  \\begin{itemize}\n    \\item Let $\\lambda_1(\\tilde{\\mW})$ and $\\lambda_{2d}(\\tilde{\\mW})$ denote its largest and smallest eigenvalues, respectively. Then $\\lambda_1(\\tilde{\\mW})$ is the maximal singular value of $\\mW$ and $\\lambda_{2d}(\\tilde{\\mW}) = -\\lambda_1(\\tilde{\\mW})$. \n    \\item Let $\\tilde{\\vv} \\in \\reals^{2d}$ denote the eigenvector associated with $\\lambda_1(\\tilde{\\mW})$ and partition it as $\\tilde{\\vv} = (\\vv,\\vv')$, where $\\vv,\\vv' \\in \\reals^d$. Then ${\\vv}$ and ${\\vv'}$ are the left and right singular vectors associated with the maximal singular value of $\\mW$, respectively.\n  \\end{itemize}\n  \n  %\n\\end{lemma}\n\n\\begin{lemma}\\label{lem:maxsvec_bound}\n  Let $\\gamma$ and $\\mS$ be the output of $\\mathsf{MaxSvec}(\\mW;\\delta,q)$ in Subroutine~\\ref{alg:lanczos_singular}.  %\n  %\n  Then with probability at least $1-q$, they satisfy the properties in Definition~\\ref{def:MaxSvec}.  \n  %\n  %\n  %\n  %\n\\end{lemma}\n\n\\begin{proof}\n  %\n  %\n  %\n  %\n  %\n  Note that in Subroutine~\\ref{alg:lanczos_singular}, we run the Lanczos method for $\\Bigl\\lceil \\frac{1}{4}\\epsilon^{-1/2}\\log\\frac{22d}{q^2}+\\frac{1}{2}\\Bigr\\rceil$ iterations, where $\\epsilon = \\frac{\\delta}{2(1+\\delta)}$. \n  Thus, by Proposition~\\ref{prop:lanczos}, with probability at least $1-q$ we can find a unit vector $\\tilde{\\vv} \\in \\reals^{2d}$ such that \n  %\n  $\\tilde{\\lambda}_1 \\triangleq \\langle \\tilde{\\mW} \\tilde{\\vv}, \\tilde{\\vv}\\rangle \\geq  \\lambda_1(\\tilde{\\mW})-\\frac{\\delta}{2(1+\\delta)}(\\lambda_1(\\tilde{\\mW})-\\lambda_{2d}(\\tilde{\\mW}))$, %\n    %\n    %\n  %\n  %\n  %\n  Moreover, notice that $\\lambda_{2d}(\\tilde{\\mW}) = -\\lambda_1(\\tilde{\\mW})$ by Lemma~\\ref{lem:singular_to_eigen}. Hence, the above further implies that $\\tilde{\\lambda}_1 \\geq \\frac{1}{1+\\delta} \\lambda_1(\\tilde{\\mW})$, which is equivalent to $\\lambda_1(\\tilde{\\mW}) \\leq (1+\\delta)\\tilde{\\lambda}$. \n  %\n  %\n  %\n  Moreover, note that $\\lambda_1(\\tilde{\\mW}) = \\|\\mW\\|_{\\op}$ and $\\gamma = \\tilde{\\lambda}_1/3$. Hence, the above is further equivalent to $\\|\\mW\\|_{\\op} \\leq 3(1+\\delta)\\gamma$. \n  %\n  %\n  %\n  %\n  %\n  %\n  Now we distinguish two cases. \n  \\begin{enumerate}[(a)]\n    \\item If $\\gamma \\leq 1$, then we are in \\textbf{Case I} and the $\\mathsf{MaxSvec}$ oracle outputs $\\gamma$ and $\\mS = 0$. In this case, we indeed have $\\|\\mW\\|_\\op \\leq  3(1+\\delta)\\gamma \\leq 3(1+\\delta)$. \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    \\item If $\\gamma >1$, then we are in \\textbf{Case II} and the $\\mathsf{MaxSvec}$ oracle returns $\\gamma $ and $\\mS = \\frac{2}{3}\\vv'\\vv^\\top$. \n    In this case, since $\\|\\mW\\|_\\op \\leq 3(1+\\delta)\\gamma$, we have $\\|\\mW/\\gamma\\|_{\\op} \\leq 3(1+\\delta)$. Also, since $\\tilde{\\vv}$ is a unit vector, we have $\\|\\mS\\|_F = \\frac{2}{3} \\|\\vv\\|\\|\\vv'\\| \\leq \\frac{1}{3}(\\|\\vv\\|^2 + \\|\\vv'\\|^2) = \\frac{1}{3}\\|\\tilde{\\vv}\\|^2 =\\frac{1}{3}$. Finally, it holds that $\\langle \\mS, \\mW \\rangle = \\frac{2}{3} \\vv^\\top \\mW \\vv' = \\frac{1}{3}\\tilde{\\vv}^\\top \\tilde{\\mW} \\tilde{\\vv} = \\frac{\\tilde{\\lambda}_1}{3} = \\gamma$. Moreover, for any $\\hat{\\mB}$ such that $\\|\\hat{\\mB}\\|_\\op \\leq 3$, we have\n    %\n    $\\langle \\mS,\\hat{\\mB}\\rangle =  \\frac{2}{3}\\vv^\\top \\hat{\\mB} \\vv' \\leq \\frac{2}{3}\\|\\vv\\| \\|\\mB\\|_{\\op} \\|\\vv'\\| \\leq 2\\|\\vv'\\| \\|\\vv\\| \\leq \\|\\vv\\|^2 + \\|\\vv'\\|^2 = 1$. \n    %\n    Hence, we obtain $\\langle \\mS,\\mW-\\hat{\\mB}\\rangle \\geq \\gamma -1$. This shows that $\\gamma$ and $\\mS$ satisfy all the properties in \\textbf{Case II}. \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n  \\end{enumerate}\n  This completes the proof.\n\\end{proof}\n\nNow we move to the proof of Theorem~\\ref{thm:main}. First of all, by using a simple union bound, we can show that \n the  $\\mathsf{ExtEvec}$ and $\\mathsf{MaxSvec}$ oracles used in Subroutine~\\ref{alg:jacobian_approx} are successful in all rounds with probability at least $1-p$. \n %\n Specifically, \nsince both  oracles have a failure probability of $ \\frac{q_t}{2} = \\frac{p}{5(t+1)\\log^2(t+1)}$ in the $t$-th round, we can use the union bound to upper bound the total failure probability by \n\\begin{equation*}\n  \\sum_{t=1}^{T-1} q_t %\n   \\leq  \\frac{p}{2.5}\\sum_{t=2}^{\\infty} \\frac{1}{t \\log^2 t} \\leq \\frac{p}{2.5}\\left(\\frac{1}{2\\log^2 2}+\\int_{2}^{+\\infty}\\frac{1}{t\\log^2t}\\,dt\\right) \\leq p.\n\\end{equation*}\n Thus, throughout the proof, we assume every call of $\\mathsf{ExtEvec}$ and $\\mathsf{MaxSvec}$ is successful. \n\n\n\\begin{proof}[Proof of Theorem~\\ref{thm:main}(a)]\n  \n\nWe first prove the linear convergence rate in (a). To begin with, note that Subroutine~\\ref{alg:jacobian_approx} with \\textbf{Option I} guarantees that  $\\hat{\\mB}_k \\in (1+\\frac{\\mu}{2L_1}) \\mathcal{C}$ by Lemma~\\ref{lem:regret_reduction}.  As a corollary of Lemma~\\ref{lem:transform}, we have the following result showing the boundedness of the Jacobian approximation matrix $\\mB_k$. \n\n%\n\n \n\\begin{corollary}\\label{lem:feasible_Bk}\n  Let $\\delta_t = \\frac{\\mu}{2L_1}$ for all $t\\geq 0$. Then for any $k \\geq 0$, we have $\\frac{1}{2}(\\mB_k + \\mB_k^\\top )\\succeq \\frac{\\mu}{2} \\mI$ and $\\|\\mB_k\\|_{\\op} \\leq 6.5L_1$. \n\\end{corollary}\nCombining this with Lemma~\\ref{lem:step size_lb}, we obtain the following universal lower bound on the step size $\\eta_k$. \n\\begin{lemma}\\label{lem:stepsize_const_bound}\n  %\n  In Subroutine~\\ref{alg:ls}, choose $\\sigma_0 \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$. For any $k\\geq 0$, we have $\\eta_k \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$. \n\\end{lemma}\n\\begin{proof}\n  Recall that $\\calB = \\{k: \\eta_k < \\sigma_k\\}$.\n  We first establish that $\\eta_k \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$ for $k\\in \\mathcal{B}$. To see this, suppose $k\\in \\mathcal{B}$ and recall from Lemma~\\ref{lem:step size_lb} that\n\\begin{equation}\\label{eq:step size_lb_appen}\n  \\eta_k > \\frac{\\alpha_2 \\beta\\|{\\tilde{\\vz}_k}-\\vz_k\\|}{\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\|}.\n\\end{equation}\n%\n  %\n  %\n  %\n  Moreover, by using the triangle inequality, we have \n  %\n  $\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\| \\leq \\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)\\| + \\|\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\|$.\n  %\n  By using Assumption~\\ref{assum:operator_lips}, we have $\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)\\| \\leq L_1\\|\\tilde{\\vz}_k-\\vz_k\\|$. Moreover, since $\\|\\mB_k\\|_{\\op} \\leq 6.5L_1$ by Corollary~\\ref{lem:feasible_Bk}, we also get $\\|\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\| \\leq 6.5 \\|{\\tilde{\\vz}_k}-\\vz_k\\|$. \n  %\n  Thus, this implies that $\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\| \\leq 7.5\\|{\\tilde{\\vz}_k}-\\vz_k\\|$,  \n  %\n  %\n  %\n  which leads to $\\eta_k > \\frac{\\alpha_2\\beta}{7.5L_1}$ from \\eqref{eq:step size_lb_appen}. \n\n  Now we can prove that $\\eta_k \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$ for all $k\\geq 0$ by induction. To show that this holds for $k=0$, we distinguish two cases. If $0\\notin \\mathcal{B}$, then we have $\\eta_0 = \\sigma_0 > \\frac{\\alpha_2\\beta}{7.5L_1}$ by our choice of $\\sigma_0$. Otherwise, if $0\\in \\mathcal{B}$, then it directly follows from our result in the previous paragraph. Moreover, assume that $\\eta_{l-1} \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$ where $l\\geq 1$. Similarly, we again distinguish two cases: if $l\\notin \\mathcal{B}$, then we have $\\eta_l = \\sigma_l = \\frac{\\eta_{l-1}}{\\beta} >\\frac{\\alpha_2}{7.5L_1}>\\frac{\\alpha_2\\beta}{7.5L_1}$; otherwise, if $l\\in \\mathcal{B}$, it follows from the result above that $\\eta_l \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$. This completes the induction. \n\\end{proof}\nIn light of Lemma~\\ref{lem:stepsize_const_bound}, it follows from Proposition~\\ref{prop:HPE}(a) that  $\\|\\vz_{k+1}-\\vz^*\\|^2 \\leq \\|\\vz_k-\\vz^*\\|^2 (1+2\\eta_k \\mu)^{-1} \\leq \\|\\vz_k-\\vz^*\\|^2 \\left(1+\\frac{4\\alpha_2\\beta\\mu}{15L_1}\\right)^{-1}$. \nThis proves the linear convergence result in (a). \n\\end{proof}\n\nNext, we prove the superlinear convergence rate in (b). Our starting point is the observations in \\eqref{eq:after_jensen} and \\eqref{eq:goal}, which demonstrate that it is sufficient to prove an upper bound on the cumulative loss $\\sum_{k=0}^{N-1} \\ell_k(\\mB_k)$. \n%\n%\nMoreover, since $\\ell_k(\\mB_k) = 0$ when $k \\notin \\calB$ by the definition in \\eqref{eq:loss_of_jacobian}, it is equivalent to bounding the sum $\\sum_{k \\in \\calB} \\ell_k(\\mB_k)$.  \n%\n%\n\\blue{Recall that we relabel the indices in $\\mathcal{B}$ by $t=0,\\dots,T-1$ with $T \\leq N$. %\n%\n} \n\n\n\\begin{proof}[Proof of Theorem~\\ref{thm:main}(b)] We consider the following three steps.\n\n\\vspace{0em}\\noindent\\textbf{Step 1:} Using regret analysis, we bound the cumulative loss $\\sum_{t=0}^{T-1}\\ell_{t}(\\mB_{t})$ incurred by our online learning algorithm in Subroutine~\\ref{alg:jacobian_approx}. \nTo begin with, we present the following lemma showing a self-bounding property of the loss function $\\ell_t$. {It is similar to the standard inequality $\\frac{1}{2L_1} \\|\\nabla g (\\vx)\\|^2 \\leq g(\\vx) - g^* $ for a $L_1$-smooth function $g$, where $g^*$ denotes the minimum of $g$.} This will be the key to proving a constant upper bound on the cumulative loss incurred by Subroutine~\\ref{alg:jacobian_approx}.\n\\begin{lemma}\\label{lem:loss}\n  Recall the loss function $\\ell_k$ defined in \\eqref{eq:loss_of_jacobian}. For $k\\in \\mathcal{B}$, we have\n  \\begin{equation}\\label{eq:gradient}\n    \\nabla \\ell_k(\\mB) = - \\frac{2(\\vu_k-\\mB\\vs_k)\\vs_k^\\mathsf{T}}{\\|\\vs_k\\|^2}.\n  \\end{equation}\n  Moreover, for any $\\mB\\in \\reals^{d\\times d}$, it holds that \n    \\begin{equation}\\label{eq:gradient_norm_bound}\n        \\|\\nabla \\ell_k(\\mB)\\|_F \\leq \\|\\nabla \\ell_k(\\mB)\\|_* = 2\\sqrt{\\ell_k(\\mB)},\n    \\end{equation} \n    where $\\|\\cdot\\|_F$ and $\\|\\cdot\\|_{*}$ denote the Frobenius norm and the nuclear norm, respectively.  \n\\end{lemma}\n\\begin{proof}\n  %\n  The expression in \\eqref{eq:gradient} follows from the direct calculation. The first inequality in \\eqref{eq:gradient_norm_bound} follows from the fact that $\\|\\mA\\|_F \\leq \\|\\mA\\|_*$ for any matrix $\\mA\\in \\reals^{d\\times d}$. For the equality, note that \n  \\begin{align*}\n    \\|\\nabla \\ell_k(\\mB)\\|_*   = \\frac{2}{\\|\\vs_k\\|^2}  \\|\\vu_k-\\mB\\vs_k\\|\\|\\vs_k\\|  = \\frac{2\\|\\vu_k-\\mB\\vs_k\\|}{\\|\\vs_k\\|} = 2\\sqrt{\\ell_k(\\mB)}, \n  \\end{align*}\n  where we used the fact that the rank-one matrix $\\va\\vb^\\top$ has only one nonzero singular value $\\|\\va\\|\\|\\vb\\|$. \n\\end{proof}\n\n\n%\nIn particular, by exploiting the self-bounding property of the loss function $\\ell_t$, we prove a ``small-loss bound'' \\cite{srebro2010smoothness} in the following lemma, where the cumulative loss of the learner is bounded by that of a fixed action in the competitor set. \n%\n  \\begin{lemma}\n  \\label{lem:small_loss}\n  %\n  %\n  In Subroutine~\\ref{alg:jacobian_approx}, choose \\textbf{Option I} with $\\rho = \\frac{1}{121}$ and $\\delta = \\frac{\\mu}{2L_1}$. \n  For any ${\\mH}\\in \\mathcal{{Z}}$, we have\n  %\n    $\\sum_{t=0}^{T-1} \\ell_t({\\mB}_t)  \\leq 121\\|\\mB_0-{\\mH}\\|_F^2+ 2 \\sum_{t=0}^{T-1} \\ell_t({\\mH})$.\n  %\n\\end{lemma}\n\n\\begin{proof}\nRecall that $\\calZ$ is contained in the linear subspace $\\calL$ (see \\eqref{eq:feasible_set}) and $\\mP$ denotes the orthogonal projection matrix associated with $\\calL$.    \nBy letting $\\vx_t = \\hat{\\mB}_t$, $\\vu = \\hat{\\mH} \\triangleq \\frac{1}{L_1}(\\mH-(L_1+\\mu)\\mI)$, \n%\n$\\vg_t = \\mG_t \\triangleq \\frac{1}{L_1}\\mP \\nabla \\ell_t(\\mB_t)$, $\\tilde{\\vg}_t = \\tilde{\\mG}_t$, $\\vw_t = \\mW_t$ in Lemma~\\ref{lem:regret_reduction}, we obtain:  \n\\begin{enumerate}[(i)]\n  \\item $\\hat{\\mB}_t \\in (1+\\delta)\\mathcal{C}$, which means $\\|\\hat{\\mB}_t\\|_{\\op} \\leq 3(1+\\delta)\\leq 4.5$ since $\\delta = \\frac{\\mu}{2L_1}\\leq \\frac{1}{2}$.\n  \\item It holds that \n  \\begin{align}\n    \\langle \\mG_t, \\hat{\\mB}_t-\\hat{\\mH} \\rangle &\\leq \\frac{1}{2\\rho}\\|\\mW_t-\\hat{\\mH}\\|_F^2-\\frac{1}{2\\rho}\\|\\mW_{t+1}-\\hat{\\mH}\\|_F^2+\\frac{\\rho}{2}\\|\\tilde{\\mG}_t\\|_F^2, \\label{eq:linearized_loss_matrix} \\\\%\\\\\n    %\n    \\|\\tilde{\\mG}_t\\|_F &\\leq \\|\\mG_t\\|_F + |\\langle \\mG_t, \\hat{\\mB}_t\\rangle|\\|\\mS_t\\|_F. \\label{eq:surrogate_gradient_bound}\n  \\end{align}\n\\end{enumerate}\n%\n%\n%\n%\n%\n%\n%\n%\n%\nFirst, note that $\\|\\mS_t\\|_F \\leq 1$ by Definition~\\ref{def:extevec} and\n%\n$|\\langle \\mG_t, \\hat{\\mB}_t\\rangle| \\leq \\|\\mG_t\\|_* \\|\\hat{\\mB}_t\\|_{\\op} \\leq 4.5 \\|\\mG_t\\|_*$. %\n%\nTogether with \\eqref{eq:surrogate_gradient_bound}, we get \n\\begin{equation}\\label{eq:surrogate_bound_loss}\n\\|\\tilde{\\mG}_t\\|_F \\leq \\|\\mG_t\\|_F+4.5\\|\\mG_t\\|_* \\leq 5.5\\|\\mG_t\\|_* \\leq \\frac{5.5}{L_1}\\|\\nabla \\ell_t(\\mB_t)\\|_* \\leq \\frac{11}{L_1}\\sqrt{\\ell_t({\\mB}_t)},\n\\end{equation}\nwhere we used $\\mG_t = \\frac{1}{L_1}\\mP \\nabla \\ell_t(\\mB_t)$ and Lemma~\\ref{lem:loss} in the last inequality. \nFurthermore, since $\\ell_t$ is convex and $\\mB_t,\\mH \\in \\calL$, we have \n\\begin{equation}\\label{eq:convexity_subspace}\n    \\ell_t({\\mB}_t) - \\ell_t(\\mH) \\!\\leq\\! \\langle \\nabla \\ell_t(\\mB_t), {\\mB}_t-\\mH \\rangle \\!=\\! \\langle \\nabla \\ell_t(\\mB_t), \\mP\\left({\\mB}_t-\\mH\\right) \\rangle \\!=\\! L_1^2 \\langle \\mG_t, \\hat{\\mB}_t-\\hat{\\mH} \\rangle,\n\\end{equation}\nwhere we used $\\mG_t = \\frac{1}{L_1}\\mP\\nabla \\ell_t(\\mB_t)$, $\\hat{\\mB}_t \\triangleq \\frac{1}{L_1}(\\mB_t-{(L_1+\\mu)}\\mI)$, and $\\hat{\\mH} \\triangleq \\frac{1}{L_1}(\\mH-{(L_1+\\mu)}\\mI)$. \nTherefore, by \\eqref{eq:linearized_loss_matrix} and \\eqref{eq:surrogate_bound_loss} we get \n\\begin{align*}\n\\ell_t({\\mB}_t) - \\ell_t(\\mH) &\\leq \\frac{L_1^2}{2\\rho}\\|\\mW_t-\\hat{\\mH}\\|_F^2-\\frac{L_1^2}{2\\rho}\\|\\mW_{t+1}-\\hat{\\mH}\\|_F^2+\\frac{\\rho L_1^2}{2} \\|\\tilde{\\mG}_t\\|_F^2\\\\\n&\\leq \\frac{L_1^2}{2\\rho}\\|\\mW_t-\\hat{\\mH}\\|_F^2-\\frac{L_1^2}{2\\rho}\\|\\mW_{t+1}-\\hat{\\mH}\\|_F^2+60.5\\rho\\ell_t({\\mB}_t). %\n%\n\\end{align*}\nSince $\\rho = \\frac{1}{121}$, by rearranging and simplifying terms in the above inequality, we obtain\n%\n%\n%\n$\\ell_t({\\mB}_t) \\leq 2\\ell_t({\\mH})+ {121 L_1^2}\\|\\mW_t-\\hat{\\mH}\\|_F^2-{121 L_1^2}\\|\\mW_{t+1}-\\hat{\\mH}\\|_F^2$.\n%\nBy summing the above inequality from $t=0$ to $T-1$, we further have \n\\begin{equation*}\n  \\sum_{t=0}^{T-1} \\ell_t({\\mB}_t)  \\leq {121 L_1^2}\\|\\mW_0-\\hat{\\mH}\\|_F^2+ 2 \\sum_{t=0}^{T-1} \\ell_t({\\mH}) = 121\\|\\mB_0-{\\mH}\\|_F^2+ 2 \\sum_{t=0}^{T-1} \\ell_t({\\mH}),\n\\end{equation*}\nwhere the last equality is due to ${\\mW}_0 \\triangleq \\frac{1}{L_1}(\\mB_0-{(L_1+\\mu)}\\mI)$ and $\\hat{\\mH} \\triangleq \\frac{1}{L_1}(\\mH-{(L_1+\\mu)}\\mI)$. This completes the proof. \n\\end{proof}\n\n%\nNote that in Lemma~\\ref{lem:small_loss}, we have the freedom to choose any competitor $\\mH$ in the set $\\mathcal{Z}$. To further obtain an explicit bound, a natural choice would be $\\mH^* \\triangleq \\nabla \\vF(\\vz^*)$, which leads to our next step.\n\n%\n\n\\vspace{0em}\\noindent \\textbf{Step 2:} We upper bound the cumulative loss $\\sum_{t=0}^{T-1} \\ell_t({\\mH}^*)$. The proof relies crucially on Assumption~\\ref{assum:jacobian_lips} as well as the linear convergence result we proved in (a).\nTo begin with, we prove the following useful lemma. \n\\begin{lemma}\\label{lem:sum_of_squares}\n If Assumption~\\ref{assum:strong_monotone} holds, then $\\sum_{k=0}^{N-1} \\|\\hat{\\vz}_{k}\\!-\\!\\vz_k\\|^2 \\leq \\frac{1}{1-(\\alpha_1+\\alpha_2)}\\|\\vz_0\\!-\\!\\vz^*\\|^2$. \n\\end{lemma}\n\\begin{proof}\n  Recall the inequality~\\eqref{eq:one_step_strongly} in the proof of Proposition~\\ref{prop:HPE}. Since $\\frac{1}{2}\\|\\vz_{k+1}-\\vz^*\\|^2 \\!\\leq \\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1}-\\vz^*\\|^2$, we can further derive that \n  \\begin{equation}\\label{eq:one_step_strongly_coro}\n    \\frac{1-\\alpha_1-\\alpha_2}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2) \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1}{2}\\|\\vz_{k+1} - \\vz^*\\|^2.\n\\end{equation} \nSince $\\alpha_1+\\alpha_2 < 1$, we further have $\\frac{1-\\alpha_1-\\alpha_2}{2} \\|\\hat{\\vz}_k - \\vz_k\\|^2 \\leq \\frac{1-\\alpha_1-\\alpha_2}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2) \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1}{2}\\|\\vz_{k+1} - \\vz^*\\|^2$. By summing the above inequality from $k=0$ to $k=N-1$, we obtain that \n\\begin{equation*}\n  \\frac{1-\\alpha_1-\\alpha_2}{2} \\sum_{k=0}^{N-1} \\|\\hat{\\vz}_k - \\vz_k\\|^2 \\leq \\frac{1}{2}\\|\\vz_0-\\vz^*\\|^2 - \\frac{1}{2}\\|\\vz_N-\\vz^*\\|^2 \\leq \\frac{1}{2}\\|\\vz_0-\\vz^*\\|^2. \n\\end{equation*}\nThus, we arrive at Lemma~\\ref{lem:sum_of_squares} by dividing both sides by $\\frac{1-\\alpha_1-\\alpha_2}{2}$. \n\\end{proof}\n\nWith the help of Lemma~\\ref{lem:sum_of_squares}, we are ready to upper bound $\\sum_{t=0}^{T-1} \\ell_t({\\mH}^*)$. \n  \\begin{lemma}\\label{lem:comparator}\n  Recall the constant $C$ defined in \\eqref{eq:def_c1}. \n  %\n  Then \n    %\n      %\n    %\n    %\n    %\n    %\n    $\\sum_{t=0}^{T-1} \\ell_t(\\mH^*) \\leq \\left(C+2+\\frac{15L_1}{2\\alpha_2 \\beta \\mu}\\right)L_2^2\\|\\vz_0-\\vz^*\\|^2$.\n    %\n  \\end{lemma}\n  \\begin{proof}\n    Recall the definition of $\\ell_t$ in \\eqref{eq:loss_of_jacobian}. \n    By the fundamental theorem of calculus, we can write $\\vu_t = \\vF({\\tilde{\\vz}_t})-\\vF(\\vz_t) = \\bar{\\mH}_t (\\tilde{\\vz}_t-\\vz_t)$, where $\\bar{\\mH}_t = \\int_{0}^1 \\nabla \\vF(\\vz_t+ \\lambda \\vs_t) \\,d\\lambda$. Moreover, using the triangle inequality, we have \n\\begin{align*}\n  \\|\\bar{\\mH}_t-\\mH^*\\|_{\\op} &\\leq \\int_{0}^1 \\|\\nabla \\vF(\\vz_t+ \\lambda \\vs_t)-\\nabla \\vF(\\vz^*)\\|_{\\op}\\, d\\lambda \\\\\n  &\\leq L_2\\int_{0}^1 \\|\\vz_t-\\lambda \\vs_t + \\vz^*\\|\\, d\\lambda \\\\\n  &\\leq L_2\\int_{0}^1 (\\|\\vz_t-\\vz^*\\| + \\lambda \\|\\vs_t\\|)\\, d\\lambda = L_2\\Bigl(\\|\\vz_t-\\vz^*\\|+\\frac{1}{2}\\|\\vs_t\\|\\Bigr),\n\\end{align*}\nwhere we used Assumption~\\ref{assum:jacobian_lips} in the second inequality. \n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\nTherefore, we have $\\|\\vu_t-\\mH^*\\vs_t\\| = \\|(\\bar{\\mH}_t-\\mH^*)\\vs_t\\| \\leq \\|\\bar{\\mH}_t-\\mH^*\\|_{\\op}\\|\\vs_t\\| \\leq L_2\\|\\vs_t\\|\\Bigl(\\|\\vz_t-\\vz^*\\|+\\frac{1}{2}\\|\\vs_t\\|\\Bigr)$. This further implies that \n\\begin{align}\n  \\sum_{t=0}^{T-1} \\ell_t(\\mH^*) = \\sum_{t=0}^{T-1} \\frac{\\|\\vu_t-\\mH^*\\vs_t\\|^2}{\\|\\vs_t\\|^2} &\\leq {L_2^2}\\sum_{t=0}^{T-1}\\Bigl(\\|\\vz_t-\\vz^*\\|+\\frac{1}{2}\\|\\vs_t\\|\\Bigr)^2 \\nonumber\\\\ \n  &\\leq \\frac{L_2^2}{2} \\sum_{t=0}^{T-1} \\|\\vs_t\\|^2 + 2 L_2^2\\sum_{t=0}^{T-1} \\|{\\vz}_t-\\vz^*\\|^2. \\label{eq:comparator_1}\n\\end{align}\n\\blue{Note that by our notation, we have $\\sum_{t=0}^{T-1} \\|\\vs_t\\|^2 = \\sum_{k \\in \\calB} \\|\\vs_k\\|^2 = \\sum_{k \\in \\calB} \\|\\tilde{\\vz}_k - \\vz_k\\|^2 $ and $\\sum_{t=0}^{T-1} \\|{\\vz}_t-\\vz^*\\|^2 = \\sum_{k\\in \\calB} \\|{\\vz}_k-\\vz^*\\|^2$.}\nTo bound the first sum $\\sum_{k \\in \\calB} \\|\\tilde{\\vz}_k - \\vz_k\\|^2$, we use the last result in Lemma~\\ref{lem:step size_lb} and the inequality in Lemma~\\ref{lem:sum_of_squares} to get \n\\begin{equation}\\label{eq:comparator_2}\n  \\begin{aligned}\n    \\sum_{k\\in \\calB} \\|\\tilde{\\vz}_k-\\vz_k\\|^2  \\leq \\frac{(1+\\alpha_1)^2}{\\beta^2 (1-\\alpha_1)^2} \\sum_{k\\in \\calB} \\|\\hat{\\vz}_k-\\vz_k\\|^2 \n    %\n    %\n    &\\leq \\frac{(1+\\alpha_1)^2}{\\beta^2 (1-\\alpha_1)^2} \\sum_{k=0}^{N-1} \\|\\hat{\\vz}_k-\\vz_k\\|^2 \\\\\n      &\\leq \\frac{(1+\\alpha_1)^2\\|\\vz_0-\\vz^*\\|^2}{(1-\\alpha_1)^2\\beta^2 (1-(\\alpha_1+\\alpha_2))} = 2C \\|\\vz_0-\\vz^*\\|^2. \n  \\end{aligned}\n  %\n\\end{equation}\nTo bound the second sum $\\sum_{k\\in \\calB} \\|{\\vz}_k-\\vz^*\\|^2$, we use the linear convergence result in Part (a) of Theorem~\\ref{thm:main}:\n\\begin{equation}\\label{eq:comparator_3}\n \\begin{aligned}\n  \\sum_{k\\in \\calB} \\|{\\vz}_k-\\vz^*\\|^2 \\leq \\sum_{k=0}^{N-1} \\|\\vz_k-\\vz^*\\|^2 &\\leq \\|\\vz_0-\\vz^*\\|^2\\sum_{k=0}^{N-1} \\left(1+\\frac{4\\alpha_2\\beta\\mu}{15L_1}\\right)^{-k}\\\\ &\\leq  \\|\\vz_0-\\vz^*\\|^2\\left(1+\\frac{15L_1}{4\\alpha_2 \\beta \\mu}\\right).\n \\end{aligned}\n\\end{equation}\nLemma~\\ref{lem:comparator} follows immediately from \\eqref{eq:comparator_1}, \\eqref{eq:comparator_2}, and \\eqref{eq:comparator_3}. \n  \\end{proof}\n %\n%\n  %\n\n\\noindent\\textbf{Step 3}: Combining Lemmas~\\ref{lem:comparator} and~\\ref{lem:small_loss},  we obtain a constant upper bound on the cumulative loss as\n%\n$\\sum_{t=0}^{T-1} \\ell_t({\\mB}_t) \\leq 121\\|\\mB_0-{\\mH^*}\\|_F^2+ \\left(C+2+\\frac{15L_1}{2\\alpha_2 \\beta \\mu}\\right)L_2^2\\|\\vz_0-\\vz^*\\|^2 = L_1^2 M$, where we used the definition of $M$ in \\eqref{eq:def_M}. \n%\n%\nTogether with the fact that $\\sigma_0 \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$, it follows from  Lemma~\\ref{lem:stepsize_bnd} that\n\\begin{equation*}\n  \\sum_{i=0}^{k-1}\\frac{1}{\\eta_i} \\leq \\frac{7.5L_1}{(1-\\beta)\\alpha_2\\beta}+ \\frac{L_1\\sqrt{kM}}{(1-\\beta)\\alpha_2\\beta} \\leq \n  \\frac{2L_1 \\max\\{7.5, \\sqrt{kM}\\}}{(1-\\beta)\\alpha_2\\beta}. \n  %\n\\end{equation*}\nThis leads to the superlinear convergence result in (b) by Proposition~\\ref{prop:HPE} and the observation in \\eqref{eq:after_jensen}.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:ls_terminates}\n  Suppose Assumption~\\ref{assum:operator_lips} holds. If ${\\eta}_+ <\\frac{\\alpha_2}{L_1+\\|\\mB\\|_{\\op}}$ and $\\hat{\\vz}_+$ is computed according to \\eqref{eq:linear_solver_update}, then the pair $({\\eta}_+,\\hat{\\vz}_+)$ satisfies the conditions in \\eqref{eq:x_plus_update} and \\eqref{eq:step size_condition}. \n\\end{lemma}\n\n\\begin{proof}\n  Since $\\hat{\\vz}_+$ follows the update rule in \\eqref{eq:linear_solver_update}, \n  by Definition~\\ref{def:linear_solver}, the pair $({\\eta}_+,\\hat{\\vz}_+)$ will always satisfy the condition in \\eqref{eq:x_plus_update}. \n  %\n  Hence, it is sufficient to prove that the condition in \\eqref{eq:step size_condition} also holds. \n  Recall that $\\vg$ is defined as $\\vg = \\vF(\\vz)$. By Assumption~\\ref{assum:operator_lips}, the operator $\\vF$ is $L_1$-Lipschitz and thus it follow that  \n  %\n  $\\|\\vF(\\hat{\\vz}_+)-\\vg\\| = \\|\\vF(\\hat{\\vz}_+)-\\vF(\\vz)\\| \\leq L_1 \\|\\hat{\\vz}_+ - \\vz\\|$. \n  %\n  Further, applying the triangle inequality yields \n  \\begin{equation*}\n      \\|\\vF(\\hat{\\vz}_+)-\\vg-\\mB(\\hat{\\vz}_+-\\vz)\\| \\leq \\|\\vF(\\hat{\\vz}_+)-\\vg\\| + \\|\\mB(\\hat{\\vz}_+-\\vz)\\| \\leq (L_1  + \\|\\mB\\|_{\\op}) \\|\\hat{\\vz}_+ - \\vz\\|.\n  \\end{equation*}\n  Hence, if ${\\eta}_+ \\leq \\frac{\\alpha_2}{L_1+\\|\\mB\\|_{\\op}}$, we have \n     \\begin{equation}\\label{eq:approx_error}\n         {\\eta}_+ \\|\\vF(\\hat{\\vz}_+)-\\vg-\\mB(\\hat{\\vz}_+-\\vz)\\| \\leq \\alpha_2 \\|\\hat{\\vz}_+ - \\vz\\|.\n     \\end{equation}\n  Finally, by using the triangle inequality again,  we combine \\eqref{eq:x_plus_update} and \\eqref{eq:approx_error} to establish that  \n  \\begin{align*}\n      \\|\\hat{\\vz}_{+}-\\vz+{\\eta}_+ \\vF(\\hat{\\vz}_{+})\\| &= \\|\\hat{\\vz}_{+}-\\vz+{{\\eta}_+}(\\vg+\\mB({\\hat{\\vz}_{+}}-\\vz)) + {\\eta}_+(\\vF(\\hat{\\vz}_+)-\\vg-\\mB(\\hat{\\vz}_+-\\vz))\\| \\\\\n      &\\leq \\|\\hat{\\vz}_{+}-\\vz+{{\\eta}_+}(\\vg+\\mB({\\hat{\\vz}_{+}}-\\vz))\\| + \\|{\\eta}_+(\\vF(\\hat{\\vz}_+)-\\vg-\\mB(\\hat{\\vz}_+-\\vz))\\| \\\\\n      &\\leq \\alpha_1 \\sqrt{1+\\eta_+ \\mu}\\|\\hat{\\vz}_+-\\vz\\|+\\alpha_2 \\|\\hat{\\vz}_+-\\vz\\| \\\\\n      &\\leq (\\alpha_1+\\alpha_2)\\sqrt{1+\\eta_+ \\mu}\\|\\hat{\\vz}_+-\\vz\\|, \n  \\end{align*}\n  which confirms that the condition in \\eqref{eq:step size_condition} is satisfied. This completes the proof. \n  \\end{proof}","units":[{"start":37905,"end":38371,"start_line":997,"end_line":999,"raw_tex":"\\begin{lemma}\\label{lem:jacobian}\nUnder Assumption~\\ref{assum:strong_monotone}, we have $\\frac{1}{2}(\\nabla \\vF(\\vz) + \\nabla \\vF(\\vz)^\\top) \\succeq \\mu \\mI$ for any $\\vz\\in \\reals^d$, while under Assumption~\\ref{assum:monotone}, we have $\\frac{1}{2}(\\nabla \\vF(\\vz) + \\nabla \\vF(\\vz)^\\top) \\succeq 0$ for any $\\vz\\in \\reals^d$. Further, under Assumption~\\ref{assum:operator_lips}, we have $\\|\\nabla \\vF(\\vz)\\|_{\\op} \\leq L_1$ for any $\\vz\\in \\reals^d$. \n\\end{lemma}","sha256_utf8":"cd516f6bb85fcbed89ce8b2be1130d5b49887337866b350cd01edc759a434d47"},{"start":45560,"end":46434,"start_line":1073,"end_line":1082,"raw_tex":"\\begin{lemma}\\label{lem:averaging}\n  Suppose Assumption~\\ref{assum:monotone} holds. \n    Let $\\theta_0, \\dots, \\theta_{T-1} \\geq 0$ with $\\sum_{t=0}^{T-1} \\theta_t = 1$ and let $\\vz_0,\\dots,\\vz_{T-1}\\in \\reals^d$. Define the averaged iterates as $\\bar{\\vz}_T = \\sum_{t=0}^{T-1} \\theta_t \\vz_t$. Then: \n    \\begin{enumerate}[(i)]\n      \\item For Problem~\\eqref{eq:monotone}, $\\gap_{\\mathrm{w}}(\\vz; \\mathcal{D}) \\leq \\max_{\\vz \\in \\calD}\\sum_{t=0}^{T-1} \\theta_t \\langle \\vF(\\vz_t), \\vz_t-\\vz\\rangle$. \n      \\item For Problem~\\eqref{eq:minimization}, $\\gap_{f}(\\vz; \\vz^*) \\leq \\sum_{t=0}^{T-1} \\theta_t \\langle \\vF(\\vz_t), \\vz_t-\\vz^*\\rangle$. \n      \\item For Problem~\\eqref{eq:minimax}, $\\gap_{\\mathrm{pd}}(\\vz;\\calX\\times \\calY) \\leq \\max_{\\vz \\in \\calX \\times \\calY}\\sum_{t=0}^{T-1} \\theta_t \\langle \\vF(\\vz_t), \\vz_t-\\vz\\rangle$. \n    \\end{enumerate} \n   %\n\\end{lemma}","sha256_utf8":"4bbcec8bbfca61ff7b174b8a93fe3f93891eb37e07127c6e497392ea63feea89"},{"start":66350,"end":67172,"start_line":1330,"end_line":1341,"raw_tex":"\\begin{lemma}\\label{lem:one_step}\n  Suppose that the operator $\\vF$ in \\eqref{eq:monotone} is $\\mu$-strongly monotone with $\\mu \\geq 0$.  \n  Let $\\{\\vz_k\\}_{k\\geq 0}$ and $\\{\\hat{\\vz}_k\\}_{k\\geq 0}$ be the iterates generated by Algorithm~\\ref{alg:Full_Equasi-Newton}, where $\\alpha_1\\in [0,1)$, $\\alpha_2 \\in (0,1)$ and $\\alpha_1+\\alpha_2 <1$. Then for any $\\vz \\in \\reals^d$ and $k \\geq 0$, it holds that\n  \\begin{equation}\\label{eq:one_step}\n    \\begin{aligned}\n      \\eta_k \\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz \\rangle &\\leq \\frac{\\|\\vz_k\\!-\\! \\vz\\|^2}{2} - \\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1} - \\vz\\|^2 + \\eta_k\\mu \\|\\hat{\\vz}_k - \\vz\\|^2 \\\\\n      & \\phantom{{}\\leq{}} \n      - \\frac{1-\\alpha_1-\\alpha_2}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2).\n    \\end{aligned}\n  \\end{equation} \n\n\\end{lemma}","sha256_utf8":"24070a5d43f78d01e0f2493287b514e7f2418998a49703dbc700ed67879b3f35"},{"start":67173,"end":70972,"start_line":1342,"end_line":1389,"raw_tex":"\\begin{proof}\n  To simplify the notation, let $\\alpha \\triangleq \\alpha_1+\\alpha_2\\in (0,1)$. For any $\\vz\\in \\reals^d$, we have \n\\begin{equation}\\label{eq:inexact_pp_decomp}\n  \\eta_k \\langle \\vF(\\hat{\\vz}_k), \\hat{\\vz}_k-\\vz \\rangle =  \\langle \\hat{\\vz}_k-\\vz_k+\\eta_k \\vF(\\hat{\\vz}_k), \\hat{\\vz}_k-\\vz \\rangle\n  +\\langle \\vz_k-\\hat{\\vz}_k, \\hat{\\vz}_k-\\vz\\rangle. %\n   %\n   %\n\\end{equation}\nFor the first term in \\eqref{eq:inexact_pp_decomp}, we can bound it by\n\\begin{align}\n  \\langle \\hat{\\vz}_k-\\vz_k+\\eta_k \\vF(\\hat{\\vz}_k), \\hat{\\vz}_k-\\vz \\rangle &\\leq \\|\\hat{\\vz}_k-\\vz_k+\\eta_k\\vF(\\hat{\\vz}_k)\\|\\|\\hat{\\vz}_k-\\vz\\| \\nonumber\\\\\n  &\\leq \\alpha\\sqrt{1+\\eta_k\\mu}\\|\\hat{\\vz}_k-\\vz_k\\|\\|\\hat{\\vz}_k-\\vz\\| \\nonumber\\\\\n  &\\leq {\\frac{\\alpha}{2}\\|\\hat{\\vz}_k-\\vz_k\\|^2+\\frac{\\alpha(1+\\eta_k \\mu)}{2}\\|\\hat{\\vz}_k-\\vz\\|^2}.\\label{eq:triangle_ipp}\n\\end{align}\nHere, the first inequality is due to the Cauchy-Schwarz inequality, the second inequality comes from \\eqref{eq:inexact_proximal_point}, and the last inequality is a result of Young's inequality. \nAdditionally, to handle the second term in \\eqref{eq:inexact_pp_decomp}, we apply the three-point equality, yielding:\n\\begin{equation}\\label{eq:three_point}\n  \\langle \\vz_k-\\hat{\\vz}_k, \\hat{\\vz}_k-\\vz\\rangle = \\frac{1}{2}\\|\\vz_k-\\vz\\|^2-\\frac{1}{2}\\|\\hat{\\vz}_k - \\vz_k\\|^2-\\frac{1}{2}\\|\\hat{\\vz}_k-\\vz\\|^2.\n\\end{equation}\n%\nBy combining \\eqref{eq:inexact_pp_decomp}, \\eqref{eq:triangle_ipp} and \\eqref{eq:three_point}, we obtain that    \n\\begin{equation}\\label{eq:intermediate_1}\n  \\eta_k \\langle \\vF(\\hat{\\vz}_k), \\hat{\\vz}_k-\\vz \\rangle \\leq \\frac{\\|\\vz_k-\\vz\\|^2 }{2}-\\frac{1-\\alpha}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz\\|^2) + \\frac{\\alpha\\eta_k \\mu}{2}\\|\\hat{\\vz}_k-\\vz\\|^2.\n\\end{equation}\nFurthermore, it follows from the update rule in \\eqref{eq:extragradient_mixing} that $\\eta_k\\vF(\\hat{\\vz}_k) = \\vz_k-\\vz_{k+1}+2\\eta_k\\mu (\\hat{\\vz}_k-\\vz_{k+1})$. Thus, for any $\\vz\\in \\reals^d$, it also holds that\n%\n\\begin{equation}\n  \\begin{aligned}\\label{eq:intermediate_2}\n    \\phantom{{}={}}\\eta_k \\langle \\vF(\\hat{\\vz}_k), \\vz_{k+1}-\\vz \\rangle &= \\langle \\vz_k-\\vz_{k+1}, \\vz_{k+1}-\\vz \\rangle+2\\eta_k\\mu \\langle \\hat{\\vz}_k-\\vz_{k+1}, \\vz_{k+1}-\\vz\\rangle %\n    \\\\\n    & = \\frac{\\|\\vz_{k}-\\vz\\|^2}{2}-\\frac{\\|\\vz_k-\\vz_{k+1}\\|^2}{2}-\\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1}-\\vz\\|^2\\\\\n    & \\phantom{{}={}}+ {\\eta_k\\mu}\\|\\hat{\\vz}_{k}-\\vz\\|^2-{\\eta_k\\mu}\\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2, \n  \\end{aligned}\n\\end{equation}\nwhere we applied the three-point equality twice in the last equality. Hence, by adding the inequality in \\eqref{eq:intermediate_1} with $\\vz = \\vz_{k+1}$ to the inequality in \\eqref{eq:intermediate_2}, we obtain \n\\begin{equation}\\label{eq:linearized_loss}\n  \\begin{aligned}\n    \\eta_k \\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz \\rangle \n    &= \\eta_k \\langle \\vF(\\hat{\\vz}_k),{\\vz}_{k+1} - \\vz \\rangle \n    +\\eta_k \\langle \\vF(\\hat{\\vz}_k), \\hat{\\vz}_{k} - \\vz_{k+1} \\rangle \\\\\n    &\\leq \\frac{\\|\\vz_{k}-\\vz\\|^2}{2}-\\bcancel{\\frac{\\|\\vz_k-\\vz_{k+1}\\|^2}{2}}-\\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1}-\\vz\\|^2 \\\\\n    &\\phantom{{}={}} + {\\eta_k\\mu}\\|\\hat{\\vz}_{k}-\\vz\\|^2-{{\\eta_k\\mu}\\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2} + \\bcancel{\\frac{\\|\\vz_k-\\vz_{k+1}\\|^2}{2}}\\\\\n    &\\phantom{{}={}} -\\frac{1\\!-\\!\\alpha}{2}(\\|\\hat{\\vz}_k \\!-\\! \\vz_k\\|^2 \\!+\\! \\|\\hat{\\vz}_k\\!-\\!\\vz_{k+1}\\|^2 )+{\\frac{\\alpha \\eta_k\\mu}{2}\\|\\hat{\\vz}_k\\!-\\!\\vz_{k+1}\\|^2} \\!. %\n    %\n  \\end{aligned}\n\\end{equation} \nMoreover, since $\\alpha <1$, we further have $-{{\\eta_k\\mu}\\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2} + \\frac{\\alpha \\eta_k \\mu}{2}\\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2 \\leq -\\frac{\\eta_k\\mu}{2}\\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2 \\leq 0$.  Combining this with \\eqref{eq:linearized_loss}  and rearranging the terms, we arrive at the desired result in~\\eqref{eq:one_step}. \n\\end{proof}","sha256_utf8":"dbbfc1228b3dbbcf45955aaece4fb00c3c8bb7b2bb5b7a65fcf6b2e92fd538a8"},{"start":71104,"end":72037,"start_line":1395,"end_line":1410,"raw_tex":"\\begin{proposition}\\label{prop:HPE}\n  Let $\\{\\vz_k\\}_{k\\geq 0}$ and $\\{\\hat{\\vz}_k\\}_{k\\geq 0}$ be the iterates generated by Algorithm~\\ref{alg:Full_Equasi-Newton}, where $\\alpha_1\\in [0,1)$, $\\alpha_2 \\in (0,1)$ and $\\alpha_1+\\alpha_2 <1$. \n  %\n  \\begin{enumerate}[(a)]\n    \\item Under Assumption~\\ref{assum:strong_monotone}, we have $\\|\\vz_{k+1}-\\vz^*\\|^2 \\leq \\|\\vz_k-\\vz^*\\|^2 (1+2\\eta_k \\mu)^{-1}$ for any $k \\geq 0$. \n    \\item Under Assumption~\\ref{assum:monotone}, we have  $\\|\\vz_{k+1}-\\vz^*\\| \\leq \\|\\vz_k-\\vz^*\\|$ for any $k \\geq 0$. Moreover, define the averaged iterate $\\bar{\\vz}_N$ by $\\bar{\\vz}_N = \\frac{\\sum_{k=0}^{N-1} \\eta_k \\hat{\\vz}_k}{\\sum_{k=0}^{N-1} \\eta_k}$. Then for any compact set $\\calD \\subset \\reals^d$, we have \n    %\n    $\\gap(\\bar{\\vz}_N; \\calD) \\leq \\frac{\\max_{\\vz \\in \\calD}\\,\\|\\vz_0-\\vz\\|^2}{2 \\sum_{k=0}^{N-1} \\eta_k}$.  \n    %\n    %\n    %\n    %\n    %\n    %\n  \\end{enumerate}\n\\end{proposition}","sha256_utf8":"486c3bac178afd87c4f699dff8b7ae5345ae7dffdaa07916c17063be695293d3"},{"start":72039,"end":74850,"start_line":1412,"end_line":1440,"raw_tex":"\\begin{proof}\nLet $\\alpha = \\alpha_1 + \\alpha_2$. First, we prove Part (a). Since $\\mF(\\vz^*) = 0$ and $\\vF$ is $\\mu$-strongly monotone by Assumption~\\ref{assum:strong_monotone}, it holds that  \n\\begin{equation}\\label{eq:strong_monotone}\n\\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz^* \\rangle = \\langle \\vF(\\hat{\\vz}_k)-\\vF({\\vz^*}),\\hat{\\vz}_k - \\vz^* \\rangle \\geq \\mu\\|\\hat{\\vz}_k - \\vz^*\\|^2.\n\\end{equation}\nBy setting $\\vz = \\vz^*$ in \\eqref{eq:one_step} and applying \\eqref{eq:strong_monotone}, it yields \n\\begin{equation}\\label{eq:one_step_strongly}\n    0 \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1} - \\vz^*\\|^2  - \\frac{1-\\alpha}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2).\n\\end{equation} \nSince $\\alpha = \\alpha_1 + \\alpha_2 <1$, we can drop the non-positive last term in \\eqref{eq:one_step_strongly} and obtain $0 \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1} - \\vz^*\\|^2$, which is equivalent to $\\|\\vz_{k+1} - \\vz^*\\|^2 \\leq \\|\\vz_k- \\vz^*\\|^2 (1+2\\eta_k\\mu)^{-1}$. This completes the proof of Part (a). \n\nNext, we prove Part (b). Since $\\vF$ is monotone by Assumption~\\ref{assum:monotone}, we have\n  \\begin{equation}\\label{eq:monotone_star}\n    \\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz^* \\rangle = \\langle \\vF(\\hat{\\vz}_k)-\\vF({\\vz^*}),\\hat{\\vz}_k - \\vz^* \\rangle \\geq 0.\n    \\end{equation}\n    By setting $\\vz = \\vz^*$ in \\eqref{eq:one_step} with $\\mu = 0$ and applying \\eqref{eq:monotone_star}, it yields \n    \\begin{equation}\\label{eq:one_step_monotone}\n      0 \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1}{2}\\|\\vz_{k+1} - \\vz^*\\|^2  - \\frac{1-\\alpha}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2).\n    \\end{equation}\n    Again, since the last term in \\eqref{eq:one_step_monotone} is non-positive, this immediately implies $\\frac{1}{2}\\|\\vz_{k+1} - \\vz^*\\|^2 \\leq \\frac{1}{2}\\|\\vz_{k} - \\vz^*\\|^2$, which is equivalent to $\\|\\vz_{k+1} - \\vz^*\\| \\leq \\|\\vz_{k} - \\vz^*\\|$. Additionally, by dropping the non-positive last term in \\eqref{eq:one_step} and noting that $\\mu = 0$, we also have \n    %\n    $\\eta_k \\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz \\rangle \\leq \\frac{1}{2}\\|\\vz_k- \\vz\\|^2 - \\frac{1}{2}\\|\\vz_{k+1} - \\vz\\|^2$.\n    %\n    By summing this inequality from $k=0$ to $k=N-1$, we further get \n    %\n    $\\sum_{k=0}^{N-1} \\eta_k \\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz \\rangle \\leq \\frac{1}{2}\\|\\vz_0- \\vz\\|^2 - \\frac{1}{2}\\|\\vz_{N} - \\vz\\|^2 \\leq \\frac{1}{2}\\|\\vz_0- \\vz\\|^2$. \n    %\n    Hence, it follows from Lemma~\\ref{lem:averaging} that $\\gap(\\bar{\\vz}_N; \\calD) \\leq \\frac{\\max_{\\vz \\in \\calD}\\,\\|\\vz_0-\\vz\\|^2}{2 \\sum_{k=0}^{N-1} \\eta_k}$, where $\\bar{\\vz}_N$ is the averaged iterate given by $\\bar{\\vz}_N = \\frac{\\sum_{k=0}^{N-1} \\eta_k \\hat{\\vz}_k}{\\sum_{k=0}^{N-1} \\eta_k}$.  \n\\end{proof}","sha256_utf8":"863b5e0f97cb02e7314d8bfaef834c3289939b0c84df0ef2feafc10ac8a475c9"},{"start":75941,"end":76616,"start_line":1449,"end_line":1468,"raw_tex":"\\begin{lemma}\\label{lem:step size_lb}\n  %\n  %\n    %\n   %\n   %\n    Recall that ${\\tilde{\\vz}_k}$ is the auxiliary iterate returned by our line search scheme in Subroutine~\\ref{alg:ls}. For $k\\notin \\mathcal{B}$ we have $\\eta_k = \\sigma_k$, while for $k\\in \\mathcal{B}$ we have \n   %\n    %\n     $ \\eta_k \\!> \\!\n      %\n      {\\frac{{\\alpha_2} \\beta\\|{\\tilde{\\vz}_k}-\\vz_k\\|}{\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\|}}$.\n      %\n      %\n     %\n    %\n    Moreover, if $\\frac{1}{2}(\\mB_k + \\mB_k^\\top) \\succeq \\frac{1}{2}\\mu \\mI$, we have $\\|{\\tilde{\\vz}_k}-\\vz_k\\| \\leq \\frac{1+\\alpha_1}{\\beta(1-\\alpha_1)} \\|{\\hat{\\vz}_k}-\\vz_k\\|$. \n    %\n  %\n\\end{lemma}","sha256_utf8":"25a16e55d4d01aa378184ede6a0124f29d723d85d2a6439de6016639c2516a60"},{"start":76618,"end":84452,"start_line":1470,"end_line":1596,"raw_tex":"\\begin{proof}\n  If $k\\notin \\mathcal{B}$, as per the definition, the line search scheme adopts the initial trial step size at the $k$-th iteration and thus $\\eta_k = \\sigma_k$. On the other hand, if $k \\in \\mathcal{B}$, we go through the backtracking procedure in Subroutine~\\ref{alg:ls}.\n  %\n  Let $\\tilde{\\vz}_k$ denote the last rejected point in the line search scheme, which is calculated from \\eqref{eq:linear_solver_update} using the step size $\\tilde{\\eta}_k = \\eta_k/\\beta$.\n  %\n  This means that the pair $(\\tilde{\\vz}_k,\\tilde{\\eta}_k)$ satisfies \\eqref{eq:x_plus_update} but not  \\eqref{eq:step size_condition}, i.e.,$\\|\\tilde{\\vz}_k-\\vz_k+\\tilde{\\eta}_k(\\vF(\\vz_k)+\\mB_k(\\tilde{\\vz}_k-\\vz))\\| \\leq  {\\alpha_1}\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|\\tilde{\\vz}_k-\\vz_k\\|$ and $\\|\\tilde{\\vz}_k - \\vz_k + \\tilde{\\eta}_k \\vF(\\tilde{\\vz}_k)\\| > (\\alpha_1+\\alpha_2)\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|{\\tilde{\\vz}_{k}}-\\vz_k\\|$. Moreover, note that $\\tilde{\\eta}_k (\\vF(\\tilde{\\vz}_k) - \\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)) = \\left(\\tilde{\\vz}_k - \\vz_k + \\tilde{\\eta}_k \\vF(\\tilde{\\vz}_k)\\right) - ({\\tilde{\\vz}_k}-\\vz_k+\\tilde{\\eta}_k(\\vF(\\vz_k)+\\mB_k({\\tilde{\\vz}_k}-\\vz_k)))$.  \n  %\n  %\n  %\n  %\n  %\n  Thus, it follows from the triangle inequality that $\\tilde{\\eta}_k \\|\\vF(\\tilde{\\vz}_k) - \\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\| > (\\alpha_1+\\alpha_2)\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|{\\tilde{\\vz}_{k}}-\\vz_k\\| - \\alpha_1\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|{\\tilde{\\vz}_{k}}-\\vz_k\\| = \\alpha_2\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|{\\tilde{\\vz}_{k}}-\\vz_k\\|$.\n  %\n  %\n  %\n  %\n  %\n  This further implies that \n  %\n  $\\tilde{\\eta}_k > \\frac{\\alpha_2\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|{\\tilde{\\vz}_{k}}-\\vz_k\\|}{\\|\\vF(\\tilde{\\vz}_k) - \\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\|}$. \n  %\n  %\n  %\n  %\n  %\n    Since $1 + \\tilde{\\eta}_k \\mu \\geq 1$, \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    we obtain the first result in Lemma~\\ref{lem:step size_lb}.\n    To prove the second result in Lemma~\\ref{lem:step size_lb}, \n    recall from~\\eqref{eq:x_plus_update} that $\\hat{\\vz}_k$ and $\\tilde{\\vz}_k$ are inexact solutions of the linear system of equations:  \n    \\begin{equation*}\n      (\\mI+{\\eta_k}\\mB_k)({\\vz}-\\vz_k) = -\\eta_k\\vF(\\vz_k) \\quad \\text{and} \\quad (\\mI+\\tilde{\\eta}_k\\mB_k)({\\vz}-\\vz_k) = -\\tilde{\\eta}_k\\vF(\\vz_k),\n    \\end{equation*}\n    %\n    respectively. \n    Define $\\hat{\\vz}_k^* = \\vz_k - {\\eta}_k(\\mI+ {\\eta}_k\\mB_k)^{-1} \\vF(\\vz_k)$ and $\\tilde{\\vz}_k^* = \\vz_k - \\tilde{\\eta}_k(\\mI+ \\tilde{\\eta}_k\\mB_k)^{-1} \\vF(\\vz_k)$, i.e., the exact solutions of the above linear systems. Since $(\\hat{\\vz}_k,\\eta_k)$ and $(\\tilde{\\vz}_k,\\tilde{\\eta}_k)$ satisfy the condition in \\eqref{eq:x_plus_update}, we have \n    \\begin{align}\\label{eq:inexact_condition_linear}\n      \\|(\\mI+\\eta_k\\mB_k)(\\hat{\\vz}_k-\\hat{\\vz}_k^*)\\| &\\leq \\alpha_1 \\sqrt{1+\\eta_k \\mu} \\|\\hat{\\vz}_k - \\vz_k\\| \\\\\n      \\quad \\text{and} \\quad  \\|(\\mI+\\tilde{\\eta}_k\\mB_k)(\\tilde{\\vz}_k-\\tilde{\\vz}_k^*)\\| &\\leq \\alpha_1 \\sqrt{1+\\tilde{\\eta}_k \\mu}  \\|\\tilde{\\vz}_k - \\vz_k\\|.\n    \\end{align} \n    We divide the proof of the second result {in Lemma~\\ref{lem:step size_lb}} into the following three steps.\n    %\n      First, we  show that \n      %\n      %\n        \\begin{align}\n                    (1-\\alpha_1)\\|\\hat{\\vz}_k - \\vz_k\\| &\\leq \\|\\hat{\\vz}_k^*-{\\vz}_k\\| \\leq (1+\\alpha_1)\\|\\hat{\\vz}_k - \\vz_k\\|, \\label{eq:relation_exact_inexact1}\\\\ (1-\\alpha_1)\\|\\tilde{\\vz}_k - \\vz_k\\| &\\leq \\|\\tilde{\\vz}_k^*-{\\vz}_k\\| \\leq (1+\\alpha_1)\\|\\tilde{\\vz}_k - \\vz_k\\|.\\label{eq:relation_exact_inexact2}\n        \\end{align}\n      %\n      %\n      In the following, we will only prove \\eqref{eq:relation_exact_inexact1}, since the proof of \\eqref{eq:relation_exact_inexact2} follows similarly.\n      Since we assume $\\frac{1}{2}(\\mB_k + \\mB_k^\\top) \\succeq \\frac{\\mu}{2} \\mI$, we have $\\vv^\\top \\mB_k \\vv \\geq \\frac{\\mu}{2} \\|\\vv\\|^2$ for any $\\vv \\in \\reals^d$. Therefore, by using Cauchy-Schwarz inequality, we get   \n      %\n      $\\|\\hat{\\vz}_k-\\hat{\\vz}_k^*\\| \\|(\\mI+\\eta_k\\mB_k)(\\hat{\\vz}_k-\\hat{\\vz}_k^*)\\| \\geq (\\hat{\\vz}_k-\\hat{\\vz}_k^*)^\\top (\\mI+\\eta_k\\mB_k)(\\hat{\\vz}_k-\\hat{\\vz}_k^*) \\geq (1+ \\frac{\\eta_k \\mu}{2})\\|\\hat{\\vz}_k-\\hat{\\vz}_k^*\\|^2$,\n      %\n      which further implies that $\\|(\\mI+\\eta_k\\mB_k)(\\hat{\\vz}_k-\\hat{\\vz}_k^*)\\| \\geq (1+ \\frac{1}{2}\\eta_k \\mu)\\|\\hat{\\vz}_k-\\hat{\\vz}_k^*\\|$. \n      Moreover, since $ \\sqrt{1+\\eta_k\\mu} \\leq 1+ \\frac{1}{2} \\eta_k\\mu $, by~\\eqref{eq:inexact_condition_linear} we also have $\\|(\\mI+\\eta_k\\mB_k)(\\hat{\\vz}_k-\\hat{\\vz}_k^*)\\| \\leq \\alpha_1 {(1+ \\frac{1}{2}\\eta_k \\mu)} \\|\\hat{\\vz}_k - \\vz_k\\|$. \n      %\n      %\n      %\n      %\n      Hence, combining these two inequalities, we get $\\|\\hat{\\vz}_k-\\hat{\\vz}_k^*\\| \\leq \\alpha_1 \\|\\hat{\\vz}_k-\\vz_k\\|$. It then follows from the triangle inequality that\n      \\begin{align*}\n        \\|\\hat{\\vz}_k^*-\\vz_k\\| \\leq \\|\\hat{\\vz}_k - \\vz_k\\| + \\|\\hat{\\vz}_k^*-\\hat{\\vz}_k\\| \\leq (1+\\alpha_1)\\|\\hat{\\vz}_k - \\vz_k\\|, \\\\\n        \\|\\hat{\\vz}_k^*-\\vz_k\\| \\geq \\|\\hat{\\vz}_k - \\vz_k\\| -  \\|\\hat{\\vz}_k^*-\\hat{\\vz}_k\\| \\geq (1-\\alpha_1)\\|\\hat{\\vz}_k - \\vz_k\\|,\n      \\end{align*}\n      which proves \\eqref{eq:relation_exact_inexact1}.\n    %\n  Next, we show that \n     \\begin{equation}\\label{eq:relation_displacement}\n      \\|{\\tilde{\\vz}^*_k}-\\vz_k\\|\\leq \\frac{1}{\\beta}\\|{\\hat{\\vz}^*_k}-\\vz_k\\|.\n     \\end{equation}\n     This follows from \\cite[Lemma 7.8]{monteiro2010complexity}. For completeness, we present its proof below. Note that by definition, we have $(\\mI+{\\eta_k}\\mB_k)(\\hat{\\vz}_k^*-\\vz_k) = -\\eta_k\\vF(\\vz_k)$ and $(\\mI+\\tilde{\\eta}_k\\mB_k)(\\tilde{\\vz}_k^*-\\vz_k) = -\\tilde{\\eta}_k\\vF(\\vz_k)$.\n    %\n    %\n    %\n     Hence, we further have \n     \\begin{equation*}\n      \\mB_k(\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*) = \\frac{1}{\\tilde{\\eta}_k}(\\tilde{\\vz}_k^*-\\vz_k) - \\frac{1}{\\eta_k}(\\hat{\\vz}_k^*-\\vz_k) = \\frac{1}{\\tilde{\\eta}_k} (\\tilde{\\vz}_k^* - \\hat{\\vz}_k^*) + \\left(\\frac{1}{\\tilde{\\eta}_k}-\\frac{1}{\\eta_k}\\right) (\\hat{\\vz}_k^*-\\vz_k).\n     \\end{equation*}\n     Since $(\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*)^\\top \\mB_k(\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*) \\geq 0$, by taking the inner product with $\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*$ on both sides of the above inequality, we obtain\n     %\n    $\\left(\\frac{1}{\\tilde{\\eta}_k}-\\frac{1}{\\eta_k}\\right) (\\hat{\\vz}_k^*-\\vz_k)^\\top (\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*)  \\geq \\frac{1}{\\tilde{\\eta}_k} \\|\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*\\|^2$.\n     %\n     Since ${\\eta}_k = \\beta \\tilde{\\eta}_k$, using the Cauchy-Schwarz inequality, this further leads to \n     %\n    $\\|\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*\\|^2 \\leq \\left(\\frac{1}{\\beta}-1\\right)(\\vz_k - \\hat{\\vz}_k^*)^\\top (\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*) \\leq  \\left(\\frac{1}{\\beta}-1\\right) \\|\\hat{\\vz}_k^* - \\vz_k\\| \\|\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*\\|$.\n     %\n     Hence, we further have $\\|\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*\\| \\leq \\left(\\frac{1}{\\beta}-1\\right) \\|\\hat{\\vz}_k^* - \\vz_k\\|$, which implies that $\\|{\\tilde{\\vz}^*_k}-\\vz_k\\| \\leq \\|{\\tilde{\\vz}^*_k}-\\hat{\\vz}^*_k\\|+\\|\\hat{\\vz}^*_k-\\vz_k\\| \\leq \\frac{1}{\\beta} \\|\\hat{\\vz}^*_k-\\vz_k\\|$. \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    This completes the proof of \\eqref{eq:relation_displacement}. Finally, by combining \\eqref{eq:relation_exact_inexact1}, \\eqref{eq:relation_exact_inexact2}, and \\eqref{eq:relation_displacement}, it follows that $\\|{\\tilde{\\vz}_k}-\\vz_k\\| \\leq \\frac{1}{1-\\alpha_1}\\|\\tilde{\\vz}_k^*-{\\vz}_k\\| \\leq \\frac{1}{(1-\\alpha_1)\\beta}\\|{\\hat{\\vz}^*_k}-\\vz_k\\| \\leq \\frac{1+\\alpha_1}{1-\\alpha_1}\\|\\tilde{\\vz}_k^*-{\\vz}_k\\| \\leq \\frac{1+\\alpha_1}{(1-\\alpha_1)\\beta}\\|{\\hat{\\vz}_k}-\\vz_k\\|$. This proves the second result {in Lemma~\\ref{lem:step size_lb}}. \n  \\end{proof}","sha256_utf8":"d6febe9b8dfa0d136fde819207196f93d4ce968f89d8b4a1dab806469d04743d"},{"start":85305,"end":85766,"start_line":1606,"end_line":1612,"raw_tex":"To begin with, \\blue{in the strongly monotone setting,} by repeatedly applying Proposition~\\ref{prop:HPE}(a), we obtain that \n%\n\\begin{equation}\\label{eq:after_jensen}\n   \\frac{\\|\\vz_{N}-\\vz^*\\|^2}{\\|\\vz_0-\\vz^*\\|^2} \\leq \\prod_{k=0}^{N-1} (1+2\\eta_k\\mu)^{-1} \\leq \\biggl(1+{\\frac{2\\mu N}{\\sum_{k=0}^{N-1} 1/\\eta_k}}\\biggr)^{-N}, \n   %\n\\end{equation}\nwhere the last inequality is obtained by applying Jensen's inequality to the function $\\log(1+\\frac{1}{t})$. \n","sha256_utf8":"270a4b83eef54ff529f5a3b1a202e3484a8f20b9754a7bfc151095f881e4cf80"},{"start":86348,"end":86980,"start_line":1622,"end_line":1633,"raw_tex":"\\begin{lemma}\\label{lem:stepsize_bnd}\nLet $\\{\\eta_k\\}_{k=0}^{N-1}$ be the step sizes in Algorithm~\\ref{alg:Full_Equasi-Newton} obtained by the line search in Subroutine~\\ref{alg:ls}.~Then,\n%\n\\begin{equation}\\label{eq:goal}\n     \\sum_{k=0}^{N-1}\\frac{1}{\\eta_k} \\leq \\frac{1}{(1-\\beta)\\sigma_0}+ \\frac{1}{(1-\\beta)\\alpha_2\\beta} \n    %\n     \\sqrt{N \\sum_{k\\in \\mathcal{B}} \\frac{\\|\\vu_k-\\mB_k\\vs_k\\|^2}{\\|\\vs_k\\|^2}}, \n     %\n      %\n\\end{equation}\nwhere $\\vu_k \\triangleq \\vF(\\tilde{\\vz}_k) -\\vF({\\vz_k})$, $\\vs_k \\triangleq \\tilde{\\vz}_k-\\vz_k$, and $\\tilde{\\vz}_k$ is the auxiliary iterate in Subroutine~\\ref{alg:ls}. \n\\end{lemma}","sha256_utf8":"538785db9710fca65f2932a7e3b82cdc645db46fee5fa26c4fbf095a97c1de81"},{"start":86981,"end":88958,"start_line":1634,"end_line":1686,"raw_tex":"\\begin{proof}\n  Let $\\vu_k \\triangleq \\vF(\\tilde{\\vz}_k) -\\vF({\\vz_k})$ and $\\vs_k \\triangleq \\tilde{\\vz}_k-\\vz_k$. \n  In Lemma~\\ref{lem:step size_lb}, we showed that $\\eta_k = \\sigma_k$ if $k\\notin \\mathcal{B}$ and $\\eta_k> \\frac{\\alpha_2 \\beta \\|\\vs_k\\|}{\\|\\vu_k-\\mB_k\\vs_k\\|}$ otherwise. %\n%\n%\nUsing the observations above, we can write \n\\begin{equation}\\label{eq:intermediate_bound}\n  %\n    \\sum_{k=0}^{N-1}\\frac{1}{\\eta_k} \n  = \\sum_{k\\notin \\mathcal{B}}\\frac{1}{\\eta_k} + \\sum_{k\\in \\mathcal{B}}\\frac{1}{\\eta_k} \\leq\n  \\sum_{k\\notin \\mathcal{B}}\\frac{1}{\\sigma_k} + \\sum_{k\\in \\mathcal{B}}\\frac{1}{\\eta_k}\n  \\leq \\frac{1}{\\sigma_0} + \n  \\beta\\sum_{k\\notin \\mathcal{B},k\\geq 1}\\frac{1}{\\eta_{k-1}}+\\sum_{k\\in \\mathcal{B}}\\frac{1}{\\eta_k},\n  %\n\\end{equation}\n%\n%\n%\n%\n%\n%\n%\n%\n%\nwhere we used $\\sigma_k = \\frac{\\eta_{k-1}}{\\beta}$ for $k\\geq 1$ in the last equality. \nSince\n%\n$\\sum_{k\\notin \\mathcal{B},k\\geq 1}\\frac{1}{\\eta_{k-1}} \n%\n\\leq \\sum_{k= 0}^{N-1}\\frac{1}{\\eta_{k}}$,\n%\nrearranging and simplifying the terms in \\eqref{eq:intermediate_bound}, we arrive at \n\\begin{equation}\\label{eq:all_vs_B}\n  \\sum_{k=0}^{N-1}\\frac{1}{\\eta_k} \\leq \\frac{1}{(1-\\beta)\\sigma_0} + \\frac{1}{1-\\beta} \\sum_{k\\in \\mathcal{B}}\\frac{1}{\\eta_k}. \n\\end{equation}\n%\nNow using $\\eta_k > \\frac{\\alpha_2 \\beta \\|\\vs_k\\|}{\\|\\vu_k-\\mB_k\\vs_k\\|}$ for $k \\in \\calB$, we further have \n\\begin{align}\n  \\sum_{k \\in \\calB} \\frac{1}{\\eta_k} \\leq \\sum_{k \\in \\calB} \\frac{\\|\\vu_k-\\mB_k\\vs_k\\|}{\\alpha_2 \\beta \\|\\vs_k\\|} &\\leq \\frac{1}{\\alpha_2\\beta} \\sqrt{|\\calB| \\sum_{k\\in \\mathcal{B}} \\frac{\\|\\vu_k-\\mB_k\\vs_k\\|^2}{\\|\\vs_k\\|^2}} \n  %\n  %\n  \\leq \\frac{1}{\\alpha_2\\beta} \n  \\sqrt{N \\sum_{k\\in \\mathcal{B}} \\frac{\\|\\vu_k-\\mB_k\\vs_k\\|^2}{\\|\\vs_k\\|^2}}, \\label{eq:B_bound1}\n\\end{align}\nwhere the second inequality is due to the generalized mean inequality. \n%\n%\n%\n%\nFinally, the inequality in \\eqref{eq:goal} follows from \\eqref{eq:all_vs_B} and \\eqref{eq:B_bound1}. \n%\nThis completes the proof.\n\\end{proof}","sha256_utf8":"b7c5c17ed5e4e7ffc5e41506e45dec6ba2eaa8b314b9b16ee8218eb0c11e2208"},{"start":106628,"end":108240,"start_line":1886,"end_line":1923,"raw_tex":"\\begin{lemma}\\label{lem:regret_reduction}\n Suppose the loss function $\\ell_t$ is convex for any $t\\geq 0$ and let $\\{\\vx_t\\}_{t=0}^{T-1}$ and $\\{\\vw_t\\}_{t=0}^{T-1}$ be the iterates generated by Algorithm~\\ref{alg:projection_free_online_learning}. Then we have $\\vw_t \\in \\vspan(\\calC)$ for any $t \\geq 0$. Moreover: \n  \\begin{enumerate}[(a)]\n    \\item In \\textbf{Option I}, we have $\\vx_t \\in (1+\\delta)\\calC$ for $t=0,1,\\dots,T-1$. Also, for any $\\vu\\in \\mathcal{C}$, it holds that \n    \\begin{equation}\n    %\n    \\ell_t(\\vx_t) - \\ell_t(\\vu) %\n    \\leq \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle %\n    \\leq \\frac{\\|\\vw_t-\\vu\\|^2_2}{2\\rho}-\\frac{\\|\\vw_{t+1}-\\vu\\|^2_2}{2\\rho}+ \\frac{\\rho}{2}\\|\\tilde{\\vg}_t\\|_2^2,\\label{eq:regret_reduction} %\n\\end{equation}\n%\n\\begin{equation}\\label{eq:surrogate_loss}\n   \\|\\tilde{\\vg}_t\\| \\leq \\|\\vg_t\\|+|\\langle \\vg_t, \\vx_t\\rangle|\\|\\vs_t\\|.\n\\end{equation}\n\\item In \\textbf{Option II}, we have $\\vx_t \\in \\calC$ for $t=0,1,\\dots,T-1$. Also, for any $\\vu\\in \\mathcal{C}$, it holds that \n  \\begin{align}\n  %\n  \\ell_t(\\vx_t) - \\ell_t(\\vu) \n  %\n  &\\leq \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle - \\delta \\langle \\vg_t, \\vx_t \\rangle \\label{eq:regret_reduction_II_1}\\\\\n  &\\leq \\frac{\\|\\vw_t-\\vu\\|^2_2}{2\\rho}-\\frac{\\|\\vw_{t+1}-\\vu\\|^2_2}{2\\rho}+ \\frac{\\rho}{2}\\|\\tilde{\\vg}_t\\|_2^2 - \\delta \\langle \\vg_t, \\vx_t \\rangle,\\label{eq:regret_reduction_II_2} %\n\\end{align}\n%\n\\begin{equation}\\label{eq:surrogate_loss_II}\n  \\|\\tilde{\\vg}_t\\| \\leq \\|\\vg_t\\|+(1+\\delta)|\\langle \\vg_t, \\vx_t\\rangle|\\|\\vs_t\\|.\n\\end{equation}\n  \\end{enumerate}\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n\\end{lemma}","sha256_utf8":"d47d97fab8ff9066a60a53106a8a92cc5b91186cbcf7719d78617514ed93aed6"},{"start":108241,"end":115842,"start_line":1924,"end_line":2018,"raw_tex":"\\begin{proof}\n  %\n  %\n  %\n  %\n%\n%\nTo begin with, we prove by induction that $\\vw_t \\in \\vspan(\\calC)$. By initialization, we have $\\vw_0 \\in \\vspan(\\calC)$. Now suppose $\\vw_t \\in \\vspan(\\calC)$ for some $t \\geq 0$. Since $\\vg_t =  \\mP\\nabla \\ell_t(\\vx_t) \\in \\vspan(\\calC)$ and $\\vs_t \\in \\vspan(\\calC)$ by Definition~\\ref{def:gauge}, we obtain that $\\tilde{\\vg}_t \\in \\vspan(\\calC)$ in both \\textbf{Options I} and \\textbf{II}. Moreover, since $\\vw_{t+1}$ is expressed as a linear combination of $\\vw_t$ and $\\tilde{\\vg}_t$ according to \\eqref{eq:w_update}, we obtain that $\\vw_{t+1} \\in \\vspan(\\calC)$. Hence, we conclude by induction that $\\vw_t \\in \\vspan(\\calC)$ for any $t\\geq 0$.  \n\nNext, we consider the result in Part (a) for \\textbf{Option I}.  We distinguish two cases depending on the outcome of $\\mathsf{SEP}(\\vw_t;\\delta)$. \n\\begin{itemize}\n  \\item If $\\gamma_t \\leq 1$,  we have $\\vw_t \\in (1+\\delta) \\mathcal{C}$ by Definition~\\ref{def:gauge}. Thus, we have $\\vx_t=\\vw_t\\in (1+\\delta) \\mathcal{C}$ and $\\tilde{\\vg}_t = \\vg_t$, which immediately implies \\eqref{eq:surrogate_loss}. Moreover, since $\\ell_t$ is convex, we have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\nabla \\ell_t(\\vx_t), \\vx_t - \\vu \\rangle$. Note that both $\\vx_t$ and $\\vu$ are in $\\vspan(\\calC)$ and recall that $\\mP$ denotes the orthogonal projection matrix associated with $\\vspan(\\calC)$. Thus, we further have $\\vx_t - \\vu = \\mP(\\vx_t - \\vu)$, which implies that $\\langle \\nabla \\ell_t(\\vx_t), \\vx_t - \\vu \\rangle = \\langle \\nabla \\ell_t(\\vx_t), \\mP(\\vx_t - \\vu )\\rangle = \\langle \\mP\\nabla \\ell_t(\\vx_t), \\vx_t - \\vu \\rangle = \\langle \\vg_t, \\vx_t - \\vu \\rangle$. \n  Therefore, we have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle = \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle$, which proves the first inequality in \\eqref{eq:regret_reduction}. \n  \\item Otherwise, if $\\gamma_t>1$,  we have $\\frac{\\vw_t}{\\gamma_t} \\in (1+\\delta)\\mathcal{C}$ by Definition~\\ref{def:gauge} and $\\langle \\vs_t, \\vw_t-\\vx \\rangle \\geq {\\gamma_t-1}$,  $ \\forall\\vx\\in \\mathcal{C}$. In this case, Algorithm~\\ref{alg:projection_free_online_learning} chooses $\\vx_t  = \\frac{\\vw_t}{\\gamma_t} \\in (1+\\delta)\\mathcal{C}$ and $\\tilde{\\vg}_t = \\vg_t+\\max\\{0, -\\frac{1}{\\gamma_t}\\langle \\vg_t, \\vw_t \\rangle\\} \\vs_t = \\vg_t+\\max\\{0, -\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t$. To prove the first inequality in \\eqref{eq:regret_reduction}, \n  first note that we also have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle$ using similar arguments as above. Moreover, for any $\\vu\\in \\mathcal{C}$,\n  \\begin{align*}\n    \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle &= \\langle\\vg_t+\\max\\{0, -\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t, \\vw_t-\\vu \\rangle \\\\\n    & = \\langle \\vg_t, \\gamma_t\\vx_t-\\vu \\rangle + \\max\\{0, -\\langle \\vg_t, \\vx_t \\rangle\\} \\langle \\vs_t, \\vw_t-\\vu\\rangle \\\\\n    & \\geq \\langle \\vg_t, \\vx_t-\\vu\\rangle + (\\gamma_t-1)\\langle\\vg_t,\\vx_t\\rangle + (\\gamma_t-1)\\max\\{0, -\\langle \\vg_t, \\vx_t \\rangle\\}  \\\\\n    %\n    & \\geq \\langle \\vg_t, \\vx_t-\\vu\\rangle, \n  \\end{align*}\n  where we used $\\vw_t = \\gamma_t \\vx_t$ in the second equality and $\\langle \\vs_t, \\vw_t-\\vu\\rangle \\geq \\gamma_t -1$ in the first inequality. \n  Also, by the triangle inequality we obtain \n  \\begin{equation*}\n    \\|\\tilde{\\vg}_t\\|_2 =\\|\\vg_t+\\max\\{0, -\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t\\|_2 %\n    \\leq  \\|\\vg_t\\|_2+|\\langle \\vg_t, \\vx_t \\rangle|\\|\\vs_t\\|_2,\n  \\end{equation*}\n  which proves \\eqref{eq:surrogate_loss}. \n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n\\end{itemize}\nFinally, from the update rule of $\\vw_{t+1}$ in \\eqref{eq:w_update}, for any $\\vu\\in \\mathcal{C} \\subset \\mathcal{B}_R(0)$, we have \n$\n  \\langle\\vw_t-\\rho\\tilde{\\vg}_t-\\vw_{t+1}, \\vw_{t+1}-\\vu\\rangle \\geq 0\n$. This further implies that \n\\begin{align}\n  \\langle \\tilde{\\vg}_t, \\vw_{t}-\\vu \\rangle &\\leq  \\langle \\tilde{\\vg}_t, \\vw_{t}-\\vw_{t+1}\\rangle+ \\frac{1}{\\rho}\\langle \\vw_t-\\vw_{t+1}, \\vw_{t+1}-\\vu \\rangle \\nonumber\\\\\n  & =  \\langle \\tilde{\\vg}_t, \\vw_{t}-\\vw_{t+1}\\rangle+ \\frac{\\|\\vw_t-\\vu\\|_2^2}{2\\rho}-\\frac{\\|\\vw_{t+1}-\\vu\\|_2^2}{2\\rho}-\\frac{\\|\\vw_t-\\vw_{t+1}\\|_2^2}{2\\rho} \\nonumber\\\\\n  &\\leq \\frac{1}{2\\rho}\\|\\vw_t-\\vu\\|_2^2-\\frac{1}{2\\rho}\\|\\vw_{t+1}-\\vu\\|_2^2 +\\frac{\\rho}{2}\\|\\tilde{\\vg}_t\\|_2^2, \\label{eq:regret_wt}\n\\end{align}\nwhere we used the Young's inequality $\\langle \\tilde{\\vg}_t, \\vw_{t}-\\vw_{t+1}\\rangle \\leq \\frac{\\|\\vw_{t}-\\vw_{t+1}\\|^2}{2\\rho} + \\frac{\\rho}{2}\\|\\tilde{\\vg}_t\\|^2$ in the last inequality. \nThis proves the second inequality in \\eqref{eq:regret_reduction}. \n\n\\blue{Next, we consider Part (b) for \\textbf{Option II}. Similarly, we distinguish two cases depending on the outcome of $\\mathsf{SEP}(\\vw_t;\\delta)$. \n\\begin{itemize}\n  \\item If $\\gamma_t \\leq 1$,  we have $\\vw_t \\in (1+\\delta) \\mathcal{C}$ by Definition~\\ref{def:gauge}. According to \\textbf{Option II} in Algorithm~\\ref{alg:projection_free_online_learning}, we have $\\vx_t= \\frac{\\vw_t}{1+\\delta}\\in \\mathcal{C}$ and $\\tilde{\\vg}_t = \\vg_t$, which immediately implies~\\eqref{eq:surrogate_loss_II}. Using similar arguments as in \\textbf{Option I}, we have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle$. Moreover, since $\\vw_t = (1+\\delta)\\vx_t$, we have $\\langle {\\vg}_t, \\vw_t - \\vu \\rangle = \\langle {\\vg}_t, (1+\\delta)\\vx_t - \\vu \\rangle = \\langle {\\vg}_t, \\vx_t-\\vu \\rangle + \\delta \\langle {\\vg}_t, \\vx_t\\rangle$. Combining these two, we further have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle \\leq\\langle {\\vg}_t, \\vw_t - \\vu \\rangle- \\delta \\langle {\\vg}_t, \\vx_t\\rangle$. This proves the inequality in \\eqref{eq:regret_reduction_II_1}. \n  %\n  \\item Otherwise, if $\\gamma_t>1$, By Definition~\\ref{def:gauge} we have $ \\frac{\\vw_t}{\\gamma_t} \\in (1+\\delta)\\mathcal{C}$ and $\\langle \\vs_t, \\vw_t-\\vx \\rangle \\geq {\\gamma_t-1}$  $ \\forall\\vx\\in \\mathcal{C}$. According to \\textbf{Option II} in Algorithm~\\ref{alg:projection_free_online_learning}, we have $\\vx_t  = \\frac{\\vw_t}{\\gamma_t(1+\\delta)} \\in \\mathcal{C}$ and $\\tilde{\\vg}_t = \\vg_t+\\max\\{0, -\\frac{1}{\\gamma_t}\\langle \\vg_t, \\vw_t \\rangle\\} \\vs_t = \\vg_t+\\max\\{0, -(1+\\delta)\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t$. \n  %\n  Following similar arguments as in \\textbf{Option I}, we have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle$. Moreover, for any $\\vu\\in \\mathcal{C}$, \n  \\begin{align*}\n    \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle &= \\langle\\vg_t+\\max\\{0, -(1+\\delta)\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t, \\vw_t-\\vu \\rangle \\\\\n    & = \\langle \\vg_t, (1+\\delta)\\gamma_t\\vx_t-\\vu \\rangle + \\max\\{0, -(1+\\delta)\\langle \\vg_t, \\vx_t \\rangle\\} \\langle \\vs_t, \\vw_t-\\vu\\rangle \\\\\n    %\n    %\n    & \\geq \\langle \\vg_t, (1+\\delta)\\vx_t-\\vu\\rangle, \n  \\end{align*}\n  where we used $\\langle \\vs_t, \\vw_t-\\vu\\rangle \\geq \\gamma_t -1$ in the first inequality. By rearranging the above inequality, we obtain that $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle \\leq \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle - \\delta \\langle \\vg_t, \\vx_t \\rangle$, which proves~\\eqref{eq:regret_reduction_II_1}.  \n  Also, by the triangle inequality we obtain \n  \\begin{equation*}\n    \\|\\tilde{\\vg}_t\\|_2 =\\|\\vg_t+\\max\\{0, -(1+\\delta)\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t\\|_2 %\n    \\leq  \\|\\vg_t\\|_2+(1+\\delta)|\\langle \\vg_t, \\vx_t \\rangle|\\|\\vs_t\\|_2,\n  \\end{equation*}\n  which proves \\eqref{eq:surrogate_loss_II}. \n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n\\end{itemize}\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\nFinally, the second inequality in \\eqref{eq:regret_reduction_II_2} can be shown similarly as in \\textbf{Option I}. \n }\n\\end{proof}","sha256_utf8":"a221e01134a1df26426c6875f90be06934c89404e75747b01b832abe00f58ea3"},{"start":116561,"end":117450,"start_line":2061,"end_line":2070,"raw_tex":"\\begin{lemma}\\label{lem:sep_intersect}\n  Assume that $\\mathcal{C} = \\mathcal{C}_1 \\cap \\mathcal{C}_2 \\cap \\mathcal{L}$, where $0 \\in \\calC_1 \\cap \\calC_2$ and $\\mathcal{L}= \\vspan(\\calC)$ is a linear subspace. Suppose we have access to $\\mathsf{SEP}_{\\mathcal{C}_1}(\\vw; \\delta)$, $\\mathsf{SEP}_{\\mathcal{C}_2}(\\vw; \\delta)$ and the orthogonal projection matrix $\\mP_{\\calL}$ associated with the subspace $\\calL$. Then given inputs $\\vw \\in \\calL$ and $\\delta>0$, the oracle $\\mathsf{SEP}_{\\mathcal{C}}(\\vw; \\delta)$ can be constructed in the following ways: \n  \\begin{enumerate}[(a)]\n  %\n    \\item  let $(\\gamma_1,\\vs_1) = \\mathsf{SEP}_{\\mathcal{C}_1}(\\vw; \\delta)$ and $(\\gamma_2,\\vs_2) = \\mathsf{SEP}_{\\mathcal{C}_2}(\\vw; \\delta)$.\n    %\n    \\item Let $i = \\argmax_{i \\in \\{1,2\\}} \\gamma_i$ and then output $(\\gamma,\\vs) = (\\gamma_i,\\mP_{\\calL}\\vs_i)$.\n  \\end{enumerate}\n  %\n\\end{lemma}","sha256_utf8":"d7186e88f8604530cf37f2c4ec9cea56d72720809cb29631a5286538a0339ec5"},{"start":117451,"end":119611,"start_line":2071,"end_line":2082,"raw_tex":"\\begin{proof}\nWithout loss of generality, we assume $\\gamma_1 \\geq \\gamma_2$ and thus the output is given by $(\\gamma,\\vs) = (\\gamma_1,\\mP_{\\calL}\\vs_1)$. We consider two cases depending on whether $\\gamma_1 \\leq 1$ or not. \n\nIn the first case where $\\gamma = \\gamma_1 \\leq 1$, we have $\\gamma_2 \\leq \\gamma_1 \\leq 1$. According to the definitions of $\\SEP_{\\calC_1}(\\vw;\\delta)$ and $ \\SEP_{\\calC_2}(\\vw;\\delta)$, we have $ \\vw \\in (1+\\delta)\\calC_1$ and $\\vw \\in (1+\\delta)\\calC_2$. This implies that $\\vw \\in (1+\\delta) (\\calC_1 \\cap \\calC_2)$. Moreover, since we assume that $\\vw \\in \\calL$, this leads to $ \\vw \\in (1+\\delta) (\\calC_1 \\cap \\calC_2 \\cap \\calL) = (1+\\delta) \\calC$. \n\nIn the second case where $ \\gamma = \\gamma_1 > 1$, according to the definition of $\\SEP_{\\calC_1}(\\vw;\\delta)$ we have $\\vw/\\gamma_1 \\in (1+\\delta)\\calC_1$ and $ \\langle \\vs_1, \\vw-\\vx \\rangle \\geq \\gamma_1 - 1$ for all $\\vx \\in \\calC_1$. Moreover, by the definition of $\\SEP_{\\calC_2}(\\vw;\\delta)$, we have $\\vw/\\gamma_2 \\in (1+\\delta)\\calC_2$.\nFirst, we show that $\\vw/\\gamma \\in (1+\\delta)\\calC$. \nSince $\\gamma_2 \\leq \\gamma_1$, we can write $ \\vw/\\gamma_1$ as a convex combination of the origin and $\\vw/\\gamma_2$, and thus it follows from the convexity of $\\calC_2$ that $ \\vw/\\gamma_1 \\in (1+\\delta) \\calC_2$. Similar to the arguments in the first case, we obtain $\\vw/\\gamma =  \\vw/\\gamma_1 \\in (1+\\delta) \\calC$. \nNext, we will prove that $\\langle \\vs, \\vw-\\vx \\rangle \\geq \\gamma-1$.  Since $\\calC_1 \\subset \\calC$, we have $\\vx \\in \\calC_1$ for any $\\vx \\in \\calC$. Therefore,  it implies that $\\langle \\vs_1, \\vw - \\vx \\rangle \\geq \\gamma_1-1 = \\gamma-1$. Moreover, since $\\vw \\in \\calL$ and $\\vx \\in \\calL$, we have $\\mP_{\\calL}(\\vw-\\vx) = \\vw-\\vx$. Therefore, we conclude that $\\langle \\vs, \\vw-\\vx \\rangle = \\langle \\mP_{\\calL}\\vs_1, \\vw-\\vx \\rangle = \\langle \\vs_1, \\mP_{\\calL}(\\vw-\\vx) \\rangle = \\langle \\vs_1, \\vw-\\vx \\rangle \\geq \\gamma_1 -1 = \\gamma-1$. \n\nCombining both cases, we observe that the procedure described in Lemma~\\ref{lem:sep_intersect} indeed satisfies the definition in Definition~\\ref{def:gauge}. This completes the proof. \n\\end{proof}","sha256_utf8":"7e006391bd771e7d5dee56d86fdd8ecc2cd0c41949c348053ca14df0b4f6bd73"},{"start":124876,"end":125639,"start_line":2217,"end_line":2223,"raw_tex":"\\begin{lemma}\\label{lem:transform}\n   Recall the set $\\calZ$ defined in \\eqref{eq:feasible_set} and suppose that the linear subspace $\\calL$ satisfies $\\mI \\in \\calL$.  Define the set $\\calC$ as the following: \n   \\begin{equation}\\label{eq:translated_set}\n  {\\mathcal{C}} \\mydef \\left\\{\\hat{\\mB} \\in \\reals^{d\\times d}: -\\mI \\preceq \\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top ) \\preceq \\mI , \\ \\|\\hat{\\mB}\\|_{\\op} \\leq 3, \\ \\hat{\\mB} \\in \\calL \\right\\}. \n  \\end{equation}\n    Let $\\hat{\\mB} \\mydef \\frac{1}{L_1}(\\mB-(L_1+\\mu)\\mI)$. If $\\mB \\in \\calZ$, then $\\hat{\\mB} \\in \\mathcal{C}$. Conversely, if $\\hat{\\mB} \\in (1+\\delta)\\mathcal{C}$, then  $\\frac{1}{2}(\\mB + \\mB^\\top )\\succeq (\\mu -L_1\\delta) \\mI$ and $\\|\\mB\\|_{\\op} \\leq 4L_1 + \\mu+ 3\\delta L_1$. \n\\end{lemma}","sha256_utf8":"692e52c0e0a2a3218755c120316ce6dee5fcb848ac4a062812a6f63ebf566f22"},{"start":125640,"end":127142,"start_line":2224,"end_line":2240,"raw_tex":"\\begin{proof}\n  First, we show that $\\mB\\in \\calZ$ implies $\\hat{\\mB} \\in \\calC$. By the definition of $\\hat{\\mB}$, we have  \n%\n$\\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top ) = \\frac{1}{2L_1}(\\mB + \\mB^\\top) - \\frac{L_1+\\mu}{L_1}\\mI \\succeq \\frac{\\mu}{L_1} \\mI - \\frac{L_1+\\mu}{L_1}\\mI = -\\mI$. \n%\nMoreover, we also have \n%\n$\\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top ) \\preceq  \\frac{1}{2L_1}(\\mB + \\mB^\\top) \\preceq \\mI$.\n%\nCombining these two, we obtain that $ -\\mI \\preceq \\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top ) \\preceq \\mI$. Additionally, we can bound $\\|\\hat{\\mB}\\|_{\\op} \\leq \\frac{1}{L_1} (\\|\\mB\\|_{\\op} + (L_1+\\mu)) \\leq 3$ since $\\|\\mB\\|_{\\op} \\leq L_1$ and $\\mu \\leq L_1$. Finally, note that $\\hat{\\mB}$ is a linear combination of $\\mB$ and $\\mI$. Since $\\mI \\in \\calL$ and $\\mB \\in \\calL$, we also have $\\hat{\\mB} \\in \\calL$. Hence, we conclude that $\\hat{\\mB} \\in \\calC$. \n\nFor the other direction, assume $\\hat{\\mB} \\in (1+\\delta)\\calC$ and note that $ \\mB = L_1 \\hat{\\mB} + (L_1+\\mu)\\mI$. Since $\\hat{\\mB} \\in (1+\\delta)\\calC$ implies that $\\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top) \\succeq -(1+\\delta)\\mI$, we have \n%\n$\\frac{1}{2}(\\mB + \\mB^\\top) \\succeq \\frac{L_1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top) + (L_1 + \\mu) \\mI \\succeq (\\mu-L_1 \\delta)\\mI$.\n%\nFinally, $\\hat{\\mB} \\in (1+\\delta)\\calC$ also implies that $\\|\\hat{\\mB}\\|_{\\op} \\leq 3(1+\\delta)$, and thus it holds that $\\|\\mB\\|_{\\op}  \\leq L_1\\|\\hat{\\mB}\\|_{\\op} + (L_1+\\mu) \\leq 4L_1 + \\mu+ 3\\delta L_1$. This completes the proof.  \n\\end{proof}","sha256_utf8":"1b7c54a0ab7dfcaaa80dcbb32176530d07c7ced9ed13ed446351a2eda271c709"},{"start":133406,"end":133888,"start_line":2314,"end_line":2315,"raw_tex":"The oracles in Definitions~\\ref{def:extevec} and~\\ref{def:MaxSvec} provide the approximate separation oracles for $\\calC_1$ and $\\calC_2$, respectively. Using these two building blocks, we can construct the approximate separation oracle for $\\calC$ by following the procedure in Lemma~\\ref{lem:sep_intersect}. \nBy instantiating Algorithm~\\ref{alg:projection_free_online_learning}, we obtain the complete Jacobian approximation update given in Subroutine~\\ref{alg:jacobian_approx}. \n","sha256_utf8":"922164f7f1dcc0e9b2ddf17a9d1f8cc7c289a07e1b606d58d5204654d502c654"},{"start":142284,"end":143426,"start_line":2492,"end_line":2506,"raw_tex":"\\begin{proposition}[{\\cite[Theorem~4.2]{kuczynski1992estimating}}]\\label{prop:lanczos}\n  Consider a symmetric matrix $\\mW$ and let $\\lambda_1(\\mW)$ and $\\lambda_d(\\mW)$ denote its largest and smallest eigenvalues, respectively. Then after $k$ iterations of the Lanczos method with a random start, we find unit vectors $\\vu^{(1)}$ and $\\vu^{(d)}$  such that \n  \\begin{align*}\n    \\mathbb{P}(\\langle\\mW \\vu^{(1)}, \\vu^{(1)}\\rangle\\leq \\lambda_1(\\mW)-\\epsilon(\\lambda_1(\\mW)-\\lambda_d(\\mW))) \\leq 1.648\\sqrt{d}e^{-\\sqrt{\\epsilon}(2k-1)},\\\\\n    \\mathbb{P}(\\langle\\mW \\vu^{(d)}, \\vu^{(d)}\\rangle\\geq \\lambda_d(\\mW)+\\epsilon(\\lambda_1(\\mW)-\\lambda_d(\\mW))) \\leq 1.648\\sqrt{d}e^{-\\sqrt{\\epsilon}(2k-1)},\n  \\end{align*}\n  As a corollary, to ensure that, with probability at least $1-q$, $ \\langle\\mW \\vu^{(1)}, \\vu^{(1)}\\rangle> \\lambda_1(\\mW)-\\epsilon(\\lambda_1(\\mW)-\\lambda_d(\\mW))$ and $\\langle\\mW \\vu^{(d)}, \\vu^{(d)}\\rangle< \\lambda_n(\\mW)+\\epsilon(\\lambda_1(\\mW)-\\lambda_d(\\mW))$,\n  %\n  %\n  %\n  %\n  %\n  %\n  the number of iterations can be bounded by  $\\lceil\\frac{1}{4}\\epsilon^{-1/2}\\log(\\frac{11d}{q^2})+\\frac{1}{2} \\rceil$.\n\\end{proposition}","sha256_utf8":"ab545e3db076077295efb9a9df4d463e88c3a06eacd8250a48c02bf57eb0a25b"},{"start":146984,"end":147272,"start_line":2544,"end_line":2550,"raw_tex":"\\begin{lemma}\\label{lem:extevec_bound}\n  Let $\\gamma$ and $\\mS$ be the output of $\\mathsf{ExtEvec}(\\mW;\\delta,q)$ in Subroutine~\\ref{alg:lanczos}. Then with probability at least $1-q$, they satisfy one of the properties given in Definition~\\ref{def:extevec}.  \n  %\n  %\n  %\n  %\n\\end{lemma}","sha256_utf8":"1afecf65dafe57856e9710cda6c9b631309923b54239bd76a117608e81e5182a"},{"start":147273,"end":151105,"start_line":2551,"end_line":2614,"raw_tex":"\\begin{proof}\n  Note that in Subroutine~\\ref{alg:lanczos}, we run the Lanczos method for $\\Bigl\\lceil \\frac{1}{4}\\epsilon^{-1/2}\\log\\frac{11d}{q^2}+\\frac{1}{2}\\Bigr\\rceil$ iterations, where $\\epsilon = \\frac{\\delta}{2(1+\\delta)}$. \n  Thus, by Proposition~\\ref{prop:lanczos}, with probability at least $1-q$ we have \n  \\begin{align}\n    \\hat{\\lambda}_1 \\triangleq \\langle\\bar{\\mW} \\vu^{(1)}, \\vu^{(1)}\\rangle \\geq  \\lambda_1(\\bar{\\mW})-\\frac{\\delta}{2(1+\\delta)}(\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})), \\label{eq:lambda_1}\\\\ \n    %\n    \\hat{\\lambda}_d \\triangleq \\langle\\bar{\\mW} \\vu^{(d)}, \\vu^{(d)}\\rangle \\leq  \\lambda_d(\\bar{\\mW})+\\frac{\\delta}{2(1+\\delta)}(\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})). \\label{eq:lambda_d}\n  \\end{align}\n  %\n  %\n  Combining \\eqref{eq:lambda_1} and \\eqref{eq:lambda_d}, we obtain $\\frac{\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})}{1+\\delta}\\leq \\hat{\\lambda}_1-\\hat{\\lambda}_d$, which further implies that $\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})\\leq (1+\\delta)(\\hat{\\lambda}_1-\\hat{\\lambda}_d)$. \n    %\n    %\n    %\n  By plugging the above inequality back into \\eqref{eq:lambda_1} and \\eqref{eq:lambda_d}, we further have \n  \\begin{align}\n    \\lambda_1(\\bar{\\mW}) \\leq \\hat{\\lambda}_1+\\frac{\\delta}{2(1+\\delta)}(\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})) \\leq \\hat{\\lambda}_1+\\frac{\\delta}{2}(\\hat{\\lambda}_1-\\hat{\\lambda}_d), \\label{eq:upper_bound_lambda_1} \\\\\n    \\lambda_d(\\bar{\\mW}) \\geq \\hat{\\lambda}_d-\\frac{\\delta}{2(1+\\delta)}(\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})) \\geq \\hat{\\lambda}_d-\\frac{\\delta}{2}(\\hat{\\lambda}_1-\\hat{\\lambda}_d). \\label{eq:lower_bound_lambda_d}\n  \\end{align}\n  Recall that $\\gamma = \\max\\{\\hat{\\lambda}_1,-\\hat{\\lambda}_d\\}$. By \\eqref{eq:upper_bound_lambda_1} and \\eqref{eq:lower_bound_lambda_d}, \n  we can further bound the eigenvalues of $\\bar{\\mW}$ by  \n  \\begin{equation}\n    \\lambda_1(\\bar{\\mW}) \\leq \\gamma +\\frac{\\delta}{2}\\cdot 2\\gamma ={(1+\\delta)\\gamma} \\quad \\text{and} \\quad \\lambda_d(\\bar{\\mW}) \\geq -\\gamma -\\frac{\\delta}{2}\\cdot 2\\gamma = -(1+\\delta){\\gamma}.\n  \\end{equation}\n  Hence, we can see that $-(1+\\delta)\\gamma \\mI\\preceq \\bar{\\mW} = \\frac{1}{2}(\\mW + \\mW^\\top) \\preceq (1+\\delta)\\gamma \\mI$. \n  %\n  Now we distinguish three cases. \n  \\begin{enumerate}[(a)]\n    \\item If $\\gamma \\leq 1$, then we are in \\textbf{Case I} and the $\\mathsf{ExtEvec}$ oracle outputs $\\gamma$ and $\\mS = 0$. In this case, we indeed have $-(1+\\delta) \\mI\\preceq \\frac{1}{2}(\\mW + \\mW^\\top) \\preceq (1+\\delta) \\mI$. \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    \\item If $\\gamma >1$ and $\\hat{\\lambda}_1 \\geq -\\hat{\\lambda}_d$, then we are in \\textbf{Case II} and the $\\mathsf{ExtEvec}$ oracle returns $\\gamma $ and $\\mS = \\vu^{(1)}(\\vu^{(1)})^\\top$. In this case, since $\\max\\{\\lambda_1(\\bar{\\mW}),-\\lambda_d(\\bar{\\mW})\\} \\leq \\gamma(1+\\delta)$, we have $-(1+\\delta) \\mI\\preceq \\frac{1}{2\\gamma}(\\mW + \\mW^\\top) \\preceq (1+\\delta) \\mI$. Also, since $\\vu^{(1)}$ is a unit vector, we have $\\|\\mS\\|_F = \\|\\vu^{(1)}\\|^2 = 1$. Finally, for any $\\hat{\\mB}$ such that $-\\mI \\preceq \\frac{1}{2}(\\hat{\\mB}+\\hat{\\mB}^\\top) \\preceq \\mI$, \n    %\n    we have \n    %\n    $\\langle \\mS,\\mW-\\hat{\\mB}\\rangle = \\vu_1^\\top{\\mW} \\vu_1-\\vu_1^\\top \\hat{\\mB} \\vu_1 = \\vu_1^\\top{\\bar{\\mW}} \\vu_1 -\\frac{1}{2}\\vu_1^\\top (\\hat{\\mB}+\\hat{\\mB}^\\top) \\vu_1 \\geq \\hat{\\lambda}_1-1 = \\gamma-1$. Thus, $\\gamma$ and $\\mS$ satisfy all the properties in \\textbf{Case II}. \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    \\item If $\\gamma >1$ and $-\\hat{\\lambda}_d \\geq \\hat{\\lambda}_1$, then we are also in \\textbf{Case II} and the $\\mathsf{ExtEvec}$ oracle returns $\\gamma$ and $\\mS = -\\vu^{(d)}(\\vu^{(d)})^\\top$. The rest follows similarly to the case above.\n    %\n    %\n    %\n    %\n    %\n  \\end{enumerate}\n  This completes the proof.\n\\end{proof}","sha256_utf8":"3b6c01e2a7f0c56f18271af17f2b5f0698234a69667e1d4d0abf3d4c46867769"},{"start":151879,"end":152856,"start_line":2618,"end_line":2631,"raw_tex":"\\begin{lemma}[{\\cite[Section 8.6.1]{Golub2013Matrix}}]\\label{lem:singular_to_eigen}\n  Let $\\mW\\in \\reals^{d\\times d}$ be an arbitrary matrix. Define the augmented matrix $\\tilde{\\mW}$ by \n  \\begin{equation}\\label{eq:augmented}\n    \\tilde{\\mW} = \\begin{bmatrix} 0 & \\mW \\\\ \\mW^\\top & 0\n    \\end{bmatrix} \\in \\mathbb{S}^{2d}. \n  \\end{equation}\n  %\n  \\begin{itemize}\n    \\item Let $\\lambda_1(\\tilde{\\mW})$ and $\\lambda_{2d}(\\tilde{\\mW})$ denote its largest and smallest eigenvalues, respectively. Then $\\lambda_1(\\tilde{\\mW})$ is the maximal singular value of $\\mW$ and $\\lambda_{2d}(\\tilde{\\mW}) = -\\lambda_1(\\tilde{\\mW})$. \n    \\item Let $\\tilde{\\vv} \\in \\reals^{2d}$ denote the eigenvector associated with $\\lambda_1(\\tilde{\\mW})$ and partition it as $\\tilde{\\vv} = (\\vv,\\vv')$, where $\\vv,\\vv' \\in \\reals^d$. Then ${\\vv}$ and ${\\vv'}$ are the left and right singular vectors associated with the maximal singular value of $\\mW$, respectively.\n  \\end{itemize}\n  \n  %\n\\end{lemma}","sha256_utf8":"7df775f8601729cc58f5259777ad9ce0e1b1aeec1470e34e43948dd4c35984e8"},{"start":154612,"end":154905,"start_line":2673,"end_line":2681,"raw_tex":"\\begin{lemma}\\label{lem:maxsvec_bound}\n  Let $\\gamma$ and $\\mS$ be the output of $\\mathsf{MaxSvec}(\\mW;\\delta,q)$ in Subroutine~\\ref{alg:lanczos_singular}.  %\n  %\n  Then with probability at least $1-q$, they satisfy the properties in Definition~\\ref{def:MaxSvec}.  \n  %\n  %\n  %\n  %\n\\end{lemma}","sha256_utf8":"ef5366a927b51e144b30d9cbf9cfcf00df054415f7fa8ebe95edf1a5280b7525"},{"start":154906,"end":157584,"start_line":2682,"end_line":2747,"raw_tex":"\\begin{proof}\n  %\n  %\n  %\n  %\n  %\n  Note that in Subroutine~\\ref{alg:lanczos_singular}, we run the Lanczos method for $\\Bigl\\lceil \\frac{1}{4}\\epsilon^{-1/2}\\log\\frac{22d}{q^2}+\\frac{1}{2}\\Bigr\\rceil$ iterations, where $\\epsilon = \\frac{\\delta}{2(1+\\delta)}$. \n  Thus, by Proposition~\\ref{prop:lanczos}, with probability at least $1-q$ we can find a unit vector $\\tilde{\\vv} \\in \\reals^{2d}$ such that \n  %\n  $\\tilde{\\lambda}_1 \\triangleq \\langle \\tilde{\\mW} \\tilde{\\vv}, \\tilde{\\vv}\\rangle \\geq  \\lambda_1(\\tilde{\\mW})-\\frac{\\delta}{2(1+\\delta)}(\\lambda_1(\\tilde{\\mW})-\\lambda_{2d}(\\tilde{\\mW}))$, %\n    %\n    %\n  %\n  %\n  %\n  Moreover, notice that $\\lambda_{2d}(\\tilde{\\mW}) = -\\lambda_1(\\tilde{\\mW})$ by Lemma~\\ref{lem:singular_to_eigen}. Hence, the above further implies that $\\tilde{\\lambda}_1 \\geq \\frac{1}{1+\\delta} \\lambda_1(\\tilde{\\mW})$, which is equivalent to $\\lambda_1(\\tilde{\\mW}) \\leq (1+\\delta)\\tilde{\\lambda}$. \n  %\n  %\n  %\n  Moreover, note that $\\lambda_1(\\tilde{\\mW}) = \\|\\mW\\|_{\\op}$ and $\\gamma = \\tilde{\\lambda}_1/3$. Hence, the above is further equivalent to $\\|\\mW\\|_{\\op} \\leq 3(1+\\delta)\\gamma$. \n  %\n  %\n  %\n  %\n  %\n  %\n  Now we distinguish two cases. \n  \\begin{enumerate}[(a)]\n    \\item If $\\gamma \\leq 1$, then we are in \\textbf{Case I} and the $\\mathsf{MaxSvec}$ oracle outputs $\\gamma$ and $\\mS = 0$. In this case, we indeed have $\\|\\mW\\|_\\op \\leq  3(1+\\delta)\\gamma \\leq 3(1+\\delta)$. \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    \\item If $\\gamma >1$, then we are in \\textbf{Case II} and the $\\mathsf{MaxSvec}$ oracle returns $\\gamma $ and $\\mS = \\frac{2}{3}\\vv'\\vv^\\top$. \n    In this case, since $\\|\\mW\\|_\\op \\leq 3(1+\\delta)\\gamma$, we have $\\|\\mW/\\gamma\\|_{\\op} \\leq 3(1+\\delta)$. Also, since $\\tilde{\\vv}$ is a unit vector, we have $\\|\\mS\\|_F = \\frac{2}{3} \\|\\vv\\|\\|\\vv'\\| \\leq \\frac{1}{3}(\\|\\vv\\|^2 + \\|\\vv'\\|^2) = \\frac{1}{3}\\|\\tilde{\\vv}\\|^2 =\\frac{1}{3}$. Finally, it holds that $\\langle \\mS, \\mW \\rangle = \\frac{2}{3} \\vv^\\top \\mW \\vv' = \\frac{1}{3}\\tilde{\\vv}^\\top \\tilde{\\mW} \\tilde{\\vv} = \\frac{\\tilde{\\lambda}_1}{3} = \\gamma$. Moreover, for any $\\hat{\\mB}$ such that $\\|\\hat{\\mB}\\|_\\op \\leq 3$, we have\n    %\n    $\\langle \\mS,\\hat{\\mB}\\rangle =  \\frac{2}{3}\\vv^\\top \\hat{\\mB} \\vv' \\leq \\frac{2}{3}\\|\\vv\\| \\|\\mB\\|_{\\op} \\|\\vv'\\| \\leq 2\\|\\vv'\\| \\|\\vv\\| \\leq \\|\\vv\\|^2 + \\|\\vv'\\|^2 = 1$. \n    %\n    Hence, we obtain $\\langle \\mS,\\mW-\\hat{\\mB}\\rangle \\geq \\gamma -1$. This shows that $\\gamma$ and $\\mS$ satisfy all the properties in \\textbf{Case II}. \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n  \\end{enumerate}\n  This completes the proof.\n\\end{proof}","sha256_utf8":"3974313989a3db634e6ef7a63a3c3ed20a39b7690867166b0a11ec578a4c4940"},{"start":161821,"end":162646,"start_line":2832,"end_line":2841,"raw_tex":"Now we move to the proof of Theorem~\\ref{thm:main}. First of all, by using a simple union bound, we can show that \n the  $\\mathsf{ExtEvec}$ and $\\mathsf{MaxSvec}$ oracles used in Subroutine~\\ref{alg:jacobian_approx} are successful in all rounds with probability at least $1-p$. \n %\n Specifically, \nsince both  oracles have a failure probability of $ \\frac{q_t}{2} = \\frac{p}{5(t+1)\\log^2(t+1)}$ in the $t$-th round, we can use the union bound to upper bound the total failure probability by \n\\begin{equation*}\n  \\sum_{t=1}^{T-1} q_t %\n   \\leq  \\frac{p}{2.5}\\sum_{t=2}^{\\infty} \\frac{1}{t \\log^2 t} \\leq \\frac{p}{2.5}\\left(\\frac{1}{2\\log^2 2}+\\int_{2}^{+\\infty}\\frac{1}{t\\log^2t}\\,dt\\right) \\leq p.\n\\end{equation*}\n Thus, throughout the proof, we assume every call of $\\mathsf{ExtEvec}$ and $\\mathsf{MaxSvec}$ is successful. \n","sha256_utf8":"4c24c444a630ae21074f635ba6f53cb25f519bb551ef6f12bd2d68e43fd60b8b"},{"start":162649,"end":165988,"start_line":2844,"end_line":2886,"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm:main}(a)]\n  \n\nWe first prove the linear convergence rate in (a). To begin with, note that Subroutine~\\ref{alg:jacobian_approx} with \\textbf{Option I} guarantees that  $\\hat{\\mB}_k \\in (1+\\frac{\\mu}{2L_1}) \\mathcal{C}$ by Lemma~\\ref{lem:regret_reduction}.  As a corollary of Lemma~\\ref{lem:transform}, we have the following result showing the boundedness of the Jacobian approximation matrix $\\mB_k$. \n\n%\n\n \n\\begin{corollary}\\label{lem:feasible_Bk}\n  Let $\\delta_t = \\frac{\\mu}{2L_1}$ for all $t\\geq 0$. Then for any $k \\geq 0$, we have $\\frac{1}{2}(\\mB_k + \\mB_k^\\top )\\succeq \\frac{\\mu}{2} \\mI$ and $\\|\\mB_k\\|_{\\op} \\leq 6.5L_1$. \n\\end{corollary}\nCombining this with Lemma~\\ref{lem:step size_lb}, we obtain the following universal lower bound on the step size $\\eta_k$. \n\\begin{lemma}\\label{lem:stepsize_const_bound}\n  %\n  In Subroutine~\\ref{alg:ls}, choose $\\sigma_0 \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$. For any $k\\geq 0$, we have $\\eta_k \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$. \n\\end{lemma}\n\\begin{proof}\n  Recall that $\\calB = \\{k: \\eta_k < \\sigma_k\\}$.\n  We first establish that $\\eta_k \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$ for $k\\in \\mathcal{B}$. To see this, suppose $k\\in \\mathcal{B}$ and recall from Lemma~\\ref{lem:step size_lb} that\n\\begin{equation}\\label{eq:step size_lb_appen}\n  \\eta_k > \\frac{\\alpha_2 \\beta\\|{\\tilde{\\vz}_k}-\\vz_k\\|}{\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\|}.\n\\end{equation}\n%\n  %\n  %\n  %\n  Moreover, by using the triangle inequality, we have \n  %\n  $\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\| \\leq \\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)\\| + \\|\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\|$.\n  %\n  By using Assumption~\\ref{assum:operator_lips}, we have $\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)\\| \\leq L_1\\|\\tilde{\\vz}_k-\\vz_k\\|$. Moreover, since $\\|\\mB_k\\|_{\\op} \\leq 6.5L_1$ by Corollary~\\ref{lem:feasible_Bk}, we also get $\\|\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\| \\leq 6.5 \\|{\\tilde{\\vz}_k}-\\vz_k\\|$. \n  %\n  Thus, this implies that $\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\| \\leq 7.5\\|{\\tilde{\\vz}_k}-\\vz_k\\|$,  \n  %\n  %\n  %\n  which leads to $\\eta_k > \\frac{\\alpha_2\\beta}{7.5L_1}$ from \\eqref{eq:step size_lb_appen}. \n\n  Now we can prove that $\\eta_k \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$ for all $k\\geq 0$ by induction. To show that this holds for $k=0$, we distinguish two cases. If $0\\notin \\mathcal{B}$, then we have $\\eta_0 = \\sigma_0 > \\frac{\\alpha_2\\beta}{7.5L_1}$ by our choice of $\\sigma_0$. Otherwise, if $0\\in \\mathcal{B}$, then it directly follows from our result in the previous paragraph. Moreover, assume that $\\eta_{l-1} \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$ where $l\\geq 1$. Similarly, we again distinguish two cases: if $l\\notin \\mathcal{B}$, then we have $\\eta_l = \\sigma_l = \\frac{\\eta_{l-1}}{\\beta} >\\frac{\\alpha_2}{7.5L_1}>\\frac{\\alpha_2\\beta}{7.5L_1}$; otherwise, if $l\\in \\mathcal{B}$, it follows from the result above that $\\eta_l \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$. This completes the induction. \n\\end{proof}\nIn light of Lemma~\\ref{lem:stepsize_const_bound}, it follows from Proposition~\\ref{prop:HPE}(a) that  $\\|\\vz_{k+1}-\\vz^*\\|^2 \\leq \\|\\vz_k-\\vz^*\\|^2 (1+2\\eta_k \\mu)^{-1} \\leq \\|\\vz_k-\\vz^*\\|^2 \\left(1+\\frac{4\\alpha_2\\beta\\mu}{15L_1}\\right)^{-1}$. \nThis proves the linear convergence result in (a). \n\\end{proof}","sha256_utf8":"8423e5d994c1c0e719d8d9f101e4a0264e41702f301aa5d365951b38ad88ea1f"},{"start":165990,"end":166547,"start_line":2888,"end_line":2896,"raw_tex":"Next, we prove the superlinear convergence rate in (b). Our starting point is the observations in \\eqref{eq:after_jensen} and \\eqref{eq:goal}, which demonstrate that it is sufficient to prove an upper bound on the cumulative loss $\\sum_{k=0}^{N-1} \\ell_k(\\mB_k)$. \n%\n%\nMoreover, since $\\ell_k(\\mB_k) = 0$ when $k \\notin \\calB$ by the definition in \\eqref{eq:loss_of_jacobian}, it is equivalent to bounding the sum $\\sum_{k \\in \\calB} \\ell_k(\\mB_k)$.  \n%\n%\n\\blue{Recall that we relabel the indices in $\\mathcal{B}$ by $t=0,\\dots,T-1$ with $T \\leq N$. %\n%\n} \n","sha256_utf8":"2fd3d39fb9b5521500733e26ef36522e5dde38106848392dcb039479c8cb0335"},{"start":166581,"end":178460,"start_line":2915,"end_line":3111,"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm:main}(b)] We consider the following three steps.\n\n\\vspace{0em}\\noindent\\textbf{Step 1:} Using regret analysis, we bound the cumulative loss $\\sum_{t=0}^{T-1}\\ell_{t}(\\mB_{t})$ incurred by our online learning algorithm in Subroutine~\\ref{alg:jacobian_approx}. \nTo begin with, we present the following lemma showing a self-bounding property of the loss function $\\ell_t$. {It is similar to the standard inequality $\\frac{1}{2L_1} \\|\\nabla g (\\vx)\\|^2 \\leq g(\\vx) - g^* $ for a $L_1$-smooth function $g$, where $g^*$ denotes the minimum of $g$.} This will be the key to proving a constant upper bound on the cumulative loss incurred by Subroutine~\\ref{alg:jacobian_approx}.\n\\begin{lemma}\\label{lem:loss}\n  Recall the loss function $\\ell_k$ defined in \\eqref{eq:loss_of_jacobian}. For $k\\in \\mathcal{B}$, we have\n  \\begin{equation}\\label{eq:gradient}\n    \\nabla \\ell_k(\\mB) = - \\frac{2(\\vu_k-\\mB\\vs_k)\\vs_k^\\mathsf{T}}{\\|\\vs_k\\|^2}.\n  \\end{equation}\n  Moreover, for any $\\mB\\in \\reals^{d\\times d}$, it holds that \n    \\begin{equation}\\label{eq:gradient_norm_bound}\n        \\|\\nabla \\ell_k(\\mB)\\|_F \\leq \\|\\nabla \\ell_k(\\mB)\\|_* = 2\\sqrt{\\ell_k(\\mB)},\n    \\end{equation} \n    where $\\|\\cdot\\|_F$ and $\\|\\cdot\\|_{*}$ denote the Frobenius norm and the nuclear norm, respectively.  \n\\end{lemma}\n\\begin{proof}\n  %\n  The expression in \\eqref{eq:gradient} follows from the direct calculation. The first inequality in \\eqref{eq:gradient_norm_bound} follows from the fact that $\\|\\mA\\|_F \\leq \\|\\mA\\|_*$ for any matrix $\\mA\\in \\reals^{d\\times d}$. For the equality, note that \n  \\begin{align*}\n    \\|\\nabla \\ell_k(\\mB)\\|_*   = \\frac{2}{\\|\\vs_k\\|^2}  \\|\\vu_k-\\mB\\vs_k\\|\\|\\vs_k\\|  = \\frac{2\\|\\vu_k-\\mB\\vs_k\\|}{\\|\\vs_k\\|} = 2\\sqrt{\\ell_k(\\mB)}, \n  \\end{align*}\n  where we used the fact that the rank-one matrix $\\va\\vb^\\top$ has only one nonzero singular value $\\|\\va\\|\\|\\vb\\|$. \n\\end{proof}\n\n\n%\nIn particular, by exploiting the self-bounding property of the loss function $\\ell_t$, we prove a ``small-loss bound'' \\cite{srebro2010smoothness} in the following lemma, where the cumulative loss of the learner is bounded by that of a fixed action in the competitor set. \n%\n  \\begin{lemma}\n  \\label{lem:small_loss}\n  %\n  %\n  In Subroutine~\\ref{alg:jacobian_approx}, choose \\textbf{Option I} with $\\rho = \\frac{1}{121}$ and $\\delta = \\frac{\\mu}{2L_1}$. \n  For any ${\\mH}\\in \\mathcal{{Z}}$, we have\n  %\n    $\\sum_{t=0}^{T-1} \\ell_t({\\mB}_t)  \\leq 121\\|\\mB_0-{\\mH}\\|_F^2+ 2 \\sum_{t=0}^{T-1} \\ell_t({\\mH})$.\n  %\n\\end{lemma}\n\n\\begin{proof}\nRecall that $\\calZ$ is contained in the linear subspace $\\calL$ (see \\eqref{eq:feasible_set}) and $\\mP$ denotes the orthogonal projection matrix associated with $\\calL$.    \nBy letting $\\vx_t = \\hat{\\mB}_t$, $\\vu = \\hat{\\mH} \\triangleq \\frac{1}{L_1}(\\mH-(L_1+\\mu)\\mI)$, \n%\n$\\vg_t = \\mG_t \\triangleq \\frac{1}{L_1}\\mP \\nabla \\ell_t(\\mB_t)$, $\\tilde{\\vg}_t = \\tilde{\\mG}_t$, $\\vw_t = \\mW_t$ in Lemma~\\ref{lem:regret_reduction}, we obtain:  \n\\begin{enumerate}[(i)]\n  \\item $\\hat{\\mB}_t \\in (1+\\delta)\\mathcal{C}$, which means $\\|\\hat{\\mB}_t\\|_{\\op} \\leq 3(1+\\delta)\\leq 4.5$ since $\\delta = \\frac{\\mu}{2L_1}\\leq \\frac{1}{2}$.\n  \\item It holds that \n  \\begin{align}\n    \\langle \\mG_t, \\hat{\\mB}_t-\\hat{\\mH} \\rangle &\\leq \\frac{1}{2\\rho}\\|\\mW_t-\\hat{\\mH}\\|_F^2-\\frac{1}{2\\rho}\\|\\mW_{t+1}-\\hat{\\mH}\\|_F^2+\\frac{\\rho}{2}\\|\\tilde{\\mG}_t\\|_F^2, \\label{eq:linearized_loss_matrix} \\\\%\\\\\n    %\n    \\|\\tilde{\\mG}_t\\|_F &\\leq \\|\\mG_t\\|_F + |\\langle \\mG_t, \\hat{\\mB}_t\\rangle|\\|\\mS_t\\|_F. \\label{eq:surrogate_gradient_bound}\n  \\end{align}\n\\end{enumerate}\n%\n%\n%\n%\n%\n%\n%\n%\n%\nFirst, note that $\\|\\mS_t\\|_F \\leq 1$ by Definition~\\ref{def:extevec} and\n%\n$|\\langle \\mG_t, \\hat{\\mB}_t\\rangle| \\leq \\|\\mG_t\\|_* \\|\\hat{\\mB}_t\\|_{\\op} \\leq 4.5 \\|\\mG_t\\|_*$. %\n%\nTogether with \\eqref{eq:surrogate_gradient_bound}, we get \n\\begin{equation}\\label{eq:surrogate_bound_loss}\n\\|\\tilde{\\mG}_t\\|_F \\leq \\|\\mG_t\\|_F+4.5\\|\\mG_t\\|_* \\leq 5.5\\|\\mG_t\\|_* \\leq \\frac{5.5}{L_1}\\|\\nabla \\ell_t(\\mB_t)\\|_* \\leq \\frac{11}{L_1}\\sqrt{\\ell_t({\\mB}_t)},\n\\end{equation}\nwhere we used $\\mG_t = \\frac{1}{L_1}\\mP \\nabla \\ell_t(\\mB_t)$ and Lemma~\\ref{lem:loss} in the last inequality. \nFurthermore, since $\\ell_t$ is convex and $\\mB_t,\\mH \\in \\calL$, we have \n\\begin{equation}\\label{eq:convexity_subspace}\n    \\ell_t({\\mB}_t) - \\ell_t(\\mH) \\!\\leq\\! \\langle \\nabla \\ell_t(\\mB_t), {\\mB}_t-\\mH \\rangle \\!=\\! \\langle \\nabla \\ell_t(\\mB_t), \\mP\\left({\\mB}_t-\\mH\\right) \\rangle \\!=\\! L_1^2 \\langle \\mG_t, \\hat{\\mB}_t-\\hat{\\mH} \\rangle,\n\\end{equation}\nwhere we used $\\mG_t = \\frac{1}{L_1}\\mP\\nabla \\ell_t(\\mB_t)$, $\\hat{\\mB}_t \\triangleq \\frac{1}{L_1}(\\mB_t-{(L_1+\\mu)}\\mI)$, and $\\hat{\\mH} \\triangleq \\frac{1}{L_1}(\\mH-{(L_1+\\mu)}\\mI)$. \nTherefore, by \\eqref{eq:linearized_loss_matrix} and \\eqref{eq:surrogate_bound_loss} we get \n\\begin{align*}\n\\ell_t({\\mB}_t) - \\ell_t(\\mH) &\\leq \\frac{L_1^2}{2\\rho}\\|\\mW_t-\\hat{\\mH}\\|_F^2-\\frac{L_1^2}{2\\rho}\\|\\mW_{t+1}-\\hat{\\mH}\\|_F^2+\\frac{\\rho L_1^2}{2} \\|\\tilde{\\mG}_t\\|_F^2\\\\\n&\\leq \\frac{L_1^2}{2\\rho}\\|\\mW_t-\\hat{\\mH}\\|_F^2-\\frac{L_1^2}{2\\rho}\\|\\mW_{t+1}-\\hat{\\mH}\\|_F^2+60.5\\rho\\ell_t({\\mB}_t). %\n%\n\\end{align*}\nSince $\\rho = \\frac{1}{121}$, by rearranging and simplifying terms in the above inequality, we obtain\n%\n%\n%\n$\\ell_t({\\mB}_t) \\leq 2\\ell_t({\\mH})+ {121 L_1^2}\\|\\mW_t-\\hat{\\mH}\\|_F^2-{121 L_1^2}\\|\\mW_{t+1}-\\hat{\\mH}\\|_F^2$.\n%\nBy summing the above inequality from $t=0$ to $T-1$, we further have \n\\begin{equation*}\n  \\sum_{t=0}^{T-1} \\ell_t({\\mB}_t)  \\leq {121 L_1^2}\\|\\mW_0-\\hat{\\mH}\\|_F^2+ 2 \\sum_{t=0}^{T-1} \\ell_t({\\mH}) = 121\\|\\mB_0-{\\mH}\\|_F^2+ 2 \\sum_{t=0}^{T-1} \\ell_t({\\mH}),\n\\end{equation*}\nwhere the last equality is due to ${\\mW}_0 \\triangleq \\frac{1}{L_1}(\\mB_0-{(L_1+\\mu)}\\mI)$ and $\\hat{\\mH} \\triangleq \\frac{1}{L_1}(\\mH-{(L_1+\\mu)}\\mI)$. This completes the proof. \n\\end{proof}\n\n%\nNote that in Lemma~\\ref{lem:small_loss}, we have the freedom to choose any competitor $\\mH$ in the set $\\mathcal{Z}$. To further obtain an explicit bound, a natural choice would be $\\mH^* \\triangleq \\nabla \\vF(\\vz^*)$, which leads to our next step.\n\n%\n\n\\vspace{0em}\\noindent \\textbf{Step 2:} We upper bound the cumulative loss $\\sum_{t=0}^{T-1} \\ell_t({\\mH}^*)$. The proof relies crucially on Assumption~\\ref{assum:jacobian_lips} as well as the linear convergence result we proved in (a).\nTo begin with, we prove the following useful lemma. \n\\begin{lemma}\\label{lem:sum_of_squares}\n If Assumption~\\ref{assum:strong_monotone} holds, then $\\sum_{k=0}^{N-1} \\|\\hat{\\vz}_{k}\\!-\\!\\vz_k\\|^2 \\leq \\frac{1}{1-(\\alpha_1+\\alpha_2)}\\|\\vz_0\\!-\\!\\vz^*\\|^2$. \n\\end{lemma}\n\\begin{proof}\n  Recall the inequality~\\eqref{eq:one_step_strongly} in the proof of Proposition~\\ref{prop:HPE}. Since $\\frac{1}{2}\\|\\vz_{k+1}-\\vz^*\\|^2 \\!\\leq \\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1}-\\vz^*\\|^2$, we can further derive that \n  \\begin{equation}\\label{eq:one_step_strongly_coro}\n    \\frac{1-\\alpha_1-\\alpha_2}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2) \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1}{2}\\|\\vz_{k+1} - \\vz^*\\|^2.\n\\end{equation} \nSince $\\alpha_1+\\alpha_2 < 1$, we further have $\\frac{1-\\alpha_1-\\alpha_2}{2} \\|\\hat{\\vz}_k - \\vz_k\\|^2 \\leq \\frac{1-\\alpha_1-\\alpha_2}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2) \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1}{2}\\|\\vz_{k+1} - \\vz^*\\|^2$. By summing the above inequality from $k=0$ to $k=N-1$, we obtain that \n\\begin{equation*}\n  \\frac{1-\\alpha_1-\\alpha_2}{2} \\sum_{k=0}^{N-1} \\|\\hat{\\vz}_k - \\vz_k\\|^2 \\leq \\frac{1}{2}\\|\\vz_0-\\vz^*\\|^2 - \\frac{1}{2}\\|\\vz_N-\\vz^*\\|^2 \\leq \\frac{1}{2}\\|\\vz_0-\\vz^*\\|^2. \n\\end{equation*}\nThus, we arrive at Lemma~\\ref{lem:sum_of_squares} by dividing both sides by $\\frac{1-\\alpha_1-\\alpha_2}{2}$. \n\\end{proof}\n\nWith the help of Lemma~\\ref{lem:sum_of_squares}, we are ready to upper bound $\\sum_{t=0}^{T-1} \\ell_t({\\mH}^*)$. \n  \\begin{lemma}\\label{lem:comparator}\n  Recall the constant $C$ defined in \\eqref{eq:def_c1}. \n  %\n  Then \n    %\n      %\n    %\n    %\n    %\n    %\n    $\\sum_{t=0}^{T-1} \\ell_t(\\mH^*) \\leq \\left(C+2+\\frac{15L_1}{2\\alpha_2 \\beta \\mu}\\right)L_2^2\\|\\vz_0-\\vz^*\\|^2$.\n    %\n  \\end{lemma}\n  \\begin{proof}\n    Recall the definition of $\\ell_t$ in \\eqref{eq:loss_of_jacobian}. \n    By the fundamental theorem of calculus, we can write $\\vu_t = \\vF({\\tilde{\\vz}_t})-\\vF(\\vz_t) = \\bar{\\mH}_t (\\tilde{\\vz}_t-\\vz_t)$, where $\\bar{\\mH}_t = \\int_{0}^1 \\nabla \\vF(\\vz_t+ \\lambda \\vs_t) \\,d\\lambda$. Moreover, using the triangle inequality, we have \n\\begin{align*}\n  \\|\\bar{\\mH}_t-\\mH^*\\|_{\\op} &\\leq \\int_{0}^1 \\|\\nabla \\vF(\\vz_t+ \\lambda \\vs_t)-\\nabla \\vF(\\vz^*)\\|_{\\op}\\, d\\lambda \\\\\n  &\\leq L_2\\int_{0}^1 \\|\\vz_t-\\lambda \\vs_t + \\vz^*\\|\\, d\\lambda \\\\\n  &\\leq L_2\\int_{0}^1 (\\|\\vz_t-\\vz^*\\| + \\lambda \\|\\vs_t\\|)\\, d\\lambda = L_2\\Bigl(\\|\\vz_t-\\vz^*\\|+\\frac{1}{2}\\|\\vs_t\\|\\Bigr),\n\\end{align*}\nwhere we used Assumption~\\ref{assum:jacobian_lips} in the second inequality. \n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\nTherefore, we have $\\|\\vu_t-\\mH^*\\vs_t\\| = \\|(\\bar{\\mH}_t-\\mH^*)\\vs_t\\| \\leq \\|\\bar{\\mH}_t-\\mH^*\\|_{\\op}\\|\\vs_t\\| \\leq L_2\\|\\vs_t\\|\\Bigl(\\|\\vz_t-\\vz^*\\|+\\frac{1}{2}\\|\\vs_t\\|\\Bigr)$. This further implies that \n\\begin{align}\n  \\sum_{t=0}^{T-1} \\ell_t(\\mH^*) = \\sum_{t=0}^{T-1} \\frac{\\|\\vu_t-\\mH^*\\vs_t\\|^2}{\\|\\vs_t\\|^2} &\\leq {L_2^2}\\sum_{t=0}^{T-1}\\Bigl(\\|\\vz_t-\\vz^*\\|+\\frac{1}{2}\\|\\vs_t\\|\\Bigr)^2 \\nonumber\\\\ \n  &\\leq \\frac{L_2^2}{2} \\sum_{t=0}^{T-1} \\|\\vs_t\\|^2 + 2 L_2^2\\sum_{t=0}^{T-1} \\|{\\vz}_t-\\vz^*\\|^2. \\label{eq:comparator_1}\n\\end{align}\n\\blue{Note that by our notation, we have $\\sum_{t=0}^{T-1} \\|\\vs_t\\|^2 = \\sum_{k \\in \\calB} \\|\\vs_k\\|^2 = \\sum_{k \\in \\calB} \\|\\tilde{\\vz}_k - \\vz_k\\|^2 $ and $\\sum_{t=0}^{T-1} \\|{\\vz}_t-\\vz^*\\|^2 = \\sum_{k\\in \\calB} \\|{\\vz}_k-\\vz^*\\|^2$.}\nTo bound the first sum $\\sum_{k \\in \\calB} \\|\\tilde{\\vz}_k - \\vz_k\\|^2$, we use the last result in Lemma~\\ref{lem:step size_lb} and the inequality in Lemma~\\ref{lem:sum_of_squares} to get \n\\begin{equation}\\label{eq:comparator_2}\n  \\begin{aligned}\n    \\sum_{k\\in \\calB} \\|\\tilde{\\vz}_k-\\vz_k\\|^2  \\leq \\frac{(1+\\alpha_1)^2}{\\beta^2 (1-\\alpha_1)^2} \\sum_{k\\in \\calB} \\|\\hat{\\vz}_k-\\vz_k\\|^2 \n    %\n    %\n    &\\leq \\frac{(1+\\alpha_1)^2}{\\beta^2 (1-\\alpha_1)^2} \\sum_{k=0}^{N-1} \\|\\hat{\\vz}_k-\\vz_k\\|^2 \\\\\n      &\\leq \\frac{(1+\\alpha_1)^2\\|\\vz_0-\\vz^*\\|^2}{(1-\\alpha_1)^2\\beta^2 (1-(\\alpha_1+\\alpha_2))} = 2C \\|\\vz_0-\\vz^*\\|^2. \n  \\end{aligned}\n  %\n\\end{equation}\nTo bound the second sum $\\sum_{k\\in \\calB} \\|{\\vz}_k-\\vz^*\\|^2$, we use the linear convergence result in Part (a) of Theorem~\\ref{thm:main}:\n\\begin{equation}\\label{eq:comparator_3}\n \\begin{aligned}\n  \\sum_{k\\in \\calB} \\|{\\vz}_k-\\vz^*\\|^2 \\leq \\sum_{k=0}^{N-1} \\|\\vz_k-\\vz^*\\|^2 &\\leq \\|\\vz_0-\\vz^*\\|^2\\sum_{k=0}^{N-1} \\left(1+\\frac{4\\alpha_2\\beta\\mu}{15L_1}\\right)^{-k}\\\\ &\\leq  \\|\\vz_0-\\vz^*\\|^2\\left(1+\\frac{15L_1}{4\\alpha_2 \\beta \\mu}\\right).\n \\end{aligned}\n\\end{equation}\nLemma~\\ref{lem:comparator} follows immediately from \\eqref{eq:comparator_1}, \\eqref{eq:comparator_2}, and \\eqref{eq:comparator_3}. \n  \\end{proof}\n %\n%\n  %\n\n\\noindent\\textbf{Step 3}: Combining Lemmas~\\ref{lem:comparator} and~\\ref{lem:small_loss},  we obtain a constant upper bound on the cumulative loss as\n%\n$\\sum_{t=0}^{T-1} \\ell_t({\\mB}_t) \\leq 121\\|\\mB_0-{\\mH^*}\\|_F^2+ \\left(C+2+\\frac{15L_1}{2\\alpha_2 \\beta \\mu}\\right)L_2^2\\|\\vz_0-\\vz^*\\|^2 = L_1^2 M$, where we used the definition of $M$ in \\eqref{eq:def_M}. \n%\n%\nTogether with the fact that $\\sigma_0 \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$, it follows from  Lemma~\\ref{lem:stepsize_bnd} that\n\\begin{equation*}\n  \\sum_{i=0}^{k-1}\\frac{1}{\\eta_i} \\leq \\frac{7.5L_1}{(1-\\beta)\\alpha_2\\beta}+ \\frac{L_1\\sqrt{kM}}{(1-\\beta)\\alpha_2\\beta} \\leq \n  \\frac{2L_1 \\max\\{7.5, \\sqrt{kM}\\}}{(1-\\beta)\\alpha_2\\beta}. \n  %\n\\end{equation*}\nThis leads to the superlinear convergence result in (b) by Proposition~\\ref{prop:HPE} and the observation in \\eqref{eq:after_jensen}.\n\\end{proof}","sha256_utf8":"b954560b2079e2e574051606420b092ddcb70bfbb1bf6f3d9b80f7f59fe7e554"},{"start":222212,"end":222571,"start_line":3862,"end_line":3864,"raw_tex":"\\begin{lemma}\\label{lem:ls_terminates}\n  Suppose Assumption~\\ref{assum:operator_lips} holds. If ${\\eta}_+ <\\frac{\\alpha_2}{L_1+\\|\\mB\\|_{\\op}}$ and $\\hat{\\vz}_+$ is computed according to \\eqref{eq:linear_solver_update}, then the pair $({\\eta}_+,\\hat{\\vz}_+)$ satisfies the conditions in \\eqref{eq:x_plus_update} and \\eqref{eq:step size_condition}. \n\\end{lemma}","sha256_utf8":"0ba34cb72cb7f20bc6a3c7ddd7c29ae20962c2b3faff0f4e74bdd501867bc980"},{"start":222573,"end":224433,"start_line":3866,"end_line":3891,"raw_tex":"\\begin{proof}\n  Since $\\hat{\\vz}_+$ follows the update rule in \\eqref{eq:linear_solver_update}, \n  by Definition~\\ref{def:linear_solver}, the pair $({\\eta}_+,\\hat{\\vz}_+)$ will always satisfy the condition in \\eqref{eq:x_plus_update}. \n  %\n  Hence, it is sufficient to prove that the condition in \\eqref{eq:step size_condition} also holds. \n  Recall that $\\vg$ is defined as $\\vg = \\vF(\\vz)$. By Assumption~\\ref{assum:operator_lips}, the operator $\\vF$ is $L_1$-Lipschitz and thus it follow that  \n  %\n  $\\|\\vF(\\hat{\\vz}_+)-\\vg\\| = \\|\\vF(\\hat{\\vz}_+)-\\vF(\\vz)\\| \\leq L_1 \\|\\hat{\\vz}_+ - \\vz\\|$. \n  %\n  Further, applying the triangle inequality yields \n  \\begin{equation*}\n      \\|\\vF(\\hat{\\vz}_+)-\\vg-\\mB(\\hat{\\vz}_+-\\vz)\\| \\leq \\|\\vF(\\hat{\\vz}_+)-\\vg\\| + \\|\\mB(\\hat{\\vz}_+-\\vz)\\| \\leq (L_1  + \\|\\mB\\|_{\\op}) \\|\\hat{\\vz}_+ - \\vz\\|.\n  \\end{equation*}\n  Hence, if ${\\eta}_+ \\leq \\frac{\\alpha_2}{L_1+\\|\\mB\\|_{\\op}}$, we have \n     \\begin{equation}\\label{eq:approx_error}\n         {\\eta}_+ \\|\\vF(\\hat{\\vz}_+)-\\vg-\\mB(\\hat{\\vz}_+-\\vz)\\| \\leq \\alpha_2 \\|\\hat{\\vz}_+ - \\vz\\|.\n     \\end{equation}\n  Finally, by using the triangle inequality again,  we combine \\eqref{eq:x_plus_update} and \\eqref{eq:approx_error} to establish that  \n  \\begin{align*}\n      \\|\\hat{\\vz}_{+}-\\vz+{\\eta}_+ \\vF(\\hat{\\vz}_{+})\\| &= \\|\\hat{\\vz}_{+}-\\vz+{{\\eta}_+}(\\vg+\\mB({\\hat{\\vz}_{+}}-\\vz)) + {\\eta}_+(\\vF(\\hat{\\vz}_+)-\\vg-\\mB(\\hat{\\vz}_+-\\vz))\\| \\\\\n      &\\leq \\|\\hat{\\vz}_{+}-\\vz+{{\\eta}_+}(\\vg+\\mB({\\hat{\\vz}_{+}}-\\vz))\\| + \\|{\\eta}_+(\\vF(\\hat{\\vz}_+)-\\vg-\\mB(\\hat{\\vz}_+-\\vz))\\| \\\\\n      &\\leq \\alpha_1 \\sqrt{1+\\eta_+ \\mu}\\|\\hat{\\vz}_+-\\vz\\|+\\alpha_2 \\|\\hat{\\vz}_+-\\vz\\| \\\\\n      &\\leq (\\alpha_1+\\alpha_2)\\sqrt{1+\\eta_+ \\mu}\\|\\hat{\\vz}_+-\\vz\\|, \n  \\end{align*}\n  which confirms that the condition in \\eqref{eq:step size_condition} is satisfied. This completes the proof. \n  \\end{proof}","sha256_utf8":"27dd749ac1252e72f9039dd8e4d74576645cb311ea350505210650d9aedd753d"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"强单调情形的全局收敛定理","source_units":[{"start":158832,"end":160678,"start_line":2788,"end_line":2824,"raw_tex":"\\begin{theorem}\\label{thm:main}\nSuppose Assumptions~\\ref{assum:strong_monotone},~\\ref{assum:operator_lips} and~\\ref{assum:jacobian_lips} hold. Let $\\{\\vz_k\\}$ and $\\{\\hat{\\vz}_k\\}$ be the iterates generated by Algorithm~\\ref{alg:Full_Equasi-Newton} using the line search scheme in Subroutine~\\ref{alg:ls}, where \\blue{$\\alpha_1, \\alpha_2 \\in (0,\\frac{1}{2})$, $\\beta\\in(0,1)$,}\n%\nand $\\sigma_0 \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$. In addition, the Jacobian approximation matrices are updated in Subroutine~\\ref{alg:jacobian_approx} with \\textbf{Option I}, where {$\\rho = \\frac{1}{121}$}, $\\delta_t = {\\frac{\\mu}{2L_1}}$, and $q_t = \\frac{p}{2.5(t+1)\\log^2(t+1)}$ for $t \\geq 1$. With probability at least $1-p$, the following holds:  \n  \\begin{enumerate}[(a)]\n  %\n    \\item (Linear convergence) For any $k\\geq 0$, we have\n    %\n        $ \\frac{\\|\\vz_{k+1}-\\vz^*\\|^2}{\\|\\vz_k-\\vz^*\\|^2 }\\leq  \\left(1+\\frac{4\\alpha_2\\beta\\mu}{15L_1}\\right)^{-1}$.\n        %\n    \\item (Superlinear convergence) We have $\\lim_{k\\rightarrow \\infty}\\frac{\\|\\vz_{k+1}-\\vz^*\\|^2}{\\|\\vz_k-\\vz^*\\|^2 } = 0$. Furthermore, define the absolute constant $C$ and the quantity $M$ by  \n    \\begin{align}\\label{eq:def_c1}\n      C &= \\frac{(1+\\alpha_1)^2}{2(1-\\alpha_1)^2\\beta^2 (1-(\\alpha_1+\\alpha_2))}, \\\\\n       M &= \\frac{121\\|\\mB_0-\\nabla^2f(\\vz^*)\\|_F^2}{L_1^2}+ \\left(C+2+\\frac{15L_1}{2\\alpha_2 \\beta \\mu}\\right)\\frac{L_2^2\\|\\vz_0-\\vz^*\\|^2}{L_1^2}. \\label{eq:def_M}\n    \\end{align}\n    Then for any $k\\geq 0$, we have \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    \\begin{equation*}\n      {%\n      \\frac{\\|\\vz_k-\\vz^*\\|^2}{\\|\\vz_0-\\vz^*\\|^2} \\leq   \\Biggl(1+ \n       \\frac{(1-\\beta)\\alpha_2\\beta\\mu}{L_1} \\min\\left\\{ \\frac{2}{15}k, \\sqrt{\\frac{k}{M}}\\right\\}\n      %\n      \\Biggr)^{-k}%\n      .}\n      %\n    \\end{equation*}\n    %\n  \\end{enumerate}\n\\end{theorem}","sha256_utf8":"ef3d7df826ffe0c96f6286b91f998f458b473f6896c20060d59e9a960b60d314"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"主定理完整证明","source_units":[{"start":161821,"end":162646,"start_line":2832,"end_line":2841,"raw_tex":"Now we move to the proof of Theorem~\\ref{thm:main}. First of all, by using a simple union bound, we can show that \n the  $\\mathsf{ExtEvec}$ and $\\mathsf{MaxSvec}$ oracles used in Subroutine~\\ref{alg:jacobian_approx} are successful in all rounds with probability at least $1-p$. \n %\n Specifically, \nsince both  oracles have a failure probability of $ \\frac{q_t}{2} = \\frac{p}{5(t+1)\\log^2(t+1)}$ in the $t$-th round, we can use the union bound to upper bound the total failure probability by \n\\begin{equation*}\n  \\sum_{t=1}^{T-1} q_t %\n   \\leq  \\frac{p}{2.5}\\sum_{t=2}^{\\infty} \\frac{1}{t \\log^2 t} \\leq \\frac{p}{2.5}\\left(\\frac{1}{2\\log^2 2}+\\int_{2}^{+\\infty}\\frac{1}{t\\log^2t}\\,dt\\right) \\leq p.\n\\end{equation*}\n Thus, throughout the proof, we assume every call of $\\mathsf{ExtEvec}$ and $\\mathsf{MaxSvec}$ is successful. \n","sha256_utf8":"4c24c444a630ae21074f635ba6f53cb25f519bb551ef6f12bd2d68e43fd60b8b"},{"start":162649,"end":165988,"start_line":2844,"end_line":2886,"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm:main}(a)]\n  \n\nWe first prove the linear convergence rate in (a). To begin with, note that Subroutine~\\ref{alg:jacobian_approx} with \\textbf{Option I} guarantees that  $\\hat{\\mB}_k \\in (1+\\frac{\\mu}{2L_1}) \\mathcal{C}$ by Lemma~\\ref{lem:regret_reduction}.  As a corollary of Lemma~\\ref{lem:transform}, we have the following result showing the boundedness of the Jacobian approximation matrix $\\mB_k$. \n\n%\n\n \n\\begin{corollary}\\label{lem:feasible_Bk}\n  Let $\\delta_t = \\frac{\\mu}{2L_1}$ for all $t\\geq 0$. Then for any $k \\geq 0$, we have $\\frac{1}{2}(\\mB_k + \\mB_k^\\top )\\succeq \\frac{\\mu}{2} \\mI$ and $\\|\\mB_k\\|_{\\op} \\leq 6.5L_1$. \n\\end{corollary}\nCombining this with Lemma~\\ref{lem:step size_lb}, we obtain the following universal lower bound on the step size $\\eta_k$. \n\\begin{lemma}\\label{lem:stepsize_const_bound}\n  %\n  In Subroutine~\\ref{alg:ls}, choose $\\sigma_0 \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$. For any $k\\geq 0$, we have $\\eta_k \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$. \n\\end{lemma}\n\\begin{proof}\n  Recall that $\\calB = \\{k: \\eta_k < \\sigma_k\\}$.\n  We first establish that $\\eta_k \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$ for $k\\in \\mathcal{B}$. To see this, suppose $k\\in \\mathcal{B}$ and recall from Lemma~\\ref{lem:step size_lb} that\n\\begin{equation}\\label{eq:step size_lb_appen}\n  \\eta_k > \\frac{\\alpha_2 \\beta\\|{\\tilde{\\vz}_k}-\\vz_k\\|}{\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\|}.\n\\end{equation}\n%\n  %\n  %\n  %\n  Moreover, by using the triangle inequality, we have \n  %\n  $\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\| \\leq \\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)\\| + \\|\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\|$.\n  %\n  By using Assumption~\\ref{assum:operator_lips}, we have $\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)\\| \\leq L_1\\|\\tilde{\\vz}_k-\\vz_k\\|$. Moreover, since $\\|\\mB_k\\|_{\\op} \\leq 6.5L_1$ by Corollary~\\ref{lem:feasible_Bk}, we also get $\\|\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\| \\leq 6.5 \\|{\\tilde{\\vz}_k}-\\vz_k\\|$. \n  %\n  Thus, this implies that $\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\| \\leq 7.5\\|{\\tilde{\\vz}_k}-\\vz_k\\|$,  \n  %\n  %\n  %\n  which leads to $\\eta_k > \\frac{\\alpha_2\\beta}{7.5L_1}$ from \\eqref{eq:step size_lb_appen}. \n\n  Now we can prove that $\\eta_k \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$ for all $k\\geq 0$ by induction. To show that this holds for $k=0$, we distinguish two cases. If $0\\notin \\mathcal{B}$, then we have $\\eta_0 = \\sigma_0 > \\frac{\\alpha_2\\beta}{7.5L_1}$ by our choice of $\\sigma_0$. Otherwise, if $0\\in \\mathcal{B}$, then it directly follows from our result in the previous paragraph. Moreover, assume that $\\eta_{l-1} \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$ where $l\\geq 1$. Similarly, we again distinguish two cases: if $l\\notin \\mathcal{B}$, then we have $\\eta_l = \\sigma_l = \\frac{\\eta_{l-1}}{\\beta} >\\frac{\\alpha_2}{7.5L_1}>\\frac{\\alpha_2\\beta}{7.5L_1}$; otherwise, if $l\\in \\mathcal{B}$, it follows from the result above that $\\eta_l \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$. This completes the induction. \n\\end{proof}\nIn light of Lemma~\\ref{lem:stepsize_const_bound}, it follows from Proposition~\\ref{prop:HPE}(a) that  $\\|\\vz_{k+1}-\\vz^*\\|^2 \\leq \\|\\vz_k-\\vz^*\\|^2 (1+2\\eta_k \\mu)^{-1} \\leq \\|\\vz_k-\\vz^*\\|^2 \\left(1+\\frac{4\\alpha_2\\beta\\mu}{15L_1}\\right)^{-1}$. \nThis proves the linear convergence result in (a). \n\\end{proof}","sha256_utf8":"8423e5d994c1c0e719d8d9f101e4a0264e41702f301aa5d365951b38ad88ea1f"},{"start":165990,"end":166547,"start_line":2888,"end_line":2896,"raw_tex":"Next, we prove the superlinear convergence rate in (b). Our starting point is the observations in \\eqref{eq:after_jensen} and \\eqref{eq:goal}, which demonstrate that it is sufficient to prove an upper bound on the cumulative loss $\\sum_{k=0}^{N-1} \\ell_k(\\mB_k)$. \n%\n%\nMoreover, since $\\ell_k(\\mB_k) = 0$ when $k \\notin \\calB$ by the definition in \\eqref{eq:loss_of_jacobian}, it is equivalent to bounding the sum $\\sum_{k \\in \\calB} \\ell_k(\\mB_k)$.  \n%\n%\n\\blue{Recall that we relabel the indices in $\\mathcal{B}$ by $t=0,\\dots,T-1$ with $T \\leq N$. %\n%\n} \n","sha256_utf8":"2fd3d39fb9b5521500733e26ef36522e5dde38106848392dcb039479c8cb0335"},{"start":166581,"end":178460,"start_line":2915,"end_line":3111,"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm:main}(b)] We consider the following three steps.\n\n\\vspace{0em}\\noindent\\textbf{Step 1:} Using regret analysis, we bound the cumulative loss $\\sum_{t=0}^{T-1}\\ell_{t}(\\mB_{t})$ incurred by our online learning algorithm in Subroutine~\\ref{alg:jacobian_approx}. \nTo begin with, we present the following lemma showing a self-bounding property of the loss function $\\ell_t$. {It is similar to the standard inequality $\\frac{1}{2L_1} \\|\\nabla g (\\vx)\\|^2 \\leq g(\\vx) - g^* $ for a $L_1$-smooth function $g$, where $g^*$ denotes the minimum of $g$.} This will be the key to proving a constant upper bound on the cumulative loss incurred by Subroutine~\\ref{alg:jacobian_approx}.\n\\begin{lemma}\\label{lem:loss}\n  Recall the loss function $\\ell_k$ defined in \\eqref{eq:loss_of_jacobian}. For $k\\in \\mathcal{B}$, we have\n  \\begin{equation}\\label{eq:gradient}\n    \\nabla \\ell_k(\\mB) = - \\frac{2(\\vu_k-\\mB\\vs_k)\\vs_k^\\mathsf{T}}{\\|\\vs_k\\|^2}.\n  \\end{equation}\n  Moreover, for any $\\mB\\in \\reals^{d\\times d}$, it holds that \n    \\begin{equation}\\label{eq:gradient_norm_bound}\n        \\|\\nabla \\ell_k(\\mB)\\|_F \\leq \\|\\nabla \\ell_k(\\mB)\\|_* = 2\\sqrt{\\ell_k(\\mB)},\n    \\end{equation} \n    where $\\|\\cdot\\|_F$ and $\\|\\cdot\\|_{*}$ denote the Frobenius norm and the nuclear norm, respectively.  \n\\end{lemma}\n\\begin{proof}\n  %\n  The expression in \\eqref{eq:gradient} follows from the direct calculation. The first inequality in \\eqref{eq:gradient_norm_bound} follows from the fact that $\\|\\mA\\|_F \\leq \\|\\mA\\|_*$ for any matrix $\\mA\\in \\reals^{d\\times d}$. For the equality, note that \n  \\begin{align*}\n    \\|\\nabla \\ell_k(\\mB)\\|_*   = \\frac{2}{\\|\\vs_k\\|^2}  \\|\\vu_k-\\mB\\vs_k\\|\\|\\vs_k\\|  = \\frac{2\\|\\vu_k-\\mB\\vs_k\\|}{\\|\\vs_k\\|} = 2\\sqrt{\\ell_k(\\mB)}, \n  \\end{align*}\n  where we used the fact that the rank-one matrix $\\va\\vb^\\top$ has only one nonzero singular value $\\|\\va\\|\\|\\vb\\|$. \n\\end{proof}\n\n\n%\nIn particular, by exploiting the self-bounding property of the loss function $\\ell_t$, we prove a ``small-loss bound'' \\cite{srebro2010smoothness} in the following lemma, where the cumulative loss of the learner is bounded by that of a fixed action in the competitor set. \n%\n  \\begin{lemma}\n  \\label{lem:small_loss}\n  %\n  %\n  In Subroutine~\\ref{alg:jacobian_approx}, choose \\textbf{Option I} with $\\rho = \\frac{1}{121}$ and $\\delta = \\frac{\\mu}{2L_1}$. \n  For any ${\\mH}\\in \\mathcal{{Z}}$, we have\n  %\n    $\\sum_{t=0}^{T-1} \\ell_t({\\mB}_t)  \\leq 121\\|\\mB_0-{\\mH}\\|_F^2+ 2 \\sum_{t=0}^{T-1} \\ell_t({\\mH})$.\n  %\n\\end{lemma}\n\n\\begin{proof}\nRecall that $\\calZ$ is contained in the linear subspace $\\calL$ (see \\eqref{eq:feasible_set}) and $\\mP$ denotes the orthogonal projection matrix associated with $\\calL$.    \nBy letting $\\vx_t = \\hat{\\mB}_t$, $\\vu = \\hat{\\mH} \\triangleq \\frac{1}{L_1}(\\mH-(L_1+\\mu)\\mI)$, \n%\n$\\vg_t = \\mG_t \\triangleq \\frac{1}{L_1}\\mP \\nabla \\ell_t(\\mB_t)$, $\\tilde{\\vg}_t = \\tilde{\\mG}_t$, $\\vw_t = \\mW_t$ in Lemma~\\ref{lem:regret_reduction}, we obtain:  \n\\begin{enumerate}[(i)]\n  \\item $\\hat{\\mB}_t \\in (1+\\delta)\\mathcal{C}$, which means $\\|\\hat{\\mB}_t\\|_{\\op} \\leq 3(1+\\delta)\\leq 4.5$ since $\\delta = \\frac{\\mu}{2L_1}\\leq \\frac{1}{2}$.\n  \\item It holds that \n  \\begin{align}\n    \\langle \\mG_t, \\hat{\\mB}_t-\\hat{\\mH} \\rangle &\\leq \\frac{1}{2\\rho}\\|\\mW_t-\\hat{\\mH}\\|_F^2-\\frac{1}{2\\rho}\\|\\mW_{t+1}-\\hat{\\mH}\\|_F^2+\\frac{\\rho}{2}\\|\\tilde{\\mG}_t\\|_F^2, \\label{eq:linearized_loss_matrix} \\\\%\\\\\n    %\n    \\|\\tilde{\\mG}_t\\|_F &\\leq \\|\\mG_t\\|_F + |\\langle \\mG_t, \\hat{\\mB}_t\\rangle|\\|\\mS_t\\|_F. \\label{eq:surrogate_gradient_bound}\n  \\end{align}\n\\end{enumerate}\n%\n%\n%\n%\n%\n%\n%\n%\n%\nFirst, note that $\\|\\mS_t\\|_F \\leq 1$ by Definition~\\ref{def:extevec} and\n%\n$|\\langle \\mG_t, \\hat{\\mB}_t\\rangle| \\leq \\|\\mG_t\\|_* \\|\\hat{\\mB}_t\\|_{\\op} \\leq 4.5 \\|\\mG_t\\|_*$. %\n%\nTogether with \\eqref{eq:surrogate_gradient_bound}, we get \n\\begin{equation}\\label{eq:surrogate_bound_loss}\n\\|\\tilde{\\mG}_t\\|_F \\leq \\|\\mG_t\\|_F+4.5\\|\\mG_t\\|_* \\leq 5.5\\|\\mG_t\\|_* \\leq \\frac{5.5}{L_1}\\|\\nabla \\ell_t(\\mB_t)\\|_* \\leq \\frac{11}{L_1}\\sqrt{\\ell_t({\\mB}_t)},\n\\end{equation}\nwhere we used $\\mG_t = \\frac{1}{L_1}\\mP \\nabla \\ell_t(\\mB_t)$ and Lemma~\\ref{lem:loss} in the last inequality. \nFurthermore, since $\\ell_t$ is convex and $\\mB_t,\\mH \\in \\calL$, we have \n\\begin{equation}\\label{eq:convexity_subspace}\n    \\ell_t({\\mB}_t) - \\ell_t(\\mH) \\!\\leq\\! \\langle \\nabla \\ell_t(\\mB_t), {\\mB}_t-\\mH \\rangle \\!=\\! \\langle \\nabla \\ell_t(\\mB_t), \\mP\\left({\\mB}_t-\\mH\\right) \\rangle \\!=\\! L_1^2 \\langle \\mG_t, \\hat{\\mB}_t-\\hat{\\mH} \\rangle,\n\\end{equation}\nwhere we used $\\mG_t = \\frac{1}{L_1}\\mP\\nabla \\ell_t(\\mB_t)$, $\\hat{\\mB}_t \\triangleq \\frac{1}{L_1}(\\mB_t-{(L_1+\\mu)}\\mI)$, and $\\hat{\\mH} \\triangleq \\frac{1}{L_1}(\\mH-{(L_1+\\mu)}\\mI)$. \nTherefore, by \\eqref{eq:linearized_loss_matrix} and \\eqref{eq:surrogate_bound_loss} we get \n\\begin{align*}\n\\ell_t({\\mB}_t) - \\ell_t(\\mH) &\\leq \\frac{L_1^2}{2\\rho}\\|\\mW_t-\\hat{\\mH}\\|_F^2-\\frac{L_1^2}{2\\rho}\\|\\mW_{t+1}-\\hat{\\mH}\\|_F^2+\\frac{\\rho L_1^2}{2} \\|\\tilde{\\mG}_t\\|_F^2\\\\\n&\\leq \\frac{L_1^2}{2\\rho}\\|\\mW_t-\\hat{\\mH}\\|_F^2-\\frac{L_1^2}{2\\rho}\\|\\mW_{t+1}-\\hat{\\mH}\\|_F^2+60.5\\rho\\ell_t({\\mB}_t). %\n%\n\\end{align*}\nSince $\\rho = \\frac{1}{121}$, by rearranging and simplifying terms in the above inequality, we obtain\n%\n%\n%\n$\\ell_t({\\mB}_t) \\leq 2\\ell_t({\\mH})+ {121 L_1^2}\\|\\mW_t-\\hat{\\mH}\\|_F^2-{121 L_1^2}\\|\\mW_{t+1}-\\hat{\\mH}\\|_F^2$.\n%\nBy summing the above inequality from $t=0$ to $T-1$, we further have \n\\begin{equation*}\n  \\sum_{t=0}^{T-1} \\ell_t({\\mB}_t)  \\leq {121 L_1^2}\\|\\mW_0-\\hat{\\mH}\\|_F^2+ 2 \\sum_{t=0}^{T-1} \\ell_t({\\mH}) = 121\\|\\mB_0-{\\mH}\\|_F^2+ 2 \\sum_{t=0}^{T-1} \\ell_t({\\mH}),\n\\end{equation*}\nwhere the last equality is due to ${\\mW}_0 \\triangleq \\frac{1}{L_1}(\\mB_0-{(L_1+\\mu)}\\mI)$ and $\\hat{\\mH} \\triangleq \\frac{1}{L_1}(\\mH-{(L_1+\\mu)}\\mI)$. This completes the proof. \n\\end{proof}\n\n%\nNote that in Lemma~\\ref{lem:small_loss}, we have the freedom to choose any competitor $\\mH$ in the set $\\mathcal{Z}$. To further obtain an explicit bound, a natural choice would be $\\mH^* \\triangleq \\nabla \\vF(\\vz^*)$, which leads to our next step.\n\n%\n\n\\vspace{0em}\\noindent \\textbf{Step 2:} We upper bound the cumulative loss $\\sum_{t=0}^{T-1} \\ell_t({\\mH}^*)$. The proof relies crucially on Assumption~\\ref{assum:jacobian_lips} as well as the linear convergence result we proved in (a).\nTo begin with, we prove the following useful lemma. \n\\begin{lemma}\\label{lem:sum_of_squares}\n If Assumption~\\ref{assum:strong_monotone} holds, then $\\sum_{k=0}^{N-1} \\|\\hat{\\vz}_{k}\\!-\\!\\vz_k\\|^2 \\leq \\frac{1}{1-(\\alpha_1+\\alpha_2)}\\|\\vz_0\\!-\\!\\vz^*\\|^2$. \n\\end{lemma}\n\\begin{proof}\n  Recall the inequality~\\eqref{eq:one_step_strongly} in the proof of Proposition~\\ref{prop:HPE}. Since $\\frac{1}{2}\\|\\vz_{k+1}-\\vz^*\\|^2 \\!\\leq \\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1}-\\vz^*\\|^2$, we can further derive that \n  \\begin{equation}\\label{eq:one_step_strongly_coro}\n    \\frac{1-\\alpha_1-\\alpha_2}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2) \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1}{2}\\|\\vz_{k+1} - \\vz^*\\|^2.\n\\end{equation} \nSince $\\alpha_1+\\alpha_2 < 1$, we further have $\\frac{1-\\alpha_1-\\alpha_2}{2} \\|\\hat{\\vz}_k - \\vz_k\\|^2 \\leq \\frac{1-\\alpha_1-\\alpha_2}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2) \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1}{2}\\|\\vz_{k+1} - \\vz^*\\|^2$. By summing the above inequality from $k=0$ to $k=N-1$, we obtain that \n\\begin{equation*}\n  \\frac{1-\\alpha_1-\\alpha_2}{2} \\sum_{k=0}^{N-1} \\|\\hat{\\vz}_k - \\vz_k\\|^2 \\leq \\frac{1}{2}\\|\\vz_0-\\vz^*\\|^2 - \\frac{1}{2}\\|\\vz_N-\\vz^*\\|^2 \\leq \\frac{1}{2}\\|\\vz_0-\\vz^*\\|^2. \n\\end{equation*}\nThus, we arrive at Lemma~\\ref{lem:sum_of_squares} by dividing both sides by $\\frac{1-\\alpha_1-\\alpha_2}{2}$. \n\\end{proof}\n\nWith the help of Lemma~\\ref{lem:sum_of_squares}, we are ready to upper bound $\\sum_{t=0}^{T-1} \\ell_t({\\mH}^*)$. \n  \\begin{lemma}\\label{lem:comparator}\n  Recall the constant $C$ defined in \\eqref{eq:def_c1}. \n  %\n  Then \n    %\n      %\n    %\n    %\n    %\n    %\n    $\\sum_{t=0}^{T-1} \\ell_t(\\mH^*) \\leq \\left(C+2+\\frac{15L_1}{2\\alpha_2 \\beta \\mu}\\right)L_2^2\\|\\vz_0-\\vz^*\\|^2$.\n    %\n  \\end{lemma}\n  \\begin{proof}\n    Recall the definition of $\\ell_t$ in \\eqref{eq:loss_of_jacobian}. \n    By the fundamental theorem of calculus, we can write $\\vu_t = \\vF({\\tilde{\\vz}_t})-\\vF(\\vz_t) = \\bar{\\mH}_t (\\tilde{\\vz}_t-\\vz_t)$, where $\\bar{\\mH}_t = \\int_{0}^1 \\nabla \\vF(\\vz_t+ \\lambda \\vs_t) \\,d\\lambda$. Moreover, using the triangle inequality, we have \n\\begin{align*}\n  \\|\\bar{\\mH}_t-\\mH^*\\|_{\\op} &\\leq \\int_{0}^1 \\|\\nabla \\vF(\\vz_t+ \\lambda \\vs_t)-\\nabla \\vF(\\vz^*)\\|_{\\op}\\, d\\lambda \\\\\n  &\\leq L_2\\int_{0}^1 \\|\\vz_t-\\lambda \\vs_t + \\vz^*\\|\\, d\\lambda \\\\\n  &\\leq L_2\\int_{0}^1 (\\|\\vz_t-\\vz^*\\| + \\lambda \\|\\vs_t\\|)\\, d\\lambda = L_2\\Bigl(\\|\\vz_t-\\vz^*\\|+\\frac{1}{2}\\|\\vs_t\\|\\Bigr),\n\\end{align*}\nwhere we used Assumption~\\ref{assum:jacobian_lips} in the second inequality. \n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\nTherefore, we have $\\|\\vu_t-\\mH^*\\vs_t\\| = \\|(\\bar{\\mH}_t-\\mH^*)\\vs_t\\| \\leq \\|\\bar{\\mH}_t-\\mH^*\\|_{\\op}\\|\\vs_t\\| \\leq L_2\\|\\vs_t\\|\\Bigl(\\|\\vz_t-\\vz^*\\|+\\frac{1}{2}\\|\\vs_t\\|\\Bigr)$. This further implies that \n\\begin{align}\n  \\sum_{t=0}^{T-1} \\ell_t(\\mH^*) = \\sum_{t=0}^{T-1} \\frac{\\|\\vu_t-\\mH^*\\vs_t\\|^2}{\\|\\vs_t\\|^2} &\\leq {L_2^2}\\sum_{t=0}^{T-1}\\Bigl(\\|\\vz_t-\\vz^*\\|+\\frac{1}{2}\\|\\vs_t\\|\\Bigr)^2 \\nonumber\\\\ \n  &\\leq \\frac{L_2^2}{2} \\sum_{t=0}^{T-1} \\|\\vs_t\\|^2 + 2 L_2^2\\sum_{t=0}^{T-1} \\|{\\vz}_t-\\vz^*\\|^2. \\label{eq:comparator_1}\n\\end{align}\n\\blue{Note that by our notation, we have $\\sum_{t=0}^{T-1} \\|\\vs_t\\|^2 = \\sum_{k \\in \\calB} \\|\\vs_k\\|^2 = \\sum_{k \\in \\calB} \\|\\tilde{\\vz}_k - \\vz_k\\|^2 $ and $\\sum_{t=0}^{T-1} \\|{\\vz}_t-\\vz^*\\|^2 = \\sum_{k\\in \\calB} \\|{\\vz}_k-\\vz^*\\|^2$.}\nTo bound the first sum $\\sum_{k \\in \\calB} \\|\\tilde{\\vz}_k - \\vz_k\\|^2$, we use the last result in Lemma~\\ref{lem:step size_lb} and the inequality in Lemma~\\ref{lem:sum_of_squares} to get \n\\begin{equation}\\label{eq:comparator_2}\n  \\begin{aligned}\n    \\sum_{k\\in \\calB} \\|\\tilde{\\vz}_k-\\vz_k\\|^2  \\leq \\frac{(1+\\alpha_1)^2}{\\beta^2 (1-\\alpha_1)^2} \\sum_{k\\in \\calB} \\|\\hat{\\vz}_k-\\vz_k\\|^2 \n    %\n    %\n    &\\leq \\frac{(1+\\alpha_1)^2}{\\beta^2 (1-\\alpha_1)^2} \\sum_{k=0}^{N-1} \\|\\hat{\\vz}_k-\\vz_k\\|^2 \\\\\n      &\\leq \\frac{(1+\\alpha_1)^2\\|\\vz_0-\\vz^*\\|^2}{(1-\\alpha_1)^2\\beta^2 (1-(\\alpha_1+\\alpha_2))} = 2C \\|\\vz_0-\\vz^*\\|^2. \n  \\end{aligned}\n  %\n\\end{equation}\nTo bound the second sum $\\sum_{k\\in \\calB} \\|{\\vz}_k-\\vz^*\\|^2$, we use the linear convergence result in Part (a) of Theorem~\\ref{thm:main}:\n\\begin{equation}\\label{eq:comparator_3}\n \\begin{aligned}\n  \\sum_{k\\in \\calB} \\|{\\vz}_k-\\vz^*\\|^2 \\leq \\sum_{k=0}^{N-1} \\|\\vz_k-\\vz^*\\|^2 &\\leq \\|\\vz_0-\\vz^*\\|^2\\sum_{k=0}^{N-1} \\left(1+\\frac{4\\alpha_2\\beta\\mu}{15L_1}\\right)^{-k}\\\\ &\\leq  \\|\\vz_0-\\vz^*\\|^2\\left(1+\\frac{15L_1}{4\\alpha_2 \\beta \\mu}\\right).\n \\end{aligned}\n\\end{equation}\nLemma~\\ref{lem:comparator} follows immediately from \\eqref{eq:comparator_1}, \\eqref{eq:comparator_2}, and \\eqref{eq:comparator_3}. \n  \\end{proof}\n %\n%\n  %\n\n\\noindent\\textbf{Step 3}: Combining Lemmas~\\ref{lem:comparator} and~\\ref{lem:small_loss},  we obtain a constant upper bound on the cumulative loss as\n%\n$\\sum_{t=0}^{T-1} \\ell_t({\\mB}_t) \\leq 121\\|\\mB_0-{\\mH^*}\\|_F^2+ \\left(C+2+\\frac{15L_1}{2\\alpha_2 \\beta \\mu}\\right)L_2^2\\|\\vz_0-\\vz^*\\|^2 = L_1^2 M$, where we used the definition of $M$ in \\eqref{eq:def_M}. \n%\n%\nTogether with the fact that $\\sigma_0 \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$, it follows from  Lemma~\\ref{lem:stepsize_bnd} that\n\\begin{equation*}\n  \\sum_{i=0}^{k-1}\\frac{1}{\\eta_i} \\leq \\frac{7.5L_1}{(1-\\beta)\\alpha_2\\beta}+ \\frac{L_1\\sqrt{kM}}{(1-\\beta)\\alpha_2\\beta} \\leq \n  \\frac{2L_1 \\max\\{7.5, \\sqrt{kM}\\}}{(1-\\beta)\\alpha_2\\beta}. \n  %\n\\end{equation*}\nThis leads to the superlinear convergence result in (b) by Proposition~\\ref{prop:HPE} and the observation in \\eqref{eq:after_jensen}.\n\\end{proof}","sha256_utf8":"b954560b2079e2e574051606420b092ddcb70bfbb1bf6f3d9b80f7f59fe7e554"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"非线性方程与连续可微算子","source_units":[{"start":18753,"end":18908,"start_line":684,"end_line":689,"raw_tex":"In this paper, we consider the problem of solving a system of nonlinear equations:\n%\n\\begin{equation}\\label{eq:monotone}\n  \\vF(\\vz) = 0,\n  %\n\\end{equation}","sha256_utf8":"daf1a1184722314aa1224af98de70c5ff325fae57d5f0a2c3a2671fd8d3efccc"},{"start":18909,"end":18996,"start_line":690,"end_line":690,"raw_tex":"where $\\vF : \\reals^d \\rightarrow \\reals^d$ is a continuously differentiable operator.\n","sha256_utf8":"6f2b7816885eaaf550349fa39cfe7d92b3899c362a2a7e3a2ad815f7663bfbea"}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","title":"单调性与两类连续性假设","source_units":[{"start":35042,"end":35273,"start_line":946,"end_line":949,"raw_tex":"\\begin{assumption}\\label{assum:strong_monotone}  \nThe operator $\\vF$ is $\\mu$-strongly monotone, i.e., $\\langle \\vF(\\vz)-\\vF(\\vz'), \\vz-\\vz' \\rangle \\geq \\mu \\|\\vz-\\vz'\\|^2$,  $\\forall \\vz,\\vz' \\in \\reals^d$. \n   %\n\\end{assumption}","sha256_utf8":"9cde0702e46789b444eb0089a787f31a9584a70a8c42012f5dd103b55905de6d"},{"start":35275,"end":35455,"start_line":951,"end_line":954,"raw_tex":"\\begin{assumption}\\label{assum:monotone}  \n  $\\vF$ is monotone, i.e., $\\langle \\vF(\\vz)-\\vF(\\vz'), \\vz-\\vz' \\rangle \\geq 0$,  $\\forall \\vz,\\vz' \\in \\reals^d$. \n  %\n\\end{assumption}","sha256_utf8":"5c43752a1c1ac1e2729d97cbaeea71f0a040f4e8dced29374f4b622cdfd2c375"},{"start":35524,"end":36069,"start_line":958,"end_line":959,"raw_tex":"Note that under Assumption~\\ref{assum:strong_monotone}, Problem \\eqref{eq:monotone} has a unique solution, which we denote by $\\vz^*$ throughout the paper.\nMoreover, Assumption~\\ref{assum:monotone} can be regarded as a special case of Assumption~\\ref{assum:strong_monotone} by setting $\\mu=0$. Therefore, we will sometimes use this observation to present our results in both settings in a united manner, avoiding unnecessary repetition. In addition, we make the following two assumptions that the operator $\\vF$ and its Jacobian are Lipschitz. \n","sha256_utf8":"db085d32777cc9f0f8a97f31ed63c69e555cf7c00959c67c521e3783bb97e500"},{"start":36070,"end":36262,"start_line":961,"end_line":964,"raw_tex":"\\begin{assumption}\\label{assum:operator_lips}\nThe operator $\\vF$ is $L_1$-Lipschitz, i.e., $\\|\\vF(\\vz)-\\vF(\\vz')\\| \\leq L_1\\|\\vz-\\vz'\\|$, $\\forall \\vz,\\vz' \\in \\reals^d$.  \n %\n\\end{assumption}","sha256_utf8":"657178a9c67b0fa6178e0f0a573269dee27ac684a9b847a158dcc97c13a77c68"},{"start":36264,"end":36541,"start_line":966,"end_line":969,"raw_tex":"\\begin{assumption}\\label{assum:jacobian_lips}\n  The Jacobian of $\\vF$ is $L_2$-Lipschitz, i.e., $\\|\\nabla \\vF(\\vz)-\\nabla \\vF(\\vz')\\|_{\\op} \\leq L_2\\|\\vz-\\vz'\\|$, $\\forall \\vz,\\vz'\\in \\reals^d$, where $\\|\\mA\\|_{\\op} \\mydef \\sup_{\\|\\vz\\| = 1} \\|\\mA \\vz\\|$. \n  %\n\\end{assumption}","sha256_utf8":"808362cdb6abe2379b3f1deb854c685a0552b1d2131dcc40625bdd89355ff65a"}]},{"id":"S005","semantic_type":"problem_setting","kind":"problem_setting","title":"优化特例与雅可比结构","source_units":[{"start":38834,"end":39051,"start_line":1008,"end_line":1011,"raw_tex":"\\myalert{{Minimization.}} As a special instance of Problem \\eqref{eq:monotone}, consider the unconstrained minimization problem \n\\begin{equation}\\label{eq:minimization}\n  \\min_{\\vz \\in \\reals^d} f(\\vz),\n\\end{equation}","sha256_utf8":"509fe7789c34cd4926cb09dff0ddfb894784f86d7520bac2f7f379b08812a7be"},{"start":39052,"end":40186,"start_line":1012,"end_line":1013,"raw_tex":"%\nwhere $f\\!:\\! \\reals^d\\! \\rightarrow \\!\\reals$ is a (strongly) convex function. By the first-order optimality condition, the optimal solution of \\eqref{eq:minimization} satisfies the nonlinear equation $\\nabla f(\\vz) = 0$. Thus, in this case, $\\vF$ is  the gradient operator, i.e., $\\vF(\\vz) = \\nabla f(\\vz)$, and its Jacobian is the Hessian matrix of $f$, i.e., $\\nabla \\vF(\\vz) = \\nabla^2 f(\\vz)$. In particular, we observe that the Jacobian is a symmetric matrix, which allows us to simplify the conditions in Lemma~\\ref{lem:jacobian}. Specifically, under Assumptions~\\ref{assum:strong_monotone} and~\\ref{assum:operator_lips},  $\\nabla \\vF$ belongs to the set $\\{\\mA \\in \\mathbb{S}^d\\!: \\mu \\mI \\preceq\\! \\mA\\! \\preceq L\\mI\\}$. Meanwhile, under Assumptions~\\ref{assum:monotone} and~\\ref{assum:operator_lips}, it belongs to the set $\\{\\mA \\in \\mathbb{S}^d\\!: 0 \\preceq\\! \\mA\\! \\preceq L\\mI\\}$. As discussed in Section~\\ref{subsec:nonlinear_eqs}, in our method, we ensure that the Jacobian approximation matrix stays symmetric and positive semidefinite, which is a key property for the efficient implementation of the subroutines.\n","sha256_utf8":"a72f16538da8be2c1ab426fe4823f616866d57e071172d37f21dab9ebd784ba8"},{"start":40200,"end":40452,"start_line":1016,"end_line":1019,"raw_tex":"\\myalert{Minimax optimization.} Another important instance of Problem \\eqref{eq:monotone} is the unconstrained minimax optimization problem \n\\begin{equation}\\label{eq:minimax}\n  \\min_{\\vx \\in \\reals^m} \\max_{\\vy \\in \\reals^n} f(\\vx,\\vy),\n\\end{equation}","sha256_utf8":"6d2857885db78116cd31e2798a1b65f319331bf03ac11704aae29203ad58fb7f"},{"start":40453,"end":41808,"start_line":1020,"end_line":1034,"raw_tex":"%\nwhere $f$ is (strongly) convex with respect to $\\vx$ and (strongly) concave with respect to $\\vy$. To reformulate this problem as \\eqref{eq:monotone}, we define the saddle point operator as $\\vF(\\vz) = (\\nabla f_{\\vx}(\\vx,\\vy),-\\nabla f_{\\vy}(\\vx,\\vy)) \\in \\reals^{m+n}$, where $\\vz = (\\vx,\\vy) \\in \\reals^{m+n}$. Indeed, finding a solution of $\\vF(\\vz)$ is equivalent to finding a stationary point of the minimax problem, which is the optimal solution. In this case, the Jacobian is   \n$\n    \\nabla \\vF(\\vz) = \n    \\begin{bmatrix}\n        \\nabla^2 f_{\\vx\\vx}(\\vx,\\vy) & \\nabla^2 f_{\\vx \\vy}(\\vx,\\vy) \\\\\n        -\\nabla^2 f_{\\vx \\vy}(\\vx,\\vy)^\\top & -\\nabla^2 f_{\\vy\\vy}(\\vx,\\vy)\n    \\end{bmatrix}$.\nAs noted in~\\cite{asl2023j}, $\\nabla \\vF(\\vz)$ exhibits $\\mJ$-symmetric structure~\\cite{mackey2003structured}, where it is symmetric on the main diagonal blocks and anti-symmetric on the off-diagonal blocks. \nSpecifically, for  $\n    \\mJ = \\begin{bmatrix}\n        \\mI_{m\\times m} & 0 \\\\\n        0 & - \\mI_{n\\times n}\n    \\end{bmatrix}$,  matrix $\\mM$ is $\\mJ$-symmetric if $ \\mJ\\mM$ is symmetric, i.e., $ \\mJ \\mM = \\mM^\\top \\mJ$. \nSimilarly, as discussed in Section~\\ref{subsec:nonlinear_eqs}, we can ensure that our Jacobian approximation matrix in our method respects the $\\mJ$-symmetry property, simplifying the implementation of some subroutines.  \n","sha256_utf8":"75f3422cab9a8c3c7cb89a7d9fe392d74d27d584ec7a5872b7798b95e5ea6fe5"},{"start":42198,"end":42687,"start_line":1041,"end_line":1045,"raw_tex":"Specifically, assume that we know the sparsity pattern of the Jacobian matrices, defined as the set\n$\n  \\Omega = \\{(i,j):\\,\\exists\\, \\vz \\in \\reals^{d}\\,\\mathrm{s.t.}\\,[\\nabla \\vF(\\vz)]_{ij} \\neq 0, \\;1\\leq i \\leq d,\\; 1\\leq j \\leq d\\}$,\nwhere $[\\nabla \\vF(\\vz)]_{ij}$ denotes the $(i,j)$-the entry of the matrix $\\nabla \\vF(\\vz)$. \nMoreover, we say a matrix $\\mM\\in \\reals^{d \\times d}$ has the sparsity pattern $\\Omega$ if $[\\mM]_{ij} = 0$ for all $(i,j) \\notin \\Omega$ and $i \\neq j$. \n","sha256_utf8":"749ae7cedec0d39aff3ca14a07826cb27ad1bb9e477e58bbaf7e30307351b31b"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"距离与各类间隙函数","source_units":[{"start":43154,"end":43385,"start_line":1054,"end_line":1055,"raw_tex":"In the strongly monotone setting (under Assumption~\\ref{assum:strong_monotone}), \nrecall that there is a unique solution $\\vz^*$ to Problem~\\eqref{eq:monotone}. Hence, we measure the suboptimality in terms of distance to $\\vz^*$. \n","sha256_utf8":"1617f1e1aca3bcec10a3b61dc2f5f190b3c897c859151ded6d997b535e374b36"},{"start":44061,"end":44199,"start_line":1058,"end_line":1060,"raw_tex":"\\begin{equation}\n  \\gap_{\\text{w}}(\\vz; \\mathcal{D}) = \\max_{\\vz' \\in \\mathcal{D}} \\langle \\vF(\\vz'),   \\vz - \\vz' \\rangle,\n\\end{equation}","sha256_utf8":"dbf4c5202bb4be6bf8f99c4a1035671ba7d48a4ef628ff01202a098005da34f4"},{"start":44996,"end":45070,"start_line":1064,"end_line":1066,"raw_tex":"\\begin{equation}\n  \\gap_f(\\vz; \\vz^*) = f(\\vz) - f(\\vz^*). \n\\end{equation}","sha256_utf8":"a6d6d77a3f06068a5c1d60e7dd52a7238bda6d6cee932a704667c330b7b42851"},{"start":45164,"end":45309,"start_line":1068,"end_line":1070,"raw_tex":"\\begin{equation}\n  \\gap_{\\text{pd}}(\\vz;\\calX\\times \\calY) = \\max_{\\vy' \\in \\calY} f(\\vx,\\vy') - \\min_{\\vx'\\in \\calX} f(\\vx',\\vy),\n\\end{equation}","sha256_utf8":"ab4efe42efcaa9c624d08262eaa92c4b8d201e3dac52483a71bc6d246326d7d5"},{"start":45310,"end":45426,"start_line":1071,"end_line":1071,"raw_tex":"where $\\vz = (\\vx,\\vy)$ and $\\calX$ and $\\calY$ are given compact sets in $\\reals^m$ and $\\reals^n$, respectively. \n","sha256_utf8":"f6f367b3be238d27e95dbcc6241af6e756228e60edf7165d83fec5688c03afa9"},{"start":46435,"end":47009,"start_line":1083,"end_line":1090,"raw_tex":"In Lemma~\\ref{lem:averaging}, we observe that the gap functions in the three cases can all be upper bounded by a similar quantity. Thus, to unify these different notions, we define  \n\\begin{equation}\\label{eq:gap}\n  \\gap(\\vz; \\mathcal{D}) = \\begin{cases}\n    \\gap_{\\mathrm{w}}(\\vz; \\mathcal{D}), & \\text{for Problem~\\eqref{eq:monotone};}\\\\\n    \\max_{\\vz' \\in \\mathcal{D}}\\{f(\\vz) - f(\\vz')\\}, & \\text{for Problem~\\eqref{eq:minimization}};\\\\\n    \\max_{(\\vx',\\vy') \\in \\calD} \\{f(\\vx,\\vy') - f(\\vx',\\vy)\\}, & \\text{for Problem~\\eqref{eq:minimax}}.\n  \\end{cases}\n\\end{equation}","sha256_utf8":"7527f8e5ce23798431eae4f36fc2359590f810d1a43cc4a2f13b58b6787f0148"},{"start":47010,"end":47245,"start_line":1091,"end_line":1091,"raw_tex":"Specifically, note that $\\gap(\\vz; \\{\\vz^*\\})$ reduces to $\\gap_f(\\vz;\\vz^*)$ for Problem~\\eqref{eq:minimization} and $\\gap(\\vz; \\calX\\times \\calY)$ reduces to $\\gap_{\\text{pd}}(\\vz;\\calX\\times \\calY)$ for Problem~\\eqref{eq:minimax}. \n","sha256_utf8":"32ee5175992efe4b2c153e4b3699e14a6f74089bdd220d7ec7e60ecc72d63888"}]},{"id":"S007","semantic_type":"algorithm","kind":"algorithm","title":"QNPE迭代及非精确条件","source_units":[{"start":51153,"end":53695,"start_line":1142,"end_line":1184,"raw_tex":"\\begin{algorithm}[t!]\\small\n  \\caption{Quasi-Newton Proximal Extragradient (QNPE) Method (informal)}\\label{alg:Full_Equasi-Newton}\n    \\begin{algorithmic}[1]\n        \\STATE \\textbf{Input:} strong monotonicity parameter $\\mu \\geq 0$, Jacobian feasible set $\\mathcal{Z}$, line search parameters $\\alpha_1\\geq 0$ and $\\alpha_2>0$ such that $\\alpha_1+\\alpha_2 < 1$,  and initial trial step size $\\sigma_0>0$ \n        \\STATE \\textbf{Initialization:} initial point $\\vz_0\\in \\mathbb{R}^d$ and initial Jacobian approximation $\\mB_0\\in \\mathcal{Z}$ %\n        %\n        \\FOR{iteration $k=0,\\ldots,N-1$}\n        %\n        %\n        \\STATE Let $\\eta_k$ be the largest possible step size in $\\{\\sigma_k\\beta^i:i\\geq 0\\}$ such that  \\tikzmark{top} \n        %\n        \\begin{gather*}\n          \\hspace{-8em}\\|{\\hat{\\vz}_k}-\\vz_k+{\\eta_k}(\\vF(\\vz_k)+\\mB_k({\\hat{\\vz}_k}-\\vz_k))\\| \\leq \\alpha_1 \\sqrt{1+\\eta_k \\mu} \\|\\hat{\\vz}_k-\\vz_k\\|, \\tikzmark{right} %\n          \\\\\n          \\hspace{-8em}\\|\\hat{\\vz}_k - \\vz_k + \\eta_k \\vF(\\hat{\\vz}_k)\\| \\leq { (\\alpha_1+\\alpha_2)\\sqrt{1+\\eta_k \\mu} \\|{\\hat{\\vz}_k}-\\vz_k\\|}\n           %\n      \\end{gather*}%\n\n%\n        \\STATE Set $\\sigma_{k+1} \\leftarrow \\eta_{k}/\\beta$ \\tikzmark{bottom}\n        \\STATE Update \n        $\n          \\vz_{k+1} \\leftarrow \\theta_k (\\vz_k - \\eta_k \\vF(\\hat{\\vz}_k))+(1-\\theta_k) \\hat{\\vz}_k\n        $, where $\\theta_k = \\frac{1}{1+2\\eta_k\\mu}$\n        %\n        %\n        \\IF[Line search accepted the initial trial step size]{$\\eta_k = \\sigma_k$ \\tikzmark{top2}}\n        \\STATE Set $\\mB_{k+1} \\leftarrow \\mB_k$ \\label{line:Hessian_approx_unchanged}\n        \\ELSE[Line search bactracked]\n        \\STATE Let $\\tilde{\\vz}_k$ be the last rejected iterate in the line search  %\n        \\STATE Set $\\vu_k \\leftarrow \\vF(\\tilde{\\vz}_k) -\\vF({\\vz_k})$, $\\vs_k \\leftarrow \\tilde{\\vz}_k-\\vz_k$\n        \\STATE Define the loss function $\\ell_k(\\mB) = \\frac{\\|\\vu_k-\\mB\\vs_k\\|^2}{\\|\\vs_k\\|^2}$ \n        \\STATE Feed $\\ell_k(\\mB)$ to an online learning algorithm $\\mathcal{A}$ and obtain $\\mB_{k+1}$ \\quad \\tikzmark{right2}\n        \\ENDIF \\tikzmark{bottom2}\n        \\ENDFOR\n                 \n    \\end{algorithmic}\n    \\AddNote{top}{bottom}{right}{\\hspace{-.5em}\\color{comment}\\textit{\\quad Line search subroutine;\\\\\n             \\hspace{.2em} see Section~\\ref{subsec:ls}}}\n    \\AddNote{top2}{bottom2}{right2}{\\hspace{-.5em}\\color{comment}\\textit{\\quad Jacobian approximation\\\\ \\hspace{.2em} update subroutine; see \\\\\n    \\hspace{.2em} Section~\\ref{sec:jacobian_online_learning}}}\n    %\n  \\end{algorithm}","sha256_utf8":"19ee4be9420cc8edbad5dec6ace028193cc015eec22d51bcae60830abbcdadcc"},{"start":55418,"end":55779,"start_line":1201,"end_line":1205,"raw_tex":"\\begin{align}\n    \\|{\\hat{\\vz}_k}-\\vz_k+{\\eta_k}(\\vF(\\vz_k)+\\mB_k({\\hat{\\vz}_k}-\\vz_k))\\| &\\leq {\\alpha_1} \\sqrt{1+\\eta_k \\mu} \\|\\hat{\\vz}_k-\\vz_k\\|, \\label{eq:inexact_linear_solver} \\\\\n    \\|\\hat{\\vz}_k - \\vz_k + \\eta_k \\vF(\\hat{\\vz}_k)\\| &\\leq {(\\alpha_1+\\alpha_2) \\sqrt{1+\\eta_k \\mu} \\|{\\hat{\\vz}_k}-\\vz_k\\|} \\label{eq:inexact_proximal_point}, \n%\n\\end{align}","sha256_utf8":"8bf0213f4df432f3cb4f25863edafaf966164c5ffc426563138063849b686e57"},{"start":55780,"end":55891,"start_line":1206,"end_line":1206,"raw_tex":"where $\\alpha_1\\in [0,1)$ and $\\alpha_2 \\in (0,1)$ are user-specified parameters with $\\alpha_1+\\alpha_2<1$.  \n","sha256_utf8":"3ee73c0f130270baaa7e3aa8121066811e5f4bf6f38443797a4270697ddd1d66"},{"start":57402,"end":57547,"start_line":1216,"end_line":1219,"raw_tex":"\\begin{equation}\\label{eq:extragradient_mixing}\n%\n  \\vz_{k+1} = \\theta_k(\\vz_k - \\eta_k \\vF(\\hat{\\vz}_k))+(1-\\theta_k)\\hat{\\vz}_k,\n\\end{equation}","sha256_utf8":"b722d49886605a1fc7aba479ee78ea1dea024bcff0ae6fe3838fa206b92241ef"},{"start":57548,"end":57962,"start_line":1220,"end_line":1220,"raw_tex":"where $\\theta_k \\in [0,1]$ is chosen based on our convergence analysis. Specifically, in the monotone setting (under Assumption~\\ref{assum:monotone}), we choose $\\theta_k = 1$ and \\eqref{eq:extragradient_mixing} reduces to the extragradient step in \\eqref{eq:extragradient}. Moreover, in the strongly monotone setting (under Assumption~\\ref{assum:strong_monotone}), we choose $\\theta_k  = \\frac{1}{1+2\\eta_k\\mu}$.\n","sha256_utf8":"272e0e3aaa81a78fed9ebbd990cc500428e6d42bd243a7eb6657895e2caaec5e"}]},{"id":"S008","semantic_type":"algorithm","kind":"algorithm","title":"回溯搜索与线性求解接口","source_units":[{"start":59742,"end":61008,"start_line":1244,"end_line":1259,"raw_tex":"\\begin{algorithmic}[1]\n      \\STATE \\textbf{Input:} iterate $\\vz \\in \\mathbb{R}^d$,  operator $\\vg\\in \\reals^d$,  Jacobian approximation $\\mB$, initial trial step size $\\sigma>0$\n      \\STATE \\textbf{Parameters:} line search parameters $\\beta\\in (0,1)$, $\\alpha_1\\geq 0$ and $\\alpha_2>0$ such that $\\alpha_1+\\alpha_2<1$\n      %\n      \\STATE Set ${\\eta}_{+} \\leftarrow \\sigma$, {$\\vs_{+} \\leftarrow \\mathsf{LinearSolver}(\\mI+\\eta_{+}\\mB, -\\eta_{+}\\vg; {\\alpha_1}\\sqrt{1+\\eta_+ \\mu})$ and $\\hat{\\vz}_{+} \\leftarrow \\vz+\\vs_{+}$}\n      \\WHILE{$\\|\\hat{\\vz}_+ - \\vz + \\eta_+ \\vF(\\hat{\\vz}_+)\\| \\geq (\\alpha_1+\\alpha_2)\\sqrt{1+\\eta_+ \\mu} \\|{\\hat{\\vz}_{+}}-\\vz\\|$\\label{line:check_condition}}\n        \\STATE Set $\\tilde{\\vz} \\leftarrow \\hat{\\vz}_{+}$ %\n          and $\\eta_{+} \\leftarrow \\beta\\eta_+ $ \\label{line:decreasing_stepsize}\n      \\STATE Compute $\\vs_{+} \\leftarrow \\mathsf{LinearSolver}(\\mI+\\eta_{+}\\mB, -\\eta_{+}\\vg; \\alpha_1\\sqrt{1+\\eta_+ \\mu})$ and $\\hat{\\vz}_{+} \\leftarrow \\vz+\\vs_{+}$\n      \\ENDWHILE\n      \\IF{$\\eta_+ = \\sigma$}\n      \\STATE \\textbf{Return} $\\eta_{+}$ and  $\\hat{\\vz}_{+}$ \\label{line:return_1}\n      \\ELSE \n      \\STATE \\textbf{Return} $\\eta_{+}$, $\\hat{\\vz}_{+}$ and $\\tilde{\\vz}$ \\label{line:return_2}\n      \\ENDIF\n  \\end{algorithmic}","sha256_utf8":"ad485b8be3aefeba0506cbbe2dcbca10de2e4af693aab892c30cff2f622356d2"},{"start":61027,"end":61797,"start_line":1262,"end_line":1270,"raw_tex":"Next, we present the backtracking line search scheme for selecting $\\eta_k$ and the iterate $\\hat{\\vz}_k$ in the first stage of QNPE. For brevity, we denote $\\vF(\\vz_k)$ by $\\vg$ and omit the iteration subscript $k$ from both $\\vz_k$ and $\\mB_k$. %\nIn light of \\eqref{eq:inexact_linear_solver} and \\eqref{eq:inexact_proximal_point}, our goal at the $k$-th iteration is to identify a pair $(\\eta_{+}, \\hat{\\vz}_{+})$ satisfying  \n%\n\\begin{align}\n  %\n  \\|\\hat{\\vz}_{+}-\\vz+{\\eta_{+}}(\\vg+\\mB({\\hat{\\vz}_{+}}-\\vz))\\| &\\leq  {\\alpha_1}\\sqrt{1+\\eta_+ \\mu} \\|{\\hat{\\vz}_{+}}-\\vz\\|,\n  \\label{eq:x_plus_update} \\\\\n  \\|\\hat{\\vz}_+ - \\vz + \\eta_+ \\vF(\\hat{\\vz}_+)\\| &\\leq (\\alpha_1+\\alpha_2)\\sqrt{1+\\eta_+ \\mu} \\|{\\hat{\\vz}_{+}}-\\vz\\|  \\label{eq:step size_condition}. \n\\end{align}","sha256_utf8":"42ff624ab576ef76e8032c30f4a403d27581526d6dfa0ad66759fcb9769c3976"},{"start":61798,"end":62007,"start_line":1271,"end_line":1271,"raw_tex":"As previously discussed, with $\\eta_{+}$ fixed, the first condition in \\eqref{eq:x_plus_update} can be met by solving the linear system $(\\mI+\\eta_{+}\\mB)(\\hat{\\vz}_+-\\vz) = -\\eta_+\\vg$ to a desired accuracy.\n","sha256_utf8":"a76cf956036650002274442ee3ec8497cfd4dec305888f59a7db9e7f8c6fd9ef"},{"start":62036,"end":62250,"start_line":1276,"end_line":1279,"raw_tex":"\\begin{equation}\\label{eq:linear_solver_update}\n  \\vs_{+} = \\mathsf{LinearSolver}(\\mI+\\eta_{+}\\mB, -\\eta_{+}\\vg; {\\alpha_1}\\sqrt{1+\\eta_+ \\mu}) \\quad \\text{and} \\quad \\hat{\\vz}_{+} = \\vz+\\vs_{+},\n  %\n\\end{equation}","sha256_utf8":"c850ccd4d186c4a64114664219bb184ac7047431030aab70024e147896d9abc2"},{"start":62251,"end":62317,"start_line":1280,"end_line":1280,"raw_tex":"where the $\\mathsf{LinearSolver}$ oracle is defined as follows. %\n","sha256_utf8":"38920ab11ac75d3afb53e6ff4937c8ee825ce2c6249f936c4c483dcfc7566d78"},{"start":62319,"end":62660,"start_line":1283,"end_line":1293,"raw_tex":"\\begin{definition}\\label{def:linear_solver}\n  The oracle $\\mathsf{LinearSolver}(\\mA,\\vb; \\rho)$ takes  a matrix $\\mA \\in \\reals^{d\\times d}$, a vector $\\vb\\in \\reals^d$ and $\\rho>0$ as input, and returns an approximate solution $\\vs_{+}$ \n  %\n  satisfying $\\|\\mA\\vs_{+}-\\vb\\| \\leq \\rho \\|\\vs_{+}\\|$. \n  %\n  %\n  %\n  %\n  %\n  %\n\\end{definition}","sha256_utf8":"cc80dca79490851acf0f6091c231e39ebc0b45ddf359524fec1971383df50ff8"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"回溯指标集与矩阵损失","source_units":[{"start":75808,"end":75938,"start_line":1446,"end_line":1446,"raw_tex":"Recall that $\\calB$ is the set of indices where the line search subroutine backtracks, i.e., $\\calB = \\{k: \\eta_k < \\sigma_k\\}$. \n","sha256_utf8":"2b41e200b5f235eff60d5f1cad78868fa020143f41ecb547417ef7890b04d443"},{"start":90071,"end":90176,"start_line":1700,"end_line":1700,"raw_tex":"Formally, define the loss function $\\ell_k : \\reals^{d \\times d} \\rightarrow \\reals$ at iteration $k$ as\n","sha256_utf8":"efe574605b474ad0f35986bbb49712219315391c1b9228c387d67e0584a1dc93"},{"start":90178,"end":90406,"start_line":1702,"end_line":1709,"raw_tex":"\\begin{equation}\\label{eq:loss_of_jacobian}\n  \\ell_k(\\mB) \\triangleq \n  \\begin{cases}\n    0, & \\text{if } k\\notin \\mathcal{B}, \\\\\n    \\frac{\\|\\vu_k-\\mB\\vs_k\\|^2}{\\|\\vs_k\\|^2}, & \\text{otherwise}.\n  \\end{cases}\n  %\n\\end{equation}","sha256_utf8":"8270f5a03c70c54a33f580344b2b77c6f1c609bc2461b97e63b5c63d5ab293f1"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"雅可比近似的可行集合","source_units":[{"start":94569,"end":94791,"start_line":1739,"end_line":1741,"raw_tex":"Given these points and the discussions in Section~\\ref{subsec:structures}, we choose the feasible set~$\\calZ$ as follows: \n\\begin{enumerate}[(a)]\n  \\item For the minimization problem in \\eqref{eq:minimization}, we choose \n","sha256_utf8":"f50cb4d526ff2dd0ad9d3b565e2d5de1e9b2685e05920a79cca1751a4935a709"},{"start":94793,"end":94941,"start_line":1742,"end_line":1744,"raw_tex":"\\begin{equation}\\label{eq:feasible_set_min}\n    \\calZ = \\{\\mB \\in \\sS^d: \\; \\mu \\mI \\preceq \\mB \\preceq L_1 \\mI,\\; \\mB \\in \\calL\\},\n  \\end{equation}","sha256_utf8":"0dcb3297008d2a242861c66e09ab623017932657422c2725860e3b760128172c"},{"start":94942,"end":95349,"start_line":1745,"end_line":1746,"raw_tex":"  where $\\calL \\subset \\sS^d$ is a linear subspace. in the general case, we set $\\calL = \\sS^d$. However, when the Hessians of $f$ exhibit the sparsity pattern $\\Omega$, we define $\\calL$ as $\\calL=\\{\\mB\\in \\sS^{d}: [\\mB]_{ij} = 0 \\text{ for all } (i,j)\\notin \\Omega \\text{ and }i\\neq j\\}$.\n  \\item For the minimax problem in \\eqref{eq:minimax} and the nonlinear equation in \\eqref{eq:monotone}, we choose \n","sha256_utf8":"051949f810115eb6fb6c34c1089fcbed42660fac5b6e4697d36334df0353f67c"},{"start":95351,"end":95583,"start_line":1747,"end_line":1750,"raw_tex":"\\begin{equation}\\label{eq:feasible_set}\n    \\mathcal{Z} = \\left\\{ \\mB \\in \\reals^{d\\times d}:\\mu \\mI \\preceq \\frac{1}{2}(\\mB + \\mB^\\top )\\preceq L_1\\mI , \\;\\|\\mB\\|_{\\op} \\leq L_1, \\mB \\in \\mathcal{L} \\right\\},\n    %\n  \\end{equation}","sha256_utf8":"36bbf1824d2d3b204cd9a5edc17a61391ed8aae60259a858fe872dc27b629e95"},{"start":95584,"end":95978,"start_line":1751,"end_line":1752,"raw_tex":"  where $\\mathcal{L} \\subset \\reals^{d\\times d}$ is a linear subspace. For general nonlinear equations $\\mathcal{L} = \\reals^{d \\times d}$, for minimax problems $\\mathcal{L} = \\{\\mB\\in \\reals^{d\\times d}: \\mB = \\mJ \\mB^\\top \\mJ\\}$, and for sparse nonlinear equations $\\calL=\\{\\mB\\in \\reals^{d\\times d}: [\\mB]_{ij} = 0 \\text{ for all } (i,j)\\notin \\Omega \\text{ and }i\\neq j\\}$.\n\\end{enumerate}\n","sha256_utf8":"7d3008c6941a1d10b92662356417784e5e15b31fcbc02773a24e83c93c65a91b"}]},{"id":"S011","semantic_type":"algorithm","kind":"algorithm","title":"分离预言机在线学习框架","source_units":[{"start":96824,"end":98508,"start_line":1793,"end_line":1821,"raw_tex":"\\begin{algorithm}[!t]\\small\n  \\caption{Projection-Free Online Learning}\\label{alg:projection_free_online_learning}\n  \\begin{algorithmic}[1]\n      \\STATE \\textbf{Input:} Initial point $\\vw_0 \\in  \\vspan(\\calC)$, the orthogonal projection matrix $\\mP$ associated with the subspace $\\vspan(\\calC)$, step size $\\rho>0$, $\\delta>0$, radius $R>0$\n      %\n      %\n      \\FOR{$t=0,1,\\dots T-1$}\n      \\STATE Query the oracle $(\\gamma_t,\\vs_t) \\leftarrow \\mathsf{SEP}_{\\calC}(\\vw_t;\\delta)$\n      \\IF[Case I: we have $\\vw_t\\in (1+\\delta)\\mathcal{C}$]{$\\gamma_t \\leq 1$}\n      %\n      %\n        \\STATE Set $\\vx_t \\leftarrow \\begin{cases} \\vw_t & (\\textbf{Option I}) \\\\ \\frac{\\vw_t}{1+\\delta} & (\\textbf{Option II}) \\end{cases}$ and play the action $\\vx_t$ \n        \\STATE Receive the loss $\\ell_t(\\vx_t)$ and the subspace gradient $\\vg_t = \\mP\\nabla \\ell_t(\\vx_t)$\n        \\STATE Set $\\tilde{\\vg}_t \\leftarrow \\vg_t$\n      %\n      %\n      \\ELSE[Case II: we have $\\vw_t/\\gamma_t\\in (1+\\delta)\\mathcal{C}$] \n      %\n        \\STATE Set $\\vx_t \\leftarrow \\begin{cases} \\frac{\\vw_t}{\\gamma_t} & (\\textbf{Option I}) \\\\ \\frac{\\vw_t}{(1+\\delta)\\gamma_t} & (\\textbf{Option II}) \\end{cases}$ and play the action $\\vx_t$ \n      \\STATE Receive the loss $\\ell_t(\\vx_t)$ and the subspace gradient $\\vg_t = \\mP\\nabla \\ell_t(\\vx_t)$\n      \\STATE Set $\\tilde{\\vg}_t \\leftarrow \\vg_t+\\max\\left\\{0, - \\frac{1}{\\gamma_t}\\langle \\vg_t, \\vw_t \\rangle\\right\\} \\vs_t$\n      %\n      \\ENDIF\n      %\n      \\STATE Update $\\vw_{t+1} \\leftarrow \\frac{R(\\vw_t-\\rho \\tilde{\\vg}_t)}{\\max\\{\\|\\vw_t-\\rho \\tilde{\\vg}_t\\|_2,R\\}} $ \n      \\COMMENT{Online projected gradient descent}\n      \\ENDFOR\n  \\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"c89799edb9105168a6deb4dbfa4fa3a659cafd3f443ad2a3d68f7ff6bcf90779"},{"start":98510,"end":99510,"start_line":1823,"end_line":1825,"raw_tex":"To set the stage for our Jacobian approximation update algorithm, we take a detour and consider a general online learning problem over a compact feasible set $\\calC \\subset \\reals^D$. For $T$ consecutive rounds $t=0,\\dots,T-1$, a learner chooses an action $\\vx_t \\in \\reals^D$ and then observes a convex loss function $\\ell_t:\\reals^D \\rightarrow \\reals$. \n\\blue{The goal is to minimize either the \\emph{static regret} defined by $\\mathrm{Reg}_T(\\vu; \\calC) \\triangleq \\sum_{t=0}^{T-1} \\ell_t(\\vx_t) - \\sum_{t=0}^{T-1} \\ell_t(\\vu)$ with a fixed competitor $\\vx \\in \\calC$, or the \\emph{dynamic regret} defined by $\\dreg_T(\\vu_0,\\dots,\\vu_{T-1}; \\calC) \\triangleq \\sum_{t=0}^{T-1} \\ell_t(\\vx_t) - \\sum_{t=0}^{T-1} \\ell_t(\\vu_t)$ with a sequence of competitors $\\vu_t \\in \\calC$ for any $t \\geq 0$.}\nWe assume $0\\in \\mathcal{C}$ without loss of generality. Moreover, we assume that the set $\\calC$ is contained in the Euclidean ball $B_R(0):= \\{\\vw\\in \\reals^D: \\|\\vw\\|\\leq R\\}$ for some radius $R>0$.\n","sha256_utf8":"2c031611f3a1e292c21c8f5ba1c5217495d6336f5fecb076a56ddb2a10742310"},{"start":99829,"end":100234,"start_line":1830,"end_line":1830,"raw_tex":"To start, we will define the approximate separation oracle $\\SEP_{\\cal{C}}$ for a compact set $\\calC$. We use $\\vspan(\\mathcal{C})$ to denote the linear span of $\\mathcal{C}$, i.e., $\\vspan(\\calC) = \\left\\{\\sum_{i=1}^{k}  \\alpha_i \\vx_i : k \\in \\mathbb{Z}_+,\\;\\vx_i \\in \\calC,\\; \\alpha_i \\in \\reals  \\right\\}$. Intuitively, $\\vspan(\\calC)$ is the smallest linear subspace that contains the set $\\calC$. %\n","sha256_utf8":"a14e07dcb3e5b88f8468a798dc7bb49b15b42a7f20536e90ce7d01cbd2a3febc"},{"start":100236,"end":100914,"start_line":1833,"end_line":1848,"raw_tex":"\\begin{definition}\\label{def:gauge}\n  Let $\\calC$ be a compact convex set in $\\reals^D$ containing the origin. \n  The oracle $\\mathsf{SEP}_{\\mathcal{C}}(\\vw; \\delta)$ takes $\\vw\\!\\in\\! \\vspan(\\calC)$ and $\\delta>0$ and returns $\\gamma>0$ and a vector $\\vs\\in \\vspan(\\calC)$ with one of these outcomes:\n   %\n   %\n    \\begin{itemize}\n      \\item Case I: $\\gamma \\leq 1$, which implies that $\\vw \\in (1+\\delta)\\mathcal{C}$;\n       %\n      \\item Case II: $\\gamma>1$, which implies that $\\vw/\\gamma \\in (1+\\delta)\\mathcal{C} \\ $ and $\\ \\langle \\vs, \\vw-\\vx \\rangle \\geq {\\gamma-1},$  $ \\forall\\vx\\in \\mathcal{C}$. \n    \\end{itemize}\n    %\n    %\n    %\n    %\n    %\n    \\end{definition}","sha256_utf8":"a3d4087636b44d0500ad61daacbf36e44d2b3e9ae86271c8e27af902c50e399d"},{"start":105818,"end":106047,"start_line":1879,"end_line":1881,"raw_tex":"\\begin{equation}\\label{eq:w_update}\n  \\vw_{t+1}  = \\mathrm{\\Pi}_{\\mathcal{B}_R(0)}\\bigl(\\vw_t-\\rho \\nabla \\tilde{\\ell}_t(\\vw_t)\\bigr) = \\frac{R}{\\max\\{\\|\\vw_t-\\rho \\tilde{\\vg}_t\\|_2,R\\}} (\\vw_t-\\rho \\tilde{\\vg}_t).\n\\end{equation}","sha256_utf8":"143712ad471dde04a35a8607f3ccd4ad911559f0c2aa6b65410458f591d071d8"}]},{"id":"S012","semantic_type":"algorithm","kind":"algorithm","title":"在线雅可比更新与指标重编号","source_units":[{"start":120265,"end":122644,"start_line":2155,"end_line":2198,"raw_tex":"\\begin{algorithmic}[1]\n      \\STATE \\textbf{Input:} Initial matrix $\\mB_0\\in \\mathcal{Z}$, the orthogonal projection matrix $\\mP$, step size $\\rho>0$, $\\delta>0$, failure probabilities $\\{q_t\\}_{t=1}^{T-1}$, Lipschitz constant $L_1>0$ and strong monotonicity parameter $\\mu \\geq 0$\n      \\STATE \\textbf{Initialize:} set $\\mW_0 \\leftarrow \\frac{1}{L_1}(\\mB-(L_1+\\mu)\\mI)$, $\\mG_0 \\leftarrow \\frac{1}{L_1} \\mP\\nabla \\ell_0(\\mB_0)$ and $\\tilde{\\mG}_0 \\leftarrow \\mG_0$\n      \\STATE {Update $\\mW_{1} \\leftarrow \\frac{\\sqrt{d}(\\mW_0 - \\rho \\tilde{\\mG}_0)}{\\max\\{\\sqrt{d},\\|\\mW_0 - \\rho \\tilde{\\mG}_0\\|_F\\}}$ \\COMMENT{Projection onto $\\mathcal{B}_{\\sqrt{d}}(0)$}\\vspace{-.5em}\n      \\FOR{$t=1,\\dots,T-1$}\n      \\STATE Query $(\\gamma^{(1)}_t,\\mS^{(1)}_t) \\leftarrow \\mathsf{ExtEvec}(\\frac{\\mW_t+\\mW_t^\\top}{2};\\delta, \\frac{q_t}{2} )$ and $(\\gamma^{(2)}_t,\\mS^{(2)}_t) \\leftarrow \\mathsf{MaxSvec}(\\mW_t;\\delta, \\frac{q_t}{2} )$ %\n      \\STATE Let $i = \\argmax_{i\\in \\{1,2\\}} \\gamma_t^{(i)}$, and set $\\gamma_t \\leftarrow \\gamma_t^{(i)}$ and $\\mS_t \\leftarrow \\mP \\mS_t^{(i)}$\n      \\IF[Case I]{$\\gamma_t \\leq 1$}\n      %\n        \\STATE Set $\\hat{\\mB}_t \\leftarrow \\begin{cases} \\mW_t & \\text{if }\\mu>0\\;(\\textbf{Option I})\\\\ \\frac{\\mW_t}{1+\\delta} & \\text{if }\\mu=0\\; (\\textbf{Option II})\\end{cases}$ and $\\mB_t \\leftarrow {L_1}\\hat{\\mB}_t+{(L_1+\\mu)}\\mI$\n        \\STATE Set $\\mG_t \\leftarrow \\frac{1}{L_1}\\mP\\nabla \\ell_t(\\mB_t)$ and $\\tilde{\\mG}_t \\leftarrow \\mG_t$\n        %\n      \\ELSE[Case II]\n        \\STATE Set $\\hat{\\mB}_t \\leftarrow \\begin{cases} \\frac{\\mW_t}{\\gamma_t} & \\text{if }\\mu>0\\;(\\textbf{Option I})\\\\ \\frac{\\mW_t}{(1+\\delta)\\gamma_t} & \\text{if }\\mu=0\\; (\\textbf{Option II})\\end{cases}$ and $\\mB_t \\leftarrow {L_1}\\hat{\\mB}_t+{(L_1+\\mu)}\\mI$ \n        \\STATE Set $\\mG_t \\leftarrow \\frac{1}{L_1}\\mP\\nabla \\ell_t(\\mB_t)$ and $\\tilde{\\mG}_t \\leftarrow \\mG_t+\\max\\{0,-\\frac{1}{\\gamma_t}\\langle \\mG_t, \\mW_t \\rangle\\} \\mS_t$\n        %\n      \\ENDIF\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      %\n      \\STATE %\n      %\n      Update \n      $\\mW_{t+1} \\leftarrow \\frac{\\sqrt{d}(\\mW_t - \\rho \\tilde{\\mG}_t)}{\\max\\{\\sqrt{d},\\|\\mW_t - \\rho \\tilde{\\mG}_t\\|_F\\}}$ \\COMMENT{Projection onto $\\mathcal{B}_{\\sqrt{d}}(0)$}\n        %\n        %\n        %\n        %\n      %\n      \\ENDFOR}\n  \\end{algorithmic}","sha256_utf8":"ed907b4d314da5147448e9ddf6bebaf5a2b07f16db15b500c93e2f149deffe2b"},{"start":122663,"end":124471,"start_line":2201,"end_line":2206,"raw_tex":"Next, we present our online learning algorithm for updating $\\{\\mB_k\\}_{k \\geq 0}$. We follow the projection-free online learning algorithm outlined in Section~\\ref{subsec:projection_free_online_learning} and apply it to the online learning problem in Section~\\ref{subsec:convergence}.\n\\blue{Moreover, recall that $\\calB$ denotes the set of indices where the line search subroutine backtracks, and we have $\\ell_k(\\mB)=0$ when $k\\notin \\mathcal{B}$. Thus, we can simply keep $\\mB_{k+1}$ unchanged for these iterations (cf. Line~\\ref{line:Hessian_approx_unchanged} in Algorithm~\\ref{alg:Full_Equasi-Newton}). Suppose $\\calB = \\{k_0,k_1,\\dots,k_{T-1}\\}$, where $T \\leq N$. With a slight abuse of notation, in the following, we relabel the indices in $\\mathcal{B}$ as $t=0,\\dots,T-1$.}\n\nThe remaining question is how to construct the approximate separation oracle, required by the projection-free scheme and defined in Definition~\\ref{def:gauge}, for our specific set $\\calZ$.\nTo start, we consider the nonlinear monotone equation in \\eqref{eq:monotone}; we will further discuss the special cases in the next section.  \nRecall that the feasible set~$\\calZ$ for our online learning problem in Section~\\ref{subsec:convergence} is given in \\eqref{eq:feasible_set} with a linear subspace $\\calL$. In the following, we assume that the orthogonal projection matrix $\\mP$ associated with the linear subspace $\\calL$ is given. Specifically, for a general nonlinear monotone equation, we have $\\calL = \\reals^d$ and the matrix $\\mP$ is simply the identity matrix. Moreover, for those special cases discussed in Section~\\ref{subsec:structures}, the linear subspace is specified at the end of Section~\\ref{subsec:convergence} and we will discuss the corresponding projection matrix in Section~\\ref{subsec:special_cases}. \n","sha256_utf8":"a5b84d1dd7908e76ef591b68b200b56598a61e241625c81c220071459ee27e5e"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"谱分离预言机及集合分解","source_units":[{"start":127167,"end":127575,"start_line":2255,"end_line":2257,"raw_tex":"Hence, we will work with the new set $\\calC$ in our online learning algorithm. Moreover, note that we can write $\\calC = \\calC_1 \\cap \\calC_2 \\cap \\calL$, where we define \n%\n  $\\mathcal{C}_1 \\mydef \\{\\hat{\\mB} \\in \\mathbb{R}^{d\\times d}:  -\\mI \\preceq \\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top )\\preceq \\mI \\}$ and $\\mathcal{C}_2 \\mydef \\{\\hat{\\mB} \\in \\mathbb{R}^{d\\times d}: \\|\\hat{\\mB}\\|_{\\op} \\leq 3 \\}$.  \n","sha256_utf8":"9ca3e0174df69f07fa317f8b9cc7e58e8759e76ff394c229730c471be16e024c"},{"start":129770,"end":130629,"start_line":2275,"end_line":2294,"raw_tex":"\\begin{definition}\\label{def:extevec}\n  The oracle $\\mathsf{ExtEvec}(\\mW;\\delta,q)$ takes $\\mW \\in \\reals^{d\\times d}$, $\\delta>0$, and $q\\in (0,1)$ as input and returns a scalar $\\gamma>0$ and a matrix $\\mS\\in \\mathbb{S}^d$. With probability at least $1-q$, the returned $\\gamma$ and $\\mS$ satisfy one of the following properties::\n  %\n  \\begin{itemize}\n    \\item Case I: $\\gamma \\leq 1$, which implies that\n    %\n    $-(1+\\delta)\\mI \\preceq \\frac{1}{2}(\\mW+\\mW^\\top) \\preceq (1+\\delta)\\mI$;\n    %\n    \\item Case II: $\\gamma>1$, which implies that %\n    %\n    $-(1+\\delta)\\mI \\preceq \\frac{1}{2\\gamma}(\\mW+\\mW^\\top) \\preceq (1+\\delta)\\mI$, \n    $\\|\\mS\\|_F \\leq 1$ and $\\langle \\mS,\\mW-\\hat{\\mB}\\rangle \\geq \\gamma -1$ for any $\\hat{\\mB}$ such that $-\\mI \\preceq \\frac{1}{2}(\\hat{\\mB}+\\hat{\\mB}^\\top) \\preceq \\mI$.\n  \\end{itemize}\n%\n%\n%\n%\n%\n%\n\\end{definition}","sha256_utf8":"4472affc88894f00a9a0304669b7f605934042fedc5ed16a4091fff71ab65593"},{"start":132399,"end":133148,"start_line":2302,"end_line":2310,"raw_tex":"\\begin{definition}\\label{def:MaxSvec}\n  The oracle $\\mathsf{MaxSvec}(\\mW;\\delta,q)$ takes $\\mW \\in \\mathbb{R}^{d\\times d}$, $\\delta>0$, and $q\\in (0,1)$ as input and returns a scalar $\\gamma>0$ and a matrix $\\mS\\in \\mathbb{R}^{d\\times d}$. With probability at least $1-q$, the returned $\\gamma$ and $\\mS$ satisfy one of the following properties::\n  %\n  \\begin{itemize}\n    \\item Case I: $\\gamma \\leq 1$, which implies that $ \\|\\mW\\|_{\\op} \\leq 3(1+\\delta) $;\n    %\n    \\item Case II: $\\gamma>1$, which implies that $ \\|\\mW/\\gamma \\|_{\\op} \\leq 3(1+\\delta)$, $\\|\\mS\\|_F \\leq 1$ and $\\langle \\mS,\\mW-\\hat{\\mB}\\rangle \\geq \\gamma -1$ for any $\\hat{\\mB} \\in \\reals^{d\\times d}$ such that $ \\|\\hat{\\mB}\\|_{\\op}  \\leq 3 $.\n  \\end{itemize}\n\\end{definition}","sha256_utf8":"8732120fb2ec7f35f54d591add6397ec16060bf8200dc7b882c8f9d3c785944a"}]},{"id":"S014","semantic_type":"algorithm","kind":"algorithm","title":"两个随机Lanczos预言机","source_units":[{"start":144946,"end":146889,"start_line":2515,"end_line":2541,"raw_tex":"\\begin{algorithmic}[1]\n      \\STATE \\textbf{Input:} $\\mW \\in \\reals^{d\\times d}$, $\\delta>0$, $q\\in (0,1)$\n      \\STATE \\textbf{Initialize:} sample $\\vv_1\\in \\reals^d$ uniformly from the unit sphere, $\\beta_1 \\leftarrow 0$, $\\vv_0\\leftarrow 0$\n      \\STATE Set $\\bar{\\mW} \\leftarrow \\frac{1}{2}(\\mW + \\mW^\\top)$ and set the number of iterations \n      $N \\leftarrow \\Bigl\\lceil\\frac{1}{4}\\sqrt{2(1+\\frac{1}{\\delta})}\\log(\\frac{11d}{q^2})+\\frac{1}{2} \\rceil$\n      %\n      %\n      %\n      %\n      \\FOR{$k=1,\\dots,N$ \\tikzmark{top}}\n      \\STATE Set $\\vw_k \\leftarrow \\bar{\\mW} \\vv_k-\\beta_k \\vv_{k-1}$ \n      \\STATE Set $\\alpha_k \\leftarrow \\langle \\vw_k,\\vv_k \\rangle $ and $\\vw_k \\leftarrow \\vw_k-\\alpha_k\\vv_k$\n      \\STATE Set $\\beta_{k+1} \\leftarrow \\|\\vw_k\\|$ and $\\vv_{k+1}\\leftarrow \\vw_k/\\beta_{k+1}$\n      \\ENDFOR\n    \\STATE Form a tridiagonal matrix $\\mT \\leftarrow \\mathsf{tridiag}(\\beta_{2:N},\\alpha_{1:N},\\beta_{2:N})$\n    \\STATE \\hspace{-1em}\\COMMENT{Use the tridiagonal structure to compute eigenvectors of $\\mT$}\n    \\STATE Compute $(\\hat{\\lambda}_1,\\vz^{(1)}) \\leftarrow \\mathsf{MaxEvec}(\\mT)$ and $(\\hat{\\lambda}_d, \\vz^{(d)}) \\leftarrow \\mathsf{MinEvec}(\\mT)$ \\tikzmark{right}\n    \\STATE Set $\\vu^{(1)} \\leftarrow \\sum_{k=1}^N z^{(1)}_k\\vv_k$ and $\\vu^{(d)} \\leftarrow \\sum_{k=1}^N z^{(d)}_k\\vv_k$ \\tikzmark{bottom}\n    \\STATE Set $\\gamma \\leftarrow \\max\\{\\hat{\\lambda}_1,-\\hat{\\lambda}_d\\}$ \n    \\IF{$\\gamma \\leq 1$}\n    \\STATE Return $\\gamma$ and $\\mS = 0$ \\COMMENT{Case I: $\\gamma\\leq 1$, which implies $\\|\\mW\\|_{\\op} \\leq 1+\\delta$}\n    \\ELSIF{$\\hat{\\lambda}_1 \\geq -\\hat{\\lambda}_d$}\n      \\STATE Return $\\gamma$ and $\\mS = \\vu^{(1)}(\\vu^{(1)})^\\top$ \\hspace{.75em}\\COMMENT{Case II: $\\gamma> 1$ and $\\mS$ defines a separating hyperplane}\n    \\ELSE\n      \\STATE Return $\\gamma$ and $\\mS = -\\vu^{(d)}(\\vu^{(d)})^\\top$ \\COMMENT{Case II: $\\gamma> 1$ and $\\mS$ defines a separating hyperplane}\n    \\ENDIF\n  \\end{algorithmic}","sha256_utf8":"d877a217bde80ca0e7ed667edf5116d837ab2d393420e904cc0c2542b283b527"},{"start":153362,"end":154589,"start_line":2637,"end_line":2669,"raw_tex":"\\begin{algorithmic}[1]\n    \\STATE \\textbf{Input:} $\\mW \\in \\mathbb{R}^{d\\times d}$, $\\delta>0$, $q\\in (0,1)$\n    %\n    \\STATE Set $\\tilde{\\mW} \\leftarrow \\begin{bmatrix} 0 & \\mW \\\\ \\mW^\\top & 0\n    \\end{bmatrix} \\in \\mathbb{S}^{2d}$ and the number of iterations $N  \\leftarrow \n    %\n    \\Bigl\\lceil \\frac{1}{4}\\sqrt{2(1+\\frac{1}{\\delta})}\\log\\frac{22d}{q^2}+\\frac{1}{2}\\Bigr\n    \\rceil\n    %\n    $\n    \n    \\STATE Compute approximate largest eigenvalue $\\tilde{\\lambda}_1$ and its eigenvector $\\tilde{\\vv} \\in \\reals^{2d}$, by running the Lanczos method on $\\tilde{\\mW}$ for $N$ iterations \\COMMENT{See Subroutine~\\ref{alg:lanczos}}\n    \\STATE Partition $\\tilde{\\vv}$ as $\\tilde{\\vv} = [\\vv,\\vv']$ where $\\vv, \\vv' \\in \\reals^d$ \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    \\STATE Set $\\gamma \\leftarrow \\tilde{\\lambda}_1/3$ \n    \\IF{$\\gamma \\leq 1$}\n    \\STATE Return $\\gamma$ and $\\mS = 0$ \\COMMENT{Case I: $\\gamma\\leq 1$, which implies $\\|\\mW\\|_{\\op} \\leq 3(1+\\delta)$}\n    \\ELSE\n      \\STATE Return $\\gamma$ and $\\mS = \\frac{2}{3}\\vv'\\vv^\\top \\in \\reals^{d\\times d}$ \\hspace{.75em}\\COMMENT{Case II: $\\gamma> 1$ and $\\mS$ defines a separating hyperplane}\n    %\n    %\n    \\ENDIF\n  \\end{algorithmic}","sha256_utf8":"478467f444bb80a4a3d4ac943543b8191c8f50854bbff199e2fc406e52070986"}]},{"id":"S015","semantic_type":"lemma","kind":"lemma","title":"雅可比的谱约束","source_units":[{"start":37905,"end":38371,"start_line":997,"end_line":999,"raw_tex":"\\begin{lemma}\\label{lem:jacobian}\nUnder Assumption~\\ref{assum:strong_monotone}, we have $\\frac{1}{2}(\\nabla \\vF(\\vz) + \\nabla \\vF(\\vz)^\\top) \\succeq \\mu \\mI$ for any $\\vz\\in \\reals^d$, while under Assumption~\\ref{assum:monotone}, we have $\\frac{1}{2}(\\nabla \\vF(\\vz) + \\nabla \\vF(\\vz)^\\top) \\succeq 0$ for any $\\vz\\in \\reals^d$. Further, under Assumption~\\ref{assum:operator_lips}, we have $\\|\\nabla \\vF(\\vz)\\|_{\\op} \\leq L_1$ for any $\\vz\\in \\reals^d$. \n\\end{lemma}","sha256_utf8":"cd516f6bb85fcbed89ce8b2be1130d5b49887337866b350cd01edc759a434d47"}]},{"id":"S016","semantic_type":"lemma","kind":"lemma","title":"加权平均的间隙界","source_units":[{"start":45560,"end":46434,"start_line":1073,"end_line":1082,"raw_tex":"\\begin{lemma}\\label{lem:averaging}\n  Suppose Assumption~\\ref{assum:monotone} holds. \n    Let $\\theta_0, \\dots, \\theta_{T-1} \\geq 0$ with $\\sum_{t=0}^{T-1} \\theta_t = 1$ and let $\\vz_0,\\dots,\\vz_{T-1}\\in \\reals^d$. Define the averaged iterates as $\\bar{\\vz}_T = \\sum_{t=0}^{T-1} \\theta_t \\vz_t$. Then: \n    \\begin{enumerate}[(i)]\n      \\item For Problem~\\eqref{eq:monotone}, $\\gap_{\\mathrm{w}}(\\vz; \\mathcal{D}) \\leq \\max_{\\vz \\in \\calD}\\sum_{t=0}^{T-1} \\theta_t \\langle \\vF(\\vz_t), \\vz_t-\\vz\\rangle$. \n      \\item For Problem~\\eqref{eq:minimization}, $\\gap_{f}(\\vz; \\vz^*) \\leq \\sum_{t=0}^{T-1} \\theta_t \\langle \\vF(\\vz_t), \\vz_t-\\vz^*\\rangle$. \n      \\item For Problem~\\eqref{eq:minimax}, $\\gap_{\\mathrm{pd}}(\\vz;\\calX\\times \\calY) \\leq \\max_{\\vz \\in \\calX \\times \\calY}\\sum_{t=0}^{T-1} \\theta_t \\langle \\vF(\\vz_t), \\vz_t-\\vz\\rangle$. \n    \\end{enumerate} \n   %\n\\end{lemma}","sha256_utf8":"4bbcec8bbfca61ff7b174b8a93fe3f93891eb37e07127c6e497392ea63feea89"}]},{"id":"S017","semantic_type":"lemma","kind":"lemma","title":"QNPE单步能量不等式","source_units":[{"start":66350,"end":67172,"start_line":1330,"end_line":1341,"raw_tex":"\\begin{lemma}\\label{lem:one_step}\n  Suppose that the operator $\\vF$ in \\eqref{eq:monotone} is $\\mu$-strongly monotone with $\\mu \\geq 0$.  \n  Let $\\{\\vz_k\\}_{k\\geq 0}$ and $\\{\\hat{\\vz}_k\\}_{k\\geq 0}$ be the iterates generated by Algorithm~\\ref{alg:Full_Equasi-Newton}, where $\\alpha_1\\in [0,1)$, $\\alpha_2 \\in (0,1)$ and $\\alpha_1+\\alpha_2 <1$. Then for any $\\vz \\in \\reals^d$ and $k \\geq 0$, it holds that\n  \\begin{equation}\\label{eq:one_step}\n    \\begin{aligned}\n      \\eta_k \\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz \\rangle &\\leq \\frac{\\|\\vz_k\\!-\\! \\vz\\|^2}{2} - \\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1} - \\vz\\|^2 + \\eta_k\\mu \\|\\hat{\\vz}_k - \\vz\\|^2 \\\\\n      & \\phantom{{}\\leq{}} \n      - \\frac{1-\\alpha_1-\\alpha_2}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2).\n    \\end{aligned}\n  \\end{equation} \n\n\\end{lemma}","sha256_utf8":"24070a5d43f78d01e0f2493287b514e7f2418998a49703dbc700ed67879b3f35"}]},{"id":"S018","semantic_type":"proof","kind":"proof","title":"单步能量引理证明","source_units":[{"start":67173,"end":70972,"start_line":1342,"end_line":1389,"raw_tex":"\\begin{proof}\n  To simplify the notation, let $\\alpha \\triangleq \\alpha_1+\\alpha_2\\in (0,1)$. For any $\\vz\\in \\reals^d$, we have \n\\begin{equation}\\label{eq:inexact_pp_decomp}\n  \\eta_k \\langle \\vF(\\hat{\\vz}_k), \\hat{\\vz}_k-\\vz \\rangle =  \\langle \\hat{\\vz}_k-\\vz_k+\\eta_k \\vF(\\hat{\\vz}_k), \\hat{\\vz}_k-\\vz \\rangle\n  +\\langle \\vz_k-\\hat{\\vz}_k, \\hat{\\vz}_k-\\vz\\rangle. %\n   %\n   %\n\\end{equation}\nFor the first term in \\eqref{eq:inexact_pp_decomp}, we can bound it by\n\\begin{align}\n  \\langle \\hat{\\vz}_k-\\vz_k+\\eta_k \\vF(\\hat{\\vz}_k), \\hat{\\vz}_k-\\vz \\rangle &\\leq \\|\\hat{\\vz}_k-\\vz_k+\\eta_k\\vF(\\hat{\\vz}_k)\\|\\|\\hat{\\vz}_k-\\vz\\| \\nonumber\\\\\n  &\\leq \\alpha\\sqrt{1+\\eta_k\\mu}\\|\\hat{\\vz}_k-\\vz_k\\|\\|\\hat{\\vz}_k-\\vz\\| \\nonumber\\\\\n  &\\leq {\\frac{\\alpha}{2}\\|\\hat{\\vz}_k-\\vz_k\\|^2+\\frac{\\alpha(1+\\eta_k \\mu)}{2}\\|\\hat{\\vz}_k-\\vz\\|^2}.\\label{eq:triangle_ipp}\n\\end{align}\nHere, the first inequality is due to the Cauchy-Schwarz inequality, the second inequality comes from \\eqref{eq:inexact_proximal_point}, and the last inequality is a result of Young's inequality. \nAdditionally, to handle the second term in \\eqref{eq:inexact_pp_decomp}, we apply the three-point equality, yielding:\n\\begin{equation}\\label{eq:three_point}\n  \\langle \\vz_k-\\hat{\\vz}_k, \\hat{\\vz}_k-\\vz\\rangle = \\frac{1}{2}\\|\\vz_k-\\vz\\|^2-\\frac{1}{2}\\|\\hat{\\vz}_k - \\vz_k\\|^2-\\frac{1}{2}\\|\\hat{\\vz}_k-\\vz\\|^2.\n\\end{equation}\n%\nBy combining \\eqref{eq:inexact_pp_decomp}, \\eqref{eq:triangle_ipp} and \\eqref{eq:three_point}, we obtain that    \n\\begin{equation}\\label{eq:intermediate_1}\n  \\eta_k \\langle \\vF(\\hat{\\vz}_k), \\hat{\\vz}_k-\\vz \\rangle \\leq \\frac{\\|\\vz_k-\\vz\\|^2 }{2}-\\frac{1-\\alpha}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz\\|^2) + \\frac{\\alpha\\eta_k \\mu}{2}\\|\\hat{\\vz}_k-\\vz\\|^2.\n\\end{equation}\nFurthermore, it follows from the update rule in \\eqref{eq:extragradient_mixing} that $\\eta_k\\vF(\\hat{\\vz}_k) = \\vz_k-\\vz_{k+1}+2\\eta_k\\mu (\\hat{\\vz}_k-\\vz_{k+1})$. Thus, for any $\\vz\\in \\reals^d$, it also holds that\n%\n\\begin{equation}\n  \\begin{aligned}\\label{eq:intermediate_2}\n    \\phantom{{}={}}\\eta_k \\langle \\vF(\\hat{\\vz}_k), \\vz_{k+1}-\\vz \\rangle &= \\langle \\vz_k-\\vz_{k+1}, \\vz_{k+1}-\\vz \\rangle+2\\eta_k\\mu \\langle \\hat{\\vz}_k-\\vz_{k+1}, \\vz_{k+1}-\\vz\\rangle %\n    \\\\\n    & = \\frac{\\|\\vz_{k}-\\vz\\|^2}{2}-\\frac{\\|\\vz_k-\\vz_{k+1}\\|^2}{2}-\\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1}-\\vz\\|^2\\\\\n    & \\phantom{{}={}}+ {\\eta_k\\mu}\\|\\hat{\\vz}_{k}-\\vz\\|^2-{\\eta_k\\mu}\\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2, \n  \\end{aligned}\n\\end{equation}\nwhere we applied the three-point equality twice in the last equality. Hence, by adding the inequality in \\eqref{eq:intermediate_1} with $\\vz = \\vz_{k+1}$ to the inequality in \\eqref{eq:intermediate_2}, we obtain \n\\begin{equation}\\label{eq:linearized_loss}\n  \\begin{aligned}\n    \\eta_k \\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz \\rangle \n    &= \\eta_k \\langle \\vF(\\hat{\\vz}_k),{\\vz}_{k+1} - \\vz \\rangle \n    +\\eta_k \\langle \\vF(\\hat{\\vz}_k), \\hat{\\vz}_{k} - \\vz_{k+1} \\rangle \\\\\n    &\\leq \\frac{\\|\\vz_{k}-\\vz\\|^2}{2}-\\bcancel{\\frac{\\|\\vz_k-\\vz_{k+1}\\|^2}{2}}-\\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1}-\\vz\\|^2 \\\\\n    &\\phantom{{}={}} + {\\eta_k\\mu}\\|\\hat{\\vz}_{k}-\\vz\\|^2-{{\\eta_k\\mu}\\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2} + \\bcancel{\\frac{\\|\\vz_k-\\vz_{k+1}\\|^2}{2}}\\\\\n    &\\phantom{{}={}} -\\frac{1\\!-\\!\\alpha}{2}(\\|\\hat{\\vz}_k \\!-\\! \\vz_k\\|^2 \\!+\\! \\|\\hat{\\vz}_k\\!-\\!\\vz_{k+1}\\|^2 )+{\\frac{\\alpha \\eta_k\\mu}{2}\\|\\hat{\\vz}_k\\!-\\!\\vz_{k+1}\\|^2} \\!. %\n    %\n  \\end{aligned}\n\\end{equation} \nMoreover, since $\\alpha <1$, we further have $-{{\\eta_k\\mu}\\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2} + \\frac{\\alpha \\eta_k \\mu}{2}\\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2 \\leq -\\frac{\\eta_k\\mu}{2}\\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2 \\leq 0$.  Combining this with \\eqref{eq:linearized_loss}  and rearranging the terms, we arrive at the desired result in~\\eqref{eq:one_step}. \n\\end{proof}","sha256_utf8":"dbbfc1228b3dbbcf45955aaece4fb00c3c8bb7b2bb5b7a65fcf6b2e92fd538a8"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"分解内积并控制非精确残差","source_units":[{"start":67173,"end":68229,"start_line":1342,"end_line":1356,"raw_tex":"\\begin{proof}\n  To simplify the notation, let $\\alpha \\triangleq \\alpha_1+\\alpha_2\\in (0,1)$. For any $\\vz\\in \\reals^d$, we have \n\\begin{equation}\\label{eq:inexact_pp_decomp}\n  \\eta_k \\langle \\vF(\\hat{\\vz}_k), \\hat{\\vz}_k-\\vz \\rangle =  \\langle \\hat{\\vz}_k-\\vz_k+\\eta_k \\vF(\\hat{\\vz}_k), \\hat{\\vz}_k-\\vz \\rangle\n  +\\langle \\vz_k-\\hat{\\vz}_k, \\hat{\\vz}_k-\\vz\\rangle. %\n   %\n   %\n\\end{equation}\nFor the first term in \\eqref{eq:inexact_pp_decomp}, we can bound it by\n\\begin{align}\n  \\langle \\hat{\\vz}_k-\\vz_k+\\eta_k \\vF(\\hat{\\vz}_k), \\hat{\\vz}_k-\\vz \\rangle &\\leq \\|\\hat{\\vz}_k-\\vz_k+\\eta_k\\vF(\\hat{\\vz}_k)\\|\\|\\hat{\\vz}_k-\\vz\\| \\nonumber\\\\\n  &\\leq \\alpha\\sqrt{1+\\eta_k\\mu}\\|\\hat{\\vz}_k-\\vz_k\\|\\|\\hat{\\vz}_k-\\vz\\| \\nonumber\\\\\n  &\\leq {\\frac{\\alpha}{2}\\|\\hat{\\vz}_k-\\vz_k\\|^2+\\frac{\\alpha(1+\\eta_k \\mu)}{2}\\|\\hat{\\vz}_k-\\vz\\|^2}.\\label{eq:triangle_ipp}\n\\end{align}\nHere, the first inequality is due to the Cauchy-Schwarz inequality, the second inequality comes from \\eqref{eq:inexact_proximal_point}, and the last inequality is a result of Young's inequality. \n","sha256_utf8":"793f9a3f7f4f97c74c4471f093860f77813a8d885ae3e0c7c03d8ddd7357085a"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"三点恒等式给出首个能量界","source_units":[{"start":68229,"end":68938,"start_line":1357,"end_line":1365,"raw_tex":"Additionally, to handle the second term in \\eqref{eq:inexact_pp_decomp}, we apply the three-point equality, yielding:\n\\begin{equation}\\label{eq:three_point}\n  \\langle \\vz_k-\\hat{\\vz}_k, \\hat{\\vz}_k-\\vz\\rangle = \\frac{1}{2}\\|\\vz_k-\\vz\\|^2-\\frac{1}{2}\\|\\hat{\\vz}_k - \\vz_k\\|^2-\\frac{1}{2}\\|\\hat{\\vz}_k-\\vz\\|^2.\n\\end{equation}\n%\nBy combining \\eqref{eq:inexact_pp_decomp}, \\eqref{eq:triangle_ipp} and \\eqref{eq:three_point}, we obtain that    \n\\begin{equation}\\label{eq:intermediate_1}\n  \\eta_k \\langle \\vF(\\hat{\\vz}_k), \\hat{\\vz}_k-\\vz \\rangle \\leq \\frac{\\|\\vz_k-\\vz\\|^2 }{2}-\\frac{1-\\alpha}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz\\|^2) + \\frac{\\alpha\\eta_k \\mu}{2}\\|\\hat{\\vz}_k-\\vz\\|^2.\n\\end{equation}\n","sha256_utf8":"5790a4264f0e8d187adb098fd0ebdaaae4c4fc5a3948d9b2860646a332d0091d"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"混合更新产生第二个恒等式","source_units":[{"start":68938,"end":69654,"start_line":1366,"end_line":1375,"raw_tex":"Furthermore, it follows from the update rule in \\eqref{eq:extragradient_mixing} that $\\eta_k\\vF(\\hat{\\vz}_k) = \\vz_k-\\vz_{k+1}+2\\eta_k\\mu (\\hat{\\vz}_k-\\vz_{k+1})$. Thus, for any $\\vz\\in \\reals^d$, it also holds that\n%\n\\begin{equation}\n  \\begin{aligned}\\label{eq:intermediate_2}\n    \\phantom{{}={}}\\eta_k \\langle \\vF(\\hat{\\vz}_k), \\vz_{k+1}-\\vz \\rangle &= \\langle \\vz_k-\\vz_{k+1}, \\vz_{k+1}-\\vz \\rangle+2\\eta_k\\mu \\langle \\hat{\\vz}_k-\\vz_{k+1}, \\vz_{k+1}-\\vz\\rangle %\n    \\\\\n    & = \\frac{\\|\\vz_{k}-\\vz\\|^2}{2}-\\frac{\\|\\vz_k-\\vz_{k+1}\\|^2}{2}-\\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1}-\\vz\\|^2\\\\\n    & \\phantom{{}={}}+ {\\eta_k\\mu}\\|\\hat{\\vz}_{k}-\\vz\\|^2-{\\eta_k\\mu}\\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2, \n  \\end{aligned}\n\\end{equation}\n","sha256_utf8":"276df69cdf5ca748559f3b560f7a9022ea0fe1aa4c71c710242cd2ac7946bf6a"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"合并两式并删除非正项","source_units":[{"start":69654,"end":70972,"start_line":1376,"end_line":1389,"raw_tex":"where we applied the three-point equality twice in the last equality. Hence, by adding the inequality in \\eqref{eq:intermediate_1} with $\\vz = \\vz_{k+1}$ to the inequality in \\eqref{eq:intermediate_2}, we obtain \n\\begin{equation}\\label{eq:linearized_loss}\n  \\begin{aligned}\n    \\eta_k \\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz \\rangle \n    &= \\eta_k \\langle \\vF(\\hat{\\vz}_k),{\\vz}_{k+1} - \\vz \\rangle \n    +\\eta_k \\langle \\vF(\\hat{\\vz}_k), \\hat{\\vz}_{k} - \\vz_{k+1} \\rangle \\\\\n    &\\leq \\frac{\\|\\vz_{k}-\\vz\\|^2}{2}-\\bcancel{\\frac{\\|\\vz_k-\\vz_{k+1}\\|^2}{2}}-\\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1}-\\vz\\|^2 \\\\\n    &\\phantom{{}={}} + {\\eta_k\\mu}\\|\\hat{\\vz}_{k}-\\vz\\|^2-{{\\eta_k\\mu}\\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2} + \\bcancel{\\frac{\\|\\vz_k-\\vz_{k+1}\\|^2}{2}}\\\\\n    &\\phantom{{}={}} -\\frac{1\\!-\\!\\alpha}{2}(\\|\\hat{\\vz}_k \\!-\\! \\vz_k\\|^2 \\!+\\! \\|\\hat{\\vz}_k\\!-\\!\\vz_{k+1}\\|^2 )+{\\frac{\\alpha \\eta_k\\mu}{2}\\|\\hat{\\vz}_k\\!-\\!\\vz_{k+1}\\|^2} \\!. %\n    %\n  \\end{aligned}\n\\end{equation} \nMoreover, since $\\alpha <1$, we further have $-{{\\eta_k\\mu}\\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2} + \\frac{\\alpha \\eta_k \\mu}{2}\\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2 \\leq -\\frac{\\eta_k\\mu}{2}\\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2 \\leq 0$.  Combining this with \\eqref{eq:linearized_loss}  and rearranging the terms, we arrive at the desired result in~\\eqref{eq:one_step}. \n\\end{proof}","sha256_utf8":"50d2259d7ef5837aacdfd66c770743c67ca68bd5ccc2d6e5ca13bc5b78b3f2f4"}]},{"id":"S023","semantic_type":"proposition","kind":"proposition","title":"步长控制的收敛与间隙界","source_units":[{"start":71104,"end":72037,"start_line":1395,"end_line":1410,"raw_tex":"\\begin{proposition}\\label{prop:HPE}\n  Let $\\{\\vz_k\\}_{k\\geq 0}$ and $\\{\\hat{\\vz}_k\\}_{k\\geq 0}$ be the iterates generated by Algorithm~\\ref{alg:Full_Equasi-Newton}, where $\\alpha_1\\in [0,1)$, $\\alpha_2 \\in (0,1)$ and $\\alpha_1+\\alpha_2 <1$. \n  %\n  \\begin{enumerate}[(a)]\n    \\item Under Assumption~\\ref{assum:strong_monotone}, we have $\\|\\vz_{k+1}-\\vz^*\\|^2 \\leq \\|\\vz_k-\\vz^*\\|^2 (1+2\\eta_k \\mu)^{-1}$ for any $k \\geq 0$. \n    \\item Under Assumption~\\ref{assum:monotone}, we have  $\\|\\vz_{k+1}-\\vz^*\\| \\leq \\|\\vz_k-\\vz^*\\|$ for any $k \\geq 0$. Moreover, define the averaged iterate $\\bar{\\vz}_N$ by $\\bar{\\vz}_N = \\frac{\\sum_{k=0}^{N-1} \\eta_k \\hat{\\vz}_k}{\\sum_{k=0}^{N-1} \\eta_k}$. Then for any compact set $\\calD \\subset \\reals^d$, we have \n    %\n    $\\gap(\\bar{\\vz}_N; \\calD) \\leq \\frac{\\max_{\\vz \\in \\calD}\\,\\|\\vz_0-\\vz\\|^2}{2 \\sum_{k=0}^{N-1} \\eta_k}$.  \n    %\n    %\n    %\n    %\n    %\n    %\n  \\end{enumerate}\n\\end{proposition}","sha256_utf8":"486c3bac178afd87c4f699dff8b7ae5345ae7dffdaa07916c17063be695293d3"}]},{"id":"S024","semantic_type":"proof","kind":"proof","title":"步长收敛命题证明","source_units":[{"start":72039,"end":74850,"start_line":1412,"end_line":1440,"raw_tex":"\\begin{proof}\nLet $\\alpha = \\alpha_1 + \\alpha_2$. First, we prove Part (a). Since $\\mF(\\vz^*) = 0$ and $\\vF$ is $\\mu$-strongly monotone by Assumption~\\ref{assum:strong_monotone}, it holds that  \n\\begin{equation}\\label{eq:strong_monotone}\n\\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz^* \\rangle = \\langle \\vF(\\hat{\\vz}_k)-\\vF({\\vz^*}),\\hat{\\vz}_k - \\vz^* \\rangle \\geq \\mu\\|\\hat{\\vz}_k - \\vz^*\\|^2.\n\\end{equation}\nBy setting $\\vz = \\vz^*$ in \\eqref{eq:one_step} and applying \\eqref{eq:strong_monotone}, it yields \n\\begin{equation}\\label{eq:one_step_strongly}\n    0 \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1} - \\vz^*\\|^2  - \\frac{1-\\alpha}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2).\n\\end{equation} \nSince $\\alpha = \\alpha_1 + \\alpha_2 <1$, we can drop the non-positive last term in \\eqref{eq:one_step_strongly} and obtain $0 \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1} - \\vz^*\\|^2$, which is equivalent to $\\|\\vz_{k+1} - \\vz^*\\|^2 \\leq \\|\\vz_k- \\vz^*\\|^2 (1+2\\eta_k\\mu)^{-1}$. This completes the proof of Part (a). \n\nNext, we prove Part (b). Since $\\vF$ is monotone by Assumption~\\ref{assum:monotone}, we have\n  \\begin{equation}\\label{eq:monotone_star}\n    \\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz^* \\rangle = \\langle \\vF(\\hat{\\vz}_k)-\\vF({\\vz^*}),\\hat{\\vz}_k - \\vz^* \\rangle \\geq 0.\n    \\end{equation}\n    By setting $\\vz = \\vz^*$ in \\eqref{eq:one_step} with $\\mu = 0$ and applying \\eqref{eq:monotone_star}, it yields \n    \\begin{equation}\\label{eq:one_step_monotone}\n      0 \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1}{2}\\|\\vz_{k+1} - \\vz^*\\|^2  - \\frac{1-\\alpha}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2).\n    \\end{equation}\n    Again, since the last term in \\eqref{eq:one_step_monotone} is non-positive, this immediately implies $\\frac{1}{2}\\|\\vz_{k+1} - \\vz^*\\|^2 \\leq \\frac{1}{2}\\|\\vz_{k} - \\vz^*\\|^2$, which is equivalent to $\\|\\vz_{k+1} - \\vz^*\\| \\leq \\|\\vz_{k} - \\vz^*\\|$. Additionally, by dropping the non-positive last term in \\eqref{eq:one_step} and noting that $\\mu = 0$, we also have \n    %\n    $\\eta_k \\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz \\rangle \\leq \\frac{1}{2}\\|\\vz_k- \\vz\\|^2 - \\frac{1}{2}\\|\\vz_{k+1} - \\vz\\|^2$.\n    %\n    By summing this inequality from $k=0$ to $k=N-1$, we further get \n    %\n    $\\sum_{k=0}^{N-1} \\eta_k \\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz \\rangle \\leq \\frac{1}{2}\\|\\vz_0- \\vz\\|^2 - \\frac{1}{2}\\|\\vz_{N} - \\vz\\|^2 \\leq \\frac{1}{2}\\|\\vz_0- \\vz\\|^2$. \n    %\n    Hence, it follows from Lemma~\\ref{lem:averaging} that $\\gap(\\bar{\\vz}_N; \\calD) \\leq \\frac{\\max_{\\vz \\in \\calD}\\,\\|\\vz_0-\\vz\\|^2}{2 \\sum_{k=0}^{N-1} \\eta_k}$, where $\\bar{\\vz}_N$ is the averaged iterate given by $\\bar{\\vz}_N = \\frac{\\sum_{k=0}^{N-1} \\eta_k \\hat{\\vz}_k}{\\sum_{k=0}^{N-1} \\eta_k}$.  \n\\end{proof}","sha256_utf8":"863b5e0f97cb02e7314d8bfaef834c3289939b0c84df0ef2feafc10ac8a475c9"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"强单调性导出距离收缩","source_units":[{"start":72039,"end":73122,"start_line":1412,"end_line":1421,"raw_tex":"\\begin{proof}\nLet $\\alpha = \\alpha_1 + \\alpha_2$. First, we prove Part (a). Since $\\mF(\\vz^*) = 0$ and $\\vF$ is $\\mu$-strongly monotone by Assumption~\\ref{assum:strong_monotone}, it holds that  \n\\begin{equation}\\label{eq:strong_monotone}\n\\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz^* \\rangle = \\langle \\vF(\\hat{\\vz}_k)-\\vF({\\vz^*}),\\hat{\\vz}_k - \\vz^* \\rangle \\geq \\mu\\|\\hat{\\vz}_k - \\vz^*\\|^2.\n\\end{equation}\nBy setting $\\vz = \\vz^*$ in \\eqref{eq:one_step} and applying \\eqref{eq:strong_monotone}, it yields \n\\begin{equation}\\label{eq:one_step_strongly}\n    0 \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1} - \\vz^*\\|^2  - \\frac{1-\\alpha}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2).\n\\end{equation} \nSince $\\alpha = \\alpha_1 + \\alpha_2 <1$, we can drop the non-positive last term in \\eqref{eq:one_step_strongly} and obtain $0 \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1} - \\vz^*\\|^2$, which is equivalent to $\\|\\vz_{k+1} - \\vz^*\\|^2 \\leq \\|\\vz_k- \\vz^*\\|^2 (1+2\\eta_k\\mu)^{-1}$. This completes the proof of Part (a). \n","sha256_utf8":"050b8746316afd1badd591b1742b187c0142be3acc3fb159fe2b850dc494e83e"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"单调情形的距离不增","source_units":[{"start":73122,"end":74127,"start_line":1422,"end_line":1431,"raw_tex":"\nNext, we prove Part (b). Since $\\vF$ is monotone by Assumption~\\ref{assum:monotone}, we have\n  \\begin{equation}\\label{eq:monotone_star}\n    \\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz^* \\rangle = \\langle \\vF(\\hat{\\vz}_k)-\\vF({\\vz^*}),\\hat{\\vz}_k - \\vz^* \\rangle \\geq 0.\n    \\end{equation}\n    By setting $\\vz = \\vz^*$ in \\eqref{eq:one_step} with $\\mu = 0$ and applying \\eqref{eq:monotone_star}, it yields \n    \\begin{equation}\\label{eq:one_step_monotone}\n      0 \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1}{2}\\|\\vz_{k+1} - \\vz^*\\|^2  - \\frac{1-\\alpha}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2).\n    \\end{equation}\n    Again, since the last term in \\eqref{eq:one_step_monotone} is non-positive, this immediately implies $\\frac{1}{2}\\|\\vz_{k+1} - \\vz^*\\|^2 \\leq \\frac{1}{2}\\|\\vz_{k} - \\vz^*\\|^2$, which is equivalent to $\\|\\vz_{k+1} - \\vz^*\\| \\leq \\|\\vz_{k} - \\vz^*\\|$. Additionally, by dropping the non-positive last term in \\eqref{eq:one_step} and noting that $\\mu = 0$, we also have \n","sha256_utf8":"d25a04da35ee6ac1bb2798199bbe2dcb9f96ce19faad683f3af8738f26786963"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"求和并应用平均间隙引理","source_units":[{"start":74127,"end":74850,"start_line":1432,"end_line":1440,"raw_tex":"    %\n    $\\eta_k \\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz \\rangle \\leq \\frac{1}{2}\\|\\vz_k- \\vz\\|^2 - \\frac{1}{2}\\|\\vz_{k+1} - \\vz\\|^2$.\n    %\n    By summing this inequality from $k=0$ to $k=N-1$, we further get \n    %\n    $\\sum_{k=0}^{N-1} \\eta_k \\langle \\vF(\\hat{\\vz}_k),\\hat{\\vz}_k - \\vz \\rangle \\leq \\frac{1}{2}\\|\\vz_0- \\vz\\|^2 - \\frac{1}{2}\\|\\vz_{N} - \\vz\\|^2 \\leq \\frac{1}{2}\\|\\vz_0- \\vz\\|^2$. \n    %\n    Hence, it follows from Lemma~\\ref{lem:averaging} that $\\gap(\\bar{\\vz}_N; \\calD) \\leq \\frac{\\max_{\\vz \\in \\calD}\\,\\|\\vz_0-\\vz\\|^2}{2 \\sum_{k=0}^{N-1} \\eta_k}$, where $\\bar{\\vz}_N$ is the averaged iterate given by $\\bar{\\vz}_N = \\frac{\\sum_{k=0}^{N-1} \\eta_k \\hat{\\vz}_k}{\\sum_{k=0}^{N-1} \\eta_k}$.  \n\\end{proof}","sha256_utf8":"666e8efec0abe74a2e45d40fe0bee547eb63152d7ad37fb7aad2877079281bb6"}]},{"id":"S028","semantic_type":"lemma","kind":"lemma","title":"回溯步长与位移比较","source_units":[{"start":75941,"end":76616,"start_line":1449,"end_line":1468,"raw_tex":"\\begin{lemma}\\label{lem:step size_lb}\n  %\n  %\n    %\n   %\n   %\n    Recall that ${\\tilde{\\vz}_k}$ is the auxiliary iterate returned by our line search scheme in Subroutine~\\ref{alg:ls}. For $k\\notin \\mathcal{B}$ we have $\\eta_k = \\sigma_k$, while for $k\\in \\mathcal{B}$ we have \n   %\n    %\n     $ \\eta_k \\!> \\!\n      %\n      {\\frac{{\\alpha_2} \\beta\\|{\\tilde{\\vz}_k}-\\vz_k\\|}{\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\|}}$.\n      %\n      %\n     %\n    %\n    Moreover, if $\\frac{1}{2}(\\mB_k + \\mB_k^\\top) \\succeq \\frac{1}{2}\\mu \\mI$, we have $\\|{\\tilde{\\vz}_k}-\\vz_k\\| \\leq \\frac{1+\\alpha_1}{\\beta(1-\\alpha_1)} \\|{\\hat{\\vz}_k}-\\vz_k\\|$. \n    %\n  %\n\\end{lemma}","sha256_utf8":"25a16e55d4d01aa378184ede6a0124f29d723d85d2a6439de6016639c2516a60"}]},{"id":"S029","semantic_type":"proof","kind":"proof","title":"步长及位移引理证明","source_units":[{"start":76618,"end":84452,"start_line":1470,"end_line":1596,"raw_tex":"\\begin{proof}\n  If $k\\notin \\mathcal{B}$, as per the definition, the line search scheme adopts the initial trial step size at the $k$-th iteration and thus $\\eta_k = \\sigma_k$. On the other hand, if $k \\in \\mathcal{B}$, we go through the backtracking procedure in Subroutine~\\ref{alg:ls}.\n  %\n  Let $\\tilde{\\vz}_k$ denote the last rejected point in the line search scheme, which is calculated from \\eqref{eq:linear_solver_update} using the step size $\\tilde{\\eta}_k = \\eta_k/\\beta$.\n  %\n  This means that the pair $(\\tilde{\\vz}_k,\\tilde{\\eta}_k)$ satisfies \\eqref{eq:x_plus_update} but not  \\eqref{eq:step size_condition}, i.e.,$\\|\\tilde{\\vz}_k-\\vz_k+\\tilde{\\eta}_k(\\vF(\\vz_k)+\\mB_k(\\tilde{\\vz}_k-\\vz))\\| \\leq  {\\alpha_1}\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|\\tilde{\\vz}_k-\\vz_k\\|$ and $\\|\\tilde{\\vz}_k - \\vz_k + \\tilde{\\eta}_k \\vF(\\tilde{\\vz}_k)\\| > (\\alpha_1+\\alpha_2)\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|{\\tilde{\\vz}_{k}}-\\vz_k\\|$. Moreover, note that $\\tilde{\\eta}_k (\\vF(\\tilde{\\vz}_k) - \\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)) = \\left(\\tilde{\\vz}_k - \\vz_k + \\tilde{\\eta}_k \\vF(\\tilde{\\vz}_k)\\right) - ({\\tilde{\\vz}_k}-\\vz_k+\\tilde{\\eta}_k(\\vF(\\vz_k)+\\mB_k({\\tilde{\\vz}_k}-\\vz_k)))$.  \n  %\n  %\n  %\n  %\n  %\n  Thus, it follows from the triangle inequality that $\\tilde{\\eta}_k \\|\\vF(\\tilde{\\vz}_k) - \\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\| > (\\alpha_1+\\alpha_2)\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|{\\tilde{\\vz}_{k}}-\\vz_k\\| - \\alpha_1\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|{\\tilde{\\vz}_{k}}-\\vz_k\\| = \\alpha_2\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|{\\tilde{\\vz}_{k}}-\\vz_k\\|$.\n  %\n  %\n  %\n  %\n  %\n  This further implies that \n  %\n  $\\tilde{\\eta}_k > \\frac{\\alpha_2\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|{\\tilde{\\vz}_{k}}-\\vz_k\\|}{\\|\\vF(\\tilde{\\vz}_k) - \\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\|}$. \n  %\n  %\n  %\n  %\n  %\n    Since $1 + \\tilde{\\eta}_k \\mu \\geq 1$, \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    we obtain the first result in Lemma~\\ref{lem:step size_lb}.\n    To prove the second result in Lemma~\\ref{lem:step size_lb}, \n    recall from~\\eqref{eq:x_plus_update} that $\\hat{\\vz}_k$ and $\\tilde{\\vz}_k$ are inexact solutions of the linear system of equations:  \n    \\begin{equation*}\n      (\\mI+{\\eta_k}\\mB_k)({\\vz}-\\vz_k) = -\\eta_k\\vF(\\vz_k) \\quad \\text{and} \\quad (\\mI+\\tilde{\\eta}_k\\mB_k)({\\vz}-\\vz_k) = -\\tilde{\\eta}_k\\vF(\\vz_k),\n    \\end{equation*}\n    %\n    respectively. \n    Define $\\hat{\\vz}_k^* = \\vz_k - {\\eta}_k(\\mI+ {\\eta}_k\\mB_k)^{-1} \\vF(\\vz_k)$ and $\\tilde{\\vz}_k^* = \\vz_k - \\tilde{\\eta}_k(\\mI+ \\tilde{\\eta}_k\\mB_k)^{-1} \\vF(\\vz_k)$, i.e., the exact solutions of the above linear systems. Since $(\\hat{\\vz}_k,\\eta_k)$ and $(\\tilde{\\vz}_k,\\tilde{\\eta}_k)$ satisfy the condition in \\eqref{eq:x_plus_update}, we have \n    \\begin{align}\\label{eq:inexact_condition_linear}\n      \\|(\\mI+\\eta_k\\mB_k)(\\hat{\\vz}_k-\\hat{\\vz}_k^*)\\| &\\leq \\alpha_1 \\sqrt{1+\\eta_k \\mu} \\|\\hat{\\vz}_k - \\vz_k\\| \\\\\n      \\quad \\text{and} \\quad  \\|(\\mI+\\tilde{\\eta}_k\\mB_k)(\\tilde{\\vz}_k-\\tilde{\\vz}_k^*)\\| &\\leq \\alpha_1 \\sqrt{1+\\tilde{\\eta}_k \\mu}  \\|\\tilde{\\vz}_k - \\vz_k\\|.\n    \\end{align} \n    We divide the proof of the second result {in Lemma~\\ref{lem:step size_lb}} into the following three steps.\n    %\n      First, we  show that \n      %\n      %\n        \\begin{align}\n                    (1-\\alpha_1)\\|\\hat{\\vz}_k - \\vz_k\\| &\\leq \\|\\hat{\\vz}_k^*-{\\vz}_k\\| \\leq (1+\\alpha_1)\\|\\hat{\\vz}_k - \\vz_k\\|, \\label{eq:relation_exact_inexact1}\\\\ (1-\\alpha_1)\\|\\tilde{\\vz}_k - \\vz_k\\| &\\leq \\|\\tilde{\\vz}_k^*-{\\vz}_k\\| \\leq (1+\\alpha_1)\\|\\tilde{\\vz}_k - \\vz_k\\|.\\label{eq:relation_exact_inexact2}\n        \\end{align}\n      %\n      %\n      In the following, we will only prove \\eqref{eq:relation_exact_inexact1}, since the proof of \\eqref{eq:relation_exact_inexact2} follows similarly.\n      Since we assume $\\frac{1}{2}(\\mB_k + \\mB_k^\\top) \\succeq \\frac{\\mu}{2} \\mI$, we have $\\vv^\\top \\mB_k \\vv \\geq \\frac{\\mu}{2} \\|\\vv\\|^2$ for any $\\vv \\in \\reals^d$. Therefore, by using Cauchy-Schwarz inequality, we get   \n      %\n      $\\|\\hat{\\vz}_k-\\hat{\\vz}_k^*\\| \\|(\\mI+\\eta_k\\mB_k)(\\hat{\\vz}_k-\\hat{\\vz}_k^*)\\| \\geq (\\hat{\\vz}_k-\\hat{\\vz}_k^*)^\\top (\\mI+\\eta_k\\mB_k)(\\hat{\\vz}_k-\\hat{\\vz}_k^*) \\geq (1+ \\frac{\\eta_k \\mu}{2})\\|\\hat{\\vz}_k-\\hat{\\vz}_k^*\\|^2$,\n      %\n      which further implies that $\\|(\\mI+\\eta_k\\mB_k)(\\hat{\\vz}_k-\\hat{\\vz}_k^*)\\| \\geq (1+ \\frac{1}{2}\\eta_k \\mu)\\|\\hat{\\vz}_k-\\hat{\\vz}_k^*\\|$. \n      Moreover, since $ \\sqrt{1+\\eta_k\\mu} \\leq 1+ \\frac{1}{2} \\eta_k\\mu $, by~\\eqref{eq:inexact_condition_linear} we also have $\\|(\\mI+\\eta_k\\mB_k)(\\hat{\\vz}_k-\\hat{\\vz}_k^*)\\| \\leq \\alpha_1 {(1+ \\frac{1}{2}\\eta_k \\mu)} \\|\\hat{\\vz}_k - \\vz_k\\|$. \n      %\n      %\n      %\n      %\n      Hence, combining these two inequalities, we get $\\|\\hat{\\vz}_k-\\hat{\\vz}_k^*\\| \\leq \\alpha_1 \\|\\hat{\\vz}_k-\\vz_k\\|$. It then follows from the triangle inequality that\n      \\begin{align*}\n        \\|\\hat{\\vz}_k^*-\\vz_k\\| \\leq \\|\\hat{\\vz}_k - \\vz_k\\| + \\|\\hat{\\vz}_k^*-\\hat{\\vz}_k\\| \\leq (1+\\alpha_1)\\|\\hat{\\vz}_k - \\vz_k\\|, \\\\\n        \\|\\hat{\\vz}_k^*-\\vz_k\\| \\geq \\|\\hat{\\vz}_k - \\vz_k\\| -  \\|\\hat{\\vz}_k^*-\\hat{\\vz}_k\\| \\geq (1-\\alpha_1)\\|\\hat{\\vz}_k - \\vz_k\\|,\n      \\end{align*}\n      which proves \\eqref{eq:relation_exact_inexact1}.\n    %\n  Next, we show that \n     \\begin{equation}\\label{eq:relation_displacement}\n      \\|{\\tilde{\\vz}^*_k}-\\vz_k\\|\\leq \\frac{1}{\\beta}\\|{\\hat{\\vz}^*_k}-\\vz_k\\|.\n     \\end{equation}\n     This follows from \\cite[Lemma 7.8]{monteiro2010complexity}. For completeness, we present its proof below. Note that by definition, we have $(\\mI+{\\eta_k}\\mB_k)(\\hat{\\vz}_k^*-\\vz_k) = -\\eta_k\\vF(\\vz_k)$ and $(\\mI+\\tilde{\\eta}_k\\mB_k)(\\tilde{\\vz}_k^*-\\vz_k) = -\\tilde{\\eta}_k\\vF(\\vz_k)$.\n    %\n    %\n    %\n     Hence, we further have \n     \\begin{equation*}\n      \\mB_k(\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*) = \\frac{1}{\\tilde{\\eta}_k}(\\tilde{\\vz}_k^*-\\vz_k) - \\frac{1}{\\eta_k}(\\hat{\\vz}_k^*-\\vz_k) = \\frac{1}{\\tilde{\\eta}_k} (\\tilde{\\vz}_k^* - \\hat{\\vz}_k^*) + \\left(\\frac{1}{\\tilde{\\eta}_k}-\\frac{1}{\\eta_k}\\right) (\\hat{\\vz}_k^*-\\vz_k).\n     \\end{equation*}\n     Since $(\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*)^\\top \\mB_k(\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*) \\geq 0$, by taking the inner product with $\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*$ on both sides of the above inequality, we obtain\n     %\n    $\\left(\\frac{1}{\\tilde{\\eta}_k}-\\frac{1}{\\eta_k}\\right) (\\hat{\\vz}_k^*-\\vz_k)^\\top (\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*)  \\geq \\frac{1}{\\tilde{\\eta}_k} \\|\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*\\|^2$.\n     %\n     Since ${\\eta}_k = \\beta \\tilde{\\eta}_k$, using the Cauchy-Schwarz inequality, this further leads to \n     %\n    $\\|\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*\\|^2 \\leq \\left(\\frac{1}{\\beta}-1\\right)(\\vz_k - \\hat{\\vz}_k^*)^\\top (\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*) \\leq  \\left(\\frac{1}{\\beta}-1\\right) \\|\\hat{\\vz}_k^* - \\vz_k\\| \\|\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*\\|$.\n     %\n     Hence, we further have $\\|\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*\\| \\leq \\left(\\frac{1}{\\beta}-1\\right) \\|\\hat{\\vz}_k^* - \\vz_k\\|$, which implies that $\\|{\\tilde{\\vz}^*_k}-\\vz_k\\| \\leq \\|{\\tilde{\\vz}^*_k}-\\hat{\\vz}^*_k\\|+\\|\\hat{\\vz}^*_k-\\vz_k\\| \\leq \\frac{1}{\\beta} \\|\\hat{\\vz}^*_k-\\vz_k\\|$. \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    This completes the proof of \\eqref{eq:relation_displacement}. Finally, by combining \\eqref{eq:relation_exact_inexact1}, \\eqref{eq:relation_exact_inexact2}, and \\eqref{eq:relation_displacement}, it follows that $\\|{\\tilde{\\vz}_k}-\\vz_k\\| \\leq \\frac{1}{1-\\alpha_1}\\|\\tilde{\\vz}_k^*-{\\vz}_k\\| \\leq \\frac{1}{(1-\\alpha_1)\\beta}\\|{\\hat{\\vz}^*_k}-\\vz_k\\| \\leq \\frac{1+\\alpha_1}{1-\\alpha_1}\\|\\tilde{\\vz}_k^*-{\\vz}_k\\| \\leq \\frac{1+\\alpha_1}{(1-\\alpha_1)\\beta}\\|{\\hat{\\vz}_k}-\\vz_k\\|$. This proves the second result {in Lemma~\\ref{lem:step size_lb}}. \n  \\end{proof}","sha256_utf8":"d6febe9b8dfa0d136fde819207196f93d4ce968f89d8b4a1dab806469d04743d"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"利用最后拒绝点下界步长","source_units":[{"start":76618,"end":78550,"start_line":1470,"end_line":1504,"raw_tex":"\\begin{proof}\n  If $k\\notin \\mathcal{B}$, as per the definition, the line search scheme adopts the initial trial step size at the $k$-th iteration and thus $\\eta_k = \\sigma_k$. On the other hand, if $k \\in \\mathcal{B}$, we go through the backtracking procedure in Subroutine~\\ref{alg:ls}.\n  %\n  Let $\\tilde{\\vz}_k$ denote the last rejected point in the line search scheme, which is calculated from \\eqref{eq:linear_solver_update} using the step size $\\tilde{\\eta}_k = \\eta_k/\\beta$.\n  %\n  This means that the pair $(\\tilde{\\vz}_k,\\tilde{\\eta}_k)$ satisfies \\eqref{eq:x_plus_update} but not  \\eqref{eq:step size_condition}, i.e.,$\\|\\tilde{\\vz}_k-\\vz_k+\\tilde{\\eta}_k(\\vF(\\vz_k)+\\mB_k(\\tilde{\\vz}_k-\\vz))\\| \\leq  {\\alpha_1}\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|\\tilde{\\vz}_k-\\vz_k\\|$ and $\\|\\tilde{\\vz}_k - \\vz_k + \\tilde{\\eta}_k \\vF(\\tilde{\\vz}_k)\\| > (\\alpha_1+\\alpha_2)\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|{\\tilde{\\vz}_{k}}-\\vz_k\\|$. Moreover, note that $\\tilde{\\eta}_k (\\vF(\\tilde{\\vz}_k) - \\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)) = \\left(\\tilde{\\vz}_k - \\vz_k + \\tilde{\\eta}_k \\vF(\\tilde{\\vz}_k)\\right) - ({\\tilde{\\vz}_k}-\\vz_k+\\tilde{\\eta}_k(\\vF(\\vz_k)+\\mB_k({\\tilde{\\vz}_k}-\\vz_k)))$.  \n  %\n  %\n  %\n  %\n  %\n  Thus, it follows from the triangle inequality that $\\tilde{\\eta}_k \\|\\vF(\\tilde{\\vz}_k) - \\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\| > (\\alpha_1+\\alpha_2)\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|{\\tilde{\\vz}_{k}}-\\vz_k\\| - \\alpha_1\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|{\\tilde{\\vz}_{k}}-\\vz_k\\| = \\alpha_2\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|{\\tilde{\\vz}_{k}}-\\vz_k\\|$.\n  %\n  %\n  %\n  %\n  %\n  This further implies that \n  %\n  $\\tilde{\\eta}_k > \\frac{\\alpha_2\\sqrt{1+\\tilde{\\eta}_k \\mu} \\|{\\tilde{\\vz}_{k}}-\\vz_k\\|}{\\|\\vF(\\tilde{\\vz}_k) - \\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\|}$. \n  %\n  %\n  %\n  %\n  %\n    Since $1 + \\tilde{\\eta}_k \\mu \\geq 1$, \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    we obtain the first result in Lemma~\\ref{lem:step size_lb}.\n","sha256_utf8":"01b67c9705ad17d7e0400b390454eac3ec715045cb6409f810e8512c26234fda"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"引入精确线性系统解","source_units":[{"start":78550,"end":79784,"start_line":1505,"end_line":1517,"raw_tex":"    To prove the second result in Lemma~\\ref{lem:step size_lb}, \n    recall from~\\eqref{eq:x_plus_update} that $\\hat{\\vz}_k$ and $\\tilde{\\vz}_k$ are inexact solutions of the linear system of equations:  \n    \\begin{equation*}\n      (\\mI+{\\eta_k}\\mB_k)({\\vz}-\\vz_k) = -\\eta_k\\vF(\\vz_k) \\quad \\text{and} \\quad (\\mI+\\tilde{\\eta}_k\\mB_k)({\\vz}-\\vz_k) = -\\tilde{\\eta}_k\\vF(\\vz_k),\n    \\end{equation*}\n    %\n    respectively. \n    Define $\\hat{\\vz}_k^* = \\vz_k - {\\eta}_k(\\mI+ {\\eta}_k\\mB_k)^{-1} \\vF(\\vz_k)$ and $\\tilde{\\vz}_k^* = \\vz_k - \\tilde{\\eta}_k(\\mI+ \\tilde{\\eta}_k\\mB_k)^{-1} \\vF(\\vz_k)$, i.e., the exact solutions of the above linear systems. Since $(\\hat{\\vz}_k,\\eta_k)$ and $(\\tilde{\\vz}_k,\\tilde{\\eta}_k)$ satisfy the condition in \\eqref{eq:x_plus_update}, we have \n    \\begin{align}\\label{eq:inexact_condition_linear}\n      \\|(\\mI+\\eta_k\\mB_k)(\\hat{\\vz}_k-\\hat{\\vz}_k^*)\\| &\\leq \\alpha_1 \\sqrt{1+\\eta_k \\mu} \\|\\hat{\\vz}_k - \\vz_k\\| \\\\\n      \\quad \\text{and} \\quad  \\|(\\mI+\\tilde{\\eta}_k\\mB_k)(\\tilde{\\vz}_k-\\tilde{\\vz}_k^*)\\| &\\leq \\alpha_1 \\sqrt{1+\\tilde{\\eta}_k \\mu}  \\|\\tilde{\\vz}_k - \\vz_k\\|.\n    \\end{align} \n    We divide the proof of the second result {in Lemma~\\ref{lem:step size_lb}} into the following three steps.\n","sha256_utf8":"342635c5280d184787f65b932dcaa14038426a53ff9485e14475015304b97492"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"比较精确与非精确位移","source_units":[{"start":79784,"end":81805,"start_line":1518,"end_line":1543,"raw_tex":"    %\n      First, we  show that \n      %\n      %\n        \\begin{align}\n                    (1-\\alpha_1)\\|\\hat{\\vz}_k - \\vz_k\\| &\\leq \\|\\hat{\\vz}_k^*-{\\vz}_k\\| \\leq (1+\\alpha_1)\\|\\hat{\\vz}_k - \\vz_k\\|, \\label{eq:relation_exact_inexact1}\\\\ (1-\\alpha_1)\\|\\tilde{\\vz}_k - \\vz_k\\| &\\leq \\|\\tilde{\\vz}_k^*-{\\vz}_k\\| \\leq (1+\\alpha_1)\\|\\tilde{\\vz}_k - \\vz_k\\|.\\label{eq:relation_exact_inexact2}\n        \\end{align}\n      %\n      %\n      In the following, we will only prove \\eqref{eq:relation_exact_inexact1}, since the proof of \\eqref{eq:relation_exact_inexact2} follows similarly.\n      Since we assume $\\frac{1}{2}(\\mB_k + \\mB_k^\\top) \\succeq \\frac{\\mu}{2} \\mI$, we have $\\vv^\\top \\mB_k \\vv \\geq \\frac{\\mu}{2} \\|\\vv\\|^2$ for any $\\vv \\in \\reals^d$. Therefore, by using Cauchy-Schwarz inequality, we get   \n      %\n      $\\|\\hat{\\vz}_k-\\hat{\\vz}_k^*\\| \\|(\\mI+\\eta_k\\mB_k)(\\hat{\\vz}_k-\\hat{\\vz}_k^*)\\| \\geq (\\hat{\\vz}_k-\\hat{\\vz}_k^*)^\\top (\\mI+\\eta_k\\mB_k)(\\hat{\\vz}_k-\\hat{\\vz}_k^*) \\geq (1+ \\frac{\\eta_k \\mu}{2})\\|\\hat{\\vz}_k-\\hat{\\vz}_k^*\\|^2$,\n      %\n      which further implies that $\\|(\\mI+\\eta_k\\mB_k)(\\hat{\\vz}_k-\\hat{\\vz}_k^*)\\| \\geq (1+ \\frac{1}{2}\\eta_k \\mu)\\|\\hat{\\vz}_k-\\hat{\\vz}_k^*\\|$. \n      Moreover, since $ \\sqrt{1+\\eta_k\\mu} \\leq 1+ \\frac{1}{2} \\eta_k\\mu $, by~\\eqref{eq:inexact_condition_linear} we also have $\\|(\\mI+\\eta_k\\mB_k)(\\hat{\\vz}_k-\\hat{\\vz}_k^*)\\| \\leq \\alpha_1 {(1+ \\frac{1}{2}\\eta_k \\mu)} \\|\\hat{\\vz}_k - \\vz_k\\|$. \n      %\n      %\n      %\n      %\n      Hence, combining these two inequalities, we get $\\|\\hat{\\vz}_k-\\hat{\\vz}_k^*\\| \\leq \\alpha_1 \\|\\hat{\\vz}_k-\\vz_k\\|$. It then follows from the triangle inequality that\n      \\begin{align*}\n        \\|\\hat{\\vz}_k^*-\\vz_k\\| \\leq \\|\\hat{\\vz}_k - \\vz_k\\| + \\|\\hat{\\vz}_k^*-\\hat{\\vz}_k\\| \\leq (1+\\alpha_1)\\|\\hat{\\vz}_k - \\vz_k\\|, \\\\\n        \\|\\hat{\\vz}_k^*-\\vz_k\\| \\geq \\|\\hat{\\vz}_k - \\vz_k\\| -  \\|\\hat{\\vz}_k^*-\\hat{\\vz}_k\\| \\geq (1-\\alpha_1)\\|\\hat{\\vz}_k - \\vz_k\\|,\n      \\end{align*}\n      which proves \\eqref{eq:relation_exact_inexact1}.\n","sha256_utf8":"a4408d21067759d39de8ea3b512ecf2d8e069d6dd4e8b6d5508cbf807bfa5fef"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"比较两个步长下的精确位移","source_units":[{"start":81805,"end":83892,"start_line":1544,"end_line":1594,"raw_tex":"    %\n  Next, we show that \n     \\begin{equation}\\label{eq:relation_displacement}\n      \\|{\\tilde{\\vz}^*_k}-\\vz_k\\|\\leq \\frac{1}{\\beta}\\|{\\hat{\\vz}^*_k}-\\vz_k\\|.\n     \\end{equation}\n     This follows from \\cite[Lemma 7.8]{monteiro2010complexity}. For completeness, we present its proof below. Note that by definition, we have $(\\mI+{\\eta_k}\\mB_k)(\\hat{\\vz}_k^*-\\vz_k) = -\\eta_k\\vF(\\vz_k)$ and $(\\mI+\\tilde{\\eta}_k\\mB_k)(\\tilde{\\vz}_k^*-\\vz_k) = -\\tilde{\\eta}_k\\vF(\\vz_k)$.\n    %\n    %\n    %\n     Hence, we further have \n     \\begin{equation*}\n      \\mB_k(\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*) = \\frac{1}{\\tilde{\\eta}_k}(\\tilde{\\vz}_k^*-\\vz_k) - \\frac{1}{\\eta_k}(\\hat{\\vz}_k^*-\\vz_k) = \\frac{1}{\\tilde{\\eta}_k} (\\tilde{\\vz}_k^* - \\hat{\\vz}_k^*) + \\left(\\frac{1}{\\tilde{\\eta}_k}-\\frac{1}{\\eta_k}\\right) (\\hat{\\vz}_k^*-\\vz_k).\n     \\end{equation*}\n     Since $(\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*)^\\top \\mB_k(\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*) \\geq 0$, by taking the inner product with $\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*$ on both sides of the above inequality, we obtain\n     %\n    $\\left(\\frac{1}{\\tilde{\\eta}_k}-\\frac{1}{\\eta_k}\\right) (\\hat{\\vz}_k^*-\\vz_k)^\\top (\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*)  \\geq \\frac{1}{\\tilde{\\eta}_k} \\|\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*\\|^2$.\n     %\n     Since ${\\eta}_k = \\beta \\tilde{\\eta}_k$, using the Cauchy-Schwarz inequality, this further leads to \n     %\n    $\\|\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*\\|^2 \\leq \\left(\\frac{1}{\\beta}-1\\right)(\\vz_k - \\hat{\\vz}_k^*)^\\top (\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*) \\leq  \\left(\\frac{1}{\\beta}-1\\right) \\|\\hat{\\vz}_k^* - \\vz_k\\| \\|\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*\\|$.\n     %\n     Hence, we further have $\\|\\hat{\\vz}_k^* - \\tilde{\\vz}_k^*\\| \\leq \\left(\\frac{1}{\\beta}-1\\right) \\|\\hat{\\vz}_k^* - \\vz_k\\|$, which implies that $\\|{\\tilde{\\vz}^*_k}-\\vz_k\\| \\leq \\|{\\tilde{\\vz}^*_k}-\\hat{\\vz}^*_k\\|+\\|\\hat{\\vz}^*_k-\\vz_k\\| \\leq \\frac{1}{\\beta} \\|\\hat{\\vz}^*_k-\\vz_k\\|$. \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n","sha256_utf8":"178907baff7b48d473660604fbb395f51f607a6f5499b71c80a444a35b0047bb"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"合并位移比较式","source_units":[{"start":83892,"end":84452,"start_line":1595,"end_line":1596,"raw_tex":"    This completes the proof of \\eqref{eq:relation_displacement}. Finally, by combining \\eqref{eq:relation_exact_inexact1}, \\eqref{eq:relation_exact_inexact2}, and \\eqref{eq:relation_displacement}, it follows that $\\|{\\tilde{\\vz}_k}-\\vz_k\\| \\leq \\frac{1}{1-\\alpha_1}\\|\\tilde{\\vz}_k^*-{\\vz}_k\\| \\leq \\frac{1}{(1-\\alpha_1)\\beta}\\|{\\hat{\\vz}^*_k}-\\vz_k\\| \\leq \\frac{1+\\alpha_1}{1-\\alpha_1}\\|\\tilde{\\vz}_k^*-{\\vz}_k\\| \\leq \\frac{1+\\alpha_1}{(1-\\alpha_1)\\beta}\\|{\\hat{\\vz}_k}-\\vz_k\\|$. This proves the second result {in Lemma~\\ref{lem:step size_lb}}. \n  \\end{proof}","sha256_utf8":"394c90aad2c352eb19139a453fafab48b6f67c320ccc3d9c40050e257496a8c5"}]},{"id":"S035","semantic_type":"proof","kind":"proof","title":"调和步长收敛界推导","source_units":[{"start":85305,"end":85766,"start_line":1606,"end_line":1612,"raw_tex":"To begin with, \\blue{in the strongly monotone setting,} by repeatedly applying Proposition~\\ref{prop:HPE}(a), we obtain that \n%\n\\begin{equation}\\label{eq:after_jensen}\n   \\frac{\\|\\vz_{N}-\\vz^*\\|^2}{\\|\\vz_0-\\vz^*\\|^2} \\leq \\prod_{k=0}^{N-1} (1+2\\eta_k\\mu)^{-1} \\leq \\biggl(1+{\\frac{2\\mu N}{\\sum_{k=0}^{N-1} 1/\\eta_k}}\\biggr)^{-N}, \n   %\n\\end{equation}\nwhere the last inequality is obtained by applying Jensen's inequality to the function $\\log(1+\\frac{1}{t})$. \n","sha256_utf8":"270a4b83eef54ff529f5a3b1a202e3484a8f20b9754a7bfc151095f881e4cf80"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"迭代收缩并应用Jensen不等式","source_units":[{"start":85305,"end":85766,"start_line":1606,"end_line":1612,"raw_tex":"To begin with, \\blue{in the strongly monotone setting,} by repeatedly applying Proposition~\\ref{prop:HPE}(a), we obtain that \n%\n\\begin{equation}\\label{eq:after_jensen}\n   \\frac{\\|\\vz_{N}-\\vz^*\\|^2}{\\|\\vz_0-\\vz^*\\|^2} \\leq \\prod_{k=0}^{N-1} (1+2\\eta_k\\mu)^{-1} \\leq \\biggl(1+{\\frac{2\\mu N}{\\sum_{k=0}^{N-1} 1/\\eta_k}}\\biggr)^{-N}, \n   %\n\\end{equation}\nwhere the last inequality is obtained by applying Jensen's inequality to the function $\\log(1+\\frac{1}{t})$. \n","sha256_utf8":"270a4b83eef54ff529f5a3b1a202e3484a8f20b9754a7bfc151095f881e4cf80"}]},{"id":"S037","semantic_type":"lemma","kind":"lemma","title":"累计逆步长的损失界","source_units":[{"start":86348,"end":86980,"start_line":1622,"end_line":1633,"raw_tex":"\\begin{lemma}\\label{lem:stepsize_bnd}\nLet $\\{\\eta_k\\}_{k=0}^{N-1}$ be the step sizes in Algorithm~\\ref{alg:Full_Equasi-Newton} obtained by the line search in Subroutine~\\ref{alg:ls}.~Then,\n%\n\\begin{equation}\\label{eq:goal}\n     \\sum_{k=0}^{N-1}\\frac{1}{\\eta_k} \\leq \\frac{1}{(1-\\beta)\\sigma_0}+ \\frac{1}{(1-\\beta)\\alpha_2\\beta} \n    %\n     \\sqrt{N \\sum_{k\\in \\mathcal{B}} \\frac{\\|\\vu_k-\\mB_k\\vs_k\\|^2}{\\|\\vs_k\\|^2}}, \n     %\n      %\n\\end{equation}\nwhere $\\vu_k \\triangleq \\vF(\\tilde{\\vz}_k) -\\vF({\\vz_k})$, $\\vs_k \\triangleq \\tilde{\\vz}_k-\\vz_k$, and $\\tilde{\\vz}_k$ is the auxiliary iterate in Subroutine~\\ref{alg:ls}. \n\\end{lemma}","sha256_utf8":"538785db9710fca65f2932a7e3b82cdc645db46fee5fa26c4fbf095a97c1de81"}]},{"id":"S038","semantic_type":"proof","kind":"proof","title":"累计逆步长引理证明","source_units":[{"start":86981,"end":88958,"start_line":1634,"end_line":1686,"raw_tex":"\\begin{proof}\n  Let $\\vu_k \\triangleq \\vF(\\tilde{\\vz}_k) -\\vF({\\vz_k})$ and $\\vs_k \\triangleq \\tilde{\\vz}_k-\\vz_k$. \n  In Lemma~\\ref{lem:step size_lb}, we showed that $\\eta_k = \\sigma_k$ if $k\\notin \\mathcal{B}$ and $\\eta_k> \\frac{\\alpha_2 \\beta \\|\\vs_k\\|}{\\|\\vu_k-\\mB_k\\vs_k\\|}$ otherwise. %\n%\n%\nUsing the observations above, we can write \n\\begin{equation}\\label{eq:intermediate_bound}\n  %\n    \\sum_{k=0}^{N-1}\\frac{1}{\\eta_k} \n  = \\sum_{k\\notin \\mathcal{B}}\\frac{1}{\\eta_k} + \\sum_{k\\in \\mathcal{B}}\\frac{1}{\\eta_k} \\leq\n  \\sum_{k\\notin \\mathcal{B}}\\frac{1}{\\sigma_k} + \\sum_{k\\in \\mathcal{B}}\\frac{1}{\\eta_k}\n  \\leq \\frac{1}{\\sigma_0} + \n  \\beta\\sum_{k\\notin \\mathcal{B},k\\geq 1}\\frac{1}{\\eta_{k-1}}+\\sum_{k\\in \\mathcal{B}}\\frac{1}{\\eta_k},\n  %\n\\end{equation}\n%\n%\n%\n%\n%\n%\n%\n%\n%\nwhere we used $\\sigma_k = \\frac{\\eta_{k-1}}{\\beta}$ for $k\\geq 1$ in the last equality. \nSince\n%\n$\\sum_{k\\notin \\mathcal{B},k\\geq 1}\\frac{1}{\\eta_{k-1}} \n%\n\\leq \\sum_{k= 0}^{N-1}\\frac{1}{\\eta_{k}}$,\n%\nrearranging and simplifying the terms in \\eqref{eq:intermediate_bound}, we arrive at \n\\begin{equation}\\label{eq:all_vs_B}\n  \\sum_{k=0}^{N-1}\\frac{1}{\\eta_k} \\leq \\frac{1}{(1-\\beta)\\sigma_0} + \\frac{1}{1-\\beta} \\sum_{k\\in \\mathcal{B}}\\frac{1}{\\eta_k}. \n\\end{equation}\n%\nNow using $\\eta_k > \\frac{\\alpha_2 \\beta \\|\\vs_k\\|}{\\|\\vu_k-\\mB_k\\vs_k\\|}$ for $k \\in \\calB$, we further have \n\\begin{align}\n  \\sum_{k \\in \\calB} \\frac{1}{\\eta_k} \\leq \\sum_{k \\in \\calB} \\frac{\\|\\vu_k-\\mB_k\\vs_k\\|}{\\alpha_2 \\beta \\|\\vs_k\\|} &\\leq \\frac{1}{\\alpha_2\\beta} \\sqrt{|\\calB| \\sum_{k\\in \\mathcal{B}} \\frac{\\|\\vu_k-\\mB_k\\vs_k\\|^2}{\\|\\vs_k\\|^2}} \n  %\n  %\n  \\leq \\frac{1}{\\alpha_2\\beta} \n  \\sqrt{N \\sum_{k\\in \\mathcal{B}} \\frac{\\|\\vu_k-\\mB_k\\vs_k\\|^2}{\\|\\vs_k\\|^2}}, \\label{eq:B_bound1}\n\\end{align}\nwhere the second inequality is due to the generalized mean inequality. \n%\n%\n%\n%\nFinally, the inequality in \\eqref{eq:goal} follows from \\eqref{eq:all_vs_B} and \\eqref{eq:B_bound1}. \n%\nThis completes the proof.\n\\end{proof}","sha256_utf8":"b7c5c17ed5e4e7ffc5e41506e45dec6ba2eaa8b314b9b16ee8218eb0c11e2208"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"按回溯与否分解逆步长和","source_units":[{"start":86981,"end":87851,"start_line":1634,"end_line":1658,"raw_tex":"\\begin{proof}\n  Let $\\vu_k \\triangleq \\vF(\\tilde{\\vz}_k) -\\vF({\\vz_k})$ and $\\vs_k \\triangleq \\tilde{\\vz}_k-\\vz_k$. \n  In Lemma~\\ref{lem:step size_lb}, we showed that $\\eta_k = \\sigma_k$ if $k\\notin \\mathcal{B}$ and $\\eta_k> \\frac{\\alpha_2 \\beta \\|\\vs_k\\|}{\\|\\vu_k-\\mB_k\\vs_k\\|}$ otherwise. %\n%\n%\nUsing the observations above, we can write \n\\begin{equation}\\label{eq:intermediate_bound}\n  %\n    \\sum_{k=0}^{N-1}\\frac{1}{\\eta_k} \n  = \\sum_{k\\notin \\mathcal{B}}\\frac{1}{\\eta_k} + \\sum_{k\\in \\mathcal{B}}\\frac{1}{\\eta_k} \\leq\n  \\sum_{k\\notin \\mathcal{B}}\\frac{1}{\\sigma_k} + \\sum_{k\\in \\mathcal{B}}\\frac{1}{\\eta_k}\n  \\leq \\frac{1}{\\sigma_0} + \n  \\beta\\sum_{k\\notin \\mathcal{B},k\\geq 1}\\frac{1}{\\eta_{k-1}}+\\sum_{k\\in \\mathcal{B}}\\frac{1}{\\eta_k},\n  %\n\\end{equation}\n%\n%\n%\n%\n%\n%\n%\n%\n%\nwhere we used $\\sigma_k = \\frac{\\eta_{k-1}}{\\beta}$ for $k\\geq 1$ in the last equality. \n","sha256_utf8":"966418aa6ac5e16f5b7d334068d2d041426ce321af8b580e7cf61fcaa3f80e7e"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"利用热启动递推整理总和","source_units":[{"start":87851,"end":88230,"start_line":1659,"end_line":1668,"raw_tex":"Since\n%\n$\\sum_{k\\notin \\mathcal{B},k\\geq 1}\\frac{1}{\\eta_{k-1}} \n%\n\\leq \\sum_{k= 0}^{N-1}\\frac{1}{\\eta_{k}}$,\n%\nrearranging and simplifying the terms in \\eqref{eq:intermediate_bound}, we arrive at \n\\begin{equation}\\label{eq:all_vs_B}\n  \\sum_{k=0}^{N-1}\\frac{1}{\\eta_k} \\leq \\frac{1}{(1-\\beta)\\sigma_0} + \\frac{1}{1-\\beta} \\sum_{k\\in \\mathcal{B}}\\frac{1}{\\eta_k}. \n\\end{equation}\n","sha256_utf8":"69de0709564e3200c77886df03938d001a8c91e79a033ae981b36b9cb6ace503"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"以累计平方损失控制回溯项","source_units":[{"start":88230,"end":88958,"start_line":1669,"end_line":1686,"raw_tex":"%\nNow using $\\eta_k > \\frac{\\alpha_2 \\beta \\|\\vs_k\\|}{\\|\\vu_k-\\mB_k\\vs_k\\|}$ for $k \\in \\calB$, we further have \n\\begin{align}\n  \\sum_{k \\in \\calB} \\frac{1}{\\eta_k} \\leq \\sum_{k \\in \\calB} \\frac{\\|\\vu_k-\\mB_k\\vs_k\\|}{\\alpha_2 \\beta \\|\\vs_k\\|} &\\leq \\frac{1}{\\alpha_2\\beta} \\sqrt{|\\calB| \\sum_{k\\in \\mathcal{B}} \\frac{\\|\\vu_k-\\mB_k\\vs_k\\|^2}{\\|\\vs_k\\|^2}} \n  %\n  %\n  \\leq \\frac{1}{\\alpha_2\\beta} \n  \\sqrt{N \\sum_{k\\in \\mathcal{B}} \\frac{\\|\\vu_k-\\mB_k\\vs_k\\|^2}{\\|\\vs_k\\|^2}}, \\label{eq:B_bound1}\n\\end{align}\nwhere the second inequality is due to the generalized mean inequality. \n%\n%\n%\n%\nFinally, the inequality in \\eqref{eq:goal} follows from \\eqref{eq:all_vs_B} and \\eqref{eq:B_bound1}. \n%\nThis completes the proof.\n\\end{proof}","sha256_utf8":"021ac435b220f94a0ca64a1c7873fd3e0d30a98021146767997f281aa1bedd14"}]},{"id":"S042","semantic_type":"lemma","kind":"lemma","title":"分离预言机的遗憾归约","source_units":[{"start":106628,"end":108240,"start_line":1886,"end_line":1923,"raw_tex":"\\begin{lemma}\\label{lem:regret_reduction}\n Suppose the loss function $\\ell_t$ is convex for any $t\\geq 0$ and let $\\{\\vx_t\\}_{t=0}^{T-1}$ and $\\{\\vw_t\\}_{t=0}^{T-1}$ be the iterates generated by Algorithm~\\ref{alg:projection_free_online_learning}. Then we have $\\vw_t \\in \\vspan(\\calC)$ for any $t \\geq 0$. Moreover: \n  \\begin{enumerate}[(a)]\n    \\item In \\textbf{Option I}, we have $\\vx_t \\in (1+\\delta)\\calC$ for $t=0,1,\\dots,T-1$. Also, for any $\\vu\\in \\mathcal{C}$, it holds that \n    \\begin{equation}\n    %\n    \\ell_t(\\vx_t) - \\ell_t(\\vu) %\n    \\leq \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle %\n    \\leq \\frac{\\|\\vw_t-\\vu\\|^2_2}{2\\rho}-\\frac{\\|\\vw_{t+1}-\\vu\\|^2_2}{2\\rho}+ \\frac{\\rho}{2}\\|\\tilde{\\vg}_t\\|_2^2,\\label{eq:regret_reduction} %\n\\end{equation}\n%\n\\begin{equation}\\label{eq:surrogate_loss}\n   \\|\\tilde{\\vg}_t\\| \\leq \\|\\vg_t\\|+|\\langle \\vg_t, \\vx_t\\rangle|\\|\\vs_t\\|.\n\\end{equation}\n\\item In \\textbf{Option II}, we have $\\vx_t \\in \\calC$ for $t=0,1,\\dots,T-1$. Also, for any $\\vu\\in \\mathcal{C}$, it holds that \n  \\begin{align}\n  %\n  \\ell_t(\\vx_t) - \\ell_t(\\vu) \n  %\n  &\\leq \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle - \\delta \\langle \\vg_t, \\vx_t \\rangle \\label{eq:regret_reduction_II_1}\\\\\n  &\\leq \\frac{\\|\\vw_t-\\vu\\|^2_2}{2\\rho}-\\frac{\\|\\vw_{t+1}-\\vu\\|^2_2}{2\\rho}+ \\frac{\\rho}{2}\\|\\tilde{\\vg}_t\\|_2^2 - \\delta \\langle \\vg_t, \\vx_t \\rangle,\\label{eq:regret_reduction_II_2} %\n\\end{align}\n%\n\\begin{equation}\\label{eq:surrogate_loss_II}\n  \\|\\tilde{\\vg}_t\\| \\leq \\|\\vg_t\\|+(1+\\delta)|\\langle \\vg_t, \\vx_t\\rangle|\\|\\vs_t\\|.\n\\end{equation}\n  \\end{enumerate}\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n\\end{lemma}","sha256_utf8":"d47d97fab8ff9066a60a53106a8a92cc5b91186cbcf7719d78617514ed93aed6"}]},{"id":"S043","semantic_type":"proof","kind":"proof","title":"遗憾归约引理证明","source_units":[{"start":108241,"end":115842,"start_line":1924,"end_line":2018,"raw_tex":"\\begin{proof}\n  %\n  %\n  %\n  %\n%\n%\nTo begin with, we prove by induction that $\\vw_t \\in \\vspan(\\calC)$. By initialization, we have $\\vw_0 \\in \\vspan(\\calC)$. Now suppose $\\vw_t \\in \\vspan(\\calC)$ for some $t \\geq 0$. Since $\\vg_t =  \\mP\\nabla \\ell_t(\\vx_t) \\in \\vspan(\\calC)$ and $\\vs_t \\in \\vspan(\\calC)$ by Definition~\\ref{def:gauge}, we obtain that $\\tilde{\\vg}_t \\in \\vspan(\\calC)$ in both \\textbf{Options I} and \\textbf{II}. Moreover, since $\\vw_{t+1}$ is expressed as a linear combination of $\\vw_t$ and $\\tilde{\\vg}_t$ according to \\eqref{eq:w_update}, we obtain that $\\vw_{t+1} \\in \\vspan(\\calC)$. Hence, we conclude by induction that $\\vw_t \\in \\vspan(\\calC)$ for any $t\\geq 0$.  \n\nNext, we consider the result in Part (a) for \\textbf{Option I}.  We distinguish two cases depending on the outcome of $\\mathsf{SEP}(\\vw_t;\\delta)$. \n\\begin{itemize}\n  \\item If $\\gamma_t \\leq 1$,  we have $\\vw_t \\in (1+\\delta) \\mathcal{C}$ by Definition~\\ref{def:gauge}. Thus, we have $\\vx_t=\\vw_t\\in (1+\\delta) \\mathcal{C}$ and $\\tilde{\\vg}_t = \\vg_t$, which immediately implies \\eqref{eq:surrogate_loss}. Moreover, since $\\ell_t$ is convex, we have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\nabla \\ell_t(\\vx_t), \\vx_t - \\vu \\rangle$. Note that both $\\vx_t$ and $\\vu$ are in $\\vspan(\\calC)$ and recall that $\\mP$ denotes the orthogonal projection matrix associated with $\\vspan(\\calC)$. Thus, we further have $\\vx_t - \\vu = \\mP(\\vx_t - \\vu)$, which implies that $\\langle \\nabla \\ell_t(\\vx_t), \\vx_t - \\vu \\rangle = \\langle \\nabla \\ell_t(\\vx_t), \\mP(\\vx_t - \\vu )\\rangle = \\langle \\mP\\nabla \\ell_t(\\vx_t), \\vx_t - \\vu \\rangle = \\langle \\vg_t, \\vx_t - \\vu \\rangle$. \n  Therefore, we have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle = \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle$, which proves the first inequality in \\eqref{eq:regret_reduction}. \n  \\item Otherwise, if $\\gamma_t>1$,  we have $\\frac{\\vw_t}{\\gamma_t} \\in (1+\\delta)\\mathcal{C}$ by Definition~\\ref{def:gauge} and $\\langle \\vs_t, \\vw_t-\\vx \\rangle \\geq {\\gamma_t-1}$,  $ \\forall\\vx\\in \\mathcal{C}$. In this case, Algorithm~\\ref{alg:projection_free_online_learning} chooses $\\vx_t  = \\frac{\\vw_t}{\\gamma_t} \\in (1+\\delta)\\mathcal{C}$ and $\\tilde{\\vg}_t = \\vg_t+\\max\\{0, -\\frac{1}{\\gamma_t}\\langle \\vg_t, \\vw_t \\rangle\\} \\vs_t = \\vg_t+\\max\\{0, -\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t$. To prove the first inequality in \\eqref{eq:regret_reduction}, \n  first note that we also have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle$ using similar arguments as above. Moreover, for any $\\vu\\in \\mathcal{C}$,\n  \\begin{align*}\n    \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle &= \\langle\\vg_t+\\max\\{0, -\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t, \\vw_t-\\vu \\rangle \\\\\n    & = \\langle \\vg_t, \\gamma_t\\vx_t-\\vu \\rangle + \\max\\{0, -\\langle \\vg_t, \\vx_t \\rangle\\} \\langle \\vs_t, \\vw_t-\\vu\\rangle \\\\\n    & \\geq \\langle \\vg_t, \\vx_t-\\vu\\rangle + (\\gamma_t-1)\\langle\\vg_t,\\vx_t\\rangle + (\\gamma_t-1)\\max\\{0, -\\langle \\vg_t, \\vx_t \\rangle\\}  \\\\\n    %\n    & \\geq \\langle \\vg_t, \\vx_t-\\vu\\rangle, \n  \\end{align*}\n  where we used $\\vw_t = \\gamma_t \\vx_t$ in the second equality and $\\langle \\vs_t, \\vw_t-\\vu\\rangle \\geq \\gamma_t -1$ in the first inequality. \n  Also, by the triangle inequality we obtain \n  \\begin{equation*}\n    \\|\\tilde{\\vg}_t\\|_2 =\\|\\vg_t+\\max\\{0, -\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t\\|_2 %\n    \\leq  \\|\\vg_t\\|_2+|\\langle \\vg_t, \\vx_t \\rangle|\\|\\vs_t\\|_2,\n  \\end{equation*}\n  which proves \\eqref{eq:surrogate_loss}. \n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n\\end{itemize}\nFinally, from the update rule of $\\vw_{t+1}$ in \\eqref{eq:w_update}, for any $\\vu\\in \\mathcal{C} \\subset \\mathcal{B}_R(0)$, we have \n$\n  \\langle\\vw_t-\\rho\\tilde{\\vg}_t-\\vw_{t+1}, \\vw_{t+1}-\\vu\\rangle \\geq 0\n$. This further implies that \n\\begin{align}\n  \\langle \\tilde{\\vg}_t, \\vw_{t}-\\vu \\rangle &\\leq  \\langle \\tilde{\\vg}_t, \\vw_{t}-\\vw_{t+1}\\rangle+ \\frac{1}{\\rho}\\langle \\vw_t-\\vw_{t+1}, \\vw_{t+1}-\\vu \\rangle \\nonumber\\\\\n  & =  \\langle \\tilde{\\vg}_t, \\vw_{t}-\\vw_{t+1}\\rangle+ \\frac{\\|\\vw_t-\\vu\\|_2^2}{2\\rho}-\\frac{\\|\\vw_{t+1}-\\vu\\|_2^2}{2\\rho}-\\frac{\\|\\vw_t-\\vw_{t+1}\\|_2^2}{2\\rho} \\nonumber\\\\\n  &\\leq \\frac{1}{2\\rho}\\|\\vw_t-\\vu\\|_2^2-\\frac{1}{2\\rho}\\|\\vw_{t+1}-\\vu\\|_2^2 +\\frac{\\rho}{2}\\|\\tilde{\\vg}_t\\|_2^2, \\label{eq:regret_wt}\n\\end{align}\nwhere we used the Young's inequality $\\langle \\tilde{\\vg}_t, \\vw_{t}-\\vw_{t+1}\\rangle \\leq \\frac{\\|\\vw_{t}-\\vw_{t+1}\\|^2}{2\\rho} + \\frac{\\rho}{2}\\|\\tilde{\\vg}_t\\|^2$ in the last inequality. \nThis proves the second inequality in \\eqref{eq:regret_reduction}. \n\n\\blue{Next, we consider Part (b) for \\textbf{Option II}. Similarly, we distinguish two cases depending on the outcome of $\\mathsf{SEP}(\\vw_t;\\delta)$. \n\\begin{itemize}\n  \\item If $\\gamma_t \\leq 1$,  we have $\\vw_t \\in (1+\\delta) \\mathcal{C}$ by Definition~\\ref{def:gauge}. According to \\textbf{Option II} in Algorithm~\\ref{alg:projection_free_online_learning}, we have $\\vx_t= \\frac{\\vw_t}{1+\\delta}\\in \\mathcal{C}$ and $\\tilde{\\vg}_t = \\vg_t$, which immediately implies~\\eqref{eq:surrogate_loss_II}. Using similar arguments as in \\textbf{Option I}, we have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle$. Moreover, since $\\vw_t = (1+\\delta)\\vx_t$, we have $\\langle {\\vg}_t, \\vw_t - \\vu \\rangle = \\langle {\\vg}_t, (1+\\delta)\\vx_t - \\vu \\rangle = \\langle {\\vg}_t, \\vx_t-\\vu \\rangle + \\delta \\langle {\\vg}_t, \\vx_t\\rangle$. Combining these two, we further have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle \\leq\\langle {\\vg}_t, \\vw_t - \\vu \\rangle- \\delta \\langle {\\vg}_t, \\vx_t\\rangle$. This proves the inequality in \\eqref{eq:regret_reduction_II_1}. \n  %\n  \\item Otherwise, if $\\gamma_t>1$, By Definition~\\ref{def:gauge} we have $ \\frac{\\vw_t}{\\gamma_t} \\in (1+\\delta)\\mathcal{C}$ and $\\langle \\vs_t, \\vw_t-\\vx \\rangle \\geq {\\gamma_t-1}$  $ \\forall\\vx\\in \\mathcal{C}$. According to \\textbf{Option II} in Algorithm~\\ref{alg:projection_free_online_learning}, we have $\\vx_t  = \\frac{\\vw_t}{\\gamma_t(1+\\delta)} \\in \\mathcal{C}$ and $\\tilde{\\vg}_t = \\vg_t+\\max\\{0, -\\frac{1}{\\gamma_t}\\langle \\vg_t, \\vw_t \\rangle\\} \\vs_t = \\vg_t+\\max\\{0, -(1+\\delta)\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t$. \n  %\n  Following similar arguments as in \\textbf{Option I}, we have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle$. Moreover, for any $\\vu\\in \\mathcal{C}$, \n  \\begin{align*}\n    \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle &= \\langle\\vg_t+\\max\\{0, -(1+\\delta)\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t, \\vw_t-\\vu \\rangle \\\\\n    & = \\langle \\vg_t, (1+\\delta)\\gamma_t\\vx_t-\\vu \\rangle + \\max\\{0, -(1+\\delta)\\langle \\vg_t, \\vx_t \\rangle\\} \\langle \\vs_t, \\vw_t-\\vu\\rangle \\\\\n    %\n    %\n    & \\geq \\langle \\vg_t, (1+\\delta)\\vx_t-\\vu\\rangle, \n  \\end{align*}\n  where we used $\\langle \\vs_t, \\vw_t-\\vu\\rangle \\geq \\gamma_t -1$ in the first inequality. By rearranging the above inequality, we obtain that $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle \\leq \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle - \\delta \\langle \\vg_t, \\vx_t \\rangle$, which proves~\\eqref{eq:regret_reduction_II_1}.  \n  Also, by the triangle inequality we obtain \n  \\begin{equation*}\n    \\|\\tilde{\\vg}_t\\|_2 =\\|\\vg_t+\\max\\{0, -(1+\\delta)\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t\\|_2 %\n    \\leq  \\|\\vg_t\\|_2+(1+\\delta)|\\langle \\vg_t, \\vx_t \\rangle|\\|\\vs_t\\|_2,\n  \\end{equation*}\n  which proves \\eqref{eq:surrogate_loss_II}. \n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n\\end{itemize}\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\nFinally, the second inequality in \\eqref{eq:regret_reduction_II_2} can be shown similarly as in \\textbf{Option I}. \n }\n\\end{proof}","sha256_utf8":"a221e01134a1df26426c6875f90be06934c89404e75747b01b832abe00f58ea3"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"归纳保持线性子空间","source_units":[{"start":108241,"end":108930,"start_line":1924,"end_line":1931,"raw_tex":"\\begin{proof}\n  %\n  %\n  %\n  %\n%\n%\nTo begin with, we prove by induction that $\\vw_t \\in \\vspan(\\calC)$. By initialization, we have $\\vw_0 \\in \\vspan(\\calC)$. Now suppose $\\vw_t \\in \\vspan(\\calC)$ for some $t \\geq 0$. Since $\\vg_t =  \\mP\\nabla \\ell_t(\\vx_t) \\in \\vspan(\\calC)$ and $\\vs_t \\in \\vspan(\\calC)$ by Definition~\\ref{def:gauge}, we obtain that $\\tilde{\\vg}_t \\in \\vspan(\\calC)$ in both \\textbf{Options I} and \\textbf{II}. Moreover, since $\\vw_{t+1}$ is expressed as a linear combination of $\\vw_t$ and $\\tilde{\\vg}_t$ according to \\eqref{eq:w_update}, we obtain that $\\vw_{t+1} \\in \\vspan(\\calC)$. Hence, we conclude by induction that $\\vw_t \\in \\vspan(\\calC)$ for any $t\\geq 0$.  \n","sha256_utf8":"c654f4d9c13bd0984254b797f6ab8a9495d2244b9f71be0c95d7319c6b0bd3c0"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"选项一的近可行情形","source_units":[{"start":108931,"end":110099,"start_line":1933,"end_line":1936,"raw_tex":"Next, we consider the result in Part (a) for \\textbf{Option I}.  We distinguish two cases depending on the outcome of $\\mathsf{SEP}(\\vw_t;\\delta)$. \n\\begin{itemize}\n  \\item If $\\gamma_t \\leq 1$,  we have $\\vw_t \\in (1+\\delta) \\mathcal{C}$ by Definition~\\ref{def:gauge}. Thus, we have $\\vx_t=\\vw_t\\in (1+\\delta) \\mathcal{C}$ and $\\tilde{\\vg}_t = \\vg_t$, which immediately implies \\eqref{eq:surrogate_loss}. Moreover, since $\\ell_t$ is convex, we have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\nabla \\ell_t(\\vx_t), \\vx_t - \\vu \\rangle$. Note that both $\\vx_t$ and $\\vu$ are in $\\vspan(\\calC)$ and recall that $\\mP$ denotes the orthogonal projection matrix associated with $\\vspan(\\calC)$. Thus, we further have $\\vx_t - \\vu = \\mP(\\vx_t - \\vu)$, which implies that $\\langle \\nabla \\ell_t(\\vx_t), \\vx_t - \\vu \\rangle = \\langle \\nabla \\ell_t(\\vx_t), \\mP(\\vx_t - \\vu )\\rangle = \\langle \\mP\\nabla \\ell_t(\\vx_t), \\vx_t - \\vu \\rangle = \\langle \\vg_t, \\vx_t - \\vu \\rangle$. \n  Therefore, we have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle = \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle$, which proves the first inequality in \\eqref{eq:regret_reduction}. \n","sha256_utf8":"7b2c5798f2022cd23d8afb11fe1d85c964f34519ac0bd82c8f97846cbb1934b1"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"选项一的分离情形与梯度界","source_units":[{"start":110099,"end":111790,"start_line":1937,"end_line":1962,"raw_tex":"  \\item Otherwise, if $\\gamma_t>1$,  we have $\\frac{\\vw_t}{\\gamma_t} \\in (1+\\delta)\\mathcal{C}$ by Definition~\\ref{def:gauge} and $\\langle \\vs_t, \\vw_t-\\vx \\rangle \\geq {\\gamma_t-1}$,  $ \\forall\\vx\\in \\mathcal{C}$. In this case, Algorithm~\\ref{alg:projection_free_online_learning} chooses $\\vx_t  = \\frac{\\vw_t}{\\gamma_t} \\in (1+\\delta)\\mathcal{C}$ and $\\tilde{\\vg}_t = \\vg_t+\\max\\{0, -\\frac{1}{\\gamma_t}\\langle \\vg_t, \\vw_t \\rangle\\} \\vs_t = \\vg_t+\\max\\{0, -\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t$. To prove the first inequality in \\eqref{eq:regret_reduction}, \n  first note that we also have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle$ using similar arguments as above. Moreover, for any $\\vu\\in \\mathcal{C}$,\n  \\begin{align*}\n    \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle &= \\langle\\vg_t+\\max\\{0, -\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t, \\vw_t-\\vu \\rangle \\\\\n    & = \\langle \\vg_t, \\gamma_t\\vx_t-\\vu \\rangle + \\max\\{0, -\\langle \\vg_t, \\vx_t \\rangle\\} \\langle \\vs_t, \\vw_t-\\vu\\rangle \\\\\n    & \\geq \\langle \\vg_t, \\vx_t-\\vu\\rangle + (\\gamma_t-1)\\langle\\vg_t,\\vx_t\\rangle + (\\gamma_t-1)\\max\\{0, -\\langle \\vg_t, \\vx_t \\rangle\\}  \\\\\n    %\n    & \\geq \\langle \\vg_t, \\vx_t-\\vu\\rangle, \n  \\end{align*}\n  where we used $\\vw_t = \\gamma_t \\vx_t$ in the second equality and $\\langle \\vs_t, \\vw_t-\\vu\\rangle \\geq \\gamma_t -1$ in the first inequality. \n  Also, by the triangle inequality we obtain \n  \\begin{equation*}\n    \\|\\tilde{\\vg}_t\\|_2 =\\|\\vg_t+\\max\\{0, -\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t\\|_2 %\n    \\leq  \\|\\vg_t\\|_2+|\\langle \\vg_t, \\vx_t \\rangle|\\|\\vs_t\\|_2,\n  \\end{equation*}\n  which proves \\eqref{eq:surrogate_loss}. \n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n\\end{itemize}\n","sha256_utf8":"4b53fc0904bfe38e9c0cfc41ab7249caab9b1a45f810b28c18011423a0518eee"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"投影梯度步的势函数界","source_units":[{"start":111790,"end":112796,"start_line":1963,"end_line":1973,"raw_tex":"Finally, from the update rule of $\\vw_{t+1}$ in \\eqref{eq:w_update}, for any $\\vu\\in \\mathcal{C} \\subset \\mathcal{B}_R(0)$, we have \n$\n  \\langle\\vw_t-\\rho\\tilde{\\vg}_t-\\vw_{t+1}, \\vw_{t+1}-\\vu\\rangle \\geq 0\n$. This further implies that \n\\begin{align}\n  \\langle \\tilde{\\vg}_t, \\vw_{t}-\\vu \\rangle &\\leq  \\langle \\tilde{\\vg}_t, \\vw_{t}-\\vw_{t+1}\\rangle+ \\frac{1}{\\rho}\\langle \\vw_t-\\vw_{t+1}, \\vw_{t+1}-\\vu \\rangle \\nonumber\\\\\n  & =  \\langle \\tilde{\\vg}_t, \\vw_{t}-\\vw_{t+1}\\rangle+ \\frac{\\|\\vw_t-\\vu\\|_2^2}{2\\rho}-\\frac{\\|\\vw_{t+1}-\\vu\\|_2^2}{2\\rho}-\\frac{\\|\\vw_t-\\vw_{t+1}\\|_2^2}{2\\rho} \\nonumber\\\\\n  &\\leq \\frac{1}{2\\rho}\\|\\vw_t-\\vu\\|_2^2-\\frac{1}{2\\rho}\\|\\vw_{t+1}-\\vu\\|_2^2 +\\frac{\\rho}{2}\\|\\tilde{\\vg}_t\\|_2^2, \\label{eq:regret_wt}\n\\end{align}\nwhere we used the Young's inequality $\\langle \\tilde{\\vg}_t, \\vw_{t}-\\vw_{t+1}\\rangle \\leq \\frac{\\|\\vw_{t}-\\vw_{t+1}\\|^2}{2\\rho} + \\frac{\\rho}{2}\\|\\tilde{\\vg}_t\\|^2$ in the last inequality. \nThis proves the second inequality in \\eqref{eq:regret_reduction}. \n","sha256_utf8":"29bbe3e243d2c4cccf4f2da97cb0d5c8d5474429ef34c41db302c2e3427a0669"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"选项二的近可行情形","source_units":[{"start":112797,"end":113898,"start_line":1975,"end_line":1978,"raw_tex":"\\blue{Next, we consider Part (b) for \\textbf{Option II}. Similarly, we distinguish two cases depending on the outcome of $\\mathsf{SEP}(\\vw_t;\\delta)$. \n\\begin{itemize}\n  \\item If $\\gamma_t \\leq 1$,  we have $\\vw_t \\in (1+\\delta) \\mathcal{C}$ by Definition~\\ref{def:gauge}. According to \\textbf{Option II} in Algorithm~\\ref{alg:projection_free_online_learning}, we have $\\vx_t= \\frac{\\vw_t}{1+\\delta}\\in \\mathcal{C}$ and $\\tilde{\\vg}_t = \\vg_t$, which immediately implies~\\eqref{eq:surrogate_loss_II}. Using similar arguments as in \\textbf{Option I}, we have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle$. Moreover, since $\\vw_t = (1+\\delta)\\vx_t$, we have $\\langle {\\vg}_t, \\vw_t - \\vu \\rangle = \\langle {\\vg}_t, (1+\\delta)\\vx_t - \\vu \\rangle = \\langle {\\vg}_t, \\vx_t-\\vu \\rangle + \\delta \\langle {\\vg}_t, \\vx_t\\rangle$. Combining these two, we further have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle \\leq\\langle {\\vg}_t, \\vw_t - \\vu \\rangle- \\delta \\langle {\\vg}_t, \\vx_t\\rangle$. This proves the inequality in \\eqref{eq:regret_reduction_II_1}. \n  %\n","sha256_utf8":"9eaedb80f8c3016edc5869c6da68f109067fcca3da1e299c733231ca22a54788"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"选项二的分离情形与梯度界","source_units":[{"start":113898,"end":115712,"start_line":1979,"end_line":2015,"raw_tex":"  \\item Otherwise, if $\\gamma_t>1$, By Definition~\\ref{def:gauge} we have $ \\frac{\\vw_t}{\\gamma_t} \\in (1+\\delta)\\mathcal{C}$ and $\\langle \\vs_t, \\vw_t-\\vx \\rangle \\geq {\\gamma_t-1}$  $ \\forall\\vx\\in \\mathcal{C}$. According to \\textbf{Option II} in Algorithm~\\ref{alg:projection_free_online_learning}, we have $\\vx_t  = \\frac{\\vw_t}{\\gamma_t(1+\\delta)} \\in \\mathcal{C}$ and $\\tilde{\\vg}_t = \\vg_t+\\max\\{0, -\\frac{1}{\\gamma_t}\\langle \\vg_t, \\vw_t \\rangle\\} \\vs_t = \\vg_t+\\max\\{0, -(1+\\delta)\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t$. \n  %\n  Following similar arguments as in \\textbf{Option I}, we have $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle$. Moreover, for any $\\vu\\in \\mathcal{C}$, \n  \\begin{align*}\n    \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle &= \\langle\\vg_t+\\max\\{0, -(1+\\delta)\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t, \\vw_t-\\vu \\rangle \\\\\n    & = \\langle \\vg_t, (1+\\delta)\\gamma_t\\vx_t-\\vu \\rangle + \\max\\{0, -(1+\\delta)\\langle \\vg_t, \\vx_t \\rangle\\} \\langle \\vs_t, \\vw_t-\\vu\\rangle \\\\\n    %\n    %\n    & \\geq \\langle \\vg_t, (1+\\delta)\\vx_t-\\vu\\rangle, \n  \\end{align*}\n  where we used $\\langle \\vs_t, \\vw_t-\\vu\\rangle \\geq \\gamma_t -1$ in the first inequality. By rearranging the above inequality, we obtain that $\\ell_t(\\vx_t) - \\ell_t(\\vu) \\leq \\langle \\vg_t, \\vx_t - \\vu \\rangle \\leq \\langle \\tilde{\\vg}_t, \\vw_t-\\vu \\rangle - \\delta \\langle \\vg_t, \\vx_t \\rangle$, which proves~\\eqref{eq:regret_reduction_II_1}.  \n  Also, by the triangle inequality we obtain \n  \\begin{equation*}\n    \\|\\tilde{\\vg}_t\\|_2 =\\|\\vg_t+\\max\\{0, -(1+\\delta)\\langle \\vg_t, \\vx_t \\rangle\\} \\vs_t\\|_2 %\n    \\leq  \\|\\vg_t\\|_2+(1+\\delta)|\\langle \\vg_t, \\vx_t \\rangle|\\|\\vs_t\\|_2,\n  \\end{equation*}\n  which proves \\eqref{eq:surrogate_loss_II}. \n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n  %\n\\end{itemize}\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n","sha256_utf8":"4f2b655cd5c7488cc0e8e38e2c6f4c612b29001f63a43002dd3e9c2d79899427"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"完成选项二的势函数界","source_units":[{"start":115712,"end":115842,"start_line":2016,"end_line":2018,"raw_tex":"Finally, the second inequality in \\eqref{eq:regret_reduction_II_2} can be shown similarly as in \\textbf{Option I}. \n }\n\\end{proof}","sha256_utf8":"eb8634921c16b30d72992edea4b6989f297f2330d94c0dddc5c883e2c80273d1"}]},{"id":"S051","semantic_type":"lemma","kind":"lemma","title":"交集分离预言机","source_units":[{"start":116561,"end":117450,"start_line":2061,"end_line":2070,"raw_tex":"\\begin{lemma}\\label{lem:sep_intersect}\n  Assume that $\\mathcal{C} = \\mathcal{C}_1 \\cap \\mathcal{C}_2 \\cap \\mathcal{L}$, where $0 \\in \\calC_1 \\cap \\calC_2$ and $\\mathcal{L}= \\vspan(\\calC)$ is a linear subspace. Suppose we have access to $\\mathsf{SEP}_{\\mathcal{C}_1}(\\vw; \\delta)$, $\\mathsf{SEP}_{\\mathcal{C}_2}(\\vw; \\delta)$ and the orthogonal projection matrix $\\mP_{\\calL}$ associated with the subspace $\\calL$. Then given inputs $\\vw \\in \\calL$ and $\\delta>0$, the oracle $\\mathsf{SEP}_{\\mathcal{C}}(\\vw; \\delta)$ can be constructed in the following ways: \n  \\begin{enumerate}[(a)]\n  %\n    \\item  let $(\\gamma_1,\\vs_1) = \\mathsf{SEP}_{\\mathcal{C}_1}(\\vw; \\delta)$ and $(\\gamma_2,\\vs_2) = \\mathsf{SEP}_{\\mathcal{C}_2}(\\vw; \\delta)$.\n    %\n    \\item Let $i = \\argmax_{i \\in \\{1,2\\}} \\gamma_i$ and then output $(\\gamma,\\vs) = (\\gamma_i,\\mP_{\\calL}\\vs_i)$.\n  \\end{enumerate}\n  %\n\\end{lemma}","sha256_utf8":"d7186e88f8604530cf37f2c4ec9cea56d72720809cb29631a5286538a0339ec5"}]},{"id":"S052","semantic_type":"proof","kind":"proof","title":"交集预言机引理证明","source_units":[{"start":117451,"end":119611,"start_line":2071,"end_line":2082,"raw_tex":"\\begin{proof}\nWithout loss of generality, we assume $\\gamma_1 \\geq \\gamma_2$ and thus the output is given by $(\\gamma,\\vs) = (\\gamma_1,\\mP_{\\calL}\\vs_1)$. We consider two cases depending on whether $\\gamma_1 \\leq 1$ or not. \n\nIn the first case where $\\gamma = \\gamma_1 \\leq 1$, we have $\\gamma_2 \\leq \\gamma_1 \\leq 1$. According to the definitions of $\\SEP_{\\calC_1}(\\vw;\\delta)$ and $ \\SEP_{\\calC_2}(\\vw;\\delta)$, we have $ \\vw \\in (1+\\delta)\\calC_1$ and $\\vw \\in (1+\\delta)\\calC_2$. This implies that $\\vw \\in (1+\\delta) (\\calC_1 \\cap \\calC_2)$. Moreover, since we assume that $\\vw \\in \\calL$, this leads to $ \\vw \\in (1+\\delta) (\\calC_1 \\cap \\calC_2 \\cap \\calL) = (1+\\delta) \\calC$. \n\nIn the second case where $ \\gamma = \\gamma_1 > 1$, according to the definition of $\\SEP_{\\calC_1}(\\vw;\\delta)$ we have $\\vw/\\gamma_1 \\in (1+\\delta)\\calC_1$ and $ \\langle \\vs_1, \\vw-\\vx \\rangle \\geq \\gamma_1 - 1$ for all $\\vx \\in \\calC_1$. Moreover, by the definition of $\\SEP_{\\calC_2}(\\vw;\\delta)$, we have $\\vw/\\gamma_2 \\in (1+\\delta)\\calC_2$.\nFirst, we show that $\\vw/\\gamma \\in (1+\\delta)\\calC$. \nSince $\\gamma_2 \\leq \\gamma_1$, we can write $ \\vw/\\gamma_1$ as a convex combination of the origin and $\\vw/\\gamma_2$, and thus it follows from the convexity of $\\calC_2$ that $ \\vw/\\gamma_1 \\in (1+\\delta) \\calC_2$. Similar to the arguments in the first case, we obtain $\\vw/\\gamma =  \\vw/\\gamma_1 \\in (1+\\delta) \\calC$. \nNext, we will prove that $\\langle \\vs, \\vw-\\vx \\rangle \\geq \\gamma-1$.  Since $\\calC_1 \\subset \\calC$, we have $\\vx \\in \\calC_1$ for any $\\vx \\in \\calC$. Therefore,  it implies that $\\langle \\vs_1, \\vw - \\vx \\rangle \\geq \\gamma_1-1 = \\gamma-1$. Moreover, since $\\vw \\in \\calL$ and $\\vx \\in \\calL$, we have $\\mP_{\\calL}(\\vw-\\vx) = \\vw-\\vx$. Therefore, we conclude that $\\langle \\vs, \\vw-\\vx \\rangle = \\langle \\mP_{\\calL}\\vs_1, \\vw-\\vx \\rangle = \\langle \\vs_1, \\mP_{\\calL}(\\vw-\\vx) \\rangle = \\langle \\vs_1, \\vw-\\vx \\rangle \\geq \\gamma_1 -1 = \\gamma-1$. \n\nCombining both cases, we observe that the procedure described in Lemma~\\ref{lem:sep_intersect} indeed satisfies the definition in Definition~\\ref{def:gauge}. This completes the proof. \n\\end{proof}","sha256_utf8":"7e006391bd771e7d5dee56d86fdd8ecc2cd0c41949c348053ca14df0b4f6bd73"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"选择较大的缩放因子","source_units":[{"start":117451,"end":117676,"start_line":2071,"end_line":2072,"raw_tex":"\\begin{proof}\nWithout loss of generality, we assume $\\gamma_1 \\geq \\gamma_2$ and thus the output is given by $(\\gamma,\\vs) = (\\gamma_1,\\mP_{\\calL}\\vs_1)$. We consider two cases depending on whether $\\gamma_1 \\leq 1$ or not. \n","sha256_utf8":"c8200f853a044ade08c01e3e8630fbb48dcc1efaceaea147ffa2e4458f8e2b4b"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"两个近可行证书合并","source_units":[{"start":117677,"end":118138,"start_line":2074,"end_line":2074,"raw_tex":"In the first case where $\\gamma = \\gamma_1 \\leq 1$, we have $\\gamma_2 \\leq \\gamma_1 \\leq 1$. According to the definitions of $\\SEP_{\\calC_1}(\\vw;\\delta)$ and $ \\SEP_{\\calC_2}(\\vw;\\delta)$, we have $ \\vw \\in (1+\\delta)\\calC_1$ and $\\vw \\in (1+\\delta)\\calC_2$. This implies that $\\vw \\in (1+\\delta) (\\calC_1 \\cap \\calC_2)$. Moreover, since we assume that $\\vw \\in \\calL$, this leads to $ \\vw \\in (1+\\delta) (\\calC_1 \\cap \\calC_2 \\cap \\calL) = (1+\\delta) \\calC$. \n","sha256_utf8":"c58d05163eb103dda1809a443d78ca3f386a192cbccba0c87820168225142548"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"凸性保证缩放后属于交集","source_units":[{"start":118139,"end":118862,"start_line":2076,"end_line":2078,"raw_tex":"In the second case where $ \\gamma = \\gamma_1 > 1$, according to the definition of $\\SEP_{\\calC_1}(\\vw;\\delta)$ we have $\\vw/\\gamma_1 \\in (1+\\delta)\\calC_1$ and $ \\langle \\vs_1, \\vw-\\vx \\rangle \\geq \\gamma_1 - 1$ for all $\\vx \\in \\calC_1$. Moreover, by the definition of $\\SEP_{\\calC_2}(\\vw;\\delta)$, we have $\\vw/\\gamma_2 \\in (1+\\delta)\\calC_2$.\nFirst, we show that $\\vw/\\gamma \\in (1+\\delta)\\calC$. \nSince $\\gamma_2 \\leq \\gamma_1$, we can write $ \\vw/\\gamma_1$ as a convex combination of the origin and $\\vw/\\gamma_2$, and thus it follows from the convexity of $\\calC_2$ that $ \\vw/\\gamma_1 \\in (1+\\delta) \\calC_2$. Similar to the arguments in the first case, we obtain $\\vw/\\gamma =  \\vw/\\gamma_1 \\in (1+\\delta) \\calC$. \n","sha256_utf8":"5a088edb21493560b1224de06863e120a104f98f26c039a5e3d08d7aa989f505"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"投影保持分离不等式","source_units":[{"start":118862,"end":119611,"start_line":2079,"end_line":2082,"raw_tex":"Next, we will prove that $\\langle \\vs, \\vw-\\vx \\rangle \\geq \\gamma-1$.  Since $\\calC_1 \\subset \\calC$, we have $\\vx \\in \\calC_1$ for any $\\vx \\in \\calC$. Therefore,  it implies that $\\langle \\vs_1, \\vw - \\vx \\rangle \\geq \\gamma_1-1 = \\gamma-1$. Moreover, since $\\vw \\in \\calL$ and $\\vx \\in \\calL$, we have $\\mP_{\\calL}(\\vw-\\vx) = \\vw-\\vx$. Therefore, we conclude that $\\langle \\vs, \\vw-\\vx \\rangle = \\langle \\mP_{\\calL}\\vs_1, \\vw-\\vx \\rangle = \\langle \\vs_1, \\mP_{\\calL}(\\vw-\\vx) \\rangle = \\langle \\vs_1, \\vw-\\vx \\rangle \\geq \\gamma_1 -1 = \\gamma-1$. \n\nCombining both cases, we observe that the procedure described in Lemma~\\ref{lem:sep_intersect} indeed satisfies the definition in Definition~\\ref{def:gauge}. This completes the proof. \n\\end{proof}","sha256_utf8":"91511b86f8fea8c8808fc620770241355263678558bafc09900727092f0f455f"}]},{"id":"S057","semantic_type":"lemma","kind":"lemma","title":"可行集的平移缩放","source_units":[{"start":124876,"end":125639,"start_line":2217,"end_line":2223,"raw_tex":"\\begin{lemma}\\label{lem:transform}\n   Recall the set $\\calZ$ defined in \\eqref{eq:feasible_set} and suppose that the linear subspace $\\calL$ satisfies $\\mI \\in \\calL$.  Define the set $\\calC$ as the following: \n   \\begin{equation}\\label{eq:translated_set}\n  {\\mathcal{C}} \\mydef \\left\\{\\hat{\\mB} \\in \\reals^{d\\times d}: -\\mI \\preceq \\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top ) \\preceq \\mI , \\ \\|\\hat{\\mB}\\|_{\\op} \\leq 3, \\ \\hat{\\mB} \\in \\calL \\right\\}. \n  \\end{equation}\n    Let $\\hat{\\mB} \\mydef \\frac{1}{L_1}(\\mB-(L_1+\\mu)\\mI)$. If $\\mB \\in \\calZ$, then $\\hat{\\mB} \\in \\mathcal{C}$. Conversely, if $\\hat{\\mB} \\in (1+\\delta)\\mathcal{C}$, then  $\\frac{1}{2}(\\mB + \\mB^\\top )\\succeq (\\mu -L_1\\delta) \\mI$ and $\\|\\mB\\|_{\\op} \\leq 4L_1 + \\mu+ 3\\delta L_1$. \n\\end{lemma}","sha256_utf8":"692e52c0e0a2a3218755c120316ce6dee5fcb848ac4a062812a6f63ebf566f22"}]},{"id":"S058","semantic_type":"proof","kind":"proof","title":"平移缩放引理证明","source_units":[{"start":125640,"end":127142,"start_line":2224,"end_line":2240,"raw_tex":"\\begin{proof}\n  First, we show that $\\mB\\in \\calZ$ implies $\\hat{\\mB} \\in \\calC$. By the definition of $\\hat{\\mB}$, we have  \n%\n$\\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top ) = \\frac{1}{2L_1}(\\mB + \\mB^\\top) - \\frac{L_1+\\mu}{L_1}\\mI \\succeq \\frac{\\mu}{L_1} \\mI - \\frac{L_1+\\mu}{L_1}\\mI = -\\mI$. \n%\nMoreover, we also have \n%\n$\\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top ) \\preceq  \\frac{1}{2L_1}(\\mB + \\mB^\\top) \\preceq \\mI$.\n%\nCombining these two, we obtain that $ -\\mI \\preceq \\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top ) \\preceq \\mI$. Additionally, we can bound $\\|\\hat{\\mB}\\|_{\\op} \\leq \\frac{1}{L_1} (\\|\\mB\\|_{\\op} + (L_1+\\mu)) \\leq 3$ since $\\|\\mB\\|_{\\op} \\leq L_1$ and $\\mu \\leq L_1$. Finally, note that $\\hat{\\mB}$ is a linear combination of $\\mB$ and $\\mI$. Since $\\mI \\in \\calL$ and $\\mB \\in \\calL$, we also have $\\hat{\\mB} \\in \\calL$. Hence, we conclude that $\\hat{\\mB} \\in \\calC$. \n\nFor the other direction, assume $\\hat{\\mB} \\in (1+\\delta)\\calC$ and note that $ \\mB = L_1 \\hat{\\mB} + (L_1+\\mu)\\mI$. Since $\\hat{\\mB} \\in (1+\\delta)\\calC$ implies that $\\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top) \\succeq -(1+\\delta)\\mI$, we have \n%\n$\\frac{1}{2}(\\mB + \\mB^\\top) \\succeq \\frac{L_1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top) + (L_1 + \\mu) \\mI \\succeq (\\mu-L_1 \\delta)\\mI$.\n%\nFinally, $\\hat{\\mB} \\in (1+\\delta)\\calC$ also implies that $\\|\\hat{\\mB}\\|_{\\op} \\leq 3(1+\\delta)$, and thus it holds that $\\|\\mB\\|_{\\op}  \\leq L_1\\|\\hat{\\mB}\\|_{\\op} + (L_1+\\mu) \\leq 4L_1 + \\mu+ 3\\delta L_1$. This completes the proof.  \n\\end{proof}","sha256_utf8":"1b7c54a0ab7dfcaaa80dcbb32176530d07c7ced9ed13ed446351a2eda271c709"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"原可行矩阵映入中心化集合","source_units":[{"start":125640,"end":126519,"start_line":2224,"end_line":2233,"raw_tex":"\\begin{proof}\n  First, we show that $\\mB\\in \\calZ$ implies $\\hat{\\mB} \\in \\calC$. By the definition of $\\hat{\\mB}$, we have  \n%\n$\\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top ) = \\frac{1}{2L_1}(\\mB + \\mB^\\top) - \\frac{L_1+\\mu}{L_1}\\mI \\succeq \\frac{\\mu}{L_1} \\mI - \\frac{L_1+\\mu}{L_1}\\mI = -\\mI$. \n%\nMoreover, we also have \n%\n$\\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top ) \\preceq  \\frac{1}{2L_1}(\\mB + \\mB^\\top) \\preceq \\mI$.\n%\nCombining these two, we obtain that $ -\\mI \\preceq \\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top ) \\preceq \\mI$. Additionally, we can bound $\\|\\hat{\\mB}\\|_{\\op} \\leq \\frac{1}{L_1} (\\|\\mB\\|_{\\op} + (L_1+\\mu)) \\leq 3$ since $\\|\\mB\\|_{\\op} \\leq L_1$ and $\\mu \\leq L_1$. Finally, note that $\\hat{\\mB}$ is a linear combination of $\\mB$ and $\\mI$. Since $\\mI \\in \\calL$ and $\\mB \\in \\calL$, we also have $\\hat{\\mB} \\in \\calL$. Hence, we conclude that $\\hat{\\mB} \\in \\calC$. \n","sha256_utf8":"1248ac691d3f91d90410d55ff28ed87e862d1daa9dd4975382b9c568c5939954"}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","title":"反向变换给出谱界与范数界","source_units":[{"start":126520,"end":127142,"start_line":2235,"end_line":2240,"raw_tex":"For the other direction, assume $\\hat{\\mB} \\in (1+\\delta)\\calC$ and note that $ \\mB = L_1 \\hat{\\mB} + (L_1+\\mu)\\mI$. Since $\\hat{\\mB} \\in (1+\\delta)\\calC$ implies that $\\frac{1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top) \\succeq -(1+\\delta)\\mI$, we have \n%\n$\\frac{1}{2}(\\mB + \\mB^\\top) \\succeq \\frac{L_1}{2}(\\hat{\\mB} + \\hat{\\mB}^\\top) + (L_1 + \\mu) \\mI \\succeq (\\mu-L_1 \\delta)\\mI$.\n%\nFinally, $\\hat{\\mB} \\in (1+\\delta)\\calC$ also implies that $\\|\\hat{\\mB}\\|_{\\op} \\leq 3(1+\\delta)$, and thus it holds that $\\|\\mB\\|_{\\op}  \\leq L_1\\|\\hat{\\mB}\\|_{\\op} + (L_1+\\mu) \\leq 4L_1 + \\mu+ 3\\delta L_1$. This completes the proof.  \n\\end{proof}","sha256_utf8":"80a08fbdffdc946b7e660e6cfbe0ebd3055673a8f497b5bf5b526f7b53b2eb8d"}]},{"id":"S061","semantic_type":"proof","kind":"proof","title":"矩阵在线更新的预言机实例化","source_units":[{"start":133406,"end":133888,"start_line":2314,"end_line":2315,"raw_tex":"The oracles in Definitions~\\ref{def:extevec} and~\\ref{def:MaxSvec} provide the approximate separation oracles for $\\calC_1$ and $\\calC_2$, respectively. Using these two building blocks, we can construct the approximate separation oracle for $\\calC$ by following the procedure in Lemma~\\ref{lem:sep_intersect}. \nBy instantiating Algorithm~\\ref{alg:projection_free_online_learning}, we obtain the complete Jacobian approximation update given in Subroutine~\\ref{alg:jacobian_approx}. \n","sha256_utf8":"922164f7f1dcc0e9b2ddf17a9d1f8cc7c289a07e1b606d58d5204654d502c654"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"组合谱预言机并实例化在线算法","source_units":[{"start":133406,"end":133888,"start_line":2314,"end_line":2315,"raw_tex":"The oracles in Definitions~\\ref{def:extevec} and~\\ref{def:MaxSvec} provide the approximate separation oracles for $\\calC_1$ and $\\calC_2$, respectively. Using these two building blocks, we can construct the approximate separation oracle for $\\calC$ by following the procedure in Lemma~\\ref{lem:sep_intersect}. \nBy instantiating Algorithm~\\ref{alg:projection_free_online_learning}, we obtain the complete Jacobian approximation update given in Subroutine~\\ref{alg:jacobian_approx}. \n","sha256_utf8":"922164f7f1dcc0e9b2ddf17a9d1f8cc7c289a07e1b606d58d5204654d502c654"}]},{"id":"S063","semantic_type":"proposition","kind":"proposition","title":"随机Lanczos误差界","source_units":[{"start":142284,"end":143426,"start_line":2492,"end_line":2506,"raw_tex":"\\begin{proposition}[{\\cite[Theorem~4.2]{kuczynski1992estimating}}]\\label{prop:lanczos}\n  Consider a symmetric matrix $\\mW$ and let $\\lambda_1(\\mW)$ and $\\lambda_d(\\mW)$ denote its largest and smallest eigenvalues, respectively. Then after $k$ iterations of the Lanczos method with a random start, we find unit vectors $\\vu^{(1)}$ and $\\vu^{(d)}$  such that \n  \\begin{align*}\n    \\mathbb{P}(\\langle\\mW \\vu^{(1)}, \\vu^{(1)}\\rangle\\leq \\lambda_1(\\mW)-\\epsilon(\\lambda_1(\\mW)-\\lambda_d(\\mW))) \\leq 1.648\\sqrt{d}e^{-\\sqrt{\\epsilon}(2k-1)},\\\\\n    \\mathbb{P}(\\langle\\mW \\vu^{(d)}, \\vu^{(d)}\\rangle\\geq \\lambda_d(\\mW)+\\epsilon(\\lambda_1(\\mW)-\\lambda_d(\\mW))) \\leq 1.648\\sqrt{d}e^{-\\sqrt{\\epsilon}(2k-1)},\n  \\end{align*}\n  As a corollary, to ensure that, with probability at least $1-q$, $ \\langle\\mW \\vu^{(1)}, \\vu^{(1)}\\rangle> \\lambda_1(\\mW)-\\epsilon(\\lambda_1(\\mW)-\\lambda_d(\\mW))$ and $\\langle\\mW \\vu^{(d)}, \\vu^{(d)}\\rangle< \\lambda_n(\\mW)+\\epsilon(\\lambda_1(\\mW)-\\lambda_d(\\mW))$,\n  %\n  %\n  %\n  %\n  %\n  %\n  the number of iterations can be bounded by  $\\lceil\\frac{1}{4}\\epsilon^{-1/2}\\log(\\frac{11d}{q^2})+\\frac{1}{2} \\rceil$.\n\\end{proposition}","sha256_utf8":"ab545e3db076077295efb9a9df4d463e88c3a06eacd8250a48c02bf57eb0a25b"}]},{"id":"S064","semantic_type":"lemma","kind":"lemma","title":"极端特征值预言机正确性","source_units":[{"start":146984,"end":147272,"start_line":2544,"end_line":2550,"raw_tex":"\\begin{lemma}\\label{lem:extevec_bound}\n  Let $\\gamma$ and $\\mS$ be the output of $\\mathsf{ExtEvec}(\\mW;\\delta,q)$ in Subroutine~\\ref{alg:lanczos}. Then with probability at least $1-q$, they satisfy one of the properties given in Definition~\\ref{def:extevec}.  \n  %\n  %\n  %\n  %\n\\end{lemma}","sha256_utf8":"1afecf65dafe57856e9710cda6c9b631309923b54239bd76a117608e81e5182a"}]},{"id":"S065","semantic_type":"proof","kind":"proof","title":"特征值预言机正确性证明","source_units":[{"start":147273,"end":151105,"start_line":2551,"end_line":2614,"raw_tex":"\\begin{proof}\n  Note that in Subroutine~\\ref{alg:lanczos}, we run the Lanczos method for $\\Bigl\\lceil \\frac{1}{4}\\epsilon^{-1/2}\\log\\frac{11d}{q^2}+\\frac{1}{2}\\Bigr\\rceil$ iterations, where $\\epsilon = \\frac{\\delta}{2(1+\\delta)}$. \n  Thus, by Proposition~\\ref{prop:lanczos}, with probability at least $1-q$ we have \n  \\begin{align}\n    \\hat{\\lambda}_1 \\triangleq \\langle\\bar{\\mW} \\vu^{(1)}, \\vu^{(1)}\\rangle \\geq  \\lambda_1(\\bar{\\mW})-\\frac{\\delta}{2(1+\\delta)}(\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})), \\label{eq:lambda_1}\\\\ \n    %\n    \\hat{\\lambda}_d \\triangleq \\langle\\bar{\\mW} \\vu^{(d)}, \\vu^{(d)}\\rangle \\leq  \\lambda_d(\\bar{\\mW})+\\frac{\\delta}{2(1+\\delta)}(\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})). \\label{eq:lambda_d}\n  \\end{align}\n  %\n  %\n  Combining \\eqref{eq:lambda_1} and \\eqref{eq:lambda_d}, we obtain $\\frac{\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})}{1+\\delta}\\leq \\hat{\\lambda}_1-\\hat{\\lambda}_d$, which further implies that $\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})\\leq (1+\\delta)(\\hat{\\lambda}_1-\\hat{\\lambda}_d)$. \n    %\n    %\n    %\n  By plugging the above inequality back into \\eqref{eq:lambda_1} and \\eqref{eq:lambda_d}, we further have \n  \\begin{align}\n    \\lambda_1(\\bar{\\mW}) \\leq \\hat{\\lambda}_1+\\frac{\\delta}{2(1+\\delta)}(\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})) \\leq \\hat{\\lambda}_1+\\frac{\\delta}{2}(\\hat{\\lambda}_1-\\hat{\\lambda}_d), \\label{eq:upper_bound_lambda_1} \\\\\n    \\lambda_d(\\bar{\\mW}) \\geq \\hat{\\lambda}_d-\\frac{\\delta}{2(1+\\delta)}(\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})) \\geq \\hat{\\lambda}_d-\\frac{\\delta}{2}(\\hat{\\lambda}_1-\\hat{\\lambda}_d). \\label{eq:lower_bound_lambda_d}\n  \\end{align}\n  Recall that $\\gamma = \\max\\{\\hat{\\lambda}_1,-\\hat{\\lambda}_d\\}$. By \\eqref{eq:upper_bound_lambda_1} and \\eqref{eq:lower_bound_lambda_d}, \n  we can further bound the eigenvalues of $\\bar{\\mW}$ by  \n  \\begin{equation}\n    \\lambda_1(\\bar{\\mW}) \\leq \\gamma +\\frac{\\delta}{2}\\cdot 2\\gamma ={(1+\\delta)\\gamma} \\quad \\text{and} \\quad \\lambda_d(\\bar{\\mW}) \\geq -\\gamma -\\frac{\\delta}{2}\\cdot 2\\gamma = -(1+\\delta){\\gamma}.\n  \\end{equation}\n  Hence, we can see that $-(1+\\delta)\\gamma \\mI\\preceq \\bar{\\mW} = \\frac{1}{2}(\\mW + \\mW^\\top) \\preceq (1+\\delta)\\gamma \\mI$. \n  %\n  Now we distinguish three cases. \n  \\begin{enumerate}[(a)]\n    \\item If $\\gamma \\leq 1$, then we are in \\textbf{Case I} and the $\\mathsf{ExtEvec}$ oracle outputs $\\gamma$ and $\\mS = 0$. In this case, we indeed have $-(1+\\delta) \\mI\\preceq \\frac{1}{2}(\\mW + \\mW^\\top) \\preceq (1+\\delta) \\mI$. \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    \\item If $\\gamma >1$ and $\\hat{\\lambda}_1 \\geq -\\hat{\\lambda}_d$, then we are in \\textbf{Case II} and the $\\mathsf{ExtEvec}$ oracle returns $\\gamma $ and $\\mS = \\vu^{(1)}(\\vu^{(1)})^\\top$. In this case, since $\\max\\{\\lambda_1(\\bar{\\mW}),-\\lambda_d(\\bar{\\mW})\\} \\leq \\gamma(1+\\delta)$, we have $-(1+\\delta) \\mI\\preceq \\frac{1}{2\\gamma}(\\mW + \\mW^\\top) \\preceq (1+\\delta) \\mI$. Also, since $\\vu^{(1)}$ is a unit vector, we have $\\|\\mS\\|_F = \\|\\vu^{(1)}\\|^2 = 1$. Finally, for any $\\hat{\\mB}$ such that $-\\mI \\preceq \\frac{1}{2}(\\hat{\\mB}+\\hat{\\mB}^\\top) \\preceq \\mI$, \n    %\n    we have \n    %\n    $\\langle \\mS,\\mW-\\hat{\\mB}\\rangle = \\vu_1^\\top{\\mW} \\vu_1-\\vu_1^\\top \\hat{\\mB} \\vu_1 = \\vu_1^\\top{\\bar{\\mW}} \\vu_1 -\\frac{1}{2}\\vu_1^\\top (\\hat{\\mB}+\\hat{\\mB}^\\top) \\vu_1 \\geq \\hat{\\lambda}_1-1 = \\gamma-1$. Thus, $\\gamma$ and $\\mS$ satisfy all the properties in \\textbf{Case II}. \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    \\item If $\\gamma >1$ and $-\\hat{\\lambda}_d \\geq \\hat{\\lambda}_1$, then we are also in \\textbf{Case II} and the $\\mathsf{ExtEvec}$ oracle returns $\\gamma$ and $\\mS = -\\vu^{(d)}(\\vu^{(d)})^\\top$. The rest follows similarly to the case above.\n    %\n    %\n    %\n    %\n    %\n  \\end{enumerate}\n  This completes the proof.\n\\end{proof}","sha256_utf8":"3b6c01e2a7f0c56f18271af17f2b5f0698234a69667e1d4d0abf3d4c46867769"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"应用Lanczos的概率误差界","source_units":[{"start":147273,"end":148016,"start_line":2551,"end_line":2558,"raw_tex":"\\begin{proof}\n  Note that in Subroutine~\\ref{alg:lanczos}, we run the Lanczos method for $\\Bigl\\lceil \\frac{1}{4}\\epsilon^{-1/2}\\log\\frac{11d}{q^2}+\\frac{1}{2}\\Bigr\\rceil$ iterations, where $\\epsilon = \\frac{\\delta}{2(1+\\delta)}$. \n  Thus, by Proposition~\\ref{prop:lanczos}, with probability at least $1-q$ we have \n  \\begin{align}\n    \\hat{\\lambda}_1 \\triangleq \\langle\\bar{\\mW} \\vu^{(1)}, \\vu^{(1)}\\rangle \\geq  \\lambda_1(\\bar{\\mW})-\\frac{\\delta}{2(1+\\delta)}(\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})), \\label{eq:lambda_1}\\\\ \n    %\n    \\hat{\\lambda}_d \\triangleq \\langle\\bar{\\mW} \\vu^{(d)}, \\vu^{(d)}\\rangle \\leq  \\lambda_d(\\bar{\\mW})+\\frac{\\delta}{2(1+\\delta)}(\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})). \\label{eq:lambda_d}\n  \\end{align}\n","sha256_utf8":"4d2704b2a2094a93e2b938c8b0d9a021d7765af36561d823f5a05a6629ff3ce4"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"由近似特征值控制真实谱区间","source_units":[{"start":148016,"end":149469,"start_line":2559,"end_line":2575,"raw_tex":"  %\n  %\n  Combining \\eqref{eq:lambda_1} and \\eqref{eq:lambda_d}, we obtain $\\frac{\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})}{1+\\delta}\\leq \\hat{\\lambda}_1-\\hat{\\lambda}_d$, which further implies that $\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})\\leq (1+\\delta)(\\hat{\\lambda}_1-\\hat{\\lambda}_d)$. \n    %\n    %\n    %\n  By plugging the above inequality back into \\eqref{eq:lambda_1} and \\eqref{eq:lambda_d}, we further have \n  \\begin{align}\n    \\lambda_1(\\bar{\\mW}) \\leq \\hat{\\lambda}_1+\\frac{\\delta}{2(1+\\delta)}(\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})) \\leq \\hat{\\lambda}_1+\\frac{\\delta}{2}(\\hat{\\lambda}_1-\\hat{\\lambda}_d), \\label{eq:upper_bound_lambda_1} \\\\\n    \\lambda_d(\\bar{\\mW}) \\geq \\hat{\\lambda}_d-\\frac{\\delta}{2(1+\\delta)}(\\lambda_1(\\bar{\\mW})-\\lambda_d(\\bar{\\mW})) \\geq \\hat{\\lambda}_d-\\frac{\\delta}{2}(\\hat{\\lambda}_1-\\hat{\\lambda}_d). \\label{eq:lower_bound_lambda_d}\n  \\end{align}\n  Recall that $\\gamma = \\max\\{\\hat{\\lambda}_1,-\\hat{\\lambda}_d\\}$. By \\eqref{eq:upper_bound_lambda_1} and \\eqref{eq:lower_bound_lambda_d}, \n  we can further bound the eigenvalues of $\\bar{\\mW}$ by  \n  \\begin{equation}\n    \\lambda_1(\\bar{\\mW}) \\leq \\gamma +\\frac{\\delta}{2}\\cdot 2\\gamma ={(1+\\delta)\\gamma} \\quad \\text{and} \\quad \\lambda_d(\\bar{\\mW}) \\geq -\\gamma -\\frac{\\delta}{2}\\cdot 2\\gamma = -(1+\\delta){\\gamma}.\n  \\end{equation}\n  Hence, we can see that $-(1+\\delta)\\gamma \\mI\\preceq \\bar{\\mW} = \\frac{1}{2}(\\mW + \\mW^\\top) \\preceq (1+\\delta)\\gamma \\mI$. \n","sha256_utf8":"2a207cb545a91ab39726a973f2ff3ec0ded0a7d1b35e392d8ca325d084c15bfd"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"验证近可行输出","source_units":[{"start":149473,"end":149839,"start_line":2577,"end_line":2591,"raw_tex":"  Now we distinguish three cases. \n  \\begin{enumerate}[(a)]\n    \\item If $\\gamma \\leq 1$, then we are in \\textbf{Case I} and the $\\mathsf{ExtEvec}$ oracle outputs $\\gamma$ and $\\mS = 0$. In this case, we indeed have $-(1+\\delta) \\mI\\preceq \\frac{1}{2}(\\mW + \\mW^\\top) \\preceq (1+\\delta) \\mI$. \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n","sha256_utf8":"eeba4c0ca9df7e8bc608f85ca2457fee2b7c43ac9c284724eeb3f9963de5620c"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"验证最大特征向量分离证书","source_units":[{"start":149839,"end":150774,"start_line":2592,"end_line":2605,"raw_tex":"    \\item If $\\gamma >1$ and $\\hat{\\lambda}_1 \\geq -\\hat{\\lambda}_d$, then we are in \\textbf{Case II} and the $\\mathsf{ExtEvec}$ oracle returns $\\gamma $ and $\\mS = \\vu^{(1)}(\\vu^{(1)})^\\top$. In this case, since $\\max\\{\\lambda_1(\\bar{\\mW}),-\\lambda_d(\\bar{\\mW})\\} \\leq \\gamma(1+\\delta)$, we have $-(1+\\delta) \\mI\\preceq \\frac{1}{2\\gamma}(\\mW + \\mW^\\top) \\preceq (1+\\delta) \\mI$. Also, since $\\vu^{(1)}$ is a unit vector, we have $\\|\\mS\\|_F = \\|\\vu^{(1)}\\|^2 = 1$. Finally, for any $\\hat{\\mB}$ such that $-\\mI \\preceq \\frac{1}{2}(\\hat{\\mB}+\\hat{\\mB}^\\top) \\preceq \\mI$, \n    %\n    we have \n    %\n    $\\langle \\mS,\\mW-\\hat{\\mB}\\rangle = \\vu_1^\\top{\\mW} \\vu_1-\\vu_1^\\top \\hat{\\mB} \\vu_1 = \\vu_1^\\top{\\bar{\\mW}} \\vu_1 -\\frac{1}{2}\\vu_1^\\top (\\hat{\\mB}+\\hat{\\mB}^\\top) \\vu_1 \\geq \\hat{\\lambda}_1-1 = \\gamma-1$. Thus, $\\gamma$ and $\\mS$ satisfy all the properties in \\textbf{Case II}. \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n","sha256_utf8":"0574abdb753c96e67352cc288dfd75725e628c6547c43723bf2d3d80d88bcc77"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"验证最小特征向量分离证书","source_units":[{"start":150774,"end":151105,"start_line":2606,"end_line":2614,"raw_tex":"    \\item If $\\gamma >1$ and $-\\hat{\\lambda}_d \\geq \\hat{\\lambda}_1$, then we are also in \\textbf{Case II} and the $\\mathsf{ExtEvec}$ oracle returns $\\gamma$ and $\\mS = -\\vu^{(d)}(\\vu^{(d)})^\\top$. The rest follows similarly to the case above.\n    %\n    %\n    %\n    %\n    %\n  \\end{enumerate}\n  This completes the proof.\n\\end{proof}","sha256_utf8":"195157b2c3456f53e066a5398a9de76c0ecd601d70f0f90f27321e1d155540e4"}]},{"id":"S071","semantic_type":"lemma","kind":"lemma","title":"奇异值与对称增广谱","source_units":[{"start":151879,"end":152856,"start_line":2618,"end_line":2631,"raw_tex":"\\begin{lemma}[{\\cite[Section 8.6.1]{Golub2013Matrix}}]\\label{lem:singular_to_eigen}\n  Let $\\mW\\in \\reals^{d\\times d}$ be an arbitrary matrix. Define the augmented matrix $\\tilde{\\mW}$ by \n  \\begin{equation}\\label{eq:augmented}\n    \\tilde{\\mW} = \\begin{bmatrix} 0 & \\mW \\\\ \\mW^\\top & 0\n    \\end{bmatrix} \\in \\mathbb{S}^{2d}. \n  \\end{equation}\n  %\n  \\begin{itemize}\n    \\item Let $\\lambda_1(\\tilde{\\mW})$ and $\\lambda_{2d}(\\tilde{\\mW})$ denote its largest and smallest eigenvalues, respectively. Then $\\lambda_1(\\tilde{\\mW})$ is the maximal singular value of $\\mW$ and $\\lambda_{2d}(\\tilde{\\mW}) = -\\lambda_1(\\tilde{\\mW})$. \n    \\item Let $\\tilde{\\vv} \\in \\reals^{2d}$ denote the eigenvector associated with $\\lambda_1(\\tilde{\\mW})$ and partition it as $\\tilde{\\vv} = (\\vv,\\vv')$, where $\\vv,\\vv' \\in \\reals^d$. Then ${\\vv}$ and ${\\vv'}$ are the left and right singular vectors associated with the maximal singular value of $\\mW$, respectively.\n  \\end{itemize}\n  \n  %\n\\end{lemma}","sha256_utf8":"7df775f8601729cc58f5259777ad9ce0e1b1aeec1470e34e43948dd4c35984e8"}]},{"id":"S072","semantic_type":"lemma","kind":"lemma","title":"最大奇异值预言机正确性","source_units":[{"start":154612,"end":154905,"start_line":2673,"end_line":2681,"raw_tex":"\\begin{lemma}\\label{lem:maxsvec_bound}\n  Let $\\gamma$ and $\\mS$ be the output of $\\mathsf{MaxSvec}(\\mW;\\delta,q)$ in Subroutine~\\ref{alg:lanczos_singular}.  %\n  %\n  Then with probability at least $1-q$, they satisfy the properties in Definition~\\ref{def:MaxSvec}.  \n  %\n  %\n  %\n  %\n\\end{lemma}","sha256_utf8":"ef5366a927b51e144b30d9cbf9cfcf00df054415f7fa8ebe95edf1a5280b7525"}]},{"id":"S073","semantic_type":"proof","kind":"proof","title":"奇异值预言机正确性证明","source_units":[{"start":154906,"end":157584,"start_line":2682,"end_line":2747,"raw_tex":"\\begin{proof}\n  %\n  %\n  %\n  %\n  %\n  Note that in Subroutine~\\ref{alg:lanczos_singular}, we run the Lanczos method for $\\Bigl\\lceil \\frac{1}{4}\\epsilon^{-1/2}\\log\\frac{22d}{q^2}+\\frac{1}{2}\\Bigr\\rceil$ iterations, where $\\epsilon = \\frac{\\delta}{2(1+\\delta)}$. \n  Thus, by Proposition~\\ref{prop:lanczos}, with probability at least $1-q$ we can find a unit vector $\\tilde{\\vv} \\in \\reals^{2d}$ such that \n  %\n  $\\tilde{\\lambda}_1 \\triangleq \\langle \\tilde{\\mW} \\tilde{\\vv}, \\tilde{\\vv}\\rangle \\geq  \\lambda_1(\\tilde{\\mW})-\\frac{\\delta}{2(1+\\delta)}(\\lambda_1(\\tilde{\\mW})-\\lambda_{2d}(\\tilde{\\mW}))$, %\n    %\n    %\n  %\n  %\n  %\n  Moreover, notice that $\\lambda_{2d}(\\tilde{\\mW}) = -\\lambda_1(\\tilde{\\mW})$ by Lemma~\\ref{lem:singular_to_eigen}. Hence, the above further implies that $\\tilde{\\lambda}_1 \\geq \\frac{1}{1+\\delta} \\lambda_1(\\tilde{\\mW})$, which is equivalent to $\\lambda_1(\\tilde{\\mW}) \\leq (1+\\delta)\\tilde{\\lambda}$. \n  %\n  %\n  %\n  Moreover, note that $\\lambda_1(\\tilde{\\mW}) = \\|\\mW\\|_{\\op}$ and $\\gamma = \\tilde{\\lambda}_1/3$. Hence, the above is further equivalent to $\\|\\mW\\|_{\\op} \\leq 3(1+\\delta)\\gamma$. \n  %\n  %\n  %\n  %\n  %\n  %\n  Now we distinguish two cases. \n  \\begin{enumerate}[(a)]\n    \\item If $\\gamma \\leq 1$, then we are in \\textbf{Case I} and the $\\mathsf{MaxSvec}$ oracle outputs $\\gamma$ and $\\mS = 0$. In this case, we indeed have $\\|\\mW\\|_\\op \\leq  3(1+\\delta)\\gamma \\leq 3(1+\\delta)$. \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    \\item If $\\gamma >1$, then we are in \\textbf{Case II} and the $\\mathsf{MaxSvec}$ oracle returns $\\gamma $ and $\\mS = \\frac{2}{3}\\vv'\\vv^\\top$. \n    In this case, since $\\|\\mW\\|_\\op \\leq 3(1+\\delta)\\gamma$, we have $\\|\\mW/\\gamma\\|_{\\op} \\leq 3(1+\\delta)$. Also, since $\\tilde{\\vv}$ is a unit vector, we have $\\|\\mS\\|_F = \\frac{2}{3} \\|\\vv\\|\\|\\vv'\\| \\leq \\frac{1}{3}(\\|\\vv\\|^2 + \\|\\vv'\\|^2) = \\frac{1}{3}\\|\\tilde{\\vv}\\|^2 =\\frac{1}{3}$. Finally, it holds that $\\langle \\mS, \\mW \\rangle = \\frac{2}{3} \\vv^\\top \\mW \\vv' = \\frac{1}{3}\\tilde{\\vv}^\\top \\tilde{\\mW} \\tilde{\\vv} = \\frac{\\tilde{\\lambda}_1}{3} = \\gamma$. Moreover, for any $\\hat{\\mB}$ such that $\\|\\hat{\\mB}\\|_\\op \\leq 3$, we have\n    %\n    $\\langle \\mS,\\hat{\\mB}\\rangle =  \\frac{2}{3}\\vv^\\top \\hat{\\mB} \\vv' \\leq \\frac{2}{3}\\|\\vv\\| \\|\\mB\\|_{\\op} \\|\\vv'\\| \\leq 2\\|\\vv'\\| \\|\\vv\\| \\leq \\|\\vv\\|^2 + \\|\\vv'\\|^2 = 1$. \n    %\n    Hence, we obtain $\\langle \\mS,\\mW-\\hat{\\mB}\\rangle \\geq \\gamma -1$. This shows that $\\gamma$ and $\\mS$ satisfy all the properties in \\textbf{Case II}. \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n  \\end{enumerate}\n  This completes the proof.\n\\end{proof}","sha256_utf8":"3974313989a3db634e6ef7a63a3c3ed20a39b7690867166b0a11ec578a4c4940"}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","title":"增广谱误差给出算子范数界","source_units":[{"start":154906,"end":156052,"start_line":2682,"end_line":2707,"raw_tex":"\\begin{proof}\n  %\n  %\n  %\n  %\n  %\n  Note that in Subroutine~\\ref{alg:lanczos_singular}, we run the Lanczos method for $\\Bigl\\lceil \\frac{1}{4}\\epsilon^{-1/2}\\log\\frac{22d}{q^2}+\\frac{1}{2}\\Bigr\\rceil$ iterations, where $\\epsilon = \\frac{\\delta}{2(1+\\delta)}$. \n  Thus, by Proposition~\\ref{prop:lanczos}, with probability at least $1-q$ we can find a unit vector $\\tilde{\\vv} \\in \\reals^{2d}$ such that \n  %\n  $\\tilde{\\lambda}_1 \\triangleq \\langle \\tilde{\\mW} \\tilde{\\vv}, \\tilde{\\vv}\\rangle \\geq  \\lambda_1(\\tilde{\\mW})-\\frac{\\delta}{2(1+\\delta)}(\\lambda_1(\\tilde{\\mW})-\\lambda_{2d}(\\tilde{\\mW}))$, %\n    %\n    %\n  %\n  %\n  %\n  Moreover, notice that $\\lambda_{2d}(\\tilde{\\mW}) = -\\lambda_1(\\tilde{\\mW})$ by Lemma~\\ref{lem:singular_to_eigen}. Hence, the above further implies that $\\tilde{\\lambda}_1 \\geq \\frac{1}{1+\\delta} \\lambda_1(\\tilde{\\mW})$, which is equivalent to $\\lambda_1(\\tilde{\\mW}) \\leq (1+\\delta)\\tilde{\\lambda}$. \n  %\n  %\n  %\n  Moreover, note that $\\lambda_1(\\tilde{\\mW}) = \\|\\mW\\|_{\\op}$ and $\\gamma = \\tilde{\\lambda}_1/3$. Hence, the above is further equivalent to $\\|\\mW\\|_{\\op} \\leq 3(1+\\delta)\\gamma$. \n  %\n  %\n  %\n  %\n  %\n  %\n","sha256_utf8":"770b18d5f3062f479d5cd952d764773d5f30353b08e2829d7450aefa05c3bc7a"}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","title":"验证奇异值近可行证书","source_units":[{"start":156052,"end":156389,"start_line":2708,"end_line":2721,"raw_tex":"  Now we distinguish two cases. \n  \\begin{enumerate}[(a)]\n    \\item If $\\gamma \\leq 1$, then we are in \\textbf{Case I} and the $\\mathsf{MaxSvec}$ oracle outputs $\\gamma$ and $\\mS = 0$. In this case, we indeed have $\\|\\mW\\|_\\op \\leq  3(1+\\delta)\\gamma \\leq 3(1+\\delta)$. \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n","sha256_utf8":"6dd7d9e00bb7849e94d51fb40b8190a32ed9a00d2e5f5fe40cf280c2f17da117"}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","title":"验证秩一分离矩阵及其范数","source_units":[{"start":156389,"end":157584,"start_line":2722,"end_line":2747,"raw_tex":"    \\item If $\\gamma >1$, then we are in \\textbf{Case II} and the $\\mathsf{MaxSvec}$ oracle returns $\\gamma $ and $\\mS = \\frac{2}{3}\\vv'\\vv^\\top$. \n    In this case, since $\\|\\mW\\|_\\op \\leq 3(1+\\delta)\\gamma$, we have $\\|\\mW/\\gamma\\|_{\\op} \\leq 3(1+\\delta)$. Also, since $\\tilde{\\vv}$ is a unit vector, we have $\\|\\mS\\|_F = \\frac{2}{3} \\|\\vv\\|\\|\\vv'\\| \\leq \\frac{1}{3}(\\|\\vv\\|^2 + \\|\\vv'\\|^2) = \\frac{1}{3}\\|\\tilde{\\vv}\\|^2 =\\frac{1}{3}$. Finally, it holds that $\\langle \\mS, \\mW \\rangle = \\frac{2}{3} \\vv^\\top \\mW \\vv' = \\frac{1}{3}\\tilde{\\vv}^\\top \\tilde{\\mW} \\tilde{\\vv} = \\frac{\\tilde{\\lambda}_1}{3} = \\gamma$. Moreover, for any $\\hat{\\mB}$ such that $\\|\\hat{\\mB}\\|_\\op \\leq 3$, we have\n    %\n    $\\langle \\mS,\\hat{\\mB}\\rangle =  \\frac{2}{3}\\vv^\\top \\hat{\\mB} \\vv' \\leq \\frac{2}{3}\\|\\vv\\| \\|\\mB\\|_{\\op} \\|\\vv'\\| \\leq 2\\|\\vv'\\| \\|\\vv\\| \\leq \\|\\vv\\|^2 + \\|\\vv'\\|^2 = 1$. \n    %\n    Hence, we obtain $\\langle \\mS,\\mW-\\hat{\\mB}\\rangle \\geq \\gamma -1$. This shows that $\\gamma$ and $\\mS$ satisfy all the properties in \\textbf{Case II}. \n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n    %\n  \\end{enumerate}\n  This completes the proof.\n\\end{proof}","sha256_utf8":"178fcf73edf95fbd16bc076ac7b76062b5fe426e7d435affb082ac9462845d58"}]},{"id":"S077","semantic_type":"corollary","kind":"corollary","title":"雅可比近似的一致谱界","source_units":[{"start":163096,"end":163336,"start_line":2852,"end_line":2854,"raw_tex":"\\begin{corollary}\\label{lem:feasible_Bk}\n  Let $\\delta_t = \\frac{\\mu}{2L_1}$ for all $t\\geq 0$. Then for any $k \\geq 0$, we have $\\frac{1}{2}(\\mB_k + \\mB_k^\\top )\\succeq \\frac{\\mu}{2} \\mI$ and $\\|\\mB_k\\|_{\\op} \\leq 6.5L_1$. \n\\end{corollary}","sha256_utf8":"63bc9ef96d9f21ed53923402c58c8fe39359683ef6f5358dd731b9c6827d850a"}]},{"id":"S078","semantic_type":"proof","kind":"proof","title":"一致谱界的正文推导","source_units":[{"start":162703,"end":163090,"start_line":2847,"end_line":2847,"raw_tex":"We first prove the linear convergence rate in (a). To begin with, note that Subroutine~\\ref{alg:jacobian_approx} with \\textbf{Option I} guarantees that  $\\hat{\\mB}_k \\in (1+\\frac{\\mu}{2L_1}) \\mathcal{C}$ by Lemma~\\ref{lem:regret_reduction}.  As a corollary of Lemma~\\ref{lem:transform}, we have the following result showing the boundedness of the Jacobian approximation matrix $\\mB_k$. \n","sha256_utf8":"4dd5603a0aa0d87a23f603a83e05c341ff81408e626dd54bc134f75ba126861c"}]},{"id":"S079","semantic_type":"proof_step","kind":"proof_step","title":"近可行性经反变换产生谱界","source_units":[{"start":162703,"end":163090,"start_line":2847,"end_line":2847,"raw_tex":"We first prove the linear convergence rate in (a). To begin with, note that Subroutine~\\ref{alg:jacobian_approx} with \\textbf{Option I} guarantees that  $\\hat{\\mB}_k \\in (1+\\frac{\\mu}{2L_1}) \\mathcal{C}$ by Lemma~\\ref{lem:regret_reduction}.  As a corollary of Lemma~\\ref{lem:transform}, we have the following result showing the boundedness of the Jacobian approximation matrix $\\mB_k$. \n","sha256_utf8":"4dd5603a0aa0d87a23f603a83e05c341ff81408e626dd54bc134f75ba126861c"}]},{"id":"S080","semantic_type":"lemma","kind":"lemma","title":"所有迭代的一致步长下界","source_units":[{"start":163461,"end":163677,"start_line":2856,"end_line":2859,"raw_tex":"\\begin{lemma}\\label{lem:stepsize_const_bound}\n  %\n  In Subroutine~\\ref{alg:ls}, choose $\\sigma_0 \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$. For any $k\\geq 0$, we have $\\eta_k \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$. \n\\end{lemma}","sha256_utf8":"e71844e4fb2a8435011cd77c9a5ad2b067bbf98dc21343580b49e996ca8aa91b"}]},{"id":"S081","semantic_type":"proof","kind":"proof","title":"一致步长下界证明","source_units":[{"start":163678,"end":165678,"start_line":2860,"end_line":2883,"raw_tex":"\\begin{proof}\n  Recall that $\\calB = \\{k: \\eta_k < \\sigma_k\\}$.\n  We first establish that $\\eta_k \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$ for $k\\in \\mathcal{B}$. To see this, suppose $k\\in \\mathcal{B}$ and recall from Lemma~\\ref{lem:step size_lb} that\n\\begin{equation}\\label{eq:step size_lb_appen}\n  \\eta_k > \\frac{\\alpha_2 \\beta\\|{\\tilde{\\vz}_k}-\\vz_k\\|}{\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\|}.\n\\end{equation}\n%\n  %\n  %\n  %\n  Moreover, by using the triangle inequality, we have \n  %\n  $\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\| \\leq \\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)\\| + \\|\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\|$.\n  %\n  By using Assumption~\\ref{assum:operator_lips}, we have $\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)\\| \\leq L_1\\|\\tilde{\\vz}_k-\\vz_k\\|$. Moreover, since $\\|\\mB_k\\|_{\\op} \\leq 6.5L_1$ by Corollary~\\ref{lem:feasible_Bk}, we also get $\\|\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\| \\leq 6.5 \\|{\\tilde{\\vz}_k}-\\vz_k\\|$. \n  %\n  Thus, this implies that $\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\| \\leq 7.5\\|{\\tilde{\\vz}_k}-\\vz_k\\|$,  \n  %\n  %\n  %\n  which leads to $\\eta_k > \\frac{\\alpha_2\\beta}{7.5L_1}$ from \\eqref{eq:step size_lb_appen}. \n\n  Now we can prove that $\\eta_k \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$ for all $k\\geq 0$ by induction. To show that this holds for $k=0$, we distinguish two cases. If $0\\notin \\mathcal{B}$, then we have $\\eta_0 = \\sigma_0 > \\frac{\\alpha_2\\beta}{7.5L_1}$ by our choice of $\\sigma_0$. Otherwise, if $0\\in \\mathcal{B}$, then it directly follows from our result in the previous paragraph. Moreover, assume that $\\eta_{l-1} \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$ where $l\\geq 1$. Similarly, we again distinguish two cases: if $l\\notin \\mathcal{B}$, then we have $\\eta_l = \\sigma_l = \\frac{\\eta_{l-1}}{\\beta} >\\frac{\\alpha_2}{7.5L_1}>\\frac{\\alpha_2\\beta}{7.5L_1}$; otherwise, if $l\\in \\mathcal{B}$, it follows from the result above that $\\eta_l \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$. This completes the induction. \n\\end{proof}","sha256_utf8":"8c76e7580418f5cacd5a923c8b83fac9ca4d2b2f6d72af1faa9cb1a1768699fb"}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","title":"谱范数界控制回溯误差","source_units":[{"start":163678,"end":164868,"start_line":2860,"end_line":2880,"raw_tex":"\\begin{proof}\n  Recall that $\\calB = \\{k: \\eta_k < \\sigma_k\\}$.\n  We first establish that $\\eta_k \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$ for $k\\in \\mathcal{B}$. To see this, suppose $k\\in \\mathcal{B}$ and recall from Lemma~\\ref{lem:step size_lb} that\n\\begin{equation}\\label{eq:step size_lb_appen}\n  \\eta_k > \\frac{\\alpha_2 \\beta\\|{\\tilde{\\vz}_k}-\\vz_k\\|}{\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\|}.\n\\end{equation}\n%\n  %\n  %\n  %\n  Moreover, by using the triangle inequality, we have \n  %\n  $\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\| \\leq \\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)\\| + \\|\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\|$.\n  %\n  By using Assumption~\\ref{assum:operator_lips}, we have $\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)\\| \\leq L_1\\|\\tilde{\\vz}_k-\\vz_k\\|$. Moreover, since $\\|\\mB_k\\|_{\\op} \\leq 6.5L_1$ by Corollary~\\ref{lem:feasible_Bk}, we also get $\\|\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\| \\leq 6.5 \\|{\\tilde{\\vz}_k}-\\vz_k\\|$. \n  %\n  Thus, this implies that $\\|\\vF({\\tilde{\\vz}_k})-\\vF(\\vz_k)-\\mB_k({\\tilde{\\vz}_k}-\\vz_k)\\| \\leq 7.5\\|{\\tilde{\\vz}_k}-\\vz_k\\|$,  \n  %\n  %\n  %\n  which leads to $\\eta_k > \\frac{\\alpha_2\\beta}{7.5L_1}$ from \\eqref{eq:step size_lb_appen}. \n","sha256_utf8":"206bb7a46a1e90f04af3ec73b7adb5b4715b0f998ab609c91f92654ba86eebb3"}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","title":"对热启动步长作归纳","source_units":[{"start":164868,"end":165679,"start_line":2881,"end_line":2883,"raw_tex":"\n  Now we can prove that $\\eta_k \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$ for all $k\\geq 0$ by induction. To show that this holds for $k=0$, we distinguish two cases. If $0\\notin \\mathcal{B}$, then we have $\\eta_0 = \\sigma_0 > \\frac{\\alpha_2\\beta}{7.5L_1}$ by our choice of $\\sigma_0$. Otherwise, if $0\\in \\mathcal{B}$, then it directly follows from our result in the previous paragraph. Moreover, assume that $\\eta_{l-1} \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$ where $l\\geq 1$. Similarly, we again distinguish two cases: if $l\\notin \\mathcal{B}$, then we have $\\eta_l = \\sigma_l = \\frac{\\eta_{l-1}}{\\beta} >\\frac{\\alpha_2}{7.5L_1}>\\frac{\\alpha_2\\beta}{7.5L_1}$; otherwise, if $l\\in \\mathcal{B}$, it follows from the result above that $\\eta_l \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$. This completes the induction. \n\\end{proof}\n","sha256_utf8":"a1a6e92dc1765bf675c0598cd14c4fc926a0625cb69bd9aa1397c79384bccce7"}]},{"id":"S084","semantic_type":"lemma","kind":"lemma","title":"平方损失的梯度自界","source_units":[{"start":167292,"end":167907,"start_line":2919,"end_line":2929,"raw_tex":"\\begin{lemma}\\label{lem:loss}\n  Recall the loss function $\\ell_k$ defined in \\eqref{eq:loss_of_jacobian}. For $k\\in \\mathcal{B}$, we have\n  \\begin{equation}\\label{eq:gradient}\n    \\nabla \\ell_k(\\mB) = - \\frac{2(\\vu_k-\\mB\\vs_k)\\vs_k^\\mathsf{T}}{\\|\\vs_k\\|^2}.\n  \\end{equation}\n  Moreover, for any $\\mB\\in \\reals^{d\\times d}$, it holds that \n    \\begin{equation}\\label{eq:gradient_norm_bound}\n        \\|\\nabla \\ell_k(\\mB)\\|_F \\leq \\|\\nabla \\ell_k(\\mB)\\|_* = 2\\sqrt{\\ell_k(\\mB)},\n    \\end{equation} \n    where $\\|\\cdot\\|_F$ and $\\|\\cdot\\|_{*}$ denote the Frobenius norm and the nuclear norm, respectively.  \n\\end{lemma}","sha256_utf8":"3426c0825ce9e3c62fdf7e382c7c87e5658c81314727a1e698c253434f081757"}]},{"id":"S085","semantic_type":"proof","kind":"proof","title":"损失梯度自界证明","source_units":[{"start":167908,"end":168496,"start_line":2930,"end_line":2937,"raw_tex":"\\begin{proof}\n  %\n  The expression in \\eqref{eq:gradient} follows from the direct calculation. The first inequality in \\eqref{eq:gradient_norm_bound} follows from the fact that $\\|\\mA\\|_F \\leq \\|\\mA\\|_*$ for any matrix $\\mA\\in \\reals^{d\\times d}$. For the equality, note that \n  \\begin{align*}\n    \\|\\nabla \\ell_k(\\mB)\\|_*   = \\frac{2}{\\|\\vs_k\\|^2}  \\|\\vu_k-\\mB\\vs_k\\|\\|\\vs_k\\|  = \\frac{2\\|\\vu_k-\\mB\\vs_k\\|}{\\|\\vs_k\\|} = 2\\sqrt{\\ell_k(\\mB)}, \n  \\end{align*}\n  where we used the fact that the rank-one matrix $\\va\\vb^\\top$ has only one nonzero singular value $\\|\\va\\|\\|\\vb\\|$. \n\\end{proof}","sha256_utf8":"68a96c90d82bd9f4e11a0827c0a829eb40aea56b4aee3b872bcc2e37deb68afa"}]},{"id":"S086","semantic_type":"proof_step","kind":"proof_step","title":"求导并计算秩一梯度范数","source_units":[{"start":167908,"end":168497,"start_line":2930,"end_line":2937,"raw_tex":"\\begin{proof}\n  %\n  The expression in \\eqref{eq:gradient} follows from the direct calculation. The first inequality in \\eqref{eq:gradient_norm_bound} follows from the fact that $\\|\\mA\\|_F \\leq \\|\\mA\\|_*$ for any matrix $\\mA\\in \\reals^{d\\times d}$. For the equality, note that \n  \\begin{align*}\n    \\|\\nabla \\ell_k(\\mB)\\|_*   = \\frac{2}{\\|\\vs_k\\|^2}  \\|\\vu_k-\\mB\\vs_k\\|\\|\\vs_k\\|  = \\frac{2\\|\\vu_k-\\mB\\vs_k\\|}{\\|\\vs_k\\|} = 2\\sqrt{\\ell_k(\\mB)}, \n  \\end{align*}\n  where we used the fact that the rank-one matrix $\\va\\vb^\\top$ has only one nonzero singular value $\\|\\va\\|\\|\\vb\\|$. \n\\end{proof}\n","sha256_utf8":"b10f269e0a5ea848167b477fca026fbff070b15d5bbb8a4c6d327576fe66c884"}]},{"id":"S087","semantic_type":"lemma","kind":"lemma","title":"固定比较矩阵的小损失界","source_units":[{"start":168778,"end":169121,"start_line":2943,"end_line":2952,"raw_tex":"\\begin{lemma}\n  \\label{lem:small_loss}\n  %\n  %\n  In Subroutine~\\ref{alg:jacobian_approx}, choose \\textbf{Option I} with $\\rho = \\frac{1}{121}$ and $\\delta = \\frac{\\mu}{2L_1}$. \n  For any ${\\mH}\\in \\mathcal{{Z}}$, we have\n  %\n    $\\sum_{t=0}^{T-1} \\ell_t({\\mB}_t)  \\leq 121\\|\\mB_0-{\\mH}\\|_F^2+ 2 \\sum_{t=0}^{T-1} \\ell_t({\\mH})$.\n  %\n\\end{lemma}","sha256_utf8":"7b4e6d67fa430f582742fe7a4245813a09ac943878a042e2b116e3b60ebfa7ea"}]},{"id":"S088","semantic_type":"proof","kind":"proof","title":"小损失引理证明","source_units":[{"start":169123,"end":172419,"start_line":2954,"end_line":3008,"raw_tex":"\\begin{proof}\nRecall that $\\calZ$ is contained in the linear subspace $\\calL$ (see \\eqref{eq:feasible_set}) and $\\mP$ denotes the orthogonal projection matrix associated with $\\calL$.    \nBy letting $\\vx_t = \\hat{\\mB}_t$, $\\vu = \\hat{\\mH} \\triangleq \\frac{1}{L_1}(\\mH-(L_1+\\mu)\\mI)$, \n%\n$\\vg_t = \\mG_t \\triangleq \\frac{1}{L_1}\\mP \\nabla \\ell_t(\\mB_t)$, $\\tilde{\\vg}_t = \\tilde{\\mG}_t$, $\\vw_t = \\mW_t$ in Lemma~\\ref{lem:regret_reduction}, we obtain:  \n\\begin{enumerate}[(i)]\n  \\item $\\hat{\\mB}_t \\in (1+\\delta)\\mathcal{C}$, which means $\\|\\hat{\\mB}_t\\|_{\\op} \\leq 3(1+\\delta)\\leq 4.5$ since $\\delta = \\frac{\\mu}{2L_1}\\leq \\frac{1}{2}$.\n  \\item It holds that \n  \\begin{align}\n    \\langle \\mG_t, \\hat{\\mB}_t-\\hat{\\mH} \\rangle &\\leq \\frac{1}{2\\rho}\\|\\mW_t-\\hat{\\mH}\\|_F^2-\\frac{1}{2\\rho}\\|\\mW_{t+1}-\\hat{\\mH}\\|_F^2+\\frac{\\rho}{2}\\|\\tilde{\\mG}_t\\|_F^2, \\label{eq:linearized_loss_matrix} \\\\%\\\\\n    %\n    \\|\\tilde{\\mG}_t\\|_F &\\leq \\|\\mG_t\\|_F + |\\langle \\mG_t, \\hat{\\mB}_t\\rangle|\\|\\mS_t\\|_F. \\label{eq:surrogate_gradient_bound}\n  \\end{align}\n\\end{enumerate}\n%\n%\n%\n%\n%\n%\n%\n%\n%\nFirst, note that $\\|\\mS_t\\|_F \\leq 1$ by Definition~\\ref{def:extevec} and\n%\n$|\\langle \\mG_t, \\hat{\\mB}_t\\rangle| \\leq \\|\\mG_t\\|_* \\|\\hat{\\mB}_t\\|_{\\op} \\leq 4.5 \\|\\mG_t\\|_*$. %\n%\nTogether with \\eqref{eq:surrogate_gradient_bound}, we get \n\\begin{equation}\\label{eq:surrogate_bound_loss}\n\\|\\tilde{\\mG}_t\\|_F \\leq \\|\\mG_t\\|_F+4.5\\|\\mG_t\\|_* \\leq 5.5\\|\\mG_t\\|_* \\leq \\frac{5.5}{L_1}\\|\\nabla \\ell_t(\\mB_t)\\|_* \\leq \\frac{11}{L_1}\\sqrt{\\ell_t({\\mB}_t)},\n\\end{equation}\nwhere we used $\\mG_t = \\frac{1}{L_1}\\mP \\nabla \\ell_t(\\mB_t)$ and Lemma~\\ref{lem:loss} in the last inequality. \nFurthermore, since $\\ell_t$ is convex and $\\mB_t,\\mH \\in \\calL$, we have \n\\begin{equation}\\label{eq:convexity_subspace}\n    \\ell_t({\\mB}_t) - \\ell_t(\\mH) \\!\\leq\\! \\langle \\nabla \\ell_t(\\mB_t), {\\mB}_t-\\mH \\rangle \\!=\\! \\langle \\nabla \\ell_t(\\mB_t), \\mP\\left({\\mB}_t-\\mH\\right) \\rangle \\!=\\! L_1^2 \\langle \\mG_t, \\hat{\\mB}_t-\\hat{\\mH} \\rangle,\n\\end{equation}\nwhere we used $\\mG_t = \\frac{1}{L_1}\\mP\\nabla \\ell_t(\\mB_t)$, $\\hat{\\mB}_t \\triangleq \\frac{1}{L_1}(\\mB_t-{(L_1+\\mu)}\\mI)$, and $\\hat{\\mH} \\triangleq \\frac{1}{L_1}(\\mH-{(L_1+\\mu)}\\mI)$. \nTherefore, by \\eqref{eq:linearized_loss_matrix} and \\eqref{eq:surrogate_bound_loss} we get \n\\begin{align*}\n\\ell_t({\\mB}_t) - \\ell_t(\\mH) &\\leq \\frac{L_1^2}{2\\rho}\\|\\mW_t-\\hat{\\mH}\\|_F^2-\\frac{L_1^2}{2\\rho}\\|\\mW_{t+1}-\\hat{\\mH}\\|_F^2+\\frac{\\rho L_1^2}{2} \\|\\tilde{\\mG}_t\\|_F^2\\\\\n&\\leq \\frac{L_1^2}{2\\rho}\\|\\mW_t-\\hat{\\mH}\\|_F^2-\\frac{L_1^2}{2\\rho}\\|\\mW_{t+1}-\\hat{\\mH}\\|_F^2+60.5\\rho\\ell_t({\\mB}_t). %\n%\n\\end{align*}\nSince $\\rho = \\frac{1}{121}$, by rearranging and simplifying terms in the above inequality, we obtain\n%\n%\n%\n$\\ell_t({\\mB}_t) \\leq 2\\ell_t({\\mH})+ {121 L_1^2}\\|\\mW_t-\\hat{\\mH}\\|_F^2-{121 L_1^2}\\|\\mW_{t+1}-\\hat{\\mH}\\|_F^2$.\n%\nBy summing the above inequality from $t=0$ to $T-1$, we further have \n\\begin{equation*}\n  \\sum_{t=0}^{T-1} \\ell_t({\\mB}_t)  \\leq {121 L_1^2}\\|\\mW_0-\\hat{\\mH}\\|_F^2+ 2 \\sum_{t=0}^{T-1} \\ell_t({\\mH}) = 121\\|\\mB_0-{\\mH}\\|_F^2+ 2 \\sum_{t=0}^{T-1} \\ell_t({\\mH}),\n\\end{equation*}\nwhere the last equality is due to ${\\mW}_0 \\triangleq \\frac{1}{L_1}(\\mB_0-{(L_1+\\mu)}\\mI)$ and $\\hat{\\mH} \\triangleq \\frac{1}{L_1}(\\mH-{(L_1+\\mu)}\\mI)$. This completes the proof. \n\\end{proof}","sha256_utf8":"5c3a4e31f6b6b3fd279cdbe317982d2f32c3eca409420ccf58bca71c8186d853"}]},{"id":"S089","semantic_type":"proof_step","kind":"proof_step","title":"将遗憾归约用于矩阵变量","source_units":[{"start":169123,"end":170194,"start_line":2954,"end_line":2976,"raw_tex":"\\begin{proof}\nRecall that $\\calZ$ is contained in the linear subspace $\\calL$ (see \\eqref{eq:feasible_set}) and $\\mP$ denotes the orthogonal projection matrix associated with $\\calL$.    \nBy letting $\\vx_t = \\hat{\\mB}_t$, $\\vu = \\hat{\\mH} \\triangleq \\frac{1}{L_1}(\\mH-(L_1+\\mu)\\mI)$, \n%\n$\\vg_t = \\mG_t \\triangleq \\frac{1}{L_1}\\mP \\nabla \\ell_t(\\mB_t)$, $\\tilde{\\vg}_t = \\tilde{\\mG}_t$, $\\vw_t = \\mW_t$ in Lemma~\\ref{lem:regret_reduction}, we obtain:  \n\\begin{enumerate}[(i)]\n  \\item $\\hat{\\mB}_t \\in (1+\\delta)\\mathcal{C}$, which means $\\|\\hat{\\mB}_t\\|_{\\op} \\leq 3(1+\\delta)\\leq 4.5$ since $\\delta = \\frac{\\mu}{2L_1}\\leq \\frac{1}{2}$.\n  \\item It holds that \n  \\begin{align}\n    \\langle \\mG_t, \\hat{\\mB}_t-\\hat{\\mH} \\rangle &\\leq \\frac{1}{2\\rho}\\|\\mW_t-\\hat{\\mH}\\|_F^2-\\frac{1}{2\\rho}\\|\\mW_{t+1}-\\hat{\\mH}\\|_F^2+\\frac{\\rho}{2}\\|\\tilde{\\mG}_t\\|_F^2, \\label{eq:linearized_loss_matrix} \\\\%\\\\\n    %\n    \\|\\tilde{\\mG}_t\\|_F &\\leq \\|\\mG_t\\|_F + |\\langle \\mG_t, \\hat{\\mB}_t\\rangle|\\|\\mS_t\\|_F. \\label{eq:surrogate_gradient_bound}\n  \\end{align}\n\\end{enumerate}\n%\n%\n%\n%\n%\n%\n%\n%\n%\n","sha256_utf8":"f458b6251675a2a333f383b673cfc2269e79cc0c651cec2c5dd86129f42516a8"}]},{"id":"S090","semantic_type":"proof_step","kind":"proof_step","title":"以当前损失控制替代梯度","source_units":[{"start":170194,"end":170769,"start_line":2977,"end_line":2985,"raw_tex":"First, note that $\\|\\mS_t\\|_F \\leq 1$ by Definition~\\ref{def:extevec} and\n%\n$|\\langle \\mG_t, \\hat{\\mB}_t\\rangle| \\leq \\|\\mG_t\\|_* \\|\\hat{\\mB}_t\\|_{\\op} \\leq 4.5 \\|\\mG_t\\|_*$. %\n%\nTogether with \\eqref{eq:surrogate_gradient_bound}, we get \n\\begin{equation}\\label{eq:surrogate_bound_loss}\n\\|\\tilde{\\mG}_t\\|_F \\leq \\|\\mG_t\\|_F+4.5\\|\\mG_t\\|_* \\leq 5.5\\|\\mG_t\\|_* \\leq \\frac{5.5}{L_1}\\|\\nabla \\ell_t(\\mB_t)\\|_* \\leq \\frac{11}{L_1}\\sqrt{\\ell_t({\\mB}_t)},\n\\end{equation}\nwhere we used $\\mG_t = \\frac{1}{L_1}\\mP \\nabla \\ell_t(\\mB_t)$ and Lemma~\\ref{lem:loss} in the last inequality. \n","sha256_utf8":"7142392f75628650d30579b8305af18f42b830b5d5ff5db639c5eef71a7a0d9b"}]},{"id":"S091","semantic_type":"proof_step","kind":"proof_step","title":"凸性连接损失差与势函数","source_units":[{"start":170769,"end":171730,"start_line":2986,"end_line":2996,"raw_tex":"Furthermore, since $\\ell_t$ is convex and $\\mB_t,\\mH \\in \\calL$, we have \n\\begin{equation}\\label{eq:convexity_subspace}\n    \\ell_t({\\mB}_t) - \\ell_t(\\mH) \\!\\leq\\! \\langle \\nabla \\ell_t(\\mB_t), {\\mB}_t-\\mH \\rangle \\!=\\! \\langle \\nabla \\ell_t(\\mB_t), \\mP\\left({\\mB}_t-\\mH\\right) \\rangle \\!=\\! L_1^2 \\langle \\mG_t, \\hat{\\mB}_t-\\hat{\\mH} \\rangle,\n\\end{equation}\nwhere we used $\\mG_t = \\frac{1}{L_1}\\mP\\nabla \\ell_t(\\mB_t)$, $\\hat{\\mB}_t \\triangleq \\frac{1}{L_1}(\\mB_t-{(L_1+\\mu)}\\mI)$, and $\\hat{\\mH} \\triangleq \\frac{1}{L_1}(\\mH-{(L_1+\\mu)}\\mI)$. \nTherefore, by \\eqref{eq:linearized_loss_matrix} and \\eqref{eq:surrogate_bound_loss} we get \n\\begin{align*}\n\\ell_t({\\mB}_t) - \\ell_t(\\mH) &\\leq \\frac{L_1^2}{2\\rho}\\|\\mW_t-\\hat{\\mH}\\|_F^2-\\frac{L_1^2}{2\\rho}\\|\\mW_{t+1}-\\hat{\\mH}\\|_F^2+\\frac{\\rho L_1^2}{2} \\|\\tilde{\\mG}_t\\|_F^2\\\\\n&\\leq \\frac{L_1^2}{2\\rho}\\|\\mW_t-\\hat{\\mH}\\|_F^2-\\frac{L_1^2}{2\\rho}\\|\\mW_{t+1}-\\hat{\\mH}\\|_F^2+60.5\\rho\\ell_t({\\mB}_t). %\n%\n\\end{align*}\n","sha256_utf8":"16f1ac2586e529222141918e5898d13a922ef5c18589540798e4853d4fa55fd8"}]},{"id":"S092","semantic_type":"proof_step","kind":"proof_step","title":"选取学习率并望远镜求和","source_units":[{"start":171730,"end":172420,"start_line":2997,"end_line":3008,"raw_tex":"Since $\\rho = \\frac{1}{121}$, by rearranging and simplifying terms in the above inequality, we obtain\n%\n%\n%\n$\\ell_t({\\mB}_t) \\leq 2\\ell_t({\\mH})+ {121 L_1^2}\\|\\mW_t-\\hat{\\mH}\\|_F^2-{121 L_1^2}\\|\\mW_{t+1}-\\hat{\\mH}\\|_F^2$.\n%\nBy summing the above inequality from $t=0$ to $T-1$, we further have \n\\begin{equation*}\n  \\sum_{t=0}^{T-1} \\ell_t({\\mB}_t)  \\leq {121 L_1^2}\\|\\mW_0-\\hat{\\mH}\\|_F^2+ 2 \\sum_{t=0}^{T-1} \\ell_t({\\mH}) = 121\\|\\mB_0-{\\mH}\\|_F^2+ 2 \\sum_{t=0}^{T-1} \\ell_t({\\mH}),\n\\end{equation*}\nwhere the last equality is due to ${\\mW}_0 \\triangleq \\frac{1}{L_1}(\\mB_0-{(L_1+\\mu)}\\mI)$ and $\\hat{\\mH} \\triangleq \\frac{1}{L_1}(\\mH-{(L_1+\\mu)}\\mI)$. This completes the proof. \n\\end{proof}\n","sha256_utf8":"bd29df4669aae1a186b50bada32c1d153ff1fb6923c8b28bb3f87285a63a29a6"}]},{"id":"S093","semantic_type":"lemma","kind":"lemma","title":"近端位移的平方可和性","source_units":[{"start":172965,"end":173180,"start_line":3017,"end_line":3019,"raw_tex":"\\begin{lemma}\\label{lem:sum_of_squares}\n If Assumption~\\ref{assum:strong_monotone} holds, then $\\sum_{k=0}^{N-1} \\|\\hat{\\vz}_{k}\\!-\\!\\vz_k\\|^2 \\leq \\frac{1}{1-(\\alpha_1+\\alpha_2)}\\|\\vz_0\\!-\\!\\vz^*\\|^2$. \n\\end{lemma}","sha256_utf8":"2290769be92581e905ebcc01b0cb188f03c5b4d779015b4a0213e7b09b5cecfd"}]},{"id":"S094","semantic_type":"proof","kind":"proof","title":"位移平方和引理证明","source_units":[{"start":173181,"end":174317,"start_line":3020,"end_line":3030,"raw_tex":"\\begin{proof}\n  Recall the inequality~\\eqref{eq:one_step_strongly} in the proof of Proposition~\\ref{prop:HPE}. Since $\\frac{1}{2}\\|\\vz_{k+1}-\\vz^*\\|^2 \\!\\leq \\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1}-\\vz^*\\|^2$, we can further derive that \n  \\begin{equation}\\label{eq:one_step_strongly_coro}\n    \\frac{1-\\alpha_1-\\alpha_2}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2) \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1}{2}\\|\\vz_{k+1} - \\vz^*\\|^2.\n\\end{equation} \nSince $\\alpha_1+\\alpha_2 < 1$, we further have $\\frac{1-\\alpha_1-\\alpha_2}{2} \\|\\hat{\\vz}_k - \\vz_k\\|^2 \\leq \\frac{1-\\alpha_1-\\alpha_2}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2) \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1}{2}\\|\\vz_{k+1} - \\vz^*\\|^2$. By summing the above inequality from $k=0$ to $k=N-1$, we obtain that \n\\begin{equation*}\n  \\frac{1-\\alpha_1-\\alpha_2}{2} \\sum_{k=0}^{N-1} \\|\\hat{\\vz}_k - \\vz_k\\|^2 \\leq \\frac{1}{2}\\|\\vz_0-\\vz^*\\|^2 - \\frac{1}{2}\\|\\vz_N-\\vz^*\\|^2 \\leq \\frac{1}{2}\\|\\vz_0-\\vz^*\\|^2. \n\\end{equation*}\nThus, we arrive at Lemma~\\ref{lem:sum_of_squares} by dividing both sides by $\\frac{1-\\alpha_1-\\alpha_2}{2}$. \n\\end{proof}","sha256_utf8":"27c1c61ff0ed6ef116a698420a7dcb870313ec8825bcfbc580cbb430b1d71e60"}]},{"id":"S095","semantic_type":"proof_step","kind":"proof_step","title":"从单步能量式提取位移项","source_units":[{"start":173181,"end":173645,"start_line":3020,"end_line":3024,"raw_tex":"\\begin{proof}\n  Recall the inequality~\\eqref{eq:one_step_strongly} in the proof of Proposition~\\ref{prop:HPE}. Since $\\frac{1}{2}\\|\\vz_{k+1}-\\vz^*\\|^2 \\!\\leq \\frac{1+2\\eta_k\\mu}{2}\\|\\vz_{k+1}-\\vz^*\\|^2$, we can further derive that \n  \\begin{equation}\\label{eq:one_step_strongly_coro}\n    \\frac{1-\\alpha_1-\\alpha_2}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2) \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1}{2}\\|\\vz_{k+1} - \\vz^*\\|^2.\n\\end{equation} \n","sha256_utf8":"037637021bfbebaf58c4ca7274de400e55022e0169d347df08575af236211ac2"}]},{"id":"S096","semantic_type":"proof_step","kind":"proof_step","title":"望远镜求和得到平方和界","source_units":[{"start":173645,"end":174318,"start_line":3025,"end_line":3030,"raw_tex":"Since $\\alpha_1+\\alpha_2 < 1$, we further have $\\frac{1-\\alpha_1-\\alpha_2}{2} \\|\\hat{\\vz}_k - \\vz_k\\|^2 \\leq \\frac{1-\\alpha_1-\\alpha_2}{2}(\\|\\hat{\\vz}_k - \\vz_k\\|^2 + \\|\\hat{\\vz}_k-\\vz_{k+1}\\|^2) \\leq \\frac{1}{2}\\|\\vz_k- \\vz^*\\|^2 - \\frac{1}{2}\\|\\vz_{k+1} - \\vz^*\\|^2$. By summing the above inequality from $k=0$ to $k=N-1$, we obtain that \n\\begin{equation*}\n  \\frac{1-\\alpha_1-\\alpha_2}{2} \\sum_{k=0}^{N-1} \\|\\hat{\\vz}_k - \\vz_k\\|^2 \\leq \\frac{1}{2}\\|\\vz_0-\\vz^*\\|^2 - \\frac{1}{2}\\|\\vz_N-\\vz^*\\|^2 \\leq \\frac{1}{2}\\|\\vz_0-\\vz^*\\|^2. \n\\end{equation*}\nThus, we arrive at Lemma~\\ref{lem:sum_of_squares} by dividing both sides by $\\frac{1-\\alpha_1-\\alpha_2}{2}$. \n\\end{proof}\n","sha256_utf8":"c1a27f1343eae1dae3d16409259a3383e56cea424e76f688f25d3e97aeaa74b8"}]},{"id":"S097","semantic_type":"lemma","kind":"lemma","title":"最优点雅可比的累计损失界","source_units":[{"start":174435,"end":174713,"start_line":3033,"end_line":3045,"raw_tex":"\\begin{lemma}\\label{lem:comparator}\n  Recall the constant $C$ defined in \\eqref{eq:def_c1}. \n  %\n  Then \n    %\n      %\n    %\n    %\n    %\n    %\n    $\\sum_{t=0}^{T-1} \\ell_t(\\mH^*) \\leq \\left(C+2+\\frac{15L_1}{2\\alpha_2 \\beta \\mu}\\right)L_2^2\\|\\vz_0-\\vz^*\\|^2$.\n    %\n  \\end{lemma}","sha256_utf8":"c3c89eaf8f01308c48a2a37bc4f68a337dd5bb7c0cbc365086d84e02c4813e1f"}]},{"id":"S098","semantic_type":"proof","kind":"proof","title":"比较矩阵累计损失证明","source_units":[{"start":174716,"end":177585,"start_line":3046,"end_line":3094,"raw_tex":"\\begin{proof}\n    Recall the definition of $\\ell_t$ in \\eqref{eq:loss_of_jacobian}. \n    By the fundamental theorem of calculus, we can write $\\vu_t = \\vF({\\tilde{\\vz}_t})-\\vF(\\vz_t) = \\bar{\\mH}_t (\\tilde{\\vz}_t-\\vz_t)$, where $\\bar{\\mH}_t = \\int_{0}^1 \\nabla \\vF(\\vz_t+ \\lambda \\vs_t) \\,d\\lambda$. Moreover, using the triangle inequality, we have \n\\begin{align*}\n  \\|\\bar{\\mH}_t-\\mH^*\\|_{\\op} &\\leq \\int_{0}^1 \\|\\nabla \\vF(\\vz_t+ \\lambda \\vs_t)-\\nabla \\vF(\\vz^*)\\|_{\\op}\\, d\\lambda \\\\\n  &\\leq L_2\\int_{0}^1 \\|\\vz_t-\\lambda \\vs_t + \\vz^*\\|\\, d\\lambda \\\\\n  &\\leq L_2\\int_{0}^1 (\\|\\vz_t-\\vz^*\\| + \\lambda \\|\\vs_t\\|)\\, d\\lambda = L_2\\Bigl(\\|\\vz_t-\\vz^*\\|+\\frac{1}{2}\\|\\vs_t\\|\\Bigr),\n\\end{align*}\nwhere we used Assumption~\\ref{assum:jacobian_lips} in the second inequality. \n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\nTherefore, we have $\\|\\vu_t-\\mH^*\\vs_t\\| = \\|(\\bar{\\mH}_t-\\mH^*)\\vs_t\\| \\leq \\|\\bar{\\mH}_t-\\mH^*\\|_{\\op}\\|\\vs_t\\| \\leq L_2\\|\\vs_t\\|\\Bigl(\\|\\vz_t-\\vz^*\\|+\\frac{1}{2}\\|\\vs_t\\|\\Bigr)$. This further implies that \n\\begin{align}\n  \\sum_{t=0}^{T-1} \\ell_t(\\mH^*) = \\sum_{t=0}^{T-1} \\frac{\\|\\vu_t-\\mH^*\\vs_t\\|^2}{\\|\\vs_t\\|^2} &\\leq {L_2^2}\\sum_{t=0}^{T-1}\\Bigl(\\|\\vz_t-\\vz^*\\|+\\frac{1}{2}\\|\\vs_t\\|\\Bigr)^2 \\nonumber\\\\ \n  &\\leq \\frac{L_2^2}{2} \\sum_{t=0}^{T-1} \\|\\vs_t\\|^2 + 2 L_2^2\\sum_{t=0}^{T-1} \\|{\\vz}_t-\\vz^*\\|^2. \\label{eq:comparator_1}\n\\end{align}\n\\blue{Note that by our notation, we have $\\sum_{t=0}^{T-1} \\|\\vs_t\\|^2 = \\sum_{k \\in \\calB} \\|\\vs_k\\|^2 = \\sum_{k \\in \\calB} \\|\\tilde{\\vz}_k - \\vz_k\\|^2 $ and $\\sum_{t=0}^{T-1} \\|{\\vz}_t-\\vz^*\\|^2 = \\sum_{k\\in \\calB} \\|{\\vz}_k-\\vz^*\\|^2$.}\nTo bound the first sum $\\sum_{k \\in \\calB} \\|\\tilde{\\vz}_k - \\vz_k\\|^2$, we use the last result in Lemma~\\ref{lem:step size_lb} and the inequality in Lemma~\\ref{lem:sum_of_squares} to get \n\\begin{equation}\\label{eq:comparator_2}\n  \\begin{aligned}\n    \\sum_{k\\in \\calB} \\|\\tilde{\\vz}_k-\\vz_k\\|^2  \\leq \\frac{(1+\\alpha_1)^2}{\\beta^2 (1-\\alpha_1)^2} \\sum_{k\\in \\calB} \\|\\hat{\\vz}_k-\\vz_k\\|^2 \n    %\n    %\n    &\\leq \\frac{(1+\\alpha_1)^2}{\\beta^2 (1-\\alpha_1)^2} \\sum_{k=0}^{N-1} \\|\\hat{\\vz}_k-\\vz_k\\|^2 \\\\\n      &\\leq \\frac{(1+\\alpha_1)^2\\|\\vz_0-\\vz^*\\|^2}{(1-\\alpha_1)^2\\beta^2 (1-(\\alpha_1+\\alpha_2))} = 2C \\|\\vz_0-\\vz^*\\|^2. \n  \\end{aligned}\n  %\n\\end{equation}\nTo bound the second sum $\\sum_{k\\in \\calB} \\|{\\vz}_k-\\vz^*\\|^2$, we use the linear convergence result in Part (a) of Theorem~\\ref{thm:main}:\n\\begin{equation}\\label{eq:comparator_3}\n \\begin{aligned}\n  \\sum_{k\\in \\calB} \\|{\\vz}_k-\\vz^*\\|^2 \\leq \\sum_{k=0}^{N-1} \\|\\vz_k-\\vz^*\\|^2 &\\leq \\|\\vz_0-\\vz^*\\|^2\\sum_{k=0}^{N-1} \\left(1+\\frac{4\\alpha_2\\beta\\mu}{15L_1}\\right)^{-k}\\\\ &\\leq  \\|\\vz_0-\\vz^*\\|^2\\left(1+\\frac{15L_1}{4\\alpha_2 \\beta \\mu}\\right).\n \\end{aligned}\n\\end{equation}\nLemma~\\ref{lem:comparator} follows immediately from \\eqref{eq:comparator_1}, \\eqref{eq:comparator_2}, and \\eqref{eq:comparator_3}. \n  \\end{proof}","sha256_utf8":"49052916aa17a421a709dbe4a8f64baecd1c7fa1213964b969395e417b8af5ee"}]},{"id":"S099","semantic_type":"proof_step","kind":"proof_step","title":"积分雅可比逼近最优点雅可比","source_units":[{"start":174714,"end":175517,"start_line":3046,"end_line":3069,"raw_tex":"  \\begin{proof}\n    Recall the definition of $\\ell_t$ in \\eqref{eq:loss_of_jacobian}. \n    By the fundamental theorem of calculus, we can write $\\vu_t = \\vF({\\tilde{\\vz}_t})-\\vF(\\vz_t) = \\bar{\\mH}_t (\\tilde{\\vz}_t-\\vz_t)$, where $\\bar{\\mH}_t = \\int_{0}^1 \\nabla \\vF(\\vz_t+ \\lambda \\vs_t) \\,d\\lambda$. Moreover, using the triangle inequality, we have \n\\begin{align*}\n  \\|\\bar{\\mH}_t-\\mH^*\\|_{\\op} &\\leq \\int_{0}^1 \\|\\nabla \\vF(\\vz_t+ \\lambda \\vs_t)-\\nabla \\vF(\\vz^*)\\|_{\\op}\\, d\\lambda \\\\\n  &\\leq L_2\\int_{0}^1 \\|\\vz_t-\\lambda \\vs_t + \\vz^*\\|\\, d\\lambda \\\\\n  &\\leq L_2\\int_{0}^1 (\\|\\vz_t-\\vz^*\\| + \\lambda \\|\\vs_t\\|)\\, d\\lambda = L_2\\Bigl(\\|\\vz_t-\\vz^*\\|+\\frac{1}{2}\\|\\vs_t\\|\\Bigr),\n\\end{align*}\nwhere we used Assumption~\\ref{assum:jacobian_lips} in the second inequality. \n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n%\n","sha256_utf8":"c5abdd047f654472c5b42f45de129f800802f3bb7b0bf69a9764ca1de4a310ed"}]},{"id":"S100","semantic_type":"proof_step","kind":"proof_step","title":"将比较损失拆成两个平方和","source_units":[{"start":175517,"end":176304,"start_line":3070,"end_line":3075,"raw_tex":"Therefore, we have $\\|\\vu_t-\\mH^*\\vs_t\\| = \\|(\\bar{\\mH}_t-\\mH^*)\\vs_t\\| \\leq \\|\\bar{\\mH}_t-\\mH^*\\|_{\\op}\\|\\vs_t\\| \\leq L_2\\|\\vs_t\\|\\Bigl(\\|\\vz_t-\\vz^*\\|+\\frac{1}{2}\\|\\vs_t\\|\\Bigr)$. This further implies that \n\\begin{align}\n  \\sum_{t=0}^{T-1} \\ell_t(\\mH^*) = \\sum_{t=0}^{T-1} \\frac{\\|\\vu_t-\\mH^*\\vs_t\\|^2}{\\|\\vs_t\\|^2} &\\leq {L_2^2}\\sum_{t=0}^{T-1}\\Bigl(\\|\\vz_t-\\vz^*\\|+\\frac{1}{2}\\|\\vs_t\\|\\Bigr)^2 \\nonumber\\\\ \n  &\\leq \\frac{L_2^2}{2} \\sum_{t=0}^{T-1} \\|\\vs_t\\|^2 + 2 L_2^2\\sum_{t=0}^{T-1} \\|{\\vz}_t-\\vz^*\\|^2. \\label{eq:comparator_1}\n\\end{align}\n\\blue{Note that by our notation, we have $\\sum_{t=0}^{T-1} \\|\\vs_t\\|^2 = \\sum_{k \\in \\calB} \\|\\vs_k\\|^2 = \\sum_{k \\in \\calB} \\|\\tilde{\\vz}_k - \\vz_k\\|^2 $ and $\\sum_{t=0}^{T-1} \\|{\\vz}_t-\\vz^*\\|^2 = \\sum_{k\\in \\calB} \\|{\\vz}_k-\\vz^*\\|^2$.}\n","sha256_utf8":"a352cb544c1937d29e780dd6f006cff0a6060adaff6e2bb9f521b36fe87f87d1"}]},{"id":"S101","semantic_type":"proof_step","kind":"proof_step","title":"控制拒绝点位移平方和","source_units":[{"start":176304,"end":176964,"start_line":3076,"end_line":3086,"raw_tex":"To bound the first sum $\\sum_{k \\in \\calB} \\|\\tilde{\\vz}_k - \\vz_k\\|^2$, we use the last result in Lemma~\\ref{lem:step size_lb} and the inequality in Lemma~\\ref{lem:sum_of_squares} to get \n\\begin{equation}\\label{eq:comparator_2}\n  \\begin{aligned}\n    \\sum_{k\\in \\calB} \\|\\tilde{\\vz}_k-\\vz_k\\|^2  \\leq \\frac{(1+\\alpha_1)^2}{\\beta^2 (1-\\alpha_1)^2} \\sum_{k\\in \\calB} \\|\\hat{\\vz}_k-\\vz_k\\|^2 \n    %\n    %\n    &\\leq \\frac{(1+\\alpha_1)^2}{\\beta^2 (1-\\alpha_1)^2} \\sum_{k=0}^{N-1} \\|\\hat{\\vz}_k-\\vz_k\\|^2 \\\\\n      &\\leq \\frac{(1+\\alpha_1)^2\\|\\vz_0-\\vz^*\\|^2}{(1-\\alpha_1)^2\\beta^2 (1-(\\alpha_1+\\alpha_2))} = 2C \\|\\vz_0-\\vz^*\\|^2. \n  \\end{aligned}\n  %\n\\end{equation}\n","sha256_utf8":"1fb774b7e998a26c6bd0ff1c01b88f2c998b808fb5538a8c542c7d6b569e2998"}]},{"id":"S102","semantic_type":"proof_step","kind":"proof_step","title":"利用已证线性率控制距离和","source_units":[{"start":176964,"end":177586,"start_line":3087,"end_line":3094,"raw_tex":"To bound the second sum $\\sum_{k\\in \\calB} \\|{\\vz}_k-\\vz^*\\|^2$, we use the linear convergence result in Part (a) of Theorem~\\ref{thm:main}:\n\\begin{equation}\\label{eq:comparator_3}\n \\begin{aligned}\n  \\sum_{k\\in \\calB} \\|{\\vz}_k-\\vz^*\\|^2 \\leq \\sum_{k=0}^{N-1} \\|\\vz_k-\\vz^*\\|^2 &\\leq \\|\\vz_0-\\vz^*\\|^2\\sum_{k=0}^{N-1} \\left(1+\\frac{4\\alpha_2\\beta\\mu}{15L_1}\\right)^{-k}\\\\ &\\leq  \\|\\vz_0-\\vz^*\\|^2\\left(1+\\frac{15L_1}{4\\alpha_2 \\beta \\mu}\\right).\n \\end{aligned}\n\\end{equation}\nLemma~\\ref{lem:comparator} follows immediately from \\eqref{eq:comparator_1}, \\eqref{eq:comparator_2}, and \\eqref{eq:comparator_3}. \n  \\end{proof}\n","sha256_utf8":"bd8cbff375f382114a53da2e22718ae41e0b34c40c7694facf70ec743df73d22"}]},{"id":"S103","semantic_type":"lemma","kind":"lemma","title":"回溯搜索的接受阈值","source_units":[{"start":222212,"end":222571,"start_line":3862,"end_line":3864,"raw_tex":"\\begin{lemma}\\label{lem:ls_terminates}\n  Suppose Assumption~\\ref{assum:operator_lips} holds. If ${\\eta}_+ <\\frac{\\alpha_2}{L_1+\\|\\mB\\|_{\\op}}$ and $\\hat{\\vz}_+$ is computed according to \\eqref{eq:linear_solver_update}, then the pair $({\\eta}_+,\\hat{\\vz}_+)$ satisfies the conditions in \\eqref{eq:x_plus_update} and \\eqref{eq:step size_condition}. \n\\end{lemma}","sha256_utf8":"0ba34cb72cb7f20bc6a3c7ddd7c29ae20962c2b3faff0f4e74bdd501867bc980"}]},{"id":"S104","semantic_type":"proof","kind":"proof","title":"回溯终止引理证明","source_units":[{"start":222573,"end":224433,"start_line":3866,"end_line":3891,"raw_tex":"\\begin{proof}\n  Since $\\hat{\\vz}_+$ follows the update rule in \\eqref{eq:linear_solver_update}, \n  by Definition~\\ref{def:linear_solver}, the pair $({\\eta}_+,\\hat{\\vz}_+)$ will always satisfy the condition in \\eqref{eq:x_plus_update}. \n  %\n  Hence, it is sufficient to prove that the condition in \\eqref{eq:step size_condition} also holds. \n  Recall that $\\vg$ is defined as $\\vg = \\vF(\\vz)$. By Assumption~\\ref{assum:operator_lips}, the operator $\\vF$ is $L_1$-Lipschitz and thus it follow that  \n  %\n  $\\|\\vF(\\hat{\\vz}_+)-\\vg\\| = \\|\\vF(\\hat{\\vz}_+)-\\vF(\\vz)\\| \\leq L_1 \\|\\hat{\\vz}_+ - \\vz\\|$. \n  %\n  Further, applying the triangle inequality yields \n  \\begin{equation*}\n      \\|\\vF(\\hat{\\vz}_+)-\\vg-\\mB(\\hat{\\vz}_+-\\vz)\\| \\leq \\|\\vF(\\hat{\\vz}_+)-\\vg\\| + \\|\\mB(\\hat{\\vz}_+-\\vz)\\| \\leq (L_1  + \\|\\mB\\|_{\\op}) \\|\\hat{\\vz}_+ - \\vz\\|.\n  \\end{equation*}\n  Hence, if ${\\eta}_+ \\leq \\frac{\\alpha_2}{L_1+\\|\\mB\\|_{\\op}}$, we have \n     \\begin{equation}\\label{eq:approx_error}\n         {\\eta}_+ \\|\\vF(\\hat{\\vz}_+)-\\vg-\\mB(\\hat{\\vz}_+-\\vz)\\| \\leq \\alpha_2 \\|\\hat{\\vz}_+ - \\vz\\|.\n     \\end{equation}\n  Finally, by using the triangle inequality again,  we combine \\eqref{eq:x_plus_update} and \\eqref{eq:approx_error} to establish that  \n  \\begin{align*}\n      \\|\\hat{\\vz}_{+}-\\vz+{\\eta}_+ \\vF(\\hat{\\vz}_{+})\\| &= \\|\\hat{\\vz}_{+}-\\vz+{{\\eta}_+}(\\vg+\\mB({\\hat{\\vz}_{+}}-\\vz)) + {\\eta}_+(\\vF(\\hat{\\vz}_+)-\\vg-\\mB(\\hat{\\vz}_+-\\vz))\\| \\\\\n      &\\leq \\|\\hat{\\vz}_{+}-\\vz+{{\\eta}_+}(\\vg+\\mB({\\hat{\\vz}_{+}}-\\vz))\\| + \\|{\\eta}_+(\\vF(\\hat{\\vz}_+)-\\vg-\\mB(\\hat{\\vz}_+-\\vz))\\| \\\\\n      &\\leq \\alpha_1 \\sqrt{1+\\eta_+ \\mu}\\|\\hat{\\vz}_+-\\vz\\|+\\alpha_2 \\|\\hat{\\vz}_+-\\vz\\| \\\\\n      &\\leq (\\alpha_1+\\alpha_2)\\sqrt{1+\\eta_+ \\mu}\\|\\hat{\\vz}_+-\\vz\\|, \n  \\end{align*}\n  which confirms that the condition in \\eqref{eq:step size_condition} is satisfied. This completes the proof. \n  \\end{proof}","sha256_utf8":"27dd749ac1252e72f9039dd8e4d74576645cb311ea350505210650d9aedd753d"}]},{"id":"S105","semantic_type":"proof_step","kind":"proof_step","title":"求解接口保证首个残差条件","source_units":[{"start":222573,"end":223169,"start_line":3866,"end_line":3873,"raw_tex":"\\begin{proof}\n  Since $\\hat{\\vz}_+$ follows the update rule in \\eqref{eq:linear_solver_update}, \n  by Definition~\\ref{def:linear_solver}, the pair $({\\eta}_+,\\hat{\\vz}_+)$ will always satisfy the condition in \\eqref{eq:x_plus_update}. \n  %\n  Hence, it is sufficient to prove that the condition in \\eqref{eq:step size_condition} also holds. \n  Recall that $\\vg$ is defined as $\\vg = \\vF(\\vz)$. By Assumption~\\ref{assum:operator_lips}, the operator $\\vF$ is $L_1$-Lipschitz and thus it follow that  \n  %\n  $\\|\\vF(\\hat{\\vz}_+)-\\vg\\| = \\|\\vF(\\hat{\\vz}_+)-\\vF(\\vz)\\| \\leq L_1 \\|\\hat{\\vz}_+ - \\vz\\|$. \n","sha256_utf8":"6cab4ba4757f185aa58b8a8873aaae124dff2d527f0ada35fc800dd9e43e2e5e"}]},{"id":"S106","semantic_type":"proof_step","kind":"proof_step","title":"小步长控制线性化误差","source_units":[{"start":223169,"end":223662,"start_line":3874,"end_line":3882,"raw_tex":"  %\n  Further, applying the triangle inequality yields \n  \\begin{equation*}\n      \\|\\vF(\\hat{\\vz}_+)-\\vg-\\mB(\\hat{\\vz}_+-\\vz)\\| \\leq \\|\\vF(\\hat{\\vz}_+)-\\vg\\| + \\|\\mB(\\hat{\\vz}_+-\\vz)\\| \\leq (L_1  + \\|\\mB\\|_{\\op}) \\|\\hat{\\vz}_+ - \\vz\\|.\n  \\end{equation*}\n  Hence, if ${\\eta}_+ \\leq \\frac{\\alpha_2}{L_1+\\|\\mB\\|_{\\op}}$, we have \n     \\begin{equation}\\label{eq:approx_error}\n         {\\eta}_+ \\|\\vF(\\hat{\\vz}_+)-\\vg-\\mB(\\hat{\\vz}_+-\\vz)\\| \\leq \\alpha_2 \\|\\hat{\\vz}_+ - \\vz\\|.\n     \\end{equation}\n","sha256_utf8":"b356a29eb406cd56563c0dc45285db0a40daebd8cbc5ffbd232b5db177581248"}]},{"id":"S107","semantic_type":"proof_step","kind":"proof_step","title":"合并残差得到接受条件","source_units":[{"start":223662,"end":224433,"start_line":3883,"end_line":3891,"raw_tex":"  Finally, by using the triangle inequality again,  we combine \\eqref{eq:x_plus_update} and \\eqref{eq:approx_error} to establish that  \n  \\begin{align*}\n      \\|\\hat{\\vz}_{+}-\\vz+{\\eta}_+ \\vF(\\hat{\\vz}_{+})\\| &= \\|\\hat{\\vz}_{+}-\\vz+{{\\eta}_+}(\\vg+\\mB({\\hat{\\vz}_{+}}-\\vz)) + {\\eta}_+(\\vF(\\hat{\\vz}_+)-\\vg-\\mB(\\hat{\\vz}_+-\\vz))\\| \\\\\n      &\\leq \\|\\hat{\\vz}_{+}-\\vz+{{\\eta}_+}(\\vg+\\mB({\\hat{\\vz}_{+}}-\\vz))\\| + \\|{\\eta}_+(\\vF(\\hat{\\vz}_+)-\\vg-\\mB(\\hat{\\vz}_+-\\vz))\\| \\\\\n      &\\leq \\alpha_1 \\sqrt{1+\\eta_+ \\mu}\\|\\hat{\\vz}_+-\\vz\\|+\\alpha_2 \\|\\hat{\\vz}_+-\\vz\\| \\\\\n      &\\leq (\\alpha_1+\\alpha_2)\\sqrt{1+\\eta_+ \\mu}\\|\\hat{\\vz}_+-\\vz\\|, \n  \\end{align*}\n  which confirms that the condition in \\eqref{eq:step size_condition} is satisfied. This completes the proof. \n  \\end{proof}","sha256_utf8":"692e032a858e72afe8ad7184de9082c2aa9b4451e390cb051aa1bb5241046058"}]},{"id":"S108","semantic_type":"proof_step","kind":"proof_step","title":"并集界保证全部预言机成功","source_units":[{"start":161821,"end":162646,"start_line":2832,"end_line":2841,"raw_tex":"Now we move to the proof of Theorem~\\ref{thm:main}. First of all, by using a simple union bound, we can show that \n the  $\\mathsf{ExtEvec}$ and $\\mathsf{MaxSvec}$ oracles used in Subroutine~\\ref{alg:jacobian_approx} are successful in all rounds with probability at least $1-p$. \n %\n Specifically, \nsince both  oracles have a failure probability of $ \\frac{q_t}{2} = \\frac{p}{5(t+1)\\log^2(t+1)}$ in the $t$-th round, we can use the union bound to upper bound the total failure probability by \n\\begin{equation*}\n  \\sum_{t=1}^{T-1} q_t %\n   \\leq  \\frac{p}{2.5}\\sum_{t=2}^{\\infty} \\frac{1}{t \\log^2 t} \\leq \\frac{p}{2.5}\\left(\\frac{1}{2\\log^2 2}+\\int_{2}^{+\\infty}\\frac{1}{t\\log^2t}\\,dt\\right) \\leq p.\n\\end{equation*}\n Thus, throughout the proof, we assume every call of $\\mathsf{ExtEvec}$ and $\\mathsf{MaxSvec}$ is successful. \n","sha256_utf8":"4c24c444a630ae21074f635ba6f53cb25f519bb551ef6f12bd2d68e43fd60b8b"}]},{"id":"S109","semantic_type":"proof_step","kind":"proof_step","title":"由谱界转向一致步长下界","source_units":[{"start":163337,"end":163461,"start_line":2855,"end_line":2855,"raw_tex":"Combining this with Lemma~\\ref{lem:step size_lb}, we obtain the following universal lower bound on the step size $\\eta_k$. \n","sha256_utf8":"f494f78ad8beb87caee752d7f50c7ca9bd5aaabd76088b3a4bd367a9561c3b1e"}]},{"id":"S110","semantic_type":"proof_step","kind":"proof_step","title":"组合单步收缩与步长下界","source_units":[{"start":165679,"end":165988,"start_line":2884,"end_line":2886,"raw_tex":"In light of Lemma~\\ref{lem:stepsize_const_bound}, it follows from Proposition~\\ref{prop:HPE}(a) that  $\\|\\vz_{k+1}-\\vz^*\\|^2 \\leq \\|\\vz_k-\\vz^*\\|^2 (1+2\\eta_k \\mu)^{-1} \\leq \\|\\vz_k-\\vz^*\\|^2 \\left(1+\\frac{4\\alpha_2\\beta\\mu}{15L_1}\\right)^{-1}$. \nThis proves the linear convergence result in (a). \n\\end{proof}","sha256_utf8":"05acedfbd85936fe058ef870a641e70bb2e84fe8045a712b55149056d425e1a9"}]},{"id":"S111","semantic_type":"proof_step","kind":"proof_step","title":"将超线性率归约为累计损失","source_units":[{"start":165990,"end":166547,"start_line":2888,"end_line":2896,"raw_tex":"Next, we prove the superlinear convergence rate in (b). Our starting point is the observations in \\eqref{eq:after_jensen} and \\eqref{eq:goal}, which demonstrate that it is sufficient to prove an upper bound on the cumulative loss $\\sum_{k=0}^{N-1} \\ell_k(\\mB_k)$. \n%\n%\nMoreover, since $\\ell_k(\\mB_k) = 0$ when $k \\notin \\calB$ by the definition in \\eqref{eq:loss_of_jacobian}, it is equivalent to bounding the sum $\\sum_{k \\in \\calB} \\ell_k(\\mB_k)$.  \n%\n%\n\\blue{Recall that we relabel the indices in $\\mathcal{B}$ by $t=0,\\dots,T-1$ with $T \\leq N$. %\n%\n} \n","sha256_utf8":"2fd3d39fb9b5521500733e26ef36522e5dde38106848392dcb039479c8cb0335"}]},{"id":"S112","semantic_type":"proof_step","kind":"proof_step","title":"以梯度自界启动遗憾分析","source_units":[{"start":166581,"end":167292,"start_line":2915,"end_line":2918,"raw_tex":"\\begin{proof}[Proof of Theorem~\\ref{thm:main}(b)] We consider the following three steps.\n\n\\vspace{0em}\\noindent\\textbf{Step 1:} Using regret analysis, we bound the cumulative loss $\\sum_{t=0}^{T-1}\\ell_{t}(\\mB_{t})$ incurred by our online learning algorithm in Subroutine~\\ref{alg:jacobian_approx}. \nTo begin with, we present the following lemma showing a self-bounding property of the loss function $\\ell_t$. {It is similar to the standard inequality $\\frac{1}{2L_1} \\|\\nabla g (\\vx)\\|^2 \\leq g(\\vx) - g^* $ for a $L_1$-smooth function $g$, where $g^*$ denotes the minimum of $g$.} This will be the key to proving a constant upper bound on the cumulative loss incurred by Subroutine~\\ref{alg:jacobian_approx}.\n","sha256_utf8":"f18581c24f16e704f7836e3823f4589a5a275d242554ffaff33d842a16c0924c"}]},{"id":"S113","semantic_type":"proof_step","kind":"proof_step","title":"由自界性质引入小损失分析","source_units":[{"start":168499,"end":168776,"start_line":2940,"end_line":2942,"raw_tex":"%\nIn particular, by exploiting the self-bounding property of the loss function $\\ell_t$, we prove a ``small-loss bound'' \\cite{srebro2010smoothness} in the following lemma, where the cumulative loss of the learner is bounded by that of a fixed action in the competitor set. \n%\n","sha256_utf8":"fc24ea2f66515fb35d8aaec456cfcbf6864beea66f9e070bc88b7d6514eeb191"}]},{"id":"S114","semantic_type":"proof_step","kind":"proof_step","title":"选择最优点雅可比作为比较矩阵","source_units":[{"start":172421,"end":172965,"start_line":3010,"end_line":3016,"raw_tex":"%\nNote that in Lemma~\\ref{lem:small_loss}, we have the freedom to choose any competitor $\\mH$ in the set $\\mathcal{Z}$. To further obtain an explicit bound, a natural choice would be $\\mH^* \\triangleq \\nabla \\vF(\\vz^*)$, which leads to our next step.\n\n%\n\n\\vspace{0em}\\noindent \\textbf{Step 2:} We upper bound the cumulative loss $\\sum_{t=0}^{T-1} \\ell_t({\\mH}^*)$. The proof relies crucially on Assumption~\\ref{assum:jacobian_lips} as well as the linear convergence result we proved in (a).\nTo begin with, we prove the following useful lemma. \n","sha256_utf8":"9674673bfb92dc27c874e0be052f3c81266713d3ea9e95cdb6c7c44dc3fbe79c"},{"start":174319,"end":174433,"start_line":3032,"end_line":3032,"raw_tex":"With the help of Lemma~\\ref{lem:sum_of_squares}, we are ready to upper bound $\\sum_{t=0}^{T-1} \\ell_t({\\mH}^*)$. \n","sha256_utf8":"68672d03c49f07cb7ff1692aab0e03dd9058927853c4c2ea330acd103a5c801f"}]},{"id":"S115","semantic_type":"proof_step","kind":"proof_step","title":"累计损失界导出全局超线性率","source_units":[{"start":177586,"end":178460,"start_line":3095,"end_line":3111,"raw_tex":" %\n%\n  %\n\n\\noindent\\textbf{Step 3}: Combining Lemmas~\\ref{lem:comparator} and~\\ref{lem:small_loss},  we obtain a constant upper bound on the cumulative loss as\n%\n$\\sum_{t=0}^{T-1} \\ell_t({\\mB}_t) \\leq 121\\|\\mB_0-{\\mH^*}\\|_F^2+ \\left(C+2+\\frac{15L_1}{2\\alpha_2 \\beta \\mu}\\right)L_2^2\\|\\vz_0-\\vz^*\\|^2 = L_1^2 M$, where we used the definition of $M$ in \\eqref{eq:def_M}. \n%\n%\nTogether with the fact that $\\sigma_0 \\geq \\frac{\\alpha_2\\beta}{7.5L_1}$, it follows from  Lemma~\\ref{lem:stepsize_bnd} that\n\\begin{equation*}\n  \\sum_{i=0}^{k-1}\\frac{1}{\\eta_i} \\leq \\frac{7.5L_1}{(1-\\beta)\\alpha_2\\beta}+ \\frac{L_1\\sqrt{kM}}{(1-\\beta)\\alpha_2\\beta} \\leq \n  \\frac{2L_1 \\max\\{7.5, \\sqrt{kM}\\}}{(1-\\beta)\\alpha_2\\beta}. \n  %\n\\end{equation*}\nThis leads to the superlinear convergence result in (b) by Proposition~\\ref{prop:HPE} and the observation in \\eqref{eq:after_jensen}.\n\\end{proof}","sha256_utf8":"4d7baefcea368ea70987e41068481148da244dea38f1ca503d2cdf528cc69e08"}]},{"id":"S116","semantic_type":"external_fact","kind":"external_fact","title":"单调算子的雅可比刻画","source_units":[],"declared_external":true,"declaration_text":["对连续可微 F，μ-强单调性等价于 (∇F(z)+∇F(z)^T)/2 ⪰ μI；单调情形取 μ=0。F 为 L1-Lipschitz 时，所有 z 均满足 ||∇F(z)||op≤L1。该标准边界事实对应作者引用的 ryu2022large 第2节。"]},{"id":"S117","semantic_type":"external_fact","kind":"external_fact","title":"凸性与单调性的平均间隙界","source_units":[],"declared_external":true,"declaration_text":["对非负且和为1的权重，单调算子的受限弱间隙、凸目标的函数值间隙及凸凹目标的受限原始对偶间隙，均由相应加权线性化内积和上界控制，即作者 B0021 所列经典平均不等式。"]},{"id":"S118","semantic_type":"external_fact","kind":"external_fact","title":"随机Lanczos的标准概率界","source_units":[],"declared_external":true,"declaration_text":["对 d 维实对称矩阵，随机单位初始向量的 k 步 Lanczos 极端 Rayleigh 商偏离极端特征值超过 ε 倍谱宽的概率，各不超过 1.648√d exp(-√ε(2k-1))；联合成功概率和相应迭代数采用 B0102 中引自 kuczynski1992estimating 定理4.2的界。"]},{"id":"S119","semantic_type":"external_fact","kind":"external_fact","title":"对称增广矩阵的奇异谱关系","source_units":[],"declared_external":true,"declaration_text":["实矩阵 W 的对称增广矩阵 [[0,W],[W^T,0]] 的最大、最小特征值分别为 σmax(W) 与 -σmax(W)；最大特征值对应特征向量的两个分块给出相应左右奇异向量方向。对应 Golub2013Matrix 第8.6.1节。"]},{"id":"S120","semantic_type":"external_fact","kind":"external_fact","title":"欧氏不等式与投影性质","source_units":[],"declared_external":true,"declaration_text":["使用 Cauchy–Schwarz、Young、三角不等式、欧氏三点恒等式及凸函数一阶不等式；对闭凸集合 K，y=ΠK(x) 满足 ⟨x-y,y-u⟩≥0，对所有 u∈K 成立。"]},{"id":"S121","semantic_type":"external_fact","kind":"external_fact","title":"矩阵范数与正交投影事实","source_units":[],"declared_external":true,"declaration_text":["使用 ||A||F≤||A||*、|⟨A,B⟩|≤||A||*||B||op、||uv^T||F=||uv^T||*=||u||||v||，以及实对称矩阵的 Rayleigh 商谱界；正交投影自伴、保持子空间内向量并收缩欧氏范数。"]},{"id":"S122","semantic_type":"external_fact","kind":"external_fact","title":"微积分与概率求和事实","source_units":[],"declared_external":true,"declaration_text":["使用向量值微积分基本定理 F(z+s)-F(z)=∫₀¹∇F(z+λs)s dλ、积分三角不等式、对凸函数 log(1+1/t) 的 Jensen 不等式、几何级数公式、可数概率并集界及单调正项级数的积分上界。"]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S025","kind":"supports"},{"source_id":"S003","target_id":"S026","kind":"supports"},{"source_id":"S004","target_id":"S015","kind":"supports"},{"source_id":"S004","target_id":"S025","kind":"supports"},{"source_id":"S004","target_id":"S026","kind":"supports"},{"source_id":"S004","target_id":"S082","kind":"supports"},{"source_id":"S004","target_id":"S099","kind":"supports"},{"source_id":"S004","target_id":"S105","kind":"supports"},{"source_id":"S005","target_id":"S016","kind":"supports"},{"source_id":"S005","target_id":"S010","kind":"supports"},{"source_id":"S006","target_id":"S016","kind":"supports"},{"source_id":"S006","target_id":"S027","kind":"supports"},{"source_id":"S007","target_id":"S019","kind":"supports"},{"source_id":"S007","target_id":"S021","kind":"supports"},{"source_id":"S007","target_id":"S039","kind":"supports"},{"source_id":"S007","target_id":"S083","kind":"supports"},{"source_id":"S008","target_id":"S019","kind":"supports"},{"source_id":"S008","target_id":"S030","kind":"supports"},{"source_id":"S008","target_id":"S031","kind":"supports"},{"source_id":"S008","target_id":"S083","kind":"supports"},{"source_id":"S008","target_id":"S105","kind":"supports"},{"source_id":"S008","target_id":"S107","kind":"supports"},{"source_id":"S009","target_id":"S030","kind":"supports"},{"source_id":"S009","target_id":"S039","kind":"supports"},{"source_id":"S009","target_id":"S086","kind":"supports"},{"source_id":"S009","target_id":"S099","kind":"supports"},{"source_id":"S009","target_id":"S111","kind":"supports"},{"source_id":"S010","target_id":"S059","kind":"supports"},{"source_id":"S010","target_id":"S089","kind":"supports"},{"source_id":"S010","target_id":"S114","kind":"supports"},{"source_id":"S011","target_id":"S044","kind":"supports"},{"source_id":"S011","target_id":"S045","kind":"supports"},{"source_id":"S011","target_id":"S046","kind":"supports"},{"source_id":"S011","target_id":"S047","kind":"supports"},{"source_id":"S011","target_id":"S048","kind":"supports"},{"source_id":"S011","target_id":"S049","kind":"supports"},{"source_id":"S011","target_id":"S062","kind":"supports"},{"source_id":"S012","target_id":"S062","kind":"supports"},{"source_id":"S012","target_id":"S079","kind":"supports"},{"source_id":"S012","target_id":"S089","kind":"supports"},{"source_id":"S012","target_id":"S092","kind":"supports"},{"source_id":"S012","target_id":"S100","kind":"supports"},{"source_id":"S013","target_id":"S062","kind":"supports"},{"source_id":"S013","target_id":"S068","kind":"supports"},{"source_id":"S013","target_id":"S069","kind":"supports"},{"source_id":"S013","target_id":"S070","kind":"supports"},{"source_id":"S013","target_id":"S075","kind":"supports"},{"source_id":"S013","target_id":"S076","kind":"supports"},{"source_id":"S013","target_id":"S090","kind":"supports"},{"source_id":"S014","target_id":"S066","kind":"supports"},{"source_id":"S014","target_id":"S069","kind":"supports"},{"source_id":"S014","target_id":"S070","kind":"supports"},{"source_id":"S014","target_id":"S074","kind":"supports"},{"source_id":"S014","target_id":"S075","kind":"supports"},{"source_id":"S014","target_id":"S076","kind":"supports"},{"source_id":"S015","target_id":"S114","kind":"supports"},{"source_id":"S016","target_id":"S027","kind":"supports"},{"source_id":"S017","target_id":"S025","kind":"supports"},{"source_id":"S017","target_id":"S026","kind":"supports"},{"source_id":"S017","target_id":"S027","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S018","kind":"supports"},{"source_id":"S020","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S018","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S018","kind":"supports"},{"source_id":"S023","target_id":"S036","kind":"supports"},{"source_id":"S023","target_id":"S110","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S025","target_id":"S095","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S024","kind":"supports"},{"source_id":"S028","target_id":"S039","kind":"supports"},{"source_id":"S028","target_id":"S041","kind":"supports"},{"source_id":"S028","target_id":"S082","kind":"supports"},{"source_id":"S028","target_id":"S101","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S029","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S029","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S029","kind":"supports"},{"source_id":"S035","target_id":"S111","kind":"supports"},{"source_id":"S035","target_id":"S115","kind":"supports"},{"source_id":"S036","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S111","kind":"supports"},{"source_id":"S037","target_id":"S115","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S038","kind":"supports"},{"source_id":"S042","target_id":"S079","kind":"supports"},{"source_id":"S042","target_id":"S089","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S046","kind":"supports"},{"source_id":"S044","target_id":"S048","kind":"supports"},{"source_id":"S044","target_id":"S049","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"},{"source_id":"S046","target_id":"S043","kind":"supports"},{"source_id":"S047","target_id":"S043","kind":"supports"},{"source_id":"S047","target_id":"S050","kind":"supports"},{"source_id":"S048","target_id":"S043","kind":"supports"},{"source_id":"S049","target_id":"S043","kind":"supports"},{"source_id":"S050","target_id":"S043","kind":"supports"},{"source_id":"S051","target_id":"S062","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S052","kind":"supports"},{"source_id":"S055","target_id":"S052","kind":"supports"},{"source_id":"S056","target_id":"S052","kind":"supports"},{"source_id":"S057","target_id":"S079","kind":"supports"},{"source_id":"S057","target_id":"S089","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S059","target_id":"S058","kind":"supports"},{"source_id":"S060","target_id":"S058","kind":"supports"},{"source_id":"S061","target_id":"S079","kind":"supports"},{"source_id":"S061","target_id":"S089","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S063","target_id":"S066","kind":"supports"},{"source_id":"S063","target_id":"S074","kind":"supports"},{"source_id":"S064","target_id":"S108","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S067","target_id":"S065","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S069","kind":"supports"},{"source_id":"S067","target_id":"S070","kind":"supports"},{"source_id":"S068","target_id":"S065","kind":"supports"},{"source_id":"S069","target_id":"S065","kind":"supports"},{"source_id":"S070","target_id":"S065","kind":"supports"},{"source_id":"S071","target_id":"S074","kind":"supports"},{"source_id":"S072","target_id":"S108","kind":"supports"},{"source_id":"S073","target_id":"S072","kind":"supports"},{"source_id":"S074","target_id":"S073","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S076","kind":"supports"},{"source_id":"S075","target_id":"S073","kind":"supports"},{"source_id":"S076","target_id":"S073","kind":"supports"},{"source_id":"S077","target_id":"S082","kind":"supports"},{"source_id":"S077","target_id":"S101","kind":"supports"},{"source_id":"S078","target_id":"S077","kind":"supports"},{"source_id":"S079","target_id":"S078","kind":"supports"},{"source_id":"S080","target_id":"S110","kind":"supports"},{"source_id":"S081","target_id":"S080","kind":"supports"},{"source_id":"S082","target_id":"S081","kind":"supports"},{"source_id":"S082","target_id":"S083","kind":"supports"},{"source_id":"S083","target_id":"S081","kind":"supports"},{"source_id":"S084","target_id":"S090","kind":"supports"},{"source_id":"S084","target_id":"S113","kind":"supports"},{"source_id":"S085","target_id":"S084","kind":"supports"},{"source_id":"S086","target_id":"S085","kind":"supports"},{"source_id":"S087","target_id":"S115","kind":"supports"},{"source_id":"S088","target_id":"S087","kind":"supports"},{"source_id":"S089","target_id":"S088","kind":"supports"},{"source_id":"S089","target_id":"S090","kind":"supports"},{"source_id":"S089","target_id":"S091","kind":"supports"},{"source_id":"S090","target_id":"S088","kind":"supports"},{"source_id":"S090","target_id":"S091","kind":"supports"},{"source_id":"S091","target_id":"S088","kind":"supports"},{"source_id":"S091","target_id":"S092","kind":"supports"},{"source_id":"S092","target_id":"S088","kind":"supports"},{"source_id":"S093","target_id":"S101","kind":"supports"},{"source_id":"S094","target_id":"S093","kind":"supports"},{"source_id":"S095","target_id":"S094","kind":"supports"},{"source_id":"S095","target_id":"S096","kind":"supports"},{"source_id":"S096","target_id":"S094","kind":"supports"},{"source_id":"S097","target_id":"S115","kind":"supports"},{"source_id":"S098","target_id":"S097","kind":"supports"},{"source_id":"S099","target_id":"S098","kind":"supports"},{"source_id":"S099","target_id":"S100","kind":"supports"},{"source_id":"S100","target_id":"S098","kind":"supports"},{"source_id":"S100","target_id":"S102","kind":"supports"},{"source_id":"S101","target_id":"S098","kind":"supports"},{"source_id":"S101","target_id":"S102","kind":"supports"},{"source_id":"S102","target_id":"S098","kind":"supports"},{"source_id":"S103","target_id":"S030","kind":"supports"},{"source_id":"S104","target_id":"S103","kind":"supports"},{"source_id":"S105","target_id":"S104","kind":"supports"},{"source_id":"S105","target_id":"S106","kind":"supports"},{"source_id":"S105","target_id":"S107","kind":"supports"},{"source_id":"S106","target_id":"S104","kind":"supports"},{"source_id":"S106","target_id":"S107","kind":"supports"},{"source_id":"S107","target_id":"S104","kind":"supports"},{"source_id":"S108","target_id":"S002","kind":"supports"},{"source_id":"S108","target_id":"S079","kind":"supports"},{"source_id":"S108","target_id":"S089","kind":"supports"},{"source_id":"S077","target_id":"S109","kind":"supports"},{"source_id":"S028","target_id":"S109","kind":"supports"},{"source_id":"S109","target_id":"S002","kind":"supports"},{"source_id":"S110","target_id":"S002","kind":"supports"},{"source_id":"S110","target_id":"S102","kind":"supports"},{"source_id":"S111","target_id":"S002","kind":"supports"},{"source_id":"S112","target_id":"S002","kind":"supports"},{"source_id":"S113","target_id":"S002","kind":"supports"},{"source_id":"S114","target_id":"S002","kind":"supports"},{"source_id":"S114","target_id":"S099","kind":"supports"},{"source_id":"S115","target_id":"S002","kind":"supports"},{"source_id":"S116","target_id":"S015","kind":"supports"},{"source_id":"S117","target_id":"S016","kind":"supports"},{"source_id":"S118","target_id":"S063","kind":"supports"},{"source_id":"S119","target_id":"S071","kind":"supports"},{"source_id":"S120","target_id":"S019","kind":"supports"},{"source_id":"S120","target_id":"S020","kind":"supports"},{"source_id":"S120","target_id":"S021","kind":"supports"},{"source_id":"S120","target_id":"S030","kind":"supports"},{"source_id":"S120","target_id":"S032","kind":"supports"},{"source_id":"S120","target_id":"S033","kind":"supports"},{"source_id":"S120","target_id":"S041","kind":"supports"},{"source_id":"S120","target_id":"S045","kind":"supports"},{"source_id":"S120","target_id":"S046","kind":"supports"},{"source_id":"S120","target_id":"S047","kind":"supports"},{"source_id":"S120","target_id":"S048","kind":"supports"},{"source_id":"S120","target_id":"S049","kind":"supports"},{"source_id":"S120","target_id":"S055","kind":"supports"},{"source_id":"S120","target_id":"S082","kind":"supports"},{"source_id":"S120","target_id":"S091","kind":"supports"},{"source_id":"S120","target_id":"S100","kind":"supports"},{"source_id":"S120","target_id":"S106","kind":"supports"},{"source_id":"S120","target_id":"S107","kind":"supports"},{"source_id":"S121","target_id":"S056","kind":"supports"},{"source_id":"S121","target_id":"S059","kind":"supports"},{"source_id":"S121","target_id":"S060","kind":"supports"},{"source_id":"S121","target_id":"S067","kind":"supports"},{"source_id":"S121","target_id":"S069","kind":"supports"},{"source_id":"S121","target_id":"S076","kind":"supports"},{"source_id":"S121","target_id":"S086","kind":"supports"},{"source_id":"S121","target_id":"S090","kind":"supports"},{"source_id":"S121","target_id":"S091","kind":"supports"},{"source_id":"S122","target_id":"S036","kind":"supports"},{"source_id":"S122","target_id":"S099","kind":"supports"},{"source_id":"S122","target_id":"S102","kind":"supports"},{"source_id":"S122","target_id":"S108","kind":"supports"}]}}
{"record_id":"2412.00585v4","paper":{"title":"Primal-dual proximal bundle and conditional gradient methods for convex problems","authors":["Jiaming Liang"],"arxiv_id":"2412.00585","version":"v4","journal":"Mathematical Programming","doi":"10.1007/s10107-025-02307-z","abstract_url":"https://arxiv.org/abs/2412.00585v4","pdf_url":"https://arxiv.org/pdf/2412.00585v4"},"context":{"raw_tex":"This paper considers two nonsmooth problems with convex structures: 1) the convex nonsmooth composite optimization (CNCO) problem\n\\begin{equation}\\label{eq:problem}\n    \\phi_*:=\\min \\{\\phi(x):=f(x)+h(x): x \\in \\R^n\\},\n\\end{equation}\nwhere $f,h:\\R^n\\to \\R \\cup \\{+\\infty\\}$ are proper lower semi-continuous convex functions such that $\\dom h \\subset \\dom f$;\n\n\n    Let $\\R$ denote the set of real numbers. Let $\\R_{++}$ denote the set of positive real numbers. Let $\\R^n$ denote the standard $n$-dimensional Euclidean space equipped with inner product and norm denoted by $\\left\\langle \\cdot,\\cdot\\right\\rangle $ and $\\|\\cdot\\|$, respectively. \n    \n    For given $f: \\R^n\\rightarrow (-\\infty,+\\infty]$, let $\\dom f:=\\{x \\in \\R^n: f (x) <\\infty\\}$ denote the effective domain of $f$. We say $f$ is proper if $\\dom f \\ne \\emptyset$. A proper function $f: \\R^n\\rightarrow (-\\infty,+\\infty]$ is $\\mu$-strongly convex for some $\\mu > 0$ if for every $x, y \\in \\dom f$ and $t \\in [0,1]$\n    \\[\n    f(t x+(1-t) y)\\le t f(x)+(1-t)f(y) - \\frac{t(1-t) \\mu}{2}\\|x-y\\|^2.\n    \\]\nLet $\\bConv{n}$ denote the set of all proper lower-semicontinuous convex functions.\n\tFor $\\varepsilon \\ge 0$, the \\emph{$\\varepsilon$-subdifferential} of $ f $ at $x \\in \\dom f$ is denoted by\n\t\\begin{equation}\\label{def:subdiff}\n\t    \\partial_\\varepsilon f (x):=\\left\\{ s \\in\\R^n: f(y)\\geq f(x)+\\left\\langle s,y-x\\right\\rangle -\\varepsilon, \\forall y\\in\\R^n\\right\\}.\n\t\\end{equation}\n\tWe denote the subdifferential of $f$ at $x \\in \\dom f$ by $\\partial f (x)$, which is the set  $\\partial_0 f(x)$ by definition.\n    For a given subgradient\n$f'(x) \\in \\partial f(x)$, we denote the linearization of convex function $f$ at $x$ by $\\ell_f(\\cdot;x)$, which is defined as\n\\begin{equation}\\label{linfdef}\n\\ell_f(\\cdot;x):=f(x)+\\langle f'(x),\\cdot-x\\rangle.\n\\end{equation}\nThe infimum convolution of proper functions $f_1, f_2: \\R^n\\rightarrow (-\\infty,+\\infty]$ is given by\n\\begin{equation}\\label{def:infconv}\n    (f_1 \\square f_2)(x)\n= \\min_{u\\in \\R^n}\\{f_1(u) + f_2(x-u)\\}.\n\\end{equation}\n\n\nIn this section, we consider the CNCO problem \\eqref{eq:problem}.\nMore specifically, we assume the following conditions hold:\n\\begin{itemize}\n    \\item[(A1)] a subgradient oracle, i.e., a function $f': \\dom h \\to \\R^n$ satisfying $f^{\\prime}(x) \\in \\partial f(x)$ for every $x \\in \\dom h$, is available;\n    \\item[(A2)] $\\|f'(x)\\|\\le M$ for every $x\\in \\dom h$ and some $M>0$;\n    \\item[(A3)] the set of optimal solutions $X_*$ of problem \\eqref{eq:problem} is nonempty.\n\\end{itemize}\nDefine the linearization of $f$ at $x\\in \\dom h$, $\\ell_f:\\dom h \\to \\R$ as\n\\[\n\\ell_f(\\cdot;x):=f(x)+\\inner{f'(x)}{\\cdot-x}.\n\\]\nClearly, it follows from (A2) that for every $x, y \\in \\dom h$,\n\\begin{equation}\\label{ineq:f}\n  f(x)-\\ell_f(x ; y) \\le 2 M\\|x-y\\|.  \n\\end{equation}\nFor a given initial point $\\hat x_0 \\in \\dom h$, we denote its distance to $X_*$ as\n\\begin{equation}\\label{eq:d0}\n    d_0:=\\left\\|x_0^*-\\hat x_0\\right\\|, \\quad x_0^*:=\\underset{x_* \\in X_*}\\argmin\\{\\left\\| x_*-\\hat x_0\\right\\|\\}.\n\\end{equation}\n\n\nThe primal-dual subgradient method denoted by PDS$(\\hat x_0,\\lam)$, where $\\hat x_0\\in \\dom h$ is the initial point and $\\lam>0$ is the prox stepsize, recursively computes\n\\begin{equation}\\label{eq:pds}\n    s_k = f'(x_{k-1}) \\in \\partial f(x_{k-1}), \\quad \\hat x_k = \\underset{u\\in\\R^n}{\\argmin}\\left\\{\\ell_f(u;\\hat x_{k-1}) + h(u) + \\frac{1}{2\\lam}\\|u-\\hat x_{k-1}\\|^2 \\right\\}.\n\\end{equation}\n\n\nThe PDPB method solves a sequence of proximal subproblems of the form\n\\begin{equation}\\label{eq:phi-lam}\n    \\min_{u \\in \\R^n} \\left\\{\\phi^\\lam(u) :=\\phi(u)+\\frac{1}{2 \\lambda}\\left\\|u-\\hat x_{k-1}\\right\\|^2\\right\\},\n\\end{equation}\nwhere $\\lam$ is the prox stepsize and $\\hat x_{k-1}$ is the prox center in the $k$-th proximal subproblem (or cycle). We omit the index $k$ in $\\phi^\\lam$ since the prox center is always fixed to be $\\hat x_{k-1}$ in this subsection.\nEach proximal subproblem invokes the PDCP scheme to find an approximate solution. Hence, PDPB can be viewed as a generalization of PDS, which only takes one proximal subgradient step (i.e., \\eqref{eq:pds}) to solve every proximal subproblem \\eqref{eq:phi-lam}.\nThe goal of this subsection is to describe the key subroutine PDCP for solving \\eqref{eq:phi-lam} and present its primal-dual convergence analysis.\n\n\nIn the rest of this subsection, we consider subproblem \\eqref{eq:phi-lam} with fixed prox center $\\hat x_{k-1}$. For simplicity, we denote $\\hat x_{k-1}$ as $x_0$ from a local perspective within the current cycle, as it is also the initial point of PDCP.  At the $j$-th iteration of PDCP, given some prox stepsize $\\lam>0$ and prox center $x_0$, PDCP computes a primal-dual pair $(x_j,s_j)$ as follows\n\\begin{equation}\\label{eq:xj}\n    x_j=\\underset{u \\in \\R^n}{\\argmin}\\left\\{\\Gamma_j(u) + h(u) +\\frac{1}{2 \\lam}\\|u-x_0\\|^2\\right\\}, \\quad s_j \\in \\partial \\Gamma_j(x_j) \\cap (-\\partial h^\\lam(x_j)),  \n\\end{equation}\nwhere $\\Gamma_j$ is a proper, closed and convex function satisfying $\\Gamma_j \\le f$ for every $j\\ge 1$, and\n\\begin{equation}\\label{eq:hlam}\n    h^\\lam (\\cdot) := h(\\cdot) + \\frac{1}{2\\lam}\\|\\cdot - x_0\\|^2.\n\\end{equation}\nStarting from $\\Gamma_1(\\cdot) = \\ell_f(\\cdot ; x_0)$, and for every $j\\ge 1$,  $\\Gamma_{j+1}$ is obtained from the following generic bundle management (GBM), which is motivated by BU given in Subsection~3.1 of \\cite{liang2024unified}.\n\n\n\\begin{algorithm}[H]\n\\caption{Generic Bundle Management, GBM$(\\lambda, \\tau_j, x_0, x_{j}, \\Gamma_j)$}\\label{alg:GBM}\n\\begin{algorithmic}\n\\REQUIRE $(\\lam, \\tau_j)\\in \\R_{++} \\times [0,1]$, $(x_0,x_j) \\in \\R^n \\times \\R^n$, and $\\Gamma_j \\in \\bConv{n}$ satisfying $\\Gamma_j \\le f$\n\n\\STATE $\\bullet$ find a bundle model $\\Gamma_{j+1}\\in \\bConv{n}$ satisfying $\\Gamma_{j+1} \\le f$ and \n        \\begin{equation}\\label{def:Gammaj}\n\t     \\Gamma_{j+1}(\\cdot) \\ge \\tau_j \\bar \\Gamma_j(\\cdot) + (1-\\tau_j)\\ell_f(\\cdot; x_j),\n\t\\end{equation}\nwhere $\\bar \\Gamma_j \\in \\bConv{n}$ satisfies $\\bar \\Gamma_j \\le f$ and\n        \\begin{equation}\\label{def:bar Gamma}\n            \\bar \\Gamma_j(x_j) = \\Gamma_j(x_j), \\quad x_j = \\underset{u\\in \\R^n}\\argmin \\left \\{\\bar \\Gamma_j(u) + h(u) + \\frac{1}{2\\lam}\\|u-x_0\\|^2\\right\\}.\n        \\end{equation}\n\n\\ENSURE $\\Gamma_{j+1}$.\n\n\\end{algorithmic} \n\\end{algorithm}\n\nPDCP computes an auxiliary sequence $\\{\\tx_j\\}$ to determine termination. \nIt generated $\\tx_j$ such that\n\\begin{equation}\\label{ineq:txj}\n    \\tilde x_1 = x_1, \\quad \\mbox{and} \\quad \\phi^\\lam(\\tx_{j+1}) \\le \\tau_j\\phi^\\lam(\\tx_j) + (1-\\tau_j)\\phi^\\lam(x_{j+1}), \\, \\forall j\\ge 1,\n\\end{equation}\nwhere $\\phi^\\lam$ is as in \\eqref{eq:phi-lam}. \nPDCP also computes\n\\begin{equation}\\label{eq:mj}\n    m_j =\\underset{u \\in \\R^n}{\\min}\\left\\{\\Gamma_j(u) + h(u) +\\frac{1}{2 \\lam}\\|u-x_0\\|^2\\right\\}, \\quad t_j=\\phi^\\lam \\left(\\tilde x_j\\right)-m_j.\n\\end{equation}\nFor given tolerance $\\varepsilon>0$, PDCP terminates the current cycle when $t_j \\le \\varepsilon$.\n\n\n\\begin{algorithm}[H]\n\\caption{Primal-Dual Cutting-Plane, PDCP$(x_0,\\lam,\\varepsilon)$}\\label{alg:PDCP}\n\\begin{algorithmic}\n\\REQUIRE given $x_0\\in \\dom h$, $\\lam>0$, $\\varepsilon>0$, set $t_0 = 2\\varepsilon$, $\\Gamma_1(\\cdot)=\\ell_f(\\cdot;x_0)$, and $j=1$;\n\n\\WHILE{$t_{j-1} > \\varepsilon$}\n\n\\STATE {\\bf 1.} compute $(x_j,s_j)$ by \\eqref{eq:xj}, choose $\\tilde x_j$ as in \\eqref{ineq:txj}, and set $t_j$ as in~\\eqref{eq:mj};\n\\STATE {\\bf 2.} select $\\tau_j \\in [0,1]$ and update $\\Gamma_{j+1}$ by GBM$(\\lambda, \\tau_j, x_0, x_{j}, \\Gamma_j)$ and $j \\leftarrow j+1$;\n\n\\ENDWHILE\n\\ENSURE $(x_{j-1}, \\tilde x_{j-1},  s_{j-1})$.\n\n\\end{algorithmic} \n\\end{algorithm}\n\nRecall the definitions of $d_0$ and $x_0^*$ in \\eqref{eq:d0}. Since $x_0^* \\in B(\\hat x_0, 6d_0)$, which is the ball centered at $\\hat x_0$ and with radius $6d_0$, it is easy to see that to solve \\eqref{eq:problem}, it suffices to solve\n\\begin{equation}\\label{eq:new}\n    \\min \\left\\{\\hat \\phi(x):=f(x)+\\hat h(x): x \\in \\R^n\\right\\}\n    = \\min \\left\\{\\phi(x): x \\in Q\\right\\},\n\\end{equation}\nwhere $\\hat h = h + I_Q$ and $I_Q$ is the indicator function of $Q = B(\\hat x_0, 6d_0)$.\n\n\n\\begin{algorithm}[H]\n\\caption{Primal-Dual Proximal Bundle, PDPB$(\\hat x_0,\\lam,\\bar \\varepsilon)$}\\label{alg:PDPB}\n\\begin{algorithmic}\n\\REQUIRE given $(\\hat x_0,\\lam,\\bar \\varepsilon) \\in \\dom h \\times \\R_{++} \\times \\R_{++}$\n\\FOR{$k=1,2,\\cdots$}\n\\STATE $\\bullet$ call oracle $(\\hat x_k,\\tx_k,s_k) = \\text{PDCP}(\\hat x_{k-1},\\lam, \\bar \\varepsilon)$ and compute \n\\begin{equation}\\label{eq:bxk}\n    \\bar x_k = \\frac{1}{k}\\sum_{i=1}^{k} \\tx_i , \\quad \\bar s_k = \\frac{1}{k}\\sum_{i=1}^{k} s_i.\n\\end{equation}\n\\ENDFOR\n\\end{algorithmic} \n\\end{algorithm}\n\nRecall from Subsection~\\ref{subsec:PDCP} that \\eqref{eq:phi-lam} is approximately solved by invoking PDCP. The (global) iteration indices in PDCP are regarded as the $k$-th cycle, denoted by ${\\cal C}_k=\\left\\{i_k, \\ldots, j_k\\right\\}$, where $j_k$ is the last iteration index of the $k$-th call to PDCP, $j_0=0$, and $i_k = j_{k-1}+1$.\nHence, for the $j_k$-th iteration of PDCP, we have\n\\begin{equation}\\label{eq:translate}\n    \\hat x_k = x_{j_k}, \\quad \\tilde x_k = \\tilde x_{j_k}, \\quad s_k = s_{j_k}, \\quad \\Gamma_k  = \\Gamma_{j_k}, \\quad m_k = m_{j_k},\n\\end{equation}\nand \\eqref{eq:xj} becomes\n\\begin{equation}\\label{eq:xk}\n    \\hat x_k=\\underset{u \\in \\R^n}{\\argmin}\\left\\{\\Gamma_k(u) + h(u) +\\frac{1}{2 \\lambda}\\left\\|u-\\hat x_{k-1}\\right\\|^2\\right\\}, \\quad s_k \\in \\partial \\Gamma_k(\\hat x_k) \\cap (-\\partial h^\\lam(\\hat x_k)).\n\\end{equation}\n\n\nRecall the definitions of $d_0$ and $x_0^*$ in \\eqref{eq:d0}. Since $x_0^* \\in B(\\hat x_0, 4d_0)$, which is the ball centered at $\\hat x_0$ and with radius $4d_0$, it is easy to see that to solve \\eqref{eq:problem}, it suffices to solve\n\\begin{equation}\\label{eq:constrained}\n    \\min \\left\\{\\hat \\phi(x):=f(x)+\\hat h(x): x \\in \\R^n\\right\\}\n    = \\min \\left\\{\\phi(x): x \\in Q\\right\\},\n\\end{equation}\nwhere $\\hat h = h + I_Q$ and $I_Q$ is the indicator function of $Q = B(\\hat x_0, 4d_0)$.\nHence, it is convenient to consider a slightly modified version of PDS$(\\hat x_0,\\lam)$ with $h$ replaced by $\\hat h$ in \\eqref{eq:pds}, denoted by MPDS$(\\hat x_0,\\lam)$, i.e.,\n\\begin{equation}\\label{eq:pds-1}\n    s_k = f'(\\hat x_{k-1}), \\quad \\hat x_k = \\underset{u\\in\\R^n}{\\argmin}\\left\\{\\ell_f(u;\\hat x_{k-1}) + \\hat h(u) + \\frac{1}{2\\lam}\\|u-\\hat x_{k-1}\\|^2 \\right\\}.\n\\end{equation}\n","units":[{"start":9135,"end":9493,"start_line":243,"end_line":247,"raw_tex":"This paper considers two nonsmooth problems with convex structures: 1) the convex nonsmooth composite optimization (CNCO) problem\n\\begin{equation}\\label{eq:problem}\n    \\phi_*:=\\min \\{\\phi(x):=f(x)+h(x): x \\in \\R^n\\},\n\\end{equation}\nwhere $f,h:\\R^n\\to \\R \\cup \\{+\\infty\\}$ are proper lower semi-continuous convex functions such that $\\dom h \\subset \\dom f$;\n","sha256_utf8":"61b250a5db4e75a1ac99e9e0ac17d60bea27a7360b2bc3045a8dfc4f86ac15d3"},{"start":19878,"end":21569,"start_line":314,"end_line":335,"raw_tex":"    Let $\\R$ denote the set of real numbers. Let $\\R_{++}$ denote the set of positive real numbers. Let $\\R^n$ denote the standard $n$-dimensional Euclidean space equipped with inner product and norm denoted by $\\left\\langle \\cdot,\\cdot\\right\\rangle $ and $\\|\\cdot\\|$, respectively. \n    \n    For given $f: \\R^n\\rightarrow (-\\infty,+\\infty]$, let $\\dom f:=\\{x \\in \\R^n: f (x) <\\infty\\}$ denote the effective domain of $f$. We say $f$ is proper if $\\dom f \\ne \\emptyset$. A proper function $f: \\R^n\\rightarrow (-\\infty,+\\infty]$ is $\\mu$-strongly convex for some $\\mu > 0$ if for every $x, y \\in \\dom f$ and $t \\in [0,1]$\n    \\[\n    f(t x+(1-t) y)\\le t f(x)+(1-t)f(y) - \\frac{t(1-t) \\mu}{2}\\|x-y\\|^2.\n    \\]\nLet $\\bConv{n}$ denote the set of all proper lower-semicontinuous convex functions.\n\tFor $\\varepsilon \\ge 0$, the \\emph{$\\varepsilon$-subdifferential} of $ f $ at $x \\in \\dom f$ is denoted by\n\t\\begin{equation}\\label{def:subdiff}\n\t    \\partial_\\varepsilon f (x):=\\left\\{ s \\in\\R^n: f(y)\\geq f(x)+\\left\\langle s,y-x\\right\\rangle -\\varepsilon, \\forall y\\in\\R^n\\right\\}.\n\t\\end{equation}\n\tWe denote the subdifferential of $f$ at $x \\in \\dom f$ by $\\partial f (x)$, which is the set  $\\partial_0 f(x)$ by definition.\n    For a given subgradient\n$f'(x) \\in \\partial f(x)$, we denote the linearization of convex function $f$ at $x$ by $\\ell_f(\\cdot;x)$, which is defined as\n\\begin{equation}\\label{linfdef}\n\\ell_f(\\cdot;x):=f(x)+\\langle f'(x),\\cdot-x\\rangle.\n\\end{equation}\nThe infimum convolution of proper functions $f_1, f_2: \\R^n\\rightarrow (-\\infty,+\\infty]$ is given by\n\\begin{equation}\\label{def:infconv}\n    (f_1 \\square f_2)(x)\n= \\min_{u\\in \\R^n}\\{f_1(u) + f_2(x-u)\\}.\n\\end{equation}\n","sha256_utf8":"25d858fb9fcf9903e73f598f9af536672e2dbb5d932398ba7f424c65c79c826f"},{"start":21642,"end":22649,"start_line":339,"end_line":357,"raw_tex":"In this section, we consider the CNCO problem \\eqref{eq:problem}.\nMore specifically, we assume the following conditions hold:\n\\begin{itemize}\n    \\item[(A1)] a subgradient oracle, i.e., a function $f': \\dom h \\to \\R^n$ satisfying $f^{\\prime}(x) \\in \\partial f(x)$ for every $x \\in \\dom h$, is available;\n    \\item[(A2)] $\\|f'(x)\\|\\le M$ for every $x\\in \\dom h$ and some $M>0$;\n    \\item[(A3)] the set of optimal solutions $X_*$ of problem \\eqref{eq:problem} is nonempty.\n\\end{itemize}\nDefine the linearization of $f$ at $x\\in \\dom h$, $\\ell_f:\\dom h \\to \\R$ as\n\\[\n\\ell_f(\\cdot;x):=f(x)+\\inner{f'(x)}{\\cdot-x}.\n\\]\nClearly, it follows from (A2) that for every $x, y \\in \\dom h$,\n\\begin{equation}\\label{ineq:f}\n  f(x)-\\ell_f(x ; y) \\le 2 M\\|x-y\\|.  \n\\end{equation}\nFor a given initial point $\\hat x_0 \\in \\dom h$, we denote its distance to $X_*$ as\n\\begin{equation}\\label{eq:d0}\n    d_0:=\\left\\|x_0^*-\\hat x_0\\right\\|, \\quad x_0^*:=\\underset{x_* \\in X_*}\\argmin\\{\\left\\| x_*-\\hat x_0\\right\\|\\}.\n\\end{equation}\n","sha256_utf8":"484c5dd18b439be0c2b65f7b2334b4f69c34ebff476482036d793d3f6ee0c18e"},{"start":22650,"end":23045,"start_line":359,"end_line":362,"raw_tex":"The primal-dual subgradient method denoted by PDS$(\\hat x_0,\\lam)$, where $\\hat x_0\\in \\dom h$ is the initial point and $\\lam>0$ is the prox stepsize, recursively computes\n\\begin{equation}\\label{eq:pds}\n    s_k = f'(x_{k-1}) \\in \\partial f(x_{k-1}), \\quad \\hat x_k = \\underset{u\\in\\R^n}{\\argmin}\\left\\{\\ell_f(u;\\hat x_{k-1}) + h(u) + \\frac{1}{2\\lam}\\|u-\\hat x_{k-1}\\|^2 \\right\\}.\n\\end{equation}\n","sha256_utf8":"b1e77eb6f837dbb486e5418f07b3b91458eab3db4853401e4faa42b5be85b882"},{"start":23467,"end":24342,"start_line":369,"end_line":375,"raw_tex":"The PDPB method solves a sequence of proximal subproblems of the form\n\\begin{equation}\\label{eq:phi-lam}\n    \\min_{u \\in \\R^n} \\left\\{\\phi^\\lam(u) :=\\phi(u)+\\frac{1}{2 \\lambda}\\left\\|u-\\hat x_{k-1}\\right\\|^2\\right\\},\n\\end{equation}\nwhere $\\lam$ is the prox stepsize and $\\hat x_{k-1}$ is the prox center in the $k$-th proximal subproblem (or cycle). We omit the index $k$ in $\\phi^\\lam$ since the prox center is always fixed to be $\\hat x_{k-1}$ in this subsection.\nEach proximal subproblem invokes the PDCP scheme to find an approximate solution. Hence, PDPB can be viewed as a generalization of PDS, which only takes one proximal subgradient step (i.e., \\eqref{eq:pds}) to solve every proximal subproblem \\eqref{eq:phi-lam}.\nThe goal of this subsection is to describe the key subroutine PDCP for solving \\eqref{eq:phi-lam} and present its primal-dual convergence analysis.\n","sha256_utf8":"1c7de1c327e3f3386f4d8bfb4d2a7af21de4182f1d6ea3ea79714480757b853f"},{"start":24343,"end":25420,"start_line":377,"end_line":385,"raw_tex":"In the rest of this subsection, we consider subproblem \\eqref{eq:phi-lam} with fixed prox center $\\hat x_{k-1}$. For simplicity, we denote $\\hat x_{k-1}$ as $x_0$ from a local perspective within the current cycle, as it is also the initial point of PDCP.  At the $j$-th iteration of PDCP, given some prox stepsize $\\lam>0$ and prox center $x_0$, PDCP computes a primal-dual pair $(x_j,s_j)$ as follows\n\\begin{equation}\\label{eq:xj}\n    x_j=\\underset{u \\in \\R^n}{\\argmin}\\left\\{\\Gamma_j(u) + h(u) +\\frac{1}{2 \\lam}\\|u-x_0\\|^2\\right\\}, \\quad s_j \\in \\partial \\Gamma_j(x_j) \\cap (-\\partial h^\\lam(x_j)),  \n\\end{equation}\nwhere $\\Gamma_j$ is a proper, closed and convex function satisfying $\\Gamma_j \\le f$ for every $j\\ge 1$, and\n\\begin{equation}\\label{eq:hlam}\n    h^\\lam (\\cdot) := h(\\cdot) + \\frac{1}{2\\lam}\\|\\cdot - x_0\\|^2.\n\\end{equation}\nStarting from $\\Gamma_1(\\cdot) = \\ell_f(\\cdot ; x_0)$, and for every $j\\ge 1$,  $\\Gamma_{j+1}$ is obtained from the following generic bundle management (GBM), which is motivated by BU given in Subsection~3.1 of \\cite{liang2024unified}.\n","sha256_utf8":"d23530640d26259c4bc88b1eb737f5384318e0031b52e09f9fbebe048784e157"},{"start":25753,"end":26644,"start_line":389,"end_line":406,"raw_tex":"\\begin{algorithm}[H]\n\\caption{Generic Bundle Management, GBM$(\\lambda, \\tau_j, x_0, x_{j}, \\Gamma_j)$}\\label{alg:GBM}\n\\begin{algorithmic}\n\\REQUIRE $(\\lam, \\tau_j)\\in \\R_{++} \\times [0,1]$, $(x_0,x_j) \\in \\R^n \\times \\R^n$, and $\\Gamma_j \\in \\bConv{n}$ satisfying $\\Gamma_j \\le f$\n\n\\STATE $\\bullet$ find a bundle model $\\Gamma_{j+1}\\in \\bConv{n}$ satisfying $\\Gamma_{j+1} \\le f$ and \n        \\begin{equation}\\label{def:Gammaj}\n\t     \\Gamma_{j+1}(\\cdot) \\ge \\tau_j \\bar \\Gamma_j(\\cdot) + (1-\\tau_j)\\ell_f(\\cdot; x_j),\n\t\\end{equation}\nwhere $\\bar \\Gamma_j \\in \\bConv{n}$ satisfies $\\bar \\Gamma_j \\le f$ and\n        \\begin{equation}\\label{def:bar Gamma}\n            \\bar \\Gamma_j(x_j) = \\Gamma_j(x_j), \\quad x_j = \\underset{u\\in \\R^n}\\argmin \\left \\{\\bar \\Gamma_j(u) + h(u) + \\frac{1}{2\\lam}\\|u-x_0\\|^2\\right\\}.\n        \\end{equation}\n\n\\ENSURE $\\Gamma_{j+1}$.\n\n\\end{algorithmic} \n\\end{algorithm}","sha256_utf8":"91bb65095751239966d9b36f80d0385b31a82186ffde7f8bc391b12c46c46bae"},{"start":26649,"end":27307,"start_line":409,"end_line":419,"raw_tex":"PDCP computes an auxiliary sequence $\\{\\tx_j\\}$ to determine termination. \nIt generated $\\tx_j$ such that\n\\begin{equation}\\label{ineq:txj}\n    \\tilde x_1 = x_1, \\quad \\mbox{and} \\quad \\phi^\\lam(\\tx_{j+1}) \\le \\tau_j\\phi^\\lam(\\tx_j) + (1-\\tau_j)\\phi^\\lam(x_{j+1}), \\, \\forall j\\ge 1,\n\\end{equation}\nwhere $\\phi^\\lam$ is as in \\eqref{eq:phi-lam}. \nPDCP also computes\n\\begin{equation}\\label{eq:mj}\n    m_j =\\underset{u \\in \\R^n}{\\min}\\left\\{\\Gamma_j(u) + h(u) +\\frac{1}{2 \\lam}\\|u-x_0\\|^2\\right\\}, \\quad t_j=\\phi^\\lam \\left(\\tilde x_j\\right)-m_j.\n\\end{equation}\nFor given tolerance $\\varepsilon>0$, PDCP terminates the current cycle when $t_j \\le \\varepsilon$.\n","sha256_utf8":"38039996b3b201d36fc6367c8ee370dc6c7ddf7df1d3bac05d687a237191ff53"},{"start":27341,"end":27997,"start_line":424,"end_line":438,"raw_tex":"\\begin{algorithm}[H]\n\\caption{Primal-Dual Cutting-Plane, PDCP$(x_0,\\lam,\\varepsilon)$}\\label{alg:PDCP}\n\\begin{algorithmic}\n\\REQUIRE given $x_0\\in \\dom h$, $\\lam>0$, $\\varepsilon>0$, set $t_0 = 2\\varepsilon$, $\\Gamma_1(\\cdot)=\\ell_f(\\cdot;x_0)$, and $j=1$;\n\n\\WHILE{$t_{j-1} > \\varepsilon$}\n\n\\STATE {\\bf 1.} compute $(x_j,s_j)$ by \\eqref{eq:xj}, choose $\\tilde x_j$ as in \\eqref{ineq:txj}, and set $t_j$ as in~\\eqref{eq:mj};\n\\STATE {\\bf 2.} select $\\tau_j \\in [0,1]$ and update $\\Gamma_{j+1}$ by GBM$(\\lambda, \\tau_j, x_0, x_{j}, \\Gamma_j)$ and $j \\leftarrow j+1$;\n\n\\ENDWHILE\n\\ENSURE $(x_{j-1}, \\tilde x_{j-1},  s_{j-1})$.\n\n\\end{algorithmic} \n\\end{algorithm}","sha256_utf8":"57666bad01f247e616e660bf486cd23fa3ed86465d47822881eac41c69e6b4f7"},{"start":34089,"end":34570,"start_line":551,"end_line":556,"raw_tex":"Recall the definitions of $d_0$ and $x_0^*$ in \\eqref{eq:d0}. Since $x_0^* \\in B(\\hat x_0, 6d_0)$, which is the ball centered at $\\hat x_0$ and with radius $6d_0$, it is easy to see that to solve \\eqref{eq:problem}, it suffices to solve\n\\begin{equation}\\label{eq:new}\n    \\min \\left\\{\\hat \\phi(x):=f(x)+\\hat h(x): x \\in \\R^n\\right\\}\n    = \\min \\left\\{\\phi(x): x \\in Q\\right\\},\n\\end{equation}\nwhere $\\hat h = h + I_Q$ and $I_Q$ is the indicator function of $Q = B(\\hat x_0, 6d_0)$.\n","sha256_utf8":"a3f879e02e77a91c8ab95b914787c86257addc5a0a7bd2946c4a4fe15f1da4f5"},{"start":34734,"end":35282,"start_line":561,"end_line":572,"raw_tex":"\\begin{algorithm}[H]\n\\caption{Primal-Dual Proximal Bundle, PDPB$(\\hat x_0,\\lam,\\bar \\varepsilon)$}\\label{alg:PDPB}\n\\begin{algorithmic}\n\\REQUIRE given $(\\hat x_0,\\lam,\\bar \\varepsilon) \\in \\dom h \\times \\R_{++} \\times \\R_{++}$\n\\FOR{$k=1,2,\\cdots$}\n\\STATE $\\bullet$ call oracle $(\\hat x_k,\\tx_k,s_k) = \\text{PDCP}(\\hat x_{k-1},\\lam, \\bar \\varepsilon)$ and compute \n\\begin{equation}\\label{eq:bxk}\n    \\bar x_k = \\frac{1}{k}\\sum_{i=1}^{k} \\tx_i , \\quad \\bar s_k = \\frac{1}{k}\\sum_{i=1}^{k} s_i.\n\\end{equation}\n\\ENDFOR\n\\end{algorithmic} \n\\end{algorithm}","sha256_utf8":"4fd77d6297dc54993b06c31d23002fee37c45cc8d86350331aebbbd80d184dd3"},{"start":35847,"end":36698,"start_line":581,"end_line":589,"raw_tex":"Recall from Subsection~\\ref{subsec:PDCP} that \\eqref{eq:phi-lam} is approximately solved by invoking PDCP. The (global) iteration indices in PDCP are regarded as the $k$-th cycle, denoted by ${\\cal C}_k=\\left\\{i_k, \\ldots, j_k\\right\\}$, where $j_k$ is the last iteration index of the $k$-th call to PDCP, $j_0=0$, and $i_k = j_{k-1}+1$.\nHence, for the $j_k$-th iteration of PDCP, we have\n\\begin{equation}\\label{eq:translate}\n    \\hat x_k = x_{j_k}, \\quad \\tilde x_k = \\tilde x_{j_k}, \\quad s_k = s_{j_k}, \\quad \\Gamma_k  = \\Gamma_{j_k}, \\quad m_k = m_{j_k},\n\\end{equation}\nand \\eqref{eq:xj} becomes\n\\begin{equation}\\label{eq:xk}\n    \\hat x_k=\\underset{u \\in \\R^n}{\\argmin}\\left\\{\\Gamma_k(u) + h(u) +\\frac{1}{2 \\lambda}\\left\\|u-\\hat x_{k-1}\\right\\|^2\\right\\}, \\quad s_k \\in \\partial \\Gamma_k(\\hat x_k) \\cap (-\\partial h^\\lam(\\hat x_k)).\n\\end{equation}\n","sha256_utf8":"a342af5e590c8cb9a0bbc130f38c56faf85598f4e27a5a4c97f87ac31b7a5052"},{"start":122385,"end":123262,"start_line":1906,"end_line":1915,"raw_tex":"Recall the definitions of $d_0$ and $x_0^*$ in \\eqref{eq:d0}. Since $x_0^* \\in B(\\hat x_0, 4d_0)$, which is the ball centered at $\\hat x_0$ and with radius $4d_0$, it is easy to see that to solve \\eqref{eq:problem}, it suffices to solve\n\\begin{equation}\\label{eq:constrained}\n    \\min \\left\\{\\hat \\phi(x):=f(x)+\\hat h(x): x \\in \\R^n\\right\\}\n    = \\min \\left\\{\\phi(x): x \\in Q\\right\\},\n\\end{equation}\nwhere $\\hat h = h + I_Q$ and $I_Q$ is the indicator function of $Q = B(\\hat x_0, 4d_0)$.\nHence, it is convenient to consider a slightly modified version of PDS$(\\hat x_0,\\lam)$ with $h$ replaced by $\\hat h$ in \\eqref{eq:pds}, denoted by MPDS$(\\hat x_0,\\lam)$, i.e.,\n\\begin{equation}\\label{eq:pds-1}\n    s_k = f'(\\hat x_{k-1}), \\quad \\hat x_k = \\underset{u\\in\\R^n}{\\argmin}\\left\\{\\ell_f(u;\\hat x_{k-1}) + \\hat h(u) + \\frac{1}{2\\lam}\\|u-\\hat x_{k-1}\\|^2 \\right\\}.\n\\end{equation}\n","sha256_utf8":"dcd98a30357d68a7441e09a40a816c1a5d4bdc33f3e1bc76c0ff4cf0a608095c"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\n    Given $(\\hat x_0,\\bar \\varepsilon) \\in \\R^n \\times \\R_{++}$, assuming that $\\lam$ satisfies\n    \\begin{equation}\\label{ineq:lam}\n        \\frac{\\sqrt{\\bar \\varepsilon d_0}}{M^{3/2}} \\le \\lam \\le \\frac{2d_0^2}{\\bar \\varepsilon},\n    \\end{equation}\n    then the total iteration-complexity of PDPB$(\\hat x_0,\\lam,\\bar \\varepsilon)$ to find $(\\bar x_k,\\bar s_k)$ satisfying \\eqref{ineq:pd} is\n    \\begin{equation}\\label{cmplx:PDPB}\n        {\\cal O}\\left( \\frac{M^2 d_0^2}{\\bar \\varepsilon^2} + 1\\right).\n    \\end{equation}\n\\end{theorem}","units":[{"start":48311,"end":48862,"start_line":792,"end_line":801,"raw_tex":"\\begin{theorem}\n    Given $(\\hat x_0,\\bar \\varepsilon) \\in \\R^n \\times \\R_{++}$, assuming that $\\lam$ satisfies\n    \\begin{equation}\\label{ineq:lam}\n        \\frac{\\sqrt{\\bar \\varepsilon d_0}}{M^{3/2}} \\le \\lam \\le \\frac{2d_0^2}{\\bar \\varepsilon},\n    \\end{equation}\n    then the total iteration-complexity of PDPB$(\\hat x_0,\\lam,\\bar \\varepsilon)$ to find $(\\bar x_k,\\bar s_k)$ satisfying \\eqref{ineq:pd} is\n    \\begin{equation}\\label{cmplx:PDPB}\n        {\\cal O}\\left( \\frac{M^2 d_0^2}{\\bar \\varepsilon^2} + 1\\right).\n    \\end{equation}\n\\end{theorem}","sha256_utf8":"57b1afbb657153e5717090c82f51f0dd302474bb919aee4747268eb9679a6831"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\\label{lem:Gamma}\nFor every $j \\ge 1$, there exists $\\bar \\Gamma_j \\in \\bConv{n}$ such that for every $u\\in \\R^n$,\n\\begin{equation}\\label{ineq:bGammaj}\n    \\bar{\\Gamma}_j(u) + h^\\lam(u) \\ge m_j + \\frac{1}{2\\lam}\\|u-x_j\\|^2.\n\\end{equation}\n\\end{lemma}\n\n\\begin{proof}\n    Since the objective function in \\eqref{def:bar Gamma} is $\\lam^{-1}$-strongly convex, it follows from \\eqref{def:bar Gamma} that \n    \\[\n    \\bar \\Gamma_j(u) + h(u) + \\frac{1}{2\\lam}\\|u-x_0\\|^2 \\ge \\bar \\Gamma_j(x_j) + h(x_j) + \\frac{1}{2\\lam}\\|x_j-x_0\\|^2 + \\frac{1}{2\\lam}\\|u-x_j\\|^2.\n    \\]\n    Inequality \\eqref{ineq:bGammaj} immediately follows from the above inequality, the definition of $m_j$ in \\eqref{eq:mj}, and the fact that $h^\\lam(\\cdot) = h(\\cdot) + \\|\\cdot-x_0\\|^2/(2\\lam)$.\n\\end{proof}\n\n\\begin{proposition}\\label{prop:inner}\nConsidering Algorithm \\ref{alg:PDCP} with $\\tau_j = j/(j+2)$, then for every $j \\ge 1$, we have\n\\begin{equation}\\label{ineq:tj}\n    t_j\\le \\frac{2t_1}{j(j+1)} + \\frac{16\\lam M^2}{j+1},\n\\end{equation}\nwhere $t_j$ is as in \\eqref{eq:mj}.\nMoreover, the number of iterations for PDCP to obtain $t_j \\le \\varepsilon$ is at most\n\\[\n{\\cal O}\\left(\\frac{\\sqrt{t_1}}{\\sqrt{\\varepsilon}} + \\frac{\\lam M^2}{\\varepsilon} + 1\\right).\n\\]\n\\end{proposition}\n\n\\begin{proof}\nWe first note that for every $j\\ge 1$, $\\tau_j=A_j/A_{j+1}$ where $A_{j+1}=A_j+j+1$ and $A_0=0$, i.e., $A_j=j(j+1)/2$ for every $j\\ge 0$.\nIt follows from this observation, the definition of $m_j$ in \\eqref{eq:mj}, and relation \\eqref{def:Gammaj} that\n\\begin{align*}\n    A_{j+1}m_{j+1} & \\stackrel{\\eqref{eq:mj}}{=} A_{j+1}(\\Gamma_{j+1}+h^\\lam)(x_{j+1}) \\\\\n    &\\stackrel{\\eqref{def:Gammaj}}{\\ge} A_j \\left[(\\bar \\Gamma_j+ h^\\lam)(x_{j+1}) \\right] + (j+1)\\left[\\ell_f(x_{j+1};x_j) + h^\\lam(x_{j+1})\\right].\n\\end{align*}\nApplying Lemma~\\ref{lem:Gamma} in the above inequality and using \\eqref{ineq:f}, we have\n\\begin{align*}\n    A_{j+1}m_{j+1} \n    &\\stackrel{\\eqref{ineq:bGammaj}}{\\ge} A_j \\left[ m_j+\\frac{1}{2\\lam}\\|x_{j+1} - x_j\\|^2 \\right] + (j+1)\\left[\\ell_f(x_{j+1};x_j) + h^\\lam(x_{j+1})\\right] \\\\\n    &= A_j m_j + (j+1)\\left[\\ell_f(x_{j+1};x_j) + h^\\lam(x_{j+1}) + \\frac{A_j}{2\\lam (j+1)}\\|x_{j+1} - x_j\\|^2 \\right] \\\\\n    &\\stackrel{\\eqref{ineq:f}}{\\ge} A_j m_j + (j+1)\\left[\\phi^\\lam(x_{j+1}) - 2M\\|x_{j+1}-x_j\\| + \\frac{A_j}{2\\lam (j+1)}\\|x_{j+1} - x_j\\|^2 \\right] \\\\\n    &\\ge A_j m_j + (j+1)\\phi^\\lam(x_{j+1}) - \\frac{2\\lam M^2 (j+1)^2}{A_j}\n\\end{align*}\nwhere the last inequality is due to the Young's inequality $a^2+b^2\\ge 2ab$.\nIt follows from the fact that $A_j=j(j+1)/2$ that for every $j\\ge 1$,\n\\[\n    A_{j+1}m_{j+1} \\ge A_j m_j + (j+1)\\phi^\\lam(x_{j+1}) - 8\\lam M^2.\n\\]\nReplacing the index $j$ in the above inequality by $i$, summing the resulting inequality from $i=1$ to $j-1$, and using the definition of $t_j$ in \\eqref{eq:mj} and the fact that $\\tx_1=x_1$, we obtain\n\\begin{align*}\n    A_j m_j\n    &\\ge A_1 m_1 + 2 \\phi^\\lam(x_2) + \\dots + j\\phi^\\lam(x_j) - 8\\lam M^2 (j-1) \\\\\n    &\\stackrel{\\eqref{eq:mj}}{=} -A_1 t_1 + A_1 \\phi^\\lam(x_1) + 2 \\phi^\\lam(x_2) + \\dots + j\\phi^\\lam(x_j) - 8\\lam M^2 (j-1) \\\\\n    &\\stackrel{\\eqref{ineq:txj}}{\\ge} -A_1 t_1 + A_j \\phi^\\lam(\\tx_j) - 8\\lam M^2 (j-1),\n\\end{align*}\nwhere the last inequality follows from \\eqref{ineq:txj} and the fact that $A_j=A_{j-1}+j$.\nRearranging the terms and using the definition of $t_j$ in \\eqref{eq:mj} again, we have\n\\begin{equation}\\label{ineq:Aj-tj}\n    A_j t_j \\le A_1 t_1 + 8\\lam M^2 (j-1).\n\\end{equation}\nHence, \\eqref{ineq:tj} follows from the fact that $A_j=j(j+1)/2$.\nFinally, the complexity bound immediately follows from \\eqref{ineq:tj}.\n\\end{proof}\n\n\\begin{lemma}\\label{ser-lemma1}\n    The following statements hold for every $k \\ge 1$:\n    \\begin{enumerate}\n        \\item [(a)] $\\Gamma_k \\le f$ and $f^*\\le \\Gamma_k^*$;\n        \\item [(b)] $s_k \\in \\partial \\Gamma_k(\\hat x_k)$ and $g_k \\in \\partial h(\\hat x_k)$ where $g_k = - s_k + (\\hat x_{k-1}-\\hat x_k)/\\lam$;\n        \\item [(c)] $\\phi^\\lam(\\tilde x_k) \\le \\bar \\varepsilon + m_k = \\bar \\varepsilon + (\\Gamma_k + h)(\\hat x_k) + \\|\\hat x_k - \\hat x_{k-1}\\|^2/(2\\lam)$.\n    \\end{enumerate}\n\\end{lemma}\n\n\\begin{proof}\n    (a) It follows from the facts that $\\Gamma_j \\le f$ for every $j\\ge 1$ and $\\Gamma_k=\\Gamma_{j_k}$ that the first inequality holds. The second one immediately follows from the first one and the definition of the conjugate function.\n\n    (b) This statement follows from \\eqref{eq:xk} and the definitions in \\eqref{eq:translate}.\n\n    (c) This statement follows from the termination criterion of the $k$-th cycle, that is, $t_{j_k}\\le \\bar \\varepsilon$, and the definitions in \\eqref{eq:mj} and \\eqref{eq:translate}.\n\\end{proof}\n\n\\begin{proposition}\\label{prop:pd-key}\nFor every $k \\ge 1$, we have\n\\begin{equation}\\label{ineq:conjugate}\n    \\phi(\\bar{x}_k) + f^*(\\bar s_k) + \\hat h ^*(-\\bar s_k) \\le \\bar \\varepsilon + \\frac{18 d_0^2}{\\lam k}.\n\\end{equation}\nwhere $\\bar x_k$ and $\\bar s_k$ are as in \\eqref{eq:bxk}.\n\\end{proposition}\n\n\\begin{proof}\nIt follows from Lemma \\ref{ser-lemma1}(b) and Theorem 4.20 of \\cite{Beck2017} that for every $k \\ge 1$,\n\\[ \n\\Gamma_k(\\hat x_k) +  \\Gamma_k^*( s_k) = \\inner{\\hat x_k}{s_k}, \\quad  h(\\hat x_k) +  h^*(g_k) = \\inner{\\hat x_k}{g_k}.\n\\]\nSumming the above two equations yields\n\\begin{equation}\\label{eq:Gamma-h}\n    (\\Gamma_k + h)(\\hat x_k) + {\\Gamma}_k^*(s_k) + h^*(g_k) = \\frac{1}{\\lam}\\inner{\\hat x_k}{\\hat x_{k-1}-\\hat x_k}.\n\\end{equation}\nUsing the above identity and Lemma \\ref{ser-lemma1}(a) and (c), we have for every $k \\ge 1$,\n\\begin{align*}\n& \\phi(\\tilde x_k) + f^*(s_k) +  {h}^*(g_k) \\le \\phi(\\tilde x_k) + \\Gamma_k^*(s_k) +  {h}^*(g_k) \\\\\n\\le & \\bar \\varepsilon +(\\Gamma_k + h)(\\hat x_k) + \\frac{1}{2\\lam}\\|\\hat x_k-\\hat x_{k-1}\\|^2 + \\Gamma_k^*(s_k) + {h}^*(g_k)\\\\\n\\stackrel{\\eqref{eq:Gamma-h}}{=}& \\bar \\varepsilon + \\frac{1}{2\\lam}(\\|\\hat x_{k-1}\\|^2-\\|\\hat x_k\\|^2).\n\\end{align*}\nReplacing the index $k$ in the above inequality by $i$, summing the resulting inequality from $i=1$ to $k$, and using convexity and the definitions in \\eqref{eq:bxk}, we obtain \n\\begin{equation}\\label{ineq:avg}\n    \\phi(\\bar x_k) + f^*(\\bar s_k) + h^*(\\bar g_k)\\le \\bar \\varepsilon + \\frac{1}{2\\lam k }\\left(\\|\\hat x_0\\|^2-\\|\\hat x_k\\|^2\\right),\n\\end{equation}\nwhere $\\bar g_k = (\\sum_{i=1}^k g_i)/k$.\nDefine\n\\begin{equation}\\label{eq:zeta}\n    \\eta_k(u) = \\frac{1}{2\\lam k} \\|u-\\hat x_0\\|^2, \\quad \\hat \\eta_k(u) = \\eta_k(u) - I_Q(u).\n\\end{equation}\nNoting that $\\nabla \\eta_k(\\hat x_k) = (\\hat x_k - \\hat x_0)/(\\lam k) = -\\bar g_k - \\bar s_k$, and hence it follows from Theorem 4.20 of \\cite{Beck2017} that\n\\[\n\\eta_k^*(-\\bar g_k - \\bar s_k) = \\frac{1}{\\lam k}\\inner{\\hat x_k-\\hat x_0}{\\hat x_k} - \\eta_k(\\hat x_k) = \\frac{1}{2\\lam k}\\left(\\|\\hat x_k\\|^2 - \\| \\hat x_0\\|^2\\right).\n\\]\nThe above observation and \\eqref{ineq:avg} together imply that\n\\begin{equation}\\label{ineq:bar}\n    \\phi(\\bar x_k) + f^*(\\bar s_k) + h^*(\\bar g_k) + \\eta_k^*(-\\bar g_k - \\bar s_k) \\le \\bar \\varepsilon.\n\\end{equation}\nIt follows from Theorem 4.17 of \\cite{Beck2017} and the definition of infimum convolution in \\eqref{def:infconv} that\n\\[\n(h + \\eta_k)^*(-\\bar s_k) = (h^* \\square \\eta_k^*)(-\\bar s_k)\n\\stackrel{\\eqref{def:infconv}}= \\min_{u\\in \\R^n}\\{h^*(u) + \\eta_k^*(-\\bar s_k-u)\\}\n\\le h^*(\\bar g_k) + \\eta_k^*(-\\bar g_k - \\bar s_k).\n\\]\nNoting from \\eqref{eq:zeta} that $\\hat h = h + \\eta_k -\\hat \\eta_k$ and applying Theorem 4.17 of \\cite{Beck2017} again, we obtain\n\\begin{align*}\n    \\hat h ^*(-\\bar s_k) &= [(h + \\eta_k)^* \\square (-\\hat \\eta_k)^*](-\\bar s_k)\n= \\min_{u\\in \\R^n}\\{(h + \\eta_k)^*(u) + (-\\hat \\eta_k)^*(-\\bar s_k-u)\\}\\\\\n&\\le (h + \\eta_k)^*(-\\bar s_k) + (-\\hat \\eta_k)^*(0).\n\\end{align*}\nSumming the above two inequalities, we have\n\\[\n\\hat h ^*(-\\bar s_k) \\le h^*(\\bar g_k) + \\eta_k^*(-\\bar g_k - \\bar s_k) + (-\\hat \\eta_k)^*(0),\n\\]\nwhich together with \\eqref{ineq:bar} implies that\n\\[\n\\phi(\\bar x_k) + f^*(\\bar s_k) + \\hat h ^*(-\\bar s_k) \\le \\bar \\varepsilon + (-\\hat \\eta_k)^*(0).\n\\]\nIt follows from \\eqref{eq:zeta} that\n\\[\n(-\\hat \\eta_k)^*(0) = \\max_{u \\in \\R^n}\\left\\{\\inner{0}{u} - \\left( -\\frac{\\|u-\\hat x_0\\|^2}{2\\lam k} +I_Q(u)\\right)\\right\\}\n=\\frac{\\max_{u \\in Q}\\|u-\\hat x_0\\|^2}{2\\lam k}\n= \\frac{18d_0^2}{\\lam k},\n\\]\nwhere the last identity follows from the fact that $Q = B(\\hat x_0, 6d_0)$.\nTherefore, \\eqref{ineq:conjugate} holds in view of the above two relations.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:bound}\n    Given $(\\hat x_0,\\bar \\varepsilon) \\in \\R^n \\times \\R_{++}$, if $\\lam \\le 2d_0^2/\\bar \\varepsilon$ and $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$, then the sequences $\\{\\hat x_k\\}$ and $\\{\\tx_k\\}$ generated by PDPB$(\\hat x_0,\\lam, \\bar \\varepsilon)$ satisfy\n    \\begin{equation}\\label{incl:xk}\n        \\hat x_k \\in Q, \\quad \\tx_k \\in Q.\n    \\end{equation}\n\\end{lemma}\n\n\\begin{proof}\n    Noticing that the objective function in \\eqref{eq:xk} is $\\lam^{-1}$-strongly convex, it thus follows from Theorem 5.25(b) of \\cite{Beck2017} that for every $u \\in \\dom h$,\n    \\begin{equation}\\label{ineq:key}\n        m_k + \\frac{1}{2\\lam} \\|u-\\hat x_{k}\\|^2 \\le \\Gamma_k(u) + h(u) + \\frac1{2\\lam}\\|u-\\hat x_{k-1}\\|^2 \\le \\phi(u)+ \\frac1{2\\lam}\\|u-\\hat x_{k-1}\\|^2, \n    \\end{equation}\n    where the second inequality follows from the first one in Lemma \\ref{ser-lemma1}(a).\n    Taking $u=x_0^*$ in \\eqref{ineq:key}, we have\n    \\[\n    m_k + \\frac{1}{2\\lam} \\|\\hat x_k-x_0^*\\|^2 \\le \\phi_* + \\frac1{2\\lam}\\|\\hat x_{k-1}-x_0^*\\|^2. \n    \\]\n    This inequality and Lemma \\ref{ser-lemma1}(c) then imply that\n    \\begin{align*}\n        \\frac{1}{2\\lam}\\|\\hat x_k - x_0^*\\|^2 &\\le \\phi(\\tx_k)- \\phi_* + \\frac{1}{2\\lam}\\|\\hat x_k - x_0^*\\|^2 \\\\\n        &\\le \\phi(\\tx_k) - m_k + \\frac{1}{2\\lam}\\|\\hat x_{k-1}-x_0^*\\|^2 \\le \\bar \\varepsilon + \\frac{1}{2\\lam}\\|\\hat x_{k-1}-x_0^*\\|^2.\n    \\end{align*}\n    Replacing the index $k$ in the above inequality by $i$ and summing the resulting inequality from $i=1$ to $k$, we have\n    \\[\n    \\|\\hat x_k - x_0^*\\|^2 \\le \\|\\hat x_0-x_0^*\\|^2 + 2 k \\lam \\bar \\varepsilon.\n    \\]\n    Using the fact that $\\sqrt{a+b} \\le \\sqrt{a} + \\sqrt{b}$ for $a,b \\ge 0$ and the assumption that $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$, we further obtain\n    \\begin{equation}\\label{ineq:xk}\n        \\|\\hat x_k - x_0^*\\| \\le d_0 + \\sqrt{2k\\lam \\bar \\varepsilon} \\le 3d_0.\n    \\end{equation}\n    Taking $u=\\tx_k$ in \\eqref{ineq:key} and using Lemma \\ref{ser-lemma1}(c), we have\n    \\[\n    \\frac{1}{2\\lam} \\|\\tx_k-\\hat x_k\\|^2 \\le \\phi(\\tx_k) + \\frac1{2\\lam}\\|\\tx_k - \\hat x_{k-1}\\|^2 - m_k \\le \\bar \\varepsilon.\n    \\]\n    Under the assumption that $\\lam \\le 2d_0^2/\\bar \\varepsilon$, using \\eqref{ineq:xk}, the above inequality, and the triangle inequality, we have\n    \\begin{align*}\n        \\|\\hat x_k - \\hat x_0\\| &\\le \\|\\hat x_k - x_0^*\\| + \\|x_0^* - \\hat x_0\\| \\stackrel{\\eqref{ineq:xk}}{\\le} 4d_0, \\\\\n        \\|\\tx_k - \\hat x_0\\| &\\le \\|\\hat x_k - \\hat x_0\\| + \\|\\tx_k - \\hat x_k\\| \\le 4d_0 + \\sqrt{2\\lam \\bar \\varepsilon} \\le 6d_0.\n    \\end{align*}\n    Hence, \\eqref{incl:xk} follows immediately.\n\\end{proof}\n\n\\begin{proposition}\\label{prop:pdpb}\n    Given $(\\hat x_0,\\bar \\varepsilon) \\in \\R^n \\times \\R_{++}$, if $\\lam \\le 2d_0^2/\\bar \\varepsilon$, then the number of iterations for PDPB$(\\hat x_0,\\lam, \\bar \\varepsilon)$ to generate $(\\bar x_k, \\bar s_k)$ satisfying \n    \\begin{equation}\\label{ineq:pd}\n        \\hat \\phi(\\bar x_k) + f^*(\\bar s_k) + \\hat h^*(-\\bar s_k) \\le 10\\bar \\varepsilon\n    \\end{equation}\n    is at most $2 d_0^2/(\\lam \\bar \\varepsilon)$.\n\\end{proposition}\n\n\\begin{proof}\n    Since $Q$ is a convex set, it follows from the definition of $\\bar x_k$ in \\eqref{eq:bxk} and Lemma \\ref{lem:bound} that $\\bar x_k \\in Q$ for every $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$.\n    This observation and the fact that $\\hat h = h+ I_Q$ imply that $\\hat h(\\bar x_k) = h(\\bar x_k)$.\n    Hence, using Proposition~\\ref{prop:pd-key}, we have for every $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$,\n    \\[\n    \\hat \\phi(\\bar x_k) + f^*(\\bar s_k) + \\hat h ^*(-\\bar s_k) \\le \\bar \\varepsilon + \\frac{18 d_0^2}{\\lam k}.\n    \\]\n    Therefore, the conclusion of the proposition follows immediately.\n\\end{proof}\n\n\\begin{lemma}\\label{lem:t1}\nFor $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$, we have\n\\begin{equation}\\label{ineq:t1}\n    t_{i_k} \\le \\bar{t}:= 4M(3d_0 + \\lam M),\n\\end{equation}\nwhere $i_k$ is the first iteration index in the cycle ${\\cal C}_k$.\n\\end{lemma}\n\n\\begin{proof}\nUsing \\eqref{ineq:f}, definitions of $m_j$ and $t_j$ in \\eqref{eq:mj}, and the facts that $\\tx_{i_k}=x_{i_k}$ and $\\Gamma_{i_k}=\\ell_f(\\cdot;x_{k-1})$, we have\n\\begin{align}\n    t_{i_k} &\\stackrel{\\eqref{eq:mj}}{=} \\phi^\\lam \\left(\\tx_{i_k}\\right)-m_{i_k} = \\phi^\\lam (x_{i_k}) -m_{i_k} \\nn \\\\\n    &\\stackrel{\\eqref{eq:problem},\\eqref{eq:phi-lam},\\eqref{eq:mj}}{=} f(x_{i_k})-\\ell_f(x_{i_k};\\hat x_{k-1}) \\stackrel{\\eqref{ineq:f}}{\\le} 2M\\left\\|x_{i_k}-\\hat x_{k-1}\\right\\|, \\label{ineq:tik}\n\\end{align}\nwhere we have also used the definitions of $\\phi$ and $\\phi^\\lam$ in \\eqref{eq:problem} and \\eqref{eq:phi-lam}, respectively, in the last identity.\nIn view of \\eqref{eq:xj} and the fact that $\\Gamma_{i_k}=\\ell_f(\\cdot;\\hat x_{k-1})$, we know the first iteration of PDCP is the same as PDS$(\\hat x_0,\\lam)$ (see \\eqref{eq:pds}). Hence, following an argument similar to the proof of Lemma~\\ref{pds-lemma1}, we can prove for every $u \\in \\dom h$,\n\\[\n\\phi(x_{i_k}) - \\ell_f(u;\\hat x_{k-1}) -h(u)\n\\stackrel{\\eqref{ineq:hat-phi-recur}}{\\le} 2\\lam M^2 + \\frac{1}{2 \\lambda}\\|u -\\hat x_{k-1}\\|^2 - \\frac{1}{2\\lambda}\\left\\|u-x_{i_k}\\right\\|^2.\n\\]\nIt follows from the above inequality with $u=x_0^*$ and the convexity of $f$ that\n\\begin{align*}\n    0 &\\le \\phi(x_{i_k}) -\\phi_*\n    \\le \\phi(x_{i_k}) - \\ell_f(x_0^*;\\hat x_{k-1}) -h(x_0^*) \\\\\n    &\\le 2\\lam M^2 + \\frac{1}{2 \\lambda}\\|x_0^* -\\hat x_{k-1}\\|^2 - \\frac{1}{2\\lambda}\\left\\|x_0^*-x_{i_k}\\right\\|^2.\n\\end{align*}\nRearranging the terms and using the inequality $\\sqrt{a+b} \\le \\sqrt{a} + \\sqrt{b}$ for any $a, b \\ge 0$, we have\n\\[\n\\|x_0^*-x_{i_k}\\| \\le\\|x_0^*-\\hat x_{k-1}\\|+2\\lam M.\n\\]\nThis inequality and the triangle inequality then imply that\n\\[\n\\|x_{i_k}-\\hat x_{k-1}\\| \\le \\|x_{i_k}-x_0^*\\|+\\|x_0^*-\\hat x_{k-1}\\| \\le 2\\|\\hat x_{k-1}-x_0^*\\|+2 \\lam M.\n\\]\nRecall from the proof of Lemma \\ref{lem:bound} that \\eqref{ineq:xk} gives $\\|\\hat x_k-x_0^*\\| \\le 3d_0$ for $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$.\nHence, we have\n\\[\n\\|x_{i_k}-\\hat x_{k-1}\\| \\le 2(3d_0 + \\lam M).\n\\]\nTherefore, \\eqref{ineq:t1} follows from \\eqref{ineq:tik} and the above inequality.\n\\end{proof}\n\n\\begin{proof}\n    In view of Proposition \\ref{prop:pdpb}, PDPB takes \n    \\begin{equation}\\label{cmplx:outer}\n        {\\cal O}\\left(\\frac{d_0^2}{\\lam \\bar \\varepsilon}+1\\right)\n    \\end{equation}\n    cycles to find $(\\bar x_k,\\bar s_k)$ satisfying \\eqref{ineq:pd}.\n    It follows from Proposition \\ref{prop:inner} and Lemma \\ref{lem:t1} that for every cycle in PDPB before termination, the number of iterations in the cycle is \n    \\[\n    {\\cal O}\\left(\\frac{\\sqrt{Md_0 + \\lam M^2}}{\\sqrt{\\bar \\varepsilon}} + \\frac{\\lam M^2}{\\bar \\varepsilon} + 1\\right)=\n    {\\cal O}\\left(\\frac{\\sqrt{Md_0}}{\\sqrt{\\bar \\varepsilon}} + \\frac{\\lam M^2}{\\bar \\varepsilon} + 1\\right),\n    \\]\n    which together with the assumption that $\\sqrt{\\bar \\varepsilon d_0}/M^{3/2} \\le \\lam$ becomes\n    \\begin{equation}\\label{cmplx:inner}\n        {\\cal O}\\left(\\frac{\\lam M^2}{\\bar \\varepsilon} + 1\\right).\n    \\end{equation}\n    Combining \\eqref{cmplx:outer} and \\eqref{cmplx:inner}, and using \\eqref{ineq:lam}, we conclude that \\eqref{cmplx:PDPB} holds.\n\\end{proof}\n\n\\begin{lemma}\\label{pds-lemma1}\nGiven $\\hat x_0 \\in \\R^n$, for every $k \\ge 1$ and $u \\in \\dom \\hat h$, the sequence $\\{\\hat x_k\\}$ generated by MPDS$(\\hat x_0,\\lam)$ satisfies\n\\begin{equation}\\label{ineq:hat-phi-recur}\n    \\hat \\phi(\\hat x_k) - \\ell_f(u;\\hat x_{k-1}) - \\hat h(u) \\le 2\\lam M^2 + \\frac{1}{2\\lam}\\|u- \\hat x_{k-1}\\|^2 - \\frac1{2\\lam}\\|u - \\hat x_k\\|^2.\n\\end{equation}\n\\end{lemma}\n\n\\begin{proof}\nNoticing that the objective function in \\eqref{eq:pds-1} is $\\lam^{-1}$-strongly convex, it then follows from Theorem 5.25(b) of \\cite{Beck2017} that for every $u \\in \\dom \\hat h$,\n\\begin{equation}\\label{ineq:ell-h-mk}\n    \\ell_f(u;\\hat x_{k-1}) + \\hat h(u) + \\frac1{2\\lam}\\|u-\\hat x_{k-1}\\|^2 \\ge m_k + \\frac{1}{2\\lam} \\|u-\\hat x_k\\|^2,\n\\end{equation}\nwhere $m_k = \\ell_f(\\hat x_k;\\hat x_{k-1}) + \\hat h(\\hat x_k) + \\|\\hat x_k-\\hat x_{k-1}\\|^2/(2\\lam)$.\nUsing \\eqref{ineq:f} with $(x,y)=(\\hat x_k,\\hat x_{k-1})$, we have\n\\[\n    \\hat \\phi(\\hat x_k) - m_k = f(\\hat x_k) - \\ell_f(\\hat x_k;\\hat x_{k-1}) \\stackrel{\\eqref{ineq:f}}{\\le} 2M\\|\\hat x_k - \\hat x_{k-1}\\| - \\frac1{2\\lam}\\|\\hat x_k-\\hat x_{k-1}\\|^2 \\le 2\\lam M^2,\n\\]\nwhere the last inequality is due to Young's inequality $a^2+b^2\\ge 2ab$.\nHence, \\eqref{ineq:hat-phi-recur} follows from combining the above inequality and \\eqref{ineq:ell-h-mk}.\n\\end{proof}","units":[{"start":30159,"end":30422,"start_line":483,"end_line":488,"raw_tex":"\\begin{lemma}\\label{lem:Gamma}\nFor every $j \\ge 1$, there exists $\\bar \\Gamma_j \\in \\bConv{n}$ such that for every $u\\in \\R^n$,\n\\begin{equation}\\label{ineq:bGammaj}\n    \\bar{\\Gamma}_j(u) + h^\\lam(u) \\ge m_j + \\frac{1}{2\\lam}\\|u-x_j\\|^2.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"332caa9f0ad73aa0d96682658d53b903b981be937069d823300b44f897ceed98"},{"start":30424,"end":30944,"start_line":490,"end_line":496,"raw_tex":"\\begin{proof}\n    Since the objective function in \\eqref{def:bar Gamma} is $\\lam^{-1}$-strongly convex, it follows from \\eqref{def:bar Gamma} that \n    \\[\n    \\bar \\Gamma_j(u) + h(u) + \\frac{1}{2\\lam}\\|u-x_0\\|^2 \\ge \\bar \\Gamma_j(x_j) + h(x_j) + \\frac{1}{2\\lam}\\|x_j-x_0\\|^2 + \\frac{1}{2\\lam}\\|u-x_j\\|^2.\n    \\]\n    Inequality \\eqref{ineq:bGammaj} immediately follows from the above inequality, the definition of $m_j$ in \\eqref{eq:mj}, and the fact that $h^\\lam(\\cdot) = h(\\cdot) + \\|\\cdot-x_0\\|^2/(2\\lam)$.\n\\end{proof}","sha256_utf8":"258dcd2c2a4c931184b0ad8a457256bf942c40f2b4c5cf83a7aa36f20403939d"},{"start":31171,"end":31650,"start_line":501,"end_line":511,"raw_tex":"\\begin{proposition}\\label{prop:inner}\nConsidering Algorithm \\ref{alg:PDCP} with $\\tau_j = j/(j+2)$, then for every $j \\ge 1$, we have\n\\begin{equation}\\label{ineq:tj}\n    t_j\\le \\frac{2t_1}{j(j+1)} + \\frac{16\\lam M^2}{j+1},\n\\end{equation}\nwhere $t_j$ is as in \\eqref{eq:mj}.\nMoreover, the number of iterations for PDCP to obtain $t_j \\le \\varepsilon$ is at most\n\\[\n{\\cal O}\\left(\\frac{\\sqrt{t_1}}{\\sqrt{\\varepsilon}} + \\frac{\\lam M^2}{\\varepsilon} + 1\\right).\n\\]\n\\end{proposition}","sha256_utf8":"c4113160384754314eb4d13decbb6c32e815634573ee8c649a03dc107154690b"},{"start":31652,"end":34019,"start_line":513,"end_line":547,"raw_tex":"\\begin{proof}\nWe first note that for every $j\\ge 1$, $\\tau_j=A_j/A_{j+1}$ where $A_{j+1}=A_j+j+1$ and $A_0=0$, i.e., $A_j=j(j+1)/2$ for every $j\\ge 0$.\nIt follows from this observation, the definition of $m_j$ in \\eqref{eq:mj}, and relation \\eqref{def:Gammaj} that\n\\begin{align*}\n    A_{j+1}m_{j+1} & \\stackrel{\\eqref{eq:mj}}{=} A_{j+1}(\\Gamma_{j+1}+h^\\lam)(x_{j+1}) \\\\\n    &\\stackrel{\\eqref{def:Gammaj}}{\\ge} A_j \\left[(\\bar \\Gamma_j+ h^\\lam)(x_{j+1}) \\right] + (j+1)\\left[\\ell_f(x_{j+1};x_j) + h^\\lam(x_{j+1})\\right].\n\\end{align*}\nApplying Lemma~\\ref{lem:Gamma} in the above inequality and using \\eqref{ineq:f}, we have\n\\begin{align*}\n    A_{j+1}m_{j+1} \n    &\\stackrel{\\eqref{ineq:bGammaj}}{\\ge} A_j \\left[ m_j+\\frac{1}{2\\lam}\\|x_{j+1} - x_j\\|^2 \\right] + (j+1)\\left[\\ell_f(x_{j+1};x_j) + h^\\lam(x_{j+1})\\right] \\\\\n    &= A_j m_j + (j+1)\\left[\\ell_f(x_{j+1};x_j) + h^\\lam(x_{j+1}) + \\frac{A_j}{2\\lam (j+1)}\\|x_{j+1} - x_j\\|^2 \\right] \\\\\n    &\\stackrel{\\eqref{ineq:f}}{\\ge} A_j m_j + (j+1)\\left[\\phi^\\lam(x_{j+1}) - 2M\\|x_{j+1}-x_j\\| + \\frac{A_j}{2\\lam (j+1)}\\|x_{j+1} - x_j\\|^2 \\right] \\\\\n    &\\ge A_j m_j + (j+1)\\phi^\\lam(x_{j+1}) - \\frac{2\\lam M^2 (j+1)^2}{A_j}\n\\end{align*}\nwhere the last inequality is due to the Young's inequality $a^2+b^2\\ge 2ab$.\nIt follows from the fact that $A_j=j(j+1)/2$ that for every $j\\ge 1$,\n\\[\n    A_{j+1}m_{j+1} \\ge A_j m_j + (j+1)\\phi^\\lam(x_{j+1}) - 8\\lam M^2.\n\\]\nReplacing the index $j$ in the above inequality by $i$, summing the resulting inequality from $i=1$ to $j-1$, and using the definition of $t_j$ in \\eqref{eq:mj} and the fact that $\\tx_1=x_1$, we obtain\n\\begin{align*}\n    A_j m_j\n    &\\ge A_1 m_1 + 2 \\phi^\\lam(x_2) + \\dots + j\\phi^\\lam(x_j) - 8\\lam M^2 (j-1) \\\\\n    &\\stackrel{\\eqref{eq:mj}}{=} -A_1 t_1 + A_1 \\phi^\\lam(x_1) + 2 \\phi^\\lam(x_2) + \\dots + j\\phi^\\lam(x_j) - 8\\lam M^2 (j-1) \\\\\n    &\\stackrel{\\eqref{ineq:txj}}{\\ge} -A_1 t_1 + A_j \\phi^\\lam(\\tx_j) - 8\\lam M^2 (j-1),\n\\end{align*}\nwhere the last inequality follows from \\eqref{ineq:txj} and the fact that $A_j=A_{j-1}+j$.\nRearranging the terms and using the definition of $t_j$ in \\eqref{eq:mj} again, we have\n\\begin{equation}\\label{ineq:Aj-tj}\n    A_j t_j \\le A_1 t_1 + 8\\lam M^2 (j-1).\n\\end{equation}\nHence, \\eqref{ineq:tj} follows from the fact that $A_j=j(j+1)/2$.\nFinally, the complexity bound immediately follows from \\eqref{ineq:tj}.\n\\end{proof}","sha256_utf8":"ca59a5c045500495415d9c37d9417e18220e40e444275a1ef601c490f2ef3c24"},{"start":36820,"end":37325,"start_line":593,"end_line":600,"raw_tex":"\\begin{lemma}\\label{ser-lemma1}\n    The following statements hold for every $k \\ge 1$:\n    \\begin{enumerate}\n        \\item [(a)] $\\Gamma_k \\le f$ and $f^*\\le \\Gamma_k^*$;\n        \\item [(b)] $s_k \\in \\partial \\Gamma_k(\\hat x_k)$ and $g_k \\in \\partial h(\\hat x_k)$ where $g_k = - s_k + (\\hat x_{k-1}-\\hat x_k)/\\lam$;\n        \\item [(c)] $\\phi^\\lam(\\tilde x_k) \\le \\bar \\varepsilon + m_k = \\bar \\varepsilon + (\\Gamma_k + h)(\\hat x_k) + \\|\\hat x_k - \\hat x_{k-1}\\|^2/(2\\lam)$.\n    \\end{enumerate}\n\\end{lemma}","sha256_utf8":"84facb44c24a0af445e031fd5937ffe2fb4f52b8197b609d105060d745584ba7"},{"start":37327,"end":37871,"start_line":602,"end_line":608,"raw_tex":"\\begin{proof}\n    (a) It follows from the facts that $\\Gamma_j \\le f$ for every $j\\ge 1$ and $\\Gamma_k=\\Gamma_{j_k}$ that the first inequality holds. The second one immediately follows from the first one and the definition of the conjugate function.\n\n    (b) This statement follows from \\eqref{eq:xk} and the definitions in \\eqref{eq:translate}.\n\n    (c) This statement follows from the termination criterion of the $k$-th cycle, that is, $t_{j_k}\\le \\bar \\varepsilon$, and the definitions in \\eqref{eq:mj} and \\eqref{eq:translate}.\n\\end{proof}","sha256_utf8":"bcd9c03578202f7009093057dbd4b316d2148f8044d875426e6ac28c7bd03fef"},{"start":38018,"end":38322,"start_line":612,"end_line":618,"raw_tex":"\\begin{proposition}\\label{prop:pd-key}\nFor every $k \\ge 1$, we have\n\\begin{equation}\\label{ineq:conjugate}\n    \\phi(\\bar{x}_k) + f^*(\\bar s_k) + \\hat h ^*(-\\bar s_k) \\le \\bar \\varepsilon + \\frac{18 d_0^2}{\\lam k}.\n\\end{equation}\nwhere $\\bar x_k$ and $\\bar s_k$ are as in \\eqref{eq:bxk}.\n\\end{proposition}","sha256_utf8":"a3446358430845e2b69b50d993bb01d27d2c1fa9740464136b19eaa67ef2fb3b"},{"start":38324,"end":41722,"start_line":620,"end_line":680,"raw_tex":"\\begin{proof}\nIt follows from Lemma \\ref{ser-lemma1}(b) and Theorem 4.20 of \\cite{Beck2017} that for every $k \\ge 1$,\n\\[ \n\\Gamma_k(\\hat x_k) +  \\Gamma_k^*( s_k) = \\inner{\\hat x_k}{s_k}, \\quad  h(\\hat x_k) +  h^*(g_k) = \\inner{\\hat x_k}{g_k}.\n\\]\nSumming the above two equations yields\n\\begin{equation}\\label{eq:Gamma-h}\n    (\\Gamma_k + h)(\\hat x_k) + {\\Gamma}_k^*(s_k) + h^*(g_k) = \\frac{1}{\\lam}\\inner{\\hat x_k}{\\hat x_{k-1}-\\hat x_k}.\n\\end{equation}\nUsing the above identity and Lemma \\ref{ser-lemma1}(a) and (c), we have for every $k \\ge 1$,\n\\begin{align*}\n& \\phi(\\tilde x_k) + f^*(s_k) +  {h}^*(g_k) \\le \\phi(\\tilde x_k) + \\Gamma_k^*(s_k) +  {h}^*(g_k) \\\\\n\\le & \\bar \\varepsilon +(\\Gamma_k + h)(\\hat x_k) + \\frac{1}{2\\lam}\\|\\hat x_k-\\hat x_{k-1}\\|^2 + \\Gamma_k^*(s_k) + {h}^*(g_k)\\\\\n\\stackrel{\\eqref{eq:Gamma-h}}{=}& \\bar \\varepsilon + \\frac{1}{2\\lam}(\\|\\hat x_{k-1}\\|^2-\\|\\hat x_k\\|^2).\n\\end{align*}\nReplacing the index $k$ in the above inequality by $i$, summing the resulting inequality from $i=1$ to $k$, and using convexity and the definitions in \\eqref{eq:bxk}, we obtain \n\\begin{equation}\\label{ineq:avg}\n    \\phi(\\bar x_k) + f^*(\\bar s_k) + h^*(\\bar g_k)\\le \\bar \\varepsilon + \\frac{1}{2\\lam k }\\left(\\|\\hat x_0\\|^2-\\|\\hat x_k\\|^2\\right),\n\\end{equation}\nwhere $\\bar g_k = (\\sum_{i=1}^k g_i)/k$.\nDefine\n\\begin{equation}\\label{eq:zeta}\n    \\eta_k(u) = \\frac{1}{2\\lam k} \\|u-\\hat x_0\\|^2, \\quad \\hat \\eta_k(u) = \\eta_k(u) - I_Q(u).\n\\end{equation}\nNoting that $\\nabla \\eta_k(\\hat x_k) = (\\hat x_k - \\hat x_0)/(\\lam k) = -\\bar g_k - \\bar s_k$, and hence it follows from Theorem 4.20 of \\cite{Beck2017} that\n\\[\n\\eta_k^*(-\\bar g_k - \\bar s_k) = \\frac{1}{\\lam k}\\inner{\\hat x_k-\\hat x_0}{\\hat x_k} - \\eta_k(\\hat x_k) = \\frac{1}{2\\lam k}\\left(\\|\\hat x_k\\|^2 - \\| \\hat x_0\\|^2\\right).\n\\]\nThe above observation and \\eqref{ineq:avg} together imply that\n\\begin{equation}\\label{ineq:bar}\n    \\phi(\\bar x_k) + f^*(\\bar s_k) + h^*(\\bar g_k) + \\eta_k^*(-\\bar g_k - \\bar s_k) \\le \\bar \\varepsilon.\n\\end{equation}\nIt follows from Theorem 4.17 of \\cite{Beck2017} and the definition of infimum convolution in \\eqref{def:infconv} that\n\\[\n(h + \\eta_k)^*(-\\bar s_k) = (h^* \\square \\eta_k^*)(-\\bar s_k)\n\\stackrel{\\eqref{def:infconv}}= \\min_{u\\in \\R^n}\\{h^*(u) + \\eta_k^*(-\\bar s_k-u)\\}\n\\le h^*(\\bar g_k) + \\eta_k^*(-\\bar g_k - \\bar s_k).\n\\]\nNoting from \\eqref{eq:zeta} that $\\hat h = h + \\eta_k -\\hat \\eta_k$ and applying Theorem 4.17 of \\cite{Beck2017} again, we obtain\n\\begin{align*}\n    \\hat h ^*(-\\bar s_k) &= [(h + \\eta_k)^* \\square (-\\hat \\eta_k)^*](-\\bar s_k)\n= \\min_{u\\in \\R^n}\\{(h + \\eta_k)^*(u) + (-\\hat \\eta_k)^*(-\\bar s_k-u)\\}\\\\\n&\\le (h + \\eta_k)^*(-\\bar s_k) + (-\\hat \\eta_k)^*(0).\n\\end{align*}\nSumming the above two inequalities, we have\n\\[\n\\hat h ^*(-\\bar s_k) \\le h^*(\\bar g_k) + \\eta_k^*(-\\bar g_k - \\bar s_k) + (-\\hat \\eta_k)^*(0),\n\\]\nwhich together with \\eqref{ineq:bar} implies that\n\\[\n\\phi(\\bar x_k) + f^*(\\bar s_k) + \\hat h ^*(-\\bar s_k) \\le \\bar \\varepsilon + (-\\hat \\eta_k)^*(0).\n\\]\nIt follows from \\eqref{eq:zeta} that\n\\[\n(-\\hat \\eta_k)^*(0) = \\max_{u \\in \\R^n}\\left\\{\\inner{0}{u} - \\left( -\\frac{\\|u-\\hat x_0\\|^2}{2\\lam k} +I_Q(u)\\right)\\right\\}\n=\\frac{\\max_{u \\in Q}\\|u-\\hat x_0\\|^2}{2\\lam k}\n= \\frac{18d_0^2}{\\lam k},\n\\]\nwhere the last identity follows from the fact that $Q = B(\\hat x_0, 6d_0)$.\nTherefore, \\eqref{ineq:conjugate} holds in view of the above two relations.\n\\end{proof}","sha256_utf8":"5d2603c3f1233d54202b610d39a0be95444e7343600954f38c79e04f1a33e003"},{"start":41863,"end":42261,"start_line":684,"end_line":689,"raw_tex":"\\begin{lemma}\\label{lem:bound}\n    Given $(\\hat x_0,\\bar \\varepsilon) \\in \\R^n \\times \\R_{++}$, if $\\lam \\le 2d_0^2/\\bar \\varepsilon$ and $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$, then the sequences $\\{\\hat x_k\\}$ and $\\{\\tx_k\\}$ generated by PDPB$(\\hat x_0,\\lam, \\bar \\varepsilon)$ satisfy\n    \\begin{equation}\\label{incl:xk}\n        \\hat x_k \\in Q, \\quad \\tx_k \\in Q.\n    \\end{equation}\n\\end{lemma}","sha256_utf8":"c29364c56802e276a966fa5ebc30b735a17e636589f57081f7e58cf105da6836"},{"start":42263,"end":44509,"start_line":691,"end_line":724,"raw_tex":"\\begin{proof}\n    Noticing that the objective function in \\eqref{eq:xk} is $\\lam^{-1}$-strongly convex, it thus follows from Theorem 5.25(b) of \\cite{Beck2017} that for every $u \\in \\dom h$,\n    \\begin{equation}\\label{ineq:key}\n        m_k + \\frac{1}{2\\lam} \\|u-\\hat x_{k}\\|^2 \\le \\Gamma_k(u) + h(u) + \\frac1{2\\lam}\\|u-\\hat x_{k-1}\\|^2 \\le \\phi(u)+ \\frac1{2\\lam}\\|u-\\hat x_{k-1}\\|^2, \n    \\end{equation}\n    where the second inequality follows from the first one in Lemma \\ref{ser-lemma1}(a).\n    Taking $u=x_0^*$ in \\eqref{ineq:key}, we have\n    \\[\n    m_k + \\frac{1}{2\\lam} \\|\\hat x_k-x_0^*\\|^2 \\le \\phi_* + \\frac1{2\\lam}\\|\\hat x_{k-1}-x_0^*\\|^2. \n    \\]\n    This inequality and Lemma \\ref{ser-lemma1}(c) then imply that\n    \\begin{align*}\n        \\frac{1}{2\\lam}\\|\\hat x_k - x_0^*\\|^2 &\\le \\phi(\\tx_k)- \\phi_* + \\frac{1}{2\\lam}\\|\\hat x_k - x_0^*\\|^2 \\\\\n        &\\le \\phi(\\tx_k) - m_k + \\frac{1}{2\\lam}\\|\\hat x_{k-1}-x_0^*\\|^2 \\le \\bar \\varepsilon + \\frac{1}{2\\lam}\\|\\hat x_{k-1}-x_0^*\\|^2.\n    \\end{align*}\n    Replacing the index $k$ in the above inequality by $i$ and summing the resulting inequality from $i=1$ to $k$, we have\n    \\[\n    \\|\\hat x_k - x_0^*\\|^2 \\le \\|\\hat x_0-x_0^*\\|^2 + 2 k \\lam \\bar \\varepsilon.\n    \\]\n    Using the fact that $\\sqrt{a+b} \\le \\sqrt{a} + \\sqrt{b}$ for $a,b \\ge 0$ and the assumption that $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$, we further obtain\n    \\begin{equation}\\label{ineq:xk}\n        \\|\\hat x_k - x_0^*\\| \\le d_0 + \\sqrt{2k\\lam \\bar \\varepsilon} \\le 3d_0.\n    \\end{equation}\n    Taking $u=\\tx_k$ in \\eqref{ineq:key} and using Lemma \\ref{ser-lemma1}(c), we have\n    \\[\n    \\frac{1}{2\\lam} \\|\\tx_k-\\hat x_k\\|^2 \\le \\phi(\\tx_k) + \\frac1{2\\lam}\\|\\tx_k - \\hat x_{k-1}\\|^2 - m_k \\le \\bar \\varepsilon.\n    \\]\n    Under the assumption that $\\lam \\le 2d_0^2/\\bar \\varepsilon$, using \\eqref{ineq:xk}, the above inequality, and the triangle inequality, we have\n    \\begin{align*}\n        \\|\\hat x_k - \\hat x_0\\| &\\le \\|\\hat x_k - x_0^*\\| + \\|x_0^* - \\hat x_0\\| \\stackrel{\\eqref{ineq:xk}}{\\le} 4d_0, \\\\\n        \\|\\tx_k - \\hat x_0\\| &\\le \\|\\hat x_k - \\hat x_0\\| + \\|\\tx_k - \\hat x_k\\| \\le 4d_0 + \\sqrt{2\\lam \\bar \\varepsilon} \\le 6d_0.\n    \\end{align*}\n    Hence, \\eqref{incl:xk} follows immediately.\n\\end{proof}","sha256_utf8":"5c6a108ccf3533bf1b1aded5425dfda928ab030c781855de9aa1e441312cd698"},{"start":44617,"end":45090,"start_line":728,"end_line":734,"raw_tex":"\\begin{proposition}\\label{prop:pdpb}\n    Given $(\\hat x_0,\\bar \\varepsilon) \\in \\R^n \\times \\R_{++}$, if $\\lam \\le 2d_0^2/\\bar \\varepsilon$, then the number of iterations for PDPB$(\\hat x_0,\\lam, \\bar \\varepsilon)$ to generate $(\\bar x_k, \\bar s_k)$ satisfying \n    \\begin{equation}\\label{ineq:pd}\n        \\hat \\phi(\\bar x_k) + f^*(\\bar s_k) + \\hat h^*(-\\bar s_k) \\le 10\\bar \\varepsilon\n    \\end{equation}\n    is at most $2 d_0^2/(\\lam \\bar \\varepsilon)$.\n\\end{proposition}","sha256_utf8":"5a8d58402d4310ccaffe43da5034c3556e1ebc194d75b27ab2a21ff851b3d23c"},{"start":45092,"end":45712,"start_line":736,"end_line":744,"raw_tex":"\\begin{proof}\n    Since $Q$ is a convex set, it follows from the definition of $\\bar x_k$ in \\eqref{eq:bxk} and Lemma \\ref{lem:bound} that $\\bar x_k \\in Q$ for every $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$.\n    This observation and the fact that $\\hat h = h+ I_Q$ imply that $\\hat h(\\bar x_k) = h(\\bar x_k)$.\n    Hence, using Proposition~\\ref{prop:pd-key}, we have for every $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$,\n    \\[\n    \\hat \\phi(\\bar x_k) + f^*(\\bar s_k) + \\hat h ^*(-\\bar s_k) \\le \\bar \\varepsilon + \\frac{18 d_0^2}{\\lam k}.\n    \\]\n    Therefore, the conclusion of the proposition follows immediately.\n\\end{proof}","sha256_utf8":"d5d3d31672c341aadbc9328670b644e0b66df505f2a50546be35b51251750e18"},{"start":45841,"end":46092,"start_line":748,"end_line":754,"raw_tex":"\\begin{lemma}\\label{lem:t1}\nFor $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$, we have\n\\begin{equation}\\label{ineq:t1}\n    t_{i_k} \\le \\bar{t}:= 4M(3d_0 + \\lam M),\n\\end{equation}\nwhere $i_k$ is the first iteration index in the cycle ${\\cal C}_k$.\n\\end{lemma}","sha256_utf8":"0758ec57f7de3c270bc2e1b152da757d8eb1c8e112cd2a72cae80c5e1ed37774"},{"start":46094,"end":48233,"start_line":756,"end_line":788,"raw_tex":"\\begin{proof}\nUsing \\eqref{ineq:f}, definitions of $m_j$ and $t_j$ in \\eqref{eq:mj}, and the facts that $\\tx_{i_k}=x_{i_k}$ and $\\Gamma_{i_k}=\\ell_f(\\cdot;x_{k-1})$, we have\n\\begin{align}\n    t_{i_k} &\\stackrel{\\eqref{eq:mj}}{=} \\phi^\\lam \\left(\\tx_{i_k}\\right)-m_{i_k} = \\phi^\\lam (x_{i_k}) -m_{i_k} \\nn \\\\\n    &\\stackrel{\\eqref{eq:problem},\\eqref{eq:phi-lam},\\eqref{eq:mj}}{=} f(x_{i_k})-\\ell_f(x_{i_k};\\hat x_{k-1}) \\stackrel{\\eqref{ineq:f}}{\\le} 2M\\left\\|x_{i_k}-\\hat x_{k-1}\\right\\|, \\label{ineq:tik}\n\\end{align}\nwhere we have also used the definitions of $\\phi$ and $\\phi^\\lam$ in \\eqref{eq:problem} and \\eqref{eq:phi-lam}, respectively, in the last identity.\nIn view of \\eqref{eq:xj} and the fact that $\\Gamma_{i_k}=\\ell_f(\\cdot;\\hat x_{k-1})$, we know the first iteration of PDCP is the same as PDS$(\\hat x_0,\\lam)$ (see \\eqref{eq:pds}). Hence, following an argument similar to the proof of Lemma~\\ref{pds-lemma1}, we can prove for every $u \\in \\dom h$,\n\\[\n\\phi(x_{i_k}) - \\ell_f(u;\\hat x_{k-1}) -h(u)\n\\stackrel{\\eqref{ineq:hat-phi-recur}}{\\le} 2\\lam M^2 + \\frac{1}{2 \\lambda}\\|u -\\hat x_{k-1}\\|^2 - \\frac{1}{2\\lambda}\\left\\|u-x_{i_k}\\right\\|^2.\n\\]\nIt follows from the above inequality with $u=x_0^*$ and the convexity of $f$ that\n\\begin{align*}\n    0 &\\le \\phi(x_{i_k}) -\\phi_*\n    \\le \\phi(x_{i_k}) - \\ell_f(x_0^*;\\hat x_{k-1}) -h(x_0^*) \\\\\n    &\\le 2\\lam M^2 + \\frac{1}{2 \\lambda}\\|x_0^* -\\hat x_{k-1}\\|^2 - \\frac{1}{2\\lambda}\\left\\|x_0^*-x_{i_k}\\right\\|^2.\n\\end{align*}\nRearranging the terms and using the inequality $\\sqrt{a+b} \\le \\sqrt{a} + \\sqrt{b}$ for any $a, b \\ge 0$, we have\n\\[\n\\|x_0^*-x_{i_k}\\| \\le\\|x_0^*-\\hat x_{k-1}\\|+2\\lam M.\n\\]\nThis inequality and the triangle inequality then imply that\n\\[\n\\|x_{i_k}-\\hat x_{k-1}\\| \\le \\|x_{i_k}-x_0^*\\|+\\|x_0^*-\\hat x_{k-1}\\| \\le 2\\|\\hat x_{k-1}-x_0^*\\|+2 \\lam M.\n\\]\nRecall from the proof of Lemma \\ref{lem:bound} that \\eqref{ineq:xk} gives $\\|\\hat x_k-x_0^*\\| \\le 3d_0$ for $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$.\nHence, we have\n\\[\n\\|x_{i_k}-\\hat x_{k-1}\\| \\le 2(3d_0 + \\lam M).\n\\]\nTherefore, \\eqref{ineq:t1} follows from \\eqref{ineq:tik} and the above inequality.\n\\end{proof}","sha256_utf8":"f82bb3a33333e17c60a08513db2ab163227711f71530c4cb1eba401e3d96e168"},{"start":48864,"end":49904,"start_line":803,"end_line":819,"raw_tex":"\\begin{proof}\n    In view of Proposition \\ref{prop:pdpb}, PDPB takes \n    \\begin{equation}\\label{cmplx:outer}\n        {\\cal O}\\left(\\frac{d_0^2}{\\lam \\bar \\varepsilon}+1\\right)\n    \\end{equation}\n    cycles to find $(\\bar x_k,\\bar s_k)$ satisfying \\eqref{ineq:pd}.\n    It follows from Proposition \\ref{prop:inner} and Lemma \\ref{lem:t1} that for every cycle in PDPB before termination, the number of iterations in the cycle is \n    \\[\n    {\\cal O}\\left(\\frac{\\sqrt{Md_0 + \\lam M^2}}{\\sqrt{\\bar \\varepsilon}} + \\frac{\\lam M^2}{\\bar \\varepsilon} + 1\\right)=\n    {\\cal O}\\left(\\frac{\\sqrt{Md_0}}{\\sqrt{\\bar \\varepsilon}} + \\frac{\\lam M^2}{\\bar \\varepsilon} + 1\\right),\n    \\]\n    which together with the assumption that $\\sqrt{\\bar \\varepsilon d_0}/M^{3/2} \\le \\lam$ becomes\n    \\begin{equation}\\label{cmplx:inner}\n        {\\cal O}\\left(\\frac{\\lam M^2}{\\bar \\varepsilon} + 1\\right).\n    \\end{equation}\n    Combining \\eqref{cmplx:outer} and \\eqref{cmplx:inner}, and using \\eqref{ineq:lam}, we conclude that \\eqref{cmplx:PDPB} holds.\n\\end{proof}","sha256_utf8":"5350f226d52fb67e9130c742e37f987f44acd8ec0bd37eb556101c6ed1535056"},{"start":124018,"end":124413,"start_line":1923,"end_line":1928,"raw_tex":"\\begin{lemma}\\label{pds-lemma1}\nGiven $\\hat x_0 \\in \\R^n$, for every $k \\ge 1$ and $u \\in \\dom \\hat h$, the sequence $\\{\\hat x_k\\}$ generated by MPDS$(\\hat x_0,\\lam)$ satisfies\n\\begin{equation}\\label{ineq:hat-phi-recur}\n    \\hat \\phi(\\hat x_k) - \\ell_f(u;\\hat x_{k-1}) - \\hat h(u) \\le 2\\lam M^2 + \\frac{1}{2\\lam}\\|u- \\hat x_{k-1}\\|^2 - \\frac1{2\\lam}\\|u - \\hat x_k\\|^2.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"39ca5924c622c5cec4ad9c1a95f442932c8f9523c7bf6cf9ff50beb501b7c94b"},{"start":124414,"end":125340,"start_line":1929,"end_line":1941,"raw_tex":"\\begin{proof}\nNoticing that the objective function in \\eqref{eq:pds-1} is $\\lam^{-1}$-strongly convex, it then follows from Theorem 5.25(b) of \\cite{Beck2017} that for every $u \\in \\dom \\hat h$,\n\\begin{equation}\\label{ineq:ell-h-mk}\n    \\ell_f(u;\\hat x_{k-1}) + \\hat h(u) + \\frac1{2\\lam}\\|u-\\hat x_{k-1}\\|^2 \\ge m_k + \\frac{1}{2\\lam} \\|u-\\hat x_k\\|^2,\n\\end{equation}\nwhere $m_k = \\ell_f(\\hat x_k;\\hat x_{k-1}) + \\hat h(\\hat x_k) + \\|\\hat x_k-\\hat x_{k-1}\\|^2/(2\\lam)$.\nUsing \\eqref{ineq:f} with $(x,y)=(\\hat x_k,\\hat x_{k-1})$, we have\n\\[\n    \\hat \\phi(\\hat x_k) - m_k = f(\\hat x_k) - \\ell_f(\\hat x_k;\\hat x_{k-1}) \\stackrel{\\eqref{ineq:f}}{\\le} 2M\\|\\hat x_k - \\hat x_{k-1}\\| - \\frac1{2\\lam}\\|\\hat x_k-\\hat x_{k-1}\\|^2 \\le 2\\lam M^2,\n\\]\nwhere the last inequality is due to Young's inequality $a^2+b^2\\ge 2ab$.\nHence, \\eqref{ineq:hat-phi-recur} follows from combining the above inequality and \\eqref{ineq:ell-h-mk}.\n\\end{proof}","sha256_utf8":"c0a94a9934e9ad60eba9712b60222e1c6465e6375afa04a5a265c15debd7b9f3"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"PDPB原始对偶复杂度界","source_units":[{"start":48311,"end":48862,"start_line":792,"end_line":801,"raw_tex":"\\begin{theorem}\n    Given $(\\hat x_0,\\bar \\varepsilon) \\in \\R^n \\times \\R_{++}$, assuming that $\\lam$ satisfies\n    \\begin{equation}\\label{ineq:lam}\n        \\frac{\\sqrt{\\bar \\varepsilon d_0}}{M^{3/2}} \\le \\lam \\le \\frac{2d_0^2}{\\bar \\varepsilon},\n    \\end{equation}\n    then the total iteration-complexity of PDPB$(\\hat x_0,\\lam,\\bar \\varepsilon)$ to find $(\\bar x_k,\\bar s_k)$ satisfying \\eqref{ineq:pd} is\n    \\begin{equation}\\label{cmplx:PDPB}\n        {\\cal O}\\left( \\frac{M^2 d_0^2}{\\bar \\varepsilon^2} + 1\\right).\n    \\end{equation}\n\\end{theorem}","sha256_utf8":"57b1afbb657153e5717090c82f51f0dd302474bb919aee4747268eb9679a6831"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"总复杂度证明","source_units":[{"start":48864,"end":49904,"start_line":803,"end_line":819,"raw_tex":"\\begin{proof}\n    In view of Proposition \\ref{prop:pdpb}, PDPB takes \n    \\begin{equation}\\label{cmplx:outer}\n        {\\cal O}\\left(\\frac{d_0^2}{\\lam \\bar \\varepsilon}+1\\right)\n    \\end{equation}\n    cycles to find $(\\bar x_k,\\bar s_k)$ satisfying \\eqref{ineq:pd}.\n    It follows from Proposition \\ref{prop:inner} and Lemma \\ref{lem:t1} that for every cycle in PDPB before termination, the number of iterations in the cycle is \n    \\[\n    {\\cal O}\\left(\\frac{\\sqrt{Md_0 + \\lam M^2}}{\\sqrt{\\bar \\varepsilon}} + \\frac{\\lam M^2}{\\bar \\varepsilon} + 1\\right)=\n    {\\cal O}\\left(\\frac{\\sqrt{Md_0}}{\\sqrt{\\bar \\varepsilon}} + \\frac{\\lam M^2}{\\bar \\varepsilon} + 1\\right),\n    \\]\n    which together with the assumption that $\\sqrt{\\bar \\varepsilon d_0}/M^{3/2} \\le \\lam$ becomes\n    \\begin{equation}\\label{cmplx:inner}\n        {\\cal O}\\left(\\frac{\\lam M^2}{\\bar \\varepsilon} + 1\\right).\n    \\end{equation}\n    Combining \\eqref{cmplx:outer} and \\eqref{cmplx:inner}, and using \\eqref{ineq:lam}, we conclude that \\eqref{cmplx:PDPB} holds.\n\\end{proof}","sha256_utf8":"5350f226d52fb67e9130c742e37f987f44acd8ec0bd37eb556101c6ed1535056"}]},{"id":"S003","semantic_type":"assumption","kind":"assumption","title":"初值容差与步长区间","source_units":[{"start":48327,"end":48577,"start_line":793,"end_line":796,"raw_tex":"    Given $(\\hat x_0,\\bar \\varepsilon) \\in \\R^n \\times \\R_{++}$, assuming that $\\lam$ satisfies\n    \\begin{equation}\\label{ineq:lam}\n        \\frac{\\sqrt{\\bar \\varepsilon d_0}}{M^{3/2}} \\le \\lam \\le \\frac{2d_0^2}{\\bar \\varepsilon},\n    \\end{equation}\n","sha256_utf8":"e3fe2dac1a62ec1c48a152cafd9e1253596770dd2af603c99d924e4b48ff5fdb"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"界定外层周期数","source_units":[{"start":48878,"end":49129,"start_line":804,"end_line":808,"raw_tex":"    In view of Proposition \\ref{prop:pdpb}, PDPB takes \n    \\begin{equation}\\label{cmplx:outer}\n        {\\cal O}\\left(\\frac{d_0^2}{\\lam \\bar \\varepsilon}+1\\right)\n    \\end{equation}\n    cycles to find $(\\bar x_k,\\bar s_k)$ satisfying \\eqref{ineq:pd}.\n","sha256_utf8":"02c9105da2f8714ec951eb435138c61c9354d517191fc58908ecdbbd38f69eee"}]},{"id":"S005","semantic_type":"proof_step","kind":"proof_step","title":"界定每周期内层迭代数","source_units":[{"start":49129,"end":49537,"start_line":809,"end_line":813,"raw_tex":"    It follows from Proposition \\ref{prop:inner} and Lemma \\ref{lem:t1} that for every cycle in PDPB before termination, the number of iterations in the cycle is \n    \\[\n    {\\cal O}\\left(\\frac{\\sqrt{Md_0 + \\lam M^2}}{\\sqrt{\\bar \\varepsilon}} + \\frac{\\lam M^2}{\\bar \\varepsilon} + 1\\right)=\n    {\\cal O}\\left(\\frac{\\sqrt{Md_0}}{\\sqrt{\\bar \\varepsilon}} + \\frac{\\lam M^2}{\\bar \\varepsilon} + 1\\right),\n    \\]\n","sha256_utf8":"41d64ea99883159fc0a22fea80c2b2ab53951b29ba4c274145bf2ad6fb80e9b7"}]},{"id":"S006","semantic_type":"proof_step","kind":"proof_step","title":"利用步长下界简化内层界","source_units":[{"start":49537,"end":49763,"start_line":814,"end_line":817,"raw_tex":"    which together with the assumption that $\\sqrt{\\bar \\varepsilon d_0}/M^{3/2} \\le \\lam$ becomes\n    \\begin{equation}\\label{cmplx:inner}\n        {\\cal O}\\left(\\frac{\\lam M^2}{\\bar \\varepsilon} + 1\\right).\n    \\end{equation}\n","sha256_utf8":"436e3a2a90d944ea6270ef29b7a427702d0764f010005db4e42ca08e1ac1f10d"}]},{"id":"S007","semantic_type":"proof_step","kind":"proof_step","title":"合并内外层复杂度","source_units":[{"start":49763,"end":49893,"start_line":818,"end_line":818,"raw_tex":"    Combining \\eqref{cmplx:outer} and \\eqref{cmplx:inner}, and using \\eqref{ineq:lam}, we conclude that \\eqref{cmplx:PDPB} holds.\n","sha256_utf8":"0f3d373930bf59592381ae1258e20e02cfbd883fb7b592ad99128831fd7f4fca"}]},{"id":"S008","semantic_type":"problem_setting","kind":"problem_setting","title":"凸非光滑复合优化问题","source_units":[{"start":9135,"end":9493,"start_line":243,"end_line":247,"raw_tex":"This paper considers two nonsmooth problems with convex structures: 1) the convex nonsmooth composite optimization (CNCO) problem\n\\begin{equation}\\label{eq:problem}\n    \\phi_*:=\\min \\{\\phi(x):=f(x)+h(x): x \\in \\R^n\\},\n\\end{equation}\nwhere $f,h:\\R^n\\to \\R \\cup \\{+\\infty\\}$ are proper lower semi-continuous convex functions such that $\\dom h \\subset \\dom f$;\n","sha256_utf8":"61b250a5db4e75a1ac99e9e0ac17d60bea27a7360b2bc3045a8dfc4f86ac15d3"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"欧氏空间与强凸性","source_units":[{"start":19878,"end":20669,"start_line":314,"end_line":320,"raw_tex":"    Let $\\R$ denote the set of real numbers. Let $\\R_{++}$ denote the set of positive real numbers. Let $\\R^n$ denote the standard $n$-dimensional Euclidean space equipped with inner product and norm denoted by $\\left\\langle \\cdot,\\cdot\\right\\rangle $ and $\\|\\cdot\\|$, respectively. \n    \n    For given $f: \\R^n\\rightarrow (-\\infty,+\\infty]$, let $\\dom f:=\\{x \\in \\R^n: f (x) <\\infty\\}$ denote the effective domain of $f$. We say $f$ is proper if $\\dom f \\ne \\emptyset$. A proper function $f: \\R^n\\rightarrow (-\\infty,+\\infty]$ is $\\mu$-strongly convex for some $\\mu > 0$ if for every $x, y \\in \\dom f$ and $t \\in [0,1]$\n    \\[\n    f(t x+(1-t) y)\\le t f(x)+(1-t)f(y) - \\frac{t(1-t) \\mu}{2}\\|x-y\\|^2.\n    \\]\nLet $\\bConv{n}$ denote the set of all proper lower-semicontinuous convex functions.\n","sha256_utf8":"2ee6ac604f2fd9c96e1dad186ebff31974801c3bb1ad9f0fa0d3eea6820c8170"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"次微分与线性化","source_units":[{"start":20669,"end":21350,"start_line":321,"end_line":330,"raw_tex":"\tFor $\\varepsilon \\ge 0$, the \\emph{$\\varepsilon$-subdifferential} of $ f $ at $x \\in \\dom f$ is denoted by\n\t\\begin{equation}\\label{def:subdiff}\n\t    \\partial_\\varepsilon f (x):=\\left\\{ s \\in\\R^n: f(y)\\geq f(x)+\\left\\langle s,y-x\\right\\rangle -\\varepsilon, \\forall y\\in\\R^n\\right\\}.\n\t\\end{equation}\n\tWe denote the subdifferential of $f$ at $x \\in \\dom f$ by $\\partial f (x)$, which is the set  $\\partial_0 f(x)$ by definition.\n    For a given subgradient\n$f'(x) \\in \\partial f(x)$, we denote the linearization of convex function $f$ at $x$ by $\\ell_f(\\cdot;x)$, which is defined as\n\\begin{equation}\\label{linfdef}\n\\ell_f(\\cdot;x):=f(x)+\\langle f'(x),\\cdot-x\\rangle.\n\\end{equation}\n","sha256_utf8":"bbd4fb8fbb28dcaaa8f3a647a096868f6eb2c88b67903ab31c999642d2249450"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"下确界卷积","source_units":[{"start":21350,"end":21569,"start_line":331,"end_line":335,"raw_tex":"The infimum convolution of proper functions $f_1, f_2: \\R^n\\rightarrow (-\\infty,+\\infty]$ is given by\n\\begin{equation}\\label{def:infconv}\n    (f_1 \\square f_2)(x)\n= \\min_{u\\in \\R^n}\\{f_1(u) + f_2(x-u)\\}.\n\\end{equation}\n","sha256_utf8":"0e0599fd52ad70472de495f65eb5ccc7b6ceabfab5f8a913b500dc5fe826e904"}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","title":"次梯度有界与最优解存在","source_units":[{"start":21642,"end":22127,"start_line":339,"end_line":345,"raw_tex":"In this section, we consider the CNCO problem \\eqref{eq:problem}.\nMore specifically, we assume the following conditions hold:\n\\begin{itemize}\n    \\item[(A1)] a subgradient oracle, i.e., a function $f': \\dom h \\to \\R^n$ satisfying $f^{\\prime}(x) \\in \\partial f(x)$ for every $x \\in \\dom h$, is available;\n    \\item[(A2)] $\\|f'(x)\\|\\le M$ for every $x\\in \\dom h$ and some $M>0$;\n    \\item[(A3)] the set of optimal solutions $X_*$ of problem \\eqref{eq:problem} is nonempty.\n\\end{itemize}\n","sha256_utf8":"2de1faf188cf0e8cf0698aedf7d1ee807653b31c5a58cf99be09d46528e375e1"}]},{"id":"S013","semantic_type":"derived_fact","kind":"derived_fact","title":"线性化误差上界","source_units":[{"start":22127,"end":22404,"start_line":346,"end_line":353,"raw_tex":"Define the linearization of $f$ at $x\\in \\dom h$, $\\ell_f:\\dom h \\to \\R$ as\n\\[\n\\ell_f(\\cdot;x):=f(x)+\\inner{f'(x)}{\\cdot-x}.\n\\]\nClearly, it follows from (A2) that for every $x, y \\in \\dom h$,\n\\begin{equation}\\label{ineq:f}\n  f(x)-\\ell_f(x ; y) \\le 2 M\\|x-y\\|.  \n\\end{equation}\n","sha256_utf8":"ebe8553d59d860fa9cae25688efa9c8be02162bdcb5b03b3925911c4036e12aa"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"初始最优距离","source_units":[{"start":22404,"end":22649,"start_line":354,"end_line":357,"raw_tex":"For a given initial point $\\hat x_0 \\in \\dom h$, we denote its distance to $X_*$ as\n\\begin{equation}\\label{eq:d0}\n    d_0:=\\left\\|x_0^*-\\hat x_0\\right\\|, \\quad x_0^*:=\\underset{x_* \\in X_*}\\argmin\\{\\left\\| x_*-\\hat x_0\\right\\|\\}.\n\\end{equation}\n","sha256_utf8":"03c9ceed1cc90a5815cdd810fafc429b1218acef291a45b5b60128ff51e27728"}]},{"id":"S015","semantic_type":"algorithm","kind":"algorithm","title":"原始对偶次梯度迭代","source_units":[{"start":22650,"end":23045,"start_line":359,"end_line":362,"raw_tex":"The primal-dual subgradient method denoted by PDS$(\\hat x_0,\\lam)$, where $\\hat x_0\\in \\dom h$ is the initial point and $\\lam>0$ is the prox stepsize, recursively computes\n\\begin{equation}\\label{eq:pds}\n    s_k = f'(x_{k-1}) \\in \\partial f(x_{k-1}), \\quad \\hat x_k = \\underset{u\\in\\R^n}{\\argmin}\\left\\{\\ell_f(u;\\hat x_{k-1}) + h(u) + \\frac{1}{2\\lam}\\|u-\\hat x_{k-1}\\|^2 \\right\\}.\n\\end{equation}\n","sha256_utf8":"b1e77eb6f837dbb486e5418f07b3b91458eab3db4853401e4faa42b5be85b882"}]},{"id":"S016","semantic_type":"problem_setting","kind":"problem_setting","title":"近端子问题","source_units":[{"start":23467,"end":24342,"start_line":369,"end_line":375,"raw_tex":"The PDPB method solves a sequence of proximal subproblems of the form\n\\begin{equation}\\label{eq:phi-lam}\n    \\min_{u \\in \\R^n} \\left\\{\\phi^\\lam(u) :=\\phi(u)+\\frac{1}{2 \\lambda}\\left\\|u-\\hat x_{k-1}\\right\\|^2\\right\\},\n\\end{equation}\nwhere $\\lam$ is the prox stepsize and $\\hat x_{k-1}$ is the prox center in the $k$-th proximal subproblem (or cycle). We omit the index $k$ in $\\phi^\\lam$ since the prox center is always fixed to be $\\hat x_{k-1}$ in this subsection.\nEach proximal subproblem invokes the PDCP scheme to find an approximate solution. Hence, PDPB can be viewed as a generalization of PDS, which only takes one proximal subgradient step (i.e., \\eqref{eq:pds}) to solve every proximal subproblem \\eqref{eq:phi-lam}.\nThe goal of this subsection is to describe the key subroutine PDCP for solving \\eqref{eq:phi-lam} and present its primal-dual convergence analysis.\n","sha256_utf8":"1c7de1c327e3f3386f4d8bfb4d2a7af21de4182f1d6ea3ea79714480757b853f"}]},{"id":"S017","semantic_type":"definition","kind":"definition","title":"内层原始对偶点与模型","source_units":[{"start":24343,"end":25420,"start_line":377,"end_line":385,"raw_tex":"In the rest of this subsection, we consider subproblem \\eqref{eq:phi-lam} with fixed prox center $\\hat x_{k-1}$. For simplicity, we denote $\\hat x_{k-1}$ as $x_0$ from a local perspective within the current cycle, as it is also the initial point of PDCP.  At the $j$-th iteration of PDCP, given some prox stepsize $\\lam>0$ and prox center $x_0$, PDCP computes a primal-dual pair $(x_j,s_j)$ as follows\n\\begin{equation}\\label{eq:xj}\n    x_j=\\underset{u \\in \\R^n}{\\argmin}\\left\\{\\Gamma_j(u) + h(u) +\\frac{1}{2 \\lam}\\|u-x_0\\|^2\\right\\}, \\quad s_j \\in \\partial \\Gamma_j(x_j) \\cap (-\\partial h^\\lam(x_j)),  \n\\end{equation}\nwhere $\\Gamma_j$ is a proper, closed and convex function satisfying $\\Gamma_j \\le f$ for every $j\\ge 1$, and\n\\begin{equation}\\label{eq:hlam}\n    h^\\lam (\\cdot) := h(\\cdot) + \\frac{1}{2\\lam}\\|\\cdot - x_0\\|^2.\n\\end{equation}\nStarting from $\\Gamma_1(\\cdot) = \\ell_f(\\cdot ; x_0)$, and for every $j\\ge 1$,  $\\Gamma_{j+1}$ is obtained from the following generic bundle management (GBM), which is motivated by BU given in Subsection~3.1 of \\cite{liang2024unified}.\n","sha256_utf8":"d23530640d26259c4bc88b1eb737f5384318e0031b52e09f9fbebe048784e157"}]},{"id":"S018","semantic_type":"algorithm","kind":"algorithm","title":"通用束模型更新","source_units":[{"start":25753,"end":26644,"start_line":389,"end_line":406,"raw_tex":"\\begin{algorithm}[H]\n\\caption{Generic Bundle Management, GBM$(\\lambda, \\tau_j, x_0, x_{j}, \\Gamma_j)$}\\label{alg:GBM}\n\\begin{algorithmic}\n\\REQUIRE $(\\lam, \\tau_j)\\in \\R_{++} \\times [0,1]$, $(x_0,x_j) \\in \\R^n \\times \\R^n$, and $\\Gamma_j \\in \\bConv{n}$ satisfying $\\Gamma_j \\le f$\n\n\\STATE $\\bullet$ find a bundle model $\\Gamma_{j+1}\\in \\bConv{n}$ satisfying $\\Gamma_{j+1} \\le f$ and \n        \\begin{equation}\\label{def:Gammaj}\n\t     \\Gamma_{j+1}(\\cdot) \\ge \\tau_j \\bar \\Gamma_j(\\cdot) + (1-\\tau_j)\\ell_f(\\cdot; x_j),\n\t\\end{equation}\nwhere $\\bar \\Gamma_j \\in \\bConv{n}$ satisfies $\\bar \\Gamma_j \\le f$ and\n        \\begin{equation}\\label{def:bar Gamma}\n            \\bar \\Gamma_j(x_j) = \\Gamma_j(x_j), \\quad x_j = \\underset{u\\in \\R^n}\\argmin \\left \\{\\bar \\Gamma_j(u) + h(u) + \\frac{1}{2\\lam}\\|u-x_0\\|^2\\right\\}.\n        \\end{equation}\n\n\\ENSURE $\\Gamma_{j+1}$.\n\n\\end{algorithmic} \n\\end{algorithm}","sha256_utf8":"91bb65095751239966d9b36f80d0385b31a82186ffde7f8bc391b12c46c46bae"}]},{"id":"S019","semantic_type":"definition","kind":"definition","title":"辅助点与内层停止间隙","source_units":[{"start":26649,"end":27307,"start_line":409,"end_line":419,"raw_tex":"PDCP computes an auxiliary sequence $\\{\\tx_j\\}$ to determine termination. \nIt generated $\\tx_j$ such that\n\\begin{equation}\\label{ineq:txj}\n    \\tilde x_1 = x_1, \\quad \\mbox{and} \\quad \\phi^\\lam(\\tx_{j+1}) \\le \\tau_j\\phi^\\lam(\\tx_j) + (1-\\tau_j)\\phi^\\lam(x_{j+1}), \\, \\forall j\\ge 1,\n\\end{equation}\nwhere $\\phi^\\lam$ is as in \\eqref{eq:phi-lam}. \nPDCP also computes\n\\begin{equation}\\label{eq:mj}\n    m_j =\\underset{u \\in \\R^n}{\\min}\\left\\{\\Gamma_j(u) + h(u) +\\frac{1}{2 \\lam}\\|u-x_0\\|^2\\right\\}, \\quad t_j=\\phi^\\lam \\left(\\tilde x_j\\right)-m_j.\n\\end{equation}\nFor given tolerance $\\varepsilon>0$, PDCP terminates the current cycle when $t_j \\le \\varepsilon$.\n","sha256_utf8":"38039996b3b201d36fc6367c8ee370dc6c7ddf7df1d3bac05d687a237191ff53"}]},{"id":"S020","semantic_type":"algorithm","kind":"algorithm","title":"原始对偶切平面子程序","source_units":[{"start":27341,"end":27997,"start_line":424,"end_line":438,"raw_tex":"\\begin{algorithm}[H]\n\\caption{Primal-Dual Cutting-Plane, PDCP$(x_0,\\lam,\\varepsilon)$}\\label{alg:PDCP}\n\\begin{algorithmic}\n\\REQUIRE given $x_0\\in \\dom h$, $\\lam>0$, $\\varepsilon>0$, set $t_0 = 2\\varepsilon$, $\\Gamma_1(\\cdot)=\\ell_f(\\cdot;x_0)$, and $j=1$;\n\n\\WHILE{$t_{j-1} > \\varepsilon$}\n\n\\STATE {\\bf 1.} compute $(x_j,s_j)$ by \\eqref{eq:xj}, choose $\\tilde x_j$ as in \\eqref{ineq:txj}, and set $t_j$ as in~\\eqref{eq:mj};\n\\STATE {\\bf 2.} select $\\tau_j \\in [0,1]$ and update $\\Gamma_{j+1}$ by GBM$(\\lambda, \\tau_j, x_0, x_{j}, \\Gamma_j)$ and $j \\leftarrow j+1$;\n\n\\ENDWHILE\n\\ENSURE $(x_{j-1}, \\tilde x_{j-1},  s_{j-1})$.\n\n\\end{algorithmic} \n\\end{algorithm}","sha256_utf8":"57666bad01f247e616e660bf486cd23fa3ed86465d47822881eac41c69e6b4f7"}]},{"id":"S021","semantic_type":"definition","kind":"definition","title":"六倍距离球上的受限问题","source_units":[{"start":34089,"end":34570,"start_line":551,"end_line":556,"raw_tex":"Recall the definitions of $d_0$ and $x_0^*$ in \\eqref{eq:d0}. Since $x_0^* \\in B(\\hat x_0, 6d_0)$, which is the ball centered at $\\hat x_0$ and with radius $6d_0$, it is easy to see that to solve \\eqref{eq:problem}, it suffices to solve\n\\begin{equation}\\label{eq:new}\n    \\min \\left\\{\\hat \\phi(x):=f(x)+\\hat h(x): x \\in \\R^n\\right\\}\n    = \\min \\left\\{\\phi(x): x \\in Q\\right\\},\n\\end{equation}\nwhere $\\hat h = h + I_Q$ and $I_Q$ is the indicator function of $Q = B(\\hat x_0, 6d_0)$.\n","sha256_utf8":"a3f879e02e77a91c8ab95b914787c86257addc5a0a7bd2946c4a4fe15f1da4f5"}]},{"id":"S022","semantic_type":"algorithm","kind":"algorithm","title":"PDPB及原始对偶平均","source_units":[{"start":34734,"end":35282,"start_line":561,"end_line":572,"raw_tex":"\\begin{algorithm}[H]\n\\caption{Primal-Dual Proximal Bundle, PDPB$(\\hat x_0,\\lam,\\bar \\varepsilon)$}\\label{alg:PDPB}\n\\begin{algorithmic}\n\\REQUIRE given $(\\hat x_0,\\lam,\\bar \\varepsilon) \\in \\dom h \\times \\R_{++} \\times \\R_{++}$\n\\FOR{$k=1,2,\\cdots$}\n\\STATE $\\bullet$ call oracle $(\\hat x_k,\\tx_k,s_k) = \\text{PDCP}(\\hat x_{k-1},\\lam, \\bar \\varepsilon)$ and compute \n\\begin{equation}\\label{eq:bxk}\n    \\bar x_k = \\frac{1}{k}\\sum_{i=1}^{k} \\tx_i , \\quad \\bar s_k = \\frac{1}{k}\\sum_{i=1}^{k} s_i.\n\\end{equation}\n\\ENDFOR\n\\end{algorithmic} \n\\end{algorithm}","sha256_utf8":"4fd77d6297dc54993b06c31d23002fee37c45cc8d86350331aebbbd80d184dd3"}]},{"id":"S023","semantic_type":"notation","kind":"notation","title":"周期编号与内外层变量对应","source_units":[{"start":35847,"end":36698,"start_line":581,"end_line":589,"raw_tex":"Recall from Subsection~\\ref{subsec:PDCP} that \\eqref{eq:phi-lam} is approximately solved by invoking PDCP. The (global) iteration indices in PDCP are regarded as the $k$-th cycle, denoted by ${\\cal C}_k=\\left\\{i_k, \\ldots, j_k\\right\\}$, where $j_k$ is the last iteration index of the $k$-th call to PDCP, $j_0=0$, and $i_k = j_{k-1}+1$.\nHence, for the $j_k$-th iteration of PDCP, we have\n\\begin{equation}\\label{eq:translate}\n    \\hat x_k = x_{j_k}, \\quad \\tilde x_k = \\tilde x_{j_k}, \\quad s_k = s_{j_k}, \\quad \\Gamma_k  = \\Gamma_{j_k}, \\quad m_k = m_{j_k},\n\\end{equation}\nand \\eqref{eq:xj} becomes\n\\begin{equation}\\label{eq:xk}\n    \\hat x_k=\\underset{u \\in \\R^n}{\\argmin}\\left\\{\\Gamma_k(u) + h(u) +\\frac{1}{2 \\lambda}\\left\\|u-\\hat x_{k-1}\\right\\|^2\\right\\}, \\quad s_k \\in \\partial \\Gamma_k(\\hat x_k) \\cap (-\\partial h^\\lam(\\hat x_k)).\n\\end{equation}\n","sha256_utf8":"a342af5e590c8cb9a0bbc130f38c56faf85598f4e27a5a4c97f87ac31b7a5052"}]},{"id":"S024","semantic_type":"definition","kind":"definition","title":"附录次梯度法的受限问题","source_units":[{"start":122385,"end":122874,"start_line":1906,"end_line":1911,"raw_tex":"Recall the definitions of $d_0$ and $x_0^*$ in \\eqref{eq:d0}. Since $x_0^* \\in B(\\hat x_0, 4d_0)$, which is the ball centered at $\\hat x_0$ and with radius $4d_0$, it is easy to see that to solve \\eqref{eq:problem}, it suffices to solve\n\\begin{equation}\\label{eq:constrained}\n    \\min \\left\\{\\hat \\phi(x):=f(x)+\\hat h(x): x \\in \\R^n\\right\\}\n    = \\min \\left\\{\\phi(x): x \\in Q\\right\\},\n\\end{equation}\nwhere $\\hat h = h + I_Q$ and $I_Q$ is the indicator function of $Q = B(\\hat x_0, 4d_0)$.\n","sha256_utf8":"820267a124983e481c7d9c252759346c4a30dceba7c7702accb2d4cb516c5621"}]},{"id":"S025","semantic_type":"algorithm","kind":"algorithm","title":"修正原始对偶次梯度法","source_units":[{"start":122874,"end":123262,"start_line":1912,"end_line":1915,"raw_tex":"Hence, it is convenient to consider a slightly modified version of PDS$(\\hat x_0,\\lam)$ with $h$ replaced by $\\hat h$ in \\eqref{eq:pds}, denoted by MPDS$(\\hat x_0,\\lam)$, i.e.,\n\\begin{equation}\\label{eq:pds-1}\n    s_k = f'(\\hat x_{k-1}), \\quad \\hat x_k = \\underset{u\\in\\R^n}{\\argmin}\\left\\{\\ell_f(u;\\hat x_{k-1}) + \\hat h(u) + \\frac{1}{2\\lam}\\|u-\\hat x_{k-1}\\|^2 \\right\\}.\n\\end{equation}\n","sha256_utf8":"b54a892db79ef6223cbf74d1c2ff716cc9b05ed712f37e7fcaf286680382ace4"}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","title":"辅助束模型的强凸下界","source_units":[{"start":30159,"end":30422,"start_line":483,"end_line":488,"raw_tex":"\\begin{lemma}\\label{lem:Gamma}\nFor every $j \\ge 1$, there exists $\\bar \\Gamma_j \\in \\bConv{n}$ such that for every $u\\in \\R^n$,\n\\begin{equation}\\label{ineq:bGammaj}\n    \\bar{\\Gamma}_j(u) + h^\\lam(u) \\ge m_j + \\frac{1}{2\\lam}\\|u-x_j\\|^2.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"332caa9f0ad73aa0d96682658d53b903b981be937069d823300b44f897ceed98"}]},{"id":"S027","semantic_type":"proof","kind":"proof","title":"模型强凸下界证明","source_units":[{"start":30424,"end":30944,"start_line":490,"end_line":496,"raw_tex":"\\begin{proof}\n    Since the objective function in \\eqref{def:bar Gamma} is $\\lam^{-1}$-strongly convex, it follows from \\eqref{def:bar Gamma} that \n    \\[\n    \\bar \\Gamma_j(u) + h(u) + \\frac{1}{2\\lam}\\|u-x_0\\|^2 \\ge \\bar \\Gamma_j(x_j) + h(x_j) + \\frac{1}{2\\lam}\\|x_j-x_0\\|^2 + \\frac{1}{2\\lam}\\|u-x_j\\|^2.\n    \\]\n    Inequality \\eqref{ineq:bGammaj} immediately follows from the above inequality, the definition of $m_j$ in \\eqref{eq:mj}, and the fact that $h^\\lam(\\cdot) = h(\\cdot) + \\|\\cdot-x_0\\|^2/(2\\lam)$.\n\\end{proof}","sha256_utf8":"258dcd2c2a4c931184b0ad8a457256bf942c40f2b4c5cf83a7aa36f20403939d"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"应用辅助模型目标的强凸性","source_units":[{"start":30438,"end":30736,"start_line":491,"end_line":494,"raw_tex":"    Since the objective function in \\eqref{def:bar Gamma} is $\\lam^{-1}$-strongly convex, it follows from \\eqref{def:bar Gamma} that \n    \\[\n    \\bar \\Gamma_j(u) + h(u) + \\frac{1}{2\\lam}\\|u-x_0\\|^2 \\ge \\bar \\Gamma_j(x_j) + h(x_j) + \\frac{1}{2\\lam}\\|x_j-x_0\\|^2 + \\frac{1}{2\\lam}\\|u-x_j\\|^2.\n    \\]\n","sha256_utf8":"41027212f977a1c79f95613b87278100756f5e3471dc4a549deb56b1e0c88d3e"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"代入模型最优值与近端项","source_units":[{"start":30736,"end":30933,"start_line":495,"end_line":495,"raw_tex":"    Inequality \\eqref{ineq:bGammaj} immediately follows from the above inequality, the definition of $m_j$ in \\eqref{eq:mj}, and the fact that $h^\\lam(\\cdot) = h(\\cdot) + \\|\\cdot-x_0\\|^2/(2\\lam)$.\n","sha256_utf8":"4682a897d54dbdcf24826b01c26f2c3c33479ac9f82af8abf0c1e2adccbaf897"}]},{"id":"S030","semantic_type":"proposition","kind":"proposition","title":"PDCP间隙收敛与复杂度","source_units":[{"start":31171,"end":31650,"start_line":501,"end_line":511,"raw_tex":"\\begin{proposition}\\label{prop:inner}\nConsidering Algorithm \\ref{alg:PDCP} with $\\tau_j = j/(j+2)$, then for every $j \\ge 1$, we have\n\\begin{equation}\\label{ineq:tj}\n    t_j\\le \\frac{2t_1}{j(j+1)} + \\frac{16\\lam M^2}{j+1},\n\\end{equation}\nwhere $t_j$ is as in \\eqref{eq:mj}.\nMoreover, the number of iterations for PDCP to obtain $t_j \\le \\varepsilon$ is at most\n\\[\n{\\cal O}\\left(\\frac{\\sqrt{t_1}}{\\sqrt{\\varepsilon}} + \\frac{\\lam M^2}{\\varepsilon} + 1\\right).\n\\]\n\\end{proposition}","sha256_utf8":"c4113160384754314eb4d13decbb6c32e815634573ee8c649a03dc107154690b"}]},{"id":"S031","semantic_type":"proof","kind":"proof","title":"内层间隙收敛证明","source_units":[{"start":31652,"end":34019,"start_line":513,"end_line":547,"raw_tex":"\\begin{proof}\nWe first note that for every $j\\ge 1$, $\\tau_j=A_j/A_{j+1}$ where $A_{j+1}=A_j+j+1$ and $A_0=0$, i.e., $A_j=j(j+1)/2$ for every $j\\ge 0$.\nIt follows from this observation, the definition of $m_j$ in \\eqref{eq:mj}, and relation \\eqref{def:Gammaj} that\n\\begin{align*}\n    A_{j+1}m_{j+1} & \\stackrel{\\eqref{eq:mj}}{=} A_{j+1}(\\Gamma_{j+1}+h^\\lam)(x_{j+1}) \\\\\n    &\\stackrel{\\eqref{def:Gammaj}}{\\ge} A_j \\left[(\\bar \\Gamma_j+ h^\\lam)(x_{j+1}) \\right] + (j+1)\\left[\\ell_f(x_{j+1};x_j) + h^\\lam(x_{j+1})\\right].\n\\end{align*}\nApplying Lemma~\\ref{lem:Gamma} in the above inequality and using \\eqref{ineq:f}, we have\n\\begin{align*}\n    A_{j+1}m_{j+1} \n    &\\stackrel{\\eqref{ineq:bGammaj}}{\\ge} A_j \\left[ m_j+\\frac{1}{2\\lam}\\|x_{j+1} - x_j\\|^2 \\right] + (j+1)\\left[\\ell_f(x_{j+1};x_j) + h^\\lam(x_{j+1})\\right] \\\\\n    &= A_j m_j + (j+1)\\left[\\ell_f(x_{j+1};x_j) + h^\\lam(x_{j+1}) + \\frac{A_j}{2\\lam (j+1)}\\|x_{j+1} - x_j\\|^2 \\right] \\\\\n    &\\stackrel{\\eqref{ineq:f}}{\\ge} A_j m_j + (j+1)\\left[\\phi^\\lam(x_{j+1}) - 2M\\|x_{j+1}-x_j\\| + \\frac{A_j}{2\\lam (j+1)}\\|x_{j+1} - x_j\\|^2 \\right] \\\\\n    &\\ge A_j m_j + (j+1)\\phi^\\lam(x_{j+1}) - \\frac{2\\lam M^2 (j+1)^2}{A_j}\n\\end{align*}\nwhere the last inequality is due to the Young's inequality $a^2+b^2\\ge 2ab$.\nIt follows from the fact that $A_j=j(j+1)/2$ that for every $j\\ge 1$,\n\\[\n    A_{j+1}m_{j+1} \\ge A_j m_j + (j+1)\\phi^\\lam(x_{j+1}) - 8\\lam M^2.\n\\]\nReplacing the index $j$ in the above inequality by $i$, summing the resulting inequality from $i=1$ to $j-1$, and using the definition of $t_j$ in \\eqref{eq:mj} and the fact that $\\tx_1=x_1$, we obtain\n\\begin{align*}\n    A_j m_j\n    &\\ge A_1 m_1 + 2 \\phi^\\lam(x_2) + \\dots + j\\phi^\\lam(x_j) - 8\\lam M^2 (j-1) \\\\\n    &\\stackrel{\\eqref{eq:mj}}{=} -A_1 t_1 + A_1 \\phi^\\lam(x_1) + 2 \\phi^\\lam(x_2) + \\dots + j\\phi^\\lam(x_j) - 8\\lam M^2 (j-1) \\\\\n    &\\stackrel{\\eqref{ineq:txj}}{\\ge} -A_1 t_1 + A_j \\phi^\\lam(\\tx_j) - 8\\lam M^2 (j-1),\n\\end{align*}\nwhere the last inequality follows from \\eqref{ineq:txj} and the fact that $A_j=A_{j-1}+j$.\nRearranging the terms and using the definition of $t_j$ in \\eqref{eq:mj} again, we have\n\\begin{equation}\\label{ineq:Aj-tj}\n    A_j t_j \\le A_1 t_1 + 8\\lam M^2 (j-1).\n\\end{equation}\nHence, \\eqref{ineq:tj} follows from the fact that $A_j=j(j+1)/2$.\nFinally, the complexity bound immediately follows from \\eqref{ineq:tj}.\n\\end{proof}","sha256_utf8":"ca59a5c045500495415d9c37d9417e18220e40e444275a1ef601c490f2ef3c24"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"构造权重并比较模型最优值","source_units":[{"start":31666,"end":32185,"start_line":514,"end_line":519,"raw_tex":"We first note that for every $j\\ge 1$, $\\tau_j=A_j/A_{j+1}$ where $A_{j+1}=A_j+j+1$ and $A_0=0$, i.e., $A_j=j(j+1)/2$ for every $j\\ge 0$.\nIt follows from this observation, the definition of $m_j$ in \\eqref{eq:mj}, and relation \\eqref{def:Gammaj} that\n\\begin{align*}\n    A_{j+1}m_{j+1} & \\stackrel{\\eqref{eq:mj}}{=} A_{j+1}(\\Gamma_{j+1}+h^\\lam)(x_{j+1}) \\\\\n    &\\stackrel{\\eqref{def:Gammaj}}{\\ge} A_j \\left[(\\bar \\Gamma_j+ h^\\lam)(x_{j+1}) \\right] + (j+1)\\left[\\ell_f(x_{j+1};x_j) + h^\\lam(x_{j+1})\\right].\n\\end{align*}\n","sha256_utf8":"ac77f66f7cd0b5c54841ebfa13f1e4d16eaace005d3d7ae57b308342ac234ae8"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"用强凸性吸收线性化误差","source_units":[{"start":32185,"end":32909,"start_line":520,"end_line":528,"raw_tex":"Applying Lemma~\\ref{lem:Gamma} in the above inequality and using \\eqref{ineq:f}, we have\n\\begin{align*}\n    A_{j+1}m_{j+1} \n    &\\stackrel{\\eqref{ineq:bGammaj}}{\\ge} A_j \\left[ m_j+\\frac{1}{2\\lam}\\|x_{j+1} - x_j\\|^2 \\right] + (j+1)\\left[\\ell_f(x_{j+1};x_j) + h^\\lam(x_{j+1})\\right] \\\\\n    &= A_j m_j + (j+1)\\left[\\ell_f(x_{j+1};x_j) + h^\\lam(x_{j+1}) + \\frac{A_j}{2\\lam (j+1)}\\|x_{j+1} - x_j\\|^2 \\right] \\\\\n    &\\stackrel{\\eqref{ineq:f}}{\\ge} A_j m_j + (j+1)\\left[\\phi^\\lam(x_{j+1}) - 2M\\|x_{j+1}-x_j\\| + \\frac{A_j}{2\\lam (j+1)}\\|x_{j+1} - x_j\\|^2 \\right] \\\\\n    &\\ge A_j m_j + (j+1)\\phi^\\lam(x_{j+1}) - \\frac{2\\lam M^2 (j+1)^2}{A_j}\n\\end{align*}\nwhere the last inequality is due to the Young's inequality $a^2+b^2\\ge 2ab$.\n","sha256_utf8":"e7f86ac181f4731c422972c4503547fcae0534128e7b52a65791d53c659a2428"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"得到统一单步误差界","source_units":[{"start":32909,"end":33055,"start_line":529,"end_line":532,"raw_tex":"It follows from the fact that $A_j=j(j+1)/2$ that for every $j\\ge 1$,\n\\[\n    A_{j+1}m_{j+1} \\ge A_j m_j + (j+1)\\phi^\\lam(x_{j+1}) - 8\\lam M^2.\n\\]\n","sha256_utf8":"e24e82bc2b697da45b40d9a463819ac997df121f99bbbc8b377beed81d4dffad"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"累加模型递推并使用辅助点条件","source_units":[{"start":33055,"end":33689,"start_line":533,"end_line":540,"raw_tex":"Replacing the index $j$ in the above inequality by $i$, summing the resulting inequality from $i=1$ to $j-1$, and using the definition of $t_j$ in \\eqref{eq:mj} and the fact that $\\tx_1=x_1$, we obtain\n\\begin{align*}\n    A_j m_j\n    &\\ge A_1 m_1 + 2 \\phi^\\lam(x_2) + \\dots + j\\phi^\\lam(x_j) - 8\\lam M^2 (j-1) \\\\\n    &\\stackrel{\\eqref{eq:mj}}{=} -A_1 t_1 + A_1 \\phi^\\lam(x_1) + 2 \\phi^\\lam(x_2) + \\dots + j\\phi^\\lam(x_j) - 8\\lam M^2 (j-1) \\\\\n    &\\stackrel{\\eqref{ineq:txj}}{\\ge} -A_1 t_1 + A_j \\phi^\\lam(\\tx_j) - 8\\lam M^2 (j-1),\n\\end{align*}\nwhere the last inequality follows from \\eqref{ineq:txj} and the fact that $A_j=A_{j-1}+j$.\n","sha256_utf8":"d574367fde7c70a279e2ab7eb9aac76a0a7d8b7405776aca3c637b5983dd3304"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"推出间隙速率与停止复杂度","source_units":[{"start":33689,"end":34008,"start_line":541,"end_line":546,"raw_tex":"Rearranging the terms and using the definition of $t_j$ in \\eqref{eq:mj} again, we have\n\\begin{equation}\\label{ineq:Aj-tj}\n    A_j t_j \\le A_1 t_1 + 8\\lam M^2 (j-1).\n\\end{equation}\nHence, \\eqref{ineq:tj} follows from the fact that $A_j=j(j+1)/2$.\nFinally, the complexity bound immediately follows from \\eqref{ineq:tj}.\n","sha256_utf8":"935edae456a3c0dfea51fecd45ea43ec807bc83eb391d3cbf2e3a46f5e27c5d1"}]},{"id":"S037","semantic_type":"lemma","kind":"lemma","title":"PDPB模型次梯度与停止性质","source_units":[{"start":36820,"end":37325,"start_line":593,"end_line":600,"raw_tex":"\\begin{lemma}\\label{ser-lemma1}\n    The following statements hold for every $k \\ge 1$:\n    \\begin{enumerate}\n        \\item [(a)] $\\Gamma_k \\le f$ and $f^*\\le \\Gamma_k^*$;\n        \\item [(b)] $s_k \\in \\partial \\Gamma_k(\\hat x_k)$ and $g_k \\in \\partial h(\\hat x_k)$ where $g_k = - s_k + (\\hat x_{k-1}-\\hat x_k)/\\lam$;\n        \\item [(c)] $\\phi^\\lam(\\tilde x_k) \\le \\bar \\varepsilon + m_k = \\bar \\varepsilon + (\\Gamma_k + h)(\\hat x_k) + \\|\\hat x_k - \\hat x_{k-1}\\|^2/(2\\lam)$.\n    \\end{enumerate}\n\\end{lemma}","sha256_utf8":"84facb44c24a0af445e031fd5937ffe2fb4f52b8197b609d105060d745584ba7"}]},{"id":"S038","semantic_type":"proof","kind":"proof","title":"PDPB基本性质证明","source_units":[{"start":37327,"end":37871,"start_line":602,"end_line":608,"raw_tex":"\\begin{proof}\n    (a) It follows from the facts that $\\Gamma_j \\le f$ for every $j\\ge 1$ and $\\Gamma_k=\\Gamma_{j_k}$ that the first inequality holds. The second one immediately follows from the first one and the definition of the conjugate function.\n\n    (b) This statement follows from \\eqref{eq:xk} and the definitions in \\eqref{eq:translate}.\n\n    (c) This statement follows from the termination criterion of the $k$-th cycle, that is, $t_{j_k}\\le \\bar \\varepsilon$, and the definitions in \\eqref{eq:mj} and \\eqref{eq:translate}.\n\\end{proof}","sha256_utf8":"bcd9c03578202f7009093057dbd4b316d2148f8044d875426e6ac28c7bd03fef"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"传递模型下界与共轭次序","source_units":[{"start":37341,"end":37577,"start_line":603,"end_line":603,"raw_tex":"    (a) It follows from the facts that $\\Gamma_j \\le f$ for every $j\\ge 1$ and $\\Gamma_k=\\Gamma_{j_k}$ that the first inequality holds. The second one immediately follows from the first one and the definition of the conjugate function.\n","sha256_utf8":"453475afdccf78706a475b00ce6c7e9685296e0041fcb95ee23c9df36538ab7d"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"由最优性得到两类次梯度","source_units":[{"start":37578,"end":37673,"start_line":605,"end_line":605,"raw_tex":"    (b) This statement follows from \\eqref{eq:xk} and the definitions in \\eqref{eq:translate}.\n","sha256_utf8":"656c4df26929575e40d2833e35e5d0e4aab91bd8a36f6a7f5674b28a0a6f83ef"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"将周期停止条件转为函数值界","source_units":[{"start":37674,"end":37860,"start_line":607,"end_line":607,"raw_tex":"    (c) This statement follows from the termination criterion of the $k$-th cycle, that is, $t_{j_k}\\le \\bar \\varepsilon$, and the definitions in \\eqref{eq:mj} and \\eqref{eq:translate}.\n","sha256_utf8":"2f3b870e9a0f99a8b4796b9048c5ba16a612c96b124ae0c251a3ae4d7a9180c5"}]},{"id":"S042","semantic_type":"proposition","kind":"proposition","title":"平均点的共轭间隙界","source_units":[{"start":38018,"end":38322,"start_line":612,"end_line":618,"raw_tex":"\\begin{proposition}\\label{prop:pd-key}\nFor every $k \\ge 1$, we have\n\\begin{equation}\\label{ineq:conjugate}\n    \\phi(\\bar{x}_k) + f^*(\\bar s_k) + \\hat h ^*(-\\bar s_k) \\le \\bar \\varepsilon + \\frac{18 d_0^2}{\\lam k}.\n\\end{equation}\nwhere $\\bar x_k$ and $\\bar s_k$ are as in \\eqref{eq:bxk}.\n\\end{proposition}","sha256_utf8":"a3446358430845e2b69b50d993bb01d27d2c1fa9740464136b19eaa67ef2fb3b"}]},{"id":"S043","semantic_type":"proof","kind":"proof","title":"平均共轭间隙证明","source_units":[{"start":38324,"end":41722,"start_line":620,"end_line":680,"raw_tex":"\\begin{proof}\nIt follows from Lemma \\ref{ser-lemma1}(b) and Theorem 4.20 of \\cite{Beck2017} that for every $k \\ge 1$,\n\\[ \n\\Gamma_k(\\hat x_k) +  \\Gamma_k^*( s_k) = \\inner{\\hat x_k}{s_k}, \\quad  h(\\hat x_k) +  h^*(g_k) = \\inner{\\hat x_k}{g_k}.\n\\]\nSumming the above two equations yields\n\\begin{equation}\\label{eq:Gamma-h}\n    (\\Gamma_k + h)(\\hat x_k) + {\\Gamma}_k^*(s_k) + h^*(g_k) = \\frac{1}{\\lam}\\inner{\\hat x_k}{\\hat x_{k-1}-\\hat x_k}.\n\\end{equation}\nUsing the above identity and Lemma \\ref{ser-lemma1}(a) and (c), we have for every $k \\ge 1$,\n\\begin{align*}\n& \\phi(\\tilde x_k) + f^*(s_k) +  {h}^*(g_k) \\le \\phi(\\tilde x_k) + \\Gamma_k^*(s_k) +  {h}^*(g_k) \\\\\n\\le & \\bar \\varepsilon +(\\Gamma_k + h)(\\hat x_k) + \\frac{1}{2\\lam}\\|\\hat x_k-\\hat x_{k-1}\\|^2 + \\Gamma_k^*(s_k) + {h}^*(g_k)\\\\\n\\stackrel{\\eqref{eq:Gamma-h}}{=}& \\bar \\varepsilon + \\frac{1}{2\\lam}(\\|\\hat x_{k-1}\\|^2-\\|\\hat x_k\\|^2).\n\\end{align*}\nReplacing the index $k$ in the above inequality by $i$, summing the resulting inequality from $i=1$ to $k$, and using convexity and the definitions in \\eqref{eq:bxk}, we obtain \n\\begin{equation}\\label{ineq:avg}\n    \\phi(\\bar x_k) + f^*(\\bar s_k) + h^*(\\bar g_k)\\le \\bar \\varepsilon + \\frac{1}{2\\lam k }\\left(\\|\\hat x_0\\|^2-\\|\\hat x_k\\|^2\\right),\n\\end{equation}\nwhere $\\bar g_k = (\\sum_{i=1}^k g_i)/k$.\nDefine\n\\begin{equation}\\label{eq:zeta}\n    \\eta_k(u) = \\frac{1}{2\\lam k} \\|u-\\hat x_0\\|^2, \\quad \\hat \\eta_k(u) = \\eta_k(u) - I_Q(u).\n\\end{equation}\nNoting that $\\nabla \\eta_k(\\hat x_k) = (\\hat x_k - \\hat x_0)/(\\lam k) = -\\bar g_k - \\bar s_k$, and hence it follows from Theorem 4.20 of \\cite{Beck2017} that\n\\[\n\\eta_k^*(-\\bar g_k - \\bar s_k) = \\frac{1}{\\lam k}\\inner{\\hat x_k-\\hat x_0}{\\hat x_k} - \\eta_k(\\hat x_k) = \\frac{1}{2\\lam k}\\left(\\|\\hat x_k\\|^2 - \\| \\hat x_0\\|^2\\right).\n\\]\nThe above observation and \\eqref{ineq:avg} together imply that\n\\begin{equation}\\label{ineq:bar}\n    \\phi(\\bar x_k) + f^*(\\bar s_k) + h^*(\\bar g_k) + \\eta_k^*(-\\bar g_k - \\bar s_k) \\le \\bar \\varepsilon.\n\\end{equation}\nIt follows from Theorem 4.17 of \\cite{Beck2017} and the definition of infimum convolution in \\eqref{def:infconv} that\n\\[\n(h + \\eta_k)^*(-\\bar s_k) = (h^* \\square \\eta_k^*)(-\\bar s_k)\n\\stackrel{\\eqref{def:infconv}}= \\min_{u\\in \\R^n}\\{h^*(u) + \\eta_k^*(-\\bar s_k-u)\\}\n\\le h^*(\\bar g_k) + \\eta_k^*(-\\bar g_k - \\bar s_k).\n\\]\nNoting from \\eqref{eq:zeta} that $\\hat h = h + \\eta_k -\\hat \\eta_k$ and applying Theorem 4.17 of \\cite{Beck2017} again, we obtain\n\\begin{align*}\n    \\hat h ^*(-\\bar s_k) &= [(h + \\eta_k)^* \\square (-\\hat \\eta_k)^*](-\\bar s_k)\n= \\min_{u\\in \\R^n}\\{(h + \\eta_k)^*(u) + (-\\hat \\eta_k)^*(-\\bar s_k-u)\\}\\\\\n&\\le (h + \\eta_k)^*(-\\bar s_k) + (-\\hat \\eta_k)^*(0).\n\\end{align*}\nSumming the above two inequalities, we have\n\\[\n\\hat h ^*(-\\bar s_k) \\le h^*(\\bar g_k) + \\eta_k^*(-\\bar g_k - \\bar s_k) + (-\\hat \\eta_k)^*(0),\n\\]\nwhich together with \\eqref{ineq:bar} implies that\n\\[\n\\phi(\\bar x_k) + f^*(\\bar s_k) + \\hat h ^*(-\\bar s_k) \\le \\bar \\varepsilon + (-\\hat \\eta_k)^*(0).\n\\]\nIt follows from \\eqref{eq:zeta} that\n\\[\n(-\\hat \\eta_k)^*(0) = \\max_{u \\in \\R^n}\\left\\{\\inner{0}{u} - \\left( -\\frac{\\|u-\\hat x_0\\|^2}{2\\lam k} +I_Q(u)\\right)\\right\\}\n=\\frac{\\max_{u \\in Q}\\|u-\\hat x_0\\|^2}{2\\lam k}\n= \\frac{18d_0^2}{\\lam k},\n\\]\nwhere the last identity follows from the fact that $Q = B(\\hat x_0, 6d_0)$.\nTherefore, \\eqref{ineq:conjugate} holds in view of the above two relations.\n\\end{proof}","sha256_utf8":"5d2603c3f1233d54202b610d39a0be95444e7343600954f38c79e04f1a33e003"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"求和Fenchel等式","source_units":[{"start":38338,"end":38775,"start_line":621,"end_line":628,"raw_tex":"It follows from Lemma \\ref{ser-lemma1}(b) and Theorem 4.20 of \\cite{Beck2017} that for every $k \\ge 1$,\n\\[ \n\\Gamma_k(\\hat x_k) +  \\Gamma_k^*( s_k) = \\inner{\\hat x_k}{s_k}, \\quad  h(\\hat x_k) +  h^*(g_k) = \\inner{\\hat x_k}{g_k}.\n\\]\nSumming the above two equations yields\n\\begin{equation}\\label{eq:Gamma-h}\n    (\\Gamma_k + h)(\\hat x_k) + {\\Gamma}_k^*(s_k) + h^*(g_k) = \\frac{1}{\\lam}\\inner{\\hat x_k}{\\hat x_{k-1}-\\hat x_k}.\n\\end{equation}\n","sha256_utf8":"ffb0edfabac07a6fa449c3516fdf902ea4ee568c9cd4b2868c9a19d914478d2d"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"建立单周期共轭势函数下降","source_units":[{"start":38775,"end":39228,"start_line":629,"end_line":634,"raw_tex":"Using the above identity and Lemma \\ref{ser-lemma1}(a) and (c), we have for every $k \\ge 1$,\n\\begin{align*}\n& \\phi(\\tilde x_k) + f^*(s_k) +  {h}^*(g_k) \\le \\phi(\\tilde x_k) + \\Gamma_k^*(s_k) +  {h}^*(g_k) \\\\\n\\le & \\bar \\varepsilon +(\\Gamma_k + h)(\\hat x_k) + \\frac{1}{2\\lam}\\|\\hat x_k-\\hat x_{k-1}\\|^2 + \\Gamma_k^*(s_k) + {h}^*(g_k)\\\\\n\\stackrel{\\eqref{eq:Gamma-h}}{=}& \\bar \\varepsilon + \\frac{1}{2\\lam}(\\|\\hat x_{k-1}\\|^2-\\|\\hat x_k\\|^2).\n\\end{align*}\n","sha256_utf8":"22853941aacd59fc8a51150cf6ac26c330817e87e52d2e15a03c497cce6bea7b"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"累加并对原始对偶变量取平均","source_units":[{"start":39228,"end":39630,"start_line":635,"end_line":639,"raw_tex":"Replacing the index $k$ in the above inequality by $i$, summing the resulting inequality from $i=1$ to $k$, and using convexity and the definitions in \\eqref{eq:bxk}, we obtain \n\\begin{equation}\\label{ineq:avg}\n    \\phi(\\bar x_k) + f^*(\\bar s_k) + h^*(\\bar g_k)\\le \\bar \\varepsilon + \\frac{1}{2\\lam k }\\left(\\|\\hat x_0\\|^2-\\|\\hat x_k\\|^2\\right),\n\\end{equation}\nwhere $\\bar g_k = (\\sum_{i=1}^k g_i)/k$.\n","sha256_utf8":"310f52ed597f859a156341ee27c9c10276efc028081b83f4a9fc14ceccbb9031"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"定义二次修正并计算其共轭值","source_units":[{"start":39630,"end":40113,"start_line":640,"end_line":647,"raw_tex":"Define\n\\begin{equation}\\label{eq:zeta}\n    \\eta_k(u) = \\frac{1}{2\\lam k} \\|u-\\hat x_0\\|^2, \\quad \\hat \\eta_k(u) = \\eta_k(u) - I_Q(u).\n\\end{equation}\nNoting that $\\nabla \\eta_k(\\hat x_k) = (\\hat x_k - \\hat x_0)/(\\lam k) = -\\bar g_k - \\bar s_k$, and hence it follows from Theorem 4.20 of \\cite{Beck2017} that\n\\[\n\\eta_k^*(-\\bar g_k - \\bar s_k) = \\frac{1}{\\lam k}\\inner{\\hat x_k-\\hat x_0}{\\hat x_k} - \\eta_k(\\hat x_k) = \\frac{1}{2\\lam k}\\left(\\|\\hat x_k\\|^2 - \\| \\hat x_0\\|^2\\right).\n\\]\n","sha256_utf8":"21a555ad027ec984262df6b231fed1a81df380614c000550944d4e698c59587c"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"二次共轭项抵消端点势函数","source_units":[{"start":40113,"end":40330,"start_line":648,"end_line":651,"raw_tex":"The above observation and \\eqref{ineq:avg} together imply that\n\\begin{equation}\\label{ineq:bar}\n    \\phi(\\bar x_k) + f^*(\\bar s_k) + h^*(\\bar g_k) + \\eta_k^*(-\\bar g_k - \\bar s_k) \\le \\bar \\varepsilon.\n\\end{equation}\n","sha256_utf8":"32fab0f47b90dd135412ba878981edebe7fe4b6a590a5048c7bdff3a7f286a1d"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"用下确界卷积控制正则化共轭","source_units":[{"start":40330,"end":40651,"start_line":652,"end_line":657,"raw_tex":"It follows from Theorem 4.17 of \\cite{Beck2017} and the definition of infimum convolution in \\eqref{def:infconv} that\n\\[\n(h + \\eta_k)^*(-\\bar s_k) = (h^* \\square \\eta_k^*)(-\\bar s_k)\n\\stackrel{\\eqref{def:infconv}}= \\min_{u\\in \\R^n}\\{h^*(u) + \\eta_k^*(-\\bar s_k-u)\\}\n\\le h^*(\\bar g_k) + \\eta_k^*(-\\bar g_k - \\bar s_k).\n\\]\n","sha256_utf8":"2eddd82a0880403482ffff9d0238fac42d33204012ba5812ebac731a89306941"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"对受限函数再次应用共轭分解","source_units":[{"start":40651,"end":41018,"start_line":658,"end_line":663,"raw_tex":"Noting from \\eqref{eq:zeta} that $\\hat h = h + \\eta_k -\\hat \\eta_k$ and applying Theorem 4.17 of \\cite{Beck2017} again, we obtain\n\\begin{align*}\n    \\hat h ^*(-\\bar s_k) &= [(h + \\eta_k)^* \\square (-\\hat \\eta_k)^*](-\\bar s_k)\n= \\min_{u\\in \\R^n}\\{(h + \\eta_k)^*(u) + (-\\hat \\eta_k)^*(-\\bar s_k-u)\\}\\\\\n&\\le (h + \\eta_k)^*(-\\bar s_k) + (-\\hat \\eta_k)^*(0).\n\\end{align*}\n","sha256_utf8":"be16a9d7058e717dd1686992072795eed93b6130a61c15623bfd39b9fa558ba9"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"合并共轭界得到受限间隙","source_units":[{"start":41018,"end":41317,"start_line":664,"end_line":671,"raw_tex":"Summing the above two inequalities, we have\n\\[\n\\hat h ^*(-\\bar s_k) \\le h^*(\\bar g_k) + \\eta_k^*(-\\bar g_k - \\bar s_k) + (-\\hat \\eta_k)^*(0),\n\\]\nwhich together with \\eqref{ineq:bar} implies that\n\\[\n\\phi(\\bar x_k) + f^*(\\bar s_k) + \\hat h ^*(-\\bar s_k) \\le \\bar \\varepsilon + (-\\hat \\eta_k)^*(0).\n\\]\n","sha256_utf8":"9cf5be8f89d14cd040aff338383351ce0e7b1f2e9af827c156748db9cb85f3ca"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"利用球半径计算修正误差","source_units":[{"start":41317,"end":41711,"start_line":672,"end_line":679,"raw_tex":"It follows from \\eqref{eq:zeta} that\n\\[\n(-\\hat \\eta_k)^*(0) = \\max_{u \\in \\R^n}\\left\\{\\inner{0}{u} - \\left( -\\frac{\\|u-\\hat x_0\\|^2}{2\\lam k} +I_Q(u)\\right)\\right\\}\n=\\frac{\\max_{u \\in Q}\\|u-\\hat x_0\\|^2}{2\\lam k}\n= \\frac{18d_0^2}{\\lam k},\n\\]\nwhere the last identity follows from the fact that $Q = B(\\hat x_0, 6d_0)$.\nTherefore, \\eqref{ineq:conjugate} holds in view of the above two relations.\n","sha256_utf8":"d3d709ca0d79948c9c6397e7638d718ccfa178d4cedd3676eae4a59db6f56d44"}]},{"id":"S053","semantic_type":"lemma","kind":"lemma","title":"外层点与辅助点的球内有界性","source_units":[{"start":41863,"end":42261,"start_line":684,"end_line":689,"raw_tex":"\\begin{lemma}\\label{lem:bound}\n    Given $(\\hat x_0,\\bar \\varepsilon) \\in \\R^n \\times \\R_{++}$, if $\\lam \\le 2d_0^2/\\bar \\varepsilon$ and $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$, then the sequences $\\{\\hat x_k\\}$ and $\\{\\tx_k\\}$ generated by PDPB$(\\hat x_0,\\lam, \\bar \\varepsilon)$ satisfy\n    \\begin{equation}\\label{incl:xk}\n        \\hat x_k \\in Q, \\quad \\tx_k \\in Q.\n    \\end{equation}\n\\end{lemma}","sha256_utf8":"c29364c56802e276a966fa5ebc30b735a17e636589f57081f7e58cf105da6836"}]},{"id":"S054","semantic_type":"proof","kind":"proof","title":"迭代点有界性证明","source_units":[{"start":42263,"end":44509,"start_line":691,"end_line":724,"raw_tex":"\\begin{proof}\n    Noticing that the objective function in \\eqref{eq:xk} is $\\lam^{-1}$-strongly convex, it thus follows from Theorem 5.25(b) of \\cite{Beck2017} that for every $u \\in \\dom h$,\n    \\begin{equation}\\label{ineq:key}\n        m_k + \\frac{1}{2\\lam} \\|u-\\hat x_{k}\\|^2 \\le \\Gamma_k(u) + h(u) + \\frac1{2\\lam}\\|u-\\hat x_{k-1}\\|^2 \\le \\phi(u)+ \\frac1{2\\lam}\\|u-\\hat x_{k-1}\\|^2, \n    \\end{equation}\n    where the second inequality follows from the first one in Lemma \\ref{ser-lemma1}(a).\n    Taking $u=x_0^*$ in \\eqref{ineq:key}, we have\n    \\[\n    m_k + \\frac{1}{2\\lam} \\|\\hat x_k-x_0^*\\|^2 \\le \\phi_* + \\frac1{2\\lam}\\|\\hat x_{k-1}-x_0^*\\|^2. \n    \\]\n    This inequality and Lemma \\ref{ser-lemma1}(c) then imply that\n    \\begin{align*}\n        \\frac{1}{2\\lam}\\|\\hat x_k - x_0^*\\|^2 &\\le \\phi(\\tx_k)- \\phi_* + \\frac{1}{2\\lam}\\|\\hat x_k - x_0^*\\|^2 \\\\\n        &\\le \\phi(\\tx_k) - m_k + \\frac{1}{2\\lam}\\|\\hat x_{k-1}-x_0^*\\|^2 \\le \\bar \\varepsilon + \\frac{1}{2\\lam}\\|\\hat x_{k-1}-x_0^*\\|^2.\n    \\end{align*}\n    Replacing the index $k$ in the above inequality by $i$ and summing the resulting inequality from $i=1$ to $k$, we have\n    \\[\n    \\|\\hat x_k - x_0^*\\|^2 \\le \\|\\hat x_0-x_0^*\\|^2 + 2 k \\lam \\bar \\varepsilon.\n    \\]\n    Using the fact that $\\sqrt{a+b} \\le \\sqrt{a} + \\sqrt{b}$ for $a,b \\ge 0$ and the assumption that $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$, we further obtain\n    \\begin{equation}\\label{ineq:xk}\n        \\|\\hat x_k - x_0^*\\| \\le d_0 + \\sqrt{2k\\lam \\bar \\varepsilon} \\le 3d_0.\n    \\end{equation}\n    Taking $u=\\tx_k$ in \\eqref{ineq:key} and using Lemma \\ref{ser-lemma1}(c), we have\n    \\[\n    \\frac{1}{2\\lam} \\|\\tx_k-\\hat x_k\\|^2 \\le \\phi(\\tx_k) + \\frac1{2\\lam}\\|\\tx_k - \\hat x_{k-1}\\|^2 - m_k \\le \\bar \\varepsilon.\n    \\]\n    Under the assumption that $\\lam \\le 2d_0^2/\\bar \\varepsilon$, using \\eqref{ineq:xk}, the above inequality, and the triangle inequality, we have\n    \\begin{align*}\n        \\|\\hat x_k - \\hat x_0\\| &\\le \\|\\hat x_k - x_0^*\\| + \\|x_0^* - \\hat x_0\\| \\stackrel{\\eqref{ineq:xk}}{\\le} 4d_0, \\\\\n        \\|\\tx_k - \\hat x_0\\| &\\le \\|\\hat x_k - \\hat x_0\\| + \\|\\tx_k - \\hat x_k\\| \\le 4d_0 + \\sqrt{2\\lam \\bar \\varepsilon} \\le 6d_0.\n    \\end{align*}\n    Hence, \\eqref{incl:xk} follows immediately.\n\\end{proof}","sha256_utf8":"5c6a108ccf3533bf1b1aded5425dfda928ab030c781855de9aa1e441312cd698"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"建立模型最优值的三点界","source_units":[{"start":42277,"end":42756,"start_line":692,"end_line":696,"raw_tex":"    Noticing that the objective function in \\eqref{eq:xk} is $\\lam^{-1}$-strongly convex, it thus follows from Theorem 5.25(b) of \\cite{Beck2017} that for every $u \\in \\dom h$,\n    \\begin{equation}\\label{ineq:key}\n        m_k + \\frac{1}{2\\lam} \\|u-\\hat x_{k}\\|^2 \\le \\Gamma_k(u) + h(u) + \\frac1{2\\lam}\\|u-\\hat x_{k-1}\\|^2 \\le \\phi(u)+ \\frac1{2\\lam}\\|u-\\hat x_{k-1}\\|^2, \n    \\end{equation}\n    where the second inequality follows from the first one in Lemma \\ref{ser-lemma1}(a).\n","sha256_utf8":"4c83cb09d3233202b571807b534df2b97ed4153f8c107327b7211645d65ec38d"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"与最近最优解比较得到距离递推","source_units":[{"start":42756,"end":43273,"start_line":697,"end_line":705,"raw_tex":"    Taking $u=x_0^*$ in \\eqref{ineq:key}, we have\n    \\[\n    m_k + \\frac{1}{2\\lam} \\|\\hat x_k-x_0^*\\|^2 \\le \\phi_* + \\frac1{2\\lam}\\|\\hat x_{k-1}-x_0^*\\|^2. \n    \\]\n    This inequality and Lemma \\ref{ser-lemma1}(c) then imply that\n    \\begin{align*}\n        \\frac{1}{2\\lam}\\|\\hat x_k - x_0^*\\|^2 &\\le \\phi(\\tx_k)- \\phi_* + \\frac{1}{2\\lam}\\|\\hat x_k - x_0^*\\|^2 \\\\\n        &\\le \\phi(\\tx_k) - m_k + \\frac{1}{2\\lam}\\|\\hat x_{k-1}-x_0^*\\|^2 \\le \\bar \\varepsilon + \\frac{1}{2\\lam}\\|\\hat x_{k-1}-x_0^*\\|^2.\n    \\end{align*}\n","sha256_utf8":"949c9f2b2cb627a5bb1e95656c849c983fa6dc59052e5bb644bcbfdee1fc1dbe"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"累加得到三倍最优距离界","source_units":[{"start":43273,"end":43785,"start_line":706,"end_line":713,"raw_tex":"    Replacing the index $k$ in the above inequality by $i$ and summing the resulting inequality from $i=1$ to $k$, we have\n    \\[\n    \\|\\hat x_k - x_0^*\\|^2 \\le \\|\\hat x_0-x_0^*\\|^2 + 2 k \\lam \\bar \\varepsilon.\n    \\]\n    Using the fact that $\\sqrt{a+b} \\le \\sqrt{a} + \\sqrt{b}$ for $a,b \\ge 0$ and the assumption that $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$, we further obtain\n    \\begin{equation}\\label{ineq:xk}\n        \\|\\hat x_k - x_0^*\\| \\le d_0 + \\sqrt{2k\\lam \\bar \\varepsilon} \\le 3d_0.\n    \\end{equation}\n","sha256_utf8":"e749d9aae4d4b65c7642173598cb585d1c53d07e6a6c5d5786fbb60d9985aef4"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"界定辅助点与近端点距离","source_units":[{"start":43785,"end":44012,"start_line":714,"end_line":717,"raw_tex":"    Taking $u=\\tx_k$ in \\eqref{ineq:key} and using Lemma \\ref{ser-lemma1}(c), we have\n    \\[\n    \\frac{1}{2\\lam} \\|\\tx_k-\\hat x_k\\|^2 \\le \\phi(\\tx_k) + \\frac1{2\\lam}\\|\\tx_k - \\hat x_{k-1}\\|^2 - m_k \\le \\bar \\varepsilon.\n    \\]\n","sha256_utf8":"6626eca9676e385680c8a8411fff24f705d6e4e69b92a43d7878ae2f27ea43c8"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"由三角不等式得到球内包含","source_units":[{"start":44012,"end":44498,"start_line":718,"end_line":723,"raw_tex":"    Under the assumption that $\\lam \\le 2d_0^2/\\bar \\varepsilon$, using \\eqref{ineq:xk}, the above inequality, and the triangle inequality, we have\n    \\begin{align*}\n        \\|\\hat x_k - \\hat x_0\\| &\\le \\|\\hat x_k - x_0^*\\| + \\|x_0^* - \\hat x_0\\| \\stackrel{\\eqref{ineq:xk}}{\\le} 4d_0, \\\\\n        \\|\\tx_k - \\hat x_0\\| &\\le \\|\\hat x_k - \\hat x_0\\| + \\|\\tx_k - \\hat x_k\\| \\le 4d_0 + \\sqrt{2\\lam \\bar \\varepsilon} \\le 6d_0.\n    \\end{align*}\n    Hence, \\eqref{incl:xk} follows immediately.\n","sha256_utf8":"3bf226afe856765eeb1b2d5ad651f895a4f93b09823a1cf28b8cea2502f738dd"}]},{"id":"S060","semantic_type":"proposition","kind":"proposition","title":"PDPB外层原始对偶复杂度","source_units":[{"start":44617,"end":45090,"start_line":728,"end_line":734,"raw_tex":"\\begin{proposition}\\label{prop:pdpb}\n    Given $(\\hat x_0,\\bar \\varepsilon) \\in \\R^n \\times \\R_{++}$, if $\\lam \\le 2d_0^2/\\bar \\varepsilon$, then the number of iterations for PDPB$(\\hat x_0,\\lam, \\bar \\varepsilon)$ to generate $(\\bar x_k, \\bar s_k)$ satisfying \n    \\begin{equation}\\label{ineq:pd}\n        \\hat \\phi(\\bar x_k) + f^*(\\bar s_k) + \\hat h^*(-\\bar s_k) \\le 10\\bar \\varepsilon\n    \\end{equation}\n    is at most $2 d_0^2/(\\lam \\bar \\varepsilon)$.\n\\end{proposition}","sha256_utf8":"5a8d58402d4310ccaffe43da5034c3556e1ebc194d75b27ab2a21ff851b3d23c"}]},{"id":"S061","semantic_type":"proof","kind":"proof","title":"外层复杂度证明","source_units":[{"start":45092,"end":45712,"start_line":736,"end_line":744,"raw_tex":"\\begin{proof}\n    Since $Q$ is a convex set, it follows from the definition of $\\bar x_k$ in \\eqref{eq:bxk} and Lemma \\ref{lem:bound} that $\\bar x_k \\in Q$ for every $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$.\n    This observation and the fact that $\\hat h = h+ I_Q$ imply that $\\hat h(\\bar x_k) = h(\\bar x_k)$.\n    Hence, using Proposition~\\ref{prop:pd-key}, we have for every $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$,\n    \\[\n    \\hat \\phi(\\bar x_k) + f^*(\\bar s_k) + \\hat h ^*(-\\bar s_k) \\le \\bar \\varepsilon + \\frac{18 d_0^2}{\\lam k}.\n    \\]\n    Therefore, the conclusion of the proposition follows immediately.\n\\end{proof}","sha256_utf8":"d5d3d31672c341aadbc9328670b644e0b66df505f2a50546be35b51251750e18"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"平均点位于球内并消去指示项","source_units":[{"start":45106,"end":45400,"start_line":737,"end_line":738,"raw_tex":"    Since $Q$ is a convex set, it follows from the definition of $\\bar x_k$ in \\eqref{eq:bxk} and Lemma \\ref{lem:bound} that $\\bar x_k \\in Q$ for every $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$.\n    This observation and the fact that $\\hat h = h+ I_Q$ imply that $\\hat h(\\bar x_k) = h(\\bar x_k)$.\n","sha256_utf8":"f7253b4ada754cd769f894a28706d258a6d025ae1ef09892643c4c47cf1434a9"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"由共轭间隙界确定周期数","source_units":[{"start":45400,"end":45701,"start_line":739,"end_line":743,"raw_tex":"    Hence, using Proposition~\\ref{prop:pd-key}, we have for every $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$,\n    \\[\n    \\hat \\phi(\\bar x_k) + f^*(\\bar s_k) + \\hat h ^*(-\\bar s_k) \\le \\bar \\varepsilon + \\frac{18 d_0^2}{\\lam k}.\n    \\]\n    Therefore, the conclusion of the proposition follows immediately.\n","sha256_utf8":"972fce26ebc18a274e8ad3ca82dcdd3aa08246c875b4373cebc1d9f1cf590426"}]},{"id":"S064","semantic_type":"lemma","kind":"lemma","title":"各周期初始间隙的统一界","source_units":[{"start":45841,"end":46092,"start_line":748,"end_line":754,"raw_tex":"\\begin{lemma}\\label{lem:t1}\nFor $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$, we have\n\\begin{equation}\\label{ineq:t1}\n    t_{i_k} \\le \\bar{t}:= 4M(3d_0 + \\lam M),\n\\end{equation}\nwhere $i_k$ is the first iteration index in the cycle ${\\cal C}_k$.\n\\end{lemma}","sha256_utf8":"0758ec57f7de3c270bc2e1b152da757d8eb1c8e112cd2a72cae80c5e1ed37774"}]},{"id":"S065","semantic_type":"proof","kind":"proof","title":"周期初始间隙证明","source_units":[{"start":46094,"end":48233,"start_line":756,"end_line":788,"raw_tex":"\\begin{proof}\nUsing \\eqref{ineq:f}, definitions of $m_j$ and $t_j$ in \\eqref{eq:mj}, and the facts that $\\tx_{i_k}=x_{i_k}$ and $\\Gamma_{i_k}=\\ell_f(\\cdot;x_{k-1})$, we have\n\\begin{align}\n    t_{i_k} &\\stackrel{\\eqref{eq:mj}}{=} \\phi^\\lam \\left(\\tx_{i_k}\\right)-m_{i_k} = \\phi^\\lam (x_{i_k}) -m_{i_k} \\nn \\\\\n    &\\stackrel{\\eqref{eq:problem},\\eqref{eq:phi-lam},\\eqref{eq:mj}}{=} f(x_{i_k})-\\ell_f(x_{i_k};\\hat x_{k-1}) \\stackrel{\\eqref{ineq:f}}{\\le} 2M\\left\\|x_{i_k}-\\hat x_{k-1}\\right\\|, \\label{ineq:tik}\n\\end{align}\nwhere we have also used the definitions of $\\phi$ and $\\phi^\\lam$ in \\eqref{eq:problem} and \\eqref{eq:phi-lam}, respectively, in the last identity.\nIn view of \\eqref{eq:xj} and the fact that $\\Gamma_{i_k}=\\ell_f(\\cdot;\\hat x_{k-1})$, we know the first iteration of PDCP is the same as PDS$(\\hat x_0,\\lam)$ (see \\eqref{eq:pds}). Hence, following an argument similar to the proof of Lemma~\\ref{pds-lemma1}, we can prove for every $u \\in \\dom h$,\n\\[\n\\phi(x_{i_k}) - \\ell_f(u;\\hat x_{k-1}) -h(u)\n\\stackrel{\\eqref{ineq:hat-phi-recur}}{\\le} 2\\lam M^2 + \\frac{1}{2 \\lambda}\\|u -\\hat x_{k-1}\\|^2 - \\frac{1}{2\\lambda}\\left\\|u-x_{i_k}\\right\\|^2.\n\\]\nIt follows from the above inequality with $u=x_0^*$ and the convexity of $f$ that\n\\begin{align*}\n    0 &\\le \\phi(x_{i_k}) -\\phi_*\n    \\le \\phi(x_{i_k}) - \\ell_f(x_0^*;\\hat x_{k-1}) -h(x_0^*) \\\\\n    &\\le 2\\lam M^2 + \\frac{1}{2 \\lambda}\\|x_0^* -\\hat x_{k-1}\\|^2 - \\frac{1}{2\\lambda}\\left\\|x_0^*-x_{i_k}\\right\\|^2.\n\\end{align*}\nRearranging the terms and using the inequality $\\sqrt{a+b} \\le \\sqrt{a} + \\sqrt{b}$ for any $a, b \\ge 0$, we have\n\\[\n\\|x_0^*-x_{i_k}\\| \\le\\|x_0^*-\\hat x_{k-1}\\|+2\\lam M.\n\\]\nThis inequality and the triangle inequality then imply that\n\\[\n\\|x_{i_k}-\\hat x_{k-1}\\| \\le \\|x_{i_k}-x_0^*\\|+\\|x_0^*-\\hat x_{k-1}\\| \\le 2\\|\\hat x_{k-1}-x_0^*\\|+2 \\lam M.\n\\]\nRecall from the proof of Lemma \\ref{lem:bound} that \\eqref{ineq:xk} gives $\\|\\hat x_k-x_0^*\\| \\le 3d_0$ for $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$.\nHence, we have\n\\[\n\\|x_{i_k}-\\hat x_{k-1}\\| \\le 2(3d_0 + \\lam M).\n\\]\nTherefore, \\eqref{ineq:t1} follows from \\eqref{ineq:tik} and the above inequality.\n\\end{proof}","sha256_utf8":"f82bb3a33333e17c60a08513db2ab163227711f71530c4cb1eba401e3d96e168"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"将初始间隙化为首步位移","source_units":[{"start":46108,"end":46760,"start_line":757,"end_line":762,"raw_tex":"Using \\eqref{ineq:f}, definitions of $m_j$ and $t_j$ in \\eqref{eq:mj}, and the facts that $\\tx_{i_k}=x_{i_k}$ and $\\Gamma_{i_k}=\\ell_f(\\cdot;x_{k-1})$, we have\n\\begin{align}\n    t_{i_k} &\\stackrel{\\eqref{eq:mj}}{=} \\phi^\\lam \\left(\\tx_{i_k}\\right)-m_{i_k} = \\phi^\\lam (x_{i_k}) -m_{i_k} \\nn \\\\\n    &\\stackrel{\\eqref{eq:problem},\\eqref{eq:phi-lam},\\eqref{eq:mj}}{=} f(x_{i_k})-\\ell_f(x_{i_k};\\hat x_{k-1}) \\stackrel{\\eqref{ineq:f}}{\\le} 2M\\left\\|x_{i_k}-\\hat x_{k-1}\\right\\|, \\label{ineq:tik}\n\\end{align}\nwhere we have also used the definitions of $\\phi$ and $\\phi^\\lam$ in \\eqref{eq:problem} and \\eqref{eq:phi-lam}, respectively, in the last identity.\n","sha256_utf8":"09af7d894bd0f1374cc6941f279617888a7b187595188423d61aafba6cfb67e2"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"调用次梯度首步递推估计","source_units":[{"start":46760,"end":47251,"start_line":763,"end_line":767,"raw_tex":"In view of \\eqref{eq:xj} and the fact that $\\Gamma_{i_k}=\\ell_f(\\cdot;\\hat x_{k-1})$, we know the first iteration of PDCP is the same as PDS$(\\hat x_0,\\lam)$ (see \\eqref{eq:pds}). Hence, following an argument similar to the proof of Lemma~\\ref{pds-lemma1}, we can prove for every $u \\in \\dom h$,\n\\[\n\\phi(x_{i_k}) - \\ell_f(u;\\hat x_{k-1}) -h(u)\n\\stackrel{\\eqref{ineq:hat-phi-recur}}{\\le} 2\\lam M^2 + \\frac{1}{2 \\lambda}\\|u -\\hat x_{k-1}\\|^2 - \\frac{1}{2\\lambda}\\left\\|u-x_{i_k}\\right\\|^2.\n\\]\n","sha256_utf8":"5c2149dca0659ecc015196e436c572f5434efceb532262ede98ee45d99d18774"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"与最优解比较界定首步距离","source_units":[{"start":47251,"end":47749,"start_line":768,"end_line":777,"raw_tex":"It follows from the above inequality with $u=x_0^*$ and the convexity of $f$ that\n\\begin{align*}\n    0 &\\le \\phi(x_{i_k}) -\\phi_*\n    \\le \\phi(x_{i_k}) - \\ell_f(x_0^*;\\hat x_{k-1}) -h(x_0^*) \\\\\n    &\\le 2\\lam M^2 + \\frac{1}{2 \\lambda}\\|x_0^* -\\hat x_{k-1}\\|^2 - \\frac{1}{2\\lambda}\\left\\|x_0^*-x_{i_k}\\right\\|^2.\n\\end{align*}\nRearranging the terms and using the inequality $\\sqrt{a+b} \\le \\sqrt{a} + \\sqrt{b}$ for any $a, b \\ge 0$, we have\n\\[\n\\|x_0^*-x_{i_k}\\| \\le\\|x_0^*-\\hat x_{k-1}\\|+2\\lam M.\n\\]\n","sha256_utf8":"9e4998c60d26a4cc7c870d2993ee28c75348280060ff54d0d1430d67dbebe2fb"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"由三角不等式界定首步位移","source_units":[{"start":47749,"end":47923,"start_line":778,"end_line":781,"raw_tex":"This inequality and the triangle inequality then imply that\n\\[\n\\|x_{i_k}-\\hat x_{k-1}\\| \\le \\|x_{i_k}-x_0^*\\|+\\|x_0^*-\\hat x_{k-1}\\| \\le 2\\|\\hat x_{k-1}-x_0^*\\|+2 \\lam M.\n\\]\n","sha256_utf8":"691646033d8e50927e64493009dff6529c14ee3693961b6de6ba9039448ad8fb"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"使用外层距离界完成初始间隙估计","source_units":[{"start":47923,"end":48222,"start_line":782,"end_line":787,"raw_tex":"Recall from the proof of Lemma \\ref{lem:bound} that \\eqref{ineq:xk} gives $\\|\\hat x_k-x_0^*\\| \\le 3d_0$ for $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$.\nHence, we have\n\\[\n\\|x_{i_k}-\\hat x_{k-1}\\| \\le 2(3d_0 + \\lam M).\n\\]\nTherefore, \\eqref{ineq:t1} follows from \\eqref{ineq:tik} and the above inequality.\n","sha256_utf8":"4d1b6e77cea1fd4780d660af297e8b5cfead9f5081d9e09db90abf72ce032fab"}]},{"id":"S071","semantic_type":"lemma","kind":"lemma","title":"修正次梯度法单步递推","source_units":[{"start":124018,"end":124413,"start_line":1923,"end_line":1928,"raw_tex":"\\begin{lemma}\\label{pds-lemma1}\nGiven $\\hat x_0 \\in \\R^n$, for every $k \\ge 1$ and $u \\in \\dom \\hat h$, the sequence $\\{\\hat x_k\\}$ generated by MPDS$(\\hat x_0,\\lam)$ satisfies\n\\begin{equation}\\label{ineq:hat-phi-recur}\n    \\hat \\phi(\\hat x_k) - \\ell_f(u;\\hat x_{k-1}) - \\hat h(u) \\le 2\\lam M^2 + \\frac{1}{2\\lam}\\|u- \\hat x_{k-1}\\|^2 - \\frac1{2\\lam}\\|u - \\hat x_k\\|^2.\n\\end{equation}\n\\end{lemma}","sha256_utf8":"39ca5924c622c5cec4ad9c1a95f442932c8f9523c7bf6cf9ff50beb501b7c94b"}]},{"id":"S072","semantic_type":"proof","kind":"proof","title":"次梯度单步递推证明","source_units":[{"start":124414,"end":125340,"start_line":1929,"end_line":1941,"raw_tex":"\\begin{proof}\nNoticing that the objective function in \\eqref{eq:pds-1} is $\\lam^{-1}$-strongly convex, it then follows from Theorem 5.25(b) of \\cite{Beck2017} that for every $u \\in \\dom \\hat h$,\n\\begin{equation}\\label{ineq:ell-h-mk}\n    \\ell_f(u;\\hat x_{k-1}) + \\hat h(u) + \\frac1{2\\lam}\\|u-\\hat x_{k-1}\\|^2 \\ge m_k + \\frac{1}{2\\lam} \\|u-\\hat x_k\\|^2,\n\\end{equation}\nwhere $m_k = \\ell_f(\\hat x_k;\\hat x_{k-1}) + \\hat h(\\hat x_k) + \\|\\hat x_k-\\hat x_{k-1}\\|^2/(2\\lam)$.\nUsing \\eqref{ineq:f} with $(x,y)=(\\hat x_k,\\hat x_{k-1})$, we have\n\\[\n    \\hat \\phi(\\hat x_k) - m_k = f(\\hat x_k) - \\ell_f(\\hat x_k;\\hat x_{k-1}) \\stackrel{\\eqref{ineq:f}}{\\le} 2M\\|\\hat x_k - \\hat x_{k-1}\\| - \\frac1{2\\lam}\\|\\hat x_k-\\hat x_{k-1}\\|^2 \\le 2\\lam M^2,\n\\]\nwhere the last inequality is due to Young's inequality $a^2+b^2\\ge 2ab$.\nHence, \\eqref{ineq:hat-phi-recur} follows from combining the above inequality and \\eqref{ineq:ell-h-mk}.\n\\end{proof}","sha256_utf8":"c0a94a9934e9ad60eba9712b60222e1c6465e6375afa04a5a265c15debd7b9f3"}]},{"id":"S073","semantic_type":"proof_step","kind":"proof_step","title":"利用强凸性比较次梯度模型最优值","source_units":[{"start":124428,"end":124883,"start_line":1930,"end_line":1934,"raw_tex":"Noticing that the objective function in \\eqref{eq:pds-1} is $\\lam^{-1}$-strongly convex, it then follows from Theorem 5.25(b) of \\cite{Beck2017} that for every $u \\in \\dom \\hat h$,\n\\begin{equation}\\label{ineq:ell-h-mk}\n    \\ell_f(u;\\hat x_{k-1}) + \\hat h(u) + \\frac1{2\\lam}\\|u-\\hat x_{k-1}\\|^2 \\ge m_k + \\frac{1}{2\\lam} \\|u-\\hat x_k\\|^2,\n\\end{equation}\nwhere $m_k = \\ell_f(\\hat x_k;\\hat x_{k-1}) + \\hat h(\\hat x_k) + \\|\\hat x_k-\\hat x_{k-1}\\|^2/(2\\lam)$.\n","sha256_utf8":"571710738f46087115775ca96560d8c175a0d770e45396c6d7250144f9136402"}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","title":"用Young不等式控制模型误差","source_units":[{"start":124883,"end":125224,"start_line":1935,"end_line":1939,"raw_tex":"Using \\eqref{ineq:f} with $(x,y)=(\\hat x_k,\\hat x_{k-1})$, we have\n\\[\n    \\hat \\phi(\\hat x_k) - m_k = f(\\hat x_k) - \\ell_f(\\hat x_k;\\hat x_{k-1}) \\stackrel{\\eqref{ineq:f}}{\\le} 2M\\|\\hat x_k - \\hat x_{k-1}\\| - \\frac1{2\\lam}\\|\\hat x_k-\\hat x_{k-1}\\|^2 \\le 2\\lam M^2,\n\\]\nwhere the last inequality is due to Young's inequality $a^2+b^2\\ge 2ab$.\n","sha256_utf8":"739e9686fec96cdbfc30b68a19cc6e0cff102d04d4c61e9b394ca59c225d7872"}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","title":"合并模型比较与误差界","source_units":[{"start":125224,"end":125329,"start_line":1940,"end_line":1940,"raw_tex":"Hence, \\eqref{ineq:hat-phi-recur} follows from combining the above inequality and \\eqref{ineq:ell-h-mk}.\n","sha256_utf8":"70f68ba6492a8cdcc16937720317473ee53ab67884097c1328a82ff4e2cb6631"}]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","title":"共轭定义与Fenchel等式","source_units":[],"declared_external":true,"declaration_text":["For a proper function a, a*(s)=sup_x{<s,x>-a(x)}. Conjugates are convex and reverse pointwise order. For proper closed convex a, s in partial a(x) implies a(x)+a*(s)=<s,x> (Beck2017, Theorem 4.20)."]},{"id":"S077","semantic_type":"external_fact","kind":"external_fact","title":"强凸最小值界与最优性","source_units":[],"declared_external":true,"declaration_text":["If F is mu-strongly convex and x minimizes F, then F(u)>=F(x)+(mu/2)||u-x||^2. Addition of a differentiable quadratic obeys the subdifferential sum rule; minimizers satisfy the corresponding zero-subgradient condition (including Beck2017, Theorem 5.25(b))."]},{"id":"S078","semantic_type":"external_fact","kind":"external_fact","title":"和函数的共轭卷积公式","source_units":[],"declared_external":true,"declaration_text":["The conjugate-of-sum formula invoked as Beck2017, Theorem 4.17 is (a+b)*(s)=min_u{a*(u)+b*(s-u)}, under its qualification conditions; evaluating the minimization at any admissible u gives an upper bound."]},{"id":"S079","semantic_type":"external_fact","kind":"external_fact","title":"凸平均与欧氏基本不等式","source_units":[],"declared_external":true,"declaration_text":["Jensen's inequality for convex combinations, Young's inequality a^2+b^2>=2ab, the triangle inequality, expansion of squared Euclidean norms, and sqrt(a+b)<=sqrt(a)+sqrt(b) for nonnegative a,b."]},{"id":"S080","semantic_type":"assumption","kind":"assumption","title":"球内有界性的局部步长与周期条件","source_units":[{"start":41894,"end":42152,"start_line":685,"end_line":685,"raw_tex":"    Given $(\\hat x_0,\\bar \\varepsilon) \\in \\R^n \\times \\R_{++}$, if $\\lam \\le 2d_0^2/\\bar \\varepsilon$ and $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$, then the sequences $\\{\\hat x_k\\}$ and $\\{\\tx_k\\}$ generated by PDPB$(\\hat x_0,\\lam, \\bar \\varepsilon)$ satisfy\n","sha256_utf8":"80ab82a169e669ca7cc961914331ee99f2a13ab265b9754aa59a43abc03a9602"}]},{"id":"S081","semantic_type":"assumption","kind":"assumption","title":"初始间隙引理的周期范围","source_units":[{"start":45869,"end":45921,"start_line":749,"end_line":749,"raw_tex":"For $k \\le 2d_0^2/(\\lam \\bar \\varepsilon)$, we have\n","sha256_utf8":"704972d539ebe5214aad3d0574e271a6d9829e00599e52a81ec3694a27186859"}]},{"id":"S082","semantic_type":"assumption","kind":"assumption","title":"内层权重选择","source_units":[{"start":31209,"end":31305,"start_line":502,"end_line":502,"raw_tex":"Considering Algorithm \\ref{alg:PDCP} with $\\tau_j = j/(j+2)$, then for every $j \\ge 1$, we have\n","sha256_utf8":"80855ffbdc3160d19a9492452cd84439c6d00036cad298ca729df706b1254199"}]},{"id":"S083","semantic_type":"assumption","kind":"assumption","title":"外层复杂度的步长限制","source_units":[{"start":44654,"end":44879,"start_line":729,"end_line":729,"raw_tex":"    Given $(\\hat x_0,\\bar \\varepsilon) \\in \\R^n \\times \\R_{++}$, if $\\lam \\le 2d_0^2/\\bar \\varepsilon$, then the number of iterations for PDPB$(\\hat x_0,\\lam, \\bar \\varepsilon)$ to generate $(\\bar x_k, \\bar s_k)$ satisfying \n","sha256_utf8":"ea14d0c740b4d7bfd60b16f644a37a4555caac34c65c16d703c45d3d56050761"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S006","kind":"supports"},{"source_id":"S003","target_id":"S007","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S002","kind":"supports"},{"source_id":"S006","target_id":"S002","kind":"supports"},{"source_id":"S007","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S004","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S008","target_id":"S016","kind":"supports"},{"source_id":"S008","target_id":"S045","kind":"supports"},{"source_id":"S008","target_id":"S056","kind":"supports"},{"source_id":"S009","target_id":"S028","kind":"supports"},{"source_id":"S009","target_id":"S055","kind":"supports"},{"source_id":"S009","target_id":"S073","kind":"supports"},{"source_id":"S010","target_id":"S013","kind":"supports"},{"source_id":"S010","target_id":"S040","kind":"supports"},{"source_id":"S010","target_id":"S066","kind":"supports"},{"source_id":"S011","target_id":"S049","kind":"supports"},{"source_id":"S011","target_id":"S050","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S056","kind":"supports"},{"source_id":"S012","target_id":"S068","kind":"supports"},{"source_id":"S013","target_id":"S033","kind":"supports"},{"source_id":"S013","target_id":"S066","kind":"supports"},{"source_id":"S013","target_id":"S074","kind":"supports"},{"source_id":"S014","target_id":"S021","kind":"supports"},{"source_id":"S014","target_id":"S024","kind":"supports"},{"source_id":"S014","target_id":"S052","kind":"supports"},{"source_id":"S014","target_id":"S056","kind":"supports"},{"source_id":"S014","target_id":"S057","kind":"supports"},{"source_id":"S015","target_id":"S067","kind":"supports"},{"source_id":"S016","target_id":"S045","kind":"supports"},{"source_id":"S016","target_id":"S066","kind":"supports"},{"source_id":"S017","target_id":"S028","kind":"supports"},{"source_id":"S017","target_id":"S032","kind":"supports"},{"source_id":"S017","target_id":"S040","kind":"supports"},{"source_id":"S017","target_id":"S067","kind":"supports"},{"source_id":"S018","target_id":"S028","kind":"supports"},{"source_id":"S018","target_id":"S032","kind":"supports"},{"source_id":"S018","target_id":"S039","kind":"supports"},{"source_id":"S019","target_id":"S029","kind":"supports"},{"source_id":"S019","target_id":"S032","kind":"supports"},{"source_id":"S019","target_id":"S035","kind":"supports"},{"source_id":"S019","target_id":"S036","kind":"supports"},{"source_id":"S019","target_id":"S041","kind":"supports"},{"source_id":"S019","target_id":"S066","kind":"supports"},{"source_id":"S020","target_id":"S041","kind":"supports"},{"source_id":"S020","target_id":"S066","kind":"supports"},{"source_id":"S020","target_id":"S067","kind":"supports"},{"source_id":"S021","target_id":"S050","kind":"supports"},{"source_id":"S021","target_id":"S052","kind":"supports"},{"source_id":"S021","target_id":"S059","kind":"supports"},{"source_id":"S021","target_id":"S062","kind":"supports"},{"source_id":"S022","target_id":"S040","kind":"supports"},{"source_id":"S022","target_id":"S041","kind":"supports"},{"source_id":"S022","target_id":"S046","kind":"supports"},{"source_id":"S022","target_id":"S062","kind":"supports"},{"source_id":"S023","target_id":"S039","kind":"supports"},{"source_id":"S023","target_id":"S040","kind":"supports"},{"source_id":"S023","target_id":"S041","kind":"supports"},{"source_id":"S023","target_id":"S055","kind":"supports"},{"source_id":"S023","target_id":"S066","kind":"supports"},{"source_id":"S024","target_id":"S073","kind":"supports"},{"source_id":"S024","target_id":"S074","kind":"supports"},{"source_id":"S025","target_id":"S073","kind":"supports"},{"source_id":"S026","target_id":"S033","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S027","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S030","target_id":"S005","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S034","target_id":"S031","kind":"supports"},{"source_id":"S035","target_id":"S031","kind":"supports"},{"source_id":"S036","target_id":"S031","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S037","target_id":"S044","kind":"supports"},{"source_id":"S037","target_id":"S045","kind":"supports"},{"source_id":"S037","target_id":"S055","kind":"supports"},{"source_id":"S037","target_id":"S056","kind":"supports"},{"source_id":"S037","target_id":"S058","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S040","target_id":"S038","kind":"supports"},{"source_id":"S041","target_id":"S038","kind":"supports"},{"source_id":"S042","target_id":"S063","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S043","kind":"supports"},{"source_id":"S046","target_id":"S043","kind":"supports"},{"source_id":"S047","target_id":"S043","kind":"supports"},{"source_id":"S048","target_id":"S043","kind":"supports"},{"source_id":"S049","target_id":"S043","kind":"supports"},{"source_id":"S050","target_id":"S043","kind":"supports"},{"source_id":"S051","target_id":"S043","kind":"supports"},{"source_id":"S052","target_id":"S043","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S047","target_id":"S050","kind":"supports"},{"source_id":"S048","target_id":"S051","kind":"supports"},{"source_id":"S049","target_id":"S051","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S047","target_id":"S052","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S053","target_id":"S062","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S055","target_id":"S054","kind":"supports"},{"source_id":"S056","target_id":"S054","kind":"supports"},{"source_id":"S057","target_id":"S054","kind":"supports"},{"source_id":"S058","target_id":"S054","kind":"supports"},{"source_id":"S059","target_id":"S054","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S055","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S057","target_id":"S070","kind":"supports"},{"source_id":"S060","target_id":"S004","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S063","target_id":"S061","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S064","target_id":"S005","kind":"supports"},{"source_id":"S065","target_id":"S064","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"},{"source_id":"S067","target_id":"S065","kind":"supports"},{"source_id":"S068","target_id":"S065","kind":"supports"},{"source_id":"S069","target_id":"S065","kind":"supports"},{"source_id":"S070","target_id":"S065","kind":"supports"},{"source_id":"S067","target_id":"S068","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S069","target_id":"S070","kind":"supports"},{"source_id":"S066","target_id":"S070","kind":"supports"},{"source_id":"S071","target_id":"S067","kind":"supports"},{"source_id":"S072","target_id":"S071","kind":"supports"},{"source_id":"S073","target_id":"S072","kind":"supports"},{"source_id":"S074","target_id":"S072","kind":"supports"},{"source_id":"S075","target_id":"S072","kind":"supports"},{"source_id":"S073","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S076","target_id":"S039","kind":"supports"},{"source_id":"S076","target_id":"S044","kind":"supports"},{"source_id":"S076","target_id":"S046","kind":"supports"},{"source_id":"S076","target_id":"S047","kind":"supports"},{"source_id":"S076","target_id":"S052","kind":"supports"},{"source_id":"S077","target_id":"S028","kind":"supports"},{"source_id":"S077","target_id":"S040","kind":"supports"},{"source_id":"S077","target_id":"S055","kind":"supports"},{"source_id":"S077","target_id":"S073","kind":"supports"},{"source_id":"S078","target_id":"S049","kind":"supports"},{"source_id":"S078","target_id":"S050","kind":"supports"},{"source_id":"S079","target_id":"S033","kind":"supports"},{"source_id":"S079","target_id":"S035","kind":"supports"},{"source_id":"S079","target_id":"S045","kind":"supports"},{"source_id":"S079","target_id":"S046","kind":"supports"},{"source_id":"S079","target_id":"S057","kind":"supports"},{"source_id":"S079","target_id":"S059","kind":"supports"},{"source_id":"S079","target_id":"S062","kind":"supports"},{"source_id":"S079","target_id":"S068","kind":"supports"},{"source_id":"S079","target_id":"S069","kind":"supports"},{"source_id":"S079","target_id":"S074","kind":"supports"},{"source_id":"S080","target_id":"S057","kind":"supports"},{"source_id":"S080","target_id":"S059","kind":"supports"},{"source_id":"S081","target_id":"S070","kind":"supports"},{"source_id":"S082","target_id":"S032","kind":"supports"},{"source_id":"S083","target_id":"S062","kind":"supports"},{"source_id":"S083","target_id":"S063","kind":"supports"}]}}
{"record_id":"2412.06730v4","paper":{"title":"Stochastic and incremental subgradient methods for convex optimization on Hadamard spaces","authors":["Ariel Goodwin","Adrian S. Lewis","Genaro López-Acedo","Adriana Nicolae"],"arxiv_id":"2412.06730","version":"v4","journal":"Mathematical Programming","doi":"10.1007/s10107-026-02334-4","abstract_url":"https://arxiv.org/abs/2412.06730v4","pdf_url":"https://arxiv.org/pdf/2412.06730v4"},"context":{"raw_tex":"A metric space $(X,d)$ is a\n\t\t\\textit{geodesic metric space} if every two points $x,y\\in X$ can be joined by a \n\t\t \\textit{geodesic},\n\t\twhich is to say a map $\\gamma$ from a closed interval $[a,b]$ into $X$ with $d(\\gamma(t),\\gamma(t')) = |t-t'|$\n\t\tfor all $t,t'\\in [a,b]$. A \\textit{ray} is a geodesic with domain $\\R_+$, and we say $r$ \\textit{issues from}\n\t\t$x$ if $r(0) = x$. A geodesic metric space is said to be \\textit{CAT(0)} if the map $t\\mapsto \\frac{1}{2}d(\\gamma(t),a)^2$ is $1$-strongly convex for every $a\\in X$ and geodesic $\\gamma$. A \\textit{Hadamard space} is a complete CAT(0) space. \n\t\tIn a Hadamard space, there is exactly one geodesic joining $x$ to $y$ for every $x,y\\in X$. \n\t\tWe say that\n\t\ta Hadamard space has the \\textit{geodesic extension property} if for every $x\\neq y\\in X$ there exists a \n\t\tray $r\\from\\R_+\\to X$ with $r(0) = x$ and $r(t) = y$ for some $t>0$. A subset $C$ of a Hadamard \n\t\tspace $X$ is \\textit{geodesically\n\t\tconvex} if the geodesic between any two points in $C$ is contained in $C$. A metric space $(X,d)$ is \\textit{proper} if the closed ball $B_r(x)$ is compact for every $x\\in X, r > 0$.\n\n\t\tHenceforth, $(X,d)$ will be a Hadamard space with the geodesic extension property, and to avoid degenerate\n\t\tcase-splitting in forthcoming results we will assume $X$ has at least two points. The significance of this technical\n\t\tassumption is that there always exist rays.\n\t\tGiven a ray $r$ on $X$ we may associate the \n\t\tcorresponding \\textit{Busemann function} $b_r\\from X\\to \\R$ defined by\n\t\t\\[b_r(z) = \\lim_{t\\to\\infty}(d(z,r(t)) - t) ~ ~ ~ (z\\in X).\\]\nIt is easy to see that the Busemann function $b_r$ is 1-Lipschitz, convex, and satisfies $b_r(r(0)) = 0$.\n\n\n\t\tTwo rays $r,r'$ are said to be \\textit{asymptotic}\n\t\tif there exists a positive constant $K\\geq 0$ such that $d(r(t),r'(t)) \\leq K$ for all $t\\geq 0$. \n\t\tThis defines an equivalence relation on rays in $X$: the set of equivalence classes is \n\t\tdenoted by $X^{\\infty}$ and called the \\textit{boundary of $X$ at infinity}, or simply the \\textit{boundary}. Note our assumption that $X$ has at least two points and the geodesic extension property guarantees the boundary is nonempty.\n\t\tThe equivalence class of a particular \n\t\tray $r$ is denoted by $r(\\infty)$, and we say a ray $r$ has \\textit{direction} $\\xi \\in X^\\infty$ if $r$\n\t\tbelongs to the equivalence class $\\xi$. Given $x\\in X$ and $\\xi \\in X^\\infty$, there exists a unique\n\t\tray $r$ issuing from $x$ such that $r(\\infty) = \\xi$ \\cite[II.8.2]{bridson}. \n\t\tIf we fix an arbitrary \n\t\treference point $\\bar x \\in X$, we can thus identify any $\\xi \\in X^\\infty$ with the unique ray $r_{\\bar x,\\xi}$ \n\t\tissuing from $\\bar x$ with direction $\\xi$, and hence to the unique\n\t\tcorresponding Busemann function which we may denote by either $b_{r_{\\bar x,\\xi}}$ or $b_{\\bar x,\\xi}$. \n\t\tTo avoid cumbersome notation, we henceforth\n\t\tfix such a reference point $\\bar x \\in X$ and denote the corresponding Busemann function \n\t\t$b_{r_{\\bar x,\\xi}}$ by simply $b_\\xi$.  We note the following fact \\cite[Corollary II.8.20]{bridson}.\n\n\n\t\tThe space $X^\\infty$ is naturally endowed with the so-called \\textit{cone topology} \n\t\t\\cite[Definition II.8.6]{bridson}, and notably this space is first-countable \n\t\t\\cite[Proof of Theorem II.8.13]{bridson}. \n\t\t\t\tIn particular, the cone topology is completely specified by \n\t\tthe convergent sequences, so let us mention that $\\xi_n \\to \\xi \\in X^\\infty$ if and only if \n\t\t$b_{\\xi_n} \\to b_{\\xi}$ uniformly on bounded subsets of $X$.  The latter property, and hence the cone topology, is independent of the implicit basepoint (cf.\\ \\cite[Proposition II.8.8]{bridson}), due to Proposition \\ref{additive}.\n\n\n\t\tConsider the product space $X^\\infty\\times \\R_+$ endowed with the product of the cone topology\n\t\ton $X^\\infty$ and the usual topology on $\\R_+$. Define an equivalence relation $\\sim$ on $X^\\infty\\times\\R_+$\n\t\tby $(\\xi,s) \\sim (\\xi',s')$ if $s = s' = 0$ or $(\\xi,s) = (\\xi',s')$.\n\t\tNow define the \\textit{boundary cone} $CX^\\infty$ as the quotient of $X^\\infty \\times\\R_+$ by $\\sim$, endowed with the \t\t\tquotient topology which we will also refer to as the cone topology (context should always prevent any confusion). \n\t\tThe equivalence\n\t\tclass of $(\\xi,s) \\in X^\\infty\\times \\R_+$ is denoted by $[\\xi,s]$. We will sometimes use the notation $[0]$ to denote the \t\tequivalence class corresponding to $s = 0$. Note that continuity of a function defined on $CX^\\infty$\n\t\tis equivalent to sequential continuity because of the sequentially characterized topology on $X^\\infty \\times \\R_+$. \n\n\n\t\tNow define the \\textit{pairing} on $X \\times CX^\\infty$ to be the function \n\t\t\\[\\ip{\\cdot,\\cdot} \\from X\\times CX^\\infty \\to \\R,~ ~ ~ \\ip{x,[\\xi,s]} = \\begin{cases}\n\t\t\t\tsb_\\xi(x), & s > 0\\\\\n\t\t0, & s = 0.\\end{cases}\\]\n\n\n\\begin{definition}\n\t\t\t\\label{def:epihoro}\n\t\t\tConsider a nonempty subset $C$ of $X$ and a real-valued function $f\\from C \\to \\R$.\n\t\t\tA \\textit{Busemann subgradient} of $f$ at a point $x\\in C$ is an element $[\\xi,s] \\in CX^\\infty$\n\t\t\tsuch that $x$ minimizes $y\\mapsto f(y) - \\ip{y,[\\xi,s]}$ over $C$. The function $f$ is \\textit{Busemann subdifferentiable} if it has a Busemann subgradient at every point in $C$.\n\n\t\\end{definition}\n\nAn immediate consequence of the definition is that a point $x \\in C$ minimizes a function $f\\from C\\to\\R$ if and only if $[0]$ is a Busemann subgradient of $f$ at $x$.\n\n\n\t\t\\begin{equation}\n\t\t\t\\label{eqn:busesub}\n\t\t\tf(y) \\geq f(x) + sb_{x,\\xi}(y) \\text{ for all } y\\in C,\n\t\t\\end{equation}\n\t\ta version of the subgradient inequality in Hadamard space.\n\n\nFor simplicity, we restrict attention to our usual setting of a Hadamard space $X$.  We briefly review some necessary tangent space constructions:  for details, see \\cite{bridson}.  For any point $x \\in X$, we denote by $\\Theta_x X$  the set of all nonconstant geodesics issuing at~$x$. \n\n\n\t\tGiven three points $x, y, z \\in X$, a \\textit{geodesic triangle} $\\Delta = \\Delta(x, y, z)$\n\tis the union of three geodesic segments (its sides) joining each pair of points. A\n\t\\textit{comparison triangle} for $\\Delta$ is a triangle  $\\Delta(\\bar x, \\bar y,  \\bar z)$ in $\\R^2$\n\tthat has side lengths equal\n\tto those of $\\Delta$. \n\tTake two geodesics $\\gamma,\\eta \\in \\Theta_x$, let $\\gamma_t = \\gamma(t), \\eta_s = \\eta(s)$ for $t,s\\geq 0$ and \n\tlet $\\Delta(\\bar \\gamma_t,\\bar x,\\bar \\eta_s)$ be a comparison triangle in $\\R^2$ for $\\Delta(\\gamma_t,x,\\eta_s)$. Then the\n\tangle $\\angle \\bar \\gamma_t\\bar x\\bar \\eta_s$ is a nondecreasing function of both $t$ and $s$ and the\n\t\\textit{Alexandrov angle} between $\\gamma$ and $\\eta$ is well-defined as the following limit: \n\t\\[\\angle(\\gamma,\\eta) := \\lim_{t,s\\downarrow 0} \\angle \\bar \\gamma_t \\bar x\\bar \\eta_s.\\]\n\t\tThe Alexandrov angle induces a metric on the set $\\Sigma_x X$ of equivalence classes of geodesics in $\\Theta_x X$, where two geodesics $\\gamma, \\eta\\in \\Theta_x X$ are considered equivalent if $\\angle(\\gamma,\\eta) = 0$. \nFor a geodesic $\\gamma \\in \\Theta_x X$, we denote by $[\\gamma]$ its equivalence class. \n The {\\it tangent space} $T_x X$ of $X$ at $x$ is the Euclidean cone over the metric space $(\\Sigma_x X, \\angle)$ (see \\cite[Chapter I, Definition 5.6]{bridson}). \n For $v = ([\\gamma],r), w = ([\\eta],s) \\in T_x X$ the {\\it scalar product} of $v$ and $w$ is given by $\\langle \\langle v,w \\rangle \n \\rangle = rs\\cos \\angle(\\gamma, \\eta)$. For a geodesically convex function $f\\from X\\to \\R$, the \\textit{subdifferential} of $f$ at $x$, denoted by\n $\\p f(x)$, can be characterized (see \\cite[Proposition 4.4]{sumrule})\n as the elements $([\\eta],s)\\in T_xX$ satisfying the \\textit{subgradient inequality}\n \\begin{equation}\n\t \\label{eqn:cat0subgrad}\n \\ip{\\langle ([\\eta],s),([\\gamma_y],d(x,y))\\rangle} + f(x) \\leq f(y) \\text{ for all } y\\in X,\n \\end{equation}\n where $\\gamma_y\\from[0,d(x,y)]\\to X$ denotes the geodesic from $x$ to $y$. Any element $v\\in \\p f(x)$ is called a \\textit{subgradient}\n of $f$ at $x$.  A related idea used in the study of gradient flows in Alexandrov spaces is the \\textit{minus-subdifferential} \\cite[Definition 3.6]{GigliNobili2021}. \n\n\nWe denote the projection\n\t\t\tonto a nonempty closed geodesically convex set $C$ by $P_C$ \n\t\t\t(well-defined by \\cite[Proposition II.2.4]{bridson}).\t\n\t\t\tDefine an oracle \\texttt{Busemann}\n\t\t\tthat accepts a Busemann subdifferentiable function $g\\from C\\to \\R$ and a point $x\\in C$, and \n\t\t\treturns a Busemann subgradient\n\t\t\tfor $g$ at $x$:\n\t\t\t$[\\xi,s] \\leftarrow \\texttt{Busemann}(g,x)$. If $[\\xi,s]$ is a Busemann subgradient for $g$\n\t\t\tat $x$, we will use the notation $r_{x,\\xi}$ to denote the ray \n\t\t\tissuing from $x$ with direction $\\xi$.\n\t\t\t\\begin{customthm}{A}\n\t\t\t\t\\label{assumptionA}~\\\\\n\t\t\t\t\\vspace{-0.5cm}\n\t\t\t\\begin{enumerate}[label=(\\roman*)]\n\t\t\t\\item $(X,d)$ is a Hadamard space with the geodesic extension property.\n\t\t\t\\item $C\\seq X$ is nonempty, closed, and geodesically convex.\n\n\t\t\t\\item The function $f$ decomposes as $f = \\sum_{i=1}^m f_i$, where each \n\t\t\t\t$f_i, i = 1,\\dots,m$ is Busemann subdifferentiable on an open set \n\t\t\tcontaining $C$.\n\t\t\t\t\t\t\\item The optimal set of $\\inf_{x\\in C} f(x)$ is nonempty, and denoted by $X^*$. The optimal value of the\n\t\t\t\tproblem is denoted by $f_{\\text{opt}}$.\n\t\t\t\\item There is a constant $L \\geq 0$ such that\n\t\t\t\tfor all $i=1,\\dots,m$, every Busemann subgradient $[\\xi,s]$ for $f_i$ at every point in $C$\n\t\t\t\thas $s\\leq L$.\n\\end{enumerate}\n\t\\end{customthm}\n\n\n\\begin{algorithm}[h!]\n\t\t\\caption{Stochastic Busemann Subgradient Method}\n\t\t\t\\begin{algorithmic}\n\t\t\t\t\\Require $x^0 \\in C, \\st{t_k}_{k=0}^\\infty \\subset (0,+\\infty), \\st{i(k)}_{k=0}^\\infty$ independent Uniform$\\st{1,\n\t\t\t\t\\dots,m}$ \n\t\t\t\t\\For{$k=0,1,2\\dots$}\n\t\t\t\t%\\State $x^{k,0} \\leftarrow x^k$\n\t\t\t\t%\\For{$i=0,1,\\dots,m-1$}\n\t\t\t\t\\State $[\\xi_{k},s_{k}] \\leftarrow \\texttt{Busemann}(f_{i(k)}, x^{k})$\n\t\t\t\t\\State $x^{k+1} \\leftarrow P_C\\left(r_{x_{k},\\xi_{k}}\\left(s_{k}t_k\\right)\\right)$\n\t\t\t\t%\\EndFor\n\t\t\t\t%\\State $x^{k+1} \\leftarrow x^{k,m}$\n\t\t\t\t\\EndFor \t\t\n\t\t\t\t\\end{algorithmic}\n\\label{alg:stochsub}\n\t\t\\end{algorithm}\n\n\t\t\t$f^k_{\\text{best}} := \\min_{i=1,\\dots,k} f(x^i) - f_{\\text{opt}}$.  For stochastic terminology, see \\cite{resnick}.\n","units":[{"start":19489,"end":21195,"start_line":243,"end_line":262,"raw_tex":"A metric space $(X,d)$ is a\n\t\t\\textit{geodesic metric space} if every two points $x,y\\in X$ can be joined by a \n\t\t \\textit{geodesic},\n\t\twhich is to say a map $\\gamma$ from a closed interval $[a,b]$ into $X$ with $d(\\gamma(t),\\gamma(t')) = |t-t'|$\n\t\tfor all $t,t'\\in [a,b]$. A \\textit{ray} is a geodesic with domain $\\R_+$, and we say $r$ \\textit{issues from}\n\t\t$x$ if $r(0) = x$. A geodesic metric space is said to be \\textit{CAT(0)} if the map $t\\mapsto \\frac{1}{2}d(\\gamma(t),a)^2$ is $1$-strongly convex for every $a\\in X$ and geodesic $\\gamma$. A \\textit{Hadamard space} is a complete CAT(0) space. \n\t\tIn a Hadamard space, there is exactly one geodesic joining $x$ to $y$ for every $x,y\\in X$. \n\t\tWe say that\n\t\ta Hadamard space has the \\textit{geodesic extension property} if for every $x\\neq y\\in X$ there exists a \n\t\tray $r\\from\\R_+\\to X$ with $r(0) = x$ and $r(t) = y$ for some $t>0$. A subset $C$ of a Hadamard \n\t\tspace $X$ is \\textit{geodesically\n\t\tconvex} if the geodesic between any two points in $C$ is contained in $C$. A metric space $(X,d)$ is \\textit{proper} if the closed ball $B_r(x)$ is compact for every $x\\in X, r > 0$.\n\n\t\tHenceforth, $(X,d)$ will be a Hadamard space with the geodesic extension property, and to avoid degenerate\n\t\tcase-splitting in forthcoming results we will assume $X$ has at least two points. The significance of this technical\n\t\tassumption is that there always exist rays.\n\t\tGiven a ray $r$ on $X$ we may associate the \n\t\tcorresponding \\textit{Busemann function} $b_r\\from X\\to \\R$ defined by\n\t\t\\[b_r(z) = \\lim_{t\\to\\infty}(d(z,r(t)) - t) ~ ~ ~ (z\\in X).\\]\nIt is easy to see that the Busemann function $b_r$ is 1-Lipschitz, convex, and satisfies $b_r(r(0)) = 0$.\n","sha256_utf8":"ee83b4e123ffcae4c96949a2d13af0bea9e4d70512ae460411c3f17ef19a2244"},{"start":21992,"end":23367,"start_line":278,"end_line":292,"raw_tex":"\t\tTwo rays $r,r'$ are said to be \\textit{asymptotic}\n\t\tif there exists a positive constant $K\\geq 0$ such that $d(r(t),r'(t)) \\leq K$ for all $t\\geq 0$. \n\t\tThis defines an equivalence relation on rays in $X$: the set of equivalence classes is \n\t\tdenoted by $X^{\\infty}$ and called the \\textit{boundary of $X$ at infinity}, or simply the \\textit{boundary}. Note our assumption that $X$ has at least two points and the geodesic extension property guarantees the boundary is nonempty.\n\t\tThe equivalence class of a particular \n\t\tray $r$ is denoted by $r(\\infty)$, and we say a ray $r$ has \\textit{direction} $\\xi \\in X^\\infty$ if $r$\n\t\tbelongs to the equivalence class $\\xi$. Given $x\\in X$ and $\\xi \\in X^\\infty$, there exists a unique\n\t\tray $r$ issuing from $x$ such that $r(\\infty) = \\xi$ \\cite[II.8.2]{bridson}. \n\t\tIf we fix an arbitrary \n\t\treference point $\\bar x \\in X$, we can thus identify any $\\xi \\in X^\\infty$ with the unique ray $r_{\\bar x,\\xi}$ \n\t\tissuing from $\\bar x$ with direction $\\xi$, and hence to the unique\n\t\tcorresponding Busemann function which we may denote by either $b_{r_{\\bar x,\\xi}}$ or $b_{\\bar x,\\xi}$. \n\t\tTo avoid cumbersome notation, we henceforth\n\t\tfix such a reference point $\\bar x \\in X$ and denote the corresponding Busemann function \n\t\t$b_{r_{\\bar x,\\xi}}$ by simply $b_\\xi$.  We note the following fact \\cite[Corollary II.8.20]{bridson}.\n","sha256_utf8":"34d4ffa7f95f31fa86b0d1cf0f8c52d885d540d29951f674198ade9f0fac743f"},{"start":23553,"end":24159,"start_line":299,"end_line":304,"raw_tex":"\t\tThe space $X^\\infty$ is naturally endowed with the so-called \\textit{cone topology} \n\t\t\\cite[Definition II.8.6]{bridson}, and notably this space is first-countable \n\t\t\\cite[Proof of Theorem II.8.13]{bridson}. \n\t\t\t\tIn particular, the cone topology is completely specified by \n\t\tthe convergent sequences, so let us mention that $\\xi_n \\to \\xi \\in X^\\infty$ if and only if \n\t\t$b_{\\xi_n} \\to b_{\\xi}$ uniformly on bounded subsets of $X$.  The latter property, and hence the cone topology, is independent of the implicit basepoint (cf.\\ \\cite[Proposition II.8.8]{bridson}), due to Proposition \\ref{additive}.\n","sha256_utf8":"773dd5efd7294b37a27e3c917b7dd0c4f275d3fd9ea701bfcbfc0886055d027f"},{"start":29207,"end":30094,"start_line":388,"end_line":394,"raw_tex":"\t\tConsider the product space $X^\\infty\\times \\R_+$ endowed with the product of the cone topology\n\t\ton $X^\\infty$ and the usual topology on $\\R_+$. Define an equivalence relation $\\sim$ on $X^\\infty\\times\\R_+$\n\t\tby $(\\xi,s) \\sim (\\xi',s')$ if $s = s' = 0$ or $(\\xi,s) = (\\xi',s')$.\n\t\tNow define the \\textit{boundary cone} $CX^\\infty$ as the quotient of $X^\\infty \\times\\R_+$ by $\\sim$, endowed with the \t\t\tquotient topology which we will also refer to as the cone topology (context should always prevent any confusion). \n\t\tThe equivalence\n\t\tclass of $(\\xi,s) \\in X^\\infty\\times \\R_+$ is denoted by $[\\xi,s]$. We will sometimes use the notation $[0]$ to denote the \t\tequivalence class corresponding to $s = 0$. Note that continuity of a function defined on $CX^\\infty$\n\t\tis equivalent to sequential continuity because of the sequentially characterized topology on $X^\\infty \\times \\R_+$. \n","sha256_utf8":"010174ec874d12255390587bfd2ad03c84ffb95870960c35df3912b30d2f935c"},{"start":31402,"end":31621,"start_line":416,"end_line":419,"raw_tex":"\t\tNow define the \\textit{pairing} on $X \\times CX^\\infty$ to be the function \n\t\t\\[\\ip{\\cdot,\\cdot} \\from X\\times CX^\\infty \\to \\R,~ ~ ~ \\ip{x,[\\xi,s]} = \\begin{cases}\n\t\t\t\tsb_\\xi(x), & s > 0\\\\\n\t\t0, & s = 0.\\end{cases}\\]\n","sha256_utf8":"47909b8d561ab5bce66bdfa655ae182bfb737b7033d93499762c17fded7673e1"},{"start":34484,"end":34913,"start_line":463,"end_line":469,"raw_tex":"\\begin{definition}\n\t\t\t\\label{def:epihoro}\n\t\t\tConsider a nonempty subset $C$ of $X$ and a real-valued function $f\\from C \\to \\R$.\n\t\t\tA \\textit{Busemann subgradient} of $f$ at a point $x\\in C$ is an element $[\\xi,s] \\in CX^\\infty$\n\t\t\tsuch that $x$ minimizes $y\\mapsto f(y) - \\ip{y,[\\xi,s]}$ over $C$. The function $f$ is \\textit{Busemann subdifferentiable} if it has a Busemann subgradient at every point in $C$.\n\n\t\\end{definition}","sha256_utf8":"13727b4b5d065e8960269f749ea041e0f7772d41365aac4ff23e93ce98914008"},{"start":34926,"end":35094,"start_line":472,"end_line":472,"raw_tex":"An immediate consequence of the definition is that a point $x \\in C$ minimizes a function $f\\from C\\to\\R$ if and only if $[0]$ is a Busemann subgradient of $f$ at $x$.\n","sha256_utf8":"badd08378483e2ec6482dc4ffa140ed9dd652cb6869dabea146896f97cf516f1"},{"start":36352,"end":36531,"start_line":494,"end_line":498,"raw_tex":"\t\t\\begin{equation}\n\t\t\t\\label{eqn:busesub}\n\t\t\tf(y) \\geq f(x) + sb_{x,\\xi}(y) \\text{ for all } y\\in C,\n\t\t\\end{equation}\n\t\ta version of the subgradient inequality in Hadamard space.\n","sha256_utf8":"032cec872a7efaa2bb86f6aaf4b0d234e158d3249b07f952a05489882505c1db"},{"start":69065,"end":69353,"start_line":991,"end_line":991,"raw_tex":"For simplicity, we restrict attention to our usual setting of a Hadamard space $X$.  We briefly review some necessary tangent space constructions:  for details, see \\cite{bridson}.  For any point $x \\in X$, we denote by $\\Theta_x X$  the set of all nonconstant geodesics issuing at~$x$. \n","sha256_utf8":"40680256200a43cf8cdd1f7c15f0290cd8052e04a13227099805eb74a2afb812"},{"start":69355,"end":71587,"start_line":994,"end_line":1016,"raw_tex":"\t\tGiven three points $x, y, z \\in X$, a \\textit{geodesic triangle} $\\Delta = \\Delta(x, y, z)$\n\tis the union of three geodesic segments (its sides) joining each pair of points. A\n\t\\textit{comparison triangle} for $\\Delta$ is a triangle  $\\Delta(\\bar x, \\bar y,  \\bar z)$ in $\\R^2$\n\tthat has side lengths equal\n\tto those of $\\Delta$. \n\tTake two geodesics $\\gamma,\\eta \\in \\Theta_x$, let $\\gamma_t = \\gamma(t), \\eta_s = \\eta(s)$ for $t,s\\geq 0$ and \n\tlet $\\Delta(\\bar \\gamma_t,\\bar x,\\bar \\eta_s)$ be a comparison triangle in $\\R^2$ for $\\Delta(\\gamma_t,x,\\eta_s)$. Then the\n\tangle $\\angle \\bar \\gamma_t\\bar x\\bar \\eta_s$ is a nondecreasing function of both $t$ and $s$ and the\n\t\\textit{Alexandrov angle} between $\\gamma$ and $\\eta$ is well-defined as the following limit: \n\t\\[\\angle(\\gamma,\\eta) := \\lim_{t,s\\downarrow 0} \\angle \\bar \\gamma_t \\bar x\\bar \\eta_s.\\]\n\t\tThe Alexandrov angle induces a metric on the set $\\Sigma_x X$ of equivalence classes of geodesics in $\\Theta_x X$, where two geodesics $\\gamma, \\eta\\in \\Theta_x X$ are considered equivalent if $\\angle(\\gamma,\\eta) = 0$. \nFor a geodesic $\\gamma \\in \\Theta_x X$, we denote by $[\\gamma]$ its equivalence class. \n The {\\it tangent space} $T_x X$ of $X$ at $x$ is the Euclidean cone over the metric space $(\\Sigma_x X, \\angle)$ (see \\cite[Chapter I, Definition 5.6]{bridson}). \n For $v = ([\\gamma],r), w = ([\\eta],s) \\in T_x X$ the {\\it scalar product} of $v$ and $w$ is given by $\\langle \\langle v,w \\rangle \n \\rangle = rs\\cos \\angle(\\gamma, \\eta)$. For a geodesically convex function $f\\from X\\to \\R$, the \\textit{subdifferential} of $f$ at $x$, denoted by\n $\\p f(x)$, can be characterized (see \\cite[Proposition 4.4]{sumrule})\n as the elements $([\\eta],s)\\in T_xX$ satisfying the \\textit{subgradient inequality}\n \\begin{equation}\n\t \\label{eqn:cat0subgrad}\n \\ip{\\langle ([\\eta],s),([\\gamma_y],d(x,y))\\rangle} + f(x) \\leq f(y) \\text{ for all } y\\in X,\n \\end{equation}\n where $\\gamma_y\\from[0,d(x,y)]\\to X$ denotes the geodesic from $x$ to $y$. Any element $v\\in \\p f(x)$ is called a \\textit{subgradient}\n of $f$ at $x$.  A related idea used in the study of gradient flows in Alexandrov spaces is the \\textit{minus-subdifferential} \\cite[Definition 3.6]{GigliNobili2021}. \n","sha256_utf8":"a17f078939c4b6d820ffbfce5de9698629e3291339e130cfb5498c09ffb6d4b0"},{"start":75870,"end":77180,"start_line":1094,"end_line":1120,"raw_tex":"We denote the projection\n\t\t\tonto a nonempty closed geodesically convex set $C$ by $P_C$ \n\t\t\t(well-defined by \\cite[Proposition II.2.4]{bridson}).\t\n\t\t\tDefine an oracle \\texttt{Busemann}\n\t\t\tthat accepts a Busemann subdifferentiable function $g\\from C\\to \\R$ and a point $x\\in C$, and \n\t\t\treturns a Busemann subgradient\n\t\t\tfor $g$ at $x$:\n\t\t\t$[\\xi,s] \\leftarrow \\texttt{Busemann}(g,x)$. If $[\\xi,s]$ is a Busemann subgradient for $g$\n\t\t\tat $x$, we will use the notation $r_{x,\\xi}$ to denote the ray \n\t\t\tissuing from $x$ with direction $\\xi$.\n\t\t\t\\begin{customthm}{A}\n\t\t\t\t\\label{assumptionA}~\\\\\n\t\t\t\t\\vspace{-0.5cm}\n\t\t\t\\begin{enumerate}[label=(\\roman*)]\n\t\t\t\\item $(X,d)$ is a Hadamard space with the geodesic extension property.\n\t\t\t\\item $C\\seq X$ is nonempty, closed, and geodesically convex.\n\n\t\t\t\\item The function $f$ decomposes as $f = \\sum_{i=1}^m f_i$, where each \n\t\t\t\t$f_i, i = 1,\\dots,m$ is Busemann subdifferentiable on an open set \n\t\t\tcontaining $C$.\n\t\t\t\t\t\t\\item The optimal set of $\\inf_{x\\in C} f(x)$ is nonempty, and denoted by $X^*$. The optimal value of the\n\t\t\t\tproblem is denoted by $f_{\\text{opt}}$.\n\t\t\t\\item There is a constant $L \\geq 0$ such that\n\t\t\t\tfor all $i=1,\\dots,m$, every Busemann subgradient $[\\xi,s]$ for $f_i$ at every point in $C$\n\t\t\t\thas $s\\leq L$.\n\\end{enumerate}\n\t\\end{customthm}\n","sha256_utf8":"30a1e320c8ee6eee9d68fa490b4ec04b2be82c3d47be8e6ad056a3c10c0fa632"},{"start":77232,"end":77844,"start_line":1124,"end_line":1139,"raw_tex":"\\begin{algorithm}[h!]\n\t\t\\caption{Stochastic Busemann Subgradient Method}\n\t\t\t\\begin{algorithmic}\n\t\t\t\t\\Require $x^0 \\in C, \\st{t_k}_{k=0}^\\infty \\subset (0,+\\infty), \\st{i(k)}_{k=0}^\\infty$ independent Uniform$\\st{1,\n\t\t\t\t\\dots,m}$ \n\t\t\t\t\\For{$k=0,1,2\\dots$}\n\t\t\t\t%\\State $x^{k,0} \\leftarrow x^k$\n\t\t\t\t%\\For{$i=0,1,\\dots,m-1$}\n\t\t\t\t\\State $[\\xi_{k},s_{k}] \\leftarrow \\texttt{Busemann}(f_{i(k)}, x^{k})$\n\t\t\t\t\\State $x^{k+1} \\leftarrow P_C\\left(r_{x_{k},\\xi_{k}}\\left(s_{k}t_k\\right)\\right)$\n\t\t\t\t%\\EndFor\n\t\t\t\t%\\State $x^{k+1} \\leftarrow x^{k,m}$\n\t\t\t\t\\EndFor \t\t\n\t\t\t\t\\end{algorithmic}\n\\label{alg:stochsub}\n\t\t\\end{algorithm}","sha256_utf8":"c0cfd00b7d0a365f4e2f507b634191c3dd304fed76aa1957d21d0addd3215174"},{"start":80256,"end":80375,"start_line":1192,"end_line":1192,"raw_tex":"\t\t\t$f^k_{\\text{best}} := \\min_{i=1,\\dots,k} f(x^i) - f_{\\text{opt}}$.  For stochastic terminology, see \\cite{resnick}.\n","sha256_utf8":"352220d52bb7bd7179b6c9392582e4b5d0bfc80ab85f40acc97213d749d8de6a"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\\textbf{(Complexity of stochastic Busemann subgradient method)}\n\t\t\t\\label{thm:stochcomplexity}\n\t\t\tSuppose Assumption \\ref{assumptionA} holds, and let $\\st{t_k}_{k=0}^\\infty$ be a sequence of positive \n\t\t\tstepsizes and $\\st{i(k)}_{k=0}^\\infty$ a sequence of independent random variables distributed\n\t\t\tuniformly over $\\st{1,\\dots,m}$.\n\t\t\tLet $\\st{x^k}_{k=0}^\\infty$ be the sequence of iterates \n\t\t\tgenerated by Algorithm \\ref{alg:stochsub} with the above parameters.\n\t\t\tSuppose furthermore that the diameter of $C$ is bounded above by $D > 0$.\n\t\t\tIf $t_k = \\frac{D}{Lm\\sqrt{k+1}}$ then for all $k\\geq 2$,\n\t\t\t\\[\\mathbb{E}\\left[\\min_{i=1,\\dots,k} f(x^i) - f_{\\text{opt}}\\right] \\leq \\frac{2(1+\\log(3))mLD}{\\sqrt{k+2}}.\\]\n\t\t\tMore generally, if $\\sum_{k=0}^nt_k^2/\\sum_{k=0}^nt_k \\to 0$ as $n\\to \\infty$ then $\\mathbb{E}[f^k_{\\text{best}}] \\to f_{\\text{opt}}$ as $k\\to \\infty$ even if $C$ is unbounded.\n\t\\end{theorem}","units":[{"start":80378,"end":81305,"start_line":1193,"end_line":1204,"raw_tex":"\\begin{theorem}\\textbf{(Complexity of stochastic Busemann subgradient method)}\n\t\t\t\\label{thm:stochcomplexity}\n\t\t\tSuppose Assumption \\ref{assumptionA} holds, and let $\\st{t_k}_{k=0}^\\infty$ be a sequence of positive \n\t\t\tstepsizes and $\\st{i(k)}_{k=0}^\\infty$ a sequence of independent random variables distributed\n\t\t\tuniformly over $\\st{1,\\dots,m}$.\n\t\t\tLet $\\st{x^k}_{k=0}^\\infty$ be the sequence of iterates \n\t\t\tgenerated by Algorithm \\ref{alg:stochsub} with the above parameters.\n\t\t\tSuppose furthermore that the diameter of $C$ is bounded above by $D > 0$.\n\t\t\tIf $t_k = \\frac{D}{Lm\\sqrt{k+1}}$ then for all $k\\geq 2$,\n\t\t\t\\[\\mathbb{E}\\left[\\min_{i=1,\\dots,k} f(x^i) - f_{\\text{opt}}\\right] \\leq \\frac{2(1+\\log(3))mLD}{\\sqrt{k+2}}.\\]\n\t\t\tMore generally, if $\\sum_{k=0}^nt_k^2/\\sum_{k=0}^nt_k \\to 0$ as $n\\to \\infty$ then $\\mathbb{E}[f^k_{\\text{best}}] \\to f_{\\text{opt}}$ as $k\\to \\infty$ even if $C$ is unbounded.\n\t\\end{theorem}","sha256_utf8":"3db08a9f967417a1f44e0f72cbd3852ae2cf993cc010bbb7520011f10811b4e8"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\t\tMore generally, $[\\xi,s]$ is a Busemann subgradient of $f\\from C\\to\\R$ at $x$ if and only if\n\t\t\\begin{equation}\n\t\t\t\\label{eqn:busesubpre}\n\t\t\tf(y) - sb_\\xi(y) \\geq f(x) - sb_\\xi(x) \\text{ for all } y\\in C.\n\t\t\\end{equation}\n\t\tWe adopt the same convention that $sb_\\xi$ is identically $0$ if $s = 0$.\n\t\tThe Busemann functions for asymptotic rays differ only by an additive constant \n\t\t(Proposition \\ref{additive}), so it follows that inequality\n\t\t(\\ref{eqn:busesubpre}) is equivalent to \n\n\n\\begin{theorem}\n\t\t\t\\label{thm:subgrad}\nGiven a Hadamard space $X$ and $C \\seq X$, suppose that the function $f\\from C\\to \\R$ is geodesically convex, with a Busemann subgradient $[\\xi,s]$ at $x\\in C$. \n\t\t\tLet $r$ be a ray issuing from $x$ with direction $\\xi$.\n\t\t\tExtend $r$ to a geodesic line\n\t\t\t$\\tilde{r}\\from \\R\\to X$ and define a new ray $r_-\\from \\R_+\\to X$ by $r_-(t) = \\tilde{r}(-t)$.\n\t\t\tThen $([r_-],s) \\in \\p f(x)$.\t\n\t\t\\end{theorem}\n\n\\begin{proof}\n\t\t\tBy definition of the scalar product on $T_xX$, the subgradient inequality we wish to prove is\n\t\t\t\\[s d(x,y)\\cos\\angle(r_-,\\gamma_y) + f(x)\\leq f(y) \\text{ for all } y\\in X.\\]\n\t\t\tBusemann subdifferentiability of $f$ implies, by (\\ref{eqn:busesub}), \n\t\t\t\\[s b_r(y) + f(x) \\leq f(y) \\text{ for all } y\\in X.\\]\n\t\t\tThus it is enough to prove (the case $s = 0$ being trivial):\n\t\t\t\\begin{equation}\n\t\t\t\t\\label{eqn:desired}\n\t\t\t\tb_r(y) \\geq d(x,y)\\cos\\angle(r_-,\\gamma_y) \\text{ for all } y\\in X.\n\t\t\t\\end{equation}\n\t\t\tWe proceed in two steps, first proving \n\t\t\t$\\cos\\angle(r_-,\\gamma_y) \\leq -\\cos\\angle(r,\\gamma_y)$ $(*)$.\n\t\t\tUsing the triangle inequality for angles we find:\n\t\t\t\\[\\pi = \\angle(r,r_-) \\leq \\angle(r_-,\\gamma_y) + \\angle(r,\\gamma_y).\\]\n\t\t\tHence $\\pi - \\angle(r,\\gamma_y)\\leq \\angle(r_-,\\gamma_y)$, and taking the cosine reverses the inequality: \n\\begin{equation} \\label{star}\n\\cos\\angle(r_-,\\gamma_y) \\leq \\cos(\\pi - \\angle(r,\\gamma_y)) = -\\cos\\angle(r,\\gamma_y).\n\\end{equation}\n\n\t\t\tWe next prove \n\\begin{equation} \\label{starstar}\nb_r(y) \\geq -d(x,y)\\cos\\angle(r,\\gamma_y).\n\\end{equation}\n\t\t\tBy the law of cosines we have:\n\t\t\t\\[d(r(t),y)^2 \\geq d(x,r(t))^2 + d(x,y)^2 - 2d(x,r(t))d(x,y)\\cos\\angle(r,\\gamma_y).\\]\n\t\t\tRearrangement leads to\n\t\t\t\\begin{equation}\n\t\t\t\t\\label{eqn:ineqprelim}\n\t\t\t\t\\frac{d(r(t),y)^2 - t^2}{2t} \\geq \\frac{d(x,y)^2}{2t} - d(x,y)\\cos\\angle(r,\\gamma_y).\n\t\t\\end{equation}\n\t\tThe lefthand side of \\eqref{eqn:ineqprelim} simplifies in the limit because $t\\mapsto d(r(t),y)-t$ is bounded:\n\t\t\t\\begin{align}\n\t\t\t\t\\label{eqn:quadraticbuse}\n\t\t\t\\lim_{t\\to \\infty}\\frac{d(r(t),y)^2 - t^2}{2t} &= \n\t\t\t\\lim_{t\\to \\infty}(d(r(t),y)-t)\\lim_{t\\to \\infty}\\frac{d(r(t),y) +t}{2t} \\nonumber\\\\\n\t\t\t&= b_r(y)\\lim_{t\\to \\infty}\\left(\\frac{d(r(t),y)-t}{2t} + 1 \\right)  \\nonumber \\\\\n\t\t\t&= b_r(y).\n\t\\end{align}\n\tUsing \\eqref{eqn:quadraticbuse} and sending $t\\to \\infty$ in (\\ref{eqn:ineqprelim}) yields \n\t(\\ref{starstar}).\n\t\t\tCombining (\\ref{star}) and (\\ref{starstar}) gives\n\t\t\t\\[d(x,y)\\cos\\angle(r_-,\\gamma_y) \\leq -d(x,y)\\cos\\angle(r,\\gamma_y) \\leq b_r(y).\\]\n\t\t\tThis proves (\\ref{eqn:desired}).\n\t\t\\end{proof}\n\n\\begin{lemma}\\textbf{(Projected Busemann subgradient inequality)}\n\t\t\t\\label{lem:projsub}\n\t\t\tSuppose the function $f\\from X\\to \\R$ is Busemann subdifferentiable on a nonempty, closed, and \n\t\t\tgeodesically convex set $C$. Let $x\\in C$, $t > 0$ and choose a\n\t\t\tBusemann subgradient $[\\xi,s]$ for $f$ at $x$.\n\t\t\tDefine the new point\n\t\t\t\\[x^+ = \\begin{cases}\n\t\t\t\t\tP_C\\left(r_{x,\\xi}\\left( s t\\right)\\right), & s > 0\\\\\n\t\t\t\t\tx, & s = 0,\n\t\t\\end{cases}\\]\n\t\twhere $r_{x,\\xi}$ is the ray issuing from $x$ with direction $\\xi$.\n\tThen for any $y \\in C$,\n\t\\[d(x^{+},y)^2\\leq d(x,y)^2 - 2t (f(x)- f(y)) +s^2t^2.\\]\n\t\t\\end{lemma}\n\n\\begin{proof}\n\t\t\tFor brevity we denote $r = r_{x,\\xi}$. If $s = 0$ then the desired inequality reduces\n\t\t\tto $f(x)\\leq f(y)$ for all $y\\in C$, which holds because $x$ minimizes $f$ over $C$ \n\t\t\t(cf.\\ the discussion\n\t\t\t\tfollowing Definition \\ref{def:epihoro}). Thus we may assume\n\t\t\t$s > 0$.\n\t\t\tSince $X$ is Hadamard, $x\\mapsto \\frac{1}{2}d(x,y)^2$ is 1-strongly convex for any $y\\in C$.\n\t\t\tIt follows that for all\n\t\t\t$\\delta \\geq s t$ we have\n\t\t\t\\begin{align*}\n\t\t\t\td(x^{+},y)^2 &= d\\left(P_C(r\\hspace{-0.05cm}\\left(st\\right)),P_C(y)\\right)^2\\\\\n\t\t\t\t&\\leq d\\left(r\\hspace{-0.05cm}\\left(st\\right),y\\right)^2\\\\\n\t\t\t\t&\\leq \\left(1-\\frac{s t}{\\delta}\\right)d(x,y)^2 + \\frac{s t}{\\delta}\n\t\t\t\td(r(\\delta),y)^2 - \\left(1 - \\frac{s t}{\\delta}\\right)\\frac{s t}{\\delta}\n\t\t\t\td(x,r(\\delta))^2\\\\\n\t\t\t\t&= \\left(1-\\frac{s t}{\\delta}\\right)d(x,y)^2 + \\frac{s t}{\\delta}\n\t\t\t\td(r(\\delta),y)^2 + s^2t^2 - \\delta s t\\\\\n\t\t\t\t&= \\left(1-\\frac{s t}{\\delta}\\right)d(x,y)^2 + \\frac{s t}{\\delta}(\n\t\t\td(r(\\delta),y)^2 - \\delta^2) + \n\t\t\ts^2t^2.\n\t\t\t\\end{align*}\n\t\t\tThe second line uses nonexpansivity of the projection $P_C$, see \\cite[Theorem 2.1.12]{bacakbook}.\n\t\t\tLetting $\\delta\\to \\infty$ and using once more \\eqref{eqn:quadraticbuse} we deduce\n\t\t\t\t\t\t\\[d(x^{+},y)^2 \\leq d(x,y)^2 +  2 s t b_r(y) + s^2t^2.\\]\n\t\t\tBy (\\ref{eqn:busesub}) we have\n\t\t\t$s b_r(y) \n\t\t\t\\leq -(f(x) - f(y))$.\n\t\t\tWe conclude:\n\t\t\t\\[d(x^{+},y)^2 \\leq d(x,y)^2 +  2s t b_r(y) + s^2t^2 \\leq\n\t\t\td(x,y)^2 - 2 t(f(x)-f(y)) +s^2t^2.\\]\n\t\t\\end{proof}\n\n\\begin{proof}\nDenoting the nonnegative integers by $\\N_0$, the underlying probability space is $\\Omega = \\st{1,\\dots,m}^{\\N_0}$ equipped with the product of the \nuniform $\\st{1,\\dots,m}$ probability measures.\nAt step $k \\in \\N_0$, choose a Busemann subgradient $[\\xi_{k},s_{k}]$ for $f_{i(k)}$ at $x^k$. \nThen define $x^{k+1} := P_C(r_{x^k, \\xi_{k}}(s_{k}t_k))$, as in Algorithm \\ref{alg:stochsub}.\n%We will show that \n%$\\E\\left[ \\min_{k=0,\\dots,n} f(x^k)\\right] - f_{\\text{opt}} = O(n^{-1/2})$.\nConsider a given (random) iterate $x^k$. \nLemma \\ref{lem:projsub} and Assumption \\ref{assumptionA}(v) together imply\n\\[d(x^{k+1},x^*)^2 \\leq d(x^k,x^*)^2 -2t_k(f_{i(k)}(x^k) - f_{i(k)}(x^*)) + L^2t_k^2.\\]\nLet $\\fami_k = \\sigma(x^0,\\dots,x^k)$ be the $\\sigma$-algebra generated by the variables $x^0,\\dots,x^k$.\nTaking the conditional expectation with respect to $\\fami_k$ gives\n\\begin{align*}\n\t\\E[d(x^{k+1},x^*)^2 \\mid \\fami_k] &\\leq d(x^k,x^*)^2 -2t_k\\E[f_{i(k)}(x^k)-f_{i(k)}(x^*) \\mid \\fami_k] + L^2t_k^2\\\\\n&= d(x^k,x^*)^2 -\\frac{2t_k}{m}\\sum_{i=1}^m(f_{i}(x^k)-f_{i}(x^*)) + L^2t_k^2\\\\\n&= d(x^k,x^*)^2 -\\frac{2t_k}{m}(f(x^k)-f_{\\text{opt}}) + L^2t_k^2.\n\\end{align*}\nNow taking the expectation, we arrive at\n\\[\\E[d(x^{k+1},x^*)^2] \\leq \\E[d(x^k,x^*)^2]-\\frac{2t_k}{m} \\E[f(x^k) - f_{\\text{opt}}] + L^2t_k^2.\\]\nSumming from $k= j$ to $n\\geq j$ and rearranging:\n\\[\\frac{2}{m}\\sum_{k=j}^n t_k \\E[f(x^k)-f_{\\text{opt}}] \\leq \\E[d(x^j,x^*)^2] + L^2\\sum_{k=j}^nt_k^2.\\]\nBy the elementary inequality $\\E[\\min_{k=j,\\dots,n}f(x^k)] \\leq \\min_{k=j,\\dots,n}\\E[f(x^k)]$, we derive\n\\[\\E\\left[\\min_{k=0,\\dots,n} f(x^k) - f_{\\text{opt}}\\right] \n\\leq \\E\\left[\\min_{k=j,\\dots,n}f(x^k) - f_{\\text{opt}}\\right] \\leq \\frac{m (\\E[d(x^j,x^*)^2] + L^2\\sum_{k=j}^nt_k^2)}{2\\sum_{k=j}^n t_k}.\\]\nFinally, since the diameter of $C$ is at most $D$ we have $\\E[d(x^j,x^*)^2] \\leq D^2$. Choosing $t_k = \\frac{D}{L\\sqrt{k+1}}$, $j = \\ceil{n/2}$,\nand appealing to \n\\cite[Lemma 8.27(b)]{beckfirstorder} gives \n\\[\\E\\left[\\min_{k=0,\\dots,n} f(x^k) - f_{\\text{opt}}\\right]  \\leq \\frac{2(1+\\log(3))mDL}{\\sqrt{n+2}} ~ ~ ~ \\text{ for all } n \\geq 2.\\]\nThe last statement of the theorem can be proven in exactly the same way\n\t\t\t as \\cite[Theorem 8.35(a)]{beckfirstorder}.\n\\end{proof}","units":[{"start":35865,"end":36352,"start_line":485,"end_line":493,"raw_tex":"\t\tMore generally, $[\\xi,s]$ is a Busemann subgradient of $f\\from C\\to\\R$ at $x$ if and only if\n\t\t\\begin{equation}\n\t\t\t\\label{eqn:busesubpre}\n\t\t\tf(y) - sb_\\xi(y) \\geq f(x) - sb_\\xi(x) \\text{ for all } y\\in C.\n\t\t\\end{equation}\n\t\tWe adopt the same convention that $sb_\\xi$ is identically $0$ if $s = 0$.\n\t\tThe Busemann functions for asymptotic rays differ only by an additive constant \n\t\t(Proposition \\ref{additive}), so it follows that inequality\n\t\t(\\ref{eqn:busesubpre}) is equivalent to \n","sha256_utf8":"4064503f8922cd494e7b5b618526c49d061e54cc6ed483dd56bd99b05c7e3d86"},{"start":71736,"end":72177,"start_line":1020,"end_line":1027,"raw_tex":"\\begin{theorem}\n\t\t\t\\label{thm:subgrad}\nGiven a Hadamard space $X$ and $C \\seq X$, suppose that the function $f\\from C\\to \\R$ is geodesically convex, with a Busemann subgradient $[\\xi,s]$ at $x\\in C$. \n\t\t\tLet $r$ be a ray issuing from $x$ with direction $\\xi$.\n\t\t\tExtend $r$ to a geodesic line\n\t\t\t$\\tilde{r}\\from \\R\\to X$ and define a new ray $r_-\\from \\R_+\\to X$ by $r_-(t) = \\tilde{r}(-t)$.\n\t\t\tThen $([r_-],s) \\in \\p f(x)$.\t\n\t\t\\end{theorem}","sha256_utf8":"3ea2cee13090e1dbd4ff7178e7f0486e1971c52b59be76612e9097e2f2a3c584"},{"start":72180,"end":74310,"start_line":1028,"end_line":1071,"raw_tex":"\\begin{proof}\n\t\t\tBy definition of the scalar product on $T_xX$, the subgradient inequality we wish to prove is\n\t\t\t\\[s d(x,y)\\cos\\angle(r_-,\\gamma_y) + f(x)\\leq f(y) \\text{ for all } y\\in X.\\]\n\t\t\tBusemann subdifferentiability of $f$ implies, by (\\ref{eqn:busesub}), \n\t\t\t\\[s b_r(y) + f(x) \\leq f(y) \\text{ for all } y\\in X.\\]\n\t\t\tThus it is enough to prove (the case $s = 0$ being trivial):\n\t\t\t\\begin{equation}\n\t\t\t\t\\label{eqn:desired}\n\t\t\t\tb_r(y) \\geq d(x,y)\\cos\\angle(r_-,\\gamma_y) \\text{ for all } y\\in X.\n\t\t\t\\end{equation}\n\t\t\tWe proceed in two steps, first proving \n\t\t\t$\\cos\\angle(r_-,\\gamma_y) \\leq -\\cos\\angle(r,\\gamma_y)$ $(*)$.\n\t\t\tUsing the triangle inequality for angles we find:\n\t\t\t\\[\\pi = \\angle(r,r_-) \\leq \\angle(r_-,\\gamma_y) + \\angle(r,\\gamma_y).\\]\n\t\t\tHence $\\pi - \\angle(r,\\gamma_y)\\leq \\angle(r_-,\\gamma_y)$, and taking the cosine reverses the inequality: \n\\begin{equation} \\label{star}\n\\cos\\angle(r_-,\\gamma_y) \\leq \\cos(\\pi - \\angle(r,\\gamma_y)) = -\\cos\\angle(r,\\gamma_y).\n\\end{equation}\n\n\t\t\tWe next prove \n\\begin{equation} \\label{starstar}\nb_r(y) \\geq -d(x,y)\\cos\\angle(r,\\gamma_y).\n\\end{equation}\n\t\t\tBy the law of cosines we have:\n\t\t\t\\[d(r(t),y)^2 \\geq d(x,r(t))^2 + d(x,y)^2 - 2d(x,r(t))d(x,y)\\cos\\angle(r,\\gamma_y).\\]\n\t\t\tRearrangement leads to\n\t\t\t\\begin{equation}\n\t\t\t\t\\label{eqn:ineqprelim}\n\t\t\t\t\\frac{d(r(t),y)^2 - t^2}{2t} \\geq \\frac{d(x,y)^2}{2t} - d(x,y)\\cos\\angle(r,\\gamma_y).\n\t\t\\end{equation}\n\t\tThe lefthand side of \\eqref{eqn:ineqprelim} simplifies in the limit because $t\\mapsto d(r(t),y)-t$ is bounded:\n\t\t\t\\begin{align}\n\t\t\t\t\\label{eqn:quadraticbuse}\n\t\t\t\\lim_{t\\to \\infty}\\frac{d(r(t),y)^2 - t^2}{2t} &= \n\t\t\t\\lim_{t\\to \\infty}(d(r(t),y)-t)\\lim_{t\\to \\infty}\\frac{d(r(t),y) +t}{2t} \\nonumber\\\\\n\t\t\t&= b_r(y)\\lim_{t\\to \\infty}\\left(\\frac{d(r(t),y)-t}{2t} + 1 \\right)  \\nonumber \\\\\n\t\t\t&= b_r(y).\n\t\\end{align}\n\tUsing \\eqref{eqn:quadraticbuse} and sending $t\\to \\infty$ in (\\ref{eqn:ineqprelim}) yields \n\t(\\ref{starstar}).\n\t\t\tCombining (\\ref{star}) and (\\ref{starstar}) gives\n\t\t\t\\[d(x,y)\\cos\\angle(r_-,\\gamma_y) \\leq -d(x,y)\\cos\\angle(r,\\gamma_y) \\leq b_r(y).\\]\n\t\t\tThis proves (\\ref{eqn:desired}).\n\t\t\\end{proof}","sha256_utf8":"fbf344a351495188e4f0932ef62e72af54977b90439540a617b31f868e3228b9"},{"start":77997,"end":78609,"start_line":1144,"end_line":1157,"raw_tex":"\\begin{lemma}\\textbf{(Projected Busemann subgradient inequality)}\n\t\t\t\\label{lem:projsub}\n\t\t\tSuppose the function $f\\from X\\to \\R$ is Busemann subdifferentiable on a nonempty, closed, and \n\t\t\tgeodesically convex set $C$. Let $x\\in C$, $t > 0$ and choose a\n\t\t\tBusemann subgradient $[\\xi,s]$ for $f$ at $x$.\n\t\t\tDefine the new point\n\t\t\t\\[x^+ = \\begin{cases}\n\t\t\t\t\tP_C\\left(r_{x,\\xi}\\left( s t\\right)\\right), & s > 0\\\\\n\t\t\t\t\tx, & s = 0,\n\t\t\\end{cases}\\]\n\t\twhere $r_{x,\\xi}$ is the ray issuing from $x$ with direction $\\xi$.\n\tThen for any $y \\in C$,\n\t\\[d(x^{+},y)^2\\leq d(x,y)^2 - 2t (f(x)- f(y)) +s^2t^2.\\]\n\t\t\\end{lemma}","sha256_utf8":"6031f1e48305510e0ec28b8fdd3290f0b4fb8bf901c5d4a754e3ab98b111397d"},{"start":78612,"end":80094,"start_line":1158,"end_line":1188,"raw_tex":"\\begin{proof}\n\t\t\tFor brevity we denote $r = r_{x,\\xi}$. If $s = 0$ then the desired inequality reduces\n\t\t\tto $f(x)\\leq f(y)$ for all $y\\in C$, which holds because $x$ minimizes $f$ over $C$ \n\t\t\t(cf.\\ the discussion\n\t\t\t\tfollowing Definition \\ref{def:epihoro}). Thus we may assume\n\t\t\t$s > 0$.\n\t\t\tSince $X$ is Hadamard, $x\\mapsto \\frac{1}{2}d(x,y)^2$ is 1-strongly convex for any $y\\in C$.\n\t\t\tIt follows that for all\n\t\t\t$\\delta \\geq s t$ we have\n\t\t\t\\begin{align*}\n\t\t\t\td(x^{+},y)^2 &= d\\left(P_C(r\\hspace{-0.05cm}\\left(st\\right)),P_C(y)\\right)^2\\\\\n\t\t\t\t&\\leq d\\left(r\\hspace{-0.05cm}\\left(st\\right),y\\right)^2\\\\\n\t\t\t\t&\\leq \\left(1-\\frac{s t}{\\delta}\\right)d(x,y)^2 + \\frac{s t}{\\delta}\n\t\t\t\td(r(\\delta),y)^2 - \\left(1 - \\frac{s t}{\\delta}\\right)\\frac{s t}{\\delta}\n\t\t\t\td(x,r(\\delta))^2\\\\\n\t\t\t\t&= \\left(1-\\frac{s t}{\\delta}\\right)d(x,y)^2 + \\frac{s t}{\\delta}\n\t\t\t\td(r(\\delta),y)^2 + s^2t^2 - \\delta s t\\\\\n\t\t\t\t&= \\left(1-\\frac{s t}{\\delta}\\right)d(x,y)^2 + \\frac{s t}{\\delta}(\n\t\t\td(r(\\delta),y)^2 - \\delta^2) + \n\t\t\ts^2t^2.\n\t\t\t\\end{align*}\n\t\t\tThe second line uses nonexpansivity of the projection $P_C$, see \\cite[Theorem 2.1.12]{bacakbook}.\n\t\t\tLetting $\\delta\\to \\infty$ and using once more \\eqref{eqn:quadraticbuse} we deduce\n\t\t\t\t\t\t\\[d(x^{+},y)^2 \\leq d(x,y)^2 +  2 s t b_r(y) + s^2t^2.\\]\n\t\t\tBy (\\ref{eqn:busesub}) we have\n\t\t\t$s b_r(y) \n\t\t\t\\leq -(f(x) - f(y))$.\n\t\t\tWe conclude:\n\t\t\t\\[d(x^{+},y)^2 \\leq d(x,y)^2 +  2s t b_r(y) + s^2t^2 \\leq\n\t\t\td(x,y)^2 - 2 t(f(x)-f(y)) +s^2t^2.\\]\n\t\t\\end{proof}","sha256_utf8":"f3e4316357c87faea016498d0caa177e35cf4b55ac60326e8c539b64ba941327"},{"start":81306,"end":83548,"start_line":1205,"end_line":1235,"raw_tex":"\\begin{proof}\nDenoting the nonnegative integers by $\\N_0$, the underlying probability space is $\\Omega = \\st{1,\\dots,m}^{\\N_0}$ equipped with the product of the \nuniform $\\st{1,\\dots,m}$ probability measures.\nAt step $k \\in \\N_0$, choose a Busemann subgradient $[\\xi_{k},s_{k}]$ for $f_{i(k)}$ at $x^k$. \nThen define $x^{k+1} := P_C(r_{x^k, \\xi_{k}}(s_{k}t_k))$, as in Algorithm \\ref{alg:stochsub}.\n%We will show that \n%$\\E\\left[ \\min_{k=0,\\dots,n} f(x^k)\\right] - f_{\\text{opt}} = O(n^{-1/2})$.\nConsider a given (random) iterate $x^k$. \nLemma \\ref{lem:projsub} and Assumption \\ref{assumptionA}(v) together imply\n\\[d(x^{k+1},x^*)^2 \\leq d(x^k,x^*)^2 -2t_k(f_{i(k)}(x^k) - f_{i(k)}(x^*)) + L^2t_k^2.\\]\nLet $\\fami_k = \\sigma(x^0,\\dots,x^k)$ be the $\\sigma$-algebra generated by the variables $x^0,\\dots,x^k$.\nTaking the conditional expectation with respect to $\\fami_k$ gives\n\\begin{align*}\n\t\\E[d(x^{k+1},x^*)^2 \\mid \\fami_k] &\\leq d(x^k,x^*)^2 -2t_k\\E[f_{i(k)}(x^k)-f_{i(k)}(x^*) \\mid \\fami_k] + L^2t_k^2\\\\\n&= d(x^k,x^*)^2 -\\frac{2t_k}{m}\\sum_{i=1}^m(f_{i}(x^k)-f_{i}(x^*)) + L^2t_k^2\\\\\n&= d(x^k,x^*)^2 -\\frac{2t_k}{m}(f(x^k)-f_{\\text{opt}}) + L^2t_k^2.\n\\end{align*}\nNow taking the expectation, we arrive at\n\\[\\E[d(x^{k+1},x^*)^2] \\leq \\E[d(x^k,x^*)^2]-\\frac{2t_k}{m} \\E[f(x^k) - f_{\\text{opt}}] + L^2t_k^2.\\]\nSumming from $k= j$ to $n\\geq j$ and rearranging:\n\\[\\frac{2}{m}\\sum_{k=j}^n t_k \\E[f(x^k)-f_{\\text{opt}}] \\leq \\E[d(x^j,x^*)^2] + L^2\\sum_{k=j}^nt_k^2.\\]\nBy the elementary inequality $\\E[\\min_{k=j,\\dots,n}f(x^k)] \\leq \\min_{k=j,\\dots,n}\\E[f(x^k)]$, we derive\n\\[\\E\\left[\\min_{k=0,\\dots,n} f(x^k) - f_{\\text{opt}}\\right] \n\\leq \\E\\left[\\min_{k=j,\\dots,n}f(x^k) - f_{\\text{opt}}\\right] \\leq \\frac{m (\\E[d(x^j,x^*)^2] + L^2\\sum_{k=j}^nt_k^2)}{2\\sum_{k=j}^n t_k}.\\]\nFinally, since the diameter of $C$ is at most $D$ we have $\\E[d(x^j,x^*)^2] \\leq D^2$. Choosing $t_k = \\frac{D}{L\\sqrt{k+1}}$, $j = \\ceil{n/2}$,\nand appealing to \n\\cite[Lemma 8.27(b)]{beckfirstorder} gives \n\\[\\E\\left[\\min_{k=0,\\dots,n} f(x^k) - f_{\\text{opt}}\\right]  \\leq \\frac{2(1+\\log(3))mDL}{\\sqrt{n+2}} ~ ~ ~ \\text{ for all } n \\geq 2.\\]\nThe last statement of the theorem can be proven in exactly the same way\n\t\t\t as \\cite[Theorem 8.35(a)]{beckfirstorder}.\n\\end{proof}","sha256_utf8":"a1a86e35dcccdd3ee4933609d41f66162ffa761075e8befcea03f61b64bd11ec"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"definition","kind":"definition","title":"Hadamard空间与Busemann函数","source_units":[{"start":19489,"end":21195,"start_line":243,"end_line":262,"raw_tex":"A metric space $(X,d)$ is a\n\t\t\\textit{geodesic metric space} if every two points $x,y\\in X$ can be joined by a \n\t\t \\textit{geodesic},\n\t\twhich is to say a map $\\gamma$ from a closed interval $[a,b]$ into $X$ with $d(\\gamma(t),\\gamma(t')) = |t-t'|$\n\t\tfor all $t,t'\\in [a,b]$. A \\textit{ray} is a geodesic with domain $\\R_+$, and we say $r$ \\textit{issues from}\n\t\t$x$ if $r(0) = x$. A geodesic metric space is said to be \\textit{CAT(0)} if the map $t\\mapsto \\frac{1}{2}d(\\gamma(t),a)^2$ is $1$-strongly convex for every $a\\in X$ and geodesic $\\gamma$. A \\textit{Hadamard space} is a complete CAT(0) space. \n\t\tIn a Hadamard space, there is exactly one geodesic joining $x$ to $y$ for every $x,y\\in X$. \n\t\tWe say that\n\t\ta Hadamard space has the \\textit{geodesic extension property} if for every $x\\neq y\\in X$ there exists a \n\t\tray $r\\from\\R_+\\to X$ with $r(0) = x$ and $r(t) = y$ for some $t>0$. A subset $C$ of a Hadamard \n\t\tspace $X$ is \\textit{geodesically\n\t\tconvex} if the geodesic between any two points in $C$ is contained in $C$. A metric space $(X,d)$ is \\textit{proper} if the closed ball $B_r(x)$ is compact for every $x\\in X, r > 0$.\n\n\t\tHenceforth, $(X,d)$ will be a Hadamard space with the geodesic extension property, and to avoid degenerate\n\t\tcase-splitting in forthcoming results we will assume $X$ has at least two points. The significance of this technical\n\t\tassumption is that there always exist rays.\n\t\tGiven a ray $r$ on $X$ we may associate the \n\t\tcorresponding \\textit{Busemann function} $b_r\\from X\\to \\R$ defined by\n\t\t\\[b_r(z) = \\lim_{t\\to\\infty}(d(z,r(t)) - t) ~ ~ ~ (z\\in X).\\]\nIt is easy to see that the Busemann function $b_r$ is 1-Lipschitz, convex, and satisfies $b_r(r(0)) = 0$.\n","sha256_utf8":"ee83b4e123ffcae4c96949a2d13af0bea9e4d70512ae460411c3f17ef19a2244"}]},{"id":"S002","semantic_type":"definition","kind":"definition","title":"无穷远方向与基点","source_units":[{"start":21992,"end":23367,"start_line":278,"end_line":292,"raw_tex":"\t\tTwo rays $r,r'$ are said to be \\textit{asymptotic}\n\t\tif there exists a positive constant $K\\geq 0$ such that $d(r(t),r'(t)) \\leq K$ for all $t\\geq 0$. \n\t\tThis defines an equivalence relation on rays in $X$: the set of equivalence classes is \n\t\tdenoted by $X^{\\infty}$ and called the \\textit{boundary of $X$ at infinity}, or simply the \\textit{boundary}. Note our assumption that $X$ has at least two points and the geodesic extension property guarantees the boundary is nonempty.\n\t\tThe equivalence class of a particular \n\t\tray $r$ is denoted by $r(\\infty)$, and we say a ray $r$ has \\textit{direction} $\\xi \\in X^\\infty$ if $r$\n\t\tbelongs to the equivalence class $\\xi$. Given $x\\in X$ and $\\xi \\in X^\\infty$, there exists a unique\n\t\tray $r$ issuing from $x$ such that $r(\\infty) = \\xi$ \\cite[II.8.2]{bridson}. \n\t\tIf we fix an arbitrary \n\t\treference point $\\bar x \\in X$, we can thus identify any $\\xi \\in X^\\infty$ with the unique ray $r_{\\bar x,\\xi}$ \n\t\tissuing from $\\bar x$ with direction $\\xi$, and hence to the unique\n\t\tcorresponding Busemann function which we may denote by either $b_{r_{\\bar x,\\xi}}$ or $b_{\\bar x,\\xi}$. \n\t\tTo avoid cumbersome notation, we henceforth\n\t\tfix such a reference point $\\bar x \\in X$ and denote the corresponding Busemann function \n\t\t$b_{r_{\\bar x,\\xi}}$ by simply $b_\\xi$.  We note the following fact \\cite[Corollary II.8.20]{bridson}.\n","sha256_utf8":"34d4ffa7f95f31fa86b0d1cf0f8c52d885d540d29951f674198ade9f0fac743f"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"边界的锥拓扑","source_units":[{"start":23553,"end":24159,"start_line":299,"end_line":304,"raw_tex":"\t\tThe space $X^\\infty$ is naturally endowed with the so-called \\textit{cone topology} \n\t\t\\cite[Definition II.8.6]{bridson}, and notably this space is first-countable \n\t\t\\cite[Proof of Theorem II.8.13]{bridson}. \n\t\t\t\tIn particular, the cone topology is completely specified by \n\t\tthe convergent sequences, so let us mention that $\\xi_n \\to \\xi \\in X^\\infty$ if and only if \n\t\t$b_{\\xi_n} \\to b_{\\xi}$ uniformly on bounded subsets of $X$.  The latter property, and hence the cone topology, is independent of the implicit basepoint (cf.\\ \\cite[Proposition II.8.8]{bridson}), due to Proposition \\ref{additive}.\n","sha256_utf8":"773dd5efd7294b37a27e3c917b7dd0c4f275d3fd9ea701bfcbfc0886055d027f"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"边界锥与零元","source_units":[{"start":29207,"end":30094,"start_line":388,"end_line":394,"raw_tex":"\t\tConsider the product space $X^\\infty\\times \\R_+$ endowed with the product of the cone topology\n\t\ton $X^\\infty$ and the usual topology on $\\R_+$. Define an equivalence relation $\\sim$ on $X^\\infty\\times\\R_+$\n\t\tby $(\\xi,s) \\sim (\\xi',s')$ if $s = s' = 0$ or $(\\xi,s) = (\\xi',s')$.\n\t\tNow define the \\textit{boundary cone} $CX^\\infty$ as the quotient of $X^\\infty \\times\\R_+$ by $\\sim$, endowed with the \t\t\tquotient topology which we will also refer to as the cone topology (context should always prevent any confusion). \n\t\tThe equivalence\n\t\tclass of $(\\xi,s) \\in X^\\infty\\times \\R_+$ is denoted by $[\\xi,s]$. We will sometimes use the notation $[0]$ to denote the \t\tequivalence class corresponding to $s = 0$. Note that continuity of a function defined on $CX^\\infty$\n\t\tis equivalent to sequential continuity because of the sequentially characterized topology on $X^\\infty \\times \\R_+$. \n","sha256_utf8":"010174ec874d12255390587bfd2ad03c84ffb95870960c35df3912b30d2f935c"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"空间与边界锥的配对","source_units":[{"start":31402,"end":31621,"start_line":416,"end_line":419,"raw_tex":"\t\tNow define the \\textit{pairing} on $X \\times CX^\\infty$ to be the function \n\t\t\\[\\ip{\\cdot,\\cdot} \\from X\\times CX^\\infty \\to \\R,~ ~ ~ \\ip{x,[\\xi,s]} = \\begin{cases}\n\t\t\t\tsb_\\xi(x), & s > 0\\\\\n\t\t0, & s = 0.\\end{cases}\\]\n","sha256_utf8":"47909b8d561ab5bce66bdfa655ae182bfb737b7033d93499762c17fded7673e1"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"Busemann子梯度","source_units":[{"start":34484,"end":34913,"start_line":463,"end_line":469,"raw_tex":"\\begin{definition}\n\t\t\t\\label{def:epihoro}\n\t\t\tConsider a nonempty subset $C$ of $X$ and a real-valued function $f\\from C \\to \\R$.\n\t\t\tA \\textit{Busemann subgradient} of $f$ at a point $x\\in C$ is an element $[\\xi,s] \\in CX^\\infty$\n\t\t\tsuch that $x$ minimizes $y\\mapsto f(y) - \\ip{y,[\\xi,s]}$ over $C$. The function $f$ is \\textit{Busemann subdifferentiable} if it has a Busemann subgradient at every point in $C$.\n\n\t\\end{definition}","sha256_utf8":"13727b4b5d065e8960269f749ea041e0f7772d41365aac4ff23e93ce98914008"}]},{"id":"S007","semantic_type":"derived_fact","kind":"derived_fact","title":"零子梯度与最优点","source_units":[{"start":34926,"end":35094,"start_line":472,"end_line":472,"raw_tex":"An immediate consequence of the definition is that a point $x \\in C$ minimizes a function $f\\from C\\to\\R$ if and only if $[0]$ is a Busemann subgradient of $f$ at $x$.\n","sha256_utf8":"badd08378483e2ec6482dc4ffa140ed9dd652cb6869dabea146896f97cf516f1"}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","title":"归一化子梯度不等式","source_units":[{"start":36352,"end":36531,"start_line":494,"end_line":498,"raw_tex":"\t\t\\begin{equation}\n\t\t\t\\label{eqn:busesub}\n\t\t\tf(y) \\geq f(x) + sb_{x,\\xi}(y) \\text{ for all } y\\in C,\n\t\t\\end{equation}\n\t\ta version of the subgradient inequality in Hadamard space.\n","sha256_utf8":"032cec872a7efaa2bb86f6aaf4b0d234e158d3249b07f952a05489882505c1db"}]},{"id":"S009","semantic_type":"proof","kind":"proof","title":"子梯度不等式等价推导","source_units":[{"start":35865,"end":36352,"start_line":485,"end_line":493,"raw_tex":"\t\tMore generally, $[\\xi,s]$ is a Busemann subgradient of $f\\from C\\to\\R$ at $x$ if and only if\n\t\t\\begin{equation}\n\t\t\t\\label{eqn:busesubpre}\n\t\t\tf(y) - sb_\\xi(y) \\geq f(x) - sb_\\xi(x) \\text{ for all } y\\in C.\n\t\t\\end{equation}\n\t\tWe adopt the same convention that $sb_\\xi$ is identically $0$ if $s = 0$.\n\t\tThe Busemann functions for asymptotic rays differ only by an additive constant \n\t\t(Proposition \\ref{additive}), so it follows that inequality\n\t\t(\\ref{eqn:busesubpre}) is equivalent to \n","sha256_utf8":"4064503f8922cd494e7b5b618526c49d061e54cc6ed483dd56bd99b05c7e3d86"}]},{"id":"S010","semantic_type":"proof_step","kind":"proof_step","title":"展开配对与零速约定","source_units":[{"start":35865,"end":36165,"start_line":485,"end_line":490,"raw_tex":"\t\tMore generally, $[\\xi,s]$ is a Busemann subgradient of $f\\from C\\to\\R$ at $x$ if and only if\n\t\t\\begin{equation}\n\t\t\t\\label{eqn:busesubpre}\n\t\t\tf(y) - sb_\\xi(y) \\geq f(x) - sb_\\xi(x) \\text{ for all } y\\in C.\n\t\t\\end{equation}\n\t\tWe adopt the same convention that $sb_\\xi$ is identically $0$ if $s = 0$.\n","sha256_utf8":"f097e1b84221082f20493f2e946f3c130912544a3dda7710ce39e87114876385"}]},{"id":"S011","semantic_type":"proof_step","kind":"proof_step","title":"利用基点加性常数","source_units":[{"start":36165,"end":36352,"start_line":491,"end_line":493,"raw_tex":"\t\tThe Busemann functions for asymptotic rays differ only by an additive constant \n\t\t(Proposition \\ref{additive}), so it follows that inequality\n\t\t(\\ref{eqn:busesubpre}) is equivalent to \n","sha256_utf8":"2b120b92c7b1efd2478d5d193e939b395da9bacabd388d1a9283504d4e8bbcfb"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"角度切空间与Alexandrov子梯度","source_units":[{"start":69065,"end":69353,"start_line":991,"end_line":991,"raw_tex":"For simplicity, we restrict attention to our usual setting of a Hadamard space $X$.  We briefly review some necessary tangent space constructions:  for details, see \\cite{bridson}.  For any point $x \\in X$, we denote by $\\Theta_x X$  the set of all nonconstant geodesics issuing at~$x$. \n","sha256_utf8":"40680256200a43cf8cdd1f7c15f0290cd8052e04a13227099805eb74a2afb812"},{"start":69355,"end":71587,"start_line":994,"end_line":1016,"raw_tex":"\t\tGiven three points $x, y, z \\in X$, a \\textit{geodesic triangle} $\\Delta = \\Delta(x, y, z)$\n\tis the union of three geodesic segments (its sides) joining each pair of points. A\n\t\\textit{comparison triangle} for $\\Delta$ is a triangle  $\\Delta(\\bar x, \\bar y,  \\bar z)$ in $\\R^2$\n\tthat has side lengths equal\n\tto those of $\\Delta$. \n\tTake two geodesics $\\gamma,\\eta \\in \\Theta_x$, let $\\gamma_t = \\gamma(t), \\eta_s = \\eta(s)$ for $t,s\\geq 0$ and \n\tlet $\\Delta(\\bar \\gamma_t,\\bar x,\\bar \\eta_s)$ be a comparison triangle in $\\R^2$ for $\\Delta(\\gamma_t,x,\\eta_s)$. Then the\n\tangle $\\angle \\bar \\gamma_t\\bar x\\bar \\eta_s$ is a nondecreasing function of both $t$ and $s$ and the\n\t\\textit{Alexandrov angle} between $\\gamma$ and $\\eta$ is well-defined as the following limit: \n\t\\[\\angle(\\gamma,\\eta) := \\lim_{t,s\\downarrow 0} \\angle \\bar \\gamma_t \\bar x\\bar \\eta_s.\\]\n\t\tThe Alexandrov angle induces a metric on the set $\\Sigma_x X$ of equivalence classes of geodesics in $\\Theta_x X$, where two geodesics $\\gamma, \\eta\\in \\Theta_x X$ are considered equivalent if $\\angle(\\gamma,\\eta) = 0$. \nFor a geodesic $\\gamma \\in \\Theta_x X$, we denote by $[\\gamma]$ its equivalence class. \n The {\\it tangent space} $T_x X$ of $X$ at $x$ is the Euclidean cone over the metric space $(\\Sigma_x X, \\angle)$ (see \\cite[Chapter I, Definition 5.6]{bridson}). \n For $v = ([\\gamma],r), w = ([\\eta],s) \\in T_x X$ the {\\it scalar product} of $v$ and $w$ is given by $\\langle \\langle v,w \\rangle \n \\rangle = rs\\cos \\angle(\\gamma, \\eta)$. For a geodesically convex function $f\\from X\\to \\R$, the \\textit{subdifferential} of $f$ at $x$, denoted by\n $\\p f(x)$, can be characterized (see \\cite[Proposition 4.4]{sumrule})\n as the elements $([\\eta],s)\\in T_xX$ satisfying the \\textit{subgradient inequality}\n \\begin{equation}\n\t \\label{eqn:cat0subgrad}\n \\ip{\\langle ([\\eta],s),([\\gamma_y],d(x,y))\\rangle} + f(x) \\leq f(y) \\text{ for all } y\\in X,\n \\end{equation}\n where $\\gamma_y\\from[0,d(x,y)]\\to X$ denotes the geodesic from $x$ to $y$. Any element $v\\in \\p f(x)$ is called a \\textit{subgradient}\n of $f$ at $x$.  A related idea used in the study of gradient flows in Alexandrov spaces is the \\textit{minus-subdifferential} \\cite[Definition 3.6]{GigliNobili2021}. \n","sha256_utf8":"a17f078939c4b6d820ffbfce5de9698629e3291339e130cfb5498c09ffb6d4b0"}]},{"id":"S013","semantic_type":"theorem","kind":"theorem","title":"两类子梯度的兼容性","source_units":[{"start":71736,"end":72177,"start_line":1020,"end_line":1027,"raw_tex":"\\begin{theorem}\n\t\t\t\\label{thm:subgrad}\nGiven a Hadamard space $X$ and $C \\seq X$, suppose that the function $f\\from C\\to \\R$ is geodesically convex, with a Busemann subgradient $[\\xi,s]$ at $x\\in C$. \n\t\t\tLet $r$ be a ray issuing from $x$ with direction $\\xi$.\n\t\t\tExtend $r$ to a geodesic line\n\t\t\t$\\tilde{r}\\from \\R\\to X$ and define a new ray $r_-\\from \\R_+\\to X$ by $r_-(t) = \\tilde{r}(-t)$.\n\t\t\tThen $([r_-],s) \\in \\p f(x)$.\t\n\t\t\\end{theorem}","sha256_utf8":"3ea2cee13090e1dbd4ff7178e7f0486e1971c52b59be76612e9097e2f2a3c584"}]},{"id":"S014","semantic_type":"proof","kind":"proof","title":"兼容性及平方距离极限的完整证明","source_units":[{"start":72180,"end":74310,"start_line":1028,"end_line":1071,"raw_tex":"\\begin{proof}\n\t\t\tBy definition of the scalar product on $T_xX$, the subgradient inequality we wish to prove is\n\t\t\t\\[s d(x,y)\\cos\\angle(r_-,\\gamma_y) + f(x)\\leq f(y) \\text{ for all } y\\in X.\\]\n\t\t\tBusemann subdifferentiability of $f$ implies, by (\\ref{eqn:busesub}), \n\t\t\t\\[s b_r(y) + f(x) \\leq f(y) \\text{ for all } y\\in X.\\]\n\t\t\tThus it is enough to prove (the case $s = 0$ being trivial):\n\t\t\t\\begin{equation}\n\t\t\t\t\\label{eqn:desired}\n\t\t\t\tb_r(y) \\geq d(x,y)\\cos\\angle(r_-,\\gamma_y) \\text{ for all } y\\in X.\n\t\t\t\\end{equation}\n\t\t\tWe proceed in two steps, first proving \n\t\t\t$\\cos\\angle(r_-,\\gamma_y) \\leq -\\cos\\angle(r,\\gamma_y)$ $(*)$.\n\t\t\tUsing the triangle inequality for angles we find:\n\t\t\t\\[\\pi = \\angle(r,r_-) \\leq \\angle(r_-,\\gamma_y) + \\angle(r,\\gamma_y).\\]\n\t\t\tHence $\\pi - \\angle(r,\\gamma_y)\\leq \\angle(r_-,\\gamma_y)$, and taking the cosine reverses the inequality: \n\\begin{equation} \\label{star}\n\\cos\\angle(r_-,\\gamma_y) \\leq \\cos(\\pi - \\angle(r,\\gamma_y)) = -\\cos\\angle(r,\\gamma_y).\n\\end{equation}\n\n\t\t\tWe next prove \n\\begin{equation} \\label{starstar}\nb_r(y) \\geq -d(x,y)\\cos\\angle(r,\\gamma_y).\n\\end{equation}\n\t\t\tBy the law of cosines we have:\n\t\t\t\\[d(r(t),y)^2 \\geq d(x,r(t))^2 + d(x,y)^2 - 2d(x,r(t))d(x,y)\\cos\\angle(r,\\gamma_y).\\]\n\t\t\tRearrangement leads to\n\t\t\t\\begin{equation}\n\t\t\t\t\\label{eqn:ineqprelim}\n\t\t\t\t\\frac{d(r(t),y)^2 - t^2}{2t} \\geq \\frac{d(x,y)^2}{2t} - d(x,y)\\cos\\angle(r,\\gamma_y).\n\t\t\\end{equation}\n\t\tThe lefthand side of \\eqref{eqn:ineqprelim} simplifies in the limit because $t\\mapsto d(r(t),y)-t$ is bounded:\n\t\t\t\\begin{align}\n\t\t\t\t\\label{eqn:quadraticbuse}\n\t\t\t\\lim_{t\\to \\infty}\\frac{d(r(t),y)^2 - t^2}{2t} &= \n\t\t\t\\lim_{t\\to \\infty}(d(r(t),y)-t)\\lim_{t\\to \\infty}\\frac{d(r(t),y) +t}{2t} \\nonumber\\\\\n\t\t\t&= b_r(y)\\lim_{t\\to \\infty}\\left(\\frac{d(r(t),y)-t}{2t} + 1 \\right)  \\nonumber \\\\\n\t\t\t&= b_r(y).\n\t\\end{align}\n\tUsing \\eqref{eqn:quadraticbuse} and sending $t\\to \\infty$ in (\\ref{eqn:ineqprelim}) yields \n\t(\\ref{starstar}).\n\t\t\tCombining (\\ref{star}) and (\\ref{starstar}) gives\n\t\t\t\\[d(x,y)\\cos\\angle(r_-,\\gamma_y) \\leq -d(x,y)\\cos\\angle(r,\\gamma_y) \\leq b_r(y).\\]\n\t\t\tThis proves (\\ref{eqn:desired}).\n\t\t\\end{proof}","sha256_utf8":"fbf344a351495188e4f0932ef62e72af54977b90439540a617b31f868e3228b9"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"归约为Busemann角度下界","source_units":[{"start":72180,"end":72702,"start_line":1028,"end_line":1037,"raw_tex":"\\begin{proof}\n\t\t\tBy definition of the scalar product on $T_xX$, the subgradient inequality we wish to prove is\n\t\t\t\\[s d(x,y)\\cos\\angle(r_-,\\gamma_y) + f(x)\\leq f(y) \\text{ for all } y\\in X.\\]\n\t\t\tBusemann subdifferentiability of $f$ implies, by (\\ref{eqn:busesub}), \n\t\t\t\\[s b_r(y) + f(x) \\leq f(y) \\text{ for all } y\\in X.\\]\n\t\t\tThus it is enough to prove (the case $s = 0$ being trivial):\n\t\t\t\\begin{equation}\n\t\t\t\t\\label{eqn:desired}\n\t\t\t\tb_r(y) \\geq d(x,y)\\cos\\angle(r_-,\\gamma_y) \\text{ for all } y\\in X.\n\t\t\t\\end{equation}\n","sha256_utf8":"25b42accfece0ff7b31043f28df25b5f0eca1de5419aa855835b3de49c7f2ff4"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"比较正反射线的夹角","source_units":[{"start":72702,"end":73182,"start_line":1038,"end_line":1045,"raw_tex":"\t\t\tWe proceed in two steps, first proving \n\t\t\t$\\cos\\angle(r_-,\\gamma_y) \\leq -\\cos\\angle(r,\\gamma_y)$ $(*)$.\n\t\t\tUsing the triangle inequality for angles we find:\n\t\t\t\\[\\pi = \\angle(r,r_-) \\leq \\angle(r_-,\\gamma_y) + \\angle(r,\\gamma_y).\\]\n\t\t\tHence $\\pi - \\angle(r,\\gamma_y)\\leq \\angle(r_-,\\gamma_y)$, and taking the cosine reverses the inequality: \n\\begin{equation} \\label{star}\n\\cos\\angle(r_-,\\gamma_y) \\leq \\cos(\\pi - \\angle(r,\\gamma_y)) = -\\cos\\angle(r,\\gamma_y).\n\\end{equation}\n","sha256_utf8":"43b4dee1144d8839f102a5c5d58b1bd515963d73c1726dd0e57f3027a51e1c4b"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"余弦比较得到距离下界","source_units":[{"start":73183,"end":73596,"start_line":1047,"end_line":1057,"raw_tex":"\t\t\tWe next prove \n\\begin{equation} \\label{starstar}\nb_r(y) \\geq -d(x,y)\\cos\\angle(r,\\gamma_y).\n\\end{equation}\n\t\t\tBy the law of cosines we have:\n\t\t\t\\[d(r(t),y)^2 \\geq d(x,r(t))^2 + d(x,y)^2 - 2d(x,r(t))d(x,y)\\cos\\angle(r,\\gamma_y).\\]\n\t\t\tRearrangement leads to\n\t\t\t\\begin{equation}\n\t\t\t\t\\label{eqn:ineqprelim}\n\t\t\t\t\\frac{d(r(t),y)^2 - t^2}{2t} \\geq \\frac{d(x,y)^2}{2t} - d(x,y)\\cos\\angle(r,\\gamma_y).\n\t\t\\end{equation}\n","sha256_utf8":"637d49243f49cbad001cd7153f8c0481b842890883a3cc85772e4181f3629a96"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"平方距离差的Busemann极限","source_units":[{"start":73596,"end":74010,"start_line":1058,"end_line":1065,"raw_tex":"\t\tThe lefthand side of \\eqref{eqn:ineqprelim} simplifies in the limit because $t\\mapsto d(r(t),y)-t$ is bounded:\n\t\t\t\\begin{align}\n\t\t\t\t\\label{eqn:quadraticbuse}\n\t\t\t\\lim_{t\\to \\infty}\\frac{d(r(t),y)^2 - t^2}{2t} &= \n\t\t\t\\lim_{t\\to \\infty}(d(r(t),y)-t)\\lim_{t\\to \\infty}\\frac{d(r(t),y) +t}{2t} \\nonumber\\\\\n\t\t\t&= b_r(y)\\lim_{t\\to \\infty}\\left(\\frac{d(r(t),y)-t}{2t} + 1 \\right)  \\nonumber \\\\\n\t\t\t&= b_r(y).\n\t\\end{align}\n","sha256_utf8":"2b97abd0bdf4f328519db0e25b804a8ea7f13b013a5b17a1b49c50106695377b"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"取极限并完成兼容性证明","source_units":[{"start":74010,"end":74310,"start_line":1066,"end_line":1071,"raw_tex":"\tUsing \\eqref{eqn:quadraticbuse} and sending $t\\to \\infty$ in (\\ref{eqn:ineqprelim}) yields \n\t(\\ref{starstar}).\n\t\t\tCombining (\\ref{star}) and (\\ref{starstar}) gives\n\t\t\t\\[d(x,y)\\cos\\angle(r_-,\\gamma_y) \\leq -d(x,y)\\cos\\angle(r,\\gamma_y) \\leq b_r(y).\\]\n\t\t\tThis proves (\\ref{eqn:desired}).\n\t\t\\end{proof}","sha256_utf8":"f29a534205488604e439610d703c769f9c3f94a7329b3c91c123b125f61a1c85"}]},{"id":"S020","semantic_type":"notation","kind":"notation","title":"投影子梯度预言机与射线记号","source_units":[{"start":75870,"end":76410,"start_line":1094,"end_line":1103,"raw_tex":"We denote the projection\n\t\t\tonto a nonempty closed geodesically convex set $C$ by $P_C$ \n\t\t\t(well-defined by \\cite[Proposition II.2.4]{bridson}).\t\n\t\t\tDefine an oracle \\texttt{Busemann}\n\t\t\tthat accepts a Busemann subdifferentiable function $g\\from C\\to \\R$ and a point $x\\in C$, and \n\t\t\treturns a Busemann subgradient\n\t\t\tfor $g$ at $x$:\n\t\t\t$[\\xi,s] \\leftarrow \\texttt{Busemann}(g,x)$. If $[\\xi,s]$ is a Busemann subgradient for $g$\n\t\t\tat $x$, we will use the notation $r_{x,\\xi}$ to denote the ray \n\t\t\tissuing from $x$ with direction $\\xi$.\n","sha256_utf8":"610185fbabaf5c5a1b826ffc0e3f6b6276f1f153b4bb10e2444bc7b254c2ee6d"}]},{"id":"S021","semantic_type":"assumption","kind":"assumption","title":"结构优化的假设A","source_units":[{"start":76410,"end":77180,"start_line":1104,"end_line":1120,"raw_tex":"\t\t\t\\begin{customthm}{A}\n\t\t\t\t\\label{assumptionA}~\\\\\n\t\t\t\t\\vspace{-0.5cm}\n\t\t\t\\begin{enumerate}[label=(\\roman*)]\n\t\t\t\\item $(X,d)$ is a Hadamard space with the geodesic extension property.\n\t\t\t\\item $C\\seq X$ is nonempty, closed, and geodesically convex.\n\n\t\t\t\\item The function $f$ decomposes as $f = \\sum_{i=1}^m f_i$, where each \n\t\t\t\t$f_i, i = 1,\\dots,m$ is Busemann subdifferentiable on an open set \n\t\t\tcontaining $C$.\n\t\t\t\t\t\t\\item The optimal set of $\\inf_{x\\in C} f(x)$ is nonempty, and denoted by $X^*$. The optimal value of the\n\t\t\t\tproblem is denoted by $f_{\\text{opt}}$.\n\t\t\t\\item There is a constant $L \\geq 0$ such that\n\t\t\t\tfor all $i=1,\\dots,m$, every Busemann subgradient $[\\xi,s]$ for $f_i$ at every point in $C$\n\t\t\t\thas $s\\leq L$.\n\\end{enumerate}\n\t\\end{customthm}\n","sha256_utf8":"0c3cec5cb52bb72d5cbb493f2d679fa2d5f5cea6027a3aed90cd8690d02a155d"}]},{"id":"S022","semantic_type":"algorithm","kind":"algorithm","title":"随机Busemann子梯度法","source_units":[{"start":77232,"end":77844,"start_line":1124,"end_line":1139,"raw_tex":"\\begin{algorithm}[h!]\n\t\t\\caption{Stochastic Busemann Subgradient Method}\n\t\t\t\\begin{algorithmic}\n\t\t\t\t\\Require $x^0 \\in C, \\st{t_k}_{k=0}^\\infty \\subset (0,+\\infty), \\st{i(k)}_{k=0}^\\infty$ independent Uniform$\\st{1,\n\t\t\t\t\\dots,m}$ \n\t\t\t\t\\For{$k=0,1,2\\dots$}\n\t\t\t\t%\\State $x^{k,0} \\leftarrow x^k$\n\t\t\t\t%\\For{$i=0,1,\\dots,m-1$}\n\t\t\t\t\\State $[\\xi_{k},s_{k}] \\leftarrow \\texttt{Busemann}(f_{i(k)}, x^{k})$\n\t\t\t\t\\State $x^{k+1} \\leftarrow P_C\\left(r_{x_{k},\\xi_{k}}\\left(s_{k}t_k\\right)\\right)$\n\t\t\t\t%\\EndFor\n\t\t\t\t%\\State $x^{k+1} \\leftarrow x^{k,m}$\n\t\t\t\t\\EndFor \t\t\n\t\t\t\t\\end{algorithmic}\n\\label{alg:stochsub}\n\t\t\\end{algorithm}","sha256_utf8":"c0cfd00b7d0a365f4e2f507b634191c3dd304fed76aa1957d21d0addd3215174"}]},{"id":"S023","semantic_type":"lemma","kind":"lemma","title":"投影子梯度单步不等式","source_units":[{"start":77997,"end":78609,"start_line":1144,"end_line":1157,"raw_tex":"\\begin{lemma}\\textbf{(Projected Busemann subgradient inequality)}\n\t\t\t\\label{lem:projsub}\n\t\t\tSuppose the function $f\\from X\\to \\R$ is Busemann subdifferentiable on a nonempty, closed, and \n\t\t\tgeodesically convex set $C$. Let $x\\in C$, $t > 0$ and choose a\n\t\t\tBusemann subgradient $[\\xi,s]$ for $f$ at $x$.\n\t\t\tDefine the new point\n\t\t\t\\[x^+ = \\begin{cases}\n\t\t\t\t\tP_C\\left(r_{x,\\xi}\\left( s t\\right)\\right), & s > 0\\\\\n\t\t\t\t\tx, & s = 0,\n\t\t\\end{cases}\\]\n\t\twhere $r_{x,\\xi}$ is the ray issuing from $x$ with direction $\\xi$.\n\tThen for any $y \\in C$,\n\t\\[d(x^{+},y)^2\\leq d(x,y)^2 - 2t (f(x)- f(y)) +s^2t^2.\\]\n\t\t\\end{lemma}","sha256_utf8":"6031f1e48305510e0ec28b8fdd3290f0b4fb8bf901c5d4a754e3ab98b111397d"}]},{"id":"S024","semantic_type":"proof","kind":"proof","title":"投影单步不等式证明","source_units":[{"start":78612,"end":80094,"start_line":1158,"end_line":1188,"raw_tex":"\\begin{proof}\n\t\t\tFor brevity we denote $r = r_{x,\\xi}$. If $s = 0$ then the desired inequality reduces\n\t\t\tto $f(x)\\leq f(y)$ for all $y\\in C$, which holds because $x$ minimizes $f$ over $C$ \n\t\t\t(cf.\\ the discussion\n\t\t\t\tfollowing Definition \\ref{def:epihoro}). Thus we may assume\n\t\t\t$s > 0$.\n\t\t\tSince $X$ is Hadamard, $x\\mapsto \\frac{1}{2}d(x,y)^2$ is 1-strongly convex for any $y\\in C$.\n\t\t\tIt follows that for all\n\t\t\t$\\delta \\geq s t$ we have\n\t\t\t\\begin{align*}\n\t\t\t\td(x^{+},y)^2 &= d\\left(P_C(r\\hspace{-0.05cm}\\left(st\\right)),P_C(y)\\right)^2\\\\\n\t\t\t\t&\\leq d\\left(r\\hspace{-0.05cm}\\left(st\\right),y\\right)^2\\\\\n\t\t\t\t&\\leq \\left(1-\\frac{s t}{\\delta}\\right)d(x,y)^2 + \\frac{s t}{\\delta}\n\t\t\t\td(r(\\delta),y)^2 - \\left(1 - \\frac{s t}{\\delta}\\right)\\frac{s t}{\\delta}\n\t\t\t\td(x,r(\\delta))^2\\\\\n\t\t\t\t&= \\left(1-\\frac{s t}{\\delta}\\right)d(x,y)^2 + \\frac{s t}{\\delta}\n\t\t\t\td(r(\\delta),y)^2 + s^2t^2 - \\delta s t\\\\\n\t\t\t\t&= \\left(1-\\frac{s t}{\\delta}\\right)d(x,y)^2 + \\frac{s t}{\\delta}(\n\t\t\td(r(\\delta),y)^2 - \\delta^2) + \n\t\t\ts^2t^2.\n\t\t\t\\end{align*}\n\t\t\tThe second line uses nonexpansivity of the projection $P_C$, see \\cite[Theorem 2.1.12]{bacakbook}.\n\t\t\tLetting $\\delta\\to \\infty$ and using once more \\eqref{eqn:quadraticbuse} we deduce\n\t\t\t\t\t\t\\[d(x^{+},y)^2 \\leq d(x,y)^2 +  2 s t b_r(y) + s^2t^2.\\]\n\t\t\tBy (\\ref{eqn:busesub}) we have\n\t\t\t$s b_r(y) \n\t\t\t\\leq -(f(x) - f(y))$.\n\t\t\tWe conclude:\n\t\t\t\\[d(x^{+},y)^2 \\leq d(x,y)^2 +  2s t b_r(y) + s^2t^2 \\leq\n\t\t\td(x,y)^2 - 2 t(f(x)-f(y)) +s^2t^2.\\]\n\t\t\\end{proof}","sha256_utf8":"f3e4316357c87faea016498d0caa177e35cf4b55ac60326e8c539b64ba941327"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"处理零速子梯度","source_units":[{"start":78612,"end":78903,"start_line":1158,"end_line":1163,"raw_tex":"\\begin{proof}\n\t\t\tFor brevity we denote $r = r_{x,\\xi}$. If $s = 0$ then the desired inequality reduces\n\t\t\tto $f(x)\\leq f(y)$ for all $y\\in C$, which holds because $x$ minimizes $f$ over $C$ \n\t\t\t(cf.\\ the discussion\n\t\t\t\tfollowing Definition \\ref{def:epihoro}). Thus we may assume\n\t\t\t$s > 0$.\n","sha256_utf8":"6be21fe0fafacb1b972b9b6e87fa8f25fb8fccf5a443aaf677817dc420da031f"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"强凸性与投影非扩张估计","source_units":[{"start":78903,"end":79742,"start_line":1164,"end_line":1179,"raw_tex":"\t\t\tSince $X$ is Hadamard, $x\\mapsto \\frac{1}{2}d(x,y)^2$ is 1-strongly convex for any $y\\in C$.\n\t\t\tIt follows that for all\n\t\t\t$\\delta \\geq s t$ we have\n\t\t\t\\begin{align*}\n\t\t\t\td(x^{+},y)^2 &= d\\left(P_C(r\\hspace{-0.05cm}\\left(st\\right)),P_C(y)\\right)^2\\\\\n\t\t\t\t&\\leq d\\left(r\\hspace{-0.05cm}\\left(st\\right),y\\right)^2\\\\\n\t\t\t\t&\\leq \\left(1-\\frac{s t}{\\delta}\\right)d(x,y)^2 + \\frac{s t}{\\delta}\n\t\t\t\td(r(\\delta),y)^2 - \\left(1 - \\frac{s t}{\\delta}\\right)\\frac{s t}{\\delta}\n\t\t\t\td(x,r(\\delta))^2\\\\\n\t\t\t\t&= \\left(1-\\frac{s t}{\\delta}\\right)d(x,y)^2 + \\frac{s t}{\\delta}\n\t\t\t\td(r(\\delta),y)^2 + s^2t^2 - \\delta s t\\\\\n\t\t\t\t&= \\left(1-\\frac{s t}{\\delta}\\right)d(x,y)^2 + \\frac{s t}{\\delta}(\n\t\t\td(r(\\delta),y)^2 - \\delta^2) + \n\t\t\ts^2t^2.\n\t\t\t\\end{align*}\n\t\t\tThe second line uses nonexpansivity of the projection $P_C$, see \\cite[Theorem 2.1.12]{bacakbook}.\n","sha256_utf8":"c523d03227c27de2e98ba59e16a845f21dae664f920fb4482bc7211732a73fe6"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"沿射线取无穷远极限","source_units":[{"start":79742,"end":79891,"start_line":1180,"end_line":1181,"raw_tex":"\t\t\tLetting $\\delta\\to \\infty$ and using once more \\eqref{eqn:quadraticbuse} we deduce\n\t\t\t\t\t\t\\[d(x^{+},y)^2 \\leq d(x,y)^2 +  2 s t b_r(y) + s^2t^2.\\]\n","sha256_utf8":"db66ccd3d19b17edb88b4f740c2cfabff6fb2aa9382583f88adf169da3c66afe"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"用子梯度不等式完成单步界","source_units":[{"start":79891,"end":80094,"start_line":1182,"end_line":1188,"raw_tex":"\t\t\tBy (\\ref{eqn:busesub}) we have\n\t\t\t$s b_r(y) \n\t\t\t\\leq -(f(x) - f(y))$.\n\t\t\tWe conclude:\n\t\t\t\\[d(x^{+},y)^2 \\leq d(x,y)^2 +  2s t b_r(y) + s^2t^2 \\leq\n\t\t\td(x,y)^2 - 2 t(f(x)-f(y)) +s^2t^2.\\]\n\t\t\\end{proof}","sha256_utf8":"679462d93fad8f3a1f5215e3d52c4b8825d009bb7b46d729dc0d8bce474530ce"}]},{"id":"S029","semantic_type":"notation","kind":"notation","title":"历史最优目标差记号","source_units":[{"start":80256,"end":80375,"start_line":1192,"end_line":1192,"raw_tex":"\t\t\t$f^k_{\\text{best}} := \\min_{i=1,\\dots,k} f(x^i) - f_{\\text{opt}}$.  For stochastic terminology, see \\cite{resnick}.\n","sha256_utf8":"352220d52bb7bd7179b6c9392582e4b5d0bfc80ab85f40acc97213d749d8de6a"}]},{"id":"S030","semantic_type":"theorem","kind":"theorem","title":"随机子梯度法的复杂度与收敛","source_units":[{"start":80378,"end":81305,"start_line":1193,"end_line":1204,"raw_tex":"\\begin{theorem}\\textbf{(Complexity of stochastic Busemann subgradient method)}\n\t\t\t\\label{thm:stochcomplexity}\n\t\t\tSuppose Assumption \\ref{assumptionA} holds, and let $\\st{t_k}_{k=0}^\\infty$ be a sequence of positive \n\t\t\tstepsizes and $\\st{i(k)}_{k=0}^\\infty$ a sequence of independent random variables distributed\n\t\t\tuniformly over $\\st{1,\\dots,m}$.\n\t\t\tLet $\\st{x^k}_{k=0}^\\infty$ be the sequence of iterates \n\t\t\tgenerated by Algorithm \\ref{alg:stochsub} with the above parameters.\n\t\t\tSuppose furthermore that the diameter of $C$ is bounded above by $D > 0$.\n\t\t\tIf $t_k = \\frac{D}{Lm\\sqrt{k+1}}$ then for all $k\\geq 2$,\n\t\t\t\\[\\mathbb{E}\\left[\\min_{i=1,\\dots,k} f(x^i) - f_{\\text{opt}}\\right] \\leq \\frac{2(1+\\log(3))mLD}{\\sqrt{k+2}}.\\]\n\t\t\tMore generally, if $\\sum_{k=0}^nt_k^2/\\sum_{k=0}^nt_k \\to 0$ as $n\\to \\infty$ then $\\mathbb{E}[f^k_{\\text{best}}] \\to f_{\\text{opt}}$ as $k\\to \\infty$ even if $C$ is unbounded.\n\t\\end{theorem}","sha256_utf8":"3db08a9f967417a1f44e0f72cbd3852ae2cf993cc010bbb7520011f10811b4e8"}]},{"id":"S031","semantic_type":"proof","kind":"proof","title":"随机复杂度定理的完整证明","source_units":[{"start":81306,"end":83548,"start_line":1205,"end_line":1235,"raw_tex":"\\begin{proof}\nDenoting the nonnegative integers by $\\N_0$, the underlying probability space is $\\Omega = \\st{1,\\dots,m}^{\\N_0}$ equipped with the product of the \nuniform $\\st{1,\\dots,m}$ probability measures.\nAt step $k \\in \\N_0$, choose a Busemann subgradient $[\\xi_{k},s_{k}]$ for $f_{i(k)}$ at $x^k$. \nThen define $x^{k+1} := P_C(r_{x^k, \\xi_{k}}(s_{k}t_k))$, as in Algorithm \\ref{alg:stochsub}.\n%We will show that \n%$\\E\\left[ \\min_{k=0,\\dots,n} f(x^k)\\right] - f_{\\text{opt}} = O(n^{-1/2})$.\nConsider a given (random) iterate $x^k$. \nLemma \\ref{lem:projsub} and Assumption \\ref{assumptionA}(v) together imply\n\\[d(x^{k+1},x^*)^2 \\leq d(x^k,x^*)^2 -2t_k(f_{i(k)}(x^k) - f_{i(k)}(x^*)) + L^2t_k^2.\\]\nLet $\\fami_k = \\sigma(x^0,\\dots,x^k)$ be the $\\sigma$-algebra generated by the variables $x^0,\\dots,x^k$.\nTaking the conditional expectation with respect to $\\fami_k$ gives\n\\begin{align*}\n\t\\E[d(x^{k+1},x^*)^2 \\mid \\fami_k] &\\leq d(x^k,x^*)^2 -2t_k\\E[f_{i(k)}(x^k)-f_{i(k)}(x^*) \\mid \\fami_k] + L^2t_k^2\\\\\n&= d(x^k,x^*)^2 -\\frac{2t_k}{m}\\sum_{i=1}^m(f_{i}(x^k)-f_{i}(x^*)) + L^2t_k^2\\\\\n&= d(x^k,x^*)^2 -\\frac{2t_k}{m}(f(x^k)-f_{\\text{opt}}) + L^2t_k^2.\n\\end{align*}\nNow taking the expectation, we arrive at\n\\[\\E[d(x^{k+1},x^*)^2] \\leq \\E[d(x^k,x^*)^2]-\\frac{2t_k}{m} \\E[f(x^k) - f_{\\text{opt}}] + L^2t_k^2.\\]\nSumming from $k= j$ to $n\\geq j$ and rearranging:\n\\[\\frac{2}{m}\\sum_{k=j}^n t_k \\E[f(x^k)-f_{\\text{opt}}] \\leq \\E[d(x^j,x^*)^2] + L^2\\sum_{k=j}^nt_k^2.\\]\nBy the elementary inequality $\\E[\\min_{k=j,\\dots,n}f(x^k)] \\leq \\min_{k=j,\\dots,n}\\E[f(x^k)]$, we derive\n\\[\\E\\left[\\min_{k=0,\\dots,n} f(x^k) - f_{\\text{opt}}\\right] \n\\leq \\E\\left[\\min_{k=j,\\dots,n}f(x^k) - f_{\\text{opt}}\\right] \\leq \\frac{m (\\E[d(x^j,x^*)^2] + L^2\\sum_{k=j}^nt_k^2)}{2\\sum_{k=j}^n t_k}.\\]\nFinally, since the diameter of $C$ is at most $D$ we have $\\E[d(x^j,x^*)^2] \\leq D^2$. Choosing $t_k = \\frac{D}{L\\sqrt{k+1}}$, $j = \\ceil{n/2}$,\nand appealing to \n\\cite[Lemma 8.27(b)]{beckfirstorder} gives \n\\[\\E\\left[\\min_{k=0,\\dots,n} f(x^k) - f_{\\text{opt}}\\right]  \\leq \\frac{2(1+\\log(3))mDL}{\\sqrt{n+2}} ~ ~ ~ \\text{ for all } n \\geq 2.\\]\nThe last statement of the theorem can be proven in exactly the same way\n\t\t\t as \\cite[Theorem 8.35(a)]{beckfirstorder}.\n\\end{proof}","sha256_utf8":"a1a86e35dcccdd3ee4933609d41f66162ffa761075e8befcea03f61b64bd11ec"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"建立概率空间与随机更新","source_units":[{"start":81306,"end":81705,"start_line":1205,"end_line":1209,"raw_tex":"\\begin{proof}\nDenoting the nonnegative integers by $\\N_0$, the underlying probability space is $\\Omega = \\st{1,\\dots,m}^{\\N_0}$ equipped with the product of the \nuniform $\\st{1,\\dots,m}$ probability measures.\nAt step $k \\in \\N_0$, choose a Busemann subgradient $[\\xi_{k},s_{k}]$ for $f_{i(k)}$ at $x^k$. \nThen define $x^{k+1} := P_C(r_{x^k, \\xi_{k}}(s_{k}t_k))$, as in Algorithm \\ref{alg:stochsub}.\n","sha256_utf8":"bfd82253f884ce784c94bc8c1c3d853a44cd1122a7c1b609bfbb783eefbe7ee1"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"应用单步界与速度上界","source_units":[{"start":81802,"end":82007,"start_line":1212,"end_line":1214,"raw_tex":"Consider a given (random) iterate $x^k$. \nLemma \\ref{lem:projsub} and Assumption \\ref{assumptionA}(v) together imply\n\\[d(x^{k+1},x^*)^2 \\leq d(x^k,x^*)^2 -2t_k(f_{i(k)}(x^k) - f_{i(k)}(x^*)) + L^2t_k^2.\\]\n","sha256_utf8":"1dd8f2df87042c0c9b4bc3c64b7bd15f0b2ade159833367a85b3f2d7c4ffbd8c"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"条件期望恢复完整目标差","source_units":[{"start":82007,"end":82472,"start_line":1215,"end_line":1221,"raw_tex":"Let $\\fami_k = \\sigma(x^0,\\dots,x^k)$ be the $\\sigma$-algebra generated by the variables $x^0,\\dots,x^k$.\nTaking the conditional expectation with respect to $\\fami_k$ gives\n\\begin{align*}\n\t\\E[d(x^{k+1},x^*)^2 \\mid \\fami_k] &\\leq d(x^k,x^*)^2 -2t_k\\E[f_{i(k)}(x^k)-f_{i(k)}(x^*) \\mid \\fami_k] + L^2t_k^2\\\\\n&= d(x^k,x^*)^2 -\\frac{2t_k}{m}\\sum_{i=1}^m(f_{i}(x^k)-f_{i}(x^*)) + L^2t_k^2\\\\\n&= d(x^k,x^*)^2 -\\frac{2t_k}{m}(f(x^k)-f_{\\text{opt}}) + L^2t_k^2.\n\\end{align*}\n","sha256_utf8":"0a2be137eb44faa503f252763ffb84297172dc0c71db4f22f9fd36ca738506bb"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"全期望与区间望远镜求和","source_units":[{"start":82472,"end":82769,"start_line":1222,"end_line":1225,"raw_tex":"Now taking the expectation, we arrive at\n\\[\\E[d(x^{k+1},x^*)^2] \\leq \\E[d(x^k,x^*)^2]-\\frac{2t_k}{m} \\E[f(x^k) - f_{\\text{opt}}] + L^2t_k^2.\\]\nSumming from $k= j$ to $n\\geq j$ and rearranging:\n\\[\\frac{2}{m}\\sum_{k=j}^n t_k \\E[f(x^k)-f_{\\text{opt}}] \\leq \\E[d(x^j,x^*)^2] + L^2\\sum_{k=j}^nt_k^2.\\]\n","sha256_utf8":"88f2b24a4c708a461da3ea371f0c1b3a68334f4ab164bc5a3a43e45b00b0b476"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"控制期望历史最优目标差","source_units":[{"start":82769,"end":83075,"start_line":1226,"end_line":1228,"raw_tex":"By the elementary inequality $\\E[\\min_{k=j,\\dots,n}f(x^k)] \\leq \\min_{k=j,\\dots,n}\\E[f(x^k)]$, we derive\n\\[\\E\\left[\\min_{k=0,\\dots,n} f(x^k) - f_{\\text{opt}}\\right] \n\\leq \\E\\left[\\min_{k=j,\\dots,n}f(x^k) - f_{\\text{opt}}\\right] \\leq \\frac{m (\\E[d(x^j,x^*)^2] + L^2\\sum_{k=j}^nt_k^2)}{2\\sum_{k=j}^n t_k}.\\]\n","sha256_utf8":"5edb363c5a62d7aee085b458ad4728cb132815c3326ce71d250ecd6cffaca3b2"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"代入直径步长与求和估计","source_units":[{"start":83075,"end":83418,"start_line":1229,"end_line":1232,"raw_tex":"Finally, since the diameter of $C$ is at most $D$ we have $\\E[d(x^j,x^*)^2] \\leq D^2$. Choosing $t_k = \\frac{D}{L\\sqrt{k+1}}$, $j = \\ceil{n/2}$,\nand appealing to \n\\cite[Lemma 8.27(b)]{beckfirstorder} gives \n\\[\\E\\left[\\min_{k=0,\\dots,n} f(x^k) - f_{\\text{opt}}\\right]  \\leq \\frac{2(1+\\log(3))mDL}{\\sqrt{n+2}} ~ ~ ~ \\text{ for all } n \\geq 2.\\]\n","sha256_utf8":"58b0c7c23d082cb983311ab7766359b27b7f37a67d7d7510e6054549be44bb65"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"转引一般步长的收敛论证","source_units":[{"start":83418,"end":83548,"start_line":1233,"end_line":1235,"raw_tex":"The last statement of the theorem can be proven in exactly the same way\n\t\t\t as \\cite[Theorem 8.35(a)]{beckfirstorder}.\n\\end{proof}","sha256_utf8":"a542ac83c46631d3f53b262d4df9b40570d3f99dc01e5a0fdbde63e28832d042"}]},{"id":"S039","semantic_type":"external_fact","kind":"external_fact","title":"方向射线唯一性与基点变换","source_units":[],"declared_external":true,"declaration_text":["For each x in a Hadamard space and each boundary direction xi, there is a unique unit-speed ray r_{x,xi} issuing from x with direction xi. Busemann functions of asymptotic rays differ by an additive constant; in particular b_{x,xi}(y)=b_xi(y)-b_xi(x). These are the standard facts cited from Bridson–Haefliger II.8.2 and Corollary II.8.20, the latter restated as B0009."]},{"id":"S040","semantic_type":"external_fact","kind":"external_fact","title":"CAT零角度与余弦比较","source_units":[],"declared_external":true,"declaration_text":["Alexandrov angles in a CAT(0) space satisfy the triangle inequality. Opposite unit-speed rays along a geodesic line have angle pi. For a unit-speed ray r issuing from x and the geodesic gamma_y from x to y, d(r(t),y)^2 >= t^2+d(x,y)^2-2t d(x,y) cos(angle(r,gamma_y))."]},{"id":"S041","semantic_type":"external_fact","kind":"external_fact","title":"闭凸集投影的非扩张性","source_units":[],"declared_external":true,"declaration_text":["For a nonempty closed geodesically convex subset C of a Hadamard space, the metric projection P_C exists uniquely, satisfies P_C(y)=y for y in C, and d(P_C(u),P_C(v)) <= d(u,v). The selected text cites Bridson–Haefliger Proposition II.2.4 and Bačák Theorem 2.1.12."]},{"id":"S042","semantic_type":"external_fact","kind":"external_fact","title":"调和与平方根求和界","source_units":[],"declared_external":true,"declaration_text":["The numerical estimate cited as Beck Lemma 8.27(b) gives, for n >= 2 and j=ceil(n/2), (1+sum_{k=j}^n 1/(k+1))/(sum_{k=j}^n 1/sqrt(k+1)) <= 4(1+log(3))/sqrt(n+2)."]},{"id":"S043","semantic_type":"external_fact","kind":"external_fact","title":"一般步长的加权递推收敛","source_units":[],"declared_external":true,"declaration_text":["The standard argument cited from Beck Theorem 8.35(a) applies to a recurrence a_{k+1} <= a_k-c t_k E[u_k]+C t_k^2, with a_k >= 0, u_k >= 0, t_k > 0, c > 0, C >= 0 and finite a_0. If sum_{k=0}^n t_k^2/sum_{k=0}^n t_k tends to zero, the expectation of the running minimum of u_k tends to zero."]},{"id":"S044","semantic_type":"assumption","kind":"assumption","title":"可行集直径上界","source_units":[{"start":80859,"end":80936,"start_line":1200,"end_line":1200,"raw_tex":"\t\t\tSuppose furthermore that the diameter of $C$ is bounded above by $D > 0$.\n","sha256_utf8":"e143a048b4525e677bbc425425e913ae8558956c675f4785a17f4a3e2905a7b5"}]},{"id":"S045","semantic_type":"assumption","kind":"assumption","title":"兼容性证明的局部设定","source_units":[{"start":71775,"end":72128,"start_line":1022,"end_line":1025,"raw_tex":"Given a Hadamard space $X$ and $C \\seq X$, suppose that the function $f\\from C\\to \\R$ is geodesically convex, with a Busemann subgradient $[\\xi,s]$ at $x\\in C$. \n\t\t\tLet $r$ be a ray issuing from $x$ with direction $\\xi$.\n\t\t\tExtend $r$ to a geodesic line\n\t\t\t$\\tilde{r}\\from \\R\\to X$ and define a new ray $r_-\\from \\R_+\\to X$ by $r_-(t) = \\tilde{r}(-t)$.\n","sha256_utf8":"34b59b428364484da28127a661f3c751f4be0b224bc884095df2c6a923cabce3"}]},{"id":"S046","semantic_type":"assumption","kind":"assumption","title":"投影步的局部设定","source_units":[{"start":78086,"end":78513,"start_line":1146,"end_line":1154,"raw_tex":"\t\t\tSuppose the function $f\\from X\\to \\R$ is Busemann subdifferentiable on a nonempty, closed, and \n\t\t\tgeodesically convex set $C$. Let $x\\in C$, $t > 0$ and choose a\n\t\t\tBusemann subgradient $[\\xi,s]$ for $f$ at $x$.\n\t\t\tDefine the new point\n\t\t\t\\[x^+ = \\begin{cases}\n\t\t\t\t\tP_C\\left(r_{x,\\xi}\\left( s t\\right)\\right), & s > 0\\\\\n\t\t\t\t\tx, & s = 0,\n\t\t\\end{cases}\\]\n\t\twhere $r_{x,\\xi}$ is the ray issuing from $x$ with direction $\\xi$.\n","sha256_utf8":"d17679026d2c625026e8b2372fed8f1dd42f5f117bb7f34e3ac2095e4d09b842"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S030","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S001","target_id":"S005","kind":"supports"},{"source_id":"S002","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S010","kind":"supports"},{"source_id":"S005","target_id":"S010","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S039","target_id":"S011","kind":"supports"},{"source_id":"S010","target_id":"S009","kind":"supports"},{"source_id":"S011","target_id":"S009","kind":"supports"},{"source_id":"S009","target_id":"S008","kind":"supports"},{"source_id":"S001","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S015","kind":"supports"},{"source_id":"S008","target_id":"S015","kind":"supports"},{"source_id":"S045","target_id":"S015","kind":"supports"},{"source_id":"S045","target_id":"S016","kind":"supports"},{"source_id":"S040","target_id":"S016","kind":"supports"},{"source_id":"S012","target_id":"S016","kind":"supports"},{"source_id":"S040","target_id":"S017","kind":"supports"},{"source_id":"S045","target_id":"S017","kind":"supports"},{"source_id":"S001","target_id":"S018","kind":"supports"},{"source_id":"S045","target_id":"S018","kind":"supports"},{"source_id":"S015","target_id":"S019","kind":"supports"},{"source_id":"S016","target_id":"S019","kind":"supports"},{"source_id":"S017","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S016","target_id":"S014","kind":"supports"},{"source_id":"S017","target_id":"S014","kind":"supports"},{"source_id":"S018","target_id":"S014","kind":"supports"},{"source_id":"S019","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S041","target_id":"S020","kind":"supports"},{"source_id":"S006","target_id":"S020","kind":"supports"},{"source_id":"S039","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S007","target_id":"S025","kind":"supports"},{"source_id":"S046","target_id":"S025","kind":"supports"},{"source_id":"S001","target_id":"S026","kind":"supports"},{"source_id":"S041","target_id":"S026","kind":"supports"},{"source_id":"S046","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S018","target_id":"S027","kind":"supports"},{"source_id":"S014","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S008","target_id":"S028","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S027","target_id":"S024","kind":"supports"},{"source_id":"S028","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S032","kind":"supports"},{"source_id":"S020","target_id":"S032","kind":"supports"},{"source_id":"S021","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S023","target_id":"S033","kind":"supports"},{"source_id":"S021","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S021","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S044","target_id":"S037","kind":"supports"},{"source_id":"S042","target_id":"S037","kind":"supports"},{"source_id":"S029","target_id":"S037","kind":"supports"},{"source_id":"S035","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S043","target_id":"S038","kind":"supports"},{"source_id":"S029","target_id":"S038","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S034","target_id":"S031","kind":"supports"},{"source_id":"S035","target_id":"S031","kind":"supports"},{"source_id":"S036","target_id":"S031","kind":"supports"},{"source_id":"S037","target_id":"S031","kind":"supports"},{"source_id":"S038","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"}]}}
{"record_id":"2502.11125v3","paper":{"title":"Convergence Analysis of Stochastic Accelerated Gradient Methods for Generalized Smooth Optimizations","authors":["Chenhao Yu","Yusu Hong","Junhong Lin"],"arxiv_id":"2502.11125","version":"v3","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-026-03102-2","abstract_url":"https://arxiv.org/abs/2502.11125v3","pdf_url":"https://arxiv.org/pdf/2502.11125v3"},"context":{"raw_tex":"\\newcommand{\\lsmooth}[1]{\\left(L_0+L_1\\|\\nabla f(x_{#1})\\|\\right)}\n\n\n\\newcommand{\\Gconstant}{8\\overline{\\Delta}_1+8+32\\log^2\\frac{T}{\\delta}+8\\frac{\\sqrt{2C}}{\\sqrt{L_0T}}\\log\\frac{T}{\\delta}+\\frac{303L_0}{224L_1^2}}\n\n\n\\newcommand{\\functiongapboundconstant}{\\overline{\\Delta}_c}\n\n\nWe consider the stochastic optimization problem that \n\\begin{align}\n  \\label{problem}\n  \\min\\limits_{x\\in\\mathbb{R}^d} f(x)=\\mathbb{E}_{z \\sim \\mathcal{D}}[f_z(x;z)],\n\\end{align}\n\nwhere the objective function $f(x)$ is possibly non-convex and $\\mathcal{D}$ is a probability distribution from which the random vector $z$ is drawn.\n\n\n\\begin{definition}[$\\left(L_0,L_1\\right)$-smoothness]\n  \\label{definition 1}\n  Let $f(x)$ be a differentiable function. For some $L_0,L_1>0$,\n   $f(x)$ is $(L_0,L_1)$-smooth iff for any $x,y\\in\\mathbb{R}^d$ with $\\|x-y\\|\\leq 1/L_1$,\n  \\begin{align}\n    \\label{def generalized smooth}\n  \\|\\nabla f(x)-\\nabla f(y)\\|\\leq \\left(L_0+L_1\\nablaf{x}\\right)\\|x-y\\|.\n  \\end{align}\n\\end{definition}\n\nIn this paper, we denote the set $\\{1,\\cdots,T\\}$ as $[T]$. \nWe use $\\mathbb{E}_t[\\cdot]\\triangleq\\mathbb{E}[\\cdot|z_1,\\cdots,z_{t-1}]$ to represent the conditional expectation, where $z_i$ is the random sample in the $i$-th gradient oracle. \nThe notation $a\\sim \\mathcal{O}(b)$ and $a\\leq \\mathcal{O}(b)$ refer to $c_1 b\\leq a\\leq c_2 b$ and $a\\leq c_3 b$ with $c_1, c_2, c_3$ being positive constants, respectively. \nAlso, we write $\\tilde{\\mathcal{O}}(b)$ as shorthand for $\\mathcal{O}(b \\cdot \\text{poly}(\\log b))$.\n\n\nWe consider Problem \\eqref{problem} over the Euclidean space $\\mathbb{R}^d$ with the $l_2$ norm, denoted as $\\|\\cdot\\|$.\n\n\n  \\item \\label{assumption 1} \\textbf{Below bounded:} The objective function is bounded from below, i.e.,  $f^*:=\\inf_{x\\in\\mathbb{R}^d}f(x)>-\\infty.$\n  \\item \\label{assumption 2}\\textbf{Unbiased estimator:} The gradient oracle returns an unbiased estimator of $\\nabla f(x)$, i.e., for all $x\\in\\mathbb{R}^d$, \n  $$\\mathbb{E}_z\\left[\\nabla f_z(x;z)\\right]=\\nabla f(x).$$\n  \\item \\label{assumption 3}\\textbf{Relaxed affine variance (almost-sure version):} The gradient oracle satisfies that for some constants $A,B,C>0$, $$\\|\\nabla f_z(x;z)-\\nabla f(x)\\|^2\\leq A\\left(f(x)-f^*\\right)+B\\nablaf{x}^2+C \\quad a.s., \\forall x\\in\\mathbb{R}^d.$$\n\n\n\\begin{algorithm}[H]\n  \\caption{Randomized Stochastic Accelerated Gradient (RSAG)}\n  \\label{algorithm1}\n  \\begin{algorithmic}[1]\n  \\Require Horizon $T$, $\\tilde{x}_1=x_1\\in \\mathbb{R}^d$, $\\alpha_t \\in \\left(\\left.0,1\\right.\\right]$, step sizes $\\{\\theta_t\\}_{t\\in [T]}$, $\\{\\gamma_t\\}_{t\\in [T]}$.\n  \\For{$t=1,\\cdots, T$}\n  \\State $\\overline{x}_t=\\alpha_tx_t+\\left(1-\\alpha_t\\right)\\tilde{x}_t$;\n  \\State \\textbf{Set} $g_t=\\nabla f_z(\\overline{x}_t;z_t)$;\n  \\State $x_{t+1}=x_t-\\theta_t g_t$;\n  \\State $\\tilde{x}_{t+1}=\\overline{x}_t-\\gamma_t g_t$.\n  \\EndFor\n  \\end{algorithmic}\n\\end{algorithm}\n\nFor simplicity, we denote $\\overline{\\Delta}_t=f(\\overline{x}_t)-f^*$ and $\\xi_t=g_t-\\overline{g}_t$ where $\\overline{g}_t=\\nabla f(\\overline{x}_t)$.\n\n\n\\begin{align}\n    \\label{Gconstant}\n    \\functiongapboundconstant=\\Gconstant.\n  \\end{align}","units":[{"start":1640,"end":1707,"start_line":62,"end_line":62,"raw_tex":"\\newcommand{\\lsmooth}[1]{\\left(L_0+L_1\\|\\nabla f(x_{#1})\\|\\right)}\n","sha256_utf8":"8370e7aa3fd61f530a023af464d7ac9d2b56a7b4b128eae68cab9232123af483"},{"start":2027,"end":2175,"start_line":75,"end_line":75,"raw_tex":"\\newcommand{\\Gconstant}{8\\overline{\\Delta}_1+8+32\\log^2\\frac{T}{\\delta}+8\\frac{\\sqrt{2C}}{\\sqrt{L_0T}}\\log\\frac{T}{\\delta}+\\frac{303L_0}{224L_1^2}}\n","sha256_utf8":"cbd279d409c5af23bd5292dca00726c5a95ed60060c10fcaa004434ed9e613c6"},{"start":2302,"end":2362,"start_line":79,"end_line":79,"raw_tex":"\\newcommand{\\functiongapboundconstant}{\\overline{\\Delta}_c}\n","sha256_utf8":"6f3b5ed9b8b574d5579357b6c51cdc6c46b3a1eb58fc4027658efbca901bff80"},{"start":4228,"end":4406,"start_line":111,"end_line":115,"raw_tex":"We consider the stochastic optimization problem that \n\\begin{align}\n  \\label{problem}\n  \\min\\limits_{x\\in\\mathbb{R}^d} f(x)=\\mathbb{E}_{z \\sim \\mathcal{D}}[f_z(x;z)],\n\\end{align}","sha256_utf8":"c60014fa98498c55bdbc6d4f99287e88e9b38f56573a09f9b8732a1d30f6d895"},{"start":4407,"end":4557,"start_line":116,"end_line":116,"raw_tex":"where the objective function $f(x)$ is possibly non-convex and $\\mathcal{D}$ is a probability distribution from which the random vector $z$ is drawn.\n","sha256_utf8":"dc1ad599b2a83ecdbeb539acecfe070fadd59dbc51834f1b0583fc34d2229eb1"},{"start":11247,"end":11634,"start_line":196,"end_line":204,"raw_tex":"\\begin{definition}[$\\left(L_0,L_1\\right)$-smoothness]\n  \\label{definition 1}\n  Let $f(x)$ be a differentiable function. For some $L_0,L_1>0$,\n   $f(x)$ is $(L_0,L_1)$-smooth iff for any $x,y\\in\\mathbb{R}^d$ with $\\|x-y\\|\\leq 1/L_1$,\n  \\begin{align}\n    \\label{def generalized smooth}\n  \\|\\nabla f(x)-\\nabla f(y)\\|\\leq \\left(L_0+L_1\\nablaf{x}\\right)\\|x-y\\|.\n  \\end{align}\n\\end{definition}","sha256_utf8":"d94e33c114145d189945ab1220d2c5c2dc54f4f70a0b6b40f16b0729a9b64843"},{"start":19002,"end":19522,"start_line":278,"end_line":281,"raw_tex":"In this paper, we denote the set $\\{1,\\cdots,T\\}$ as $[T]$. \nWe use $\\mathbb{E}_t[\\cdot]\\triangleq\\mathbb{E}[\\cdot|z_1,\\cdots,z_{t-1}]$ to represent the conditional expectation, where $z_i$ is the random sample in the $i$-th gradient oracle. \nThe notation $a\\sim \\mathcal{O}(b)$ and $a\\leq \\mathcal{O}(b)$ refer to $c_1 b\\leq a\\leq c_2 b$ and $a\\leq c_3 b$ with $c_1, c_2, c_3$ being positive constants, respectively. \nAlso, we write $\\tilde{\\mathcal{O}}(b)$ as shorthand for $\\mathcal{O}(b \\cdot \\text{poly}(\\log b))$.\n","sha256_utf8":"0f753a3599127f9c5fc2888760e9b8cd1dde28d6014838cfe38f19ee93de95b6"},{"start":29253,"end":29374,"start_line":371,"end_line":371,"raw_tex":"We consider Problem \\eqref{problem} over the Euclidean space $\\mathbb{R}^d$ with the $l_2$ norm, denoted as $\\|\\cdot\\|$.\n","sha256_utf8":"664928cf310395cb7bb3d5dfbf7b9da3c8b9897d94711d7c0b90a89ea9c03459"},{"start":29470,"end":30108,"start_line":379,"end_line":382,"raw_tex":"  \\item \\label{assumption 1} \\textbf{Below bounded:} The objective function is bounded from below, i.e.,  $f^*:=\\inf_{x\\in\\mathbb{R}^d}f(x)>-\\infty.$\n  \\item \\label{assumption 2}\\textbf{Unbiased estimator:} The gradient oracle returns an unbiased estimator of $\\nabla f(x)$, i.e., for all $x\\in\\mathbb{R}^d$, \n  $$\\mathbb{E}_z\\left[\\nabla f_z(x;z)\\right]=\\nabla f(x).$$\n  \\item \\label{assumption 3}\\textbf{Relaxed affine variance (almost-sure version):} The gradient oracle satisfies that for some constants $A,B,C>0$, $$\\|\\nabla f_z(x;z)-\\nabla f(x)\\|^2\\leq A\\left(f(x)-f^*\\right)+B\\nablaf{x}^2+C \\quad a.s., \\forall x\\in\\mathbb{R}^d.$$\n","sha256_utf8":"685ba2fb3916fcba8718bcb60e1a6ae7edbb1512891e28f075cc9c01d7583e37"},{"start":31180,"end":31775,"start_line":394,"end_line":406,"raw_tex":"\\begin{algorithm}[H]\n  \\caption{Randomized Stochastic Accelerated Gradient (RSAG)}\n  \\label{algorithm1}\n  \\begin{algorithmic}[1]\n  \\Require Horizon $T$, $\\tilde{x}_1=x_1\\in \\mathbb{R}^d$, $\\alpha_t \\in \\left(\\left.0,1\\right.\\right]$, step sizes $\\{\\theta_t\\}_{t\\in [T]}$, $\\{\\gamma_t\\}_{t\\in [T]}$.\n  \\For{$t=1,\\cdots, T$}\n  \\State $\\overline{x}_t=\\alpha_tx_t+\\left(1-\\alpha_t\\right)\\tilde{x}_t$;\n  \\State \\textbf{Set} $g_t=\\nabla f_z(\\overline{x}_t;z_t)$;\n  \\State $x_{t+1}=x_t-\\theta_t g_t$;\n  \\State $\\tilde{x}_{t+1}=\\overline{x}_t-\\gamma_t g_t$.\n  \\EndFor\n  \\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"2e3a6e7bbc95a601227978bedb7e5df489c2072a070a8118d9015fda2af5b0f5"},{"start":34901,"end":35051,"start_line":434,"end_line":434,"raw_tex":"For simplicity, we denote $\\overline{\\Delta}_t=f(\\overline{x}_t)-f^*$ and $\\xi_t=g_t-\\overline{g}_t$ where $\\overline{g}_t=\\nabla f(\\overline{x}_t)$.\n","sha256_utf8":"4ab257f7bae61bb02e7d8fd0e03316c38dded2de10f262c3784d69340a654a87"},{"start":62856,"end":62947,"start_line":722,"end_line":725,"raw_tex":"\\begin{align}\n    \\label{Gconstant}\n    \\functiongapboundconstant=\\Gconstant.\n  \\end{align}","sha256_utf8":"32338567636a42632924719ed0f6d0b0636ef9c0e59e251b7f67ce714a6221c6"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\n  \\label{theorem 1}\n  Let  $T>0$ and $\\delta\\in(0,1)$. Suppose that $f$ is an $\\left(L_0,L_1\\right)$-smooth function, $\\{\\overline{x}_t\\}_{x\\in[T]}$ is a sequence generated by Algorithm \\ref{algorithm1} with $\\alpha_t=\\frac{2}{t+1},\\gamma_t=\\left(1+\\alpha_t\\right)\\theta_t$ or $\\gamma_t=\\theta_t$ (SGD), $\\forall t\\in[T]$, and the step size $\\theta_t$ satisfies\n  \\begin{align}\n    \\label{eta constant non-convex}\n    \\theta_t=\\eta = \\min\\left\\{\\frac{1}{\\mathcal{A}\\sqrt{T}},\\frac{1}{\\mathcal{B}},\\frac{1}{\\mathcal{C}\\sqrt{T}}, \\frac{1}{\\mathcal{G}},\\frac{\\functiongapboundconstant^{-1/2}}{3\\left(A+4BL_0\\right)}, \\frac{\\functiongapboundconstant^{-3/2}}{12BL_1^2}\\right\\},\n  \\end{align}\n    where $\\mathcal{A}, \\mathcal{B}, \\mathcal{C}, \\mathcal{G}$ are defined as\n  \\begin{align}\n    \\label{theorem 1 eta}\n    \\mathcal{A}=4\\sqrt{\\left(L_0+L_1\\mathcal{M}_c\\right)A}, \\qquad &\\mathcal{B}=8\\left(L_0+L_1\\mathcal{M}_c\\right)\\left(B+1\\right), \\notag\\\\\n    \\mathcal{C}=2\\sqrt{\\left(L_0+L_1\\mathcal{M}_c\\right)C},\\qquad &\\mathcal{G}=8L_1 \\left(\\sqrt{A\\functiongapboundconstant}+2\\left(\\sqrt{B}+1\\right)\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}+\\sqrt{C}\\right),  \n  \\end{align}\n     and $\\functiongapboundconstant$, $\\mathcal{M}_c$ are given in the following order\\footnote{The explicit expressions of $\\functiongapboundconstant$ and $\\mathcal{M}_c$ are in \\eqref{Gconstant} and \\eqref{nabla fx_t constant step size}, respectively.},\n  \\begin{align*} \n  \\functiongapboundconstant \\sim \\mathcal{O}\\left(\\overline{\\Delta}_1+\\log^2\\frac{T}{\\delta}+\\frac{\\sqrt{C}}{\\sqrt{L_0T}}\\log\\frac{T}{\\delta}+\\frac{L_0}{L_1^2 }\\right), \\qquad  \\mathcal{M}_c \\sim \\mathcal{O}\\left(L_1\\functiongapboundconstant\\right).\n  \\end{align*}\n  Under Assumptions \\ref{assumption 1}, \\ref{assumption 2} and \\ref{assumption 3}, with probability at least $1-\\delta$, we have\\footnote{The detail convergence rate is stated in \\eqref{5.34}.}\n  \\begin{align}\n    \\label{4.1}\n    \\frac{1}{T}\\sumlimits{t=1}{T}\\|\\nabla f(\\overline{x}_t)\\|^2\\leq & \\mathcal{O}\\left(\\frac{\\functiongapboundconstant\\left(\\functiongapboundconstant B\\log \\frac{T}{\\delta}+\\sqrt{B+1}\\right)}{T}+\\frac{\\sqrt{\\functiongapboundconstant}\\left(\\functiongapboundconstant\\sqrt{A}+\\sqrt{C}\\right)}{\\sqrt{T}}\\right).\n  \\end{align}\n\\end{theorem}","units":[{"start":35438,"end":37754,"start_line":440,"end_line":462,"raw_tex":"\\begin{theorem}\n  \\label{theorem 1}\n  Let  $T>0$ and $\\delta\\in(0,1)$. Suppose that $f$ is an $\\left(L_0,L_1\\right)$-smooth function, $\\{\\overline{x}_t\\}_{x\\in[T]}$ is a sequence generated by Algorithm \\ref{algorithm1} with $\\alpha_t=\\frac{2}{t+1},\\gamma_t=\\left(1+\\alpha_t\\right)\\theta_t$ or $\\gamma_t=\\theta_t$ (SGD), $\\forall t\\in[T]$, and the step size $\\theta_t$ satisfies\n  \\begin{align}\n    \\label{eta constant non-convex}\n    \\theta_t=\\eta = \\min\\left\\{\\frac{1}{\\mathcal{A}\\sqrt{T}},\\frac{1}{\\mathcal{B}},\\frac{1}{\\mathcal{C}\\sqrt{T}}, \\frac{1}{\\mathcal{G}},\\frac{\\functiongapboundconstant^{-1/2}}{3\\left(A+4BL_0\\right)}, \\frac{\\functiongapboundconstant^{-3/2}}{12BL_1^2}\\right\\},\n  \\end{align}\n    where $\\mathcal{A}, \\mathcal{B}, \\mathcal{C}, \\mathcal{G}$ are defined as\n  \\begin{align}\n    \\label{theorem 1 eta}\n    \\mathcal{A}=4\\sqrt{\\left(L_0+L_1\\mathcal{M}_c\\right)A}, \\qquad &\\mathcal{B}=8\\left(L_0+L_1\\mathcal{M}_c\\right)\\left(B+1\\right), \\notag\\\\\n    \\mathcal{C}=2\\sqrt{\\left(L_0+L_1\\mathcal{M}_c\\right)C},\\qquad &\\mathcal{G}=8L_1 \\left(\\sqrt{A\\functiongapboundconstant}+2\\left(\\sqrt{B}+1\\right)\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}+\\sqrt{C}\\right),  \n  \\end{align}\n     and $\\functiongapboundconstant$, $\\mathcal{M}_c$ are given in the following order\\footnote{The explicit expressions of $\\functiongapboundconstant$ and $\\mathcal{M}_c$ are in \\eqref{Gconstant} and \\eqref{nabla fx_t constant step size}, respectively.},\n  \\begin{align*} \n  \\functiongapboundconstant \\sim \\mathcal{O}\\left(\\overline{\\Delta}_1+\\log^2\\frac{T}{\\delta}+\\frac{\\sqrt{C}}{\\sqrt{L_0T}}\\log\\frac{T}{\\delta}+\\frac{L_0}{L_1^2 }\\right), \\qquad  \\mathcal{M}_c \\sim \\mathcal{O}\\left(L_1\\functiongapboundconstant\\right).\n  \\end{align*}\n  Under Assumptions \\ref{assumption 1}, \\ref{assumption 2} and \\ref{assumption 3}, with probability at least $1-\\delta$, we have\\footnote{The detail convergence rate is stated in \\eqref{5.34}.}\n  \\begin{align}\n    \\label{4.1}\n    \\frac{1}{T}\\sumlimits{t=1}{T}\\|\\nabla f(\\overline{x}_t)\\|^2\\leq & \\mathcal{O}\\left(\\frac{\\functiongapboundconstant\\left(\\functiongapboundconstant B\\log \\frac{T}{\\delta}+\\sqrt{B+1}\\right)}{T}+\\frac{\\sqrt{\\functiongapboundconstant}\\left(\\functiongapboundconstant\\sqrt{A}+\\sqrt{C}\\right)}{\\sqrt{T}}\\right).\n  \\end{align}\n\\end{theorem}","sha256_utf8":"4640427d09d604749c34ab54f5c037d2bacc44c1e963e8ed8bb08515e3ece30d"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{proposition}[Proposition 5.2 in \\cite{kavis2022high}]\n  \\label{proposition 1}\n  Denote $\\alpha_t=\\frac{2}{t+1}$ and $\\varGamma_t=\\left(1-\\alpha_t\\right)\\varGamma_{t-1}$ with $\\varGamma_1=1$. We have that for all $t\\in[T],$\n  \\begin{align}\n    \\label{proposition 1.1}\n   \\varGamma_{t}\\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}=1,\n  \\end{align}\n  and \n  \\begin{align}\n    \\label{proposition 1.2}\n    \\left[\\sumlimits{k=t}{T}\\left(1-\\alpha_k\\right)\\varGamma_k\\right]\\frac{\\alpha_t}{\\varGamma_t}\\leq 2.\n  \\end{align}\n\\end{proposition}\n\n\\begin{proposition}\n  \\label{proposition overline x_t - x_t}\n  Let $\\{x_t\\}_{t\\in[T]}$ and $\\{\\overline{x}_t\\}_{t\\in[T]}$ be generated by Algorithm \\ref{algorithm1}. We have\n  \\begin{align}\n    \\label{overline x_t-x_t}\n    \\overline{x}_t-x_t =\\left(1-\\alpha_t\\right)\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\frac{\\left(\\theta_k-\\gamma_k\\right)}{\\alpha_k}g_k,\n  \\end{align}\n  and\n  \\begin{align*}\n  \\|\\overline{x}_t-x_t\\|^2\\leq \\left(1-\\alpha_t\\right)\\varGamma_t\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\frac{\\left(\\theta_k-\\gamma_k\\right)^2}{\\alpha_k^2}\\|g_k\\|^2.\n\\end{align*}\n\\end{proposition}\n\n\\begin{lemma}\n  \\label{high probability sum non-convex constant}\n  Given $T\\geq 1 $ and $\\delta\\in \\left(0,1\\right)$, if Assumptions \\ref{assumption 2} and \\ref{assumption 3} hold, then with probability at least $1-\\delta$, $\\forall l\\in[T]$,\n  \\begin{align}\n    \\label{inequality high probability non-convex constant}\n    \\sumlimits{t=1}{l}-\\la\\overline{g}_t,\\xi_t\\ra\\leq \\frac{1}{4}\\sumlimits{t=1}{l}\\frac{P_t^2}{P_c^2}\\|\\overline{g}_t\\|^2+3P_c^2\\log\\frac{T}{\\delta},\n  \\end{align}\n  where \n  \\begin{align}\n    \\label{P_t}\n    P_t=\\sqrt{A\\overline{\\Delta}_t+4B\\left(L_0\\overline{\\Delta}_t+L_1^2\\overline{\\Delta}_t^2\\right)+C},\n  \\end{align}\nand \n\\begin{align}\n  \\label{P}\n  P_c=\\sqrt{A\\functiongapboundconstant+4B\\left(L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2\\right)+C}.\n\\end{align}\n\\end{lemma}\n\n\\begin{proposition}\n  \\label{constant bound}\n  Under the conditions of Theorem \\ref{theorem 1}, $\\overline{\\Delta}_t\\leq \\functiongapboundconstant,P_t\\leq P_c, \\forall t\\in[T]$, hold with probability at least $1-\\delta$, where  $P_t$, $P_c$ are in \\eqref{P_t}, \\eqref{P} and $\\functiongapboundconstant$ is defined as\n  \\begin{align}\n    \\label{Gconstant}\n    \\functiongapboundconstant=\\Gconstant.\n  \\end{align} \n\\end{proposition}\n\n\\begin{proof}\n  We assume that \\eqref{inequality high probability non-convex constant} always happens and then deduce $\\overline{\\Delta}_t\\leq \\functiongapboundconstant$, $P_t\\leq P_c$ for all $t\\in[T]$. \n  Since \\eqref{inequality high probability non-convex constant} happens with probability at least $1-\\delta$, $\\overline{\\Delta}_t\\leq \\functiongapboundconstant$ and $P_t\\leq P_c$, $\\forall t\\in[T]$,  hold with probability at least $1-\\delta$. \n  It is obvious that $f(\\overline{x}_1)-f^*\\leq \\functiongapboundconstant$. Therefore, $P_1\\leq P_c$.\n  Suppose that for some $l\\in[T]$, $$f(\\overline{x}_t)-f^*\\leq \\functiongapboundconstant,\\quad\\forall t\\in[l], \\quad \\text{thus}, \\quad P_t\\leq P_c, \\quad \\forall t\\in[l].$$ \n  By the triangle inequality of the norm function,\n  \\begin{align*}\n    \\|g_t\\|\\leq  \\|g_t-\\overline{g}_t\\|+\\|\\overline{g}_t\\|\n    \\leq  \\sqrt{A\\overline{\\Delta}_t+B\\|\\overline{g}_t\\|^2+C}+\\|\\overline{g}_t\\|\n    \\leq  \\sqrt{A\\overline{\\Delta}_t}+\\left(\\sqrt{B}+1\\right)\\|\\overline{g}_t\\|+\\sqrt{C},\n  \\end{align*}\n  where the second inequality follows from Assumption \\ref{assumption 3} and the last inequality follows from Lemma \\ref{sqrt sum}.\n  Combining with Lemma \\ref{lemma 6.2} and the assumption that $\\overline{\\Delta}_t\\leq \\functiongapboundconstant, \\forall t\\in[l]$, we have\n  \\begin{align} \n    \\label{g_t}\n    \\|g_t\\| \\leq  \\sqrt{A\\functiongapboundconstant}+2\\left(\\sqrt{B}+1\\right)\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}+\\sqrt{C}, \\quad \\forall t\\in[l].\n  \\end{align}\n  For simplicity, let \n  \\begin{align}\n    \\label{Y_c}\n    Y_c=\\eta \\left(\\sqrt{A\\functiongapboundconstant}+2\\left(\\sqrt{B}+1\\right)\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}+\\sqrt{C}\\right).\n  \\end{align}\n    By the iteration step of Algorithm \\ref{algorithm1}, we have\n  \\begin{align*}\n  \\|x_{t+1}-x_t\\|=\\theta_t\\|g_t\\|=\\eta \\|g_t\\|\\leq Y_c\\leq 1/8L_1,\n  \\end{align*}\n  where the last inequality follows from the restriction of $\\eta$.\n Applying Lemma \\ref{descent lemma}, we have \n\\begin{align*}\n  & f(x_{t+1})-f(x_t) \\notag\\\\\n  \\leq & \\la \\nabla f(x_t),x_{t+1}-x_t\\ra+\\frac{L_0+L_1\\nablaf{x_t}}{2}\\|x_{t+1}-x_t\\|^2\\notag\\\\\n   = & -\\theta_t\\la \\nabla f(x_t),g_t\\ra+\\frac{L_0+L_1\\nablaf{x_t}}{2}\\theta_t^2\\|g_t\\|^2\\notag\\\\\n   = & -\\theta_t\\la \\overline{g}_t+\\nabla f(x_t)-\\overline{g}_t,\\overline{g}_t+\\xi_t\\ra + \\frac{L_0+L_1\\nablaf{x_t}}{2}\\theta_t^2\\|g_t\\|^2\\notag\\\\\n   = & -\\theta_t \\|\\overline{g}_t\\|^2-\\theta_t\\la \\overline{g}_t,\\xi_t\\ra -\\theta_t \\la \\nabla f(x_t)-\\overline{g}_t,g_t\\ra+\\frac{L_0+L_1\\nablaf{x_t}}{2}\\theta_t^2\\|g_t\\|^2\\notag\\\\\n  \\leq & -\\theta_t \\|\\overline{g}_t\\|^2-\\theta_t\\la \\overline{g}_t,\\xi_t\\ra +\\theta_t \\| \\nabla f(x_t)-\\overline{g}_t\\|\\|g_t\\|+\\frac{L_0+L_1\\nablaf{x_t}}{2}\\theta_t^2\\|g_t\\|^2,\n\\end{align*} \nwhere the first equation follows from the update rule in Algorithm \\ref{algorithm1} and the last line follows from Cauchy-Schwarz inequality. \nRecall the constraints of the step size for RSAG with the constant step size and SGD, we have \n\\begin{align}\n  \\label{constant step size constraint}\n  0\\leq \\frac{\\gamma_t-\\theta_t}{\\alpha_t}\\leq \\theta_t=\\eta, \\quad \\forall t\\in[T].\n\\end{align}\nCombining \\eqref{overline x_t-x_t} with \\eqref{proposition 1.1}, \\eqref{g_t}, \\eqref{constant step size constraint}, we have\n\\begin{align}\n  \\label{bound overline x_t-x_t}\n  \\|\\overline{x}_t-x_t\\| \\leq & \\eta\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\|g_k\\|\\notag\\\\\n  \\leq & \\eta\\left(\\sqrt{A\\functiongapboundconstant}+2\\left(\\sqrt{B}+1\\right)\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}+\\sqrt{C}\\right)\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\notag\\\\\n  = & Y_c \\leq 1/8L_1.\n\\end{align}\nNote that \\eqref{bound overline x_t-x_t} holds for all $t\\in[l+1]$. \nThus, applying Definition \\ref{definition 1}, we have that for all $t\\in[l]$,\n\\begin{align}  \n  \\label{inequality base}\n  & f(x_{t+1})-f(x_t) \\notag\\\\\n  \\leq & -\\theta_t \\|\\overline{g}_t\\|^2-\\theta_t\\la \\overline{g}_t,\\xi_t\\ra +\\theta_t\\left(L_0+L_1\\nablaf{x_t}\\right)\\|\\overline{x}_t-x_t\\|\\|g_t\\| +\\frac{L_0+L_1\\nablaf{x_t}}{2}\\theta_t^2\\|g_t\\|^2\\notag\\\\\n  \\leq & -\\theta_t\\|\\overline{g}_t\\|^2-\\theta_t\\la\\overline{g}_t,\\xi_t\\ra+\\frac{1}{2}\\left(L_0+L_1\\nablaf{x_t}\\right)\\|\\overline{x}_t-x_t\\|^2+\\left(L_0+L_1\\nablaf{x_t}\\right)\\theta_t^2\\|g_t\\|^2\\notag\\\\\n  \\leq & -\\theta_t\\|\\overline{g}_t\\|^2-\\theta_t\\la\\overline{g}_t,\\xi_t\\ra+\\frac{1}{2}\\left(L_0+L_1\\nablaf{x_t}\\right)\\left(1-\\alpha_t\\right)\\varGamma_t\\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}\\theta_k^2\\|g_k\\|^2\\notag\\\\\n  & + \\left(L_0+L_1\\nablaf{x_t}\\right)\\theta_t^2\\|g_t\\|^2,\n\\end{align}\nwhere we use Young's inequality in the second inequality and apply Proposition \\ref{proposition overline x_t - x_t} with \\eqref{constant step size constraint} in the last inequality. \nSumming over $t\\in [l]$ with $\\theta_t=\\eta$, we have\n\\begin{align}\n\\label{telescope}\nf(x_{l+1})-f(x_1)  \\leq& \\frac{1}{2}\\sum\\limits^{l}_{t=1}\\left[\\lsmooth{t}\\left(1-\\alpha_t\\right)\\varGamma_t\\sum\\limits_{k=1}^t\\frac{\\alpha_k}{\\varGamma_k}\\eta^2\\|g_k\\|^2\\right]\\notag\\\\\n&+\\eta^2\\sumlimits{t=1}{l}\\lsmooth{t}\\|g_t\\|^2-\\eta\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2-\\eta\\sumlimits{t=1}{l}\\la \\overline{g}_t,\\xi_t\\ra\\notag\\\\\n=& \\frac{1}{2}\\eta^2 \\sumlimits{t=1}{l}\\left[\\sumlimits{k=t}{l}\\lsmooth{k}\\left(1-\\alpha_k\\right)\\varGamma_k\\right]\\frac{\\alpha_t}{\\varGamma_t}\\|g_t\\|^2\\notag\\\\\n&+\\eta^2\\sumlimits{t=1}{l}\\lsmooth{t}\\|g_t\\|^2-\\eta\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2-\\eta\\sumlimits{t=1}{l}\\la \\overline{g}_t,\\xi_t\\ra.\n\\end{align}\nApplying the triangle inequality of $\\|\\cdot\\|$, we have that for all $t\\in[l]$,\n\\begin{align}\n  \\label{nabla fx_t}\n  \\nablaf{x_t} \\leq  \\|\\overline{g}_t-\\nabla f(x_t)\\|+\\|\\overline{g}_t\\| \n  \\leq  \\left(L_0+L_1\\|\\overline{g}_t\\|\\right)\\|\\overline{x}_t-x_t\\|+\\|\\overline{g}_t\\|,\n\\end{align}\nwhere the second inequality holds since $\\|\\overline{x}_t-x_t\\|\\leq 1/L_1$. Combining \\eqref{bound overline x_t-x_t} and \\eqref{nabla fx_t}, for all $t\\in[l]$,\n\\begin{align}\n  \\label{nabla fx_t constant step size}\n   \\nablaf{x_t}\n  \\leq & \\frac{1}{8L_1}\\left(L_0+2L_1\\sqrt{L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2}\\right)+2\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}\\notag\\\\\n  = & \\frac{L_0}{8L_1}+\\frac{9}{4} \\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}=\\mathcal{M}_c,\n\\end{align}\nwhere we apply Lemma \\ref{lemma 6.2} to the inequality. \nApplying the triangle inequality again, we have\n\\begin{align}\n  \\label{triangle square}\n  \\|g_t\\|^2\\leq 2\\|g_t-\\overline{g}_t\\|^2+2\\|\\overline{g}_t\\|^2\\leq 2\\left(A \\overline{\\Delta}_t+\\left(B+1\\right)\\|\\overline{g}_t\\|^2+C\\right).\n\\end{align}\nCombining  \\eqref{proposition 1.2}, \\eqref{inequality high probability non-convex constant}, \\eqref{telescope} and \\eqref{triangle square}, together with the bound of $\\nablaf{x_t}$ in \\eqref{nabla fx_t constant step size}, we have \n\\begin{align}\n  \\label{4.26}\nf(x_{l+1})-f(x_1)  \\leq & 2\\left(L_0+L_1\\mathcal{M}_c\\right)\\eta^2\\sumlimits{t=1}{l}\\|g_t\\|^2-\\eta\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2-\\eta\\sumlimits{t=1}{l}\\la\\overline{g}_t,\\xi_t\\ra\\notag\\\\\n\\leq & 4\\left(L_0+L_1\\mathcal{M}_c\\right)\\eta^2\\sumlimits{t=1}{l}\\left(A\\overline{\\Delta}_t+\\left(B+1\\right)\\|\\overline{g}_t\\|^2+C\\right)-\\eta\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2\\notag\\\\\n&+\\frac{1}{4}\\eta\\sumlimits{t=1}{l}\\frac{P_t^2}{P_c^2}\\|\\overline{g}_t\\|^2+3\\eta P_c^2\\log\\frac{T}{\\delta}\\notag\\\\\n\\leq & 4\\left(L_0+L_1\\mathcal{M}_c\\right)\\eta^2 lA\\functiongapboundconstant+4\\left(L_0+L_1\\mathcal{M}_c\\right)\\eta^2lC \\notag\\\\\n&+\\left(4\\left(L_0+L_1\\mathcal{M}_c\\right)\\eta\\left(B+1\\right)+\\frac{1}{4}-1\\right)\\eta\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2+3\\eta P_c^2\\log\\frac{T}{\\delta},\n\\end{align}\nwhere the last inequality holds since the assumption that $P_t\\leq P_c, \\forall t\\in[l]$. \nApplying the restrictions of $\\eta$ in \\eqref{eta constant non-convex}, it is apparent that \n\\begin{align*}\n  f(x_{l+1})\\leq f(x_1)+\\frac{1}{4}\\functiongapboundconstant+1+3\\eta P_c^2\\log\\frac{T}{\\delta}.\n\\end{align*}\nSince $P_c^2 = A\\functiongapboundconstant+4B\\left(L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2\\right)+C= \\left(A+4BL_0\\right)\\functiongapboundconstant+4BL_1^2\\functiongapboundconstant^2+C$,\n\\begin{align}\n  \\label{f x+1-f x_1 constant step size}\n  f(x_{l+1}) \\leq & f(x_1)+\\frac{1}{4}\\functiongapboundconstant+1+3\\eta \\left(A+4BL_0\\right)\\functiongapboundconstant\\log\\frac{T}{\\delta}+12\\eta BL_1^2\\functiongapboundconstant^2\\log\\frac{T}{\\delta}+3\\eta C \\log\\frac{T}{\\delta}\\notag\\\\\n  \\leq & f(x_1)+\\frac{1}{4}\\functiongapboundconstant+1+2\\sqrt{\\functiongapboundconstant}\\log\\frac{T}{\\delta}+3\\eta C\\log\\frac{T}{\\delta}     \\notag\\\\\n  \\leq & f(x_1)+\\frac{1}{2}\\functiongapboundconstant+1+ 4\\log^2\\frac{T}{\\delta}+3\\eta C \\log\\frac{T}{\\delta},\n\\end{align}\nwhere the second inequality is due to the restrictions of $\\eta$ in \\eqref{eta constant non-convex} and the last inequality follows from Young's inequality. Next we will bound the gap between $f(x_{l+1})$ and $f(\\overline{x}_{l+1})$. By \\eqref{bound overline x_t-x_t}, we have $\\|\\overline{x}_{l+1}-x_{l+1}\\|<1/L_1$. Using Lemma \\ref{descent lemma} again,\n\\begin{align}\n  \\label{gap between f overline x_t+1 and f x_t+1 constant step size}\n  f(\\overline{x}_{l+1}) \\leq & f(x_{l+1})+\\la \\overline{g}_{l+1}, \\overline{x}_{l+1}-x_{l+1}\\ra+\\frac{L_0+L_1\\|\\overline{g}_{l+1}\\|}{2}\\|\\overline{x}_{l+1}-x_{l+1}\\|^2\\notag\\\\\n  \\leq & f(x_{l+1})+\\|\\overline{g}_{l+1}\\|\\|\\overline{x}_{l+1}-x_{l+1}\\|+\\frac{L_0+L_1\\|\\overline{g}_{l+1}\\|}{2}\\|\\overline{x}_{l+1}-x_{l+1}\\|^2,\n\\end{align}\nwhere the second inequality follows from Cauchy-Schwarz inequality. Substituting \\eqref{bound overline x_t-x_t} into \\eqref{gap between f overline x_t+1 and f x_t+1 constant step size}, we have\n\\begin{align}\n  \\label{descent lemma 2}\n  f(\\overline{x}_{l+1}) \\leq & f(x_{l+1})+\\frac{1}{8L_1}\\|\\overline{g}_{l+1}\\|+\\frac{1}{128L_1^2}\\left(L_0+L_1\\|\\overline{g}_{l+1}\\|\\right)\\notag\\\\\n  = & f(x_{l+1})+\\frac{17}{128L_1}\\|\\overline{g}_{l+1}\\|+\\frac{L_0}{128L_1^2}.\n\\end{align}\nCombining \\eqref{f x+1-f x_1 constant step size} and \\eqref{descent lemma 2}, and subtracting $f^*$ from both sides, we have\n\\begin{align}\n  \\overline{\\Delta}_{l+1}\\leq & \\overline{\\Delta}_1+\\frac{1}{2}\\functiongapboundconstant+1+4\\log^2\\frac{T}{\\delta}+3\\eta C \\log\\frac{T}{\\delta} +\\frac{17}{128L_1}\\|\\overline{g}_{l+1}\\|+\\frac{L_0}{128L_1^2}\\notag\\\\\n  \\leq & \\overline{\\Delta}_1+\\frac{1}{2}\\functiongapboundconstant+1+4\\log^2\\frac{T}{\\delta}+3\\eta C \\log\\frac{T}{\\delta} + \\frac{17}{64L_1}\\left(\\sqrt{L_0\\overline{\\Delta}_{l+1}}+L_1\\overline{\\Delta}_{l+1}\\right)+  \\frac{L_0}{128L_1^2}\\notag\\\\\n  \\leq &   \\overline{\\Delta}_1+\\frac{1}{2}\\functiongapboundconstant+1+4\\log^2\\frac{T}{\\delta}+\\frac{\\sqrt{2C}}{\\sqrt{L_0T}}\\log\\frac{T}{\\delta} + \\frac{7}{64}\\overline{\\Delta}_{l+1}+\\frac{289L_0}{1792L_1^2}+\\frac{17}{64}\\overline{\\Delta}_{l+1}+\\frac{L_0}{128L_1^2}.\n\\end{align}\nThe second inequality follows from Lemma \\ref{lemma 6.2} and Lemma \\ref{sqrt sum}. The third inequality follows from Young's inequality and the constraint of $\\eta$ that $\\eta\\leq \\frac{1}{2\\sqrt{\\left(L_0+L_1\\mathcal{M}_c\\right)CT}}$, where $\\mathcal{M}_c\\geq \\frac{L_0}{8L_1}$.\nRe-arranging the above inequality, we have \n\\begin{align*}\n  \\frac{5}{8}\\overline{\\Delta}_{l+1}\\leq \\overline{\\Delta}_1+\\frac{1}{2}\\functiongapboundconstant+1+4\\log^2\\frac{T}{\\delta}+\\frac{\\sqrt{2C}}{\\sqrt{L_0T}}+\\frac{303L_0}{1792L_1^2}\\leq \\frac{5}{8}\\functiongapboundconstant.\n\\end{align*}\nThe second inequality holds since \n$\\functiongapboundconstant=\\Gconstant$.\nTherefore, $P_{l+1}\\leq P_c$ also holds. \nSo the induction is complete and we obtain the desired result. \n\\end{proof}\n\n\\begin{proof}[Proof of Theorem \\ref{theorem 1}]\nWe assume \\eqref{inequality high probability non-convex constant} always holds then we deduce \\eqref{4.1}. Note that \\eqref{inequality high probability non-convex constant} holds with probability at least $1-\\delta$. Therefore, \\eqref{4.1} holds with probability at least $1-\\delta$.\nRecalling \\eqref{inequality base} and dividing $\\theta_t$ on both sides, we have  \n\\begin{align*}\n  \\|\\overline{g}_t\\|^2\\leq &\\frac{f(x_t)-f(x_{t+1})}{\\theta_t}+\\frac{1}{2\\theta_t}\\left(L_0+L_1\\nablaf{x_t}\\right)\\left(1-\\alpha_t\\right)\\varGamma_t\\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}\\theta_k^2\\|g_k\\|^2\\notag\\\\\n&+\\left(L_0+L_1\\nablaf{x_t}\\right)\\theta_t\\|g_t\\|^2-\\la\\overline{g}_t,\\xi_t\\ra.\n\\end{align*}\nSumming over $t\\in[T]$ with $\\theta_t=\\eta$,\n\\begin{align}\n  \\label{constant gt}\n  \\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2\\leq & \\frac{1}{\\eta}\\sumlimits{t=1}{T}\\left(f(x_t)-f(x_{t+1})\\right)+\\frac{\\eta}{2}\\sumlimits{t=1}{T}\\left[\\left(L_0+L_1\\nablaf{x_t}\\right)\\left(1-\\alpha_t\\right)\\varGamma_t\\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}\\|g_k\\|^2\\right]\\notag\\\\\n  &+\\eta\\sumlimits{t=1}{T}\\left(L_0+L_1\\nablaf{x_t}\\right)\\|g_t\\|^2-\\sumlimits{t=1}{T}\\la \\overline{g}_t,\\xi_t\\ra\\notag\\\\\n  \\leq &\\frac{\\overline{\\Delta}_1}{\\eta}+\\frac{\\eta}{2} \\sumlimits{t=1}{T}\\left[\\sumlimits{k=t}{T}\\lsmooth{k}\\left(1-\\alpha_k\\right)\\varGamma_k\\right]\\frac{\\alpha_t}{\\varGamma_t}\\|g_t\\|^2\\notag\\\\\n  & +\\eta\\sumlimits{t=1}{T}\\left(L_0+L_1\\nablaf{x_t}\\right)\\|g_t\\|^2-\\sumlimits{t=1}{T}\\la \\overline{g}_t,\\xi_t\\ra.\n\\end{align}\nApplying Lemma \\ref{high probability sum non-convex constant},\n\\begin{align*}\n  \\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2\\leq &\\frac{\\overline{\\Delta}_1}{\\eta}+\\frac{\\eta}{2} \\sumlimits{t=1}{T}\\left[\\sumlimits{k=t}{T}\\lsmooth{k}\\left(1-\\alpha_k\\right)\\varGamma_k\\right]\\frac{\\alpha_t}{\\varGamma_t}\\|g_t\\|^2\\notag\\\\\n  & +\\eta\\sumlimits{t=1}{T}\\left(L_0+L_1\\nablaf{x_t}\\right)\\|g_t\\|^2+\\frac{1}{4P_c^2}\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2 P_t^2+3P_c^2\\log\\frac{T}{\\delta}.\n\\end{align*}\nCombining with \\eqref{proposition 1.2}, \\eqref{nabla fx_t constant step size}, \\eqref{triangle square} and Proposition \\ref{constant bound}, we have\n\\begin{align*}\n  \\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2 \\leq & \\frac{\\overline{\\Delta}_1}{\\eta}+2\\eta\\sumlimits{t=1}{T}\\left(L_0+L_1\\mathcal{M}_c\\right)\\|g_t\\|^2+\\frac{1}{4}\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2+3P_c^2\\log\\frac{T}{\\delta}\\notag\\\\\n  \\leq & \\frac{\\overline{\\Delta}_1}{\\eta}+4\\eta\\left(L_0+L_1\\mathcal{M}_c\\right) T \\left(A\\functiongapboundconstant+C\\right)+4\\left(L_0+L_1\\mathcal{M}_c\\right)\\left(B+1\\right)\\eta\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2\\notag\\\\\n  & + \\frac{1}{4}\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2+3\\left(A\\functiongapboundconstant + 4B\\left(L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2\\right)+C\\right)\\log\\frac{T}{\\delta}.\n\\end{align*}\nRe-arranging the above inequality with the constraint of $\\eta$ in \\eqref{eta constant non-convex}, we obtain \n\\begin{align*}\n  \\frac{1}{T}\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2\\leq & \\frac{4\\overline{\\Delta}_1}{\\eta T}+ \\frac{4}{\\sqrt{T}}\\sqrt{L_0+L_1\\mathcal{M}_c}\\left(\\sqrt{A\\functiongapboundconstant^2}+2\\sqrt{C}\\right)\\notag\\\\\n  &+\\frac{12}{T}\\left(A\\functiongapboundconstant + 4B\\left(L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2\\right)+C\\right)\\log\\frac{T}{\\delta}.\n\\end{align*}\nCombining with the restrictions of $\\eta$ in \\eqref{eta constant non-convex} again, we get the final result,\n\\begin{align}\n  \\label{5.34}\n  \\frac{1}{T}\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2\\leq & \\frac{4}{\\sqrt{T}}\\sqrt{L_0+L_1\\mathcal{M}_c}\\left(\\sqrt{A}\\left(\\functiongapboundconstant+4\\overline{\\Delta}_1\\right)+2\\sqrt{C}\\left(1+\\overline{\\Delta}_1\\right)\\right)\\notag\\\\\n  &+ \\frac{32\\overline{\\Delta}_1}{T}\\left(L_0+L_1\\mathcal{M}_c\\right)\\left(B+1\\right) +  \\frac{12\\overline{\\Delta}_1}{T}\\left(\\left(A+4BL_0\\right)\\functiongapboundconstant^{1/2}+4BL_1^2\\functiongapboundconstant^{3/2}\\right) \\notag\\\\\n  &+ \\frac{32\\overline{\\Delta}_1}{T}L_1 \\left(\\sqrt{A\\functiongapboundconstant}+2\\left(\\sqrt{B}+1\\right)\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}+\\sqrt{C}\\right) \\notag\\\\\n  &+\\frac{12}{T}\\left(A\\functiongapboundconstant + 4B\\left(L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2\\right)+C\\right)\\log\\frac{T}{\\delta}.\n\\end{align}\n\\end{proof}\n\n\\begin{lemma}[Remark 2.3 in \\cite{zhang2020improved}]\n  \\label{descent lemma}\n  Suppose that $f:\\mathbb{R}^d \\rightarrow \\mathbb{R}$ is an $(L_0,L_1)$-generalized smooth function. When $\\|x-y\\|\\leq 1/L_1$, we have \n  $$\\left\\lvert f(y)- f(x)-\\la \\nabla f(x),y-x\\ra\\right\\rvert \\leq \\frac{L_0+L_1\\nablaf{x}}{2}\\|x-y\\|^2.$$\n\\end{lemma}\n\n\\begin{lemma}[Lemma 1 in \\cite{li2020high}]\n  \\label{lemma high probability}\n  Assume that $\\{Z_t\\}_{t\\in[T]}$ is a martingale difference sequence with respect to $\\gamma_1, \\gamma_2, \\cdots, \\gamma_T$ and $\\mathbb{E}_t[\\exp\\left(Z_t^2/\\sigma_t^2\\right)]\\leq \\exp(1)$ for all $1 \\leq t \\leq T$, where $\\sigma_t$ is a sequence of measurable random variables with respect to $\\gamma_1, \\gamma_2, \\cdots, \\gamma_{t-1}$. Then, for any fixed $\\lambda >0$ and $\\delta\\in(0,1)$, with probability at least $1-\\delta$, we have\n  $$\\sumlimits{t=1}{T}Z_t\\leq \\frac{3\\lambda}{4}\\sumlimits{t=1}{T}\\sigma_t^2+\\frac{1}{\\lambda}\\log\\frac{1}{\\delta}.$$\n\\end{lemma}\n\n\\begin{lemma}\n  \\label{lemma 6.2}  \n  Let $f(x) : \\mathbb{R}^d\\rightarrow \\mathbb{R}$ be an $(L_0,L_1)$-smooth function with minimum $f^*$. \n  Then, $$\\|\\nabla f(x_t)\\|^2\\leq 2\\left(L_0+L_1\\|\\nabla f(x_t)\\|\\right)\\Delta_t,$$ and $$\\nablaf{x_t}^2\\leq 4L_0\\Delta_t+4L_1^2\\Delta_t^2,$$\n  where $\\Delta_t=f(x_t)-f^*$.\n  \\end{lemma}\n\n\\begin{proof}\n    Let $x=x_t-\\frac{1}{L_0+L_1\\|\\nabla f(x_t)\\|}\\nabla f(x_t)$.\n    It is easy to verify that $\\|x-x_t\\|\\leq 1/L_1$.\n    From Lemma \\ref{descent lemma}, we have that\n    $$f(x) \\leq f(x_t)  + \\la \\nabla f(x_t), x - x_t \\ra + {L_0+L_1 \\|\\nabla f(x)\\|  \\over 2} \\|x - x_t\\|^2.$$\n    Hence, \n    \\begin{align*}\n      &\\frac{\\|\\nabla f(x_t)\\|^2}{2\\left(L_0+L_1\\|\\nabla f(x_t)\\|\\right)}\\leq f(x_t)-f(x)\\leq \\Delta_t.\n    \\end{align*}\n  Re-arranging the above inequality and applying Cauchy-Schwarz inequality and Young's inequality, we have\n  \\begin{align*}\n    \\nablaf{x_{t}}^2 \\leq & 2L_0\\Delta_{t}+\\frac{1}{2}\\|\\nabla f(x_{t})\\|^2+2L_1^2\\Delta_{t}^2.\n  \\end{align*}\n  We obtain the desired result.\n  \\end{proof}\n\n\\begin{lemma}\n    \\label{sqrt sum}\n    Let $\\{a_t\\}_{t\\in[n]}$ be a sequence of non-negative real numbers. We have\n    \\begin{align*}\n      \\sqrt{\\sum_{i=1}^n a_i}\\leq \\sumlimits{i=1}{n}\\sqrt{a_i}.\n    \\end{align*}\n  \\end{lemma}\n\n\\begin{proof}[Proof of Proposition \\ref{proposition 1}]\n  From \\cite{ghadimi2016accelerated}, we have\n  \\begin{align*}\n    \\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}=\\frac{\\alpha_1}{\\varGamma_1}+\\sumlimits{k=2}{t}\\frac{\\alpha_k}{\\varGamma_k}=\\frac{1}{\\varGamma_1}+\\sumlimits{k=2}{t}\\frac{1}{\\varGamma_k}\\left(1-\\frac{\\varGamma_{k}}{\\varGamma_{k-1}}\\right)=\\frac{1}{\\varGamma_1}+\\sumlimits{k=2}{t}\\left(\\frac{1}{\\varGamma_k}-\\frac{1}{\\varGamma_{k-1}}\\right)=\\frac{1}{\\varGamma_t}.\n  \\end{align*}\n  Therefore, \n  \\begin{align*}\n    \\varGamma_{t}\\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}=1.\n  \\end{align*}\n  For any $t\\geq 2$, we have\n \\begin{align*}\n  \\varGamma_t=\\left(1-\\alpha_t\\right)\\varGamma_{t-1}=\\frac{t-1}{t+1}\\varGamma_{t-1}=\\cdots=\\frac{t-1}{t+1}\\cdot\\frac{t-2}{t}\\cdots \\frac{2-1}{2+1}=\\frac{2}{t\\left(t+1\\right)}.\n\\end{align*}\nHence,\n  \\begin{align*}\n    \\left[\\sumlimits{k=t}{T}\\left(1-\\alpha_k\\right)\\varGamma_k\\right]\\frac{\\alpha_t}{\\varGamma_t} & =\\left[\\sumlimits{k=t}{T}\\frac{k-1}{k+1}\\frac{2}{k\\left(k+1\\right)}\\right]\\cdot t\\notag\\\\\n    \\leq & 2t \\sumlimits{k=t}{T}\\frac{1}{k\\left(k+1\\right)}\\notag\\\\\n    = & 2t\\left(\\frac{1}{t}-\\frac{1}{T+1}\\right) \\leq 2.\n  \\end{align*}\n\\end{proof}\n\n\\begin{proof}[Proof of Proposition \\ref{proposition overline x_t - x_t}]\n  Following from the iteration steps in Algorithm \\ref{algorithm1} and referring to the proof in \\cite{ghadimi2016accelerated}, we have \n  \\begin{align*}\n    \\overline{x}_t-x_t = & \\left(1-\\alpha_t\\right)[\\tilde{x}_t-x_t]\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)[\\overline{x}_{t-1}-x_{t-1}+\\left(\\theta_{t-1}-\\gamma_{t-1}\\right)g_{t-1}]\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)[\\left(1-\\alpha_{t-1}\\right)\\left(\\tilde{x}_{t-1}-x_{t-1}\\right)+\\left(\\theta_{t-1}-\\gamma_{t-1}\\right)g_{t-1}]\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)\\sumlimits{k=1}{t-1}\\left(\\prod\\limits_{j=k+1}^{t-1}\\left(1-\\alpha_j\\right)\\right)\\left(\\theta_k-\\gamma_k\\right)g_k\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)\\sumlimits{k=1}{t-1}\\frac{\\varGamma_{t-1}}{\\varGamma_{k}}\\left(\\theta_k-\\gamma_k\\right)g_k\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\frac{\\left(\\theta_k-\\gamma_k\\right)}{\\alpha_k}g_k.\n  \\end{align*}\n  By the convexity of square of the norm and \\eqref{proposition 1.1}, we have \n  \\begin{align}\n    \\label{overline x_t-x_t square norm}\n    \\|\\overline{x}_t-x_t\\|^2 = & \\left\\Vert\\left(1-\\alpha_t\\right)\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k} \\frac{\\left(\\theta_k-\\gamma_k\\right)}{\\alpha_k}  g_k\\right\\Vert^2\\notag\\\\\n    \\leq & \\left(1-\\alpha_t\\right)^2\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\frac{\\left(\\theta_k-\\gamma_k\\right)^2}{\\alpha_k^2}\\|g_k\\|^2\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)\\varGamma_t\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\frac{\\left(\\theta_k-\\gamma_k\\right)^2}{\\alpha_k^2}\\|g_k\\|^2.\n  \\end{align}\n\\end{proof}\n\n\\begin{proof}[Proof of Lemma \\ref{high probability sum non-convex constant}]\n  Let $Z_t=-\\la \\overline{g}_t,\\xi_t\\ra$. \n  Note that $\\overline{g}_t$ is a random variable dependent on $z_1,\\cdots,z_{t-1}$ and $\\xi_t$ is dependent on $z_1,\\cdots,z_t$. Therefore, it is apparent that $Z_t$ is a martingale difference sequence since\n  $$\\mathbb{E}\\left[- \\la \\overline{g}_t,\\xi_t\\ra|z_1,\\cdots,z_{t-1}\\right]=-\\la \\overline{g}_t,\\mathbb{E}_t[\\xi_t]\\ra=0.$$\n  Using Cauchy-Schwarz inequality,\n  \\begin{align*}\n    \\mathbb{E}_t\\left[\\exp\\left(\\frac{Z_t^2}{\\|\\overline{g}_t\\|^2\\left(A\\overline{\\Delta}_t+B\\|\\overline{g}_t\\|^2+C\\right)}\\right)\\right]\\leq\\mathbb{E}_t\\left[\\exp\\left(\\frac{\\|\\xi_t\\|^2}{A\\overline{\\Delta}_t+B\\|\\overline{g}_t\\|^2+C}\\right)\\right] \\leq \\text{e}.\n  \\end{align*}\n  Therefore, given any $l\\in [T]$, applying Lemma \\ref{lemma high probability}, we have that for any $\\lambda>0$, with probability at least $1-\\delta$,\n  \\begin{align*}\n    \\sumlimits{t=1}{l}Z_t\\leq &\\frac{3\\lambda}{4}\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2\\left(A\\overline{\\Delta}_t+B\\|\\overline{g}_t\\|^2+C\\right)+\\frac{1}{\\lambda}\\log\\frac{1}{\\delta}\\notag\\\\\n    \\leq &\\frac{3\\lambda}{4}\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2 P_t^2+\\frac{1}{\\lambda}\\log\\frac{1}{\\delta},\n  \\end{align*}\n  where the last inequality follow from Lemma \\ref{lemma 6.2} and $P_t$ defined in \\eqref{P_t}.\n For any fixed $\\lambda$, we can  re-scale over $\\delta$ and have that  with probability at least $1-\\delta$, for all $l\\in[T]$,\n \\begin{align}\n  \\label{4.14}\n  \\sumlimits{t=1}{l}-\\la\\overline{g}_t,\\xi_t\\ra\\leq \\frac{3\\lambda}{4}\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2 P_t^2+\\frac{1}{\\lambda}\\log\\frac{T}{\\delta}.\n \\end{align}\n Let $\\lambda=\\frac{1}{3 P_c^2}$, and we obtain the desired result.\n\\end{proof}","units":[{"start":60335,"end":60876,"start_line":672,"end_line":684,"raw_tex":"\\begin{proposition}[Proposition 5.2 in \\cite{kavis2022high}]\n  \\label{proposition 1}\n  Denote $\\alpha_t=\\frac{2}{t+1}$ and $\\varGamma_t=\\left(1-\\alpha_t\\right)\\varGamma_{t-1}$ with $\\varGamma_1=1$. We have that for all $t\\in[T],$\n  \\begin{align}\n    \\label{proposition 1.1}\n   \\varGamma_{t}\\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}=1,\n  \\end{align}\n  and \n  \\begin{align}\n    \\label{proposition 1.2}\n    \\left[\\sumlimits{k=t}{T}\\left(1-\\alpha_k\\right)\\varGamma_k\\right]\\frac{\\alpha_t}{\\varGamma_t}\\leq 2.\n  \\end{align}\n\\end{proposition}","sha256_utf8":"9b4b234fa7d6c03ef3f97f07b264259d654c201e28b311db034350b876babc33"},{"start":60980,"end":61603,"start_line":687,"end_line":698,"raw_tex":"\\begin{proposition}\n  \\label{proposition overline x_t - x_t}\n  Let $\\{x_t\\}_{t\\in[T]}$ and $\\{\\overline{x}_t\\}_{t\\in[T]}$ be generated by Algorithm \\ref{algorithm1}. We have\n  \\begin{align}\n    \\label{overline x_t-x_t}\n    \\overline{x}_t-x_t =\\left(1-\\alpha_t\\right)\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\frac{\\left(\\theta_k-\\gamma_k\\right)}{\\alpha_k}g_k,\n  \\end{align}\n  and\n  \\begin{align*}\n  \\|\\overline{x}_t-x_t\\|^2\\leq \\left(1-\\alpha_t\\right)\\varGamma_t\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\frac{\\left(\\theta_k-\\gamma_k\\right)^2}{\\alpha_k^2}\\|g_k\\|^2.\n\\end{align*}\n\\end{proposition}","sha256_utf8":"92d4e72657d85962db728dce0fed735a00a11ffd81c96f00feb4e7f0a28f8e69"},{"start":61604,"end":62421,"start_line":699,"end_line":716,"raw_tex":"\\begin{lemma}\n  \\label{high probability sum non-convex constant}\n  Given $T\\geq 1 $ and $\\delta\\in \\left(0,1\\right)$, if Assumptions \\ref{assumption 2} and \\ref{assumption 3} hold, then with probability at least $1-\\delta$, $\\forall l\\in[T]$,\n  \\begin{align}\n    \\label{inequality high probability non-convex constant}\n    \\sumlimits{t=1}{l}-\\la\\overline{g}_t,\\xi_t\\ra\\leq \\frac{1}{4}\\sumlimits{t=1}{l}\\frac{P_t^2}{P_c^2}\\|\\overline{g}_t\\|^2+3P_c^2\\log\\frac{T}{\\delta},\n  \\end{align}\n  where \n  \\begin{align}\n    \\label{P_t}\n    P_t=\\sqrt{A\\overline{\\Delta}_t+4B\\left(L_0\\overline{\\Delta}_t+L_1^2\\overline{\\Delta}_t^2\\right)+C},\n  \\end{align}\nand \n\\begin{align}\n  \\label{P}\n  P_c=\\sqrt{A\\functiongapboundconstant+4B\\left(L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2\\right)+C}.\n\\end{align}\n\\end{lemma}","sha256_utf8":"f305991c925bdf609ab3952fdc1bb70186eda4e6245d78eca9c48800c471cc57"},{"start":62537,"end":62966,"start_line":719,"end_line":726,"raw_tex":"\\begin{proposition}\n  \\label{constant bound}\n  Under the conditions of Theorem \\ref{theorem 1}, $\\overline{\\Delta}_t\\leq \\functiongapboundconstant,P_t\\leq P_c, \\forall t\\in[T]$, hold with probability at least $1-\\delta$, where  $P_t$, $P_c$ are in \\eqref{P_t}, \\eqref{P} and $\\functiongapboundconstant$ is defined as\n  \\begin{align}\n    \\label{Gconstant}\n    \\functiongapboundconstant=\\Gconstant.\n  \\end{align} \n\\end{proposition}","sha256_utf8":"33f2d89b307dea7ec57f97330fbca0f7d5ee3aa0d8d8f41cada63c9ca2c6a2a9"},{"start":62967,"end":74738,"start_line":727,"end_line":862,"raw_tex":"\\begin{proof}\n  We assume that \\eqref{inequality high probability non-convex constant} always happens and then deduce $\\overline{\\Delta}_t\\leq \\functiongapboundconstant$, $P_t\\leq P_c$ for all $t\\in[T]$. \n  Since \\eqref{inequality high probability non-convex constant} happens with probability at least $1-\\delta$, $\\overline{\\Delta}_t\\leq \\functiongapboundconstant$ and $P_t\\leq P_c$, $\\forall t\\in[T]$,  hold with probability at least $1-\\delta$. \n  It is obvious that $f(\\overline{x}_1)-f^*\\leq \\functiongapboundconstant$. Therefore, $P_1\\leq P_c$.\n  Suppose that for some $l\\in[T]$, $$f(\\overline{x}_t)-f^*\\leq \\functiongapboundconstant,\\quad\\forall t\\in[l], \\quad \\text{thus}, \\quad P_t\\leq P_c, \\quad \\forall t\\in[l].$$ \n  By the triangle inequality of the norm function,\n  \\begin{align*}\n    \\|g_t\\|\\leq  \\|g_t-\\overline{g}_t\\|+\\|\\overline{g}_t\\|\n    \\leq  \\sqrt{A\\overline{\\Delta}_t+B\\|\\overline{g}_t\\|^2+C}+\\|\\overline{g}_t\\|\n    \\leq  \\sqrt{A\\overline{\\Delta}_t}+\\left(\\sqrt{B}+1\\right)\\|\\overline{g}_t\\|+\\sqrt{C},\n  \\end{align*}\n  where the second inequality follows from Assumption \\ref{assumption 3} and the last inequality follows from Lemma \\ref{sqrt sum}.\n  Combining with Lemma \\ref{lemma 6.2} and the assumption that $\\overline{\\Delta}_t\\leq \\functiongapboundconstant, \\forall t\\in[l]$, we have\n  \\begin{align} \n    \\label{g_t}\n    \\|g_t\\| \\leq  \\sqrt{A\\functiongapboundconstant}+2\\left(\\sqrt{B}+1\\right)\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}+\\sqrt{C}, \\quad \\forall t\\in[l].\n  \\end{align}\n  For simplicity, let \n  \\begin{align}\n    \\label{Y_c}\n    Y_c=\\eta \\left(\\sqrt{A\\functiongapboundconstant}+2\\left(\\sqrt{B}+1\\right)\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}+\\sqrt{C}\\right).\n  \\end{align}\n    By the iteration step of Algorithm \\ref{algorithm1}, we have\n  \\begin{align*}\n  \\|x_{t+1}-x_t\\|=\\theta_t\\|g_t\\|=\\eta \\|g_t\\|\\leq Y_c\\leq 1/8L_1,\n  \\end{align*}\n  where the last inequality follows from the restriction of $\\eta$.\n Applying Lemma \\ref{descent lemma}, we have \n\\begin{align*}\n  & f(x_{t+1})-f(x_t) \\notag\\\\\n  \\leq & \\la \\nabla f(x_t),x_{t+1}-x_t\\ra+\\frac{L_0+L_1\\nablaf{x_t}}{2}\\|x_{t+1}-x_t\\|^2\\notag\\\\\n   = & -\\theta_t\\la \\nabla f(x_t),g_t\\ra+\\frac{L_0+L_1\\nablaf{x_t}}{2}\\theta_t^2\\|g_t\\|^2\\notag\\\\\n   = & -\\theta_t\\la \\overline{g}_t+\\nabla f(x_t)-\\overline{g}_t,\\overline{g}_t+\\xi_t\\ra + \\frac{L_0+L_1\\nablaf{x_t}}{2}\\theta_t^2\\|g_t\\|^2\\notag\\\\\n   = & -\\theta_t \\|\\overline{g}_t\\|^2-\\theta_t\\la \\overline{g}_t,\\xi_t\\ra -\\theta_t \\la \\nabla f(x_t)-\\overline{g}_t,g_t\\ra+\\frac{L_0+L_1\\nablaf{x_t}}{2}\\theta_t^2\\|g_t\\|^2\\notag\\\\\n  \\leq & -\\theta_t \\|\\overline{g}_t\\|^2-\\theta_t\\la \\overline{g}_t,\\xi_t\\ra +\\theta_t \\| \\nabla f(x_t)-\\overline{g}_t\\|\\|g_t\\|+\\frac{L_0+L_1\\nablaf{x_t}}{2}\\theta_t^2\\|g_t\\|^2,\n\\end{align*} \nwhere the first equation follows from the update rule in Algorithm \\ref{algorithm1} and the last line follows from Cauchy-Schwarz inequality. \nRecall the constraints of the step size for RSAG with the constant step size and SGD, we have \n\\begin{align}\n  \\label{constant step size constraint}\n  0\\leq \\frac{\\gamma_t-\\theta_t}{\\alpha_t}\\leq \\theta_t=\\eta, \\quad \\forall t\\in[T].\n\\end{align}\nCombining \\eqref{overline x_t-x_t} with \\eqref{proposition 1.1}, \\eqref{g_t}, \\eqref{constant step size constraint}, we have\n\\begin{align}\n  \\label{bound overline x_t-x_t}\n  \\|\\overline{x}_t-x_t\\| \\leq & \\eta\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\|g_k\\|\\notag\\\\\n  \\leq & \\eta\\left(\\sqrt{A\\functiongapboundconstant}+2\\left(\\sqrt{B}+1\\right)\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}+\\sqrt{C}\\right)\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\notag\\\\\n  = & Y_c \\leq 1/8L_1.\n\\end{align}\nNote that \\eqref{bound overline x_t-x_t} holds for all $t\\in[l+1]$. \nThus, applying Definition \\ref{definition 1}, we have that for all $t\\in[l]$,\n\\begin{align}  \n  \\label{inequality base}\n  & f(x_{t+1})-f(x_t) \\notag\\\\\n  \\leq & -\\theta_t \\|\\overline{g}_t\\|^2-\\theta_t\\la \\overline{g}_t,\\xi_t\\ra +\\theta_t\\left(L_0+L_1\\nablaf{x_t}\\right)\\|\\overline{x}_t-x_t\\|\\|g_t\\| +\\frac{L_0+L_1\\nablaf{x_t}}{2}\\theta_t^2\\|g_t\\|^2\\notag\\\\\n  \\leq & -\\theta_t\\|\\overline{g}_t\\|^2-\\theta_t\\la\\overline{g}_t,\\xi_t\\ra+\\frac{1}{2}\\left(L_0+L_1\\nablaf{x_t}\\right)\\|\\overline{x}_t-x_t\\|^2+\\left(L_0+L_1\\nablaf{x_t}\\right)\\theta_t^2\\|g_t\\|^2\\notag\\\\\n  \\leq & -\\theta_t\\|\\overline{g}_t\\|^2-\\theta_t\\la\\overline{g}_t,\\xi_t\\ra+\\frac{1}{2}\\left(L_0+L_1\\nablaf{x_t}\\right)\\left(1-\\alpha_t\\right)\\varGamma_t\\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}\\theta_k^2\\|g_k\\|^2\\notag\\\\\n  & + \\left(L_0+L_1\\nablaf{x_t}\\right)\\theta_t^2\\|g_t\\|^2,\n\\end{align}\nwhere we use Young's inequality in the second inequality and apply Proposition \\ref{proposition overline x_t - x_t} with \\eqref{constant step size constraint} in the last inequality. \nSumming over $t\\in [l]$ with $\\theta_t=\\eta$, we have\n\\begin{align}\n\\label{telescope}\nf(x_{l+1})-f(x_1)  \\leq& \\frac{1}{2}\\sum\\limits^{l}_{t=1}\\left[\\lsmooth{t}\\left(1-\\alpha_t\\right)\\varGamma_t\\sum\\limits_{k=1}^t\\frac{\\alpha_k}{\\varGamma_k}\\eta^2\\|g_k\\|^2\\right]\\notag\\\\\n&+\\eta^2\\sumlimits{t=1}{l}\\lsmooth{t}\\|g_t\\|^2-\\eta\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2-\\eta\\sumlimits{t=1}{l}\\la \\overline{g}_t,\\xi_t\\ra\\notag\\\\\n=& \\frac{1}{2}\\eta^2 \\sumlimits{t=1}{l}\\left[\\sumlimits{k=t}{l}\\lsmooth{k}\\left(1-\\alpha_k\\right)\\varGamma_k\\right]\\frac{\\alpha_t}{\\varGamma_t}\\|g_t\\|^2\\notag\\\\\n&+\\eta^2\\sumlimits{t=1}{l}\\lsmooth{t}\\|g_t\\|^2-\\eta\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2-\\eta\\sumlimits{t=1}{l}\\la \\overline{g}_t,\\xi_t\\ra.\n\\end{align}\nApplying the triangle inequality of $\\|\\cdot\\|$, we have that for all $t\\in[l]$,\n\\begin{align}\n  \\label{nabla fx_t}\n  \\nablaf{x_t} \\leq  \\|\\overline{g}_t-\\nabla f(x_t)\\|+\\|\\overline{g}_t\\| \n  \\leq  \\left(L_0+L_1\\|\\overline{g}_t\\|\\right)\\|\\overline{x}_t-x_t\\|+\\|\\overline{g}_t\\|,\n\\end{align}\nwhere the second inequality holds since $\\|\\overline{x}_t-x_t\\|\\leq 1/L_1$. Combining \\eqref{bound overline x_t-x_t} and \\eqref{nabla fx_t}, for all $t\\in[l]$,\n\\begin{align}\n  \\label{nabla fx_t constant step size}\n   \\nablaf{x_t}\n  \\leq & \\frac{1}{8L_1}\\left(L_0+2L_1\\sqrt{L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2}\\right)+2\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}\\notag\\\\\n  = & \\frac{L_0}{8L_1}+\\frac{9}{4} \\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}=\\mathcal{M}_c,\n\\end{align}\nwhere we apply Lemma \\ref{lemma 6.2} to the inequality. \nApplying the triangle inequality again, we have\n\\begin{align}\n  \\label{triangle square}\n  \\|g_t\\|^2\\leq 2\\|g_t-\\overline{g}_t\\|^2+2\\|\\overline{g}_t\\|^2\\leq 2\\left(A \\overline{\\Delta}_t+\\left(B+1\\right)\\|\\overline{g}_t\\|^2+C\\right).\n\\end{align}\nCombining  \\eqref{proposition 1.2}, \\eqref{inequality high probability non-convex constant}, \\eqref{telescope} and \\eqref{triangle square}, together with the bound of $\\nablaf{x_t}$ in \\eqref{nabla fx_t constant step size}, we have \n\\begin{align}\n  \\label{4.26}\nf(x_{l+1})-f(x_1)  \\leq & 2\\left(L_0+L_1\\mathcal{M}_c\\right)\\eta^2\\sumlimits{t=1}{l}\\|g_t\\|^2-\\eta\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2-\\eta\\sumlimits{t=1}{l}\\la\\overline{g}_t,\\xi_t\\ra\\notag\\\\\n\\leq & 4\\left(L_0+L_1\\mathcal{M}_c\\right)\\eta^2\\sumlimits{t=1}{l}\\left(A\\overline{\\Delta}_t+\\left(B+1\\right)\\|\\overline{g}_t\\|^2+C\\right)-\\eta\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2\\notag\\\\\n&+\\frac{1}{4}\\eta\\sumlimits{t=1}{l}\\frac{P_t^2}{P_c^2}\\|\\overline{g}_t\\|^2+3\\eta P_c^2\\log\\frac{T}{\\delta}\\notag\\\\\n\\leq & 4\\left(L_0+L_1\\mathcal{M}_c\\right)\\eta^2 lA\\functiongapboundconstant+4\\left(L_0+L_1\\mathcal{M}_c\\right)\\eta^2lC \\notag\\\\\n&+\\left(4\\left(L_0+L_1\\mathcal{M}_c\\right)\\eta\\left(B+1\\right)+\\frac{1}{4}-1\\right)\\eta\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2+3\\eta P_c^2\\log\\frac{T}{\\delta},\n\\end{align}\nwhere the last inequality holds since the assumption that $P_t\\leq P_c, \\forall t\\in[l]$. \nApplying the restrictions of $\\eta$ in \\eqref{eta constant non-convex}, it is apparent that \n\\begin{align*}\n  f(x_{l+1})\\leq f(x_1)+\\frac{1}{4}\\functiongapboundconstant+1+3\\eta P_c^2\\log\\frac{T}{\\delta}.\n\\end{align*}\nSince $P_c^2 = A\\functiongapboundconstant+4B\\left(L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2\\right)+C= \\left(A+4BL_0\\right)\\functiongapboundconstant+4BL_1^2\\functiongapboundconstant^2+C$,\n\\begin{align}\n  \\label{f x+1-f x_1 constant step size}\n  f(x_{l+1}) \\leq & f(x_1)+\\frac{1}{4}\\functiongapboundconstant+1+3\\eta \\left(A+4BL_0\\right)\\functiongapboundconstant\\log\\frac{T}{\\delta}+12\\eta BL_1^2\\functiongapboundconstant^2\\log\\frac{T}{\\delta}+3\\eta C \\log\\frac{T}{\\delta}\\notag\\\\\n  \\leq & f(x_1)+\\frac{1}{4}\\functiongapboundconstant+1+2\\sqrt{\\functiongapboundconstant}\\log\\frac{T}{\\delta}+3\\eta C\\log\\frac{T}{\\delta}     \\notag\\\\\n  \\leq & f(x_1)+\\frac{1}{2}\\functiongapboundconstant+1+ 4\\log^2\\frac{T}{\\delta}+3\\eta C \\log\\frac{T}{\\delta},\n\\end{align}\nwhere the second inequality is due to the restrictions of $\\eta$ in \\eqref{eta constant non-convex} and the last inequality follows from Young's inequality. Next we will bound the gap between $f(x_{l+1})$ and $f(\\overline{x}_{l+1})$. By \\eqref{bound overline x_t-x_t}, we have $\\|\\overline{x}_{l+1}-x_{l+1}\\|<1/L_1$. Using Lemma \\ref{descent lemma} again,\n\\begin{align}\n  \\label{gap between f overline x_t+1 and f x_t+1 constant step size}\n  f(\\overline{x}_{l+1}) \\leq & f(x_{l+1})+\\la \\overline{g}_{l+1}, \\overline{x}_{l+1}-x_{l+1}\\ra+\\frac{L_0+L_1\\|\\overline{g}_{l+1}\\|}{2}\\|\\overline{x}_{l+1}-x_{l+1}\\|^2\\notag\\\\\n  \\leq & f(x_{l+1})+\\|\\overline{g}_{l+1}\\|\\|\\overline{x}_{l+1}-x_{l+1}\\|+\\frac{L_0+L_1\\|\\overline{g}_{l+1}\\|}{2}\\|\\overline{x}_{l+1}-x_{l+1}\\|^2,\n\\end{align}\nwhere the second inequality follows from Cauchy-Schwarz inequality. Substituting \\eqref{bound overline x_t-x_t} into \\eqref{gap between f overline x_t+1 and f x_t+1 constant step size}, we have\n\\begin{align}\n  \\label{descent lemma 2}\n  f(\\overline{x}_{l+1}) \\leq & f(x_{l+1})+\\frac{1}{8L_1}\\|\\overline{g}_{l+1}\\|+\\frac{1}{128L_1^2}\\left(L_0+L_1\\|\\overline{g}_{l+1}\\|\\right)\\notag\\\\\n  = & f(x_{l+1})+\\frac{17}{128L_1}\\|\\overline{g}_{l+1}\\|+\\frac{L_0}{128L_1^2}.\n\\end{align}\nCombining \\eqref{f x+1-f x_1 constant step size} and \\eqref{descent lemma 2}, and subtracting $f^*$ from both sides, we have\n\\begin{align}\n  \\overline{\\Delta}_{l+1}\\leq & \\overline{\\Delta}_1+\\frac{1}{2}\\functiongapboundconstant+1+4\\log^2\\frac{T}{\\delta}+3\\eta C \\log\\frac{T}{\\delta} +\\frac{17}{128L_1}\\|\\overline{g}_{l+1}\\|+\\frac{L_0}{128L_1^2}\\notag\\\\\n  \\leq & \\overline{\\Delta}_1+\\frac{1}{2}\\functiongapboundconstant+1+4\\log^2\\frac{T}{\\delta}+3\\eta C \\log\\frac{T}{\\delta} + \\frac{17}{64L_1}\\left(\\sqrt{L_0\\overline{\\Delta}_{l+1}}+L_1\\overline{\\Delta}_{l+1}\\right)+  \\frac{L_0}{128L_1^2}\\notag\\\\\n  \\leq &   \\overline{\\Delta}_1+\\frac{1}{2}\\functiongapboundconstant+1+4\\log^2\\frac{T}{\\delta}+\\frac{\\sqrt{2C}}{\\sqrt{L_0T}}\\log\\frac{T}{\\delta} + \\frac{7}{64}\\overline{\\Delta}_{l+1}+\\frac{289L_0}{1792L_1^2}+\\frac{17}{64}\\overline{\\Delta}_{l+1}+\\frac{L_0}{128L_1^2}.\n\\end{align}\nThe second inequality follows from Lemma \\ref{lemma 6.2} and Lemma \\ref{sqrt sum}. The third inequality follows from Young's inequality and the constraint of $\\eta$ that $\\eta\\leq \\frac{1}{2\\sqrt{\\left(L_0+L_1\\mathcal{M}_c\\right)CT}}$, where $\\mathcal{M}_c\\geq \\frac{L_0}{8L_1}$.\nRe-arranging the above inequality, we have \n\\begin{align*}\n  \\frac{5}{8}\\overline{\\Delta}_{l+1}\\leq \\overline{\\Delta}_1+\\frac{1}{2}\\functiongapboundconstant+1+4\\log^2\\frac{T}{\\delta}+\\frac{\\sqrt{2C}}{\\sqrt{L_0T}}+\\frac{303L_0}{1792L_1^2}\\leq \\frac{5}{8}\\functiongapboundconstant.\n\\end{align*}\nThe second inequality holds since \n$\\functiongapboundconstant=\\Gconstant$.\nTherefore, $P_{l+1}\\leq P_c$ also holds. \nSo the induction is complete and we obtain the desired result. \n\\end{proof}","sha256_utf8":"1b090d59789ef66835a3540a4985f4bbeae4eccab1cb7703f404b7e6719a3f9d"},{"start":74876,"end":79242,"start_line":864,"end_line":903,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{theorem 1}]\nWe assume \\eqref{inequality high probability non-convex constant} always holds then we deduce \\eqref{4.1}. Note that \\eqref{inequality high probability non-convex constant} holds with probability at least $1-\\delta$. Therefore, \\eqref{4.1} holds with probability at least $1-\\delta$.\nRecalling \\eqref{inequality base} and dividing $\\theta_t$ on both sides, we have  \n\\begin{align*}\n  \\|\\overline{g}_t\\|^2\\leq &\\frac{f(x_t)-f(x_{t+1})}{\\theta_t}+\\frac{1}{2\\theta_t}\\left(L_0+L_1\\nablaf{x_t}\\right)\\left(1-\\alpha_t\\right)\\varGamma_t\\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}\\theta_k^2\\|g_k\\|^2\\notag\\\\\n&+\\left(L_0+L_1\\nablaf{x_t}\\right)\\theta_t\\|g_t\\|^2-\\la\\overline{g}_t,\\xi_t\\ra.\n\\end{align*}\nSumming over $t\\in[T]$ with $\\theta_t=\\eta$,\n\\begin{align}\n  \\label{constant gt}\n  \\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2\\leq & \\frac{1}{\\eta}\\sumlimits{t=1}{T}\\left(f(x_t)-f(x_{t+1})\\right)+\\frac{\\eta}{2}\\sumlimits{t=1}{T}\\left[\\left(L_0+L_1\\nablaf{x_t}\\right)\\left(1-\\alpha_t\\right)\\varGamma_t\\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}\\|g_k\\|^2\\right]\\notag\\\\\n  &+\\eta\\sumlimits{t=1}{T}\\left(L_0+L_1\\nablaf{x_t}\\right)\\|g_t\\|^2-\\sumlimits{t=1}{T}\\la \\overline{g}_t,\\xi_t\\ra\\notag\\\\\n  \\leq &\\frac{\\overline{\\Delta}_1}{\\eta}+\\frac{\\eta}{2} \\sumlimits{t=1}{T}\\left[\\sumlimits{k=t}{T}\\lsmooth{k}\\left(1-\\alpha_k\\right)\\varGamma_k\\right]\\frac{\\alpha_t}{\\varGamma_t}\\|g_t\\|^2\\notag\\\\\n  & +\\eta\\sumlimits{t=1}{T}\\left(L_0+L_1\\nablaf{x_t}\\right)\\|g_t\\|^2-\\sumlimits{t=1}{T}\\la \\overline{g}_t,\\xi_t\\ra.\n\\end{align}\nApplying Lemma \\ref{high probability sum non-convex constant},\n\\begin{align*}\n  \\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2\\leq &\\frac{\\overline{\\Delta}_1}{\\eta}+\\frac{\\eta}{2} \\sumlimits{t=1}{T}\\left[\\sumlimits{k=t}{T}\\lsmooth{k}\\left(1-\\alpha_k\\right)\\varGamma_k\\right]\\frac{\\alpha_t}{\\varGamma_t}\\|g_t\\|^2\\notag\\\\\n  & +\\eta\\sumlimits{t=1}{T}\\left(L_0+L_1\\nablaf{x_t}\\right)\\|g_t\\|^2+\\frac{1}{4P_c^2}\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2 P_t^2+3P_c^2\\log\\frac{T}{\\delta}.\n\\end{align*}\nCombining with \\eqref{proposition 1.2}, \\eqref{nabla fx_t constant step size}, \\eqref{triangle square} and Proposition \\ref{constant bound}, we have\n\\begin{align*}\n  \\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2 \\leq & \\frac{\\overline{\\Delta}_1}{\\eta}+2\\eta\\sumlimits{t=1}{T}\\left(L_0+L_1\\mathcal{M}_c\\right)\\|g_t\\|^2+\\frac{1}{4}\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2+3P_c^2\\log\\frac{T}{\\delta}\\notag\\\\\n  \\leq & \\frac{\\overline{\\Delta}_1}{\\eta}+4\\eta\\left(L_0+L_1\\mathcal{M}_c\\right) T \\left(A\\functiongapboundconstant+C\\right)+4\\left(L_0+L_1\\mathcal{M}_c\\right)\\left(B+1\\right)\\eta\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2\\notag\\\\\n  & + \\frac{1}{4}\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2+3\\left(A\\functiongapboundconstant + 4B\\left(L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2\\right)+C\\right)\\log\\frac{T}{\\delta}.\n\\end{align*}\nRe-arranging the above inequality with the constraint of $\\eta$ in \\eqref{eta constant non-convex}, we obtain \n\\begin{align*}\n  \\frac{1}{T}\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2\\leq & \\frac{4\\overline{\\Delta}_1}{\\eta T}+ \\frac{4}{\\sqrt{T}}\\sqrt{L_0+L_1\\mathcal{M}_c}\\left(\\sqrt{A\\functiongapboundconstant^2}+2\\sqrt{C}\\right)\\notag\\\\\n  &+\\frac{12}{T}\\left(A\\functiongapboundconstant + 4B\\left(L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2\\right)+C\\right)\\log\\frac{T}{\\delta}.\n\\end{align*}\nCombining with the restrictions of $\\eta$ in \\eqref{eta constant non-convex} again, we get the final result,\n\\begin{align}\n  \\label{5.34}\n  \\frac{1}{T}\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2\\leq & \\frac{4}{\\sqrt{T}}\\sqrt{L_0+L_1\\mathcal{M}_c}\\left(\\sqrt{A}\\left(\\functiongapboundconstant+4\\overline{\\Delta}_1\\right)+2\\sqrt{C}\\left(1+\\overline{\\Delta}_1\\right)\\right)\\notag\\\\\n  &+ \\frac{32\\overline{\\Delta}_1}{T}\\left(L_0+L_1\\mathcal{M}_c\\right)\\left(B+1\\right) +  \\frac{12\\overline{\\Delta}_1}{T}\\left(\\left(A+4BL_0\\right)\\functiongapboundconstant^{1/2}+4BL_1^2\\functiongapboundconstant^{3/2}\\right) \\notag\\\\\n  &+ \\frac{32\\overline{\\Delta}_1}{T}L_1 \\left(\\sqrt{A\\functiongapboundconstant}+2\\left(\\sqrt{B}+1\\right)\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}+\\sqrt{C}\\right) \\notag\\\\\n  &+\\frac{12}{T}\\left(A\\functiongapboundconstant + 4B\\left(L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2\\right)+C\\right)\\log\\frac{T}{\\delta}.\n\\end{align}\n\\end{proof}","sha256_utf8":"ca4c6790f25f4c443d0522acbfa916372b5ba003738a89965374cef17be99fac"},{"start":173853,"end":174186,"start_line":2109,"end_line":2113,"raw_tex":"\\begin{lemma}[Remark 2.3 in \\cite{zhang2020improved}]\n  \\label{descent lemma}\n  Suppose that $f:\\mathbb{R}^d \\rightarrow \\mathbb{R}$ is an $(L_0,L_1)$-generalized smooth function. When $\\|x-y\\|\\leq 1/L_1$, we have \n  $$\\left\\lvert f(y)- f(x)-\\la \\nabla f(x),y-x\\ra\\right\\rvert \\leq \\frac{L_0+L_1\\nablaf{x}}{2}\\|x-y\\|^2.$$\n\\end{lemma}","sha256_utf8":"92d6432dfc587318eb50e683be34c3f83e5ecf5da83b8ed1b961d9399277476c"},{"start":174331,"end":174978,"start_line":2116,"end_line":2120,"raw_tex":"\\begin{lemma}[Lemma 1 in \\cite{li2020high}]\n  \\label{lemma high probability}\n  Assume that $\\{Z_t\\}_{t\\in[T]}$ is a martingale difference sequence with respect to $\\gamma_1, \\gamma_2, \\cdots, \\gamma_T$ and $\\mathbb{E}_t[\\exp\\left(Z_t^2/\\sigma_t^2\\right)]\\leq \\exp(1)$ for all $1 \\leq t \\leq T$, where $\\sigma_t$ is a sequence of measurable random variables with respect to $\\gamma_1, \\gamma_2, \\cdots, \\gamma_{t-1}$. Then, for any fixed $\\lambda >0$ and $\\delta\\in(0,1)$, with probability at least $1-\\delta$, we have\n  $$\\sumlimits{t=1}{T}Z_t\\leq \\frac{3\\lambda}{4}\\sumlimits{t=1}{T}\\sigma_t^2+\\frac{1}{\\lambda}\\log\\frac{1}{\\delta}.$$\n\\end{lemma}","sha256_utf8":"527ba6a30522eb9d78b966832366b8838adac27efce793544a71a232d0d28425"},{"start":175792,"end":176119,"start_line":2135,"end_line":2140,"raw_tex":"\\begin{lemma}\n  \\label{lemma 6.2}  \n  Let $f(x) : \\mathbb{R}^d\\rightarrow \\mathbb{R}$ be an $(L_0,L_1)$-smooth function with minimum $f^*$. \n  Then, $$\\|\\nabla f(x_t)\\|^2\\leq 2\\left(L_0+L_1\\|\\nabla f(x_t)\\|\\right)\\Delta_t,$$ and $$\\nablaf{x_t}^2\\leq 4L_0\\Delta_t+4L_1^2\\Delta_t^2,$$\n  where $\\Delta_t=f(x_t)-f^*$.\n  \\end{lemma}","sha256_utf8":"9bfc5c39bea72f7daae6f6430eba1baf23b91f2ffe85c38076c3c8e2f712ca5b"},{"start":176122,"end":176846,"start_line":2141,"end_line":2155,"raw_tex":"\\begin{proof}\n    Let $x=x_t-\\frac{1}{L_0+L_1\\|\\nabla f(x_t)\\|}\\nabla f(x_t)$.\n    It is easy to verify that $\\|x-x_t\\|\\leq 1/L_1$.\n    From Lemma \\ref{descent lemma}, we have that\n    $$f(x) \\leq f(x_t)  + \\la \\nabla f(x_t), x - x_t \\ra + {L_0+L_1 \\|\\nabla f(x)\\|  \\over 2} \\|x - x_t\\|^2.$$\n    Hence, \n    \\begin{align*}\n      &\\frac{\\|\\nabla f(x_t)\\|^2}{2\\left(L_0+L_1\\|\\nabla f(x_t)\\|\\right)}\\leq f(x_t)-f(x)\\leq \\Delta_t.\n    \\end{align*}\n  Re-arranging the above inequality and applying Cauchy-Schwarz inequality and Young's inequality, we have\n  \\begin{align*}\n    \\nablaf{x_{t}}^2 \\leq & 2L_0\\Delta_{t}+\\frac{1}{2}\\|\\nabla f(x_{t})\\|^2+2L_1^2\\Delta_{t}^2.\n  \\end{align*}\n  We obtain the desired result.\n  \\end{proof}","sha256_utf8":"8168f2acfe09b7f2db5ba004dad79fd19615b9d751a119ccc68e3be66f7a3d44"},{"start":177250,"end":177478,"start_line":2168,"end_line":2174,"raw_tex":"\\begin{lemma}\n    \\label{sqrt sum}\n    Let $\\{a_t\\}_{t\\in[n]}$ be a sequence of non-negative real numbers. We have\n    \\begin{align*}\n      \\sqrt{\\sum_{i=1}^n a_i}\\leq \\sumlimits{i=1}{n}\\sqrt{a_i}.\n    \\end{align*}\n  \\end{lemma}","sha256_utf8":"1a4c86dc1978bfe25a6b7434a13be92ba54ae5ae1b55a3d310df63dc96e973d1"},{"start":193290,"end":194501,"start_line":2396,"end_line":2415,"raw_tex":"\\begin{proof}[Proof of Proposition \\ref{proposition 1}]\n  From \\cite{ghadimi2016accelerated}, we have\n  \\begin{align*}\n    \\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}=\\frac{\\alpha_1}{\\varGamma_1}+\\sumlimits{k=2}{t}\\frac{\\alpha_k}{\\varGamma_k}=\\frac{1}{\\varGamma_1}+\\sumlimits{k=2}{t}\\frac{1}{\\varGamma_k}\\left(1-\\frac{\\varGamma_{k}}{\\varGamma_{k-1}}\\right)=\\frac{1}{\\varGamma_1}+\\sumlimits{k=2}{t}\\left(\\frac{1}{\\varGamma_k}-\\frac{1}{\\varGamma_{k-1}}\\right)=\\frac{1}{\\varGamma_t}.\n  \\end{align*}\n  Therefore, \n  \\begin{align*}\n    \\varGamma_{t}\\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}=1.\n  \\end{align*}\n  For any $t\\geq 2$, we have\n \\begin{align*}\n  \\varGamma_t=\\left(1-\\alpha_t\\right)\\varGamma_{t-1}=\\frac{t-1}{t+1}\\varGamma_{t-1}=\\cdots=\\frac{t-1}{t+1}\\cdot\\frac{t-2}{t}\\cdots \\frac{2-1}{2+1}=\\frac{2}{t\\left(t+1\\right)}.\n\\end{align*}\nHence,\n  \\begin{align*}\n    \\left[\\sumlimits{k=t}{T}\\left(1-\\alpha_k\\right)\\varGamma_k\\right]\\frac{\\alpha_t}{\\varGamma_t} & =\\left[\\sumlimits{k=t}{T}\\frac{k-1}{k+1}\\frac{2}{k\\left(k+1\\right)}\\right]\\cdot t\\notag\\\\\n    \\leq & 2t \\sumlimits{k=t}{T}\\frac{1}{k\\left(k+1\\right)}\\notag\\\\\n    = & 2t\\left(\\frac{1}{t}-\\frac{1}{T+1}\\right) \\leq 2.\n  \\end{align*}\n\\end{proof}","sha256_utf8":"91464696456c79cdc59678f1c05b78476b7e76dc860e604611dbd5f469e052d7"},{"start":194503,"end":196202,"start_line":2417,"end_line":2434,"raw_tex":"\\begin{proof}[Proof of Proposition \\ref{proposition overline x_t - x_t}]\n  Following from the iteration steps in Algorithm \\ref{algorithm1} and referring to the proof in \\cite{ghadimi2016accelerated}, we have \n  \\begin{align*}\n    \\overline{x}_t-x_t = & \\left(1-\\alpha_t\\right)[\\tilde{x}_t-x_t]\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)[\\overline{x}_{t-1}-x_{t-1}+\\left(\\theta_{t-1}-\\gamma_{t-1}\\right)g_{t-1}]\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)[\\left(1-\\alpha_{t-1}\\right)\\left(\\tilde{x}_{t-1}-x_{t-1}\\right)+\\left(\\theta_{t-1}-\\gamma_{t-1}\\right)g_{t-1}]\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)\\sumlimits{k=1}{t-1}\\left(\\prod\\limits_{j=k+1}^{t-1}\\left(1-\\alpha_j\\right)\\right)\\left(\\theta_k-\\gamma_k\\right)g_k\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)\\sumlimits{k=1}{t-1}\\frac{\\varGamma_{t-1}}{\\varGamma_{k}}\\left(\\theta_k-\\gamma_k\\right)g_k\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\frac{\\left(\\theta_k-\\gamma_k\\right)}{\\alpha_k}g_k.\n  \\end{align*}\n  By the convexity of square of the norm and \\eqref{proposition 1.1}, we have \n  \\begin{align}\n    \\label{overline x_t-x_t square norm}\n    \\|\\overline{x}_t-x_t\\|^2 = & \\left\\Vert\\left(1-\\alpha_t\\right)\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k} \\frac{\\left(\\theta_k-\\gamma_k\\right)}{\\alpha_k}  g_k\\right\\Vert^2\\notag\\\\\n    \\leq & \\left(1-\\alpha_t\\right)^2\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\frac{\\left(\\theta_k-\\gamma_k\\right)^2}{\\alpha_k^2}\\|g_k\\|^2\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)\\varGamma_t\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\frac{\\left(\\theta_k-\\gamma_k\\right)^2}{\\alpha_k^2}\\|g_k\\|^2.\n  \\end{align}\n\\end{proof}","sha256_utf8":"f60684d600a00876aef4a5d63c8a73da219c20255780ed4f3138db878d5c3352"},{"start":196206,"end":197977,"start_line":2438,"end_line":2458,"raw_tex":"\\begin{proof}[Proof of Lemma \\ref{high probability sum non-convex constant}]\n  Let $Z_t=-\\la \\overline{g}_t,\\xi_t\\ra$. \n  Note that $\\overline{g}_t$ is a random variable dependent on $z_1,\\cdots,z_{t-1}$ and $\\xi_t$ is dependent on $z_1,\\cdots,z_t$. Therefore, it is apparent that $Z_t$ is a martingale difference sequence since\n  $$\\mathbb{E}\\left[- \\la \\overline{g}_t,\\xi_t\\ra|z_1,\\cdots,z_{t-1}\\right]=-\\la \\overline{g}_t,\\mathbb{E}_t[\\xi_t]\\ra=0.$$\n  Using Cauchy-Schwarz inequality,\n  \\begin{align*}\n    \\mathbb{E}_t\\left[\\exp\\left(\\frac{Z_t^2}{\\|\\overline{g}_t\\|^2\\left(A\\overline{\\Delta}_t+B\\|\\overline{g}_t\\|^2+C\\right)}\\right)\\right]\\leq\\mathbb{E}_t\\left[\\exp\\left(\\frac{\\|\\xi_t\\|^2}{A\\overline{\\Delta}_t+B\\|\\overline{g}_t\\|^2+C}\\right)\\right] \\leq \\text{e}.\n  \\end{align*}\n  Therefore, given any $l\\in [T]$, applying Lemma \\ref{lemma high probability}, we have that for any $\\lambda>0$, with probability at least $1-\\delta$,\n  \\begin{align*}\n    \\sumlimits{t=1}{l}Z_t\\leq &\\frac{3\\lambda}{4}\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2\\left(A\\overline{\\Delta}_t+B\\|\\overline{g}_t\\|^2+C\\right)+\\frac{1}{\\lambda}\\log\\frac{1}{\\delta}\\notag\\\\\n    \\leq &\\frac{3\\lambda}{4}\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2 P_t^2+\\frac{1}{\\lambda}\\log\\frac{1}{\\delta},\n  \\end{align*}\n  where the last inequality follow from Lemma \\ref{lemma 6.2} and $P_t$ defined in \\eqref{P_t}.\n For any fixed $\\lambda$, we can  re-scale over $\\delta$ and have that  with probability at least $1-\\delta$, for all $l\\in[T]$,\n \\begin{align}\n  \\label{4.14}\n  \\sumlimits{t=1}{l}-\\la\\overline{g}_t,\\xi_t\\ra\\leq \\frac{3\\lambda}{4}\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2 P_t^2+\\frac{1}{\\lambda}\\log\\frac{T}{\\delta}.\n \\end{align}\n Let $\\lambda=\\frac{1}{3 P_c^2}$, and we obtain the desired result.\n\\end{proof}","sha256_utf8":"44718ceaa37f544290d23ec24e0ff3ce716c0567f2d44dda3c41060cf9d27a8c"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"常步长随机加速法的高概率收敛","source_units":[{"start":35438,"end":37754,"start_line":440,"end_line":462,"raw_tex":"\\begin{theorem}\n  \\label{theorem 1}\n  Let  $T>0$ and $\\delta\\in(0,1)$. Suppose that $f$ is an $\\left(L_0,L_1\\right)$-smooth function, $\\{\\overline{x}_t\\}_{x\\in[T]}$ is a sequence generated by Algorithm \\ref{algorithm1} with $\\alpha_t=\\frac{2}{t+1},\\gamma_t=\\left(1+\\alpha_t\\right)\\theta_t$ or $\\gamma_t=\\theta_t$ (SGD), $\\forall t\\in[T]$, and the step size $\\theta_t$ satisfies\n  \\begin{align}\n    \\label{eta constant non-convex}\n    \\theta_t=\\eta = \\min\\left\\{\\frac{1}{\\mathcal{A}\\sqrt{T}},\\frac{1}{\\mathcal{B}},\\frac{1}{\\mathcal{C}\\sqrt{T}}, \\frac{1}{\\mathcal{G}},\\frac{\\functiongapboundconstant^{-1/2}}{3\\left(A+4BL_0\\right)}, \\frac{\\functiongapboundconstant^{-3/2}}{12BL_1^2}\\right\\},\n  \\end{align}\n    where $\\mathcal{A}, \\mathcal{B}, \\mathcal{C}, \\mathcal{G}$ are defined as\n  \\begin{align}\n    \\label{theorem 1 eta}\n    \\mathcal{A}=4\\sqrt{\\left(L_0+L_1\\mathcal{M}_c\\right)A}, \\qquad &\\mathcal{B}=8\\left(L_0+L_1\\mathcal{M}_c\\right)\\left(B+1\\right), \\notag\\\\\n    \\mathcal{C}=2\\sqrt{\\left(L_0+L_1\\mathcal{M}_c\\right)C},\\qquad &\\mathcal{G}=8L_1 \\left(\\sqrt{A\\functiongapboundconstant}+2\\left(\\sqrt{B}+1\\right)\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}+\\sqrt{C}\\right),  \n  \\end{align}\n     and $\\functiongapboundconstant$, $\\mathcal{M}_c$ are given in the following order\\footnote{The explicit expressions of $\\functiongapboundconstant$ and $\\mathcal{M}_c$ are in \\eqref{Gconstant} and \\eqref{nabla fx_t constant step size}, respectively.},\n  \\begin{align*} \n  \\functiongapboundconstant \\sim \\mathcal{O}\\left(\\overline{\\Delta}_1+\\log^2\\frac{T}{\\delta}+\\frac{\\sqrt{C}}{\\sqrt{L_0T}}\\log\\frac{T}{\\delta}+\\frac{L_0}{L_1^2 }\\right), \\qquad  \\mathcal{M}_c \\sim \\mathcal{O}\\left(L_1\\functiongapboundconstant\\right).\n  \\end{align*}\n  Under Assumptions \\ref{assumption 1}, \\ref{assumption 2} and \\ref{assumption 3}, with probability at least $1-\\delta$, we have\\footnote{The detail convergence rate is stated in \\eqref{5.34}.}\n  \\begin{align}\n    \\label{4.1}\n    \\frac{1}{T}\\sumlimits{t=1}{T}\\|\\nabla f(\\overline{x}_t)\\|^2\\leq & \\mathcal{O}\\left(\\frac{\\functiongapboundconstant\\left(\\functiongapboundconstant B\\log \\frac{T}{\\delta}+\\sqrt{B+1}\\right)}{T}+\\frac{\\sqrt{\\functiongapboundconstant}\\left(\\functiongapboundconstant\\sqrt{A}+\\sqrt{C}\\right)}{\\sqrt{T}}\\right).\n  \\end{align}\n\\end{theorem}","sha256_utf8":"4640427d09d604749c34ab54f5c037d2bacc44c1e963e8ed8bb08515e3ece30d"}]},{"id":"S002","semantic_type":"problem_setting","kind":"problem_setting","title":"随机优化问题","source_units":[{"start":4228,"end":4406,"start_line":111,"end_line":115,"raw_tex":"We consider the stochastic optimization problem that \n\\begin{align}\n  \\label{problem}\n  \\min\\limits_{x\\in\\mathbb{R}^d} f(x)=\\mathbb{E}_{z \\sim \\mathcal{D}}[f_z(x;z)],\n\\end{align}","sha256_utf8":"c60014fa98498c55bdbc6d4f99287e88e9b38f56573a09f9b8732a1d30f6d895"},{"start":4407,"end":4557,"start_line":116,"end_line":116,"raw_tex":"where the objective function $f(x)$ is possibly non-convex and $\\mathcal{D}$ is a probability distribution from which the random vector $z$ is drawn.\n","sha256_utf8":"dc1ad599b2a83ecdbeb539acecfe070fadd59dbc51834f1b0583fc34d2229eb1"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"广义光滑性","source_units":[{"start":11247,"end":11634,"start_line":196,"end_line":204,"raw_tex":"\\begin{definition}[$\\left(L_0,L_1\\right)$-smoothness]\n  \\label{definition 1}\n  Let $f(x)$ be a differentiable function. For some $L_0,L_1>0$,\n   $f(x)$ is $(L_0,L_1)$-smooth iff for any $x,y\\in\\mathbb{R}^d$ with $\\|x-y\\|\\leq 1/L_1$,\n  \\begin{align}\n    \\label{def generalized smooth}\n  \\|\\nabla f(x)-\\nabla f(y)\\|\\leq \\left(L_0+L_1\\nablaf{x}\\right)\\|x-y\\|.\n  \\end{align}\n\\end{definition}","sha256_utf8":"d94e33c114145d189945ab1220d2c5c2dc54f4f70a0b6b40f16b0729a9b64843"}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","title":"下界、无偏性与几乎必然噪声界","source_units":[{"start":29470,"end":30108,"start_line":379,"end_line":382,"raw_tex":"  \\item \\label{assumption 1} \\textbf{Below bounded:} The objective function is bounded from below, i.e.,  $f^*:=\\inf_{x\\in\\mathbb{R}^d}f(x)>-\\infty.$\n  \\item \\label{assumption 2}\\textbf{Unbiased estimator:} The gradient oracle returns an unbiased estimator of $\\nabla f(x)$, i.e., for all $x\\in\\mathbb{R}^d$, \n  $$\\mathbb{E}_z\\left[\\nabla f_z(x;z)\\right]=\\nabla f(x).$$\n  \\item \\label{assumption 3}\\textbf{Relaxed affine variance (almost-sure version):} The gradient oracle satisfies that for some constants $A,B,C>0$, $$\\|\\nabla f_z(x;z)-\\nabla f(x)\\|^2\\leq A\\left(f(x)-f^*\\right)+B\\nablaf{x}^2+C \\quad a.s., \\forall x\\in\\mathbb{R}^d.$$\n","sha256_utf8":"685ba2fb3916fcba8718bcb60e1a6ae7edbb1512891e28f075cc9c01d7583e37"}]},{"id":"S005","semantic_type":"algorithm","kind":"algorithm","title":"随机加速梯度迭代","source_units":[{"start":31180,"end":31775,"start_line":394,"end_line":406,"raw_tex":"\\begin{algorithm}[H]\n  \\caption{Randomized Stochastic Accelerated Gradient (RSAG)}\n  \\label{algorithm1}\n  \\begin{algorithmic}[1]\n  \\Require Horizon $T$, $\\tilde{x}_1=x_1\\in \\mathbb{R}^d$, $\\alpha_t \\in \\left(\\left.0,1\\right.\\right]$, step sizes $\\{\\theta_t\\}_{t\\in [T]}$, $\\{\\gamma_t\\}_{t\\in [T]}$.\n  \\For{$t=1,\\cdots, T$}\n  \\State $\\overline{x}_t=\\alpha_tx_t+\\left(1-\\alpha_t\\right)\\tilde{x}_t$;\n  \\State \\textbf{Set} $g_t=\\nabla f_z(\\overline{x}_t;z_t)$;\n  \\State $x_{t+1}=x_t-\\theta_t g_t$;\n  \\State $\\tilde{x}_{t+1}=\\overline{x}_t-\\gamma_t g_t$.\n  \\EndFor\n  \\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"2e3a6e7bbc95a601227978bedb7e5df489c2072a070a8118d9015fda2af5b0f5"}]},{"id":"S006","semantic_type":"notation","kind":"notation","title":"条件期望、范数与梯度记号","source_units":[{"start":19002,"end":19522,"start_line":278,"end_line":281,"raw_tex":"In this paper, we denote the set $\\{1,\\cdots,T\\}$ as $[T]$. \nWe use $\\mathbb{E}_t[\\cdot]\\triangleq\\mathbb{E}[\\cdot|z_1,\\cdots,z_{t-1}]$ to represent the conditional expectation, where $z_i$ is the random sample in the $i$-th gradient oracle. \nThe notation $a\\sim \\mathcal{O}(b)$ and $a\\leq \\mathcal{O}(b)$ refer to $c_1 b\\leq a\\leq c_2 b$ and $a\\leq c_3 b$ with $c_1, c_2, c_3$ being positive constants, respectively. \nAlso, we write $\\tilde{\\mathcal{O}}(b)$ as shorthand for $\\mathcal{O}(b \\cdot \\text{poly}(\\log b))$.\n","sha256_utf8":"0f753a3599127f9c5fc2888760e9b8cd1dde28d6014838cfe38f19ee93de95b6"},{"start":29253,"end":29374,"start_line":371,"end_line":371,"raw_tex":"We consider Problem \\eqref{problem} over the Euclidean space $\\mathbb{R}^d$ with the $l_2$ norm, denoted as $\\|\\cdot\\|$.\n","sha256_utf8":"664928cf310395cb7bb3d5dfbf7b9da3c8b9897d94711d7c0b90a89ea9c03459"},{"start":34901,"end":35051,"start_line":434,"end_line":434,"raw_tex":"For simplicity, we denote $\\overline{\\Delta}_t=f(\\overline{x}_t)-f^*$ and $\\xi_t=g_t-\\overline{g}_t$ where $\\overline{g}_t=\\nabla f(\\overline{x}_t)$.\n","sha256_utf8":"4ab257f7bae61bb02e7d8fd0e03316c38dded2de10f262c3784d69340a654a87"}]},{"id":"S007","semantic_type":"notation","kind":"notation","title":"光滑系数与函数差常数宏","source_units":[{"start":1640,"end":1707,"start_line":62,"end_line":62,"raw_tex":"\\newcommand{\\lsmooth}[1]{\\left(L_0+L_1\\|\\nabla f(x_{#1})\\|\\right)}\n","sha256_utf8":"8370e7aa3fd61f530a023af464d7ac9d2b56a7b4b128eae68cab9232123af483"},{"start":2027,"end":2175,"start_line":75,"end_line":75,"raw_tex":"\\newcommand{\\Gconstant}{8\\overline{\\Delta}_1+8+32\\log^2\\frac{T}{\\delta}+8\\frac{\\sqrt{2C}}{\\sqrt{L_0T}}\\log\\frac{T}{\\delta}+\\frac{303L_0}{224L_1^2}}\n","sha256_utf8":"cbd279d409c5af23bd5292dca00726c5a95ed60060c10fcaa004434ed9e613c6"},{"start":2302,"end":2362,"start_line":79,"end_line":79,"raw_tex":"\\newcommand{\\functiongapboundconstant}{\\overline{\\Delta}_c}\n","sha256_utf8":"6f3b5ed9b8b574d5579357b6c51cdc6c46b3a1eb58fc4027658efbca901bff80"}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","title":"迭代参数与常步长条件","source_units":[{"start":35474,"end":36654,"start_line":442,"end_line":452,"raw_tex":"  Let  $T>0$ and $\\delta\\in(0,1)$. Suppose that $f$ is an $\\left(L_0,L_1\\right)$-smooth function, $\\{\\overline{x}_t\\}_{x\\in[T]}$ is a sequence generated by Algorithm \\ref{algorithm1} with $\\alpha_t=\\frac{2}{t+1},\\gamma_t=\\left(1+\\alpha_t\\right)\\theta_t$ or $\\gamma_t=\\theta_t$ (SGD), $\\forall t\\in[T]$, and the step size $\\theta_t$ satisfies\n  \\begin{align}\n    \\label{eta constant non-convex}\n    \\theta_t=\\eta = \\min\\left\\{\\frac{1}{\\mathcal{A}\\sqrt{T}},\\frac{1}{\\mathcal{B}},\\frac{1}{\\mathcal{C}\\sqrt{T}}, \\frac{1}{\\mathcal{G}},\\frac{\\functiongapboundconstant^{-1/2}}{3\\left(A+4BL_0\\right)}, \\frac{\\functiongapboundconstant^{-3/2}}{12BL_1^2}\\right\\},\n  \\end{align}\n    where $\\mathcal{A}, \\mathcal{B}, \\mathcal{C}, \\mathcal{G}$ are defined as\n  \\begin{align}\n    \\label{theorem 1 eta}\n    \\mathcal{A}=4\\sqrt{\\left(L_0+L_1\\mathcal{M}_c\\right)A}, \\qquad &\\mathcal{B}=8\\left(L_0+L_1\\mathcal{M}_c\\right)\\left(B+1\\right), \\notag\\\\\n    \\mathcal{C}=2\\sqrt{\\left(L_0+L_1\\mathcal{M}_c\\right)C},\\qquad &\\mathcal{G}=8L_1 \\left(\\sqrt{A\\functiongapboundconstant}+2\\left(\\sqrt{B}+1\\right)\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}+\\sqrt{C}\\right),  \n  \\end{align}\n","sha256_utf8":"a494416c969023f2acafb14c286c0d08d4192a0dcf2d019d8365f87c7d43f833"}]},{"id":"S009","semantic_type":"lemma","kind":"lemma","title":"广义光滑下降界","source_units":[{"start":173853,"end":174186,"start_line":2109,"end_line":2113,"raw_tex":"\\begin{lemma}[Remark 2.3 in \\cite{zhang2020improved}]\n  \\label{descent lemma}\n  Suppose that $f:\\mathbb{R}^d \\rightarrow \\mathbb{R}$ is an $(L_0,L_1)$-generalized smooth function. When $\\|x-y\\|\\leq 1/L_1$, we have \n  $$\\left\\lvert f(y)- f(x)-\\la \\nabla f(x),y-x\\ra\\right\\rvert \\leq \\frac{L_0+L_1\\nablaf{x}}{2}\\|x-y\\|^2.$$\n\\end{lemma}","sha256_utf8":"92d6432dfc587318eb50e683be34c3f83e5ecf5da83b8ed1b961d9399277476c"}]},{"id":"S010","semantic_type":"lemma","kind":"lemma","title":"鞅差指数集中界","source_units":[{"start":174331,"end":174978,"start_line":2116,"end_line":2120,"raw_tex":"\\begin{lemma}[Lemma 1 in \\cite{li2020high}]\n  \\label{lemma high probability}\n  Assume that $\\{Z_t\\}_{t\\in[T]}$ is a martingale difference sequence with respect to $\\gamma_1, \\gamma_2, \\cdots, \\gamma_T$ and $\\mathbb{E}_t[\\exp\\left(Z_t^2/\\sigma_t^2\\right)]\\leq \\exp(1)$ for all $1 \\leq t \\leq T$, where $\\sigma_t$ is a sequence of measurable random variables with respect to $\\gamma_1, \\gamma_2, \\cdots, \\gamma_{t-1}$. Then, for any fixed $\\lambda >0$ and $\\delta\\in(0,1)$, with probability at least $1-\\delta$, we have\n  $$\\sumlimits{t=1}{T}Z_t\\leq \\frac{3\\lambda}{4}\\sumlimits{t=1}{T}\\sigma_t^2+\\frac{1}{\\lambda}\\log\\frac{1}{\\delta}.$$\n\\end{lemma}","sha256_utf8":"527ba6a30522eb9d78b966832366b8838adac27efce793544a71a232d0d28425"}]},{"id":"S011","semantic_type":"lemma","kind":"lemma","title":"平方根和不等式","source_units":[{"start":177250,"end":177478,"start_line":2168,"end_line":2174,"raw_tex":"\\begin{lemma}\n    \\label{sqrt sum}\n    Let $\\{a_t\\}_{t\\in[n]}$ be a sequence of non-negative real numbers. We have\n    \\begin{align*}\n      \\sqrt{\\sum_{i=1}^n a_i}\\leq \\sumlimits{i=1}{n}\\sqrt{a_i}.\n    \\end{align*}\n  \\end{lemma}","sha256_utf8":"1a4c86dc1978bfe25a6b7434a13be92ba54ae5ae1b55a3d310df63dc96e973d1"}]},{"id":"S012","semantic_type":"proposition","kind":"proposition","title":"加速权重的求和恒等式","source_units":[{"start":60335,"end":60876,"start_line":672,"end_line":684,"raw_tex":"\\begin{proposition}[Proposition 5.2 in \\cite{kavis2022high}]\n  \\label{proposition 1}\n  Denote $\\alpha_t=\\frac{2}{t+1}$ and $\\varGamma_t=\\left(1-\\alpha_t\\right)\\varGamma_{t-1}$ with $\\varGamma_1=1$. We have that for all $t\\in[T],$\n  \\begin{align}\n    \\label{proposition 1.1}\n   \\varGamma_{t}\\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}=1,\n  \\end{align}\n  and \n  \\begin{align}\n    \\label{proposition 1.2}\n    \\left[\\sumlimits{k=t}{T}\\left(1-\\alpha_k\\right)\\varGamma_k\\right]\\frac{\\alpha_t}{\\varGamma_t}\\leq 2.\n  \\end{align}\n\\end{proposition}","sha256_utf8":"9b4b234fa7d6c03ef3f97f07b264259d654c201e28b311db034350b876babc33"}]},{"id":"S013","semantic_type":"proposition","kind":"proposition","title":"两条迭代序列的距离界","source_units":[{"start":60980,"end":61603,"start_line":687,"end_line":698,"raw_tex":"\\begin{proposition}\n  \\label{proposition overline x_t - x_t}\n  Let $\\{x_t\\}_{t\\in[T]}$ and $\\{\\overline{x}_t\\}_{t\\in[T]}$ be generated by Algorithm \\ref{algorithm1}. We have\n  \\begin{align}\n    \\label{overline x_t-x_t}\n    \\overline{x}_t-x_t =\\left(1-\\alpha_t\\right)\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\frac{\\left(\\theta_k-\\gamma_k\\right)}{\\alpha_k}g_k,\n  \\end{align}\n  and\n  \\begin{align*}\n  \\|\\overline{x}_t-x_t\\|^2\\leq \\left(1-\\alpha_t\\right)\\varGamma_t\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\frac{\\left(\\theta_k-\\gamma_k\\right)^2}{\\alpha_k^2}\\|g_k\\|^2.\n\\end{align*}\n\\end{proposition}","sha256_utf8":"92d4e72657d85962db728dce0fed735a00a11ffd81c96f00feb4e7f0a28f8e69"}]},{"id":"S014","semantic_type":"lemma","kind":"lemma","title":"函数差控制梯度范数","source_units":[{"start":175792,"end":176119,"start_line":2135,"end_line":2140,"raw_tex":"\\begin{lemma}\n  \\label{lemma 6.2}  \n  Let $f(x) : \\mathbb{R}^d\\rightarrow \\mathbb{R}$ be an $(L_0,L_1)$-smooth function with minimum $f^*$. \n  Then, $$\\|\\nabla f(x_t)\\|^2\\leq 2\\left(L_0+L_1\\|\\nabla f(x_t)\\|\\right)\\Delta_t,$$ and $$\\nablaf{x_t}^2\\leq 4L_0\\Delta_t+4L_1^2\\Delta_t^2,$$\n  where $\\Delta_t=f(x_t)-f^*$.\n  \\end{lemma}","sha256_utf8":"9bfc5c39bea72f7daae6f6430eba1baf23b91f2ffe85c38076c3c8e2f712ca5b"}]},{"id":"S015","semantic_type":"lemma","kind":"lemma","title":"梯度噪声内积的统一概率界","source_units":[{"start":61604,"end":62421,"start_line":699,"end_line":716,"raw_tex":"\\begin{lemma}\n  \\label{high probability sum non-convex constant}\n  Given $T\\geq 1 $ and $\\delta\\in \\left(0,1\\right)$, if Assumptions \\ref{assumption 2} and \\ref{assumption 3} hold, then with probability at least $1-\\delta$, $\\forall l\\in[T]$,\n  \\begin{align}\n    \\label{inequality high probability non-convex constant}\n    \\sumlimits{t=1}{l}-\\la\\overline{g}_t,\\xi_t\\ra\\leq \\frac{1}{4}\\sumlimits{t=1}{l}\\frac{P_t^2}{P_c^2}\\|\\overline{g}_t\\|^2+3P_c^2\\log\\frac{T}{\\delta},\n  \\end{align}\n  where \n  \\begin{align}\n    \\label{P_t}\n    P_t=\\sqrt{A\\overline{\\Delta}_t+4B\\left(L_0\\overline{\\Delta}_t+L_1^2\\overline{\\Delta}_t^2\\right)+C},\n  \\end{align}\nand \n\\begin{align}\n  \\label{P}\n  P_c=\\sqrt{A\\functiongapboundconstant+4B\\left(L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2\\right)+C}.\n\\end{align}\n\\end{lemma}","sha256_utf8":"f305991c925bdf609ab3952fdc1bb70186eda4e6245d78eca9c48800c471cc57"}]},{"id":"S016","semantic_type":"proposition","kind":"proposition","title":"函数差与噪声尺度的一致有界性","source_units":[{"start":62537,"end":62966,"start_line":719,"end_line":726,"raw_tex":"\\begin{proposition}\n  \\label{constant bound}\n  Under the conditions of Theorem \\ref{theorem 1}, $\\overline{\\Delta}_t\\leq \\functiongapboundconstant,P_t\\leq P_c, \\forall t\\in[T]$, hold with probability at least $1-\\delta$, where  $P_t$, $P_c$ are in \\eqref{P_t}, \\eqref{P} and $\\functiongapboundconstant$ is defined as\n  \\begin{align}\n    \\label{Gconstant}\n    \\functiongapboundconstant=\\Gconstant.\n  \\end{align} \n\\end{proposition}","sha256_utf8":"33f2d89b307dea7ec57f97330fbca0f7d5ee3aa0d8d8f41cada63c9ca2c6a2a9"}]},{"id":"S017","semantic_type":"proof","kind":"proof","title":"主收敛定理的完整证明","source_units":[{"start":74876,"end":79242,"start_line":864,"end_line":903,"raw_tex":"\\begin{proof}[Proof of Theorem \\ref{theorem 1}]\nWe assume \\eqref{inequality high probability non-convex constant} always holds then we deduce \\eqref{4.1}. Note that \\eqref{inequality high probability non-convex constant} holds with probability at least $1-\\delta$. Therefore, \\eqref{4.1} holds with probability at least $1-\\delta$.\nRecalling \\eqref{inequality base} and dividing $\\theta_t$ on both sides, we have  \n\\begin{align*}\n  \\|\\overline{g}_t\\|^2\\leq &\\frac{f(x_t)-f(x_{t+1})}{\\theta_t}+\\frac{1}{2\\theta_t}\\left(L_0+L_1\\nablaf{x_t}\\right)\\left(1-\\alpha_t\\right)\\varGamma_t\\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}\\theta_k^2\\|g_k\\|^2\\notag\\\\\n&+\\left(L_0+L_1\\nablaf{x_t}\\right)\\theta_t\\|g_t\\|^2-\\la\\overline{g}_t,\\xi_t\\ra.\n\\end{align*}\nSumming over $t\\in[T]$ with $\\theta_t=\\eta$,\n\\begin{align}\n  \\label{constant gt}\n  \\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2\\leq & \\frac{1}{\\eta}\\sumlimits{t=1}{T}\\left(f(x_t)-f(x_{t+1})\\right)+\\frac{\\eta}{2}\\sumlimits{t=1}{T}\\left[\\left(L_0+L_1\\nablaf{x_t}\\right)\\left(1-\\alpha_t\\right)\\varGamma_t\\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}\\|g_k\\|^2\\right]\\notag\\\\\n  &+\\eta\\sumlimits{t=1}{T}\\left(L_0+L_1\\nablaf{x_t}\\right)\\|g_t\\|^2-\\sumlimits{t=1}{T}\\la \\overline{g}_t,\\xi_t\\ra\\notag\\\\\n  \\leq &\\frac{\\overline{\\Delta}_1}{\\eta}+\\frac{\\eta}{2} \\sumlimits{t=1}{T}\\left[\\sumlimits{k=t}{T}\\lsmooth{k}\\left(1-\\alpha_k\\right)\\varGamma_k\\right]\\frac{\\alpha_t}{\\varGamma_t}\\|g_t\\|^2\\notag\\\\\n  & +\\eta\\sumlimits{t=1}{T}\\left(L_0+L_1\\nablaf{x_t}\\right)\\|g_t\\|^2-\\sumlimits{t=1}{T}\\la \\overline{g}_t,\\xi_t\\ra.\n\\end{align}\nApplying Lemma \\ref{high probability sum non-convex constant},\n\\begin{align*}\n  \\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2\\leq &\\frac{\\overline{\\Delta}_1}{\\eta}+\\frac{\\eta}{2} \\sumlimits{t=1}{T}\\left[\\sumlimits{k=t}{T}\\lsmooth{k}\\left(1-\\alpha_k\\right)\\varGamma_k\\right]\\frac{\\alpha_t}{\\varGamma_t}\\|g_t\\|^2\\notag\\\\\n  & +\\eta\\sumlimits{t=1}{T}\\left(L_0+L_1\\nablaf{x_t}\\right)\\|g_t\\|^2+\\frac{1}{4P_c^2}\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2 P_t^2+3P_c^2\\log\\frac{T}{\\delta}.\n\\end{align*}\nCombining with \\eqref{proposition 1.2}, \\eqref{nabla fx_t constant step size}, \\eqref{triangle square} and Proposition \\ref{constant bound}, we have\n\\begin{align*}\n  \\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2 \\leq & \\frac{\\overline{\\Delta}_1}{\\eta}+2\\eta\\sumlimits{t=1}{T}\\left(L_0+L_1\\mathcal{M}_c\\right)\\|g_t\\|^2+\\frac{1}{4}\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2+3P_c^2\\log\\frac{T}{\\delta}\\notag\\\\\n  \\leq & \\frac{\\overline{\\Delta}_1}{\\eta}+4\\eta\\left(L_0+L_1\\mathcal{M}_c\\right) T \\left(A\\functiongapboundconstant+C\\right)+4\\left(L_0+L_1\\mathcal{M}_c\\right)\\left(B+1\\right)\\eta\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2\\notag\\\\\n  & + \\frac{1}{4}\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2+3\\left(A\\functiongapboundconstant + 4B\\left(L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2\\right)+C\\right)\\log\\frac{T}{\\delta}.\n\\end{align*}\nRe-arranging the above inequality with the constraint of $\\eta$ in \\eqref{eta constant non-convex}, we obtain \n\\begin{align*}\n  \\frac{1}{T}\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2\\leq & \\frac{4\\overline{\\Delta}_1}{\\eta T}+ \\frac{4}{\\sqrt{T}}\\sqrt{L_0+L_1\\mathcal{M}_c}\\left(\\sqrt{A\\functiongapboundconstant^2}+2\\sqrt{C}\\right)\\notag\\\\\n  &+\\frac{12}{T}\\left(A\\functiongapboundconstant + 4B\\left(L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2\\right)+C\\right)\\log\\frac{T}{\\delta}.\n\\end{align*}\nCombining with the restrictions of $\\eta$ in \\eqref{eta constant non-convex} again, we get the final result,\n\\begin{align}\n  \\label{5.34}\n  \\frac{1}{T}\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2\\leq & \\frac{4}{\\sqrt{T}}\\sqrt{L_0+L_1\\mathcal{M}_c}\\left(\\sqrt{A}\\left(\\functiongapboundconstant+4\\overline{\\Delta}_1\\right)+2\\sqrt{C}\\left(1+\\overline{\\Delta}_1\\right)\\right)\\notag\\\\\n  &+ \\frac{32\\overline{\\Delta}_1}{T}\\left(L_0+L_1\\mathcal{M}_c\\right)\\left(B+1\\right) +  \\frac{12\\overline{\\Delta}_1}{T}\\left(\\left(A+4BL_0\\right)\\functiongapboundconstant^{1/2}+4BL_1^2\\functiongapboundconstant^{3/2}\\right) \\notag\\\\\n  &+ \\frac{32\\overline{\\Delta}_1}{T}L_1 \\left(\\sqrt{A\\functiongapboundconstant}+2\\left(\\sqrt{B}+1\\right)\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}+\\sqrt{C}\\right) \\notag\\\\\n  &+\\frac{12}{T}\\left(A\\functiongapboundconstant + 4B\\left(L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2\\right)+C\\right)\\log\\frac{T}{\\delta}.\n\\end{align}\n\\end{proof}","sha256_utf8":"ca4c6790f25f4c443d0522acbfa916372b5ba003738a89965374cef17be99fac"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"固定高概率事件并整理单步下降","source_units":[{"start":74924,"end":75621,"start_line":865,"end_line":870,"raw_tex":"We assume \\eqref{inequality high probability non-convex constant} always holds then we deduce \\eqref{4.1}. Note that \\eqref{inequality high probability non-convex constant} holds with probability at least $1-\\delta$. Therefore, \\eqref{4.1} holds with probability at least $1-\\delta$.\nRecalling \\eqref{inequality base} and dividing $\\theta_t$ on both sides, we have  \n\\begin{align*}\n  \\|\\overline{g}_t\\|^2\\leq &\\frac{f(x_t)-f(x_{t+1})}{\\theta_t}+\\frac{1}{2\\theta_t}\\left(L_0+L_1\\nablaf{x_t}\\right)\\left(1-\\alpha_t\\right)\\varGamma_t\\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}\\theta_k^2\\|g_k\\|^2\\notag\\\\\n&+\\left(L_0+L_1\\nablaf{x_t}\\right)\\theta_t\\|g_t\\|^2-\\la\\overline{g}_t,\\xi_t\\ra.\n\\end{align*}\n","sha256_utf8":"b516863ac6cc227a12135c3b7bc19e2150119130c5d87eabd3c652e429d8a845"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"望远镜求和与权重换序","source_units":[{"start":75621,"end":76433,"start_line":871,"end_line":878,"raw_tex":"Summing over $t\\in[T]$ with $\\theta_t=\\eta$,\n\\begin{align}\n  \\label{constant gt}\n  \\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2\\leq & \\frac{1}{\\eta}\\sumlimits{t=1}{T}\\left(f(x_t)-f(x_{t+1})\\right)+\\frac{\\eta}{2}\\sumlimits{t=1}{T}\\left[\\left(L_0+L_1\\nablaf{x_t}\\right)\\left(1-\\alpha_t\\right)\\varGamma_t\\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}\\|g_k\\|^2\\right]\\notag\\\\\n  &+\\eta\\sumlimits{t=1}{T}\\left(L_0+L_1\\nablaf{x_t}\\right)\\|g_t\\|^2-\\sumlimits{t=1}{T}\\la \\overline{g}_t,\\xi_t\\ra\\notag\\\\\n  \\leq &\\frac{\\overline{\\Delta}_1}{\\eta}+\\frac{\\eta}{2} \\sumlimits{t=1}{T}\\left[\\sumlimits{k=t}{T}\\lsmooth{k}\\left(1-\\alpha_k\\right)\\varGamma_k\\right]\\frac{\\alpha_t}{\\varGamma_t}\\|g_t\\|^2\\notag\\\\\n  & +\\eta\\sumlimits{t=1}{T}\\left(L_0+L_1\\nablaf{x_t}\\right)\\|g_t\\|^2-\\sumlimits{t=1}{T}\\la \\overline{g}_t,\\xi_t\\ra.\n\\end{align}\n","sha256_utf8":"f4690e840aacac1474b69177c36ae74dfcc7338fddcea3f511dfbdd8e95f48a5"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"控制累计梯度噪声","source_units":[{"start":76433,"end":76916,"start_line":879,"end_line":883,"raw_tex":"Applying Lemma \\ref{high probability sum non-convex constant},\n\\begin{align*}\n  \\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2\\leq &\\frac{\\overline{\\Delta}_1}{\\eta}+\\frac{\\eta}{2} \\sumlimits{t=1}{T}\\left[\\sumlimits{k=t}{T}\\lsmooth{k}\\left(1-\\alpha_k\\right)\\varGamma_k\\right]\\frac{\\alpha_t}{\\varGamma_t}\\|g_t\\|^2\\notag\\\\\n  & +\\eta\\sumlimits{t=1}{T}\\left(L_0+L_1\\nablaf{x_t}\\right)\\|g_t\\|^2+\\frac{1}{4P_c^2}\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2 P_t^2+3P_c^2\\log\\frac{T}{\\delta}.\n\\end{align*}\n","sha256_utf8":"2140e4d719a95d100efb47673ec6ca0d0a330a77ebe0aa87ed359acbd0e976d0"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"代入函数差与梯度上界","source_units":[{"start":76916,"end":77750,"start_line":884,"end_line":889,"raw_tex":"Combining with \\eqref{proposition 1.2}, \\eqref{nabla fx_t constant step size}, \\eqref{triangle square} and Proposition \\ref{constant bound}, we have\n\\begin{align*}\n  \\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2 \\leq & \\frac{\\overline{\\Delta}_1}{\\eta}+2\\eta\\sumlimits{t=1}{T}\\left(L_0+L_1\\mathcal{M}_c\\right)\\|g_t\\|^2+\\frac{1}{4}\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2+3P_c^2\\log\\frac{T}{\\delta}\\notag\\\\\n  \\leq & \\frac{\\overline{\\Delta}_1}{\\eta}+4\\eta\\left(L_0+L_1\\mathcal{M}_c\\right) T \\left(A\\functiongapboundconstant+C\\right)+4\\left(L_0+L_1\\mathcal{M}_c\\right)\\left(B+1\\right)\\eta\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2\\notag\\\\\n  & + \\frac{1}{4}\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2+3\\left(A\\functiongapboundconstant + 4B\\left(L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2\\right)+C\\right)\\log\\frac{T}{\\delta}.\n\\end{align*}\n","sha256_utf8":"ffbc4e37482f35d850cec7c3d9d4a402989c422c6bcbe78e03c8ed503823c095"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"吸收累计梯度项","source_units":[{"start":77750,"end":78254,"start_line":890,"end_line":894,"raw_tex":"Re-arranging the above inequality with the constraint of $\\eta$ in \\eqref{eta constant non-convex}, we obtain \n\\begin{align*}\n  \\frac{1}{T}\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2\\leq & \\frac{4\\overline{\\Delta}_1}{\\eta T}+ \\frac{4}{\\sqrt{T}}\\sqrt{L_0+L_1\\mathcal{M}_c}\\left(\\sqrt{A\\functiongapboundconstant^2}+2\\sqrt{C}\\right)\\notag\\\\\n  &+\\frac{12}{T}\\left(A\\functiongapboundconstant + 4B\\left(L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2\\right)+C\\right)\\log\\frac{T}{\\delta}.\n\\end{align*}\n","sha256_utf8":"e1d90698431d84a3cd67f3a4ad4483f55cf00266f4184f0e96519fbf4734b1cf"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"展开步长得到显式收敛率","source_units":[{"start":78254,"end":79242,"start_line":895,"end_line":903,"raw_tex":"Combining with the restrictions of $\\eta$ in \\eqref{eta constant non-convex} again, we get the final result,\n\\begin{align}\n  \\label{5.34}\n  \\frac{1}{T}\\sumlimits{t=1}{T}\\|\\overline{g}_t\\|^2\\leq & \\frac{4}{\\sqrt{T}}\\sqrt{L_0+L_1\\mathcal{M}_c}\\left(\\sqrt{A}\\left(\\functiongapboundconstant+4\\overline{\\Delta}_1\\right)+2\\sqrt{C}\\left(1+\\overline{\\Delta}_1\\right)\\right)\\notag\\\\\n  &+ \\frac{32\\overline{\\Delta}_1}{T}\\left(L_0+L_1\\mathcal{M}_c\\right)\\left(B+1\\right) +  \\frac{12\\overline{\\Delta}_1}{T}\\left(\\left(A+4BL_0\\right)\\functiongapboundconstant^{1/2}+4BL_1^2\\functiongapboundconstant^{3/2}\\right) \\notag\\\\\n  &+ \\frac{32\\overline{\\Delta}_1}{T}L_1 \\left(\\sqrt{A\\functiongapboundconstant}+2\\left(\\sqrt{B}+1\\right)\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}+\\sqrt{C}\\right) \\notag\\\\\n  &+\\frac{12}{T}\\left(A\\functiongapboundconstant + 4B\\left(L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2\\right)+C\\right)\\log\\frac{T}{\\delta}.\n\\end{align}\n\\end{proof}","sha256_utf8":"aae96f15ae53e7c826ace5029196f18135db1a6cbc5a60a87c68ebdfda100d20"}]},{"id":"S024","semantic_type":"proof","kind":"proof","title":"函数差一致有界性的完整证明","source_units":[{"start":62967,"end":74738,"start_line":727,"end_line":862,"raw_tex":"\\begin{proof}\n  We assume that \\eqref{inequality high probability non-convex constant} always happens and then deduce $\\overline{\\Delta}_t\\leq \\functiongapboundconstant$, $P_t\\leq P_c$ for all $t\\in[T]$. \n  Since \\eqref{inequality high probability non-convex constant} happens with probability at least $1-\\delta$, $\\overline{\\Delta}_t\\leq \\functiongapboundconstant$ and $P_t\\leq P_c$, $\\forall t\\in[T]$,  hold with probability at least $1-\\delta$. \n  It is obvious that $f(\\overline{x}_1)-f^*\\leq \\functiongapboundconstant$. Therefore, $P_1\\leq P_c$.\n  Suppose that for some $l\\in[T]$, $$f(\\overline{x}_t)-f^*\\leq \\functiongapboundconstant,\\quad\\forall t\\in[l], \\quad \\text{thus}, \\quad P_t\\leq P_c, \\quad \\forall t\\in[l].$$ \n  By the triangle inequality of the norm function,\n  \\begin{align*}\n    \\|g_t\\|\\leq  \\|g_t-\\overline{g}_t\\|+\\|\\overline{g}_t\\|\n    \\leq  \\sqrt{A\\overline{\\Delta}_t+B\\|\\overline{g}_t\\|^2+C}+\\|\\overline{g}_t\\|\n    \\leq  \\sqrt{A\\overline{\\Delta}_t}+\\left(\\sqrt{B}+1\\right)\\|\\overline{g}_t\\|+\\sqrt{C},\n  \\end{align*}\n  where the second inequality follows from Assumption \\ref{assumption 3} and the last inequality follows from Lemma \\ref{sqrt sum}.\n  Combining with Lemma \\ref{lemma 6.2} and the assumption that $\\overline{\\Delta}_t\\leq \\functiongapboundconstant, \\forall t\\in[l]$, we have\n  \\begin{align} \n    \\label{g_t}\n    \\|g_t\\| \\leq  \\sqrt{A\\functiongapboundconstant}+2\\left(\\sqrt{B}+1\\right)\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}+\\sqrt{C}, \\quad \\forall t\\in[l].\n  \\end{align}\n  For simplicity, let \n  \\begin{align}\n    \\label{Y_c}\n    Y_c=\\eta \\left(\\sqrt{A\\functiongapboundconstant}+2\\left(\\sqrt{B}+1\\right)\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}+\\sqrt{C}\\right).\n  \\end{align}\n    By the iteration step of Algorithm \\ref{algorithm1}, we have\n  \\begin{align*}\n  \\|x_{t+1}-x_t\\|=\\theta_t\\|g_t\\|=\\eta \\|g_t\\|\\leq Y_c\\leq 1/8L_1,\n  \\end{align*}\n  where the last inequality follows from the restriction of $\\eta$.\n Applying Lemma \\ref{descent lemma}, we have \n\\begin{align*}\n  & f(x_{t+1})-f(x_t) \\notag\\\\\n  \\leq & \\la \\nabla f(x_t),x_{t+1}-x_t\\ra+\\frac{L_0+L_1\\nablaf{x_t}}{2}\\|x_{t+1}-x_t\\|^2\\notag\\\\\n   = & -\\theta_t\\la \\nabla f(x_t),g_t\\ra+\\frac{L_0+L_1\\nablaf{x_t}}{2}\\theta_t^2\\|g_t\\|^2\\notag\\\\\n   = & -\\theta_t\\la \\overline{g}_t+\\nabla f(x_t)-\\overline{g}_t,\\overline{g}_t+\\xi_t\\ra + \\frac{L_0+L_1\\nablaf{x_t}}{2}\\theta_t^2\\|g_t\\|^2\\notag\\\\\n   = & -\\theta_t \\|\\overline{g}_t\\|^2-\\theta_t\\la \\overline{g}_t,\\xi_t\\ra -\\theta_t \\la \\nabla f(x_t)-\\overline{g}_t,g_t\\ra+\\frac{L_0+L_1\\nablaf{x_t}}{2}\\theta_t^2\\|g_t\\|^2\\notag\\\\\n  \\leq & -\\theta_t \\|\\overline{g}_t\\|^2-\\theta_t\\la \\overline{g}_t,\\xi_t\\ra +\\theta_t \\| \\nabla f(x_t)-\\overline{g}_t\\|\\|g_t\\|+\\frac{L_0+L_1\\nablaf{x_t}}{2}\\theta_t^2\\|g_t\\|^2,\n\\end{align*} \nwhere the first equation follows from the update rule in Algorithm \\ref{algorithm1} and the last line follows from Cauchy-Schwarz inequality. \nRecall the constraints of the step size for RSAG with the constant step size and SGD, we have \n\\begin{align}\n  \\label{constant step size constraint}\n  0\\leq \\frac{\\gamma_t-\\theta_t}{\\alpha_t}\\leq \\theta_t=\\eta, \\quad \\forall t\\in[T].\n\\end{align}\nCombining \\eqref{overline x_t-x_t} with \\eqref{proposition 1.1}, \\eqref{g_t}, \\eqref{constant step size constraint}, we have\n\\begin{align}\n  \\label{bound overline x_t-x_t}\n  \\|\\overline{x}_t-x_t\\| \\leq & \\eta\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\|g_k\\|\\notag\\\\\n  \\leq & \\eta\\left(\\sqrt{A\\functiongapboundconstant}+2\\left(\\sqrt{B}+1\\right)\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}+\\sqrt{C}\\right)\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\notag\\\\\n  = & Y_c \\leq 1/8L_1.\n\\end{align}\nNote that \\eqref{bound overline x_t-x_t} holds for all $t\\in[l+1]$. \nThus, applying Definition \\ref{definition 1}, we have that for all $t\\in[l]$,\n\\begin{align}  \n  \\label{inequality base}\n  & f(x_{t+1})-f(x_t) \\notag\\\\\n  \\leq & -\\theta_t \\|\\overline{g}_t\\|^2-\\theta_t\\la \\overline{g}_t,\\xi_t\\ra +\\theta_t\\left(L_0+L_1\\nablaf{x_t}\\right)\\|\\overline{x}_t-x_t\\|\\|g_t\\| +\\frac{L_0+L_1\\nablaf{x_t}}{2}\\theta_t^2\\|g_t\\|^2\\notag\\\\\n  \\leq & -\\theta_t\\|\\overline{g}_t\\|^2-\\theta_t\\la\\overline{g}_t,\\xi_t\\ra+\\frac{1}{2}\\left(L_0+L_1\\nablaf{x_t}\\right)\\|\\overline{x}_t-x_t\\|^2+\\left(L_0+L_1\\nablaf{x_t}\\right)\\theta_t^2\\|g_t\\|^2\\notag\\\\\n  \\leq & -\\theta_t\\|\\overline{g}_t\\|^2-\\theta_t\\la\\overline{g}_t,\\xi_t\\ra+\\frac{1}{2}\\left(L_0+L_1\\nablaf{x_t}\\right)\\left(1-\\alpha_t\\right)\\varGamma_t\\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}\\theta_k^2\\|g_k\\|^2\\notag\\\\\n  & + \\left(L_0+L_1\\nablaf{x_t}\\right)\\theta_t^2\\|g_t\\|^2,\n\\end{align}\nwhere we use Young's inequality in the second inequality and apply Proposition \\ref{proposition overline x_t - x_t} with \\eqref{constant step size constraint} in the last inequality. \nSumming over $t\\in [l]$ with $\\theta_t=\\eta$, we have\n\\begin{align}\n\\label{telescope}\nf(x_{l+1})-f(x_1)  \\leq& \\frac{1}{2}\\sum\\limits^{l}_{t=1}\\left[\\lsmooth{t}\\left(1-\\alpha_t\\right)\\varGamma_t\\sum\\limits_{k=1}^t\\frac{\\alpha_k}{\\varGamma_k}\\eta^2\\|g_k\\|^2\\right]\\notag\\\\\n&+\\eta^2\\sumlimits{t=1}{l}\\lsmooth{t}\\|g_t\\|^2-\\eta\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2-\\eta\\sumlimits{t=1}{l}\\la \\overline{g}_t,\\xi_t\\ra\\notag\\\\\n=& \\frac{1}{2}\\eta^2 \\sumlimits{t=1}{l}\\left[\\sumlimits{k=t}{l}\\lsmooth{k}\\left(1-\\alpha_k\\right)\\varGamma_k\\right]\\frac{\\alpha_t}{\\varGamma_t}\\|g_t\\|^2\\notag\\\\\n&+\\eta^2\\sumlimits{t=1}{l}\\lsmooth{t}\\|g_t\\|^2-\\eta\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2-\\eta\\sumlimits{t=1}{l}\\la \\overline{g}_t,\\xi_t\\ra.\n\\end{align}\nApplying the triangle inequality of $\\|\\cdot\\|$, we have that for all $t\\in[l]$,\n\\begin{align}\n  \\label{nabla fx_t}\n  \\nablaf{x_t} \\leq  \\|\\overline{g}_t-\\nabla f(x_t)\\|+\\|\\overline{g}_t\\| \n  \\leq  \\left(L_0+L_1\\|\\overline{g}_t\\|\\right)\\|\\overline{x}_t-x_t\\|+\\|\\overline{g}_t\\|,\n\\end{align}\nwhere the second inequality holds since $\\|\\overline{x}_t-x_t\\|\\leq 1/L_1$. Combining \\eqref{bound overline x_t-x_t} and \\eqref{nabla fx_t}, for all $t\\in[l]$,\n\\begin{align}\n  \\label{nabla fx_t constant step size}\n   \\nablaf{x_t}\n  \\leq & \\frac{1}{8L_1}\\left(L_0+2L_1\\sqrt{L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2}\\right)+2\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}\\notag\\\\\n  = & \\frac{L_0}{8L_1}+\\frac{9}{4} \\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}=\\mathcal{M}_c,\n\\end{align}\nwhere we apply Lemma \\ref{lemma 6.2} to the inequality. \nApplying the triangle inequality again, we have\n\\begin{align}\n  \\label{triangle square}\n  \\|g_t\\|^2\\leq 2\\|g_t-\\overline{g}_t\\|^2+2\\|\\overline{g}_t\\|^2\\leq 2\\left(A \\overline{\\Delta}_t+\\left(B+1\\right)\\|\\overline{g}_t\\|^2+C\\right).\n\\end{align}\nCombining  \\eqref{proposition 1.2}, \\eqref{inequality high probability non-convex constant}, \\eqref{telescope} and \\eqref{triangle square}, together with the bound of $\\nablaf{x_t}$ in \\eqref{nabla fx_t constant step size}, we have \n\\begin{align}\n  \\label{4.26}\nf(x_{l+1})-f(x_1)  \\leq & 2\\left(L_0+L_1\\mathcal{M}_c\\right)\\eta^2\\sumlimits{t=1}{l}\\|g_t\\|^2-\\eta\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2-\\eta\\sumlimits{t=1}{l}\\la\\overline{g}_t,\\xi_t\\ra\\notag\\\\\n\\leq & 4\\left(L_0+L_1\\mathcal{M}_c\\right)\\eta^2\\sumlimits{t=1}{l}\\left(A\\overline{\\Delta}_t+\\left(B+1\\right)\\|\\overline{g}_t\\|^2+C\\right)-\\eta\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2\\notag\\\\\n&+\\frac{1}{4}\\eta\\sumlimits{t=1}{l}\\frac{P_t^2}{P_c^2}\\|\\overline{g}_t\\|^2+3\\eta P_c^2\\log\\frac{T}{\\delta}\\notag\\\\\n\\leq & 4\\left(L_0+L_1\\mathcal{M}_c\\right)\\eta^2 lA\\functiongapboundconstant+4\\left(L_0+L_1\\mathcal{M}_c\\right)\\eta^2lC \\notag\\\\\n&+\\left(4\\left(L_0+L_1\\mathcal{M}_c\\right)\\eta\\left(B+1\\right)+\\frac{1}{4}-1\\right)\\eta\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2+3\\eta P_c^2\\log\\frac{T}{\\delta},\n\\end{align}\nwhere the last inequality holds since the assumption that $P_t\\leq P_c, \\forall t\\in[l]$. \nApplying the restrictions of $\\eta$ in \\eqref{eta constant non-convex}, it is apparent that \n\\begin{align*}\n  f(x_{l+1})\\leq f(x_1)+\\frac{1}{4}\\functiongapboundconstant+1+3\\eta P_c^2\\log\\frac{T}{\\delta}.\n\\end{align*}\nSince $P_c^2 = A\\functiongapboundconstant+4B\\left(L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2\\right)+C= \\left(A+4BL_0\\right)\\functiongapboundconstant+4BL_1^2\\functiongapboundconstant^2+C$,\n\\begin{align}\n  \\label{f x+1-f x_1 constant step size}\n  f(x_{l+1}) \\leq & f(x_1)+\\frac{1}{4}\\functiongapboundconstant+1+3\\eta \\left(A+4BL_0\\right)\\functiongapboundconstant\\log\\frac{T}{\\delta}+12\\eta BL_1^2\\functiongapboundconstant^2\\log\\frac{T}{\\delta}+3\\eta C \\log\\frac{T}{\\delta}\\notag\\\\\n  \\leq & f(x_1)+\\frac{1}{4}\\functiongapboundconstant+1+2\\sqrt{\\functiongapboundconstant}\\log\\frac{T}{\\delta}+3\\eta C\\log\\frac{T}{\\delta}     \\notag\\\\\n  \\leq & f(x_1)+\\frac{1}{2}\\functiongapboundconstant+1+ 4\\log^2\\frac{T}{\\delta}+3\\eta C \\log\\frac{T}{\\delta},\n\\end{align}\nwhere the second inequality is due to the restrictions of $\\eta$ in \\eqref{eta constant non-convex} and the last inequality follows from Young's inequality. Next we will bound the gap between $f(x_{l+1})$ and $f(\\overline{x}_{l+1})$. By \\eqref{bound overline x_t-x_t}, we have $\\|\\overline{x}_{l+1}-x_{l+1}\\|<1/L_1$. Using Lemma \\ref{descent lemma} again,\n\\begin{align}\n  \\label{gap between f overline x_t+1 and f x_t+1 constant step size}\n  f(\\overline{x}_{l+1}) \\leq & f(x_{l+1})+\\la \\overline{g}_{l+1}, \\overline{x}_{l+1}-x_{l+1}\\ra+\\frac{L_0+L_1\\|\\overline{g}_{l+1}\\|}{2}\\|\\overline{x}_{l+1}-x_{l+1}\\|^2\\notag\\\\\n  \\leq & f(x_{l+1})+\\|\\overline{g}_{l+1}\\|\\|\\overline{x}_{l+1}-x_{l+1}\\|+\\frac{L_0+L_1\\|\\overline{g}_{l+1}\\|}{2}\\|\\overline{x}_{l+1}-x_{l+1}\\|^2,\n\\end{align}\nwhere the second inequality follows from Cauchy-Schwarz inequality. Substituting \\eqref{bound overline x_t-x_t} into \\eqref{gap between f overline x_t+1 and f x_t+1 constant step size}, we have\n\\begin{align}\n  \\label{descent lemma 2}\n  f(\\overline{x}_{l+1}) \\leq & f(x_{l+1})+\\frac{1}{8L_1}\\|\\overline{g}_{l+1}\\|+\\frac{1}{128L_1^2}\\left(L_0+L_1\\|\\overline{g}_{l+1}\\|\\right)\\notag\\\\\n  = & f(x_{l+1})+\\frac{17}{128L_1}\\|\\overline{g}_{l+1}\\|+\\frac{L_0}{128L_1^2}.\n\\end{align}\nCombining \\eqref{f x+1-f x_1 constant step size} and \\eqref{descent lemma 2}, and subtracting $f^*$ from both sides, we have\n\\begin{align}\n  \\overline{\\Delta}_{l+1}\\leq & \\overline{\\Delta}_1+\\frac{1}{2}\\functiongapboundconstant+1+4\\log^2\\frac{T}{\\delta}+3\\eta C \\log\\frac{T}{\\delta} +\\frac{17}{128L_1}\\|\\overline{g}_{l+1}\\|+\\frac{L_0}{128L_1^2}\\notag\\\\\n  \\leq & \\overline{\\Delta}_1+\\frac{1}{2}\\functiongapboundconstant+1+4\\log^2\\frac{T}{\\delta}+3\\eta C \\log\\frac{T}{\\delta} + \\frac{17}{64L_1}\\left(\\sqrt{L_0\\overline{\\Delta}_{l+1}}+L_1\\overline{\\Delta}_{l+1}\\right)+  \\frac{L_0}{128L_1^2}\\notag\\\\\n  \\leq &   \\overline{\\Delta}_1+\\frac{1}{2}\\functiongapboundconstant+1+4\\log^2\\frac{T}{\\delta}+\\frac{\\sqrt{2C}}{\\sqrt{L_0T}}\\log\\frac{T}{\\delta} + \\frac{7}{64}\\overline{\\Delta}_{l+1}+\\frac{289L_0}{1792L_1^2}+\\frac{17}{64}\\overline{\\Delta}_{l+1}+\\frac{L_0}{128L_1^2}.\n\\end{align}\nThe second inequality follows from Lemma \\ref{lemma 6.2} and Lemma \\ref{sqrt sum}. The third inequality follows from Young's inequality and the constraint of $\\eta$ that $\\eta\\leq \\frac{1}{2\\sqrt{\\left(L_0+L_1\\mathcal{M}_c\\right)CT}}$, where $\\mathcal{M}_c\\geq \\frac{L_0}{8L_1}$.\nRe-arranging the above inequality, we have \n\\begin{align*}\n  \\frac{5}{8}\\overline{\\Delta}_{l+1}\\leq \\overline{\\Delta}_1+\\frac{1}{2}\\functiongapboundconstant+1+4\\log^2\\frac{T}{\\delta}+\\frac{\\sqrt{2C}}{\\sqrt{L_0T}}+\\frac{303L_0}{1792L_1^2}\\leq \\frac{5}{8}\\functiongapboundconstant.\n\\end{align*}\nThe second inequality holds since \n$\\functiongapboundconstant=\\Gconstant$.\nTherefore, $P_{l+1}\\leq P_c$ also holds. \nSo the induction is complete and we obtain the desired result. \n\\end{proof}","sha256_utf8":"1b090d59789ef66835a3540a4985f4bbeae4eccab1cb7703f404b7e6719a3f9d"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"固定集中事件并启动归纳","source_units":[{"start":62981,"end":63694,"start_line":728,"end_line":731,"raw_tex":"  We assume that \\eqref{inequality high probability non-convex constant} always happens and then deduce $\\overline{\\Delta}_t\\leq \\functiongapboundconstant$, $P_t\\leq P_c$ for all $t\\in[T]$. \n  Since \\eqref{inequality high probability non-convex constant} happens with probability at least $1-\\delta$, $\\overline{\\Delta}_t\\leq \\functiongapboundconstant$ and $P_t\\leq P_c$, $\\forall t\\in[T]$,  hold with probability at least $1-\\delta$. \n  It is obvious that $f(\\overline{x}_1)-f^*\\leq \\functiongapboundconstant$. Therefore, $P_1\\leq P_c$.\n  Suppose that for some $l\\in[T]$, $$f(\\overline{x}_t)-f^*\\leq \\functiongapboundconstant,\\quad\\forall t\\in[l], \\quad \\text{thus}, \\quad P_t\\leq P_c, \\quad \\forall t\\in[l].$$ \n","sha256_utf8":"53ade454a6e41b0dbeb40fd79f428ed878544426bbb7bea23a75adb8a5ebeecd"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"由噪声假设估计随机梯度","source_units":[{"start":63694,"end":64139,"start_line":732,"end_line":738,"raw_tex":"  By the triangle inequality of the norm function,\n  \\begin{align*}\n    \\|g_t\\|\\leq  \\|g_t-\\overline{g}_t\\|+\\|\\overline{g}_t\\|\n    \\leq  \\sqrt{A\\overline{\\Delta}_t+B\\|\\overline{g}_t\\|^2+C}+\\|\\overline{g}_t\\|\n    \\leq  \\sqrt{A\\overline{\\Delta}_t}+\\left(\\sqrt{B}+1\\right)\\|\\overline{g}_t\\|+\\sqrt{C},\n  \\end{align*}\n  where the second inequality follows from Assumption \\ref{assumption 3} and the last inequality follows from Lemma \\ref{sqrt sum}.\n","sha256_utf8":"4d97e53bbea30c5e5ef38729d9dd8d17f0bac1fffca37c3b79d4b67526acd1d3"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"建立随机梯度界并定义位移尺度","source_units":[{"start":64139,"end":64741,"start_line":739,"end_line":748,"raw_tex":"  Combining with Lemma \\ref{lemma 6.2} and the assumption that $\\overline{\\Delta}_t\\leq \\functiongapboundconstant, \\forall t\\in[l]$, we have\n  \\begin{align} \n    \\label{g_t}\n    \\|g_t\\| \\leq  \\sqrt{A\\functiongapboundconstant}+2\\left(\\sqrt{B}+1\\right)\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}+\\sqrt{C}, \\quad \\forall t\\in[l].\n  \\end{align}\n  For simplicity, let \n  \\begin{align}\n    \\label{Y_c}\n    Y_c=\\eta \\left(\\sqrt{A\\functiongapboundconstant}+2\\left(\\sqrt{B}+1\\right)\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}+\\sqrt{C}\\right).\n  \\end{align}\n","sha256_utf8":"8a69a8df83e11183110447bc7f0fea0148a4bd33a2d4097dee6d4f412800af40"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"验证迭代步的局部光滑半径","source_units":[{"start":64741,"end":64973,"start_line":749,"end_line":753,"raw_tex":"    By the iteration step of Algorithm \\ref{algorithm1}, we have\n  \\begin{align*}\n  \\|x_{t+1}-x_t\\|=\\theta_t\\|g_t\\|=\\eta \\|g_t\\|\\leq Y_c\\leq 1/8L_1,\n  \\end{align*}\n  where the last inequality follows from the restriction of $\\eta$.\n","sha256_utf8":"22a310f984d2031be73523438a46d53d7ce4ffe2e7dbc6c27d728f1275dec2d3"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"展开单步函数下降","source_units":[{"start":64973,"end":65922,"start_line":754,"end_line":763,"raw_tex":" Applying Lemma \\ref{descent lemma}, we have \n\\begin{align*}\n  & f(x_{t+1})-f(x_t) \\notag\\\\\n  \\leq & \\la \\nabla f(x_t),x_{t+1}-x_t\\ra+\\frac{L_0+L_1\\nablaf{x_t}}{2}\\|x_{t+1}-x_t\\|^2\\notag\\\\\n   = & -\\theta_t\\la \\nabla f(x_t),g_t\\ra+\\frac{L_0+L_1\\nablaf{x_t}}{2}\\theta_t^2\\|g_t\\|^2\\notag\\\\\n   = & -\\theta_t\\la \\overline{g}_t+\\nabla f(x_t)-\\overline{g}_t,\\overline{g}_t+\\xi_t\\ra + \\frac{L_0+L_1\\nablaf{x_t}}{2}\\theta_t^2\\|g_t\\|^2\\notag\\\\\n   = & -\\theta_t \\|\\overline{g}_t\\|^2-\\theta_t\\la \\overline{g}_t,\\xi_t\\ra -\\theta_t \\la \\nabla f(x_t)-\\overline{g}_t,g_t\\ra+\\frac{L_0+L_1\\nablaf{x_t}}{2}\\theta_t^2\\|g_t\\|^2\\notag\\\\\n  \\leq & -\\theta_t \\|\\overline{g}_t\\|^2-\\theta_t\\la \\overline{g}_t,\\xi_t\\ra +\\theta_t \\| \\nabla f(x_t)-\\overline{g}_t\\|\\|g_t\\|+\\frac{L_0+L_1\\nablaf{x_t}}{2}\\theta_t^2\\|g_t\\|^2,\n\\end{align*} \nwhere the first equation follows from the update rule in Algorithm \\ref{algorithm1} and the last line follows from Cauchy-Schwarz inequality. \n","sha256_utf8":"b5b09e9db9614f3ef40f550d77c5f625156d1deeda300a48f2d916180d77f6ff"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"控制加速点与主迭代点的距离","source_units":[{"start":65922,"end":66794,"start_line":764,"end_line":776,"raw_tex":"Recall the constraints of the step size for RSAG with the constant step size and SGD, we have \n\\begin{align}\n  \\label{constant step size constraint}\n  0\\leq \\frac{\\gamma_t-\\theta_t}{\\alpha_t}\\leq \\theta_t=\\eta, \\quad \\forall t\\in[T].\n\\end{align}\nCombining \\eqref{overline x_t-x_t} with \\eqref{proposition 1.1}, \\eqref{g_t}, \\eqref{constant step size constraint}, we have\n\\begin{align}\n  \\label{bound overline x_t-x_t}\n  \\|\\overline{x}_t-x_t\\| \\leq & \\eta\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\|g_k\\|\\notag\\\\\n  \\leq & \\eta\\left(\\sqrt{A\\functiongapboundconstant}+2\\left(\\sqrt{B}+1\\right)\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}+\\sqrt{C}\\right)\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\notag\\\\\n  = & Y_c \\leq 1/8L_1.\n\\end{align}\nNote that \\eqref{bound overline x_t-x_t} holds for all $t\\in[l+1]$. \n","sha256_utf8":"e19b657a4bd56ea19391b081c7b45252948311532a9c68672fbcabd1d869a52f"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"建立含加速误差的下降基本式","source_units":[{"start":66794,"end":67832,"start_line":777,"end_line":786,"raw_tex":"Thus, applying Definition \\ref{definition 1}, we have that for all $t\\in[l]$,\n\\begin{align}  \n  \\label{inequality base}\n  & f(x_{t+1})-f(x_t) \\notag\\\\\n  \\leq & -\\theta_t \\|\\overline{g}_t\\|^2-\\theta_t\\la \\overline{g}_t,\\xi_t\\ra +\\theta_t\\left(L_0+L_1\\nablaf{x_t}\\right)\\|\\overline{x}_t-x_t\\|\\|g_t\\| +\\frac{L_0+L_1\\nablaf{x_t}}{2}\\theta_t^2\\|g_t\\|^2\\notag\\\\\n  \\leq & -\\theta_t\\|\\overline{g}_t\\|^2-\\theta_t\\la\\overline{g}_t,\\xi_t\\ra+\\frac{1}{2}\\left(L_0+L_1\\nablaf{x_t}\\right)\\|\\overline{x}_t-x_t\\|^2+\\left(L_0+L_1\\nablaf{x_t}\\right)\\theta_t^2\\|g_t\\|^2\\notag\\\\\n  \\leq & -\\theta_t\\|\\overline{g}_t\\|^2-\\theta_t\\la\\overline{g}_t,\\xi_t\\ra+\\frac{1}{2}\\left(L_0+L_1\\nablaf{x_t}\\right)\\left(1-\\alpha_t\\right)\\varGamma_t\\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}\\theta_k^2\\|g_k\\|^2\\notag\\\\\n  & + \\left(L_0+L_1\\nablaf{x_t}\\right)\\theta_t^2\\|g_t\\|^2,\n\\end{align}\nwhere we use Young's inequality in the second inequality and apply Proposition \\ref{proposition overline x_t - x_t} with \\eqref{constant step size constraint} in the last inequality. \n","sha256_utf8":"cece57cb727cbe0ffcad3e4a41b013890cf56802ff031e939277783241d9013f"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"累计下降式并交换双重求和","source_units":[{"start":67832,"end":68566,"start_line":787,"end_line":794,"raw_tex":"Summing over $t\\in [l]$ with $\\theta_t=\\eta$, we have\n\\begin{align}\n\\label{telescope}\nf(x_{l+1})-f(x_1)  \\leq& \\frac{1}{2}\\sum\\limits^{l}_{t=1}\\left[\\lsmooth{t}\\left(1-\\alpha_t\\right)\\varGamma_t\\sum\\limits_{k=1}^t\\frac{\\alpha_k}{\\varGamma_k}\\eta^2\\|g_k\\|^2\\right]\\notag\\\\\n&+\\eta^2\\sumlimits{t=1}{l}\\lsmooth{t}\\|g_t\\|^2-\\eta\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2-\\eta\\sumlimits{t=1}{l}\\la \\overline{g}_t,\\xi_t\\ra\\notag\\\\\n=& \\frac{1}{2}\\eta^2 \\sumlimits{t=1}{l}\\left[\\sumlimits{k=t}{l}\\lsmooth{k}\\left(1-\\alpha_k\\right)\\varGamma_k\\right]\\frac{\\alpha_t}{\\varGamma_t}\\|g_t\\|^2\\notag\\\\\n&+\\eta^2\\sumlimits{t=1}{l}\\lsmooth{t}\\|g_t\\|^2-\\eta\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2-\\eta\\sumlimits{t=1}{l}\\la \\overline{g}_t,\\xi_t\\ra.\n\\end{align}\n","sha256_utf8":"2258573e9c0afd77fc777f425f97809fb6b0302b5e12fd4954aeb7ecb6b4796a"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"控制主迭代点梯度并定义梯度常数","source_units":[{"start":68566,"end":69470,"start_line":795,"end_line":808,"raw_tex":"Applying the triangle inequality of $\\|\\cdot\\|$, we have that for all $t\\in[l]$,\n\\begin{align}\n  \\label{nabla fx_t}\n  \\nablaf{x_t} \\leq  \\|\\overline{g}_t-\\nabla f(x_t)\\|+\\|\\overline{g}_t\\| \n  \\leq  \\left(L_0+L_1\\|\\overline{g}_t\\|\\right)\\|\\overline{x}_t-x_t\\|+\\|\\overline{g}_t\\|,\n\\end{align}\nwhere the second inequality holds since $\\|\\overline{x}_t-x_t\\|\\leq 1/L_1$. Combining \\eqref{bound overline x_t-x_t} and \\eqref{nabla fx_t}, for all $t\\in[l]$,\n\\begin{align}\n  \\label{nabla fx_t constant step size}\n   \\nablaf{x_t}\n  \\leq & \\frac{1}{8L_1}\\left(L_0+2L_1\\sqrt{L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2}\\right)+2\\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}\\notag\\\\\n  = & \\frac{L_0}{8L_1}+\\frac{9}{4} \\sqrt{L_0 \\functiongapboundconstant+L_1^2 \\functiongapboundconstant^2}=\\mathcal{M}_c,\n\\end{align}\nwhere we apply Lemma \\ref{lemma 6.2} to the inequality. \n","sha256_utf8":"65eb0fdfb7f2745d140173ea131dddf72f75a2ef380b893bb39439ab7bd90240"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"随机梯度平方的噪声分解界","source_units":[{"start":69470,"end":69714,"start_line":809,"end_line":813,"raw_tex":"Applying the triangle inequality again, we have\n\\begin{align}\n  \\label{triangle square}\n  \\|g_t\\|^2\\leq 2\\|g_t-\\overline{g}_t\\|^2+2\\|\\overline{g}_t\\|^2\\leq 2\\left(A \\overline{\\Delta}_t+\\left(B+1\\right)\\|\\overline{g}_t\\|^2+C\\right).\n\\end{align}\n","sha256_utf8":"a67b96c3e35ab9d1e16babab64efafc13e290e36cf2eaf84ef5c7095348e2daf"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"合并权重、噪声与归纳上界","source_units":[{"start":69714,"end":70864,"start_line":814,"end_line":823,"raw_tex":"Combining  \\eqref{proposition 1.2}, \\eqref{inequality high probability non-convex constant}, \\eqref{telescope} and \\eqref{triangle square}, together with the bound of $\\nablaf{x_t}$ in \\eqref{nabla fx_t constant step size}, we have \n\\begin{align}\n  \\label{4.26}\nf(x_{l+1})-f(x_1)  \\leq & 2\\left(L_0+L_1\\mathcal{M}_c\\right)\\eta^2\\sumlimits{t=1}{l}\\|g_t\\|^2-\\eta\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2-\\eta\\sumlimits{t=1}{l}\\la\\overline{g}_t,\\xi_t\\ra\\notag\\\\\n\\leq & 4\\left(L_0+L_1\\mathcal{M}_c\\right)\\eta^2\\sumlimits{t=1}{l}\\left(A\\overline{\\Delta}_t+\\left(B+1\\right)\\|\\overline{g}_t\\|^2+C\\right)-\\eta\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2\\notag\\\\\n&+\\frac{1}{4}\\eta\\sumlimits{t=1}{l}\\frac{P_t^2}{P_c^2}\\|\\overline{g}_t\\|^2+3\\eta P_c^2\\log\\frac{T}{\\delta}\\notag\\\\\n\\leq & 4\\left(L_0+L_1\\mathcal{M}_c\\right)\\eta^2 lA\\functiongapboundconstant+4\\left(L_0+L_1\\mathcal{M}_c\\right)\\eta^2lC \\notag\\\\\n&+\\left(4\\left(L_0+L_1\\mathcal{M}_c\\right)\\eta\\left(B+1\\right)+\\frac{1}{4}-1\\right)\\eta\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2+3\\eta P_c^2\\log\\frac{T}{\\delta},\n\\end{align}\nwhere the last inequality holds since the assumption that $P_t\\leq P_c, \\forall t\\in[l]$. \n","sha256_utf8":"ee1908d73e2324c108d8eb0556ca85401a22d10da431fb4c70cf9676eaf9ede8"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"利用步长限制控制主点函数值","source_units":[{"start":70864,"end":71851,"start_line":824,"end_line":834,"raw_tex":"Applying the restrictions of $\\eta$ in \\eqref{eta constant non-convex}, it is apparent that \n\\begin{align*}\n  f(x_{l+1})\\leq f(x_1)+\\frac{1}{4}\\functiongapboundconstant+1+3\\eta P_c^2\\log\\frac{T}{\\delta}.\n\\end{align*}\nSince $P_c^2 = A\\functiongapboundconstant+4B\\left(L_0\\functiongapboundconstant+L_1^2\\functiongapboundconstant^2\\right)+C= \\left(A+4BL_0\\right)\\functiongapboundconstant+4BL_1^2\\functiongapboundconstant^2+C$,\n\\begin{align}\n  \\label{f x+1-f x_1 constant step size}\n  f(x_{l+1}) \\leq & f(x_1)+\\frac{1}{4}\\functiongapboundconstant+1+3\\eta \\left(A+4BL_0\\right)\\functiongapboundconstant\\log\\frac{T}{\\delta}+12\\eta BL_1^2\\functiongapboundconstant^2\\log\\frac{T}{\\delta}+3\\eta C \\log\\frac{T}{\\delta}\\notag\\\\\n  \\leq & f(x_1)+\\frac{1}{4}\\functiongapboundconstant+1+2\\sqrt{\\functiongapboundconstant}\\log\\frac{T}{\\delta}+3\\eta C\\log\\frac{T}{\\delta}     \\notag\\\\\n  \\leq & f(x_1)+\\frac{1}{2}\\functiongapboundconstant+1+ 4\\log^2\\frac{T}{\\delta}+3\\eta C \\log\\frac{T}{\\delta},\n\\end{align}\n","sha256_utf8":"b0bf0a56b3dd9c6b2534132393b980d9dc1c0aa51346d75a76940202e1694f0f"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"比较下一加速点与主点函数值","source_units":[{"start":71851,"end":72819,"start_line":835,"end_line":841,"raw_tex":"where the second inequality is due to the restrictions of $\\eta$ in \\eqref{eta constant non-convex} and the last inequality follows from Young's inequality. Next we will bound the gap between $f(x_{l+1})$ and $f(\\overline{x}_{l+1})$. By \\eqref{bound overline x_t-x_t}, we have $\\|\\overline{x}_{l+1}-x_{l+1}\\|<1/L_1$. Using Lemma \\ref{descent lemma} again,\n\\begin{align}\n  \\label{gap between f overline x_t+1 and f x_t+1 constant step size}\n  f(\\overline{x}_{l+1}) \\leq & f(x_{l+1})+\\la \\overline{g}_{l+1}, \\overline{x}_{l+1}-x_{l+1}\\ra+\\frac{L_0+L_1\\|\\overline{g}_{l+1}\\|}{2}\\|\\overline{x}_{l+1}-x_{l+1}\\|^2\\notag\\\\\n  \\leq & f(x_{l+1})+\\|\\overline{g}_{l+1}\\|\\|\\overline{x}_{l+1}-x_{l+1}\\|+\\frac{L_0+L_1\\|\\overline{g}_{l+1}\\|}{2}\\|\\overline{x}_{l+1}-x_{l+1}\\|^2,\n\\end{align}\nwhere the second inequality follows from Cauchy-Schwarz inequality. Substituting \\eqref{bound overline x_t-x_t} into \\eqref{gap between f overline x_t+1 and f x_t+1 constant step size}, we have\n","sha256_utf8":"08e104db524be5bdc9c111ae917c8d333678974fb3818f6aee827cdb61d79dc1"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"代入位移界控制函数值偏差","source_units":[{"start":72819,"end":73098,"start_line":842,"end_line":846,"raw_tex":"\\begin{align}\n  \\label{descent lemma 2}\n  f(\\overline{x}_{l+1}) \\leq & f(x_{l+1})+\\frac{1}{8L_1}\\|\\overline{g}_{l+1}\\|+\\frac{1}{128L_1^2}\\left(L_0+L_1\\|\\overline{g}_{l+1}\\|\\right)\\notag\\\\\n  = & f(x_{l+1})+\\frac{17}{128L_1}\\|\\overline{g}_{l+1}\\|+\\frac{L_0}{128L_1^2}.\n\\end{align}\n","sha256_utf8":"9ec8847d2ee3f5a45831eacdbdda9d722717cf846da9c42f0fc9988c83f496e3"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"用函数差吸收下一步梯度项","source_units":[{"start":73098,"end":74253,"start_line":847,"end_line":853,"raw_tex":"Combining \\eqref{f x+1-f x_1 constant step size} and \\eqref{descent lemma 2}, and subtracting $f^*$ from both sides, we have\n\\begin{align}\n  \\overline{\\Delta}_{l+1}\\leq & \\overline{\\Delta}_1+\\frac{1}{2}\\functiongapboundconstant+1+4\\log^2\\frac{T}{\\delta}+3\\eta C \\log\\frac{T}{\\delta} +\\frac{17}{128L_1}\\|\\overline{g}_{l+1}\\|+\\frac{L_0}{128L_1^2}\\notag\\\\\n  \\leq & \\overline{\\Delta}_1+\\frac{1}{2}\\functiongapboundconstant+1+4\\log^2\\frac{T}{\\delta}+3\\eta C \\log\\frac{T}{\\delta} + \\frac{17}{64L_1}\\left(\\sqrt{L_0\\overline{\\Delta}_{l+1}}+L_1\\overline{\\Delta}_{l+1}\\right)+  \\frac{L_0}{128L_1^2}\\notag\\\\\n  \\leq &   \\overline{\\Delta}_1+\\frac{1}{2}\\functiongapboundconstant+1+4\\log^2\\frac{T}{\\delta}+\\frac{\\sqrt{2C}}{\\sqrt{L_0T}}\\log\\frac{T}{\\delta} + \\frac{7}{64}\\overline{\\Delta}_{l+1}+\\frac{289L_0}{1792L_1^2}+\\frac{17}{64}\\overline{\\Delta}_{l+1}+\\frac{L_0}{128L_1^2}.\n\\end{align}\nThe second inequality follows from Lemma \\ref{lemma 6.2} and Lemma \\ref{sqrt sum}. The third inequality follows from Young's inequality and the constraint of $\\eta$ that $\\eta\\leq \\frac{1}{2\\sqrt{\\left(L_0+L_1\\mathcal{M}_c\\right)CT}}$, where $\\mathcal{M}_c\\geq \\frac{L_0}{8L_1}$.\n","sha256_utf8":"7b2647af32d44c598ac2a7ddcc3545c5beb02e234fc21292e998ce5e9241ada6"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"整理常数闭合函数差归纳","source_units":[{"start":74253,"end":74621,"start_line":854,"end_line":859,"raw_tex":"Re-arranging the above inequality, we have \n\\begin{align*}\n  \\frac{5}{8}\\overline{\\Delta}_{l+1}\\leq \\overline{\\Delta}_1+\\frac{1}{2}\\functiongapboundconstant+1+4\\log^2\\frac{T}{\\delta}+\\frac{\\sqrt{2C}}{\\sqrt{L_0T}}+\\frac{303L_0}{1792L_1^2}\\leq \\frac{5}{8}\\functiongapboundconstant.\n\\end{align*}\nThe second inequality holds since \n$\\functiongapboundconstant=\\Gconstant$.\n","sha256_utf8":"eda4f189e48751572dabd3aebc317603b3ab5044bd0651cce1f34427f7f0a06f"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"闭合噪声尺度归纳","source_units":[{"start":74621,"end":74738,"start_line":860,"end_line":862,"raw_tex":"Therefore, $P_{l+1}\\leq P_c$ also holds. \nSo the induction is complete and we obtain the desired result. \n\\end{proof}","sha256_utf8":"af26ff392dfdc7f3588618864b6f5d392de1f7dfcaa92e88da62a16b7379025c"}]},{"id":"S042","semantic_type":"proof","kind":"proof","title":"加速权重命题的完整证明","source_units":[{"start":193290,"end":194501,"start_line":2396,"end_line":2415,"raw_tex":"\\begin{proof}[Proof of Proposition \\ref{proposition 1}]\n  From \\cite{ghadimi2016accelerated}, we have\n  \\begin{align*}\n    \\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}=\\frac{\\alpha_1}{\\varGamma_1}+\\sumlimits{k=2}{t}\\frac{\\alpha_k}{\\varGamma_k}=\\frac{1}{\\varGamma_1}+\\sumlimits{k=2}{t}\\frac{1}{\\varGamma_k}\\left(1-\\frac{\\varGamma_{k}}{\\varGamma_{k-1}}\\right)=\\frac{1}{\\varGamma_1}+\\sumlimits{k=2}{t}\\left(\\frac{1}{\\varGamma_k}-\\frac{1}{\\varGamma_{k-1}}\\right)=\\frac{1}{\\varGamma_t}.\n  \\end{align*}\n  Therefore, \n  \\begin{align*}\n    \\varGamma_{t}\\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}=1.\n  \\end{align*}\n  For any $t\\geq 2$, we have\n \\begin{align*}\n  \\varGamma_t=\\left(1-\\alpha_t\\right)\\varGamma_{t-1}=\\frac{t-1}{t+1}\\varGamma_{t-1}=\\cdots=\\frac{t-1}{t+1}\\cdot\\frac{t-2}{t}\\cdots \\frac{2-1}{2+1}=\\frac{2}{t\\left(t+1\\right)}.\n\\end{align*}\nHence,\n  \\begin{align*}\n    \\left[\\sumlimits{k=t}{T}\\left(1-\\alpha_k\\right)\\varGamma_k\\right]\\frac{\\alpha_t}{\\varGamma_t} & =\\left[\\sumlimits{k=t}{T}\\frac{k-1}{k+1}\\frac{2}{k\\left(k+1\\right)}\\right]\\cdot t\\notag\\\\\n    \\leq & 2t \\sumlimits{k=t}{T}\\frac{1}{k\\left(k+1\\right)}\\notag\\\\\n    = & 2t\\left(\\frac{1}{t}-\\frac{1}{T+1}\\right) \\leq 2.\n  \\end{align*}\n\\end{proof}","sha256_utf8":"91464696456c79cdc59678f1c05b78476b7e76dc860e604611dbd5f469e052d7"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"权重倒数的望远镜恒等式","source_units":[{"start":193346,"end":193789,"start_line":2397,"end_line":2400,"raw_tex":"  From \\cite{ghadimi2016accelerated}, we have\n  \\begin{align*}\n    \\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}=\\frac{\\alpha_1}{\\varGamma_1}+\\sumlimits{k=2}{t}\\frac{\\alpha_k}{\\varGamma_k}=\\frac{1}{\\varGamma_1}+\\sumlimits{k=2}{t}\\frac{1}{\\varGamma_k}\\left(1-\\frac{\\varGamma_{k}}{\\varGamma_{k-1}}\\right)=\\frac{1}{\\varGamma_1}+\\sumlimits{k=2}{t}\\left(\\frac{1}{\\varGamma_k}-\\frac{1}{\\varGamma_{k-1}}\\right)=\\frac{1}{\\varGamma_t}.\n  \\end{align*}\n","sha256_utf8":"cd3c272a181b0b223547c13f1b23cfc08cda00a831a1005665ca49a76594832c"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"得到归一化权重和","source_units":[{"start":193789,"end":193902,"start_line":2401,"end_line":2404,"raw_tex":"  Therefore, \n  \\begin{align*}\n    \\varGamma_{t}\\sumlimits{k=1}{t}\\frac{\\alpha_k}{\\varGamma_k}=1.\n  \\end{align*}\n","sha256_utf8":"62fc96f265a5a0c17b5417508233165d73e111f4ac51574945265aad2b2d5d21"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"计算权重乘积的显式形式","source_units":[{"start":193902,"end":194136,"start_line":2405,"end_line":2408,"raw_tex":"  For any $t\\geq 2$, we have\n \\begin{align*}\n  \\varGamma_t=\\left(1-\\alpha_t\\right)\\varGamma_{t-1}=\\frac{t-1}{t+1}\\varGamma_{t-1}=\\cdots=\\frac{t-1}{t+1}\\cdot\\frac{t-2}{t}\\cdots \\frac{2-1}{2+1}=\\frac{2}{t\\left(t+1\\right)}.\n\\end{align*}\n","sha256_utf8":"07c04dae2c46671e856ce2ff9f6144a3cb35695528ce8f70aa39abbdde81bffb"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"控制权重尾和","source_units":[{"start":194136,"end":194501,"start_line":2409,"end_line":2415,"raw_tex":"Hence,\n  \\begin{align*}\n    \\left[\\sumlimits{k=t}{T}\\left(1-\\alpha_k\\right)\\varGamma_k\\right]\\frac{\\alpha_t}{\\varGamma_t} & =\\left[\\sumlimits{k=t}{T}\\frac{k-1}{k+1}\\frac{2}{k\\left(k+1\\right)}\\right]\\cdot t\\notag\\\\\n    \\leq & 2t \\sumlimits{k=t}{T}\\frac{1}{k\\left(k+1\\right)}\\notag\\\\\n    = & 2t\\left(\\frac{1}{t}-\\frac{1}{T+1}\\right) \\leq 2.\n  \\end{align*}\n\\end{proof}","sha256_utf8":"348b69dd58ba557648ddad249e2faf570607def1e40c6bb508e37ac610a11f1e"}]},{"id":"S047","semantic_type":"proof","kind":"proof","title":"迭代序列距离命题的完整证明","source_units":[{"start":194503,"end":196202,"start_line":2417,"end_line":2434,"raw_tex":"\\begin{proof}[Proof of Proposition \\ref{proposition overline x_t - x_t}]\n  Following from the iteration steps in Algorithm \\ref{algorithm1} and referring to the proof in \\cite{ghadimi2016accelerated}, we have \n  \\begin{align*}\n    \\overline{x}_t-x_t = & \\left(1-\\alpha_t\\right)[\\tilde{x}_t-x_t]\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)[\\overline{x}_{t-1}-x_{t-1}+\\left(\\theta_{t-1}-\\gamma_{t-1}\\right)g_{t-1}]\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)[\\left(1-\\alpha_{t-1}\\right)\\left(\\tilde{x}_{t-1}-x_{t-1}\\right)+\\left(\\theta_{t-1}-\\gamma_{t-1}\\right)g_{t-1}]\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)\\sumlimits{k=1}{t-1}\\left(\\prod\\limits_{j=k+1}^{t-1}\\left(1-\\alpha_j\\right)\\right)\\left(\\theta_k-\\gamma_k\\right)g_k\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)\\sumlimits{k=1}{t-1}\\frac{\\varGamma_{t-1}}{\\varGamma_{k}}\\left(\\theta_k-\\gamma_k\\right)g_k\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\frac{\\left(\\theta_k-\\gamma_k\\right)}{\\alpha_k}g_k.\n  \\end{align*}\n  By the convexity of square of the norm and \\eqref{proposition 1.1}, we have \n  \\begin{align}\n    \\label{overline x_t-x_t square norm}\n    \\|\\overline{x}_t-x_t\\|^2 = & \\left\\Vert\\left(1-\\alpha_t\\right)\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k} \\frac{\\left(\\theta_k-\\gamma_k\\right)}{\\alpha_k}  g_k\\right\\Vert^2\\notag\\\\\n    \\leq & \\left(1-\\alpha_t\\right)^2\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\frac{\\left(\\theta_k-\\gamma_k\\right)^2}{\\alpha_k^2}\\|g_k\\|^2\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)\\varGamma_t\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\frac{\\left(\\theta_k-\\gamma_k\\right)^2}{\\alpha_k^2}\\|g_k\\|^2.\n  \\end{align}\n\\end{proof}","sha256_utf8":"f60684d600a00876aef4a5d63c8a73da219c20255780ed4f3138db878d5c3352"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"递推展开两条序列之差","source_units":[{"start":194576,"end":195517,"start_line":2418,"end_line":2426,"raw_tex":"  Following from the iteration steps in Algorithm \\ref{algorithm1} and referring to the proof in \\cite{ghadimi2016accelerated}, we have \n  \\begin{align*}\n    \\overline{x}_t-x_t = & \\left(1-\\alpha_t\\right)[\\tilde{x}_t-x_t]\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)[\\overline{x}_{t-1}-x_{t-1}+\\left(\\theta_{t-1}-\\gamma_{t-1}\\right)g_{t-1}]\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)[\\left(1-\\alpha_{t-1}\\right)\\left(\\tilde{x}_{t-1}-x_{t-1}\\right)+\\left(\\theta_{t-1}-\\gamma_{t-1}\\right)g_{t-1}]\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)\\sumlimits{k=1}{t-1}\\left(\\prod\\limits_{j=k+1}^{t-1}\\left(1-\\alpha_j\\right)\\right)\\left(\\theta_k-\\gamma_k\\right)g_k\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)\\sumlimits{k=1}{t-1}\\frac{\\varGamma_{t-1}}{\\varGamma_{k}}\\left(\\theta_k-\\gamma_k\\right)g_k\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\frac{\\left(\\theta_k-\\gamma_k\\right)}{\\alpha_k}g_k.\n  \\end{align*}\n","sha256_utf8":"a0cc9dcbd0fad7168225840f414c8e743f9fe7cfeee23d52dd8c4f7faff9278d"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"利用平方范数凸性得到距离平方界","source_units":[{"start":195517,"end":196202,"start_line":2427,"end_line":2434,"raw_tex":"  By the convexity of square of the norm and \\eqref{proposition 1.1}, we have \n  \\begin{align}\n    \\label{overline x_t-x_t square norm}\n    \\|\\overline{x}_t-x_t\\|^2 = & \\left\\Vert\\left(1-\\alpha_t\\right)\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k} \\frac{\\left(\\theta_k-\\gamma_k\\right)}{\\alpha_k}  g_k\\right\\Vert^2\\notag\\\\\n    \\leq & \\left(1-\\alpha_t\\right)^2\\varGamma_{t-1}\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\frac{\\left(\\theta_k-\\gamma_k\\right)^2}{\\alpha_k^2}\\|g_k\\|^2\\notag\\\\\n    = & \\left(1-\\alpha_t\\right)\\varGamma_t\\sumlimits{k=1}{t-1}\\frac{\\alpha_k}{\\varGamma_k}\\frac{\\left(\\theta_k-\\gamma_k\\right)^2}{\\alpha_k^2}\\|g_k\\|^2.\n  \\end{align}\n\\end{proof}","sha256_utf8":"0ab595366a808c34d247f2afc6791beaa8088bd183b5782a7518317cede87623"}]},{"id":"S050","semantic_type":"proof","kind":"proof","title":"梯度函数差引理的完整证明","source_units":[{"start":176122,"end":176846,"start_line":2141,"end_line":2155,"raw_tex":"\\begin{proof}\n    Let $x=x_t-\\frac{1}{L_0+L_1\\|\\nabla f(x_t)\\|}\\nabla f(x_t)$.\n    It is easy to verify that $\\|x-x_t\\|\\leq 1/L_1$.\n    From Lemma \\ref{descent lemma}, we have that\n    $$f(x) \\leq f(x_t)  + \\la \\nabla f(x_t), x - x_t \\ra + {L_0+L_1 \\|\\nabla f(x)\\|  \\over 2} \\|x - x_t\\|^2.$$\n    Hence, \n    \\begin{align*}\n      &\\frac{\\|\\nabla f(x_t)\\|^2}{2\\left(L_0+L_1\\|\\nabla f(x_t)\\|\\right)}\\leq f(x_t)-f(x)\\leq \\Delta_t.\n    \\end{align*}\n  Re-arranging the above inequality and applying Cauchy-Schwarz inequality and Young's inequality, we have\n  \\begin{align*}\n    \\nablaf{x_{t}}^2 \\leq & 2L_0\\Delta_{t}+\\frac{1}{2}\\|\\nabla f(x_{t})\\|^2+2L_1^2\\Delta_{t}^2.\n  \\end{align*}\n  We obtain the desired result.\n  \\end{proof}","sha256_utf8":"8168f2acfe09b7f2db5ba004dad79fd19615b9d751a119ccc68e3be66f7a3d44"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"构造局部梯度步并应用下降引理","source_units":[{"start":176136,"end":176414,"start_line":2142,"end_line":2145,"raw_tex":"    Let $x=x_t-\\frac{1}{L_0+L_1\\|\\nabla f(x_t)\\|}\\nabla f(x_t)$.\n    It is easy to verify that $\\|x-x_t\\|\\leq 1/L_1$.\n    From Lemma \\ref{descent lemma}, we have that\n    $$f(x) \\leq f(x_t)  + \\la \\nabla f(x_t), x - x_t \\ra + {L_0+L_1 \\|\\nabla f(x)\\|  \\over 2} \\|x - x_t\\|^2.$$\n","sha256_utf8":"9c1ad0a3a7f26e6784f6a824ce81436edcb29d4570688041d7cf7b37b762270d"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"由函数下界得到梯度隐式界","source_units":[{"start":176414,"end":176566,"start_line":2146,"end_line":2149,"raw_tex":"    Hence, \n    \\begin{align*}\n      &\\frac{\\|\\nabla f(x_t)\\|^2}{2\\left(L_0+L_1\\|\\nabla f(x_t)\\|\\right)}\\leq f(x_t)-f(x)\\leq \\Delta_t.\n    \\end{align*}\n","sha256_utf8":"4ff542eb2284b8a6d4d002b4cf8574b3390133892bc1e039ccbf56cf358da6d8"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"吸收梯度项得到显式平方界","source_units":[{"start":176566,"end":176846,"start_line":2150,"end_line":2155,"raw_tex":"  Re-arranging the above inequality and applying Cauchy-Schwarz inequality and Young's inequality, we have\n  \\begin{align*}\n    \\nablaf{x_{t}}^2 \\leq & 2L_0\\Delta_{t}+\\frac{1}{2}\\|\\nabla f(x_{t})\\|^2+2L_1^2\\Delta_{t}^2.\n  \\end{align*}\n  We obtain the desired result.\n  \\end{proof}","sha256_utf8":"d78ca28efba9843051145a77375f87eb81bd87857b199e9ca29192ce7ef00f26"}]},{"id":"S054","semantic_type":"proof","kind":"proof","title":"统一噪声集中引理的完整证明","source_units":[{"start":196206,"end":197977,"start_line":2438,"end_line":2458,"raw_tex":"\\begin{proof}[Proof of Lemma \\ref{high probability sum non-convex constant}]\n  Let $Z_t=-\\la \\overline{g}_t,\\xi_t\\ra$. \n  Note that $\\overline{g}_t$ is a random variable dependent on $z_1,\\cdots,z_{t-1}$ and $\\xi_t$ is dependent on $z_1,\\cdots,z_t$. Therefore, it is apparent that $Z_t$ is a martingale difference sequence since\n  $$\\mathbb{E}\\left[- \\la \\overline{g}_t,\\xi_t\\ra|z_1,\\cdots,z_{t-1}\\right]=-\\la \\overline{g}_t,\\mathbb{E}_t[\\xi_t]\\ra=0.$$\n  Using Cauchy-Schwarz inequality,\n  \\begin{align*}\n    \\mathbb{E}_t\\left[\\exp\\left(\\frac{Z_t^2}{\\|\\overline{g}_t\\|^2\\left(A\\overline{\\Delta}_t+B\\|\\overline{g}_t\\|^2+C\\right)}\\right)\\right]\\leq\\mathbb{E}_t\\left[\\exp\\left(\\frac{\\|\\xi_t\\|^2}{A\\overline{\\Delta}_t+B\\|\\overline{g}_t\\|^2+C}\\right)\\right] \\leq \\text{e}.\n  \\end{align*}\n  Therefore, given any $l\\in [T]$, applying Lemma \\ref{lemma high probability}, we have that for any $\\lambda>0$, with probability at least $1-\\delta$,\n  \\begin{align*}\n    \\sumlimits{t=1}{l}Z_t\\leq &\\frac{3\\lambda}{4}\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2\\left(A\\overline{\\Delta}_t+B\\|\\overline{g}_t\\|^2+C\\right)+\\frac{1}{\\lambda}\\log\\frac{1}{\\delta}\\notag\\\\\n    \\leq &\\frac{3\\lambda}{4}\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2 P_t^2+\\frac{1}{\\lambda}\\log\\frac{1}{\\delta},\n  \\end{align*}\n  where the last inequality follow from Lemma \\ref{lemma 6.2} and $P_t$ defined in \\eqref{P_t}.\n For any fixed $\\lambda$, we can  re-scale over $\\delta$ and have that  with probability at least $1-\\delta$, for all $l\\in[T]$,\n \\begin{align}\n  \\label{4.14}\n  \\sumlimits{t=1}{l}-\\la\\overline{g}_t,\\xi_t\\ra\\leq \\frac{3\\lambda}{4}\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2 P_t^2+\\frac{1}{\\lambda}\\log\\frac{T}{\\delta}.\n \\end{align}\n Let $\\lambda=\\frac{1}{3 P_c^2}$, and we obtain the desired result.\n\\end{proof}","sha256_utf8":"44718ceaa37f544290d23ec24e0ff3ce716c0567f2d44dda3c41060cf9d27a8c"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"验证梯度噪声内积为鞅差","source_units":[{"start":196283,"end":196659,"start_line":2439,"end_line":2441,"raw_tex":"  Let $Z_t=-\\la \\overline{g}_t,\\xi_t\\ra$. \n  Note that $\\overline{g}_t$ is a random variable dependent on $z_1,\\cdots,z_{t-1}$ and $\\xi_t$ is dependent on $z_1,\\cdots,z_t$. Therefore, it is apparent that $Z_t$ is a martingale difference sequence since\n  $$\\mathbb{E}\\left[- \\la \\overline{g}_t,\\xi_t\\ra|z_1,\\cdots,z_{t-1}\\right]=-\\la \\overline{g}_t,\\mathbb{E}_t[\\xi_t]\\ra=0.$$\n","sha256_utf8":"d02c3c3154a69d7fda97a5da2b6e7971592adc6b8826393b29c194899da8fc8f"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"验证条件指数矩界","source_units":[{"start":196659,"end":196989,"start_line":2442,"end_line":2445,"raw_tex":"  Using Cauchy-Schwarz inequality,\n  \\begin{align*}\n    \\mathbb{E}_t\\left[\\exp\\left(\\frac{Z_t^2}{\\|\\overline{g}_t\\|^2\\left(A\\overline{\\Delta}_t+B\\|\\overline{g}_t\\|^2+C\\right)}\\right)\\right]\\leq\\mathbb{E}_t\\left[\\exp\\left(\\frac{\\|\\xi_t\\|^2}{A\\overline{\\Delta}_t+B\\|\\overline{g}_t\\|^2+C}\\right)\\right] \\leq \\text{e}.\n  \\end{align*}\n","sha256_utf8":"fdfaf8b20a7f953e3d14e4f7c424d860a44219ee005b50d0b29f3bb743960d62"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"应用集中界并代入梯度函数差界","source_units":[{"start":196989,"end":197572,"start_line":2446,"end_line":2451,"raw_tex":"  Therefore, given any $l\\in [T]$, applying Lemma \\ref{lemma high probability}, we have that for any $\\lambda>0$, with probability at least $1-\\delta$,\n  \\begin{align*}\n    \\sumlimits{t=1}{l}Z_t\\leq &\\frac{3\\lambda}{4}\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2\\left(A\\overline{\\Delta}_t+B\\|\\overline{g}_t\\|^2+C\\right)+\\frac{1}{\\lambda}\\log\\frac{1}{\\delta}\\notag\\\\\n    \\leq &\\frac{3\\lambda}{4}\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2 P_t^2+\\frac{1}{\\lambda}\\log\\frac{1}{\\delta},\n  \\end{align*}\n  where the last inequality follow from Lemma \\ref{lemma 6.2} and $P_t$ defined in \\eqref{P_t}.\n","sha256_utf8":"a0c102dd7de10b5b0ff8f8159f5250f289db215d6e4c3cdc9c1c3cd22de5a1cf"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"对所有部分和统一概率事件","source_units":[{"start":197572,"end":197898,"start_line":2452,"end_line":2456,"raw_tex":" For any fixed $\\lambda$, we can  re-scale over $\\delta$ and have that  with probability at least $1-\\delta$, for all $l\\in[T]$,\n \\begin{align}\n  \\label{4.14}\n  \\sumlimits{t=1}{l}-\\la\\overline{g}_t,\\xi_t\\ra\\leq \\frac{3\\lambda}{4}\\sumlimits{t=1}{l}\\|\\overline{g}_t\\|^2 P_t^2+\\frac{1}{\\lambda}\\log\\frac{T}{\\delta}.\n \\end{align}\n","sha256_utf8":"b3843ffa9a298dd791c668c30129bf46944dc77db7663c75451d5eb24675e2bb"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"选取集中参数得到所需界","source_units":[{"start":197898,"end":197977,"start_line":2457,"end_line":2458,"raw_tex":" Let $\\lambda=\\frac{1}{3 P_c^2}$, and we obtain the desired result.\n\\end{proof}","sha256_utf8":"31b0a8f6f6c5a570b217dfe7a19322985893a0b9967fbb953a22f17c187cb69b"}]},{"id":"S060","semantic_type":"external_fact","kind":"external_fact","title":"引用的广义光滑下降引理","source_units":[],"declared_external":true,"declaration_text":["若 f 为 (L0,L1)-光滑且 ||x-y||≤1/L1，则 |f(y)-f(x)-〈∇f(x),y-x〉|≤(L0+L1||∇f(x)||)||x-y||²/2；这是 B0338 所引用的 Zhang 等人的 Remark 2.3。"]},{"id":"S061","semantic_type":"external_fact","kind":"external_fact","title":"引用的鞅差集中不等式","source_units":[],"declared_external":true,"declaration_text":["若鞅差序列满足 E_t exp(Z_t²/σ_t²)≤e，且 σ_t 对过去可测，则任意固定 λ>0 下，以至少 1-δ 的概率有 ΣZ_t≤(3λ/4)Σσ_t²+λ⁻¹log(1/δ)；这是 B0340 所引用的 Li 与 Orabona 的 Lemma 1。"]},{"id":"S062","semantic_type":"external_fact","kind":"external_fact","title":"标准平方根次可加性","source_units":[],"declared_external":true,"declaration_text":["任意有限非负数列满足 √(Σa_i)≤Σ√a_i。"]},{"id":"S063","semantic_type":"external_fact","kind":"external_fact","title":"基本范数不等式与并集界","source_units":[],"declared_external":true,"declaration_text":["欧氏范数满足三角不等式与 Cauchy–Schwarz 不等式；实数乘积可用 Young 不等式控制；平方范数为凸函数；有限个事件的失败概率不超过各失败概率之和。"]},{"id":"S064","semantic_type":"definition","kind":"definition","title":"确定性函数差上界常数","source_units":[{"start":62856,"end":62947,"start_line":722,"end_line":725,"raw_tex":"\\begin{align}\n    \\label{Gconstant}\n    \\functiongapboundconstant=\\Gconstant.\n  \\end{align}","sha256_utf8":"32338567636a42632924719ed0f6d0b0636ef9c0e59e251b7f67ce714a6221c6"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S017","target_id":"S001","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S017","kind":"supports"},{"source_id":"S020","target_id":"S017","kind":"supports"},{"source_id":"S021","target_id":"S017","kind":"supports"},{"source_id":"S022","target_id":"S017","kind":"supports"},{"source_id":"S023","target_id":"S017","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S004","target_id":"S019","kind":"supports"},{"source_id":"S006","target_id":"S019","kind":"supports"},{"source_id":"S008","target_id":"S019","kind":"supports"},{"source_id":"S008","target_id":"S022","kind":"supports"},{"source_id":"S008","target_id":"S023","kind":"supports"},{"source_id":"S015","target_id":"S018","kind":"supports"},{"source_id":"S015","target_id":"S020","kind":"supports"},{"source_id":"S012","target_id":"S021","kind":"supports"},{"source_id":"S016","target_id":"S021","kind":"supports"},{"source_id":"S031","target_id":"S018","kind":"supports"},{"source_id":"S033","target_id":"S021","kind":"supports"},{"source_id":"S034","target_id":"S021","kind":"supports"},{"source_id":"S024","target_id":"S016","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S024","kind":"supports"},{"source_id":"S027","target_id":"S024","kind":"supports"},{"source_id":"S028","target_id":"S024","kind":"supports"},{"source_id":"S029","target_id":"S024","kind":"supports"},{"source_id":"S030","target_id":"S024","kind":"supports"},{"source_id":"S031","target_id":"S024","kind":"supports"},{"source_id":"S032","target_id":"S024","kind":"supports"},{"source_id":"S033","target_id":"S024","kind":"supports"},{"source_id":"S034","target_id":"S024","kind":"supports"},{"source_id":"S035","target_id":"S024","kind":"supports"},{"source_id":"S036","target_id":"S024","kind":"supports"},{"source_id":"S037","target_id":"S024","kind":"supports"},{"source_id":"S038","target_id":"S024","kind":"supports"},{"source_id":"S039","target_id":"S024","kind":"supports"},{"source_id":"S040","target_id":"S024","kind":"supports"},{"source_id":"S041","target_id":"S024","kind":"supports"},{"source_id":"S015","target_id":"S025","kind":"supports"},{"source_id":"S064","target_id":"S025","kind":"supports"},{"source_id":"S004","target_id":"S026","kind":"supports"},{"source_id":"S011","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S014","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S005","target_id":"S028","kind":"supports"},{"source_id":"S008","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S009","target_id":"S029","kind":"supports"},{"source_id":"S005","target_id":"S029","kind":"supports"},{"source_id":"S006","target_id":"S029","kind":"supports"},{"source_id":"S008","target_id":"S030","kind":"supports"},{"source_id":"S012","target_id":"S030","kind":"supports"},{"source_id":"S013","target_id":"S030","kind":"supports"},{"source_id":"S027","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S003","target_id":"S031","kind":"supports"},{"source_id":"S013","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S008","target_id":"S032","kind":"supports"},{"source_id":"S003","target_id":"S033","kind":"supports"},{"source_id":"S030","target_id":"S033","kind":"supports"},{"source_id":"S014","target_id":"S033","kind":"supports"},{"source_id":"S025","target_id":"S033","kind":"supports"},{"source_id":"S004","target_id":"S034","kind":"supports"},{"source_id":"S006","target_id":"S034","kind":"supports"},{"source_id":"S012","target_id":"S035","kind":"supports"},{"source_id":"S015","target_id":"S035","kind":"supports"},{"source_id":"S025","target_id":"S035","kind":"supports"},{"source_id":"S032","target_id":"S035","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S008","target_id":"S036","kind":"supports"},{"source_id":"S015","target_id":"S036","kind":"supports"},{"source_id":"S030","target_id":"S037","kind":"supports"},{"source_id":"S009","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S030","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S014","target_id":"S039","kind":"supports"},{"source_id":"S011","target_id":"S039","kind":"supports"},{"source_id":"S008","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S064","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S015","target_id":"S041","kind":"supports"},{"source_id":"S042","target_id":"S012","kind":"supports"},{"source_id":"S043","target_id":"S042","kind":"supports"},{"source_id":"S044","target_id":"S042","kind":"supports"},{"source_id":"S045","target_id":"S042","kind":"supports"},{"source_id":"S046","target_id":"S042","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S047","target_id":"S013","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S049","target_id":"S047","kind":"supports"},{"source_id":"S005","target_id":"S048","kind":"supports"},{"source_id":"S012","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S012","target_id":"S049","kind":"supports"},{"source_id":"S050","target_id":"S014","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S052","target_id":"S050","kind":"supports"},{"source_id":"S053","target_id":"S050","kind":"supports"},{"source_id":"S003","target_id":"S051","kind":"supports"},{"source_id":"S009","target_id":"S051","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S004","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S054","target_id":"S015","kind":"supports"},{"source_id":"S055","target_id":"S054","kind":"supports"},{"source_id":"S056","target_id":"S054","kind":"supports"},{"source_id":"S057","target_id":"S054","kind":"supports"},{"source_id":"S058","target_id":"S054","kind":"supports"},{"source_id":"S059","target_id":"S054","kind":"supports"},{"source_id":"S004","target_id":"S055","kind":"supports"},{"source_id":"S005","target_id":"S055","kind":"supports"},{"source_id":"S006","target_id":"S055","kind":"supports"},{"source_id":"S004","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S057","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S010","target_id":"S057","kind":"supports"},{"source_id":"S014","target_id":"S057","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S064","target_id":"S059","kind":"supports"},{"source_id":"S060","target_id":"S009","kind":"supports"},{"source_id":"S061","target_id":"S010","kind":"supports"},{"source_id":"S062","target_id":"S011","kind":"supports"},{"source_id":"S063","target_id":"S029","kind":"supports"},{"source_id":"S063","target_id":"S030","kind":"supports"},{"source_id":"S063","target_id":"S031","kind":"supports"},{"source_id":"S063","target_id":"S033","kind":"supports"},{"source_id":"S063","target_id":"S034","kind":"supports"},{"source_id":"S063","target_id":"S036","kind":"supports"},{"source_id":"S063","target_id":"S037","kind":"supports"},{"source_id":"S063","target_id":"S039","kind":"supports"},{"source_id":"S063","target_id":"S049","kind":"supports"},{"source_id":"S063","target_id":"S053","kind":"supports"},{"source_id":"S063","target_id":"S056","kind":"supports"},{"source_id":"S063","target_id":"S058","kind":"supports"},{"source_id":"S007","target_id":"S064","kind":"supports"},{"source_id":"S007","target_id":"S032","kind":"supports"},{"source_id":"S002","target_id":"S005","kind":"supports"}]}}
{"record_id":"2502.16020v1","paper":{"title":"Interior-point algorithms with full Newton steps for nonsymmetric convex conic optimization","authors":["Dávid Papp","Anita Varga"],"arxiv_id":"2502.16020","version":"v1","journal":"SIAM Journal on Optimization","doi":"10.1137/25m1736396","abstract_url":"https://arxiv.org/abs/2502.16020v1","pdf_url":"https://arxiv.org/pdf/2502.16020v1"},"context":{"raw_tex":"\\begin{equation} \\label{eq:PD_problem}\n\\left.\n\\begin{aligned}\n    \\inf\\;\\, & \\mathbf{c}^\\top \\mathbf{x} \\\\\n    \\st\\  & A\\mathbf{x} = \\mathbf{b} \\\\\n    & \\mathbf{x} \\in \\mathcal{K}\n\\end{aligned}\n\\quad \\right\\} \\quad \\text{(P)}\n\\qquad \\qquad \\qquad \\left.\n\\begin{aligned}\n    \\sup \\;\\, & \\mathbf{b}^\\top \\mathbf{y} \\\\\n    \\st\\;\\; & A^\\top \\mathbf{y} + \\mathbf{s} = \\mathbf{c} \\\\\n    & \\mathbf{s} \\in \\mathcal{K}^*, \\  \\mathbf{y} \\in \\mathbb{R}^m \\\\\n\\end{aligned}\n\\quad \\right\\}  \\quad\\text{(D)}\n\\end{equation}\n\nwhere $\\mathbf{c} \\in \\mathbb{R}^n$ and $\\mathbf{b} \\in \\mathbb{R}^m$ are given vectors, $A \\in \\mathbb{R}^{m \\times n}$ is a given matrix with full row rank, and $\\mathcal{K} \\subseteq \\mathbb{R}^n$ is a proper convex cone whose dual cone is denoted by $\\mathcal{K}^*$. The only assumption we make about $\\K$ is that its interior, denoted by $\\K^\\circ$, is the domain of a known logarithmically homogeneous self-concordant barrier function (LHSCB), see Appendix \\ref{sec:lhscb}, whose gradient and Hessian are computable efficiently. The cone need not have any other desirable properties; in particular, we do not assume that $\\K$ is symmetric, homogeneous, or hyperbolic. Nor do we make any assumptions about its dual.\n\n\nLet $\\mathcal{F}$ denote the set of primal-dual feasible solutions:\n\\[\n\\mathcal{F} = \\left\\{ (\\mathbf{x}, \\mathbf{y}, \\mathbf{s})\\in\\mathcal{K}\\times\\mathbb{R}^m\\times\\mathcal{K}^* \\mid A\\mathbf{x} = \\mathbf{b}, \\ A^{\\top}\\mathbf{y} + \\mathbf{s} = \\mathbf{c} \\right\\},\n\\]\n\nand let $\\mathcal{F}^{\\circ}$ be the set of strictly feasible solutions:\n\\[\n\\mathcal{F}^{\\circ} = \\left\\{ (\\mathbf{x}, \\mathbf{y}, \\mathbf{s})\\in\\mathcal{K}^\\circ\\times\\mathbb{R}^m\\times(\\mathcal{K}^*)^\\circ \\mid A\\mathbf{x} = \\mathbf{b}, \\ A^{\\top}\\mathbf{y} + \\mathbf{s} = \\mathbf{c} \\right\\}.\n\\]\n\nSuppose that the interior point (aka.~Slater) condition is satisfied: $\\mathcal{F}^{\\circ} \\neq \\emptyset$. Then strong duality holds, and the optimality conditions can be formulated as\n\\begin{equation}\n\\left.\n\\begin{aligned}\n(\\vx,\\vy,\\vs)&\\in\\mathcal{F}\\\\\n\\vx^\\T\\vs&=0\n\\end{aligned} \\right\\}. \\tag{OC}\\label{eq:OC}\n\\end{equation}\n\nWe assume that an LHSCB $f: \\mathcal{K}^{\\circ} \\rightarrow \\mathbb{R}$ with barrier parameter $\\nu$ is given for the primal cone $\\mathcal{K}$. Let $g(\\mathbf{x})$ and $H(\\mathbf{x})$ denote the gradient and Hessian of $f$ at $\\mathbf{x} \\in \\mathcal{K}^{\\circ}$, respectively. With this notation in place, we define the central path problem, parametrized by the positive scalar $\\tau$, called the \\emph{central path parameter}, as follows:\n\\begin{equation}\n\\left.\n\\begin{aligned}\n(\\vx,\\vy,\\vs)&\\in\\mathcal{F}^\\circ\\\\\n    \\mathbf{s} + \\tau g(\\mathbf{x}) \\ & = \\mathbf{0} \\\\\n\\end{aligned} \\right\\}. \\tag{CP}\\label{eq:CP}\n\\end{equation}\n\nIn the analysis,  we consider the following \\emph{neighborhood} of the central path:\n\\begin{equation}\n\\label{eq:nbd}\n\\mathcal{N}(\\eta,\\tau) = \\left\\{ (\\mathbf{x}, \\mathbf{y}, \\mathbf{s}) \\in \\mathcal{F}^\\circ : \\| \\mathbf{s} + \\tau g(\\mathbf{x}) \\|_{\\mathbf{x}}^* \\leq \\eta \\tau \\right\\},\n\\end{equation}\n\nGiven an $(\\mathbf{x},\\mathbf{y},\\mathbf{s})\\in\\mathcal{F}^\\circ$, the \\emph{Newton system} associated with the central path \\eqref{eq:CP} is\n\\begin{equation}\n\\left.\n\\begin{aligned}\n    A \\Delta \\mathbf{x} & = \\mathbf{0} \\\\\n    A^{\\top} \\Delta \\mathbf{y} + \\Delta \\mathbf{s} & = \\mathbf{0} \\\\\n    \\tau H(\\mathbf{x}) \\Delta \\mathbf{x} + \\Delta \\mathbf{s} & = -(\\mathbf{s} + \\tau g(\\mathbf{x}))\n\\end{aligned} \\right\\} .\n\\tag{NS}\\label{eq:NS}\n\\end{equation}\n\nIts solution ($\\Delta \\mathbf{x}$, $\\Delta \\mathbf{y}$, $\\Delta \\mathbf{s}$)\nis called the \\emph{Newton direction} at $(\\vx,\\vy,\\vs)$.\n\n\n\\begin{algorithm}[h]\n\\caption{A full-step IPA for nonsymmetric cone programming}\n\\label{alg:simple_short_step}\n\\begin{algorithmic}\n\\vspace{0.5em}\n\\STATE \\textbf{inputs:}\n$A \\in \\mathbb{R}^{m \\times n}$ with full row rank; $\\mathbf{b} \\in \\mathbb{R}^m$;  $\\mathbf{c} \\in \\mathbb{R}^n$;\n\\\\ \\hspace{1em}an oracle for computing the Hessian $H$ of the $\\nu$-LHSCB $f: \\ \\mathcal{K}^{\\circ} \\rightarrow \\mathbb{R}$;\n\\\\ \\hspace{1em}an initial $\\tau_0>0$ and\n$(\\mathbf{x}_0, \\mathbf{y}_0, \\mathbf{s}_0) \\in \\mathcal{F}^\\circ$ such that $\\| \\mathbf{s}_0 + \\tau_0 g(\\mathbf{x}_0) \\|_{\\mathbf{x}_0}^* \\leq \\eta \\tau_0$;\n\\\\ \\hspace{1em}$\\varepsilon > 0$ optimality tolerance.\n\n\\vspace{0.5em}\n\\STATE \\textbf{parameters:}\nneighborhood parameter $\\eta \\in (0, 1)$;\nupdate parameter $\\vartheta$.\n\\vspace{0.5em}\n\n\\STATE \\textbf{output:} a triplet $(\\vx,\\vy,\\vs)\\in\\mathcal{F}^\\circ$ satisfying $\\vx^\\T\\vs\\leq\\varepsilon$.\n\n\\vspace{1em}\n\\STATE Initialize $\\mathbf{x} := \\mathbf{x}_0$, $\\mathbf{y} := \\mathbf{y}_0$, $\\mathbf{s} := \\mathbf{s}_0$, $\\tau := \\tau_0$.\n\\vspace{0.5em}\n\\WHILE{$\\mathbf{x}^\\top \\mathbf{s} > \\varepsilon$}\n    \\STATE Solve \\eqref{eq:NS} for $(\\Delta \\mathbf{x}, \\Delta \\mathbf{y}, \\Delta \\mathbf{s})$.\n    \\STATE Set $\\mathbf{x} := \\mathbf{x} + \\Delta \\mathbf{x}$, $\\mathbf{y} := \\mathbf{y} + \\Delta \\mathbf{y}$, $\\mathbf{s} := \\mathbf{s} + \\Delta \\mathbf{s}$.\n    \\STATE Set $\\tau := (1 - \\vartheta) \\tau$.\n\\ENDWHILE\n\\vspace{0.5em}\n\\STATE \\textbf{return} $(\\mathbf{x}, \\mathbf{y}, \\mathbf{s})$.\n\\vspace{0.5em}\n\\end{algorithmic}\n\\end{algorithm}\n\n\\begin{definition}[Self-concordant barrier]\nLet $\\K\\subseteq\\Rn$ be a convex cone with a non-empty interior $\\K^\\circ$. A function $f:\\K^\\circ\\to\\R$ is called a \\emph{strongly nondegenerate, self-concordant barrier} for $\\K$ if $f$ is strictly convex, at least three times differentiable, and has the following two properties (with $\\operatorname{bd}(\\K)$ denoting the boundary of $\\K$, and $g$ and $H$ denoting the gradient and Hessian of $f$):\n\\begin{enumerate}[\\hspace{1em} 1.]\n\\item $f(\\vx)\\to\\infty$  as $\\vx\\to\\operatorname{bd}(\\K)$,\n\\item $|D^3 f(\\vx)[\\vh,\\vh,\\vh]| \\leq 2\\left(D^2f(\\vx)[\\vh,\\vh]\\right)^{3/2}$,\n\\item $\\nu := \\sup_{\\vx\\in\\K^\\circ} g(\\vx)^\\T H(\\vx)^{-1}g(\\vx) < \\infty.$\n\\end{enumerate}\nWe refer to the parameter $\\nu$ above as the \\emph{barrier parameter} of $f$; we also use the shorthand \\emph{$\\nu$-SCB} to mean ``an SCB with barrier parameter $\\nu$''.\n\\end{definition}\n\n\\begin{definition}[Logarithmic homogeneity and LHSCBs]\n    Let $f: \\ \\mathcal{K}^{\\circ} \\rightarrow \\mathbb{R}$ be a $\\nu$-SCB function for the proper convex cone $ \\mathcal{K}$.  We say that $f$ is a \\emph{logarithmically homogeneous self-concordant barrier (LHSCB)} with barrier parameter $\\nu$ for $\\mathcal{K}$, or a \\emph{$\\nu$-LHSCB} for short, if it satisfies the \\emph{logarithmic homogeneity property}\n    \\[ f(t \\mathbf{x}) = f(\\mathbf{x}) - \\nu \\ln (t) \\quad \\text{for all } \\mathbf{x} \\in \\mathcal{K}^{\\circ} \\text{ and } t > 0.\\]\n\\end{definition}\n\nWe associate with each $\\vx\\in\\K^\\circ$ the \\emph{local inner product} $\\langle\\cdot,\\cdot\\rangle_\\vx : \\K \\times \\K \\to \\R$ defined as $\\langle \\vu,\\vv\\rangle_\\vx := \\vu^\\T H(\\vx) \\vv$ and the \\emph{local norm} $\\|\\cdot\\|_\\vx$ induced by this local inner product. Thus, $\\|\\vv\\|_\\vx = \\|H(\\vx)^{1/2}\\vv\\|$. We define the local (open) ball centered at $\\vx$ with radius $r$ by $\\mathcal{B}(\\vx,r) := \\{\\vv\\in\\R^n\\,|\\, \\|\\vv-\\vx\\|_{\\vx} < r\\}$. When $r=1$, these balls are also called the \\emph{Dikin ellipsoids}. Crucially, for every $\\vx\\in\\K^\\circ$, we have $\\mathcal{B}(\\vx,1)\\subset\\K^\\circ$.\n\nAnalogously, we define the \\emph{dual local inner product} $\\langle \\cdot,\\cdot \\rangle_\\vx^*: \\K^*\\times\\K^*\\to\\R$ by  $\\langle \\vs, \\vt \\rangle_\\vx^* := \\vs^\\T H(\\vx)^{-1}\\vt$, its induced \\emph{dual local norm} $\\|\\cdot\\|_{\\vx}^*$, and the corresponding open balls $\\mathcal{B}^*(\\vx,r)$. The dual local norm, by definition, satisfies the identity $\\|\\vt\\|_{\\vx}^* = \\|H(\\vx)^{-1/2}\\vt\\|$.\n","units":[{"start":3783,"end":4290,"start_line":75,"end_line":90,"raw_tex":"\\begin{equation} \\label{eq:PD_problem}\n\\left.\n\\begin{aligned}\n    \\inf\\;\\, & \\mathbf{c}^\\top \\mathbf{x} \\\\\n    \\st\\  & A\\mathbf{x} = \\mathbf{b} \\\\\n    & \\mathbf{x} \\in \\mathcal{K}\n\\end{aligned}\n\\quad \\right\\} \\quad \\text{(P)}\n\\qquad \\qquad \\qquad \\left.\n\\begin{aligned}\n    \\sup \\;\\, & \\mathbf{b}^\\top \\mathbf{y} \\\\\n    \\st\\;\\; & A^\\top \\mathbf{y} + \\mathbf{s} = \\mathbf{c} \\\\\n    & \\mathbf{s} \\in \\mathcal{K}^*, \\  \\mathbf{y} \\in \\mathbb{R}^m \\\\\n\\end{aligned}\n\\quad \\right\\}  \\quad\\text{(D)}\n\\end{equation}","sha256_utf8":"fcbe4bc416581530a1925f9e9339165a3162325c22aa0d8a2366266ec5b4197a"},{"start":4291,"end":5012,"start_line":91,"end_line":91,"raw_tex":"where $\\mathbf{c} \\in \\mathbb{R}^n$ and $\\mathbf{b} \\in \\mathbb{R}^m$ are given vectors, $A \\in \\mathbb{R}^{m \\times n}$ is a given matrix with full row rank, and $\\mathcal{K} \\subseteq \\mathbb{R}^n$ is a proper convex cone whose dual cone is denoted by $\\mathcal{K}^*$. The only assumption we make about $\\K$ is that its interior, denoted by $\\K^\\circ$, is the domain of a known logarithmically homogeneous self-concordant barrier function (LHSCB), see Appendix \\ref{sec:lhscb}, whose gradient and Hessian are computable efficiently. The cone need not have any other desirable properties; in particular, we do not assume that $\\K$ is symmetric, homogeneous, or hyperbolic. Nor do we make any assumptions about its dual.\n","sha256_utf8":"45fb6b6bceaf8df40db46fb36ca4b2168df1177ccfc98eb413a746dfe9d844a9"},{"start":14676,"end":14947,"start_line":129,"end_line":132,"raw_tex":"Let $\\mathcal{F}$ denote the set of primal-dual feasible solutions:\n\\[\n\\mathcal{F} = \\left\\{ (\\mathbf{x}, \\mathbf{y}, \\mathbf{s})\\in\\mathcal{K}\\times\\mathbb{R}^m\\times\\mathcal{K}^* \\mid A\\mathbf{x} = \\mathbf{b}, \\ A^{\\top}\\mathbf{y} + \\mathbf{s} = \\mathbf{c} \\right\\},\n\\]","sha256_utf8":"97103c9e5761290f6c609376681720817b08c0ec1db95a7341fae58c17c38de9"},{"start":14948,"end":15246,"start_line":133,"end_line":136,"raw_tex":"and let $\\mathcal{F}^{\\circ}$ be the set of strictly feasible solutions:\n\\[\n\\mathcal{F}^{\\circ} = \\left\\{ (\\mathbf{x}, \\mathbf{y}, \\mathbf{s})\\in\\mathcal{K}^\\circ\\times\\mathbb{R}^m\\times(\\mathcal{K}^*)^\\circ \\mid A\\mathbf{x} = \\mathbf{b}, \\ A^{\\top}\\mathbf{y} + \\mathbf{s} = \\mathbf{c} \\right\\}.\n\\]","sha256_utf8":"abdf5de9b4e279e817bf459decf3d335e8f5a6b2d01ae48851f62027ed2a7894"},{"start":15247,"end":15577,"start_line":137,"end_line":144,"raw_tex":"Suppose that the interior point (aka.~Slater) condition is satisfied: $\\mathcal{F}^{\\circ} \\neq \\emptyset$. Then strong duality holds, and the optimality conditions can be formulated as\n\\begin{equation}\n\\left.\n\\begin{aligned}\n(\\vx,\\vy,\\vs)&\\in\\mathcal{F}\\\\\n\\vx^\\T\\vs&=0\n\\end{aligned} \\right\\}. \\tag{OC}\\label{eq:OC}\n\\end{equation}","sha256_utf8":"f026961469bcd5ba2de30ec0598142977032d60e4d19775474bb64df80c65537"},{"start":15579,"end":16214,"start_line":146,"end_line":153,"raw_tex":"We assume that an LHSCB $f: \\mathcal{K}^{\\circ} \\rightarrow \\mathbb{R}$ with barrier parameter $\\nu$ is given for the primal cone $\\mathcal{K}$. Let $g(\\mathbf{x})$ and $H(\\mathbf{x})$ denote the gradient and Hessian of $f$ at $\\mathbf{x} \\in \\mathcal{K}^{\\circ}$, respectively. With this notation in place, we define the central path problem, parametrized by the positive scalar $\\tau$, called the \\emph{central path parameter}, as follows:\n\\begin{equation}\n\\left.\n\\begin{aligned}\n(\\vx,\\vy,\\vs)&\\in\\mathcal{F}^\\circ\\\\\n    \\mathbf{s} + \\tau g(\\mathbf{x}) \\ & = \\mathbf{0} \\\\\n\\end{aligned} \\right\\}. \\tag{CP}\\label{eq:CP}\n\\end{equation}","sha256_utf8":"cbb752c5b525d1a44b5e8a2a5be45f67c64d40db1dde824405937e31366622a7"},{"start":16416,"end":16719,"start_line":156,"end_line":160,"raw_tex":"In the analysis,  we consider the following \\emph{neighborhood} of the central path:\n\\begin{equation}\n\\label{eq:nbd}\n\\mathcal{N}(\\eta,\\tau) = \\left\\{ (\\mathbf{x}, \\mathbf{y}, \\mathbf{s}) \\in \\mathcal{F}^\\circ : \\| \\mathbf{s} + \\tau g(\\mathbf{x}) \\|_{\\mathbf{x}}^* \\leq \\eta \\tau \\right\\},\n\\end{equation}","sha256_utf8":"1ef94e2cdfa3bd6409b45b8435e297c81f3384a8a609474300e34d757b5d4efa"},{"start":17495,"end":17949,"start_line":163,"end_line":172,"raw_tex":"Given an $(\\mathbf{x},\\mathbf{y},\\mathbf{s})\\in\\mathcal{F}^\\circ$, the \\emph{Newton system} associated with the central path \\eqref{eq:CP} is\n\\begin{equation}\n\\left.\n\\begin{aligned}\n    A \\Delta \\mathbf{x} & = \\mathbf{0} \\\\\n    A^{\\top} \\Delta \\mathbf{y} + \\Delta \\mathbf{s} & = \\mathbf{0} \\\\\n    \\tau H(\\mathbf{x}) \\Delta \\mathbf{x} + \\Delta \\mathbf{s} & = -(\\mathbf{s} + \\tau g(\\mathbf{x}))\n\\end{aligned} \\right\\} .\n\\tag{NS}\\label{eq:NS}\n\\end{equation}","sha256_utf8":"e858a9d55bdb4a45a38eea0417fd528d089e56f315eb8467f81aa9b2bde8719c"},{"start":17950,"end":18085,"start_line":173,"end_line":174,"raw_tex":"Its solution ($\\Delta \\mathbf{x}$, $\\Delta \\mathbf{y}$, $\\Delta \\mathbf{s}$)\nis called the \\emph{Newton direction} at $(\\vx,\\vy,\\vs)$.\n","sha256_utf8":"30c7b8bbd175e9bec24154f3abba6f3f0206d2b8e316dce1f6cf50685fa8c0fc"},{"start":18699,"end":20248,"start_line":178,"end_line":210,"raw_tex":"\\begin{algorithm}[h]\n\\caption{A full-step IPA for nonsymmetric cone programming}\n\\label{alg:simple_short_step}\n\\begin{algorithmic}\n\\vspace{0.5em}\n\\STATE \\textbf{inputs:}\n$A \\in \\mathbb{R}^{m \\times n}$ with full row rank; $\\mathbf{b} \\in \\mathbb{R}^m$;  $\\mathbf{c} \\in \\mathbb{R}^n$;\n\\\\ \\hspace{1em}an oracle for computing the Hessian $H$ of the $\\nu$-LHSCB $f: \\ \\mathcal{K}^{\\circ} \\rightarrow \\mathbb{R}$;\n\\\\ \\hspace{1em}an initial $\\tau_0>0$ and\n$(\\mathbf{x}_0, \\mathbf{y}_0, \\mathbf{s}_0) \\in \\mathcal{F}^\\circ$ such that $\\| \\mathbf{s}_0 + \\tau_0 g(\\mathbf{x}_0) \\|_{\\mathbf{x}_0}^* \\leq \\eta \\tau_0$;\n\\\\ \\hspace{1em}$\\varepsilon > 0$ optimality tolerance.\n\n\\vspace{0.5em}\n\\STATE \\textbf{parameters:}\nneighborhood parameter $\\eta \\in (0, 1)$;\nupdate parameter $\\vartheta$.\n\\vspace{0.5em}\n\n\\STATE \\textbf{output:} a triplet $(\\vx,\\vy,\\vs)\\in\\mathcal{F}^\\circ$ satisfying $\\vx^\\T\\vs\\leq\\varepsilon$.\n\n\\vspace{1em}\n\\STATE Initialize $\\mathbf{x} := \\mathbf{x}_0$, $\\mathbf{y} := \\mathbf{y}_0$, $\\mathbf{s} := \\mathbf{s}_0$, $\\tau := \\tau_0$.\n\\vspace{0.5em}\n\\WHILE{$\\mathbf{x}^\\top \\mathbf{s} > \\varepsilon$}\n    \\STATE Solve \\eqref{eq:NS} for $(\\Delta \\mathbf{x}, \\Delta \\mathbf{y}, \\Delta \\mathbf{s})$.\n    \\STATE Set $\\mathbf{x} := \\mathbf{x} + \\Delta \\mathbf{x}$, $\\mathbf{y} := \\mathbf{y} + \\Delta \\mathbf{y}$, $\\mathbf{s} := \\mathbf{s} + \\Delta \\mathbf{s}$.\n    \\STATE Set $\\tau := (1 - \\vartheta) \\tau$.\n\\ENDWHILE\n\\vspace{0.5em}\n\\STATE \\textbf{return} $(\\mathbf{x}, \\mathbf{y}, \\mathbf{s})$.\n\\vspace{0.5em}\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"54dcf6ef43fe880bb1ceaa7f8aa2e4704c129ea1a77f4662f015d689f9516a48"},{"start":94107,"end":95003,"start_line":1003,"end_line":1011,"raw_tex":"\\begin{definition}[Self-concordant barrier]\nLet $\\K\\subseteq\\Rn$ be a convex cone with a non-empty interior $\\K^\\circ$. A function $f:\\K^\\circ\\to\\R$ is called a \\emph{strongly nondegenerate, self-concordant barrier} for $\\K$ if $f$ is strictly convex, at least three times differentiable, and has the following two properties (with $\\operatorname{bd}(\\K)$ denoting the boundary of $\\K$, and $g$ and $H$ denoting the gradient and Hessian of $f$):\n\\begin{enumerate}[\\hspace{1em} 1.]\n\\item $f(\\vx)\\to\\infty$  as $\\vx\\to\\operatorname{bd}(\\K)$,\n\\item $|D^3 f(\\vx)[\\vh,\\vh,\\vh]| \\leq 2\\left(D^2f(\\vx)[\\vh,\\vh]\\right)^{3/2}$,\n\\item $\\nu := \\sup_{\\vx\\in\\K^\\circ} g(\\vx)^\\T H(\\vx)^{-1}g(\\vx) < \\infty.$\n\\end{enumerate}\nWe refer to the parameter $\\nu$ above as the \\emph{barrier parameter} of $f$; we also use the shorthand \\emph{$\\nu$-SCB} to mean ``an SCB with barrier parameter $\\nu$''.\n\\end{definition}","sha256_utf8":"884217bf405df639aefa4a2fb7aa07605fd7f5e61973e2d08b8fa22b34ef5296"},{"start":95005,"end":95565,"start_line":1013,"end_line":1016,"raw_tex":"\\begin{definition}[Logarithmic homogeneity and LHSCBs]\n    Let $f: \\ \\mathcal{K}^{\\circ} \\rightarrow \\mathbb{R}$ be a $\\nu$-SCB function for the proper convex cone $ \\mathcal{K}$.  We say that $f$ is a \\emph{logarithmically homogeneous self-concordant barrier (LHSCB)} with barrier parameter $\\nu$ for $\\mathcal{K}$, or a \\emph{$\\nu$-LHSCB} for short, if it satisfies the \\emph{logarithmic homogeneity property}\n    \\[ f(t \\mathbf{x}) = f(\\mathbf{x}) - \\nu \\ln (t) \\quad \\text{for all } \\mathbf{x} \\in \\mathcal{K}^{\\circ} \\text{ and } t > 0.\\]\n\\end{definition}","sha256_utf8":"ee24932e14fbbb79973ed2f0154585e0ea95e838c4ce5fee031249c26e6d576c"},{"start":95567,"end":96558,"start_line":1018,"end_line":1020,"raw_tex":"We associate with each $\\vx\\in\\K^\\circ$ the \\emph{local inner product} $\\langle\\cdot,\\cdot\\rangle_\\vx : \\K \\times \\K \\to \\R$ defined as $\\langle \\vu,\\vv\\rangle_\\vx := \\vu^\\T H(\\vx) \\vv$ and the \\emph{local norm} $\\|\\cdot\\|_\\vx$ induced by this local inner product. Thus, $\\|\\vv\\|_\\vx = \\|H(\\vx)^{1/2}\\vv\\|$. We define the local (open) ball centered at $\\vx$ with radius $r$ by $\\mathcal{B}(\\vx,r) := \\{\\vv\\in\\R^n\\,|\\, \\|\\vv-\\vx\\|_{\\vx} < r\\}$. When $r=1$, these balls are also called the \\emph{Dikin ellipsoids}. Crucially, for every $\\vx\\in\\K^\\circ$, we have $\\mathcal{B}(\\vx,1)\\subset\\K^\\circ$.\n\nAnalogously, we define the \\emph{dual local inner product} $\\langle \\cdot,\\cdot \\rangle_\\vx^*: \\K^*\\times\\K^*\\to\\R$ by  $\\langle \\vs, \\vt \\rangle_\\vx^* := \\vs^\\T H(\\vx)^{-1}\\vt$, its induced \\emph{dual local norm} $\\|\\cdot\\|_{\\vx}^*$, and the corresponding open balls $\\mathcal{B}^*(\\vx,r)$. The dual local norm, by definition, satisfies the identity $\\|\\vt\\|_{\\vx}^* = \\|H(\\vx)^{-1/2}\\vt\\|$.\n","sha256_utf8":"a7661ac8da44247fddd6d4c58e49097cbc01d9dcdf272a39cc7dcd4df4e8aff8"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem} \\label{theorem:compl1}\n\nIf $0 < \\eta \\leq \\frac{1}{4}$ and $\n\\vartheta = \\frac{\\eta / 2}{\\sqrt{\\nu} + 1}$, then Algorithm \\ref{alg:simple_short_step} produces a feasible solution to \\eqref{eq:PD_problem} with duality gap below $\\varepsilon>0$ in no more than $\\frac{2}{\\eta}\\left(\\sqrt{\\nu}+1 \\right)  \\ln \\left(\\frac{\\tau_0 \\nu}{\\varepsilon}\\right) +1$\niterations.\n\nIn particular, if the initial point satisfies $\\tau_0 = \\frac{\\vx_0^\\T\\vs_0}{\\nu}$, as in the initialization strategies discussed in Section \\ref{sec:init}, then the required number of iterations is\n\\[\\mathcal{O}\\left(\\sqrt\\nu\\ln\\frac{\\vx_0^\\T\\vs_0}{\\varepsilon}\\right).\\]\n\\end{theorem}","units":[{"start":29530,"end":30199,"start_line":337,"end_line":345,"raw_tex":"\\begin{theorem} \\label{theorem:compl1}\n\nIf $0 < \\eta \\leq \\frac{1}{4}$ and $\n\\vartheta = \\frac{\\eta / 2}{\\sqrt{\\nu} + 1}$, then Algorithm \\ref{alg:simple_short_step} produces a feasible solution to \\eqref{eq:PD_problem} with duality gap below $\\varepsilon>0$ in no more than $\\frac{2}{\\eta}\\left(\\sqrt{\\nu}+1 \\right)  \\ln \\left(\\frac{\\tau_0 \\nu}{\\varepsilon}\\right) +1$\niterations.\n\nIn particular, if the initial point satisfies $\\tau_0 = \\frac{\\vx_0^\\T\\vs_0}{\\nu}$, as in the initialization strategies discussed in Section \\ref{sec:init}, then the required number of iterations is\n\\[\\mathcal{O}\\left(\\sqrt\\nu\\ln\\frac{\\vx_0^\\T\\vs_0}{\\varepsilon}\\right).\\]\n\\end{theorem}","sha256_utf8":"71eea6017f2ca4a09777a4ce59afce1810b0bb268b4da2abaf9d94bac4063a21"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}[\\cite{PappYildiz2017, Serrano2015,SkajaaYe2015}]\\label{lemma:sd_properties}\nLet $(\\mathbf{x}, \\mathbf{y}, \\mathbf{s}) \\in \\mathcal{N}(\\eta, \\tau)$ and let $(\\Delta \\mathbf{x}, \\Delta \\mathbf{y}, \\Delta \\mathbf{s})$ be the solution of \\eqref{eq:NS}. Then\n\\begin{enumerate}[label=(\\roman*)]\n    \\item  $\\Delta \\mathbf{x}^{\\top} \\Delta \\mathbf{s} = 0$, \\label{sd_prop:first}\n    \\item $\n\\| \\Delta \\mathbf{x} \\|_{\\mathbf{x}} \\leq \\eta$ and $\\| \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^* \\leq \\eta \\tau\n$,  \\label{sd_prop:second}\n    \\item $\\| \\mathbf{s} + \\tau g(\\mathbf{x}) + \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^* \\leq \\eta \\tau$. \\label{sd_prop:third}\n\\end{enumerate}\n\\end{lemma}\n\n\\begin{proof}\n    \\begin{enumerate}[label=(\\roman*)]\n        \\item Using the first two equations of \\eqref{eq:NS}, we get\n    \\[ 0 = \\Delta \\mathbf{x}^{\\top} (A^{\\top} \\Delta \\mathbf{y} + \\Delta \\mathbf{s}) = \\Delta \\mathbf{y}^{\\top} A \\Delta \\mathbf{x} + \\Delta \\mathbf{x}^{\\top} \\Delta \\mathbf{s} = \\Delta \\mathbf{x}^{\\top} \\Delta \\mathbf{s}. \\]\n        \\item Using the neighborhood definition, the orthogonality of $\\Delta \\mathbf{x}$ and $\\Delta \\mathbf{s}$, and the third equality of the Newton system, we can derive the upper bounds the following way:\n\\[ \\begin{aligned}\n\\eta^2 \\tau^2 &\\geq \\left( \\| \\mathbf{s} + \\tau g(\\mathbf{x}) \\|_{\\mathbf{x}}^* \\right)^2 = \\left( \\| \\tau H(\\mathbf{x}) \\Delta \\mathbf{x} + \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^* \\right)^2 \\\\\n&= \\tau^2 \\Delta \\mathbf{x}^\\top H(\\mathbf{x}) \\Delta \\mathbf{x} + \\Delta \\mathbf{s}^\\top H(\\mathbf{x})^{-1} \\Delta \\mathbf{s}\n= \\tau^2 \\| \\Delta \\mathbf{x} \\|_{\\mathbf{x}}^2 + (\\| \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^*)^2.\n\\end{aligned}\n\\]\n\nAs both terms on the right-hand side are nonnegative, it follows that $\\| \\Delta \\mathbf{x} \\|_{\\mathbf{x}} \\leq \\eta$ and $\\| \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^* \\leq \\eta \\tau$.\n    \\item Using the third equation of \\eqref{eq:NS} and the upper bound on $\\|\\Delta \\mathbf{x}\\|_{\\mathbf{x}}$ from \\ref{sd_prop:second}, we get\n    \\[\n\\| \\mathbf{s} + \\tau g(\\mathbf{x}) + \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^* = \\| \\tau H(\\mathbf{x}) \\Delta \\mathbf{x} \\|_{\\mathbf{x}}^*\n= \\tau \\| \\Delta \\mathbf{x} \\|_{\\mathbf{x}} \\leq \\eta \\tau.\n  \\]\n\n    \\end{enumerate}\n\\end{proof}\n\n\\begin{lemma}\n\\label{lemma:zplus-in-Nthetaplus}\nLet $(\\mathbf{x}, \\mathbf{y}, \\mathbf{s}) \\in \\mathcal{N}(\\eta, \\tau)$, let $(\\Delta \\mathbf{x}, \\Delta \\mathbf{y}, \\Delta \\mathbf{s})$ be the solution of \\eqref{eq:NS}, and let $(\\mathbf{x}^+, \\mathbf{y}^+, \\mathbf{s}^+) = (\\mathbf{x}, \\mathbf{y}, \\mathbf{s}) + (\\Delta \\mathbf{x}, \\Delta \\mathbf{y}, \\Delta \\mathbf{s})$. If $0 < \\eta \\leq \\frac{1}{4}$ and $\n\\vartheta = \\frac{\\eta / 2}{\\sqrt{\\nu} + 1}$,\nthen the new point satisfies $(\\mathbf{x}^+, \\mathbf{y}^+, \\mathbf{s}^+) \\in \\mathcal{N}(\\eta, \\tau^+)$ with $\\tau^+ = (1 - \\vartheta) \\tau$.\n\\end{lemma}\n\n\\begin{proof}\n    Let us define the function $\\varphi : \\mathcal{K}^\\circ \\to \\mathbb{R}$:\n\\[\n\\varphi(\\mathbf{v}) = \\frac{1}{\\tau} (\\mathbf{s} + \\Delta \\mathbf{s})^\\top \\mathbf{v} + f(\\mathbf{v}),\n\\]\nwhere $f$ is the known LHSCB for $\\mathcal{K}^\\circ$, with gradient $g$ and Hessian $H$.\n\nThen $\\varphi$ is self-concordant, and its Hessian is also $H$.\nLet $\\mathbf{n}_{\\varphi}(\\mathbf{v})$ denote the (unconstrained) Newton step for $\\varphi$ at $\\mathbf{v} \\in \\mathcal{K}^\\circ$:\n\\[\n\\mathbf{n}_{\\varphi}(\\mathbf{v}) = -\\tau^{-1} H(\\mathbf{v})^{-1} \\left( \\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{v}) \\right).\n\\]\nThus,\n\\[\n\\mathbf{n}_{\\varphi}(\\mathbf{x}) = -\\tau^{-1} H(\\mathbf{x})^{-1} \\left( \\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{x}) \\right) = \\Delta \\mathbf{x},\n\\]\nand therefore,\n\\[\n\\mathbf{x}^+ = \\mathbf{x} + \\Delta \\mathbf{x} = \\mathbf{x} + \\mathbf{n}_{\\varphi}(\\mathbf{x}).\n\\]\n\nFor any $\\mathbf{v} \\in \\mathcal{K}^\\circ$,\n\\[\n\\|\\mathbf{n}_{\\varphi}(\\mathbf{v})\\|_{\\mathbf{v}} = \\left\\| \\tau^{-1} H(\\mathbf{v})^{-1} \\left( \\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{v}) \\right) \\right\\|_{\\mathbf{v}}\n= \\tau^{-1} \\|\\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{v})\\|_{\\mathbf{v}}^*.\n\\]\nFor $\\mathbf{v} = \\mathbf{x}$, $\\|\\mathbf{n}_{\\varphi}(\\mathbf{x})\\|_{\\mathbf{x}} = \\|\\Delta \\mathbf{x}\\|_{\\mathbf{x}} \\leq \\eta$, using Lemma \\ref{lemma:sd_properties}, part \\ref{sd_prop:second}. As $\\eta < 1$, we can use \\ref{prop:scb_NS}:\n\\begin{equation} \\label{eq:UB_NS}\n    \\tau^{-1} \\|\\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{x}^+)\\|^*_{\\mathbf{x}^+} = \\|\\mathbf{n}_{\\varphi}(\\mathbf{x}^+)\\|_{\\mathbf{x}^+}\n\\leq \\left( \\frac{\\|\\mathbf{n}_{\\varphi}(\\mathbf{x})\\|_{\\mathbf{x}}}{1 - \\|\\mathbf{n}_{\\varphi}(\\mathbf{x})\\|_{\\mathbf{x}}} \\right)^2 \\leq \\frac{\\eta^2}{(1 - \\eta)^2}.\n\\end{equation}\n\nThus,\n\\begin{align*}\n\\frac{1}{\\tau^+} \\left\\| \\mathbf{s}^+ + \\tau^+ g(\\mathbf{x}^+) \\right\\|_{\\mathbf{x}^+}^*\n&= \\frac{1}{1 - \\vartheta} \\tau^{-1} \\| \\mathbf{s} + \\Delta \\mathbf{s} + \\tau^+ g(\\mathbf{x}^+) \\|_{\\mathbf{x}^+}^* \\\\\n&\\leq \\frac{1}{1 - \\vartheta} \\tau^{-1} \\| \\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{x}^+)\\|_{\\mathbf{x}^+}^* + \\frac{\\vartheta}{1 - \\vartheta} \\| g(\\mathbf{x}^+) \\|^*_{\\mathbf{x}^+} \\\\\n&\\stackrel{\\eqref{eq:UB_NS},\\ref{prop:lhscb_norm}}{\\leq} \\frac{1}{1 - \\vartheta}  \\frac{\\eta^2}{(1 - \\eta)^2} + \\frac{\\vartheta}{1 - \\vartheta} \\sqrt{\\nu} \\\\\n&= \\frac{\\sqrt{\\nu} + 1}{\\sqrt{\\nu} + 1 - \\eta/2} \\cdot \\frac{\\eta^2}{(1 - \\eta)^2} + \\frac{\\eta / 2}{\\sqrt{\\nu} + 1 - \\eta/2} \\sqrt{\\nu} \\\\\n&\\leq \\frac{2}{2 - \\eta / 2} \\cdot \\frac{\\eta^2}{(1 - \\eta)^2} + \\frac{\\eta}{2}.\n\\end{align*}\n\nThe last expression is less than $\\eta$ if $\\eta \\leq \\frac{1}{4}$ holds.\n\\end{proof}\n\n\\begin{lemma} \\label{lemma:DG}\nLet $(\\mathbf{x}, \\mathbf{y}, \\mathbf{s}) \\in \\mathcal{N}(\\eta, \\tau)$ and let $(\\Delta \\mathbf{x}, \\Delta \\mathbf{y}, \\Delta \\mathbf{s})$ be the solution of \\eqref{eq:NS}. We have the following lower and upper bounds on the duality gap after a full Newton step:\n    \\[ \\tau(\\nu - \\eta^2) \\leq (\\mathbf{x}^+)^\\top \\mathbf{s}^+ \\leq \\tau \\nu. \\]\n\\end{lemma}\n\n\\begin{proof}\nUsing \\ref{sd_prop:first} from Lemma \\ref{lemma:sd_properties},\n    \\[ (\\mathbf{x}^+)^\\top \\mathbf{s}^+ = (\\mathbf{x} + \\Delta \\mathbf{x})^\\top (\\mathbf{s} + \\Delta \\mathbf{s}) = \\mathbf{x}^\\top \\mathbf{s} + \\mathbf{s}^\\top \\Delta \\mathbf{x} + \\mathbf{x}^\\top \\Delta \\mathbf{s}. \\]\nFrom the third equation of \\eqref{eq:NS}, we have\n\\begin{align*}\n    \\Delta \\mathbf{x}^\\top \\mathbf{s} &= \\Delta \\mathbf{x}^\\top \\mathbf{s} + \\Delta \\mathbf{x}^\\top \\Delta \\mathbf{s} \\\\\n&= \\Delta \\mathbf{x}^\\top \\mathbf{s} - \\Delta \\mathbf{x}^\\top (\\mathbf{s} + \\tau g(\\mathbf{x}) + \\tau H(\\mathbf{x}) \\Delta \\mathbf{x}) \\\\\n&= -\\tau (\\Delta \\mathbf{x}^\\top g(\\mathbf{x}) + \\|\\Delta \\mathbf{x}\\|_{\\mathbf{x}}^2).\n\\end{align*}\nSimilarly, we can express $\\mathbf{x}^\\top \\Delta \\mathbf{s}$ as\n\\begin{align*}\n\\mathbf{x}^\\top \\Delta \\mathbf{s} &= - \\mathbf{x}^\\top \\left( \\mathbf{s} + \\tau g(\\mathbf{x}) + \\tau H(\\mathbf{x}) \\Delta \\mathbf{x} \\right) \\\\\n&\\stackrel{\\ref{prop:lhscb_Hessian},\\ref{prop:lhscb_gradient}}{=} -\\mathbf{x}^\\top \\mathbf{s} + \\tau \\nu + \\tau \\Delta \\mathbf{x}^\\top g(\\mathbf{x}).\n\\end{align*}\n\nCombining the previous equations, we can write $(\\mathbf{x}^+)^\\top \\mathbf{s}^+$ as\n\\begin{align*}\n(\\mathbf{x}^+)^\\top \\mathbf{s}^+ &= \\mathbf{x}^\\top \\mathbf{s} - \\tau \\Delta \\mathbf{x}^\\top g(\\mathbf{x}) - \\tau \\|\\Delta \\mathbf{x}\\|_{\\mathbf{x}}^2 - \\mathbf{x}^\\top \\mathbf{s} + \\tau \\nu + \\tau \\Delta \\mathbf{x}^\\top g(\\mathbf{x}) \\\\\n&= \\tau (\\nu -  \\|\\Delta \\mathbf{x}\\|_{\\mathbf{x}}^2).\n\\end{align*}\nUsing part \\ref{sd_prop:second} of Lemma \\ref{lemma:sd_properties}, we get\n\\[ \\tau(\\nu-\\eta^2) \\leq (\\mathbf{x}^+)^\\top \\mathbf{s}^+ \\leq \\tau\\nu.  \\]\n\n\\end{proof}\n\n\\begin{proof}\n    At the beginning of the $k^{\\text{th}}$ iteration, $\\tau = \\tau_k = (1-\\vartheta)^{k-1} \\tau_0$. According to Lemma \\ref{lemma:DG}, we have\n    \\[ \\mathbf{x}_k^{\\top} \\mathbf{s}_k  \\leq \\tau_k \\nu = (1-\\vartheta)^{k-1} \\tau_0 \\nu. \\]\n    We would like to find the smallest $k$ for which  $(1-\\vartheta)^{k-1} \\tau_0 \\nu \\leq \\varepsilon$ holds, which is equivalent to\n    \\[ (k-1) \\ln (1-\\vartheta) + \\ln ( \\tau_0 \\nu ) \\leq \\ln \\varepsilon.\\]\n    Using the inequality $-\\ln(1 - \\vartheta) \\geq \\vartheta$ for $\\vartheta \\in [0,1)$, we can require the fulfillment of the stronger inequality\n    \\[ -(k-1) \\vartheta + \\ln (\\tau_0 \\nu ) \\leq \\ln \\varepsilon,\\]\nwhich is true if\n\\[\nk \\geq \\frac{1}{\\vartheta} \\ln \\left( \\frac{\\tau_0 \\nu}{\\varepsilon} \\right) +1\n= \\frac{2}{\\eta} \\left( \\sqrt{\\nu} + 1 \\right)  \\ln \\left(  \\frac{\\tau_0\\nu}{\\varepsilon} \\right) +1.\n\\]\n\n\\end{proof}\n\n\\begin{proposition}[\\protect{\\cite[Chapters 2--3]{Renegar2001}, \\cite[Chapter 5]{Nesterov2018}, and \\cite[Lemma 1]{Nesterov2012}}]\n    Let $f$ be a $\\nu$-SCB for the proper convex cone $\\mathcal{K}$, let $\\vx\\in\\K^\\circ$, and let $\\mathbf{n} (\\mathbf{x} ) = - H (\\mathbf{x})^{-1} g (\\mathbf{x})$ be the (unconstrained) Newton step for $f$ at $\\mathbf{x}$.\n       \\setlist[enumerate,1]{label=\\textup{(A\\arabic*)}, leftmargin=5em, labelsep=0.5em, itemindent=0pt}\n    \\begin{enumerate}\n        \\item \\label{prop:scb_NS}\n        If $\\|\\mathbf{n}(\\mathbf{x})\\|_{\\mathbf{x}} < 1$, then $\\mathbf{x}^+ := \\mathbf{x} + \\mathbf{n} (\\mathbf{x})$ satisfies\n\\[\n\\|\\mathbf{n} (\\mathbf{x}^+)\\|_{\\mathbf{x}^+} \\leq \\left( \\frac{\\|\\mathbf{n}(\\mathbf{x})\\|_{\\mathbf{x}}}{1 - \\|\\mathbf{n}(\\mathbf{x})\\|_{\\mathbf{x}}} \\right)^2.\n\\]\n    \\item \\label{prop:scb_norm1}\n   If $\\mathbf{u} \\in \\mathcal{B}_{\\mathbf{x}}(\\mathbf{x},1)$, then $\\vu\\in\\K^\\circ$, and for all $\\mathbf{v} \\neq \\mathbf{0}$, one has\n\\[\n\\frac{\\|\\mathbf{v}\\|_{\\mathbf{u}}^*}{\\|\\mathbf{v}\\|_{\\mathbf{x}}^*} \\leq \\frac{1}{1 - \\|\\mathbf{u} - \\mathbf{x}\\|_{\\mathbf{x}}}.\n\\]\n\\item \\label{prop:scb_norm2} Let $\\mathbf{x}, \\mathbf{u} \\in \\mathcal{K}^\\circ$, and let $r:= \\| \\mathbf{x} - \\mathbf{u} \\|_\\mathbf{u} < 1$. Then\n\\[ \\| g(\\mathbf{x}) - g(\\mathbf{u}) - H(\\mathbf{u}) (\\mathbf{x} - \\mathbf{u}) \\|_{\\mathbf{u}}^* \\leq \\frac{r^2}{1-r}. \\]\n    \\end{enumerate}\n\\end{proposition}\n\n\\begin{proposition}[\\protect{\\cite[Chapters 2--3]{Renegar2001}, \\cite[Chapter 5]{Nesterov2018}}]\n    Let $f$ be a $\\nu$-LHSCB for the proper convex cone $\\mathcal{K}$ and let $\\vx\\in\\K^{\\circ}$. Then:\n       \\setlist[enumerate,1]{label=\\textup{(A\\arabic*)}, leftmargin=5em, labelsep=0.5em, itemindent=0pt, start=4}\n    \\begin{enumerate}\n        \\item \\label{prop:lhscb_Hessian} $H(\\mathbf{x}) \\mathbf{x} = - g(\\mathbf{x})$;\n        \\item \\label{prop:lhscb_gradient} $ g(\\mathbf{x})^{\\top} \\mathbf{x} = - \\nu$;\n        \\item \\label{prop:lhscb_lh} $g( \\alpha \\mathbf{x}) = \\alpha^{-1} g(\\mathbf{x})$ and $H( \\alpha \\mathbf{x} ) = \\alpha^{-2} H(\\mathbf{x})$ for all $\\alpha > 0$;\n        \\item  \\label{prop:lhscb_norm} $\\| g(\\mathbf{x}) \\|_x^* = \\sqrt{\\nu}$.\n    \\end{enumerate}\n\\end{proposition}","units":[{"start":21207,"end":21889,"start_line":216,"end_line":225,"raw_tex":"\\begin{lemma}[\\cite{PappYildiz2017, Serrano2015,SkajaaYe2015}]\\label{lemma:sd_properties}\nLet $(\\mathbf{x}, \\mathbf{y}, \\mathbf{s}) \\in \\mathcal{N}(\\eta, \\tau)$ and let $(\\Delta \\mathbf{x}, \\Delta \\mathbf{y}, \\Delta \\mathbf{s})$ be the solution of \\eqref{eq:NS}. Then\n\\begin{enumerate}[label=(\\roman*)]\n    \\item  $\\Delta \\mathbf{x}^{\\top} \\Delta \\mathbf{s} = 0$, \\label{sd_prop:first}\n    \\item $\n\\| \\Delta \\mathbf{x} \\|_{\\mathbf{x}} \\leq \\eta$ and $\\| \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^* \\leq \\eta \\tau\n$,  \\label{sd_prop:second}\n    \\item $\\| \\mathbf{s} + \\tau g(\\mathbf{x}) + \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^* \\leq \\eta \\tau$. \\label{sd_prop:third}\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"8f0572e0eef658e86453ea49858979f6138cacf9f28410afaf159768a21d6c29"},{"start":21891,"end":23460,"start_line":227,"end_line":247,"raw_tex":"\\begin{proof}\n    \\begin{enumerate}[label=(\\roman*)]\n        \\item Using the first two equations of \\eqref{eq:NS}, we get\n    \\[ 0 = \\Delta \\mathbf{x}^{\\top} (A^{\\top} \\Delta \\mathbf{y} + \\Delta \\mathbf{s}) = \\Delta \\mathbf{y}^{\\top} A \\Delta \\mathbf{x} + \\Delta \\mathbf{x}^{\\top} \\Delta \\mathbf{s} = \\Delta \\mathbf{x}^{\\top} \\Delta \\mathbf{s}. \\]\n        \\item Using the neighborhood definition, the orthogonality of $\\Delta \\mathbf{x}$ and $\\Delta \\mathbf{s}$, and the third equality of the Newton system, we can derive the upper bounds the following way:\n\\[ \\begin{aligned}\n\\eta^2 \\tau^2 &\\geq \\left( \\| \\mathbf{s} + \\tau g(\\mathbf{x}) \\|_{\\mathbf{x}}^* \\right)^2 = \\left( \\| \\tau H(\\mathbf{x}) \\Delta \\mathbf{x} + \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^* \\right)^2 \\\\\n&= \\tau^2 \\Delta \\mathbf{x}^\\top H(\\mathbf{x}) \\Delta \\mathbf{x} + \\Delta \\mathbf{s}^\\top H(\\mathbf{x})^{-1} \\Delta \\mathbf{s}\n= \\tau^2 \\| \\Delta \\mathbf{x} \\|_{\\mathbf{x}}^2 + (\\| \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^*)^2.\n\\end{aligned}\n\\]\n\nAs both terms on the right-hand side are nonnegative, it follows that $\\| \\Delta \\mathbf{x} \\|_{\\mathbf{x}} \\leq \\eta$ and $\\| \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^* \\leq \\eta \\tau$.\n    \\item Using the third equation of \\eqref{eq:NS} and the upper bound on $\\|\\Delta \\mathbf{x}\\|_{\\mathbf{x}}$ from \\ref{sd_prop:second}, we get\n    \\[\n\\| \\mathbf{s} + \\tau g(\\mathbf{x}) + \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^* = \\| \\tau H(\\mathbf{x}) \\Delta \\mathbf{x} \\|_{\\mathbf{x}}^*\n= \\tau \\| \\Delta \\mathbf{x} \\|_{\\mathbf{x}} \\leq \\eta \\tau.\n  \\]\n\n    \\end{enumerate}\n\\end{proof}","sha256_utf8":"1cafa11156314525b85fa802174a49d0cd674174502d29ac0011c6bba97a5288"},{"start":23685,"end":24292,"start_line":251,"end_line":256,"raw_tex":"\\begin{lemma}\n\\label{lemma:zplus-in-Nthetaplus}\nLet $(\\mathbf{x}, \\mathbf{y}, \\mathbf{s}) \\in \\mathcal{N}(\\eta, \\tau)$, let $(\\Delta \\mathbf{x}, \\Delta \\mathbf{y}, \\Delta \\mathbf{s})$ be the solution of \\eqref{eq:NS}, and let $(\\mathbf{x}^+, \\mathbf{y}^+, \\mathbf{s}^+) = (\\mathbf{x}, \\mathbf{y}, \\mathbf{s}) + (\\Delta \\mathbf{x}, \\Delta \\mathbf{y}, \\Delta \\mathbf{s})$. If $0 < \\eta \\leq \\frac{1}{4}$ and $\n\\vartheta = \\frac{\\eta / 2}{\\sqrt{\\nu} + 1}$,\nthen the new point satisfies $(\\mathbf{x}^+, \\mathbf{y}^+, \\mathbf{s}^+) \\in \\mathcal{N}(\\eta, \\tau^+)$ with $\\tau^+ = (1 - \\vartheta) \\tau$.\n\\end{lemma}","sha256_utf8":"9d57772d4bbdc874590c5a3de82660ec67f22c698438ccf418858f7a732d7364"},{"start":24294,"end":27017,"start_line":258,"end_line":301,"raw_tex":"\\begin{proof}\n    Let us define the function $\\varphi : \\mathcal{K}^\\circ \\to \\mathbb{R}$:\n\\[\n\\varphi(\\mathbf{v}) = \\frac{1}{\\tau} (\\mathbf{s} + \\Delta \\mathbf{s})^\\top \\mathbf{v} + f(\\mathbf{v}),\n\\]\nwhere $f$ is the known LHSCB for $\\mathcal{K}^\\circ$, with gradient $g$ and Hessian $H$.\n\nThen $\\varphi$ is self-concordant, and its Hessian is also $H$.\nLet $\\mathbf{n}_{\\varphi}(\\mathbf{v})$ denote the (unconstrained) Newton step for $\\varphi$ at $\\mathbf{v} \\in \\mathcal{K}^\\circ$:\n\\[\n\\mathbf{n}_{\\varphi}(\\mathbf{v}) = -\\tau^{-1} H(\\mathbf{v})^{-1} \\left( \\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{v}) \\right).\n\\]\nThus,\n\\[\n\\mathbf{n}_{\\varphi}(\\mathbf{x}) = -\\tau^{-1} H(\\mathbf{x})^{-1} \\left( \\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{x}) \\right) = \\Delta \\mathbf{x},\n\\]\nand therefore,\n\\[\n\\mathbf{x}^+ = \\mathbf{x} + \\Delta \\mathbf{x} = \\mathbf{x} + \\mathbf{n}_{\\varphi}(\\mathbf{x}).\n\\]\n\nFor any $\\mathbf{v} \\in \\mathcal{K}^\\circ$,\n\\[\n\\|\\mathbf{n}_{\\varphi}(\\mathbf{v})\\|_{\\mathbf{v}} = \\left\\| \\tau^{-1} H(\\mathbf{v})^{-1} \\left( \\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{v}) \\right) \\right\\|_{\\mathbf{v}}\n= \\tau^{-1} \\|\\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{v})\\|_{\\mathbf{v}}^*.\n\\]\nFor $\\mathbf{v} = \\mathbf{x}$, $\\|\\mathbf{n}_{\\varphi}(\\mathbf{x})\\|_{\\mathbf{x}} = \\|\\Delta \\mathbf{x}\\|_{\\mathbf{x}} \\leq \\eta$, using Lemma \\ref{lemma:sd_properties}, part \\ref{sd_prop:second}. As $\\eta < 1$, we can use \\ref{prop:scb_NS}:\n\\begin{equation} \\label{eq:UB_NS}\n    \\tau^{-1} \\|\\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{x}^+)\\|^*_{\\mathbf{x}^+} = \\|\\mathbf{n}_{\\varphi}(\\mathbf{x}^+)\\|_{\\mathbf{x}^+}\n\\leq \\left( \\frac{\\|\\mathbf{n}_{\\varphi}(\\mathbf{x})\\|_{\\mathbf{x}}}{1 - \\|\\mathbf{n}_{\\varphi}(\\mathbf{x})\\|_{\\mathbf{x}}} \\right)^2 \\leq \\frac{\\eta^2}{(1 - \\eta)^2}.\n\\end{equation}\n\nThus,\n\\begin{align*}\n\\frac{1}{\\tau^+} \\left\\| \\mathbf{s}^+ + \\tau^+ g(\\mathbf{x}^+) \\right\\|_{\\mathbf{x}^+}^*\n&= \\frac{1}{1 - \\vartheta} \\tau^{-1} \\| \\mathbf{s} + \\Delta \\mathbf{s} + \\tau^+ g(\\mathbf{x}^+) \\|_{\\mathbf{x}^+}^* \\\\\n&\\leq \\frac{1}{1 - \\vartheta} \\tau^{-1} \\| \\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{x}^+)\\|_{\\mathbf{x}^+}^* + \\frac{\\vartheta}{1 - \\vartheta} \\| g(\\mathbf{x}^+) \\|^*_{\\mathbf{x}^+} \\\\\n&\\stackrel{\\eqref{eq:UB_NS},\\ref{prop:lhscb_norm}}{\\leq} \\frac{1}{1 - \\vartheta}  \\frac{\\eta^2}{(1 - \\eta)^2} + \\frac{\\vartheta}{1 - \\vartheta} \\sqrt{\\nu} \\\\\n&= \\frac{\\sqrt{\\nu} + 1}{\\sqrt{\\nu} + 1 - \\eta/2} \\cdot \\frac{\\eta^2}{(1 - \\eta)^2} + \\frac{\\eta / 2}{\\sqrt{\\nu} + 1 - \\eta/2} \\sqrt{\\nu} \\\\\n&\\leq \\frac{2}{2 - \\eta / 2} \\cdot \\frac{\\eta^2}{(1 - \\eta)^2} + \\frac{\\eta}{2}.\n\\end{align*}\n\nThe last expression is less than $\\eta$ if $\\eta \\leq \\frac{1}{4}$ holds.\n\\end{proof}","sha256_utf8":"74c83afcd11d969d738c1ca4f1d3778cf8bcd93ba84180096e268f0b399066f2"},{"start":27273,"end":27660,"start_line":305,"end_line":308,"raw_tex":"\\begin{lemma} \\label{lemma:DG}\nLet $(\\mathbf{x}, \\mathbf{y}, \\mathbf{s}) \\in \\mathcal{N}(\\eta, \\tau)$ and let $(\\Delta \\mathbf{x}, \\Delta \\mathbf{y}, \\Delta \\mathbf{s})$ be the solution of \\eqref{eq:NS}. We have the following lower and upper bounds on the duality gap after a full Newton step:\n    \\[ \\tau(\\nu - \\eta^2) \\leq (\\mathbf{x}^+)^\\top \\mathbf{s}^+ \\leq \\tau \\nu. \\]\n\\end{lemma}","sha256_utf8":"4f3303ed136706e50fc800569765ae003eaae36a4306266423c37b6275292079"},{"start":27662,"end":29340,"start_line":310,"end_line":333,"raw_tex":"\\begin{proof}\nUsing \\ref{sd_prop:first} from Lemma \\ref{lemma:sd_properties},\n    \\[ (\\mathbf{x}^+)^\\top \\mathbf{s}^+ = (\\mathbf{x} + \\Delta \\mathbf{x})^\\top (\\mathbf{s} + \\Delta \\mathbf{s}) = \\mathbf{x}^\\top \\mathbf{s} + \\mathbf{s}^\\top \\Delta \\mathbf{x} + \\mathbf{x}^\\top \\Delta \\mathbf{s}. \\]\nFrom the third equation of \\eqref{eq:NS}, we have\n\\begin{align*}\n    \\Delta \\mathbf{x}^\\top \\mathbf{s} &= \\Delta \\mathbf{x}^\\top \\mathbf{s} + \\Delta \\mathbf{x}^\\top \\Delta \\mathbf{s} \\\\\n&= \\Delta \\mathbf{x}^\\top \\mathbf{s} - \\Delta \\mathbf{x}^\\top (\\mathbf{s} + \\tau g(\\mathbf{x}) + \\tau H(\\mathbf{x}) \\Delta \\mathbf{x}) \\\\\n&= -\\tau (\\Delta \\mathbf{x}^\\top g(\\mathbf{x}) + \\|\\Delta \\mathbf{x}\\|_{\\mathbf{x}}^2).\n\\end{align*}\nSimilarly, we can express $\\mathbf{x}^\\top \\Delta \\mathbf{s}$ as\n\\begin{align*}\n\\mathbf{x}^\\top \\Delta \\mathbf{s} &= - \\mathbf{x}^\\top \\left( \\mathbf{s} + \\tau g(\\mathbf{x}) + \\tau H(\\mathbf{x}) \\Delta \\mathbf{x} \\right) \\\\\n&\\stackrel{\\ref{prop:lhscb_Hessian},\\ref{prop:lhscb_gradient}}{=} -\\mathbf{x}^\\top \\mathbf{s} + \\tau \\nu + \\tau \\Delta \\mathbf{x}^\\top g(\\mathbf{x}).\n\\end{align*}\n\nCombining the previous equations, we can write $(\\mathbf{x}^+)^\\top \\mathbf{s}^+$ as\n\\begin{align*}\n(\\mathbf{x}^+)^\\top \\mathbf{s}^+ &= \\mathbf{x}^\\top \\mathbf{s} - \\tau \\Delta \\mathbf{x}^\\top g(\\mathbf{x}) - \\tau \\|\\Delta \\mathbf{x}\\|_{\\mathbf{x}}^2 - \\mathbf{x}^\\top \\mathbf{s} + \\tau \\nu + \\tau \\Delta \\mathbf{x}^\\top g(\\mathbf{x}) \\\\\n&= \\tau (\\nu -  \\|\\Delta \\mathbf{x}\\|_{\\mathbf{x}}^2).\n\\end{align*}\nUsing part \\ref{sd_prop:second} of Lemma \\ref{lemma:sd_properties}, we get\n\\[ \\tau(\\nu-\\eta^2) \\leq (\\mathbf{x}^+)^\\top \\mathbf{s}^+ \\leq \\tau\\nu.  \\]\n\n\\end{proof}","sha256_utf8":"edafde0adc6822121351ec3d26e7625b6235bad24e4876f86e5f096f537796a5"},{"start":30201,"end":31095,"start_line":347,"end_line":360,"raw_tex":"\\begin{proof}\n    At the beginning of the $k^{\\text{th}}$ iteration, $\\tau = \\tau_k = (1-\\vartheta)^{k-1} \\tau_0$. According to Lemma \\ref{lemma:DG}, we have\n    \\[ \\mathbf{x}_k^{\\top} \\mathbf{s}_k  \\leq \\tau_k \\nu = (1-\\vartheta)^{k-1} \\tau_0 \\nu. \\]\n    We would like to find the smallest $k$ for which  $(1-\\vartheta)^{k-1} \\tau_0 \\nu \\leq \\varepsilon$ holds, which is equivalent to\n    \\[ (k-1) \\ln (1-\\vartheta) + \\ln ( \\tau_0 \\nu ) \\leq \\ln \\varepsilon.\\]\n    Using the inequality $-\\ln(1 - \\vartheta) \\geq \\vartheta$ for $\\vartheta \\in [0,1)$, we can require the fulfillment of the stronger inequality\n    \\[ -(k-1) \\vartheta + \\ln (\\tau_0 \\nu ) \\leq \\ln \\varepsilon,\\]\nwhich is true if\n\\[\nk \\geq \\frac{1}{\\vartheta} \\ln \\left( \\frac{\\tau_0 \\nu}{\\varepsilon} \\right) +1\n= \\frac{2}{\\eta} \\left( \\sqrt{\\nu} + 1 \\right)  \\ln \\left(  \\frac{\\tau_0\\nu}{\\varepsilon} \\right) +1.\n\\]\n\n\\end{proof}","sha256_utf8":"70d3e7159bd2d3dc4209ac63c681ab74d7b81f5423e1e781db5b2a77f95d4dd2"},{"start":96694,"end":98113,"start_line":1024,"end_line":1041,"raw_tex":"\\begin{proposition}[\\protect{\\cite[Chapters 2--3]{Renegar2001}, \\cite[Chapter 5]{Nesterov2018}, and \\cite[Lemma 1]{Nesterov2012}}]\n    Let $f$ be a $\\nu$-SCB for the proper convex cone $\\mathcal{K}$, let $\\vx\\in\\K^\\circ$, and let $\\mathbf{n} (\\mathbf{x} ) = - H (\\mathbf{x})^{-1} g (\\mathbf{x})$ be the (unconstrained) Newton step for $f$ at $\\mathbf{x}$.\n       \\setlist[enumerate,1]{label=\\textup{(A\\arabic*)}, leftmargin=5em, labelsep=0.5em, itemindent=0pt}\n    \\begin{enumerate}\n        \\item \\label{prop:scb_NS}\n        If $\\|\\mathbf{n}(\\mathbf{x})\\|_{\\mathbf{x}} < 1$, then $\\mathbf{x}^+ := \\mathbf{x} + \\mathbf{n} (\\mathbf{x})$ satisfies\n\\[\n\\|\\mathbf{n} (\\mathbf{x}^+)\\|_{\\mathbf{x}^+} \\leq \\left( \\frac{\\|\\mathbf{n}(\\mathbf{x})\\|_{\\mathbf{x}}}{1 - \\|\\mathbf{n}(\\mathbf{x})\\|_{\\mathbf{x}}} \\right)^2.\n\\]\n    \\item \\label{prop:scb_norm1}\n   If $\\mathbf{u} \\in \\mathcal{B}_{\\mathbf{x}}(\\mathbf{x},1)$, then $\\vu\\in\\K^\\circ$, and for all $\\mathbf{v} \\neq \\mathbf{0}$, one has\n\\[\n\\frac{\\|\\mathbf{v}\\|_{\\mathbf{u}}^*}{\\|\\mathbf{v}\\|_{\\mathbf{x}}^*} \\leq \\frac{1}{1 - \\|\\mathbf{u} - \\mathbf{x}\\|_{\\mathbf{x}}}.\n\\]\n\\item \\label{prop:scb_norm2} Let $\\mathbf{x}, \\mathbf{u} \\in \\mathcal{K}^\\circ$, and let $r:= \\| \\mathbf{x} - \\mathbf{u} \\|_\\mathbf{u} < 1$. Then\n\\[ \\| g(\\mathbf{x}) - g(\\mathbf{u}) - H(\\mathbf{u}) (\\mathbf{x} - \\mathbf{u}) \\|_{\\mathbf{u}}^* \\leq \\frac{r^2}{1-r}. \\]\n    \\end{enumerate}\n\\end{proposition}","sha256_utf8":"170a38f90f6fc334fd92a558877a2e6f4ec10668ec3f9aa9962263ee0069bbac"},{"start":98115,"end":98908,"start_line":1043,"end_line":1052,"raw_tex":"\\begin{proposition}[\\protect{\\cite[Chapters 2--3]{Renegar2001}, \\cite[Chapter 5]{Nesterov2018}}]\n    Let $f$ be a $\\nu$-LHSCB for the proper convex cone $\\mathcal{K}$ and let $\\vx\\in\\K^{\\circ}$. Then:\n       \\setlist[enumerate,1]{label=\\textup{(A\\arabic*)}, leftmargin=5em, labelsep=0.5em, itemindent=0pt, start=4}\n    \\begin{enumerate}\n        \\item \\label{prop:lhscb_Hessian} $H(\\mathbf{x}) \\mathbf{x} = - g(\\mathbf{x})$;\n        \\item \\label{prop:lhscb_gradient} $ g(\\mathbf{x})^{\\top} \\mathbf{x} = - \\nu$;\n        \\item \\label{prop:lhscb_lh} $g( \\alpha \\mathbf{x}) = \\alpha^{-1} g(\\mathbf{x})$ and $H( \\alpha \\mathbf{x} ) = \\alpha^{-2} H(\\mathbf{x})$ for all $\\alpha > 0$;\n        \\item  \\label{prop:lhscb_norm} $\\| g(\\mathbf{x}) \\|_x^* = \\sqrt{\\nu}$.\n    \\end{enumerate}\n\\end{proposition}","sha256_utf8":"b923171e77a78d505a77abb5ea874fd57c703174ca2a22ee0a8347343ef8005a"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"原始对偶锥规划","source_units":[{"start":3783,"end":4290,"start_line":75,"end_line":90,"raw_tex":"\\begin{equation} \\label{eq:PD_problem}\n\\left.\n\\begin{aligned}\n    \\inf\\;\\, & \\mathbf{c}^\\top \\mathbf{x} \\\\\n    \\st\\  & A\\mathbf{x} = \\mathbf{b} \\\\\n    & \\mathbf{x} \\in \\mathcal{K}\n\\end{aligned}\n\\quad \\right\\} \\quad \\text{(P)}\n\\qquad \\qquad \\qquad \\left.\n\\begin{aligned}\n    \\sup \\;\\, & \\mathbf{b}^\\top \\mathbf{y} \\\\\n    \\st\\;\\; & A^\\top \\mathbf{y} + \\mathbf{s} = \\mathbf{c} \\\\\n    & \\mathbf{s} \\in \\mathcal{K}^*, \\  \\mathbf{y} \\in \\mathbb{R}^m \\\\\n\\end{aligned}\n\\quad \\right\\}  \\quad\\text{(D)}\n\\end{equation}","sha256_utf8":"fcbe4bc416581530a1925f9e9339165a3162325c22aa0d8a2366266ec5b4197a"},{"start":4291,"end":5012,"start_line":91,"end_line":91,"raw_tex":"where $\\mathbf{c} \\in \\mathbb{R}^n$ and $\\mathbf{b} \\in \\mathbb{R}^m$ are given vectors, $A \\in \\mathbb{R}^{m \\times n}$ is a given matrix with full row rank, and $\\mathcal{K} \\subseteq \\mathbb{R}^n$ is a proper convex cone whose dual cone is denoted by $\\mathcal{K}^*$. The only assumption we make about $\\K$ is that its interior, denoted by $\\K^\\circ$, is the domain of a known logarithmically homogeneous self-concordant barrier function (LHSCB), see Appendix \\ref{sec:lhscb}, whose gradient and Hessian are computable efficiently. The cone need not have any other desirable properties; in particular, we do not assume that $\\K$ is symmetric, homogeneous, or hyperbolic. Nor do we make any assumptions about its dual.\n","sha256_utf8":"45fb6b6bceaf8df40db46fb36ca4b2168df1177ccfc98eb413a746dfe9d844a9"}]},{"id":"S002","semantic_type":"definition","kind":"definition","title":"可行集与严格可行集","source_units":[{"start":14676,"end":14947,"start_line":129,"end_line":132,"raw_tex":"Let $\\mathcal{F}$ denote the set of primal-dual feasible solutions:\n\\[\n\\mathcal{F} = \\left\\{ (\\mathbf{x}, \\mathbf{y}, \\mathbf{s})\\in\\mathcal{K}\\times\\mathbb{R}^m\\times\\mathcal{K}^* \\mid A\\mathbf{x} = \\mathbf{b}, \\ A^{\\top}\\mathbf{y} + \\mathbf{s} = \\mathbf{c} \\right\\},\n\\]","sha256_utf8":"97103c9e5761290f6c609376681720817b08c0ec1db95a7341fae58c17c38de9"},{"start":14948,"end":15246,"start_line":133,"end_line":136,"raw_tex":"and let $\\mathcal{F}^{\\circ}$ be the set of strictly feasible solutions:\n\\[\n\\mathcal{F}^{\\circ} = \\left\\{ (\\mathbf{x}, \\mathbf{y}, \\mathbf{s})\\in\\mathcal{K}^\\circ\\times\\mathbb{R}^m\\times(\\mathcal{K}^*)^\\circ \\mid A\\mathbf{x} = \\mathbf{b}, \\ A^{\\top}\\mathbf{y} + \\mathbf{s} = \\mathbf{c} \\right\\}.\n\\]","sha256_utf8":"abdf5de9b4e279e817bf459decf3d335e8f5a6b2d01ae48851f62027ed2a7894"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"严格可行性与最优条件","source_units":[{"start":15247,"end":15577,"start_line":137,"end_line":144,"raw_tex":"Suppose that the interior point (aka.~Slater) condition is satisfied: $\\mathcal{F}^{\\circ} \\neq \\emptyset$. Then strong duality holds, and the optimality conditions can be formulated as\n\\begin{equation}\n\\left.\n\\begin{aligned}\n(\\vx,\\vy,\\vs)&\\in\\mathcal{F}\\\\\n\\vx^\\T\\vs&=0\n\\end{aligned} \\right\\}. \\tag{OC}\\label{eq:OC}\n\\end{equation}","sha256_utf8":"f026961469bcd5ba2de30ec0598142977032d60e4d19775474bb64df80c65537"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"障碍函数与中心路径","source_units":[{"start":15579,"end":16214,"start_line":146,"end_line":153,"raw_tex":"We assume that an LHSCB $f: \\mathcal{K}^{\\circ} \\rightarrow \\mathbb{R}$ with barrier parameter $\\nu$ is given for the primal cone $\\mathcal{K}$. Let $g(\\mathbf{x})$ and $H(\\mathbf{x})$ denote the gradient and Hessian of $f$ at $\\mathbf{x} \\in \\mathcal{K}^{\\circ}$, respectively. With this notation in place, we define the central path problem, parametrized by the positive scalar $\\tau$, called the \\emph{central path parameter}, as follows:\n\\begin{equation}\n\\left.\n\\begin{aligned}\n(\\vx,\\vy,\\vs)&\\in\\mathcal{F}^\\circ\\\\\n    \\mathbf{s} + \\tau g(\\mathbf{x}) \\ & = \\mathbf{0} \\\\\n\\end{aligned} \\right\\}. \\tag{CP}\\label{eq:CP}\n\\end{equation}","sha256_utf8":"cbb752c5b525d1a44b5e8a2a5be45f67c64d40db1dde824405937e31366622a7"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"中心路径邻域","source_units":[{"start":16416,"end":16719,"start_line":156,"end_line":160,"raw_tex":"In the analysis,  we consider the following \\emph{neighborhood} of the central path:\n\\begin{equation}\n\\label{eq:nbd}\n\\mathcal{N}(\\eta,\\tau) = \\left\\{ (\\mathbf{x}, \\mathbf{y}, \\mathbf{s}) \\in \\mathcal{F}^\\circ : \\| \\mathbf{s} + \\tau g(\\mathbf{x}) \\|_{\\mathbf{x}}^* \\leq \\eta \\tau \\right\\},\n\\end{equation}","sha256_utf8":"1ef94e2cdfa3bd6409b45b8435e297c81f3384a8a609474300e34d757b5d4efa"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"牛顿系统与方向","source_units":[{"start":17495,"end":17949,"start_line":163,"end_line":172,"raw_tex":"Given an $(\\mathbf{x},\\mathbf{y},\\mathbf{s})\\in\\mathcal{F}^\\circ$, the \\emph{Newton system} associated with the central path \\eqref{eq:CP} is\n\\begin{equation}\n\\left.\n\\begin{aligned}\n    A \\Delta \\mathbf{x} & = \\mathbf{0} \\\\\n    A^{\\top} \\Delta \\mathbf{y} + \\Delta \\mathbf{s} & = \\mathbf{0} \\\\\n    \\tau H(\\mathbf{x}) \\Delta \\mathbf{x} + \\Delta \\mathbf{s} & = -(\\mathbf{s} + \\tau g(\\mathbf{x}))\n\\end{aligned} \\right\\} .\n\\tag{NS}\\label{eq:NS}\n\\end{equation}","sha256_utf8":"e858a9d55bdb4a45a38eea0417fd528d089e56f315eb8467f81aa9b2bde8719c"},{"start":17950,"end":18085,"start_line":173,"end_line":174,"raw_tex":"Its solution ($\\Delta \\mathbf{x}$, $\\Delta \\mathbf{y}$, $\\Delta \\mathbf{s}$)\nis called the \\emph{Newton direction} at $(\\vx,\\vy,\\vs)$.\n","sha256_utf8":"30c7b8bbd175e9bec24154f3abba6f3f0206d2b8e316dce1f6cf50685fa8c0fc"}]},{"id":"S007","semantic_type":"algorithm","kind":"algorithm","title":"固定缩减全步长算法","source_units":[{"start":18699,"end":20248,"start_line":178,"end_line":210,"raw_tex":"\\begin{algorithm}[h]\n\\caption{A full-step IPA for nonsymmetric cone programming}\n\\label{alg:simple_short_step}\n\\begin{algorithmic}\n\\vspace{0.5em}\n\\STATE \\textbf{inputs:}\n$A \\in \\mathbb{R}^{m \\times n}$ with full row rank; $\\mathbf{b} \\in \\mathbb{R}^m$;  $\\mathbf{c} \\in \\mathbb{R}^n$;\n\\\\ \\hspace{1em}an oracle for computing the Hessian $H$ of the $\\nu$-LHSCB $f: \\ \\mathcal{K}^{\\circ} \\rightarrow \\mathbb{R}$;\n\\\\ \\hspace{1em}an initial $\\tau_0>0$ and\n$(\\mathbf{x}_0, \\mathbf{y}_0, \\mathbf{s}_0) \\in \\mathcal{F}^\\circ$ such that $\\| \\mathbf{s}_0 + \\tau_0 g(\\mathbf{x}_0) \\|_{\\mathbf{x}_0}^* \\leq \\eta \\tau_0$;\n\\\\ \\hspace{1em}$\\varepsilon > 0$ optimality tolerance.\n\n\\vspace{0.5em}\n\\STATE \\textbf{parameters:}\nneighborhood parameter $\\eta \\in (0, 1)$;\nupdate parameter $\\vartheta$.\n\\vspace{0.5em}\n\n\\STATE \\textbf{output:} a triplet $(\\vx,\\vy,\\vs)\\in\\mathcal{F}^\\circ$ satisfying $\\vx^\\T\\vs\\leq\\varepsilon$.\n\n\\vspace{1em}\n\\STATE Initialize $\\mathbf{x} := \\mathbf{x}_0$, $\\mathbf{y} := \\mathbf{y}_0$, $\\mathbf{s} := \\mathbf{s}_0$, $\\tau := \\tau_0$.\n\\vspace{0.5em}\n\\WHILE{$\\mathbf{x}^\\top \\mathbf{s} > \\varepsilon$}\n    \\STATE Solve \\eqref{eq:NS} for $(\\Delta \\mathbf{x}, \\Delta \\mathbf{y}, \\Delta \\mathbf{s})$.\n    \\STATE Set $\\mathbf{x} := \\mathbf{x} + \\Delta \\mathbf{x}$, $\\mathbf{y} := \\mathbf{y} + \\Delta \\mathbf{y}$, $\\mathbf{s} := \\mathbf{s} + \\Delta \\mathbf{s}$.\n    \\STATE Set $\\tau := (1 - \\vartheta) \\tau$.\n\\ENDWHILE\n\\vspace{0.5em}\n\\STATE \\textbf{return} $(\\mathbf{x}, \\mathbf{y}, \\mathbf{s})$.\n\\vspace{0.5em}\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"54dcf6ef43fe880bb1ceaa7f8aa2e4704c129ea1a77f4662f015d689f9516a48"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"自协调障碍","source_units":[{"start":94107,"end":95003,"start_line":1003,"end_line":1011,"raw_tex":"\\begin{definition}[Self-concordant barrier]\nLet $\\K\\subseteq\\Rn$ be a convex cone with a non-empty interior $\\K^\\circ$. A function $f:\\K^\\circ\\to\\R$ is called a \\emph{strongly nondegenerate, self-concordant barrier} for $\\K$ if $f$ is strictly convex, at least three times differentiable, and has the following two properties (with $\\operatorname{bd}(\\K)$ denoting the boundary of $\\K$, and $g$ and $H$ denoting the gradient and Hessian of $f$):\n\\begin{enumerate}[\\hspace{1em} 1.]\n\\item $f(\\vx)\\to\\infty$  as $\\vx\\to\\operatorname{bd}(\\K)$,\n\\item $|D^3 f(\\vx)[\\vh,\\vh,\\vh]| \\leq 2\\left(D^2f(\\vx)[\\vh,\\vh]\\right)^{3/2}$,\n\\item $\\nu := \\sup_{\\vx\\in\\K^\\circ} g(\\vx)^\\T H(\\vx)^{-1}g(\\vx) < \\infty.$\n\\end{enumerate}\nWe refer to the parameter $\\nu$ above as the \\emph{barrier parameter} of $f$; we also use the shorthand \\emph{$\\nu$-SCB} to mean ``an SCB with barrier parameter $\\nu$''.\n\\end{definition}","sha256_utf8":"884217bf405df639aefa4a2fb7aa07605fd7f5e61973e2d08b8fa22b34ef5296"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"对数齐次障碍","source_units":[{"start":95005,"end":95565,"start_line":1013,"end_line":1016,"raw_tex":"\\begin{definition}[Logarithmic homogeneity and LHSCBs]\n    Let $f: \\ \\mathcal{K}^{\\circ} \\rightarrow \\mathbb{R}$ be a $\\nu$-SCB function for the proper convex cone $ \\mathcal{K}$.  We say that $f$ is a \\emph{logarithmically homogeneous self-concordant barrier (LHSCB)} with barrier parameter $\\nu$ for $\\mathcal{K}$, or a \\emph{$\\nu$-LHSCB} for short, if it satisfies the \\emph{logarithmic homogeneity property}\n    \\[ f(t \\mathbf{x}) = f(\\mathbf{x}) - \\nu \\ln (t) \\quad \\text{for all } \\mathbf{x} \\in \\mathcal{K}^{\\circ} \\text{ and } t > 0.\\]\n\\end{definition}","sha256_utf8":"ee24932e14fbbb79973ed2f0154585e0ea95e838c4ce5fee031249c26e6d576c"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"局部范数与对偶范数","source_units":[{"start":95567,"end":96558,"start_line":1018,"end_line":1020,"raw_tex":"We associate with each $\\vx\\in\\K^\\circ$ the \\emph{local inner product} $\\langle\\cdot,\\cdot\\rangle_\\vx : \\K \\times \\K \\to \\R$ defined as $\\langle \\vu,\\vv\\rangle_\\vx := \\vu^\\T H(\\vx) \\vv$ and the \\emph{local norm} $\\|\\cdot\\|_\\vx$ induced by this local inner product. Thus, $\\|\\vv\\|_\\vx = \\|H(\\vx)^{1/2}\\vv\\|$. We define the local (open) ball centered at $\\vx$ with radius $r$ by $\\mathcal{B}(\\vx,r) := \\{\\vv\\in\\R^n\\,|\\, \\|\\vv-\\vx\\|_{\\vx} < r\\}$. When $r=1$, these balls are also called the \\emph{Dikin ellipsoids}. Crucially, for every $\\vx\\in\\K^\\circ$, we have $\\mathcal{B}(\\vx,1)\\subset\\K^\\circ$.\n\nAnalogously, we define the \\emph{dual local inner product} $\\langle \\cdot,\\cdot \\rangle_\\vx^*: \\K^*\\times\\K^*\\to\\R$ by  $\\langle \\vs, \\vt \\rangle_\\vx^* := \\vs^\\T H(\\vx)^{-1}\\vt$, its induced \\emph{dual local norm} $\\|\\cdot\\|_{\\vx}^*$, and the corresponding open balls $\\mathcal{B}^*(\\vx,r)$. The dual local norm, by definition, satisfies the identity $\\|\\vt\\|_{\\vx}^* = \\|H(\\vx)^{-1/2}\\vt\\|$.\n","sha256_utf8":"a7661ac8da44247fddd6d4c58e49097cbc01d9dcdf272a39cc7dcd4df4e8aff8"}]},{"id":"S011","semantic_type":"proposition","kind":"proposition","title":"自协调牛顿估计","source_units":[{"start":96694,"end":98113,"start_line":1024,"end_line":1041,"raw_tex":"\\begin{proposition}[\\protect{\\cite[Chapters 2--3]{Renegar2001}, \\cite[Chapter 5]{Nesterov2018}, and \\cite[Lemma 1]{Nesterov2012}}]\n    Let $f$ be a $\\nu$-SCB for the proper convex cone $\\mathcal{K}$, let $\\vx\\in\\K^\\circ$, and let $\\mathbf{n} (\\mathbf{x} ) = - H (\\mathbf{x})^{-1} g (\\mathbf{x})$ be the (unconstrained) Newton step for $f$ at $\\mathbf{x}$.\n       \\setlist[enumerate,1]{label=\\textup{(A\\arabic*)}, leftmargin=5em, labelsep=0.5em, itemindent=0pt}\n    \\begin{enumerate}\n        \\item \\label{prop:scb_NS}\n        If $\\|\\mathbf{n}(\\mathbf{x})\\|_{\\mathbf{x}} < 1$, then $\\mathbf{x}^+ := \\mathbf{x} + \\mathbf{n} (\\mathbf{x})$ satisfies\n\\[\n\\|\\mathbf{n} (\\mathbf{x}^+)\\|_{\\mathbf{x}^+} \\leq \\left( \\frac{\\|\\mathbf{n}(\\mathbf{x})\\|_{\\mathbf{x}}}{1 - \\|\\mathbf{n}(\\mathbf{x})\\|_{\\mathbf{x}}} \\right)^2.\n\\]\n    \\item \\label{prop:scb_norm1}\n   If $\\mathbf{u} \\in \\mathcal{B}_{\\mathbf{x}}(\\mathbf{x},1)$, then $\\vu\\in\\K^\\circ$, and for all $\\mathbf{v} \\neq \\mathbf{0}$, one has\n\\[\n\\frac{\\|\\mathbf{v}\\|_{\\mathbf{u}}^*}{\\|\\mathbf{v}\\|_{\\mathbf{x}}^*} \\leq \\frac{1}{1 - \\|\\mathbf{u} - \\mathbf{x}\\|_{\\mathbf{x}}}.\n\\]\n\\item \\label{prop:scb_norm2} Let $\\mathbf{x}, \\mathbf{u} \\in \\mathcal{K}^\\circ$, and let $r:= \\| \\mathbf{x} - \\mathbf{u} \\|_\\mathbf{u} < 1$. Then\n\\[ \\| g(\\mathbf{x}) - g(\\mathbf{u}) - H(\\mathbf{u}) (\\mathbf{x} - \\mathbf{u}) \\|_{\\mathbf{u}}^* \\leq \\frac{r^2}{1-r}. \\]\n    \\end{enumerate}\n\\end{proposition}","sha256_utf8":"170a38f90f6fc334fd92a558877a2e6f4ec10668ec3f9aa9962263ee0069bbac"}]},{"id":"S012","semantic_type":"proposition","kind":"proposition","title":"齐次障碍恒等式","source_units":[{"start":98115,"end":98908,"start_line":1043,"end_line":1052,"raw_tex":"\\begin{proposition}[\\protect{\\cite[Chapters 2--3]{Renegar2001}, \\cite[Chapter 5]{Nesterov2018}}]\n    Let $f$ be a $\\nu$-LHSCB for the proper convex cone $\\mathcal{K}$ and let $\\vx\\in\\K^{\\circ}$. Then:\n       \\setlist[enumerate,1]{label=\\textup{(A\\arabic*)}, leftmargin=5em, labelsep=0.5em, itemindent=0pt, start=4}\n    \\begin{enumerate}\n        \\item \\label{prop:lhscb_Hessian} $H(\\mathbf{x}) \\mathbf{x} = - g(\\mathbf{x})$;\n        \\item \\label{prop:lhscb_gradient} $ g(\\mathbf{x})^{\\top} \\mathbf{x} = - \\nu$;\n        \\item \\label{prop:lhscb_lh} $g( \\alpha \\mathbf{x}) = \\alpha^{-1} g(\\mathbf{x})$ and $H( \\alpha \\mathbf{x} ) = \\alpha^{-2} H(\\mathbf{x})$ for all $\\alpha > 0$;\n        \\item  \\label{prop:lhscb_norm} $\\| g(\\mathbf{x}) \\|_x^* = \\sqrt{\\nu}$.\n    \\end{enumerate}\n\\end{proposition}","sha256_utf8":"b923171e77a78d505a77abb5ea874fd57c703174ca2a22ee0a8347343ef8005a"}]},{"id":"S013","semantic_type":"external_fact","kind":"external_fact","title":"标准自协调理论","source_units":[],"declared_external":true,"declaration_text":["The standard self-concordance results cited in B0113 give its Newton-decrement contraction and local-norm estimates. An affine perturbation of a self-concordant function remains self-concordant with the same Hessian, and the same Newton-decrement contraction applies."]},{"id":"S014","semantic_type":"external_fact","kind":"external_fact","title":"标准齐次障碍理论","source_units":[],"declared_external":true,"declaration_text":["The standard logarithmically homogeneous barrier results cited in B0117 give H(x)x=-g(x), g(x)ᵀx=-ν, the gradient and Hessian scaling identities, and ||g(x)||_x^*=√ν."]},{"id":"S015","semantic_type":"external_fact","kind":"external_fact","title":"障碍参数下界","source_units":[],"declared_external":true,"declaration_text":["The parameter of a logarithmically homogeneous self-concordant barrier on a proper cone satisfies ν≥1."]},{"id":"S016","semantic_type":"theorem","kind":"theorem","title":"全步长算法迭代复杂度","source_units":[{"start":29530,"end":30199,"start_line":337,"end_line":345,"raw_tex":"\\begin{theorem} \\label{theorem:compl1}\n\nIf $0 < \\eta \\leq \\frac{1}{4}$ and $\n\\vartheta = \\frac{\\eta / 2}{\\sqrt{\\nu} + 1}$, then Algorithm \\ref{alg:simple_short_step} produces a feasible solution to \\eqref{eq:PD_problem} with duality gap below $\\varepsilon>0$ in no more than $\\frac{2}{\\eta}\\left(\\sqrt{\\nu}+1 \\right)  \\ln \\left(\\frac{\\tau_0 \\nu}{\\varepsilon}\\right) +1$\niterations.\n\nIn particular, if the initial point satisfies $\\tau_0 = \\frac{\\vx_0^\\T\\vs_0}{\\nu}$, as in the initialization strategies discussed in Section \\ref{sec:init}, then the required number of iterations is\n\\[\\mathcal{O}\\left(\\sqrt\\nu\\ln\\frac{\\vx_0^\\T\\vs_0}{\\varepsilon}\\right).\\]\n\\end{theorem}","sha256_utf8":"71eea6017f2ca4a09777a4ce59afce1810b0bb268b4da2abaf9d94bac4063a21"}]},{"id":"S017","semantic_type":"proof","kind":"proof","title":"迭代复杂度完整证明","source_units":[{"start":30201,"end":31095,"start_line":347,"end_line":360,"raw_tex":"\\begin{proof}\n    At the beginning of the $k^{\\text{th}}$ iteration, $\\tau = \\tau_k = (1-\\vartheta)^{k-1} \\tau_0$. According to Lemma \\ref{lemma:DG}, we have\n    \\[ \\mathbf{x}_k^{\\top} \\mathbf{s}_k  \\leq \\tau_k \\nu = (1-\\vartheta)^{k-1} \\tau_0 \\nu. \\]\n    We would like to find the smallest $k$ for which  $(1-\\vartheta)^{k-1} \\tau_0 \\nu \\leq \\varepsilon$ holds, which is equivalent to\n    \\[ (k-1) \\ln (1-\\vartheta) + \\ln ( \\tau_0 \\nu ) \\leq \\ln \\varepsilon.\\]\n    Using the inequality $-\\ln(1 - \\vartheta) \\geq \\vartheta$ for $\\vartheta \\in [0,1)$, we can require the fulfillment of the stronger inequality\n    \\[ -(k-1) \\vartheta + \\ln (\\tau_0 \\nu ) \\leq \\ln \\varepsilon,\\]\nwhich is true if\n\\[\nk \\geq \\frac{1}{\\vartheta} \\ln \\left( \\frac{\\tau_0 \\nu}{\\varepsilon} \\right) +1\n= \\frac{2}{\\eta} \\left( \\sqrt{\\nu} + 1 \\right)  \\ln \\left(  \\frac{\\tau_0\\nu}{\\varepsilon} \\right) +1.\n\\]\n\n\\end{proof}","sha256_utf8":"70d3e7159bd2d3dc4209ac63c681ab74d7b81f5423e1e781db5b2a77f95d4dd2"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"间隙的几何上界","source_units":[{"start":30215,"end":30453,"start_line":348,"end_line":349,"raw_tex":"    At the beginning of the $k^{\\text{th}}$ iteration, $\\tau = \\tau_k = (1-\\vartheta)^{k-1} \\tau_0$. According to Lemma \\ref{lemma:DG}, we have\n    \\[ \\mathbf{x}_k^{\\top} \\mathbf{s}_k  \\leq \\tau_k \\nu = (1-\\vartheta)^{k-1} \\tau_0 \\nu. \\]\n","sha256_utf8":"9d62ae4e3a69520391ad5f5a82fba85828477171b56668157b6a35b6421a6841"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"停止条件取对数","source_units":[{"start":30453,"end":30663,"start_line":350,"end_line":351,"raw_tex":"    We would like to find the smallest $k$ for which  $(1-\\vartheta)^{k-1} \\tau_0 \\nu \\leq \\varepsilon$ holds, which is equivalent to\n    \\[ (k-1) \\ln (1-\\vartheta) + \\ln ( \\tau_0 \\nu ) \\leq \\ln \\varepsilon.\\]\n","sha256_utf8":"69d4da7b3d7bbd16c11dfd3933b1f4f50d94b151698ad154d34dad529d55d0b4"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"代入缩减率得到次数界","source_units":[{"start":30663,"end":31083,"start_line":352,"end_line":358,"raw_tex":"    Using the inequality $-\\ln(1 - \\vartheta) \\geq \\vartheta$ for $\\vartheta \\in [0,1)$, we can require the fulfillment of the stronger inequality\n    \\[ -(k-1) \\vartheta + \\ln (\\tau_0 \\nu ) \\leq \\ln \\varepsilon,\\]\nwhich is true if\n\\[\nk \\geq \\frac{1}{\\vartheta} \\ln \\left( \\frac{\\tau_0 \\nu}{\\varepsilon} \\right) +1\n= \\frac{2}{\\eta} \\left( \\sqrt{\\nu} + 1 \\right)  \\ln \\left(  \\frac{\\tau_0\\nu}{\\varepsilon} \\right) +1.\n\\]\n","sha256_utf8":"01ba30abefffb7d951e09cfad5283763133e3b8ec528ba20bdbbf71d59504f4e"}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","title":"牛顿方向正交与范数界","source_units":[{"start":21207,"end":21889,"start_line":216,"end_line":225,"raw_tex":"\\begin{lemma}[\\cite{PappYildiz2017, Serrano2015,SkajaaYe2015}]\\label{lemma:sd_properties}\nLet $(\\mathbf{x}, \\mathbf{y}, \\mathbf{s}) \\in \\mathcal{N}(\\eta, \\tau)$ and let $(\\Delta \\mathbf{x}, \\Delta \\mathbf{y}, \\Delta \\mathbf{s})$ be the solution of \\eqref{eq:NS}. Then\n\\begin{enumerate}[label=(\\roman*)]\n    \\item  $\\Delta \\mathbf{x}^{\\top} \\Delta \\mathbf{s} = 0$, \\label{sd_prop:first}\n    \\item $\n\\| \\Delta \\mathbf{x} \\|_{\\mathbf{x}} \\leq \\eta$ and $\\| \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^* \\leq \\eta \\tau\n$,  \\label{sd_prop:second}\n    \\item $\\| \\mathbf{s} + \\tau g(\\mathbf{x}) + \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^* \\leq \\eta \\tau$. \\label{sd_prop:third}\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"8f0572e0eef658e86453ea49858979f6138cacf9f28410afaf159768a21d6c29"}]},{"id":"S022","semantic_type":"proof","kind":"proof","title":"牛顿方向性质完整证明","source_units":[{"start":21891,"end":23460,"start_line":227,"end_line":247,"raw_tex":"\\begin{proof}\n    \\begin{enumerate}[label=(\\roman*)]\n        \\item Using the first two equations of \\eqref{eq:NS}, we get\n    \\[ 0 = \\Delta \\mathbf{x}^{\\top} (A^{\\top} \\Delta \\mathbf{y} + \\Delta \\mathbf{s}) = \\Delta \\mathbf{y}^{\\top} A \\Delta \\mathbf{x} + \\Delta \\mathbf{x}^{\\top} \\Delta \\mathbf{s} = \\Delta \\mathbf{x}^{\\top} \\Delta \\mathbf{s}. \\]\n        \\item Using the neighborhood definition, the orthogonality of $\\Delta \\mathbf{x}$ and $\\Delta \\mathbf{s}$, and the third equality of the Newton system, we can derive the upper bounds the following way:\n\\[ \\begin{aligned}\n\\eta^2 \\tau^2 &\\geq \\left( \\| \\mathbf{s} + \\tau g(\\mathbf{x}) \\|_{\\mathbf{x}}^* \\right)^2 = \\left( \\| \\tau H(\\mathbf{x}) \\Delta \\mathbf{x} + \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^* \\right)^2 \\\\\n&= \\tau^2 \\Delta \\mathbf{x}^\\top H(\\mathbf{x}) \\Delta \\mathbf{x} + \\Delta \\mathbf{s}^\\top H(\\mathbf{x})^{-1} \\Delta \\mathbf{s}\n= \\tau^2 \\| \\Delta \\mathbf{x} \\|_{\\mathbf{x}}^2 + (\\| \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^*)^2.\n\\end{aligned}\n\\]\n\nAs both terms on the right-hand side are nonnegative, it follows that $\\| \\Delta \\mathbf{x} \\|_{\\mathbf{x}} \\leq \\eta$ and $\\| \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^* \\leq \\eta \\tau$.\n    \\item Using the third equation of \\eqref{eq:NS} and the upper bound on $\\|\\Delta \\mathbf{x}\\|_{\\mathbf{x}}$ from \\ref{sd_prop:second}, we get\n    \\[\n\\| \\mathbf{s} + \\tau g(\\mathbf{x}) + \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^* = \\| \\tau H(\\mathbf{x}) \\Delta \\mathbf{x} \\|_{\\mathbf{x}}^*\n= \\tau \\| \\Delta \\mathbf{x} \\|_{\\mathbf{x}} \\leq \\eta \\tau.\n  \\]\n\n    \\end{enumerate}\n\\end{proof}","sha256_utf8":"1cafa11156314525b85fa802174a49d0cd674174502d29ac0011c6bba97a5288"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"线性约束推出方向正交","source_units":[{"start":21905,"end":22239,"start_line":228,"end_line":230,"raw_tex":"    \\begin{enumerate}[label=(\\roman*)]\n        \\item Using the first two equations of \\eqref{eq:NS}, we get\n    \\[ 0 = \\Delta \\mathbf{x}^{\\top} (A^{\\top} \\Delta \\mathbf{y} + \\Delta \\mathbf{s}) = \\Delta \\mathbf{y}^{\\top} A \\Delta \\mathbf{x} + \\Delta \\mathbf{x}^{\\top} \\Delta \\mathbf{s} = \\Delta \\mathbf{x}^{\\top} \\Delta \\mathbf{s}. \\]\n","sha256_utf8":"739a6963bc00c9316313e41d53e3562be8718f8ef5fd5ee42e520a721eafc0d6"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"残差平方分解控制方向","source_units":[{"start":22239,"end":23077,"start_line":231,"end_line":239,"raw_tex":"        \\item Using the neighborhood definition, the orthogonality of $\\Delta \\mathbf{x}$ and $\\Delta \\mathbf{s}$, and the third equality of the Newton system, we can derive the upper bounds the following way:\n\\[ \\begin{aligned}\n\\eta^2 \\tau^2 &\\geq \\left( \\| \\mathbf{s} + \\tau g(\\mathbf{x}) \\|_{\\mathbf{x}}^* \\right)^2 = \\left( \\| \\tau H(\\mathbf{x}) \\Delta \\mathbf{x} + \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^* \\right)^2 \\\\\n&= \\tau^2 \\Delta \\mathbf{x}^\\top H(\\mathbf{x}) \\Delta \\mathbf{x} + \\Delta \\mathbf{s}^\\top H(\\mathbf{x})^{-1} \\Delta \\mathbf{s}\n= \\tau^2 \\| \\Delta \\mathbf{x} \\|_{\\mathbf{x}}^2 + (\\| \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^*)^2.\n\\end{aligned}\n\\]\n\nAs both terms on the right-hand side are nonnegative, it follows that $\\| \\Delta \\mathbf{x} \\|_{\\mathbf{x}} \\leq \\eta$ and $\\| \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^* \\leq \\eta \\tau$.\n","sha256_utf8":"94f1b361d66faf9d27fe672b8efa2c6596e72aa5c19fe143fd75b4ce1c7e4e9d"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"控制修正后的线性残差","source_units":[{"start":23077,"end":23449,"start_line":240,"end_line":246,"raw_tex":"    \\item Using the third equation of \\eqref{eq:NS} and the upper bound on $\\|\\Delta \\mathbf{x}\\|_{\\mathbf{x}}$ from \\ref{sd_prop:second}, we get\n    \\[\n\\| \\mathbf{s} + \\tau g(\\mathbf{x}) + \\Delta \\mathbf{s} \\|_{\\mathbf{x}}^* = \\| \\tau H(\\mathbf{x}) \\Delta \\mathbf{x} \\|_{\\mathbf{x}}^*\n= \\tau \\| \\Delta \\mathbf{x} \\|_{\\mathbf{x}} \\leq \\eta \\tau.\n  \\]\n\n    \\end{enumerate}\n","sha256_utf8":"0c62f77b99b08cc31ea0569341a65b7fee5eb14a1b55534000449576806a3597"}]},{"id":"S026","semantic_type":"lemma","kind":"lemma","title":"全步长保持中心邻域","source_units":[{"start":23685,"end":24292,"start_line":251,"end_line":256,"raw_tex":"\\begin{lemma}\n\\label{lemma:zplus-in-Nthetaplus}\nLet $(\\mathbf{x}, \\mathbf{y}, \\mathbf{s}) \\in \\mathcal{N}(\\eta, \\tau)$, let $(\\Delta \\mathbf{x}, \\Delta \\mathbf{y}, \\Delta \\mathbf{s})$ be the solution of \\eqref{eq:NS}, and let $(\\mathbf{x}^+, \\mathbf{y}^+, \\mathbf{s}^+) = (\\mathbf{x}, \\mathbf{y}, \\mathbf{s}) + (\\Delta \\mathbf{x}, \\Delta \\mathbf{y}, \\Delta \\mathbf{s})$. If $0 < \\eta \\leq \\frac{1}{4}$ and $\n\\vartheta = \\frac{\\eta / 2}{\\sqrt{\\nu} + 1}$,\nthen the new point satisfies $(\\mathbf{x}^+, \\mathbf{y}^+, \\mathbf{s}^+) \\in \\mathcal{N}(\\eta, \\tau^+)$ with $\\tau^+ = (1 - \\vartheta) \\tau$.\n\\end{lemma}","sha256_utf8":"9d57772d4bbdc874590c5a3de82660ec67f22c698438ccf418858f7a732d7364"}]},{"id":"S027","semantic_type":"proof","kind":"proof","title":"邻域保持完整证明","source_units":[{"start":24294,"end":27017,"start_line":258,"end_line":301,"raw_tex":"\\begin{proof}\n    Let us define the function $\\varphi : \\mathcal{K}^\\circ \\to \\mathbb{R}$:\n\\[\n\\varphi(\\mathbf{v}) = \\frac{1}{\\tau} (\\mathbf{s} + \\Delta \\mathbf{s})^\\top \\mathbf{v} + f(\\mathbf{v}),\n\\]\nwhere $f$ is the known LHSCB for $\\mathcal{K}^\\circ$, with gradient $g$ and Hessian $H$.\n\nThen $\\varphi$ is self-concordant, and its Hessian is also $H$.\nLet $\\mathbf{n}_{\\varphi}(\\mathbf{v})$ denote the (unconstrained) Newton step for $\\varphi$ at $\\mathbf{v} \\in \\mathcal{K}^\\circ$:\n\\[\n\\mathbf{n}_{\\varphi}(\\mathbf{v}) = -\\tau^{-1} H(\\mathbf{v})^{-1} \\left( \\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{v}) \\right).\n\\]\nThus,\n\\[\n\\mathbf{n}_{\\varphi}(\\mathbf{x}) = -\\tau^{-1} H(\\mathbf{x})^{-1} \\left( \\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{x}) \\right) = \\Delta \\mathbf{x},\n\\]\nand therefore,\n\\[\n\\mathbf{x}^+ = \\mathbf{x} + \\Delta \\mathbf{x} = \\mathbf{x} + \\mathbf{n}_{\\varphi}(\\mathbf{x}).\n\\]\n\nFor any $\\mathbf{v} \\in \\mathcal{K}^\\circ$,\n\\[\n\\|\\mathbf{n}_{\\varphi}(\\mathbf{v})\\|_{\\mathbf{v}} = \\left\\| \\tau^{-1} H(\\mathbf{v})^{-1} \\left( \\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{v}) \\right) \\right\\|_{\\mathbf{v}}\n= \\tau^{-1} \\|\\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{v})\\|_{\\mathbf{v}}^*.\n\\]\nFor $\\mathbf{v} = \\mathbf{x}$, $\\|\\mathbf{n}_{\\varphi}(\\mathbf{x})\\|_{\\mathbf{x}} = \\|\\Delta \\mathbf{x}\\|_{\\mathbf{x}} \\leq \\eta$, using Lemma \\ref{lemma:sd_properties}, part \\ref{sd_prop:second}. As $\\eta < 1$, we can use \\ref{prop:scb_NS}:\n\\begin{equation} \\label{eq:UB_NS}\n    \\tau^{-1} \\|\\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{x}^+)\\|^*_{\\mathbf{x}^+} = \\|\\mathbf{n}_{\\varphi}(\\mathbf{x}^+)\\|_{\\mathbf{x}^+}\n\\leq \\left( \\frac{\\|\\mathbf{n}_{\\varphi}(\\mathbf{x})\\|_{\\mathbf{x}}}{1 - \\|\\mathbf{n}_{\\varphi}(\\mathbf{x})\\|_{\\mathbf{x}}} \\right)^2 \\leq \\frac{\\eta^2}{(1 - \\eta)^2}.\n\\end{equation}\n\nThus,\n\\begin{align*}\n\\frac{1}{\\tau^+} \\left\\| \\mathbf{s}^+ + \\tau^+ g(\\mathbf{x}^+) \\right\\|_{\\mathbf{x}^+}^*\n&= \\frac{1}{1 - \\vartheta} \\tau^{-1} \\| \\mathbf{s} + \\Delta \\mathbf{s} + \\tau^+ g(\\mathbf{x}^+) \\|_{\\mathbf{x}^+}^* \\\\\n&\\leq \\frac{1}{1 - \\vartheta} \\tau^{-1} \\| \\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{x}^+)\\|_{\\mathbf{x}^+}^* + \\frac{\\vartheta}{1 - \\vartheta} \\| g(\\mathbf{x}^+) \\|^*_{\\mathbf{x}^+} \\\\\n&\\stackrel{\\eqref{eq:UB_NS},\\ref{prop:lhscb_norm}}{\\leq} \\frac{1}{1 - \\vartheta}  \\frac{\\eta^2}{(1 - \\eta)^2} + \\frac{\\vartheta}{1 - \\vartheta} \\sqrt{\\nu} \\\\\n&= \\frac{\\sqrt{\\nu} + 1}{\\sqrt{\\nu} + 1 - \\eta/2} \\cdot \\frac{\\eta^2}{(1 - \\eta)^2} + \\frac{\\eta / 2}{\\sqrt{\\nu} + 1 - \\eta/2} \\sqrt{\\nu} \\\\\n&\\leq \\frac{2}{2 - \\eta / 2} \\cdot \\frac{\\eta^2}{(1 - \\eta)^2} + \\frac{\\eta}{2}.\n\\end{align*}\n\nThe last expression is less than $\\eta$ if $\\eta \\leq \\frac{1}{4}$ holds.\n\\end{proof}","sha256_utf8":"74c83afcd11d969d738c1ca4f1d3778cf8bcd93ba84180096e268f0b399066f2"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"构造仿射扰动障碍","source_units":[{"start":24308,"end":24583,"start_line":259,"end_line":263,"raw_tex":"    Let us define the function $\\varphi : \\mathcal{K}^\\circ \\to \\mathbb{R}$:\n\\[\n\\varphi(\\mathbf{v}) = \\frac{1}{\\tau} (\\mathbf{s} + \\Delta \\mathbf{s})^\\top \\mathbf{v} + f(\\mathbf{v}),\n\\]\nwhere $f$ is the known LHSCB for $\\mathcal{K}^\\circ$, with gradient $g$ and Hessian $H$.\n","sha256_utf8":"c4df2bacd980e74a60a430b5a506491a57076087be81ef4c18b2c13114455bcd"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"识别无约束牛顿步","source_units":[{"start":24584,"end":25199,"start_line":265,"end_line":277,"raw_tex":"Then $\\varphi$ is self-concordant, and its Hessian is also $H$.\nLet $\\mathbf{n}_{\\varphi}(\\mathbf{v})$ denote the (unconstrained) Newton step for $\\varphi$ at $\\mathbf{v} \\in \\mathcal{K}^\\circ$:\n\\[\n\\mathbf{n}_{\\varphi}(\\mathbf{v}) = -\\tau^{-1} H(\\mathbf{v})^{-1} \\left( \\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{v}) \\right).\n\\]\nThus,\n\\[\n\\mathbf{n}_{\\varphi}(\\mathbf{x}) = -\\tau^{-1} H(\\mathbf{x})^{-1} \\left( \\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{x}) \\right) = \\Delta \\mathbf{x},\n\\]\nand therefore,\n\\[\n\\mathbf{x}^+ = \\mathbf{x} + \\Delta \\mathbf{x} = \\mathbf{x} + \\mathbf{n}_{\\varphi}(\\mathbf{x}).\n\\]\n","sha256_utf8":"161ea89e030bba2c2f550592dfa06f48e6dd5be11d39c1a6cf7e433379a88131"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"牛顿收缩控制新残差","source_units":[{"start":25200,"end":26116,"start_line":279,"end_line":288,"raw_tex":"For any $\\mathbf{v} \\in \\mathcal{K}^\\circ$,\n\\[\n\\|\\mathbf{n}_{\\varphi}(\\mathbf{v})\\|_{\\mathbf{v}} = \\left\\| \\tau^{-1} H(\\mathbf{v})^{-1} \\left( \\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{v}) \\right) \\right\\|_{\\mathbf{v}}\n= \\tau^{-1} \\|\\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{v})\\|_{\\mathbf{v}}^*.\n\\]\nFor $\\mathbf{v} = \\mathbf{x}$, $\\|\\mathbf{n}_{\\varphi}(\\mathbf{x})\\|_{\\mathbf{x}} = \\|\\Delta \\mathbf{x}\\|_{\\mathbf{x}} \\leq \\eta$, using Lemma \\ref{lemma:sd_properties}, part \\ref{sd_prop:second}. As $\\eta < 1$, we can use \\ref{prop:scb_NS}:\n\\begin{equation} \\label{eq:UB_NS}\n    \\tau^{-1} \\|\\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{x}^+)\\|^*_{\\mathbf{x}^+} = \\|\\mathbf{n}_{\\varphi}(\\mathbf{x}^+)\\|_{\\mathbf{x}^+}\n\\leq \\left( \\frac{\\|\\mathbf{n}_{\\varphi}(\\mathbf{x})\\|_{\\mathbf{x}}}{1 - \\|\\mathbf{n}_{\\varphi}(\\mathbf{x})\\|_{\\mathbf{x}}} \\right)^2 \\leq \\frac{\\eta^2}{(1 - \\eta)^2}.\n\\end{equation}\n","sha256_utf8":"f28eee4011b71b9988143de0c0173f170b9dc8912e5f18f30f9c2344e3e0902a"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"缩减路径参数仍在邻域","source_units":[{"start":26117,"end":27006,"start_line":290,"end_line":300,"raw_tex":"Thus,\n\\begin{align*}\n\\frac{1}{\\tau^+} \\left\\| \\mathbf{s}^+ + \\tau^+ g(\\mathbf{x}^+) \\right\\|_{\\mathbf{x}^+}^*\n&= \\frac{1}{1 - \\vartheta} \\tau^{-1} \\| \\mathbf{s} + \\Delta \\mathbf{s} + \\tau^+ g(\\mathbf{x}^+) \\|_{\\mathbf{x}^+}^* \\\\\n&\\leq \\frac{1}{1 - \\vartheta} \\tau^{-1} \\| \\mathbf{s} + \\Delta \\mathbf{s} + \\tau g(\\mathbf{x}^+)\\|_{\\mathbf{x}^+}^* + \\frac{\\vartheta}{1 - \\vartheta} \\| g(\\mathbf{x}^+) \\|^*_{\\mathbf{x}^+} \\\\\n&\\stackrel{\\eqref{eq:UB_NS},\\ref{prop:lhscb_norm}}{\\leq} \\frac{1}{1 - \\vartheta}  \\frac{\\eta^2}{(1 - \\eta)^2} + \\frac{\\vartheta}{1 - \\vartheta} \\sqrt{\\nu} \\\\\n&= \\frac{\\sqrt{\\nu} + 1}{\\sqrt{\\nu} + 1 - \\eta/2} \\cdot \\frac{\\eta^2}{(1 - \\eta)^2} + \\frac{\\eta / 2}{\\sqrt{\\nu} + 1 - \\eta/2} \\sqrt{\\nu} \\\\\n&\\leq \\frac{2}{2 - \\eta / 2} \\cdot \\frac{\\eta^2}{(1 - \\eta)^2} + \\frac{\\eta}{2}.\n\\end{align*}\n\nThe last expression is less than $\\eta$ if $\\eta \\leq \\frac{1}{4}$ holds.\n","sha256_utf8":"7287d5df683e1722969330a12c1125dd8896f3dc3cb05c2612ee6ccc97cc9503"}]},{"id":"S032","semantic_type":"lemma","kind":"lemma","title":"全步长后的对偶间隙界","source_units":[{"start":27273,"end":27660,"start_line":305,"end_line":308,"raw_tex":"\\begin{lemma} \\label{lemma:DG}\nLet $(\\mathbf{x}, \\mathbf{y}, \\mathbf{s}) \\in \\mathcal{N}(\\eta, \\tau)$ and let $(\\Delta \\mathbf{x}, \\Delta \\mathbf{y}, \\Delta \\mathbf{s})$ be the solution of \\eqref{eq:NS}. We have the following lower and upper bounds on the duality gap after a full Newton step:\n    \\[ \\tau(\\nu - \\eta^2) \\leq (\\mathbf{x}^+)^\\top \\mathbf{s}^+ \\leq \\tau \\nu. \\]\n\\end{lemma}","sha256_utf8":"4f3303ed136706e50fc800569765ae003eaae36a4306266423c37b6275292079"}]},{"id":"S033","semantic_type":"proof","kind":"proof","title":"对偶间隙界完整证明","source_units":[{"start":27662,"end":29340,"start_line":310,"end_line":333,"raw_tex":"\\begin{proof}\nUsing \\ref{sd_prop:first} from Lemma \\ref{lemma:sd_properties},\n    \\[ (\\mathbf{x}^+)^\\top \\mathbf{s}^+ = (\\mathbf{x} + \\Delta \\mathbf{x})^\\top (\\mathbf{s} + \\Delta \\mathbf{s}) = \\mathbf{x}^\\top \\mathbf{s} + \\mathbf{s}^\\top \\Delta \\mathbf{x} + \\mathbf{x}^\\top \\Delta \\mathbf{s}. \\]\nFrom the third equation of \\eqref{eq:NS}, we have\n\\begin{align*}\n    \\Delta \\mathbf{x}^\\top \\mathbf{s} &= \\Delta \\mathbf{x}^\\top \\mathbf{s} + \\Delta \\mathbf{x}^\\top \\Delta \\mathbf{s} \\\\\n&= \\Delta \\mathbf{x}^\\top \\mathbf{s} - \\Delta \\mathbf{x}^\\top (\\mathbf{s} + \\tau g(\\mathbf{x}) + \\tau H(\\mathbf{x}) \\Delta \\mathbf{x}) \\\\\n&= -\\tau (\\Delta \\mathbf{x}^\\top g(\\mathbf{x}) + \\|\\Delta \\mathbf{x}\\|_{\\mathbf{x}}^2).\n\\end{align*}\nSimilarly, we can express $\\mathbf{x}^\\top \\Delta \\mathbf{s}$ as\n\\begin{align*}\n\\mathbf{x}^\\top \\Delta \\mathbf{s} &= - \\mathbf{x}^\\top \\left( \\mathbf{s} + \\tau g(\\mathbf{x}) + \\tau H(\\mathbf{x}) \\Delta \\mathbf{x} \\right) \\\\\n&\\stackrel{\\ref{prop:lhscb_Hessian},\\ref{prop:lhscb_gradient}}{=} -\\mathbf{x}^\\top \\mathbf{s} + \\tau \\nu + \\tau \\Delta \\mathbf{x}^\\top g(\\mathbf{x}).\n\\end{align*}\n\nCombining the previous equations, we can write $(\\mathbf{x}^+)^\\top \\mathbf{s}^+$ as\n\\begin{align*}\n(\\mathbf{x}^+)^\\top \\mathbf{s}^+ &= \\mathbf{x}^\\top \\mathbf{s} - \\tau \\Delta \\mathbf{x}^\\top g(\\mathbf{x}) - \\tau \\|\\Delta \\mathbf{x}\\|_{\\mathbf{x}}^2 - \\mathbf{x}^\\top \\mathbf{s} + \\tau \\nu + \\tau \\Delta \\mathbf{x}^\\top g(\\mathbf{x}) \\\\\n&= \\tau (\\nu -  \\|\\Delta \\mathbf{x}\\|_{\\mathbf{x}}^2).\n\\end{align*}\nUsing part \\ref{sd_prop:second} of Lemma \\ref{lemma:sd_properties}, we get\n\\[ \\tau(\\nu-\\eta^2) \\leq (\\mathbf{x}^+)^\\top \\mathbf{s}^+ \\leq \\tau\\nu.  \\]\n\n\\end{proof}","sha256_utf8":"edafde0adc6822121351ec3d26e7625b6235bad24e4876f86e5f096f537796a5"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"正交性展开新间隙","source_units":[{"start":27676,"end":27958,"start_line":311,"end_line":312,"raw_tex":"Using \\ref{sd_prop:first} from Lemma \\ref{lemma:sd_properties},\n    \\[ (\\mathbf{x}^+)^\\top \\mathbf{s}^+ = (\\mathbf{x} + \\Delta \\mathbf{x})^\\top (\\mathbf{s} + \\Delta \\mathbf{s}) = \\mathbf{x}^\\top \\mathbf{s} + \\mathbf{s}^\\top \\Delta \\mathbf{x} + \\mathbf{x}^\\top \\Delta \\mathbf{s}. \\]\n","sha256_utf8":"6da591feb062cf9b12230939c8c6e7b97d26b6b8b9c488d9cb7b0d62013bf364"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"化简原始方向交叉项","source_units":[{"start":27958,"end":28383,"start_line":313,"end_line":318,"raw_tex":"From the third equation of \\eqref{eq:NS}, we have\n\\begin{align*}\n    \\Delta \\mathbf{x}^\\top \\mathbf{s} &= \\Delta \\mathbf{x}^\\top \\mathbf{s} + \\Delta \\mathbf{x}^\\top \\Delta \\mathbf{s} \\\\\n&= \\Delta \\mathbf{x}^\\top \\mathbf{s} - \\Delta \\mathbf{x}^\\top (\\mathbf{s} + \\tau g(\\mathbf{x}) + \\tau H(\\mathbf{x}) \\Delta \\mathbf{x}) \\\\\n&= -\\tau (\\Delta \\mathbf{x}^\\top g(\\mathbf{x}) + \\|\\Delta \\mathbf{x}\\|_{\\mathbf{x}}^2).\n\\end{align*}\n","sha256_utf8":"400b63aa43a0fcba7da05f3177d6885a494214d84890c39f49e8694a6fa8d363"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"齐次恒等式化简对偶项","source_units":[{"start":28383,"end":28770,"start_line":319,"end_line":323,"raw_tex":"Similarly, we can express $\\mathbf{x}^\\top \\Delta \\mathbf{s}$ as\n\\begin{align*}\n\\mathbf{x}^\\top \\Delta \\mathbf{s} &= - \\mathbf{x}^\\top \\left( \\mathbf{s} + \\tau g(\\mathbf{x}) + \\tau H(\\mathbf{x}) \\Delta \\mathbf{x} \\right) \\\\\n&\\stackrel{\\ref{prop:lhscb_Hessian},\\ref{prop:lhscb_gradient}}{=} -\\mathbf{x}^\\top \\mathbf{s} + \\tau \\nu + \\tau \\Delta \\mathbf{x}^\\top g(\\mathbf{x}).\n\\end{align*}\n","sha256_utf8":"2a1ac21ed96450d0789cc1fb1493a01caa483bb81adbb32fb08aa08f51e1bb44"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"合并恒等式得到间隙界","source_units":[{"start":28771,"end":29328,"start_line":325,"end_line":331,"raw_tex":"Combining the previous equations, we can write $(\\mathbf{x}^+)^\\top \\mathbf{s}^+$ as\n\\begin{align*}\n(\\mathbf{x}^+)^\\top \\mathbf{s}^+ &= \\mathbf{x}^\\top \\mathbf{s} - \\tau \\Delta \\mathbf{x}^\\top g(\\mathbf{x}) - \\tau \\|\\Delta \\mathbf{x}\\|_{\\mathbf{x}}^2 - \\mathbf{x}^\\top \\mathbf{s} + \\tau \\nu + \\tau \\Delta \\mathbf{x}^\\top g(\\mathbf{x}) \\\\\n&= \\tau (\\nu -  \\|\\Delta \\mathbf{x}\\|_{\\mathbf{x}}^2).\n\\end{align*}\nUsing part \\ref{sd_prop:second} of Lemma \\ref{lemma:sd_properties}, we get\n\\[ \\tau(\\nu-\\eta^2) \\leq (\\mathbf{x}^+)^\\top \\mathbf{s}^+ \\leq \\tau\\nu.  \\]\n","sha256_utf8":"f17aecfe569921a2f89750721a797731e2a83b8999a691413e9a7e977c318834"}]},{"id":"S038","semantic_type":"external_fact","kind":"external_fact","title":"原始对偶椭球内点性质","source_units":[],"declared_external":true,"declaration_text":["For x∈K°, the primal unit Dikin ellipsoid lies in K°. For a logarithmically homogeneous self-concordant barrier, {s: ||s+g(x)||_x^*<1}⊂(K*)°. Positive scaling preserves both cone interiors."]},{"id":"S039","semantic_type":"external_fact","kind":"external_fact","title":"锥规划对偶与间隙","source_units":[],"declared_external":true,"declaration_text":["Primal and dual strict feasibility imply strong conic duality. For a feasible pair, cᵀx-bᵀy=xᵀs≥0; this gap bounds both primal and dual objective suboptimalities."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S016","edges":[{"source_id":"S001","target_id":"S002","kind":"supports"},{"source_id":"S001","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S004","kind":"supports"},{"source_id":"S002","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S005","kind":"supports"},{"source_id":"S004","target_id":"S006","kind":"supports"},{"source_id":"S001","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S008","target_id":"S010","kind":"supports"},{"source_id":"S008","target_id":"S011","kind":"supports"},{"source_id":"S009","target_id":"S004","kind":"supports"},{"source_id":"S009","target_id":"S012","kind":"supports"},{"source_id":"S010","target_id":"S005","kind":"supports"},{"source_id":"S013","target_id":"S011","kind":"supports"},{"source_id":"S013","target_id":"S029","kind":"supports"},{"source_id":"S014","target_id":"S012","kind":"supports"},{"source_id":"S015","target_id":"S031","kind":"supports"},{"source_id":"S006","target_id":"S023","kind":"supports"},{"source_id":"S005","target_id":"S024","kind":"supports"},{"source_id":"S006","target_id":"S024","kind":"supports"},{"source_id":"S010","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S006","target_id":"S025","kind":"supports"},{"source_id":"S010","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S004","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S006","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S010","target_id":"S030","kind":"supports"},{"source_id":"S021","target_id":"S030","kind":"supports"},{"source_id":"S011","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S012","target_id":"S031","kind":"supports"},{"source_id":"S005","target_id":"S031","kind":"supports"},{"source_id":"S038","target_id":"S030","kind":"supports"},{"source_id":"S038","target_id":"S031","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S027","kind":"supports"},{"source_id":"S031","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S026","kind":"supports"},{"source_id":"S021","target_id":"S034","kind":"supports"},{"source_id":"S021","target_id":"S035","kind":"supports"},{"source_id":"S006","target_id":"S035","kind":"supports"},{"source_id":"S010","target_id":"S035","kind":"supports"},{"source_id":"S006","target_id":"S036","kind":"supports"},{"source_id":"S012","target_id":"S036","kind":"supports"},{"source_id":"S034","target_id":"S037","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"},{"source_id":"S021","target_id":"S037","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S035","target_id":"S033","kind":"supports"},{"source_id":"S036","target_id":"S033","kind":"supports"},{"source_id":"S037","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S007","target_id":"S018","kind":"supports"},{"source_id":"S026","target_id":"S018","kind":"supports"},{"source_id":"S032","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S019","target_id":"S017","kind":"supports"},{"source_id":"S020","target_id":"S017","kind":"supports"},{"source_id":"S039","target_id":"S003","kind":"supports"},{"source_id":"S003","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"}]}}
{"record_id":"2505.03050v1","paper":{"title":"Convergence of First-Order Algorithms with Momentum from the Perspective of an Inexact Gradient Descent Method","authors":["Pham Duy Khanh","Boris Mordukhovich","Dat Ba Tran"],"arxiv_id":"2505.03050","version":"v1","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-025-02864-5","abstract_url":"https://arxiv.org/abs/2505.03050v1","pdf_url":"https://arxiv.org/pdf/2505.03050v1"},"context":{"raw_tex":"This paper addresses the design and convergence analysis of first-order optimization algorithms with momentum for solving unconstrained optimization problems of the type\n\\begin{align}\\label{main problem}\n{\\rm minimize}\\quad f(x)\\quad \\text{ subject to }x\\in\\R^n,\n\\end{align}\nwhere $f:\\R^n\\rightarrow\\overline{\\R}$ is a continuously differentiable function on some open and convex subset $X$ of $\\R^n$ having a Lipschitz continuous gradient with constant $L>0$ on $X$, i.e.,\n\\begin{align*}\n\\norm{\\nabla f(x)-\\nabla f(y)}\\le L\\norm{x-y}\\;\\text{ for all }\\;x,y\\in X.\n\\end{align*}\n\n\nAs always, $\\N:=\\{1,2,\\ldots\\}$ signifies the collections of natural numbers. Given $r>0$, the open ball centered at $\\bar x$ with radius $r$ is denoted by $\\mathbb{B}(\\bar x,\\rho)$ and its closure is $\\overline{\\mathbb{B}}(\\bar x,\\rho)$.\nThe symbol $x^k\\st{J}{\\to}\\ox$ means that $x^k\\to\\ox$ as $k\\to\\infty$ with $k\\in J\\subset\\N$. Recall that $\\bar x$ is a \\textit{stationary point} and that $f(\\bar x)$ is a \\textit{critical value} of a $\\mathcal{C}^1$-smooth function $f\\colon\\R^n\\rightarrow\\R$ if $\\nabla f(\\bar x)=0$. A $\\C^1-$smooth function $f:\\Omega\\rightarrow {\\R}$ is said to have a Lipschitz continuous gradient with the uniform constant $L>0$ on some subset $\\Omega$ of $\\R^n$ if \n\\begin{align}\\label{Lips def}\n\\norm{\\nabla f(x)-\\nabla f(y)}\\le L\\norm{x-y}\\;\\text{ for all }\\;x,y\\in\\Omega.\n\\end{align}\nA function $f\\colon\\R^n\\rightarrow\\R$ is {\\em locally Lipschitz continuous} around $\\bar x$  if there is a neighborhood $\\Omega$ of $\\bar x$ such that \\eqref{Lips def} holds; $f$ {\\em globally Lipschitz continuous} if \\eqref{Lips def} holds for $\\Omega=\\R^n$. In what follows, we denote by $\\mathcal{C}^{1,1}$ the class of smooth functions that have a {\\em locally Lipschitz continuous gradient} around $\\bar x$, and by $\\mathcal{C}^{1,1}_L$ the class of smooth functions that have a {\\em globally Lipschitz continuous gradient with constant} $L>0$.\n\nNext we recall the subdifferential constructions for proper extended-real-valued functions $h:\\R^n\\rightarrow\\Bar\\R:=(-\\infty,\\infty]$ with $\\dom h:=\\{x\\in\\R^n\\;|\\;h(x)<\\infty\\}\\ne\\emp$ taken from \\cite{Mordukhovich2006,Mordukhovich2018,RockafellarWets} and used below. The (Fr\\'echet) \\textit{regular subdifferential} of $ h $ at $\\ox \\in\\dom  h $ is \n\\begin{align}\\label{frechet}\n\\widehat{\\partial} h (\\bar x):=\\Big\\{v\\in \\R^n\\;\\Big|\\;\\liminf_{x\\rightarrow\\bar x}\\dfrac{ h (x)- h (\\bar x)-\\dotproduct{v,x-\\bar x}}{\\norm{x-\\bar x}}\\ge 0\\Big\\},\n\\end{align}\nand the (Mordukhovich) \\textit{limiting subdifferential} of $ h $ at $\\bar x\\in\\dom  h $ is defined by \n\\begin{align}\\label{limiting subdiff}\n\\partial  h (\\bar x):=\\set{v\\in\\R^n\\;\\big|\\;\\exists\\,x^k\\rightarrow \\ox,\\  h (x^k)\\rightarrow h (\\bar x),\\ \\widehat{\\partial}h(x^k)\\ni v^k\\rightarrow v}.\n\\end{align}\nWhen $ h $ is $\\mathcal{C}^1$-smooth around $\\bar x$, both subdifferentials in \\eqref{frechet} and \\eqref{limiting subdiff} reduce to the gradient $\\nabla  h (\\bar x)$, while for convex functions they agree with the classical subdifferential of convex analysis. A necessary condition for $\\bar x\\in\\R^n$ to be a local minimizer of $ h $ is \n$0\\in \\partial h (\\bar x)$. Any point $\\bar x$ satisfying the latter condition is known in the literature as an {\\em M$($ordukhovich$)$-stationary} (or {\\em limiting-stationary}) point of $h$. Since this paper does not deal with other stationary concepts for nonsmooth functions, we simply use the {\\em stationary} point term in what follows.\n\n\n\\begin{Definition}\\rm \\label{KL ine}\\rm We say that a lower semicontinuous (l.s.c.) function $f:\\R^n\\rightarrow\\overline{\\R}$ satisfies the $($basic$)$ \\textit{Polyak-\\L ojasiewicz-Kurdyka} $(PLK)$ condition at $\\bar x\\in\\dom \\partial f$ if there exist $\\eta\\in (0,\\infty]$, a neighborhood $U$ of $\\bar x$, and a desingularizing concave function $\\varphi:[0,\\eta)\\rightarrow[0,\\infty)$ such that:\n\n{\\bf(i)} $\\varphi(0)=0$.\n\n{\\bf(ii)} $\\varphi$ is $\\mathcal{C}^1$-smooth on $(0,\\eta)$.\n\n{\\bf(iii)} $\\varphi'>0$ on $(0,\\eta)$.\n\n{\\bf(iv)} For all $x\\in U$ with $f(\\bar{x})<f(x)<f(\\bar{x})+\\eta$, we have \n\\begin{align}\\label{KL 2}\n\\varphi'\\big(f(x)-f(\\bar{x})\\big)d(0,\\partial f(x))\\ge 1.\n\\end{align}\n\\end{Definition}\n\n\\begin{Remark}\\rm\\label{algebraic}If $f$ satisfies the PLK property at $\\bar x$ with a neighborhood $U$, it is clear that the same property holds for any $\\tilde x\\in U\\cap {\\rm dom\\; }\\partial f$ satisfying $f(\\tilde x)=f(\\bar x).$\nIt has been realized that the PLK condition is satisfied in rather  broad settings. In particular, it holds at every \nnonstationary point of $f$, for semianalytic and more general functions known as definable in o-minimal structures; see, e.g., \\cite{AbsilMahonyAndrews2005,AttouchBolteRedontSoubeyran2010,AttouchBolteSvaiter2013} and the references therein. As demonstrated in \\cite[Section~2]{KhanhMordukhovichTran2023}, the PLK condition formulated in \\cite{AttouchBolteRedontSoubeyran2010} is stronger than the one in Definition~\\ref{KL ine}.\n\\end{Remark}\n\nNote that condition (ii) ensures that, for any fixed $x\\in(0,\\eta)$ and $a\\in(0,x]$, the function $\\varphi'$ is integrable on $[a,x]$ and thus $\\varphi(x)=\\varphi(a)+\\int_a^{x}\\varphi'(t)dt$. Taking the limit as $a\\rightarrow0^+$ and using  \n$\\varphi(0)=0$ give us $\\varphi(x)=\\int_0^{x}\\varphi'(t)dt$. The following result plays an important role in our local convergence analysis.\n\n\n\\begin{Remark}\\rm\\label{rmk general rate} It is easy to see  that condition \\eqref{two conditions} in Proposition~\\ref{general rate} yields the fulfillment of (H2) and of (H1) with $\\sigma=\\frac{\\alpha}{\\beta}$  in Proposition~\\ref{general convergence under KL}. Proposition~\\ref{general rate} obviously holds true even for extended-real-valued functions $f$ provided that $\\{x^k\\}$ is convergent and $f$ is ${\\cal C}^1$-smooth on an open set containing $\\{x^k\\}$.\n\\end{Remark}\n\nTo proceed further, we recall the following fundamental construction that is usually employed in the analysis of momentum methods. Given a number $\\alpha>0$, consider the {\\em Lyapunov function} $H_\\alpha:\\R^n\\times \\R^n\\rightarrow\\R$ associated with the objective function $f$ that is defined by\n\\begin{align}\\label{lyapunov func}\nH_\\alpha(x,y):=f(x)+\\alpha\\norm{x-y}^2\\text{ for all }x,y\\in\\R^n.\n\\end{align}\n\n\\begin{align}\\label{construction varphi}\n\\varphi(t):=\\sqrt{\\frac{8}{\\varepsilon}}\\int_0^{t/2} \\max\\set{\\psi'(s),\\frac{1}{\\sqrt{s}}}ds\\;\\text{ for all }\\;t\\in (0,\\eta).\n\\end{align}\n\n\\begin{align}\\label{tilde M}\n\\tilde{M}:=\\begin{cases}\n1&\\text{ if }\\quad q\\in \\left[0,\\frac{1}{2}\\right),\\\\\n\\max \\set{M(1-q),1}&\\text{ if }\\quad q=\\frac{1}{2},\\\\\nM(1-q)&\\text{ if }\\quad q\\in \\left(\\frac{1}{2},1\\right).\n\\end{cases}\n\\end{align}\n\n\\begin{Algorithm}[IGDm]\\hlabel{IGDm}\\quad\n\\begin{enumerate}[-]\n\\item \\textbf{Optimization problem: } \n\\begin{align*}\n\\eqref{main problem}\\;\\text{ with }\\begin{cases}X=\\R^n\\;\\;\\text{ for global minima,}\\\\\nX=\\B(\\bar x,r)\\;\\text{ for a local minimizer }\\bar x\\;\\text{ with some }\\;r>0.\n\\end{cases}\n\\end{align*}\n\\vspace{-0.5cm}\n\\item \\textbf{Initialization: }Choose  $x^0=x^1\\in\\Omega,\\;\\set{\\beta_k}\\subset [0,\\infty),\\;\\set{\\gamma_k}\\subset[0,\\infty)$,  $\\nu \\in(0,1),$ $\\tau>0$.\n\\item \\textbf{Parameter conditions:} $\\bar{\\beta}:=\\sup \\beta_k<1,$ $\\bar \\delta:=\\sup|\\beta_k-\\gamma_k|<\\infty.$\n\\item \\textbf{Iteration: } $(k\\ge 1)$ Update:\n\\begin{align}\n&x^k_{\\rm in}:=x^k+\\beta_k(x^k-x^{k-1}),\\label{step xin}\\\\\n&x^k_{\\rm ex}:=x^k+\\gamma_k(x^k-x^{k-1}),\\label{step xex}\\\\\n&x^{k+1}:=x^k_{\\rm in}-\\tau  g^k,\\;\\text{ where }\\;\\norm{g^k-\\nabla f(x^k_\\rex)}\\le \\nu \\norm{g^k}.\\label{iteration update}\n\\end{align}\n\\end{enumerate}\n\\end{Algorithm}\n\nDefine further the sequence $\\{z^k\\} \\subset \\mathbb{R}^{n}\\times \\mathbb{R}^n$ by $z^{k}:= (x^{k},x^{k-1})$ for all $k\\in\\N$ along which the fundamental properties of the Lyapunov function $H_\\alpha$ are establishes below. These properties are crucial for deriving and unifying local and global convergence properties of Algorithm~\\ref{IGDm}.\n\n\n\\begin{align}\\label{C1C2}\n\\alpha:&=\\frac{(L \\tau+1)\\bar \\beta^2+1-\\nu }{4\\tau},\\nonumber\\\\\nC_1:&=\\frac{1-\\nu-(L \\tau+1)\\bar \\beta^2-2L\\tau \\bar \\delta}{4\\tau},\\\\\nC_2:&=\\sqrt{2}\\max\\set{\\frac{\\nu+1}{\\tau },\\frac{\\bar\\beta(L\\tau+\\nu+1)+L\\tau\\bar\\delta}{\\tau }}+4{\\alpha}.\\nonumber\n\\end{align}\n\n\\begin{Assumption}\\rm\\label{assu}\n$\\nabla f(x^k_{\\rm ex})\\ne 0$ for all $k\\in\\N$.\n\\end{Assumption}","units":[{"start":6615,"end":7192,"start_line":172,"end_line":179,"raw_tex":"This paper addresses the design and convergence analysis of first-order optimization algorithms with momentum for solving unconstrained optimization problems of the type\n\\begin{align}\\label{main problem}\n{\\rm minimize}\\quad f(x)\\quad \\text{ subject to }x\\in\\R^n,\n\\end{align}\nwhere $f:\\R^n\\rightarrow\\overline{\\R}$ is a continuously differentiable function on some open and convex subset $X$ of $\\R^n$ having a Lipschitz continuous gradient with constant $L>0$ on $X$, i.e.,\n\\begin{align*}\n\\norm{\\nabla f(x)-\\nabla f(y)}\\le L\\norm{x-y}\\;\\text{ for all }\\;x,y\\in X.\n\\end{align*}\n","sha256_utf8":"47d52bfd3761fd0005119ec329a1dd0e3f9183d935730ed6e95f5b8699345bb2"},{"start":18460,"end":21375,"start_line":230,"end_line":246,"raw_tex":"As always, $\\N:=\\{1,2,\\ldots\\}$ signifies the collections of natural numbers. Given $r>0$, the open ball centered at $\\bar x$ with radius $r$ is denoted by $\\mathbb{B}(\\bar x,\\rho)$ and its closure is $\\overline{\\mathbb{B}}(\\bar x,\\rho)$.\nThe symbol $x^k\\st{J}{\\to}\\ox$ means that $x^k\\to\\ox$ as $k\\to\\infty$ with $k\\in J\\subset\\N$. Recall that $\\bar x$ is a \\textit{stationary point} and that $f(\\bar x)$ is a \\textit{critical value} of a $\\mathcal{C}^1$-smooth function $f\\colon\\R^n\\rightarrow\\R$ if $\\nabla f(\\bar x)=0$. A $\\C^1-$smooth function $f:\\Omega\\rightarrow {\\R}$ is said to have a Lipschitz continuous gradient with the uniform constant $L>0$ on some subset $\\Omega$ of $\\R^n$ if \n\\begin{align}\\label{Lips def}\n\\norm{\\nabla f(x)-\\nabla f(y)}\\le L\\norm{x-y}\\;\\text{ for all }\\;x,y\\in\\Omega.\n\\end{align}\nA function $f\\colon\\R^n\\rightarrow\\R$ is {\\em locally Lipschitz continuous} around $\\bar x$  if there is a neighborhood $\\Omega$ of $\\bar x$ such that \\eqref{Lips def} holds; $f$ {\\em globally Lipschitz continuous} if \\eqref{Lips def} holds for $\\Omega=\\R^n$. In what follows, we denote by $\\mathcal{C}^{1,1}$ the class of smooth functions that have a {\\em locally Lipschitz continuous gradient} around $\\bar x$, and by $\\mathcal{C}^{1,1}_L$ the class of smooth functions that have a {\\em globally Lipschitz continuous gradient with constant} $L>0$.\n\nNext we recall the subdifferential constructions for proper extended-real-valued functions $h:\\R^n\\rightarrow\\Bar\\R:=(-\\infty,\\infty]$ with $\\dom h:=\\{x\\in\\R^n\\;|\\;h(x)<\\infty\\}\\ne\\emp$ taken from \\cite{Mordukhovich2006,Mordukhovich2018,RockafellarWets} and used below. The (Fr\\'echet) \\textit{regular subdifferential} of $ h $ at $\\ox \\in\\dom  h $ is \n\\begin{align}\\label{frechet}\n\\widehat{\\partial} h (\\bar x):=\\Big\\{v\\in \\R^n\\;\\Big|\\;\\liminf_{x\\rightarrow\\bar x}\\dfrac{ h (x)- h (\\bar x)-\\dotproduct{v,x-\\bar x}}{\\norm{x-\\bar x}}\\ge 0\\Big\\},\n\\end{align}\nand the (Mordukhovich) \\textit{limiting subdifferential} of $ h $ at $\\bar x\\in\\dom  h $ is defined by \n\\begin{align}\\label{limiting subdiff}\n\\partial  h (\\bar x):=\\set{v\\in\\R^n\\;\\big|\\;\\exists\\,x^k\\rightarrow \\ox,\\  h (x^k)\\rightarrow h (\\bar x),\\ \\widehat{\\partial}h(x^k)\\ni v^k\\rightarrow v}.\n\\end{align}\nWhen $ h $ is $\\mathcal{C}^1$-smooth around $\\bar x$, both subdifferentials in \\eqref{frechet} and \\eqref{limiting subdiff} reduce to the gradient $\\nabla  h (\\bar x)$, while for convex functions they agree with the classical subdifferential of convex analysis. A necessary condition for $\\bar x\\in\\R^n$ to be a local minimizer of $ h $ is \n$0\\in \\partial h (\\bar x)$. Any point $\\bar x$ satisfying the latter condition is known in the literature as an {\\em M$($ordukhovich$)$-stationary} (or {\\em limiting-stationary}) point of $h$. Since this paper does not deal with other stationary concepts for nonsmooth functions, we simply use the {\\em stationary} point term in what follows.\n","sha256_utf8":"4380c46273343e54eff60ba8c17f8d688768a912cba58b2dc1624673fbe88c44"},{"start":22313,"end":23027,"start_line":259,"end_line":271,"raw_tex":"\\begin{Definition}\\rm \\label{KL ine}\\rm We say that a lower semicontinuous (l.s.c.) function $f:\\R^n\\rightarrow\\overline{\\R}$ satisfies the $($basic$)$ \\textit{Polyak-\\L ojasiewicz-Kurdyka} $(PLK)$ condition at $\\bar x\\in\\dom \\partial f$ if there exist $\\eta\\in (0,\\infty]$, a neighborhood $U$ of $\\bar x$, and a desingularizing concave function $\\varphi:[0,\\eta)\\rightarrow[0,\\infty)$ such that:\n\n{\\bf(i)} $\\varphi(0)=0$.\n\n{\\bf(ii)} $\\varphi$ is $\\mathcal{C}^1$-smooth on $(0,\\eta)$.\n\n{\\bf(iii)} $\\varphi'>0$ on $(0,\\eta)$.\n\n{\\bf(iv)} For all $x\\in U$ with $f(\\bar{x})<f(x)<f(\\bar{x})+\\eta$, we have \n\\begin{align}\\label{KL 2}\n\\varphi'\\big(f(x)-f(\\bar{x})\\big)d(0,\\partial f(x))\\ge 1.\n\\end{align}\n\\end{Definition}","sha256_utf8":"7f0750dc4b6353642beebc0f0eb509297d14f6b31b39af084f3ff7b592c36e5d"},{"start":23029,"end":23821,"start_line":273,"end_line":276,"raw_tex":"\\begin{Remark}\\rm\\label{algebraic}If $f$ satisfies the PLK property at $\\bar x$ with a neighborhood $U$, it is clear that the same property holds for any $\\tilde x\\in U\\cap {\\rm dom\\; }\\partial f$ satisfying $f(\\tilde x)=f(\\bar x).$\nIt has been realized that the PLK condition is satisfied in rather  broad settings. In particular, it holds at every \nnonstationary point of $f$, for semianalytic and more general functions known as definable in o-minimal structures; see, e.g., \\cite{AbsilMahonyAndrews2005,AttouchBolteRedontSoubeyran2010,AttouchBolteSvaiter2013} and the references therein. As demonstrated in \\cite[Section~2]{KhanhMordukhovichTran2023}, the PLK condition formulated in \\cite{AttouchBolteRedontSoubeyran2010} is stronger than the one in Definition~\\ref{KL ine}.\n\\end{Remark}","sha256_utf8":"5c1e542db7a10f11cdfc3ec8c18c0ba967d0bbf0e1b236f0e4fa34a28b02fd4f"},{"start":23823,"end":24206,"start_line":278,"end_line":279,"raw_tex":"Note that condition (ii) ensures that, for any fixed $x\\in(0,\\eta)$ and $a\\in(0,x]$, the function $\\varphi'$ is integrable on $[a,x]$ and thus $\\varphi(x)=\\varphi(a)+\\int_a^{x}\\varphi'(t)dt$. Taking the limit as $a\\rightarrow0^+$ and using  \n$\\varphi(0)=0$ give us $\\varphi(x)=\\int_0^{x}\\varphi'(t)dt$. The following result plays an important role in our local convergence analysis.\n","sha256_utf8":"dc6036cc0bfc97041a56d4c6b5233807df5f5987ffcd55d5848f6a65ff1b8d55"},{"start":27436,"end":27913,"start_line":325,"end_line":326,"raw_tex":"\\begin{Remark}\\rm\\label{rmk general rate} It is easy to see  that condition \\eqref{two conditions} in Proposition~\\ref{general rate} yields the fulfillment of (H2) and of (H1) with $\\sigma=\\frac{\\alpha}{\\beta}$  in Proposition~\\ref{general convergence under KL}. Proposition~\\ref{general rate} obviously holds true even for extended-real-valued functions $f$ provided that $\\{x^k\\}$ is convergent and $f$ is ${\\cal C}^1$-smooth on an open set containing $\\{x^k\\}$.\n\\end{Remark}","sha256_utf8":"f1f3e1f2588eee9734bc220f67f58552179d77651ff5b529d9fc116ae935bf5c"},{"start":27915,"end":28324,"start_line":328,"end_line":331,"raw_tex":"To proceed further, we recall the following fundamental construction that is usually employed in the analysis of momentum methods. Given a number $\\alpha>0$, consider the {\\em Lyapunov function} $H_\\alpha:\\R^n\\times \\R^n\\rightarrow\\R$ associated with the objective function $f$ that is defined by\n\\begin{align}\\label{lyapunov func}\nH_\\alpha(x,y):=f(x)+\\alpha\\norm{x-y}^2\\text{ for all }x,y\\in\\R^n.\n\\end{align}","sha256_utf8":"0d81c7d71248b723c5df3f9f827b8edca6dc3bb8f3d26118d530736312930301"},{"start":30498,"end":30677,"start_line":349,"end_line":351,"raw_tex":"\\begin{align}\\label{construction varphi}\n\\varphi(t):=\\sqrt{\\frac{8}{\\varepsilon}}\\int_0^{t/2} \\max\\set{\\psi'(s),\\frac{1}{\\sqrt{s}}}ds\\;\\text{ for all }\\;t\\in (0,\\eta).\n\\end{align}","sha256_utf8":"94048c5c1ebccd272147cf61617a1ea54d88ba4509ab5785bdef88ec65bbb38f"},{"start":31033,"end":31275,"start_line":354,"end_line":360,"raw_tex":"\\begin{align}\\label{tilde M}\n\\tilde{M}:=\\begin{cases}\n1&\\text{ if }\\quad q\\in \\left[0,\\frac{1}{2}\\right),\\\\\n\\max \\set{M(1-q),1}&\\text{ if }\\quad q=\\frac{1}{2},\\\\\nM(1-q)&\\text{ if }\\quad q\\in \\left(\\frac{1}{2},1\\right).\n\\end{cases}\n\\end{align}","sha256_utf8":"5e86627c8f7468910c570634ebd3163f72d4859a17ac190bf60d212c2ab64f40"},{"start":38438,"end":39376,"start_line":454,"end_line":472,"raw_tex":"\\begin{Algorithm}[IGDm]\\hlabel{IGDm}\\quad\n\\begin{enumerate}[-]\n\\item \\textbf{Optimization problem: } \n\\begin{align*}\n\\eqref{main problem}\\;\\text{ with }\\begin{cases}X=\\R^n\\;\\;\\text{ for global minima,}\\\\\nX=\\B(\\bar x,r)\\;\\text{ for a local minimizer }\\bar x\\;\\text{ with some }\\;r>0.\n\\end{cases}\n\\end{align*}\n\\vspace{-0.5cm}\n\\item \\textbf{Initialization: }Choose  $x^0=x^1\\in\\Omega,\\;\\set{\\beta_k}\\subset [0,\\infty),\\;\\set{\\gamma_k}\\subset[0,\\infty)$,  $\\nu \\in(0,1),$ $\\tau>0$.\n\\item \\textbf{Parameter conditions:} $\\bar{\\beta}:=\\sup \\beta_k<1,$ $\\bar \\delta:=\\sup|\\beta_k-\\gamma_k|<\\infty.$\n\\item \\textbf{Iteration: } $(k\\ge 1)$ Update:\n\\begin{align}\n&x^k_{\\rm in}:=x^k+\\beta_k(x^k-x^{k-1}),\\label{step xin}\\\\\n&x^k_{\\rm ex}:=x^k+\\gamma_k(x^k-x^{k-1}),\\label{step xex}\\\\\n&x^{k+1}:=x^k_{\\rm in}-\\tau  g^k,\\;\\text{ where }\\;\\norm{g^k-\\nabla f(x^k_\\rex)}\\le \\nu \\norm{g^k}.\\label{iteration update}\n\\end{align}\n\\end{enumerate}\n\\end{Algorithm}","sha256_utf8":"febf84450593a7bbfac6ae084bfed1df0c2adde0c7404e220e25a61792c77dcb"},{"start":41141,"end":41485,"start_line":497,"end_line":497,"raw_tex":"Define further the sequence $\\{z^k\\} \\subset \\mathbb{R}^{n}\\times \\mathbb{R}^n$ by $z^{k}:= (x^{k},x^{k-1})$ for all $k\\in\\N$ along which the fundamental properties of the Lyapunov function $H_\\alpha$ are establishes below. These properties are crucial for deriving and unifying local and global convergence properties of Algorithm~\\ref{IGDm}.\n","sha256_utf8":"29db792bfe5610fc3a91b71e6fdc0f90968be2dab2e5b9b742309166e9ef9884"},{"start":41820,"end":42110,"start_line":505,"end_line":509,"raw_tex":"\\begin{align}\\label{C1C2}\n\\alpha:&=\\frac{(L \\tau+1)\\bar \\beta^2+1-\\nu }{4\\tau},\\nonumber\\\\\nC_1:&=\\frac{1-\\nu-(L \\tau+1)\\bar \\beta^2-2L\\tau \\bar \\delta}{4\\tau},\\\\\nC_2:&=\\sqrt{2}\\max\\set{\\frac{\\nu+1}{\\tau },\\frac{\\bar\\beta(L\\tau+\\nu+1)+L\\tau\\bar\\delta}{\\tau }}+4{\\alpha}.\\nonumber\n\\end{align}","sha256_utf8":"cb76a598f004f1665a80dcc827ba20b2564442058e5f8b74a9faa8ccfe0aa456"},{"start":53484,"end":53582,"start_line":655,"end_line":657,"raw_tex":"\\begin{Assumption}\\rm\\label{assu}\n$\\nabla f(x^k_{\\rm ex})\\ne 0$ for all $k\\in\\N$.\n\\end{Assumption}","sha256_utf8":"191a0a25d4bd19c5e44ac87b22bb27b3d5b6976f807d68937829829f49991d46"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{Theorem}\\label{local con IGDm}\nLet the objective function $f:X\\rightarrow\\R$ in \\eqref{main problem} satisfy the basic PLK condition at its local minimizer $\\bar x$, and let $\\set{x^k}$ be the sequence of iterates generated by Algorithm~{\\rm\\ref{IGDm}} with $X=\\B(\\bar x,r)$ for some $r>0$ and $\\bar \\beta\\in [0,\\sqrt{1-\\nu})$. Then there exist $\\xi \\in(0,r]$ and $T>0$ such that for any initial point $x^0\\in \\mathbb{B}(\\bar x, \\xi)$ and $\\tau\\in (0, T]$, we have the convergence properties$:$\n\\begin{enumerate}\n\\item[\\bf(i)] The sequences $\\set{x^k},\\set{x^k_{\\rm in}},\\set{x^k_{\\rm ex}}$ stay in $\\mathbb{B}(\\bar x,r)$ and converge to some local minimizer $\\tilde x\\in\\B(\\bar x,r)$ of $f$ with $f(\\tilde x)=f(\\bar x)$.\n\n\\item[\\bf(ii)] If the exponential PLK condition is satisfied with the desingularizing function $\\psi(t)= Mt^{1-q}$ for some $M>0$ and $q\\in(0,1)$, then the following convergence rates are guaranteed as $k\\to\\infty:$\n\\end{enumerate}\\vspace*{-0.1in}\n\n $\\bullet$ For $q=1/2$, the sequences $\\set{x^k},\\set{\\nabla f(x^k)},\\set{f(x^k)}$ converge linearly to $\\tilde x,0,f(\\tilde x),$ respectively.\n\n $\\bullet$ For $q\\in(1/2,1)$, we have the asymptotic estimates\n $$\n \\norm{x^k-\\tilde x }=\\mathcal{O}\\brac{k^{-\\frac{1-q}{2q-1}}},\\quad\\norm{\\nabla f(x^k)}=\\mathcal{O}\\brac{k^{-\\frac{1-q}{2q-1}}},\\quad f(x^k)-f(\\tilde x)=\\mathcal{O}\\brac{k^{-\\frac{2-2q}{2q-1}}}.\n $$\n \\end{Theorem}","units":[{"start":61301,"end":62705,"start_line":722,"end_line":736,"raw_tex":"\\begin{Theorem}\\label{local con IGDm}\nLet the objective function $f:X\\rightarrow\\R$ in \\eqref{main problem} satisfy the basic PLK condition at its local minimizer $\\bar x$, and let $\\set{x^k}$ be the sequence of iterates generated by Algorithm~{\\rm\\ref{IGDm}} with $X=\\B(\\bar x,r)$ for some $r>0$ and $\\bar \\beta\\in [0,\\sqrt{1-\\nu})$. Then there exist $\\xi \\in(0,r]$ and $T>0$ such that for any initial point $x^0\\in \\mathbb{B}(\\bar x, \\xi)$ and $\\tau\\in (0, T]$, we have the convergence properties$:$\n\\begin{enumerate}\n\\item[\\bf(i)] The sequences $\\set{x^k},\\set{x^k_{\\rm in}},\\set{x^k_{\\rm ex}}$ stay in $\\mathbb{B}(\\bar x,r)$ and converge to some local minimizer $\\tilde x\\in\\B(\\bar x,r)$ of $f$ with $f(\\tilde x)=f(\\bar x)$.\n\n\\item[\\bf(ii)] If the exponential PLK condition is satisfied with the desingularizing function $\\psi(t)= Mt^{1-q}$ for some $M>0$ and $q\\in(0,1)$, then the following convergence rates are guaranteed as $k\\to\\infty:$\n\\end{enumerate}\\vspace*{-0.1in}\n\n $\\bullet$ For $q=1/2$, the sequences $\\set{x^k},\\set{\\nabla f(x^k)},\\set{f(x^k)}$ converge linearly to $\\tilde x,0,f(\\tilde x),$ respectively.\n\n $\\bullet$ For $q\\in(1/2,1)$, we have the asymptotic estimates\n $$\n \\norm{x^k-\\tilde x }=\\mathcal{O}\\brac{k^{-\\frac{1-q}{2q-1}}},\\quad\\norm{\\nabla f(x^k)}=\\mathcal{O}\\brac{k^{-\\frac{1-q}{2q-1}}},\\quad f(x^k)-f(\\tilde x)=\\mathcal{O}\\brac{k^{-\\frac{2-2q}{2q-1}}}.\n $$\n \\end{Theorem}","sha256_utf8":"d3b4c48fb3fd270179056599ea04b32873b3b26c84712568a5f47d134fab8900"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{Lemma}\\label{lemma descent}\nGiven $f:\\R^n\\rightarrow\\R$ and $x,y\\in\\Omega$, assume that $f$ is differentiable on the line segment $[x,y]$ and its derivative is Lipschitz continuous on this segment with constant $L>0$. Then we have\n\\begin{align}\\label{eq descent}\n\\abs{f(y)-f(x)-\\dotproduct{\\nabla f(x),y-x}}\\le \\frac{L}{2}\\norm{y-x}^2.\n\\end{align}\n\\end{Lemma}\n\n\\begin{Lemma}\\label{isolation lemma}\nLet $f:\\R^n\\rightarrow\\overline{\\R}$ be continuous around some local minimizer $\\bar x$ and satisfy the PLK condition at $\\bar x$. Then there is a neighborhood $U$ of $\\bar x$ such that $f(\\bar x)$ is the unique critical value of $f$ in $U$.\n\\end{Lemma}\n\n\\begin{proof}\nThe PLK condition for $f$ at $\\bar x$ gives us $\\eta\\in(0,\\infty],$ a neighborhood $U$ of $\\bar x$, and a function $\\varphi:[0,\\eta)\\rightarrow[0,\\infty)$ satisfying all the conditions in Definition~\\ref{KL ine}. Since $f$ is continuous around the local minimizer $\\bar x,$ we assume, by shrinking $U$ if necessary, that $f(\\bar x)\\le f(x)<f(\\bar x)+\\eta.$ Then condition (iv) in Definition~\\ref{KL 2} can be rewritten as\n\\begin{align*}\n\\varphi'(f(x)-f(\\bar x))d(0,\\partial f(x))\\ge 1\\text{ whenever }x\\in U,\\;f(x)\\ne f(\\bar x).\n\\end{align*}\nTherefore, $0\\notin \\partial f(x)$ for all such $x$, i.e., $f(\\bar x)$ is the unique critical value of $f$ on $U.$ \n\\end{proof}\n\n\\begin{Proposition}\\label{general convergence under KL}\nLet $f:\\R^n\\rightarrow\\R$ be a $\\mathcal{C}^1$-smooth function, and let the following conditions hold along a sequence of iterates $\\set{x^k}\\subset\\R^n$ for the function $f$:\n\\begin{itemize}\n\\item[\\bf(H1)] {\\rm(primary descent condition)}. There exists $\\sigma>0$ such that for sufficiently large $k\\in\\N$, we have \n\\begin{align*}\n f(x^k)-f(x^{k+1})\\ge\\sigma\\norm{\\nabla f(x^k)}\\cdot\\norm{x^{k+1}-x^k}.\n\\end{align*}\n\\item[\\bf(H2)] {\\rm(complementary descent condition)}. For sufficiently large $k\\in\\N$, we have\n\\begin{align*}\n \\big[f(x^{k+1})=f(x^k)\\big]\\Longrightarrow [x^{k+1}=x^k].\n\\end{align*}\n\\end{itemize}\nIf $\\bar x$ is an accumulation point of $\\set{x^k}$  and $f$ satisfies the PLK property at $\\bar x$, then $x^k\\rightarrow\\bar x$ as $k\\to\\infty$.\n\\end{Proposition}\n\n\\begin{Proposition}\\label{general rate}\nLet $f:\\R^n\\rightarrow\\R$ be a $\\mathcal{C}^1$-smooth function, and let $\\set{x^k}$ satisfy the conditions \n\\begin{align}\\label{two conditions}\nf(x^k)-f(x^{k+1})\\ge \\alpha\\norm{x^{k+1}-x^k}^2\\;\\text{ and }\\;\\norm{\\nabla f(x^k)}\\le \\beta\\norm{x^{k+1}-x^k}\n\\end{align}\nfor all $k\\in\\N$ sufficiently large with some constants $\\alpha,\\beta>0$. Suppose  in addition that $\\bar x$ is an accumulation point of $\\set{x^k}$ and that $f$ satisfies the $($exponential$)$ PLK property at $\\bar x$ with $\\varphi(t)=Mt^{1-q}$ for some $M>0$ and $q\\in(0,1)$. Then the following convergence rates are guaranteed:\n\n{\\bf (i)} For $q\\in(0,1/2)$, the sequence $\\set{x^k}$ terminates at $\\bar x$ in finite steps.\n\n{\\bf (ii)} For $q=1/2$, the sequence $\\set{x^k}$ converges linearly to $\\bar x$ as $k\\to\\infty$.\n\n{\\bf (iii)} For $q\\in(1/2,1)$, we have  $\\norm{x^k-\\bar x}=\\mathcal{O}( k^{-\\frac{1-q}{2q-1}})$ as $k\\to\\infty$\n\\end{Proposition}\n\n\\begin{Proposition}\\label{isolated} Let $f:\\R^n\\rightarrow\\overline\\R$ be a ${\\cal C}^1$-smooth function around its local minimizer $\\bar x$. Assume that $f$ satisfies the PLK condition at $\\bar x$ with the desingularizing function $\\psi:[0,\\eta)\\rightarrow[0,\\infty)$ for some $\\eta\\in (0,\\infty]$. Then the following assertions hold whenever $\\varepsilon\\in (0,1/4]$:\n   \n{\\bf(i)} The collection of functions $\\set{H_\\alpha}_{\\alpha\\ge \\varepsilon}$ satisfies the same PLK condition at $\\bar z:=(\\bar x,\\bar x)$ with the desingularizing function $\\varphi:[0,\\eta)\\rightarrow[0,\\infty)$ defined by $\\varphi(0):=0$ and \n\\begin{align}\\label{construction varphi}\n\\varphi(t):=\\sqrt{\\frac{8}{\\varepsilon}}\\int_0^{t/2} \\max\\set{\\psi'(s),\\frac{1}{\\sqrt{s}}}ds\\;\\text{ for all }\\;t\\in (0,\\eta).\n\\end{align}\n        \n{\\bf(ii)} If $\\psi(t)=Mt^{1-q}$ for some $M>0$ and $q\\in[0,1)$, then the function $\\varphi$ in \\eqref{construction varphi} is given by $\\varphi(t)=\\frac{2^{\\theta+1/2}\\tilde M}{\\sqrt{\\varepsilon}(1-\\theta)}t^{1-\\theta}$ when $t\\in (0,\\delta)$ for some $\\delta\\in(0,\\eta)$, where $\\theta:=\\max\\set{q,\\frac{1}{2}}$ and $\\tilde{M}>0$ is defined as \n\\begin{align}\\label{tilde M}\n\\tilde{M}:=\\begin{cases}\n1&\\text{ if }\\quad q\\in \\left[0,\\frac{1}{2}\\right),\\\\\n\\max \\set{M(1-q),1}&\\text{ if }\\quad q=\\frac{1}{2},\\\\\nM(1-q)&\\text{ if }\\quad q\\in \\left(\\frac{1}{2},1\\right).\n\\end{cases}\n\\end{align}\n\\end{Proposition}\n\n\\begin{proof}\nTo verify (i) pick any $\\varepsilon \\in (0,1/4]$ and use the PLK property of $f$ at its local minimizer $\\bar x$ to find a neighborhood $U$ of $\\bar x$ on which $f$ satisfies all the conditions in Definition~\\ref{KL ine} with the corresponding desingularizing function $\\psi: [0,\\eta) \\rightarrow [0,\\infty)$. In particular, we have\n\\begin{align}\\label{desingularizing}\n\\psi'\\big(f(x)-f(\\bar{x})\\big)\\norm{\\nabla f(x)}\\ge 1\\;\\mbox{  for all }\\;x \\in U\\;\\mbox{ with }\\;f(\\bar{x}) < f(x) < f(\\bar{x}) + \\eta.\n\\end{align}\nThe function $\\varphi$ in \\eqref{construction varphi} can be written as $\\varphi(t)=\\sqrt{\\frac{8}{\\varepsilon}}\\phi\\brac{\\frac{t}{2}}$, where $\\phi:[0,\\eta)\\rightarrow[0,\\infty)$ is defined by \n\\begin{align}\\label{construction phi t}\n\\phi(t):=\\int_0^t \\max\\set{\\psi'(s),\\frac{1}{\\sqrt{s}}}ds\\;\\text{ for }\\;t\\in(0,\\eta)\\;\\mbox{ with }\\;\\phi(0):=0.\n\\end{align}\nIt follows from the continuity of $\\psi$ at $0$ that\n\\begin{align*}\n0\\le \\phi(t)\\le \\int_0^t \\psi'(s)ds+ \\int_0^t\\frac{1}{\\sqrt{s}}ds=\\psi(t)+2\\sqrt{t}\\;\\text{ for }\\;t\\in(0,\\eta).\n\\end{align*}\nPassing the limit above as $t\\rightarrow0^+$ and using the continuity of $\\psi$ reveal that $\\phi$ is also continuous at $0$. Moreover, we deduce from \\eqref{construction phi t} that the derivative $\\phi'(t)=\\max \\set{\\psi'(t),\\frac{1}{\\sqrt{t}}}>0$ is positive, continuous, and nonincreasing on $(0,\\eta)$, which ensures the concavity of $\\phi$ on this interval. In addition, it follows that \n\\begin{align}\\label{phi'>1}\n\\phi'(f(x)-f(\\bar x))\\norm{\\nabla f(x)}\\ge \\psi'(f(x)-f(\\bar x))\\norm{\\nabla f(x)}\\ge 1\n\\end{align}\nfor $x\\in U$ with $f(\\bar x)< f(x)<f(x)+\\eta$. Since $\\varphi(t)=\\sqrt{\\frac{8}{\\varepsilon}}\\phi\\brac{\\frac{t}{2}}$, the domain of $\\varphi$ is $[0,2\\eta).$ It is clear that $\\varphi$ satisfies all the conditions as a desingularizing function on $[0,2\\eta)$ with\n\\begin{align}\\label{deri varphi}\n\\varphi'(t)=\\sqrt{\\frac{2}{\\varepsilon}}\\phi'\\brac{\\frac{t}{2}}>0 \\text{ for all }t\\in (0,2\\eta).\n\\end{align}\nTaking any $\\alpha\\ge \\varepsilon$ and $z=(x,y)\\in Z:= U\\times U$ with \n\\begin{align}\\label{halpha}\nH_\\alpha(\\bar z)<H_\\alpha(z)<H_\\alpha(\\bar z)+\\eta,\n\\end{align}\nwe aim at verifying the inequality\n\\begin{align}\\label{KL R2n}\n\\varphi'(H_\\alpha(z)-H_\\alpha(\\bar z))\\norm{\\nabla H_\\alpha (z)}\\ge 1.\n\\end{align}\nTo proceed, observe first the estimates\n\\begin{subequations}\n\\begin{align}\n\\norm{\\nabla H_\\alpha(x,y)}&=\\sqrt{\\norm{\\nabla f(x)+2\\alpha(x-y)}^2+\\norm{2\\alpha(x-y)}^2} \\label{g_Hal.a}\\\\\n&\\ge \\sqrt{\\frac{1}{4}\\norm{\\nabla f(x)}^2+\\alpha^2\\norm{x-y}^2}\\label{g_Hal.b}\\\\\n&\\ge \\sqrt{\\frac{1}{4}\\norm{\\nabla f(x)}^2+\\alpha\\varepsilon\\norm{x-y}^2}\\label{g_Hal.c}\\\\\n&\\ge \\sqrt{\\frac{\\varepsilon}{2}}\\brac{\\norm{\\nabla f(x)}+\\sqrt{\\alpha}\\norm{x-y}},\\label{g_Hal.d}\n\\end{align}\n\\end{subequations}\nwhere \\eqref{g_Hal.a} is comes from the construction of $H_\\alpha$, \\eqref{g_Hal.b} follows from the inequality $\\norm{a+b}^2\\ge \\frac{1}{4}\\norm{a}^2-\\frac{3}{4}\\norm{b}^2$ for $a,b\\in\\R^n$, \\eqref{g_Hal.c} follows from $\\alpha\\ge \\varepsilon$, and \\eqref{g_Hal.d} follows from $a^2+b^2\\ge \\frac{1}{2}(a+b)^2$ for $a,b\\in\\R$ and $\\varepsilon\\le 1/4$.  Now let $S:=\\max \\set{\\alpha\\norm{x-y}^2,f(x)-f(\\bar x)}$ and deduce from $f(x)\\ge f(\\bar x)$ as $x\\in U$ that\n\\begin{align*}\nH_\\alpha(z)-H_\\alpha(\\bar z)=f(x)+\\alpha\\norm{x-y}^2-f(\\bar x) \\in [S,2S].\n\\end{align*}\nCombining the latter with $0< H_\\alpha(z)-H_\\alpha(\\bar z)<\\eta$ by \\eqref{halpha} gives us $S\\in (0,\\eta).$ Using the nonincreasing property of $\\varphi'$ and the relation $\\varphi'(t)=\\sqrt{\\frac{2}{\\varepsilon}}\\phi'\\brac{\\frac{t}{2}}$ from \\eqref{deri varphi} yields\n\\begin{align*}\n\\varphi'(H_\\alpha(z)-H_\\alpha(\\bar z))&\\ge \\varphi'\\brac{2S}=\\sqrt{\\frac{2}{\\varepsilon}}\\phi'(S).\n\\end{align*}\nFurthermore, it follows from  \\eqref{deri varphi} and  \\eqref{g_Hal.d} that\n\\begin{align*}\n\\varphi'(H_\\alpha(z)-H_\\alpha(\\bar z))\\norm{\\nabla H_\\alpha(x,y)}&\\ge \\sqrt{\\frac{2}{\\varepsilon}}\\phi'(S)\\sqrt{\\frac{\\varepsilon}{2}}\\brac{\\norm{\\nabla f(x)}+\\sqrt{\\alpha}\\norm{x-y}}\\\\\n&=\\phi'(S)\\norm{\\nabla f(x)}+\\phi'(S)\\sqrt{\\alpha}\\norm{x-y}.\n\\end{align*}\nBy $\\phi'\\big(f(x)-f(\\bar{x})\\big)\\norm{\\nabla f(x)}\\ge 1$ coming from \\eqref{phi'>1} and $\\phi'(\\alpha \\norm{x-y}^2)\\ge \\frac{1}{\\sqrt{\\alpha}\\norm{x-y}}$ due to $\\phi'(t)\\ge \\frac{1}{\\sqrt{t}}$ for $t\\in (0,\\eta),$ we get the estimate\n\\begin{align*}\n\\phi'(S)\\norm{\\nabla f(x)}+\\phi'(S)\\sqrt{\\alpha}\\norm{x-y}\\ge 1,\n\\end{align*}\nwhich therefore verifies assertion (i) of the proposition.\n\nTo prove assertion (ii)  is easier. Indeed, let $\\psi(t)=Mt^{1-q}$ for $q\\in[0,1)$, let $\\theta:=\\max\\set{\\frac{1}{2},q}\\in (0,1)$, and take $\\tilde M$ from \\eqref{tilde M}. It readily follows that\n\\begin{align}\\label{formu phi}\n\\max\\set{\\psi'(s),\\frac{1}{\\sqrt{s}}}=\\tilde Ms^{-\\theta}\\text{ for all }s\\in(0,\\delta)\n\\end{align}\nfor some $\\delta\\in(0,\\eta)$ sufficiently small.\nCombining the latter with \\eqref{construction phi t}, we arrive at \n\\begin{align*}\n \\varphi(t)=\\sqrt{\\frac{8}{\\varepsilon}}\\int_0^{t/2}\\tilde{M}s^{-\\theta}ds=\\frac{2^{\\theta+1/2}\\tilde M}{\\sqrt{\\varepsilon}(1-\\theta)}t^{1-\\theta} \\text{ for all }t\\in (0,\\delta),\n\\end{align*}\nwhich thus verifies (ii) and completes the proof of the proposition.\n\\end{proof}\n\n\\begin{Proposition}\\label{convergence rate deduce}\nLet $f:\\Omega\\rightarrow{\\R}$ be a function defined on open and convex set of $\\Omega\\subset\\R^n$, and let $\\bar x \\in\\Omega$. Assume that $f$ has a Lipschitz continuous gradient on $\\Omega$ and take a sequence $\\set{x^k}\\subset\\Omega $ converging to $\\bar x$. Suppose in addition that there exists $\\alpha>0$ such that \n\\begin{align}\\label{main condition rate}\n\\alpha\\norm{\\nabla f(x^k)}^2\\le f(x^k)-f(x^{k+1})\\;\\text{ for all large }\\;k\\in\\N.\n\\end{align}\nThen the linear convergence of $x^k\\rightarrow\\bar x$ ensures the linear convergences of $f(x^k)$ to $f(\\bar x)$ and $\\nabla f(x^k)$ to $0$ as $k\\to\\infty$. Moreover, the convergence rate $\\norm{x^k-\\bar x}=\\mathcal{O}(m(k))$ with $m(k)\\downarrow 0$ yields\n$$\n\\abs{f(x^k)-f(\\bar x)}=\\mathcal{O}(m^2(k))\\;\\mbox{ and }\\;\\norm{\\nabla f(x^k)}=\\mathcal{O}(m(k))\\;\\mbox{ as }\\;\nk\\to\\infty. \n$$\n\\end{Proposition}\n\n\\begin{Proposition}\\label{prop AIGD tech}\nLet $\\set{x^k}$ be a sequence generated by Algorithm~{\\rm\\ref{IGDm}} with \n\\begin{align}\\label{max Lt<1-nu}\n\\max\\set{L\\tau,2L\\tau \\bar \\delta+(L\\tau+1)\\bar \\beta^2}< 1-\\nu\n\\end{align}\nsuch that $x^k,x^k_{\\rm in},x^k_{\\rm ex}\\in X$ for all $k=1,\\ldots,K\\in\\N$. Consider the positive constants\n\\begin{align}\\label{C1C2}\n\\alpha:&=\\frac{(L \\tau+1)\\bar \\beta^2+1-\\nu }{4\\tau},\\nonumber\\\\\nC_1:&=\\frac{1-\\nu-(L \\tau+1)\\bar \\beta^2-2L\\tau \\bar \\delta}{4\\tau},\\\\\nC_2:&=\\sqrt{2}\\max\\set{\\frac{\\nu+1}{\\tau },\\frac{\\bar\\beta(L\\tau+\\nu+1)+L\\tau\\bar\\delta}{\\tau }}+4{\\alpha}.\\nonumber\n\\end{align}\nThen we have the following estimates whenever $k=1,\\ldots,K-1:$\n\\begin{align}\nC_1\\norm{z^{k+1}-z^k}^2&\\le H_{\\alpha}(z^k)-H_{\\alpha}(z^{k+1}),\\label{3.2(i)}\\\\\n\\norm{\\nabla H_{\\alpha}(z^{k})}&\\le C_2\\norm{z^{k+1}-z^k}\\label{3.2(ii)}.\n\\end{align}\n\\end{Proposition}\n\n\\begin{proof}\nBy Lemma~\\ref{lemma descent}, the Lipschitz continuity with constant $L>0$ of $\\nabla f$ on $X $ yields\n\\begin{align*}\n-\\frac{L}{2}\\norm{y-x}^2\\le f(y)-f(x)-\\dotproduct{\\nabla f(x),y-x}\\le \\frac{L}{2}\\norm{y-x}^2 \\text{ for all }x,y\\in X .\n\\end{align*}\nFor any $k=1,\\ldots,K-1$, we use  the left-hand side inequality above with $y=x^k,x=x^k_{\\rm in}$ and the right-hand side inequality with $y=x^{k+1},x=x^k_{\\rm in}$ to get\n\\begin{align*}\n f(x^k)-f(x^k_{\\rm in})-\\dotproduct{\\nabla f(x^k_{\\rm in}),x^k-x^k_{\\rm in}}&\\ge -\\frac{L }{2}\\norm{x^k-x^k_{\\rm in}}^2,\n \\\\\n f(x^{k+1})- f(x^k_{\\rm in}) -\\dotproduct{\\nabla f(x^k_{\\rm in}),x^{k+1}-x^k_{\\rm in}}&\\le \\frac{L}{2}\\norm{x^{k+1}-x^k_{\\rm in}}^2.\n\\end{align*}\nSubtracting the first inequality from the second one gives us \n\\begin{align*}\nf(x^{k+1})-f(x^k)-\\dotproduct{\\nabla f(x^k_{\\rm in}),x^{k+1}-x^k}\\le \\frac{L}{2}\\norm{x^{k+1}-x^k_{\\rm in}}^2+\\frac{L }{2}\\norm{x^k-x^k_{\\rm in}}^2,\n\\end{align*}\nwhich can be rewritten in the form\n\\begin{align}\\label{prop: 1 chain}\n\\begin{array}{ll}\n\\disp f(x^{k+1})-f(x^k)\\le \\frac{L}{2}\\norm{x^{k+1}-x^k_{\\rm in}}^2+\\frac{L }{2}\\norm{x^k-x^k_{\\rm in}}^2+\\dotproduct{\\nabla f(x^k_{\\rm in}),x^{k+1}-x^k}\\\\\n=\\disp\\frac{L}{2} \\norm{x^{k+1}-x^k_{\\rm in}}^2+\\frac{L \\beta_k^2}{2}\\norm{x^k-x^{k-1}}^2+\\dotproduct{\\nabla f(x^k_{\\rm in})-\\nabla f(x^k_{\\rm ex}),x^{k+1}-x^k}\\\\\n\\disp+\\dotproduct{\\nabla f(x^k_{\\rm ex})- g^k,x^{k+1}-x^k}+\\dotproduct{g^k,x^{k+1}-x^k}.\n\\end{array}\n\\end{align}\nBy the Cauchy-Schwarz inequality, the Lipschitz continuity of $\\nabla f$ on $X $, and the constructions of $x^k_{\\rm in}$ and $x^k_{\\rm ex}$ in \\eqref{step xin} and \\eqref{step xex}, we deduce the estimates\n\\begin{align}\\label{prop: 1.1 chain}\n\\begin{array}{ll}\n\\dotproduct{\\nabla f(x^k_{\\rm in})-\\nabla f(x^k_{\\rm ex}),x^{k+1}-x^k}\\le \\norm{\\nabla f(x^k_{\\rm in})-\\nabla f(x^k_{\\rm ex})}\\cdot\\norm{x^{k+1}-x^k}\\\\\n\\le L\\norm{x^k_{\\rm in}-x^k_{\\rm ex}}\\cdot\\norm{x^{k+1}-x^k}\\le L\\abs{\\beta_k-\\gamma_k}\\norm{x^k-x^{k-1}}\\cdot\\norm{x^{k+1}-x^k}\\\\\n\\le\\disp\\frac{L\\bar \\delta}{2}\\brac{\\norm{x^k-x^{k-1}}^2+\\norm{x^{k+1}-x^k}^2}.\n\\end{array}\n\\end{align}\nIt follows from the Cauchy-Schwarz and AM-GM inequalities together with the algorithm design $x^{k+1}:=x^k_{\\rm in}-\\tau  g^k$ and $\\norm{g^k-\\nabla f(x^k_{\\rm ex})}\\le \\nu\\norm{g^k}$ in \\eqref{iteration update} that\n\\begin{align}\\label{prop: 3 chain}\n\\begin{array}{ll}\n\\dotproduct{\\nabla f(x^k_{\\rm ex})- g^k,x^{k+1}-x^k}\\le \\norm{\\nabla f(x^k_{\\rm ex})- g^k}\\cdot\\norm{x^{k+1}-x^k}\\\\\n\\disp\\le \\nu \\norm{g^k}\\cdot\\norm{x^{k+1}-x^k}=\\frac{\\nu}{\\tau }\\norm{x^k_{\\rm in}-x^{k+1}}\\cdot\\norm{x^{k+1}-x^k}\\\\\n\\le\\disp \\frac{\\nu}{2\\tau }\\brac{\\norm{x^{k+1}-x^k_{\\rm in}}^2+\\norm{x^{k+1}-x^k}^2}.\n\\end{array}\n\\end{align}\nUsing further \\eqref{step xin} and \\eqref{iteration update},  we arrive at\n\\begin{align}\\label{prop: 2 chain}\n\\begin{array}{ll}\n\\dotproduct{g^k,x^{k+1}-x^k}=\\disp\\frac{1}{\\tau }\\dotproduct{x^k_{\\rm in}-x^{k+1},x^{k+1}-x^k}\\\\\n=\\disp\\frac{1}{2\\tau }\\brac{\\norm{x^k_{\\rm in}-x^k}^2-\\norm{x^{k+1}-x^k_{\\rm in}}^2-\\norm{x^k-x^{k+1}}^2}\\\\\n=\\disp\\frac{1}{2\\tau }\\brac{\\beta_k^2\\norm{x^{k}-x^{k-1}}^2-\\norm{x^{k+1}-x^k_{\\rm in}}^2-\\norm{x^k-x^{k+1}}^2}.\n\\end{array}\n\\end{align}\nCombining \\eqref{prop: 1 chain}--\\eqref{prop: 2 chain} with taking into account $L\\tau +\\nu<1$ and $\\beta_k\\le \\bar \\beta$ gives us\n\\begin{align}\\label{estimane conse}\n\\begin{array}{ll}\nf(x^{k+1})-f(x^k)&\\le\\disp\\frac{L\\tau +\\nu-1}{2\\tau }\\norm{x^{k+1}-x^k_{\\rm in}}^2+\\frac{L\\tau\\bar \\delta+\\nu-1}{2\\tau }\\norm{x^{k+1}-x^k}^2\\\\\n&+\\disp\\frac{L \\tau \\beta_k^2+\\beta_k^2+L\\tau\\bar \\delta}{2\\tau }\\norm{x^k-x^{k-1}}^2\\\\\n&\\le\\disp\\frac{L\\tau\\bar \\delta+\\nu-1}{2\\tau }\\norm{x^{k+1}-x^k}^2+\\frac{(L \\tau +1)\\bar \\beta^2+L\\tau\\bar \\delta}{2\\tau }\\norm{x^k-x^{k-1}}^2.\n\\end{array}\n\\end{align}\nSince $\\bar\\beta^2<\\frac{1-\\nu}{L \\tau+1}$, it follows that $\\frac{(L \\tau+1)\\bar \\beta^2+L\\tau\\bar \\delta}{2\\tau }<\\frac{1-\\nu-L\\tau\\bar \\delta}{2\\tau }$, and by $\\alpha=\\frac{(L \\tau+1)\\bar \\beta^2+1-\\nu}{4\\tau}$ we get \n\\begin{align*}\nC_1=\\frac{1-\\nu-(L \\tau+1)\\bar \\beta^2-2L\\tau \\bar \\delta}{4\\tau}=\\frac{1-\\nu-L\\tau\\bar\\delta}{2\\tau}-\\alpha=\\alpha-\\frac{(L  \\tau+1)\\bar\\beta^2+L\\tau\\bar \\delta}{2\\tau}\n\\end{align*}\nfor $C_1$ in \\eqref{C1C2}. Using the latter together with \\eqref{estimane conse} and the construction of $H_\\alpha$ in \\eqref{lyapunov func}  brings us to\n\\begin{align*}\nH_{\\alpha}(z^{k+1})-H_{\\alpha}(z^k)&= f(x^{k+1})+\\alpha\\norm{x^{k+1}-x^k}^2-f(x^{k})-\\alpha\\norm{x^{k}-x^{k-1}}^2\\\\\n&\\le\\brac{ \\frac{\\nu-1+L\\tau\\bar \\delta}{2\\tau }+\\alpha}\\norm{x^{k+1}-x^k}^2+\\brac{\\frac{(L \\tau+1)\\bar  \\beta^2+L\\tau\\bar \\delta}{2\\tau }-\\alpha}\\norm{x^k-x^{k-1}}^2\\\\\n&\\le- C_1\\brac{\\norm{x^{k+1}-x^k}^2+\\norm{x^k-x^{k-1}}^2}=-C_1\\norm{z^{k+1}-z^k}^2,\n\\end{align*}\nwhich therefore justifies the inequality in \\eqref{3.2(i)}. \n\nTo verify \\eqref{3.2(ii)}, take any $k=1,\\ldots,K-1$ and obtain the relationships\n\\begin{subequations}\n\\begin{align}\n\\norm{\\nabla f(x^k)}&\\le \\norm{\\nabla f(x^k)-\\nabla f(x^k_{\\rm in})} +\\norm{\\nabla f(x^k_{\\rm in})-\\nabla f(x^k_{\\rm ex})}+\\norm{\\nabla f(x^k_{\\rm ex})-g^k}+\\norm{g^k}\\label{nab.1 triang}\\\\\n&\\le(L\\bar \\beta+L\\bar\\delta)\\norm{x^k-x^{k-1}}+(\\nu+1)\\norm{g^k}\\label{nab.2}\\\\\n&=(L\\bar \\beta+L\\bar\\delta)\\norm{x^k-x^{k-1}}+\\frac{\\nu+1}{\\tau }\\norm{x^k+\\beta_k(x^k-x^{k-1})-x^{k+1}}\\label{nab.3}\\\\\n&\\le \\brac{L\\bar \\beta+L\\bar\\delta+\\frac{\\nu+1}{\\tau }\\bar \\beta}\\norm{x^k-x^{k-1}}+\\frac{\\nu+1}{\\tau }\\norm{x^{k+1}-x^k}\\nonumber\\\\\n&\\le \\max\\set{\\frac{\\nu+1}{\\tau },L\\bar \\beta+L\\bar\\delta+\\frac{\\nu+1}{\\tau }\\bar \\beta}\\brac{\\norm{x^{k+1}-x^k}+\\norm{x^k-x^{k-1}}}\\nonumber\\\\\n&\\le \\sqrt{2}\\max\\set{\\frac{\\nu+1}{\\tau },L\\bar \\beta+L\\bar\\delta+\\frac{\\nu+1}{\\tau }\\bar \\beta}\\norm{z^{k+1}-z^k}\\label{nab.6},\n\\end{align}\n\\end{subequations}\nwhere \\eqref{nab.1 triang} follows from the triangle inequality; \\eqref{nab.2} follows from $\\norm{g^k-\\nabla f(x^k_{\\rm ex})}\\le \\nu\\norm{g^k}$ in \\eqref{iteration update}, the updates in \\eqref{step xin}, \\eqref{step xex}, and the Lipschitz continuity of $\\nabla f$ on $X $ containing $ x^k_\\rin,x^k_\\rex$; \\eqref{nab.3} follows from \\eqref{step xin} and \\eqref{step xex}; \\eqref{nab.6} follows from $a+b\\le \\sqrt{2(a^2+b^2)}$ and $z^k=(x^k,x^{k-1})$. By using the estimates above, we deduce from the construction of $H_\\alpha$ and inequality $\\sqrt{a^2+b^2}\\le a+b$ for $a,b\\ge 0$ that\n\\begin{align*}\n\\norm{\\nabla H_{\\alpha}(z^k)}&= \\sqrt{\\norm{\\nabla f(x^k)+2{\\alpha}(x^k-x^{k-1})}^2+\\norm{2{\\alpha} (x^k-x^{k-1})}^2}\\\\\n&\\le \\norm{\\nabla f(x^k)+2{\\alpha}(x^k-x^{k-1})}+\\norm{2{\\alpha} (x^k-x^{k-1})}\\\\\n& \\le \\norm{\\nabla f(x^k)}+4{\\alpha}\\norm{x^k-x^{k-1}}\\\\\n&\\le \\brac{\\sqrt{2}\\max\\set{\\frac{\\nu+1}{\\tau },L\\bar \\beta+L\\bar\\delta+\\frac{\\nu+1}{\\tau }\\bar \\beta}+4{\\alpha}}\\norm{z^{k+1}-z^k},\n\\end{align*}\nwhich justifies \\eqref{3.2(ii)}  and thus completes the proof of the proposition.\n\\end{proof}\n\n\\begin{Proposition}\\label{proposition distance}\nLet $\\set{x^k}$ be the sequence of iterates generated by Algorithm~{\\rm\\ref{IGDm}} such that $x^k,x^k_{\\rm ex}\\in \\B(\\bar x,\\rho)\\subset X$ for all $k=1,\\ldots,K$. Then we have for all $k=1,\\ldots,K$ that\n\\begin{align}\n\\norm{x^{k+1}-x^k}&\\le   \\frac{L\\tau\\rho }{(1-\\bar \\beta)(1-\\nu)},\\label{3.3(i)}\\\\\n\\norm{z^{k+1}-z^k}&\\le  \\frac{\\sqrt{2} L\\tau\\rho }{(1-\\bar \\beta)(1-\\nu)}.\\label{3.3(ii)}\n\\end{align}\n\\end{Proposition}\n\n\\begin{proof}\nPick any $k=1,\\ldots,K$ and observe that the iteration update in Algorithm~\\ref{IGDm} can be rewritten as $x^{i+1}=x^i+\\beta_k(x^i-x^{i-1})-\\tau g^i$, which yields\n\\begin{align}\\label{new update}\n\\norm{x^{i+1}-x^i}\\le \\bar \\beta \\norm{x^i-x^{i-1}}+\\tau\\norm{g^i}\\;\\text{ for all }\\;i\\le k.\n\\end{align}\nUsing the triangle inequality together with $\\norm{g^i-\\nabla f(x^i_{\\rm ex})}\\le \\nu \\norm{g^k}$ gives us\n\\begin{align*}\n\\norm{\\nabla f(x^i_{\\rm ex})}\\ge \\norm{g^i}-\\norm{\\nabla f(x^i_{\\rm ex})-g^i}\\ge (1-\\nu)\\norm{g^i}\\;\\text{ for all }\\;i\\le k.\n\\end{align*} \nProceeding inductively from \\eqref{new update} for $i=k,k-1,\\ldots,1$ with the usage of the Lipschitz continuity of $\\nabla f$ on the ball $\\B(\\bar x,\\rho)$ containing $\\bar x$ and $x^{k-i}_\\rex,i=0,\\ldots,k-1$ yields\n\\begin{align*}\n\\norm{x^{k+1}-x^k}&\\le \\bar \\beta\\norm{x^k-x^{k-1}}+\\tau \\norm{g^k}\\\\\n&=\\bar \\beta^2\\norm{x^{k-1}-x^{k-2}}+\\tau\\brac{\\bar \\beta \\norm{g^{k-1}}+\\norm{g^k}}\\\\\n&\\ldots\\\\\n&\\le\\bar \\beta^{k}\\norm{x^1-x^0}+\\tau \\sum_{i=0}^{k-1}\\bar \\beta^i\\norm{g^{k-i}}\\\\\n&\\le \\frac{\\tau}{1-\\nu} \\sum_{i=0}^{k-1}\\bar \\beta^i\\norm{\\nabla f(x_\\rex^{k-i})} = \\frac{\\tau}{1-\\nu}\\sum_{i=0}^{k-1}\\bar \\beta^i\\norm{\\nabla f(x^{k-i}_\\rex)-\\nabla f(\\bar x)}\\\\\n&\\le \\frac{\\tau}{1-\\nu}\\sum_{i=0}^{k-1}\\bar \\beta^i L\\norm{x^{k-i}_\\rex -\\bar x}\\le \\frac{\\tau}{1-\\nu}\\sum_{i=0}^{k-1}\\bar \\beta^i L\\rho\\\\\n&\\le \\frac{L\\tau\\rho}{1-\\nu}\\sum_{i=0}^{\\infty}\\bar \\beta^i = \\frac{L\\tau\\rho}{(1-\\nu)(1-\\bar \\beta)},\n\\end{align*}\nwhere the latter inequality follows from $\\bar \\beta\\in [0,1)$. This readily verifies \\eqref{3.3(i)}. Combining \\eqref{3.3(i)}  with $\\norm{z^{k+1}-z^k}=\\sqrt{\\norm{x^{k+1}-x^k}^2+\\norm{x^{k}-x^{k-1}}^2}$ brings us to \\eqref{3.3(ii)} and thus completes the proof.\n\\end{proof}\n\n\\begin{Proposition}\\label{convergence xk+1 xk} Let $\\set{x^k}$ be the sequence of iterates generated by Algorithm~{\\rm\\ref{IGDm}} that converges to some $\\tilde x\\in X$. Then $\\tilde x$ is a stationary point of \\eqref{main problem} and the sequences $\\set{x^k_\\rex},\\set{x^k_\\rin}$ converge to $\\tilde x$ as well.\n\\end{Proposition}\n\n\\begin{proof} Since the sequence $\\set{x^k}$ is convergent,  we have $\\norm{x^{k+1}-x^k}\\rightarrow 0$ as $k\\rightarrow\\infty.$ Combining this with the updates in \\eqref{step xin} and \\eqref{step xex} together with the boundedness of $\\set{\\beta_k} and \\set{\\gamma_k}$ tells us that\n\\begin{align*}\n{x^k_{\\rm in}}&=x^k+\\beta_k({x^{k}-x^{k-1}})\\rightarrow\\tilde x,\\\\\n{x^k_{\\rm ex}}&=x^k+\\gamma_k({x^{k}-x^{k-1}})\\rightarrow\\tilde x.\n\\end{align*}\nUsing $\\norm{x^{k+1}-x^k}\\rightarrow 0$ and the updates in \\eqref{step xin}, \\eqref{iteration update} of Algorithm~\\ref{IGDm} yields\n\\begin{align*}\ng^k=\\frac{1}{\\tau}\\brac{x^k-x^{k+1}+\\beta_k(x^k-x^{k-1})}\\rightarrow 0\\;\\text{ as }\\;k\\rightarrow\\infty.\n\\end{align*}\nIt follows from $\\norm{\\nabla f(x^k_{\\rm ex})-g^k}\\le \\nu\\norm{g^k}$ in \\eqref{iteration update} that ${\\nabla f(x^k_{\\rm ex})}\\rightarrow 0$, and we are done with the proof.\n\\end{proof}\n\n\\begin{proof} To verify (i), deduce from $\\bar \\beta \\in[0,\\sqrt{1-\\nu})$ that there exists $T>0$ such that\n\\begin{align}\\label{LT<1}\nT< \\frac{(1-\\bar \\beta)(1-\\nu)}{4L\\sqrt{2}(1+\\bar \\delta)},\\quad(LT+1)\\bar\\beta^2+2LT\\bar \\delta<1-\\nu,\\quad LT\\le 1-\\nu.\n\\end{align}\nDefine further $\\varepsilon:=\\min\\set{\\frac{1-\\nu}{4T},\\frac{1}{4}}\\in \\big(0,\\frac{1}{4}\\big]$ and deduce from Lemma~\\ref{isolation lemma} due to the imposed assumptions that there exists a neighborhood $U\\subset\\B(\\bar x,r)$ of $\\bar x$ such that $\\bar x$ minimizes $f$ on $U$ and that $f(\\bar x)$ is the only critical value within this set. Remembering that $f$ satisfies the basic PLK condition at $\\bar x$ tells us by Proposition~\\ref{isolated} that the Lyapunov function $\\set{H_\\alpha}_{\\alpha\\ge \\varepsilon}$ enjoys the same PLK property at $\\bar z:=(\\bar x,\\bar x)$ for some $\\eta\\in (0,\\infty]$, some neighborhood $Z$ of $\\bar z:=(\\bar x,\\bar x)$, and a concave desingularizing function $\\varphi:[0,\\eta)\\rightarrow[0,\\infty)$ with $\\varphi(0)=0$ which is ${\\cal C}^1$-smooth on $(0,\\eta)$. This means that for any $\\alpha\\ge \\varepsilon$ and any $z\\in Z$ with $H_\\alpha(\\bar z)< H_\\alpha( z)<H_\\alpha(\\bar z)+\\eta$, we get the inequality\n\\begin{align}\\label{KL R2n in proof}\n\\varphi'\\big(H_\\alpha(z )-H_\\alpha(\\bar z )\\big)\\norm{\\nabla H_\\alpha(z)}\\ge 1.\n\\end{align}\nThe continuity of $f$ allows us to find $\\rho\\in(0,r)$ such that \n\\begin{align}\\label{choice rho}\nf(x)+\\frac{L^2T\\rho^2}{2(1-\\bar \\beta)^2(1-\\nu)^2}< f(\\bar x)+\\eta\\text{ for all }x \\in \\mathbb{B}(\\bar x,\\rho).\n\\end{align}\nInvoking further the continuity of $\\varphi$ at $0$ with $\\varphi(0)=0$ gives us $\\xi\\in (0,\\rho]$ for which\n\\begin{align}\\label{defi xi 1}\n\\sqrt{2}\\norm{x-\\bar x}+\\frac{4\\sqrt{2}\\max\\set{\\nu+1,\\bar\\beta(\\nu+2)+\\bar\\delta}+8}{1-\\nu-(L  T+1)\\bar\\beta^2-2LT \\bar \\delta}\\varphi(f(x)-f(\\bar x))&<\\frac{\\rho}{4}\\;\\text{ for all }\\;x\\in\\mathbb{B}(\\bar x,\\xi).\n\\end{align}\nGiven the sequence of iterates $\\set{x^k}$ generated by Algorithm~\\ref{IGDm} with $x^0\\in \\mathbb{B}(\\bar x, \\xi)$ and $\\tau\\in (0, T]$, take the constant $\\alpha,C_1,C_2$ from \\eqref{C1C2} and deduce that\n\\begin{align}\\label{sele alpha}\n\\alpha=\\frac{(L\\tau+1)\\bar\\beta^2+1-\\nu}{4\\tau}\\ge \\frac{1-\\nu}{4T}\\ge \\varepsilon.\n\\end{align}\nSince $\\tau\\le T$, we get from the selection of $T$ in \\eqref{LT<1} the inequalities\n\\begin{align}\\label{ineq 1-nu}\n(L\\tau+1)\\bar \\beta^2+2L\\tau \\bar \\delta\\le (LT+1)\\bar \\beta^2+2LT \\bar \\delta <1-\\nu.\n\\end{align}\nwhich bring us to the relationships  \n\\begin{align}\\label{ineq 1/2}\n\\alpha=\\frac{(L\\tau+1)\\bar\\beta^2+1-\\nu}{4\\tau}\\le \\frac{2(1-\\nu)-2L\\tau \\bar \\delta}{4\\tau}=\\frac{1-\\nu-L\\tau \\bar \\delta}{2\\tau}\\le \\frac{1}{2\\tau}.\n\\end{align} \nIt follows from $L\\tau \\le LT\\le 1$ by \\eqref{LT<1} and the estimates in \\eqref{ineq 1/2}, \\eqref{ineq 1-nu} that\n\\begin{align*}\n\\frac{C_2}{C_1}&= \\frac{\\sqrt{2}\\max\\set{\\frac{\\nu+1}{\\tau },\\frac{\\bar\\beta(L\\tau+\\nu+1)+L\\tau \\bar \\delta}{\\tau }}+4\\alpha}{\\frac{1-\\nu-(L \\tau+1)\\bar \\beta^2-2L\\tau \\bar \\delta}{4\\tau}}\\\\\n&= \\frac{4\\sqrt{2}\\max\\set{{\\nu+1}, \\bar\\beta(L\\tau+\\nu+1)+L\\tau \\bar \\delta}+16\\alpha\\tau}{1-\\nu-(L \\tau+1)\\bar\\beta^2-2L\\tau \\bar \\delta}\\\\\n&\\le \\frac{4\\sqrt{2}\\max\\set{\\nu+1,\\bar\\beta(\\nu+2)+\\bar\\delta}+8}{1-\\nu-(L  T+1)\\bar\\beta^2-2LT \\bar \\delta}.\n\\end{align*}\nThis implies by \\eqref{defi xi 1} that \n\\begin{align}\\label{defi xi 2}\n\\sqrt{2}\\norm{x-\\bar x}+\\frac{C_2}{C_1}\\varphi(f(x)-f(\\bar x))&<\\frac{\\rho}{4} \\text{ for all }x\\in\\mathbb{B}(\\bar x,\\xi).\n\\end{align}\nThe rest of the proof of assertion (i) is split into two claims.\n\n\\begin{Claim} The sequences $\\set{x^k},\\set{x^k_{\\rm in}}$, and $\\set{x^k_{\\rm ex}}$ stay inside $\\mathbb{B}(\\bar x,\\rho)$ for all $k\\in\\N$.\n\\end{Claim}\n\\noindent Since $x^0=x^1\\in \\mathbb{B}(\\bar x,\\xi),$ the updates in \\eqref{step xin}, \\eqref{step xex} of Algorithm~\\ref{IGDm} tell us that $x^1_{\\rm in}=x^1_{\\rm ex}=x^1\\in \\mathbb{B}(\\bar x,\\xi)\\subset \\mathbb{B}(\\bar x,\\rho).$ We proceed now by induction to show that $x^k,x^k_{\\rm in},x^k_{\\rm ex}\\in \\mathbb{B}(\\bar x,\\rho)$ for all $k\\in\\N.$ Assuming that $x^k,x^k_\\rin,x^k_{\\rm ex}\\in \\mathbb{B}(\\bar x,\\rho)$ for $k=1,\\ldots,K$, let us verify that $x^{K+1},x_\\rin^{K+1},x^{K+1}_\\rex\\in \\mathbb{B}(\\bar x,\\rho).$ It follows from \\eqref{ineq 1-nu} and $L\\tau\\le LT<1-\\nu$ by \\eqref{LT<1} that assumption \\eqref{max Lt<1-nu} in Proposition~\\ref{prop AIGD tech} is satisfied. Therefore, we get the estimates\n\\begin{align}\\label{descent con for rate 1}\n&C_1 \\norm{z^{k+1}-z^k}^2\\le H_{\\alpha}(z^k)-H_{\\alpha}(z^{k+1}),\\\\\n&\\norm{\\nabla H_{\\alpha}(z^{k})}\\le C_2\\norm{z^{k+1}-z^k}\\text{ for all }k=1,\\ldots,K-1,\\label{descent con for rate 2}\n\\end{align}\nwhere $z^k=(x^k,x^{k-1})$ as before. Fix any such $k,$ we deduce from the above inequalities that\n\\begin{align}\\label{primary descent condi}\n\\norm{z^{k+1}-z^k}\\cdot\\norm{\\nabla H_\\alpha(z^k)}\\le C_2  \\norm{z^{k+1}-z^k}^2\\le \\frac{C_2}{C_1}(H_\\alpha(z^k)-H_\\alpha(z^{k+1})).\n\\end{align}\nTo use inequality \\eqref{KL R2n in proof} with $z:=z^k$, it is sufficient to check that $z^k\\in Z$ and $ H_\\alpha(\\bar z)<H_\\alpha(z^k)<H_\\alpha(\\bar z)+\\eta$. Note first that the former condition is satisfied due to\n\\begin{align*}\nz^k=(x^k,x^{k-1})\\in \\mathbb{B}(\\bar x,\\rho)\\times \\mathbb{B}(\\bar x,\\rho)\\subset Z.\n\\end{align*}\n To verify the latter condition, deduce from Proposition~\\ref{proposition distance} that $\\norm{x^k-x^{k-1}}\\le \\frac{L\\tau\\rho}{(1-\\bar \\beta)(1-\\nu)}$. Combining this with the construction of $H_\\alpha$, the inequality $\\alpha\\le \\frac{1}{2\\tau}$ from \\eqref{ineq 1/2}, the algorithm inclusions $x^k,x^{k-1}\\in\\mathbb{B}(\\bar x,\\rho)$, and the choice of $\\rho$ in \\eqref{choice rho} tells us that\n\\begin{align*}\nH_\\alpha(z^k)&=f(x^k)+\\alpha\\norm{x^k-x^{k-1}}^2\\\\\n&\\le f(x^k)+\\frac{1}{2\\tau}\\cdot\\frac{L^2\\tau^2\\rho^2}{(1-\\bar \\beta)^2(1-\\nu)^2}\\\\\n&= f(x^k)+\\frac{L^2T\\rho^2}{2(1-\\bar \\beta)^2(1-\\nu)^2}< f(\\bar x)+\\eta=H_\\alpha(\\bar z)+\\eta.\n\\end{align*}\nObserve that the inequality $H_\\alpha(\\bar z)<H_\\alpha(z^k)$ is also satisfied. Indeed, its failure means that \n$$\nf(x^k)+\\alpha\\norm{x^{k}-x^{k-1}}^2\\le H_\\alpha(\\bar z)=f(\\bar x).\n$$\nBy taking into account that $f(x^k)\\ge f(\\bar x)$ by $x^k\\in\\mathbb{B}(\\bar x,\\rho)$, the latter implies that\n\\begin{align*}\nf(x^k)=f(\\bar x) \\text{ and }x^{k}=x^{k-1},\n\\end{align*}\nwhich  means that $\\nabla f(x^k)=0$ since $\\bar x$ is a local minimizer of $f$ within $\\B(\\bar x,\\rho)$. The equalities above also imply that\n$x^k_{\\rm ex}=x^k$ by \\eqref{step xex}, and so $\\nabla f(x^k_{\\rm ex})=0,$ which contradicts Assumption~\\ref{assu} and thus justifies \\eqref{KL R2n in proof} with $z=z^k$. Multiplying both sides of \\eqref{primary descent condi} by $\\varphi'(H_\\alpha(z^k)-H_\\alpha(\\bar z))$ and using \\eqref{KL R2n in proof} give us\n\\begin{align}\\label{ineq alter}\n\\norm{z^{k+1}-z^k}& \\le \\varphi'(H_\\alpha(z^k)-H_\\alpha(\\bar z))\\norm{\\nabla H_\\alpha(z^k)}\\cdot\\norm{z^{k+1}-z^k}\\\\\n&\\le \\frac{C_2}{C_1}\\varphi'(H_\\alpha(z^k)-H_\\alpha(\\bar z))\\brac{ H_{\\alpha}(z^k)-H_{\\alpha}(z^{k+1})}\\nonumber\\\\\n&\\le\\frac{C_2}{C_1}\\brac{\\varphi(H_\\alpha(z^k)-H_\\alpha(\\bar z))-\\varphi(H_\\alpha(z^{k+1})-H_\\alpha(\\bar z))} \\text{ for all }k=1,\\ldots,K-1,\n\\end{align}\nwhere the inequality  in \\eqref{chain ineq} follows from the concavity of $\\varphi.$\nCombining further the obtained estimate with $\\tau\\le T<\\frac{(1-\\bar \\beta)(1-\\nu)}{4L\\sqrt{2}}$ from \\eqref{LT<1} and estimate \\eqref{3.3(ii)} from Proposition~\\ref{proposition distance},  we have\n\\begin{align}\n\\norm{z^{K+1}-\\bar z}&\\le\\norm{z^1-\\bar z}+\\norm{z^{K+1}-z^K}+\\sum_{k=1}^{K-1}\\norm{z^{k+1}-z^k}\\nonumber\\\\\n&\\le \\norm{z^1-\\bar z}+\\norm{z^{K+1}-z^K}+\\frac{C_2}{C_1}\\sum_{k=1}^{K-1}\\brac{\\varphi(H_\\alpha(z^k)-H_\\alpha(\\bar z))-\\varphi(H_\\alpha(z^{k+1})-H_\\alpha(\\bar z))}\\nonumber\\\\\n&=\\sqrt{2}\\norm{x^1-\\bar x}+\\frac{\\sqrt{2}L\\rho\\tau}{(1-\\bar \\beta)(1-\\nu)}+\\frac{C_2}{C_1}\\brac{\\varphi(H_\\alpha(z^1)-H_\\alpha(\\bar z))-\\varphi(H_\\alpha(z^{K})-H_\\alpha(\\bar z))}\\nonumber\\\\\n&\\le \\sqrt{2}\\norm{x^1-\\bar x}+\\frac{\\rho}{4}+\\frac{C_2}{C_1}\\varphi(f(x^1)-f(\\bar x))<\\rho/2,\\label{chain ineq}\n\\end{align}\nwhere the last inequality is a consequence of \\eqref{defi xi 2} and $x^1\\in\\B(\\bar x,\\xi)$. \nTherefore, \n\\begin{align*}\n\\norm{x^{K+1}-\\bar x}\\le \\norm{z^{K+1}-\\bar z}\\le \\rho/2,\n\\end{align*}\nwhich means that $x^{K+1}\\in\\mathbb{B}(\\bar x,\\rho/2)$. Using the updates in \\eqref{step xin} and \\eqref{step xex} at the $(K+1)^{\\rm th}$ iteration with taking into account \\eqref{3.3(ii)} from Proposition~\\ref{proposition distance}, $\\tau\\le T<\\frac{(1-\\bar \\beta)(1-\\nu)}{4L\\sqrt{2}(1+\\bar\\delta)}$ from \\eqref{LT<1}, and  \n\\begin{align*}\n0\\le \\beta_{K+1}<1,\\quad 0\\le \\delta_{K+1}\\le \\delta_{K+1}-\\beta_{K+1}+\\beta_{K+1}\\le \\bar \\delta +1,\n\\end{align*}\nwe arrive at the estimates\n\\begin{align*}\n\\norm{x^{K+1}_\\rin-\\bar x}&\\le \\norm{x^{K+1}-\\bar x}+\\norm{x^{K+1}-x^K}\\le \\rho/2+ \\frac{2L\\tau\\rho}{(1-\\bar \\beta)(1-\\nu)}<\\rho,\\\\\n\\norm{x^{K+1}_\\rex-\\bar x}&\\le \\norm{x^{K+1}-\\bar x}+(1+\\bar \\delta)\\norm{x^{K+1}-x^K}\\le \\rho/2+ \\frac{2L\\tau\\rho(1+\\bar\\delta)}{(1-\\bar \\beta)(1-\\nu)}<\\rho.\n\\end{align*}\nThus $x^{K+1}_\\rin,x^{K+1}_\\rex\\in\\mathbb{B}(\\bar x,\\rho)$, which completes the claimed assertion by induction. \n\\medskip\n\n\\noindent\\textbf{Claim~2.} \\textit{The sequences $\\set{x^k},\\set{x^k_{\\rm in}},\\set{x^k_{\\rm ex}}$ converge to a local minimizer $\\tilde x$ of $f$ with $f(\\tilde x)=f(\\bar x)$.}\n\\medskip\n\n\\noindent Indeed, it follows from \\eqref{ineq alter} and $x^k,x^k_\\rin,x^k_{\\rm ex}\\in\\mathbb{B}(\\bar x,\\rho)$ from Claim 1 that \n\\begin{align*}\n\\norm{z^{k+1}-z^k}&\\le \\frac{C_2}{C_1}\\brac{\\varphi(H_\\alpha(z^k)-H_\\alpha(\\bar z))-\\varphi(H_\\alpha(z^{k+1})-H_\\alpha(\\bar z))} \\text{ for all }k\\in\\N.\n\\end{align*}\nThis readily implies that \n\\begin{align*}\n\\sum_{k=1}^\\infty \\norm{z^{k+1}-z^k}\\le \\frac{C_2}{C_1}\\varphi(H_\\alpha(z^1)-H_\\alpha(\\bar z))<\\infty.\n\\end{align*}\nTherefore, $\\set{z^k}$ is convergent, and thus $\\set{x^k}$ converges to some $\\tilde x\\in\\overline{\\mathbb{B}}(\\bar x,\\rho)\\subset U$. Using  Proposition~\\ref{convergence xk+1 xk}, we justify the convergence of $\\set{x^k_\\rex},\\set{x^k_\\rin}$ to $\\tilde x$ and that $\\tilde x$ is a stationary point of $f.$ Since $f(\\bar x)$ is the only critical value of $f$ within $U,$ it gives us $f(\\tilde x)=f(\\bar x)$ and shows that $\\tilde x$ is a local minimizer of $f$.\\vspace*{0.05in} \n\nTo verify (ii), we employ Proposition~\\ref{general rate}. Letting $\\tilde z:=(\\tilde x,\\tilde x)$ and using \\eqref{descent con for rate 1}, \\eqref{descent con for rate 2} from the proof of Claim~1 in (i) together with $\\set{x^k},\\set{x^k_{\\rm in}},\\set{x^k_{\\rm ex}}\\subset\\mathbb{B}(\\bar x,\\rho)$ give us the estimate\n\\begin{align}\\label{ineq rate grad}\n\\begin{array}{ll}\nC_1 \\norm{z^{k+1}-z^k}^2\\le H_{\\alpha}(z^k)-H_{\\alpha}(z^{k+1}),\\\\\n\\norm{\\nabla H_{\\alpha}(z^{k})}\\le C_2\\norm{z^{k+1}-z^k},\\quad k\\in\\N.\n\\end{array}\n\\end{align}\nObserve from $\\nabla f(x^k_\\rex)\\ne 0$ and $\\norm{g^k-\\nabla f(x^k_\\rex)}\\le \\nu \\norm{g^k}$ that $z^{k+1}\\ne z^k$ for all $k\\in\\N.$ Indeed, the contrary implies by $z^k=(x^{k},x^{k-1})$ that $x^{k+1}=x^k=x^{k-1}$ for some $k\\in\\N$. Using the updates in \\eqref{step xin}, \\eqref{step xex} of Algorithm~\\ref{IGDm} ensures that $x^k_{\\rm ex}=x^k_{\\rm in}=x^k=x^{k+1}$. Then $g^k=0$ by \\eqref{iteration update}, and hence $\\nabla f(x^k_{\\rm ex})=0$ by $\\norm{g^k-\\nabla f(x^k_\\rex)}\\le \\nu \\norm{g^k}$. This contradicts the assumption $\\nabla f(x^k_{\\rm ex})\\ne 0$ for all $k\\in\\N$.\n\nRecalling that $f$ satisfies the PLK condition at $\\bar x$ with the desingularizing function $\\psi(t)=Mt^{1-q}$, we deduce from Proposition~\\ref{isolated}(ii) that $H_\\alpha$ satisfies the PLK condition at $(\\bar x,\\bar x)$ with the neighborhood $Z$ taking from \\eqref{KL R2n in proof} and the desingularizing function $\\varphi(t)=\\overline Mt^{1-\\theta}$, where $\\theta=\\max \\set{q,\\frac{1}{2}}\\in (0,1),\\overline{M}>0.$ Taking into account Remark~\\ref{algebraic}, this PLK property also holds at $\\tilde z\\in\\overline{\\mathbb{B}}(\\bar x,\\rho)\\times \\overline{\\mathbb{B}}(\\bar x,\\rho)\\subset Z$ due to $H_\\alpha(\\tilde z)=f(\\tilde x)=f(\\bar x)=H_\\alpha(\\bar z).$  Then Proposition~\\ref{general rate} applied to the function $H_\\alpha$ and sequence $\\set{z^k}\\subset\\R^{n}\\times \\R^n$, with taking into account Remark~\\ref{rmk general rate}, brings us to the following:\n\\begin{itemize}\n\\item For $q\\in[0,1/2],$ we get $\\theta=1/2$, which ensures that $\\set{z^k}$ converges linearly to $(\\tilde x,\\tilde x)$. This implies that $\\set{x^k}$ converges linearly to $\\tilde x.$\n\n\\item For $q\\in (1/2,1),$ we get $\\theta=q\\in (1/2,1)$, which tells us that $\\norm{z^k-(\\tilde x,\\tilde x)}=\\mathcal{O}(k^{-\\frac{1-q}{2q-1}})$ and thus yields the convergence rate $\\norm{x^k-\\tilde x}=\\mathcal{O}(k^{-\\frac{1-q}{2q-1}})$ as $k\\rightarrow\\infty$.\n\\end{itemize}\n\nFinally, we employ Proposition~\\ref{convergence rate deduce} to verify the convergence rates of $\\set{\\nabla f(x^k)}$ and $\\set{f(x^k)}.$ Since $f$ is of class ${\\cal C}^{1,1}$ on $\\B(\\bar x,r)$ and since $H_\\alpha$ is of class ${\\cal C}^{1,1}$ on $\\B(\\bar x,r)\\times \\B(\\bar x,r)\\supset \\set{z^k}$, condition \\eqref{main condition rate} follows from \\eqref{ineq rate grad}. This clearly leads us to the conclusions:\n\\begin{itemize}\n\\item For $q\\in[0,1/2),$ we get $\\theta=1/2$. Therefore, $\\{H_\\alpha(z^k)\\}$ converges linearly to $H_\\alpha(\\bar z)$ and $\\{\\norm{\\nabla H_\\alpha(z^k)}\\}$ converges linearly to $0$ as $k\\to\\infty$.\n\n\\item For $q\\in (1/2,1),$ we have $H_\\alpha(z^k)-H_\\alpha(\\tilde z)=\\mathcal{O}(k^{-\\frac{2-2q}{2q-1}})$ and $\\norm{\\nabla H_\\alpha(z^k)}=\\mathcal{O}(k^{-\\frac{1-q}{2q-1}})$ as $k\\to\\infty$.\n\\end{itemize}\nThe claimed convergence rates of the sequences $\\{\\norm{\\nabla f(x^k)}\\}$ and $\\set{f(x^k)}$ are now deduced from those for $\\set{\\norm{\\nabla H_\\alpha (z^k)}},\\set{H_\\alpha (z^k)}$ and the estimates\n\\begin{align*}\n\\norm{\\nabla H_\\alpha(x,y)}&=\\sqrt{\\norm{\\nabla f(x)+2\\alpha (x-y)}^2+4\\alpha^2\\norm{x-y}^2}\\ge \\norm{\\nabla f(x)},\\\\\nH_\\alpha(x,y)&=f(x)+\\alpha\\norm{x-y}^2\\ge f(x)\\;\\text{ for all }\\;x,y\\in\\R^n.\n\\end{align*}\nThis completes the proof of the theorem.\n\\end{proof}","units":[{"start":21658,"end":22024,"start_line":250,"end_line":255,"raw_tex":"\\begin{Lemma}\\label{lemma descent}\nGiven $f:\\R^n\\rightarrow\\R$ and $x,y\\in\\Omega$, assume that $f$ is differentiable on the line segment $[x,y]$ and its derivative is Lipschitz continuous on this segment with constant $L>0$. Then we have\n\\begin{align}\\label{eq descent}\n\\abs{f(y)-f(x)-\\dotproduct{\\nabla f(x),y-x}}\\le \\frac{L}{2}\\norm{y-x}^2.\n\\end{align}\n\\end{Lemma}","sha256_utf8":"4198f1dbb9ce50c022341b6aef82c7a6f337f62c63dd852c50beba1de6a9c229"},{"start":24207,"end":24497,"start_line":281,"end_line":283,"raw_tex":"\\begin{Lemma}\\label{isolation lemma}\nLet $f:\\R^n\\rightarrow\\overline{\\R}$ be continuous around some local minimizer $\\bar x$ and satisfy the PLK condition at $\\bar x$. Then there is a neighborhood $U$ of $\\bar x$ such that $f(\\bar x)$ is the unique critical value of $f$ in $U$.\n\\end{Lemma}","sha256_utf8":"027eacf5203057f8845f669ec97e00ca04e8d8a7424a7aa52c282668dd311ac4"},{"start":24498,"end":25181,"start_line":284,"end_line":290,"raw_tex":"\\begin{proof}\nThe PLK condition for $f$ at $\\bar x$ gives us $\\eta\\in(0,\\infty],$ a neighborhood $U$ of $\\bar x$, and a function $\\varphi:[0,\\eta)\\rightarrow[0,\\infty)$ satisfying all the conditions in Definition~\\ref{KL ine}. Since $f$ is continuous around the local minimizer $\\bar x,$ we assume, by shrinking $U$ if necessary, that $f(\\bar x)\\le f(x)<f(\\bar x)+\\eta.$ Then condition (iv) in Definition~\\ref{KL 2} can be rewritten as\n\\begin{align*}\n\\varphi'(f(x)-f(\\bar x))d(0,\\partial f(x))\\ge 1\\text{ whenever }x\\in U,\\;f(x)\\ne f(\\bar x).\n\\end{align*}\nTherefore, $0\\notin \\partial f(x)$ for all such $x$, i.e., $f(\\bar x)$ is the unique critical value of $f$ on $U.$ \n\\end{proof}","sha256_utf8":"41e8e265773d8d27be5006dca5481258b86bb3365800fffc9e161c46807e4e5d"},{"start":25360,"end":26193,"start_line":294,"end_line":307,"raw_tex":"\\begin{Proposition}\\label{general convergence under KL}\nLet $f:\\R^n\\rightarrow\\R$ be a $\\mathcal{C}^1$-smooth function, and let the following conditions hold along a sequence of iterates $\\set{x^k}\\subset\\R^n$ for the function $f$:\n\\begin{itemize}\n\\item[\\bf(H1)] {\\rm(primary descent condition)}. There exists $\\sigma>0$ such that for sufficiently large $k\\in\\N$, we have \n\\begin{align*}\n f(x^k)-f(x^{k+1})\\ge\\sigma\\norm{\\nabla f(x^k)}\\cdot\\norm{x^{k+1}-x^k}.\n\\end{align*}\n\\item[\\bf(H2)] {\\rm(complementary descent condition)}. For sufficiently large $k\\in\\N$, we have\n\\begin{align*}\n \\big[f(x^{k+1})=f(x^k)\\big]\\Longrightarrow [x^{k+1}=x^k].\n\\end{align*}\n\\end{itemize}\nIf $\\bar x$ is an accumulation point of $\\set{x^k}$  and $f$ satisfies the PLK property at $\\bar x$, then $x^k\\rightarrow\\bar x$ as $k\\to\\infty$.\n\\end{Proposition}","sha256_utf8":"eaaa9b37fbe4a4db0f08b387e35d605ef5cb42622348e6fd7681df6e9ca59adf"},{"start":26473,"end":27434,"start_line":311,"end_line":323,"raw_tex":"\\begin{Proposition}\\label{general rate}\nLet $f:\\R^n\\rightarrow\\R$ be a $\\mathcal{C}^1$-smooth function, and let $\\set{x^k}$ satisfy the conditions \n\\begin{align}\\label{two conditions}\nf(x^k)-f(x^{k+1})\\ge \\alpha\\norm{x^{k+1}-x^k}^2\\;\\text{ and }\\;\\norm{\\nabla f(x^k)}\\le \\beta\\norm{x^{k+1}-x^k}\n\\end{align}\nfor all $k\\in\\N$ sufficiently large with some constants $\\alpha,\\beta>0$. Suppose  in addition that $\\bar x$ is an accumulation point of $\\set{x^k}$ and that $f$ satisfies the $($exponential$)$ PLK property at $\\bar x$ with $\\varphi(t)=Mt^{1-q}$ for some $M>0$ and $q\\in(0,1)$. Then the following convergence rates are guaranteed:\n\n{\\bf (i)} For $q\\in(0,1/2)$, the sequence $\\set{x^k}$ terminates at $\\bar x$ in finite steps.\n\n{\\bf (ii)} For $q=1/2$, the sequence $\\set{x^k}$ converges linearly to $\\bar x$ as $k\\to\\infty$.\n\n{\\bf (iii)} For $q\\in(1/2,1)$, we have  $\\norm{x^k-\\bar x}=\\mathcal{O}( k^{-\\frac{1-q}{2q-1}})$ as $k\\to\\infty$\n\\end{Proposition}","sha256_utf8":"be3f4fb0460bd5028140959b524ea56fb16ee70b58fcc4b4b5e89276d6ba3e09"},{"start":29878,"end":31293,"start_line":346,"end_line":361,"raw_tex":"\\begin{Proposition}\\label{isolated} Let $f:\\R^n\\rightarrow\\overline\\R$ be a ${\\cal C}^1$-smooth function around its local minimizer $\\bar x$. Assume that $f$ satisfies the PLK condition at $\\bar x$ with the desingularizing function $\\psi:[0,\\eta)\\rightarrow[0,\\infty)$ for some $\\eta\\in (0,\\infty]$. Then the following assertions hold whenever $\\varepsilon\\in (0,1/4]$:\n   \n{\\bf(i)} The collection of functions $\\set{H_\\alpha}_{\\alpha\\ge \\varepsilon}$ satisfies the same PLK condition at $\\bar z:=(\\bar x,\\bar x)$ with the desingularizing function $\\varphi:[0,\\eta)\\rightarrow[0,\\infty)$ defined by $\\varphi(0):=0$ and \n\\begin{align}\\label{construction varphi}\n\\varphi(t):=\\sqrt{\\frac{8}{\\varepsilon}}\\int_0^{t/2} \\max\\set{\\psi'(s),\\frac{1}{\\sqrt{s}}}ds\\;\\text{ for all }\\;t\\in (0,\\eta).\n\\end{align}\n        \n{\\bf(ii)} If $\\psi(t)=Mt^{1-q}$ for some $M>0$ and $q\\in[0,1)$, then the function $\\varphi$ in \\eqref{construction varphi} is given by $\\varphi(t)=\\frac{2^{\\theta+1/2}\\tilde M}{\\sqrt{\\varepsilon}(1-\\theta)}t^{1-\\theta}$ when $t\\in (0,\\delta)$ for some $\\delta\\in(0,\\eta)$, where $\\theta:=\\max\\set{q,\\frac{1}{2}}$ and $\\tilde{M}>0$ is defined as \n\\begin{align}\\label{tilde M}\n\\tilde{M}:=\\begin{cases}\n1&\\text{ if }\\quad q\\in \\left[0,\\frac{1}{2}\\right),\\\\\n\\max \\set{M(1-q),1}&\\text{ if }\\quad q=\\frac{1}{2},\\\\\nM(1-q)&\\text{ if }\\quad q\\in \\left(\\frac{1}{2},1\\right).\n\\end{cases}\n\\end{align}\n\\end{Proposition}","sha256_utf8":"43c2d7da56317a0d8745a9079da20d991f4eaae4c79dc7465b2422d4d1c9c10f"},{"start":31294,"end":36550,"start_line":362,"end_line":429,"raw_tex":"\\begin{proof}\nTo verify (i) pick any $\\varepsilon \\in (0,1/4]$ and use the PLK property of $f$ at its local minimizer $\\bar x$ to find a neighborhood $U$ of $\\bar x$ on which $f$ satisfies all the conditions in Definition~\\ref{KL ine} with the corresponding desingularizing function $\\psi: [0,\\eta) \\rightarrow [0,\\infty)$. In particular, we have\n\\begin{align}\\label{desingularizing}\n\\psi'\\big(f(x)-f(\\bar{x})\\big)\\norm{\\nabla f(x)}\\ge 1\\;\\mbox{  for all }\\;x \\in U\\;\\mbox{ with }\\;f(\\bar{x}) < f(x) < f(\\bar{x}) + \\eta.\n\\end{align}\nThe function $\\varphi$ in \\eqref{construction varphi} can be written as $\\varphi(t)=\\sqrt{\\frac{8}{\\varepsilon}}\\phi\\brac{\\frac{t}{2}}$, where $\\phi:[0,\\eta)\\rightarrow[0,\\infty)$ is defined by \n\\begin{align}\\label{construction phi t}\n\\phi(t):=\\int_0^t \\max\\set{\\psi'(s),\\frac{1}{\\sqrt{s}}}ds\\;\\text{ for }\\;t\\in(0,\\eta)\\;\\mbox{ with }\\;\\phi(0):=0.\n\\end{align}\nIt follows from the continuity of $\\psi$ at $0$ that\n\\begin{align*}\n0\\le \\phi(t)\\le \\int_0^t \\psi'(s)ds+ \\int_0^t\\frac{1}{\\sqrt{s}}ds=\\psi(t)+2\\sqrt{t}\\;\\text{ for }\\;t\\in(0,\\eta).\n\\end{align*}\nPassing the limit above as $t\\rightarrow0^+$ and using the continuity of $\\psi$ reveal that $\\phi$ is also continuous at $0$. Moreover, we deduce from \\eqref{construction phi t} that the derivative $\\phi'(t)=\\max \\set{\\psi'(t),\\frac{1}{\\sqrt{t}}}>0$ is positive, continuous, and nonincreasing on $(0,\\eta)$, which ensures the concavity of $\\phi$ on this interval. In addition, it follows that \n\\begin{align}\\label{phi'>1}\n\\phi'(f(x)-f(\\bar x))\\norm{\\nabla f(x)}\\ge \\psi'(f(x)-f(\\bar x))\\norm{\\nabla f(x)}\\ge 1\n\\end{align}\nfor $x\\in U$ with $f(\\bar x)< f(x)<f(x)+\\eta$. Since $\\varphi(t)=\\sqrt{\\frac{8}{\\varepsilon}}\\phi\\brac{\\frac{t}{2}}$, the domain of $\\varphi$ is $[0,2\\eta).$ It is clear that $\\varphi$ satisfies all the conditions as a desingularizing function on $[0,2\\eta)$ with\n\\begin{align}\\label{deri varphi}\n\\varphi'(t)=\\sqrt{\\frac{2}{\\varepsilon}}\\phi'\\brac{\\frac{t}{2}}>0 \\text{ for all }t\\in (0,2\\eta).\n\\end{align}\nTaking any $\\alpha\\ge \\varepsilon$ and $z=(x,y)\\in Z:= U\\times U$ with \n\\begin{align}\\label{halpha}\nH_\\alpha(\\bar z)<H_\\alpha(z)<H_\\alpha(\\bar z)+\\eta,\n\\end{align}\nwe aim at verifying the inequality\n\\begin{align}\\label{KL R2n}\n\\varphi'(H_\\alpha(z)-H_\\alpha(\\bar z))\\norm{\\nabla H_\\alpha (z)}\\ge 1.\n\\end{align}\nTo proceed, observe first the estimates\n\\begin{subequations}\n\\begin{align}\n\\norm{\\nabla H_\\alpha(x,y)}&=\\sqrt{\\norm{\\nabla f(x)+2\\alpha(x-y)}^2+\\norm{2\\alpha(x-y)}^2} \\label{g_Hal.a}\\\\\n&\\ge \\sqrt{\\frac{1}{4}\\norm{\\nabla f(x)}^2+\\alpha^2\\norm{x-y}^2}\\label{g_Hal.b}\\\\\n&\\ge \\sqrt{\\frac{1}{4}\\norm{\\nabla f(x)}^2+\\alpha\\varepsilon\\norm{x-y}^2}\\label{g_Hal.c}\\\\\n&\\ge \\sqrt{\\frac{\\varepsilon}{2}}\\brac{\\norm{\\nabla f(x)}+\\sqrt{\\alpha}\\norm{x-y}},\\label{g_Hal.d}\n\\end{align}\n\\end{subequations}\nwhere \\eqref{g_Hal.a} is comes from the construction of $H_\\alpha$, \\eqref{g_Hal.b} follows from the inequality $\\norm{a+b}^2\\ge \\frac{1}{4}\\norm{a}^2-\\frac{3}{4}\\norm{b}^2$ for $a,b\\in\\R^n$, \\eqref{g_Hal.c} follows from $\\alpha\\ge \\varepsilon$, and \\eqref{g_Hal.d} follows from $a^2+b^2\\ge \\frac{1}{2}(a+b)^2$ for $a,b\\in\\R$ and $\\varepsilon\\le 1/4$.  Now let $S:=\\max \\set{\\alpha\\norm{x-y}^2,f(x)-f(\\bar x)}$ and deduce from $f(x)\\ge f(\\bar x)$ as $x\\in U$ that\n\\begin{align*}\nH_\\alpha(z)-H_\\alpha(\\bar z)=f(x)+\\alpha\\norm{x-y}^2-f(\\bar x) \\in [S,2S].\n\\end{align*}\nCombining the latter with $0< H_\\alpha(z)-H_\\alpha(\\bar z)<\\eta$ by \\eqref{halpha} gives us $S\\in (0,\\eta).$ Using the nonincreasing property of $\\varphi'$ and the relation $\\varphi'(t)=\\sqrt{\\frac{2}{\\varepsilon}}\\phi'\\brac{\\frac{t}{2}}$ from \\eqref{deri varphi} yields\n\\begin{align*}\n\\varphi'(H_\\alpha(z)-H_\\alpha(\\bar z))&\\ge \\varphi'\\brac{2S}=\\sqrt{\\frac{2}{\\varepsilon}}\\phi'(S).\n\\end{align*}\nFurthermore, it follows from  \\eqref{deri varphi} and  \\eqref{g_Hal.d} that\n\\begin{align*}\n\\varphi'(H_\\alpha(z)-H_\\alpha(\\bar z))\\norm{\\nabla H_\\alpha(x,y)}&\\ge \\sqrt{\\frac{2}{\\varepsilon}}\\phi'(S)\\sqrt{\\frac{\\varepsilon}{2}}\\brac{\\norm{\\nabla f(x)}+\\sqrt{\\alpha}\\norm{x-y}}\\\\\n&=\\phi'(S)\\norm{\\nabla f(x)}+\\phi'(S)\\sqrt{\\alpha}\\norm{x-y}.\n\\end{align*}\nBy $\\phi'\\big(f(x)-f(\\bar{x})\\big)\\norm{\\nabla f(x)}\\ge 1$ coming from \\eqref{phi'>1} and $\\phi'(\\alpha \\norm{x-y}^2)\\ge \\frac{1}{\\sqrt{\\alpha}\\norm{x-y}}$ due to $\\phi'(t)\\ge \\frac{1}{\\sqrt{t}}$ for $t\\in (0,\\eta),$ we get the estimate\n\\begin{align*}\n\\phi'(S)\\norm{\\nabla f(x)}+\\phi'(S)\\sqrt{\\alpha}\\norm{x-y}\\ge 1,\n\\end{align*}\nwhich therefore verifies assertion (i) of the proposition.\n\nTo prove assertion (ii)  is easier. Indeed, let $\\psi(t)=Mt^{1-q}$ for $q\\in[0,1)$, let $\\theta:=\\max\\set{\\frac{1}{2},q}\\in (0,1)$, and take $\\tilde M$ from \\eqref{tilde M}. It readily follows that\n\\begin{align}\\label{formu phi}\n\\max\\set{\\psi'(s),\\frac{1}{\\sqrt{s}}}=\\tilde Ms^{-\\theta}\\text{ for all }s\\in(0,\\delta)\n\\end{align}\nfor some $\\delta\\in(0,\\eta)$ sufficiently small.\nCombining the latter with \\eqref{construction phi t}, we arrive at \n\\begin{align*}\n \\varphi(t)=\\sqrt{\\frac{8}{\\varepsilon}}\\int_0^{t/2}\\tilde{M}s^{-\\theta}ds=\\frac{2^{\\theta+1/2}\\tilde M}{\\sqrt{\\varepsilon}(1-\\theta)}t^{1-\\theta} \\text{ for all }t\\in (0,\\delta),\n\\end{align*}\nwhich thus verifies (ii) and completes the proof of the proposition.\n\\end{proof}","sha256_utf8":"e7c5ec95a1bbc36f3fe942704b9263d00286e1542ce850783bb0f936d0a6f201"},{"start":36689,"end":37602,"start_line":433,"end_line":443,"raw_tex":"\\begin{Proposition}\\label{convergence rate deduce}\nLet $f:\\Omega\\rightarrow{\\R}$ be a function defined on open and convex set of $\\Omega\\subset\\R^n$, and let $\\bar x \\in\\Omega$. Assume that $f$ has a Lipschitz continuous gradient on $\\Omega$ and take a sequence $\\set{x^k}\\subset\\Omega $ converging to $\\bar x$. Suppose in addition that there exists $\\alpha>0$ such that \n\\begin{align}\\label{main condition rate}\n\\alpha\\norm{\\nabla f(x^k)}^2\\le f(x^k)-f(x^{k+1})\\;\\text{ for all large }\\;k\\in\\N.\n\\end{align}\nThen the linear convergence of $x^k\\rightarrow\\bar x$ ensures the linear convergences of $f(x^k)$ to $f(\\bar x)$ and $\\nabla f(x^k)$ to $0$ as $k\\to\\infty$. Moreover, the convergence rate $\\norm{x^k-\\bar x}=\\mathcal{O}(m(k))$ with $m(k)\\downarrow 0$ yields\n$$\n\\abs{f(x^k)-f(\\bar x)}=\\mathcal{O}(m^2(k))\\;\\mbox{ and }\\;\\norm{\\nabla f(x^k)}=\\mathcal{O}(m(k))\\;\\mbox{ as }\\;\nk\\to\\infty. \n$$\n\\end{Proposition}","sha256_utf8":"52ab37c2f5d6972e11a6e3f395a159691a49ca3428c0d241246ab9e5040f99da"},{"start":41486,"end":42373,"start_line":499,"end_line":515,"raw_tex":"\\begin{Proposition}\\label{prop AIGD tech}\nLet $\\set{x^k}$ be a sequence generated by Algorithm~{\\rm\\ref{IGDm}} with \n\\begin{align}\\label{max Lt<1-nu}\n\\max\\set{L\\tau,2L\\tau \\bar \\delta+(L\\tau+1)\\bar \\beta^2}< 1-\\nu\n\\end{align}\nsuch that $x^k,x^k_{\\rm in},x^k_{\\rm ex}\\in X$ for all $k=1,\\ldots,K\\in\\N$. Consider the positive constants\n\\begin{align}\\label{C1C2}\n\\alpha:&=\\frac{(L \\tau+1)\\bar \\beta^2+1-\\nu }{4\\tau},\\nonumber\\\\\nC_1:&=\\frac{1-\\nu-(L \\tau+1)\\bar \\beta^2-2L\\tau \\bar \\delta}{4\\tau},\\\\\nC_2:&=\\sqrt{2}\\max\\set{\\frac{\\nu+1}{\\tau },\\frac{\\bar\\beta(L\\tau+\\nu+1)+L\\tau\\bar\\delta}{\\tau }}+4{\\alpha}.\\nonumber\n\\end{align}\nThen we have the following estimates whenever $k=1,\\ldots,K-1:$\n\\begin{align}\nC_1\\norm{z^{k+1}-z^k}^2&\\le H_{\\alpha}(z^k)-H_{\\alpha}(z^{k+1}),\\label{3.2(i)}\\\\\n\\norm{\\nabla H_{\\alpha}(z^{k})}&\\le C_2\\norm{z^{k+1}-z^k}\\label{3.2(ii)}.\n\\end{align}\n\\end{Proposition}","sha256_utf8":"7117e4af895cc3bbe0bf9274ecbe3eddaaa7ebce9567def3e98ccf3f97bc1633"},{"start":42374,"end":49258,"start_line":516,"end_line":602,"raw_tex":"\\begin{proof}\nBy Lemma~\\ref{lemma descent}, the Lipschitz continuity with constant $L>0$ of $\\nabla f$ on $X $ yields\n\\begin{align*}\n-\\frac{L}{2}\\norm{y-x}^2\\le f(y)-f(x)-\\dotproduct{\\nabla f(x),y-x}\\le \\frac{L}{2}\\norm{y-x}^2 \\text{ for all }x,y\\in X .\n\\end{align*}\nFor any $k=1,\\ldots,K-1$, we use  the left-hand side inequality above with $y=x^k,x=x^k_{\\rm in}$ and the right-hand side inequality with $y=x^{k+1},x=x^k_{\\rm in}$ to get\n\\begin{align*}\n f(x^k)-f(x^k_{\\rm in})-\\dotproduct{\\nabla f(x^k_{\\rm in}),x^k-x^k_{\\rm in}}&\\ge -\\frac{L }{2}\\norm{x^k-x^k_{\\rm in}}^2,\n \\\\\n f(x^{k+1})- f(x^k_{\\rm in}) -\\dotproduct{\\nabla f(x^k_{\\rm in}),x^{k+1}-x^k_{\\rm in}}&\\le \\frac{L}{2}\\norm{x^{k+1}-x^k_{\\rm in}}^2.\n\\end{align*}\nSubtracting the first inequality from the second one gives us \n\\begin{align*}\nf(x^{k+1})-f(x^k)-\\dotproduct{\\nabla f(x^k_{\\rm in}),x^{k+1}-x^k}\\le \\frac{L}{2}\\norm{x^{k+1}-x^k_{\\rm in}}^2+\\frac{L }{2}\\norm{x^k-x^k_{\\rm in}}^2,\n\\end{align*}\nwhich can be rewritten in the form\n\\begin{align}\\label{prop: 1 chain}\n\\begin{array}{ll}\n\\disp f(x^{k+1})-f(x^k)\\le \\frac{L}{2}\\norm{x^{k+1}-x^k_{\\rm in}}^2+\\frac{L }{2}\\norm{x^k-x^k_{\\rm in}}^2+\\dotproduct{\\nabla f(x^k_{\\rm in}),x^{k+1}-x^k}\\\\\n=\\disp\\frac{L}{2} \\norm{x^{k+1}-x^k_{\\rm in}}^2+\\frac{L \\beta_k^2}{2}\\norm{x^k-x^{k-1}}^2+\\dotproduct{\\nabla f(x^k_{\\rm in})-\\nabla f(x^k_{\\rm ex}),x^{k+1}-x^k}\\\\\n\\disp+\\dotproduct{\\nabla f(x^k_{\\rm ex})- g^k,x^{k+1}-x^k}+\\dotproduct{g^k,x^{k+1}-x^k}.\n\\end{array}\n\\end{align}\nBy the Cauchy-Schwarz inequality, the Lipschitz continuity of $\\nabla f$ on $X $, and the constructions of $x^k_{\\rm in}$ and $x^k_{\\rm ex}$ in \\eqref{step xin} and \\eqref{step xex}, we deduce the estimates\n\\begin{align}\\label{prop: 1.1 chain}\n\\begin{array}{ll}\n\\dotproduct{\\nabla f(x^k_{\\rm in})-\\nabla f(x^k_{\\rm ex}),x^{k+1}-x^k}\\le \\norm{\\nabla f(x^k_{\\rm in})-\\nabla f(x^k_{\\rm ex})}\\cdot\\norm{x^{k+1}-x^k}\\\\\n\\le L\\norm{x^k_{\\rm in}-x^k_{\\rm ex}}\\cdot\\norm{x^{k+1}-x^k}\\le L\\abs{\\beta_k-\\gamma_k}\\norm{x^k-x^{k-1}}\\cdot\\norm{x^{k+1}-x^k}\\\\\n\\le\\disp\\frac{L\\bar \\delta}{2}\\brac{\\norm{x^k-x^{k-1}}^2+\\norm{x^{k+1}-x^k}^2}.\n\\end{array}\n\\end{align}\nIt follows from the Cauchy-Schwarz and AM-GM inequalities together with the algorithm design $x^{k+1}:=x^k_{\\rm in}-\\tau  g^k$ and $\\norm{g^k-\\nabla f(x^k_{\\rm ex})}\\le \\nu\\norm{g^k}$ in \\eqref{iteration update} that\n\\begin{align}\\label{prop: 3 chain}\n\\begin{array}{ll}\n\\dotproduct{\\nabla f(x^k_{\\rm ex})- g^k,x^{k+1}-x^k}\\le \\norm{\\nabla f(x^k_{\\rm ex})- g^k}\\cdot\\norm{x^{k+1}-x^k}\\\\\n\\disp\\le \\nu \\norm{g^k}\\cdot\\norm{x^{k+1}-x^k}=\\frac{\\nu}{\\tau }\\norm{x^k_{\\rm in}-x^{k+1}}\\cdot\\norm{x^{k+1}-x^k}\\\\\n\\le\\disp \\frac{\\nu}{2\\tau }\\brac{\\norm{x^{k+1}-x^k_{\\rm in}}^2+\\norm{x^{k+1}-x^k}^2}.\n\\end{array}\n\\end{align}\nUsing further \\eqref{step xin} and \\eqref{iteration update},  we arrive at\n\\begin{align}\\label{prop: 2 chain}\n\\begin{array}{ll}\n\\dotproduct{g^k,x^{k+1}-x^k}=\\disp\\frac{1}{\\tau }\\dotproduct{x^k_{\\rm in}-x^{k+1},x^{k+1}-x^k}\\\\\n=\\disp\\frac{1}{2\\tau }\\brac{\\norm{x^k_{\\rm in}-x^k}^2-\\norm{x^{k+1}-x^k_{\\rm in}}^2-\\norm{x^k-x^{k+1}}^2}\\\\\n=\\disp\\frac{1}{2\\tau }\\brac{\\beta_k^2\\norm{x^{k}-x^{k-1}}^2-\\norm{x^{k+1}-x^k_{\\rm in}}^2-\\norm{x^k-x^{k+1}}^2}.\n\\end{array}\n\\end{align}\nCombining \\eqref{prop: 1 chain}--\\eqref{prop: 2 chain} with taking into account $L\\tau +\\nu<1$ and $\\beta_k\\le \\bar \\beta$ gives us\n\\begin{align}\\label{estimane conse}\n\\begin{array}{ll}\nf(x^{k+1})-f(x^k)&\\le\\disp\\frac{L\\tau +\\nu-1}{2\\tau }\\norm{x^{k+1}-x^k_{\\rm in}}^2+\\frac{L\\tau\\bar \\delta+\\nu-1}{2\\tau }\\norm{x^{k+1}-x^k}^2\\\\\n&+\\disp\\frac{L \\tau \\beta_k^2+\\beta_k^2+L\\tau\\bar \\delta}{2\\tau }\\norm{x^k-x^{k-1}}^2\\\\\n&\\le\\disp\\frac{L\\tau\\bar \\delta+\\nu-1}{2\\tau }\\norm{x^{k+1}-x^k}^2+\\frac{(L \\tau +1)\\bar \\beta^2+L\\tau\\bar \\delta}{2\\tau }\\norm{x^k-x^{k-1}}^2.\n\\end{array}\n\\end{align}\nSince $\\bar\\beta^2<\\frac{1-\\nu}{L \\tau+1}$, it follows that $\\frac{(L \\tau+1)\\bar \\beta^2+L\\tau\\bar \\delta}{2\\tau }<\\frac{1-\\nu-L\\tau\\bar \\delta}{2\\tau }$, and by $\\alpha=\\frac{(L \\tau+1)\\bar \\beta^2+1-\\nu}{4\\tau}$ we get \n\\begin{align*}\nC_1=\\frac{1-\\nu-(L \\tau+1)\\bar \\beta^2-2L\\tau \\bar \\delta}{4\\tau}=\\frac{1-\\nu-L\\tau\\bar\\delta}{2\\tau}-\\alpha=\\alpha-\\frac{(L  \\tau+1)\\bar\\beta^2+L\\tau\\bar \\delta}{2\\tau}\n\\end{align*}\nfor $C_1$ in \\eqref{C1C2}. Using the latter together with \\eqref{estimane conse} and the construction of $H_\\alpha$ in \\eqref{lyapunov func}  brings us to\n\\begin{align*}\nH_{\\alpha}(z^{k+1})-H_{\\alpha}(z^k)&= f(x^{k+1})+\\alpha\\norm{x^{k+1}-x^k}^2-f(x^{k})-\\alpha\\norm{x^{k}-x^{k-1}}^2\\\\\n&\\le\\brac{ \\frac{\\nu-1+L\\tau\\bar \\delta}{2\\tau }+\\alpha}\\norm{x^{k+1}-x^k}^2+\\brac{\\frac{(L \\tau+1)\\bar  \\beta^2+L\\tau\\bar \\delta}{2\\tau }-\\alpha}\\norm{x^k-x^{k-1}}^2\\\\\n&\\le- C_1\\brac{\\norm{x^{k+1}-x^k}^2+\\norm{x^k-x^{k-1}}^2}=-C_1\\norm{z^{k+1}-z^k}^2,\n\\end{align*}\nwhich therefore justifies the inequality in \\eqref{3.2(i)}. \n\nTo verify \\eqref{3.2(ii)}, take any $k=1,\\ldots,K-1$ and obtain the relationships\n\\begin{subequations}\n\\begin{align}\n\\norm{\\nabla f(x^k)}&\\le \\norm{\\nabla f(x^k)-\\nabla f(x^k_{\\rm in})} +\\norm{\\nabla f(x^k_{\\rm in})-\\nabla f(x^k_{\\rm ex})}+\\norm{\\nabla f(x^k_{\\rm ex})-g^k}+\\norm{g^k}\\label{nab.1 triang}\\\\\n&\\le(L\\bar \\beta+L\\bar\\delta)\\norm{x^k-x^{k-1}}+(\\nu+1)\\norm{g^k}\\label{nab.2}\\\\\n&=(L\\bar \\beta+L\\bar\\delta)\\norm{x^k-x^{k-1}}+\\frac{\\nu+1}{\\tau }\\norm{x^k+\\beta_k(x^k-x^{k-1})-x^{k+1}}\\label{nab.3}\\\\\n&\\le \\brac{L\\bar \\beta+L\\bar\\delta+\\frac{\\nu+1}{\\tau }\\bar \\beta}\\norm{x^k-x^{k-1}}+\\frac{\\nu+1}{\\tau }\\norm{x^{k+1}-x^k}\\nonumber\\\\\n&\\le \\max\\set{\\frac{\\nu+1}{\\tau },L\\bar \\beta+L\\bar\\delta+\\frac{\\nu+1}{\\tau }\\bar \\beta}\\brac{\\norm{x^{k+1}-x^k}+\\norm{x^k-x^{k-1}}}\\nonumber\\\\\n&\\le \\sqrt{2}\\max\\set{\\frac{\\nu+1}{\\tau },L\\bar \\beta+L\\bar\\delta+\\frac{\\nu+1}{\\tau }\\bar \\beta}\\norm{z^{k+1}-z^k}\\label{nab.6},\n\\end{align}\n\\end{subequations}\nwhere \\eqref{nab.1 triang} follows from the triangle inequality; \\eqref{nab.2} follows from $\\norm{g^k-\\nabla f(x^k_{\\rm ex})}\\le \\nu\\norm{g^k}$ in \\eqref{iteration update}, the updates in \\eqref{step xin}, \\eqref{step xex}, and the Lipschitz continuity of $\\nabla f$ on $X $ containing $ x^k_\\rin,x^k_\\rex$; \\eqref{nab.3} follows from \\eqref{step xin} and \\eqref{step xex}; \\eqref{nab.6} follows from $a+b\\le \\sqrt{2(a^2+b^2)}$ and $z^k=(x^k,x^{k-1})$. By using the estimates above, we deduce from the construction of $H_\\alpha$ and inequality $\\sqrt{a^2+b^2}\\le a+b$ for $a,b\\ge 0$ that\n\\begin{align*}\n\\norm{\\nabla H_{\\alpha}(z^k)}&= \\sqrt{\\norm{\\nabla f(x^k)+2{\\alpha}(x^k-x^{k-1})}^2+\\norm{2{\\alpha} (x^k-x^{k-1})}^2}\\\\\n&\\le \\norm{\\nabla f(x^k)+2{\\alpha}(x^k-x^{k-1})}+\\norm{2{\\alpha} (x^k-x^{k-1})}\\\\\n& \\le \\norm{\\nabla f(x^k)}+4{\\alpha}\\norm{x^k-x^{k-1}}\\\\\n&\\le \\brac{\\sqrt{2}\\max\\set{\\frac{\\nu+1}{\\tau },L\\bar \\beta+L\\bar\\delta+\\frac{\\nu+1}{\\tau }\\bar \\beta}+4{\\alpha}}\\norm{z^{k+1}-z^k},\n\\end{align*}\nwhich justifies \\eqref{3.2(ii)}  and thus completes the proof of the proposition.\n\\end{proof}","sha256_utf8":"790488d3bdb597a7f92743b90eecf4a7192583c35d8c0b803c61b1a25530da12"},{"start":49432,"end":49901,"start_line":606,"end_line":612,"raw_tex":"\\begin{Proposition}\\label{proposition distance}\nLet $\\set{x^k}$ be the sequence of iterates generated by Algorithm~{\\rm\\ref{IGDm}} such that $x^k,x^k_{\\rm ex}\\in \\B(\\bar x,\\rho)\\subset X$ for all $k=1,\\ldots,K$. Then we have for all $k=1,\\ldots,K$ that\n\\begin{align}\n\\norm{x^{k+1}-x^k}&\\le   \\frac{L\\tau\\rho }{(1-\\bar \\beta)(1-\\nu)},\\label{3.3(i)}\\\\\n\\norm{z^{k+1}-z^k}&\\le  \\frac{\\sqrt{2} L\\tau\\rho }{(1-\\bar \\beta)(1-\\nu)}.\\label{3.3(ii)}\n\\end{align}\n\\end{Proposition}","sha256_utf8":"d8a23aa74c1081ff7f5fa8997ea5ba27fd47f14adbb8112c2ede13a15b06f96f"},{"start":49902,"end":51672,"start_line":613,"end_line":633,"raw_tex":"\\begin{proof}\nPick any $k=1,\\ldots,K$ and observe that the iteration update in Algorithm~\\ref{IGDm} can be rewritten as $x^{i+1}=x^i+\\beta_k(x^i-x^{i-1})-\\tau g^i$, which yields\n\\begin{align}\\label{new update}\n\\norm{x^{i+1}-x^i}\\le \\bar \\beta \\norm{x^i-x^{i-1}}+\\tau\\norm{g^i}\\;\\text{ for all }\\;i\\le k.\n\\end{align}\nUsing the triangle inequality together with $\\norm{g^i-\\nabla f(x^i_{\\rm ex})}\\le \\nu \\norm{g^k}$ gives us\n\\begin{align*}\n\\norm{\\nabla f(x^i_{\\rm ex})}\\ge \\norm{g^i}-\\norm{\\nabla f(x^i_{\\rm ex})-g^i}\\ge (1-\\nu)\\norm{g^i}\\;\\text{ for all }\\;i\\le k.\n\\end{align*} \nProceeding inductively from \\eqref{new update} for $i=k,k-1,\\ldots,1$ with the usage of the Lipschitz continuity of $\\nabla f$ on the ball $\\B(\\bar x,\\rho)$ containing $\\bar x$ and $x^{k-i}_\\rex,i=0,\\ldots,k-1$ yields\n\\begin{align*}\n\\norm{x^{k+1}-x^k}&\\le \\bar \\beta\\norm{x^k-x^{k-1}}+\\tau \\norm{g^k}\\\\\n&=\\bar \\beta^2\\norm{x^{k-1}-x^{k-2}}+\\tau\\brac{\\bar \\beta \\norm{g^{k-1}}+\\norm{g^k}}\\\\\n&\\ldots\\\\\n&\\le\\bar \\beta^{k}\\norm{x^1-x^0}+\\tau \\sum_{i=0}^{k-1}\\bar \\beta^i\\norm{g^{k-i}}\\\\\n&\\le \\frac{\\tau}{1-\\nu} \\sum_{i=0}^{k-1}\\bar \\beta^i\\norm{\\nabla f(x_\\rex^{k-i})} = \\frac{\\tau}{1-\\nu}\\sum_{i=0}^{k-1}\\bar \\beta^i\\norm{\\nabla f(x^{k-i}_\\rex)-\\nabla f(\\bar x)}\\\\\n&\\le \\frac{\\tau}{1-\\nu}\\sum_{i=0}^{k-1}\\bar \\beta^i L\\norm{x^{k-i}_\\rex -\\bar x}\\le \\frac{\\tau}{1-\\nu}\\sum_{i=0}^{k-1}\\bar \\beta^i L\\rho\\\\\n&\\le \\frac{L\\tau\\rho}{1-\\nu}\\sum_{i=0}^{\\infty}\\bar \\beta^i = \\frac{L\\tau\\rho}{(1-\\nu)(1-\\bar \\beta)},\n\\end{align*}\nwhere the latter inequality follows from $\\bar \\beta\\in [0,1)$. This readily verifies \\eqref{3.3(i)}. Combining \\eqref{3.3(i)}  with $\\norm{z^{k+1}-z^k}=\\sqrt{\\norm{x^{k+1}-x^k}^2+\\norm{x^{k}-x^{k-1}}^2}$ brings us to \\eqref{3.3(ii)} and thus completes the proof.\n\\end{proof}","sha256_utf8":"3baf4a198c6a0d4efb67a09a64d9cd00234c93fd05f24c3339cb6e53ea250d3a"},{"start":51798,"end":52129,"start_line":637,"end_line":638,"raw_tex":"\\begin{Proposition}\\label{convergence xk+1 xk} Let $\\set{x^k}$ be the sequence of iterates generated by Algorithm~{\\rm\\ref{IGDm}} that converges to some $\\tilde x\\in X$. Then $\\tilde x$ is a stationary point of \\eqref{main problem} and the sequences $\\set{x^k_\\rex},\\set{x^k_\\rin}$ converge to $\\tilde x$ as well.\n\\end{Proposition}","sha256_utf8":"e80482e37b66bfd4e63b4c2b5d9d2b80b32dec367452d4821813c17602504105"},{"start":52130,"end":53026,"start_line":639,"end_line":649,"raw_tex":"\\begin{proof} Since the sequence $\\set{x^k}$ is convergent,  we have $\\norm{x^{k+1}-x^k}\\rightarrow 0$ as $k\\rightarrow\\infty.$ Combining this with the updates in \\eqref{step xin} and \\eqref{step xex} together with the boundedness of $\\set{\\beta_k} and \\set{\\gamma_k}$ tells us that\n\\begin{align*}\n{x^k_{\\rm in}}&=x^k+\\beta_k({x^{k}-x^{k-1}})\\rightarrow\\tilde x,\\\\\n{x^k_{\\rm ex}}&=x^k+\\gamma_k({x^{k}-x^{k-1}})\\rightarrow\\tilde x.\n\\end{align*}\nUsing $\\norm{x^{k+1}-x^k}\\rightarrow 0$ and the updates in \\eqref{step xin}, \\eqref{iteration update} of Algorithm~\\ref{IGDm} yields\n\\begin{align*}\ng^k=\\frac{1}{\\tau}\\brac{x^k-x^{k+1}+\\beta_k(x^k-x^{k-1})}\\rightarrow 0\\;\\text{ as }\\;k\\rightarrow\\infty.\n\\end{align*}\nIt follows from $\\norm{\\nabla f(x^k_{\\rm ex})-g^k}\\le \\nu\\norm{g^k}$ in \\eqref{iteration update} that ${\\nabla f(x^k_{\\rm ex})}\\rightarrow 0$, and we are done with the proof.\n\\end{proof}","sha256_utf8":"d1f8439d07901e96be5a6208eef49290c1814574a3c87867d320dffa6344de9e"},{"start":62706,"end":76838,"start_line":737,"end_line":879,"raw_tex":"\\begin{proof} To verify (i), deduce from $\\bar \\beta \\in[0,\\sqrt{1-\\nu})$ that there exists $T>0$ such that\n\\begin{align}\\label{LT<1}\nT< \\frac{(1-\\bar \\beta)(1-\\nu)}{4L\\sqrt{2}(1+\\bar \\delta)},\\quad(LT+1)\\bar\\beta^2+2LT\\bar \\delta<1-\\nu,\\quad LT\\le 1-\\nu.\n\\end{align}\nDefine further $\\varepsilon:=\\min\\set{\\frac{1-\\nu}{4T},\\frac{1}{4}}\\in \\big(0,\\frac{1}{4}\\big]$ and deduce from Lemma~\\ref{isolation lemma} due to the imposed assumptions that there exists a neighborhood $U\\subset\\B(\\bar x,r)$ of $\\bar x$ such that $\\bar x$ minimizes $f$ on $U$ and that $f(\\bar x)$ is the only critical value within this set. Remembering that $f$ satisfies the basic PLK condition at $\\bar x$ tells us by Proposition~\\ref{isolated} that the Lyapunov function $\\set{H_\\alpha}_{\\alpha\\ge \\varepsilon}$ enjoys the same PLK property at $\\bar z:=(\\bar x,\\bar x)$ for some $\\eta\\in (0,\\infty]$, some neighborhood $Z$ of $\\bar z:=(\\bar x,\\bar x)$, and a concave desingularizing function $\\varphi:[0,\\eta)\\rightarrow[0,\\infty)$ with $\\varphi(0)=0$ which is ${\\cal C}^1$-smooth on $(0,\\eta)$. This means that for any $\\alpha\\ge \\varepsilon$ and any $z\\in Z$ with $H_\\alpha(\\bar z)< H_\\alpha( z)<H_\\alpha(\\bar z)+\\eta$, we get the inequality\n\\begin{align}\\label{KL R2n in proof}\n\\varphi'\\big(H_\\alpha(z )-H_\\alpha(\\bar z )\\big)\\norm{\\nabla H_\\alpha(z)}\\ge 1.\n\\end{align}\nThe continuity of $f$ allows us to find $\\rho\\in(0,r)$ such that \n\\begin{align}\\label{choice rho}\nf(x)+\\frac{L^2T\\rho^2}{2(1-\\bar \\beta)^2(1-\\nu)^2}< f(\\bar x)+\\eta\\text{ for all }x \\in \\mathbb{B}(\\bar x,\\rho).\n\\end{align}\nInvoking further the continuity of $\\varphi$ at $0$ with $\\varphi(0)=0$ gives us $\\xi\\in (0,\\rho]$ for which\n\\begin{align}\\label{defi xi 1}\n\\sqrt{2}\\norm{x-\\bar x}+\\frac{4\\sqrt{2}\\max\\set{\\nu+1,\\bar\\beta(\\nu+2)+\\bar\\delta}+8}{1-\\nu-(L  T+1)\\bar\\beta^2-2LT \\bar \\delta}\\varphi(f(x)-f(\\bar x))&<\\frac{\\rho}{4}\\;\\text{ for all }\\;x\\in\\mathbb{B}(\\bar x,\\xi).\n\\end{align}\nGiven the sequence of iterates $\\set{x^k}$ generated by Algorithm~\\ref{IGDm} with $x^0\\in \\mathbb{B}(\\bar x, \\xi)$ and $\\tau\\in (0, T]$, take the constant $\\alpha,C_1,C_2$ from \\eqref{C1C2} and deduce that\n\\begin{align}\\label{sele alpha}\n\\alpha=\\frac{(L\\tau+1)\\bar\\beta^2+1-\\nu}{4\\tau}\\ge \\frac{1-\\nu}{4T}\\ge \\varepsilon.\n\\end{align}\nSince $\\tau\\le T$, we get from the selection of $T$ in \\eqref{LT<1} the inequalities\n\\begin{align}\\label{ineq 1-nu}\n(L\\tau+1)\\bar \\beta^2+2L\\tau \\bar \\delta\\le (LT+1)\\bar \\beta^2+2LT \\bar \\delta <1-\\nu.\n\\end{align}\nwhich bring us to the relationships  \n\\begin{align}\\label{ineq 1/2}\n\\alpha=\\frac{(L\\tau+1)\\bar\\beta^2+1-\\nu}{4\\tau}\\le \\frac{2(1-\\nu)-2L\\tau \\bar \\delta}{4\\tau}=\\frac{1-\\nu-L\\tau \\bar \\delta}{2\\tau}\\le \\frac{1}{2\\tau}.\n\\end{align} \nIt follows from $L\\tau \\le LT\\le 1$ by \\eqref{LT<1} and the estimates in \\eqref{ineq 1/2}, \\eqref{ineq 1-nu} that\n\\begin{align*}\n\\frac{C_2}{C_1}&= \\frac{\\sqrt{2}\\max\\set{\\frac{\\nu+1}{\\tau },\\frac{\\bar\\beta(L\\tau+\\nu+1)+L\\tau \\bar \\delta}{\\tau }}+4\\alpha}{\\frac{1-\\nu-(L \\tau+1)\\bar \\beta^2-2L\\tau \\bar \\delta}{4\\tau}}\\\\\n&= \\frac{4\\sqrt{2}\\max\\set{{\\nu+1}, \\bar\\beta(L\\tau+\\nu+1)+L\\tau \\bar \\delta}+16\\alpha\\tau}{1-\\nu-(L \\tau+1)\\bar\\beta^2-2L\\tau \\bar \\delta}\\\\\n&\\le \\frac{4\\sqrt{2}\\max\\set{\\nu+1,\\bar\\beta(\\nu+2)+\\bar\\delta}+8}{1-\\nu-(L  T+1)\\bar\\beta^2-2LT \\bar \\delta}.\n\\end{align*}\nThis implies by \\eqref{defi xi 1} that \n\\begin{align}\\label{defi xi 2}\n\\sqrt{2}\\norm{x-\\bar x}+\\frac{C_2}{C_1}\\varphi(f(x)-f(\\bar x))&<\\frac{\\rho}{4} \\text{ for all }x\\in\\mathbb{B}(\\bar x,\\xi).\n\\end{align}\nThe rest of the proof of assertion (i) is split into two claims.\n\n\\begin{Claim} The sequences $\\set{x^k},\\set{x^k_{\\rm in}}$, and $\\set{x^k_{\\rm ex}}$ stay inside $\\mathbb{B}(\\bar x,\\rho)$ for all $k\\in\\N$.\n\\end{Claim}\n\\noindent Since $x^0=x^1\\in \\mathbb{B}(\\bar x,\\xi),$ the updates in \\eqref{step xin}, \\eqref{step xex} of Algorithm~\\ref{IGDm} tell us that $x^1_{\\rm in}=x^1_{\\rm ex}=x^1\\in \\mathbb{B}(\\bar x,\\xi)\\subset \\mathbb{B}(\\bar x,\\rho).$ We proceed now by induction to show that $x^k,x^k_{\\rm in},x^k_{\\rm ex}\\in \\mathbb{B}(\\bar x,\\rho)$ for all $k\\in\\N.$ Assuming that $x^k,x^k_\\rin,x^k_{\\rm ex}\\in \\mathbb{B}(\\bar x,\\rho)$ for $k=1,\\ldots,K$, let us verify that $x^{K+1},x_\\rin^{K+1},x^{K+1}_\\rex\\in \\mathbb{B}(\\bar x,\\rho).$ It follows from \\eqref{ineq 1-nu} and $L\\tau\\le LT<1-\\nu$ by \\eqref{LT<1} that assumption \\eqref{max Lt<1-nu} in Proposition~\\ref{prop AIGD tech} is satisfied. Therefore, we get the estimates\n\\begin{align}\\label{descent con for rate 1}\n&C_1 \\norm{z^{k+1}-z^k}^2\\le H_{\\alpha}(z^k)-H_{\\alpha}(z^{k+1}),\\\\\n&\\norm{\\nabla H_{\\alpha}(z^{k})}\\le C_2\\norm{z^{k+1}-z^k}\\text{ for all }k=1,\\ldots,K-1,\\label{descent con for rate 2}\n\\end{align}\nwhere $z^k=(x^k,x^{k-1})$ as before. Fix any such $k,$ we deduce from the above inequalities that\n\\begin{align}\\label{primary descent condi}\n\\norm{z^{k+1}-z^k}\\cdot\\norm{\\nabla H_\\alpha(z^k)}\\le C_2  \\norm{z^{k+1}-z^k}^2\\le \\frac{C_2}{C_1}(H_\\alpha(z^k)-H_\\alpha(z^{k+1})).\n\\end{align}\nTo use inequality \\eqref{KL R2n in proof} with $z:=z^k$, it is sufficient to check that $z^k\\in Z$ and $ H_\\alpha(\\bar z)<H_\\alpha(z^k)<H_\\alpha(\\bar z)+\\eta$. Note first that the former condition is satisfied due to\n\\begin{align*}\nz^k=(x^k,x^{k-1})\\in \\mathbb{B}(\\bar x,\\rho)\\times \\mathbb{B}(\\bar x,\\rho)\\subset Z.\n\\end{align*}\n To verify the latter condition, deduce from Proposition~\\ref{proposition distance} that $\\norm{x^k-x^{k-1}}\\le \\frac{L\\tau\\rho}{(1-\\bar \\beta)(1-\\nu)}$. Combining this with the construction of $H_\\alpha$, the inequality $\\alpha\\le \\frac{1}{2\\tau}$ from \\eqref{ineq 1/2}, the algorithm inclusions $x^k,x^{k-1}\\in\\mathbb{B}(\\bar x,\\rho)$, and the choice of $\\rho$ in \\eqref{choice rho} tells us that\n\\begin{align*}\nH_\\alpha(z^k)&=f(x^k)+\\alpha\\norm{x^k-x^{k-1}}^2\\\\\n&\\le f(x^k)+\\frac{1}{2\\tau}\\cdot\\frac{L^2\\tau^2\\rho^2}{(1-\\bar \\beta)^2(1-\\nu)^2}\\\\\n&= f(x^k)+\\frac{L^2T\\rho^2}{2(1-\\bar \\beta)^2(1-\\nu)^2}< f(\\bar x)+\\eta=H_\\alpha(\\bar z)+\\eta.\n\\end{align*}\nObserve that the inequality $H_\\alpha(\\bar z)<H_\\alpha(z^k)$ is also satisfied. Indeed, its failure means that \n$$\nf(x^k)+\\alpha\\norm{x^{k}-x^{k-1}}^2\\le H_\\alpha(\\bar z)=f(\\bar x).\n$$\nBy taking into account that $f(x^k)\\ge f(\\bar x)$ by $x^k\\in\\mathbb{B}(\\bar x,\\rho)$, the latter implies that\n\\begin{align*}\nf(x^k)=f(\\bar x) \\text{ and }x^{k}=x^{k-1},\n\\end{align*}\nwhich  means that $\\nabla f(x^k)=0$ since $\\bar x$ is a local minimizer of $f$ within $\\B(\\bar x,\\rho)$. The equalities above also imply that\n$x^k_{\\rm ex}=x^k$ by \\eqref{step xex}, and so $\\nabla f(x^k_{\\rm ex})=0,$ which contradicts Assumption~\\ref{assu} and thus justifies \\eqref{KL R2n in proof} with $z=z^k$. Multiplying both sides of \\eqref{primary descent condi} by $\\varphi'(H_\\alpha(z^k)-H_\\alpha(\\bar z))$ and using \\eqref{KL R2n in proof} give us\n\\begin{align}\\label{ineq alter}\n\\norm{z^{k+1}-z^k}& \\le \\varphi'(H_\\alpha(z^k)-H_\\alpha(\\bar z))\\norm{\\nabla H_\\alpha(z^k)}\\cdot\\norm{z^{k+1}-z^k}\\\\\n&\\le \\frac{C_2}{C_1}\\varphi'(H_\\alpha(z^k)-H_\\alpha(\\bar z))\\brac{ H_{\\alpha}(z^k)-H_{\\alpha}(z^{k+1})}\\nonumber\\\\\n&\\le\\frac{C_2}{C_1}\\brac{\\varphi(H_\\alpha(z^k)-H_\\alpha(\\bar z))-\\varphi(H_\\alpha(z^{k+1})-H_\\alpha(\\bar z))} \\text{ for all }k=1,\\ldots,K-1,\n\\end{align}\nwhere the inequality  in \\eqref{chain ineq} follows from the concavity of $\\varphi.$\nCombining further the obtained estimate with $\\tau\\le T<\\frac{(1-\\bar \\beta)(1-\\nu)}{4L\\sqrt{2}}$ from \\eqref{LT<1} and estimate \\eqref{3.3(ii)} from Proposition~\\ref{proposition distance},  we have\n\\begin{align}\n\\norm{z^{K+1}-\\bar z}&\\le\\norm{z^1-\\bar z}+\\norm{z^{K+1}-z^K}+\\sum_{k=1}^{K-1}\\norm{z^{k+1}-z^k}\\nonumber\\\\\n&\\le \\norm{z^1-\\bar z}+\\norm{z^{K+1}-z^K}+\\frac{C_2}{C_1}\\sum_{k=1}^{K-1}\\brac{\\varphi(H_\\alpha(z^k)-H_\\alpha(\\bar z))-\\varphi(H_\\alpha(z^{k+1})-H_\\alpha(\\bar z))}\\nonumber\\\\\n&=\\sqrt{2}\\norm{x^1-\\bar x}+\\frac{\\sqrt{2}L\\rho\\tau}{(1-\\bar \\beta)(1-\\nu)}+\\frac{C_2}{C_1}\\brac{\\varphi(H_\\alpha(z^1)-H_\\alpha(\\bar z))-\\varphi(H_\\alpha(z^{K})-H_\\alpha(\\bar z))}\\nonumber\\\\\n&\\le \\sqrt{2}\\norm{x^1-\\bar x}+\\frac{\\rho}{4}+\\frac{C_2}{C_1}\\varphi(f(x^1)-f(\\bar x))<\\rho/2,\\label{chain ineq}\n\\end{align}\nwhere the last inequality is a consequence of \\eqref{defi xi 2} and $x^1\\in\\B(\\bar x,\\xi)$. \nTherefore, \n\\begin{align*}\n\\norm{x^{K+1}-\\bar x}\\le \\norm{z^{K+1}-\\bar z}\\le \\rho/2,\n\\end{align*}\nwhich means that $x^{K+1}\\in\\mathbb{B}(\\bar x,\\rho/2)$. Using the updates in \\eqref{step xin} and \\eqref{step xex} at the $(K+1)^{\\rm th}$ iteration with taking into account \\eqref{3.3(ii)} from Proposition~\\ref{proposition distance}, $\\tau\\le T<\\frac{(1-\\bar \\beta)(1-\\nu)}{4L\\sqrt{2}(1+\\bar\\delta)}$ from \\eqref{LT<1}, and  \n\\begin{align*}\n0\\le \\beta_{K+1}<1,\\quad 0\\le \\delta_{K+1}\\le \\delta_{K+1}-\\beta_{K+1}+\\beta_{K+1}\\le \\bar \\delta +1,\n\\end{align*}\nwe arrive at the estimates\n\\begin{align*}\n\\norm{x^{K+1}_\\rin-\\bar x}&\\le \\norm{x^{K+1}-\\bar x}+\\norm{x^{K+1}-x^K}\\le \\rho/2+ \\frac{2L\\tau\\rho}{(1-\\bar \\beta)(1-\\nu)}<\\rho,\\\\\n\\norm{x^{K+1}_\\rex-\\bar x}&\\le \\norm{x^{K+1}-\\bar x}+(1+\\bar \\delta)\\norm{x^{K+1}-x^K}\\le \\rho/2+ \\frac{2L\\tau\\rho(1+\\bar\\delta)}{(1-\\bar \\beta)(1-\\nu)}<\\rho.\n\\end{align*}\nThus $x^{K+1}_\\rin,x^{K+1}_\\rex\\in\\mathbb{B}(\\bar x,\\rho)$, which completes the claimed assertion by induction. \n\\medskip\n\n\\noindent\\textbf{Claim~2.} \\textit{The sequences $\\set{x^k},\\set{x^k_{\\rm in}},\\set{x^k_{\\rm ex}}$ converge to a local minimizer $\\tilde x$ of $f$ with $f(\\tilde x)=f(\\bar x)$.}\n\\medskip\n\n\\noindent Indeed, it follows from \\eqref{ineq alter} and $x^k,x^k_\\rin,x^k_{\\rm ex}\\in\\mathbb{B}(\\bar x,\\rho)$ from Claim 1 that \n\\begin{align*}\n\\norm{z^{k+1}-z^k}&\\le \\frac{C_2}{C_1}\\brac{\\varphi(H_\\alpha(z^k)-H_\\alpha(\\bar z))-\\varphi(H_\\alpha(z^{k+1})-H_\\alpha(\\bar z))} \\text{ for all }k\\in\\N.\n\\end{align*}\nThis readily implies that \n\\begin{align*}\n\\sum_{k=1}^\\infty \\norm{z^{k+1}-z^k}\\le \\frac{C_2}{C_1}\\varphi(H_\\alpha(z^1)-H_\\alpha(\\bar z))<\\infty.\n\\end{align*}\nTherefore, $\\set{z^k}$ is convergent, and thus $\\set{x^k}$ converges to some $\\tilde x\\in\\overline{\\mathbb{B}}(\\bar x,\\rho)\\subset U$. Using  Proposition~\\ref{convergence xk+1 xk}, we justify the convergence of $\\set{x^k_\\rex},\\set{x^k_\\rin}$ to $\\tilde x$ and that $\\tilde x$ is a stationary point of $f.$ Since $f(\\bar x)$ is the only critical value of $f$ within $U,$ it gives us $f(\\tilde x)=f(\\bar x)$ and shows that $\\tilde x$ is a local minimizer of $f$.\\vspace*{0.05in} \n\nTo verify (ii), we employ Proposition~\\ref{general rate}. Letting $\\tilde z:=(\\tilde x,\\tilde x)$ and using \\eqref{descent con for rate 1}, \\eqref{descent con for rate 2} from the proof of Claim~1 in (i) together with $\\set{x^k},\\set{x^k_{\\rm in}},\\set{x^k_{\\rm ex}}\\subset\\mathbb{B}(\\bar x,\\rho)$ give us the estimate\n\\begin{align}\\label{ineq rate grad}\n\\begin{array}{ll}\nC_1 \\norm{z^{k+1}-z^k}^2\\le H_{\\alpha}(z^k)-H_{\\alpha}(z^{k+1}),\\\\\n\\norm{\\nabla H_{\\alpha}(z^{k})}\\le C_2\\norm{z^{k+1}-z^k},\\quad k\\in\\N.\n\\end{array}\n\\end{align}\nObserve from $\\nabla f(x^k_\\rex)\\ne 0$ and $\\norm{g^k-\\nabla f(x^k_\\rex)}\\le \\nu \\norm{g^k}$ that $z^{k+1}\\ne z^k$ for all $k\\in\\N.$ Indeed, the contrary implies by $z^k=(x^{k},x^{k-1})$ that $x^{k+1}=x^k=x^{k-1}$ for some $k\\in\\N$. Using the updates in \\eqref{step xin}, \\eqref{step xex} of Algorithm~\\ref{IGDm} ensures that $x^k_{\\rm ex}=x^k_{\\rm in}=x^k=x^{k+1}$. Then $g^k=0$ by \\eqref{iteration update}, and hence $\\nabla f(x^k_{\\rm ex})=0$ by $\\norm{g^k-\\nabla f(x^k_\\rex)}\\le \\nu \\norm{g^k}$. This contradicts the assumption $\\nabla f(x^k_{\\rm ex})\\ne 0$ for all $k\\in\\N$.\n\nRecalling that $f$ satisfies the PLK condition at $\\bar x$ with the desingularizing function $\\psi(t)=Mt^{1-q}$, we deduce from Proposition~\\ref{isolated}(ii) that $H_\\alpha$ satisfies the PLK condition at $(\\bar x,\\bar x)$ with the neighborhood $Z$ taking from \\eqref{KL R2n in proof} and the desingularizing function $\\varphi(t)=\\overline Mt^{1-\\theta}$, where $\\theta=\\max \\set{q,\\frac{1}{2}}\\in (0,1),\\overline{M}>0.$ Taking into account Remark~\\ref{algebraic}, this PLK property also holds at $\\tilde z\\in\\overline{\\mathbb{B}}(\\bar x,\\rho)\\times \\overline{\\mathbb{B}}(\\bar x,\\rho)\\subset Z$ due to $H_\\alpha(\\tilde z)=f(\\tilde x)=f(\\bar x)=H_\\alpha(\\bar z).$  Then Proposition~\\ref{general rate} applied to the function $H_\\alpha$ and sequence $\\set{z^k}\\subset\\R^{n}\\times \\R^n$, with taking into account Remark~\\ref{rmk general rate}, brings us to the following:\n\\begin{itemize}\n\\item For $q\\in[0,1/2],$ we get $\\theta=1/2$, which ensures that $\\set{z^k}$ converges linearly to $(\\tilde x,\\tilde x)$. This implies that $\\set{x^k}$ converges linearly to $\\tilde x.$\n\n\\item For $q\\in (1/2,1),$ we get $\\theta=q\\in (1/2,1)$, which tells us that $\\norm{z^k-(\\tilde x,\\tilde x)}=\\mathcal{O}(k^{-\\frac{1-q}{2q-1}})$ and thus yields the convergence rate $\\norm{x^k-\\tilde x}=\\mathcal{O}(k^{-\\frac{1-q}{2q-1}})$ as $k\\rightarrow\\infty$.\n\\end{itemize}\n\nFinally, we employ Proposition~\\ref{convergence rate deduce} to verify the convergence rates of $\\set{\\nabla f(x^k)}$ and $\\set{f(x^k)}.$ Since $f$ is of class ${\\cal C}^{1,1}$ on $\\B(\\bar x,r)$ and since $H_\\alpha$ is of class ${\\cal C}^{1,1}$ on $\\B(\\bar x,r)\\times \\B(\\bar x,r)\\supset \\set{z^k}$, condition \\eqref{main condition rate} follows from \\eqref{ineq rate grad}. This clearly leads us to the conclusions:\n\\begin{itemize}\n\\item For $q\\in[0,1/2),$ we get $\\theta=1/2$. Therefore, $\\{H_\\alpha(z^k)\\}$ converges linearly to $H_\\alpha(\\bar z)$ and $\\{\\norm{\\nabla H_\\alpha(z^k)}\\}$ converges linearly to $0$ as $k\\to\\infty$.\n\n\\item For $q\\in (1/2,1),$ we have $H_\\alpha(z^k)-H_\\alpha(\\tilde z)=\\mathcal{O}(k^{-\\frac{2-2q}{2q-1}})$ and $\\norm{\\nabla H_\\alpha(z^k)}=\\mathcal{O}(k^{-\\frac{1-q}{2q-1}})$ as $k\\to\\infty$.\n\\end{itemize}\nThe claimed convergence rates of the sequences $\\{\\norm{\\nabla f(x^k)}\\}$ and $\\set{f(x^k)}$ are now deduced from those for $\\set{\\norm{\\nabla H_\\alpha (z^k)}},\\set{H_\\alpha (z^k)}$ and the estimates\n\\begin{align*}\n\\norm{\\nabla H_\\alpha(x,y)}&=\\sqrt{\\norm{\\nabla f(x)+2\\alpha (x-y)}^2+4\\alpha^2\\norm{x-y}^2}\\ge \\norm{\\nabla f(x)},\\\\\nH_\\alpha(x,y)&=f(x)+\\alpha\\norm{x-y}^2\\ge f(x)\\;\\text{ for all }\\;x,y\\in\\R^n.\n\\end{align*}\nThis completes the proof of the theorem.\n\\end{proof}","sha256_utf8":"90d6228cf14513e8620ca3ff4fe20ee0190ccca50edc6a4c0e5cbcf423d0d60f"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"IGDm局部收敛与速率","source_units":[{"start":61301,"end":62705,"start_line":722,"end_line":736,"raw_tex":"\\begin{Theorem}\\label{local con IGDm}\nLet the objective function $f:X\\rightarrow\\R$ in \\eqref{main problem} satisfy the basic PLK condition at its local minimizer $\\bar x$, and let $\\set{x^k}$ be the sequence of iterates generated by Algorithm~{\\rm\\ref{IGDm}} with $X=\\B(\\bar x,r)$ for some $r>0$ and $\\bar \\beta\\in [0,\\sqrt{1-\\nu})$. Then there exist $\\xi \\in(0,r]$ and $T>0$ such that for any initial point $x^0\\in \\mathbb{B}(\\bar x, \\xi)$ and $\\tau\\in (0, T]$, we have the convergence properties$:$\n\\begin{enumerate}\n\\item[\\bf(i)] The sequences $\\set{x^k},\\set{x^k_{\\rm in}},\\set{x^k_{\\rm ex}}$ stay in $\\mathbb{B}(\\bar x,r)$ and converge to some local minimizer $\\tilde x\\in\\B(\\bar x,r)$ of $f$ with $f(\\tilde x)=f(\\bar x)$.\n\n\\item[\\bf(ii)] If the exponential PLK condition is satisfied with the desingularizing function $\\psi(t)= Mt^{1-q}$ for some $M>0$ and $q\\in(0,1)$, then the following convergence rates are guaranteed as $k\\to\\infty:$\n\\end{enumerate}\\vspace*{-0.1in}\n\n $\\bullet$ For $q=1/2$, the sequences $\\set{x^k},\\set{\\nabla f(x^k)},\\set{f(x^k)}$ converge linearly to $\\tilde x,0,f(\\tilde x),$ respectively.\n\n $\\bullet$ For $q\\in(1/2,1)$, we have the asymptotic estimates\n $$\n \\norm{x^k-\\tilde x }=\\mathcal{O}\\brac{k^{-\\frac{1-q}{2q-1}}},\\quad\\norm{\\nabla f(x^k)}=\\mathcal{O}\\brac{k^{-\\frac{1-q}{2q-1}}},\\quad f(x^k)-f(\\tilde x)=\\mathcal{O}\\brac{k^{-\\frac{2-2q}{2q-1}}}.\n $$\n \\end{Theorem}","sha256_utf8":"d3b4c48fb3fd270179056599ea04b32873b3b26c84712568a5f47d134fab8900"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"局部收敛完整证明","source_units":[{"start":62706,"end":76838,"start_line":737,"end_line":879,"raw_tex":"\\begin{proof} To verify (i), deduce from $\\bar \\beta \\in[0,\\sqrt{1-\\nu})$ that there exists $T>0$ such that\n\\begin{align}\\label{LT<1}\nT< \\frac{(1-\\bar \\beta)(1-\\nu)}{4L\\sqrt{2}(1+\\bar \\delta)},\\quad(LT+1)\\bar\\beta^2+2LT\\bar \\delta<1-\\nu,\\quad LT\\le 1-\\nu.\n\\end{align}\nDefine further $\\varepsilon:=\\min\\set{\\frac{1-\\nu}{4T},\\frac{1}{4}}\\in \\big(0,\\frac{1}{4}\\big]$ and deduce from Lemma~\\ref{isolation lemma} due to the imposed assumptions that there exists a neighborhood $U\\subset\\B(\\bar x,r)$ of $\\bar x$ such that $\\bar x$ minimizes $f$ on $U$ and that $f(\\bar x)$ is the only critical value within this set. Remembering that $f$ satisfies the basic PLK condition at $\\bar x$ tells us by Proposition~\\ref{isolated} that the Lyapunov function $\\set{H_\\alpha}_{\\alpha\\ge \\varepsilon}$ enjoys the same PLK property at $\\bar z:=(\\bar x,\\bar x)$ for some $\\eta\\in (0,\\infty]$, some neighborhood $Z$ of $\\bar z:=(\\bar x,\\bar x)$, and a concave desingularizing function $\\varphi:[0,\\eta)\\rightarrow[0,\\infty)$ with $\\varphi(0)=0$ which is ${\\cal C}^1$-smooth on $(0,\\eta)$. This means that for any $\\alpha\\ge \\varepsilon$ and any $z\\in Z$ with $H_\\alpha(\\bar z)< H_\\alpha( z)<H_\\alpha(\\bar z)+\\eta$, we get the inequality\n\\begin{align}\\label{KL R2n in proof}\n\\varphi'\\big(H_\\alpha(z )-H_\\alpha(\\bar z )\\big)\\norm{\\nabla H_\\alpha(z)}\\ge 1.\n\\end{align}\nThe continuity of $f$ allows us to find $\\rho\\in(0,r)$ such that \n\\begin{align}\\label{choice rho}\nf(x)+\\frac{L^2T\\rho^2}{2(1-\\bar \\beta)^2(1-\\nu)^2}< f(\\bar x)+\\eta\\text{ for all }x \\in \\mathbb{B}(\\bar x,\\rho).\n\\end{align}\nInvoking further the continuity of $\\varphi$ at $0$ with $\\varphi(0)=0$ gives us $\\xi\\in (0,\\rho]$ for which\n\\begin{align}\\label{defi xi 1}\n\\sqrt{2}\\norm{x-\\bar x}+\\frac{4\\sqrt{2}\\max\\set{\\nu+1,\\bar\\beta(\\nu+2)+\\bar\\delta}+8}{1-\\nu-(L  T+1)\\bar\\beta^2-2LT \\bar \\delta}\\varphi(f(x)-f(\\bar x))&<\\frac{\\rho}{4}\\;\\text{ for all }\\;x\\in\\mathbb{B}(\\bar x,\\xi).\n\\end{align}\nGiven the sequence of iterates $\\set{x^k}$ generated by Algorithm~\\ref{IGDm} with $x^0\\in \\mathbb{B}(\\bar x, \\xi)$ and $\\tau\\in (0, T]$, take the constant $\\alpha,C_1,C_2$ from \\eqref{C1C2} and deduce that\n\\begin{align}\\label{sele alpha}\n\\alpha=\\frac{(L\\tau+1)\\bar\\beta^2+1-\\nu}{4\\tau}\\ge \\frac{1-\\nu}{4T}\\ge \\varepsilon.\n\\end{align}\nSince $\\tau\\le T$, we get from the selection of $T$ in \\eqref{LT<1} the inequalities\n\\begin{align}\\label{ineq 1-nu}\n(L\\tau+1)\\bar \\beta^2+2L\\tau \\bar \\delta\\le (LT+1)\\bar \\beta^2+2LT \\bar \\delta <1-\\nu.\n\\end{align}\nwhich bring us to the relationships  \n\\begin{align}\\label{ineq 1/2}\n\\alpha=\\frac{(L\\tau+1)\\bar\\beta^2+1-\\nu}{4\\tau}\\le \\frac{2(1-\\nu)-2L\\tau \\bar \\delta}{4\\tau}=\\frac{1-\\nu-L\\tau \\bar \\delta}{2\\tau}\\le \\frac{1}{2\\tau}.\n\\end{align} \nIt follows from $L\\tau \\le LT\\le 1$ by \\eqref{LT<1} and the estimates in \\eqref{ineq 1/2}, \\eqref{ineq 1-nu} that\n\\begin{align*}\n\\frac{C_2}{C_1}&= \\frac{\\sqrt{2}\\max\\set{\\frac{\\nu+1}{\\tau },\\frac{\\bar\\beta(L\\tau+\\nu+1)+L\\tau \\bar \\delta}{\\tau }}+4\\alpha}{\\frac{1-\\nu-(L \\tau+1)\\bar \\beta^2-2L\\tau \\bar \\delta}{4\\tau}}\\\\\n&= \\frac{4\\sqrt{2}\\max\\set{{\\nu+1}, \\bar\\beta(L\\tau+\\nu+1)+L\\tau \\bar \\delta}+16\\alpha\\tau}{1-\\nu-(L \\tau+1)\\bar\\beta^2-2L\\tau \\bar \\delta}\\\\\n&\\le \\frac{4\\sqrt{2}\\max\\set{\\nu+1,\\bar\\beta(\\nu+2)+\\bar\\delta}+8}{1-\\nu-(L  T+1)\\bar\\beta^2-2LT \\bar \\delta}.\n\\end{align*}\nThis implies by \\eqref{defi xi 1} that \n\\begin{align}\\label{defi xi 2}\n\\sqrt{2}\\norm{x-\\bar x}+\\frac{C_2}{C_1}\\varphi(f(x)-f(\\bar x))&<\\frac{\\rho}{4} \\text{ for all }x\\in\\mathbb{B}(\\bar x,\\xi).\n\\end{align}\nThe rest of the proof of assertion (i) is split into two claims.\n\n\\begin{Claim} The sequences $\\set{x^k},\\set{x^k_{\\rm in}}$, and $\\set{x^k_{\\rm ex}}$ stay inside $\\mathbb{B}(\\bar x,\\rho)$ for all $k\\in\\N$.\n\\end{Claim}\n\\noindent Since $x^0=x^1\\in \\mathbb{B}(\\bar x,\\xi),$ the updates in \\eqref{step xin}, \\eqref{step xex} of Algorithm~\\ref{IGDm} tell us that $x^1_{\\rm in}=x^1_{\\rm ex}=x^1\\in \\mathbb{B}(\\bar x,\\xi)\\subset \\mathbb{B}(\\bar x,\\rho).$ We proceed now by induction to show that $x^k,x^k_{\\rm in},x^k_{\\rm ex}\\in \\mathbb{B}(\\bar x,\\rho)$ for all $k\\in\\N.$ Assuming that $x^k,x^k_\\rin,x^k_{\\rm ex}\\in \\mathbb{B}(\\bar x,\\rho)$ for $k=1,\\ldots,K$, let us verify that $x^{K+1},x_\\rin^{K+1},x^{K+1}_\\rex\\in \\mathbb{B}(\\bar x,\\rho).$ It follows from \\eqref{ineq 1-nu} and $L\\tau\\le LT<1-\\nu$ by \\eqref{LT<1} that assumption \\eqref{max Lt<1-nu} in Proposition~\\ref{prop AIGD tech} is satisfied. Therefore, we get the estimates\n\\begin{align}\\label{descent con for rate 1}\n&C_1 \\norm{z^{k+1}-z^k}^2\\le H_{\\alpha}(z^k)-H_{\\alpha}(z^{k+1}),\\\\\n&\\norm{\\nabla H_{\\alpha}(z^{k})}\\le C_2\\norm{z^{k+1}-z^k}\\text{ for all }k=1,\\ldots,K-1,\\label{descent con for rate 2}\n\\end{align}\nwhere $z^k=(x^k,x^{k-1})$ as before. Fix any such $k,$ we deduce from the above inequalities that\n\\begin{align}\\label{primary descent condi}\n\\norm{z^{k+1}-z^k}\\cdot\\norm{\\nabla H_\\alpha(z^k)}\\le C_2  \\norm{z^{k+1}-z^k}^2\\le \\frac{C_2}{C_1}(H_\\alpha(z^k)-H_\\alpha(z^{k+1})).\n\\end{align}\nTo use inequality \\eqref{KL R2n in proof} with $z:=z^k$, it is sufficient to check that $z^k\\in Z$ and $ H_\\alpha(\\bar z)<H_\\alpha(z^k)<H_\\alpha(\\bar z)+\\eta$. Note first that the former condition is satisfied due to\n\\begin{align*}\nz^k=(x^k,x^{k-1})\\in \\mathbb{B}(\\bar x,\\rho)\\times \\mathbb{B}(\\bar x,\\rho)\\subset Z.\n\\end{align*}\n To verify the latter condition, deduce from Proposition~\\ref{proposition distance} that $\\norm{x^k-x^{k-1}}\\le \\frac{L\\tau\\rho}{(1-\\bar \\beta)(1-\\nu)}$. Combining this with the construction of $H_\\alpha$, the inequality $\\alpha\\le \\frac{1}{2\\tau}$ from \\eqref{ineq 1/2}, the algorithm inclusions $x^k,x^{k-1}\\in\\mathbb{B}(\\bar x,\\rho)$, and the choice of $\\rho$ in \\eqref{choice rho} tells us that\n\\begin{align*}\nH_\\alpha(z^k)&=f(x^k)+\\alpha\\norm{x^k-x^{k-1}}^2\\\\\n&\\le f(x^k)+\\frac{1}{2\\tau}\\cdot\\frac{L^2\\tau^2\\rho^2}{(1-\\bar \\beta)^2(1-\\nu)^2}\\\\\n&= f(x^k)+\\frac{L^2T\\rho^2}{2(1-\\bar \\beta)^2(1-\\nu)^2}< f(\\bar x)+\\eta=H_\\alpha(\\bar z)+\\eta.\n\\end{align*}\nObserve that the inequality $H_\\alpha(\\bar z)<H_\\alpha(z^k)$ is also satisfied. Indeed, its failure means that \n$$\nf(x^k)+\\alpha\\norm{x^{k}-x^{k-1}}^2\\le H_\\alpha(\\bar z)=f(\\bar x).\n$$\nBy taking into account that $f(x^k)\\ge f(\\bar x)$ by $x^k\\in\\mathbb{B}(\\bar x,\\rho)$, the latter implies that\n\\begin{align*}\nf(x^k)=f(\\bar x) \\text{ and }x^{k}=x^{k-1},\n\\end{align*}\nwhich  means that $\\nabla f(x^k)=0$ since $\\bar x$ is a local minimizer of $f$ within $\\B(\\bar x,\\rho)$. The equalities above also imply that\n$x^k_{\\rm ex}=x^k$ by \\eqref{step xex}, and so $\\nabla f(x^k_{\\rm ex})=0,$ which contradicts Assumption~\\ref{assu} and thus justifies \\eqref{KL R2n in proof} with $z=z^k$. Multiplying both sides of \\eqref{primary descent condi} by $\\varphi'(H_\\alpha(z^k)-H_\\alpha(\\bar z))$ and using \\eqref{KL R2n in proof} give us\n\\begin{align}\\label{ineq alter}\n\\norm{z^{k+1}-z^k}& \\le \\varphi'(H_\\alpha(z^k)-H_\\alpha(\\bar z))\\norm{\\nabla H_\\alpha(z^k)}\\cdot\\norm{z^{k+1}-z^k}\\\\\n&\\le \\frac{C_2}{C_1}\\varphi'(H_\\alpha(z^k)-H_\\alpha(\\bar z))\\brac{ H_{\\alpha}(z^k)-H_{\\alpha}(z^{k+1})}\\nonumber\\\\\n&\\le\\frac{C_2}{C_1}\\brac{\\varphi(H_\\alpha(z^k)-H_\\alpha(\\bar z))-\\varphi(H_\\alpha(z^{k+1})-H_\\alpha(\\bar z))} \\text{ for all }k=1,\\ldots,K-1,\n\\end{align}\nwhere the inequality  in \\eqref{chain ineq} follows from the concavity of $\\varphi.$\nCombining further the obtained estimate with $\\tau\\le T<\\frac{(1-\\bar \\beta)(1-\\nu)}{4L\\sqrt{2}}$ from \\eqref{LT<1} and estimate \\eqref{3.3(ii)} from Proposition~\\ref{proposition distance},  we have\n\\begin{align}\n\\norm{z^{K+1}-\\bar z}&\\le\\norm{z^1-\\bar z}+\\norm{z^{K+1}-z^K}+\\sum_{k=1}^{K-1}\\norm{z^{k+1}-z^k}\\nonumber\\\\\n&\\le \\norm{z^1-\\bar z}+\\norm{z^{K+1}-z^K}+\\frac{C_2}{C_1}\\sum_{k=1}^{K-1}\\brac{\\varphi(H_\\alpha(z^k)-H_\\alpha(\\bar z))-\\varphi(H_\\alpha(z^{k+1})-H_\\alpha(\\bar z))}\\nonumber\\\\\n&=\\sqrt{2}\\norm{x^1-\\bar x}+\\frac{\\sqrt{2}L\\rho\\tau}{(1-\\bar \\beta)(1-\\nu)}+\\frac{C_2}{C_1}\\brac{\\varphi(H_\\alpha(z^1)-H_\\alpha(\\bar z))-\\varphi(H_\\alpha(z^{K})-H_\\alpha(\\bar z))}\\nonumber\\\\\n&\\le \\sqrt{2}\\norm{x^1-\\bar x}+\\frac{\\rho}{4}+\\frac{C_2}{C_1}\\varphi(f(x^1)-f(\\bar x))<\\rho/2,\\label{chain ineq}\n\\end{align}\nwhere the last inequality is a consequence of \\eqref{defi xi 2} and $x^1\\in\\B(\\bar x,\\xi)$. \nTherefore, \n\\begin{align*}\n\\norm{x^{K+1}-\\bar x}\\le \\norm{z^{K+1}-\\bar z}\\le \\rho/2,\n\\end{align*}\nwhich means that $x^{K+1}\\in\\mathbb{B}(\\bar x,\\rho/2)$. Using the updates in \\eqref{step xin} and \\eqref{step xex} at the $(K+1)^{\\rm th}$ iteration with taking into account \\eqref{3.3(ii)} from Proposition~\\ref{proposition distance}, $\\tau\\le T<\\frac{(1-\\bar \\beta)(1-\\nu)}{4L\\sqrt{2}(1+\\bar\\delta)}$ from \\eqref{LT<1}, and  \n\\begin{align*}\n0\\le \\beta_{K+1}<1,\\quad 0\\le \\delta_{K+1}\\le \\delta_{K+1}-\\beta_{K+1}+\\beta_{K+1}\\le \\bar \\delta +1,\n\\end{align*}\nwe arrive at the estimates\n\\begin{align*}\n\\norm{x^{K+1}_\\rin-\\bar x}&\\le \\norm{x^{K+1}-\\bar x}+\\norm{x^{K+1}-x^K}\\le \\rho/2+ \\frac{2L\\tau\\rho}{(1-\\bar \\beta)(1-\\nu)}<\\rho,\\\\\n\\norm{x^{K+1}_\\rex-\\bar x}&\\le \\norm{x^{K+1}-\\bar x}+(1+\\bar \\delta)\\norm{x^{K+1}-x^K}\\le \\rho/2+ \\frac{2L\\tau\\rho(1+\\bar\\delta)}{(1-\\bar \\beta)(1-\\nu)}<\\rho.\n\\end{align*}\nThus $x^{K+1}_\\rin,x^{K+1}_\\rex\\in\\mathbb{B}(\\bar x,\\rho)$, which completes the claimed assertion by induction. \n\\medskip\n\n\\noindent\\textbf{Claim~2.} \\textit{The sequences $\\set{x^k},\\set{x^k_{\\rm in}},\\set{x^k_{\\rm ex}}$ converge to a local minimizer $\\tilde x$ of $f$ with $f(\\tilde x)=f(\\bar x)$.}\n\\medskip\n\n\\noindent Indeed, it follows from \\eqref{ineq alter} and $x^k,x^k_\\rin,x^k_{\\rm ex}\\in\\mathbb{B}(\\bar x,\\rho)$ from Claim 1 that \n\\begin{align*}\n\\norm{z^{k+1}-z^k}&\\le \\frac{C_2}{C_1}\\brac{\\varphi(H_\\alpha(z^k)-H_\\alpha(\\bar z))-\\varphi(H_\\alpha(z^{k+1})-H_\\alpha(\\bar z))} \\text{ for all }k\\in\\N.\n\\end{align*}\nThis readily implies that \n\\begin{align*}\n\\sum_{k=1}^\\infty \\norm{z^{k+1}-z^k}\\le \\frac{C_2}{C_1}\\varphi(H_\\alpha(z^1)-H_\\alpha(\\bar z))<\\infty.\n\\end{align*}\nTherefore, $\\set{z^k}$ is convergent, and thus $\\set{x^k}$ converges to some $\\tilde x\\in\\overline{\\mathbb{B}}(\\bar x,\\rho)\\subset U$. Using  Proposition~\\ref{convergence xk+1 xk}, we justify the convergence of $\\set{x^k_\\rex},\\set{x^k_\\rin}$ to $\\tilde x$ and that $\\tilde x$ is a stationary point of $f.$ Since $f(\\bar x)$ is the only critical value of $f$ within $U,$ it gives us $f(\\tilde x)=f(\\bar x)$ and shows that $\\tilde x$ is a local minimizer of $f$.\\vspace*{0.05in} \n\nTo verify (ii), we employ Proposition~\\ref{general rate}. Letting $\\tilde z:=(\\tilde x,\\tilde x)$ and using \\eqref{descent con for rate 1}, \\eqref{descent con for rate 2} from the proof of Claim~1 in (i) together with $\\set{x^k},\\set{x^k_{\\rm in}},\\set{x^k_{\\rm ex}}\\subset\\mathbb{B}(\\bar x,\\rho)$ give us the estimate\n\\begin{align}\\label{ineq rate grad}\n\\begin{array}{ll}\nC_1 \\norm{z^{k+1}-z^k}^2\\le H_{\\alpha}(z^k)-H_{\\alpha}(z^{k+1}),\\\\\n\\norm{\\nabla H_{\\alpha}(z^{k})}\\le C_2\\norm{z^{k+1}-z^k},\\quad k\\in\\N.\n\\end{array}\n\\end{align}\nObserve from $\\nabla f(x^k_\\rex)\\ne 0$ and $\\norm{g^k-\\nabla f(x^k_\\rex)}\\le \\nu \\norm{g^k}$ that $z^{k+1}\\ne z^k$ for all $k\\in\\N.$ Indeed, the contrary implies by $z^k=(x^{k},x^{k-1})$ that $x^{k+1}=x^k=x^{k-1}$ for some $k\\in\\N$. Using the updates in \\eqref{step xin}, \\eqref{step xex} of Algorithm~\\ref{IGDm} ensures that $x^k_{\\rm ex}=x^k_{\\rm in}=x^k=x^{k+1}$. Then $g^k=0$ by \\eqref{iteration update}, and hence $\\nabla f(x^k_{\\rm ex})=0$ by $\\norm{g^k-\\nabla f(x^k_\\rex)}\\le \\nu \\norm{g^k}$. This contradicts the assumption $\\nabla f(x^k_{\\rm ex})\\ne 0$ for all $k\\in\\N$.\n\nRecalling that $f$ satisfies the PLK condition at $\\bar x$ with the desingularizing function $\\psi(t)=Mt^{1-q}$, we deduce from Proposition~\\ref{isolated}(ii) that $H_\\alpha$ satisfies the PLK condition at $(\\bar x,\\bar x)$ with the neighborhood $Z$ taking from \\eqref{KL R2n in proof} and the desingularizing function $\\varphi(t)=\\overline Mt^{1-\\theta}$, where $\\theta=\\max \\set{q,\\frac{1}{2}}\\in (0,1),\\overline{M}>0.$ Taking into account Remark~\\ref{algebraic}, this PLK property also holds at $\\tilde z\\in\\overline{\\mathbb{B}}(\\bar x,\\rho)\\times \\overline{\\mathbb{B}}(\\bar x,\\rho)\\subset Z$ due to $H_\\alpha(\\tilde z)=f(\\tilde x)=f(\\bar x)=H_\\alpha(\\bar z).$  Then Proposition~\\ref{general rate} applied to the function $H_\\alpha$ and sequence $\\set{z^k}\\subset\\R^{n}\\times \\R^n$, with taking into account Remark~\\ref{rmk general rate}, brings us to the following:\n\\begin{itemize}\n\\item For $q\\in[0,1/2],$ we get $\\theta=1/2$, which ensures that $\\set{z^k}$ converges linearly to $(\\tilde x,\\tilde x)$. This implies that $\\set{x^k}$ converges linearly to $\\tilde x.$\n\n\\item For $q\\in (1/2,1),$ we get $\\theta=q\\in (1/2,1)$, which tells us that $\\norm{z^k-(\\tilde x,\\tilde x)}=\\mathcal{O}(k^{-\\frac{1-q}{2q-1}})$ and thus yields the convergence rate $\\norm{x^k-\\tilde x}=\\mathcal{O}(k^{-\\frac{1-q}{2q-1}})$ as $k\\rightarrow\\infty$.\n\\end{itemize}\n\nFinally, we employ Proposition~\\ref{convergence rate deduce} to verify the convergence rates of $\\set{\\nabla f(x^k)}$ and $\\set{f(x^k)}.$ Since $f$ is of class ${\\cal C}^{1,1}$ on $\\B(\\bar x,r)$ and since $H_\\alpha$ is of class ${\\cal C}^{1,1}$ on $\\B(\\bar x,r)\\times \\B(\\bar x,r)\\supset \\set{z^k}$, condition \\eqref{main condition rate} follows from \\eqref{ineq rate grad}. This clearly leads us to the conclusions:\n\\begin{itemize}\n\\item For $q\\in[0,1/2),$ we get $\\theta=1/2$. Therefore, $\\{H_\\alpha(z^k)\\}$ converges linearly to $H_\\alpha(\\bar z)$ and $\\{\\norm{\\nabla H_\\alpha(z^k)}\\}$ converges linearly to $0$ as $k\\to\\infty$.\n\n\\item For $q\\in (1/2,1),$ we have $H_\\alpha(z^k)-H_\\alpha(\\tilde z)=\\mathcal{O}(k^{-\\frac{2-2q}{2q-1}})$ and $\\norm{\\nabla H_\\alpha(z^k)}=\\mathcal{O}(k^{-\\frac{1-q}{2q-1}})$ as $k\\to\\infty$.\n\\end{itemize}\nThe claimed convergence rates of the sequences $\\{\\norm{\\nabla f(x^k)}\\}$ and $\\set{f(x^k)}$ are now deduced from those for $\\set{\\norm{\\nabla H_\\alpha (z^k)}},\\set{H_\\alpha (z^k)}$ and the estimates\n\\begin{align*}\n\\norm{\\nabla H_\\alpha(x,y)}&=\\sqrt{\\norm{\\nabla f(x)+2\\alpha (x-y)}^2+4\\alpha^2\\norm{x-y}^2}\\ge \\norm{\\nabla f(x)},\\\\\nH_\\alpha(x,y)&=f(x)+\\alpha\\norm{x-y}^2\\ge f(x)\\;\\text{ for all }\\;x,y\\in\\R^n.\n\\end{align*}\nThis completes the proof of the theorem.\n\\end{proof}","sha256_utf8":"90d6228cf14513e8620ca3ff4fe20ee0190ccca50edc6a4c0e5cbcf423d0d60f"}]},{"id":"S003","semantic_type":"problem_setting","kind":"problem_setting","title":"光滑无约束优化问题","source_units":[{"start":6615,"end":7192,"start_line":172,"end_line":179,"raw_tex":"This paper addresses the design and convergence analysis of first-order optimization algorithms with momentum for solving unconstrained optimization problems of the type\n\\begin{align}\\label{main problem}\n{\\rm minimize}\\quad f(x)\\quad \\text{ subject to }x\\in\\R^n,\n\\end{align}\nwhere $f:\\R^n\\rightarrow\\overline{\\R}$ is a continuously differentiable function on some open and convex subset $X$ of $\\R^n$ having a Lipschitz continuous gradient with constant $L>0$ on $X$, i.e.,\n\\begin{align*}\n\\norm{\\nabla f(x)-\\nabla f(y)}\\le L\\norm{x-y}\\;\\text{ for all }\\;x,y\\in X.\n\\end{align*}\n","sha256_utf8":"47d52bfd3761fd0005119ec329a1dd0e3f9183d935730ed6e95f5b8699345bb2"}]},{"id":"S004","semantic_type":"notation","kind":"notation","title":"邻域驻点与光滑类别","source_units":[{"start":18460,"end":19825,"start_line":230,"end_line":235,"raw_tex":"As always, $\\N:=\\{1,2,\\ldots\\}$ signifies the collections of natural numbers. Given $r>0$, the open ball centered at $\\bar x$ with radius $r$ is denoted by $\\mathbb{B}(\\bar x,\\rho)$ and its closure is $\\overline{\\mathbb{B}}(\\bar x,\\rho)$.\nThe symbol $x^k\\st{J}{\\to}\\ox$ means that $x^k\\to\\ox$ as $k\\to\\infty$ with $k\\in J\\subset\\N$. Recall that $\\bar x$ is a \\textit{stationary point} and that $f(\\bar x)$ is a \\textit{critical value} of a $\\mathcal{C}^1$-smooth function $f\\colon\\R^n\\rightarrow\\R$ if $\\nabla f(\\bar x)=0$. A $\\C^1-$smooth function $f:\\Omega\\rightarrow {\\R}$ is said to have a Lipschitz continuous gradient with the uniform constant $L>0$ on some subset $\\Omega$ of $\\R^n$ if \n\\begin{align}\\label{Lips def}\n\\norm{\\nabla f(x)-\\nabla f(y)}\\le L\\norm{x-y}\\;\\text{ for all }\\;x,y\\in\\Omega.\n\\end{align}\nA function $f\\colon\\R^n\\rightarrow\\R$ is {\\em locally Lipschitz continuous} around $\\bar x$  if there is a neighborhood $\\Omega$ of $\\bar x$ such that \\eqref{Lips def} holds; $f$ {\\em globally Lipschitz continuous} if \\eqref{Lips def} holds for $\\Omega=\\R^n$. In what follows, we denote by $\\mathcal{C}^{1,1}$ the class of smooth functions that have a {\\em locally Lipschitz continuous gradient} around $\\bar x$, and by $\\mathcal{C}^{1,1}_L$ the class of smooth functions that have a {\\em globally Lipschitz continuous gradient with constant} $L>0$.\n","sha256_utf8":"b73dc168e60c3204b4188db185bff1565cac339c7ae3abe2bc381444ac3e1c12"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"次微分与驻点定义","source_units":[{"start":19826,"end":21375,"start_line":237,"end_line":246,"raw_tex":"Next we recall the subdifferential constructions for proper extended-real-valued functions $h:\\R^n\\rightarrow\\Bar\\R:=(-\\infty,\\infty]$ with $\\dom h:=\\{x\\in\\R^n\\;|\\;h(x)<\\infty\\}\\ne\\emp$ taken from \\cite{Mordukhovich2006,Mordukhovich2018,RockafellarWets} and used below. The (Fr\\'echet) \\textit{regular subdifferential} of $ h $ at $\\ox \\in\\dom  h $ is \n\\begin{align}\\label{frechet}\n\\widehat{\\partial} h (\\bar x):=\\Big\\{v\\in \\R^n\\;\\Big|\\;\\liminf_{x\\rightarrow\\bar x}\\dfrac{ h (x)- h (\\bar x)-\\dotproduct{v,x-\\bar x}}{\\norm{x-\\bar x}}\\ge 0\\Big\\},\n\\end{align}\nand the (Mordukhovich) \\textit{limiting subdifferential} of $ h $ at $\\bar x\\in\\dom  h $ is defined by \n\\begin{align}\\label{limiting subdiff}\n\\partial  h (\\bar x):=\\set{v\\in\\R^n\\;\\big|\\;\\exists\\,x^k\\rightarrow \\ox,\\  h (x^k)\\rightarrow h (\\bar x),\\ \\widehat{\\partial}h(x^k)\\ni v^k\\rightarrow v}.\n\\end{align}\nWhen $ h $ is $\\mathcal{C}^1$-smooth around $\\bar x$, both subdifferentials in \\eqref{frechet} and \\eqref{limiting subdiff} reduce to the gradient $\\nabla  h (\\bar x)$, while for convex functions they agree with the classical subdifferential of convex analysis. A necessary condition for $\\bar x\\in\\R^n$ to be a local minimizer of $ h $ is \n$0\\in \\partial h (\\bar x)$. Any point $\\bar x$ satisfying the latter condition is known in the literature as an {\\em M$($ordukhovich$)$-stationary} (or {\\em limiting-stationary}) point of $h$. Since this paper does not deal with other stationary concepts for nonsmooth functions, we simply use the {\\em stationary} point term in what follows.\n","sha256_utf8":"1e05ee9ef546f19fb148f7f21c5c2b1fc320ff213b514df3382dad4c44894951"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"基本PLK条件","source_units":[{"start":22313,"end":23027,"start_line":259,"end_line":271,"raw_tex":"\\begin{Definition}\\rm \\label{KL ine}\\rm We say that a lower semicontinuous (l.s.c.) function $f:\\R^n\\rightarrow\\overline{\\R}$ satisfies the $($basic$)$ \\textit{Polyak-\\L ojasiewicz-Kurdyka} $(PLK)$ condition at $\\bar x\\in\\dom \\partial f$ if there exist $\\eta\\in (0,\\infty]$, a neighborhood $U$ of $\\bar x$, and a desingularizing concave function $\\varphi:[0,\\eta)\\rightarrow[0,\\infty)$ such that:\n\n{\\bf(i)} $\\varphi(0)=0$.\n\n{\\bf(ii)} $\\varphi$ is $\\mathcal{C}^1$-smooth on $(0,\\eta)$.\n\n{\\bf(iii)} $\\varphi'>0$ on $(0,\\eta)$.\n\n{\\bf(iv)} For all $x\\in U$ with $f(\\bar{x})<f(x)<f(\\bar{x})+\\eta$, we have \n\\begin{align}\\label{KL 2}\n\\varphi'\\big(f(x)-f(\\bar{x})\\big)d(0,\\partial f(x))\\ge 1.\n\\end{align}\n\\end{Definition}","sha256_utf8":"7f0750dc4b6353642beebc0f0eb509297d14f6b31b39af084f3ff7b592c36e5d"}]},{"id":"S007","semantic_type":"derived_fact","kind":"derived_fact","title":"PLK适用性与同能级传递","source_units":[{"start":23029,"end":23821,"start_line":273,"end_line":276,"raw_tex":"\\begin{Remark}\\rm\\label{algebraic}If $f$ satisfies the PLK property at $\\bar x$ with a neighborhood $U$, it is clear that the same property holds for any $\\tilde x\\in U\\cap {\\rm dom\\; }\\partial f$ satisfying $f(\\tilde x)=f(\\bar x).$\nIt has been realized that the PLK condition is satisfied in rather  broad settings. In particular, it holds at every \nnonstationary point of $f$, for semianalytic and more general functions known as definable in o-minimal structures; see, e.g., \\cite{AbsilMahonyAndrews2005,AttouchBolteRedontSoubeyran2010,AttouchBolteSvaiter2013} and the references therein. As demonstrated in \\cite[Section~2]{KhanhMordukhovichTran2023}, the PLK condition formulated in \\cite{AttouchBolteRedontSoubeyran2010} is stronger than the one in Definition~\\ref{KL ine}.\n\\end{Remark}","sha256_utf8":"5c1e542db7a10f11cdfc3ec8c18c0ba967d0bbf0e1b236f0e4fa34a28b02fd4f"}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","title":"去奇异函数积分表示","source_units":[{"start":23823,"end":24206,"start_line":278,"end_line":279,"raw_tex":"Note that condition (ii) ensures that, for any fixed $x\\in(0,\\eta)$ and $a\\in(0,x]$, the function $\\varphi'$ is integrable on $[a,x]$ and thus $\\varphi(x)=\\varphi(a)+\\int_a^{x}\\varphi'(t)dt$. Taking the limit as $a\\rightarrow0^+$ and using  \n$\\varphi(0)=0$ give us $\\varphi(x)=\\int_0^{x}\\varphi'(t)dt$. The following result plays an important role in our local convergence analysis.\n","sha256_utf8":"dc6036cc0bfc97041a56d4c6b5233807df5f5987ffcd55d5848f6a65ff1b8d55"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"Lyapunov函数","source_units":[{"start":27915,"end":28324,"start_line":328,"end_line":331,"raw_tex":"To proceed further, we recall the following fundamental construction that is usually employed in the analysis of momentum methods. Given a number $\\alpha>0$, consider the {\\em Lyapunov function} $H_\\alpha:\\R^n\\times \\R^n\\rightarrow\\R$ associated with the objective function $f$ that is defined by\n\\begin{align}\\label{lyapunov func}\nH_\\alpha(x,y):=f(x)+\\alpha\\norm{x-y}^2\\text{ for all }x,y\\in\\R^n.\n\\end{align}","sha256_utf8":"0d81c7d71248b723c5df3f9f827b8edca6dc3bb8f3d26118d530736312930301"}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","title":"相对误差动量算法","source_units":[{"start":38438,"end":39376,"start_line":454,"end_line":472,"raw_tex":"\\begin{Algorithm}[IGDm]\\hlabel{IGDm}\\quad\n\\begin{enumerate}[-]\n\\item \\textbf{Optimization problem: } \n\\begin{align*}\n\\eqref{main problem}\\;\\text{ with }\\begin{cases}X=\\R^n\\;\\;\\text{ for global minima,}\\\\\nX=\\B(\\bar x,r)\\;\\text{ for a local minimizer }\\bar x\\;\\text{ with some }\\;r>0.\n\\end{cases}\n\\end{align*}\n\\vspace{-0.5cm}\n\\item \\textbf{Initialization: }Choose  $x^0=x^1\\in\\Omega,\\;\\set{\\beta_k}\\subset [0,\\infty),\\;\\set{\\gamma_k}\\subset[0,\\infty)$,  $\\nu \\in(0,1),$ $\\tau>0$.\n\\item \\textbf{Parameter conditions:} $\\bar{\\beta}:=\\sup \\beta_k<1,$ $\\bar \\delta:=\\sup|\\beta_k-\\gamma_k|<\\infty.$\n\\item \\textbf{Iteration: } $(k\\ge 1)$ Update:\n\\begin{align}\n&x^k_{\\rm in}:=x^k+\\beta_k(x^k-x^{k-1}),\\label{step xin}\\\\\n&x^k_{\\rm ex}:=x^k+\\gamma_k(x^k-x^{k-1}),\\label{step xex}\\\\\n&x^{k+1}:=x^k_{\\rm in}-\\tau  g^k,\\;\\text{ where }\\;\\norm{g^k-\\nabla f(x^k_\\rex)}\\le \\nu \\norm{g^k}.\\label{iteration update}\n\\end{align}\n\\end{enumerate}\n\\end{Algorithm}","sha256_utf8":"febf84450593a7bbfac6ae084bfed1df0c2adde0c7404e220e25a61792c77dcb"}]},{"id":"S011","semantic_type":"notation","kind":"notation","title":"提升迭代变量","source_units":[{"start":41141,"end":41485,"start_line":497,"end_line":497,"raw_tex":"Define further the sequence $\\{z^k\\} \\subset \\mathbb{R}^{n}\\times \\mathbb{R}^n$ by $z^{k}:= (x^{k},x^{k-1})$ for all $k\\in\\N$ along which the fundamental properties of the Lyapunov function $H_\\alpha$ are establishes below. These properties are crucial for deriving and unifying local and global convergence properties of Algorithm~\\ref{IGDm}.\n","sha256_utf8":"29db792bfe5610fc3a91b71e6fdc0f90968be2dab2e5b9b742309166e9ef9884"}]},{"id":"S012","semantic_type":"assumption","kind":"assumption","title":"外推梯度非零","source_units":[{"start":53484,"end":53582,"start_line":655,"end_line":657,"raw_tex":"\\begin{Assumption}\\rm\\label{assu}\n$\\nabla f(x^k_{\\rm ex})\\ne 0$ for all $k\\in\\N$.\n\\end{Assumption}","sha256_utf8":"191a0a25d4bd19c5e44ac87b22bb27b3d5b6976f807d68937829829f49991d46"}]},{"id":"S013","semantic_type":"derived_fact","kind":"derived_fact","title":"速率定理的局部适用性","source_units":[{"start":27436,"end":27913,"start_line":325,"end_line":326,"raw_tex":"\\begin{Remark}\\rm\\label{rmk general rate} It is easy to see  that condition \\eqref{two conditions} in Proposition~\\ref{general rate} yields the fulfillment of (H2) and of (H1) with $\\sigma=\\frac{\\alpha}{\\beta}$  in Proposition~\\ref{general convergence under KL}. Proposition~\\ref{general rate} obviously holds true even for extended-real-valued functions $f$ provided that $\\{x^k\\}$ is convergent and $f$ is ${\\cal C}^1$-smooth on an open set containing $\\{x^k\\}$.\n\\end{Remark}","sha256_utf8":"f1f3e1f2588eee9734bc220f67f58552179d77651ff5b529d9fc116ae935bf5c"}]},{"id":"S014","semantic_type":"lemma","kind":"lemma","title":"光滑函数下降引理","source_units":[{"start":21658,"end":22024,"start_line":250,"end_line":255,"raw_tex":"\\begin{Lemma}\\label{lemma descent}\nGiven $f:\\R^n\\rightarrow\\R$ and $x,y\\in\\Omega$, assume that $f$ is differentiable on the line segment $[x,y]$ and its derivative is Lipschitz continuous on this segment with constant $L>0$. Then we have\n\\begin{align}\\label{eq descent}\n\\abs{f(y)-f(x)-\\dotproduct{\\nabla f(x),y-x}}\\le \\frac{L}{2}\\norm{y-x}^2.\n\\end{align}\n\\end{Lemma}","sha256_utf8":"4198f1dbb9ce50c022341b6aef82c7a6f337f62c63dd852c50beba1de6a9c229"}]},{"id":"S015","semantic_type":"proposition","kind":"proposition","title":"PLK下降收敛准则","source_units":[{"start":25360,"end":26193,"start_line":294,"end_line":307,"raw_tex":"\\begin{Proposition}\\label{general convergence under KL}\nLet $f:\\R^n\\rightarrow\\R$ be a $\\mathcal{C}^1$-smooth function, and let the following conditions hold along a sequence of iterates $\\set{x^k}\\subset\\R^n$ for the function $f$:\n\\begin{itemize}\n\\item[\\bf(H1)] {\\rm(primary descent condition)}. There exists $\\sigma>0$ such that for sufficiently large $k\\in\\N$, we have \n\\begin{align*}\n f(x^k)-f(x^{k+1})\\ge\\sigma\\norm{\\nabla f(x^k)}\\cdot\\norm{x^{k+1}-x^k}.\n\\end{align*}\n\\item[\\bf(H2)] {\\rm(complementary descent condition)}. For sufficiently large $k\\in\\N$, we have\n\\begin{align*}\n \\big[f(x^{k+1})=f(x^k)\\big]\\Longrightarrow [x^{k+1}=x^k].\n\\end{align*}\n\\end{itemize}\nIf $\\bar x$ is an accumulation point of $\\set{x^k}$  and $f$ satisfies the PLK property at $\\bar x$, then $x^k\\rightarrow\\bar x$ as $k\\to\\infty$.\n\\end{Proposition}","sha256_utf8":"eaaa9b37fbe4a4db0f08b387e35d605ef5cb42622348e6fd7681df6e9ca59adf"}]},{"id":"S016","semantic_type":"proposition","kind":"proposition","title":"指数PLK迭代速率","source_units":[{"start":26473,"end":27434,"start_line":311,"end_line":323,"raw_tex":"\\begin{Proposition}\\label{general rate}\nLet $f:\\R^n\\rightarrow\\R$ be a $\\mathcal{C}^1$-smooth function, and let $\\set{x^k}$ satisfy the conditions \n\\begin{align}\\label{two conditions}\nf(x^k)-f(x^{k+1})\\ge \\alpha\\norm{x^{k+1}-x^k}^2\\;\\text{ and }\\;\\norm{\\nabla f(x^k)}\\le \\beta\\norm{x^{k+1}-x^k}\n\\end{align}\nfor all $k\\in\\N$ sufficiently large with some constants $\\alpha,\\beta>0$. Suppose  in addition that $\\bar x$ is an accumulation point of $\\set{x^k}$ and that $f$ satisfies the $($exponential$)$ PLK property at $\\bar x$ with $\\varphi(t)=Mt^{1-q}$ for some $M>0$ and $q\\in(0,1)$. Then the following convergence rates are guaranteed:\n\n{\\bf (i)} For $q\\in(0,1/2)$, the sequence $\\set{x^k}$ terminates at $\\bar x$ in finite steps.\n\n{\\bf (ii)} For $q=1/2$, the sequence $\\set{x^k}$ converges linearly to $\\bar x$ as $k\\to\\infty$.\n\n{\\bf (iii)} For $q\\in(1/2,1)$, we have  $\\norm{x^k-\\bar x}=\\mathcal{O}( k^{-\\frac{1-q}{2q-1}})$ as $k\\to\\infty$\n\\end{Proposition}","sha256_utf8":"be3f4fb0460bd5028140959b524ea56fb16ee70b58fcc4b4b5e89276d6ba3e09"}]},{"id":"S017","semantic_type":"proposition","kind":"proposition","title":"函数值与梯度速率转换","source_units":[{"start":36689,"end":37602,"start_line":433,"end_line":443,"raw_tex":"\\begin{Proposition}\\label{convergence rate deduce}\nLet $f:\\Omega\\rightarrow{\\R}$ be a function defined on open and convex set of $\\Omega\\subset\\R^n$, and let $\\bar x \\in\\Omega$. Assume that $f$ has a Lipschitz continuous gradient on $\\Omega$ and take a sequence $\\set{x^k}\\subset\\Omega $ converging to $\\bar x$. Suppose in addition that there exists $\\alpha>0$ such that \n\\begin{align}\\label{main condition rate}\n\\alpha\\norm{\\nabla f(x^k)}^2\\le f(x^k)-f(x^{k+1})\\;\\text{ for all large }\\;k\\in\\N.\n\\end{align}\nThen the linear convergence of $x^k\\rightarrow\\bar x$ ensures the linear convergences of $f(x^k)$ to $f(\\bar x)$ and $\\nabla f(x^k)$ to $0$ as $k\\to\\infty$. Moreover, the convergence rate $\\norm{x^k-\\bar x}=\\mathcal{O}(m(k))$ with $m(k)\\downarrow 0$ yields\n$$\n\\abs{f(x^k)-f(\\bar x)}=\\mathcal{O}(m^2(k))\\;\\mbox{ and }\\;\\norm{\\nabla f(x^k)}=\\mathcal{O}(m(k))\\;\\mbox{ as }\\;\nk\\to\\infty. \n$$\n\\end{Proposition}","sha256_utf8":"52ab37c2f5d6972e11a6e3f395a159691a49ca3428c0d241246ab9e5040f99da"}]},{"id":"S018","semantic_type":"lemma","kind":"lemma","title":"局部临界值唯一性","source_units":[{"start":24207,"end":24497,"start_line":281,"end_line":283,"raw_tex":"\\begin{Lemma}\\label{isolation lemma}\nLet $f:\\R^n\\rightarrow\\overline{\\R}$ be continuous around some local minimizer $\\bar x$ and satisfy the PLK condition at $\\bar x$. Then there is a neighborhood $U$ of $\\bar x$ such that $f(\\bar x)$ is the unique critical value of $f$ in $U$.\n\\end{Lemma}","sha256_utf8":"027eacf5203057f8845f669ec97e00ca04e8d8a7424a7aa52c282668dd311ac4"}]},{"id":"S019","semantic_type":"proof","kind":"proof","title":"临界值唯一性证明","source_units":[{"start":24498,"end":25181,"start_line":284,"end_line":290,"raw_tex":"\\begin{proof}\nThe PLK condition for $f$ at $\\bar x$ gives us $\\eta\\in(0,\\infty],$ a neighborhood $U$ of $\\bar x$, and a function $\\varphi:[0,\\eta)\\rightarrow[0,\\infty)$ satisfying all the conditions in Definition~\\ref{KL ine}. Since $f$ is continuous around the local minimizer $\\bar x,$ we assume, by shrinking $U$ if necessary, that $f(\\bar x)\\le f(x)<f(\\bar x)+\\eta.$ Then condition (iv) in Definition~\\ref{KL 2} can be rewritten as\n\\begin{align*}\n\\varphi'(f(x)-f(\\bar x))d(0,\\partial f(x))\\ge 1\\text{ whenever }x\\in U,\\;f(x)\\ne f(\\bar x).\n\\end{align*}\nTherefore, $0\\notin \\partial f(x)$ for all such $x$, i.e., $f(\\bar x)$ is the unique critical value of $f$ on $U.$ \n\\end{proof}","sha256_utf8":"41e8e265773d8d27be5006dca5481258b86bb3365800fffc9e161c46807e4e5d"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"缩小邻域应用PLK","source_units":[{"start":24512,"end":25054,"start_line":285,"end_line":288,"raw_tex":"The PLK condition for $f$ at $\\bar x$ gives us $\\eta\\in(0,\\infty],$ a neighborhood $U$ of $\\bar x$, and a function $\\varphi:[0,\\eta)\\rightarrow[0,\\infty)$ satisfying all the conditions in Definition~\\ref{KL ine}. Since $f$ is continuous around the local minimizer $\\bar x,$ we assume, by shrinking $U$ if necessary, that $f(\\bar x)\\le f(x)<f(\\bar x)+\\eta.$ Then condition (iv) in Definition~\\ref{KL 2} can be rewritten as\n\\begin{align*}\n\\varphi'(f(x)-f(\\bar x))d(0,\\partial f(x))\\ge 1\\text{ whenever }x\\in U,\\;f(x)\\ne f(\\bar x).\n\\end{align*}\n","sha256_utf8":"481b5292ad8e68a58e3b32646a4c8282c1a27ab02b5a1047a0ff1ad8d0f72a38"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"排除其他临界能级","source_units":[{"start":25054,"end":25170,"start_line":289,"end_line":289,"raw_tex":"Therefore, $0\\notin \\partial f(x)$ for all such $x$, i.e., $f(\\bar x)$ is the unique critical value of $f$ on $U.$ \n","sha256_utf8":"1e63101e4c6e5fce3a88caa9c986d0bda875c0b60c64b0d8b705fd7f2b794e66"}]},{"id":"S022","semantic_type":"proposition","kind":"proposition","title":"Lyapunov族一致PLK性质","source_units":[{"start":29878,"end":31293,"start_line":346,"end_line":361,"raw_tex":"\\begin{Proposition}\\label{isolated} Let $f:\\R^n\\rightarrow\\overline\\R$ be a ${\\cal C}^1$-smooth function around its local minimizer $\\bar x$. Assume that $f$ satisfies the PLK condition at $\\bar x$ with the desingularizing function $\\psi:[0,\\eta)\\rightarrow[0,\\infty)$ for some $\\eta\\in (0,\\infty]$. Then the following assertions hold whenever $\\varepsilon\\in (0,1/4]$:\n   \n{\\bf(i)} The collection of functions $\\set{H_\\alpha}_{\\alpha\\ge \\varepsilon}$ satisfies the same PLK condition at $\\bar z:=(\\bar x,\\bar x)$ with the desingularizing function $\\varphi:[0,\\eta)\\rightarrow[0,\\infty)$ defined by $\\varphi(0):=0$ and \n\\begin{align}\\label{construction varphi}\n\\varphi(t):=\\sqrt{\\frac{8}{\\varepsilon}}\\int_0^{t/2} \\max\\set{\\psi'(s),\\frac{1}{\\sqrt{s}}}ds\\;\\text{ for all }\\;t\\in (0,\\eta).\n\\end{align}\n        \n{\\bf(ii)} If $\\psi(t)=Mt^{1-q}$ for some $M>0$ and $q\\in[0,1)$, then the function $\\varphi$ in \\eqref{construction varphi} is given by $\\varphi(t)=\\frac{2^{\\theta+1/2}\\tilde M}{\\sqrt{\\varepsilon}(1-\\theta)}t^{1-\\theta}$ when $t\\in (0,\\delta)$ for some $\\delta\\in(0,\\eta)$, where $\\theta:=\\max\\set{q,\\frac{1}{2}}$ and $\\tilde{M}>0$ is defined as \n\\begin{align}\\label{tilde M}\n\\tilde{M}:=\\begin{cases}\n1&\\text{ if }\\quad q\\in \\left[0,\\frac{1}{2}\\right),\\\\\n\\max \\set{M(1-q),1}&\\text{ if }\\quad q=\\frac{1}{2},\\\\\nM(1-q)&\\text{ if }\\quad q\\in \\left(\\frac{1}{2},1\\right).\n\\end{cases}\n\\end{align}\n\\end{Proposition}","sha256_utf8":"43c2d7da56317a0d8745a9079da20d991f4eaae4c79dc7465b2422d4d1c9c10f"}]},{"id":"S023","semantic_type":"proof","kind":"proof","title":"一致PLK性质完整证明","source_units":[{"start":31294,"end":36550,"start_line":362,"end_line":429,"raw_tex":"\\begin{proof}\nTo verify (i) pick any $\\varepsilon \\in (0,1/4]$ and use the PLK property of $f$ at its local minimizer $\\bar x$ to find a neighborhood $U$ of $\\bar x$ on which $f$ satisfies all the conditions in Definition~\\ref{KL ine} with the corresponding desingularizing function $\\psi: [0,\\eta) \\rightarrow [0,\\infty)$. In particular, we have\n\\begin{align}\\label{desingularizing}\n\\psi'\\big(f(x)-f(\\bar{x})\\big)\\norm{\\nabla f(x)}\\ge 1\\;\\mbox{  for all }\\;x \\in U\\;\\mbox{ with }\\;f(\\bar{x}) < f(x) < f(\\bar{x}) + \\eta.\n\\end{align}\nThe function $\\varphi$ in \\eqref{construction varphi} can be written as $\\varphi(t)=\\sqrt{\\frac{8}{\\varepsilon}}\\phi\\brac{\\frac{t}{2}}$, where $\\phi:[0,\\eta)\\rightarrow[0,\\infty)$ is defined by \n\\begin{align}\\label{construction phi t}\n\\phi(t):=\\int_0^t \\max\\set{\\psi'(s),\\frac{1}{\\sqrt{s}}}ds\\;\\text{ for }\\;t\\in(0,\\eta)\\;\\mbox{ with }\\;\\phi(0):=0.\n\\end{align}\nIt follows from the continuity of $\\psi$ at $0$ that\n\\begin{align*}\n0\\le \\phi(t)\\le \\int_0^t \\psi'(s)ds+ \\int_0^t\\frac{1}{\\sqrt{s}}ds=\\psi(t)+2\\sqrt{t}\\;\\text{ for }\\;t\\in(0,\\eta).\n\\end{align*}\nPassing the limit above as $t\\rightarrow0^+$ and using the continuity of $\\psi$ reveal that $\\phi$ is also continuous at $0$. Moreover, we deduce from \\eqref{construction phi t} that the derivative $\\phi'(t)=\\max \\set{\\psi'(t),\\frac{1}{\\sqrt{t}}}>0$ is positive, continuous, and nonincreasing on $(0,\\eta)$, which ensures the concavity of $\\phi$ on this interval. In addition, it follows that \n\\begin{align}\\label{phi'>1}\n\\phi'(f(x)-f(\\bar x))\\norm{\\nabla f(x)}\\ge \\psi'(f(x)-f(\\bar x))\\norm{\\nabla f(x)}\\ge 1\n\\end{align}\nfor $x\\in U$ with $f(\\bar x)< f(x)<f(x)+\\eta$. Since $\\varphi(t)=\\sqrt{\\frac{8}{\\varepsilon}}\\phi\\brac{\\frac{t}{2}}$, the domain of $\\varphi$ is $[0,2\\eta).$ It is clear that $\\varphi$ satisfies all the conditions as a desingularizing function on $[0,2\\eta)$ with\n\\begin{align}\\label{deri varphi}\n\\varphi'(t)=\\sqrt{\\frac{2}{\\varepsilon}}\\phi'\\brac{\\frac{t}{2}}>0 \\text{ for all }t\\in (0,2\\eta).\n\\end{align}\nTaking any $\\alpha\\ge \\varepsilon$ and $z=(x,y)\\in Z:= U\\times U$ with \n\\begin{align}\\label{halpha}\nH_\\alpha(\\bar z)<H_\\alpha(z)<H_\\alpha(\\bar z)+\\eta,\n\\end{align}\nwe aim at verifying the inequality\n\\begin{align}\\label{KL R2n}\n\\varphi'(H_\\alpha(z)-H_\\alpha(\\bar z))\\norm{\\nabla H_\\alpha (z)}\\ge 1.\n\\end{align}\nTo proceed, observe first the estimates\n\\begin{subequations}\n\\begin{align}\n\\norm{\\nabla H_\\alpha(x,y)}&=\\sqrt{\\norm{\\nabla f(x)+2\\alpha(x-y)}^2+\\norm{2\\alpha(x-y)}^2} \\label{g_Hal.a}\\\\\n&\\ge \\sqrt{\\frac{1}{4}\\norm{\\nabla f(x)}^2+\\alpha^2\\norm{x-y}^2}\\label{g_Hal.b}\\\\\n&\\ge \\sqrt{\\frac{1}{4}\\norm{\\nabla f(x)}^2+\\alpha\\varepsilon\\norm{x-y}^2}\\label{g_Hal.c}\\\\\n&\\ge \\sqrt{\\frac{\\varepsilon}{2}}\\brac{\\norm{\\nabla f(x)}+\\sqrt{\\alpha}\\norm{x-y}},\\label{g_Hal.d}\n\\end{align}\n\\end{subequations}\nwhere \\eqref{g_Hal.a} is comes from the construction of $H_\\alpha$, \\eqref{g_Hal.b} follows from the inequality $\\norm{a+b}^2\\ge \\frac{1}{4}\\norm{a}^2-\\frac{3}{4}\\norm{b}^2$ for $a,b\\in\\R^n$, \\eqref{g_Hal.c} follows from $\\alpha\\ge \\varepsilon$, and \\eqref{g_Hal.d} follows from $a^2+b^2\\ge \\frac{1}{2}(a+b)^2$ for $a,b\\in\\R$ and $\\varepsilon\\le 1/4$.  Now let $S:=\\max \\set{\\alpha\\norm{x-y}^2,f(x)-f(\\bar x)}$ and deduce from $f(x)\\ge f(\\bar x)$ as $x\\in U$ that\n\\begin{align*}\nH_\\alpha(z)-H_\\alpha(\\bar z)=f(x)+\\alpha\\norm{x-y}^2-f(\\bar x) \\in [S,2S].\n\\end{align*}\nCombining the latter with $0< H_\\alpha(z)-H_\\alpha(\\bar z)<\\eta$ by \\eqref{halpha} gives us $S\\in (0,\\eta).$ Using the nonincreasing property of $\\varphi'$ and the relation $\\varphi'(t)=\\sqrt{\\frac{2}{\\varepsilon}}\\phi'\\brac{\\frac{t}{2}}$ from \\eqref{deri varphi} yields\n\\begin{align*}\n\\varphi'(H_\\alpha(z)-H_\\alpha(\\bar z))&\\ge \\varphi'\\brac{2S}=\\sqrt{\\frac{2}{\\varepsilon}}\\phi'(S).\n\\end{align*}\nFurthermore, it follows from  \\eqref{deri varphi} and  \\eqref{g_Hal.d} that\n\\begin{align*}\n\\varphi'(H_\\alpha(z)-H_\\alpha(\\bar z))\\norm{\\nabla H_\\alpha(x,y)}&\\ge \\sqrt{\\frac{2}{\\varepsilon}}\\phi'(S)\\sqrt{\\frac{\\varepsilon}{2}}\\brac{\\norm{\\nabla f(x)}+\\sqrt{\\alpha}\\norm{x-y}}\\\\\n&=\\phi'(S)\\norm{\\nabla f(x)}+\\phi'(S)\\sqrt{\\alpha}\\norm{x-y}.\n\\end{align*}\nBy $\\phi'\\big(f(x)-f(\\bar{x})\\big)\\norm{\\nabla f(x)}\\ge 1$ coming from \\eqref{phi'>1} and $\\phi'(\\alpha \\norm{x-y}^2)\\ge \\frac{1}{\\sqrt{\\alpha}\\norm{x-y}}$ due to $\\phi'(t)\\ge \\frac{1}{\\sqrt{t}}$ for $t\\in (0,\\eta),$ we get the estimate\n\\begin{align*}\n\\phi'(S)\\norm{\\nabla f(x)}+\\phi'(S)\\sqrt{\\alpha}\\norm{x-y}\\ge 1,\n\\end{align*}\nwhich therefore verifies assertion (i) of the proposition.\n\nTo prove assertion (ii)  is easier. Indeed, let $\\psi(t)=Mt^{1-q}$ for $q\\in[0,1)$, let $\\theta:=\\max\\set{\\frac{1}{2},q}\\in (0,1)$, and take $\\tilde M$ from \\eqref{tilde M}. It readily follows that\n\\begin{align}\\label{formu phi}\n\\max\\set{\\psi'(s),\\frac{1}{\\sqrt{s}}}=\\tilde Ms^{-\\theta}\\text{ for all }s\\in(0,\\delta)\n\\end{align}\nfor some $\\delta\\in(0,\\eta)$ sufficiently small.\nCombining the latter with \\eqref{construction phi t}, we arrive at \n\\begin{align*}\n \\varphi(t)=\\sqrt{\\frac{8}{\\varepsilon}}\\int_0^{t/2}\\tilde{M}s^{-\\theta}ds=\\frac{2^{\\theta+1/2}\\tilde M}{\\sqrt{\\varepsilon}(1-\\theta)}t^{1-\\theta} \\text{ for all }t\\in (0,\\delta),\n\\end{align*}\nwhich thus verifies (ii) and completes the proof of the proposition.\n\\end{proof}","sha256_utf8":"e7c5ec95a1bbc36f3fe942704b9263d00286e1542ce850783bb0f936d0a6f201"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"构造积分辅助函数","source_units":[{"start":31308,"end":32382,"start_line":363,"end_line":374,"raw_tex":"To verify (i) pick any $\\varepsilon \\in (0,1/4]$ and use the PLK property of $f$ at its local minimizer $\\bar x$ to find a neighborhood $U$ of $\\bar x$ on which $f$ satisfies all the conditions in Definition~\\ref{KL ine} with the corresponding desingularizing function $\\psi: [0,\\eta) \\rightarrow [0,\\infty)$. In particular, we have\n\\begin{align}\\label{desingularizing}\n\\psi'\\big(f(x)-f(\\bar{x})\\big)\\norm{\\nabla f(x)}\\ge 1\\;\\mbox{  for all }\\;x \\in U\\;\\mbox{ with }\\;f(\\bar{x}) < f(x) < f(\\bar{x}) + \\eta.\n\\end{align}\nThe function $\\varphi$ in \\eqref{construction varphi} can be written as $\\varphi(t)=\\sqrt{\\frac{8}{\\varepsilon}}\\phi\\brac{\\frac{t}{2}}$, where $\\phi:[0,\\eta)\\rightarrow[0,\\infty)$ is defined by \n\\begin{align}\\label{construction phi t}\n\\phi(t):=\\int_0^t \\max\\set{\\psi'(s),\\frac{1}{\\sqrt{s}}}ds\\;\\text{ for }\\;t\\in(0,\\eta)\\;\\mbox{ with }\\;\\phi(0):=0.\n\\end{align}\nIt follows from the continuity of $\\psi$ at $0$ that\n\\begin{align*}\n0\\le \\phi(t)\\le \\int_0^t \\psi'(s)ds+ \\int_0^t\\frac{1}{\\sqrt{s}}ds=\\psi(t)+2\\sqrt{t}\\;\\text{ for }\\;t\\in(0,\\eta).\n\\end{align*}\n","sha256_utf8":"a9c8281d6867ecab7e3c4253353ba697b65f767a463f41e307107800eb646391"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"验证凹性与导数下界","source_units":[{"start":32382,"end":33311,"start_line":375,"end_line":382,"raw_tex":"Passing the limit above as $t\\rightarrow0^+$ and using the continuity of $\\psi$ reveal that $\\phi$ is also continuous at $0$. Moreover, we deduce from \\eqref{construction phi t} that the derivative $\\phi'(t)=\\max \\set{\\psi'(t),\\frac{1}{\\sqrt{t}}}>0$ is positive, continuous, and nonincreasing on $(0,\\eta)$, which ensures the concavity of $\\phi$ on this interval. In addition, it follows that \n\\begin{align}\\label{phi'>1}\n\\phi'(f(x)-f(\\bar x))\\norm{\\nabla f(x)}\\ge \\psi'(f(x)-f(\\bar x))\\norm{\\nabla f(x)}\\ge 1\n\\end{align}\nfor $x\\in U$ with $f(\\bar x)< f(x)<f(x)+\\eta$. Since $\\varphi(t)=\\sqrt{\\frac{8}{\\varepsilon}}\\phi\\brac{\\frac{t}{2}}$, the domain of $\\varphi$ is $[0,2\\eta).$ It is clear that $\\varphi$ satisfies all the conditions as a desingularizing function on $[0,2\\eta)$ with\n\\begin{align}\\label{deri varphi}\n\\varphi'(t)=\\sqrt{\\frac{2}{\\varepsilon}}\\phi'\\brac{\\frac{t}{2}}>0 \\text{ for all }t\\in (0,2\\eta).\n\\end{align}\n","sha256_utf8":"6ea939850d403ff734b6226f3b996b876550b7dead52c29968483f63a0f1c03f"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"选取提升点与能量窗口","source_units":[{"start":33311,"end":33621,"start_line":383,"end_line":390,"raw_tex":"Taking any $\\alpha\\ge \\varepsilon$ and $z=(x,y)\\in Z:= U\\times U$ with \n\\begin{align}\\label{halpha}\nH_\\alpha(\\bar z)<H_\\alpha(z)<H_\\alpha(\\bar z)+\\eta,\n\\end{align}\nwe aim at verifying the inequality\n\\begin{align}\\label{KL R2n}\n\\varphi'(H_\\alpha(z)-H_\\alpha(\\bar z))\\norm{\\nabla H_\\alpha (z)}\\ge 1.\n\\end{align}\n","sha256_utf8":"f4975b427df945e35244adad430b6211c15918400c203c95e4013b2a0eb33463"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"梯度下界与最大能量项","source_units":[{"start":33621,"end":34676,"start_line":391,"end_line":403,"raw_tex":"To proceed, observe first the estimates\n\\begin{subequations}\n\\begin{align}\n\\norm{\\nabla H_\\alpha(x,y)}&=\\sqrt{\\norm{\\nabla f(x)+2\\alpha(x-y)}^2+\\norm{2\\alpha(x-y)}^2} \\label{g_Hal.a}\\\\\n&\\ge \\sqrt{\\frac{1}{4}\\norm{\\nabla f(x)}^2+\\alpha^2\\norm{x-y}^2}\\label{g_Hal.b}\\\\\n&\\ge \\sqrt{\\frac{1}{4}\\norm{\\nabla f(x)}^2+\\alpha\\varepsilon\\norm{x-y}^2}\\label{g_Hal.c}\\\\\n&\\ge \\sqrt{\\frac{\\varepsilon}{2}}\\brac{\\norm{\\nabla f(x)}+\\sqrt{\\alpha}\\norm{x-y}},\\label{g_Hal.d}\n\\end{align}\n\\end{subequations}\nwhere \\eqref{g_Hal.a} is comes from the construction of $H_\\alpha$, \\eqref{g_Hal.b} follows from the inequality $\\norm{a+b}^2\\ge \\frac{1}{4}\\norm{a}^2-\\frac{3}{4}\\norm{b}^2$ for $a,b\\in\\R^n$, \\eqref{g_Hal.c} follows from $\\alpha\\ge \\varepsilon$, and \\eqref{g_Hal.d} follows from $a^2+b^2\\ge \\frac{1}{2}(a+b)^2$ for $a,b\\in\\R$ and $\\varepsilon\\le 1/4$.  Now let $S:=\\max \\set{\\alpha\\norm{x-y}^2,f(x)-f(\\bar x)}$ and deduce from $f(x)\\ge f(\\bar x)$ as $x\\in U$ that\n\\begin{align*}\nH_\\alpha(z)-H_\\alpha(\\bar z)=f(x)+\\alpha\\norm{x-y}^2-f(\\bar x) \\in [S,2S].\n\\end{align*}\n","sha256_utf8":"9cb23155e62026a87fffbd1677fd5c493676d40e89a1f62f5d3198cfe29caf32"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"组合能量与梯度估计","source_units":[{"start":34676,"end":35426,"start_line":404,"end_line":412,"raw_tex":"Combining the latter with $0< H_\\alpha(z)-H_\\alpha(\\bar z)<\\eta$ by \\eqref{halpha} gives us $S\\in (0,\\eta).$ Using the nonincreasing property of $\\varphi'$ and the relation $\\varphi'(t)=\\sqrt{\\frac{2}{\\varepsilon}}\\phi'\\brac{\\frac{t}{2}}$ from \\eqref{deri varphi} yields\n\\begin{align*}\n\\varphi'(H_\\alpha(z)-H_\\alpha(\\bar z))&\\ge \\varphi'\\brac{2S}=\\sqrt{\\frac{2}{\\varepsilon}}\\phi'(S).\n\\end{align*}\nFurthermore, it follows from  \\eqref{deri varphi} and  \\eqref{g_Hal.d} that\n\\begin{align*}\n\\varphi'(H_\\alpha(z)-H_\\alpha(\\bar z))\\norm{\\nabla H_\\alpha(x,y)}&\\ge \\sqrt{\\frac{2}{\\varepsilon}}\\phi'(S)\\sqrt{\\frac{\\varepsilon}{2}}\\brac{\\norm{\\nabla f(x)}+\\sqrt{\\alpha}\\norm{x-y}}\\\\\n&=\\phi'(S)\\norm{\\nabla f(x)}+\\phi'(S)\\sqrt{\\alpha}\\norm{x-y}.\n\\end{align*}\n","sha256_utf8":"f103836af15d0ebe81d5e71400e058a257a46dddb182110beebcc0bd93a5b2bc"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"完成一致PLK不等式","source_units":[{"start":35426,"end":35815,"start_line":413,"end_line":417,"raw_tex":"By $\\phi'\\big(f(x)-f(\\bar{x})\\big)\\norm{\\nabla f(x)}\\ge 1$ coming from \\eqref{phi'>1} and $\\phi'(\\alpha \\norm{x-y}^2)\\ge \\frac{1}{\\sqrt{\\alpha}\\norm{x-y}}$ due to $\\phi'(t)\\ge \\frac{1}{\\sqrt{t}}$ for $t\\in (0,\\eta),$ we get the estimate\n\\begin{align*}\n\\phi'(S)\\norm{\\nabla f(x)}+\\phi'(S)\\sqrt{\\alpha}\\norm{x-y}\\ge 1,\n\\end{align*}\nwhich therefore verifies assertion (i) of the proposition.\n","sha256_utf8":"0a364c78a08f8f774e5c68ddcd98a1f9828dc3e6c999cd8f3417971ed1a01009"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"计算幂型去奇异函数","source_units":[{"start":35816,"end":36539,"start_line":419,"end_line":428,"raw_tex":"To prove assertion (ii)  is easier. Indeed, let $\\psi(t)=Mt^{1-q}$ for $q\\in[0,1)$, let $\\theta:=\\max\\set{\\frac{1}{2},q}\\in (0,1)$, and take $\\tilde M$ from \\eqref{tilde M}. It readily follows that\n\\begin{align}\\label{formu phi}\n\\max\\set{\\psi'(s),\\frac{1}{\\sqrt{s}}}=\\tilde Ms^{-\\theta}\\text{ for all }s\\in(0,\\delta)\n\\end{align}\nfor some $\\delta\\in(0,\\eta)$ sufficiently small.\nCombining the latter with \\eqref{construction phi t}, we arrive at \n\\begin{align*}\n \\varphi(t)=\\sqrt{\\frac{8}{\\varepsilon}}\\int_0^{t/2}\\tilde{M}s^{-\\theta}ds=\\frac{2^{\\theta+1/2}\\tilde M}{\\sqrt{\\varepsilon}(1-\\theta)}t^{1-\\theta} \\text{ for all }t\\in (0,\\delta),\n\\end{align*}\nwhich thus verifies (ii) and completes the proof of the proposition.\n","sha256_utf8":"73b34c4dad295fa22fa8992ec3ab3bbd315c50239b5d705caccb2cf221b5f93c"}]},{"id":"S031","semantic_type":"notation","kind":"notation","title":"一致PLK构造参数","source_units":[{"start":30498,"end":30677,"start_line":349,"end_line":351,"raw_tex":"\\begin{align}\\label{construction varphi}\n\\varphi(t):=\\sqrt{\\frac{8}{\\varepsilon}}\\int_0^{t/2} \\max\\set{\\psi'(s),\\frac{1}{\\sqrt{s}}}ds\\;\\text{ for all }\\;t\\in (0,\\eta).\n\\end{align}","sha256_utf8":"94048c5c1ebccd272147cf61617a1ea54d88ba4509ab5785bdef88ec65bbb38f"},{"start":31033,"end":31275,"start_line":354,"end_line":360,"raw_tex":"\\begin{align}\\label{tilde M}\n\\tilde{M}:=\\begin{cases}\n1&\\text{ if }\\quad q\\in \\left[0,\\frac{1}{2}\\right),\\\\\n\\max \\set{M(1-q),1}&\\text{ if }\\quad q=\\frac{1}{2},\\\\\nM(1-q)&\\text{ if }\\quad q\\in \\left(\\frac{1}{2},1\\right).\n\\end{cases}\n\\end{align}","sha256_utf8":"5e86627c8f7468910c570634ebd3163f72d4859a17ac190bf60d212c2ab64f40"}]},{"id":"S032","semantic_type":"proposition","kind":"proposition","title":"能量下降与梯度控制","source_units":[{"start":41486,"end":42373,"start_line":499,"end_line":515,"raw_tex":"\\begin{Proposition}\\label{prop AIGD tech}\nLet $\\set{x^k}$ be a sequence generated by Algorithm~{\\rm\\ref{IGDm}} with \n\\begin{align}\\label{max Lt<1-nu}\n\\max\\set{L\\tau,2L\\tau \\bar \\delta+(L\\tau+1)\\bar \\beta^2}< 1-\\nu\n\\end{align}\nsuch that $x^k,x^k_{\\rm in},x^k_{\\rm ex}\\in X$ for all $k=1,\\ldots,K\\in\\N$. Consider the positive constants\n\\begin{align}\\label{C1C2}\n\\alpha:&=\\frac{(L \\tau+1)\\bar \\beta^2+1-\\nu }{4\\tau},\\nonumber\\\\\nC_1:&=\\frac{1-\\nu-(L \\tau+1)\\bar \\beta^2-2L\\tau \\bar \\delta}{4\\tau},\\\\\nC_2:&=\\sqrt{2}\\max\\set{\\frac{\\nu+1}{\\tau },\\frac{\\bar\\beta(L\\tau+\\nu+1)+L\\tau\\bar\\delta}{\\tau }}+4{\\alpha}.\\nonumber\n\\end{align}\nThen we have the following estimates whenever $k=1,\\ldots,K-1:$\n\\begin{align}\nC_1\\norm{z^{k+1}-z^k}^2&\\le H_{\\alpha}(z^k)-H_{\\alpha}(z^{k+1}),\\label{3.2(i)}\\\\\n\\norm{\\nabla H_{\\alpha}(z^{k})}&\\le C_2\\norm{z^{k+1}-z^k}\\label{3.2(ii)}.\n\\end{align}\n\\end{Proposition}","sha256_utf8":"7117e4af895cc3bbe0bf9274ecbe3eddaaa7ebce9567def3e98ccf3f97bc1633"}]},{"id":"S033","semantic_type":"proof","kind":"proof","title":"能量估计完整证明","source_units":[{"start":42374,"end":49258,"start_line":516,"end_line":602,"raw_tex":"\\begin{proof}\nBy Lemma~\\ref{lemma descent}, the Lipschitz continuity with constant $L>0$ of $\\nabla f$ on $X $ yields\n\\begin{align*}\n-\\frac{L}{2}\\norm{y-x}^2\\le f(y)-f(x)-\\dotproduct{\\nabla f(x),y-x}\\le \\frac{L}{2}\\norm{y-x}^2 \\text{ for all }x,y\\in X .\n\\end{align*}\nFor any $k=1,\\ldots,K-1$, we use  the left-hand side inequality above with $y=x^k,x=x^k_{\\rm in}$ and the right-hand side inequality with $y=x^{k+1},x=x^k_{\\rm in}$ to get\n\\begin{align*}\n f(x^k)-f(x^k_{\\rm in})-\\dotproduct{\\nabla f(x^k_{\\rm in}),x^k-x^k_{\\rm in}}&\\ge -\\frac{L }{2}\\norm{x^k-x^k_{\\rm in}}^2,\n \\\\\n f(x^{k+1})- f(x^k_{\\rm in}) -\\dotproduct{\\nabla f(x^k_{\\rm in}),x^{k+1}-x^k_{\\rm in}}&\\le \\frac{L}{2}\\norm{x^{k+1}-x^k_{\\rm in}}^2.\n\\end{align*}\nSubtracting the first inequality from the second one gives us \n\\begin{align*}\nf(x^{k+1})-f(x^k)-\\dotproduct{\\nabla f(x^k_{\\rm in}),x^{k+1}-x^k}\\le \\frac{L}{2}\\norm{x^{k+1}-x^k_{\\rm in}}^2+\\frac{L }{2}\\norm{x^k-x^k_{\\rm in}}^2,\n\\end{align*}\nwhich can be rewritten in the form\n\\begin{align}\\label{prop: 1 chain}\n\\begin{array}{ll}\n\\disp f(x^{k+1})-f(x^k)\\le \\frac{L}{2}\\norm{x^{k+1}-x^k_{\\rm in}}^2+\\frac{L }{2}\\norm{x^k-x^k_{\\rm in}}^2+\\dotproduct{\\nabla f(x^k_{\\rm in}),x^{k+1}-x^k}\\\\\n=\\disp\\frac{L}{2} \\norm{x^{k+1}-x^k_{\\rm in}}^2+\\frac{L \\beta_k^2}{2}\\norm{x^k-x^{k-1}}^2+\\dotproduct{\\nabla f(x^k_{\\rm in})-\\nabla f(x^k_{\\rm ex}),x^{k+1}-x^k}\\\\\n\\disp+\\dotproduct{\\nabla f(x^k_{\\rm ex})- g^k,x^{k+1}-x^k}+\\dotproduct{g^k,x^{k+1}-x^k}.\n\\end{array}\n\\end{align}\nBy the Cauchy-Schwarz inequality, the Lipschitz continuity of $\\nabla f$ on $X $, and the constructions of $x^k_{\\rm in}$ and $x^k_{\\rm ex}$ in \\eqref{step xin} and \\eqref{step xex}, we deduce the estimates\n\\begin{align}\\label{prop: 1.1 chain}\n\\begin{array}{ll}\n\\dotproduct{\\nabla f(x^k_{\\rm in})-\\nabla f(x^k_{\\rm ex}),x^{k+1}-x^k}\\le \\norm{\\nabla f(x^k_{\\rm in})-\\nabla f(x^k_{\\rm ex})}\\cdot\\norm{x^{k+1}-x^k}\\\\\n\\le L\\norm{x^k_{\\rm in}-x^k_{\\rm ex}}\\cdot\\norm{x^{k+1}-x^k}\\le L\\abs{\\beta_k-\\gamma_k}\\norm{x^k-x^{k-1}}\\cdot\\norm{x^{k+1}-x^k}\\\\\n\\le\\disp\\frac{L\\bar \\delta}{2}\\brac{\\norm{x^k-x^{k-1}}^2+\\norm{x^{k+1}-x^k}^2}.\n\\end{array}\n\\end{align}\nIt follows from the Cauchy-Schwarz and AM-GM inequalities together with the algorithm design $x^{k+1}:=x^k_{\\rm in}-\\tau  g^k$ and $\\norm{g^k-\\nabla f(x^k_{\\rm ex})}\\le \\nu\\norm{g^k}$ in \\eqref{iteration update} that\n\\begin{align}\\label{prop: 3 chain}\n\\begin{array}{ll}\n\\dotproduct{\\nabla f(x^k_{\\rm ex})- g^k,x^{k+1}-x^k}\\le \\norm{\\nabla f(x^k_{\\rm ex})- g^k}\\cdot\\norm{x^{k+1}-x^k}\\\\\n\\disp\\le \\nu \\norm{g^k}\\cdot\\norm{x^{k+1}-x^k}=\\frac{\\nu}{\\tau }\\norm{x^k_{\\rm in}-x^{k+1}}\\cdot\\norm{x^{k+1}-x^k}\\\\\n\\le\\disp \\frac{\\nu}{2\\tau }\\brac{\\norm{x^{k+1}-x^k_{\\rm in}}^2+\\norm{x^{k+1}-x^k}^2}.\n\\end{array}\n\\end{align}\nUsing further \\eqref{step xin} and \\eqref{iteration update},  we arrive at\n\\begin{align}\\label{prop: 2 chain}\n\\begin{array}{ll}\n\\dotproduct{g^k,x^{k+1}-x^k}=\\disp\\frac{1}{\\tau }\\dotproduct{x^k_{\\rm in}-x^{k+1},x^{k+1}-x^k}\\\\\n=\\disp\\frac{1}{2\\tau }\\brac{\\norm{x^k_{\\rm in}-x^k}^2-\\norm{x^{k+1}-x^k_{\\rm in}}^2-\\norm{x^k-x^{k+1}}^2}\\\\\n=\\disp\\frac{1}{2\\tau }\\brac{\\beta_k^2\\norm{x^{k}-x^{k-1}}^2-\\norm{x^{k+1}-x^k_{\\rm in}}^2-\\norm{x^k-x^{k+1}}^2}.\n\\end{array}\n\\end{align}\nCombining \\eqref{prop: 1 chain}--\\eqref{prop: 2 chain} with taking into account $L\\tau +\\nu<1$ and $\\beta_k\\le \\bar \\beta$ gives us\n\\begin{align}\\label{estimane conse}\n\\begin{array}{ll}\nf(x^{k+1})-f(x^k)&\\le\\disp\\frac{L\\tau +\\nu-1}{2\\tau }\\norm{x^{k+1}-x^k_{\\rm in}}^2+\\frac{L\\tau\\bar \\delta+\\nu-1}{2\\tau }\\norm{x^{k+1}-x^k}^2\\\\\n&+\\disp\\frac{L \\tau \\beta_k^2+\\beta_k^2+L\\tau\\bar \\delta}{2\\tau }\\norm{x^k-x^{k-1}}^2\\\\\n&\\le\\disp\\frac{L\\tau\\bar \\delta+\\nu-1}{2\\tau }\\norm{x^{k+1}-x^k}^2+\\frac{(L \\tau +1)\\bar \\beta^2+L\\tau\\bar \\delta}{2\\tau }\\norm{x^k-x^{k-1}}^2.\n\\end{array}\n\\end{align}\nSince $\\bar\\beta^2<\\frac{1-\\nu}{L \\tau+1}$, it follows that $\\frac{(L \\tau+1)\\bar \\beta^2+L\\tau\\bar \\delta}{2\\tau }<\\frac{1-\\nu-L\\tau\\bar \\delta}{2\\tau }$, and by $\\alpha=\\frac{(L \\tau+1)\\bar \\beta^2+1-\\nu}{4\\tau}$ we get \n\\begin{align*}\nC_1=\\frac{1-\\nu-(L \\tau+1)\\bar \\beta^2-2L\\tau \\bar \\delta}{4\\tau}=\\frac{1-\\nu-L\\tau\\bar\\delta}{2\\tau}-\\alpha=\\alpha-\\frac{(L  \\tau+1)\\bar\\beta^2+L\\tau\\bar \\delta}{2\\tau}\n\\end{align*}\nfor $C_1$ in \\eqref{C1C2}. Using the latter together with \\eqref{estimane conse} and the construction of $H_\\alpha$ in \\eqref{lyapunov func}  brings us to\n\\begin{align*}\nH_{\\alpha}(z^{k+1})-H_{\\alpha}(z^k)&= f(x^{k+1})+\\alpha\\norm{x^{k+1}-x^k}^2-f(x^{k})-\\alpha\\norm{x^{k}-x^{k-1}}^2\\\\\n&\\le\\brac{ \\frac{\\nu-1+L\\tau\\bar \\delta}{2\\tau }+\\alpha}\\norm{x^{k+1}-x^k}^2+\\brac{\\frac{(L \\tau+1)\\bar  \\beta^2+L\\tau\\bar \\delta}{2\\tau }-\\alpha}\\norm{x^k-x^{k-1}}^2\\\\\n&\\le- C_1\\brac{\\norm{x^{k+1}-x^k}^2+\\norm{x^k-x^{k-1}}^2}=-C_1\\norm{z^{k+1}-z^k}^2,\n\\end{align*}\nwhich therefore justifies the inequality in \\eqref{3.2(i)}. \n\nTo verify \\eqref{3.2(ii)}, take any $k=1,\\ldots,K-1$ and obtain the relationships\n\\begin{subequations}\n\\begin{align}\n\\norm{\\nabla f(x^k)}&\\le \\norm{\\nabla f(x^k)-\\nabla f(x^k_{\\rm in})} +\\norm{\\nabla f(x^k_{\\rm in})-\\nabla f(x^k_{\\rm ex})}+\\norm{\\nabla f(x^k_{\\rm ex})-g^k}+\\norm{g^k}\\label{nab.1 triang}\\\\\n&\\le(L\\bar \\beta+L\\bar\\delta)\\norm{x^k-x^{k-1}}+(\\nu+1)\\norm{g^k}\\label{nab.2}\\\\\n&=(L\\bar \\beta+L\\bar\\delta)\\norm{x^k-x^{k-1}}+\\frac{\\nu+1}{\\tau }\\norm{x^k+\\beta_k(x^k-x^{k-1})-x^{k+1}}\\label{nab.3}\\\\\n&\\le \\brac{L\\bar \\beta+L\\bar\\delta+\\frac{\\nu+1}{\\tau }\\bar \\beta}\\norm{x^k-x^{k-1}}+\\frac{\\nu+1}{\\tau }\\norm{x^{k+1}-x^k}\\nonumber\\\\\n&\\le \\max\\set{\\frac{\\nu+1}{\\tau },L\\bar \\beta+L\\bar\\delta+\\frac{\\nu+1}{\\tau }\\bar \\beta}\\brac{\\norm{x^{k+1}-x^k}+\\norm{x^k-x^{k-1}}}\\nonumber\\\\\n&\\le \\sqrt{2}\\max\\set{\\frac{\\nu+1}{\\tau },L\\bar \\beta+L\\bar\\delta+\\frac{\\nu+1}{\\tau }\\bar \\beta}\\norm{z^{k+1}-z^k}\\label{nab.6},\n\\end{align}\n\\end{subequations}\nwhere \\eqref{nab.1 triang} follows from the triangle inequality; \\eqref{nab.2} follows from $\\norm{g^k-\\nabla f(x^k_{\\rm ex})}\\le \\nu\\norm{g^k}$ in \\eqref{iteration update}, the updates in \\eqref{step xin}, \\eqref{step xex}, and the Lipschitz continuity of $\\nabla f$ on $X $ containing $ x^k_\\rin,x^k_\\rex$; \\eqref{nab.3} follows from \\eqref{step xin} and \\eqref{step xex}; \\eqref{nab.6} follows from $a+b\\le \\sqrt{2(a^2+b^2)}$ and $z^k=(x^k,x^{k-1})$. By using the estimates above, we deduce from the construction of $H_\\alpha$ and inequality $\\sqrt{a^2+b^2}\\le a+b$ for $a,b\\ge 0$ that\n\\begin{align*}\n\\norm{\\nabla H_{\\alpha}(z^k)}&= \\sqrt{\\norm{\\nabla f(x^k)+2{\\alpha}(x^k-x^{k-1})}^2+\\norm{2{\\alpha} (x^k-x^{k-1})}^2}\\\\\n&\\le \\norm{\\nabla f(x^k)+2{\\alpha}(x^k-x^{k-1})}+\\norm{2{\\alpha} (x^k-x^{k-1})}\\\\\n& \\le \\norm{\\nabla f(x^k)}+4{\\alpha}\\norm{x^k-x^{k-1}}\\\\\n&\\le \\brac{\\sqrt{2}\\max\\set{\\frac{\\nu+1}{\\tau },L\\bar \\beta+L\\bar\\delta+\\frac{\\nu+1}{\\tau }\\bar \\beta}+4{\\alpha}}\\norm{z^{k+1}-z^k},\n\\end{align*}\nwhich justifies \\eqref{3.2(ii)}  and thus completes the proof of the proposition.\n\\end{proof}","sha256_utf8":"790488d3bdb597a7f92743b90eecf4a7192583c35d8c0b803c61b1a25530da12"}]},{"id":"S034","semantic_type":"notation","kind":"notation","title":"能量权重与控制常数","source_units":[{"start":41820,"end":42110,"start_line":505,"end_line":509,"raw_tex":"\\begin{align}\\label{C1C2}\n\\alpha:&=\\frac{(L \\tau+1)\\bar \\beta^2+1-\\nu }{4\\tau},\\nonumber\\\\\nC_1:&=\\frac{1-\\nu-(L \\tau+1)\\bar \\beta^2-2L\\tau \\bar \\delta}{4\\tau},\\\\\nC_2:&=\\sqrt{2}\\max\\set{\\frac{\\nu+1}{\\tau },\\frac{\\bar\\beta(L\\tau+\\nu+1)+L\\tau\\bar\\delta}{\\tau }}+4{\\alpha}.\\nonumber\n\\end{align}","sha256_utf8":"cb76a598f004f1665a80dcc827ba20b2564442058e5f8b74a9faa8ccfe0aa456"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"应用双侧光滑估计","source_units":[{"start":42388,"end":43099,"start_line":517,"end_line":526,"raw_tex":"By Lemma~\\ref{lemma descent}, the Lipschitz continuity with constant $L>0$ of $\\nabla f$ on $X $ yields\n\\begin{align*}\n-\\frac{L}{2}\\norm{y-x}^2\\le f(y)-f(x)-\\dotproduct{\\nabla f(x),y-x}\\le \\frac{L}{2}\\norm{y-x}^2 \\text{ for all }x,y\\in X .\n\\end{align*}\nFor any $k=1,\\ldots,K-1$, we use  the left-hand side inequality above with $y=x^k,x=x^k_{\\rm in}$ and the right-hand side inequality with $y=x^{k+1},x=x^k_{\\rm in}$ to get\n\\begin{align*}\n f(x^k)-f(x^k_{\\rm in})-\\dotproduct{\\nabla f(x^k_{\\rm in}),x^k-x^k_{\\rm in}}&\\ge -\\frac{L }{2}\\norm{x^k-x^k_{\\rm in}}^2,\n \\\\\n f(x^{k+1})- f(x^k_{\\rm in}) -\\dotproduct{\\nabla f(x^k_{\\rm in}),x^{k+1}-x^k_{\\rm in}}&\\le \\frac{L}{2}\\norm{x^{k+1}-x^k_{\\rm in}}^2.\n\\end{align*}\n","sha256_utf8":"339d531d87f83aaf125878646ee27a0032871ebae024c290818e4fab0069ea45"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"分解单步目标变化","source_units":[{"start":43099,"end":43859,"start_line":527,"end_line":538,"raw_tex":"Subtracting the first inequality from the second one gives us \n\\begin{align*}\nf(x^{k+1})-f(x^k)-\\dotproduct{\\nabla f(x^k_{\\rm in}),x^{k+1}-x^k}\\le \\frac{L}{2}\\norm{x^{k+1}-x^k_{\\rm in}}^2+\\frac{L }{2}\\norm{x^k-x^k_{\\rm in}}^2,\n\\end{align*}\nwhich can be rewritten in the form\n\\begin{align}\\label{prop: 1 chain}\n\\begin{array}{ll}\n\\disp f(x^{k+1})-f(x^k)\\le \\frac{L}{2}\\norm{x^{k+1}-x^k_{\\rm in}}^2+\\frac{L }{2}\\norm{x^k-x^k_{\\rm in}}^2+\\dotproduct{\\nabla f(x^k_{\\rm in}),x^{k+1}-x^k}\\\\\n=\\disp\\frac{L}{2} \\norm{x^{k+1}-x^k_{\\rm in}}^2+\\frac{L \\beta_k^2}{2}\\norm{x^k-x^{k-1}}^2+\\dotproduct{\\nabla f(x^k_{\\rm in})-\\nabla f(x^k_{\\rm ex}),x^{k+1}-x^k}\\\\\n\\disp+\\dotproduct{\\nabla f(x^k_{\\rm ex})- g^k,x^{k+1}-x^k}+\\dotproduct{g^k,x^{k+1}-x^k}.\n\\end{array}\n\\end{align}\n","sha256_utf8":"c597def245c41a3b6be78cb705d06257939e126ba4a87c758e4db6d37f6e450b"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"控制两类动量差异","source_units":[{"start":43859,"end":44508,"start_line":539,"end_line":546,"raw_tex":"By the Cauchy-Schwarz inequality, the Lipschitz continuity of $\\nabla f$ on $X $, and the constructions of $x^k_{\\rm in}$ and $x^k_{\\rm ex}$ in \\eqref{step xin} and \\eqref{step xex}, we deduce the estimates\n\\begin{align}\\label{prop: 1.1 chain}\n\\begin{array}{ll}\n\\dotproduct{\\nabla f(x^k_{\\rm in})-\\nabla f(x^k_{\\rm ex}),x^{k+1}-x^k}\\le \\norm{\\nabla f(x^k_{\\rm in})-\\nabla f(x^k_{\\rm ex})}\\cdot\\norm{x^{k+1}-x^k}\\\\\n\\le L\\norm{x^k_{\\rm in}-x^k_{\\rm ex}}\\cdot\\norm{x^{k+1}-x^k}\\le L\\abs{\\beta_k-\\gamma_k}\\norm{x^k-x^{k-1}}\\cdot\\norm{x^{k+1}-x^k}\\\\\n\\le\\disp\\frac{L\\bar \\delta}{2}\\brac{\\norm{x^k-x^{k-1}}^2+\\norm{x^{k+1}-x^k}^2}.\n\\end{array}\n\\end{align}\n","sha256_utf8":"bb0d902dd65386266974a9757723c6357e20e232dbe62cd90fb301e29e0ce4a7"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"控制相对梯度误差","source_units":[{"start":44508,"end":45121,"start_line":547,"end_line":554,"raw_tex":"It follows from the Cauchy-Schwarz and AM-GM inequalities together with the algorithm design $x^{k+1}:=x^k_{\\rm in}-\\tau  g^k$ and $\\norm{g^k-\\nabla f(x^k_{\\rm ex})}\\le \\nu\\norm{g^k}$ in \\eqref{iteration update} that\n\\begin{align}\\label{prop: 3 chain}\n\\begin{array}{ll}\n\\dotproduct{\\nabla f(x^k_{\\rm ex})- g^k,x^{k+1}-x^k}\\le \\norm{\\nabla f(x^k_{\\rm ex})- g^k}\\cdot\\norm{x^{k+1}-x^k}\\\\\n\\disp\\le \\nu \\norm{g^k}\\cdot\\norm{x^{k+1}-x^k}=\\frac{\\nu}{\\tau }\\norm{x^k_{\\rm in}-x^{k+1}}\\cdot\\norm{x^{k+1}-x^k}\\\\\n\\le\\disp \\frac{\\nu}{2\\tau }\\brac{\\norm{x^{k+1}-x^k_{\\rm in}}^2+\\norm{x^{k+1}-x^k}^2}.\n\\end{array}\n\\end{align}\n","sha256_utf8":"51df619d0ad6d66bd5944f22c4ef36fecd2b47f88ec5c13e6fa532bd4677dbc0"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"展开更新方向内积","source_units":[{"start":45121,"end":45591,"start_line":555,"end_line":562,"raw_tex":"Using further \\eqref{step xin} and \\eqref{iteration update},  we arrive at\n\\begin{align}\\label{prop: 2 chain}\n\\begin{array}{ll}\n\\dotproduct{g^k,x^{k+1}-x^k}=\\disp\\frac{1}{\\tau }\\dotproduct{x^k_{\\rm in}-x^{k+1},x^{k+1}-x^k}\\\\\n=\\disp\\frac{1}{2\\tau }\\brac{\\norm{x^k_{\\rm in}-x^k}^2-\\norm{x^{k+1}-x^k_{\\rm in}}^2-\\norm{x^k-x^{k+1}}^2}\\\\\n=\\disp\\frac{1}{2\\tau }\\brac{\\beta_k^2\\norm{x^{k}-x^{k-1}}^2-\\norm{x^{k+1}-x^k_{\\rm in}}^2-\\norm{x^k-x^{k+1}}^2}.\n\\end{array}\n\\end{align}\n","sha256_utf8":"3fa6f963a32385e464f25f0560dc6fb9c847db50c4e2a210b133859e6510b24b"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"合并目标下降系数","source_units":[{"start":45591,"end":46597,"start_line":563,"end_line":574,"raw_tex":"Combining \\eqref{prop: 1 chain}--\\eqref{prop: 2 chain} with taking into account $L\\tau +\\nu<1$ and $\\beta_k\\le \\bar \\beta$ gives us\n\\begin{align}\\label{estimane conse}\n\\begin{array}{ll}\nf(x^{k+1})-f(x^k)&\\le\\disp\\frac{L\\tau +\\nu-1}{2\\tau }\\norm{x^{k+1}-x^k_{\\rm in}}^2+\\frac{L\\tau\\bar \\delta+\\nu-1}{2\\tau }\\norm{x^{k+1}-x^k}^2\\\\\n&+\\disp\\frac{L \\tau \\beta_k^2+\\beta_k^2+L\\tau\\bar \\delta}{2\\tau }\\norm{x^k-x^{k-1}}^2\\\\\n&\\le\\disp\\frac{L\\tau\\bar \\delta+\\nu-1}{2\\tau }\\norm{x^{k+1}-x^k}^2+\\frac{(L \\tau +1)\\bar \\beta^2+L\\tau\\bar \\delta}{2\\tau }\\norm{x^k-x^{k-1}}^2.\n\\end{array}\n\\end{align}\nSince $\\bar\\beta^2<\\frac{1-\\nu}{L \\tau+1}$, it follows that $\\frac{(L \\tau+1)\\bar \\beta^2+L\\tau\\bar \\delta}{2\\tau }<\\frac{1-\\nu-L\\tau\\bar \\delta}{2\\tau }$, and by $\\alpha=\\frac{(L \\tau+1)\\bar \\beta^2+1-\\nu}{4\\tau}$ we get \n\\begin{align*}\nC_1=\\frac{1-\\nu-(L \\tau+1)\\bar \\beta^2-2L\\tau \\bar \\delta}{4\\tau}=\\frac{1-\\nu-L\\tau\\bar\\delta}{2\\tau}-\\alpha=\\alpha-\\frac{(L  \\tau+1)\\bar\\beta^2+L\\tau\\bar \\delta}{2\\tau}\n\\end{align*}\n","sha256_utf8":"323001ca1c75953a2df0bc8beac08cad76fbbeddabe9e16a3abf33b2990e079e"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"得到Lyapunov充分下降","source_units":[{"start":46597,"end":47210,"start_line":575,"end_line":581,"raw_tex":"for $C_1$ in \\eqref{C1C2}. Using the latter together with \\eqref{estimane conse} and the construction of $H_\\alpha$ in \\eqref{lyapunov func}  brings us to\n\\begin{align*}\nH_{\\alpha}(z^{k+1})-H_{\\alpha}(z^k)&= f(x^{k+1})+\\alpha\\norm{x^{k+1}-x^k}^2-f(x^{k})-\\alpha\\norm{x^{k}-x^{k-1}}^2\\\\\n&\\le\\brac{ \\frac{\\nu-1+L\\tau\\bar \\delta}{2\\tau }+\\alpha}\\norm{x^{k+1}-x^k}^2+\\brac{\\frac{(L \\tau+1)\\bar  \\beta^2+L\\tau\\bar \\delta}{2\\tau }-\\alpha}\\norm{x^k-x^{k-1}}^2\\\\\n&\\le- C_1\\brac{\\norm{x^{k+1}-x^k}^2+\\norm{x^k-x^{k-1}}^2}=-C_1\\norm{z^{k+1}-z^k}^2,\n\\end{align*}\nwhich therefore justifies the inequality in \\eqref{3.2(i)}. \n","sha256_utf8":"dc0b18b2479aac44db3eb04fe195176aa119ffd5eed32ac7de69b77458cdb865"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"以步长控制目标梯度","source_units":[{"start":47211,"end":48745,"start_line":583,"end_line":594,"raw_tex":"To verify \\eqref{3.2(ii)}, take any $k=1,\\ldots,K-1$ and obtain the relationships\n\\begin{subequations}\n\\begin{align}\n\\norm{\\nabla f(x^k)}&\\le \\norm{\\nabla f(x^k)-\\nabla f(x^k_{\\rm in})} +\\norm{\\nabla f(x^k_{\\rm in})-\\nabla f(x^k_{\\rm ex})}+\\norm{\\nabla f(x^k_{\\rm ex})-g^k}+\\norm{g^k}\\label{nab.1 triang}\\\\\n&\\le(L\\bar \\beta+L\\bar\\delta)\\norm{x^k-x^{k-1}}+(\\nu+1)\\norm{g^k}\\label{nab.2}\\\\\n&=(L\\bar \\beta+L\\bar\\delta)\\norm{x^k-x^{k-1}}+\\frac{\\nu+1}{\\tau }\\norm{x^k+\\beta_k(x^k-x^{k-1})-x^{k+1}}\\label{nab.3}\\\\\n&\\le \\brac{L\\bar \\beta+L\\bar\\delta+\\frac{\\nu+1}{\\tau }\\bar \\beta}\\norm{x^k-x^{k-1}}+\\frac{\\nu+1}{\\tau }\\norm{x^{k+1}-x^k}\\nonumber\\\\\n&\\le \\max\\set{\\frac{\\nu+1}{\\tau },L\\bar \\beta+L\\bar\\delta+\\frac{\\nu+1}{\\tau }\\bar \\beta}\\brac{\\norm{x^{k+1}-x^k}+\\norm{x^k-x^{k-1}}}\\nonumber\\\\\n&\\le \\sqrt{2}\\max\\set{\\frac{\\nu+1}{\\tau },L\\bar \\beta+L\\bar\\delta+\\frac{\\nu+1}{\\tau }\\bar \\beta}\\norm{z^{k+1}-z^k}\\label{nab.6},\n\\end{align}\n\\end{subequations}\nwhere \\eqref{nab.1 triang} follows from the triangle inequality; \\eqref{nab.2} follows from $\\norm{g^k-\\nabla f(x^k_{\\rm ex})}\\le \\nu\\norm{g^k}$ in \\eqref{iteration update}, the updates in \\eqref{step xin}, \\eqref{step xex}, and the Lipschitz continuity of $\\nabla f$ on $X $ containing $ x^k_\\rin,x^k_\\rex$; \\eqref{nab.3} follows from \\eqref{step xin} and \\eqref{step xex}; \\eqref{nab.6} follows from $a+b\\le \\sqrt{2(a^2+b^2)}$ and $z^k=(x^k,x^{k-1})$. By using the estimates above, we deduce from the construction of $H_\\alpha$ and inequality $\\sqrt{a^2+b^2}\\le a+b$ for $a,b\\ge 0$ that\n","sha256_utf8":"804afb90c37677db56ce206c779103065cafd400057774d273078965ff178233"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"控制提升函数梯度","source_units":[{"start":48745,"end":49247,"start_line":595,"end_line":601,"raw_tex":"\\begin{align*}\n\\norm{\\nabla H_{\\alpha}(z^k)}&= \\sqrt{\\norm{\\nabla f(x^k)+2{\\alpha}(x^k-x^{k-1})}^2+\\norm{2{\\alpha} (x^k-x^{k-1})}^2}\\\\\n&\\le \\norm{\\nabla f(x^k)+2{\\alpha}(x^k-x^{k-1})}+\\norm{2{\\alpha} (x^k-x^{k-1})}\\\\\n& \\le \\norm{\\nabla f(x^k)}+4{\\alpha}\\norm{x^k-x^{k-1}}\\\\\n&\\le \\brac{\\sqrt{2}\\max\\set{\\frac{\\nu+1}{\\tau },L\\bar \\beta+L\\bar\\delta+\\frac{\\nu+1}{\\tau }\\bar \\beta}+4{\\alpha}}\\norm{z^{k+1}-z^k},\n\\end{align*}\nwhich justifies \\eqref{3.2(ii)}  and thus completes the proof of the proposition.\n","sha256_utf8":"7c93760b920fbf04ebd3637b6dba05caa729125d1effba41779980f77df63089"}]},{"id":"S044","semantic_type":"proposition","kind":"proposition","title":"邻域内迭代距离界","source_units":[{"start":49432,"end":49901,"start_line":606,"end_line":612,"raw_tex":"\\begin{Proposition}\\label{proposition distance}\nLet $\\set{x^k}$ be the sequence of iterates generated by Algorithm~{\\rm\\ref{IGDm}} such that $x^k,x^k_{\\rm ex}\\in \\B(\\bar x,\\rho)\\subset X$ for all $k=1,\\ldots,K$. Then we have for all $k=1,\\ldots,K$ that\n\\begin{align}\n\\norm{x^{k+1}-x^k}&\\le   \\frac{L\\tau\\rho }{(1-\\bar \\beta)(1-\\nu)},\\label{3.3(i)}\\\\\n\\norm{z^{k+1}-z^k}&\\le  \\frac{\\sqrt{2} L\\tau\\rho }{(1-\\bar \\beta)(1-\\nu)}.\\label{3.3(ii)}\n\\end{align}\n\\end{Proposition}","sha256_utf8":"d8a23aa74c1081ff7f5fa8997ea5ba27fd47f14adbb8112c2ede13a15b06f96f"}]},{"id":"S045","semantic_type":"proof","kind":"proof","title":"迭代距离界完整证明","source_units":[{"start":49902,"end":51672,"start_line":613,"end_line":633,"raw_tex":"\\begin{proof}\nPick any $k=1,\\ldots,K$ and observe that the iteration update in Algorithm~\\ref{IGDm} can be rewritten as $x^{i+1}=x^i+\\beta_k(x^i-x^{i-1})-\\tau g^i$, which yields\n\\begin{align}\\label{new update}\n\\norm{x^{i+1}-x^i}\\le \\bar \\beta \\norm{x^i-x^{i-1}}+\\tau\\norm{g^i}\\;\\text{ for all }\\;i\\le k.\n\\end{align}\nUsing the triangle inequality together with $\\norm{g^i-\\nabla f(x^i_{\\rm ex})}\\le \\nu \\norm{g^k}$ gives us\n\\begin{align*}\n\\norm{\\nabla f(x^i_{\\rm ex})}\\ge \\norm{g^i}-\\norm{\\nabla f(x^i_{\\rm ex})-g^i}\\ge (1-\\nu)\\norm{g^i}\\;\\text{ for all }\\;i\\le k.\n\\end{align*} \nProceeding inductively from \\eqref{new update} for $i=k,k-1,\\ldots,1$ with the usage of the Lipschitz continuity of $\\nabla f$ on the ball $\\B(\\bar x,\\rho)$ containing $\\bar x$ and $x^{k-i}_\\rex,i=0,\\ldots,k-1$ yields\n\\begin{align*}\n\\norm{x^{k+1}-x^k}&\\le \\bar \\beta\\norm{x^k-x^{k-1}}+\\tau \\norm{g^k}\\\\\n&=\\bar \\beta^2\\norm{x^{k-1}-x^{k-2}}+\\tau\\brac{\\bar \\beta \\norm{g^{k-1}}+\\norm{g^k}}\\\\\n&\\ldots\\\\\n&\\le\\bar \\beta^{k}\\norm{x^1-x^0}+\\tau \\sum_{i=0}^{k-1}\\bar \\beta^i\\norm{g^{k-i}}\\\\\n&\\le \\frac{\\tau}{1-\\nu} \\sum_{i=0}^{k-1}\\bar \\beta^i\\norm{\\nabla f(x_\\rex^{k-i})} = \\frac{\\tau}{1-\\nu}\\sum_{i=0}^{k-1}\\bar \\beta^i\\norm{\\nabla f(x^{k-i}_\\rex)-\\nabla f(\\bar x)}\\\\\n&\\le \\frac{\\tau}{1-\\nu}\\sum_{i=0}^{k-1}\\bar \\beta^i L\\norm{x^{k-i}_\\rex -\\bar x}\\le \\frac{\\tau}{1-\\nu}\\sum_{i=0}^{k-1}\\bar \\beta^i L\\rho\\\\\n&\\le \\frac{L\\tau\\rho}{1-\\nu}\\sum_{i=0}^{\\infty}\\bar \\beta^i = \\frac{L\\tau\\rho}{(1-\\nu)(1-\\bar \\beta)},\n\\end{align*}\nwhere the latter inequality follows from $\\bar \\beta\\in [0,1)$. This readily verifies \\eqref{3.3(i)}. Combining \\eqref{3.3(i)}  with $\\norm{z^{k+1}-z^k}=\\sqrt{\\norm{x^{k+1}-x^k}^2+\\norm{x^{k}-x^{k-1}}^2}$ brings us to \\eqref{3.3(ii)} and thus completes the proof.\n\\end{proof}","sha256_utf8":"3baf4a198c6a0d4efb67a09a64d9cd00234c93fd05f24c3339cb6e53ea250d3a"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"步长递推与误差反估计","source_units":[{"start":49916,"end":50480,"start_line":614,"end_line":621,"raw_tex":"Pick any $k=1,\\ldots,K$ and observe that the iteration update in Algorithm~\\ref{IGDm} can be rewritten as $x^{i+1}=x^i+\\beta_k(x^i-x^{i-1})-\\tau g^i$, which yields\n\\begin{align}\\label{new update}\n\\norm{x^{i+1}-x^i}\\le \\bar \\beta \\norm{x^i-x^{i-1}}+\\tau\\norm{g^i}\\;\\text{ for all }\\;i\\le k.\n\\end{align}\nUsing the triangle inequality together with $\\norm{g^i-\\nabla f(x^i_{\\rm ex})}\\le \\nu \\norm{g^k}$ gives us\n\\begin{align*}\n\\norm{\\nabla f(x^i_{\\rm ex})}\\ge \\norm{g^i}-\\norm{\\nabla f(x^i_{\\rm ex})-g^i}\\ge (1-\\nu)\\norm{g^i}\\;\\text{ for all }\\;i\\le k.\n\\end{align*} \n","sha256_utf8":"dbed6b494476f17945cdf021388fa2f1dc50e925e5abbbc3af226642f3343eac"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"几何求和得到距离界","source_units":[{"start":50480,"end":51661,"start_line":622,"end_line":632,"raw_tex":"Proceeding inductively from \\eqref{new update} for $i=k,k-1,\\ldots,1$ with the usage of the Lipschitz continuity of $\\nabla f$ on the ball $\\B(\\bar x,\\rho)$ containing $\\bar x$ and $x^{k-i}_\\rex,i=0,\\ldots,k-1$ yields\n\\begin{align*}\n\\norm{x^{k+1}-x^k}&\\le \\bar \\beta\\norm{x^k-x^{k-1}}+\\tau \\norm{g^k}\\\\\n&=\\bar \\beta^2\\norm{x^{k-1}-x^{k-2}}+\\tau\\brac{\\bar \\beta \\norm{g^{k-1}}+\\norm{g^k}}\\\\\n&\\ldots\\\\\n&\\le\\bar \\beta^{k}\\norm{x^1-x^0}+\\tau \\sum_{i=0}^{k-1}\\bar \\beta^i\\norm{g^{k-i}}\\\\\n&\\le \\frac{\\tau}{1-\\nu} \\sum_{i=0}^{k-1}\\bar \\beta^i\\norm{\\nabla f(x_\\rex^{k-i})} = \\frac{\\tau}{1-\\nu}\\sum_{i=0}^{k-1}\\bar \\beta^i\\norm{\\nabla f(x^{k-i}_\\rex)-\\nabla f(\\bar x)}\\\\\n&\\le \\frac{\\tau}{1-\\nu}\\sum_{i=0}^{k-1}\\bar \\beta^i L\\norm{x^{k-i}_\\rex -\\bar x}\\le \\frac{\\tau}{1-\\nu}\\sum_{i=0}^{k-1}\\bar \\beta^i L\\rho\\\\\n&\\le \\frac{L\\tau\\rho}{1-\\nu}\\sum_{i=0}^{\\infty}\\bar \\beta^i = \\frac{L\\tau\\rho}{(1-\\nu)(1-\\bar \\beta)},\n\\end{align*}\nwhere the latter inequality follows from $\\bar \\beta\\in [0,1)$. This readily verifies \\eqref{3.3(i)}. Combining \\eqref{3.3(i)}  with $\\norm{z^{k+1}-z^k}=\\sqrt{\\norm{x^{k+1}-x^k}^2+\\norm{x^{k}-x^{k-1}}^2}$ brings us to \\eqref{3.3(ii)} and thus completes the proof.\n","sha256_utf8":"d41bce644988ea87d976c7855931ad5206a5231bffaaa9abccc62d6fe1c2f9ba"}]},{"id":"S048","semantic_type":"proposition","kind":"proposition","title":"收敛极限的驻点性质","source_units":[{"start":51798,"end":52129,"start_line":637,"end_line":638,"raw_tex":"\\begin{Proposition}\\label{convergence xk+1 xk} Let $\\set{x^k}$ be the sequence of iterates generated by Algorithm~{\\rm\\ref{IGDm}} that converges to some $\\tilde x\\in X$. Then $\\tilde x$ is a stationary point of \\eqref{main problem} and the sequences $\\set{x^k_\\rex},\\set{x^k_\\rin}$ converge to $\\tilde x$ as well.\n\\end{Proposition}","sha256_utf8":"e80482e37b66bfd4e63b4c2b5d9d2b80b32dec367452d4821813c17602504105"}]},{"id":"S049","semantic_type":"proof","kind":"proof","title":"极限驻点完整证明","source_units":[{"start":52130,"end":53026,"start_line":639,"end_line":649,"raw_tex":"\\begin{proof} Since the sequence $\\set{x^k}$ is convergent,  we have $\\norm{x^{k+1}-x^k}\\rightarrow 0$ as $k\\rightarrow\\infty.$ Combining this with the updates in \\eqref{step xin} and \\eqref{step xex} together with the boundedness of $\\set{\\beta_k} and \\set{\\gamma_k}$ tells us that\n\\begin{align*}\n{x^k_{\\rm in}}&=x^k+\\beta_k({x^{k}-x^{k-1}})\\rightarrow\\tilde x,\\\\\n{x^k_{\\rm ex}}&=x^k+\\gamma_k({x^{k}-x^{k-1}})\\rightarrow\\tilde x.\n\\end{align*}\nUsing $\\norm{x^{k+1}-x^k}\\rightarrow 0$ and the updates in \\eqref{step xin}, \\eqref{iteration update} of Algorithm~\\ref{IGDm} yields\n\\begin{align*}\ng^k=\\frac{1}{\\tau}\\brac{x^k-x^{k+1}+\\beta_k(x^k-x^{k-1})}\\rightarrow 0\\;\\text{ as }\\;k\\rightarrow\\infty.\n\\end{align*}\nIt follows from $\\norm{\\nabla f(x^k_{\\rm ex})-g^k}\\le \\nu\\norm{g^k}$ in \\eqref{iteration update} that ${\\nabla f(x^k_{\\rm ex})}\\rightarrow 0$, and we are done with the proof.\n\\end{proof}","sha256_utf8":"d1f8439d07901e96be5a6208eef49290c1814574a3c87867d320dffa6344de9e"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"内外推点具有共同极限","source_units":[{"start":52130,"end":52574,"start_line":639,"end_line":643,"raw_tex":"\\begin{proof} Since the sequence $\\set{x^k}$ is convergent,  we have $\\norm{x^{k+1}-x^k}\\rightarrow 0$ as $k\\rightarrow\\infty.$ Combining this with the updates in \\eqref{step xin} and \\eqref{step xex} together with the boundedness of $\\set{\\beta_k} and \\set{\\gamma_k}$ tells us that\n\\begin{align*}\n{x^k_{\\rm in}}&=x^k+\\beta_k({x^{k}-x^{k-1}})\\rightarrow\\tilde x,\\\\\n{x^k_{\\rm ex}}&=x^k+\\gamma_k({x^{k}-x^{k-1}})\\rightarrow\\tilde x.\n\\end{align*}\n","sha256_utf8":"f993fa1be36a575aea6e86c8788cb4800cd07bc632fddaa4d8d75fc8bbcb8f10"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"近似梯度趋于零","source_units":[{"start":52574,"end":52840,"start_line":644,"end_line":647,"raw_tex":"Using $\\norm{x^{k+1}-x^k}\\rightarrow 0$ and the updates in \\eqref{step xin}, \\eqref{iteration update} of Algorithm~\\ref{IGDm} yields\n\\begin{align*}\ng^k=\\frac{1}{\\tau}\\brac{x^k-x^{k+1}+\\beta_k(x^k-x^{k-1})}\\rightarrow 0\\;\\text{ as }\\;k\\rightarrow\\infty.\n\\end{align*}\n","sha256_utf8":"af56e1cf31c91daf3f887c19daeb093c08ff848703614a1f146c7525adc57444"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"由相对误差获得驻点","source_units":[{"start":52840,"end":53015,"start_line":648,"end_line":648,"raw_tex":"It follows from $\\norm{\\nabla f(x^k_{\\rm ex})-g^k}\\le \\nu\\norm{g^k}$ in \\eqref{iteration update} that ${\\nabla f(x^k_{\\rm ex})}\\rightarrow 0$, and we are done with the proof.\n","sha256_utf8":"6b598f7ed514d3ec09bea91eaeb907d238fd707ee5aedfc0b0991e1c603d6db8"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"选择统一步长上界","source_units":[{"start":62706,"end":62974,"start_line":737,"end_line":740,"raw_tex":"\\begin{proof} To verify (i), deduce from $\\bar \\beta \\in[0,\\sqrt{1-\\nu})$ that there exists $T>0$ such that\n\\begin{align}\\label{LT<1}\nT< \\frac{(1-\\bar \\beta)(1-\\nu)}{4L\\sqrt{2}(1+\\bar \\delta)},\\quad(LT+1)\\bar\\beta^2+2LT\\bar \\delta<1-\\nu,\\quad LT\\le 1-\\nu.\n\\end{align}\n","sha256_utf8":"0a8b5527b4cf22bfc1948fd8afd1f2958329fcc0b5ee2e217b59141ee5d44d7e"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"建立临界值隔离与一致PLK","source_units":[{"start":62974,"end":64053,"start_line":741,"end_line":744,"raw_tex":"Define further $\\varepsilon:=\\min\\set{\\frac{1-\\nu}{4T},\\frac{1}{4}}\\in \\big(0,\\frac{1}{4}\\big]$ and deduce from Lemma~\\ref{isolation lemma} due to the imposed assumptions that there exists a neighborhood $U\\subset\\B(\\bar x,r)$ of $\\bar x$ such that $\\bar x$ minimizes $f$ on $U$ and that $f(\\bar x)$ is the only critical value within this set. Remembering that $f$ satisfies the basic PLK condition at $\\bar x$ tells us by Proposition~\\ref{isolated} that the Lyapunov function $\\set{H_\\alpha}_{\\alpha\\ge \\varepsilon}$ enjoys the same PLK property at $\\bar z:=(\\bar x,\\bar x)$ for some $\\eta\\in (0,\\infty]$, some neighborhood $Z$ of $\\bar z:=(\\bar x,\\bar x)$, and a concave desingularizing function $\\varphi:[0,\\eta)\\rightarrow[0,\\infty)$ with $\\varphi(0)=0$ which is ${\\cal C}^1$-smooth on $(0,\\eta)$. This means that for any $\\alpha\\ge \\varepsilon$ and any $z\\in Z$ with $H_\\alpha(\\bar z)< H_\\alpha( z)<H_\\alpha(\\bar z)+\\eta$, we get the inequality\n\\begin{align}\\label{KL R2n in proof}\n\\varphi'\\big(H_\\alpha(z )-H_\\alpha(\\bar z )\\big)\\norm{\\nabla H_\\alpha(z)}\\ge 1.\n\\end{align}\n","sha256_utf8":"426e3c61c30968cbd85c8d077f638a384394f912ceda6049995ddc6484f03c1f"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"选择能量邻域与初始邻域","source_units":[{"start":64053,"end":64643,"start_line":745,"end_line":752,"raw_tex":"The continuity of $f$ allows us to find $\\rho\\in(0,r)$ such that \n\\begin{align}\\label{choice rho}\nf(x)+\\frac{L^2T\\rho^2}{2(1-\\bar \\beta)^2(1-\\nu)^2}< f(\\bar x)+\\eta\\text{ for all }x \\in \\mathbb{B}(\\bar x,\\rho).\n\\end{align}\nInvoking further the continuity of $\\varphi$ at $0$ with $\\varphi(0)=0$ gives us $\\xi\\in (0,\\rho]$ for which\n\\begin{align}\\label{defi xi 1}\n\\sqrt{2}\\norm{x-\\bar x}+\\frac{4\\sqrt{2}\\max\\set{\\nu+1,\\bar\\beta(\\nu+2)+\\bar\\delta}+8}{1-\\nu-(L  T+1)\\bar\\beta^2-2LT \\bar \\delta}\\varphi(f(x)-f(\\bar x))&<\\frac{\\rho}{4}\\;\\text{ for all }\\;x\\in\\mathbb{B}(\\bar x,\\xi).\n\\end{align}\n","sha256_utf8":"05623a7ef1736cb76d7857407f44fb3e17a46508f09cb76f5f3f792b7b53f4d7"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"控制能量权重与下降余量","source_units":[{"start":64643,"end":65424,"start_line":753,"end_line":764,"raw_tex":"Given the sequence of iterates $\\set{x^k}$ generated by Algorithm~\\ref{IGDm} with $x^0\\in \\mathbb{B}(\\bar x, \\xi)$ and $\\tau\\in (0, T]$, take the constant $\\alpha,C_1,C_2$ from \\eqref{C1C2} and deduce that\n\\begin{align}\\label{sele alpha}\n\\alpha=\\frac{(L\\tau+1)\\bar\\beta^2+1-\\nu}{4\\tau}\\ge \\frac{1-\\nu}{4T}\\ge \\varepsilon.\n\\end{align}\nSince $\\tau\\le T$, we get from the selection of $T$ in \\eqref{LT<1} the inequalities\n\\begin{align}\\label{ineq 1-nu}\n(L\\tau+1)\\bar \\beta^2+2L\\tau \\bar \\delta\\le (LT+1)\\bar \\beta^2+2LT \\bar \\delta <1-\\nu.\n\\end{align}\nwhich bring us to the relationships  \n\\begin{align}\\label{ineq 1/2}\n\\alpha=\\frac{(L\\tau+1)\\bar\\beta^2+1-\\nu}{4\\tau}\\le \\frac{2(1-\\nu)-2L\\tau \\bar \\delta}{4\\tau}=\\frac{1-\\nu-L\\tau \\bar \\delta}{2\\tau}\\le \\frac{1}{2\\tau}.\n\\end{align} \n","sha256_utf8":"36fd229ff532be1df5428d48714cc21fc59cc5260629f3d5ea29e10b5ab2a96b"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"统一控制常数比值","source_units":[{"start":65424,"end":66216,"start_line":765,"end_line":774,"raw_tex":"It follows from $L\\tau \\le LT\\le 1$ by \\eqref{LT<1} and the estimates in \\eqref{ineq 1/2}, \\eqref{ineq 1-nu} that\n\\begin{align*}\n\\frac{C_2}{C_1}&= \\frac{\\sqrt{2}\\max\\set{\\frac{\\nu+1}{\\tau },\\frac{\\bar\\beta(L\\tau+\\nu+1)+L\\tau \\bar \\delta}{\\tau }}+4\\alpha}{\\frac{1-\\nu-(L \\tau+1)\\bar \\beta^2-2L\\tau \\bar \\delta}{4\\tau}}\\\\\n&= \\frac{4\\sqrt{2}\\max\\set{{\\nu+1}, \\bar\\beta(L\\tau+\\nu+1)+L\\tau \\bar \\delta}+16\\alpha\\tau}{1-\\nu-(L \\tau+1)\\bar\\beta^2-2L\\tau \\bar \\delta}\\\\\n&\\le \\frac{4\\sqrt{2}\\max\\set{\\nu+1,\\bar\\beta(\\nu+2)+\\bar\\delta}+8}{1-\\nu-(L  T+1)\\bar\\beta^2-2LT \\bar \\delta}.\n\\end{align*}\nThis implies by \\eqref{defi xi 1} that \n\\begin{align}\\label{defi xi 2}\n\\sqrt{2}\\norm{x-\\bar x}+\\frac{C_2}{C_1}\\varphi(f(x)-f(\\bar x))&<\\frac{\\rho}{4} \\text{ for all }x\\in\\mathbb{B}(\\bar x,\\xi).\n\\end{align}\n","sha256_utf8":"6e9a1e5739b25b49c6f4ac12897757015261675dbf773c02774345491bbb33e3"}]},{"id":"S058","semantic_type":"derived_fact","kind":"derived_fact","title":"三类迭代保持在邻域内","source_units":[{"start":66216,"end":66435,"start_line":775,"end_line":778,"raw_tex":"The rest of the proof of assertion (i) is split into two claims.\n\n\\begin{Claim} The sequences $\\set{x^k},\\set{x^k_{\\rm in}}$, and $\\set{x^k_{\\rm ex}}$ stay inside $\\mathbb{B}(\\bar x,\\rho)$ for all $k\\in\\N$.\n\\end{Claim}\n","sha256_utf8":"5ac745bc06931364c4589823a0e6d2b5a15cafbdf7f841ca8a9746c5229a2bf3"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"归纳起点与能量下降","source_units":[{"start":66435,"end":67676,"start_line":779,"end_line":787,"raw_tex":"\\noindent Since $x^0=x^1\\in \\mathbb{B}(\\bar x,\\xi),$ the updates in \\eqref{step xin}, \\eqref{step xex} of Algorithm~\\ref{IGDm} tell us that $x^1_{\\rm in}=x^1_{\\rm ex}=x^1\\in \\mathbb{B}(\\bar x,\\xi)\\subset \\mathbb{B}(\\bar x,\\rho).$ We proceed now by induction to show that $x^k,x^k_{\\rm in},x^k_{\\rm ex}\\in \\mathbb{B}(\\bar x,\\rho)$ for all $k\\in\\N.$ Assuming that $x^k,x^k_\\rin,x^k_{\\rm ex}\\in \\mathbb{B}(\\bar x,\\rho)$ for $k=1,\\ldots,K$, let us verify that $x^{K+1},x_\\rin^{K+1},x^{K+1}_\\rex\\in \\mathbb{B}(\\bar x,\\rho).$ It follows from \\eqref{ineq 1-nu} and $L\\tau\\le LT<1-\\nu$ by \\eqref{LT<1} that assumption \\eqref{max Lt<1-nu} in Proposition~\\ref{prop AIGD tech} is satisfied. Therefore, we get the estimates\n\\begin{align}\\label{descent con for rate 1}\n&C_1 \\norm{z^{k+1}-z^k}^2\\le H_{\\alpha}(z^k)-H_{\\alpha}(z^{k+1}),\\\\\n&\\norm{\\nabla H_{\\alpha}(z^{k})}\\le C_2\\norm{z^{k+1}-z^k}\\text{ for all }k=1,\\ldots,K-1,\\label{descent con for rate 2}\n\\end{align}\nwhere $z^k=(x^k,x^{k-1})$ as before. Fix any such $k,$ we deduce from the above inequalities that\n\\begin{align}\\label{primary descent condi}\n\\norm{z^{k+1}-z^k}\\cdot\\norm{\\nabla H_\\alpha(z^k)}\\le C_2  \\norm{z^{k+1}-z^k}^2\\le \\frac{C_2}{C_1}(H_\\alpha(z^k)-H_\\alpha(z^{k+1})).\n\\end{align}\n","sha256_utf8":"e17e677b266dd732c605a9c6780dde626243ccb9ce2d662b377bc7f5c31f0865"}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","title":"验证PLK邻域与能量上界","source_units":[{"start":67676,"end":68663,"start_line":788,"end_line":797,"raw_tex":"To use inequality \\eqref{KL R2n in proof} with $z:=z^k$, it is sufficient to check that $z^k\\in Z$ and $ H_\\alpha(\\bar z)<H_\\alpha(z^k)<H_\\alpha(\\bar z)+\\eta$. Note first that the former condition is satisfied due to\n\\begin{align*}\nz^k=(x^k,x^{k-1})\\in \\mathbb{B}(\\bar x,\\rho)\\times \\mathbb{B}(\\bar x,\\rho)\\subset Z.\n\\end{align*}\n To verify the latter condition, deduce from Proposition~\\ref{proposition distance} that $\\norm{x^k-x^{k-1}}\\le \\frac{L\\tau\\rho}{(1-\\bar \\beta)(1-\\nu)}$. Combining this with the construction of $H_\\alpha$, the inequality $\\alpha\\le \\frac{1}{2\\tau}$ from \\eqref{ineq 1/2}, the algorithm inclusions $x^k,x^{k-1}\\in\\mathbb{B}(\\bar x,\\rho)$, and the choice of $\\rho$ in \\eqref{choice rho} tells us that\n\\begin{align*}\nH_\\alpha(z^k)&=f(x^k)+\\alpha\\norm{x^k-x^{k-1}}^2\\\\\n&\\le f(x^k)+\\frac{1}{2\\tau}\\cdot\\frac{L^2\\tau^2\\rho^2}{(1-\\bar \\beta)^2(1-\\nu)^2}\\\\\n&= f(x^k)+\\frac{L^2T\\rho^2}{2(1-\\bar \\beta)^2(1-\\nu)^2}< f(\\bar x)+\\eta=H_\\alpha(\\bar z)+\\eta.\n\\end{align*}\n","sha256_utf8":"58964878a28c649277335eebfc6de5ce5cd402dcf5248f8619c5048732ccef1d"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"排除最低能级处停滞","source_units":[{"start":68663,"end":69488,"start_line":798,"end_line":807,"raw_tex":"Observe that the inequality $H_\\alpha(\\bar z)<H_\\alpha(z^k)$ is also satisfied. Indeed, its failure means that \n$$\nf(x^k)+\\alpha\\norm{x^{k}-x^{k-1}}^2\\le H_\\alpha(\\bar z)=f(\\bar x).\n$$\nBy taking into account that $f(x^k)\\ge f(\\bar x)$ by $x^k\\in\\mathbb{B}(\\bar x,\\rho)$, the latter implies that\n\\begin{align*}\nf(x^k)=f(\\bar x) \\text{ and }x^{k}=x^{k-1},\n\\end{align*}\nwhich  means that $\\nabla f(x^k)=0$ since $\\bar x$ is a local minimizer of $f$ within $\\B(\\bar x,\\rho)$. The equalities above also imply that\n$x^k_{\\rm ex}=x^k$ by \\eqref{step xex}, and so $\\nabla f(x^k_{\\rm ex})=0,$ which contradicts Assumption~\\ref{assu} and thus justifies \\eqref{KL R2n in proof} with $z=z^k$. Multiplying both sides of \\eqref{primary descent condi} by $\\varphi'(H_\\alpha(z^k)-H_\\alpha(\\bar z))$ and using \\eqref{KL R2n in proof} give us\n","sha256_utf8":"8361cd867c856c5f9dad7bcf7842d227c29ef56720b24281d378c7cc962973cf"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"PLK与凹性给出长度差分","source_units":[{"start":69488,"end":69991,"start_line":808,"end_line":813,"raw_tex":"\\begin{align}\\label{ineq alter}\n\\norm{z^{k+1}-z^k}& \\le \\varphi'(H_\\alpha(z^k)-H_\\alpha(\\bar z))\\norm{\\nabla H_\\alpha(z^k)}\\cdot\\norm{z^{k+1}-z^k}\\\\\n&\\le \\frac{C_2}{C_1}\\varphi'(H_\\alpha(z^k)-H_\\alpha(\\bar z))\\brac{ H_{\\alpha}(z^k)-H_{\\alpha}(z^{k+1})}\\nonumber\\\\\n&\\le\\frac{C_2}{C_1}\\brac{\\varphi(H_\\alpha(z^k)-H_\\alpha(\\bar z))-\\varphi(H_\\alpha(z^{k+1})-H_\\alpha(\\bar z))} \\text{ for all }k=1,\\ldots,K-1,\n\\end{align}\nwhere the inequality  in \\eqref{chain ineq} follows from the concavity of $\\varphi.$\n","sha256_utf8":"2b6d226141c73295d3a7276d065e53efb38524ff48cff94142b50d329aba1202"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"望远镜求和控制偏移","source_units":[{"start":69991,"end":70896,"start_line":814,"end_line":821,"raw_tex":"Combining further the obtained estimate with $\\tau\\le T<\\frac{(1-\\bar \\beta)(1-\\nu)}{4L\\sqrt{2}}$ from \\eqref{LT<1} and estimate \\eqref{3.3(ii)} from Proposition~\\ref{proposition distance},  we have\n\\begin{align}\n\\norm{z^{K+1}-\\bar z}&\\le\\norm{z^1-\\bar z}+\\norm{z^{K+1}-z^K}+\\sum_{k=1}^{K-1}\\norm{z^{k+1}-z^k}\\nonumber\\\\\n&\\le \\norm{z^1-\\bar z}+\\norm{z^{K+1}-z^K}+\\frac{C_2}{C_1}\\sum_{k=1}^{K-1}\\brac{\\varphi(H_\\alpha(z^k)-H_\\alpha(\\bar z))-\\varphi(H_\\alpha(z^{k+1})-H_\\alpha(\\bar z))}\\nonumber\\\\\n&=\\sqrt{2}\\norm{x^1-\\bar x}+\\frac{\\sqrt{2}L\\rho\\tau}{(1-\\bar \\beta)(1-\\nu)}+\\frac{C_2}{C_1}\\brac{\\varphi(H_\\alpha(z^1)-H_\\alpha(\\bar z))-\\varphi(H_\\alpha(z^{K})-H_\\alpha(\\bar z))}\\nonumber\\\\\n&\\le \\sqrt{2}\\norm{x^1-\\bar x}+\\frac{\\rho}{4}+\\frac{C_2}{C_1}\\varphi(f(x^1)-f(\\bar x))<\\rho/2,\\label{chain ineq}\n\\end{align}\nwhere the last inequality is a consequence of \\eqref{defi xi 2} and $x^1\\in\\B(\\bar x,\\xi)$. \n","sha256_utf8":"1011c3d3cc8d260e99256d4b02eb089d6990b3986b28836f82d79025ff049c08"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"闭合三类迭代的不变性归纳","source_units":[{"start":70896,"end":71920,"start_line":822,"end_line":837,"raw_tex":"Therefore, \n\\begin{align*}\n\\norm{x^{K+1}-\\bar x}\\le \\norm{z^{K+1}-\\bar z}\\le \\rho/2,\n\\end{align*}\nwhich means that $x^{K+1}\\in\\mathbb{B}(\\bar x,\\rho/2)$. Using the updates in \\eqref{step xin} and \\eqref{step xex} at the $(K+1)^{\\rm th}$ iteration with taking into account \\eqref{3.3(ii)} from Proposition~\\ref{proposition distance}, $\\tau\\le T<\\frac{(1-\\bar \\beta)(1-\\nu)}{4L\\sqrt{2}(1+\\bar\\delta)}$ from \\eqref{LT<1}, and  \n\\begin{align*}\n0\\le \\beta_{K+1}<1,\\quad 0\\le \\delta_{K+1}\\le \\delta_{K+1}-\\beta_{K+1}+\\beta_{K+1}\\le \\bar \\delta +1,\n\\end{align*}\nwe arrive at the estimates\n\\begin{align*}\n\\norm{x^{K+1}_\\rin-\\bar x}&\\le \\norm{x^{K+1}-\\bar x}+\\norm{x^{K+1}-x^K}\\le \\rho/2+ \\frac{2L\\tau\\rho}{(1-\\bar \\beta)(1-\\nu)}<\\rho,\\\\\n\\norm{x^{K+1}_\\rex-\\bar x}&\\le \\norm{x^{K+1}-\\bar x}+(1+\\bar \\delta)\\norm{x^{K+1}-x^K}\\le \\rho/2+ \\frac{2L\\tau\\rho(1+\\bar\\delta)}{(1-\\bar \\beta)(1-\\nu)}<\\rho.\n\\end{align*}\nThus $x^{K+1}_\\rin,x^{K+1}_\\rex\\in\\mathbb{B}(\\bar x,\\rho)$, which completes the claimed assertion by induction. \n\\medskip\n\n","sha256_utf8":"b9b3d1d8ae4cd70a52f3917ef54187fd149d9134d4a84b14d56204089a7038b3"}]},{"id":"S065","semantic_type":"derived_fact","kind":"derived_fact","title":"共同收敛至同值局部极小点","source_units":[{"start":71920,"end":72107,"start_line":838,"end_line":839,"raw_tex":"\\noindent\\textbf{Claim~2.} \\textit{The sequences $\\set{x^k},\\set{x^k_{\\rm in}},\\set{x^k_{\\rm ex}}$ converge to a local minimizer $\\tilde x$ of $f$ with $f(\\tilde x)=f(\\bar x)$.}\n\\medskip\n","sha256_utf8":"8ff3c858876eb03d6ae57c1c7cf1dbf4d01a36d3301fcde90e8b7b543bd6d0a4"}]},{"id":"S066","semantic_type":"proof_step","kind":"proof_step","title":"无限求和获得有限轨迹长度","source_units":[{"start":72108,"end":72577,"start_line":841,"end_line":848,"raw_tex":"\\noindent Indeed, it follows from \\eqref{ineq alter} and $x^k,x^k_\\rin,x^k_{\\rm ex}\\in\\mathbb{B}(\\bar x,\\rho)$ from Claim 1 that \n\\begin{align*}\n\\norm{z^{k+1}-z^k}&\\le \\frac{C_2}{C_1}\\brac{\\varphi(H_\\alpha(z^k)-H_\\alpha(\\bar z))-\\varphi(H_\\alpha(z^{k+1})-H_\\alpha(\\bar z))} \\text{ for all }k\\in\\N.\n\\end{align*}\nThis readily implies that \n\\begin{align*}\n\\sum_{k=1}^\\infty \\norm{z^{k+1}-z^k}\\le \\frac{C_2}{C_1}\\varphi(H_\\alpha(z^1)-H_\\alpha(\\bar z))<\\infty.\n\\end{align*}\n","sha256_utf8":"b38971977090787b63b9f3e058d747938898769add72b82b960abfadc5dcbcaf"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"识别极限及其极小值","source_units":[{"start":72577,"end":73056,"start_line":849,"end_line":849,"raw_tex":"Therefore, $\\set{z^k}$ is convergent, and thus $\\set{x^k}$ converges to some $\\tilde x\\in\\overline{\\mathbb{B}}(\\bar x,\\rho)\\subset U$. Using  Proposition~\\ref{convergence xk+1 xk}, we justify the convergence of $\\set{x^k_\\rex},\\set{x^k_\\rin}$ to $\\tilde x$ and that $\\tilde x$ is a stationary point of $f.$ Since $f(\\bar x)$ is the only critical value of $f$ within $U,$ it gives us $f(\\tilde x)=f(\\bar x)$ and shows that $\\tilde x$ is a local minimizer of $f$.\\vspace*{0.05in} \n","sha256_utf8":"51f1bd271ccbd045d650f65db4ec87ad87bb01c05d1d23ef15046bfa0a19f059"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"验证速率条件与非停滞性","source_units":[{"start":73057,"end":74172,"start_line":851,"end_line":858,"raw_tex":"To verify (ii), we employ Proposition~\\ref{general rate}. Letting $\\tilde z:=(\\tilde x,\\tilde x)$ and using \\eqref{descent con for rate 1}, \\eqref{descent con for rate 2} from the proof of Claim~1 in (i) together with $\\set{x^k},\\set{x^k_{\\rm in}},\\set{x^k_{\\rm ex}}\\subset\\mathbb{B}(\\bar x,\\rho)$ give us the estimate\n\\begin{align}\\label{ineq rate grad}\n\\begin{array}{ll}\nC_1 \\norm{z^{k+1}-z^k}^2\\le H_{\\alpha}(z^k)-H_{\\alpha}(z^{k+1}),\\\\\n\\norm{\\nabla H_{\\alpha}(z^{k})}\\le C_2\\norm{z^{k+1}-z^k},\\quad k\\in\\N.\n\\end{array}\n\\end{align}\nObserve from $\\nabla f(x^k_\\rex)\\ne 0$ and $\\norm{g^k-\\nabla f(x^k_\\rex)}\\le \\nu \\norm{g^k}$ that $z^{k+1}\\ne z^k$ for all $k\\in\\N.$ Indeed, the contrary implies by $z^k=(x^{k},x^{k-1})$ that $x^{k+1}=x^k=x^{k-1}$ for some $k\\in\\N$. Using the updates in \\eqref{step xin}, \\eqref{step xex} of Algorithm~\\ref{IGDm} ensures that $x^k_{\\rm ex}=x^k_{\\rm in}=x^k=x^{k+1}$. Then $g^k=0$ by \\eqref{iteration update}, and hence $\\nabla f(x^k_{\\rm ex})=0$ by $\\norm{g^k-\\nabla f(x^k_\\rex)}\\le \\nu \\norm{g^k}$. This contradicts the assumption $\\nabla f(x^k_{\\rm ex})\\ne 0$ for all $k\\in\\N$.\n","sha256_utf8":"d22fb7557f48a029385daf9183497cc928ddf0074dd417fc7cb4b259c2948b51"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"转移PLK指数并得到迭代速率","source_units":[{"start":74173,"end":75523,"start_line":860,"end_line":865,"raw_tex":"Recalling that $f$ satisfies the PLK condition at $\\bar x$ with the desingularizing function $\\psi(t)=Mt^{1-q}$, we deduce from Proposition~\\ref{isolated}(ii) that $H_\\alpha$ satisfies the PLK condition at $(\\bar x,\\bar x)$ with the neighborhood $Z$ taking from \\eqref{KL R2n in proof} and the desingularizing function $\\varphi(t)=\\overline Mt^{1-\\theta}$, where $\\theta=\\max \\set{q,\\frac{1}{2}}\\in (0,1),\\overline{M}>0.$ Taking into account Remark~\\ref{algebraic}, this PLK property also holds at $\\tilde z\\in\\overline{\\mathbb{B}}(\\bar x,\\rho)\\times \\overline{\\mathbb{B}}(\\bar x,\\rho)\\subset Z$ due to $H_\\alpha(\\tilde z)=f(\\tilde x)=f(\\bar x)=H_\\alpha(\\bar z).$  Then Proposition~\\ref{general rate} applied to the function $H_\\alpha$ and sequence $\\set{z^k}\\subset\\R^{n}\\times \\R^n$, with taking into account Remark~\\ref{rmk general rate}, brings us to the following:\n\\begin{itemize}\n\\item For $q\\in[0,1/2],$ we get $\\theta=1/2$, which ensures that $\\set{z^k}$ converges linearly to $(\\tilde x,\\tilde x)$. This implies that $\\set{x^k}$ converges linearly to $\\tilde x.$\n\n\\item For $q\\in (1/2,1),$ we get $\\theta=q\\in (1/2,1)$, which tells us that $\\norm{z^k-(\\tilde x,\\tilde x)}=\\mathcal{O}(k^{-\\frac{1-q}{2q-1}})$ and thus yields the convergence rate $\\norm{x^k-\\tilde x}=\\mathcal{O}(k^{-\\frac{1-q}{2q-1}})$ as $k\\rightarrow\\infty$.\n\\end{itemize}\n","sha256_utf8":"4c2b5c3ae24fdab042f504f5b3aa5e89e741de08fdb28989798af4fd5842858d"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"推导提升函数与梯度速率","source_units":[{"start":75524,"end":76362,"start_line":867,"end_line":872,"raw_tex":"Finally, we employ Proposition~\\ref{convergence rate deduce} to verify the convergence rates of $\\set{\\nabla f(x^k)}$ and $\\set{f(x^k)}.$ Since $f$ is of class ${\\cal C}^{1,1}$ on $\\B(\\bar x,r)$ and since $H_\\alpha$ is of class ${\\cal C}^{1,1}$ on $\\B(\\bar x,r)\\times \\B(\\bar x,r)\\supset \\set{z^k}$, condition \\eqref{main condition rate} follows from \\eqref{ineq rate grad}. This clearly leads us to the conclusions:\n\\begin{itemize}\n\\item For $q\\in[0,1/2),$ we get $\\theta=1/2$. Therefore, $\\{H_\\alpha(z^k)\\}$ converges linearly to $H_\\alpha(\\bar z)$ and $\\{\\norm{\\nabla H_\\alpha(z^k)}\\}$ converges linearly to $0$ as $k\\to\\infty$.\n\n\\item For $q\\in (1/2,1),$ we have $H_\\alpha(z^k)-H_\\alpha(\\tilde z)=\\mathcal{O}(k^{-\\frac{2-2q}{2q-1}})$ and $\\norm{\\nabla H_\\alpha(z^k)}=\\mathcal{O}(k^{-\\frac{1-q}{2q-1}})$ as $k\\to\\infty$.\n\\end{itemize}\n","sha256_utf8":"ea5e666ba4d88cefd0abad4cf24704afdfea8d88290e8449387e88790a2e8411"}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","title":"返回原目标并结束证明","source_units":[{"start":76362,"end":76827,"start_line":873,"end_line":878,"raw_tex":"The claimed convergence rates of the sequences $\\{\\norm{\\nabla f(x^k)}\\}$ and $\\set{f(x^k)}$ are now deduced from those for $\\set{\\norm{\\nabla H_\\alpha (z^k)}},\\set{H_\\alpha (z^k)}$ and the estimates\n\\begin{align*}\n\\norm{\\nabla H_\\alpha(x,y)}&=\\sqrt{\\norm{\\nabla f(x)+2\\alpha (x-y)}^2+4\\alpha^2\\norm{x-y}^2}\\ge \\norm{\\nabla f(x)},\\\\\nH_\\alpha(x,y)&=f(x)+\\alpha\\norm{x-y}^2\\ge f(x)\\;\\text{ for all }\\;x,y\\in\\R^n.\n\\end{align*}\nThis completes the proof of the theorem.\n","sha256_utf8":"541b4e2cfc6cc7ea238fe3368d322fb938ed45516cdb5356eb50550fadc7fd4d"}]},{"id":"S072","semantic_type":"external_fact","kind":"external_fact","title":"标准光滑下降估计","source_units":[],"declared_external":true,"declaration_text":["If the gradient of a differentiable function is L-Lipschitz on the segment joining x and y, then |f(y)-f(x)-<∇f(x),y-x>| ≤ (L/2)||y-x||². This is the cited descent lemma recalled in B0011."]},{"id":"S073","semantic_type":"external_fact","kind":"external_fact","title":"标准PLK下降收敛准则","source_units":[],"declared_external":true,"declaration_text":["For a C¹ function and an iterate sequence, eventual primary descent f(x^k)-f(x^{k+1}) ≥ σ||∇f(x^k)|| ||x^{k+1}-x^k|| together with complementary descent f(x^{k+1})=f(x^k) ⇒ x^{k+1}=x^k ensures convergence to an accumulation point at which the PLK property holds. This is the cited criterion recalled in B0019."]},{"id":"S074","semantic_type":"external_fact","kind":"external_fact","title":"标准指数PLK速率定理","source_units":[],"declared_external":true,"declaration_text":["Under eventual sufficient quadratic descent and the gradient-step bound stated in B0022, an accumulation point satisfying PLK with desingularizing function Mt^(1-q) gives finite termination for 0<q<1/2, linear iterate convergence for q=1/2, and iterate error O(k^(-(1-q)/(2q-1))) for 1/2<q<1. The local smooth-domain application for an already convergent sequence is as recalled in B0024."]},{"id":"S075","semantic_type":"external_fact","kind":"external_fact","title":"标准收敛速率转换","source_units":[],"declared_external":true,"declaration_text":["For a convergent sequence in an open convex domain where ∇f is Lipschitz, if a||∇f(x^k)||² ≤ f(x^k)-f(x^{k+1}) eventually for a>0, linear iterate convergence gives linear function-value and gradient convergence; iterate error O(m(k)) with m(k) decreasing to zero gives function-value error O(m(k)²) and gradient norm O(m(k)). This is the cited result recalled in B0045."]},{"id":"S076","semantic_type":"external_fact","kind":"external_fact","title":"初等分析与欧氏空间事实","source_units":[],"declared_external":true,"declaration_text":["Use the triangle, Cauchy–Schwarz, AM–GM and elementary squared-norm inequalities; convergence of geometric series with ratio in [0,1); convergence of Euclidean sequences with summable successive distances; continuity and differentiation of smooth functions; zero gradient at an interior differentiable local minimizer; reduction of smooth subdifferentials to the gradient; and elementary integration, monotonicity and first-order inequalities for differentiable concave functions."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S010","kind":"supports"},{"source_id":"S004","target_id":"S035","kind":"supports"},{"source_id":"S004","target_id":"S037","kind":"supports"},{"source_id":"S004","target_id":"S042","kind":"supports"},{"source_id":"S004","target_id":"S047","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S006","target_id":"S007","kind":"supports"},{"source_id":"S006","target_id":"S020","kind":"supports"},{"source_id":"S006","target_id":"S024","kind":"supports"},{"source_id":"S006","target_id":"S054","kind":"supports"},{"source_id":"S007","target_id":"S069","kind":"supports"},{"source_id":"S008","target_id":"S024","kind":"supports"},{"source_id":"S009","target_id":"S026","kind":"supports"},{"source_id":"S009","target_id":"S027","kind":"supports"},{"source_id":"S009","target_id":"S041","kind":"supports"},{"source_id":"S009","target_id":"S043","kind":"supports"},{"source_id":"S009","target_id":"S060","kind":"supports"},{"source_id":"S009","target_id":"S061","kind":"supports"},{"source_id":"S009","target_id":"S071","kind":"supports"},{"source_id":"S010","target_id":"S036","kind":"supports"},{"source_id":"S010","target_id":"S037","kind":"supports"},{"source_id":"S010","target_id":"S038","kind":"supports"},{"source_id":"S010","target_id":"S039","kind":"supports"},{"source_id":"S010","target_id":"S042","kind":"supports"},{"source_id":"S010","target_id":"S046","kind":"supports"},{"source_id":"S010","target_id":"S047","kind":"supports"},{"source_id":"S010","target_id":"S050","kind":"supports"},{"source_id":"S010","target_id":"S051","kind":"supports"},{"source_id":"S010","target_id":"S052","kind":"supports"},{"source_id":"S010","target_id":"S053","kind":"supports"},{"source_id":"S010","target_id":"S059","kind":"supports"},{"source_id":"S010","target_id":"S061","kind":"supports"},{"source_id":"S010","target_id":"S064","kind":"supports"},{"source_id":"S010","target_id":"S068","kind":"supports"},{"source_id":"S011","target_id":"S041","kind":"supports"},{"source_id":"S011","target_id":"S042","kind":"supports"},{"source_id":"S011","target_id":"S047","kind":"supports"},{"source_id":"S011","target_id":"S059","kind":"supports"},{"source_id":"S011","target_id":"S063","kind":"supports"},{"source_id":"S011","target_id":"S068","kind":"supports"},{"source_id":"S012","target_id":"S061","kind":"supports"},{"source_id":"S012","target_id":"S068","kind":"supports"},{"source_id":"S013","target_id":"S069","kind":"supports"},{"source_id":"S014","target_id":"S035","kind":"supports"},{"source_id":"S015","target_id":"S013","kind":"supports"},{"source_id":"S016","target_id":"S013","kind":"supports"},{"source_id":"S016","target_id":"S069","kind":"supports"},{"source_id":"S017","target_id":"S070","kind":"supports"},{"source_id":"S018","target_id":"S054","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S019","kind":"supports"},{"source_id":"S022","target_id":"S054","kind":"supports"},{"source_id":"S022","target_id":"S069","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S030","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S028","kind":"supports"},{"source_id":"S025","target_id":"S029","kind":"supports"},{"source_id":"S025","target_id":"S023","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S028","kind":"supports"},{"source_id":"S026","target_id":"S023","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S023","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S023","kind":"supports"},{"source_id":"S029","target_id":"S023","kind":"supports"},{"source_id":"S030","target_id":"S023","kind":"supports"},{"source_id":"S031","target_id":"S024","kind":"supports"},{"source_id":"S031","target_id":"S030","kind":"supports"},{"source_id":"S032","target_id":"S059","kind":"supports"},{"source_id":"S032","target_id":"S068","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S034","target_id":"S040","kind":"supports"},{"source_id":"S034","target_id":"S041","kind":"supports"},{"source_id":"S034","target_id":"S043","kind":"supports"},{"source_id":"S034","target_id":"S056","kind":"supports"},{"source_id":"S034","target_id":"S057","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S033","kind":"supports"},{"source_id":"S036","target_id":"S040","kind":"supports"},{"source_id":"S036","target_id":"S033","kind":"supports"},{"source_id":"S037","target_id":"S040","kind":"supports"},{"source_id":"S037","target_id":"S033","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S038","target_id":"S033","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S033","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S033","kind":"supports"},{"source_id":"S041","target_id":"S033","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S033","kind":"supports"},{"source_id":"S043","target_id":"S033","kind":"supports"},{"source_id":"S044","target_id":"S060","kind":"supports"},{"source_id":"S044","target_id":"S063","kind":"supports"},{"source_id":"S044","target_id":"S064","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S048","target_id":"S067","kind":"supports"},{"source_id":"S049","target_id":"S048","kind":"supports"},{"source_id":"S050","target_id":"S052","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S051","target_id":"S052","kind":"supports"},{"source_id":"S051","target_id":"S049","kind":"supports"},{"source_id":"S052","target_id":"S049","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S053","target_id":"S056","kind":"supports"},{"source_id":"S053","target_id":"S057","kind":"supports"},{"source_id":"S053","target_id":"S059","kind":"supports"},{"source_id":"S053","target_id":"S063","kind":"supports"},{"source_id":"S053","target_id":"S064","kind":"supports"},{"source_id":"S053","target_id":"S002","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S060","kind":"supports"},{"source_id":"S054","target_id":"S061","kind":"supports"},{"source_id":"S054","target_id":"S062","kind":"supports"},{"source_id":"S054","target_id":"S067","kind":"supports"},{"source_id":"S054","target_id":"S069","kind":"supports"},{"source_id":"S054","target_id":"S002","kind":"supports"},{"source_id":"S055","target_id":"S057","kind":"supports"},{"source_id":"S055","target_id":"S060","kind":"supports"},{"source_id":"S055","target_id":"S002","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S056","target_id":"S059","kind":"supports"},{"source_id":"S056","target_id":"S060","kind":"supports"},{"source_id":"S056","target_id":"S062","kind":"supports"},{"source_id":"S056","target_id":"S002","kind":"supports"},{"source_id":"S057","target_id":"S063","kind":"supports"},{"source_id":"S057","target_id":"S002","kind":"supports"},{"source_id":"S058","target_id":"S066","kind":"supports"},{"source_id":"S058","target_id":"S067","kind":"supports"},{"source_id":"S058","target_id":"S068","kind":"supports"},{"source_id":"S058","target_id":"S002","kind":"supports"},{"source_id":"S059","target_id":"S062","kind":"supports"},{"source_id":"S059","target_id":"S002","kind":"supports"},{"source_id":"S060","target_id":"S062","kind":"supports"},{"source_id":"S060","target_id":"S002","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S061","target_id":"S002","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S062","target_id":"S066","kind":"supports"},{"source_id":"S062","target_id":"S002","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S063","target_id":"S002","kind":"supports"},{"source_id":"S064","target_id":"S058","kind":"supports"},{"source_id":"S064","target_id":"S002","kind":"supports"},{"source_id":"S065","target_id":"S069","kind":"supports"},{"source_id":"S065","target_id":"S070","kind":"supports"},{"source_id":"S065","target_id":"S002","kind":"supports"},{"source_id":"S066","target_id":"S067","kind":"supports"},{"source_id":"S066","target_id":"S002","kind":"supports"},{"source_id":"S067","target_id":"S065","kind":"supports"},{"source_id":"S067","target_id":"S002","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S070","kind":"supports"},{"source_id":"S068","target_id":"S002","kind":"supports"},{"source_id":"S069","target_id":"S070","kind":"supports"},{"source_id":"S069","target_id":"S071","kind":"supports"},{"source_id":"S069","target_id":"S002","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S002","kind":"supports"},{"source_id":"S071","target_id":"S002","kind":"supports"},{"source_id":"S072","target_id":"S014","kind":"supports"},{"source_id":"S073","target_id":"S015","kind":"supports"},{"source_id":"S074","target_id":"S016","kind":"supports"},{"source_id":"S075","target_id":"S017","kind":"supports"},{"source_id":"S076","target_id":"S005","kind":"supports"},{"source_id":"S076","target_id":"S008","kind":"supports"},{"source_id":"S076","target_id":"S024","kind":"supports"},{"source_id":"S076","target_id":"S025","kind":"supports"},{"source_id":"S076","target_id":"S027","kind":"supports"},{"source_id":"S076","target_id":"S028","kind":"supports"},{"source_id":"S076","target_id":"S029","kind":"supports"},{"source_id":"S076","target_id":"S030","kind":"supports"},{"source_id":"S076","target_id":"S037","kind":"supports"},{"source_id":"S076","target_id":"S038","kind":"supports"},{"source_id":"S076","target_id":"S039","kind":"supports"},{"source_id":"S076","target_id":"S042","kind":"supports"},{"source_id":"S076","target_id":"S043","kind":"supports"},{"source_id":"S076","target_id":"S047","kind":"supports"},{"source_id":"S076","target_id":"S052","kind":"supports"},{"source_id":"S076","target_id":"S055","kind":"supports"},{"source_id":"S076","target_id":"S061","kind":"supports"},{"source_id":"S076","target_id":"S062","kind":"supports"},{"source_id":"S076","target_id":"S067","kind":"supports"}]}}
{"record_id":"2506.10849v4","paper":{"title":"Bregman proximal gradient method for linear optimization under entropic constraints","authors":["Luis M. Briceño-Arias","Maël Le Treust"],"arxiv_id":"2506.10849","version":"v4","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-026-03017-y","abstract_url":"https://arxiv.org/abs/2506.10849v4","pdf_url":"https://arxiv.org/pdf/2506.10849v4"},"context":{"raw_tex":" For every nonempty finite \nset $I$, we denote by \n$\\Delta(I)=\\menge{x\\in[0,1]^{{I}}}{\\sum_{i\\in \nI}x_i=1}$ the set of probability distributions over $I$, \n{$\\reli(C)=\\menge{x\\in C}{{\\rm cone}(C-x)={\\rm \nspan}(C-x)}$ stands \nfor the relative interior of a set $C\\subset \\R^I$, where ${\\rm \ncone}(C)$ and ${\\rm span}(C)$ are the smallest cone (conical \nhull) and linear subspace (span) containing $C$, \nrespectively.}\n\\begin{problem}\\label{prob:main}\nLet $A$, $B$, and $S$ be nonempty finite sets such that $|A|\\ge \n2$, let $(c_{ab}^s)_{a\\in A,\\, b\\in B}^{s\\in S}\\in \\R^{(A\\times \nB)^S}$ a vector of cost values, and let $p=(p_s)_{s\\in \nS}\\in{\\reli(\\Delta(S))}$ a probability distribution. \nThe problem is to\n\\begin{align}\n\\label{e:PrimalProblem}\n\\minimize{q\\in C}{\\sum_{s\\in \nS}p_s\\sum_{a\\in A}\\sum_{b\\in B}c_{ab}^sq_{ab}^s},\n\\end{align}\nwhere \n\\begin{equation}\n\\label{e:defC}\nC=\\Menge{q\\in(\\Delta(A\\times \nB))^S}{\\sum_{s\\in \nS}p_s\\sum_{a\\in A}\\sum_{b\\in B}q_{ab}^s\\ln \n\\Bigg(\\frac{q_{ab}^s}{\\sum\\limits_{a'\\in A}\\sum\\limits_{s'\\in \nS}p_{s'}q_{a'b}^{s'}}\\Bigg)\\leq \n0}\n\\end{equation}\nwith the convention $0\\cdot\\ln 0=0$.\n\\end{problem}\n\nFor every finite set $I$ and $q\\in \\R^I$, we define \n$\\supp(q)=\\menge{i\\in I}{q_i\\ne 0}$, for every nonempty closed \nconvex \nset $X\\subset \\R^{I}$, let\n\\begin{equation}\n\\iota_{X}\\colon q\\mapsto \n\\begin{cases}\n0,\\quad &\\text{ if }q\\in X;\\\\\n\\pinf,&\\text{ otherwise,}\n\\end{cases}\n\\end{equation}\n\n\\begin{equation}\n\\label{e:Kullback}\nD_I\\colon \\R^{I}\\times\\R^{I}\\to\\RX\\colon (q,u)\\mapsto \n\\sum\\limits_{i\\in I}\n\\psi(q_i,u_i),\n\\end{equation}\n\nwhere\n$\\psi\\colon\\R^2\\to\\RX$ is \n\\begin{equation}\n\\label{e:defpsi}\n\\psi\\colon (\\xi,\\mu)\\mapsto \n\\begin{cases}\n\\xi\\ln(\\xi/\\mu),\\quad &\\text{if}\\:\\: \n\\xi> 0\\:\\:\\text{and}\\:\\:\\mu>0;\\\\\n0,&\\text{if}\\:\\:\\xi=0\\:\\:\\text{and}\\:\\:\\mu\\ge 0;\\\\\n\\pinf,&\\text{otherwise}.\n\\end{cases}\n\\end{equation}\n\nNote that $D_I|_{\\Delta(I)\\times \\Delta(I)}$ is the \nKullback-Leibler \ndivergence \n\\cite[Section~2.3]{cover-book-2006}, which turns out to be a \nBregman distance \\cite[Example~9.10]{Beck2017}. More precisely,\n{\n\\begin{equation}\n\\label{e:Bdist}\n(\\forall q\\in \\R^I)(\\forall u\\in\\RPP^I)\\quad \nD_I(q,u)=\\varphi_I(q)-\\varphi_I(u)-\\scal{\\nabla\\varphi_I(u)}{q-u},\n\\end{equation}\n\nwhere\n\\begin{equation}\n\\label{e:defvarphi}\n\\varphi_I\\colon \\R^I\\to \\RX\\colon q\\mapsto \\sum_{i\\in I}\\phi(q_i)\n\\end{equation}\n\nand \n\\begin{equation}\n\\label{e:defphi}\n\\phi\\colon \\xi\\mapsto\n\\begin{cases}\n\\xi\\ln\\xi,\\quad&\\text{if}\\:\\: \\xi> 0;\\\\\n0,&\\text{if}\\:\\: \\xi=0;\\\\\n\\pinf,&\\text{otherwise}\n\\end{cases}\n\\end{equation}\n\nis a convex lower semicontinuous proper function\n\\cite[Example~9.35]{BauschkeCombettes17}. \nIt follows from \\cite[Example~2.1]{Teboulle18} that $\\varphi_I$ is a \nLegendre function, i.e., is \nproper, lower semicontinuous, strictly convex, and essentially \nsmooth. The last property is satisfied by functions which are\n differentiable in \nthe interior of its domain (assumed nonempty) and the norm of its \ngradient evaluated in sequences converging to the boundary \nof its domain goes to infinity (see, e.g., \n\\cite[Definition~2.1]{Teboulle18} for more details). \n}\n\n\n\\begin{equation}\n\\label{e:defg}\ng\\colon\\R^{(A\\times B)^S}\\to \\RX\\colon q \\mapsto \n\\begin{cases}\n\\sum\\limits_{s\\in S}p_s\\sum\\limits_{a\\in A}\\sum\\limits_{b\\in \nB}\\psi\\bigg(q_{ab}^s,\\sum\\limits_{a'\\in A}\\sum\\limits_{s'\\in \nS}p_{s'}q_{a'b}^{s'}\\bigg),&\\text{if}\\:\\:\n q\\in \n[0,+\\infty[^{(A\\times B)^S};\\\\\n\\pinf,&\\text{otherwise}.\n\\end{cases}\n\\end{equation}","units":[{"start":17078,"end":18212,"start_line":435,"end_line":467,"raw_tex":" For every nonempty finite \nset $I$, we denote by \n$\\Delta(I)=\\menge{x\\in[0,1]^{{I}}}{\\sum_{i\\in \nI}x_i=1}$ the set of probability distributions over $I$, \n{$\\reli(C)=\\menge{x\\in C}{{\\rm cone}(C-x)={\\rm \nspan}(C-x)}$ stands \nfor the relative interior of a set $C\\subset \\R^I$, where ${\\rm \ncone}(C)$ and ${\\rm span}(C)$ are the smallest cone (conical \nhull) and linear subspace (span) containing $C$, \nrespectively.}\n\\begin{problem}\\label{prob:main}\nLet $A$, $B$, and $S$ be nonempty finite sets such that $|A|\\ge \n2$, let $(c_{ab}^s)_{a\\in A,\\, b\\in B}^{s\\in S}\\in \\R^{(A\\times \nB)^S}$ a vector of cost values, and let $p=(p_s)_{s\\in \nS}\\in{\\reli(\\Delta(S))}$ a probability distribution. \nThe problem is to\n\\begin{align}\n\\label{e:PrimalProblem}\n\\minimize{q\\in C}{\\sum_{s\\in \nS}p_s\\sum_{a\\in A}\\sum_{b\\in B}c_{ab}^sq_{ab}^s},\n\\end{align}\nwhere \n\\begin{equation}\n\\label{e:defC}\nC=\\Menge{q\\in(\\Delta(A\\times \nB))^S}{\\sum_{s\\in \nS}p_s\\sum_{a\\in A}\\sum_{b\\in B}q_{ab}^s\\ln \n\\Bigg(\\frac{q_{ab}^s}{\\sum\\limits_{a'\\in A}\\sum\\limits_{s'\\in \nS}p_{s'}q_{a'b}^{s'}}\\Bigg)\\leq \n0}\n\\end{equation}\nwith the convention $0\\cdot\\ln 0=0$.\n\\end{problem}","sha256_utf8":"a5542eb026cd81ce393922f0007e4d5c2b2a6ca16e39e746046c2953c6539fd3"},{"start":31840,"end":32131,"start_line":782,"end_line":792,"raw_tex":"For every finite set $I$ and $q\\in \\R^I$, we define \n$\\supp(q)=\\menge{i\\in I}{q_i\\ne 0}$, for every nonempty closed \nconvex \nset $X\\subset \\R^{I}$, let\n\\begin{equation}\n\\iota_{X}\\colon q\\mapsto \n\\begin{cases}\n0,\\quad &\\text{ if }q\\in X;\\\\\n\\pinf,&\\text{ otherwise,}\n\\end{cases}\n\\end{equation}","sha256_utf8":"f580e77b1bb172079fad1f1b782c842cbe361b5d7177ca28b0cdc730aca0991e"},{"start":32144,"end":32285,"start_line":794,"end_line":799,"raw_tex":"\\begin{equation}\n\\label{e:Kullback}\nD_I\\colon \\R^{I}\\times\\R^{I}\\to\\RX\\colon (q,u)\\mapsto \n\\sum\\limits_{i\\in I}\n\\psi(q_i,u_i),\n\\end{equation}","sha256_utf8":"ba05c0f425c09395abaefc4815d86f78b3b4070d3647b70fe949ac8db7cc896b"},{"start":32286,"end":32569,"start_line":800,"end_line":811,"raw_tex":"where\n$\\psi\\colon\\R^2\\to\\RX$ is \n\\begin{equation}\n\\label{e:defpsi}\n\\psi\\colon (\\xi,\\mu)\\mapsto \n\\begin{cases}\n\\xi\\ln(\\xi/\\mu),\\quad &\\text{if}\\:\\: \n\\xi> 0\\:\\:\\text{and}\\:\\:\\mu>0;\\\\\n0,&\\text{if}\\:\\:\\xi=0\\:\\:\\text{and}\\:\\:\\mu\\ge 0;\\\\\n\\pinf,&\\text{otherwise}.\n\\end{cases}\n\\end{equation}","sha256_utf8":"3848d66fd28402dfb389c6e9e0d9b854753951c4374e26e10d6630ab78bfd2a0"},{"start":32570,"end":32941,"start_line":812,"end_line":822,"raw_tex":"Note that $D_I|_{\\Delta(I)\\times \\Delta(I)}$ is the \nKullback-Leibler \ndivergence \n\\cite[Section~2.3]{cover-book-2006}, which turns out to be a \nBregman distance \\cite[Example~9.10]{Beck2017}. More precisely,\n{\n\\begin{equation}\n\\label{e:Bdist}\n(\\forall q\\in \\R^I)(\\forall u\\in\\RPP^I)\\quad \nD_I(q,u)=\\varphi_I(q)-\\varphi_I(u)-\\scal{\\nabla\\varphi_I(u)}{q-u},\n\\end{equation}","sha256_utf8":"d9e76e8ffd5ec43f75f6b45b3dd66b0116207ff42bb1ba8fb2c2d307304ee46f"},{"start":32942,"end":33065,"start_line":823,"end_line":827,"raw_tex":"where\n\\begin{equation}\n\\label{e:defvarphi}\n\\varphi_I\\colon \\R^I\\to \\RX\\colon q\\mapsto \\sum_{i\\in I}\\phi(q_i)\n\\end{equation}","sha256_utf8":"132819a7f27e320de88dc74e6a6402b0a9a4eb804ebfd2a28b5e0815a31a5086"},{"start":33066,"end":33257,"start_line":828,"end_line":837,"raw_tex":"and \n\\begin{equation}\n\\label{e:defphi}\n\\phi\\colon \\xi\\mapsto\n\\begin{cases}\n\\xi\\ln\\xi,\\quad&\\text{if}\\:\\: \\xi> 0;\\\\\n0,&\\text{if}\\:\\: \\xi=0;\\\\\n\\pinf,&\\text{otherwise}\n\\end{cases}\n\\end{equation}","sha256_utf8":"4cad0dfa0a55b7718b182684eb2043108c9ddef10064f4c85458361ce25d9dce"},{"start":33258,"end":33822,"start_line":838,"end_line":849,"raw_tex":"is a convex lower semicontinuous proper function\n\\cite[Example~9.35]{BauschkeCombettes17}. \nIt follows from \\cite[Example~2.1]{Teboulle18} that $\\varphi_I$ is a \nLegendre function, i.e., is \nproper, lower semicontinuous, strictly convex, and essentially \nsmooth. The last property is satisfied by functions which are\n differentiable in \nthe interior of its domain (assumed nonempty) and the norm of its \ngradient evaluated in sequences converging to the boundary \nof its domain goes to infinity (see, e.g., \n\\cite[Definition~2.1]{Teboulle18} for more details). \n}\n","sha256_utf8":"b8387d3bda101c713aa947cfe9a6a72d9f8790ed747f41df06ddbeb0407db2fa"},{"start":37622,"end":37973,"start_line":967,"end_line":978,"raw_tex":"\\begin{equation}\n\\label{e:defg}\ng\\colon\\R^{(A\\times B)^S}\\to \\RX\\colon q \\mapsto \n\\begin{cases}\n\\sum\\limits_{s\\in S}p_s\\sum\\limits_{a\\in A}\\sum\\limits_{b\\in \nB}\\psi\\bigg(q_{ab}^s,\\sum\\limits_{a'\\in A}\\sum\\limits_{s'\\in \nS}p_{s'}q_{a'b}^{s'}\\bigg),&\\text{if}\\:\\:\n q\\in \n[0,+\\infty[^{(A\\times B)^S};\\\\\n\\pinf,&\\text{otherwise}.\n\\end{cases}\n\\end{equation}","sha256_utf8":"7c805f38bbc1b38cc431870bbc2c30601a36e65e0e4ce2c6a95e89d751d41cde"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\n\\label{t:convgen}\nFor every $s\\in S$, let $\\Xi_s\\subset A\\times B$ {and, \nfor every $(a,b)\\in\\Xi_s$, let $d_{ab}^s\\in\\R$.\nMoreover, let $\\eta\\ge 0$,} denote by \n$\\Xi=\\prod_{s\\in S}\\Xi_s$ and by $\\Delta_\\Xi=\\prod_{s\\in \nS}\\Delta(\\Xi_s)$, and consider the problem\n\\begin{equation}\n\\label{e:probXi}\\tag{$P_{\\Xi}$}\n\\min_{q\\in\\Delta_{\\Xi}}F(q,\\eta):={g_{\\Xi}(q)}\n+\\eta\\sum_{s\\in S}\n\\sum_{(a,b)\\in \\Xi_s}d_{ab}^sq_{ab}^s,\n\\end{equation}\nwhere {$g_{\\Xi}$ is the restriction of $g$ \ndefined in \n\\eqref{e:defg} to $\\RP^\\Xi$.} Then \nthe following hold:\n\\begin{enumerate}\n\\item\\label{t:convgeni} {The set of solutions to \n\\eqref{e:probXi} is \nnonempty.\n\\item\\label{t:convgenii}  Given $q^0\\in {\\rm \nri}(\\Delta_\\Xi)$, \nconsider the sequence generated \nby the recurrence \n\\begin{equation}\n\\label{e:recursXi}\n(\\forall n\\in\\NN)(\\forall s\\in S)(\\forall (a,b)\\in \\Xi_s)\\quad \nq_{ab}^{n+1,s}=\n\\frac{(T_\\Xi(q^n))_be^{-\\eta \nd_{ab}^s/p_s}}{\\sum\\limits_{(a',b')\\in\\Xi_s}\n(T_\\Xi(q^n))_{b'}e^{-\\eta \nd_{a'b'}^s/p_s}}, \n\\end{equation}\nwhere $T_{\\Xi}\\colon\\Delta_\\Xi\\to \n\\Delta(B)$ is defined by \n\\begin{equation}\n\\label{e:TXi}\nT{_\\Xi}\\colon \nq\\mapsto \\left(\\sum_{a\\in A}\\sum_{s\\in S}p_sq_{ab}^s\\right)_{b\\in \nB}.\n\\end{equation}\nThen, the following hold.\n\\begin{enumerate}\n\\item \\label{t:convgeniia}For every solution $\\overline{q}$ to \n\\eqref{e:probXi}, \n\\begin{equation}\n\\label{e:o1k}\n(\\forall n\\in\\NN)\\quad 0\\le \nn(F(q^{n},\\eta)\n-F(\\overline{q},\\eta))+\\sum_{k=0}^{n-1}\nD_B(T_{\\Xi}(q^{k+1}),T_{\\Xi}(q^{k}))\\le \n\\sum_{s\\in S}p_sD_{A\\times B}(\\overline{q}^s,q^{0,s}).\n\\end{equation}\n\\item\\label{t:convgeniib} For every solution $\\overline{q}$ to \n\\eqref{e:probXi}, \n$$0\\le F(q^{n},\\eta)\n-F(\\overline{q},\\eta)\\le \\frac{1}{n}\\sum_{s\\in S}p_sD_{A\\times \nB}(\\overline{q}^s,q^{0,s})$$ \nand \n$\\sum_{k=1}^{\\infty}D_B(T_{\\Xi}(q^{k+1}),T_{\\Xi}(q^{k}))<+\\infty$.\n\n\\item \\label{t:convgeniic}\nThe sequence $(q_n)_{n\\in\\NN}$ converges to some solution \n$\\overline{q}$ to \n\\eqref{e:probXi}. \n\\end{enumerate}\n}\n\n\n\n\n\n\n\n\n\\end{enumerate}\n\n\\end{theorem}","units":[{"start":40577,"end":42602,"start_line":1066,"end_line":1144,"raw_tex":"\\begin{theorem}\n\\label{t:convgen}\nFor every $s\\in S$, let $\\Xi_s\\subset A\\times B$ {and, \nfor every $(a,b)\\in\\Xi_s$, let $d_{ab}^s\\in\\R$.\nMoreover, let $\\eta\\ge 0$,} denote by \n$\\Xi=\\prod_{s\\in S}\\Xi_s$ and by $\\Delta_\\Xi=\\prod_{s\\in \nS}\\Delta(\\Xi_s)$, and consider the problem\n\\begin{equation}\n\\label{e:probXi}\\tag{$P_{\\Xi}$}\n\\min_{q\\in\\Delta_{\\Xi}}F(q,\\eta):={g_{\\Xi}(q)}\n+\\eta\\sum_{s\\in S}\n\\sum_{(a,b)\\in \\Xi_s}d_{ab}^sq_{ab}^s,\n\\end{equation}\nwhere {$g_{\\Xi}$ is the restriction of $g$ \ndefined in \n\\eqref{e:defg} to $\\RP^\\Xi$.} Then \nthe following hold:\n\\begin{enumerate}\n\\item\\label{t:convgeni} {The set of solutions to \n\\eqref{e:probXi} is \nnonempty.\n\\item\\label{t:convgenii}  Given $q^0\\in {\\rm \nri}(\\Delta_\\Xi)$, \nconsider the sequence generated \nby the recurrence \n\\begin{equation}\n\\label{e:recursXi}\n(\\forall n\\in\\NN)(\\forall s\\in S)(\\forall (a,b)\\in \\Xi_s)\\quad \nq_{ab}^{n+1,s}=\n\\frac{(T_\\Xi(q^n))_be^{-\\eta \nd_{ab}^s/p_s}}{\\sum\\limits_{(a',b')\\in\\Xi_s}\n(T_\\Xi(q^n))_{b'}e^{-\\eta \nd_{a'b'}^s/p_s}}, \n\\end{equation}\nwhere $T_{\\Xi}\\colon\\Delta_\\Xi\\to \n\\Delta(B)$ is defined by \n\\begin{equation}\n\\label{e:TXi}\nT{_\\Xi}\\colon \nq\\mapsto \\left(\\sum_{a\\in A}\\sum_{s\\in S}p_sq_{ab}^s\\right)_{b\\in \nB}.\n\\end{equation}\nThen, the following hold.\n\\begin{enumerate}\n\\item \\label{t:convgeniia}For every solution $\\overline{q}$ to \n\\eqref{e:probXi}, \n\\begin{equation}\n\\label{e:o1k}\n(\\forall n\\in\\NN)\\quad 0\\le \nn(F(q^{n},\\eta)\n-F(\\overline{q},\\eta))+\\sum_{k=0}^{n-1}\nD_B(T_{\\Xi}(q^{k+1}),T_{\\Xi}(q^{k}))\\le \n\\sum_{s\\in S}p_sD_{A\\times B}(\\overline{q}^s,q^{0,s}).\n\\end{equation}\n\\item\\label{t:convgeniib} For every solution $\\overline{q}$ to \n\\eqref{e:probXi}, \n$$0\\le F(q^{n},\\eta)\n-F(\\overline{q},\\eta)\\le \\frac{1}{n}\\sum_{s\\in S}p_sD_{A\\times \nB}(\\overline{q}^s,q^{0,s})$$ \nand \n$\\sum_{k=1}^{\\infty}D_B(T_{\\Xi}(q^{k+1}),T_{\\Xi}(q^{k}))<+\\infty$.\n\n\\item \\label{t:convgeniic}\nThe sequence $(q_n)_{n\\in\\NN}$ converges to some solution \n$\\overline{q}$ to \n\\eqref{e:probXi}. \n\\end{enumerate}\n}\n\n\n\n\n\n\n\n\n\\end{enumerate}\n\n\\end{theorem}","sha256_utf8":"5324187fc58351b520319bac3b98e30a6732cff728067bc2786257ebfa63861f"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{proposition}\n\\label{p:KLprop}\nLet $I$ be a finite set, let $\\psi$ be the function defined \nin \\eqref{e:defpsi}, and let $D_I$ the \nfunction defined in \\eqref{e:Kullback}. Then the following \nassertions \nhold:\n\\begin{enumerate}\n\\item \n\\label{p:KLpropi}\n$\\psi$ is proper, lower semicontinuous, and convex.\n\\item \n\\label{p:KLpropii}\n$D_I$ is proper, lower semicontinuous, convex, and satisfies\n\\begin{equation}\n\\label{e:fortmon}\n(\\forall q\\in\\Delta(I))(\\forall u\\in\\Delta(I))\\quad \nD_I(q,u)\\ge\\frac{1}{2}\\|q-u\\|^2_1.\n\\end{equation}\n\\end{enumerate}\n\\end{proposition}\n\n\\begin{proof}\n\\ref{p:KLpropi}: \nNoting that $\\psi$ is the {lower semicontinuous convex \nenvelope of \nthe perspective \nassociated to $\\phi$ defined in \\eqref{e:defphi},\nthe convexity and lower semicontinuity follow from \n\\cite[Proposition~9.42]{BauschkeCombettes17}. }\n\n\n\\ref{p:KLpropii}: The convexity and lower semicontinuity of $D_I$\nfollow from \\ref{p:KLpropi} in view of \\eqref{e:Kullback} and \n\\cite[Proposition~8.6]{BauschkeCombettes17}. Let $q$ and $u$ in \n$\\Delta(I)$.\nNote that, if $\\supp q\\not\\subset \\supp u$, there exists $i_0\\in I$\nsuch that $q_{i_0}>0$ and $u_{i_0}=0$, which implies from \n\\eqref{e:Kullback} and \\eqref{e:defpsi} that $D_I(q,u)=+\\infty$ and, \nthus, the inequality holds trivially. Hence, let us assume that \n$I_q=\\supp \nq\\subset \\supp u=I_u$.\nSince $u\\in\\reli\\Delta(I_u)$, there exists $\\epsilon>0$\nsuch that, for every $t\\in [-\\epsilon,1[$, \n$u+t(q-u)\\in\\reli\\Delta(I_u)$.\nLet $\\varphi\\colon\\Delta(I_u)\\to\\R\\colon z\\mapsto \n\\sum_{i\\in I_u}\\phi(z_i)$, where $\\phi$ is defined in \n\\eqref{e:defphi}, \nand set\n$\\theta\\colon[-\\epsilon,1]\\to\\R\\colon t\\mapsto \n\\varphi(u+t(q-u))$. \nNote that \n$\\theta$ is continuous in $[-\\epsilon,1]$, continuously differentiable \nin \n$]-\\epsilon,1[$,\nand $\\theta'\\colon t\\mapsto \\scal{\\nabla\\varphi(u+t(q-u))}{q-u}$.\nThen, for every $t\\in]-\\epsilon,1[$ we have $u+t(q-u)$ and $u$ are \nin \n$\\reli\\Delta(I_u)$ and \\cite[Proposition~5.1(a)]{BeckTeb03} \nimplies\n\\begin{align}\n\\theta'(t)-\\theta'(0)&=\\scal{\\nabla\\varphi(u+t(q-u))\n-\\nabla\\varphi(u)}{q-u}\\nonumber\\\\\n&=\n\\frac{1}{t}\\scal{\\nabla\\varphi(u+t(q-u))-\\nabla\\varphi(u)}{u+t(q-u)-u}\n\\nonumber\\\\\n&\\ge \n\\frac{1}{t}\\|t(q-u)\\|_1^2\\nonumber\\\\\n&=t\\|q-u\\|_1^2.\n\\end{align}\nTherefore,\n\\begin{equation}\n(\\forall s\\in\\left]0,1\\right[)\\quad \n\\theta(s)-\\theta(0)=\\int_0^s\\theta'(t)dt\\ge \ns\\theta'(0)+\\frac{s^2}{2}\\|q-u\\|_1^2,\n\\end{equation}\nand taking $s\\to 1$ we obtain from $(q,u)\\in\\Delta(I_u)^2$ that\n\\begin{align}\n\\frac{1}{2}\\|q-u\\|_1^2&\\le \n\\varphi(q)-\\varphi(u)-\\scal{\\nabla\\varphi(u)}{q-u}\\nonumber\\\\ \n&=\\sum_{i\\in I_u}\\phi(q_i)-\\sum_{i\\in \nI_u}(u_i\\ln(u_i)+(\\ln(u_i)+1)(q_i-u_i))\n\\nonumber\\\\\n&=\\sum_{i\\in I_q}q_i\\ln(q_i)-\\sum_{i\\in I_u}q_i\\ln(u_i)\\nonumber\\\\\n&=\\sum_{i\\in I_q}q_i\\ln\\left(\\frac{q_i}{u_i}\\right)\\nonumber\\\\\n&=D_I(q,u),\n\\end{align}\nand the proof is complete.\\qed\n\\end{proof}\n\n\\begin{proposition}\n\\label{p:exist}\nIn the context of Problem~\\ref{prob:main}, \nlet $g$ be the function defined in \\eqref{e:defg} and let \n$C$ be the set defined in \\eqref{e:defC}. Then, the following hold.\n\\begin{enumerate}\n\\item \n\\label{p:existi}\nThere exists $\\widetilde{q}\\in{\\reli((\\Delta(A\\times \nB))^S)}$ such that\n$g(\\widetilde{q})<0$.\n\n\\item\n\\label{p:existii+}\n$g$ is convex and lower semicontinuous.\n\\item \n\\label{p:existii}\n$C$ is a nonempty convex compact set.\n\\item \n\\label{p:existiii}\nProblem~\\ref{prob:main} admits at least a solution.\n\\end{enumerate}\n\\end{proposition}\n\n\\begin{proof}\n\\ref{p:existi}:\nDefine $\\widetilde{q}\\in(\\Delta(A\\times B))^S$ by\n\\begin{equation}\n(\\forall s\\in S)(\\forall a\\in A)(\\forall b\\in B)\\quad \n\\widetilde{q}_{ab}^s=\\dfrac{1}{|A||B|}{>0}.\n\\end{equation}\nIt follows from \\eqref{e:defg} that $g(\\widetilde{q})=-\\ln|A|<0$.\n\n\n\\ref{p:existii+}: Note that $(u_{as})_{a,s}\\mapsto \n\\sum_{a',s'}p_{s'}u_{a's'}$ is linear and, since $\\psi$ is convex and \nlower semicontinuous in view of \nProposition~\\ref{p:KLprop}\\eqref{p:KLpropi}, $g$ is convex and \nlower semicontinuous. \n\n\\ref{p:existii}: It follows from \\eqref{e:defC}, \\eqref{e:defg}, and \n\\eqref{e:defpsi} that \n\\begin{equation}\n\\label{e:C2}\nC=g^{-1}(\\RM)\\cap(\\Delta(A\\times B))^S.\n\\end{equation}\nSince \\ref{p:existii+} asserts that $g$ is\nconvex and lower semicontinuous, $g^{-1}(\\RM)$ is convex and \nclosed.\nMoreover, since $(\\Delta(A\\times B))^S$ is convex and compact,\nwe deduce from \\eqref{e:C2} that\n$C$ is convex and compact. The result follows from \\ref{p:existi}.\n\n\\ref{p:existiii}: It is a direct consequence of \n\\cite[Theorem~2.12]{Beck2017}.\\qed\n\\end{proof}\n\n\\begin{proof}\n\\ref{t:convgeni}:\nNote that, since $\\Delta_\\Xi$ is compact, the existence \nof solutions to \\eqref{e:probXi} is guaranteed by \n{\\cite[Theorem~2.12]{Beck2017}} and \nProposition~\\ref{p:exist}\\eqref{p:existii+}.\n\n\\ref{t:convgeniia}: Note that \\eqref{e:probXi} can be written \nequivalently as \n\\begin{equation}\n\\min_{q\\in [0,+\\infty[^{\\Xi}}f(q)+g_{\\Xi}(q),\n\\end{equation}\nwhere \n{\\begin{equation}\n\\begin{cases}\nf\\colon \\R^{\\Xi}\\to\\left]-\\infty,+\\infty\\right]\\colon q\\mapsto \n\\iota_{V_\\Xi}(q)+\\eta\n\\sum_{s\\in S}\\sum_{(a,b)\\in \\Xi_s}d_{ab}^sq_{ab}^s\\\\\nV_{\\Xi}=\\menge{q\\in \\R^{\\Xi}}{(\\forall s\\in \nS)\\:\\:\\sum_{(a,b)\\in\\Xi_s}q_{ab}^s=1}.\n\\end{cases}\n\\end{equation}\nNote that $g_{\\Xi}=g\\circ E$, where}\n$E\\colon \\R^{\\Xi}\\to\\R^{(A\\times B)^S}$ stands for the \ncanonical extension defined by\n\\begin{equation}\n\\label{e:defE}\n(\\forall q\\in \\R^{\\Xi})(\\forall (a,b,s)\\in A\\times B\\times S)\\quad \n(Eq)_{ab}^s=\n\\begin{cases}\nq_{ab}^s,&\\text{if}\\:\\: (a,b)\\in \\Xi_s;\\\\\n0,&\\text{otherwise}.\n\\end{cases}\n\\end{equation}\nNote that $f$ and $g_{\\Xi}$ are proper, lower semicontinuous, and \nconvex.\nMoreover, by defining \n\\begin{equation}\n\\label{e:defh}\nh\\colon \\R^{\\Xi}\\to\\left]-\\infty,+\\infty\\right]\\colon q\\mapsto \n\\sum_{s\\in \nS}p_s\\varphi_{\\Xi_s}(q^s),\n\\end{equation}\nwhere {$\\varphi_{\\Xi_s}$\nis defined in \\eqref{e:defvarphi}}, it follows from \n\\cite[Example~1]{BauMOR} that $h$ is a Legendre function \nsatisfying $\\overline{\\rm dom }\\,h=[0,+\\infty[^{\\Xi}={\\rm dom}\\, h= \n{\\rm dom}\\, g_{\\Xi}$, \nand by defining $\\widetilde{q}\\in\\R^{\\Xi}$ by\n$$(\\forall s\\in S)(\\forall (a,b)\\in\\Xi_s)\\quad \n\\widetilde{q}_{ab}^s=\\frac{1}{|\\Xi_s|},$$ we have that\n$\\widetilde{q}\\in\\dom f\\cap \\inte\\dom h$.\nMoreover, $h-g_{\\Xi}$ is convex on \n$\\inte\\dom h=\\left]0,+\\infty\\right[^{\\Xi}$. \nIndeed, \nfor every $q\\in \\left]0,+\\infty\\right[^{\\Xi}$, \nit follows from \\eqref{e:defg}, \\eqref{e:defpsi}, and \\eqref{e:defE} \nthat\n\\begin{align}\n\\label{e:hminusgconv}\n(h-g_{\\Xi})(q)&=\n\\sum_{s\\in S}p_s\\left(\\sum_{(a,b)\\in \\Xi^s}q_{ab}^s\\ln \nq_{ab}^s-\\sum_{a\\in A}\\sum_{b\\in \nB}\\psi\\left((Eq)_{ab}^s,\\sum_{a'\\in A}\\sum_{s'\\in \nS}p_{s'}(Eq)_{a'b}^{s'}\\right)\n\\right)\\nonumber\\\\\n&=\n\\sum_{s\\in S}p_s\\sum_{(a,b)\\in \\Xi^s}q_{ab}^s\\left(\\ln \nq_{ab}^s-\\ln\\left(\\frac{q_{ab}^s}{{\\sum\\limits_{a'\\in \nA}\\sum\\limits_{s'\\in \nS}}p_{s'}(Eq)_{a'b}^{s'}}\\right)\\right)\\nonumber\\\\\n&=\n\\sum_{s\\in S}p_s\\sum_{(a,b)\\in \\Xi^s}q_{ab}^s\\ln\\left({\\sum_{a'\\in \nA}\\sum_{s'\\in \nS}p_{s'}(Eq)_{a'b}^{s'}}\\right)\\nonumber\\\\\n&=\n\\sum_{b\\in \nB}\\sum_{a\\in A}\\sum_{s\\in S}p_s(Eq)_{ab}^s\\ln\\left({\\sum_{a'\\in \nA}\\sum_{s'\\in \nS}p_{s'}(Eq)_{a'b}^{s'}}\\right)\\nonumber\\\\\n&=\\varphi_B(T_{\\Xi}(Eq)),\n\\end{align}\nwhich is convex because it is the composition of the convex \nfunction $\\varphi_B$, defined in \\eqref{e:defvarphi},\nwith the linear operators\n$T{_{\\Xi}}$ and $E$\ndefined in \\eqref{e:TXi} and \\eqref{e:defE}, respectively.\nOn the other hand, note that the fixed point recursion in \n\\eqref{e:recursXi} is equivalent to \n\\begin{equation}\n\\label{e:mirrorfbXi}\n(\\forall n\\in\\NN)\\quad q^{n+1}=\\arg\\min_{q\\in \n\\R^\\Xi}f(q)+{\\scal{\\nabla\n g_{\\Xi}(q^n)}{q}}+D_h(q,q^n),\n\\end{equation}\nwhere the existence and uniqueness of the \nsolution to \\eqref{e:mirrorfbXi} are guaranteed by \n\\cite[Lemma~2]{BauMOR},  $(q^n)_{n\\in\\NN}\\subset \n\\reli(\\Delta_\\Xi)=\\prod_{s\\in \nS}\\reli(\\Delta(\\Xi_s))$, and,\nfor every $p\\in\\left]0,+\\infty\\right[^\\Xi$ and $q\\in[0,+\\infty[^\\Xi$,\n\\begin{equation}\n\\label{e:breghXi}\nD_h\\colon (q,p)\\mapsto h(q)-h(p)-\\scal{\\nabla \nh(p)}{q-p}=\\sum_{s\\in S}p_s {D_{\\Xi_s}(q^s,p^s)}\n\\end{equation} \nis the Bregman distance associated to $h$. \nIndeed, given $n\\in\\NN$, since $g_{\\Xi}$ is \ndifferentiable in \n$]0,+\\infty[^{\\Xi}$ and, for every $q\\in \\reli(\\Delta_\\Xi)$, \n$\\nabla \ng_{\\Xi}(q)= \n(p_s\\ln(q_{ab}^s/(T_{\\Xi}(q))_b))_{(a,b)\\in\\Xi_s,\\,s\\in S}$, \n\\eqref{e:mirrorfbXi} is \nequivalent \nto\n{\\begin{align}\n\\quad q^{n+1}&=\\arg\\min_{q\\in V_{\\Xi}}\n\\eta\\sum_{s\\in S}\\sum_{(a,b)\\in\\Xi_s}d_{ab}^sq_{ab}^s+\n\\sum_{s\\in S}\\sum_{(a,b)\\in\\Xi_s}p_sq_{ab}^s\n\\ln\\left(\\frac{q_{ab}^{n,s}}{(T_{\\Xi}(q^n))_b}\\right)\n+\\sum_{s\\in S}\np_sD_{\\Xi_s}(q^s,q^{n,s})\\nonumber\\\\\n&=\\arg\\min_{q\\in \nV_{\\Xi}}\\eta\\sum_{s\\in S}\\sum_{(a,b)\\in\\Xi_s}d_{ab}^sq_{ab}^s+\n\\sum_{s\\in S}p_s\\sum_{(a,b)\\in\\Xi_s}\\phi(q_{ab}^s)-\\sum_{s\\in \nS}p_s\\sum_{(a,b)\\in\\Xi_s}q_{ab}^{s}\\ln\\left((T_{\\Xi}(q^n))_b\\right).\n\\end{align}}\nHence, it follows from  \n{\\cite[Theorem~28.2]{rockafellar1997convex}} and \nProposition~\\ref{p:KLprop}\\eqref{p:KLpropi} that there exists \n$\\xi\\in\\R^S$ such that\n%\\begin{equation}\n%{0 = \\eta d_{ab}^s + p_s (\\ln q_{ab}^s +1) - p_s \\ln \n%(T_{\\Xi}(q^n))_b - \\xi_s }.\n%\\end{equation}\n\\begin{equation}\n(\\forall s\\in S)(\\forall (a,b)\\in\\Xi_s)\\quad 0=\\eta \n\\frac{d_{ab}^s}{p_s}-\\frac{\\xi_s}{p_s}+\n\\ln\\left(\\frac{q_{ab}^{n+1,s}}{T{_{\\Xi}}(q^n)_b}\\right) {+1 },\n\\end{equation}\n{which leads to}\n\\begin{equation}\n\\label{e:almos}\n(\\forall s\\in S)(\\forall (a,b)\\in\\Xi_s)\\quad\nq_{ab}^{n+1,s}=T{_{\\Xi}}(q^n)_be^{-\\eta d_{ab}^s/p_s}\ne^{\\xi_s/p_s{-1}}\n\\end{equation}\nand,  { using that $q^{n+1}\\in V_{\\Xi}$, we conclude \nthat \\eqref{e:almos} reduces to \\eqref{e:recursXi}.\nIn addition, since $h$ is differentiable in $\\RPP^{\\Xi}$,\nit follows from \\eqref{e:hminusgconv} that $\\nabla \nh=\\nabla g_{\\Xi}+T_{\\Xi}^*\\circ\\nabla\\varphi_B\\circ T_{\\Xi}$. Hence,\ngiven $n\\in\\NN$,\nwe deduce from \\eqref{e:hminusgconv} and \\eqref{e:Bdist} that\n\\begin{align}\n\\label{e:auxd}\ng_{\\Xi}(q^{n+1})-g_{\\Xi}(q^{n})&=\nh(q^{n+1})-h(q^n)-\\varphi_B(T_{\\Xi}(q^{n+1}))+\\varphi_B(T_{\\Xi}\n(q^n))\\nonumber\\\\\n&=D_h(q^{n+1},q^n)+\\scal{\\nabla h(q^n)}{q^{n+1}-q^n}\n-\\varphi_B(T_{\\Xi}(q^{n+1}))+\\varphi_B(T_{\\Xi}\n(q^n))\\nonumber\\\\\n&=D_h(q^{n+1},q^n)+\\scal{\\nabla g_{\\Xi}(q^n)}{q^{n+1}-q^n}\n-\\varphi_B(T_{\\Xi}(q^{n+1}))+\\varphi_B(T_{\\Xi}\n(q^n))\\nonumber\\\\\n&\\hspace{6cm}+\\scal{T_{\\Xi}^*\\nabla \n\\varphi_B(T_{\\Xi}(q^n))}{q^{n+1}-q^n}\\nonumber\\\\\n&=D_h(q^{n+1},q^n)+\\scal{\\nabla g_{\\Xi}(q^n)}{q^{n+1}-q^n}\n-\\varphi_B(T_{\\Xi}(q^{n+1}))+\\varphi_B(T_{\\Xi}\n(q^n))\\nonumber\\\\\n&\\hspace{5cm}+\\scal{\\nabla \n\\varphi_B(T_{\\Xi}(q^n))}{T_{\\Xi}(q^{n+1})-T_{\\Xi}(q^n)}\\nonumber\\\\\n&=D_h(q^{n+1},q^n)+\\scal{\\nabla g_{\\Xi}(q^n)}{q^{n+1}-q^n}\n-D_B(T_{\\Xi}(q^{n+1}),T_{\\Xi}(q^n)).\n\\end{align}\nMoreover, the convexity of $g_{\\Xi}$\nimplies \n$$(\\forall q\\in\\RP^{\\Xi})\\quad g_{\\Xi}(q^n)-g_{\\Xi}(q)\\le \\scal{\\nabla \ng_{\\Xi}(q^n)}{q^n-q},$$\nwhich, combined with \\eqref{e:auxd}, yields\n\\begin{align}\n\\label{e:propg}\n(\\forall q\\in\\RP^{\\Xi})\\quad \ng_{\\Xi}(q^{n+1})-g_{\\Xi}(q)\n&\\le D_h(q^{n+1},q^n)\n-D_B(T_{\\Xi}(q^{n+1}),T_{\\Xi}(q^n))+\\scal{\\nabla \ng_{\\Xi}(q^n)}{q^{n+1}-q}.\n\\end{align}\nOn the other hand, the first order optimality conditions of \n\\eqref{e:mirrorfbXi} and the three points identity \n\\cite[Lemma~3.1]{chen1993convergence}\nimply\n\\begin{align}\n\\label{e:auxf}\n(\\forall q\\in\\RP^{\\Xi})\\quad f(q^{n+1})-f(q)\n&\\le\\scal{\\nabla \nh(q^n)-\\nabla h(q^{n+1})}{q^{n+1}-q}+\\scal{\\nabla \ng_{\\Xi}(q^n)}{q-q^{n+1}}\\nonumber\\\\\n&=D_h(q,q^n)-D_h(q,q^{n+1})-D_h(q^{n+1},q^n)+\\scal{\\nabla \ng_{\\Xi}(q^n)}{q-q^{n+1}}.\n\\end{align} \nHence, by summing up \\eqref{e:auxf} and \\eqref{e:propg} we \ndeduce\n\\begin{equation}\n\\label{e:trueineq}\n(\\forall q\\in\\RP^{\\Xi})\\quad F(q^{n+1},\\eta)-F(q,\\eta)\\le \nD_h(q,q^n)-D_h(q,q^{n+1})-D_B(T_{\\Xi}(q^{n+1}),T_{\\Xi}(q^n)).\n\\end{equation}\nNote that, by taking $q=q^n$ in \\eqref{e:trueineq}, we deduce that \n$(F(q^n,\\eta))_{n\\in\\NN}$ is nonincreasing and, by taking \n$q=\\overline{q}$, where $\\overline{q}$ is any solution to \n\\eqref{e:probXi}, we obtain\n \\begin{equation}\n\\label{e:inequs}\n0\\le F(q^{n+1},\\eta)-F(\\overline{q},\\eta)\\le \nD_h(\\overline{q},q^n)-D_h(\\overline{q},q^{n+1})\n-D_B(T_{\\Xi}(q^{n+1}),T_{\\Xi}(q^n)).\n\\end{equation}\nThis} \nimplies that the sequence \n$(D_h(\\overline{q},q^n))_{n\\in\\NN}$\nis nonincreasing, {nonnegative} in view of  \nProposition~\\ref{p:KLprop}\\eqref{p:KLpropii} and, \ntherefore, { for every solution $\\overline{q}$ to \n\\eqref{e:probXi}, we have}\n\\begin{equation}\n\\label{e:convergentlamqXi}\n\\big(D_h(\\overline{q},q^n)\\big)_{n\\in\\NN}\\quad\\text{ \nis convergent}.\n\\end{equation}\n{\nTherefore, by summing from $0$ to $n-1$ in \\eqref{e:inequs}, we \ndeduce\n\\begin{align}\n\\label{e:ineq}\n0\\le n(F(q^n,\\eta)-F(\\overline{q},\\eta))\n&\\le \n\\sum_{k=0}^{n-1}\\big(F(q^{k+1},\\eta)-F(\\overline{q},\\eta)\\big)\n\\nonumber\\\\\n&\\le D_h(\\overline{q},q^0)-D_h(\\overline{q},q^{n})\n-\\sum_{k=0}^{n-1}D_B(T_{\\Xi}(q^{k+1}),T_{\\Xi}(q^k)),\n\\end{align}\n which yields the result.\n\n\\ref{t:convgeniib}: It is a direct consequence of \\ref{t:convgeniia}.\n\n\\ref{t:convgeniic}: Note that \\ref{t:convgeniib} implies that\n$F(q^{n},\\eta)\\to \nF(\\overline{q},\\eta)$.}\nFinally, since $\\Delta_{\\Xi}$ is compact and recalling that \n$(q^n)_{n\\in\\NN}\\subset \\reli(\\Delta_{\\Xi})$,\nlet $q^*$ be an accumulation point of $(q^n)_{n\\in\\NN}$, say \n$q^{k_n}\\to \nq^*$. The continuity of $F(\\cdot,\\eta)$ relatively to \n$\\Delta_{\\Xi}$ implies that\n$F(q^{k_n},\\eta)\\to \nF(q^*,\\eta)=F(\\overline{q},\\eta)\\in\\R$,\n which { implies that $q^*$ is also a solution to \n \\eqref{e:probXi}.  Furthermore, since \\eqref{e:breghXi} \ncan be written as\n\\begin{equation}\n(\\forall n\\in\\NN)\\quad D_h(q^*,q^n)=\n\\sum_{s\\in S}p_s\\sum_{(a,b)\\in\\supp \nq^{*,s}}q^{*,s}_{ab}\\ln\\left(\\frac{q^{*,s}_{ab}}{q^{n,s}_{ab}}\\right),\n\\end{equation}\n$q^{k_n}\\to q^*$ implies $D_{h}({q}^*,q^{k_n})\\to 0$, and } \nwe deduce from \n\\eqref{e:convergentlamqXi} that $D_{h}({q}^*,q^{n})\\to 0$.\nWe conclude that $q^n\\to q^*$ from \nProposition~\\ref{p:KLprop}\\eqref{p:KLpropii}.\\qed\n\\end{proof}","units":[{"start":34306,"end":34875,"start_line":863,"end_line":883,"raw_tex":"\\begin{proposition}\n\\label{p:KLprop}\nLet $I$ be a finite set, let $\\psi$ be the function defined \nin \\eqref{e:defpsi}, and let $D_I$ the \nfunction defined in \\eqref{e:Kullback}. Then the following \nassertions \nhold:\n\\begin{enumerate}\n\\item \n\\label{p:KLpropi}\n$\\psi$ is proper, lower semicontinuous, and convex.\n\\item \n\\label{p:KLpropii}\n$D_I$ is proper, lower semicontinuous, convex, and satisfies\n\\begin{equation}\n\\label{e:fortmon}\n(\\forall q\\in\\Delta(I))(\\forall u\\in\\Delta(I))\\quad \nD_I(q,u)\\ge\\frac{1}{2}\\|q-u\\|^2_1.\n\\end{equation}\n\\end{enumerate}\n\\end{proposition}","sha256_utf8":"8dafb086d02b42efd82ea8e511d2410bbae50f8db45b282a1fa6813b3cd5a297"},{"start":34876,"end":37172,"start_line":884,"end_line":950,"raw_tex":"\\begin{proof}\n\\ref{p:KLpropi}: \nNoting that $\\psi$ is the {lower semicontinuous convex \nenvelope of \nthe perspective \nassociated to $\\phi$ defined in \\eqref{e:defphi},\nthe convexity and lower semicontinuity follow from \n\\cite[Proposition~9.42]{BauschkeCombettes17}. }\n\n\n\\ref{p:KLpropii}: The convexity and lower semicontinuity of $D_I$\nfollow from \\ref{p:KLpropi} in view of \\eqref{e:Kullback} and \n\\cite[Proposition~8.6]{BauschkeCombettes17}. Let $q$ and $u$ in \n$\\Delta(I)$.\nNote that, if $\\supp q\\not\\subset \\supp u$, there exists $i_0\\in I$\nsuch that $q_{i_0}>0$ and $u_{i_0}=0$, which implies from \n\\eqref{e:Kullback} and \\eqref{e:defpsi} that $D_I(q,u)=+\\infty$ and, \nthus, the inequality holds trivially. Hence, let us assume that \n$I_q=\\supp \nq\\subset \\supp u=I_u$.\nSince $u\\in\\reli\\Delta(I_u)$, there exists $\\epsilon>0$\nsuch that, for every $t\\in [-\\epsilon,1[$, \n$u+t(q-u)\\in\\reli\\Delta(I_u)$.\nLet $\\varphi\\colon\\Delta(I_u)\\to\\R\\colon z\\mapsto \n\\sum_{i\\in I_u}\\phi(z_i)$, where $\\phi$ is defined in \n\\eqref{e:defphi}, \nand set\n$\\theta\\colon[-\\epsilon,1]\\to\\R\\colon t\\mapsto \n\\varphi(u+t(q-u))$. \nNote that \n$\\theta$ is continuous in $[-\\epsilon,1]$, continuously differentiable \nin \n$]-\\epsilon,1[$,\nand $\\theta'\\colon t\\mapsto \\scal{\\nabla\\varphi(u+t(q-u))}{q-u}$.\nThen, for every $t\\in]-\\epsilon,1[$ we have $u+t(q-u)$ and $u$ are \nin \n$\\reli\\Delta(I_u)$ and \\cite[Proposition~5.1(a)]{BeckTeb03} \nimplies\n\\begin{align}\n\\theta'(t)-\\theta'(0)&=\\scal{\\nabla\\varphi(u+t(q-u))\n-\\nabla\\varphi(u)}{q-u}\\nonumber\\\\\n&=\n\\frac{1}{t}\\scal{\\nabla\\varphi(u+t(q-u))-\\nabla\\varphi(u)}{u+t(q-u)-u}\n\\nonumber\\\\\n&\\ge \n\\frac{1}{t}\\|t(q-u)\\|_1^2\\nonumber\\\\\n&=t\\|q-u\\|_1^2.\n\\end{align}\nTherefore,\n\\begin{equation}\n(\\forall s\\in\\left]0,1\\right[)\\quad \n\\theta(s)-\\theta(0)=\\int_0^s\\theta'(t)dt\\ge \ns\\theta'(0)+\\frac{s^2}{2}\\|q-u\\|_1^2,\n\\end{equation}\nand taking $s\\to 1$ we obtain from $(q,u)\\in\\Delta(I_u)^2$ that\n\\begin{align}\n\\frac{1}{2}\\|q-u\\|_1^2&\\le \n\\varphi(q)-\\varphi(u)-\\scal{\\nabla\\varphi(u)}{q-u}\\nonumber\\\\ \n&=\\sum_{i\\in I_u}\\phi(q_i)-\\sum_{i\\in \nI_u}(u_i\\ln(u_i)+(\\ln(u_i)+1)(q_i-u_i))\n\\nonumber\\\\\n&=\\sum_{i\\in I_q}q_i\\ln(q_i)-\\sum_{i\\in I_u}q_i\\ln(u_i)\\nonumber\\\\\n&=\\sum_{i\\in I_q}q_i\\ln\\left(\\frac{q_i}{u_i}\\right)\\nonumber\\\\\n&=D_I(q,u),\n\\end{align}\nand the proof is complete.\\qed\n\\end{proof}","sha256_utf8":"4e6a9b04042d34c7e20ff37da270f46f6dd4a60bee2fd148d5da48a12755f12f"},{"start":38063,"end":38647,"start_line":983,"end_line":1005,"raw_tex":"\\begin{proposition}\n\\label{p:exist}\nIn the context of Problem~\\ref{prob:main}, \nlet $g$ be the function defined in \\eqref{e:defg} and let \n$C$ be the set defined in \\eqref{e:defC}. Then, the following hold.\n\\begin{enumerate}\n\\item \n\\label{p:existi}\nThere exists $\\widetilde{q}\\in{\\reli((\\Delta(A\\times \nB))^S)}$ such that\n$g(\\widetilde{q})<0$.\n\n\\item\n\\label{p:existii+}\n$g$ is convex and lower semicontinuous.\n\\item \n\\label{p:existii}\n$C$ is a nonempty convex compact set.\n\\item \n\\label{p:existiii}\nProblem~\\ref{prob:main} admits at least a solution.\n\\end{enumerate}\n\\end{proposition}","sha256_utf8":"7162433df7c3f996f1ee8b9b33925631828d87e8aa9223309c3f3c219c630508"},{"start":38648,"end":39721,"start_line":1006,"end_line":1037,"raw_tex":"\\begin{proof}\n\\ref{p:existi}:\nDefine $\\widetilde{q}\\in(\\Delta(A\\times B))^S$ by\n\\begin{equation}\n(\\forall s\\in S)(\\forall a\\in A)(\\forall b\\in B)\\quad \n\\widetilde{q}_{ab}^s=\\dfrac{1}{|A||B|}{>0}.\n\\end{equation}\nIt follows from \\eqref{e:defg} that $g(\\widetilde{q})=-\\ln|A|<0$.\n\n\n\\ref{p:existii+}: Note that $(u_{as})_{a,s}\\mapsto \n\\sum_{a',s'}p_{s'}u_{a's'}$ is linear and, since $\\psi$ is convex and \nlower semicontinuous in view of \nProposition~\\ref{p:KLprop}\\eqref{p:KLpropi}, $g$ is convex and \nlower semicontinuous. \n\n\\ref{p:existii}: It follows from \\eqref{e:defC}, \\eqref{e:defg}, and \n\\eqref{e:defpsi} that \n\\begin{equation}\n\\label{e:C2}\nC=g^{-1}(\\RM)\\cap(\\Delta(A\\times B))^S.\n\\end{equation}\nSince \\ref{p:existii+} asserts that $g$ is\nconvex and lower semicontinuous, $g^{-1}(\\RM)$ is convex and \nclosed.\nMoreover, since $(\\Delta(A\\times B))^S$ is convex and compact,\nwe deduce from \\eqref{e:C2} that\n$C$ is convex and compact. The result follows from \\ref{p:existi}.\n\n\\ref{p:existiii}: It is a direct consequence of \n\\cite[Theorem~2.12]{Beck2017}.\\qed\n\\end{proof}","sha256_utf8":"933432234b10a1a4312aea399dd56ae30fb8c40b39519ea7ac47be09edc5338c"},{"start":42603,"end":51951,"start_line":1145,"end_line":1416,"raw_tex":"\\begin{proof}\n\\ref{t:convgeni}:\nNote that, since $\\Delta_\\Xi$ is compact, the existence \nof solutions to \\eqref{e:probXi} is guaranteed by \n{\\cite[Theorem~2.12]{Beck2017}} and \nProposition~\\ref{p:exist}\\eqref{p:existii+}.\n\n\\ref{t:convgeniia}: Note that \\eqref{e:probXi} can be written \nequivalently as \n\\begin{equation}\n\\min_{q\\in [0,+\\infty[^{\\Xi}}f(q)+g_{\\Xi}(q),\n\\end{equation}\nwhere \n{\\begin{equation}\n\\begin{cases}\nf\\colon \\R^{\\Xi}\\to\\left]-\\infty,+\\infty\\right]\\colon q\\mapsto \n\\iota_{V_\\Xi}(q)+\\eta\n\\sum_{s\\in S}\\sum_{(a,b)\\in \\Xi_s}d_{ab}^sq_{ab}^s\\\\\nV_{\\Xi}=\\menge{q\\in \\R^{\\Xi}}{(\\forall s\\in \nS)\\:\\:\\sum_{(a,b)\\in\\Xi_s}q_{ab}^s=1}.\n\\end{cases}\n\\end{equation}\nNote that $g_{\\Xi}=g\\circ E$, where}\n$E\\colon \\R^{\\Xi}\\to\\R^{(A\\times B)^S}$ stands for the \ncanonical extension defined by\n\\begin{equation}\n\\label{e:defE}\n(\\forall q\\in \\R^{\\Xi})(\\forall (a,b,s)\\in A\\times B\\times S)\\quad \n(Eq)_{ab}^s=\n\\begin{cases}\nq_{ab}^s,&\\text{if}\\:\\: (a,b)\\in \\Xi_s;\\\\\n0,&\\text{otherwise}.\n\\end{cases}\n\\end{equation}\nNote that $f$ and $g_{\\Xi}$ are proper, lower semicontinuous, and \nconvex.\nMoreover, by defining \n\\begin{equation}\n\\label{e:defh}\nh\\colon \\R^{\\Xi}\\to\\left]-\\infty,+\\infty\\right]\\colon q\\mapsto \n\\sum_{s\\in \nS}p_s\\varphi_{\\Xi_s}(q^s),\n\\end{equation}\nwhere {$\\varphi_{\\Xi_s}$\nis defined in \\eqref{e:defvarphi}}, it follows from \n\\cite[Example~1]{BauMOR} that $h$ is a Legendre function \nsatisfying $\\overline{\\rm dom }\\,h=[0,+\\infty[^{\\Xi}={\\rm dom}\\, h= \n{\\rm dom}\\, g_{\\Xi}$, \nand by defining $\\widetilde{q}\\in\\R^{\\Xi}$ by\n$$(\\forall s\\in S)(\\forall (a,b)\\in\\Xi_s)\\quad \n\\widetilde{q}_{ab}^s=\\frac{1}{|\\Xi_s|},$$ we have that\n$\\widetilde{q}\\in\\dom f\\cap \\inte\\dom h$.\nMoreover, $h-g_{\\Xi}$ is convex on \n$\\inte\\dom h=\\left]0,+\\infty\\right[^{\\Xi}$. \nIndeed, \nfor every $q\\in \\left]0,+\\infty\\right[^{\\Xi}$, \nit follows from \\eqref{e:defg}, \\eqref{e:defpsi}, and \\eqref{e:defE} \nthat\n\\begin{align}\n\\label{e:hminusgconv}\n(h-g_{\\Xi})(q)&=\n\\sum_{s\\in S}p_s\\left(\\sum_{(a,b)\\in \\Xi^s}q_{ab}^s\\ln \nq_{ab}^s-\\sum_{a\\in A}\\sum_{b\\in \nB}\\psi\\left((Eq)_{ab}^s,\\sum_{a'\\in A}\\sum_{s'\\in \nS}p_{s'}(Eq)_{a'b}^{s'}\\right)\n\\right)\\nonumber\\\\\n&=\n\\sum_{s\\in S}p_s\\sum_{(a,b)\\in \\Xi^s}q_{ab}^s\\left(\\ln \nq_{ab}^s-\\ln\\left(\\frac{q_{ab}^s}{{\\sum\\limits_{a'\\in \nA}\\sum\\limits_{s'\\in \nS}}p_{s'}(Eq)_{a'b}^{s'}}\\right)\\right)\\nonumber\\\\\n&=\n\\sum_{s\\in S}p_s\\sum_{(a,b)\\in \\Xi^s}q_{ab}^s\\ln\\left({\\sum_{a'\\in \nA}\\sum_{s'\\in \nS}p_{s'}(Eq)_{a'b}^{s'}}\\right)\\nonumber\\\\\n&=\n\\sum_{b\\in \nB}\\sum_{a\\in A}\\sum_{s\\in S}p_s(Eq)_{ab}^s\\ln\\left({\\sum_{a'\\in \nA}\\sum_{s'\\in \nS}p_{s'}(Eq)_{a'b}^{s'}}\\right)\\nonumber\\\\\n&=\\varphi_B(T_{\\Xi}(Eq)),\n\\end{align}\nwhich is convex because it is the composition of the convex \nfunction $\\varphi_B$, defined in \\eqref{e:defvarphi},\nwith the linear operators\n$T{_{\\Xi}}$ and $E$\ndefined in \\eqref{e:TXi} and \\eqref{e:defE}, respectively.\nOn the other hand, note that the fixed point recursion in \n\\eqref{e:recursXi} is equivalent to \n\\begin{equation}\n\\label{e:mirrorfbXi}\n(\\forall n\\in\\NN)\\quad q^{n+1}=\\arg\\min_{q\\in \n\\R^\\Xi}f(q)+{\\scal{\\nabla\n g_{\\Xi}(q^n)}{q}}+D_h(q,q^n),\n\\end{equation}\nwhere the existence and uniqueness of the \nsolution to \\eqref{e:mirrorfbXi} are guaranteed by \n\\cite[Lemma~2]{BauMOR},  $(q^n)_{n\\in\\NN}\\subset \n\\reli(\\Delta_\\Xi)=\\prod_{s\\in \nS}\\reli(\\Delta(\\Xi_s))$, and,\nfor every $p\\in\\left]0,+\\infty\\right[^\\Xi$ and $q\\in[0,+\\infty[^\\Xi$,\n\\begin{equation}\n\\label{e:breghXi}\nD_h\\colon (q,p)\\mapsto h(q)-h(p)-\\scal{\\nabla \nh(p)}{q-p}=\\sum_{s\\in S}p_s {D_{\\Xi_s}(q^s,p^s)}\n\\end{equation} \nis the Bregman distance associated to $h$. \nIndeed, given $n\\in\\NN$, since $g_{\\Xi}$ is \ndifferentiable in \n$]0,+\\infty[^{\\Xi}$ and, for every $q\\in \\reli(\\Delta_\\Xi)$, \n$\\nabla \ng_{\\Xi}(q)= \n(p_s\\ln(q_{ab}^s/(T_{\\Xi}(q))_b))_{(a,b)\\in\\Xi_s,\\,s\\in S}$, \n\\eqref{e:mirrorfbXi} is \nequivalent \nto\n{\\begin{align}\n\\quad q^{n+1}&=\\arg\\min_{q\\in V_{\\Xi}}\n\\eta\\sum_{s\\in S}\\sum_{(a,b)\\in\\Xi_s}d_{ab}^sq_{ab}^s+\n\\sum_{s\\in S}\\sum_{(a,b)\\in\\Xi_s}p_sq_{ab}^s\n\\ln\\left(\\frac{q_{ab}^{n,s}}{(T_{\\Xi}(q^n))_b}\\right)\n+\\sum_{s\\in S}\np_sD_{\\Xi_s}(q^s,q^{n,s})\\nonumber\\\\\n&=\\arg\\min_{q\\in \nV_{\\Xi}}\\eta\\sum_{s\\in S}\\sum_{(a,b)\\in\\Xi_s}d_{ab}^sq_{ab}^s+\n\\sum_{s\\in S}p_s\\sum_{(a,b)\\in\\Xi_s}\\phi(q_{ab}^s)-\\sum_{s\\in \nS}p_s\\sum_{(a,b)\\in\\Xi_s}q_{ab}^{s}\\ln\\left((T_{\\Xi}(q^n))_b\\right).\n\\end{align}}\nHence, it follows from  \n{\\cite[Theorem~28.2]{rockafellar1997convex}} and \nProposition~\\ref{p:KLprop}\\eqref{p:KLpropi} that there exists \n$\\xi\\in\\R^S$ such that\n%\\begin{equation}\n%{0 = \\eta d_{ab}^s + p_s (\\ln q_{ab}^s +1) - p_s \\ln \n%(T_{\\Xi}(q^n))_b - \\xi_s }.\n%\\end{equation}\n\\begin{equation}\n(\\forall s\\in S)(\\forall (a,b)\\in\\Xi_s)\\quad 0=\\eta \n\\frac{d_{ab}^s}{p_s}-\\frac{\\xi_s}{p_s}+\n\\ln\\left(\\frac{q_{ab}^{n+1,s}}{T{_{\\Xi}}(q^n)_b}\\right) {+1 },\n\\end{equation}\n{which leads to}\n\\begin{equation}\n\\label{e:almos}\n(\\forall s\\in S)(\\forall (a,b)\\in\\Xi_s)\\quad\nq_{ab}^{n+1,s}=T{_{\\Xi}}(q^n)_be^{-\\eta d_{ab}^s/p_s}\ne^{\\xi_s/p_s{-1}}\n\\end{equation}\nand,  { using that $q^{n+1}\\in V_{\\Xi}$, we conclude \nthat \\eqref{e:almos} reduces to \\eqref{e:recursXi}.\nIn addition, since $h$ is differentiable in $\\RPP^{\\Xi}$,\nit follows from \\eqref{e:hminusgconv} that $\\nabla \nh=\\nabla g_{\\Xi}+T_{\\Xi}^*\\circ\\nabla\\varphi_B\\circ T_{\\Xi}$. Hence,\ngiven $n\\in\\NN$,\nwe deduce from \\eqref{e:hminusgconv} and \\eqref{e:Bdist} that\n\\begin{align}\n\\label{e:auxd}\ng_{\\Xi}(q^{n+1})-g_{\\Xi}(q^{n})&=\nh(q^{n+1})-h(q^n)-\\varphi_B(T_{\\Xi}(q^{n+1}))+\\varphi_B(T_{\\Xi}\n(q^n))\\nonumber\\\\\n&=D_h(q^{n+1},q^n)+\\scal{\\nabla h(q^n)}{q^{n+1}-q^n}\n-\\varphi_B(T_{\\Xi}(q^{n+1}))+\\varphi_B(T_{\\Xi}\n(q^n))\\nonumber\\\\\n&=D_h(q^{n+1},q^n)+\\scal{\\nabla g_{\\Xi}(q^n)}{q^{n+1}-q^n}\n-\\varphi_B(T_{\\Xi}(q^{n+1}))+\\varphi_B(T_{\\Xi}\n(q^n))\\nonumber\\\\\n&\\hspace{6cm}+\\scal{T_{\\Xi}^*\\nabla \n\\varphi_B(T_{\\Xi}(q^n))}{q^{n+1}-q^n}\\nonumber\\\\\n&=D_h(q^{n+1},q^n)+\\scal{\\nabla g_{\\Xi}(q^n)}{q^{n+1}-q^n}\n-\\varphi_B(T_{\\Xi}(q^{n+1}))+\\varphi_B(T_{\\Xi}\n(q^n))\\nonumber\\\\\n&\\hspace{5cm}+\\scal{\\nabla \n\\varphi_B(T_{\\Xi}(q^n))}{T_{\\Xi}(q^{n+1})-T_{\\Xi}(q^n)}\\nonumber\\\\\n&=D_h(q^{n+1},q^n)+\\scal{\\nabla g_{\\Xi}(q^n)}{q^{n+1}-q^n}\n-D_B(T_{\\Xi}(q^{n+1}),T_{\\Xi}(q^n)).\n\\end{align}\nMoreover, the convexity of $g_{\\Xi}$\nimplies \n$$(\\forall q\\in\\RP^{\\Xi})\\quad g_{\\Xi}(q^n)-g_{\\Xi}(q)\\le \\scal{\\nabla \ng_{\\Xi}(q^n)}{q^n-q},$$\nwhich, combined with \\eqref{e:auxd}, yields\n\\begin{align}\n\\label{e:propg}\n(\\forall q\\in\\RP^{\\Xi})\\quad \ng_{\\Xi}(q^{n+1})-g_{\\Xi}(q)\n&\\le D_h(q^{n+1},q^n)\n-D_B(T_{\\Xi}(q^{n+1}),T_{\\Xi}(q^n))+\\scal{\\nabla \ng_{\\Xi}(q^n)}{q^{n+1}-q}.\n\\end{align}\nOn the other hand, the first order optimality conditions of \n\\eqref{e:mirrorfbXi} and the three points identity \n\\cite[Lemma~3.1]{chen1993convergence}\nimply\n\\begin{align}\n\\label{e:auxf}\n(\\forall q\\in\\RP^{\\Xi})\\quad f(q^{n+1})-f(q)\n&\\le\\scal{\\nabla \nh(q^n)-\\nabla h(q^{n+1})}{q^{n+1}-q}+\\scal{\\nabla \ng_{\\Xi}(q^n)}{q-q^{n+1}}\\nonumber\\\\\n&=D_h(q,q^n)-D_h(q,q^{n+1})-D_h(q^{n+1},q^n)+\\scal{\\nabla \ng_{\\Xi}(q^n)}{q-q^{n+1}}.\n\\end{align} \nHence, by summing up \\eqref{e:auxf} and \\eqref{e:propg} we \ndeduce\n\\begin{equation}\n\\label{e:trueineq}\n(\\forall q\\in\\RP^{\\Xi})\\quad F(q^{n+1},\\eta)-F(q,\\eta)\\le \nD_h(q,q^n)-D_h(q,q^{n+1})-D_B(T_{\\Xi}(q^{n+1}),T_{\\Xi}(q^n)).\n\\end{equation}\nNote that, by taking $q=q^n$ in \\eqref{e:trueineq}, we deduce that \n$(F(q^n,\\eta))_{n\\in\\NN}$ is nonincreasing and, by taking \n$q=\\overline{q}$, where $\\overline{q}$ is any solution to \n\\eqref{e:probXi}, we obtain\n \\begin{equation}\n\\label{e:inequs}\n0\\le F(q^{n+1},\\eta)-F(\\overline{q},\\eta)\\le \nD_h(\\overline{q},q^n)-D_h(\\overline{q},q^{n+1})\n-D_B(T_{\\Xi}(q^{n+1}),T_{\\Xi}(q^n)).\n\\end{equation}\nThis} \nimplies that the sequence \n$(D_h(\\overline{q},q^n))_{n\\in\\NN}$\nis nonincreasing, {nonnegative} in view of  \nProposition~\\ref{p:KLprop}\\eqref{p:KLpropii} and, \ntherefore, { for every solution $\\overline{q}$ to \n\\eqref{e:probXi}, we have}\n\\begin{equation}\n\\label{e:convergentlamqXi}\n\\big(D_h(\\overline{q},q^n)\\big)_{n\\in\\NN}\\quad\\text{ \nis convergent}.\n\\end{equation}\n{\nTherefore, by summing from $0$ to $n-1$ in \\eqref{e:inequs}, we \ndeduce\n\\begin{align}\n\\label{e:ineq}\n0\\le n(F(q^n,\\eta)-F(\\overline{q},\\eta))\n&\\le \n\\sum_{k=0}^{n-1}\\big(F(q^{k+1},\\eta)-F(\\overline{q},\\eta)\\big)\n\\nonumber\\\\\n&\\le D_h(\\overline{q},q^0)-D_h(\\overline{q},q^{n})\n-\\sum_{k=0}^{n-1}D_B(T_{\\Xi}(q^{k+1}),T_{\\Xi}(q^k)),\n\\end{align}\n which yields the result.\n\n\\ref{t:convgeniib}: It is a direct consequence of \\ref{t:convgeniia}.\n\n\\ref{t:convgeniic}: Note that \\ref{t:convgeniib} implies that\n$F(q^{n},\\eta)\\to \nF(\\overline{q},\\eta)$.}\nFinally, since $\\Delta_{\\Xi}$ is compact and recalling that \n$(q^n)_{n\\in\\NN}\\subset \\reli(\\Delta_{\\Xi})$,\nlet $q^*$ be an accumulation point of $(q^n)_{n\\in\\NN}$, say \n$q^{k_n}\\to \nq^*$. The continuity of $F(\\cdot,\\eta)$ relatively to \n$\\Delta_{\\Xi}$ implies that\n$F(q^{k_n},\\eta)\\to \nF(q^*,\\eta)=F(\\overline{q},\\eta)\\in\\R$,\n which { implies that $q^*$ is also a solution to \n \\eqref{e:probXi}.  Furthermore, since \\eqref{e:breghXi} \ncan be written as\n\\begin{equation}\n(\\forall n\\in\\NN)\\quad D_h(q^*,q^n)=\n\\sum_{s\\in S}p_s\\sum_{(a,b)\\in\\supp \nq^{*,s}}q^{*,s}_{ab}\\ln\\left(\\frac{q^{*,s}_{ab}}{q^{n,s}_{ab}}\\right),\n\\end{equation}\n$q^{k_n}\\to q^*$ implies $D_{h}({q}^*,q^{k_n})\\to 0$, and } \nwe deduce from \n\\eqref{e:convergentlamqXi} that $D_{h}({q}^*,q^{n})\\to 0$.\nWe conclude that $q^n\\to q^*$ from \nProposition~\\ref{p:KLprop}\\eqref{p:KLpropii}.\\qed\n\\end{proof}","sha256_utf8":"2ac4eab0ed7b024cb22d79e26048b74fef9e2ae84fea2e0ac1d7476ce66c2515"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"熵型固定点迭代的收敛与速率","source_units":[{"start":40577,"end":42602,"start_line":1066,"end_line":1144,"raw_tex":"\\begin{theorem}\n\\label{t:convgen}\nFor every $s\\in S$, let $\\Xi_s\\subset A\\times B$ {and, \nfor every $(a,b)\\in\\Xi_s$, let $d_{ab}^s\\in\\R$.\nMoreover, let $\\eta\\ge 0$,} denote by \n$\\Xi=\\prod_{s\\in S}\\Xi_s$ and by $\\Delta_\\Xi=\\prod_{s\\in \nS}\\Delta(\\Xi_s)$, and consider the problem\n\\begin{equation}\n\\label{e:probXi}\\tag{$P_{\\Xi}$}\n\\min_{q\\in\\Delta_{\\Xi}}F(q,\\eta):={g_{\\Xi}(q)}\n+\\eta\\sum_{s\\in S}\n\\sum_{(a,b)\\in \\Xi_s}d_{ab}^sq_{ab}^s,\n\\end{equation}\nwhere {$g_{\\Xi}$ is the restriction of $g$ \ndefined in \n\\eqref{e:defg} to $\\RP^\\Xi$.} Then \nthe following hold:\n\\begin{enumerate}\n\\item\\label{t:convgeni} {The set of solutions to \n\\eqref{e:probXi} is \nnonempty.\n\\item\\label{t:convgenii}  Given $q^0\\in {\\rm \nri}(\\Delta_\\Xi)$, \nconsider the sequence generated \nby the recurrence \n\\begin{equation}\n\\label{e:recursXi}\n(\\forall n\\in\\NN)(\\forall s\\in S)(\\forall (a,b)\\in \\Xi_s)\\quad \nq_{ab}^{n+1,s}=\n\\frac{(T_\\Xi(q^n))_be^{-\\eta \nd_{ab}^s/p_s}}{\\sum\\limits_{(a',b')\\in\\Xi_s}\n(T_\\Xi(q^n))_{b'}e^{-\\eta \nd_{a'b'}^s/p_s}}, \n\\end{equation}\nwhere $T_{\\Xi}\\colon\\Delta_\\Xi\\to \n\\Delta(B)$ is defined by \n\\begin{equation}\n\\label{e:TXi}\nT{_\\Xi}\\colon \nq\\mapsto \\left(\\sum_{a\\in A}\\sum_{s\\in S}p_sq_{ab}^s\\right)_{b\\in \nB}.\n\\end{equation}\nThen, the following hold.\n\\begin{enumerate}\n\\item \\label{t:convgeniia}For every solution $\\overline{q}$ to \n\\eqref{e:probXi}, \n\\begin{equation}\n\\label{e:o1k}\n(\\forall n\\in\\NN)\\quad 0\\le \nn(F(q^{n},\\eta)\n-F(\\overline{q},\\eta))+\\sum_{k=0}^{n-1}\nD_B(T_{\\Xi}(q^{k+1}),T_{\\Xi}(q^{k}))\\le \n\\sum_{s\\in S}p_sD_{A\\times B}(\\overline{q}^s,q^{0,s}).\n\\end{equation}\n\\item\\label{t:convgeniib} For every solution $\\overline{q}$ to \n\\eqref{e:probXi}, \n$$0\\le F(q^{n},\\eta)\n-F(\\overline{q},\\eta)\\le \\frac{1}{n}\\sum_{s\\in S}p_sD_{A\\times \nB}(\\overline{q}^s,q^{0,s})$$ \nand \n$\\sum_{k=1}^{\\infty}D_B(T_{\\Xi}(q^{k+1}),T_{\\Xi}(q^{k}))<+\\infty$.\n\n\\item \\label{t:convgeniic}\nThe sequence $(q_n)_{n\\in\\NN}$ converges to some solution \n$\\overline{q}$ to \n\\eqref{e:probXi}. \n\\end{enumerate}\n}\n\n\n\n\n\n\n\n\n\\end{enumerate}\n\n\\end{theorem}","sha256_utf8":"5324187fc58351b520319bac3b98e30a6732cff728067bc2786257ebfa63861f"}]},{"id":"S002","semantic_type":"problem_setting","kind":"problem_setting","title":"熵约束线性优化问题","source_units":[{"start":17495,"end":18212,"start_line":445,"end_line":467,"raw_tex":"\\begin{problem}\\label{prob:main}\nLet $A$, $B$, and $S$ be nonempty finite sets such that $|A|\\ge \n2$, let $(c_{ab}^s)_{a\\in A,\\, b\\in B}^{s\\in S}\\in \\R^{(A\\times \nB)^S}$ a vector of cost values, and let $p=(p_s)_{s\\in \nS}\\in{\\reli(\\Delta(S))}$ a probability distribution. \nThe problem is to\n\\begin{align}\n\\label{e:PrimalProblem}\n\\minimize{q\\in C}{\\sum_{s\\in \nS}p_s\\sum_{a\\in A}\\sum_{b\\in B}c_{ab}^sq_{ab}^s},\n\\end{align}\nwhere \n\\begin{equation}\n\\label{e:defC}\nC=\\Menge{q\\in(\\Delta(A\\times \nB))^S}{\\sum_{s\\in \nS}p_s\\sum_{a\\in A}\\sum_{b\\in B}q_{ab}^s\\ln \n\\Bigg(\\frac{q_{ab}^s}{\\sum\\limits_{a'\\in A}\\sum\\limits_{s'\\in \nS}p_{s'}q_{a'b}^{s'}}\\Bigg)\\leq \n0}\n\\end{equation}\nwith the convention $0\\cdot\\ln 0=0$.\n\\end{problem}","sha256_utf8":"c43cfa237ae0cb2620db1caeae2e0a2f8bcbc6dfc108b300d95ae2942f9471a0"}]},{"id":"S003","semantic_type":"notation","kind":"notation","title":"概率单纯形与相对内部","source_units":[{"start":17078,"end":17495,"start_line":435,"end_line":444,"raw_tex":" For every nonempty finite \nset $I$, we denote by \n$\\Delta(I)=\\menge{x\\in[0,1]^{{I}}}{\\sum_{i\\in \nI}x_i=1}$ the set of probability distributions over $I$, \n{$\\reli(C)=\\menge{x\\in C}{{\\rm cone}(C-x)={\\rm \nspan}(C-x)}$ stands \nfor the relative interior of a set $C\\subset \\R^I$, where ${\\rm \ncone}(C)$ and ${\\rm span}(C)$ are the smallest cone (conical \nhull) and linear subspace (span) containing $C$, \nrespectively.}\n","sha256_utf8":"7dccfc87072e6889bfae2d561fc074e9166814b8f60c256343c218965a109173"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"支撑集与示性函数","source_units":[{"start":31840,"end":32131,"start_line":782,"end_line":792,"raw_tex":"For every finite set $I$ and $q\\in \\R^I$, we define \n$\\supp(q)=\\menge{i\\in I}{q_i\\ne 0}$, for every nonempty closed \nconvex \nset $X\\subset \\R^{I}$, let\n\\begin{equation}\n\\iota_{X}\\colon q\\mapsto \n\\begin{cases}\n0,\\quad &\\text{ if }q\\in X;\\\\\n\\pinf,&\\text{ otherwise,}\n\\end{cases}\n\\end{equation}","sha256_utf8":"f580e77b1bb172079fad1f1b782c842cbe361b5d7177ca28b0cdc730aca0991e"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"相对熵及边界约定","source_units":[{"start":32144,"end":32285,"start_line":794,"end_line":799,"raw_tex":"\\begin{equation}\n\\label{e:Kullback}\nD_I\\colon \\R^{I}\\times\\R^{I}\\to\\RX\\colon (q,u)\\mapsto \n\\sum\\limits_{i\\in I}\n\\psi(q_i,u_i),\n\\end{equation}","sha256_utf8":"ba05c0f425c09395abaefc4815d86f78b3b4070d3647b70fe949ac8db7cc896b"},{"start":32286,"end":32569,"start_line":800,"end_line":811,"raw_tex":"where\n$\\psi\\colon\\R^2\\to\\RX$ is \n\\begin{equation}\n\\label{e:defpsi}\n\\psi\\colon (\\xi,\\mu)\\mapsto \n\\begin{cases}\n\\xi\\ln(\\xi/\\mu),\\quad &\\text{if}\\:\\: \n\\xi> 0\\:\\:\\text{and}\\:\\:\\mu>0;\\\\\n0,&\\text{if}\\:\\:\\xi=0\\:\\:\\text{and}\\:\\:\\mu\\ge 0;\\\\\n\\pinf,&\\text{otherwise}.\n\\end{cases}\n\\end{equation}","sha256_utf8":"3848d66fd28402dfb389c6e9e0d9b854753951c4374e26e10d6630ab78bfd2a0"},{"start":32570,"end":32781,"start_line":812,"end_line":817,"raw_tex":"Note that $D_I|_{\\Delta(I)\\times \\Delta(I)}$ is the \nKullback-Leibler \ndivergence \n\\cite[Section~2.3]{cover-book-2006}, which turns out to be a \nBregman distance \\cite[Example~9.10]{Beck2017}. More precisely,\n{\n","sha256_utf8":"c9d79c0e58d7bbce7931a6b80cd6b35e9815fbaf3693fb4ee141114651675d89"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"熵函数与布雷格曼距离","source_units":[{"start":32781,"end":32941,"start_line":818,"end_line":822,"raw_tex":"\\begin{equation}\n\\label{e:Bdist}\n(\\forall q\\in \\R^I)(\\forall u\\in\\RPP^I)\\quad \nD_I(q,u)=\\varphi_I(q)-\\varphi_I(u)-\\scal{\\nabla\\varphi_I(u)}{q-u},\n\\end{equation}","sha256_utf8":"aca1c26ca791cf543d4af8a31f5ac5d42369c7e8f499067254aaeca882486c68"},{"start":32942,"end":33065,"start_line":823,"end_line":827,"raw_tex":"where\n\\begin{equation}\n\\label{e:defvarphi}\n\\varphi_I\\colon \\R^I\\to \\RX\\colon q\\mapsto \\sum_{i\\in I}\\phi(q_i)\n\\end{equation}","sha256_utf8":"132819a7f27e320de88dc74e6a6402b0a9a4eb804ebfd2a28b5e0815a31a5086"},{"start":33066,"end":33257,"start_line":828,"end_line":837,"raw_tex":"and \n\\begin{equation}\n\\label{e:defphi}\n\\phi\\colon \\xi\\mapsto\n\\begin{cases}\n\\xi\\ln\\xi,\\quad&\\text{if}\\:\\: \\xi> 0;\\\\\n0,&\\text{if}\\:\\: \\xi=0;\\\\\n\\pinf,&\\text{otherwise}\n\\end{cases}\n\\end{equation}","sha256_utf8":"4cad0dfa0a55b7718b182684eb2043108c9ddef10064f4c85458361ce25d9dce"},{"start":33258,"end":33822,"start_line":838,"end_line":849,"raw_tex":"is a convex lower semicontinuous proper function\n\\cite[Example~9.35]{BauschkeCombettes17}. \nIt follows from \\cite[Example~2.1]{Teboulle18} that $\\varphi_I$ is a \nLegendre function, i.e., is \nproper, lower semicontinuous, strictly convex, and essentially \nsmooth. The last property is satisfied by functions which are\n differentiable in \nthe interior of its domain (assumed nonempty) and the norm of its \ngradient evaluated in sequences converging to the boundary \nof its domain goes to infinity (see, e.g., \n\\cite[Definition~2.1]{Teboulle18} for more details). \n}\n","sha256_utf8":"b8387d3bda101c713aa947cfe9a6a72d9f8790ed747f41df06ddbeb0407db2fa"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"熵约束函数","source_units":[{"start":37622,"end":37973,"start_line":967,"end_line":978,"raw_tex":"\\begin{equation}\n\\label{e:defg}\ng\\colon\\R^{(A\\times B)^S}\\to \\RX\\colon q \\mapsto \n\\begin{cases}\n\\sum\\limits_{s\\in S}p_s\\sum\\limits_{a\\in A}\\sum\\limits_{b\\in \nB}\\psi\\bigg(q_{ab}^s,\\sum\\limits_{a'\\in A}\\sum\\limits_{s'\\in \nS}p_{s'}q_{a'b}^{s'}\\bigg),&\\text{if}\\:\\:\n q\\in \n[0,+\\infty[^{(A\\times B)^S};\\\\\n\\pinf,&\\text{otherwise}.\n\\end{cases}\n\\end{equation}","sha256_utf8":"7c805f38bbc1b38cc431870bbc2c30601a36e65e0e4ce2c6a95e89d751d41cde"}]},{"id":"S008","semantic_type":"proposition","kind":"proposition","title":"相对熵的凸性与距离下界","source_units":[{"start":34306,"end":34875,"start_line":863,"end_line":883,"raw_tex":"\\begin{proposition}\n\\label{p:KLprop}\nLet $I$ be a finite set, let $\\psi$ be the function defined \nin \\eqref{e:defpsi}, and let $D_I$ the \nfunction defined in \\eqref{e:Kullback}. Then the following \nassertions \nhold:\n\\begin{enumerate}\n\\item \n\\label{p:KLpropi}\n$\\psi$ is proper, lower semicontinuous, and convex.\n\\item \n\\label{p:KLpropii}\n$D_I$ is proper, lower semicontinuous, convex, and satisfies\n\\begin{equation}\n\\label{e:fortmon}\n(\\forall q\\in\\Delta(I))(\\forall u\\in\\Delta(I))\\quad \nD_I(q,u)\\ge\\frac{1}{2}\\|q-u\\|^2_1.\n\\end{equation}\n\\end{enumerate}\n\\end{proposition}","sha256_utf8":"8dafb086d02b42efd82ea8e511d2410bbae50f8db45b282a1fa6813b3cd5a297"}]},{"id":"S009","semantic_type":"proof","kind":"proof","title":"相对熵性质的完整证明","source_units":[{"start":34876,"end":37172,"start_line":884,"end_line":950,"raw_tex":"\\begin{proof}\n\\ref{p:KLpropi}: \nNoting that $\\psi$ is the {lower semicontinuous convex \nenvelope of \nthe perspective \nassociated to $\\phi$ defined in \\eqref{e:defphi},\nthe convexity and lower semicontinuity follow from \n\\cite[Proposition~9.42]{BauschkeCombettes17}. }\n\n\n\\ref{p:KLpropii}: The convexity and lower semicontinuity of $D_I$\nfollow from \\ref{p:KLpropi} in view of \\eqref{e:Kullback} and \n\\cite[Proposition~8.6]{BauschkeCombettes17}. Let $q$ and $u$ in \n$\\Delta(I)$.\nNote that, if $\\supp q\\not\\subset \\supp u$, there exists $i_0\\in I$\nsuch that $q_{i_0}>0$ and $u_{i_0}=0$, which implies from \n\\eqref{e:Kullback} and \\eqref{e:defpsi} that $D_I(q,u)=+\\infty$ and, \nthus, the inequality holds trivially. Hence, let us assume that \n$I_q=\\supp \nq\\subset \\supp u=I_u$.\nSince $u\\in\\reli\\Delta(I_u)$, there exists $\\epsilon>0$\nsuch that, for every $t\\in [-\\epsilon,1[$, \n$u+t(q-u)\\in\\reli\\Delta(I_u)$.\nLet $\\varphi\\colon\\Delta(I_u)\\to\\R\\colon z\\mapsto \n\\sum_{i\\in I_u}\\phi(z_i)$, where $\\phi$ is defined in \n\\eqref{e:defphi}, \nand set\n$\\theta\\colon[-\\epsilon,1]\\to\\R\\colon t\\mapsto \n\\varphi(u+t(q-u))$. \nNote that \n$\\theta$ is continuous in $[-\\epsilon,1]$, continuously differentiable \nin \n$]-\\epsilon,1[$,\nand $\\theta'\\colon t\\mapsto \\scal{\\nabla\\varphi(u+t(q-u))}{q-u}$.\nThen, for every $t\\in]-\\epsilon,1[$ we have $u+t(q-u)$ and $u$ are \nin \n$\\reli\\Delta(I_u)$ and \\cite[Proposition~5.1(a)]{BeckTeb03} \nimplies\n\\begin{align}\n\\theta'(t)-\\theta'(0)&=\\scal{\\nabla\\varphi(u+t(q-u))\n-\\nabla\\varphi(u)}{q-u}\\nonumber\\\\\n&=\n\\frac{1}{t}\\scal{\\nabla\\varphi(u+t(q-u))-\\nabla\\varphi(u)}{u+t(q-u)-u}\n\\nonumber\\\\\n&\\ge \n\\frac{1}{t}\\|t(q-u)\\|_1^2\\nonumber\\\\\n&=t\\|q-u\\|_1^2.\n\\end{align}\nTherefore,\n\\begin{equation}\n(\\forall s\\in\\left]0,1\\right[)\\quad \n\\theta(s)-\\theta(0)=\\int_0^s\\theta'(t)dt\\ge \ns\\theta'(0)+\\frac{s^2}{2}\\|q-u\\|_1^2,\n\\end{equation}\nand taking $s\\to 1$ we obtain from $(q,u)\\in\\Delta(I_u)^2$ that\n\\begin{align}\n\\frac{1}{2}\\|q-u\\|_1^2&\\le \n\\varphi(q)-\\varphi(u)-\\scal{\\nabla\\varphi(u)}{q-u}\\nonumber\\\\ \n&=\\sum_{i\\in I_u}\\phi(q_i)-\\sum_{i\\in \nI_u}(u_i\\ln(u_i)+(\\ln(u_i)+1)(q_i-u_i))\n\\nonumber\\\\\n&=\\sum_{i\\in I_q}q_i\\ln(q_i)-\\sum_{i\\in I_u}q_i\\ln(u_i)\\nonumber\\\\\n&=\\sum_{i\\in I_q}q_i\\ln\\left(\\frac{q_i}{u_i}\\right)\\nonumber\\\\\n&=D_I(q,u),\n\\end{align}\nand the proof is complete.\\qed\n\\end{proof}","sha256_utf8":"4e6a9b04042d34c7e20ff37da270f46f6dd4a60bee2fd148d5da48a12755f12f"}]},{"id":"S010","semantic_type":"proof_step","kind":"proof_step","title":"闭透视函数给出凸性","source_units":[{"start":34890,"end":35144,"start_line":885,"end_line":891,"raw_tex":"\\ref{p:KLpropi}: \nNoting that $\\psi$ is the {lower semicontinuous convex \nenvelope of \nthe perspective \nassociated to $\\phi$ defined in \\eqref{e:defphi},\nthe convexity and lower semicontinuity follow from \n\\cite[Proposition~9.42]{BauschkeCombettes17}. }\n","sha256_utf8":"011e70919f6d2aa01c141115e2d43bbbb9c9d1013c96d9c89940ce14eed80acc"}]},{"id":"S011","semantic_type":"proof_step","kind":"proof_step","title":"有限求和与支撑集分情形","source_units":[{"start":35146,"end":35650,"start_line":894,"end_line":903,"raw_tex":"\\ref{p:KLpropii}: The convexity and lower semicontinuity of $D_I$\nfollow from \\ref{p:KLpropi} in view of \\eqref{e:Kullback} and \n\\cite[Proposition~8.6]{BauschkeCombettes17}. Let $q$ and $u$ in \n$\\Delta(I)$.\nNote that, if $\\supp q\\not\\subset \\supp u$, there exists $i_0\\in I$\nsuch that $q_{i_0}>0$ and $u_{i_0}=0$, which implies from \n\\eqref{e:Kullback} and \\eqref{e:defpsi} that $D_I(q,u)=+\\infty$ and, \nthus, the inequality holds trivially. Hence, let us assume that \n$I_q=\\supp \nq\\subset \\supp u=I_u$.\n","sha256_utf8":"797fa7c51c569a36727c6704e9f7b4f13c26f53f64cd7ebffad9a7582b43a949"}]},{"id":"S012","semantic_type":"proof_step","kind":"proof_step","title":"沿线段应用熵梯度下界","source_units":[{"start":35650,"end":36553,"start_line":904,"end_line":931,"raw_tex":"Since $u\\in\\reli\\Delta(I_u)$, there exists $\\epsilon>0$\nsuch that, for every $t\\in [-\\epsilon,1[$, \n$u+t(q-u)\\in\\reli\\Delta(I_u)$.\nLet $\\varphi\\colon\\Delta(I_u)\\to\\R\\colon z\\mapsto \n\\sum_{i\\in I_u}\\phi(z_i)$, where $\\phi$ is defined in \n\\eqref{e:defphi}, \nand set\n$\\theta\\colon[-\\epsilon,1]\\to\\R\\colon t\\mapsto \n\\varphi(u+t(q-u))$. \nNote that \n$\\theta$ is continuous in $[-\\epsilon,1]$, continuously differentiable \nin \n$]-\\epsilon,1[$,\nand $\\theta'\\colon t\\mapsto \\scal{\\nabla\\varphi(u+t(q-u))}{q-u}$.\nThen, for every $t\\in]-\\epsilon,1[$ we have $u+t(q-u)$ and $u$ are \nin \n$\\reli\\Delta(I_u)$ and \\cite[Proposition~5.1(a)]{BeckTeb03} \nimplies\n\\begin{align}\n\\theta'(t)-\\theta'(0)&=\\scal{\\nabla\\varphi(u+t(q-u))\n-\\nabla\\varphi(u)}{q-u}\\nonumber\\\\\n&=\n\\frac{1}{t}\\scal{\\nabla\\varphi(u+t(q-u))-\\nabla\\varphi(u)}{u+t(q-u)-u}\n\\nonumber\\\\\n&\\ge \n\\frac{1}{t}\\|t(q-u)\\|_1^2\\nonumber\\\\\n&=t\\|q-u\\|_1^2.\n\\end{align}\n","sha256_utf8":"e7cc2f44cee5a74220de261c42a4ba180fac9b08eb4a42c79987a1f7d328c893"}]},{"id":"S013","semantic_type":"proof_step","kind":"proof_step","title":"积分并取极限得到距离下界","source_units":[{"start":36553,"end":37161,"start_line":932,"end_line":949,"raw_tex":"Therefore,\n\\begin{equation}\n(\\forall s\\in\\left]0,1\\right[)\\quad \n\\theta(s)-\\theta(0)=\\int_0^s\\theta'(t)dt\\ge \ns\\theta'(0)+\\frac{s^2}{2}\\|q-u\\|_1^2,\n\\end{equation}\nand taking $s\\to 1$ we obtain from $(q,u)\\in\\Delta(I_u)^2$ that\n\\begin{align}\n\\frac{1}{2}\\|q-u\\|_1^2&\\le \n\\varphi(q)-\\varphi(u)-\\scal{\\nabla\\varphi(u)}{q-u}\\nonumber\\\\ \n&=\\sum_{i\\in I_u}\\phi(q_i)-\\sum_{i\\in \nI_u}(u_i\\ln(u_i)+(\\ln(u_i)+1)(q_i-u_i))\n\\nonumber\\\\\n&=\\sum_{i\\in I_q}q_i\\ln(q_i)-\\sum_{i\\in I_u}q_i\\ln(u_i)\\nonumber\\\\\n&=\\sum_{i\\in I_q}q_i\\ln\\left(\\frac{q_i}{u_i}\\right)\\nonumber\\\\\n&=D_I(q,u),\n\\end{align}\nand the proof is complete.\\qed\n","sha256_utf8":"12cc26dadad8f8c001001f4751a81bf25dfcb44754d0dabb3c8c9f9e9ea753e9"}]},{"id":"S014","semantic_type":"proposition","kind":"proposition","title":"约束凸性与解的存在性","source_units":[{"start":38063,"end":38647,"start_line":983,"end_line":1005,"raw_tex":"\\begin{proposition}\n\\label{p:exist}\nIn the context of Problem~\\ref{prob:main}, \nlet $g$ be the function defined in \\eqref{e:defg} and let \n$C$ be the set defined in \\eqref{e:defC}. Then, the following hold.\n\\begin{enumerate}\n\\item \n\\label{p:existi}\nThere exists $\\widetilde{q}\\in{\\reli((\\Delta(A\\times \nB))^S)}$ such that\n$g(\\widetilde{q})<0$.\n\n\\item\n\\label{p:existii+}\n$g$ is convex and lower semicontinuous.\n\\item \n\\label{p:existii}\n$C$ is a nonempty convex compact set.\n\\item \n\\label{p:existiii}\nProblem~\\ref{prob:main} admits at least a solution.\n\\end{enumerate}\n\\end{proposition}","sha256_utf8":"7162433df7c3f996f1ee8b9b33925631828d87e8aa9223309c3f3c219c630508"}]},{"id":"S015","semantic_type":"proof","kind":"proof","title":"可行性与存在性的完整证明","source_units":[{"start":38648,"end":39721,"start_line":1006,"end_line":1037,"raw_tex":"\\begin{proof}\n\\ref{p:existi}:\nDefine $\\widetilde{q}\\in(\\Delta(A\\times B))^S$ by\n\\begin{equation}\n(\\forall s\\in S)(\\forall a\\in A)(\\forall b\\in B)\\quad \n\\widetilde{q}_{ab}^s=\\dfrac{1}{|A||B|}{>0}.\n\\end{equation}\nIt follows from \\eqref{e:defg} that $g(\\widetilde{q})=-\\ln|A|<0$.\n\n\n\\ref{p:existii+}: Note that $(u_{as})_{a,s}\\mapsto \n\\sum_{a',s'}p_{s'}u_{a's'}$ is linear and, since $\\psi$ is convex and \nlower semicontinuous in view of \nProposition~\\ref{p:KLprop}\\eqref{p:KLpropi}, $g$ is convex and \nlower semicontinuous. \n\n\\ref{p:existii}: It follows from \\eqref{e:defC}, \\eqref{e:defg}, and \n\\eqref{e:defpsi} that \n\\begin{equation}\n\\label{e:C2}\nC=g^{-1}(\\RM)\\cap(\\Delta(A\\times B))^S.\n\\end{equation}\nSince \\ref{p:existii+} asserts that $g$ is\nconvex and lower semicontinuous, $g^{-1}(\\RM)$ is convex and \nclosed.\nMoreover, since $(\\Delta(A\\times B))^S$ is convex and compact,\nwe deduce from \\eqref{e:C2} that\n$C$ is convex and compact. The result follows from \\ref{p:existi}.\n\n\\ref{p:existiii}: It is a direct consequence of \n\\cite[Theorem~2.12]{Beck2017}.\\qed\n\\end{proof}","sha256_utf8":"933432234b10a1a4312aea399dd56ae30fb8c40b39519ea7ac47be09edc5338c"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"均匀分布提供严格可行点","source_units":[{"start":38662,"end":38925,"start_line":1007,"end_line":1013,"raw_tex":"\\ref{p:existi}:\nDefine $\\widetilde{q}\\in(\\Delta(A\\times B))^S$ by\n\\begin{equation}\n(\\forall s\\in S)(\\forall a\\in A)(\\forall b\\in B)\\quad \n\\widetilde{q}_{ab}^s=\\dfrac{1}{|A||B|}{>0}.\n\\end{equation}\nIt follows from \\eqref{e:defg} that $g(\\widetilde{q})=-\\ln|A|<0$.\n","sha256_utf8":"237e61ac09165c764207e77385e409b1dfcba2c64428617ec4544843b328aa5c"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"线性复合保持约束函数凸性","source_units":[{"start":38927,"end":39170,"start_line":1016,"end_line":1020,"raw_tex":"\\ref{p:existii+}: Note that $(u_{as})_{a,s}\\mapsto \n\\sum_{a',s'}p_{s'}u_{a's'}$ is linear and, since $\\psi$ is convex and \nlower semicontinuous in view of \nProposition~\\ref{p:KLprop}\\eqref{p:KLpropi}, $g$ is convex and \nlower semicontinuous. \n","sha256_utf8":"1731b6cfd970f0ae0b2969f26b1a7b4b9fd7f4f81b050b7cb170e576657747c2"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"次水平集刻画紧凸可行域","source_units":[{"start":39171,"end":39625,"start_line":1022,"end_line":1033,"raw_tex":"\\ref{p:existii}: It follows from \\eqref{e:defC}, \\eqref{e:defg}, and \n\\eqref{e:defpsi} that \n\\begin{equation}\n\\label{e:C2}\nC=g^{-1}(\\RM)\\cap(\\Delta(A\\times B))^S.\n\\end{equation}\nSince \\ref{p:existii+} asserts that $g$ is\nconvex and lower semicontinuous, $g^{-1}(\\RM)$ is convex and \nclosed.\nMoreover, since $(\\Delta(A\\times B))^S$ is convex and compact,\nwe deduce from \\eqref{e:C2} that\n$C$ is convex and compact. The result follows from \\ref{p:existi}.\n","sha256_utf8":"694c48e31cfe1fcdf3cf394e0d9ec4cb16648847375d226164b8dec855df6599"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"紧集上的目标取得最小值","source_units":[{"start":39626,"end":39710,"start_line":1035,"end_line":1036,"raw_tex":"\\ref{p:existiii}: It is a direct consequence of \n\\cite[Theorem~2.12]{Beck2017}.\\qed\n","sha256_utf8":"5f39b5eaf899fc36ce4ce03177199d6be67f697a75e44349f27ca991c4afad69"}]},{"id":"S020","semantic_type":"proof","kind":"proof","title":"迭代收敛定理的完整证明","source_units":[{"start":42603,"end":51951,"start_line":1145,"end_line":1416,"raw_tex":"\\begin{proof}\n\\ref{t:convgeni}:\nNote that, since $\\Delta_\\Xi$ is compact, the existence \nof solutions to \\eqref{e:probXi} is guaranteed by \n{\\cite[Theorem~2.12]{Beck2017}} and \nProposition~\\ref{p:exist}\\eqref{p:existii+}.\n\n\\ref{t:convgeniia}: Note that \\eqref{e:probXi} can be written \nequivalently as \n\\begin{equation}\n\\min_{q\\in [0,+\\infty[^{\\Xi}}f(q)+g_{\\Xi}(q),\n\\end{equation}\nwhere \n{\\begin{equation}\n\\begin{cases}\nf\\colon \\R^{\\Xi}\\to\\left]-\\infty,+\\infty\\right]\\colon q\\mapsto \n\\iota_{V_\\Xi}(q)+\\eta\n\\sum_{s\\in S}\\sum_{(a,b)\\in \\Xi_s}d_{ab}^sq_{ab}^s\\\\\nV_{\\Xi}=\\menge{q\\in \\R^{\\Xi}}{(\\forall s\\in \nS)\\:\\:\\sum_{(a,b)\\in\\Xi_s}q_{ab}^s=1}.\n\\end{cases}\n\\end{equation}\nNote that $g_{\\Xi}=g\\circ E$, where}\n$E\\colon \\R^{\\Xi}\\to\\R^{(A\\times B)^S}$ stands for the \ncanonical extension defined by\n\\begin{equation}\n\\label{e:defE}\n(\\forall q\\in \\R^{\\Xi})(\\forall (a,b,s)\\in A\\times B\\times S)\\quad \n(Eq)_{ab}^s=\n\\begin{cases}\nq_{ab}^s,&\\text{if}\\:\\: (a,b)\\in \\Xi_s;\\\\\n0,&\\text{otherwise}.\n\\end{cases}\n\\end{equation}\nNote that $f$ and $g_{\\Xi}$ are proper, lower semicontinuous, and \nconvex.\nMoreover, by defining \n\\begin{equation}\n\\label{e:defh}\nh\\colon \\R^{\\Xi}\\to\\left]-\\infty,+\\infty\\right]\\colon q\\mapsto \n\\sum_{s\\in \nS}p_s\\varphi_{\\Xi_s}(q^s),\n\\end{equation}\nwhere {$\\varphi_{\\Xi_s}$\nis defined in \\eqref{e:defvarphi}}, it follows from \n\\cite[Example~1]{BauMOR} that $h$ is a Legendre function \nsatisfying $\\overline{\\rm dom }\\,h=[0,+\\infty[^{\\Xi}={\\rm dom}\\, h= \n{\\rm dom}\\, g_{\\Xi}$, \nand by defining $\\widetilde{q}\\in\\R^{\\Xi}$ by\n$$(\\forall s\\in S)(\\forall (a,b)\\in\\Xi_s)\\quad \n\\widetilde{q}_{ab}^s=\\frac{1}{|\\Xi_s|},$$ we have that\n$\\widetilde{q}\\in\\dom f\\cap \\inte\\dom h$.\nMoreover, $h-g_{\\Xi}$ is convex on \n$\\inte\\dom h=\\left]0,+\\infty\\right[^{\\Xi}$. \nIndeed, \nfor every $q\\in \\left]0,+\\infty\\right[^{\\Xi}$, \nit follows from \\eqref{e:defg}, \\eqref{e:defpsi}, and \\eqref{e:defE} \nthat\n\\begin{align}\n\\label{e:hminusgconv}\n(h-g_{\\Xi})(q)&=\n\\sum_{s\\in S}p_s\\left(\\sum_{(a,b)\\in \\Xi^s}q_{ab}^s\\ln \nq_{ab}^s-\\sum_{a\\in A}\\sum_{b\\in \nB}\\psi\\left((Eq)_{ab}^s,\\sum_{a'\\in A}\\sum_{s'\\in \nS}p_{s'}(Eq)_{a'b}^{s'}\\right)\n\\right)\\nonumber\\\\\n&=\n\\sum_{s\\in S}p_s\\sum_{(a,b)\\in \\Xi^s}q_{ab}^s\\left(\\ln \nq_{ab}^s-\\ln\\left(\\frac{q_{ab}^s}{{\\sum\\limits_{a'\\in \nA}\\sum\\limits_{s'\\in \nS}}p_{s'}(Eq)_{a'b}^{s'}}\\right)\\right)\\nonumber\\\\\n&=\n\\sum_{s\\in S}p_s\\sum_{(a,b)\\in \\Xi^s}q_{ab}^s\\ln\\left({\\sum_{a'\\in \nA}\\sum_{s'\\in \nS}p_{s'}(Eq)_{a'b}^{s'}}\\right)\\nonumber\\\\\n&=\n\\sum_{b\\in \nB}\\sum_{a\\in A}\\sum_{s\\in S}p_s(Eq)_{ab}^s\\ln\\left({\\sum_{a'\\in \nA}\\sum_{s'\\in \nS}p_{s'}(Eq)_{a'b}^{s'}}\\right)\\nonumber\\\\\n&=\\varphi_B(T_{\\Xi}(Eq)),\n\\end{align}\nwhich is convex because it is the composition of the convex \nfunction $\\varphi_B$, defined in \\eqref{e:defvarphi},\nwith the linear operators\n$T{_{\\Xi}}$ and $E$\ndefined in \\eqref{e:TXi} and \\eqref{e:defE}, respectively.\nOn the other hand, note that the fixed point recursion in \n\\eqref{e:recursXi} is equivalent to \n\\begin{equation}\n\\label{e:mirrorfbXi}\n(\\forall n\\in\\NN)\\quad q^{n+1}=\\arg\\min_{q\\in \n\\R^\\Xi}f(q)+{\\scal{\\nabla\n g_{\\Xi}(q^n)}{q}}+D_h(q,q^n),\n\\end{equation}\nwhere the existence and uniqueness of the \nsolution to \\eqref{e:mirrorfbXi} are guaranteed by \n\\cite[Lemma~2]{BauMOR},  $(q^n)_{n\\in\\NN}\\subset \n\\reli(\\Delta_\\Xi)=\\prod_{s\\in \nS}\\reli(\\Delta(\\Xi_s))$, and,\nfor every $p\\in\\left]0,+\\infty\\right[^\\Xi$ and $q\\in[0,+\\infty[^\\Xi$,\n\\begin{equation}\n\\label{e:breghXi}\nD_h\\colon (q,p)\\mapsto h(q)-h(p)-\\scal{\\nabla \nh(p)}{q-p}=\\sum_{s\\in S}p_s {D_{\\Xi_s}(q^s,p^s)}\n\\end{equation} \nis the Bregman distance associated to $h$. \nIndeed, given $n\\in\\NN$, since $g_{\\Xi}$ is \ndifferentiable in \n$]0,+\\infty[^{\\Xi}$ and, for every $q\\in \\reli(\\Delta_\\Xi)$, \n$\\nabla \ng_{\\Xi}(q)= \n(p_s\\ln(q_{ab}^s/(T_{\\Xi}(q))_b))_{(a,b)\\in\\Xi_s,\\,s\\in S}$, \n\\eqref{e:mirrorfbXi} is \nequivalent \nto\n{\\begin{align}\n\\quad q^{n+1}&=\\arg\\min_{q\\in V_{\\Xi}}\n\\eta\\sum_{s\\in S}\\sum_{(a,b)\\in\\Xi_s}d_{ab}^sq_{ab}^s+\n\\sum_{s\\in S}\\sum_{(a,b)\\in\\Xi_s}p_sq_{ab}^s\n\\ln\\left(\\frac{q_{ab}^{n,s}}{(T_{\\Xi}(q^n))_b}\\right)\n+\\sum_{s\\in S}\np_sD_{\\Xi_s}(q^s,q^{n,s})\\nonumber\\\\\n&=\\arg\\min_{q\\in \nV_{\\Xi}}\\eta\\sum_{s\\in S}\\sum_{(a,b)\\in\\Xi_s}d_{ab}^sq_{ab}^s+\n\\sum_{s\\in S}p_s\\sum_{(a,b)\\in\\Xi_s}\\phi(q_{ab}^s)-\\sum_{s\\in \nS}p_s\\sum_{(a,b)\\in\\Xi_s}q_{ab}^{s}\\ln\\left((T_{\\Xi}(q^n))_b\\right).\n\\end{align}}\nHence, it follows from  \n{\\cite[Theorem~28.2]{rockafellar1997convex}} and \nProposition~\\ref{p:KLprop}\\eqref{p:KLpropi} that there exists \n$\\xi\\in\\R^S$ such that\n%\\begin{equation}\n%{0 = \\eta d_{ab}^s + p_s (\\ln q_{ab}^s +1) - p_s \\ln \n%(T_{\\Xi}(q^n))_b - \\xi_s }.\n%\\end{equation}\n\\begin{equation}\n(\\forall s\\in S)(\\forall (a,b)\\in\\Xi_s)\\quad 0=\\eta \n\\frac{d_{ab}^s}{p_s}-\\frac{\\xi_s}{p_s}+\n\\ln\\left(\\frac{q_{ab}^{n+1,s}}{T{_{\\Xi}}(q^n)_b}\\right) {+1 },\n\\end{equation}\n{which leads to}\n\\begin{equation}\n\\label{e:almos}\n(\\forall s\\in S)(\\forall (a,b)\\in\\Xi_s)\\quad\nq_{ab}^{n+1,s}=T{_{\\Xi}}(q^n)_be^{-\\eta d_{ab}^s/p_s}\ne^{\\xi_s/p_s{-1}}\n\\end{equation}\nand,  { using that $q^{n+1}\\in V_{\\Xi}$, we conclude \nthat \\eqref{e:almos} reduces to \\eqref{e:recursXi}.\nIn addition, since $h$ is differentiable in $\\RPP^{\\Xi}$,\nit follows from \\eqref{e:hminusgconv} that $\\nabla \nh=\\nabla g_{\\Xi}+T_{\\Xi}^*\\circ\\nabla\\varphi_B\\circ T_{\\Xi}$. Hence,\ngiven $n\\in\\NN$,\nwe deduce from \\eqref{e:hminusgconv} and \\eqref{e:Bdist} that\n\\begin{align}\n\\label{e:auxd}\ng_{\\Xi}(q^{n+1})-g_{\\Xi}(q^{n})&=\nh(q^{n+1})-h(q^n)-\\varphi_B(T_{\\Xi}(q^{n+1}))+\\varphi_B(T_{\\Xi}\n(q^n))\\nonumber\\\\\n&=D_h(q^{n+1},q^n)+\\scal{\\nabla h(q^n)}{q^{n+1}-q^n}\n-\\varphi_B(T_{\\Xi}(q^{n+1}))+\\varphi_B(T_{\\Xi}\n(q^n))\\nonumber\\\\\n&=D_h(q^{n+1},q^n)+\\scal{\\nabla g_{\\Xi}(q^n)}{q^{n+1}-q^n}\n-\\varphi_B(T_{\\Xi}(q^{n+1}))+\\varphi_B(T_{\\Xi}\n(q^n))\\nonumber\\\\\n&\\hspace{6cm}+\\scal{T_{\\Xi}^*\\nabla \n\\varphi_B(T_{\\Xi}(q^n))}{q^{n+1}-q^n}\\nonumber\\\\\n&=D_h(q^{n+1},q^n)+\\scal{\\nabla g_{\\Xi}(q^n)}{q^{n+1}-q^n}\n-\\varphi_B(T_{\\Xi}(q^{n+1}))+\\varphi_B(T_{\\Xi}\n(q^n))\\nonumber\\\\\n&\\hspace{5cm}+\\scal{\\nabla \n\\varphi_B(T_{\\Xi}(q^n))}{T_{\\Xi}(q^{n+1})-T_{\\Xi}(q^n)}\\nonumber\\\\\n&=D_h(q^{n+1},q^n)+\\scal{\\nabla g_{\\Xi}(q^n)}{q^{n+1}-q^n}\n-D_B(T_{\\Xi}(q^{n+1}),T_{\\Xi}(q^n)).\n\\end{align}\nMoreover, the convexity of $g_{\\Xi}$\nimplies \n$$(\\forall q\\in\\RP^{\\Xi})\\quad g_{\\Xi}(q^n)-g_{\\Xi}(q)\\le \\scal{\\nabla \ng_{\\Xi}(q^n)}{q^n-q},$$\nwhich, combined with \\eqref{e:auxd}, yields\n\\begin{align}\n\\label{e:propg}\n(\\forall q\\in\\RP^{\\Xi})\\quad \ng_{\\Xi}(q^{n+1})-g_{\\Xi}(q)\n&\\le D_h(q^{n+1},q^n)\n-D_B(T_{\\Xi}(q^{n+1}),T_{\\Xi}(q^n))+\\scal{\\nabla \ng_{\\Xi}(q^n)}{q^{n+1}-q}.\n\\end{align}\nOn the other hand, the first order optimality conditions of \n\\eqref{e:mirrorfbXi} and the three points identity \n\\cite[Lemma~3.1]{chen1993convergence}\nimply\n\\begin{align}\n\\label{e:auxf}\n(\\forall q\\in\\RP^{\\Xi})\\quad f(q^{n+1})-f(q)\n&\\le\\scal{\\nabla \nh(q^n)-\\nabla h(q^{n+1})}{q^{n+1}-q}+\\scal{\\nabla \ng_{\\Xi}(q^n)}{q-q^{n+1}}\\nonumber\\\\\n&=D_h(q,q^n)-D_h(q,q^{n+1})-D_h(q^{n+1},q^n)+\\scal{\\nabla \ng_{\\Xi}(q^n)}{q-q^{n+1}}.\n\\end{align} \nHence, by summing up \\eqref{e:auxf} and \\eqref{e:propg} we \ndeduce\n\\begin{equation}\n\\label{e:trueineq}\n(\\forall q\\in\\RP^{\\Xi})\\quad F(q^{n+1},\\eta)-F(q,\\eta)\\le \nD_h(q,q^n)-D_h(q,q^{n+1})-D_B(T_{\\Xi}(q^{n+1}),T_{\\Xi}(q^n)).\n\\end{equation}\nNote that, by taking $q=q^n$ in \\eqref{e:trueineq}, we deduce that \n$(F(q^n,\\eta))_{n\\in\\NN}$ is nonincreasing and, by taking \n$q=\\overline{q}$, where $\\overline{q}$ is any solution to \n\\eqref{e:probXi}, we obtain\n \\begin{equation}\n\\label{e:inequs}\n0\\le F(q^{n+1},\\eta)-F(\\overline{q},\\eta)\\le \nD_h(\\overline{q},q^n)-D_h(\\overline{q},q^{n+1})\n-D_B(T_{\\Xi}(q^{n+1}),T_{\\Xi}(q^n)).\n\\end{equation}\nThis} \nimplies that the sequence \n$(D_h(\\overline{q},q^n))_{n\\in\\NN}$\nis nonincreasing, {nonnegative} in view of  \nProposition~\\ref{p:KLprop}\\eqref{p:KLpropii} and, \ntherefore, { for every solution $\\overline{q}$ to \n\\eqref{e:probXi}, we have}\n\\begin{equation}\n\\label{e:convergentlamqXi}\n\\big(D_h(\\overline{q},q^n)\\big)_{n\\in\\NN}\\quad\\text{ \nis convergent}.\n\\end{equation}\n{\nTherefore, by summing from $0$ to $n-1$ in \\eqref{e:inequs}, we \ndeduce\n\\begin{align}\n\\label{e:ineq}\n0\\le n(F(q^n,\\eta)-F(\\overline{q},\\eta))\n&\\le \n\\sum_{k=0}^{n-1}\\big(F(q^{k+1},\\eta)-F(\\overline{q},\\eta)\\big)\n\\nonumber\\\\\n&\\le D_h(\\overline{q},q^0)-D_h(\\overline{q},q^{n})\n-\\sum_{k=0}^{n-1}D_B(T_{\\Xi}(q^{k+1}),T_{\\Xi}(q^k)),\n\\end{align}\n which yields the result.\n\n\\ref{t:convgeniib}: It is a direct consequence of \\ref{t:convgeniia}.\n\n\\ref{t:convgeniic}: Note that \\ref{t:convgeniib} implies that\n$F(q^{n},\\eta)\\to \nF(\\overline{q},\\eta)$.}\nFinally, since $\\Delta_{\\Xi}$ is compact and recalling that \n$(q^n)_{n\\in\\NN}\\subset \\reli(\\Delta_{\\Xi})$,\nlet $q^*$ be an accumulation point of $(q^n)_{n\\in\\NN}$, say \n$q^{k_n}\\to \nq^*$. The continuity of $F(\\cdot,\\eta)$ relatively to \n$\\Delta_{\\Xi}$ implies that\n$F(q^{k_n},\\eta)\\to \nF(q^*,\\eta)=F(\\overline{q},\\eta)\\in\\R$,\n which { implies that $q^*$ is also a solution to \n \\eqref{e:probXi}.  Furthermore, since \\eqref{e:breghXi} \ncan be written as\n\\begin{equation}\n(\\forall n\\in\\NN)\\quad D_h(q^*,q^n)=\n\\sum_{s\\in S}p_s\\sum_{(a,b)\\in\\supp \nq^{*,s}}q^{*,s}_{ab}\\ln\\left(\\frac{q^{*,s}_{ab}}{q^{n,s}_{ab}}\\right),\n\\end{equation}\n$q^{k_n}\\to q^*$ implies $D_{h}({q}^*,q^{k_n})\\to 0$, and } \nwe deduce from \n\\eqref{e:convergentlamqXi} that $D_{h}({q}^*,q^{n})\\to 0$.\nWe conclude that $q^n\\to q^*$ from \nProposition~\\ref{p:KLprop}\\eqref{p:KLpropii}.\\qed\n\\end{proof}","sha256_utf8":"2ac4eab0ed7b024cb22d79e26048b74fef9e2ae84fea2e0ac1d7476ce66c2515"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"紧性保证受限问题有解","source_units":[{"start":42617,"end":42825,"start_line":1146,"end_line":1150,"raw_tex":"\\ref{t:convgeni}:\nNote that, since $\\Delta_\\Xi$ is compact, the existence \nof solutions to \\eqref{e:probXi} is guaranteed by \n{\\cite[Theorem~2.12]{Beck2017}} and \nProposition~\\ref{p:exist}\\eqref{p:existii+}.\n","sha256_utf8":"77be88bf8b8d626e3b09ec2c9902e5b50a08ab6e857aee589dfc2bf42322adae"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"分裂目标并定义零延拓","source_units":[{"start":42826,"end":43689,"start_line":1152,"end_line":1180,"raw_tex":"\\ref{t:convgeniia}: Note that \\eqref{e:probXi} can be written \nequivalently as \n\\begin{equation}\n\\min_{q\\in [0,+\\infty[^{\\Xi}}f(q)+g_{\\Xi}(q),\n\\end{equation}\nwhere \n{\\begin{equation}\n\\begin{cases}\nf\\colon \\R^{\\Xi}\\to\\left]-\\infty,+\\infty\\right]\\colon q\\mapsto \n\\iota_{V_\\Xi}(q)+\\eta\n\\sum_{s\\in S}\\sum_{(a,b)\\in \\Xi_s}d_{ab}^sq_{ab}^s\\\\\nV_{\\Xi}=\\menge{q\\in \\R^{\\Xi}}{(\\forall s\\in \nS)\\:\\:\\sum_{(a,b)\\in\\Xi_s}q_{ab}^s=1}.\n\\end{cases}\n\\end{equation}\nNote that $g_{\\Xi}=g\\circ E$, where}\n$E\\colon \\R^{\\Xi}\\to\\R^{(A\\times B)^S}$ stands for the \ncanonical extension defined by\n\\begin{equation}\n\\label{e:defE}\n(\\forall q\\in \\R^{\\Xi})(\\forall (a,b,s)\\in A\\times B\\times S)\\quad \n(Eq)_{ab}^s=\n\\begin{cases}\nq_{ab}^s,&\\text{if}\\:\\: (a,b)\\in \\Xi_s;\\\\\n0,&\\text{otherwise}.\n\\end{cases}\n\\end{equation}\nNote that $f$ and $g_{\\Xi}$ are proper, lower semicontinuous, and \nconvex.\n","sha256_utf8":"3c565fb936c84a003f0fb471b6f1d6ba2a65a24e816af18fe691486afffb8206"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"加权熵核满足定义域条件","source_units":[{"start":43689,"end":44281,"start_line":1181,"end_line":1196,"raw_tex":"Moreover, by defining \n\\begin{equation}\n\\label{e:defh}\nh\\colon \\R^{\\Xi}\\to\\left]-\\infty,+\\infty\\right]\\colon q\\mapsto \n\\sum_{s\\in \nS}p_s\\varphi_{\\Xi_s}(q^s),\n\\end{equation}\nwhere {$\\varphi_{\\Xi_s}$\nis defined in \\eqref{e:defvarphi}}, it follows from \n\\cite[Example~1]{BauMOR} that $h$ is a Legendre function \nsatisfying $\\overline{\\rm dom }\\,h=[0,+\\infty[^{\\Xi}={\\rm dom}\\, h= \n{\\rm dom}\\, g_{\\Xi}$, \nand by defining $\\widetilde{q}\\in\\R^{\\Xi}$ by\n$$(\\forall s\\in S)(\\forall (a,b)\\in\\Xi_s)\\quad \n\\widetilde{q}_{ab}^s=\\frac{1}{|\\Xi_s|},$$ we have that\n$\\widetilde{q}\\in\\dom f\\cap \\inte\\dom h$.\n","sha256_utf8":"2222f7dd948f0858d54000a41d7770b851c4a67c032b7bb1f8a7a1e4dc0663b4"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"边缘熵恒等式给出相对光滑性","source_units":[{"start":44281,"end":45450,"start_line":1197,"end_line":1231,"raw_tex":"Moreover, $h-g_{\\Xi}$ is convex on \n$\\inte\\dom h=\\left]0,+\\infty\\right[^{\\Xi}$. \nIndeed, \nfor every $q\\in \\left]0,+\\infty\\right[^{\\Xi}$, \nit follows from \\eqref{e:defg}, \\eqref{e:defpsi}, and \\eqref{e:defE} \nthat\n\\begin{align}\n\\label{e:hminusgconv}\n(h-g_{\\Xi})(q)&=\n\\sum_{s\\in S}p_s\\left(\\sum_{(a,b)\\in \\Xi^s}q_{ab}^s\\ln \nq_{ab}^s-\\sum_{a\\in A}\\sum_{b\\in \nB}\\psi\\left((Eq)_{ab}^s,\\sum_{a'\\in A}\\sum_{s'\\in \nS}p_{s'}(Eq)_{a'b}^{s'}\\right)\n\\right)\\nonumber\\\\\n&=\n\\sum_{s\\in S}p_s\\sum_{(a,b)\\in \\Xi^s}q_{ab}^s\\left(\\ln \nq_{ab}^s-\\ln\\left(\\frac{q_{ab}^s}{{\\sum\\limits_{a'\\in \nA}\\sum\\limits_{s'\\in \nS}}p_{s'}(Eq)_{a'b}^{s'}}\\right)\\right)\\nonumber\\\\\n&=\n\\sum_{s\\in S}p_s\\sum_{(a,b)\\in \\Xi^s}q_{ab}^s\\ln\\left({\\sum_{a'\\in \nA}\\sum_{s'\\in \nS}p_{s'}(Eq)_{a'b}^{s'}}\\right)\\nonumber\\\\\n&=\n\\sum_{b\\in \nB}\\sum_{a\\in A}\\sum_{s\\in S}p_s(Eq)_{ab}^s\\ln\\left({\\sum_{a'\\in \nA}\\sum_{s'\\in \nS}p_{s'}(Eq)_{a'b}^{s'}}\\right)\\nonumber\\\\\n&=\\varphi_B(T_{\\Xi}(Eq)),\n\\end{align}\nwhich is convex because it is the composition of the convex \nfunction $\\varphi_B$, defined in \\eqref{e:defvarphi},\nwith the linear operators\n$T{_{\\Xi}}$ and $E$\ndefined in \\eqref{e:TXi} and \\eqref{e:defE}, respectively.\n","sha256_utf8":"6911253d239095d221db9a32964d6f1eb7621dc53f4b99a5b3337147b5198bbd"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"迭代的布雷格曼近端表示","source_units":[{"start":45450,"end":46170,"start_line":1232,"end_line":1251,"raw_tex":"On the other hand, note that the fixed point recursion in \n\\eqref{e:recursXi} is equivalent to \n\\begin{equation}\n\\label{e:mirrorfbXi}\n(\\forall n\\in\\NN)\\quad q^{n+1}=\\arg\\min_{q\\in \n\\R^\\Xi}f(q)+{\\scal{\\nabla\n g_{\\Xi}(q^n)}{q}}+D_h(q,q^n),\n\\end{equation}\nwhere the existence and uniqueness of the \nsolution to \\eqref{e:mirrorfbXi} are guaranteed by \n\\cite[Lemma~2]{BauMOR},  $(q^n)_{n\\in\\NN}\\subset \n\\reli(\\Delta_\\Xi)=\\prod_{s\\in \nS}\\reli(\\Delta(\\Xi_s))$, and,\nfor every $p\\in\\left]0,+\\infty\\right[^\\Xi$ and $q\\in[0,+\\infty[^\\Xi$,\n\\begin{equation}\n\\label{e:breghXi}\nD_h\\colon (q,p)\\mapsto h(q)-h(p)-\\scal{\\nabla \nh(p)}{q-p}=\\sum_{s\\in S}p_s {D_{\\Xi_s}(q^s,p^s)}\n\\end{equation} \nis the Bregman distance associated to $h$. \n","sha256_utf8":"69b8bc66fda5ed5b0c551bd43e88c07fed9ea924cdd38d79c365f8b3ff2d3383"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"计算梯度并展开近端子问题","source_units":[{"start":46170,"end":46906,"start_line":1252,"end_line":1272,"raw_tex":"Indeed, given $n\\in\\NN$, since $g_{\\Xi}$ is \ndifferentiable in \n$]0,+\\infty[^{\\Xi}$ and, for every $q\\in \\reli(\\Delta_\\Xi)$, \n$\\nabla \ng_{\\Xi}(q)= \n(p_s\\ln(q_{ab}^s/(T_{\\Xi}(q))_b))_{(a,b)\\in\\Xi_s,\\,s\\in S}$, \n\\eqref{e:mirrorfbXi} is \nequivalent \nto\n{\\begin{align}\n\\quad q^{n+1}&=\\arg\\min_{q\\in V_{\\Xi}}\n\\eta\\sum_{s\\in S}\\sum_{(a,b)\\in\\Xi_s}d_{ab}^sq_{ab}^s+\n\\sum_{s\\in S}\\sum_{(a,b)\\in\\Xi_s}p_sq_{ab}^s\n\\ln\\left(\\frac{q_{ab}^{n,s}}{(T_{\\Xi}(q^n))_b}\\right)\n+\\sum_{s\\in S}\np_sD_{\\Xi_s}(q^s,q^{n,s})\\nonumber\\\\\n&=\\arg\\min_{q\\in \nV_{\\Xi}}\\eta\\sum_{s\\in S}\\sum_{(a,b)\\in\\Xi_s}d_{ab}^sq_{ab}^s+\n\\sum_{s\\in S}p_s\\sum_{(a,b)\\in\\Xi_s}\\phi(q_{ab}^s)-\\sum_{s\\in \nS}p_s\\sum_{(a,b)\\in\\Xi_s}q_{ab}^{s}\\ln\\left((T_{\\Xi}(q^n))_b\\right).\n\\end{align}}\n","sha256_utf8":"da5eb841b3a652be04d10a0a94a3ceca78605e03be2c609b2778f4f20199690a"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"乘子条件导出归一化递推","source_units":[{"start":46906,"end":47661,"start_line":1273,"end_line":1294,"raw_tex":"Hence, it follows from  \n{\\cite[Theorem~28.2]{rockafellar1997convex}} and \nProposition~\\ref{p:KLprop}\\eqref{p:KLpropi} that there exists \n$\\xi\\in\\R^S$ such that\n%\\begin{equation}\n%{0 = \\eta d_{ab}^s + p_s (\\ln q_{ab}^s +1) - p_s \\ln \n%(T_{\\Xi}(q^n))_b - \\xi_s }.\n%\\end{equation}\n\\begin{equation}\n(\\forall s\\in S)(\\forall (a,b)\\in\\Xi_s)\\quad 0=\\eta \n\\frac{d_{ab}^s}{p_s}-\\frac{\\xi_s}{p_s}+\n\\ln\\left(\\frac{q_{ab}^{n+1,s}}{T{_{\\Xi}}(q^n)_b}\\right) {+1 },\n\\end{equation}\n{which leads to}\n\\begin{equation}\n\\label{e:almos}\n(\\forall s\\in S)(\\forall (a,b)\\in\\Xi_s)\\quad\nq_{ab}^{n+1,s}=T{_{\\Xi}}(q^n)_be^{-\\eta d_{ab}^s/p_s}\ne^{\\xi_s/p_s{-1}}\n\\end{equation}\nand,  { using that $q^{n+1}\\in V_{\\Xi}$, we conclude \nthat \\eqref{e:almos} reduces to \\eqref{e:recursXi}.\n","sha256_utf8":"a8681580195ce1f0356199ff2e6b7e3edac322f209d7518be41deb582ee45bfe"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"推导熵目标的精确增量","source_units":[{"start":47661,"end":48719,"start_line":1295,"end_line":1320,"raw_tex":"In addition, since $h$ is differentiable in $\\RPP^{\\Xi}$,\nit follows from \\eqref{e:hminusgconv} that $\\nabla \nh=\\nabla g_{\\Xi}+T_{\\Xi}^*\\circ\\nabla\\varphi_B\\circ T_{\\Xi}$. Hence,\ngiven $n\\in\\NN$,\nwe deduce from \\eqref{e:hminusgconv} and \\eqref{e:Bdist} that\n\\begin{align}\n\\label{e:auxd}\ng_{\\Xi}(q^{n+1})-g_{\\Xi}(q^{n})&=\nh(q^{n+1})-h(q^n)-\\varphi_B(T_{\\Xi}(q^{n+1}))+\\varphi_B(T_{\\Xi}\n(q^n))\\nonumber\\\\\n&=D_h(q^{n+1},q^n)+\\scal{\\nabla h(q^n)}{q^{n+1}-q^n}\n-\\varphi_B(T_{\\Xi}(q^{n+1}))+\\varphi_B(T_{\\Xi}\n(q^n))\\nonumber\\\\\n&=D_h(q^{n+1},q^n)+\\scal{\\nabla g_{\\Xi}(q^n)}{q^{n+1}-q^n}\n-\\varphi_B(T_{\\Xi}(q^{n+1}))+\\varphi_B(T_{\\Xi}\n(q^n))\\nonumber\\\\\n&\\hspace{6cm}+\\scal{T_{\\Xi}^*\\nabla \n\\varphi_B(T_{\\Xi}(q^n))}{q^{n+1}-q^n}\\nonumber\\\\\n&=D_h(q^{n+1},q^n)+\\scal{\\nabla g_{\\Xi}(q^n)}{q^{n+1}-q^n}\n-\\varphi_B(T_{\\Xi}(q^{n+1}))+\\varphi_B(T_{\\Xi}\n(q^n))\\nonumber\\\\\n&\\hspace{5cm}+\\scal{\\nabla \n\\varphi_B(T_{\\Xi}(q^n))}{T_{\\Xi}(q^{n+1})-T_{\\Xi}(q^n)}\\nonumber\\\\\n&=D_h(q^{n+1},q^n)+\\scal{\\nabla g_{\\Xi}(q^n)}{q^{n+1}-q^n}\n-D_B(T_{\\Xi}(q^{n+1}),T_{\\Xi}(q^n)).\n\\end{align}\n","sha256_utf8":"cdae3f3a965d6648872c7ab56a15afc17af4b38789e91df192c6a66ddfae5da4"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"凸性给出光滑项比较界","source_units":[{"start":48719,"end":49103,"start_line":1321,"end_line":1333,"raw_tex":"Moreover, the convexity of $g_{\\Xi}$\nimplies \n$$(\\forall q\\in\\RP^{\\Xi})\\quad g_{\\Xi}(q^n)-g_{\\Xi}(q)\\le \\scal{\\nabla \ng_{\\Xi}(q^n)}{q^n-q},$$\nwhich, combined with \\eqref{e:auxd}, yields\n\\begin{align}\n\\label{e:propg}\n(\\forall q\\in\\RP^{\\Xi})\\quad \ng_{\\Xi}(q^{n+1})-g_{\\Xi}(q)\n&\\le D_h(q^{n+1},q^n)\n-D_B(T_{\\Xi}(q^{n+1}),T_{\\Xi}(q^n))+\\scal{\\nabla \ng_{\\Xi}(q^n)}{q^{n+1}-q}.\n\\end{align}\n","sha256_utf8":"c8292465f5cddbcf0ba9044b90341308a6311a31ffbe68dafac6a58782fad98f"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"三点恒等式控制非光滑项","source_units":[{"start":49103,"end":49537,"start_line":1334,"end_line":1346,"raw_tex":"On the other hand, the first order optimality conditions of \n\\eqref{e:mirrorfbXi} and the three points identity \n\\cite[Lemma~3.1]{chen1993convergence}\nimply\n\\begin{align}\n\\label{e:auxf}\n(\\forall q\\in\\RP^{\\Xi})\\quad f(q^{n+1})-f(q)\n&\\le\\scal{\\nabla \nh(q^n)-\\nabla h(q^{n+1})}{q^{n+1}-q}+\\scal{\\nabla \ng_{\\Xi}(q^n)}{q-q^{n+1}}\\nonumber\\\\\n&=D_h(q,q^n)-D_h(q,q^{n+1})-D_h(q^{n+1},q^n)+\\scal{\\nabla \ng_{\\Xi}(q^n)}{q-q^{n+1}}.\n\\end{align} \n","sha256_utf8":"cdcdc64902da5f87487073d0592bf9d44de397459bea1fa47af443920f08baaa"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"合并得到单步基本不等式","source_units":[{"start":49537,"end":49776,"start_line":1347,"end_line":1353,"raw_tex":"Hence, by summing up \\eqref{e:auxf} and \\eqref{e:propg} we \ndeduce\n\\begin{equation}\n\\label{e:trueineq}\n(\\forall q\\in\\RP^{\\Xi})\\quad F(q^{n+1},\\eta)-F(q,\\eta)\\le \nD_h(q,q^n)-D_h(q,q^{n+1})-D_B(T_{\\Xi}(q^{n+1}),T_{\\Xi}(q^n)).\n\\end{equation}\n","sha256_utf8":"2cf484516f16843746456c939a23940fedc2a77373036118eb9545aa04ad20b2"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"目标下降与解距收敛","source_units":[{"start":49776,"end":50544,"start_line":1354,"end_line":1375,"raw_tex":"Note that, by taking $q=q^n$ in \\eqref{e:trueineq}, we deduce that \n$(F(q^n,\\eta))_{n\\in\\NN}$ is nonincreasing and, by taking \n$q=\\overline{q}$, where $\\overline{q}$ is any solution to \n\\eqref{e:probXi}, we obtain\n \\begin{equation}\n\\label{e:inequs}\n0\\le F(q^{n+1},\\eta)-F(\\overline{q},\\eta)\\le \nD_h(\\overline{q},q^n)-D_h(\\overline{q},q^{n+1})\n-D_B(T_{\\Xi}(q^{n+1}),T_{\\Xi}(q^n)).\n\\end{equation}\nThis} \nimplies that the sequence \n$(D_h(\\overline{q},q^n))_{n\\in\\NN}$\nis nonincreasing, {nonnegative} in view of  \nProposition~\\ref{p:KLprop}\\eqref{p:KLpropii} and, \ntherefore, { for every solution $\\overline{q}$ to \n\\eqref{e:probXi}, we have}\n\\begin{equation}\n\\label{e:convergentlamqXi}\n\\big(D_h(\\overline{q},q^n)\\big)_{n\\in\\NN}\\quad\\text{ \nis convergent}.\n\\end{equation}\n","sha256_utf8":"880f9b4a68c53239deb09cc48d397fbded9b9e7935a058c135d8ddced361329b"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"望远镜求和得到速率估计","source_units":[{"start":50544,"end":50911,"start_line":1376,"end_line":1388,"raw_tex":"{\nTherefore, by summing from $0$ to $n-1$ in \\eqref{e:inequs}, we \ndeduce\n\\begin{align}\n\\label{e:ineq}\n0\\le n(F(q^n,\\eta)-F(\\overline{q},\\eta))\n&\\le \n\\sum_{k=0}^{n-1}\\big(F(q^{k+1},\\eta)-F(\\overline{q},\\eta)\\big)\n\\nonumber\\\\\n&\\le D_h(\\overline{q},q^0)-D_h(\\overline{q},q^{n})\n-\\sum_{k=0}^{n-1}D_B(T_{\\Xi}(q^{k+1}),T_{\\Xi}(q^k)),\n\\end{align}\n which yields the result.\n","sha256_utf8":"92a5e6796adb668570671ebb946f67519549e23e00bca7c2ac137b9b578effb3"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"推出次线性速率与可求和性","source_units":[{"start":50912,"end":50982,"start_line":1390,"end_line":1390,"raw_tex":"\\ref{t:convgeniib}: It is a direct consequence of \\ref{t:convgeniia}.\n","sha256_utf8":"621f8300d909cfa1ea7df5f452d49ff411240ccfab37dae4cec72b19716948c2"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"目标收敛保证聚点最优","source_units":[{"start":50983,"end":51523,"start_line":1392,"end_line":1404,"raw_tex":"\\ref{t:convgeniic}: Note that \\ref{t:convgeniib} implies that\n$F(q^{n},\\eta)\\to \nF(\\overline{q},\\eta)$.}\nFinally, since $\\Delta_{\\Xi}$ is compact and recalling that \n$(q^n)_{n\\in\\NN}\\subset \\reli(\\Delta_{\\Xi})$,\nlet $q^*$ be an accumulation point of $(q^n)_{n\\in\\NN}$, say \n$q^{k_n}\\to \nq^*$. The continuity of $F(\\cdot,\\eta)$ relatively to \n$\\Delta_{\\Xi}$ implies that\n$F(q^{k_n},\\eta)\\to \nF(q^*,\\eta)=F(\\overline{q},\\eta)\\in\\R$,\n which { implies that $q^*$ is also a solution to \n \\eqref{e:probXi}.  Furthermore, since \\eqref{e:breghXi} \n","sha256_utf8":"9ab97a8856672107701412475517fdd06a9b2a965112ad07a0973464e3dfe42a"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"相对熵下界提升为全序列收敛","source_units":[{"start":51523,"end":51940,"start_line":1405,"end_line":1415,"raw_tex":"can be written as\n\\begin{equation}\n(\\forall n\\in\\NN)\\quad D_h(q^*,q^n)=\n\\sum_{s\\in S}p_s\\sum_{(a,b)\\in\\supp \nq^{*,s}}q^{*,s}_{ab}\\ln\\left(\\frac{q^{*,s}_{ab}}{q^{n,s}_{ab}}\\right),\n\\end{equation}\n$q^{k_n}\\to q^*$ implies $D_{h}({q}^*,q^{k_n})\\to 0$, and } \nwe deduce from \n\\eqref{e:convergentlamqXi} that $D_{h}({q}^*,q^{n})\\to 0$.\nWe conclude that $q^n\\to q^*$ from \nProposition~\\ref{p:KLprop}\\eqref{p:KLpropii}.\\qed\n","sha256_utf8":"d8eb272f6399af6070a369e7aba8875659717ae704f44fbaecb59160a8fe7c8d"}]},{"id":"S037","semantic_type":"assumption","kind":"assumption","title":"受限支撑与惩罚目标设定","source_units":[{"start":40611,"end":41116,"start_line":1068,"end_line":1081,"raw_tex":"For every $s\\in S$, let $\\Xi_s\\subset A\\times B$ {and, \nfor every $(a,b)\\in\\Xi_s$, let $d_{ab}^s\\in\\R$.\nMoreover, let $\\eta\\ge 0$,} denote by \n$\\Xi=\\prod_{s\\in S}\\Xi_s$ and by $\\Delta_\\Xi=\\prod_{s\\in \nS}\\Delta(\\Xi_s)$, and consider the problem\n\\begin{equation}\n\\label{e:probXi}\\tag{$P_{\\Xi}$}\n\\min_{q\\in\\Delta_{\\Xi}}F(q,\\eta):={g_{\\Xi}(q)}\n+\\eta\\sum_{s\\in S}\n\\sum_{(a,b)\\in \\Xi_s}d_{ab}^sq_{ab}^s,\n\\end{equation}\nwhere {$g_{\\Xi}$ is the restriction of $g$ \ndefined in \n\\eqref{e:defg} to $\\RP^\\Xi$.} Then \n","sha256_utf8":"fb35bd11737bc378ef0ba1ff406276ad8997fbadf1a9389efeaad3ff60e40e88"}]},{"id":"S038","semantic_type":"assumption","kind":"assumption","title":"内部初值与固定点递推设定","source_units":[{"start":41235,"end":41797,"start_line":1087,"end_line":1107,"raw_tex":"\\item\\label{t:convgenii}  Given $q^0\\in {\\rm \nri}(\\Delta_\\Xi)$, \nconsider the sequence generated \nby the recurrence \n\\begin{equation}\n\\label{e:recursXi}\n(\\forall n\\in\\NN)(\\forall s\\in S)(\\forall (a,b)\\in \\Xi_s)\\quad \nq_{ab}^{n+1,s}=\n\\frac{(T_\\Xi(q^n))_be^{-\\eta \nd_{ab}^s/p_s}}{\\sum\\limits_{(a',b')\\in\\Xi_s}\n(T_\\Xi(q^n))_{b'}e^{-\\eta \nd_{a'b'}^s/p_s}}, \n\\end{equation}\nwhere $T_{\\Xi}\\colon\\Delta_\\Xi\\to \n\\Delta(B)$ is defined by \n\\begin{equation}\n\\label{e:TXi}\nT{_\\Xi}\\colon \nq\\mapsto \\left(\\sum_{a\\in A}\\sum_{s\\in S}p_sq_{ab}^s\\right)_{b\\in \nB}.\n\\end{equation}\n","sha256_utf8":"1e940d596ca176e953129c7c9cc12a9b8b0e0895d2eac7955076a52783778720"}]},{"id":"S039","semantic_type":"external_fact","kind":"external_fact","title":"闭透视与有限和的凸性","source_units":[],"declared_external":true,"declaration_text":["闭凸熵函数的闭透视函数是适当、下半连续、凸的；有限个此类函数的和在共同有效定义域非空时保持这些性质。对应 BauschkeCombettes17, Example 9.35, Propositions 9.42 and 8.6。"]},{"id":"S040","semantic_type":"external_fact","kind":"external_fact","title":"单纯形上的熵梯度下界","source_units":[],"declared_external":true,"declaration_text":["对概率单纯形相对内部的 x,y，负熵满足 ⟨∇φ(x)−∇φ(y),x−y⟩≥||x−y||₁²。对应 BeckTeb03, Proposition 5.1(a)。"]},{"id":"S041","semantic_type":"external_fact","kind":"external_fact","title":"紧集上的最小值存在","source_units":[],"declared_external":true,"declaration_text":["适当下半连续函数在非空紧可行集上，只要存在有限函数值，即取得最小值。对应 Beck2017, Theorem 2.12。"]},{"id":"S042","semantic_type":"external_fact","kind":"external_fact","title":"熵核的勒让德性质","source_units":[],"declared_external":true,"declaration_text":["正权重可分负熵在非负正交象限上是勒让德函数，定义域内部为正正交象限；在概率单纯形上，其布雷格曼距离为相对熵。对应 cover-book-2006, Section 2.3; Beck2017, Example 9.10; Teboulle18, Example 2.1; BauMOR, Example 1。"]},{"id":"S043","semantic_type":"external_fact","kind":"external_fact","title":"近端子问题的内部唯一解","source_units":[],"declared_external":true,"declaration_text":["在作者验证的勒让德核、凸目标及定义域相容条件下，布雷格曼近端子问题具有唯一解，且该解位于熵核定义域内部。对应 BauMOR, Lemma 2。"]},{"id":"S044","semantic_type":"external_fact","kind":"external_fact","title":"凸规划的乘子最优性条件","source_units":[],"declared_external":true,"declaration_text":["满足相对内部约束资格条件的凸规划可通过拉格朗日乘子及一阶驻点条件刻画最优解，仿射等式约束的乘子提供相应法向量。对应 rockafellar1997convex, Theorem 28.2。"]},{"id":"S045","semantic_type":"external_fact","kind":"external_fact","title":"布雷格曼三点恒等式","source_units":[],"declared_external":true,"declaration_text":["在相关梯度存在时，⟨∇h(y)−∇h(z),z−x⟩=D_h(x,y)−D_h(x,z)−D_h(z,y)。对应 chen1993convergence, Lemma 3.1。"]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S002","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S010","kind":"supports"},{"source_id":"S006","target_id":"S010","kind":"supports"},{"source_id":"S039","target_id":"S010","kind":"supports"},{"source_id":"S010","target_id":"S011","kind":"supports"},{"source_id":"S005","target_id":"S011","kind":"supports"},{"source_id":"S004","target_id":"S011","kind":"supports"},{"source_id":"S039","target_id":"S011","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S006","target_id":"S012","kind":"supports"},{"source_id":"S040","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S005","target_id":"S013","kind":"supports"},{"source_id":"S010","target_id":"S009","kind":"supports"},{"source_id":"S011","target_id":"S009","kind":"supports"},{"source_id":"S012","target_id":"S009","kind":"supports"},{"source_id":"S013","target_id":"S009","kind":"supports"},{"source_id":"S009","target_id":"S008","kind":"supports"},{"source_id":"S002","target_id":"S016","kind":"supports"},{"source_id":"S003","target_id":"S016","kind":"supports"},{"source_id":"S007","target_id":"S016","kind":"supports"},{"source_id":"S007","target_id":"S017","kind":"supports"},{"source_id":"S008","target_id":"S017","kind":"supports"},{"source_id":"S002","target_id":"S018","kind":"supports"},{"source_id":"S005","target_id":"S018","kind":"supports"},{"source_id":"S007","target_id":"S018","kind":"supports"},{"source_id":"S016","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S041","target_id":"S019","kind":"supports"},{"source_id":"S016","target_id":"S015","kind":"supports"},{"source_id":"S017","target_id":"S015","kind":"supports"},{"source_id":"S018","target_id":"S015","kind":"supports"},{"source_id":"S019","target_id":"S015","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S021","kind":"supports"},{"source_id":"S037","target_id":"S021","kind":"supports"},{"source_id":"S041","target_id":"S021","kind":"supports"},{"source_id":"S004","target_id":"S022","kind":"supports"},{"source_id":"S007","target_id":"S022","kind":"supports"},{"source_id":"S014","target_id":"S022","kind":"supports"},{"source_id":"S037","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S006","target_id":"S023","kind":"supports"},{"source_id":"S042","target_id":"S023","kind":"supports"},{"source_id":"S005","target_id":"S024","kind":"supports"},{"source_id":"S006","target_id":"S024","kind":"supports"},{"source_id":"S007","target_id":"S024","kind":"supports"},{"source_id":"S022","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S038","target_id":"S024","kind":"supports"},{"source_id":"S022","target_id":"S026","kind":"supports"},{"source_id":"S023","target_id":"S026","kind":"supports"},{"source_id":"S005","target_id":"S026","kind":"supports"},{"source_id":"S007","target_id":"S026","kind":"supports"},{"source_id":"S038","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S008","target_id":"S027","kind":"supports"},{"source_id":"S044","target_id":"S027","kind":"supports"},{"source_id":"S038","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S043","target_id":"S025","kind":"supports"},{"source_id":"S006","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S028","kind":"supports"},{"source_id":"S023","target_id":"S028","kind":"supports"},{"source_id":"S006","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S014","target_id":"S029","kind":"supports"},{"source_id":"S025","target_id":"S030","kind":"supports"},{"source_id":"S045","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S022","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S021","target_id":"S032","kind":"supports"},{"source_id":"S008","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S025","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S035","kind":"supports"},{"source_id":"S007","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S032","target_id":"S036","kind":"supports"},{"source_id":"S025","target_id":"S036","kind":"supports"},{"source_id":"S004","target_id":"S036","kind":"supports"},{"source_id":"S008","target_id":"S036","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S020","kind":"supports"},{"source_id":"S023","target_id":"S020","kind":"supports"},{"source_id":"S024","target_id":"S020","kind":"supports"},{"source_id":"S025","target_id":"S020","kind":"supports"},{"source_id":"S026","target_id":"S020","kind":"supports"},{"source_id":"S027","target_id":"S020","kind":"supports"},{"source_id":"S028","target_id":"S020","kind":"supports"},{"source_id":"S029","target_id":"S020","kind":"supports"},{"source_id":"S030","target_id":"S020","kind":"supports"},{"source_id":"S031","target_id":"S020","kind":"supports"},{"source_id":"S032","target_id":"S020","kind":"supports"},{"source_id":"S033","target_id":"S020","kind":"supports"},{"source_id":"S034","target_id":"S020","kind":"supports"},{"source_id":"S035","target_id":"S020","kind":"supports"},{"source_id":"S036","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S001","kind":"supports"}]}}
{"record_id":"2506.12397v1","paper":{"title":"Monotone and nonmonotone linearized block coordinate descent methods for nonsmooth composite optimization problems","authors":["Yassine Nabou","Lahcen El Bourkhissi","Sebastian U. Stich","Tuomo Valkonen"],"arxiv_id":"2506.12397","version":"v1","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-026-03120-0","abstract_url":"https://arxiv.org/abs/2506.12397v1","pdf_url":"https://arxiv.org/pdf/2506.12397v1"},"context":{"raw_tex":"\\begin{equation}  \n    \\label{eq:problem}  \n    \\min_{x \\in \\text{dom}\\,\\varphi} \\varphi(x): = f(x) + h(F(x)) + \\sum_{i=1}^{n_{\\text{block}}} g_i(x^i), \n\\end{equation}\n\nwhere \\( F(x): = (F_1(x), \\cdots, F_m(x)) \\) and \\( f \\) are nonlinear differentiable functions defined on \\(\\text{dom}\\,\\varphi\\). The functions \\( g_i \\) are proper, lower semicontinuous (lsc), and convex.\nThey may be nonsmooth, while \\( h \\) is a smooth and convex function. Here, \\( n = \\sum_{i=1}^{n_{\\text{block}}} n_i \\), \\( x^i \\in \\mathbb{R}^{n_i} \\), and \\( g(x) := \\sum_{i=1}^{n_{\\text{block}}} g_i(x^i) \\).\\footnote{Note that, from the point of view of general theory, $f$ is superfluous, and can be subsumed by $h \\circ F$. It is, however, practical to explicitly include it for our application examples.}\n\n\nIn what follows, we use $\\| \\cdot \\|$ to denote the 2-norm of a vector or a matrix, respectively. Let $n = \\sum_{i=1}^{n_\\text{block}} n_i$. For a differentiable function $\\phi: \\mathbb{R}^n \\to \\mathbb{R}$, we denote its gradient with respect to the $i^{\\text{th}}$ block at a point $x$ as $\\nabla_i \\phi(x) \\in \\mathbb{R}^{n_i}$, for any $i \\in \\{1, \\dots, n_\\text{block}\\}$. For a differentiable vector function $F: \\mathbb{R}^n \\to \\mathbb{R}^m$, we denote its Jacobian with respect to the $i^{\\text{th}}$ block at a given point $x$ as $\\nabla_i F(x) \\in \\mathbb{R}^{m \\times n_i}$. We define the distance from a point $x$ to a set $A$ as $\\dist(x, A) := \\min_{a \\in A} \\| x - a \\|$. Furthermore, for a proper, lower semicontinuous (lsc) function $g: \\mathbb{R}^n \\to \\mathbb{R}$, we use $\\partial g(x)$ to refer to its Mordukhovich (aka.~limiting) subdifferential at a point $x$, and $\\partial^{\\infty} g(x)$ to refer to the horizon subdifferential.\nThis, instead of the plain convex subdifferential, is required due to the nonconvexity of $h \\circ F$ and hence the composite objective $\\phi$.\nFor more details about the subdifferential of nonsmooth functions, we refer to \\cite{rockafellar2009variational}. In the following, we say that a point \\( x^* \\in \\mathbb{R}^n \\) is a stationary point of problem~\\eqref{eq:problem} if \\( 0 \\in \\partial \\varphi(x^*) \\). Moreover, for \\( \\epsilon > 0 \\), \\( x^* \\) is said to be an \\( \\epsilon \\)-stationary point of problem \\eqref{eq:problem} if \\( \\dist\\left( 0, \\partial \\varphi(x^*) \\right) \\leq \\epsilon \\).\n\n\n\\begin{assumption}\\label{ass:1}\n    \\begin{enumerate} %[itemsep=1pt]\n    \\item Given a compact set $\\mathcal{C}$, the gradients of $f$ and $F_j$ for $j=1:m$ are block coordinate-wise Lipschitz continuous in $\\mathcal{C}$, i.e., $\\forall i\\in \\{1,\\cdots,n_{\\text{block}}\\},\\;x,y\\in \\mathcal{C}$\n        \\begin{align*}\n        \\|\\nabla_i f(x) - \\nabla_if(y)\\|\\leq L^{f}_i \\|x - y\\|,\n        \\|\\nabla_i F_j(x) - \\nabla_i F_j(y)\\|\\leq L^{F_j}_i \\|x - y\\|.\n        \\end{align*}\n    \\item $h$ is convex, differentiable with Lipschitz gradient\n        \\begin{align*}\n            \\|\\nabla h(u) - \\nabla h(v)\\|\\leq L_{h} \\|u - v\\|,\\;\\forall \\;u,v\\in\\mathbb{R}^{m}.\n        \\end{align*}\n    \\item Each $g_i$ is a proper, lsc, convex function.\n    \\item $ \\forall x\\in\\mathcal{C}$, $u\\in\\mathbb{R}^m$ and $i\\in\\{1,\\cdots,n_{\\text{block}}\\}$ we have\n        $\n             f(x) \\geq f^*,\\; h(u)\\geq h^*,\\;g_i(x) \\geq g^*.% \\varphi(x) \\geq \\varphi^*,\\;\n        $\n    \\end{enumerate}\n\\end{assumption}\n\nIn the sequel, for all $i=\\{1,\\ldots,n_{\\text{block}}\\}$, $\\bar{x}^i, s\\in\\mathbb{R}^{n_i}$ and $\\bar{x}\\in\\mathbb{R}^n$, we denote\n\\begin{align*}\n    &l^i_f(s;\\bar{x}^i)\n    :=\n    f(\\bar{x})+\\langle\\nabla_i f(\\bar{x}),s-\\bar{x}^i\\rangle,\\quad\n    l^i_F \\big(s;\\bar{x}^i \\big):=F(\\bar{x})+ \\nabla_i F(\\bar{x}) \\big(s-\\bar{x}^i \\big),\\\\\n    &\\quad\\text{and}\\quad\n    \\bar{\\varphi}^i(s;\\bar{x}) = l^i_f \\big( s;\\bar{x}^i \\big)+ h \\big( l^i_F(s;\\bar{x}^i) \\big) + g_i(s).\n\\end{align*}\n\n\\begin{algorithm*}[ht]\n\\caption{Monotone LiBCoD }\n\\label{alg:libcod}\n\n\\KwIn{$x_0$, $\\beta_1 \\geq \\frac{\\beta_{\\min}}{2}>0$, $p_{\\min} \\in (0,1)$}\nSet $k = 0$\n\n\\While{termination criterion not satisfied}{\n  Pick a random index $i_k \\in \\{1,\\cdots,n_{\\text{block}}\\}$ with probability $p_{i_k} \\geq p_{\\min}$\\:\n\n  \\Repeat(\n    %\\tcp*[f]{\\label{line:dec_mon} $\\varphi(x_{k+1}) \\leq  \\varphi(x_k) - \\frac{\\beta_{k+1}}{2}\\|x_{k+1} - x_k\\|^{2}$}\n  ){\n    \\textcolor{blue}{$\\varphi(x_{k+1}) \\leq  \\varphi(x_k) - \\frac{\\beta_{k+1}}{2}\\|x_{k+1} - x_k\\|^{2}$}\n  }{\n    $\\beta_{k+1} := 2\\beta_{k+1}$ \\label{line:lin_search}\n\n    $s_{k+1} := \\arg\\min\\limits_{s \\in \\mathbb{R}^{n_{i_k}}} \\resizebox{!}{2ex}{$\\langle\\nabla_{i_k} f(x_k),s-x_k^{i_k}\\rangle + h\\left(F(x_k)+\\langle\\nabla_{i_k} F(x_k),s-x_k^{i_k}\\rangle\\right) + g_{i_k}(s) + \\frac{\\beta_{k+1}}{2}\\|s - x^{i_k}\\|^2$}$ \n    %$s_{k+1} := \\arg\\min\\limits_{s \\in \\mathbb{R}^{n_{i_k}}} \\langle\\nabla_{i_k} f(x_k),s-x_k^{i_k}\\rangle+ h\\left(F(x_k)+\\langle\\nabla_{i_k} F(x_k),s-x_k^{i_k}\\rangle\\right) + g_{i_k}(s) + \\frac{\\beta_{k+1}}{2}\\|s - x^{i_k}\\|^2$ \\label{line:libcod:subprob}\n    \n    $x^{i_k}_{k+1} := s_{k+1}$ and $x^{i}_{k+1} = x_k^i$ for $i\\neq i_k$\n  }\n\n  Set $k=k+1$ and $\\beta_{k+1} = \\max\\left\\{\\frac{\\beta_{k}}{4},\\frac{\\beta_{\\min}}{2}\\right\\}$\n}\n\\end{algorithm*}\n\n$\n\\Delta{x}_{k+1} := x_{k+1} - x_k .\n$\n\n\n\\begin{equation} \\label{eq:beta_cond}\n    \\beta_{k+1} \\geq L^f_{i_k} + L^F_{i_k}\\sqrt{2 L_h}\\sqrt{\\varphi(x_k)- \\varphi^*},\n\\end{equation}\n\nwhere $\\varphi^* = h^* + f^* + g^*$, then the process in Monotone LiBCoD \\ref{alg:libcod}, is well defined. \n\n\nNote that in Monotone LiBCoD \\ref{alg:libcod}, $\\beta_{k+1}$ is always upper bounded by\n\\begin{equation}\n\\label{bar_gamma}\n    \\bar{\\beta} := \\sup_{k\\geq 1}\\beta_k \\leq 2 \\left(L^f+ L^F\\sqrt{2L_h}\\sqrt{\\varphi(x_0)-\\varphi^*}\\right),\n\\end{equation}\n\n where $L^f:=\\max_{i\\in\\{1, \\ldots, n_{\\text{block}}\\}} L^f_{i}$  and  $L^F:=\\max_{i\\in\\{1, \\ldots, n_{\\text{block}}\\}} L^F_{i}$.\n","units":[{"start":7414,"end":7581,"start_line":188,"end_line":191,"raw_tex":"\\begin{equation}  \n    \\label{eq:problem}  \n    \\min_{x \\in \\text{dom}\\,\\varphi} \\varphi(x): = f(x) + h(F(x)) + \\sum_{i=1}^{n_{\\text{block}}} g_i(x^i), \n\\end{equation}","sha256_utf8":"e18c3e77d70f4d0cd84c7015885c59ec4ede1e2bb8c46dc8273b9458db5d4443"},{"start":7584,"end":8203,"start_line":192,"end_line":193,"raw_tex":"where \\( F(x): = (F_1(x), \\cdots, F_m(x)) \\) and \\( f \\) are nonlinear differentiable functions defined on \\(\\text{dom}\\,\\varphi\\). The functions \\( g_i \\) are proper, lower semicontinuous (lsc), and convex.\nThey may be nonsmooth, while \\( h \\) is a smooth and convex function. Here, \\( n = \\sum_{i=1}^{n_{\\text{block}}} n_i \\), \\( x^i \\in \\mathbb{R}^{n_i} \\), and \\( g(x) := \\sum_{i=1}^{n_{\\text{block}}} g_i(x^i) \\).\\footnote{Note that, from the point of view of general theory, $f$ is superfluous, and can be subsumed by $h \\circ F$. It is, however, practical to explicitly include it for our application examples.}\n","sha256_utf8":"184cb8a27f24077254a3ab0b1094c2254e0c15fd3cd1573753fcd5e1773f701b"},{"start":19731,"end":21291,"start_line":239,"end_line":241,"raw_tex":"In what follows, we use $\\| \\cdot \\|$ to denote the 2-norm of a vector or a matrix, respectively. Let $n = \\sum_{i=1}^{n_\\text{block}} n_i$. For a differentiable function $\\phi: \\mathbb{R}^n \\to \\mathbb{R}$, we denote its gradient with respect to the $i^{\\text{th}}$ block at a point $x$ as $\\nabla_i \\phi(x) \\in \\mathbb{R}^{n_i}$, for any $i \\in \\{1, \\dots, n_\\text{block}\\}$. For a differentiable vector function $F: \\mathbb{R}^n \\to \\mathbb{R}^m$, we denote its Jacobian with respect to the $i^{\\text{th}}$ block at a given point $x$ as $\\nabla_i F(x) \\in \\mathbb{R}^{m \\times n_i}$. We define the distance from a point $x$ to a set $A$ as $\\dist(x, A) := \\min_{a \\in A} \\| x - a \\|$. Furthermore, for a proper, lower semicontinuous (lsc) function $g: \\mathbb{R}^n \\to \\mathbb{R}$, we use $\\partial g(x)$ to refer to its Mordukhovich (aka.~limiting) subdifferential at a point $x$, and $\\partial^{\\infty} g(x)$ to refer to the horizon subdifferential.\nThis, instead of the plain convex subdifferential, is required due to the nonconvexity of $h \\circ F$ and hence the composite objective $\\phi$.\nFor more details about the subdifferential of nonsmooth functions, we refer to \\cite{rockafellar2009variational}. In the following, we say that a point \\( x^* \\in \\mathbb{R}^n \\) is a stationary point of problem~\\eqref{eq:problem} if \\( 0 \\in \\partial \\varphi(x^*) \\). Moreover, for \\( \\epsilon > 0 \\), \\( x^* \\) is said to be an \\( \\epsilon \\)-stationary point of problem \\eqref{eq:problem} if \\( \\dist\\left( 0, \\partial \\varphi(x^*) \\right) \\leq \\epsilon \\).\n","sha256_utf8":"68c4f0098bd951184e943d7c29388de6db4ac1f39617c6ebcf2b8064d22149d7"},{"start":21626,"end":22612,"start_line":246,"end_line":263,"raw_tex":"\\begin{assumption}\\label{ass:1}\n    \\begin{enumerate} %[itemsep=1pt]\n    \\item Given a compact set $\\mathcal{C}$, the gradients of $f$ and $F_j$ for $j=1:m$ are block coordinate-wise Lipschitz continuous in $\\mathcal{C}$, i.e., $\\forall i\\in \\{1,\\cdots,n_{\\text{block}}\\},\\;x,y\\in \\mathcal{C}$\n        \\begin{align*}\n        \\|\\nabla_i f(x) - \\nabla_if(y)\\|\\leq L^{f}_i \\|x - y\\|,\n        \\|\\nabla_i F_j(x) - \\nabla_i F_j(y)\\|\\leq L^{F_j}_i \\|x - y\\|.\n        \\end{align*}\n    \\item $h$ is convex, differentiable with Lipschitz gradient\n        \\begin{align*}\n            \\|\\nabla h(u) - \\nabla h(v)\\|\\leq L_{h} \\|u - v\\|,\\;\\forall \\;u,v\\in\\mathbb{R}^{m}.\n        \\end{align*}\n    \\item Each $g_i$ is a proper, lsc, convex function.\n    \\item $ \\forall x\\in\\mathcal{C}$, $u\\in\\mathbb{R}^m$ and $i\\in\\{1,\\cdots,n_{\\text{block}}\\}$ we have\n        $\n             f(x) \\geq f^*,\\; h(u)\\geq h^*,\\;g_i(x) \\geq g^*.% \\varphi(x) \\geq \\varphi^*,\\;\n        $\n    \\end{enumerate}\n\\end{assumption}","sha256_utf8":"373c12d4cac8292afaeb04529c622b9f7838a5a5404bb121ab618d5a1c4bf46d"},{"start":22751,"end":23233,"start_line":265,"end_line":273,"raw_tex":"In the sequel, for all $i=\\{1,\\ldots,n_{\\text{block}}\\}$, $\\bar{x}^i, s\\in\\mathbb{R}^{n_i}$ and $\\bar{x}\\in\\mathbb{R}^n$, we denote\n\\begin{align*}\n    &l^i_f(s;\\bar{x}^i)\n    :=\n    f(\\bar{x})+\\langle\\nabla_i f(\\bar{x}),s-\\bar{x}^i\\rangle,\\quad\n    l^i_F \\big(s;\\bar{x}^i \\big):=F(\\bar{x})+ \\nabla_i F(\\bar{x}) \\big(s-\\bar{x}^i \\big),\\\\\n    &\\quad\\text{and}\\quad\n    \\bar{\\varphi}^i(s;\\bar{x}) = l^i_f \\big( s;\\bar{x}^i \\big)+ h \\big( l^i_F(s;\\bar{x}^i) \\big) + g_i(s).\n\\end{align*}","sha256_utf8":"c39ad7517393b517670dac67f0c5ae2604d157b2925278cba0ab30328d5fead5"},{"start":23345,"end":24669,"start_line":276,"end_line":301,"raw_tex":"\\begin{algorithm*}[ht]\n\\caption{Monotone LiBCoD }\n\\label{alg:libcod}\n\n\\KwIn{$x_0$, $\\beta_1 \\geq \\frac{\\beta_{\\min}}{2}>0$, $p_{\\min} \\in (0,1)$}\nSet $k = 0$\n\n\\While{termination criterion not satisfied}{\n  Pick a random index $i_k \\in \\{1,\\cdots,n_{\\text{block}}\\}$ with probability $p_{i_k} \\geq p_{\\min}$\\:\n\n  \\Repeat(\n    %\\tcp*[f]{\\label{line:dec_mon} $\\varphi(x_{k+1}) \\leq  \\varphi(x_k) - \\frac{\\beta_{k+1}}{2}\\|x_{k+1} - x_k\\|^{2}$}\n  ){\n    \\textcolor{blue}{$\\varphi(x_{k+1}) \\leq  \\varphi(x_k) - \\frac{\\beta_{k+1}}{2}\\|x_{k+1} - x_k\\|^{2}$}\n  }{\n    $\\beta_{k+1} := 2\\beta_{k+1}$ \\label{line:lin_search}\n\n    $s_{k+1} := \\arg\\min\\limits_{s \\in \\mathbb{R}^{n_{i_k}}} \\resizebox{!}{2ex}{$\\langle\\nabla_{i_k} f(x_k),s-x_k^{i_k}\\rangle + h\\left(F(x_k)+\\langle\\nabla_{i_k} F(x_k),s-x_k^{i_k}\\rangle\\right) + g_{i_k}(s) + \\frac{\\beta_{k+1}}{2}\\|s - x^{i_k}\\|^2$}$ \n    %$s_{k+1} := \\arg\\min\\limits_{s \\in \\mathbb{R}^{n_{i_k}}} \\langle\\nabla_{i_k} f(x_k),s-x_k^{i_k}\\rangle+ h\\left(F(x_k)+\\langle\\nabla_{i_k} F(x_k),s-x_k^{i_k}\\rangle\\right) + g_{i_k}(s) + \\frac{\\beta_{k+1}}{2}\\|s - x^{i_k}\\|^2$ \\label{line:libcod:subprob}\n    \n    $x^{i_k}_{k+1} := s_{k+1}$ and $x^{i}_{k+1} = x_k^i$ for $i\\neq i_k$\n  }\n\n  Set $k=k+1$ and $\\beta_{k+1} = \\max\\left\\{\\frac{\\beta_{k}}{4},\\frac{\\beta_{\\min}}{2}\\right\\}$\n}\n\\end{algorithm*}","sha256_utf8":"3286c01a903650d7658fc8b52d3b4c1f3232bb2298b4e826117537c951eeb319"},{"start":25151,"end":25190,"start_line":303,"end_line":305,"raw_tex":"$\n\\Delta{x}_{k+1} := x_{k+1} - x_k .\n$\n","sha256_utf8":"3b0dcde8ebcb34181ffa107a197574d72a0927863714d2f08acfe1ff39280acf"},{"start":25505,"end":25643,"start_line":309,"end_line":311,"raw_tex":"\\begin{equation} \\label{eq:beta_cond}\n    \\beta_{k+1} \\geq L^f_{i_k} + L^F_{i_k}\\sqrt{2 L_h}\\sqrt{\\varphi(x_k)- \\varphi^*},\n\\end{equation}","sha256_utf8":"70222bdb74e09e01e3334f0123061c2724c60bdd02e7668820ed6e911593ad23"},{"start":25644,"end":25753,"start_line":312,"end_line":312,"raw_tex":"where $\\varphi^* = h^* + f^* + g^*$, then the process in Monotone LiBCoD \\ref{alg:libcod}, is well defined. \n","sha256_utf8":"0e9945a4f9a9887fcb4bc016adeb5ae2bedde47e3d3c3989f286aacbc903681b"},{"start":33491,"end":33739,"start_line":446,"end_line":450,"raw_tex":"Note that in Monotone LiBCoD \\ref{alg:libcod}, $\\beta_{k+1}$ is always upper bounded by\n\\begin{equation}\n\\label{bar_gamma}\n    \\bar{\\beta} := \\sup_{k\\geq 1}\\beta_k \\leq 2 \\left(L^f+ L^F\\sqrt{2L_h}\\sqrt{\\varphi(x_0)-\\varphi^*}\\right),\n\\end{equation}","sha256_utf8":"1f39fb8d444304cb841bf10478e2d9ea70c28da67adb69b2f9f7d205b3b36836"},{"start":33740,"end":33870,"start_line":451,"end_line":451,"raw_tex":" where $L^f:=\\max_{i\\in\\{1, \\ldots, n_{\\text{block}}\\}} L^f_{i}$  and  $L^F:=\\max_{i\\in\\{1, \\ldots, n_{\\text{block}}\\}} L^F_{i}$.\n","sha256_utf8":"4ab1f514c0388d5dcd2565f91b325651a642a369332c603037633e13a6ed7f87"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}[Iteration complexity]\n    \\label{thm:convergence:monotone}\n    Let $(x_k)_{k\\geq 0}$ be generated by Monotone LiBCoD \\ref{alg:libcod}. If \\cref{ass:1} holds on a compact set to which the iterates belong, then, for any $\\epsilon>0$,  Monotone LiBCoD \\ref{alg:libcod} yields, in expectation, an $\\epsilon$-stationary point of \\eqref{eq:problem}, i.e.,\n    \\[\n     \\E\\left[\\dist(0, \\partial\\varphi(x_{k+1})) \\right] \\leq \\epsilon,\n    \\]\n%  \\textcolor{red}{where $\\partial \\varphi(x)$ denotes the Mordukhovich (aka.~limiting) subdifferential of $\\varphi$ at $x$,}\n    within  \n    \\begin{gather*}\n        \\left(\\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min} \\epsilon^2} + \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min} \\epsilon} \\right)\\left(\\varphi(x_0) - \\varphi^*\\right)\n    \\end{gather*}  \n    iterations.\n\\end{theorem}","units":[{"start":43343,"end":44239,"start_line":597,"end_line":609,"raw_tex":"\\begin{theorem}[Iteration complexity]\n    \\label{thm:convergence:monotone}\n    Let $(x_k)_{k\\geq 0}$ be generated by Monotone LiBCoD \\ref{alg:libcod}. If \\cref{ass:1} holds on a compact set to which the iterates belong, then, for any $\\epsilon>0$,  Monotone LiBCoD \\ref{alg:libcod} yields, in expectation, an $\\epsilon$-stationary point of \\eqref{eq:problem}, i.e.,\n    \\[\n     \\E\\left[\\dist(0, \\partial\\varphi(x_{k+1})) \\right] \\leq \\epsilon,\n    \\]\n%  \\textcolor{red}{where $\\partial \\varphi(x)$ denotes the Mordukhovich (aka.~limiting) subdifferential of $\\varphi$ at $x$,}\n    within  \n    \\begin{gather*}\n        \\left(\\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min} \\epsilon^2} + \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min} \\epsilon} \\right)\\left(\\varphi(x_0) - \\varphi^*\\right)\n    \\end{gather*}  \n    iterations.\n\\end{theorem}","sha256_utf8":"7a2b6c2872a12f4cf7c3c6e3df8c7cff496e537116453d47c6d1e930cc1b893c"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\n    \\label{lemma1}\n    Let $(x_k)_{k\\geq 0}$ be generated by Monotone LiBCoD \\ref{alg:libcod}. If \\cref{ass:1} holds on a compact set $\\mathcal{C}$ containing the iterates, and $\\beta_{k+1}$ satisfies \\eqref{eq:beta_cond}, then\n    \\begin{align}\n        \\label{decrease}\n         \\varphi(x_{k+1}) \\leq  \\varphi(x_{k}) - \\frac{\\beta_{k+1}}{2} \\|x_{k+1} - x_k\\|^2.\n    \\end{align}\n\\end{lemma}\n\n\\begin{proof}\n    This proof follows a reasoning similar to that in \\cite{marumo2024accelerated} [Lemma 3.1], adapted to the coordinate descent framework with an additional smooth function $f$.  \n    %\\seb{it would be nice to repeat there some of the definitions (for better readability)}\n    %\\seb{it seems the proof (for the second term) follows exactly \\cite{marumo2024accelerated}? maybe mention this here, and explain the differences}\n    Note that the subproblem's objective function\n    \\[\n        q(s) := \\bar{\\varphi}^{i_k}(s;x_k)+\\frac{\\beta_{k+1}}{2}\\|s-x^i_k\\|^2,\n    \\]\n    By the strong convexity of this objective and $s_{k+1}$ being a minimiser, i.e., $0 \\in \\partial q(s_{k+1})$,\n    \\[\n        q(x_k^i)\n        \\ge\n        q(s_{k+1}) + \\frac{\\beta_{k+1}}{2}\\|s_{k+1}-x^i_k\\|^2.\n    \\]\n    Hence, expanding $q$,\n    % is strongly convex with a strong convexity constant $\\beta_{k+1}$, then,\n    % \\[\n    %    \\bar{\\varphi}^{i_k}(x_k;x_k)+\\frac{\\beta_{k+1}}{2}\\|x^i_k-x^i_k\\|^2 \\geq \\bar{\\varphi}^{i_k}(s_{k+1};x_k)+\\frac{\\beta_{k+1}}{2}\\|s_{k+1}-x^i_k\\|^2 + \\frac{\\beta_{k+1}}{2}\\norm{x_{k+1} - x_k}^2 \n    % \\]\n    % combining this with the optimality of $s_{k+1}$,  \n    the fact that $\\bar{\\varphi}^{i_k}(x^{i_k}_k;x_k)=\\varphi(x_{k}) - \\sum_{i\\neq i_k} g_i(x^i_{k})$ and making use of the fact that $x^i_{k+1} = x^i_k$ for $i\\neq i_k$, we get\n    \\begin{equation} \\label{initial_decrease}\n        \\bar{\\varphi}^{i_k}(s_{k+1};x_k)\\leq \\varphi(x_{k}) - \\sum_{i\\neq i_k} g_i(x^i_{k+1}) -\\beta_{k+1}\\|x_{k+1}-x_k\\|^2.\n    \\end{equation}\n%    \\seb{is it $\\frac{\\beta_{k+1}}{2}$? \\textcolor{red}{just $\\beta_{k+1}$ since we use the strong convexity of $\\bar{\\varphi}^{i_k}(\\cdot;x_k)$}}%\n%    \\seb{I still don't see this. Strong convexity gives $\\varphi(x) \\geq \\varphi(s_{k+1}) + \\frac{\\beta}{2} \\| x - s_{k+1}\\|^2$ ? \\textcolor{red}{ It is \\(\\varphi(x) + \\frac{\\beta}{2}\\|x - x_k\\|^2\\) that is strongly convex, whereas \\(\\varphi(x)\\) is not. Thus, we have:\n% \\[\n% \\left(\\varphi(s) + \\frac{\\beta}{2} \\|s - x_{k}\\|^2\\right) \\geq \\left(\\varphi(s_{k+1}) + \\frac{\\beta}{2} \\|s_{k+1} - x_{k}\\|^2\\right) + \\frac{\\beta}{2} \\|s_{k+1} - x_{k}\\|^2.\n% \\]\n% %%SEB:\n% ok, tricky; \n% } }%\n    Furthermore, since \\( f \\) has a block Lipschitz gradient with a Lipschitz constant \\( L_{i_k}^f \\), it follows that \\cite{nesterov2018lectures}\n        \\begin{equation}\n        \\label{smooth_object}\n        f(x_{k+1}) - l_f^{i_k}(s_{k+1};x_k) \\leq \\frac{L^f_{i_k}}{2}\\|x_{k+1}-x_k\\|^2.\n    \\end{equation}\n    %\\seb{can you add a reference to the equation that you are using? (also it would help to remind the readers of the definition of $l_f$)\n    %why is there no term with $\\nabla f$ ? (a similar equation is also used in (14) below). $s_{k+1}$ is not a minimizer of $f$}\n     Moreover, since \\( h \\) has a Lipschitz gradient with a Lipschitz constant \\( L_h \\), we have\n    \\begin{equation}\n    \\label{decrease_feasib}\n    \\begin{split}\n         h(F(x_{k+1}))& -  h(l^{i_k}_F(s_{k+1};x_k)) \n        \\\\\n        &\\leq   \\langle \\nabla h \\left( l^{i_k}_F(s_{k+1};x_k)\\right), F(x_{k+1})-l^{i_k}_F(s_{k+1};x_k) \\rangle \\\\\n        \\MoveEqLeft[-1] + \\frac{L_h}{2}\\|F(x_{k+1})-l^{i_k}_F(s_{k+1};x_k)\\|^2 \n        \\\\\n        & \\leq \\left\\|\\nabla h \\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right\\| \\|F(x_{k+1})-l^{i_k}_F(s_{k+1};x_k) \\|\\\\\n        \\MoveEqLeft[-1] + \\frac{L_h}{2}\\|F(x_{k+1})-l^{i_k}_F(s_{k+1};x_k)\\|^2 \n        \\\\\n        &\\overset{\\text{Ass. } \\ref{ass:1}}{\\leq} \\left\\|\\nabla h \\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right\\|  \\frac{L^F_{i_k}}{2}\\|\\Delta x_{k+1}\\|^2 + \\frac{L_h}{2} \\left(\\frac{L^F_{i_k}}{2}\\|\\Delta x_{k+1}\\|^2\\right)^2. \n    \\end{split}\n    \\end{equation}\n    Using the Young's inequality $2\\sqrt{a b} - a \\leq b$ for any $a, b \\geq 0$, and the following relation for $u =l^{i_k}_F(s_{k+1};x_k) \\in \\mathbb{R}^m$, with $v = u - \\frac{1}{L_h} \\nabla h(u)$\n    \\[\n        h^* - h(u) \\leq h(v) - h(u)\n        \\leq \\langle \\nabla h(u), v - u \\rangle + \\frac{L_h}{2} \\norm{v - u}^2 = -\\frac{1}{2L_h} \\norm{\\nabla h(u)}^2,\n    \\]\n    we bound\n    \\begin{equation}\n        \\label{bound_grad_norm_squared}\n        \\begin{split}\n            &\n            \\left\\|\\nabla h \\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right\\| - \\frac{ \\beta_{k+1} - L^f_{i_k}}{2L^F_{i_k}} \\leq  \\frac{L^F_{i_k}}{ 2(\\beta_{k+1} - L^f_{i_k})}    \\left\\|\\nabla h \\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right\\|^2\n            \\\\\n            &\n            \\leq\n            \\frac{L^F_{i_k}L_h}{ \\beta_{k+1} - L^f_{i_k}} \\left( h(l^{i_k}_F(s_{k+1};x_k)) - h^*  \\right)\n            %\\\\\n            %&\n            =\n            \\frac{L^F_{i_k}L_h}{ \\beta_{k+1} - L^f_{i_k}}\n            \\\\\n            \\MoveEqLeft[-4]\n            \\left( \\bar{\\varphi}^{i_k}(s_{k+1};x_k) - h^* - g_{i_k}(s_{k+1})- f(x_{k+1}) + f(x_{k+1}) - l_f^{i_k}(s_{k+1};x_k)  \\right)\n            \\\\\n            &\n            \\overset{\\eqref{initial_decrease}, \\eqref{smooth_object}}{\\leq}\n            \\frac{L^F_{i_k} L_h}{\\beta_{k+1} - L^f_{i_k}} \\left( \\varphi(x_k) - \\varphi^* - \\frac{2\\beta_{k+1} - L^F_{i_k}}{2}\\|\\Delta x_{k+1}\\|^2 \\right)\n            \\\\\n            &\n            \\leq\n            \\frac{L^F_{i_k} L_h}{\\beta_{k+1} - L^f_{i_k}} \\left( \\varphi(x_k) - \\varphi^*\\right) - L^F_{i_k} L_h\\|\\Delta x_{k+1}\\|^2\n            \\\\\n            &\n            \\overset{\\eqref{eq:beta_cond}}{\\leq}\\frac{\\beta_{k+1}-L^f_{i_k}}{2L^F_{i_k}} - L^F_{i_k} L_h\\|\\Delta x_{k+1}\\|^2,\n        \\end{split}\n    \\end{equation}\n    where third inequality follows from the definition of $\\bar{\\varphi}^{i_k}$. Moreover, Using \\eqref{bound_grad_norm_squared} in \\eqref{decrease_feasib}, we deduce that\n    \\begin{equation}\n        \\label{to_use_next}\n        \\begin{split}\n                    h\\left(F(x_{k+1})\\right) &- h\\left(l^{i_k}_F(x_{k+1};x_k) \\right)\\\\\n        & \\leq\n        \\frac{\\beta_{k+1}-L^f_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2 - \\left(\\frac{L_h (L^F_{i_k})^2}{2} - \\frac{L_h (L^{F}_{i_k})^2}{8} \\right) \\|\\Delta x_{k+1}\\|^4\n        \\\\\n        &=\n        \\frac{\\beta_{k+1}-L^f_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2-  \\frac{3 L_h (L^{F}_{i_k})^2}{8}  \\|\\Delta x_{k+1}\\|^4.\n        \\end{split}\n    \\end{equation}\n    Moreover, we have\n    \\begin{align*}\n    \\begin{split}\n        \\varphi(x_{k+1}) -\\sum_{i\\neq i_k} g_i(x^i_{k+1}) - \\bar{\\varphi}^{i_k}(s_{k+1};x_k)\n        &=\n        f(x_{k+1}) - l^{i_k}_f(s_{k+1};x_k) + h(F(x_{k+1}))\\\\\n        \\MoveEqLeft [-1] - h(l^{i_k}_F(x_{k+1};x_k)).\n    \\end{split}\n    \\end{align*}\n    Using  \\eqref{smooth_object} and \\eqref{to_use_next} in the previous relation, it follows that\n    \\[\n        \\varphi(x_{k+1}) \\leq \\bar{\\varphi}^{i_k}(s_{k+1};x_k) + \\sum_{i\\neq i_k} g_i(x^i_{k+1}) + \\frac{\\beta_{k+1}}{2} \\|\\Delta x_{k+1}\\|^2.\n    \\]\n    Finally, using \\eqref{initial_decrease}, we get\n    \\begin{align*}\n        \\varphi(x_{k+1}) \\leq  \\varphi(x_{k}) - \\frac{\\beta_{k+1}}{2} \\|\\Delta x_{k+1}\\|^2. \n    \\end{align*}\n    This proves our statement. \\hfill\\qed\n\\end{proof}\n\n\\begin{lemma} \\label{bound_subgradient}\nLet \\((x_k)_{k \\geq 0}\\) be generated by Monotone LiBCoD \\ref{alg:libcod}. If \\cref{ass:1} holds on a compact set to which the iterates belong, then, for all \\(k \\geq 0\\), we have\n\\begin{equation}\\label{bounde_gradient1}\n    \\begin{split}\n     \\dist\\left(0, \\partial_{i_k} \\varphi (x_{k+1})  \\right)\n     & \\leq \\left( L^f_{i_k} + L^F_{i_k} \\sqrt{2L_h} \\sqrt{ \\varphi(x_0) - \\varphi^*} + \\beta_{k+1}\\right)  \\|\\Delta x_{k+1}\\|\n     \\\\\n     \\MoveEqLeft[-1] + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2,\n    \\end{split}\n\\end{equation}\nwhere $\\|\\nabla_{i_k}F(x_k)\\| \\leq M^{F}_{i_k}$.% \\textcolor{red}{and $\\partial \\varphi(x)$ denotes the Mordukhovich (aka.~limiting) subdifferential of $\\varphi$ at $x$.} \n\\end{lemma}\n\n\\begin{proof}\n    By the optimality condition corresponding to the $s_{k+1}$ determining subproblem on Line 6 of Monotone LiBCoD \\ref{alg:libcod}, we have\n    \\[\n        -\\nabla_{i_k} f(x_{k})-\\nabla_{i_k} F(x_{k})^T \\nabla h\\left( l^{i_k}_F(s_{k+1};x_k)\\right)-\\beta_{k+1}(x_{k+1}-x_{k})\\in \\partial g_{i_k}(s_{k+1}).\n    \\]\n    It then follows, by exploiting the definition of $\\varphi$ and the properties of the derivative, that\n    \\begin{align*}\n        &\\dist\\left(0,\\partial_{i_k} \\varphi (x_{k+1})\\right)\\\\\n        &=  \\dist\\left(-\\nabla_{i_k} f(x_{k+1}) - \\nabla_{i_k} F(x_{k+1})^T \\nabla h \\left( F(x_{k+1})\\right) , \\partial g_{i_k}(s_{k+1})\\right)\\\\\n        & \\leq  \\Bigg \\|\\nabla_{i_k} f(x_{k+1}) - \\nabla_{i_k} f(x_k) + \\Big(\\nabla_{i_k} F(x_{k+1}) - \\nabla_{i_k} F(x_{k})\\Big)^T\\nabla h\\left(  F(x_{k+1})\\right)  \\\\\n        & -\\beta_{k+1}\\Delta x_{k+1} +\\nabla_{i_k} F(x_{k})^T\\left( \\nabla h \\left( F(x_{k+1}) \\right)-\\nabla h\\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right) \\Bigg\\|.\n    \\end{align*}\n    Using the triangle inequality, we further get\n    \\begin{equation}\n        \\label{eq:lem2-bndsub}\n        \\begin{split}\n            &\n            \\dist\\left(0, \\partial_{i_k} \\varphi (x_{k+1}) \\right) \\leq  \\|\\nabla_{i_k} f(x_{k+1})-\\nabla_{i_k} f(x_k)\\|\n            \\\\\n            \\MoveEqLeft[-1] + \\left\\|\\nabla h\\left( F(x_{k+1})\\right)\\right\\|\\left\\|\\nabla_{i_k} F(x_{k+1}) - \\nabla_{i_k} F(x_{k})\\right\\|+\\beta_{k+1}\\|\\Delta x_{k+1}\\|\n            \\\\\n            \\MoveEqLeft[-1]\n            +\\|{\\nabla_{i_k} F(x_{k})}\\|\\left\\| \\nabla h \\left( F(x_{k+1}) \\right)-\\nabla h\\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right\\|\n            \\\\\n            &\n            {\\overset{{\\text{Ass. } \\ref{ass:1}}}{\\leq}} \\left(L^f_{i_k}+\\left\\|\\nabla h\\left( F(x_{k+1})\\right)\\right\\| L^F_{i_k} +\\beta_{k+1}\\right)\\|\\Delta x_{k+1}\\|\n            + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2\n            \\\\\n            &\\leq \\left( L^f_{i_k} + L^F_{i_k} \\sqrt{2L_h \\left(h(F(x_k)) \\!-\\! h^*\\right)} + \\beta_{k+1}\\right) \\|\\Delta x_{k+1}\\|  + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2\n            \\\\\n            &\n            \\leq  \\left( L^f_{i_k} + L^F_{i_k} \\sqrt{2L_h \\left(\\varphi(x_k) - \\varphi^*\\right)} + \\beta_{k+1}\\right)  \\|\\Delta x_{k+1}\\| + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2\n            \\\\\n            &\n            \\leq  \\left( L^f_{i_k} + L^F_{i_k} \\sqrt{2L_h \\left(\\varphi(x_0) - \\varphi^*\\right)} + \\beta_{k+1}\\right)  \\|\\Delta x_{k+1}\\| + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2\n             \\\\\n            &\n            \\leq  \\left( L^f + L^F \\sqrt{2L_h \\left(\\varphi(x_0) - \\varphi^*\\right)} + \\bar{\\beta}\\right)  \\|\\Delta x_{k+1}\\| + \\frac{L_h M^{F}L^{F}}{2} \\|\\Delta x_{k+1}\\|^2,\n        \\end{split}\n    \\end{equation}\n        where the third inequality follows from the following relation for $u= F(x_{k}) \\in \\mathbb{R}^m$, and $v = u - \\frac{1}{L_h} \\nabla h(u)$\n    \\[\n        h^* - h(u) \\leq h(v) - h(u)\n        \\leq \\langle \\nabla h(u), v - u \\rangle + \\frac{L_ h}{2} \\norm{v - u}^2 = -\\frac{1}{2L_h} \\norm{\\nabla h(u)}^2,\n    \\]\n    and $M^F:=\\max_{i\\in\\{1, \\ldots, n_{\\text{block}}\\}} M^F_{i}$, and the fourth inequality follows from the fact that \\( \\varphi(x) = f(x) + h(F(x)) + g(x) \\) and \\( \\varphi^* = h^* + f^* + g^* \\), where \\( h^*, g^*, \\) and \\( f^* \\) are the lower bounds of \\( h(x) \\), \\( g(x) \\), and \\( f(x) \\), respectively, as defined in \\cref{ass:1}.\n    This proves our claim. \\hfill\\qed\n\\end{proof}\n\n\\begin{proposition}[Asymptotic convergence]\n    \\label{propos:asymtotic:monotone}\n    Let $(x_k)_{k\\geq 0}$ be generated by Monotone LiBCoD \\ref{alg:libcod}. If \\cref{ass:1} holds on a compact set to which the iterates belong, then we have \n    \\[\n        \\lim_{k\\to \\infty} \\E \\left[\\dist(0, \\partial\\varphi(x_{k})) \\right] = 0,\n    \\]\n\\end{proposition}\n\n\\begin{proof}\n     Let $k\\geq0$. From Monotone LiBCoD \\ref{alg:libcod}, we have\n    \\begin{align*}\n        \\frac{\\beta_{\\min}}{2}\\|x_{k+1} - x_k\\|^{2}\\leq \\frac{\\beta_{k+1}}{2}\\|x_{k+1} - x_k\\|^{2}   \\leq  \\varphi(x_k) - \\varphi(x_{k+1}).\n    \\end{align*}\n    Taking the expectation, we obtain\n    \\begin{align*}\n        \\frac{\\beta_{\\min}}{2} \\E\\left[\\|x_{k+1} - x_k\\|^{2}\\right]  \\leq  \\E\\left[\\varphi(x_k)\\right] - \\E[\\varphi(x_{k+1})].\n    \\end{align*}\n    Summing the above inequality from $k=0$ to $K\\geq 0$, we get\n    \\begin{align*}\n        \\frac{\\beta_{\\min}}{2 }\\sum_{k=0}^{K}\\E\\left[\\|x_{k+1} - x_k\\|^{2}\\right]  &\\leq  \\sum_{k=0}^{K}\\left(\\E\\left[\\varphi(x_k)\\right] - \\E[\\varphi(x_{k+1})]\\right) \\\\\n        & = \\E\\left[\\varphi(x_0)\\right] - \\E[\\varphi(x_{K+1})]\\\\\n        & \\leq \\varphi(x_0) - \\varphi^*.\n        \\end{align*}\n    Taking the limit $K \\to \\infty$, we get\n    \\begin{align*}\n        \\frac{\\beta_{\\min}}{2 }\\sum_{k=0}^{\\infty}\\E\\left[\\|x_{k+1} - x_k\\|^{2}\\right]  \\leq \\varphi(x_0) - \\varphi^* < \\infty.\n    \\end{align*}\n    Since $\\beta_{\\min} > 0$, it follows that\n    \\begin{equation}\\label{zero_limit}\n        \\lim_{k\\to\\infty}{\\E\\left[\\|x_{k+1} - x_{k}\\|^2\\right]}=0 \\quad \\text{ and } \\quad \\lim_{k\\to\\infty}{\\E\\left[\\|x_{k+1} - x_{k}\\|\\right]}=0, \n    \\end{equation}\n    where the limit on the right-hand side of \\eqref{zero_limit} follows from Jensen's inequality. For simplicity, we denote $\\E_k[\\cdot] =  \\E[\\cdot|x_k]$.  On the other, using \\ref{bound_subgradient}, we deduce\n      \\allowdisplaybreaks\n    \\begin{equation}\n        \\label{subgrad_bound_asymp}\n        \\begin{split}\n            &\\dist(0, \\partial\\varphi(x_{k+1}))\n            \\leq\n            \\sum_{i=1}^{n_{\\text{block}}}{\\dist \\left(0, \\partial_i\\varphi(x_{k+1})\\right)}\n            \\\\\n            &\n            \\leq\n            \\frac{1}{p_{\\min}}\\sum_{i=1}^{n_{\\text{block}}}{p_i \\; \\dist \\left(0, \\partial_{i_k}\\varphi(x_{k+1})\\right)}\n            \\\\\n            &\n            = \\frac1p_{\\min}  \\E_{k}\\left[\\dist \\left(0, \\partial_{i_k}\\varphi(x_{k+1})\\right)\\right]\n            \\\\\n            &\n            \\overset{\\eqref{bounde_gradient1}}{\\leq }\n            \\frac1p_{\\min}\\E_{k}\\left[  \\left( L^f_{i_k} + L^F_{i_k} \\sqrt{2L_h} \\sqrt{ \\left(\\varphi(x_0) - \\varphi^*\\right)} + \\beta_{k+1}\\right)  \\|\\Delta x_{k+1}\\| \\right.\\\\\n            \\MoveEqLeft[-1] \\left. + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2 \\right]\n            \\\\\n            &\n            \\overset{\\eqref{bar_gamma}}{\\leq}\n            \\frac{1}{p_{\\min}} \\left( L^f + L^F \\sqrt{2L_h} \\sqrt{ \\left(\\varphi(x_0) - \\varphi^*\\right)} + \\bar{\\beta} \\right)  \\E_{k}\\left[\\|\\Delta x_{k+1}\\|\\right]\\\\\n             \\MoveEqLeft[-1] + \\frac{L_h M^F L^{F}}{2p_{\\min}} \\E_{k}\\left[\\|\\Delta x_{k+1}\\|^2 \\right].\n        \\end{split}\n    \\end{equation}\n  Taking expectation in \\eqref{subgrad_bound_asymp}, we obtain\n    \\begin{align*}\n            \\E\\left[\\dist(0, \\partial\\varphi(x_{k+1}))\\right] \n            %&\\leq             \\frac{1}{p_{\\min}} \\left( L^f + L^F \\sqrt{2L_h} \\sqrt{ \\left(\\varphi(x_0) - \\varphi^*\\right)} + \\bar{\\beta} \\right)  \\E\\left[\\E_{k}\\left[\\|\\Delta x_{k+1}\\|\\right]\\right] +  \\frac{L_h M^F L^{F}}{2p_{\\min}} \\E\\left[\\E_{k}\\left[\\|\\Delta x_{k+1}\\|^2 \\right]\\right] \\\\\n            &\\leq\n            \\frac{1}{p_{\\min}} \\left( L^f \\!+\\! L^F \\sqrt{2L_h} \\sqrt{ \\left(\\varphi(x_0) \\!-\\! \\varphi^*\\right)} + \\bar{\\beta} \\right)  \\E\\left[\\|\\Delta x_{k+1}\\|\\right]\\\\\n             \\MoveEqLeft[-1] +  \\frac{L_h M^F L^{F}}{2p_{\\min}} \\E\\left[\\|\\Delta x_{k+1}\\|^2 \\right]\n    \\end{align*}\n    Passing to the limit and using \\eqref{zero_limit}, we get\n      $  \\lim_{k\\to \\infty} \\E\\left[\\dist(0, \\partial\\varphi(x_{k})) \\right] = 0.$ \\hfill\\qed\n\\end{proof}\n\n\\begin{proof}\n    From \\cref{propos:asymtotic:monotone}, we know that  $\\E\\left[\\dist(0, \\partial\\varphi(x_{k})) \\right] \\to 0$ as $k\\to \\infty$. Let $\\epsilon>0$ and let $N\\geq 0$ be the first integer such that\n    \\begin{equation}\n    \\E\\left[\\dist(0, \\partial\\varphi(x_{N+1})) \\right] \\leq \\epsilon.\n    \\end{equation}\n    Using  \\eqref{bounde_gradient1} from  \\cref{bound_subgradient}, it follows that at each iteration, $k \\in [0:N]$, we encounter one of the following two cases:\\\\\n    \\noindent \\textit{Case 1: } If the following holds\n    \\[\n        \\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)\\|\\Delta x_{k+1}\\|\\geq \\frac{L_h M^F L^F}{2}\\|\\Delta x_{k+1}\\|^2,\n    \\]\n    then we get\n    \\begin{align} \\label{case1}\n        & \\dist^2\\left(0, \\partial_{i_k} \\varphi (x_{k+1}) \\right) \\leq 2 \\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2\\|\\Delta x_{k+1}\\|^2 \\nonumber\\\\\n        &\\overset{\\eqref{decrease}}{\\leq} \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{\\beta_{k+1}}\\left(\\varphi(x_{k}) - \\varphi(x_{k+1})\\right).\n    \\end{align}\n    \\noindent \\textit{Case 2: } Otherwise,  the following is valid \n    \\[\n        \\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)\\|\\Delta x_{k+1}\\|< \\frac{L_h M^F L^F}{2}\\|\\Delta x_{k+1}\\|^2,\n    \\]\n    which yields\n    \\begin{equation}\\label{case2}\n        \\dist\\left(0, \\partial_{i_k} \\varphi (x_{k+1}) \\right) \\leq L_h M^F L^F \\|\\Delta x_{k+1}\\|^2\\overset{\\eqref{decrease}}{\\leq} \\frac{2 L_h M^F L^F}{\\beta_{k+1}}\\left(\\varphi(x_{k}) - \\varphi(x_{k+1})\\right).\n    \\end{equation}\n    Define $\\mathcal{I}_1$ as the set of iterations $k \\in [0:N-1]$ at which   \\textit{Case 1} holds, and  $\\mathcal{I}_2$ as the set of iterations $k\\in [0:N-1]$ at which   \\textit{Case 2} holds. Clearly: $N=|\\mathcal{I}_1|+|\\mathcal{I}_2|+1$. We first derive an upper bound for $|\\mathcal{I}_1|$.  Let $k \\in \\mathcal{I}_1$, we have\n    \\allowdisplaybreaks\n    \\begin{equation}\n        \\label{criticality1_squared}\n        \\begin{split}\n        &\\dist^2(0, \\partial\\varphi(x_{k+1}))\n        =\n        \\sum_{i=1}^{n_{\\text{block}}}{\\dist^2 \\left(0, \\partial_i\\varphi(x_{k+1})\\right)}\n        \\\\\n        &\n        \\leq\n        \\frac{1}{p_{\\min}}\\sum_{i=1}^{n_{\\text{block}}}{p_i \\; \\dist^2 \\left(0, \\partial_i\\varphi(x_{k+1})\\right)}\n        \\\\\n        &\n        = \\frac1p_{\\min}  \\E_{k}\\left[\\dist^2 \\left(0, \\partial_{i_k}\\varphi(x_{k+1})\\right)\\right]\n        \\\\\n        &\n        \\overset{\\eqref{case1} }{\\leq }\n        \\frac1p_{\\min}  \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{\\beta_{k+1}}      \\E_{k} \\left[\\varphi(x_{k}) - \\varphi(x_{k+1}) \\right]\n        \\\\\n        &\n        \\leq\n        \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}      \n        \\left(\\varphi(x_k) - \\E_{k} [\\varphi(x_{k+1})]\\right).\n        \\end{split}\n    \\end{equation}\n    Taking the expectation in the inequality \\eqref{criticality1_squared} and summing it over $\\mathcal{I}_1$ yields\n    \\begin{align*}\n        |\\mathcal{I}_1|\\epsilon^2  & < \\sum_{k\\in\\mathcal{I}_1} \\left(\\E\\left[\\dist(0, \\partial\\varphi(x_{k+1}))\\right]\\right)^2\\leq \\sum_{k\\in\\mathcal{I}_1} \\E\\left[\\dist^2(0, \\partial\\varphi(x_{k+1}))\\right]  \\\\\n        & \\leq  \\sum_{k\\in\\mathcal{I}_1}  \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}      \n        \\E\\left[\\varphi(x_k) - \\E_{k} [\\varphi(x_{k+1})]\\right] \\\\\n        & = \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}   \\sum_{k\\in\\mathcal{I}_1}      \n        \\left(\\E\\left[\\varphi(x_k)\\right] - \\E [\\varphi(x_{k+1})]\\right)\\\\\n        & \\overset{\\eqref{decrease}}{\\leq}  \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}   \\sum_{k=0}^{N-1}      \n        \\left(\\E\\left[\\varphi(x_k)\\right] - \\E [\\varphi(x_{k+1})]\\right)  \\\\\n        & = \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}  \\left(\\E[\\varphi(x_0)] - \\E[\\varphi(x_N)]\\right)  \\\\ \n        & \\leq \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}  \\left(\\varphi(x_0) - \\varphi^*\\right) .\n    \\end{align*}\n    Thus, we have: $|\\mathcal{I}_1|< \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min} \\epsilon^2}  \\left(\\varphi(x_0) - \\varphi^*\\right)$. Similarly, we derive an upper bound for $|\\mathcal{I}_2|$. Let $k\\in \\mathcal{I}_2$, we have\n    \\allowdisplaybreaks\n    \\begin{equation}\n        \\label{criticality1}\n        \\begin{split}\n        \\dist(0, \\partial\\varphi(x_{k+1}))\n        &\n        \\leq\n        \\sum_{i=1}^{n_{\\text{block}}}{\\dist(0, \\partial_i\\varphi(x_{k+1}))}\n        \\\\\n        &\n        \\leq\n        \\frac{1}{p_{\\min}}\\sum_{i=1}^{n_{\\text{block}}}{p_i \\; \\dist(0, \\partial\\varphi_i(x_{k+1}))}\n        \\\\\n        &\n        = \\frac1p_{\\min}  \\E_{k}\\left[\\dist(0, \\partial\\varphi_{i_k}(x_{k+1}))\\right]\n        \\\\\n        &\n        \\overset{\\eqref{case2} }{\\leq }\n        \\frac1p_{\\min}  \\frac{2 L_h M^F L^F}{\\beta_{k+1}}      \\E_{k} \\left[\\varphi(x_{k}) - \\varphi(x_{k+1}) \\right]\n        \\\\\n        &\n        \\leq\n         \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}}      \n        \\left(\\varphi(x_k) - \\E_{k} [\\varphi(x_{k+1})]\\right).\n        \\end{split}\n    \\end{equation}\n    Taking the expectation in the inequality \\eqref{criticality1} and summing it over $\\mathcal{I}_2$ yields\n    \\begin{align*}\n        &  |\\mathcal{I}_2|\\epsilon  < \\sum_{k\\in\\mathcal{I}_2} \\E\\left[\\dist(0, \\partial\\varphi(x_{k+1}))\\right]  \\leq  \\sum_{k\\in\\mathcal{I}_2} \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}} \\E\\left[\\varphi(x_k) - \\E_{k} [\\varphi(x_{k+1})]\\right]\\\\\n        &  = \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}} \\sum_{k\\in\\mathcal{I}_2}  \\left(\\E\\left[\\varphi(x_k)\\right] - \\E [\\varphi(x_{k+1})]\\right)\\\\\n        & \\overset{\\eqref{decrease}}{\\leq}  \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}} \\sum_{k=0}^{N-1}  \\left(\\E\\left[\\varphi(x_k)\\right] - \\E [\\varphi(x_{k+1})]\\right)\\\\\n        & \\leq  \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}}   \\left(\\E\\left[\\varphi(x_0)\\right] - \\E [\\varphi(x_{N+1})]\\right)\\leq \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}}   \\left(\\varphi(x_0) - \\varphi^*\\right).\n    \\end{align*}\n    Therefore, we obtain: $|\\mathcal{I}_2|< \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min} \\epsilon}   \\left(\\varphi(x_0) - \\varphi^*\\right) $.  Consequently, we obtain\n    \\[\n        N \\leq  \\left(\\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min} \\epsilon^2}  + \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min} \\epsilon} \\right)\\left(\\varphi(x_0) - \\varphi^*\\right).\n        \\]\n    This proves our claim. \\hfill\\qed\n\\end{proof}","units":[{"start":26108,"end":26512,"start_line":316,"end_line":323,"raw_tex":"\\begin{lemma}\n    \\label{lemma1}\n    Let $(x_k)_{k\\geq 0}$ be generated by Monotone LiBCoD \\ref{alg:libcod}. If \\cref{ass:1} holds on a compact set $\\mathcal{C}$ containing the iterates, and $\\beta_{k+1}$ satisfies \\eqref{eq:beta_cond}, then\n    \\begin{align}\n        \\label{decrease}\n         \\varphi(x_{k+1}) \\leq  \\varphi(x_{k}) - \\frac{\\beta_{k+1}}{2} \\|x_{k+1} - x_k\\|^2.\n    \\end{align}\n\\end{lemma}","sha256_utf8":"c6e928fb95bd436e21a080ebb687429269362e0beb5c5a9473618349ba0ed2c3"},{"start":26518,"end":33462,"start_line":325,"end_line":444,"raw_tex":"\\begin{proof}\n    This proof follows a reasoning similar to that in \\cite{marumo2024accelerated} [Lemma 3.1], adapted to the coordinate descent framework with an additional smooth function $f$.  \n    %\\seb{it would be nice to repeat there some of the definitions (for better readability)}\n    %\\seb{it seems the proof (for the second term) follows exactly \\cite{marumo2024accelerated}? maybe mention this here, and explain the differences}\n    Note that the subproblem's objective function\n    \\[\n        q(s) := \\bar{\\varphi}^{i_k}(s;x_k)+\\frac{\\beta_{k+1}}{2}\\|s-x^i_k\\|^2,\n    \\]\n    By the strong convexity of this objective and $s_{k+1}$ being a minimiser, i.e., $0 \\in \\partial q(s_{k+1})$,\n    \\[\n        q(x_k^i)\n        \\ge\n        q(s_{k+1}) + \\frac{\\beta_{k+1}}{2}\\|s_{k+1}-x^i_k\\|^2.\n    \\]\n    Hence, expanding $q$,\n    % is strongly convex with a strong convexity constant $\\beta_{k+1}$, then,\n    % \\[\n    %    \\bar{\\varphi}^{i_k}(x_k;x_k)+\\frac{\\beta_{k+1}}{2}\\|x^i_k-x^i_k\\|^2 \\geq \\bar{\\varphi}^{i_k}(s_{k+1};x_k)+\\frac{\\beta_{k+1}}{2}\\|s_{k+1}-x^i_k\\|^2 + \\frac{\\beta_{k+1}}{2}\\norm{x_{k+1} - x_k}^2 \n    % \\]\n    % combining this with the optimality of $s_{k+1}$,  \n    the fact that $\\bar{\\varphi}^{i_k}(x^{i_k}_k;x_k)=\\varphi(x_{k}) - \\sum_{i\\neq i_k} g_i(x^i_{k})$ and making use of the fact that $x^i_{k+1} = x^i_k$ for $i\\neq i_k$, we get\n    \\begin{equation} \\label{initial_decrease}\n        \\bar{\\varphi}^{i_k}(s_{k+1};x_k)\\leq \\varphi(x_{k}) - \\sum_{i\\neq i_k} g_i(x^i_{k+1}) -\\beta_{k+1}\\|x_{k+1}-x_k\\|^2.\n    \\end{equation}\n%    \\seb{is it $\\frac{\\beta_{k+1}}{2}$? \\textcolor{red}{just $\\beta_{k+1}$ since we use the strong convexity of $\\bar{\\varphi}^{i_k}(\\cdot;x_k)$}}%\n%    \\seb{I still don't see this. Strong convexity gives $\\varphi(x) \\geq \\varphi(s_{k+1}) + \\frac{\\beta}{2} \\| x - s_{k+1}\\|^2$ ? \\textcolor{red}{ It is \\(\\varphi(x) + \\frac{\\beta}{2}\\|x - x_k\\|^2\\) that is strongly convex, whereas \\(\\varphi(x)\\) is not. Thus, we have:\n% \\[\n% \\left(\\varphi(s) + \\frac{\\beta}{2} \\|s - x_{k}\\|^2\\right) \\geq \\left(\\varphi(s_{k+1}) + \\frac{\\beta}{2} \\|s_{k+1} - x_{k}\\|^2\\right) + \\frac{\\beta}{2} \\|s_{k+1} - x_{k}\\|^2.\n% \\]\n% %%SEB:\n% ok, tricky; \n% } }%\n    Furthermore, since \\( f \\) has a block Lipschitz gradient with a Lipschitz constant \\( L_{i_k}^f \\), it follows that \\cite{nesterov2018lectures}\n        \\begin{equation}\n        \\label{smooth_object}\n        f(x_{k+1}) - l_f^{i_k}(s_{k+1};x_k) \\leq \\frac{L^f_{i_k}}{2}\\|x_{k+1}-x_k\\|^2.\n    \\end{equation}\n    %\\seb{can you add a reference to the equation that you are using? (also it would help to remind the readers of the definition of $l_f$)\n    %why is there no term with $\\nabla f$ ? (a similar equation is also used in (14) below). $s_{k+1}$ is not a minimizer of $f$}\n     Moreover, since \\( h \\) has a Lipschitz gradient with a Lipschitz constant \\( L_h \\), we have\n    \\begin{equation}\n    \\label{decrease_feasib}\n    \\begin{split}\n         h(F(x_{k+1}))& -  h(l^{i_k}_F(s_{k+1};x_k)) \n        \\\\\n        &\\leq   \\langle \\nabla h \\left( l^{i_k}_F(s_{k+1};x_k)\\right), F(x_{k+1})-l^{i_k}_F(s_{k+1};x_k) \\rangle \\\\\n        \\MoveEqLeft[-1] + \\frac{L_h}{2}\\|F(x_{k+1})-l^{i_k}_F(s_{k+1};x_k)\\|^2 \n        \\\\\n        & \\leq \\left\\|\\nabla h \\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right\\| \\|F(x_{k+1})-l^{i_k}_F(s_{k+1};x_k) \\|\\\\\n        \\MoveEqLeft[-1] + \\frac{L_h}{2}\\|F(x_{k+1})-l^{i_k}_F(s_{k+1};x_k)\\|^2 \n        \\\\\n        &\\overset{\\text{Ass. } \\ref{ass:1}}{\\leq} \\left\\|\\nabla h \\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right\\|  \\frac{L^F_{i_k}}{2}\\|\\Delta x_{k+1}\\|^2 + \\frac{L_h}{2} \\left(\\frac{L^F_{i_k}}{2}\\|\\Delta x_{k+1}\\|^2\\right)^2. \n    \\end{split}\n    \\end{equation}\n    Using the Young's inequality $2\\sqrt{a b} - a \\leq b$ for any $a, b \\geq 0$, and the following relation for $u =l^{i_k}_F(s_{k+1};x_k) \\in \\mathbb{R}^m$, with $v = u - \\frac{1}{L_h} \\nabla h(u)$\n    \\[\n        h^* - h(u) \\leq h(v) - h(u)\n        \\leq \\langle \\nabla h(u), v - u \\rangle + \\frac{L_h}{2} \\norm{v - u}^2 = -\\frac{1}{2L_h} \\norm{\\nabla h(u)}^2,\n    \\]\n    we bound\n    \\begin{equation}\n        \\label{bound_grad_norm_squared}\n        \\begin{split}\n            &\n            \\left\\|\\nabla h \\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right\\| - \\frac{ \\beta_{k+1} - L^f_{i_k}}{2L^F_{i_k}} \\leq  \\frac{L^F_{i_k}}{ 2(\\beta_{k+1} - L^f_{i_k})}    \\left\\|\\nabla h \\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right\\|^2\n            \\\\\n            &\n            \\leq\n            \\frac{L^F_{i_k}L_h}{ \\beta_{k+1} - L^f_{i_k}} \\left( h(l^{i_k}_F(s_{k+1};x_k)) - h^*  \\right)\n            %\\\\\n            %&\n            =\n            \\frac{L^F_{i_k}L_h}{ \\beta_{k+1} - L^f_{i_k}}\n            \\\\\n            \\MoveEqLeft[-4]\n            \\left( \\bar{\\varphi}^{i_k}(s_{k+1};x_k) - h^* - g_{i_k}(s_{k+1})- f(x_{k+1}) + f(x_{k+1}) - l_f^{i_k}(s_{k+1};x_k)  \\right)\n            \\\\\n            &\n            \\overset{\\eqref{initial_decrease}, \\eqref{smooth_object}}{\\leq}\n            \\frac{L^F_{i_k} L_h}{\\beta_{k+1} - L^f_{i_k}} \\left( \\varphi(x_k) - \\varphi^* - \\frac{2\\beta_{k+1} - L^F_{i_k}}{2}\\|\\Delta x_{k+1}\\|^2 \\right)\n            \\\\\n            &\n            \\leq\n            \\frac{L^F_{i_k} L_h}{\\beta_{k+1} - L^f_{i_k}} \\left( \\varphi(x_k) - \\varphi^*\\right) - L^F_{i_k} L_h\\|\\Delta x_{k+1}\\|^2\n            \\\\\n            &\n            \\overset{\\eqref{eq:beta_cond}}{\\leq}\\frac{\\beta_{k+1}-L^f_{i_k}}{2L^F_{i_k}} - L^F_{i_k} L_h\\|\\Delta x_{k+1}\\|^2,\n        \\end{split}\n    \\end{equation}\n    where third inequality follows from the definition of $\\bar{\\varphi}^{i_k}$. Moreover, Using \\eqref{bound_grad_norm_squared} in \\eqref{decrease_feasib}, we deduce that\n    \\begin{equation}\n        \\label{to_use_next}\n        \\begin{split}\n                    h\\left(F(x_{k+1})\\right) &- h\\left(l^{i_k}_F(x_{k+1};x_k) \\right)\\\\\n        & \\leq\n        \\frac{\\beta_{k+1}-L^f_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2 - \\left(\\frac{L_h (L^F_{i_k})^2}{2} - \\frac{L_h (L^{F}_{i_k})^2}{8} \\right) \\|\\Delta x_{k+1}\\|^4\n        \\\\\n        &=\n        \\frac{\\beta_{k+1}-L^f_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2-  \\frac{3 L_h (L^{F}_{i_k})^2}{8}  \\|\\Delta x_{k+1}\\|^4.\n        \\end{split}\n    \\end{equation}\n    Moreover, we have\n    \\begin{align*}\n    \\begin{split}\n        \\varphi(x_{k+1}) -\\sum_{i\\neq i_k} g_i(x^i_{k+1}) - \\bar{\\varphi}^{i_k}(s_{k+1};x_k)\n        &=\n        f(x_{k+1}) - l^{i_k}_f(s_{k+1};x_k) + h(F(x_{k+1}))\\\\\n        \\MoveEqLeft [-1] - h(l^{i_k}_F(x_{k+1};x_k)).\n    \\end{split}\n    \\end{align*}\n    Using  \\eqref{smooth_object} and \\eqref{to_use_next} in the previous relation, it follows that\n    \\[\n        \\varphi(x_{k+1}) \\leq \\bar{\\varphi}^{i_k}(s_{k+1};x_k) + \\sum_{i\\neq i_k} g_i(x^i_{k+1}) + \\frac{\\beta_{k+1}}{2} \\|\\Delta x_{k+1}\\|^2.\n    \\]\n    Finally, using \\eqref{initial_decrease}, we get\n    \\begin{align*}\n        \\varphi(x_{k+1}) \\leq  \\varphi(x_{k}) - \\frac{\\beta_{k+1}}{2} \\|\\Delta x_{k+1}\\|^2. \n    \\end{align*}\n    This proves our statement. \\hfill\\qed\n\\end{proof}","sha256_utf8":"24280f3ca4829d7214db8491a9d38f4783cfedfea7bb9453662f5e0d2f8223e1"},{"start":34246,"end":35017,"start_line":459,"end_line":470,"raw_tex":"\\begin{lemma} \\label{bound_subgradient}\nLet \\((x_k)_{k \\geq 0}\\) be generated by Monotone LiBCoD \\ref{alg:libcod}. If \\cref{ass:1} holds on a compact set to which the iterates belong, then, for all \\(k \\geq 0\\), we have\n\\begin{equation}\\label{bounde_gradient1}\n    \\begin{split}\n     \\dist\\left(0, \\partial_{i_k} \\varphi (x_{k+1})  \\right)\n     & \\leq \\left( L^f_{i_k} + L^F_{i_k} \\sqrt{2L_h} \\sqrt{ \\varphi(x_0) - \\varphi^*} + \\beta_{k+1}\\right)  \\|\\Delta x_{k+1}\\|\n     \\\\\n     \\MoveEqLeft[-1] + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2,\n    \\end{split}\n\\end{equation}\nwhere $\\|\\nabla_{i_k}F(x_k)\\| \\leq M^{F}_{i_k}$.% \\textcolor{red}{and $\\partial \\varphi(x)$ denotes the Mordukhovich (aka.~limiting) subdifferential of $\\varphi$ at $x$.} \n\\end{lemma}","sha256_utf8":"e7def4eb5be23f646b3b2a42306a91d3a856f92f07ed1cf0528c3149541ea805"},{"start":35086,"end":38605,"start_line":473,"end_line":520,"raw_tex":"\\begin{proof}\n    By the optimality condition corresponding to the $s_{k+1}$ determining subproblem on Line 6 of Monotone LiBCoD \\ref{alg:libcod}, we have\n    \\[\n        -\\nabla_{i_k} f(x_{k})-\\nabla_{i_k} F(x_{k})^T \\nabla h\\left( l^{i_k}_F(s_{k+1};x_k)\\right)-\\beta_{k+1}(x_{k+1}-x_{k})\\in \\partial g_{i_k}(s_{k+1}).\n    \\]\n    It then follows, by exploiting the definition of $\\varphi$ and the properties of the derivative, that\n    \\begin{align*}\n        &\\dist\\left(0,\\partial_{i_k} \\varphi (x_{k+1})\\right)\\\\\n        &=  \\dist\\left(-\\nabla_{i_k} f(x_{k+1}) - \\nabla_{i_k} F(x_{k+1})^T \\nabla h \\left( F(x_{k+1})\\right) , \\partial g_{i_k}(s_{k+1})\\right)\\\\\n        & \\leq  \\Bigg \\|\\nabla_{i_k} f(x_{k+1}) - \\nabla_{i_k} f(x_k) + \\Big(\\nabla_{i_k} F(x_{k+1}) - \\nabla_{i_k} F(x_{k})\\Big)^T\\nabla h\\left(  F(x_{k+1})\\right)  \\\\\n        & -\\beta_{k+1}\\Delta x_{k+1} +\\nabla_{i_k} F(x_{k})^T\\left( \\nabla h \\left( F(x_{k+1}) \\right)-\\nabla h\\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right) \\Bigg\\|.\n    \\end{align*}\n    Using the triangle inequality, we further get\n    \\begin{equation}\n        \\label{eq:lem2-bndsub}\n        \\begin{split}\n            &\n            \\dist\\left(0, \\partial_{i_k} \\varphi (x_{k+1}) \\right) \\leq  \\|\\nabla_{i_k} f(x_{k+1})-\\nabla_{i_k} f(x_k)\\|\n            \\\\\n            \\MoveEqLeft[-1] + \\left\\|\\nabla h\\left( F(x_{k+1})\\right)\\right\\|\\left\\|\\nabla_{i_k} F(x_{k+1}) - \\nabla_{i_k} F(x_{k})\\right\\|+\\beta_{k+1}\\|\\Delta x_{k+1}\\|\n            \\\\\n            \\MoveEqLeft[-1]\n            +\\|{\\nabla_{i_k} F(x_{k})}\\|\\left\\| \\nabla h \\left( F(x_{k+1}) \\right)-\\nabla h\\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right\\|\n            \\\\\n            &\n            {\\overset{{\\text{Ass. } \\ref{ass:1}}}{\\leq}} \\left(L^f_{i_k}+\\left\\|\\nabla h\\left( F(x_{k+1})\\right)\\right\\| L^F_{i_k} +\\beta_{k+1}\\right)\\|\\Delta x_{k+1}\\|\n            + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2\n            \\\\\n            &\\leq \\left( L^f_{i_k} + L^F_{i_k} \\sqrt{2L_h \\left(h(F(x_k)) \\!-\\! h^*\\right)} + \\beta_{k+1}\\right) \\|\\Delta x_{k+1}\\|  + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2\n            \\\\\n            &\n            \\leq  \\left( L^f_{i_k} + L^F_{i_k} \\sqrt{2L_h \\left(\\varphi(x_k) - \\varphi^*\\right)} + \\beta_{k+1}\\right)  \\|\\Delta x_{k+1}\\| + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2\n            \\\\\n            &\n            \\leq  \\left( L^f_{i_k} + L^F_{i_k} \\sqrt{2L_h \\left(\\varphi(x_0) - \\varphi^*\\right)} + \\beta_{k+1}\\right)  \\|\\Delta x_{k+1}\\| + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2\n             \\\\\n            &\n            \\leq  \\left( L^f + L^F \\sqrt{2L_h \\left(\\varphi(x_0) - \\varphi^*\\right)} + \\bar{\\beta}\\right)  \\|\\Delta x_{k+1}\\| + \\frac{L_h M^{F}L^{F}}{2} \\|\\Delta x_{k+1}\\|^2,\n        \\end{split}\n    \\end{equation}\n        where the third inequality follows from the following relation for $u= F(x_{k}) \\in \\mathbb{R}^m$, and $v = u - \\frac{1}{L_h} \\nabla h(u)$\n    \\[\n        h^* - h(u) \\leq h(v) - h(u)\n        \\leq \\langle \\nabla h(u), v - u \\rangle + \\frac{L_ h}{2} \\norm{v - u}^2 = -\\frac{1}{2L_h} \\norm{\\nabla h(u)}^2,\n    \\]\n    and $M^F:=\\max_{i\\in\\{1, \\ldots, n_{\\text{block}}\\}} M^F_{i}$, and the fourth inequality follows from the fact that \\( \\varphi(x) = f(x) + h(F(x)) + g(x) \\) and \\( \\varphi^* = h^* + f^* + g^* \\), where \\( h^*, g^*, \\) and \\( f^* \\) are the lower bounds of \\( h(x) \\), \\( g(x) \\), and \\( f(x) \\), respectively, as defined in \\cref{ass:1}.\n    This proves our claim. \\hfill\\qed\n\\end{proof}","sha256_utf8":"90e16efc9188ce1781c851c9f42b27e4a384161b4da213edd99d4e6bf4a55272"},{"start":39036,"end":39390,"start_line":526,"end_line":532,"raw_tex":"\\begin{proposition}[Asymptotic convergence]\n    \\label{propos:asymtotic:monotone}\n    Let $(x_k)_{k\\geq 0}$ be generated by Monotone LiBCoD \\ref{alg:libcod}. If \\cref{ass:1} holds on a compact set to which the iterates belong, then we have \n    \\[\n        \\lim_{k\\to \\infty} \\E \\left[\\dist(0, \\partial\\varphi(x_{k})) \\right] = 0,\n    \\]\n\\end{proposition}","sha256_utf8":"a47be2a2f571a188b3928b20439e7616f1fa9fe6f75aad29bcd8b762c76bcf89"},{"start":39391,"end":43132,"start_line":533,"end_line":593,"raw_tex":"\\begin{proof}\n     Let $k\\geq0$. From Monotone LiBCoD \\ref{alg:libcod}, we have\n    \\begin{align*}\n        \\frac{\\beta_{\\min}}{2}\\|x_{k+1} - x_k\\|^{2}\\leq \\frac{\\beta_{k+1}}{2}\\|x_{k+1} - x_k\\|^{2}   \\leq  \\varphi(x_k) - \\varphi(x_{k+1}).\n    \\end{align*}\n    Taking the expectation, we obtain\n    \\begin{align*}\n        \\frac{\\beta_{\\min}}{2} \\E\\left[\\|x_{k+1} - x_k\\|^{2}\\right]  \\leq  \\E\\left[\\varphi(x_k)\\right] - \\E[\\varphi(x_{k+1})].\n    \\end{align*}\n    Summing the above inequality from $k=0$ to $K\\geq 0$, we get\n    \\begin{align*}\n        \\frac{\\beta_{\\min}}{2 }\\sum_{k=0}^{K}\\E\\left[\\|x_{k+1} - x_k\\|^{2}\\right]  &\\leq  \\sum_{k=0}^{K}\\left(\\E\\left[\\varphi(x_k)\\right] - \\E[\\varphi(x_{k+1})]\\right) \\\\\n        & = \\E\\left[\\varphi(x_0)\\right] - \\E[\\varphi(x_{K+1})]\\\\\n        & \\leq \\varphi(x_0) - \\varphi^*.\n        \\end{align*}\n    Taking the limit $K \\to \\infty$, we get\n    \\begin{align*}\n        \\frac{\\beta_{\\min}}{2 }\\sum_{k=0}^{\\infty}\\E\\left[\\|x_{k+1} - x_k\\|^{2}\\right]  \\leq \\varphi(x_0) - \\varphi^* < \\infty.\n    \\end{align*}\n    Since $\\beta_{\\min} > 0$, it follows that\n    \\begin{equation}\\label{zero_limit}\n        \\lim_{k\\to\\infty}{\\E\\left[\\|x_{k+1} - x_{k}\\|^2\\right]}=0 \\quad \\text{ and } \\quad \\lim_{k\\to\\infty}{\\E\\left[\\|x_{k+1} - x_{k}\\|\\right]}=0, \n    \\end{equation}\n    where the limit on the right-hand side of \\eqref{zero_limit} follows from Jensen's inequality. For simplicity, we denote $\\E_k[\\cdot] =  \\E[\\cdot|x_k]$.  On the other, using \\ref{bound_subgradient}, we deduce\n      \\allowdisplaybreaks\n    \\begin{equation}\n        \\label{subgrad_bound_asymp}\n        \\begin{split}\n            &\\dist(0, \\partial\\varphi(x_{k+1}))\n            \\leq\n            \\sum_{i=1}^{n_{\\text{block}}}{\\dist \\left(0, \\partial_i\\varphi(x_{k+1})\\right)}\n            \\\\\n            &\n            \\leq\n            \\frac{1}{p_{\\min}}\\sum_{i=1}^{n_{\\text{block}}}{p_i \\; \\dist \\left(0, \\partial_{i_k}\\varphi(x_{k+1})\\right)}\n            \\\\\n            &\n            = \\frac1p_{\\min}  \\E_{k}\\left[\\dist \\left(0, \\partial_{i_k}\\varphi(x_{k+1})\\right)\\right]\n            \\\\\n            &\n            \\overset{\\eqref{bounde_gradient1}}{\\leq }\n            \\frac1p_{\\min}\\E_{k}\\left[  \\left( L^f_{i_k} + L^F_{i_k} \\sqrt{2L_h} \\sqrt{ \\left(\\varphi(x_0) - \\varphi^*\\right)} + \\beta_{k+1}\\right)  \\|\\Delta x_{k+1}\\| \\right.\\\\\n            \\MoveEqLeft[-1] \\left. + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2 \\right]\n            \\\\\n            &\n            \\overset{\\eqref{bar_gamma}}{\\leq}\n            \\frac{1}{p_{\\min}} \\left( L^f + L^F \\sqrt{2L_h} \\sqrt{ \\left(\\varphi(x_0) - \\varphi^*\\right)} + \\bar{\\beta} \\right)  \\E_{k}\\left[\\|\\Delta x_{k+1}\\|\\right]\\\\\n             \\MoveEqLeft[-1] + \\frac{L_h M^F L^{F}}{2p_{\\min}} \\E_{k}\\left[\\|\\Delta x_{k+1}\\|^2 \\right].\n        \\end{split}\n    \\end{equation}\n  Taking expectation in \\eqref{subgrad_bound_asymp}, we obtain\n    \\begin{align*}\n            \\E\\left[\\dist(0, \\partial\\varphi(x_{k+1}))\\right] \n            %&\\leq             \\frac{1}{p_{\\min}} \\left( L^f + L^F \\sqrt{2L_h} \\sqrt{ \\left(\\varphi(x_0) - \\varphi^*\\right)} + \\bar{\\beta} \\right)  \\E\\left[\\E_{k}\\left[\\|\\Delta x_{k+1}\\|\\right]\\right] +  \\frac{L_h M^F L^{F}}{2p_{\\min}} \\E\\left[\\E_{k}\\left[\\|\\Delta x_{k+1}\\|^2 \\right]\\right] \\\\\n            &\\leq\n            \\frac{1}{p_{\\min}} \\left( L^f \\!+\\! L^F \\sqrt{2L_h} \\sqrt{ \\left(\\varphi(x_0) \\!-\\! \\varphi^*\\right)} + \\bar{\\beta} \\right)  \\E\\left[\\|\\Delta x_{k+1}\\|\\right]\\\\\n             \\MoveEqLeft[-1] +  \\frac{L_h M^F L^{F}}{2p_{\\min}} \\E\\left[\\|\\Delta x_{k+1}\\|^2 \\right]\n    \\end{align*}\n    Passing to the limit and using \\eqref{zero_limit}, we get\n      $  \\lim_{k\\to \\infty} \\E\\left[\\dist(0, \\partial\\varphi(x_{k})) \\right] = 0.$ \\hfill\\qed\n\\end{proof}","sha256_utf8":"87e2718e4bdfe583e546dd2ee61d5324d0cb6c68537c01e114489f29d17be990"},{"start":44282,"end":51300,"start_line":612,"end_line":712,"raw_tex":"\\begin{proof}\n    From \\cref{propos:asymtotic:monotone}, we know that  $\\E\\left[\\dist(0, \\partial\\varphi(x_{k})) \\right] \\to 0$ as $k\\to \\infty$. Let $\\epsilon>0$ and let $N\\geq 0$ be the first integer such that\n    \\begin{equation}\n    \\E\\left[\\dist(0, \\partial\\varphi(x_{N+1})) \\right] \\leq \\epsilon.\n    \\end{equation}\n    Using  \\eqref{bounde_gradient1} from  \\cref{bound_subgradient}, it follows that at each iteration, $k \\in [0:N]$, we encounter one of the following two cases:\\\\\n    \\noindent \\textit{Case 1: } If the following holds\n    \\[\n        \\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)\\|\\Delta x_{k+1}\\|\\geq \\frac{L_h M^F L^F}{2}\\|\\Delta x_{k+1}\\|^2,\n    \\]\n    then we get\n    \\begin{align} \\label{case1}\n        & \\dist^2\\left(0, \\partial_{i_k} \\varphi (x_{k+1}) \\right) \\leq 2 \\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2\\|\\Delta x_{k+1}\\|^2 \\nonumber\\\\\n        &\\overset{\\eqref{decrease}}{\\leq} \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{\\beta_{k+1}}\\left(\\varphi(x_{k}) - \\varphi(x_{k+1})\\right).\n    \\end{align}\n    \\noindent \\textit{Case 2: } Otherwise,  the following is valid \n    \\[\n        \\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)\\|\\Delta x_{k+1}\\|< \\frac{L_h M^F L^F}{2}\\|\\Delta x_{k+1}\\|^2,\n    \\]\n    which yields\n    \\begin{equation}\\label{case2}\n        \\dist\\left(0, \\partial_{i_k} \\varphi (x_{k+1}) \\right) \\leq L_h M^F L^F \\|\\Delta x_{k+1}\\|^2\\overset{\\eqref{decrease}}{\\leq} \\frac{2 L_h M^F L^F}{\\beta_{k+1}}\\left(\\varphi(x_{k}) - \\varphi(x_{k+1})\\right).\n    \\end{equation}\n    Define $\\mathcal{I}_1$ as the set of iterations $k \\in [0:N-1]$ at which   \\textit{Case 1} holds, and  $\\mathcal{I}_2$ as the set of iterations $k\\in [0:N-1]$ at which   \\textit{Case 2} holds. Clearly: $N=|\\mathcal{I}_1|+|\\mathcal{I}_2|+1$. We first derive an upper bound for $|\\mathcal{I}_1|$.  Let $k \\in \\mathcal{I}_1$, we have\n    \\allowdisplaybreaks\n    \\begin{equation}\n        \\label{criticality1_squared}\n        \\begin{split}\n        &\\dist^2(0, \\partial\\varphi(x_{k+1}))\n        =\n        \\sum_{i=1}^{n_{\\text{block}}}{\\dist^2 \\left(0, \\partial_i\\varphi(x_{k+1})\\right)}\n        \\\\\n        &\n        \\leq\n        \\frac{1}{p_{\\min}}\\sum_{i=1}^{n_{\\text{block}}}{p_i \\; \\dist^2 \\left(0, \\partial_i\\varphi(x_{k+1})\\right)}\n        \\\\\n        &\n        = \\frac1p_{\\min}  \\E_{k}\\left[\\dist^2 \\left(0, \\partial_{i_k}\\varphi(x_{k+1})\\right)\\right]\n        \\\\\n        &\n        \\overset{\\eqref{case1} }{\\leq }\n        \\frac1p_{\\min}  \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{\\beta_{k+1}}      \\E_{k} \\left[\\varphi(x_{k}) - \\varphi(x_{k+1}) \\right]\n        \\\\\n        &\n        \\leq\n        \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}      \n        \\left(\\varphi(x_k) - \\E_{k} [\\varphi(x_{k+1})]\\right).\n        \\end{split}\n    \\end{equation}\n    Taking the expectation in the inequality \\eqref{criticality1_squared} and summing it over $\\mathcal{I}_1$ yields\n    \\begin{align*}\n        |\\mathcal{I}_1|\\epsilon^2  & < \\sum_{k\\in\\mathcal{I}_1} \\left(\\E\\left[\\dist(0, \\partial\\varphi(x_{k+1}))\\right]\\right)^2\\leq \\sum_{k\\in\\mathcal{I}_1} \\E\\left[\\dist^2(0, \\partial\\varphi(x_{k+1}))\\right]  \\\\\n        & \\leq  \\sum_{k\\in\\mathcal{I}_1}  \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}      \n        \\E\\left[\\varphi(x_k) - \\E_{k} [\\varphi(x_{k+1})]\\right] \\\\\n        & = \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}   \\sum_{k\\in\\mathcal{I}_1}      \n        \\left(\\E\\left[\\varphi(x_k)\\right] - \\E [\\varphi(x_{k+1})]\\right)\\\\\n        & \\overset{\\eqref{decrease}}{\\leq}  \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}   \\sum_{k=0}^{N-1}      \n        \\left(\\E\\left[\\varphi(x_k)\\right] - \\E [\\varphi(x_{k+1})]\\right)  \\\\\n        & = \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}  \\left(\\E[\\varphi(x_0)] - \\E[\\varphi(x_N)]\\right)  \\\\ \n        & \\leq \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}  \\left(\\varphi(x_0) - \\varphi^*\\right) .\n    \\end{align*}\n    Thus, we have: $|\\mathcal{I}_1|< \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min} \\epsilon^2}  \\left(\\varphi(x_0) - \\varphi^*\\right)$. Similarly, we derive an upper bound for $|\\mathcal{I}_2|$. Let $k\\in \\mathcal{I}_2$, we have\n    \\allowdisplaybreaks\n    \\begin{equation}\n        \\label{criticality1}\n        \\begin{split}\n        \\dist(0, \\partial\\varphi(x_{k+1}))\n        &\n        \\leq\n        \\sum_{i=1}^{n_{\\text{block}}}{\\dist(0, \\partial_i\\varphi(x_{k+1}))}\n        \\\\\n        &\n        \\leq\n        \\frac{1}{p_{\\min}}\\sum_{i=1}^{n_{\\text{block}}}{p_i \\; \\dist(0, \\partial\\varphi_i(x_{k+1}))}\n        \\\\\n        &\n        = \\frac1p_{\\min}  \\E_{k}\\left[\\dist(0, \\partial\\varphi_{i_k}(x_{k+1}))\\right]\n        \\\\\n        &\n        \\overset{\\eqref{case2} }{\\leq }\n        \\frac1p_{\\min}  \\frac{2 L_h M^F L^F}{\\beta_{k+1}}      \\E_{k} \\left[\\varphi(x_{k}) - \\varphi(x_{k+1}) \\right]\n        \\\\\n        &\n        \\leq\n         \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}}      \n        \\left(\\varphi(x_k) - \\E_{k} [\\varphi(x_{k+1})]\\right).\n        \\end{split}\n    \\end{equation}\n    Taking the expectation in the inequality \\eqref{criticality1} and summing it over $\\mathcal{I}_2$ yields\n    \\begin{align*}\n        &  |\\mathcal{I}_2|\\epsilon  < \\sum_{k\\in\\mathcal{I}_2} \\E\\left[\\dist(0, \\partial\\varphi(x_{k+1}))\\right]  \\leq  \\sum_{k\\in\\mathcal{I}_2} \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}} \\E\\left[\\varphi(x_k) - \\E_{k} [\\varphi(x_{k+1})]\\right]\\\\\n        &  = \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}} \\sum_{k\\in\\mathcal{I}_2}  \\left(\\E\\left[\\varphi(x_k)\\right] - \\E [\\varphi(x_{k+1})]\\right)\\\\\n        & \\overset{\\eqref{decrease}}{\\leq}  \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}} \\sum_{k=0}^{N-1}  \\left(\\E\\left[\\varphi(x_k)\\right] - \\E [\\varphi(x_{k+1})]\\right)\\\\\n        & \\leq  \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}}   \\left(\\E\\left[\\varphi(x_0)\\right] - \\E [\\varphi(x_{N+1})]\\right)\\leq \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}}   \\left(\\varphi(x_0) - \\varphi^*\\right).\n    \\end{align*}\n    Therefore, we obtain: $|\\mathcal{I}_2|< \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min} \\epsilon}   \\left(\\varphi(x_0) - \\varphi^*\\right) $.  Consequently, we obtain\n    \\[\n        N \\leq  \\left(\\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min} \\epsilon^2}  + \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min} \\epsilon} \\right)\\left(\\varphi(x_0) - \\varphi^*\\right).\n        \\]\n    This proves our claim. \\hfill\\qed\n\\end{proof}","sha256_utf8":"e5a258c20ba5951dbaa2e179193de77c3448701724995ab80e802cfbb9875b95"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"分块复合优化问题","source_units":[{"start":7414,"end":7581,"start_line":188,"end_line":191,"raw_tex":"\\begin{equation}  \n    \\label{eq:problem}  \n    \\min_{x \\in \\text{dom}\\,\\varphi} \\varphi(x): = f(x) + h(F(x)) + \\sum_{i=1}^{n_{\\text{block}}} g_i(x^i), \n\\end{equation}","sha256_utf8":"e18c3e77d70f4d0cd84c7015885c59ec4ede1e2bb8c46dc8273b9458db5d4443"},{"start":7584,"end":8203,"start_line":192,"end_line":193,"raw_tex":"where \\( F(x): = (F_1(x), \\cdots, F_m(x)) \\) and \\( f \\) are nonlinear differentiable functions defined on \\(\\text{dom}\\,\\varphi\\). The functions \\( g_i \\) are proper, lower semicontinuous (lsc), and convex.\nThey may be nonsmooth, while \\( h \\) is a smooth and convex function. Here, \\( n = \\sum_{i=1}^{n_{\\text{block}}} n_i \\), \\( x^i \\in \\mathbb{R}^{n_i} \\), and \\( g(x) := \\sum_{i=1}^{n_{\\text{block}}} g_i(x^i) \\).\\footnote{Note that, from the point of view of general theory, $f$ is superfluous, and can be subsumed by $h \\circ F$. It is, however, practical to explicitly include it for our application examples.}\n","sha256_utf8":"184cb8a27f24077254a3ab0b1094c2254e0c15fd3cd1573753fcd5e1773f701b"}]},{"id":"S002","semantic_type":"notation","kind":"notation","title":"范数与分块次微分记号","source_units":[{"start":19731,"end":20830,"start_line":239,"end_line":240,"raw_tex":"In what follows, we use $\\| \\cdot \\|$ to denote the 2-norm of a vector or a matrix, respectively. Let $n = \\sum_{i=1}^{n_\\text{block}} n_i$. For a differentiable function $\\phi: \\mathbb{R}^n \\to \\mathbb{R}$, we denote its gradient with respect to the $i^{\\text{th}}$ block at a point $x$ as $\\nabla_i \\phi(x) \\in \\mathbb{R}^{n_i}$, for any $i \\in \\{1, \\dots, n_\\text{block}\\}$. For a differentiable vector function $F: \\mathbb{R}^n \\to \\mathbb{R}^m$, we denote its Jacobian with respect to the $i^{\\text{th}}$ block at a given point $x$ as $\\nabla_i F(x) \\in \\mathbb{R}^{m \\times n_i}$. We define the distance from a point $x$ to a set $A$ as $\\dist(x, A) := \\min_{a \\in A} \\| x - a \\|$. Furthermore, for a proper, lower semicontinuous (lsc) function $g: \\mathbb{R}^n \\to \\mathbb{R}$, we use $\\partial g(x)$ to refer to its Mordukhovich (aka.~limiting) subdifferential at a point $x$, and $\\partial^{\\infty} g(x)$ to refer to the horizon subdifferential.\nThis, instead of the plain convex subdifferential, is required due to the nonconvexity of $h \\circ F$ and hence the composite objective $\\phi$.\n","sha256_utf8":"18d83795d49602866319ea7b881924d77897feb9fd94c8c728aaa375f3af6161"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"驻点与近似驻点","source_units":[{"start":20830,"end":21291,"start_line":241,"end_line":241,"raw_tex":"For more details about the subdifferential of nonsmooth functions, we refer to \\cite{rockafellar2009variational}. In the following, we say that a point \\( x^* \\in \\mathbb{R}^n \\) is a stationary point of problem~\\eqref{eq:problem} if \\( 0 \\in \\partial \\varphi(x^*) \\). Moreover, for \\( \\epsilon > 0 \\), \\( x^* \\) is said to be an \\( \\epsilon \\)-stationary point of problem \\eqref{eq:problem} if \\( \\dist\\left( 0, \\partial \\varphi(x^*) \\right) \\leq \\epsilon \\).\n","sha256_utf8":"39c116483b873d19a2a14f80a337555d982d7db2771caacd2db9575fb8989bf7"}]},{"id":"S004","semantic_type":"assumption","kind":"assumption","title":"光滑性凸性及下界假设","source_units":[{"start":21626,"end":22612,"start_line":246,"end_line":263,"raw_tex":"\\begin{assumption}\\label{ass:1}\n    \\begin{enumerate} %[itemsep=1pt]\n    \\item Given a compact set $\\mathcal{C}$, the gradients of $f$ and $F_j$ for $j=1:m$ are block coordinate-wise Lipschitz continuous in $\\mathcal{C}$, i.e., $\\forall i\\in \\{1,\\cdots,n_{\\text{block}}\\},\\;x,y\\in \\mathcal{C}$\n        \\begin{align*}\n        \\|\\nabla_i f(x) - \\nabla_if(y)\\|\\leq L^{f}_i \\|x - y\\|,\n        \\|\\nabla_i F_j(x) - \\nabla_i F_j(y)\\|\\leq L^{F_j}_i \\|x - y\\|.\n        \\end{align*}\n    \\item $h$ is convex, differentiable with Lipschitz gradient\n        \\begin{align*}\n            \\|\\nabla h(u) - \\nabla h(v)\\|\\leq L_{h} \\|u - v\\|,\\;\\forall \\;u,v\\in\\mathbb{R}^{m}.\n        \\end{align*}\n    \\item Each $g_i$ is a proper, lsc, convex function.\n    \\item $ \\forall x\\in\\mathcal{C}$, $u\\in\\mathbb{R}^m$ and $i\\in\\{1,\\cdots,n_{\\text{block}}\\}$ we have\n        $\n             f(x) \\geq f^*,\\; h(u)\\geq h^*,\\;g_i(x) \\geq g^*.% \\varphi(x) \\geq \\varphi^*,\\;\n        $\n    \\end{enumerate}\n\\end{assumption}","sha256_utf8":"373c12d4cac8292afaeb04529c622b9f7838a5a5404bb121ab618d5a1c4bf46d"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"分块线性化模型","source_units":[{"start":22751,"end":23233,"start_line":265,"end_line":273,"raw_tex":"In the sequel, for all $i=\\{1,\\ldots,n_{\\text{block}}\\}$, $\\bar{x}^i, s\\in\\mathbb{R}^{n_i}$ and $\\bar{x}\\in\\mathbb{R}^n$, we denote\n\\begin{align*}\n    &l^i_f(s;\\bar{x}^i)\n    :=\n    f(\\bar{x})+\\langle\\nabla_i f(\\bar{x}),s-\\bar{x}^i\\rangle,\\quad\n    l^i_F \\big(s;\\bar{x}^i \\big):=F(\\bar{x})+ \\nabla_i F(\\bar{x}) \\big(s-\\bar{x}^i \\big),\\\\\n    &\\quad\\text{and}\\quad\n    \\bar{\\varphi}^i(s;\\bar{x}) = l^i_f \\big( s;\\bar{x}^i \\big)+ h \\big( l^i_F(s;\\bar{x}^i) \\big) + g_i(s).\n\\end{align*}","sha256_utf8":"c39ad7517393b517670dac67f0c5ae2604d157b2925278cba0ab30328d5fead5"}]},{"id":"S006","semantic_type":"algorithm","kind":"algorithm","title":"单调随机分块算法","source_units":[{"start":23345,"end":24669,"start_line":276,"end_line":301,"raw_tex":"\\begin{algorithm*}[ht]\n\\caption{Monotone LiBCoD }\n\\label{alg:libcod}\n\n\\KwIn{$x_0$, $\\beta_1 \\geq \\frac{\\beta_{\\min}}{2}>0$, $p_{\\min} \\in (0,1)$}\nSet $k = 0$\n\n\\While{termination criterion not satisfied}{\n  Pick a random index $i_k \\in \\{1,\\cdots,n_{\\text{block}}\\}$ with probability $p_{i_k} \\geq p_{\\min}$\\:\n\n  \\Repeat(\n    %\\tcp*[f]{\\label{line:dec_mon} $\\varphi(x_{k+1}) \\leq  \\varphi(x_k) - \\frac{\\beta_{k+1}}{2}\\|x_{k+1} - x_k\\|^{2}$}\n  ){\n    \\textcolor{blue}{$\\varphi(x_{k+1}) \\leq  \\varphi(x_k) - \\frac{\\beta_{k+1}}{2}\\|x_{k+1} - x_k\\|^{2}$}\n  }{\n    $\\beta_{k+1} := 2\\beta_{k+1}$ \\label{line:lin_search}\n\n    $s_{k+1} := \\arg\\min\\limits_{s \\in \\mathbb{R}^{n_{i_k}}} \\resizebox{!}{2ex}{$\\langle\\nabla_{i_k} f(x_k),s-x_k^{i_k}\\rangle + h\\left(F(x_k)+\\langle\\nabla_{i_k} F(x_k),s-x_k^{i_k}\\rangle\\right) + g_{i_k}(s) + \\frac{\\beta_{k+1}}{2}\\|s - x^{i_k}\\|^2$}$ \n    %$s_{k+1} := \\arg\\min\\limits_{s \\in \\mathbb{R}^{n_{i_k}}} \\langle\\nabla_{i_k} f(x_k),s-x_k^{i_k}\\rangle+ h\\left(F(x_k)+\\langle\\nabla_{i_k} F(x_k),s-x_k^{i_k}\\rangle\\right) + g_{i_k}(s) + \\frac{\\beta_{k+1}}{2}\\|s - x^{i_k}\\|^2$ \\label{line:libcod:subprob}\n    \n    $x^{i_k}_{k+1} := s_{k+1}$ and $x^{i}_{k+1} = x_k^i$ for $i\\neq i_k$\n  }\n\n  Set $k=k+1$ and $\\beta_{k+1} = \\max\\left\\{\\frac{\\beta_{k}}{4},\\frac{\\beta_{\\min}}{2}\\right\\}$\n}\n\\end{algorithm*}","sha256_utf8":"3286c01a903650d7658fc8b52d3b4c1f3232bb2298b4e826117537c951eeb319"}]},{"id":"S007","semantic_type":"notation","kind":"notation","title":"迭代增量","source_units":[{"start":25151,"end":25190,"start_line":303,"end_line":305,"raw_tex":"$\n\\Delta{x}_{k+1} := x_{k+1} - x_k .\n$\n","sha256_utf8":"3b0dcde8ebcb34181ffa107a197574d72a0927863714d2f08acfe1ff39280acf"}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","title":"充分正则化阈值及目标下界","source_units":[{"start":25505,"end":25643,"start_line":309,"end_line":311,"raw_tex":"\\begin{equation} \\label{eq:beta_cond}\n    \\beta_{k+1} \\geq L^f_{i_k} + L^F_{i_k}\\sqrt{2 L_h}\\sqrt{\\varphi(x_k)- \\varphi^*},\n\\end{equation}","sha256_utf8":"70222bdb74e09e01e3334f0123061c2724c60bdd02e7668820ed6e911593ad23"},{"start":25644,"end":25753,"start_line":312,"end_line":312,"raw_tex":"where $\\varphi^* = h^* + f^* + g^*$, then the process in Monotone LiBCoD \\ref{alg:libcod}, is well defined. \n","sha256_utf8":"0e9945a4f9a9887fcb4bc016adeb5ae2bedde47e3d3c3989f286aacbc903681b"}]},{"id":"S009","semantic_type":"derived_fact","kind":"derived_fact","title":"正则化参数统一上界","source_units":[{"start":33491,"end":33739,"start_line":446,"end_line":450,"raw_tex":"Note that in Monotone LiBCoD \\ref{alg:libcod}, $\\beta_{k+1}$ is always upper bounded by\n\\begin{equation}\n\\label{bar_gamma}\n    \\bar{\\beta} := \\sup_{k\\geq 1}\\beta_k \\leq 2 \\left(L^f+ L^F\\sqrt{2L_h}\\sqrt{\\varphi(x_0)-\\varphi^*}\\right),\n\\end{equation}","sha256_utf8":"1f39fb8d444304cb841bf10478e2d9ea70c28da67adb69b2f9f7d205b3b36836"},{"start":33740,"end":33870,"start_line":451,"end_line":451,"raw_tex":" where $L^f:=\\max_{i\\in\\{1, \\ldots, n_{\\text{block}}\\}} L^f_{i}$  and  $L^F:=\\max_{i\\in\\{1, \\ldots, n_{\\text{block}}\\}} L^F_{i}$.\n","sha256_utf8":"4ab1f514c0388d5dcd2565f91b325651a642a369332c603037633e13a6ed7f87"}]},{"id":"S010","semantic_type":"theorem","kind":"theorem","title":"单调随机分块迭代复杂度","source_units":[{"start":43343,"end":44239,"start_line":597,"end_line":609,"raw_tex":"\\begin{theorem}[Iteration complexity]\n    \\label{thm:convergence:monotone}\n    Let $(x_k)_{k\\geq 0}$ be generated by Monotone LiBCoD \\ref{alg:libcod}. If \\cref{ass:1} holds on a compact set to which the iterates belong, then, for any $\\epsilon>0$,  Monotone LiBCoD \\ref{alg:libcod} yields, in expectation, an $\\epsilon$-stationary point of \\eqref{eq:problem}, i.e.,\n    \\[\n     \\E\\left[\\dist(0, \\partial\\varphi(x_{k+1})) \\right] \\leq \\epsilon,\n    \\]\n%  \\textcolor{red}{where $\\partial \\varphi(x)$ denotes the Mordukhovich (aka.~limiting) subdifferential of $\\varphi$ at $x$,}\n    within  \n    \\begin{gather*}\n        \\left(\\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min} \\epsilon^2} + \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min} \\epsilon} \\right)\\left(\\varphi(x_0) - \\varphi^*\\right)\n    \\end{gather*}  \n    iterations.\n\\end{theorem}","sha256_utf8":"7a2b6c2872a12f4cf7c3c6e3df8c7cff496e537116453d47c6d1e930cc1b893c"}]},{"id":"S011","semantic_type":"proof","kind":"proof","title":"迭代复杂度完整证明","source_units":[{"start":44282,"end":51300,"start_line":612,"end_line":712,"raw_tex":"\\begin{proof}\n    From \\cref{propos:asymtotic:monotone}, we know that  $\\E\\left[\\dist(0, \\partial\\varphi(x_{k})) \\right] \\to 0$ as $k\\to \\infty$. Let $\\epsilon>0$ and let $N\\geq 0$ be the first integer such that\n    \\begin{equation}\n    \\E\\left[\\dist(0, \\partial\\varphi(x_{N+1})) \\right] \\leq \\epsilon.\n    \\end{equation}\n    Using  \\eqref{bounde_gradient1} from  \\cref{bound_subgradient}, it follows that at each iteration, $k \\in [0:N]$, we encounter one of the following two cases:\\\\\n    \\noindent \\textit{Case 1: } If the following holds\n    \\[\n        \\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)\\|\\Delta x_{k+1}\\|\\geq \\frac{L_h M^F L^F}{2}\\|\\Delta x_{k+1}\\|^2,\n    \\]\n    then we get\n    \\begin{align} \\label{case1}\n        & \\dist^2\\left(0, \\partial_{i_k} \\varphi (x_{k+1}) \\right) \\leq 2 \\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2\\|\\Delta x_{k+1}\\|^2 \\nonumber\\\\\n        &\\overset{\\eqref{decrease}}{\\leq} \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{\\beta_{k+1}}\\left(\\varphi(x_{k}) - \\varphi(x_{k+1})\\right).\n    \\end{align}\n    \\noindent \\textit{Case 2: } Otherwise,  the following is valid \n    \\[\n        \\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)\\|\\Delta x_{k+1}\\|< \\frac{L_h M^F L^F}{2}\\|\\Delta x_{k+1}\\|^2,\n    \\]\n    which yields\n    \\begin{equation}\\label{case2}\n        \\dist\\left(0, \\partial_{i_k} \\varphi (x_{k+1}) \\right) \\leq L_h M^F L^F \\|\\Delta x_{k+1}\\|^2\\overset{\\eqref{decrease}}{\\leq} \\frac{2 L_h M^F L^F}{\\beta_{k+1}}\\left(\\varphi(x_{k}) - \\varphi(x_{k+1})\\right).\n    \\end{equation}\n    Define $\\mathcal{I}_1$ as the set of iterations $k \\in [0:N-1]$ at which   \\textit{Case 1} holds, and  $\\mathcal{I}_2$ as the set of iterations $k\\in [0:N-1]$ at which   \\textit{Case 2} holds. Clearly: $N=|\\mathcal{I}_1|+|\\mathcal{I}_2|+1$. We first derive an upper bound for $|\\mathcal{I}_1|$.  Let $k \\in \\mathcal{I}_1$, we have\n    \\allowdisplaybreaks\n    \\begin{equation}\n        \\label{criticality1_squared}\n        \\begin{split}\n        &\\dist^2(0, \\partial\\varphi(x_{k+1}))\n        =\n        \\sum_{i=1}^{n_{\\text{block}}}{\\dist^2 \\left(0, \\partial_i\\varphi(x_{k+1})\\right)}\n        \\\\\n        &\n        \\leq\n        \\frac{1}{p_{\\min}}\\sum_{i=1}^{n_{\\text{block}}}{p_i \\; \\dist^2 \\left(0, \\partial_i\\varphi(x_{k+1})\\right)}\n        \\\\\n        &\n        = \\frac1p_{\\min}  \\E_{k}\\left[\\dist^2 \\left(0, \\partial_{i_k}\\varphi(x_{k+1})\\right)\\right]\n        \\\\\n        &\n        \\overset{\\eqref{case1} }{\\leq }\n        \\frac1p_{\\min}  \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{\\beta_{k+1}}      \\E_{k} \\left[\\varphi(x_{k}) - \\varphi(x_{k+1}) \\right]\n        \\\\\n        &\n        \\leq\n        \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}      \n        \\left(\\varphi(x_k) - \\E_{k} [\\varphi(x_{k+1})]\\right).\n        \\end{split}\n    \\end{equation}\n    Taking the expectation in the inequality \\eqref{criticality1_squared} and summing it over $\\mathcal{I}_1$ yields\n    \\begin{align*}\n        |\\mathcal{I}_1|\\epsilon^2  & < \\sum_{k\\in\\mathcal{I}_1} \\left(\\E\\left[\\dist(0, \\partial\\varphi(x_{k+1}))\\right]\\right)^2\\leq \\sum_{k\\in\\mathcal{I}_1} \\E\\left[\\dist^2(0, \\partial\\varphi(x_{k+1}))\\right]  \\\\\n        & \\leq  \\sum_{k\\in\\mathcal{I}_1}  \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}      \n        \\E\\left[\\varphi(x_k) - \\E_{k} [\\varphi(x_{k+1})]\\right] \\\\\n        & = \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}   \\sum_{k\\in\\mathcal{I}_1}      \n        \\left(\\E\\left[\\varphi(x_k)\\right] - \\E [\\varphi(x_{k+1})]\\right)\\\\\n        & \\overset{\\eqref{decrease}}{\\leq}  \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}   \\sum_{k=0}^{N-1}      \n        \\left(\\E\\left[\\varphi(x_k)\\right] - \\E [\\varphi(x_{k+1})]\\right)  \\\\\n        & = \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}  \\left(\\E[\\varphi(x_0)] - \\E[\\varphi(x_N)]\\right)  \\\\ \n        & \\leq \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}  \\left(\\varphi(x_0) - \\varphi^*\\right) .\n    \\end{align*}\n    Thus, we have: $|\\mathcal{I}_1|< \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min} \\epsilon^2}  \\left(\\varphi(x_0) - \\varphi^*\\right)$. Similarly, we derive an upper bound for $|\\mathcal{I}_2|$. Let $k\\in \\mathcal{I}_2$, we have\n    \\allowdisplaybreaks\n    \\begin{equation}\n        \\label{criticality1}\n        \\begin{split}\n        \\dist(0, \\partial\\varphi(x_{k+1}))\n        &\n        \\leq\n        \\sum_{i=1}^{n_{\\text{block}}}{\\dist(0, \\partial_i\\varphi(x_{k+1}))}\n        \\\\\n        &\n        \\leq\n        \\frac{1}{p_{\\min}}\\sum_{i=1}^{n_{\\text{block}}}{p_i \\; \\dist(0, \\partial\\varphi_i(x_{k+1}))}\n        \\\\\n        &\n        = \\frac1p_{\\min}  \\E_{k}\\left[\\dist(0, \\partial\\varphi_{i_k}(x_{k+1}))\\right]\n        \\\\\n        &\n        \\overset{\\eqref{case2} }{\\leq }\n        \\frac1p_{\\min}  \\frac{2 L_h M^F L^F}{\\beta_{k+1}}      \\E_{k} \\left[\\varphi(x_{k}) - \\varphi(x_{k+1}) \\right]\n        \\\\\n        &\n        \\leq\n         \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}}      \n        \\left(\\varphi(x_k) - \\E_{k} [\\varphi(x_{k+1})]\\right).\n        \\end{split}\n    \\end{equation}\n    Taking the expectation in the inequality \\eqref{criticality1} and summing it over $\\mathcal{I}_2$ yields\n    \\begin{align*}\n        &  |\\mathcal{I}_2|\\epsilon  < \\sum_{k\\in\\mathcal{I}_2} \\E\\left[\\dist(0, \\partial\\varphi(x_{k+1}))\\right]  \\leq  \\sum_{k\\in\\mathcal{I}_2} \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}} \\E\\left[\\varphi(x_k) - \\E_{k} [\\varphi(x_{k+1})]\\right]\\\\\n        &  = \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}} \\sum_{k\\in\\mathcal{I}_2}  \\left(\\E\\left[\\varphi(x_k)\\right] - \\E [\\varphi(x_{k+1})]\\right)\\\\\n        & \\overset{\\eqref{decrease}}{\\leq}  \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}} \\sum_{k=0}^{N-1}  \\left(\\E\\left[\\varphi(x_k)\\right] - \\E [\\varphi(x_{k+1})]\\right)\\\\\n        & \\leq  \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}}   \\left(\\E\\left[\\varphi(x_0)\\right] - \\E [\\varphi(x_{N+1})]\\right)\\leq \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}}   \\left(\\varphi(x_0) - \\varphi^*\\right).\n    \\end{align*}\n    Therefore, we obtain: $|\\mathcal{I}_2|< \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min} \\epsilon}   \\left(\\varphi(x_0) - \\varphi^*\\right) $.  Consequently, we obtain\n    \\[\n        N \\leq  \\left(\\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min} \\epsilon^2}  + \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min} \\epsilon} \\right)\\left(\\varphi(x_0) - \\varphi^*\\right).\n        \\]\n    This proves our claim. \\hfill\\qed\n\\end{proof}","sha256_utf8":"e5a258c20ba5951dbaa2e179193de77c3448701724995ab80e802cfbb9875b95"}]},{"id":"S012","semantic_type":"lemma","kind":"lemma","title":"充分下降引理","source_units":[{"start":26108,"end":26512,"start_line":316,"end_line":323,"raw_tex":"\\begin{lemma}\n    \\label{lemma1}\n    Let $(x_k)_{k\\geq 0}$ be generated by Monotone LiBCoD \\ref{alg:libcod}. If \\cref{ass:1} holds on a compact set $\\mathcal{C}$ containing the iterates, and $\\beta_{k+1}$ satisfies \\eqref{eq:beta_cond}, then\n    \\begin{align}\n        \\label{decrease}\n         \\varphi(x_{k+1}) \\leq  \\varphi(x_{k}) - \\frac{\\beta_{k+1}}{2} \\|x_{k+1} - x_k\\|^2.\n    \\end{align}\n\\end{lemma}","sha256_utf8":"c6e928fb95bd436e21a080ebb687429269362e0beb5c5a9473618349ba0ed2c3"}]},{"id":"S013","semantic_type":"proof","kind":"proof","title":"充分下降完整证明","source_units":[{"start":26518,"end":33462,"start_line":325,"end_line":444,"raw_tex":"\\begin{proof}\n    This proof follows a reasoning similar to that in \\cite{marumo2024accelerated} [Lemma 3.1], adapted to the coordinate descent framework with an additional smooth function $f$.  \n    %\\seb{it would be nice to repeat there some of the definitions (for better readability)}\n    %\\seb{it seems the proof (for the second term) follows exactly \\cite{marumo2024accelerated}? maybe mention this here, and explain the differences}\n    Note that the subproblem's objective function\n    \\[\n        q(s) := \\bar{\\varphi}^{i_k}(s;x_k)+\\frac{\\beta_{k+1}}{2}\\|s-x^i_k\\|^2,\n    \\]\n    By the strong convexity of this objective and $s_{k+1}$ being a minimiser, i.e., $0 \\in \\partial q(s_{k+1})$,\n    \\[\n        q(x_k^i)\n        \\ge\n        q(s_{k+1}) + \\frac{\\beta_{k+1}}{2}\\|s_{k+1}-x^i_k\\|^2.\n    \\]\n    Hence, expanding $q$,\n    % is strongly convex with a strong convexity constant $\\beta_{k+1}$, then,\n    % \\[\n    %    \\bar{\\varphi}^{i_k}(x_k;x_k)+\\frac{\\beta_{k+1}}{2}\\|x^i_k-x^i_k\\|^2 \\geq \\bar{\\varphi}^{i_k}(s_{k+1};x_k)+\\frac{\\beta_{k+1}}{2}\\|s_{k+1}-x^i_k\\|^2 + \\frac{\\beta_{k+1}}{2}\\norm{x_{k+1} - x_k}^2 \n    % \\]\n    % combining this with the optimality of $s_{k+1}$,  \n    the fact that $\\bar{\\varphi}^{i_k}(x^{i_k}_k;x_k)=\\varphi(x_{k}) - \\sum_{i\\neq i_k} g_i(x^i_{k})$ and making use of the fact that $x^i_{k+1} = x^i_k$ for $i\\neq i_k$, we get\n    \\begin{equation} \\label{initial_decrease}\n        \\bar{\\varphi}^{i_k}(s_{k+1};x_k)\\leq \\varphi(x_{k}) - \\sum_{i\\neq i_k} g_i(x^i_{k+1}) -\\beta_{k+1}\\|x_{k+1}-x_k\\|^2.\n    \\end{equation}\n%    \\seb{is it $\\frac{\\beta_{k+1}}{2}$? \\textcolor{red}{just $\\beta_{k+1}$ since we use the strong convexity of $\\bar{\\varphi}^{i_k}(\\cdot;x_k)$}}%\n%    \\seb{I still don't see this. Strong convexity gives $\\varphi(x) \\geq \\varphi(s_{k+1}) + \\frac{\\beta}{2} \\| x - s_{k+1}\\|^2$ ? \\textcolor{red}{ It is \\(\\varphi(x) + \\frac{\\beta}{2}\\|x - x_k\\|^2\\) that is strongly convex, whereas \\(\\varphi(x)\\) is not. Thus, we have:\n% \\[\n% \\left(\\varphi(s) + \\frac{\\beta}{2} \\|s - x_{k}\\|^2\\right) \\geq \\left(\\varphi(s_{k+1}) + \\frac{\\beta}{2} \\|s_{k+1} - x_{k}\\|^2\\right) + \\frac{\\beta}{2} \\|s_{k+1} - x_{k}\\|^2.\n% \\]\n% %%SEB:\n% ok, tricky; \n% } }%\n    Furthermore, since \\( f \\) has a block Lipschitz gradient with a Lipschitz constant \\( L_{i_k}^f \\), it follows that \\cite{nesterov2018lectures}\n        \\begin{equation}\n        \\label{smooth_object}\n        f(x_{k+1}) - l_f^{i_k}(s_{k+1};x_k) \\leq \\frac{L^f_{i_k}}{2}\\|x_{k+1}-x_k\\|^2.\n    \\end{equation}\n    %\\seb{can you add a reference to the equation that you are using? (also it would help to remind the readers of the definition of $l_f$)\n    %why is there no term with $\\nabla f$ ? (a similar equation is also used in (14) below). $s_{k+1}$ is not a minimizer of $f$}\n     Moreover, since \\( h \\) has a Lipschitz gradient with a Lipschitz constant \\( L_h \\), we have\n    \\begin{equation}\n    \\label{decrease_feasib}\n    \\begin{split}\n         h(F(x_{k+1}))& -  h(l^{i_k}_F(s_{k+1};x_k)) \n        \\\\\n        &\\leq   \\langle \\nabla h \\left( l^{i_k}_F(s_{k+1};x_k)\\right), F(x_{k+1})-l^{i_k}_F(s_{k+1};x_k) \\rangle \\\\\n        \\MoveEqLeft[-1] + \\frac{L_h}{2}\\|F(x_{k+1})-l^{i_k}_F(s_{k+1};x_k)\\|^2 \n        \\\\\n        & \\leq \\left\\|\\nabla h \\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right\\| \\|F(x_{k+1})-l^{i_k}_F(s_{k+1};x_k) \\|\\\\\n        \\MoveEqLeft[-1] + \\frac{L_h}{2}\\|F(x_{k+1})-l^{i_k}_F(s_{k+1};x_k)\\|^2 \n        \\\\\n        &\\overset{\\text{Ass. } \\ref{ass:1}}{\\leq} \\left\\|\\nabla h \\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right\\|  \\frac{L^F_{i_k}}{2}\\|\\Delta x_{k+1}\\|^2 + \\frac{L_h}{2} \\left(\\frac{L^F_{i_k}}{2}\\|\\Delta x_{k+1}\\|^2\\right)^2. \n    \\end{split}\n    \\end{equation}\n    Using the Young's inequality $2\\sqrt{a b} - a \\leq b$ for any $a, b \\geq 0$, and the following relation for $u =l^{i_k}_F(s_{k+1};x_k) \\in \\mathbb{R}^m$, with $v = u - \\frac{1}{L_h} \\nabla h(u)$\n    \\[\n        h^* - h(u) \\leq h(v) - h(u)\n        \\leq \\langle \\nabla h(u), v - u \\rangle + \\frac{L_h}{2} \\norm{v - u}^2 = -\\frac{1}{2L_h} \\norm{\\nabla h(u)}^2,\n    \\]\n    we bound\n    \\begin{equation}\n        \\label{bound_grad_norm_squared}\n        \\begin{split}\n            &\n            \\left\\|\\nabla h \\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right\\| - \\frac{ \\beta_{k+1} - L^f_{i_k}}{2L^F_{i_k}} \\leq  \\frac{L^F_{i_k}}{ 2(\\beta_{k+1} - L^f_{i_k})}    \\left\\|\\nabla h \\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right\\|^2\n            \\\\\n            &\n            \\leq\n            \\frac{L^F_{i_k}L_h}{ \\beta_{k+1} - L^f_{i_k}} \\left( h(l^{i_k}_F(s_{k+1};x_k)) - h^*  \\right)\n            %\\\\\n            %&\n            =\n            \\frac{L^F_{i_k}L_h}{ \\beta_{k+1} - L^f_{i_k}}\n            \\\\\n            \\MoveEqLeft[-4]\n            \\left( \\bar{\\varphi}^{i_k}(s_{k+1};x_k) - h^* - g_{i_k}(s_{k+1})- f(x_{k+1}) + f(x_{k+1}) - l_f^{i_k}(s_{k+1};x_k)  \\right)\n            \\\\\n            &\n            \\overset{\\eqref{initial_decrease}, \\eqref{smooth_object}}{\\leq}\n            \\frac{L^F_{i_k} L_h}{\\beta_{k+1} - L^f_{i_k}} \\left( \\varphi(x_k) - \\varphi^* - \\frac{2\\beta_{k+1} - L^F_{i_k}}{2}\\|\\Delta x_{k+1}\\|^2 \\right)\n            \\\\\n            &\n            \\leq\n            \\frac{L^F_{i_k} L_h}{\\beta_{k+1} - L^f_{i_k}} \\left( \\varphi(x_k) - \\varphi^*\\right) - L^F_{i_k} L_h\\|\\Delta x_{k+1}\\|^2\n            \\\\\n            &\n            \\overset{\\eqref{eq:beta_cond}}{\\leq}\\frac{\\beta_{k+1}-L^f_{i_k}}{2L^F_{i_k}} - L^F_{i_k} L_h\\|\\Delta x_{k+1}\\|^2,\n        \\end{split}\n    \\end{equation}\n    where third inequality follows from the definition of $\\bar{\\varphi}^{i_k}$. Moreover, Using \\eqref{bound_grad_norm_squared} in \\eqref{decrease_feasib}, we deduce that\n    \\begin{equation}\n        \\label{to_use_next}\n        \\begin{split}\n                    h\\left(F(x_{k+1})\\right) &- h\\left(l^{i_k}_F(x_{k+1};x_k) \\right)\\\\\n        & \\leq\n        \\frac{\\beta_{k+1}-L^f_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2 - \\left(\\frac{L_h (L^F_{i_k})^2}{2} - \\frac{L_h (L^{F}_{i_k})^2}{8} \\right) \\|\\Delta x_{k+1}\\|^4\n        \\\\\n        &=\n        \\frac{\\beta_{k+1}-L^f_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2-  \\frac{3 L_h (L^{F}_{i_k})^2}{8}  \\|\\Delta x_{k+1}\\|^4.\n        \\end{split}\n    \\end{equation}\n    Moreover, we have\n    \\begin{align*}\n    \\begin{split}\n        \\varphi(x_{k+1}) -\\sum_{i\\neq i_k} g_i(x^i_{k+1}) - \\bar{\\varphi}^{i_k}(s_{k+1};x_k)\n        &=\n        f(x_{k+1}) - l^{i_k}_f(s_{k+1};x_k) + h(F(x_{k+1}))\\\\\n        \\MoveEqLeft [-1] - h(l^{i_k}_F(x_{k+1};x_k)).\n    \\end{split}\n    \\end{align*}\n    Using  \\eqref{smooth_object} and \\eqref{to_use_next} in the previous relation, it follows that\n    \\[\n        \\varphi(x_{k+1}) \\leq \\bar{\\varphi}^{i_k}(s_{k+1};x_k) + \\sum_{i\\neq i_k} g_i(x^i_{k+1}) + \\frac{\\beta_{k+1}}{2} \\|\\Delta x_{k+1}\\|^2.\n    \\]\n    Finally, using \\eqref{initial_decrease}, we get\n    \\begin{align*}\n        \\varphi(x_{k+1}) \\leq  \\varphi(x_{k}) - \\frac{\\beta_{k+1}}{2} \\|\\Delta x_{k+1}\\|^2. \n    \\end{align*}\n    This proves our statement. \\hfill\\qed\n\\end{proof}","sha256_utf8":"24280f3ca4829d7214db8491a9d38f4783cfedfea7bb9453662f5e0d2f8223e1"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"强凸子问题给出模型下降","source_units":[{"start":26518,"end":28709,"start_line":325,"end_line":356,"raw_tex":"\\begin{proof}\n    This proof follows a reasoning similar to that in \\cite{marumo2024accelerated} [Lemma 3.1], adapted to the coordinate descent framework with an additional smooth function $f$.  \n    %\\seb{it would be nice to repeat there some of the definitions (for better readability)}\n    %\\seb{it seems the proof (for the second term) follows exactly \\cite{marumo2024accelerated}? maybe mention this here, and explain the differences}\n    Note that the subproblem's objective function\n    \\[\n        q(s) := \\bar{\\varphi}^{i_k}(s;x_k)+\\frac{\\beta_{k+1}}{2}\\|s-x^i_k\\|^2,\n    \\]\n    By the strong convexity of this objective and $s_{k+1}$ being a minimiser, i.e., $0 \\in \\partial q(s_{k+1})$,\n    \\[\n        q(x_k^i)\n        \\ge\n        q(s_{k+1}) + \\frac{\\beta_{k+1}}{2}\\|s_{k+1}-x^i_k\\|^2.\n    \\]\n    Hence, expanding $q$,\n    % is strongly convex with a strong convexity constant $\\beta_{k+1}$, then,\n    % \\[\n    %    \\bar{\\varphi}^{i_k}(x_k;x_k)+\\frac{\\beta_{k+1}}{2}\\|x^i_k-x^i_k\\|^2 \\geq \\bar{\\varphi}^{i_k}(s_{k+1};x_k)+\\frac{\\beta_{k+1}}{2}\\|s_{k+1}-x^i_k\\|^2 + \\frac{\\beta_{k+1}}{2}\\norm{x_{k+1} - x_k}^2 \n    % \\]\n    % combining this with the optimality of $s_{k+1}$,  \n    the fact that $\\bar{\\varphi}^{i_k}(x^{i_k}_k;x_k)=\\varphi(x_{k}) - \\sum_{i\\neq i_k} g_i(x^i_{k})$ and making use of the fact that $x^i_{k+1} = x^i_k$ for $i\\neq i_k$, we get\n    \\begin{equation} \\label{initial_decrease}\n        \\bar{\\varphi}^{i_k}(s_{k+1};x_k)\\leq \\varphi(x_{k}) - \\sum_{i\\neq i_k} g_i(x^i_{k+1}) -\\beta_{k+1}\\|x_{k+1}-x_k\\|^2.\n    \\end{equation}\n%    \\seb{is it $\\frac{\\beta_{k+1}}{2}$? \\textcolor{red}{just $\\beta_{k+1}$ since we use the strong convexity of $\\bar{\\varphi}^{i_k}(\\cdot;x_k)$}}%\n%    \\seb{I still don't see this. Strong convexity gives $\\varphi(x) \\geq \\varphi(s_{k+1}) + \\frac{\\beta}{2} \\| x - s_{k+1}\\|^2$ ? \\textcolor{red}{ It is \\(\\varphi(x) + \\frac{\\beta}{2}\\|x - x_k\\|^2\\) that is strongly convex, whereas \\(\\varphi(x)\\) is not. Thus, we have:\n% \\[\n% \\left(\\varphi(s) + \\frac{\\beta}{2} \\|s - x_{k}\\|^2\\right) \\geq \\left(\\varphi(s_{k+1}) + \\frac{\\beta}{2} \\|s_{k+1} - x_{k}\\|^2\\right) + \\frac{\\beta}{2} \\|s_{k+1} - x_{k}\\|^2.\n% \\]\n% %%SEB:\n% ok, tricky; \n% } }%\n","sha256_utf8":"b2bf4d1d4398fe071c50a41b0dc2caf4d7203d37b2b3a89fa81c194fe5513a11"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"光滑项线性化误差","source_units":[{"start":28709,"end":29289,"start_line":357,"end_line":363,"raw_tex":"    Furthermore, since \\( f \\) has a block Lipschitz gradient with a Lipschitz constant \\( L_{i_k}^f \\), it follows that \\cite{nesterov2018lectures}\n        \\begin{equation}\n        \\label{smooth_object}\n        f(x_{k+1}) - l_f^{i_k}(s_{k+1};x_k) \\leq \\frac{L^f_{i_k}}{2}\\|x_{k+1}-x_k\\|^2.\n    \\end{equation}\n    %\\seb{can you add a reference to the equation that you are using? (also it would help to remind the readers of the definition of $l_f$)\n    %why is there no term with $\\nabla f$ ? (a similar equation is also used in (14) below). $s_{k+1}$ is not a minimizer of $f$}\n","sha256_utf8":"404cb36b6864d546e49309012640210275f807c689dfd92c6066b7512984d4c4"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"复合项误差初步估计","source_units":[{"start":29289,"end":30195,"start_line":364,"end_line":378,"raw_tex":"     Moreover, since \\( h \\) has a Lipschitz gradient with a Lipschitz constant \\( L_h \\), we have\n    \\begin{equation}\n    \\label{decrease_feasib}\n    \\begin{split}\n         h(F(x_{k+1}))& -  h(l^{i_k}_F(s_{k+1};x_k)) \n        \\\\\n        &\\leq   \\langle \\nabla h \\left( l^{i_k}_F(s_{k+1};x_k)\\right), F(x_{k+1})-l^{i_k}_F(s_{k+1};x_k) \\rangle \\\\\n        \\MoveEqLeft[-1] + \\frac{L_h}{2}\\|F(x_{k+1})-l^{i_k}_F(s_{k+1};x_k)\\|^2 \n        \\\\\n        & \\leq \\left\\|\\nabla h \\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right\\| \\|F(x_{k+1})-l^{i_k}_F(s_{k+1};x_k) \\|\\\\\n        \\MoveEqLeft[-1] + \\frac{L_h}{2}\\|F(x_{k+1})-l^{i_k}_F(s_{k+1};x_k)\\|^2 \n        \\\\\n        &\\overset{\\text{Ass. } \\ref{ass:1}}{\\leq} \\left\\|\\nabla h \\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right\\|  \\frac{L^F_{i_k}}{2}\\|\\Delta x_{k+1}\\|^2 + \\frac{L_h}{2} \\left(\\frac{L^F_{i_k}}{2}\\|\\Delta x_{k+1}\\|^2\\right)^2. \n    \\end{split}\n    \\end{equation}\n","sha256_utf8":"ef3bc47a8992df62ca452fad45015c6c80f1b59c73ae0fb4ec9862e44fa4264e"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"利用函数下界控制外层梯度","source_units":[{"start":30195,"end":31977,"start_line":379,"end_line":413,"raw_tex":"    Using the Young's inequality $2\\sqrt{a b} - a \\leq b$ for any $a, b \\geq 0$, and the following relation for $u =l^{i_k}_F(s_{k+1};x_k) \\in \\mathbb{R}^m$, with $v = u - \\frac{1}{L_h} \\nabla h(u)$\n    \\[\n        h^* - h(u) \\leq h(v) - h(u)\n        \\leq \\langle \\nabla h(u), v - u \\rangle + \\frac{L_h}{2} \\norm{v - u}^2 = -\\frac{1}{2L_h} \\norm{\\nabla h(u)}^2,\n    \\]\n    we bound\n    \\begin{equation}\n        \\label{bound_grad_norm_squared}\n        \\begin{split}\n            &\n            \\left\\|\\nabla h \\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right\\| - \\frac{ \\beta_{k+1} - L^f_{i_k}}{2L^F_{i_k}} \\leq  \\frac{L^F_{i_k}}{ 2(\\beta_{k+1} - L^f_{i_k})}    \\left\\|\\nabla h \\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right\\|^2\n            \\\\\n            &\n            \\leq\n            \\frac{L^F_{i_k}L_h}{ \\beta_{k+1} - L^f_{i_k}} \\left( h(l^{i_k}_F(s_{k+1};x_k)) - h^*  \\right)\n            %\\\\\n            %&\n            =\n            \\frac{L^F_{i_k}L_h}{ \\beta_{k+1} - L^f_{i_k}}\n            \\\\\n            \\MoveEqLeft[-4]\n            \\left( \\bar{\\varphi}^{i_k}(s_{k+1};x_k) - h^* - g_{i_k}(s_{k+1})- f(x_{k+1}) + f(x_{k+1}) - l_f^{i_k}(s_{k+1};x_k)  \\right)\n            \\\\\n            &\n            \\overset{\\eqref{initial_decrease}, \\eqref{smooth_object}}{\\leq}\n            \\frac{L^F_{i_k} L_h}{\\beta_{k+1} - L^f_{i_k}} \\left( \\varphi(x_k) - \\varphi^* - \\frac{2\\beta_{k+1} - L^F_{i_k}}{2}\\|\\Delta x_{k+1}\\|^2 \\right)\n            \\\\\n            &\n            \\leq\n            \\frac{L^F_{i_k} L_h}{\\beta_{k+1} - L^f_{i_k}} \\left( \\varphi(x_k) - \\varphi^*\\right) - L^F_{i_k} L_h\\|\\Delta x_{k+1}\\|^2\n            \\\\\n            &\n            \\overset{\\eqref{eq:beta_cond}}{\\leq}\\frac{\\beta_{k+1}-L^f_{i_k}}{2L^F_{i_k}} - L^F_{i_k} L_h\\|\\Delta x_{k+1}\\|^2,\n        \\end{split}\n    \\end{equation}\n","sha256_utf8":"5f574bf63b91e12644b1b0f18227ae77a58edbff32a9dc55965a7046f90b3cc2"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"正则化控制复合项误差","source_units":[{"start":31977,"end":32660,"start_line":414,"end_line":425,"raw_tex":"    where third inequality follows from the definition of $\\bar{\\varphi}^{i_k}$. Moreover, Using \\eqref{bound_grad_norm_squared} in \\eqref{decrease_feasib}, we deduce that\n    \\begin{equation}\n        \\label{to_use_next}\n        \\begin{split}\n                    h\\left(F(x_{k+1})\\right) &- h\\left(l^{i_k}_F(x_{k+1};x_k) \\right)\\\\\n        & \\leq\n        \\frac{\\beta_{k+1}-L^f_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2 - \\left(\\frac{L_h (L^F_{i_k})^2}{2} - \\frac{L_h (L^{F}_{i_k})^2}{8} \\right) \\|\\Delta x_{k+1}\\|^4\n        \\\\\n        &=\n        \\frac{\\beta_{k+1}-L^f_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2-  \\frac{3 L_h (L^{F}_{i_k})^2}{8}  \\|\\Delta x_{k+1}\\|^4.\n        \\end{split}\n    \\end{equation}\n","sha256_utf8":"2f26ad2acf4da577c576a1013288a6d6c9cbe7055b4128705102501285437fd4"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"真实目标与模型差值分解","source_units":[{"start":32660,"end":32972,"start_line":426,"end_line":434,"raw_tex":"    Moreover, we have\n    \\begin{align*}\n    \\begin{split}\n        \\varphi(x_{k+1}) -\\sum_{i\\neq i_k} g_i(x^i_{k+1}) - \\bar{\\varphi}^{i_k}(s_{k+1};x_k)\n        &=\n        f(x_{k+1}) - l^{i_k}_f(s_{k+1};x_k) + h(F(x_{k+1}))\\\\\n        \\MoveEqLeft [-1] - h(l^{i_k}_F(x_{k+1};x_k)).\n    \\end{split}\n    \\end{align*}\n","sha256_utf8":"6b2611f75b05d6fefab3b17f6295eedcda00aecd085bea08c526abe658e94b11"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"合并误差得到充分下降","source_units":[{"start":32972,"end":33462,"start_line":435,"end_line":444,"raw_tex":"    Using  \\eqref{smooth_object} and \\eqref{to_use_next} in the previous relation, it follows that\n    \\[\n        \\varphi(x_{k+1}) \\leq \\bar{\\varphi}^{i_k}(s_{k+1};x_k) + \\sum_{i\\neq i_k} g_i(x^i_{k+1}) + \\frac{\\beta_{k+1}}{2} \\|\\Delta x_{k+1}\\|^2.\n    \\]\n    Finally, using \\eqref{initial_decrease}, we get\n    \\begin{align*}\n        \\varphi(x_{k+1}) \\leq  \\varphi(x_{k}) - \\frac{\\beta_{k+1}}{2} \\|\\Delta x_{k+1}\\|^2. \n    \\end{align*}\n    This proves our statement. \\hfill\\qed\n\\end{proof}","sha256_utf8":"c17eecfd23bd45ba8d1de22493ad3a433781bf55ede924a957b3702f4d187137"}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","title":"所选分块次梯度界","source_units":[{"start":34246,"end":35017,"start_line":459,"end_line":470,"raw_tex":"\\begin{lemma} \\label{bound_subgradient}\nLet \\((x_k)_{k \\geq 0}\\) be generated by Monotone LiBCoD \\ref{alg:libcod}. If \\cref{ass:1} holds on a compact set to which the iterates belong, then, for all \\(k \\geq 0\\), we have\n\\begin{equation}\\label{bounde_gradient1}\n    \\begin{split}\n     \\dist\\left(0, \\partial_{i_k} \\varphi (x_{k+1})  \\right)\n     & \\leq \\left( L^f_{i_k} + L^F_{i_k} \\sqrt{2L_h} \\sqrt{ \\varphi(x_0) - \\varphi^*} + \\beta_{k+1}\\right)  \\|\\Delta x_{k+1}\\|\n     \\\\\n     \\MoveEqLeft[-1] + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2,\n    \\end{split}\n\\end{equation}\nwhere $\\|\\nabla_{i_k}F(x_k)\\| \\leq M^{F}_{i_k}$.% \\textcolor{red}{and $\\partial \\varphi(x)$ denotes the Mordukhovich (aka.~limiting) subdifferential of $\\varphi$ at $x$.} \n\\end{lemma}","sha256_utf8":"e7def4eb5be23f646b3b2a42306a91d3a856f92f07ed1cf0528c3149541ea805"}]},{"id":"S022","semantic_type":"proof","kind":"proof","title":"分块次梯度界完整证明","source_units":[{"start":35086,"end":38605,"start_line":473,"end_line":520,"raw_tex":"\\begin{proof}\n    By the optimality condition corresponding to the $s_{k+1}$ determining subproblem on Line 6 of Monotone LiBCoD \\ref{alg:libcod}, we have\n    \\[\n        -\\nabla_{i_k} f(x_{k})-\\nabla_{i_k} F(x_{k})^T \\nabla h\\left( l^{i_k}_F(s_{k+1};x_k)\\right)-\\beta_{k+1}(x_{k+1}-x_{k})\\in \\partial g_{i_k}(s_{k+1}).\n    \\]\n    It then follows, by exploiting the definition of $\\varphi$ and the properties of the derivative, that\n    \\begin{align*}\n        &\\dist\\left(0,\\partial_{i_k} \\varphi (x_{k+1})\\right)\\\\\n        &=  \\dist\\left(-\\nabla_{i_k} f(x_{k+1}) - \\nabla_{i_k} F(x_{k+1})^T \\nabla h \\left( F(x_{k+1})\\right) , \\partial g_{i_k}(s_{k+1})\\right)\\\\\n        & \\leq  \\Bigg \\|\\nabla_{i_k} f(x_{k+1}) - \\nabla_{i_k} f(x_k) + \\Big(\\nabla_{i_k} F(x_{k+1}) - \\nabla_{i_k} F(x_{k})\\Big)^T\\nabla h\\left(  F(x_{k+1})\\right)  \\\\\n        & -\\beta_{k+1}\\Delta x_{k+1} +\\nabla_{i_k} F(x_{k})^T\\left( \\nabla h \\left( F(x_{k+1}) \\right)-\\nabla h\\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right) \\Bigg\\|.\n    \\end{align*}\n    Using the triangle inequality, we further get\n    \\begin{equation}\n        \\label{eq:lem2-bndsub}\n        \\begin{split}\n            &\n            \\dist\\left(0, \\partial_{i_k} \\varphi (x_{k+1}) \\right) \\leq  \\|\\nabla_{i_k} f(x_{k+1})-\\nabla_{i_k} f(x_k)\\|\n            \\\\\n            \\MoveEqLeft[-1] + \\left\\|\\nabla h\\left( F(x_{k+1})\\right)\\right\\|\\left\\|\\nabla_{i_k} F(x_{k+1}) - \\nabla_{i_k} F(x_{k})\\right\\|+\\beta_{k+1}\\|\\Delta x_{k+1}\\|\n            \\\\\n            \\MoveEqLeft[-1]\n            +\\|{\\nabla_{i_k} F(x_{k})}\\|\\left\\| \\nabla h \\left( F(x_{k+1}) \\right)-\\nabla h\\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right\\|\n            \\\\\n            &\n            {\\overset{{\\text{Ass. } \\ref{ass:1}}}{\\leq}} \\left(L^f_{i_k}+\\left\\|\\nabla h\\left( F(x_{k+1})\\right)\\right\\| L^F_{i_k} +\\beta_{k+1}\\right)\\|\\Delta x_{k+1}\\|\n            + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2\n            \\\\\n            &\\leq \\left( L^f_{i_k} + L^F_{i_k} \\sqrt{2L_h \\left(h(F(x_k)) \\!-\\! h^*\\right)} + \\beta_{k+1}\\right) \\|\\Delta x_{k+1}\\|  + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2\n            \\\\\n            &\n            \\leq  \\left( L^f_{i_k} + L^F_{i_k} \\sqrt{2L_h \\left(\\varphi(x_k) - \\varphi^*\\right)} + \\beta_{k+1}\\right)  \\|\\Delta x_{k+1}\\| + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2\n            \\\\\n            &\n            \\leq  \\left( L^f_{i_k} + L^F_{i_k} \\sqrt{2L_h \\left(\\varphi(x_0) - \\varphi^*\\right)} + \\beta_{k+1}\\right)  \\|\\Delta x_{k+1}\\| + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2\n             \\\\\n            &\n            \\leq  \\left( L^f + L^F \\sqrt{2L_h \\left(\\varphi(x_0) - \\varphi^*\\right)} + \\bar{\\beta}\\right)  \\|\\Delta x_{k+1}\\| + \\frac{L_h M^{F}L^{F}}{2} \\|\\Delta x_{k+1}\\|^2,\n        \\end{split}\n    \\end{equation}\n        where the third inequality follows from the following relation for $u= F(x_{k}) \\in \\mathbb{R}^m$, and $v = u - \\frac{1}{L_h} \\nabla h(u)$\n    \\[\n        h^* - h(u) \\leq h(v) - h(u)\n        \\leq \\langle \\nabla h(u), v - u \\rangle + \\frac{L_ h}{2} \\norm{v - u}^2 = -\\frac{1}{2L_h} \\norm{\\nabla h(u)}^2,\n    \\]\n    and $M^F:=\\max_{i\\in\\{1, \\ldots, n_{\\text{block}}\\}} M^F_{i}$, and the fourth inequality follows from the fact that \\( \\varphi(x) = f(x) + h(F(x)) + g(x) \\) and \\( \\varphi^* = h^* + f^* + g^* \\), where \\( h^*, g^*, \\) and \\( f^* \\) are the lower bounds of \\( h(x) \\), \\( g(x) \\), and \\( f(x) \\), respectively, as defined in \\cref{ass:1}.\n    This proves our claim. \\hfill\\qed\n\\end{proof}","sha256_utf8":"90e16efc9188ce1781c851c9f42b27e4a384161b4da213edd99d4e6bf4a55272"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"写出子问题最优性条件","source_units":[{"start":35086,"end":35412,"start_line":473,"end_line":477,"raw_tex":"\\begin{proof}\n    By the optimality condition corresponding to the $s_{k+1}$ determining subproblem on Line 6 of Monotone LiBCoD \\ref{alg:libcod}, we have\n    \\[\n        -\\nabla_{i_k} f(x_{k})-\\nabla_{i_k} F(x_{k})^T \\nabla h\\left( l^{i_k}_F(s_{k+1};x_k)\\right)-\\beta_{k+1}(x_{k+1}-x_{k})\\in \\partial g_{i_k}(s_{k+1}).\n    \\]\n","sha256_utf8":"c9053400b68e00501feabb7034d472b3bc8f4db4a1af160fe67bc87988215d28"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"构造真实目标次梯度残差","source_units":[{"start":35412,"end":36098,"start_line":478,"end_line":484,"raw_tex":"    It then follows, by exploiting the definition of $\\varphi$ and the properties of the derivative, that\n    \\begin{align*}\n        &\\dist\\left(0,\\partial_{i_k} \\varphi (x_{k+1})\\right)\\\\\n        &=  \\dist\\left(-\\nabla_{i_k} f(x_{k+1}) - \\nabla_{i_k} F(x_{k+1})^T \\nabla h \\left( F(x_{k+1})\\right) , \\partial g_{i_k}(s_{k+1})\\right)\\\\\n        & \\leq  \\Bigg \\|\\nabla_{i_k} f(x_{k+1}) - \\nabla_{i_k} f(x_k) + \\Big(\\nabla_{i_k} F(x_{k+1}) - \\nabla_{i_k} F(x_{k})\\Big)^T\\nabla h\\left(  F(x_{k+1})\\right)  \\\\\n        & -\\beta_{k+1}\\Delta x_{k+1} +\\nabla_{i_k} F(x_{k})^T\\left( \\nabla h \\left( F(x_{k+1}) \\right)-\\nabla h\\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right) \\Bigg\\|.\n    \\end{align*}\n","sha256_utf8":"0acff3d4c0d8f7e292e582475c6f3643087561a6bce605a361b57b6062983966"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"逐项估计残差并统一常数","source_units":[{"start":36098,"end":37897,"start_line":485,"end_line":512,"raw_tex":"    Using the triangle inequality, we further get\n    \\begin{equation}\n        \\label{eq:lem2-bndsub}\n        \\begin{split}\n            &\n            \\dist\\left(0, \\partial_{i_k} \\varphi (x_{k+1}) \\right) \\leq  \\|\\nabla_{i_k} f(x_{k+1})-\\nabla_{i_k} f(x_k)\\|\n            \\\\\n            \\MoveEqLeft[-1] + \\left\\|\\nabla h\\left( F(x_{k+1})\\right)\\right\\|\\left\\|\\nabla_{i_k} F(x_{k+1}) - \\nabla_{i_k} F(x_{k})\\right\\|+\\beta_{k+1}\\|\\Delta x_{k+1}\\|\n            \\\\\n            \\MoveEqLeft[-1]\n            +\\|{\\nabla_{i_k} F(x_{k})}\\|\\left\\| \\nabla h \\left( F(x_{k+1}) \\right)-\\nabla h\\left( l^{i_k}_F(s_{k+1};x_k)\\right)\\right\\|\n            \\\\\n            &\n            {\\overset{{\\text{Ass. } \\ref{ass:1}}}{\\leq}} \\left(L^f_{i_k}+\\left\\|\\nabla h\\left( F(x_{k+1})\\right)\\right\\| L^F_{i_k} +\\beta_{k+1}\\right)\\|\\Delta x_{k+1}\\|\n            + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2\n            \\\\\n            &\\leq \\left( L^f_{i_k} + L^F_{i_k} \\sqrt{2L_h \\left(h(F(x_k)) \\!-\\! h^*\\right)} + \\beta_{k+1}\\right) \\|\\Delta x_{k+1}\\|  + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2\n            \\\\\n            &\n            \\leq  \\left( L^f_{i_k} + L^F_{i_k} \\sqrt{2L_h \\left(\\varphi(x_k) - \\varphi^*\\right)} + \\beta_{k+1}\\right)  \\|\\Delta x_{k+1}\\| + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2\n            \\\\\n            &\n            \\leq  \\left( L^f_{i_k} + L^F_{i_k} \\sqrt{2L_h \\left(\\varphi(x_0) - \\varphi^*\\right)} + \\beta_{k+1}\\right)  \\|\\Delta x_{k+1}\\| + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2\n             \\\\\n            &\n            \\leq  \\left( L^f + L^F \\sqrt{2L_h \\left(\\varphi(x_0) - \\varphi^*\\right)} + \\bar{\\beta}\\right)  \\|\\Delta x_{k+1}\\| + \\frac{L_h M^{F}L^{F}}{2} \\|\\Delta x_{k+1}\\|^2,\n        \\end{split}\n    \\end{equation}\n","sha256_utf8":"1a54e1e5a918c80c1ef9a37f6992cb3115b17fabba3ef1e30ebd58e962e80122"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"补充梯度估计及常数定义","source_units":[{"start":37897,"end":38605,"start_line":513,"end_line":520,"raw_tex":"        where the third inequality follows from the following relation for $u= F(x_{k}) \\in \\mathbb{R}^m$, and $v = u - \\frac{1}{L_h} \\nabla h(u)$\n    \\[\n        h^* - h(u) \\leq h(v) - h(u)\n        \\leq \\langle \\nabla h(u), v - u \\rangle + \\frac{L_ h}{2} \\norm{v - u}^2 = -\\frac{1}{2L_h} \\norm{\\nabla h(u)}^2,\n    \\]\n    and $M^F:=\\max_{i\\in\\{1, \\ldots, n_{\\text{block}}\\}} M^F_{i}$, and the fourth inequality follows from the fact that \\( \\varphi(x) = f(x) + h(F(x)) + g(x) \\) and \\( \\varphi^* = h^* + f^* + g^* \\), where \\( h^*, g^*, \\) and \\( f^* \\) are the lower bounds of \\( h(x) \\), \\( g(x) \\), and \\( f(x) \\), respectively, as defined in \\cref{ass:1}.\n    This proves our claim. \\hfill\\qed\n\\end{proof}","sha256_utf8":"9e7c287e92c9de5a1db05fd70f1438ec6b16ea83e9fc6c561773747f04b1d318"}]},{"id":"S027","semantic_type":"proposition","kind":"proposition","title":"期望驻点残差渐近消失","source_units":[{"start":39036,"end":39390,"start_line":526,"end_line":532,"raw_tex":"\\begin{proposition}[Asymptotic convergence]\n    \\label{propos:asymtotic:monotone}\n    Let $(x_k)_{k\\geq 0}$ be generated by Monotone LiBCoD \\ref{alg:libcod}. If \\cref{ass:1} holds on a compact set to which the iterates belong, then we have \n    \\[\n        \\lim_{k\\to \\infty} \\E \\left[\\dist(0, \\partial\\varphi(x_{k})) \\right] = 0,\n    \\]\n\\end{proposition}","sha256_utf8":"a47be2a2f571a188b3928b20439e7616f1fa9fe6f75aad29bcd8b762c76bcf89"}]},{"id":"S028","semantic_type":"proof","kind":"proof","title":"期望渐近收敛完整证明","source_units":[{"start":39391,"end":43132,"start_line":533,"end_line":593,"raw_tex":"\\begin{proof}\n     Let $k\\geq0$. From Monotone LiBCoD \\ref{alg:libcod}, we have\n    \\begin{align*}\n        \\frac{\\beta_{\\min}}{2}\\|x_{k+1} - x_k\\|^{2}\\leq \\frac{\\beta_{k+1}}{2}\\|x_{k+1} - x_k\\|^{2}   \\leq  \\varphi(x_k) - \\varphi(x_{k+1}).\n    \\end{align*}\n    Taking the expectation, we obtain\n    \\begin{align*}\n        \\frac{\\beta_{\\min}}{2} \\E\\left[\\|x_{k+1} - x_k\\|^{2}\\right]  \\leq  \\E\\left[\\varphi(x_k)\\right] - \\E[\\varphi(x_{k+1})].\n    \\end{align*}\n    Summing the above inequality from $k=0$ to $K\\geq 0$, we get\n    \\begin{align*}\n        \\frac{\\beta_{\\min}}{2 }\\sum_{k=0}^{K}\\E\\left[\\|x_{k+1} - x_k\\|^{2}\\right]  &\\leq  \\sum_{k=0}^{K}\\left(\\E\\left[\\varphi(x_k)\\right] - \\E[\\varphi(x_{k+1})]\\right) \\\\\n        & = \\E\\left[\\varphi(x_0)\\right] - \\E[\\varphi(x_{K+1})]\\\\\n        & \\leq \\varphi(x_0) - \\varphi^*.\n        \\end{align*}\n    Taking the limit $K \\to \\infty$, we get\n    \\begin{align*}\n        \\frac{\\beta_{\\min}}{2 }\\sum_{k=0}^{\\infty}\\E\\left[\\|x_{k+1} - x_k\\|^{2}\\right]  \\leq \\varphi(x_0) - \\varphi^* < \\infty.\n    \\end{align*}\n    Since $\\beta_{\\min} > 0$, it follows that\n    \\begin{equation}\\label{zero_limit}\n        \\lim_{k\\to\\infty}{\\E\\left[\\|x_{k+1} - x_{k}\\|^2\\right]}=0 \\quad \\text{ and } \\quad \\lim_{k\\to\\infty}{\\E\\left[\\|x_{k+1} - x_{k}\\|\\right]}=0, \n    \\end{equation}\n    where the limit on the right-hand side of \\eqref{zero_limit} follows from Jensen's inequality. For simplicity, we denote $\\E_k[\\cdot] =  \\E[\\cdot|x_k]$.  On the other, using \\ref{bound_subgradient}, we deduce\n      \\allowdisplaybreaks\n    \\begin{equation}\n        \\label{subgrad_bound_asymp}\n        \\begin{split}\n            &\\dist(0, \\partial\\varphi(x_{k+1}))\n            \\leq\n            \\sum_{i=1}^{n_{\\text{block}}}{\\dist \\left(0, \\partial_i\\varphi(x_{k+1})\\right)}\n            \\\\\n            &\n            \\leq\n            \\frac{1}{p_{\\min}}\\sum_{i=1}^{n_{\\text{block}}}{p_i \\; \\dist \\left(0, \\partial_{i_k}\\varphi(x_{k+1})\\right)}\n            \\\\\n            &\n            = \\frac1p_{\\min}  \\E_{k}\\left[\\dist \\left(0, \\partial_{i_k}\\varphi(x_{k+1})\\right)\\right]\n            \\\\\n            &\n            \\overset{\\eqref{bounde_gradient1}}{\\leq }\n            \\frac1p_{\\min}\\E_{k}\\left[  \\left( L^f_{i_k} + L^F_{i_k} \\sqrt{2L_h} \\sqrt{ \\left(\\varphi(x_0) - \\varphi^*\\right)} + \\beta_{k+1}\\right)  \\|\\Delta x_{k+1}\\| \\right.\\\\\n            \\MoveEqLeft[-1] \\left. + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2 \\right]\n            \\\\\n            &\n            \\overset{\\eqref{bar_gamma}}{\\leq}\n            \\frac{1}{p_{\\min}} \\left( L^f + L^F \\sqrt{2L_h} \\sqrt{ \\left(\\varphi(x_0) - \\varphi^*\\right)} + \\bar{\\beta} \\right)  \\E_{k}\\left[\\|\\Delta x_{k+1}\\|\\right]\\\\\n             \\MoveEqLeft[-1] + \\frac{L_h M^F L^{F}}{2p_{\\min}} \\E_{k}\\left[\\|\\Delta x_{k+1}\\|^2 \\right].\n        \\end{split}\n    \\end{equation}\n  Taking expectation in \\eqref{subgrad_bound_asymp}, we obtain\n    \\begin{align*}\n            \\E\\left[\\dist(0, \\partial\\varphi(x_{k+1}))\\right] \n            %&\\leq             \\frac{1}{p_{\\min}} \\left( L^f + L^F \\sqrt{2L_h} \\sqrt{ \\left(\\varphi(x_0) - \\varphi^*\\right)} + \\bar{\\beta} \\right)  \\E\\left[\\E_{k}\\left[\\|\\Delta x_{k+1}\\|\\right]\\right] +  \\frac{L_h M^F L^{F}}{2p_{\\min}} \\E\\left[\\E_{k}\\left[\\|\\Delta x_{k+1}\\|^2 \\right]\\right] \\\\\n            &\\leq\n            \\frac{1}{p_{\\min}} \\left( L^f \\!+\\! L^F \\sqrt{2L_h} \\sqrt{ \\left(\\varphi(x_0) \\!-\\! \\varphi^*\\right)} + \\bar{\\beta} \\right)  \\E\\left[\\|\\Delta x_{k+1}\\|\\right]\\\\\n             \\MoveEqLeft[-1] +  \\frac{L_h M^F L^{F}}{2p_{\\min}} \\E\\left[\\|\\Delta x_{k+1}\\|^2 \\right]\n    \\end{align*}\n    Passing to the limit and using \\eqref{zero_limit}, we get\n      $  \\lim_{k\\to \\infty} \\E\\left[\\dist(0, \\partial\\varphi(x_{k})) \\right] = 0.$ \\hfill\\qed\n\\end{proof}","sha256_utf8":"87e2718e4bdfe583e546dd2ee61d5324d0cb6c68537c01e114489f29d17be990"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"下降量控制期望步长平方","source_units":[{"start":39391,"end":39848,"start_line":533,"end_line":541,"raw_tex":"\\begin{proof}\n     Let $k\\geq0$. From Monotone LiBCoD \\ref{alg:libcod}, we have\n    \\begin{align*}\n        \\frac{\\beta_{\\min}}{2}\\|x_{k+1} - x_k\\|^{2}\\leq \\frac{\\beta_{k+1}}{2}\\|x_{k+1} - x_k\\|^{2}   \\leq  \\varphi(x_k) - \\varphi(x_{k+1}).\n    \\end{align*}\n    Taking the expectation, we obtain\n    \\begin{align*}\n        \\frac{\\beta_{\\min}}{2} \\E\\left[\\|x_{k+1} - x_k\\|^{2}\\right]  \\leq  \\E\\left[\\varphi(x_k)\\right] - \\E[\\varphi(x_{k+1})].\n    \\end{align*}\n","sha256_utf8":"90b1294b182552105d765609da6d745d8102400cf28b66c57047d72db506e559"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"望远镜求和得到可求和性","source_units":[{"start":39848,"end":40438,"start_line":542,"end_line":551,"raw_tex":"    Summing the above inequality from $k=0$ to $K\\geq 0$, we get\n    \\begin{align*}\n        \\frac{\\beta_{\\min}}{2 }\\sum_{k=0}^{K}\\E\\left[\\|x_{k+1} - x_k\\|^{2}\\right]  &\\leq  \\sum_{k=0}^{K}\\left(\\E\\left[\\varphi(x_k)\\right] - \\E[\\varphi(x_{k+1})]\\right) \\\\\n        & = \\E\\left[\\varphi(x_0)\\right] - \\E[\\varphi(x_{K+1})]\\\\\n        & \\leq \\varphi(x_0) - \\varphi^*.\n        \\end{align*}\n    Taking the limit $K \\to \\infty$, we get\n    \\begin{align*}\n        \\frac{\\beta_{\\min}}{2 }\\sum_{k=0}^{\\infty}\\E\\left[\\|x_{k+1} - x_k\\|^{2}\\right]  \\leq \\varphi(x_0) - \\varphi^* < \\infty.\n    \\end{align*}\n","sha256_utf8":"1b26186a861d72edbe461f7eb9a27470238a548bdf05c5a5f7b80c648604bb62"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"步长矩消失及条件期望记号","source_units":[{"start":40438,"end":40904,"start_line":552,"end_line":556,"raw_tex":"    Since $\\beta_{\\min} > 0$, it follows that\n    \\begin{equation}\\label{zero_limit}\n        \\lim_{k\\to\\infty}{\\E\\left[\\|x_{k+1} - x_{k}\\|^2\\right]}=0 \\quad \\text{ and } \\quad \\lim_{k\\to\\infty}{\\E\\left[\\|x_{k+1} - x_{k}\\|\\right]}=0, \n    \\end{equation}\n    where the limit on the right-hand side of \\eqref{zero_limit} follows from Jensen's inequality. For simplicity, we denote $\\E_k[\\cdot] =  \\E[\\cdot|x_k]$.  On the other, using \\ref{bound_subgradient}, we deduce\n","sha256_utf8":"7fe96aa6f75e0048be5e37975a2151d238feaf6369b787fd1b829d4f8da15fee"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"作者的全次梯度条件期望估计","source_units":[{"start":40904,"end":42216,"start_line":557,"end_line":582,"raw_tex":"      \\allowdisplaybreaks\n    \\begin{equation}\n        \\label{subgrad_bound_asymp}\n        \\begin{split}\n            &\\dist(0, \\partial\\varphi(x_{k+1}))\n            \\leq\n            \\sum_{i=1}^{n_{\\text{block}}}{\\dist \\left(0, \\partial_i\\varphi(x_{k+1})\\right)}\n            \\\\\n            &\n            \\leq\n            \\frac{1}{p_{\\min}}\\sum_{i=1}^{n_{\\text{block}}}{p_i \\; \\dist \\left(0, \\partial_{i_k}\\varphi(x_{k+1})\\right)}\n            \\\\\n            &\n            = \\frac1p_{\\min}  \\E_{k}\\left[\\dist \\left(0, \\partial_{i_k}\\varphi(x_{k+1})\\right)\\right]\n            \\\\\n            &\n            \\overset{\\eqref{bounde_gradient1}}{\\leq }\n            \\frac1p_{\\min}\\E_{k}\\left[  \\left( L^f_{i_k} + L^F_{i_k} \\sqrt{2L_h} \\sqrt{ \\left(\\varphi(x_0) - \\varphi^*\\right)} + \\beta_{k+1}\\right)  \\|\\Delta x_{k+1}\\| \\right.\\\\\n            \\MoveEqLeft[-1] \\left. + \\frac{L_h M^{F}_{i_k}L^{F}_{i_k}}{2} \\|\\Delta x_{k+1}\\|^2 \\right]\n            \\\\\n            &\n            \\overset{\\eqref{bar_gamma}}{\\leq}\n            \\frac{1}{p_{\\min}} \\left( L^f + L^F \\sqrt{2L_h} \\sqrt{ \\left(\\varphi(x_0) - \\varphi^*\\right)} + \\bar{\\beta} \\right)  \\E_{k}\\left[\\|\\Delta x_{k+1}\\|\\right]\\\\\n             \\MoveEqLeft[-1] + \\frac{L_h M^F L^{F}}{2p_{\\min}} \\E_{k}\\left[\\|\\Delta x_{k+1}\\|^2 \\right].\n        \\end{split}\n    \\end{equation}\n","sha256_utf8":"cdc4beb0ef601a3556f5639d321e73b8dcfed069ba8a18a9a4ab9b979d8edce5"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"取总期望并令迭代趋于无穷","source_units":[{"start":42216,"end":43132,"start_line":583,"end_line":593,"raw_tex":"  Taking expectation in \\eqref{subgrad_bound_asymp}, we obtain\n    \\begin{align*}\n            \\E\\left[\\dist(0, \\partial\\varphi(x_{k+1}))\\right] \n            %&\\leq             \\frac{1}{p_{\\min}} \\left( L^f + L^F \\sqrt{2L_h} \\sqrt{ \\left(\\varphi(x_0) - \\varphi^*\\right)} + \\bar{\\beta} \\right)  \\E\\left[\\E_{k}\\left[\\|\\Delta x_{k+1}\\|\\right]\\right] +  \\frac{L_h M^F L^{F}}{2p_{\\min}} \\E\\left[\\E_{k}\\left[\\|\\Delta x_{k+1}\\|^2 \\right]\\right] \\\\\n            &\\leq\n            \\frac{1}{p_{\\min}} \\left( L^f \\!+\\! L^F \\sqrt{2L_h} \\sqrt{ \\left(\\varphi(x_0) \\!-\\! \\varphi^*\\right)} + \\bar{\\beta} \\right)  \\E\\left[\\|\\Delta x_{k+1}\\|\\right]\\\\\n             \\MoveEqLeft[-1] +  \\frac{L_h M^F L^{F}}{2p_{\\min}} \\E\\left[\\|\\Delta x_{k+1}\\|^2 \\right]\n    \\end{align*}\n    Passing to the limit and using \\eqref{zero_limit}, we get\n      $  \\lim_{k\\to \\infty} \\E\\left[\\dist(0, \\partial\\varphi(x_{k})) \\right] = 0.$ \\hfill\\qed\n\\end{proof}","sha256_utf8":"23cd0b33ca3b29bb0fab57f60dc398caf904f1800cab2b04a8cfc146fe0a3aa9"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"确定首次满足精度的迭代","source_units":[{"start":44282,"end":44769,"start_line":612,"end_line":617,"raw_tex":"\\begin{proof}\n    From \\cref{propos:asymtotic:monotone}, we know that  $\\E\\left[\\dist(0, \\partial\\varphi(x_{k})) \\right] \\to 0$ as $k\\to \\infty$. Let $\\epsilon>0$ and let $N\\geq 0$ be the first integer such that\n    \\begin{equation}\n    \\E\\left[\\dist(0, \\partial\\varphi(x_{N+1})) \\right] \\leq \\epsilon.\n    \\end{equation}\n    Using  \\eqref{bounde_gradient1} from  \\cref{bound_subgradient}, it follows that at each iteration, $k \\in [0:N]$, we encounter one of the following two cases:\\\\\n","sha256_utf8":"7c758ad90579806098c28bee33f29648e11592aa11569c331c4fadc1b0d8aef3"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"线性步长项主导的情形","source_units":[{"start":44769,"end":45432,"start_line":618,"end_line":626,"raw_tex":"    \\noindent \\textit{Case 1: } If the following holds\n    \\[\n        \\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)\\|\\Delta x_{k+1}\\|\\geq \\frac{L_h M^F L^F}{2}\\|\\Delta x_{k+1}\\|^2,\n    \\]\n    then we get\n    \\begin{align} \\label{case1}\n        & \\dist^2\\left(0, \\partial_{i_k} \\varphi (x_{k+1}) \\right) \\leq 2 \\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2\\|\\Delta x_{k+1}\\|^2 \\nonumber\\\\\n        &\\overset{\\eqref{decrease}}{\\leq} \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{\\beta_{k+1}}\\left(\\varphi(x_{k}) - \\varphi(x_{k+1})\\right).\n    \\end{align}\n","sha256_utf8":"ffd3a577425c6b50ad9caf05a62d56389b8f4c088820bf7d3d78b1560bf83059"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"二次步长项主导的情形","source_units":[{"start":45432,"end":45947,"start_line":627,"end_line":634,"raw_tex":"    \\noindent \\textit{Case 2: } Otherwise,  the following is valid \n    \\[\n        \\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)\\|\\Delta x_{k+1}\\|< \\frac{L_h M^F L^F}{2}\\|\\Delta x_{k+1}\\|^2,\n    \\]\n    which yields\n    \\begin{equation}\\label{case2}\n        \\dist\\left(0, \\partial_{i_k} \\varphi (x_{k+1}) \\right) \\leq L_h M^F L^F \\|\\Delta x_{k+1}\\|^2\\overset{\\eqref{decrease}}{\\leq} \\frac{2 L_h M^F L^F}{\\beta_{k+1}}\\left(\\varphi(x_{k}) - \\varphi(x_{k+1})\\right).\n    \\end{equation}\n","sha256_utf8":"cb8a3ff8d645d085417a2e9c53651d75a3b3a3721182f1891d91351047e46ef7"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"划分迭代并估计第一类残差","source_units":[{"start":45947,"end":47311,"start_line":635,"end_line":660,"raw_tex":"    Define $\\mathcal{I}_1$ as the set of iterations $k \\in [0:N-1]$ at which   \\textit{Case 1} holds, and  $\\mathcal{I}_2$ as the set of iterations $k\\in [0:N-1]$ at which   \\textit{Case 2} holds. Clearly: $N=|\\mathcal{I}_1|+|\\mathcal{I}_2|+1$. We first derive an upper bound for $|\\mathcal{I}_1|$.  Let $k \\in \\mathcal{I}_1$, we have\n    \\allowdisplaybreaks\n    \\begin{equation}\n        \\label{criticality1_squared}\n        \\begin{split}\n        &\\dist^2(0, \\partial\\varphi(x_{k+1}))\n        =\n        \\sum_{i=1}^{n_{\\text{block}}}{\\dist^2 \\left(0, \\partial_i\\varphi(x_{k+1})\\right)}\n        \\\\\n        &\n        \\leq\n        \\frac{1}{p_{\\min}}\\sum_{i=1}^{n_{\\text{block}}}{p_i \\; \\dist^2 \\left(0, \\partial_i\\varphi(x_{k+1})\\right)}\n        \\\\\n        &\n        = \\frac1p_{\\min}  \\E_{k}\\left[\\dist^2 \\left(0, \\partial_{i_k}\\varphi(x_{k+1})\\right)\\right]\n        \\\\\n        &\n        \\overset{\\eqref{case1} }{\\leq }\n        \\frac1p_{\\min}  \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{\\beta_{k+1}}      \\E_{k} \\left[\\varphi(x_{k}) - \\varphi(x_{k+1}) \\right]\n        \\\\\n        &\n        \\leq\n        \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}      \n        \\left(\\varphi(x_k) - \\E_{k} [\\varphi(x_{k+1})]\\right).\n        \\end{split}\n    \\end{equation}\n","sha256_utf8":"c391a2c5bcd15ed13c14afea7b65c243dc2396569d37e601580a7111449f6d88"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"第一类迭代的期望求和","source_units":[{"start":47311,"end":48742,"start_line":661,"end_line":672,"raw_tex":"    Taking the expectation in the inequality \\eqref{criticality1_squared} and summing it over $\\mathcal{I}_1$ yields\n    \\begin{align*}\n        |\\mathcal{I}_1|\\epsilon^2  & < \\sum_{k\\in\\mathcal{I}_1} \\left(\\E\\left[\\dist(0, \\partial\\varphi(x_{k+1}))\\right]\\right)^2\\leq \\sum_{k\\in\\mathcal{I}_1} \\E\\left[\\dist^2(0, \\partial\\varphi(x_{k+1}))\\right]  \\\\\n        & \\leq  \\sum_{k\\in\\mathcal{I}_1}  \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}      \n        \\E\\left[\\varphi(x_k) - \\E_{k} [\\varphi(x_{k+1})]\\right] \\\\\n        & = \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}   \\sum_{k\\in\\mathcal{I}_1}      \n        \\left(\\E\\left[\\varphi(x_k)\\right] - \\E [\\varphi(x_{k+1})]\\right)\\\\\n        & \\overset{\\eqref{decrease}}{\\leq}  \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}   \\sum_{k=0}^{N-1}      \n        \\left(\\E\\left[\\varphi(x_k)\\right] - \\E [\\varphi(x_{k+1})]\\right)  \\\\\n        & = \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}  \\left(\\E[\\varphi(x_0)] - \\E[\\varphi(x_N)]\\right)  \\\\ \n        & \\leq \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min}}  \\left(\\varphi(x_0) - \\varphi^*\\right) .\n    \\end{align*}\n","sha256_utf8":"6a06f44b55318d15f12c38b9704406f38a6b197ab4762c9efa5a6de9bf7c2955"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"第一类计数及第二类残差估计","source_units":[{"start":48742,"end":49890,"start_line":673,"end_line":699,"raw_tex":"    Thus, we have: $|\\mathcal{I}_1|< \\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min} \\epsilon^2}  \\left(\\varphi(x_0) - \\varphi^*\\right)$. Similarly, we derive an upper bound for $|\\mathcal{I}_2|$. Let $k\\in \\mathcal{I}_2$, we have\n    \\allowdisplaybreaks\n    \\begin{equation}\n        \\label{criticality1}\n        \\begin{split}\n        \\dist(0, \\partial\\varphi(x_{k+1}))\n        &\n        \\leq\n        \\sum_{i=1}^{n_{\\text{block}}}{\\dist(0, \\partial_i\\varphi(x_{k+1}))}\n        \\\\\n        &\n        \\leq\n        \\frac{1}{p_{\\min}}\\sum_{i=1}^{n_{\\text{block}}}{p_i \\; \\dist(0, \\partial\\varphi_i(x_{k+1}))}\n        \\\\\n        &\n        = \\frac1p_{\\min}  \\E_{k}\\left[\\dist(0, \\partial\\varphi_{i_k}(x_{k+1}))\\right]\n        \\\\\n        &\n        \\overset{\\eqref{case2} }{\\leq }\n        \\frac1p_{\\min}  \\frac{2 L_h M^F L^F}{\\beta_{k+1}}      \\E_{k} \\left[\\varphi(x_{k}) - \\varphi(x_{k+1}) \\right]\n        \\\\\n        &\n        \\leq\n         \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}}      \n        \\left(\\varphi(x_k) - \\E_{k} [\\varphi(x_{k+1})]\\right).\n        \\end{split}\n    \\end{equation}\n","sha256_utf8":"fa046af7b58d400b4f7a4bda6e562694b40c6ab1ce5efb20804d7a1f22ca1564"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"第二类迭代的期望求和","source_units":[{"start":49890,"end":50821,"start_line":700,"end_line":706,"raw_tex":"    Taking the expectation in the inequality \\eqref{criticality1} and summing it over $\\mathcal{I}_2$ yields\n    \\begin{align*}\n        &  |\\mathcal{I}_2|\\epsilon  < \\sum_{k\\in\\mathcal{I}_2} \\E\\left[\\dist(0, \\partial\\varphi(x_{k+1}))\\right]  \\leq  \\sum_{k\\in\\mathcal{I}_2} \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}} \\E\\left[\\varphi(x_k) - \\E_{k} [\\varphi(x_{k+1})]\\right]\\\\\n        &  = \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}} \\sum_{k\\in\\mathcal{I}_2}  \\left(\\E\\left[\\varphi(x_k)\\right] - \\E [\\varphi(x_{k+1})]\\right)\\\\\n        & \\overset{\\eqref{decrease}}{\\leq}  \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}} \\sum_{k=0}^{N-1}  \\left(\\E\\left[\\varphi(x_k)\\right] - \\E [\\varphi(x_{k+1})]\\right)\\\\\n        & \\leq  \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}}   \\left(\\E\\left[\\varphi(x_0)\\right] - \\E [\\varphi(x_{N+1})]\\right)\\leq \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min}}   \\left(\\varphi(x_0) - \\varphi^*\\right).\n    \\end{align*}\n","sha256_utf8":"7877302b6417ce761dee9f3f633a6bd2d560d7ef8748eee20a5f18ed2339a875"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"合并两类计数得到复杂度","source_units":[{"start":50821,"end":51300,"start_line":707,"end_line":712,"raw_tex":"    Therefore, we obtain: $|\\mathcal{I}_2|< \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min} \\epsilon}   \\left(\\varphi(x_0) - \\varphi^*\\right) $.  Consequently, we obtain\n    \\[\n        N \\leq  \\left(\\frac{4\\left(L^f+ L^F \\sqrt{2L_h} \\sqrt{\\varphi(x_0)- \\varphi^*} + \\bar{\\beta}\\right)^2}{p_{\\min} \\beta_{\\min} \\epsilon^2}  + \\frac{2 L_h M^F L^F}{p_{\\min} \\beta_{\\min} \\epsilon} \\right)\\left(\\varphi(x_0) - \\varphi^*\\right).\n        \\]\n    This proves our claim. \\hfill\\qed\n\\end{proof}","sha256_utf8":"b2de98ccaba073208b00b2bbd2d4ae05956cb303f74d238bb522706b5af733c8"}]},{"id":"S042","semantic_type":"external_fact","kind":"external_fact","title":"光滑函数误差与范数不等式","source_units":[],"declared_external":true,"declaration_text":["Lipschitz 连续梯度给出二次下降引理与一阶 Taylor 余项界；同时使用 Cauchy–Schwarz、三角不等式和 Young 不等式。"]},{"id":"S043","semantic_type":"external_fact","kind":"external_fact","title":"强凸性与最优性条件","source_units":[],"declared_external":true,"declaration_text":["若 q 为 beta 强凸函数且 s* 为其极小点，则 0 属于 partial q(s*)，且 q(s) >= q(s*) + beta ||s-s*||^2/2。"]},{"id":"S044","semantic_type":"external_fact","kind":"external_fact","title":"次微分求和与分块乘积法则","source_units":[],"declared_external":true,"declaration_text":["光滑函数 ell 与适当下半连续凸函数 g 满足 partial(ell+g)=grad ell+partial g；光滑复合函数满足链式法则，分块可分凸函数的次微分为各块次微分的 Cartesian 乘积。"]},{"id":"S045","semantic_type":"external_fact","kind":"external_fact","title":"期望运算与非负级数性质","source_units":[],"declared_external":true,"declaration_text":["使用 Jensen 不等式、条件期望塔式性质、有限和的望远镜恒等式，以及收敛非负级数的各项趋于零。"]},{"id":"S046","semantic_type":"external_fact","kind":"external_fact","title":"紧集上的导数有界性","source_units":[],"declared_external":true,"declaration_text":["连续 Jacobian 在紧集上有界，因此存在有限的分块 Jacobian 范数上界。"]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S010","edges":[{"source_id":"S001","target_id":"S005","kind":"supports"},{"source_id":"S001","target_id":"S019","kind":"supports"},{"source_id":"S001","target_id":"S024","kind":"supports"},{"source_id":"S002","target_id":"S024","kind":"supports"},{"source_id":"S002","target_id":"S032","kind":"supports"},{"source_id":"S003","target_id":"S010","kind":"supports"},{"source_id":"S004","target_id":"S014","kind":"supports"},{"source_id":"S004","target_id":"S015","kind":"supports"},{"source_id":"S004","target_id":"S016","kind":"supports"},{"source_id":"S004","target_id":"S017","kind":"supports"},{"source_id":"S004","target_id":"S025","kind":"supports"},{"source_id":"S004","target_id":"S026","kind":"supports"},{"source_id":"S004","target_id":"S030","kind":"supports"},{"source_id":"S004","target_id":"S038","kind":"supports"},{"source_id":"S004","target_id":"S040","kind":"supports"},{"source_id":"S005","target_id":"S014","kind":"supports"},{"source_id":"S005","target_id":"S019","kind":"supports"},{"source_id":"S005","target_id":"S023","kind":"supports"},{"source_id":"S006","target_id":"S014","kind":"supports"},{"source_id":"S006","target_id":"S009","kind":"supports"},{"source_id":"S006","target_id":"S023","kind":"supports"},{"source_id":"S006","target_id":"S029","kind":"supports"},{"source_id":"S006","target_id":"S032","kind":"supports"},{"source_id":"S006","target_id":"S037","kind":"supports"},{"source_id":"S006","target_id":"S039","kind":"supports"},{"source_id":"S007","target_id":"S016","kind":"supports"},{"source_id":"S007","target_id":"S025","kind":"supports"},{"source_id":"S007","target_id":"S035","kind":"supports"},{"source_id":"S007","target_id":"S036","kind":"supports"},{"source_id":"S008","target_id":"S017","kind":"supports"},{"source_id":"S008","target_id":"S009","kind":"supports"},{"source_id":"S009","target_id":"S025","kind":"supports"},{"source_id":"S009","target_id":"S032","kind":"supports"},{"source_id":"S009","target_id":"S035","kind":"supports"},{"source_id":"S009","target_id":"S036","kind":"supports"},{"source_id":"S011","target_id":"S010","kind":"supports"},{"source_id":"S012","target_id":"S009","kind":"supports"},{"source_id":"S012","target_id":"S025","kind":"supports"},{"source_id":"S012","target_id":"S029","kind":"supports"},{"source_id":"S012","target_id":"S035","kind":"supports"},{"source_id":"S012","target_id":"S036","kind":"supports"},{"source_id":"S012","target_id":"S038","kind":"supports"},{"source_id":"S012","target_id":"S040","kind":"supports"},{"source_id":"S013","target_id":"S012","kind":"supports"},{"source_id":"S014","target_id":"S017","kind":"supports"},{"source_id":"S014","target_id":"S020","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S015","target_id":"S020","kind":"supports"},{"source_id":"S016","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S015","target_id":"S013","kind":"supports"},{"source_id":"S016","target_id":"S013","kind":"supports"},{"source_id":"S017","target_id":"S013","kind":"supports"},{"source_id":"S018","target_id":"S013","kind":"supports"},{"source_id":"S019","target_id":"S013","kind":"supports"},{"source_id":"S020","target_id":"S013","kind":"supports"},{"source_id":"S021","target_id":"S032","kind":"supports"},{"source_id":"S021","target_id":"S034","kind":"supports"},{"source_id":"S021","target_id":"S035","kind":"supports"},{"source_id":"S021","target_id":"S036","kind":"supports"},{"source_id":"S022","target_id":"S021","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S024","target_id":"S022","kind":"supports"},{"source_id":"S025","target_id":"S022","kind":"supports"},{"source_id":"S026","target_id":"S022","kind":"supports"},{"source_id":"S027","target_id":"S034","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S028","kind":"supports"},{"source_id":"S031","target_id":"S028","kind":"supports"},{"source_id":"S032","target_id":"S028","kind":"supports"},{"source_id":"S033","target_id":"S028","kind":"supports"},{"source_id":"S034","target_id":"S038","kind":"supports"},{"source_id":"S034","target_id":"S040","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S036","target_id":"S039","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S041","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S034","target_id":"S011","kind":"supports"},{"source_id":"S035","target_id":"S011","kind":"supports"},{"source_id":"S036","target_id":"S011","kind":"supports"},{"source_id":"S037","target_id":"S011","kind":"supports"},{"source_id":"S038","target_id":"S011","kind":"supports"},{"source_id":"S039","target_id":"S011","kind":"supports"},{"source_id":"S040","target_id":"S011","kind":"supports"},{"source_id":"S041","target_id":"S011","kind":"supports"},{"source_id":"S042","target_id":"S015","kind":"supports"},{"source_id":"S042","target_id":"S016","kind":"supports"},{"source_id":"S042","target_id":"S017","kind":"supports"},{"source_id":"S042","target_id":"S025","kind":"supports"},{"source_id":"S042","target_id":"S026","kind":"supports"},{"source_id":"S043","target_id":"S014","kind":"supports"},{"source_id":"S043","target_id":"S023","kind":"supports"},{"source_id":"S044","target_id":"S024","kind":"supports"},{"source_id":"S044","target_id":"S032","kind":"supports"},{"source_id":"S044","target_id":"S037","kind":"supports"},{"source_id":"S044","target_id":"S039","kind":"supports"},{"source_id":"S045","target_id":"S029","kind":"supports"},{"source_id":"S045","target_id":"S030","kind":"supports"},{"source_id":"S045","target_id":"S031","kind":"supports"},{"source_id":"S045","target_id":"S033","kind":"supports"},{"source_id":"S045","target_id":"S038","kind":"supports"},{"source_id":"S045","target_id":"S040","kind":"supports"},{"source_id":"S046","target_id":"S025","kind":"supports"}]}}
{"record_id":"2508.01353v2","paper":{"title":"Linesearch-free adaptive Bregman proximal gradient for convex minimization under local relative smoothness","authors":["Hongjia Ou","Puya Latafat","Andreas Themelis"],"arxiv_id":"2508.01353","version":"v2","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-026-03101-3","abstract_url":"https://arxiv.org/abs/2508.01353v2","pdf_url":"https://arxiv.org/pdf/2508.01353v2"},"context":{"raw_tex":"\\[\\label{eq:P} \\tag{\\text{P}}%\n\t\t\t\\minimize_{x\\in\\overline{C}}\n\t\t\t\\varphi(x)\n\t\t\t\\coloneqq\n\t\t\tf(x) + g(x),\n\t\t\\]\n\n\t\twhere \\(\\overline{C}\\) denotes the closure of \\(C \\coloneqq \\interior\\dom\\kernel\\) for a proper \\(1\\)-coercive function \\(\\kernel:\\R^n\\to\\Rinf\\) of Legendre type, \\(f:\\R^n\\to\\Rinf\\), \\(g:\\R^n\\to\\Rinf\\)  are proper closed convex, while \\(f\\) is only assumed to be \\emph{locally} smooth relative to the \\emph{distance-generating function} (dgf).\n\t\tBy local smoothness relative to \\(\\kernel\\) we refer to the existence, for any compact and convex set \\(\\K\\subset\\interior\\dom\\kernel\\), of a constant \\(L_{f,\\K}^{\\kernel}\\geq0\\) such that \\(L_{f,\\K}^{\\kernel}\\kernel-f\\) is convex over \\(\\K\\); see \\cref{ass:basic}.\n\n\n\\begin{equation}\\label{eq:BPG}\n\t\t\tx^{k+1}\n\t\t=\n\t\t\t\\argmin_{w\\in\\R^n}\\left\\{\n\t\t\t\tf(x^k)+\\innprod{\\nabla f(x^k)}{w-x^k}+g(w)+\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(w,x^k)\n\t\t\t\\right\\},\n\t\t\\end{equation}\n\n\\begin{equation}\\label{eq:D}\n\t\t\t\\D_{\\kernel}(x, y)\n\t\t\\coloneqq\n\t\t\t\\begin{cases}\n\t\t\t\t\\kernel(x) - \\kernel(y) - \\innprod{\\nabla\\kernel(y)}{x - y} & \\text{if }(x,y)\\in\\dom\\kernel\\times\\interior\\dom\\kernel\n\t\t\t\\\\\n\t\t\t\t\\infty & \\text{otherwise}\n\t\t\t\\end{cases}\n\t\t\\end{equation}\n\n\t\t\tThe set of natural numbers is \\(\\N\\coloneqq\\{0,1,2,\\dots\\}\\), while \\(\\R\\), \\(\\R_{++}\\coloneqq(0,\\infty)\\), and \\(\\Rinf\\coloneqq\\R\\cup\\{\\pm\\infty\\}\\) denote the set of real, strictly positive, and extended-real numbers, respectively.\n\t\t\tFor \\(t\\in\\R\\), we define \\([t]_+\\coloneqq\\max\\{t,0\\}\\).\n\n\n\t\t\tThe \\emph{domain} and \\emph{epigraph} of an extended-real-valued function \\(h:\\R^n\\to\\Rinf\\) are the sets\n\t\t\t\\(\n\t\t\t\t\\dom h\n\t\t\t\\coloneqq\n\t\t\t\t\\{x\\in\\R^n \\mid h(x)<\\infty\\}\n\t\t\t\\)\n\t\t\tand\n\t\t\t\\(\n\t\t\t\t\\epi h\n\t\t\t\\coloneqq\n\t\t\t\t\\{(x, c)\\in\\R^n\\times\\R \\mid h(x)\\leq c\\}\n\t\t\t\\).\n\t\t\tFunction \\(h\\) is said to be: \\emph{proper} if \\(h>-\\infty\\) and \\(\\dom h\\neq\\emptyset\\); \\emph{lower semicontinuous (lsc)} if \\(\\epi h\\) is a closed subset of \\(\\R^{n+1}\\); \\emph{1-coercive} if\n\t\t\t\\(\n\t\t\t\t\\lim_{\\norm{x}\\to\\infty}\\frac{h(x)}{\\norm{x}}=\\infty\n\t\t\t\\).\n\n\t\t\tThe \\emph{conjugate} of a proper, lsc, convex function \\(h:\\R^n\\to\\Rinf\\) is the proper, lsc, convex function \\(h^*:\\R^n\\to\\Rinf\\) defined by\n\t\t\t\\(\n\t\t\t\th^*(\\xi)\n\t\t\t\\coloneqq\n\t\t\t\t\\sup_{x\\in\\R^n}\\{\\innprod{\\xi}{x}-h(x)\\}\n\t\t\t\\).\n\t\t\tThe \\emph{(convex) subdifferential} of \\(h\\) at \\(x\\in\\dom h\\) is the set\n\t\t\t\\[\n\t\t\t\t\\partial h(x)\n\t\t\t\\coloneqq\n\t\t\t\t\\{u\\in\\R^n \\mid h(x')\\geq h(x)+\\innprod{u}{x'-x} ~ \\forall x'\\in\\R^n\\},\n\t\t\t\\]\n\t\t\twhile \\(\\partial h(x)=\\emptyset\\) for \\(x\\notin\\dom h\\).\n\t\t\t\\(h\\) is differentiable at \\(x\\) iff \\(\\partial h(x)\\) is a singleton, and in this case one has that \\(\\partial f(x)=\\{\\nabla h(x)\\}\\).\n\n\n\t\t\t\tWe say that a proper, lsc, convex function \\(\\kernel:\\R^n\\to\\Rinf\\) is \\emph{of Legendre type} (or simply \\emph{Legendre}) if it is\n\t\t\t\t(i) \\emph{essentially smooth}, namely differentiable on \\(\\interior\\dom\\kernel\\neq\\emptyset\\) and such that \\(\\norm{\\nabla\\kernel(x^k)}\\to\\infty\\) whenever \\(\\interior\\dom\\kernel\\ni x^k\\to x\\in\\dom\\kernel\\setminus\\interior\\dom\\kernel\\), and\n\t\t\t\t(ii) \\emph{essentially strictly convex}, namely strictly convex on every convex subset of \\(\\dom\\partial\\kernel\\).\n\n\n\\begin{equation}\\label{eq:DD}\n\t\t\t\t\t\\DD_{\\kernel}(x,y)\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\D_{\\kernel}(x,y)+\\D_{\\kernel}(y,x)\n\t\t\t\t=\n\t\t\t\t\t\\begin{cases}\n\t\t\t\t\t\t\\innprod{\\nabla\\kernel(x)-\\nabla\\kernel(y)}{x - y} & \\hspace*{-7pt}\\text{if } x,y\\in\\interior\\dom\\kernel\n\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\infty & \\hspace*{-7pt}\\text{otherwise.}\n\t\t\t\t\t\\end{cases}\n\t\t\t\t\\end{equation}\n\n\\begin{assumption}\\label{ass:basic}%\n\t\t\tThe following hold in \\eqref{eq:P}:\n\t\t\t\\begin{enumerate}\n\t\t\t\\item \\label{ass:h}%\n\t\t\t\t\\(C=\\interior\\dom\\kernel\\) for a proper, convex, 1-coercive function \\(\\kernel\\) of Legendre type, which is twice continuously differentiable with \\(\\nabla^2\\kernel \\succ 0\\) on \\(C\\).\n\t\t\t\\item \\label{ass:f}%\n\t\t\t\t\\(f:\\R^n\\to\\Rinf\\) is proper, convex, lsc, and \\emph{locally smooth relative to \\(\\kernel\\)}:\n\t\t\t\tthat is, \\(\\dom\\kernel\\subseteq\\dom f\\), and for every convex and compact set \\(\\K\\subset C\\) there exists \\(L_{f,\\K}^{\\kernel}>0\\) such that \\(L_{f,\\K}^{\\kernel}\\kernel-f\\) is convex on \\(\\K\\).\n\t\t\t\\item \\label{ass:g}%\n\t\t\t\t\\(g:\\R^n\\to\\Rinf\\) is proper, lsc, and convex with \\(\\dom g\\cap C\\neq\\emptyset\\).\n\t\t\t\\item \\label{ass:argmin}%\n\t\t\t\tA solution exists: \\(\\argmin_{\\overline C}\\varphi\\neq\\emptyset\\).\n\t\t\t\\end{enumerate}\n\t\t\\end{assumption}\n\n\t\tThe local relative smoothness in \\cref{ass:f} is tantamount to saying that \\(f\\) is differentiable on \\(C\\),\\footnote{%\n\t\t\tSee \\cite[Prop. 3.7]{wang2024mirror} or \\cite[Prop. 2.5]{ahookhosh2021bregman}.\n\t\t}\n\t\tand that\n\n\n\\begin{equation}\\label{eq:Df<D}\n\t\t\t\\D_f(x, y)\\leq L_{f,\\K}^{\\kernel}\\D_{\\kernel}(x, y)\n\t\t\\quad\n\t\t\t\\forall x,y\\in\\K,\n\t\t\\end{equation}\n\n\\begin{assumption}[Bregman with zone \\(C\\) {\\cite[Def. 2.1]{solodov2000inexact}}]\\label{ass:zone}%\n\t\t\tThe dgf \\(\\kernel\\) satisfies the following:\n\t\t\t\\begin{enumerate}\n\t\t\t\\item \\label{ass:Dhto0}%\n\t\t\t\t\\(\\D_{\\kernel}(x, x^k) \\to 0\\) whenever \\(C \\ni x^k \\to x\\) (in particular, \\(\\dom\\kernel\\) is closed).\n\t\t\t\\item \\label{ass:xkbounded}%\n\t\t\t\t\\(\\D_{\\kernel}(x,{}\\cdot{})\\) is level bounded for any \\(x\\in\\overline{C}\\setminus C\\).\n\t\t\t\\end{enumerate}\n\t\t\\end{assumption}\n\n\\begin{equation}\\label{eq:Hk}\n\t\t\t\tH_k\\coloneqq\\nabla\\kernel-\\gamma_k\\nabla f,\n\t\t\t\\end{equation}\n\n\\begin{equation}\\label{eq:lk}\n\t\t\t\t\t\t\\ell_k\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\frac{\\DD_f(x^k,x^{k-1})}{\\DD_{\\kernel}(x^k,x^{k-1})}\n\t\t\t\t\t\\end{equation}\n\n\\begin{equation}\\label{eq:Lamk}\n\t\t\t\t\t\t\\Lambda_{k,\\delta}\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\t2\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(x^k)+\\delta\\bigl[H_k(x^k)-H_k(x^{k-1})\\bigr],\\nabla\\kernel(x^k)\\bigr)\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\\delta^2\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t\t\t}\n\t\t\t\t\t\\end{equation}\n\n\t\t\t\t\tdepending on some parameter \\(\\delta>0\\) (specified later).\n\n\n\\begin{equation}\\label{eq:ak}\n\t\t\t\t\t\t\\alpha_k\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\frac{\\D_{\\kernel}(x^k,x^{k-1})}{\\D_{\\kernel}(x^{k-1},x^k)}\n\t\t\t\t\t\\in\n\t\t\t\t\t\t(0,\\infty)\n\t\t\t\t\t\\end{equation}\n\n\\begin{equation}\n\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t=\n\t\t\t\t\\tfrac{1+\\alpha_k}{\\alpha_k}\\D_{\\kernel}(x^k,x^{k-1}).\n\t\t\t\\end{equation}\n\n\\begin{equation}\\label{eq:alpha}\n\t\t\t\t\\alpha(\\kernel)\n\t\t\t\\coloneqq\n\t\t\t\t\\inf_{\\substack{x,y\\in\\interior\\dom\\kernel\\\\x\\neq y}}\\frac{\\D_{\\kernel}(x,y)}{\\D_{\\kernel}(y,x)}.\n\t\t\t\\end{equation}\n\n\t\t\tBased on these three quantities, choose a stepsize \\(\\gamma_{k+1}\\) and proceed with a Bregman proximal gradient update \\eqref{eq:BPG}.\n\t\t\tWe propose the following two options, where we let\n\t\t\t\\[\n\t\t\t\t\\rho_{k+1}\n\t\t\t\\coloneqq\n\t\t\t\t\\tfrac{\\gamma_{k+1}}{\\gamma_k}\n\t\t\t\\]\n\t\t\tdenote the ratio of consecutive stepsizes (so that \\(\\gamma_{k+1}=\\rho_{k+1}\\gamma_k\\)):\n\n\t\t\t\\medskip\n\t\t\t\\par\\noindent\\phantomsection \\label{alg:BY}%\n\t\t\t\t\\parbox[t]{0.15\\linewidth}{\\bf\\BY}%\n\t\t\t\t\\hfill\n\t\t\t\t\\fbox{\\parbox[t]{0.82\\linewidth}{%\n\t\t\t\t\tset \\(\\hat\\rho_{k+1}=\\sqrt{1+\\rho_k}\\) and \\(\\delta=2\\hat\\rho_{k+1}\\), and update\n\t\t\t\t\t\\begin{equation}\\label{eq:BY:rhok*}\n\t\t\t\t\t\t\\rho_{k+1}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\min\\left\\{\n\t\t\t\t\t\t\t\\hat\\rho_{k+1}\n\t\t\t\t\t\t,\n\t\t\t\t\t\t\t\\frac{\\alpha_k}{1+\\alpha_k}\n\t\t\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\t\t1\n\t\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\t2\\hat\\rho_{k+1}\n\t\t\t\t\t\t\t\t\\left[\n\t\t\t\t\t\t\t\t\t\\Lambda_{k,\\delta}\n\t\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\t\t\t\\right]_+\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\right\\}\n\t\t\t\t\t\\end{equation}\n\t\t\t\t}}%\n\n\t\t\t\\medskip\n\t\t\t\\noindent\n\t\t\tand, in case \\(\\kernel\\) enjoys a symmetry coefficient \\(\\alpha=\\alpha(\\kernel)>0\\),\n\t\t\t\\medskip\n\n\t\t\t\\par\\noindent\\phantomsection \\label{alg:FNEa}%\n\t\t\t\t\\parbox[t]{0.15\\linewidth}{%\n\t\t\t\t\t{\\bf\\FNEa}\n\n\t\t\t\t\t\\footnotesize\n\t\t\t\t\t(if \\(\\alpha(\\kernel)>0\\))%\n\t\t\t\t}\n\t\t\t\t\\hfill\n\t\t\t\t\\fbox{\\parbox[t]{0.82\\linewidth}{%\n\t\t\t\t\tset \\(\\hat\\rho_{k+1}=\\sqrt{\\frac{1+\\alpha}{2}+\\rho_k}\\) and \\(\\delta=\\frac{2}{1+\\alpha}\\hat\\rho_{k+1}\\), and update\n\t\t\t\t\t\\begin{equation}\\label{eq:FNEa:rhok*}\n\t\t\t\t\t\t\\rho_{k+1}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\min\\left\\{\n\t\t\t\t\t\t\t\\hat\\rho_{k+1}\n\t\t\t\t\t\t,\n\t\t\t\t\t\t\t\\frac{\\alpha}{\n\t\t\t\t\t\t\t\t2\\hat\\rho_{k+1}\n\t\t\t\t\t\t\t\t\\left[\n\t\t\t\t\t\t\t\t\t\\Lambda_{k,\\delta}\n\t\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\t\t\t\\right]_+\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\right\\}.\n\t\t\t\t\t\\end{equation}\n\t\t\t\t}}%\n\n\n\t\t\t\tWe use the convention that \\(\\frac{1}{0}=\\infty\\), and remind that \\([t]_+=\\max\\{0,t\\}\\).\n\t\t\t\tIn particular, whenever \\(\\Lambda_{k,\\delta}\\leq1-\\gamma_k\\ell_k\\) all updates reduce to \\(\\rho_{k+1}=\\hat\\rho_{k+1}\\).\n\t\t\t\tIt is implied that a starting point \\(x^0\\in C\\) should be provided, as well as two stepsizes \\(\\gamma_0,\\gamma_1>0\\) for the first two iterations.\n\n\n\t\tTo ease the subsequent discussion, we introduce some convenient notational shorthands and remind some of those already encountered.\n\t\tRelative to the iterates generated by \\eqref{eq:BPG}, for any \\(k\\in\\N\\) and \\(x\\in\\dom\\varphi\\) we denote\n\t\t\\begin{align}\n\t\t\\label{eq:Pk}\n\t\t\tP_k(x)\n\t\t\\coloneqq{} &\n\t\t\t\\varphi(x^k)-\\varphi(x)\n\t\t\\shortintertext{and}\n\t\t\\label{eq:Bk}\n\t\t\tB_{k+1}\n\t\t\\coloneqq{} &\n\t\t\t\\rho_{k+1}\n\t\t\t\\innprod{x^{k+1}-x^k}{H_k(x^k)-H_k(x^{k-1})},\n\t\t\\end{align}\n\t\twhere \\(\\rho_{k+1}=\\tfrac{\\gamma_{k+1}}{\\gamma_k}\\) and\n\t\t\\begin{equation}\\label{eq:H:def}\n\t\t\tH_k\\coloneqq\\nabla\\kernel-\\gamma_k\\nabla f\n\t\t\\end{equation}\n\t\tis the ``forward'' operator.\n\t\tDue to convexity of \\(g\\), the BPG updates in \\eqref{eq:BPG} are characterized by the subgradient inclusion\n\t\t\\begin{subequations}\n\t\t\t\\begin{align}\n\t\t\t\\label{eq:subgrad}\n\t\t\t\t\\tilde{\\nabla}g(x^{k+1})\n\t\t\t\\coloneqq{} &\n\t\t\t\t\\tfrac{\\nabla\\kernel(x^k)-\\nabla\\kernel(x^{k+1})}{\\gamma_{k+1}}\n\t\t\t\t-\n\t\t\t\t\\nabla f(x^k)\n\t\t\t\\in\n\t\t\t\t\\partial g(x^{k+1}).\n\t\t\t\\intertext{%\n\t\t\t\tThroughout, we use the notation \\(\\tilde{\\nabla}g(x^{k+1})\\) to indicate this particular element of the subgradient of \\(g\\) along the iterates.\n\t\t\t\tSimilarly, we use\n\t\t\t}\n\t\t\t\\nonumber\n\t\t\t\t\\tilde{\\nabla}\\varphi(x^{k+1})\n\t\t\t\\coloneqq{} &\n\t\t\t\t\\tilde{\\nabla} g(x^{k+1})\n\t\t\t\t+\n\t\t\t\t\\nabla f(x^{k+1})\n\t\t\t\\\\\n\t\t\t\\label{eq:subgradvarphi}\n\t\t\t={} &\n\t\t\t\t\\tfrac{H_{k+1}(x^k)-H_{k+1}(x^{k+1})}{\\gamma_{k+1}}\n\t\t\t\\in\n\t\t\t\t\\partial\\varphi(x^{k+1}).\n\t\t\t\\end{align}\n\t\t\\end{subequations}\n\t\tIn light of these, we adopt the notation\n\t\t\\begin{gather*}\n\t\t\t\\tD_g(w,x^k)\n\t\t\\coloneqq\n\t\t\tg(w)-g(x^k)-\\innprod{\\tilde{\\nabla}g(x^k)}{w-x^k},\n\t\t\\shortintertext{and similarly}\n\t\t\t\\tD_{\\varphi}(w,x^k)\n\t\t\\coloneqq\n\t\t\t{\\varphi}(w)-{\\varphi}(x^k)-\\innprod{\\tilde{\\nabla}{\\varphi}(x^k)}{w-x^k},\n\t\t\\end{gather*}\n\t\twhich are both positive quantities for any \\(k\\in\\N\\) and \\(w\\in\\R^n\\).\n","units":[{"start":12920,"end":13030,"start_line":472,"end_line":477,"raw_tex":"\\[\\label{eq:P} \\tag{\\text{P}}%\n\t\t\t\\minimize_{x\\in\\overline{C}}\n\t\t\t\\varphi(x)\n\t\t\t\\coloneqq\n\t\t\tf(x) + g(x),\n\t\t\\]","sha256_utf8":"d87984e121d9ef8aa602df631ada174bc50121ac8e54cb4839aecc45b5b80b13"},{"start":13031,"end":13645,"start_line":478,"end_line":479,"raw_tex":"\t\twhere \\(\\overline{C}\\) denotes the closure of \\(C \\coloneqq \\interior\\dom\\kernel\\) for a proper \\(1\\)-coercive function \\(\\kernel:\\R^n\\to\\Rinf\\) of Legendre type, \\(f:\\R^n\\to\\Rinf\\), \\(g:\\R^n\\to\\Rinf\\)  are proper closed convex, while \\(f\\) is only assumed to be \\emph{locally} smooth relative to the \\emph{distance-generating function} (dgf).\n\t\tBy local smoothness relative to \\(\\kernel\\) we refer to the existence, for any compact and convex set \\(\\K\\subset\\interior\\dom\\kernel\\), of a constant \\(L_{f,\\K}^{\\kernel}\\geq0\\) such that \\(L_{f,\\K}^{\\kernel}\\kernel-f\\) is convex over \\(\\K\\); see \\cref{ass:basic}.\n","sha256_utf8":"355a3ad8e7c572f0a97891c950f434632b64ae00b2ed3c6dff32a82107d09de3"},{"start":14365,"end":14559,"start_line":490,"end_line":496,"raw_tex":"\\begin{equation}\\label{eq:BPG}\n\t\t\tx^{k+1}\n\t\t=\n\t\t\t\\argmin_{w\\in\\R^n}\\left\\{\n\t\t\t\tf(x^k)+\\innprod{\\nabla f(x^k)}{w-x^k}+g(w)+\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(w,x^k)\n\t\t\t\\right\\},\n\t\t\\end{equation}","sha256_utf8":"b9fb09a63cd22e353fb2e948dd7af854a59fdd1910874cfaa96e65b28b239926"},{"start":14619,"end":14888,"start_line":498,"end_line":506,"raw_tex":"\\begin{equation}\\label{eq:D}\n\t\t\t\\D_{\\kernel}(x, y)\n\t\t\\coloneqq\n\t\t\t\\begin{cases}\n\t\t\t\t\\kernel(x) - \\kernel(y) - \\innprod{\\nabla\\kernel(y)}{x - y} & \\text{if }(x,y)\\in\\dom\\kernel\\times\\interior\\dom\\kernel\n\t\t\t\\\\\n\t\t\t\t\\infty & \\text{otherwise}\n\t\t\t\\end{cases}\n\t\t\\end{equation}","sha256_utf8":"bdc0f8612a83857a570e0b3ca397ce9f87485b640d83267ba72757ffd75495dc"},{"start":22422,"end":22719,"start_line":561,"end_line":562,"raw_tex":"\t\t\tThe set of natural numbers is \\(\\N\\coloneqq\\{0,1,2,\\dots\\}\\), while \\(\\R\\), \\(\\R_{++}\\coloneqq(0,\\infty)\\), and \\(\\Rinf\\coloneqq\\R\\cup\\{\\pm\\infty\\}\\) denote the set of real, strictly positive, and extended-real numbers, respectively.\n\t\t\tFor \\(t\\in\\R\\), we define \\([t]_+\\coloneqq\\max\\{t,0\\}\\).\n","sha256_utf8":"9339900c415443c4e141bdcd80acd2a09a45869d25ca0b6429050ce1fad2fac6"},{"start":23267,"end":24429,"start_line":571,"end_line":601,"raw_tex":"\t\t\tThe \\emph{domain} and \\emph{epigraph} of an extended-real-valued function \\(h:\\R^n\\to\\Rinf\\) are the sets\n\t\t\t\\(\n\t\t\t\t\\dom h\n\t\t\t\\coloneqq\n\t\t\t\t\\{x\\in\\R^n \\mid h(x)<\\infty\\}\n\t\t\t\\)\n\t\t\tand\n\t\t\t\\(\n\t\t\t\t\\epi h\n\t\t\t\\coloneqq\n\t\t\t\t\\{(x, c)\\in\\R^n\\times\\R \\mid h(x)\\leq c\\}\n\t\t\t\\).\n\t\t\tFunction \\(h\\) is said to be: \\emph{proper} if \\(h>-\\infty\\) and \\(\\dom h\\neq\\emptyset\\); \\emph{lower semicontinuous (lsc)} if \\(\\epi h\\) is a closed subset of \\(\\R^{n+1}\\); \\emph{1-coercive} if\n\t\t\t\\(\n\t\t\t\t\\lim_{\\norm{x}\\to\\infty}\\frac{h(x)}{\\norm{x}}=\\infty\n\t\t\t\\).\n\n\t\t\tThe \\emph{conjugate} of a proper, lsc, convex function \\(h:\\R^n\\to\\Rinf\\) is the proper, lsc, convex function \\(h^*:\\R^n\\to\\Rinf\\) defined by\n\t\t\t\\(\n\t\t\t\th^*(\\xi)\n\t\t\t\\coloneqq\n\t\t\t\t\\sup_{x\\in\\R^n}\\{\\innprod{\\xi}{x}-h(x)\\}\n\t\t\t\\).\n\t\t\tThe \\emph{(convex) subdifferential} of \\(h\\) at \\(x\\in\\dom h\\) is the set\n\t\t\t\\[\n\t\t\t\t\\partial h(x)\n\t\t\t\\coloneqq\n\t\t\t\t\\{u\\in\\R^n \\mid h(x')\\geq h(x)+\\innprod{u}{x'-x} ~ \\forall x'\\in\\R^n\\},\n\t\t\t\\]\n\t\t\twhile \\(\\partial h(x)=\\emptyset\\) for \\(x\\notin\\dom h\\).\n\t\t\t\\(h\\) is differentiable at \\(x\\) iff \\(\\partial h(x)\\) is a singleton, and in this case one has that \\(\\partial f(x)=\\{\\nabla h(x)\\}\\).\n","sha256_utf8":"9f597d3ca01c632196ee9d2878be5cc84f35cb72583c1b292c9477f095c983a4"},{"start":25029,"end":25529,"start_line":621,"end_line":623,"raw_tex":"\t\t\t\tWe say that a proper, lsc, convex function \\(\\kernel:\\R^n\\to\\Rinf\\) is \\emph{of Legendre type} (or simply \\emph{Legendre}) if it is\n\t\t\t\t(i) \\emph{essentially smooth}, namely differentiable on \\(\\interior\\dom\\kernel\\neq\\emptyset\\) and such that \\(\\norm{\\nabla\\kernel(x^k)}\\to\\infty\\) whenever \\(\\interior\\dom\\kernel\\ni x^k\\to x\\in\\dom\\kernel\\setminus\\interior\\dom\\kernel\\), and\n\t\t\t\t(ii) \\emph{essentially strictly convex}, namely strictly convex on every convex subset of \\(\\dom\\partial\\kernel\\).\n","sha256_utf8":"1cec3791deef5393d2046d8801b78979ac0c76c0c4d8f09e6a848c2abbaef3fa"},{"start":27128,"end":27463,"start_line":652,"end_line":662,"raw_tex":"\\begin{equation}\\label{eq:DD}\n\t\t\t\t\t\\DD_{\\kernel}(x,y)\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\D_{\\kernel}(x,y)+\\D_{\\kernel}(y,x)\n\t\t\t\t=\n\t\t\t\t\t\\begin{cases}\n\t\t\t\t\t\t\\innprod{\\nabla\\kernel(x)-\\nabla\\kernel(y)}{x - y} & \\hspace*{-7pt}\\text{if } x,y\\in\\interior\\dom\\kernel\n\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\infty & \\hspace*{-7pt}\\text{otherwise.}\n\t\t\t\t\t\\end{cases}\n\t\t\t\t\\end{equation}","sha256_utf8":"4fc636784c945f993d9ce0d018a08f91e3e51fc51717cd5c46e82930b2ee21fa"},{"start":27715,"end":28592,"start_line":672,"end_line":685,"raw_tex":"\\begin{assumption}\\label{ass:basic}%\n\t\t\tThe following hold in \\eqref{eq:P}:\n\t\t\t\\begin{enumerate}\n\t\t\t\\item \\label{ass:h}%\n\t\t\t\t\\(C=\\interior\\dom\\kernel\\) for a proper, convex, 1-coercive function \\(\\kernel\\) of Legendre type, which is twice continuously differentiable with \\(\\nabla^2\\kernel \\succ 0\\) on \\(C\\).\n\t\t\t\\item \\label{ass:f}%\n\t\t\t\t\\(f:\\R^n\\to\\Rinf\\) is proper, convex, lsc, and \\emph{locally smooth relative to \\(\\kernel\\)}:\n\t\t\t\tthat is, \\(\\dom\\kernel\\subseteq\\dom f\\), and for every convex and compact set \\(\\K\\subset C\\) there exists \\(L_{f,\\K}^{\\kernel}>0\\) such that \\(L_{f,\\K}^{\\kernel}\\kernel-f\\) is convex on \\(\\K\\).\n\t\t\t\\item \\label{ass:g}%\n\t\t\t\t\\(g:\\R^n\\to\\Rinf\\) is proper, lsc, and convex with \\(\\dom g\\cap C\\neq\\emptyset\\).\n\t\t\t\\item \\label{ass:argmin}%\n\t\t\t\tA solution exists: \\(\\argmin_{\\overline C}\\varphi\\neq\\emptyset\\).\n\t\t\t\\end{enumerate}\n\t\t\\end{assumption}","sha256_utf8":"739b5fa0c562f4c9aece72d0d25c9fd278a6103afa18035f01ae1cf765e31f14"},{"start":28688,"end":28908,"start_line":688,"end_line":691,"raw_tex":"\t\tThe local relative smoothness in \\cref{ass:f} is tantamount to saying that \\(f\\) is differentiable on \\(C\\),\\footnote{%\n\t\t\tSee \\cite[Prop. 3.7]{wang2024mirror} or \\cite[Prop. 2.5]{ahookhosh2021bregman}.\n\t\t}\n\t\tand that\n","sha256_utf8":"9e0fac7d415041a5f2a1de7cce320ee30d4792f3c14d5b8b14ea625373319b51"},{"start":28910,"end":29042,"start_line":692,"end_line":696,"raw_tex":"\\begin{equation}\\label{eq:Df<D}\n\t\t\t\\D_f(x, y)\\leq L_{f,\\K}^{\\kernel}\\D_{\\kernel}(x, y)\n\t\t\\quad\n\t\t\t\\forall x,y\\in\\K,\n\t\t\\end{equation}","sha256_utf8":"675a9a01df75f5216625dac60c581397a4e1aa317b1e9cee2e03342f9c554b94"},{"start":31262,"end":31727,"start_line":719,"end_line":727,"raw_tex":"\\begin{assumption}[Bregman with zone \\(C\\) {\\cite[Def. 2.1]{solodov2000inexact}}]\\label{ass:zone}%\n\t\t\tThe dgf \\(\\kernel\\) satisfies the following:\n\t\t\t\\begin{enumerate}\n\t\t\t\\item \\label{ass:Dhto0}%\n\t\t\t\t\\(\\D_{\\kernel}(x, x^k) \\to 0\\) whenever \\(C \\ni x^k \\to x\\) (in particular, \\(\\dom\\kernel\\) is closed).\n\t\t\t\\item \\label{ass:xkbounded}%\n\t\t\t\t\\(\\D_{\\kernel}(x,{}\\cdot{})\\) is level bounded for any \\(x\\in\\overline{C}\\setminus C\\).\n\t\t\t\\end{enumerate}\n\t\t\\end{assumption}","sha256_utf8":"85cf2280edde4580eb05194b968fceb8eab1ddc1efe47ba8fed735e43f6c6aae"},{"start":34138,"end":34233,"start_line":775,"end_line":777,"raw_tex":"\\begin{equation}\\label{eq:Hk}\n\t\t\t\tH_k\\coloneqq\\nabla\\kernel-\\gamma_k\\nabla f,\n\t\t\t\\end{equation}","sha256_utf8":"4810118b2dbf87f840da9f60ac045f3a0d24172fd74d472fb5ff5b4344931cf7"},{"start":34584,"end":34721,"start_line":784,"end_line":788,"raw_tex":"\\begin{equation}\\label{eq:lk}\n\t\t\t\t\t\t\\ell_k\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\frac{\\DD_f(x^k,x^{k-1})}{\\DD_{\\kernel}(x^k,x^{k-1})}\n\t\t\t\t\t\\end{equation}","sha256_utf8":"e874be7e7ad474193fe1e0ce3110cd68f4df4e93018414433cc79ea2cee46e3d"},{"start":35579,"end":35854,"start_line":796,"end_line":804,"raw_tex":"\\begin{equation}\\label{eq:Lamk}\n\t\t\t\t\t\t\\Lambda_{k,\\delta}\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\t2\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(x^k)+\\delta\\bigl[H_k(x^k)-H_k(x^{k-1})\\bigr],\\nabla\\kernel(x^k)\\bigr)\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\\delta^2\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t\t\t}\n\t\t\t\t\t\\end{equation}","sha256_utf8":"82b82c81b2fdaba188e2696da8c389aba34f4aed0182105817b873fbd61868cf"},{"start":35855,"end":35920,"start_line":805,"end_line":805,"raw_tex":"\t\t\t\t\tdepending on some parameter \\(\\delta>0\\) (specified later).\n","sha256_utf8":"40b90223838e3fd82e2577b6ff9f9e9732142caf67d715df2bae03afa5964b9c"},{"start":36858,"end":37029,"start_line":823,"end_line":829,"raw_tex":"\\begin{equation}\\label{eq:ak}\n\t\t\t\t\t\t\\alpha_k\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\frac{\\D_{\\kernel}(x^k,x^{k-1})}{\\D_{\\kernel}(x^{k-1},x^k)}\n\t\t\t\t\t\\in\n\t\t\t\t\t\t(0,\\infty)\n\t\t\t\t\t\\end{equation}","sha256_utf8":"125e157f005d9705fcd1a56577d36bc38ff4401d35df71a7630469fe83a3f58b"},{"start":37081,"end":37210,"start_line":832,"end_line":836,"raw_tex":"\\begin{equation}\n\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t=\n\t\t\t\t\\tfrac{1+\\alpha_k}{\\alpha_k}\\D_{\\kernel}(x^k,x^{k-1}).\n\t\t\t\\end{equation}","sha256_utf8":"eb7e3441ac85b3935623c50fd6eb24679c7a55c2416cd5f8eeedda17882dd864"},{"start":37417,"end":37602,"start_line":838,"end_line":842,"raw_tex":"\\begin{equation}\\label{eq:alpha}\n\t\t\t\t\\alpha(\\kernel)\n\t\t\t\\coloneqq\n\t\t\t\t\\inf_{\\substack{x,y\\in\\interior\\dom\\kernel\\\\x\\neq y}}\\frac{\\D_{\\kernel}(x,y)}{\\D_{\\kernel}(y,x)}.\n\t\t\t\\end{equation}","sha256_utf8":"e1242262d4d760e284ec7e5a1546ea3592e5456d434c15b9a0b058690a3f72cd"},{"start":38292,"end":40018,"start_line":850,"end_line":917,"raw_tex":"\t\t\tBased on these three quantities, choose a stepsize \\(\\gamma_{k+1}\\) and proceed with a Bregman proximal gradient update \\eqref{eq:BPG}.\n\t\t\tWe propose the following two options, where we let\n\t\t\t\\[\n\t\t\t\t\\rho_{k+1}\n\t\t\t\\coloneqq\n\t\t\t\t\\tfrac{\\gamma_{k+1}}{\\gamma_k}\n\t\t\t\\]\n\t\t\tdenote the ratio of consecutive stepsizes (so that \\(\\gamma_{k+1}=\\rho_{k+1}\\gamma_k\\)):\n\n\t\t\t\\medskip\n\t\t\t\\par\\noindent\\phantomsection \\label{alg:BY}%\n\t\t\t\t\\parbox[t]{0.15\\linewidth}{\\bf\\BY}%\n\t\t\t\t\\hfill\n\t\t\t\t\\fbox{\\parbox[t]{0.82\\linewidth}{%\n\t\t\t\t\tset \\(\\hat\\rho_{k+1}=\\sqrt{1+\\rho_k}\\) and \\(\\delta=2\\hat\\rho_{k+1}\\), and update\n\t\t\t\t\t\\begin{equation}\\label{eq:BY:rhok*}\n\t\t\t\t\t\t\\rho_{k+1}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\min\\left\\{\n\t\t\t\t\t\t\t\\hat\\rho_{k+1}\n\t\t\t\t\t\t,\n\t\t\t\t\t\t\t\\frac{\\alpha_k}{1+\\alpha_k}\n\t\t\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\t\t1\n\t\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\t2\\hat\\rho_{k+1}\n\t\t\t\t\t\t\t\t\\left[\n\t\t\t\t\t\t\t\t\t\\Lambda_{k,\\delta}\n\t\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\t\t\t\\right]_+\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\right\\}\n\t\t\t\t\t\\end{equation}\n\t\t\t\t}}%\n\n\t\t\t\\medskip\n\t\t\t\\noindent\n\t\t\tand, in case \\(\\kernel\\) enjoys a symmetry coefficient \\(\\alpha=\\alpha(\\kernel)>0\\),\n\t\t\t\\medskip\n\n\t\t\t\\par\\noindent\\phantomsection \\label{alg:FNEa}%\n\t\t\t\t\\parbox[t]{0.15\\linewidth}{%\n\t\t\t\t\t{\\bf\\FNEa}\n\n\t\t\t\t\t\\footnotesize\n\t\t\t\t\t(if \\(\\alpha(\\kernel)>0\\))%\n\t\t\t\t}\n\t\t\t\t\\hfill\n\t\t\t\t\\fbox{\\parbox[t]{0.82\\linewidth}{%\n\t\t\t\t\tset \\(\\hat\\rho_{k+1}=\\sqrt{\\frac{1+\\alpha}{2}+\\rho_k}\\) and \\(\\delta=\\frac{2}{1+\\alpha}\\hat\\rho_{k+1}\\), and update\n\t\t\t\t\t\\begin{equation}\\label{eq:FNEa:rhok*}\n\t\t\t\t\t\t\\rho_{k+1}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\min\\left\\{\n\t\t\t\t\t\t\t\\hat\\rho_{k+1}\n\t\t\t\t\t\t,\n\t\t\t\t\t\t\t\\frac{\\alpha}{\n\t\t\t\t\t\t\t\t2\\hat\\rho_{k+1}\n\t\t\t\t\t\t\t\t\\left[\n\t\t\t\t\t\t\t\t\t\\Lambda_{k,\\delta}\n\t\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\t\t\t\\right]_+\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\right\\}.\n\t\t\t\t\t\\end{equation}\n\t\t\t\t}}%\n","sha256_utf8":"3491deec115d3f4d6feb1e959fbd820daab25db5f2efd0edffa692a0d8ce2209"},{"start":40065,"end":40435,"start_line":921,"end_line":923,"raw_tex":"\t\t\t\tWe use the convention that \\(\\frac{1}{0}=\\infty\\), and remind that \\([t]_+=\\max\\{0,t\\}\\).\n\t\t\t\tIn particular, whenever \\(\\Lambda_{k,\\delta}\\leq1-\\gamma_k\\ell_k\\) all updates reduce to \\(\\rho_{k+1}=\\hat\\rho_{k+1}\\).\n\t\t\t\tIt is implied that a starting point \\(x^0\\in C\\) should be provided, as well as two stepsizes \\(\\gamma_0,\\gamma_1>0\\) for the first two iterations.\n","sha256_utf8":"cfbba540cf9f58665a0c354ec6b08d24f717850c11ffbf5bb2097f4a8047ba23"},{"start":48285,"end":50152,"start_line":1125,"end_line":1183,"raw_tex":"\t\tTo ease the subsequent discussion, we introduce some convenient notational shorthands and remind some of those already encountered.\n\t\tRelative to the iterates generated by \\eqref{eq:BPG}, for any \\(k\\in\\N\\) and \\(x\\in\\dom\\varphi\\) we denote\n\t\t\\begin{align}\n\t\t\\label{eq:Pk}\n\t\t\tP_k(x)\n\t\t\\coloneqq{} &\n\t\t\t\\varphi(x^k)-\\varphi(x)\n\t\t\\shortintertext{and}\n\t\t\\label{eq:Bk}\n\t\t\tB_{k+1}\n\t\t\\coloneqq{} &\n\t\t\t\\rho_{k+1}\n\t\t\t\\innprod{x^{k+1}-x^k}{H_k(x^k)-H_k(x^{k-1})},\n\t\t\\end{align}\n\t\twhere \\(\\rho_{k+1}=\\tfrac{\\gamma_{k+1}}{\\gamma_k}\\) and\n\t\t\\begin{equation}\\label{eq:H:def}\n\t\t\tH_k\\coloneqq\\nabla\\kernel-\\gamma_k\\nabla f\n\t\t\\end{equation}\n\t\tis the ``forward'' operator.\n\t\tDue to convexity of \\(g\\), the BPG updates in \\eqref{eq:BPG} are characterized by the subgradient inclusion\n\t\t\\begin{subequations}\n\t\t\t\\begin{align}\n\t\t\t\\label{eq:subgrad}\n\t\t\t\t\\tilde{\\nabla}g(x^{k+1})\n\t\t\t\\coloneqq{} &\n\t\t\t\t\\tfrac{\\nabla\\kernel(x^k)-\\nabla\\kernel(x^{k+1})}{\\gamma_{k+1}}\n\t\t\t\t-\n\t\t\t\t\\nabla f(x^k)\n\t\t\t\\in\n\t\t\t\t\\partial g(x^{k+1}).\n\t\t\t\\intertext{%\n\t\t\t\tThroughout, we use the notation \\(\\tilde{\\nabla}g(x^{k+1})\\) to indicate this particular element of the subgradient of \\(g\\) along the iterates.\n\t\t\t\tSimilarly, we use\n\t\t\t}\n\t\t\t\\nonumber\n\t\t\t\t\\tilde{\\nabla}\\varphi(x^{k+1})\n\t\t\t\\coloneqq{} &\n\t\t\t\t\\tilde{\\nabla} g(x^{k+1})\n\t\t\t\t+\n\t\t\t\t\\nabla f(x^{k+1})\n\t\t\t\\\\\n\t\t\t\\label{eq:subgradvarphi}\n\t\t\t={} &\n\t\t\t\t\\tfrac{H_{k+1}(x^k)-H_{k+1}(x^{k+1})}{\\gamma_{k+1}}\n\t\t\t\\in\n\t\t\t\t\\partial\\varphi(x^{k+1}).\n\t\t\t\\end{align}\n\t\t\\end{subequations}\n\t\tIn light of these, we adopt the notation\n\t\t\\begin{gather*}\n\t\t\t\\tD_g(w,x^k)\n\t\t\\coloneqq\n\t\t\tg(w)-g(x^k)-\\innprod{\\tilde{\\nabla}g(x^k)}{w-x^k},\n\t\t\\shortintertext{and similarly}\n\t\t\t\\tD_{\\varphi}(w,x^k)\n\t\t\\coloneqq\n\t\t\t{\\varphi}(w)-{\\varphi}(x^k)-\\innprod{\\tilde{\\nabla}{\\varphi}(x^k)}{w-x^k},\n\t\t\\end{gather*}\n\t\twhich are both positive quantities for any \\(k\\in\\N\\) and \\(w\\in\\R^n\\).\n","sha256_utf8":"43945b1d2feb9d034abdb6a180bd79c922bdd838bca846b17b297a4f2f80062a"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}[summary of main results]\\label{thm:main}%\n\t\t\t\t\tSuppose that \\cref{ass:basic} holds, and consider the iterates generated by \\refBY{}.\n\t\t\t\t\tThen, one always has that\n\t\t\t\t\t\\begin{equation}\\label{eq:main:inf}\n\t\t\t\t\t\t\\smash{\n\t\t\t\t\t\t\t\\inf_{k\\in\\N}\\varphi(x^k)=\\inf_{\\overline{C}}\\varphi.\n\t\t\t\t\t\t}\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\tMoreover,\n\t\t\t\t\t\\begin{enumerate}\n\t\t\t\t\t\\item \\label{thm:main:C}%\n\t\t\t\t\t\tIf \\(C\\cap\\argmin_{\\overline{C}}\\varphi\\neq \\emptyset\\) (equivalently, if \\(C\\cap\\argmin\\varphi\\neq\\emptyset\\)), then there exists \\(x^\\star\\in C\\cap \\argmin\\varphi\\) such that \\(x^k\\to x^\\star\\).\n\t\t\t\t\t\\item \\label{thm:main:zone}%\n\t\t\t\t\t\tIf \\cref{ass:zone} holds, then \\((x^k)_{k\\in\\N}\\) is bounded and admits a unique optimal limit point.\n\t\t\t\t\t\\end{enumerate}\n\t\t\t\t\tWhen \\(\\kernel\\) has symmetry coefficient \\(\\alpha>0\\), the same is true for \\refFNEa.\n\t\t\t\t\\end{theorem}","units":[{"start":40790,"end":41658,"start_line":932,"end_line":948,"raw_tex":"\\begin{theorem}[summary of main results]\\label{thm:main}%\n\t\t\t\t\tSuppose that \\cref{ass:basic} holds, and consider the iterates generated by \\refBY{}.\n\t\t\t\t\tThen, one always has that\n\t\t\t\t\t\\begin{equation}\\label{eq:main:inf}\n\t\t\t\t\t\t\\smash{\n\t\t\t\t\t\t\t\\inf_{k\\in\\N}\\varphi(x^k)=\\inf_{\\overline{C}}\\varphi.\n\t\t\t\t\t\t}\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\tMoreover,\n\t\t\t\t\t\\begin{enumerate}\n\t\t\t\t\t\\item \\label{thm:main:C}%\n\t\t\t\t\t\tIf \\(C\\cap\\argmin_{\\overline{C}}\\varphi\\neq \\emptyset\\) (equivalently, if \\(C\\cap\\argmin\\varphi\\neq\\emptyset\\)), then there exists \\(x^\\star\\in C\\cap \\argmin\\varphi\\) such that \\(x^k\\to x^\\star\\).\n\t\t\t\t\t\\item \\label{thm:main:zone}%\n\t\t\t\t\t\tIf \\cref{ass:zone} holds, then \\((x^k)_{k\\in\\N}\\) is bounded and admits a unique optimal limit point.\n\t\t\t\t\t\\end{enumerate}\n\t\t\t\t\tWhen \\(\\kernel\\) has symmetry coefficient \\(\\alpha>0\\), the same is true for \\refFNEa.\n\t\t\t\t\\end{theorem}","sha256_utf8":"27ef109641b935d3bdc3c90221d37a8ddb8eb17fd43bc47d3924f760723a3edf"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{fact}[three-point identity {\\cite[Lem. 3.1]{chen1993convergence}}]\\label{thm:3p}%\n\t\t\t\t\tLet \\(\\kernel:\\R^n\\to\\Rinf\\) be a proper lsc convex function differentiable on \\(\\interior\\dom\\kernel\\).\n\t\t\t\t\tFor any \\(x\\in\\dom\\kernel\\), and \\(y,z\\in\\interior\\dom\\kernel\\) the following holds:\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\D_{\\kernel}(x,z)\n\t\t\t\t\t=\n\t\t\t\t\t\t\\D_{\\kernel}(x,y)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\D_{\\kernel}(y,z)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{x-y}{\\nabla\\kernel(y)-\\nabla\\kernel(z)}.\n\t\t\t\t\t\\]\n\t\t\t\t\\end{fact}\n\n\\begin{fact}\\label{fact:D}%\n\t\t\t\t\tLet \\(\\kernel:\\R^n\\to\\Rinf\\) be of Legendre type, and let \\(C\\coloneqq\\interior\\dom\\kernel\\).\n\t\t\t\t\t\\begin{enumerate}\n\t\t\t\t\t\\item \\label{fact:D:coercive}%\n\t\t\t\t\t\t\t{\\upshape \\cite[Prop. 2.16]{bauschke1997legendre}, \\cite[Thm. 3.7(vi)]{bauschke1997legendre}}\n\t\t\t\t\t\t\tIf \\(\\kernel\\) is 1-coercive, then \\(\\dom\\kernel^*=\\R^n\\) and \\(\\D_{\\kernel}(x,{}\\cdot{})\\) is coercive for any \\(x\\in C\\).\n\n\t\t\t\t\t\\item \\label{fact:D:boundary}%\n\t\t\t\t\t\t{\\upshape \\cite[Thm. 3.8(i)]{bauschke1997legendre}}\n\t\t\t\t\t\tIf a sequence \\((x^k)_{k\\in\\N}\\) converges to a boundary point of \\(C\\), then \\(\\D_{\\kernel}(x,x^k)\\to\\infty\\) for any \\(x\\in C\\).\n\n\t\t\t\t\t\\item \\label{fact:D:solodov}%\n\t\t\t\t\t\t{\\upshape \\cite[Thm. 2.4]{solodov2000inexact}}\n\t\t\t\t\t\tSuppose that \\(\\dom\\kernel\\) is closed and that \\(\\D_{\\kernel}(x^k,y^k)\\to0\\).\n\t\t\t\t\t\tIf either \\((x^k)_{k\\in\\N}\\) or \\((y^k)_{k\\in\\N}\\) converges, then the other sequence too converges to the same limit.\n\n\t\t\t\t\t\\item \\label{fact:D:Dphi*}%\n\t\t\t\t\t\t{\\upshape \\cite[Thm. 26.5]{rockafellar1970convex}}\n\t\t\t\t\t\tThe conjugate \\(\\kernel^*\\) is continuously differentiable and strictly convex on \\(\\interior\\dom\\kernel^*\\), with \\(\\nabla\\kernel^* = \\nabla\\kernel^{-1}:\\interior\\dom\\kernel^*\\to\\interior\\dom\\kernel\\).\n\n\t\t\t\t\t\\item \\label{fact:D:DtoD*}%\n\t\t\t\t\t\t{\\upshape \\cite[Thm. 3.7(v)]{bauschke1997legendre}}\n\t\t\t\t\t\t\\(\\D_{\\kernel}(x,y) = \\D_{\\kernel^*}(\\nabla \\kernel(y), \\nabla \\kernel(x))\\) for any \\(x,y\\in C\\).\n\t\t\t\t\t\\end{enumerate}\n\t\t\t\t\\end{fact}\n\n\\begin{lemma}[main identity]\\label{thm:eq}%\n\t\t\tSuppose that \\cref{ass:basic} holds, and starting from \\(x^0\\in C\\) consider Bregman proximal gradient iterations \\eqref{eq:BPG} with stepsizes \\(\\gamma_k>0\\).\n\t\t\tThen, for any \\(x\\in\\dom\\varphi\\cap\\dom\\kernel\\), \\(\\vartheta_{k+1}\\geq0\\), \\(k\\in\\N\\), it holds that\n\t\t\t\\begin{align*}\n\t\t\t&\n\t\t\t\t\\D_{\\kernel}(x,x^{k+1})\n\t\t\t\t+\n\t\t\t\t\\gamma_{k+1}(1+\\vartheta_{k+1})P_k(x)\n\t\t\t\t+\n\t\t\t\t\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\\\\\n\t\t\t={} &\n\t\t\t\t\\D_{\\kernel}(x,x^k)\n\t\t\t\t+\n\t\t\t\t\\gamma_{k+1}\\vartheta_{k+1} P_{k-1}(x)\n\t\t\t\t-\n\t\t\t\t\\rho_{k+1}\\vartheta_{k+1}(1-\\gamma_k\\ell_k)\n\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t+\n\t\t\t\tB_{k+1}\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\gamma_{k+1}\n\t\t\t\t\\Bigl\\{\n\t\t\t\t\t\\D_f(x,x^k)\n\t\t\t\t\t+\n\t\t\t\t\t\\tD_g(x^{k+1},x^k)\n\t\t\t\t\t+\n\t\t\t\t\t\\tD_g(x,x^{k+1})\n\t\t\t\t\t+\n\t\t\t\t\t\\vartheta_{k+1}\\tD_\\varphi(x^{k-1},x^k)\n\t\t\t\t\\Bigr\\},\n\t\t\t\\end{align*}\n\t\t\twhere \\(P_k(x)\\), \\(B_{k+1}\\), and \\(\\ell_k\\) are as in \\eqref{eq:Pk}, \\eqref{eq:Bk}, and \\eqref{eq:lk}.\n\t\t\tIn particular, denoting\n\t\t\t\\[\n\t\t\t\t\\widehat{\\U}_k(x)\n\t\t\t\\coloneqq\n\t\t\t\t\\D_{\\kernel}(x,x^k)\n\t\t\t\t+\n\t\t\t\t\\gamma_k(1+\\vartheta_k)P_{k-1}(x)\n\t\t\t\t+\n\t\t\t\t\\D_{\\kernel}(x^k,x^{k-1}),\n\t\t\t\\]\n\t\t\tone has that\n\t\t\t\\begin{align}\n\t\t\t\\nonumber\n\t\t\t\t\\widehat{\\U}_{k+1}(x)\n\t\t\t\\leq{} &\n\t\t\t\t\\widehat{\\U}_k(x)\n\t\t\t\t-\n\t\t\t\t\\gamma_k(1+\\vartheta_k-\\rho_{k+1}\\vartheta_{k+1})P_{k-1}(x)\n\t\t\t\t+\n\t\t\t\tB_{k+1}\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\left[\n\t\t\t\t\t\\tfrac{\\alpha_k}{1+\\alpha_k}\n\t\t\t\t\t+\n\t\t\t\t\t\\rho_{k+1}\\vartheta_{k+1}(1-\\gamma_k\\ell_k)\n\t\t\t\t\\right]\n\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}).\n\t\t\t\\label{eq:ineq0}\n\t\t\t\\end{align}\n\t\t\\end{lemma}\n\n\\begin{proof}\n\t\t\tSee \\cref{proof:thm:eq}.\n\t\t\\end{proof}\n\n\\begin{lemma}[Young's inequality in the Bregman sense]\\label{thm:BY}%\n\t\t\t\tLet \\(\\kernel:\\R^n\\to\\Rinf\\) be Legendre (not necessarily 1-coercive).\n\t\t\t\tThen, for any \\(x\\in\\dom\\kernel\\), \\(y\\in\\interior\\dom\\kernel\\), \\(v\\in\\R^n\\), and \\(\\delta>0\\) one has\n\t\t\t\t\\begin{equation}\\label{eq:BY}\n\t\t\t\t\t\\innprod{x-y}{v}\n\t\t\t\t\\leq\n\t\t\t\t\t\\tfrac{1}{\\delta}\n\t\t\t\t\t\\D_{\\kernel}(x,y)\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{1}{\\delta}\n\t\t\t\t\t\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(y)+\\delta v,\\nabla\\kernel(y)\\bigr).\n\t\t\t\t\\end{equation}\n\t\t\t\\end{lemma}\n\n\\begin{proof}\n\t\t\t\tWithout loss of generality we may assume that \\(\\nabla\\kernel(y)+\\delta v\\in\\dom\\kernel^*\\), for otherwise the right-hand side of \\eqref{eq:BY} is infinite and the inequality trivializes.\n\t\t\t\tSince \\(\\nabla\\kernel(y)\\in\\interior\\dom\\kernel^*\\) by \\cref{fact:D:Dphi*}, for any \\(t\\in(0,1)\\) one has that\n\t\t\t\t\\(\n\t\t\t\t\t\\nabla\\kernel(y)+t\\delta v\\in\\interior\\dom\\kernel^*\n\t\t\t\t\\)\n\t\t\t\tby \\cite[Prop. 3.44]{bauschke2017convex}.\n\t\t\t\tWe have\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\innprod{x-y}{v}\n\t\t\t\t={} &\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\innprod{x-y}{t\\delta v}\n\t\t\t\t\\\\\n\t\t\t\t={} &\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\innprod{x-y}{\\nabla\\kernel(\\nabla\\kernel^*\\bigl(\\nabla\\kernel(y)+t\\delta v)\\bigr)-\\nabla\\kernel(y)}\n\t\t\t\t\\\\\n\t\t\t\t={} &\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\D_{\\kernel}(x,y)\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\D_{\\kernel}\\bigl(y,\\nabla\\kernel^*\\bigl(\\nabla\\kernel(y)+t\\delta v\\bigr)\\bigr)\n\t\t\t\t\\\\\n\t\t\t\t&\n\t\t\t\t\t-\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\D_{\\kernel}\\bigl(x,\\nabla\\kernel^*\\bigl(\\nabla\\kernel(y)+t\\delta v\\bigr)\\bigr)\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\D_{\\kernel}(x,y)\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(y)+t\\delta v,\\nabla\\kernel(y)\\bigr).\n\t\t\t\\end{align*}\n\t\t\t\tThe second equality follows from Legendreness of \\(\\kernel\\) via \\cref{fact:D:Dphi*}, and the third one is the three point identity of \\cref{thm:3p}.\n\t\t\t\tThe inequality follows from \\cref{fact:D:DtoD*} and the fact that \\(\\D_\\kernel\\geq0\\).\n\t\t\t\tBy taking the limit as \\(t\\to1^-\\) and appealing to \\cite[Thm. 7.5]{rockafellar1970convex}, the claimed inequality \\eqref{eq:BY} is obtained.\n\t\t\t\\end{proof}\n\n\t\t\t\tA direct application of Young's inequality of \\cref{thm:BY} allows us to bound the inner product \\(B_{k+1}\\) as\n\t\t\t\t\\begin{align}\n\t\t\t\t\\nonumber\n\t\t\t\t\tB_{k+1}\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\\\\\n\t\t\t\t\\nonumber\n\t\t\t\t&\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(x^k)+\\delta_{k+1}\\bigl[H_k(x^k)-H_k(x^{k-1})\\bigr],\\nabla\\kernel(x^k)\\bigl)\n\t\t\t\t\\shortintertext{%\n\t\t\t\t\tfor any \\(\\delta_{k+1}>0\\), which in terms of the Lipschitz-like estimate \\(\\Lambda_{k,\\delta}\\) as in \\eqref{eq:Lamk} reads\n\t\t\t\t}\n\t\t\t\t={} &\n\t\t\t\t\t\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\delta_{k+1}\\rho_{k+1}}{2}\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}).\n\t\t\t\t\\label{eq:Bk:BY}\n\t\t\t\t\\end{align}\n\t\t\t\tAs we will see, the employment of this inequality combined with a specific choice of \\(\\delta_{k+1}\\) will lead to the stepsize update in the \\refBY{} variant.\n\n\n\\begin{lemma}\\label{thm:FNE}\n\t\t\t\t\tLet \\(\\kernel:\\R^n\\to\\Rinf\\) be Legendre, and \\(f,g:\\R^n\\to\\Rinf\\) be proper, lsc, and convex, with \\(f\\) differentiable on \\(C\\coloneqq\\interior\\dom\\kernel\\) and \\(\\dom g\\cap C\\neq\\emptyset\\).\n\t\t\t\t\tThen, denoting \\(H_k\\coloneqq\\nabla\\kernel-\\gamma_k\\nabla f\\), Bregman proximal gradient iterations \\eqref{eq:BPG} with stepsizes \\(\\gamma_k>0\\) and starting at some \\(x^0\\in C\\) satisfy\n\t\t\t\t\t\\[\n\t\t\t\t\t\tB_{k+1}\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\tfrac{\\gamma_{k+1}}{\\gamma_k}\n\t\t\t\t\t\t\\innprod{x^{k+1}-x^k}{H_k(x^k)-H_k(x^{k-1})}\n\t\t\t\t\t\\geq\n\t\t\t\t\t\t\\DD_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\t\\quad\n\t\t\t\t\t\t\\forall k\\geq 1.\n\t\t\t\t\t\\]\n\t\t\t\t\\end{lemma}\n\n\\begin{proof}\n\t\t\t\t\tFollows by observing that\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t&\n\t\t\t\t\t\t\\tfrac{\\gamma_{k+1}}{\\gamma_k}\n\t\t\t\t\t\t\\bigl(H_k(x^k)-H_k(x^{k-1})\\bigr)\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\nabla\\kernel(x^{k+1})-\\nabla\\kernel(x^k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\gamma_{k+1}\n\t\t\t\t\t\t\\Bigl[\n\t\t\t\t\t\t\t\\smash{\n\t\t\t\t\t\t\t\t\\overbrace{\n\t\t\t\t\t\t\t\t\t\\tfrac{H_{k+1}(x^k)-\\nabla\\kernel(x^{k+1})}{\\gamma_{k+1}}\n\t\t\t\t\t\t\t\t}^{\\tilde{\\nabla}g(x^{k+1})}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\overbrace{\n\t\t\t\t\t\t\t\t\t\\tfrac{H_k(x^{k-1})-\\nabla\\kernel(x^k)}{\\gamma_k}\n\t\t\t\t\t\t\t\t}^{\\tilde{\\nabla}g(x^k)}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\Bigr],\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\thence that\n\t\t\t\t\t\\(\n\t\t\t\t\t\tB_{k+1}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\DD_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\gamma_{k+1}\\tD_g(x^{k+1},x^k)\n\t\t\t\t\t\\geq\n\t\t\t\t\t\t\\DD_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\t\\).\n\t\t\t\t\\end{proof}\n\n\t\t\t\tWe may thus complement \\eqref{eq:Bk:BY} with a lower bound as\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\DD_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\\leq{} &\n\t\t\t\t\tB_{k+1}\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\delta_{k+1}\\rho_{k+1}}{2}\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}).\n\t\t\t\t\\end{align*}\n\t\t\t\tIf \\(\\kernel\\) has symmetry coefficient \\(\\alpha>0\\) as in \\eqref{eq:alpha}, then note that\n\t\t\t\t\\[\n\t\t\t\t\t\\DD_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\\geq\n\t\t\t\t\t(1+\\alpha)\\D_{\\kernel}(x^{k+1},x^k).\n\t\t\t\t\\]\n\t\t\t\tCombined with the previous inequality, we obtain\n\t\t\t\t\\[\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t1+\\alpha-\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\n\t\t\t\t\t\\right)\n\t\t\t\t\t\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\\leq\n\t\t\t\t\t\\tfrac{\\delta_{k+1}\\rho_{k+1}}{2}\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}),\n\t\t\t\t\\]\n\t\t\t\twhich plugged into \\eqref{eq:Bk:BY} leads to\n\t\t\t\t\\begin{equation}\n\t\t\t\t\\label{eq:Bk:FNEa}\n\t\t\t\t\tB_{k+1}\n\t\t\t\t\\leq\n\t\t\t\t\t\\tfrac{\n\t\t\t\t\t\t(1+\\alpha)\\delta_{k+1}^2\n\t\t\t\t\t}{\n\t\t\t\t\t\t(1+\\alpha)\\delta_{k+1}-\\rho_{k+1}\n\t\t\t\t\t}\n\t\t\t\t\t\\tfrac{\\rho_{k+1}}{2}\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}),\n\t\t\t\t\\end{equation}\n\t\t\t\tholding for any \\(\\delta_{k+1}\\) such that \\((1+\\alpha)\\delta_{k+1}-\\rho_{k+1}>0\\).\n\n\n\t\t\tBy bounding the inner product term \\(B_{k+1}\\) with \\eqref{eq:Bk:BY}, the inequality \\eqref{eq:ineq0} in \\cref{thm:eq} reveals that Bregman proximal gradient iterations \\eqref{eq:BPG} with arbitrary stepsizes \\(\\gamma_k>0\\) satisfy\n\t\t\t\\begin{align}\n\t\t\t\\nonumber\n\t\t\t&\n\t\t\t\t\\widehat{\\U}_{k+1}(x)\n\t\t\t\t-\n\t\t\t\t\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\n\t\t\t\t\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\\\\\n\t\t\t\\nonumber\n\t\t\t\\leq{} &\n\t\t\t\t\\widehat{\\U}_k(x)\n\t\t\t\t-\n\t\t\t\t\\tfrac{\\rho_k}{\\delta_k}\n\t\t\t\t\\D_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t-\n\t\t\t\t\\gamma_k(1+\\vartheta_k-\\rho_{k+1}\\vartheta_{k+1})P_{k-1}(x)\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\left[\n\t\t\t\t\t\\bigl(1-\\tfrac{\\rho_k}{\\delta_k}\\bigr)\n\t\t\t\t\t-\n\t\t\t\t\t\\rho_{k+1}(1+\\alpha_k)\n\t\t\t\t\t\\tfrac{\n\t\t\t\t\t\t\\frac{\\delta_{k+1}}{2}\n\t\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\vartheta_{k+1}\n\t\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\t}{\\alpha_k}\n\t\t\t\t\\right]\n\t\t\t\t\\D_{\\kernel}(x^k,x^{k-1})\n\t\t\t\\label{eq:UkBYdt:leq}\n\t\t\t\\end{align}\n\t\t\tfor any \\(x\\in\\dom\\varphi\\cap\\dom\\kernel\\), \\(\\vartheta_k\\geq0\\), and \\(\\delta_k>0\\), \\(k\\in\\N\\).\n\t\t\tImposing that the terms in brackets multiplying \\(P_{k-1}(x)\\) and \\(\\D_{\\kernel}(x^k,x^{k-1})\\) in the right-hand side of \\eqref{eq:UkBYdt:leq} are negative amounts to the following two conditions:\n\t\t\t\\begin{subequations}\\label{eq:gamk:ineq12}\n\t\t\t\t\\begin{gather}\n\t\t\t\t\\label{eq:gamk:ineq1}\n\t\t\t\t\t\\vartheta_{k+1}\\rho_{k+1}\\leq 1+\\vartheta_k\n\t\t\t\t\\shortintertext{and}\n\t\t\t\t\\label{eq:gamk:ineq2}\n\t\t\t\t\t1-\\tfrac{\\rho_k}{\\delta_k}\n\t\t\t\t\\geq\n\t\t\t\t\t\\rho_{k+1}\n\t\t\t\t\t\\tfrac{1+\\alpha_k}{\\alpha_k}\n\t\t\t\t\t\\bigl[\n\t\t\t\t\t\t\\tfrac{\\delta_{k+1}}{2}\n\t\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\vartheta_{k+1}(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\\bigr].\n\t\t\t\t\\end{gather}\n\t\t\t\\end{subequations}\n\t\t\tIn line with the analyses of \\cite{malitsky2020adaptive,latafat2024adaptive}, a convenient choice for the parameter \\(\\delta_{k+1}\\) is \\(\\delta_{k+1} = 2\\rho_{k+1}\\).\n\t\t\tHowever, this choice is not feasible in our more general setting.\n\t\t\tSpecifically, selecting \\(\\rho_{k+1}\\) to satisfy \\eqref{eq:gamk:ineq2} requires knowledge of the quantity \\(\\Lambda_{k,\\delta_{k+1}}\\), which generally depends on \\(\\delta_{k+1}\\) itself (except in the special case where \\(\\kernel\\) is quadratic).\n\t\t\tAs a result, setting \\(\\delta_{k+1}=2\\rho_{k+1}\\) would create a circular dependency between the two parameters.\n\t\t\tTo complicate things further, note that the left-hand side of \\eqref{eq:gamk:ineq2} indicates that a constraint \\(\\delta_k>\\rho_k\\) must be in place in order to ensure the existence of a \\(\\rho_{k+1}>0\\) satisfying the inequality.\n\n\n\t\t\tIn order to resolve this circular dependence, we introduce a parameter \\(\\hat\\rho_{k+1}\\) that shall provide an overestimate\n\t\t\t\\begin{equation}\\label{eq:rhokhatgeq}\n\t\t\t\t\\hat\\rho_{k+1}\\geq\\rho_{k+1}\n\t\t\t\\end{equation}\n\t\t\twhile only being based on information available at iteration \\(k\\).\n\t\t\tIts explicit value will be revealed shortly after.\n\t\t\tThen, we may conveniently select \\(\\delta_k=2\\hat\\rho_k\\) and \\(\\vartheta_k=\\hat\\rho_k\\) so that \\eqref{eq:gamk:ineq12} simplifies as\n\t\t\t\\[\n\t\t\t\t\\rho_{k+1}\\hat\\rho_{k+1}\n\t\t\t\\leq\n\t\t\t\t1+\\hat\\rho_k\n\t\t\t\\quad\\text{and}\\quad\n\t\t\t\t1-\\tfrac{\\rho_k}{2\\hat\\rho_k}\n\t\t\t\\geq\n\t\t\t\t\\rho_{k+1}\n\t\t\t\t\\hat\\rho_{k+1}\n\t\t\t\t\\tfrac{1+\\alpha_k}{\\alpha_k}\n\t\t\t\t\\bigl[\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t-\n\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\\bigr],\n\t\t\t\\]\n\t\t\tthat is,\n\t\t\t\\begin{equation}\\label{eq:BY:rhok*+}\n\t\t\t\t\\rho_{k+1}\n\t\t\t\\leq\n\t\t\t\t\\min\\left\\{\n\t\t\t\t\t\\frac{1+\\hat\\rho_k}{\\hat\\rho_{k+1}}\n\t\t\t\t\t,\\,\n\t\t\t\t\t\\frac{\\alpha_k}{1+\\alpha_k}\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t1+ \\frac{\\hat\\rho_k - \\rho_k}{\\hat\\rho_k}\n\t\t\t\t\t}{\n\t\t\t\t\t\t2\\hat\\rho_{k+1}\n\t\t\t\t\t\t\\bigl[\n\t\t\t\t\t\t\t\\Lambda_{k,2\\hat\\rho_{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\t\\bigr]_+\n\t\t\t\t\t}\n\t\t\t\t\\right\\}.\n\t\t\t\\end{equation}\n\t\t\tDue to \\eqref{eq:rhokhatgeq} the term \\(\\frac{\\hat\\rho_k - \\rho_k}{\\hat\\rho_k}\\) in the numerator of the second update is always nonnegative.\n\t\t\tUsing this, we show in the next lemma that the update rule of \\refBY{} always complies with the above bound.\n\t\t\tIn fact, while it is possible to retain the term \\(\\frac{\\hat\\rho_k - \\rho_k}{\\hat\\rho_k}\\) in the update of \\refBY{}, for the sake of a neater expression we opted to omit it at the cost of introducing slight conservatism.\n\n\n\\begin{lemma}\\label{thm:BY:leq}%\n\t\t\t\tSuppose that \\cref{ass:basic} holds, and consider the iterates generated by Bregman proximal gradient iterations \\eqref{eq:BPG} with \\(\\gamma_k\\) and \\(\\hat\\rho_k\\) selected according to \\refBY.\n\t\t\t\tThen, both \\eqref{eq:rhokhatgeq} and \\eqref{eq:BY:rhok*+} are satisfied for any \\(k\\in\\N\\).\n\n\t\t\t\tMoreover, denoting\n\t\t\t\t\\begin{equation}\\label{eq:UkBY}\n\t\t\t\t\t\\U_k(x)\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\D_{\\kernel}(x,x^k)\n\t\t\t\t\t+\n\t\t\t\t\t\\gamma_k(1+\\hat\\rho_k)P_{k-1}(x)\n\t\t\t\t\t+\n\t\t\t\t\t(1-\\tfrac{\\rho_k}{2\\hat\\rho_k})\n\t\t\t\t\t\\D_{\\kernel}(x^k,x^{k-1}),\n\t\t\t\t\\end{equation}\n\t\t\t\tone has that\n\t\t\t\t\\begin{equation}\\label{eq:UkBY:leq}\n\t\t\t\t\t\\U_{k+1}(x)\n\t\t\t\t\\leq\n\t\t\t\t\t\\U_k(x)\n\t\t\t\t\t-\n\t\t\t\t\t\\gamma_k(\n\t\t\t\t\t\t\\smash{\n\t\t\t\t\t\t\t\\overbrace{1+\\hat\\rho_k-\\rho_{k+1}\\hat\\rho_{k+1}}^{\\geq\\hat\\rho_k-\\rho_k\\geq0}\n\t\t\t\t\t\t}\n\t\t\t\t\t)P_{k-1}(x)\n\t\t\t\t\\end{equation}\n\t\t\t\tholds for any \\(x\\in\\dom\\varphi\\cap\\dom\\kernel\\) and \\(k\\in\\N\\).\n\t\t\t\\end{lemma}\n\n\\begin{proof}\n\t\t\t\tThe update \\eqref{eq:BY:rhok*} clearly ensures that \\(\\rho_{k+1}\\leq\\hat\\rho_{k+1}\\) always holds.\n\t\t\t\tNote that the second element in the minimum within \\eqref{eq:BY:rhok*+} coincides with that in \\eqref{eq:BY:rhok*};\n\t\t\t\tmoreover, since \\(\\rho_{k+1}\\leq\\hat\\rho_{k+1}\\) one has that\n\t\t\t\t\\(\n\t\t\t\t\t\\rho_{k+1}\\hat\\rho_{k+1}\n\t\t\t\t\\leq\n\t\t\t\t\t\\hat\\rho_{k+1}^2\n\t\t\t\t=\n\t\t\t\t\t1+\\rho_k\n\t\t\t\t\\leq\n\t\t\t\t\t1+\\hat\\rho_k\n\t\t\t\t\\),\n\t\t\t\taltogether confirming that the validity of \\eqref{eq:BY:rhok*+}.\n\n\t\t\t\tObserve further that the bound \\(\\hat\\rho_k\\geq\\rho_k\\) ensures that both elements in the minimum of \\eqref{eq:BY:rhok*} are strictly positive, and thus so are the generated stepsizes \\(\\gamma_k\\).\n\t\t\t\tFinally, \\eqref{eq:UkBY:leq} follows from \\eqref{eq:UkBYdt:leq} with the specified choices of \\(\\vartheta_k\\) and \\(\\delta_k\\).\n\t\t\t\\end{proof}\n\n\t\t\tIn case \\(\\kernel\\) has a symmetry coefficient \\(\\alpha>0\\), we may leverage the bound \\eqref{eq:Bk:FNEa} for \\(B_{k+1}\\).\n\t\t\tBy doing so, the inequality \\eqref{eq:ineq0} in \\cref{thm:eq} becomes\n\t\t\t\\begin{multline*}\n\t\t\t\t\\widehat{\\U}_{k+1}(x)\n\t\t\t\\leq\n\t\t\t\t\\widehat{\\U}_k(x)\n\t\t\t\t-\n\t\t\t\t\\gamma_k(1+\\vartheta_k-\\rho_{k+1}\\vartheta_{k+1})P_{k-1}(x)\n\t\t\t\\\\\n\t\t\t\t-\n\t\t\t\t\\left\\{\n\t\t\t\t\t\\tfrac{\\alpha_k}{1+\\alpha_k}\n\t\t\t\t\t-\n\t\t\t\t\t\\rho_{k+1}\n\t\t\t\t\t\\left[\n\t\t\t\t\t\t\\tfrac{\n\t\t\t\t\t\t\t(1+\\alpha)\\delta_{k+1}^2\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t(1+\\alpha)\\delta_{k+1}-\\rho_{k+1}\n\t\t\t\t\t\t}\n\t\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\vartheta_{k+1}(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\\right]\n\t\t\t\t\\right\\}\n\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}).\n\t\t\t\\end{multline*}\n\n\t\t\tOnce again, we introduce a parameter \\(\\hat\\rho_{k+1}\\) (to be specified later) that satisfies \\eqref{eq:rhokhatgeq} whilst based on information available at iteration \\(k\\).\n\t\t\tWe can conveniently set \\(\\delta_{k+1}=\\vartheta_{k+1}=\\frac{2}{1+\\alpha}\\hat\\rho_{k+1}\\), and use the bound\n\t\t\t\\(\n\t\t\t\t(1+\\alpha)\\delta_{k+1}-\\rho_{k+1}\n\t\t\t\\geq\n\t\t\t\t(1+\\alpha)\\delta_{k+1}-\\hat\\rho_{k+1}\n\t\t\t\\)\n\t\t\tto simplify the coefficient of \\(\\Lambda_{k,\\delta_{k+1}}\\).\n\t\t\tCombined with the fact that\n\t\t\t\\(\n\t\t\t\t\\tfrac{\\alpha_k}{1+\\alpha_k}\n\t\t\t\\geq\n\t\t\t\t\\tfrac{\\alpha}{1+\\alpha}\n\t\t\t\\),\n\t\t\tthe inequality simplifies as\n\t\t\t\\begin{align*}\n\t\t\t\t\\U^{\\alpha}_{k+1}(x)\n\t\t\t\\coloneqq{} &\n\t\t\t\t\\D_{\\kernel}(x,x^{k+1})\n\t\t\t\t+\n\t\t\t\t\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t+\n\t\t\t\t\\gamma_{k+1}\\left(1+\\tfrac{2}{1+\\alpha}\\hat\\rho_{k+1}\\right)P_k(x)\n\t\t\t\\\\\n\t\t\t\\leq{} &\n\t\t\t\t\\U^{\\alpha}_k(x)\n\t\t\t\t-\n\t\t\t\t\\gamma_k(1+\\tfrac{2}{1+\\alpha}\\hat\\rho_k-\\tfrac{2}{1+\\alpha}\\hat\\rho_{k+1}\\rho_{k+1})P_{k-1}(x)\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\left\\{\n\t\t\t\t\t\\tfrac{\\alpha}{1+\\alpha}\n\t\t\t\t\t-\n\t\t\t\t\t\\tfrac{2}{1+\\alpha}\n\t\t\t\t\t\\hat\\rho_{k+1}\n\t\t\t\t\t\\rho_{k+1}\n\t\t\t\t\t\\left[\n\t\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\\right]\n\t\t\t\t\\right\\}\n\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}),\n\t\t\t\\end{align*}\n\t\t\twhere \\(\\U^{\\alpha}_k(x)\\) corresponds to \\(\\widehat{\\U}_k(x)\\) as in \\cref{thm:eq} with \\(\\vartheta_k=\\tfrac{2}{1+\\alpha}\\hat\\rho_k\\).\n\t\t\tBy imposing negativity of the coefficients of \\(P_{k-1}(x)\\) and \\(\\DD_{\\kernel}(x^k,x^{k-1})\\) as done in the previous subsection, the following analogue of \\cref{thm:BY:leq} for \\refFNEa{} is derived.\n\n\n\\begin{lemma}\\label{thm:FNEa:leq}%\n\t\t\t\tAdditionally to \\cref{ass:basic}, suppose that \\(\\kernel\\) has symmetry coefficient \\(\\alpha>0\\) and consider the iterates generated by Bregman proximal gradient iterations \\eqref{eq:BPG} with \\(\\gamma_k\\) and \\(\\hat\\rho_k\\) selected according to \\refFNEa.\n\t\t\t\tThen, denoting\n\t\t\t\t\\begin{equation}\\label{eq:UkFNEa}\n\t\t\t\t\t\\U^{\\alpha}_k(x)\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\D_{\\kernel}(x,x^k)\n\t\t\t\t\t+\n\t\t\t\t\t\\D_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t\t+\n\t\t\t\t\t\\gamma_k\\left(1+\\tfrac{2}{1+\\alpha}\\hat\\rho_k\\right)P_{k-1}(x),\n\t\t\t\t\\end{equation}\n\t\t\t\tone has that\n\t\t\t\t\\begin{equation}\\label{eq:UkFNEa:leq}\n\t\t\t\t\t\\U^{\\alpha}_{k+1}(x)\n\t\t\t\t\\leq\n\t\t\t\t\t\\U^{\\alpha}_k(x)\n\t\t\t\t\t-\n\t\t\t\t\t\\gamma_k\\Bigl(\n\t\t\t\t\t\t\\smash{\n\t\t\t\t\t\t\t\\overbrace{\n\t\t\t\t\t\t\t\t1+\\tfrac{2}{1+\\alpha}\\hat\\rho_k-\\tfrac{2}{1+\\alpha}\\hat\\rho_{k+1}\\rho_{k+1}\n\t\t\t\t\t\t\t}^{\n\t\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t\\tfrac{2}{1+\\alpha}(\\hat\\rho_k-\\rho_k)\n\t\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t\\Bigr)P_{k-1}(x)\n\t\t\t\t\\end{equation}\n\t\t\t\tholds for any \\(x\\in\\dom\\varphi\\cap\\dom\\kernel\\) and \\(k\\in\\N\\).\n\t\t\t\\end{lemma}\n\n\\begin{lemma}\\label{thm:descent}%\n\t\t\tSuppose that \\cref{ass:basic} holds, and consider the iterates generated by Bregman proximal gradient iterations \\eqref{eq:BPG} with \\(\\gamma_k\\) and \\(\\hat\\rho_k\\) selected according to \\refBY.\n\t\t\tThen, with \\(\\U_k\\) as in \\eqref{eq:UkBY},\n\t\t\tthe following hold for any \\(x\\in\\dom\\kernel\\) satisfying \\(\\varphi(x)\\leq\\inf_{k\\in\\N}\\varphi(x^k)\\):\n\t\t\t\\begin{enumerate}\n\t\t\t\\item \\label{thm:SD}%\n\t\t\t\t\\(\\bigl(\\U_k(x)\\bigr)_{k\\in\\N}\\) decreases and converges to a finite value.\n\t\t\t\\item \\label{thm:Pkmin}%\n\t\t\t\t\\(P_K^{\\rm min}(x)\\leq\\frac{\\U_0(x)}{\\sum_{k=1}^{K+1}\\gamma_k}\\) for every \\(K\\geq 1\\), where \\(P_K^{\\rm min}(x)\\coloneqq\\min_{k\\leq K}P_k(x)\\).\n\t\t\t\\end{enumerate}\n\t\t\tWhen \\(\\kernel\\) has symmetry coefficient \\(\\alpha>0\\), all remains true for the updates of \\refFNEa, with \\(\\U_k\\gets\\U^{\\alpha}_k\\) as in \\eqref{eq:UkFNEa}.\n\t\t\\end{lemma}\n\n\\begin{proof}\n\t\t\tSee \\cref{proof:thm:descent}.\n\t\t\\end{proof}\n\n\\begin{lemma}\\label{thm:Lamkto1}%\n\t\t\t\tSuppose that \\cref{ass:basic} holds, and consider a sequence \\((x^k)_{k\\in\\N}\\) generated by Bregman proximal gradient iterations \\eqref{eq:BPG} with stepsizes \\(\\gamma_k>0\\).\n\t\t\t\tSuppose that \\((x^k)_{k\\in\\N}\\) is contained in a compact set \\(\\K\\subset C\\), and consider the ratio\n\t\t\t\t\\[\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t=\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t2\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(x^k)+\\delta_{k+1}\\bigl[H_k(x^k)-H_k(x^{k-1})\\bigr],\\nabla\\kernel(x^k)\\bigr)\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\delta_{k+1}^2\n\t\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t\t}\n\t\t\t\t\\]\n\t\t\t\tas in \\eqref{eq:Lamk}, where \\((\\delta_k)_{k\\in\\N}\\subset\\R_{++}\\) is a bounded sequence.\n\t\t\t\tIf \\(\\gamma_k\\to0\\), then \\(\\Lambda_{k,\\delta_{k+1}}\\to1\\).\n\t\t\t\\end{lemma}\n\n\\begin{proof}\n\t\t\t\tFor notational conciseness, let us denote \\(u^k\\coloneqq\\nabla\\kernel(x^k)-\\nabla\\kernel(x^{k-1})\\) and \\(v^k\\coloneqq\\nabla f(x^k)-\\nabla f(x^{k-1})\\), so that \\(H_k(x^k)-H_k(x^{k-1})=u^k-\\gamma_k v^k\\).\n\t\t\t\tFor any \\(k\\in\\N\\) there exists\n\t\t\t\t\\(\\xi^k\\) on the line segment between \\(\\nabla\\kernel(x^k)\\) and \\(\\nabla\\kernel(x^k)+\\delta_{k+1}(H_k(x^k)-H_k(x^{k-1}))\\)\n\t\t\t\tand\n\t\t\t\t\\(\\eta^k\\) on the line segment between \\(\\nabla\\kernel(x^k)\\) and \\(\\nabla\\kernel(x^{k-1})\\)\n\t\t\t\tsuch that\n\t\t\t\t\\begin{align}\n\t\t\t\t\\label{eq:Lamknum}\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t={} &\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t2\n\t\t\t\t\t\t\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(x^k)+\\delta_{k+1}\\bigl[H_k(x^k)-H_k(x^{k-1})\\bigr],\\nabla\\kernel(x^k)\\bigr)\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\delta_{k+1}^2\n\t\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t\t}\n\t\t\t\t\\\\\n\t\t\t\t\\nonumber\n\t\t\t\t={} &\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\xi_k)\n\t\t\t\t\t\t\t\\bigl[H_k(x^k)-H_k(x^{k-1})\\bigr]\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tH_k(x^k)-H_k(x^{k-1})\n\t\t\t\t\t\t}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\eta_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\\\\\n\t\t\t\t\\nonumber\n\t\t\t\t={} &\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\xi_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\eta_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\t-\n\t\t\t\t\t2\\gamma_k\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\xi_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tv^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\eta_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\t+\n\t\t\t\t\t\\gamma_k^2\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\xi_k)\n\t\t\t\t\t\t\tv^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tv^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\eta_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}.\n\t\t\t\t\\end{align}\n\t\t\t\tSince \\(\\kernel\\) is twice continuously differentiable with \\(\\nabla^2\\kernel\\succ0\\) on \\(C\\), and \\(\\K\\subset C\\) is compact and convex, one has that\n\t\t\t\t\\[\n\t\t\t\t\\textstyle\n\t\t\t\t\tL_{\\kernel,\\K}\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\sup_{\\K}\\norm{\\nabla^2\\kernel}\n\t\t\t\t=\n\t\t\t\t\t\\left(\\inf_{\\nabla\\kernel(\\K)}\\lambda_{\\rm min}(\\nabla^2\\kernel^*)\\right)^{-1}\n\t\t\t\t\\]\n\t\t\t\tis finite (\\(L_{\\kernel,\\K}\\) being the Lipschitz modulus of \\(\\nabla\\kernel\\) on \\(\\K\\)).\n\t\t\t\tAs such, one has that\n\t\t\t\t\\(\n\t\t\t\t\t\\innprod{\\nabla^2\\kernel^*(\\eta_k)u^k}{u^k}\n\t\t\t\t\\geq\n\t\t\t\t\tL_{\\kernel,\\K}^{-1}\\norm{u^k}^2\n\t\t\t\t\\)\n\t\t\t\tfor all \\(k\\).\n\t\t\t\tMoreover, letting \\(L_{f,\\K}^{\\kernel}\\) denote a smoothness modulus of \\(f\\) relative to \\(\\kernel\\) on \\(\\K\\), ensured to exist by \\cref{ass:f}, we infer from \\cite[Prop. 2.5(ii)]{ahookhosh2021bregman} that\n\t\t\t\t\\(\n\t\t\t\t\t\\norm{v^k}\\leq L_{f,\\K}^{\\kernel} L_{\\kernel,\\K}\\norm{u^k}\n\t\t\t\t\\)\n\t\t\t\tholds for all \\(k\\).\n\t\t\t\tTherefore,\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\abs{\\Lambda_{k,\\delta_{k+1}}-1}\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\norm{\\nabla^2\\kernel^*(\\xi_k)-\\nabla^2\\kernel^*(\\eta_k)}\n\t\t\t\t\tL_{\\kernel,\\K}\n\t\t\t\t\t+\n\t\t\t\t\t2\\gamma_k\n\t\t\t\t\tL_{f,\\K}^{\\kernel} L_{\\kernel,\\K}^2\n\t\t\t\t\t\\norm{\\nabla^2\\kernel^*(\\xi_k)}\n\t\t\t\t\\\\\n\t\t\t\t&\n\t\t\t\t\t+\n\t\t\t\t\t\\gamma_k^2\n\t\t\t\t\tL_{\\kernel,\\K}^3\n\t\t\t\t\t(L_{f,\\K}^{\\kernel})^2\n\t\t\t\t\t\\norm{\\nabla^2\\kernel^*(\\xi_k)}.\n\t\t\t\t\\end{align*}\n\t\t\t\tIf \\(\\gamma_k\\to0\\), then eventually \\(\\gamma_k<\\frac{1}{L_{f,\\K}^{\\kernel}}\\) and standard results ensure that \\((x^k)_{k\\in\\N}\\) converges to some point \\(x\\in\\K\\).\n\t\t\t\tIn this case, \\((\\xi^k)_{k\\in\\N}\\) and \\((\\eta^k)_{k\\in\\N}\\) converge to \\(\\nabla\\kernel(x)\\) (the former sequence because \\((\\delta_k)_{k\\in\\N}\\) is bounded), and by continuity of \\(\\nabla^2\\kernel^*\\) on \\(\\K\\) the right-hand side in the above inequality vanishes.\n\t\t\t\\end{proof}\n\n\\begin{lemma}\\label{lem:prox:property:optimal}%\n\t\t\t\tAdditionally to \\cref{ass:basic} suppose that \\(\\argmin_C\\varphi\\neq\\emptyset\\).\n\t\t\t\tLet a sequence \\((y^k)_{k\\in\\N}\\) contained in \\(C\\) and converging to a point \\(y^\\star\\in\\argmin_C\\varphi\\) be fixed, and for every \\(k\\in\\N\\) let\n\t\t\t\t\\begin{equation}\\label{eq:baryk}\n\t\t\t\t\t\\bar y^k\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\argmin_w\\left\\{\n\t\t\t\t\t\tg(w)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla f(y^k)}{w-y^k}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(w,y^k)\n\t\t\t\t\t\\right\\}\n\t\t\t\t\\end{equation}\n\t\t\t\twhere \\((\\gamma_k)_{k\\in\\N}\\) is a bounded sequence of strictly positive stepsizes.\n\t\t\t\tThen, \\(\\bar y^k\\to y^\\star\\) and \\(\\gamma_{k+1}(\\varphi(\\bar y^k) - \\min\\varphi)\\to0\\).\n\t\t\t\\end{lemma}\n\n\\begin{proof}\n\t\t\t\tConsider the ``left'' Bregman proximal operator \\(\\prox_{g}:\\interior\\dom\\kernel\\to\\dom\\kernel\\) defined as\n\t\t\t\t\\[\n\t\t\t\t\t\\prox_{g}(y)\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\argmin_{w\\in\\R^n}\\{g(w)+\\D_{\\kernel}(w,y)\\}\n\t\t\t\t\\]\n\t\t\t\t(note that \\(\\prox_{g}(y)\\in\\interior\\dom\\kernel\\subseteq\\dom\\kernel\\) for any \\(y\\in\\interior\\dom\\kernel\\), owing to \\cref{ass:basic}).\n\t\t\t\tThe mapping \\(\\prox_{\\gamma_{k+1}g}\\circ\\nabla\\kernel^*\\) is \\(\\nabla\\kernel\\)-firmly-nonexpansive, in the sense that\n\t\t\t\t\\begin{equation}\\label{eq:BPG:FNE}\n\t\t\t\t\t\\DD_{\\kernel}(\\bar y_1,\\bar y_2)\n\t\t\t\t\\leq\n\t\t\t\t\t\\innprod{\\eta_1-\\eta_2}{\\bar y_1-\\bar y_2}\n\t\t\t\t\\end{equation}\n\t\t\t\tholds for any \\(\\eta_i\\in\\R^2\\) and \\(\\bar y_i = \\prox_{\\gamma_{k+1} g}\\circ\\nabla\\kernel^*(\\eta_i)\\), \\(i=1,2\\);\n\t\t\t\ta proof of this fact can be found in \\cite[Lem. 4.2]{wang2022bregman} (see also \\cite[Thm. 4.9]{wang2025bregman} for its \\emph{equivalence} to convexity of \\(g\\)).\n\t\t\t\tNote that \\eqref{eq:baryk} can equivalently be written as\n\t\t\t\t\\[\n\t\t\t\t\t\\bar y^k\n\t\t\t\t=\n\t\t\t\t\t\\prox_{\\gamma_{k+1} g}\\circ\\nabla\\kernel^*(H_{k+1}(y^k)),\n\t\t\t\t\\]\n\t\t\t\twhere we remind that \\(H_{k+1}=\\nabla\\kernel-\\gamma_{k+1}\\nabla f\\).\n\t\t\t\tUsing \\(\\nabla\\kernel\\)-firm nonexpansiveness and recalling that \\(y^\\star = \\prox_{\\gamma_{k+1} g}\\circ\\nabla\\kernel^*(H_{k+1}(y^\\star))\\) we have\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\DD_{\\kernel}(\\bar y^k,y^\\star)\n\t\t\t\t\\stackrel{\\text{\\clap{\\eqref{eq:BPG:FNE}}}}{\\leq}{} &\n\t\t\t\t\t\\innprod{H_{k+1}(y^k) - H_{k+1}(y^\\star)}{\\bar y^k - y^\\star}\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\Bigl(\n\t\t\t\t\t\t\\norm{\\nabla \\kernel(y^k) - \\nabla \\kernel(y^\\star)}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\gamma_{k+1}\\norm{\\nabla f(y^k) - \\nabla f(y^\\star)}\n\t\t\t\t\t\\Bigr)\n\t\t\t\t\t\\norm{\\bar y^k - y^\\star},\n\t\t\t\t\\end{align*}\n\t\t\t\twhich vanishes as \\(k\\to\\infty\\).\n\t\t\t\tThen, by essential strict convexity of \\(\\kernel\\), it follows that \\(\\bar y^k\\to y^\\star\\).\n\t\t\t\tMoreover, by the subgradient inequality, for every \\(k \\in \\N\\) it holds that\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t0\n\t\t\t\t\\leq\n\t\t\t\t\t\\gamma_{k+1}(\\varphi(\\bar y^k) - \\min\\varphi)\n\t\t\t\t={} &\n\t\t\t\t\t\\gamma_{k+1}(f(\\bar y^k) + g(\\bar y^k) - \\min\\varphi)\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\gamma_{k+1}(f(\\bar y^k) - f(y^\\star))\n\t\t\t\t\\\\\n\t\t\t\t&\n\t\t\t\t\t-\\innprod{\\nabla \\kernel(y^k) - \\gamma_{k+1}\\nabla f(y^k) - \\nabla \\kernel(\\bar y^k)}{y^\\star - \\bar y^k}.\n\t\t\t\t\\end{align*}\n\t\t\t\tThe proof then follows from continuity of \\(f\\) and the fact that the inner product vanishes, since both \\(y^k\\) and \\(\\bar y^k\\) converge to \\(y^\\star\\).\n\t\t\t\\end{proof}\n\n\\begin{theorem}\\label{thm:xinC}%\n\t\t\t\tAdditionally to \\cref{ass:basic}, suppose that\n\t\t\t\t\\begin{equation}\\label{eq:xinC}\n\t\t\t\t\t\\text{there exists \\(x\\in C\\) such that \\(\\varphi(x)\\leq\\inf_{k\\in\\N}\\varphi(x^k)\\)}\n\t\t\t\t\\end{equation}\n\t\t\t\tholds for the iterates \\(x^k\\) generated by \\refBY.\n\t\t\t\tThen, the sequence \\((x^k)_{k\\in\\N}\\) converges to a solution \\(x^\\star\\in\\argmin_C\\varphi\\neq\\emptyset\\) and \\(\\varphi(x^k)\\to\\inf_{\\overline C}\\varphi\\).\n\n\t\t\t\tWhen \\(\\kernel\\) has symmetry coefficient \\(\\alpha>0\\), the same is true for \\refFNEa.\n\t\t\t\\end{theorem}\n\n\t\t\tThe fact that this result subsumes \\cref{thm:main:C} is obvious by observing that the validity of \\cref{thm:xinC} implies that any \\(x\\) complying with \\eqref{eq:xinC} must necessarily belong to \\(\\argmin_{\\overline C}\\varphi\\cap C\\).\n\t\t\tThe proof of \\cref{thm:xinC} will be carried out via intermediate claims.\n\t\t\tIn what follows, we consider \\(\\U_k\\) as in \\eqref{eq:UkBY} in case of \\refBY{}; under the needed symmetry assumption, the same proof applies to \\refFNEa{} as well by simply replacing \\(\\U_k\\gets\\U^{\\alpha}_k\\).\n\n\n\\begin{claim}\\label{thm:xinC:boundedseq}\n\t\t\t\tThere exists a compact set \\(\\K\\subset C\\) containing all the iterates \\(x^k\\).\n\t\t\t\\end{claim}\n\n\\begin{proof}\n\t\t\t\tIn view of \\eqref{eq:xinC}, \\cref{thm:SD} implies that\n\t\t\t\t\\[\n\t\t\t\t\t\\D_{\\kernel}(x,x^k)\n\t\t\t\t\\leq\n\t\t\t\t\t\\U_k(x)\n\t\t\t\t\\leq\n\t\t\t\t\t\\U_0(x)\n\t\t\t\t\\]\n\t\t\t\tholds for any \\(k\\in\\N\\).\n\t\t\t\tThe assertion then follows from \\cref{fact:D:coercive}.\n\t\t\t\\end{proof}\n\n\\begin{claim}\n\t\t\t\t\\(\\gamma_k\\not\\to0\\).\n\t\t\t\\end{claim}\n\n\\begin{proof}\n\t\t\t\tTo arrive to a contradiction, suppose that \\(\\gamma_k\\to0\\).\n\t\t\t\tThen, \\cref{thm:Lamkto1} implies that eventually the quantity\n\t\t\t\t\\(\n\t\t\t\t\t[\\Lambda_{k,\\delta}-(1-\\gamma_k\\ell_k)]_+\n\t\t\t\t\\)\n\t\t\t\tappearing in both \\eqref{eq:BY:rhok*} and \\eqref{eq:FNEa:rhok*} vanishes.\n\t\t\t\tSince \\((x^k)_{k\\in\\N}\\) is bounded and bounded away from the boundary of \\(C\\), regardless of whether or not \\(\\kernel\\) has a symmetry coefficient \\(\\alpha>0\\), it holds that \\(\\inf_{k\\in\\N}\\alpha_k>0\\);\n\t\t\t\tfurthermore, for both \\refBY{} and \\refFNEa{} note that \\(\\hat\\rho_{k+1}\\leq\\sqrt{1+\\rho_k}\\leq\\sqrt{1+\\hat\\rho_k}\\), implying that \\(\\hat\\rho_k\\leq\\frac{1+\\sqrt{5}}{2}\\) holds for any \\(k\\).\n\t\t\t\tAs such, in both \\eqref{eq:BY:rhok*} and \\eqref{eq:FNEa:rhok*} eventually the second element in the minimum is infinite, implying that the update reduces to \\(\\gamma_{k+1}=\\hat\\rho_{k+1}\\gamma_k\\) and is thus divergent, a contradiction.\n\t\t\t\\end{proof}\n\n\\begin{claim}\n\t\t\t\t\\(\\inf_{k\\in\\N}\\varphi(x^k)=\\inf_{\\overline C}\\varphi\\) (in particular, necessarily \\(x\\in\\argmin_C\\varphi\\) for any \\(x\\) satisfying \\eqref{eq:xinC}).\n\t\t\t\\end{claim}\n\n\\begin{proof}\n\t\t\t\tHaving shown that \\(\\gamma_k\\not\\to0\\), one has that \\(\\sum_{k\\in\\N}\\gamma_k=\\infty\\).\n\t\t\t\t\\Cref{thm:Pkmin} then implies that \\(\\inf_{k\\in\\N}\\varphi(x^k)=\\varphi(x)\\).\n\t\t\t\tFrom the arbitrariness of the point \\(x\\) as in \\eqref{eq:xinC} we conclude that necessarily \\(x\\in\\argmin_C\\varphi\\subseteq\\argmin_{\\overline C}\\varphi\\), with inclusion holding by virtue of \\cite[Prop. 11.1(iv)]{bauschke2017convex}.\n\t\t\t\tHence \\(\\inf_{k\\in\\N}\\varphi(x^k)=\\inf_{\\overline C}\\varphi\\).\n\t\t\t\\end{proof}\n\n\\begin{claim}\\label{claim:unique}%\n\t\t\t\tThere exists exactly one optimal limit point \\(x^\\star\\in\\argmin_C\\varphi\\).\n\t\t\t\\end{claim}\n\n\\begin{proof}\n\t\t\t\tThe existence of an optimal limit point is guaranteed by the previous claim, since \\((x^k)_{k\\in\\N}\\) is bounded (and bounded away from the boundary of \\(C\\)) and \\(\\varphi\\) is lsc.\n\t\t\t\tConsider two optimal limit points \\(\\bar x_1\\) and \\(\\bar x_2\\), and two corresponding subsequences \\((x^k)_{k\\in K_1}\\to\\bar x_1\\) and \\((x^k)_{k\\in K_2}\\to\\bar x_2\\).\n\t\t\t\tNotice that\n\t\t\t\t\\[\n\t\t\t\t\t\\U_k(\\bar x_1)-\\U_k(\\bar x_2)\n\t\t\t\t=\n\t\t\t\t\t\\D_{\\kernel}(\\bar x_1,x^k)-\\D_{\\kernel}(\\bar x_2,x^k)\n\t\t\t\t\\]\n\t\t\t\tconverges to some finite value \\(V\\), because both \\(\\U_k(\\bar x_1)\\) and \\(\\U_k(\\bar x_2)\\) are convergent.\n\t\t\t\tConsidering the limits along \\(k\\in K_1\\) and \\(k\\in K_2\\) yields that\n\t\t\t\t\\[\n\t\t\t\t\tV\n\t\t\t\t=\n\t\t\t\t\t\\D_{\\kernel}(\\bar x_1,\\bar x_1)-\\D_{\\kernel}(\\bar x_2,\\bar x_1)\n\t\t\t\t=\n\t\t\t\t\t\\D_{\\kernel}(\\bar x_1,\\bar x_2)-\\D_{\\kernel}(\\bar x_2,\\bar x_2),\n\t\t\t\t\\]\n\t\t\t\thence that \\(-\\D_{\\kernel}(\\bar x_2,\\bar x_1)=\\D_{\\kernel}(\\bar x_1,\\bar x_2)\\).\n\t\t\t\tThus \\(\\DD_{\\kernel}(\\bar x_1,\\bar x_2)=0\\), implying that \\(\\bar x_1=\\bar x_2\\).\n\t\t\t\\end{proof}\n\n\t\t\tIn light of the previous claim, the proof of \\cref{thm:xinC} is completed once we show that \\(x^k\\to x^\\star\\).\n\t\t\tTo this end, owing to the fact that \\(\\D_{\\kernel}(x^\\star,x^k)\\leq\\U_k(x^\\star)\\) it will suffice to show that \\(U\\coloneqq\\lim_{k\\to\\infty}\\U_k(x^\\star)\\) is zero, (existence and finiteness of \\(U\\) is ensured by \\cref{thm:SD}).\n\t\t\t\\begin{itemize}\n\t\t\t\\item\n\t\t\t\tLet us consider a subsequence \\((x^k)_{k\\in K}\\) converging to \\(x^\\star\\).\n\t\t\t\tIf the corresponding subsequence \\((\\gamma_{k+1})_{k\\in K}\\) is bounded, then it follows from \\cref{lem:prox:property:optimal} that  \\(x^{k+1}\\to x^\\star\\) and \\(\\gamma_{k+1} P_k(x^\\star) \\to 0\\) as well, and from the expression of \\(\\U_k\\) together with the fact that \\(\\hat\\rho_{k+1}\\) is bounded and \\(\\inf_k \\alpha_k>0\\), it is clear that\n\t\t\t\t\\(\\U_{k+1}(x^\\star)\\to0\\) as \\(K\\ni k\\to\\infty\\).\n\n\t\t\t\\item\n\t\t\t\tIn what follows, let us instead consider the complementary case in which for any subsequence \\(x^k\\) converging to \\(x^\\star\\) the corresponding subsequence of stepsizes \\((\\gamma_{k+1})_{k\\in K}\\) is divergent.\n\t\t\t\tIn this case, note that since \\(\\gamma_{k+1}P_k(x^\\star)\\leq\\U_{k+1}(x^\\star)\\leq\\U_0(x^\\star)\\), we have\n\t\t\t\t\\begin{align}\n\t\t\t\t\\nonumber\n\t\t\t\t\t(x^k)_{k\\in K}\\to x^\\star\n\t\t\t\t\\Leftrightarrow{} &\n\t\t\t\t\t(\\gamma_{k+1})_{k\\in K}\\to\\infty\n\t\t\t\t\\Leftrightarrow\n\t\t\t\t\t\\bigl(P_k(x^\\star)\\bigr)_{k\\in K}\\to0\n\t\t\t\t\\\\\n\t\t\t\t\\nonumber\n\t\t\t\t\\Rightarrow{} &\n\t\t\t\t\t(\\gamma_k)_{k\\in K}\\to\\infty\n\t\t\t\t\\\\\n\t\t\t\t\\Leftrightarrow{} &\n\t\t\t\t\t(x^{k-1})_{k\\in K}\\to x^\\star\n\t\t\t\t\\Leftrightarrow\n\t\t\t\t\t\\bigl(P_{k-1}(x^\\star)\\bigr)_{k\\in K}\\to0,\n\t\t\t\t\\label{eq:gamk:infty\"gamk-1}\n\t\t\t\t\\end{align}\n\t\t\t\twhere the right implication follows from the fact that \\(\\gamma_{k+1}\\leq\\hat\\rho_{k+1}\\gamma_k\\leq\\rho_{\\rm max}\\gamma_k\\), for some \\(\\rho_{\\rm max}\\leq\\frac{1+\\sqrt{5}}{2}\\).\n\t\t\t\tSince an optimal limit exists, it follows that \\(\\sup_{k\\in\\N}\\gamma_k=\\infty\\).\n\n\t\t\t\tIn what follows, we construct a specific subsequence \\(K\\coloneqq\\{k_0,k_1,\\dots\\}\\) along which \\((\\gamma_k)_{k\\in K}\\) diverges.\n\t\t\t\tIn doing so, we expand upon the arguments in the proof of \\cite[Thm. 2.3]{latafat2024adaptive} to account for the complications of the non-Euclidean setting investigated here.\n\t\t\t\tFor \\(i\\geq0\\) let\n\t\t\t\t\\begin{equation}\\label{eq:ki+}\n\t\t\t\t\tk_{i+1}=\\min\\{k\\geq k_i \\mid \\gamma_k \\geq \\rho_{\\rm max}\\gamma_{k_i}\\}.\n\t\t\t\t\\end{equation}\n\t\t\t\tThen, \\(\\bigl(\\gamma_{k_i}\\bigr)_{i\\in\\N}\\to\\infty\\), which by \\eqref{eq:gamk:infty\"gamk-1} implies that\n\t\t\t\t\\begin{equation}\\label{eq:xkitox*}\n\t\t\t\t\t\\lim_{i\\to\\infty}x^{k_i-s}\n\t\t\t\t=\n\t\t\t\t\tx^\\star\n\t\t\t\t\\quad\n\t\t\t\t\t\\forall s=1,2,3\n\t\t\t\t\\end{equation}\n\t\t\t\t(in fact, for any \\(s\\in\\N\\)).\n\t\t\t\tIn light of \\cref{thm:SD} we have that\n\t\t\t\t\\begin{equation}\\label{eq:U:gamkPk-1}\n\t\t\t\t\tU= \\lim_{k \\to \\infty} \\U_k= \\lim_{i \\to \\infty} \\U_{k_i-1}(x^\\star) = \\lim_{i \\to \\infty} \\gamma_{k_i-1}(1+ \\vartheta_{k_i-1})P_{k_i-2}(x^\\star).\n\t\t\t\t\\end{equation}\n\n\t\t\t\tWe proceed to show that \\(\\gamma_{k-1} P_{k-2}(x^\\star)\\) converges to zero along the same subsequence.\n\t\t\t\t\\begin{subequations}\n\t\t\t\t\tFor every \\(i\\in\\N\\), note that\n\t\t\t\t\t\\begin{gather}\n\t\t\t\t\t\t\\rho_{k_i}>1\n\t\t\t\t\t\t~\\text{ and }~\n\t\t\t\t\t\tk_{i-1}\\leq k_i-1\n\t\t\t\t\t\\shortintertext{by minimality of \\(k_i\\) and the fact that \\(\\rho_{\\rm max}>1\\), hence that}\n\t\t\t\t\t\\label{eq:k_i-1}\n\t\t\t\t\t\t\\text{either }~\n\t\t\t\t\t\t\\rho_{k_i-1}>1\n\t\t\t\t\t\t~\\text{ or }~\n\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\tk_i-1\\notin K\n\t\t\t\t\t\t\t\\text{ and thus }\n\t\t\t\t\t\t\tk_{i-1}\\leq k_i-2\n\t\t\t\t\t\t\\bigr).\n\t\t\t\t\t\\shortintertext{These combined imply also that}\n\t\t\t\t\t\\label{eq:k-1}\n\t\t\t\t\t\t\\rho_{k_i-1}\\geq\\rho_{\\rm max}^{-1}.\n\t\t\t\t\t\\end{gather}\n\t\t\t\t\tIndeed, if not, then \\(\\rho_{k_i-1}<\\rho_{\\rm max}^{-1}<1\\), implying by \\eqref{eq:k_i-1} that \\(k_{i-1}\\leq k_i-2\\);\n\t\t\t\t\tthis would lead to the contradiction\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\rho_{\\rm max}\\gamma_{k_{i-1}}\n\t\t\t\t\t\\stackrel{\\text{\\clap{\\eqref{eq:ki+}}}}{\\leq}\n\t\t\t\t\t\t\\gamma_{k_i}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\rho_{\\rm max}\\gamma_{k_i-1}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\rho_{\\rm max}\\rho_{k_i-1}\\gamma_{k_i-2}\n\t\t\t\t\t\\stackrel{\\mathclap{\\bot}}{<}\n\t\t\t\t\t\t\\gamma_{k_i-2}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\rho_{\\rm max}\\gamma_{k_{i-1}},\n\t\t\t\t\t\\]\n\t\t\t\t\twhere ``\\(\\bot\\)'' marks where the contradictory inequality is used, and the last inequality follows both in case \\(k_i-2=k_{i-1}\\) (since \\(\\rho_{\\rm max}>1\\)) or when \\(k_{i-1}<k_i-2<k_i\\) (from minimality in the definition of \\(k_i\\)).\n\t\t\t\t\\end{subequations}\n\t\t\t\tTherefore, \\eqref{eq:k-1} holds true, implying in particular that the sequence\n\t\t\t\t\\begin{equation}\\label{eq:Lambda:k-2:bounded}\n\t\t\t\t\t\\bigl(\\Lambda_{k_i-2, 2\\hat\\rho_{k_i-1}}\\bigr)_{i\\in\\N}\n\t\t\t\t\\quad\n\t\t\t\t\t\\text{is bounded,}\n\t\t\t\t\\end{equation}\n\t\t\t\tfor otherwise the second term in either \\eqref{eq:BY:rhok*} or \\eqref{eq:FNEa:rhok*} would vanish, owing to the fact that \\(\\hat\\rho_k\\) is lower bounded by \\(\\sqrt{1/2}\\) and similarly \\(\\alpha_k\\) is bounded away from zero.\n\n\t\t\t\tLet \\(v^k\\coloneqq H_k(x^{k-1})- H_k(x^k)\\in\\gamma_k\\partial\\varphi(x^k)\\), and observe that\n\t\t\t\t\\begin{align*}\n\t\t\t\t&\n\t\t\t\t\t\\gamma_k P_{k-1}(x^\\star)\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\rho_k\\innprod{x^\\star - x^{k-1}}{-v^{k-1}}\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{\\rho_k}{2\\hat\\rho_k}\\D_{\\kernel}(x^\\star, x^{k-1})\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\rho_k}{2\\hat\\rho_k}\n\t\t\t\t\t\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(x^{k-1})-2\\hat\\rho_kv^{k-1}\\bigr), \\nabla \\kernel(x^{k-1})\\bigr)\n\t\t\t\t\\\\\n\t\t\t\t={} &\n\t\t\t\t\t\\tfrac{\\rho_k}{2\\hat\\rho_k}\\D_{\\kernel}(x^\\star, x^{k-1})\n\t\t\t\t\t+\n\t\t\t\t\t2\\rho_k \\hat\\rho_k^2\\Lambda_{k-1, 2\\hat\\rho_k} \\DD_{\\kernel}(x^{k-1},x^{k-2})\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{1}{2}\\D_{\\kernel}(x^\\star, x^{k-1})\n\t\t\t\t\t+\n\t\t\t\t\t2\\rho_{\\rm max}^3 \\Lambda_{k-1, 2\\hat\\rho_k} \\DD_{\\kernel}(x^{k-1},x^{k-2})\n\t\t\t\t\\end{align*}\n\t\t\t\tholds for any \\(k\\), where we used the Bregman--Young inequality \\eqref{eq:BY} with parameter \\(\\delta= \\hat\\rho_k\\) in the second inequality, and \\(\\rho_k\\leq \\hat\\rho_k\\leq \\rho_{\\rm max}\\) in the last one.\n\t\t\t\tRecall that \\(\\bigl(x^{k_i-3}\\bigr)_{i\\in\\N} \\to x^\\star\\) by \\eqref{eq:xkitox*}.\n\t\t\t\tThus,\n\t\t\t\t\\[\n\t\t\t\t\t\\gamma_{k_i-1} P_{k_i-2}(x^\\star)\n\t\t\t\t\\leq\n\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t\\underbrace{\n\t\t\t\t\t\t\\D_{\\kernel}(x^\\star, x^{k_i-2})\n\t\t\t\t\t\t\\vphantom{\\Lambda_{k_i-2, 2\\rho_{k_i-1}}}\n\t\t\t\t\t}_{\\to 0 \\text{ by \\eqref{eq:xkitox*}}}\n\t\t\t\t\t+\n\t\t\t\t\t2\\rho_{\\rm max}^3\n\t\t\t\t\t\\underbrace{\n\t\t\t\t\t\t\\Lambda_{k_i-2, 2\\rho_{k_i-1}}\n\t\t\t\t\t}_{\\text{\\rm bounded}}\n\t\t\t\t\t\\,\n\t\t\t\t\t\\underbrace{\n\t\t\t\t\t\t\\DD_{\\kernel}(x^{k_i-2},x^{k_i-3})\n\t\t\t\t\t\t\\vphantom{\\Lambda_{k_i-2, 2\\rho_{k_i-1}}}\n\t\t\t\t\t}_{\\to 0 \\text{ by \\eqref{eq:xkitox*}}}.\n\t\t\t\t\\]\n\t\t\t\tUsing this in \\eqref{eq:U:gamkPk-1} and noting that \\(\\theta_{k_i-1}\\) being a multiple of \\(\\hat{\\rho}_{k_i-1}\\) in both \\refBY{} and \\refFNEa{} is bounded, completes the proof.\n\t\t\t\\end{itemize}\n\n\n\\subsection{Proof of \\texorpdfstring{\\cref{eq:main:inf}}{Equation \\ref*{eq:main:inf}}}\n\t\t\tIn this subsection we prove that the iterates generated by either \\refBY{} or \\refFNEa{} in the generality of \\cref{ass:basic} (in addition to \\(\\kernel\\) having a symmetry coefficient \\(\\alpha>0\\) in the latter case) are such that \\(\\inf_{k\\in\\N}\\varphi(x^k)=\\inf_{\\overline C}\\varphi\\).\n\n\t\t\tTo see this, contrary to the claim suppose that \\(\\inf_{k\\in\\N}\\varphi(x^k)>\\inf_{\\overline C}\\varphi\\).\n\t\t\tThen, since \\(\\inf_{\\overline C}\\varphi=\\inf_C\\varphi\\) by \\cite[Prop. 11.1(iv)]{bauschke2017convex}, there exists \\(x\\in C\\) such that \\(\\varphi(x^k)\\geq\\varphi(x)\\) holds for all \\(k\\in\\N\\).\n\t\t\tInvoking \\cref{thm:xinC} yields a contradiction.\n\n\n\n\t\t%% ░░░░ 4.5 Proof of Theorem 2.7(ii) ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░\n\t\t\\subsection{Proof of \\texorpdfstring{\\cref{thm:main:zone}}{Theorem \\ref*{thm:main:zone}}}\n\t\t\tIn this subsection we prove that whenever \\cref{ass:basic,ass:zone} are satisfied, the iterates generated by \\refBY{} are bounded and admit exactly one optimal limit point.\\footnote{%\n\t\t\t\t\\label{foot:noFNEa}%\n\t\t\t\tAs commented after \\cref{thm:main}, there is no loss of generality in considering only \\refBY{} and disregard the claim for \\refFNEa.\n\t\t\t}\n\t\t\tWe can without loss of generality assume that \\(\\argmin_{C}\\varphi=\\emptyset\\), for otherwise a stronger result is already covered by \\cref{thm:main:C}.\n\n\t\t\tUnder \\cref{ass:zone}, \\(\\dom\\kernel=\\overline C\\), and therefore in \\cref{thm:SD} one can take any \\(x\\in\\argmin_{\\overline C}\\varphi\\).\n\t\t\tIn particular, \\(\\D_{\\kernel}(x,x^k)\\leq\\U_k(x)\\leq\\U_0(x)\\) holds for any \\(k\\).\n\t\t\tSince \\(\\D_{\\kernel}(x,{}\\cdot{})\\) is level bounded by \\cref{ass:xkbounded}, boundedness of \\((x^k)_{k\\in\\N}\\) follows.\n\t\t\tMoreover, we know from \\eqref{eq:main:inf} that \\(\\varphi(x^k)\\to\\inf_{\\overline C}\\varphi\\) along a subsequence, and therefore an optimal limit point \\(x^\\star\\) exists.\n\t\t\tTo assess its uniqueness, we can argue similarly to the proof of \\cref{claim:unique}, with the minor catch that now such limit points are on the boundary of \\(C\\).\n\t\t\tConsidering two optimal limit points \\(\\bar x_1\\) and \\(\\bar x_2\\), and two corresponding subsequences \\((x^k)_{k\\in K_1}\\to\\bar x_1\\) and \\((x^k)_{k\\in K_2}\\to\\bar x_2\\), we still have that\n\t\t\t\\[\n\t\t\t\t\\U_k(\\bar x_1)-\\U_k(\\bar x_2)\n\t\t\t=\n\t\t\t\t\\D_{\\kernel}(\\bar x_1,x^k)-\\D_{\\kernel}(\\bar x_2,x^k)\n\t\t\t\\]\n\t\t\tconverges to some finite value \\(V\\).\n\t\t\tAgain by considering the limit along the two subsequences \\(k\\in K_1\\) and \\(k\\in K_2\\), \\cref{ass:Dhto0} yields that\n\t\t\t\\[\n\t\t\t\t-\\lim_{k\\in K_2}\\D_{\\kernel}(\\bar x_1,x^k)\n\t\t\t=\n\t\t\t\t\\lim_{k\\in K_1}\\D_{\\kernel}(\\bar x_2,x^k),\n\t\t\t\\]\n\t\t\thence that\n\t\t\t\\[\n\t\t\t\t\\lim_{k\\in K_2}\\D_{\\kernel}(\\bar x_1,x^k)\n\t\t\t=\n\t\t\t\t0\n\t\t\t=\n\t\t\t\t\\lim_{k\\in K_1}\\D_{\\kernel}(\\bar x_2,x^k),\n\t\t\t\\]\n\t\t\twhich by \\cref{fact:D:solodov} implies that \\(\\bar x_1=\\bar x_2\\).\n\n\n\n\t\t%% ░░░░ 4.6 Bregman proximal gradient with backtracking ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░\n\t\t\n\n\\begin{proof}[\\textbf{Proof of \\cref{thm:eq}}]%\n\t\t\tBased on the subgradient characterization \\eqref{eq:subgradvarphi} and the definition of \\(\\ell_k\\) in \\eqref{eq:lk}, we have\n\t\t\t\\begin{align}\n\t\t\t\\nonumber\n\t\t\t\t0\n\t\t\t={} &\n\t\t\t\t\\varphi(x^{k-1})\n\t\t\t\t-\n\t\t\t\t\\varphi(x^k)\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_k}\n\t\t\t\t\\innprod{H_k(x^{k-1})-H_k(x^k)}{x^{k-1}-x^k}\n\t\t\t\t-\n\t\t\t\t\\tD_\\varphi(x^{k-1},x^k)\n\t\t\t\\\\\n\t\t\t={} &\n\t\t\t\tP_{k-1}(x)\n\t\t\t\t-\n\t\t\t\tP_k(x)\n\t\t\t\t-\n\t\t\t\t\\tfrac{1-\\gamma_k\\ell_k}{\\gamma_k}\n\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t-\n\t\t\t\t\\tD_\\varphi(x^{k-1},x^k).\n\t\t\t\\label{eq:thetk}\n\t\t\t\\end{align}\n\t\t\tFurthermore, by leveraging the subgradient characterization \\eqref{eq:subgrad} and applying the three point identity of \\cref{thm:3p},\n\t\t\t\\begin{align}\n\t\t\t\\nonumber\n\t\t\t\t0\n\t\t\t={} &\n\t\t\t\tg(x)-g(x^{k+1})-\\innprod{\\tilde{\\nabla}g(x^{k+1})}{x-x^{k+1}}-\\tD_g(x,x^{k+1})\n\t\t\t\\\\\n\t\t\t\\nonumber\n\t\t\t={} &\n\t\t\t\tg(x)-g(x^{k+1})\n\t\t\t\t+\n\t\t\t\t\\innprod{\\nabla f(x^k)}{x- x^{k+1}}\n\t\t\t\\\\\n\t\t\t\\nonumber\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\innprod{\\nabla\\kernel(x^k)-\\nabla\\kernel(x^{k+1})}{x-x^{k+1}}\n\t\t\t\t-\n\t\t\t\t\\tD_g(x,x^{k+1})\n\t\t\t\\\\\n\t\t\t\\nonumber\n\t\t\t={} &\n\t\t\t\tg(x)-g(x^{k+1})\n\t\t\t\t+\n\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x- x^{k+1}}\n\t\t\t\t}_{\\text{(A)}}\n\t\t\t\t-\n\t\t\t\t\\tD_g(x,x^{k+1})\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x,x^k)\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x,x^{k+1})\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x^{k+1},x^k).\n\t\t\t\\label{eq:main}\n\t\t\t\\end{align}\n\t\t\tWe next proceed to expand the term (A) as\n\t\t\t\\begin{align*}\n\t\t\t\t\\text{(A)}\n\t\t\t={} &\n\t\t\t\t\\innprod{\\nabla f(x^k)}{x-x^k}\n\t\t\t\t-\n\t\t\t\t\\innprod{\\nabla f(x^k)}{x^{k+1}-x^k}\n\t\t\t\\\\\n\t\t\t={} &\n\t\t\t\t\\innprod{\\nabla f(x^k)}{x-x^k}\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_k}\n\t\t\t\t\\innprod{H_k(x^{k-1})-\\nabla\\kernel(x^k)}{x^{k+1}-x^k}\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_k}\n\t\t\t\t\\innprod{H_k(x^{k-1})-H_k(x^k)}{x^k-x^{k+1}}\n\t\t\t\\\\\n\t\t\t={} &\n\t\t\t\t\\Bigl[\n\t\t\t\t\tf(x)-f(x^k)-\\D_f(x,x^k)\n\t\t\t\t\\Bigr]\n\t\t\t\t+\n\t\t\t\t\\Bigl[\n\t\t\t\t\tg(x^{k+1})-g(x^k)-\\tD_g(x^{k+1},x^k)\n\t\t\t\t\\Bigr]\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\n\t\t\t\tB_{k+1},\n\t\t\t\\end{align*}\n\t\t\twhich combined with \\eqref{eq:main} gives\n\t\t\t\\begin{align*}\n\t\t\t\t0\n\t\t\t={} &\n\t\t\t\t-P_k(x)\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x,x^k)\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x,x^{k+1})\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\\D_f(x,x^k)\n\t\t\t\t-\\tD_g(x^{k+1},x^k)\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\n\t\t\t\tB_{k+1}\n\t\t\t\t-\n\t\t\t\t\\tD_g(x,x^{k+1}).\n\t\t\t\\end{align*}\n\t\t\tWe may now add \\eqref{eq:main} to \\eqref{eq:thetk} scaled by \\(\\vartheta_{k+1}\\) and multiply everything by \\(\\gamma_{k+1}\\) to obtain\n\t\t\t\\begin{align*}\n\t\t\t\t0\n\t\t\t={} &\n\t\t\t\t-\\gamma_{k+1} P_k(x)\n\t\t\t\t+\n\t\t\t\t\\D_{\\kernel}(x,x^k)\n\t\t\t\t-\n\t\t\t\t\\D_{\\kernel}(x,x^{k+1})\n\t\t\t\t-\n\t\t\t\t\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\gamma_{k+1}\\D_f(x,x^k)\n\t\t\t\t-\n\t\t\t\t\\gamma_{k+1}\\tD_g(x^{k+1},x^k)\n\t\t\t\t+\n\t\t\t\tB_{k+1}\n\t\t\t\t-\n\t\t\t\t\\gamma_{k+1}\\tD_g(x,x^{k+1})\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t+\n\t\t\t\t\\gamma_{k+1}\\vartheta_{k+1}\\left(\n\t\t\t\t\tP_{k-1}(x)\n\t\t\t\t\t-\n\t\t\t\t\tP_k(x)\n\t\t\t\t\t-\n\t\t\t\t\t\\tfrac{1-\\gamma_k\\ell_k}{\\gamma_k}\n\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t\t-\n\t\t\t\t\t\\tD_\\varphi(x^{k-1},x^k)\n\t\t\t\t\\right).\n\t\t\t\\end{align*}\n\t\t\tAfter suitably rearranging, the claimed identity is obtained.\n\n\t\t\tThe inequality follows by neglecting the terms between curly brackets, and further using the identity \\(\\D_{\\kernel}(x^k,x^{k-1})=\\frac{\\alpha_k}{1+\\alpha_k}\\DD_{\\kernel}(x^k,x^{k-1})\\).\n\t\t\\end{proof}\n\n\\begin{proof}[\\textbf{Proof of \\cref{thm:descent}}]%\n\t\t\tIt suffices to prove the claim for \\refBY; the case of \\refFNEa{} under the needed assumptions follows by simply replacing \\(\\U_k\\gets\\U^{\\alpha}_k\\).\n\t\t\tThe assumption on \\(x\\) ensures that \\(P_k(x)\\geq0\\) holds for all \\(k\\), implying both that \\(\\U_k(x)\\geq0\\) and the claimed monotonic decrease of \\(\\bigl(\\U_k(x)\\bigr)_{k\\in\\N}\\) with finite limit by virtue of \\cref{thm:BY:leq,thm:FNEa:leq}.\n\t\t\tMore precisely, one has that\n\t\t\t\\[\n\t\t\t\t0\n\t\t\t\\leq\n\t\t\t\t\\U_{k+1}(x)\n\t\t\t\\leq\n\t\t\t\t\\U_k(x)\n\t\t\t\t-\n\t\t\t\t\\gamma_k(1+q\\hat\\rho_k-q\\hat\\rho_{k+1}\\rho_{k+1})P_{k-1}(x),\n\t\t\t\\]\n\t\t\twhere \\(q=1\\) for \\refBY{} and \\(q=\\frac{1+\\alpha}{2\\alpha}\\) for \\refFNEa{}.\n\t\t\tA telescoping argument yields that\n\t\t\t\\begin{align*}\n\t\t\t\tP_K^{\\min}(x)\n\t\t\t\t\\sum_{k=1}^K\n\t\t\t\t\\gamma_k(1+q\\hat\\rho_k-q\\hat\\rho_{k+1}\\rho_{k+1})\n\t\t\t\\leq{} &\n\t\t\t\t\\sum_{k=1}^K\n\t\t\t\t\\gamma_k(1+q\\hat\\rho_k-q\\hat\\rho_{k+1}\\rho_{k+1})P_{k-1}(x)\\\\\n\t\t\t\\leq{} &\n\t\t\t\t\\U_1(x) - \\U_{K+1}(x)\\\\\n\t\t\t\\leq{} &\n\t\t\t\t\\U_1(x) - \\gamma_{K+1}(1+q\\hat\\rho_{K+1})P_K^{\\rm min}(x),\n\t\t\t\\end{align*}\n\t\t\thence that\n\t\t\t\\begin{align*}\n\t\t\t\tP_K^{\\min}(x)\n\t\t\t\\leq{} &\n\t\t\t\t\\frac{\\U_1(x)\n\t\t\t\t\t}{\n\t\t\t\t\t\\sum_{k=1}^K(\\gamma_k + q\\gamma_k\\hat\\rho_k- q\\gamma_{k+1}\\hat\\rho_{k+1}) + \\gamma_{K+1} + q\\gamma_{K+1}\\hat\\rho_{K+1}\n\t\t\t\t\t}\\\\\n\t\t\t\\leq{} &\n\t\t\t\t\\frac{\\U_1(x)\n\t\t\t\t\t}{\n\t\t\t\t\t\\gamma_1\\hat\\rho_1\n\t\t\t\t\t+\n\t\t\t\t\t\\sum_{k=1}^{K+1}\\gamma_k\n\t\t\t\t\t}.\n\t\t\t\\end{align*}\n\t\t\tSince \\(\\U_1(x)\\leq\\U_0(x)\\), the claimed inequality follows.\n\t\t\\end{proof}","units":[{"start":24557,"end":25027,"start_line":607,"end_line":619,"raw_tex":"\\begin{fact}[three-point identity {\\cite[Lem. 3.1]{chen1993convergence}}]\\label{thm:3p}%\n\t\t\t\t\tLet \\(\\kernel:\\R^n\\to\\Rinf\\) be a proper lsc convex function differentiable on \\(\\interior\\dom\\kernel\\).\n\t\t\t\t\tFor any \\(x\\in\\dom\\kernel\\), and \\(y,z\\in\\interior\\dom\\kernel\\) the following holds:\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\D_{\\kernel}(x,z)\n\t\t\t\t\t=\n\t\t\t\t\t\t\\D_{\\kernel}(x,y)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\D_{\\kernel}(y,z)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{x-y}{\\nabla\\kernel(y)-\\nabla\\kernel(z)}.\n\t\t\t\t\t\\]\n\t\t\t\t\\end{fact}","sha256_utf8":"b1fa0c0449b0818138548d76c3712adce4f3f0fe33847bcd32be73c5e3ef4394"},{"start":25534,"end":27015,"start_line":625,"end_line":649,"raw_tex":"\\begin{fact}\\label{fact:D}%\n\t\t\t\t\tLet \\(\\kernel:\\R^n\\to\\Rinf\\) be of Legendre type, and let \\(C\\coloneqq\\interior\\dom\\kernel\\).\n\t\t\t\t\t\\begin{enumerate}\n\t\t\t\t\t\\item \\label{fact:D:coercive}%\n\t\t\t\t\t\t\t{\\upshape \\cite[Prop. 2.16]{bauschke1997legendre}, \\cite[Thm. 3.7(vi)]{bauschke1997legendre}}\n\t\t\t\t\t\t\tIf \\(\\kernel\\) is 1-coercive, then \\(\\dom\\kernel^*=\\R^n\\) and \\(\\D_{\\kernel}(x,{}\\cdot{})\\) is coercive for any \\(x\\in C\\).\n\n\t\t\t\t\t\\item \\label{fact:D:boundary}%\n\t\t\t\t\t\t{\\upshape \\cite[Thm. 3.8(i)]{bauschke1997legendre}}\n\t\t\t\t\t\tIf a sequence \\((x^k)_{k\\in\\N}\\) converges to a boundary point of \\(C\\), then \\(\\D_{\\kernel}(x,x^k)\\to\\infty\\) for any \\(x\\in C\\).\n\n\t\t\t\t\t\\item \\label{fact:D:solodov}%\n\t\t\t\t\t\t{\\upshape \\cite[Thm. 2.4]{solodov2000inexact}}\n\t\t\t\t\t\tSuppose that \\(\\dom\\kernel\\) is closed and that \\(\\D_{\\kernel}(x^k,y^k)\\to0\\).\n\t\t\t\t\t\tIf either \\((x^k)_{k\\in\\N}\\) or \\((y^k)_{k\\in\\N}\\) converges, then the other sequence too converges to the same limit.\n\n\t\t\t\t\t\\item \\label{fact:D:Dphi*}%\n\t\t\t\t\t\t{\\upshape \\cite[Thm. 26.5]{rockafellar1970convex}}\n\t\t\t\t\t\tThe conjugate \\(\\kernel^*\\) is continuously differentiable and strictly convex on \\(\\interior\\dom\\kernel^*\\), with \\(\\nabla\\kernel^* = \\nabla\\kernel^{-1}:\\interior\\dom\\kernel^*\\to\\interior\\dom\\kernel\\).\n\n\t\t\t\t\t\\item \\label{fact:D:DtoD*}%\n\t\t\t\t\t\t{\\upshape \\cite[Thm. 3.7(v)]{bauschke1997legendre}}\n\t\t\t\t\t\t\\(\\D_{\\kernel}(x,y) = \\D_{\\kernel^*}(\\nabla \\kernel(y), \\nabla \\kernel(x))\\) for any \\(x,y\\in C\\).\n\t\t\t\t\t\\end{enumerate}\n\t\t\t\t\\end{fact}","sha256_utf8":"3f934067b11b38b4caf981f0672b947772a8dfb090e62709103a5340764c6a73"},{"start":50393,"end":51926,"start_line":1188,"end_line":1254,"raw_tex":"\\begin{lemma}[main identity]\\label{thm:eq}%\n\t\t\tSuppose that \\cref{ass:basic} holds, and starting from \\(x^0\\in C\\) consider Bregman proximal gradient iterations \\eqref{eq:BPG} with stepsizes \\(\\gamma_k>0\\).\n\t\t\tThen, for any \\(x\\in\\dom\\varphi\\cap\\dom\\kernel\\), \\(\\vartheta_{k+1}\\geq0\\), \\(k\\in\\N\\), it holds that\n\t\t\t\\begin{align*}\n\t\t\t&\n\t\t\t\t\\D_{\\kernel}(x,x^{k+1})\n\t\t\t\t+\n\t\t\t\t\\gamma_{k+1}(1+\\vartheta_{k+1})P_k(x)\n\t\t\t\t+\n\t\t\t\t\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\\\\\n\t\t\t={} &\n\t\t\t\t\\D_{\\kernel}(x,x^k)\n\t\t\t\t+\n\t\t\t\t\\gamma_{k+1}\\vartheta_{k+1} P_{k-1}(x)\n\t\t\t\t-\n\t\t\t\t\\rho_{k+1}\\vartheta_{k+1}(1-\\gamma_k\\ell_k)\n\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t+\n\t\t\t\tB_{k+1}\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\gamma_{k+1}\n\t\t\t\t\\Bigl\\{\n\t\t\t\t\t\\D_f(x,x^k)\n\t\t\t\t\t+\n\t\t\t\t\t\\tD_g(x^{k+1},x^k)\n\t\t\t\t\t+\n\t\t\t\t\t\\tD_g(x,x^{k+1})\n\t\t\t\t\t+\n\t\t\t\t\t\\vartheta_{k+1}\\tD_\\varphi(x^{k-1},x^k)\n\t\t\t\t\\Bigr\\},\n\t\t\t\\end{align*}\n\t\t\twhere \\(P_k(x)\\), \\(B_{k+1}\\), and \\(\\ell_k\\) are as in \\eqref{eq:Pk}, \\eqref{eq:Bk}, and \\eqref{eq:lk}.\n\t\t\tIn particular, denoting\n\t\t\t\\[\n\t\t\t\t\\widehat{\\U}_k(x)\n\t\t\t\\coloneqq\n\t\t\t\t\\D_{\\kernel}(x,x^k)\n\t\t\t\t+\n\t\t\t\t\\gamma_k(1+\\vartheta_k)P_{k-1}(x)\n\t\t\t\t+\n\t\t\t\t\\D_{\\kernel}(x^k,x^{k-1}),\n\t\t\t\\]\n\t\t\tone has that\n\t\t\t\\begin{align}\n\t\t\t\\nonumber\n\t\t\t\t\\widehat{\\U}_{k+1}(x)\n\t\t\t\\leq{} &\n\t\t\t\t\\widehat{\\U}_k(x)\n\t\t\t\t-\n\t\t\t\t\\gamma_k(1+\\vartheta_k-\\rho_{k+1}\\vartheta_{k+1})P_{k-1}(x)\n\t\t\t\t+\n\t\t\t\tB_{k+1}\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\left[\n\t\t\t\t\t\\tfrac{\\alpha_k}{1+\\alpha_k}\n\t\t\t\t\t+\n\t\t\t\t\t\\rho_{k+1}\\vartheta_{k+1}(1-\\gamma_k\\ell_k)\n\t\t\t\t\\right]\n\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}).\n\t\t\t\\label{eq:ineq0}\n\t\t\t\\end{align}\n\t\t\\end{lemma}","sha256_utf8":"ff68de82de8338522fc770eb184e7548093d0583953de6042ec1d18d00461111"},{"start":51929,"end":51984,"start_line":1255,"end_line":1257,"raw_tex":"\\begin{proof}\n\t\t\tSee \\cref{proof:thm:eq}.\n\t\t\\end{proof}","sha256_utf8":"a61d0d9d44fb017950498d972454b620776a901638ad0597ae289597556939d5"},{"start":53252,"end":53754,"start_line":1282,"end_line":1294,"raw_tex":"\\begin{lemma}[Young's inequality in the Bregman sense]\\label{thm:BY}%\n\t\t\t\tLet \\(\\kernel:\\R^n\\to\\Rinf\\) be Legendre (not necessarily 1-coercive).\n\t\t\t\tThen, for any \\(x\\in\\dom\\kernel\\), \\(y\\in\\interior\\dom\\kernel\\), \\(v\\in\\R^n\\), and \\(\\delta>0\\) one has\n\t\t\t\t\\begin{equation}\\label{eq:BY}\n\t\t\t\t\t\\innprod{x-y}{v}\n\t\t\t\t\\leq\n\t\t\t\t\t\\tfrac{1}{\\delta}\n\t\t\t\t\t\\D_{\\kernel}(x,y)\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{1}{\\delta}\n\t\t\t\t\t\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(y)+\\delta v,\\nabla\\kernel(y)\\bigr).\n\t\t\t\t\\end{equation}\n\t\t\t\\end{lemma}","sha256_utf8":"d8658d38713ca9b38ce6b3bf39ae72c8ab0f13215bcc1529f9942b6b3cbd3018"},{"start":53758,"end":55364,"start_line":1295,"end_line":1335,"raw_tex":"\\begin{proof}\n\t\t\t\tWithout loss of generality we may assume that \\(\\nabla\\kernel(y)+\\delta v\\in\\dom\\kernel^*\\), for otherwise the right-hand side of \\eqref{eq:BY} is infinite and the inequality trivializes.\n\t\t\t\tSince \\(\\nabla\\kernel(y)\\in\\interior\\dom\\kernel^*\\) by \\cref{fact:D:Dphi*}, for any \\(t\\in(0,1)\\) one has that\n\t\t\t\t\\(\n\t\t\t\t\t\\nabla\\kernel(y)+t\\delta v\\in\\interior\\dom\\kernel^*\n\t\t\t\t\\)\n\t\t\t\tby \\cite[Prop. 3.44]{bauschke2017convex}.\n\t\t\t\tWe have\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\innprod{x-y}{v}\n\t\t\t\t={} &\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\innprod{x-y}{t\\delta v}\n\t\t\t\t\\\\\n\t\t\t\t={} &\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\innprod{x-y}{\\nabla\\kernel(\\nabla\\kernel^*\\bigl(\\nabla\\kernel(y)+t\\delta v)\\bigr)-\\nabla\\kernel(y)}\n\t\t\t\t\\\\\n\t\t\t\t={} &\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\D_{\\kernel}(x,y)\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\D_{\\kernel}\\bigl(y,\\nabla\\kernel^*\\bigl(\\nabla\\kernel(y)+t\\delta v\\bigr)\\bigr)\n\t\t\t\t\\\\\n\t\t\t\t&\n\t\t\t\t\t-\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\D_{\\kernel}\\bigl(x,\\nabla\\kernel^*\\bigl(\\nabla\\kernel(y)+t\\delta v\\bigr)\\bigr)\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\D_{\\kernel}(x,y)\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(y)+t\\delta v,\\nabla\\kernel(y)\\bigr).\n\t\t\t\\end{align*}\n\t\t\t\tThe second equality follows from Legendreness of \\(\\kernel\\) via \\cref{fact:D:Dphi*}, and the third one is the three point identity of \\cref{thm:3p}.\n\t\t\t\tThe inequality follows from \\cref{fact:D:DtoD*} and the fact that \\(\\D_\\kernel\\geq0\\).\n\t\t\t\tBy taking the limit as \\(t\\to1^-\\) and appealing to \\cite[Thm. 7.5]{rockafellar1970convex}, the claimed inequality \\eqref{eq:BY} is obtained.\n\t\t\t\\end{proof}","sha256_utf8":"361539ce716aae034b5b45969f8f8c1eb12f375af8cedb98101d9322dc6898e9"},{"start":61183,"end":62142,"start_line":1463,"end_line":1485,"raw_tex":"\t\t\t\tA direct application of Young's inequality of \\cref{thm:BY} allows us to bound the inner product \\(B_{k+1}\\) as\n\t\t\t\t\\begin{align}\n\t\t\t\t\\nonumber\n\t\t\t\t\tB_{k+1}\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\\\\\n\t\t\t\t\\nonumber\n\t\t\t\t&\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(x^k)+\\delta_{k+1}\\bigl[H_k(x^k)-H_k(x^{k-1})\\bigr],\\nabla\\kernel(x^k)\\bigl)\n\t\t\t\t\\shortintertext{%\n\t\t\t\t\tfor any \\(\\delta_{k+1}>0\\), which in terms of the Lipschitz-like estimate \\(\\Lambda_{k,\\delta}\\) as in \\eqref{eq:Lamk} reads\n\t\t\t\t}\n\t\t\t\t={} &\n\t\t\t\t\t\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\delta_{k+1}\\rho_{k+1}}{2}\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}).\n\t\t\t\t\\label{eq:Bk:BY}\n\t\t\t\t\\end{align}\n\t\t\t\tAs we will see, the employment of this inequality combined with a specific choice of \\(\\delta_{k+1}\\) will lead to the stepsize update in the \\refBY{} variant.\n","sha256_utf8":"e79eeccb1ed2a7fa20fdf858e4190c7fa022841c11fcd6b0b8db17dda3a1c5f9"},{"start":62419,"end":63064,"start_line":1491,"end_line":1504,"raw_tex":"\\begin{lemma}\\label{thm:FNE}\n\t\t\t\t\tLet \\(\\kernel:\\R^n\\to\\Rinf\\) be Legendre, and \\(f,g:\\R^n\\to\\Rinf\\) be proper, lsc, and convex, with \\(f\\) differentiable on \\(C\\coloneqq\\interior\\dom\\kernel\\) and \\(\\dom g\\cap C\\neq\\emptyset\\).\n\t\t\t\t\tThen, denoting \\(H_k\\coloneqq\\nabla\\kernel-\\gamma_k\\nabla f\\), Bregman proximal gradient iterations \\eqref{eq:BPG} with stepsizes \\(\\gamma_k>0\\) and starting at some \\(x^0\\in C\\) satisfy\n\t\t\t\t\t\\[\n\t\t\t\t\t\tB_{k+1}\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\tfrac{\\gamma_{k+1}}{\\gamma_k}\n\t\t\t\t\t\t\\innprod{x^{k+1}-x^k}{H_k(x^k)-H_k(x^{k-1})}\n\t\t\t\t\t\\geq\n\t\t\t\t\t\t\\DD_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\t\\quad\n\t\t\t\t\t\t\\forall k\\geq 1.\n\t\t\t\t\t\\]\n\t\t\t\t\\end{lemma}","sha256_utf8":"822afe267c861899332017e6c103164ec161ea014a91d09309465bbbe0e43e07"},{"start":63069,"end":63817,"start_line":1505,"end_line":1538,"raw_tex":"\\begin{proof}\n\t\t\t\t\tFollows by observing that\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t&\n\t\t\t\t\t\t\\tfrac{\\gamma_{k+1}}{\\gamma_k}\n\t\t\t\t\t\t\\bigl(H_k(x^k)-H_k(x^{k-1})\\bigr)\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\nabla\\kernel(x^{k+1})-\\nabla\\kernel(x^k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\gamma_{k+1}\n\t\t\t\t\t\t\\Bigl[\n\t\t\t\t\t\t\t\\smash{\n\t\t\t\t\t\t\t\t\\overbrace{\n\t\t\t\t\t\t\t\t\t\\tfrac{H_{k+1}(x^k)-\\nabla\\kernel(x^{k+1})}{\\gamma_{k+1}}\n\t\t\t\t\t\t\t\t}^{\\tilde{\\nabla}g(x^{k+1})}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\overbrace{\n\t\t\t\t\t\t\t\t\t\\tfrac{H_k(x^{k-1})-\\nabla\\kernel(x^k)}{\\gamma_k}\n\t\t\t\t\t\t\t\t}^{\\tilde{\\nabla}g(x^k)}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\Bigr],\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\thence that\n\t\t\t\t\t\\(\n\t\t\t\t\t\tB_{k+1}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\DD_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\gamma_{k+1}\\tD_g(x^{k+1},x^k)\n\t\t\t\t\t\\geq\n\t\t\t\t\t\t\\DD_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\t\\).\n\t\t\t\t\\end{proof}","sha256_utf8":"7dc461d3e1e31040331cdd9c9d538e6db54f936bdadca62b7662c7d8f5d43748"},{"start":63819,"end":65057,"start_line":1540,"end_line":1584,"raw_tex":"\t\t\t\tWe may thus complement \\eqref{eq:Bk:BY} with a lower bound as\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\DD_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\\leq{} &\n\t\t\t\t\tB_{k+1}\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\delta_{k+1}\\rho_{k+1}}{2}\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}).\n\t\t\t\t\\end{align*}\n\t\t\t\tIf \\(\\kernel\\) has symmetry coefficient \\(\\alpha>0\\) as in \\eqref{eq:alpha}, then note that\n\t\t\t\t\\[\n\t\t\t\t\t\\DD_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\\geq\n\t\t\t\t\t(1+\\alpha)\\D_{\\kernel}(x^{k+1},x^k).\n\t\t\t\t\\]\n\t\t\t\tCombined with the previous inequality, we obtain\n\t\t\t\t\\[\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t1+\\alpha-\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\n\t\t\t\t\t\\right)\n\t\t\t\t\t\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\\leq\n\t\t\t\t\t\\tfrac{\\delta_{k+1}\\rho_{k+1}}{2}\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}),\n\t\t\t\t\\]\n\t\t\t\twhich plugged into \\eqref{eq:Bk:BY} leads to\n\t\t\t\t\\begin{equation}\n\t\t\t\t\\label{eq:Bk:FNEa}\n\t\t\t\t\tB_{k+1}\n\t\t\t\t\\leq\n\t\t\t\t\t\\tfrac{\n\t\t\t\t\t\t(1+\\alpha)\\delta_{k+1}^2\n\t\t\t\t\t}{\n\t\t\t\t\t\t(1+\\alpha)\\delta_{k+1}-\\rho_{k+1}\n\t\t\t\t\t}\n\t\t\t\t\t\\tfrac{\\rho_{k+1}}{2}\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}),\n\t\t\t\t\\end{equation}\n\t\t\t\tholding for any \\(\\delta_{k+1}\\) such that \\((1+\\alpha)\\delta_{k+1}-\\rho_{k+1}>0\\).\n","sha256_utf8":"1f2642f6a89e055df222b5e2078e029a1fdadd4b83c9002930d1635586d946e6"},{"start":65241,"end":69383,"start_line":1590,"end_line":1697,"raw_tex":"\t\t\tBy bounding the inner product term \\(B_{k+1}\\) with \\eqref{eq:Bk:BY}, the inequality \\eqref{eq:ineq0} in \\cref{thm:eq} reveals that Bregman proximal gradient iterations \\eqref{eq:BPG} with arbitrary stepsizes \\(\\gamma_k>0\\) satisfy\n\t\t\t\\begin{align}\n\t\t\t\\nonumber\n\t\t\t&\n\t\t\t\t\\widehat{\\U}_{k+1}(x)\n\t\t\t\t-\n\t\t\t\t\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\n\t\t\t\t\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\\\\\n\t\t\t\\nonumber\n\t\t\t\\leq{} &\n\t\t\t\t\\widehat{\\U}_k(x)\n\t\t\t\t-\n\t\t\t\t\\tfrac{\\rho_k}{\\delta_k}\n\t\t\t\t\\D_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t-\n\t\t\t\t\\gamma_k(1+\\vartheta_k-\\rho_{k+1}\\vartheta_{k+1})P_{k-1}(x)\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\left[\n\t\t\t\t\t\\bigl(1-\\tfrac{\\rho_k}{\\delta_k}\\bigr)\n\t\t\t\t\t-\n\t\t\t\t\t\\rho_{k+1}(1+\\alpha_k)\n\t\t\t\t\t\\tfrac{\n\t\t\t\t\t\t\\frac{\\delta_{k+1}}{2}\n\t\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\vartheta_{k+1}\n\t\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\t}{\\alpha_k}\n\t\t\t\t\\right]\n\t\t\t\t\\D_{\\kernel}(x^k,x^{k-1})\n\t\t\t\\label{eq:UkBYdt:leq}\n\t\t\t\\end{align}\n\t\t\tfor any \\(x\\in\\dom\\varphi\\cap\\dom\\kernel\\), \\(\\vartheta_k\\geq0\\), and \\(\\delta_k>0\\), \\(k\\in\\N\\).\n\t\t\tImposing that the terms in brackets multiplying \\(P_{k-1}(x)\\) and \\(\\D_{\\kernel}(x^k,x^{k-1})\\) in the right-hand side of \\eqref{eq:UkBYdt:leq} are negative amounts to the following two conditions:\n\t\t\t\\begin{subequations}\\label{eq:gamk:ineq12}\n\t\t\t\t\\begin{gather}\n\t\t\t\t\\label{eq:gamk:ineq1}\n\t\t\t\t\t\\vartheta_{k+1}\\rho_{k+1}\\leq 1+\\vartheta_k\n\t\t\t\t\\shortintertext{and}\n\t\t\t\t\\label{eq:gamk:ineq2}\n\t\t\t\t\t1-\\tfrac{\\rho_k}{\\delta_k}\n\t\t\t\t\\geq\n\t\t\t\t\t\\rho_{k+1}\n\t\t\t\t\t\\tfrac{1+\\alpha_k}{\\alpha_k}\n\t\t\t\t\t\\bigl[\n\t\t\t\t\t\t\\tfrac{\\delta_{k+1}}{2}\n\t\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\vartheta_{k+1}(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\\bigr].\n\t\t\t\t\\end{gather}\n\t\t\t\\end{subequations}\n\t\t\tIn line with the analyses of \\cite{malitsky2020adaptive,latafat2024adaptive}, a convenient choice for the parameter \\(\\delta_{k+1}\\) is \\(\\delta_{k+1} = 2\\rho_{k+1}\\).\n\t\t\tHowever, this choice is not feasible in our more general setting.\n\t\t\tSpecifically, selecting \\(\\rho_{k+1}\\) to satisfy \\eqref{eq:gamk:ineq2} requires knowledge of the quantity \\(\\Lambda_{k,\\delta_{k+1}}\\), which generally depends on \\(\\delta_{k+1}\\) itself (except in the special case where \\(\\kernel\\) is quadratic).\n\t\t\tAs a result, setting \\(\\delta_{k+1}=2\\rho_{k+1}\\) would create a circular dependency between the two parameters.\n\t\t\tTo complicate things further, note that the left-hand side of \\eqref{eq:gamk:ineq2} indicates that a constraint \\(\\delta_k>\\rho_k\\) must be in place in order to ensure the existence of a \\(\\rho_{k+1}>0\\) satisfying the inequality.\n\n\n\t\t\tIn order to resolve this circular dependence, we introduce a parameter \\(\\hat\\rho_{k+1}\\) that shall provide an overestimate\n\t\t\t\\begin{equation}\\label{eq:rhokhatgeq}\n\t\t\t\t\\hat\\rho_{k+1}\\geq\\rho_{k+1}\n\t\t\t\\end{equation}\n\t\t\twhile only being based on information available at iteration \\(k\\).\n\t\t\tIts explicit value will be revealed shortly after.\n\t\t\tThen, we may conveniently select \\(\\delta_k=2\\hat\\rho_k\\) and \\(\\vartheta_k=\\hat\\rho_k\\) so that \\eqref{eq:gamk:ineq12} simplifies as\n\t\t\t\\[\n\t\t\t\t\\rho_{k+1}\\hat\\rho_{k+1}\n\t\t\t\\leq\n\t\t\t\t1+\\hat\\rho_k\n\t\t\t\\quad\\text{and}\\quad\n\t\t\t\t1-\\tfrac{\\rho_k}{2\\hat\\rho_k}\n\t\t\t\\geq\n\t\t\t\t\\rho_{k+1}\n\t\t\t\t\\hat\\rho_{k+1}\n\t\t\t\t\\tfrac{1+\\alpha_k}{\\alpha_k}\n\t\t\t\t\\bigl[\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t-\n\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\\bigr],\n\t\t\t\\]\n\t\t\tthat is,\n\t\t\t\\begin{equation}\\label{eq:BY:rhok*+}\n\t\t\t\t\\rho_{k+1}\n\t\t\t\\leq\n\t\t\t\t\\min\\left\\{\n\t\t\t\t\t\\frac{1+\\hat\\rho_k}{\\hat\\rho_{k+1}}\n\t\t\t\t\t,\\,\n\t\t\t\t\t\\frac{\\alpha_k}{1+\\alpha_k}\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t1+ \\frac{\\hat\\rho_k - \\rho_k}{\\hat\\rho_k}\n\t\t\t\t\t}{\n\t\t\t\t\t\t2\\hat\\rho_{k+1}\n\t\t\t\t\t\t\\bigl[\n\t\t\t\t\t\t\t\\Lambda_{k,2\\hat\\rho_{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\t\\bigr]_+\n\t\t\t\t\t}\n\t\t\t\t\\right\\}.\n\t\t\t\\end{equation}\n\t\t\tDue to \\eqref{eq:rhokhatgeq} the term \\(\\frac{\\hat\\rho_k - \\rho_k}{\\hat\\rho_k}\\) in the numerator of the second update is always nonnegative.\n\t\t\tUsing this, we show in the next lemma that the update rule of \\refBY{} always complies with the above bound.\n\t\t\tIn fact, while it is possible to retain the term \\(\\frac{\\hat\\rho_k - \\rho_k}{\\hat\\rho_k}\\) in the update of \\refBY{}, for the sake of a neater expression we opted to omit it at the cost of introducing slight conservatism.\n","sha256_utf8":"8c3175070e13880a535d989531a7ff0a629f0ee24f35a3ceaf6a48993b36661f"},{"start":69387,"end":70312,"start_line":1699,"end_line":1727,"raw_tex":"\\begin{lemma}\\label{thm:BY:leq}%\n\t\t\t\tSuppose that \\cref{ass:basic} holds, and consider the iterates generated by Bregman proximal gradient iterations \\eqref{eq:BPG} with \\(\\gamma_k\\) and \\(\\hat\\rho_k\\) selected according to \\refBY.\n\t\t\t\tThen, both \\eqref{eq:rhokhatgeq} and \\eqref{eq:BY:rhok*+} are satisfied for any \\(k\\in\\N\\).\n\n\t\t\t\tMoreover, denoting\n\t\t\t\t\\begin{equation}\\label{eq:UkBY}\n\t\t\t\t\t\\U_k(x)\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\D_{\\kernel}(x,x^k)\n\t\t\t\t\t+\n\t\t\t\t\t\\gamma_k(1+\\hat\\rho_k)P_{k-1}(x)\n\t\t\t\t\t+\n\t\t\t\t\t(1-\\tfrac{\\rho_k}{2\\hat\\rho_k})\n\t\t\t\t\t\\D_{\\kernel}(x^k,x^{k-1}),\n\t\t\t\t\\end{equation}\n\t\t\t\tone has that\n\t\t\t\t\\begin{equation}\\label{eq:UkBY:leq}\n\t\t\t\t\t\\U_{k+1}(x)\n\t\t\t\t\\leq\n\t\t\t\t\t\\U_k(x)\n\t\t\t\t\t-\n\t\t\t\t\t\\gamma_k(\n\t\t\t\t\t\t\\smash{\n\t\t\t\t\t\t\t\\overbrace{1+\\hat\\rho_k-\\rho_{k+1}\\hat\\rho_{k+1}}^{\\geq\\hat\\rho_k-\\rho_k\\geq0}\n\t\t\t\t\t\t}\n\t\t\t\t\t)P_{k-1}(x)\n\t\t\t\t\\end{equation}\n\t\t\t\tholds for any \\(x\\in\\dom\\varphi\\cap\\dom\\kernel\\) and \\(k\\in\\N\\).\n\t\t\t\\end{lemma}","sha256_utf8":"e8296d101bc63c013ef2843513a408ff95ffd8a2b3caaafbdc3989c81e530b6f"},{"start":70316,"end":71160,"start_line":1728,"end_line":1745,"raw_tex":"\\begin{proof}\n\t\t\t\tThe update \\eqref{eq:BY:rhok*} clearly ensures that \\(\\rho_{k+1}\\leq\\hat\\rho_{k+1}\\) always holds.\n\t\t\t\tNote that the second element in the minimum within \\eqref{eq:BY:rhok*+} coincides with that in \\eqref{eq:BY:rhok*};\n\t\t\t\tmoreover, since \\(\\rho_{k+1}\\leq\\hat\\rho_{k+1}\\) one has that\n\t\t\t\t\\(\n\t\t\t\t\t\\rho_{k+1}\\hat\\rho_{k+1}\n\t\t\t\t\\leq\n\t\t\t\t\t\\hat\\rho_{k+1}^2\n\t\t\t\t=\n\t\t\t\t\t1+\\rho_k\n\t\t\t\t\\leq\n\t\t\t\t\t1+\\hat\\rho_k\n\t\t\t\t\\),\n\t\t\t\taltogether confirming that the validity of \\eqref{eq:BY:rhok*+}.\n\n\t\t\t\tObserve further that the bound \\(\\hat\\rho_k\\geq\\rho_k\\) ensures that both elements in the minimum of \\eqref{eq:BY:rhok*} are strictly positive, and thus so are the generated stepsizes \\(\\gamma_k\\).\n\t\t\t\tFinally, \\eqref{eq:UkBY:leq} follows from \\eqref{eq:UkBYdt:leq} with the specified choices of \\(\\vartheta_k\\) and \\(\\delta_k\\).\n\t\t\t\\end{proof}","sha256_utf8":"87b85d76648f9d7bf37dbea25ac6db8fc041c4ca76e181b2c1c652bc1fe2b7d1"},{"start":71326,"end":73613,"start_line":1751,"end_line":1826,"raw_tex":"\t\t\tIn case \\(\\kernel\\) has a symmetry coefficient \\(\\alpha>0\\), we may leverage the bound \\eqref{eq:Bk:FNEa} for \\(B_{k+1}\\).\n\t\t\tBy doing so, the inequality \\eqref{eq:ineq0} in \\cref{thm:eq} becomes\n\t\t\t\\begin{multline*}\n\t\t\t\t\\widehat{\\U}_{k+1}(x)\n\t\t\t\\leq\n\t\t\t\t\\widehat{\\U}_k(x)\n\t\t\t\t-\n\t\t\t\t\\gamma_k(1+\\vartheta_k-\\rho_{k+1}\\vartheta_{k+1})P_{k-1}(x)\n\t\t\t\\\\\n\t\t\t\t-\n\t\t\t\t\\left\\{\n\t\t\t\t\t\\tfrac{\\alpha_k}{1+\\alpha_k}\n\t\t\t\t\t-\n\t\t\t\t\t\\rho_{k+1}\n\t\t\t\t\t\\left[\n\t\t\t\t\t\t\\tfrac{\n\t\t\t\t\t\t\t(1+\\alpha)\\delta_{k+1}^2\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t(1+\\alpha)\\delta_{k+1}-\\rho_{k+1}\n\t\t\t\t\t\t}\n\t\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\vartheta_{k+1}(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\\right]\n\t\t\t\t\\right\\}\n\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}).\n\t\t\t\\end{multline*}\n\n\t\t\tOnce again, we introduce a parameter \\(\\hat\\rho_{k+1}\\) (to be specified later) that satisfies \\eqref{eq:rhokhatgeq} whilst based on information available at iteration \\(k\\).\n\t\t\tWe can conveniently set \\(\\delta_{k+1}=\\vartheta_{k+1}=\\frac{2}{1+\\alpha}\\hat\\rho_{k+1}\\), and use the bound\n\t\t\t\\(\n\t\t\t\t(1+\\alpha)\\delta_{k+1}-\\rho_{k+1}\n\t\t\t\\geq\n\t\t\t\t(1+\\alpha)\\delta_{k+1}-\\hat\\rho_{k+1}\n\t\t\t\\)\n\t\t\tto simplify the coefficient of \\(\\Lambda_{k,\\delta_{k+1}}\\).\n\t\t\tCombined with the fact that\n\t\t\t\\(\n\t\t\t\t\\tfrac{\\alpha_k}{1+\\alpha_k}\n\t\t\t\\geq\n\t\t\t\t\\tfrac{\\alpha}{1+\\alpha}\n\t\t\t\\),\n\t\t\tthe inequality simplifies as\n\t\t\t\\begin{align*}\n\t\t\t\t\\U^{\\alpha}_{k+1}(x)\n\t\t\t\\coloneqq{} &\n\t\t\t\t\\D_{\\kernel}(x,x^{k+1})\n\t\t\t\t+\n\t\t\t\t\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t+\n\t\t\t\t\\gamma_{k+1}\\left(1+\\tfrac{2}{1+\\alpha}\\hat\\rho_{k+1}\\right)P_k(x)\n\t\t\t\\\\\n\t\t\t\\leq{} &\n\t\t\t\t\\U^{\\alpha}_k(x)\n\t\t\t\t-\n\t\t\t\t\\gamma_k(1+\\tfrac{2}{1+\\alpha}\\hat\\rho_k-\\tfrac{2}{1+\\alpha}\\hat\\rho_{k+1}\\rho_{k+1})P_{k-1}(x)\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\left\\{\n\t\t\t\t\t\\tfrac{\\alpha}{1+\\alpha}\n\t\t\t\t\t-\n\t\t\t\t\t\\tfrac{2}{1+\\alpha}\n\t\t\t\t\t\\hat\\rho_{k+1}\n\t\t\t\t\t\\rho_{k+1}\n\t\t\t\t\t\\left[\n\t\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\\right]\n\t\t\t\t\\right\\}\n\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}),\n\t\t\t\\end{align*}\n\t\t\twhere \\(\\U^{\\alpha}_k(x)\\) corresponds to \\(\\widehat{\\U}_k(x)\\) as in \\cref{thm:eq} with \\(\\vartheta_k=\\tfrac{2}{1+\\alpha}\\hat\\rho_k\\).\n\t\t\tBy imposing negativity of the coefficients of \\(P_{k-1}(x)\\) and \\(\\DD_{\\kernel}(x^k,x^{k-1})\\) as done in the previous subsection, the following analogue of \\cref{thm:BY:leq} for \\refFNEa{} is derived.\n","sha256_utf8":"fa495447373c0ff208586740e356aaf390e7da20aa3bf8ff622b49c65676f8fb"},{"start":73617,"end":74659,"start_line":1828,"end_line":1860,"raw_tex":"\\begin{lemma}\\label{thm:FNEa:leq}%\n\t\t\t\tAdditionally to \\cref{ass:basic}, suppose that \\(\\kernel\\) has symmetry coefficient \\(\\alpha>0\\) and consider the iterates generated by Bregman proximal gradient iterations \\eqref{eq:BPG} with \\(\\gamma_k\\) and \\(\\hat\\rho_k\\) selected according to \\refFNEa.\n\t\t\t\tThen, denoting\n\t\t\t\t\\begin{equation}\\label{eq:UkFNEa}\n\t\t\t\t\t\\U^{\\alpha}_k(x)\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\D_{\\kernel}(x,x^k)\n\t\t\t\t\t+\n\t\t\t\t\t\\D_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t\t+\n\t\t\t\t\t\\gamma_k\\left(1+\\tfrac{2}{1+\\alpha}\\hat\\rho_k\\right)P_{k-1}(x),\n\t\t\t\t\\end{equation}\n\t\t\t\tone has that\n\t\t\t\t\\begin{equation}\\label{eq:UkFNEa:leq}\n\t\t\t\t\t\\U^{\\alpha}_{k+1}(x)\n\t\t\t\t\\leq\n\t\t\t\t\t\\U^{\\alpha}_k(x)\n\t\t\t\t\t-\n\t\t\t\t\t\\gamma_k\\Bigl(\n\t\t\t\t\t\t\\smash{\n\t\t\t\t\t\t\t\\overbrace{\n\t\t\t\t\t\t\t\t1+\\tfrac{2}{1+\\alpha}\\hat\\rho_k-\\tfrac{2}{1+\\alpha}\\hat\\rho_{k+1}\\rho_{k+1}\n\t\t\t\t\t\t\t}^{\n\t\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t\\tfrac{2}{1+\\alpha}(\\hat\\rho_k-\\rho_k)\n\t\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t\\Bigr)P_{k-1}(x)\n\t\t\t\t\\end{equation}\n\t\t\t\tholds for any \\(x\\in\\dom\\varphi\\cap\\dom\\kernel\\) and \\(k\\in\\N\\).\n\t\t\t\\end{lemma}","sha256_utf8":"eeca379d1b18ba5709869c3cc5e05f2ae7c80695ba9bb1c6dd9534b1de1da3ed"},{"start":75180,"end":76061,"start_line":1873,"end_line":1884,"raw_tex":"\\begin{lemma}\\label{thm:descent}%\n\t\t\tSuppose that \\cref{ass:basic} holds, and consider the iterates generated by Bregman proximal gradient iterations \\eqref{eq:BPG} with \\(\\gamma_k\\) and \\(\\hat\\rho_k\\) selected according to \\refBY.\n\t\t\tThen, with \\(\\U_k\\) as in \\eqref{eq:UkBY},\n\t\t\tthe following hold for any \\(x\\in\\dom\\kernel\\) satisfying \\(\\varphi(x)\\leq\\inf_{k\\in\\N}\\varphi(x^k)\\):\n\t\t\t\\begin{enumerate}\n\t\t\t\\item \\label{thm:SD}%\n\t\t\t\t\\(\\bigl(\\U_k(x)\\bigr)_{k\\in\\N}\\) decreases and converges to a finite value.\n\t\t\t\\item \\label{thm:Pkmin}%\n\t\t\t\t\\(P_K^{\\rm min}(x)\\leq\\frac{\\U_0(x)}{\\sum_{k=1}^{K+1}\\gamma_k}\\) for every \\(K\\geq 1\\), where \\(P_K^{\\rm min}(x)\\coloneqq\\min_{k\\leq K}P_k(x)\\).\n\t\t\t\\end{enumerate}\n\t\t\tWhen \\(\\kernel\\) has symmetry coefficient \\(\\alpha>0\\), all remains true for the updates of \\refFNEa, with \\(\\U_k\\gets\\U^{\\alpha}_k\\) as in \\eqref{eq:UkFNEa}.\n\t\t\\end{lemma}","sha256_utf8":"3a0d04fde66610eecf4c628b0433176a548f2129a85b4144c97288cd58a66d08"},{"start":76064,"end":76124,"start_line":1885,"end_line":1887,"raw_tex":"\\begin{proof}\n\t\t\tSee \\cref{proof:thm:descent}.\n\t\t\\end{proof}","sha256_utf8":"24ff1bb006606ef6458c5466a1bdccdfd0d1628f0af60158f74a72e17d70652d"},{"start":80067,"end":80807,"start_line":1957,"end_line":1972,"raw_tex":"\\begin{lemma}\\label{thm:Lamkto1}%\n\t\t\t\tSuppose that \\cref{ass:basic} holds, and consider a sequence \\((x^k)_{k\\in\\N}\\) generated by Bregman proximal gradient iterations \\eqref{eq:BPG} with stepsizes \\(\\gamma_k>0\\).\n\t\t\t\tSuppose that \\((x^k)_{k\\in\\N}\\) is contained in a compact set \\(\\K\\subset C\\), and consider the ratio\n\t\t\t\t\\[\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t=\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t2\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(x^k)+\\delta_{k+1}\\bigl[H_k(x^k)-H_k(x^{k-1})\\bigr],\\nabla\\kernel(x^k)\\bigr)\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\delta_{k+1}^2\n\t\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t\t}\n\t\t\t\t\\]\n\t\t\t\tas in \\eqref{eq:Lamk}, where \\((\\delta_k)_{k\\in\\N}\\subset\\R_{++}\\) is a bounded sequence.\n\t\t\t\tIf \\(\\gamma_k\\to0\\), then \\(\\Lambda_{k,\\delta_{k+1}}\\to1\\).\n\t\t\t\\end{lemma}","sha256_utf8":"449b4c900cbeda7e1f578759419e56a1f241341b4662a46e4d124180a5b2186b"},{"start":80811,"end":84359,"start_line":1973,"end_line":2104,"raw_tex":"\\begin{proof}\n\t\t\t\tFor notational conciseness, let us denote \\(u^k\\coloneqq\\nabla\\kernel(x^k)-\\nabla\\kernel(x^{k-1})\\) and \\(v^k\\coloneqq\\nabla f(x^k)-\\nabla f(x^{k-1})\\), so that \\(H_k(x^k)-H_k(x^{k-1})=u^k-\\gamma_k v^k\\).\n\t\t\t\tFor any \\(k\\in\\N\\) there exists\n\t\t\t\t\\(\\xi^k\\) on the line segment between \\(\\nabla\\kernel(x^k)\\) and \\(\\nabla\\kernel(x^k)+\\delta_{k+1}(H_k(x^k)-H_k(x^{k-1}))\\)\n\t\t\t\tand\n\t\t\t\t\\(\\eta^k\\) on the line segment between \\(\\nabla\\kernel(x^k)\\) and \\(\\nabla\\kernel(x^{k-1})\\)\n\t\t\t\tsuch that\n\t\t\t\t\\begin{align}\n\t\t\t\t\\label{eq:Lamknum}\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t={} &\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t2\n\t\t\t\t\t\t\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(x^k)+\\delta_{k+1}\\bigl[H_k(x^k)-H_k(x^{k-1})\\bigr],\\nabla\\kernel(x^k)\\bigr)\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\delta_{k+1}^2\n\t\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t\t}\n\t\t\t\t\\\\\n\t\t\t\t\\nonumber\n\t\t\t\t={} &\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\xi_k)\n\t\t\t\t\t\t\t\\bigl[H_k(x^k)-H_k(x^{k-1})\\bigr]\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tH_k(x^k)-H_k(x^{k-1})\n\t\t\t\t\t\t}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\eta_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\\\\\n\t\t\t\t\\nonumber\n\t\t\t\t={} &\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\xi_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\eta_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\t-\n\t\t\t\t\t2\\gamma_k\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\xi_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tv^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\eta_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\t+\n\t\t\t\t\t\\gamma_k^2\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\xi_k)\n\t\t\t\t\t\t\tv^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tv^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\eta_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}.\n\t\t\t\t\\end{align}\n\t\t\t\tSince \\(\\kernel\\) is twice continuously differentiable with \\(\\nabla^2\\kernel\\succ0\\) on \\(C\\), and \\(\\K\\subset C\\) is compact and convex, one has that\n\t\t\t\t\\[\n\t\t\t\t\\textstyle\n\t\t\t\t\tL_{\\kernel,\\K}\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\sup_{\\K}\\norm{\\nabla^2\\kernel}\n\t\t\t\t=\n\t\t\t\t\t\\left(\\inf_{\\nabla\\kernel(\\K)}\\lambda_{\\rm min}(\\nabla^2\\kernel^*)\\right)^{-1}\n\t\t\t\t\\]\n\t\t\t\tis finite (\\(L_{\\kernel,\\K}\\) being the Lipschitz modulus of \\(\\nabla\\kernel\\) on \\(\\K\\)).\n\t\t\t\tAs such, one has that\n\t\t\t\t\\(\n\t\t\t\t\t\\innprod{\\nabla^2\\kernel^*(\\eta_k)u^k}{u^k}\n\t\t\t\t\\geq\n\t\t\t\t\tL_{\\kernel,\\K}^{-1}\\norm{u^k}^2\n\t\t\t\t\\)\n\t\t\t\tfor all \\(k\\).\n\t\t\t\tMoreover, letting \\(L_{f,\\K}^{\\kernel}\\) denote a smoothness modulus of \\(f\\) relative to \\(\\kernel\\) on \\(\\K\\), ensured to exist by \\cref{ass:f}, we infer from \\cite[Prop. 2.5(ii)]{ahookhosh2021bregman} that\n\t\t\t\t\\(\n\t\t\t\t\t\\norm{v^k}\\leq L_{f,\\K}^{\\kernel} L_{\\kernel,\\K}\\norm{u^k}\n\t\t\t\t\\)\n\t\t\t\tholds for all \\(k\\).\n\t\t\t\tTherefore,\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\abs{\\Lambda_{k,\\delta_{k+1}}-1}\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\norm{\\nabla^2\\kernel^*(\\xi_k)-\\nabla^2\\kernel^*(\\eta_k)}\n\t\t\t\t\tL_{\\kernel,\\K}\n\t\t\t\t\t+\n\t\t\t\t\t2\\gamma_k\n\t\t\t\t\tL_{f,\\K}^{\\kernel} L_{\\kernel,\\K}^2\n\t\t\t\t\t\\norm{\\nabla^2\\kernel^*(\\xi_k)}\n\t\t\t\t\\\\\n\t\t\t\t&\n\t\t\t\t\t+\n\t\t\t\t\t\\gamma_k^2\n\t\t\t\t\tL_{\\kernel,\\K}^3\n\t\t\t\t\t(L_{f,\\K}^{\\kernel})^2\n\t\t\t\t\t\\norm{\\nabla^2\\kernel^*(\\xi_k)}.\n\t\t\t\t\\end{align*}\n\t\t\t\tIf \\(\\gamma_k\\to0\\), then eventually \\(\\gamma_k<\\frac{1}{L_{f,\\K}^{\\kernel}}\\) and standard results ensure that \\((x^k)_{k\\in\\N}\\) converges to some point \\(x\\in\\K\\).\n\t\t\t\tIn this case, \\((\\xi^k)_{k\\in\\N}\\) and \\((\\eta^k)_{k\\in\\N}\\) converge to \\(\\nabla\\kernel(x)\\) (the former sequence because \\((\\delta_k)_{k\\in\\N}\\) is bounded), and by continuity of \\(\\nabla^2\\kernel^*\\) on \\(\\K\\) the right-hand side in the above inequality vanishes.\n\t\t\t\\end{proof}","sha256_utf8":"fcab76f068a38fe1da06435243fad0a9a94627b989a76f8551cf34e3cf16cc15"},{"start":84729,"end":85443,"start_line":2109,"end_line":2125,"raw_tex":"\\begin{lemma}\\label{lem:prox:property:optimal}%\n\t\t\t\tAdditionally to \\cref{ass:basic} suppose that \\(\\argmin_C\\varphi\\neq\\emptyset\\).\n\t\t\t\tLet a sequence \\((y^k)_{k\\in\\N}\\) contained in \\(C\\) and converging to a point \\(y^\\star\\in\\argmin_C\\varphi\\) be fixed, and for every \\(k\\in\\N\\) let\n\t\t\t\t\\begin{equation}\\label{eq:baryk}\n\t\t\t\t\t\\bar y^k\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\argmin_w\\left\\{\n\t\t\t\t\t\tg(w)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla f(y^k)}{w-y^k}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(w,y^k)\n\t\t\t\t\t\\right\\}\n\t\t\t\t\\end{equation}\n\t\t\t\twhere \\((\\gamma_k)_{k\\in\\N}\\) is a bounded sequence of strictly positive stepsizes.\n\t\t\t\tThen, \\(\\bar y^k\\to y^\\star\\) and \\(\\gamma_{k+1}(\\varphi(\\bar y^k) - \\min\\varphi)\\to0\\).\n\t\t\t\\end{lemma}","sha256_utf8":"00974878c62b3bda6564c12b288eb8520a8d90d2eb375f2c44fb8e902e419c1b"},{"start":85447,"end":87901,"start_line":2126,"end_line":2179,"raw_tex":"\\begin{proof}\n\t\t\t\tConsider the ``left'' Bregman proximal operator \\(\\prox_{g}:\\interior\\dom\\kernel\\to\\dom\\kernel\\) defined as\n\t\t\t\t\\[\n\t\t\t\t\t\\prox_{g}(y)\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\argmin_{w\\in\\R^n}\\{g(w)+\\D_{\\kernel}(w,y)\\}\n\t\t\t\t\\]\n\t\t\t\t(note that \\(\\prox_{g}(y)\\in\\interior\\dom\\kernel\\subseteq\\dom\\kernel\\) for any \\(y\\in\\interior\\dom\\kernel\\), owing to \\cref{ass:basic}).\n\t\t\t\tThe mapping \\(\\prox_{\\gamma_{k+1}g}\\circ\\nabla\\kernel^*\\) is \\(\\nabla\\kernel\\)-firmly-nonexpansive, in the sense that\n\t\t\t\t\\begin{equation}\\label{eq:BPG:FNE}\n\t\t\t\t\t\\DD_{\\kernel}(\\bar y_1,\\bar y_2)\n\t\t\t\t\\leq\n\t\t\t\t\t\\innprod{\\eta_1-\\eta_2}{\\bar y_1-\\bar y_2}\n\t\t\t\t\\end{equation}\n\t\t\t\tholds for any \\(\\eta_i\\in\\R^2\\) and \\(\\bar y_i = \\prox_{\\gamma_{k+1} g}\\circ\\nabla\\kernel^*(\\eta_i)\\), \\(i=1,2\\);\n\t\t\t\ta proof of this fact can be found in \\cite[Lem. 4.2]{wang2022bregman} (see also \\cite[Thm. 4.9]{wang2025bregman} for its \\emph{equivalence} to convexity of \\(g\\)).\n\t\t\t\tNote that \\eqref{eq:baryk} can equivalently be written as\n\t\t\t\t\\[\n\t\t\t\t\t\\bar y^k\n\t\t\t\t=\n\t\t\t\t\t\\prox_{\\gamma_{k+1} g}\\circ\\nabla\\kernel^*(H_{k+1}(y^k)),\n\t\t\t\t\\]\n\t\t\t\twhere we remind that \\(H_{k+1}=\\nabla\\kernel-\\gamma_{k+1}\\nabla f\\).\n\t\t\t\tUsing \\(\\nabla\\kernel\\)-firm nonexpansiveness and recalling that \\(y^\\star = \\prox_{\\gamma_{k+1} g}\\circ\\nabla\\kernel^*(H_{k+1}(y^\\star))\\) we have\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\DD_{\\kernel}(\\bar y^k,y^\\star)\n\t\t\t\t\\stackrel{\\text{\\clap{\\eqref{eq:BPG:FNE}}}}{\\leq}{} &\n\t\t\t\t\t\\innprod{H_{k+1}(y^k) - H_{k+1}(y^\\star)}{\\bar y^k - y^\\star}\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\Bigl(\n\t\t\t\t\t\t\\norm{\\nabla \\kernel(y^k) - \\nabla \\kernel(y^\\star)}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\gamma_{k+1}\\norm{\\nabla f(y^k) - \\nabla f(y^\\star)}\n\t\t\t\t\t\\Bigr)\n\t\t\t\t\t\\norm{\\bar y^k - y^\\star},\n\t\t\t\t\\end{align*}\n\t\t\t\twhich vanishes as \\(k\\to\\infty\\).\n\t\t\t\tThen, by essential strict convexity of \\(\\kernel\\), it follows that \\(\\bar y^k\\to y^\\star\\).\n\t\t\t\tMoreover, by the subgradient inequality, for every \\(k \\in \\N\\) it holds that\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t0\n\t\t\t\t\\leq\n\t\t\t\t\t\\gamma_{k+1}(\\varphi(\\bar y^k) - \\min\\varphi)\n\t\t\t\t={} &\n\t\t\t\t\t\\gamma_{k+1}(f(\\bar y^k) + g(\\bar y^k) - \\min\\varphi)\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\gamma_{k+1}(f(\\bar y^k) - f(y^\\star))\n\t\t\t\t\\\\\n\t\t\t\t&\n\t\t\t\t\t-\\innprod{\\nabla \\kernel(y^k) - \\gamma_{k+1}\\nabla f(y^k) - \\nabla \\kernel(\\bar y^k)}{y^\\star - \\bar y^k}.\n\t\t\t\t\\end{align*}\n\t\t\t\tThe proof then follows from continuity of \\(f\\) and the fact that the inner product vanishes, since both \\(y^k\\) and \\(\\bar y^k\\) converge to \\(y^\\star\\).\n\t\t\t\\end{proof}","sha256_utf8":"489cfd3f4c53d3d85d2474560dd0175a0d3dff751f8acb495f6ead06fb2823d7"},{"start":88954,"end":89507,"start_line":2193,"end_line":2202,"raw_tex":"\\begin{theorem}\\label{thm:xinC}%\n\t\t\t\tAdditionally to \\cref{ass:basic}, suppose that\n\t\t\t\t\\begin{equation}\\label{eq:xinC}\n\t\t\t\t\t\\text{there exists \\(x\\in C\\) such that \\(\\varphi(x)\\leq\\inf_{k\\in\\N}\\varphi(x^k)\\)}\n\t\t\t\t\\end{equation}\n\t\t\t\tholds for the iterates \\(x^k\\) generated by \\refBY.\n\t\t\t\tThen, the sequence \\((x^k)_{k\\in\\N}\\) converges to a solution \\(x^\\star\\in\\argmin_C\\varphi\\neq\\emptyset\\) and \\(\\varphi(x^k)\\to\\inf_{\\overline C}\\varphi\\).\n\n\t\t\t\tWhen \\(\\kernel\\) has symmetry coefficient \\(\\alpha>0\\), the same is true for \\refFNEa.\n\t\t\t\\end{theorem}","sha256_utf8":"dd1e936fe39fc2d02797c56e4b537b999c8e598957a6e9a41d0fadb41409c77e"},{"start":89509,"end":90039,"start_line":2204,"end_line":2206,"raw_tex":"\t\t\tThe fact that this result subsumes \\cref{thm:main:C} is obvious by observing that the validity of \\cref{thm:xinC} implies that any \\(x\\) complying with \\eqref{eq:xinC} must necessarily belong to \\(\\argmin_{\\overline C}\\varphi\\cap C\\).\n\t\t\tThe proof of \\cref{thm:xinC} will be carried out via intermediate claims.\n\t\t\tIn what follows, we consider \\(\\U_k\\) as in \\eqref{eq:UkBY} in case of \\refBY{}; under the needed symmetry assumption, the same proof applies to \\refFNEa{} as well by simply replacing \\(\\U_k\\gets\\U^{\\alpha}_k\\).\n","sha256_utf8":"7d75d0bed31e1c4c3f11ce0291dc628e2d355a74677eb3698559f8c567139e12"},{"start":90043,"end":90182,"start_line":2208,"end_line":2210,"raw_tex":"\\begin{claim}\\label{thm:xinC:boundedseq}\n\t\t\t\tThere exists a compact set \\(\\K\\subset C\\) containing all the iterates \\(x^k\\).\n\t\t\t\\end{claim}","sha256_utf8":"4111c27e8715dc17976d42104418a5addd33cb1552a6b8450eecb7454bdda21f"},{"start":90186,"end":90446,"start_line":2211,"end_line":2222,"raw_tex":"\\begin{proof}\n\t\t\t\tIn view of \\eqref{eq:xinC}, \\cref{thm:SD} implies that\n\t\t\t\t\\[\n\t\t\t\t\t\\D_{\\kernel}(x,x^k)\n\t\t\t\t\\leq\n\t\t\t\t\t\\U_k(x)\n\t\t\t\t\\leq\n\t\t\t\t\t\\U_0(x)\n\t\t\t\t\\]\n\t\t\t\tholds for any \\(k\\in\\N\\).\n\t\t\t\tThe assertion then follows from \\cref{fact:D:coercive}.\n\t\t\t\\end{proof}","sha256_utf8":"545119792f6143a81f6c4fb394d88287158eb41d137a0e7a9f44e742f3f0bcee"},{"start":90451,"end":90505,"start_line":2224,"end_line":2226,"raw_tex":"\\begin{claim}\n\t\t\t\t\\(\\gamma_k\\not\\to0\\).\n\t\t\t\\end{claim}","sha256_utf8":"ab722e4a594378ef824942d5760cd44b279e86a8b059497ac1e11ab2cda50053"},{"start":90509,"end":91454,"start_line":2227,"end_line":2237,"raw_tex":"\\begin{proof}\n\t\t\t\tTo arrive to a contradiction, suppose that \\(\\gamma_k\\to0\\).\n\t\t\t\tThen, \\cref{thm:Lamkto1} implies that eventually the quantity\n\t\t\t\t\\(\n\t\t\t\t\t[\\Lambda_{k,\\delta}-(1-\\gamma_k\\ell_k)]_+\n\t\t\t\t\\)\n\t\t\t\tappearing in both \\eqref{eq:BY:rhok*} and \\eqref{eq:FNEa:rhok*} vanishes.\n\t\t\t\tSince \\((x^k)_{k\\in\\N}\\) is bounded and bounded away from the boundary of \\(C\\), regardless of whether or not \\(\\kernel\\) has a symmetry coefficient \\(\\alpha>0\\), it holds that \\(\\inf_{k\\in\\N}\\alpha_k>0\\);\n\t\t\t\tfurthermore, for both \\refBY{} and \\refFNEa{} note that \\(\\hat\\rho_{k+1}\\leq\\sqrt{1+\\rho_k}\\leq\\sqrt{1+\\hat\\rho_k}\\), implying that \\(\\hat\\rho_k\\leq\\frac{1+\\sqrt{5}}{2}\\) holds for any \\(k\\).\n\t\t\t\tAs such, in both \\eqref{eq:BY:rhok*} and \\eqref{eq:FNEa:rhok*} eventually the second element in the minimum is infinite, implying that the update reduces to \\(\\gamma_{k+1}=\\hat\\rho_{k+1}\\gamma_k\\) and is thus divergent, a contradiction.\n\t\t\t\\end{proof}","sha256_utf8":"4241f59f3d23c3da07b74a483129cecab38665fde950a530d58bcff53d3e6adf"},{"start":91459,"end":91643,"start_line":2239,"end_line":2241,"raw_tex":"\\begin{claim}\n\t\t\t\t\\(\\inf_{k\\in\\N}\\varphi(x^k)=\\inf_{\\overline C}\\varphi\\) (in particular, necessarily \\(x\\in\\argmin_C\\varphi\\) for any \\(x\\) satisfying \\eqref{eq:xinC}).\n\t\t\t\\end{claim}","sha256_utf8":"88613ace57d3d63016be59ec1e6b3010d6436d9056ccdc0d1e20437efd332094"},{"start":91647,"end":92153,"start_line":2242,"end_line":2247,"raw_tex":"\\begin{proof}\n\t\t\t\tHaving shown that \\(\\gamma_k\\not\\to0\\), one has that \\(\\sum_{k\\in\\N}\\gamma_k=\\infty\\).\n\t\t\t\t\\Cref{thm:Pkmin} then implies that \\(\\inf_{k\\in\\N}\\varphi(x^k)=\\varphi(x)\\).\n\t\t\t\tFrom the arbitrariness of the point \\(x\\) as in \\eqref{eq:xinC} we conclude that necessarily \\(x\\in\\argmin_C\\varphi\\subseteq\\argmin_{\\overline C}\\varphi\\), with inclusion holding by virtue of \\cite[Prop. 11.1(iv)]{bauschke2017convex}.\n\t\t\t\tHence \\(\\inf_{k\\in\\N}\\varphi(x^k)=\\inf_{\\overline C}\\varphi\\).\n\t\t\t\\end{proof}","sha256_utf8":"3d9ba31e772de9c7d997dda4129ad30cefd8d17687471cd65e0716d2ae1faaed"},{"start":92158,"end":92288,"start_line":2249,"end_line":2251,"raw_tex":"\\begin{claim}\\label{claim:unique}%\n\t\t\t\tThere exists exactly one optimal limit point \\(x^\\star\\in\\argmin_C\\varphi\\).\n\t\t\t\\end{claim}","sha256_utf8":"a8e76f83f828859b7f7a7216e8206c0214fd689eb11ae9134171760c11d97349"},{"start":92292,"end":93341,"start_line":2252,"end_line":2272,"raw_tex":"\\begin{proof}\n\t\t\t\tThe existence of an optimal limit point is guaranteed by the previous claim, since \\((x^k)_{k\\in\\N}\\) is bounded (and bounded away from the boundary of \\(C\\)) and \\(\\varphi\\) is lsc.\n\t\t\t\tConsider two optimal limit points \\(\\bar x_1\\) and \\(\\bar x_2\\), and two corresponding subsequences \\((x^k)_{k\\in K_1}\\to\\bar x_1\\) and \\((x^k)_{k\\in K_2}\\to\\bar x_2\\).\n\t\t\t\tNotice that\n\t\t\t\t\\[\n\t\t\t\t\t\\U_k(\\bar x_1)-\\U_k(\\bar x_2)\n\t\t\t\t=\n\t\t\t\t\t\\D_{\\kernel}(\\bar x_1,x^k)-\\D_{\\kernel}(\\bar x_2,x^k)\n\t\t\t\t\\]\n\t\t\t\tconverges to some finite value \\(V\\), because both \\(\\U_k(\\bar x_1)\\) and \\(\\U_k(\\bar x_2)\\) are convergent.\n\t\t\t\tConsidering the limits along \\(k\\in K_1\\) and \\(k\\in K_2\\) yields that\n\t\t\t\t\\[\n\t\t\t\t\tV\n\t\t\t\t=\n\t\t\t\t\t\\D_{\\kernel}(\\bar x_1,\\bar x_1)-\\D_{\\kernel}(\\bar x_2,\\bar x_1)\n\t\t\t\t=\n\t\t\t\t\t\\D_{\\kernel}(\\bar x_1,\\bar x_2)-\\D_{\\kernel}(\\bar x_2,\\bar x_2),\n\t\t\t\t\\]\n\t\t\t\thence that \\(-\\D_{\\kernel}(\\bar x_2,\\bar x_1)=\\D_{\\kernel}(\\bar x_1,\\bar x_2)\\).\n\t\t\t\tThus \\(\\DD_{\\kernel}(\\bar x_1,\\bar x_2)=0\\), implying that \\(\\bar x_1=\\bar x_2\\).\n\t\t\t\\end{proof}","sha256_utf8":"b90c97ee5c77737e8154e46c5725e5b74472fee6a64706e394a540ceb4f04487"},{"start":93343,"end":99972,"start_line":2274,"end_line":2419,"raw_tex":"\t\t\tIn light of the previous claim, the proof of \\cref{thm:xinC} is completed once we show that \\(x^k\\to x^\\star\\).\n\t\t\tTo this end, owing to the fact that \\(\\D_{\\kernel}(x^\\star,x^k)\\leq\\U_k(x^\\star)\\) it will suffice to show that \\(U\\coloneqq\\lim_{k\\to\\infty}\\U_k(x^\\star)\\) is zero, (existence and finiteness of \\(U\\) is ensured by \\cref{thm:SD}).\n\t\t\t\\begin{itemize}\n\t\t\t\\item\n\t\t\t\tLet us consider a subsequence \\((x^k)_{k\\in K}\\) converging to \\(x^\\star\\).\n\t\t\t\tIf the corresponding subsequence \\((\\gamma_{k+1})_{k\\in K}\\) is bounded, then it follows from \\cref{lem:prox:property:optimal} that  \\(x^{k+1}\\to x^\\star\\) and \\(\\gamma_{k+1} P_k(x^\\star) \\to 0\\) as well, and from the expression of \\(\\U_k\\) together with the fact that \\(\\hat\\rho_{k+1}\\) is bounded and \\(\\inf_k \\alpha_k>0\\), it is clear that\n\t\t\t\t\\(\\U_{k+1}(x^\\star)\\to0\\) as \\(K\\ni k\\to\\infty\\).\n\n\t\t\t\\item\n\t\t\t\tIn what follows, let us instead consider the complementary case in which for any subsequence \\(x^k\\) converging to \\(x^\\star\\) the corresponding subsequence of stepsizes \\((\\gamma_{k+1})_{k\\in K}\\) is divergent.\n\t\t\t\tIn this case, note that since \\(\\gamma_{k+1}P_k(x^\\star)\\leq\\U_{k+1}(x^\\star)\\leq\\U_0(x^\\star)\\), we have\n\t\t\t\t\\begin{align}\n\t\t\t\t\\nonumber\n\t\t\t\t\t(x^k)_{k\\in K}\\to x^\\star\n\t\t\t\t\\Leftrightarrow{} &\n\t\t\t\t\t(\\gamma_{k+1})_{k\\in K}\\to\\infty\n\t\t\t\t\\Leftrightarrow\n\t\t\t\t\t\\bigl(P_k(x^\\star)\\bigr)_{k\\in K}\\to0\n\t\t\t\t\\\\\n\t\t\t\t\\nonumber\n\t\t\t\t\\Rightarrow{} &\n\t\t\t\t\t(\\gamma_k)_{k\\in K}\\to\\infty\n\t\t\t\t\\\\\n\t\t\t\t\\Leftrightarrow{} &\n\t\t\t\t\t(x^{k-1})_{k\\in K}\\to x^\\star\n\t\t\t\t\\Leftrightarrow\n\t\t\t\t\t\\bigl(P_{k-1}(x^\\star)\\bigr)_{k\\in K}\\to0,\n\t\t\t\t\\label{eq:gamk:infty\"gamk-1}\n\t\t\t\t\\end{align}\n\t\t\t\twhere the right implication follows from the fact that \\(\\gamma_{k+1}\\leq\\hat\\rho_{k+1}\\gamma_k\\leq\\rho_{\\rm max}\\gamma_k\\), for some \\(\\rho_{\\rm max}\\leq\\frac{1+\\sqrt{5}}{2}\\).\n\t\t\t\tSince an optimal limit exists, it follows that \\(\\sup_{k\\in\\N}\\gamma_k=\\infty\\).\n\n\t\t\t\tIn what follows, we construct a specific subsequence \\(K\\coloneqq\\{k_0,k_1,\\dots\\}\\) along which \\((\\gamma_k)_{k\\in K}\\) diverges.\n\t\t\t\tIn doing so, we expand upon the arguments in the proof of \\cite[Thm. 2.3]{latafat2024adaptive} to account for the complications of the non-Euclidean setting investigated here.\n\t\t\t\tFor \\(i\\geq0\\) let\n\t\t\t\t\\begin{equation}\\label{eq:ki+}\n\t\t\t\t\tk_{i+1}=\\min\\{k\\geq k_i \\mid \\gamma_k \\geq \\rho_{\\rm max}\\gamma_{k_i}\\}.\n\t\t\t\t\\end{equation}\n\t\t\t\tThen, \\(\\bigl(\\gamma_{k_i}\\bigr)_{i\\in\\N}\\to\\infty\\), which by \\eqref{eq:gamk:infty\"gamk-1} implies that\n\t\t\t\t\\begin{equation}\\label{eq:xkitox*}\n\t\t\t\t\t\\lim_{i\\to\\infty}x^{k_i-s}\n\t\t\t\t=\n\t\t\t\t\tx^\\star\n\t\t\t\t\\quad\n\t\t\t\t\t\\forall s=1,2,3\n\t\t\t\t\\end{equation}\n\t\t\t\t(in fact, for any \\(s\\in\\N\\)).\n\t\t\t\tIn light of \\cref{thm:SD} we have that\n\t\t\t\t\\begin{equation}\\label{eq:U:gamkPk-1}\n\t\t\t\t\tU= \\lim_{k \\to \\infty} \\U_k= \\lim_{i \\to \\infty} \\U_{k_i-1}(x^\\star) = \\lim_{i \\to \\infty} \\gamma_{k_i-1}(1+ \\vartheta_{k_i-1})P_{k_i-2}(x^\\star).\n\t\t\t\t\\end{equation}\n\n\t\t\t\tWe proceed to show that \\(\\gamma_{k-1} P_{k-2}(x^\\star)\\) converges to zero along the same subsequence.\n\t\t\t\t\\begin{subequations}\n\t\t\t\t\tFor every \\(i\\in\\N\\), note that\n\t\t\t\t\t\\begin{gather}\n\t\t\t\t\t\t\\rho_{k_i}>1\n\t\t\t\t\t\t~\\text{ and }~\n\t\t\t\t\t\tk_{i-1}\\leq k_i-1\n\t\t\t\t\t\\shortintertext{by minimality of \\(k_i\\) and the fact that \\(\\rho_{\\rm max}>1\\), hence that}\n\t\t\t\t\t\\label{eq:k_i-1}\n\t\t\t\t\t\t\\text{either }~\n\t\t\t\t\t\t\\rho_{k_i-1}>1\n\t\t\t\t\t\t~\\text{ or }~\n\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\tk_i-1\\notin K\n\t\t\t\t\t\t\t\\text{ and thus }\n\t\t\t\t\t\t\tk_{i-1}\\leq k_i-2\n\t\t\t\t\t\t\\bigr).\n\t\t\t\t\t\\shortintertext{These combined imply also that}\n\t\t\t\t\t\\label{eq:k-1}\n\t\t\t\t\t\t\\rho_{k_i-1}\\geq\\rho_{\\rm max}^{-1}.\n\t\t\t\t\t\\end{gather}\n\t\t\t\t\tIndeed, if not, then \\(\\rho_{k_i-1}<\\rho_{\\rm max}^{-1}<1\\), implying by \\eqref{eq:k_i-1} that \\(k_{i-1}\\leq k_i-2\\);\n\t\t\t\t\tthis would lead to the contradiction\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\rho_{\\rm max}\\gamma_{k_{i-1}}\n\t\t\t\t\t\\stackrel{\\text{\\clap{\\eqref{eq:ki+}}}}{\\leq}\n\t\t\t\t\t\t\\gamma_{k_i}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\rho_{\\rm max}\\gamma_{k_i-1}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\rho_{\\rm max}\\rho_{k_i-1}\\gamma_{k_i-2}\n\t\t\t\t\t\\stackrel{\\mathclap{\\bot}}{<}\n\t\t\t\t\t\t\\gamma_{k_i-2}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\rho_{\\rm max}\\gamma_{k_{i-1}},\n\t\t\t\t\t\\]\n\t\t\t\t\twhere ``\\(\\bot\\)'' marks where the contradictory inequality is used, and the last inequality follows both in case \\(k_i-2=k_{i-1}\\) (since \\(\\rho_{\\rm max}>1\\)) or when \\(k_{i-1}<k_i-2<k_i\\) (from minimality in the definition of \\(k_i\\)).\n\t\t\t\t\\end{subequations}\n\t\t\t\tTherefore, \\eqref{eq:k-1} holds true, implying in particular that the sequence\n\t\t\t\t\\begin{equation}\\label{eq:Lambda:k-2:bounded}\n\t\t\t\t\t\\bigl(\\Lambda_{k_i-2, 2\\hat\\rho_{k_i-1}}\\bigr)_{i\\in\\N}\n\t\t\t\t\\quad\n\t\t\t\t\t\\text{is bounded,}\n\t\t\t\t\\end{equation}\n\t\t\t\tfor otherwise the second term in either \\eqref{eq:BY:rhok*} or \\eqref{eq:FNEa:rhok*} would vanish, owing to the fact that \\(\\hat\\rho_k\\) is lower bounded by \\(\\sqrt{1/2}\\) and similarly \\(\\alpha_k\\) is bounded away from zero.\n\n\t\t\t\tLet \\(v^k\\coloneqq H_k(x^{k-1})- H_k(x^k)\\in\\gamma_k\\partial\\varphi(x^k)\\), and observe that\n\t\t\t\t\\begin{align*}\n\t\t\t\t&\n\t\t\t\t\t\\gamma_k P_{k-1}(x^\\star)\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\rho_k\\innprod{x^\\star - x^{k-1}}{-v^{k-1}}\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{\\rho_k}{2\\hat\\rho_k}\\D_{\\kernel}(x^\\star, x^{k-1})\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\rho_k}{2\\hat\\rho_k}\n\t\t\t\t\t\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(x^{k-1})-2\\hat\\rho_kv^{k-1}\\bigr), \\nabla \\kernel(x^{k-1})\\bigr)\n\t\t\t\t\\\\\n\t\t\t\t={} &\n\t\t\t\t\t\\tfrac{\\rho_k}{2\\hat\\rho_k}\\D_{\\kernel}(x^\\star, x^{k-1})\n\t\t\t\t\t+\n\t\t\t\t\t2\\rho_k \\hat\\rho_k^2\\Lambda_{k-1, 2\\hat\\rho_k} \\DD_{\\kernel}(x^{k-1},x^{k-2})\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{1}{2}\\D_{\\kernel}(x^\\star, x^{k-1})\n\t\t\t\t\t+\n\t\t\t\t\t2\\rho_{\\rm max}^3 \\Lambda_{k-1, 2\\hat\\rho_k} \\DD_{\\kernel}(x^{k-1},x^{k-2})\n\t\t\t\t\\end{align*}\n\t\t\t\tholds for any \\(k\\), where we used the Bregman--Young inequality \\eqref{eq:BY} with parameter \\(\\delta= \\hat\\rho_k\\) in the second inequality, and \\(\\rho_k\\leq \\hat\\rho_k\\leq \\rho_{\\rm max}\\) in the last one.\n\t\t\t\tRecall that \\(\\bigl(x^{k_i-3}\\bigr)_{i\\in\\N} \\to x^\\star\\) by \\eqref{eq:xkitox*}.\n\t\t\t\tThus,\n\t\t\t\t\\[\n\t\t\t\t\t\\gamma_{k_i-1} P_{k_i-2}(x^\\star)\n\t\t\t\t\\leq\n\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t\\underbrace{\n\t\t\t\t\t\t\\D_{\\kernel}(x^\\star, x^{k_i-2})\n\t\t\t\t\t\t\\vphantom{\\Lambda_{k_i-2, 2\\rho_{k_i-1}}}\n\t\t\t\t\t}_{\\to 0 \\text{ by \\eqref{eq:xkitox*}}}\n\t\t\t\t\t+\n\t\t\t\t\t2\\rho_{\\rm max}^3\n\t\t\t\t\t\\underbrace{\n\t\t\t\t\t\t\\Lambda_{k_i-2, 2\\rho_{k_i-1}}\n\t\t\t\t\t}_{\\text{\\rm bounded}}\n\t\t\t\t\t\\,\n\t\t\t\t\t\\underbrace{\n\t\t\t\t\t\t\\DD_{\\kernel}(x^{k_i-2},x^{k_i-3})\n\t\t\t\t\t\t\\vphantom{\\Lambda_{k_i-2, 2\\rho_{k_i-1}}}\n\t\t\t\t\t}_{\\to 0 \\text{ by \\eqref{eq:xkitox*}}}.\n\t\t\t\t\\]\n\t\t\t\tUsing this in \\eqref{eq:U:gamkPk-1} and noting that \\(\\theta_{k_i-1}\\) being a multiple of \\(\\hat{\\rho}_{k_i-1}\\) in both \\refBY{} and \\refFNEa{} is bounded, completes the proof.\n\t\t\t\\end{itemize}\n","sha256_utf8":"5ac113cffd92a46dcf4621f11f713ee18684a7819e85457d4b47fbc13f4957ce"},{"start":100096,"end":103169,"start_line":2424,"end_line":2472,"raw_tex":"\\subsection{Proof of \\texorpdfstring{\\cref{eq:main:inf}}{Equation \\ref*{eq:main:inf}}}\n\t\t\tIn this subsection we prove that the iterates generated by either \\refBY{} or \\refFNEa{} in the generality of \\cref{ass:basic} (in addition to \\(\\kernel\\) having a symmetry coefficient \\(\\alpha>0\\) in the latter case) are such that \\(\\inf_{k\\in\\N}\\varphi(x^k)=\\inf_{\\overline C}\\varphi\\).\n\n\t\t\tTo see this, contrary to the claim suppose that \\(\\inf_{k\\in\\N}\\varphi(x^k)>\\inf_{\\overline C}\\varphi\\).\n\t\t\tThen, since \\(\\inf_{\\overline C}\\varphi=\\inf_C\\varphi\\) by \\cite[Prop. 11.1(iv)]{bauschke2017convex}, there exists \\(x\\in C\\) such that \\(\\varphi(x^k)\\geq\\varphi(x)\\) holds for all \\(k\\in\\N\\).\n\t\t\tInvoking \\cref{thm:xinC} yields a contradiction.\n\n\n\n\t\t%% ░░░░ 4.5 Proof of Theorem 2.7(ii) ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░\n\t\t\\subsection{Proof of \\texorpdfstring{\\cref{thm:main:zone}}{Theorem \\ref*{thm:main:zone}}}\n\t\t\tIn this subsection we prove that whenever \\cref{ass:basic,ass:zone} are satisfied, the iterates generated by \\refBY{} are bounded and admit exactly one optimal limit point.\\footnote{%\n\t\t\t\t\\label{foot:noFNEa}%\n\t\t\t\tAs commented after \\cref{thm:main}, there is no loss of generality in considering only \\refBY{} and disregard the claim for \\refFNEa.\n\t\t\t}\n\t\t\tWe can without loss of generality assume that \\(\\argmin_{C}\\varphi=\\emptyset\\), for otherwise a stronger result is already covered by \\cref{thm:main:C}.\n\n\t\t\tUnder \\cref{ass:zone}, \\(\\dom\\kernel=\\overline C\\), and therefore in \\cref{thm:SD} one can take any \\(x\\in\\argmin_{\\overline C}\\varphi\\).\n\t\t\tIn particular, \\(\\D_{\\kernel}(x,x^k)\\leq\\U_k(x)\\leq\\U_0(x)\\) holds for any \\(k\\).\n\t\t\tSince \\(\\D_{\\kernel}(x,{}\\cdot{})\\) is level bounded by \\cref{ass:xkbounded}, boundedness of \\((x^k)_{k\\in\\N}\\) follows.\n\t\t\tMoreover, we know from \\eqref{eq:main:inf} that \\(\\varphi(x^k)\\to\\inf_{\\overline C}\\varphi\\) along a subsequence, and therefore an optimal limit point \\(x^\\star\\) exists.\n\t\t\tTo assess its uniqueness, we can argue similarly to the proof of \\cref{claim:unique}, with the minor catch that now such limit points are on the boundary of \\(C\\).\n\t\t\tConsidering two optimal limit points \\(\\bar x_1\\) and \\(\\bar x_2\\), and two corresponding subsequences \\((x^k)_{k\\in K_1}\\to\\bar x_1\\) and \\((x^k)_{k\\in K_2}\\to\\bar x_2\\), we still have that\n\t\t\t\\[\n\t\t\t\t\\U_k(\\bar x_1)-\\U_k(\\bar x_2)\n\t\t\t=\n\t\t\t\t\\D_{\\kernel}(\\bar x_1,x^k)-\\D_{\\kernel}(\\bar x_2,x^k)\n\t\t\t\\]\n\t\t\tconverges to some finite value \\(V\\).\n\t\t\tAgain by considering the limit along the two subsequences \\(k\\in K_1\\) and \\(k\\in K_2\\), \\cref{ass:Dhto0} yields that\n\t\t\t\\[\n\t\t\t\t-\\lim_{k\\in K_2}\\D_{\\kernel}(\\bar x_1,x^k)\n\t\t\t=\n\t\t\t\t\\lim_{k\\in K_1}\\D_{\\kernel}(\\bar x_2,x^k),\n\t\t\t\\]\n\t\t\thence that\n\t\t\t\\[\n\t\t\t\t\\lim_{k\\in K_2}\\D_{\\kernel}(\\bar x_1,x^k)\n\t\t\t=\n\t\t\t\t0\n\t\t\t=\n\t\t\t\t\\lim_{k\\in K_1}\\D_{\\kernel}(\\bar x_2,x^k),\n\t\t\t\\]\n\t\t\twhich by \\cref{fact:D:solodov} implies that \\(\\bar x_1=\\bar x_2\\).\n\n\n\n\t\t%% ░░░░ 4.6 Bregman proximal gradient with backtracking ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░\n\t\t","sha256_utf8":"13ec605652008cef927879b5ca802fa4a13586c1fa934e8cb41f9abfd9abb72e"},{"start":151530,"end":154945,"start_line":3240,"end_line":3397,"raw_tex":"\\begin{proof}[\\textbf{Proof of \\cref{thm:eq}}]%\n\t\t\tBased on the subgradient characterization \\eqref{eq:subgradvarphi} and the definition of \\(\\ell_k\\) in \\eqref{eq:lk}, we have\n\t\t\t\\begin{align}\n\t\t\t\\nonumber\n\t\t\t\t0\n\t\t\t={} &\n\t\t\t\t\\varphi(x^{k-1})\n\t\t\t\t-\n\t\t\t\t\\varphi(x^k)\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_k}\n\t\t\t\t\\innprod{H_k(x^{k-1})-H_k(x^k)}{x^{k-1}-x^k}\n\t\t\t\t-\n\t\t\t\t\\tD_\\varphi(x^{k-1},x^k)\n\t\t\t\\\\\n\t\t\t={} &\n\t\t\t\tP_{k-1}(x)\n\t\t\t\t-\n\t\t\t\tP_k(x)\n\t\t\t\t-\n\t\t\t\t\\tfrac{1-\\gamma_k\\ell_k}{\\gamma_k}\n\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t-\n\t\t\t\t\\tD_\\varphi(x^{k-1},x^k).\n\t\t\t\\label{eq:thetk}\n\t\t\t\\end{align}\n\t\t\tFurthermore, by leveraging the subgradient characterization \\eqref{eq:subgrad} and applying the three point identity of \\cref{thm:3p},\n\t\t\t\\begin{align}\n\t\t\t\\nonumber\n\t\t\t\t0\n\t\t\t={} &\n\t\t\t\tg(x)-g(x^{k+1})-\\innprod{\\tilde{\\nabla}g(x^{k+1})}{x-x^{k+1}}-\\tD_g(x,x^{k+1})\n\t\t\t\\\\\n\t\t\t\\nonumber\n\t\t\t={} &\n\t\t\t\tg(x)-g(x^{k+1})\n\t\t\t\t+\n\t\t\t\t\\innprod{\\nabla f(x^k)}{x- x^{k+1}}\n\t\t\t\\\\\n\t\t\t\\nonumber\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\innprod{\\nabla\\kernel(x^k)-\\nabla\\kernel(x^{k+1})}{x-x^{k+1}}\n\t\t\t\t-\n\t\t\t\t\\tD_g(x,x^{k+1})\n\t\t\t\\\\\n\t\t\t\\nonumber\n\t\t\t={} &\n\t\t\t\tg(x)-g(x^{k+1})\n\t\t\t\t+\n\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x- x^{k+1}}\n\t\t\t\t}_{\\text{(A)}}\n\t\t\t\t-\n\t\t\t\t\\tD_g(x,x^{k+1})\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x,x^k)\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x,x^{k+1})\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x^{k+1},x^k).\n\t\t\t\\label{eq:main}\n\t\t\t\\end{align}\n\t\t\tWe next proceed to expand the term (A) as\n\t\t\t\\begin{align*}\n\t\t\t\t\\text{(A)}\n\t\t\t={} &\n\t\t\t\t\\innprod{\\nabla f(x^k)}{x-x^k}\n\t\t\t\t-\n\t\t\t\t\\innprod{\\nabla f(x^k)}{x^{k+1}-x^k}\n\t\t\t\\\\\n\t\t\t={} &\n\t\t\t\t\\innprod{\\nabla f(x^k)}{x-x^k}\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_k}\n\t\t\t\t\\innprod{H_k(x^{k-1})-\\nabla\\kernel(x^k)}{x^{k+1}-x^k}\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_k}\n\t\t\t\t\\innprod{H_k(x^{k-1})-H_k(x^k)}{x^k-x^{k+1}}\n\t\t\t\\\\\n\t\t\t={} &\n\t\t\t\t\\Bigl[\n\t\t\t\t\tf(x)-f(x^k)-\\D_f(x,x^k)\n\t\t\t\t\\Bigr]\n\t\t\t\t+\n\t\t\t\t\\Bigl[\n\t\t\t\t\tg(x^{k+1})-g(x^k)-\\tD_g(x^{k+1},x^k)\n\t\t\t\t\\Bigr]\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\n\t\t\t\tB_{k+1},\n\t\t\t\\end{align*}\n\t\t\twhich combined with \\eqref{eq:main} gives\n\t\t\t\\begin{align*}\n\t\t\t\t0\n\t\t\t={} &\n\t\t\t\t-P_k(x)\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x,x^k)\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x,x^{k+1})\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\\D_f(x,x^k)\n\t\t\t\t-\\tD_g(x^{k+1},x^k)\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\n\t\t\t\tB_{k+1}\n\t\t\t\t-\n\t\t\t\t\\tD_g(x,x^{k+1}).\n\t\t\t\\end{align*}\n\t\t\tWe may now add \\eqref{eq:main} to \\eqref{eq:thetk} scaled by \\(\\vartheta_{k+1}\\) and multiply everything by \\(\\gamma_{k+1}\\) to obtain\n\t\t\t\\begin{align*}\n\t\t\t\t0\n\t\t\t={} &\n\t\t\t\t-\\gamma_{k+1} P_k(x)\n\t\t\t\t+\n\t\t\t\t\\D_{\\kernel}(x,x^k)\n\t\t\t\t-\n\t\t\t\t\\D_{\\kernel}(x,x^{k+1})\n\t\t\t\t-\n\t\t\t\t\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\gamma_{k+1}\\D_f(x,x^k)\n\t\t\t\t-\n\t\t\t\t\\gamma_{k+1}\\tD_g(x^{k+1},x^k)\n\t\t\t\t+\n\t\t\t\tB_{k+1}\n\t\t\t\t-\n\t\t\t\t\\gamma_{k+1}\\tD_g(x,x^{k+1})\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t+\n\t\t\t\t\\gamma_{k+1}\\vartheta_{k+1}\\left(\n\t\t\t\t\tP_{k-1}(x)\n\t\t\t\t\t-\n\t\t\t\t\tP_k(x)\n\t\t\t\t\t-\n\t\t\t\t\t\\tfrac{1-\\gamma_k\\ell_k}{\\gamma_k}\n\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t\t-\n\t\t\t\t\t\\tD_\\varphi(x^{k-1},x^k)\n\t\t\t\t\\right).\n\t\t\t\\end{align*}\n\t\t\tAfter suitably rearranging, the claimed identity is obtained.\n\n\t\t\tThe inequality follows by neglecting the terms between curly brackets, and further using the identity \\(\\D_{\\kernel}(x^k,x^{k-1})=\\frac{\\alpha_k}{1+\\alpha_k}\\DD_{\\kernel}(x^k,x^{k-1})\\).\n\t\t\\end{proof}","sha256_utf8":"47a4da804197d0a39e1189225c2fc6cdb0c13659b891771f78dc1a0405c12892"},{"start":155187,"end":156680,"start_line":3405,"end_line":3449,"raw_tex":"\\begin{proof}[\\textbf{Proof of \\cref{thm:descent}}]%\n\t\t\tIt suffices to prove the claim for \\refBY; the case of \\refFNEa{} under the needed assumptions follows by simply replacing \\(\\U_k\\gets\\U^{\\alpha}_k\\).\n\t\t\tThe assumption on \\(x\\) ensures that \\(P_k(x)\\geq0\\) holds for all \\(k\\), implying both that \\(\\U_k(x)\\geq0\\) and the claimed monotonic decrease of \\(\\bigl(\\U_k(x)\\bigr)_{k\\in\\N}\\) with finite limit by virtue of \\cref{thm:BY:leq,thm:FNEa:leq}.\n\t\t\tMore precisely, one has that\n\t\t\t\\[\n\t\t\t\t0\n\t\t\t\\leq\n\t\t\t\t\\U_{k+1}(x)\n\t\t\t\\leq\n\t\t\t\t\\U_k(x)\n\t\t\t\t-\n\t\t\t\t\\gamma_k(1+q\\hat\\rho_k-q\\hat\\rho_{k+1}\\rho_{k+1})P_{k-1}(x),\n\t\t\t\\]\n\t\t\twhere \\(q=1\\) for \\refBY{} and \\(q=\\frac{1+\\alpha}{2\\alpha}\\) for \\refFNEa{}.\n\t\t\tA telescoping argument yields that\n\t\t\t\\begin{align*}\n\t\t\t\tP_K^{\\min}(x)\n\t\t\t\t\\sum_{k=1}^K\n\t\t\t\t\\gamma_k(1+q\\hat\\rho_k-q\\hat\\rho_{k+1}\\rho_{k+1})\n\t\t\t\\leq{} &\n\t\t\t\t\\sum_{k=1}^K\n\t\t\t\t\\gamma_k(1+q\\hat\\rho_k-q\\hat\\rho_{k+1}\\rho_{k+1})P_{k-1}(x)\\\\\n\t\t\t\\leq{} &\n\t\t\t\t\\U_1(x) - \\U_{K+1}(x)\\\\\n\t\t\t\\leq{} &\n\t\t\t\t\\U_1(x) - \\gamma_{K+1}(1+q\\hat\\rho_{K+1})P_K^{\\rm min}(x),\n\t\t\t\\end{align*}\n\t\t\thence that\n\t\t\t\\begin{align*}\n\t\t\t\tP_K^{\\min}(x)\n\t\t\t\\leq{} &\n\t\t\t\t\\frac{\\U_1(x)\n\t\t\t\t\t}{\n\t\t\t\t\t\\sum_{k=1}^K(\\gamma_k + q\\gamma_k\\hat\\rho_k- q\\gamma_{k+1}\\hat\\rho_{k+1}) + \\gamma_{K+1} + q\\gamma_{K+1}\\hat\\rho_{K+1}\n\t\t\t\t\t}\\\\\n\t\t\t\\leq{} &\n\t\t\t\t\\frac{\\U_1(x)\n\t\t\t\t\t}{\n\t\t\t\t\t\\gamma_1\\hat\\rho_1\n\t\t\t\t\t+\n\t\t\t\t\t\\sum_{k=1}^{K+1}\\gamma_k\n\t\t\t\t\t}.\n\t\t\t\\end{align*}\n\t\t\tSince \\(\\U_1(x)\\leq\\U_0(x)\\), the claimed inequality follows.\n\t\t\\end{proof}","sha256_utf8":"849745db63590884dd216c077c603f285c2ee29905d85fbec871d03d5ad96eae"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"复合凸最小化问题","source_units":[{"start":12920,"end":13030,"start_line":472,"end_line":477,"raw_tex":"\\[\\label{eq:P} \\tag{\\text{P}}%\n\t\t\t\\minimize_{x\\in\\overline{C}}\n\t\t\t\\varphi(x)\n\t\t\t\\coloneqq\n\t\t\tf(x) + g(x),\n\t\t\\]","sha256_utf8":"d87984e121d9ef8aa602df631ada174bc50121ac8e54cb4839aecc45b5b80b13"},{"start":13031,"end":13645,"start_line":478,"end_line":479,"raw_tex":"\t\twhere \\(\\overline{C}\\) denotes the closure of \\(C \\coloneqq \\interior\\dom\\kernel\\) for a proper \\(1\\)-coercive function \\(\\kernel:\\R^n\\to\\Rinf\\) of Legendre type, \\(f:\\R^n\\to\\Rinf\\), \\(g:\\R^n\\to\\Rinf\\)  are proper closed convex, while \\(f\\) is only assumed to be \\emph{locally} smooth relative to the \\emph{distance-generating function} (dgf).\n\t\tBy local smoothness relative to \\(\\kernel\\) we refer to the existence, for any compact and convex set \\(\\K\\subset\\interior\\dom\\kernel\\), of a constant \\(L_{f,\\K}^{\\kernel}\\geq0\\) such that \\(L_{f,\\K}^{\\kernel}\\kernel-f\\) is convex over \\(\\K\\); see \\cref{ass:basic}.\n","sha256_utf8":"355a3ad8e7c572f0a97891c950f434632b64ae00b2ed3c6dff32a82107d09de3"}]},{"id":"S002","semantic_type":"algorithm","kind":"algorithm","title":"Bregman近端梯度迭代","source_units":[{"start":14365,"end":14559,"start_line":490,"end_line":496,"raw_tex":"\\begin{equation}\\label{eq:BPG}\n\t\t\tx^{k+1}\n\t\t=\n\t\t\t\\argmin_{w\\in\\R^n}\\left\\{\n\t\t\t\tf(x^k)+\\innprod{\\nabla f(x^k)}{w-x^k}+g(w)+\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(w,x^k)\n\t\t\t\\right\\},\n\t\t\\end{equation}","sha256_utf8":"b9fb09a63cd22e353fb2e948dd7af854a59fdd1910874cfaa96e65b28b239926"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"Bregman距离","source_units":[{"start":14619,"end":14888,"start_line":498,"end_line":506,"raw_tex":"\\begin{equation}\\label{eq:D}\n\t\t\t\\D_{\\kernel}(x, y)\n\t\t\\coloneqq\n\t\t\t\\begin{cases}\n\t\t\t\t\\kernel(x) - \\kernel(y) - \\innprod{\\nabla\\kernel(y)}{x - y} & \\text{if }(x,y)\\in\\dom\\kernel\\times\\interior\\dom\\kernel\n\t\t\t\\\\\n\t\t\t\t\\infty & \\text{otherwise}\n\t\t\t\\end{cases}\n\t\t\\end{equation}","sha256_utf8":"bdc0f8612a83857a570e0b3ca397ce9f87485b640d83267ba72757ffd75495dc"}]},{"id":"S004","semantic_type":"notation","kind":"notation","title":"凸分析基本记号","source_units":[{"start":22422,"end":22719,"start_line":561,"end_line":562,"raw_tex":"\t\t\tThe set of natural numbers is \\(\\N\\coloneqq\\{0,1,2,\\dots\\}\\), while \\(\\R\\), \\(\\R_{++}\\coloneqq(0,\\infty)\\), and \\(\\Rinf\\coloneqq\\R\\cup\\{\\pm\\infty\\}\\) denote the set of real, strictly positive, and extended-real numbers, respectively.\n\t\t\tFor \\(t\\in\\R\\), we define \\([t]_+\\coloneqq\\max\\{t,0\\}\\).\n","sha256_utf8":"9339900c415443c4e141bdcd80acd2a09a45869d25ca0b6429050ce1fad2fac6"},{"start":23267,"end":24429,"start_line":571,"end_line":601,"raw_tex":"\t\t\tThe \\emph{domain} and \\emph{epigraph} of an extended-real-valued function \\(h:\\R^n\\to\\Rinf\\) are the sets\n\t\t\t\\(\n\t\t\t\t\\dom h\n\t\t\t\\coloneqq\n\t\t\t\t\\{x\\in\\R^n \\mid h(x)<\\infty\\}\n\t\t\t\\)\n\t\t\tand\n\t\t\t\\(\n\t\t\t\t\\epi h\n\t\t\t\\coloneqq\n\t\t\t\t\\{(x, c)\\in\\R^n\\times\\R \\mid h(x)\\leq c\\}\n\t\t\t\\).\n\t\t\tFunction \\(h\\) is said to be: \\emph{proper} if \\(h>-\\infty\\) and \\(\\dom h\\neq\\emptyset\\); \\emph{lower semicontinuous (lsc)} if \\(\\epi h\\) is a closed subset of \\(\\R^{n+1}\\); \\emph{1-coercive} if\n\t\t\t\\(\n\t\t\t\t\\lim_{\\norm{x}\\to\\infty}\\frac{h(x)}{\\norm{x}}=\\infty\n\t\t\t\\).\n\n\t\t\tThe \\emph{conjugate} of a proper, lsc, convex function \\(h:\\R^n\\to\\Rinf\\) is the proper, lsc, convex function \\(h^*:\\R^n\\to\\Rinf\\) defined by\n\t\t\t\\(\n\t\t\t\th^*(\\xi)\n\t\t\t\\coloneqq\n\t\t\t\t\\sup_{x\\in\\R^n}\\{\\innprod{\\xi}{x}-h(x)\\}\n\t\t\t\\).\n\t\t\tThe \\emph{(convex) subdifferential} of \\(h\\) at \\(x\\in\\dom h\\) is the set\n\t\t\t\\[\n\t\t\t\t\\partial h(x)\n\t\t\t\\coloneqq\n\t\t\t\t\\{u\\in\\R^n \\mid h(x')\\geq h(x)+\\innprod{u}{x'-x} ~ \\forall x'\\in\\R^n\\},\n\t\t\t\\]\n\t\t\twhile \\(\\partial h(x)=\\emptyset\\) for \\(x\\notin\\dom h\\).\n\t\t\t\\(h\\) is differentiable at \\(x\\) iff \\(\\partial h(x)\\) is a singleton, and in this case one has that \\(\\partial f(x)=\\{\\nabla h(x)\\}\\).\n","sha256_utf8":"9f597d3ca01c632196ee9d2878be5cc84f35cb72583c1b292c9477f095c983a4"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"Legendre核","source_units":[{"start":25029,"end":25529,"start_line":621,"end_line":623,"raw_tex":"\t\t\t\tWe say that a proper, lsc, convex function \\(\\kernel:\\R^n\\to\\Rinf\\) is \\emph{of Legendre type} (or simply \\emph{Legendre}) if it is\n\t\t\t\t(i) \\emph{essentially smooth}, namely differentiable on \\(\\interior\\dom\\kernel\\neq\\emptyset\\) and such that \\(\\norm{\\nabla\\kernel(x^k)}\\to\\infty\\) whenever \\(\\interior\\dom\\kernel\\ni x^k\\to x\\in\\dom\\kernel\\setminus\\interior\\dom\\kernel\\), and\n\t\t\t\t(ii) \\emph{essentially strictly convex}, namely strictly convex on every convex subset of \\(\\dom\\partial\\kernel\\).\n","sha256_utf8":"1cec3791deef5393d2046d8801b78979ac0c76c0c4d8f09e6a848c2abbaef3fa"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"对称Bregman距离","source_units":[{"start":27128,"end":27463,"start_line":652,"end_line":662,"raw_tex":"\\begin{equation}\\label{eq:DD}\n\t\t\t\t\t\\DD_{\\kernel}(x,y)\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\D_{\\kernel}(x,y)+\\D_{\\kernel}(y,x)\n\t\t\t\t=\n\t\t\t\t\t\\begin{cases}\n\t\t\t\t\t\t\\innprod{\\nabla\\kernel(x)-\\nabla\\kernel(y)}{x - y} & \\hspace*{-7pt}\\text{if } x,y\\in\\interior\\dom\\kernel\n\t\t\t\t\t\\\\\n\t\t\t\t\t\t\\infty & \\hspace*{-7pt}\\text{otherwise.}\n\t\t\t\t\t\\end{cases}\n\t\t\t\t\\end{equation}","sha256_utf8":"4fc636784c945f993d9ce0d018a08f91e3e51fc51717cd5c46e82930b2ee21fa"}]},{"id":"S007","semantic_type":"assumption","kind":"assumption","title":"基本工作假设","source_units":[{"start":27715,"end":28592,"start_line":672,"end_line":685,"raw_tex":"\\begin{assumption}\\label{ass:basic}%\n\t\t\tThe following hold in \\eqref{eq:P}:\n\t\t\t\\begin{enumerate}\n\t\t\t\\item \\label{ass:h}%\n\t\t\t\t\\(C=\\interior\\dom\\kernel\\) for a proper, convex, 1-coercive function \\(\\kernel\\) of Legendre type, which is twice continuously differentiable with \\(\\nabla^2\\kernel \\succ 0\\) on \\(C\\).\n\t\t\t\\item \\label{ass:f}%\n\t\t\t\t\\(f:\\R^n\\to\\Rinf\\) is proper, convex, lsc, and \\emph{locally smooth relative to \\(\\kernel\\)}:\n\t\t\t\tthat is, \\(\\dom\\kernel\\subseteq\\dom f\\), and for every convex and compact set \\(\\K\\subset C\\) there exists \\(L_{f,\\K}^{\\kernel}>0\\) such that \\(L_{f,\\K}^{\\kernel}\\kernel-f\\) is convex on \\(\\K\\).\n\t\t\t\\item \\label{ass:g}%\n\t\t\t\t\\(g:\\R^n\\to\\Rinf\\) is proper, lsc, and convex with \\(\\dom g\\cap C\\neq\\emptyset\\).\n\t\t\t\\item \\label{ass:argmin}%\n\t\t\t\tA solution exists: \\(\\argmin_{\\overline C}\\varphi\\neq\\emptyset\\).\n\t\t\t\\end{enumerate}\n\t\t\\end{assumption}","sha256_utf8":"739b5fa0c562f4c9aece72d0d25c9fd278a6103afa18035f01ae1cf765e31f14"}]},{"id":"S008","semantic_type":"derived_fact","kind":"derived_fact","title":"局部相对光滑上界","source_units":[{"start":28688,"end":28908,"start_line":688,"end_line":691,"raw_tex":"\t\tThe local relative smoothness in \\cref{ass:f} is tantamount to saying that \\(f\\) is differentiable on \\(C\\),\\footnote{%\n\t\t\tSee \\cite[Prop. 3.7]{wang2024mirror} or \\cite[Prop. 2.5]{ahookhosh2021bregman}.\n\t\t}\n\t\tand that\n","sha256_utf8":"9e0fac7d415041a5f2a1de7cce320ee30d4792f3c14d5b8b14ea625373319b51"},{"start":28910,"end":29042,"start_line":692,"end_line":696,"raw_tex":"\\begin{equation}\\label{eq:Df<D}\n\t\t\t\\D_f(x, y)\\leq L_{f,\\K}^{\\kernel}\\D_{\\kernel}(x, y)\n\t\t\\quad\n\t\t\t\\forall x,y\\in\\K,\n\t\t\\end{equation}","sha256_utf8":"675a9a01df75f5216625dac60c581397a4e1aa317b1e9cee2e03342f9c554b94"}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","title":"Bregman区域假设","source_units":[{"start":31262,"end":31727,"start_line":719,"end_line":727,"raw_tex":"\\begin{assumption}[Bregman with zone \\(C\\) {\\cite[Def. 2.1]{solodov2000inexact}}]\\label{ass:zone}%\n\t\t\tThe dgf \\(\\kernel\\) satisfies the following:\n\t\t\t\\begin{enumerate}\n\t\t\t\\item \\label{ass:Dhto0}%\n\t\t\t\t\\(\\D_{\\kernel}(x, x^k) \\to 0\\) whenever \\(C \\ni x^k \\to x\\) (in particular, \\(\\dom\\kernel\\) is closed).\n\t\t\t\\item \\label{ass:xkbounded}%\n\t\t\t\t\\(\\D_{\\kernel}(x,{}\\cdot{})\\) is level bounded for any \\(x\\in\\overline{C}\\setminus C\\).\n\t\t\t\\end{enumerate}\n\t\t\\end{assumption}","sha256_utf8":"85cf2280edde4580eb05194b968fceb8eab1ddc1efe47ba8fed735e43f6c6aae"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"局部曲率与对称系数","source_units":[{"start":34138,"end":34233,"start_line":775,"end_line":777,"raw_tex":"\\begin{equation}\\label{eq:Hk}\n\t\t\t\tH_k\\coloneqq\\nabla\\kernel-\\gamma_k\\nabla f,\n\t\t\t\\end{equation}","sha256_utf8":"4810118b2dbf87f840da9f60ac045f3a0d24172fd74d472fb5ff5b4344931cf7"},{"start":34584,"end":34721,"start_line":784,"end_line":788,"raw_tex":"\\begin{equation}\\label{eq:lk}\n\t\t\t\t\t\t\\ell_k\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\frac{\\DD_f(x^k,x^{k-1})}{\\DD_{\\kernel}(x^k,x^{k-1})}\n\t\t\t\t\t\\end{equation}","sha256_utf8":"e874be7e7ad474193fe1e0ce3110cd68f4df4e93018414433cc79ea2cee46e3d"},{"start":35579,"end":35854,"start_line":796,"end_line":804,"raw_tex":"\\begin{equation}\\label{eq:Lamk}\n\t\t\t\t\t\t\\Lambda_{k,\\delta}\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\t2\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(x^k)+\\delta\\bigl[H_k(x^k)-H_k(x^{k-1})\\bigr],\\nabla\\kernel(x^k)\\bigr)\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\\delta^2\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t\t\t}\n\t\t\t\t\t\\end{equation}","sha256_utf8":"82b82c81b2fdaba188e2696da8c389aba34f4aed0182105817b873fbd61868cf"},{"start":35855,"end":35920,"start_line":805,"end_line":805,"raw_tex":"\t\t\t\t\tdepending on some parameter \\(\\delta>0\\) (specified later).\n","sha256_utf8":"40b90223838e3fd82e2577b6ff9f9e9732142caf67d715df2bae03afa5964b9c"},{"start":36858,"end":37029,"start_line":823,"end_line":829,"raw_tex":"\\begin{equation}\\label{eq:ak}\n\t\t\t\t\t\t\\alpha_k\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\frac{\\D_{\\kernel}(x^k,x^{k-1})}{\\D_{\\kernel}(x^{k-1},x^k)}\n\t\t\t\t\t\\in\n\t\t\t\t\t\t(0,\\infty)\n\t\t\t\t\t\\end{equation}","sha256_utf8":"125e157f005d9705fcd1a56577d36bc38ff4401d35df71a7630469fe83a3f58b"},{"start":37081,"end":37210,"start_line":832,"end_line":836,"raw_tex":"\\begin{equation}\n\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t=\n\t\t\t\t\\tfrac{1+\\alpha_k}{\\alpha_k}\\D_{\\kernel}(x^k,x^{k-1}).\n\t\t\t\\end{equation}","sha256_utf8":"eb7e3441ac85b3935623c50fd6eb24679c7a55c2416cd5f8eeedda17882dd864"},{"start":37417,"end":37602,"start_line":838,"end_line":842,"raw_tex":"\\begin{equation}\\label{eq:alpha}\n\t\t\t\t\\alpha(\\kernel)\n\t\t\t\\coloneqq\n\t\t\t\t\\inf_{\\substack{x,y\\in\\interior\\dom\\kernel\\\\x\\neq y}}\\frac{\\D_{\\kernel}(x,y)}{\\D_{\\kernel}(y,x)}.\n\t\t\t\\end{equation}","sha256_utf8":"e1242262d4d760e284ec7e5a1546ea3592e5456d434c15b9a0b058690a3f72cd"}]},{"id":"S011","semantic_type":"notation","kind":"notation","title":"相邻步长比","source_units":[{"start":38292,"end":38652,"start_line":850,"end_line":857,"raw_tex":"\t\t\tBased on these three quantities, choose a stepsize \\(\\gamma_{k+1}\\) and proceed with a Bregman proximal gradient update \\eqref{eq:BPG}.\n\t\t\tWe propose the following two options, where we let\n\t\t\t\\[\n\t\t\t\t\\rho_{k+1}\n\t\t\t\\coloneqq\n\t\t\t\t\\tfrac{\\gamma_{k+1}}{\\gamma_k}\n\t\t\t\\]\n\t\t\tdenote the ratio of consecutive stepsizes (so that \\(\\gamma_{k+1}=\\rho_{k+1}\\gamma_k\\)):\n","sha256_utf8":"3615df966b9122daa9729115d6bc1ec2eee5c5f41ca56ccaf30d7fc6b340dade"}]},{"id":"S012","semantic_type":"algorithm","kind":"algorithm","title":"无搜索自适应规则","source_units":[{"start":38652,"end":39248,"start_line":858,"end_line":884,"raw_tex":"\n\t\t\t\\medskip\n\t\t\t\\par\\noindent\\phantomsection \\label{alg:BY}%\n\t\t\t\t\\parbox[t]{0.15\\linewidth}{\\bf\\BY}%\n\t\t\t\t\\hfill\n\t\t\t\t\\fbox{\\parbox[t]{0.82\\linewidth}{%\n\t\t\t\t\tset \\(\\hat\\rho_{k+1}=\\sqrt{1+\\rho_k}\\) and \\(\\delta=2\\hat\\rho_{k+1}\\), and update\n\t\t\t\t\t\\begin{equation}\\label{eq:BY:rhok*}\n\t\t\t\t\t\t\\rho_{k+1}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\min\\left\\{\n\t\t\t\t\t\t\t\\hat\\rho_{k+1}\n\t\t\t\t\t\t,\n\t\t\t\t\t\t\t\\frac{\\alpha_k}{1+\\alpha_k}\n\t\t\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\t\t1\n\t\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t\t2\\hat\\rho_{k+1}\n\t\t\t\t\t\t\t\t\\left[\n\t\t\t\t\t\t\t\t\t\\Lambda_{k,\\delta}\n\t\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\t\t\t\\right]_+\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\right\\}\n\t\t\t\t\t\\end{equation}\n\t\t\t\t}}%\n","sha256_utf8":"bb3ba3b4de70b8a21e30bec84bd52c8a80784565a35a6ed08a81f2f6b988175d"}]},{"id":"S013","semantic_type":"algorithm","kind":"algorithm","title":"正对称系数变体","source_units":[{"start":39248,"end":40018,"start_line":885,"end_line":917,"raw_tex":"\n\t\t\t\\medskip\n\t\t\t\\noindent\n\t\t\tand, in case \\(\\kernel\\) enjoys a symmetry coefficient \\(\\alpha=\\alpha(\\kernel)>0\\),\n\t\t\t\\medskip\n\n\t\t\t\\par\\noindent\\phantomsection \\label{alg:FNEa}%\n\t\t\t\t\\parbox[t]{0.15\\linewidth}{%\n\t\t\t\t\t{\\bf\\FNEa}\n\n\t\t\t\t\t\\footnotesize\n\t\t\t\t\t(if \\(\\alpha(\\kernel)>0\\))%\n\t\t\t\t}\n\t\t\t\t\\hfill\n\t\t\t\t\\fbox{\\parbox[t]{0.82\\linewidth}{%\n\t\t\t\t\tset \\(\\hat\\rho_{k+1}=\\sqrt{\\frac{1+\\alpha}{2}+\\rho_k}\\) and \\(\\delta=\\frac{2}{1+\\alpha}\\hat\\rho_{k+1}\\), and update\n\t\t\t\t\t\\begin{equation}\\label{eq:FNEa:rhok*}\n\t\t\t\t\t\t\\rho_{k+1}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\min\\left\\{\n\t\t\t\t\t\t\t\\hat\\rho_{k+1}\n\t\t\t\t\t\t,\n\t\t\t\t\t\t\t\\frac{\\alpha}{\n\t\t\t\t\t\t\t\t2\\hat\\rho_{k+1}\n\t\t\t\t\t\t\t\t\\left[\n\t\t\t\t\t\t\t\t\t\\Lambda_{k,\\delta}\n\t\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\t\t\t\\right]_+\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\right\\}.\n\t\t\t\t\t\\end{equation}\n\t\t\t\t}}%\n","sha256_utf8":"bb84b6a062bc0a872dd526b9aa8754900449bc75bd996736c7e78646e35642cd"}]},{"id":"S014","semantic_type":"notation","kind":"notation","title":"初始化与零分母约定","source_units":[{"start":40065,"end":40435,"start_line":921,"end_line":923,"raw_tex":"\t\t\t\tWe use the convention that \\(\\frac{1}{0}=\\infty\\), and remind that \\([t]_+=\\max\\{0,t\\}\\).\n\t\t\t\tIn particular, whenever \\(\\Lambda_{k,\\delta}\\leq1-\\gamma_k\\ell_k\\) all updates reduce to \\(\\rho_{k+1}=\\hat\\rho_{k+1}\\).\n\t\t\t\tIt is implied that a starting point \\(x^0\\in C\\) should be provided, as well as two stepsizes \\(\\gamma_0,\\gamma_1>0\\) for the first two iterations.\n","sha256_utf8":"cfbba540cf9f58665a0c354ec6b08d24f717850c11ffbf5bb2097f4a8047ba23"}]},{"id":"S015","semantic_type":"definition","kind":"definition","title":"目标差与迭代次梯度","source_units":[{"start":48285,"end":50152,"start_line":1125,"end_line":1183,"raw_tex":"\t\tTo ease the subsequent discussion, we introduce some convenient notational shorthands and remind some of those already encountered.\n\t\tRelative to the iterates generated by \\eqref{eq:BPG}, for any \\(k\\in\\N\\) and \\(x\\in\\dom\\varphi\\) we denote\n\t\t\\begin{align}\n\t\t\\label{eq:Pk}\n\t\t\tP_k(x)\n\t\t\\coloneqq{} &\n\t\t\t\\varphi(x^k)-\\varphi(x)\n\t\t\\shortintertext{and}\n\t\t\\label{eq:Bk}\n\t\t\tB_{k+1}\n\t\t\\coloneqq{} &\n\t\t\t\\rho_{k+1}\n\t\t\t\\innprod{x^{k+1}-x^k}{H_k(x^k)-H_k(x^{k-1})},\n\t\t\\end{align}\n\t\twhere \\(\\rho_{k+1}=\\tfrac{\\gamma_{k+1}}{\\gamma_k}\\) and\n\t\t\\begin{equation}\\label{eq:H:def}\n\t\t\tH_k\\coloneqq\\nabla\\kernel-\\gamma_k\\nabla f\n\t\t\\end{equation}\n\t\tis the ``forward'' operator.\n\t\tDue to convexity of \\(g\\), the BPG updates in \\eqref{eq:BPG} are characterized by the subgradient inclusion\n\t\t\\begin{subequations}\n\t\t\t\\begin{align}\n\t\t\t\\label{eq:subgrad}\n\t\t\t\t\\tilde{\\nabla}g(x^{k+1})\n\t\t\t\\coloneqq{} &\n\t\t\t\t\\tfrac{\\nabla\\kernel(x^k)-\\nabla\\kernel(x^{k+1})}{\\gamma_{k+1}}\n\t\t\t\t-\n\t\t\t\t\\nabla f(x^k)\n\t\t\t\\in\n\t\t\t\t\\partial g(x^{k+1}).\n\t\t\t\\intertext{%\n\t\t\t\tThroughout, we use the notation \\(\\tilde{\\nabla}g(x^{k+1})\\) to indicate this particular element of the subgradient of \\(g\\) along the iterates.\n\t\t\t\tSimilarly, we use\n\t\t\t}\n\t\t\t\\nonumber\n\t\t\t\t\\tilde{\\nabla}\\varphi(x^{k+1})\n\t\t\t\\coloneqq{} &\n\t\t\t\t\\tilde{\\nabla} g(x^{k+1})\n\t\t\t\t+\n\t\t\t\t\\nabla f(x^{k+1})\n\t\t\t\\\\\n\t\t\t\\label{eq:subgradvarphi}\n\t\t\t={} &\n\t\t\t\t\\tfrac{H_{k+1}(x^k)-H_{k+1}(x^{k+1})}{\\gamma_{k+1}}\n\t\t\t\\in\n\t\t\t\t\\partial\\varphi(x^{k+1}).\n\t\t\t\\end{align}\n\t\t\\end{subequations}\n\t\tIn light of these, we adopt the notation\n\t\t\\begin{gather*}\n\t\t\t\\tD_g(w,x^k)\n\t\t\\coloneqq\n\t\t\tg(w)-g(x^k)-\\innprod{\\tilde{\\nabla}g(x^k)}{w-x^k},\n\t\t\\shortintertext{and similarly}\n\t\t\t\\tD_{\\varphi}(w,x^k)\n\t\t\\coloneqq\n\t\t\t{\\varphi}(w)-{\\varphi}(x^k)-\\innprod{\\tilde{\\nabla}{\\varphi}(x^k)}{w-x^k},\n\t\t\\end{gather*}\n\t\twhich are both positive quantities for any \\(k\\in\\N\\) and \\(w\\in\\R^n\\).\n","sha256_utf8":"43945b1d2feb9d034abdb6a180bd79c922bdd838bca846b17b297a4f2f80062a"}]},{"id":"S016","semantic_type":"derived_fact","kind":"derived_fact","title":"三点恒等式","source_units":[{"start":24557,"end":25027,"start_line":607,"end_line":619,"raw_tex":"\\begin{fact}[three-point identity {\\cite[Lem. 3.1]{chen1993convergence}}]\\label{thm:3p}%\n\t\t\t\t\tLet \\(\\kernel:\\R^n\\to\\Rinf\\) be a proper lsc convex function differentiable on \\(\\interior\\dom\\kernel\\).\n\t\t\t\t\tFor any \\(x\\in\\dom\\kernel\\), and \\(y,z\\in\\interior\\dom\\kernel\\) the following holds:\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\D_{\\kernel}(x,z)\n\t\t\t\t\t=\n\t\t\t\t\t\t\\D_{\\kernel}(x,y)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\D_{\\kernel}(y,z)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{x-y}{\\nabla\\kernel(y)-\\nabla\\kernel(z)}.\n\t\t\t\t\t\\]\n\t\t\t\t\\end{fact}","sha256_utf8":"b1fa0c0449b0818138548d76c3712adce4f3f0fe33847bcd32be73c5e3ef4394"}]},{"id":"S017","semantic_type":"derived_fact","kind":"derived_fact","title":"Legendre距离标准性质","source_units":[{"start":25534,"end":27015,"start_line":625,"end_line":649,"raw_tex":"\\begin{fact}\\label{fact:D}%\n\t\t\t\t\tLet \\(\\kernel:\\R^n\\to\\Rinf\\) be of Legendre type, and let \\(C\\coloneqq\\interior\\dom\\kernel\\).\n\t\t\t\t\t\\begin{enumerate}\n\t\t\t\t\t\\item \\label{fact:D:coercive}%\n\t\t\t\t\t\t\t{\\upshape \\cite[Prop. 2.16]{bauschke1997legendre}, \\cite[Thm. 3.7(vi)]{bauschke1997legendre}}\n\t\t\t\t\t\t\tIf \\(\\kernel\\) is 1-coercive, then \\(\\dom\\kernel^*=\\R^n\\) and \\(\\D_{\\kernel}(x,{}\\cdot{})\\) is coercive for any \\(x\\in C\\).\n\n\t\t\t\t\t\\item \\label{fact:D:boundary}%\n\t\t\t\t\t\t{\\upshape \\cite[Thm. 3.8(i)]{bauschke1997legendre}}\n\t\t\t\t\t\tIf a sequence \\((x^k)_{k\\in\\N}\\) converges to a boundary point of \\(C\\), then \\(\\D_{\\kernel}(x,x^k)\\to\\infty\\) for any \\(x\\in C\\).\n\n\t\t\t\t\t\\item \\label{fact:D:solodov}%\n\t\t\t\t\t\t{\\upshape \\cite[Thm. 2.4]{solodov2000inexact}}\n\t\t\t\t\t\tSuppose that \\(\\dom\\kernel\\) is closed and that \\(\\D_{\\kernel}(x^k,y^k)\\to0\\).\n\t\t\t\t\t\tIf either \\((x^k)_{k\\in\\N}\\) or \\((y^k)_{k\\in\\N}\\) converges, then the other sequence too converges to the same limit.\n\n\t\t\t\t\t\\item \\label{fact:D:Dphi*}%\n\t\t\t\t\t\t{\\upshape \\cite[Thm. 26.5]{rockafellar1970convex}}\n\t\t\t\t\t\tThe conjugate \\(\\kernel^*\\) is continuously differentiable and strictly convex on \\(\\interior\\dom\\kernel^*\\), with \\(\\nabla\\kernel^* = \\nabla\\kernel^{-1}:\\interior\\dom\\kernel^*\\to\\interior\\dom\\kernel\\).\n\n\t\t\t\t\t\\item \\label{fact:D:DtoD*}%\n\t\t\t\t\t\t{\\upshape \\cite[Thm. 3.7(v)]{bauschke1997legendre}}\n\t\t\t\t\t\t\\(\\D_{\\kernel}(x,y) = \\D_{\\kernel^*}(\\nabla \\kernel(y), \\nabla \\kernel(x))\\) for any \\(x,y\\in C\\).\n\t\t\t\t\t\\end{enumerate}\n\t\t\t\t\\end{fact}","sha256_utf8":"3f934067b11b38b4caf981f0672b947772a8dfb090e62709103a5340764c6a73"}]},{"id":"S018","semantic_type":"theorem","kind":"theorem","title":"自适应算法主收敛定理","source_units":[{"start":40790,"end":41658,"start_line":932,"end_line":948,"raw_tex":"\\begin{theorem}[summary of main results]\\label{thm:main}%\n\t\t\t\t\tSuppose that \\cref{ass:basic} holds, and consider the iterates generated by \\refBY{}.\n\t\t\t\t\tThen, one always has that\n\t\t\t\t\t\\begin{equation}\\label{eq:main:inf}\n\t\t\t\t\t\t\\smash{\n\t\t\t\t\t\t\t\\inf_{k\\in\\N}\\varphi(x^k)=\\inf_{\\overline{C}}\\varphi.\n\t\t\t\t\t\t}\n\t\t\t\t\t\\end{equation}\n\t\t\t\t\tMoreover,\n\t\t\t\t\t\\begin{enumerate}\n\t\t\t\t\t\\item \\label{thm:main:C}%\n\t\t\t\t\t\tIf \\(C\\cap\\argmin_{\\overline{C}}\\varphi\\neq \\emptyset\\) (equivalently, if \\(C\\cap\\argmin\\varphi\\neq\\emptyset\\)), then there exists \\(x^\\star\\in C\\cap \\argmin\\varphi\\) such that \\(x^k\\to x^\\star\\).\n\t\t\t\t\t\\item \\label{thm:main:zone}%\n\t\t\t\t\t\tIf \\cref{ass:zone} holds, then \\((x^k)_{k\\in\\N}\\) is bounded and admits a unique optimal limit point.\n\t\t\t\t\t\\end{enumerate}\n\t\t\t\t\tWhen \\(\\kernel\\) has symmetry coefficient \\(\\alpha>0\\), the same is true for \\refFNEa.\n\t\t\t\t\\end{theorem}","sha256_utf8":"27ef109641b935d3bdc3c90221d37a8ddb8eb17fd43bc47d3924f760723a3edf"}]},{"id":"S019","semantic_type":"proof","kind":"proof","title":"主定理完整证明","source_units":[{"start":89509,"end":89747,"start_line":2204,"end_line":2204,"raw_tex":"\t\t\tThe fact that this result subsumes \\cref{thm:main:C} is obvious by observing that the validity of \\cref{thm:xinC} implies that any \\(x\\) complying with \\eqref{eq:xinC} must necessarily belong to \\(\\argmin_{\\overline C}\\varphi\\cap C\\).\n","sha256_utf8":"5df1a54c9d6c0df7fd6fe8ee3d81e2f3442c62c0043558bce198a7025a80c271"},{"start":100096,"end":103169,"start_line":2424,"end_line":2472,"raw_tex":"\\subsection{Proof of \\texorpdfstring{\\cref{eq:main:inf}}{Equation \\ref*{eq:main:inf}}}\n\t\t\tIn this subsection we prove that the iterates generated by either \\refBY{} or \\refFNEa{} in the generality of \\cref{ass:basic} (in addition to \\(\\kernel\\) having a symmetry coefficient \\(\\alpha>0\\) in the latter case) are such that \\(\\inf_{k\\in\\N}\\varphi(x^k)=\\inf_{\\overline C}\\varphi\\).\n\n\t\t\tTo see this, contrary to the claim suppose that \\(\\inf_{k\\in\\N}\\varphi(x^k)>\\inf_{\\overline C}\\varphi\\).\n\t\t\tThen, since \\(\\inf_{\\overline C}\\varphi=\\inf_C\\varphi\\) by \\cite[Prop. 11.1(iv)]{bauschke2017convex}, there exists \\(x\\in C\\) such that \\(\\varphi(x^k)\\geq\\varphi(x)\\) holds for all \\(k\\in\\N\\).\n\t\t\tInvoking \\cref{thm:xinC} yields a contradiction.\n\n\n\n\t\t%% ░░░░ 4.5 Proof of Theorem 2.7(ii) ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░\n\t\t\\subsection{Proof of \\texorpdfstring{\\cref{thm:main:zone}}{Theorem \\ref*{thm:main:zone}}}\n\t\t\tIn this subsection we prove that whenever \\cref{ass:basic,ass:zone} are satisfied, the iterates generated by \\refBY{} are bounded and admit exactly one optimal limit point.\\footnote{%\n\t\t\t\t\\label{foot:noFNEa}%\n\t\t\t\tAs commented after \\cref{thm:main}, there is no loss of generality in considering only \\refBY{} and disregard the claim for \\refFNEa.\n\t\t\t}\n\t\t\tWe can without loss of generality assume that \\(\\argmin_{C}\\varphi=\\emptyset\\), for otherwise a stronger result is already covered by \\cref{thm:main:C}.\n\n\t\t\tUnder \\cref{ass:zone}, \\(\\dom\\kernel=\\overline C\\), and therefore in \\cref{thm:SD} one can take any \\(x\\in\\argmin_{\\overline C}\\varphi\\).\n\t\t\tIn particular, \\(\\D_{\\kernel}(x,x^k)\\leq\\U_k(x)\\leq\\U_0(x)\\) holds for any \\(k\\).\n\t\t\tSince \\(\\D_{\\kernel}(x,{}\\cdot{})\\) is level bounded by \\cref{ass:xkbounded}, boundedness of \\((x^k)_{k\\in\\N}\\) follows.\n\t\t\tMoreover, we know from \\eqref{eq:main:inf} that \\(\\varphi(x^k)\\to\\inf_{\\overline C}\\varphi\\) along a subsequence, and therefore an optimal limit point \\(x^\\star\\) exists.\n\t\t\tTo assess its uniqueness, we can argue similarly to the proof of \\cref{claim:unique}, with the minor catch that now such limit points are on the boundary of \\(C\\).\n\t\t\tConsidering two optimal limit points \\(\\bar x_1\\) and \\(\\bar x_2\\), and two corresponding subsequences \\((x^k)_{k\\in K_1}\\to\\bar x_1\\) and \\((x^k)_{k\\in K_2}\\to\\bar x_2\\), we still have that\n\t\t\t\\[\n\t\t\t\t\\U_k(\\bar x_1)-\\U_k(\\bar x_2)\n\t\t\t=\n\t\t\t\t\\D_{\\kernel}(\\bar x_1,x^k)-\\D_{\\kernel}(\\bar x_2,x^k)\n\t\t\t\\]\n\t\t\tconverges to some finite value \\(V\\).\n\t\t\tAgain by considering the limit along the two subsequences \\(k\\in K_1\\) and \\(k\\in K_2\\), \\cref{ass:Dhto0} yields that\n\t\t\t\\[\n\t\t\t\t-\\lim_{k\\in K_2}\\D_{\\kernel}(\\bar x_1,x^k)\n\t\t\t=\n\t\t\t\t\\lim_{k\\in K_1}\\D_{\\kernel}(\\bar x_2,x^k),\n\t\t\t\\]\n\t\t\thence that\n\t\t\t\\[\n\t\t\t\t\\lim_{k\\in K_2}\\D_{\\kernel}(\\bar x_1,x^k)\n\t\t\t=\n\t\t\t\t0\n\t\t\t=\n\t\t\t\t\\lim_{k\\in K_1}\\D_{\\kernel}(\\bar x_2,x^k),\n\t\t\t\\]\n\t\t\twhich by \\cref{fact:D:solodov} implies that \\(\\bar x_1=\\bar x_2\\).\n\n\n\n\t\t%% ░░░░ 4.6 Bregman proximal gradient with backtracking ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░\n\t\t","sha256_utf8":"13ec605652008cef927879b5ca802fa4a13586c1fa934e8cb41f9abfd9abb72e"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"内部解情形的归结","source_units":[{"start":89509,"end":89747,"start_line":2204,"end_line":2204,"raw_tex":"\t\t\tThe fact that this result subsumes \\cref{thm:main:C} is obvious by observing that the validity of \\cref{thm:xinC} implies that any \\(x\\) complying with \\eqref{eq:xinC} must necessarily belong to \\(\\argmin_{\\overline C}\\varphi\\cap C\\).\n","sha256_utf8":"5df1a54c9d6c0df7fd6fe8ee3d81e2f3442c62c0043558bce198a7025a80c271"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"确定一般下确界目标","source_units":[{"start":100183,"end":100476,"start_line":2425,"end_line":2426,"raw_tex":"\t\t\tIn this subsection we prove that the iterates generated by either \\refBY{} or \\refFNEa{} in the generality of \\cref{ass:basic} (in addition to \\(\\kernel\\) having a symmetry coefficient \\(\\alpha>0\\) in the latter case) are such that \\(\\inf_{k\\in\\N}\\varphi(x^k)=\\inf_{\\overline C}\\varphi\\).\n\n","sha256_utf8":"29ad7c88589b369cede236739d60f144570988835de58cab04ffca47aec4b66d"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"反证得到最优下确界","source_units":[{"start":100476,"end":100832,"start_line":2427,"end_line":2429,"raw_tex":"\t\t\tTo see this, contrary to the claim suppose that \\(\\inf_{k\\in\\N}\\varphi(x^k)>\\inf_{\\overline C}\\varphi\\).\n\t\t\tThen, since \\(\\inf_{\\overline C}\\varphi=\\inf_C\\varphi\\) by \\cite[Prop. 11.1(iv)]{bauschke2017convex}, there exists \\(x\\in C\\) such that \\(\\varphi(x^k)\\geq\\varphi(x)\\) holds for all \\(k\\in\\N\\).\n\t\t\tInvoking \\cref{thm:xinC} yields a contradiction.\n","sha256_utf8":"e6c83536c5dcf9b13e2e87f6e265d496dcdc21adda53d22547411d6f9a15cf16"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"区域结论化为边界情形","source_units":[{"start":101044,"end":101555,"start_line":2435,"end_line":2439,"raw_tex":"\t\t\tIn this subsection we prove that whenever \\cref{ass:basic,ass:zone} are satisfied, the iterates generated by \\refBY{} are bounded and admit exactly one optimal limit point.\\footnote{%\n\t\t\t\t\\label{foot:noFNEa}%\n\t\t\t\tAs commented after \\cref{thm:main}, there is no loss of generality in considering only \\refBY{} and disregard the claim for \\refFNEa.\n\t\t\t}\n\t\t\tWe can without loss of generality assume that \\(\\argmin_{C}\\varphi=\\emptyset\\), for otherwise a stronger result is already covered by \\cref{thm:main:C}.\n","sha256_utf8":"3b894a71d515a9b84e9964902f64665b4f39eda0050851325599acc4aac67ce8"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"有界性与最优聚点存在","source_units":[{"start":101556,"end":102080,"start_line":2441,"end_line":2444,"raw_tex":"\t\t\tUnder \\cref{ass:zone}, \\(\\dom\\kernel=\\overline C\\), and therefore in \\cref{thm:SD} one can take any \\(x\\in\\argmin_{\\overline C}\\varphi\\).\n\t\t\tIn particular, \\(\\D_{\\kernel}(x,x^k)\\leq\\U_k(x)\\leq\\U_0(x)\\) holds for any \\(k\\).\n\t\t\tSince \\(\\D_{\\kernel}(x,{}\\cdot{})\\) is level bounded by \\cref{ass:xkbounded}, boundedness of \\((x^k)_{k\\in\\N}\\) follows.\n\t\t\tMoreover, we know from \\eqref{eq:main:inf} that \\(\\varphi(x^k)\\to\\inf_{\\overline C}\\varphi\\) along a subsequence, and therefore an optimal limit point \\(x^\\star\\) exists.\n","sha256_utf8":"48cd7295daea07aaab9844b95af9d8d2c3b0e2e7133ecdc0d2e1dd6d90451cd9"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"两个聚点的距离差收敛","source_units":[{"start":102080,"end":102591,"start_line":2445,"end_line":2452,"raw_tex":"\t\t\tTo assess its uniqueness, we can argue similarly to the proof of \\cref{claim:unique}, with the minor catch that now such limit points are on the boundary of \\(C\\).\n\t\t\tConsidering two optimal limit points \\(\\bar x_1\\) and \\(\\bar x_2\\), and two corresponding subsequences \\((x^k)_{k\\in K_1}\\to\\bar x_1\\) and \\((x^k)_{k\\in K_2}\\to\\bar x_2\\), we still have that\n\t\t\t\\[\n\t\t\t\t\\U_k(\\bar x_1)-\\U_k(\\bar x_2)\n\t\t\t=\n\t\t\t\t\\D_{\\kernel}(\\bar x_1,x^k)-\\D_{\\kernel}(\\bar x_2,x^k)\n\t\t\t\\]\n\t\t\tconverges to some finite value \\(V\\).\n","sha256_utf8":"79d98c4f98416d9c36107a72b16ca00ac03968ad2a46a0f14cd57d492df60b16"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"区域连续性保证聚点唯一","source_units":[{"start":102591,"end":103028,"start_line":2453,"end_line":2467,"raw_tex":"\t\t\tAgain by considering the limit along the two subsequences \\(k\\in K_1\\) and \\(k\\in K_2\\), \\cref{ass:Dhto0} yields that\n\t\t\t\\[\n\t\t\t\t-\\lim_{k\\in K_2}\\D_{\\kernel}(\\bar x_1,x^k)\n\t\t\t=\n\t\t\t\t\\lim_{k\\in K_1}\\D_{\\kernel}(\\bar x_2,x^k),\n\t\t\t\\]\n\t\t\thence that\n\t\t\t\\[\n\t\t\t\t\\lim_{k\\in K_2}\\D_{\\kernel}(\\bar x_1,x^k)\n\t\t\t=\n\t\t\t\t0\n\t\t\t=\n\t\t\t\t\\lim_{k\\in K_1}\\D_{\\kernel}(\\bar x_2,x^k),\n\t\t\t\\]\n\t\t\twhich by \\cref{fact:D:solodov} implies that \\(\\bar x_1=\\bar x_2\\).\n","sha256_utf8":"4c272111a9c0f22a5f403c968a2dcd8923c58e1a5688fff6034dfa91e663f256"}]},{"id":"S027","semantic_type":"lemma","kind":"lemma","title":"主恒等式与势函数界","source_units":[{"start":50393,"end":51926,"start_line":1188,"end_line":1254,"raw_tex":"\\begin{lemma}[main identity]\\label{thm:eq}%\n\t\t\tSuppose that \\cref{ass:basic} holds, and starting from \\(x^0\\in C\\) consider Bregman proximal gradient iterations \\eqref{eq:BPG} with stepsizes \\(\\gamma_k>0\\).\n\t\t\tThen, for any \\(x\\in\\dom\\varphi\\cap\\dom\\kernel\\), \\(\\vartheta_{k+1}\\geq0\\), \\(k\\in\\N\\), it holds that\n\t\t\t\\begin{align*}\n\t\t\t&\n\t\t\t\t\\D_{\\kernel}(x,x^{k+1})\n\t\t\t\t+\n\t\t\t\t\\gamma_{k+1}(1+\\vartheta_{k+1})P_k(x)\n\t\t\t\t+\n\t\t\t\t\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\\\\\n\t\t\t={} &\n\t\t\t\t\\D_{\\kernel}(x,x^k)\n\t\t\t\t+\n\t\t\t\t\\gamma_{k+1}\\vartheta_{k+1} P_{k-1}(x)\n\t\t\t\t-\n\t\t\t\t\\rho_{k+1}\\vartheta_{k+1}(1-\\gamma_k\\ell_k)\n\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t+\n\t\t\t\tB_{k+1}\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\gamma_{k+1}\n\t\t\t\t\\Bigl\\{\n\t\t\t\t\t\\D_f(x,x^k)\n\t\t\t\t\t+\n\t\t\t\t\t\\tD_g(x^{k+1},x^k)\n\t\t\t\t\t+\n\t\t\t\t\t\\tD_g(x,x^{k+1})\n\t\t\t\t\t+\n\t\t\t\t\t\\vartheta_{k+1}\\tD_\\varphi(x^{k-1},x^k)\n\t\t\t\t\\Bigr\\},\n\t\t\t\\end{align*}\n\t\t\twhere \\(P_k(x)\\), \\(B_{k+1}\\), and \\(\\ell_k\\) are as in \\eqref{eq:Pk}, \\eqref{eq:Bk}, and \\eqref{eq:lk}.\n\t\t\tIn particular, denoting\n\t\t\t\\[\n\t\t\t\t\\widehat{\\U}_k(x)\n\t\t\t\\coloneqq\n\t\t\t\t\\D_{\\kernel}(x,x^k)\n\t\t\t\t+\n\t\t\t\t\\gamma_k(1+\\vartheta_k)P_{k-1}(x)\n\t\t\t\t+\n\t\t\t\t\\D_{\\kernel}(x^k,x^{k-1}),\n\t\t\t\\]\n\t\t\tone has that\n\t\t\t\\begin{align}\n\t\t\t\\nonumber\n\t\t\t\t\\widehat{\\U}_{k+1}(x)\n\t\t\t\\leq{} &\n\t\t\t\t\\widehat{\\U}_k(x)\n\t\t\t\t-\n\t\t\t\t\\gamma_k(1+\\vartheta_k-\\rho_{k+1}\\vartheta_{k+1})P_{k-1}(x)\n\t\t\t\t+\n\t\t\t\tB_{k+1}\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\left[\n\t\t\t\t\t\\tfrac{\\alpha_k}{1+\\alpha_k}\n\t\t\t\t\t+\n\t\t\t\t\t\\rho_{k+1}\\vartheta_{k+1}(1-\\gamma_k\\ell_k)\n\t\t\t\t\\right]\n\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}).\n\t\t\t\\label{eq:ineq0}\n\t\t\t\\end{align}\n\t\t\\end{lemma}","sha256_utf8":"ff68de82de8338522fc770eb184e7548093d0583953de6042ec1d18d00461111"}]},{"id":"S028","semantic_type":"proof","kind":"proof","title":"主恒等式完整证明","source_units":[{"start":51929,"end":51984,"start_line":1255,"end_line":1257,"raw_tex":"\\begin{proof}\n\t\t\tSee \\cref{proof:thm:eq}.\n\t\t\\end{proof}","sha256_utf8":"a61d0d9d44fb017950498d972454b620776a901638ad0597ae289597556939d5"},{"start":151530,"end":154945,"start_line":3240,"end_line":3397,"raw_tex":"\\begin{proof}[\\textbf{Proof of \\cref{thm:eq}}]%\n\t\t\tBased on the subgradient characterization \\eqref{eq:subgradvarphi} and the definition of \\(\\ell_k\\) in \\eqref{eq:lk}, we have\n\t\t\t\\begin{align}\n\t\t\t\\nonumber\n\t\t\t\t0\n\t\t\t={} &\n\t\t\t\t\\varphi(x^{k-1})\n\t\t\t\t-\n\t\t\t\t\\varphi(x^k)\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_k}\n\t\t\t\t\\innprod{H_k(x^{k-1})-H_k(x^k)}{x^{k-1}-x^k}\n\t\t\t\t-\n\t\t\t\t\\tD_\\varphi(x^{k-1},x^k)\n\t\t\t\\\\\n\t\t\t={} &\n\t\t\t\tP_{k-1}(x)\n\t\t\t\t-\n\t\t\t\tP_k(x)\n\t\t\t\t-\n\t\t\t\t\\tfrac{1-\\gamma_k\\ell_k}{\\gamma_k}\n\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t-\n\t\t\t\t\\tD_\\varphi(x^{k-1},x^k).\n\t\t\t\\label{eq:thetk}\n\t\t\t\\end{align}\n\t\t\tFurthermore, by leveraging the subgradient characterization \\eqref{eq:subgrad} and applying the three point identity of \\cref{thm:3p},\n\t\t\t\\begin{align}\n\t\t\t\\nonumber\n\t\t\t\t0\n\t\t\t={} &\n\t\t\t\tg(x)-g(x^{k+1})-\\innprod{\\tilde{\\nabla}g(x^{k+1})}{x-x^{k+1}}-\\tD_g(x,x^{k+1})\n\t\t\t\\\\\n\t\t\t\\nonumber\n\t\t\t={} &\n\t\t\t\tg(x)-g(x^{k+1})\n\t\t\t\t+\n\t\t\t\t\\innprod{\\nabla f(x^k)}{x- x^{k+1}}\n\t\t\t\\\\\n\t\t\t\\nonumber\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\innprod{\\nabla\\kernel(x^k)-\\nabla\\kernel(x^{k+1})}{x-x^{k+1}}\n\t\t\t\t-\n\t\t\t\t\\tD_g(x,x^{k+1})\n\t\t\t\\\\\n\t\t\t\\nonumber\n\t\t\t={} &\n\t\t\t\tg(x)-g(x^{k+1})\n\t\t\t\t+\n\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x- x^{k+1}}\n\t\t\t\t}_{\\text{(A)}}\n\t\t\t\t-\n\t\t\t\t\\tD_g(x,x^{k+1})\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x,x^k)\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x,x^{k+1})\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x^{k+1},x^k).\n\t\t\t\\label{eq:main}\n\t\t\t\\end{align}\n\t\t\tWe next proceed to expand the term (A) as\n\t\t\t\\begin{align*}\n\t\t\t\t\\text{(A)}\n\t\t\t={} &\n\t\t\t\t\\innprod{\\nabla f(x^k)}{x-x^k}\n\t\t\t\t-\n\t\t\t\t\\innprod{\\nabla f(x^k)}{x^{k+1}-x^k}\n\t\t\t\\\\\n\t\t\t={} &\n\t\t\t\t\\innprod{\\nabla f(x^k)}{x-x^k}\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_k}\n\t\t\t\t\\innprod{H_k(x^{k-1})-\\nabla\\kernel(x^k)}{x^{k+1}-x^k}\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_k}\n\t\t\t\t\\innprod{H_k(x^{k-1})-H_k(x^k)}{x^k-x^{k+1}}\n\t\t\t\\\\\n\t\t\t={} &\n\t\t\t\t\\Bigl[\n\t\t\t\t\tf(x)-f(x^k)-\\D_f(x,x^k)\n\t\t\t\t\\Bigr]\n\t\t\t\t+\n\t\t\t\t\\Bigl[\n\t\t\t\t\tg(x^{k+1})-g(x^k)-\\tD_g(x^{k+1},x^k)\n\t\t\t\t\\Bigr]\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\n\t\t\t\tB_{k+1},\n\t\t\t\\end{align*}\n\t\t\twhich combined with \\eqref{eq:main} gives\n\t\t\t\\begin{align*}\n\t\t\t\t0\n\t\t\t={} &\n\t\t\t\t-P_k(x)\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x,x^k)\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x,x^{k+1})\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\\D_f(x,x^k)\n\t\t\t\t-\\tD_g(x^{k+1},x^k)\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\n\t\t\t\tB_{k+1}\n\t\t\t\t-\n\t\t\t\t\\tD_g(x,x^{k+1}).\n\t\t\t\\end{align*}\n\t\t\tWe may now add \\eqref{eq:main} to \\eqref{eq:thetk} scaled by \\(\\vartheta_{k+1}\\) and multiply everything by \\(\\gamma_{k+1}\\) to obtain\n\t\t\t\\begin{align*}\n\t\t\t\t0\n\t\t\t={} &\n\t\t\t\t-\\gamma_{k+1} P_k(x)\n\t\t\t\t+\n\t\t\t\t\\D_{\\kernel}(x,x^k)\n\t\t\t\t-\n\t\t\t\t\\D_{\\kernel}(x,x^{k+1})\n\t\t\t\t-\n\t\t\t\t\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\gamma_{k+1}\\D_f(x,x^k)\n\t\t\t\t-\n\t\t\t\t\\gamma_{k+1}\\tD_g(x^{k+1},x^k)\n\t\t\t\t+\n\t\t\t\tB_{k+1}\n\t\t\t\t-\n\t\t\t\t\\gamma_{k+1}\\tD_g(x,x^{k+1})\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t+\n\t\t\t\t\\gamma_{k+1}\\vartheta_{k+1}\\left(\n\t\t\t\t\tP_{k-1}(x)\n\t\t\t\t\t-\n\t\t\t\t\tP_k(x)\n\t\t\t\t\t-\n\t\t\t\t\t\\tfrac{1-\\gamma_k\\ell_k}{\\gamma_k}\n\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t\t-\n\t\t\t\t\t\\tD_\\varphi(x^{k-1},x^k)\n\t\t\t\t\\right).\n\t\t\t\\end{align*}\n\t\t\tAfter suitably rearranging, the claimed identity is obtained.\n\n\t\t\tThe inequality follows by neglecting the terms between curly brackets, and further using the identity \\(\\D_{\\kernel}(x^k,x^{k-1})=\\frac{\\alpha_k}{1+\\alpha_k}\\DD_{\\kernel}(x^k,x^{k-1})\\).\n\t\t\\end{proof}","sha256_utf8":"47a4da804197d0a39e1189225c2fc6cdb0c13659b891771f78dc1a0405c12892"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"转引附录证明","source_units":[{"start":51943,"end":51971,"start_line":1256,"end_line":1256,"raw_tex":"\t\t\tSee \\cref{proof:thm:eq}.\n","sha256_utf8":"907fa000691a5c5b6965d6e644ce80ec0985a9cfd418f3c367c11aec174c4b04"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"展开前一目标差","source_units":[{"start":151578,"end":152104,"start_line":3241,"end_line":3265,"raw_tex":"\t\t\tBased on the subgradient characterization \\eqref{eq:subgradvarphi} and the definition of \\(\\ell_k\\) in \\eqref{eq:lk}, we have\n\t\t\t\\begin{align}\n\t\t\t\\nonumber\n\t\t\t\t0\n\t\t\t={} &\n\t\t\t\t\\varphi(x^{k-1})\n\t\t\t\t-\n\t\t\t\t\\varphi(x^k)\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_k}\n\t\t\t\t\\innprod{H_k(x^{k-1})-H_k(x^k)}{x^{k-1}-x^k}\n\t\t\t\t-\n\t\t\t\t\\tD_\\varphi(x^{k-1},x^k)\n\t\t\t\\\\\n\t\t\t={} &\n\t\t\t\tP_{k-1}(x)\n\t\t\t\t-\n\t\t\t\tP_k(x)\n\t\t\t\t-\n\t\t\t\t\\tfrac{1-\\gamma_k\\ell_k}{\\gamma_k}\n\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t-\n\t\t\t\t\\tD_\\varphi(x^{k-1},x^k).\n\t\t\t\\label{eq:thetk}\n\t\t\t\\end{align}\n","sha256_utf8":"c06f5c20d71e493279e8d91404f41dcbc6f9bef658cfbdcc3050531622e6be06"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"三点公式展开近端条件","source_units":[{"start":152104,"end":152993,"start_line":3266,"end_line":3304,"raw_tex":"\t\t\tFurthermore, by leveraging the subgradient characterization \\eqref{eq:subgrad} and applying the three point identity of \\cref{thm:3p},\n\t\t\t\\begin{align}\n\t\t\t\\nonumber\n\t\t\t\t0\n\t\t\t={} &\n\t\t\t\tg(x)-g(x^{k+1})-\\innprod{\\tilde{\\nabla}g(x^{k+1})}{x-x^{k+1}}-\\tD_g(x,x^{k+1})\n\t\t\t\\\\\n\t\t\t\\nonumber\n\t\t\t={} &\n\t\t\t\tg(x)-g(x^{k+1})\n\t\t\t\t+\n\t\t\t\t\\innprod{\\nabla f(x^k)}{x- x^{k+1}}\n\t\t\t\\\\\n\t\t\t\\nonumber\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\innprod{\\nabla\\kernel(x^k)-\\nabla\\kernel(x^{k+1})}{x-x^{k+1}}\n\t\t\t\t-\n\t\t\t\t\\tD_g(x,x^{k+1})\n\t\t\t\\\\\n\t\t\t\\nonumber\n\t\t\t={} &\n\t\t\t\tg(x)-g(x^{k+1})\n\t\t\t\t+\n\t\t\t\t\\underbracket[0.5pt]{\n\t\t\t\t\t\\innprod{\\nabla f(x^k)}{x- x^{k+1}}\n\t\t\t\t}_{\\text{(A)}}\n\t\t\t\t-\n\t\t\t\t\\tD_g(x,x^{k+1})\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x,x^k)\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x,x^{k+1})\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x^{k+1},x^k).\n\t\t\t\\label{eq:main}\n\t\t\t\\end{align}\n","sha256_utf8":"78dbcac902fab476c752492781a4dd6e930ae008a6101ba6afa71cdf406a58ac"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"分解关键梯度内积","source_units":[{"start":152993,"end":153601,"start_line":3305,"end_line":3337,"raw_tex":"\t\t\tWe next proceed to expand the term (A) as\n\t\t\t\\begin{align*}\n\t\t\t\t\\text{(A)}\n\t\t\t={} &\n\t\t\t\t\\innprod{\\nabla f(x^k)}{x-x^k}\n\t\t\t\t-\n\t\t\t\t\\innprod{\\nabla f(x^k)}{x^{k+1}-x^k}\n\t\t\t\\\\\n\t\t\t={} &\n\t\t\t\t\\innprod{\\nabla f(x^k)}{x-x^k}\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_k}\n\t\t\t\t\\innprod{H_k(x^{k-1})-\\nabla\\kernel(x^k)}{x^{k+1}-x^k}\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_k}\n\t\t\t\t\\innprod{H_k(x^{k-1})-H_k(x^k)}{x^k-x^{k+1}}\n\t\t\t\\\\\n\t\t\t={} &\n\t\t\t\t\\Bigl[\n\t\t\t\t\tf(x)-f(x^k)-\\D_f(x,x^k)\n\t\t\t\t\\Bigr]\n\t\t\t\t+\n\t\t\t\t\\Bigl[\n\t\t\t\t\tg(x^{k+1})-g(x^k)-\\tD_g(x^{k+1},x^k)\n\t\t\t\t\\Bigr]\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\n\t\t\t\tB_{k+1},\n\t\t\t\\end{align*}\n","sha256_utf8":"adf5073eb54259c908e6009d69d31d16ae54dfa52c3cd492e156b5e39d702fa9"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"合并目标差与距离项","source_units":[{"start":153601,"end":154002,"start_line":3338,"end_line":3358,"raw_tex":"\t\t\twhich combined with \\eqref{eq:main} gives\n\t\t\t\\begin{align*}\n\t\t\t\t0\n\t\t\t={} &\n\t\t\t\t-P_k(x)\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x,x^k)\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x,x^{k+1})\n\t\t\t\t-\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\\D_f(x,x^k)\n\t\t\t\t-\\tD_g(x^{k+1},x^k)\n\t\t\t\t+\n\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\n\t\t\t\tB_{k+1}\n\t\t\t\t-\n\t\t\t\t\\tD_g(x,x^{k+1}).\n\t\t\t\\end{align*}\n","sha256_utf8":"6f2a6003786027a633ab9af5a46dff8e00ae2f255577e73c5510ce53aea3a0b2"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"加权重排得到恒等式","source_units":[{"start":154002,"end":154741,"start_line":3359,"end_line":3394,"raw_tex":"\t\t\tWe may now add \\eqref{eq:main} to \\eqref{eq:thetk} scaled by \\(\\vartheta_{k+1}\\) and multiply everything by \\(\\gamma_{k+1}\\) to obtain\n\t\t\t\\begin{align*}\n\t\t\t\t0\n\t\t\t={} &\n\t\t\t\t-\\gamma_{k+1} P_k(x)\n\t\t\t\t+\n\t\t\t\t\\D_{\\kernel}(x,x^k)\n\t\t\t\t-\n\t\t\t\t\\D_{\\kernel}(x,x^{k+1})\n\t\t\t\t-\n\t\t\t\t\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\gamma_{k+1}\\D_f(x,x^k)\n\t\t\t\t-\n\t\t\t\t\\gamma_{k+1}\\tD_g(x^{k+1},x^k)\n\t\t\t\t+\n\t\t\t\tB_{k+1}\n\t\t\t\t-\n\t\t\t\t\\gamma_{k+1}\\tD_g(x,x^{k+1})\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t+\n\t\t\t\t\\gamma_{k+1}\\vartheta_{k+1}\\left(\n\t\t\t\t\tP_{k-1}(x)\n\t\t\t\t\t-\n\t\t\t\t\tP_k(x)\n\t\t\t\t\t-\n\t\t\t\t\t\\tfrac{1-\\gamma_k\\ell_k}{\\gamma_k}\n\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t\t-\n\t\t\t\t\t\\tD_\\varphi(x^{k-1},x^k)\n\t\t\t\t\\right).\n\t\t\t\\end{align*}\n\t\t\tAfter suitably rearranging, the claimed identity is obtained.\n","sha256_utf8":"e832eca7552af930cd9e2bec75c35095867bca611b7fa7b9ff1d15ac2661188a"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"舍去非负余项","source_units":[{"start":154742,"end":154932,"start_line":3396,"end_line":3396,"raw_tex":"\t\t\tThe inequality follows by neglecting the terms between curly brackets, and further using the identity \\(\\D_{\\kernel}(x^k,x^{k-1})=\\frac{\\alpha_k}{1+\\alpha_k}\\DD_{\\kernel}(x^k,x^{k-1})\\).\n","sha256_utf8":"231816d1967ca45ab30b228c05a2f3c2c9283ed0a180d858b36dfd5df841a073"}]},{"id":"S036","semantic_type":"lemma","kind":"lemma","title":"Bregman杨氏不等式","source_units":[{"start":53252,"end":53754,"start_line":1282,"end_line":1294,"raw_tex":"\\begin{lemma}[Young's inequality in the Bregman sense]\\label{thm:BY}%\n\t\t\t\tLet \\(\\kernel:\\R^n\\to\\Rinf\\) be Legendre (not necessarily 1-coercive).\n\t\t\t\tThen, for any \\(x\\in\\dom\\kernel\\), \\(y\\in\\interior\\dom\\kernel\\), \\(v\\in\\R^n\\), and \\(\\delta>0\\) one has\n\t\t\t\t\\begin{equation}\\label{eq:BY}\n\t\t\t\t\t\\innprod{x-y}{v}\n\t\t\t\t\\leq\n\t\t\t\t\t\\tfrac{1}{\\delta}\n\t\t\t\t\t\\D_{\\kernel}(x,y)\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{1}{\\delta}\n\t\t\t\t\t\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(y)+\\delta v,\\nabla\\kernel(y)\\bigr).\n\t\t\t\t\\end{equation}\n\t\t\t\\end{lemma}","sha256_utf8":"d8658d38713ca9b38ce6b3bf39ae72c8ab0f13215bcc1529f9942b6b3cbd3018"}]},{"id":"S037","semantic_type":"proof","kind":"proof","title":"Bregman杨氏不等式证明","source_units":[{"start":53758,"end":55364,"start_line":1295,"end_line":1335,"raw_tex":"\\begin{proof}\n\t\t\t\tWithout loss of generality we may assume that \\(\\nabla\\kernel(y)+\\delta v\\in\\dom\\kernel^*\\), for otherwise the right-hand side of \\eqref{eq:BY} is infinite and the inequality trivializes.\n\t\t\t\tSince \\(\\nabla\\kernel(y)\\in\\interior\\dom\\kernel^*\\) by \\cref{fact:D:Dphi*}, for any \\(t\\in(0,1)\\) one has that\n\t\t\t\t\\(\n\t\t\t\t\t\\nabla\\kernel(y)+t\\delta v\\in\\interior\\dom\\kernel^*\n\t\t\t\t\\)\n\t\t\t\tby \\cite[Prop. 3.44]{bauschke2017convex}.\n\t\t\t\tWe have\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\innprod{x-y}{v}\n\t\t\t\t={} &\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\innprod{x-y}{t\\delta v}\n\t\t\t\t\\\\\n\t\t\t\t={} &\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\innprod{x-y}{\\nabla\\kernel(\\nabla\\kernel^*\\bigl(\\nabla\\kernel(y)+t\\delta v)\\bigr)-\\nabla\\kernel(y)}\n\t\t\t\t\\\\\n\t\t\t\t={} &\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\D_{\\kernel}(x,y)\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\D_{\\kernel}\\bigl(y,\\nabla\\kernel^*\\bigl(\\nabla\\kernel(y)+t\\delta v\\bigr)\\bigr)\n\t\t\t\t\\\\\n\t\t\t\t&\n\t\t\t\t\t-\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\D_{\\kernel}\\bigl(x,\\nabla\\kernel^*\\bigl(\\nabla\\kernel(y)+t\\delta v\\bigr)\\bigr)\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\D_{\\kernel}(x,y)\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(y)+t\\delta v,\\nabla\\kernel(y)\\bigr).\n\t\t\t\\end{align*}\n\t\t\t\tThe second equality follows from Legendreness of \\(\\kernel\\) via \\cref{fact:D:Dphi*}, and the third one is the three point identity of \\cref{thm:3p}.\n\t\t\t\tThe inequality follows from \\cref{fact:D:DtoD*} and the fact that \\(\\D_\\kernel\\geq0\\).\n\t\t\t\tBy taking the limit as \\(t\\to1^-\\) and appealing to \\cite[Thm. 7.5]{rockafellar1970convex}, the claimed inequality \\eqref{eq:BY} is obtained.\n\t\t\t\\end{proof}","sha256_utf8":"361539ce716aae034b5b45969f8f8c1eb12f375af8cedb98101d9322dc6898e9"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"对偶域内点正则化","source_units":[{"start":53772,"end":54208,"start_line":1296,"end_line":1302,"raw_tex":"\t\t\t\tWithout loss of generality we may assume that \\(\\nabla\\kernel(y)+\\delta v\\in\\dom\\kernel^*\\), for otherwise the right-hand side of \\eqref{eq:BY} is infinite and the inequality trivializes.\n\t\t\t\tSince \\(\\nabla\\kernel(y)\\in\\interior\\dom\\kernel^*\\) by \\cref{fact:D:Dphi*}, for any \\(t\\in(0,1)\\) one has that\n\t\t\t\t\\(\n\t\t\t\t\t\\nabla\\kernel(y)+t\\delta v\\in\\interior\\dom\\kernel^*\n\t\t\t\t\\)\n\t\t\t\tby \\cite[Prop. 3.44]{bauschke2017convex}.\n\t\t\t\tWe have\n","sha256_utf8":"92ad7ef2775d17179e51231dd3665e8cc497a8448039a14db839e7317b93c181"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"三点恒等式导出上界","source_units":[{"start":54208,"end":55204,"start_line":1303,"end_line":1333,"raw_tex":"\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\innprod{x-y}{v}\n\t\t\t\t={} &\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\innprod{x-y}{t\\delta v}\n\t\t\t\t\\\\\n\t\t\t\t={} &\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\innprod{x-y}{\\nabla\\kernel(\\nabla\\kernel^*\\bigl(\\nabla\\kernel(y)+t\\delta v)\\bigr)-\\nabla\\kernel(y)}\n\t\t\t\t\\\\\n\t\t\t\t={} &\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\D_{\\kernel}(x,y)\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\D_{\\kernel}\\bigl(y,\\nabla\\kernel^*\\bigl(\\nabla\\kernel(y)+t\\delta v\\bigr)\\bigr)\n\t\t\t\t\\\\\n\t\t\t\t&\n\t\t\t\t\t-\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\D_{\\kernel}\\bigl(x,\\nabla\\kernel^*\\bigl(\\nabla\\kernel(y)+t\\delta v\\bigr)\\bigr)\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\D_{\\kernel}(x,y)\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{1}{t\\delta}\n\t\t\t\t\t\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(y)+t\\delta v,\\nabla\\kernel(y)\\bigr).\n\t\t\t\\end{align*}\n\t\t\t\tThe second equality follows from Legendreness of \\(\\kernel\\) via \\cref{fact:D:Dphi*}, and the third one is the three point identity of \\cref{thm:3p}.\n\t\t\t\tThe inequality follows from \\cref{fact:D:DtoD*} and the fact that \\(\\D_\\kernel\\geq0\\).\n","sha256_utf8":"9a197cbcd57651aebec68d50a1c98f9d4d5dc2e506fecb45e8cb48d1fa3f41b0"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"端点取极限","source_units":[{"start":55204,"end":55350,"start_line":1334,"end_line":1334,"raw_tex":"\t\t\t\tBy taking the limit as \\(t\\to1^-\\) and appealing to \\cite[Thm. 7.5]{rockafellar1970convex}, the claimed inequality \\eqref{eq:BY} is obtained.\n","sha256_utf8":"ae30ff553372b78cd379307d9664a2f424189d780a43d601f3bfa3d0b06d7eec"}]},{"id":"S041","semantic_type":"proof","kind":"proof","title":"基本内积上界推导","source_units":[{"start":61183,"end":62142,"start_line":1463,"end_line":1485,"raw_tex":"\t\t\t\tA direct application of Young's inequality of \\cref{thm:BY} allows us to bound the inner product \\(B_{k+1}\\) as\n\t\t\t\t\\begin{align}\n\t\t\t\t\\nonumber\n\t\t\t\t\tB_{k+1}\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\\\\\n\t\t\t\t\\nonumber\n\t\t\t\t&\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(x^k)+\\delta_{k+1}\\bigl[H_k(x^k)-H_k(x^{k-1})\\bigr],\\nabla\\kernel(x^k)\\bigl)\n\t\t\t\t\\shortintertext{%\n\t\t\t\t\tfor any \\(\\delta_{k+1}>0\\), which in terms of the Lipschitz-like estimate \\(\\Lambda_{k,\\delta}\\) as in \\eqref{eq:Lamk} reads\n\t\t\t\t}\n\t\t\t\t={} &\n\t\t\t\t\t\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\delta_{k+1}\\rho_{k+1}}{2}\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}).\n\t\t\t\t\\label{eq:Bk:BY}\n\t\t\t\t\\end{align}\n\t\t\t\tAs we will see, the employment of this inequality combined with a specific choice of \\(\\delta_{k+1}\\) will lead to the stepsize update in the \\refBY{} variant.\n","sha256_utf8":"e79eeccb1ed2a7fa20fdf858e4190c7fa022841c11fcd6b0b8db17dda3a1c5f9"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"应用杨氏界与曲率定义","source_units":[{"start":61183,"end":62142,"start_line":1463,"end_line":1485,"raw_tex":"\t\t\t\tA direct application of Young's inequality of \\cref{thm:BY} allows us to bound the inner product \\(B_{k+1}\\) as\n\t\t\t\t\\begin{align}\n\t\t\t\t\\nonumber\n\t\t\t\t\tB_{k+1}\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\\\\\n\t\t\t\t\\nonumber\n\t\t\t\t&\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(x^k)+\\delta_{k+1}\\bigl[H_k(x^k)-H_k(x^{k-1})\\bigr],\\nabla\\kernel(x^k)\\bigl)\n\t\t\t\t\\shortintertext{%\n\t\t\t\t\tfor any \\(\\delta_{k+1}>0\\), which in terms of the Lipschitz-like estimate \\(\\Lambda_{k,\\delta}\\) as in \\eqref{eq:Lamk} reads\n\t\t\t\t}\n\t\t\t\t={} &\n\t\t\t\t\t\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\delta_{k+1}\\rho_{k+1}}{2}\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}).\n\t\t\t\t\\label{eq:Bk:BY}\n\t\t\t\t\\end{align}\n\t\t\t\tAs we will see, the employment of this inequality combined with a specific choice of \\(\\delta_{k+1}\\) will lead to the stepsize update in the \\refBY{} variant.\n","sha256_utf8":"e79eeccb1ed2a7fa20fdf858e4190c7fa022841c11fcd6b0b8db17dda3a1c5f9"}]},{"id":"S043","semantic_type":"lemma","kind":"lemma","title":"自适应坚固非扩张下界","source_units":[{"start":62419,"end":63064,"start_line":1491,"end_line":1504,"raw_tex":"\\begin{lemma}\\label{thm:FNE}\n\t\t\t\t\tLet \\(\\kernel:\\R^n\\to\\Rinf\\) be Legendre, and \\(f,g:\\R^n\\to\\Rinf\\) be proper, lsc, and convex, with \\(f\\) differentiable on \\(C\\coloneqq\\interior\\dom\\kernel\\) and \\(\\dom g\\cap C\\neq\\emptyset\\).\n\t\t\t\t\tThen, denoting \\(H_k\\coloneqq\\nabla\\kernel-\\gamma_k\\nabla f\\), Bregman proximal gradient iterations \\eqref{eq:BPG} with stepsizes \\(\\gamma_k>0\\) and starting at some \\(x^0\\in C\\) satisfy\n\t\t\t\t\t\\[\n\t\t\t\t\t\tB_{k+1}\n\t\t\t\t\t\\coloneqq\n\t\t\t\t\t\t\\tfrac{\\gamma_{k+1}}{\\gamma_k}\n\t\t\t\t\t\t\\innprod{x^{k+1}-x^k}{H_k(x^k)-H_k(x^{k-1})}\n\t\t\t\t\t\\geq\n\t\t\t\t\t\t\\DD_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\t\\quad\n\t\t\t\t\t\t\\forall k\\geq 1.\n\t\t\t\t\t\\]\n\t\t\t\t\\end{lemma}","sha256_utf8":"822afe267c861899332017e6c103164ec161ea014a91d09309465bbbe0e43e07"}]},{"id":"S044","semantic_type":"proof","kind":"proof","title":"内积下界证明","source_units":[{"start":63069,"end":63817,"start_line":1505,"end_line":1538,"raw_tex":"\\begin{proof}\n\t\t\t\t\tFollows by observing that\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t&\n\t\t\t\t\t\t\\tfrac{\\gamma_{k+1}}{\\gamma_k}\n\t\t\t\t\t\t\\bigl(H_k(x^k)-H_k(x^{k-1})\\bigr)\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\nabla\\kernel(x^{k+1})-\\nabla\\kernel(x^k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\gamma_{k+1}\n\t\t\t\t\t\t\\Bigl[\n\t\t\t\t\t\t\t\\smash{\n\t\t\t\t\t\t\t\t\\overbrace{\n\t\t\t\t\t\t\t\t\t\\tfrac{H_{k+1}(x^k)-\\nabla\\kernel(x^{k+1})}{\\gamma_{k+1}}\n\t\t\t\t\t\t\t\t}^{\\tilde{\\nabla}g(x^{k+1})}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\overbrace{\n\t\t\t\t\t\t\t\t\t\\tfrac{H_k(x^{k-1})-\\nabla\\kernel(x^k)}{\\gamma_k}\n\t\t\t\t\t\t\t\t}^{\\tilde{\\nabla}g(x^k)}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\Bigr],\n\t\t\t\t\t\\end{align*}\n\t\t\t\t\thence that\n\t\t\t\t\t\\(\n\t\t\t\t\t\tB_{k+1}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\DD_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\gamma_{k+1}\\tD_g(x^{k+1},x^k)\n\t\t\t\t\t\\geq\n\t\t\t\t\t\t\\DD_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\t\\).\n\t\t\t\t\\end{proof}","sha256_utf8":"7dc461d3e1e31040331cdd9c9d538e6db54f936bdadca62b7662c7d8f5d43748"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"分解相邻次梯度差","source_units":[{"start":63083,"end":63627,"start_line":1506,"end_line":1527,"raw_tex":"\t\t\t\t\tFollows by observing that\n\t\t\t\t\t\\begin{align*}\n\t\t\t\t\t&\n\t\t\t\t\t\t\\tfrac{\\gamma_{k+1}}{\\gamma_k}\n\t\t\t\t\t\t\\bigl(H_k(x^k)-H_k(x^{k-1})\\bigr)\n\t\t\t\t\t\\\\\n\t\t\t\t\t={} &\n\t\t\t\t\t\t\\nabla\\kernel(x^{k+1})-\\nabla\\kernel(x^k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\gamma_{k+1}\n\t\t\t\t\t\t\\Bigl[\n\t\t\t\t\t\t\t\\smash{\n\t\t\t\t\t\t\t\t\\overbrace{\n\t\t\t\t\t\t\t\t\t\\tfrac{H_{k+1}(x^k)-\\nabla\\kernel(x^{k+1})}{\\gamma_{k+1}}\n\t\t\t\t\t\t\t\t}^{\\tilde{\\nabla}g(x^{k+1})}\n\t\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t\t\\overbrace{\n\t\t\t\t\t\t\t\t\t\\tfrac{H_k(x^{k-1})-\\nabla\\kernel(x^k)}{\\gamma_k}\n\t\t\t\t\t\t\t\t}^{\\tilde{\\nabla}g(x^k)}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\\Bigr],\n\t\t\t\t\t\\end{align*}\n","sha256_utf8":"bc9bd4cc6c0dd059745e72bf1d017f9761e01d9b895ca3c9b5acea98fa9d4c04"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"凸性给出内积下界","source_units":[{"start":63627,"end":63802,"start_line":1528,"end_line":1537,"raw_tex":"\t\t\t\t\thence that\n\t\t\t\t\t\\(\n\t\t\t\t\t\tB_{k+1}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\DD_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\gamma_{k+1}\\tD_g(x^{k+1},x^k)\n\t\t\t\t\t\\geq\n\t\t\t\t\t\t\\DD_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\t\\).\n","sha256_utf8":"5ac6182092532e1157545e58146877ec5052527c3c45b59304b9ea0ca8cc4dec"}]},{"id":"S047","semantic_type":"proof","kind":"proof","title":"对称系数内积界推导","source_units":[{"start":63819,"end":65057,"start_line":1540,"end_line":1584,"raw_tex":"\t\t\t\tWe may thus complement \\eqref{eq:Bk:BY} with a lower bound as\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\DD_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\\leq{} &\n\t\t\t\t\tB_{k+1}\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\delta_{k+1}\\rho_{k+1}}{2}\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}).\n\t\t\t\t\\end{align*}\n\t\t\t\tIf \\(\\kernel\\) has symmetry coefficient \\(\\alpha>0\\) as in \\eqref{eq:alpha}, then note that\n\t\t\t\t\\[\n\t\t\t\t\t\\DD_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\\geq\n\t\t\t\t\t(1+\\alpha)\\D_{\\kernel}(x^{k+1},x^k).\n\t\t\t\t\\]\n\t\t\t\tCombined with the previous inequality, we obtain\n\t\t\t\t\\[\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t1+\\alpha-\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\n\t\t\t\t\t\\right)\n\t\t\t\t\t\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\\leq\n\t\t\t\t\t\\tfrac{\\delta_{k+1}\\rho_{k+1}}{2}\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}),\n\t\t\t\t\\]\n\t\t\t\twhich plugged into \\eqref{eq:Bk:BY} leads to\n\t\t\t\t\\begin{equation}\n\t\t\t\t\\label{eq:Bk:FNEa}\n\t\t\t\t\tB_{k+1}\n\t\t\t\t\\leq\n\t\t\t\t\t\\tfrac{\n\t\t\t\t\t\t(1+\\alpha)\\delta_{k+1}^2\n\t\t\t\t\t}{\n\t\t\t\t\t\t(1+\\alpha)\\delta_{k+1}-\\rho_{k+1}\n\t\t\t\t\t}\n\t\t\t\t\t\\tfrac{\\rho_{k+1}}{2}\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}),\n\t\t\t\t\\end{equation}\n\t\t\t\tholding for any \\(\\delta_{k+1}\\) such that \\((1+\\alpha)\\delta_{k+1}-\\rho_{k+1}>0\\).\n","sha256_utf8":"1f2642f6a89e055df222b5e2078e029a1fdadd4b83c9002930d1635586d946e6"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"合并内积上下界","source_units":[{"start":63819,"end":64171,"start_line":1540,"end_line":1552,"raw_tex":"\t\t\t\tWe may thus complement \\eqref{eq:Bk:BY} with a lower bound as\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\DD_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\\leq{} &\n\t\t\t\t\tB_{k+1}\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\delta_{k+1}\\rho_{k+1}}{2}\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}).\n\t\t\t\t\\end{align*}\n","sha256_utf8":"aea15d82d3ca735c7514fe2ccd0e343d66df442caebf9d584db23ec3fc4cf563"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"利用正对称系数","source_units":[{"start":64171,"end":64364,"start_line":1553,"end_line":1558,"raw_tex":"\t\t\t\tIf \\(\\kernel\\) has symmetry coefficient \\(\\alpha>0\\) as in \\eqref{eq:alpha}, then note that\n\t\t\t\t\\[\n\t\t\t\t\t\\DD_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\\geq\n\t\t\t\t\t(1+\\alpha)\\D_{\\kernel}(x^{k+1},x^k).\n\t\t\t\t\\]\n","sha256_utf8":"dda5a4d2df1f50dd9f36fe30cfa7fb3d907edd9b08abb3fa370d502f7ad170d7"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"消去新迭代距离","source_units":[{"start":64364,"end":65057,"start_line":1559,"end_line":1584,"raw_tex":"\t\t\t\tCombined with the previous inequality, we obtain\n\t\t\t\t\\[\n\t\t\t\t\t\\left(\n\t\t\t\t\t\t1+\\alpha-\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\n\t\t\t\t\t\\right)\n\t\t\t\t\t\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t\\leq\n\t\t\t\t\t\\tfrac{\\delta_{k+1}\\rho_{k+1}}{2}\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}),\n\t\t\t\t\\]\n\t\t\t\twhich plugged into \\eqref{eq:Bk:BY} leads to\n\t\t\t\t\\begin{equation}\n\t\t\t\t\\label{eq:Bk:FNEa}\n\t\t\t\t\tB_{k+1}\n\t\t\t\t\\leq\n\t\t\t\t\t\\tfrac{\n\t\t\t\t\t\t(1+\\alpha)\\delta_{k+1}^2\n\t\t\t\t\t}{\n\t\t\t\t\t\t(1+\\alpha)\\delta_{k+1}-\\rho_{k+1}\n\t\t\t\t\t}\n\t\t\t\t\t\\tfrac{\\rho_{k+1}}{2}\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}),\n\t\t\t\t\\end{equation}\n\t\t\t\tholding for any \\(\\delta_{k+1}\\) such that \\((1+\\alpha)\\delta_{k+1}-\\rho_{k+1}>0\\).\n","sha256_utf8":"b2412341a06627587a568efae0ac701580d56515af08e5d055cc6ef8bc393832"}]},{"id":"S051","semantic_type":"proof","kind":"proof","title":"基本势函数条件推导","source_units":[{"start":65241,"end":69383,"start_line":1590,"end_line":1697,"raw_tex":"\t\t\tBy bounding the inner product term \\(B_{k+1}\\) with \\eqref{eq:Bk:BY}, the inequality \\eqref{eq:ineq0} in \\cref{thm:eq} reveals that Bregman proximal gradient iterations \\eqref{eq:BPG} with arbitrary stepsizes \\(\\gamma_k>0\\) satisfy\n\t\t\t\\begin{align}\n\t\t\t\\nonumber\n\t\t\t&\n\t\t\t\t\\widehat{\\U}_{k+1}(x)\n\t\t\t\t-\n\t\t\t\t\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\n\t\t\t\t\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\\\\\n\t\t\t\\nonumber\n\t\t\t\\leq{} &\n\t\t\t\t\\widehat{\\U}_k(x)\n\t\t\t\t-\n\t\t\t\t\\tfrac{\\rho_k}{\\delta_k}\n\t\t\t\t\\D_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t-\n\t\t\t\t\\gamma_k(1+\\vartheta_k-\\rho_{k+1}\\vartheta_{k+1})P_{k-1}(x)\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\left[\n\t\t\t\t\t\\bigl(1-\\tfrac{\\rho_k}{\\delta_k}\\bigr)\n\t\t\t\t\t-\n\t\t\t\t\t\\rho_{k+1}(1+\\alpha_k)\n\t\t\t\t\t\\tfrac{\n\t\t\t\t\t\t\\frac{\\delta_{k+1}}{2}\n\t\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\vartheta_{k+1}\n\t\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\t}{\\alpha_k}\n\t\t\t\t\\right]\n\t\t\t\t\\D_{\\kernel}(x^k,x^{k-1})\n\t\t\t\\label{eq:UkBYdt:leq}\n\t\t\t\\end{align}\n\t\t\tfor any \\(x\\in\\dom\\varphi\\cap\\dom\\kernel\\), \\(\\vartheta_k\\geq0\\), and \\(\\delta_k>0\\), \\(k\\in\\N\\).\n\t\t\tImposing that the terms in brackets multiplying \\(P_{k-1}(x)\\) and \\(\\D_{\\kernel}(x^k,x^{k-1})\\) in the right-hand side of \\eqref{eq:UkBYdt:leq} are negative amounts to the following two conditions:\n\t\t\t\\begin{subequations}\\label{eq:gamk:ineq12}\n\t\t\t\t\\begin{gather}\n\t\t\t\t\\label{eq:gamk:ineq1}\n\t\t\t\t\t\\vartheta_{k+1}\\rho_{k+1}\\leq 1+\\vartheta_k\n\t\t\t\t\\shortintertext{and}\n\t\t\t\t\\label{eq:gamk:ineq2}\n\t\t\t\t\t1-\\tfrac{\\rho_k}{\\delta_k}\n\t\t\t\t\\geq\n\t\t\t\t\t\\rho_{k+1}\n\t\t\t\t\t\\tfrac{1+\\alpha_k}{\\alpha_k}\n\t\t\t\t\t\\bigl[\n\t\t\t\t\t\t\\tfrac{\\delta_{k+1}}{2}\n\t\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\vartheta_{k+1}(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\\bigr].\n\t\t\t\t\\end{gather}\n\t\t\t\\end{subequations}\n\t\t\tIn line with the analyses of \\cite{malitsky2020adaptive,latafat2024adaptive}, a convenient choice for the parameter \\(\\delta_{k+1}\\) is \\(\\delta_{k+1} = 2\\rho_{k+1}\\).\n\t\t\tHowever, this choice is not feasible in our more general setting.\n\t\t\tSpecifically, selecting \\(\\rho_{k+1}\\) to satisfy \\eqref{eq:gamk:ineq2} requires knowledge of the quantity \\(\\Lambda_{k,\\delta_{k+1}}\\), which generally depends on \\(\\delta_{k+1}\\) itself (except in the special case where \\(\\kernel\\) is quadratic).\n\t\t\tAs a result, setting \\(\\delta_{k+1}=2\\rho_{k+1}\\) would create a circular dependency between the two parameters.\n\t\t\tTo complicate things further, note that the left-hand side of \\eqref{eq:gamk:ineq2} indicates that a constraint \\(\\delta_k>\\rho_k\\) must be in place in order to ensure the existence of a \\(\\rho_{k+1}>0\\) satisfying the inequality.\n\n\n\t\t\tIn order to resolve this circular dependence, we introduce a parameter \\(\\hat\\rho_{k+1}\\) that shall provide an overestimate\n\t\t\t\\begin{equation}\\label{eq:rhokhatgeq}\n\t\t\t\t\\hat\\rho_{k+1}\\geq\\rho_{k+1}\n\t\t\t\\end{equation}\n\t\t\twhile only being based on information available at iteration \\(k\\).\n\t\t\tIts explicit value will be revealed shortly after.\n\t\t\tThen, we may conveniently select \\(\\delta_k=2\\hat\\rho_k\\) and \\(\\vartheta_k=\\hat\\rho_k\\) so that \\eqref{eq:gamk:ineq12} simplifies as\n\t\t\t\\[\n\t\t\t\t\\rho_{k+1}\\hat\\rho_{k+1}\n\t\t\t\\leq\n\t\t\t\t1+\\hat\\rho_k\n\t\t\t\\quad\\text{and}\\quad\n\t\t\t\t1-\\tfrac{\\rho_k}{2\\hat\\rho_k}\n\t\t\t\\geq\n\t\t\t\t\\rho_{k+1}\n\t\t\t\t\\hat\\rho_{k+1}\n\t\t\t\t\\tfrac{1+\\alpha_k}{\\alpha_k}\n\t\t\t\t\\bigl[\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t-\n\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\\bigr],\n\t\t\t\\]\n\t\t\tthat is,\n\t\t\t\\begin{equation}\\label{eq:BY:rhok*+}\n\t\t\t\t\\rho_{k+1}\n\t\t\t\\leq\n\t\t\t\t\\min\\left\\{\n\t\t\t\t\t\\frac{1+\\hat\\rho_k}{\\hat\\rho_{k+1}}\n\t\t\t\t\t,\\,\n\t\t\t\t\t\\frac{\\alpha_k}{1+\\alpha_k}\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t1+ \\frac{\\hat\\rho_k - \\rho_k}{\\hat\\rho_k}\n\t\t\t\t\t}{\n\t\t\t\t\t\t2\\hat\\rho_{k+1}\n\t\t\t\t\t\t\\bigl[\n\t\t\t\t\t\t\t\\Lambda_{k,2\\hat\\rho_{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\t\\bigr]_+\n\t\t\t\t\t}\n\t\t\t\t\\right\\}.\n\t\t\t\\end{equation}\n\t\t\tDue to \\eqref{eq:rhokhatgeq} the term \\(\\frac{\\hat\\rho_k - \\rho_k}{\\hat\\rho_k}\\) in the numerator of the second update is always nonnegative.\n\t\t\tUsing this, we show in the next lemma that the update rule of \\refBY{} always complies with the above bound.\n\t\t\tIn fact, while it is possible to retain the term \\(\\frac{\\hat\\rho_k - \\rho_k}{\\hat\\rho_k}\\) in the update of \\refBY{}, for the sake of a neater expression we opted to omit it at the cost of introducing slight conservatism.\n","sha256_utf8":"8c3175070e13880a535d989531a7ff0a629f0ee24f35a3ceaf6a48993b36661f"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"内积界代入主恒等式","source_units":[{"start":65241,"end":66435,"start_line":1590,"end_line":1626,"raw_tex":"\t\t\tBy bounding the inner product term \\(B_{k+1}\\) with \\eqref{eq:Bk:BY}, the inequality \\eqref{eq:ineq0} in \\cref{thm:eq} reveals that Bregman proximal gradient iterations \\eqref{eq:BPG} with arbitrary stepsizes \\(\\gamma_k>0\\) satisfy\n\t\t\t\\begin{align}\n\t\t\t\\nonumber\n\t\t\t&\n\t\t\t\t\\widehat{\\U}_{k+1}(x)\n\t\t\t\t-\n\t\t\t\t\\tfrac{\\rho_{k+1}}{\\delta_{k+1}}\n\t\t\t\t\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\\\\\n\t\t\t\\nonumber\n\t\t\t\\leq{} &\n\t\t\t\t\\widehat{\\U}_k(x)\n\t\t\t\t-\n\t\t\t\t\\tfrac{\\rho_k}{\\delta_k}\n\t\t\t\t\\D_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t-\n\t\t\t\t\\gamma_k(1+\\vartheta_k-\\rho_{k+1}\\vartheta_{k+1})P_{k-1}(x)\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\left[\n\t\t\t\t\t\\bigl(1-\\tfrac{\\rho_k}{\\delta_k}\\bigr)\n\t\t\t\t\t-\n\t\t\t\t\t\\rho_{k+1}(1+\\alpha_k)\n\t\t\t\t\t\\tfrac{\n\t\t\t\t\t\t\\frac{\\delta_{k+1}}{2}\n\t\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\vartheta_{k+1}\n\t\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\t}{\\alpha_k}\n\t\t\t\t\\right]\n\t\t\t\t\\D_{\\kernel}(x^k,x^{k-1})\n\t\t\t\\label{eq:UkBYdt:leq}\n\t\t\t\\end{align}\n\t\t\tfor any \\(x\\in\\dom\\varphi\\cap\\dom\\kernel\\), \\(\\vartheta_k\\geq0\\), and \\(\\delta_k>0\\), \\(k\\in\\N\\).\n\t\t\tImposing that the terms in brackets multiplying \\(P_{k-1}(x)\\) and \\(\\D_{\\kernel}(x^k,x^{k-1})\\) in the right-hand side of \\eqref{eq:UkBYdt:leq} are negative amounts to the following two conditions:\n","sha256_utf8":"db574ecf46bde0f02eabcdac4925a98a1878cc8cba47cd6570f8f356dd638541"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"提取下降条件及参数障碍","source_units":[{"start":66435,"end":67732,"start_line":1627,"end_line":1649,"raw_tex":"\t\t\t\\begin{subequations}\\label{eq:gamk:ineq12}\n\t\t\t\t\\begin{gather}\n\t\t\t\t\\label{eq:gamk:ineq1}\n\t\t\t\t\t\\vartheta_{k+1}\\rho_{k+1}\\leq 1+\\vartheta_k\n\t\t\t\t\\shortintertext{and}\n\t\t\t\t\\label{eq:gamk:ineq2}\n\t\t\t\t\t1-\\tfrac{\\rho_k}{\\delta_k}\n\t\t\t\t\\geq\n\t\t\t\t\t\\rho_{k+1}\n\t\t\t\t\t\\tfrac{1+\\alpha_k}{\\alpha_k}\n\t\t\t\t\t\\bigl[\n\t\t\t\t\t\t\\tfrac{\\delta_{k+1}}{2}\n\t\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\vartheta_{k+1}(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\\bigr].\n\t\t\t\t\\end{gather}\n\t\t\t\\end{subequations}\n\t\t\tIn line with the analyses of \\cite{malitsky2020adaptive,latafat2024adaptive}, a convenient choice for the parameter \\(\\delta_{k+1}\\) is \\(\\delta_{k+1} = 2\\rho_{k+1}\\).\n\t\t\tHowever, this choice is not feasible in our more general setting.\n\t\t\tSpecifically, selecting \\(\\rho_{k+1}\\) to satisfy \\eqref{eq:gamk:ineq2} requires knowledge of the quantity \\(\\Lambda_{k,\\delta_{k+1}}\\), which generally depends on \\(\\delta_{k+1}\\) itself (except in the special case where \\(\\kernel\\) is quadratic).\n\t\t\tAs a result, setting \\(\\delta_{k+1}=2\\rho_{k+1}\\) would create a circular dependency between the two parameters.\n\t\t\tTo complicate things further, note that the left-hand side of \\eqref{eq:gamk:ineq2} indicates that a constraint \\(\\delta_k>\\rho_k\\) must be in place in order to ensure the existence of a \\(\\rho_{k+1}>0\\) satisfying the inequality.\n","sha256_utf8":"878bc5ab9360225fa96be042eec19834ce792fd028a3056e936d11d97e898082"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"引入步长比上估计","source_units":[{"start":67734,"end":68079,"start_line":1652,"end_line":1657,"raw_tex":"\t\t\tIn order to resolve this circular dependence, we introduce a parameter \\(\\hat\\rho_{k+1}\\) that shall provide an overestimate\n\t\t\t\\begin{equation}\\label{eq:rhokhatgeq}\n\t\t\t\t\\hat\\rho_{k+1}\\geq\\rho_{k+1}\n\t\t\t\\end{equation}\n\t\t\twhile only being based on information available at iteration \\(k\\).\n\t\t\tIts explicit value will be revealed shortly after.\n","sha256_utf8":"ec0b46f3e3f86f603cea9d51e7d58c85223b8763f11f5ddf0aa39a123057760d"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"固定参数得到可行步长界","source_units":[{"start":68079,"end":68900,"start_line":1658,"end_line":1694,"raw_tex":"\t\t\tThen, we may conveniently select \\(\\delta_k=2\\hat\\rho_k\\) and \\(\\vartheta_k=\\hat\\rho_k\\) so that \\eqref{eq:gamk:ineq12} simplifies as\n\t\t\t\\[\n\t\t\t\t\\rho_{k+1}\\hat\\rho_{k+1}\n\t\t\t\\leq\n\t\t\t\t1+\\hat\\rho_k\n\t\t\t\\quad\\text{and}\\quad\n\t\t\t\t1-\\tfrac{\\rho_k}{2\\hat\\rho_k}\n\t\t\t\\geq\n\t\t\t\t\\rho_{k+1}\n\t\t\t\t\\hat\\rho_{k+1}\n\t\t\t\t\\tfrac{1+\\alpha_k}{\\alpha_k}\n\t\t\t\t\\bigl[\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t-\n\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\\bigr],\n\t\t\t\\]\n\t\t\tthat is,\n\t\t\t\\begin{equation}\\label{eq:BY:rhok*+}\n\t\t\t\t\\rho_{k+1}\n\t\t\t\\leq\n\t\t\t\t\\min\\left\\{\n\t\t\t\t\t\\frac{1+\\hat\\rho_k}{\\hat\\rho_{k+1}}\n\t\t\t\t\t,\\,\n\t\t\t\t\t\\frac{\\alpha_k}{1+\\alpha_k}\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t1+ \\frac{\\hat\\rho_k - \\rho_k}{\\hat\\rho_k}\n\t\t\t\t\t}{\n\t\t\t\t\t\t2\\hat\\rho_{k+1}\n\t\t\t\t\t\t\\bigl[\n\t\t\t\t\t\t\t\\Lambda_{k,2\\hat\\rho_{k+1}}\n\t\t\t\t\t\t\t-\n\t\t\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\t\\bigr]_+\n\t\t\t\t\t}\n\t\t\t\t\\right\\}.\n\t\t\t\\end{equation}\n","sha256_utf8":"a125bee1ba58c11a99002dab1705959fe76a37ddc9ab8053732de4c2ed6b7325"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"放松非负修正项","source_units":[{"start":68900,"end":69383,"start_line":1695,"end_line":1697,"raw_tex":"\t\t\tDue to \\eqref{eq:rhokhatgeq} the term \\(\\frac{\\hat\\rho_k - \\rho_k}{\\hat\\rho_k}\\) in the numerator of the second update is always nonnegative.\n\t\t\tUsing this, we show in the next lemma that the update rule of \\refBY{} always complies with the above bound.\n\t\t\tIn fact, while it is possible to retain the term \\(\\frac{\\hat\\rho_k - \\rho_k}{\\hat\\rho_k}\\) in the update of \\refBY{}, for the sake of a neater expression we opted to omit it at the cost of introducing slight conservatism.\n","sha256_utf8":"078d8366cd4e6891a07ccc3e13caf9cbae9c116029535cfb93e30292f09ef107"}]},{"id":"S057","semantic_type":"lemma","kind":"lemma","title":"基本算法势函数下降","source_units":[{"start":69387,"end":70312,"start_line":1699,"end_line":1727,"raw_tex":"\\begin{lemma}\\label{thm:BY:leq}%\n\t\t\t\tSuppose that \\cref{ass:basic} holds, and consider the iterates generated by Bregman proximal gradient iterations \\eqref{eq:BPG} with \\(\\gamma_k\\) and \\(\\hat\\rho_k\\) selected according to \\refBY.\n\t\t\t\tThen, both \\eqref{eq:rhokhatgeq} and \\eqref{eq:BY:rhok*+} are satisfied for any \\(k\\in\\N\\).\n\n\t\t\t\tMoreover, denoting\n\t\t\t\t\\begin{equation}\\label{eq:UkBY}\n\t\t\t\t\t\\U_k(x)\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\D_{\\kernel}(x,x^k)\n\t\t\t\t\t+\n\t\t\t\t\t\\gamma_k(1+\\hat\\rho_k)P_{k-1}(x)\n\t\t\t\t\t+\n\t\t\t\t\t(1-\\tfrac{\\rho_k}{2\\hat\\rho_k})\n\t\t\t\t\t\\D_{\\kernel}(x^k,x^{k-1}),\n\t\t\t\t\\end{equation}\n\t\t\t\tone has that\n\t\t\t\t\\begin{equation}\\label{eq:UkBY:leq}\n\t\t\t\t\t\\U_{k+1}(x)\n\t\t\t\t\\leq\n\t\t\t\t\t\\U_k(x)\n\t\t\t\t\t-\n\t\t\t\t\t\\gamma_k(\n\t\t\t\t\t\t\\smash{\n\t\t\t\t\t\t\t\\overbrace{1+\\hat\\rho_k-\\rho_{k+1}\\hat\\rho_{k+1}}^{\\geq\\hat\\rho_k-\\rho_k\\geq0}\n\t\t\t\t\t\t}\n\t\t\t\t\t)P_{k-1}(x)\n\t\t\t\t\\end{equation}\n\t\t\t\tholds for any \\(x\\in\\dom\\varphi\\cap\\dom\\kernel\\) and \\(k\\in\\N\\).\n\t\t\t\\end{lemma}","sha256_utf8":"e8296d101bc63c013ef2843513a408ff95ffd8a2b3caaafbdc3989c81e530b6f"}]},{"id":"S058","semantic_type":"proof","kind":"proof","title":"基本势函数下降证明","source_units":[{"start":70316,"end":71160,"start_line":1728,"end_line":1745,"raw_tex":"\\begin{proof}\n\t\t\t\tThe update \\eqref{eq:BY:rhok*} clearly ensures that \\(\\rho_{k+1}\\leq\\hat\\rho_{k+1}\\) always holds.\n\t\t\t\tNote that the second element in the minimum within \\eqref{eq:BY:rhok*+} coincides with that in \\eqref{eq:BY:rhok*};\n\t\t\t\tmoreover, since \\(\\rho_{k+1}\\leq\\hat\\rho_{k+1}\\) one has that\n\t\t\t\t\\(\n\t\t\t\t\t\\rho_{k+1}\\hat\\rho_{k+1}\n\t\t\t\t\\leq\n\t\t\t\t\t\\hat\\rho_{k+1}^2\n\t\t\t\t=\n\t\t\t\t\t1+\\rho_k\n\t\t\t\t\\leq\n\t\t\t\t\t1+\\hat\\rho_k\n\t\t\t\t\\),\n\t\t\t\taltogether confirming that the validity of \\eqref{eq:BY:rhok*+}.\n\n\t\t\t\tObserve further that the bound \\(\\hat\\rho_k\\geq\\rho_k\\) ensures that both elements in the minimum of \\eqref{eq:BY:rhok*} are strictly positive, and thus so are the generated stepsizes \\(\\gamma_k\\).\n\t\t\t\tFinally, \\eqref{eq:UkBY:leq} follows from \\eqref{eq:UkBYdt:leq} with the specified choices of \\(\\vartheta_k\\) and \\(\\delta_k\\).\n\t\t\t\\end{proof}","sha256_utf8":"87b85d76648f9d7bf37dbea25ac6db8fc041c4ca76e181b2c1c652bc1fe2b7d1"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"验证步长可行条件","source_units":[{"start":70330,"end":70811,"start_line":1729,"end_line":1741,"raw_tex":"\t\t\t\tThe update \\eqref{eq:BY:rhok*} clearly ensures that \\(\\rho_{k+1}\\leq\\hat\\rho_{k+1}\\) always holds.\n\t\t\t\tNote that the second element in the minimum within \\eqref{eq:BY:rhok*+} coincides with that in \\eqref{eq:BY:rhok*};\n\t\t\t\tmoreover, since \\(\\rho_{k+1}\\leq\\hat\\rho_{k+1}\\) one has that\n\t\t\t\t\\(\n\t\t\t\t\t\\rho_{k+1}\\hat\\rho_{k+1}\n\t\t\t\t\\leq\n\t\t\t\t\t\\hat\\rho_{k+1}^2\n\t\t\t\t=\n\t\t\t\t\t1+\\rho_k\n\t\t\t\t\\leq\n\t\t\t\t\t1+\\hat\\rho_k\n\t\t\t\t\\),\n\t\t\t\taltogether confirming that the validity of \\eqref{eq:BY:rhok*+}.\n","sha256_utf8":"50d0c2a4261bef3646773ea8fc5545e8017006a8b9015f47923336540ddb824e"}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","title":"正步长与势函数下降","source_units":[{"start":70812,"end":71146,"start_line":1743,"end_line":1744,"raw_tex":"\t\t\t\tObserve further that the bound \\(\\hat\\rho_k\\geq\\rho_k\\) ensures that both elements in the minimum of \\eqref{eq:BY:rhok*} are strictly positive, and thus so are the generated stepsizes \\(\\gamma_k\\).\n\t\t\t\tFinally, \\eqref{eq:UkBY:leq} follows from \\eqref{eq:UkBYdt:leq} with the specified choices of \\(\\vartheta_k\\) and \\(\\delta_k\\).\n","sha256_utf8":"1f9684aab40bbbd44043debcbf7aab31d0985f93d7dd6eae4da704a34370f77b"}]},{"id":"S061","semantic_type":"proof","kind":"proof","title":"对称变体下降界推导","source_units":[{"start":71326,"end":73613,"start_line":1751,"end_line":1826,"raw_tex":"\t\t\tIn case \\(\\kernel\\) has a symmetry coefficient \\(\\alpha>0\\), we may leverage the bound \\eqref{eq:Bk:FNEa} for \\(B_{k+1}\\).\n\t\t\tBy doing so, the inequality \\eqref{eq:ineq0} in \\cref{thm:eq} becomes\n\t\t\t\\begin{multline*}\n\t\t\t\t\\widehat{\\U}_{k+1}(x)\n\t\t\t\\leq\n\t\t\t\t\\widehat{\\U}_k(x)\n\t\t\t\t-\n\t\t\t\t\\gamma_k(1+\\vartheta_k-\\rho_{k+1}\\vartheta_{k+1})P_{k-1}(x)\n\t\t\t\\\\\n\t\t\t\t-\n\t\t\t\t\\left\\{\n\t\t\t\t\t\\tfrac{\\alpha_k}{1+\\alpha_k}\n\t\t\t\t\t-\n\t\t\t\t\t\\rho_{k+1}\n\t\t\t\t\t\\left[\n\t\t\t\t\t\t\\tfrac{\n\t\t\t\t\t\t\t(1+\\alpha)\\delta_{k+1}^2\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t(1+\\alpha)\\delta_{k+1}-\\rho_{k+1}\n\t\t\t\t\t\t}\n\t\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\vartheta_{k+1}(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\\right]\n\t\t\t\t\\right\\}\n\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}).\n\t\t\t\\end{multline*}\n\n\t\t\tOnce again, we introduce a parameter \\(\\hat\\rho_{k+1}\\) (to be specified later) that satisfies \\eqref{eq:rhokhatgeq} whilst based on information available at iteration \\(k\\).\n\t\t\tWe can conveniently set \\(\\delta_{k+1}=\\vartheta_{k+1}=\\frac{2}{1+\\alpha}\\hat\\rho_{k+1}\\), and use the bound\n\t\t\t\\(\n\t\t\t\t(1+\\alpha)\\delta_{k+1}-\\rho_{k+1}\n\t\t\t\\geq\n\t\t\t\t(1+\\alpha)\\delta_{k+1}-\\hat\\rho_{k+1}\n\t\t\t\\)\n\t\t\tto simplify the coefficient of \\(\\Lambda_{k,\\delta_{k+1}}\\).\n\t\t\tCombined with the fact that\n\t\t\t\\(\n\t\t\t\t\\tfrac{\\alpha_k}{1+\\alpha_k}\n\t\t\t\\geq\n\t\t\t\t\\tfrac{\\alpha}{1+\\alpha}\n\t\t\t\\),\n\t\t\tthe inequality simplifies as\n\t\t\t\\begin{align*}\n\t\t\t\t\\U^{\\alpha}_{k+1}(x)\n\t\t\t\\coloneqq{} &\n\t\t\t\t\\D_{\\kernel}(x,x^{k+1})\n\t\t\t\t+\n\t\t\t\t\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t+\n\t\t\t\t\\gamma_{k+1}\\left(1+\\tfrac{2}{1+\\alpha}\\hat\\rho_{k+1}\\right)P_k(x)\n\t\t\t\\\\\n\t\t\t\\leq{} &\n\t\t\t\t\\U^{\\alpha}_k(x)\n\t\t\t\t-\n\t\t\t\t\\gamma_k(1+\\tfrac{2}{1+\\alpha}\\hat\\rho_k-\\tfrac{2}{1+\\alpha}\\hat\\rho_{k+1}\\rho_{k+1})P_{k-1}(x)\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\left\\{\n\t\t\t\t\t\\tfrac{\\alpha}{1+\\alpha}\n\t\t\t\t\t-\n\t\t\t\t\t\\tfrac{2}{1+\\alpha}\n\t\t\t\t\t\\hat\\rho_{k+1}\n\t\t\t\t\t\\rho_{k+1}\n\t\t\t\t\t\\left[\n\t\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\\right]\n\t\t\t\t\\right\\}\n\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}),\n\t\t\t\\end{align*}\n\t\t\twhere \\(\\U^{\\alpha}_k(x)\\) corresponds to \\(\\widehat{\\U}_k(x)\\) as in \\cref{thm:eq} with \\(\\vartheta_k=\\tfrac{2}{1+\\alpha}\\hat\\rho_k\\).\n\t\t\tBy imposing negativity of the coefficients of \\(P_{k-1}(x)\\) and \\(\\DD_{\\kernel}(x^k,x^{k-1})\\) as done in the previous subsection, the following analogue of \\cref{thm:BY:leq} for \\refFNEa{} is derived.\n","sha256_utf8":"fa495447373c0ff208586740e356aaf390e7da20aa3bf8ff622b49c65676f8fb"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"改进内积界代入势函数","source_units":[{"start":71326,"end":72044,"start_line":1751,"end_line":1778,"raw_tex":"\t\t\tIn case \\(\\kernel\\) has a symmetry coefficient \\(\\alpha>0\\), we may leverage the bound \\eqref{eq:Bk:FNEa} for \\(B_{k+1}\\).\n\t\t\tBy doing so, the inequality \\eqref{eq:ineq0} in \\cref{thm:eq} becomes\n\t\t\t\\begin{multline*}\n\t\t\t\t\\widehat{\\U}_{k+1}(x)\n\t\t\t\\leq\n\t\t\t\t\\widehat{\\U}_k(x)\n\t\t\t\t-\n\t\t\t\t\\gamma_k(1+\\vartheta_k-\\rho_{k+1}\\vartheta_{k+1})P_{k-1}(x)\n\t\t\t\\\\\n\t\t\t\t-\n\t\t\t\t\\left\\{\n\t\t\t\t\t\\tfrac{\\alpha_k}{1+\\alpha_k}\n\t\t\t\t\t-\n\t\t\t\t\t\\rho_{k+1}\n\t\t\t\t\t\\left[\n\t\t\t\t\t\t\\tfrac{\n\t\t\t\t\t\t\t(1+\\alpha)\\delta_{k+1}^2\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\t(1+\\alpha)\\delta_{k+1}-\\rho_{k+1}\n\t\t\t\t\t\t}\n\t\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t\\vartheta_{k+1}(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\\right]\n\t\t\t\t\\right\\}\n\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}).\n\t\t\t\\end{multline*}\n","sha256_utf8":"51b9978e5a6e881a3f048dc2eac75e6481a72ef7686fa45d29558ed9e88657cf"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"选择参数并放松系数","source_units":[{"start":72045,"end":72645,"start_line":1780,"end_line":1794,"raw_tex":"\t\t\tOnce again, we introduce a parameter \\(\\hat\\rho_{k+1}\\) (to be specified later) that satisfies \\eqref{eq:rhokhatgeq} whilst based on information available at iteration \\(k\\).\n\t\t\tWe can conveniently set \\(\\delta_{k+1}=\\vartheta_{k+1}=\\frac{2}{1+\\alpha}\\hat\\rho_{k+1}\\), and use the bound\n\t\t\t\\(\n\t\t\t\t(1+\\alpha)\\delta_{k+1}-\\rho_{k+1}\n\t\t\t\\geq\n\t\t\t\t(1+\\alpha)\\delta_{k+1}-\\hat\\rho_{k+1}\n\t\t\t\\)\n\t\t\tto simplify the coefficient of \\(\\Lambda_{k,\\delta_{k+1}}\\).\n\t\t\tCombined with the fact that\n\t\t\t\\(\n\t\t\t\t\\tfrac{\\alpha_k}{1+\\alpha_k}\n\t\t\t\\geq\n\t\t\t\t\\tfrac{\\alpha}{1+\\alpha}\n\t\t\t\\),\n\t\t\tthe inequality simplifies as\n","sha256_utf8":"b9b88e49f1a87c58fc9266b624ebccfc04caee24249af49a19dd17fcb85ac74b"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"得到变体势函数下降条件","source_units":[{"start":72645,"end":73613,"start_line":1795,"end_line":1826,"raw_tex":"\t\t\t\\begin{align*}\n\t\t\t\t\\U^{\\alpha}_{k+1}(x)\n\t\t\t\\coloneqq{} &\n\t\t\t\t\\D_{\\kernel}(x,x^{k+1})\n\t\t\t\t+\n\t\t\t\t\\D_{\\kernel}(x^{k+1},x^k)\n\t\t\t\t+\n\t\t\t\t\\gamma_{k+1}\\left(1+\\tfrac{2}{1+\\alpha}\\hat\\rho_{k+1}\\right)P_k(x)\n\t\t\t\\\\\n\t\t\t\\leq{} &\n\t\t\t\t\\U^{\\alpha}_k(x)\n\t\t\t\t-\n\t\t\t\t\\gamma_k(1+\\tfrac{2}{1+\\alpha}\\hat\\rho_k-\\tfrac{2}{1+\\alpha}\\hat\\rho_{k+1}\\rho_{k+1})P_{k-1}(x)\n\t\t\t\\\\\n\t\t\t&\n\t\t\t\t-\n\t\t\t\t\\left\\{\n\t\t\t\t\t\\tfrac{\\alpha}{1+\\alpha}\n\t\t\t\t\t-\n\t\t\t\t\t\\tfrac{2}{1+\\alpha}\n\t\t\t\t\t\\hat\\rho_{k+1}\n\t\t\t\t\t\\rho_{k+1}\n\t\t\t\t\t\\left[\n\t\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t\t\t-\n\t\t\t\t\t\t(1-\\gamma_k\\ell_k)\n\t\t\t\t\t\\right]\n\t\t\t\t\\right\\}\n\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1}),\n\t\t\t\\end{align*}\n\t\t\twhere \\(\\U^{\\alpha}_k(x)\\) corresponds to \\(\\widehat{\\U}_k(x)\\) as in \\cref{thm:eq} with \\(\\vartheta_k=\\tfrac{2}{1+\\alpha}\\hat\\rho_k\\).\n\t\t\tBy imposing negativity of the coefficients of \\(P_{k-1}(x)\\) and \\(\\DD_{\\kernel}(x^k,x^{k-1})\\) as done in the previous subsection, the following analogue of \\cref{thm:BY:leq} for \\refFNEa{} is derived.\n","sha256_utf8":"d36d727b28857d75d75e76ae3a08e81096b9e8ff67fdc03eac21db3db34b784f"}]},{"id":"S065","semantic_type":"lemma","kind":"lemma","title":"对称变体势函数下降","source_units":[{"start":73617,"end":74659,"start_line":1828,"end_line":1860,"raw_tex":"\\begin{lemma}\\label{thm:FNEa:leq}%\n\t\t\t\tAdditionally to \\cref{ass:basic}, suppose that \\(\\kernel\\) has symmetry coefficient \\(\\alpha>0\\) and consider the iterates generated by Bregman proximal gradient iterations \\eqref{eq:BPG} with \\(\\gamma_k\\) and \\(\\hat\\rho_k\\) selected according to \\refFNEa.\n\t\t\t\tThen, denoting\n\t\t\t\t\\begin{equation}\\label{eq:UkFNEa}\n\t\t\t\t\t\\U^{\\alpha}_k(x)\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\D_{\\kernel}(x,x^k)\n\t\t\t\t\t+\n\t\t\t\t\t\\D_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t\t+\n\t\t\t\t\t\\gamma_k\\left(1+\\tfrac{2}{1+\\alpha}\\hat\\rho_k\\right)P_{k-1}(x),\n\t\t\t\t\\end{equation}\n\t\t\t\tone has that\n\t\t\t\t\\begin{equation}\\label{eq:UkFNEa:leq}\n\t\t\t\t\t\\U^{\\alpha}_{k+1}(x)\n\t\t\t\t\\leq\n\t\t\t\t\t\\U^{\\alpha}_k(x)\n\t\t\t\t\t-\n\t\t\t\t\t\\gamma_k\\Bigl(\n\t\t\t\t\t\t\\smash{\n\t\t\t\t\t\t\t\\overbrace{\n\t\t\t\t\t\t\t\t1+\\tfrac{2}{1+\\alpha}\\hat\\rho_k-\\tfrac{2}{1+\\alpha}\\hat\\rho_{k+1}\\rho_{k+1}\n\t\t\t\t\t\t\t}^{\n\t\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t\\tfrac{2}{1+\\alpha}(\\hat\\rho_k-\\rho_k)\n\t\t\t\t\t\t\t\t\\geq\n\t\t\t\t\t\t\t\t0\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t\\Bigr)P_{k-1}(x)\n\t\t\t\t\\end{equation}\n\t\t\t\tholds for any \\(x\\in\\dom\\varphi\\cap\\dom\\kernel\\) and \\(k\\in\\N\\).\n\t\t\t\\end{lemma}","sha256_utf8":"eeca379d1b18ba5709869c3cc5e05f2ae7c80695ba9bb1c6dd9534b1de1da3ed"}]},{"id":"S066","semantic_type":"lemma","kind":"lemma","title":"势函数极限与最小目标差界","source_units":[{"start":75180,"end":76061,"start_line":1873,"end_line":1884,"raw_tex":"\\begin{lemma}\\label{thm:descent}%\n\t\t\tSuppose that \\cref{ass:basic} holds, and consider the iterates generated by Bregman proximal gradient iterations \\eqref{eq:BPG} with \\(\\gamma_k\\) and \\(\\hat\\rho_k\\) selected according to \\refBY.\n\t\t\tThen, with \\(\\U_k\\) as in \\eqref{eq:UkBY},\n\t\t\tthe following hold for any \\(x\\in\\dom\\kernel\\) satisfying \\(\\varphi(x)\\leq\\inf_{k\\in\\N}\\varphi(x^k)\\):\n\t\t\t\\begin{enumerate}\n\t\t\t\\item \\label{thm:SD}%\n\t\t\t\t\\(\\bigl(\\U_k(x)\\bigr)_{k\\in\\N}\\) decreases and converges to a finite value.\n\t\t\t\\item \\label{thm:Pkmin}%\n\t\t\t\t\\(P_K^{\\rm min}(x)\\leq\\frac{\\U_0(x)}{\\sum_{k=1}^{K+1}\\gamma_k}\\) for every \\(K\\geq 1\\), where \\(P_K^{\\rm min}(x)\\coloneqq\\min_{k\\leq K}P_k(x)\\).\n\t\t\t\\end{enumerate}\n\t\t\tWhen \\(\\kernel\\) has symmetry coefficient \\(\\alpha>0\\), all remains true for the updates of \\refFNEa, with \\(\\U_k\\gets\\U^{\\alpha}_k\\) as in \\eqref{eq:UkFNEa}.\n\t\t\\end{lemma}","sha256_utf8":"3a0d04fde66610eecf4c628b0433176a548f2129a85b4144c97288cd58a66d08"}]},{"id":"S067","semantic_type":"proof","kind":"proof","title":"下降与目标差界完整证明","source_units":[{"start":76064,"end":76124,"start_line":1885,"end_line":1887,"raw_tex":"\\begin{proof}\n\t\t\tSee \\cref{proof:thm:descent}.\n\t\t\\end{proof}","sha256_utf8":"24ff1bb006606ef6458c5466a1bdccdfd0d1628f0af60158f74a72e17d70652d"},{"start":155187,"end":156680,"start_line":3405,"end_line":3449,"raw_tex":"\\begin{proof}[\\textbf{Proof of \\cref{thm:descent}}]%\n\t\t\tIt suffices to prove the claim for \\refBY; the case of \\refFNEa{} under the needed assumptions follows by simply replacing \\(\\U_k\\gets\\U^{\\alpha}_k\\).\n\t\t\tThe assumption on \\(x\\) ensures that \\(P_k(x)\\geq0\\) holds for all \\(k\\), implying both that \\(\\U_k(x)\\geq0\\) and the claimed monotonic decrease of \\(\\bigl(\\U_k(x)\\bigr)_{k\\in\\N}\\) with finite limit by virtue of \\cref{thm:BY:leq,thm:FNEa:leq}.\n\t\t\tMore precisely, one has that\n\t\t\t\\[\n\t\t\t\t0\n\t\t\t\\leq\n\t\t\t\t\\U_{k+1}(x)\n\t\t\t\\leq\n\t\t\t\t\\U_k(x)\n\t\t\t\t-\n\t\t\t\t\\gamma_k(1+q\\hat\\rho_k-q\\hat\\rho_{k+1}\\rho_{k+1})P_{k-1}(x),\n\t\t\t\\]\n\t\t\twhere \\(q=1\\) for \\refBY{} and \\(q=\\frac{1+\\alpha}{2\\alpha}\\) for \\refFNEa{}.\n\t\t\tA telescoping argument yields that\n\t\t\t\\begin{align*}\n\t\t\t\tP_K^{\\min}(x)\n\t\t\t\t\\sum_{k=1}^K\n\t\t\t\t\\gamma_k(1+q\\hat\\rho_k-q\\hat\\rho_{k+1}\\rho_{k+1})\n\t\t\t\\leq{} &\n\t\t\t\t\\sum_{k=1}^K\n\t\t\t\t\\gamma_k(1+q\\hat\\rho_k-q\\hat\\rho_{k+1}\\rho_{k+1})P_{k-1}(x)\\\\\n\t\t\t\\leq{} &\n\t\t\t\t\\U_1(x) - \\U_{K+1}(x)\\\\\n\t\t\t\\leq{} &\n\t\t\t\t\\U_1(x) - \\gamma_{K+1}(1+q\\hat\\rho_{K+1})P_K^{\\rm min}(x),\n\t\t\t\\end{align*}\n\t\t\thence that\n\t\t\t\\begin{align*}\n\t\t\t\tP_K^{\\min}(x)\n\t\t\t\\leq{} &\n\t\t\t\t\\frac{\\U_1(x)\n\t\t\t\t\t}{\n\t\t\t\t\t\\sum_{k=1}^K(\\gamma_k + q\\gamma_k\\hat\\rho_k- q\\gamma_{k+1}\\hat\\rho_{k+1}) + \\gamma_{K+1} + q\\gamma_{K+1}\\hat\\rho_{K+1}\n\t\t\t\t\t}\\\\\n\t\t\t\\leq{} &\n\t\t\t\t\\frac{\\U_1(x)\n\t\t\t\t\t}{\n\t\t\t\t\t\\gamma_1\\hat\\rho_1\n\t\t\t\t\t+\n\t\t\t\t\t\\sum_{k=1}^{K+1}\\gamma_k\n\t\t\t\t\t}.\n\t\t\t\\end{align*}\n\t\t\tSince \\(\\U_1(x)\\leq\\U_0(x)\\), the claimed inequality follows.\n\t\t\\end{proof}","sha256_utf8":"849745db63590884dd216c077c603f285c2ee29905d85fbec871d03d5ad96eae"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"转引下降引理附录证明","source_units":[{"start":76078,"end":76111,"start_line":1886,"end_line":1886,"raw_tex":"\t\t\tSee \\cref{proof:thm:descent}.\n","sha256_utf8":"d48b9882effd29ba5b5e2211f7cb5a5d422c3729cffc5ce20fe1de7aa1bae3fe"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"非负性与势函数单调收敛","source_units":[{"start":155240,"end":155887,"start_line":3406,"end_line":3418,"raw_tex":"\t\t\tIt suffices to prove the claim for \\refBY; the case of \\refFNEa{} under the needed assumptions follows by simply replacing \\(\\U_k\\gets\\U^{\\alpha}_k\\).\n\t\t\tThe assumption on \\(x\\) ensures that \\(P_k(x)\\geq0\\) holds for all \\(k\\), implying both that \\(\\U_k(x)\\geq0\\) and the claimed monotonic decrease of \\(\\bigl(\\U_k(x)\\bigr)_{k\\in\\N}\\) with finite limit by virtue of \\cref{thm:BY:leq,thm:FNEa:leq}.\n\t\t\tMore precisely, one has that\n\t\t\t\\[\n\t\t\t\t0\n\t\t\t\\leq\n\t\t\t\t\\U_{k+1}(x)\n\t\t\t\\leq\n\t\t\t\t\\U_k(x)\n\t\t\t\t-\n\t\t\t\t\\gamma_k(1+q\\hat\\rho_k-q\\hat\\rho_{k+1}\\rho_{k+1})P_{k-1}(x),\n\t\t\t\\]\n\t\t\twhere \\(q=1\\) for \\refBY{} and \\(q=\\frac{1+\\alpha}{2\\alpha}\\) for \\refFNEa{}.\n","sha256_utf8":"b32faebff85b4e8b64be117a49f77fcb5ba9a40cf624ed16db55639384872f80"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"望远镜求和控制目标差","source_units":[{"start":155887,"end":156258,"start_line":3419,"end_line":3431,"raw_tex":"\t\t\tA telescoping argument yields that\n\t\t\t\\begin{align*}\n\t\t\t\tP_K^{\\min}(x)\n\t\t\t\t\\sum_{k=1}^K\n\t\t\t\t\\gamma_k(1+q\\hat\\rho_k-q\\hat\\rho_{k+1}\\rho_{k+1})\n\t\t\t\\leq{} &\n\t\t\t\t\\sum_{k=1}^K\n\t\t\t\t\\gamma_k(1+q\\hat\\rho_k-q\\hat\\rho_{k+1}\\rho_{k+1})P_{k-1}(x)\\\\\n\t\t\t\\leq{} &\n\t\t\t\t\\U_1(x) - \\U_{K+1}(x)\\\\\n\t\t\t\\leq{} &\n\t\t\t\t\\U_1(x) - \\gamma_{K+1}(1+q\\hat\\rho_{K+1})P_K^{\\rm min}(x),\n\t\t\t\\end{align*}\n","sha256_utf8":"d53e55af8e1f40b65651af28907c83a3f7cf2dbee58ffb4f02e2b7a1bca7acf9"}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","title":"消去步长比边界项","source_units":[{"start":156258,"end":156667,"start_line":3432,"end_line":3448,"raw_tex":"\t\t\thence that\n\t\t\t\\begin{align*}\n\t\t\t\tP_K^{\\min}(x)\n\t\t\t\\leq{} &\n\t\t\t\t\\frac{\\U_1(x)\n\t\t\t\t\t}{\n\t\t\t\t\t\\sum_{k=1}^K(\\gamma_k + q\\gamma_k\\hat\\rho_k- q\\gamma_{k+1}\\hat\\rho_{k+1}) + \\gamma_{K+1} + q\\gamma_{K+1}\\hat\\rho_{K+1}\n\t\t\t\t\t}\\\\\n\t\t\t\\leq{} &\n\t\t\t\t\\frac{\\U_1(x)\n\t\t\t\t\t}{\n\t\t\t\t\t\\gamma_1\\hat\\rho_1\n\t\t\t\t\t+\n\t\t\t\t\t\\sum_{k=1}^{K+1}\\gamma_k\n\t\t\t\t\t}.\n\t\t\t\\end{align*}\n\t\t\tSince \\(\\U_1(x)\\leq\\U_0(x)\\), the claimed inequality follows.\n","sha256_utf8":"65e1dcdb2dfc73980912b7a03feee6f7c63d3ccf9ab7df7b3704ae9077ef997e"}]},{"id":"S072","semantic_type":"lemma","kind":"lemma","title":"小步长下曲率估计趋一","source_units":[{"start":80067,"end":80807,"start_line":1957,"end_line":1972,"raw_tex":"\\begin{lemma}\\label{thm:Lamkto1}%\n\t\t\t\tSuppose that \\cref{ass:basic} holds, and consider a sequence \\((x^k)_{k\\in\\N}\\) generated by Bregman proximal gradient iterations \\eqref{eq:BPG} with stepsizes \\(\\gamma_k>0\\).\n\t\t\t\tSuppose that \\((x^k)_{k\\in\\N}\\) is contained in a compact set \\(\\K\\subset C\\), and consider the ratio\n\t\t\t\t\\[\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t=\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t2\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(x^k)+\\delta_{k+1}\\bigl[H_k(x^k)-H_k(x^{k-1})\\bigr],\\nabla\\kernel(x^k)\\bigr)\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\delta_{k+1}^2\n\t\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t\t}\n\t\t\t\t\\]\n\t\t\t\tas in \\eqref{eq:Lamk}, where \\((\\delta_k)_{k\\in\\N}\\subset\\R_{++}\\) is a bounded sequence.\n\t\t\t\tIf \\(\\gamma_k\\to0\\), then \\(\\Lambda_{k,\\delta_{k+1}}\\to1\\).\n\t\t\t\\end{lemma}","sha256_utf8":"449b4c900cbeda7e1f578759419e56a1f241341b4662a46e4d124180a5b2186b"}]},{"id":"S073","semantic_type":"proof","kind":"proof","title":"曲率估计极限证明","source_units":[{"start":80811,"end":84359,"start_line":1973,"end_line":2104,"raw_tex":"\\begin{proof}\n\t\t\t\tFor notational conciseness, let us denote \\(u^k\\coloneqq\\nabla\\kernel(x^k)-\\nabla\\kernel(x^{k-1})\\) and \\(v^k\\coloneqq\\nabla f(x^k)-\\nabla f(x^{k-1})\\), so that \\(H_k(x^k)-H_k(x^{k-1})=u^k-\\gamma_k v^k\\).\n\t\t\t\tFor any \\(k\\in\\N\\) there exists\n\t\t\t\t\\(\\xi^k\\) on the line segment between \\(\\nabla\\kernel(x^k)\\) and \\(\\nabla\\kernel(x^k)+\\delta_{k+1}(H_k(x^k)-H_k(x^{k-1}))\\)\n\t\t\t\tand\n\t\t\t\t\\(\\eta^k\\) on the line segment between \\(\\nabla\\kernel(x^k)\\) and \\(\\nabla\\kernel(x^{k-1})\\)\n\t\t\t\tsuch that\n\t\t\t\t\\begin{align}\n\t\t\t\t\\label{eq:Lamknum}\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t={} &\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t2\n\t\t\t\t\t\t\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(x^k)+\\delta_{k+1}\\bigl[H_k(x^k)-H_k(x^{k-1})\\bigr],\\nabla\\kernel(x^k)\\bigr)\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\delta_{k+1}^2\n\t\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t\t}\n\t\t\t\t\\\\\n\t\t\t\t\\nonumber\n\t\t\t\t={} &\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\xi_k)\n\t\t\t\t\t\t\t\\bigl[H_k(x^k)-H_k(x^{k-1})\\bigr]\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tH_k(x^k)-H_k(x^{k-1})\n\t\t\t\t\t\t}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\eta_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\\\\\n\t\t\t\t\\nonumber\n\t\t\t\t={} &\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\xi_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\eta_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\t-\n\t\t\t\t\t2\\gamma_k\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\xi_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tv^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\eta_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\t+\n\t\t\t\t\t\\gamma_k^2\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\xi_k)\n\t\t\t\t\t\t\tv^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tv^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\eta_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}.\n\t\t\t\t\\end{align}\n\t\t\t\tSince \\(\\kernel\\) is twice continuously differentiable with \\(\\nabla^2\\kernel\\succ0\\) on \\(C\\), and \\(\\K\\subset C\\) is compact and convex, one has that\n\t\t\t\t\\[\n\t\t\t\t\\textstyle\n\t\t\t\t\tL_{\\kernel,\\K}\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\sup_{\\K}\\norm{\\nabla^2\\kernel}\n\t\t\t\t=\n\t\t\t\t\t\\left(\\inf_{\\nabla\\kernel(\\K)}\\lambda_{\\rm min}(\\nabla^2\\kernel^*)\\right)^{-1}\n\t\t\t\t\\]\n\t\t\t\tis finite (\\(L_{\\kernel,\\K}\\) being the Lipschitz modulus of \\(\\nabla\\kernel\\) on \\(\\K\\)).\n\t\t\t\tAs such, one has that\n\t\t\t\t\\(\n\t\t\t\t\t\\innprod{\\nabla^2\\kernel^*(\\eta_k)u^k}{u^k}\n\t\t\t\t\\geq\n\t\t\t\t\tL_{\\kernel,\\K}^{-1}\\norm{u^k}^2\n\t\t\t\t\\)\n\t\t\t\tfor all \\(k\\).\n\t\t\t\tMoreover, letting \\(L_{f,\\K}^{\\kernel}\\) denote a smoothness modulus of \\(f\\) relative to \\(\\kernel\\) on \\(\\K\\), ensured to exist by \\cref{ass:f}, we infer from \\cite[Prop. 2.5(ii)]{ahookhosh2021bregman} that\n\t\t\t\t\\(\n\t\t\t\t\t\\norm{v^k}\\leq L_{f,\\K}^{\\kernel} L_{\\kernel,\\K}\\norm{u^k}\n\t\t\t\t\\)\n\t\t\t\tholds for all \\(k\\).\n\t\t\t\tTherefore,\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\abs{\\Lambda_{k,\\delta_{k+1}}-1}\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\norm{\\nabla^2\\kernel^*(\\xi_k)-\\nabla^2\\kernel^*(\\eta_k)}\n\t\t\t\t\tL_{\\kernel,\\K}\n\t\t\t\t\t+\n\t\t\t\t\t2\\gamma_k\n\t\t\t\t\tL_{f,\\K}^{\\kernel} L_{\\kernel,\\K}^2\n\t\t\t\t\t\\norm{\\nabla^2\\kernel^*(\\xi_k)}\n\t\t\t\t\\\\\n\t\t\t\t&\n\t\t\t\t\t+\n\t\t\t\t\t\\gamma_k^2\n\t\t\t\t\tL_{\\kernel,\\K}^3\n\t\t\t\t\t(L_{f,\\K}^{\\kernel})^2\n\t\t\t\t\t\\norm{\\nabla^2\\kernel^*(\\xi_k)}.\n\t\t\t\t\\end{align*}\n\t\t\t\tIf \\(\\gamma_k\\to0\\), then eventually \\(\\gamma_k<\\frac{1}{L_{f,\\K}^{\\kernel}}\\) and standard results ensure that \\((x^k)_{k\\in\\N}\\) converges to some point \\(x\\in\\K\\).\n\t\t\t\tIn this case, \\((\\xi^k)_{k\\in\\N}\\) and \\((\\eta^k)_{k\\in\\N}\\) converge to \\(\\nabla\\kernel(x)\\) (the former sequence because \\((\\delta_k)_{k\\in\\N}\\) is bounded), and by continuity of \\(\\nabla^2\\kernel^*\\) on \\(\\K\\) the right-hand side in the above inequality vanishes.\n\t\t\t\\end{proof}","sha256_utf8":"fcab76f068a38fe1da06435243fad0a9a94627b989a76f8551cf34e3cf16cc15"}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","title":"梯度增量与二阶展开","source_units":[{"start":80825,"end":82583,"start_line":1974,"end_line":2061,"raw_tex":"\t\t\t\tFor notational conciseness, let us denote \\(u^k\\coloneqq\\nabla\\kernel(x^k)-\\nabla\\kernel(x^{k-1})\\) and \\(v^k\\coloneqq\\nabla f(x^k)-\\nabla f(x^{k-1})\\), so that \\(H_k(x^k)-H_k(x^{k-1})=u^k-\\gamma_k v^k\\).\n\t\t\t\tFor any \\(k\\in\\N\\) there exists\n\t\t\t\t\\(\\xi^k\\) on the line segment between \\(\\nabla\\kernel(x^k)\\) and \\(\\nabla\\kernel(x^k)+\\delta_{k+1}(H_k(x^k)-H_k(x^{k-1}))\\)\n\t\t\t\tand\n\t\t\t\t\\(\\eta^k\\) on the line segment between \\(\\nabla\\kernel(x^k)\\) and \\(\\nabla\\kernel(x^{k-1})\\)\n\t\t\t\tsuch that\n\t\t\t\t\\begin{align}\n\t\t\t\t\\label{eq:Lamknum}\n\t\t\t\t\t\\Lambda_{k,\\delta_{k+1}}\n\t\t\t\t={} &\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t2\n\t\t\t\t\t\t\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(x^k)+\\delta_{k+1}\\bigl[H_k(x^k)-H_k(x^{k-1})\\bigr],\\nabla\\kernel(x^k)\\bigr)\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\delta_{k+1}^2\n\t\t\t\t\t\t\\DD_{\\kernel}(x^k,x^{k-1})\n\t\t\t\t\t}\n\t\t\t\t\\\\\n\t\t\t\t\\nonumber\n\t\t\t\t={} &\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\xi_k)\n\t\t\t\t\t\t\t\\bigl[H_k(x^k)-H_k(x^{k-1})\\bigr]\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tH_k(x^k)-H_k(x^{k-1})\n\t\t\t\t\t\t}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\eta_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\\\\\n\t\t\t\t\\nonumber\n\t\t\t\t={} &\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\xi_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\eta_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\t-\n\t\t\t\t\t2\\gamma_k\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\xi_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tv^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\eta_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\t+\n\t\t\t\t\t\\gamma_k^2\n\t\t\t\t\t\\frac{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\xi_k)\n\t\t\t\t\t\t\tv^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tv^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}{\n\t\t\t\t\t\t\\innprod{\n\t\t\t\t\t\t\t\\nabla^2\\kernel^*(\\eta_k)\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}{\n\t\t\t\t\t\t\tu^k\n\t\t\t\t\t\t}\n\t\t\t\t\t}.\n\t\t\t\t\\end{align}\n","sha256_utf8":"5fffc574bc85c8e5b1b8abb4bc6bac5bbb7862afb8a500b1d2ca34a99c86f300"}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","title":"紧集上的分母谱下界","source_units":[{"start":82583,"end":83178,"start_line":2062,"end_line":2078,"raw_tex":"\t\t\t\tSince \\(\\kernel\\) is twice continuously differentiable with \\(\\nabla^2\\kernel\\succ0\\) on \\(C\\), and \\(\\K\\subset C\\) is compact and convex, one has that\n\t\t\t\t\\[\n\t\t\t\t\\textstyle\n\t\t\t\t\tL_{\\kernel,\\K}\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\sup_{\\K}\\norm{\\nabla^2\\kernel}\n\t\t\t\t=\n\t\t\t\t\t\\left(\\inf_{\\nabla\\kernel(\\K)}\\lambda_{\\rm min}(\\nabla^2\\kernel^*)\\right)^{-1}\n\t\t\t\t\\]\n\t\t\t\tis finite (\\(L_{\\kernel,\\K}\\) being the Lipschitz modulus of \\(\\nabla\\kernel\\) on \\(\\K\\)).\n\t\t\t\tAs such, one has that\n\t\t\t\t\\(\n\t\t\t\t\t\\innprod{\\nabla^2\\kernel^*(\\eta_k)u^k}{u^k}\n\t\t\t\t\\geq\n\t\t\t\t\tL_{\\kernel,\\K}^{-1}\\norm{u^k}^2\n\t\t\t\t\\)\n\t\t\t\tfor all \\(k\\).\n","sha256_utf8":"06cc40c91c50c62bcf0ca46ce3174f67fb0920a9f28636e7c651e1a9e32eeb83"}]},{"id":"S076","semantic_type":"proof_step","kind":"proof_step","title":"控制目标梯度增量","source_units":[{"start":83178,"end":83494,"start_line":2079,"end_line":2083,"raw_tex":"\t\t\t\tMoreover, letting \\(L_{f,\\K}^{\\kernel}\\) denote a smoothness modulus of \\(f\\) relative to \\(\\kernel\\) on \\(\\K\\), ensured to exist by \\cref{ass:f}, we infer from \\cite[Prop. 2.5(ii)]{ahookhosh2021bregman} that\n\t\t\t\t\\(\n\t\t\t\t\t\\norm{v^k}\\leq L_{f,\\K}^{\\kernel} L_{\\kernel,\\K}\\norm{u^k}\n\t\t\t\t\\)\n\t\t\t\tholds for all \\(k\\).\n","sha256_utf8":"efcc7d1eae44bbd94ba7ed906c69a15829e2d09c7defb4be736160ae64bdb608"}]},{"id":"S077","semantic_type":"proof_step","kind":"proof_step","title":"估计曲率偏差","source_units":[{"start":83494,"end":83903,"start_line":2084,"end_line":2101,"raw_tex":"\t\t\t\tTherefore,\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\abs{\\Lambda_{k,\\delta_{k+1}}-1}\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\norm{\\nabla^2\\kernel^*(\\xi_k)-\\nabla^2\\kernel^*(\\eta_k)}\n\t\t\t\t\tL_{\\kernel,\\K}\n\t\t\t\t\t+\n\t\t\t\t\t2\\gamma_k\n\t\t\t\t\tL_{f,\\K}^{\\kernel} L_{\\kernel,\\K}^2\n\t\t\t\t\t\\norm{\\nabla^2\\kernel^*(\\xi_k)}\n\t\t\t\t\\\\\n\t\t\t\t&\n\t\t\t\t\t+\n\t\t\t\t\t\\gamma_k^2\n\t\t\t\t\tL_{\\kernel,\\K}^3\n\t\t\t\t\t(L_{f,\\K}^{\\kernel})^2\n\t\t\t\t\t\\norm{\\nabla^2\\kernel^*(\\xi_k)}.\n\t\t\t\t\\end{align*}\n","sha256_utf8":"db961e4d74ed38c25f2b8ef772dd4917f9d87d5030cef3f048ef4ce84ce25e4b"}]},{"id":"S078","semantic_type":"proof_step","kind":"proof_step","title":"收敛与连续性消去偏差","source_units":[{"start":83903,"end":84345,"start_line":2102,"end_line":2103,"raw_tex":"\t\t\t\tIf \\(\\gamma_k\\to0\\), then eventually \\(\\gamma_k<\\frac{1}{L_{f,\\K}^{\\kernel}}\\) and standard results ensure that \\((x^k)_{k\\in\\N}\\) converges to some point \\(x\\in\\K\\).\n\t\t\t\tIn this case, \\((\\xi^k)_{k\\in\\N}\\) and \\((\\eta^k)_{k\\in\\N}\\) converge to \\(\\nabla\\kernel(x)\\) (the former sequence because \\((\\delta_k)_{k\\in\\N}\\) is bounded), and by continuity of \\(\\nabla^2\\kernel^*\\) on \\(\\K\\) the right-hand side in the above inequality vanishes.\n","sha256_utf8":"21451ab2b2179905f4a227abbeda16dc05ffd1cee9f40a39c986ddfc89fbc96d"}]},{"id":"S079","semantic_type":"lemma","kind":"lemma","title":"最优点附近近端步稳定性","source_units":[{"start":84729,"end":85443,"start_line":2109,"end_line":2125,"raw_tex":"\\begin{lemma}\\label{lem:prox:property:optimal}%\n\t\t\t\tAdditionally to \\cref{ass:basic} suppose that \\(\\argmin_C\\varphi\\neq\\emptyset\\).\n\t\t\t\tLet a sequence \\((y^k)_{k\\in\\N}\\) contained in \\(C\\) and converging to a point \\(y^\\star\\in\\argmin_C\\varphi\\) be fixed, and for every \\(k\\in\\N\\) let\n\t\t\t\t\\begin{equation}\\label{eq:baryk}\n\t\t\t\t\t\\bar y^k\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\argmin_w\\left\\{\n\t\t\t\t\t\tg(w)\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\innprod{\\nabla f(y^k)}{w-y^k}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\tfrac{1}{\\gamma_{k+1}}\\D_{\\kernel}(w,y^k)\n\t\t\t\t\t\\right\\}\n\t\t\t\t\\end{equation}\n\t\t\t\twhere \\((\\gamma_k)_{k\\in\\N}\\) is a bounded sequence of strictly positive stepsizes.\n\t\t\t\tThen, \\(\\bar y^k\\to y^\\star\\) and \\(\\gamma_{k+1}(\\varphi(\\bar y^k) - \\min\\varphi)\\to0\\).\n\t\t\t\\end{lemma}","sha256_utf8":"00974878c62b3bda6564c12b288eb8520a8d90d2eb375f2c44fb8e902e419c1b"}]},{"id":"S080","semantic_type":"proof","kind":"proof","title":"近端步稳定性证明","source_units":[{"start":85447,"end":87901,"start_line":2126,"end_line":2179,"raw_tex":"\\begin{proof}\n\t\t\t\tConsider the ``left'' Bregman proximal operator \\(\\prox_{g}:\\interior\\dom\\kernel\\to\\dom\\kernel\\) defined as\n\t\t\t\t\\[\n\t\t\t\t\t\\prox_{g}(y)\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\argmin_{w\\in\\R^n}\\{g(w)+\\D_{\\kernel}(w,y)\\}\n\t\t\t\t\\]\n\t\t\t\t(note that \\(\\prox_{g}(y)\\in\\interior\\dom\\kernel\\subseteq\\dom\\kernel\\) for any \\(y\\in\\interior\\dom\\kernel\\), owing to \\cref{ass:basic}).\n\t\t\t\tThe mapping \\(\\prox_{\\gamma_{k+1}g}\\circ\\nabla\\kernel^*\\) is \\(\\nabla\\kernel\\)-firmly-nonexpansive, in the sense that\n\t\t\t\t\\begin{equation}\\label{eq:BPG:FNE}\n\t\t\t\t\t\\DD_{\\kernel}(\\bar y_1,\\bar y_2)\n\t\t\t\t\\leq\n\t\t\t\t\t\\innprod{\\eta_1-\\eta_2}{\\bar y_1-\\bar y_2}\n\t\t\t\t\\end{equation}\n\t\t\t\tholds for any \\(\\eta_i\\in\\R^2\\) and \\(\\bar y_i = \\prox_{\\gamma_{k+1} g}\\circ\\nabla\\kernel^*(\\eta_i)\\), \\(i=1,2\\);\n\t\t\t\ta proof of this fact can be found in \\cite[Lem. 4.2]{wang2022bregman} (see also \\cite[Thm. 4.9]{wang2025bregman} for its \\emph{equivalence} to convexity of \\(g\\)).\n\t\t\t\tNote that \\eqref{eq:baryk} can equivalently be written as\n\t\t\t\t\\[\n\t\t\t\t\t\\bar y^k\n\t\t\t\t=\n\t\t\t\t\t\\prox_{\\gamma_{k+1} g}\\circ\\nabla\\kernel^*(H_{k+1}(y^k)),\n\t\t\t\t\\]\n\t\t\t\twhere we remind that \\(H_{k+1}=\\nabla\\kernel-\\gamma_{k+1}\\nabla f\\).\n\t\t\t\tUsing \\(\\nabla\\kernel\\)-firm nonexpansiveness and recalling that \\(y^\\star = \\prox_{\\gamma_{k+1} g}\\circ\\nabla\\kernel^*(H_{k+1}(y^\\star))\\) we have\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\DD_{\\kernel}(\\bar y^k,y^\\star)\n\t\t\t\t\\stackrel{\\text{\\clap{\\eqref{eq:BPG:FNE}}}}{\\leq}{} &\n\t\t\t\t\t\\innprod{H_{k+1}(y^k) - H_{k+1}(y^\\star)}{\\bar y^k - y^\\star}\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\Bigl(\n\t\t\t\t\t\t\\norm{\\nabla \\kernel(y^k) - \\nabla \\kernel(y^\\star)}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\gamma_{k+1}\\norm{\\nabla f(y^k) - \\nabla f(y^\\star)}\n\t\t\t\t\t\\Bigr)\n\t\t\t\t\t\\norm{\\bar y^k - y^\\star},\n\t\t\t\t\\end{align*}\n\t\t\t\twhich vanishes as \\(k\\to\\infty\\).\n\t\t\t\tThen, by essential strict convexity of \\(\\kernel\\), it follows that \\(\\bar y^k\\to y^\\star\\).\n\t\t\t\tMoreover, by the subgradient inequality, for every \\(k \\in \\N\\) it holds that\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t0\n\t\t\t\t\\leq\n\t\t\t\t\t\\gamma_{k+1}(\\varphi(\\bar y^k) - \\min\\varphi)\n\t\t\t\t={} &\n\t\t\t\t\t\\gamma_{k+1}(f(\\bar y^k) + g(\\bar y^k) - \\min\\varphi)\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\gamma_{k+1}(f(\\bar y^k) - f(y^\\star))\n\t\t\t\t\\\\\n\t\t\t\t&\n\t\t\t\t\t-\\innprod{\\nabla \\kernel(y^k) - \\gamma_{k+1}\\nabla f(y^k) - \\nabla \\kernel(\\bar y^k)}{y^\\star - \\bar y^k}.\n\t\t\t\t\\end{align*}\n\t\t\t\tThe proof then follows from continuity of \\(f\\) and the fact that the inner product vanishes, since both \\(y^k\\) and \\(\\bar y^k\\) converge to \\(y^\\star\\).\n\t\t\t\\end{proof}","sha256_utf8":"489cfd3f4c53d3d85d2474560dd0175a0d3dff751f8acb495f6ead06fb2823d7"}]},{"id":"S081","semantic_type":"proof_step","kind":"proof_step","title":"近端算子及坚固非扩张性","source_units":[{"start":85461,"end":86371,"start_line":2127,"end_line":2141,"raw_tex":"\t\t\t\tConsider the ``left'' Bregman proximal operator \\(\\prox_{g}:\\interior\\dom\\kernel\\to\\dom\\kernel\\) defined as\n\t\t\t\t\\[\n\t\t\t\t\t\\prox_{g}(y)\n\t\t\t\t\\coloneqq\n\t\t\t\t\t\\argmin_{w\\in\\R^n}\\{g(w)+\\D_{\\kernel}(w,y)\\}\n\t\t\t\t\\]\n\t\t\t\t(note that \\(\\prox_{g}(y)\\in\\interior\\dom\\kernel\\subseteq\\dom\\kernel\\) for any \\(y\\in\\interior\\dom\\kernel\\), owing to \\cref{ass:basic}).\n\t\t\t\tThe mapping \\(\\prox_{\\gamma_{k+1}g}\\circ\\nabla\\kernel^*\\) is \\(\\nabla\\kernel\\)-firmly-nonexpansive, in the sense that\n\t\t\t\t\\begin{equation}\\label{eq:BPG:FNE}\n\t\t\t\t\t\\DD_{\\kernel}(\\bar y_1,\\bar y_2)\n\t\t\t\t\\leq\n\t\t\t\t\t\\innprod{\\eta_1-\\eta_2}{\\bar y_1-\\bar y_2}\n\t\t\t\t\\end{equation}\n\t\t\t\tholds for any \\(\\eta_i\\in\\R^2\\) and \\(\\bar y_i = \\prox_{\\gamma_{k+1} g}\\circ\\nabla\\kernel^*(\\eta_i)\\), \\(i=1,2\\);\n\t\t\t\ta proof of this fact can be found in \\cite[Lem. 4.2]{wang2022bregman} (see also \\cite[Thm. 4.9]{wang2025bregman} for its \\emph{equivalence} to convexity of \\(g\\)).\n","sha256_utf8":"02ceb74bdc2ec64c1bbbaa78214aea890f142871aecf8b088e1c938685017f89"}]},{"id":"S082","semantic_type":"proof_step","kind":"proof_step","title":"固定点比较得到迭代极限","source_units":[{"start":86371,"end":87290,"start_line":2142,"end_line":2164,"raw_tex":"\t\t\t\tNote that \\eqref{eq:baryk} can equivalently be written as\n\t\t\t\t\\[\n\t\t\t\t\t\\bar y^k\n\t\t\t\t=\n\t\t\t\t\t\\prox_{\\gamma_{k+1} g}\\circ\\nabla\\kernel^*(H_{k+1}(y^k)),\n\t\t\t\t\\]\n\t\t\t\twhere we remind that \\(H_{k+1}=\\nabla\\kernel-\\gamma_{k+1}\\nabla f\\).\n\t\t\t\tUsing \\(\\nabla\\kernel\\)-firm nonexpansiveness and recalling that \\(y^\\star = \\prox_{\\gamma_{k+1} g}\\circ\\nabla\\kernel^*(H_{k+1}(y^\\star))\\) we have\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t\\DD_{\\kernel}(\\bar y^k,y^\\star)\n\t\t\t\t\\stackrel{\\text{\\clap{\\eqref{eq:BPG:FNE}}}}{\\leq}{} &\n\t\t\t\t\t\\innprod{H_{k+1}(y^k) - H_{k+1}(y^\\star)}{\\bar y^k - y^\\star}\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\Bigl(\n\t\t\t\t\t\t\\norm{\\nabla \\kernel(y^k) - \\nabla \\kernel(y^\\star)}\n\t\t\t\t\t\t+\n\t\t\t\t\t\t\\gamma_{k+1}\\norm{\\nabla f(y^k) - \\nabla f(y^\\star)}\n\t\t\t\t\t\\Bigr)\n\t\t\t\t\t\\norm{\\bar y^k - y^\\star},\n\t\t\t\t\\end{align*}\n\t\t\t\twhich vanishes as \\(k\\to\\infty\\).\n\t\t\t\tThen, by essential strict convexity of \\(\\kernel\\), it follows that \\(\\bar y^k\\to y^\\star\\).\n","sha256_utf8":"04e424d7504af050bb57dd3cb515a3afe3cae6cbb43750381b96dc306b6d2745"}]},{"id":"S083","semantic_type":"proof_step","kind":"proof_step","title":"次梯度界控制加权目标差","source_units":[{"start":87290,"end":87887,"start_line":2165,"end_line":2178,"raw_tex":"\t\t\t\tMoreover, by the subgradient inequality, for every \\(k \\in \\N\\) it holds that\n\t\t\t\t\\begin{align*}\n\t\t\t\t\t0\n\t\t\t\t\\leq\n\t\t\t\t\t\\gamma_{k+1}(\\varphi(\\bar y^k) - \\min\\varphi)\n\t\t\t\t={} &\n\t\t\t\t\t\\gamma_{k+1}(f(\\bar y^k) + g(\\bar y^k) - \\min\\varphi)\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\gamma_{k+1}(f(\\bar y^k) - f(y^\\star))\n\t\t\t\t\\\\\n\t\t\t\t&\n\t\t\t\t\t-\\innprod{\\nabla \\kernel(y^k) - \\gamma_{k+1}\\nabla f(y^k) - \\nabla \\kernel(\\bar y^k)}{y^\\star - \\bar y^k}.\n\t\t\t\t\\end{align*}\n\t\t\t\tThe proof then follows from continuity of \\(f\\) and the fact that the inner product vanishes, since both \\(y^k\\) and \\(\\bar y^k\\) converge to \\(y^\\star\\).\n","sha256_utf8":"6937bdb2a63e237eeff4b3e01d47c4fe0a755caf9207533f2b5faf231bd072dc"}]},{"id":"S084","semantic_type":"theorem","kind":"theorem","title":"内部下界点蕴含全序列收敛","source_units":[{"start":88954,"end":89507,"start_line":2193,"end_line":2202,"raw_tex":"\\begin{theorem}\\label{thm:xinC}%\n\t\t\t\tAdditionally to \\cref{ass:basic}, suppose that\n\t\t\t\t\\begin{equation}\\label{eq:xinC}\n\t\t\t\t\t\\text{there exists \\(x\\in C\\) such that \\(\\varphi(x)\\leq\\inf_{k\\in\\N}\\varphi(x^k)\\)}\n\t\t\t\t\\end{equation}\n\t\t\t\tholds for the iterates \\(x^k\\) generated by \\refBY.\n\t\t\t\tThen, the sequence \\((x^k)_{k\\in\\N}\\) converges to a solution \\(x^\\star\\in\\argmin_C\\varphi\\neq\\emptyset\\) and \\(\\varphi(x^k)\\to\\inf_{\\overline C}\\varphi\\).\n\n\t\t\t\tWhen \\(\\kernel\\) has symmetry coefficient \\(\\alpha>0\\), the same is true for \\refFNEa.\n\t\t\t\\end{theorem}","sha256_utf8":"dd1e936fe39fc2d02797c56e4b537b999c8e598957a6e9a41d0fadb41409c77e"}]},{"id":"S085","semantic_type":"assumption","kind":"assumption","title":"存在内部目标下界点","source_units":[{"start":88987,"end":89239,"start_line":2194,"end_line":2198,"raw_tex":"\t\t\t\tAdditionally to \\cref{ass:basic}, suppose that\n\t\t\t\t\\begin{equation}\\label{eq:xinC}\n\t\t\t\t\t\\text{there exists \\(x\\in C\\) such that \\(\\varphi(x)\\leq\\inf_{k\\in\\N}\\varphi(x^k)\\)}\n\t\t\t\t\\end{equation}\n\t\t\t\tholds for the iterates \\(x^k\\) generated by \\refBY.\n","sha256_utf8":"348970d36804b3570660f659ccc60149d15f5d3d34b2b059ddb36355e91a0dd0"}]},{"id":"S086","semantic_type":"proof","kind":"proof","title":"内部情形完整收敛证明","source_units":[{"start":89747,"end":90039,"start_line":2205,"end_line":2206,"raw_tex":"\t\t\tThe proof of \\cref{thm:xinC} will be carried out via intermediate claims.\n\t\t\tIn what follows, we consider \\(\\U_k\\) as in \\eqref{eq:UkBY} in case of \\refBY{}; under the needed symmetry assumption, the same proof applies to \\refFNEa{} as well by simply replacing \\(\\U_k\\gets\\U^{\\alpha}_k\\).\n","sha256_utf8":"706ee4e92af82a435bc25476be7fec2afac077cf61bdbf387c11316764ad54ac"},{"start":90186,"end":90446,"start_line":2211,"end_line":2222,"raw_tex":"\\begin{proof}\n\t\t\t\tIn view of \\eqref{eq:xinC}, \\cref{thm:SD} implies that\n\t\t\t\t\\[\n\t\t\t\t\t\\D_{\\kernel}(x,x^k)\n\t\t\t\t\\leq\n\t\t\t\t\t\\U_k(x)\n\t\t\t\t\\leq\n\t\t\t\t\t\\U_0(x)\n\t\t\t\t\\]\n\t\t\t\tholds for any \\(k\\in\\N\\).\n\t\t\t\tThe assertion then follows from \\cref{fact:D:coercive}.\n\t\t\t\\end{proof}","sha256_utf8":"545119792f6143a81f6c4fb394d88287158eb41d137a0e7a9f44e742f3f0bcee"},{"start":90509,"end":91454,"start_line":2227,"end_line":2237,"raw_tex":"\\begin{proof}\n\t\t\t\tTo arrive to a contradiction, suppose that \\(\\gamma_k\\to0\\).\n\t\t\t\tThen, \\cref{thm:Lamkto1} implies that eventually the quantity\n\t\t\t\t\\(\n\t\t\t\t\t[\\Lambda_{k,\\delta}-(1-\\gamma_k\\ell_k)]_+\n\t\t\t\t\\)\n\t\t\t\tappearing in both \\eqref{eq:BY:rhok*} and \\eqref{eq:FNEa:rhok*} vanishes.\n\t\t\t\tSince \\((x^k)_{k\\in\\N}\\) is bounded and bounded away from the boundary of \\(C\\), regardless of whether or not \\(\\kernel\\) has a symmetry coefficient \\(\\alpha>0\\), it holds that \\(\\inf_{k\\in\\N}\\alpha_k>0\\);\n\t\t\t\tfurthermore, for both \\refBY{} and \\refFNEa{} note that \\(\\hat\\rho_{k+1}\\leq\\sqrt{1+\\rho_k}\\leq\\sqrt{1+\\hat\\rho_k}\\), implying that \\(\\hat\\rho_k\\leq\\frac{1+\\sqrt{5}}{2}\\) holds for any \\(k\\).\n\t\t\t\tAs such, in both \\eqref{eq:BY:rhok*} and \\eqref{eq:FNEa:rhok*} eventually the second element in the minimum is infinite, implying that the update reduces to \\(\\gamma_{k+1}=\\hat\\rho_{k+1}\\gamma_k\\) and is thus divergent, a contradiction.\n\t\t\t\\end{proof}","sha256_utf8":"4241f59f3d23c3da07b74a483129cecab38665fde950a530d58bcff53d3e6adf"},{"start":91647,"end":92153,"start_line":2242,"end_line":2247,"raw_tex":"\\begin{proof}\n\t\t\t\tHaving shown that \\(\\gamma_k\\not\\to0\\), one has that \\(\\sum_{k\\in\\N}\\gamma_k=\\infty\\).\n\t\t\t\t\\Cref{thm:Pkmin} then implies that \\(\\inf_{k\\in\\N}\\varphi(x^k)=\\varphi(x)\\).\n\t\t\t\tFrom the arbitrariness of the point \\(x\\) as in \\eqref{eq:xinC} we conclude that necessarily \\(x\\in\\argmin_C\\varphi\\subseteq\\argmin_{\\overline C}\\varphi\\), with inclusion holding by virtue of \\cite[Prop. 11.1(iv)]{bauschke2017convex}.\n\t\t\t\tHence \\(\\inf_{k\\in\\N}\\varphi(x^k)=\\inf_{\\overline C}\\varphi\\).\n\t\t\t\\end{proof}","sha256_utf8":"3d9ba31e772de9c7d997dda4129ad30cefd8d17687471cd65e0716d2ae1faaed"},{"start":92292,"end":93341,"start_line":2252,"end_line":2272,"raw_tex":"\\begin{proof}\n\t\t\t\tThe existence of an optimal limit point is guaranteed by the previous claim, since \\((x^k)_{k\\in\\N}\\) is bounded (and bounded away from the boundary of \\(C\\)) and \\(\\varphi\\) is lsc.\n\t\t\t\tConsider two optimal limit points \\(\\bar x_1\\) and \\(\\bar x_2\\), and two corresponding subsequences \\((x^k)_{k\\in K_1}\\to\\bar x_1\\) and \\((x^k)_{k\\in K_2}\\to\\bar x_2\\).\n\t\t\t\tNotice that\n\t\t\t\t\\[\n\t\t\t\t\t\\U_k(\\bar x_1)-\\U_k(\\bar x_2)\n\t\t\t\t=\n\t\t\t\t\t\\D_{\\kernel}(\\bar x_1,x^k)-\\D_{\\kernel}(\\bar x_2,x^k)\n\t\t\t\t\\]\n\t\t\t\tconverges to some finite value \\(V\\), because both \\(\\U_k(\\bar x_1)\\) and \\(\\U_k(\\bar x_2)\\) are convergent.\n\t\t\t\tConsidering the limits along \\(k\\in K_1\\) and \\(k\\in K_2\\) yields that\n\t\t\t\t\\[\n\t\t\t\t\tV\n\t\t\t\t=\n\t\t\t\t\t\\D_{\\kernel}(\\bar x_1,\\bar x_1)-\\D_{\\kernel}(\\bar x_2,\\bar x_1)\n\t\t\t\t=\n\t\t\t\t\t\\D_{\\kernel}(\\bar x_1,\\bar x_2)-\\D_{\\kernel}(\\bar x_2,\\bar x_2),\n\t\t\t\t\\]\n\t\t\t\thence that \\(-\\D_{\\kernel}(\\bar x_2,\\bar x_1)=\\D_{\\kernel}(\\bar x_1,\\bar x_2)\\).\n\t\t\t\tThus \\(\\DD_{\\kernel}(\\bar x_1,\\bar x_2)=0\\), implying that \\(\\bar x_1=\\bar x_2\\).\n\t\t\t\\end{proof}","sha256_utf8":"b90c97ee5c77737e8154e46c5725e5b74472fee6a64706e394a540ceb4f04487"},{"start":93343,"end":99972,"start_line":2274,"end_line":2419,"raw_tex":"\t\t\tIn light of the previous claim, the proof of \\cref{thm:xinC} is completed once we show that \\(x^k\\to x^\\star\\).\n\t\t\tTo this end, owing to the fact that \\(\\D_{\\kernel}(x^\\star,x^k)\\leq\\U_k(x^\\star)\\) it will suffice to show that \\(U\\coloneqq\\lim_{k\\to\\infty}\\U_k(x^\\star)\\) is zero, (existence and finiteness of \\(U\\) is ensured by \\cref{thm:SD}).\n\t\t\t\\begin{itemize}\n\t\t\t\\item\n\t\t\t\tLet us consider a subsequence \\((x^k)_{k\\in K}\\) converging to \\(x^\\star\\).\n\t\t\t\tIf the corresponding subsequence \\((\\gamma_{k+1})_{k\\in K}\\) is bounded, then it follows from \\cref{lem:prox:property:optimal} that  \\(x^{k+1}\\to x^\\star\\) and \\(\\gamma_{k+1} P_k(x^\\star) \\to 0\\) as well, and from the expression of \\(\\U_k\\) together with the fact that \\(\\hat\\rho_{k+1}\\) is bounded and \\(\\inf_k \\alpha_k>0\\), it is clear that\n\t\t\t\t\\(\\U_{k+1}(x^\\star)\\to0\\) as \\(K\\ni k\\to\\infty\\).\n\n\t\t\t\\item\n\t\t\t\tIn what follows, let us instead consider the complementary case in which for any subsequence \\(x^k\\) converging to \\(x^\\star\\) the corresponding subsequence of stepsizes \\((\\gamma_{k+1})_{k\\in K}\\) is divergent.\n\t\t\t\tIn this case, note that since \\(\\gamma_{k+1}P_k(x^\\star)\\leq\\U_{k+1}(x^\\star)\\leq\\U_0(x^\\star)\\), we have\n\t\t\t\t\\begin{align}\n\t\t\t\t\\nonumber\n\t\t\t\t\t(x^k)_{k\\in K}\\to x^\\star\n\t\t\t\t\\Leftrightarrow{} &\n\t\t\t\t\t(\\gamma_{k+1})_{k\\in K}\\to\\infty\n\t\t\t\t\\Leftrightarrow\n\t\t\t\t\t\\bigl(P_k(x^\\star)\\bigr)_{k\\in K}\\to0\n\t\t\t\t\\\\\n\t\t\t\t\\nonumber\n\t\t\t\t\\Rightarrow{} &\n\t\t\t\t\t(\\gamma_k)_{k\\in K}\\to\\infty\n\t\t\t\t\\\\\n\t\t\t\t\\Leftrightarrow{} &\n\t\t\t\t\t(x^{k-1})_{k\\in K}\\to x^\\star\n\t\t\t\t\\Leftrightarrow\n\t\t\t\t\t\\bigl(P_{k-1}(x^\\star)\\bigr)_{k\\in K}\\to0,\n\t\t\t\t\\label{eq:gamk:infty\"gamk-1}\n\t\t\t\t\\end{align}\n\t\t\t\twhere the right implication follows from the fact that \\(\\gamma_{k+1}\\leq\\hat\\rho_{k+1}\\gamma_k\\leq\\rho_{\\rm max}\\gamma_k\\), for some \\(\\rho_{\\rm max}\\leq\\frac{1+\\sqrt{5}}{2}\\).\n\t\t\t\tSince an optimal limit exists, it follows that \\(\\sup_{k\\in\\N}\\gamma_k=\\infty\\).\n\n\t\t\t\tIn what follows, we construct a specific subsequence \\(K\\coloneqq\\{k_0,k_1,\\dots\\}\\) along which \\((\\gamma_k)_{k\\in K}\\) diverges.\n\t\t\t\tIn doing so, we expand upon the arguments in the proof of \\cite[Thm. 2.3]{latafat2024adaptive} to account for the complications of the non-Euclidean setting investigated here.\n\t\t\t\tFor \\(i\\geq0\\) let\n\t\t\t\t\\begin{equation}\\label{eq:ki+}\n\t\t\t\t\tk_{i+1}=\\min\\{k\\geq k_i \\mid \\gamma_k \\geq \\rho_{\\rm max}\\gamma_{k_i}\\}.\n\t\t\t\t\\end{equation}\n\t\t\t\tThen, \\(\\bigl(\\gamma_{k_i}\\bigr)_{i\\in\\N}\\to\\infty\\), which by \\eqref{eq:gamk:infty\"gamk-1} implies that\n\t\t\t\t\\begin{equation}\\label{eq:xkitox*}\n\t\t\t\t\t\\lim_{i\\to\\infty}x^{k_i-s}\n\t\t\t\t=\n\t\t\t\t\tx^\\star\n\t\t\t\t\\quad\n\t\t\t\t\t\\forall s=1,2,3\n\t\t\t\t\\end{equation}\n\t\t\t\t(in fact, for any \\(s\\in\\N\\)).\n\t\t\t\tIn light of \\cref{thm:SD} we have that\n\t\t\t\t\\begin{equation}\\label{eq:U:gamkPk-1}\n\t\t\t\t\tU= \\lim_{k \\to \\infty} \\U_k= \\lim_{i \\to \\infty} \\U_{k_i-1}(x^\\star) = \\lim_{i \\to \\infty} \\gamma_{k_i-1}(1+ \\vartheta_{k_i-1})P_{k_i-2}(x^\\star).\n\t\t\t\t\\end{equation}\n\n\t\t\t\tWe proceed to show that \\(\\gamma_{k-1} P_{k-2}(x^\\star)\\) converges to zero along the same subsequence.\n\t\t\t\t\\begin{subequations}\n\t\t\t\t\tFor every \\(i\\in\\N\\), note that\n\t\t\t\t\t\\begin{gather}\n\t\t\t\t\t\t\\rho_{k_i}>1\n\t\t\t\t\t\t~\\text{ and }~\n\t\t\t\t\t\tk_{i-1}\\leq k_i-1\n\t\t\t\t\t\\shortintertext{by minimality of \\(k_i\\) and the fact that \\(\\rho_{\\rm max}>1\\), hence that}\n\t\t\t\t\t\\label{eq:k_i-1}\n\t\t\t\t\t\t\\text{either }~\n\t\t\t\t\t\t\\rho_{k_i-1}>1\n\t\t\t\t\t\t~\\text{ or }~\n\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\tk_i-1\\notin K\n\t\t\t\t\t\t\t\\text{ and thus }\n\t\t\t\t\t\t\tk_{i-1}\\leq k_i-2\n\t\t\t\t\t\t\\bigr).\n\t\t\t\t\t\\shortintertext{These combined imply also that}\n\t\t\t\t\t\\label{eq:k-1}\n\t\t\t\t\t\t\\rho_{k_i-1}\\geq\\rho_{\\rm max}^{-1}.\n\t\t\t\t\t\\end{gather}\n\t\t\t\t\tIndeed, if not, then \\(\\rho_{k_i-1}<\\rho_{\\rm max}^{-1}<1\\), implying by \\eqref{eq:k_i-1} that \\(k_{i-1}\\leq k_i-2\\);\n\t\t\t\t\tthis would lead to the contradiction\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\rho_{\\rm max}\\gamma_{k_{i-1}}\n\t\t\t\t\t\\stackrel{\\text{\\clap{\\eqref{eq:ki+}}}}{\\leq}\n\t\t\t\t\t\t\\gamma_{k_i}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\rho_{\\rm max}\\gamma_{k_i-1}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\rho_{\\rm max}\\rho_{k_i-1}\\gamma_{k_i-2}\n\t\t\t\t\t\\stackrel{\\mathclap{\\bot}}{<}\n\t\t\t\t\t\t\\gamma_{k_i-2}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\rho_{\\rm max}\\gamma_{k_{i-1}},\n\t\t\t\t\t\\]\n\t\t\t\t\twhere ``\\(\\bot\\)'' marks where the contradictory inequality is used, and the last inequality follows both in case \\(k_i-2=k_{i-1}\\) (since \\(\\rho_{\\rm max}>1\\)) or when \\(k_{i-1}<k_i-2<k_i\\) (from minimality in the definition of \\(k_i\\)).\n\t\t\t\t\\end{subequations}\n\t\t\t\tTherefore, \\eqref{eq:k-1} holds true, implying in particular that the sequence\n\t\t\t\t\\begin{equation}\\label{eq:Lambda:k-2:bounded}\n\t\t\t\t\t\\bigl(\\Lambda_{k_i-2, 2\\hat\\rho_{k_i-1}}\\bigr)_{i\\in\\N}\n\t\t\t\t\\quad\n\t\t\t\t\t\\text{is bounded,}\n\t\t\t\t\\end{equation}\n\t\t\t\tfor otherwise the second term in either \\eqref{eq:BY:rhok*} or \\eqref{eq:FNEa:rhok*} would vanish, owing to the fact that \\(\\hat\\rho_k\\) is lower bounded by \\(\\sqrt{1/2}\\) and similarly \\(\\alpha_k\\) is bounded away from zero.\n\n\t\t\t\tLet \\(v^k\\coloneqq H_k(x^{k-1})- H_k(x^k)\\in\\gamma_k\\partial\\varphi(x^k)\\), and observe that\n\t\t\t\t\\begin{align*}\n\t\t\t\t&\n\t\t\t\t\t\\gamma_k P_{k-1}(x^\\star)\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\rho_k\\innprod{x^\\star - x^{k-1}}{-v^{k-1}}\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{\\rho_k}{2\\hat\\rho_k}\\D_{\\kernel}(x^\\star, x^{k-1})\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\rho_k}{2\\hat\\rho_k}\n\t\t\t\t\t\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(x^{k-1})-2\\hat\\rho_kv^{k-1}\\bigr), \\nabla \\kernel(x^{k-1})\\bigr)\n\t\t\t\t\\\\\n\t\t\t\t={} &\n\t\t\t\t\t\\tfrac{\\rho_k}{2\\hat\\rho_k}\\D_{\\kernel}(x^\\star, x^{k-1})\n\t\t\t\t\t+\n\t\t\t\t\t2\\rho_k \\hat\\rho_k^2\\Lambda_{k-1, 2\\hat\\rho_k} \\DD_{\\kernel}(x^{k-1},x^{k-2})\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{1}{2}\\D_{\\kernel}(x^\\star, x^{k-1})\n\t\t\t\t\t+\n\t\t\t\t\t2\\rho_{\\rm max}^3 \\Lambda_{k-1, 2\\hat\\rho_k} \\DD_{\\kernel}(x^{k-1},x^{k-2})\n\t\t\t\t\\end{align*}\n\t\t\t\tholds for any \\(k\\), where we used the Bregman--Young inequality \\eqref{eq:BY} with parameter \\(\\delta= \\hat\\rho_k\\) in the second inequality, and \\(\\rho_k\\leq \\hat\\rho_k\\leq \\rho_{\\rm max}\\) in the last one.\n\t\t\t\tRecall that \\(\\bigl(x^{k_i-3}\\bigr)_{i\\in\\N} \\to x^\\star\\) by \\eqref{eq:xkitox*}.\n\t\t\t\tThus,\n\t\t\t\t\\[\n\t\t\t\t\t\\gamma_{k_i-1} P_{k_i-2}(x^\\star)\n\t\t\t\t\\leq\n\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t\\underbrace{\n\t\t\t\t\t\t\\D_{\\kernel}(x^\\star, x^{k_i-2})\n\t\t\t\t\t\t\\vphantom{\\Lambda_{k_i-2, 2\\rho_{k_i-1}}}\n\t\t\t\t\t}_{\\to 0 \\text{ by \\eqref{eq:xkitox*}}}\n\t\t\t\t\t+\n\t\t\t\t\t2\\rho_{\\rm max}^3\n\t\t\t\t\t\\underbrace{\n\t\t\t\t\t\t\\Lambda_{k_i-2, 2\\rho_{k_i-1}}\n\t\t\t\t\t}_{\\text{\\rm bounded}}\n\t\t\t\t\t\\,\n\t\t\t\t\t\\underbrace{\n\t\t\t\t\t\t\\DD_{\\kernel}(x^{k_i-2},x^{k_i-3})\n\t\t\t\t\t\t\\vphantom{\\Lambda_{k_i-2, 2\\rho_{k_i-1}}}\n\t\t\t\t\t}_{\\to 0 \\text{ by \\eqref{eq:xkitox*}}}.\n\t\t\t\t\\]\n\t\t\t\tUsing this in \\eqref{eq:U:gamkPk-1} and noting that \\(\\theta_{k_i-1}\\) being a multiple of \\(\\hat{\\rho}_{k_i-1}\\) in both \\refBY{} and \\refFNEa{} is bounded, completes the proof.\n\t\t\t\\end{itemize}\n","sha256_utf8":"5ac113cffd92a46dcf4621f11f713ee18684a7819e85457d4b47fbc13f4957ce"}]},{"id":"S087","semantic_type":"proof_step","kind":"proof_step","title":"声明分步证明及变体替换","source_units":[{"start":89747,"end":90039,"start_line":2205,"end_line":2206,"raw_tex":"\t\t\tThe proof of \\cref{thm:xinC} will be carried out via intermediate claims.\n\t\t\tIn what follows, we consider \\(\\U_k\\) as in \\eqref{eq:UkBY} in case of \\refBY{}; under the needed symmetry assumption, the same proof applies to \\refFNEa{} as well by simply replacing \\(\\U_k\\gets\\U^{\\alpha}_k\\).\n","sha256_utf8":"706ee4e92af82a435bc25476be7fec2afac077cf61bdbf387c11316764ad54ac"}]},{"id":"S088","semantic_type":"proposition","kind":"proposition","title":"迭代位于内部紧集","source_units":[{"start":90043,"end":90182,"start_line":2208,"end_line":2210,"raw_tex":"\\begin{claim}\\label{thm:xinC:boundedseq}\n\t\t\t\tThere exists a compact set \\(\\K\\subset C\\) containing all the iterates \\(x^k\\).\n\t\t\t\\end{claim}","sha256_utf8":"4111c27e8715dc17976d42104418a5addd33cb1552a6b8450eecb7454bdda21f"}]},{"id":"S089","semantic_type":"proof","kind":"proof","title":"内部紧集性质证明","source_units":[{"start":90186,"end":90446,"start_line":2211,"end_line":2222,"raw_tex":"\\begin{proof}\n\t\t\t\tIn view of \\eqref{eq:xinC}, \\cref{thm:SD} implies that\n\t\t\t\t\\[\n\t\t\t\t\t\\D_{\\kernel}(x,x^k)\n\t\t\t\t\\leq\n\t\t\t\t\t\\U_k(x)\n\t\t\t\t\\leq\n\t\t\t\t\t\\U_0(x)\n\t\t\t\t\\]\n\t\t\t\tholds for any \\(k\\in\\N\\).\n\t\t\t\tThe assertion then follows from \\cref{fact:D:coercive}.\n\t\t\t\\end{proof}","sha256_utf8":"545119792f6143a81f6c4fb394d88287158eb41d137a0e7a9f44e742f3f0bcee"}]},{"id":"S090","semantic_type":"proof_step","kind":"proof_step","title":"势函数有界与距离强制性","source_units":[{"start":90200,"end":90432,"start_line":2212,"end_line":2221,"raw_tex":"\t\t\t\tIn view of \\eqref{eq:xinC}, \\cref{thm:SD} implies that\n\t\t\t\t\\[\n\t\t\t\t\t\\D_{\\kernel}(x,x^k)\n\t\t\t\t\\leq\n\t\t\t\t\t\\U_k(x)\n\t\t\t\t\\leq\n\t\t\t\t\t\\U_0(x)\n\t\t\t\t\\]\n\t\t\t\tholds for any \\(k\\in\\N\\).\n\t\t\t\tThe assertion then follows from \\cref{fact:D:coercive}.\n","sha256_utf8":"7e03707c78f3baea52dfa1ebd4997b226224e8c1be659198dc4d1bb90c13f5c6"}]},{"id":"S091","semantic_type":"proposition","kind":"proposition","title":"步长不趋于零","source_units":[{"start":90451,"end":90505,"start_line":2224,"end_line":2226,"raw_tex":"\\begin{claim}\n\t\t\t\t\\(\\gamma_k\\not\\to0\\).\n\t\t\t\\end{claim}","sha256_utf8":"ab722e4a594378ef824942d5760cd44b279e86a8b059497ac1e11ab2cda50053"}]},{"id":"S092","semantic_type":"proof","kind":"proof","title":"步长不消失证明","source_units":[{"start":90509,"end":91454,"start_line":2227,"end_line":2237,"raw_tex":"\\begin{proof}\n\t\t\t\tTo arrive to a contradiction, suppose that \\(\\gamma_k\\to0\\).\n\t\t\t\tThen, \\cref{thm:Lamkto1} implies that eventually the quantity\n\t\t\t\t\\(\n\t\t\t\t\t[\\Lambda_{k,\\delta}-(1-\\gamma_k\\ell_k)]_+\n\t\t\t\t\\)\n\t\t\t\tappearing in both \\eqref{eq:BY:rhok*} and \\eqref{eq:FNEa:rhok*} vanishes.\n\t\t\t\tSince \\((x^k)_{k\\in\\N}\\) is bounded and bounded away from the boundary of \\(C\\), regardless of whether or not \\(\\kernel\\) has a symmetry coefficient \\(\\alpha>0\\), it holds that \\(\\inf_{k\\in\\N}\\alpha_k>0\\);\n\t\t\t\tfurthermore, for both \\refBY{} and \\refFNEa{} note that \\(\\hat\\rho_{k+1}\\leq\\sqrt{1+\\rho_k}\\leq\\sqrt{1+\\hat\\rho_k}\\), implying that \\(\\hat\\rho_k\\leq\\frac{1+\\sqrt{5}}{2}\\) holds for any \\(k\\).\n\t\t\t\tAs such, in both \\eqref{eq:BY:rhok*} and \\eqref{eq:FNEa:rhok*} eventually the second element in the minimum is infinite, implying that the update reduces to \\(\\gamma_{k+1}=\\hat\\rho_{k+1}\\gamma_k\\) and is thus divergent, a contradiction.\n\t\t\t\\end{proof}","sha256_utf8":"4241f59f3d23c3da07b74a483129cecab38665fde950a530d58bcff53d3e6adf"}]},{"id":"S093","semantic_type":"proof_step","kind":"proof_step","title":"假设消失并应用曲率极限","source_units":[{"start":90523,"end":90793,"start_line":2228,"end_line":2233,"raw_tex":"\t\t\t\tTo arrive to a contradiction, suppose that \\(\\gamma_k\\to0\\).\n\t\t\t\tThen, \\cref{thm:Lamkto1} implies that eventually the quantity\n\t\t\t\t\\(\n\t\t\t\t\t[\\Lambda_{k,\\delta}-(1-\\gamma_k\\ell_k)]_+\n\t\t\t\t\\)\n\t\t\t\tappearing in both \\eqref{eq:BY:rhok*} and \\eqref{eq:FNEa:rhok*} vanishes.\n","sha256_utf8":"743cdd617f3b55312cb54e867a90d8740d7d4987f4967bcf423db5fd5d428143"}]},{"id":"S094","semantic_type":"proof_step","kind":"proof_step","title":"增长步长规则导出矛盾","source_units":[{"start":90793,"end":91440,"start_line":2234,"end_line":2236,"raw_tex":"\t\t\t\tSince \\((x^k)_{k\\in\\N}\\) is bounded and bounded away from the boundary of \\(C\\), regardless of whether or not \\(\\kernel\\) has a symmetry coefficient \\(\\alpha>0\\), it holds that \\(\\inf_{k\\in\\N}\\alpha_k>0\\);\n\t\t\t\tfurthermore, for both \\refBY{} and \\refFNEa{} note that \\(\\hat\\rho_{k+1}\\leq\\sqrt{1+\\rho_k}\\leq\\sqrt{1+\\hat\\rho_k}\\), implying that \\(\\hat\\rho_k\\leq\\frac{1+\\sqrt{5}}{2}\\) holds for any \\(k\\).\n\t\t\t\tAs such, in both \\eqref{eq:BY:rhok*} and \\eqref{eq:FNEa:rhok*} eventually the second element in the minimum is infinite, implying that the update reduces to \\(\\gamma_{k+1}=\\hat\\rho_{k+1}\\gamma_k\\) and is thus divergent, a contradiction.\n","sha256_utf8":"5068dfe44bd9ad5612088c1e2ad5ce5904b4d3d25ea76468a966b3cc42f9d27e"}]},{"id":"S095","semantic_type":"proposition","kind":"proposition","title":"内部情形的最优下确界","source_units":[{"start":91459,"end":91643,"start_line":2239,"end_line":2241,"raw_tex":"\\begin{claim}\n\t\t\t\t\\(\\inf_{k\\in\\N}\\varphi(x^k)=\\inf_{\\overline C}\\varphi\\) (in particular, necessarily \\(x\\in\\argmin_C\\varphi\\) for any \\(x\\) satisfying \\eqref{eq:xinC}).\n\t\t\t\\end{claim}","sha256_utf8":"88613ace57d3d63016be59ec1e6b3010d6436d9056ccdc0d1e20437efd332094"}]},{"id":"S096","semantic_type":"proof","kind":"proof","title":"最优下确界证明","source_units":[{"start":91647,"end":92153,"start_line":2242,"end_line":2247,"raw_tex":"\\begin{proof}\n\t\t\t\tHaving shown that \\(\\gamma_k\\not\\to0\\), one has that \\(\\sum_{k\\in\\N}\\gamma_k=\\infty\\).\n\t\t\t\t\\Cref{thm:Pkmin} then implies that \\(\\inf_{k\\in\\N}\\varphi(x^k)=\\varphi(x)\\).\n\t\t\t\tFrom the arbitrariness of the point \\(x\\) as in \\eqref{eq:xinC} we conclude that necessarily \\(x\\in\\argmin_C\\varphi\\subseteq\\argmin_{\\overline C}\\varphi\\), with inclusion holding by virtue of \\cite[Prop. 11.1(iv)]{bauschke2017convex}.\n\t\t\t\tHence \\(\\inf_{k\\in\\N}\\varphi(x^k)=\\inf_{\\overline C}\\varphi\\).\n\t\t\t\\end{proof}","sha256_utf8":"3d9ba31e772de9c7d997dda4129ad30cefd8d17687471cd65e0716d2ae1faaed"}]},{"id":"S097","semantic_type":"proof_step","kind":"proof_step","title":"步长和发散消去目标差","source_units":[{"start":91661,"end":92139,"start_line":2243,"end_line":2246,"raw_tex":"\t\t\t\tHaving shown that \\(\\gamma_k\\not\\to0\\), one has that \\(\\sum_{k\\in\\N}\\gamma_k=\\infty\\).\n\t\t\t\t\\Cref{thm:Pkmin} then implies that \\(\\inf_{k\\in\\N}\\varphi(x^k)=\\varphi(x)\\).\n\t\t\t\tFrom the arbitrariness of the point \\(x\\) as in \\eqref{eq:xinC} we conclude that necessarily \\(x\\in\\argmin_C\\varphi\\subseteq\\argmin_{\\overline C}\\varphi\\), with inclusion holding by virtue of \\cite[Prop. 11.1(iv)]{bauschke2017convex}.\n\t\t\t\tHence \\(\\inf_{k\\in\\N}\\varphi(x^k)=\\inf_{\\overline C}\\varphi\\).\n","sha256_utf8":"cb3479ab5da072012c1caf350e27b1ba738ce4c3984a7ba6369fb135c668dc59"}]},{"id":"S098","semantic_type":"proposition","kind":"proposition","title":"内部最优聚点唯一","source_units":[{"start":92158,"end":92288,"start_line":2249,"end_line":2251,"raw_tex":"\\begin{claim}\\label{claim:unique}%\n\t\t\t\tThere exists exactly one optimal limit point \\(x^\\star\\in\\argmin_C\\varphi\\).\n\t\t\t\\end{claim}","sha256_utf8":"a8e76f83f828859b7f7a7216e8206c0214fd689eb11ae9134171760c11d97349"}]},{"id":"S099","semantic_type":"proof","kind":"proof","title":"内部最优聚点唯一性证明","source_units":[{"start":92292,"end":93341,"start_line":2252,"end_line":2272,"raw_tex":"\\begin{proof}\n\t\t\t\tThe existence of an optimal limit point is guaranteed by the previous claim, since \\((x^k)_{k\\in\\N}\\) is bounded (and bounded away from the boundary of \\(C\\)) and \\(\\varphi\\) is lsc.\n\t\t\t\tConsider two optimal limit points \\(\\bar x_1\\) and \\(\\bar x_2\\), and two corresponding subsequences \\((x^k)_{k\\in K_1}\\to\\bar x_1\\) and \\((x^k)_{k\\in K_2}\\to\\bar x_2\\).\n\t\t\t\tNotice that\n\t\t\t\t\\[\n\t\t\t\t\t\\U_k(\\bar x_1)-\\U_k(\\bar x_2)\n\t\t\t\t=\n\t\t\t\t\t\\D_{\\kernel}(\\bar x_1,x^k)-\\D_{\\kernel}(\\bar x_2,x^k)\n\t\t\t\t\\]\n\t\t\t\tconverges to some finite value \\(V\\), because both \\(\\U_k(\\bar x_1)\\) and \\(\\U_k(\\bar x_2)\\) are convergent.\n\t\t\t\tConsidering the limits along \\(k\\in K_1\\) and \\(k\\in K_2\\) yields that\n\t\t\t\t\\[\n\t\t\t\t\tV\n\t\t\t\t=\n\t\t\t\t\t\\D_{\\kernel}(\\bar x_1,\\bar x_1)-\\D_{\\kernel}(\\bar x_2,\\bar x_1)\n\t\t\t\t=\n\t\t\t\t\t\\D_{\\kernel}(\\bar x_1,\\bar x_2)-\\D_{\\kernel}(\\bar x_2,\\bar x_2),\n\t\t\t\t\\]\n\t\t\t\thence that \\(-\\D_{\\kernel}(\\bar x_2,\\bar x_1)=\\D_{\\kernel}(\\bar x_1,\\bar x_2)\\).\n\t\t\t\tThus \\(\\DD_{\\kernel}(\\bar x_1,\\bar x_2)=0\\), implying that \\(\\bar x_1=\\bar x_2\\).\n\t\t\t\\end{proof}","sha256_utf8":"b90c97ee5c77737e8154e46c5725e5b74472fee6a64706e394a540ceb4f04487"}]},{"id":"S100","semantic_type":"proof_step","kind":"proof_step","title":"聚点存在及距离差极限","source_units":[{"start":92306,"end":92909,"start_line":2253,"end_line":2261,"raw_tex":"\t\t\t\tThe existence of an optimal limit point is guaranteed by the previous claim, since \\((x^k)_{k\\in\\N}\\) is bounded (and bounded away from the boundary of \\(C\\)) and \\(\\varphi\\) is lsc.\n\t\t\t\tConsider two optimal limit points \\(\\bar x_1\\) and \\(\\bar x_2\\), and two corresponding subsequences \\((x^k)_{k\\in K_1}\\to\\bar x_1\\) and \\((x^k)_{k\\in K_2}\\to\\bar x_2\\).\n\t\t\t\tNotice that\n\t\t\t\t\\[\n\t\t\t\t\t\\U_k(\\bar x_1)-\\U_k(\\bar x_2)\n\t\t\t\t=\n\t\t\t\t\t\\D_{\\kernel}(\\bar x_1,x^k)-\\D_{\\kernel}(\\bar x_2,x^k)\n\t\t\t\t\\]\n\t\t\t\tconverges to some finite value \\(V\\), because both \\(\\U_k(\\bar x_1)\\) and \\(\\U_k(\\bar x_2)\\) are convergent.\n","sha256_utf8":"4d283b1453ae2c8d4de2cdd06be205243902115db4358e9d530f91254651ec14"}]},{"id":"S101","semantic_type":"proof_step","kind":"proof_step","title":"两个子列极限迫使聚点相同","source_units":[{"start":92909,"end":93327,"start_line":2262,"end_line":2271,"raw_tex":"\t\t\t\tConsidering the limits along \\(k\\in K_1\\) and \\(k\\in K_2\\) yields that\n\t\t\t\t\\[\n\t\t\t\t\tV\n\t\t\t\t=\n\t\t\t\t\t\\D_{\\kernel}(\\bar x_1,\\bar x_1)-\\D_{\\kernel}(\\bar x_2,\\bar x_1)\n\t\t\t\t=\n\t\t\t\t\t\\D_{\\kernel}(\\bar x_1,\\bar x_2)-\\D_{\\kernel}(\\bar x_2,\\bar x_2),\n\t\t\t\t\\]\n\t\t\t\thence that \\(-\\D_{\\kernel}(\\bar x_2,\\bar x_1)=\\D_{\\kernel}(\\bar x_1,\\bar x_2)\\).\n\t\t\t\tThus \\(\\DD_{\\kernel}(\\bar x_1,\\bar x_2)=0\\), implying that \\(\\bar x_1=\\bar x_2\\).\n","sha256_utf8":"0f42654bdf0548e073b7d195cb7d710f33201cd0a288e02f94d61192fd15659e"}]},{"id":"S102","semantic_type":"proof_step","kind":"proof_step","title":"有界步长子列令势函数归零","source_units":[{"start":93343,"end":94211,"start_line":2274,"end_line":2282,"raw_tex":"\t\t\tIn light of the previous claim, the proof of \\cref{thm:xinC} is completed once we show that \\(x^k\\to x^\\star\\).\n\t\t\tTo this end, owing to the fact that \\(\\D_{\\kernel}(x^\\star,x^k)\\leq\\U_k(x^\\star)\\) it will suffice to show that \\(U\\coloneqq\\lim_{k\\to\\infty}\\U_k(x^\\star)\\) is zero, (existence and finiteness of \\(U\\) is ensured by \\cref{thm:SD}).\n\t\t\t\\begin{itemize}\n\t\t\t\\item\n\t\t\t\tLet us consider a subsequence \\((x^k)_{k\\in K}\\) converging to \\(x^\\star\\).\n\t\t\t\tIf the corresponding subsequence \\((\\gamma_{k+1})_{k\\in K}\\) is bounded, then it follows from \\cref{lem:prox:property:optimal} that  \\(x^{k+1}\\to x^\\star\\) and \\(\\gamma_{k+1} P_k(x^\\star) \\to 0\\) as well, and from the expression of \\(\\U_k\\) together with the fact that \\(\\hat\\rho_{k+1}\\) is bounded and \\(\\inf_k \\alpha_k>0\\), it is clear that\n\t\t\t\t\\(\\U_{k+1}(x^\\star)\\to0\\) as \\(K\\ni k\\to\\infty\\).\n\n\t\t\t\\item\n","sha256_utf8":"335208a267c32548a6cc604390a96d479585cfde98d816fafdb2205a6c5d8a92"}]},{"id":"S103","semantic_type":"proof_step","kind":"proof_step","title":"发散步长与最优子列等价","source_units":[{"start":94211,"end":95250,"start_line":2283,"end_line":2304,"raw_tex":"\t\t\t\tIn what follows, let us instead consider the complementary case in which for any subsequence \\(x^k\\) converging to \\(x^\\star\\) the corresponding subsequence of stepsizes \\((\\gamma_{k+1})_{k\\in K}\\) is divergent.\n\t\t\t\tIn this case, note that since \\(\\gamma_{k+1}P_k(x^\\star)\\leq\\U_{k+1}(x^\\star)\\leq\\U_0(x^\\star)\\), we have\n\t\t\t\t\\begin{align}\n\t\t\t\t\\nonumber\n\t\t\t\t\t(x^k)_{k\\in K}\\to x^\\star\n\t\t\t\t\\Leftrightarrow{} &\n\t\t\t\t\t(\\gamma_{k+1})_{k\\in K}\\to\\infty\n\t\t\t\t\\Leftrightarrow\n\t\t\t\t\t\\bigl(P_k(x^\\star)\\bigr)_{k\\in K}\\to0\n\t\t\t\t\\\\\n\t\t\t\t\\nonumber\n\t\t\t\t\\Rightarrow{} &\n\t\t\t\t\t(\\gamma_k)_{k\\in K}\\to\\infty\n\t\t\t\t\\\\\n\t\t\t\t\\Leftrightarrow{} &\n\t\t\t\t\t(x^{k-1})_{k\\in K}\\to x^\\star\n\t\t\t\t\\Leftrightarrow\n\t\t\t\t\t\\bigl(P_{k-1}(x^\\star)\\bigr)_{k\\in K}\\to0,\n\t\t\t\t\\label{eq:gamk:infty\"gamk-1}\n\t\t\t\t\\end{align}\n\t\t\t\twhere the right implication follows from the fact that \\(\\gamma_{k+1}\\leq\\hat\\rho_{k+1}\\gamma_k\\leq\\rho_{\\rm max}\\gamma_k\\), for some \\(\\rho_{\\rm max}\\leq\\frac{1+\\sqrt{5}}{2}\\).\n\t\t\t\tSince an optimal limit exists, it follows that \\(\\sup_{k\\in\\N}\\gamma_k=\\infty\\).\n","sha256_utf8":"6675fa0c70c6da468935b18f420b3dbf91553c59f6a4399253fce2e82e18ff50"}]},{"id":"S104","semantic_type":"proof_step","kind":"proof_step","title":"构造增长子列并表示势极限","source_units":[{"start":95251,"end":96261,"start_line":2306,"end_line":2324,"raw_tex":"\t\t\t\tIn what follows, we construct a specific subsequence \\(K\\coloneqq\\{k_0,k_1,\\dots\\}\\) along which \\((\\gamma_k)_{k\\in K}\\) diverges.\n\t\t\t\tIn doing so, we expand upon the arguments in the proof of \\cite[Thm. 2.3]{latafat2024adaptive} to account for the complications of the non-Euclidean setting investigated here.\n\t\t\t\tFor \\(i\\geq0\\) let\n\t\t\t\t\\begin{equation}\\label{eq:ki+}\n\t\t\t\t\tk_{i+1}=\\min\\{k\\geq k_i \\mid \\gamma_k \\geq \\rho_{\\rm max}\\gamma_{k_i}\\}.\n\t\t\t\t\\end{equation}\n\t\t\t\tThen, \\(\\bigl(\\gamma_{k_i}\\bigr)_{i\\in\\N}\\to\\infty\\), which by \\eqref{eq:gamk:infty\"gamk-1} implies that\n\t\t\t\t\\begin{equation}\\label{eq:xkitox*}\n\t\t\t\t\t\\lim_{i\\to\\infty}x^{k_i-s}\n\t\t\t\t=\n\t\t\t\t\tx^\\star\n\t\t\t\t\\quad\n\t\t\t\t\t\\forall s=1,2,3\n\t\t\t\t\\end{equation}\n\t\t\t\t(in fact, for any \\(s\\in\\N\\)).\n\t\t\t\tIn light of \\cref{thm:SD} we have that\n\t\t\t\t\\begin{equation}\\label{eq:U:gamkPk-1}\n\t\t\t\t\tU= \\lim_{k \\to \\infty} \\U_k= \\lim_{i \\to \\infty} \\U_{k_i-1}(x^\\star) = \\lim_{i \\to \\infty} \\gamma_{k_i-1}(1+ \\vartheta_{k_i-1})P_{k_i-2}(x^\\star).\n\t\t\t\t\\end{equation}\n","sha256_utf8":"76c128106a2949c68fed90317665d1bc48da96f1f0ccf4bdb8a236dc18de9fc4"}]},{"id":"S105","semantic_type":"proof_step","kind":"proof_step","title":"提取增长子列的步长比性质","source_units":[{"start":96262,"end":96931,"start_line":2326,"end_line":2346,"raw_tex":"\t\t\t\tWe proceed to show that \\(\\gamma_{k-1} P_{k-2}(x^\\star)\\) converges to zero along the same subsequence.\n\t\t\t\t\\begin{subequations}\n\t\t\t\t\tFor every \\(i\\in\\N\\), note that\n\t\t\t\t\t\\begin{gather}\n\t\t\t\t\t\t\\rho_{k_i}>1\n\t\t\t\t\t\t~\\text{ and }~\n\t\t\t\t\t\tk_{i-1}\\leq k_i-1\n\t\t\t\t\t\\shortintertext{by minimality of \\(k_i\\) and the fact that \\(\\rho_{\\rm max}>1\\), hence that}\n\t\t\t\t\t\\label{eq:k_i-1}\n\t\t\t\t\t\t\\text{either }~\n\t\t\t\t\t\t\\rho_{k_i-1}>1\n\t\t\t\t\t\t~\\text{ or }~\n\t\t\t\t\t\t\\bigl(\n\t\t\t\t\t\t\tk_i-1\\notin K\n\t\t\t\t\t\t\t\\text{ and thus }\n\t\t\t\t\t\t\tk_{i-1}\\leq k_i-2\n\t\t\t\t\t\t\\bigr).\n\t\t\t\t\t\\shortintertext{These combined imply also that}\n\t\t\t\t\t\\label{eq:k-1}\n\t\t\t\t\t\t\\rho_{k_i-1}\\geq\\rho_{\\rm max}^{-1}.\n\t\t\t\t\t\\end{gather}\n","sha256_utf8":"f216108f973713ac2021bcc87799a566352711bd4758f8396b57a87603b78ca9"}]},{"id":"S106","semantic_type":"proof_step","kind":"proof_step","title":"最小索引反证步长比下界","source_units":[{"start":96931,"end":97689,"start_line":2347,"end_line":2363,"raw_tex":"\t\t\t\t\tIndeed, if not, then \\(\\rho_{k_i-1}<\\rho_{\\rm max}^{-1}<1\\), implying by \\eqref{eq:k_i-1} that \\(k_{i-1}\\leq k_i-2\\);\n\t\t\t\t\tthis would lead to the contradiction\n\t\t\t\t\t\\[\n\t\t\t\t\t\t\\rho_{\\rm max}\\gamma_{k_{i-1}}\n\t\t\t\t\t\\stackrel{\\text{\\clap{\\eqref{eq:ki+}}}}{\\leq}\n\t\t\t\t\t\t\\gamma_{k_i}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\rho_{\\rm max}\\gamma_{k_i-1}\n\t\t\t\t\t=\n\t\t\t\t\t\t\\rho_{\\rm max}\\rho_{k_i-1}\\gamma_{k_i-2}\n\t\t\t\t\t\\stackrel{\\mathclap{\\bot}}{<}\n\t\t\t\t\t\t\\gamma_{k_i-2}\n\t\t\t\t\t\\leq\n\t\t\t\t\t\t\\rho_{\\rm max}\\gamma_{k_{i-1}},\n\t\t\t\t\t\\]\n\t\t\t\t\twhere ``\\(\\bot\\)'' marks where the contradictory inequality is used, and the last inequality follows both in case \\(k_i-2=k_{i-1}\\) (since \\(\\rho_{\\rm max}>1\\)) or when \\(k_{i-1}<k_i-2<k_i\\) (from minimality in the definition of \\(k_i\\)).\n\t\t\t\t\\end{subequations}\n","sha256_utf8":"2ecd9b3e5cad45ea46410f27b0678eb32230e94863a6ffc4621f64edc2e1194a"}]},{"id":"S107","semantic_type":"proof_step","kind":"proof_step","title":"更新规则给出曲率子列有界性","source_units":[{"start":97689,"end":98166,"start_line":2364,"end_line":2370,"raw_tex":"\t\t\t\tTherefore, \\eqref{eq:k-1} holds true, implying in particular that the sequence\n\t\t\t\t\\begin{equation}\\label{eq:Lambda:k-2:bounded}\n\t\t\t\t\t\\bigl(\\Lambda_{k_i-2, 2\\hat\\rho_{k_i-1}}\\bigr)_{i\\in\\N}\n\t\t\t\t\\quad\n\t\t\t\t\t\\text{is bounded,}\n\t\t\t\t\\end{equation}\n\t\t\t\tfor otherwise the second term in either \\eqref{eq:BY:rhok*} or \\eqref{eq:FNEa:rhok*} would vanish, owing to the fact that \\(\\hat\\rho_k\\) is lower bounded by \\(\\sqrt{1/2}\\) and similarly \\(\\alpha_k\\) is bounded away from zero.\n","sha256_utf8":"f2b432150ca680732dfb2650db6209d46eb29fa06c36f16a260eb4abe919761e"}]},{"id":"S108","semantic_type":"proof_step","kind":"proof_step","title":"杨氏界控制加权目标差","source_units":[{"start":98167,"end":99258,"start_line":2372,"end_line":2397,"raw_tex":"\t\t\t\tLet \\(v^k\\coloneqq H_k(x^{k-1})- H_k(x^k)\\in\\gamma_k\\partial\\varphi(x^k)\\), and observe that\n\t\t\t\t\\begin{align*}\n\t\t\t\t&\n\t\t\t\t\t\\gamma_k P_{k-1}(x^\\star)\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\rho_k\\innprod{x^\\star - x^{k-1}}{-v^{k-1}}\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{\\rho_k}{2\\hat\\rho_k}\\D_{\\kernel}(x^\\star, x^{k-1})\n\t\t\t\t\t+\n\t\t\t\t\t\\tfrac{\\rho_k}{2\\hat\\rho_k}\n\t\t\t\t\t\\D_{\\kernel^*}\\bigl(\\nabla\\kernel(x^{k-1})-2\\hat\\rho_kv^{k-1}\\bigr), \\nabla \\kernel(x^{k-1})\\bigr)\n\t\t\t\t\\\\\n\t\t\t\t={} &\n\t\t\t\t\t\\tfrac{\\rho_k}{2\\hat\\rho_k}\\D_{\\kernel}(x^\\star, x^{k-1})\n\t\t\t\t\t+\n\t\t\t\t\t2\\rho_k \\hat\\rho_k^2\\Lambda_{k-1, 2\\hat\\rho_k} \\DD_{\\kernel}(x^{k-1},x^{k-2})\n\t\t\t\t\\\\\n\t\t\t\t\\leq{} &\n\t\t\t\t\t\\tfrac{1}{2}\\D_{\\kernel}(x^\\star, x^{k-1})\n\t\t\t\t\t+\n\t\t\t\t\t2\\rho_{\\rm max}^3 \\Lambda_{k-1, 2\\hat\\rho_k} \\DD_{\\kernel}(x^{k-1},x^{k-2})\n\t\t\t\t\\end{align*}\n\t\t\t\tholds for any \\(k\\), where we used the Bregman--Young inequality \\eqref{eq:BY} with parameter \\(\\delta= \\hat\\rho_k\\) in the second inequality, and \\(\\rho_k\\leq \\hat\\rho_k\\leq \\rho_{\\rm max}\\) in the last one.\n\t\t\t\tRecall that \\(\\bigl(x^{k_i-3}\\bigr)_{i\\in\\N} \\to x^\\star\\) by \\eqref{eq:xkitox*}.\n","sha256_utf8":"576befe20891adac2dbd5cf632252c689222aa56b06ed7b3ad1f96ea082628c9"}]},{"id":"S109","semantic_type":"proof_step","kind":"proof_step","title":"沿增长子列消去势函数极限","source_units":[{"start":99258,"end":99972,"start_line":2398,"end_line":2419,"raw_tex":"\t\t\t\tThus,\n\t\t\t\t\\[\n\t\t\t\t\t\\gamma_{k_i-1} P_{k_i-2}(x^\\star)\n\t\t\t\t\\leq\n\t\t\t\t\t\\tfrac{1}{2}\n\t\t\t\t\t\\underbrace{\n\t\t\t\t\t\t\\D_{\\kernel}(x^\\star, x^{k_i-2})\n\t\t\t\t\t\t\\vphantom{\\Lambda_{k_i-2, 2\\rho_{k_i-1}}}\n\t\t\t\t\t}_{\\to 0 \\text{ by \\eqref{eq:xkitox*}}}\n\t\t\t\t\t+\n\t\t\t\t\t2\\rho_{\\rm max}^3\n\t\t\t\t\t\\underbrace{\n\t\t\t\t\t\t\\Lambda_{k_i-2, 2\\rho_{k_i-1}}\n\t\t\t\t\t}_{\\text{\\rm bounded}}\n\t\t\t\t\t\\,\n\t\t\t\t\t\\underbrace{\n\t\t\t\t\t\t\\DD_{\\kernel}(x^{k_i-2},x^{k_i-3})\n\t\t\t\t\t\t\\vphantom{\\Lambda_{k_i-2, 2\\rho_{k_i-1}}}\n\t\t\t\t\t}_{\\to 0 \\text{ by \\eqref{eq:xkitox*}}}.\n\t\t\t\t\\]\n\t\t\t\tUsing this in \\eqref{eq:U:gamkPk-1} and noting that \\(\\theta_{k_i-1}\\) being a multiple of \\(\\hat{\\rho}_{k_i-1}\\) in both \\refBY{} and \\refFNEa{} is bounded, completes the proof.\n\t\t\t\\end{itemize}\n","sha256_utf8":"d625bbeeb705d4c535a7f7e152a08ac67c4e2658e89d331fd59d2ed4addea9ad"}]},{"id":"S110","semantic_type":"external_fact","kind":"external_fact","title":"标准三点公式","source_units":[],"declared_external":true,"declaration_text":["对适当可微凸核，D_phi(x,z)=D_phi(x,y)+D_phi(y,z)+<x-y,grad phi(y)-grad phi(z)>；适用域条件如B0013。"]},{"id":"S111","semantic_type":"external_fact","kind":"external_fact","title":"标准Legendre对偶与边界事实","source_units":[],"declared_external":true,"declaration_text":["采用B0015逐项列出的已引用标准结论：1-强制Legendre核的共轭满定义域、反向距离强制性、边界发散、闭域下距离消失导致同极限、共轭梯度互逆及原始与对偶Bregman距离恒等式。"]},{"id":"S112","semantic_type":"external_fact","kind":"external_fact","title":"凸域线段与端点连续性","source_units":[],"declared_external":true,"declaration_text":["凸集内点与集合中一点之间的开线段仍在内点集中；适当闭凸函数沿从定义域内点到定义域中一点的线段趋近端点时，函数值趋于该端点函数值。"]},{"id":"S113","semantic_type":"external_fact","kind":"external_fact","title":"相对光滑的梯度增量界","source_units":[],"declared_external":true,"declaration_text":["采用作者在L688–L696及L2079–L2083援引的局部相对光滑标准性质：f在C上可微且D_f受局部常数乘D_phi控制；在该证明的记号下，||v^k||<=L_{f,K}^{phi} L_{phi,K} ||u^k||。"]},{"id":"S114","semantic_type":"external_fact","kind":"external_fact","title":"最终小步长BPG收敛","source_units":[],"declared_external":true,"declaration_text":["采用作者L2102明确调用的标准收敛事实：在B0115的凸性、内部紧集及局部相对光滑条件下，若gamma_k趋于零并最终小于1/L_{f,K}^{phi}，则BPG迭代收敛于K中的一点。"]},{"id":"S115","semantic_type":"external_fact","kind":"external_fact","title":"凸Bregman近端坚固非扩张","source_units":[],"declared_external":true,"declaration_text":["若bar y_i=prox_{gamma g}^{phi}(grad phi^*(eta_i))且g凸，则Delta_phi(bar y_1,bar y_2)<=<eta_1-eta_2,bar y_1-bar y_2>，在作者L2134–L2141所列条件下成立。"]},{"id":"S116","semantic_type":"external_fact","kind":"external_fact","title":"凸目标的内部逼近","source_units":[],"declared_external":true,"declaration_text":["在本文的凸域与目标定义域相交条件下，inf_C varphi=inf_closure(C) varphi；C内的最小点也是闭包上的最小点。"]},{"id":"S117","semantic_type":"external_fact","kind":"external_fact","title":"正对称系数蕴含开定义域","source_units":[],"declared_external":true,"declaration_text":["Legendre核若具有严格正的全局对称系数，则其定义域是开集；再结合区域假设的定义域闭性及非空性，定义域必为R^n。"]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S018","edges":[{"source_id":"S001","target_id":"S007","kind":"supports"},{"source_id":"S002","target_id":"S015","kind":"supports"},{"source_id":"S003","target_id":"S006","kind":"supports"},{"source_id":"S003","target_id":"S039","kind":"supports"},{"source_id":"S004","target_id":"S010","kind":"supports"},{"source_id":"S004","target_id":"S015","kind":"supports"},{"source_id":"S005","target_id":"S007","kind":"supports"},{"source_id":"S005","target_id":"S082","kind":"supports"},{"source_id":"S006","target_id":"S030","kind":"supports"},{"source_id":"S006","target_id":"S045","kind":"supports"},{"source_id":"S006","target_id":"S101","kind":"supports"},{"source_id":"S007","target_id":"S030","kind":"supports"},{"source_id":"S007","target_id":"S075","kind":"supports"},{"source_id":"S007","target_id":"S076","kind":"supports"},{"source_id":"S007","target_id":"S078","kind":"supports"},{"source_id":"S007","target_id":"S081","kind":"supports"},{"source_id":"S007","target_id":"S094","kind":"supports"},{"source_id":"S008","target_id":"S076","kind":"supports"},{"source_id":"S009","target_id":"S024","kind":"supports"},{"source_id":"S009","target_id":"S026","kind":"supports"},{"source_id":"S010","target_id":"S030","kind":"supports"},{"source_id":"S010","target_id":"S035","kind":"supports"},{"source_id":"S010","target_id":"S042","kind":"supports"},{"source_id":"S010","target_id":"S049","kind":"supports"},{"source_id":"S010","target_id":"S055","kind":"supports"},{"source_id":"S010","target_id":"S074","kind":"supports"},{"source_id":"S010","target_id":"S094","kind":"supports"},{"source_id":"S010","target_id":"S107","kind":"supports"},{"source_id":"S010","target_id":"S108","kind":"supports"},{"source_id":"S011","target_id":"S059","kind":"supports"},{"source_id":"S011","target_id":"S103","kind":"supports"},{"source_id":"S011","target_id":"S105","kind":"supports"},{"source_id":"S012","target_id":"S059","kind":"supports"},{"source_id":"S012","target_id":"S094","kind":"supports"},{"source_id":"S012","target_id":"S107","kind":"supports"},{"source_id":"S013","target_id":"S063","kind":"supports"},{"source_id":"S013","target_id":"S064","kind":"supports"},{"source_id":"S013","target_id":"S094","kind":"supports"},{"source_id":"S013","target_id":"S107","kind":"supports"},{"source_id":"S014","target_id":"S094","kind":"supports"},{"source_id":"S015","target_id":"S030","kind":"supports"},{"source_id":"S015","target_id":"S031","kind":"supports"},{"source_id":"S015","target_id":"S032","kind":"supports"},{"source_id":"S015","target_id":"S045","kind":"supports"},{"source_id":"S015","target_id":"S046","kind":"supports"},{"source_id":"S015","target_id":"S083","kind":"supports"},{"source_id":"S015","target_id":"S108","kind":"supports"},{"source_id":"S016","target_id":"S031","kind":"supports"},{"source_id":"S016","target_id":"S039","kind":"supports"},{"source_id":"S017","target_id":"S038","kind":"supports"},{"source_id":"S017","target_id":"S039","kind":"supports"},{"source_id":"S017","target_id":"S075","kind":"supports"},{"source_id":"S017","target_id":"S090","kind":"supports"},{"source_id":"S017","target_id":"S026","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S020","target_id":"S023","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S019","kind":"supports"},{"source_id":"S022","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S019","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S019","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S019","kind":"supports"},{"source_id":"S026","target_id":"S019","kind":"supports"},{"source_id":"S027","target_id":"S052","kind":"supports"},{"source_id":"S027","target_id":"S062","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S034","kind":"supports"},{"source_id":"S030","target_id":"S028","kind":"supports"},{"source_id":"S031","target_id":"S033","kind":"supports"},{"source_id":"S031","target_id":"S028","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S028","kind":"supports"},{"source_id":"S033","target_id":"S034","kind":"supports"},{"source_id":"S033","target_id":"S028","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S028","kind":"supports"},{"source_id":"S035","target_id":"S028","kind":"supports"},{"source_id":"S036","target_id":"S042","kind":"supports"},{"source_id":"S036","target_id":"S108","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S037","kind":"supports"},{"source_id":"S039","target_id":"S040","kind":"supports"},{"source_id":"S039","target_id":"S037","kind":"supports"},{"source_id":"S040","target_id":"S037","kind":"supports"},{"source_id":"S041","target_id":"S048","kind":"supports"},{"source_id":"S041","target_id":"S052","kind":"supports"},{"source_id":"S042","target_id":"S041","kind":"supports"},{"source_id":"S043","target_id":"S048","kind":"supports"},{"source_id":"S044","target_id":"S043","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S046","target_id":"S044","kind":"supports"},{"source_id":"S047","target_id":"S062","kind":"supports"},{"source_id":"S048","target_id":"S050","kind":"supports"},{"source_id":"S048","target_id":"S047","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S047","kind":"supports"},{"source_id":"S050","target_id":"S047","kind":"supports"},{"source_id":"S051","target_id":"S060","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S052","target_id":"S051","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S053","target_id":"S055","kind":"supports"},{"source_id":"S053","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S056","kind":"supports"},{"source_id":"S054","target_id":"S051","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S059","kind":"supports"},{"source_id":"S055","target_id":"S051","kind":"supports"},{"source_id":"S056","target_id":"S059","kind":"supports"},{"source_id":"S056","target_id":"S051","kind":"supports"},{"source_id":"S057","target_id":"S069","kind":"supports"},{"source_id":"S058","target_id":"S057","kind":"supports"},{"source_id":"S059","target_id":"S060","kind":"supports"},{"source_id":"S059","target_id":"S058","kind":"supports"},{"source_id":"S060","target_id":"S058","kind":"supports"},{"source_id":"S061","target_id":"S065","kind":"supports"},{"source_id":"S062","target_id":"S064","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S054","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S063","target_id":"S061","kind":"supports"},{"source_id":"S064","target_id":"S061","kind":"supports"},{"source_id":"S065","target_id":"S069","kind":"supports"},{"source_id":"S066","target_id":"S090","kind":"supports"},{"source_id":"S066","target_id":"S097","kind":"supports"},{"source_id":"S066","target_id":"S100","kind":"supports"},{"source_id":"S066","target_id":"S102","kind":"supports"},{"source_id":"S066","target_id":"S104","kind":"supports"},{"source_id":"S066","target_id":"S024","kind":"supports"},{"source_id":"S066","target_id":"S025","kind":"supports"},{"source_id":"S067","target_id":"S066","kind":"supports"},{"source_id":"S068","target_id":"S067","kind":"supports"},{"source_id":"S069","target_id":"S070","kind":"supports"},{"source_id":"S069","target_id":"S067","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S067","kind":"supports"},{"source_id":"S071","target_id":"S067","kind":"supports"},{"source_id":"S072","target_id":"S093","kind":"supports"},{"source_id":"S073","target_id":"S072","kind":"supports"},{"source_id":"S074","target_id":"S077","kind":"supports"},{"source_id":"S074","target_id":"S073","kind":"supports"},{"source_id":"S075","target_id":"S077","kind":"supports"},{"source_id":"S075","target_id":"S073","kind":"supports"},{"source_id":"S076","target_id":"S077","kind":"supports"},{"source_id":"S076","target_id":"S073","kind":"supports"},{"source_id":"S077","target_id":"S078","kind":"supports"},{"source_id":"S077","target_id":"S073","kind":"supports"},{"source_id":"S078","target_id":"S073","kind":"supports"},{"source_id":"S079","target_id":"S102","kind":"supports"},{"source_id":"S080","target_id":"S079","kind":"supports"},{"source_id":"S081","target_id":"S082","kind":"supports"},{"source_id":"S081","target_id":"S080","kind":"supports"},{"source_id":"S082","target_id":"S083","kind":"supports"},{"source_id":"S082","target_id":"S080","kind":"supports"},{"source_id":"S083","target_id":"S080","kind":"supports"},{"source_id":"S084","target_id":"S020","kind":"supports"},{"source_id":"S084","target_id":"S022","kind":"supports"},{"source_id":"S085","target_id":"S090","kind":"supports"},{"source_id":"S085","target_id":"S097","kind":"supports"},{"source_id":"S086","target_id":"S084","kind":"supports"},{"source_id":"S087","target_id":"S086","kind":"supports"},{"source_id":"S088","target_id":"S093","kind":"supports"},{"source_id":"S088","target_id":"S094","kind":"supports"},{"source_id":"S088","target_id":"S097","kind":"supports"},{"source_id":"S088","target_id":"S100","kind":"supports"},{"source_id":"S088","target_id":"S103","kind":"supports"},{"source_id":"S089","target_id":"S088","kind":"supports"},{"source_id":"S090","target_id":"S089","kind":"supports"},{"source_id":"S091","target_id":"S097","kind":"supports"},{"source_id":"S092","target_id":"S091","kind":"supports"},{"source_id":"S093","target_id":"S094","kind":"supports"},{"source_id":"S093","target_id":"S092","kind":"supports"},{"source_id":"S094","target_id":"S092","kind":"supports"},{"source_id":"S095","target_id":"S100","kind":"supports"},{"source_id":"S096","target_id":"S095","kind":"supports"},{"source_id":"S097","target_id":"S096","kind":"supports"},{"source_id":"S098","target_id":"S102","kind":"supports"},{"source_id":"S098","target_id":"S103","kind":"supports"},{"source_id":"S099","target_id":"S098","kind":"supports"},{"source_id":"S100","target_id":"S101","kind":"supports"},{"source_id":"S100","target_id":"S099","kind":"supports"},{"source_id":"S101","target_id":"S099","kind":"supports"},{"source_id":"S102","target_id":"S086","kind":"supports"},{"source_id":"S103","target_id":"S104","kind":"supports"},{"source_id":"S103","target_id":"S086","kind":"supports"},{"source_id":"S104","target_id":"S105","kind":"supports"},{"source_id":"S104","target_id":"S109","kind":"supports"},{"source_id":"S104","target_id":"S086","kind":"supports"},{"source_id":"S105","target_id":"S106","kind":"supports"},{"source_id":"S105","target_id":"S086","kind":"supports"},{"source_id":"S106","target_id":"S107","kind":"supports"},{"source_id":"S106","target_id":"S086","kind":"supports"},{"source_id":"S107","target_id":"S109","kind":"supports"},{"source_id":"S107","target_id":"S086","kind":"supports"},{"source_id":"S108","target_id":"S109","kind":"supports"},{"source_id":"S108","target_id":"S086","kind":"supports"},{"source_id":"S109","target_id":"S086","kind":"supports"},{"source_id":"S110","target_id":"S016","kind":"supports"},{"source_id":"S111","target_id":"S017","kind":"supports"},{"source_id":"S112","target_id":"S038","kind":"supports"},{"source_id":"S112","target_id":"S040","kind":"supports"},{"source_id":"S113","target_id":"S008","kind":"supports"},{"source_id":"S113","target_id":"S076","kind":"supports"},{"source_id":"S114","target_id":"S078","kind":"supports"},{"source_id":"S115","target_id":"S081","kind":"supports"},{"source_id":"S116","target_id":"S097","kind":"supports"},{"source_id":"S116","target_id":"S022","kind":"supports"},{"source_id":"S117","target_id":"S023","kind":"supports"}]}}
{"record_id":"2509.00737v3","paper":{"title":"Convergence Analysis of the ProbAbilistic Gradient Estimator Algorithm for Weakly Convex Finite-Sum Optimization","authors":["Laurent Condat","Peter Richtárik"],"arxiv_id":"2509.00737","version":"v3","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-026-03065-4","abstract_url":"https://arxiv.org/abs/2509.00737v3","pdf_url":"https://arxiv.org/pdf/2509.00737v3"},"context":{"raw_tex":"Formally, let $\\mathcal{X}$ be a finite-dimensional real Hilbert space. Given a number  $n\\geq 1$ of smooth functions $f_i:\\mathcal{X}\\rightarrow \\mathbb{R}$, $i\\in[n]\\coloneqq \\{1,\\ldots,n\\}$, \n the finite-sum optimization problem is\n\\begin{equation}\n\\min_{x\\in\\mathcal{X}} \\,f(x)\\coloneqq \\frac{1}{n}\\sum_{i=1}^n f_i(x).\\label{eqp1}\n\\end{equation}\n\nWe make the following assumptions on the functions.\n\\begin{itemize}\n\\item There exists $L>0$ such that $f_i$ is $L$-smooth for every $i\\in[n]$; that is, $f_i$ is differentiable on $\\mathcal{X}$ and for every $(x,y)\\in\\mathcal{X}^2$, $\\|\\nabla f_i(x)-\\nabla f_i(y)\\|\\leq L \\|x-y\\|$.\n\\item There exists $\\tau\\in [0,L]$ such that $f_i$ is $\\tau$-weakly convex for every $i\\in[n]$; that is, $f_i+\\frac{\\tau}{2} \\|\\cdot\\|^2$ is convex \\cite{nur73}.\n\\item $f$ is lower bounded: $f^\\star \\coloneqq \\inf f > -\\infty$. \n\\end{itemize}\n\n\nIn addition, we establish linear convergence when \n$f$ satisfies the \nPolyak--{\\L}ojasiewicz (P{\\L}) condition with constant $\\mu>0$, or in short is $\\mu$-P{\\L} \\cite{kar16}:\n \\begin{equation*}\n\\|\\nabla f(x)\\|^2\\geq 2\\mu\\big(f(x)-f^\\star\\big),\\ \\mbox{for every }x\\in\\mathcal{X}.\n\\end{equation*}\nIn this case, $\\mu\\leq L$ and we define the condition number $\\kappa\\coloneqq \\frac{L}{\\mu}\\geq 1$.  \n\n\n\\begin{algorithm}[H] \n\t\t\\caption{\\algn{PAGE}}\\label{alg1}\n\t\t\\begin{algorithmic}\n\t\t\t\\STATE  \\textbf{input:} initial estimate $x^0$, initial gradient estimate $g^0$, stepsize $\\gamma>0$, probability $p\\in (0,1]$.\n\t\t\t\\FOR{$t=0, 1, \\ldots$}\n\t\t\t\\STATE $x^{t+1} \\coloneqq x^t - \\gamma g^t$\n\t\t\t\\STATE flip a coin $\\theta^t = (1$ with probability $p$, $0$ otherwise)\n\t\t\t\\IF{$\\theta^t=0$}\n\t\t\t\\STATE pick $i^t \\in [n]$ uniformly at random\n\t\t\t\\STATE $g^{t+1} \\coloneqq g^t +\\nabla f_{i^t}(x^{t+1})-\\nabla f_{i^t}(x^{t})$\n\t\t\t\\ELSE\n\t\t\t\\STATE $g^{t+1} \\coloneqq \\nabla f(x^{t+1})$\n\t\t\t\\ENDIF \n\t\t\t\\ENDFOR\n\t\t\\end{algorithmic}\n\t\\end{algorithm}\n\n\\begin{equation}\n\\Psi^t \\coloneqq f(x^t)-f^\\star -b\\gamma \\sqnorm{\\nabla f(x^t)} + \\frac{a \\gamma}{p} \\sqnorm{g^t-\\nabla f(x^t)} + \\frac{b \\gamma}{p} \\sqnorm{g^t},\\label{eqly}\n\\end{equation}\n\nfor some constants $a>0$ and $b\\in[ 0, \\frac{1}{2\\gamma L}]$. Since $f$ is $L$-smooth, we have \n$f(x^t)-f^\\star \\geq  \\frac{1}{2L}\\sqnorm{\\nabla f(x^t)}$\n\\cite[eq. 3.5]{bub15}, \nso that the condition $b\\gamma  \\leq \\frac{1}{2L}$ guarantees that $\\Psi^t$ is a valid Lyapunov function: we have $\\Psi^t\\geq 0$, and $\\Psi^t \\rightarrow 0$ implies $\\sqnorm{g^t}\\rightarrow 0$, $\\sqnorm{\\nabla f(x^t)}\\rightarrow 0$, and\n$f(x^t)-f^\\star \\rightarrow 0$.\n","units":[{"start":2545,"end":2894,"start_line":78,"end_line":82,"raw_tex":"Formally, let $\\mathcal{X}$ be a finite-dimensional real Hilbert space. Given a number  $n\\geq 1$ of smooth functions $f_i:\\mathcal{X}\\rightarrow \\mathbb{R}$, $i\\in[n]\\coloneqq \\{1,\\ldots,n\\}$, \n the finite-sum optimization problem is\n\\begin{equation}\n\\min_{x\\in\\mathcal{X}} \\,f(x)\\coloneqq \\frac{1}{n}\\sum_{i=1}^n f_i(x).\\label{eqp1}\n\\end{equation}","sha256_utf8":"4afc86b46047d0664d30505a4b2cf6e267d2ecd1da421618441e92350d1e4eb5"},{"start":3481,"end":4006,"start_line":92,"end_line":97,"raw_tex":"We make the following assumptions on the functions.\n\\begin{itemize}\n\\item There exists $L>0$ such that $f_i$ is $L$-smooth for every $i\\in[n]$; that is, $f_i$ is differentiable on $\\mathcal{X}$ and for every $(x,y)\\in\\mathcal{X}^2$, $\\|\\nabla f_i(x)-\\nabla f_i(y)\\|\\leq L \\|x-y\\|$.\n\\item There exists $\\tau\\in [0,L]$ such that $f_i$ is $\\tau$-weakly convex for every $i\\in[n]$; that is, $f_i+\\frac{\\tau}{2} \\|\\cdot\\|^2$ is convex \\cite{nur73}.\n\\item $f$ is lower bounded: $f^\\star \\coloneqq \\inf f > -\\infty$. \n\\end{itemize}\n","sha256_utf8":"e0ba2d99ef6ce98acb89fb811d1af4cb9d78dfbc19f5cb6e94f87b2e633a7628"},{"start":4532,"end":4929,"start_line":103,"end_line":109,"raw_tex":"In addition, we establish linear convergence when \n$f$ satisfies the \nPolyak--{\\L}ojasiewicz (P{\\L}) condition with constant $\\mu>0$, or in short is $\\mu$-P{\\L} \\cite{kar16}:\n \\begin{equation*}\n\\|\\nabla f(x)\\|^2\\geq 2\\mu\\big(f(x)-f^\\star\\big),\\ \\mbox{for every }x\\in\\mathcal{X}.\n\\end{equation*}\nIn this case, $\\mu\\leq L$ and we define the condition number $\\kappa\\coloneqq \\frac{L}{\\mu}\\geq 1$.  \n","sha256_utf8":"b77b58e5accbda8e305e7d5eb731069835de910485e62b7bc65ffc22ee9f8671"},{"start":6003,"end":6628,"start_line":119,"end_line":134,"raw_tex":"\\begin{algorithm}[H] \n\t\t\\caption{\\algn{PAGE}}\\label{alg1}\n\t\t\\begin{algorithmic}\n\t\t\t\\STATE  \\textbf{input:} initial estimate $x^0$, initial gradient estimate $g^0$, stepsize $\\gamma>0$, probability $p\\in (0,1]$.\n\t\t\t\\FOR{$t=0, 1, \\ldots$}\n\t\t\t\\STATE $x^{t+1} \\coloneqq x^t - \\gamma g^t$\n\t\t\t\\STATE flip a coin $\\theta^t = (1$ with probability $p$, $0$ otherwise)\n\t\t\t\\IF{$\\theta^t=0$}\n\t\t\t\\STATE pick $i^t \\in [n]$ uniformly at random\n\t\t\t\\STATE $g^{t+1} \\coloneqq g^t +\\nabla f_{i^t}(x^{t+1})-\\nabla f_{i^t}(x^{t})$\n\t\t\t\\ELSE\n\t\t\t\\STATE $g^{t+1} \\coloneqq \\nabla f(x^{t+1})$\n\t\t\t\\ENDIF \n\t\t\t\\ENDFOR\n\t\t\\end{algorithmic}\n\t\\end{algorithm}","sha256_utf8":"de38d00aa5ce8387b9ac40f72893cf48e4a6517fce5a68781f8814f624ea82ab"},{"start":11318,"end":11508,"start_line":186,"end_line":188,"raw_tex":"\\begin{equation}\n\\Psi^t \\coloneqq f(x^t)-f^\\star -b\\gamma \\sqnorm{\\nabla f(x^t)} + \\frac{a \\gamma}{p} \\sqnorm{g^t-\\nabla f(x^t)} + \\frac{b \\gamma}{p} \\sqnorm{g^t},\\label{eqly}\n\\end{equation}","sha256_utf8":"93711fbd1100afb29279830fdff1a730b2f087e32a730b299f23458840157399"},{"start":11509,"end":11956,"start_line":189,"end_line":193,"raw_tex":"for some constants $a>0$ and $b\\in[ 0, \\frac{1}{2\\gamma L}]$. Since $f$ is $L$-smooth, we have \n$f(x^t)-f^\\star \\geq  \\frac{1}{2L}\\sqnorm{\\nabla f(x^t)}$\n\\cite[eq. 3.5]{bub15}, \nso that the condition $b\\gamma  \\leq \\frac{1}{2L}$ guarantees that $\\Psi^t$ is a valid Lyapunov function: we have $\\Psi^t\\geq 0$, and $\\Psi^t \\rightarrow 0$ implies $\\sqnorm{g^t}\\rightarrow 0$, $\\sqnorm{\\nabla f(x^t)}\\rightarrow 0$, and\n$f(x^t)-f^\\star \\rightarrow 0$.\n","sha256_utf8":"3adc4532dcced3c5cc5cc0c7f3e2199a1188735a5a1906da2daed0b519f27ba4"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}[linear convergence]\\label{theo1}\nSuppose that $f$ is $\\mu$-P{\\L} for some $\\mu>0$. \nIn \\algn{PAGE} with Lyapunov function \\eqref{eqly}, suppose that\n\\begin{equation*}\n\\gamma \\leq %\\min\\left(\n\\frac{1}{L\\left(1+\\sqrt{ \\frac{4\\tau}{L+\\tau}}\\sqrt{  \\frac{1-p}{p}}\\right)}, \n\\end{equation*}\nwith the additional requirement $\\gamma<\\frac{1}{L}$ if $\\tau=0$, \n\\begin{equation*}\na= 1-\\frac{\\gamma (L-\\tau)}{2}\\in \\left( \\frac{1}{2},1\\right], \\quad b= \\frac{\\gamma (L-\\tau)}{2}\\in \\left[0, \\frac{1}{2}\\right).\n\\end{equation*}\nThen $b\\gamma  < \\frac{1}{2L}$, and \n\\algn{PAGE}\nconverges linearly:  for every $t\\geq 0$, \n\\begin{equation*}\n\\Exp{\\Psi^{t}}\\leq \\rho^t \\Psi^0,\n\\end{equation*}\nwhere \n\\begin{equation*}\n\\rho\\coloneqq \\max\\left(1-(1-2b)\\gamma\\mu,1-p\\left(1-\\frac{1}{2a}\\right)\\right)<1.\n\\end{equation*}\nAlso, $\\Psi^t\\rightarrow 0$ \nalmost surely.\n\\end{theorem}","units":[{"start":12854,"end":13728,"start_line":210,"end_line":233,"raw_tex":"\\begin{theorem}[linear convergence]\\label{theo1}\nSuppose that $f$ is $\\mu$-P{\\L} for some $\\mu>0$. \nIn \\algn{PAGE} with Lyapunov function \\eqref{eqly}, suppose that\n\\begin{equation*}\n\\gamma \\leq %\\min\\left(\n\\frac{1}{L\\left(1+\\sqrt{ \\frac{4\\tau}{L+\\tau}}\\sqrt{  \\frac{1-p}{p}}\\right)}, \n\\end{equation*}\nwith the additional requirement $\\gamma<\\frac{1}{L}$ if $\\tau=0$, \n\\begin{equation*}\na= 1-\\frac{\\gamma (L-\\tau)}{2}\\in \\left( \\frac{1}{2},1\\right], \\quad b= \\frac{\\gamma (L-\\tau)}{2}\\in \\left[0, \\frac{1}{2}\\right).\n\\end{equation*}\nThen $b\\gamma  < \\frac{1}{2L}$, and \n\\algn{PAGE}\nconverges linearly:  for every $t\\geq 0$, \n\\begin{equation*}\n\\Exp{\\Psi^{t}}\\leq \\rho^t \\Psi^0,\n\\end{equation*}\nwhere \n\\begin{equation*}\n\\rho\\coloneqq \\max\\left(1-(1-2b)\\gamma\\mu,1-p\\left(1-\\frac{1}{2a}\\right)\\right)<1.\n\\end{equation*}\nAlso, $\\Psi^t\\rightarrow 0$ \nalmost surely.\n\\end{theorem}","sha256_utf8":"ed48dc7cf4a123fa90acf78fe15d846d32730907dc59055fdfd46d9bda5abe54"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\\label{lemm2}\nLet $g:\\mathcal{X}\\rightarrow \\mathbb{R}$ be an $L$-smooth and $\\tau$-weakly convex function for some $L>0$ and $\\tau\\in [0,L]$. Then, for every $(x,y)\\in\\mathcal{X}^2$, \n\\begin{align*}\n\\sqnorm{\\nabla g(x)- \\nabla g(y)}&\\leq (L-\\tau)\\langle \\nabla g(x) - \\nabla g(y),x-y\\rangle + L\\tau \\sqnorm{x-y}.\n\\end{align*}\n\\end{lemma}\n\n\\subsection{Proof of Lemma~\\ref{lemm2}}\n\nLet $(x,y)\\in\\mathcal{X}^2$. Since $g+\\frac{\\tau}{2}\\|\\cdot\\|^2$ is convex and $(L+\\tau)$-smooth, its gradient is $\\frac{1}{L+\\tau}$-cocoercive according to the Baillon--Haddad theorem \\cite[Corollary 18.17]{bau17}, so that\n\\begin{align*}\n\\sqnorm{\\nabla g(x)+\\tau x - \\nabla g(y)-\\tau y} &\\leq (L+\\tau)\\langle \\nabla g(x) +\\tau x- \\nabla g(y)-\\tau y,x-y\\rangle\\\\\n&=(L+\\tau)\\langle \\nabla g(x) - \\nabla g(y),x-y\\rangle +  (L+\\tau)\\tau \\sqnorm{x-y}.\n\\end{align*}\nHence,\n\\begin{align*}\n\\sqnorm{\\nabla g(x)- \\nabla g(y)}&=\\sqnorm{\\nabla g(x)+\\tau x- \\nabla g(y)-\\tau y}\\!-\\!2\\tau\n\\langle \\nabla g(x) - \\nabla g(y),x-y\\rangle \n\\!-\\!\\tau^2\\sqnorm{x-y}\\\\\n&\\leq (L-\\tau)\\langle \\nabla g(x) - \\nabla g(y),x-y\\rangle + L\\tau \\sqnorm{x-y}.\n\\end{align*}\\qed\n\n\n\n\\subsection{Proof of Theorem~\\ref{theo1}}\\label{secproof1}\n\nFor every $t\\geq 0$, let $\\mathcal{F}^t$ denote the $\\sigma$-algebra generated by the random variables $x^0,g^0,\\ldots, x^t,g^t$. For the moment, assume that $a$ and $b$ in \\eqref{eqly} satisfy  $a>0$, \n$0\\leq b\\leq \\frac{1}{2}$, $b\\gamma  \\leq \\frac{1}{2L}$, without additional restrictions.\n\n\nLet $t\\geq 0$. We now derive upper bounds for the terms in $\\Exp{\\Psi^{t+1}\\;|\\;\\mathcal{F}^t}$.\n\n\\noindent \\textbf{First}, we have the descent property \\cite[Lemma 4]{ric21}\n\\begin{align*}\nf(x^{t+1}) -f^\\star &\\leq f(x^t)  -f^\\star -\\frac{\\gamma}{2} \\sqnorm{\\nabla f(x^t)} +\\frac{ \\gamma }{2}\\sqnorm{g^{t}-\\nabla f(x^t)}+ \\left(\\frac{ L}{2}-\\frac{1}{2\\gamma}\\right)\\sqnorm{x^{t+1}-x^t}\\\\\n&=f(x^t)  -f^\\star -\\frac{\\gamma}{2} \\sqnorm{\\nabla f(x^t)} +\\frac{ \\gamma }{2}\\sqnorm{g^{t}-\\nabla f(x^t)}+ \\left(\\frac{ L\\gamma^2}{2}-\\frac{\\gamma}{2}\\right)\\sqnorm{g^t}.\n\\end{align*}\n\\textbf{Second}, we have \n\\begin{align*}\n\\Exp{\\sqnorm{g^{t+1}-\\nabla f(x^{t+1})}\\;|\\;\\mathcal{F}^t} &= (1-p)\\Exp{\\sqnorm{g^{t+1}-\\nabla f(x^{t+1})}\\;|\\;\\theta^t=0,\\mathcal{F}^t},\n\\end{align*}\nand by decomposing \nthe expected squared norm into its squared mean and variance, we obtain\n\\begin{align*}\n&\\Exp{\\sqnorm{g^{t+1}-\\nabla f(x^{t+1})}\\;|\\;\\mathcal{F}^t} = (1-p)\\sqnorm{g^{t}-\\nabla f(x^t)}\\\\\n&\\qquad +\\frac{1-p}{n}\\sum_{i=1}^n\\sqnorm{\\nabla f_i(x^{t+1})-\\nabla f_i(x^{t})-\\nabla f(x^{t+1})+\\nabla f(x^t)}.\n\\end{align*}\n\\textbf{Third}, we have\n\\begin{align*}\n\\Exp{\\sqnorm{g^{t+1}}\\;|\\;\\mathcal{F}^t} &= p \\sqnorm{\\nabla f(x^{t+1})}+ (1-p)\\Exp{\\sqnorm{g^{t+1}}\\;|\\;\\theta^t=0,\\mathcal{F}^t}\\\\\n&=p \\sqnorm{\\nabla f(x^{t+1})}+ (1-p)\\sqnorm{g^{t}+\\nabla f(x^{t+1})-\\nabla f(x^t)}\\\\\n&\\quad +\\frac{1-p}{n}\\sum_{i=1}^n\\sqnorm{\\nabla f_i(x^{t+1})-\\nabla f_i(x^{t})-\\nabla f(x^{t+1})+\\nabla f(x^t)}.\n\\end{align*}\nUsing the fact that $g^t=-\\frac{1}{\\gamma}(x^{t+1}-x^t)$, we obtain\n\\begin{align*}\n\\sqnorm{g^{t}+\\nabla f(x^{t+1})-\\nabla f(x^t)} &= \\sqnorm{g^{t}} + \\sqnorm{\\nabla f(x^{t+1})-\\nabla f(x^t)} +2\\!\\left\\langle g^{t},\\nabla f(x^{t+1})-\\nabla f(x^t)\\right\\rangle\\\\\n&=\\sqnorm{g^{t}} + \\sqnorm{\\nabla f(x^{t+1})-\\nabla f(x^t)}\\\\\n&\\quad -\\frac{2}{\\gamma}\\left\\langle x^{t+1}-x^t,\\nabla f(x^{t+1})-\\nabla f(x^t)\\right\\rangle.\n\\end{align*}\nThe negative inner product $-\\frac{2}{\\gamma}\\left\\langle x^{t+1}-x^t,\\nabla f(x^{t+1})-\\nabla f(x^t)\\right\\rangle$ that appears is crucial, as it enables the cancellation of positive terms using Lemma~\\ref{lemm2}.\n\nSince $f=\\frac{1}{n}\\sum_{i=1}^n f_i$, we have\n\\begin{align*}\n&\\frac{1}{n}\\sum_{i=1}^n\\sqnorm{\\nabla f_i(x^{t+1})-\\nabla f_i(x^{t})-\\nabla f(x^{t+1})+\\nabla f(x^t)}\\\\\n&\\quad= \\frac{1}{n}\\sum_{i=1}^n\\sqnorm{\\nabla f_i(x^{t+1})-\\nabla f_i(x^{t})} -\\sqnorm{\\nabla f(x^{t+1})-\\nabla f(x^t)}.\n\\end{align*}\nTherefore,\n\\begin{align*}\n\\Exp{\\sqnorm{g^{t+1}}\\;|\\;\\mathcal{F}^t} &=p \\sqnorm{\\nabla f(x^{t+1})}+ (1-p)\\sqnorm{g^{t}}\\\\\n&\\quad -\\frac{2(1-p)}{\\gamma}\\left\\langle x^{t+1}-x^t,\\nabla f(x^{t+1})-\\nabla f(x^t)\\right\\rangle\\\\\n &\\quad +\\frac{1-p}{n}\\sum_{i=1}^n\\sqnorm{\\nabla f_i(x^{t+1})-\\nabla f_i(x^{t})}.\n\\end{align*}\n\nMoreover, by Lemma~\\ref{lemm2} and the smoothness and weak convexity of the functions $f_i$, we obtain\n\\begin{align*}\n\\frac{1}{n}\\sum_{i=1}^n \\sqnorm{\\nabla f_i(x^{t+1})- \\nabla f_i(x^t)}&\\leq\\frac{1}{n}\\sum_{i=1}^n \\Big( (L-\\tau)\\big\\langle \\nabla f_i(x^{t+1}) - \\nabla f_i(x^t),x^{t+1}-x^t\\big\\rangle \\\\\n&\\quad\\left.{}+ L\\tau \\sqnorm{x^{t+1}-x^t}\\right)\\\\\n&= (L-\\tau)\\big\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\big\\rangle \\\\\n&\\quad+ L\\tau \\sqnorm{x^{t+1}-x^t}.\n\\end{align*}\nThe inner products $(L-\\tau)\\big\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\big\\rangle$ will be canceled using the negative inner product $-\\frac{2}{\\gamma}\\left\\langle x^{t+1}-x^t,\\nabla f(x^{t+1})-\\nabla f(x^t)\\right\\rangle$ that appeared above, while the terms \n$\\sqnorm{x^{t+1}-x^t}$ will be canceled by choosing $\\gamma$ sufficiently small and using the negative term $(\\frac{ L}{2}-\\frac{1}{2\\gamma})\\sqnorm{x^{t+1}-x^t}$ arising in the descent property.\n\nThus, \n\\begin{align*}\n\\Exp{\\sqnorm{g^{t+1}-\\nabla f(x^{t+1})}\\;|\\;\\mathcal{F}^t} \n&\\leq (1-p)\\sqnorm{g^{t}-\\nabla f(x^t)}\\!+\\!\n\\frac{1-p}{n}\\sum_{i=1}^n \\sqnorm{\\nabla f_i(x^{t+1})- \\nabla f_i(x^t)}\\\\\n&\\leq (1-p)\\sqnorm{g^{t}-\\nabla f(x^t)}+ (1-p)L\\tau \\sqnorm{x^{t+1}-x^t}\\\\\n&\\quad +(1-p)(L-\\tau)  \\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle\n\\end{align*}\nand\n\\begin{align*}\n\\Exp{\\sqnorm{g^{t+1}}\\;|\\;\\mathcal{F}^t} \n&\\leq p \\sqnorm{\\nabla f(x^{t+1})}+ (1-p)\\sqnorm{g^{t}}\\\\\n&\\quad -\\frac{2(1-p)}{\\gamma}\\left\\langle \\nabla f(x^{t+1})-\\nabla f(x^t),x^{t+1}-x^t\\right\\rangle\\\\\n &\\quad +(1-p)(L-\\tau)\\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle\\\\\n&\\quad +(1-p)L\\tau \\sqnorm{x^{t+1}-x^t}\\\\\n &= p\\sqnorm{\\nabla f(x^{t+1})}+ (1-p)\\big(1+L\\tau\\gamma^2\\big)\n\\sqnorm{g^{t}} \\\\\n&\\quad +(1-p)\\left(L- \\tau-\\frac{2}{\\gamma}\\right) \\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle .\n\\end{align*}\nHence, combining the inequalities for these terms, we obtain\n\\begin{align*}\n\\Exp{\\Psi^{t+1}\\;|\\;\\mathcal{F}^t}&\\leq\nf(x^t)  -f^\\star -\\frac{\\gamma}{2} \\sqnorm{\\nabla f(x^t)} +\\frac{ \\gamma }{2}\\sqnorm{g^{t}-\\nabla f(x^t)}+ \\left(\\frac{ L\\gamma^2}{2}-\\frac{\\gamma}{2}\\right)\\sqnorm{g^t} \\\\\n&\\quad -b\\gamma\\sqnorm{\\nabla f(x^{t+1})}\n+\\frac{a\\gamma}{p} (1-p)\\sqnorm{g^{t}-\\nabla f(x^t)}\n+ a\\gamma^3 L\\tau \\frac{1-p}{p}\\sqnorm{g^t}\\\\\n&\\quad+a\\gamma (L-\\tau) \\frac{1-p}{p}\\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle+b\\gamma\\sqnorm{\\nabla f(x^{t+1})}\\notag\\\\\n&\\quad+\\frac{b\\gamma}{p}  (1-p)\\big(1+L\\tau\\gamma^2\\big)\\sqnorm{g^{t}}\\\\\n&\\quad+ \\big(b\\gamma (L-\\tau)-2b\\big) \\frac{1-p}{p}\\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle\\notag\\\\\n&= f(x^t)  -f^\\star  -\\frac{\\gamma}{2} \\sqnorm{\\nabla f(x^t)}+\\frac{a\\gamma}{p}\\left(1-p\\left(1-\\frac{1 }{2a}\\right)  \\right)\\sqnorm{g^{t}-\\nabla f(x^t)}\\notag\\\\\n&\\quad+\\frac{b\\gamma}{p}  (1-p)\\sqnorm{g^{t}}\n+\\frac{\\gamma}{2}\\left(L\\gamma-1+ 2(a+b)\\gamma^2 L\\tau \\frac{1-p}{p}\\right)\\sqnorm{g^{t}}\\notag\\\\\n&\\quad+ \\big((a+b)\\gamma (L-\\tau)-2b\\big) \\frac{1-p}{p}\\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle.\n\\end{align*}\nTo cancel the inner product, we set\n\\begin{equation}\nb\\coloneqq \\frac{a\\gamma (L-\\tau)}{2-\\gamma(L-\\tau)},\\label{eqbb}\n\\end{equation}\nso that $(a+b)\\gamma (L-\\tau)-2b=0$. Note that $\\gamma\\leq \\frac{1}{L}$ implies $2-\\gamma(L-\\tau)\\geq 1$. Moreover,\na sufficient condition for $L\\gamma-1+ 2(a+b)\\gamma^2 L\\tau \\frac{1-p}{p}\\leq 0$ is  \\cite[Lemma 5]{ric21}\n\\begin{equation}\n\\gamma \\leq \\frac{1}{L+\\sqrt{2(a+b)L\\tau \\frac{1-p}{p}}}.\\label{eqlllo}\n\\end{equation}\nTo avoid the dependence of $b$ on $\\gamma$, we derive from $\\gamma\\leq \\frac{1}{L}$ that \n\\begin{equation*}\nb\\leq  \\frac{a (L-\\tau)}{L+\\tau},%\\label{eqww}\n\\end{equation*}\nso that $a+b\\leq  \\frac{2aL}{L+\\tau}$. Therefore, a sufficient condition for \\eqref{eqlllo} is\n\\begin{equation}\n\\gamma \\leq \\frac{1}{L+\\sqrt{ \\frac{4aL}{L+\\tau}L\\tau \\frac{1-p}{p}}}=\\frac{1}{L\\left(1+\\sqrt{ \\frac{4a\\tau}{L+\\tau}}\\sqrt{  \\frac{1-p}{p}}\\right)}.\\label{eqlllo2}\n\\end{equation}\nIf we additionally assume $a\\leq 1$,  the dependence on $a$ can be eliminated with the sufficient condition \n\\begin{equation*}\n\\gamma \\leq \\frac{1}{L\\left(1+\\sqrt{ \\frac{4\\tau}{L+\\tau}}\\sqrt{  \\frac{1-p}{p}}\\right)},\n\\end{equation*}\nas stated in the theorem.\n\nTherefore, since the sufficient conditions above are supposed to hold, we have\n\\begin{align*}\n\\Exp{\\Psi^{t+1}\\;|\\;\\mathcal{F}^t}&\\leq\n f(x^t)  -f^\\star  -\\frac{\\gamma}{2} \\sqnorm{\\nabla f(x^t)}+\\frac{a\\gamma}{p}\\left(1-p\\left(1-\\frac{1 }{2a}\\right)  \\right)\\sqnorm{g^{t}-\\nabla f(x^t)}\\\\\n&\\quad+\\frac{b\\gamma}{p}  (1-p)\\sqnorm{g^{t}}\\\\\n&= f(x^t)  -f^\\star  -\\gamma\\left(\\frac{1}{2}-b\\right)\\sqnorm{\\nabla f(x^t)} -b\\gamma \\sqnorm{\\nabla f(x^t)}\\\\\n&\\quad+\\frac{a\\gamma}{p}\\left(1-p\\left(1-\\frac{1}{2a}\\right)  \\right)\\sqnorm{g^{t}-\\nabla f(x^t)}+\\frac{b\\gamma}{p}  (1-p)\\sqnorm{g^{t}}.\n\\end{align*}\nTo obtain a contraction, we need $a>\\frac{1}{2}$ and $b<\\frac{1}{2}$. The choice \n\\begin{equation*}\na= 1-\\frac{\\gamma (L-\\tau)}{2}, \\quad b= \\frac{\\gamma (L-\\tau)}{2},\n\\end{equation*}\nwith the additional condition $\\gamma<\\frac{1}{L}$ if $\\tau=0$, satisfies  these two inequalities, as well as \\eqref{eqbb}, $a\\leq 1$, and $b \\gamma \\leq \\frac{1}{2L}$.\n\nFinally, under the assumption that $f$ is $\\mu$-P{\\L}, we have\n\\begin{equation*}\n\\|\\nabla f(x^t)\\|^2\\geq 2\\mu\\big(f(x^t)-f^\\star\\big).\n\\end{equation*}\nTherefore,\n\\begin{align}\n\\Exp{\\Psi^{t+1}\\;|\\;\\mathcal{F}^t}&\\leq  \\big(1-(1-2b)\\gamma\\mu\\big)\\big(f(x^t)  -f^\\star\\big)  -b\\gamma \\sqnorm{\\nabla f(x^t)}\\notag\\\\\n&\\quad+\\frac{a\\gamma}{p}\\left(1-p\\left(1-\\frac{1}{2a}\\right)  \\right)\\sqnorm{g^{t}-\\nabla f(x^t)}+\\frac{b\\gamma}{p}  (1-p)\\sqnorm{g^{t}}\\notag\\\\\n&\\leq  \\big(1-(1-2b)\\gamma\\mu\\big)\\big(f(x^t)  -f^\\star  -b\\gamma \\sqnorm{\\nabla f(x^t)}\\big)\\notag\\\\\n&\\quad+\\frac{a\\gamma}{p}\\left(1-p\\left(1-\\frac{1}{2a}\\right)  \\right)\\sqnorm{g^{t}-\\nabla f(x^t)}+\\frac{b\\gamma}{p}  (1-p)\\sqnorm{g^{t}}\\notag\\\\\n&\\leq \\max\\left(1-(1-2b)\\gamma\\mu,1-p\\left(1-\\frac{1}{2a}\\right),1-p\\right)\\Psi^t\\notag\\\\\n&= \\max\\left(1-(1-2b)\\gamma\\mu,1-p\\left(1-\\frac{1}{2a}\\right)\\right)\\Psi^t.\\label{eqrec2b}\n\\end{align}\nUsing the tower rule of expectations, we can unroll the recursion in \\eqref{eqrec2b} to obtain the unconditional expectation of $\\Psi^{t+1}$. Moreover, using classical results on supermartingale convergence \\cite[Proposition A.4.5]{ber15}, it follows from \\eqref{eqrec2b} that $\\Psi^t\\rightarrow 0$ almost surely. \\qed\n%\\hfill$\\square$\n\n\n\n\n","units":[{"start":12257,"end":12608,"start_line":198,"end_line":203,"raw_tex":"\\begin{lemma}\\label{lemm2}\nLet $g:\\mathcal{X}\\rightarrow \\mathbb{R}$ be an $L$-smooth and $\\tau$-weakly convex function for some $L>0$ and $\\tau\\in [0,L]$. Then, for every $(x,y)\\in\\mathcal{X}^2$, \n\\begin{align*}\n\\sqnorm{\\nabla g(x)- \\nabla g(y)}&\\leq (L-\\tau)\\langle \\nabla g(x) - \\nabla g(y),x-y\\rangle + L\\tau \\sqnorm{x-y}.\n\\end{align*}\n\\end{lemma}","sha256_utf8":"04b6c3f67fae46ca1f7079ec87a213e45b1818b4d29c915e7ec35ec448c7f740"},{"start":24311,"end":34733,"start_line":413,"end_line":578,"raw_tex":"\\subsection{Proof of Lemma~\\ref{lemm2}}\n\nLet $(x,y)\\in\\mathcal{X}^2$. Since $g+\\frac{\\tau}{2}\\|\\cdot\\|^2$ is convex and $(L+\\tau)$-smooth, its gradient is $\\frac{1}{L+\\tau}$-cocoercive according to the Baillon--Haddad theorem \\cite[Corollary 18.17]{bau17}, so that\n\\begin{align*}\n\\sqnorm{\\nabla g(x)+\\tau x - \\nabla g(y)-\\tau y} &\\leq (L+\\tau)\\langle \\nabla g(x) +\\tau x- \\nabla g(y)-\\tau y,x-y\\rangle\\\\\n&=(L+\\tau)\\langle \\nabla g(x) - \\nabla g(y),x-y\\rangle +  (L+\\tau)\\tau \\sqnorm{x-y}.\n\\end{align*}\nHence,\n\\begin{align*}\n\\sqnorm{\\nabla g(x)- \\nabla g(y)}&=\\sqnorm{\\nabla g(x)+\\tau x- \\nabla g(y)-\\tau y}\\!-\\!2\\tau\n\\langle \\nabla g(x) - \\nabla g(y),x-y\\rangle \n\\!-\\!\\tau^2\\sqnorm{x-y}\\\\\n&\\leq (L-\\tau)\\langle \\nabla g(x) - \\nabla g(y),x-y\\rangle + L\\tau \\sqnorm{x-y}.\n\\end{align*}\\qed\n\n\n\n\\subsection{Proof of Theorem~\\ref{theo1}}\\label{secproof1}\n\nFor every $t\\geq 0$, let $\\mathcal{F}^t$ denote the $\\sigma$-algebra generated by the random variables $x^0,g^0,\\ldots, x^t,g^t$. For the moment, assume that $a$ and $b$ in \\eqref{eqly} satisfy  $a>0$, \n$0\\leq b\\leq \\frac{1}{2}$, $b\\gamma  \\leq \\frac{1}{2L}$, without additional restrictions.\n\n\nLet $t\\geq 0$. We now derive upper bounds for the terms in $\\Exp{\\Psi^{t+1}\\;|\\;\\mathcal{F}^t}$.\n\n\\noindent \\textbf{First}, we have the descent property \\cite[Lemma 4]{ric21}\n\\begin{align*}\nf(x^{t+1}) -f^\\star &\\leq f(x^t)  -f^\\star -\\frac{\\gamma}{2} \\sqnorm{\\nabla f(x^t)} +\\frac{ \\gamma }{2}\\sqnorm{g^{t}-\\nabla f(x^t)}+ \\left(\\frac{ L}{2}-\\frac{1}{2\\gamma}\\right)\\sqnorm{x^{t+1}-x^t}\\\\\n&=f(x^t)  -f^\\star -\\frac{\\gamma}{2} \\sqnorm{\\nabla f(x^t)} +\\frac{ \\gamma }{2}\\sqnorm{g^{t}-\\nabla f(x^t)}+ \\left(\\frac{ L\\gamma^2}{2}-\\frac{\\gamma}{2}\\right)\\sqnorm{g^t}.\n\\end{align*}\n\\textbf{Second}, we have \n\\begin{align*}\n\\Exp{\\sqnorm{g^{t+1}-\\nabla f(x^{t+1})}\\;|\\;\\mathcal{F}^t} &= (1-p)\\Exp{\\sqnorm{g^{t+1}-\\nabla f(x^{t+1})}\\;|\\;\\theta^t=0,\\mathcal{F}^t},\n\\end{align*}\nand by decomposing \nthe expected squared norm into its squared mean and variance, we obtain\n\\begin{align*}\n&\\Exp{\\sqnorm{g^{t+1}-\\nabla f(x^{t+1})}\\;|\\;\\mathcal{F}^t} = (1-p)\\sqnorm{g^{t}-\\nabla f(x^t)}\\\\\n&\\qquad +\\frac{1-p}{n}\\sum_{i=1}^n\\sqnorm{\\nabla f_i(x^{t+1})-\\nabla f_i(x^{t})-\\nabla f(x^{t+1})+\\nabla f(x^t)}.\n\\end{align*}\n\\textbf{Third}, we have\n\\begin{align*}\n\\Exp{\\sqnorm{g^{t+1}}\\;|\\;\\mathcal{F}^t} &= p \\sqnorm{\\nabla f(x^{t+1})}+ (1-p)\\Exp{\\sqnorm{g^{t+1}}\\;|\\;\\theta^t=0,\\mathcal{F}^t}\\\\\n&=p \\sqnorm{\\nabla f(x^{t+1})}+ (1-p)\\sqnorm{g^{t}+\\nabla f(x^{t+1})-\\nabla f(x^t)}\\\\\n&\\quad +\\frac{1-p}{n}\\sum_{i=1}^n\\sqnorm{\\nabla f_i(x^{t+1})-\\nabla f_i(x^{t})-\\nabla f(x^{t+1})+\\nabla f(x^t)}.\n\\end{align*}\nUsing the fact that $g^t=-\\frac{1}{\\gamma}(x^{t+1}-x^t)$, we obtain\n\\begin{align*}\n\\sqnorm{g^{t}+\\nabla f(x^{t+1})-\\nabla f(x^t)} &= \\sqnorm{g^{t}} + \\sqnorm{\\nabla f(x^{t+1})-\\nabla f(x^t)} +2\\!\\left\\langle g^{t},\\nabla f(x^{t+1})-\\nabla f(x^t)\\right\\rangle\\\\\n&=\\sqnorm{g^{t}} + \\sqnorm{\\nabla f(x^{t+1})-\\nabla f(x^t)}\\\\\n&\\quad -\\frac{2}{\\gamma}\\left\\langle x^{t+1}-x^t,\\nabla f(x^{t+1})-\\nabla f(x^t)\\right\\rangle.\n\\end{align*}\nThe negative inner product $-\\frac{2}{\\gamma}\\left\\langle x^{t+1}-x^t,\\nabla f(x^{t+1})-\\nabla f(x^t)\\right\\rangle$ that appears is crucial, as it enables the cancellation of positive terms using Lemma~\\ref{lemm2}.\n\nSince $f=\\frac{1}{n}\\sum_{i=1}^n f_i$, we have\n\\begin{align*}\n&\\frac{1}{n}\\sum_{i=1}^n\\sqnorm{\\nabla f_i(x^{t+1})-\\nabla f_i(x^{t})-\\nabla f(x^{t+1})+\\nabla f(x^t)}\\\\\n&\\quad= \\frac{1}{n}\\sum_{i=1}^n\\sqnorm{\\nabla f_i(x^{t+1})-\\nabla f_i(x^{t})} -\\sqnorm{\\nabla f(x^{t+1})-\\nabla f(x^t)}.\n\\end{align*}\nTherefore,\n\\begin{align*}\n\\Exp{\\sqnorm{g^{t+1}}\\;|\\;\\mathcal{F}^t} &=p \\sqnorm{\\nabla f(x^{t+1})}+ (1-p)\\sqnorm{g^{t}}\\\\\n&\\quad -\\frac{2(1-p)}{\\gamma}\\left\\langle x^{t+1}-x^t,\\nabla f(x^{t+1})-\\nabla f(x^t)\\right\\rangle\\\\\n &\\quad +\\frac{1-p}{n}\\sum_{i=1}^n\\sqnorm{\\nabla f_i(x^{t+1})-\\nabla f_i(x^{t})}.\n\\end{align*}\n\nMoreover, by Lemma~\\ref{lemm2} and the smoothness and weak convexity of the functions $f_i$, we obtain\n\\begin{align*}\n\\frac{1}{n}\\sum_{i=1}^n \\sqnorm{\\nabla f_i(x^{t+1})- \\nabla f_i(x^t)}&\\leq\\frac{1}{n}\\sum_{i=1}^n \\Big( (L-\\tau)\\big\\langle \\nabla f_i(x^{t+1}) - \\nabla f_i(x^t),x^{t+1}-x^t\\big\\rangle \\\\\n&\\quad\\left.{}+ L\\tau \\sqnorm{x^{t+1}-x^t}\\right)\\\\\n&= (L-\\tau)\\big\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\big\\rangle \\\\\n&\\quad+ L\\tau \\sqnorm{x^{t+1}-x^t}.\n\\end{align*}\nThe inner products $(L-\\tau)\\big\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\big\\rangle$ will be canceled using the negative inner product $-\\frac{2}{\\gamma}\\left\\langle x^{t+1}-x^t,\\nabla f(x^{t+1})-\\nabla f(x^t)\\right\\rangle$ that appeared above, while the terms \n$\\sqnorm{x^{t+1}-x^t}$ will be canceled by choosing $\\gamma$ sufficiently small and using the negative term $(\\frac{ L}{2}-\\frac{1}{2\\gamma})\\sqnorm{x^{t+1}-x^t}$ arising in the descent property.\n\nThus, \n\\begin{align*}\n\\Exp{\\sqnorm{g^{t+1}-\\nabla f(x^{t+1})}\\;|\\;\\mathcal{F}^t} \n&\\leq (1-p)\\sqnorm{g^{t}-\\nabla f(x^t)}\\!+\\!\n\\frac{1-p}{n}\\sum_{i=1}^n \\sqnorm{\\nabla f_i(x^{t+1})- \\nabla f_i(x^t)}\\\\\n&\\leq (1-p)\\sqnorm{g^{t}-\\nabla f(x^t)}+ (1-p)L\\tau \\sqnorm{x^{t+1}-x^t}\\\\\n&\\quad +(1-p)(L-\\tau)  \\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle\n\\end{align*}\nand\n\\begin{align*}\n\\Exp{\\sqnorm{g^{t+1}}\\;|\\;\\mathcal{F}^t} \n&\\leq p \\sqnorm{\\nabla f(x^{t+1})}+ (1-p)\\sqnorm{g^{t}}\\\\\n&\\quad -\\frac{2(1-p)}{\\gamma}\\left\\langle \\nabla f(x^{t+1})-\\nabla f(x^t),x^{t+1}-x^t\\right\\rangle\\\\\n &\\quad +(1-p)(L-\\tau)\\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle\\\\\n&\\quad +(1-p)L\\tau \\sqnorm{x^{t+1}-x^t}\\\\\n &= p\\sqnorm{\\nabla f(x^{t+1})}+ (1-p)\\big(1+L\\tau\\gamma^2\\big)\n\\sqnorm{g^{t}} \\\\\n&\\quad +(1-p)\\left(L- \\tau-\\frac{2}{\\gamma}\\right) \\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle .\n\\end{align*}\nHence, combining the inequalities for these terms, we obtain\n\\begin{align*}\n\\Exp{\\Psi^{t+1}\\;|\\;\\mathcal{F}^t}&\\leq\nf(x^t)  -f^\\star -\\frac{\\gamma}{2} \\sqnorm{\\nabla f(x^t)} +\\frac{ \\gamma }{2}\\sqnorm{g^{t}-\\nabla f(x^t)}+ \\left(\\frac{ L\\gamma^2}{2}-\\frac{\\gamma}{2}\\right)\\sqnorm{g^t} \\\\\n&\\quad -b\\gamma\\sqnorm{\\nabla f(x^{t+1})}\n+\\frac{a\\gamma}{p} (1-p)\\sqnorm{g^{t}-\\nabla f(x^t)}\n+ a\\gamma^3 L\\tau \\frac{1-p}{p}\\sqnorm{g^t}\\\\\n&\\quad+a\\gamma (L-\\tau) \\frac{1-p}{p}\\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle+b\\gamma\\sqnorm{\\nabla f(x^{t+1})}\\notag\\\\\n&\\quad+\\frac{b\\gamma}{p}  (1-p)\\big(1+L\\tau\\gamma^2\\big)\\sqnorm{g^{t}}\\\\\n&\\quad+ \\big(b\\gamma (L-\\tau)-2b\\big) \\frac{1-p}{p}\\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle\\notag\\\\\n&= f(x^t)  -f^\\star  -\\frac{\\gamma}{2} \\sqnorm{\\nabla f(x^t)}+\\frac{a\\gamma}{p}\\left(1-p\\left(1-\\frac{1 }{2a}\\right)  \\right)\\sqnorm{g^{t}-\\nabla f(x^t)}\\notag\\\\\n&\\quad+\\frac{b\\gamma}{p}  (1-p)\\sqnorm{g^{t}}\n+\\frac{\\gamma}{2}\\left(L\\gamma-1+ 2(a+b)\\gamma^2 L\\tau \\frac{1-p}{p}\\right)\\sqnorm{g^{t}}\\notag\\\\\n&\\quad+ \\big((a+b)\\gamma (L-\\tau)-2b\\big) \\frac{1-p}{p}\\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle.\n\\end{align*}\nTo cancel the inner product, we set\n\\begin{equation}\nb\\coloneqq \\frac{a\\gamma (L-\\tau)}{2-\\gamma(L-\\tau)},\\label{eqbb}\n\\end{equation}\nso that $(a+b)\\gamma (L-\\tau)-2b=0$. Note that $\\gamma\\leq \\frac{1}{L}$ implies $2-\\gamma(L-\\tau)\\geq 1$. Moreover,\na sufficient condition for $L\\gamma-1+ 2(a+b)\\gamma^2 L\\tau \\frac{1-p}{p}\\leq 0$ is  \\cite[Lemma 5]{ric21}\n\\begin{equation}\n\\gamma \\leq \\frac{1}{L+\\sqrt{2(a+b)L\\tau \\frac{1-p}{p}}}.\\label{eqlllo}\n\\end{equation}\nTo avoid the dependence of $b$ on $\\gamma$, we derive from $\\gamma\\leq \\frac{1}{L}$ that \n\\begin{equation*}\nb\\leq  \\frac{a (L-\\tau)}{L+\\tau},%\\label{eqww}\n\\end{equation*}\nso that $a+b\\leq  \\frac{2aL}{L+\\tau}$. Therefore, a sufficient condition for \\eqref{eqlllo} is\n\\begin{equation}\n\\gamma \\leq \\frac{1}{L+\\sqrt{ \\frac{4aL}{L+\\tau}L\\tau \\frac{1-p}{p}}}=\\frac{1}{L\\left(1+\\sqrt{ \\frac{4a\\tau}{L+\\tau}}\\sqrt{  \\frac{1-p}{p}}\\right)}.\\label{eqlllo2}\n\\end{equation}\nIf we additionally assume $a\\leq 1$,  the dependence on $a$ can be eliminated with the sufficient condition \n\\begin{equation*}\n\\gamma \\leq \\frac{1}{L\\left(1+\\sqrt{ \\frac{4\\tau}{L+\\tau}}\\sqrt{  \\frac{1-p}{p}}\\right)},\n\\end{equation*}\nas stated in the theorem.\n\nTherefore, since the sufficient conditions above are supposed to hold, we have\n\\begin{align*}\n\\Exp{\\Psi^{t+1}\\;|\\;\\mathcal{F}^t}&\\leq\n f(x^t)  -f^\\star  -\\frac{\\gamma}{2} \\sqnorm{\\nabla f(x^t)}+\\frac{a\\gamma}{p}\\left(1-p\\left(1-\\frac{1 }{2a}\\right)  \\right)\\sqnorm{g^{t}-\\nabla f(x^t)}\\\\\n&\\quad+\\frac{b\\gamma}{p}  (1-p)\\sqnorm{g^{t}}\\\\\n&= f(x^t)  -f^\\star  -\\gamma\\left(\\frac{1}{2}-b\\right)\\sqnorm{\\nabla f(x^t)} -b\\gamma \\sqnorm{\\nabla f(x^t)}\\\\\n&\\quad+\\frac{a\\gamma}{p}\\left(1-p\\left(1-\\frac{1}{2a}\\right)  \\right)\\sqnorm{g^{t}-\\nabla f(x^t)}+\\frac{b\\gamma}{p}  (1-p)\\sqnorm{g^{t}}.\n\\end{align*}\nTo obtain a contraction, we need $a>\\frac{1}{2}$ and $b<\\frac{1}{2}$. The choice \n\\begin{equation*}\na= 1-\\frac{\\gamma (L-\\tau)}{2}, \\quad b= \\frac{\\gamma (L-\\tau)}{2},\n\\end{equation*}\nwith the additional condition $\\gamma<\\frac{1}{L}$ if $\\tau=0$, satisfies  these two inequalities, as well as \\eqref{eqbb}, $a\\leq 1$, and $b \\gamma \\leq \\frac{1}{2L}$.\n\nFinally, under the assumption that $f$ is $\\mu$-P{\\L}, we have\n\\begin{equation*}\n\\|\\nabla f(x^t)\\|^2\\geq 2\\mu\\big(f(x^t)-f^\\star\\big).\n\\end{equation*}\nTherefore,\n\\begin{align}\n\\Exp{\\Psi^{t+1}\\;|\\;\\mathcal{F}^t}&\\leq  \\big(1-(1-2b)\\gamma\\mu\\big)\\big(f(x^t)  -f^\\star\\big)  -b\\gamma \\sqnorm{\\nabla f(x^t)}\\notag\\\\\n&\\quad+\\frac{a\\gamma}{p}\\left(1-p\\left(1-\\frac{1}{2a}\\right)  \\right)\\sqnorm{g^{t}-\\nabla f(x^t)}+\\frac{b\\gamma}{p}  (1-p)\\sqnorm{g^{t}}\\notag\\\\\n&\\leq  \\big(1-(1-2b)\\gamma\\mu\\big)\\big(f(x^t)  -f^\\star  -b\\gamma \\sqnorm{\\nabla f(x^t)}\\big)\\notag\\\\\n&\\quad+\\frac{a\\gamma}{p}\\left(1-p\\left(1-\\frac{1}{2a}\\right)  \\right)\\sqnorm{g^{t}-\\nabla f(x^t)}+\\frac{b\\gamma}{p}  (1-p)\\sqnorm{g^{t}}\\notag\\\\\n&\\leq \\max\\left(1-(1-2b)\\gamma\\mu,1-p\\left(1-\\frac{1}{2a}\\right),1-p\\right)\\Psi^t\\notag\\\\\n&= \\max\\left(1-(1-2b)\\gamma\\mu,1-p\\left(1-\\frac{1}{2a}\\right)\\right)\\Psi^t.\\label{eqrec2b}\n\\end{align}\nUsing the tower rule of expectations, we can unroll the recursion in \\eqref{eqrec2b} to obtain the unconditional expectation of $\\Psi^{t+1}$. Moreover, using classical results on supermartingale convergence \\cite[Proposition A.4.5]{ber15}, it follows from \\eqref{eqrec2b} that $\\Psi^t\\rightarrow 0$ almost surely. \\qed\n%\\hfill$\\square$\n\n\n\n\n","sha256_utf8":"2d5fc8f0bf5afbbfcd136bb3c4dc85bdc3f5aa4b80ffdeca1bf075815e5596c3"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"有限和优化问题","source_units":[{"start":2545,"end":2894,"start_line":78,"end_line":82,"raw_tex":"Formally, let $\\mathcal{X}$ be a finite-dimensional real Hilbert space. Given a number  $n\\geq 1$ of smooth functions $f_i:\\mathcal{X}\\rightarrow \\mathbb{R}$, $i\\in[n]\\coloneqq \\{1,\\ldots,n\\}$, \n the finite-sum optimization problem is\n\\begin{equation}\n\\min_{x\\in\\mathcal{X}} \\,f(x)\\coloneqq \\frac{1}{n}\\sum_{i=1}^n f_i(x).\\label{eqp1}\n\\end{equation}","sha256_utf8":"4afc86b46047d0664d30505a4b2cf6e267d2ecd1da421618441e92350d1e4eb5"}]},{"id":"S002","semantic_type":"assumption","kind":"assumption","title":"光滑弱凸与下界假设","source_units":[{"start":3481,"end":4006,"start_line":92,"end_line":97,"raw_tex":"We make the following assumptions on the functions.\n\\begin{itemize}\n\\item There exists $L>0$ such that $f_i$ is $L$-smooth for every $i\\in[n]$; that is, $f_i$ is differentiable on $\\mathcal{X}$ and for every $(x,y)\\in\\mathcal{X}^2$, $\\|\\nabla f_i(x)-\\nabla f_i(y)\\|\\leq L \\|x-y\\|$.\n\\item There exists $\\tau\\in [0,L]$ such that $f_i$ is $\\tau$-weakly convex for every $i\\in[n]$; that is, $f_i+\\frac{\\tau}{2} \\|\\cdot\\|^2$ is convex \\cite{nur73}.\n\\item $f$ is lower bounded: $f^\\star \\coloneqq \\inf f > -\\infty$. \n\\end{itemize}\n","sha256_utf8":"e0ba2d99ef6ce98acb89fb811d1af4cb9d78dfbc19f5cb6e94f87b2e633a7628"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"PŁ条件与条件数","source_units":[{"start":4532,"end":4929,"start_line":103,"end_line":109,"raw_tex":"In addition, we establish linear convergence when \n$f$ satisfies the \nPolyak--{\\L}ojasiewicz (P{\\L}) condition with constant $\\mu>0$, or in short is $\\mu$-P{\\L} \\cite{kar16}:\n \\begin{equation*}\n\\|\\nabla f(x)\\|^2\\geq 2\\mu\\big(f(x)-f^\\star\\big),\\ \\mbox{for every }x\\in\\mathcal{X}.\n\\end{equation*}\nIn this case, $\\mu\\leq L$ and we define the condition number $\\kappa\\coloneqq \\frac{L}{\\mu}\\geq 1$.  \n","sha256_utf8":"b77b58e5accbda8e305e7d5eb731069835de910485e62b7bc65ffc22ee9f8671"}]},{"id":"S004","semantic_type":"algorithm","kind":"algorithm","title":"PAGE随机迭代","source_units":[{"start":6003,"end":6628,"start_line":119,"end_line":134,"raw_tex":"\\begin{algorithm}[H] \n\t\t\\caption{\\algn{PAGE}}\\label{alg1}\n\t\t\\begin{algorithmic}\n\t\t\t\\STATE  \\textbf{input:} initial estimate $x^0$, initial gradient estimate $g^0$, stepsize $\\gamma>0$, probability $p\\in (0,1]$.\n\t\t\t\\FOR{$t=0, 1, \\ldots$}\n\t\t\t\\STATE $x^{t+1} \\coloneqq x^t - \\gamma g^t$\n\t\t\t\\STATE flip a coin $\\theta^t = (1$ with probability $p$, $0$ otherwise)\n\t\t\t\\IF{$\\theta^t=0$}\n\t\t\t\\STATE pick $i^t \\in [n]$ uniformly at random\n\t\t\t\\STATE $g^{t+1} \\coloneqq g^t +\\nabla f_{i^t}(x^{t+1})-\\nabla f_{i^t}(x^{t})$\n\t\t\t\\ELSE\n\t\t\t\\STATE $g^{t+1} \\coloneqq \\nabla f(x^{t+1})$\n\t\t\t\\ENDIF \n\t\t\t\\ENDFOR\n\t\t\\end{algorithmic}\n\t\\end{algorithm}","sha256_utf8":"de38d00aa5ce8387b9ac40f72893cf48e4a6517fce5a68781f8814f624ea82ab"}]},{"id":"S005","semantic_type":"definition","kind":"definition","title":"李雅普诺夫函数","source_units":[{"start":11318,"end":11508,"start_line":186,"end_line":188,"raw_tex":"\\begin{equation}\n\\Psi^t \\coloneqq f(x^t)-f^\\star -b\\gamma \\sqnorm{\\nabla f(x^t)} + \\frac{a \\gamma}{p} \\sqnorm{g^t-\\nabla f(x^t)} + \\frac{b \\gamma}{p} \\sqnorm{g^t},\\label{eqly}\n\\end{equation}","sha256_utf8":"93711fbd1100afb29279830fdff1a730b2f087e32a730b299f23458840157399"}]},{"id":"S006","semantic_type":"derived_fact","kind":"derived_fact","title":"势函数的非负性","source_units":[{"start":11509,"end":11956,"start_line":189,"end_line":193,"raw_tex":"for some constants $a>0$ and $b\\in[ 0, \\frac{1}{2\\gamma L}]$. Since $f$ is $L$-smooth, we have \n$f(x^t)-f^\\star \\geq  \\frac{1}{2L}\\sqnorm{\\nabla f(x^t)}$\n\\cite[eq. 3.5]{bub15}, \nso that the condition $b\\gamma  \\leq \\frac{1}{2L}$ guarantees that $\\Psi^t$ is a valid Lyapunov function: we have $\\Psi^t\\geq 0$, and $\\Psi^t \\rightarrow 0$ implies $\\sqnorm{g^t}\\rightarrow 0$, $\\sqnorm{\\nabla f(x^t)}\\rightarrow 0$, and\n$f(x^t)-f^\\star \\rightarrow 0$.\n","sha256_utf8":"3adc4532dcced3c5cc5cc0c7f3e2199a1188735a5a1906da2daed0b519f27ba4"}]},{"id":"S007","semantic_type":"theorem","kind":"theorem","title":"PAGE线性收敛定理","source_units":[{"start":12854,"end":13728,"start_line":210,"end_line":233,"raw_tex":"\\begin{theorem}[linear convergence]\\label{theo1}\nSuppose that $f$ is $\\mu$-P{\\L} for some $\\mu>0$. \nIn \\algn{PAGE} with Lyapunov function \\eqref{eqly}, suppose that\n\\begin{equation*}\n\\gamma \\leq %\\min\\left(\n\\frac{1}{L\\left(1+\\sqrt{ \\frac{4\\tau}{L+\\tau}}\\sqrt{  \\frac{1-p}{p}}\\right)}, \n\\end{equation*}\nwith the additional requirement $\\gamma<\\frac{1}{L}$ if $\\tau=0$, \n\\begin{equation*}\na= 1-\\frac{\\gamma (L-\\tau)}{2}\\in \\left( \\frac{1}{2},1\\right], \\quad b= \\frac{\\gamma (L-\\tau)}{2}\\in \\left[0, \\frac{1}{2}\\right).\n\\end{equation*}\nThen $b\\gamma  < \\frac{1}{2L}$, and \n\\algn{PAGE}\nconverges linearly:  for every $t\\geq 0$, \n\\begin{equation*}\n\\Exp{\\Psi^{t}}\\leq \\rho^t \\Psi^0,\n\\end{equation*}\nwhere \n\\begin{equation*}\n\\rho\\coloneqq \\max\\left(1-(1-2b)\\gamma\\mu,1-p\\left(1-\\frac{1}{2a}\\right)\\right)<1.\n\\end{equation*}\nAlso, $\\Psi^t\\rightarrow 0$ \nalmost surely.\n\\end{theorem}","sha256_utf8":"ed48dc7cf4a123fa90acf78fe15d846d32730907dc59055fdfd46d9bda5abe54"}]},{"id":"S008","semantic_type":"assumption","kind":"assumption","title":"主定理的参数条件","source_units":[{"start":12903,"end":13387,"start_line":211,"end_line":220,"raw_tex":"Suppose that $f$ is $\\mu$-P{\\L} for some $\\mu>0$. \nIn \\algn{PAGE} with Lyapunov function \\eqref{eqly}, suppose that\n\\begin{equation*}\n\\gamma \\leq %\\min\\left(\n\\frac{1}{L\\left(1+\\sqrt{ \\frac{4\\tau}{L+\\tau}}\\sqrt{  \\frac{1-p}{p}}\\right)}, \n\\end{equation*}\nwith the additional requirement $\\gamma<\\frac{1}{L}$ if $\\tau=0$, \n\\begin{equation*}\na= 1-\\frac{\\gamma (L-\\tau)}{2}\\in \\left( \\frac{1}{2},1\\right], \\quad b= \\frac{\\gamma (L-\\tau)}{2}\\in \\left[0, \\frac{1}{2}\\right).\n\\end{equation*}\n","sha256_utf8":"5755325fc0f68db66c817d9a9de23b5d4fc4339d879384ddad6f3dedadb9554b"}]},{"id":"S009","semantic_type":"lemma","kind":"lemma","title":"弱凸梯度差不等式","source_units":[{"start":12257,"end":12608,"start_line":198,"end_line":203,"raw_tex":"\\begin{lemma}\\label{lemm2}\nLet $g:\\mathcal{X}\\rightarrow \\mathbb{R}$ be an $L$-smooth and $\\tau$-weakly convex function for some $L>0$ and $\\tau\\in [0,L]$. Then, for every $(x,y)\\in\\mathcal{X}^2$, \n\\begin{align*}\n\\sqnorm{\\nabla g(x)- \\nabla g(y)}&\\leq (L-\\tau)\\langle \\nabla g(x) - \\nabla g(y),x-y\\rangle + L\\tau \\sqnorm{x-y}.\n\\end{align*}\n\\end{lemma}","sha256_utf8":"04b6c3f67fae46ca1f7079ec87a213e45b1818b4d29c915e7ec35ec448c7f740"}]},{"id":"S010","semantic_type":"proof","kind":"proof","title":"梯度差引理完整证明","source_units":[{"start":24311,"end":25101,"start_line":413,"end_line":429,"raw_tex":"\\subsection{Proof of Lemma~\\ref{lemm2}}\n\nLet $(x,y)\\in\\mathcal{X}^2$. Since $g+\\frac{\\tau}{2}\\|\\cdot\\|^2$ is convex and $(L+\\tau)$-smooth, its gradient is $\\frac{1}{L+\\tau}$-cocoercive according to the Baillon--Haddad theorem \\cite[Corollary 18.17]{bau17}, so that\n\\begin{align*}\n\\sqnorm{\\nabla g(x)+\\tau x - \\nabla g(y)-\\tau y} &\\leq (L+\\tau)\\langle \\nabla g(x) +\\tau x- \\nabla g(y)-\\tau y,x-y\\rangle\\\\\n&=(L+\\tau)\\langle \\nabla g(x) - \\nabla g(y),x-y\\rangle +  (L+\\tau)\\tau \\sqnorm{x-y}.\n\\end{align*}\nHence,\n\\begin{align*}\n\\sqnorm{\\nabla g(x)- \\nabla g(y)}&=\\sqnorm{\\nabla g(x)+\\tau x- \\nabla g(y)-\\tau y}\\!-\\!2\\tau\n\\langle \\nabla g(x) - \\nabla g(y),x-y\\rangle \n\\!-\\!\\tau^2\\sqnorm{x-y}\\\\\n&\\leq (L-\\tau)\\langle \\nabla g(x) - \\nabla g(y),x-y\\rangle + L\\tau \\sqnorm{x-y}.\n\\end{align*}\\qed\n\n\n\n","sha256_utf8":"70dc77bc9944cd09444770f187fde753598dbb1ed100372b52e9a27c4e771ae6"}]},{"id":"S011","semantic_type":"proof_step","kind":"proof_step","title":"凸化后应用余强单调性","source_units":[{"start":24311,"end":24813,"start_line":413,"end_line":419,"raw_tex":"\\subsection{Proof of Lemma~\\ref{lemm2}}\n\nLet $(x,y)\\in\\mathcal{X}^2$. Since $g+\\frac{\\tau}{2}\\|\\cdot\\|^2$ is convex and $(L+\\tau)$-smooth, its gradient is $\\frac{1}{L+\\tau}$-cocoercive according to the Baillon--Haddad theorem \\cite[Corollary 18.17]{bau17}, so that\n\\begin{align*}\n\\sqnorm{\\nabla g(x)+\\tau x - \\nabla g(y)-\\tau y} &\\leq (L+\\tau)\\langle \\nabla g(x) +\\tau x- \\nabla g(y)-\\tau y,x-y\\rangle\\\\\n&=(L+\\tau)\\langle \\nabla g(x) - \\nabla g(y),x-y\\rangle +  (L+\\tau)\\tau \\sqnorm{x-y}.\n\\end{align*}\n","sha256_utf8":"e34f0f413c74a2fe74ad3988c13999ded6bb75cf0401715b8578610c59d2bf02"}]},{"id":"S012","semantic_type":"proof_step","kind":"proof_step","title":"展开平方得到梯度界","source_units":[{"start":24813,"end":25098,"start_line":420,"end_line":426,"raw_tex":"Hence,\n\\begin{align*}\n\\sqnorm{\\nabla g(x)- \\nabla g(y)}&=\\sqnorm{\\nabla g(x)+\\tau x- \\nabla g(y)-\\tau y}\\!-\\!2\\tau\n\\langle \\nabla g(x) - \\nabla g(y),x-y\\rangle \n\\!-\\!\\tau^2\\sqnorm{x-y}\\\\\n&\\leq (L-\\tau)\\langle \\nabla g(x) - \\nabla g(y),x-y\\rangle + L\\tau \\sqnorm{x-y}.\n\\end{align*}\\qed\n","sha256_utf8":"7baf0521fd6f56bfedbc4d628f4537f68f902999f0396f7bccbc186044864da6"}]},{"id":"S013","semantic_type":"proof","kind":"proof","title":"线性收敛完整证明","source_units":[{"start":25101,"end":34733,"start_line":430,"end_line":578,"raw_tex":"\\subsection{Proof of Theorem~\\ref{theo1}}\\label{secproof1}\n\nFor every $t\\geq 0$, let $\\mathcal{F}^t$ denote the $\\sigma$-algebra generated by the random variables $x^0,g^0,\\ldots, x^t,g^t$. For the moment, assume that $a$ and $b$ in \\eqref{eqly} satisfy  $a>0$, \n$0\\leq b\\leq \\frac{1}{2}$, $b\\gamma  \\leq \\frac{1}{2L}$, without additional restrictions.\n\n\nLet $t\\geq 0$. We now derive upper bounds for the terms in $\\Exp{\\Psi^{t+1}\\;|\\;\\mathcal{F}^t}$.\n\n\\noindent \\textbf{First}, we have the descent property \\cite[Lemma 4]{ric21}\n\\begin{align*}\nf(x^{t+1}) -f^\\star &\\leq f(x^t)  -f^\\star -\\frac{\\gamma}{2} \\sqnorm{\\nabla f(x^t)} +\\frac{ \\gamma }{2}\\sqnorm{g^{t}-\\nabla f(x^t)}+ \\left(\\frac{ L}{2}-\\frac{1}{2\\gamma}\\right)\\sqnorm{x^{t+1}-x^t}\\\\\n&=f(x^t)  -f^\\star -\\frac{\\gamma}{2} \\sqnorm{\\nabla f(x^t)} +\\frac{ \\gamma }{2}\\sqnorm{g^{t}-\\nabla f(x^t)}+ \\left(\\frac{ L\\gamma^2}{2}-\\frac{\\gamma}{2}\\right)\\sqnorm{g^t}.\n\\end{align*}\n\\textbf{Second}, we have \n\\begin{align*}\n\\Exp{\\sqnorm{g^{t+1}-\\nabla f(x^{t+1})}\\;|\\;\\mathcal{F}^t} &= (1-p)\\Exp{\\sqnorm{g^{t+1}-\\nabla f(x^{t+1})}\\;|\\;\\theta^t=0,\\mathcal{F}^t},\n\\end{align*}\nand by decomposing \nthe expected squared norm into its squared mean and variance, we obtain\n\\begin{align*}\n&\\Exp{\\sqnorm{g^{t+1}-\\nabla f(x^{t+1})}\\;|\\;\\mathcal{F}^t} = (1-p)\\sqnorm{g^{t}-\\nabla f(x^t)}\\\\\n&\\qquad +\\frac{1-p}{n}\\sum_{i=1}^n\\sqnorm{\\nabla f_i(x^{t+1})-\\nabla f_i(x^{t})-\\nabla f(x^{t+1})+\\nabla f(x^t)}.\n\\end{align*}\n\\textbf{Third}, we have\n\\begin{align*}\n\\Exp{\\sqnorm{g^{t+1}}\\;|\\;\\mathcal{F}^t} &= p \\sqnorm{\\nabla f(x^{t+1})}+ (1-p)\\Exp{\\sqnorm{g^{t+1}}\\;|\\;\\theta^t=0,\\mathcal{F}^t}\\\\\n&=p \\sqnorm{\\nabla f(x^{t+1})}+ (1-p)\\sqnorm{g^{t}+\\nabla f(x^{t+1})-\\nabla f(x^t)}\\\\\n&\\quad +\\frac{1-p}{n}\\sum_{i=1}^n\\sqnorm{\\nabla f_i(x^{t+1})-\\nabla f_i(x^{t})-\\nabla f(x^{t+1})+\\nabla f(x^t)}.\n\\end{align*}\nUsing the fact that $g^t=-\\frac{1}{\\gamma}(x^{t+1}-x^t)$, we obtain\n\\begin{align*}\n\\sqnorm{g^{t}+\\nabla f(x^{t+1})-\\nabla f(x^t)} &= \\sqnorm{g^{t}} + \\sqnorm{\\nabla f(x^{t+1})-\\nabla f(x^t)} +2\\!\\left\\langle g^{t},\\nabla f(x^{t+1})-\\nabla f(x^t)\\right\\rangle\\\\\n&=\\sqnorm{g^{t}} + \\sqnorm{\\nabla f(x^{t+1})-\\nabla f(x^t)}\\\\\n&\\quad -\\frac{2}{\\gamma}\\left\\langle x^{t+1}-x^t,\\nabla f(x^{t+1})-\\nabla f(x^t)\\right\\rangle.\n\\end{align*}\nThe negative inner product $-\\frac{2}{\\gamma}\\left\\langle x^{t+1}-x^t,\\nabla f(x^{t+1})-\\nabla f(x^t)\\right\\rangle$ that appears is crucial, as it enables the cancellation of positive terms using Lemma~\\ref{lemm2}.\n\nSince $f=\\frac{1}{n}\\sum_{i=1}^n f_i$, we have\n\\begin{align*}\n&\\frac{1}{n}\\sum_{i=1}^n\\sqnorm{\\nabla f_i(x^{t+1})-\\nabla f_i(x^{t})-\\nabla f(x^{t+1})+\\nabla f(x^t)}\\\\\n&\\quad= \\frac{1}{n}\\sum_{i=1}^n\\sqnorm{\\nabla f_i(x^{t+1})-\\nabla f_i(x^{t})} -\\sqnorm{\\nabla f(x^{t+1})-\\nabla f(x^t)}.\n\\end{align*}\nTherefore,\n\\begin{align*}\n\\Exp{\\sqnorm{g^{t+1}}\\;|\\;\\mathcal{F}^t} &=p \\sqnorm{\\nabla f(x^{t+1})}+ (1-p)\\sqnorm{g^{t}}\\\\\n&\\quad -\\frac{2(1-p)}{\\gamma}\\left\\langle x^{t+1}-x^t,\\nabla f(x^{t+1})-\\nabla f(x^t)\\right\\rangle\\\\\n &\\quad +\\frac{1-p}{n}\\sum_{i=1}^n\\sqnorm{\\nabla f_i(x^{t+1})-\\nabla f_i(x^{t})}.\n\\end{align*}\n\nMoreover, by Lemma~\\ref{lemm2} and the smoothness and weak convexity of the functions $f_i$, we obtain\n\\begin{align*}\n\\frac{1}{n}\\sum_{i=1}^n \\sqnorm{\\nabla f_i(x^{t+1})- \\nabla f_i(x^t)}&\\leq\\frac{1}{n}\\sum_{i=1}^n \\Big( (L-\\tau)\\big\\langle \\nabla f_i(x^{t+1}) - \\nabla f_i(x^t),x^{t+1}-x^t\\big\\rangle \\\\\n&\\quad\\left.{}+ L\\tau \\sqnorm{x^{t+1}-x^t}\\right)\\\\\n&= (L-\\tau)\\big\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\big\\rangle \\\\\n&\\quad+ L\\tau \\sqnorm{x^{t+1}-x^t}.\n\\end{align*}\nThe inner products $(L-\\tau)\\big\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\big\\rangle$ will be canceled using the negative inner product $-\\frac{2}{\\gamma}\\left\\langle x^{t+1}-x^t,\\nabla f(x^{t+1})-\\nabla f(x^t)\\right\\rangle$ that appeared above, while the terms \n$\\sqnorm{x^{t+1}-x^t}$ will be canceled by choosing $\\gamma$ sufficiently small and using the negative term $(\\frac{ L}{2}-\\frac{1}{2\\gamma})\\sqnorm{x^{t+1}-x^t}$ arising in the descent property.\n\nThus, \n\\begin{align*}\n\\Exp{\\sqnorm{g^{t+1}-\\nabla f(x^{t+1})}\\;|\\;\\mathcal{F}^t} \n&\\leq (1-p)\\sqnorm{g^{t}-\\nabla f(x^t)}\\!+\\!\n\\frac{1-p}{n}\\sum_{i=1}^n \\sqnorm{\\nabla f_i(x^{t+1})- \\nabla f_i(x^t)}\\\\\n&\\leq (1-p)\\sqnorm{g^{t}-\\nabla f(x^t)}+ (1-p)L\\tau \\sqnorm{x^{t+1}-x^t}\\\\\n&\\quad +(1-p)(L-\\tau)  \\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle\n\\end{align*}\nand\n\\begin{align*}\n\\Exp{\\sqnorm{g^{t+1}}\\;|\\;\\mathcal{F}^t} \n&\\leq p \\sqnorm{\\nabla f(x^{t+1})}+ (1-p)\\sqnorm{g^{t}}\\\\\n&\\quad -\\frac{2(1-p)}{\\gamma}\\left\\langle \\nabla f(x^{t+1})-\\nabla f(x^t),x^{t+1}-x^t\\right\\rangle\\\\\n &\\quad +(1-p)(L-\\tau)\\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle\\\\\n&\\quad +(1-p)L\\tau \\sqnorm{x^{t+1}-x^t}\\\\\n &= p\\sqnorm{\\nabla f(x^{t+1})}+ (1-p)\\big(1+L\\tau\\gamma^2\\big)\n\\sqnorm{g^{t}} \\\\\n&\\quad +(1-p)\\left(L- \\tau-\\frac{2}{\\gamma}\\right) \\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle .\n\\end{align*}\nHence, combining the inequalities for these terms, we obtain\n\\begin{align*}\n\\Exp{\\Psi^{t+1}\\;|\\;\\mathcal{F}^t}&\\leq\nf(x^t)  -f^\\star -\\frac{\\gamma}{2} \\sqnorm{\\nabla f(x^t)} +\\frac{ \\gamma }{2}\\sqnorm{g^{t}-\\nabla f(x^t)}+ \\left(\\frac{ L\\gamma^2}{2}-\\frac{\\gamma}{2}\\right)\\sqnorm{g^t} \\\\\n&\\quad -b\\gamma\\sqnorm{\\nabla f(x^{t+1})}\n+\\frac{a\\gamma}{p} (1-p)\\sqnorm{g^{t}-\\nabla f(x^t)}\n+ a\\gamma^3 L\\tau \\frac{1-p}{p}\\sqnorm{g^t}\\\\\n&\\quad+a\\gamma (L-\\tau) \\frac{1-p}{p}\\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle+b\\gamma\\sqnorm{\\nabla f(x^{t+1})}\\notag\\\\\n&\\quad+\\frac{b\\gamma}{p}  (1-p)\\big(1+L\\tau\\gamma^2\\big)\\sqnorm{g^{t}}\\\\\n&\\quad+ \\big(b\\gamma (L-\\tau)-2b\\big) \\frac{1-p}{p}\\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle\\notag\\\\\n&= f(x^t)  -f^\\star  -\\frac{\\gamma}{2} \\sqnorm{\\nabla f(x^t)}+\\frac{a\\gamma}{p}\\left(1-p\\left(1-\\frac{1 }{2a}\\right)  \\right)\\sqnorm{g^{t}-\\nabla f(x^t)}\\notag\\\\\n&\\quad+\\frac{b\\gamma}{p}  (1-p)\\sqnorm{g^{t}}\n+\\frac{\\gamma}{2}\\left(L\\gamma-1+ 2(a+b)\\gamma^2 L\\tau \\frac{1-p}{p}\\right)\\sqnorm{g^{t}}\\notag\\\\\n&\\quad+ \\big((a+b)\\gamma (L-\\tau)-2b\\big) \\frac{1-p}{p}\\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle.\n\\end{align*}\nTo cancel the inner product, we set\n\\begin{equation}\nb\\coloneqq \\frac{a\\gamma (L-\\tau)}{2-\\gamma(L-\\tau)},\\label{eqbb}\n\\end{equation}\nso that $(a+b)\\gamma (L-\\tau)-2b=0$. Note that $\\gamma\\leq \\frac{1}{L}$ implies $2-\\gamma(L-\\tau)\\geq 1$. Moreover,\na sufficient condition for $L\\gamma-1+ 2(a+b)\\gamma^2 L\\tau \\frac{1-p}{p}\\leq 0$ is  \\cite[Lemma 5]{ric21}\n\\begin{equation}\n\\gamma \\leq \\frac{1}{L+\\sqrt{2(a+b)L\\tau \\frac{1-p}{p}}}.\\label{eqlllo}\n\\end{equation}\nTo avoid the dependence of $b$ on $\\gamma$, we derive from $\\gamma\\leq \\frac{1}{L}$ that \n\\begin{equation*}\nb\\leq  \\frac{a (L-\\tau)}{L+\\tau},%\\label{eqww}\n\\end{equation*}\nso that $a+b\\leq  \\frac{2aL}{L+\\tau}$. Therefore, a sufficient condition for \\eqref{eqlllo} is\n\\begin{equation}\n\\gamma \\leq \\frac{1}{L+\\sqrt{ \\frac{4aL}{L+\\tau}L\\tau \\frac{1-p}{p}}}=\\frac{1}{L\\left(1+\\sqrt{ \\frac{4a\\tau}{L+\\tau}}\\sqrt{  \\frac{1-p}{p}}\\right)}.\\label{eqlllo2}\n\\end{equation}\nIf we additionally assume $a\\leq 1$,  the dependence on $a$ can be eliminated with the sufficient condition \n\\begin{equation*}\n\\gamma \\leq \\frac{1}{L\\left(1+\\sqrt{ \\frac{4\\tau}{L+\\tau}}\\sqrt{  \\frac{1-p}{p}}\\right)},\n\\end{equation*}\nas stated in the theorem.\n\nTherefore, since the sufficient conditions above are supposed to hold, we have\n\\begin{align*}\n\\Exp{\\Psi^{t+1}\\;|\\;\\mathcal{F}^t}&\\leq\n f(x^t)  -f^\\star  -\\frac{\\gamma}{2} \\sqnorm{\\nabla f(x^t)}+\\frac{a\\gamma}{p}\\left(1-p\\left(1-\\frac{1 }{2a}\\right)  \\right)\\sqnorm{g^{t}-\\nabla f(x^t)}\\\\\n&\\quad+\\frac{b\\gamma}{p}  (1-p)\\sqnorm{g^{t}}\\\\\n&= f(x^t)  -f^\\star  -\\gamma\\left(\\frac{1}{2}-b\\right)\\sqnorm{\\nabla f(x^t)} -b\\gamma \\sqnorm{\\nabla f(x^t)}\\\\\n&\\quad+\\frac{a\\gamma}{p}\\left(1-p\\left(1-\\frac{1}{2a}\\right)  \\right)\\sqnorm{g^{t}-\\nabla f(x^t)}+\\frac{b\\gamma}{p}  (1-p)\\sqnorm{g^{t}}.\n\\end{align*}\nTo obtain a contraction, we need $a>\\frac{1}{2}$ and $b<\\frac{1}{2}$. The choice \n\\begin{equation*}\na= 1-\\frac{\\gamma (L-\\tau)}{2}, \\quad b= \\frac{\\gamma (L-\\tau)}{2},\n\\end{equation*}\nwith the additional condition $\\gamma<\\frac{1}{L}$ if $\\tau=0$, satisfies  these two inequalities, as well as \\eqref{eqbb}, $a\\leq 1$, and $b \\gamma \\leq \\frac{1}{2L}$.\n\nFinally, under the assumption that $f$ is $\\mu$-P{\\L}, we have\n\\begin{equation*}\n\\|\\nabla f(x^t)\\|^2\\geq 2\\mu\\big(f(x^t)-f^\\star\\big).\n\\end{equation*}\nTherefore,\n\\begin{align}\n\\Exp{\\Psi^{t+1}\\;|\\;\\mathcal{F}^t}&\\leq  \\big(1-(1-2b)\\gamma\\mu\\big)\\big(f(x^t)  -f^\\star\\big)  -b\\gamma \\sqnorm{\\nabla f(x^t)}\\notag\\\\\n&\\quad+\\frac{a\\gamma}{p}\\left(1-p\\left(1-\\frac{1}{2a}\\right)  \\right)\\sqnorm{g^{t}-\\nabla f(x^t)}+\\frac{b\\gamma}{p}  (1-p)\\sqnorm{g^{t}}\\notag\\\\\n&\\leq  \\big(1-(1-2b)\\gamma\\mu\\big)\\big(f(x^t)  -f^\\star  -b\\gamma \\sqnorm{\\nabla f(x^t)}\\big)\\notag\\\\\n&\\quad+\\frac{a\\gamma}{p}\\left(1-p\\left(1-\\frac{1}{2a}\\right)  \\right)\\sqnorm{g^{t}-\\nabla f(x^t)}+\\frac{b\\gamma}{p}  (1-p)\\sqnorm{g^{t}}\\notag\\\\\n&\\leq \\max\\left(1-(1-2b)\\gamma\\mu,1-p\\left(1-\\frac{1}{2a}\\right),1-p\\right)\\Psi^t\\notag\\\\\n&= \\max\\left(1-(1-2b)\\gamma\\mu,1-p\\left(1-\\frac{1}{2a}\\right)\\right)\\Psi^t.\\label{eqrec2b}\n\\end{align}\nUsing the tower rule of expectations, we can unroll the recursion in \\eqref{eqrec2b} to obtain the unconditional expectation of $\\Psi^{t+1}$. Moreover, using classical results on supermartingale convergence \\cite[Proposition A.4.5]{ber15}, it follows from \\eqref{eqrec2b} that $\\Psi^t\\rightarrow 0$ almost surely. \\qed\n%\\hfill$\\square$\n\n\n\n\n","sha256_utf8":"52cfb2b7ef2719d46f95aec5be7a02c456db795aa9c23da03ba509235d390441"}]},{"id":"S014","semantic_type":"proof_step","kind":"proof_step","title":"建立滤过与临时参数范围","source_units":[{"start":25101,"end":25554,"start_line":430,"end_line":437,"raw_tex":"\\subsection{Proof of Theorem~\\ref{theo1}}\\label{secproof1}\n\nFor every $t\\geq 0$, let $\\mathcal{F}^t$ denote the $\\sigma$-algebra generated by the random variables $x^0,g^0,\\ldots, x^t,g^t$. For the moment, assume that $a$ and $b$ in \\eqref{eqly} satisfy  $a>0$, \n$0\\leq b\\leq \\frac{1}{2}$, $b\\gamma  \\leq \\frac{1}{2L}$, without additional restrictions.\n\n\nLet $t\\geq 0$. We now derive upper bounds for the terms in $\\Exp{\\Psi^{t+1}\\;|\\;\\mathcal{F}^t}$.\n\n","sha256_utf8":"82d00cb28553d8330b195d0ca014617b074a68235caaefac52c7fe9cfc1f9848"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"目标函数的单步下降","source_units":[{"start":25554,"end":26031,"start_line":438,"end_line":442,"raw_tex":"\\noindent \\textbf{First}, we have the descent property \\cite[Lemma 4]{ric21}\n\\begin{align*}\nf(x^{t+1}) -f^\\star &\\leq f(x^t)  -f^\\star -\\frac{\\gamma}{2} \\sqnorm{\\nabla f(x^t)} +\\frac{ \\gamma }{2}\\sqnorm{g^{t}-\\nabla f(x^t)}+ \\left(\\frac{ L}{2}-\\frac{1}{2\\gamma}\\right)\\sqnorm{x^{t+1}-x^t}\\\\\n&=f(x^t)  -f^\\star -\\frac{\\gamma}{2} \\sqnorm{\\nabla f(x^t)} +\\frac{ \\gamma }{2}\\sqnorm{g^{t}-\\nabla f(x^t)}+ \\left(\\frac{ L\\gamma^2}{2}-\\frac{\\gamma}{2}\\right)\\sqnorm{g^t}.\n\\end{align*}\n","sha256_utf8":"61df050982d54052b7d525d22ff0120cbdb0f6819eb0ed75fceb971f29a85de4"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"估计误差的条件方差分解","source_units":[{"start":26031,"end":26555,"start_line":443,"end_line":452,"raw_tex":"\\textbf{Second}, we have \n\\begin{align*}\n\\Exp{\\sqnorm{g^{t+1}-\\nabla f(x^{t+1})}\\;|\\;\\mathcal{F}^t} &= (1-p)\\Exp{\\sqnorm{g^{t+1}-\\nabla f(x^{t+1})}\\;|\\;\\theta^t=0,\\mathcal{F}^t},\n\\end{align*}\nand by decomposing \nthe expected squared norm into its squared mean and variance, we obtain\n\\begin{align*}\n&\\Exp{\\sqnorm{g^{t+1}-\\nabla f(x^{t+1})}\\;|\\;\\mathcal{F}^t} = (1-p)\\sqnorm{g^{t}-\\nabla f(x^t)}\\\\\n&\\qquad +\\frac{1-p}{n}\\sum_{i=1}^n\\sqnorm{\\nabla f_i(x^{t+1})-\\nabla f_i(x^{t})-\\nabla f(x^{t+1})+\\nabla f(x^t)}.\n\\end{align*}\n","sha256_utf8":"b0cef8dda365c0abab5875c35ed1d153248f41de81541bb69e22a1062845e661"}]},{"id":"S017","semantic_type":"proof_step","kind":"proof_step","title":"估计量二阶矩中的负内积","source_units":[{"start":26555,"end":27585,"start_line":453,"end_line":465,"raw_tex":"\\textbf{Third}, we have\n\\begin{align*}\n\\Exp{\\sqnorm{g^{t+1}}\\;|\\;\\mathcal{F}^t} &= p \\sqnorm{\\nabla f(x^{t+1})}+ (1-p)\\Exp{\\sqnorm{g^{t+1}}\\;|\\;\\theta^t=0,\\mathcal{F}^t}\\\\\n&=p \\sqnorm{\\nabla f(x^{t+1})}+ (1-p)\\sqnorm{g^{t}+\\nabla f(x^{t+1})-\\nabla f(x^t)}\\\\\n&\\quad +\\frac{1-p}{n}\\sum_{i=1}^n\\sqnorm{\\nabla f_i(x^{t+1})-\\nabla f_i(x^{t})-\\nabla f(x^{t+1})+\\nabla f(x^t)}.\n\\end{align*}\nUsing the fact that $g^t=-\\frac{1}{\\gamma}(x^{t+1}-x^t)$, we obtain\n\\begin{align*}\n\\sqnorm{g^{t}+\\nabla f(x^{t+1})-\\nabla f(x^t)} &= \\sqnorm{g^{t}} + \\sqnorm{\\nabla f(x^{t+1})-\\nabla f(x^t)} +2\\!\\left\\langle g^{t},\\nabla f(x^{t+1})-\\nabla f(x^t)\\right\\rangle\\\\\n&=\\sqnorm{g^{t}} + \\sqnorm{\\nabla f(x^{t+1})-\\nabla f(x^t)}\\\\\n&\\quad -\\frac{2}{\\gamma}\\left\\langle x^{t+1}-x^t,\\nabla f(x^{t+1})-\\nabla f(x^t)\\right\\rangle.\n\\end{align*}\nThe negative inner product $-\\frac{2}{\\gamma}\\left\\langle x^{t+1}-x^t,\\nabla f(x^{t+1})-\\nabla f(x^t)\\right\\rangle$ that appears is crucial, as it enables the cancellation of positive terms using Lemma~\\ref{lemm2}.\n","sha256_utf8":"e0aad70a3c4c68852ce6f2f61b3d6e7b724c25cf13d480f67a27b053c3126e30"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"消去平均梯度差平方项","source_units":[{"start":27586,"end":28204,"start_line":467,"end_line":477,"raw_tex":"Since $f=\\frac{1}{n}\\sum_{i=1}^n f_i$, we have\n\\begin{align*}\n&\\frac{1}{n}\\sum_{i=1}^n\\sqnorm{\\nabla f_i(x^{t+1})-\\nabla f_i(x^{t})-\\nabla f(x^{t+1})+\\nabla f(x^t)}\\\\\n&\\quad= \\frac{1}{n}\\sum_{i=1}^n\\sqnorm{\\nabla f_i(x^{t+1})-\\nabla f_i(x^{t})} -\\sqnorm{\\nabla f(x^{t+1})-\\nabla f(x^t)}.\n\\end{align*}\nTherefore,\n\\begin{align*}\n\\Exp{\\sqnorm{g^{t+1}}\\;|\\;\\mathcal{F}^t} &=p \\sqnorm{\\nabla f(x^{t+1})}+ (1-p)\\sqnorm{g^{t}}\\\\\n&\\quad -\\frac{2(1-p)}{\\gamma}\\left\\langle x^{t+1}-x^t,\\nabla f(x^{t+1})-\\nabla f(x^t)\\right\\rangle\\\\\n &\\quad +\\frac{1-p}{n}\\sum_{i=1}^n\\sqnorm{\\nabla f_i(x^{t+1})-\\nabla f_i(x^{t})}.\n\\end{align*}\n","sha256_utf8":"f1af693dedb506540200f8fc9521e43125df3ea59936af4e22c4d793f9f3d070"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"应用弱凸梯度界安排抵消","source_units":[{"start":28205,"end":29166,"start_line":479,"end_line":487,"raw_tex":"Moreover, by Lemma~\\ref{lemm2} and the smoothness and weak convexity of the functions $f_i$, we obtain\n\\begin{align*}\n\\frac{1}{n}\\sum_{i=1}^n \\sqnorm{\\nabla f_i(x^{t+1})- \\nabla f_i(x^t)}&\\leq\\frac{1}{n}\\sum_{i=1}^n \\Big( (L-\\tau)\\big\\langle \\nabla f_i(x^{t+1}) - \\nabla f_i(x^t),x^{t+1}-x^t\\big\\rangle \\\\\n&\\quad\\left.{}+ L\\tau \\sqnorm{x^{t+1}-x^t}\\right)\\\\\n&= (L-\\tau)\\big\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\big\\rangle \\\\\n&\\quad+ L\\tau \\sqnorm{x^{t+1}-x^t}.\n\\end{align*}\nThe inner products $(L-\\tau)\\big\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\big\\rangle$ will be canceled using the negative inner product $-\\frac{2}{\\gamma}\\left\\langle x^{t+1}-x^t,\\nabla f(x^{t+1})-\\nabla f(x^t)\\right\\rangle$ that appeared above, while the terms \n$\\sqnorm{x^{t+1}-x^t}$ will be canceled by choosing $\\gamma$ sufficiently small and using the negative term $(\\frac{ L}{2}-\\frac{1}{2\\gamma})\\sqnorm{x^{t+1}-x^t}$ arising in the descent property.\n","sha256_utf8":"1f67ea0132aa21d5b3950ddeff024d4f17e1c249f1e3e7ad80fbc2a7ef2ee3fd"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"控制下一步估计误差","source_units":[{"start":29167,"end":29551,"start_line":489,"end_line":496,"raw_tex":"Thus, \n\\begin{align*}\n\\Exp{\\sqnorm{g^{t+1}-\\nabla f(x^{t+1})}\\;|\\;\\mathcal{F}^t} \n&\\leq (1-p)\\sqnorm{g^{t}-\\nabla f(x^t)}\\!+\\!\n\\frac{1-p}{n}\\sum_{i=1}^n \\sqnorm{\\nabla f_i(x^{t+1})- \\nabla f_i(x^t)}\\\\\n&\\leq (1-p)\\sqnorm{g^{t}-\\nabla f(x^t)}+ (1-p)L\\tau \\sqnorm{x^{t+1}-x^t}\\\\\n&\\quad +(1-p)(L-\\tau)  \\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle\n\\end{align*}\n","sha256_utf8":"2ea5abcb7e4b6b8fbef2c25de5e135e5f47e907be25732a131d27a47e81bc60c"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"控制下一步估计量范数","source_units":[{"start":29551,"end":30129,"start_line":497,"end_line":507,"raw_tex":"and\n\\begin{align*}\n\\Exp{\\sqnorm{g^{t+1}}\\;|\\;\\mathcal{F}^t} \n&\\leq p \\sqnorm{\\nabla f(x^{t+1})}+ (1-p)\\sqnorm{g^{t}}\\\\\n&\\quad -\\frac{2(1-p)}{\\gamma}\\left\\langle \\nabla f(x^{t+1})-\\nabla f(x^t),x^{t+1}-x^t\\right\\rangle\\\\\n &\\quad +(1-p)(L-\\tau)\\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle\\\\\n&\\quad +(1-p)L\\tau \\sqnorm{x^{t+1}-x^t}\\\\\n &= p\\sqnorm{\\nabla f(x^{t+1})}+ (1-p)\\big(1+L\\tau\\gamma^2\\big)\n\\sqnorm{g^{t}} \\\\\n&\\quad +(1-p)\\left(L- \\tau-\\frac{2}{\\gamma}\\right) \\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle .\n\\end{align*}\n","sha256_utf8":"839ab6e044d32c37b25bcf421fdb11f6ba4565d0c35fe5a93987e85169451589"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"合并势函数各项条件期望","source_units":[{"start":30129,"end":31361,"start_line":508,"end_line":522,"raw_tex":"Hence, combining the inequalities for these terms, we obtain\n\\begin{align*}\n\\Exp{\\Psi^{t+1}\\;|\\;\\mathcal{F}^t}&\\leq\nf(x^t)  -f^\\star -\\frac{\\gamma}{2} \\sqnorm{\\nabla f(x^t)} +\\frac{ \\gamma }{2}\\sqnorm{g^{t}-\\nabla f(x^t)}+ \\left(\\frac{ L\\gamma^2}{2}-\\frac{\\gamma}{2}\\right)\\sqnorm{g^t} \\\\\n&\\quad -b\\gamma\\sqnorm{\\nabla f(x^{t+1})}\n+\\frac{a\\gamma}{p} (1-p)\\sqnorm{g^{t}-\\nabla f(x^t)}\n+ a\\gamma^3 L\\tau \\frac{1-p}{p}\\sqnorm{g^t}\\\\\n&\\quad+a\\gamma (L-\\tau) \\frac{1-p}{p}\\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle+b\\gamma\\sqnorm{\\nabla f(x^{t+1})}\\notag\\\\\n&\\quad+\\frac{b\\gamma}{p}  (1-p)\\big(1+L\\tau\\gamma^2\\big)\\sqnorm{g^{t}}\\\\\n&\\quad+ \\big(b\\gamma (L-\\tau)-2b\\big) \\frac{1-p}{p}\\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle\\notag\\\\\n&= f(x^t)  -f^\\star  -\\frac{\\gamma}{2} \\sqnorm{\\nabla f(x^t)}+\\frac{a\\gamma}{p}\\left(1-p\\left(1-\\frac{1 }{2a}\\right)  \\right)\\sqnorm{g^{t}-\\nabla f(x^t)}\\notag\\\\\n&\\quad+\\frac{b\\gamma}{p}  (1-p)\\sqnorm{g^{t}}\n+\\frac{\\gamma}{2}\\left(L\\gamma-1+ 2(a+b)\\gamma^2 L\\tau \\frac{1-p}{p}\\right)\\sqnorm{g^{t}}\\notag\\\\\n&\\quad+ \\big((a+b)\\gamma (L-\\tau)-2b\\big) \\frac{1-p}{p}\\left\\langle \\nabla f(x^{t+1}) - \\nabla f(x^t),x^{t+1}-x^t\\right\\rangle.\n\\end{align*}\n","sha256_utf8":"7ab5b1a278c79e225ebafa625e22c9167486df84a6da7d6559aac3bf07bb1678"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"选取权重消去内积","source_units":[{"start":31361,"end":31611,"start_line":523,"end_line":527,"raw_tex":"To cancel the inner product, we set\n\\begin{equation}\nb\\coloneqq \\frac{a\\gamma (L-\\tau)}{2-\\gamma(L-\\tau)},\\label{eqbb}\n\\end{equation}\nso that $(a+b)\\gamma (L-\\tau)-2b=0$. Note that $\\gamma\\leq \\frac{1}{L}$ implies $2-\\gamma(L-\\tau)\\geq 1$. Moreover,\n","sha256_utf8":"12d0da7e442495c25a940a995d43ec8ed465c9ca5e05ea1aa39133af59e02d04"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"步长保证范数项非正","source_units":[{"start":31611,"end":31822,"start_line":528,"end_line":531,"raw_tex":"a sufficient condition for $L\\gamma-1+ 2(a+b)\\gamma^2 L\\tau \\frac{1-p}{p}\\leq 0$ is  \\cite[Lemma 5]{ric21}\n\\begin{equation}\n\\gamma \\leq \\frac{1}{L+\\sqrt{2(a+b)L\\tau \\frac{1-p}{p}}}.\\label{eqlllo}\n\\end{equation}\n","sha256_utf8":"9a14f0c503160b591ea2f653ba9540fa1153be765b6da422d6bffc759069e27e"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"消除步长界中的隐式权重","source_units":[{"start":31822,"end":32284,"start_line":532,"end_line":539,"raw_tex":"To avoid the dependence of $b$ on $\\gamma$, we derive from $\\gamma\\leq \\frac{1}{L}$ that \n\\begin{equation*}\nb\\leq  \\frac{a (L-\\tau)}{L+\\tau},%\\label{eqww}\n\\end{equation*}\nso that $a+b\\leq  \\frac{2aL}{L+\\tau}$. Therefore, a sufficient condition for \\eqref{eqlllo} is\n\\begin{equation}\n\\gamma \\leq \\frac{1}{L+\\sqrt{ \\frac{4aL}{L+\\tau}L\\tau \\frac{1-p}{p}}}=\\frac{1}{L\\left(1+\\sqrt{ \\frac{4a\\tau}{L+\\tau}}\\sqrt{  \\frac{1-p}{p}}\\right)}.\\label{eqlllo2}\n\\end{equation}\n","sha256_utf8":"876152ad7205b41a6699a39fdf598c4118267418f4a0adc282cbd8c85db4d20b"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"得到定理中的显式步长界","source_units":[{"start":32284,"end":32543,"start_line":540,"end_line":544,"raw_tex":"If we additionally assume $a\\leq 1$,  the dependence on $a$ can be eliminated with the sufficient condition \n\\begin{equation*}\n\\gamma \\leq \\frac{1}{L\\left(1+\\sqrt{ \\frac{4\\tau}{L+\\tau}}\\sqrt{  \\frac{1-p}{p}}\\right)},\n\\end{equation*}\nas stated in the theorem.\n","sha256_utf8":"5c9dd9b190fd04eff23ae90ebcac16205e8e474afe0f1a3ee4aeb97c6d1882f2"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"化简势函数单步上界","source_units":[{"start":32544,"end":33142,"start_line":546,"end_line":553,"raw_tex":"Therefore, since the sufficient conditions above are supposed to hold, we have\n\\begin{align*}\n\\Exp{\\Psi^{t+1}\\;|\\;\\mathcal{F}^t}&\\leq\n f(x^t)  -f^\\star  -\\frac{\\gamma}{2} \\sqnorm{\\nabla f(x^t)}+\\frac{a\\gamma}{p}\\left(1-p\\left(1-\\frac{1 }{2a}\\right)  \\right)\\sqnorm{g^{t}-\\nabla f(x^t)}\\\\\n&\\quad+\\frac{b\\gamma}{p}  (1-p)\\sqnorm{g^{t}}\\\\\n&= f(x^t)  -f^\\star  -\\gamma\\left(\\frac{1}{2}-b\\right)\\sqnorm{\\nabla f(x^t)} -b\\gamma \\sqnorm{\\nabla f(x^t)}\\\\\n&\\quad+\\frac{a\\gamma}{p}\\left(1-p\\left(1-\\frac{1}{2a}\\right)  \\right)\\sqnorm{g^{t}-\\nabla f(x^t)}+\\frac{b\\gamma}{p}  (1-p)\\sqnorm{g^{t}}.\n\\end{align*}\n","sha256_utf8":"a8892c0009af0776cc1007fb87786dcac041150c1d49949c5c39b69bdd40ee57"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"验证收缩所需权重条件","source_units":[{"start":33142,"end":33495,"start_line":554,"end_line":558,"raw_tex":"To obtain a contraction, we need $a>\\frac{1}{2}$ and $b<\\frac{1}{2}$. The choice \n\\begin{equation*}\na= 1-\\frac{\\gamma (L-\\tau)}{2}, \\quad b= \\frac{\\gamma (L-\\tau)}{2},\n\\end{equation*}\nwith the additional condition $\\gamma<\\frac{1}{L}$ if $\\tau=0$, satisfies  these two inequalities, as well as \\eqref{eqbb}, $a\\leq 1$, and $b \\gamma \\leq \\frac{1}{2L}$.\n","sha256_utf8":"8f12453f03c7d291edf76b0e3bb00b341062a4e292267bfe760c20a71beea3b3"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"利用PŁ条件建立条件收缩","source_units":[{"start":33496,"end":34393,"start_line":560,"end_line":572,"raw_tex":"Finally, under the assumption that $f$ is $\\mu$-P{\\L}, we have\n\\begin{equation*}\n\\|\\nabla f(x^t)\\|^2\\geq 2\\mu\\big(f(x^t)-f^\\star\\big).\n\\end{equation*}\nTherefore,\n\\begin{align}\n\\Exp{\\Psi^{t+1}\\;|\\;\\mathcal{F}^t}&\\leq  \\big(1-(1-2b)\\gamma\\mu\\big)\\big(f(x^t)  -f^\\star\\big)  -b\\gamma \\sqnorm{\\nabla f(x^t)}\\notag\\\\\n&\\quad+\\frac{a\\gamma}{p}\\left(1-p\\left(1-\\frac{1}{2a}\\right)  \\right)\\sqnorm{g^{t}-\\nabla f(x^t)}+\\frac{b\\gamma}{p}  (1-p)\\sqnorm{g^{t}}\\notag\\\\\n&\\leq  \\big(1-(1-2b)\\gamma\\mu\\big)\\big(f(x^t)  -f^\\star  -b\\gamma \\sqnorm{\\nabla f(x^t)}\\big)\\notag\\\\\n&\\quad+\\frac{a\\gamma}{p}\\left(1-p\\left(1-\\frac{1}{2a}\\right)  \\right)\\sqnorm{g^{t}-\\nabla f(x^t)}+\\frac{b\\gamma}{p}  (1-p)\\sqnorm{g^{t}}\\notag\\\\\n&\\leq \\max\\left(1-(1-2b)\\gamma\\mu,1-p\\left(1-\\frac{1}{2a}\\right),1-p\\right)\\Psi^t\\notag\\\\\n&= \\max\\left(1-(1-2b)\\gamma\\mu,1-p\\left(1-\\frac{1}{2a}\\right)\\right)\\Psi^t.\\label{eqrec2b}\n\\end{align}\n","sha256_utf8":"afd4532d8b416eafed3464e471549f1b44d702a009b2b65e6d5dc2893d65f01d"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"推出期望速率与几乎处处收敛","source_units":[{"start":34393,"end":34712,"start_line":573,"end_line":573,"raw_tex":"Using the tower rule of expectations, we can unroll the recursion in \\eqref{eqrec2b} to obtain the unconditional expectation of $\\Psi^{t+1}$. Moreover, using classical results on supermartingale convergence \\cite[Proposition A.4.5]{ber15}, it follows from \\eqref{eqrec2b} that $\\Psi^t\\rightarrow 0$ almost surely. \\qed\n","sha256_utf8":"cb98e549f6956635f4f0c8311f2b055a85bd7a3eebca069b8bc1859098e43ef5"}]},{"id":"S031","semantic_type":"external_fact","kind":"external_fact","title":"Baillon–Haddad定理","source_units":[],"declared_external":true,"declaration_text":["If h is convex and M-smooth, then ||∇h(x)−∇h(y)||² ≤ M⟨∇h(x)−∇h(y),x−y⟩. Cited as bau17, Corollary 18.17."]},{"id":"S032","semantic_type":"external_fact","kind":"external_fact","title":"梯度范数与函数差界","source_units":[],"declared_external":true,"declaration_text":["If f is L-smooth and bounded below, then ||∇f(x)||² ≤ 2L(f(x)−inf f). Cited as bub15, equation 3.5."]},{"id":"S033","semantic_type":"external_fact","kind":"external_fact","title":"非精确梯度下降不等式","source_units":[],"declared_external":true,"declaration_text":["For L-smooth f and x⁺=x−γg with γ>0, f(x⁺) ≤ f(x)−(γ/2)||∇f(x)||²+(γ/2)||g−∇f(x)||²+(L/2−1/(2γ))||x⁺−x||². Cited as ric21, Lemma 4."]},{"id":"S034","semantic_type":"external_fact","kind":"external_fact","title":"二次系数的充分步长条件","source_units":[],"declared_external":true,"declaration_text":["For A>0 and B≥0, the condition 0<γ≤1/(A+√B) implies Aγ+Bγ²≤1. Cited as ric21, Lemma 5."]},{"id":"S035","semantic_type":"external_fact","kind":"external_fact","title":"条件期望与方差恒等式","source_units":[],"declared_external":true,"declaration_text":["Conditional total expectation permits decomposition over the refresh coin. For a square-integrable Hilbert-space-valued random variable Z, E[||Z||²|F]=||E[Z|F]||²+E[||Z−E[Z|F]||²|F]."]},{"id":"S036","semantic_type":"external_fact","kind":"external_fact","title":"非负随机过程的收缩收敛","source_units":[],"declared_external":true,"declaration_text":["If a nonnegative adapted integrable process V^t satisfies E[V^(t+1)|F^t]≤ρV^t for fixed 0≤ρ<1, the tower rule yields E[V^t]≤ρ^t E[V^0], and nonnegative supermartingale convergence yields V^t→0 almost surely. The author cites ber15, Proposition A.4.5."]},{"id":"S037","semantic_type":"assumption","kind":"assumption","title":"梯度差引理的局部假设","source_units":[{"start":12284,"end":12455,"start_line":199,"end_line":199,"raw_tex":"Let $g:\\mathcal{X}\\rightarrow \\mathbb{R}$ be an $L$-smooth and $\\tau$-weakly convex function for some $L>0$ and $\\tau\\in [0,L]$. Then, for every $(x,y)\\in\\mathcal{X}^2$, \n","sha256_utf8":"9eff7a392246b15ae179718799f8176a15bd5af9c54fbccabd3692e3af26836d"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S007","edges":[{"source_id":"S002","target_id":"S006","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S032","target_id":"S006","kind":"supports"},{"source_id":"S037","target_id":"S011","kind":"supports"},{"source_id":"S031","target_id":"S011","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S010","kind":"supports"},{"source_id":"S012","target_id":"S010","kind":"supports"},{"source_id":"S010","target_id":"S009","kind":"supports"},{"source_id":"S004","target_id":"S014","kind":"supports"},{"source_id":"S005","target_id":"S014","kind":"supports"},{"source_id":"S001","target_id":"S015","kind":"supports"},{"source_id":"S002","target_id":"S015","kind":"supports"},{"source_id":"S004","target_id":"S015","kind":"supports"},{"source_id":"S033","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S016","kind":"supports"},{"source_id":"S004","target_id":"S016","kind":"supports"},{"source_id":"S035","target_id":"S016","kind":"supports"},{"source_id":"S014","target_id":"S017","kind":"supports"},{"source_id":"S004","target_id":"S017","kind":"supports"},{"source_id":"S035","target_id":"S017","kind":"supports"},{"source_id":"S001","target_id":"S018","kind":"supports"},{"source_id":"S017","target_id":"S018","kind":"supports"},{"source_id":"S001","target_id":"S019","kind":"supports"},{"source_id":"S002","target_id":"S019","kind":"supports"},{"source_id":"S009","target_id":"S019","kind":"supports"},{"source_id":"S015","target_id":"S019","kind":"supports"},{"source_id":"S018","target_id":"S019","kind":"supports"},{"source_id":"S016","target_id":"S020","kind":"supports"},{"source_id":"S019","target_id":"S020","kind":"supports"},{"source_id":"S004","target_id":"S021","kind":"supports"},{"source_id":"S018","target_id":"S021","kind":"supports"},{"source_id":"S019","target_id":"S021","kind":"supports"},{"source_id":"S005","target_id":"S022","kind":"supports"},{"source_id":"S015","target_id":"S022","kind":"supports"},{"source_id":"S020","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S014","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S024","kind":"supports"},{"source_id":"S034","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S008","target_id":"S026","kind":"supports"},{"source_id":"S022","target_id":"S027","kind":"supports"},{"source_id":"S023","target_id":"S027","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S008","target_id":"S028","kind":"supports"},{"source_id":"S023","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S026","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S003","target_id":"S029","kind":"supports"},{"source_id":"S005","target_id":"S029","kind":"supports"},{"source_id":"S006","target_id":"S029","kind":"supports"},{"source_id":"S008","target_id":"S029","kind":"supports"},{"source_id":"S027","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S006","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S036","target_id":"S030","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S015","target_id":"S013","kind":"supports"},{"source_id":"S016","target_id":"S013","kind":"supports"},{"source_id":"S017","target_id":"S013","kind":"supports"},{"source_id":"S018","target_id":"S013","kind":"supports"},{"source_id":"S019","target_id":"S013","kind":"supports"},{"source_id":"S020","target_id":"S013","kind":"supports"},{"source_id":"S021","target_id":"S013","kind":"supports"},{"source_id":"S022","target_id":"S013","kind":"supports"},{"source_id":"S023","target_id":"S013","kind":"supports"},{"source_id":"S024","target_id":"S013","kind":"supports"},{"source_id":"S025","target_id":"S013","kind":"supports"},{"source_id":"S026","target_id":"S013","kind":"supports"},{"source_id":"S027","target_id":"S013","kind":"supports"},{"source_id":"S028","target_id":"S013","kind":"supports"},{"source_id":"S029","target_id":"S013","kind":"supports"},{"source_id":"S030","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S007","kind":"supports"}]}}
{"record_id":"2510.16680v3","paper":{"title":"HNAG$^{++}$: An Accelerated Gradient Method with a Refined Asymptotic Rate for Strongly Convex Optimization","authors":["Long Chen","Zeyi Xu"],"arxiv_id":"2510.16680","version":"v3","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-026-03131-x","abstract_url":"https://arxiv.org/abs/2510.16680v3","pdf_url":"https://arxiv.org/pdf/2510.16680v3"},"context":{"raw_tex":"We consider the unconstrained smooth convex optimization problem\n\\begin{equation*}\n\\min_{x\\in\\mathbb{R}^d} f(x),\n\\end{equation*}\nwhere $f$ belongs to the class $\\mathcal S_{\\mu,L}$ of continuously differentiable, $\\mu$-strongly convex functions with $L$-Lipschitz continuous gradients. Strong convexity guarantees the existence\nof a unique global minimizer $x^{\\star}$. We study first-order methods, which\nrely solely on gradient information, and generate iterates \n$\\{x_k\\}$ converging to $x^{\\star}$.\n\n\nConvergence is measured by a nonnegative error quantity $E_k$, such as $\\|x_k-x^\\star\\|^2$, $f(x_k)-f(x^\\star)$, or a Lyapunov function with comparable scaling. We say that the sequence $\\{E_k\\}$\nconverges linearly if\n$$\nE_k \\le C r^k, \\qquad k \\ge 1,\n$$\nfor some constants $C>0$ and $r\\in(0,1)$.\nMore precisely, $E_k$ is said to converge \\emph{R-linearly} with asymptotic rate\n$r$ if\n$$\n\\limsup_{k\\to\\infty} E_k^{1/k} \\le r.\n$$\n\n\nThe condition number of $f$ is $\\kappa = L/\\mu$, and we denote its reciprocal by\n$\\rho = 1/\\kappa = \\mu/L$. \n\n\nLet $f: \\mathbb{R}^d \\to \\mathbb{R}$ be differentiable. We use $\\langle \\cdot, \\cdot \\rangle$ for the inner product of $\\mathbb R^d$.\n\nThe Bregman divergence of $f$ between $x, y \\in \\mathbb{R}^d$ is \n\\begin{equation*}\nD_f(y,x) := f(y) - f(x) - \\langle \\nabla f(x),\\, y - x \\rangle .\n\\end{equation*}\nIn general, $D_f$ is not symmetric, i.e., $D_f(y,x) \\neq D_f(x,y)$.  \nIts symmetrization, called the symmetrized Bregman divergence, is  \n\\begin{equation}\\label{eq:symBre}\nD_f(y,x) + D_f(x,y)\n= \\langle \\nabla f(y) - \\nabla f(x),\\, y - x \\rangle .\n\\end{equation}\nDefine the difference of Bregman divergences\n\\begin{equation*}\n\\Delta_f(x,y) := D_f(y,x) - D_f(x,y).\n\\end{equation*}\nWe can rewrite the symmetric Bregman divergence via the asymmetry:\n\\begin{equation}\\label{eq:Brelation}\n\\langle \\nabla f(x) - \\nabla f(y),\\, x - y \\rangle \n= 2D_f(x,y) + \\Delta_f(x,y).\n\\end{equation}\n\n\nThe following three-point Bregman identity holds~\\cite{chen1993convergence}:\n\\begin{equation}\\label{eq:Bregmanidentity}\n\\langle \\nabla f(y) - \\nabla f(x),\\, y - z \\rangle \n= D_f(y,x) + D_f(z,y) - D_f(z,x),\n\\end{equation}\nwhich is a generalization of the classical identity of squares:\n\\begin{equation}\\label{eq:squares}\n\\langle y-x,\\, y-z \\rangle\n=\n\\frac12\\bigl(\n\\|y-x\\|^2 + \\|z-y\\|^2 - \\|z-x\\|^2\n\\bigr).\n\\end{equation}\n\n\nThe function $f$ is $\\mu$-strongly convex if for some $\\mu > 0$,\n\\begin{equation*}\nD_f(y,x) \\geq \\frac{\\mu}{2} \\|y - x\\|^2, \n\\qquad \\forall x, y \\in \\mathbb{R}^d.\n\\end{equation*}\nIt is $L$-smooth, for some $L > 0$, if its gradient is $L$-Lipschitz:\n\\begin{equation*}\n\\|\\nabla f(y) - \\nabla f(x)\\| \\leq L \\|y - x\\|, \n\\qquad \\forall x, y \\in \\mathbb{R}^d.\n\\end{equation*}\n\n\nLet $\\mathcal{S}_{\\mu,L}$ be the class of all differentiable functions that are both $\\mu$-strongly convex and $L$-smooth.  \nFor $f \\in \\mathcal{S}_{\\mu,L}$, the Bregman divergence satisfies~\\cite{nesterov2003introductory}\n\\begin{equation}\\label{eq:muLsquares}\n\\frac{\\mu}{2} \\| x - y \\|^2 \n\\leq D_f(x,y) \n\\leq \\frac{L}{2} \\| x - y \\|^2, \n\\qquad \\forall x, y \\in \\mathbb{R}^d,\n\\end{equation}\nand, in terms of gradient differences,\n\\begin{equation}\\label{eq:cocov}\n\\frac{1}{2L} \\|\\nabla f(x) - \\nabla f(y)\\|^2 \n\\leq D_f(x,y) \n\\leq \\frac{1}{2\\mu} \\|\\nabla f(x) - \\nabla f(y)\\|^2, \n\\qquad \\forall x, y \\in \\mathbb{R}^d.\n\\end{equation}\n\n\nThe continuous-time HNAG flow is\n\\begin{equation}\\label{eq:flow}\n\\left\\{\n\\begin{aligned}\nx' &= y - x - \\beta \\nabla f(x), \\\\\ny' &= x - y - \\tfrac{1}{\\mu}\\nabla f(x),\n\\end{aligned}\n\\right.\n\\end{equation}\n\nwhere $\\mu$ is the strong convexity parameter of $f$.\nLet $\\boldsymbol z=(x,y)$ and define\n\\begin{equation}\\label{eq:G}\n\\mathcal G(\\boldsymbol z)\n=\\bigl(y-x-\\beta\\nabla f(x),\\; x-y-\\tfrac{1}{\\mu}\\nabla f(x)\\bigr)^{\\intercal},\n\\end{equation}\n\nso that \\eqref{eq:flow} can be written compactly as $\\boldsymbol z'=\\mathcal G(\\boldsymbol z)$.\n\n\nAn implicit--explicit (IMEX) discretization of~\\eqref{eq:flow} yields\n\\begin{subequations}\\label{eq:HNAG}\n\\begin{align}\n\\label{eq:HNAGx}\n\\frac{x_{k+1}-x_k}{\\alpha} &= y_k - x_{k+1} - \\beta \\nabla f(x_k), \\\\\n\\label{eq:HNAGy}\n\\frac{y_{k+1}-y_k}{\\alpha} &= x_{k+1} - y_{k+1} - \\tfrac{1}{\\mu}\\nabla f(x_{k+1}),\n\\end{align}\n\n\\end{subequations}\nwhere $x_{k+1}$ is updated first in \\eqref{eq:HNAGx} and then used to compute $y_{k+1}$ in \\eqref{eq:HNAGy}.\n\n\nWe use the following Lyapunov notation throughout the paper. First, define\n\\begin{equation}\\label{eq:lyapunov_strong_HNAG}\nE(\\boldsymbol z)=E(x,y)\n:=D_f(x,x^\\star)+\\frac{\\mu}{2}\\|y-x^\\star\\|^2\n=\nf(x)-f(x^\\star)+\\frac{\\mu}{2}\\|y-x^\\star\\|^2,\n\\end{equation}\n\nwhere \\(D_f(x,x^\\star)=f(x)-f(x^\\star)\\) since \\(\\nabla f(x^\\star)=0\\). Since\n\\(f\\in\\mathcal S_{\\mu,L}\\), we have \\(E(\\boldsymbol z)\\ge0\\), with equality if\nand only if \\(x=y=x^\\star\\).\n\n\n\\begin{algorithm}\n\\caption{HNAG$^{++}$}\n\\label{alg:HNAGpp}\n\\linespread{1.3}\\selectfont\n\\begin{algorithmic}[1]\n\\Require $x_0,v_0\\in\\mathbb R^d$, parameters $L\\ge \\mu>0$\n\\State $\\alpha \\gets \\sqrt{2\\mu/L}$\n\\For{$k=0,1,2,\\ldots$}\n\\State $v_{k+1}\\gets \\tfrac{1}{1+\\alpha}\n\\bigl(v_k+\\alpha^2 x_k-\\tfrac{2}{L}\\nabla f(x_k)\\bigr)$\n\\State $x_{k+1}\\gets \\tfrac{1}{1+\\alpha}\n\\bigl(x_k+v_{k+1}-\\tfrac{1}{L}\\nabla f(x_k)\\bigr)$\n\\EndFor\n\\State \\textbf{Return:} $x_{k+1}$ and $y_{k} = v_{k+1}/\\alpha$\n\\end{algorithmic}\n\\end{algorithm}\n\nSince $f$ is $\\mu$-strongly convex, for any $\\hat{\\mu}\\leq \\mu$, the shifted function\n$$\nf_{-\\hat{\\mu}}(x)\n:= f(x) - \\frac{\\hat{\\mu}}{2}\\|x - x^{\\star}\\|^2\n$$\nis convex and satisfies $\\nabla f_{-\\hat{\\mu}}(x^{\\star}) = 0$. \n\n\n\\begin{equation}\\label{eq:lya-muhat}\n\\mathcal{E}(\\bs z; \\hat{\\mu})\n:= D_{f_{-\\hat{\\mu}}}(x, x^{\\star})\n+ \\tfrac{\\mu}{2}\\|y - x^{\\star}\\|^2.\n\\end{equation}\n\n{Notice that the partial shift \\(\\hat\\mu\\) is used only in \\(f\\), while the \\(y\\)-component still uses the fixed coefficient \\(\\mu\\).} \n\n\n\\begin{definition}[Local asymptotic symmetry]\nLet $f\\in\\mathcal S_{\\mu,L}$ with minimizer $x^\\star$. The function $f$ is said to be \\emph{locally asymptotically symmetric at $x^\\star$}, abbreviated \\emph{LAS at $x^\\star$}, if\n$$\n\\lim_{x\\to x^\\star}\n\\RV{\\frac{|\\Delta_f(x,x^\\star)|}{D_f(x,x^\\star)}}\n=\n0.\n$$\n%where\n%$$\n%\\Delta_f(x,x^\\star)\n%:=\n%D_f(x^\\star,x)-D_f(x,x^\\star).\n%$$\nDefine\n$$\n\\omega(R)\n:=\n\\sup_{0<\\|x-x^\\star\\|\\leq R}\n\\frac{|\\Delta_f(x,x^\\star)|}{D_f(x,x^\\star)},\n\\qquad R>0.\n$$\nThen $\\omega$ is monotone nondecreasing, and the LAS condition is equivalent to\n$$\n\\lim_{R\\to0^+}\\omega(R)=0.\n$$\n\\end{definition}\n\nWe define\n$$\n\\mathcal S_{\\mu,L}^{\\mathrm{LAS}}(x^\\star)\n:=\n\\Bigl\\{\nf\\in\\mathcal S_{\\mu,L}:\nf\\text{ is LAS at }x^\\star\n\\Bigr\\}.\n$$\n\n\nLet \\(\\{\\mu_k\\}\\) be a nondecreasing sequence with \\(\\mu_k\\uparrow\\mu\\). Define\n\\begin{equation*}\n\\mathcal E(\\bs z_k;\\mu_k)\n:=\nD_{f_k}(x_k,x^\\star)+\\frac{\\mu}{2}\\|y_k-x^\\star\\|^2,\n\\qquad\nf_k:=f_{-\\mu_k}.\n\\end{equation*}\nThe parameter \\(\\mu_k\\) is used only in the shift of \\(f_k\\); the quadratic term in \\(y_k\\) always uses the fixed coefficient \\(\\mu\\). We write\n$$\n\\delta_k:=\\frac{\\mu-\\mu_k}{\\mu},\n\\qquad\n\\Delta\\delta_k:=\\delta_k-\\delta_{k+1}\\ge0.\n$$\n\n\nIf $f\\in\\mathcal S_{\\mu,L}^{\\rm LAS}(x^\\star)$, then\n\\begin{equation}\\label{eq:def-omegak}\n\\omega(\\|x_{k+1}-x^\\star\\|)\n\\leq\n\\omega\\left(\nC_1\\left(1+\\sqrt{2\\rho}\\right)^{-\\frac{k+1}{2}}\n\\right)\n=:\\omega_k.\n\\end{equation}","units":[{"start":4998,"end":5501,"start_line":112,"end_line":119,"raw_tex":"We consider the unconstrained smooth convex optimization problem\n\\begin{equation*}\n\\min_{x\\in\\mathbb{R}^d} f(x),\n\\end{equation*}\nwhere $f$ belongs to the class $\\mathcal S_{\\mu,L}$ of continuously differentiable, $\\mu$-strongly convex functions with $L$-Lipschitz continuous gradients. Strong convexity guarantees the existence\nof a unique global minimizer $x^{\\star}$. We study first-order methods, which\nrely solely on gradient information, and generate iterates \n$\\{x_k\\}$ converging to $x^{\\star}$.\n","sha256_utf8":"5ee12bddb36bdc31c1311bad888e6f7d22cde0e8d591fef0ac2f65c81443e350"},{"start":5502,"end":5931,"start_line":121,"end_line":131,"raw_tex":"Convergence is measured by a nonnegative error quantity $E_k$, such as $\\|x_k-x^\\star\\|^2$, $f(x_k)-f(x^\\star)$, or a Lyapunov function with comparable scaling. We say that the sequence $\\{E_k\\}$\nconverges linearly if\n$$\nE_k \\le C r^k, \\qquad k \\ge 1,\n$$\nfor some constants $C>0$ and $r\\in(0,1)$.\nMore precisely, $E_k$ is said to converge \\emph{R-linearly} with asymptotic rate\n$r$ if\n$$\n\\limsup_{k\\to\\infty} E_k^{1/k} \\le r.\n$$\n","sha256_utf8":"207246691003b2416418aec4dcab98192b1ececbdd4a6b4761e7c2d68b201e37"},{"start":6225,"end":6334,"start_line":140,"end_line":141,"raw_tex":"The condition number of $f$ is $\\kappa = L/\\mu$, and we denote its reciprocal by\n$\\rho = 1/\\kappa = \\mu/L$. \n","sha256_utf8":"0a0cb50621164b7dfa355d5b577ece2f5d1a8ca109d43e1b7dd7c2d1f06ea098"},{"start":13029,"end":13908,"start_line":236,"end_line":256,"raw_tex":"Let $f: \\mathbb{R}^d \\to \\mathbb{R}$ be differentiable. We use $\\langle \\cdot, \\cdot \\rangle$ for the inner product of $\\mathbb R^d$.\n\nThe Bregman divergence of $f$ between $x, y \\in \\mathbb{R}^d$ is \n\\begin{equation*}\nD_f(y,x) := f(y) - f(x) - \\langle \\nabla f(x),\\, y - x \\rangle .\n\\end{equation*}\nIn general, $D_f$ is not symmetric, i.e., $D_f(y,x) \\neq D_f(x,y)$.  \nIts symmetrization, called the symmetrized Bregman divergence, is  \n\\begin{equation}\\label{eq:symBre}\nD_f(y,x) + D_f(x,y)\n= \\langle \\nabla f(y) - \\nabla f(x),\\, y - x \\rangle .\n\\end{equation}\nDefine the difference of Bregman divergences\n\\begin{equation*}\n\\Delta_f(x,y) := D_f(y,x) - D_f(x,y).\n\\end{equation*}\nWe can rewrite the symmetric Bregman divergence via the asymmetry:\n\\begin{equation}\\label{eq:Brelation}\n\\langle \\nabla f(x) - \\nabla f(y),\\, x - y \\rangle \n= 2D_f(x,y) + \\Delta_f(x,y).\n\\end{equation}\n","sha256_utf8":"628b1212dc24d38b565ff4ef5c7b7fc34aa69c45821159d63aca52c4db4ac254"},{"start":14008,"end":14428,"start_line":259,"end_line":271,"raw_tex":"The following three-point Bregman identity holds~\\cite{chen1993convergence}:\n\\begin{equation}\\label{eq:Bregmanidentity}\n\\langle \\nabla f(y) - \\nabla f(x),\\, y - z \\rangle \n= D_f(y,x) + D_f(z,y) - D_f(z,x),\n\\end{equation}\nwhich is a generalization of the classical identity of squares:\n\\begin{equation}\\label{eq:squares}\n\\langle y-x,\\, y-z \\rangle\n=\n\\frac12\\bigl(\n\\|y-x\\|^2 + \\|z-y\\|^2 - \\|z-x\\|^2\n\\bigr).\n\\end{equation}\n","sha256_utf8":"20640f2a20c7ad010f3f42ca7e39974e16f45b44324cd33db8662824de8c9d50"},{"start":14430,"end":14800,"start_line":274,"end_line":283,"raw_tex":"The function $f$ is $\\mu$-strongly convex if for some $\\mu > 0$,\n\\begin{equation*}\nD_f(y,x) \\geq \\frac{\\mu}{2} \\|y - x\\|^2, \n\\qquad \\forall x, y \\in \\mathbb{R}^d.\n\\end{equation*}\nIt is $L$-smooth, for some $L > 0$, if its gradient is $L$-Lipschitz:\n\\begin{equation*}\n\\|\\nabla f(y) - \\nabla f(x)\\| \\leq L \\|y - x\\|, \n\\qquad \\forall x, y \\in \\mathbb{R}^d.\n\\end{equation*}\n","sha256_utf8":"285054ef020908ad3ee5cf8e324b97dd6f0b3d422b469ce3386f619afbac9dec"},{"start":14981,"end":15612,"start_line":287,"end_line":301,"raw_tex":"Let $\\mathcal{S}_{\\mu,L}$ be the class of all differentiable functions that are both $\\mu$-strongly convex and $L$-smooth.  \nFor $f \\in \\mathcal{S}_{\\mu,L}$, the Bregman divergence satisfies~\\cite{nesterov2003introductory}\n\\begin{equation}\\label{eq:muLsquares}\n\\frac{\\mu}{2} \\| x - y \\|^2 \n\\leq D_f(x,y) \n\\leq \\frac{L}{2} \\| x - y \\|^2, \n\\qquad \\forall x, y \\in \\mathbb{R}^d,\n\\end{equation}\nand, in terms of gradient differences,\n\\begin{equation}\\label{eq:cocov}\n\\frac{1}{2L} \\|\\nabla f(x) - \\nabla f(y)\\|^2 \n\\leq D_f(x,y) \n\\leq \\frac{1}{2\\mu} \\|\\nabla f(x) - \\nabla f(y)\\|^2, \n\\qquad \\forall x, y \\in \\mathbb{R}^d.\n\\end{equation}\n","sha256_utf8":"4f283765b6cce5b65de580b635e0bb4734d5894229f176424e5b6b5a65f4b521"},{"start":17062,"end":17264,"start_line":331,"end_line":339,"raw_tex":"The continuous-time HNAG flow is\n\\begin{equation}\\label{eq:flow}\n\\left\\{\n\\begin{aligned}\nx' &= y - x - \\beta \\nabla f(x), \\\\\ny' &= x - y - \\tfrac{1}{\\mu}\\nabla f(x),\n\\end{aligned}\n\\right.\n\\end{equation}","sha256_utf8":"d6c9c0794a6e2c541d80e698c076263923f5b46dc891ebfae4bdef67fb25354b"},{"start":17265,"end":17505,"start_line":340,"end_line":345,"raw_tex":"where $\\mu$ is the strong convexity parameter of $f$.\nLet $\\boldsymbol z=(x,y)$ and define\n\\begin{equation}\\label{eq:G}\n\\mathcal G(\\boldsymbol z)\n=\\bigl(y-x-\\beta\\nabla f(x),\\; x-y-\\tfrac{1}{\\mu}\\nabla f(x)\\bigr)^{\\intercal},\n\\end{equation}","sha256_utf8":"ae84f2239bcdf4840d9a4d9b81ec1568338b5e050ec5639cfaddafa7f5f4f7c2"},{"start":17506,"end":17602,"start_line":346,"end_line":346,"raw_tex":"so that \\eqref{eq:flow} can be written compactly as $\\boldsymbol z'=\\mathcal G(\\boldsymbol z)$.\n","sha256_utf8":"60f01ce00a4b893ea494992f97941e24db0b76546de8cc265afa23de494a8d54"},{"start":17603,"end":17921,"start_line":348,"end_line":355,"raw_tex":"An implicit--explicit (IMEX) discretization of~\\eqref{eq:flow} yields\n\\begin{subequations}\\label{eq:HNAG}\n\\begin{align}\n\\label{eq:HNAGx}\n\\frac{x_{k+1}-x_k}{\\alpha} &= y_k - x_{k+1} - \\beta \\nabla f(x_k), \\\\\n\\label{eq:HNAGy}\n\\frac{y_{k+1}-y_k}{\\alpha} &= x_{k+1} - y_{k+1} - \\tfrac{1}{\\mu}\\nabla f(x_{k+1}),\n\\end{align}","sha256_utf8":"67b2bd65bb21e92ccf0e35a6104a9665a42485cad42df784544d826e642c979d"},{"start":17922,"end":18050,"start_line":356,"end_line":357,"raw_tex":"\\end{subequations}\nwhere $x_{k+1}$ is updated first in \\eqref{eq:HNAGx} and then used to compute $y_{k+1}$ in \\eqref{eq:HNAGy}.\n","sha256_utf8":"f4153922d90cede69be73ae9c267e0906ccbaa9f2222ab19bc2cebdd08fbf79c"},{"start":19767,"end":20022,"start_line":402,"end_line":408,"raw_tex":"We use the following Lyapunov notation throughout the paper. First, define\n\\begin{equation}\\label{eq:lyapunov_strong_HNAG}\nE(\\boldsymbol z)=E(x,y)\n:=D_f(x,x^\\star)+\\frac{\\mu}{2}\\|y-x^\\star\\|^2\n=\nf(x)-f(x^\\star)+\\frac{\\mu}{2}\\|y-x^\\star\\|^2,\n\\end{equation}","sha256_utf8":"339265d7f16033c6a4eb3748db7ea5731752fb99b42ff569d58f8fa692cd7720"},{"start":20023,"end":20209,"start_line":409,"end_line":411,"raw_tex":"where \\(D_f(x,x^\\star)=f(x)-f(x^\\star)\\) since \\(\\nabla f(x^\\star)=0\\). Since\n\\(f\\in\\mathcal S_{\\mu,L}\\), we have \\(E(\\boldsymbol z)\\ge0\\), with equality if\nand only if \\(x=y=x^\\star\\).\n","sha256_utf8":"778748c7c4fa41fabcf2100dd54c2637d493cc5fa240c143481f6a43c8a266b9"},{"start":64328,"end":64848,"start_line":1637,"end_line":1652,"raw_tex":"\\begin{algorithm}\n\\caption{HNAG$^{++}$}\n\\label{alg:HNAGpp}\n\\linespread{1.3}\\selectfont\n\\begin{algorithmic}[1]\n\\Require $x_0,v_0\\in\\mathbb R^d$, parameters $L\\ge \\mu>0$\n\\State $\\alpha \\gets \\sqrt{2\\mu/L}$\n\\For{$k=0,1,2,\\ldots$}\n\\State $v_{k+1}\\gets \\tfrac{1}{1+\\alpha}\n\\bigl(v_k+\\alpha^2 x_k-\\tfrac{2}{L}\\nabla f(x_k)\\bigr)$\n\\State $x_{k+1}\\gets \\tfrac{1}{1+\\alpha}\n\\bigl(x_k+v_{k+1}-\\tfrac{1}{L}\\nabla f(x_k)\\bigr)$\n\\EndFor\n\\State \\textbf{Return:} $x_{k+1}$ and $y_{k} = v_{k+1}/\\alpha$\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"d3688d7a53361b8ee9fab83a0c811d3fa9a2dfc8b75f06c41dee5a3296d50121"},{"start":71365,"end":71589,"start_line":1842,"end_line":1847,"raw_tex":"Since $f$ is $\\mu$-strongly convex, for any $\\hat{\\mu}\\leq \\mu$, the shifted function\n$$\nf_{-\\hat{\\mu}}(x)\n:= f(x) - \\frac{\\hat{\\mu}}{2}\\|x - x^{\\star}\\|^2\n$$\nis convex and satisfies $\\nabla f_{-\\hat{\\mu}}(x^{\\star}) = 0$. \n","sha256_utf8":"40aa77b48170c47ac0727dd2f37f8e49a419a92af9dd642259bfca50d9089b9e"},{"start":72169,"end":72323,"start_line":1860,"end_line":1864,"raw_tex":"\\begin{equation}\\label{eq:lya-muhat}\n\\mathcal{E}(\\bs z; \\hat{\\mu})\n:= D_{f_{-\\hat{\\mu}}}(x, x^{\\star})\n+ \\tfrac{\\mu}{2}\\|y - x^{\\star}\\|^2.\n\\end{equation}","sha256_utf8":"41b0441cda25ad13113a844170c532ab9513743587187e0ca5f162363c2d3e6a"},{"start":72324,"end":72460,"start_line":1865,"end_line":1865,"raw_tex":"{Notice that the partial shift \\(\\hat\\mu\\) is used only in \\(f\\), while the \\(y\\)-component still uses the fixed coefficient \\(\\mu\\).} \n","sha256_utf8":"6152258f7da1ec9df04672b39416385b11ce6b0f80ac9b6b044015075dc0784b"},{"start":86257,"end":86878,"start_line":2369,"end_line":2395,"raw_tex":"\\begin{definition}[Local asymptotic symmetry]\nLet $f\\in\\mathcal S_{\\mu,L}$ with minimizer $x^\\star$. The function $f$ is said to be \\emph{locally asymptotically symmetric at $x^\\star$}, abbreviated \\emph{LAS at $x^\\star$}, if\n$$\n\\lim_{x\\to x^\\star}\n\\RV{\\frac{|\\Delta_f(x,x^\\star)|}{D_f(x,x^\\star)}}\n=\n0.\n$$\n%where\n%$$\n%\\Delta_f(x,x^\\star)\n%:=\n%D_f(x^\\star,x)-D_f(x,x^\\star).\n%$$\nDefine\n$$\n\\omega(R)\n:=\n\\sup_{0<\\|x-x^\\star\\|\\leq R}\n\\frac{|\\Delta_f(x,x^\\star)|}{D_f(x,x^\\star)},\n\\qquad R>0.\n$$\nThen $\\omega$ is monotone nondecreasing, and the LAS condition is equivalent to\n$$\n\\lim_{R\\to0^+}\\omega(R)=0.\n$$\n\\end{definition}","sha256_utf8":"599028ac9bd4fd5d0a560f9fce5cf80a285663848c789bef186f9d83653c60ea"},{"start":86880,"end":87010,"start_line":2397,"end_line":2405,"raw_tex":"We define\n$$\n\\mathcal S_{\\mu,L}^{\\mathrm{LAS}}(x^\\star)\n:=\n\\Bigl\\{\nf\\in\\mathcal S_{\\mu,L}:\nf\\text{ is LAS at }x^\\star\n\\Bigr\\}.\n$$\n","sha256_utf8":"dc593262bc94a164be1fb49cb55a4757cc73a641234e0d9e7aca5100c32b7a6a"},{"start":92522,"end":92975,"start_line":2614,"end_line":2627,"raw_tex":"Let \\(\\{\\mu_k\\}\\) be a nondecreasing sequence with \\(\\mu_k\\uparrow\\mu\\). Define\n\\begin{equation*}\n\\mathcal E(\\bs z_k;\\mu_k)\n:=\nD_{f_k}(x_k,x^\\star)+\\frac{\\mu}{2}\\|y_k-x^\\star\\|^2,\n\\qquad\nf_k:=f_{-\\mu_k}.\n\\end{equation*}\nThe parameter \\(\\mu_k\\) is used only in the shift of \\(f_k\\); the quadratic term in \\(y_k\\) always uses the fixed coefficient \\(\\mu\\). We write\n$$\n\\delta_k:=\\frac{\\mu-\\mu_k}{\\mu},\n\\qquad\n\\Delta\\delta_k:=\\delta_k-\\delta_{k+1}\\ge0.\n$$\n","sha256_utf8":"3d8e6646880d49f732b5b566a16d01e9797496eba0a42773d82e68752c586dc2"},{"start":94195,"end":94414,"start_line":2684,"end_line":2692,"raw_tex":"If $f\\in\\mathcal S_{\\mu,L}^{\\rm LAS}(x^\\star)$, then\n\\begin{equation}\\label{eq:def-omegak}\n\\omega(\\|x_{k+1}-x^\\star\\|)\n\\leq\n\\omega\\left(\nC_1\\left(1+\\sqrt{2\\rho}\\right)^{-\\frac{k+1}{2}}\n\\right)\n=:\\omega_k.\n\\end{equation}","sha256_utf8":"7c4df7ba2f0c2ce636efcfc4f4e55b71545b34899e9f90f873f9af76f38e9b21"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}[Asymptotic rate $1-2\\sqrt{2/\\kappa}$]\\label{thm:main-thm-C21}\nAssume $f\\in\\mathcal S_{\\mu,L}^{\\rm LAS}(x^\\star)$. Let $(x_k,y_k)$ be generated by Algorithm~\\ref{alg:HNAGpp}, let $\\omega_k$ be defined by \\eqref{eq:def-omegak}, and set $K:=\\min\\{k\\geq0:\\omega_k\\leq1/8\\}$. Then there exist nonnegative sequences $\\{\\mu_k\\}$, $\\{\\delta_k\\}$, $\\{b_k\\}$, and $\\{c_k\\}$, and a constant $C>0$, such that\n$$\n\\mu_k=(1-\\delta_k)\\mu,\\qquad\n\\delta_k\\downarrow0,\\qquad\nb_k\\to0,\\qquad\nc_k=2-\\sqrt{\\delta_k}-b_k\\to2,\n$$\nand\n%$$\n%\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n%\\leq\n%C\\prod_{i=0}^{k}r_i.\n%$$\n\\RV{\n$$\n\\frac{\\mu}{2}\\|y_{k+1}-x^\\star\\|^2\n\\leq\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\nC\\prod_{i=0}^{k}r_i.\n$$}\nHere\n$$\n\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k)\n:=\n\\mathcal E(\\bs z_k;\\mu_k)\n-\\frac1{2L}\\|\\nabla f_k(x_k)\\|^2,\n\\qquad\nf_k:=f_{-\\mu_k},\n$$\nand\n$$\nr_k:=\n\\begin{cases}\n(1+\\alpha)^{-1}, & 0\\leq k<K,\\\\[1mm]\n\\left(1+c_k\\alpha-\\Delta\\delta_k\\alpha^2\\right)^{-1}, & k\\geq K,\n\\end{cases}\n\\quad\n\\alpha=\\sqrt{\\frac{2}{\\kappa}},\n\\quad\n\\Delta\\delta_k:=\\delta_k-\\delta_{k+1}.\n$$\nConsequently, $\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k)$ \\RV{and $\\|y_k-x^\\star\\|^2$} converge $R$-linearly with asymptotic rate\n$$\nr_\\ast\n=\n\\frac1{1+2\\sqrt{2/\\kappa}}\n=\n1-2\\sqrt{2/\\kappa}+O(\\kappa^{-1})\n\\qquad\\text{as }\\kappa\\to\\infty.\n$$\n\\end{theorem}","units":[{"start":99631,"end":100976,"start_line":2897,"end_line":2949,"raw_tex":"\\begin{theorem}[Asymptotic rate $1-2\\sqrt{2/\\kappa}$]\\label{thm:main-thm-C21}\nAssume $f\\in\\mathcal S_{\\mu,L}^{\\rm LAS}(x^\\star)$. Let $(x_k,y_k)$ be generated by Algorithm~\\ref{alg:HNAGpp}, let $\\omega_k$ be defined by \\eqref{eq:def-omegak}, and set $K:=\\min\\{k\\geq0:\\omega_k\\leq1/8\\}$. Then there exist nonnegative sequences $\\{\\mu_k\\}$, $\\{\\delta_k\\}$, $\\{b_k\\}$, and $\\{c_k\\}$, and a constant $C>0$, such that\n$$\n\\mu_k=(1-\\delta_k)\\mu,\\qquad\n\\delta_k\\downarrow0,\\qquad\nb_k\\to0,\\qquad\nc_k=2-\\sqrt{\\delta_k}-b_k\\to2,\n$$\nand\n%$$\n%\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n%\\leq\n%C\\prod_{i=0}^{k}r_i.\n%$$\n\\RV{\n$$\n\\frac{\\mu}{2}\\|y_{k+1}-x^\\star\\|^2\n\\leq\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\nC\\prod_{i=0}^{k}r_i.\n$$}\nHere\n$$\n\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k)\n:=\n\\mathcal E(\\bs z_k;\\mu_k)\n-\\frac1{2L}\\|\\nabla f_k(x_k)\\|^2,\n\\qquad\nf_k:=f_{-\\mu_k},\n$$\nand\n$$\nr_k:=\n\\begin{cases}\n(1+\\alpha)^{-1}, & 0\\leq k<K,\\\\[1mm]\n\\left(1+c_k\\alpha-\\Delta\\delta_k\\alpha^2\\right)^{-1}, & k\\geq K,\n\\end{cases}\n\\quad\n\\alpha=\\sqrt{\\frac{2}{\\kappa}},\n\\quad\n\\Delta\\delta_k:=\\delta_k-\\delta_{k+1}.\n$$\nConsequently, $\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k)$ \\RV{and $\\|y_k-x^\\star\\|^2$} converge $R$-linearly with asymptotic rate\n$$\nr_\\ast\n=\n\\frac1{1+2\\sqrt{2/\\kappa}}\n=\n1-2\\sqrt{2/\\kappa}+O(\\kappa^{-1})\n\\qquad\\text{as }\\kappa\\to\\infty.\n$$\n\\end{theorem}","sha256_utf8":"b638a657dfb57c84b0e4f83355ae7c4e197230ef9681ffc75729682bb1ed752a"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\\label{lem:cont_strong}\nLet $\\mathcal G(\\boldsymbol z)$ be the vector field of the HNAG flow~\\eqref{eq:flow}.\nThen $E(\\boldsymbol z)$ satisfies the following strong Lyapunov condition\n\\begin{equation}\\label{eq:A-HNAG}\n-\\langle\\nabla E(\\boldsymbol z),\\mathcal G(\\boldsymbol z)\\rangle\n\\;\\ge\\;\nE(\\boldsymbol z)+\\beta\\|\\nabla f(x)\\|^2+\\frac{\\mu}{2}\\|x-y\\|^2 .\n\\end{equation}\n\\end{lemma}\n\n\\begin{proof}\nA direct computation yields\n$$\n\\begin{aligned}\n&\\quad -\\langle\\nabla E(\\boldsymbol z),\\mathcal G(\\boldsymbol z)\\rangle \\\\ \n&=\n\\dual{\\nabla f(x),x-x^\\star}\n+\\beta\\|\\nabla f(x)\\|^2\n+\\mu\\|y-x^\\star\\|^2\n-\\mu\\dual{y-x^\\star,x-x^\\star} \\\\\n&=\nE(\\boldsymbol z)\n+D_f(x^\\star,x)\n+\\beta\\|\\nabla f(x)\\|^2\n+\\frac{\\mu}{2}\\|x-y\\|^2\n-\\frac{\\mu}{2}\\|x-x^\\star\\|^2,\n\\end{aligned}\n$$\nwhere we use identities of squares~\\eqref{eq:squares} to expand the cross term. \nUsing the $\\mu$-convexity $D_f(x^\\star,x)\\ge \\tfrac{\\mu}{2}\\|x-x^\\star\\|^2$ gives \\eqref{eq:A-HNAG}.\n$\\Box$ \\end{proof}\n\n\\begin{lemma}\\label{lem:disc_strong}\nLet $(x_k,y_k)$ be the iterates generated by~\\eqref{eq:HNAG} and $E$ defined by \\eqref{eq:lyapunov_strong_HNAG}. Then\n\\begin{equation}\\label{eq:onestep}\n\\begin{aligned}\n(1+\\alpha)E(\\bs z_{k+1}) \\;\\le\\;& E(\\bs z_k)\n+ \\Bigl(\\frac{\\alpha^2}{2\\mu}-\\frac{\\alpha\\beta}{2}\\Bigr)\\|\\nabla f(x_{k+1})\\|^2\n- \\frac{\\alpha\\beta}{2}\\|\\nabla f(x_k)\\|^2 \\\\\n\\qquad-\\frac{\\alpha \\mu}{2}\\|x_{k+1} - y_{k+1}\\|^2 &+ \\frac{\\alpha\\beta}{2}\\|\\nabla f(x_{k+1})-\\nabla f(x_k)\\|^2\n- D_f(x_k,x_{k+1}).\n\\end{aligned}\n\\end{equation}\n\\end{lemma}\n\n\\begin{proof}\nExpand the difference of $E$ by the definition of the Bregman divergence:\n\\begin{equation*}\nE(\\bs z_{k+1})-E(\\bs z_k)\n= \\langle \\nabla E(\\bs z_{k+1}),\\,\\bs z_{k+1}-\\bs z_k\\rangle\n- D_E(\\bs z_k,\\bs z_{k+1}),\n\\end{equation*}\nwhere\n\\begin{equation}\\label{eq:DEHNAG}\n- D_E(\\bs z_k,\\bs z_{k+1})\n= -D_f(x_k,x_{k+1})-\\frac{\\mu}{2}\\|y_k-y_{k+1}\\|^2 .\n\\end{equation}\n\nWrite the deviation from the implicit Euler step as\n$$\n\\bs z_{k+1}-\\bs z_k\n= \\alpha \\mathcal G(\\bs z_{k+1})\n+ \\alpha\n\\begin{pmatrix}\ny_k-y_{k+1}+\\beta(\\nabla f(x_{k+1})-\\nabla f(x_k))\\\\\n0\n\\end{pmatrix}.\n$$\n\\medskip\n\\noindent\n\\textbf{The implicit Euler term.} We apply the strong Lyapunov property \\eqref{eq:A-HNAG} at $\\bs z_{k+1}$ to the implicit Euler term\n\\begin{equation*}\n\\langle \\nabla E(\\bs z_{k+1}),\\,\\alpha \\mathcal G(\\bs z_{k+1}) \\rangle \\leq - \\alpha E(\\boldsymbol z_{k+1}) - \\alpha\\beta\\|\\nabla f(x_{k+1})\\|^2-\\frac{\\alpha \\mu}{2}\\|x_{k+1} - y_{k+1}\\|^2.\n\\end{equation*}\n\n\\medskip\n\\noindent\n\\textbf{Lagging of \\(y\\).}\n%\n%This yields two cross terms in the Lyapunov difference:\n%\\begin{align*}\n%\\alpha\\langle \\nabla f(x_{k+1}),\\,y_k-y_{k+1}\\rangle\\quad\n%\\mathrm{and}\\quad \\alpha\\beta\\langle \\nabla f(x_{k+1}),\\,\\nabla f(x_{k+1})-\\nabla f(x_k)\\rangle .\n%\\end{align*}\nFor the term $\\alpha\\langle \\nabla f(x_{k+1}),\\,y_k-y_{k+1}\\rangle$ arising from the lagging of $y$, the Cauchy--Schwarz and Young inequalities give\n\\begin{equation}\\label{eq:firstcross}\n\\alpha\\langle \\nabla f(x_{k+1}),\\,y_k-y_{k+1}\\rangle\n\\le \\frac{\\alpha^2}{2\\mu}\\|\\nabla f(x_{k+1})\\|^2\n+ \\frac{\\mu}{2}\\|y_k-y_{k+1}\\|^2,\n\\end{equation}\nwhose second term cancels with \\eqref{eq:DEHNAG}. \n\n\\medskip\n\\noindent\n\\textbf{Lagging of \\(\\nabla f(x)\\).}\nFor the gradient cross term, arising from the lagging of $\\nabla f(x)$,\nthe identity of squares yields\n$$\n\\begin{aligned}\n\\alpha\\beta\\langle \\nabla f(x_{k+1}),\\,\\nabla f(x_{k+1})-\\nabla f(x_k)\\rangle\n&= -\\frac{\\alpha\\beta}{2}\\|\\nabla f(x_k)\\|^2\n+ \\frac{\\alpha\\beta}{2}\\|\\nabla f(x_{k+1})\\|^2 \\\\\n&\\quad + \\frac{\\alpha\\beta}{2}\\|\\nabla f(x_{k+1})-\\nabla f(x_k)\\|^2 .\n\\end{aligned}\n$$\nCombining and rearranging terms yields \\eqref{eq:onestep}.\n$\\Box$ \\end{proof}\n\n\\begin{theorem}[\\((1-\\sqrt{2/\\kappa})\\)-linear convergence]\\label{thm:linearconvHNAG-funcval}\nLet \\(f\\in\\mathcal S_{\\mu,L}\\), and let \\(E\\) be defined by \\eqref{eq:lyapunov_strong_HNAG}. For the iterates \\((x_k,y_k)\\) generated by the HNAG scheme~\\eqref{eq:HNAG} with\n$$\n\\alpha=\\sqrt{2\\mu/L},\\quad \\alpha\\beta=1/L,\n$$\nor equivalently by Algorithm~\\ref{alg:HNAGpp}, define\n$$\n\\widetilde E_k:=E(x_k,y_k)-\\frac1{2L}\\|\\nabla f(x_k)\\|^2.\n$$\n%By co-coercivity,\n%$$\n%\\widetilde E_k\n%\\geq\n%\\frac{\\mu}{2}\\|y_k-x^\\star\\|^2\n%\\geq0.\n%$$\nThen, for all \\(k\\ge0\\),\n\\begin{equation}\\label{eq:HNAGrate}\n\\frac{\\mu}{2}\\|y_{k+1}-x^\\star\\|^2 \\leq \\widetilde E_{k+1}\n\\le\n\\frac1{1+\\sqrt{2/\\kappa}}\\widetilde E_k\n\\le\n\\widetilde E_0\\left(\\frac1{1+\\sqrt{2/\\kappa}}\\right)^{k+1}.\n\\end{equation}\nMoreover, for all \\(k\\ge1\\),\n\\begin{equation}\\label{eq:HNAG-grad-bound}\n\\|\\nabla f(x_k)\\|^2\n\\le\n\\frac{2L(1+\\alpha)}{\\alpha}\\,\n\\widetilde E_0\n\\left(\\frac1{1+\\alpha}\\right)^k,\n\\end{equation}\nand\n\\begin{equation}\\label{eq:HNAG-strong-components}\n\\|x_k-x^\\star\\|^2\n\\le\n\\frac{2(1+2\\alpha)}{\\alpha\\mu}\\,\n\\widetilde E_0\n\\left(\\frac1{1+\\alpha}\\right)^k .\n\\end{equation}\n\\end{theorem}\n\n\\begin{proof}\nWith \\(\\alpha\\beta=1/L\\) and \\(\\alpha^2=2\\mu/L\\), the one-step estimate \\eqref{eq:onestep} becomes\n$$\n\\begin{aligned}\n(1+\\alpha)E(\\bs z_{k+1})\n\\le{}&\nE(\\bs z_k)\n+\\frac1{2L}\\|\\nabla f(x_{k+1})\\|^2\n-\\frac1{2L}\\|\\nabla f(x_k)\\|^2\\\\\n&\\  -\\frac{\\alpha\\mu}{2}\\|x_{k+1}-y_{k+1}\\|^2 .\n\\end{aligned}\n$$\nEquivalently,\n\\begin{equation}\\label{eq:one-step-energy-E}\n(1+\\alpha)\\widetilde E_{k+1}\n\\le\n\\widetilde E_k\n-\\frac{\\alpha\\mu}{2}\\|x_{k+1}-y_{k+1}\\|^2\n-\\frac{\\alpha}{2L}\\|\\nabla f(x_{k+1})\\|^2 .\n\\end{equation}\nDropping the negative terms gives \\eqref{eq:HNAGrate}.\n\nNext, moving the negative terms in \\eqref{eq:one-step-energy-E} to the left gives\n$$\n(1+\\alpha)\\widetilde E_{k+1}\n+\\frac{\\alpha\\mu}{2}\\|x_{k+1}-y_{k+1}\\|^2\n+\\frac{\\alpha}{2L}\\|\\nabla f(x_{k+1})\\|^2\n\\le\n\\widetilde E_k .\n$$\nShifting the index, we get, for \\(k\\ge1\\),\n$$\n\\frac{\\alpha}{2L}\\|\\nabla f(x_k)\\|^2\n\\le\n\\widetilde E_{k-1}\n\\le\n(1+\\alpha)\\widetilde E_0\\left(\\frac1{1+\\alpha}\\right)^k .\n$$\nThis proves \\eqref{eq:HNAG-grad-bound}. \n%Similarly,\n%\\[\n%\\frac{\\alpha\\mu}{2}\\|x_k-y_k\\|^2\n%\\le\n%\\widetilde E_{k-1}\n%\\le\n%(1+\\alpha)\\widetilde E_0\\left(\\frac1{1+\\alpha}\\right)^k .\n%\\]\n%Moreover, since \\(f\\in\\mathcal S_{\\mu,L}\\), $f(x_k)-f(x^\\star)-\\frac1{2L}\\|\\nabla f(x_k)\\|^2\\ge0,$\n%and hence\n%\\[\n%\\frac{\\mu}{2}\\|y_k-x^\\star\\|^2\n%\\le\n%\\widetilde E_k\n%\\le\n%\\widetilde E_0\\left(\\frac1{1+\\alpha}\\right)^k .\n%\\]\n%Therefore, by the triangle inequality,\n%\\[\n%\\begin{aligned}\n%\\frac{\\alpha\\mu}{2}\\|x_k-x^\\star\\|^2\n%&\\le\n%\\alpha\\mu\\|x_k-y_k\\|^2\n%+\n%\\alpha\\mu\\|y_k-x^\\star\\|^2 \n%\\leq \n%2(1+2\\alpha)\\widetilde E_0\\left(\\frac1{1+\\alpha}\\right)^k .\n%\\end{aligned}\n%\\]\nWe use the $\\mu$-strong convexity of $f$ to conclude \n$$\n\\frac{\\mu}{2}\\|x_k-x^{\\star}\\|^2 \\leq \\widetilde E_k + \\frac1{2L}\\|\\nabla f(x_k)\\|^2\\leq\n\\frac{(1+2\\alpha)}{\\alpha}\\,\n\\widetilde E_0\n\\left(\\frac1{1+\\alpha}\\right)^k .\n$$\nThis gives \\eqref{eq:HNAG-strong-components}.\n$\\Box$\\end{proof}\n\n\\begin{lemma}\\label{lem:energy_identity}\nLet $0\\leq\\hat\\mu\\leq\\mu$ and set\n$\\delta:=(\\mu-\\hat\\mu)/\\mu\\in[0,1]$.\nFor the Lyapunov function $\\mathcal E(\\bs z;\\hat\\mu)$ defined in \\eqref{eq:lya-muhat} and the vector field $\\mathcal G$ defined in \\eqref{eq:G}, we have\n\\begin{equation}\\label{eq:energy_identity}\n\\begin{aligned}\n-\\big\\langle \\nabla \\mathcal E(\\bs z;\\hat\\mu),\\mathcal G(\\bs z)\\big\\rangle\n={}&\n\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n+\\mu\\|y-x^\\star\\|^2\n-\\delta\\mu\\langle y-x^\\star,x-x^\\star\\rangle\n\\\\\n&\\quad\n+\\beta\\|\\nabla f_{-\\hat\\mu}(x)\\|^2\n+\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n\\\\\n\\geq{}&\n\\left(2-\\sqrt{\\delta}-b\\right)\\mathcal E(\\bs z;\\hat\\mu)\n+\\RV{bD_{f_{-\\hat\\mu}}(x,x^\\star)}+\\beta\\|\\nabla f_{-\\hat\\mu}(x)\\|^2\\\\\n&\\quad\n+\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n+\\left(1-\\sqrt{\\delta}\\right)\\Delta_f(x,x^\\star),\n\\end{aligned}\n\\end{equation}\nwhere $0\\leq b<2-\\sqrt{\\delta}$.\n\\end{lemma}\n\n\\begin{proof}\nA direct calculation gives\n$$\n\\begin{aligned}\n-\\big\\langle \\nabla \\mathcal E(\\bs z;\\hat\\mu),\\mathcal G(\\bs z)\\big\\rangle\n={}&\n\\left\\langle\n\\begin{pmatrix}\n\\nabla f_{-\\hat\\mu}(x)\\\\[2pt]\n\\mu(y-x^\\star)\n\\end{pmatrix},\n\\begin{pmatrix}\nx-y+\\beta\\nabla f_{-\\hat\\mu}(x)+\\beta\\hat\\mu(x-x^\\star)\\\\\ny-x^\\star+\\dfrac1\\mu\\nabla f_{-\\hat\\mu}(x)-\\delta(x-x^\\star)\n\\end{pmatrix}\n\\right\\rangle\n\\\\\n={}&\n\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n+\\mu\\|y-x^\\star\\|^2\n-\\delta\\mu\\langle y-x^\\star,x-x^\\star\\rangle\n\\\\\n&\\quad\n+\\beta\\|\\nabla f_{-\\hat\\mu}(x)\\|^2\n+\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle.\n\\end{aligned}\n$$\nThis proves the identity. \\RV{We keep the term $\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle$ separate, since it controls the $\\hat \\mu$-shift term in Lemma~\\ref{lm:decay1}.}\n\nSince $f_{-\\hat\\mu}$ is $\\delta\\mu$-strongly convex,\n\\begin{equation}\\label{eq:deltaconvexity}\nD_{f_{-\\hat\\mu}}(x^\\star,x)\n\\geq\n\\frac{\\mu-\\hat\\mu}{2}\\|x-x^\\star\\|^2\n=\n\\frac{\\delta\\mu}{2}\\|x-x^\\star\\|^2.\n\\end{equation}\nYoung's inequality and \\eqref{eq:deltaconvexity} give\n$$\n\\begin{aligned}\n\\delta\\mu\\langle y-x^\\star,x-x^\\star\\rangle\n&\\leq\n\\sqrt{\\delta}\\,\\frac{\\mu}{2}\\|y-x^\\star\\|^2\n+\n\\frac{\\delta^{3/2}\\mu}{2}\\|x-x^\\star\\|^2\n\\\\\n&\\leq\n\\sqrt{\\delta}\\,\\frac{\\mu}{2}\\|y-x^\\star\\|^2\n+\n\\sqrt{\\delta}\\,D_{f_{-\\hat\\mu}}(x^\\star,x).\n\\end{aligned}\n$$\nTherefore,\n$$\n\\begin{aligned}\n-\\big\\langle \\nabla \\mathcal E(\\bs z;\\hat\\mu),\\mathcal G(\\bs z)\\big\\rangle\n\\geq{}&\n\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n-\n\\sqrt{\\delta}\\,D_{f_{-\\hat\\mu}}(x^\\star,x)\n+\n\\beta\\|\\nabla f_{-\\hat\\mu}(x)\\|^2\n\\\\\n&\\quad\n+\n\\left(2-\\sqrt{\\delta}\\right)\\frac{\\mu}{2}\\|y-x^\\star\\|^2\n+\n\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle.\n\\end{aligned}\n$$\nBy the symmetrization identity \\eqref{eq:symBre} and the invariance of the Bregman asymmetry under quadratic shifts,\n$D_{f_{-\\hat\\mu}}(x^\\star,x)\n=\nD_{f_{-\\hat\\mu}}(x,x^\\star)\n+\n\\Delta_f(x,x^\\star).$\nHence\n$$\n\\begin{aligned}\n&\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n-\n\\sqrt{\\delta}\\,D_{f_{-\\hat\\mu}}(x^\\star,x)\n\\\\\n={}&\nD_{f_{-\\hat\\mu}}(x,x^\\star)\n+\n\\left(1-\\sqrt{\\delta}\\right)D_{f_{-\\hat\\mu}}(x^\\star,x)\n\\\\\n={}&\n\\left(2-\\sqrt{\\delta}\\right)D_{f_{-\\hat\\mu}}(x,x^\\star)\n+\n\\left(1-\\sqrt{\\delta}\\right)\\Delta_f(x,x^\\star).\n\\end{aligned}\n$$\nCombining the first term with $\\frac{\\mu}{2}\\|y-x^\\star\\|^2$ yields\n$$\n\\begin{aligned}\n-\\big\\langle \\nabla \\mathcal E(\\bs z;\\hat\\mu),\\mathcal G(\\bs z)\\big\\rangle\n\\geq{}&\n\\left(2-\\sqrt{\\delta}\\right)\\mathcal E(\\bs z;\\hat\\mu)\n+\n\\beta\\|\\nabla f_{-\\hat\\mu}(x)\\|^2\n\\\\\n&\\quad\n+\n\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n+\n\\left(1-\\sqrt{\\delta}\\right)\\Delta_f(x,x^\\star).\n\\end{aligned}\n$$\nFinally,\n$$\n\\left(2-\\sqrt{\\delta}\\right)\\mathcal E(\\bs z;\\hat\\mu)\n=\n\\left(2-\\sqrt{\\delta}-b\\right)\\mathcal E(\\bs z;\\hat\\mu)\n+\nb\\mathcal E(\\bs z;\\hat\\mu).\n$$\nSince $b\\geq0$, \n% and\n%$$\n%\\mathcal E(\\bs z;\\hat\\mu)\n%\\geq\n%D_{f_{-\\hat\\mu}}(x,x^\\star),\n%$$\nusing\n$$\nb\\mathcal E(\\bs z;\\hat\\mu)\n\\geq\nbD_{f_{-\\hat\\mu}}(x,x^\\star),\n$$\nwe obtain \\eqref{eq:energy_identity}.\n\\end{proof}\n\n\\begin{lemma}\\label{lm:decay1}\nLet $0\\leq\\hat\\mu\\leq\\mu$, and let $(x_k,y_k)$ be generated by the HNAG scheme~\\eqref{eq:HNAG} with\n$\\alpha=\\sqrt{2\\mu/L}$ and $\\alpha\\beta=1/L$.\nThen, for all $k\\geq0$,\n\\begin{equation}\\label{eq:decay1}\n\\begin{aligned}\n(1+c_L\\alpha)\\mathcal E(\\bs z_{k+1};\\hat\\mu)\n\\leq{}&\n\\mathcal E(\\bs z_k;\\hat\\mu)\n+\\frac1{2L}\\|\\nabla f_{-\\hat\\mu}(x_{k+1})\\|^2\n-\\frac1{2L}\\|\\nabla f_{-\\hat\\mu}(x_k)\\|^2\n\\\\\n&\\quad\n-\\alpha\\left[\nd_\\delta\\Delta_f(x_{k+1},x^\\star)\n+\\RV{bD_{f_{-\\hat\\mu}}(x_{k+1},x^\\star)}\n\\right ],\n\\end{aligned}\n\\end{equation}\nwhere\n$$\nb\\geq0,\\quad\n\\delta=\\frac{\\mu-\\hat\\mu}{\\mu}\\in[0,1],\\quad\nc_L=2-\\sqrt{\\delta}-b>0,\\quad\nd_\\delta=1-\\sqrt{\\delta}.\n$$\n\\end{lemma}\n\n\\begin{proof}\nUsing the Bregman expansion,\n$$\n\\mathcal E(\\bs z_{k+1};\\hat\\mu)-\\mathcal E(\\bs z_k;\\hat\\mu)\n=\n\\big\\langle \\nabla \\mathcal E(\\bs z_{k+1};\\hat\\mu),\\bs z_{k+1}-\\bs z_k\\big\\rangle\n-\nD_{\\mathcal E(\\cdot;\\hat\\mu)}(\\bs z_k,\\bs z_{k+1}).\n$$\nLet $g_k:=\\nabla f_{-\\hat\\mu}(x_k)$. \\RV{Since $-\\beta\\nabla f(x_{k+1})$ is used in $\\mathcal G(\\bs z_{k+1})$ and\n$$\n\\beta(g_{k+1}-g_k)+\\beta\\hat\\mu(x_{k+1}-x_k)\n=\n\\beta\\bigl(\\nabla f(x_{k+1})-\\nabla f(x_k)\\bigr),\n$$}\nthe update can be written as\n$$\n\\bs z_{k+1}-\\bs z_k\n=\n\\alpha\\mathcal G(\\bs z_{k+1})\n+\n\\alpha\n\\binom{\n(y_k-y_{k+1})+\\beta(g_{k+1}-g_k)+\\beta\\hat\\mu(x_{k+1}-x_k)\n}{0}.\n$$\n\nThe implicit Euler part is controlled by Lemma~\\ref{lem:energy_identity}:\n\\begin{equation}\\label{eq:IE-HNAGpp}\n\\begin{aligned}\n&\\alpha\\big\\langle \\nabla \\mathcal E(\\bs z_{k+1};\\hat\\mu),\\mathcal G(\\bs z_{k+1})\\big\\rangle \\\\\n\\leq{}&\n-c_L\\alpha\\mathcal E(\\bs z_{k+1};\\hat\\mu)\n-\\alpha\\beta\\hat\\mu\\langle g_{k+1},x_{k+1}-x^\\star\\rangle\n\\\\\n&\\quad\n-\\alpha\\beta\\|g_{k+1}\\|^2\n-\\alpha\\left [\nd_\\delta\\Delta_f(x_{k+1},x^\\star)\n+\\RV{bD_{f_{-\\hat\\mu}}(x_{k+1},x^\\star)}\n\\right ].\n\\end{aligned}\n\\end{equation}\n\n\\medskip\n\\noindent\n\\textbf{The $\\hat\\mu$-shift term.}\nBy the three-point Bregman identity~\\eqref{eq:Bregmanidentity},\n$$\n\\begin{aligned}\n&\\alpha\\beta\\hat\\mu\\langle g_{k+1},x_{k+1}-x_k\\rangle\n\\\\\n={}&\n\\alpha\\beta\\hat\\mu\\Bigl(\nD_{f_{-\\hat\\mu}}(x_{k+1},x^\\star)\n+\nD_{f_{-\\hat\\mu}}(x_k,x_{k+1})\n-\nD_{f_{-\\hat\\mu}}(x_k,x^\\star)\n\\Bigr).\n\\end{aligned}\n$$\n%Using\n%$$\n%\\langle g_{k+1},x_{k+1}-x^\\star\\rangle\n%=\n%D_{f_{-\\hat\\mu}}(x_{k+1},x^\\star)\n%+\n%D_{f_{-\\hat\\mu}}(x^\\star,x_{k+1}),\n%$$\nThe first term cancels the corresponding part of the shift term in \\eqref{eq:IE-HNAGpp}, leaving a nonpositive term, which we may discard.\n%$$\n%-\\alpha\\beta\\hat\\mu D_{f_{-\\hat\\mu}}(x^\\star,x_{k+1})\\leq0.\n%$$\nThe second term combines with\n$-D_{\\mathcal E(\\cdot;\\hat\\mu)}(\\bs z_k,\\bs z_{k+1})$ to give\n$$\n-(1-\\alpha\\beta\\hat\\mu)D_{f_{-\\hat\\mu}}(x_k,x_{k+1}).\n$$\nThe last term is also nonpositive and may be discarded.\n\n\\medskip\n\\noindent\n\\textbf{Lagging of $y$.}\nBy Cauchy--Schwarz and Young's inequality,\n$$\n\\alpha\\langle g_{k+1},y_k-y_{k+1}\\rangle\n\\leq\n\\frac{\\alpha^2}{2\\mu}\\|g_{k+1}\\|^2\n+\n\\frac{\\mu}{2}\\|y_k-y_{k+1}\\|^2.\n$$\nThe second term cancels the $y$-component of\n$-D_{\\mathcal E(\\cdot;\\hat\\mu)}(\\bs z_k,\\bs z_{k+1})$.\n\n\\medskip\n\\noindent\n\\textbf{Lagging of the gradient.}\nThe identity of squares gives\n$$\n\\begin{aligned}\n\\alpha\\beta\\langle g_{k+1},g_{k+1}-g_k\\rangle\n={}&\n-\\frac{\\alpha\\beta}{2}\\|g_k\\|^2\n+\\frac{\\alpha\\beta}{2}\\|g_{k+1}\\|^2\n+\\frac{\\alpha\\beta}{2}\\|g_{k+1}-g_k\\|^2.\n\\end{aligned}\n$$\nSince $f_{-\\hat\\mu}$ is $(L-\\hat\\mu)$-smooth and $\\alpha\\beta=1/L$,\n$$\n\\frac{\\alpha\\beta}{2}\\|g_{k+1}-g_k\\|^2\n\\leq\n(1-\\alpha\\beta\\hat\\mu)D_{f_{-\\hat\\mu}}(x_k,x_{k+1}).\n$$\nTherefore,\n$$\n\\begin{aligned}\n&\\alpha\\beta\\langle g_{k+1},g_{k+1}-g_k\\rangle\n-\n(1-\\alpha\\beta\\hat\\mu)D_{f_{-\\hat\\mu}}(x_k,x_{k+1})\n\\\\\n\\leq{}&\n-\\frac{\\alpha\\beta}{2}\\|g_k\\|^2\n+\n\\frac{\\alpha\\beta}{2}\\|g_{k+1}\\|^2.\n\\end{aligned}\n$$\n\nCombining the above estimates gives\n$$\n\\begin{aligned}\n(1+c_L\\alpha)\\mathcal E(\\bs z_{k+1};\\hat\\mu)\n\\leq{}&\n\\mathcal E(\\bs z_k;\\hat\\mu)\n+\n\\left(\n\\frac{\\alpha^2}{2\\mu}\n-\\frac{\\alpha\\beta}{2}\n\\right)\\|g_{k+1}\\|^2\n-\n\\frac{\\alpha\\beta}{2}\\|g_k\\|^2\n\\\\\n&\\quad\n-\\alpha bD_{f_{-\\hat\\mu}}(x_{k+1},x^\\star)\n-\\alpha d_\\delta\\Delta_f(x_{k+1},x^\\star).\n\\end{aligned}\n$$\nFinally, using $\\alpha^2=2\\mu/L$ and $\\alpha\\beta=1/L$, we have\n%$$\n%\\frac{\\alpha^2}{2\\mu}-\\frac{\\alpha\\beta}{2}\n%=\n%\\frac1{2L},\n%\\qquad\n%\\frac{\\alpha\\beta}{2}\n%=\n%\\frac1{2L}.\n%$$\nthe desired result \\eqref{eq:decay1}. $\\Box$\n\\end{proof}\n\n\\begin{lemma}\\label{lm:gradient_norm_ineq}\nFor all \\(k\\ge 0\\),\n$$\n\\|\\nabla f_k(x_{k+1})\\|^2\n\\le\n\\|\\nabla f_{k+1}(x_{k+1})\\|^2\n+\nL\\Delta\\delta_k\\alpha^2\n\\langle \\nabla f_k(x_{k+1}),x_{k+1}-x^\\star\\rangle.\n$$\n\\end{lemma}\n\n\\begin{proof}\nBy definition,\n$$\n\\nabla f_{k+1}(x_{k+1})\n=\n\\nabla f_k(x_{k+1})-(\\mu_{k+1}-\\mu_k)(x_{k+1}-x^\\star).\n$$\nExpanding the square and dropping the positive last term, we obtain\n$$\n\\begin{aligned}\n\\|\\nabla f_{k+1}(x_{k+1})\\|^2\n&=\n\\bigl\\|\\nabla f_k(x_{k+1})-(\\mu_{k+1}-\\mu_k)(x_{k+1}-x^\\star)\\bigr\\|^2\\\\\n&\\ge\n\\|\\nabla f_k(x_{k+1})\\|^2\n-\n2(\\mu_{k+1}-\\mu_k)\n\\langle \\nabla f_k(x_{k+1}),x_{k+1}-x^\\star\\rangle.\n\\end{aligned}\n$$\nSince $\\mu_{k+1}-\\mu_k=\\Delta\\delta_k\\,\\mu, \n\\alpha^2=2\\mu/L,$\nwe have\n$$\n2(\\mu_{k+1}-\\mu_k)=L\\Delta\\delta_k\\alpha^2.\n$$\nThis proves the claim.\n$\\Box$\n\\end{proof}\n\nBy Theorem~\\ref{thm:linearconvHNAG-funcval},\n$$\n\\|x_{k+1}-x^\\star\\|\n\\leq\nC_1\\left(1+\\sqrt{2\\rho}\\right)^{-\\frac{k+1}{2}},\n\\qquad\nC_1:=\n\\left(\n\\frac{2(1+2\\alpha)}{\\alpha\\mu}\\widetilde E_0\n\\right)^{1/2}.\n$$\n\n\nSince $\\omega$ is nondecreasing and its argument decreases to zero, $\\{\\omega_k\\}$ is nonincreasing and converges to zero.\n\n\n\\begin{lemma}\\label{lm:shifted_energy_recursion}\nAssume $f\\in\\mathcal S_{\\mu,L}^{\\rm LAS}(x^\\star)$. Let $(x_k,y_k)$ be generated by Algorithm~\\ref{alg:HNAGpp}. Let $\\{\\delta_k\\}\\subset[0,1]$ be nonincreasing with $\\delta_k\\to0$, and let $\\{b_k\\}$ be nonnegative with $b_k\\to0$. Set\n$$\n\\mu_k=(1-\\delta_k)\\mu,\\quad\nf_k:=f_{-\\mu_k},\\quad\nd_k:=1-\\sqrt{\\delta_k},\\quad\n\\Delta\\delta_k:=\\delta_k-\\delta_{k+1}\\geq0,\n$$\nand define\n$$\n\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k)\n:=\n\\mathcal E(\\bs z_k;\\mu_k)\n-\\frac1{2L}\\|\\nabla f_k(x_k)\\|^2\\geq 0.\n$$\n%\\RV{Since $f_k$ is convex and $L$-smooth, co-coercivity gives\n%$$\n%D_{f_k}(x_k,x^\\star)\n%\\geq\n%\\frac1{2L}\\|\\nabla f_k(x_k)\\|^2.\n%$$\n%Consequently,\n%$$\n%\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k)\n%\\geq\n%\\frac{\\mu}{2}\\|y_k-x^\\star\\|^2\n%\\geq0.\n%$$}\nLet $\\omega_k$ be defined by \\eqref{eq:def-omegak}, and set\n$$\nc_k:=2-\\sqrt{\\delta_k}-b_k\\geq0,\n\\qquad\nr_k:=\n\\left(1+c_k\\alpha-\\Delta\\delta_k\\alpha^2\\right)^{-1}.\n$$\nAssume $r_k^{-1}\\geq1$ and\n\\begin{equation}\\label{eq:omega_condition}\n\\left|\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\right|\\omega_k\n\\leq\nb_k\\delta_k.\n\\end{equation}\nThen\n$$\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\nr_k\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k).\n$$\n\\end{lemma}\n\n\\begin{proof}\nApply Lemma~\\ref{lm:decay1} with $\\hat\\mu=\\mu_k$. Then\n$$\n\\begin{aligned}\n(1+c_k\\alpha)\\mathcal E(\\bs z_{k+1};\\mu_k)\n\\leq{}&\n\\mathcal E(\\bs z_k;\\mu_k)\n+\\frac1{2L}\\|\\nabla f_k(x_{k+1})\\|^2\n-\\frac1{2L}\\|\\nabla f_k(x_k)\\|^2\n\\\\\n&\\quad\n-\\alpha\\left(\nd_k\\Delta_f(x_{k+1},x^\\star)\n+\nb_kD_{f_k}(x_{k+1},x^\\star)\n\\right).\n\\end{aligned}\n$$\nWe first change the shifted gradient using Lemma~\\ref{lm:gradient_norm_ineq}:\n$$\n\\frac1{2L}\\|\\nabla f_k(x_{k+1})\\|^2\n\\leq\n\\frac1{2L}\\|\\nabla f_{k+1}(x_{k+1})\\|^2\n+\n\\frac{\\Delta\\delta_k\\alpha^2}{2}\n\\langle\\nabla f_k(x_{k+1}),x_{k+1}-x^\\star\\rangle.\n$$\nUsing\n$$\n\\begin{aligned}\n\\frac{\\Delta\\delta_k\\alpha^2}{2}\n\\langle\\nabla f_k(x_{k+1}),x_{k+1}-x^\\star\\rangle\n={}&\n\\Delta\\delta_k\\alpha^2D_{f_k}(x_{k+1},x^\\star)\n+\\frac{\\Delta\\delta_k\\alpha^2}{2}\n\\Delta_f(x_{k+1},x^\\star),\n\\end{aligned}\n$$\nand\n$$\n\\Delta\\delta_k\\alpha^2D_{f_k}(x_{k+1},x^\\star)\n\\leq\n\\Delta\\delta_k\\alpha^2\\mathcal E(\\bs z_{k+1};\\mu_k),\n$$\nthe first term can be absorbed into the left-hand side, which slightly weakens the contraction factor from $1+c_k\\alpha$ to $r_k$. \nWe then obtain\n$$\n\\begin{aligned}\nr_k^{-1}\\mathcal E(\\bs z_{k+1};\\mu_k)\n\\leq{}&\n\\mathcal E(\\bs z_k;\\mu_k)\n+\\frac1{2L}\\|\\nabla f_{k+1}(x_{k+1})\\|^2\n-\\frac1{2L}\\|\\nabla f_k(x_k)\\|^2\n\\\\\n&\\quad\n-\\alpha\\left[\n\\left(\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\right)\\Delta_f(x_{k+1},x^\\star)\n+\nb_kD_{f_k}(x_{k+1},x^\\star)\n\\right].\n\\end{aligned}\n$$\nSince $\\mu_{k+1}\\geq\\mu_k$,\n$$\n\\mathcal E(\\bs z_{k+1};\\mu_{k+1})\n\\leq\n\\mathcal E(\\bs z_{k+1};\\mu_k).\n$$\nThus the same inequality holds with $\\mathcal E(\\bs z_{k+1};\\mu_{k+1})$ on the left-hand side.\n\nBy the definition of $\\omega_k$,\n$$\n|\\Delta_f(x_{k+1},x^\\star)|\n\\leq\n\\omega_kD_f(x_{k+1},x^\\star).\n$$\nMoreover, since $\\mu_k=(1-\\delta_k)\\mu$ and $f$ is $\\mu$-strongly convex,\n$$\n\\begin{aligned}\nD_{f_k}(x_{k+1},x^\\star)\n&=\nD_f(x_{k+1},x^\\star)\n-\\frac{\\mu_k}{2}\\|x_{k+1}-x^\\star\\|^2\n\\\\\n&\\geq\nD_f(x_{k+1},x^\\star)\n-\\frac{\\mu_k}{\\mu}D_f(x_{k+1},x^\\star)\n=\n\\delta_kD_f(x_{k+1},x^\\star).\n\\end{aligned}\n$$\nTherefore,\n$$\n\\begin{aligned}\n&\n\\left(\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\right)\\Delta_f(x_{k+1},x^\\star)\n+\nb_kD_{f_k}(x_{k+1},x^\\star)\n\\\\\n&\\geq\n\\left[\nb_k\\delta_k\n-\n\\left|\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\right|\\omega_k\n\\right]\nD_f(x_{k+1},x^\\star)\n\\geq0,\n\\end{aligned}\n$$\nwhere the last inequality follows from \\eqref{eq:omega_condition}. It follows that\n$$\n\\begin{aligned}\nr_k^{-1}\\mathcal E(\\bs z_{k+1};\\mu_{k+1})\n\\leq{}&\n\\mathcal E(\\bs z_k;\\mu_k)\n+\\frac1{2L}\\|\\nabla f_{k+1}(x_{k+1})\\|^2\n-\\frac1{2L}\\|\\nabla f_k(x_k)\\|^2.\n\\end{aligned}\n$$\nSubtracting $r_k^{-1}(2L)^{-1}\\|\\nabla f_{k+1}(x_{k+1})\\|^2$ from both sides gives\n$$\n\\begin{aligned}\nr_k^{-1}\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq{}&\n\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k)\n+\n\\frac{1-r_k^{-1}}{2L}\n\\|\\nabla f_{k+1}(x_{k+1})\\|^2.\n\\end{aligned}\n$$\nSince $r_k^{-1}\\geq1$, the last term is nonpositive. Therefore,\n$$\nr_k^{-1}\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\n\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k),\n$$\nwhich proves the claim.\n\\end{proof}\n\n\\begin{proof}\nSince $f\\in\\mathcal S_{\\mu,L}^{\\rm LAS}(x^\\star)$, we have $\\omega_k\\downarrow0$, so the index $K$ in the theorem statement is finite. The threshold $1/8$ is only a convenient choice ensuring $q_k\\leq1/2$; any fixed sufficiently small threshold would work.\n\n\\medskip\n\\noindent\n\\textbf{Step 1: Initial finite segment.}\nFor $0\\leq k<K$, we use the unshifted choice\n$$\n\\delta_k=1,\\qquad\n\\mu_k=0,\\qquad\nb_k=0,\\qquad\nd_k=0,\\qquad\nc_k=1.\n$$\nThen\n$$\n\\widetilde{\\mathcal E}(\\bs z_{k+1};0)\n\\leq\n(1+\\alpha)^{-1}\n\\widetilde{\\mathcal E}(\\bs z_k;0),\n\\qquad\n0\\leq k<K.\n$$\n\n\\medskip\n\\noindent\n\\textbf{Step 2: Choice of the partial shifts.}\nFor $k\\geq K$, set\n$$\nq_k:=\\omega_k^{1/3},\\qquad\n\\delta_k:=q_k^2,\\qquad\nd_k:=1-q_k,\\qquad\nb_k:=d_kq_k.\n$$\nThis choice gives\n$$\nb_k\\delta_k=d_k\\omega_k.\n$$\nSince $\\omega_k\\leq1/8$, we have $0\\leq q_k\\leq1/2$, and hence $d_k,b_k\\geq0$. Moreover,\n$$\n\\begin{aligned}\nc_k\n&:=\n2-\\sqrt{\\delta_k}-b_k\n=\n1+(1-q_k)^2\n\\geq\\frac54,\\\\\n&q_k\\to0,\\qquad\n\\delta_k\\downarrow0,\\qquad\nb_k\\to0,\\qquad\nc_k\\to2.\n\\end{aligned}\n$$\n\n\\medskip\n\\noindent\n\\textbf{Step 3: Verification of Lemma~\\ref{lm:shifted_energy_recursion}.}\nSet\n$$\nr_k\n:=\n\\left(1+c_k\\alpha-\\Delta\\delta_k\\alpha^2\\right)^{-1}.\n$$\nSince\n$$\nc_k\\geq\\frac54,\n\\qquad\n0\\leq\\Delta\\delta_k\\leq\\delta_k=q_k^2\\leq\\frac14,\n$$\nand $\\kappa=L/\\mu\\geq1$ implies $\\alpha=\\sqrt{2/\\kappa}\\leq\\sqrt2$, we have\n\\begin{equation*}\n\\begin{aligned}\nr_k^{-1}\n&=\n1+c_k\\alpha-\\Delta\\delta_k\\alpha^2\n\\geq\n1+\\frac54\\alpha-\\frac14\\alpha^2\n=\n1+\\frac{\\alpha}{4}(5-\\alpha)\n>1.\n\\end{aligned}\n\\end{equation*}\n\nIt remains to verify \\eqref{eq:omega_condition}. Since $d_k\\geq1/2$,\n$$\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\geq\n\\frac12-\\frac{\\sqrt2}{8}\n>0.\n$$\n%Moreover,\n%$$\n%d_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n%\\leq\n%d_k.\n%$$\nTherefore,\n\\begin{equation*}\n\\left|\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\right|\\omega_k\n\\leq\nd_k\\omega_k\n=\nb_k\\delta_k.\n\\end{equation*}\nLemma~\\ref{lm:shifted_energy_recursion} now gives\n$$\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\nr_k\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k),\n\\qquad\nk\\geq K.\n$$\nHence\n$$\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\n\\widetilde{\\mathcal E}(\\bs z_K;\\mu_K)\n\\prod_{i=K}^{k}r_i.\n$$\nThe finitely many initial terms can be absorbed into the constant. Specifically, let\n$$\nC\n:=\n1+\n\\max_{0\\leq j\\leq K}\n\\left\\{\n\\widetilde{\\mathcal E}(\\bs z_{j+1};\\mu_{j+1})\n\\left(\\prod_{i=0}^{j}r_i\\right)^{-1}\n\\right\\},\n$$\nThen, for all $k\\geq0$,\n$$\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\nC\\prod_{i=0}^{k}r_i.\n$$\n\n\\medskip\n\\noindent\n\\textbf{Step 4: Asymptotic rate.}\nSince $c_k\\to2$ and $\\Delta\\delta_k\\to0$,\n$$\nr_k\n\\to\nr_\\ast\n:=\n\\left(1+2\\sqrt{2/\\kappa}\\right)^{-1}.\n$$\nTherefore,\n%$$\n%\\left(\\prod_{i=0}^{k}r_i\\right)^{1/k}\n%\\to\n%r_\\ast.\n%$$\n%It follows that\n$$\n\\begin{aligned}\n\\limsup_{k\\to\\infty}\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})^{1/k}\n&\\leq\n\\limsup_{k\\to\\infty}\n\\left(\nC\\prod_{i=0}^{k}r_i\n\\right)^{1/k}\n=\nr_\\ast.\n\\end{aligned}\n$$\nHence $\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k)$ converges $R$-linearly with asymptotic rate $r_\\ast$. \\RV{The bound in the theorem then gives the same asymptotic rate for $\\|y_k-x^\\star\\|^2$.} $\\Box$\n\\end{proof}","units":[{"start":20949,"end":21344,"start_line":437,"end_line":445,"raw_tex":"\\begin{lemma}\\label{lem:cont_strong}\nLet $\\mathcal G(\\boldsymbol z)$ be the vector field of the HNAG flow~\\eqref{eq:flow}.\nThen $E(\\boldsymbol z)$ satisfies the following strong Lyapunov condition\n\\begin{equation}\\label{eq:A-HNAG}\n-\\langle\\nabla E(\\boldsymbol z),\\mathcal G(\\boldsymbol z)\\rangle\n\\;\\ge\\;\nE(\\boldsymbol z)+\\beta\\|\\nabla f(x)\\|^2+\\frac{\\mu}{2}\\|x-y\\|^2 .\n\\end{equation}\n\\end{lemma}","sha256_utf8":"89d6e69e428c231a07f911f47c25795aad26a3c77c6ba077eb808a53b2bca626"},{"start":21346,"end":21925,"start_line":447,"end_line":467,"raw_tex":"\\begin{proof}\nA direct computation yields\n$$\n\\begin{aligned}\n&\\quad -\\langle\\nabla E(\\boldsymbol z),\\mathcal G(\\boldsymbol z)\\rangle \\\\ \n&=\n\\dual{\\nabla f(x),x-x^\\star}\n+\\beta\\|\\nabla f(x)\\|^2\n+\\mu\\|y-x^\\star\\|^2\n-\\mu\\dual{y-x^\\star,x-x^\\star} \\\\\n&=\nE(\\boldsymbol z)\n+D_f(x^\\star,x)\n+\\beta\\|\\nabla f(x)\\|^2\n+\\frac{\\mu}{2}\\|x-y\\|^2\n-\\frac{\\mu}{2}\\|x-x^\\star\\|^2,\n\\end{aligned}\n$$\nwhere we use identities of squares~\\eqref{eq:squares} to expand the cross term. \nUsing the $\\mu$-convexity $D_f(x^\\star,x)\\ge \\tfrac{\\mu}{2}\\|x-x^\\star\\|^2$ gives \\eqref{eq:A-HNAG}.\n$\\Box$ \\end{proof}","sha256_utf8":"72840eb651fc14242fec17786c90555e53bfadf1d82431ff90239509839590db"},{"start":24043,"end":24594,"start_line":499,"end_line":510,"raw_tex":"\\begin{lemma}\\label{lem:disc_strong}\nLet $(x_k,y_k)$ be the iterates generated by~\\eqref{eq:HNAG} and $E$ defined by \\eqref{eq:lyapunov_strong_HNAG}. Then\n\\begin{equation}\\label{eq:onestep}\n\\begin{aligned}\n(1+\\alpha)E(\\bs z_{k+1}) \\;\\le\\;& E(\\bs z_k)\n+ \\Bigl(\\frac{\\alpha^2}{2\\mu}-\\frac{\\alpha\\beta}{2}\\Bigr)\\|\\nabla f(x_{k+1})\\|^2\n- \\frac{\\alpha\\beta}{2}\\|\\nabla f(x_k)\\|^2 \\\\\n\\qquad-\\frac{\\alpha \\mu}{2}\\|x_{k+1} - y_{k+1}\\|^2 &+ \\frac{\\alpha\\beta}{2}\\|\\nabla f(x_{k+1})-\\nabla f(x_k)\\|^2\n- D_f(x_k,x_{k+1}).\n\\end{aligned}\n\\end{equation}\n\\end{lemma}","sha256_utf8":"e4e860b3359e837c4f8211f708044da4b95fa8d5d2dd70bd2c6d946e28271a59"},{"start":24595,"end":26750,"start_line":511,"end_line":572,"raw_tex":"\\begin{proof}\nExpand the difference of $E$ by the definition of the Bregman divergence:\n\\begin{equation*}\nE(\\bs z_{k+1})-E(\\bs z_k)\n= \\langle \\nabla E(\\bs z_{k+1}),\\,\\bs z_{k+1}-\\bs z_k\\rangle\n- D_E(\\bs z_k,\\bs z_{k+1}),\n\\end{equation*}\nwhere\n\\begin{equation}\\label{eq:DEHNAG}\n- D_E(\\bs z_k,\\bs z_{k+1})\n= -D_f(x_k,x_{k+1})-\\frac{\\mu}{2}\\|y_k-y_{k+1}\\|^2 .\n\\end{equation}\n\nWrite the deviation from the implicit Euler step as\n$$\n\\bs z_{k+1}-\\bs z_k\n= \\alpha \\mathcal G(\\bs z_{k+1})\n+ \\alpha\n\\begin{pmatrix}\ny_k-y_{k+1}+\\beta(\\nabla f(x_{k+1})-\\nabla f(x_k))\\\\\n0\n\\end{pmatrix}.\n$$\n\\medskip\n\\noindent\n\\textbf{The implicit Euler term.} We apply the strong Lyapunov property \\eqref{eq:A-HNAG} at $\\bs z_{k+1}$ to the implicit Euler term\n\\begin{equation*}\n\\langle \\nabla E(\\bs z_{k+1}),\\,\\alpha \\mathcal G(\\bs z_{k+1}) \\rangle \\leq - \\alpha E(\\boldsymbol z_{k+1}) - \\alpha\\beta\\|\\nabla f(x_{k+1})\\|^2-\\frac{\\alpha \\mu}{2}\\|x_{k+1} - y_{k+1}\\|^2.\n\\end{equation*}\n\n\\medskip\n\\noindent\n\\textbf{Lagging of \\(y\\).}\n%\n%This yields two cross terms in the Lyapunov difference:\n%\\begin{align*}\n%\\alpha\\langle \\nabla f(x_{k+1}),\\,y_k-y_{k+1}\\rangle\\quad\n%\\mathrm{and}\\quad \\alpha\\beta\\langle \\nabla f(x_{k+1}),\\,\\nabla f(x_{k+1})-\\nabla f(x_k)\\rangle .\n%\\end{align*}\nFor the term $\\alpha\\langle \\nabla f(x_{k+1}),\\,y_k-y_{k+1}\\rangle$ arising from the lagging of $y$, the Cauchy--Schwarz and Young inequalities give\n\\begin{equation}\\label{eq:firstcross}\n\\alpha\\langle \\nabla f(x_{k+1}),\\,y_k-y_{k+1}\\rangle\n\\le \\frac{\\alpha^2}{2\\mu}\\|\\nabla f(x_{k+1})\\|^2\n+ \\frac{\\mu}{2}\\|y_k-y_{k+1}\\|^2,\n\\end{equation}\nwhose second term cancels with \\eqref{eq:DEHNAG}. \n\n\\medskip\n\\noindent\n\\textbf{Lagging of \\(\\nabla f(x)\\).}\nFor the gradient cross term, arising from the lagging of $\\nabla f(x)$,\nthe identity of squares yields\n$$\n\\begin{aligned}\n\\alpha\\beta\\langle \\nabla f(x_{k+1}),\\,\\nabla f(x_{k+1})-\\nabla f(x_k)\\rangle\n&= -\\frac{\\alpha\\beta}{2}\\|\\nabla f(x_k)\\|^2\n+ \\frac{\\alpha\\beta}{2}\\|\\nabla f(x_{k+1})\\|^2 \\\\\n&\\quad + \\frac{\\alpha\\beta}{2}\\|\\nabla f(x_{k+1})-\\nabla f(x_k)\\|^2 .\n\\end{aligned}\n$$\nCombining and rearranging terms yields \\eqref{eq:onestep}.\n$\\Box$ \\end{proof}","sha256_utf8":"ef0f0f2a8235da1f56d2ead1e64a1ed84459a6eaf724c79f98b7727240f6d3dc"},{"start":65547,"end":66690,"start_line":1664,"end_line":1704,"raw_tex":"\\begin{theorem}[\\((1-\\sqrt{2/\\kappa})\\)-linear convergence]\\label{thm:linearconvHNAG-funcval}\nLet \\(f\\in\\mathcal S_{\\mu,L}\\), and let \\(E\\) be defined by \\eqref{eq:lyapunov_strong_HNAG}. For the iterates \\((x_k,y_k)\\) generated by the HNAG scheme~\\eqref{eq:HNAG} with\n$$\n\\alpha=\\sqrt{2\\mu/L},\\quad \\alpha\\beta=1/L,\n$$\nor equivalently by Algorithm~\\ref{alg:HNAGpp}, define\n$$\n\\widetilde E_k:=E(x_k,y_k)-\\frac1{2L}\\|\\nabla f(x_k)\\|^2.\n$$\n%By co-coercivity,\n%$$\n%\\widetilde E_k\n%\\geq\n%\\frac{\\mu}{2}\\|y_k-x^\\star\\|^2\n%\\geq0.\n%$$\nThen, for all \\(k\\ge0\\),\n\\begin{equation}\\label{eq:HNAGrate}\n\\frac{\\mu}{2}\\|y_{k+1}-x^\\star\\|^2 \\leq \\widetilde E_{k+1}\n\\le\n\\frac1{1+\\sqrt{2/\\kappa}}\\widetilde E_k\n\\le\n\\widetilde E_0\\left(\\frac1{1+\\sqrt{2/\\kappa}}\\right)^{k+1}.\n\\end{equation}\nMoreover, for all \\(k\\ge1\\),\n\\begin{equation}\\label{eq:HNAG-grad-bound}\n\\|\\nabla f(x_k)\\|^2\n\\le\n\\frac{2L(1+\\alpha)}{\\alpha}\\,\n\\widetilde E_0\n\\left(\\frac1{1+\\alpha}\\right)^k,\n\\end{equation}\nand\n\\begin{equation}\\label{eq:HNAG-strong-components}\n\\|x_k-x^\\star\\|^2\n\\le\n\\frac{2(1+2\\alpha)}{\\alpha\\mu}\\,\n\\widetilde E_0\n\\left(\\frac1{1+\\alpha}\\right)^k .\n\\end{equation}\n\\end{theorem}","sha256_utf8":"d45b9fd77183964397bafc24c8536225de1dfbd4485fad785a2ae96c8155dbd5"},{"start":66692,"end":68605,"start_line":1706,"end_line":1782,"raw_tex":"\\begin{proof}\nWith \\(\\alpha\\beta=1/L\\) and \\(\\alpha^2=2\\mu/L\\), the one-step estimate \\eqref{eq:onestep} becomes\n$$\n\\begin{aligned}\n(1+\\alpha)E(\\bs z_{k+1})\n\\le{}&\nE(\\bs z_k)\n+\\frac1{2L}\\|\\nabla f(x_{k+1})\\|^2\n-\\frac1{2L}\\|\\nabla f(x_k)\\|^2\\\\\n&\\  -\\frac{\\alpha\\mu}{2}\\|x_{k+1}-y_{k+1}\\|^2 .\n\\end{aligned}\n$$\nEquivalently,\n\\begin{equation}\\label{eq:one-step-energy-E}\n(1+\\alpha)\\widetilde E_{k+1}\n\\le\n\\widetilde E_k\n-\\frac{\\alpha\\mu}{2}\\|x_{k+1}-y_{k+1}\\|^2\n-\\frac{\\alpha}{2L}\\|\\nabla f(x_{k+1})\\|^2 .\n\\end{equation}\nDropping the negative terms gives \\eqref{eq:HNAGrate}.\n\nNext, moving the negative terms in \\eqref{eq:one-step-energy-E} to the left gives\n$$\n(1+\\alpha)\\widetilde E_{k+1}\n+\\frac{\\alpha\\mu}{2}\\|x_{k+1}-y_{k+1}\\|^2\n+\\frac{\\alpha}{2L}\\|\\nabla f(x_{k+1})\\|^2\n\\le\n\\widetilde E_k .\n$$\nShifting the index, we get, for \\(k\\ge1\\),\n$$\n\\frac{\\alpha}{2L}\\|\\nabla f(x_k)\\|^2\n\\le\n\\widetilde E_{k-1}\n\\le\n(1+\\alpha)\\widetilde E_0\\left(\\frac1{1+\\alpha}\\right)^k .\n$$\nThis proves \\eqref{eq:HNAG-grad-bound}. \n%Similarly,\n%\\[\n%\\frac{\\alpha\\mu}{2}\\|x_k-y_k\\|^2\n%\\le\n%\\widetilde E_{k-1}\n%\\le\n%(1+\\alpha)\\widetilde E_0\\left(\\frac1{1+\\alpha}\\right)^k .\n%\\]\n%Moreover, since \\(f\\in\\mathcal S_{\\mu,L}\\), $f(x_k)-f(x^\\star)-\\frac1{2L}\\|\\nabla f(x_k)\\|^2\\ge0,$\n%and hence\n%\\[\n%\\frac{\\mu}{2}\\|y_k-x^\\star\\|^2\n%\\le\n%\\widetilde E_k\n%\\le\n%\\widetilde E_0\\left(\\frac1{1+\\alpha}\\right)^k .\n%\\]\n%Therefore, by the triangle inequality,\n%\\[\n%\\begin{aligned}\n%\\frac{\\alpha\\mu}{2}\\|x_k-x^\\star\\|^2\n%&\\le\n%\\alpha\\mu\\|x_k-y_k\\|^2\n%+\n%\\alpha\\mu\\|y_k-x^\\star\\|^2 \n%\\leq \n%2(1+2\\alpha)\\widetilde E_0\\left(\\frac1{1+\\alpha}\\right)^k .\n%\\end{aligned}\n%\\]\nWe use the $\\mu$-strong convexity of $f$ to conclude \n$$\n\\frac{\\mu}{2}\\|x_k-x^{\\star}\\|^2 \\leq \\widetilde E_k + \\frac1{2L}\\|\\nabla f(x_k)\\|^2\\leq\n\\frac{(1+2\\alpha)}{\\alpha}\\,\n\\widetilde E_0\n\\left(\\frac1{1+\\alpha}\\right)^k .\n$$\nThis gives \\eqref{eq:HNAG-strong-components}.\n$\\Box$\\end{proof}","sha256_utf8":"1c18e7cffd8ac837f8546f55e276301879f8d3c7dee81379b933aa8588756393"},{"start":72631,"end":73583,"start_line":1869,"end_line":1894,"raw_tex":"\\begin{lemma}\\label{lem:energy_identity}\nLet $0\\leq\\hat\\mu\\leq\\mu$ and set\n$\\delta:=(\\mu-\\hat\\mu)/\\mu\\in[0,1]$.\nFor the Lyapunov function $\\mathcal E(\\bs z;\\hat\\mu)$ defined in \\eqref{eq:lya-muhat} and the vector field $\\mathcal G$ defined in \\eqref{eq:G}, we have\n\\begin{equation}\\label{eq:energy_identity}\n\\begin{aligned}\n-\\big\\langle \\nabla \\mathcal E(\\bs z;\\hat\\mu),\\mathcal G(\\bs z)\\big\\rangle\n={}&\n\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n+\\mu\\|y-x^\\star\\|^2\n-\\delta\\mu\\langle y-x^\\star,x-x^\\star\\rangle\n\\\\\n&\\quad\n+\\beta\\|\\nabla f_{-\\hat\\mu}(x)\\|^2\n+\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n\\\\\n\\geq{}&\n\\left(2-\\sqrt{\\delta}-b\\right)\\mathcal E(\\bs z;\\hat\\mu)\n+\\RV{bD_{f_{-\\hat\\mu}}(x,x^\\star)}+\\beta\\|\\nabla f_{-\\hat\\mu}(x)\\|^2\\\\\n&\\quad\n+\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n+\\left(1-\\sqrt{\\delta}\\right)\\Delta_f(x,x^\\star),\n\\end{aligned}\n\\end{equation}\nwhere $0\\leq b<2-\\sqrt{\\delta}$.\n\\end{lemma}","sha256_utf8":"3fda101df0209402e556531b555c8900debe9c184caa6f55e3c6f119afe0f14b"},{"start":73585,"end":76667,"start_line":1896,"end_line":2028,"raw_tex":"\\begin{proof}\nA direct calculation gives\n$$\n\\begin{aligned}\n-\\big\\langle \\nabla \\mathcal E(\\bs z;\\hat\\mu),\\mathcal G(\\bs z)\\big\\rangle\n={}&\n\\left\\langle\n\\begin{pmatrix}\n\\nabla f_{-\\hat\\mu}(x)\\\\[2pt]\n\\mu(y-x^\\star)\n\\end{pmatrix},\n\\begin{pmatrix}\nx-y+\\beta\\nabla f_{-\\hat\\mu}(x)+\\beta\\hat\\mu(x-x^\\star)\\\\\ny-x^\\star+\\dfrac1\\mu\\nabla f_{-\\hat\\mu}(x)-\\delta(x-x^\\star)\n\\end{pmatrix}\n\\right\\rangle\n\\\\\n={}&\n\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n+\\mu\\|y-x^\\star\\|^2\n-\\delta\\mu\\langle y-x^\\star,x-x^\\star\\rangle\n\\\\\n&\\quad\n+\\beta\\|\\nabla f_{-\\hat\\mu}(x)\\|^2\n+\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle.\n\\end{aligned}\n$$\nThis proves the identity. \\RV{We keep the term $\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle$ separate, since it controls the $\\hat \\mu$-shift term in Lemma~\\ref{lm:decay1}.}\n\nSince $f_{-\\hat\\mu}$ is $\\delta\\mu$-strongly convex,\n\\begin{equation}\\label{eq:deltaconvexity}\nD_{f_{-\\hat\\mu}}(x^\\star,x)\n\\geq\n\\frac{\\mu-\\hat\\mu}{2}\\|x-x^\\star\\|^2\n=\n\\frac{\\delta\\mu}{2}\\|x-x^\\star\\|^2.\n\\end{equation}\nYoung's inequality and \\eqref{eq:deltaconvexity} give\n$$\n\\begin{aligned}\n\\delta\\mu\\langle y-x^\\star,x-x^\\star\\rangle\n&\\leq\n\\sqrt{\\delta}\\,\\frac{\\mu}{2}\\|y-x^\\star\\|^2\n+\n\\frac{\\delta^{3/2}\\mu}{2}\\|x-x^\\star\\|^2\n\\\\\n&\\leq\n\\sqrt{\\delta}\\,\\frac{\\mu}{2}\\|y-x^\\star\\|^2\n+\n\\sqrt{\\delta}\\,D_{f_{-\\hat\\mu}}(x^\\star,x).\n\\end{aligned}\n$$\nTherefore,\n$$\n\\begin{aligned}\n-\\big\\langle \\nabla \\mathcal E(\\bs z;\\hat\\mu),\\mathcal G(\\bs z)\\big\\rangle\n\\geq{}&\n\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n-\n\\sqrt{\\delta}\\,D_{f_{-\\hat\\mu}}(x^\\star,x)\n+\n\\beta\\|\\nabla f_{-\\hat\\mu}(x)\\|^2\n\\\\\n&\\quad\n+\n\\left(2-\\sqrt{\\delta}\\right)\\frac{\\mu}{2}\\|y-x^\\star\\|^2\n+\n\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle.\n\\end{aligned}\n$$\nBy the symmetrization identity \\eqref{eq:symBre} and the invariance of the Bregman asymmetry under quadratic shifts,\n$D_{f_{-\\hat\\mu}}(x^\\star,x)\n=\nD_{f_{-\\hat\\mu}}(x,x^\\star)\n+\n\\Delta_f(x,x^\\star).$\nHence\n$$\n\\begin{aligned}\n&\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n-\n\\sqrt{\\delta}\\,D_{f_{-\\hat\\mu}}(x^\\star,x)\n\\\\\n={}&\nD_{f_{-\\hat\\mu}}(x,x^\\star)\n+\n\\left(1-\\sqrt{\\delta}\\right)D_{f_{-\\hat\\mu}}(x^\\star,x)\n\\\\\n={}&\n\\left(2-\\sqrt{\\delta}\\right)D_{f_{-\\hat\\mu}}(x,x^\\star)\n+\n\\left(1-\\sqrt{\\delta}\\right)\\Delta_f(x,x^\\star).\n\\end{aligned}\n$$\nCombining the first term with $\\frac{\\mu}{2}\\|y-x^\\star\\|^2$ yields\n$$\n\\begin{aligned}\n-\\big\\langle \\nabla \\mathcal E(\\bs z;\\hat\\mu),\\mathcal G(\\bs z)\\big\\rangle\n\\geq{}&\n\\left(2-\\sqrt{\\delta}\\right)\\mathcal E(\\bs z;\\hat\\mu)\n+\n\\beta\\|\\nabla f_{-\\hat\\mu}(x)\\|^2\n\\\\\n&\\quad\n+\n\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n+\n\\left(1-\\sqrt{\\delta}\\right)\\Delta_f(x,x^\\star).\n\\end{aligned}\n$$\nFinally,\n$$\n\\left(2-\\sqrt{\\delta}\\right)\\mathcal E(\\bs z;\\hat\\mu)\n=\n\\left(2-\\sqrt{\\delta}-b\\right)\\mathcal E(\\bs z;\\hat\\mu)\n+\nb\\mathcal E(\\bs z;\\hat\\mu).\n$$\nSince $b\\geq0$, \n% and\n%$$\n%\\mathcal E(\\bs z;\\hat\\mu)\n%\\geq\n%D_{f_{-\\hat\\mu}}(x,x^\\star),\n%$$\nusing\n$$\nb\\mathcal E(\\bs z;\\hat\\mu)\n\\geq\nbD_{f_{-\\hat\\mu}}(x,x^\\star),\n$$\nwe obtain \\eqref{eq:energy_identity}.\n\\end{proof}","sha256_utf8":"e0df610491a9d130e67d1f698b57ce4ddf632bf04c944ec8f19495812e298b67"},{"start":76925,"end":77620,"start_line":2032,"end_line":2058,"raw_tex":"\\begin{lemma}\\label{lm:decay1}\nLet $0\\leq\\hat\\mu\\leq\\mu$, and let $(x_k,y_k)$ be generated by the HNAG scheme~\\eqref{eq:HNAG} with\n$\\alpha=\\sqrt{2\\mu/L}$ and $\\alpha\\beta=1/L$.\nThen, for all $k\\geq0$,\n\\begin{equation}\\label{eq:decay1}\n\\begin{aligned}\n(1+c_L\\alpha)\\mathcal E(\\bs z_{k+1};\\hat\\mu)\n\\leq{}&\n\\mathcal E(\\bs z_k;\\hat\\mu)\n+\\frac1{2L}\\|\\nabla f_{-\\hat\\mu}(x_{k+1})\\|^2\n-\\frac1{2L}\\|\\nabla f_{-\\hat\\mu}(x_k)\\|^2\n\\\\\n&\\quad\n-\\alpha\\left[\nd_\\delta\\Delta_f(x_{k+1},x^\\star)\n+\\RV{bD_{f_{-\\hat\\mu}}(x_{k+1},x^\\star)}\n\\right ],\n\\end{aligned}\n\\end{equation}\nwhere\n$$\nb\\geq0,\\quad\n\\delta=\\frac{\\mu-\\hat\\mu}{\\mu}\\in[0,1],\\quad\nc_L=2-\\sqrt{\\delta}-b>0,\\quad\nd_\\delta=1-\\sqrt{\\delta}.\n$$\n\\end{lemma}","sha256_utf8":"96abee9f9a2cf7c59ad1f7491cc94d13e73f8b0ed85eda1d5c33f802a99560d5"},{"start":77622,"end":81232,"start_line":2060,"end_line":2218,"raw_tex":"\\begin{proof}\nUsing the Bregman expansion,\n$$\n\\mathcal E(\\bs z_{k+1};\\hat\\mu)-\\mathcal E(\\bs z_k;\\hat\\mu)\n=\n\\big\\langle \\nabla \\mathcal E(\\bs z_{k+1};\\hat\\mu),\\bs z_{k+1}-\\bs z_k\\big\\rangle\n-\nD_{\\mathcal E(\\cdot;\\hat\\mu)}(\\bs z_k,\\bs z_{k+1}).\n$$\nLet $g_k:=\\nabla f_{-\\hat\\mu}(x_k)$. \\RV{Since $-\\beta\\nabla f(x_{k+1})$ is used in $\\mathcal G(\\bs z_{k+1})$ and\n$$\n\\beta(g_{k+1}-g_k)+\\beta\\hat\\mu(x_{k+1}-x_k)\n=\n\\beta\\bigl(\\nabla f(x_{k+1})-\\nabla f(x_k)\\bigr),\n$$}\nthe update can be written as\n$$\n\\bs z_{k+1}-\\bs z_k\n=\n\\alpha\\mathcal G(\\bs z_{k+1})\n+\n\\alpha\n\\binom{\n(y_k-y_{k+1})+\\beta(g_{k+1}-g_k)+\\beta\\hat\\mu(x_{k+1}-x_k)\n}{0}.\n$$\n\nThe implicit Euler part is controlled by Lemma~\\ref{lem:energy_identity}:\n\\begin{equation}\\label{eq:IE-HNAGpp}\n\\begin{aligned}\n&\\alpha\\big\\langle \\nabla \\mathcal E(\\bs z_{k+1};\\hat\\mu),\\mathcal G(\\bs z_{k+1})\\big\\rangle \\\\\n\\leq{}&\n-c_L\\alpha\\mathcal E(\\bs z_{k+1};\\hat\\mu)\n-\\alpha\\beta\\hat\\mu\\langle g_{k+1},x_{k+1}-x^\\star\\rangle\n\\\\\n&\\quad\n-\\alpha\\beta\\|g_{k+1}\\|^2\n-\\alpha\\left [\nd_\\delta\\Delta_f(x_{k+1},x^\\star)\n+\\RV{bD_{f_{-\\hat\\mu}}(x_{k+1},x^\\star)}\n\\right ].\n\\end{aligned}\n\\end{equation}\n\n\\medskip\n\\noindent\n\\textbf{The $\\hat\\mu$-shift term.}\nBy the three-point Bregman identity~\\eqref{eq:Bregmanidentity},\n$$\n\\begin{aligned}\n&\\alpha\\beta\\hat\\mu\\langle g_{k+1},x_{k+1}-x_k\\rangle\n\\\\\n={}&\n\\alpha\\beta\\hat\\mu\\Bigl(\nD_{f_{-\\hat\\mu}}(x_{k+1},x^\\star)\n+\nD_{f_{-\\hat\\mu}}(x_k,x_{k+1})\n-\nD_{f_{-\\hat\\mu}}(x_k,x^\\star)\n\\Bigr).\n\\end{aligned}\n$$\n%Using\n%$$\n%\\langle g_{k+1},x_{k+1}-x^\\star\\rangle\n%=\n%D_{f_{-\\hat\\mu}}(x_{k+1},x^\\star)\n%+\n%D_{f_{-\\hat\\mu}}(x^\\star,x_{k+1}),\n%$$\nThe first term cancels the corresponding part of the shift term in \\eqref{eq:IE-HNAGpp}, leaving a nonpositive term, which we may discard.\n%$$\n%-\\alpha\\beta\\hat\\mu D_{f_{-\\hat\\mu}}(x^\\star,x_{k+1})\\leq0.\n%$$\nThe second term combines with\n$-D_{\\mathcal E(\\cdot;\\hat\\mu)}(\\bs z_k,\\bs z_{k+1})$ to give\n$$\n-(1-\\alpha\\beta\\hat\\mu)D_{f_{-\\hat\\mu}}(x_k,x_{k+1}).\n$$\nThe last term is also nonpositive and may be discarded.\n\n\\medskip\n\\noindent\n\\textbf{Lagging of $y$.}\nBy Cauchy--Schwarz and Young's inequality,\n$$\n\\alpha\\langle g_{k+1},y_k-y_{k+1}\\rangle\n\\leq\n\\frac{\\alpha^2}{2\\mu}\\|g_{k+1}\\|^2\n+\n\\frac{\\mu}{2}\\|y_k-y_{k+1}\\|^2.\n$$\nThe second term cancels the $y$-component of\n$-D_{\\mathcal E(\\cdot;\\hat\\mu)}(\\bs z_k,\\bs z_{k+1})$.\n\n\\medskip\n\\noindent\n\\textbf{Lagging of the gradient.}\nThe identity of squares gives\n$$\n\\begin{aligned}\n\\alpha\\beta\\langle g_{k+1},g_{k+1}-g_k\\rangle\n={}&\n-\\frac{\\alpha\\beta}{2}\\|g_k\\|^2\n+\\frac{\\alpha\\beta}{2}\\|g_{k+1}\\|^2\n+\\frac{\\alpha\\beta}{2}\\|g_{k+1}-g_k\\|^2.\n\\end{aligned}\n$$\nSince $f_{-\\hat\\mu}$ is $(L-\\hat\\mu)$-smooth and $\\alpha\\beta=1/L$,\n$$\n\\frac{\\alpha\\beta}{2}\\|g_{k+1}-g_k\\|^2\n\\leq\n(1-\\alpha\\beta\\hat\\mu)D_{f_{-\\hat\\mu}}(x_k,x_{k+1}).\n$$\nTherefore,\n$$\n\\begin{aligned}\n&\\alpha\\beta\\langle g_{k+1},g_{k+1}-g_k\\rangle\n-\n(1-\\alpha\\beta\\hat\\mu)D_{f_{-\\hat\\mu}}(x_k,x_{k+1})\n\\\\\n\\leq{}&\n-\\frac{\\alpha\\beta}{2}\\|g_k\\|^2\n+\n\\frac{\\alpha\\beta}{2}\\|g_{k+1}\\|^2.\n\\end{aligned}\n$$\n\nCombining the above estimates gives\n$$\n\\begin{aligned}\n(1+c_L\\alpha)\\mathcal E(\\bs z_{k+1};\\hat\\mu)\n\\leq{}&\n\\mathcal E(\\bs z_k;\\hat\\mu)\n+\n\\left(\n\\frac{\\alpha^2}{2\\mu}\n-\\frac{\\alpha\\beta}{2}\n\\right)\\|g_{k+1}\\|^2\n-\n\\frac{\\alpha\\beta}{2}\\|g_k\\|^2\n\\\\\n&\\quad\n-\\alpha bD_{f_{-\\hat\\mu}}(x_{k+1},x^\\star)\n-\\alpha d_\\delta\\Delta_f(x_{k+1},x^\\star).\n\\end{aligned}\n$$\nFinally, using $\\alpha^2=2\\mu/L$ and $\\alpha\\beta=1/L$, we have\n%$$\n%\\frac{\\alpha^2}{2\\mu}-\\frac{\\alpha\\beta}{2}\n%=\n%\\frac1{2L},\n%\\qquad\n%\\frac{\\alpha\\beta}{2}\n%=\n%\\frac1{2L}.\n%$$\nthe desired result \\eqref{eq:decay1}. $\\Box$\n\\end{proof}","sha256_utf8":"28eb2cedadcb274211c364393f005f6759834571795b496bbb76a308323e0cbb"},{"start":93084,"end":93302,"start_line":2630,"end_line":2640,"raw_tex":"\\begin{lemma}\\label{lm:gradient_norm_ineq}\nFor all \\(k\\ge 0\\),\n$$\n\\|\\nabla f_k(x_{k+1})\\|^2\n\\le\n\\|\\nabla f_{k+1}(x_{k+1})\\|^2\n+\nL\\Delta\\delta_k\\alpha^2\n\\langle \\nabla f_k(x_{k+1}),x_{k+1}-x^\\star\\rangle.\n$$\n\\end{lemma}","sha256_utf8":"33078542c639aa89ae65d408ad459825feed31762d84c67af6b08543a640158a"},{"start":93303,"end":93896,"start_line":2641,"end_line":2669,"raw_tex":"\\begin{proof}\nBy definition,\n$$\n\\nabla f_{k+1}(x_{k+1})\n=\n\\nabla f_k(x_{k+1})-(\\mu_{k+1}-\\mu_k)(x_{k+1}-x^\\star).\n$$\nExpanding the square and dropping the positive last term, we obtain\n$$\n\\begin{aligned}\n\\|\\nabla f_{k+1}(x_{k+1})\\|^2\n&=\n\\bigl\\|\\nabla f_k(x_{k+1})-(\\mu_{k+1}-\\mu_k)(x_{k+1}-x^\\star)\\bigr\\|^2\\\\\n&\\ge\n\\|\\nabla f_k(x_{k+1})\\|^2\n-\n2(\\mu_{k+1}-\\mu_k)\n\\langle \\nabla f_k(x_{k+1}),x_{k+1}-x^\\star\\rangle.\n\\end{aligned}\n$$\nSince $\\mu_{k+1}-\\mu_k=\\Delta\\delta_k\\,\\mu, \n\\alpha^2=2\\mu/L,$\nwe have\n$$\n2(\\mu_{k+1}-\\mu_k)=L\\Delta\\delta_k\\alpha^2.\n$$\nThis proves the claim.\n$\\Box$\n\\end{proof}","sha256_utf8":"a6a9d880bda214bbe775bb71e1e1e88fea3cc575f1fdc9e225f591276529d746"},{"start":93990,"end":94195,"start_line":2673,"end_line":2683,"raw_tex":"By Theorem~\\ref{thm:linearconvHNAG-funcval},\n$$\n\\|x_{k+1}-x^\\star\\|\n\\leq\nC_1\\left(1+\\sqrt{2\\rho}\\right)^{-\\frac{k+1}{2}},\n\\qquad\nC_1:=\n\\left(\n\\frac{2(1+2\\alpha)}{\\alpha\\mu}\\widetilde E_0\n\\right)^{1/2}.\n$$\n","sha256_utf8":"59330d8fe4fb804948beecbe8d6e5c62f7301e8927dc87b27cb2d6c4c29ca16b"},{"start":94415,"end":94538,"start_line":2693,"end_line":2693,"raw_tex":"Since $\\omega$ is nondecreasing and its argument decreases to zero, $\\{\\omega_k\\}$ is nonincreasing and converges to zero.\n","sha256_utf8":"5e7ddeb586082814805bbef702489ada0b98a82f906772a0bc046ab2dbe4552a"},{"start":94982,"end":96202,"start_line":2697,"end_line":2746,"raw_tex":"\\begin{lemma}\\label{lm:shifted_energy_recursion}\nAssume $f\\in\\mathcal S_{\\mu,L}^{\\rm LAS}(x^\\star)$. Let $(x_k,y_k)$ be generated by Algorithm~\\ref{alg:HNAGpp}. Let $\\{\\delta_k\\}\\subset[0,1]$ be nonincreasing with $\\delta_k\\to0$, and let $\\{b_k\\}$ be nonnegative with $b_k\\to0$. Set\n$$\n\\mu_k=(1-\\delta_k)\\mu,\\quad\nf_k:=f_{-\\mu_k},\\quad\nd_k:=1-\\sqrt{\\delta_k},\\quad\n\\Delta\\delta_k:=\\delta_k-\\delta_{k+1}\\geq0,\n$$\nand define\n$$\n\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k)\n:=\n\\mathcal E(\\bs z_k;\\mu_k)\n-\\frac1{2L}\\|\\nabla f_k(x_k)\\|^2\\geq 0.\n$$\n%\\RV{Since $f_k$ is convex and $L$-smooth, co-coercivity gives\n%$$\n%D_{f_k}(x_k,x^\\star)\n%\\geq\n%\\frac1{2L}\\|\\nabla f_k(x_k)\\|^2.\n%$$\n%Consequently,\n%$$\n%\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k)\n%\\geq\n%\\frac{\\mu}{2}\\|y_k-x^\\star\\|^2\n%\\geq0.\n%$$}\nLet $\\omega_k$ be defined by \\eqref{eq:def-omegak}, and set\n$$\nc_k:=2-\\sqrt{\\delta_k}-b_k\\geq0,\n\\qquad\nr_k:=\n\\left(1+c_k\\alpha-\\Delta\\delta_k\\alpha^2\\right)^{-1}.\n$$\nAssume $r_k^{-1}\\geq1$ and\n\\begin{equation}\\label{eq:omega_condition}\n\\left|\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\right|\\omega_k\n\\leq\nb_k\\delta_k.\n\\end{equation}\nThen\n$$\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\nr_k\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k).\n$$\n\\end{lemma}","sha256_utf8":"7d63ab769438fa58b59177d5e237699da5736da55af4a3ae5f339c4b836a6e9d"},{"start":96204,"end":99244,"start_line":2748,"end_line":2891,"raw_tex":"\\begin{proof}\nApply Lemma~\\ref{lm:decay1} with $\\hat\\mu=\\mu_k$. Then\n$$\n\\begin{aligned}\n(1+c_k\\alpha)\\mathcal E(\\bs z_{k+1};\\mu_k)\n\\leq{}&\n\\mathcal E(\\bs z_k;\\mu_k)\n+\\frac1{2L}\\|\\nabla f_k(x_{k+1})\\|^2\n-\\frac1{2L}\\|\\nabla f_k(x_k)\\|^2\n\\\\\n&\\quad\n-\\alpha\\left(\nd_k\\Delta_f(x_{k+1},x^\\star)\n+\nb_kD_{f_k}(x_{k+1},x^\\star)\n\\right).\n\\end{aligned}\n$$\nWe first change the shifted gradient using Lemma~\\ref{lm:gradient_norm_ineq}:\n$$\n\\frac1{2L}\\|\\nabla f_k(x_{k+1})\\|^2\n\\leq\n\\frac1{2L}\\|\\nabla f_{k+1}(x_{k+1})\\|^2\n+\n\\frac{\\Delta\\delta_k\\alpha^2}{2}\n\\langle\\nabla f_k(x_{k+1}),x_{k+1}-x^\\star\\rangle.\n$$\nUsing\n$$\n\\begin{aligned}\n\\frac{\\Delta\\delta_k\\alpha^2}{2}\n\\langle\\nabla f_k(x_{k+1}),x_{k+1}-x^\\star\\rangle\n={}&\n\\Delta\\delta_k\\alpha^2D_{f_k}(x_{k+1},x^\\star)\n+\\frac{\\Delta\\delta_k\\alpha^2}{2}\n\\Delta_f(x_{k+1},x^\\star),\n\\end{aligned}\n$$\nand\n$$\n\\Delta\\delta_k\\alpha^2D_{f_k}(x_{k+1},x^\\star)\n\\leq\n\\Delta\\delta_k\\alpha^2\\mathcal E(\\bs z_{k+1};\\mu_k),\n$$\nthe first term can be absorbed into the left-hand side, which slightly weakens the contraction factor from $1+c_k\\alpha$ to $r_k$. \nWe then obtain\n$$\n\\begin{aligned}\nr_k^{-1}\\mathcal E(\\bs z_{k+1};\\mu_k)\n\\leq{}&\n\\mathcal E(\\bs z_k;\\mu_k)\n+\\frac1{2L}\\|\\nabla f_{k+1}(x_{k+1})\\|^2\n-\\frac1{2L}\\|\\nabla f_k(x_k)\\|^2\n\\\\\n&\\quad\n-\\alpha\\left[\n\\left(\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\right)\\Delta_f(x_{k+1},x^\\star)\n+\nb_kD_{f_k}(x_{k+1},x^\\star)\n\\right].\n\\end{aligned}\n$$\nSince $\\mu_{k+1}\\geq\\mu_k$,\n$$\n\\mathcal E(\\bs z_{k+1};\\mu_{k+1})\n\\leq\n\\mathcal E(\\bs z_{k+1};\\mu_k).\n$$\nThus the same inequality holds with $\\mathcal E(\\bs z_{k+1};\\mu_{k+1})$ on the left-hand side.\n\nBy the definition of $\\omega_k$,\n$$\n|\\Delta_f(x_{k+1},x^\\star)|\n\\leq\n\\omega_kD_f(x_{k+1},x^\\star).\n$$\nMoreover, since $\\mu_k=(1-\\delta_k)\\mu$ and $f$ is $\\mu$-strongly convex,\n$$\n\\begin{aligned}\nD_{f_k}(x_{k+1},x^\\star)\n&=\nD_f(x_{k+1},x^\\star)\n-\\frac{\\mu_k}{2}\\|x_{k+1}-x^\\star\\|^2\n\\\\\n&\\geq\nD_f(x_{k+1},x^\\star)\n-\\frac{\\mu_k}{\\mu}D_f(x_{k+1},x^\\star)\n=\n\\delta_kD_f(x_{k+1},x^\\star).\n\\end{aligned}\n$$\nTherefore,\n$$\n\\begin{aligned}\n&\n\\left(\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\right)\\Delta_f(x_{k+1},x^\\star)\n+\nb_kD_{f_k}(x_{k+1},x^\\star)\n\\\\\n&\\geq\n\\left[\nb_k\\delta_k\n-\n\\left|\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\right|\\omega_k\n\\right]\nD_f(x_{k+1},x^\\star)\n\\geq0,\n\\end{aligned}\n$$\nwhere the last inequality follows from \\eqref{eq:omega_condition}. It follows that\n$$\n\\begin{aligned}\nr_k^{-1}\\mathcal E(\\bs z_{k+1};\\mu_{k+1})\n\\leq{}&\n\\mathcal E(\\bs z_k;\\mu_k)\n+\\frac1{2L}\\|\\nabla f_{k+1}(x_{k+1})\\|^2\n-\\frac1{2L}\\|\\nabla f_k(x_k)\\|^2.\n\\end{aligned}\n$$\nSubtracting $r_k^{-1}(2L)^{-1}\\|\\nabla f_{k+1}(x_{k+1})\\|^2$ from both sides gives\n$$\n\\begin{aligned}\nr_k^{-1}\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq{}&\n\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k)\n+\n\\frac{1-r_k^{-1}}{2L}\n\\|\\nabla f_{k+1}(x_{k+1})\\|^2.\n\\end{aligned}\n$$\nSince $r_k^{-1}\\geq1$, the last term is nonpositive. Therefore,\n$$\nr_k^{-1}\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\n\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k),\n$$\nwhich proves the claim.\n\\end{proof}","sha256_utf8":"b31e64de243c573240a803939e77aa531e627b7848ae604ed489abf87312a247"},{"start":100978,"end":104131,"start_line":2951,"end_line":3122,"raw_tex":"\\begin{proof}\nSince $f\\in\\mathcal S_{\\mu,L}^{\\rm LAS}(x^\\star)$, we have $\\omega_k\\downarrow0$, so the index $K$ in the theorem statement is finite. The threshold $1/8$ is only a convenient choice ensuring $q_k\\leq1/2$; any fixed sufficiently small threshold would work.\n\n\\medskip\n\\noindent\n\\textbf{Step 1: Initial finite segment.}\nFor $0\\leq k<K$, we use the unshifted choice\n$$\n\\delta_k=1,\\qquad\n\\mu_k=0,\\qquad\nb_k=0,\\qquad\nd_k=0,\\qquad\nc_k=1.\n$$\nThen\n$$\n\\widetilde{\\mathcal E}(\\bs z_{k+1};0)\n\\leq\n(1+\\alpha)^{-1}\n\\widetilde{\\mathcal E}(\\bs z_k;0),\n\\qquad\n0\\leq k<K.\n$$\n\n\\medskip\n\\noindent\n\\textbf{Step 2: Choice of the partial shifts.}\nFor $k\\geq K$, set\n$$\nq_k:=\\omega_k^{1/3},\\qquad\n\\delta_k:=q_k^2,\\qquad\nd_k:=1-q_k,\\qquad\nb_k:=d_kq_k.\n$$\nThis choice gives\n$$\nb_k\\delta_k=d_k\\omega_k.\n$$\nSince $\\omega_k\\leq1/8$, we have $0\\leq q_k\\leq1/2$, and hence $d_k,b_k\\geq0$. Moreover,\n$$\n\\begin{aligned}\nc_k\n&:=\n2-\\sqrt{\\delta_k}-b_k\n=\n1+(1-q_k)^2\n\\geq\\frac54,\\\\\n&q_k\\to0,\\qquad\n\\delta_k\\downarrow0,\\qquad\nb_k\\to0,\\qquad\nc_k\\to2.\n\\end{aligned}\n$$\n\n\\medskip\n\\noindent\n\\textbf{Step 3: Verification of Lemma~\\ref{lm:shifted_energy_recursion}.}\nSet\n$$\nr_k\n:=\n\\left(1+c_k\\alpha-\\Delta\\delta_k\\alpha^2\\right)^{-1}.\n$$\nSince\n$$\nc_k\\geq\\frac54,\n\\qquad\n0\\leq\\Delta\\delta_k\\leq\\delta_k=q_k^2\\leq\\frac14,\n$$\nand $\\kappa=L/\\mu\\geq1$ implies $\\alpha=\\sqrt{2/\\kappa}\\leq\\sqrt2$, we have\n\\begin{equation*}\n\\begin{aligned}\nr_k^{-1}\n&=\n1+c_k\\alpha-\\Delta\\delta_k\\alpha^2\n\\geq\n1+\\frac54\\alpha-\\frac14\\alpha^2\n=\n1+\\frac{\\alpha}{4}(5-\\alpha)\n>1.\n\\end{aligned}\n\\end{equation*}\n\nIt remains to verify \\eqref{eq:omega_condition}. Since $d_k\\geq1/2$,\n$$\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\geq\n\\frac12-\\frac{\\sqrt2}{8}\n>0.\n$$\n%Moreover,\n%$$\n%d_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n%\\leq\n%d_k.\n%$$\nTherefore,\n\\begin{equation*}\n\\left|\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\right|\\omega_k\n\\leq\nd_k\\omega_k\n=\nb_k\\delta_k.\n\\end{equation*}\nLemma~\\ref{lm:shifted_energy_recursion} now gives\n$$\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\nr_k\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k),\n\\qquad\nk\\geq K.\n$$\nHence\n$$\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\n\\widetilde{\\mathcal E}(\\bs z_K;\\mu_K)\n\\prod_{i=K}^{k}r_i.\n$$\nThe finitely many initial terms can be absorbed into the constant. Specifically, let\n$$\nC\n:=\n1+\n\\max_{0\\leq j\\leq K}\n\\left\\{\n\\widetilde{\\mathcal E}(\\bs z_{j+1};\\mu_{j+1})\n\\left(\\prod_{i=0}^{j}r_i\\right)^{-1}\n\\right\\},\n$$\nThen, for all $k\\geq0$,\n$$\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\nC\\prod_{i=0}^{k}r_i.\n$$\n\n\\medskip\n\\noindent\n\\textbf{Step 4: Asymptotic rate.}\nSince $c_k\\to2$ and $\\Delta\\delta_k\\to0$,\n$$\nr_k\n\\to\nr_\\ast\n:=\n\\left(1+2\\sqrt{2/\\kappa}\\right)^{-1}.\n$$\nTherefore,\n%$$\n%\\left(\\prod_{i=0}^{k}r_i\\right)^{1/k}\n%\\to\n%r_\\ast.\n%$$\n%It follows that\n$$\n\\begin{aligned}\n\\limsup_{k\\to\\infty}\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})^{1/k}\n&\\leq\n\\limsup_{k\\to\\infty}\n\\left(\nC\\prod_{i=0}^{k}r_i\n\\right)^{1/k}\n=\nr_\\ast.\n\\end{aligned}\n$$\nHence $\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k)$ converges $R$-linearly with asymptotic rate $r_\\ast$. \\RV{The bound in the theorem then gives the same asymptotic rate for $\\|y_k-x^\\star\\|^2$.} $\\Box$\n\\end{proof}","sha256_utf8":"1d15ed46f626f93b91938f36581f3438de63ee00ee0b8f76a7e553b18259d14b"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"强凸光滑优化问题","source_units":[{"start":4998,"end":5501,"start_line":112,"end_line":119,"raw_tex":"We consider the unconstrained smooth convex optimization problem\n\\begin{equation*}\n\\min_{x\\in\\mathbb{R}^d} f(x),\n\\end{equation*}\nwhere $f$ belongs to the class $\\mathcal S_{\\mu,L}$ of continuously differentiable, $\\mu$-strongly convex functions with $L$-Lipschitz continuous gradients. Strong convexity guarantees the existence\nof a unique global minimizer $x^{\\star}$. We study first-order methods, which\nrely solely on gradient information, and generate iterates \n$\\{x_k\\}$ converging to $x^{\\star}$.\n","sha256_utf8":"5ee12bddb36bdc31c1311bad888e6f7d22cde0e8d591fef0ac2f65c81443e350"}]},{"id":"S002","semantic_type":"definition","kind":"definition","title":"线性与渐近收敛率","source_units":[{"start":5502,"end":5931,"start_line":121,"end_line":131,"raw_tex":"Convergence is measured by a nonnegative error quantity $E_k$, such as $\\|x_k-x^\\star\\|^2$, $f(x_k)-f(x^\\star)$, or a Lyapunov function with comparable scaling. We say that the sequence $\\{E_k\\}$\nconverges linearly if\n$$\nE_k \\le C r^k, \\qquad k \\ge 1,\n$$\nfor some constants $C>0$ and $r\\in(0,1)$.\nMore precisely, $E_k$ is said to converge \\emph{R-linearly} with asymptotic rate\n$r$ if\n$$\n\\limsup_{k\\to\\infty} E_k^{1/k} \\le r.\n$$\n","sha256_utf8":"207246691003b2416418aec4dcab98192b1ececbdd4a6b4761e7c2d68b201e37"}]},{"id":"S003","semantic_type":"notation","kind":"notation","title":"条件数及其倒数","source_units":[{"start":6225,"end":6334,"start_line":140,"end_line":141,"raw_tex":"The condition number of $f$ is $\\kappa = L/\\mu$, and we denote its reciprocal by\n$\\rho = 1/\\kappa = \\mu/L$. \n","sha256_utf8":"0a0cb50621164b7dfa355d5b577ece2f5d1a8ca109d43e1b7dd7c2d1f06ea098"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"布雷格曼散度与非对称量","source_units":[{"start":13029,"end":13908,"start_line":236,"end_line":256,"raw_tex":"Let $f: \\mathbb{R}^d \\to \\mathbb{R}$ be differentiable. We use $\\langle \\cdot, \\cdot \\rangle$ for the inner product of $\\mathbb R^d$.\n\nThe Bregman divergence of $f$ between $x, y \\in \\mathbb{R}^d$ is \n\\begin{equation*}\nD_f(y,x) := f(y) - f(x) - \\langle \\nabla f(x),\\, y - x \\rangle .\n\\end{equation*}\nIn general, $D_f$ is not symmetric, i.e., $D_f(y,x) \\neq D_f(x,y)$.  \nIts symmetrization, called the symmetrized Bregman divergence, is  \n\\begin{equation}\\label{eq:symBre}\nD_f(y,x) + D_f(x,y)\n= \\langle \\nabla f(y) - \\nabla f(x),\\, y - x \\rangle .\n\\end{equation}\nDefine the difference of Bregman divergences\n\\begin{equation*}\n\\Delta_f(x,y) := D_f(y,x) - D_f(x,y).\n\\end{equation*}\nWe can rewrite the symmetric Bregman divergence via the asymmetry:\n\\begin{equation}\\label{eq:Brelation}\n\\langle \\nabla f(x) - \\nabla f(y),\\, x - y \\rangle \n= 2D_f(x,y) + \\Delta_f(x,y).\n\\end{equation}\n","sha256_utf8":"628b1212dc24d38b565ff4ef5c7b7fc34aa69c45821159d63aca52c4db4ac254"}]},{"id":"S005","semantic_type":"derived_fact","kind":"derived_fact","title":"三点恒等式与平方恒等式","source_units":[{"start":14008,"end":14428,"start_line":259,"end_line":271,"raw_tex":"The following three-point Bregman identity holds~\\cite{chen1993convergence}:\n\\begin{equation}\\label{eq:Bregmanidentity}\n\\langle \\nabla f(y) - \\nabla f(x),\\, y - z \\rangle \n= D_f(y,x) + D_f(z,y) - D_f(z,x),\n\\end{equation}\nwhich is a generalization of the classical identity of squares:\n\\begin{equation}\\label{eq:squares}\n\\langle y-x,\\, y-z \\rangle\n=\n\\frac12\\bigl(\n\\|y-x\\|^2 + \\|z-y\\|^2 - \\|z-x\\|^2\n\\bigr).\n\\end{equation}\n","sha256_utf8":"20640f2a20c7ad010f3f42ca7e39974e16f45b44324cd33db8662824de8c9d50"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"函数类与基本散度界","source_units":[{"start":14430,"end":14800,"start_line":274,"end_line":283,"raw_tex":"The function $f$ is $\\mu$-strongly convex if for some $\\mu > 0$,\n\\begin{equation*}\nD_f(y,x) \\geq \\frac{\\mu}{2} \\|y - x\\|^2, \n\\qquad \\forall x, y \\in \\mathbb{R}^d.\n\\end{equation*}\nIt is $L$-smooth, for some $L > 0$, if its gradient is $L$-Lipschitz:\n\\begin{equation*}\n\\|\\nabla f(y) - \\nabla f(x)\\| \\leq L \\|y - x\\|, \n\\qquad \\forall x, y \\in \\mathbb{R}^d.\n\\end{equation*}\n","sha256_utf8":"285054ef020908ad3ee5cf8e324b97dd6f0b3d422b469ce3386f619afbac9dec"},{"start":14981,"end":15612,"start_line":287,"end_line":301,"raw_tex":"Let $\\mathcal{S}_{\\mu,L}$ be the class of all differentiable functions that are both $\\mu$-strongly convex and $L$-smooth.  \nFor $f \\in \\mathcal{S}_{\\mu,L}$, the Bregman divergence satisfies~\\cite{nesterov2003introductory}\n\\begin{equation}\\label{eq:muLsquares}\n\\frac{\\mu}{2} \\| x - y \\|^2 \n\\leq D_f(x,y) \n\\leq \\frac{L}{2} \\| x - y \\|^2, \n\\qquad \\forall x, y \\in \\mathbb{R}^d,\n\\end{equation}\nand, in terms of gradient differences,\n\\begin{equation}\\label{eq:cocov}\n\\frac{1}{2L} \\|\\nabla f(x) - \\nabla f(y)\\|^2 \n\\leq D_f(x,y) \n\\leq \\frac{1}{2\\mu} \\|\\nabla f(x) - \\nabla f(y)\\|^2, \n\\qquad \\forall x, y \\in \\mathbb{R}^d.\n\\end{equation}\n","sha256_utf8":"4f283765b6cce5b65de580b635e0bb4734d5894229f176424e5b6b5a65f4b521"}]},{"id":"S007","semantic_type":"definition","kind":"definition","title":"HNAG流与向量场","source_units":[{"start":17062,"end":17264,"start_line":331,"end_line":339,"raw_tex":"The continuous-time HNAG flow is\n\\begin{equation}\\label{eq:flow}\n\\left\\{\n\\begin{aligned}\nx' &= y - x - \\beta \\nabla f(x), \\\\\ny' &= x - y - \\tfrac{1}{\\mu}\\nabla f(x),\n\\end{aligned}\n\\right.\n\\end{equation}","sha256_utf8":"d6c9c0794a6e2c541d80e698c076263923f5b46dc891ebfae4bdef67fb25354b"},{"start":17265,"end":17505,"start_line":340,"end_line":345,"raw_tex":"where $\\mu$ is the strong convexity parameter of $f$.\nLet $\\boldsymbol z=(x,y)$ and define\n\\begin{equation}\\label{eq:G}\n\\mathcal G(\\boldsymbol z)\n=\\bigl(y-x-\\beta\\nabla f(x),\\; x-y-\\tfrac{1}{\\mu}\\nabla f(x)\\bigr)^{\\intercal},\n\\end{equation}","sha256_utf8":"ae84f2239bcdf4840d9a4d9b81ec1568338b5e050ec5639cfaddafa7f5f4f7c2"},{"start":17506,"end":17602,"start_line":346,"end_line":346,"raw_tex":"so that \\eqref{eq:flow} can be written compactly as $\\boldsymbol z'=\\mathcal G(\\boldsymbol z)$.\n","sha256_utf8":"60f01ce00a4b893ea494992f97941e24db0b76546de8cc265afa23de494a8d54"}]},{"id":"S008","semantic_type":"algorithm","kind":"algorithm","title":"HNAG隐显离散格式","source_units":[{"start":17603,"end":17921,"start_line":348,"end_line":355,"raw_tex":"An implicit--explicit (IMEX) discretization of~\\eqref{eq:flow} yields\n\\begin{subequations}\\label{eq:HNAG}\n\\begin{align}\n\\label{eq:HNAGx}\n\\frac{x_{k+1}-x_k}{\\alpha} &= y_k - x_{k+1} - \\beta \\nabla f(x_k), \\\\\n\\label{eq:HNAGy}\n\\frac{y_{k+1}-y_k}{\\alpha} &= x_{k+1} - y_{k+1} - \\tfrac{1}{\\mu}\\nabla f(x_{k+1}),\n\\end{align}","sha256_utf8":"67b2bd65bb21e92ccf0e35a6104a9665a42485cad42df784544d826e642c979d"},{"start":17922,"end":18050,"start_line":356,"end_line":357,"raw_tex":"\\end{subequations}\nwhere $x_{k+1}$ is updated first in \\eqref{eq:HNAGx} and then used to compute $y_{k+1}$ in \\eqref{eq:HNAGy}.\n","sha256_utf8":"f4153922d90cede69be73ae9c267e0906ccbaa9f2222ab19bc2cebdd08fbf79c"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"未移位李雅普诺夫能量","source_units":[{"start":19767,"end":20022,"start_line":402,"end_line":408,"raw_tex":"We use the following Lyapunov notation throughout the paper. First, define\n\\begin{equation}\\label{eq:lyapunov_strong_HNAG}\nE(\\boldsymbol z)=E(x,y)\n:=D_f(x,x^\\star)+\\frac{\\mu}{2}\\|y-x^\\star\\|^2\n=\nf(x)-f(x^\\star)+\\frac{\\mu}{2}\\|y-x^\\star\\|^2,\n\\end{equation}","sha256_utf8":"339265d7f16033c6a4eb3748db7ea5731752fb99b42ff569d58f8fa692cd7720"},{"start":20023,"end":20209,"start_line":409,"end_line":411,"raw_tex":"where \\(D_f(x,x^\\star)=f(x)-f(x^\\star)\\) since \\(\\nabla f(x^\\star)=0\\). Since\n\\(f\\in\\mathcal S_{\\mu,L}\\), we have \\(E(\\boldsymbol z)\\ge0\\), with equality if\nand only if \\(x=y=x^\\star\\).\n","sha256_utf8":"778748c7c4fa41fabcf2100dd54c2637d493cc5fa240c143481f6a43c8a266b9"}]},{"id":"S010","semantic_type":"algorithm","kind":"algorithm","title":"HNAG加大步长算法","source_units":[{"start":64328,"end":64848,"start_line":1637,"end_line":1652,"raw_tex":"\\begin{algorithm}\n\\caption{HNAG$^{++}$}\n\\label{alg:HNAGpp}\n\\linespread{1.3}\\selectfont\n\\begin{algorithmic}[1]\n\\Require $x_0,v_0\\in\\mathbb R^d$, parameters $L\\ge \\mu>0$\n\\State $\\alpha \\gets \\sqrt{2\\mu/L}$\n\\For{$k=0,1,2,\\ldots$}\n\\State $v_{k+1}\\gets \\tfrac{1}{1+\\alpha}\n\\bigl(v_k+\\alpha^2 x_k-\\tfrac{2}{L}\\nabla f(x_k)\\bigr)$\n\\State $x_{k+1}\\gets \\tfrac{1}{1+\\alpha}\n\\bigl(x_k+v_{k+1}-\\tfrac{1}{L}\\nabla f(x_k)\\bigr)$\n\\EndFor\n\\State \\textbf{Return:} $x_{k+1}$ and $y_{k} = v_{k+1}/\\alpha$\n\\end{algorithmic}\n\\end{algorithm}","sha256_utf8":"d3688d7a53361b8ee9fab83a0c811d3fa9a2dfc8b75f06c41dee5a3296d50121"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"部分原始移位及其能量","source_units":[{"start":71365,"end":71589,"start_line":1842,"end_line":1847,"raw_tex":"Since $f$ is $\\mu$-strongly convex, for any $\\hat{\\mu}\\leq \\mu$, the shifted function\n$$\nf_{-\\hat{\\mu}}(x)\n:= f(x) - \\frac{\\hat{\\mu}}{2}\\|x - x^{\\star}\\|^2\n$$\nis convex and satisfies $\\nabla f_{-\\hat{\\mu}}(x^{\\star}) = 0$. \n","sha256_utf8":"40aa77b48170c47ac0727dd2f37f8e49a419a92af9dd642259bfca50d9089b9e"},{"start":72169,"end":72323,"start_line":1860,"end_line":1864,"raw_tex":"\\begin{equation}\\label{eq:lya-muhat}\n\\mathcal{E}(\\bs z; \\hat{\\mu})\n:= D_{f_{-\\hat{\\mu}}}(x, x^{\\star})\n+ \\tfrac{\\mu}{2}\\|y - x^{\\star}\\|^2.\n\\end{equation}","sha256_utf8":"41b0441cda25ad13113a844170c532ab9513743587187e0ca5f162363c2d3e6a"},{"start":72324,"end":72460,"start_line":1865,"end_line":1865,"raw_tex":"{Notice that the partial shift \\(\\hat\\mu\\) is used only in \\(f\\), while the \\(y\\)-component still uses the fixed coefficient \\(\\mu\\).} \n","sha256_utf8":"6152258f7da1ec9df04672b39416385b11ce6b0f80ac9b6b044015075dc0784b"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"局部渐近对称函数类","source_units":[{"start":86257,"end":86878,"start_line":2369,"end_line":2395,"raw_tex":"\\begin{definition}[Local asymptotic symmetry]\nLet $f\\in\\mathcal S_{\\mu,L}$ with minimizer $x^\\star$. The function $f$ is said to be \\emph{locally asymptotically symmetric at $x^\\star$}, abbreviated \\emph{LAS at $x^\\star$}, if\n$$\n\\lim_{x\\to x^\\star}\n\\RV{\\frac{|\\Delta_f(x,x^\\star)|}{D_f(x,x^\\star)}}\n=\n0.\n$$\n%where\n%$$\n%\\Delta_f(x,x^\\star)\n%:=\n%D_f(x^\\star,x)-D_f(x,x^\\star).\n%$$\nDefine\n$$\n\\omega(R)\n:=\n\\sup_{0<\\|x-x^\\star\\|\\leq R}\n\\frac{|\\Delta_f(x,x^\\star)|}{D_f(x,x^\\star)},\n\\qquad R>0.\n$$\nThen $\\omega$ is monotone nondecreasing, and the LAS condition is equivalent to\n$$\n\\lim_{R\\to0^+}\\omega(R)=0.\n$$\n\\end{definition}","sha256_utf8":"599028ac9bd4fd5d0a560f9fce5cf80a285663848c789bef186f9d83653c60ea"},{"start":86880,"end":87010,"start_line":2397,"end_line":2405,"raw_tex":"We define\n$$\n\\mathcal S_{\\mu,L}^{\\mathrm{LAS}}(x^\\star)\n:=\n\\Bigl\\{\nf\\in\\mathcal S_{\\mu,L}:\nf\\text{ is LAS at }x^\\star\n\\Bigr\\}.\n$$\n","sha256_utf8":"dc593262bc94a164be1fb49cb55a4757cc73a641234e0d9e7aca5100c32b7a6a"}]},{"id":"S013","semantic_type":"notation","kind":"notation","title":"变化移位与参数差分","source_units":[{"start":92522,"end":92975,"start_line":2614,"end_line":2627,"raw_tex":"Let \\(\\{\\mu_k\\}\\) be a nondecreasing sequence with \\(\\mu_k\\uparrow\\mu\\). Define\n\\begin{equation*}\n\\mathcal E(\\bs z_k;\\mu_k)\n:=\nD_{f_k}(x_k,x^\\star)+\\frac{\\mu}{2}\\|y_k-x^\\star\\|^2,\n\\qquad\nf_k:=f_{-\\mu_k}.\n\\end{equation*}\nThe parameter \\(\\mu_k\\) is used only in the shift of \\(f_k\\); the quadratic term in \\(y_k\\) always uses the fixed coefficient \\(\\mu\\). We write\n$$\n\\delta_k:=\\frac{\\mu-\\mu_k}{\\mu},\n\\qquad\n\\Delta\\delta_k:=\\delta_k-\\delta_{k+1}\\ge0.\n$$\n","sha256_utf8":"3d8e6646880d49f732b5b566a16d01e9797496eba0a42773d82e68752c586dc2"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"沿迭代的对称性模量","source_units":[{"start":94195,"end":94414,"start_line":2684,"end_line":2692,"raw_tex":"If $f\\in\\mathcal S_{\\mu,L}^{\\rm LAS}(x^\\star)$, then\n\\begin{equation}\\label{eq:def-omegak}\n\\omega(\\|x_{k+1}-x^\\star\\|)\n\\leq\n\\omega\\left(\nC_1\\left(1+\\sqrt{2\\rho}\\right)^{-\\frac{k+1}{2}}\n\\right)\n=:\\omega_k.\n\\end{equation}","sha256_utf8":"7c4df7ba2f0c2ce636efcfc4f4e55b71545b34899e9f90f873f9af76f38e9b21"}]},{"id":"S015","semantic_type":"external_fact","kind":"external_fact","title":"移位光滑性与余强制性","source_units":[],"declared_external":true,"declaration_text":["For convex M-smooth h, D_h(u,v) >= ||∇h(u)-∇h(v)||²/(2M). If f belongs to S_{μ,L} and 0 <= m <= μ, then f(x)-m||x-x*||²/2 is convex and (L-m)-smooth. At its minimizer x*, the gradient vanishes, so subtracting ||∇h(x)||²/(2M) from D_h(x,x*) leaves a nonnegative quantity."]},{"id":"S016","semantic_type":"external_fact","kind":"external_fact","title":"内积的标准不等式","source_units":[],"declared_external":true,"declaration_text":["For Euclidean vectors u,v and ε>0, |⟨u,v⟩| <= ||u||||v|| <= ε||u||²/2 + ||v||²/(2ε)."]},{"id":"S017","semantic_type":"external_fact","kind":"external_fact","title":"收缩因子乘积的根极限","source_units":[],"declared_external":true,"declaration_text":["If r_k>0 and r_k tends to r*>0, then (∏_{i=0}^k r_i)^{1/k} tends to r*. A fixed positive multiplier and a finite shift of the sequence index do not change this asymptotic root bound."]},{"id":"S018","semantic_type":"theorem","kind":"theorem","title":"HNAG双加号的精细渐近率","source_units":[{"start":99631,"end":100976,"start_line":2897,"end_line":2949,"raw_tex":"\\begin{theorem}[Asymptotic rate $1-2\\sqrt{2/\\kappa}$]\\label{thm:main-thm-C21}\nAssume $f\\in\\mathcal S_{\\mu,L}^{\\rm LAS}(x^\\star)$. Let $(x_k,y_k)$ be generated by Algorithm~\\ref{alg:HNAGpp}, let $\\omega_k$ be defined by \\eqref{eq:def-omegak}, and set $K:=\\min\\{k\\geq0:\\omega_k\\leq1/8\\}$. Then there exist nonnegative sequences $\\{\\mu_k\\}$, $\\{\\delta_k\\}$, $\\{b_k\\}$, and $\\{c_k\\}$, and a constant $C>0$, such that\n$$\n\\mu_k=(1-\\delta_k)\\mu,\\qquad\n\\delta_k\\downarrow0,\\qquad\nb_k\\to0,\\qquad\nc_k=2-\\sqrt{\\delta_k}-b_k\\to2,\n$$\nand\n%$$\n%\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n%\\leq\n%C\\prod_{i=0}^{k}r_i.\n%$$\n\\RV{\n$$\n\\frac{\\mu}{2}\\|y_{k+1}-x^\\star\\|^2\n\\leq\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\nC\\prod_{i=0}^{k}r_i.\n$$}\nHere\n$$\n\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k)\n:=\n\\mathcal E(\\bs z_k;\\mu_k)\n-\\frac1{2L}\\|\\nabla f_k(x_k)\\|^2,\n\\qquad\nf_k:=f_{-\\mu_k},\n$$\nand\n$$\nr_k:=\n\\begin{cases}\n(1+\\alpha)^{-1}, & 0\\leq k<K,\\\\[1mm]\n\\left(1+c_k\\alpha-\\Delta\\delta_k\\alpha^2\\right)^{-1}, & k\\geq K,\n\\end{cases}\n\\quad\n\\alpha=\\sqrt{\\frac{2}{\\kappa}},\n\\quad\n\\Delta\\delta_k:=\\delta_k-\\delta_{k+1}.\n$$\nConsequently, $\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k)$ \\RV{and $\\|y_k-x^\\star\\|^2$} converge $R$-linearly with asymptotic rate\n$$\nr_\\ast\n=\n\\frac1{1+2\\sqrt{2/\\kappa}}\n=\n1-2\\sqrt{2/\\kappa}+O(\\kappa^{-1})\n\\qquad\\text{as }\\kappa\\to\\infty.\n$$\n\\end{theorem}","sha256_utf8":"b638a657dfb57c84b0e4f83355ae7c4e197230ef9681ffc75729682bb1ed752a"}]},{"id":"S019","semantic_type":"proof","kind":"proof","title":"精细渐近率的完整证明","source_units":[{"start":100978,"end":104131,"start_line":2951,"end_line":3122,"raw_tex":"\\begin{proof}\nSince $f\\in\\mathcal S_{\\mu,L}^{\\rm LAS}(x^\\star)$, we have $\\omega_k\\downarrow0$, so the index $K$ in the theorem statement is finite. The threshold $1/8$ is only a convenient choice ensuring $q_k\\leq1/2$; any fixed sufficiently small threshold would work.\n\n\\medskip\n\\noindent\n\\textbf{Step 1: Initial finite segment.}\nFor $0\\leq k<K$, we use the unshifted choice\n$$\n\\delta_k=1,\\qquad\n\\mu_k=0,\\qquad\nb_k=0,\\qquad\nd_k=0,\\qquad\nc_k=1.\n$$\nThen\n$$\n\\widetilde{\\mathcal E}(\\bs z_{k+1};0)\n\\leq\n(1+\\alpha)^{-1}\n\\widetilde{\\mathcal E}(\\bs z_k;0),\n\\qquad\n0\\leq k<K.\n$$\n\n\\medskip\n\\noindent\n\\textbf{Step 2: Choice of the partial shifts.}\nFor $k\\geq K$, set\n$$\nq_k:=\\omega_k^{1/3},\\qquad\n\\delta_k:=q_k^2,\\qquad\nd_k:=1-q_k,\\qquad\nb_k:=d_kq_k.\n$$\nThis choice gives\n$$\nb_k\\delta_k=d_k\\omega_k.\n$$\nSince $\\omega_k\\leq1/8$, we have $0\\leq q_k\\leq1/2$, and hence $d_k,b_k\\geq0$. Moreover,\n$$\n\\begin{aligned}\nc_k\n&:=\n2-\\sqrt{\\delta_k}-b_k\n=\n1+(1-q_k)^2\n\\geq\\frac54,\\\\\n&q_k\\to0,\\qquad\n\\delta_k\\downarrow0,\\qquad\nb_k\\to0,\\qquad\nc_k\\to2.\n\\end{aligned}\n$$\n\n\\medskip\n\\noindent\n\\textbf{Step 3: Verification of Lemma~\\ref{lm:shifted_energy_recursion}.}\nSet\n$$\nr_k\n:=\n\\left(1+c_k\\alpha-\\Delta\\delta_k\\alpha^2\\right)^{-1}.\n$$\nSince\n$$\nc_k\\geq\\frac54,\n\\qquad\n0\\leq\\Delta\\delta_k\\leq\\delta_k=q_k^2\\leq\\frac14,\n$$\nand $\\kappa=L/\\mu\\geq1$ implies $\\alpha=\\sqrt{2/\\kappa}\\leq\\sqrt2$, we have\n\\begin{equation*}\n\\begin{aligned}\nr_k^{-1}\n&=\n1+c_k\\alpha-\\Delta\\delta_k\\alpha^2\n\\geq\n1+\\frac54\\alpha-\\frac14\\alpha^2\n=\n1+\\frac{\\alpha}{4}(5-\\alpha)\n>1.\n\\end{aligned}\n\\end{equation*}\n\nIt remains to verify \\eqref{eq:omega_condition}. Since $d_k\\geq1/2$,\n$$\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\geq\n\\frac12-\\frac{\\sqrt2}{8}\n>0.\n$$\n%Moreover,\n%$$\n%d_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n%\\leq\n%d_k.\n%$$\nTherefore,\n\\begin{equation*}\n\\left|\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\right|\\omega_k\n\\leq\nd_k\\omega_k\n=\nb_k\\delta_k.\n\\end{equation*}\nLemma~\\ref{lm:shifted_energy_recursion} now gives\n$$\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\nr_k\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k),\n\\qquad\nk\\geq K.\n$$\nHence\n$$\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\n\\widetilde{\\mathcal E}(\\bs z_K;\\mu_K)\n\\prod_{i=K}^{k}r_i.\n$$\nThe finitely many initial terms can be absorbed into the constant. Specifically, let\n$$\nC\n:=\n1+\n\\max_{0\\leq j\\leq K}\n\\left\\{\n\\widetilde{\\mathcal E}(\\bs z_{j+1};\\mu_{j+1})\n\\left(\\prod_{i=0}^{j}r_i\\right)^{-1}\n\\right\\},\n$$\nThen, for all $k\\geq0$,\n$$\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\nC\\prod_{i=0}^{k}r_i.\n$$\n\n\\medskip\n\\noindent\n\\textbf{Step 4: Asymptotic rate.}\nSince $c_k\\to2$ and $\\Delta\\delta_k\\to0$,\n$$\nr_k\n\\to\nr_\\ast\n:=\n\\left(1+2\\sqrt{2/\\kappa}\\right)^{-1}.\n$$\nTherefore,\n%$$\n%\\left(\\prod_{i=0}^{k}r_i\\right)^{1/k}\n%\\to\n%r_\\ast.\n%$$\n%It follows that\n$$\n\\begin{aligned}\n\\limsup_{k\\to\\infty}\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})^{1/k}\n&\\leq\n\\limsup_{k\\to\\infty}\n\\left(\nC\\prod_{i=0}^{k}r_i\n\\right)^{1/k}\n=\nr_\\ast.\n\\end{aligned}\n$$\nHence $\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k)$ converges $R$-linearly with asymptotic rate $r_\\ast$. \\RV{The bound in the theorem then gives the same asymptotic rate for $\\|y_k-x^\\star\\|^2$.} $\\Box$\n\\end{proof}","sha256_utf8":"1d15ed46f626f93b91938f36581f3438de63ee00ee0b8f76a7e553b18259d14b"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"确定有限进入时刻","source_units":[{"start":100978,"end":101250,"start_line":2951,"end_line":2953,"raw_tex":"\\begin{proof}\nSince $f\\in\\mathcal S_{\\mu,L}^{\\rm LAS}(x^\\star)$, we have $\\omega_k\\downarrow0$, so the index $K$ in the theorem statement is finite. The threshold $1/8$ is only a convenient choice ensuring $q_k\\leq1/2$; any fixed sufficiently small threshold would work.\n\n","sha256_utf8":"9ea86bdb1b39bf1fc0b503f88b607e4965ccc613df79ee56be2240ced6d287bc"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"处理有限初始区间","source_units":[{"start":101250,"end":101551,"start_line":2954,"end_line":2974,"raw_tex":"\\medskip\n\\noindent\n\\textbf{Step 1: Initial finite segment.}\nFor $0\\leq k<K$, we use the unshifted choice\n$$\n\\delta_k=1,\\qquad\n\\mu_k=0,\\qquad\nb_k=0,\\qquad\nd_k=0,\\qquad\nc_k=1.\n$$\nThen\n$$\n\\widetilde{\\mathcal E}(\\bs z_{k+1};0)\n\\leq\n(1+\\alpha)^{-1}\n\\widetilde{\\mathcal E}(\\bs z_k;0),\n\\qquad\n0\\leq k<K.\n$$\n\n","sha256_utf8":"3cbab12afa45a1d5d3e2199c9256fce0391d6618c1b032619af9edcc1617ec38"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"构造渐近部分移位","source_units":[{"start":101551,"end":102024,"start_line":2975,"end_line":3004,"raw_tex":"\\medskip\n\\noindent\n\\textbf{Step 2: Choice of the partial shifts.}\nFor $k\\geq K$, set\n$$\nq_k:=\\omega_k^{1/3},\\qquad\n\\delta_k:=q_k^2,\\qquad\nd_k:=1-q_k,\\qquad\nb_k:=d_kq_k.\n$$\nThis choice gives\n$$\nb_k\\delta_k=d_k\\omega_k.\n$$\nSince $\\omega_k\\leq1/8$, we have $0\\leq q_k\\leq1/2$, and hence $d_k,b_k\\geq0$. Moreover,\n$$\n\\begin{aligned}\nc_k\n&:=\n2-\\sqrt{\\delta_k}-b_k\n=\n1+(1-q_k)^2\n\\geq\\frac54,\\\\\n&q_k\\to0,\\qquad\n\\delta_k\\downarrow0,\\qquad\nb_k\\to0,\\qquad\nc_k\\to2.\n\\end{aligned}\n$$\n\n","sha256_utf8":"e83ee47985add8096ba28d7e55e3387647bcdf4e1603c96b19d3ca1bec82a13e"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"验证收缩分母大于一","source_units":[{"start":102024,"end":102533,"start_line":3005,"end_line":3033,"raw_tex":"\\medskip\n\\noindent\n\\textbf{Step 3: Verification of Lemma~\\ref{lm:shifted_energy_recursion}.}\nSet\n$$\nr_k\n:=\n\\left(1+c_k\\alpha-\\Delta\\delta_k\\alpha^2\\right)^{-1}.\n$$\nSince\n$$\nc_k\\geq\\frac54,\n\\qquad\n0\\leq\\Delta\\delta_k\\leq\\delta_k=q_k^2\\leq\\frac14,\n$$\nand $\\kappa=L/\\mu\\geq1$ implies $\\alpha=\\sqrt{2/\\kappa}\\leq\\sqrt2$, we have\n\\begin{equation*}\n\\begin{aligned}\nr_k^{-1}\n&=\n1+c_k\\alpha-\\Delta\\delta_k\\alpha^2\n\\geq\n1+\\frac54\\alpha-\\frac14\\alpha^2\n=\n1+\\frac{\\alpha}{4}(5-\\alpha)\n>1.\n\\end{aligned}\n\\end{equation*}\n\n","sha256_utf8":"b31224d6c2b84f554eeb4412a7a03b9e7a4802d9feb5fc793c73b55e016908a6"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"验证非对称扰动吸收条件","source_units":[{"start":102533,"end":102879,"start_line":3034,"end_line":3056,"raw_tex":"It remains to verify \\eqref{eq:omega_condition}. Since $d_k\\geq1/2$,\n$$\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\geq\n\\frac12-\\frac{\\sqrt2}{8}\n>0.\n$$\n%Moreover,\n%$$\n%d_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n%\\leq\n%d_k.\n%$$\nTherefore,\n\\begin{equation*}\n\\left|\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\right|\\omega_k\n\\leq\nd_k\\omega_k\n=\nb_k\\delta_k.\n\\end{equation*}\n","sha256_utf8":"9f9d7d754db2dfb2623680988f4bf6ab372762ae2c63a11dcbade6abfc22cebe"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"应用变化移位能量递推","source_units":[{"start":102879,"end":103165,"start_line":3057,"end_line":3071,"raw_tex":"Lemma~\\ref{lm:shifted_energy_recursion} now gives\n$$\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\nr_k\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k),\n\\qquad\nk\\geq K.\n$$\nHence\n$$\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\n\\widetilde{\\mathcal E}(\\bs z_K;\\mu_K)\n\\prod_{i=K}^{k}r_i.\n$$\n","sha256_utf8":"26d3418521729f6bc1b62b167caf1e50c7a340c75e079509280d1ddef2dd1784"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"合并初始项得到乘积界","source_units":[{"start":103165,"end":103489,"start_line":3072,"end_line":3089,"raw_tex":"The finitely many initial terms can be absorbed into the constant. Specifically, let\n$$\nC\n:=\n1+\n\\max_{0\\leq j\\leq K}\n\\left\\{\n\\widetilde{\\mathcal E}(\\bs z_{j+1};\\mu_{j+1})\n\\left(\\prod_{i=0}^{j}r_i\\right)^{-1}\n\\right\\},\n$$\nThen, for all $k\\geq0$,\n$$\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\nC\\prod_{i=0}^{k}r_i.\n$$\n\n","sha256_utf8":"628c76e03fc75f80dd18f8e6f23a47b0e6fcb524cce9d1e60888dfd94f547a00"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"取根极限得到渐近率","source_units":[{"start":103489,"end":104131,"start_line":3090,"end_line":3122,"raw_tex":"\\medskip\n\\noindent\n\\textbf{Step 4: Asymptotic rate.}\nSince $c_k\\to2$ and $\\Delta\\delta_k\\to0$,\n$$\nr_k\n\\to\nr_\\ast\n:=\n\\left(1+2\\sqrt{2/\\kappa}\\right)^{-1}.\n$$\nTherefore,\n%$$\n%\\left(\\prod_{i=0}^{k}r_i\\right)^{1/k}\n%\\to\n%r_\\ast.\n%$$\n%It follows that\n$$\n\\begin{aligned}\n\\limsup_{k\\to\\infty}\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})^{1/k}\n&\\leq\n\\limsup_{k\\to\\infty}\n\\left(\nC\\prod_{i=0}^{k}r_i\n\\right)^{1/k}\n=\nr_\\ast.\n\\end{aligned}\n$$\nHence $\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k)$ converges $R$-linearly with asymptotic rate $r_\\ast$. \\RV{The bound in the theorem then gives the same asymptotic rate for $\\|y_k-x^\\star\\|^2$.} $\\Box$\n\\end{proof}","sha256_utf8":"13a99d2c79fcc1c30cebd928571c0bd6366071546d5c49bed47cabfc1349121a"}]},{"id":"S028","semantic_type":"lemma","kind":"lemma","title":"未移位强李雅普诺夫性质","source_units":[{"start":20949,"end":21344,"start_line":437,"end_line":445,"raw_tex":"\\begin{lemma}\\label{lem:cont_strong}\nLet $\\mathcal G(\\boldsymbol z)$ be the vector field of the HNAG flow~\\eqref{eq:flow}.\nThen $E(\\boldsymbol z)$ satisfies the following strong Lyapunov condition\n\\begin{equation}\\label{eq:A-HNAG}\n-\\langle\\nabla E(\\boldsymbol z),\\mathcal G(\\boldsymbol z)\\rangle\n\\;\\ge\\;\nE(\\boldsymbol z)+\\beta\\|\\nabla f(x)\\|^2+\\frac{\\mu}{2}\\|x-y\\|^2 .\n\\end{equation}\n\\end{lemma}","sha256_utf8":"89d6e69e428c231a07f911f47c25795aad26a3c77c6ba077eb808a53b2bca626"}]},{"id":"S029","semantic_type":"proof","kind":"proof","title":"未移位强耗散性质证明","source_units":[{"start":21346,"end":21925,"start_line":447,"end_line":467,"raw_tex":"\\begin{proof}\nA direct computation yields\n$$\n\\begin{aligned}\n&\\quad -\\langle\\nabla E(\\boldsymbol z),\\mathcal G(\\boldsymbol z)\\rangle \\\\ \n&=\n\\dual{\\nabla f(x),x-x^\\star}\n+\\beta\\|\\nabla f(x)\\|^2\n+\\mu\\|y-x^\\star\\|^2\n-\\mu\\dual{y-x^\\star,x-x^\\star} \\\\\n&=\nE(\\boldsymbol z)\n+D_f(x^\\star,x)\n+\\beta\\|\\nabla f(x)\\|^2\n+\\frac{\\mu}{2}\\|x-y\\|^2\n-\\frac{\\mu}{2}\\|x-x^\\star\\|^2,\n\\end{aligned}\n$$\nwhere we use identities of squares~\\eqref{eq:squares} to expand the cross term. \nUsing the $\\mu$-convexity $D_f(x^\\star,x)\\ge \\tfrac{\\mu}{2}\\|x-x^\\star\\|^2$ gives \\eqref{eq:A-HNAG}.\n$\\Box$ \\end{proof}","sha256_utf8":"72840eb651fc14242fec17786c90555e53bfadf1d82431ff90239509839590db"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"展开能量沿流的导数","source_units":[{"start":21346,"end":21806,"start_line":447,"end_line":465,"raw_tex":"\\begin{proof}\nA direct computation yields\n$$\n\\begin{aligned}\n&\\quad -\\langle\\nabla E(\\boldsymbol z),\\mathcal G(\\boldsymbol z)\\rangle \\\\ \n&=\n\\dual{\\nabla f(x),x-x^\\star}\n+\\beta\\|\\nabla f(x)\\|^2\n+\\mu\\|y-x^\\star\\|^2\n-\\mu\\dual{y-x^\\star,x-x^\\star} \\\\\n&=\nE(\\boldsymbol z)\n+D_f(x^\\star,x)\n+\\beta\\|\\nabla f(x)\\|^2\n+\\frac{\\mu}{2}\\|x-y\\|^2\n-\\frac{\\mu}{2}\\|x-x^\\star\\|^2,\n\\end{aligned}\n$$\nwhere we use identities of squares~\\eqref{eq:squares} to expand the cross term. \n","sha256_utf8":"01a7783576d613603a43439d6ab534d4cdc156170e18a6fc683a7172a9054223"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"以强凸性消去负平方项","source_units":[{"start":21806,"end":21925,"start_line":466,"end_line":467,"raw_tex":"Using the $\\mu$-convexity $D_f(x^\\star,x)\\ge \\tfrac{\\mu}{2}\\|x-x^\\star\\|^2$ gives \\eqref{eq:A-HNAG}.\n$\\Box$ \\end{proof}","sha256_utf8":"9a585dd359ca2394392195ba473ac0dc3bf096889e04cb2dfbd6cdc380ff269f"}]},{"id":"S032","semantic_type":"lemma","kind":"lemma","title":"HNAG基本单步能量界","source_units":[{"start":24043,"end":24594,"start_line":499,"end_line":510,"raw_tex":"\\begin{lemma}\\label{lem:disc_strong}\nLet $(x_k,y_k)$ be the iterates generated by~\\eqref{eq:HNAG} and $E$ defined by \\eqref{eq:lyapunov_strong_HNAG}. Then\n\\begin{equation}\\label{eq:onestep}\n\\begin{aligned}\n(1+\\alpha)E(\\bs z_{k+1}) \\;\\le\\;& E(\\bs z_k)\n+ \\Bigl(\\frac{\\alpha^2}{2\\mu}-\\frac{\\alpha\\beta}{2}\\Bigr)\\|\\nabla f(x_{k+1})\\|^2\n- \\frac{\\alpha\\beta}{2}\\|\\nabla f(x_k)\\|^2 \\\\\n\\qquad-\\frac{\\alpha \\mu}{2}\\|x_{k+1} - y_{k+1}\\|^2 &+ \\frac{\\alpha\\beta}{2}\\|\\nabla f(x_{k+1})-\\nabla f(x_k)\\|^2\n- D_f(x_k,x_{k+1}).\n\\end{aligned}\n\\end{equation}\n\\end{lemma}","sha256_utf8":"e4e860b3359e837c4f8211f708044da4b95fa8d5d2dd70bd2c6d946e28271a59"}]},{"id":"S033","semantic_type":"proof","kind":"proof","title":"基本单步能量界证明","source_units":[{"start":24595,"end":26750,"start_line":511,"end_line":572,"raw_tex":"\\begin{proof}\nExpand the difference of $E$ by the definition of the Bregman divergence:\n\\begin{equation*}\nE(\\bs z_{k+1})-E(\\bs z_k)\n= \\langle \\nabla E(\\bs z_{k+1}),\\,\\bs z_{k+1}-\\bs z_k\\rangle\n- D_E(\\bs z_k,\\bs z_{k+1}),\n\\end{equation*}\nwhere\n\\begin{equation}\\label{eq:DEHNAG}\n- D_E(\\bs z_k,\\bs z_{k+1})\n= -D_f(x_k,x_{k+1})-\\frac{\\mu}{2}\\|y_k-y_{k+1}\\|^2 .\n\\end{equation}\n\nWrite the deviation from the implicit Euler step as\n$$\n\\bs z_{k+1}-\\bs z_k\n= \\alpha \\mathcal G(\\bs z_{k+1})\n+ \\alpha\n\\begin{pmatrix}\ny_k-y_{k+1}+\\beta(\\nabla f(x_{k+1})-\\nabla f(x_k))\\\\\n0\n\\end{pmatrix}.\n$$\n\\medskip\n\\noindent\n\\textbf{The implicit Euler term.} We apply the strong Lyapunov property \\eqref{eq:A-HNAG} at $\\bs z_{k+1}$ to the implicit Euler term\n\\begin{equation*}\n\\langle \\nabla E(\\bs z_{k+1}),\\,\\alpha \\mathcal G(\\bs z_{k+1}) \\rangle \\leq - \\alpha E(\\boldsymbol z_{k+1}) - \\alpha\\beta\\|\\nabla f(x_{k+1})\\|^2-\\frac{\\alpha \\mu}{2}\\|x_{k+1} - y_{k+1}\\|^2.\n\\end{equation*}\n\n\\medskip\n\\noindent\n\\textbf{Lagging of \\(y\\).}\n%\n%This yields two cross terms in the Lyapunov difference:\n%\\begin{align*}\n%\\alpha\\langle \\nabla f(x_{k+1}),\\,y_k-y_{k+1}\\rangle\\quad\n%\\mathrm{and}\\quad \\alpha\\beta\\langle \\nabla f(x_{k+1}),\\,\\nabla f(x_{k+1})-\\nabla f(x_k)\\rangle .\n%\\end{align*}\nFor the term $\\alpha\\langle \\nabla f(x_{k+1}),\\,y_k-y_{k+1}\\rangle$ arising from the lagging of $y$, the Cauchy--Schwarz and Young inequalities give\n\\begin{equation}\\label{eq:firstcross}\n\\alpha\\langle \\nabla f(x_{k+1}),\\,y_k-y_{k+1}\\rangle\n\\le \\frac{\\alpha^2}{2\\mu}\\|\\nabla f(x_{k+1})\\|^2\n+ \\frac{\\mu}{2}\\|y_k-y_{k+1}\\|^2,\n\\end{equation}\nwhose second term cancels with \\eqref{eq:DEHNAG}. \n\n\\medskip\n\\noindent\n\\textbf{Lagging of \\(\\nabla f(x)\\).}\nFor the gradient cross term, arising from the lagging of $\\nabla f(x)$,\nthe identity of squares yields\n$$\n\\begin{aligned}\n\\alpha\\beta\\langle \\nabla f(x_{k+1}),\\,\\nabla f(x_{k+1})-\\nabla f(x_k)\\rangle\n&= -\\frac{\\alpha\\beta}{2}\\|\\nabla f(x_k)\\|^2\n+ \\frac{\\alpha\\beta}{2}\\|\\nabla f(x_{k+1})\\|^2 \\\\\n&\\quad + \\frac{\\alpha\\beta}{2}\\|\\nabla f(x_{k+1})-\\nabla f(x_k)\\|^2 .\n\\end{aligned}\n$$\nCombining and rearranging terms yields \\eqref{eq:onestep}.\n$\\Box$ \\end{proof}","sha256_utf8":"ef0f0f2a8235da1f56d2ead1e64a1ed84459a6eaf724c79f98b7727240f6d3dc"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"展开离散能量差","source_units":[{"start":24595,"end":24968,"start_line":511,"end_line":523,"raw_tex":"\\begin{proof}\nExpand the difference of $E$ by the definition of the Bregman divergence:\n\\begin{equation*}\nE(\\bs z_{k+1})-E(\\bs z_k)\n= \\langle \\nabla E(\\bs z_{k+1}),\\,\\bs z_{k+1}-\\bs z_k\\rangle\n- D_E(\\bs z_k,\\bs z_{k+1}),\n\\end{equation*}\nwhere\n\\begin{equation}\\label{eq:DEHNAG}\n- D_E(\\bs z_k,\\bs z_{k+1})\n= -D_f(x_k,x_{k+1})-\\frac{\\mu}{2}\\|y_k-y_{k+1}\\|^2 .\n\\end{equation}\n\n","sha256_utf8":"b390034851d89083926680a63f754e24caa4baab5decb9581b4ec1ed25f9da39"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"分解隐式步与滞后修正","source_units":[{"start":24968,"end":25193,"start_line":524,"end_line":535,"raw_tex":"Write the deviation from the implicit Euler step as\n$$\n\\bs z_{k+1}-\\bs z_k\n= \\alpha \\mathcal G(\\bs z_{k+1})\n+ \\alpha\n\\begin{pmatrix}\ny_k-y_{k+1}+\\beta(\\nabla f(x_{k+1})-\\nabla f(x_k))\\\\\n0\n\\end{pmatrix}.\n$$\n\\medskip\n\\noindent\n","sha256_utf8":"14632a8b6206dea348b41827c3f6bc899e2736980ef5ddcf1e94edfa955a0982"}]},{"id":"S036","semantic_type":"proof_step","kind":"proof_step","title":"估计隐式步耗散","source_units":[{"start":25193,"end":25571,"start_line":536,"end_line":542,"raw_tex":"\\textbf{The implicit Euler term.} We apply the strong Lyapunov property \\eqref{eq:A-HNAG} at $\\bs z_{k+1}$ to the implicit Euler term\n\\begin{equation*}\n\\langle \\nabla E(\\bs z_{k+1}),\\,\\alpha \\mathcal G(\\bs z_{k+1}) \\rangle \\leq - \\alpha E(\\boldsymbol z_{k+1}) - \\alpha\\beta\\|\\nabla f(x_{k+1})\\|^2-\\frac{\\alpha \\mu}{2}\\|x_{k+1} - y_{k+1}\\|^2.\n\\end{equation*}\n\n\\medskip\n\\noindent\n","sha256_utf8":"2a2c313b372ea671f264e1cd6375e7f1a29a193dca9fd13c4a3f3272065ee328"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"控制辅助变量滞后项","source_units":[{"start":25571,"end":26254,"start_line":543,"end_line":559,"raw_tex":"\\textbf{Lagging of \\(y\\).}\n%\n%This yields two cross terms in the Lyapunov difference:\n%\\begin{align*}\n%\\alpha\\langle \\nabla f(x_{k+1}),\\,y_k-y_{k+1}\\rangle\\quad\n%\\mathrm{and}\\quad \\alpha\\beta\\langle \\nabla f(x_{k+1}),\\,\\nabla f(x_{k+1})-\\nabla f(x_k)\\rangle .\n%\\end{align*}\nFor the term $\\alpha\\langle \\nabla f(x_{k+1}),\\,y_k-y_{k+1}\\rangle$ arising from the lagging of $y$, the Cauchy--Schwarz and Young inequalities give\n\\begin{equation}\\label{eq:firstcross}\n\\alpha\\langle \\nabla f(x_{k+1}),\\,y_k-y_{k+1}\\rangle\n\\le \\frac{\\alpha^2}{2\\mu}\\|\\nabla f(x_{k+1})\\|^2\n+ \\frac{\\mu}{2}\\|y_k-y_{k+1}\\|^2,\n\\end{equation}\nwhose second term cancels with \\eqref{eq:DEHNAG}. \n\n\\medskip\n\\noindent\n","sha256_utf8":"ebccfea9c9358f99bb6b29df0c03c8a9939f5a6d4548a4937cd268f90540810a"}]},{"id":"S038","semantic_type":"proof_step","kind":"proof_step","title":"展开梯度滞后并合并","source_units":[{"start":26254,"end":26750,"start_line":560,"end_line":572,"raw_tex":"\\textbf{Lagging of \\(\\nabla f(x)\\).}\nFor the gradient cross term, arising from the lagging of $\\nabla f(x)$,\nthe identity of squares yields\n$$\n\\begin{aligned}\n\\alpha\\beta\\langle \\nabla f(x_{k+1}),\\,\\nabla f(x_{k+1})-\\nabla f(x_k)\\rangle\n&= -\\frac{\\alpha\\beta}{2}\\|\\nabla f(x_k)\\|^2\n+ \\frac{\\alpha\\beta}{2}\\|\\nabla f(x_{k+1})\\|^2 \\\\\n&\\quad + \\frac{\\alpha\\beta}{2}\\|\\nabla f(x_{k+1})-\\nabla f(x_k)\\|^2 .\n\\end{aligned}\n$$\nCombining and rearranging terms yields \\eqref{eq:onestep}.\n$\\Box$ \\end{proof}","sha256_utf8":"ae6cad8d4ac87778442126d4cdc6d6556a4577d6ba8153a510401920aa6a3795"}]},{"id":"S039","semantic_type":"theorem","kind":"theorem","title":"大步长算法的全局基准率","source_units":[{"start":65547,"end":66690,"start_line":1664,"end_line":1704,"raw_tex":"\\begin{theorem}[\\((1-\\sqrt{2/\\kappa})\\)-linear convergence]\\label{thm:linearconvHNAG-funcval}\nLet \\(f\\in\\mathcal S_{\\mu,L}\\), and let \\(E\\) be defined by \\eqref{eq:lyapunov_strong_HNAG}. For the iterates \\((x_k,y_k)\\) generated by the HNAG scheme~\\eqref{eq:HNAG} with\n$$\n\\alpha=\\sqrt{2\\mu/L},\\quad \\alpha\\beta=1/L,\n$$\nor equivalently by Algorithm~\\ref{alg:HNAGpp}, define\n$$\n\\widetilde E_k:=E(x_k,y_k)-\\frac1{2L}\\|\\nabla f(x_k)\\|^2.\n$$\n%By co-coercivity,\n%$$\n%\\widetilde E_k\n%\\geq\n%\\frac{\\mu}{2}\\|y_k-x^\\star\\|^2\n%\\geq0.\n%$$\nThen, for all \\(k\\ge0\\),\n\\begin{equation}\\label{eq:HNAGrate}\n\\frac{\\mu}{2}\\|y_{k+1}-x^\\star\\|^2 \\leq \\widetilde E_{k+1}\n\\le\n\\frac1{1+\\sqrt{2/\\kappa}}\\widetilde E_k\n\\le\n\\widetilde E_0\\left(\\frac1{1+\\sqrt{2/\\kappa}}\\right)^{k+1}.\n\\end{equation}\nMoreover, for all \\(k\\ge1\\),\n\\begin{equation}\\label{eq:HNAG-grad-bound}\n\\|\\nabla f(x_k)\\|^2\n\\le\n\\frac{2L(1+\\alpha)}{\\alpha}\\,\n\\widetilde E_0\n\\left(\\frac1{1+\\alpha}\\right)^k,\n\\end{equation}\nand\n\\begin{equation}\\label{eq:HNAG-strong-components}\n\\|x_k-x^\\star\\|^2\n\\le\n\\frac{2(1+2\\alpha)}{\\alpha\\mu}\\,\n\\widetilde E_0\n\\left(\\frac1{1+\\alpha}\\right)^k .\n\\end{equation}\n\\end{theorem}","sha256_utf8":"d45b9fd77183964397bafc24c8536225de1dfbd4485fad785a2ae96c8155dbd5"}]},{"id":"S040","semantic_type":"proof","kind":"proof","title":"全局基准率及分量界证明","source_units":[{"start":66692,"end":68605,"start_line":1706,"end_line":1782,"raw_tex":"\\begin{proof}\nWith \\(\\alpha\\beta=1/L\\) and \\(\\alpha^2=2\\mu/L\\), the one-step estimate \\eqref{eq:onestep} becomes\n$$\n\\begin{aligned}\n(1+\\alpha)E(\\bs z_{k+1})\n\\le{}&\nE(\\bs z_k)\n+\\frac1{2L}\\|\\nabla f(x_{k+1})\\|^2\n-\\frac1{2L}\\|\\nabla f(x_k)\\|^2\\\\\n&\\  -\\frac{\\alpha\\mu}{2}\\|x_{k+1}-y_{k+1}\\|^2 .\n\\end{aligned}\n$$\nEquivalently,\n\\begin{equation}\\label{eq:one-step-energy-E}\n(1+\\alpha)\\widetilde E_{k+1}\n\\le\n\\widetilde E_k\n-\\frac{\\alpha\\mu}{2}\\|x_{k+1}-y_{k+1}\\|^2\n-\\frac{\\alpha}{2L}\\|\\nabla f(x_{k+1})\\|^2 .\n\\end{equation}\nDropping the negative terms gives \\eqref{eq:HNAGrate}.\n\nNext, moving the negative terms in \\eqref{eq:one-step-energy-E} to the left gives\n$$\n(1+\\alpha)\\widetilde E_{k+1}\n+\\frac{\\alpha\\mu}{2}\\|x_{k+1}-y_{k+1}\\|^2\n+\\frac{\\alpha}{2L}\\|\\nabla f(x_{k+1})\\|^2\n\\le\n\\widetilde E_k .\n$$\nShifting the index, we get, for \\(k\\ge1\\),\n$$\n\\frac{\\alpha}{2L}\\|\\nabla f(x_k)\\|^2\n\\le\n\\widetilde E_{k-1}\n\\le\n(1+\\alpha)\\widetilde E_0\\left(\\frac1{1+\\alpha}\\right)^k .\n$$\nThis proves \\eqref{eq:HNAG-grad-bound}. \n%Similarly,\n%\\[\n%\\frac{\\alpha\\mu}{2}\\|x_k-y_k\\|^2\n%\\le\n%\\widetilde E_{k-1}\n%\\le\n%(1+\\alpha)\\widetilde E_0\\left(\\frac1{1+\\alpha}\\right)^k .\n%\\]\n%Moreover, since \\(f\\in\\mathcal S_{\\mu,L}\\), $f(x_k)-f(x^\\star)-\\frac1{2L}\\|\\nabla f(x_k)\\|^2\\ge0,$\n%and hence\n%\\[\n%\\frac{\\mu}{2}\\|y_k-x^\\star\\|^2\n%\\le\n%\\widetilde E_k\n%\\le\n%\\widetilde E_0\\left(\\frac1{1+\\alpha}\\right)^k .\n%\\]\n%Therefore, by the triangle inequality,\n%\\[\n%\\begin{aligned}\n%\\frac{\\alpha\\mu}{2}\\|x_k-x^\\star\\|^2\n%&\\le\n%\\alpha\\mu\\|x_k-y_k\\|^2\n%+\n%\\alpha\\mu\\|y_k-x^\\star\\|^2 \n%\\leq \n%2(1+2\\alpha)\\widetilde E_0\\left(\\frac1{1+\\alpha}\\right)^k .\n%\\end{aligned}\n%\\]\nWe use the $\\mu$-strong convexity of $f$ to conclude \n$$\n\\frac{\\mu}{2}\\|x_k-x^{\\star}\\|^2 \\leq \\widetilde E_k + \\frac1{2L}\\|\\nabla f(x_k)\\|^2\\leq\n\\frac{(1+2\\alpha)}{\\alpha}\\,\n\\widetilde E_0\n\\left(\\frac1{1+\\alpha}\\right)^k .\n$$\nThis gives \\eqref{eq:HNAG-strong-components}.\n$\\Box$\\end{proof}","sha256_utf8":"1c18e7cffd8ac837f8546f55e276301879f8d3c7dee81379b933aa8588756393"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"对偶移位得到全局收缩","source_units":[{"start":66692,"end":67264,"start_line":1706,"end_line":1727,"raw_tex":"\\begin{proof}\nWith \\(\\alpha\\beta=1/L\\) and \\(\\alpha^2=2\\mu/L\\), the one-step estimate \\eqref{eq:onestep} becomes\n$$\n\\begin{aligned}\n(1+\\alpha)E(\\bs z_{k+1})\n\\le{}&\nE(\\bs z_k)\n+\\frac1{2L}\\|\\nabla f(x_{k+1})\\|^2\n-\\frac1{2L}\\|\\nabla f(x_k)\\|^2\\\\\n&\\  -\\frac{\\alpha\\mu}{2}\\|x_{k+1}-y_{k+1}\\|^2 .\n\\end{aligned}\n$$\nEquivalently,\n\\begin{equation}\\label{eq:one-step-energy-E}\n(1+\\alpha)\\widetilde E_{k+1}\n\\le\n\\widetilde E_k\n-\\frac{\\alpha\\mu}{2}\\|x_{k+1}-y_{k+1}\\|^2\n-\\frac{\\alpha}{2L}\\|\\nabla f(x_{k+1})\\|^2 .\n\\end{equation}\nDropping the negative terms gives \\eqref{eq:HNAGrate}.\n\n","sha256_utf8":"97e36d68e8bbe688fbfcd65e08371f4dcbd350d5f5ce41116c28bab7e6590b24"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"保留耗散得到梯度界","source_units":[{"start":67264,"end":67698,"start_line":1728,"end_line":1744,"raw_tex":"Next, moving the negative terms in \\eqref{eq:one-step-energy-E} to the left gives\n$$\n(1+\\alpha)\\widetilde E_{k+1}\n+\\frac{\\alpha\\mu}{2}\\|x_{k+1}-y_{k+1}\\|^2\n+\\frac{\\alpha}{2L}\\|\\nabla f(x_{k+1})\\|^2\n\\le\n\\widetilde E_k .\n$$\nShifting the index, we get, for \\(k\\ge1\\),\n$$\n\\frac{\\alpha}{2L}\\|\\nabla f(x_k)\\|^2\n\\le\n\\widetilde E_{k-1}\n\\le\n(1+\\alpha)\\widetilde E_0\\left(\\frac1{1+\\alpha}\\right)^k .\n$$\nThis proves \\eqref{eq:HNAG-grad-bound}. \n","sha256_utf8":"ad824dbac4eeea9b5ba5df43052b507bdd0aa13e6752c43aacb3c876460a53c1"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"强凸性给出原变量误差界","source_units":[{"start":67698,"end":68605,"start_line":1745,"end_line":1782,"raw_tex":"%Similarly,\n%\\[\n%\\frac{\\alpha\\mu}{2}\\|x_k-y_k\\|^2\n%\\le\n%\\widetilde E_{k-1}\n%\\le\n%(1+\\alpha)\\widetilde E_0\\left(\\frac1{1+\\alpha}\\right)^k .\n%\\]\n%Moreover, since \\(f\\in\\mathcal S_{\\mu,L}\\), $f(x_k)-f(x^\\star)-\\frac1{2L}\\|\\nabla f(x_k)\\|^2\\ge0,$\n%and hence\n%\\[\n%\\frac{\\mu}{2}\\|y_k-x^\\star\\|^2\n%\\le\n%\\widetilde E_k\n%\\le\n%\\widetilde E_0\\left(\\frac1{1+\\alpha}\\right)^k .\n%\\]\n%Therefore, by the triangle inequality,\n%\\[\n%\\begin{aligned}\n%\\frac{\\alpha\\mu}{2}\\|x_k-x^\\star\\|^2\n%&\\le\n%\\alpha\\mu\\|x_k-y_k\\|^2\n%+\n%\\alpha\\mu\\|y_k-x^\\star\\|^2 \n%\\leq \n%2(1+2\\alpha)\\widetilde E_0\\left(\\frac1{1+\\alpha}\\right)^k .\n%\\end{aligned}\n%\\]\nWe use the $\\mu$-strong convexity of $f$ to conclude \n$$\n\\frac{\\mu}{2}\\|x_k-x^{\\star}\\|^2 \\leq \\widetilde E_k + \\frac1{2L}\\|\\nabla f(x_k)\\|^2\\leq\n\\frac{(1+2\\alpha)}{\\alpha}\\,\n\\widetilde E_0\n\\left(\\frac1{1+\\alpha}\\right)^k .\n$$\nThis gives \\eqref{eq:HNAG-strong-components}.\n$\\Box$\\end{proof}","sha256_utf8":"b4f4e194258d2ac31974c23a5c0ccb84852addce4353520082bb503ae8eb583b"}]},{"id":"S044","semantic_type":"lemma","kind":"lemma","title":"部分移位的精细耗散界","source_units":[{"start":72631,"end":73583,"start_line":1869,"end_line":1894,"raw_tex":"\\begin{lemma}\\label{lem:energy_identity}\nLet $0\\leq\\hat\\mu\\leq\\mu$ and set\n$\\delta:=(\\mu-\\hat\\mu)/\\mu\\in[0,1]$.\nFor the Lyapunov function $\\mathcal E(\\bs z;\\hat\\mu)$ defined in \\eqref{eq:lya-muhat} and the vector field $\\mathcal G$ defined in \\eqref{eq:G}, we have\n\\begin{equation}\\label{eq:energy_identity}\n\\begin{aligned}\n-\\big\\langle \\nabla \\mathcal E(\\bs z;\\hat\\mu),\\mathcal G(\\bs z)\\big\\rangle\n={}&\n\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n+\\mu\\|y-x^\\star\\|^2\n-\\delta\\mu\\langle y-x^\\star,x-x^\\star\\rangle\n\\\\\n&\\quad\n+\\beta\\|\\nabla f_{-\\hat\\mu}(x)\\|^2\n+\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n\\\\\n\\geq{}&\n\\left(2-\\sqrt{\\delta}-b\\right)\\mathcal E(\\bs z;\\hat\\mu)\n+\\RV{bD_{f_{-\\hat\\mu}}(x,x^\\star)}+\\beta\\|\\nabla f_{-\\hat\\mu}(x)\\|^2\\\\\n&\\quad\n+\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n+\\left(1-\\sqrt{\\delta}\\right)\\Delta_f(x,x^\\star),\n\\end{aligned}\n\\end{equation}\nwhere $0\\leq b<2-\\sqrt{\\delta}$.\n\\end{lemma}","sha256_utf8":"3fda101df0209402e556531b555c8900debe9c184caa6f55e3c6f119afe0f14b"}]},{"id":"S045","semantic_type":"proof","kind":"proof","title":"部分移位耗散界证明","source_units":[{"start":73585,"end":76667,"start_line":1896,"end_line":2028,"raw_tex":"\\begin{proof}\nA direct calculation gives\n$$\n\\begin{aligned}\n-\\big\\langle \\nabla \\mathcal E(\\bs z;\\hat\\mu),\\mathcal G(\\bs z)\\big\\rangle\n={}&\n\\left\\langle\n\\begin{pmatrix}\n\\nabla f_{-\\hat\\mu}(x)\\\\[2pt]\n\\mu(y-x^\\star)\n\\end{pmatrix},\n\\begin{pmatrix}\nx-y+\\beta\\nabla f_{-\\hat\\mu}(x)+\\beta\\hat\\mu(x-x^\\star)\\\\\ny-x^\\star+\\dfrac1\\mu\\nabla f_{-\\hat\\mu}(x)-\\delta(x-x^\\star)\n\\end{pmatrix}\n\\right\\rangle\n\\\\\n={}&\n\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n+\\mu\\|y-x^\\star\\|^2\n-\\delta\\mu\\langle y-x^\\star,x-x^\\star\\rangle\n\\\\\n&\\quad\n+\\beta\\|\\nabla f_{-\\hat\\mu}(x)\\|^2\n+\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle.\n\\end{aligned}\n$$\nThis proves the identity. \\RV{We keep the term $\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle$ separate, since it controls the $\\hat \\mu$-shift term in Lemma~\\ref{lm:decay1}.}\n\nSince $f_{-\\hat\\mu}$ is $\\delta\\mu$-strongly convex,\n\\begin{equation}\\label{eq:deltaconvexity}\nD_{f_{-\\hat\\mu}}(x^\\star,x)\n\\geq\n\\frac{\\mu-\\hat\\mu}{2}\\|x-x^\\star\\|^2\n=\n\\frac{\\delta\\mu}{2}\\|x-x^\\star\\|^2.\n\\end{equation}\nYoung's inequality and \\eqref{eq:deltaconvexity} give\n$$\n\\begin{aligned}\n\\delta\\mu\\langle y-x^\\star,x-x^\\star\\rangle\n&\\leq\n\\sqrt{\\delta}\\,\\frac{\\mu}{2}\\|y-x^\\star\\|^2\n+\n\\frac{\\delta^{3/2}\\mu}{2}\\|x-x^\\star\\|^2\n\\\\\n&\\leq\n\\sqrt{\\delta}\\,\\frac{\\mu}{2}\\|y-x^\\star\\|^2\n+\n\\sqrt{\\delta}\\,D_{f_{-\\hat\\mu}}(x^\\star,x).\n\\end{aligned}\n$$\nTherefore,\n$$\n\\begin{aligned}\n-\\big\\langle \\nabla \\mathcal E(\\bs z;\\hat\\mu),\\mathcal G(\\bs z)\\big\\rangle\n\\geq{}&\n\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n-\n\\sqrt{\\delta}\\,D_{f_{-\\hat\\mu}}(x^\\star,x)\n+\n\\beta\\|\\nabla f_{-\\hat\\mu}(x)\\|^2\n\\\\\n&\\quad\n+\n\\left(2-\\sqrt{\\delta}\\right)\\frac{\\mu}{2}\\|y-x^\\star\\|^2\n+\n\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle.\n\\end{aligned}\n$$\nBy the symmetrization identity \\eqref{eq:symBre} and the invariance of the Bregman asymmetry under quadratic shifts,\n$D_{f_{-\\hat\\mu}}(x^\\star,x)\n=\nD_{f_{-\\hat\\mu}}(x,x^\\star)\n+\n\\Delta_f(x,x^\\star).$\nHence\n$$\n\\begin{aligned}\n&\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n-\n\\sqrt{\\delta}\\,D_{f_{-\\hat\\mu}}(x^\\star,x)\n\\\\\n={}&\nD_{f_{-\\hat\\mu}}(x,x^\\star)\n+\n\\left(1-\\sqrt{\\delta}\\right)D_{f_{-\\hat\\mu}}(x^\\star,x)\n\\\\\n={}&\n\\left(2-\\sqrt{\\delta}\\right)D_{f_{-\\hat\\mu}}(x,x^\\star)\n+\n\\left(1-\\sqrt{\\delta}\\right)\\Delta_f(x,x^\\star).\n\\end{aligned}\n$$\nCombining the first term with $\\frac{\\mu}{2}\\|y-x^\\star\\|^2$ yields\n$$\n\\begin{aligned}\n-\\big\\langle \\nabla \\mathcal E(\\bs z;\\hat\\mu),\\mathcal G(\\bs z)\\big\\rangle\n\\geq{}&\n\\left(2-\\sqrt{\\delta}\\right)\\mathcal E(\\bs z;\\hat\\mu)\n+\n\\beta\\|\\nabla f_{-\\hat\\mu}(x)\\|^2\n\\\\\n&\\quad\n+\n\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n+\n\\left(1-\\sqrt{\\delta}\\right)\\Delta_f(x,x^\\star).\n\\end{aligned}\n$$\nFinally,\n$$\n\\left(2-\\sqrt{\\delta}\\right)\\mathcal E(\\bs z;\\hat\\mu)\n=\n\\left(2-\\sqrt{\\delta}-b\\right)\\mathcal E(\\bs z;\\hat\\mu)\n+\nb\\mathcal E(\\bs z;\\hat\\mu).\n$$\nSince $b\\geq0$, \n% and\n%$$\n%\\mathcal E(\\bs z;\\hat\\mu)\n%\\geq\n%D_{f_{-\\hat\\mu}}(x,x^\\star),\n%$$\nusing\n$$\nb\\mathcal E(\\bs z;\\hat\\mu)\n\\geq\nbD_{f_{-\\hat\\mu}}(x,x^\\star),\n$$\nwe obtain \\eqref{eq:energy_identity}.\n\\end{proof}","sha256_utf8":"e0df610491a9d130e67d1f698b57ce4ddf632bf04c944ec8f19495812e298b67"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"计算部分移位能量导数","source_units":[{"start":73585,"end":74413,"start_line":1896,"end_line":1924,"raw_tex":"\\begin{proof}\nA direct calculation gives\n$$\n\\begin{aligned}\n-\\big\\langle \\nabla \\mathcal E(\\bs z;\\hat\\mu),\\mathcal G(\\bs z)\\big\\rangle\n={}&\n\\left\\langle\n\\begin{pmatrix}\n\\nabla f_{-\\hat\\mu}(x)\\\\[2pt]\n\\mu(y-x^\\star)\n\\end{pmatrix},\n\\begin{pmatrix}\nx-y+\\beta\\nabla f_{-\\hat\\mu}(x)+\\beta\\hat\\mu(x-x^\\star)\\\\\ny-x^\\star+\\dfrac1\\mu\\nabla f_{-\\hat\\mu}(x)-\\delta(x-x^\\star)\n\\end{pmatrix}\n\\right\\rangle\n\\\\\n={}&\n\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n+\\mu\\|y-x^\\star\\|^2\n-\\delta\\mu\\langle y-x^\\star,x-x^\\star\\rangle\n\\\\\n&\\quad\n+\\beta\\|\\nabla f_{-\\hat\\mu}(x)\\|^2\n+\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle.\n\\end{aligned}\n$$\nThis proves the identity. \\RV{We keep the term $\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle$ separate, since it controls the $\\hat \\mu$-shift term in Lemma~\\ref{lm:decay1}.}\n\n","sha256_utf8":"36c7df894f9420e1b2da8b5a93d1c39f908fb64c58482fe3435679f8cc8645de"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"以剩余强凸性控制混合项","source_units":[{"start":74413,"end":74957,"start_line":1925,"end_line":1947,"raw_tex":"Since $f_{-\\hat\\mu}$ is $\\delta\\mu$-strongly convex,\n\\begin{equation}\\label{eq:deltaconvexity}\nD_{f_{-\\hat\\mu}}(x^\\star,x)\n\\geq\n\\frac{\\mu-\\hat\\mu}{2}\\|x-x^\\star\\|^2\n=\n\\frac{\\delta\\mu}{2}\\|x-x^\\star\\|^2.\n\\end{equation}\nYoung's inequality and \\eqref{eq:deltaconvexity} give\n$$\n\\begin{aligned}\n\\delta\\mu\\langle y-x^\\star,x-x^\\star\\rangle\n&\\leq\n\\sqrt{\\delta}\\,\\frac{\\mu}{2}\\|y-x^\\star\\|^2\n+\n\\frac{\\delta^{3/2}\\mu}{2}\\|x-x^\\star\\|^2\n\\\\\n&\\leq\n\\sqrt{\\delta}\\,\\frac{\\mu}{2}\\|y-x^\\star\\|^2\n+\n\\sqrt{\\delta}\\,D_{f_{-\\hat\\mu}}(x^\\star,x).\n\\end{aligned}\n$$\n","sha256_utf8":"6b921ee7aaa343934ed05e382f81b3e6482b16dc0ff8ebc84ffeaee772587ca3"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"代入混合项估计","source_units":[{"start":74957,"end":75348,"start_line":1948,"end_line":1965,"raw_tex":"Therefore,\n$$\n\\begin{aligned}\n-\\big\\langle \\nabla \\mathcal E(\\bs z;\\hat\\mu),\\mathcal G(\\bs z)\\big\\rangle\n\\geq{}&\n\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n-\n\\sqrt{\\delta}\\,D_{f_{-\\hat\\mu}}(x^\\star,x)\n+\n\\beta\\|\\nabla f_{-\\hat\\mu}(x)\\|^2\n\\\\\n&\\quad\n+\n\\left(2-\\sqrt{\\delta}\\right)\\frac{\\mu}{2}\\|y-x^\\star\\|^2\n+\n\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle.\n\\end{aligned}\n$$\n","sha256_utf8":"c3f7d1b52d65837ad3b6b175cbafe47033923a1813967e049fb25ee03dd931fe"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"利用移位不改变非对称量","source_units":[{"start":75348,"end":75893,"start_line":1966,"end_line":1989,"raw_tex":"By the symmetrization identity \\eqref{eq:symBre} and the invariance of the Bregman asymmetry under quadratic shifts,\n$D_{f_{-\\hat\\mu}}(x^\\star,x)\n=\nD_{f_{-\\hat\\mu}}(x,x^\\star)\n+\n\\Delta_f(x,x^\\star).$\nHence\n$$\n\\begin{aligned}\n&\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n-\n\\sqrt{\\delta}\\,D_{f_{-\\hat\\mu}}(x^\\star,x)\n\\\\\n={}&\nD_{f_{-\\hat\\mu}}(x,x^\\star)\n+\n\\left(1-\\sqrt{\\delta}\\right)D_{f_{-\\hat\\mu}}(x^\\star,x)\n\\\\\n={}&\n\\left(2-\\sqrt{\\delta}\\right)D_{f_{-\\hat\\mu}}(x,x^\\star)\n+\n\\left(1-\\sqrt{\\delta}\\right)\\Delta_f(x,x^\\star).\n\\end{aligned}\n$$\n","sha256_utf8":"cfd116b6881b970e69b52d7eadbd63e1944c33a575da0af856720a03e4482a46"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"合并得到增强耗散系数","source_units":[{"start":75893,"end":76293,"start_line":1990,"end_line":2005,"raw_tex":"Combining the first term with $\\frac{\\mu}{2}\\|y-x^\\star\\|^2$ yields\n$$\n\\begin{aligned}\n-\\big\\langle \\nabla \\mathcal E(\\bs z;\\hat\\mu),\\mathcal G(\\bs z)\\big\\rangle\n\\geq{}&\n\\left(2-\\sqrt{\\delta}\\right)\\mathcal E(\\bs z;\\hat\\mu)\n+\n\\beta\\|\\nabla f_{-\\hat\\mu}(x)\\|^2\n\\\\\n&\\quad\n+\n\\beta\\hat\\mu\\langle \\nabla f_{-\\hat\\mu}(x),x-x^\\star\\rangle\n+\n\\left(1-\\sqrt{\\delta}\\right)\\Delta_f(x,x^\\star).\n\\end{aligned}\n$$\n","sha256_utf8":"f31edbb99a3d9ca2b8ec77958060c50f86b0c89f0c6f6a90f339cf3b3b6701b9"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"分出用于吸收扰动的能量","source_units":[{"start":76293,"end":76667,"start_line":2006,"end_line":2028,"raw_tex":"Finally,\n$$\n\\left(2-\\sqrt{\\delta}\\right)\\mathcal E(\\bs z;\\hat\\mu)\n=\n\\left(2-\\sqrt{\\delta}-b\\right)\\mathcal E(\\bs z;\\hat\\mu)\n+\nb\\mathcal E(\\bs z;\\hat\\mu).\n$$\nSince $b\\geq0$, \n% and\n%$$\n%\\mathcal E(\\bs z;\\hat\\mu)\n%\\geq\n%D_{f_{-\\hat\\mu}}(x,x^\\star),\n%$$\nusing\n$$\nb\\mathcal E(\\bs z;\\hat\\mu)\n\\geq\nbD_{f_{-\\hat\\mu}}(x,x^\\star),\n$$\nwe obtain \\eqref{eq:energy_identity}.\n\\end{proof}","sha256_utf8":"2c5833a01136a895452bc922288a43c9c747093196f00c76f284c990b5c3a70e"}]},{"id":"S052","semantic_type":"lemma","kind":"lemma","title":"部分移位单步扰动界","source_units":[{"start":76925,"end":77620,"start_line":2032,"end_line":2058,"raw_tex":"\\begin{lemma}\\label{lm:decay1}\nLet $0\\leq\\hat\\mu\\leq\\mu$, and let $(x_k,y_k)$ be generated by the HNAG scheme~\\eqref{eq:HNAG} with\n$\\alpha=\\sqrt{2\\mu/L}$ and $\\alpha\\beta=1/L$.\nThen, for all $k\\geq0$,\n\\begin{equation}\\label{eq:decay1}\n\\begin{aligned}\n(1+c_L\\alpha)\\mathcal E(\\bs z_{k+1};\\hat\\mu)\n\\leq{}&\n\\mathcal E(\\bs z_k;\\hat\\mu)\n+\\frac1{2L}\\|\\nabla f_{-\\hat\\mu}(x_{k+1})\\|^2\n-\\frac1{2L}\\|\\nabla f_{-\\hat\\mu}(x_k)\\|^2\n\\\\\n&\\quad\n-\\alpha\\left[\nd_\\delta\\Delta_f(x_{k+1},x^\\star)\n+\\RV{bD_{f_{-\\hat\\mu}}(x_{k+1},x^\\star)}\n\\right ],\n\\end{aligned}\n\\end{equation}\nwhere\n$$\nb\\geq0,\\quad\n\\delta=\\frac{\\mu-\\hat\\mu}{\\mu}\\in[0,1],\\quad\nc_L=2-\\sqrt{\\delta}-b>0,\\quad\nd_\\delta=1-\\sqrt{\\delta}.\n$$\n\\end{lemma}","sha256_utf8":"96abee9f9a2cf7c59ad1f7491cc94d13e73f8b0ed85eda1d5c33f802a99560d5"}]},{"id":"S053","semantic_type":"proof","kind":"proof","title":"部分移位单步界证明","source_units":[{"start":77622,"end":81232,"start_line":2060,"end_line":2218,"raw_tex":"\\begin{proof}\nUsing the Bregman expansion,\n$$\n\\mathcal E(\\bs z_{k+1};\\hat\\mu)-\\mathcal E(\\bs z_k;\\hat\\mu)\n=\n\\big\\langle \\nabla \\mathcal E(\\bs z_{k+1};\\hat\\mu),\\bs z_{k+1}-\\bs z_k\\big\\rangle\n-\nD_{\\mathcal E(\\cdot;\\hat\\mu)}(\\bs z_k,\\bs z_{k+1}).\n$$\nLet $g_k:=\\nabla f_{-\\hat\\mu}(x_k)$. \\RV{Since $-\\beta\\nabla f(x_{k+1})$ is used in $\\mathcal G(\\bs z_{k+1})$ and\n$$\n\\beta(g_{k+1}-g_k)+\\beta\\hat\\mu(x_{k+1}-x_k)\n=\n\\beta\\bigl(\\nabla f(x_{k+1})-\\nabla f(x_k)\\bigr),\n$$}\nthe update can be written as\n$$\n\\bs z_{k+1}-\\bs z_k\n=\n\\alpha\\mathcal G(\\bs z_{k+1})\n+\n\\alpha\n\\binom{\n(y_k-y_{k+1})+\\beta(g_{k+1}-g_k)+\\beta\\hat\\mu(x_{k+1}-x_k)\n}{0}.\n$$\n\nThe implicit Euler part is controlled by Lemma~\\ref{lem:energy_identity}:\n\\begin{equation}\\label{eq:IE-HNAGpp}\n\\begin{aligned}\n&\\alpha\\big\\langle \\nabla \\mathcal E(\\bs z_{k+1};\\hat\\mu),\\mathcal G(\\bs z_{k+1})\\big\\rangle \\\\\n\\leq{}&\n-c_L\\alpha\\mathcal E(\\bs z_{k+1};\\hat\\mu)\n-\\alpha\\beta\\hat\\mu\\langle g_{k+1},x_{k+1}-x^\\star\\rangle\n\\\\\n&\\quad\n-\\alpha\\beta\\|g_{k+1}\\|^2\n-\\alpha\\left [\nd_\\delta\\Delta_f(x_{k+1},x^\\star)\n+\\RV{bD_{f_{-\\hat\\mu}}(x_{k+1},x^\\star)}\n\\right ].\n\\end{aligned}\n\\end{equation}\n\n\\medskip\n\\noindent\n\\textbf{The $\\hat\\mu$-shift term.}\nBy the three-point Bregman identity~\\eqref{eq:Bregmanidentity},\n$$\n\\begin{aligned}\n&\\alpha\\beta\\hat\\mu\\langle g_{k+1},x_{k+1}-x_k\\rangle\n\\\\\n={}&\n\\alpha\\beta\\hat\\mu\\Bigl(\nD_{f_{-\\hat\\mu}}(x_{k+1},x^\\star)\n+\nD_{f_{-\\hat\\mu}}(x_k,x_{k+1})\n-\nD_{f_{-\\hat\\mu}}(x_k,x^\\star)\n\\Bigr).\n\\end{aligned}\n$$\n%Using\n%$$\n%\\langle g_{k+1},x_{k+1}-x^\\star\\rangle\n%=\n%D_{f_{-\\hat\\mu}}(x_{k+1},x^\\star)\n%+\n%D_{f_{-\\hat\\mu}}(x^\\star,x_{k+1}),\n%$$\nThe first term cancels the corresponding part of the shift term in \\eqref{eq:IE-HNAGpp}, leaving a nonpositive term, which we may discard.\n%$$\n%-\\alpha\\beta\\hat\\mu D_{f_{-\\hat\\mu}}(x^\\star,x_{k+1})\\leq0.\n%$$\nThe second term combines with\n$-D_{\\mathcal E(\\cdot;\\hat\\mu)}(\\bs z_k,\\bs z_{k+1})$ to give\n$$\n-(1-\\alpha\\beta\\hat\\mu)D_{f_{-\\hat\\mu}}(x_k,x_{k+1}).\n$$\nThe last term is also nonpositive and may be discarded.\n\n\\medskip\n\\noindent\n\\textbf{Lagging of $y$.}\nBy Cauchy--Schwarz and Young's inequality,\n$$\n\\alpha\\langle g_{k+1},y_k-y_{k+1}\\rangle\n\\leq\n\\frac{\\alpha^2}{2\\mu}\\|g_{k+1}\\|^2\n+\n\\frac{\\mu}{2}\\|y_k-y_{k+1}\\|^2.\n$$\nThe second term cancels the $y$-component of\n$-D_{\\mathcal E(\\cdot;\\hat\\mu)}(\\bs z_k,\\bs z_{k+1})$.\n\n\\medskip\n\\noindent\n\\textbf{Lagging of the gradient.}\nThe identity of squares gives\n$$\n\\begin{aligned}\n\\alpha\\beta\\langle g_{k+1},g_{k+1}-g_k\\rangle\n={}&\n-\\frac{\\alpha\\beta}{2}\\|g_k\\|^2\n+\\frac{\\alpha\\beta}{2}\\|g_{k+1}\\|^2\n+\\frac{\\alpha\\beta}{2}\\|g_{k+1}-g_k\\|^2.\n\\end{aligned}\n$$\nSince $f_{-\\hat\\mu}$ is $(L-\\hat\\mu)$-smooth and $\\alpha\\beta=1/L$,\n$$\n\\frac{\\alpha\\beta}{2}\\|g_{k+1}-g_k\\|^2\n\\leq\n(1-\\alpha\\beta\\hat\\mu)D_{f_{-\\hat\\mu}}(x_k,x_{k+1}).\n$$\nTherefore,\n$$\n\\begin{aligned}\n&\\alpha\\beta\\langle g_{k+1},g_{k+1}-g_k\\rangle\n-\n(1-\\alpha\\beta\\hat\\mu)D_{f_{-\\hat\\mu}}(x_k,x_{k+1})\n\\\\\n\\leq{}&\n-\\frac{\\alpha\\beta}{2}\\|g_k\\|^2\n+\n\\frac{\\alpha\\beta}{2}\\|g_{k+1}\\|^2.\n\\end{aligned}\n$$\n\nCombining the above estimates gives\n$$\n\\begin{aligned}\n(1+c_L\\alpha)\\mathcal E(\\bs z_{k+1};\\hat\\mu)\n\\leq{}&\n\\mathcal E(\\bs z_k;\\hat\\mu)\n+\n\\left(\n\\frac{\\alpha^2}{2\\mu}\n-\\frac{\\alpha\\beta}{2}\n\\right)\\|g_{k+1}\\|^2\n-\n\\frac{\\alpha\\beta}{2}\\|g_k\\|^2\n\\\\\n&\\quad\n-\\alpha bD_{f_{-\\hat\\mu}}(x_{k+1},x^\\star)\n-\\alpha d_\\delta\\Delta_f(x_{k+1},x^\\star).\n\\end{aligned}\n$$\nFinally, using $\\alpha^2=2\\mu/L$ and $\\alpha\\beta=1/L$, we have\n%$$\n%\\frac{\\alpha^2}{2\\mu}-\\frac{\\alpha\\beta}{2}\n%=\n%\\frac1{2L},\n%\\qquad\n%\\frac{\\alpha\\beta}{2}\n%=\n%\\frac1{2L}.\n%$$\nthe desired result \\eqref{eq:decay1}. $\\Box$\n\\end{proof}","sha256_utf8":"28eb2cedadcb274211c364393f005f6759834571795b496bbb76a308323e0cbb"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"展开能量并分解更新修正","source_units":[{"start":77622,"end":78257,"start_line":2060,"end_line":2086,"raw_tex":"\\begin{proof}\nUsing the Bregman expansion,\n$$\n\\mathcal E(\\bs z_{k+1};\\hat\\mu)-\\mathcal E(\\bs z_k;\\hat\\mu)\n=\n\\big\\langle \\nabla \\mathcal E(\\bs z_{k+1};\\hat\\mu),\\bs z_{k+1}-\\bs z_k\\big\\rangle\n-\nD_{\\mathcal E(\\cdot;\\hat\\mu)}(\\bs z_k,\\bs z_{k+1}).\n$$\nLet $g_k:=\\nabla f_{-\\hat\\mu}(x_k)$. \\RV{Since $-\\beta\\nabla f(x_{k+1})$ is used in $\\mathcal G(\\bs z_{k+1})$ and\n$$\n\\beta(g_{k+1}-g_k)+\\beta\\hat\\mu(x_{k+1}-x_k)\n=\n\\beta\\bigl(\\nabla f(x_{k+1})-\\nabla f(x_k)\\bigr),\n$$}\nthe update can be written as\n$$\n\\bs z_{k+1}-\\bs z_k\n=\n\\alpha\\mathcal G(\\bs z_{k+1})\n+\n\\alpha\n\\binom{\n(y_k-y_{k+1})+\\beta(g_{k+1}-g_k)+\\beta\\hat\\mu(x_{k+1}-x_k)\n}{0}.\n$$\n\n","sha256_utf8":"57cb06aa1d8794749144eb0ecaf1b86cc5c688d73fb6f6242f4cc40750d8db9e"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"应用部分移位强耗散界","source_units":[{"start":78257,"end":78754,"start_line":2087,"end_line":2103,"raw_tex":"The implicit Euler part is controlled by Lemma~\\ref{lem:energy_identity}:\n\\begin{equation}\\label{eq:IE-HNAGpp}\n\\begin{aligned}\n&\\alpha\\big\\langle \\nabla \\mathcal E(\\bs z_{k+1};\\hat\\mu),\\mathcal G(\\bs z_{k+1})\\big\\rangle \\\\\n\\leq{}&\n-c_L\\alpha\\mathcal E(\\bs z_{k+1};\\hat\\mu)\n-\\alpha\\beta\\hat\\mu\\langle g_{k+1},x_{k+1}-x^\\star\\rangle\n\\\\\n&\\quad\n-\\alpha\\beta\\|g_{k+1}\\|^2\n-\\alpha\\left [\nd_\\delta\\Delta_f(x_{k+1},x^\\star)\n+\\RV{bD_{f_{-\\hat\\mu}}(x_{k+1},x^\\star)}\n\\right ].\n\\end{aligned}\n\\end{equation}\n\n","sha256_utf8":"7ce5fcabd9573f3816c699e75e7b3957bbb14eb5a0ea0a757e3c2a8cc5ade538"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"消去原始移位修正项","source_units":[{"start":78754,"end":79650,"start_line":2104,"end_line":2140,"raw_tex":"\\medskip\n\\noindent\n\\textbf{The $\\hat\\mu$-shift term.}\nBy the three-point Bregman identity~\\eqref{eq:Bregmanidentity},\n$$\n\\begin{aligned}\n&\\alpha\\beta\\hat\\mu\\langle g_{k+1},x_{k+1}-x_k\\rangle\n\\\\\n={}&\n\\alpha\\beta\\hat\\mu\\Bigl(\nD_{f_{-\\hat\\mu}}(x_{k+1},x^\\star)\n+\nD_{f_{-\\hat\\mu}}(x_k,x_{k+1})\n-\nD_{f_{-\\hat\\mu}}(x_k,x^\\star)\n\\Bigr).\n\\end{aligned}\n$$\n%Using\n%$$\n%\\langle g_{k+1},x_{k+1}-x^\\star\\rangle\n%=\n%D_{f_{-\\hat\\mu}}(x_{k+1},x^\\star)\n%+\n%D_{f_{-\\hat\\mu}}(x^\\star,x_{k+1}),\n%$$\nThe first term cancels the corresponding part of the shift term in \\eqref{eq:IE-HNAGpp}, leaving a nonpositive term, which we may discard.\n%$$\n%-\\alpha\\beta\\hat\\mu D_{f_{-\\hat\\mu}}(x^\\star,x_{k+1})\\leq0.\n%$$\nThe second term combines with\n$-D_{\\mathcal E(\\cdot;\\hat\\mu)}(\\bs z_k,\\bs z_{k+1})$ to give\n$$\n-(1-\\alpha\\beta\\hat\\mu)D_{f_{-\\hat\\mu}}(x_k,x_{k+1}).\n$$\nThe last term is also nonpositive and may be discarded.\n\n","sha256_utf8":"84e1ac4fdebf7b3c571bedf50429ba43fbd76b08d39b2c748d8242d9033b9a3a"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"抵消辅助变量滞后误差","source_units":[{"start":79650,"end":79959,"start_line":2141,"end_line":2154,"raw_tex":"\\medskip\n\\noindent\n\\textbf{Lagging of $y$.}\nBy Cauchy--Schwarz and Young's inequality,\n$$\n\\alpha\\langle g_{k+1},y_k-y_{k+1}\\rangle\n\\leq\n\\frac{\\alpha^2}{2\\mu}\\|g_{k+1}\\|^2\n+\n\\frac{\\mu}{2}\\|y_k-y_{k+1}\\|^2.\n$$\nThe second term cancels the $y$-component of\n$-D_{\\mathcal E(\\cdot;\\hat\\mu)}(\\bs z_k,\\bs z_{k+1})$.\n\n","sha256_utf8":"9e1275ad9f47ba1e7e74e26a1b58ee49ea847c4ae1c9b5506d84448807eae265"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"用光滑性抵消梯度差项","source_units":[{"start":79959,"end":80639,"start_line":2155,"end_line":2187,"raw_tex":"\\medskip\n\\noindent\n\\textbf{Lagging of the gradient.}\nThe identity of squares gives\n$$\n\\begin{aligned}\n\\alpha\\beta\\langle g_{k+1},g_{k+1}-g_k\\rangle\n={}&\n-\\frac{\\alpha\\beta}{2}\\|g_k\\|^2\n+\\frac{\\alpha\\beta}{2}\\|g_{k+1}\\|^2\n+\\frac{\\alpha\\beta}{2}\\|g_{k+1}-g_k\\|^2.\n\\end{aligned}\n$$\nSince $f_{-\\hat\\mu}$ is $(L-\\hat\\mu)$-smooth and $\\alpha\\beta=1/L$,\n$$\n\\frac{\\alpha\\beta}{2}\\|g_{k+1}-g_k\\|^2\n\\leq\n(1-\\alpha\\beta\\hat\\mu)D_{f_{-\\hat\\mu}}(x_k,x_{k+1}).\n$$\nTherefore,\n$$\n\\begin{aligned}\n&\\alpha\\beta\\langle g_{k+1},g_{k+1}-g_k\\rangle\n-\n(1-\\alpha\\beta\\hat\\mu)D_{f_{-\\hat\\mu}}(x_k,x_{k+1})\n\\\\\n\\leq{}&\n-\\frac{\\alpha\\beta}{2}\\|g_k\\|^2\n+\n\\frac{\\alpha\\beta}{2}\\|g_{k+1}\\|^2.\n\\end{aligned}\n$$\n\n","sha256_utf8":"67460a412797b992bbf2528a7d51211a089a77e2b9c0031195bfd20ee7ef7bcd"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"合并并代入大步长参数","source_units":[{"start":80639,"end":81232,"start_line":2188,"end_line":2218,"raw_tex":"Combining the above estimates gives\n$$\n\\begin{aligned}\n(1+c_L\\alpha)\\mathcal E(\\bs z_{k+1};\\hat\\mu)\n\\leq{}&\n\\mathcal E(\\bs z_k;\\hat\\mu)\n+\n\\left(\n\\frac{\\alpha^2}{2\\mu}\n-\\frac{\\alpha\\beta}{2}\n\\right)\\|g_{k+1}\\|^2\n-\n\\frac{\\alpha\\beta}{2}\\|g_k\\|^2\n\\\\\n&\\quad\n-\\alpha bD_{f_{-\\hat\\mu}}(x_{k+1},x^\\star)\n-\\alpha d_\\delta\\Delta_f(x_{k+1},x^\\star).\n\\end{aligned}\n$$\nFinally, using $\\alpha^2=2\\mu/L$ and $\\alpha\\beta=1/L$, we have\n%$$\n%\\frac{\\alpha^2}{2\\mu}-\\frac{\\alpha\\beta}{2}\n%=\n%\\frac1{2L},\n%\\qquad\n%\\frac{\\alpha\\beta}{2}\n%=\n%\\frac1{2L}.\n%$$\nthe desired result \\eqref{eq:decay1}. $\\Box$\n\\end{proof}","sha256_utf8":"b91a7506631fde1fb257a13db2dc1fbd496493f0bd85159e22f328751f7bc576"}]},{"id":"S060","semantic_type":"lemma","kind":"lemma","title":"变化移位的梯度范数比较","source_units":[{"start":93084,"end":93302,"start_line":2630,"end_line":2640,"raw_tex":"\\begin{lemma}\\label{lm:gradient_norm_ineq}\nFor all \\(k\\ge 0\\),\n$$\n\\|\\nabla f_k(x_{k+1})\\|^2\n\\le\n\\|\\nabla f_{k+1}(x_{k+1})\\|^2\n+\nL\\Delta\\delta_k\\alpha^2\n\\langle \\nabla f_k(x_{k+1}),x_{k+1}-x^\\star\\rangle.\n$$\n\\end{lemma}","sha256_utf8":"33078542c639aa89ae65d408ad459825feed31762d84c67af6b08543a640158a"}]},{"id":"S061","semantic_type":"proof","kind":"proof","title":"梯度范数比较证明","source_units":[{"start":93303,"end":93896,"start_line":2641,"end_line":2669,"raw_tex":"\\begin{proof}\nBy definition,\n$$\n\\nabla f_{k+1}(x_{k+1})\n=\n\\nabla f_k(x_{k+1})-(\\mu_{k+1}-\\mu_k)(x_{k+1}-x^\\star).\n$$\nExpanding the square and dropping the positive last term, we obtain\n$$\n\\begin{aligned}\n\\|\\nabla f_{k+1}(x_{k+1})\\|^2\n&=\n\\bigl\\|\\nabla f_k(x_{k+1})-(\\mu_{k+1}-\\mu_k)(x_{k+1}-x^\\star)\\bigr\\|^2\\\\\n&\\ge\n\\|\\nabla f_k(x_{k+1})\\|^2\n-\n2(\\mu_{k+1}-\\mu_k)\n\\langle \\nabla f_k(x_{k+1}),x_{k+1}-x^\\star\\rangle.\n\\end{aligned}\n$$\nSince $\\mu_{k+1}-\\mu_k=\\Delta\\delta_k\\,\\mu, \n\\alpha^2=2\\mu/L,$\nwe have\n$$\n2(\\mu_{k+1}-\\mu_k)=L\\Delta\\delta_k\\alpha^2.\n$$\nThis proves the claim.\n$\\Box$\n\\end{proof}","sha256_utf8":"a6a9d880bda214bbe775bb71e1e1e88fea3cc575f1fdc9e225f591276529d746"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"写出相邻移位梯度关系","source_units":[{"start":93303,"end":93420,"start_line":2641,"end_line":2647,"raw_tex":"\\begin{proof}\nBy definition,\n$$\n\\nabla f_{k+1}(x_{k+1})\n=\n\\nabla f_k(x_{k+1})-(\\mu_{k+1}-\\mu_k)(x_{k+1}-x^\\star).\n$$\n","sha256_utf8":"43337446a241c6529ea3b962ae642e9fa8033a5c48f6ced24b278a3aff1d466b"}]},{"id":"S063","semantic_type":"proof_step","kind":"proof_step","title":"展开平方并舍去正项","source_units":[{"start":93420,"end":93734,"start_line":2648,"end_line":2660,"raw_tex":"Expanding the square and dropping the positive last term, we obtain\n$$\n\\begin{aligned}\n\\|\\nabla f_{k+1}(x_{k+1})\\|^2\n&=\n\\bigl\\|\\nabla f_k(x_{k+1})-(\\mu_{k+1}-\\mu_k)(x_{k+1}-x^\\star)\\bigr\\|^2\\\\\n&\\ge\n\\|\\nabla f_k(x_{k+1})\\|^2\n-\n2(\\mu_{k+1}-\\mu_k)\n\\langle \\nabla f_k(x_{k+1}),x_{k+1}-x^\\star\\rangle.\n\\end{aligned}\n$$\n","sha256_utf8":"5916439d7816e59de97c786566f342157613fd0228fdb67ea48341ba6e0f7770"}]},{"id":"S064","semantic_type":"proof_step","kind":"proof_step","title":"换成移位差分与步长系数","source_units":[{"start":93734,"end":93896,"start_line":2661,"end_line":2669,"raw_tex":"Since $\\mu_{k+1}-\\mu_k=\\Delta\\delta_k\\,\\mu, \n\\alpha^2=2\\mu/L,$\nwe have\n$$\n2(\\mu_{k+1}-\\mu_k)=L\\Delta\\delta_k\\alpha^2.\n$$\nThis proves the claim.\n$\\Box$\n\\end{proof}","sha256_utf8":"598135253cf7e8de93abbd7cf4c7dc27e387c57945137d4c9d1e028bc76c4a50"}]},{"id":"S065","semantic_type":"lemma","kind":"lemma","title":"变化移位能量收缩递推","source_units":[{"start":94982,"end":96202,"start_line":2697,"end_line":2746,"raw_tex":"\\begin{lemma}\\label{lm:shifted_energy_recursion}\nAssume $f\\in\\mathcal S_{\\mu,L}^{\\rm LAS}(x^\\star)$. Let $(x_k,y_k)$ be generated by Algorithm~\\ref{alg:HNAGpp}. Let $\\{\\delta_k\\}\\subset[0,1]$ be nonincreasing with $\\delta_k\\to0$, and let $\\{b_k\\}$ be nonnegative with $b_k\\to0$. Set\n$$\n\\mu_k=(1-\\delta_k)\\mu,\\quad\nf_k:=f_{-\\mu_k},\\quad\nd_k:=1-\\sqrt{\\delta_k},\\quad\n\\Delta\\delta_k:=\\delta_k-\\delta_{k+1}\\geq0,\n$$\nand define\n$$\n\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k)\n:=\n\\mathcal E(\\bs z_k;\\mu_k)\n-\\frac1{2L}\\|\\nabla f_k(x_k)\\|^2\\geq 0.\n$$\n%\\RV{Since $f_k$ is convex and $L$-smooth, co-coercivity gives\n%$$\n%D_{f_k}(x_k,x^\\star)\n%\\geq\n%\\frac1{2L}\\|\\nabla f_k(x_k)\\|^2.\n%$$\n%Consequently,\n%$$\n%\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k)\n%\\geq\n%\\frac{\\mu}{2}\\|y_k-x^\\star\\|^2\n%\\geq0.\n%$$}\nLet $\\omega_k$ be defined by \\eqref{eq:def-omegak}, and set\n$$\nc_k:=2-\\sqrt{\\delta_k}-b_k\\geq0,\n\\qquad\nr_k:=\n\\left(1+c_k\\alpha-\\Delta\\delta_k\\alpha^2\\right)^{-1}.\n$$\nAssume $r_k^{-1}\\geq1$ and\n\\begin{equation}\\label{eq:omega_condition}\n\\left|\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\right|\\omega_k\n\\leq\nb_k\\delta_k.\n\\end{equation}\nThen\n$$\n\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\nr_k\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k).\n$$\n\\end{lemma}","sha256_utf8":"7d63ab769438fa58b59177d5e237699da5736da55af4a3ae5f339c4b836a6e9d"}]},{"id":"S066","semantic_type":"proof","kind":"proof","title":"变化移位收缩递推证明","source_units":[{"start":96204,"end":99244,"start_line":2748,"end_line":2891,"raw_tex":"\\begin{proof}\nApply Lemma~\\ref{lm:decay1} with $\\hat\\mu=\\mu_k$. Then\n$$\n\\begin{aligned}\n(1+c_k\\alpha)\\mathcal E(\\bs z_{k+1};\\mu_k)\n\\leq{}&\n\\mathcal E(\\bs z_k;\\mu_k)\n+\\frac1{2L}\\|\\nabla f_k(x_{k+1})\\|^2\n-\\frac1{2L}\\|\\nabla f_k(x_k)\\|^2\n\\\\\n&\\quad\n-\\alpha\\left(\nd_k\\Delta_f(x_{k+1},x^\\star)\n+\nb_kD_{f_k}(x_{k+1},x^\\star)\n\\right).\n\\end{aligned}\n$$\nWe first change the shifted gradient using Lemma~\\ref{lm:gradient_norm_ineq}:\n$$\n\\frac1{2L}\\|\\nabla f_k(x_{k+1})\\|^2\n\\leq\n\\frac1{2L}\\|\\nabla f_{k+1}(x_{k+1})\\|^2\n+\n\\frac{\\Delta\\delta_k\\alpha^2}{2}\n\\langle\\nabla f_k(x_{k+1}),x_{k+1}-x^\\star\\rangle.\n$$\nUsing\n$$\n\\begin{aligned}\n\\frac{\\Delta\\delta_k\\alpha^2}{2}\n\\langle\\nabla f_k(x_{k+1}),x_{k+1}-x^\\star\\rangle\n={}&\n\\Delta\\delta_k\\alpha^2D_{f_k}(x_{k+1},x^\\star)\n+\\frac{\\Delta\\delta_k\\alpha^2}{2}\n\\Delta_f(x_{k+1},x^\\star),\n\\end{aligned}\n$$\nand\n$$\n\\Delta\\delta_k\\alpha^2D_{f_k}(x_{k+1},x^\\star)\n\\leq\n\\Delta\\delta_k\\alpha^2\\mathcal E(\\bs z_{k+1};\\mu_k),\n$$\nthe first term can be absorbed into the left-hand side, which slightly weakens the contraction factor from $1+c_k\\alpha$ to $r_k$. \nWe then obtain\n$$\n\\begin{aligned}\nr_k^{-1}\\mathcal E(\\bs z_{k+1};\\mu_k)\n\\leq{}&\n\\mathcal E(\\bs z_k;\\mu_k)\n+\\frac1{2L}\\|\\nabla f_{k+1}(x_{k+1})\\|^2\n-\\frac1{2L}\\|\\nabla f_k(x_k)\\|^2\n\\\\\n&\\quad\n-\\alpha\\left[\n\\left(\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\right)\\Delta_f(x_{k+1},x^\\star)\n+\nb_kD_{f_k}(x_{k+1},x^\\star)\n\\right].\n\\end{aligned}\n$$\nSince $\\mu_{k+1}\\geq\\mu_k$,\n$$\n\\mathcal E(\\bs z_{k+1};\\mu_{k+1})\n\\leq\n\\mathcal E(\\bs z_{k+1};\\mu_k).\n$$\nThus the same inequality holds with $\\mathcal E(\\bs z_{k+1};\\mu_{k+1})$ on the left-hand side.\n\nBy the definition of $\\omega_k$,\n$$\n|\\Delta_f(x_{k+1},x^\\star)|\n\\leq\n\\omega_kD_f(x_{k+1},x^\\star).\n$$\nMoreover, since $\\mu_k=(1-\\delta_k)\\mu$ and $f$ is $\\mu$-strongly convex,\n$$\n\\begin{aligned}\nD_{f_k}(x_{k+1},x^\\star)\n&=\nD_f(x_{k+1},x^\\star)\n-\\frac{\\mu_k}{2}\\|x_{k+1}-x^\\star\\|^2\n\\\\\n&\\geq\nD_f(x_{k+1},x^\\star)\n-\\frac{\\mu_k}{\\mu}D_f(x_{k+1},x^\\star)\n=\n\\delta_kD_f(x_{k+1},x^\\star).\n\\end{aligned}\n$$\nTherefore,\n$$\n\\begin{aligned}\n&\n\\left(\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\right)\\Delta_f(x_{k+1},x^\\star)\n+\nb_kD_{f_k}(x_{k+1},x^\\star)\n\\\\\n&\\geq\n\\left[\nb_k\\delta_k\n-\n\\left|\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\right|\\omega_k\n\\right]\nD_f(x_{k+1},x^\\star)\n\\geq0,\n\\end{aligned}\n$$\nwhere the last inequality follows from \\eqref{eq:omega_condition}. It follows that\n$$\n\\begin{aligned}\nr_k^{-1}\\mathcal E(\\bs z_{k+1};\\mu_{k+1})\n\\leq{}&\n\\mathcal E(\\bs z_k;\\mu_k)\n+\\frac1{2L}\\|\\nabla f_{k+1}(x_{k+1})\\|^2\n-\\frac1{2L}\\|\\nabla f_k(x_k)\\|^2.\n\\end{aligned}\n$$\nSubtracting $r_k^{-1}(2L)^{-1}\\|\\nabla f_{k+1}(x_{k+1})\\|^2$ from both sides gives\n$$\n\\begin{aligned}\nr_k^{-1}\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq{}&\n\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k)\n+\n\\frac{1-r_k^{-1}}{2L}\n\\|\\nabla f_{k+1}(x_{k+1})\\|^2.\n\\end{aligned}\n$$\nSince $r_k^{-1}\\geq1$, the last term is nonpositive. Therefore,\n$$\nr_k^{-1}\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\n\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k),\n$$\nwhich proves the claim.\n\\end{proof}","sha256_utf8":"b31e64de243c573240a803939e77aa531e627b7848ae604ed489abf87312a247"}]},{"id":"S067","semantic_type":"proof_step","kind":"proof_step","title":"应用固定部分移位单步界","source_units":[{"start":96204,"end":96548,"start_line":2748,"end_line":2765,"raw_tex":"\\begin{proof}\nApply Lemma~\\ref{lm:decay1} with $\\hat\\mu=\\mu_k$. Then\n$$\n\\begin{aligned}\n(1+c_k\\alpha)\\mathcal E(\\bs z_{k+1};\\mu_k)\n\\leq{}&\n\\mathcal E(\\bs z_k;\\mu_k)\n+\\frac1{2L}\\|\\nabla f_k(x_{k+1})\\|^2\n-\\frac1{2L}\\|\\nabla f_k(x_k)\\|^2\n\\\\\n&\\quad\n-\\alpha\\left(\nd_k\\Delta_f(x_{k+1},x^\\star)\n+\nb_kD_{f_k}(x_{k+1},x^\\star)\n\\right).\n\\end{aligned}\n$$\n","sha256_utf8":"b34d728ae2e623247c2f366ffafe4584b3c3614b6ae05da1536d8b55f80354b8"}]},{"id":"S068","semantic_type":"proof_step","kind":"proof_step","title":"控制梯度移位变化的扰动","source_units":[{"start":96548,"end":97284,"start_line":2766,"end_line":2792,"raw_tex":"We first change the shifted gradient using Lemma~\\ref{lm:gradient_norm_ineq}:\n$$\n\\frac1{2L}\\|\\nabla f_k(x_{k+1})\\|^2\n\\leq\n\\frac1{2L}\\|\\nabla f_{k+1}(x_{k+1})\\|^2\n+\n\\frac{\\Delta\\delta_k\\alpha^2}{2}\n\\langle\\nabla f_k(x_{k+1}),x_{k+1}-x^\\star\\rangle.\n$$\nUsing\n$$\n\\begin{aligned}\n\\frac{\\Delta\\delta_k\\alpha^2}{2}\n\\langle\\nabla f_k(x_{k+1}),x_{k+1}-x^\\star\\rangle\n={}&\n\\Delta\\delta_k\\alpha^2D_{f_k}(x_{k+1},x^\\star)\n+\\frac{\\Delta\\delta_k\\alpha^2}{2}\n\\Delta_f(x_{k+1},x^\\star),\n\\end{aligned}\n$$\nand\n$$\n\\Delta\\delta_k\\alpha^2D_{f_k}(x_{k+1},x^\\star)\n\\leq\n\\Delta\\delta_k\\alpha^2\\mathcal E(\\bs z_{k+1};\\mu_k),\n$$\nthe first term can be absorbed into the left-hand side, which slightly weakens the contraction factor from $1+c_k\\alpha$ to $r_k$. \n","sha256_utf8":"9c93be844d6d2572371f3f2c44e1367f5f7b58233c2c6bdb86961b120f66f1d8"}]},{"id":"S069","semantic_type":"proof_step","kind":"proof_step","title":"调整收缩因子与能量参数","source_units":[{"start":97284,"end":97819,"start_line":2793,"end_line":2819,"raw_tex":"We then obtain\n$$\n\\begin{aligned}\nr_k^{-1}\\mathcal E(\\bs z_{k+1};\\mu_k)\n\\leq{}&\n\\mathcal E(\\bs z_k;\\mu_k)\n+\\frac1{2L}\\|\\nabla f_{k+1}(x_{k+1})\\|^2\n-\\frac1{2L}\\|\\nabla f_k(x_k)\\|^2\n\\\\\n&\\quad\n-\\alpha\\left[\n\\left(\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\right)\\Delta_f(x_{k+1},x^\\star)\n+\nb_kD_{f_k}(x_{k+1},x^\\star)\n\\right].\n\\end{aligned}\n$$\nSince $\\mu_{k+1}\\geq\\mu_k$,\n$$\n\\mathcal E(\\bs z_{k+1};\\mu_{k+1})\n\\leq\n\\mathcal E(\\bs z_{k+1};\\mu_k).\n$$\nThus the same inequality holds with $\\mathcal E(\\bs z_{k+1};\\mu_{k+1})$ on the left-hand side.\n\n","sha256_utf8":"54ebc9e410bebe273bfb52669d78644dfd7dc38f146d51e5171be7f28e57246c"}]},{"id":"S070","semantic_type":"proof_step","kind":"proof_step","title":"比较非对称量与剩余散度","source_units":[{"start":97819,"end":98219,"start_line":2820,"end_line":2840,"raw_tex":"By the definition of $\\omega_k$,\n$$\n|\\Delta_f(x_{k+1},x^\\star)|\n\\leq\n\\omega_kD_f(x_{k+1},x^\\star).\n$$\nMoreover, since $\\mu_k=(1-\\delta_k)\\mu$ and $f$ is $\\mu$-strongly convex,\n$$\n\\begin{aligned}\nD_{f_k}(x_{k+1},x^\\star)\n&=\nD_f(x_{k+1},x^\\star)\n-\\frac{\\mu_k}{2}\\|x_{k+1}-x^\\star\\|^2\n\\\\\n&\\geq\nD_f(x_{k+1},x^\\star)\n-\\frac{\\mu_k}{\\mu}D_f(x_{k+1},x^\\star)\n=\n\\delta_kD_f(x_{k+1},x^\\star).\n\\end{aligned}\n$$\n","sha256_utf8":"bf3853937de62aa2565159b6ca86c08f9593bd55ad9c65ac0e0119e41c1b4a59"}]},{"id":"S071","semantic_type":"proof_step","kind":"proof_step","title":"利用模量条件吸收非对称项","source_units":[{"start":98219,"end":98767,"start_line":2841,"end_line":2872,"raw_tex":"Therefore,\n$$\n\\begin{aligned}\n&\n\\left(\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\right)\\Delta_f(x_{k+1},x^\\star)\n+\nb_kD_{f_k}(x_{k+1},x^\\star)\n\\\\\n&\\geq\n\\left[\nb_k\\delta_k\n-\n\\left|\nd_k-\\frac{\\Delta\\delta_k\\alpha}{2}\n\\right|\\omega_k\n\\right]\nD_f(x_{k+1},x^\\star)\n\\geq0,\n\\end{aligned}\n$$\nwhere the last inequality follows from \\eqref{eq:omega_condition}. It follows that\n$$\n\\begin{aligned}\nr_k^{-1}\\mathcal E(\\bs z_{k+1};\\mu_{k+1})\n\\leq{}&\n\\mathcal E(\\bs z_k;\\mu_k)\n+\\frac1{2L}\\|\\nabla f_{k+1}(x_{k+1})\\|^2\n-\\frac1{2L}\\|\\nabla f_k(x_k)\\|^2.\n\\end{aligned}\n$$\n","sha256_utf8":"f4196cba51fd75a48a473275bdc1db60ed3fd468e1488f3ac808f6876a16c2d3"}]},{"id":"S072","semantic_type":"proof_step","kind":"proof_step","title":"对偶移位得到最终递推","source_units":[{"start":98767,"end":99244,"start_line":2873,"end_line":2891,"raw_tex":"Subtracting $r_k^{-1}(2L)^{-1}\\|\\nabla f_{k+1}(x_{k+1})\\|^2$ from both sides gives\n$$\n\\begin{aligned}\nr_k^{-1}\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq{}&\n\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k)\n+\n\\frac{1-r_k^{-1}}{2L}\n\\|\\nabla f_{k+1}(x_{k+1})\\|^2.\n\\end{aligned}\n$$\nSince $r_k^{-1}\\geq1$, the last term is nonpositive. Therefore,\n$$\nr_k^{-1}\\widetilde{\\mathcal E}(\\bs z_{k+1};\\mu_{k+1})\n\\leq\n\\widetilde{\\mathcal E}(\\bs z_k;\\mu_k),\n$$\nwhich proves the claim.\n\\end{proof}","sha256_utf8":"4ac665b6bbcf2151e323bcb1502062bf19ea5a92450bfa17da76c16bcdb1f620"}]},{"id":"S073","semantic_type":"proof","kind":"proof","title":"沿迭代模量趋零的推导","source_units":[{"start":93990,"end":94195,"start_line":2673,"end_line":2683,"raw_tex":"By Theorem~\\ref{thm:linearconvHNAG-funcval},\n$$\n\\|x_{k+1}-x^\\star\\|\n\\leq\nC_1\\left(1+\\sqrt{2\\rho}\\right)^{-\\frac{k+1}{2}},\n\\qquad\nC_1:=\n\\left(\n\\frac{2(1+2\\alpha)}{\\alpha\\mu}\\widetilde E_0\n\\right)^{1/2}.\n$$\n","sha256_utf8":"59330d8fe4fb804948beecbe8d6e5c62f7301e8927dc87b27cb2d6c4c29ca16b"},{"start":94415,"end":94538,"start_line":2693,"end_line":2693,"raw_tex":"Since $\\omega$ is nondecreasing and its argument decreases to zero, $\\{\\omega_k\\}$ is nonincreasing and converges to zero.\n","sha256_utf8":"5e7ddeb586082814805bbef702489ada0b98a82f906772a0bc046ab2dbe4552a"}]},{"id":"S074","semantic_type":"proof_step","kind":"proof_step","title":"全局误差界给出局部半径","source_units":[{"start":93990,"end":94195,"start_line":2673,"end_line":2683,"raw_tex":"By Theorem~\\ref{thm:linearconvHNAG-funcval},\n$$\n\\|x_{k+1}-x^\\star\\|\n\\leq\nC_1\\left(1+\\sqrt{2\\rho}\\right)^{-\\frac{k+1}{2}},\n\\qquad\nC_1:=\n\\left(\n\\frac{2(1+2\\alpha)}{\\alpha\\mu}\\widetilde E_0\n\\right)^{1/2}.\n$$\n","sha256_utf8":"59330d8fe4fb804948beecbe8d6e5c62f7301e8927dc87b27cb2d6c4c29ca16b"}]},{"id":"S075","semantic_type":"proof_step","kind":"proof_step","title":"单调模量沿半径趋于零","source_units":[{"start":94415,"end":94538,"start_line":2693,"end_line":2693,"raw_tex":"Since $\\omega$ is nondecreasing and its argument decreases to zero, $\\{\\omega_k\\}$ is nonincreasing and converges to zero.\n","sha256_utf8":"5e7ddeb586082814805bbef702489ada0b98a82f906772a0bc046ab2dbe4552a"}]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S018","edges":[{"source_id":"S001","target_id":"S030","kind":"supports"},{"source_id":"S002","target_id":"S027","kind":"supports"},{"source_id":"S003","target_id":"S023","kind":"supports"},{"source_id":"S003","target_id":"S064","kind":"supports"},{"source_id":"S004","target_id":"S030","kind":"supports"},{"source_id":"S004","target_id":"S034","kind":"supports"},{"source_id":"S004","target_id":"S046","kind":"supports"},{"source_id":"S004","target_id":"S049","kind":"supports"},{"source_id":"S004","target_id":"S054","kind":"supports"},{"source_id":"S004","target_id":"S068","kind":"supports"},{"source_id":"S005","target_id":"S030","kind":"supports"},{"source_id":"S005","target_id":"S038","kind":"supports"},{"source_id":"S005","target_id":"S056","kind":"supports"},{"source_id":"S005","target_id":"S058","kind":"supports"},{"source_id":"S006","target_id":"S031","kind":"supports"},{"source_id":"S006","target_id":"S041","kind":"supports"},{"source_id":"S006","target_id":"S043","kind":"supports"},{"source_id":"S006","target_id":"S047","kind":"supports"},{"source_id":"S006","target_id":"S070","kind":"supports"},{"source_id":"S007","target_id":"S030","kind":"supports"},{"source_id":"S007","target_id":"S035","kind":"supports"},{"source_id":"S007","target_id":"S046","kind":"supports"},{"source_id":"S007","target_id":"S054","kind":"supports"},{"source_id":"S008","target_id":"S035","kind":"supports"},{"source_id":"S008","target_id":"S037","kind":"supports"},{"source_id":"S008","target_id":"S041","kind":"supports"},{"source_id":"S008","target_id":"S054","kind":"supports"},{"source_id":"S008","target_id":"S057","kind":"supports"},{"source_id":"S009","target_id":"S030","kind":"supports"},{"source_id":"S009","target_id":"S034","kind":"supports"},{"source_id":"S009","target_id":"S041","kind":"supports"},{"source_id":"S009","target_id":"S074","kind":"supports"},{"source_id":"S010","target_id":"S041","kind":"supports"},{"source_id":"S010","target_id":"S064","kind":"supports"},{"source_id":"S010","target_id":"S067","kind":"supports"},{"source_id":"S011","target_id":"S046","kind":"supports"},{"source_id":"S011","target_id":"S047","kind":"supports"},{"source_id":"S011","target_id":"S054","kind":"supports"},{"source_id":"S011","target_id":"S056","kind":"supports"},{"source_id":"S011","target_id":"S070","kind":"supports"},{"source_id":"S012","target_id":"S070","kind":"supports"},{"source_id":"S012","target_id":"S075","kind":"supports"},{"source_id":"S013","target_id":"S062","kind":"supports"},{"source_id":"S013","target_id":"S064","kind":"supports"},{"source_id":"S013","target_id":"S068","kind":"supports"},{"source_id":"S013","target_id":"S069","kind":"supports"},{"source_id":"S014","target_id":"S022","kind":"supports"},{"source_id":"S014","target_id":"S070","kind":"supports"},{"source_id":"S014","target_id":"S075","kind":"supports"},{"source_id":"S015","target_id":"S041","kind":"supports"},{"source_id":"S015","target_id":"S058","kind":"supports"},{"source_id":"S015","target_id":"S027","kind":"supports"},{"source_id":"S016","target_id":"S037","kind":"supports"},{"source_id":"S016","target_id":"S047","kind":"supports"},{"source_id":"S016","target_id":"S057","kind":"supports"},{"source_id":"S017","target_id":"S027","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S036","kind":"supports"},{"source_id":"S034","target_id":"S038","kind":"supports"},{"source_id":"S035","target_id":"S036","kind":"supports"},{"source_id":"S035","target_id":"S037","kind":"supports"},{"source_id":"S035","target_id":"S038","kind":"supports"},{"source_id":"S036","target_id":"S038","kind":"supports"},{"source_id":"S037","target_id":"S038","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S035","target_id":"S033","kind":"supports"},{"source_id":"S036","target_id":"S033","kind":"supports"},{"source_id":"S037","target_id":"S033","kind":"supports"},{"source_id":"S038","target_id":"S033","kind":"supports"},{"source_id":"S033","target_id":"S032","kind":"supports"},{"source_id":"S032","target_id":"S041","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S041","target_id":"S040","kind":"supports"},{"source_id":"S042","target_id":"S040","kind":"supports"},{"source_id":"S043","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S074","kind":"supports"},{"source_id":"S039","target_id":"S021","kind":"supports"},{"source_id":"S046","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S048","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S051","kind":"supports"},{"source_id":"S046","target_id":"S045","kind":"supports"},{"source_id":"S047","target_id":"S045","kind":"supports"},{"source_id":"S048","target_id":"S045","kind":"supports"},{"source_id":"S049","target_id":"S045","kind":"supports"},{"source_id":"S050","target_id":"S045","kind":"supports"},{"source_id":"S051","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S055","kind":"supports"},{"source_id":"S054","target_id":"S056","kind":"supports"},{"source_id":"S054","target_id":"S057","kind":"supports"},{"source_id":"S054","target_id":"S058","kind":"supports"},{"source_id":"S054","target_id":"S059","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S059","kind":"supports"},{"source_id":"S056","target_id":"S058","kind":"supports"},{"source_id":"S056","target_id":"S059","kind":"supports"},{"source_id":"S057","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S055","target_id":"S053","kind":"supports"},{"source_id":"S056","target_id":"S053","kind":"supports"},{"source_id":"S057","target_id":"S053","kind":"supports"},{"source_id":"S058","target_id":"S053","kind":"supports"},{"source_id":"S059","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S067","kind":"supports"},{"source_id":"S062","target_id":"S063","kind":"supports"},{"source_id":"S063","target_id":"S064","kind":"supports"},{"source_id":"S062","target_id":"S061","kind":"supports"},{"source_id":"S063","target_id":"S061","kind":"supports"},{"source_id":"S064","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S068","kind":"supports"},{"source_id":"S067","target_id":"S069","kind":"supports"},{"source_id":"S068","target_id":"S069","kind":"supports"},{"source_id":"S069","target_id":"S071","kind":"supports"},{"source_id":"S070","target_id":"S071","kind":"supports"},{"source_id":"S071","target_id":"S072","kind":"supports"},{"source_id":"S067","target_id":"S066","kind":"supports"},{"source_id":"S068","target_id":"S066","kind":"supports"},{"source_id":"S069","target_id":"S066","kind":"supports"},{"source_id":"S070","target_id":"S066","kind":"supports"},{"source_id":"S071","target_id":"S066","kind":"supports"},{"source_id":"S072","target_id":"S066","kind":"supports"},{"source_id":"S066","target_id":"S065","kind":"supports"},{"source_id":"S065","target_id":"S025","kind":"supports"},{"source_id":"S074","target_id":"S075","kind":"supports"},{"source_id":"S074","target_id":"S073","kind":"supports"},{"source_id":"S075","target_id":"S073","kind":"supports"},{"source_id":"S073","target_id":"S020","kind":"supports"},{"source_id":"S073","target_id":"S070","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S026","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S024","kind":"supports"},{"source_id":"S022","target_id":"S027","kind":"supports"},{"source_id":"S023","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S021","target_id":"S019","kind":"supports"},{"source_id":"S022","target_id":"S019","kind":"supports"},{"source_id":"S023","target_id":"S019","kind":"supports"},{"source_id":"S024","target_id":"S019","kind":"supports"},{"source_id":"S025","target_id":"S019","kind":"supports"},{"source_id":"S026","target_id":"S019","kind":"supports"},{"source_id":"S027","target_id":"S019","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"}]}}
{"record_id":"2510.27156v1","paper":{"title":"Variable Smoothing Alternating Proximal Gradient Algorithm for Coupled Composite Optimization","authors":["Xian-Jun Long","Kang Zeng","Gao-Xi Li","Minh N. Dao","Zai-Yun Peng"],"arxiv_id":"2510.27156","version":"v1","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-026-02966-8","abstract_url":"https://arxiv.org/abs/2510.27156v1","pdf_url":"https://arxiv.org/pdf/2510.27156v1"},"context":{"raw_tex":"\\begin{equation}\\label{eq:L1E1}\r\n\\min_{(x,y)\\in \\mathbb{R}^{n}\\times\\mathbb{R}^{m}} \\mathcal{L}(x,y)=f(x) + g(Ay)+H(x,y),\r\n\\end{equation}\n\nwhere $f:\\mathbb{R}^{n}\\rightarrow \\mathbb{R}$ is a (possibly nonsmooth) convex function, $g:\\mathbb{R}^{d}\\rightarrow \\mathbb{R}\\cup \\{+\\infty\\}$ is a nonsmooth and proper lower semicontinuous $\\rho$-weakly convex function, $A:\\mathbb{R}^{m} \\rightarrow \\mathbb{R}^{d}$ is a linear operator, and $H:\\mathbb{R}^{n}\\times \\mathbb{R}^{m} \\rightarrow \\mathbb{R}$ is a continuously differentiable (possibly nonconvex) function. Such problem has a wide range of applications, including compressed sensing, machine learning, nonnegative matrix factorization, image denoising, signal recovery, and multimodal learning for image classification; see, for example \\cite{BST,CP,CP1,DD,OBP,PN,TSR}.\r\n\n\n\\begin{align}\\label{guanghua}\r\n\\min_{(x,y)\\in \\mathbb{R}^{n}\\times\\mathbb{R}^{m}} \\mathcal{L}_\\mu(x,y)=f(x)+g_{\\mu}(Ay)+H(x,y).\r\n\\end{align}\n\nLet $\\mathbb{R}^n$ be a finite-dimensional Euclidean space, which equipped with standard inner product $\\langle \\cdot,\\cdot \\rangle$ and norm $\\|\\cdot \\|$, respectively. For any $x,y\\in \\mathbb{R}^n$, $\\|(x,y)\\|:=\\sqrt{\\|x\\|^2+\\|y\\|^2}$. Given a nonempty set $C\\subseteq \\mathbb{R}^n$, the distance from $z\\in \\mathbb{R}^n$, the \\emph{distance} from $z$ to $C$ is defined as\r\n$\\dist(z,C):=\\inf_{w\\in C}\\|w-z\\|$. Let $h: \\mathbb{R}^n\\rightarrow {\\mathbb{R}}\\cup \\{+\\infty\\}$ be a mapping, the \\emph{domain} of $h$ is defined by $\\dom h:=\\{x\\in \\mathbb{R}^n:h(x)<+\\infty\\}$. The function $h$ is said to be \\emph{proper} if $\\dom h \\neq\\varnothing$ and \\emph{lower semicontinuous} if $h(x)\\leq \\liminf_{z\\rightarrow x}h(z)$ for any $x\\in \\mathbb{R}^n$.\r\n\r\nLet $F:\\mathbb{R}^n\\rightarrow \\mathbb{R}\\cup \\{+\\infty\\}$ be a proper lower semicontinuous convex function and $\\lambda>0$. The \\emph{proximal operator} of $F$ at $v\\in \\mathbb{R}^n$ is defined by\r\n\\begin{align*}\r\n\\prox_{\\lambda F}(v):=\\argmin_{x\\in \\mathbb{R}^n} \\left(F(x)+\\frac{1}{2\\lambda}\\|x-v\\|^2\\right).\r\n\\end{align*}\r\nIt is known that $\\prox_{\\lambda F}$ is nonexpansive, i.e., for all $x,y\\in \\mathbb{R}^n$,\r\n\\begin{align*}\r\n\\|\\prox_{\\lambda F}(x)-\\prox_{\\lambda F}(y)\\|\\leq\\|x-y\\|.\r\n\\end{align*}\r\nThe \\emph{regular subdifferential} of $F$ at $x\\in\\dom F$ is defined by\r\n$$\\widehat{\\partial}F(x):=\\left\\{u\\in\\mathbb{R}^{n}:\\liminf_{y\\rightarrow x\\atop y\\neq x} \\frac{\\ F(y)- F(x) + \\langle u, y-x \\rangle}{\\|y-x\\|}\\geq0\\right\\}.$$\r\nThe \\emph{limiting subdifferential} of $F$ at $x\\in\\dom F$ is defined by\r\n \\begin{align*}\r\n\\partial F(x) :=\\{u \\in\\mathbb{R}^{n}: \\exists\\ x_{k}\\rightarrow x, \\widehat{\\partial}F(x_{k}) \\ni u_{k} \\rightarrow u \\ \\mathrm{with} \\  F(x_{k}) \\rightarrow F(x)\\}.\r\n\\end{align*}\r\nWhen $F$ is a convex function, both subdifferentials coincide with the subdifferential in the sense of convex analysis, i.e.,\r\n\\begin{align*}\r\n\\widehat{\\partial}F(x)=\\partial F(x)=\\{u\\in\\mathbb{R}^{n}: \\forall y\\in\\mathbb{R}^{n},\\quad F(y)-F(x)\\geq\\langle{u,y-x}\\rangle\\}.\r\n\\end{align*}\r\nClearly, $\\widehat{\\partial}F(x) \\subseteq \\partial F(x)$ for all $x\\in \\mathbb{R}^{n}$ and both of them are closed.  If $G:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ is  a continuously differentiable function, then $\\partial G(x)=\\{\\nabla G(x)\\}$ and $\\partial(F+G)(x)=\\partial F(x)+\\nabla G(x)$ for all $x\\in \\mathbb{R}^n$, where $\\nabla G(x)$ denotes the gradient of $G$ at $x$.\r\n\n\n\\begin{definition}[\\cite{V}] A function $F: \\mathbb{R}^n\\rightarrow (-\\infty,+\\infty]$  is said to be \\emph{$\\rho$-weakly convex} if $F +\\frac{\\rho}{2}\\|\\cdot\\|^2$ is convex.\r\n\\end{definition}\n\n\\begin{definition}[{\\cite[Definition 2.1]{BW}}] Let $F: \\mathbb{R}^n\\rightarrow (-\\infty,+\\infty]$ be a proper $\\rho$-weakly convex lower semicontinuous function. The Moreau envelope function of $F$ is defined as\r\n\\begin{align*}\r\nF_{\\mu}(x):=\\min_{y\\in \\mathbb{R}^n}\\{F(y)+\\frac{1}{2\\mu}\\|y-x\\|^2\\},\r\n\\end{align*}\r\nwhere $\\mu \\in(0,1/\\rho)$.\r\n\\end{definition}\n\n\\begin{align*}\r\nG_\\lambda(x,y) =\\frac{1}{\\lambda}(x -\\prox_{\\lambda f}(x -\\lambda\\nabla_x  H(x,y))).\r\n\\end{align*}\n\n\\begin{definition}\r\nLet $(x^*,y^*)\\in \\mathbb{R}^n\\times \\mathbb{R}^m$. We say that $(x^*,y^*)$ is an \\emph{$\\varepsilon$-approximate stationary point} of problem~\\eqref{eq:L1E1} if\r\n\\begin{align}\\label{wending}\r\n\\dist(0,G_\\lambda(x^*,y^*))+\\dist(-\\nabla_yH(x^*,y^*),A^*\\partial g(Ay^*))\\leq\\varepsilon.\r\n\\end{align}\r\nSimilarly, $(x^*,y^*)$ is said to be an \\emph{$\\varepsilon$-approximate stationary point} of problem~\\eqref{guanghua} if\r\n\\begin{align}\\label{wending1}\r\n\\dist(0,G_\\lambda(x^*,y^*))+\\dist(-\\nabla_yH(x^*,y^*),A^*\\nabla g_\\mu(Ay^*))\\leq\\varepsilon.\r\n\\end{align}\r\n\\end{definition}\n\n\\begin{assumption}\r\n\\label{a:H}\r\nFor any $y\\in \\mathbb{R}^m$, the partial gradient $\\nabla_x H(\\cdot,y)$ is $L_{11}$-Lipschitz continuous, i.e.,\r\n\\begin{align*}\r\n\\forall x_1,x_2\\in \\mathbb{R}^n,\\quad \\|\\nabla_x H(x_1,y)-\\nabla_x H(x_2,y)\\|\\leq L_{11}\\|x_1-x_2\\|.\r\n\\end{align*}\r\nFor any $x\\in \\mathbb{R}^n$, the partial gradient $\\nabla_x H(x,\\cdot)$ is $L_{12}$-Lipschitz continuous and the partial gradient $\\nabla_y H(x,\\cdot)$ is $L_{22}$-Lipschitz continuous, i.e.,\r\n\\begin{align*}\r\n&\\forall y_1,y_2\\in \\mathbb{R}^n,\\quad \\|\\nabla_x H(x,y_1)-\\nabla_x H(x,y_2)\\|\\leq L_{12}\\|y_1-y_2\\|,\\\\\r\n&\\forall y_1,y_2\\in\\mathbb{ R}^m,\\quad \\|\\nabla_y H(x,y_1)-\\nabla_y H(x,y_2)\\|\\leq L_{22}\\|y_1-y_2\\|.\r\n\\end{align*}\r\n\\end{assumption}\n\n\\begin{assumption}\r\n\\label{a:g}\r\nThe function $g$ is $\\rho$-weakly convex and $L_g$-Lipschitz continuous. The function $\\mathcal{L}$ is bounded below.\r\n\\end{assumption}\n\n\\begin{align}\\label{app}\r\n\\min_{(x,y)\\in \\mathbb{R}^{n}\\times\\mathbb{R}^{m}}\\mathcal{L}_{\\mu_k}(x,y)=f(x)+g_{\\mu_k}(Ay)+H(x,y).\r\n\\end{align}\n\nDenote $\\Phi_k(x,y):=g_{\\mu_k}(Ay)+H(x,y)$. For the smoothed function $\\Phi_k$, we have\r\n\\begin{align*}\r\n\\nabla_y\\Phi_k(x,y) &=\\frac{1}{\\mu_k}A^{*}(Ay-\\prox_{\\mu_k g}(Ay))+\\nabla_y H(x,y) \\text{~~and}\\\\\r\n\\nabla_x\\Phi_k(x,y) &=\r\n\\nabla_x H(x,y).\r\n\\end{align*}\n\nFrom Assumptions~\\ref{a:H} and \\ref{a:g}, it can be seen that $\\nabla_x\\Phi_k(\\cdot,y)$ and $\\nabla_y\\Phi_k(x,\\cdot)$ are two Lipschitz continuous functions with Lipschitz constants $L_{11}$ and $L_k:=L_{22}+\\|A\\|^2\\max\\{\\frac{1}{\\mu_k},\\frac{\\rho}{1-\\rho\\mu_k}\\}$, respectively. Increasing $L_{11}$, $L_{22}$, and/or $\\rho$ if necessary, we can assume that $L_{12}^2 <L_{11}(L_{22} +2\\rho\\|A\\|^2)$, which ensures the existence of $\\alpha$ in the subsequent algorithm.\r\n\n\n\\begin{algorithm}[htb]\r\n\\caption{Variable smoothing alternating proximal gradient (VsaPG)}\r\n\\label{algo}\r\n\r\n\\textbf{Step 1.} Let $x^1 =\\bar{x}^1\\in \\mathbb{R}^{n}$, $y^1 =\\bar{y}^1\\in \\mathbb{R}^{m}$, $\\mu_1\\in (0, 1/\\rho)$, and set $k =1$. Let $\\gamma\\in (0, +\\infty)$, $\\eta\\in (0, +\\infty)$, $\\theta\\in (0, 1)$, $\\beta\\in(0, 1]$, $\\sigma\\in (0, 2/L_{11})$, and let $\\alpha\\in (0, 1)$ be such that\r\n\\begin{align*}\r\n1 -\\alpha^2 -\\frac{L_{12}^2(1 +\\alpha)^2}{L_{11}(L_{22} +2\\rho\\|A\\|^2)} >0.\r\n\\end{align*}\r\n\r\n\\textbf{Step 2.} Let $\\mu_{k+1}\\in [\\mu_k/2, \\mu_k]$, $\\alpha_k\\in [-\\alpha, \\alpha]$, $\\beta_k\\in [\\beta,1]$, $\\sigma_k\\in [\\gamma k^{-\\theta}, \\sigma]$, and $\\tau_k\\in [\\eta k^{-\\theta}, 1/L_k]$. Compute\r\n\\begin{align}\r\n\\label{s21}&y^{k+1}=\\bar{y}^k-\\tau_k(A^*\\nabla g_{\\mu_k}(A\\bar{y}^{k})+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)),\\\\\r\n\\label{s41}&\\bar{y}^{k+1}=y^{k+1}+\\alpha_k(y^{k+1}-\\bar{y}^k),\\\\\r\n\\label{s31}&x^{k+1}=\\prox_{\\sigma_k f}(\\bar{x}^k-\\sigma_k\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1})),\\\\\r\n\\label{s42}&\\bar{x}^{k+1} =(1 -\\beta_k)\\bar{x}^k +\\beta_kx^{k+1}.\r\n\\end{align}\r\n\r\n\\textbf{Step 3.} If a termination criterion is not met, set $k =k+1$ and go to Step 2.\r\n\\end{algorithm}\n\nObviously, Assumption \\ref{a:g} implies that $\\liminf_{k\\rightarrow \\infty} \\mathcal{L}_{\\mu_k}(x^k,y^k)>-\\infty$. From now on, we denote\r\n\\begin{align*}\r\n\\mathcal{L}^* :=\\liminf_{k\\rightarrow \\infty} \\mathcal{L}_{\\mu_k}(x^k,y^k).\r\n\\end{align*}","units":[{"start":5196,"end":5333,"start_line":127,"end_line":129,"raw_tex":"\\begin{equation}\\label{eq:L1E1}\r\n\\min_{(x,y)\\in \\mathbb{R}^{n}\\times\\mathbb{R}^{m}} \\mathcal{L}(x,y)=f(x) + g(Ay)+H(x,y),\r\n\\end{equation}","sha256_utf8":"4c5c9f288c9f005c2b1dd00e1918551de2f5d4580a2705a0aa4c1185b1ed64f4"},{"start":5335,"end":6023,"start_line":130,"end_line":130,"raw_tex":"where $f:\\mathbb{R}^{n}\\rightarrow \\mathbb{R}$ is a (possibly nonsmooth) convex function, $g:\\mathbb{R}^{d}\\rightarrow \\mathbb{R}\\cup \\{+\\infty\\}$ is a nonsmooth and proper lower semicontinuous $\\rho$-weakly convex function, $A:\\mathbb{R}^{m} \\rightarrow \\mathbb{R}^{d}$ is a linear operator, and $H:\\mathbb{R}^{n}\\times \\mathbb{R}^{m} \\rightarrow \\mathbb{R}$ is a continuously differentiable (possibly nonconvex) function. Such problem has a wide range of applications, including compressed sensing, machine learning, nonnegative matrix factorization, image denoising, signal recovery, and multimodal learning for image classification; see, for example \\cite{BST,CP,CP1,DD,OBP,PN,TSR}.\r\n","sha256_utf8":"0edff7967c8086d5dcfde93795b55db1e7d1b3b434a8c211aba3d813a6976ecf"},{"start":8479,"end":8619,"start_line":150,"end_line":152,"raw_tex":"\\begin{align}\\label{guanghua}\r\n\\min_{(x,y)\\in \\mathbb{R}^{n}\\times\\mathbb{R}^{m}} \\mathcal{L}_\\mu(x,y)=f(x)+g_{\\mu}(Ay)+H(x,y).\r\n\\end{align}","sha256_utf8":"9c1b4f69707301ebbe6085fdeaae1ef4556e8a3904494afd38b80398dda9f465"},{"start":9551,"end":11983,"start_line":162,"end_line":183,"raw_tex":"Let $\\mathbb{R}^n$ be a finite-dimensional Euclidean space, which equipped with standard inner product $\\langle \\cdot,\\cdot \\rangle$ and norm $\\|\\cdot \\|$, respectively. For any $x,y\\in \\mathbb{R}^n$, $\\|(x,y)\\|:=\\sqrt{\\|x\\|^2+\\|y\\|^2}$. Given a nonempty set $C\\subseteq \\mathbb{R}^n$, the distance from $z\\in \\mathbb{R}^n$, the \\emph{distance} from $z$ to $C$ is defined as\r\n$\\dist(z,C):=\\inf_{w\\in C}\\|w-z\\|$. Let $h: \\mathbb{R}^n\\rightarrow {\\mathbb{R}}\\cup \\{+\\infty\\}$ be a mapping, the \\emph{domain} of $h$ is defined by $\\dom h:=\\{x\\in \\mathbb{R}^n:h(x)<+\\infty\\}$. The function $h$ is said to be \\emph{proper} if $\\dom h \\neq\\varnothing$ and \\emph{lower semicontinuous} if $h(x)\\leq \\liminf_{z\\rightarrow x}h(z)$ for any $x\\in \\mathbb{R}^n$.\r\n\r\nLet $F:\\mathbb{R}^n\\rightarrow \\mathbb{R}\\cup \\{+\\infty\\}$ be a proper lower semicontinuous convex function and $\\lambda>0$. The \\emph{proximal operator} of $F$ at $v\\in \\mathbb{R}^n$ is defined by\r\n\\begin{align*}\r\n\\prox_{\\lambda F}(v):=\\argmin_{x\\in \\mathbb{R}^n} \\left(F(x)+\\frac{1}{2\\lambda}\\|x-v\\|^2\\right).\r\n\\end{align*}\r\nIt is known that $\\prox_{\\lambda F}$ is nonexpansive, i.e., for all $x,y\\in \\mathbb{R}^n$,\r\n\\begin{align*}\r\n\\|\\prox_{\\lambda F}(x)-\\prox_{\\lambda F}(y)\\|\\leq\\|x-y\\|.\r\n\\end{align*}\r\nThe \\emph{regular subdifferential} of $F$ at $x\\in\\dom F$ is defined by\r\n$$\\widehat{\\partial}F(x):=\\left\\{u\\in\\mathbb{R}^{n}:\\liminf_{y\\rightarrow x\\atop y\\neq x} \\frac{\\ F(y)- F(x) + \\langle u, y-x \\rangle}{\\|y-x\\|}\\geq0\\right\\}.$$\r\nThe \\emph{limiting subdifferential} of $F$ at $x\\in\\dom F$ is defined by\r\n \\begin{align*}\r\n\\partial F(x) :=\\{u \\in\\mathbb{R}^{n}: \\exists\\ x_{k}\\rightarrow x, \\widehat{\\partial}F(x_{k}) \\ni u_{k} \\rightarrow u \\ \\mathrm{with} \\  F(x_{k}) \\rightarrow F(x)\\}.\r\n\\end{align*}\r\nWhen $F$ is a convex function, both subdifferentials coincide with the subdifferential in the sense of convex analysis, i.e.,\r\n\\begin{align*}\r\n\\widehat{\\partial}F(x)=\\partial F(x)=\\{u\\in\\mathbb{R}^{n}: \\forall y\\in\\mathbb{R}^{n},\\quad F(y)-F(x)\\geq\\langle{u,y-x}\\rangle\\}.\r\n\\end{align*}\r\nClearly, $\\widehat{\\partial}F(x) \\subseteq \\partial F(x)$ for all $x\\in \\mathbb{R}^{n}$ and both of them are closed.  If $G:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ is  a continuously differentiable function, then $\\partial G(x)=\\{\\nabla G(x)\\}$ and $\\partial(F+G)(x)=\\partial F(x)+\\nabla G(x)$ for all $x\\in \\mathbb{R}^n$, where $\\nabla G(x)$ denotes the gradient of $G$ at $x$.\r\n","sha256_utf8":"b460734b2f1e2927a796e7c754fa7571ca71a4b6f43376cc9af8e77b1e2f0531"},{"start":12350,"end":12542,"start_line":194,"end_line":195,"raw_tex":"\\begin{definition}[\\cite{V}] A function $F: \\mathbb{R}^n\\rightarrow (-\\infty,+\\infty]$  is said to be \\emph{$\\rho$-weakly convex} if $F +\\frac{\\rho}{2}\\|\\cdot\\|^2$ is convex.\r\n\\end{definition}","sha256_utf8":"15bc72b0a946710c6ebf4d7688b8b7402f8274fb0b874c6114e44a77efafc369"},{"start":12654,"end":13013,"start_line":201,"end_line":206,"raw_tex":"\\begin{definition}[{\\cite[Definition 2.1]{BW}}] Let $F: \\mathbb{R}^n\\rightarrow (-\\infty,+\\infty]$ be a proper $\\rho$-weakly convex lower semicontinuous function. The Moreau envelope function of $F$ is defined as\r\n\\begin{align*}\r\nF_{\\mu}(x):=\\min_{y\\in \\mathbb{R}^n}\\{F(y)+\\frac{1}{2\\mu}\\|y-x\\|^2\\},\r\n\\end{align*}\r\nwhere $\\mu \\in(0,1/\\rho)$.\r\n\\end{definition}","sha256_utf8":"4f9332665f3bddb2a4cf61dc5f82519b4d645c81c5ce0428ea6b8cc6624c2aab"},{"start":16137,"end":16251,"start_line":276,"end_line":278,"raw_tex":"\\begin{align*}\r\nG_\\lambda(x,y) =\\frac{1}{\\lambda}(x -\\prox_{\\lambda f}(x -\\lambda\\nabla_x  H(x,y))).\r\n\\end{align*}","sha256_utf8":"7510f4e485203a4746a2ae5b70686f04e738dc23f5d8060fa8d1d01376b1105a"},{"start":18156,"end":18750,"start_line":329,"end_line":338,"raw_tex":"\\begin{definition}\r\nLet $(x^*,y^*)\\in \\mathbb{R}^n\\times \\mathbb{R}^m$. We say that $(x^*,y^*)$ is an \\emph{$\\varepsilon$-approximate stationary point} of problem~\\eqref{eq:L1E1} if\r\n\\begin{align}\\label{wending}\r\n\\dist(0,G_\\lambda(x^*,y^*))+\\dist(-\\nabla_yH(x^*,y^*),A^*\\partial g(Ay^*))\\leq\\varepsilon.\r\n\\end{align}\r\nSimilarly, $(x^*,y^*)$ is said to be an \\emph{$\\varepsilon$-approximate stationary point} of problem~\\eqref{guanghua} if\r\n\\begin{align}\\label{wending1}\r\n\\dist(0,G_\\lambda(x^*,y^*))+\\dist(-\\nabla_yH(x^*,y^*),A^*\\nabla g_\\mu(Ay^*))\\leq\\varepsilon.\r\n\\end{align}\r\n\\end{definition}","sha256_utf8":"dc35e0a4f9fb4ac4504257c99efd197c7eed56d977e349b9323d94d62644f51f"},{"start":18834,"end":19559,"start_line":342,"end_line":353,"raw_tex":"\\begin{assumption}\r\n\\label{a:H}\r\nFor any $y\\in \\mathbb{R}^m$, the partial gradient $\\nabla_x H(\\cdot,y)$ is $L_{11}$-Lipschitz continuous, i.e.,\r\n\\begin{align*}\r\n\\forall x_1,x_2\\in \\mathbb{R}^n,\\quad \\|\\nabla_x H(x_1,y)-\\nabla_x H(x_2,y)\\|\\leq L_{11}\\|x_1-x_2\\|.\r\n\\end{align*}\r\nFor any $x\\in \\mathbb{R}^n$, the partial gradient $\\nabla_x H(x,\\cdot)$ is $L_{12}$-Lipschitz continuous and the partial gradient $\\nabla_y H(x,\\cdot)$ is $L_{22}$-Lipschitz continuous, i.e.,\r\n\\begin{align*}\r\n&\\forall y_1,y_2\\in \\mathbb{R}^n,\\quad \\|\\nabla_x H(x,y_1)-\\nabla_x H(x,y_2)\\|\\leq L_{12}\\|y_1-y_2\\|,\\\\\r\n&\\forall y_1,y_2\\in\\mathbb{ R}^m,\\quad \\|\\nabla_y H(x,y_1)-\\nabla_y H(x,y_2)\\|\\leq L_{22}\\|y_1-y_2\\|.\r\n\\end{align*}\r\n\\end{assumption}","sha256_utf8":"25777c1296ac3a84388759e6527d58f822adcd6df2fef6e1144b5ff3c9f38060"},{"start":19565,"end":19733,"start_line":356,"end_line":359,"raw_tex":"\\begin{assumption}\r\n\\label{a:g}\r\nThe function $g$ is $\\rho$-weakly convex and $L_g$-Lipschitz continuous. The function $\\mathcal{L}$ is bounded below.\r\n\\end{assumption}","sha256_utf8":"0cf6ffee9ee84928379890111046f75578c86bd6af0728621ab00d28193d26c3"},{"start":23193,"end":23333,"start_line":421,"end_line":423,"raw_tex":"\\begin{align}\\label{app}\r\n\\min_{(x,y)\\in \\mathbb{R}^{n}\\times\\mathbb{R}^{m}}\\mathcal{L}_{\\mu_k}(x,y)=f(x)+g_{\\mu_k}(Ay)+H(x,y).\r\n\\end{align}","sha256_utf8":"23bb12d16803b02dbf8a21d14c2f6cf2370264b4e31687a95e90247002851ea0"},{"start":23335,"end":23593,"start_line":424,"end_line":429,"raw_tex":"Denote $\\Phi_k(x,y):=g_{\\mu_k}(Ay)+H(x,y)$. For the smoothed function $\\Phi_k$, we have\r\n\\begin{align*}\r\n\\nabla_y\\Phi_k(x,y) &=\\frac{1}{\\mu_k}A^{*}(Ay-\\prox_{\\mu_k g}(Ay))+\\nabla_y H(x,y) \\text{~~and}\\\\\r\n\\nabla_x\\Phi_k(x,y) &=\r\n\\nabla_x H(x,y).\r\n\\end{align*}","sha256_utf8":"ed038839b3db83752ab5834f6b7f3b85bacfcaa16fe6645b9a2d2aeb0d5b9ff6"},{"start":23595,"end":24065,"start_line":430,"end_line":430,"raw_tex":"From Assumptions~\\ref{a:H} and \\ref{a:g}, it can be seen that $\\nabla_x\\Phi_k(\\cdot,y)$ and $\\nabla_y\\Phi_k(x,\\cdot)$ are two Lipschitz continuous functions with Lipschitz constants $L_{11}$ and $L_k:=L_{22}+\\|A\\|^2\\max\\{\\frac{1}{\\mu_k},\\frac{\\rho}{1-\\rho\\mu_k}\\}$, respectively. Increasing $L_{11}$, $L_{22}$, and/or $\\rho$ if necessary, we can assume that $L_{12}^2 <L_{11}(L_{22} +2\\rho\\|A\\|^2)$, which ensures the existence of $\\alpha$ in the subsequent algorithm.\r\n","sha256_utf8":"25657c3385707a29c27cb44d4feaa84ebb0cb4e143d473d2c9fb723910ba8e0f"},{"start":24255,"end":25444,"start_line":434,"end_line":452,"raw_tex":"\\begin{algorithm}[htb]\r\n\\caption{Variable smoothing alternating proximal gradient (VsaPG)}\r\n\\label{algo}\r\n\r\n\\textbf{Step 1.} Let $x^1 =\\bar{x}^1\\in \\mathbb{R}^{n}$, $y^1 =\\bar{y}^1\\in \\mathbb{R}^{m}$, $\\mu_1\\in (0, 1/\\rho)$, and set $k =1$. Let $\\gamma\\in (0, +\\infty)$, $\\eta\\in (0, +\\infty)$, $\\theta\\in (0, 1)$, $\\beta\\in(0, 1]$, $\\sigma\\in (0, 2/L_{11})$, and let $\\alpha\\in (0, 1)$ be such that\r\n\\begin{align*}\r\n1 -\\alpha^2 -\\frac{L_{12}^2(1 +\\alpha)^2}{L_{11}(L_{22} +2\\rho\\|A\\|^2)} >0.\r\n\\end{align*}\r\n\r\n\\textbf{Step 2.} Let $\\mu_{k+1}\\in [\\mu_k/2, \\mu_k]$, $\\alpha_k\\in [-\\alpha, \\alpha]$, $\\beta_k\\in [\\beta,1]$, $\\sigma_k\\in [\\gamma k^{-\\theta}, \\sigma]$, and $\\tau_k\\in [\\eta k^{-\\theta}, 1/L_k]$. Compute\r\n\\begin{align}\r\n\\label{s21}&y^{k+1}=\\bar{y}^k-\\tau_k(A^*\\nabla g_{\\mu_k}(A\\bar{y}^{k})+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)),\\\\\r\n\\label{s41}&\\bar{y}^{k+1}=y^{k+1}+\\alpha_k(y^{k+1}-\\bar{y}^k),\\\\\r\n\\label{s31}&x^{k+1}=\\prox_{\\sigma_k f}(\\bar{x}^k-\\sigma_k\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1})),\\\\\r\n\\label{s42}&\\bar{x}^{k+1} =(1 -\\beta_k)\\bar{x}^k +\\beta_kx^{k+1}.\r\n\\end{align}\r\n\r\n\\textbf{Step 3.} If a termination criterion is not met, set $k =k+1$ and go to Step 2.\r\n\\end{algorithm}","sha256_utf8":"590e9d6be8ae03e82695480c2de1832f7f868cd49bcbf209802fccb6c3fcbc64"},{"start":35066,"end":35310,"start_line":609,"end_line":612,"raw_tex":"Obviously, Assumption \\ref{a:g} implies that $\\liminf_{k\\rightarrow \\infty} \\mathcal{L}_{\\mu_k}(x^k,y^k)>-\\infty$. From now on, we denote\r\n\\begin{align*}\r\n\\mathcal{L}^* :=\\liminf_{k\\rightarrow \\infty} \\mathcal{L}_{\\mu_k}(x^k,y^k).\r\n\\end{align*}","sha256_utf8":"53c98c7ba79a859f1580fe9eea47ec75f01e1daeaaf0e6296ca790bbd92a3a1c"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}\r\nSuppose that Assumptions~\\ref{a:H} and \\ref{a:g} hold. Let $(\\bar{x}^k,\\bar{y}^k)$ be the sequence generated by Algorithm~\\ref{algo} with $\\mu_k =\\mu_1 k^{-\\min\\{\\theta, \\frac{1-\\theta}{2}\\}}$. Then, for $\\hat{x}^k=\\bar{x}^k$ and $\\hat{y}^k=\\bar{y}^k-A^*(AA^*)^{-1}(A\\bar{y}^k-\\prox_{\\mu g}(A\\bar{y}^k))$, it holds that\r\n\\begin{multline*}\r\n\\min_{1\\leq j\\leq k}(\\dist(-\\nabla_yH(\\hat{x}^j,\\hat{y}^j),A^*\\partial g(A\\hat{y}^j))+\\dist(0,G_{\\sigma_j}(\\hat{x}^j,\\hat{y}^j))) \\\\ \\leq (\\Theta +(L_{12}+L_{12})L_g\\sigma_{\\min}(A)^{-1}\\mu_1)k^{-\\min\\{\\theta, \\frac{1-\\theta}{2}\\}},\r\n\\end{multline*}\r\nwhere $\\Theta :=\\sqrt{2(\\ln 2)^{-1}M^{-1}(\\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}^*+\\mu_1L_g^2)}$ and\r\n\\begin{align*}\r\nM :=\\min\\left\\{\\frac{(1 -\\alpha^2)\\eta}{2} -\\frac{L_{12}^2(1+\\alpha)^2\\eta}{2L_{11}(L_{22} +2\\rho\\|A\\|^2)}, \\frac{(2 -L_{11}\\beta\\sigma)\\beta\\gamma}{4}, \\frac{(2 -L_{11}\\sigma)\\gamma}{4}\\right\\}.\r\n\\end{align*}\r\n\\end{theorem}","units":[{"start":38730,"end":39694,"start_line":662,"end_line":671,"raw_tex":"\\begin{theorem}\r\nSuppose that Assumptions~\\ref{a:H} and \\ref{a:g} hold. Let $(\\bar{x}^k,\\bar{y}^k)$ be the sequence generated by Algorithm~\\ref{algo} with $\\mu_k =\\mu_1 k^{-\\min\\{\\theta, \\frac{1-\\theta}{2}\\}}$. Then, for $\\hat{x}^k=\\bar{x}^k$ and $\\hat{y}^k=\\bar{y}^k-A^*(AA^*)^{-1}(A\\bar{y}^k-\\prox_{\\mu g}(A\\bar{y}^k))$, it holds that\r\n\\begin{multline*}\r\n\\min_{1\\leq j\\leq k}(\\dist(-\\nabla_yH(\\hat{x}^j,\\hat{y}^j),A^*\\partial g(A\\hat{y}^j))+\\dist(0,G_{\\sigma_j}(\\hat{x}^j,\\hat{y}^j))) \\\\ \\leq (\\Theta +(L_{12}+L_{12})L_g\\sigma_{\\min}(A)^{-1}\\mu_1)k^{-\\min\\{\\theta, \\frac{1-\\theta}{2}\\}},\r\n\\end{multline*}\r\nwhere $\\Theta :=\\sqrt{2(\\ln 2)^{-1}M^{-1}(\\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}^*+\\mu_1L_g^2)}$ and\r\n\\begin{align*}\r\nM :=\\min\\left\\{\\frac{(1 -\\alpha^2)\\eta}{2} -\\frac{L_{12}^2(1+\\alpha)^2\\eta}{2L_{11}(L_{22} +2\\rho\\|A\\|^2)}, \\frac{(2 -L_{11}\\beta\\sigma)\\beta\\gamma}{4}, \\frac{(2 -L_{11}\\sigma)\\gamma}{4}\\right\\}.\r\n\\end{align*}\r\n\\end{theorem}","sha256_utf8":"0a5544927e26762bca9e0b9cc4ad2e4d2212a24cd6d29d1ed3eb75b345a005b4"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}[{\\cite[Theorem 2.64]{BC}}]\r\n\\label{l:descent}\r\nLet $F:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ be a continuously differentiable function whose gradient $\\nabla F$ is $L$-Lipschitz continuous with $L>0$. Then, for all $x,y\\in \\mathbb{R}^n$,\r\n\\begin{align*}\r\n|F(y)-F(x)-\\langle\\nabla F(x),y-x\\rangle|\\leq\\frac{L}{2}\\|y-x\\|^2.\r\n\\end{align*}\r\n\\end{lemma}\n\n\\begin{lemma}\r\n\\label{l:weakcvx}\r\nLet $F: \\mathbb{R}^n\\rightarrow (-\\infty,+\\infty]$ be a proper $\\rho$-weakly convex and lower semicontinuous function, and let $\\mu \\in(0,1/\\rho)$. Then\r\n\\begin{enumerate}\r\n\\item\r\n\\cite[Corollary 3.4]{HLO} The Moreau envelope function $F_{\\mu}$ is continuously differentiable on $\\mathbb{R}^n$ and, for all $x\\in \\mathbb{R}^n$,\r\n\\begin{align*}\r\n\\nabla F_{\\mu}(x) =\\frac{1}{\\mu}(x-\\prox_{\\mu F}(x)).\r\n\\end{align*}\r\nThis gradient is $\\max\\{\\frac{1}{\\mu},\\frac{\\rho}{1-\\rho \\mu}\\}$-Lipschitz continuous.\r\n\\item\\label{l:weakcvx_inclu}\r\n\\cite[Lemma 3.2]{BW} $\\nabla{F_{\\mu}(x)}\\in \\partial F(\\prox_{\\mu F}(x))$.\r\n\\end{enumerate}\r\n\\end{lemma}\n\n\\begin{lemma}[{\\cite[Lemma 3.3]{BW}}]\\label{2.6} Let $F: \\mathbb{R}^n\\rightarrow (-\\infty,\\infty)$ be a $\\rho$-weakly convex function and $L_F$-Lipschitz continuous, and let $\\mu \\in (0,1/\\rho)$. Then the Moreau envelope $F_\\mu$ is Lipschitz continuous and, for all $x\\in \\mathbb{R}^n$,\r\n\\begin{align*}\r\n\\|\\nabla F_\\mu(x)\\|\\leq L_F \\text{~~and~~} \\|x-\\prox_{\\mu F}(x)\\|\\leq \\mu L_F.\r\n\\end{align*}\r\n\\end{lemma}\n\n\\begin{lemma}[{\\cite[Lemma~\\ref{l:LL}]{BW}}]\r\n\\label{l:MoreauEnv}\r\nLet $F: \\mathbb{R}^n\\rightarrow (-\\infty,+\\infty]$ be a proper $\\rho$-weakly convex lower semicontinuous function, let $\\mu_1$ and $\\mu_2$ be parameters such that $0< \\mu_2 \\leq \\mu_1 < 1/\\rho$. Then\r\n\\begin{align*}\r\nF_{\\mu_2}(x)\\leq F_{\\mu_1}(x)+\\frac{\\mu_1(\\mu_1-\\mu_2)}{2\\mu_2}\\|\r\n\\nabla F_{\\mu_1}(x)\\|^2.\r\n\\end{align*}\r\nIf $F$ is additionally $L_F$-Lipschitz continuous, then\r\n\\begin{align*}\r\nF_{\\mu_2}\\leq F_{\\mu_1}+\\frac{\\mu_1(\\mu_1-\\mu_2)}{2\\mu_2}(L_F)^2.\r\n\\end{align*}\r\n\\end{lemma}\n\n\\begin{lemma}\\label{l:estimate}\r\nLet $x\\in [1, +\\infty)$ and $\\alpha\\in [0, 1]$. Then\r\n\\begin{align*}\r\n(1 +x)^{\\alpha} - 1 \\geq \\alpha (\\ln 2) x^{\\alpha}.\r\n\\end{align*}\r\n\\end{lemma}\n\n\\begin{proof}\r\nLet $\\varphi(x) :=(1 +x)^{\\alpha} -1 - \\alpha (\\ln 2) x^{\\alpha}$. Then $\\psi(\\alpha) =:\\varphi(1) =2^{\\alpha} -1 -\\alpha \\ln 2$. We see that, for all $\\alpha \\geq 0$,\r\n$\\psi'(\\alpha) =(\\ln 2) (2^{\\alpha} -1) \\geq 0$. It follows that $\\varphi(1) =\\psi(\\alpha) \\geq \\psi(0) =0$.\r\n\r\nOn the other hand,\r\n\\begin{align*}\r\n\\varphi'(x) =\\alpha(1 +x)^{\\alpha -1} -\\alpha^2 (\\ln 2) x^{\\alpha -1} =\\alpha x^{\\alpha -1}\\left(\\left(1 +\\frac{1}{x}\\right)^{\\alpha -1} -\\alpha \\ln 2\\right).\r\n\\end{align*}\r\nAs $x \\geq 1$ and $\\alpha -1 \\leq 0$, it holds that\r\n\\begin{align*}\r\n\\left(1 +\\frac{1}{x}\\right)^{\\alpha -1} -\\alpha \\ln 2 &\\geq (1 + 1)^{\\alpha -1} -\\alpha \\ln 2 =2^{\\alpha} -2^{\\alpha -1} -\\alpha \\ln 2 \\\\\r\n&\\geq 2^{\\alpha} -1 -\\alpha \\ln 2 =\\varphi(1) \\geq 0.\r\n\\end{align*}\r\nTherefore, for all $x\\geq 1$, $\\varphi'(x) \\geq 0$, which implies that $\\varphi(x) \\geq \\varphi(1) =0$. The proof is complete.\r\n\\end{proof}\n\n\\begin{remark}\r\n\\label{r:G(x,.)}\r\nFrom the Lipschitz continuity of $\\nabla_x H(x,\\cdot)$ in Assumption~\\ref{a:H}, we have that for all $x\\in \\mathbb{R}^n$ and all $y_1,y_2\\in \\mathbb{R}^m$,\r\n\\begin{align*}\r\n\\|G_\\lambda(x,y_1)&-G_\\lambda(x,y_2)\\| \\notag\\\\\r\n&=\\frac{1}{\\lambda}\\|\\prox_{\\lambda f}(x-\\lambda \\nabla_x H(x,y_2))-\\prox_{\\lambda f}(x-\\lambda \\nabla_x H(x,y_1))\\| \\\\\r\n&\\leq\\|\\nabla_x H(x,y_1)-\\nabla_x H(x,y_2)\\| \\\\\r\n&\\leq L_{12}\\|y_1-y_2\\|.\r\n\\end{align*}\r\n\\end{remark}\r\n\n\n\\begin{lemma}\r\n\\label{l:transfer}\r\nSuppose that Assumptions \\ref{a:H} and \\ref{a:g} hold, $A$ is surjective, and $\\mu\\in(0,1/\\rho)$.\r\nLet $\\bar{x} =x^*\\in \\mathbb{R}^n$ and $\\bar{y}=y^*-A^*(AA^*)^{-1}(Ay^*-\\prox_{\\mu g}(Ay^*))$ with $y^*\\in \\mathbb{R}^m$. Then\r\n\\begin{align*}\r\n&\\dist(0,G_\\lambda(\\bar{x},\\bar{y}))+\\dist(-\\nabla_y H(\\bar{x},\\bar{y}),A^*\\partial g(A\\bar{y})) \\\\\r\n&\\leq \\dist(0,G_\\lambda(x^*,y^*)) +\\dist(-\\nabla_yH(x^*,y^*),A^*\\partial g(\\prox_{\\mu g}(Ay^*)))\\\\\r\n&\\quad +(L_{12}+L_{22})L_g\\sigma_{\\min}(A)^{-1}\\mu.\r\n\\end{align*}\r\nConsequently, if $(x^*,y^*)$ is an $\\varepsilon$-approximate stationary point of problem~\\eqref{guanghua}, then $(\\bar{x},\\bar{y})$ is an $\\bar{\\varepsilon}$-approximate stationary point of problem~\\eqref{eq:L1E1} with $\\bar{\\varepsilon} =\\varepsilon +(L_{12}+L_{22})L_g\\sigma_{\\min}(A)^{-1}\\mu$.\r\n\\end{lemma}\n\n\\begin{proof}\r\nSince $\\bar{y}=y^*-A^*(AA^*)^{-1}(Ay^*-\\prox_{\\mu g}(Ay^*))$, multiplying both sides by $A$ yields $A\\bar{y}=\\prox_{\\mu g}(Ay^*)$. We have that\r\n\\begin{align*}\r\n&\\dist(-\\nabla_yH(\\bar{x},\\bar{y}),A^*\\partial g(A\\bar{y})) \\\\\r\n&\\leq \\dist(-\\nabla_yH(x^*,y^*),A^*\\partial g(A\\bar{y})) +\\|\\nabla_yH(x^*,y^*)-\\nabla_yH(\\bar{x},\\bar{y})\\| \\\\\r\n&=\\dist(-\\nabla_yH(x^*,y^*),A^*\\partial g(\\prox_{\\mu g}(Ay^*))) +\\|\\nabla_yH(x^*,y^*)-\\nabla_yH(x^*,\\bar{y})\\| \\\\\r\n&\\leq \\dist(-\\nabla_yH(x^*,y^*),A^*\\nabla g_{\\mu}(Ay^*)) +L_{22}\\|y^* -\\bar{y}\\|,\r\n\\end{align*}\r\nwhere the last inequality uses $A^*\\nabla g_{\\mu}(Ay^*)\\in A^*\\partial g(\\prox_{\\mu g}(Ay^*))$ (see Lemma~\\ref{l:weakcvx}\\ref{l:weakcvx_inclu}) and the Lipschitz continuity of $\\nabla_y H(x^*,\\cdot)$.\r\n\r\nNext, by Remark~\\ref{r:G(x,.)},\r\n\\begin{align*}\r\n\\dist(0,G_\\lambda(\\bar{x},\\bar{y})) &= \\|G_\\lambda(x^*,y^*) -G_\\lambda(x^*,y^*) +G_\\lambda(\\bar{x},\\bar{y})\\| \\\\\r\n&\\leq \\|G_\\lambda(x^*,y^*)\\| +\\|G_\\lambda(x^*,y^*) -G_\\lambda(\\bar{x},\\bar{y})\\| \\\\\r\n&= \\dist(0,G_\\lambda (x^*,y^*)) +\\|G_\\lambda(x^*,y^*) -G_\\lambda(x^*,\\bar{y})\\| \\\\\r\n&\\leq \\dist(0,G_\\lambda (x^*,y^*)) +L_{12}\\|y^* -\\bar{y}\\|.\r\n\\end{align*}\r\nNote that\r\n\\begin{align*}\r\n\\|y^* -\\bar{y}\\| =\\|A^*(AA^*)^{-1}(Ay^*-\\prox_{\\mu g}(Ay^*))\\| \\leq \\sigma_{\\min}(A)^{-1}\\mu L_g\r\n\\end{align*}\r\ndue to Lemma~\\ref{2.6} and the fact that the operator norm of $A^*(AA^*)^{-1}$ is bounded by the inverse of the smallest singular value $\\sigma_{\\min}(A)$ of $A$, i.e., $\\|A^*(AA^*)^{-1}\\|\\leq\\sigma_{\\min}(A)^{-1}$. Altogether, we obtain that\r\n\\begin{align*}\r\n&\\dist(0,G_\\lambda(\\bar{x},\\bar{y})) +\\dist(-\\nabla_y H(\\bar{x},\\bar{y}),A^*\\partial g(A\\bar{y})) \\\\\r\n&\\leq   \\dist(0,G_\\lambda(x^*,y^*)) +\\dist(-\\nabla_yH(x^*,y^*),A^*\\partial g(\\prox_{\\mu g}(Ay^*))) \\\\\r\n&\\quad+(L_{12} +L_{22})L_g\\sigma_{\\min}(A)^{-1}\\mu,\r\n\\end{align*}\r\nwhich completes the proof.\r\n\\end{proof}\n\nRecalling that $G_\\lambda(x,y) =\\frac{1}{\\lambda}(x -\\prox_{\\lambda f}(x -\\lambda\\nabla_x  H(x,y)))$, we have\r\n\\begin{align}\\label{zhu4.2}\r\nx^{k+1}&=\\prox_{\\sigma_k f}(\\bar{x}^k-\\sigma_k\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1}))\\notag \\\\\r\n&=\\bar{x}^k-\\sigma_k\\left(\\frac{1}{\\sigma_k}(\\bar{x}^k-\\prox_{\\sigma_k f}(\\bar{x}^k-\\sigma_k\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1})))\\right)\\notag \\\\\r\n&=\\bar{x}^k-\\sigma_k G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1}).\r\n\\end{align}\r\n\n\n\\begin{lemma}\r\n\\label{l:LL}\r\nSuppose that Assumptions \\ref{a:H} and \\ref{a:g} hold. Let $(\\bar{x}^k,\\bar{y}^k)$ be the sequence generated by Algorithm~\\ref{algo}. Then\r\n\\begin{align*}\r\n\\mathcal{L}_{\\mu_{k}}(\\bar{x}^{k+1},\\bar{y}^{k+1}) &\\leq \\mathcal{L}_{\\mu_{k}}(\\bar{x}^k,\\bar{y}^k)\r\n-\\frac{1}{2}\\delta\\tau_k\\|A^*\\nabla g_{\\mu_k}(A\\bar{y}^k)+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)\\|^2\\notag \\\\\r\n&\\quad -\\frac{1}{4}\\kappa\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)\\|^2 +(\\mu_k-\\mu_{k+1})L_g^2,\r\n\\end{align*}\r\nwhere $\\delta :=1 -\\alpha^2 -\\frac{L_{12}^2(1+\\alpha)^2}{L_{11}(L_{22} +2\\rho\\|A\\|^2)} >0$ and $\\kappa :=\\min\\{(2 -L_{11}\\beta\\sigma)\\beta, 2 -L_{11}\\sigma\\} >0$.\r\n\\end{lemma}\n\n\\begin{proof}\r\nFrom the Lipschitz continuity of $\\nabla_x \\Phi_k(\\cdot,\\bar{y}^{k+1})$ and $\\nabla_y\\Phi_k(\\bar{x}^k, \\cdot)$, Lemma \\ref{l:descent} implies that\r\n\\begin{align*}\r\n\\Phi_k(\\bar{x}^{k+1},\\bar{y}^{k+1}) &\\leq\\Phi_k(\\bar{x}^{k},\\bar{y}^{k+1}) +\\langle\\nabla_x\\Phi_k(\\bar{x}^k,\\bar{y}^{k+1}),\\bar{x}^{k+1}-\\bar{x}^k\\rangle+\\frac{L_{11}}{2}\\|\\bar{x}^{k+1}-\\bar{x}^k\\|^2, \\\\\r\n\\Phi_k(\\bar{x}^k,\\bar{y}^{k+1}) &\\leq \\Phi_k(\\bar{x}^k,\\bar{y}^k) +\\langle\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k),\\bar{y}^{k+1}-\\bar{y}^k\\rangle+\\frac{L_k}{2}\\|\\bar{y}^{k+1}-\\bar{y}^k\\|^2.\r\n\\end{align*}\r\nWe note that\r\n\\begin{align}\\label{eq:grad_y Phi_k}\r\n\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k) =A^*\\nabla g_{\\mu_k}(A\\bar{y}^k)+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)\r\n\\end{align}\r\nand from \\eqref{s21}, \\eqref{s41}, \\eqref{s42}, and \\eqref{zhu4.2} that\r\n\\begin{align*}\r\n\\bar{x}^{k+1}-\\bar{x}^k &=\\beta_k(x^{k+1}-\\bar{x}^k)=-\\beta_k\\sigma_k G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1}), \\\\\r\n\\bar{y}^{k+1}-\\bar{y}^k &=(1+\\alpha_k)(y^{k+1}-\\bar{y}^k) =-(1+\\alpha_k)\\tau_k \\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k).\r\n\\end{align*}\r\nTherefore,\r\n\\begin{align}\\label{8.7}\r\n&\\Phi_k(\\bar{x}^{k+1},\\bar{y}^{k+1}) \\notag \\\\\r\n&\\leq \\Phi_k(\\bar{x}^k,\\bar{y}^k) +\\langle\\nabla_x\\Phi_k(\\bar{x}^k,\\bar{y}^{k+1}),\\bar{x}^{k+1}-\\bar{x}^k\\rangle+\\frac{L_{11}}{2}\\|\\bar{x}^{k+1}-\\bar{x}^k\\|^2 \\notag \\\\\r\n&\\quad +\\langle\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k),\\bar{y}^{k+1}-\\bar{y}^k\\rangle+\\frac{L_k}{2}\\|\\bar{y}^{k+1}-\\bar{y}^k\\|^2 \\notag \\\\\r\n&= \\Phi_k(\\bar{x}^k,\\bar{y}^k) -\\beta_k\\sigma_k\\langle\\nabla_x\\Phi_k(\\bar{x}^k,\\bar{y}^{k+1}),G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\rangle \\notag \\\\\r\n&\\quad +\\frac{1}{2}L_{11}\\beta_k^2\\sigma_k^2\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 \\notag\\\\\r\n&\\quad -\\frac{1}{2}(2 +2\\alpha_k -L_k(1+\\alpha_k)^2\\tau_k)\\tau_k\\|\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k)\\|^2.\r\n\\end{align}\r\nBy Lemma \\ref{l:MoreauEnv} and the fact that $\\mu_{k+1}\\in [\\mu_k/2, \\mu_k]$,\r\n\\begin{align}\\label{8.8}\r\ng_{\\mu_{k+1}}(Ay)\\leq g_{\\mu_k}(Ay) +\\frac{\\mu_k(\\mu_k-\\mu_{k+1})}{2\\mu_{k+1}}L_g^2\\leq g_{\\mu_k}(Ay) +(\\mu_k-\\mu_{k+1})L_g^2.\r\n\\end{align}\r\nAdding $H(x,y)$ to both sides of \\eqref{8.8} yields\r\n\\begin{align}\\label{3.4}\r\n\\Phi_{k+1}(x,y)\\leq \\Phi_k(x,y) +(\\mu_k-\\mu_{k+1})L_g^2.\r\n\\end{align}\r\nLetting $x =\\bar{x}^{k+1}$, $y =\\bar{y}^{k+1}$ in \\eqref{3.4} and combining it with \\eqref{8.7}, we obtain that\r\n\\begin{align}\\label{4.4}\r\n&\\Phi_{k+1}(\\bar{x}^{k+1},\\bar{y}^{k+1}) \\notag \\\\\r\n&\\leq \\Phi_k(\\bar{x}^k,\\bar{y}^k) -\\beta_k\\sigma_k\\langle\\nabla_x\\Phi_k(\\bar{x}^k,\\bar{y}^{k+1}),G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\rangle \\notag \\\\\r\n&\\quad +\\frac{1}{2}L_{11}\\beta_k^2\\sigma_k^2\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 \\notag\\\\\r\n&\\quad -\\frac{1}{2}(2 +2\\alpha_k -L_k(1+\\alpha_k)^2\\tau_k)\\tau_k\\|\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k)\\|^2 +(\\mu_k-\\mu_{k+1})L_g^2.\r\n\\end{align}\r\nBy the first-order optimality condition of \\eqref{s31},\r\n\\begin{align*}\r\n0\\in \\partial f(x^{k+1})+\\frac{1}{\\sigma_k}(x^{k+1}-\\bar{x}^k+\\sigma_k\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1})),\r\n\\end{align*}\r\nor equivalently,\r\n\\begin{align*}\r\n\\frac{1}{\\sigma_k}(\\bar{x}^k -x^{k+1}) -\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1})\\in \\partial f(x^{k+1}).\r\n\\end{align*}\r\nCombining with the convexity of $f$ and \\eqref{zhu4.2}, it follows that\r\n\\begin{align}\\label{8.9}\r\nf(x^{k+1}) &\\leq f(\\bar{x}^k) -\\left\\langle \\frac{1}{\\sigma_k}(\\bar{x}^k -x^{k+1}) -\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1}), \\bar{x}^k -x^{k+1}\\right\\rangle \\notag \\\\\r\n&= f(\\bar{x}^k) -\\frac{1}{\\sigma_k}\\|\\bar{x}^k -x^{k+1}\\|^2 +\\langle \\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1}), \\bar{x}^k -x^{k+1}\\rangle\\notag \\\\\r\n&= f(\\bar{x}^k) -\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 +\\sigma_k\\langle\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1}),G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\rangle.\r\n\\end{align}\r\nAgain by the convexity of $f$ and \\eqref{s42},\r\n\\begin{align*}\r\nf(\\bar{x}^{k+1})&=f\\big(\\beta_k x^{k+1}+(1-\\beta_k)\\bar{x}^k\\big)\r\n\\leq \\beta_k f(x^{k+1})+(1-\\beta_k)f(\\bar{x}^k).\r\n\\end{align*}\r\nThis together with \\eqref{8.9} gives\r\n\\begin{align}\\label{4.5}\r\nf(\\bar{x}^{k+1})&\\leq f(\\bar{x}^k) -\\beta_k\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 \\notag \\\\\r\n&\\quad +\\beta_k\\sigma_k\\langle\\nabla_xH(\\bar{x}^k,\\bar{y}^{k+1}),G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\rangle.\r\n\\end{align}\r\nWe now combine \\eqref{4.4} and \\eqref{4.5} to obtain\r\n\\begin{align}\\label{4.10}\r\n&\\mathcal{L}_{\\mu_{k}}(\\bar{x}^{k+1},\\bar{y}^{k+1}) \\notag \\\\\r\n&\\leq \\mathcal{L}_{\\mu_{k}}(\\bar{x}^k,\\bar{y}^k) -\\frac{1}{2}(2 +2\\alpha_k -L_k(1+\\alpha_k)^2\\tau_k)\\tau_k\\|\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k)\\|^2 \\notag \\\\ &\\quad -\\frac{1}{2}(2 -L_{11}\\beta_k\\sigma_k)\\beta_k\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 +(\\mu_k-\\mu_{k+1})L_g^2.\r\n\\end{align}\r\nNext, using Remark~\\ref{r:G(x,.)},\r\n\\begin{align*}\r\n\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)\\|&=\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)-G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})+G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|\\notag \\\\\r\n&\\leq\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)-G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|+\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|\\notag \\\\\r\n&\\leq L_{12}\\|\\bar{y}^{k+1}-\\bar{y}^k\\|+\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|.\r\n\\end{align*}\r\nSquaring and applying the inequality $\\frac{1}{2}(a+b)^2 \\leq a^2 +b^2$, we get\r\n\\begin{align*}\r\n\\frac{1}{2}\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)\\|^2 &\\leq L_{12}^2\\|\\bar{y}^{k+1}-\\bar{y}^k\\|^2 +\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 \\\\\r\n&= L_{12}^2(1+\\alpha_k)^2\\tau_k^2\\|\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k)\\|^2 +\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2.\r\n\\end{align*}\r\nwhich, combined with \\eqref{eq:grad_y Phi_k} and \\eqref{4.10}, implies that\r\n\\begin{align*}\r\n\\mathcal{L}_{\\mu_{k}}(\\bar{x}^{k+1},\\bar{y}^{k+1}) &\\leq \\mathcal{L}_{\\mu_{k}}(\\bar{x}^k,\\bar{y}^k) -\\frac{1}{2}\\delta_k\\tau_k\\|A^*\\nabla g_{\\mu_k}(A\\bar{y}^k)+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)\\|^2 \\notag \\\\\r\n&\\quad -\\frac{1}{4}\\kappa_k\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 +(\\mu_k-\\mu_{k+1})L_g^2,\r\n\\end{align*}\r\nwhere\r\n\\begin{align*}\r\n\\delta_k &:=2 +2\\alpha_k -L_k(1+\\alpha_k)^2\\tau_k -(2 -L_{11}\\beta_k\\sigma_k)\\sigma_k\\beta_kL_{12}^2(1+\\alpha_k)^2\\tau_k, \\\\\r\n\\kappa_k &:=(2 -L_{11}\\beta_k\\sigma_k)\\beta_k.\r\n\\end{align*}\r\nNoting that $\\tau_k\\leq 1/L_k$ and that\r\n\\begin{align*}\r\n(2 -L_{11}\\beta_k\\sigma_k)\\sigma_k\\beta_k\\leq \\frac{1}{L_{11}}\\left(\\frac{2 -L_{11}\\beta_k\\sigma_k +L_{11}\\beta_k\\sigma_k}{2}\\right)^2 =\\frac{1}{L_{11}},\r\n\\end{align*}\r\nwe have\r\n\\begin{align*}\r\n\\delta_k &\\geq 2 +2\\alpha_k -(1+\\alpha_k)^2 -\\frac{L_{12}^2}{L_{11}L_k}(1+\\alpha_k)^2 \\\\\r\n&= 1 -\\alpha_k^2 -\\frac{L_{12}^2}{L_{11}L_k}(1+\\alpha_k)^2 \\\\\r\n&\\geq \\delta =1 -\\alpha^2 -\\frac{L_{12}^2(1+\\alpha)^2}{L_{11}(L_{22} +2\\rho\\|A\\|^2)},\r\n\\end{align*}\r\nwhere the last inequality follows from from the fact that $\\alpha_k\\in [-\\alpha, \\alpha]\\subseteq [-1, \\alpha]$ and that $L_k =L_{22} +\\|A\\|^2\\max\\{\\frac{1}{\\mu_k}, \\frac{\\rho}{1 -\\rho\\mu_k}\\}\\geq L_{22} +2\\rho\\|A\\|^2$ since\r\n\\begin{align*}\r\n\\max\\left\\{\\frac{1}{\\mu_k}, \\frac{\\rho}{1 -\\rho\\mu_k}\\right\\} =\\begin{cases}\r\n\\frac{1}{\\mu_k} &\\text{if~} 0 < \\mu_k \\leq \\frac{1}{2\\rho}, \\\\\r\n\\frac{\\rho}{1 -\\rho\\mu_k} &\\text{if~} \\frac{1}{2\\rho} < \\mu_k < \\frac{1}{\\rho}.\r\n\\end{cases}\r\n\\end{align*}\r\nFinally, we derive from $\\sigma_k\\in \\sigma$ and $\\beta_k\\in [\\beta, 1]$ that\r\n\\begin{align*}\r\n\\kappa_k =(2 -L_{11}\\beta_k\\sigma_k)\\beta_k \\geq (2 -L_{11}\\beta_k\\sigma)\\beta_k \\geq \\kappa =\\min\\{(2 -L_{11}\\beta\\sigma)\\beta, 2 -L_{11}\\sigma\\}\r\n\\end{align*}\r\nas $(2 -L_{11}\\beta_k\\sigma)\\beta_k$ is a quadratic function of $\\beta_k$ with leading coefficient $-L_{11}\\sigma <0$. This completes the proof.\r\n\\end{proof}\n\n\\begin{lemma}\r\n\\label{l:e-stationary}\r\nSuppose that Assumptions~\\ref{a:H} and \\ref{a:g} hold. Let $(\\bar{x}^k,\\bar{y}^k)$ be the sequence generated by Algorithm~\\ref{algo}. Then\r\n\\begin{align*}\r\n\\min_{1\\leq j\\leq k}\\left(\\|A^*\\nabla g_{\\mu_j}(A\\bar{y}^j)+\\nabla_y H(\\bar{x}^j,\\bar{y}^j)\\|+\\|G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j)\\|\\right)\r\n\\leq \\Theta k^{\\frac{\\theta-1}{2}},\r\n\\end{align*}\r\nwhere $\\Theta :=\\sqrt{2(\\ln 2)^{-1}M^{-1}(\\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}^*+\\mu_1L_g^2)}$ and\r\n\\begin{align*}\r\nM :=\\min\\left\\{\\frac{(1 -\\alpha^2)\\eta}{2} -\\frac{L_{12}^2(1+\\alpha)^2\\eta}{2L_{11}(L_{22} +2\\rho\\|A\\|^2)}, \\frac{(2 -L_{11}\\beta\\sigma)\\beta\\gamma}{4}, \\frac{(2 -L_{11}\\sigma)\\gamma}{4}\\right\\}.\r\n\\end{align*}\r\n\\end{lemma}\n\n\\begin{proof}\r\nAccording to Lemma~\\ref{l:LL},\r\n\\begin{align}\\label{4.10'}\r\n\\mathcal{L}_{\\mu_{k}}(\\bar{x}^{k+1},\\bar{y}^{k+1}) &\\leq \\mathcal{L}_{\\mu_{k}}(\\bar{x}^k,\\bar{y}^k) -\\frac{1}{2}\\delta\\tau_k\\|A^*\\nabla g_{\\mu_k}(A\\bar{y}^k)+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)\\|^2\\notag \\\\\r\n&\\quad -\\frac{1}{4}\\kappa\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)\\|^2 +(\\mu_k-\\mu_{k+1})L_g^2.\r\n\\end{align}\r\nSumming \\eqref{4.10'} from $k=1$ to $k=K$, we have\r\n\\begin{align}\\label{4.11}\r\n&\\sum_{k=1}^K \\left(\\frac{1}{2}\\delta\\tau_k\\|A^*\\nabla g_{\\mu_k}(A\\bar{y}^k)+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)\\|^2 +\\frac{1}{4}\\kappa\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)\\|^2\\right) \\notag \\\\\r\n&\\leq \\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}_{\\mu_{K+1}}(\\bar{x}^{K+1},\\bar{y}^{K+1})+(\\mu_1-\\mu_{K+1})L_g^2\\notag \\\\\r\n&\\leq \\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}^*+\\mu_1L_g^2.\r\n\\end{align}\r\nSince $\\sigma_k \\geq\\gamma k^{-\\theta}$, $\\tau_k\\geq\\eta k^{-\\theta}$, and $M =\\min\\{\\frac{1}{2}\\delta\\eta, \\frac{1}{4}\\kappa\\gamma\\}$, it follows from \\eqref{4.11} that\r\n\\begin{multline*}\r\nM\\left(\\sum_{k=1}^{K}k^{-\\theta}\\right) \\min_{1\\leq j\\leq K}(\\|A^*\\nabla g_{\\mu_j}(A\\bar{y}^j)+\\nabla_yH(\\bar{x}^j,\\bar{y}^j)\\|^2+\r\n\\|G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j)\\|^2) \\\\ \\leq\\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)\r\n-\\mathcal{L}^*+\\mu_1L_g^2,\r\n\\end{multline*}\r\nNote that\r\n\\begin{align*}\r\n\\sum_{k=1}^Kk^{-\\theta}&\\geq \\sum_{k=1}^K\\int_k^{k+1}x^{-\\theta}dx=\\int_1^{K+1}x^{-\\theta}dx=\\frac{1}{1-\\theta}((1+K)^{1-\\theta}-1)\\\\\r\n&\\geq(\\ln 2)K^{1-\\theta},\r\n\\end{align*}\r\nwhere the last inequality is obtained by applying Lemma \\ref{l:estimate} with $x =K$ and $\\alpha =1 -\\theta$. We deduce that\r\n\\begin{multline*}\r\n\\min_{1\\leq j\\leq K}(\\|A^*\\nabla g_{\\mu_j}(A\\bar{y}^j)+\\nabla_yH(\\bar{x}^j,\\bar{y}^j)\\|^2+\\|G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j)\\|^2) \\\\ \\leq (\\ln 2)^{-1}M^{-1}(\\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}^*+\\mu_1L_g^2)k^{\\theta-1}.\r\n\\end{multline*}\r\nUsing the inequality $(\\|a\\|+\\|b\\|)^2\\leq2(\\|a\\|^2+\\|b\\|^2)$, we have\r\n\\begin{multline*}\r\n\\min_{1\\leq j\\leq k}(\\|A^*\\nabla g_{\\mu_j}(A\\bar{y}^j)+\\nabla_yH(\\bar{x}^j,\\bar{y}^j)\\|+\\|G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j)\\|) \\\\ \\leq \\sqrt{2(\\ln 2)^{-1}M^{-1}(\\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}^*+\\mu_1L_g^2)} k^{\\frac{\\theta-1}{2}}.\r\n\\end{multline*}\r\nThe proof is complete.\r\n\\end{proof}\n\n\\begin{proof}\r\nUsing Lemmas~\\ref{l:e-stationary} and \\ref{l:weakcvx}, we obtain\r\n\\begin{align*}\r\n&\\min_{1\\leq j\\leq k}(\\dist(-\\nabla_yH(\\bar{x}^j,\\bar{y}^j),A^*\\partial g(\\prox_{\\mu_j g}(A\\bar{y}^j))+\\dist(0,G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j))) \\\\\r\n&\\leq\\min_{1\\leq j\\leq k}(\\|A^*\\nabla g_{\\mu_j}(A\\bar{y}^j)+\\nabla_yH(\\bar{x}^j,\\bar{y}^j)\\|+\\|G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j)\\|)\\leq \\Theta k^{\\frac{\\theta-1}{2}},\r\n\\end{align*}\r\nwhich together with Lemma~\\ref{l:transfer} implies that\r\n\\begin{align*}\r\n&\\min_{1\\leq j\\leq k}(\\dist(-\\nabla_yH(\\hat{x}^j,\\hat{y}^j),A^*\\partial g(A\\hat{y}^j))+\\dist(0,G_{\\sigma_j}(\\hat{x}^j,\\hat{y}^j))) \\\\\r\n&\\leq \\Theta k^{\\frac{\\theta-1}{2}} +(L_{12}+L_{12})L_g\\sigma_{\\min}(A)^{-1}\\mu_k \\\\\r\n&= \\Theta k^{-\\frac{1-\\theta}{2}} +(L_{12}+L_{12})L_g\\sigma_{\\min}(A)^{-1}\\mu_1 k^{-\\min\\{\\theta, \\frac{1-\\theta}{2}\\}} \\\\\r\n&\\leq (\\Theta +(L_{12}+L_{12})L_g\\sigma_{\\min}(A)^{-1}\\mu_1)k^{-\\min\\{\\theta, \\frac{1-\\theta}{2}\\}}.\r\n\\end{align*}\r\nThe proof is complete.\r\n\\end{proof}","units":[{"start":11987,"end":12346,"start_line":186,"end_line":192,"raw_tex":"\\begin{lemma}[{\\cite[Theorem 2.64]{BC}}]\r\n\\label{l:descent}\r\nLet $F:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ be a continuously differentiable function whose gradient $\\nabla F$ is $L$-Lipschitz continuous with $L>0$. Then, for all $x,y\\in \\mathbb{R}^n$,\r\n\\begin{align*}\r\n|F(y)-F(x)-\\langle\\nabla F(x),y-x\\rangle|\\leq\\frac{L}{2}\\|y-x\\|^2.\r\n\\end{align*}\r\n\\end{lemma}","sha256_utf8":"37900e490b090f7e31d9716a56ff1feeee164582423be267ed38e45ed80f92ce"},{"start":13017,"end":13687,"start_line":208,"end_line":221,"raw_tex":"\\begin{lemma}\r\n\\label{l:weakcvx}\r\nLet $F: \\mathbb{R}^n\\rightarrow (-\\infty,+\\infty]$ be a proper $\\rho$-weakly convex and lower semicontinuous function, and let $\\mu \\in(0,1/\\rho)$. Then\r\n\\begin{enumerate}\r\n\\item\r\n\\cite[Corollary 3.4]{HLO} The Moreau envelope function $F_{\\mu}$ is continuously differentiable on $\\mathbb{R}^n$ and, for all $x\\in \\mathbb{R}^n$,\r\n\\begin{align*}\r\n\\nabla F_{\\mu}(x) =\\frac{1}{\\mu}(x-\\prox_{\\mu F}(x)).\r\n\\end{align*}\r\nThis gradient is $\\max\\{\\frac{1}{\\mu},\\frac{\\rho}{1-\\rho \\mu}\\}$-Lipschitz continuous.\r\n\\item\\label{l:weakcvx_inclu}\r\n\\cite[Lemma 3.2]{BW} $\\nabla{F_{\\mu}(x)}\\in \\partial F(\\prox_{\\mu F}(x))$.\r\n\\end{enumerate}\r\n\\end{lemma}","sha256_utf8":"3e0cbfb074511e3d3201a2b193f7403a37888914ef5a0eb4dfa95ec7df51124b"},{"start":13695,"end":14104,"start_line":225,"end_line":229,"raw_tex":"\\begin{lemma}[{\\cite[Lemma 3.3]{BW}}]\\label{2.6} Let $F: \\mathbb{R}^n\\rightarrow (-\\infty,\\infty)$ be a $\\rho$-weakly convex function and $L_F$-Lipschitz continuous, and let $\\mu \\in (0,1/\\rho)$. Then the Moreau envelope $F_\\mu$ is Lipschitz continuous and, for all $x\\in \\mathbb{R}^n$,\r\n\\begin{align*}\r\n\\|\\nabla F_\\mu(x)\\|\\leq L_F \\text{~~and~~} \\|x-\\prox_{\\mu F}(x)\\|\\leq \\mu L_F.\r\n\\end{align*}\r\n\\end{lemma}","sha256_utf8":"8793e469809a1c09056f810a7c621077a0e41c1d70fb3a6d48573a20536c8f06"},{"start":14108,"end":14664,"start_line":231,"end_line":242,"raw_tex":"\\begin{lemma}[{\\cite[Lemma~\\ref{l:LL}]{BW}}]\r\n\\label{l:MoreauEnv}\r\nLet $F: \\mathbb{R}^n\\rightarrow (-\\infty,+\\infty]$ be a proper $\\rho$-weakly convex lower semicontinuous function, let $\\mu_1$ and $\\mu_2$ be parameters such that $0< \\mu_2 \\leq \\mu_1 < 1/\\rho$. Then\r\n\\begin{align*}\r\nF_{\\mu_2}(x)\\leq F_{\\mu_1}(x)+\\frac{\\mu_1(\\mu_1-\\mu_2)}{2\\mu_2}\\|\r\n\\nabla F_{\\mu_1}(x)\\|^2.\r\n\\end{align*}\r\nIf $F$ is additionally $L_F$-Lipschitz continuous, then\r\n\\begin{align*}\r\nF_{\\mu_2}\\leq F_{\\mu_1}+\\frac{\\mu_1(\\mu_1-\\mu_2)}{2\\mu_2}(L_F)^2.\r\n\\end{align*}\r\n\\end{lemma}","sha256_utf8":"b14927720c253a121d05cc700f7854400e9c615525c8f27914a544a3e197753a"},{"start":14725,"end":14906,"start_line":246,"end_line":251,"raw_tex":"\\begin{lemma}\\label{l:estimate}\r\nLet $x\\in [1, +\\infty)$ and $\\alpha\\in [0, 1]$. Then\r\n\\begin{align*}\r\n(1 +x)^{\\alpha} - 1 \\geq \\alpha (\\ln 2) x^{\\alpha}.\r\n\\end{align*}\r\n\\end{lemma}","sha256_utf8":"98a8450605b34c9945448d809c1eee2f91ddf268894bf8306f9d418c426a5132"},{"start":14908,"end":15830,"start_line":252,"end_line":266,"raw_tex":"\\begin{proof}\r\nLet $\\varphi(x) :=(1 +x)^{\\alpha} -1 - \\alpha (\\ln 2) x^{\\alpha}$. Then $\\psi(\\alpha) =:\\varphi(1) =2^{\\alpha} -1 -\\alpha \\ln 2$. We see that, for all $\\alpha \\geq 0$,\r\n$\\psi'(\\alpha) =(\\ln 2) (2^{\\alpha} -1) \\geq 0$. It follows that $\\varphi(1) =\\psi(\\alpha) \\geq \\psi(0) =0$.\r\n\r\nOn the other hand,\r\n\\begin{align*}\r\n\\varphi'(x) =\\alpha(1 +x)^{\\alpha -1} -\\alpha^2 (\\ln 2) x^{\\alpha -1} =\\alpha x^{\\alpha -1}\\left(\\left(1 +\\frac{1}{x}\\right)^{\\alpha -1} -\\alpha \\ln 2\\right).\r\n\\end{align*}\r\nAs $x \\geq 1$ and $\\alpha -1 \\leq 0$, it holds that\r\n\\begin{align*}\r\n\\left(1 +\\frac{1}{x}\\right)^{\\alpha -1} -\\alpha \\ln 2 &\\geq (1 + 1)^{\\alpha -1} -\\alpha \\ln 2 =2^{\\alpha} -2^{\\alpha -1} -\\alpha \\ln 2 \\\\\r\n&\\geq 2^{\\alpha} -1 -\\alpha \\ln 2 =\\varphi(1) \\geq 0.\r\n\\end{align*}\r\nTherefore, for all $x\\geq 1$, $\\varphi'(x) \\geq 0$, which implies that $\\varphi(x) \\geq \\varphi(1) =0$. The proof is complete.\r\n\\end{proof}","sha256_utf8":"eb28c13922efddb7314d9fb353024e875c8de0fa6e7aa9431c1a3f8a0a531229"},{"start":19739,"end":20219,"start_line":362,"end_line":371,"raw_tex":"\\begin{remark}\r\n\\label{r:G(x,.)}\r\nFrom the Lipschitz continuity of $\\nabla_x H(x,\\cdot)$ in Assumption~\\ref{a:H}, we have that for all $x\\in \\mathbb{R}^n$ and all $y_1,y_2\\in \\mathbb{R}^m$,\r\n\\begin{align*}\r\n\\|G_\\lambda(x,y_1)&-G_\\lambda(x,y_2)\\| \\notag\\\\\r\n&=\\frac{1}{\\lambda}\\|\\prox_{\\lambda f}(x-\\lambda \\nabla_x H(x,y_2))-\\prox_{\\lambda f}(x-\\lambda \\nabla_x H(x,y_1))\\| \\\\\r\n&\\leq\\|\\nabla_x H(x,y_1)-\\nabla_x H(x,y_2)\\| \\\\\r\n&\\leq L_{12}\\|y_1-y_2\\|.\r\n\\end{align*}\r\n\\end{remark}\r\n","sha256_utf8":"cf5577965543126540746cfa28208ba00ff7ba7544a2a29613b87bff990d86b4"},{"start":20223,"end":21078,"start_line":374,"end_line":384,"raw_tex":"\\begin{lemma}\r\n\\label{l:transfer}\r\nSuppose that Assumptions \\ref{a:H} and \\ref{a:g} hold, $A$ is surjective, and $\\mu\\in(0,1/\\rho)$.\r\nLet $\\bar{x} =x^*\\in \\mathbb{R}^n$ and $\\bar{y}=y^*-A^*(AA^*)^{-1}(Ay^*-\\prox_{\\mu g}(Ay^*))$ with $y^*\\in \\mathbb{R}^m$. Then\r\n\\begin{align*}\r\n&\\dist(0,G_\\lambda(\\bar{x},\\bar{y}))+\\dist(-\\nabla_y H(\\bar{x},\\bar{y}),A^*\\partial g(A\\bar{y})) \\\\\r\n&\\leq \\dist(0,G_\\lambda(x^*,y^*)) +\\dist(-\\nabla_yH(x^*,y^*),A^*\\partial g(\\prox_{\\mu g}(Ay^*)))\\\\\r\n&\\quad +(L_{12}+L_{22})L_g\\sigma_{\\min}(A)^{-1}\\mu.\r\n\\end{align*}\r\nConsequently, if $(x^*,y^*)$ is an $\\varepsilon$-approximate stationary point of problem~\\eqref{guanghua}, then $(\\bar{x},\\bar{y})$ is an $\\bar{\\varepsilon}$-approximate stationary point of problem~\\eqref{eq:L1E1} with $\\bar{\\varepsilon} =\\varepsilon +(L_{12}+L_{22})L_g\\sigma_{\\min}(A)^{-1}\\mu$.\r\n\\end{lemma}","sha256_utf8":"9cf7a7411af7ccb9416d643994130b3a352524f480b489dd00c81cd997a310ef"},{"start":21080,"end":22965,"start_line":385,"end_line":413,"raw_tex":"\\begin{proof}\r\nSince $\\bar{y}=y^*-A^*(AA^*)^{-1}(Ay^*-\\prox_{\\mu g}(Ay^*))$, multiplying both sides by $A$ yields $A\\bar{y}=\\prox_{\\mu g}(Ay^*)$. We have that\r\n\\begin{align*}\r\n&\\dist(-\\nabla_yH(\\bar{x},\\bar{y}),A^*\\partial g(A\\bar{y})) \\\\\r\n&\\leq \\dist(-\\nabla_yH(x^*,y^*),A^*\\partial g(A\\bar{y})) +\\|\\nabla_yH(x^*,y^*)-\\nabla_yH(\\bar{x},\\bar{y})\\| \\\\\r\n&=\\dist(-\\nabla_yH(x^*,y^*),A^*\\partial g(\\prox_{\\mu g}(Ay^*))) +\\|\\nabla_yH(x^*,y^*)-\\nabla_yH(x^*,\\bar{y})\\| \\\\\r\n&\\leq \\dist(-\\nabla_yH(x^*,y^*),A^*\\nabla g_{\\mu}(Ay^*)) +L_{22}\\|y^* -\\bar{y}\\|,\r\n\\end{align*}\r\nwhere the last inequality uses $A^*\\nabla g_{\\mu}(Ay^*)\\in A^*\\partial g(\\prox_{\\mu g}(Ay^*))$ (see Lemma~\\ref{l:weakcvx}\\ref{l:weakcvx_inclu}) and the Lipschitz continuity of $\\nabla_y H(x^*,\\cdot)$.\r\n\r\nNext, by Remark~\\ref{r:G(x,.)},\r\n\\begin{align*}\r\n\\dist(0,G_\\lambda(\\bar{x},\\bar{y})) &= \\|G_\\lambda(x^*,y^*) -G_\\lambda(x^*,y^*) +G_\\lambda(\\bar{x},\\bar{y})\\| \\\\\r\n&\\leq \\|G_\\lambda(x^*,y^*)\\| +\\|G_\\lambda(x^*,y^*) -G_\\lambda(\\bar{x},\\bar{y})\\| \\\\\r\n&= \\dist(0,G_\\lambda (x^*,y^*)) +\\|G_\\lambda(x^*,y^*) -G_\\lambda(x^*,\\bar{y})\\| \\\\\r\n&\\leq \\dist(0,G_\\lambda (x^*,y^*)) +L_{12}\\|y^* -\\bar{y}\\|.\r\n\\end{align*}\r\nNote that\r\n\\begin{align*}\r\n\\|y^* -\\bar{y}\\| =\\|A^*(AA^*)^{-1}(Ay^*-\\prox_{\\mu g}(Ay^*))\\| \\leq \\sigma_{\\min}(A)^{-1}\\mu L_g\r\n\\end{align*}\r\ndue to Lemma~\\ref{2.6} and the fact that the operator norm of $A^*(AA^*)^{-1}$ is bounded by the inverse of the smallest singular value $\\sigma_{\\min}(A)$ of $A$, i.e., $\\|A^*(AA^*)^{-1}\\|\\leq\\sigma_{\\min}(A)^{-1}$. Altogether, we obtain that\r\n\\begin{align*}\r\n&\\dist(0,G_\\lambda(\\bar{x},\\bar{y})) +\\dist(-\\nabla_y H(\\bar{x},\\bar{y}),A^*\\partial g(A\\bar{y})) \\\\\r\n&\\leq   \\dist(0,G_\\lambda(x^*,y^*)) +\\dist(-\\nabla_yH(x^*,y^*),A^*\\partial g(\\prox_{\\mu g}(Ay^*))) \\\\\r\n&\\quad+(L_{12} +L_{22})L_g\\sigma_{\\min}(A)^{-1}\\mu,\r\n\\end{align*}\r\nwhich completes the proof.\r\n\\end{proof}","sha256_utf8":"9a443b18c5e7c67cd8876a4b943687bb2b28c7ea6007c505fec5b615c374fc64"},{"start":26326,"end":26780,"start_line":477,"end_line":482,"raw_tex":"Recalling that $G_\\lambda(x,y) =\\frac{1}{\\lambda}(x -\\prox_{\\lambda f}(x -\\lambda\\nabla_x  H(x,y)))$, we have\r\n\\begin{align}\\label{zhu4.2}\r\nx^{k+1}&=\\prox_{\\sigma_k f}(\\bar{x}^k-\\sigma_k\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1}))\\notag \\\\\r\n&=\\bar{x}^k-\\sigma_k\\left(\\frac{1}{\\sigma_k}(\\bar{x}^k-\\prox_{\\sigma_k f}(\\bar{x}^k-\\sigma_k\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1})))\\right)\\notag \\\\\r\n&=\\bar{x}^k-\\sigma_k G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1}).\r\n\\end{align}\r\n","sha256_utf8":"0f15c2dcd83973269c049970359daa72d2ad38e98c4cd91294539f254ad7da03"},{"start":26822,"end":27500,"start_line":485,"end_line":494,"raw_tex":"\\begin{lemma}\r\n\\label{l:LL}\r\nSuppose that Assumptions \\ref{a:H} and \\ref{a:g} hold. Let $(\\bar{x}^k,\\bar{y}^k)$ be the sequence generated by Algorithm~\\ref{algo}. Then\r\n\\begin{align*}\r\n\\mathcal{L}_{\\mu_{k}}(\\bar{x}^{k+1},\\bar{y}^{k+1}) &\\leq \\mathcal{L}_{\\mu_{k}}(\\bar{x}^k,\\bar{y}^k)\r\n-\\frac{1}{2}\\delta\\tau_k\\|A^*\\nabla g_{\\mu_k}(A\\bar{y}^k)+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)\\|^2\\notag \\\\\r\n&\\quad -\\frac{1}{4}\\kappa\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)\\|^2 +(\\mu_k-\\mu_{k+1})L_g^2,\r\n\\end{align*}\r\nwhere $\\delta :=1 -\\alpha^2 -\\frac{L_{12}^2(1+\\alpha)^2}{L_{11}(L_{22} +2\\rho\\|A\\|^2)} >0$ and $\\kappa :=\\min\\{(2 -L_{11}\\beta\\sigma)\\beta, 2 -L_{11}\\sigma\\} >0$.\r\n\\end{lemma}","sha256_utf8":"4adc24021618285f5b6ae1f2a48898aaa2b23c648ab89a16b586b0e01bcf7f17"},{"start":27502,"end":35060,"start_line":495,"end_line":606,"raw_tex":"\\begin{proof}\r\nFrom the Lipschitz continuity of $\\nabla_x \\Phi_k(\\cdot,\\bar{y}^{k+1})$ and $\\nabla_y\\Phi_k(\\bar{x}^k, \\cdot)$, Lemma \\ref{l:descent} implies that\r\n\\begin{align*}\r\n\\Phi_k(\\bar{x}^{k+1},\\bar{y}^{k+1}) &\\leq\\Phi_k(\\bar{x}^{k},\\bar{y}^{k+1}) +\\langle\\nabla_x\\Phi_k(\\bar{x}^k,\\bar{y}^{k+1}),\\bar{x}^{k+1}-\\bar{x}^k\\rangle+\\frac{L_{11}}{2}\\|\\bar{x}^{k+1}-\\bar{x}^k\\|^2, \\\\\r\n\\Phi_k(\\bar{x}^k,\\bar{y}^{k+1}) &\\leq \\Phi_k(\\bar{x}^k,\\bar{y}^k) +\\langle\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k),\\bar{y}^{k+1}-\\bar{y}^k\\rangle+\\frac{L_k}{2}\\|\\bar{y}^{k+1}-\\bar{y}^k\\|^2.\r\n\\end{align*}\r\nWe note that\r\n\\begin{align}\\label{eq:grad_y Phi_k}\r\n\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k) =A^*\\nabla g_{\\mu_k}(A\\bar{y}^k)+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)\r\n\\end{align}\r\nand from \\eqref{s21}, \\eqref{s41}, \\eqref{s42}, and \\eqref{zhu4.2} that\r\n\\begin{align*}\r\n\\bar{x}^{k+1}-\\bar{x}^k &=\\beta_k(x^{k+1}-\\bar{x}^k)=-\\beta_k\\sigma_k G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1}), \\\\\r\n\\bar{y}^{k+1}-\\bar{y}^k &=(1+\\alpha_k)(y^{k+1}-\\bar{y}^k) =-(1+\\alpha_k)\\tau_k \\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k).\r\n\\end{align*}\r\nTherefore,\r\n\\begin{align}\\label{8.7}\r\n&\\Phi_k(\\bar{x}^{k+1},\\bar{y}^{k+1}) \\notag \\\\\r\n&\\leq \\Phi_k(\\bar{x}^k,\\bar{y}^k) +\\langle\\nabla_x\\Phi_k(\\bar{x}^k,\\bar{y}^{k+1}),\\bar{x}^{k+1}-\\bar{x}^k\\rangle+\\frac{L_{11}}{2}\\|\\bar{x}^{k+1}-\\bar{x}^k\\|^2 \\notag \\\\\r\n&\\quad +\\langle\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k),\\bar{y}^{k+1}-\\bar{y}^k\\rangle+\\frac{L_k}{2}\\|\\bar{y}^{k+1}-\\bar{y}^k\\|^2 \\notag \\\\\r\n&= \\Phi_k(\\bar{x}^k,\\bar{y}^k) -\\beta_k\\sigma_k\\langle\\nabla_x\\Phi_k(\\bar{x}^k,\\bar{y}^{k+1}),G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\rangle \\notag \\\\\r\n&\\quad +\\frac{1}{2}L_{11}\\beta_k^2\\sigma_k^2\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 \\notag\\\\\r\n&\\quad -\\frac{1}{2}(2 +2\\alpha_k -L_k(1+\\alpha_k)^2\\tau_k)\\tau_k\\|\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k)\\|^2.\r\n\\end{align}\r\nBy Lemma \\ref{l:MoreauEnv} and the fact that $\\mu_{k+1}\\in [\\mu_k/2, \\mu_k]$,\r\n\\begin{align}\\label{8.8}\r\ng_{\\mu_{k+1}}(Ay)\\leq g_{\\mu_k}(Ay) +\\frac{\\mu_k(\\mu_k-\\mu_{k+1})}{2\\mu_{k+1}}L_g^2\\leq g_{\\mu_k}(Ay) +(\\mu_k-\\mu_{k+1})L_g^2.\r\n\\end{align}\r\nAdding $H(x,y)$ to both sides of \\eqref{8.8} yields\r\n\\begin{align}\\label{3.4}\r\n\\Phi_{k+1}(x,y)\\leq \\Phi_k(x,y) +(\\mu_k-\\mu_{k+1})L_g^2.\r\n\\end{align}\r\nLetting $x =\\bar{x}^{k+1}$, $y =\\bar{y}^{k+1}$ in \\eqref{3.4} and combining it with \\eqref{8.7}, we obtain that\r\n\\begin{align}\\label{4.4}\r\n&\\Phi_{k+1}(\\bar{x}^{k+1},\\bar{y}^{k+1}) \\notag \\\\\r\n&\\leq \\Phi_k(\\bar{x}^k,\\bar{y}^k) -\\beta_k\\sigma_k\\langle\\nabla_x\\Phi_k(\\bar{x}^k,\\bar{y}^{k+1}),G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\rangle \\notag \\\\\r\n&\\quad +\\frac{1}{2}L_{11}\\beta_k^2\\sigma_k^2\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 \\notag\\\\\r\n&\\quad -\\frac{1}{2}(2 +2\\alpha_k -L_k(1+\\alpha_k)^2\\tau_k)\\tau_k\\|\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k)\\|^2 +(\\mu_k-\\mu_{k+1})L_g^2.\r\n\\end{align}\r\nBy the first-order optimality condition of \\eqref{s31},\r\n\\begin{align*}\r\n0\\in \\partial f(x^{k+1})+\\frac{1}{\\sigma_k}(x^{k+1}-\\bar{x}^k+\\sigma_k\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1})),\r\n\\end{align*}\r\nor equivalently,\r\n\\begin{align*}\r\n\\frac{1}{\\sigma_k}(\\bar{x}^k -x^{k+1}) -\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1})\\in \\partial f(x^{k+1}).\r\n\\end{align*}\r\nCombining with the convexity of $f$ and \\eqref{zhu4.2}, it follows that\r\n\\begin{align}\\label{8.9}\r\nf(x^{k+1}) &\\leq f(\\bar{x}^k) -\\left\\langle \\frac{1}{\\sigma_k}(\\bar{x}^k -x^{k+1}) -\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1}), \\bar{x}^k -x^{k+1}\\right\\rangle \\notag \\\\\r\n&= f(\\bar{x}^k) -\\frac{1}{\\sigma_k}\\|\\bar{x}^k -x^{k+1}\\|^2 +\\langle \\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1}), \\bar{x}^k -x^{k+1}\\rangle\\notag \\\\\r\n&= f(\\bar{x}^k) -\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 +\\sigma_k\\langle\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1}),G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\rangle.\r\n\\end{align}\r\nAgain by the convexity of $f$ and \\eqref{s42},\r\n\\begin{align*}\r\nf(\\bar{x}^{k+1})&=f\\big(\\beta_k x^{k+1}+(1-\\beta_k)\\bar{x}^k\\big)\r\n\\leq \\beta_k f(x^{k+1})+(1-\\beta_k)f(\\bar{x}^k).\r\n\\end{align*}\r\nThis together with \\eqref{8.9} gives\r\n\\begin{align}\\label{4.5}\r\nf(\\bar{x}^{k+1})&\\leq f(\\bar{x}^k) -\\beta_k\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 \\notag \\\\\r\n&\\quad +\\beta_k\\sigma_k\\langle\\nabla_xH(\\bar{x}^k,\\bar{y}^{k+1}),G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\rangle.\r\n\\end{align}\r\nWe now combine \\eqref{4.4} and \\eqref{4.5} to obtain\r\n\\begin{align}\\label{4.10}\r\n&\\mathcal{L}_{\\mu_{k}}(\\bar{x}^{k+1},\\bar{y}^{k+1}) \\notag \\\\\r\n&\\leq \\mathcal{L}_{\\mu_{k}}(\\bar{x}^k,\\bar{y}^k) -\\frac{1}{2}(2 +2\\alpha_k -L_k(1+\\alpha_k)^2\\tau_k)\\tau_k\\|\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k)\\|^2 \\notag \\\\ &\\quad -\\frac{1}{2}(2 -L_{11}\\beta_k\\sigma_k)\\beta_k\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 +(\\mu_k-\\mu_{k+1})L_g^2.\r\n\\end{align}\r\nNext, using Remark~\\ref{r:G(x,.)},\r\n\\begin{align*}\r\n\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)\\|&=\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)-G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})+G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|\\notag \\\\\r\n&\\leq\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)-G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|+\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|\\notag \\\\\r\n&\\leq L_{12}\\|\\bar{y}^{k+1}-\\bar{y}^k\\|+\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|.\r\n\\end{align*}\r\nSquaring and applying the inequality $\\frac{1}{2}(a+b)^2 \\leq a^2 +b^2$, we get\r\n\\begin{align*}\r\n\\frac{1}{2}\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)\\|^2 &\\leq L_{12}^2\\|\\bar{y}^{k+1}-\\bar{y}^k\\|^2 +\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 \\\\\r\n&= L_{12}^2(1+\\alpha_k)^2\\tau_k^2\\|\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k)\\|^2 +\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2.\r\n\\end{align*}\r\nwhich, combined with \\eqref{eq:grad_y Phi_k} and \\eqref{4.10}, implies that\r\n\\begin{align*}\r\n\\mathcal{L}_{\\mu_{k}}(\\bar{x}^{k+1},\\bar{y}^{k+1}) &\\leq \\mathcal{L}_{\\mu_{k}}(\\bar{x}^k,\\bar{y}^k) -\\frac{1}{2}\\delta_k\\tau_k\\|A^*\\nabla g_{\\mu_k}(A\\bar{y}^k)+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)\\|^2 \\notag \\\\\r\n&\\quad -\\frac{1}{4}\\kappa_k\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 +(\\mu_k-\\mu_{k+1})L_g^2,\r\n\\end{align*}\r\nwhere\r\n\\begin{align*}\r\n\\delta_k &:=2 +2\\alpha_k -L_k(1+\\alpha_k)^2\\tau_k -(2 -L_{11}\\beta_k\\sigma_k)\\sigma_k\\beta_kL_{12}^2(1+\\alpha_k)^2\\tau_k, \\\\\r\n\\kappa_k &:=(2 -L_{11}\\beta_k\\sigma_k)\\beta_k.\r\n\\end{align*}\r\nNoting that $\\tau_k\\leq 1/L_k$ and that\r\n\\begin{align*}\r\n(2 -L_{11}\\beta_k\\sigma_k)\\sigma_k\\beta_k\\leq \\frac{1}{L_{11}}\\left(\\frac{2 -L_{11}\\beta_k\\sigma_k +L_{11}\\beta_k\\sigma_k}{2}\\right)^2 =\\frac{1}{L_{11}},\r\n\\end{align*}\r\nwe have\r\n\\begin{align*}\r\n\\delta_k &\\geq 2 +2\\alpha_k -(1+\\alpha_k)^2 -\\frac{L_{12}^2}{L_{11}L_k}(1+\\alpha_k)^2 \\\\\r\n&= 1 -\\alpha_k^2 -\\frac{L_{12}^2}{L_{11}L_k}(1+\\alpha_k)^2 \\\\\r\n&\\geq \\delta =1 -\\alpha^2 -\\frac{L_{12}^2(1+\\alpha)^2}{L_{11}(L_{22} +2\\rho\\|A\\|^2)},\r\n\\end{align*}\r\nwhere the last inequality follows from from the fact that $\\alpha_k\\in [-\\alpha, \\alpha]\\subseteq [-1, \\alpha]$ and that $L_k =L_{22} +\\|A\\|^2\\max\\{\\frac{1}{\\mu_k}, \\frac{\\rho}{1 -\\rho\\mu_k}\\}\\geq L_{22} +2\\rho\\|A\\|^2$ since\r\n\\begin{align*}\r\n\\max\\left\\{\\frac{1}{\\mu_k}, \\frac{\\rho}{1 -\\rho\\mu_k}\\right\\} =\\begin{cases}\r\n\\frac{1}{\\mu_k} &\\text{if~} 0 < \\mu_k \\leq \\frac{1}{2\\rho}, \\\\\r\n\\frac{\\rho}{1 -\\rho\\mu_k} &\\text{if~} \\frac{1}{2\\rho} < \\mu_k < \\frac{1}{\\rho}.\r\n\\end{cases}\r\n\\end{align*}\r\nFinally, we derive from $\\sigma_k\\in \\sigma$ and $\\beta_k\\in [\\beta, 1]$ that\r\n\\begin{align*}\r\n\\kappa_k =(2 -L_{11}\\beta_k\\sigma_k)\\beta_k \\geq (2 -L_{11}\\beta_k\\sigma)\\beta_k \\geq \\kappa =\\min\\{(2 -L_{11}\\beta\\sigma)\\beta, 2 -L_{11}\\sigma\\}\r\n\\end{align*}\r\nas $(2 -L_{11}\\beta_k\\sigma)\\beta_k$ is a quadratic function of $\\beta_k$ with leading coefficient $-L_{11}\\sigma <0$. This completes the proof.\r\n\\end{proof}","sha256_utf8":"4bc7b604fa8fb839444d2f275bd50e6fc8457ad7d96faa4bcb2bd71077f76f68"},{"start":35314,"end":36054,"start_line":614,"end_line":625,"raw_tex":"\\begin{lemma}\r\n\\label{l:e-stationary}\r\nSuppose that Assumptions~\\ref{a:H} and \\ref{a:g} hold. Let $(\\bar{x}^k,\\bar{y}^k)$ be the sequence generated by Algorithm~\\ref{algo}. Then\r\n\\begin{align*}\r\n\\min_{1\\leq j\\leq k}\\left(\\|A^*\\nabla g_{\\mu_j}(A\\bar{y}^j)+\\nabla_y H(\\bar{x}^j,\\bar{y}^j)\\|+\\|G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j)\\|\\right)\r\n\\leq \\Theta k^{\\frac{\\theta-1}{2}},\r\n\\end{align*}\r\nwhere $\\Theta :=\\sqrt{2(\\ln 2)^{-1}M^{-1}(\\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}^*+\\mu_1L_g^2)}$ and\r\n\\begin{align*}\r\nM :=\\min\\left\\{\\frac{(1 -\\alpha^2)\\eta}{2} -\\frac{L_{12}^2(1+\\alpha)^2\\eta}{2L_{11}(L_{22} +2\\rho\\|A\\|^2)}, \\frac{(2 -L_{11}\\beta\\sigma)\\beta\\gamma}{4}, \\frac{(2 -L_{11}\\sigma)\\gamma}{4}\\right\\}.\r\n\\end{align*}\r\n\\end{lemma}","sha256_utf8":"f9df5a4947eda7fb64581f3363403f6a9999d6c8e7d348631679662af89cd51e"},{"start":36056,"end":38400,"start_line":626,"end_line":658,"raw_tex":"\\begin{proof}\r\nAccording to Lemma~\\ref{l:LL},\r\n\\begin{align}\\label{4.10'}\r\n\\mathcal{L}_{\\mu_{k}}(\\bar{x}^{k+1},\\bar{y}^{k+1}) &\\leq \\mathcal{L}_{\\mu_{k}}(\\bar{x}^k,\\bar{y}^k) -\\frac{1}{2}\\delta\\tau_k\\|A^*\\nabla g_{\\mu_k}(A\\bar{y}^k)+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)\\|^2\\notag \\\\\r\n&\\quad -\\frac{1}{4}\\kappa\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)\\|^2 +(\\mu_k-\\mu_{k+1})L_g^2.\r\n\\end{align}\r\nSumming \\eqref{4.10'} from $k=1$ to $k=K$, we have\r\n\\begin{align}\\label{4.11}\r\n&\\sum_{k=1}^K \\left(\\frac{1}{2}\\delta\\tau_k\\|A^*\\nabla g_{\\mu_k}(A\\bar{y}^k)+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)\\|^2 +\\frac{1}{4}\\kappa\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)\\|^2\\right) \\notag \\\\\r\n&\\leq \\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}_{\\mu_{K+1}}(\\bar{x}^{K+1},\\bar{y}^{K+1})+(\\mu_1-\\mu_{K+1})L_g^2\\notag \\\\\r\n&\\leq \\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}^*+\\mu_1L_g^2.\r\n\\end{align}\r\nSince $\\sigma_k \\geq\\gamma k^{-\\theta}$, $\\tau_k\\geq\\eta k^{-\\theta}$, and $M =\\min\\{\\frac{1}{2}\\delta\\eta, \\frac{1}{4}\\kappa\\gamma\\}$, it follows from \\eqref{4.11} that\r\n\\begin{multline*}\r\nM\\left(\\sum_{k=1}^{K}k^{-\\theta}\\right) \\min_{1\\leq j\\leq K}(\\|A^*\\nabla g_{\\mu_j}(A\\bar{y}^j)+\\nabla_yH(\\bar{x}^j,\\bar{y}^j)\\|^2+\r\n\\|G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j)\\|^2) \\\\ \\leq\\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)\r\n-\\mathcal{L}^*+\\mu_1L_g^2,\r\n\\end{multline*}\r\nNote that\r\n\\begin{align*}\r\n\\sum_{k=1}^Kk^{-\\theta}&\\geq \\sum_{k=1}^K\\int_k^{k+1}x^{-\\theta}dx=\\int_1^{K+1}x^{-\\theta}dx=\\frac{1}{1-\\theta}((1+K)^{1-\\theta}-1)\\\\\r\n&\\geq(\\ln 2)K^{1-\\theta},\r\n\\end{align*}\r\nwhere the last inequality is obtained by applying Lemma \\ref{l:estimate} with $x =K$ and $\\alpha =1 -\\theta$. We deduce that\r\n\\begin{multline*}\r\n\\min_{1\\leq j\\leq K}(\\|A^*\\nabla g_{\\mu_j}(A\\bar{y}^j)+\\nabla_yH(\\bar{x}^j,\\bar{y}^j)\\|^2+\\|G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j)\\|^2) \\\\ \\leq (\\ln 2)^{-1}M^{-1}(\\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}^*+\\mu_1L_g^2)k^{\\theta-1}.\r\n\\end{multline*}\r\nUsing the inequality $(\\|a\\|+\\|b\\|)^2\\leq2(\\|a\\|^2+\\|b\\|^2)$, we have\r\n\\begin{multline*}\r\n\\min_{1\\leq j\\leq k}(\\|A^*\\nabla g_{\\mu_j}(A\\bar{y}^j)+\\nabla_yH(\\bar{x}^j,\\bar{y}^j)\\|+\\|G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j)\\|) \\\\ \\leq \\sqrt{2(\\ln 2)^{-1}M^{-1}(\\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}^*+\\mu_1L_g^2)} k^{\\frac{\\theta-1}{2}}.\r\n\\end{multline*}\r\nThe proof is complete.\r\n\\end{proof}","sha256_utf8":"32c2d5b95682d6de1d88e2ef2c7c67c3ceaff2cda2d0f2e9fd777a97ccb5a32b"},{"start":39696,"end":40695,"start_line":672,"end_line":686,"raw_tex":"\\begin{proof}\r\nUsing Lemmas~\\ref{l:e-stationary} and \\ref{l:weakcvx}, we obtain\r\n\\begin{align*}\r\n&\\min_{1\\leq j\\leq k}(\\dist(-\\nabla_yH(\\bar{x}^j,\\bar{y}^j),A^*\\partial g(\\prox_{\\mu_j g}(A\\bar{y}^j))+\\dist(0,G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j))) \\\\\r\n&\\leq\\min_{1\\leq j\\leq k}(\\|A^*\\nabla g_{\\mu_j}(A\\bar{y}^j)+\\nabla_yH(\\bar{x}^j,\\bar{y}^j)\\|+\\|G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j)\\|)\\leq \\Theta k^{\\frac{\\theta-1}{2}},\r\n\\end{align*}\r\nwhich together with Lemma~\\ref{l:transfer} implies that\r\n\\begin{align*}\r\n&\\min_{1\\leq j\\leq k}(\\dist(-\\nabla_yH(\\hat{x}^j,\\hat{y}^j),A^*\\partial g(A\\hat{y}^j))+\\dist(0,G_{\\sigma_j}(\\hat{x}^j,\\hat{y}^j))) \\\\\r\n&\\leq \\Theta k^{\\frac{\\theta-1}{2}} +(L_{12}+L_{12})L_g\\sigma_{\\min}(A)^{-1}\\mu_k \\\\\r\n&= \\Theta k^{-\\frac{1-\\theta}{2}} +(L_{12}+L_{12})L_g\\sigma_{\\min}(A)^{-1}\\mu_1 k^{-\\min\\{\\theta, \\frac{1-\\theta}{2}\\}} \\\\\r\n&\\leq (\\Theta +(L_{12}+L_{12})L_g\\sigma_{\\min}(A)^{-1}\\mu_1)k^{-\\min\\{\\theta, \\frac{1-\\theta}{2}\\}}.\r\n\\end{align*}\r\nThe proof is complete.\r\n\\end{proof}","sha256_utf8":"3b57760a8788d212bb5269600bae44894be253e0983addb4a0a98f3542777d93"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"变平滑交替算法的驻点速率","source_units":[{"start":38730,"end":39694,"start_line":662,"end_line":671,"raw_tex":"\\begin{theorem}\r\nSuppose that Assumptions~\\ref{a:H} and \\ref{a:g} hold. Let $(\\bar{x}^k,\\bar{y}^k)$ be the sequence generated by Algorithm~\\ref{algo} with $\\mu_k =\\mu_1 k^{-\\min\\{\\theta, \\frac{1-\\theta}{2}\\}}$. Then, for $\\hat{x}^k=\\bar{x}^k$ and $\\hat{y}^k=\\bar{y}^k-A^*(AA^*)^{-1}(A\\bar{y}^k-\\prox_{\\mu g}(A\\bar{y}^k))$, it holds that\r\n\\begin{multline*}\r\n\\min_{1\\leq j\\leq k}(\\dist(-\\nabla_yH(\\hat{x}^j,\\hat{y}^j),A^*\\partial g(A\\hat{y}^j))+\\dist(0,G_{\\sigma_j}(\\hat{x}^j,\\hat{y}^j))) \\\\ \\leq (\\Theta +(L_{12}+L_{12})L_g\\sigma_{\\min}(A)^{-1}\\mu_1)k^{-\\min\\{\\theta, \\frac{1-\\theta}{2}\\}},\r\n\\end{multline*}\r\nwhere $\\Theta :=\\sqrt{2(\\ln 2)^{-1}M^{-1}(\\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}^*+\\mu_1L_g^2)}$ and\r\n\\begin{align*}\r\nM :=\\min\\left\\{\\frac{(1 -\\alpha^2)\\eta}{2} -\\frac{L_{12}^2(1+\\alpha)^2\\eta}{2L_{11}(L_{22} +2\\rho\\|A\\|^2)}, \\frac{(2 -L_{11}\\beta\\sigma)\\beta\\gamma}{4}, \\frac{(2 -L_{11}\\sigma)\\gamma}{4}\\right\\}.\r\n\\end{align*}\r\n\\end{theorem}","sha256_utf8":"0a5544927e26762bca9e0b9cc4ad2e4d2212a24cd6d29d1ed3eb75b345a005b4"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"主定理完整证明","source_units":[{"start":39696,"end":40695,"start_line":672,"end_line":686,"raw_tex":"\\begin{proof}\r\nUsing Lemmas~\\ref{l:e-stationary} and \\ref{l:weakcvx}, we obtain\r\n\\begin{align*}\r\n&\\min_{1\\leq j\\leq k}(\\dist(-\\nabla_yH(\\bar{x}^j,\\bar{y}^j),A^*\\partial g(\\prox_{\\mu_j g}(A\\bar{y}^j))+\\dist(0,G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j))) \\\\\r\n&\\leq\\min_{1\\leq j\\leq k}(\\|A^*\\nabla g_{\\mu_j}(A\\bar{y}^j)+\\nabla_yH(\\bar{x}^j,\\bar{y}^j)\\|+\\|G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j)\\|)\\leq \\Theta k^{\\frac{\\theta-1}{2}},\r\n\\end{align*}\r\nwhich together with Lemma~\\ref{l:transfer} implies that\r\n\\begin{align*}\r\n&\\min_{1\\leq j\\leq k}(\\dist(-\\nabla_yH(\\hat{x}^j,\\hat{y}^j),A^*\\partial g(A\\hat{y}^j))+\\dist(0,G_{\\sigma_j}(\\hat{x}^j,\\hat{y}^j))) \\\\\r\n&\\leq \\Theta k^{\\frac{\\theta-1}{2}} +(L_{12}+L_{12})L_g\\sigma_{\\min}(A)^{-1}\\mu_k \\\\\r\n&= \\Theta k^{-\\frac{1-\\theta}{2}} +(L_{12}+L_{12})L_g\\sigma_{\\min}(A)^{-1}\\mu_1 k^{-\\min\\{\\theta, \\frac{1-\\theta}{2}\\}} \\\\\r\n&\\leq (\\Theta +(L_{12}+L_{12})L_g\\sigma_{\\min}(A)^{-1}\\mu_1)k^{-\\min\\{\\theta, \\frac{1-\\theta}{2}\\}}.\r\n\\end{align*}\r\nThe proof is complete.\r\n\\end{proof}","sha256_utf8":"3b57760a8788d212bb5269600bae44894be253e0983addb4a0a98f3542777d93"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"由平滑残差控制近端次梯度残差","source_units":[{"start":39711,"end":40126,"start_line":673,"end_line":677,"raw_tex":"Using Lemmas~\\ref{l:e-stationary} and \\ref{l:weakcvx}, we obtain\r\n\\begin{align*}\r\n&\\min_{1\\leq j\\leq k}(\\dist(-\\nabla_yH(\\bar{x}^j,\\bar{y}^j),A^*\\partial g(\\prox_{\\mu_j g}(A\\bar{y}^j))+\\dist(0,G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j))) \\\\\r\n&\\leq\\min_{1\\leq j\\leq k}(\\|A^*\\nabla g_{\\mu_j}(A\\bar{y}^j)+\\nabla_yH(\\bar{x}^j,\\bar{y}^j)\\|+\\|G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j)\\|)\\leq \\Theta k^{\\frac{\\theta-1}{2}},\r\n\\end{align*}\r\n","sha256_utf8":"d21e46893bebb50a18de0eb0c9923af94da7bfd10c4558d024a3cd81157fc661"}]},{"id":"S004","semantic_type":"proof_step","kind":"proof_step","title":"转移驻点残差并合并衰减指数","source_units":[{"start":40126,"end":40684,"start_line":678,"end_line":685,"raw_tex":"which together with Lemma~\\ref{l:transfer} implies that\r\n\\begin{align*}\r\n&\\min_{1\\leq j\\leq k}(\\dist(-\\nabla_yH(\\hat{x}^j,\\hat{y}^j),A^*\\partial g(A\\hat{y}^j))+\\dist(0,G_{\\sigma_j}(\\hat{x}^j,\\hat{y}^j))) \\\\\r\n&\\leq \\Theta k^{\\frac{\\theta-1}{2}} +(L_{12}+L_{12})L_g\\sigma_{\\min}(A)^{-1}\\mu_k \\\\\r\n&= \\Theta k^{-\\frac{1-\\theta}{2}} +(L_{12}+L_{12})L_g\\sigma_{\\min}(A)^{-1}\\mu_1 k^{-\\min\\{\\theta, \\frac{1-\\theta}{2}\\}} \\\\\r\n&\\leq (\\Theta +(L_{12}+L_{12})L_g\\sigma_{\\min}(A)^{-1}\\mu_1)k^{-\\min\\{\\theta, \\frac{1-\\theta}{2}\\}}.\r\n\\end{align*}\r\nThe proof is complete.\r\n","sha256_utf8":"392e9c25f11ba867a85b0adc358d4f176761f315056ef6ff1a57fde7fa917682"}]},{"id":"S005","semantic_type":"problem_setting","kind":"problem_setting","title":"耦合复合优化模型","source_units":[{"start":5196,"end":5333,"start_line":127,"end_line":129,"raw_tex":"\\begin{equation}\\label{eq:L1E1}\r\n\\min_{(x,y)\\in \\mathbb{R}^{n}\\times\\mathbb{R}^{m}} \\mathcal{L}(x,y)=f(x) + g(Ay)+H(x,y),\r\n\\end{equation}","sha256_utf8":"4c5c9f288c9f005c2b1dd00e1918551de2f5d4580a2705a0aa4c1185b1ed64f4"},{"start":5335,"end":6023,"start_line":130,"end_line":130,"raw_tex":"where $f:\\mathbb{R}^{n}\\rightarrow \\mathbb{R}$ is a (possibly nonsmooth) convex function, $g:\\mathbb{R}^{d}\\rightarrow \\mathbb{R}\\cup \\{+\\infty\\}$ is a nonsmooth and proper lower semicontinuous $\\rho$-weakly convex function, $A:\\mathbb{R}^{m} \\rightarrow \\mathbb{R}^{d}$ is a linear operator, and $H:\\mathbb{R}^{n}\\times \\mathbb{R}^{m} \\rightarrow \\mathbb{R}$ is a continuously differentiable (possibly nonconvex) function. Such problem has a wide range of applications, including compressed sensing, machine learning, nonnegative matrix factorization, image denoising, signal recovery, and multimodal learning for image classification; see, for example \\cite{BST,CP,CP1,DD,OBP,PN,TSR}.\r\n","sha256_utf8":"0edff7967c8086d5dcfde93795b55db1e7d1b3b434a8c211aba3d813a6976ecf"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"平滑目标及其梯度常数","source_units":[{"start":8479,"end":8619,"start_line":150,"end_line":152,"raw_tex":"\\begin{align}\\label{guanghua}\r\n\\min_{(x,y)\\in \\mathbb{R}^{n}\\times\\mathbb{R}^{m}} \\mathcal{L}_\\mu(x,y)=f(x)+g_{\\mu}(Ay)+H(x,y).\r\n\\end{align}","sha256_utf8":"9c1b4f69707301ebbe6085fdeaae1ef4556e8a3904494afd38b80398dda9f465"},{"start":23193,"end":23333,"start_line":421,"end_line":423,"raw_tex":"\\begin{align}\\label{app}\r\n\\min_{(x,y)\\in \\mathbb{R}^{n}\\times\\mathbb{R}^{m}}\\mathcal{L}_{\\mu_k}(x,y)=f(x)+g_{\\mu_k}(Ay)+H(x,y).\r\n\\end{align}","sha256_utf8":"23bb12d16803b02dbf8a21d14c2f6cf2370264b4e31687a95e90247002851ea0"},{"start":23335,"end":23593,"start_line":424,"end_line":429,"raw_tex":"Denote $\\Phi_k(x,y):=g_{\\mu_k}(Ay)+H(x,y)$. For the smoothed function $\\Phi_k$, we have\r\n\\begin{align*}\r\n\\nabla_y\\Phi_k(x,y) &=\\frac{1}{\\mu_k}A^{*}(Ay-\\prox_{\\mu_k g}(Ay))+\\nabla_y H(x,y) \\text{~~and}\\\\\r\n\\nabla_x\\Phi_k(x,y) &=\r\n\\nabla_x H(x,y).\r\n\\end{align*}","sha256_utf8":"ed038839b3db83752ab5834f6b7f3b85bacfcaa16fe6645b9a2d2aeb0d5b9ff6"},{"start":23595,"end":24065,"start_line":430,"end_line":430,"raw_tex":"From Assumptions~\\ref{a:H} and \\ref{a:g}, it can be seen that $\\nabla_x\\Phi_k(\\cdot,y)$ and $\\nabla_y\\Phi_k(x,\\cdot)$ are two Lipschitz continuous functions with Lipschitz constants $L_{11}$ and $L_k:=L_{22}+\\|A\\|^2\\max\\{\\frac{1}{\\mu_k},\\frac{\\rho}{1-\\rho\\mu_k}\\}$, respectively. Increasing $L_{11}$, $L_{22}$, and/or $\\rho$ if necessary, we can assume that $L_{12}^2 <L_{11}(L_{22} +2\\rho\\|A\\|^2)$, which ensures the existence of $\\alpha$ in the subsequent algorithm.\r\n","sha256_utf8":"25657c3385707a29c27cb44d4feaa84ebb0cb4e143d473d2c9fb723910ba8e0f"}]},{"id":"S007","semantic_type":"notation","kind":"notation","title":"欧氏范数距离与函数域","source_units":[{"start":9551,"end":10302,"start_line":162,"end_line":163,"raw_tex":"Let $\\mathbb{R}^n$ be a finite-dimensional Euclidean space, which equipped with standard inner product $\\langle \\cdot,\\cdot \\rangle$ and norm $\\|\\cdot \\|$, respectively. For any $x,y\\in \\mathbb{R}^n$, $\\|(x,y)\\|:=\\sqrt{\\|x\\|^2+\\|y\\|^2}$. Given a nonempty set $C\\subseteq \\mathbb{R}^n$, the distance from $z\\in \\mathbb{R}^n$, the \\emph{distance} from $z$ to $C$ is defined as\r\n$\\dist(z,C):=\\inf_{w\\in C}\\|w-z\\|$. Let $h: \\mathbb{R}^n\\rightarrow {\\mathbb{R}}\\cup \\{+\\infty\\}$ be a mapping, the \\emph{domain} of $h$ is defined by $\\dom h:=\\{x\\in \\mathbb{R}^n:h(x)<+\\infty\\}$. The function $h$ is said to be \\emph{proper} if $\\dom h \\neq\\varnothing$ and \\emph{lower semicontinuous} if $h(x)\\leq \\liminf_{z\\rightarrow x}h(z)$ for any $x\\in \\mathbb{R}^n$.\r\n","sha256_utf8":"197f05a3e9e1235bc076ba6c08dcd8fd1e33311c7b7fa54a5c673ddf16af52c6"}]},{"id":"S008","semantic_type":"definition","kind":"definition","title":"近端算子及非扩张性","source_units":[{"start":10304,"end":10812,"start_line":165,"end_line":172,"raw_tex":"Let $F:\\mathbb{R}^n\\rightarrow \\mathbb{R}\\cup \\{+\\infty\\}$ be a proper lower semicontinuous convex function and $\\lambda>0$. The \\emph{proximal operator} of $F$ at $v\\in \\mathbb{R}^n$ is defined by\r\n\\begin{align*}\r\n\\prox_{\\lambda F}(v):=\\argmin_{x\\in \\mathbb{R}^n} \\left(F(x)+\\frac{1}{2\\lambda}\\|x-v\\|^2\\right).\r\n\\end{align*}\r\nIt is known that $\\prox_{\\lambda F}$ is nonexpansive, i.e., for all $x,y\\in \\mathbb{R}^n$,\r\n\\begin{align*}\r\n\\|\\prox_{\\lambda F}(x)-\\prox_{\\lambda F}(y)\\|\\leq\\|x-y\\|.\r\n\\end{align*}\r\n","sha256_utf8":"f024f112b0ff754686a4f6cf6db5dc2cf70d2702d30f8514c73411d1ffef0ef1"}]},{"id":"S009","semantic_type":"definition","kind":"definition","title":"次微分与光滑求和规则","source_units":[{"start":10812,"end":11983,"start_line":173,"end_line":183,"raw_tex":"The \\emph{regular subdifferential} of $F$ at $x\\in\\dom F$ is defined by\r\n$$\\widehat{\\partial}F(x):=\\left\\{u\\in\\mathbb{R}^{n}:\\liminf_{y\\rightarrow x\\atop y\\neq x} \\frac{\\ F(y)- F(x) + \\langle u, y-x \\rangle}{\\|y-x\\|}\\geq0\\right\\}.$$\r\nThe \\emph{limiting subdifferential} of $F$ at $x\\in\\dom F$ is defined by\r\n \\begin{align*}\r\n\\partial F(x) :=\\{u \\in\\mathbb{R}^{n}: \\exists\\ x_{k}\\rightarrow x, \\widehat{\\partial}F(x_{k}) \\ni u_{k} \\rightarrow u \\ \\mathrm{with} \\  F(x_{k}) \\rightarrow F(x)\\}.\r\n\\end{align*}\r\nWhen $F$ is a convex function, both subdifferentials coincide with the subdifferential in the sense of convex analysis, i.e.,\r\n\\begin{align*}\r\n\\widehat{\\partial}F(x)=\\partial F(x)=\\{u\\in\\mathbb{R}^{n}: \\forall y\\in\\mathbb{R}^{n},\\quad F(y)-F(x)\\geq\\langle{u,y-x}\\rangle\\}.\r\n\\end{align*}\r\nClearly, $\\widehat{\\partial}F(x) \\subseteq \\partial F(x)$ for all $x\\in \\mathbb{R}^{n}$ and both of them are closed.  If $G:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ is  a continuously differentiable function, then $\\partial G(x)=\\{\\nabla G(x)\\}$ and $\\partial(F+G)(x)=\\partial F(x)+\\nabla G(x)$ for all $x\\in \\mathbb{R}^n$, where $\\nabla G(x)$ denotes the gradient of $G$ at $x$.\r\n","sha256_utf8":"62ef091fbae864e0cfc0bf7d7a7147ea74171bfa788bd89fe7486bd2a095c2c4"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"弱凸性定义","source_units":[{"start":12350,"end":12542,"start_line":194,"end_line":195,"raw_tex":"\\begin{definition}[\\cite{V}] A function $F: \\mathbb{R}^n\\rightarrow (-\\infty,+\\infty]$  is said to be \\emph{$\\rho$-weakly convex} if $F +\\frac{\\rho}{2}\\|\\cdot\\|^2$ is convex.\r\n\\end{definition}","sha256_utf8":"15bc72b0a946710c6ebf4d7688b8b7402f8274fb0b874c6114e44a77efafc369"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"莫罗包络定义","source_units":[{"start":12654,"end":13013,"start_line":201,"end_line":206,"raw_tex":"\\begin{definition}[{\\cite[Definition 2.1]{BW}}] Let $F: \\mathbb{R}^n\\rightarrow (-\\infty,+\\infty]$ be a proper $\\rho$-weakly convex lower semicontinuous function. The Moreau envelope function of $F$ is defined as\r\n\\begin{align*}\r\nF_{\\mu}(x):=\\min_{y\\in \\mathbb{R}^n}\\{F(y)+\\frac{1}{2\\mu}\\|y-x\\|^2\\},\r\n\\end{align*}\r\nwhere $\\mu \\in(0,1/\\rho)$.\r\n\\end{definition}","sha256_utf8":"4f9332665f3bddb2a4cf61dc5f82519b4d645c81c5ce0428ea6b8cc6624c2aab"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"近端梯度映射","source_units":[{"start":16137,"end":16251,"start_line":276,"end_line":278,"raw_tex":"\\begin{align*}\r\nG_\\lambda(x,y) =\\frac{1}{\\lambda}(x -\\prox_{\\lambda f}(x -\\lambda\\nabla_x  H(x,y))).\r\n\\end{align*}","sha256_utf8":"7510f4e485203a4746a2ae5b70686f04e738dc23f5d8060fa8d1d01376b1105a"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"原问题及平滑问题的近似驻点","source_units":[{"start":18156,"end":18750,"start_line":329,"end_line":338,"raw_tex":"\\begin{definition}\r\nLet $(x^*,y^*)\\in \\mathbb{R}^n\\times \\mathbb{R}^m$. We say that $(x^*,y^*)$ is an \\emph{$\\varepsilon$-approximate stationary point} of problem~\\eqref{eq:L1E1} if\r\n\\begin{align}\\label{wending}\r\n\\dist(0,G_\\lambda(x^*,y^*))+\\dist(-\\nabla_yH(x^*,y^*),A^*\\partial g(Ay^*))\\leq\\varepsilon.\r\n\\end{align}\r\nSimilarly, $(x^*,y^*)$ is said to be an \\emph{$\\varepsilon$-approximate stationary point} of problem~\\eqref{guanghua} if\r\n\\begin{align}\\label{wending1}\r\n\\dist(0,G_\\lambda(x^*,y^*))+\\dist(-\\nabla_yH(x^*,y^*),A^*\\nabla g_\\mu(Ay^*))\\leq\\varepsilon.\r\n\\end{align}\r\n\\end{definition}","sha256_utf8":"dc35e0a4f9fb4ac4504257c99efd197c7eed56d977e349b9323d94d62644f51f"}]},{"id":"S014","semantic_type":"assumption","kind":"assumption","title":"耦合函数的偏梯度利普希茨假设","source_units":[{"start":18834,"end":19559,"start_line":342,"end_line":353,"raw_tex":"\\begin{assumption}\r\n\\label{a:H}\r\nFor any $y\\in \\mathbb{R}^m$, the partial gradient $\\nabla_x H(\\cdot,y)$ is $L_{11}$-Lipschitz continuous, i.e.,\r\n\\begin{align*}\r\n\\forall x_1,x_2\\in \\mathbb{R}^n,\\quad \\|\\nabla_x H(x_1,y)-\\nabla_x H(x_2,y)\\|\\leq L_{11}\\|x_1-x_2\\|.\r\n\\end{align*}\r\nFor any $x\\in \\mathbb{R}^n$, the partial gradient $\\nabla_x H(x,\\cdot)$ is $L_{12}$-Lipschitz continuous and the partial gradient $\\nabla_y H(x,\\cdot)$ is $L_{22}$-Lipschitz continuous, i.e.,\r\n\\begin{align*}\r\n&\\forall y_1,y_2\\in \\mathbb{R}^n,\\quad \\|\\nabla_x H(x,y_1)-\\nabla_x H(x,y_2)\\|\\leq L_{12}\\|y_1-y_2\\|,\\\\\r\n&\\forall y_1,y_2\\in\\mathbb{ R}^m,\\quad \\|\\nabla_y H(x,y_1)-\\nabla_y H(x,y_2)\\|\\leq L_{22}\\|y_1-y_2\\|.\r\n\\end{align*}\r\n\\end{assumption}","sha256_utf8":"25777c1296ac3a84388759e6527d58f822adcd6df2fef6e1144b5ff3c9f38060"}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","title":"弱凸正则项与目标下界假设","source_units":[{"start":19565,"end":19733,"start_line":356,"end_line":359,"raw_tex":"\\begin{assumption}\r\n\\label{a:g}\r\nThe function $g$ is $\\rho$-weakly convex and $L_g$-Lipschitz continuous. The function $\\mathcal{L}$ is bounded below.\r\n\\end{assumption}","sha256_utf8":"0cf6ffee9ee84928379890111046f75578c86bd6af0728621ab00d28193d26c3"}]},{"id":"S016","semantic_type":"algorithm","kind":"algorithm","title":"变平滑交替近端梯度算法","source_units":[{"start":24255,"end":25444,"start_line":434,"end_line":452,"raw_tex":"\\begin{algorithm}[htb]\r\n\\caption{Variable smoothing alternating proximal gradient (VsaPG)}\r\n\\label{algo}\r\n\r\n\\textbf{Step 1.} Let $x^1 =\\bar{x}^1\\in \\mathbb{R}^{n}$, $y^1 =\\bar{y}^1\\in \\mathbb{R}^{m}$, $\\mu_1\\in (0, 1/\\rho)$, and set $k =1$. Let $\\gamma\\in (0, +\\infty)$, $\\eta\\in (0, +\\infty)$, $\\theta\\in (0, 1)$, $\\beta\\in(0, 1]$, $\\sigma\\in (0, 2/L_{11})$, and let $\\alpha\\in (0, 1)$ be such that\r\n\\begin{align*}\r\n1 -\\alpha^2 -\\frac{L_{12}^2(1 +\\alpha)^2}{L_{11}(L_{22} +2\\rho\\|A\\|^2)} >0.\r\n\\end{align*}\r\n\r\n\\textbf{Step 2.} Let $\\mu_{k+1}\\in [\\mu_k/2, \\mu_k]$, $\\alpha_k\\in [-\\alpha, \\alpha]$, $\\beta_k\\in [\\beta,1]$, $\\sigma_k\\in [\\gamma k^{-\\theta}, \\sigma]$, and $\\tau_k\\in [\\eta k^{-\\theta}, 1/L_k]$. Compute\r\n\\begin{align}\r\n\\label{s21}&y^{k+1}=\\bar{y}^k-\\tau_k(A^*\\nabla g_{\\mu_k}(A\\bar{y}^{k})+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)),\\\\\r\n\\label{s41}&\\bar{y}^{k+1}=y^{k+1}+\\alpha_k(y^{k+1}-\\bar{y}^k),\\\\\r\n\\label{s31}&x^{k+1}=\\prox_{\\sigma_k f}(\\bar{x}^k-\\sigma_k\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1})),\\\\\r\n\\label{s42}&\\bar{x}^{k+1} =(1 -\\beta_k)\\bar{x}^k +\\beta_kx^{k+1}.\r\n\\end{align}\r\n\r\n\\textbf{Step 3.} If a termination criterion is not met, set $k =k+1$ and go to Step 2.\r\n\\end{algorithm}","sha256_utf8":"590e9d6be8ae03e82695480c2de1832f7f868cd49bcbf209802fccb6c3fcbc64"}]},{"id":"S017","semantic_type":"notation","kind":"notation","title":"平滑目标下极限记号","source_units":[{"start":35066,"end":35310,"start_line":609,"end_line":612,"raw_tex":"Obviously, Assumption \\ref{a:g} implies that $\\liminf_{k\\rightarrow \\infty} \\mathcal{L}_{\\mu_k}(x^k,y^k)>-\\infty$. From now on, we denote\r\n\\begin{align*}\r\n\\mathcal{L}^* :=\\liminf_{k\\rightarrow \\infty} \\mathcal{L}_{\\mu_k}(x^k,y^k).\r\n\\end{align*}","sha256_utf8":"53c98c7ba79a859f1580fe9eea47ec75f01e1daeaaf0e6296ca790bbd92a3a1c"}]},{"id":"S018","semantic_type":"lemma","kind":"lemma","title":"光滑函数下降引理","source_units":[{"start":11987,"end":12346,"start_line":186,"end_line":192,"raw_tex":"\\begin{lemma}[{\\cite[Theorem 2.64]{BC}}]\r\n\\label{l:descent}\r\nLet $F:\\mathbb{R}^n\\rightarrow \\mathbb{R}$ be a continuously differentiable function whose gradient $\\nabla F$ is $L$-Lipschitz continuous with $L>0$. Then, for all $x,y\\in \\mathbb{R}^n$,\r\n\\begin{align*}\r\n|F(y)-F(x)-\\langle\\nabla F(x),y-x\\rangle|\\leq\\frac{L}{2}\\|y-x\\|^2.\r\n\\end{align*}\r\n\\end{lemma}","sha256_utf8":"37900e490b090f7e31d9716a56ff1feeee164582423be267ed38e45ed80f92ce"}]},{"id":"S019","semantic_type":"lemma","kind":"lemma","title":"包络梯度及次梯度包含关系","source_units":[{"start":13017,"end":13687,"start_line":208,"end_line":221,"raw_tex":"\\begin{lemma}\r\n\\label{l:weakcvx}\r\nLet $F: \\mathbb{R}^n\\rightarrow (-\\infty,+\\infty]$ be a proper $\\rho$-weakly convex and lower semicontinuous function, and let $\\mu \\in(0,1/\\rho)$. Then\r\n\\begin{enumerate}\r\n\\item\r\n\\cite[Corollary 3.4]{HLO} The Moreau envelope function $F_{\\mu}$ is continuously differentiable on $\\mathbb{R}^n$ and, for all $x\\in \\mathbb{R}^n$,\r\n\\begin{align*}\r\n\\nabla F_{\\mu}(x) =\\frac{1}{\\mu}(x-\\prox_{\\mu F}(x)).\r\n\\end{align*}\r\nThis gradient is $\\max\\{\\frac{1}{\\mu},\\frac{\\rho}{1-\\rho \\mu}\\}$-Lipschitz continuous.\r\n\\item\\label{l:weakcvx_inclu}\r\n\\cite[Lemma 3.2]{BW} $\\nabla{F_{\\mu}(x)}\\in \\partial F(\\prox_{\\mu F}(x))$.\r\n\\end{enumerate}\r\n\\end{lemma}","sha256_utf8":"3e0cbfb074511e3d3201a2b193f7403a37888914ef5a0eb4dfa95ec7df51124b"}]},{"id":"S020","semantic_type":"lemma","kind":"lemma","title":"包络梯度与近端位移界","source_units":[{"start":13695,"end":14104,"start_line":225,"end_line":229,"raw_tex":"\\begin{lemma}[{\\cite[Lemma 3.3]{BW}}]\\label{2.6} Let $F: \\mathbb{R}^n\\rightarrow (-\\infty,\\infty)$ be a $\\rho$-weakly convex function and $L_F$-Lipschitz continuous, and let $\\mu \\in (0,1/\\rho)$. Then the Moreau envelope $F_\\mu$ is Lipschitz continuous and, for all $x\\in \\mathbb{R}^n$,\r\n\\begin{align*}\r\n\\|\\nabla F_\\mu(x)\\|\\leq L_F \\text{~~and~~} \\|x-\\prox_{\\mu F}(x)\\|\\leq \\mu L_F.\r\n\\end{align*}\r\n\\end{lemma}","sha256_utf8":"8793e469809a1c09056f810a7c621077a0e41c1d70fb3a6d48573a20536c8f06"}]},{"id":"S021","semantic_type":"lemma","kind":"lemma","title":"不同平滑参数的包络比较","source_units":[{"start":14108,"end":14664,"start_line":231,"end_line":242,"raw_tex":"\\begin{lemma}[{\\cite[Lemma~\\ref{l:LL}]{BW}}]\r\n\\label{l:MoreauEnv}\r\nLet $F: \\mathbb{R}^n\\rightarrow (-\\infty,+\\infty]$ be a proper $\\rho$-weakly convex lower semicontinuous function, let $\\mu_1$ and $\\mu_2$ be parameters such that $0< \\mu_2 \\leq \\mu_1 < 1/\\rho$. Then\r\n\\begin{align*}\r\nF_{\\mu_2}(x)\\leq F_{\\mu_1}(x)+\\frac{\\mu_1(\\mu_1-\\mu_2)}{2\\mu_2}\\|\r\n\\nabla F_{\\mu_1}(x)\\|^2.\r\n\\end{align*}\r\nIf $F$ is additionally $L_F$-Lipschitz continuous, then\r\n\\begin{align*}\r\nF_{\\mu_2}\\leq F_{\\mu_1}+\\frac{\\mu_1(\\mu_1-\\mu_2)}{2\\mu_2}(L_F)^2.\r\n\\end{align*}\r\n\\end{lemma}","sha256_utf8":"b14927720c253a121d05cc700f7854400e9c615525c8f27914a544a3e197753a"}]},{"id":"S022","semantic_type":"lemma","kind":"lemma","title":"幂函数增长下界","source_units":[{"start":14725,"end":14906,"start_line":246,"end_line":251,"raw_tex":"\\begin{lemma}\\label{l:estimate}\r\nLet $x\\in [1, +\\infty)$ and $\\alpha\\in [0, 1]$. Then\r\n\\begin{align*}\r\n(1 +x)^{\\alpha} - 1 \\geq \\alpha (\\ln 2) x^{\\alpha}.\r\n\\end{align*}\r\n\\end{lemma}","sha256_utf8":"98a8450605b34c9945448d809c1eee2f91ddf268894bf8306f9d418c426a5132"}]},{"id":"S023","semantic_type":"proof","kind":"proof","title":"幂函数下界完整证明","source_units":[{"start":14908,"end":15830,"start_line":252,"end_line":266,"raw_tex":"\\begin{proof}\r\nLet $\\varphi(x) :=(1 +x)^{\\alpha} -1 - \\alpha (\\ln 2) x^{\\alpha}$. Then $\\psi(\\alpha) =:\\varphi(1) =2^{\\alpha} -1 -\\alpha \\ln 2$. We see that, for all $\\alpha \\geq 0$,\r\n$\\psi'(\\alpha) =(\\ln 2) (2^{\\alpha} -1) \\geq 0$. It follows that $\\varphi(1) =\\psi(\\alpha) \\geq \\psi(0) =0$.\r\n\r\nOn the other hand,\r\n\\begin{align*}\r\n\\varphi'(x) =\\alpha(1 +x)^{\\alpha -1} -\\alpha^2 (\\ln 2) x^{\\alpha -1} =\\alpha x^{\\alpha -1}\\left(\\left(1 +\\frac{1}{x}\\right)^{\\alpha -1} -\\alpha \\ln 2\\right).\r\n\\end{align*}\r\nAs $x \\geq 1$ and $\\alpha -1 \\leq 0$, it holds that\r\n\\begin{align*}\r\n\\left(1 +\\frac{1}{x}\\right)^{\\alpha -1} -\\alpha \\ln 2 &\\geq (1 + 1)^{\\alpha -1} -\\alpha \\ln 2 =2^{\\alpha} -2^{\\alpha -1} -\\alpha \\ln 2 \\\\\r\n&\\geq 2^{\\alpha} -1 -\\alpha \\ln 2 =\\varphi(1) \\geq 0.\r\n\\end{align*}\r\nTherefore, for all $x\\geq 1$, $\\varphi'(x) \\geq 0$, which implies that $\\varphi(x) \\geq \\varphi(1) =0$. The proof is complete.\r\n\\end{proof}","sha256_utf8":"eb28c13922efddb7314d9fb353024e875c8de0fa6e7aa9431c1a3f8a0a531229"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"建立比较函数的初值非负性","source_units":[{"start":14923,"end":15202,"start_line":253,"end_line":254,"raw_tex":"Let $\\varphi(x) :=(1 +x)^{\\alpha} -1 - \\alpha (\\ln 2) x^{\\alpha}$. Then $\\psi(\\alpha) =:\\varphi(1) =2^{\\alpha} -1 -\\alpha \\ln 2$. We see that, for all $\\alpha \\geq 0$,\r\n$\\psi'(\\alpha) =(\\ln 2) (2^{\\alpha} -1) \\geq 0$. It follows that $\\varphi(1) =\\psi(\\alpha) \\geq \\psi(0) =0$.\r\n","sha256_utf8":"eb5104fe0d50a49c8f7d7f5b722655829ab11a7a44b515d03862dba081656250"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"计算比较函数导数","source_units":[{"start":15204,"end":15414,"start_line":256,"end_line":259,"raw_tex":"On the other hand,\r\n\\begin{align*}\r\n\\varphi'(x) =\\alpha(1 +x)^{\\alpha -1} -\\alpha^2 (\\ln 2) x^{\\alpha -1} =\\alpha x^{\\alpha -1}\\left(\\left(1 +\\frac{1}{x}\\right)^{\\alpha -1} -\\alpha \\ln 2\\right).\r\n\\end{align*}\r\n","sha256_utf8":"0cdb395279cb6875267461095e56151aece7a5ad75d2383d71200dbbdf8fa73f"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"证明单调性并取得下界","source_units":[{"start":15414,"end":15819,"start_line":260,"end_line":265,"raw_tex":"As $x \\geq 1$ and $\\alpha -1 \\leq 0$, it holds that\r\n\\begin{align*}\r\n\\left(1 +\\frac{1}{x}\\right)^{\\alpha -1} -\\alpha \\ln 2 &\\geq (1 + 1)^{\\alpha -1} -\\alpha \\ln 2 =2^{\\alpha} -2^{\\alpha -1} -\\alpha \\ln 2 \\\\\r\n&\\geq 2^{\\alpha} -1 -\\alpha \\ln 2 =\\varphi(1) \\geq 0.\r\n\\end{align*}\r\nTherefore, for all $x\\geq 1$, $\\varphi'(x) \\geq 0$, which implies that $\\varphi(x) \\geq \\varphi(1) =0$. The proof is complete.\r\n","sha256_utf8":"69a8ffedecf5ec3048e3568bb88b737cd887933b54e2f8c84d6f2dc7a8265bf1"}]},{"id":"S027","semantic_type":"proof","kind":"proof","title":"梯度映射偏变量连续性证明","source_units":[{"start":19739,"end":20219,"start_line":362,"end_line":371,"raw_tex":"\\begin{remark}\r\n\\label{r:G(x,.)}\r\nFrom the Lipschitz continuity of $\\nabla_x H(x,\\cdot)$ in Assumption~\\ref{a:H}, we have that for all $x\\in \\mathbb{R}^n$ and all $y_1,y_2\\in \\mathbb{R}^m$,\r\n\\begin{align*}\r\n\\|G_\\lambda(x,y_1)&-G_\\lambda(x,y_2)\\| \\notag\\\\\r\n&=\\frac{1}{\\lambda}\\|\\prox_{\\lambda f}(x-\\lambda \\nabla_x H(x,y_2))-\\prox_{\\lambda f}(x-\\lambda \\nabla_x H(x,y_1))\\| \\\\\r\n&\\leq\\|\\nabla_x H(x,y_1)-\\nabla_x H(x,y_2)\\| \\\\\r\n&\\leq L_{12}\\|y_1-y_2\\|.\r\n\\end{align*}\r\n\\end{remark}\r\n","sha256_utf8":"cf5577965543126540746cfa28208ba00ff7ba7544a2a29613b87bff990d86b4"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"用近端非扩张性控制映射变化","source_units":[{"start":19739,"end":20219,"start_line":362,"end_line":371,"raw_tex":"\\begin{remark}\r\n\\label{r:G(x,.)}\r\nFrom the Lipschitz continuity of $\\nabla_x H(x,\\cdot)$ in Assumption~\\ref{a:H}, we have that for all $x\\in \\mathbb{R}^n$ and all $y_1,y_2\\in \\mathbb{R}^m$,\r\n\\begin{align*}\r\n\\|G_\\lambda(x,y_1)&-G_\\lambda(x,y_2)\\| \\notag\\\\\r\n&=\\frac{1}{\\lambda}\\|\\prox_{\\lambda f}(x-\\lambda \\nabla_x H(x,y_2))-\\prox_{\\lambda f}(x-\\lambda \\nabla_x H(x,y_1))\\| \\\\\r\n&\\leq\\|\\nabla_x H(x,y_1)-\\nabla_x H(x,y_2)\\| \\\\\r\n&\\leq L_{12}\\|y_1-y_2\\|.\r\n\\end{align*}\r\n\\end{remark}\r\n","sha256_utf8":"cf5577965543126540746cfa28208ba00ff7ba7544a2a29613b87bff990d86b4"}]},{"id":"S029","semantic_type":"lemma","kind":"lemma","title":"平滑驻点向原问题转移","source_units":[{"start":20223,"end":21078,"start_line":374,"end_line":384,"raw_tex":"\\begin{lemma}\r\n\\label{l:transfer}\r\nSuppose that Assumptions \\ref{a:H} and \\ref{a:g} hold, $A$ is surjective, and $\\mu\\in(0,1/\\rho)$.\r\nLet $\\bar{x} =x^*\\in \\mathbb{R}^n$ and $\\bar{y}=y^*-A^*(AA^*)^{-1}(Ay^*-\\prox_{\\mu g}(Ay^*))$ with $y^*\\in \\mathbb{R}^m$. Then\r\n\\begin{align*}\r\n&\\dist(0,G_\\lambda(\\bar{x},\\bar{y}))+\\dist(-\\nabla_y H(\\bar{x},\\bar{y}),A^*\\partial g(A\\bar{y})) \\\\\r\n&\\leq \\dist(0,G_\\lambda(x^*,y^*)) +\\dist(-\\nabla_yH(x^*,y^*),A^*\\partial g(\\prox_{\\mu g}(Ay^*)))\\\\\r\n&\\quad +(L_{12}+L_{22})L_g\\sigma_{\\min}(A)^{-1}\\mu.\r\n\\end{align*}\r\nConsequently, if $(x^*,y^*)$ is an $\\varepsilon$-approximate stationary point of problem~\\eqref{guanghua}, then $(\\bar{x},\\bar{y})$ is an $\\bar{\\varepsilon}$-approximate stationary point of problem~\\eqref{eq:L1E1} with $\\bar{\\varepsilon} =\\varepsilon +(L_{12}+L_{22})L_g\\sigma_{\\min}(A)^{-1}\\mu$.\r\n\\end{lemma}","sha256_utf8":"9cf7a7411af7ccb9416d643994130b3a352524f480b489dd00c81cd997a310ef"}]},{"id":"S030","semantic_type":"assumption","kind":"assumption","title":"转移引理的局部满射假设","source_units":[{"start":20258,"end":20357,"start_line":376,"end_line":376,"raw_tex":"Suppose that Assumptions \\ref{a:H} and \\ref{a:g} hold, $A$ is surjective, and $\\mu\\in(0,1/\\rho)$.\r\n","sha256_utf8":"68171efc4ed22571f6670879d4f5a36dbd87a5291acead29188b0c9bafdb97fc"}]},{"id":"S031","semantic_type":"proof","kind":"proof","title":"驻点转移完整证明","source_units":[{"start":21080,"end":22965,"start_line":385,"end_line":413,"raw_tex":"\\begin{proof}\r\nSince $\\bar{y}=y^*-A^*(AA^*)^{-1}(Ay^*-\\prox_{\\mu g}(Ay^*))$, multiplying both sides by $A$ yields $A\\bar{y}=\\prox_{\\mu g}(Ay^*)$. We have that\r\n\\begin{align*}\r\n&\\dist(-\\nabla_yH(\\bar{x},\\bar{y}),A^*\\partial g(A\\bar{y})) \\\\\r\n&\\leq \\dist(-\\nabla_yH(x^*,y^*),A^*\\partial g(A\\bar{y})) +\\|\\nabla_yH(x^*,y^*)-\\nabla_yH(\\bar{x},\\bar{y})\\| \\\\\r\n&=\\dist(-\\nabla_yH(x^*,y^*),A^*\\partial g(\\prox_{\\mu g}(Ay^*))) +\\|\\nabla_yH(x^*,y^*)-\\nabla_yH(x^*,\\bar{y})\\| \\\\\r\n&\\leq \\dist(-\\nabla_yH(x^*,y^*),A^*\\nabla g_{\\mu}(Ay^*)) +L_{22}\\|y^* -\\bar{y}\\|,\r\n\\end{align*}\r\nwhere the last inequality uses $A^*\\nabla g_{\\mu}(Ay^*)\\in A^*\\partial g(\\prox_{\\mu g}(Ay^*))$ (see Lemma~\\ref{l:weakcvx}\\ref{l:weakcvx_inclu}) and the Lipschitz continuity of $\\nabla_y H(x^*,\\cdot)$.\r\n\r\nNext, by Remark~\\ref{r:G(x,.)},\r\n\\begin{align*}\r\n\\dist(0,G_\\lambda(\\bar{x},\\bar{y})) &= \\|G_\\lambda(x^*,y^*) -G_\\lambda(x^*,y^*) +G_\\lambda(\\bar{x},\\bar{y})\\| \\\\\r\n&\\leq \\|G_\\lambda(x^*,y^*)\\| +\\|G_\\lambda(x^*,y^*) -G_\\lambda(\\bar{x},\\bar{y})\\| \\\\\r\n&= \\dist(0,G_\\lambda (x^*,y^*)) +\\|G_\\lambda(x^*,y^*) -G_\\lambda(x^*,\\bar{y})\\| \\\\\r\n&\\leq \\dist(0,G_\\lambda (x^*,y^*)) +L_{12}\\|y^* -\\bar{y}\\|.\r\n\\end{align*}\r\nNote that\r\n\\begin{align*}\r\n\\|y^* -\\bar{y}\\| =\\|A^*(AA^*)^{-1}(Ay^*-\\prox_{\\mu g}(Ay^*))\\| \\leq \\sigma_{\\min}(A)^{-1}\\mu L_g\r\n\\end{align*}\r\ndue to Lemma~\\ref{2.6} and the fact that the operator norm of $A^*(AA^*)^{-1}$ is bounded by the inverse of the smallest singular value $\\sigma_{\\min}(A)$ of $A$, i.e., $\\|A^*(AA^*)^{-1}\\|\\leq\\sigma_{\\min}(A)^{-1}$. Altogether, we obtain that\r\n\\begin{align*}\r\n&\\dist(0,G_\\lambda(\\bar{x},\\bar{y})) +\\dist(-\\nabla_y H(\\bar{x},\\bar{y}),A^*\\partial g(A\\bar{y})) \\\\\r\n&\\leq   \\dist(0,G_\\lambda(x^*,y^*)) +\\dist(-\\nabla_yH(x^*,y^*),A^*\\partial g(\\prox_{\\mu g}(Ay^*))) \\\\\r\n&\\quad+(L_{12} +L_{22})L_g\\sigma_{\\min}(A)^{-1}\\mu,\r\n\\end{align*}\r\nwhich completes the proof.\r\n\\end{proof}","sha256_utf8":"9a443b18c5e7c67cd8876a4b943687bb2b28c7ea6007c505fec5b615c374fc64"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"校正线性像并控制纵向驻点残差","source_units":[{"start":21095,"end":21846,"start_line":386,"end_line":393,"raw_tex":"Since $\\bar{y}=y^*-A^*(AA^*)^{-1}(Ay^*-\\prox_{\\mu g}(Ay^*))$, multiplying both sides by $A$ yields $A\\bar{y}=\\prox_{\\mu g}(Ay^*)$. We have that\r\n\\begin{align*}\r\n&\\dist(-\\nabla_yH(\\bar{x},\\bar{y}),A^*\\partial g(A\\bar{y})) \\\\\r\n&\\leq \\dist(-\\nabla_yH(x^*,y^*),A^*\\partial g(A\\bar{y})) +\\|\\nabla_yH(x^*,y^*)-\\nabla_yH(\\bar{x},\\bar{y})\\| \\\\\r\n&=\\dist(-\\nabla_yH(x^*,y^*),A^*\\partial g(\\prox_{\\mu g}(Ay^*))) +\\|\\nabla_yH(x^*,y^*)-\\nabla_yH(x^*,\\bar{y})\\| \\\\\r\n&\\leq \\dist(-\\nabla_yH(x^*,y^*),A^*\\nabla g_{\\mu}(Ay^*)) +L_{22}\\|y^* -\\bar{y}\\|,\r\n\\end{align*}\r\nwhere the last inequality uses $A^*\\nabla g_{\\mu}(Ay^*)\\in A^*\\partial g(\\prox_{\\mu g}(Ay^*))$ (see Lemma~\\ref{l:weakcvx}\\ref{l:weakcvx_inclu}) and the Lipschitz continuity of $\\nabla_y H(x^*,\\cdot)$.\r\n","sha256_utf8":"0d93c84ad65a03bbd39b02a86d8b847589be68e4a3d926e211aefb5f753a47fe"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"控制校正后的近端梯度残差","source_units":[{"start":21848,"end":22255,"start_line":395,"end_line":401,"raw_tex":"Next, by Remark~\\ref{r:G(x,.)},\r\n\\begin{align*}\r\n\\dist(0,G_\\lambda(\\bar{x},\\bar{y})) &= \\|G_\\lambda(x^*,y^*) -G_\\lambda(x^*,y^*) +G_\\lambda(\\bar{x},\\bar{y})\\| \\\\\r\n&\\leq \\|G_\\lambda(x^*,y^*)\\| +\\|G_\\lambda(x^*,y^*) -G_\\lambda(\\bar{x},\\bar{y})\\| \\\\\r\n&= \\dist(0,G_\\lambda (x^*,y^*)) +\\|G_\\lambda(x^*,y^*) -G_\\lambda(x^*,\\bar{y})\\| \\\\\r\n&\\leq \\dist(0,G_\\lambda (x^*,y^*)) +L_{12}\\|y^* -\\bar{y}\\|.\r\n\\end{align*}\r\n","sha256_utf8":"03688b9d49334bc3d7e357d3c1c5d9f209eb9e1db2e5b96a5cdd57d54292d14a"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"利用伪逆范数控制校正位移","source_units":[{"start":22255,"end":22638,"start_line":402,"end_line":406,"raw_tex":"Note that\r\n\\begin{align*}\r\n\\|y^* -\\bar{y}\\| =\\|A^*(AA^*)^{-1}(Ay^*-\\prox_{\\mu g}(Ay^*))\\| \\leq \\sigma_{\\min}(A)^{-1}\\mu L_g\r\n\\end{align*}\r\ndue to Lemma~\\ref{2.6} and the fact that the operator norm of $A^*(AA^*)^{-1}$ is bounded by the inverse of the smallest singular value $\\sigma_{\\min}(A)$ of $A$, i.e., $\\|A^*(AA^*)^{-1}\\|\\leq\\sigma_{\\min}(A)^{-1}$. Altogether, we obtain that\r\n","sha256_utf8":"33cdace6a8f1934e5386c0acd89da4b9e768642c0deb758b03c36c2a5a45798a"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"汇总两项残差与校正误差","source_units":[{"start":22638,"end":22954,"start_line":407,"end_line":412,"raw_tex":"\\begin{align*}\r\n&\\dist(0,G_\\lambda(\\bar{x},\\bar{y})) +\\dist(-\\nabla_y H(\\bar{x},\\bar{y}),A^*\\partial g(A\\bar{y})) \\\\\r\n&\\leq   \\dist(0,G_\\lambda(x^*,y^*)) +\\dist(-\\nabla_yH(x^*,y^*),A^*\\partial g(\\prox_{\\mu g}(Ay^*))) \\\\\r\n&\\quad+(L_{12} +L_{22})L_g\\sigma_{\\min}(A)^{-1}\\mu,\r\n\\end{align*}\r\nwhich completes the proof.\r\n","sha256_utf8":"34fac84f0f28d2b0e7584b480e43aee151a0d47ddb039d74246295da79109b25"}]},{"id":"S036","semantic_type":"proof","kind":"proof","title":"近端更新映射恒等式推导","source_units":[{"start":26326,"end":26780,"start_line":477,"end_line":482,"raw_tex":"Recalling that $G_\\lambda(x,y) =\\frac{1}{\\lambda}(x -\\prox_{\\lambda f}(x -\\lambda\\nabla_x  H(x,y)))$, we have\r\n\\begin{align}\\label{zhu4.2}\r\nx^{k+1}&=\\prox_{\\sigma_k f}(\\bar{x}^k-\\sigma_k\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1}))\\notag \\\\\r\n&=\\bar{x}^k-\\sigma_k\\left(\\frac{1}{\\sigma_k}(\\bar{x}^k-\\prox_{\\sigma_k f}(\\bar{x}^k-\\sigma_k\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1})))\\right)\\notag \\\\\r\n&=\\bar{x}^k-\\sigma_k G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1}).\r\n\\end{align}\r\n","sha256_utf8":"0f15c2dcd83973269c049970359daa72d2ad38e98c4cd91294539f254ad7da03"}]},{"id":"S037","semantic_type":"proof_step","kind":"proof_step","title":"将近端更新写成梯度映射步","source_units":[{"start":26326,"end":26780,"start_line":477,"end_line":482,"raw_tex":"Recalling that $G_\\lambda(x,y) =\\frac{1}{\\lambda}(x -\\prox_{\\lambda f}(x -\\lambda\\nabla_x  H(x,y)))$, we have\r\n\\begin{align}\\label{zhu4.2}\r\nx^{k+1}&=\\prox_{\\sigma_k f}(\\bar{x}^k-\\sigma_k\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1}))\\notag \\\\\r\n&=\\bar{x}^k-\\sigma_k\\left(\\frac{1}{\\sigma_k}(\\bar{x}^k-\\prox_{\\sigma_k f}(\\bar{x}^k-\\sigma_k\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1})))\\right)\\notag \\\\\r\n&=\\bar{x}^k-\\sigma_k G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1}).\r\n\\end{align}\r\n","sha256_utf8":"0f15c2dcd83973269c049970359daa72d2ad38e98c4cd91294539f254ad7da03"}]},{"id":"S038","semantic_type":"lemma","kind":"lemma","title":"算法的一步充分下降界","source_units":[{"start":26822,"end":27500,"start_line":485,"end_line":494,"raw_tex":"\\begin{lemma}\r\n\\label{l:LL}\r\nSuppose that Assumptions \\ref{a:H} and \\ref{a:g} hold. Let $(\\bar{x}^k,\\bar{y}^k)$ be the sequence generated by Algorithm~\\ref{algo}. Then\r\n\\begin{align*}\r\n\\mathcal{L}_{\\mu_{k}}(\\bar{x}^{k+1},\\bar{y}^{k+1}) &\\leq \\mathcal{L}_{\\mu_{k}}(\\bar{x}^k,\\bar{y}^k)\r\n-\\frac{1}{2}\\delta\\tau_k\\|A^*\\nabla g_{\\mu_k}(A\\bar{y}^k)+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)\\|^2\\notag \\\\\r\n&\\quad -\\frac{1}{4}\\kappa\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)\\|^2 +(\\mu_k-\\mu_{k+1})L_g^2,\r\n\\end{align*}\r\nwhere $\\delta :=1 -\\alpha^2 -\\frac{L_{12}^2(1+\\alpha)^2}{L_{11}(L_{22} +2\\rho\\|A\\|^2)} >0$ and $\\kappa :=\\min\\{(2 -L_{11}\\beta\\sigma)\\beta, 2 -L_{11}\\sigma\\} >0$.\r\n\\end{lemma}","sha256_utf8":"4adc24021618285f5b6ae1f2a48898aaa2b23c648ab89a16b586b0e01bcf7f17"}]},{"id":"S039","semantic_type":"proof","kind":"proof","title":"充分下降完整证明","source_units":[{"start":27502,"end":35060,"start_line":495,"end_line":606,"raw_tex":"\\begin{proof}\r\nFrom the Lipschitz continuity of $\\nabla_x \\Phi_k(\\cdot,\\bar{y}^{k+1})$ and $\\nabla_y\\Phi_k(\\bar{x}^k, \\cdot)$, Lemma \\ref{l:descent} implies that\r\n\\begin{align*}\r\n\\Phi_k(\\bar{x}^{k+1},\\bar{y}^{k+1}) &\\leq\\Phi_k(\\bar{x}^{k},\\bar{y}^{k+1}) +\\langle\\nabla_x\\Phi_k(\\bar{x}^k,\\bar{y}^{k+1}),\\bar{x}^{k+1}-\\bar{x}^k\\rangle+\\frac{L_{11}}{2}\\|\\bar{x}^{k+1}-\\bar{x}^k\\|^2, \\\\\r\n\\Phi_k(\\bar{x}^k,\\bar{y}^{k+1}) &\\leq \\Phi_k(\\bar{x}^k,\\bar{y}^k) +\\langle\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k),\\bar{y}^{k+1}-\\bar{y}^k\\rangle+\\frac{L_k}{2}\\|\\bar{y}^{k+1}-\\bar{y}^k\\|^2.\r\n\\end{align*}\r\nWe note that\r\n\\begin{align}\\label{eq:grad_y Phi_k}\r\n\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k) =A^*\\nabla g_{\\mu_k}(A\\bar{y}^k)+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)\r\n\\end{align}\r\nand from \\eqref{s21}, \\eqref{s41}, \\eqref{s42}, and \\eqref{zhu4.2} that\r\n\\begin{align*}\r\n\\bar{x}^{k+1}-\\bar{x}^k &=\\beta_k(x^{k+1}-\\bar{x}^k)=-\\beta_k\\sigma_k G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1}), \\\\\r\n\\bar{y}^{k+1}-\\bar{y}^k &=(1+\\alpha_k)(y^{k+1}-\\bar{y}^k) =-(1+\\alpha_k)\\tau_k \\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k).\r\n\\end{align*}\r\nTherefore,\r\n\\begin{align}\\label{8.7}\r\n&\\Phi_k(\\bar{x}^{k+1},\\bar{y}^{k+1}) \\notag \\\\\r\n&\\leq \\Phi_k(\\bar{x}^k,\\bar{y}^k) +\\langle\\nabla_x\\Phi_k(\\bar{x}^k,\\bar{y}^{k+1}),\\bar{x}^{k+1}-\\bar{x}^k\\rangle+\\frac{L_{11}}{2}\\|\\bar{x}^{k+1}-\\bar{x}^k\\|^2 \\notag \\\\\r\n&\\quad +\\langle\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k),\\bar{y}^{k+1}-\\bar{y}^k\\rangle+\\frac{L_k}{2}\\|\\bar{y}^{k+1}-\\bar{y}^k\\|^2 \\notag \\\\\r\n&= \\Phi_k(\\bar{x}^k,\\bar{y}^k) -\\beta_k\\sigma_k\\langle\\nabla_x\\Phi_k(\\bar{x}^k,\\bar{y}^{k+1}),G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\rangle \\notag \\\\\r\n&\\quad +\\frac{1}{2}L_{11}\\beta_k^2\\sigma_k^2\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 \\notag\\\\\r\n&\\quad -\\frac{1}{2}(2 +2\\alpha_k -L_k(1+\\alpha_k)^2\\tau_k)\\tau_k\\|\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k)\\|^2.\r\n\\end{align}\r\nBy Lemma \\ref{l:MoreauEnv} and the fact that $\\mu_{k+1}\\in [\\mu_k/2, \\mu_k]$,\r\n\\begin{align}\\label{8.8}\r\ng_{\\mu_{k+1}}(Ay)\\leq g_{\\mu_k}(Ay) +\\frac{\\mu_k(\\mu_k-\\mu_{k+1})}{2\\mu_{k+1}}L_g^2\\leq g_{\\mu_k}(Ay) +(\\mu_k-\\mu_{k+1})L_g^2.\r\n\\end{align}\r\nAdding $H(x,y)$ to both sides of \\eqref{8.8} yields\r\n\\begin{align}\\label{3.4}\r\n\\Phi_{k+1}(x,y)\\leq \\Phi_k(x,y) +(\\mu_k-\\mu_{k+1})L_g^2.\r\n\\end{align}\r\nLetting $x =\\bar{x}^{k+1}$, $y =\\bar{y}^{k+1}$ in \\eqref{3.4} and combining it with \\eqref{8.7}, we obtain that\r\n\\begin{align}\\label{4.4}\r\n&\\Phi_{k+1}(\\bar{x}^{k+1},\\bar{y}^{k+1}) \\notag \\\\\r\n&\\leq \\Phi_k(\\bar{x}^k,\\bar{y}^k) -\\beta_k\\sigma_k\\langle\\nabla_x\\Phi_k(\\bar{x}^k,\\bar{y}^{k+1}),G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\rangle \\notag \\\\\r\n&\\quad +\\frac{1}{2}L_{11}\\beta_k^2\\sigma_k^2\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 \\notag\\\\\r\n&\\quad -\\frac{1}{2}(2 +2\\alpha_k -L_k(1+\\alpha_k)^2\\tau_k)\\tau_k\\|\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k)\\|^2 +(\\mu_k-\\mu_{k+1})L_g^2.\r\n\\end{align}\r\nBy the first-order optimality condition of \\eqref{s31},\r\n\\begin{align*}\r\n0\\in \\partial f(x^{k+1})+\\frac{1}{\\sigma_k}(x^{k+1}-\\bar{x}^k+\\sigma_k\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1})),\r\n\\end{align*}\r\nor equivalently,\r\n\\begin{align*}\r\n\\frac{1}{\\sigma_k}(\\bar{x}^k -x^{k+1}) -\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1})\\in \\partial f(x^{k+1}).\r\n\\end{align*}\r\nCombining with the convexity of $f$ and \\eqref{zhu4.2}, it follows that\r\n\\begin{align}\\label{8.9}\r\nf(x^{k+1}) &\\leq f(\\bar{x}^k) -\\left\\langle \\frac{1}{\\sigma_k}(\\bar{x}^k -x^{k+1}) -\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1}), \\bar{x}^k -x^{k+1}\\right\\rangle \\notag \\\\\r\n&= f(\\bar{x}^k) -\\frac{1}{\\sigma_k}\\|\\bar{x}^k -x^{k+1}\\|^2 +\\langle \\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1}), \\bar{x}^k -x^{k+1}\\rangle\\notag \\\\\r\n&= f(\\bar{x}^k) -\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 +\\sigma_k\\langle\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1}),G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\rangle.\r\n\\end{align}\r\nAgain by the convexity of $f$ and \\eqref{s42},\r\n\\begin{align*}\r\nf(\\bar{x}^{k+1})&=f\\big(\\beta_k x^{k+1}+(1-\\beta_k)\\bar{x}^k\\big)\r\n\\leq \\beta_k f(x^{k+1})+(1-\\beta_k)f(\\bar{x}^k).\r\n\\end{align*}\r\nThis together with \\eqref{8.9} gives\r\n\\begin{align}\\label{4.5}\r\nf(\\bar{x}^{k+1})&\\leq f(\\bar{x}^k) -\\beta_k\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 \\notag \\\\\r\n&\\quad +\\beta_k\\sigma_k\\langle\\nabla_xH(\\bar{x}^k,\\bar{y}^{k+1}),G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\rangle.\r\n\\end{align}\r\nWe now combine \\eqref{4.4} and \\eqref{4.5} to obtain\r\n\\begin{align}\\label{4.10}\r\n&\\mathcal{L}_{\\mu_{k}}(\\bar{x}^{k+1},\\bar{y}^{k+1}) \\notag \\\\\r\n&\\leq \\mathcal{L}_{\\mu_{k}}(\\bar{x}^k,\\bar{y}^k) -\\frac{1}{2}(2 +2\\alpha_k -L_k(1+\\alpha_k)^2\\tau_k)\\tau_k\\|\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k)\\|^2 \\notag \\\\ &\\quad -\\frac{1}{2}(2 -L_{11}\\beta_k\\sigma_k)\\beta_k\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 +(\\mu_k-\\mu_{k+1})L_g^2.\r\n\\end{align}\r\nNext, using Remark~\\ref{r:G(x,.)},\r\n\\begin{align*}\r\n\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)\\|&=\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)-G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})+G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|\\notag \\\\\r\n&\\leq\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)-G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|+\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|\\notag \\\\\r\n&\\leq L_{12}\\|\\bar{y}^{k+1}-\\bar{y}^k\\|+\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|.\r\n\\end{align*}\r\nSquaring and applying the inequality $\\frac{1}{2}(a+b)^2 \\leq a^2 +b^2$, we get\r\n\\begin{align*}\r\n\\frac{1}{2}\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)\\|^2 &\\leq L_{12}^2\\|\\bar{y}^{k+1}-\\bar{y}^k\\|^2 +\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 \\\\\r\n&= L_{12}^2(1+\\alpha_k)^2\\tau_k^2\\|\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k)\\|^2 +\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2.\r\n\\end{align*}\r\nwhich, combined with \\eqref{eq:grad_y Phi_k} and \\eqref{4.10}, implies that\r\n\\begin{align*}\r\n\\mathcal{L}_{\\mu_{k}}(\\bar{x}^{k+1},\\bar{y}^{k+1}) &\\leq \\mathcal{L}_{\\mu_{k}}(\\bar{x}^k,\\bar{y}^k) -\\frac{1}{2}\\delta_k\\tau_k\\|A^*\\nabla g_{\\mu_k}(A\\bar{y}^k)+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)\\|^2 \\notag \\\\\r\n&\\quad -\\frac{1}{4}\\kappa_k\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 +(\\mu_k-\\mu_{k+1})L_g^2,\r\n\\end{align*}\r\nwhere\r\n\\begin{align*}\r\n\\delta_k &:=2 +2\\alpha_k -L_k(1+\\alpha_k)^2\\tau_k -(2 -L_{11}\\beta_k\\sigma_k)\\sigma_k\\beta_kL_{12}^2(1+\\alpha_k)^2\\tau_k, \\\\\r\n\\kappa_k &:=(2 -L_{11}\\beta_k\\sigma_k)\\beta_k.\r\n\\end{align*}\r\nNoting that $\\tau_k\\leq 1/L_k$ and that\r\n\\begin{align*}\r\n(2 -L_{11}\\beta_k\\sigma_k)\\sigma_k\\beta_k\\leq \\frac{1}{L_{11}}\\left(\\frac{2 -L_{11}\\beta_k\\sigma_k +L_{11}\\beta_k\\sigma_k}{2}\\right)^2 =\\frac{1}{L_{11}},\r\n\\end{align*}\r\nwe have\r\n\\begin{align*}\r\n\\delta_k &\\geq 2 +2\\alpha_k -(1+\\alpha_k)^2 -\\frac{L_{12}^2}{L_{11}L_k}(1+\\alpha_k)^2 \\\\\r\n&= 1 -\\alpha_k^2 -\\frac{L_{12}^2}{L_{11}L_k}(1+\\alpha_k)^2 \\\\\r\n&\\geq \\delta =1 -\\alpha^2 -\\frac{L_{12}^2(1+\\alpha)^2}{L_{11}(L_{22} +2\\rho\\|A\\|^2)},\r\n\\end{align*}\r\nwhere the last inequality follows from from the fact that $\\alpha_k\\in [-\\alpha, \\alpha]\\subseteq [-1, \\alpha]$ and that $L_k =L_{22} +\\|A\\|^2\\max\\{\\frac{1}{\\mu_k}, \\frac{\\rho}{1 -\\rho\\mu_k}\\}\\geq L_{22} +2\\rho\\|A\\|^2$ since\r\n\\begin{align*}\r\n\\max\\left\\{\\frac{1}{\\mu_k}, \\frac{\\rho}{1 -\\rho\\mu_k}\\right\\} =\\begin{cases}\r\n\\frac{1}{\\mu_k} &\\text{if~} 0 < \\mu_k \\leq \\frac{1}{2\\rho}, \\\\\r\n\\frac{\\rho}{1 -\\rho\\mu_k} &\\text{if~} \\frac{1}{2\\rho} < \\mu_k < \\frac{1}{\\rho}.\r\n\\end{cases}\r\n\\end{align*}\r\nFinally, we derive from $\\sigma_k\\in \\sigma$ and $\\beta_k\\in [\\beta, 1]$ that\r\n\\begin{align*}\r\n\\kappa_k =(2 -L_{11}\\beta_k\\sigma_k)\\beta_k \\geq (2 -L_{11}\\beta_k\\sigma)\\beta_k \\geq \\kappa =\\min\\{(2 -L_{11}\\beta\\sigma)\\beta, 2 -L_{11}\\sigma\\}\r\n\\end{align*}\r\nas $(2 -L_{11}\\beta_k\\sigma)\\beta_k$ is a quadratic function of $\\beta_k$ with leading coefficient $-L_{11}\\sigma <0$. This completes the proof.\r\n\\end{proof}","sha256_utf8":"4bc7b604fa8fb839444d2f275bd50e6fc8457ad7d96faa4bcb2bd71077f76f68"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"对两个变量分别应用下降引理","source_units":[{"start":27517,"end":28086,"start_line":496,"end_line":500,"raw_tex":"From the Lipschitz continuity of $\\nabla_x \\Phi_k(\\cdot,\\bar{y}^{k+1})$ and $\\nabla_y\\Phi_k(\\bar{x}^k, \\cdot)$, Lemma \\ref{l:descent} implies that\r\n\\begin{align*}\r\n\\Phi_k(\\bar{x}^{k+1},\\bar{y}^{k+1}) &\\leq\\Phi_k(\\bar{x}^{k},\\bar{y}^{k+1}) +\\langle\\nabla_x\\Phi_k(\\bar{x}^k,\\bar{y}^{k+1}),\\bar{x}^{k+1}-\\bar{x}^k\\rangle+\\frac{L_{11}}{2}\\|\\bar{x}^{k+1}-\\bar{x}^k\\|^2, \\\\\r\n\\Phi_k(\\bar{x}^k,\\bar{y}^{k+1}) &\\leq \\Phi_k(\\bar{x}^k,\\bar{y}^k) +\\langle\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k),\\bar{y}^{k+1}-\\bar{y}^k\\rangle+\\frac{L_k}{2}\\|\\bar{y}^{k+1}-\\bar{y}^k\\|^2.\r\n\\end{align*}\r\n","sha256_utf8":"9965e9500d41c0750d7c13c8de3585a3af633e7dfd2952d4289f5b1f4aa08791"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"表达平滑梯度和迭代增量","source_units":[{"start":28086,"end":28586,"start_line":501,"end_line":509,"raw_tex":"We note that\r\n\\begin{align}\\label{eq:grad_y Phi_k}\r\n\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k) =A^*\\nabla g_{\\mu_k}(A\\bar{y}^k)+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)\r\n\\end{align}\r\nand from \\eqref{s21}, \\eqref{s41}, \\eqref{s42}, and \\eqref{zhu4.2} that\r\n\\begin{align*}\r\n\\bar{x}^{k+1}-\\bar{x}^k &=\\beta_k(x^{k+1}-\\bar{x}^k)=-\\beta_k\\sigma_k G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1}), \\\\\r\n\\bar{y}^{k+1}-\\bar{y}^k &=(1+\\alpha_k)(y^{k+1}-\\bar{y}^k) =-(1+\\alpha_k)\\tau_k \\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k).\r\n\\end{align*}\r\n","sha256_utf8":"8f3298df051bfa281356ad269058066786ecc6a4b7dcb717d3c096b2cb534261"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"合并平滑部分的两步下降","source_units":[{"start":28586,"end":29347,"start_line":510,"end_line":518,"raw_tex":"Therefore,\r\n\\begin{align}\\label{8.7}\r\n&\\Phi_k(\\bar{x}^{k+1},\\bar{y}^{k+1}) \\notag \\\\\r\n&\\leq \\Phi_k(\\bar{x}^k,\\bar{y}^k) +\\langle\\nabla_x\\Phi_k(\\bar{x}^k,\\bar{y}^{k+1}),\\bar{x}^{k+1}-\\bar{x}^k\\rangle+\\frac{L_{11}}{2}\\|\\bar{x}^{k+1}-\\bar{x}^k\\|^2 \\notag \\\\\r\n&\\quad +\\langle\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k),\\bar{y}^{k+1}-\\bar{y}^k\\rangle+\\frac{L_k}{2}\\|\\bar{y}^{k+1}-\\bar{y}^k\\|^2 \\notag \\\\\r\n&= \\Phi_k(\\bar{x}^k,\\bar{y}^k) -\\beta_k\\sigma_k\\langle\\nabla_x\\Phi_k(\\bar{x}^k,\\bar{y}^{k+1}),G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\rangle \\notag \\\\\r\n&\\quad +\\frac{1}{2}L_{11}\\beta_k^2\\sigma_k^2\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 \\notag\\\\\r\n&\\quad -\\frac{1}{2}(2 +2\\alpha_k -L_k(1+\\alpha_k)^2\\tau_k)\\tau_k\\|\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k)\\|^2.\r\n\\end{align}\r\n","sha256_utf8":"ce14b3c27a63dad83f5f37edc2049ad89d87bd726603cdab319b8366b23714c8"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"估计平滑参数变化的代价","source_units":[{"start":29347,"end":29743,"start_line":519,"end_line":526,"raw_tex":"By Lemma \\ref{l:MoreauEnv} and the fact that $\\mu_{k+1}\\in [\\mu_k/2, \\mu_k]$,\r\n\\begin{align}\\label{8.8}\r\ng_{\\mu_{k+1}}(Ay)\\leq g_{\\mu_k}(Ay) +\\frac{\\mu_k(\\mu_k-\\mu_{k+1})}{2\\mu_{k+1}}L_g^2\\leq g_{\\mu_k}(Ay) +(\\mu_k-\\mu_{k+1})L_g^2.\r\n\\end{align}\r\nAdding $H(x,y)$ to both sides of \\eqref{8.8} yields\r\n\\begin{align}\\label{3.4}\r\n\\Phi_{k+1}(x,y)\\leq \\Phi_k(x,y) +(\\mu_k-\\mu_{k+1})L_g^2.\r\n\\end{align}\r\n","sha256_utf8":"7865124be1960d2b4da8104ced85c377770b5d23509e0dc26b856135f70cf858"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"加入平滑参数变化项","source_units":[{"start":29743,"end":30330,"start_line":527,"end_line":533,"raw_tex":"Letting $x =\\bar{x}^{k+1}$, $y =\\bar{y}^{k+1}$ in \\eqref{3.4} and combining it with \\eqref{8.7}, we obtain that\r\n\\begin{align}\\label{4.4}\r\n&\\Phi_{k+1}(\\bar{x}^{k+1},\\bar{y}^{k+1}) \\notag \\\\\r\n&\\leq \\Phi_k(\\bar{x}^k,\\bar{y}^k) -\\beta_k\\sigma_k\\langle\\nabla_x\\Phi_k(\\bar{x}^k,\\bar{y}^{k+1}),G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\rangle \\notag \\\\\r\n&\\quad +\\frac{1}{2}L_{11}\\beta_k^2\\sigma_k^2\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 \\notag\\\\\r\n&\\quad -\\frac{1}{2}(2 +2\\alpha_k -L_k(1+\\alpha_k)^2\\tau_k)\\tau_k\\|\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k)\\|^2 +(\\mu_k-\\mu_{k+1})L_g^2.\r\n\\end{align}\r\n","sha256_utf8":"fc107d2ed9174552b589088e6dae78e437bd1f7bcd864e90dff889d7222ee208"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"由近端最优性推得凸项下降","source_units":[{"start":30330,"end":31261,"start_line":534,"end_line":547,"raw_tex":"By the first-order optimality condition of \\eqref{s31},\r\n\\begin{align*}\r\n0\\in \\partial f(x^{k+1})+\\frac{1}{\\sigma_k}(x^{k+1}-\\bar{x}^k+\\sigma_k\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1})),\r\n\\end{align*}\r\nor equivalently,\r\n\\begin{align*}\r\n\\frac{1}{\\sigma_k}(\\bar{x}^k -x^{k+1}) -\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1})\\in \\partial f(x^{k+1}).\r\n\\end{align*}\r\nCombining with the convexity of $f$ and \\eqref{zhu4.2}, it follows that\r\n\\begin{align}\\label{8.9}\r\nf(x^{k+1}) &\\leq f(\\bar{x}^k) -\\left\\langle \\frac{1}{\\sigma_k}(\\bar{x}^k -x^{k+1}) -\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1}), \\bar{x}^k -x^{k+1}\\right\\rangle \\notag \\\\\r\n&= f(\\bar{x}^k) -\\frac{1}{\\sigma_k}\\|\\bar{x}^k -x^{k+1}\\|^2 +\\langle \\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1}), \\bar{x}^k -x^{k+1}\\rangle\\notag \\\\\r\n&= f(\\bar{x}^k) -\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 +\\sigma_k\\langle\\nabla_x H(\\bar{x}^k,\\bar{y}^{k+1}),G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\rangle.\r\n\\end{align}\r\n","sha256_utf8":"8f0167176dbbd5b9bcf11949be1efa64166124308730a700022b74db6b8c2771"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"用凸性处理松弛近端更新","source_units":[{"start":31261,"end":31751,"start_line":548,"end_line":557,"raw_tex":"Again by the convexity of $f$ and \\eqref{s42},\r\n\\begin{align*}\r\nf(\\bar{x}^{k+1})&=f\\big(\\beta_k x^{k+1}+(1-\\beta_k)\\bar{x}^k\\big)\r\n\\leq \\beta_k f(x^{k+1})+(1-\\beta_k)f(\\bar{x}^k).\r\n\\end{align*}\r\nThis together with \\eqref{8.9} gives\r\n\\begin{align}\\label{4.5}\r\nf(\\bar{x}^{k+1})&\\leq f(\\bar{x}^k) -\\beta_k\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 \\notag \\\\\r\n&\\quad +\\beta_k\\sigma_k\\langle\\nabla_xH(\\bar{x}^k,\\bar{y}^{k+1}),G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\rangle.\r\n\\end{align}\r\n","sha256_utf8":"d4c8abd585eb92c940cdaaa51e769dd713e55125b0b4e0e3375a40bc70faa440"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"合并整个目标的下降项","source_units":[{"start":31751,"end":32196,"start_line":558,"end_line":562,"raw_tex":"We now combine \\eqref{4.4} and \\eqref{4.5} to obtain\r\n\\begin{align}\\label{4.10}\r\n&\\mathcal{L}_{\\mu_{k}}(\\bar{x}^{k+1},\\bar{y}^{k+1}) \\notag \\\\\r\n&\\leq \\mathcal{L}_{\\mu_{k}}(\\bar{x}^k,\\bar{y}^k) -\\frac{1}{2}(2 +2\\alpha_k -L_k(1+\\alpha_k)^2\\tau_k)\\tau_k\\|\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k)\\|^2 \\notag \\\\ &\\quad -\\frac{1}{2}(2 -L_{11}\\beta_k\\sigma_k)\\beta_k\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 +(\\mu_k-\\mu_{k+1})L_g^2.\r\n\\end{align}\r\n","sha256_utf8":"70a2e550c4e49ffc6d29f0150cfc8c679ca4f8ef2c2ade636b6ed855dbb05730"}]},{"id":"S048","semantic_type":"proof_step","kind":"proof_step","title":"比较新旧纵变量上的梯度映射","source_units":[{"start":32196,"end":33019,"start_line":563,"end_line":573,"raw_tex":"Next, using Remark~\\ref{r:G(x,.)},\r\n\\begin{align*}\r\n\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)\\|&=\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)-G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})+G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|\\notag \\\\\r\n&\\leq\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)-G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|+\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|\\notag \\\\\r\n&\\leq L_{12}\\|\\bar{y}^{k+1}-\\bar{y}^k\\|+\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|.\r\n\\end{align*}\r\nSquaring and applying the inequality $\\frac{1}{2}(a+b)^2 \\leq a^2 +b^2$, we get\r\n\\begin{align*}\r\n\\frac{1}{2}\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)\\|^2 &\\leq L_{12}^2\\|\\bar{y}^{k+1}-\\bar{y}^k\\|^2 +\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 \\\\\r\n&= L_{12}^2(1+\\alpha_k)^2\\tau_k^2\\|\\nabla_y\\Phi_k(\\bar{x}^k,\\bar{y}^k)\\|^2 +\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2.\r\n\\end{align*}\r\n","sha256_utf8":"3ff83acfbc965e899ed8bd8e4436214b026ea86771c56fb7b3e91d4cd0d3e715"}]},{"id":"S049","semantic_type":"proof_step","kind":"proof_step","title":"引入两项下降系数","source_units":[{"start":33019,"end":33649,"start_line":574,"end_line":583,"raw_tex":"which, combined with \\eqref{eq:grad_y Phi_k} and \\eqref{4.10}, implies that\r\n\\begin{align*}\r\n\\mathcal{L}_{\\mu_{k}}(\\bar{x}^{k+1},\\bar{y}^{k+1}) &\\leq \\mathcal{L}_{\\mu_{k}}(\\bar{x}^k,\\bar{y}^k) -\\frac{1}{2}\\delta_k\\tau_k\\|A^*\\nabla g_{\\mu_k}(A\\bar{y}^k)+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)\\|^2 \\notag \\\\\r\n&\\quad -\\frac{1}{4}\\kappa_k\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^{k+1})\\|^2 +(\\mu_k-\\mu_{k+1})L_g^2,\r\n\\end{align*}\r\nwhere\r\n\\begin{align*}\r\n\\delta_k &:=2 +2\\alpha_k -L_k(1+\\alpha_k)^2\\tau_k -(2 -L_{11}\\beta_k\\sigma_k)\\sigma_k\\beta_kL_{12}^2(1+\\alpha_k)^2\\tau_k, \\\\\r\n\\kappa_k &:=(2 -L_{11}\\beta_k\\sigma_k)\\beta_k.\r\n\\end{align*}\r\n","sha256_utf8":"442b8f28260dc4b027e40cf5c8ba923ff2f8fbad43e93e9fffbcae2ca3f43999"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"建立纵向下降系数的统一正下界","source_units":[{"start":33649,"end":34646,"start_line":584,"end_line":600,"raw_tex":"Noting that $\\tau_k\\leq 1/L_k$ and that\r\n\\begin{align*}\r\n(2 -L_{11}\\beta_k\\sigma_k)\\sigma_k\\beta_k\\leq \\frac{1}{L_{11}}\\left(\\frac{2 -L_{11}\\beta_k\\sigma_k +L_{11}\\beta_k\\sigma_k}{2}\\right)^2 =\\frac{1}{L_{11}},\r\n\\end{align*}\r\nwe have\r\n\\begin{align*}\r\n\\delta_k &\\geq 2 +2\\alpha_k -(1+\\alpha_k)^2 -\\frac{L_{12}^2}{L_{11}L_k}(1+\\alpha_k)^2 \\\\\r\n&= 1 -\\alpha_k^2 -\\frac{L_{12}^2}{L_{11}L_k}(1+\\alpha_k)^2 \\\\\r\n&\\geq \\delta =1 -\\alpha^2 -\\frac{L_{12}^2(1+\\alpha)^2}{L_{11}(L_{22} +2\\rho\\|A\\|^2)},\r\n\\end{align*}\r\nwhere the last inequality follows from from the fact that $\\alpha_k\\in [-\\alpha, \\alpha]\\subseteq [-1, \\alpha]$ and that $L_k =L_{22} +\\|A\\|^2\\max\\{\\frac{1}{\\mu_k}, \\frac{\\rho}{1 -\\rho\\mu_k}\\}\\geq L_{22} +2\\rho\\|A\\|^2$ since\r\n\\begin{align*}\r\n\\max\\left\\{\\frac{1}{\\mu_k}, \\frac{\\rho}{1 -\\rho\\mu_k}\\right\\} =\\begin{cases}\r\n\\frac{1}{\\mu_k} &\\text{if~} 0 < \\mu_k \\leq \\frac{1}{2\\rho}, \\\\\r\n\\frac{\\rho}{1 -\\rho\\mu_k} &\\text{if~} \\frac{1}{2\\rho} < \\mu_k < \\frac{1}{\\rho}.\r\n\\end{cases}\r\n\\end{align*}\r\n","sha256_utf8":"355300b86acceda45e2ed60782ce08b9f196a6e05d3c820445ac7ecb3b990347"}]},{"id":"S051","semantic_type":"proof_step","kind":"proof_step","title":"建立近端下降系数的统一正下界","source_units":[{"start":34646,"end":35049,"start_line":601,"end_line":605,"raw_tex":"Finally, we derive from $\\sigma_k\\in \\sigma$ and $\\beta_k\\in [\\beta, 1]$ that\r\n\\begin{align*}\r\n\\kappa_k =(2 -L_{11}\\beta_k\\sigma_k)\\beta_k \\geq (2 -L_{11}\\beta_k\\sigma)\\beta_k \\geq \\kappa =\\min\\{(2 -L_{11}\\beta\\sigma)\\beta, 2 -L_{11}\\sigma\\}\r\n\\end{align*}\r\nas $(2 -L_{11}\\beta_k\\sigma)\\beta_k$ is a quadratic function of $\\beta_k$ with leading coefficient $-L_{11}\\sigma <0$. This completes the proof.\r\n","sha256_utf8":"86bb563cd904631a4e540d11a06a09dbef865cdb6078892f98c68dfdb504430d"}]},{"id":"S052","semantic_type":"lemma","kind":"lemma","title":"平滑驻点残差的迭代速率","source_units":[{"start":35314,"end":36054,"start_line":614,"end_line":625,"raw_tex":"\\begin{lemma}\r\n\\label{l:e-stationary}\r\nSuppose that Assumptions~\\ref{a:H} and \\ref{a:g} hold. Let $(\\bar{x}^k,\\bar{y}^k)$ be the sequence generated by Algorithm~\\ref{algo}. Then\r\n\\begin{align*}\r\n\\min_{1\\leq j\\leq k}\\left(\\|A^*\\nabla g_{\\mu_j}(A\\bar{y}^j)+\\nabla_y H(\\bar{x}^j,\\bar{y}^j)\\|+\\|G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j)\\|\\right)\r\n\\leq \\Theta k^{\\frac{\\theta-1}{2}},\r\n\\end{align*}\r\nwhere $\\Theta :=\\sqrt{2(\\ln 2)^{-1}M^{-1}(\\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}^*+\\mu_1L_g^2)}$ and\r\n\\begin{align*}\r\nM :=\\min\\left\\{\\frac{(1 -\\alpha^2)\\eta}{2} -\\frac{L_{12}^2(1+\\alpha)^2\\eta}{2L_{11}(L_{22} +2\\rho\\|A\\|^2)}, \\frac{(2 -L_{11}\\beta\\sigma)\\beta\\gamma}{4}, \\frac{(2 -L_{11}\\sigma)\\gamma}{4}\\right\\}.\r\n\\end{align*}\r\n\\end{lemma}","sha256_utf8":"f9df5a4947eda7fb64581f3363403f6a9999d6c8e7d348631679662af89cd51e"}]},{"id":"S053","semantic_type":"proof","kind":"proof","title":"平滑残差速率完整证明","source_units":[{"start":36056,"end":38400,"start_line":626,"end_line":658,"raw_tex":"\\begin{proof}\r\nAccording to Lemma~\\ref{l:LL},\r\n\\begin{align}\\label{4.10'}\r\n\\mathcal{L}_{\\mu_{k}}(\\bar{x}^{k+1},\\bar{y}^{k+1}) &\\leq \\mathcal{L}_{\\mu_{k}}(\\bar{x}^k,\\bar{y}^k) -\\frac{1}{2}\\delta\\tau_k\\|A^*\\nabla g_{\\mu_k}(A\\bar{y}^k)+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)\\|^2\\notag \\\\\r\n&\\quad -\\frac{1}{4}\\kappa\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)\\|^2 +(\\mu_k-\\mu_{k+1})L_g^2.\r\n\\end{align}\r\nSumming \\eqref{4.10'} from $k=1$ to $k=K$, we have\r\n\\begin{align}\\label{4.11}\r\n&\\sum_{k=1}^K \\left(\\frac{1}{2}\\delta\\tau_k\\|A^*\\nabla g_{\\mu_k}(A\\bar{y}^k)+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)\\|^2 +\\frac{1}{4}\\kappa\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)\\|^2\\right) \\notag \\\\\r\n&\\leq \\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}_{\\mu_{K+1}}(\\bar{x}^{K+1},\\bar{y}^{K+1})+(\\mu_1-\\mu_{K+1})L_g^2\\notag \\\\\r\n&\\leq \\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}^*+\\mu_1L_g^2.\r\n\\end{align}\r\nSince $\\sigma_k \\geq\\gamma k^{-\\theta}$, $\\tau_k\\geq\\eta k^{-\\theta}$, and $M =\\min\\{\\frac{1}{2}\\delta\\eta, \\frac{1}{4}\\kappa\\gamma\\}$, it follows from \\eqref{4.11} that\r\n\\begin{multline*}\r\nM\\left(\\sum_{k=1}^{K}k^{-\\theta}\\right) \\min_{1\\leq j\\leq K}(\\|A^*\\nabla g_{\\mu_j}(A\\bar{y}^j)+\\nabla_yH(\\bar{x}^j,\\bar{y}^j)\\|^2+\r\n\\|G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j)\\|^2) \\\\ \\leq\\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)\r\n-\\mathcal{L}^*+\\mu_1L_g^2,\r\n\\end{multline*}\r\nNote that\r\n\\begin{align*}\r\n\\sum_{k=1}^Kk^{-\\theta}&\\geq \\sum_{k=1}^K\\int_k^{k+1}x^{-\\theta}dx=\\int_1^{K+1}x^{-\\theta}dx=\\frac{1}{1-\\theta}((1+K)^{1-\\theta}-1)\\\\\r\n&\\geq(\\ln 2)K^{1-\\theta},\r\n\\end{align*}\r\nwhere the last inequality is obtained by applying Lemma \\ref{l:estimate} with $x =K$ and $\\alpha =1 -\\theta$. We deduce that\r\n\\begin{multline*}\r\n\\min_{1\\leq j\\leq K}(\\|A^*\\nabla g_{\\mu_j}(A\\bar{y}^j)+\\nabla_yH(\\bar{x}^j,\\bar{y}^j)\\|^2+\\|G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j)\\|^2) \\\\ \\leq (\\ln 2)^{-1}M^{-1}(\\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}^*+\\mu_1L_g^2)k^{\\theta-1}.\r\n\\end{multline*}\r\nUsing the inequality $(\\|a\\|+\\|b\\|)^2\\leq2(\\|a\\|^2+\\|b\\|^2)$, we have\r\n\\begin{multline*}\r\n\\min_{1\\leq j\\leq k}(\\|A^*\\nabla g_{\\mu_j}(A\\bar{y}^j)+\\nabla_yH(\\bar{x}^j,\\bar{y}^j)\\|+\\|G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j)\\|) \\\\ \\leq \\sqrt{2(\\ln 2)^{-1}M^{-1}(\\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}^*+\\mu_1L_g^2)} k^{\\frac{\\theta-1}{2}}.\r\n\\end{multline*}\r\nThe proof is complete.\r\n\\end{proof}","sha256_utf8":"32c2d5b95682d6de1d88e2ef2c7c67c3ceaff2cda2d0f2e9fd777a97ccb5a32b"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"调用一步充分下降界","source_units":[{"start":36071,"end":36447,"start_line":627,"end_line":631,"raw_tex":"According to Lemma~\\ref{l:LL},\r\n\\begin{align}\\label{4.10'}\r\n\\mathcal{L}_{\\mu_{k}}(\\bar{x}^{k+1},\\bar{y}^{k+1}) &\\leq \\mathcal{L}_{\\mu_{k}}(\\bar{x}^k,\\bar{y}^k) -\\frac{1}{2}\\delta\\tau_k\\|A^*\\nabla g_{\\mu_k}(A\\bar{y}^k)+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)\\|^2\\notag \\\\\r\n&\\quad -\\frac{1}{4}\\kappa\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)\\|^2 +(\\mu_k-\\mu_{k+1})L_g^2.\r\n\\end{align}\r\n","sha256_utf8":"93e2278dd32b84a371453013bdb407ef28e1ba681f92c142b8c4a93d9cee2919"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"累加得到残差平方和界","source_units":[{"start":36447,"end":36943,"start_line":632,"end_line":637,"raw_tex":"Summing \\eqref{4.10'} from $k=1$ to $k=K$, we have\r\n\\begin{align}\\label{4.11}\r\n&\\sum_{k=1}^K \\left(\\frac{1}{2}\\delta\\tau_k\\|A^*\\nabla g_{\\mu_k}(A\\bar{y}^k)+\\nabla_y H(\\bar{x}^k,\\bar{y}^k)\\|^2 +\\frac{1}{4}\\kappa\\sigma_k\\|G_{\\sigma_k}(\\bar{x}^k,\\bar{y}^k)\\|^2\\right) \\notag \\\\\r\n&\\leq \\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}_{\\mu_{K+1}}(\\bar{x}^{K+1},\\bar{y}^{K+1})+(\\mu_1-\\mu_{K+1})L_g^2\\notag \\\\\r\n&\\leq \\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}^*+\\mu_1L_g^2.\r\n\\end{align}\r\n","sha256_utf8":"2faf96ece739d039932b38b347f75539ba68f27698487c2336189dd3e30cfa52"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"用步长下界提取最小残差","source_units":[{"start":36943,"end":37400,"start_line":638,"end_line":643,"raw_tex":"Since $\\sigma_k \\geq\\gamma k^{-\\theta}$, $\\tau_k\\geq\\eta k^{-\\theta}$, and $M =\\min\\{\\frac{1}{2}\\delta\\eta, \\frac{1}{4}\\kappa\\gamma\\}$, it follows from \\eqref{4.11} that\r\n\\begin{multline*}\r\nM\\left(\\sum_{k=1}^{K}k^{-\\theta}\\right) \\min_{1\\leq j\\leq K}(\\|A^*\\nabla g_{\\mu_j}(A\\bar{y}^j)+\\nabla_yH(\\bar{x}^j,\\bar{y}^j)\\|^2+\r\n\\|G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j)\\|^2) \\\\ \\leq\\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)\r\n-\\mathcal{L}^*+\\mu_1L_g^2,\r\n\\end{multline*}\r\n","sha256_utf8":"d5513e1ba3897d852d72980cdd5865afc87072ed8c77e367af2d4712bec75f52"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"积分比较给出权重和下界","source_units":[{"start":37400,"end":37729,"start_line":644,"end_line":649,"raw_tex":"Note that\r\n\\begin{align*}\r\n\\sum_{k=1}^Kk^{-\\theta}&\\geq \\sum_{k=1}^K\\int_k^{k+1}x^{-\\theta}dx=\\int_1^{K+1}x^{-\\theta}dx=\\frac{1}{1-\\theta}((1+K)^{1-\\theta}-1)\\\\\r\n&\\geq(\\ln 2)K^{1-\\theta},\r\n\\end{align*}\r\nwhere the last inequality is obtained by applying Lemma \\ref{l:estimate} with $x =K$ and $\\alpha =1 -\\theta$. We deduce that\r\n","sha256_utf8":"3e9e57a8a3cd70f0c108870f7927eab3e30b350babbc7c81d1c4735e140d4c7f"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"取得最小残差平方的速率","source_units":[{"start":37729,"end":38004,"start_line":650,"end_line":652,"raw_tex":"\\begin{multline*}\r\n\\min_{1\\leq j\\leq K}(\\|A^*\\nabla g_{\\mu_j}(A\\bar{y}^j)+\\nabla_yH(\\bar{x}^j,\\bar{y}^j)\\|^2+\\|G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j)\\|^2) \\\\ \\leq (\\ln 2)^{-1}M^{-1}(\\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}^*+\\mu_1L_g^2)k^{\\theta-1}.\r\n\\end{multline*}\r\n","sha256_utf8":"f2288ca095fa1ec862a4cf6775ff4f4ca0d7b4238d17d3cc3aa9674bf08ec0eb"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"由平方残差转为范数和速率","source_units":[{"start":38004,"end":38389,"start_line":653,"end_line":657,"raw_tex":"Using the inequality $(\\|a\\|+\\|b\\|)^2\\leq2(\\|a\\|^2+\\|b\\|^2)$, we have\r\n\\begin{multline*}\r\n\\min_{1\\leq j\\leq k}(\\|A^*\\nabla g_{\\mu_j}(A\\bar{y}^j)+\\nabla_yH(\\bar{x}^j,\\bar{y}^j)\\|+\\|G_{\\sigma_j}(\\bar{x}^j,\\bar{y}^j)\\|) \\\\ \\leq \\sqrt{2(\\ln 2)^{-1}M^{-1}(\\mathcal{L}_{\\mu_1}(\\bar{x}^1,\\bar{y}^1)-\\mathcal{L}^*+\\mu_1L_g^2)} k^{\\frac{\\theta-1}{2}}.\r\n\\end{multline*}\r\nThe proof is complete.\r\n","sha256_utf8":"1b8a9b8c409c534b4b37d976b366b6347957034b592434988df38242e4b6a6da"}]},{"id":"S060","semantic_type":"external_fact","kind":"external_fact","title":"标准光滑下降不等式","source_units":[],"declared_external":true,"declaration_text":["If F has an L-Lipschitz continuous gradient, then |F(v)-F(u)-〈∇F(u),v-u〉| ≤ L‖v-u‖²/2. This is the cited BC, Theorem 2.64, boundary fact."]},{"id":"S061","semantic_type":"external_fact","kind":"external_fact","title":"弱凸包络的标准微分性质","source_units":[],"declared_external":true,"declaration_text":["For proper lower semicontinuous ρ-weakly convex F and 0<μ<1/ρ, F_μ is continuously differentiable, ∇F_μ(z)=(z-Prox_{μF}(z))/μ, its gradient is max{1/μ,ρ/(1-ρμ)}-Lipschitz, and ∇F_μ(z) belongs to ∂F(Prox_{μF}(z)). These are the cited HLO and BW boundary facts."]},{"id":"S062","semantic_type":"external_fact","kind":"external_fact","title":"利普希茨函数的包络梯度界","source_units":[],"declared_external":true,"declaration_text":["For L_F-Lipschitz, ρ-weakly convex F and 0<μ<1/ρ, ‖∇F_μ(z)‖≤L_F and ‖z-Prox_{μF}(z)‖≤μL_F, as cited from BW, Lemma 3.3."]},{"id":"S063","semantic_type":"external_fact","kind":"external_fact","title":"包络参数比较的标准事实","source_units":[],"declared_external":true,"declaration_text":["For proper lower semicontinuous ρ-weakly convex F and 0<μ₂≤μ₁<1/ρ, F_{μ₂}(z)≤F_{μ₁}(z)+μ₁(μ₁-μ₂)‖∇F_{μ₁}(z)‖²/(2μ₂). If F is L_F-Lipschitz, the squared gradient can be replaced by L_F². This is the cited BW boundary fact in B0021."]},{"id":"S064","semantic_type":"external_fact","kind":"external_fact","title":"凸近端算子的标准性质","source_units":[],"declared_external":true,"declaration_text":["For proper lower semicontinuous convex f and λ>0, p=Prox_{λf}(v) if and only if (v-p)/λ belongs to ∂f(p); convex subgradients satisfy the subgradient inequality, and Prox_{λf} is nonexpansive."]},{"id":"S065","semantic_type":"external_fact","kind":"external_fact","title":"满射伪逆与距离估计","source_units":[],"declared_external":true,"declaration_text":["For surjective A, AA* is invertible, A[A*(AA*)^{-1}]=I, and ‖A*(AA*)^{-1}‖=σ_min(A)^{-1}. For a fixed nonempty set C, dist(u,C)≤dist(v,C)+‖u-v‖; enlarging C cannot increase this distance."]},{"id":"S066","semantic_type":"external_fact","kind":"external_fact","title":"平滑目标的统一下界","source_units":[],"declared_external":true,"declaration_text":["For an L_g-Lipschitz function g and μ>0, g_μ(z)≥g(z)-μL_g²/2. Consequently, if f(x)+g(Ay)+H(x,y) is bounded below and μ_k is bounded above, the smoothed objectives have a uniform lower bound."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S004","target_id":"S002","kind":"supports"},{"source_id":"S003","target_id":"S004","kind":"supports"},{"source_id":"S052","target_id":"S003","kind":"supports"},{"source_id":"S019","target_id":"S003","kind":"supports"},{"source_id":"S029","target_id":"S004","kind":"supports"},{"source_id":"S030","target_id":"S004","kind":"supports"},{"source_id":"S005","target_id":"S006","kind":"supports"},{"source_id":"S011","target_id":"S006","kind":"supports"},{"source_id":"S019","target_id":"S006","kind":"supports"},{"source_id":"S014","target_id":"S006","kind":"supports"},{"source_id":"S015","target_id":"S006","kind":"supports"},{"source_id":"S006","target_id":"S016","kind":"supports"},{"source_id":"S008","target_id":"S012","kind":"supports"},{"source_id":"S005","target_id":"S012","kind":"supports"},{"source_id":"S007","target_id":"S013","kind":"supports"},{"source_id":"S009","target_id":"S013","kind":"supports"},{"source_id":"S012","target_id":"S013","kind":"supports"},{"source_id":"S006","target_id":"S013","kind":"supports"},{"source_id":"S013","target_id":"S035","kind":"supports"},{"source_id":"S010","target_id":"S019","kind":"supports"},{"source_id":"S011","target_id":"S019","kind":"supports"},{"source_id":"S011","target_id":"S020","kind":"supports"},{"source_id":"S011","target_id":"S021","kind":"supports"},{"source_id":"S060","target_id":"S018","kind":"supports"},{"source_id":"S061","target_id":"S019","kind":"supports"},{"source_id":"S062","target_id":"S020","kind":"supports"},{"source_id":"S063","target_id":"S021","kind":"supports"},{"source_id":"S064","target_id":"S008","kind":"supports"},{"source_id":"S024","target_id":"S023","kind":"supports"},{"source_id":"S025","target_id":"S023","kind":"supports"},{"source_id":"S026","target_id":"S023","kind":"supports"},{"source_id":"S024","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S023","target_id":"S022","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S008","target_id":"S028","kind":"supports"},{"source_id":"S012","target_id":"S028","kind":"supports"},{"source_id":"S014","target_id":"S028","kind":"supports"},{"source_id":"S032","target_id":"S031","kind":"supports"},{"source_id":"S033","target_id":"S031","kind":"supports"},{"source_id":"S034","target_id":"S031","kind":"supports"},{"source_id":"S035","target_id":"S031","kind":"supports"},{"source_id":"S031","target_id":"S029","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S030","target_id":"S034","kind":"supports"},{"source_id":"S019","target_id":"S032","kind":"supports"},{"source_id":"S014","target_id":"S032","kind":"supports"},{"source_id":"S065","target_id":"S032","kind":"supports"},{"source_id":"S027","target_id":"S033","kind":"supports"},{"source_id":"S020","target_id":"S034","kind":"supports"},{"source_id":"S015","target_id":"S034","kind":"supports"},{"source_id":"S065","target_id":"S034","kind":"supports"},{"source_id":"S032","target_id":"S035","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S012","target_id":"S037","kind":"supports"},{"source_id":"S016","target_id":"S037","kind":"supports"},{"source_id":"S040","target_id":"S039","kind":"supports"},{"source_id":"S041","target_id":"S039","kind":"supports"},{"source_id":"S042","target_id":"S039","kind":"supports"},{"source_id":"S043","target_id":"S039","kind":"supports"},{"source_id":"S044","target_id":"S039","kind":"supports"},{"source_id":"S045","target_id":"S039","kind":"supports"},{"source_id":"S046","target_id":"S039","kind":"supports"},{"source_id":"S047","target_id":"S039","kind":"supports"},{"source_id":"S048","target_id":"S039","kind":"supports"},{"source_id":"S049","target_id":"S039","kind":"supports"},{"source_id":"S050","target_id":"S039","kind":"supports"},{"source_id":"S051","target_id":"S039","kind":"supports"},{"source_id":"S039","target_id":"S038","kind":"supports"},{"source_id":"S018","target_id":"S040","kind":"supports"},{"source_id":"S006","target_id":"S040","kind":"supports"},{"source_id":"S006","target_id":"S041","kind":"supports"},{"source_id":"S016","target_id":"S041","kind":"supports"},{"source_id":"S036","target_id":"S041","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S021","target_id":"S043","kind":"supports"},{"source_id":"S015","target_id":"S043","kind":"supports"},{"source_id":"S016","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S044","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S016","target_id":"S045","kind":"supports"},{"source_id":"S036","target_id":"S045","kind":"supports"},{"source_id":"S005","target_id":"S045","kind":"supports"},{"source_id":"S009","target_id":"S045","kind":"supports"},{"source_id":"S064","target_id":"S045","kind":"supports"},{"source_id":"S045","target_id":"S046","kind":"supports"},{"source_id":"S005","target_id":"S046","kind":"supports"},{"source_id":"S016","target_id":"S046","kind":"supports"},{"source_id":"S044","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S006","target_id":"S047","kind":"supports"},{"source_id":"S027","target_id":"S048","kind":"supports"},{"source_id":"S041","target_id":"S048","kind":"supports"},{"source_id":"S047","target_id":"S049","kind":"supports"},{"source_id":"S048","target_id":"S049","kind":"supports"},{"source_id":"S041","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S050","kind":"supports"},{"source_id":"S016","target_id":"S050","kind":"supports"},{"source_id":"S006","target_id":"S050","kind":"supports"},{"source_id":"S049","target_id":"S051","kind":"supports"},{"source_id":"S016","target_id":"S051","kind":"supports"},{"source_id":"S054","target_id":"S053","kind":"supports"},{"source_id":"S055","target_id":"S053","kind":"supports"},{"source_id":"S056","target_id":"S053","kind":"supports"},{"source_id":"S057","target_id":"S053","kind":"supports"},{"source_id":"S058","target_id":"S053","kind":"supports"},{"source_id":"S059","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S052","kind":"supports"},{"source_id":"S038","target_id":"S054","kind":"supports"},{"source_id":"S054","target_id":"S055","kind":"supports"},{"source_id":"S017","target_id":"S055","kind":"supports"},{"source_id":"S015","target_id":"S017","kind":"supports"},{"source_id":"S066","target_id":"S017","kind":"supports"},{"source_id":"S016","target_id":"S017","kind":"supports"},{"source_id":"S055","target_id":"S056","kind":"supports"},{"source_id":"S016","target_id":"S056","kind":"supports"},{"source_id":"S022","target_id":"S057","kind":"supports"},{"source_id":"S016","target_id":"S057","kind":"supports"},{"source_id":"S056","target_id":"S058","kind":"supports"},{"source_id":"S057","target_id":"S058","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"}]}}
{"record_id":"2511.03296v1","paper":{"title":"Technical results on the convergence of quasi-Newton methods for nonsmooth optimization","authors":["Bennet Gebken"],"arxiv_id":"2511.03296","version":"v1","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-026-03008-z","abstract_url":"https://arxiv.org/abs/2511.03296v1","pdf_url":"https://arxiv.org/pdf/2511.03296v1"},"context":{"raw_tex":"        Consider a function $f : \\R^n \\rightarrow \\R$ and denote the set of points at which $f$ is not differentiable by $\\Omega$. For $x \\notin \\Omega$, $p \\in \\R^n$ with $\\nabla f(x)^\\top p < 0$, and $c_1, c_2 \\in (0,1)$, $c_1 < c_2$, a step length $t > 0$ satisfies the \\emph{Wolfe conditions}, if $x + t p \\notin \\Omega$ and\n\n\n\\begin{align}\n            f(x + t p) &\\leq f(x) + c_1 t \\nabla f(x)^\\top p, \\label{eq:Wolfe_1} \\tag{W1} \\\\\n            \\nabla f(x + t p)^\\top p &\\geq c_2 \\nabla f(x)^\\top p. \\label{eq:Wolfe_2} \\tag{W2}\n        \\end{align}\n\n\\begin{algorithm} \n            \\caption{Quasi-Newton method}\n            \\label{algo:QN}\n            \\begin{algorithmic}[1] \n                \\Require Initial point $x^0 \\in \\R^n \\setminus \\Omega$, initial \\spd matrix $H_0 \\in \\R^{n \\times n}$, Wolfe parameters $c_1, c_2 \\in (0,1)$, $c_1 < c_2$.\n                \\For{$k = 0, 1, \\dots$}\n                    \\State If $\\nabla f(x^{k}) = 0$ then stop.\n                    \\State Set $p^k = - H_k \\nabla f(x^k)$.\n                    \\State Set $x^{k+1} = x^k + t_k p^k$, where $t_k$ satisfies the Wolfe conditions \\eqref{eq:Wolfe_1} and \\eqref{eq:Wolfe_2}.\n                    \\State If $f$ is not differentiable at $x^{k+1}$ then stop. \\label{state:breakdown}\n                    \\State For $y^k = \\nabla f(x^{k+1}) - \\nabla f(x^k)$ and $s^k = x^{k+1} - x^k$, compute \\label{state:QN_update}\n                    \\Statex \\hspace{\\algorithmicindent}a \\spd matrix $H_{k+1}$ with $H_{k+1} y^k = s^k$.\n                \\EndFor\n            \\end{algorithmic}\n        \\end{algorithm}\n\n        As in \\cite{LO2013}, if the algorithm stops in Step \\ref{state:breakdown} with $x^{k+1} \\in \\Omega$, then we say that it \\emph{breaks down}. For $k \\in \\N \\cup \\{ 0 \\}$ we denote the eigenvalues of $H_k$ by $0 < \\lambda_1^k \\leq \\dots \\leq \\lambda_n^k$. There are many ways for computing the matrix $H_{k+1}$ in Step \\ref{state:QN_update} of Alg.\\ \\ref{algo:QN}. For all numerical experiments in this article, we use the \\emph{BFGS update}, where\n\n\n\\begin{align} \\label{eq:BFGS_update} \n            H_{k+1} = V_k H_k V_k^\\top + \\frac{s^k (s^k)^\\top}{(s^k)^\\top y^k} \\text{ for } V_k = \\id - \\frac{s^k (y^k)^\\top}{(s^k)^\\top y^k},\n        \\end{align}\n\n        with $\\id \\in \\R^{n \\times n}$ denoting the identity matrix. Throughout the article, the term \\emph{BFGS method} refers to the method that results from using \\eqref{eq:BFGS_update} for Step \\ref{state:QN_update} of Alg.\\ \\ref{algo:QN}. For computing the Wolfe step length in the numerical experiments, we use Alg.\\ 4.6 from \\cite{LO2013}. (However, for our theoretical results, the specific way in which the step length is computed does not matter.) Finally, as in \\cite{LO2013}, we do not actually check for differentiability in practice, since there is no reliable way to do so in a numerical setting.\n\n\n        A function $f : \\R^n \\rightarrow \\R$ is called a \\emph{continuous selection} of the functions $f_i : \\R^n \\rightarrow \\R$, $i \\in I := \\{1,\\dots,m\\}$, $m \\in \\N$, on $\\R^n$, if $f$ is continuous and \n        \\begin{align*}\n            f(x) \\in \\{ f_i(x) : i \\in I \\} \\quad \\forall x \\in \\R^n.\n        \\end{align*}\n        The functions $f_i$ are referred to as \\emph{selection functions} of $f$. A selection function $f_i$ is called \\emph{active} at $x$ if $f(x) = f_i(x)$. The \\emph{active set} at $x$ is defined by $I(x) := \\{ i \\in I : f(x) = f_i(x) \\}$ and the \\emph{essentially active set} is defined by\n        \\begin{align*}\n            I_e(x) := \\{ i \\in I : x \\in \\mathrm{cl}(\\mathrm{int}(\\{y \\in \\R^n : f(y) = f_i(y)\\})) \\}.\n        \\end{align*}\n        In the following, let $f$ be a continuous selection of $\\C^1$-functions. By \\cite{U2002}, Proposition 2.24, for every $x \\notin \\Omega$, the set\n\n\n\\begin{align} \\label{eq:def_Ag}\n            I_g(x) := \\{ i \\in I_e(x) : \\nabla f(x) = \\nabla f_i(x) \\}\n        \\end{align}\n\n        is non-empty. \n        By \\cite{S2012}, Corollary 4.1.1, $f$ is locally Lipschitz continuous. By \\cite{S2012}, Proposition 4.3.1, the Clarke subdifferential \\cite{C1990} of $f$ at $x$ is given by $\\partial f(x) = \\conv(\\{ \\nabla f_i(x) : i \\in I_e(x) \\})$. We say that $x$ is a \\emph{(Clarke) critical point} of $f$ if $0 \\in \\partial f(x)$, which is a necessary condition for local optimality (cf.\\ \\cite{BKM2014}, Theorem 3.17). Finally, if $f$ is a function such that for every $x \\in \\R^n$, there is an open neighborhood $U \\subseteq \\R^n$ of $x$ such that the restriction $f|_U$ is a continuous selection of $\\C^1$-functions, then $f$ is called \\emph{piecewise differentiable}.\n\n\n\\begin{assum} \\label{assum:A1}\n        The function $f : \\R^n \\rightarrow \\R$ is a continuous selection of $\\C^1$-functions $f_i$, $i \\in I = \\{1,\\dots,m\\}$, whose gradients are locally Lipschitz continuous.\n    \\end{assum}\n\n\\begin{align} \\label{eq:def_calN}\n        \\calN(\\bar{x}) := \\spn(\\{ \\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x}) : i \\in \\{2,\\dots,m\\} \\}).\n    \\end{align}\n\n    \\begin{behav} \\label{behav:B1}\n        Assume that Alg.\\ \\ref{algo:QN} is applied to a function $f$ satisfying \\ref{assum:A1} and generates a sequence $(x^k)_k$ and corresponding quasi-Newton matrices $(H_k)_k$ such that\n        \\begin{enumerate}[leftmargin=1.3cm,label=(B1.\\arabic*)]\n            \\item \\label{enum:B1_1} $(x^k)_k$ is an infinite sequence, i.e., Alg.\\ \\ref{algo:QN} did not break down and $\\nabla f(x^k) \\neq 0$ for all $k \\in \\N$,\n            \\item \\label{enum:B1_2} $(x^k)_k$ has a limit $\\bar{x} \\in \\R^n$,\n            \\item \\label{enum:B1_3} for all $i \\in I$, the set $\\{ k \\in \\N : i \\in I_g(x^k) \\}$ is infinite,\n            \\item \\label{enum:B1_4} there are $\\sigma_L, \\sigma_U \\in \\R^{>0}$ such that \n            \\begin{align*}\n                \\lambda_j^k \\in [\\sigma_L,\\sigma_U] \\quad \\forall j \\in \\{\\dim(\\calN(\\bar{x}))+1,\\dots,n\\}, k \\in \\N,\n            \\end{align*}\n            \\item \\label{enum:B1_5} it holds\n            \\begin{align*}\n                \\lim_{k \\rightarrow \\infty} H_k (\\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x})) = 0 \\quad \\forall i \\in \\{2,\\dots,m\\}.\n            \\end{align*}\n        \\end{enumerate}\n    \\end{behav}\n\n\n\\begin{assum} \\label{assum:A2}\n        The function $f : \\R^n \\rightarrow \\R$ satisfies \\ref{assum:A1} and\n        \\begin{enumerate}[leftmargin=1.3cm,label=(A2.\\arabic*)]\n            \\item \\label{enum:A2_1} $f$ has a critical point $x^*$ with\n                \\begin{itemize}\n                    \\item $\\exists \\alpha \\in \\R^m$ with $\\alpha_i > 0$ for all $i \\in I$ and $\\sum_{i = 1}^m \\alpha_i \\nabla f_i(x^*) = 0$,\n                    \\item the vectors $\\nabla f_i(x^*)$, $i \\in I$, are affinely independent,\n                \\end{itemize}\n            \\item \\label{enum:A2_2} $x^*$ is the unique global minimum and there is some $z \\in \\R^n \\setminus \\{ x^* \\}$ such that $\\calL(z) := \\{ x \\in \\R^n : f(x) \\leq f(z) \\}$ is bounded.\n        \\end{enumerate}\n    \\end{assum}","units":[{"start":11472,"end":11801,"start_line":115,"end_line":115,"raw_tex":"        Consider a function $f : \\R^n \\rightarrow \\R$ and denote the set of points at which $f$ is not differentiable by $\\Omega$. For $x \\notin \\Omega$, $p \\in \\R^n$ with $\\nabla f(x)^\\top p < 0$, and $c_1, c_2 \\in (0,1)$, $c_1 < c_2$, a step length $t > 0$ satisfies the \\emph{Wolfe conditions}, if $x + t p \\notin \\Omega$ and\n","sha256_utf8":"ff6b03d42eca73c2df18339cd5a56c89fd5f354ccbca893c7447295c6e79f515"},{"start":11809,"end":12030,"start_line":116,"end_line":119,"raw_tex":"\\begin{align}\n            f(x + t p) &\\leq f(x) + c_1 t \\nabla f(x)^\\top p, \\label{eq:Wolfe_1} \\tag{W1} \\\\\n            \\nabla f(x + t p)^\\top p &\\geq c_2 \\nabla f(x)^\\top p. \\label{eq:Wolfe_2} \\tag{W2}\n        \\end{align}","sha256_utf8":"be339070d20d2ebca0764a7ed046bc8b3d794485e783fb8511de5089d32ec15e"},{"start":12502,"end":13523,"start_line":121,"end_line":135,"raw_tex":"\\begin{algorithm} \n            \\caption{Quasi-Newton method}\n            \\label{algo:QN}\n            \\begin{algorithmic}[1] \n                \\Require Initial point $x^0 \\in \\R^n \\setminus \\Omega$, initial \\spd matrix $H_0 \\in \\R^{n \\times n}$, Wolfe parameters $c_1, c_2 \\in (0,1)$, $c_1 < c_2$.\n                \\For{$k = 0, 1, \\dots$}\n                    \\State If $\\nabla f(x^{k}) = 0$ then stop.\n                    \\State Set $p^k = - H_k \\nabla f(x^k)$.\n                    \\State Set $x^{k+1} = x^k + t_k p^k$, where $t_k$ satisfies the Wolfe conditions \\eqref{eq:Wolfe_1} and \\eqref{eq:Wolfe_2}.\n                    \\State If $f$ is not differentiable at $x^{k+1}$ then stop. \\label{state:breakdown}\n                    \\State For $y^k = \\nabla f(x^{k+1}) - \\nabla f(x^k)$ and $s^k = x^{k+1} - x^k$, compute \\label{state:QN_update}\n                    \\Statex \\hspace{\\algorithmicindent}a \\spd matrix $H_{k+1}$ with $H_{k+1} y^k = s^k$.\n                \\EndFor\n            \\end{algorithmic}\n        \\end{algorithm}","sha256_utf8":"741c2177139bc94f39d5d5a08efddb5b102bfd7fe0a1b8c243b8a66926699762"},{"start":13524,"end":13979,"start_line":136,"end_line":136,"raw_tex":"        As in \\cite{LO2013}, if the algorithm stops in Step \\ref{state:breakdown} with $x^{k+1} \\in \\Omega$, then we say that it \\emph{breaks down}. For $k \\in \\N \\cup \\{ 0 \\}$ we denote the eigenvalues of $H_k$ by $0 < \\lambda_1^k \\leq \\dots \\leq \\lambda_n^k$. There are many ways for computing the matrix $H_{k+1}$ in Step \\ref{state:QN_update} of Alg.\\ \\ref{algo:QN}. For all numerical experiments in this article, we use the \\emph{BFGS update}, where\n","sha256_utf8":"9f448e1bee81c34093582133348db13c8ccc11001eb72cf406fc3c139f84a67c"},{"start":13987,"end":14187,"start_line":137,"end_line":139,"raw_tex":"\\begin{align} \\label{eq:BFGS_update} \n            H_{k+1} = V_k H_k V_k^\\top + \\frac{s^k (s^k)^\\top}{(s^k)^\\top y^k} \\text{ for } V_k = \\id - \\frac{s^k (y^k)^\\top}{(s^k)^\\top y^k},\n        \\end{align}","sha256_utf8":"1bdb5e4b70954ed639f9bf5e9b14684159195e523244c20b53f02eab6967fce5"},{"start":14188,"end":14800,"start_line":140,"end_line":140,"raw_tex":"        with $\\id \\in \\R^{n \\times n}$ denoting the identity matrix. Throughout the article, the term \\emph{BFGS method} refers to the method that results from using \\eqref{eq:BFGS_update} for Step \\ref{state:QN_update} of Alg.\\ \\ref{algo:QN}. For computing the Wolfe step length in the numerical experiments, we use Alg.\\ 4.6 from \\cite{LO2013}. (However, for our theoretical results, the specific way in which the step length is computed does not matter.) Finally, as in \\cite{LO2013}, we do not actually check for differentiability in practice, since there is no reliable way to do so in a numerical setting.\n","sha256_utf8":"81aa1d24c9c5883d31f5459f6bab99aa5dbc92f3eafe6d8affe6991728956c45"},{"start":14854,"end":15771,"start_line":144,"end_line":152,"raw_tex":"        A function $f : \\R^n \\rightarrow \\R$ is called a \\emph{continuous selection} of the functions $f_i : \\R^n \\rightarrow \\R$, $i \\in I := \\{1,\\dots,m\\}$, $m \\in \\N$, on $\\R^n$, if $f$ is continuous and \n        \\begin{align*}\n            f(x) \\in \\{ f_i(x) : i \\in I \\} \\quad \\forall x \\in \\R^n.\n        \\end{align*}\n        The functions $f_i$ are referred to as \\emph{selection functions} of $f$. A selection function $f_i$ is called \\emph{active} at $x$ if $f(x) = f_i(x)$. The \\emph{active set} at $x$ is defined by $I(x) := \\{ i \\in I : f(x) = f_i(x) \\}$ and the \\emph{essentially active set} is defined by\n        \\begin{align*}\n            I_e(x) := \\{ i \\in I : x \\in \\mathrm{cl}(\\mathrm{int}(\\{y \\in \\R^n : f(y) = f_i(y)\\})) \\}.\n        \\end{align*}\n        In the following, let $f$ be a continuous selection of $\\C^1$-functions. By \\cite{U2002}, Proposition 2.24, for every $x \\notin \\Omega$, the set\n","sha256_utf8":"84b5658113e11cd3102f3bc01a5be6c7a56891d044ebb313fd6cc216a8c95b88"},{"start":15779,"end":15901,"start_line":153,"end_line":155,"raw_tex":"\\begin{align} \\label{eq:def_Ag}\n            I_g(x) := \\{ i \\in I_e(x) : \\nabla f(x) = \\nabla f_i(x) \\}\n        \\end{align}","sha256_utf8":"87d54a414ddb3bc2de5e6fbfd89b102c2337a28d9133f8c6da9295cb0f996f2c"},{"start":15902,"end":16593,"start_line":156,"end_line":157,"raw_tex":"        is non-empty. \n        By \\cite{S2012}, Corollary 4.1.1, $f$ is locally Lipschitz continuous. By \\cite{S2012}, Proposition 4.3.1, the Clarke subdifferential \\cite{C1990} of $f$ at $x$ is given by $\\partial f(x) = \\conv(\\{ \\nabla f_i(x) : i \\in I_e(x) \\})$. We say that $x$ is a \\emph{(Clarke) critical point} of $f$ if $0 \\in \\partial f(x)$, which is a necessary condition for local optimality (cf.\\ \\cite{BKM2014}, Theorem 3.17). Finally, if $f$ is a function such that for every $x \\in \\R^n$, there is an open neighborhood $U \\subseteq \\R^n$ of $x$ such that the restriction $f|_U$ is a continuous selection of $\\C^1$-functions, then $f$ is called \\emph{piecewise differentiable}.\n","sha256_utf8":"ab8877a0d00d3cdf27cb6cf323bb7eaecc138482790039d7ce142c5ae08dd9b3"},{"start":17275,"end":17498,"start_line":162,"end_line":164,"raw_tex":"\\begin{assum} \\label{assum:A1}\n        The function $f : \\R^n \\rightarrow \\R$ is a continuous selection of $\\C^1$-functions $f_i$, $i \\in I = \\{1,\\dots,m\\}$, whose gradients are locally Lipschitz continuous.\n    \\end{assum}","sha256_utf8":"ce2ba04a83cad366e57c6a5e696130f8604aaaaa4296a8e90ab9ee4613cef8e5"},{"start":18580,"end":18732,"start_line":167,"end_line":169,"raw_tex":"\\begin{align} \\label{eq:def_calN}\n        \\calN(\\bar{x}) := \\spn(\\{ \\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x}) : i \\in \\{2,\\dots,m\\} \\}).\n    \\end{align}","sha256_utf8":"78e430f826ce692a65c2e0dbf6a48ee4e0de4e66c4ba38865883d085e447dd52"},{"start":22655,"end":23823,"start_line":209,"end_line":224,"raw_tex":"    \\begin{behav} \\label{behav:B1}\n        Assume that Alg.\\ \\ref{algo:QN} is applied to a function $f$ satisfying \\ref{assum:A1} and generates a sequence $(x^k)_k$ and corresponding quasi-Newton matrices $(H_k)_k$ such that\n        \\begin{enumerate}[leftmargin=1.3cm,label=(B1.\\arabic*)]\n            \\item \\label{enum:B1_1} $(x^k)_k$ is an infinite sequence, i.e., Alg.\\ \\ref{algo:QN} did not break down and $\\nabla f(x^k) \\neq 0$ for all $k \\in \\N$,\n            \\item \\label{enum:B1_2} $(x^k)_k$ has a limit $\\bar{x} \\in \\R^n$,\n            \\item \\label{enum:B1_3} for all $i \\in I$, the set $\\{ k \\in \\N : i \\in I_g(x^k) \\}$ is infinite,\n            \\item \\label{enum:B1_4} there are $\\sigma_L, \\sigma_U \\in \\R^{>0}$ such that \n            \\begin{align*}\n                \\lambda_j^k \\in [\\sigma_L,\\sigma_U] \\quad \\forall j \\in \\{\\dim(\\calN(\\bar{x}))+1,\\dots,n\\}, k \\in \\N,\n            \\end{align*}\n            \\item \\label{enum:B1_5} it holds\n            \\begin{align*}\n                \\lim_{k \\rightarrow \\infty} H_k (\\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x})) = 0 \\quad \\forall i \\in \\{2,\\dots,m\\}.\n            \\end{align*}\n        \\end{enumerate}\n    \\end{behav}\n","sha256_utf8":"872891327d913391f567cfb595ac54bc63e012a023ea60d771f36dce4ea96506"},{"start":35661,"end":36432,"start_line":352,"end_line":362,"raw_tex":"\\begin{assum} \\label{assum:A2}\n        The function $f : \\R^n \\rightarrow \\R$ satisfies \\ref{assum:A1} and\n        \\begin{enumerate}[leftmargin=1.3cm,label=(A2.\\arabic*)]\n            \\item \\label{enum:A2_1} $f$ has a critical point $x^*$ with\n                \\begin{itemize}\n                    \\item $\\exists \\alpha \\in \\R^m$ with $\\alpha_i > 0$ for all $i \\in I$ and $\\sum_{i = 1}^m \\alpha_i \\nabla f_i(x^*) = 0$,\n                    \\item the vectors $\\nabla f_i(x^*)$, $i \\in I$, are affinely independent,\n                \\end{itemize}\n            \\item \\label{enum:A2_2} $x^*$ is the unique global minimum and there is some $z \\in \\R^n \\setminus \\{ x^* \\}$ such that $\\calL(z) := \\{ x \\in \\R^n : f(x) \\leq f(z) \\}$ is bounded.\n        \\end{enumerate}\n    \\end{assum}","sha256_utf8":"9dfcc8adff1332c869fc920e59ec583f98f57106889d3384a7b0a3cffd15e2d3"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem} \\label{thm:criticality}\n        Assume that $f$ satisfies \\ref{assum:A2}. There is an open neighborhood $U \\subseteq \\R^n$ of $x^*$ such that if \\ref{behav:B1} holds for an initial point $x^0 \\in U$, then the limit $\\bar{x}$ of $(x^k)_k$ is a critical point of $f$.\n    \\end{theorem}","units":[{"start":38626,"end":38925,"start_line":374,"end_line":376,"raw_tex":"\\begin{theorem} \\label{thm:criticality}\n        Assume that $f$ satisfies \\ref{assum:A2}. There is an open neighborhood $U \\subseteq \\R^n$ of $x^*$ such that if \\ref{behav:B1} holds for an initial point $x^0 \\in U$, then the limit $\\bar{x}$ of $(x^k)_k$ is a critical point of $f$.\n    \\end{theorem}","sha256_utf8":"90a0c0bc559fa7bbba69b8d32cbad0c02293de731eba564c2ad9645c384025e5"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma} \\label{lem:common_descent}\n        Assume that $f$ satisfies \\ref{assum:A1} and that \\ref{behav:B1} holds. Then\n        \\begin{align*}\n            \\lim_{k \\rightarrow \\infty} \\nabla f_i(x^k)^\\top p^k - \\nabla f_1(x^k)^\\top p^k = 0\n            \\quad \\forall i \\in \\{2,\\dots,m\\}.\n        \\end{align*}\n    \\end{lemma}\n\n\\begin{proof}\n        For $k \\in \\N$ let $i_k \\in I_g(x^k)$. Let $i \\in \\{2,\\dots,m\\}$. By definition of $p^k$ we have\n        \\begin{align*}\n            \\nabla f_i(x^k)^\\top p^k - \\nabla f_1(x^k)^\\top p^k\n            = -(\\nabla f_i(x^k) - \\nabla f_1(x^k))^\\top H_k \\nabla f_{i_k}(x^k).\n        \\end{align*}\n        Since $x^k \\rightarrow \\bar{x}$ (by \\ref{enum:B1_2}), $(H_k)_k$ is bounded (w.r.t.\\ the spectral norm, by \\ref{enum:B1_4}), and the selection functions are $\\C^1$ (by \\ref{assum:A1}), \\ref{enum:B1_5} implies \n        \\begin{align*}\n            &(\\nabla f_i(x^k) - \\nabla f_1(x^k))^\\top H_k \\\\\n            &= (\\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x}))^\\top H_k + (\\nabla f_i(x^k) - \\nabla f_1(x^k) - (\\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x})))^\\top H_k\n            \\rightarrow 0\n        \\end{align*}\n        as $k \\rightarrow \\infty$, completing the proof.\n    \\end{proof}\n\n\\begin{lemma} \\label{lem:lower_bound_Wolfe}\n        Assume that $f$ satisfies \\ref{assum:A1} and let $x \\notin \\Omega$. Let $c_2 \\in (0,1)$ and $p \\in \\R^n$ with $\\nabla f(x)^\\top p < 0$. Let $t$ be a step length satisfying the second Wolfe condition \\eqref{eq:Wolfe_2}. For $i \\in I_g(x + t p)$ let $L$ be a Lipschitz constant of $\\nabla f_i$ on $\\conv(\\{x,x + tp\\})$. Then\n        \\begin{align*}\n            t \\geq \n            \\frac{1}{L \\| p \\|^2}\n            \\left( \n            -(1 - c_2) \\nabla f(x)^\\top p\n            +\n            (\\nabla f(x) - \\nabla f_{i}(x))^\\top p\n            \\right).\n        \\end{align*}\n    \\end{lemma}\n\n\\begin{proof}\n        By the second Wolfe condition \\eqref{eq:Wolfe_2} it holds $\\nabla f_{i}(x + tp)^\\top p \\geq c_2 \\nabla f(x)^\\top p$. Subtracting $ \\nabla f_{i}(x)^\\top p$ on both sides yields\n        \\begin{equation} \\label{eq:proof_lem_lower_bound_Wolfe_1}\n            \\begin{aligned}\n                (\\nabla f_{i}(x + tp) - \\nabla f_{i}(x))^\\top p\n                &\\geq (c_2 \\nabla f(x) - \\nabla f_{i}(x))^\\top p \\\\\n                &= (c_2 \\nabla f(x) - \\nabla f(x) + \\nabla f(x) - \\nabla f_{i}(x))^\\top p \\\\\n                &= -(1 - c_2) \\nabla f(x)^\\top p + (\\nabla f(x) - \\nabla f_{i}(x))^\\top p.\n            \\end{aligned}\n        \\end{equation}\n        Since $\\nabla f_{i}$ is locally Lipschitz continuous by \\ref{assum:A1}, the left-hand side of \\eqref{eq:proof_lem_lower_bound_Wolfe_1} satisfies \n        \\begin{align} \\label{eq:proof_lem_lower_bound_Wolfe_2}\n            (\\nabla f_{i}(x + tp) - \\nabla f_{i}(x))^\\top p\n            \\leq \\| \\nabla f_{i}(x + tp) - \\nabla f_{i}(x) \\| \\| p \\|\n            \\leq t L \\| p \\|^2\n        \\end{align}\n        for a Lipschitz constant of $\\nabla f_i$ on $\\conv(\\{x,x + tp\\})$. Combining \\eqref{eq:proof_lem_lower_bound_Wolfe_1} and \\eqref{eq:proof_lem_lower_bound_Wolfe_2} completes the proof.\n    \\end{proof}\n\n\\begin{lemma} \\label{lem:ker_H_subsequence}\n        If $f$ satisfies \\ref{assum:A1} and \\ref{behav:B1} holds, then $(H_k)_k$ has an accumulation point. Furthermore, for all accumulation points $\\bar{H}$ of $(H_k)_k$, it holds $\\ker(\\bar{H}) = \\calN(\\bar{x})$.\n    \\end{lemma}\n\n\\begin{proof}\n        By \\ref{enum:B1_4} the spectral norm of $H_k$ is bounded above by $\\sigma_U$ for all $k \\in \\N$. Thus $(H_k)_k$ must have an accumulation point $\\bar{H}$. Let $(k_l)_l \\subseteq \\N$ be a strictly increasing, infinite sequence with $H_{k_l} \\rightarrow \\bar{H}$ for $l \\rightarrow \\infty$. Since $\\lambda_j^k \\geq \\sigma_L$ for all $j \\in \\{\\dim(\\calN(\\bar{x}))+1,\\dots,n\\}$, $k \\in \\N$, we have $\\dim(\\ker(\\bar{H})) \\leq \\dim(\\calN(\\bar{x}))$ due to continuity of eigenvalues (see, e.g., \\cite{A2013}, Theorem 5.2.2). Thus, it suffices to show that $\\calN(\\bar{x}) \\subseteq \\ker(\\bar{H})$. To this end, let $v \\in \\calN(\\bar{x})$. Then there are $\\alpha_i \\in \\R$, $i \\in \\{2,\\dots,m\\}$, such that $v = \\sum_{i = 2}^m \\alpha_i (\\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x}))$. By \\ref{enum:B1_5}, we obtain\n        \\begin{align*}\n            \\bar{H} v\n            = \\lim_{l \\rightarrow \\infty} H_{k_l} v\n            = \\lim_{l \\rightarrow \\infty} \\sum_{i = 2}^m \\alpha_i H_{k_l} (\\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x}))\n            = 0,\n        \\end{align*}\n        so $v \\in \\ker(\\bar{H})$, completing the proof. \n    \\end{proof}\n\n\\begin{lemma} \\label{lem:intermediate_result}\n        If $f$ satisfies \\ref{assum:A1} and \\ref{behav:B1} holds, then $I_e(\\bar{x}) = I = \\{1,\\dots,m\\}$ and\n        \\begin{align} \\label{eq:intermediate_result}\n            \\nabla f_i(\\bar{x}) \\in \\calN(\\bar{x}) \\quad \\forall i \\in I.\n        \\end{align}\n        In particular, the vectors $\\nabla f_i(\\bar{x})$, $i \\in I$, are linearly dependent.\n    \\end{lemma}\n\n\\begin{proof}\n        The equality $I_e(\\bar{x}) = \\{1,\\dots,m\\}$ follows from \\ref{enum:B1_3} and the fact that $I_g(x^j) \\subseteq I_e(x^j)$ for all $j \\in \\N$ (cf.\\ \\eqref{eq:def_Ag}). \\\\\n        \\textbf{Part 1:} We first show that $\\nabla f(x^k)^\\top p^k \\rightarrow 0$. To this end, assume that this does not hold. Then there is some $C > 0$ and a strictly increasing, infinite sequence $(k_l)_l \\subseteq \\N$ with $\\nabla f(x^{k_l})^\\top p^{k_l} < -C$ for all $l \\in \\N$. Since the number of selection functions is finite, we can assume w.l.o.g.\\ that there is some $i \\in I$ with $i \\in I_g(x^{k_l + 1})$ for all $l \\in \\N$. Furthermore, by local Lipschitz continuity of $\\nabla f_i$, we can assume w.l.o.g.\\ that there is a Lipschitz constant $L > 0$ for $\\nabla f_i$ on a superset of $\\{ x^{k_l} : l \\in \\N \\}$. By Lemma \\ref{lem:lower_bound_Wolfe} we have\n        \\begin{align*}\n            t_{k_l} \\geq \n            \\underbrace{\\frac{1}{L \\| p^{k_l} \\|^2}}_{\\text{(I)}}\n            \\left( \n            \\underbrace{-(1 - c_2) \\nabla f(x^{k_l})^\\top p^{k_l}}_{\\text{(II)}}\n            +\n            \\underbrace{(\\nabla f(x^{k_l}) - \\nabla f_{i}(x^{k_l}))^\\top p^{k_l}}_{\\text{(III)}}\n            \\right).\n        \\end{align*}\n        The fraction (I) is bounded below since $(p^{k_l})_l$ is bounded above due to \\ref{enum:B1_2} and \\ref{enum:B1_4}. The term (II) is bounded below by $(1 - c_2)C$. The term (III) vanishes by Lemma \\ref{lem:common_descent}, since\n        \\begin{align*}\n            \\nabla f(x^{k_l}) - \\nabla f_{i}(x^{k_l})\n            = \\nabla f(x^{k_l}) - \\nabla f_1(x^{k_l}) - (\\nabla f_i(x^{k_l}) - \\nabla f_1(x^{k_l})).\n        \\end{align*}\n        Thus, there is some $t_{\\text{min}} > 0$ such that $t_{k_l} \\geq t_{\\text{min}}$ for all $l \\in \\N$. By the first Wolfe condition \\eqref{eq:Wolfe_1}, this implies that\n        \\begin{align*}\n            f(x^{k_l + 1}) - f(x^{k_l}) \n            \\leq c_1 t_{k_l} \\nabla f(x^{k_l})^\\top p^{k_l}\n            < - c_1 t_{\\text{min}} C\n            < 0\n            \\quad \\forall l \\in \\N,\n        \\end{align*}\n        i.e., the objective value decreases by at least a constant amount infinitely many times. Since $(x^k)_k$ has a limit (by \\ref{enum:B1_2}), this contradicts the continuity of $f$. \\\\\n        \\textbf{Part 2:} Let $i \\in I$. By \\ref{enum:B1_3} there is a strictly increasing, infinite sequence $(k_l)_l \\subseteq \\N$ such that $i \\in I_g(x^{k_l})$ for all $l \\in \\N$. Since $(H_k)_k$ is bounded above by \\ref{enum:B1_4}, we can assume w.l.o.g.\\ that $(H_{k_l})_l$ converges to some $\\bar{H}$. By Part 1, we obtain\n        \\begin{align*}\n            0\n            &= \\lim_{l \\rightarrow \\infty} \\nabla f(x^{k_l})^\\top p^{k_l}\n            = -\\lim_{l \\rightarrow \\infty} \\nabla f(x^{k_l})^\\top H_{k_l} \\nabla f(x^{k_l})\n            = -\\lim_{l \\rightarrow \\infty} \\nabla f_i(x^{k_l})^\\top H_{k_l} \\nabla f_i(x^{k_l}) \\\\\n            &= -\\nabla f_i(\\bar{x})^\\top \\bar{H} \\nabla f_i(\\bar{x}).\n        \\end{align*}\n        Since $\\bar{H}$ is symmetric and positive semidefinite, this implies that $\\nabla f_i(\\bar{x}) \\in \\ker(\\bar{H})$ (see, e.g., \\cite{A2024}, 7.43). Application of Lemma \\ref{lem:ker_H_subsequence} completes the proof.\n    \\end{proof}\n\n\\begin{lemma} \\label{lem:crit_limit}\n        Assume that $f$ satisfies \\ref{assum:A1} and \\ref{enum:A2_1}. Then there is an open neighborhood $U \\subseteq \\R^n$ of $x^*$ such if $x \\in U$ with $I_e(x) = I = \\{1,\\dots,m\\}$ and $\\nabla f_i(x)$, $i \\in I$, linearly dependent, then $x$ is a critical point of $f$.\n    \\end{lemma}\n\n\\begin{proof}\n        Assume that this does not hold. Then there is a sequence $(x^l)_l \\subseteq \\R^n$ with $x^l \\rightarrow x^*$, $I_e(x^l) = I$, and $\\nabla f_i(x^l)$, $i \\in I$, linearly dependent for all $l \\in \\N$, such that $x^l$ is not critical for any $l \\in \\N$. Linear dependence implies that there is a sequence $(\\beta^l)_l \\subseteq \\R^m \\setminus \\{ 0 \\}$ with $\\sum_{i = 1}^m \\beta_i^l \\nabla f_i(x^l) = 0$ for all $l \\in \\N$.\n        Assume w.l.o.g.\\ (via scaling) that $\\| \\beta^l \\|_\\infty := \\max_{i \\in I} |\\beta_i| = 1$ for all $l \\in \\N$. Then we can assume w.l.o.g.\\ that $(\\beta^l)_l$ has a limit $\\bar{\\beta} \\in \\R^m$ with $\\| \\bar{\\beta} \\|_\\infty = 1$. By continuity of $\\nabla f_i$, $i \\in I$, we have $\\sum_{i = 1}^m \\bar{\\beta}_i \\nabla f_i(x^*) = 0$. By affine independence of $\\nabla f_i(x^*)$, $i \\in I$, we must have $\\sum_{i = 1}^m \\bar{\\beta}_i \\neq 0$. Let $\\beta^* := \\bar{\\beta}/(\\sum_{i = 1}^m \\bar{\\beta}_i)$. Again using the affine independence, we must have $\\beta^* = \\alpha$. Now $\\alpha_i > 0$ for all $i \\in I$ implies that there is some $N \\in \\N$ such that $\\beta_i^l > 0$ for all $i \\in I$, $l > N$. This implies that $x^l$ is critical for any $l > N$, which is a contradiction.\n    \\end{proof}\n\n\\begin{proof}\n        Let $U'$ be the open neighborhood of $x^*$ from Lemma \\ref{lem:crit_limit}. By \\ref{enum:A2_2} there must be an open neighborhood $U \\subseteq \\R^n$ of $x^*$ such that $\\calL(x^0) \\subseteq U'$ for all $x^0 \\in U$. Since Alg.\\ \\ref{algo:QN} is a descent method, $x^0 \\in U$ implies $\\lim_{k \\rightarrow \\infty} x^k = \\bar{x} \\in \\calL(x^0) \\subseteq U'$. Application of Lemma \\ref{lem:intermediate_result} shows that $I_e(\\bar{x}) = I$ and the vectors $\\nabla f_i(\\bar{x})$, $i \\in I$, must be linearly dependent. Application of Lemma \\ref{lem:crit_limit} completes the proof.\n    \\end{proof}","units":[{"start":25664,"end":25992,"start_line":235,"end_line":241,"raw_tex":"\\begin{lemma} \\label{lem:common_descent}\n        Assume that $f$ satisfies \\ref{assum:A1} and that \\ref{behav:B1} holds. Then\n        \\begin{align*}\n            \\lim_{k \\rightarrow \\infty} \\nabla f_i(x^k)^\\top p^k - \\nabla f_1(x^k)^\\top p^k = 0\n            \\quad \\forall i \\in \\{2,\\dots,m\\}.\n        \\end{align*}\n    \\end{lemma}","sha256_utf8":"68d546f31361249835e45ae4a5fc1cb6d4e4ef2d624850f68c7eedb01e2d8414"},{"start":25997,"end":26886,"start_line":242,"end_line":255,"raw_tex":"\\begin{proof}\n        For $k \\in \\N$ let $i_k \\in I_g(x^k)$. Let $i \\in \\{2,\\dots,m\\}$. By definition of $p^k$ we have\n        \\begin{align*}\n            \\nabla f_i(x^k)^\\top p^k - \\nabla f_1(x^k)^\\top p^k\n            = -(\\nabla f_i(x^k) - \\nabla f_1(x^k))^\\top H_k \\nabla f_{i_k}(x^k).\n        \\end{align*}\n        Since $x^k \\rightarrow \\bar{x}$ (by \\ref{enum:B1_2}), $(H_k)_k$ is bounded (w.r.t.\\ the spectral norm, by \\ref{enum:B1_4}), and the selection functions are $\\C^1$ (by \\ref{assum:A1}), \\ref{enum:B1_5} implies \n        \\begin{align*}\n            &(\\nabla f_i(x^k) - \\nabla f_1(x^k))^\\top H_k \\\\\n            &= (\\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x}))^\\top H_k + (\\nabla f_i(x^k) - \\nabla f_1(x^k) - (\\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x})))^\\top H_k\n            \\rightarrow 0\n        \\end{align*}\n        as $k \\rightarrow \\infty$, completing the proof.\n    \\end{proof}","sha256_utf8":"58ce6194700a7f855443205e148aac404f13f36cd3f7f760b58d5b42a5e1182e"},{"start":27640,"end":28276,"start_line":259,"end_line":270,"raw_tex":"\\begin{lemma} \\label{lem:lower_bound_Wolfe}\n        Assume that $f$ satisfies \\ref{assum:A1} and let $x \\notin \\Omega$. Let $c_2 \\in (0,1)$ and $p \\in \\R^n$ with $\\nabla f(x)^\\top p < 0$. Let $t$ be a step length satisfying the second Wolfe condition \\eqref{eq:Wolfe_2}. For $i \\in I_g(x + t p)$ let $L$ be a Lipschitz constant of $\\nabla f_i$ on $\\conv(\\{x,x + tp\\})$. Then\n        \\begin{align*}\n            t \\geq \n            \\frac{1}{L \\| p \\|^2}\n            \\left( \n            -(1 - c_2) \\nabla f(x)^\\top p\n            +\n            (\\nabla f(x) - \\nabla f_{i}(x))^\\top p\n            \\right).\n        \\end{align*}\n    \\end{lemma}","sha256_utf8":"ebb7e3041e75ed51150647ab0e24f25d345637015b6bb87239f8cff2efe2dad9"},{"start":28281,"end":29543,"start_line":271,"end_line":288,"raw_tex":"\\begin{proof}\n        By the second Wolfe condition \\eqref{eq:Wolfe_2} it holds $\\nabla f_{i}(x + tp)^\\top p \\geq c_2 \\nabla f(x)^\\top p$. Subtracting $ \\nabla f_{i}(x)^\\top p$ on both sides yields\n        \\begin{equation} \\label{eq:proof_lem_lower_bound_Wolfe_1}\n            \\begin{aligned}\n                (\\nabla f_{i}(x + tp) - \\nabla f_{i}(x))^\\top p\n                &\\geq (c_2 \\nabla f(x) - \\nabla f_{i}(x))^\\top p \\\\\n                &= (c_2 \\nabla f(x) - \\nabla f(x) + \\nabla f(x) - \\nabla f_{i}(x))^\\top p \\\\\n                &= -(1 - c_2) \\nabla f(x)^\\top p + (\\nabla f(x) - \\nabla f_{i}(x))^\\top p.\n            \\end{aligned}\n        \\end{equation}\n        Since $\\nabla f_{i}$ is locally Lipschitz continuous by \\ref{assum:A1}, the left-hand side of \\eqref{eq:proof_lem_lower_bound_Wolfe_1} satisfies \n        \\begin{align} \\label{eq:proof_lem_lower_bound_Wolfe_2}\n            (\\nabla f_{i}(x + tp) - \\nabla f_{i}(x))^\\top p\n            \\leq \\| \\nabla f_{i}(x + tp) - \\nabla f_{i}(x) \\| \\| p \\|\n            \\leq t L \\| p \\|^2\n        \\end{align}\n        for a Lipschitz constant of $\\nabla f_i$ on $\\conv(\\{x,x + tp\\})$. Combining \\eqref{eq:proof_lem_lower_bound_Wolfe_1} and \\eqref{eq:proof_lem_lower_bound_Wolfe_2} completes the proof.\n    \\end{proof}","sha256_utf8":"9d93c55a5e88c163476b4072cbf73afc8051a30ecfca075012d240757ac357a6"},{"start":29670,"end":29945,"start_line":291,"end_line":293,"raw_tex":"\\begin{lemma} \\label{lem:ker_H_subsequence}\n        If $f$ satisfies \\ref{assum:A1} and \\ref{behav:B1} holds, then $(H_k)_k$ has an accumulation point. Furthermore, for all accumulation points $\\bar{H}$ of $(H_k)_k$, it holds $\\ker(\\bar{H}) = \\calN(\\bar{x})$.\n    \\end{lemma}","sha256_utf8":"321810a2a2fa8b2b297e7e0bb3ea0193967d304c5a0b45ec56efccdd0d4c756c"},{"start":29950,"end":31101,"start_line":294,"end_line":303,"raw_tex":"\\begin{proof}\n        By \\ref{enum:B1_4} the spectral norm of $H_k$ is bounded above by $\\sigma_U$ for all $k \\in \\N$. Thus $(H_k)_k$ must have an accumulation point $\\bar{H}$. Let $(k_l)_l \\subseteq \\N$ be a strictly increasing, infinite sequence with $H_{k_l} \\rightarrow \\bar{H}$ for $l \\rightarrow \\infty$. Since $\\lambda_j^k \\geq \\sigma_L$ for all $j \\in \\{\\dim(\\calN(\\bar{x}))+1,\\dots,n\\}$, $k \\in \\N$, we have $\\dim(\\ker(\\bar{H})) \\leq \\dim(\\calN(\\bar{x}))$ due to continuity of eigenvalues (see, e.g., \\cite{A2013}, Theorem 5.2.2). Thus, it suffices to show that $\\calN(\\bar{x}) \\subseteq \\ker(\\bar{H})$. To this end, let $v \\in \\calN(\\bar{x})$. Then there are $\\alpha_i \\in \\R$, $i \\in \\{2,\\dots,m\\}$, such that $v = \\sum_{i = 2}^m \\alpha_i (\\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x}))$. By \\ref{enum:B1_5}, we obtain\n        \\begin{align*}\n            \\bar{H} v\n            = \\lim_{l \\rightarrow \\infty} H_{k_l} v\n            = \\lim_{l \\rightarrow \\infty} \\sum_{i = 2}^m \\alpha_i H_{k_l} (\\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x}))\n            = 0,\n        \\end{align*}\n        so $v \\in \\ker(\\bar{H})$, completing the proof. \n    \\end{proof}","sha256_utf8":"342b30e7d14575286032d27b8e1a47bd0b86518750ec0b766da29e4b7aaba0ad"},{"start":31275,"end":31686,"start_line":307,"end_line":313,"raw_tex":"\\begin{lemma} \\label{lem:intermediate_result}\n        If $f$ satisfies \\ref{assum:A1} and \\ref{behav:B1} holds, then $I_e(\\bar{x}) = I = \\{1,\\dots,m\\}$ and\n        \\begin{align} \\label{eq:intermediate_result}\n            \\nabla f_i(\\bar{x}) \\in \\calN(\\bar{x}) \\quad \\forall i \\in I.\n        \\end{align}\n        In particular, the vectors $\\nabla f_i(\\bar{x})$, $i \\in I$, are linearly dependent.\n    \\end{lemma}","sha256_utf8":"4e8cebfdafead375e68ed6c3c7a877bbd1d6940b7d65d002bb8e045537d1d166"},{"start":31691,"end":34923,"start_line":314,"end_line":349,"raw_tex":"\\begin{proof}\n        The equality $I_e(\\bar{x}) = \\{1,\\dots,m\\}$ follows from \\ref{enum:B1_3} and the fact that $I_g(x^j) \\subseteq I_e(x^j)$ for all $j \\in \\N$ (cf.\\ \\eqref{eq:def_Ag}). \\\\\n        \\textbf{Part 1:} We first show that $\\nabla f(x^k)^\\top p^k \\rightarrow 0$. To this end, assume that this does not hold. Then there is some $C > 0$ and a strictly increasing, infinite sequence $(k_l)_l \\subseteq \\N$ with $\\nabla f(x^{k_l})^\\top p^{k_l} < -C$ for all $l \\in \\N$. Since the number of selection functions is finite, we can assume w.l.o.g.\\ that there is some $i \\in I$ with $i \\in I_g(x^{k_l + 1})$ for all $l \\in \\N$. Furthermore, by local Lipschitz continuity of $\\nabla f_i$, we can assume w.l.o.g.\\ that there is a Lipschitz constant $L > 0$ for $\\nabla f_i$ on a superset of $\\{ x^{k_l} : l \\in \\N \\}$. By Lemma \\ref{lem:lower_bound_Wolfe} we have\n        \\begin{align*}\n            t_{k_l} \\geq \n            \\underbrace{\\frac{1}{L \\| p^{k_l} \\|^2}}_{\\text{(I)}}\n            \\left( \n            \\underbrace{-(1 - c_2) \\nabla f(x^{k_l})^\\top p^{k_l}}_{\\text{(II)}}\n            +\n            \\underbrace{(\\nabla f(x^{k_l}) - \\nabla f_{i}(x^{k_l}))^\\top p^{k_l}}_{\\text{(III)}}\n            \\right).\n        \\end{align*}\n        The fraction (I) is bounded below since $(p^{k_l})_l$ is bounded above due to \\ref{enum:B1_2} and \\ref{enum:B1_4}. The term (II) is bounded below by $(1 - c_2)C$. The term (III) vanishes by Lemma \\ref{lem:common_descent}, since\n        \\begin{align*}\n            \\nabla f(x^{k_l}) - \\nabla f_{i}(x^{k_l})\n            = \\nabla f(x^{k_l}) - \\nabla f_1(x^{k_l}) - (\\nabla f_i(x^{k_l}) - \\nabla f_1(x^{k_l})).\n        \\end{align*}\n        Thus, there is some $t_{\\text{min}} > 0$ such that $t_{k_l} \\geq t_{\\text{min}}$ for all $l \\in \\N$. By the first Wolfe condition \\eqref{eq:Wolfe_1}, this implies that\n        \\begin{align*}\n            f(x^{k_l + 1}) - f(x^{k_l}) \n            \\leq c_1 t_{k_l} \\nabla f(x^{k_l})^\\top p^{k_l}\n            < - c_1 t_{\\text{min}} C\n            < 0\n            \\quad \\forall l \\in \\N,\n        \\end{align*}\n        i.e., the objective value decreases by at least a constant amount infinitely many times. Since $(x^k)_k$ has a limit (by \\ref{enum:B1_2}), this contradicts the continuity of $f$. \\\\\n        \\textbf{Part 2:} Let $i \\in I$. By \\ref{enum:B1_3} there is a strictly increasing, infinite sequence $(k_l)_l \\subseteq \\N$ such that $i \\in I_g(x^{k_l})$ for all $l \\in \\N$. Since $(H_k)_k$ is bounded above by \\ref{enum:B1_4}, we can assume w.l.o.g.\\ that $(H_{k_l})_l$ converges to some $\\bar{H}$. By Part 1, we obtain\n        \\begin{align*}\n            0\n            &= \\lim_{l \\rightarrow \\infty} \\nabla f(x^{k_l})^\\top p^{k_l}\n            = -\\lim_{l \\rightarrow \\infty} \\nabla f(x^{k_l})^\\top H_{k_l} \\nabla f(x^{k_l})\n            = -\\lim_{l \\rightarrow \\infty} \\nabla f_i(x^{k_l})^\\top H_{k_l} \\nabla f_i(x^{k_l}) \\\\\n            &= -\\nabla f_i(\\bar{x})^\\top \\bar{H} \\nabla f_i(\\bar{x}).\n        \\end{align*}\n        Since $\\bar{H}$ is symmetric and positive semidefinite, this implies that $\\nabla f_i(\\bar{x}) \\in \\ker(\\bar{H})$ (see, e.g., \\cite{A2024}, 7.43). Application of Lemma \\ref{lem:ker_H_subsequence} completes the proof.\n    \\end{proof}","sha256_utf8":"c40a1ffb94fec09c2fe820f9d7515f6affcb30bfbdc456c63ca564a064addbce"},{"start":36633,"end":36959,"start_line":365,"end_line":367,"raw_tex":"\\begin{lemma} \\label{lem:crit_limit}\n        Assume that $f$ satisfies \\ref{assum:A1} and \\ref{enum:A2_1}. Then there is an open neighborhood $U \\subseteq \\R^n$ of $x^*$ such if $x \\in U$ with $I_e(x) = I = \\{1,\\dots,m\\}$ and $\\nabla f_i(x)$, $i \\in I$, linearly dependent, then $x$ is a critical point of $f$.\n    \\end{lemma}","sha256_utf8":"704dd97eb21641b3ab9a49194ab7d23026a5f2c86f5d2be6af8dc4a31a30fa09"},{"start":36964,"end":38210,"start_line":368,"end_line":371,"raw_tex":"\\begin{proof}\n        Assume that this does not hold. Then there is a sequence $(x^l)_l \\subseteq \\R^n$ with $x^l \\rightarrow x^*$, $I_e(x^l) = I$, and $\\nabla f_i(x^l)$, $i \\in I$, linearly dependent for all $l \\in \\N$, such that $x^l$ is not critical for any $l \\in \\N$. Linear dependence implies that there is a sequence $(\\beta^l)_l \\subseteq \\R^m \\setminus \\{ 0 \\}$ with $\\sum_{i = 1}^m \\beta_i^l \\nabla f_i(x^l) = 0$ for all $l \\in \\N$.\n        Assume w.l.o.g.\\ (via scaling) that $\\| \\beta^l \\|_\\infty := \\max_{i \\in I} |\\beta_i| = 1$ for all $l \\in \\N$. Then we can assume w.l.o.g.\\ that $(\\beta^l)_l$ has a limit $\\bar{\\beta} \\in \\R^m$ with $\\| \\bar{\\beta} \\|_\\infty = 1$. By continuity of $\\nabla f_i$, $i \\in I$, we have $\\sum_{i = 1}^m \\bar{\\beta}_i \\nabla f_i(x^*) = 0$. By affine independence of $\\nabla f_i(x^*)$, $i \\in I$, we must have $\\sum_{i = 1}^m \\bar{\\beta}_i \\neq 0$. Let $\\beta^* := \\bar{\\beta}/(\\sum_{i = 1}^m \\bar{\\beta}_i)$. Again using the affine independence, we must have $\\beta^* = \\alpha$. Now $\\alpha_i > 0$ for all $i \\in I$ implies that there is some $N \\in \\N$ such that $\\beta_i^l > 0$ for all $i \\in I$, $l > N$. This implies that $x^l$ is critical for any $l > N$, which is a contradiction.\n    \\end{proof}","sha256_utf8":"b9cca40c1df81331b85f8163651e292a54d3f4d9baac52793bd71c2137db4201"},{"start":38930,"end":39544,"start_line":377,"end_line":379,"raw_tex":"\\begin{proof}\n        Let $U'$ be the open neighborhood of $x^*$ from Lemma \\ref{lem:crit_limit}. By \\ref{enum:A2_2} there must be an open neighborhood $U \\subseteq \\R^n$ of $x^*$ such that $\\calL(x^0) \\subseteq U'$ for all $x^0 \\in U$. Since Alg.\\ \\ref{algo:QN} is a descent method, $x^0 \\in U$ implies $\\lim_{k \\rightarrow \\infty} x^k = \\bar{x} \\in \\calL(x^0) \\subseteq U'$. Application of Lemma \\ref{lem:intermediate_result} shows that $I_e(\\bar{x}) = I$ and the vectors $\\nabla f_i(\\bar{x})$, $i \\in I$, must be linearly dependent. Application of Lemma \\ref{lem:crit_limit} completes the proof.\n    \\end{proof}","sha256_utf8":"f245c5cc0c408a6b386beeeed6a5eee06b443b587f44b680542331392ee7392b"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"条件下极限点的临界性","source_units":[{"start":38626,"end":38925,"start_line":374,"end_line":376,"raw_tex":"\\begin{theorem} \\label{thm:criticality}\n        Assume that $f$ satisfies \\ref{assum:A2}. There is an open neighborhood $U \\subseteq \\R^n$ of $x^*$ such that if \\ref{behav:B1} holds for an initial point $x^0 \\in U$, then the limit $\\bar{x}$ of $(x^k)_k$ is a critical point of $f$.\n    \\end{theorem}","sha256_utf8":"90a0c0bc559fa7bbba69b8d32cbad0c02293de731eba564c2ad9645c384025e5"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"极限临界性完整证明","source_units":[{"start":38930,"end":39544,"start_line":377,"end_line":379,"raw_tex":"\\begin{proof}\n        Let $U'$ be the open neighborhood of $x^*$ from Lemma \\ref{lem:crit_limit}. By \\ref{enum:A2_2} there must be an open neighborhood $U \\subseteq \\R^n$ of $x^*$ such that $\\calL(x^0) \\subseteq U'$ for all $x^0 \\in U$. Since Alg.\\ \\ref{algo:QN} is a descent method, $x^0 \\in U$ implies $\\lim_{k \\rightarrow \\infty} x^k = \\bar{x} \\in \\calL(x^0) \\subseteq U'$. Application of Lemma \\ref{lem:intermediate_result} shows that $I_e(\\bar{x}) = I$ and the vectors $\\nabla f_i(\\bar{x})$, $i \\in I$, must be linearly dependent. Application of Lemma \\ref{lem:crit_limit} completes the proof.\n    \\end{proof}","sha256_utf8":"f245c5cc0c408a6b386beeeed6a5eee06b443b587f44b680542331392ee7392b"}]},{"id":"S003","semantic_type":"proof_step","kind":"proof_step","title":"局部化子水平集并应用两引理","source_units":[{"start":38930,"end":39544,"start_line":377,"end_line":379,"raw_tex":"\\begin{proof}\n        Let $U'$ be the open neighborhood of $x^*$ from Lemma \\ref{lem:crit_limit}. By \\ref{enum:A2_2} there must be an open neighborhood $U \\subseteq \\R^n$ of $x^*$ such that $\\calL(x^0) \\subseteq U'$ for all $x^0 \\in U$. Since Alg.\\ \\ref{algo:QN} is a descent method, $x^0 \\in U$ implies $\\lim_{k \\rightarrow \\infty} x^k = \\bar{x} \\in \\calL(x^0) \\subseteq U'$. Application of Lemma \\ref{lem:intermediate_result} shows that $I_e(\\bar{x}) = I$ and the vectors $\\nabla f_i(\\bar{x})$, $i \\in I$, must be linearly dependent. Application of Lemma \\ref{lem:crit_limit} completes the proof.\n    \\end{proof}","sha256_utf8":"f245c5cc0c408a6b386beeeed6a5eee06b443b587f44b680542331392ee7392b"}]},{"id":"S004","semantic_type":"lemma","kind":"lemma","title":"选择函数方向导数差消失","source_units":[{"start":25664,"end":25992,"start_line":235,"end_line":241,"raw_tex":"\\begin{lemma} \\label{lem:common_descent}\n        Assume that $f$ satisfies \\ref{assum:A1} and that \\ref{behav:B1} holds. Then\n        \\begin{align*}\n            \\lim_{k \\rightarrow \\infty} \\nabla f_i(x^k)^\\top p^k - \\nabla f_1(x^k)^\\top p^k = 0\n            \\quad \\forall i \\in \\{2,\\dots,m\\}.\n        \\end{align*}\n    \\end{lemma}","sha256_utf8":"68d546f31361249835e45ae4a5fc1cb6d4e4ef2d624850f68c7eedb01e2d8414"}]},{"id":"S005","semantic_type":"proof","kind":"proof","title":"方向导数差引理完整证明","source_units":[{"start":25997,"end":26886,"start_line":242,"end_line":255,"raw_tex":"\\begin{proof}\n        For $k \\in \\N$ let $i_k \\in I_g(x^k)$. Let $i \\in \\{2,\\dots,m\\}$. By definition of $p^k$ we have\n        \\begin{align*}\n            \\nabla f_i(x^k)^\\top p^k - \\nabla f_1(x^k)^\\top p^k\n            = -(\\nabla f_i(x^k) - \\nabla f_1(x^k))^\\top H_k \\nabla f_{i_k}(x^k).\n        \\end{align*}\n        Since $x^k \\rightarrow \\bar{x}$ (by \\ref{enum:B1_2}), $(H_k)_k$ is bounded (w.r.t.\\ the spectral norm, by \\ref{enum:B1_4}), and the selection functions are $\\C^1$ (by \\ref{assum:A1}), \\ref{enum:B1_5} implies \n        \\begin{align*}\n            &(\\nabla f_i(x^k) - \\nabla f_1(x^k))^\\top H_k \\\\\n            &= (\\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x}))^\\top H_k + (\\nabla f_i(x^k) - \\nabla f_1(x^k) - (\\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x})))^\\top H_k\n            \\rightarrow 0\n        \\end{align*}\n        as $k \\rightarrow \\infty$, completing the proof.\n    \\end{proof}","sha256_utf8":"58ce6194700a7f855443205e148aac404f13f36cd3f7f760b58d5b42a5e1182e"}]},{"id":"S006","semantic_type":"proof_step","kind":"proof_step","title":"展开搜索方向的导数差","source_units":[{"start":25997,"end":26305,"start_line":242,"end_line":247,"raw_tex":"\\begin{proof}\n        For $k \\in \\N$ let $i_k \\in I_g(x^k)$. Let $i \\in \\{2,\\dots,m\\}$. By definition of $p^k$ we have\n        \\begin{align*}\n            \\nabla f_i(x^k)^\\top p^k - \\nabla f_1(x^k)^\\top p^k\n            = -(\\nabla f_i(x^k) - \\nabla f_1(x^k))^\\top H_k \\nabla f_{i_k}(x^k).\n        \\end{align*}\n","sha256_utf8":"dac7cff36fac20e99d9f2ded3a350573669446fe210affefe5c11d8909a63d55"}]},{"id":"S007","semantic_type":"proof_step","kind":"proof_step","title":"利用梯度连续性与记忆假设取极限","source_units":[{"start":26305,"end":26886,"start_line":248,"end_line":255,"raw_tex":"        Since $x^k \\rightarrow \\bar{x}$ (by \\ref{enum:B1_2}), $(H_k)_k$ is bounded (w.r.t.\\ the spectral norm, by \\ref{enum:B1_4}), and the selection functions are $\\C^1$ (by \\ref{assum:A1}), \\ref{enum:B1_5} implies \n        \\begin{align*}\n            &(\\nabla f_i(x^k) - \\nabla f_1(x^k))^\\top H_k \\\\\n            &= (\\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x}))^\\top H_k + (\\nabla f_i(x^k) - \\nabla f_1(x^k) - (\\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x})))^\\top H_k\n            \\rightarrow 0\n        \\end{align*}\n        as $k \\rightarrow \\infty$, completing the proof.\n    \\end{proof}","sha256_utf8":"0e41a510cdd1ef3f072a9f4db14a2ba62e8a789f40b361ab0600d1256d1e5bae"}]},{"id":"S008","semantic_type":"lemma","kind":"lemma","title":"Wolfe步长下界","source_units":[{"start":27640,"end":28276,"start_line":259,"end_line":270,"raw_tex":"\\begin{lemma} \\label{lem:lower_bound_Wolfe}\n        Assume that $f$ satisfies \\ref{assum:A1} and let $x \\notin \\Omega$. Let $c_2 \\in (0,1)$ and $p \\in \\R^n$ with $\\nabla f(x)^\\top p < 0$. Let $t$ be a step length satisfying the second Wolfe condition \\eqref{eq:Wolfe_2}. For $i \\in I_g(x + t p)$ let $L$ be a Lipschitz constant of $\\nabla f_i$ on $\\conv(\\{x,x + tp\\})$. Then\n        \\begin{align*}\n            t \\geq \n            \\frac{1}{L \\| p \\|^2}\n            \\left( \n            -(1 - c_2) \\nabla f(x)^\\top p\n            +\n            (\\nabla f(x) - \\nabla f_{i}(x))^\\top p\n            \\right).\n        \\end{align*}\n    \\end{lemma}","sha256_utf8":"ebb7e3041e75ed51150647ab0e24f25d345637015b6bb87239f8cff2efe2dad9"}]},{"id":"S009","semantic_type":"proof","kind":"proof","title":"步长下界完整证明","source_units":[{"start":28281,"end":29543,"start_line":271,"end_line":288,"raw_tex":"\\begin{proof}\n        By the second Wolfe condition \\eqref{eq:Wolfe_2} it holds $\\nabla f_{i}(x + tp)^\\top p \\geq c_2 \\nabla f(x)^\\top p$. Subtracting $ \\nabla f_{i}(x)^\\top p$ on both sides yields\n        \\begin{equation} \\label{eq:proof_lem_lower_bound_Wolfe_1}\n            \\begin{aligned}\n                (\\nabla f_{i}(x + tp) - \\nabla f_{i}(x))^\\top p\n                &\\geq (c_2 \\nabla f(x) - \\nabla f_{i}(x))^\\top p \\\\\n                &= (c_2 \\nabla f(x) - \\nabla f(x) + \\nabla f(x) - \\nabla f_{i}(x))^\\top p \\\\\n                &= -(1 - c_2) \\nabla f(x)^\\top p + (\\nabla f(x) - \\nabla f_{i}(x))^\\top p.\n            \\end{aligned}\n        \\end{equation}\n        Since $\\nabla f_{i}$ is locally Lipschitz continuous by \\ref{assum:A1}, the left-hand side of \\eqref{eq:proof_lem_lower_bound_Wolfe_1} satisfies \n        \\begin{align} \\label{eq:proof_lem_lower_bound_Wolfe_2}\n            (\\nabla f_{i}(x + tp) - \\nabla f_{i}(x))^\\top p\n            \\leq \\| \\nabla f_{i}(x + tp) - \\nabla f_{i}(x) \\| \\| p \\|\n            \\leq t L \\| p \\|^2\n        \\end{align}\n        for a Lipschitz constant of $\\nabla f_i$ on $\\conv(\\{x,x + tp\\})$. Combining \\eqref{eq:proof_lem_lower_bound_Wolfe_1} and \\eqref{eq:proof_lem_lower_bound_Wolfe_2} completes the proof.\n    \\end{proof}","sha256_utf8":"9d93c55a5e88c163476b4072cbf73afc8051a30ecfca075012d240757ac357a6"}]},{"id":"S010","semantic_type":"proof_step","kind":"proof_step","title":"由曲率条件获得梯度差下界","source_units":[{"start":28281,"end":28938,"start_line":271,"end_line":280,"raw_tex":"\\begin{proof}\n        By the second Wolfe condition \\eqref{eq:Wolfe_2} it holds $\\nabla f_{i}(x + tp)^\\top p \\geq c_2 \\nabla f(x)^\\top p$. Subtracting $ \\nabla f_{i}(x)^\\top p$ on both sides yields\n        \\begin{equation} \\label{eq:proof_lem_lower_bound_Wolfe_1}\n            \\begin{aligned}\n                (\\nabla f_{i}(x + tp) - \\nabla f_{i}(x))^\\top p\n                &\\geq (c_2 \\nabla f(x) - \\nabla f_{i}(x))^\\top p \\\\\n                &= (c_2 \\nabla f(x) - \\nabla f(x) + \\nabla f(x) - \\nabla f_{i}(x))^\\top p \\\\\n                &= -(1 - c_2) \\nabla f(x)^\\top p + (\\nabla f(x) - \\nabla f_{i}(x))^\\top p.\n            \\end{aligned}\n        \\end{equation}\n","sha256_utf8":"c75df908691e56af9a559d41e023c6ae1354bf0199b8b4c2f6244b4c141daf49"}]},{"id":"S011","semantic_type":"proof_step","kind":"proof_step","title":"用Lipschitz条件控制梯度差","source_units":[{"start":28938,"end":29336,"start_line":281,"end_line":286,"raw_tex":"        Since $\\nabla f_{i}$ is locally Lipschitz continuous by \\ref{assum:A1}, the left-hand side of \\eqref{eq:proof_lem_lower_bound_Wolfe_1} satisfies \n        \\begin{align} \\label{eq:proof_lem_lower_bound_Wolfe_2}\n            (\\nabla f_{i}(x + tp) - \\nabla f_{i}(x))^\\top p\n            \\leq \\| \\nabla f_{i}(x + tp) - \\nabla f_{i}(x) \\| \\| p \\|\n            \\leq t L \\| p \\|^2\n        \\end{align}\n","sha256_utf8":"f9e7f7629645bbfba246af44aa7836580bf99fbc88ccac415972371d23a4fb3a"}]},{"id":"S012","semantic_type":"proof_step","kind":"proof_step","title":"合并上下界得到步长估计","source_units":[{"start":29336,"end":29543,"start_line":287,"end_line":288,"raw_tex":"        for a Lipschitz constant of $\\nabla f_i$ on $\\conv(\\{x,x + tp\\})$. Combining \\eqref{eq:proof_lem_lower_bound_Wolfe_1} and \\eqref{eq:proof_lem_lower_bound_Wolfe_2} completes the proof.\n    \\end{proof}","sha256_utf8":"2c9597661d4b87baf77fd48022fbf49611240731dee8975fb26c6fdfebf93e46"}]},{"id":"S013","semantic_type":"lemma","kind":"lemma","title":"矩阵聚点的核空间","source_units":[{"start":29670,"end":29945,"start_line":291,"end_line":293,"raw_tex":"\\begin{lemma} \\label{lem:ker_H_subsequence}\n        If $f$ satisfies \\ref{assum:A1} and \\ref{behav:B1} holds, then $(H_k)_k$ has an accumulation point. Furthermore, for all accumulation points $\\bar{H}$ of $(H_k)_k$, it holds $\\ker(\\bar{H}) = \\calN(\\bar{x})$.\n    \\end{lemma}","sha256_utf8":"321810a2a2fa8b2b297e7e0bb3ea0193967d304c5a0b45ec56efccdd0d4c756c"}]},{"id":"S014","semantic_type":"proof","kind":"proof","title":"核空间引理完整证明","source_units":[{"start":29950,"end":31101,"start_line":294,"end_line":303,"raw_tex":"\\begin{proof}\n        By \\ref{enum:B1_4} the spectral norm of $H_k$ is bounded above by $\\sigma_U$ for all $k \\in \\N$. Thus $(H_k)_k$ must have an accumulation point $\\bar{H}$. Let $(k_l)_l \\subseteq \\N$ be a strictly increasing, infinite sequence with $H_{k_l} \\rightarrow \\bar{H}$ for $l \\rightarrow \\infty$. Since $\\lambda_j^k \\geq \\sigma_L$ for all $j \\in \\{\\dim(\\calN(\\bar{x}))+1,\\dots,n\\}$, $k \\in \\N$, we have $\\dim(\\ker(\\bar{H})) \\leq \\dim(\\calN(\\bar{x}))$ due to continuity of eigenvalues (see, e.g., \\cite{A2013}, Theorem 5.2.2). Thus, it suffices to show that $\\calN(\\bar{x}) \\subseteq \\ker(\\bar{H})$. To this end, let $v \\in \\calN(\\bar{x})$. Then there are $\\alpha_i \\in \\R$, $i \\in \\{2,\\dots,m\\}$, such that $v = \\sum_{i = 2}^m \\alpha_i (\\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x}))$. By \\ref{enum:B1_5}, we obtain\n        \\begin{align*}\n            \\bar{H} v\n            = \\lim_{l \\rightarrow \\infty} H_{k_l} v\n            = \\lim_{l \\rightarrow \\infty} \\sum_{i = 2}^m \\alpha_i H_{k_l} (\\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x}))\n            = 0,\n        \\end{align*}\n        so $v \\in \\ker(\\bar{H})$, completing the proof. \n    \\end{proof}","sha256_utf8":"342b30e7d14575286032d27b8e1a47bd0b86518750ec0b766da29e4b7aaba0ad"}]},{"id":"S015","semantic_type":"proof_step","kind":"proof_step","title":"提取矩阵聚点并控制核维数","source_units":[{"start":29950,"end":30776,"start_line":294,"end_line":295,"raw_tex":"\\begin{proof}\n        By \\ref{enum:B1_4} the spectral norm of $H_k$ is bounded above by $\\sigma_U$ for all $k \\in \\N$. Thus $(H_k)_k$ must have an accumulation point $\\bar{H}$. Let $(k_l)_l \\subseteq \\N$ be a strictly increasing, infinite sequence with $H_{k_l} \\rightarrow \\bar{H}$ for $l \\rightarrow \\infty$. Since $\\lambda_j^k \\geq \\sigma_L$ for all $j \\in \\{\\dim(\\calN(\\bar{x}))+1,\\dots,n\\}$, $k \\in \\N$, we have $\\dim(\\ker(\\bar{H})) \\leq \\dim(\\calN(\\bar{x}))$ due to continuity of eigenvalues (see, e.g., \\cite{A2013}, Theorem 5.2.2). Thus, it suffices to show that $\\calN(\\bar{x}) \\subseteq \\ker(\\bar{H})$. To this end, let $v \\in \\calN(\\bar{x})$. Then there are $\\alpha_i \\in \\R$, $i \\in \\{2,\\dots,m\\}$, such that $v = \\sum_{i = 2}^m \\alpha_i (\\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x}))$. By \\ref{enum:B1_5}, we obtain\n","sha256_utf8":"f3b8193283ae3f45d17d4fa0d18106e0333c84272b2c05aeff292a66fb07d180"}]},{"id":"S016","semantic_type":"proof_step","kind":"proof_step","title":"证明梯度差空间包含于核","source_units":[{"start":30776,"end":31101,"start_line":296,"end_line":303,"raw_tex":"        \\begin{align*}\n            \\bar{H} v\n            = \\lim_{l \\rightarrow \\infty} H_{k_l} v\n            = \\lim_{l \\rightarrow \\infty} \\sum_{i = 2}^m \\alpha_i H_{k_l} (\\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x}))\n            = 0,\n        \\end{align*}\n        so $v \\in \\ker(\\bar{H})$, completing the proof. \n    \\end{proof}","sha256_utf8":"10f6a710176aacc01a4ab2d8617b66b756edc51a6c8e2446d593c10b628330de"}]},{"id":"S017","semantic_type":"lemma","kind":"lemma","title":"极限梯度落入梯度差空间","source_units":[{"start":31275,"end":31686,"start_line":307,"end_line":313,"raw_tex":"\\begin{lemma} \\label{lem:intermediate_result}\n        If $f$ satisfies \\ref{assum:A1} and \\ref{behav:B1} holds, then $I_e(\\bar{x}) = I = \\{1,\\dots,m\\}$ and\n        \\begin{align} \\label{eq:intermediate_result}\n            \\nabla f_i(\\bar{x}) \\in \\calN(\\bar{x}) \\quad \\forall i \\in I.\n        \\end{align}\n        In particular, the vectors $\\nabla f_i(\\bar{x})$, $i \\in I$, are linearly dependent.\n    \\end{lemma}","sha256_utf8":"4e8cebfdafead375e68ed6c3c7a877bbd1d6940b7d65d002bb8e045537d1d166"}]},{"id":"S018","semantic_type":"proof","kind":"proof","title":"极限梯度引理完整证明","source_units":[{"start":31691,"end":34923,"start_line":314,"end_line":349,"raw_tex":"\\begin{proof}\n        The equality $I_e(\\bar{x}) = \\{1,\\dots,m\\}$ follows from \\ref{enum:B1_3} and the fact that $I_g(x^j) \\subseteq I_e(x^j)$ for all $j \\in \\N$ (cf.\\ \\eqref{eq:def_Ag}). \\\\\n        \\textbf{Part 1:} We first show that $\\nabla f(x^k)^\\top p^k \\rightarrow 0$. To this end, assume that this does not hold. Then there is some $C > 0$ and a strictly increasing, infinite sequence $(k_l)_l \\subseteq \\N$ with $\\nabla f(x^{k_l})^\\top p^{k_l} < -C$ for all $l \\in \\N$. Since the number of selection functions is finite, we can assume w.l.o.g.\\ that there is some $i \\in I$ with $i \\in I_g(x^{k_l + 1})$ for all $l \\in \\N$. Furthermore, by local Lipschitz continuity of $\\nabla f_i$, we can assume w.l.o.g.\\ that there is a Lipschitz constant $L > 0$ for $\\nabla f_i$ on a superset of $\\{ x^{k_l} : l \\in \\N \\}$. By Lemma \\ref{lem:lower_bound_Wolfe} we have\n        \\begin{align*}\n            t_{k_l} \\geq \n            \\underbrace{\\frac{1}{L \\| p^{k_l} \\|^2}}_{\\text{(I)}}\n            \\left( \n            \\underbrace{-(1 - c_2) \\nabla f(x^{k_l})^\\top p^{k_l}}_{\\text{(II)}}\n            +\n            \\underbrace{(\\nabla f(x^{k_l}) - \\nabla f_{i}(x^{k_l}))^\\top p^{k_l}}_{\\text{(III)}}\n            \\right).\n        \\end{align*}\n        The fraction (I) is bounded below since $(p^{k_l})_l$ is bounded above due to \\ref{enum:B1_2} and \\ref{enum:B1_4}. The term (II) is bounded below by $(1 - c_2)C$. The term (III) vanishes by Lemma \\ref{lem:common_descent}, since\n        \\begin{align*}\n            \\nabla f(x^{k_l}) - \\nabla f_{i}(x^{k_l})\n            = \\nabla f(x^{k_l}) - \\nabla f_1(x^{k_l}) - (\\nabla f_i(x^{k_l}) - \\nabla f_1(x^{k_l})).\n        \\end{align*}\n        Thus, there is some $t_{\\text{min}} > 0$ such that $t_{k_l} \\geq t_{\\text{min}}$ for all $l \\in \\N$. By the first Wolfe condition \\eqref{eq:Wolfe_1}, this implies that\n        \\begin{align*}\n            f(x^{k_l + 1}) - f(x^{k_l}) \n            \\leq c_1 t_{k_l} \\nabla f(x^{k_l})^\\top p^{k_l}\n            < - c_1 t_{\\text{min}} C\n            < 0\n            \\quad \\forall l \\in \\N,\n        \\end{align*}\n        i.e., the objective value decreases by at least a constant amount infinitely many times. Since $(x^k)_k$ has a limit (by \\ref{enum:B1_2}), this contradicts the continuity of $f$. \\\\\n        \\textbf{Part 2:} Let $i \\in I$. By \\ref{enum:B1_3} there is a strictly increasing, infinite sequence $(k_l)_l \\subseteq \\N$ such that $i \\in I_g(x^{k_l})$ for all $l \\in \\N$. Since $(H_k)_k$ is bounded above by \\ref{enum:B1_4}, we can assume w.l.o.g.\\ that $(H_{k_l})_l$ converges to some $\\bar{H}$. By Part 1, we obtain\n        \\begin{align*}\n            0\n            &= \\lim_{l \\rightarrow \\infty} \\nabla f(x^{k_l})^\\top p^{k_l}\n            = -\\lim_{l \\rightarrow \\infty} \\nabla f(x^{k_l})^\\top H_{k_l} \\nabla f(x^{k_l})\n            = -\\lim_{l \\rightarrow \\infty} \\nabla f_i(x^{k_l})^\\top H_{k_l} \\nabla f_i(x^{k_l}) \\\\\n            &= -\\nabla f_i(\\bar{x})^\\top \\bar{H} \\nabla f_i(\\bar{x}).\n        \\end{align*}\n        Since $\\bar{H}$ is symmetric and positive semidefinite, this implies that $\\nabla f_i(\\bar{x}) \\in \\ker(\\bar{H})$ (see, e.g., \\cite{A2024}, 7.43). Application of Lemma \\ref{lem:ker_H_subsequence} completes the proof.\n    \\end{proof}","sha256_utf8":"c40a1ffb94fec09c2fe820f9d7515f6affcb30bfbdc456c63ca564a064addbce"}]},{"id":"S019","semantic_type":"proof_step","kind":"proof_step","title":"确认所有选择函数本质活跃","source_units":[{"start":31691,"end":31882,"start_line":314,"end_line":315,"raw_tex":"\\begin{proof}\n        The equality $I_e(\\bar{x}) = \\{1,\\dots,m\\}$ follows from \\ref{enum:B1_3} and the fact that $I_g(x^j) \\subseteq I_e(x^j)$ for all $j \\in \\N$ (cf.\\ \\eqref{eq:def_Ag}). \\\\\n","sha256_utf8":"dc0b5cdcb768705432a0b7f9cc060a92317cba73401fd2d6cd9026a6cbf3a894"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"反设方向导数不消失并估计步长","source_units":[{"start":31882,"end":32926,"start_line":316,"end_line":325,"raw_tex":"        \\textbf{Part 1:} We first show that $\\nabla f(x^k)^\\top p^k \\rightarrow 0$. To this end, assume that this does not hold. Then there is some $C > 0$ and a strictly increasing, infinite sequence $(k_l)_l \\subseteq \\N$ with $\\nabla f(x^{k_l})^\\top p^{k_l} < -C$ for all $l \\in \\N$. Since the number of selection functions is finite, we can assume w.l.o.g.\\ that there is some $i \\in I$ with $i \\in I_g(x^{k_l + 1})$ for all $l \\in \\N$. Furthermore, by local Lipschitz continuity of $\\nabla f_i$, we can assume w.l.o.g.\\ that there is a Lipschitz constant $L > 0$ for $\\nabla f_i$ on a superset of $\\{ x^{k_l} : l \\in \\N \\}$. By Lemma \\ref{lem:lower_bound_Wolfe} we have\n        \\begin{align*}\n            t_{k_l} \\geq \n            \\underbrace{\\frac{1}{L \\| p^{k_l} \\|^2}}_{\\text{(I)}}\n            \\left( \n            \\underbrace{-(1 - c_2) \\nabla f(x^{k_l})^\\top p^{k_l}}_{\\text{(II)}}\n            +\n            \\underbrace{(\\nabla f(x^{k_l}) - \\nabla f_{i}(x^{k_l}))^\\top p^{k_l}}_{\\text{(III)}}\n            \\right).\n        \\end{align*}\n","sha256_utf8":"e04cd1e6548163999814d66a8d044ca5e8479d7166151d7bc003dacb6578e428"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"控制步长下界中的三项","source_units":[{"start":32926,"end":33361,"start_line":326,"end_line":330,"raw_tex":"        The fraction (I) is bounded below since $(p^{k_l})_l$ is bounded above due to \\ref{enum:B1_2} and \\ref{enum:B1_4}. The term (II) is bounded below by $(1 - c_2)C$. The term (III) vanishes by Lemma \\ref{lem:common_descent}, since\n        \\begin{align*}\n            \\nabla f(x^{k_l}) - \\nabla f_{i}(x^{k_l})\n            = \\nabla f(x^{k_l}) - \\nabla f_1(x^{k_l}) - (\\nabla f_i(x^{k_l}) - \\nabla f_1(x^{k_l})).\n        \\end{align*}\n","sha256_utf8":"1dd72b71aaf3104c42bb181f69180a81b636b59599be67157b07c9252eef35d3"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"固定下降量与函数连续性矛盾","source_units":[{"start":33361,"end":33961,"start_line":331,"end_line":339,"raw_tex":"        Thus, there is some $t_{\\text{min}} > 0$ such that $t_{k_l} \\geq t_{\\text{min}}$ for all $l \\in \\N$. By the first Wolfe condition \\eqref{eq:Wolfe_1}, this implies that\n        \\begin{align*}\n            f(x^{k_l + 1}) - f(x^{k_l}) \n            \\leq c_1 t_{k_l} \\nabla f(x^{k_l})^\\top p^{k_l}\n            < - c_1 t_{\\text{min}} C\n            < 0\n            \\quad \\forall l \\in \\N,\n        \\end{align*}\n        i.e., the objective value decreases by at least a constant amount infinitely many times. Since $(x^k)_k$ has a limit (by \\ref{enum:B1_2}), this contradicts the continuity of $f$. \\\\\n","sha256_utf8":"60aa2be23338ebb26c8c1cec02af8e17739ff755fa34ab08368c9c259fe9744d"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"二次型极限推出梯度入核","source_units":[{"start":33961,"end":34923,"start_line":340,"end_line":349,"raw_tex":"        \\textbf{Part 2:} Let $i \\in I$. By \\ref{enum:B1_3} there is a strictly increasing, infinite sequence $(k_l)_l \\subseteq \\N$ such that $i \\in I_g(x^{k_l})$ for all $l \\in \\N$. Since $(H_k)_k$ is bounded above by \\ref{enum:B1_4}, we can assume w.l.o.g.\\ that $(H_{k_l})_l$ converges to some $\\bar{H}$. By Part 1, we obtain\n        \\begin{align*}\n            0\n            &= \\lim_{l \\rightarrow \\infty} \\nabla f(x^{k_l})^\\top p^{k_l}\n            = -\\lim_{l \\rightarrow \\infty} \\nabla f(x^{k_l})^\\top H_{k_l} \\nabla f(x^{k_l})\n            = -\\lim_{l \\rightarrow \\infty} \\nabla f_i(x^{k_l})^\\top H_{k_l} \\nabla f_i(x^{k_l}) \\\\\n            &= -\\nabla f_i(\\bar{x})^\\top \\bar{H} \\nabla f_i(\\bar{x}).\n        \\end{align*}\n        Since $\\bar{H}$ is symmetric and positive semidefinite, this implies that $\\nabla f_i(\\bar{x}) \\in \\ker(\\bar{H})$ (see, e.g., \\cite{A2024}, 7.43). Application of Lemma \\ref{lem:ker_H_subsequence} completes the proof.\n    \\end{proof}","sha256_utf8":"5db73c1e42a972dd4cddfb09a18600bf7d0ddb52fade6971aebecc18cb3d655f"}]},{"id":"S024","semantic_type":"lemma","kind":"lemma","title":"局部线性相关蕴含临界性","source_units":[{"start":36633,"end":36959,"start_line":365,"end_line":367,"raw_tex":"\\begin{lemma} \\label{lem:crit_limit}\n        Assume that $f$ satisfies \\ref{assum:A1} and \\ref{enum:A2_1}. Then there is an open neighborhood $U \\subseteq \\R^n$ of $x^*$ such if $x \\in U$ with $I_e(x) = I = \\{1,\\dots,m\\}$ and $\\nabla f_i(x)$, $i \\in I$, linearly dependent, then $x$ is a critical point of $f$.\n    \\end{lemma}","sha256_utf8":"704dd97eb21641b3ab9a49194ab7d23026a5f2c86f5d2be6af8dc4a31a30fa09"}]},{"id":"S025","semantic_type":"proof","kind":"proof","title":"局部临界性引理完整证明","source_units":[{"start":36964,"end":38210,"start_line":368,"end_line":371,"raw_tex":"\\begin{proof}\n        Assume that this does not hold. Then there is a sequence $(x^l)_l \\subseteq \\R^n$ with $x^l \\rightarrow x^*$, $I_e(x^l) = I$, and $\\nabla f_i(x^l)$, $i \\in I$, linearly dependent for all $l \\in \\N$, such that $x^l$ is not critical for any $l \\in \\N$. Linear dependence implies that there is a sequence $(\\beta^l)_l \\subseteq \\R^m \\setminus \\{ 0 \\}$ with $\\sum_{i = 1}^m \\beta_i^l \\nabla f_i(x^l) = 0$ for all $l \\in \\N$.\n        Assume w.l.o.g.\\ (via scaling) that $\\| \\beta^l \\|_\\infty := \\max_{i \\in I} |\\beta_i| = 1$ for all $l \\in \\N$. Then we can assume w.l.o.g.\\ that $(\\beta^l)_l$ has a limit $\\bar{\\beta} \\in \\R^m$ with $\\| \\bar{\\beta} \\|_\\infty = 1$. By continuity of $\\nabla f_i$, $i \\in I$, we have $\\sum_{i = 1}^m \\bar{\\beta}_i \\nabla f_i(x^*) = 0$. By affine independence of $\\nabla f_i(x^*)$, $i \\in I$, we must have $\\sum_{i = 1}^m \\bar{\\beta}_i \\neq 0$. Let $\\beta^* := \\bar{\\beta}/(\\sum_{i = 1}^m \\bar{\\beta}_i)$. Again using the affine independence, we must have $\\beta^* = \\alpha$. Now $\\alpha_i > 0$ for all $i \\in I$ implies that there is some $N \\in \\N$ such that $\\beta_i^l > 0$ for all $i \\in I$, $l > N$. This implies that $x^l$ is critical for any $l > N$, which is a contradiction.\n    \\end{proof}","sha256_utf8":"b9cca40c1df81331b85f8163651e292a54d3f4d9baac52793bd71c2137db4201"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"构造趋近极小点的反例序列","source_units":[{"start":36964,"end":37407,"start_line":368,"end_line":369,"raw_tex":"\\begin{proof}\n        Assume that this does not hold. Then there is a sequence $(x^l)_l \\subseteq \\R^n$ with $x^l \\rightarrow x^*$, $I_e(x^l) = I$, and $\\nabla f_i(x^l)$, $i \\in I$, linearly dependent for all $l \\in \\N$, such that $x^l$ is not critical for any $l \\in \\N$. Linear dependence implies that there is a sequence $(\\beta^l)_l \\subseteq \\R^m \\setminus \\{ 0 \\}$ with $\\sum_{i = 1}^m \\beta_i^l \\nabla f_i(x^l) = 0$ for all $l \\in \\N$.\n","sha256_utf8":"165f50ace27adcae53de2e332c64737055f8864173ca38eac04ea65973a77ca9"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"归一化系数并由正性导出矛盾","source_units":[{"start":37407,"end":38210,"start_line":370,"end_line":371,"raw_tex":"        Assume w.l.o.g.\\ (via scaling) that $\\| \\beta^l \\|_\\infty := \\max_{i \\in I} |\\beta_i| = 1$ for all $l \\in \\N$. Then we can assume w.l.o.g.\\ that $(\\beta^l)_l$ has a limit $\\bar{\\beta} \\in \\R^m$ with $\\| \\bar{\\beta} \\|_\\infty = 1$. By continuity of $\\nabla f_i$, $i \\in I$, we have $\\sum_{i = 1}^m \\bar{\\beta}_i \\nabla f_i(x^*) = 0$. By affine independence of $\\nabla f_i(x^*)$, $i \\in I$, we must have $\\sum_{i = 1}^m \\bar{\\beta}_i \\neq 0$. Let $\\beta^* := \\bar{\\beta}/(\\sum_{i = 1}^m \\bar{\\beta}_i)$. Again using the affine independence, we must have $\\beta^* = \\alpha$. Now $\\alpha_i > 0$ for all $i \\in I$ implies that there is some $N \\in \\N$ such that $\\beta_i^l > 0$ for all $i \\in I$, $l > N$. This implies that $x^l$ is critical for any $l > N$, which is a contradiction.\n    \\end{proof}","sha256_utf8":"b2f9029edbb4dc0a481deb37dd52e9fad4d863a0d73bb31557258c142048632f"}]},{"id":"S028","semantic_type":"definition","kind":"definition","title":"可微点与Wolfe条件","source_units":[{"start":11472,"end":11801,"start_line":115,"end_line":115,"raw_tex":"        Consider a function $f : \\R^n \\rightarrow \\R$ and denote the set of points at which $f$ is not differentiable by $\\Omega$. For $x \\notin \\Omega$, $p \\in \\R^n$ with $\\nabla f(x)^\\top p < 0$, and $c_1, c_2 \\in (0,1)$, $c_1 < c_2$, a step length $t > 0$ satisfies the \\emph{Wolfe conditions}, if $x + t p \\notin \\Omega$ and\n","sha256_utf8":"ff6b03d42eca73c2df18339cd5a56c89fd5f354ccbca893c7447295c6e79f515"},{"start":11809,"end":12030,"start_line":116,"end_line":119,"raw_tex":"\\begin{align}\n            f(x + t p) &\\leq f(x) + c_1 t \\nabla f(x)^\\top p, \\label{eq:Wolfe_1} \\tag{W1} \\\\\n            \\nabla f(x + t p)^\\top p &\\geq c_2 \\nabla f(x)^\\top p. \\label{eq:Wolfe_2} \\tag{W2}\n        \\end{align}","sha256_utf8":"be339070d20d2ebca0764a7ed046bc8b3d794485e783fb8511de5089d32ec15e"}]},{"id":"S029","semantic_type":"algorithm","kind":"algorithm","title":"一般拟牛顿算法","source_units":[{"start":12502,"end":13523,"start_line":121,"end_line":135,"raw_tex":"\\begin{algorithm} \n            \\caption{Quasi-Newton method}\n            \\label{algo:QN}\n            \\begin{algorithmic}[1] \n                \\Require Initial point $x^0 \\in \\R^n \\setminus \\Omega$, initial \\spd matrix $H_0 \\in \\R^{n \\times n}$, Wolfe parameters $c_1, c_2 \\in (0,1)$, $c_1 < c_2$.\n                \\For{$k = 0, 1, \\dots$}\n                    \\State If $\\nabla f(x^{k}) = 0$ then stop.\n                    \\State Set $p^k = - H_k \\nabla f(x^k)$.\n                    \\State Set $x^{k+1} = x^k + t_k p^k$, where $t_k$ satisfies the Wolfe conditions \\eqref{eq:Wolfe_1} and \\eqref{eq:Wolfe_2}.\n                    \\State If $f$ is not differentiable at $x^{k+1}$ then stop. \\label{state:breakdown}\n                    \\State For $y^k = \\nabla f(x^{k+1}) - \\nabla f(x^k)$ and $s^k = x^{k+1} - x^k$, compute \\label{state:QN_update}\n                    \\Statex \\hspace{\\algorithmicindent}a \\spd matrix $H_{k+1}$ with $H_{k+1} y^k = s^k$.\n                \\EndFor\n            \\end{algorithmic}\n        \\end{algorithm}","sha256_utf8":"741c2177139bc94f39d5d5a08efddb5b102bfd7fe0a1b8c243b8a66926699762"}]},{"id":"S030","semantic_type":"notation","kind":"notation","title":"终止及谱记号与BFGS更新","source_units":[{"start":13524,"end":13979,"start_line":136,"end_line":136,"raw_tex":"        As in \\cite{LO2013}, if the algorithm stops in Step \\ref{state:breakdown} with $x^{k+1} \\in \\Omega$, then we say that it \\emph{breaks down}. For $k \\in \\N \\cup \\{ 0 \\}$ we denote the eigenvalues of $H_k$ by $0 < \\lambda_1^k \\leq \\dots \\leq \\lambda_n^k$. There are many ways for computing the matrix $H_{k+1}$ in Step \\ref{state:QN_update} of Alg.\\ \\ref{algo:QN}. For all numerical experiments in this article, we use the \\emph{BFGS update}, where\n","sha256_utf8":"9f448e1bee81c34093582133348db13c8ccc11001eb72cf406fc3c139f84a67c"},{"start":13987,"end":14187,"start_line":137,"end_line":139,"raw_tex":"\\begin{align} \\label{eq:BFGS_update} \n            H_{k+1} = V_k H_k V_k^\\top + \\frac{s^k (s^k)^\\top}{(s^k)^\\top y^k} \\text{ for } V_k = \\id - \\frac{s^k (y^k)^\\top}{(s^k)^\\top y^k},\n        \\end{align}","sha256_utf8":"1bdb5e4b70954ed639f9bf5e9b14684159195e523244c20b53f02eab6967fce5"},{"start":14188,"end":14800,"start_line":140,"end_line":140,"raw_tex":"        with $\\id \\in \\R^{n \\times n}$ denoting the identity matrix. Throughout the article, the term \\emph{BFGS method} refers to the method that results from using \\eqref{eq:BFGS_update} for Step \\ref{state:QN_update} of Alg.\\ \\ref{algo:QN}. For computing the Wolfe step length in the numerical experiments, we use Alg.\\ 4.6 from \\cite{LO2013}. (However, for our theoretical results, the specific way in which the step length is computed does not matter.) Finally, as in \\cite{LO2013}, we do not actually check for differentiability in practice, since there is no reliable way to do so in a numerical setting.\n","sha256_utf8":"81aa1d24c9c5883d31f5459f6bab99aa5dbc92f3eafe6d8affe6991728956c45"}]},{"id":"S031","semantic_type":"definition","kind":"definition","title":"连续选择与选择函数","source_units":[{"start":14854,"end":15176,"start_line":144,"end_line":147,"raw_tex":"        A function $f : \\R^n \\rightarrow \\R$ is called a \\emph{continuous selection} of the functions $f_i : \\R^n \\rightarrow \\R$, $i \\in I := \\{1,\\dots,m\\}$, $m \\in \\N$, on $\\R^n$, if $f$ is continuous and \n        \\begin{align*}\n            f(x) \\in \\{ f_i(x) : i \\in I \\} \\quad \\forall x \\in \\R^n.\n        \\end{align*}\n","sha256_utf8":"b151121ba5ca60afe549ac0d9caae32725de1aa97cf911c8ae6f54009fae53c1"}]},{"id":"S032","semantic_type":"definition","kind":"definition","title":"活跃集与本质活跃集","source_units":[{"start":15176,"end":15618,"start_line":148,"end_line":151,"raw_tex":"        The functions $f_i$ are referred to as \\emph{selection functions} of $f$. A selection function $f_i$ is called \\emph{active} at $x$ if $f(x) = f_i(x)$. The \\emph{active set} at $x$ is defined by $I(x) := \\{ i \\in I : f(x) = f_i(x) \\}$ and the \\emph{essentially active set} is defined by\n        \\begin{align*}\n            I_e(x) := \\{ i \\in I : x \\in \\mathrm{cl}(\\mathrm{int}(\\{y \\in \\R^n : f(y) = f_i(y)\\})) \\}.\n        \\end{align*}\n","sha256_utf8":"f2c891b883682657b49b70c9db8b4bae10ba23e08c26205ae0e1e9b793533526"}]},{"id":"S033","semantic_type":"definition","kind":"definition","title":"梯度匹配活跃集","source_units":[{"start":15618,"end":15771,"start_line":152,"end_line":152,"raw_tex":"        In the following, let $f$ be a continuous selection of $\\C^1$-functions. By \\cite{U2002}, Proposition 2.24, for every $x \\notin \\Omega$, the set\n","sha256_utf8":"d4d91a77de1927b5519db2d6d7adcb52c931521f8bcd520d3ae545938f99a34f"},{"start":15779,"end":15901,"start_line":153,"end_line":155,"raw_tex":"\\begin{align} \\label{eq:def_Ag}\n            I_g(x) := \\{ i \\in I_e(x) : \\nabla f(x) = \\nabla f_i(x) \\}\n        \\end{align}","sha256_utf8":"87d54a414ddb3bc2de5e6fbfd89b102c2337a28d9133f8c6da9295cb0f996f2c"},{"start":15902,"end":15925,"start_line":156,"end_line":156,"raw_tex":"        is non-empty. \n","sha256_utf8":"06f7e08eddd1611f9750ffce323fd254c72720a3d0bc9a6f2de74e4185d2d7fa"}]},{"id":"S034","semantic_type":"definition","kind":"definition","title":"Clarke临界性与分片可微性","source_units":[{"start":15925,"end":16593,"start_line":157,"end_line":157,"raw_tex":"        By \\cite{S2012}, Corollary 4.1.1, $f$ is locally Lipschitz continuous. By \\cite{S2012}, Proposition 4.3.1, the Clarke subdifferential \\cite{C1990} of $f$ at $x$ is given by $\\partial f(x) = \\conv(\\{ \\nabla f_i(x) : i \\in I_e(x) \\})$. We say that $x$ is a \\emph{(Clarke) critical point} of $f$ if $0 \\in \\partial f(x)$, which is a necessary condition for local optimality (cf.\\ \\cite{BKM2014}, Theorem 3.17). Finally, if $f$ is a function such that for every $x \\in \\R^n$, there is an open neighborhood $U \\subseteq \\R^n$ of $x$ such that the restriction $f|_U$ is a continuous selection of $\\C^1$-functions, then $f$ is called \\emph{piecewise differentiable}.\n","sha256_utf8":"e8636387876a1cd7db9aacc26103e3f4c23818cb545f8b172410a35def3a2fe2"}]},{"id":"S035","semantic_type":"assumption","kind":"assumption","title":"选择函数的梯度正则性","source_units":[{"start":17275,"end":17498,"start_line":162,"end_line":164,"raw_tex":"\\begin{assum} \\label{assum:A1}\n        The function $f : \\R^n \\rightarrow \\R$ is a continuous selection of $\\C^1$-functions $f_i$, $i \\in I = \\{1,\\dots,m\\}$, whose gradients are locally Lipschitz continuous.\n    \\end{assum}","sha256_utf8":"ce2ba04a83cad366e57c6a5e696130f8604aaaaa4296a8e90ab9ee4613cef8e5"}]},{"id":"S036","semantic_type":"definition","kind":"definition","title":"极限梯度差空间","source_units":[{"start":18580,"end":18732,"start_line":167,"end_line":169,"raw_tex":"\\begin{align} \\label{eq:def_calN}\n        \\calN(\\bar{x}) := \\spn(\\{ \\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x}) : i \\in \\{2,\\dots,m\\} \\}).\n    \\end{align}","sha256_utf8":"78e430f826ce692a65c2e0dbf6a48ee4e0de4e66c4ba38865883d085e447dd52"}]},{"id":"S037","semantic_type":"assumption","kind":"assumption","title":"迭代与矩阵的渐近行为","source_units":[{"start":22655,"end":23823,"start_line":209,"end_line":224,"raw_tex":"    \\begin{behav} \\label{behav:B1}\n        Assume that Alg.\\ \\ref{algo:QN} is applied to a function $f$ satisfying \\ref{assum:A1} and generates a sequence $(x^k)_k$ and corresponding quasi-Newton matrices $(H_k)_k$ such that\n        \\begin{enumerate}[leftmargin=1.3cm,label=(B1.\\arabic*)]\n            \\item \\label{enum:B1_1} $(x^k)_k$ is an infinite sequence, i.e., Alg.\\ \\ref{algo:QN} did not break down and $\\nabla f(x^k) \\neq 0$ for all $k \\in \\N$,\n            \\item \\label{enum:B1_2} $(x^k)_k$ has a limit $\\bar{x} \\in \\R^n$,\n            \\item \\label{enum:B1_3} for all $i \\in I$, the set $\\{ k \\in \\N : i \\in I_g(x^k) \\}$ is infinite,\n            \\item \\label{enum:B1_4} there are $\\sigma_L, \\sigma_U \\in \\R^{>0}$ such that \n            \\begin{align*}\n                \\lambda_j^k \\in [\\sigma_L,\\sigma_U] \\quad \\forall j \\in \\{\\dim(\\calN(\\bar{x}))+1,\\dots,n\\}, k \\in \\N,\n            \\end{align*}\n            \\item \\label{enum:B1_5} it holds\n            \\begin{align*}\n                \\lim_{k \\rightarrow \\infty} H_k (\\nabla f_i(\\bar{x}) - \\nabla f_1(\\bar{x})) = 0 \\quad \\forall i \\in \\{2,\\dots,m\\}.\n            \\end{align*}\n        \\end{enumerate}\n    \\end{behav}\n","sha256_utf8":"872891327d913391f567cfb595ac54bc63e012a023ea60d771f36dce4ea96506"}]},{"id":"S038","semantic_type":"assumption","kind":"assumption","title":"稳定临界点与唯一全局极小点","source_units":[{"start":35661,"end":36432,"start_line":352,"end_line":362,"raw_tex":"\\begin{assum} \\label{assum:A2}\n        The function $f : \\R^n \\rightarrow \\R$ satisfies \\ref{assum:A1} and\n        \\begin{enumerate}[leftmargin=1.3cm,label=(A2.\\arabic*)]\n            \\item \\label{enum:A2_1} $f$ has a critical point $x^*$ with\n                \\begin{itemize}\n                    \\item $\\exists \\alpha \\in \\R^m$ with $\\alpha_i > 0$ for all $i \\in I$ and $\\sum_{i = 1}^m \\alpha_i \\nabla f_i(x^*) = 0$,\n                    \\item the vectors $\\nabla f_i(x^*)$, $i \\in I$, are affinely independent,\n                \\end{itemize}\n            \\item \\label{enum:A2_2} $x^*$ is the unique global minimum and there is some $z \\in \\R^n \\setminus \\{ x^* \\}$ such that $\\calL(z) := \\{ x \\in \\R^n : f(x) \\leq f(z) \\}$ is bounded.\n        \\end{enumerate}\n    \\end{assum}","sha256_utf8":"9dfcc8adff1332c869fc920e59ec583f98f57106889d3384a7b0a3cffd15e2d3"}]},{"id":"S039","semantic_type":"external_fact","kind":"external_fact","title":"梯度匹配活跃集非空","source_units":[],"declared_external":true,"declaration_text":["U2002, Proposition 2.24: 对有限个C¹函数的连续选择，在每个可微点，至少一个本质活跃选择函数的梯度等于该点的函数梯度。"]},{"id":"S040","semantic_type":"external_fact","kind":"external_fact","title":"连续选择的Clarke次微分公式","source_units":[],"declared_external":true,"declaration_text":["S2012, Corollary 4.1.1 and Proposition 4.3.1: 有限个C¹函数的连续选择局部Lipschitz，其Clarke次微分等于本质活跃选择函数梯度的凸包。"]},{"id":"S041","semantic_type":"external_fact","kind":"external_fact","title":"对称矩阵特征值连续性","source_units":[],"declared_external":true,"declaration_text":["A2013, Theorem 5.2.2: 实对称矩阵的有序特征值连续依赖于矩阵。"]},{"id":"S042","semantic_type":"external_fact","kind":"external_fact","title":"半正定二次型的零空间判据","source_units":[],"declared_external":true,"declaration_text":["A2024, 7.43: 若A为实对称半正定矩阵，则vᵀAv=0当且仅当Av=0。"]},{"id":"S043","semantic_type":"external_fact","kind":"external_fact","title":"仿射无关的系数唯一性","source_units":[],"declared_external":true,"declaration_text":["B1983, Section §1: 仿射无关向量不存在系数和为零的非平凡零线性组合；仿射表示的归一化系数唯一。"]},{"id":"S044","semantic_type":"external_fact","kind":"external_fact","title":"有限维紧性与子水平集局部化","source_units":[],"declared_external":true,"declaration_text":["有限维有界序列存在收敛子列，连续函数的子水平集闭。若连续函数具有唯一全局极小点及一个有界的非平凡子水平集，则足够低的子水平集包含于该极小点的任意给定邻域。"]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S003","target_id":"S002","kind":"supports"},{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S017","target_id":"S003","kind":"supports"},{"source_id":"S024","target_id":"S003","kind":"supports"},{"source_id":"S029","target_id":"S003","kind":"supports"},{"source_id":"S037","target_id":"S003","kind":"supports"},{"source_id":"S038","target_id":"S003","kind":"supports"},{"source_id":"S044","target_id":"S003","kind":"supports"},{"source_id":"S006","target_id":"S005","kind":"supports"},{"source_id":"S007","target_id":"S005","kind":"supports"},{"source_id":"S005","target_id":"S004","kind":"supports"},{"source_id":"S029","target_id":"S006","kind":"supports"},{"source_id":"S033","target_id":"S006","kind":"supports"},{"source_id":"S030","target_id":"S007","kind":"supports"},{"source_id":"S035","target_id":"S007","kind":"supports"},{"source_id":"S037","target_id":"S007","kind":"supports"},{"source_id":"S010","target_id":"S009","kind":"supports"},{"source_id":"S011","target_id":"S009","kind":"supports"},{"source_id":"S012","target_id":"S009","kind":"supports"},{"source_id":"S009","target_id":"S008","kind":"supports"},{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S011","target_id":"S012","kind":"supports"},{"source_id":"S028","target_id":"S010","kind":"supports"},{"source_id":"S033","target_id":"S010","kind":"supports"},{"source_id":"S035","target_id":"S011","kind":"supports"},{"source_id":"S015","target_id":"S014","kind":"supports"},{"source_id":"S016","target_id":"S014","kind":"supports"},{"source_id":"S014","target_id":"S013","kind":"supports"},{"source_id":"S015","target_id":"S016","kind":"supports"},{"source_id":"S030","target_id":"S015","kind":"supports"},{"source_id":"S036","target_id":"S015","kind":"supports"},{"source_id":"S037","target_id":"S015","kind":"supports"},{"source_id":"S041","target_id":"S015","kind":"supports"},{"source_id":"S044","target_id":"S015","kind":"supports"},{"source_id":"S036","target_id":"S016","kind":"supports"},{"source_id":"S037","target_id":"S016","kind":"supports"},{"source_id":"S019","target_id":"S018","kind":"supports"},{"source_id":"S020","target_id":"S018","kind":"supports"},{"source_id":"S021","target_id":"S018","kind":"supports"},{"source_id":"S022","target_id":"S018","kind":"supports"},{"source_id":"S023","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S032","target_id":"S019","kind":"supports"},{"source_id":"S033","target_id":"S019","kind":"supports"},{"source_id":"S037","target_id":"S019","kind":"supports"},{"source_id":"S008","target_id":"S020","kind":"supports"},{"source_id":"S029","target_id":"S020","kind":"supports"},{"source_id":"S033","target_id":"S020","kind":"supports"},{"source_id":"S035","target_id":"S020","kind":"supports"},{"source_id":"S037","target_id":"S020","kind":"supports"},{"source_id":"S004","target_id":"S021","kind":"supports"},{"source_id":"S020","target_id":"S021","kind":"supports"},{"source_id":"S030","target_id":"S021","kind":"supports"},{"source_id":"S035","target_id":"S021","kind":"supports"},{"source_id":"S037","target_id":"S021","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S028","target_id":"S022","kind":"supports"},{"source_id":"S029","target_id":"S022","kind":"supports"},{"source_id":"S035","target_id":"S022","kind":"supports"},{"source_id":"S037","target_id":"S022","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S013","target_id":"S023","kind":"supports"},{"source_id":"S029","target_id":"S023","kind":"supports"},{"source_id":"S030","target_id":"S023","kind":"supports"},{"source_id":"S033","target_id":"S023","kind":"supports"},{"source_id":"S035","target_id":"S023","kind":"supports"},{"source_id":"S037","target_id":"S023","kind":"supports"},{"source_id":"S042","target_id":"S023","kind":"supports"},{"source_id":"S044","target_id":"S023","kind":"supports"},{"source_id":"S026","target_id":"S025","kind":"supports"},{"source_id":"S027","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S024","kind":"supports"},{"source_id":"S026","target_id":"S027","kind":"supports"},{"source_id":"S034","target_id":"S026","kind":"supports"},{"source_id":"S038","target_id":"S026","kind":"supports"},{"source_id":"S034","target_id":"S027","kind":"supports"},{"source_id":"S035","target_id":"S027","kind":"supports"},{"source_id":"S038","target_id":"S027","kind":"supports"},{"source_id":"S043","target_id":"S027","kind":"supports"},{"source_id":"S044","target_id":"S027","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S035","kind":"supports"},{"source_id":"S032","target_id":"S033","kind":"supports"},{"source_id":"S032","target_id":"S034","kind":"supports"},{"source_id":"S039","target_id":"S033","kind":"supports"},{"source_id":"S040","target_id":"S034","kind":"supports"},{"source_id":"S036","target_id":"S037","kind":"supports"}]}}
{"record_id":"2511.10414v2","paper":{"title":"Minimizing Smooth Kurdyka-Łojasiewicz Functions via Generalized Descent Methods: Convergence Rate and Complexity","authors":["Masoud Ahookhosh","Susan Ghaderi","Alireza Kabgani","Morteza Rahimi"],"arxiv_id":"2511.10414","version":"v2","journal":"Journal of Optimization Theory and Applications","doi":"10.1007/s10957-026-03034-x","abstract_url":"https://arxiv.org/abs/2511.10414v2","pdf_url":"https://arxiv.org/pdf/2511.10414v2"},"context":{"raw_tex":"\\begin{equation}\\label{eq:prob}\n    \\min_{x\\in\\R^n}~f(x),\n\\end{equation}\n\n\\begin{ass}[Basic assumptions]\\label{ass:basic} For problem \\eqref{eq:prob}, we assume:\n\\begin{description}\n\\item[(a)] The function $\\func{f}{\\R^n}{\\R}$ is continuously differentiable, i.e., $f\\in\\C^1$, and possibly nonconvex;\n\\item[(b)] The set of minimizers $\\mathcal{X}^*:=\\argmin_{x\\in \\R^n} f(x)$ is nonempty, with $f^*$ denoting the optimal value.\n\\end{description}\n\\end{ass}\n\nIn this paper, $\\R^n$ denotes the $n$-dimensional real \\textit{Euclidean space} equipped with the standard \\textit{inner product} $\\innprod{\\cdot}{\\cdot}$ and the associated \\textit{Euclidean norm} $\\|\\cdot\\|=\\sqrt{\\innprod{\\cdot}{\\cdot}}$.\nThe notation $\\N$ stands for the set of \\textit{natural numbers}, and $\\N_{0}:=\\N\\cup \\{0\\}$.\nThe \\textit{open ball} of radius $r>0$ centered at $x\\in\\R^n$ is denoted by $B(x;r)$.\nThe \\textit{interior} of a set $S\\subseteq\\R^n$ is denoted by $\\interior\\, S$.\nThe \\textit{Euclidean distance} from a point $x\\in\\R^n$ to a nonempty set $S\\subseteq\\R^n$ is given by $\\dist(x;S)=\\inf_{z\\in S} \\|z-x\\|$.\nThe \\textit{ceiling function}, denoted by $\\lceil x \\rceil$, assigns to each real number $x$ the smallest integer greater than or equal to it.\nA point $\\ov{x}$ is called a \\textit{cluster point} of a sequence $\\{x^k\\}_{k \\in \\N_0}$ if there exists a subsequence converging to $\\ov{x}$.\n\n\nFor a function $h: \\R^n \\to \\Rinf:=\\R\\cup\\{+\\infty\\}$, the \\textit{effective domain} is defined as \n$\\dom{(h)}:= \\{x \\in   \\R^{n}\\mid~h(x)< + \\infty \\}$,\nand $h$ is called \\textit{proper} if $\\dom(h)\\neq \\emptyset$.\n%%%%%%%%%%%%%%%%%%%%%%%%%%\nA function $h$ is \\textit{lower semicontinuous} (lsc) at $\\ov{x} \\in \\R^n$ if, for any sequence \n$\\{x^k\\}_{k\\in \\mathbb{N}} \\subseteq \\R^n$ with $x^k \\to \\ov{x}$, it follows that\n$\\liminf_{k \\to +\\infty} h(x^k) \\geq h(\\ov{x})$.\n\nA proper function $h: \\R^n \\to \\Rinf$ is said to be \\textit{Fr\\'{e}chet differentiable} at $\\ov{x}\\in \\idom(h)$ with \\textit{Fr\\'{e}chet derivative} $\\nabla h(\\ov{x})$ if\n\\[\n\\lim_{x\\to \\ov{x}}\\tfrac{h(x) -h(\\ov{x}) - \\langle \\nabla h(\\ov{x}) , x - \\ov{x}\\rangle}{\\Vert x - \\ov{x}\\Vert}=0.\n\\]\nThe point $\\ov{x}\\in \\idom(h)$ is called a \\textit{critical point} of $h$ if $\\nabla h(\\ov x)=0$.\n\n\nThe \\textit{Fr\\'{e}chet/regular} and \\textit{Mordukhovich/limiting subdifferentials} of $h$ at $\\ov{x}\\in \\dom{(h)}$ \\cite{Mordukhovich2018,rockafellar2011variational} are defined, respectively, as\n\\[\n\\widehat{\\partial}h(\\ov{x}):=\\left\\{\\zeta\\in \\R^n\\mid~\\liminf_{x\\to \\ov{x}}\\tfrac{h(x)- h(\\ov{x}) - \\langle \\zeta, x - \\ov{x}\\rangle}{\\Vert x - \\ov{x}\\Vert}\\geq 0\\right\\},\n\\]\nand\n\\[\n\\partial h(\\ov{x}):=\\left\\{\\zeta\\in \\R^n\\mid~\\exists x^k\\to \\ov{x}, \\zeta^k\\in \\widehat{\\partial}h(x^k),~~\\text{with}~~h(x^k)\\to h(\\ov{x})~\\text{and}~ \\zeta^k\\to \\zeta\\right\\}.\n\\]\nIf $\\func{h}{\\R^n}{\\R}$ is continuously differentiable, i.e., $h\\in \\C^1$, then\n$\\partial h(\\ov{x})=\\{\\nabla h(\\ov{x})\\}$ \\cite[page 20]{Mordukhovich2018}.\n\n\n\\begin{defin}[\\textbf{Kurdyka-{\\L}ojasiewicz inequality}] \\label{def:LojaIneqDef}\n    Let $\\func{h}{\\R^n}{\\R}$ be a lsc function.\n    \\begin{enumerate}\n        \\item\\label{def:LojaIneqDef-l}\n             We say that $h$ satisfies the \\textit{Kurdyka-{\\L}ojasiewicz} (KL) inequality at $\\ov x \\in \\R^n$ if there exist constants $r, \\xi, \\tau>0$ and an exponent $\\vartheta \\in (0,1)$ such that\n            \\begin{equation}\\label{eq:LojaIneqGrad}\n                \\big(h(x)-h(\\ov x)\\big)^\\vartheta\\leq \\tau~\\dist(0;\\partial h(x)),\n            \\end{equation}\n            whenever $x\\in B(\\ov x;r)$ and $h(\\ov x)<h(x) <h(\\ov x)+\\xi$.\n        \\item\\label{def:LojaIneqDef-g}\n             Let $h^*:=\\inf_{x\\in \\R^n} h(x)>-\\infty$. The function $h$ is said to satisfy the \\textit{global Kurdyka-{\\L}ojasiewicz (global KL) inequality} if there exist constants $\\tau>0$ and an exponent $\\vartheta \\in (0,1)$ such that\n             \\begin{equation}\\label{eq:LojaIneqGrad:g}\n                \\big(h(x)-h^*\\big)^\\vartheta\\leq \\tau~\\dist(0;\\partial h(x)),\\quad\\forall x\\in \\R^n.\n            \\end{equation}\n    \\end{enumerate}\n    The function $h$ is said to be a \\textit{KL function} at $\\ov{x}$ if it satisfies the KL inequality at $\\ov{x}$.\nMoreover, $h$ is called a \\textit{global KL function} if it satisfies the global KL inequality.\n\\end{defin}\n\nIn this paper, we use the (global) KL inequality to unify the convergence analysis of generalized descent methods. For $h\\in \\mathcal{C}^1$, we have \n\\[\n\\dist(0;\\partial h(x))=\\Vert\\nabla h(x)\\Vert,\n\\]\n\n\n\\begin{algorithm}\n\\DontPrintSemicolon\n\\textbf{Initialization:} $x^0\\in\\R^n$,~$\\rho>0$,~$\\theta>1$,~ $k=0$;\n\n    \\Repeat{the stopping criteria holds\n    }{\n    Generate $x^{k+1}$ satisfying\n    \\begin{equation}\\label{eq:deal}\n    f(x^{k+1})\\leq f(x^{k})-\\rho\\Vert \\nabla f(x^{k})\\Vert^{\\theta};\n    \\end{equation}\n    \\\\Set $k=k+1$;\n    }\n\\caption{DEAL (Generalized DEscent ALgorithm) \\label{alg:deal}}\n\\end{algorithm}\n\n\\begin{ass}[Convergent subsequence]\\label{ass:bo}\nThe sequence $\\seq{x^k}$ generated by Algorithm \\ref{alg:deal} has a convergent subsequence.\n\\end{ass}","units":[{"start":8388,"end":8460,"start_line":206,"end_line":208,"raw_tex":"\\begin{equation}\\label{eq:prob}\n    \\min_{x\\in\\R^n}~f(x),\n\\end{equation}","sha256_utf8":"f02d2dcc0760a6c552abe37e37e6d2fb7556c00f635c2e58622433b7b9d92dca"},{"start":8502,"end":8883,"start_line":210,"end_line":215,"raw_tex":"\\begin{ass}[Basic assumptions]\\label{ass:basic} For problem \\eqref{eq:prob}, we assume:\n\\begin{description}\n\\item[(a)] The function $\\func{f}{\\R^n}{\\R}$ is continuously differentiable, i.e., $f\\in\\C^1$, and possibly nonconvex;\n\\item[(b)] The set of minimizers $\\mathcal{X}^*:=\\argmin_{x\\in \\R^n} f(x)$ is nonempty, with $f^*$ denoting the optimal value.\n\\end{description}\n\\end{ass}","sha256_utf8":"eff788ec80236d90c56117978f50035c85c52fa78931a3d3939cf474aea2028f"},{"start":18298,"end":19223,"start_line":297,"end_line":303,"raw_tex":"In this paper, $\\R^n$ denotes the $n$-dimensional real \\textit{Euclidean space} equipped with the standard \\textit{inner product} $\\innprod{\\cdot}{\\cdot}$ and the associated \\textit{Euclidean norm} $\\|\\cdot\\|=\\sqrt{\\innprod{\\cdot}{\\cdot}}$.\nThe notation $\\N$ stands for the set of \\textit{natural numbers}, and $\\N_{0}:=\\N\\cup \\{0\\}$.\nThe \\textit{open ball} of radius $r>0$ centered at $x\\in\\R^n$ is denoted by $B(x;r)$.\nThe \\textit{interior} of a set $S\\subseteq\\R^n$ is denoted by $\\interior\\, S$.\nThe \\textit{Euclidean distance} from a point $x\\in\\R^n$ to a nonempty set $S\\subseteq\\R^n$ is given by $\\dist(x;S)=\\inf_{z\\in S} \\|z-x\\|$.\nThe \\textit{ceiling function}, denoted by $\\lceil x \\rceil$, assigns to each real number $x$ the smallest integer greater than or equal to it.\nA point $\\ov{x}$ is called a \\textit{cluster point} of a sequence $\\{x^k\\}_{k \\in \\N_0}$ if there exists a subsequence converging to $\\ov{x}$.\n","sha256_utf8":"df14b97ba2ebe5de80e3e2d7a3fece2398329849c2cb9ec65682312d944c9040"},{"start":19289,"end":20151,"start_line":305,"end_line":317,"raw_tex":"For a function $h: \\R^n \\to \\Rinf:=\\R\\cup\\{+\\infty\\}$, the \\textit{effective domain} is defined as \n$\\dom{(h)}:= \\{x \\in   \\R^{n}\\mid~h(x)< + \\infty \\}$,\nand $h$ is called \\textit{proper} if $\\dom(h)\\neq \\emptyset$.\n%%%%%%%%%%%%%%%%%%%%%%%%%%\nA function $h$ is \\textit{lower semicontinuous} (lsc) at $\\ov{x} \\in \\R^n$ if, for any sequence \n$\\{x^k\\}_{k\\in \\mathbb{N}} \\subseteq \\R^n$ with $x^k \\to \\ov{x}$, it follows that\n$\\liminf_{k \\to +\\infty} h(x^k) \\geq h(\\ov{x})$.\n\nA proper function $h: \\R^n \\to \\Rinf$ is said to be \\textit{Fr\\'{e}chet differentiable} at $\\ov{x}\\in \\idom(h)$ with \\textit{Fr\\'{e}chet derivative} $\\nabla h(\\ov{x})$ if\n\\[\n\\lim_{x\\to \\ov{x}}\\tfrac{h(x) -h(\\ov{x}) - \\langle \\nabla h(\\ov{x}) , x - \\ov{x}\\rangle}{\\Vert x - \\ov{x}\\Vert}=0.\n\\]\nThe point $\\ov{x}\\in \\idom(h)$ is called a \\textit{critical point} of $h$ if $\\nabla h(\\ov x)=0$.\n","sha256_utf8":"4b6bb580b16cb064335e05917f2ed962dfa87edf5e05c8e66f404e175f9f9574"},{"start":20179,"end":20898,"start_line":320,"end_line":329,"raw_tex":"The \\textit{Fr\\'{e}chet/regular} and \\textit{Mordukhovich/limiting subdifferentials} of $h$ at $\\ov{x}\\in \\dom{(h)}$ \\cite{Mordukhovich2018,rockafellar2011variational} are defined, respectively, as\n\\[\n\\widehat{\\partial}h(\\ov{x}):=\\left\\{\\zeta\\in \\R^n\\mid~\\liminf_{x\\to \\ov{x}}\\tfrac{h(x)- h(\\ov{x}) - \\langle \\zeta, x - \\ov{x}\\rangle}{\\Vert x - \\ov{x}\\Vert}\\geq 0\\right\\},\n\\]\nand\n\\[\n\\partial h(\\ov{x}):=\\left\\{\\zeta\\in \\R^n\\mid~\\exists x^k\\to \\ov{x}, \\zeta^k\\in \\widehat{\\partial}h(x^k),~~\\text{with}~~h(x^k)\\to h(\\ov{x})~\\text{and}~ \\zeta^k\\to \\zeta\\right\\}.\n\\]\nIf $\\func{h}{\\R^n}{\\R}$ is continuously differentiable, i.e., $h\\in \\C^1$, then\n$\\partial h(\\ov{x})=\\{\\nabla h(\\ov{x})\\}$ \\cite[page 20]{Mordukhovich2018}.\n","sha256_utf8":"824870469a4b179ee1a811c6571cacd7aaaf90d5d5dd28ca23700e732a395d6a"},{"start":24453,"end":25786,"start_line":385,"end_line":402,"raw_tex":"\\begin{defin}[\\textbf{Kurdyka-{\\L}ojasiewicz inequality}] \\label{def:LojaIneqDef}\n    Let $\\func{h}{\\R^n}{\\R}$ be a lsc function.\n    \\begin{enumerate}\n        \\item\\label{def:LojaIneqDef-l}\n             We say that $h$ satisfies the \\textit{Kurdyka-{\\L}ojasiewicz} (KL) inequality at $\\ov x \\in \\R^n$ if there exist constants $r, \\xi, \\tau>0$ and an exponent $\\vartheta \\in (0,1)$ such that\n            \\begin{equation}\\label{eq:LojaIneqGrad}\n                \\big(h(x)-h(\\ov x)\\big)^\\vartheta\\leq \\tau~\\dist(0;\\partial h(x)),\n            \\end{equation}\n            whenever $x\\in B(\\ov x;r)$ and $h(\\ov x)<h(x) <h(\\ov x)+\\xi$.\n        \\item\\label{def:LojaIneqDef-g}\n             Let $h^*:=\\inf_{x\\in \\R^n} h(x)>-\\infty$. The function $h$ is said to satisfy the \\textit{global Kurdyka-{\\L}ojasiewicz (global KL) inequality} if there exist constants $\\tau>0$ and an exponent $\\vartheta \\in (0,1)$ such that\n             \\begin{equation}\\label{eq:LojaIneqGrad:g}\n                \\big(h(x)-h^*\\big)^\\vartheta\\leq \\tau~\\dist(0;\\partial h(x)),\\quad\\forall x\\in \\R^n.\n            \\end{equation}\n    \\end{enumerate}\n    The function $h$ is said to be a \\textit{KL function} at $\\ov{x}$ if it satisfies the KL inequality at $\\ov{x}$.\nMoreover, $h$ is called a \\textit{global KL function} if it satisfies the global KL inequality.\n\\end{defin}","sha256_utf8":"269f4049a9d746ec31dc6d7060beba6a605dbe99f9f2791600b328522f97fb06"},{"start":26529,"end":26731,"start_line":407,"end_line":410,"raw_tex":"In this paper, we use the (global) KL inequality to unify the convergence analysis of generalized descent methods. For $h\\in \\mathcal{C}^1$, we have \n\\[\n\\dist(0;\\partial h(x))=\\Vert\\nabla h(x)\\Vert,\n\\]\n","sha256_utf8":"54215ff005ca358bcf7f167b5d8c05355afa7f3d4cb8cdd4ed5d6781f3c06c61"},{"start":28136,"end":28553,"start_line":422,"end_line":435,"raw_tex":"\\begin{algorithm}\n\\DontPrintSemicolon\n\\textbf{Initialization:} $x^0\\in\\R^n$,~$\\rho>0$,~$\\theta>1$,~ $k=0$;\n\n    \\Repeat{the stopping criteria holds\n    }{\n    Generate $x^{k+1}$ satisfying\n    \\begin{equation}\\label{eq:deal}\n    f(x^{k+1})\\leq f(x^{k})-\\rho\\Vert \\nabla f(x^{k})\\Vert^{\\theta};\n    \\end{equation}\n    \\\\Set $k=k+1$;\n    }\n\\caption{DEAL (Generalized DEscent ALgorithm) \\label{alg:deal}}\n\\end{algorithm}","sha256_utf8":"101b76a033d0f565bacdf0fd7e0dc56296c794c6c7a65a7140f7103e2121183d"},{"start":29205,"end":29357,"start_line":440,"end_line":442,"raw_tex":"\\begin{ass}[Convergent subsequence]\\label{ass:bo}\nThe sequence $\\seq{x^k}$ generated by Algorithm \\ref{alg:deal} has a convergent subsequence.\n\\end{ass}","sha256_utf8":"75ee944bfc28ce4501c2c3bb3b889498ffed8e7aa0837f22f719879755456898"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{thm}[Linear convergence rate of DEAL] \n\\label{thm:LinConvWeakDeal-s}\n     Let the sequence $\\seq{x^k}$ be generated by Algorithm~\\ref{alg:deal} under Assumption~\\ref{ass:bo}. Let us assume that there exists $c>0$ such that \n     \\[\\Vert x^{k+1}-x^k\\Vert \\leq c\\Vert \\nabla f(x^k)\\Vert^{\\theta-1},\\]\n     for all sufficiently large $k\\in\\N$. If $\\ov x$ is a cluster point at which the KL inequality holds with exponent $\\vartheta$ and constant $\\tau>0$, then the following assertions hold:\n    \\begin{enumerate}\n        \\item\\label{thm:LinConvWeakDealFun-s:0}\n            If $\\theta<\\tfrac{1}{\\vartheta}$, the sequences $\\seq{f(x^k)-f(\\ov x)}$, $\\seq{\\|\\nabla f(x^k)\\|}$, and $\\seq{\\|x^k-\\ov x\\|}$ become zero in finitely many iterations;\n        \\item\\label{thm:LinConvWeakDealFun-s:1}\n            If $\\theta=\\tfrac{1}{\\vartheta}$, the sequences $\\seq{f(x^k)-f(\\ov x)}$, $\\seq{\\|\\nabla f(x^k)\\|}$, and $\\seq{\\|x^k-\\ov x\\|}$ converge linearly to $0$, i.e.,\n        \\begin{equation}\\label{eq:convRateFun-s}\n            f(x^{k+1})-f(\\ov x)\\leq q (f(x^k)-f(\\ov x)),\n        \\end{equation}\n        \\begin{equation}\\label{eq:LinIneqNablaDeal-s}\n            \\|\\nabla f(x^k)\\|\\le \\Big( \\tfrac{f(x^0)-f(\\ov x)}{\\rho}\\Big)^{\\tfrac{1}{\\theta}} q^{\\tfrac{k}{\\theta}},\n        \\end{equation}\n        \\begin{equation}\\label{eq:LinIneqIteDealp-s}\n        \\|x^k-\\ov x\\| \\le \\tfrac{c}{1-q^{\\tfrac{\\theta-1}{\\theta}}}\\Big( \\tfrac{f(x^0)-f(\\ov x)}{\\rho}\\Big)^{\\tfrac{\\theta-1}{\\theta}} q^{\\tfrac{k(\\theta-1)}{\\theta}},\n        \\end{equation}\n        as $k\\to \\infty$, where $q:= 1- \\tfrac{\\rho}{\\tau^{\\theta}}\\in (0, 1)$;\n        \\item\\label{thm:LinConvWeakDealFun-s:2}\n            If $\\theta>\\tfrac{1}{\\vartheta}$, there exists $\\mu>0$ such that \n            \\begin{equation}\\label{eq:convRateFun2-s}\n                0\\leq f(x^k)-f(\\ov x)\\leq \\mu k^{-\\tfrac{1}{\\vartheta \\theta-1}},\n            \\end{equation}\n            \\begin{equation}\\label{eq:SubLinIneqNablaDeal-s}\n            \\|\\nabla f(x^k)\\|\\leq \\big(\\tfrac{\\mu}{\\rho}\\big)^{\\tfrac{1}{\\theta}} k^{-\\tfrac{1}{\\theta(\\vartheta \\theta-1)}},\n            \\end{equation}\n            \\begin{equation}\\label{eq:SubLinIneqItarDeal-s}\n            \\|x^k-\\ov x\\|\\leq c  \\tau\\rho^{-1}(1-\\vartheta)^{-1}\\mu^{1-\\vartheta} k^{-\\tfrac{1-\\vartheta}{\\vartheta \\theta-1}},\n            \\end{equation}\n           for all sufficiently large $k\\in\\N_{0}$.\n    \\end{enumerate}\n\\end{thm}","units":[{"start":34671,"end":37080,"start_line":518,"end_line":551,"raw_tex":"\\begin{thm}[Linear convergence rate of DEAL] \n\\label{thm:LinConvWeakDeal-s}\n     Let the sequence $\\seq{x^k}$ be generated by Algorithm~\\ref{alg:deal} under Assumption~\\ref{ass:bo}. Let us assume that there exists $c>0$ such that \n     \\[\\Vert x^{k+1}-x^k\\Vert \\leq c\\Vert \\nabla f(x^k)\\Vert^{\\theta-1},\\]\n     for all sufficiently large $k\\in\\N$. If $\\ov x$ is a cluster point at which the KL inequality holds with exponent $\\vartheta$ and constant $\\tau>0$, then the following assertions hold:\n    \\begin{enumerate}\n        \\item\\label{thm:LinConvWeakDealFun-s:0}\n            If $\\theta<\\tfrac{1}{\\vartheta}$, the sequences $\\seq{f(x^k)-f(\\ov x)}$, $\\seq{\\|\\nabla f(x^k)\\|}$, and $\\seq{\\|x^k-\\ov x\\|}$ become zero in finitely many iterations;\n        \\item\\label{thm:LinConvWeakDealFun-s:1}\n            If $\\theta=\\tfrac{1}{\\vartheta}$, the sequences $\\seq{f(x^k)-f(\\ov x)}$, $\\seq{\\|\\nabla f(x^k)\\|}$, and $\\seq{\\|x^k-\\ov x\\|}$ converge linearly to $0$, i.e.,\n        \\begin{equation}\\label{eq:convRateFun-s}\n            f(x^{k+1})-f(\\ov x)\\leq q (f(x^k)-f(\\ov x)),\n        \\end{equation}\n        \\begin{equation}\\label{eq:LinIneqNablaDeal-s}\n            \\|\\nabla f(x^k)\\|\\le \\Big( \\tfrac{f(x^0)-f(\\ov x)}{\\rho}\\Big)^{\\tfrac{1}{\\theta}} q^{\\tfrac{k}{\\theta}},\n        \\end{equation}\n        \\begin{equation}\\label{eq:LinIneqIteDealp-s}\n        \\|x^k-\\ov x\\| \\le \\tfrac{c}{1-q^{\\tfrac{\\theta-1}{\\theta}}}\\Big( \\tfrac{f(x^0)-f(\\ov x)}{\\rho}\\Big)^{\\tfrac{\\theta-1}{\\theta}} q^{\\tfrac{k(\\theta-1)}{\\theta}},\n        \\end{equation}\n        as $k\\to \\infty$, where $q:= 1- \\tfrac{\\rho}{\\tau^{\\theta}}\\in (0, 1)$;\n        \\item\\label{thm:LinConvWeakDealFun-s:2}\n            If $\\theta>\\tfrac{1}{\\vartheta}$, there exists $\\mu>0$ such that \n            \\begin{equation}\\label{eq:convRateFun2-s}\n                0\\leq f(x^k)-f(\\ov x)\\leq \\mu k^{-\\tfrac{1}{\\vartheta \\theta-1}},\n            \\end{equation}\n            \\begin{equation}\\label{eq:SubLinIneqNablaDeal-s}\n            \\|\\nabla f(x^k)\\|\\leq \\big(\\tfrac{\\mu}{\\rho}\\big)^{\\tfrac{1}{\\theta}} k^{-\\tfrac{1}{\\theta(\\vartheta \\theta-1)}},\n            \\end{equation}\n            \\begin{equation}\\label{eq:SubLinIneqItarDeal-s}\n            \\|x^k-\\ov x\\|\\leq c  \\tau\\rho^{-1}(1-\\vartheta)^{-1}\\mu^{1-\\vartheta} k^{-\\tfrac{1-\\vartheta}{\\vartheta \\theta-1}},\n            \\end{equation}\n           for all sufficiently large $k\\in\\N_{0}$.\n    \\end{enumerate}\n\\end{thm}","sha256_utf8":"9e83408c65328ef92d23e5d499dc8d811b5ee4d8a4e4675d4bba19f608808297"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{fact}[\\textbf{Convergence of a sequence with positive elements}]\\cite[Lemma 1]{aragon2018accelerating}\n\\label{lem:convRate1} \n    Let $\\seq{s^k}$ be a sequence in $(0, +\\infty)$ converging to $0$, and let $\\varsigma\\ge 1$ and $\\omega>0$. Suppose that for some $k_0\\in \\N_0$,\n    \\begin{equation}\\label{eq:sk}\n        (s^k)^{\\varsigma}\\leq\\omega(s^k-s^{k+1}),\\quad\\forall k\\ge k_0.\n    \\end{equation}\n    Then the following assertions hold:\n    \\begin{description}\n        \\item[(a)] If $\\varsigma=1$, then necessarily $\\omega>1$, and the sequence $\\seq{s^k}$ converges linearly to $0$ with rate $1-\\tfrac{1}{\\omega}$, i.e.,\n        \\begin{equation}\\label{eq:sk1}\n        s^{k+1} \\le \\big(1-\\tfrac{1}{\\omega}\\big) s^{k}, \\quad \\forall k\\ge k_0;\n        \\end{equation}\n        \\item[(b)] If $\\varsigma>1$, then there exists a constant $\\mu>0$ such that \n        \\[\n            0\\leq s^k\\leq \\mu k^{-\\tfrac{1}{\\varsigma-1}},\n        \\]\n        for all sufficiently large $k\\in\\N_{0}$.\n    \\end{description}\n\\end{fact}\n\n\\begin{rem}\\label{rem:lem:convRate1}\n    In \\cite[Lemma 1]{aragon2018accelerating}, the condition \\eqref{eq:sk} is assumed to hold for all sufficiently large $k$. Since we are using a special case of that lemma, it suffices to require \\eqref{eq:sk} only for all $k\\geq k_0$ for some $k_0\\in \\N_0$. In fact, inequality \\eqref{eq:sk1} is obtained directly by setting $\\varsigma=1$ in \\eqref{eq:sk}. Moreover, since $s^{k+1}>0$, the inequality \\eqref{eq:sk1} immediately implies that $\\omega>1$.\\qed\n\\end{rem}\n\n\n\\begin{thm}[Global convergence of DEAL] \\label{thm:GlobConv-s}\nLet the sequence $\\seq{x^k}$ be generated by Algorithm \\ref{alg:deal}. The following assertions hold:\n    \\begin{enumerate}\n        \\item\\label{thm:GlobConv-s:1} \n            The sequence of function values $\\seq{f(x^k)}$ is non-increasing;\n        \\item\\label{thm:GlobConv-s:2}\n            We have $\\sum_{k=0}^\\infty\\Vert \\nabla f(x^k)\\Vert^{\\theta}< \\infty$ and $\\displaystyle\\lim_{k\\to\\infty} \\|\\nabla f(x^k)\\|=0$;\n        %, and $\\displaystyle\\lim_{k\\to\\infty} f(x^k)=f^*$;\n        \\item\\label{thm:GlobConv-s:3}\n            If Assumption~\\ref{ass:bo} holds, every cluster point $\\ov x$ of $\\seq{x^k}$ is a critical point, and $\\displaystyle \\lim_{k\\to\\infty}f(x^k)= f(\\ov x)$;\n        \\item\\label{thm:GlobConv-s:4}\n             If Assumption~\\ref{ass:bo} holds, and there exists $c>0$ such that\n            \\[\\|x^{k+1} - x^k\\| \\leq c \\|\\nabla f(x^k)\\|^{\\theta - 1},\\]\n            for all sufficiently large $k \\in \\N_{0}$, \n            and if the KL inequality holds at a cluster point $\\ov{x}$ of $\\seq{x^k}$, then $x^k\\to \\ov x$.\n    \\end{enumerate}\n\\end{thm}\n\n\\begin{proof}\nAssertion~\\ref{thm:GlobConv-s:1} follows directly from \\eqref{eq:deal} and $\\rho>0$.\n   For Assertion~\\ref{thm:GlobConv-s:2}, summing both sides of \\eqref{eq:deal} from $k=0$ to $k=N$ yields\n   \\begin{equation}\\label{eq:thm:GlobConv-s}\n       \\rho\\sum_{k=0}^N\\Vert \\nabla f(x^k)\\Vert^{\\theta}\\leq f(x^0)-f(x^{N+1})\\le f(x^0)- f^*,\n   \\end{equation}\n    which implies $\\sum_{k=0}^\\infty\\Vert \\nabla f(x^k)\\Vert^{\\theta}< \\infty$ and $\\displaystyle\\lim_{k\\to\\infty} \\|\\nabla f(x^k)\\|=0$, verifying the desired result.\nFor Assertion~\\ref{thm:GlobConv-s:3}, let a subsequence $\\subseq{x^{k_j}}$ converge to $\\ov{x}$. \nSince $f\\in \\mathcal{C}^1$, the gradient is continuous, and thus $\\nabla f(x^{k_j})\\to \\nabla f(\\ov x)$.\nHence, by Assertion~\\ref{thm:GlobConv-s:2}, $\\|\\nabla f(\\ov x)\\|=0$ and $\\ov x$ is a critical point.  \nSince $\\seq{f(x^k)}$ is non-increasing and bounded below, it converges, and therefore $f(x^k)\\to f(\\ov x)$.  \n\nFinally, for Assertion~\\ref{thm:GlobConv-s:4}, let $\\ov x$ be a cluster point where the KL inequality holds. \nLet $r,\\xi,\\tau>0$ be the constants associated with the KL inequality at $\\ov x$ (cf. Definition~\\ref{def:LojaIneqDef}). \nIf $\\nabla f(x^k)=0$ for some $k$, the result is trivial; otherwise, assume $\\nabla f(x^k)\\neq 0$ for all $k$.  \nSince $\\ov x$ is a cluster point, Assertion~\\ref{thm:GlobConv-s:3} yields that $f(x^k)\\to f(\\ov x)$. Moreover, there exists $\\ov k$ such that $x^{\\ov k}\\in\\Gamma$, where\n\\[\n\\Gamma := \\{x\\in B(\\ov x;r) : f(\\ov x)< f(x)< f(\\ov x)+\\xi\\}.\n\\]\nAdditionally, as $\\|x^{k+1}-x^k\\|\\to 0$, we have $x^k\\in\\Gamma$ for all $k\\geq \\ov k$, possibly enlarging $\\ov k$.  \nDefine $\\Delta_k:=\\left(f(x^k)-f(\\ov x)\\right)^{1-\\vartheta}$ for $k\\ge \\ov k$.\nBy concavity of $\\phi(t)=t^{1-\\vartheta}$ on $(0, +\\infty)$, together with \\eqref{eq:deal} and the KL inequality, we obtain \n \\begin{align}\\label{eq:th:conKL:z1-s}\n  \\Delta_{k}-\\Delta_{k+1} \\geq \n  (1-\\vartheta)\\left(f(x^{k})-f(\\ov x)\\right)^{-\\vartheta}\n\\left[f(x^{k})-f(x^{k +1})\\right]\\geq \n  \\tau^{-1}\\rho (1-\\vartheta)\n\\Vert\\nabla f(x^{k})\\Vert^{\\theta-1},\n\\end{align}\ni.e.,\n\\[\n\\Delta_{k}-\\Delta_{k+1} \\geq c^{-1}\\tau^{-1}\\rho (1-\\vartheta) \\|x^{k+1}-x^{k}\\|.\n\\]\nSumming both sides of this inequality over $k$ yields\n\\begin{align*}\n\\sum_{k=0}^{\\infty}\\Vert x^{k+1}-x^k\\Vert\n&\\leq \\sum_{k=0}^{\\ov k -1}\\Vert x^{k+1}-x^k\\Vert + \\sum_{k=\\ov k}^{\\infty}\\Vert x^{k+1}-x^k\\Vert\n\\le \\sum_{k=0}^{\\ov k -1}\\Vert x^{k+1}-x^k\\Vert+\\tfrac{c\\tau}{\\rho(1-\\vartheta)}\\sum_{k=\\ov k}^{\\infty}\\left(\\Delta_k-\\Delta_{k+1} \\right)\\\\\n&=\\sum_{k=0}^{\\ov k -1}\\Vert x^{k+1}-x^k\\Vert+\\tfrac{c\\tau}{\\rho(1-\\vartheta)}\\Delta_{\\ov k}<\\infty.\n\\end{align*}\nThus, $\\{x^k\\}_{k\\in \\N_0}$ is a Cauchy sequence and therefore converges to $\\ov x$.\n    \\qed\n\\end{proof}\n\n\\begin{proof}\nBy Theorem~\\ref{thm:GlobConv-s}~\\ref{thm:GlobConv-s:4}, $x^k\\to \\ov x$ and $f(x^k)\\to f(\\ov x)$.\nFrom the KL inequality, there exists $k_0\\in \\N$ such that\n\\[\n\\big(f(x^{k}) -f(\\ov x)\\big)^{\\vartheta}\\le \\tau \\|\\nabla f(x^k)\\|, \\quad\\quad \\forall k\\ge k_0.\n\\]\nCombining this with \\eqref{eq:deal} gives\n    \\begin{equation}\\label{eq:decIneqCSS1}\n        \\tfrac{\\rho}{\\tau^{\\theta}} \\big(f(x^{k}) -f(\\ov x)\\big)^{\\vartheta\\theta}\\le \\rho \\|\\nabla f(x^k)\\|^{\\theta} \\le f(x^{k}) - f(x^{k+1}) = \\big(f(x^{k}) -f(\\ov x)\\big) - \\big(f(x^{k+1})-f(\\ov x)\\big),\n    \\end{equation}\n    for all $k\\ge k_0$.\\\\\n    \\ref{thm:LinConvWeakDealFun-s:0} By indirect proof, assume that $x^{k+1}\\neq x^k$ for all $k\\in \\N_0$. It follows from \\eqref{eq:decIneqCSS1} that\n    \\[\n     \\big(f(x^k)-f(\\ov x)\\big)^{1-\\vartheta \\theta}\\geq \\tfrac{\\rho}{\\tau^\\theta}>0,\n     \\quad\\quad \\forall k\\ge k_0,\n    \\]\n    making a contradiction with $f(x^k)\\to f(\\ov x)$. Thus, all three sequences reach zero in a finite number of iterations.\\\\\n    \\ref{thm:LinConvWeakDealFun-s:1}-\\ref{thm:LinConvWeakDealFun-s:2} Considering \\eqref{eq:decIneqCSS1} and setting $s^k:=f(x^k)-f(\\ov x)$, $\\varsigma=\\vartheta\\theta$, and $\\omega:= \\tfrac{\\tau^{\\theta}}{\\rho}$, we obtain a recurrence of the form \\eqref{eq:sk}. \n    Applying Fact~\\ref{lem:convRate1} and Remark~\\ref{rem:lem:convRate1} yields $\\omega >1$, $q\\in (0,1)$, and the rates in \\eqref{eq:convRateFun-s} and \\eqref{eq:convRateFun2-s}.  Moreover, from \\eqref{eq:deal}, \n    \\begin{equation*}\n        \\rho \\|\\nabla f(x^k)\\|^\\theta \\le f(x^{k})-f(x^{k+1}) \\le f(x^k) -f(\\ov x),\n    \\end{equation*}\n   which, together with \\eqref{eq:convRateFun-s} and \\eqref{eq:convRateFun2-s}, implies \\eqref{eq:LinIneqNablaDeal-s} and \\eqref{eq:SubLinIneqNablaDeal-s}. To prove \\eqref{eq:LinIneqIteDealp-s}, using \n    $\\Vert x^{k+1}-x^k\\Vert \\leq c\\Vert \\nabla f(x^k)\\Vert^{\\theta-1}$ and \\eqref{eq:LinIneqNablaDeal-s}, we obtain\n    \\begin{align*}\n        \\|x^k - \\ov x\\| &\\le \\sum_{i\\ge k} \\|x^i -x^{i+1}\\|\\le \\sum_{i\\ge k} c\\|\\nabla f(x^i)\\|^{\\theta-1}\n        \\le c\\Big( \\tfrac{f(x^0)-f(\\ov x)}{\\rho}\\Big)^{\\tfrac{\\theta-1}{\\theta}} \\sum_{i\\ge k}   q^{\\tfrac{i(\\theta -1)}{\\theta}}\n        = c\\Big( \\tfrac{f(x^0)-f(\\ov x)}{\\rho}\\Big)^{\\tfrac{\\theta-1}{\\theta}}  \\tfrac{q^{\\tfrac{k(\\theta-1)}{\\theta}}}{1-q^{\\tfrac{\\theta -1}{\\theta}}},\n    \\end{align*}\n    for all $k\\ge k_0$.\nLet us set $\\theta>\\tfrac{1}{\\vartheta}$. \nArguing as in the proof of Theorem~\\ref{thm:GlobConv-s}~\\ref{thm:GlobConv-s:4} (see in particular the derivation of\n\\eqref{eq:th:conKL:z1-s}), we obtain\n\\begin{equation}\\label{eq:thm:LinConvWeakDealFun-s:1:a}\n    \\Delta_{k}-\\Delta_{k+1} \\geq \\tau^{-1}\\rho (1-\\vartheta)\\Vert\\nabla f(x^{k})\\Vert^{\\theta-1},\n\\end{equation}\nwhere $\\Delta_k:=\\left(f(x^k)-f(\\ov x)\\right)^{1-\\vartheta}$. \nMoreover, by \\eqref{eq:convRateFun2-s}, there exists a constant $\\mu>0$ such that, for all\nsufficiently large $k\\in\\N_{0}$,\n\\[\n\\Delta_k\\leq \\mu^{1-\\vartheta} k^{-\\tfrac{1-\\vartheta}{\\vartheta \\theta-1}}.\n\\]\nTogether with \\eqref{eq:thm:LinConvWeakDealFun-s:1:a}, this estimate yields, for all sufficiently\nlarge $k\\in\\N_{0}$,\n    \\begin{align*}\n        \\|x^k - \\ov x\\| \\le \\sum_{i\\ge k} c\\|\\nabla f(x^i)\\|^{\\theta-1} &\\le \n        c  \\tau\\rho^{-1}(1-\\vartheta)^{-1}\\sum_{i\\ge k} (\\Delta_{i}-\\Delta_{i+1})\n        \\leq c  \\tau\\rho^{-1}(1-\\vartheta)^{-1}\\Delta_{k}\n        \\\\&\n        \\le c  \\tau\\rho^{-1}(1-\\vartheta)^{-1}\\mu^{1-\\vartheta} k^{-\\tfrac{1-\\vartheta}{\\vartheta \\theta-1}},\n    \\end{align*}\nwhich completes the proof.\n     \\qed\n\\end{proof}","units":[{"start":22292,"end":23313,"start_line":351,"end_line":369,"raw_tex":"\\begin{fact}[\\textbf{Convergence of a sequence with positive elements}]\\cite[Lemma 1]{aragon2018accelerating}\n\\label{lem:convRate1} \n    Let $\\seq{s^k}$ be a sequence in $(0, +\\infty)$ converging to $0$, and let $\\varsigma\\ge 1$ and $\\omega>0$. Suppose that for some $k_0\\in \\N_0$,\n    \\begin{equation}\\label{eq:sk}\n        (s^k)^{\\varsigma}\\leq\\omega(s^k-s^{k+1}),\\quad\\forall k\\ge k_0.\n    \\end{equation}\n    Then the following assertions hold:\n    \\begin{description}\n        \\item[(a)] If $\\varsigma=1$, then necessarily $\\omega>1$, and the sequence $\\seq{s^k}$ converges linearly to $0$ with rate $1-\\tfrac{1}{\\omega}$, i.e.,\n        \\begin{equation}\\label{eq:sk1}\n        s^{k+1} \\le \\big(1-\\tfrac{1}{\\omega}\\big) s^{k}, \\quad \\forall k\\ge k_0;\n        \\end{equation}\n        \\item[(b)] If $\\varsigma>1$, then there exists a constant $\\mu>0$ such that \n        \\[\n            0\\leq s^k\\leq \\mu k^{-\\tfrac{1}{\\varsigma-1}},\n        \\]\n        for all sufficiently large $k\\in\\N_{0}$.\n    \\end{description}\n\\end{fact}","sha256_utf8":"3fe46776ba576eca0b0ea5eb47445b29515d63e31b40e9c1af0a2006610e4c5c"},{"start":23442,"end":23949,"start_line":372,"end_line":374,"raw_tex":"\\begin{rem}\\label{rem:lem:convRate1}\n    In \\cite[Lemma 1]{aragon2018accelerating}, the condition \\eqref{eq:sk} is assumed to hold for all sufficiently large $k$. Since we are using a special case of that lemma, it suffices to require \\eqref{eq:sk} only for all $k\\geq k_0$ for some $k_0\\in \\N_0$. In fact, inequality \\eqref{eq:sk1} is obtained directly by setting $\\varsigma=1$ in \\eqref{eq:sk}. Moreover, since $s^{k+1}>0$, the inequality \\eqref{eq:sk1} immediately implies that $\\omega>1$.\\qed\n\\end{rem}\n","sha256_utf8":"f8b6a1df2bdf68a690bfa0aa80096677ae0a9cdf061a2aa7f0368effbb6e8930"},{"start":30031,"end":31159,"start_line":449,"end_line":465,"raw_tex":"\\begin{thm}[Global convergence of DEAL] \\label{thm:GlobConv-s}\nLet the sequence $\\seq{x^k}$ be generated by Algorithm \\ref{alg:deal}. The following assertions hold:\n    \\begin{enumerate}\n        \\item\\label{thm:GlobConv-s:1} \n            The sequence of function values $\\seq{f(x^k)}$ is non-increasing;\n        \\item\\label{thm:GlobConv-s:2}\n            We have $\\sum_{k=0}^\\infty\\Vert \\nabla f(x^k)\\Vert^{\\theta}< \\infty$ and $\\displaystyle\\lim_{k\\to\\infty} \\|\\nabla f(x^k)\\|=0$;\n        %, and $\\displaystyle\\lim_{k\\to\\infty} f(x^k)=f^*$;\n        \\item\\label{thm:GlobConv-s:3}\n            If Assumption~\\ref{ass:bo} holds, every cluster point $\\ov x$ of $\\seq{x^k}$ is a critical point, and $\\displaystyle \\lim_{k\\to\\infty}f(x^k)= f(\\ov x)$;\n        \\item\\label{thm:GlobConv-s:4}\n             If Assumption~\\ref{ass:bo} holds, and there exists $c>0$ such that\n            \\[\\|x^{k+1} - x^k\\| \\leq c \\|\\nabla f(x^k)\\|^{\\theta - 1},\\]\n            for all sufficiently large $k \\in \\N_{0}$, \n            and if the KL inequality holds at a cluster point $\\ov{x}$ of $\\seq{x^k}$, then $x^k\\to \\ov x$.\n    \\end{enumerate}\n\\end{thm}","sha256_utf8":"8320d3d2c15d0f0e3ec5bd85fbd78565c39b5d6f6a85bbfe016e3f71e3913f21"},{"start":31169,"end":33933,"start_line":467,"end_line":509,"raw_tex":"\\begin{proof}\nAssertion~\\ref{thm:GlobConv-s:1} follows directly from \\eqref{eq:deal} and $\\rho>0$.\n   For Assertion~\\ref{thm:GlobConv-s:2}, summing both sides of \\eqref{eq:deal} from $k=0$ to $k=N$ yields\n   \\begin{equation}\\label{eq:thm:GlobConv-s}\n       \\rho\\sum_{k=0}^N\\Vert \\nabla f(x^k)\\Vert^{\\theta}\\leq f(x^0)-f(x^{N+1})\\le f(x^0)- f^*,\n   \\end{equation}\n    which implies $\\sum_{k=0}^\\infty\\Vert \\nabla f(x^k)\\Vert^{\\theta}< \\infty$ and $\\displaystyle\\lim_{k\\to\\infty} \\|\\nabla f(x^k)\\|=0$, verifying the desired result.\nFor Assertion~\\ref{thm:GlobConv-s:3}, let a subsequence $\\subseq{x^{k_j}}$ converge to $\\ov{x}$. \nSince $f\\in \\mathcal{C}^1$, the gradient is continuous, and thus $\\nabla f(x^{k_j})\\to \\nabla f(\\ov x)$.\nHence, by Assertion~\\ref{thm:GlobConv-s:2}, $\\|\\nabla f(\\ov x)\\|=0$ and $\\ov x$ is a critical point.  \nSince $\\seq{f(x^k)}$ is non-increasing and bounded below, it converges, and therefore $f(x^k)\\to f(\\ov x)$.  \n\nFinally, for Assertion~\\ref{thm:GlobConv-s:4}, let $\\ov x$ be a cluster point where the KL inequality holds. \nLet $r,\\xi,\\tau>0$ be the constants associated with the KL inequality at $\\ov x$ (cf. Definition~\\ref{def:LojaIneqDef}). \nIf $\\nabla f(x^k)=0$ for some $k$, the result is trivial; otherwise, assume $\\nabla f(x^k)\\neq 0$ for all $k$.  \nSince $\\ov x$ is a cluster point, Assertion~\\ref{thm:GlobConv-s:3} yields that $f(x^k)\\to f(\\ov x)$. Moreover, there exists $\\ov k$ such that $x^{\\ov k}\\in\\Gamma$, where\n\\[\n\\Gamma := \\{x\\in B(\\ov x;r) : f(\\ov x)< f(x)< f(\\ov x)+\\xi\\}.\n\\]\nAdditionally, as $\\|x^{k+1}-x^k\\|\\to 0$, we have $x^k\\in\\Gamma$ for all $k\\geq \\ov k$, possibly enlarging $\\ov k$.  \nDefine $\\Delta_k:=\\left(f(x^k)-f(\\ov x)\\right)^{1-\\vartheta}$ for $k\\ge \\ov k$.\nBy concavity of $\\phi(t)=t^{1-\\vartheta}$ on $(0, +\\infty)$, together with \\eqref{eq:deal} and the KL inequality, we obtain \n \\begin{align}\\label{eq:th:conKL:z1-s}\n  \\Delta_{k}-\\Delta_{k+1} \\geq \n  (1-\\vartheta)\\left(f(x^{k})-f(\\ov x)\\right)^{-\\vartheta}\n\\left[f(x^{k})-f(x^{k +1})\\right]\\geq \n  \\tau^{-1}\\rho (1-\\vartheta)\n\\Vert\\nabla f(x^{k})\\Vert^{\\theta-1},\n\\end{align}\ni.e.,\n\\[\n\\Delta_{k}-\\Delta_{k+1} \\geq c^{-1}\\tau^{-1}\\rho (1-\\vartheta) \\|x^{k+1}-x^{k}\\|.\n\\]\nSumming both sides of this inequality over $k$ yields\n\\begin{align*}\n\\sum_{k=0}^{\\infty}\\Vert x^{k+1}-x^k\\Vert\n&\\leq \\sum_{k=0}^{\\ov k -1}\\Vert x^{k+1}-x^k\\Vert + \\sum_{k=\\ov k}^{\\infty}\\Vert x^{k+1}-x^k\\Vert\n\\le \\sum_{k=0}^{\\ov k -1}\\Vert x^{k+1}-x^k\\Vert+\\tfrac{c\\tau}{\\rho(1-\\vartheta)}\\sum_{k=\\ov k}^{\\infty}\\left(\\Delta_k-\\Delta_{k+1} \\right)\\\\\n&=\\sum_{k=0}^{\\ov k -1}\\Vert x^{k+1}-x^k\\Vert+\\tfrac{c\\tau}{\\rho(1-\\vartheta)}\\Delta_{\\ov k}<\\infty.\n\\end{align*}\nThus, $\\{x^k\\}_{k\\in \\N_0}$ is a Cauchy sequence and therefore converges to $\\ov x$.\n    \\qed\n\\end{proof}","sha256_utf8":"874550a1a6ae3aabb23a4830a6238528c5398fa237dda43055c5b2685feff4a8"},{"start":37093,"end":40666,"start_line":554,"end_line":607,"raw_tex":"\\begin{proof}\nBy Theorem~\\ref{thm:GlobConv-s}~\\ref{thm:GlobConv-s:4}, $x^k\\to \\ov x$ and $f(x^k)\\to f(\\ov x)$.\nFrom the KL inequality, there exists $k_0\\in \\N$ such that\n\\[\n\\big(f(x^{k}) -f(\\ov x)\\big)^{\\vartheta}\\le \\tau \\|\\nabla f(x^k)\\|, \\quad\\quad \\forall k\\ge k_0.\n\\]\nCombining this with \\eqref{eq:deal} gives\n    \\begin{equation}\\label{eq:decIneqCSS1}\n        \\tfrac{\\rho}{\\tau^{\\theta}} \\big(f(x^{k}) -f(\\ov x)\\big)^{\\vartheta\\theta}\\le \\rho \\|\\nabla f(x^k)\\|^{\\theta} \\le f(x^{k}) - f(x^{k+1}) = \\big(f(x^{k}) -f(\\ov x)\\big) - \\big(f(x^{k+1})-f(\\ov x)\\big),\n    \\end{equation}\n    for all $k\\ge k_0$.\\\\\n    \\ref{thm:LinConvWeakDealFun-s:0} By indirect proof, assume that $x^{k+1}\\neq x^k$ for all $k\\in \\N_0$. It follows from \\eqref{eq:decIneqCSS1} that\n    \\[\n     \\big(f(x^k)-f(\\ov x)\\big)^{1-\\vartheta \\theta}\\geq \\tfrac{\\rho}{\\tau^\\theta}>0,\n     \\quad\\quad \\forall k\\ge k_0,\n    \\]\n    making a contradiction with $f(x^k)\\to f(\\ov x)$. Thus, all three sequences reach zero in a finite number of iterations.\\\\\n    \\ref{thm:LinConvWeakDealFun-s:1}-\\ref{thm:LinConvWeakDealFun-s:2} Considering \\eqref{eq:decIneqCSS1} and setting $s^k:=f(x^k)-f(\\ov x)$, $\\varsigma=\\vartheta\\theta$, and $\\omega:= \\tfrac{\\tau^{\\theta}}{\\rho}$, we obtain a recurrence of the form \\eqref{eq:sk}. \n    Applying Fact~\\ref{lem:convRate1} and Remark~\\ref{rem:lem:convRate1} yields $\\omega >1$, $q\\in (0,1)$, and the rates in \\eqref{eq:convRateFun-s} and \\eqref{eq:convRateFun2-s}.  Moreover, from \\eqref{eq:deal}, \n    \\begin{equation*}\n        \\rho \\|\\nabla f(x^k)\\|^\\theta \\le f(x^{k})-f(x^{k+1}) \\le f(x^k) -f(\\ov x),\n    \\end{equation*}\n   which, together with \\eqref{eq:convRateFun-s} and \\eqref{eq:convRateFun2-s}, implies \\eqref{eq:LinIneqNablaDeal-s} and \\eqref{eq:SubLinIneqNablaDeal-s}. To prove \\eqref{eq:LinIneqIteDealp-s}, using \n    $\\Vert x^{k+1}-x^k\\Vert \\leq c\\Vert \\nabla f(x^k)\\Vert^{\\theta-1}$ and \\eqref{eq:LinIneqNablaDeal-s}, we obtain\n    \\begin{align*}\n        \\|x^k - \\ov x\\| &\\le \\sum_{i\\ge k} \\|x^i -x^{i+1}\\|\\le \\sum_{i\\ge k} c\\|\\nabla f(x^i)\\|^{\\theta-1}\n        \\le c\\Big( \\tfrac{f(x^0)-f(\\ov x)}{\\rho}\\Big)^{\\tfrac{\\theta-1}{\\theta}} \\sum_{i\\ge k}   q^{\\tfrac{i(\\theta -1)}{\\theta}}\n        = c\\Big( \\tfrac{f(x^0)-f(\\ov x)}{\\rho}\\Big)^{\\tfrac{\\theta-1}{\\theta}}  \\tfrac{q^{\\tfrac{k(\\theta-1)}{\\theta}}}{1-q^{\\tfrac{\\theta -1}{\\theta}}},\n    \\end{align*}\n    for all $k\\ge k_0$.\nLet us set $\\theta>\\tfrac{1}{\\vartheta}$. \nArguing as in the proof of Theorem~\\ref{thm:GlobConv-s}~\\ref{thm:GlobConv-s:4} (see in particular the derivation of\n\\eqref{eq:th:conKL:z1-s}), we obtain\n\\begin{equation}\\label{eq:thm:LinConvWeakDealFun-s:1:a}\n    \\Delta_{k}-\\Delta_{k+1} \\geq \\tau^{-1}\\rho (1-\\vartheta)\\Vert\\nabla f(x^{k})\\Vert^{\\theta-1},\n\\end{equation}\nwhere $\\Delta_k:=\\left(f(x^k)-f(\\ov x)\\right)^{1-\\vartheta}$. \nMoreover, by \\eqref{eq:convRateFun2-s}, there exists a constant $\\mu>0$ such that, for all\nsufficiently large $k\\in\\N_{0}$,\n\\[\n\\Delta_k\\leq \\mu^{1-\\vartheta} k^{-\\tfrac{1-\\vartheta}{\\vartheta \\theta-1}}.\n\\]\nTogether with \\eqref{eq:thm:LinConvWeakDealFun-s:1:a}, this estimate yields, for all sufficiently\nlarge $k\\in\\N_{0}$,\n    \\begin{align*}\n        \\|x^k - \\ov x\\| \\le \\sum_{i\\ge k} c\\|\\nabla f(x^i)\\|^{\\theta-1} &\\le \n        c  \\tau\\rho^{-1}(1-\\vartheta)^{-1}\\sum_{i\\ge k} (\\Delta_{i}-\\Delta_{i+1})\n        \\leq c  \\tau\\rho^{-1}(1-\\vartheta)^{-1}\\Delta_{k}\n        \\\\&\n        \\le c  \\tau\\rho^{-1}(1-\\vartheta)^{-1}\\mu^{1-\\vartheta} k^{-\\tfrac{1-\\vartheta}{\\vartheta \\theta-1}},\n    \\end{align*}\nwhich completes the proof.\n     \\qed\n\\end{proof}","sha256_utf8":"ef533ee46049aa410d62e78206bb7e5f253a5c871c367315c881a7d7742646d1"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"theorem","kind":"theorem","title":"DEAL的统一收敛速率","source_units":[{"start":34671,"end":37080,"start_line":518,"end_line":551,"raw_tex":"\\begin{thm}[Linear convergence rate of DEAL] \n\\label{thm:LinConvWeakDeal-s}\n     Let the sequence $\\seq{x^k}$ be generated by Algorithm~\\ref{alg:deal} under Assumption~\\ref{ass:bo}. Let us assume that there exists $c>0$ such that \n     \\[\\Vert x^{k+1}-x^k\\Vert \\leq c\\Vert \\nabla f(x^k)\\Vert^{\\theta-1},\\]\n     for all sufficiently large $k\\in\\N$. If $\\ov x$ is a cluster point at which the KL inequality holds with exponent $\\vartheta$ and constant $\\tau>0$, then the following assertions hold:\n    \\begin{enumerate}\n        \\item\\label{thm:LinConvWeakDealFun-s:0}\n            If $\\theta<\\tfrac{1}{\\vartheta}$, the sequences $\\seq{f(x^k)-f(\\ov x)}$, $\\seq{\\|\\nabla f(x^k)\\|}$, and $\\seq{\\|x^k-\\ov x\\|}$ become zero in finitely many iterations;\n        \\item\\label{thm:LinConvWeakDealFun-s:1}\n            If $\\theta=\\tfrac{1}{\\vartheta}$, the sequences $\\seq{f(x^k)-f(\\ov x)}$, $\\seq{\\|\\nabla f(x^k)\\|}$, and $\\seq{\\|x^k-\\ov x\\|}$ converge linearly to $0$, i.e.,\n        \\begin{equation}\\label{eq:convRateFun-s}\n            f(x^{k+1})-f(\\ov x)\\leq q (f(x^k)-f(\\ov x)),\n        \\end{equation}\n        \\begin{equation}\\label{eq:LinIneqNablaDeal-s}\n            \\|\\nabla f(x^k)\\|\\le \\Big( \\tfrac{f(x^0)-f(\\ov x)}{\\rho}\\Big)^{\\tfrac{1}{\\theta}} q^{\\tfrac{k}{\\theta}},\n        \\end{equation}\n        \\begin{equation}\\label{eq:LinIneqIteDealp-s}\n        \\|x^k-\\ov x\\| \\le \\tfrac{c}{1-q^{\\tfrac{\\theta-1}{\\theta}}}\\Big( \\tfrac{f(x^0)-f(\\ov x)}{\\rho}\\Big)^{\\tfrac{\\theta-1}{\\theta}} q^{\\tfrac{k(\\theta-1)}{\\theta}},\n        \\end{equation}\n        as $k\\to \\infty$, where $q:= 1- \\tfrac{\\rho}{\\tau^{\\theta}}\\in (0, 1)$;\n        \\item\\label{thm:LinConvWeakDealFun-s:2}\n            If $\\theta>\\tfrac{1}{\\vartheta}$, there exists $\\mu>0$ such that \n            \\begin{equation}\\label{eq:convRateFun2-s}\n                0\\leq f(x^k)-f(\\ov x)\\leq \\mu k^{-\\tfrac{1}{\\vartheta \\theta-1}},\n            \\end{equation}\n            \\begin{equation}\\label{eq:SubLinIneqNablaDeal-s}\n            \\|\\nabla f(x^k)\\|\\leq \\big(\\tfrac{\\mu}{\\rho}\\big)^{\\tfrac{1}{\\theta}} k^{-\\tfrac{1}{\\theta(\\vartheta \\theta-1)}},\n            \\end{equation}\n            \\begin{equation}\\label{eq:SubLinIneqItarDeal-s}\n            \\|x^k-\\ov x\\|\\leq c  \\tau\\rho^{-1}(1-\\vartheta)^{-1}\\mu^{1-\\vartheta} k^{-\\tfrac{1-\\vartheta}{\\vartheta \\theta-1}},\n            \\end{equation}\n           for all sufficiently large $k\\in\\N_{0}$.\n    \\end{enumerate}\n\\end{thm}","sha256_utf8":"9e83408c65328ef92d23e5d499dc8d811b5ee4d8a4e4675d4bba19f608808297"}]},{"id":"S002","semantic_type":"proof","kind":"proof","title":"统一收敛速率完整证明","source_units":[{"start":37093,"end":40666,"start_line":554,"end_line":607,"raw_tex":"\\begin{proof}\nBy Theorem~\\ref{thm:GlobConv-s}~\\ref{thm:GlobConv-s:4}, $x^k\\to \\ov x$ and $f(x^k)\\to f(\\ov x)$.\nFrom the KL inequality, there exists $k_0\\in \\N$ such that\n\\[\n\\big(f(x^{k}) -f(\\ov x)\\big)^{\\vartheta}\\le \\tau \\|\\nabla f(x^k)\\|, \\quad\\quad \\forall k\\ge k_0.\n\\]\nCombining this with \\eqref{eq:deal} gives\n    \\begin{equation}\\label{eq:decIneqCSS1}\n        \\tfrac{\\rho}{\\tau^{\\theta}} \\big(f(x^{k}) -f(\\ov x)\\big)^{\\vartheta\\theta}\\le \\rho \\|\\nabla f(x^k)\\|^{\\theta} \\le f(x^{k}) - f(x^{k+1}) = \\big(f(x^{k}) -f(\\ov x)\\big) - \\big(f(x^{k+1})-f(\\ov x)\\big),\n    \\end{equation}\n    for all $k\\ge k_0$.\\\\\n    \\ref{thm:LinConvWeakDealFun-s:0} By indirect proof, assume that $x^{k+1}\\neq x^k$ for all $k\\in \\N_0$. It follows from \\eqref{eq:decIneqCSS1} that\n    \\[\n     \\big(f(x^k)-f(\\ov x)\\big)^{1-\\vartheta \\theta}\\geq \\tfrac{\\rho}{\\tau^\\theta}>0,\n     \\quad\\quad \\forall k\\ge k_0,\n    \\]\n    making a contradiction with $f(x^k)\\to f(\\ov x)$. Thus, all three sequences reach zero in a finite number of iterations.\\\\\n    \\ref{thm:LinConvWeakDealFun-s:1}-\\ref{thm:LinConvWeakDealFun-s:2} Considering \\eqref{eq:decIneqCSS1} and setting $s^k:=f(x^k)-f(\\ov x)$, $\\varsigma=\\vartheta\\theta$, and $\\omega:= \\tfrac{\\tau^{\\theta}}{\\rho}$, we obtain a recurrence of the form \\eqref{eq:sk}. \n    Applying Fact~\\ref{lem:convRate1} and Remark~\\ref{rem:lem:convRate1} yields $\\omega >1$, $q\\in (0,1)$, and the rates in \\eqref{eq:convRateFun-s} and \\eqref{eq:convRateFun2-s}.  Moreover, from \\eqref{eq:deal}, \n    \\begin{equation*}\n        \\rho \\|\\nabla f(x^k)\\|^\\theta \\le f(x^{k})-f(x^{k+1}) \\le f(x^k) -f(\\ov x),\n    \\end{equation*}\n   which, together with \\eqref{eq:convRateFun-s} and \\eqref{eq:convRateFun2-s}, implies \\eqref{eq:LinIneqNablaDeal-s} and \\eqref{eq:SubLinIneqNablaDeal-s}. To prove \\eqref{eq:LinIneqIteDealp-s}, using \n    $\\Vert x^{k+1}-x^k\\Vert \\leq c\\Vert \\nabla f(x^k)\\Vert^{\\theta-1}$ and \\eqref{eq:LinIneqNablaDeal-s}, we obtain\n    \\begin{align*}\n        \\|x^k - \\ov x\\| &\\le \\sum_{i\\ge k} \\|x^i -x^{i+1}\\|\\le \\sum_{i\\ge k} c\\|\\nabla f(x^i)\\|^{\\theta-1}\n        \\le c\\Big( \\tfrac{f(x^0)-f(\\ov x)}{\\rho}\\Big)^{\\tfrac{\\theta-1}{\\theta}} \\sum_{i\\ge k}   q^{\\tfrac{i(\\theta -1)}{\\theta}}\n        = c\\Big( \\tfrac{f(x^0)-f(\\ov x)}{\\rho}\\Big)^{\\tfrac{\\theta-1}{\\theta}}  \\tfrac{q^{\\tfrac{k(\\theta-1)}{\\theta}}}{1-q^{\\tfrac{\\theta -1}{\\theta}}},\n    \\end{align*}\n    for all $k\\ge k_0$.\nLet us set $\\theta>\\tfrac{1}{\\vartheta}$. \nArguing as in the proof of Theorem~\\ref{thm:GlobConv-s}~\\ref{thm:GlobConv-s:4} (see in particular the derivation of\n\\eqref{eq:th:conKL:z1-s}), we obtain\n\\begin{equation}\\label{eq:thm:LinConvWeakDealFun-s:1:a}\n    \\Delta_{k}-\\Delta_{k+1} \\geq \\tau^{-1}\\rho (1-\\vartheta)\\Vert\\nabla f(x^{k})\\Vert^{\\theta-1},\n\\end{equation}\nwhere $\\Delta_k:=\\left(f(x^k)-f(\\ov x)\\right)^{1-\\vartheta}$. \nMoreover, by \\eqref{eq:convRateFun2-s}, there exists a constant $\\mu>0$ such that, for all\nsufficiently large $k\\in\\N_{0}$,\n\\[\n\\Delta_k\\leq \\mu^{1-\\vartheta} k^{-\\tfrac{1-\\vartheta}{\\vartheta \\theta-1}}.\n\\]\nTogether with \\eqref{eq:thm:LinConvWeakDealFun-s:1:a}, this estimate yields, for all sufficiently\nlarge $k\\in\\N_{0}$,\n    \\begin{align*}\n        \\|x^k - \\ov x\\| \\le \\sum_{i\\ge k} c\\|\\nabla f(x^i)\\|^{\\theta-1} &\\le \n        c  \\tau\\rho^{-1}(1-\\vartheta)^{-1}\\sum_{i\\ge k} (\\Delta_{i}-\\Delta_{i+1})\n        \\leq c  \\tau\\rho^{-1}(1-\\vartheta)^{-1}\\Delta_{k}\n        \\\\&\n        \\le c  \\tau\\rho^{-1}(1-\\vartheta)^{-1}\\mu^{1-\\vartheta} k^{-\\tfrac{1-\\vartheta}{\\vartheta \\theta-1}},\n    \\end{align*}\nwhich completes the proof.\n     \\qed\n\\end{proof}","sha256_utf8":"ef533ee46049aa410d62e78206bb7e5f253a5c871c367315c881a7d7742646d1"}]},{"id":"S003","semantic_type":"theorem","kind":"theorem","title":"DEAL的全局收敛","source_units":[{"start":30031,"end":31159,"start_line":449,"end_line":465,"raw_tex":"\\begin{thm}[Global convergence of DEAL] \\label{thm:GlobConv-s}\nLet the sequence $\\seq{x^k}$ be generated by Algorithm \\ref{alg:deal}. The following assertions hold:\n    \\begin{enumerate}\n        \\item\\label{thm:GlobConv-s:1} \n            The sequence of function values $\\seq{f(x^k)}$ is non-increasing;\n        \\item\\label{thm:GlobConv-s:2}\n            We have $\\sum_{k=0}^\\infty\\Vert \\nabla f(x^k)\\Vert^{\\theta}< \\infty$ and $\\displaystyle\\lim_{k\\to\\infty} \\|\\nabla f(x^k)\\|=0$;\n        %, and $\\displaystyle\\lim_{k\\to\\infty} f(x^k)=f^*$;\n        \\item\\label{thm:GlobConv-s:3}\n            If Assumption~\\ref{ass:bo} holds, every cluster point $\\ov x$ of $\\seq{x^k}$ is a critical point, and $\\displaystyle \\lim_{k\\to\\infty}f(x^k)= f(\\ov x)$;\n        \\item\\label{thm:GlobConv-s:4}\n             If Assumption~\\ref{ass:bo} holds, and there exists $c>0$ such that\n            \\[\\|x^{k+1} - x^k\\| \\leq c \\|\\nabla f(x^k)\\|^{\\theta - 1},\\]\n            for all sufficiently large $k \\in \\N_{0}$, \n            and if the KL inequality holds at a cluster point $\\ov{x}$ of $\\seq{x^k}$, then $x^k\\to \\ov x$.\n    \\end{enumerate}\n\\end{thm}","sha256_utf8":"8320d3d2c15d0f0e3ec5bd85fbd78565c39b5d6f6a85bbfe016e3f71e3913f21"}]},{"id":"S004","semantic_type":"proof","kind":"proof","title":"全局收敛完整证明","source_units":[{"start":31169,"end":33933,"start_line":467,"end_line":509,"raw_tex":"\\begin{proof}\nAssertion~\\ref{thm:GlobConv-s:1} follows directly from \\eqref{eq:deal} and $\\rho>0$.\n   For Assertion~\\ref{thm:GlobConv-s:2}, summing both sides of \\eqref{eq:deal} from $k=0$ to $k=N$ yields\n   \\begin{equation}\\label{eq:thm:GlobConv-s}\n       \\rho\\sum_{k=0}^N\\Vert \\nabla f(x^k)\\Vert^{\\theta}\\leq f(x^0)-f(x^{N+1})\\le f(x^0)- f^*,\n   \\end{equation}\n    which implies $\\sum_{k=0}^\\infty\\Vert \\nabla f(x^k)\\Vert^{\\theta}< \\infty$ and $\\displaystyle\\lim_{k\\to\\infty} \\|\\nabla f(x^k)\\|=0$, verifying the desired result.\nFor Assertion~\\ref{thm:GlobConv-s:3}, let a subsequence $\\subseq{x^{k_j}}$ converge to $\\ov{x}$. \nSince $f\\in \\mathcal{C}^1$, the gradient is continuous, and thus $\\nabla f(x^{k_j})\\to \\nabla f(\\ov x)$.\nHence, by Assertion~\\ref{thm:GlobConv-s:2}, $\\|\\nabla f(\\ov x)\\|=0$ and $\\ov x$ is a critical point.  \nSince $\\seq{f(x^k)}$ is non-increasing and bounded below, it converges, and therefore $f(x^k)\\to f(\\ov x)$.  \n\nFinally, for Assertion~\\ref{thm:GlobConv-s:4}, let $\\ov x$ be a cluster point where the KL inequality holds. \nLet $r,\\xi,\\tau>0$ be the constants associated with the KL inequality at $\\ov x$ (cf. Definition~\\ref{def:LojaIneqDef}). \nIf $\\nabla f(x^k)=0$ for some $k$, the result is trivial; otherwise, assume $\\nabla f(x^k)\\neq 0$ for all $k$.  \nSince $\\ov x$ is a cluster point, Assertion~\\ref{thm:GlobConv-s:3} yields that $f(x^k)\\to f(\\ov x)$. Moreover, there exists $\\ov k$ such that $x^{\\ov k}\\in\\Gamma$, where\n\\[\n\\Gamma := \\{x\\in B(\\ov x;r) : f(\\ov x)< f(x)< f(\\ov x)+\\xi\\}.\n\\]\nAdditionally, as $\\|x^{k+1}-x^k\\|\\to 0$, we have $x^k\\in\\Gamma$ for all $k\\geq \\ov k$, possibly enlarging $\\ov k$.  \nDefine $\\Delta_k:=\\left(f(x^k)-f(\\ov x)\\right)^{1-\\vartheta}$ for $k\\ge \\ov k$.\nBy concavity of $\\phi(t)=t^{1-\\vartheta}$ on $(0, +\\infty)$, together with \\eqref{eq:deal} and the KL inequality, we obtain \n \\begin{align}\\label{eq:th:conKL:z1-s}\n  \\Delta_{k}-\\Delta_{k+1} \\geq \n  (1-\\vartheta)\\left(f(x^{k})-f(\\ov x)\\right)^{-\\vartheta}\n\\left[f(x^{k})-f(x^{k +1})\\right]\\geq \n  \\tau^{-1}\\rho (1-\\vartheta)\n\\Vert\\nabla f(x^{k})\\Vert^{\\theta-1},\n\\end{align}\ni.e.,\n\\[\n\\Delta_{k}-\\Delta_{k+1} \\geq c^{-1}\\tau^{-1}\\rho (1-\\vartheta) \\|x^{k+1}-x^{k}\\|.\n\\]\nSumming both sides of this inequality over $k$ yields\n\\begin{align*}\n\\sum_{k=0}^{\\infty}\\Vert x^{k+1}-x^k\\Vert\n&\\leq \\sum_{k=0}^{\\ov k -1}\\Vert x^{k+1}-x^k\\Vert + \\sum_{k=\\ov k}^{\\infty}\\Vert x^{k+1}-x^k\\Vert\n\\le \\sum_{k=0}^{\\ov k -1}\\Vert x^{k+1}-x^k\\Vert+\\tfrac{c\\tau}{\\rho(1-\\vartheta)}\\sum_{k=\\ov k}^{\\infty}\\left(\\Delta_k-\\Delta_{k+1} \\right)\\\\\n&=\\sum_{k=0}^{\\ov k -1}\\Vert x^{k+1}-x^k\\Vert+\\tfrac{c\\tau}{\\rho(1-\\vartheta)}\\Delta_{\\ov k}<\\infty.\n\\end{align*}\nThus, $\\{x^k\\}_{k\\in \\N_0}$ is a Cauchy sequence and therefore converges to $\\ov x$.\n    \\qed\n\\end{proof}","sha256_utf8":"874550a1a6ae3aabb23a4830a6238528c5398fa237dda43055c5b2685feff4a8"}]},{"id":"S005","semantic_type":"derived_fact","kind":"derived_fact","title":"正序列递推速率","source_units":[{"start":22292,"end":23313,"start_line":351,"end_line":369,"raw_tex":"\\begin{fact}[\\textbf{Convergence of a sequence with positive elements}]\\cite[Lemma 1]{aragon2018accelerating}\n\\label{lem:convRate1} \n    Let $\\seq{s^k}$ be a sequence in $(0, +\\infty)$ converging to $0$, and let $\\varsigma\\ge 1$ and $\\omega>0$. Suppose that for some $k_0\\in \\N_0$,\n    \\begin{equation}\\label{eq:sk}\n        (s^k)^{\\varsigma}\\leq\\omega(s^k-s^{k+1}),\\quad\\forall k\\ge k_0.\n    \\end{equation}\n    Then the following assertions hold:\n    \\begin{description}\n        \\item[(a)] If $\\varsigma=1$, then necessarily $\\omega>1$, and the sequence $\\seq{s^k}$ converges linearly to $0$ with rate $1-\\tfrac{1}{\\omega}$, i.e.,\n        \\begin{equation}\\label{eq:sk1}\n        s^{k+1} \\le \\big(1-\\tfrac{1}{\\omega}\\big) s^{k}, \\quad \\forall k\\ge k_0;\n        \\end{equation}\n        \\item[(b)] If $\\varsigma>1$, then there exists a constant $\\mu>0$ such that \n        \\[\n            0\\leq s^k\\leq \\mu k^{-\\tfrac{1}{\\varsigma-1}},\n        \\]\n        for all sufficiently large $k\\in\\N_{0}$.\n    \\end{description}\n\\end{fact}","sha256_utf8":"3fe46776ba576eca0b0ea5eb47445b29515d63e31b40e9c1af0a2006610e4c5c"}]},{"id":"S006","semantic_type":"proof","kind":"proof","title":"线性递推边界说明","source_units":[{"start":23442,"end":23949,"start_line":372,"end_line":374,"raw_tex":"\\begin{rem}\\label{rem:lem:convRate1}\n    In \\cite[Lemma 1]{aragon2018accelerating}, the condition \\eqref{eq:sk} is assumed to hold for all sufficiently large $k$. Since we are using a special case of that lemma, it suffices to require \\eqref{eq:sk} only for all $k\\geq k_0$ for some $k_0\\in \\N_0$. In fact, inequality \\eqref{eq:sk1} is obtained directly by setting $\\varsigma=1$ in \\eqref{eq:sk}. Moreover, since $s^{k+1}>0$, the inequality \\eqref{eq:sk1} immediately implies that $\\omega>1$.\\qed\n\\end{rem}\n","sha256_utf8":"f8b6a1df2bdf68a690bfa0aa80096677ae0a9cdf061a2aa7f0368effbb6e8930"}]},{"id":"S007","semantic_type":"external_fact","kind":"external_fact","title":"标准标量递推引理","source_units":[],"declared_external":true,"declaration_text":["For a positive sequence s^k→0 satisfying (s^k)^varsigma≤omega(s^k−s^{k+1}) for all k≥k0, with varsigma≥1 and omega>0: if varsigma=1, then omega>1 and s^{k+1}≤(1−1/omega)s^k; if varsigma>1, then s^k≤mu k^(−1/(varsigma−1)) eventually for some mu>0. This is the cited special case of aragon2018accelerating, Lemma 1."]},{"id":"S008","semantic_type":"problem_setting","kind":"problem_setting","title":"无约束最小化问题","source_units":[{"start":8388,"end":8460,"start_line":206,"end_line":208,"raw_tex":"\\begin{equation}\\label{eq:prob}\n    \\min_{x\\in\\R^n}~f(x),\n\\end{equation}","sha256_utf8":"f02d2dcc0760a6c552abe37e37e6d2fb7556c00f635c2e58622433b7b9d92dca"}]},{"id":"S009","semantic_type":"assumption","kind":"assumption","title":"光滑性与最优解存在","source_units":[{"start":8502,"end":8883,"start_line":210,"end_line":215,"raw_tex":"\\begin{ass}[Basic assumptions]\\label{ass:basic} For problem \\eqref{eq:prob}, we assume:\n\\begin{description}\n\\item[(a)] The function $\\func{f}{\\R^n}{\\R}$ is continuously differentiable, i.e., $f\\in\\C^1$, and possibly nonconvex;\n\\item[(b)] The set of minimizers $\\mathcal{X}^*:=\\argmin_{x\\in \\R^n} f(x)$ is nonempty, with $f^*$ denoting the optimal value.\n\\end{description}\n\\end{ass}","sha256_utf8":"eff788ec80236d90c56117978f50035c85c52fa78931a3d3939cf474aea2028f"}]},{"id":"S010","semantic_type":"notation","kind":"notation","title":"空间与序列记号","source_units":[{"start":18298,"end":19223,"start_line":297,"end_line":303,"raw_tex":"In this paper, $\\R^n$ denotes the $n$-dimensional real \\textit{Euclidean space} equipped with the standard \\textit{inner product} $\\innprod{\\cdot}{\\cdot}$ and the associated \\textit{Euclidean norm} $\\|\\cdot\\|=\\sqrt{\\innprod{\\cdot}{\\cdot}}$.\nThe notation $\\N$ stands for the set of \\textit{natural numbers}, and $\\N_{0}:=\\N\\cup \\{0\\}$.\nThe \\textit{open ball} of radius $r>0$ centered at $x\\in\\R^n$ is denoted by $B(x;r)$.\nThe \\textit{interior} of a set $S\\subseteq\\R^n$ is denoted by $\\interior\\, S$.\nThe \\textit{Euclidean distance} from a point $x\\in\\R^n$ to a nonempty set $S\\subseteq\\R^n$ is given by $\\dist(x;S)=\\inf_{z\\in S} \\|z-x\\|$.\nThe \\textit{ceiling function}, denoted by $\\lceil x \\rceil$, assigns to each real number $x$ the smallest integer greater than or equal to it.\nA point $\\ov{x}$ is called a \\textit{cluster point} of a sequence $\\{x^k\\}_{k \\in \\N_0}$ if there exists a subsequence converging to $\\ov{x}$.\n","sha256_utf8":"df14b97ba2ebe5de80e3e2d7a3fece2398329849c2cb9ec65682312d944c9040"}]},{"id":"S011","semantic_type":"definition","kind":"definition","title":"有效域与下半连续性","source_units":[{"start":19289,"end":19760,"start_line":305,"end_line":311,"raw_tex":"For a function $h: \\R^n \\to \\Rinf:=\\R\\cup\\{+\\infty\\}$, the \\textit{effective domain} is defined as \n$\\dom{(h)}:= \\{x \\in   \\R^{n}\\mid~h(x)< + \\infty \\}$,\nand $h$ is called \\textit{proper} if $\\dom(h)\\neq \\emptyset$.\n%%%%%%%%%%%%%%%%%%%%%%%%%%\nA function $h$ is \\textit{lower semicontinuous} (lsc) at $\\ov{x} \\in \\R^n$ if, for any sequence \n$\\{x^k\\}_{k\\in \\mathbb{N}} \\subseteq \\R^n$ with $x^k \\to \\ov{x}$, it follows that\n$\\liminf_{k \\to +\\infty} h(x^k) \\geq h(\\ov{x})$.\n","sha256_utf8":"d4fd2bf4ed3ebdf66ecc1f25ffb338994c0551930f0b6bb3f130e47a1d0c8c81"}]},{"id":"S012","semantic_type":"definition","kind":"definition","title":"可微性与临界点","source_units":[{"start":19761,"end":20151,"start_line":313,"end_line":317,"raw_tex":"A proper function $h: \\R^n \\to \\Rinf$ is said to be \\textit{Fr\\'{e}chet differentiable} at $\\ov{x}\\in \\idom(h)$ with \\textit{Fr\\'{e}chet derivative} $\\nabla h(\\ov{x})$ if\n\\[\n\\lim_{x\\to \\ov{x}}\\tfrac{h(x) -h(\\ov{x}) - \\langle \\nabla h(\\ov{x}) , x - \\ov{x}\\rangle}{\\Vert x - \\ov{x}\\Vert}=0.\n\\]\nThe point $\\ov{x}\\in \\idom(h)$ is called a \\textit{critical point} of $h$ if $\\nabla h(\\ov x)=0$.\n","sha256_utf8":"7f6a3c1739fb16d6e36cdfa9b516d9b227210ae355d9da5a74e520b94ed8b965"}]},{"id":"S013","semantic_type":"definition","kind":"definition","title":"次微分及光滑情形","source_units":[{"start":20179,"end":20898,"start_line":320,"end_line":329,"raw_tex":"The \\textit{Fr\\'{e}chet/regular} and \\textit{Mordukhovich/limiting subdifferentials} of $h$ at $\\ov{x}\\in \\dom{(h)}$ \\cite{Mordukhovich2018,rockafellar2011variational} are defined, respectively, as\n\\[\n\\widehat{\\partial}h(\\ov{x}):=\\left\\{\\zeta\\in \\R^n\\mid~\\liminf_{x\\to \\ov{x}}\\tfrac{h(x)- h(\\ov{x}) - \\langle \\zeta, x - \\ov{x}\\rangle}{\\Vert x - \\ov{x}\\Vert}\\geq 0\\right\\},\n\\]\nand\n\\[\n\\partial h(\\ov{x}):=\\left\\{\\zeta\\in \\R^n\\mid~\\exists x^k\\to \\ov{x}, \\zeta^k\\in \\widehat{\\partial}h(x^k),~~\\text{with}~~h(x^k)\\to h(\\ov{x})~\\text{and}~ \\zeta^k\\to \\zeta\\right\\}.\n\\]\nIf $\\func{h}{\\R^n}{\\R}$ is continuously differentiable, i.e., $h\\in \\C^1$, then\n$\\partial h(\\ov{x})=\\{\\nabla h(\\ov{x})\\}$ \\cite[page 20]{Mordukhovich2018}.\n","sha256_utf8":"824870469a4b179ee1a811c6571cacd7aaaf90d5d5dd28ca23700e732a395d6a"}]},{"id":"S014","semantic_type":"definition","kind":"definition","title":"局部与全局KL不等式","source_units":[{"start":24453,"end":25786,"start_line":385,"end_line":402,"raw_tex":"\\begin{defin}[\\textbf{Kurdyka-{\\L}ojasiewicz inequality}] \\label{def:LojaIneqDef}\n    Let $\\func{h}{\\R^n}{\\R}$ be a lsc function.\n    \\begin{enumerate}\n        \\item\\label{def:LojaIneqDef-l}\n             We say that $h$ satisfies the \\textit{Kurdyka-{\\L}ojasiewicz} (KL) inequality at $\\ov x \\in \\R^n$ if there exist constants $r, \\xi, \\tau>0$ and an exponent $\\vartheta \\in (0,1)$ such that\n            \\begin{equation}\\label{eq:LojaIneqGrad}\n                \\big(h(x)-h(\\ov x)\\big)^\\vartheta\\leq \\tau~\\dist(0;\\partial h(x)),\n            \\end{equation}\n            whenever $x\\in B(\\ov x;r)$ and $h(\\ov x)<h(x) <h(\\ov x)+\\xi$.\n        \\item\\label{def:LojaIneqDef-g}\n             Let $h^*:=\\inf_{x\\in \\R^n} h(x)>-\\infty$. The function $h$ is said to satisfy the \\textit{global Kurdyka-{\\L}ojasiewicz (global KL) inequality} if there exist constants $\\tau>0$ and an exponent $\\vartheta \\in (0,1)$ such that\n             \\begin{equation}\\label{eq:LojaIneqGrad:g}\n                \\big(h(x)-h^*\\big)^\\vartheta\\leq \\tau~\\dist(0;\\partial h(x)),\\quad\\forall x\\in \\R^n.\n            \\end{equation}\n    \\end{enumerate}\n    The function $h$ is said to be a \\textit{KL function} at $\\ov{x}$ if it satisfies the KL inequality at $\\ov{x}$.\nMoreover, $h$ is called a \\textit{global KL function} if it satisfies the global KL inequality.\n\\end{defin}","sha256_utf8":"269f4049a9d746ec31dc6d7060beba6a605dbe99f9f2791600b328522f97fb06"}]},{"id":"S015","semantic_type":"derived_fact","kind":"derived_fact","title":"光滑KL的梯度形式","source_units":[{"start":26529,"end":26731,"start_line":407,"end_line":410,"raw_tex":"In this paper, we use the (global) KL inequality to unify the convergence analysis of generalized descent methods. For $h\\in \\mathcal{C}^1$, we have \n\\[\n\\dist(0;\\partial h(x))=\\Vert\\nabla h(x)\\Vert,\n\\]\n","sha256_utf8":"54215ff005ca358bcf7f167b5d8c05355afa7f3d4cb8cdd4ed5d6781f3c06c61"}]},{"id":"S016","semantic_type":"algorithm","kind":"algorithm","title":"广义下降算法DEAL","source_units":[{"start":28136,"end":28553,"start_line":422,"end_line":435,"raw_tex":"\\begin{algorithm}\n\\DontPrintSemicolon\n\\textbf{Initialization:} $x^0\\in\\R^n$,~$\\rho>0$,~$\\theta>1$,~ $k=0$;\n\n    \\Repeat{the stopping criteria holds\n    }{\n    Generate $x^{k+1}$ satisfying\n    \\begin{equation}\\label{eq:deal}\n    f(x^{k+1})\\leq f(x^{k})-\\rho\\Vert \\nabla f(x^{k})\\Vert^{\\theta};\n    \\end{equation}\n    \\\\Set $k=k+1$;\n    }\n\\caption{DEAL (Generalized DEscent ALgorithm) \\label{alg:deal}}\n\\end{algorithm}","sha256_utf8":"101b76a033d0f565bacdf0fd7e0dc56296c794c6c7a65a7140f7103e2121183d"}]},{"id":"S017","semantic_type":"assumption","kind":"assumption","title":"收敛子序列假设","source_units":[{"start":29205,"end":29357,"start_line":440,"end_line":442,"raw_tex":"\\begin{ass}[Convergent subsequence]\\label{ass:bo}\nThe sequence $\\seq{x^k}$ generated by Algorithm \\ref{alg:deal} has a convergent subsequence.\n\\end{ass}","sha256_utf8":"75ee944bfc28ce4501c2c3bb3b889498ffed8e7aa0837f22f719879755456898"}]},{"id":"S018","semantic_type":"assumption","kind":"assumption","title":"速率定理的局部条件","source_units":[{"start":34747,"end":35167,"start_line":520,"end_line":522,"raw_tex":"     Let the sequence $\\seq{x^k}$ be generated by Algorithm~\\ref{alg:deal} under Assumption~\\ref{ass:bo}. Let us assume that there exists $c>0$ such that \n     \\[\\Vert x^{k+1}-x^k\\Vert \\leq c\\Vert \\nabla f(x^k)\\Vert^{\\theta-1},\\]\n     for all sufficiently large $k\\in\\N$. If $\\ov x$ is a cluster point at which the KL inequality holds with exponent $\\vartheta$ and constant $\\tau>0$, then the following assertions hold:\n","sha256_utf8":"7e235edf5cb498f85f372c6af3b8d1d797387b41535d0a2bccb3766325a9a16d"}]},{"id":"S019","semantic_type":"assumption","kind":"assumption","title":"全局收敛的步长控制","source_units":[{"start":30813,"end":31022,"start_line":460,"end_line":462,"raw_tex":"             If Assumption~\\ref{ass:bo} holds, and there exists $c>0$ such that\n            \\[\\|x^{k+1} - x^k\\| \\leq c \\|\\nabla f(x^k)\\|^{\\theta - 1},\\]\n            for all sufficiently large $k \\in \\N_{0}$, \n","sha256_utf8":"b692dd8cf90fb97db364b11d74011802b74c73d59f6dd1938bfc901c360531a0"}]},{"id":"S020","semantic_type":"proof_step","kind":"proof_step","title":"推出收缩率与参数下界","source_units":[{"start":23442,"end":23949,"start_line":372,"end_line":374,"raw_tex":"\\begin{rem}\\label{rem:lem:convRate1}\n    In \\cite[Lemma 1]{aragon2018accelerating}, the condition \\eqref{eq:sk} is assumed to hold for all sufficiently large $k$. Since we are using a special case of that lemma, it suffices to require \\eqref{eq:sk} only for all $k\\geq k_0$ for some $k_0\\in \\N_0$. In fact, inequality \\eqref{eq:sk1} is obtained directly by setting $\\varsigma=1$ in \\eqref{eq:sk}. Moreover, since $s^{k+1}>0$, the inequality \\eqref{eq:sk1} immediately implies that $\\omega>1$.\\qed\n\\end{rem}\n","sha256_utf8":"f8b6a1df2bdf68a690bfa0aa80096677ae0a9cdf061a2aa7f0368effbb6e8930"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"下降求和与梯度消失","source_units":[{"start":31169,"end":31699,"start_line":467,"end_line":473,"raw_tex":"\\begin{proof}\nAssertion~\\ref{thm:GlobConv-s:1} follows directly from \\eqref{eq:deal} and $\\rho>0$.\n   For Assertion~\\ref{thm:GlobConv-s:2}, summing both sides of \\eqref{eq:deal} from $k=0$ to $k=N$ yields\n   \\begin{equation}\\label{eq:thm:GlobConv-s}\n       \\rho\\sum_{k=0}^N\\Vert \\nabla f(x^k)\\Vert^{\\theta}\\leq f(x^0)-f(x^{N+1})\\le f(x^0)- f^*,\n   \\end{equation}\n    which implies $\\sum_{k=0}^\\infty\\Vert \\nabla f(x^k)\\Vert^{\\theta}< \\infty$ and $\\displaystyle\\lim_{k\\to\\infty} \\|\\nabla f(x^k)\\|=0$, verifying the desired result.\n","sha256_utf8":"294069aaaabce754574d25483eb10dce274da2a749a27161352f0cc66c8315ea"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"聚点临界性与函数值极限","source_units":[{"start":31699,"end":32115,"start_line":474,"end_line":477,"raw_tex":"For Assertion~\\ref{thm:GlobConv-s:3}, let a subsequence $\\subseq{x^{k_j}}$ converge to $\\ov{x}$. \nSince $f\\in \\mathcal{C}^1$, the gradient is continuous, and thus $\\nabla f(x^{k_j})\\to \\nabla f(\\ov x)$.\nHence, by Assertion~\\ref{thm:GlobConv-s:2}, $\\|\\nabla f(\\ov x)\\|=0$ and $\\ov x$ is a critical point.  \nSince $\\seq{f(x^k)}$ is non-increasing and bounded below, it converges, and therefore $f(x^k)\\to f(\\ov x)$.  \n","sha256_utf8":"bee66363a599c4cf4882862b717890f824d6089233f290308cbb75583bfd5d4b"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"进入KL有效邻域","source_units":[{"start":32116,"end":32816,"start_line":479,"end_line":486,"raw_tex":"Finally, for Assertion~\\ref{thm:GlobConv-s:4}, let $\\ov x$ be a cluster point where the KL inequality holds. \nLet $r,\\xi,\\tau>0$ be the constants associated with the KL inequality at $\\ov x$ (cf. Definition~\\ref{def:LojaIneqDef}). \nIf $\\nabla f(x^k)=0$ for some $k$, the result is trivial; otherwise, assume $\\nabla f(x^k)\\neq 0$ for all $k$.  \nSince $\\ov x$ is a cluster point, Assertion~\\ref{thm:GlobConv-s:3} yields that $f(x^k)\\to f(\\ov x)$. Moreover, there exists $\\ov k$ such that $x^{\\ov k}\\in\\Gamma$, where\n\\[\n\\Gamma := \\{x\\in B(\\ov x;r) : f(\\ov x)< f(x)< f(\\ov x)+\\xi\\}.\n\\]\nAdditionally, as $\\|x^{k+1}-x^k\\|\\to 0$, we have $x^k\\in\\Gamma$ for all $k\\geq \\ov k$, possibly enlarging $\\ov k$.  \n","sha256_utf8":"e465ac9e71a41f399500ff66dcae355f3307d6e5973f63ad77e695a2aa4d0a33"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"凹变换控制梯度幂","source_units":[{"start":32816,"end":33270,"start_line":487,"end_line":495,"raw_tex":"Define $\\Delta_k:=\\left(f(x^k)-f(\\ov x)\\right)^{1-\\vartheta}$ for $k\\ge \\ov k$.\nBy concavity of $\\phi(t)=t^{1-\\vartheta}$ on $(0, +\\infty)$, together with \\eqref{eq:deal} and the KL inequality, we obtain \n \\begin{align}\\label{eq:th:conKL:z1-s}\n  \\Delta_{k}-\\Delta_{k+1} \\geq \n  (1-\\vartheta)\\left(f(x^{k})-f(\\ov x)\\right)^{-\\vartheta}\n\\left[f(x^{k})-f(x^{k +1})\\right]\\geq \n  \\tau^{-1}\\rho (1-\\vartheta)\n\\Vert\\nabla f(x^{k})\\Vert^{\\theta-1},\n\\end{align}\n","sha256_utf8":"210257df3d481b9dbf4ee1f89de5efeaf84aa76e90acf0f080e8a04cefe3cd6d"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"控制相邻迭代距离","source_units":[{"start":33270,"end":33364,"start_line":496,"end_line":499,"raw_tex":"i.e.,\n\\[\n\\Delta_{k}-\\Delta_{k+1} \\geq c^{-1}\\tau^{-1}\\rho (1-\\vartheta) \\|x^{k+1}-x^{k}\\|.\n\\]\n","sha256_utf8":"d413d12137f4554ac5a57e4a63cf2c84883fc27295cb9a4afea2190d697d15b5"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"有限长度推出全序列收敛","source_units":[{"start":33364,"end":33933,"start_line":500,"end_line":509,"raw_tex":"Summing both sides of this inequality over $k$ yields\n\\begin{align*}\n\\sum_{k=0}^{\\infty}\\Vert x^{k+1}-x^k\\Vert\n&\\leq \\sum_{k=0}^{\\ov k -1}\\Vert x^{k+1}-x^k\\Vert + \\sum_{k=\\ov k}^{\\infty}\\Vert x^{k+1}-x^k\\Vert\n\\le \\sum_{k=0}^{\\ov k -1}\\Vert x^{k+1}-x^k\\Vert+\\tfrac{c\\tau}{\\rho(1-\\vartheta)}\\sum_{k=\\ov k}^{\\infty}\\left(\\Delta_k-\\Delta_{k+1} \\right)\\\\\n&=\\sum_{k=0}^{\\ov k -1}\\Vert x^{k+1}-x^k\\Vert+\\tfrac{c\\tau}{\\rho(1-\\vartheta)}\\Delta_{\\ov k}<\\infty.\n\\end{align*}\nThus, $\\{x^k\\}_{k\\in \\N_0}$ is a Cauchy sequence and therefore converges to $\\ov x$.\n    \\qed\n\\end{proof}","sha256_utf8":"2a1f396329e47fba9d09592de2e5dd04673b69342172c14cf699b5a65c963fdd"}]},{"id":"S027","semantic_type":"proof_step","kind":"proof_step","title":"收敛后应用KL不等式","source_units":[{"start":37093,"end":37366,"start_line":554,"end_line":559,"raw_tex":"\\begin{proof}\nBy Theorem~\\ref{thm:GlobConv-s}~\\ref{thm:GlobConv-s:4}, $x^k\\to \\ov x$ and $f(x^k)\\to f(\\ov x)$.\nFrom the KL inequality, there exists $k_0\\in \\N$ such that\n\\[\n\\big(f(x^{k}) -f(\\ov x)\\big)^{\\vartheta}\\le \\tau \\|\\nabla f(x^k)\\|, \\quad\\quad \\forall k\\ge k_0.\n\\]\n","sha256_utf8":"4ace3e1c481dc777e01d649f3915002095e200951c5d287a7e70b18fcac27b17"}]},{"id":"S028","semantic_type":"proof_step","kind":"proof_step","title":"建立函数差递推","source_units":[{"start":37366,"end":37704,"start_line":560,"end_line":564,"raw_tex":"Combining this with \\eqref{eq:deal} gives\n    \\begin{equation}\\label{eq:decIneqCSS1}\n        \\tfrac{\\rho}{\\tau^{\\theta}} \\big(f(x^{k}) -f(\\ov x)\\big)^{\\vartheta\\theta}\\le \\rho \\|\\nabla f(x^k)\\|^{\\theta} \\le f(x^{k}) - f(x^{k+1}) = \\big(f(x^{k}) -f(\\ov x)\\big) - \\big(f(x^{k+1})-f(\\ov x)\\big),\n    \\end{equation}\n    for all $k\\ge k_0$.\\\\\n","sha256_utf8":"62769357fd3c80d0dcccf5894680cea9acb5004085d7cf65bc479ec34383533d"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"低下降阶的有限终止","source_units":[{"start":37704,"end":38115,"start_line":565,"end_line":570,"raw_tex":"    \\ref{thm:LinConvWeakDealFun-s:0} By indirect proof, assume that $x^{k+1}\\neq x^k$ for all $k\\in \\N_0$. It follows from \\eqref{eq:decIneqCSS1} that\n    \\[\n     \\big(f(x^k)-f(\\ov x)\\big)^{1-\\vartheta \\theta}\\geq \\tfrac{\\rho}{\\tau^\\theta}>0,\n     \\quad\\quad \\forall k\\ge k_0,\n    \\]\n    making a contradiction with $f(x^k)\\to f(\\ov x)$. Thus, all three sequences reach zero in a finite number of iterations.\\\\\n","sha256_utf8":"d042f42693f2018854dd549dbae6697b8ae9890d479b20f4d8a7f2f6b1e01b1f"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"应用标量递推速率","source_units":[{"start":38115,"end":38594,"start_line":571,"end_line":572,"raw_tex":"    \\ref{thm:LinConvWeakDealFun-s:1}-\\ref{thm:LinConvWeakDealFun-s:2} Considering \\eqref{eq:decIneqCSS1} and setting $s^k:=f(x^k)-f(\\ov x)$, $\\varsigma=\\vartheta\\theta$, and $\\omega:= \\tfrac{\\tau^{\\theta}}{\\rho}$, we obtain a recurrence of the form \\eqref{eq:sk}. \n    Applying Fact~\\ref{lem:convRate1} and Remark~\\ref{rem:lem:convRate1} yields $\\omega >1$, $q\\in (0,1)$, and the rates in \\eqref{eq:convRateFun-s} and \\eqref{eq:convRateFun2-s}.  Moreover, from \\eqref{eq:deal}, \n","sha256_utf8":"cc18182ff4d22207df6a1177422f03a31900c91358783386a78ef10a4fac30a2"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"由函数差控制梯度","source_units":[{"start":38594,"end":38922,"start_line":573,"end_line":576,"raw_tex":"    \\begin{equation*}\n        \\rho \\|\\nabla f(x^k)\\|^\\theta \\le f(x^{k})-f(x^{k+1}) \\le f(x^k) -f(\\ov x),\n    \\end{equation*}\n   which, together with \\eqref{eq:convRateFun-s} and \\eqref{eq:convRateFun2-s}, implies \\eqref{eq:LinIneqNablaDeal-s} and \\eqref{eq:SubLinIneqNablaDeal-s}. To prove \\eqref{eq:LinIneqIteDealp-s}, using \n","sha256_utf8":"4fd4c8556be397b42fc81edac627cfdfde275be4a16bbfc192436c093b39c5bf"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"几何级数给出迭代线性率","source_units":[{"start":38922,"end":39489,"start_line":577,"end_line":583,"raw_tex":"    $\\Vert x^{k+1}-x^k\\Vert \\leq c\\Vert \\nabla f(x^k)\\Vert^{\\theta-1}$ and \\eqref{eq:LinIneqNablaDeal-s}, we obtain\n    \\begin{align*}\n        \\|x^k - \\ov x\\| &\\le \\sum_{i\\ge k} \\|x^i -x^{i+1}\\|\\le \\sum_{i\\ge k} c\\|\\nabla f(x^i)\\|^{\\theta-1}\n        \\le c\\Big( \\tfrac{f(x^0)-f(\\ov x)}{\\rho}\\Big)^{\\tfrac{\\theta-1}{\\theta}} \\sum_{i\\ge k}   q^{\\tfrac{i(\\theta -1)}{\\theta}}\n        = c\\Big( \\tfrac{f(x^0)-f(\\ov x)}{\\rho}\\Big)^{\\tfrac{\\theta-1}{\\theta}}  \\tfrac{q^{\\tfrac{k(\\theta-1)}{\\theta}}}{1-q^{\\tfrac{\\theta -1}{\\theta}}},\n    \\end{align*}\n    for all $k\\ge k_0$.\n","sha256_utf8":"4b56725592de0ffaf8ff8627f62e76666d3decbda5b1ba99e2d6f56ff505f439"}]},{"id":"S033","semantic_type":"proof_step","kind":"proof_step","title":"重用凹变换差分估计","source_units":[{"start":39489,"end":39917,"start_line":584,"end_line":590,"raw_tex":"Let us set $\\theta>\\tfrac{1}{\\vartheta}$. \nArguing as in the proof of Theorem~\\ref{thm:GlobConv-s}~\\ref{thm:GlobConv-s:4} (see in particular the derivation of\n\\eqref{eq:th:conKL:z1-s}), we obtain\n\\begin{equation}\\label{eq:thm:LinConvWeakDealFun-s:1:a}\n    \\Delta_{k}-\\Delta_{k+1} \\geq \\tau^{-1}\\rho (1-\\vartheta)\\Vert\\nabla f(x^{k})\\Vert^{\\theta-1},\n\\end{equation}\nwhere $\\Delta_k:=\\left(f(x^k)-f(\\ov x)\\right)^{1-\\vartheta}$. \n","sha256_utf8":"58a93bf73fac13148b1ff5a6d17642691f7be90b56a2ed3d075245c51ab023dc"}]},{"id":"S034","semantic_type":"proof_step","kind":"proof_step","title":"变换函数差的多项式界","source_units":[{"start":39917,"end":40124,"start_line":591,"end_line":595,"raw_tex":"Moreover, by \\eqref{eq:convRateFun2-s}, there exists a constant $\\mu>0$ such that, for all\nsufficiently large $k\\in\\N_{0}$,\n\\[\n\\Delta_k\\leq \\mu^{1-\\vartheta} k^{-\\tfrac{1-\\vartheta}{\\vartheta \\theta-1}}.\n\\]\n","sha256_utf8":"50cfb91a34f164307fe7b1777284571633a56b5fab1f678098b9d9b780fb1d25"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"尾和给出迭代次线性率","source_units":[{"start":40124,"end":40666,"start_line":596,"end_line":607,"raw_tex":"Together with \\eqref{eq:thm:LinConvWeakDealFun-s:1:a}, this estimate yields, for all sufficiently\nlarge $k\\in\\N_{0}$,\n    \\begin{align*}\n        \\|x^k - \\ov x\\| \\le \\sum_{i\\ge k} c\\|\\nabla f(x^i)\\|^{\\theta-1} &\\le \n        c  \\tau\\rho^{-1}(1-\\vartheta)^{-1}\\sum_{i\\ge k} (\\Delta_{i}-\\Delta_{i+1})\n        \\leq c  \\tau\\rho^{-1}(1-\\vartheta)^{-1}\\Delta_{k}\n        \\\\&\n        \\le c  \\tau\\rho^{-1}(1-\\vartheta)^{-1}\\mu^{1-\\vartheta} k^{-\\tfrac{1-\\vartheta}{\\vartheta \\theta-1}},\n    \\end{align*}\nwhich completes the proof.\n     \\qed\n\\end{proof}","sha256_utf8":"025f30d1dcffe9798867270c6eecfb48b435e6bb6453b3a354131fd7971a42df"}]},{"id":"S036","semantic_type":"assumption","kind":"assumption","title":"正序列递推假设","source_units":[{"start":22425,"end":22699,"start_line":353,"end_line":356,"raw_tex":"    Let $\\seq{s^k}$ be a sequence in $(0, +\\infty)$ converging to $0$, and let $\\varsigma\\ge 1$ and $\\omega>0$. Suppose that for some $k_0\\in \\N_0$,\n    \\begin{equation}\\label{eq:sk}\n        (s^k)^{\\varsigma}\\leq\\omega(s^k-s^{k+1}),\\quad\\forall k\\ge k_0.\n    \\end{equation}\n","sha256_utf8":"079fa1033ce1f578f3527d9609442c3bd38a2649e5f1148f83dd861b07b8b00c"}]},{"id":"S037","semantic_type":"external_fact","kind":"external_fact","title":"光滑次微分恒等式","source_units":[],"declared_external":true,"declaration_text":["If h:R^n→R is continuously differentiable, then its regular and limiting subdifferentials at x are both {∇h(x)}. Consequently dist(0;∂h(x))=||∇h(x)||, as cited to Mordukhovich2018, page 20."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S001","edges":[{"source_id":"S002","target_id":"S001","kind":"supports"},{"source_id":"S004","target_id":"S003","kind":"supports"},{"source_id":"S007","target_id":"S005","kind":"supports"},{"source_id":"S008","target_id":"S016","kind":"supports"},{"source_id":"S009","target_id":"S021","kind":"supports"},{"source_id":"S009","target_id":"S022","kind":"supports"},{"source_id":"S010","target_id":"S012","kind":"supports"},{"source_id":"S010","target_id":"S013","kind":"supports"},{"source_id":"S010","target_id":"S014","kind":"supports"},{"source_id":"S010","target_id":"S016","kind":"supports"},{"source_id":"S011","target_id":"S013","kind":"supports"},{"source_id":"S011","target_id":"S014","kind":"supports"},{"source_id":"S012","target_id":"S022","kind":"supports"},{"source_id":"S013","target_id":"S014","kind":"supports"},{"source_id":"S013","target_id":"S015","kind":"supports"},{"source_id":"S037","target_id":"S015","kind":"supports"},{"source_id":"S014","target_id":"S023","kind":"supports"},{"source_id":"S014","target_id":"S024","kind":"supports"},{"source_id":"S014","target_id":"S027","kind":"supports"},{"source_id":"S015","target_id":"S024","kind":"supports"},{"source_id":"S015","target_id":"S027","kind":"supports"},{"source_id":"S016","target_id":"S021","kind":"supports"},{"source_id":"S016","target_id":"S024","kind":"supports"},{"source_id":"S016","target_id":"S028","kind":"supports"},{"source_id":"S016","target_id":"S031","kind":"supports"},{"source_id":"S017","target_id":"S022","kind":"supports"},{"source_id":"S017","target_id":"S023","kind":"supports"},{"source_id":"S018","target_id":"S027","kind":"supports"},{"source_id":"S018","target_id":"S032","kind":"supports"},{"source_id":"S018","target_id":"S035","kind":"supports"},{"source_id":"S019","target_id":"S023","kind":"supports"},{"source_id":"S019","target_id":"S025","kind":"supports"},{"source_id":"S036","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S006","kind":"supports"},{"source_id":"S006","target_id":"S030","kind":"supports"},{"source_id":"S005","target_id":"S030","kind":"supports"},{"source_id":"S021","target_id":"S022","kind":"supports"},{"source_id":"S021","target_id":"S004","kind":"supports"},{"source_id":"S022","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S004","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S004","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S033","kind":"supports"},{"source_id":"S024","target_id":"S004","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S025","target_id":"S004","kind":"supports"},{"source_id":"S026","target_id":"S004","kind":"supports"},{"source_id":"S003","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S028","kind":"supports"},{"source_id":"S027","target_id":"S029","kind":"supports"},{"source_id":"S027","target_id":"S032","kind":"supports"},{"source_id":"S027","target_id":"S035","kind":"supports"},{"source_id":"S027","target_id":"S002","kind":"supports"},{"source_id":"S028","target_id":"S029","kind":"supports"},{"source_id":"S028","target_id":"S030","kind":"supports"},{"source_id":"S028","target_id":"S002","kind":"supports"},{"source_id":"S029","target_id":"S002","kind":"supports"},{"source_id":"S030","target_id":"S031","kind":"supports"},{"source_id":"S030","target_id":"S034","kind":"supports"},{"source_id":"S030","target_id":"S002","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S002","kind":"supports"},{"source_id":"S032","target_id":"S002","kind":"supports"},{"source_id":"S033","target_id":"S035","kind":"supports"},{"source_id":"S033","target_id":"S002","kind":"supports"},{"source_id":"S034","target_id":"S035","kind":"supports"},{"source_id":"S034","target_id":"S002","kind":"supports"},{"source_id":"S035","target_id":"S002","kind":"supports"}]}}
{"record_id":"2608.05653v1","paper":{"title":"Fast Gradient Algorithm with Dry-like Friction and Nonmonotone Line Search for Nonconvex Optimization Problems","authors":["Lien T. Nguyen","Andrew Eberhard","Xinghuo Yu","Chaojie Li"],"arxiv_id":"2608.05653","version":"v1","journal":"SIAM Journal on Optimization","doi":"10.1137/22m1532354","abstract_url":"https://arxiv.org/abs/2608.05653v1","pdf_url":"https://arxiv.org/pdf/2608.05653v1"},"context":{"raw_tex":"Throughout this paper, $\\mathcal{H}$ is a real Hilbert space equipped with the scalar product $\\langle \\cdot, \\cdot\\rangle$ and the associated norm $\\|\\cdot\\|$. We consider the nonconvex optimization problem\n\\begin{equation}\\label{eq:pb}\n\\min_{x \\in \\mathcal{H}} f(x),\n\\end{equation}\n\nWe denote the set of nonnegative integers by $\\mathbb{N}$, the set of real numbers by $\\mathbb{R}$, the set\nof nonnegative real numbers by $\\mathbb{R}_+$, and the set of the positive real numbers by $\\mathbb{R}_{++}$. For $x\\in \\mathcal{H}$ and $\\delta \\in \\mathbb{R}_+$, the closed ball in $\\mathcal{H}$ centered at $x$ with radius $\\delta$ is $B(x; \\delta) = \\{y \\in \\mathcal{H}: \\|x - y\\| \\leq \\delta\\}$. For a subset $C$ of $\\mathcal{H}$, its interior is $\\inte C =\\{x\\in C: \\exists \\delta \\in \\mathbb{R}_{++},\\ B(x; \\delta) \\subseteq C\\}$.\n\nGiven $f\\colon \\mathcal{H} \\to (-\\infty, +\\infty]$, its \\emph{domain} is denoted by $\\dom f =\\{x \\in \\mathcal{H} \\colon f(x)  < +\\infty\\}$ and its \\emph{epigraph} by $\\epi f =\\{(x,\\xi) \\colon f(x) \\leq \\xi\\}$. We say that $f$ is \\emph{proper} if $\\dom f \\neq \\varnothing$, \\emph{lower semicontinuous} if $\\epi f$ is a closed set, and \\emph{convex} if $\\epi f$ is a convex set. The function $f$ is said to be \\emph{$\\alpha$-weakly convex} ($\\alpha \\in \\mathbb{R}_+$) if $f + \\frac{\\alpha}{2}\\| \\cdot\\|^2$ is convex.\n\nLet $f\\colon \\mathcal{H} \\to (-\\infty, +\\infty]$ be proper and let $x \\in \\dom f$. The \\emph{Fr\\'echet subdifferential} of $f$ at $x$ is defined as\n\\begin{equation*}\n\\hat{\\partial} f(x) = \\left \\{u \\in \\mathcal{H} \\colon \\liminf_{y \\to x}\\frac{f(y) -f(x) -\\langle u, y-x \\rangle}{\\|y-x\\|} \\geq 0\\right \\}\n\\end{equation*}\nand the \\emph{limiting subdifferential} of $f$ at $x$ is defined as\n\\begin{equation*}\n\\partial f(x) = \\{u \\in \\mathcal{H} \\colon \\exists x_n \\to x, f(x_n) \\to f(x) \\text{~and~} \\exists u_n \\in \\hat{\\partial} f(x_n), u_n \\to u \\text{~as~} n \\to +\\infty\\}.\n\\end{equation*}\nFor $x \\notin  \\dom f$, one takes $\\hat{\\partial} f(x)=\\varnothing$ and $\\partial f(x) = \\varnothing$. The domain of $\\partial f$ is $\\dom \\partial f = \\{ x \\in \\mathcal{H}: \\partial f(x) \\neq \\varnothing\\}$. In the case when $f$ is convex, both Fr\\'echet and limiting subdifferentials coincide with the \\emph{convex subdifferential}, that is,\n $\\partial f(x) =\\{u \\in \\mathcal{H}: \\forall y \\in \\mathcal{H}, \\langle u, y-x\\rangle \\leq f(y) -f(x)\\}.$\nLet $\\xi \\in \\R_{++}$. The \\emph{Moreau envelope} of $f$ is the mapping $f_{\\xi} \\colon \\mathcal{H} \\to \\left(-\\infty, +\\infty\\right]$ given by\n\\begin{equation*}\nf_{\\xi}(x) = \\inf_{y \\in \\mathcal{H}} \\left(f(y) + \\frac{1}{2\\xi}\\|x-y\\|^2\\right)\n\\end{equation*} \nand the \\emph{proximal operator} of $f$ is the set-valued mapping $\\prox_{\\xi f} \\colon  \\mathcal{H} \\rightrightarrows \\mathcal{H}$ given by\n\\begin{equation*}\n\\prox_{\\xi f}(x) =\\argmin_{y \\in \\mathcal{H}}  \\left(f(y) + \\frac{1}{2\\xi}\\|x-y\\|^2\\right).\n\\end{equation*}\nIn a common abuse of notation, we consider $\\prox_{\\xi f}(x)$ as a point if it is a singleton. \n\n\n\\begin{definition}[Dry-like friction property]\n\\label{d:DLF}\nWe say that a proper function $\\phi\\colon \\mathcal{H} \\to \\left(-\\infty, +\\infty\\right]$ satisfies the \\emph{dry-like friction property with constant $r \\in \\mathbb{R}_{++}$} if for all $x \\in \\dom \\partial \\phi$, $\\inf_{u \\in \\partial \\phi(x)} \\langle u, x\\rangle \\geq r \\|x\\|$. The function $\\phi$ is said to satisfy the \\emph{dry-like friction property} or called a \\emph{dry-like friction function} if it satisfies dry-like friction property with some constant $r$.\n\\end{definition}\n\nIn this section, we assume that $f: \\mathcal{H} \\to \\R$ is a differentiable function whose gradient is $L$-Lipschitz continuous. Inspired by the works using dry friction in \\cite{AA20,AA21, AA22, AAL21, ABC21} and the works using nonmonotone line search in \\cite{Dai02,  GZLHY13,GLL86, WNF09,  Yan21, ZH04} to enlarge the choices of the parameters, we propose the AGDNL for solving problem \\eqref{eq:pb} with the potential function $V \\colon \\mathcal{H} \\times \\mathcal{H} \\times \\mathcal{H} \\times \\R_+ \\times \\R_+ \\to \\R $ defined by\n\\begin{align}\\label{eq:V}\nV(x,y,w,\\rho,\\sigma)= f(x) +\\rho \\|x -y\\|^2 +\\sigma \\|w\\|^2.\n\\end{align}\n\n\\begin{algo}[AGDNL]\n\\label{algo:AGDNL}\n\\step{}\\label{step:init}\nLet $\\phi\\colon \\mathcal{H}\\to \\left(-\\infty, +\\infty\\right]$ satisfy the dry-like friction property with constant $r$. \nLet $u_{-1}=x_{-1} =x_0 \\in \\mathcal{H}$. \nLet $\\mu \\in \\R_{++}$, $\\bar\\alpha =\\alpha_{-1} \\in (0, \\frac{1}{\\mu})$,  $\\bar\\beta =\\beta_{-1} \\in \\R_{+}$, and $\\bar\\gamma =\\gamma_{-1} \\in \\R_{+}$. \nLet $L_{\\min} \\in (0, L]$, and $L_{\\max} \\in [L, +\\infty)$. Let $\\lambda_{\\min} \\in [0,\\frac{1}{L_{\\max}})$, $\\lambda_{\\max} \\in [\\lambda_{\\min}, +\\infty)$, $\\kappa_{\\min} \\in (0, \\bar\\kappa]$, where $\\bar\\kappa = \\frac{2(1-\\mu \\bar\\alpha)}{\\mu L_{\\max}} -\\frac{2(1-\\mu \\bar\\alpha)\\lambda_{\\min}}{\\mu}$, and $\\kappa_{\\max} \\in [\\kappa_{\\min}, +\\infty)$. Set \n\\begin{equation*}\n\\bar c =\\frac{2(1-\\mu \\bar \\alpha) - \\mu^2 L_{\\max} \\underline{\\nu}}{2 \\mu^2 \\underline{\\nu}} \n\\text{~~and~~} \\bar d =\\frac{(1-\\mu \\bar\\alpha)\\lambda_{\\min}^2 + \\lambda_{\\min} \\mu \\kappa_{\\min}}{\\mu^2 \\underline{\\nu}},    \n\\end{equation*}\nwhere $\\underline{\\nu}=\\frac{2 \\lambda_{\\min}(1 -\\mu \\bar\\alpha)}{\\mu^2}+ \\frac{\\kappa_{\\min}}{\\mu}$. Let $c \\in [0, \\bar c ]$ and $ d \\in [0, \\bar d]$ such that $d >0$ if $\\lambda_{\\min} >0$ and $c=0$. Let $L_{-1} \\in [L_{\\min}, L_{\\max}]$, $\\lambda_{-1} \\in [\\lambda_{\\min}, \\lambda_{\\max}]$, $\\kappa_{-1} \\in [\\kappa_{\\min}, \\kappa_{\\max}]$, and set $\\rho_{-1} = \\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\gamma_{-1}}{\\nu_{-1}} + \\frac{L_{-1}\\beta_{-1} +\\gamma_{-1}}{2}$ and $\\sigma_{-1} =\\frac{\\mu \\alpha_{-1}}{2\\nu_{-1}}$, where $\\nu_{-1} = \\frac{2\\lambda_{-1}(1 -\\mu \\bar\\alpha)}{\\mu^2} +\\frac{\\kappa_{-1}}{\\mu}$. \nSet also $w_{-1} =\\frac{\\lambda_{-1} \\nabla f(u_{-1})}{\\mu}$. Let $\\delta \\in \\R_{++}$ and $\\ell_1, \\ell_2, \\ell_3 \\in (0,1)$. \nLet $N \\in \\N$ and set $n=0$.\n%$d \\in (0,\\frac{(1- \\mu \\alpha)\\lambda_{\\min}^2}{\\mu^2 \\ell L_{\\max}} )$ . \n\n\\step{}\\label{step:main}\nLet $e_n \\in \\mathcal{H}$, $\\theta_n \\in \\R$, and $\\tau_n\\in \\mathbb{R}_+$ such that $r\\tau_n \\geq |\\theta_n| +\\|e_n\\| +\\delta$. \nChoose $\\tilde{L}_n \\in [L_{\\min}, L_{\\max}] $, $\\tilde{\\alpha}_n \\in (0, \\bar\\alpha]$, and $\\tilde{\\kappa}_n \\in [\\kappa_{\\min}, \\kappa_{\\max}]$. Choose $\\tilde{\\beta}_n \\in [0, \\bar\\beta]$ such that $\\tilde{\\beta}_n  = 0$ if $c = \\bar c$. %$\\kappa_{\\min} =\\bar \\kappa$. \nChoose $\\tilde{\\gamma}_n \\in [0, \\bar\\gamma]$ such that $\\tilde{\\gamma}_n =0$ if $d=\\bar d$ or $c=\\bar c$ or $c=0$.\nChoose $\\tilde{\\lambda}_n \\in [\\lambda_{\\min}, \\lambda_{\\max}]$ such that $\\tilde{\\lambda}_n  =0$ if $\\lambda_{\\min} =0$. Set $\\beta_n =\\tilde{\\beta}_n$, $\\gamma_n =\\tilde{\\gamma}_n$, $L_n =\\tilde{L}_n$, $\\alpha_n =\\tilde{\\alpha}_n$, $\\lambda_n =\\tilde{\\lambda}_n$, and $\\kappa_n = \\tilde{\\kappa}_n$.\n\\begin{enumerate}[leftmargin=2em]\n\\item\\label{step:main_comp}\nCompute \n\\begin{equation*}\n\\small\n\\begin{cases}\nu_n = x_n + \\beta_n(x_n - x_{n-1}),\\\\ \n%z_n &= x_n + \\nu_n(x_n - x_{n-1}),\\\\\nv_n = x_n + \\gamma_n (x_n -x_{n-1}),\\\\\nw_n \\in \\prox_{\\tau_n \\phi}\\left(\\alpha_n(x_n -v_{n-1}+\\lambda_{n-1}\\nabla f(u_{n-1}))-\\theta_n \\frac{\\nabla f(u_n)}{\\|\\nabla f(u_n)\\|} -\\kappa_n \\nabla f(u_n)+ e_n\\right ),\\\\\nz_{n+1} = v_n- \\lambda_n\\nabla f(u_n) +\\mu w_n.\n\\end{cases}\n\\end{equation*}\n\\item \n\\label{step:main_line}\nSet $\\rho_n = \\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\gamma_n}{\\nu_n} + \\frac{ L_n\\beta_n +\\gamma_n}{2 }$ and $\\sigma_n =\\frac{\\mu \\alpha_n}{2 \\nu_n}$, where $\\nu_n = \\frac{2 \\lambda_n(1 -\\mu \\bar\\alpha)}{\\mu^2}+ \\frac{\\kappa_n }{\\mu}$. If\n\\begin{align}\\label{eq:linesearch}\nV(z_{n+1}, x_n, w_n,\\rho_n, \\sigma_n) +c \\|z_{n+1}-x_n\\|^2 + d \\|\\nabla f(u_n)\\|^2 +\\frac{\\delta}{\\underline{\\nu}}\\| w_n\\|\\notag\\\\\n\\leq \\max_{[n-N]_+ \\leq k \\leq n}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1}),\n\\end{align}\nthen go to Step~\\ref{step:last}.\n\\item\\label{step:main_mult}\nSet $L_n \\leftarrow \\min\\{L_n/\\ell_1, L_{\\max}\\}$, $ \\beta_n \\leftarrow \\ell_{2}\\beta_n $,  $ \\gamma_n \\leftarrow \\ell_{2}\\gamma_n $, $\\alpha_n \\leftarrow \\ell_{2}\\alpha_n $, $\\lambda_n \\leftarrow \\max\\{\\ell_3 \\lambda_n, \\lambda_{\\min}\\}$, $\\kappa_n \\leftarrow \\max\\{\\ell_3 \\kappa_n, \\kappa_{\\min}\\}$, and go to Step~\\ref{step:main}\\ref{step:main_comp}. \n\\end{enumerate}\n\n\\step{}\\label{step:last}\nSet $x_{n+1} \\leftarrow z_{n+1}$. If a termination criterion is not met, then let $n\\leftarrow n+1$ and go to Step~\\ref{step:main}.\n\\end{algo}\n\n\\begin{equation}\\label{AGNL}\n\\begin{cases}\nu_n &= x_n + \\beta_n(x_n - x_{n-1}),\\\\ \nv_n & = x_n + \\gamma_n (x_n -x_{n-1}),\\\\\nz_{n+1}\\!\\!\\! &= v_n- \\lambda_n\\nabla f(u_n)\n\\end{cases}\n\\end{equation}\n\n\\begin{align}\\label{AGNL_ls}\nV(z_{n+1}, x_n, 0,\\rho_n, \\sigma_n) +c \\|z_{n+1}-x_n\\|^2 + d \\|\\nabla f(u_n)\\|^2\n\\notag\\\\\n\\leq \\max_{[n-N]_+ \\leq k \\leq n}V(x_k, x_{k-1}, 0,\\rho_{k-1}, \\sigma_{k-1}).    \n\\end{align}\n\nWe call this version as the \\emph{fast gradient algorithm with nonmonotone line search} (AGNL).\n\n\nFor each $n \\in \\N$, set $m(n)\\in \\argmax_{[n-N]_+ \\leq k \\leq n}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1})$. The following technical lemma will be of use in our analysis.\n\n\nWe now recall the Kurdyka--{\\L}ojasiewicz (KL) property \\cite{Kur98, Loj63} which plays an important role in our convergence analysis. Let $h: \\mathcal{H} \\to (-\\infty, +\\infty]$ be a proper lower semicontinuous function. We say that $h$ satisfies the \\emph{KL property} at $\\bar x \\in \\dom \\partial h$ if there exist $\\varepsilon \\in \\mathbb{R}_{++}$, $\\nu \\in (0, +\\infty]$, and a continuous concave function $\\varphi: [0, \\nu) \\to \\R_+$ such that $\\varphi(0) =0$ and $\\varphi$ is continuously differentiable on  $(0, \\nu)$ with $\\varphi' >0$, and for all $x \\in B(\\bar x; \\varepsilon)$ with $h(\\bar x) < h(x) < h (\\bar x) + \\nu$, one has\n\\begin{equation*}\n\\varphi'(h(x)- h(\\bar x))\\dist(0, \\partial h(x)) \\geq 1.\n\\end{equation*}\nIf $h$ satisfies the KL property at each point in $\\dom \\partial h$, then $h$ is called a \\emph{KL function}.\n\nThe function $h$ is said to satisfy the \\emph{KL property at $\\bar x$ with exponent $\\theta$} if it satisfies the KL property at $\\bar x \\in \\dom \\partial h$ with corresponding function $\\varphi(s) = \\gamma s^{1-\\theta}$ for some $\\gamma \\in \\R_{++}$ and $\\theta \\in [0,1)$, i.e., there exist $c \\in \\mathbb{R}_{++}$, $\\varepsilon \\in \\mathbb{R}_{++}$, and $\\nu \\in (0, +\\infty]$ such that\n\\begin{equation*}\n\\dist(0, \\partial h(x)) \\geq c (h(x) -h(\\bar x))^{\\theta}\n\\end{equation*}\nwhenever $x \\in B(\\bar x; \\varepsilon)$ with $h(\\bar x) < h(x) < h(\\bar x) + \\nu$. If $h$ is a KL function and has the same exponent $\\theta$ at any $\\bar x \\in \\dom \\partial h$, then $h$ is called a \\emph{KL function with exponent $\\theta$}. Some typical KL functions include strongly convex functions, real analytic functions, and semialgebraic functions, see \\cite{ABRS10, Loj63}.\n","units":[{"start":6709,"end":6992,"start_line":188,"end_line":191,"raw_tex":"Throughout this paper, $\\mathcal{H}$ is a real Hilbert space equipped with the scalar product $\\langle \\cdot, \\cdot\\rangle$ and the associated norm $\\|\\cdot\\|$. We consider the nonconvex optimization problem\n\\begin{equation}\\label{eq:pb}\n\\min_{x \\in \\mathcal{H}} f(x),\n\\end{equation}","sha256_utf8":"cc091db26d2b71e7fecce9930c31b75c428b3f834d6067b0db968a81802f25c8"},{"start":16127,"end":18856,"start_line":261,"end_line":284,"raw_tex":"We denote the set of nonnegative integers by $\\mathbb{N}$, the set of real numbers by $\\mathbb{R}$, the set\nof nonnegative real numbers by $\\mathbb{R}_+$, and the set of the positive real numbers by $\\mathbb{R}_{++}$. For $x\\in \\mathcal{H}$ and $\\delta \\in \\mathbb{R}_+$, the closed ball in $\\mathcal{H}$ centered at $x$ with radius $\\delta$ is $B(x; \\delta) = \\{y \\in \\mathcal{H}: \\|x - y\\| \\leq \\delta\\}$. For a subset $C$ of $\\mathcal{H}$, its interior is $\\inte C =\\{x\\in C: \\exists \\delta \\in \\mathbb{R}_{++},\\ B(x; \\delta) \\subseteq C\\}$.\n\nGiven $f\\colon \\mathcal{H} \\to (-\\infty, +\\infty]$, its \\emph{domain} is denoted by $\\dom f =\\{x \\in \\mathcal{H} \\colon f(x)  < +\\infty\\}$ and its \\emph{epigraph} by $\\epi f =\\{(x,\\xi) \\colon f(x) \\leq \\xi\\}$. We say that $f$ is \\emph{proper} if $\\dom f \\neq \\varnothing$, \\emph{lower semicontinuous} if $\\epi f$ is a closed set, and \\emph{convex} if $\\epi f$ is a convex set. The function $f$ is said to be \\emph{$\\alpha$-weakly convex} ($\\alpha \\in \\mathbb{R}_+$) if $f + \\frac{\\alpha}{2}\\| \\cdot\\|^2$ is convex.\n\nLet $f\\colon \\mathcal{H} \\to (-\\infty, +\\infty]$ be proper and let $x \\in \\dom f$. The \\emph{Fr\\'echet subdifferential} of $f$ at $x$ is defined as\n\\begin{equation*}\n\\hat{\\partial} f(x) = \\left \\{u \\in \\mathcal{H} \\colon \\liminf_{y \\to x}\\frac{f(y) -f(x) -\\langle u, y-x \\rangle}{\\|y-x\\|} \\geq 0\\right \\}\n\\end{equation*}\nand the \\emph{limiting subdifferential} of $f$ at $x$ is defined as\n\\begin{equation*}\n\\partial f(x) = \\{u \\in \\mathcal{H} \\colon \\exists x_n \\to x, f(x_n) \\to f(x) \\text{~and~} \\exists u_n \\in \\hat{\\partial} f(x_n), u_n \\to u \\text{~as~} n \\to +\\infty\\}.\n\\end{equation*}\nFor $x \\notin  \\dom f$, one takes $\\hat{\\partial} f(x)=\\varnothing$ and $\\partial f(x) = \\varnothing$. The domain of $\\partial f$ is $\\dom \\partial f = \\{ x \\in \\mathcal{H}: \\partial f(x) \\neq \\varnothing\\}$. In the case when $f$ is convex, both Fr\\'echet and limiting subdifferentials coincide with the \\emph{convex subdifferential}, that is,\n $\\partial f(x) =\\{u \\in \\mathcal{H}: \\forall y \\in \\mathcal{H}, \\langle u, y-x\\rangle \\leq f(y) -f(x)\\}.$\nLet $\\xi \\in \\R_{++}$. The \\emph{Moreau envelope} of $f$ is the mapping $f_{\\xi} \\colon \\mathcal{H} \\to \\left(-\\infty, +\\infty\\right]$ given by\n\\begin{equation*}\nf_{\\xi}(x) = \\inf_{y \\in \\mathcal{H}} \\left(f(y) + \\frac{1}{2\\xi}\\|x-y\\|^2\\right)\n\\end{equation*} \nand the \\emph{proximal operator} of $f$ is the set-valued mapping $\\prox_{\\xi f} \\colon  \\mathcal{H} \\rightrightarrows \\mathcal{H}$ given by\n\\begin{equation*}\n\\prox_{\\xi f}(x) =\\argmin_{y \\in \\mathcal{H}}  \\left(f(y) + \\frac{1}{2\\xi}\\|x-y\\|^2\\right).\n\\end{equation*}\nIn a common abuse of notation, we consider $\\prox_{\\xi f}(x)$ as a point if it is a singleton. \n","sha256_utf8":"f16c9fe847229596d05ada0187e6037aa0e10d4bae6141db374dd54b519652c0"},{"start":19294,"end":19841,"start_line":291,"end_line":294,"raw_tex":"\\begin{definition}[Dry-like friction property]\n\\label{d:DLF}\nWe say that a proper function $\\phi\\colon \\mathcal{H} \\to \\left(-\\infty, +\\infty\\right]$ satisfies the \\emph{dry-like friction property with constant $r \\in \\mathbb{R}_{++}$} if for all $x \\in \\dom \\partial \\phi$, $\\inf_{u \\in \\partial \\phi(x)} \\langle u, x\\rangle \\geq r \\|x\\|$. The function $\\phi$ is said to satisfy the \\emph{dry-like friction property} or called a \\emph{dry-like friction function} if it satisfies dry-like friction property with some constant $r$.\n\\end{definition}","sha256_utf8":"67b9ac53a347e240c5c1f633db671a858f11a269526c5366cb8302fd548aaa65"},{"start":24218,"end":24852,"start_line":363,"end_line":366,"raw_tex":"In this section, we assume that $f: \\mathcal{H} \\to \\R$ is a differentiable function whose gradient is $L$-Lipschitz continuous. Inspired by the works using dry friction in \\cite{AA20,AA21, AA22, AAL21, ABC21} and the works using nonmonotone line search in \\cite{Dai02,  GZLHY13,GLL86, WNF09,  Yan21, ZH04} to enlarge the choices of the parameters, we propose the AGDNL for solving problem \\eqref{eq:pb} with the potential function $V \\colon \\mathcal{H} \\times \\mathcal{H} \\times \\mathcal{H} \\times \\R_+ \\times \\R_+ \\to \\R $ defined by\n\\begin{align}\\label{eq:V}\nV(x,y,w,\\rho,\\sigma)= f(x) +\\rho \\|x -y\\|^2 +\\sigma \\|w\\|^2.\n\\end{align}","sha256_utf8":"6cdd3515bf90f870387068cd66754eed3048a465af5e402c74bb49c8fc81895f"},{"start":25323,"end":29631,"start_line":373,"end_line":421,"raw_tex":"\\begin{algo}[AGDNL]\n\\label{algo:AGDNL}\n\\step{}\\label{step:init}\nLet $\\phi\\colon \\mathcal{H}\\to \\left(-\\infty, +\\infty\\right]$ satisfy the dry-like friction property with constant $r$. \nLet $u_{-1}=x_{-1} =x_0 \\in \\mathcal{H}$. \nLet $\\mu \\in \\R_{++}$, $\\bar\\alpha =\\alpha_{-1} \\in (0, \\frac{1}{\\mu})$,  $\\bar\\beta =\\beta_{-1} \\in \\R_{+}$, and $\\bar\\gamma =\\gamma_{-1} \\in \\R_{+}$. \nLet $L_{\\min} \\in (0, L]$, and $L_{\\max} \\in [L, +\\infty)$. Let $\\lambda_{\\min} \\in [0,\\frac{1}{L_{\\max}})$, $\\lambda_{\\max} \\in [\\lambda_{\\min}, +\\infty)$, $\\kappa_{\\min} \\in (0, \\bar\\kappa]$, where $\\bar\\kappa = \\frac{2(1-\\mu \\bar\\alpha)}{\\mu L_{\\max}} -\\frac{2(1-\\mu \\bar\\alpha)\\lambda_{\\min}}{\\mu}$, and $\\kappa_{\\max} \\in [\\kappa_{\\min}, +\\infty)$. Set \n\\begin{equation*}\n\\bar c =\\frac{2(1-\\mu \\bar \\alpha) - \\mu^2 L_{\\max} \\underline{\\nu}}{2 \\mu^2 \\underline{\\nu}} \n\\text{~~and~~} \\bar d =\\frac{(1-\\mu \\bar\\alpha)\\lambda_{\\min}^2 + \\lambda_{\\min} \\mu \\kappa_{\\min}}{\\mu^2 \\underline{\\nu}},    \n\\end{equation*}\nwhere $\\underline{\\nu}=\\frac{2 \\lambda_{\\min}(1 -\\mu \\bar\\alpha)}{\\mu^2}+ \\frac{\\kappa_{\\min}}{\\mu}$. Let $c \\in [0, \\bar c ]$ and $ d \\in [0, \\bar d]$ such that $d >0$ if $\\lambda_{\\min} >0$ and $c=0$. Let $L_{-1} \\in [L_{\\min}, L_{\\max}]$, $\\lambda_{-1} \\in [\\lambda_{\\min}, \\lambda_{\\max}]$, $\\kappa_{-1} \\in [\\kappa_{\\min}, \\kappa_{\\max}]$, and set $\\rho_{-1} = \\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\gamma_{-1}}{\\nu_{-1}} + \\frac{L_{-1}\\beta_{-1} +\\gamma_{-1}}{2}$ and $\\sigma_{-1} =\\frac{\\mu \\alpha_{-1}}{2\\nu_{-1}}$, where $\\nu_{-1} = \\frac{2\\lambda_{-1}(1 -\\mu \\bar\\alpha)}{\\mu^2} +\\frac{\\kappa_{-1}}{\\mu}$. \nSet also $w_{-1} =\\frac{\\lambda_{-1} \\nabla f(u_{-1})}{\\mu}$. Let $\\delta \\in \\R_{++}$ and $\\ell_1, \\ell_2, \\ell_3 \\in (0,1)$. \nLet $N \\in \\N$ and set $n=0$.\n%$d \\in (0,\\frac{(1- \\mu \\alpha)\\lambda_{\\min}^2}{\\mu^2 \\ell L_{\\max}} )$ . \n\n\\step{}\\label{step:main}\nLet $e_n \\in \\mathcal{H}$, $\\theta_n \\in \\R$, and $\\tau_n\\in \\mathbb{R}_+$ such that $r\\tau_n \\geq |\\theta_n| +\\|e_n\\| +\\delta$. \nChoose $\\tilde{L}_n \\in [L_{\\min}, L_{\\max}] $, $\\tilde{\\alpha}_n \\in (0, \\bar\\alpha]$, and $\\tilde{\\kappa}_n \\in [\\kappa_{\\min}, \\kappa_{\\max}]$. Choose $\\tilde{\\beta}_n \\in [0, \\bar\\beta]$ such that $\\tilde{\\beta}_n  = 0$ if $c = \\bar c$. %$\\kappa_{\\min} =\\bar \\kappa$. \nChoose $\\tilde{\\gamma}_n \\in [0, \\bar\\gamma]$ such that $\\tilde{\\gamma}_n =0$ if $d=\\bar d$ or $c=\\bar c$ or $c=0$.\nChoose $\\tilde{\\lambda}_n \\in [\\lambda_{\\min}, \\lambda_{\\max}]$ such that $\\tilde{\\lambda}_n  =0$ if $\\lambda_{\\min} =0$. Set $\\beta_n =\\tilde{\\beta}_n$, $\\gamma_n =\\tilde{\\gamma}_n$, $L_n =\\tilde{L}_n$, $\\alpha_n =\\tilde{\\alpha}_n$, $\\lambda_n =\\tilde{\\lambda}_n$, and $\\kappa_n = \\tilde{\\kappa}_n$.\n\\begin{enumerate}[leftmargin=2em]\n\\item\\label{step:main_comp}\nCompute \n\\begin{equation*}\n\\small\n\\begin{cases}\nu_n = x_n + \\beta_n(x_n - x_{n-1}),\\\\ \n%z_n &= x_n + \\nu_n(x_n - x_{n-1}),\\\\\nv_n = x_n + \\gamma_n (x_n -x_{n-1}),\\\\\nw_n \\in \\prox_{\\tau_n \\phi}\\left(\\alpha_n(x_n -v_{n-1}+\\lambda_{n-1}\\nabla f(u_{n-1}))-\\theta_n \\frac{\\nabla f(u_n)}{\\|\\nabla f(u_n)\\|} -\\kappa_n \\nabla f(u_n)+ e_n\\right ),\\\\\nz_{n+1} = v_n- \\lambda_n\\nabla f(u_n) +\\mu w_n.\n\\end{cases}\n\\end{equation*}\n\\item \n\\label{step:main_line}\nSet $\\rho_n = \\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\gamma_n}{\\nu_n} + \\frac{ L_n\\beta_n +\\gamma_n}{2 }$ and $\\sigma_n =\\frac{\\mu \\alpha_n}{2 \\nu_n}$, where $\\nu_n = \\frac{2 \\lambda_n(1 -\\mu \\bar\\alpha)}{\\mu^2}+ \\frac{\\kappa_n }{\\mu}$. If\n\\begin{align}\\label{eq:linesearch}\nV(z_{n+1}, x_n, w_n,\\rho_n, \\sigma_n) +c \\|z_{n+1}-x_n\\|^2 + d \\|\\nabla f(u_n)\\|^2 +\\frac{\\delta}{\\underline{\\nu}}\\| w_n\\|\\notag\\\\\n\\leq \\max_{[n-N]_+ \\leq k \\leq n}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1}),\n\\end{align}\nthen go to Step~\\ref{step:last}.\n\\item\\label{step:main_mult}\nSet $L_n \\leftarrow \\min\\{L_n/\\ell_1, L_{\\max}\\}$, $ \\beta_n \\leftarrow \\ell_{2}\\beta_n $,  $ \\gamma_n \\leftarrow \\ell_{2}\\gamma_n $, $\\alpha_n \\leftarrow \\ell_{2}\\alpha_n $, $\\lambda_n \\leftarrow \\max\\{\\ell_3 \\lambda_n, \\lambda_{\\min}\\}$, $\\kappa_n \\leftarrow \\max\\{\\ell_3 \\kappa_n, \\kappa_{\\min}\\}$, and go to Step~\\ref{step:main}\\ref{step:main_comp}. \n\\end{enumerate}\n\n\\step{}\\label{step:last}\nSet $x_{n+1} \\leftarrow z_{n+1}$. If a termination criterion is not met, then let $n\\leftarrow n+1$ and go to Step~\\ref{step:main}.\n\\end{algo}","sha256_utf8":"a8cabf2652dba5f8774e30d572a9838216a595ad46069d6c84c5074190867a82"},{"start":44602,"end":44797,"start_line":592,"end_line":598,"raw_tex":"\\begin{equation}\\label{AGNL}\n\\begin{cases}\nu_n &= x_n + \\beta_n(x_n - x_{n-1}),\\\\ \nv_n & = x_n + \\gamma_n (x_n -x_{n-1}),\\\\\nz_{n+1}\\!\\!\\! &= v_n- \\lambda_n\\nabla f(u_n)\n\\end{cases}\n\\end{equation}","sha256_utf8":"34bf2da9693fce588956e204b199236c24edef0cd3cd77398534afacb5d0b8e6"},{"start":44836,"end":45048,"start_line":600,"end_line":604,"raw_tex":"\\begin{align}\\label{AGNL_ls}\nV(z_{n+1}, x_n, 0,\\rho_n, \\sigma_n) +c \\|z_{n+1}-x_n\\|^2 + d \\|\\nabla f(u_n)\\|^2\n\\notag\\\\\n\\leq \\max_{[n-N]_+ \\leq k \\leq n}V(x_k, x_{k-1}, 0,\\rho_{k-1}, \\sigma_{k-1}).    \n\\end{align}","sha256_utf8":"cce79a2a3ede2edb3e500b868897e0598cef9fab432b93428e5515f07a5c4aeb"},{"start":45049,"end":45145,"start_line":605,"end_line":605,"raw_tex":"We call this version as the \\emph{fast gradient algorithm with nonmonotone line search} (AGNL).\n","sha256_utf8":"66cddcdaf9ed3dd1826f7f163f0e54e67c41ac29f06c4344c5330098c98281f9"},{"start":45146,"end":45325,"start_line":607,"end_line":607,"raw_tex":"For each $n \\in \\N$, set $m(n)\\in \\argmax_{[n-N]_+ \\leq k \\leq n}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1})$. The following technical lemma will be of use in our analysis.\n","sha256_utf8":"f4bd67b909c36934cc207133929f352fbbcdd5337da3260956be038bc303d4a5"},{"start":61103,"end":62812,"start_line":784,"end_line":794,"raw_tex":"We now recall the Kurdyka--{\\L}ojasiewicz (KL) property \\cite{Kur98, Loj63} which plays an important role in our convergence analysis. Let $h: \\mathcal{H} \\to (-\\infty, +\\infty]$ be a proper lower semicontinuous function. We say that $h$ satisfies the \\emph{KL property} at $\\bar x \\in \\dom \\partial h$ if there exist $\\varepsilon \\in \\mathbb{R}_{++}$, $\\nu \\in (0, +\\infty]$, and a continuous concave function $\\varphi: [0, \\nu) \\to \\R_+$ such that $\\varphi(0) =0$ and $\\varphi$ is continuously differentiable on  $(0, \\nu)$ with $\\varphi' >0$, and for all $x \\in B(\\bar x; \\varepsilon)$ with $h(\\bar x) < h(x) < h (\\bar x) + \\nu$, one has\n\\begin{equation*}\n\\varphi'(h(x)- h(\\bar x))\\dist(0, \\partial h(x)) \\geq 1.\n\\end{equation*}\nIf $h$ satisfies the KL property at each point in $\\dom \\partial h$, then $h$ is called a \\emph{KL function}.\n\nThe function $h$ is said to satisfy the \\emph{KL property at $\\bar x$ with exponent $\\theta$} if it satisfies the KL property at $\\bar x \\in \\dom \\partial h$ with corresponding function $\\varphi(s) = \\gamma s^{1-\\theta}$ for some $\\gamma \\in \\R_{++}$ and $\\theta \\in [0,1)$, i.e., there exist $c \\in \\mathbb{R}_{++}$, $\\varepsilon \\in \\mathbb{R}_{++}$, and $\\nu \\in (0, +\\infty]$ such that\n\\begin{equation*}\n\\dist(0, \\partial h(x)) \\geq c (h(x) -h(\\bar x))^{\\theta}\n\\end{equation*}\nwhenever $x \\in B(\\bar x; \\varepsilon)$ with $h(\\bar x) < h(x) < h(\\bar x) + \\nu$. If $h$ is a KL function and has the same exponent $\\theta$ at any $\\bar x \\in \\dom \\partial h$, then $h$ is called a \\emph{KL function with exponent $\\theta$}. Some typical KL functions include strongly convex functions, real analytic functions, and semialgebraic functions, see \\cite{ABRS10, Loj63}.\n","sha256_utf8":"a573c549fcce842d073ec04b90b0ed42f9998341c04f9bb004563bc3e54de0bf"}],"unit_separator":"\n\n"},"statement":{"raw_tex":"\\begin{theorem}[Sequential and linear convergence]\n\\label{t:fullseq}\nSuppose that $\\mathcal{H}$ is finite dimensional, that $f$ is a differentiable function whose gradient is $L$-Lipschitz continuous, and that $f$ is bounded below. Let $(x_n)_{n\\in \\mathbb{N}}$ be the sequence generated by AGDNL algorithm with $\\lambda_{\\min} >0$. Suppose further that the set $\\{x \\in \\mathcal{H}: f(x) \\leq f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2\\}$ is bounded, and that one of the following holds\n\\begin{enumerate}\n\\item \\label{ADFGe:i}  $N=0$ and $ g(x,y,\\rho):= f(x)+ \\rho  \\|x -y\\|^2 $ satisfies the KL property at $(\\bar x, \\bar x, \\rho)$ for all $\\bar x \\in \\mathcal{H}$ and $\\rho \\in [0, \\bar\\rho]$, where $\\bar\\rho: =\\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\bar\\gamma}{\\underline{\\nu}} + \\frac{ L_{\\max}\\bar\\beta +\\bar\\gamma}{2 }$.\n\\item \\label{ADFGe:ii} $\\frac{1}{\\lambda_{\\max} }- L \\bar\\beta -\\frac{\\bar\\gamma}{\\lambda_{\\min}}-\\frac{L}{2} >0$ and $h(x,y) =f(x) +(\\frac{L \\bar\\beta}{2} +\\frac{\\bar\\gamma}{2 \\lambda_{\\min} }) \\|x-y\\|^2$ satisfies the KL property at $(\\bar x, \\bar x)$  for all $\\bar x \\in \\mathcal{H}$.\n\\end{enumerate}\nThen $\\sum_{k=0}^{+\\infty} \\|x_{n+1} -x_n\\| < +\\infty$ and $(x_n)_{n\\in \\N}$ converges to a critical point $x^*$ of $f$. Moreover, if the KL property in \\ref{ADFGe:i} or \\ref{ADFGe:ii} is satisfied with exponent $\\theta \\leq 1/2$, then there exist $\\Gamma \\in \\R_{++}$ and $\\zeta \\in (0,1)$ such that, for all $n \\in \\N$, $\\|x_n -x^*\\| \\leq \\Gamma{\\zeta}^{n/2}$ and $|f(x_n) -f(x^*) |\\leq \\Gamma{\\zeta}^n$.\n\\end{theorem}","units":[{"start":65773,"end":67318,"start_line":825,"end_line":833,"raw_tex":"\\begin{theorem}[Sequential and linear convergence]\n\\label{t:fullseq}\nSuppose that $\\mathcal{H}$ is finite dimensional, that $f$ is a differentiable function whose gradient is $L$-Lipschitz continuous, and that $f$ is bounded below. Let $(x_n)_{n\\in \\mathbb{N}}$ be the sequence generated by AGDNL algorithm with $\\lambda_{\\min} >0$. Suppose further that the set $\\{x \\in \\mathcal{H}: f(x) \\leq f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2\\}$ is bounded, and that one of the following holds\n\\begin{enumerate}\n\\item \\label{ADFGe:i}  $N=0$ and $ g(x,y,\\rho):= f(x)+ \\rho  \\|x -y\\|^2 $ satisfies the KL property at $(\\bar x, \\bar x, \\rho)$ for all $\\bar x \\in \\mathcal{H}$ and $\\rho \\in [0, \\bar\\rho]$, where $\\bar\\rho: =\\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\bar\\gamma}{\\underline{\\nu}} + \\frac{ L_{\\max}\\bar\\beta +\\bar\\gamma}{2 }$.\n\\item \\label{ADFGe:ii} $\\frac{1}{\\lambda_{\\max} }- L \\bar\\beta -\\frac{\\bar\\gamma}{\\lambda_{\\min}}-\\frac{L}{2} >0$ and $h(x,y) =f(x) +(\\frac{L \\bar\\beta}{2} +\\frac{\\bar\\gamma}{2 \\lambda_{\\min} }) \\|x-y\\|^2$ satisfies the KL property at $(\\bar x, \\bar x)$  for all $\\bar x \\in \\mathcal{H}$.\n\\end{enumerate}\nThen $\\sum_{k=0}^{+\\infty} \\|x_{n+1} -x_n\\| < +\\infty$ and $(x_n)_{n\\in \\N}$ converges to a critical point $x^*$ of $f$. Moreover, if the KL property in \\ref{ADFGe:i} or \\ref{ADFGe:ii} is satisfied with exponent $\\theta \\leq 1/2$, then there exist $\\Gamma \\in \\R_{++}$ and $\\zeta \\in (0,1)$ such that, for all $n \\in \\N$, $\\|x_n -x^*\\| \\leq \\Gamma{\\zeta}^{n/2}$ and $|f(x_n) -f(x^*) |\\leq \\Gamma{\\zeta}^n$.\n\\end{theorem}","sha256_utf8":"f2cdbf737f13ed8c367bb81c8730a2eb70eeb4f0aa481c2a9b7a1db4e15ea0d4"}],"unit_separator":"\n\n"},"proof":{"raw_tex":"\\begin{lemma}\\label{l:dry}\nSuppose that $\\phi\\colon \\mathcal{H}\\to \\left(-\\infty, +\\infty\\right]$ satisfies the dry-like friction property with constant $r$. Let $y \\in \\mathcal{H}$ and $w \\in \\prox_{\\tau \\phi}(y)$. Then $\\langle w-y, w\\rangle  + r\\tau \\|w\\| \\leq 0.$\n\\end{lemma}\n\n\\begin{proof}\nSince $w \\in \\prox_{\\tau \\phi}(y) = \\argmin_{x \\in \\mathcal{H}} \\left(\\phi(x) +\\frac{1}{2 \\tau}\\|x-y\\|^2\\right)$, we have $0 \\in \\partial \\phi(w) + \\frac{1}{\\tau}(w-y)$, which yields $\\frac{1}{\\tau}(y-w) \\in \\partial \\phi(w)$, and so $w \\in \\dom \\partial \\phi$. By the dry-like friction property of $\\phi$, $\\frac{1}{\\tau}\\langle y -w, w\\rangle \\geq r\\|w\\|$, which completes the proof.\n\\end{proof}\n\n\\begin{lemma}\n% [Descent property of the potential function]\n\\label{l:descent}\nSuppose that $f$ is a differentiable function whose gradient is $L$-Lipschitz continuous. Under the setting of AGDNL algorithm, for all $n \\in \\N$, the following hold:\n\\begin{enumerate}\n\\item\\label{l:descent_wn}\n$\\|w_n\\|^2 - \\mu \\alpha_n \\langle w_{n-1}, w_n\\rangle  +\\kappa_n \\langle \\nabla f(u_n), w_n\\rangle  +\\delta \\| w_n\\| \\leq 0$. Consequently, $\\|w_n\\|^2 + (\\delta - k_{\\max} \\|\\nabla f(u_n)\\| - \\mu  \\bar \\alpha\\|w_{n-1}\\|) \\|w_n\\| \\leq 0$. \n\\item\\label{l:descent_Vn}\nIf $L_n \\geq L$, $L_n \\beta_n \\leq L_{n-1} \\beta_{n-1}$, $\\gamma_n \\leq \\gamma_{n-1}$, $\\frac{\\gamma_n}{\\nu_n} \\leq \\frac{\\gamma_{n-1}}{\\nu_{n-1}}$, and $\\frac{\\alpha_n}{\\nu_n} \\leq \\frac{\\alpha_{n-1}}{\\nu_{n-1}}$, then\n\\begin{align*}%\\label{eq:KL1}\n&V(z_{n+1}, x_n, w_n,\\rho_n, \\sigma_n) + \\frac{c_n}{\\nu_n}\\|z_{n+1} -x_n||^2 +  \\frac{d_n}{\\nu_n}\\|\\nabla f(u_n)\\|^2 + \\frac{\\delta}{\\nu_n} \\| w_n\\| \\\\\n&\\leq V(x_n, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}),\n\\end{align*}  \nwhere\n\\begin{align}\nc_n &= \\frac{(1 -\\mu \\bar\\alpha) (1 -2\\gamma_n )}{\\mu^2} - \\frac{(2 L_n\\beta_n +\\gamma_n + L_n)\\nu_n }{2} \\label{d:cn}\\\\\n\\text{and~~}\nd_{n} &= \\frac{2(1- \\mu \\bar\\alpha)(\\lambda_n^2 -\\gamma_n\\lambda_n)-\\mu\\gamma_n\\kappa_n+2\\mu\\lambda_n \\kappa_n}{2\\mu^2} \\label{d:dn}.\n\\end{align}\n\\end{enumerate}\n\\end{lemma}\n\n\\begin{proof}\n\\ref{l:descent_wn}:\nSet $y_n = \\alpha_n(x_n -v_{n-1}+\\lambda_n\\nabla f(u_{n-1}))-\\theta_n \\frac{\\nabla f(u_n)}{\\|\\nabla f(u_n)\\|}-\\kappa_n \\nabla f(u_n)+ e_n$. By the update of $x_n$, we have $\\mu w_{n-1}= x_n - v_{n-1} +\\lambda_{n-1} \\nabla f(u_{n-1})$, and so\n\\begin{equation*}\ny_n = \\alpha_n\\mu w_{n-1} -\\theta_n \\frac{\\nabla f(u_n)}{\\|\\nabla f(u_n)\\|} -\\kappa_n \\nabla f(u_n)+ e_n.\n\\end{equation*}\nBy Step~\\ref{step:main}\\ref{step:main_comp}, $w_n \\in \\prox_{\\tau_n \\phi}(y_n)$. Since $\\phi$ satisfies the dry-like friction property with constant $r$, we have from Lemma~\\ref{l:dry} that $\\langle w_n-y_n, w_n\\rangle  + r\\tau_n \\|w_n\\| \\leq 0$, which yields\n\\begin{equation*}\n\\|w_n\\|^2 - \\alpha_n \\mu\\langle w_{n-1}, w_n\\rangle  + \\kappa_n \\langle \\nabla f(u_n), w_n\\rangle  + (r\\tau_n - |\\theta_n| - \\|e_n\\|)\\| w_n\\| \\leq 0.    \n\\end{equation*}\nCombining this with $r\\tau_n \\geq |\\theta_n| +\\|e_n\\| +\\delta$, we get the first conclusion, which implies the second one.\n\n\\ref{l:descent_Vn}: Set $\\Delta_n = x_n - x_{n-1}$. It follows from the update of $z_{n+1}$ that\n\\begin{equation*}\n\\mu w_n=z_{n+1} -v_n +\\lambda_n \\nabla f(u_n) \n=(z_{n+1} -x_n)-\\gamma_n \\Delta_n +\\lambda_n \\nabla f(u_n),    \n\\end{equation*}\nand so\n\\begin{align}\\label{eq:57}\n\\mu^2\\|w_n\\|^2&=  \\|z_{n+1} -x_n\\|^2 +\\gamma_n^2 \\|\\Delta_n\\|^2 +\\lambda_n^2 \\|\\nabla f(u_n)\\|^2 - 2\\gamma_n \\langle z_{n+1} -x_n,  \\Delta_n\\rangle \\notag\\\\\n & \\quad +2  \\lambda_n\\langle z_{n+1} -x_n, \\nabla f(u_n)\\rangle -2 \\gamma_n\\lambda_n\\langle  \\Delta_n, \\nabla f(u_n) \\rangle \\notag\\\\\n &\\geq  \\|z_{n+1} -x_n\\|^2 +\\gamma_n^2 \\|\\Delta_n\\|^2 +\\lambda_n^2 \\|\\nabla f(u_n)\\|^2 - \\gamma_n (\\|z_{n+1}-x_n\\|^2 + \\|\\Delta_n\\|^2)  \\notag\\\\\n  & \\quad +2 \\lambda_n\\langle z_{n+1} -x_n, \\nabla f(u_n)\\rangle - \\gamma_n\\lambda_n(\\|\\Delta_n\\|^2 + \\|\\nabla f(u_n)\\|^2) \\notag\\\\\n  &=  (1-\\gamma_n)\\|z_{n+1} -x_n\\|^2 +(\\gamma_n^2-\\gamma_n -\\gamma_n\\lambda_n) \\|\\Delta_n\\|^2 + (\\lambda_n^2 -\\gamma_n\\lambda_n) \\|\\nabla f(u_n)\\|^2 \\notag\\\\\n   & \\quad +2 \\lambda_n\\langle z_{n+1} -x_n, \\nabla f(u_n)\\rangle.\n\\end{align}\nSince $\\nabla f$ is $L$-Lipschitz continuous, $L_n \\geq L$, and $u_n = x_n + \\beta_n(x_n - x_{n-1})$, we have\n\\begin{align}\\label{eq:es2}\n\\langle \\nabla f(u_n), z_{n+1} -x_n \\rangle &= \\langle \\nabla f(u_n) - \\nabla f(x_n), z_{n+1} - x_{n} \\rangle + \\langle \\nabla f(x_n), z_{n+1} -x_n \\rangle \\notag\\\\\n& \\geq -L_n\\|u_n -x_n\\| \\|z_{n+1} -x_n\\| + f(z_{n+1}) -f(x_n) -\\frac{L_n}{2}\\|z_{n+1} - x_n\\|^2 \\notag\\\\\n% & \\geq -\\frac{L_n\\beta_n}{2}(\\|\\Delta_n\\|^2 + \\|z_{n+1}-x_n\\|^2) + f(z_{n+1}) -f(x_n) -\\frac{L_n}{2}\\|z_{n+1} -x_n\\|^2\\notag\\\\\n& \\geq  -\\frac{L_n\\beta_n}{2}\\|\\Delta_n\\|^2 -\\frac{L_n\\beta_n +L_n}{2}\\|z_{n+1} -x_n\\|^2 + f(z_{n+1}) -f(x_n).\n\\end{align}\nCombining this with \\eqref{eq:57}, we derive that\n\\begin{align}\\label{eq:es1}\n &\\|w_n\\|^2 - \\mu \\alpha_n \\langle w_{n-1}, w_n\\rangle+\\kappa_n \\langle \\nabla f(u_n), w_n\\rangle \\notag\\\\\n&\\geq \\|w_n\\|^2 - \\frac{\\mu \\alpha_n}{2}(\\|w_n\\|^2 +\\|w_{n-1}\\|^2)+\\kappa_n \\langle \\nabla f(u_n), w_n\\rangle \\notag\\\\\n& \\geq  -\\frac{\\mu \\alpha_{n}}{2} \\|w_{n-1}\\|^2 +\\frac{\\mu \\alpha_n}{2} \\|w_n\\|^2 +(1-\\mu \\bar\\alpha)\\|w_n\\|^2\\notag\\\\\n&\\quad+\\frac{\\kappa_n }{\\mu} \\langle \\nabla f(u_n), z_{n+1} -x_n \\rangle -\\frac{\\gamma_n\\kappa_n }{\\mu} \\langle \\nabla f(u_n), \\Delta_n \\rangle +\\frac{\\lambda_n \\kappa_n }{\\mu}\\|\\nabla f(u_n)\\|^2\\notag\\\\\n& \\geq  -\\frac{\\mu \\alpha_{n}}{2} \\|w_{n-1}\\|^2  +\\frac{\\mu \\alpha_n}{2} \\|w_n\\|^2 \\notag\\\\\n& \\quad +\\frac{1-\\mu \\bar\\alpha }{\\mu^2}\\left( (1-\\gamma_n)\\|z_{n+1} -x_n\\|^2 +(\\gamma_n^2-\\gamma_n -\\gamma_n\\lambda_n) \\|\\Delta_n\\|^2 + (\\lambda_n^2 -\\gamma_n\\lambda_n) \\|\\nabla f(u_n)\\|^2  \\right)\\notag\\\\\n  & \\quad+ \\frac{2\\lambda_n(1-\\mu \\bar\\alpha)}{\\mu^2}\\langle z_{n+1} -x_n, \\nabla f(u_n)\\rangle +\\frac{\\kappa_n}{\\mu} \\langle \\nabla f(u_n), z_{n+1} -x_n \\rangle  \\notag\\\\\n  &\\quad -\\frac{\\gamma_n \\kappa_n }{2\\mu} (\\|\\nabla f(u_n)\\|^2 +\\|\\Delta_n \\|^2) +\\frac{\\lambda_n\\kappa_n }{\\mu}\\|\\nabla f(u_n)\\|^2\\notag\\\\\n&\\geq -\\frac{\\mu \\alpha_{n}}{2} \\|w_{n-1}\\|^2 +\\frac{\\mu \\alpha_n}{2} \\|w_n\\|^2 +\\bar c_n\\|z_{n+1}-x_n\\|^2 -\\tilde c_n\\|\\Delta_n\\|^2+ d_n\\|\\nabla f(u_n)\\|^2\\notag\\\\\n&\\quad +\\left(\\frac{2\\lambda_n(1-\\mu \\bar\\alpha)}{\\mu^2} +\\frac{\\kappa_n }{\\mu}\\right)( f(z_{n+1}) -f(x_n)), \n\\end{align}\nwhere the last inequality follows from \\eqref{eq:es2}, $d_n$ is defined in \\eqref{d:dn} and\n\\begin{align}\n\\bar c_n &=\\frac{(1-\\mu \\bar\\alpha) (1-\\gamma_n)}{\\mu^2} -\\frac{L_n\\beta_n + L_n}{2 }  \\left(\\frac{2\\lambda_n(1-\\mu \\bar\\alpha)}{\\mu^2}+ \\frac{\\kappa_n}{\\mu}\\right), \\notag\\\\\n\\tilde c_n  &=\\frac{(1 -\\mu \\bar\\alpha)\\left(\\gamma_n +\\gamma_n\\lambda_n+L_n\\lambda_n \\beta_n \\right)  + \\frac{\\mu \\kappa_n}{2}(L_n\\beta_n +\\gamma_n)}{\\mu^2} \\notag\\\\\n&=\\frac{(1 -\\mu \\bar\\alpha)\\gamma_n}{\\mu^2} + \\frac{ L_n\\beta_n +\\gamma_n}{2 } \\left(\\frac{2\\lambda_n(1-\\mu \\bar\\alpha)}{\\mu^2}+ \\frac{\\kappa_n}{\\mu}\\right) =\\rho_n \\nu_n.\\label{d:eta} \n\\end{align}\nIt follows from \\ref{l:descent_wn}, \\eqref{eq:es1}, and the definition of $\\nu_n$ that\n\\begin{align*}\n&\\frac{\\mu \\alpha_n}{2} \\|w_n\\|^2 -\\frac{\\mu \\alpha_n}{2}\\|w_{n-1}\\|^2  +  \\nu_n(f(z_{n+1}) -f(x_n))+ \\bar c_n \\|z_{n+1} -x_n||^2  - \\rho_n \\nu_n\\|\\Delta_{n}\\|^2 \\notag\\\\\n& + d_n\\|\\nabla f(u_n)\\|^2 + \\delta \\| w_n\\|  \\leq 0,\n\\end{align*}\nwhich implies that\n\\begin{align}\\label{eq:ADFGe_4}\n &f(z_{n+1})+ \\rho_n \\|z_{n+1} -x_n||^2 + \\frac{\\mu \\alpha_n}{2\\nu_n} \\|w_n\\|^2 + \\frac{c_n}{\\nu_n}\\|z_{n+1} -x_n||^2 +  \\frac{d_n}{\\nu_n}\\|\\nabla f(u_n)\\|^2 \\notag\\\\\n &\\hspace{3cm} + \\frac{\\delta}{\\nu_n} \\| w_n\\| \\leq f(x_n)+ \\rho_n \\|x_n -x_{n-1}||^2 + \\frac{\\mu \\alpha_n}{2 \\nu_n} \\|w_{n-1}\\|^2.\n  \\end{align}  \nHere, we note that\n\\begin{align*}\n\\bar c_n-\\rho_n \\nu_n&= \\frac{(1 -\\mu \\bar\\alpha) (1 -2\\gamma_n )}{\\mu^2} - \\frac{2 L_n\\beta_n +\\gamma_n + L_n}{2} \\left(\\frac{2\\lambda_n(1-\\mu \\bar\\alpha)}{\\mu^2}+ \\frac{\\kappa_n}{\\mu}\\right) \\\\\n&= \\frac{(1 -\\mu \\bar\\alpha) (1 -2\\gamma_n )}{\\mu^2} - \\frac{(2 L_n\\beta_n +\\gamma_n + L_n)\\nu_n}{2}  =c_n.\n\\end{align*}\nSince $L_n \\beta_n \\leq L_{n-1}\\beta_{n-1}$, $\\gamma_n \\leq \\gamma_{n-1}$, and $\\frac{\\gamma_n}{\\nu_n} \\leq \\frac{\\gamma_{n-1}}{\\nu_{n-1}}$, we have $\\rho_n \\leq \\rho_{n-1}$. As $\\frac{\\alpha_n}{\\nu_n} \\leq \\frac{\\alpha_{n-1}}{\\nu_{n-1}}$, using \\eqref{eq:ADFGe_4} and the definition of $V$ in \\eqref{eq:V}, we get the conclusion. \n\\end{proof}\n\n\\begin{lemma}[Well-definedness of the line search criterion]\nSuppose that $f$ is a differentiable function whose gradient is $L$-Lipschitz continuous. Then, for all $n \\in \\N$, the line search criterion \\eqref{eq:linesearch} in AGDNL algorithm is satisfied after a finite number of inner iterations.\n\\end{lemma}\n\n\\begin{proof}\nFix $n \\in \\mathbb{N}$. Since $\\ell_1 \\in (0, 1)$ and $\\tilde{L}_n > L_{\\min}$, there exists $k_0$ such that $\\ell_1^{-k_0}\\tilde L_n\\geq L_{\\max}$. It also can be seen that there exists $k_1$ such that $\\ell_3^{k_1} \\tilde{\\lambda}_n \\leq \\lambda_{\\min}$ and $\\ell_3^{k_1} \\tilde{\\kappa}_n \\leq \\kappa_{\\min}$ since $\\ell_3 \\in (0,1)$. Suppose that the line search criterion \\eqref{eq:linesearch} is not satisfied after $\\max\\{k_0, k_1\\}$ inner iterations. At the $k$-th inner iteration with $k >\\max\\{k_0, k_1\\}$, by Step~\\ref{step:main}\\ref{step:main_mult},\n\\begin{equation}\\label{eq:kth}\nL_n = L_{\\max}, \\quad \\gamma_n =\\ell_{2}^{k} \\tilde{\\gamma}_n, \\quad \\alpha_n =\\ell_{2}^{k} \\tilde{\\alpha}_n, \\quad \\beta_n =\\ell_{2}^{k} \\tilde{\\beta}_n, \\quad \\lambda_n = \\lambda_{\\min}, \\text{~and~} \\kappa_n =\\kappa_{\\min}.\n\\end{equation}\nThen $\\nu_n = \\frac{2 \\lambda_{\\min}(1 -\\mu \\bar \\alpha)}{\\mu^2}+ \\frac{\\kappa_{\\min} }{\\mu} = \\underline{\\nu}$, and so $\\frac{\\gamma_n}{\\nu_n} =\\frac{\\ell_2^k\\tilde{\\gamma}_n}{\\nu_n} =\\frac{\\ell_2^k\\tilde{\\gamma}_n}{\\underline{\\nu}}$ and $\\frac{\\alpha_n}{\\nu_n} =\\frac{\\ell_2^k\\tilde{\\alpha}_n}{\\nu_n} =\\frac{\\ell_2^k \\tilde{\\alpha}_n}{\\underline{\\nu}}$. Since $0<\\ell_2 <1$, there exists $k_2 >\\max\\{k_0, k_1\\}$ such that after $k_2$ inner iterations, $\\gamma_n \\leq \\gamma_{n-1}$, $L_n \\beta_n = L_{\\max} \\ell_{2}^{k} \\tilde{\\beta}_n  \\leq L_{n-1} \\beta_{n-1}$,  $\\frac{\\gamma_n}{\\nu_n} \\leq \\frac{\\gamma_{n-1}}{\\nu_{n-1}}$, and $\\frac{\\alpha_n}{\\nu_n} \\leq \\frac{\\alpha_{n-1}}{\\nu_{n-1}}$.\nUsing Lemma~\\ref{l:descent} and noting that $\\nu_n =\\underline{\\nu}$, we get\n\\begin{align}\n\\label{eq:lsV}\n&V(z_{n+1}, x_n, w_n,\\rho_n, \\sigma_n) + \\frac{c_n}{\\underline{\\nu}}\\|z_{n+1} -x_n||^2 +  \\frac{d_n}{\\underline{\\nu}}\\|\\nabla f(u_n)\\|^2 + \\frac{\\delta}{\\underline{\\nu}} \\| w_n\\| \\notag \\\\\n&\\leq V(x_n, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}).\n\\end{align}\nIt follows from \\eqref{eq:kth} and the definition of $c_n$ in \\eqref{d:cn} that\n\\begin{align}\\label{eq:cn}\nc_n&= \\frac{(1 -\\mu \\bar\\alpha) (1 -2\\ell_{2}^{k} \\tilde{\\gamma}_n)}{\\mu^2} - \\frac{( 2 \\ell_{2}^{k}L_{\\max} \\tilde{\\beta}_n +\\ell_{2}^{k}\\tilde{\\gamma}_n + L_{\\max})\\underline{\\nu}}{2} \\notag\\\\\n&= \\frac{ 1-\\mu \\bar\\alpha}{\\mu^2}- \\frac{L_{\\max} \\underline{\\nu}}{2}-\\ell_2^k \\left[ \\frac{2\\tilde{\\gamma}_n (1-\\mu \\bar\\alpha)}{\\mu^2}  + \\frac{2L_{\\max}\\tilde{\\beta}_n \\underline{\\nu} +\\tilde{\\gamma}_n\\underline{\\nu}}{2} \\right]\\notag\\\\\n&=\\bar c \\underline{\\nu} -\\ell_2^k \\left[ \\frac{2\\tilde{\\gamma}_n (1-\\mu \\bar\\alpha)}{\\mu^2}  + \\frac{2L_{\\max}\\tilde{\\beta}_n \\underline{\\nu} +\\tilde{\\gamma}_n\\underline{\\nu}}{2} \\right].\n\\end{align}\nSince $c \\leq \\bar c=\\frac{2(1-\\mu \\bar \\alpha) - \\mu^2 L_{\\max} \\underline{\\nu}}{2 \\mu^2 \\underline{\\nu}}$, we consider two cases: $c <\\bar c$ and $c = \\bar c$. If $c <\\bar c$, using \\eqref{eq:cn} and the fact that $0< \\ell_{2} <1$, there exists $k_3$ such that after $k_3$ inner iterations, $c_n \\geq c\\underline{\\nu}$. If $c = \\bar c$, by the choice of $\\gamma_n$, $\\beta_n$ in Step~\\ref{step:init} and Step~\\ref{step:main}, for all $n \\in \\N$, $\\tilde \\gamma_n =0$ and $\\tilde \\beta_n =0$, which together with \\eqref{eq:cn} yields $c_n \\geq c\\underline{\\nu}$.\n\nNow, we consider $d_n$. Using definition of $d_n$ and \\eqref{eq:kth}, we have \n\\begin{align*}\nd_{n}&= \\frac{2(1- \\mu \\bar\\alpha)(\\lambda_n^2 -\\ell_{2}^k \\tilde{\\gamma}_n\\lambda_n)-\\mu \\ell_{2}^{k}\\tilde{\\gamma}_n\\kappa_n+2\\lambda_n \\mu \\kappa_n }{2\\mu^2}\\notag\\\\\n& = \\frac{(1-\\mu \\bar\\alpha)\\lambda_{\\min}^2 + \\lambda_{\\min} \\mu \\kappa_{\\min}}{\\mu^2} - \\ell_2^k\\frac{\\tilde{\\gamma}_n(2(1 -\\mu \\bar\\alpha)\\lambda_{\\min} +\\mu\\kappa_{\\min})}{2\\mu^2} \\notag\\\\\n& =\\bar d \\underline{\\nu} - \\ell_2^k\\frac{\\tilde{\\gamma}_n(2(1 -\\mu \\bar\\alpha)\\lambda_{\\min} +\\mu\\kappa_{\\min})}{2\\mu^2}.\n\\end{align*}\nBy Step~\\ref{step:init} and Step~\\ref{step:main}, $d \\in [0, \\bar d]$ and if $d =\\bar d$, then $\\tilde{\\gamma}_n =0$. Since $0<\\ell_2 <1$, there exists $k_4$ such that after $k_4$ inner iterations, $d_{n} \\geq d \\underline{\\nu}$. Hence, in view of \\eqref{eq:lsV}, after at most $\\max\\{ k_2,k_3, k_4\\}$ inner iterations, the line search criterion \\eqref{eq:linesearch} holds.\n\\end{proof}\n\n\\begin{lemma}\n\\label{l:limit}\nLet $f \\colon \\mathcal{H} \\to \\R$ be a continuous differentiable function. Let $(x_n)_{n \\in \\N}$ and $(u_n)_{n \\in \\N}$ be sequences in $\\mathcal{H}$ such that $(x_n)_{n \\in \\N}$ is bounded and $x_n -u_n \\to 0$ as $n \\to +\\infty$. Then $f(x_n) -f(u_n) \\to 0$ as $n \\to +\\infty$.\n\\end{lemma}\n\n\\begin{proof}\nBy the mean value theorem, $f(x_n) -f(u_n) = \\langle \\nabla f(z_n), x_n -u_n \\rangle$, where $z_n = x_n -\\zeta_n (x_n -u_n)$ for some $\\zeta_n \\in [0,1]$. Then $(z_n)_{n \\in \\N}$ is bounded due to the assumptions on $(x_n)_{n \\in \\N}$ and $(u_n)_{n \\in \\N}$. This together with the continuity of $\\nabla f$ implies the existence of $M$ such that, for all $n \\in \\N$, $\\|\\nabla f(z_n)\\| \\leq M$, which yields $|f(x_n) -f(u_n) | \\leq M\\|x_n -u_n\\|$. Since $x_n -u_n \\to 0$ as $n \\to +\\infty$, we obtain the conclusion.\n\\end{proof}\n\n\\begin{theorem}[Subsequential convergence and structural simplification]\n\\label{t:subseq}\nLet $(x_n)_{n\\in \\mathbb{N}}$ be the sequence generated by AGDNL algorithm and $\\Omega$ be the set of strong cluster points of $(x_n)_{n \\in \\N}$. Suppose that $f$ is a differentiable function whose gradient is $L$-Lipschitz continuous, that $f$ is bounded below, and that the set $\\{x \\in \\mathcal{H}: f(x) \\leq f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2\\}$ is bounded. Then the following hold:\n\\begin{enumerate}\n\\item\\label{t:subseq_bounded}\n$(x_n)_{n \\in \\N}$ is bounded.\n\\item\\label{t:subseq_limit} \nAs $ n \\to +\\infty$, $\\|x_{n+1}-x_n\\| \\to 0$,  $\\|w_n\\| \\to 0$, $f(x_n) \\to \\zeta$, and \\\\ $V(x_{n}, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}) \\to \\zeta$ for some $\\zeta \\in \\mathbb{R}$. Moreover, for all $\\bar x \\in \\Omega$, $ f(\\bar x) =\\zeta$.  \n\\item\\label{t:subseq_simplify} \nSuppose that $\\lambda_{\\min} >0$. Then $\\lim_{n \\to +\\infty } \\nabla f(x_n) =0$ and for all $\\bar x \\in \\Omega$, $\\nabla f(\\bar x) =0$. Moreover, from some $N_0 \\in \\N$ onwards, $w_n =0$ and AGDNL algorithm becomes AGNL algorithm.\n\\end{enumerate}\n\\end{theorem}\n\n\\begin{proof}\n\\ref{t:subseq_bounded}: First, we prove by induction that, for all $n \\in \\N$,\n\\begin{equation}\\label{eq:bx1}\nV(x_{n}, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}) \\leq f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2.\n\\end{equation}\nSince $x_{-1} =x_0$, we have $V(x_{0}, x_{-1}, w_{-1},\\rho_{-1},\\sigma_{-1}) = f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2$, so \\eqref{eq:bx1} holds for $n=0$. Suppose that \\eqref{eq:bx1} holds for all $n \\leq n_0$. %We have $\\sigma_{-1} =0$ since $\\alpha_{-1} =0$.\nFrom \\eqref{eq:linesearch}, we obtain that\n\\begin{align*}\nV(x_{n_0+1}, x_{n_0}, w_{n_0},\\rho_{n_0}, \\sigma_{n_0}) &\\leq \\max_{[n_0-N]_+ \\leq k \\leq n_0}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1})\\notag\\\\\n&\\leq   f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2,\n\\end{align*}\nand hence \\eqref{eq:bx1} is satisfied for $n=n_0+1$, which completes the induction. Next, it follows from \\eqref{eq:bx1} and the definition of $V$ that $f(x_n) \\leq  f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2$. The conclusion now follows from the boundedness of the set $\\{x \\in \\mathcal{H}: f(x) \\leq f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2\\}$.\n\n\\ref{t:subseq_limit}: It follows from the line search criterion \\eqref{eq:linesearch} that\n\\begin{align}\\label{eq:lsx}\n&V(x_{n+1}, x_n, w_n,\\rho_n, \\sigma_n) +c\\|x_{n+1}-x_n\\|^2  + d \\|\\nabla f(u_n)\\|^2+ \\frac{\\delta}{\\underline{\\nu}}\\| w_n\\|\\notag\\\\ &\\leq \\max_{[n-N]_+ \\leq k \\leq n}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1})\\notag\\\\\n&= V(x_{m(n)}, x_{m(n)-1}, w_{m(n)-1},\\rho_{m(n)-1}, \\sigma_{m(n)-1}).\n\\end{align}\nTherefore,\n\\begin{align*}\n&V(x_{m(n+1)}, x_{m(n)}, w_{m(n)},\\rho_{m(n)}, \\sigma_{m(n)}) \\notag\\\\\n&= \\max_{[n+1-N]_+ \\leq k \\leq n+1}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1})\\notag\\\\\n&=\\max \\left\\{V(x_{n+1}, x_n, w_n,\\rho_n, \\sigma_n), \\max_{[n+1-N]_+ \\leq k \\leq n}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1}) \\right\\}\\notag\\\\\n& \\leq  \\max_{[n-N]_+ \\leq k \\leq n}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1}) \\notag\\\\\n&=V(x_{m(n)}, x_{m(n)-1}, w_{m(n)-1},\\rho_{m(n)-1}, \\sigma_{m(n)-1}),\n\\end{align*}\nwhich implies that the sequence $(V(x_{m(n+1)}, x_{m(n)}, w_{m(n)},\\rho_{m(n)}, \\sigma_{m(n)}))_{n\\in \\N}$ is nonincreasing. On the other hand, this sequence is bounded below due to the bounded below property of $f$ and the definition of $V$. Thus, there exists $\\zeta$ such that\n\\begin{equation}\\label{eq:limV}\n\\zeta =\\lim_{n \\to +\\infty}V(x_{m(n)}, x_{m(n)-1}, w_{m(n)-1},\\rho_{m(n)-1}, \\sigma_{m(n)-1}).\n\\end{equation} \n%\\ref{t:subseq_limit}\nIn \\eqref{eq:lsx}, replacing $n$ by $m(n)-1$ yields\n\\begin{align*}\n&V(x_{m(n)}, x_{m(n)-1}, w_{m(n)-1},\\rho_{m(n)-1}, \\sigma_{m(n)-1}) +c\\|x_{m(n)}-x_{m(n)-1}\\|^2 \\notag\\\\ \n&+d \\|\\nabla f(u_{m(n)-1})\\|^2 +\\frac{\\delta}{\\underline{\\nu}}\\| w_{m(n)-1}\\| \\notag\\\\\n&\\leq V(x_{m(m(n)-1)}, x_{m(m(n)-1)-1}, w_{m(m(n)-1)-1},\\rho_{m(m(n)-1)-1}, \\sigma_{m(m(n)-1)-1}).\n\\end{align*}\nLetting $n \\to +\\infty$ and noting that $m(n) \\geq [n-N]_{+}$, we get $ \\lim_{n \\to +\\infty}\\|w_{m(n)-1} \\| = 0$. \nIf $c >0 $, we also have $\\lim_{n \\to +\\infty}\\|x_{m(n)}-x_{m(n)-1}\\|~=~0$. If $c=0$ and $d >0$, then $\\lim_{n \\to +\\infty}\\|\\nabla f(u_{m(n)-1}) \\| = 0$, which implies that \n\\begin{equation*}\n |x_{m(n)} - x_{m(n)-1}\\| =\\|-\\lambda_{m(n)-1}\\nabla f(u_{m(n)-1}) + \\mu w_{m(n)-1} \\| \\to 0 \\text{~as~} n \\to +\\infty   \n\\end{equation*}\nIf $c=d=0$, then $\\lambda_{\\min} =0$ since  $d >0$ if $\\lambda_{\\min} >0$ and $c=0$. By Step~\\ref{step:main}, for all $n \\in \\N$, $\\lambda_n = \\gamma_n =0$, and so $\\lim_{n \\to +\\infty}\\|x_{m(n)}-x_{m(n)-1}\\| =\\lim_{n \\to +\\infty}\\mu \\|w_{m(n)-1} \\| =0$. Therefore, in any cases,\n\\begin{equation}\\label{eq:indu1}\n\\lim_{n \\to +\\infty}\\|x_{m(n)}-x_{m(n)-1}\\| = 0 \\text{~and~} \\lim_{n \\to +\\infty}\\|w_{m(n)-1} \\| = 0.\n\\end{equation}\nBy the definitions of $\\rho_n$ and $\\sigma_n$, for all $n \\in \\N$,\n$0\\leq \\rho_n \\leq  \\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\bar\\gamma}{\\underline{\\nu}} + \\frac{ L_{\\max}\\bar\\beta +\\bar\\gamma}{2 }$ and $ 0\\leq  \\sigma_n \\leq\\frac{\\mu \\alpha_{\\max}}{2 \\underline{\\nu}}.$ Combining with \\eqref{eq:limV} and \\eqref{eq:indu1}, we derive that\n\\begin{align*}\nf(x_{m(n)})&=V(x_{m(n)}, x_{m(n)-1}, w_{m(n)-1},\\rho_{m(n)-1}, \\sigma_{m(n)-1}) \\notag\\\\\n&\\quad - \\rho_{m(n)-1}\\|x_{m(n)}-x_{m(n)-1}\\|^2 - \\sigma_{m(n)-1} \\| w_{m(n)-1}\\|^2\n\\end{align*}\nalso tends to $\\zeta$ as $n \\to +\\infty$. Since $(x_n)_{n\\in \\N}$ is bounded and $\\lim_{n \\to +\\infty}\\|x_{m(n)}-x_{m(n)-1}\\| = 0$, Lemma~\\ref{l:limit} implies that \n\\begin{equation}\\label{eq:indu2}\n\\lim_{n \\to +\\infty}f(x_{m(n)-1})=\\lim_{n \\to +\\infty}f(x_{m(n)}) = \\zeta.\n\\end{equation}\nNow, we prove by induction that for all $k \\in [1, m(n)]$,\n\\begin{equation}\\label{eq:indu}\n\\lim_{n \\to +\\infty}\\|x_{m(n)-k+1}-x_{m(n)-k}\\|=0 , \\lim_{n \\to +\\infty} \\|w_{m(n)-k}\\|=0, \\text{~and~} \\lim_{n \\to +\\infty} f(x_{m(n)-k}) = \\zeta.\n\\end{equation}\nIt follows from \\eqref{eq:indu1} and \\eqref{eq:indu2} that \\eqref{eq:indu} holds for $k=1$. Suppose that \\eqref{eq:indu} holds for $k=k_0$. Replacing $n$ by $ m(n)-k_0-1:= n_m$ in \\eqref{eq:lsx} yields\n\\begin{align*}\n&V(x_{m_n+1}, x_{m_n}, w_{m_n},\\rho_{m_n}, \\sigma_{m_n}) +c\\|x_{m_n+1}-x_{m_n}\\|^2 +d \\|\\nabla f(u_{m_n})\\|^2+\\delta\\| w_{m_n}\\| \\notag\\\\\n&\\leq V(x_{m(m_n)}, x_{m(m_n)-1}, w_{m(m_n)-1},\\rho_{m(m_n)-1},\\sigma_{m(m_n)-1}).\n\\end{align*}\nUsing the definition of $V$, we have\n\\begin{align*}\n&(c+ \\rho_{m_n})\\|x_{m_n+1}-x_{m_n}\\|^2  +(\\delta + \\sigma_{m_n})\\| w_{m_n}\\| +d \\|\\nabla f(u_{m_n})\\|^2 \\notag\\\\\n&\\leq V(x_{m(m_n)}, x_{m(m_n)-1}, w_{m(m_n)-1},\\rho_{m(m_n)-1},\\sigma_{m(m_n)-1}) - f(x_{m_n+1}).\n\\end{align*}\nSimilar to the proof of \\eqref{eq:indu1}, by the induction hypothesis (\\eqref{eq:indu} holds with $k=k_0$) and \\eqref{eq:limV}, we obtain that \\eqref{eq:indu} holds for $k=k_0+1$, which completes the induction.\n\nNext, let $n \\geq N+1$. By the definition of $m(n)$, we have $n -N \\leq m(n) \\leq n$, and so $k_n := m(n) -(n -N) +1 \\in [1, m(n)]$. Then $n-N = m(n) -k_n +1$ and by \\eqref{eq:indu}, \n\\begin{align*}\n&\\lim_{n \\to +\\infty}\\|x_n-x_{n-1}\\| = \\lim_{n \\to +\\infty}\\|x_{m(n)-k_n +1}-x_{m(n)-k_n}\\| =0, \\\\\n&\\lim_{n \\to +\\infty}\\|w_{n}\\|=\\lim_{n \\to +\\infty}\\|w_{n-N-1}\\| =\\lim_{n \\to +\\infty}\\|w_{m(n)-k_n}\\|= 0, \\text{~and}\\\\\n&\\lim_{n \\to +\\infty} f(x_{n}) =\\lim_{n \\to +\\infty} f(x_{n-N-1}) = \\lim_{n \\to +\\infty} f(x_{m(n)-k_n }) = \\zeta.\n\\end{align*}\nCombining this with the definition of $V$ and the boundedness of $\\rho_n$ and $\\sigma_n$, we deduce that\n\\begin{equation}\\label{eq:limitf}\n\\lim_{n \\to +\\infty}V(x_{n}, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}) = \\lim_{n \\to +\\infty}f(x_n) = \\zeta.\n\\end{equation}\nTake an arbitrary $\\bar x \\in \\Omega$. Then there exists a sequence $(x_{k_n})_{n \\in \\N}$ such that $x_{k_n} \\to \\bar x$ as $n \\to +\\infty$. Since $f$ is continuous, we have $\\lim_{n \\to +\\infty}f(x_{k_n}) =f(\\bar x)$. This, together with \\eqref{eq:limitf}, yields that $f(x_n) \\to f(\\bar x)$ and $V(x_{n}, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}) \\to f(\\bar x)$ as $n \\to +\\infty$.\n\n\\ref{t:subseq_simplify}: Since $\\lambda_{n} \\nabla f(u_n) = v_n- x_{n+1} + \\mu w_n = -(x_{n+1} -x_n) + \\gamma_n (x_n -x_{n-1}) + \\mu w_n$, we have that\n\\begin{align*}\n\\lambda_{\\min} \\|\\nabla f(u_n)\\|\\leq \\lambda_n \\|\\nabla f(u_n)\\| \n% &\\leq \\|x_{n+1} -x_n\\| +\\gamma_n\\|x_n -x_{n-1}\\| + \\mu\\|w_n\\| \\notag\\\\\n\\leq \\|x_{n+1} -x_n\\| +\\bar\\gamma\\|x_n -x_{n-1}\\| + \\mu \\|w_n\\|.\n\\end{align*}\nCombining with \\ref{t:subseq_limit} yields $\\lim_{n \\to +\\infty} \\|\\nabla f(u_n)\\|=0$. Since $\\nabla f$ is $L$-Lipschitz continuous, it holds that\n\\begin{equation*}\n0\\leq \\left|\\|\\nabla f(u_n) \\|-\\|\\nabla f(x_n)\\|\\right| \\leq \\|\\nabla f(u_n) -\\nabla f(x_n)\\| \\leq L\\|u_n -x_n\\| = L\\beta_n\\|x_n -x_{n-1}\\|\n\\end{equation*}\nwhich implies that $\\lim_{n \\to +\\infty} \\|\\nabla f(x_n)\\| =0$, and so $\\lim_{n \\to +\\infty} \\|\\nabla f(\\bar x)\\| =0$ any $\\bar x \\in \\Omega$. \n\nIt follows from $\\lim_{n \\to +\\infty}\\|\\nabla f(u_n)\\| =0$ and $\\lim_{n \\to +\\infty}\\|w_n\\| =0$ that there exists $N_0 \\in \\mathbb{N}$ such that, for all $n \\geq N_0$, $\\kappa_{\\max} \\|\\nabla f(u_n)\\| +\\mu \\bar\\alpha \\|w_{n-1}\\| \\leq \\delta$. Combining this with Lemma~\\ref{l:descent}\\ref{l:descent_wn}, we have for all $n \\geq N_0$, $w_n =0$, which yields $ x_{n+1} -v_n +\\lambda_n \\nabla f(u_n)=0$, where $u_n = x_n +\\beta_n(x_n-x_{n-1})$ and $v_n = x_n +\\gamma_n(x_n -x_{n-1})$.\n\\end{proof}\n\n\\begin{lemma}[Abstract convergence under KL property]\n\\label{l:KL}\nLet $\\mathcal{H}$ and $\\mathcal{K}$ be two finite-dimensional real Hilbert\nspaces. Let $h\\colon \\mathcal{K}\\to \\R$ be a continuous function, $(x_n)_{n\\in \\mathbb{N}}$ and $(z_n)_{n\\in \\mathbb{N}}$ be sequences in $\\mathcal{H}$ and $\\mathcal{K}$, respectively, $(\\Delta_n)_{n\\in \\mathbb{N}}$ be a sequence of nonnegative real numbers, and $I$ be a finite set of integer indices with the smallest index not exceeding $1$. Set $\\Delta_k =0$ for $k <0$ and consider the following conditions:\n\\begin{enumerate}\n\\renewcommand\\theenumi{(\\alph{enumi})}\n\\renewcommand{\\labelenumi}{\\rm (\\alph{enumi})}\n\\item\\label{a:decrease}\nThere exists a positive real number $\\alpha$ such that, for all $n \\in \\N$, $h(z_{n+1}) + \\alpha \\Delta_n^2 \\leq  h(z_n)$.\n\\item\\label{a:error}\nThere exist nonnegative real numbers $\\beta_i$, $i\\in I$ such that, for all $n \\in \\N$, $\\dist(0, \\partial h(z_n)) \\leq \\sum_{i\\in I} \\beta_i\\Delta_{n-i}$.\n\\item\\label{a:distance}\nThere exists an integer $j$ such that, for all $n \\in \\N$, $\\|x_{n+1} -x_n\\| \\leq  \\Delta_{n+j}$.\n\\end{enumerate}\nSuppose that \\ref{a:decrease} and \\ref{a:error} hold, that the sequence $(z_n)_{n \\in \\N}$ is bounded with $\\Omega$ being the set of cluster points, and that $h$ satisfies the KL property at any point of $\\Omega$. Then the following hold:\n\\begin{enumerate}\n\\item\nFor all $\\bar z \\in \\Omega$, $0 \\in \\partial h(\\bar z)$ and $h(z_n)\\downarrow \\bar h := h(\\overline{z})$ as $n\\to +\\infty$.\n\\item \nIf \\ref{a:distance} holds, then $\\sum_{n=0}^{+\\infty} \\|x_{n+1} -x_n\\| < +\\infty$ and the sequence $(x_n)_{n \\in \\N}$ converges to a point $\\bar x \\in \\mathcal{H}$.\n\\item \nSuppose further that $h$ satisfies the KL property at any point of $\\Omega$ with exponent $\\alpha \\leq 1/2$. Then there exist $\\Gamma_1 \\in \\R_{++}$ and $\\zeta \\in (0,1)$ such that, for all $n \\in \\N$, $0 \\leq h(z_n) - \\bar h \\leq \\Gamma_1 \\zeta^{n}$. Moreover, if \\ref{a:distance} holds, then there exists $\\Gamma_2 \\in \\R_{++}$ such that, for all $n \\in \\N$, $\\|x_n -\\bar x\\| \\leq \\Gamma_2 \\zeta^{n/2}.$\n\\end{enumerate}\n\\end{lemma}\n\n\\begin{proof}\nIn view of \\cite[Lemma~5.1 and Theorem~5.2]{BDL21}, we only need to check that (I) there  exist a subsequence $(z_{k_n})_{n \\in \\N}$ and $\\tilde{z}$ such that $z_{k_n} \\to \\tilde{z}$ and $h(z_{k_n}) \\to h(\\tilde{z})$ as $n \\to +\\infty$; and (II) $h$ is constant on $\\Omega$. The first one is due to the continuity of $h$ and the boundedness of $(z_{n})_{n \\in \\N}$. We now check (II). It follows from \\ref{a:decrease} that $(h(z_n))_{n \\in \\N}$ is nonincreasing, and hence convergent due to (I). Take an arbitrary $\\bar z$ in $\\Omega$. Then, there exists a subsequence $(z_{k_n})_{n \\in \\N}$ such that $z_{k_n} \\to z$ as $n \\to +\\infty$. Since $h$ is continuous, we obtain that $h(\\bar z) =\\lim_{n \\to +\\infty} h(z_{k_n}) =\\lim_{n\\to +\\infty} h(z_n)$, and so $h$ is constant on $\\Omega$. The proof is complete.\n\\end{proof}\n\n\\begin{proof}\n\\ref{ADFGe:i}: According to Theorem~\\ref{t:subseq}\\ref{t:subseq_simplify}, there exists $N_0$ such that for all $n \\geq N_0$, $w_{n} =0$ and $x_{n+1}=v_{n}-\\lambda_n \\nabla f(u_n)$. Let $n \\geq N_0+1$. \nWe first see that there exists $\\underline{c} >0$ such that\n\\begin{equation}\\label{eq:KL1}\nV(x_{n+1}, x_n,0,\\rho_n, \\sigma_n)+ \\underline{c} \\|x_{n+1}-x_n\\|^2  \\leq V(x_n, x_{n-1},0,\\rho_{n-1},\\sigma_{n-1}).\n\\end{equation}\nIndeed, since $N=0$, it follows from \\eqref{eq:linesearch} that\n\\begin{equation}\\label{eq:KL1'}\nV(x_{n+1}, x_n,0,\\rho_n, \\sigma_n)+ c\\|x_{n+1}-x_n\\|^2  + d \\|\\nabla f(u_n)\\|^2 \\leq V(x_n, x_{n-1},0,\\rho_{n-1},\\sigma_{n-1}).\n\\end{equation}\nBy Step~\\ref{step:init}, $d >0$ if $\\lambda_{\\min} >0$ and $c=0$. Thus, we must have $c >0$ or $d>0$ since $\\lambda_{\\min} >0$. If $c>0$, then \\eqref{eq:KL1} holds with $\\underline{c}:= c$. If $c=0$ and $d >0$, then by Step~\\ref{step:main}, for all $n \\in \\N$, $\\gamma_n =0$. Thus $x_{n+1} =x_n -\\lambda_n \\nabla f(u_n)$, which implies that $\\|\\nabla f(u_n)\\| = \\frac{1}{\\lambda_n}\\|x_{n+1} -x_n\\| \\geq \\frac{1}{\\lambda_{\\max}}\\|x_{n+1} -x_n\\| $. This together with \\eqref{eq:KL1'} implies that \\eqref{eq:KL1} is satisfied with $\\underline{c} := d/\\lambda_{\\max}^2$.\n\nSet $\\mathbf z_n = (x_{n}, x_{n-1}, \\rho_{n-1})$. Noting that $g(x,y,s)=V(x,y,0,s,t)$, we have from \\eqref{eq:linesearch} that\n\\begin{equation}\\label{eq:KL_1}\ng(\\mathbf z_{n+1})+ \\underline{c} \\|x_{n+1}-x_n\\|^2  \\leq g(\\mathbf z_n).\n\\end{equation}\nOn the other hand,\n\\begin{equation*}\n\\nabla g(\\mathbf z_{n}) =(2\\rho_{n-1}(x_{n} -x_{n-1}) + \\nabla f(x_{n}), -2\\rho_{n-1}(x_{n} -x_{n-1}) ,\\|x_{n} -x_{n-1}\\|^2 ).\n\\end{equation*}\nUsing the definition of $\\rho_n$ and the fact that $\\gamma_n \\leq \\bar\\gamma$, $\\nu_n \\geq \\underline{\\nu}$, $L_n \\leq L_{\\max}$, we have\n\\begin{equation*}\n\\rho_n = \\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\gamma_n}{\\nu_n} + \\frac{ L_n\\beta_n +\\gamma_n}{2 }\\leq \\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\bar\\gamma}{\\underline{\\nu}} + \\frac{ L_{\\max}\\bar\\beta +\\bar\\gamma}{2 }=\\bar\\rho.\n\\end{equation*}\nTherefore, by using the sum extension of the natural norm to the product space,\n\\begin{align}\\label{eq:d1}\n\\|\\nabla g(\\mathbf z_{n})) \\|&\\leq \\|2\\rho_{n-1}(x_{n} -x_{n-1}) +\\nabla f(x_{n})\\| +\\|2\\rho_{n-1}(x_{n} -x_{n-1})\\| +\\|x_{n} -x_{n-1}\\|^2 \\notag\\\\\n&\\leq 4 \\bar\\rho\\|x_{n} -x_{n-1}\\| +\\|\\nabla f(x_{n})\\| +\\|x_{n} -x_{n-1}\\|^2.\n\\end{align}\nSince $\\nabla f$ is $L$-Lipschitz continuous and $\\beta_n \\leq \\bar\\beta$,\n\\begin{equation*}\n\\|\\nabla f(u_n) -\\nabla f(x_n)\\| \\leq L \\|u_n -x_n\\| = L\\|\\beta_n(x_n -x_{n-1})\\| \\leq L \\bar\\beta\\|x_n -x_{n-1}\\|.\n\\end{equation*}\nAs $\\lambda_n \\nabla f(u_n) = v_n - x_{n+1} = -(x_{n+1} -x_n) + \\gamma_n (x_n - x_{n-1})$, it follows that\n\\begin{align}\\label{eq:d2}\n\\|\\nabla f(x_n)\\|& \\leq \\|\\nabla f(u_n)\\| + \\|\\nabla f(u_n) -\\nabla f(x_n)\\|\\notag\\\\\n% & \\leq \\frac{1}{\\lambda_n}\\|x_{n+1}-v_n\\| + L \\bar\\beta\\|x_n -x_{n-1}\\| \\notag\\\\\n&\\leq \\frac{1}{\\lambda_{\\min}} (\\|x_{n+1}-x_n\\| + \\bar\\gamma\\|x_n-x_{n-1}\\|)  + L \\bar\\beta\\|x_n -x_{n-1}\\|  \\notag\\\\\n& =  \\frac{1}{\\lambda_{\\min}}\\|x_{n+1}-x_n\\|  + \\left( \\frac{\\bar\\gamma}{\\lambda_{\\min}} +  L \\bar\\beta\\right)\\|x_n - x_{n-1}\\|.\n\\end{align}\nAccording to Theorem~\\ref{t:subseq}\\ref{t:subseq_limit}, $\\lim_{n \\to +\\infty}\\|x_{n}-x_{n-1}\\| =0$, so there exists $N_1 \\geq N_0 +1$ such that, for all $n \\geq N_1$, $\\|x_{n}-x_{n-1}\\| \\leq 1$, which implies that  $\\|x_{n}-x_{n-1}\\|^2 \\leq \\|x_{n}-x_{n-1}\\|$. Combining this with \\eqref{eq:d1} and \\eqref{eq:d2}, there exists $C$ such that, for all $n \\geq N_1$,\n\\begin{equation} \\label{eq:KL2}\n\\| \\nabla g(\\mathbf z_{n}) \\| \\leq C(\\|x_{n+1}-x_{n}\\|+\\|x_{n} -x_{n-1}\\|).\n\\end{equation}\nMoreover, $(\\mathbf z_n)_{n \\in \\N}$ is bounded due to the boundedness of $(x_n)_{n \\in \\N}$ and $(\\rho_n)_{n \\in \\N}$. Combining this with \\eqref{eq:KL_1} and \\eqref{eq:KL2}, we get the conclusion due to Lemma~\\ref{l:KL}. \n\n\\ref{ADFGe:ii}: As $\\nabla f$ is $L$-Lipschitz continuous, for all $ n \\in \\N$,\n\\begin{align*}\n&f(x_{n+1}) -f(x_n) -\\frac{L}{2}\\|x_{n+1}-x_n\\|^2 \\notag\\\\\n&\\leq \\langle \\nabla f(x_n), x_{n+1} -x_n\\rangle  \\notag\\\\\n& = \\langle\\nabla f(x_n) -\\nabla f(u_n), x_{n+1} -x_n\\rangle + \\langle \\nabla f(u_n), x_{n+1} -x_n\\rangle  \\notag\\\\\n& \\leq L\\|x_n -u_n\\|\\|x_{n+1} -x_n\\| + \\frac{1}{\\lambda_n}\\langle v_n -x_{n+1}, x_{n+1} -x_n \\rangle \\notag\\\\\n& = L \\beta_n \\|x_n -x_{n-1}\\|\\|x_{n+1} -x_n\\|- \\frac{1}{\\lambda_n}\\| x_{n+1} -x_{n}\\|^2 + \\frac{\\gamma_n}{\\lambda_n}\\langle x_n -x_{n-1}, x_{n+1} -x_n \\rangle.\n\\end{align*}\nNoting that $\\langle x_n -x_{n-1}, x_{n+1} -x_n \\rangle \\leq \\frac{1}{2}\\left( \\|x_n -x_{n-1}\\|^2 + \\|x_{n+1} -x_n\\|^2 \\right)$, we derive that\n\\begin{equation*}\nf(x_{n+1}) -f(x_n) +\\left(\\frac{1}{\\lambda_n} - \\frac{L \\beta_n}{2} -\\frac{\\gamma_n}{2\\lambda_n}-\\frac{L}{2} \\right)\\|x_{n+1}-x_n\\|^2 \\leq (\\frac{L \\beta_n}{2} +\\frac{\\gamma_n}{2\\lambda_n})\\|x_n - x_{n-1}\\|^2.\n\\end{equation*}\nUsing the definition of $h$, we have\n\\begin{equation}\\label{eq:cvKL2}\nh(x_{n+1}, x_n) + \\hat c \\|x_{n+1}-x_n\\|^2 \\leq h(x_n,x_{n-1}),\n\\end{equation}\nwhere $\\hat c:= \\frac{1}{\\lambda_{\\max} }- L \\bar\\beta -\\frac{\\bar\\gamma}{\\lambda_{\\min}}-\\frac{L}{2} >0$. On the other hand,\n\\begin{equation*}\n\\nabla h(x_n, x_{n-1}) = (\\nabla f(x_{n}) + (L \\bar\\beta +\\frac{\\bar\\gamma}{ \\lambda_{\\min} })(x_n -x_{n-1}),  -(L \\bar\\beta +\\frac{\\bar\\gamma}{ \\lambda_{\\min} })(x_n -x_{n-1})).\n\\end{equation*}\nCombining this with \\eqref{eq:d2}, we obtain that\n\\begin{equation*}\n\\|\\nabla h(x_n, x_{n-1})\\| \\leq  \\frac{1}{\\lambda_{\\min}}\\|x_{n+1}-x_n\\|  + 3\\left( \\frac{\\bar\\gamma}{\\lambda_{\\min}} +  L \\bar\\beta\\right)\\|x_n - x_{n-1}\\|,\n\\end{equation*}\nwhich together with \\eqref{eq:cvKL2} and Lemma~\\ref{l:KL} completes the proof. \n\\end{proof}","units":[{"start":23449,"end":23728,"start_line":352,"end_line":354,"raw_tex":"\\begin{lemma}\\label{l:dry}\nSuppose that $\\phi\\colon \\mathcal{H}\\to \\left(-\\infty, +\\infty\\right]$ satisfies the dry-like friction property with constant $r$. Let $y \\in \\mathcal{H}$ and $w \\in \\prox_{\\tau \\phi}(y)$. Then $\\langle w-y, w\\rangle  + r\\tau \\|w\\| \\leq 0.$\n\\end{lemma}","sha256_utf8":"12d1390a7b6f9fd03b4c1cfbfc685e0bbbf59d1957822c318dc4d807a41af762"},{"start":23729,"end":24140,"start_line":355,"end_line":357,"raw_tex":"\\begin{proof}\nSince $w \\in \\prox_{\\tau \\phi}(y) = \\argmin_{x \\in \\mathcal{H}} \\left(\\phi(x) +\\frac{1}{2 \\tau}\\|x-y\\|^2\\right)$, we have $0 \\in \\partial \\phi(w) + \\frac{1}{\\tau}(w-y)$, which yields $\\frac{1}{\\tau}(y-w) \\in \\partial \\phi(w)$, and so $w \\in \\dom \\partial \\phi$. By the dry-like friction property of $\\phi$, $\\frac{1}{\\tau}\\langle y -w, w\\rangle \\geq r\\|w\\|$, which completes the proof.\n\\end{proof}","sha256_utf8":"ec06408ee31b688bbbdc6212e7e40082beafcaccb57b85a157ef6f8fc6563aaf"},{"start":31797,"end":33154,"start_line":463,"end_line":483,"raw_tex":"\\begin{lemma}\n% [Descent property of the potential function]\n\\label{l:descent}\nSuppose that $f$ is a differentiable function whose gradient is $L$-Lipschitz continuous. Under the setting of AGDNL algorithm, for all $n \\in \\N$, the following hold:\n\\begin{enumerate}\n\\item\\label{l:descent_wn}\n$\\|w_n\\|^2 - \\mu \\alpha_n \\langle w_{n-1}, w_n\\rangle  +\\kappa_n \\langle \\nabla f(u_n), w_n\\rangle  +\\delta \\| w_n\\| \\leq 0$. Consequently, $\\|w_n\\|^2 + (\\delta - k_{\\max} \\|\\nabla f(u_n)\\| - \\mu  \\bar \\alpha\\|w_{n-1}\\|) \\|w_n\\| \\leq 0$. \n\\item\\label{l:descent_Vn}\nIf $L_n \\geq L$, $L_n \\beta_n \\leq L_{n-1} \\beta_{n-1}$, $\\gamma_n \\leq \\gamma_{n-1}$, $\\frac{\\gamma_n}{\\nu_n} \\leq \\frac{\\gamma_{n-1}}{\\nu_{n-1}}$, and $\\frac{\\alpha_n}{\\nu_n} \\leq \\frac{\\alpha_{n-1}}{\\nu_{n-1}}$, then\n\\begin{align*}%\\label{eq:KL1}\n&V(z_{n+1}, x_n, w_n,\\rho_n, \\sigma_n) + \\frac{c_n}{\\nu_n}\\|z_{n+1} -x_n||^2 +  \\frac{d_n}{\\nu_n}\\|\\nabla f(u_n)\\|^2 + \\frac{\\delta}{\\nu_n} \\| w_n\\| \\\\\n&\\leq V(x_n, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}),\n\\end{align*}  \nwhere\n\\begin{align}\nc_n &= \\frac{(1 -\\mu \\bar\\alpha) (1 -2\\gamma_n )}{\\mu^2} - \\frac{(2 L_n\\beta_n +\\gamma_n + L_n)\\nu_n }{2} \\label{d:cn}\\\\\n\\text{and~~}\nd_{n} &= \\frac{2(1- \\mu \\bar\\alpha)(\\lambda_n^2 -\\gamma_n\\lambda_n)-\\mu\\gamma_n\\kappa_n+2\\mu\\lambda_n \\kappa_n}{2\\mu^2} \\label{d:dn}.\n\\end{align}\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"3eac6115b0ea06a96f911c2f731f2c4fd3b01eb2ed345e81fc922d975d805421"},{"start":33259,"end":39560,"start_line":485,"end_line":553,"raw_tex":"\\begin{proof}\n\\ref{l:descent_wn}:\nSet $y_n = \\alpha_n(x_n -v_{n-1}+\\lambda_n\\nabla f(u_{n-1}))-\\theta_n \\frac{\\nabla f(u_n)}{\\|\\nabla f(u_n)\\|}-\\kappa_n \\nabla f(u_n)+ e_n$. By the update of $x_n$, we have $\\mu w_{n-1}= x_n - v_{n-1} +\\lambda_{n-1} \\nabla f(u_{n-1})$, and so\n\\begin{equation*}\ny_n = \\alpha_n\\mu w_{n-1} -\\theta_n \\frac{\\nabla f(u_n)}{\\|\\nabla f(u_n)\\|} -\\kappa_n \\nabla f(u_n)+ e_n.\n\\end{equation*}\nBy Step~\\ref{step:main}\\ref{step:main_comp}, $w_n \\in \\prox_{\\tau_n \\phi}(y_n)$. Since $\\phi$ satisfies the dry-like friction property with constant $r$, we have from Lemma~\\ref{l:dry} that $\\langle w_n-y_n, w_n\\rangle  + r\\tau_n \\|w_n\\| \\leq 0$, which yields\n\\begin{equation*}\n\\|w_n\\|^2 - \\alpha_n \\mu\\langle w_{n-1}, w_n\\rangle  + \\kappa_n \\langle \\nabla f(u_n), w_n\\rangle  + (r\\tau_n - |\\theta_n| - \\|e_n\\|)\\| w_n\\| \\leq 0.    \n\\end{equation*}\nCombining this with $r\\tau_n \\geq |\\theta_n| +\\|e_n\\| +\\delta$, we get the first conclusion, which implies the second one.\n\n\\ref{l:descent_Vn}: Set $\\Delta_n = x_n - x_{n-1}$. It follows from the update of $z_{n+1}$ that\n\\begin{equation*}\n\\mu w_n=z_{n+1} -v_n +\\lambda_n \\nabla f(u_n) \n=(z_{n+1} -x_n)-\\gamma_n \\Delta_n +\\lambda_n \\nabla f(u_n),    \n\\end{equation*}\nand so\n\\begin{align}\\label{eq:57}\n\\mu^2\\|w_n\\|^2&=  \\|z_{n+1} -x_n\\|^2 +\\gamma_n^2 \\|\\Delta_n\\|^2 +\\lambda_n^2 \\|\\nabla f(u_n)\\|^2 - 2\\gamma_n \\langle z_{n+1} -x_n,  \\Delta_n\\rangle \\notag\\\\\n & \\quad +2  \\lambda_n\\langle z_{n+1} -x_n, \\nabla f(u_n)\\rangle -2 \\gamma_n\\lambda_n\\langle  \\Delta_n, \\nabla f(u_n) \\rangle \\notag\\\\\n &\\geq  \\|z_{n+1} -x_n\\|^2 +\\gamma_n^2 \\|\\Delta_n\\|^2 +\\lambda_n^2 \\|\\nabla f(u_n)\\|^2 - \\gamma_n (\\|z_{n+1}-x_n\\|^2 + \\|\\Delta_n\\|^2)  \\notag\\\\\n  & \\quad +2 \\lambda_n\\langle z_{n+1} -x_n, \\nabla f(u_n)\\rangle - \\gamma_n\\lambda_n(\\|\\Delta_n\\|^2 + \\|\\nabla f(u_n)\\|^2) \\notag\\\\\n  &=  (1-\\gamma_n)\\|z_{n+1} -x_n\\|^2 +(\\gamma_n^2-\\gamma_n -\\gamma_n\\lambda_n) \\|\\Delta_n\\|^2 + (\\lambda_n^2 -\\gamma_n\\lambda_n) \\|\\nabla f(u_n)\\|^2 \\notag\\\\\n   & \\quad +2 \\lambda_n\\langle z_{n+1} -x_n, \\nabla f(u_n)\\rangle.\n\\end{align}\nSince $\\nabla f$ is $L$-Lipschitz continuous, $L_n \\geq L$, and $u_n = x_n + \\beta_n(x_n - x_{n-1})$, we have\n\\begin{align}\\label{eq:es2}\n\\langle \\nabla f(u_n), z_{n+1} -x_n \\rangle &= \\langle \\nabla f(u_n) - \\nabla f(x_n), z_{n+1} - x_{n} \\rangle + \\langle \\nabla f(x_n), z_{n+1} -x_n \\rangle \\notag\\\\\n& \\geq -L_n\\|u_n -x_n\\| \\|z_{n+1} -x_n\\| + f(z_{n+1}) -f(x_n) -\\frac{L_n}{2}\\|z_{n+1} - x_n\\|^2 \\notag\\\\\n% & \\geq -\\frac{L_n\\beta_n}{2}(\\|\\Delta_n\\|^2 + \\|z_{n+1}-x_n\\|^2) + f(z_{n+1}) -f(x_n) -\\frac{L_n}{2}\\|z_{n+1} -x_n\\|^2\\notag\\\\\n& \\geq  -\\frac{L_n\\beta_n}{2}\\|\\Delta_n\\|^2 -\\frac{L_n\\beta_n +L_n}{2}\\|z_{n+1} -x_n\\|^2 + f(z_{n+1}) -f(x_n).\n\\end{align}\nCombining this with \\eqref{eq:57}, we derive that\n\\begin{align}\\label{eq:es1}\n &\\|w_n\\|^2 - \\mu \\alpha_n \\langle w_{n-1}, w_n\\rangle+\\kappa_n \\langle \\nabla f(u_n), w_n\\rangle \\notag\\\\\n&\\geq \\|w_n\\|^2 - \\frac{\\mu \\alpha_n}{2}(\\|w_n\\|^2 +\\|w_{n-1}\\|^2)+\\kappa_n \\langle \\nabla f(u_n), w_n\\rangle \\notag\\\\\n& \\geq  -\\frac{\\mu \\alpha_{n}}{2} \\|w_{n-1}\\|^2 +\\frac{\\mu \\alpha_n}{2} \\|w_n\\|^2 +(1-\\mu \\bar\\alpha)\\|w_n\\|^2\\notag\\\\\n&\\quad+\\frac{\\kappa_n }{\\mu} \\langle \\nabla f(u_n), z_{n+1} -x_n \\rangle -\\frac{\\gamma_n\\kappa_n }{\\mu} \\langle \\nabla f(u_n), \\Delta_n \\rangle +\\frac{\\lambda_n \\kappa_n }{\\mu}\\|\\nabla f(u_n)\\|^2\\notag\\\\\n& \\geq  -\\frac{\\mu \\alpha_{n}}{2} \\|w_{n-1}\\|^2  +\\frac{\\mu \\alpha_n}{2} \\|w_n\\|^2 \\notag\\\\\n& \\quad +\\frac{1-\\mu \\bar\\alpha }{\\mu^2}\\left( (1-\\gamma_n)\\|z_{n+1} -x_n\\|^2 +(\\gamma_n^2-\\gamma_n -\\gamma_n\\lambda_n) \\|\\Delta_n\\|^2 + (\\lambda_n^2 -\\gamma_n\\lambda_n) \\|\\nabla f(u_n)\\|^2  \\right)\\notag\\\\\n  & \\quad+ \\frac{2\\lambda_n(1-\\mu \\bar\\alpha)}{\\mu^2}\\langle z_{n+1} -x_n, \\nabla f(u_n)\\rangle +\\frac{\\kappa_n}{\\mu} \\langle \\nabla f(u_n), z_{n+1} -x_n \\rangle  \\notag\\\\\n  &\\quad -\\frac{\\gamma_n \\kappa_n }{2\\mu} (\\|\\nabla f(u_n)\\|^2 +\\|\\Delta_n \\|^2) +\\frac{\\lambda_n\\kappa_n }{\\mu}\\|\\nabla f(u_n)\\|^2\\notag\\\\\n&\\geq -\\frac{\\mu \\alpha_{n}}{2} \\|w_{n-1}\\|^2 +\\frac{\\mu \\alpha_n}{2} \\|w_n\\|^2 +\\bar c_n\\|z_{n+1}-x_n\\|^2 -\\tilde c_n\\|\\Delta_n\\|^2+ d_n\\|\\nabla f(u_n)\\|^2\\notag\\\\\n&\\quad +\\left(\\frac{2\\lambda_n(1-\\mu \\bar\\alpha)}{\\mu^2} +\\frac{\\kappa_n }{\\mu}\\right)( f(z_{n+1}) -f(x_n)), \n\\end{align}\nwhere the last inequality follows from \\eqref{eq:es2}, $d_n$ is defined in \\eqref{d:dn} and\n\\begin{align}\n\\bar c_n &=\\frac{(1-\\mu \\bar\\alpha) (1-\\gamma_n)}{\\mu^2} -\\frac{L_n\\beta_n + L_n}{2 }  \\left(\\frac{2\\lambda_n(1-\\mu \\bar\\alpha)}{\\mu^2}+ \\frac{\\kappa_n}{\\mu}\\right), \\notag\\\\\n\\tilde c_n  &=\\frac{(1 -\\mu \\bar\\alpha)\\left(\\gamma_n +\\gamma_n\\lambda_n+L_n\\lambda_n \\beta_n \\right)  + \\frac{\\mu \\kappa_n}{2}(L_n\\beta_n +\\gamma_n)}{\\mu^2} \\notag\\\\\n&=\\frac{(1 -\\mu \\bar\\alpha)\\gamma_n}{\\mu^2} + \\frac{ L_n\\beta_n +\\gamma_n}{2 } \\left(\\frac{2\\lambda_n(1-\\mu \\bar\\alpha)}{\\mu^2}+ \\frac{\\kappa_n}{\\mu}\\right) =\\rho_n \\nu_n.\\label{d:eta} \n\\end{align}\nIt follows from \\ref{l:descent_wn}, \\eqref{eq:es1}, and the definition of $\\nu_n$ that\n\\begin{align*}\n&\\frac{\\mu \\alpha_n}{2} \\|w_n\\|^2 -\\frac{\\mu \\alpha_n}{2}\\|w_{n-1}\\|^2  +  \\nu_n(f(z_{n+1}) -f(x_n))+ \\bar c_n \\|z_{n+1} -x_n||^2  - \\rho_n \\nu_n\\|\\Delta_{n}\\|^2 \\notag\\\\\n& + d_n\\|\\nabla f(u_n)\\|^2 + \\delta \\| w_n\\|  \\leq 0,\n\\end{align*}\nwhich implies that\n\\begin{align}\\label{eq:ADFGe_4}\n &f(z_{n+1})+ \\rho_n \\|z_{n+1} -x_n||^2 + \\frac{\\mu \\alpha_n}{2\\nu_n} \\|w_n\\|^2 + \\frac{c_n}{\\nu_n}\\|z_{n+1} -x_n||^2 +  \\frac{d_n}{\\nu_n}\\|\\nabla f(u_n)\\|^2 \\notag\\\\\n &\\hspace{3cm} + \\frac{\\delta}{\\nu_n} \\| w_n\\| \\leq f(x_n)+ \\rho_n \\|x_n -x_{n-1}||^2 + \\frac{\\mu \\alpha_n}{2 \\nu_n} \\|w_{n-1}\\|^2.\n  \\end{align}  \nHere, we note that\n\\begin{align*}\n\\bar c_n-\\rho_n \\nu_n&= \\frac{(1 -\\mu \\bar\\alpha) (1 -2\\gamma_n )}{\\mu^2} - \\frac{2 L_n\\beta_n +\\gamma_n + L_n}{2} \\left(\\frac{2\\lambda_n(1-\\mu \\bar\\alpha)}{\\mu^2}+ \\frac{\\kappa_n}{\\mu}\\right) \\\\\n&= \\frac{(1 -\\mu \\bar\\alpha) (1 -2\\gamma_n )}{\\mu^2} - \\frac{(2 L_n\\beta_n +\\gamma_n + L_n)\\nu_n}{2}  =c_n.\n\\end{align*}\nSince $L_n \\beta_n \\leq L_{n-1}\\beta_{n-1}$, $\\gamma_n \\leq \\gamma_{n-1}$, and $\\frac{\\gamma_n}{\\nu_n} \\leq \\frac{\\gamma_{n-1}}{\\nu_{n-1}}$, we have $\\rho_n \\leq \\rho_{n-1}$. As $\\frac{\\alpha_n}{\\nu_n} \\leq \\frac{\\alpha_{n-1}}{\\nu_{n-1}}$, using \\eqref{eq:ADFGe_4} and the definition of $V$ in \\eqref{eq:V}, we get the conclusion. \n\\end{proof}","sha256_utf8":"8553a8406e3dfcf98b78ea1125bfe592a6dfa3df3267b1d5b5a000a34092ee6d"},{"start":39909,"end":40220,"start_line":558,"end_line":560,"raw_tex":"\\begin{lemma}[Well-definedness of the line search criterion]\nSuppose that $f$ is a differentiable function whose gradient is $L$-Lipschitz continuous. Then, for all $n \\in \\N$, the line search criterion \\eqref{eq:linesearch} in AGDNL algorithm is satisfied after a finite number of inner iterations.\n\\end{lemma}","sha256_utf8":"5916578fefe836cfb18959e9519d5fd10dff7eea141998ad24fce4d0d916f568"},{"start":40221,"end":44414,"start_line":561,"end_line":588,"raw_tex":"\\begin{proof}\nFix $n \\in \\mathbb{N}$. Since $\\ell_1 \\in (0, 1)$ and $\\tilde{L}_n > L_{\\min}$, there exists $k_0$ such that $\\ell_1^{-k_0}\\tilde L_n\\geq L_{\\max}$. It also can be seen that there exists $k_1$ such that $\\ell_3^{k_1} \\tilde{\\lambda}_n \\leq \\lambda_{\\min}$ and $\\ell_3^{k_1} \\tilde{\\kappa}_n \\leq \\kappa_{\\min}$ since $\\ell_3 \\in (0,1)$. Suppose that the line search criterion \\eqref{eq:linesearch} is not satisfied after $\\max\\{k_0, k_1\\}$ inner iterations. At the $k$-th inner iteration with $k >\\max\\{k_0, k_1\\}$, by Step~\\ref{step:main}\\ref{step:main_mult},\n\\begin{equation}\\label{eq:kth}\nL_n = L_{\\max}, \\quad \\gamma_n =\\ell_{2}^{k} \\tilde{\\gamma}_n, \\quad \\alpha_n =\\ell_{2}^{k} \\tilde{\\alpha}_n, \\quad \\beta_n =\\ell_{2}^{k} \\tilde{\\beta}_n, \\quad \\lambda_n = \\lambda_{\\min}, \\text{~and~} \\kappa_n =\\kappa_{\\min}.\n\\end{equation}\nThen $\\nu_n = \\frac{2 \\lambda_{\\min}(1 -\\mu \\bar \\alpha)}{\\mu^2}+ \\frac{\\kappa_{\\min} }{\\mu} = \\underline{\\nu}$, and so $\\frac{\\gamma_n}{\\nu_n} =\\frac{\\ell_2^k\\tilde{\\gamma}_n}{\\nu_n} =\\frac{\\ell_2^k\\tilde{\\gamma}_n}{\\underline{\\nu}}$ and $\\frac{\\alpha_n}{\\nu_n} =\\frac{\\ell_2^k\\tilde{\\alpha}_n}{\\nu_n} =\\frac{\\ell_2^k \\tilde{\\alpha}_n}{\\underline{\\nu}}$. Since $0<\\ell_2 <1$, there exists $k_2 >\\max\\{k_0, k_1\\}$ such that after $k_2$ inner iterations, $\\gamma_n \\leq \\gamma_{n-1}$, $L_n \\beta_n = L_{\\max} \\ell_{2}^{k} \\tilde{\\beta}_n  \\leq L_{n-1} \\beta_{n-1}$,  $\\frac{\\gamma_n}{\\nu_n} \\leq \\frac{\\gamma_{n-1}}{\\nu_{n-1}}$, and $\\frac{\\alpha_n}{\\nu_n} \\leq \\frac{\\alpha_{n-1}}{\\nu_{n-1}}$.\nUsing Lemma~\\ref{l:descent} and noting that $\\nu_n =\\underline{\\nu}$, we get\n\\begin{align}\n\\label{eq:lsV}\n&V(z_{n+1}, x_n, w_n,\\rho_n, \\sigma_n) + \\frac{c_n}{\\underline{\\nu}}\\|z_{n+1} -x_n||^2 +  \\frac{d_n}{\\underline{\\nu}}\\|\\nabla f(u_n)\\|^2 + \\frac{\\delta}{\\underline{\\nu}} \\| w_n\\| \\notag \\\\\n&\\leq V(x_n, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}).\n\\end{align}\nIt follows from \\eqref{eq:kth} and the definition of $c_n$ in \\eqref{d:cn} that\n\\begin{align}\\label{eq:cn}\nc_n&= \\frac{(1 -\\mu \\bar\\alpha) (1 -2\\ell_{2}^{k} \\tilde{\\gamma}_n)}{\\mu^2} - \\frac{( 2 \\ell_{2}^{k}L_{\\max} \\tilde{\\beta}_n +\\ell_{2}^{k}\\tilde{\\gamma}_n + L_{\\max})\\underline{\\nu}}{2} \\notag\\\\\n&= \\frac{ 1-\\mu \\bar\\alpha}{\\mu^2}- \\frac{L_{\\max} \\underline{\\nu}}{2}-\\ell_2^k \\left[ \\frac{2\\tilde{\\gamma}_n (1-\\mu \\bar\\alpha)}{\\mu^2}  + \\frac{2L_{\\max}\\tilde{\\beta}_n \\underline{\\nu} +\\tilde{\\gamma}_n\\underline{\\nu}}{2} \\right]\\notag\\\\\n&=\\bar c \\underline{\\nu} -\\ell_2^k \\left[ \\frac{2\\tilde{\\gamma}_n (1-\\mu \\bar\\alpha)}{\\mu^2}  + \\frac{2L_{\\max}\\tilde{\\beta}_n \\underline{\\nu} +\\tilde{\\gamma}_n\\underline{\\nu}}{2} \\right].\n\\end{align}\nSince $c \\leq \\bar c=\\frac{2(1-\\mu \\bar \\alpha) - \\mu^2 L_{\\max} \\underline{\\nu}}{2 \\mu^2 \\underline{\\nu}}$, we consider two cases: $c <\\bar c$ and $c = \\bar c$. If $c <\\bar c$, using \\eqref{eq:cn} and the fact that $0< \\ell_{2} <1$, there exists $k_3$ such that after $k_3$ inner iterations, $c_n \\geq c\\underline{\\nu}$. If $c = \\bar c$, by the choice of $\\gamma_n$, $\\beta_n$ in Step~\\ref{step:init} and Step~\\ref{step:main}, for all $n \\in \\N$, $\\tilde \\gamma_n =0$ and $\\tilde \\beta_n =0$, which together with \\eqref{eq:cn} yields $c_n \\geq c\\underline{\\nu}$.\n\nNow, we consider $d_n$. Using definition of $d_n$ and \\eqref{eq:kth}, we have \n\\begin{align*}\nd_{n}&= \\frac{2(1- \\mu \\bar\\alpha)(\\lambda_n^2 -\\ell_{2}^k \\tilde{\\gamma}_n\\lambda_n)-\\mu \\ell_{2}^{k}\\tilde{\\gamma}_n\\kappa_n+2\\lambda_n \\mu \\kappa_n }{2\\mu^2}\\notag\\\\\n& = \\frac{(1-\\mu \\bar\\alpha)\\lambda_{\\min}^2 + \\lambda_{\\min} \\mu \\kappa_{\\min}}{\\mu^2} - \\ell_2^k\\frac{\\tilde{\\gamma}_n(2(1 -\\mu \\bar\\alpha)\\lambda_{\\min} +\\mu\\kappa_{\\min})}{2\\mu^2} \\notag\\\\\n& =\\bar d \\underline{\\nu} - \\ell_2^k\\frac{\\tilde{\\gamma}_n(2(1 -\\mu \\bar\\alpha)\\lambda_{\\min} +\\mu\\kappa_{\\min})}{2\\mu^2}.\n\\end{align*}\nBy Step~\\ref{step:init} and Step~\\ref{step:main}, $d \\in [0, \\bar d]$ and if $d =\\bar d$, then $\\tilde{\\gamma}_n =0$. Since $0<\\ell_2 <1$, there exists $k_4$ such that after $k_4$ inner iterations, $d_{n} \\geq d \\underline{\\nu}$. Hence, in view of \\eqref{eq:lsV}, after at most $\\max\\{ k_2,k_3, k_4\\}$ inner iterations, the line search criterion \\eqref{eq:linesearch} holds.\n\\end{proof}","sha256_utf8":"437ef9c7f30839e6c8d75008083820abdb9f5f9fe201630bd274b9f6f38b2f2c"},{"start":45326,"end":45647,"start_line":609,"end_line":612,"raw_tex":"\\begin{lemma}\n\\label{l:limit}\nLet $f \\colon \\mathcal{H} \\to \\R$ be a continuous differentiable function. Let $(x_n)_{n \\in \\N}$ and $(u_n)_{n \\in \\N}$ be sequences in $\\mathcal{H}$ such that $(x_n)_{n \\in \\N}$ is bounded and $x_n -u_n \\to 0$ as $n \\to +\\infty$. Then $f(x_n) -f(u_n) \\to 0$ as $n \\to +\\infty$.\n\\end{lemma}","sha256_utf8":"4063f1061928d5689ac57117208bb2ab422fe497f2f0082826407fba3cffc49b"},{"start":45648,"end":46190,"start_line":613,"end_line":615,"raw_tex":"\\begin{proof}\nBy the mean value theorem, $f(x_n) -f(u_n) = \\langle \\nabla f(z_n), x_n -u_n \\rangle$, where $z_n = x_n -\\zeta_n (x_n -u_n)$ for some $\\zeta_n \\in [0,1]$. Then $(z_n)_{n \\in \\N}$ is bounded due to the assumptions on $(x_n)_{n \\in \\N}$ and $(u_n)_{n \\in \\N}$. This together with the continuity of $\\nabla f$ implies the existence of $M$ such that, for all $n \\in \\N$, $\\|\\nabla f(z_n)\\| \\leq M$, which yields $|f(x_n) -f(u_n) | \\leq M\\|x_n -u_n\\|$. Since $x_n -u_n \\to 0$ as $n \\to +\\infty$, we obtain the conclusion.\n\\end{proof}","sha256_utf8":"4e8a577b9f9b5901ad197425fe4777892321892c56dd7e4f5b2be7b30d128f85"},{"start":46193,"end":47319,"start_line":618,"end_line":629,"raw_tex":"\\begin{theorem}[Subsequential convergence and structural simplification]\n\\label{t:subseq}\nLet $(x_n)_{n\\in \\mathbb{N}}$ be the sequence generated by AGDNL algorithm and $\\Omega$ be the set of strong cluster points of $(x_n)_{n \\in \\N}$. Suppose that $f$ is a differentiable function whose gradient is $L$-Lipschitz continuous, that $f$ is bounded below, and that the set $\\{x \\in \\mathcal{H}: f(x) \\leq f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2\\}$ is bounded. Then the following hold:\n\\begin{enumerate}\n\\item\\label{t:subseq_bounded}\n$(x_n)_{n \\in \\N}$ is bounded.\n\\item\\label{t:subseq_limit} \nAs $ n \\to +\\infty$, $\\|x_{n+1}-x_n\\| \\to 0$,  $\\|w_n\\| \\to 0$, $f(x_n) \\to \\zeta$, and \\\\ $V(x_{n}, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}) \\to \\zeta$ for some $\\zeta \\in \\mathbb{R}$. Moreover, for all $\\bar x \\in \\Omega$, $ f(\\bar x) =\\zeta$.  \n\\item\\label{t:subseq_simplify} \nSuppose that $\\lambda_{\\min} >0$. Then $\\lim_{n \\to +\\infty } \\nabla f(x_n) =0$ and for all $\\bar x \\in \\Omega$, $\\nabla f(\\bar x) =0$. Moreover, from some $N_0 \\in \\N$ onwards, $w_n =0$ and AGDNL algorithm becomes AGNL algorithm.\n\\end{enumerate}\n\\end{theorem}","sha256_utf8":"a5b3400b53c79d6ae844f44f1cd66a4ea12617e044199ffee6488ebcfe45befd"},{"start":47320,"end":55603,"start_line":630,"end_line":727,"raw_tex":"\\begin{proof}\n\\ref{t:subseq_bounded}: First, we prove by induction that, for all $n \\in \\N$,\n\\begin{equation}\\label{eq:bx1}\nV(x_{n}, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}) \\leq f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2.\n\\end{equation}\nSince $x_{-1} =x_0$, we have $V(x_{0}, x_{-1}, w_{-1},\\rho_{-1},\\sigma_{-1}) = f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2$, so \\eqref{eq:bx1} holds for $n=0$. Suppose that \\eqref{eq:bx1} holds for all $n \\leq n_0$. %We have $\\sigma_{-1} =0$ since $\\alpha_{-1} =0$.\nFrom \\eqref{eq:linesearch}, we obtain that\n\\begin{align*}\nV(x_{n_0+1}, x_{n_0}, w_{n_0},\\rho_{n_0}, \\sigma_{n_0}) &\\leq \\max_{[n_0-N]_+ \\leq k \\leq n_0}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1})\\notag\\\\\n&\\leq   f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2,\n\\end{align*}\nand hence \\eqref{eq:bx1} is satisfied for $n=n_0+1$, which completes the induction. Next, it follows from \\eqref{eq:bx1} and the definition of $V$ that $f(x_n) \\leq  f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2$. The conclusion now follows from the boundedness of the set $\\{x \\in \\mathcal{H}: f(x) \\leq f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2\\}$.\n\n\\ref{t:subseq_limit}: It follows from the line search criterion \\eqref{eq:linesearch} that\n\\begin{align}\\label{eq:lsx}\n&V(x_{n+1}, x_n, w_n,\\rho_n, \\sigma_n) +c\\|x_{n+1}-x_n\\|^2  + d \\|\\nabla f(u_n)\\|^2+ \\frac{\\delta}{\\underline{\\nu}}\\| w_n\\|\\notag\\\\ &\\leq \\max_{[n-N]_+ \\leq k \\leq n}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1})\\notag\\\\\n&= V(x_{m(n)}, x_{m(n)-1}, w_{m(n)-1},\\rho_{m(n)-1}, \\sigma_{m(n)-1}).\n\\end{align}\nTherefore,\n\\begin{align*}\n&V(x_{m(n+1)}, x_{m(n)}, w_{m(n)},\\rho_{m(n)}, \\sigma_{m(n)}) \\notag\\\\\n&= \\max_{[n+1-N]_+ \\leq k \\leq n+1}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1})\\notag\\\\\n&=\\max \\left\\{V(x_{n+1}, x_n, w_n,\\rho_n, \\sigma_n), \\max_{[n+1-N]_+ \\leq k \\leq n}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1}) \\right\\}\\notag\\\\\n& \\leq  \\max_{[n-N]_+ \\leq k \\leq n}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1}) \\notag\\\\\n&=V(x_{m(n)}, x_{m(n)-1}, w_{m(n)-1},\\rho_{m(n)-1}, \\sigma_{m(n)-1}),\n\\end{align*}\nwhich implies that the sequence $(V(x_{m(n+1)}, x_{m(n)}, w_{m(n)},\\rho_{m(n)}, \\sigma_{m(n)}))_{n\\in \\N}$ is nonincreasing. On the other hand, this sequence is bounded below due to the bounded below property of $f$ and the definition of $V$. Thus, there exists $\\zeta$ such that\n\\begin{equation}\\label{eq:limV}\n\\zeta =\\lim_{n \\to +\\infty}V(x_{m(n)}, x_{m(n)-1}, w_{m(n)-1},\\rho_{m(n)-1}, \\sigma_{m(n)-1}).\n\\end{equation} \n%\\ref{t:subseq_limit}\nIn \\eqref{eq:lsx}, replacing $n$ by $m(n)-1$ yields\n\\begin{align*}\n&V(x_{m(n)}, x_{m(n)-1}, w_{m(n)-1},\\rho_{m(n)-1}, \\sigma_{m(n)-1}) +c\\|x_{m(n)}-x_{m(n)-1}\\|^2 \\notag\\\\ \n&+d \\|\\nabla f(u_{m(n)-1})\\|^2 +\\frac{\\delta}{\\underline{\\nu}}\\| w_{m(n)-1}\\| \\notag\\\\\n&\\leq V(x_{m(m(n)-1)}, x_{m(m(n)-1)-1}, w_{m(m(n)-1)-1},\\rho_{m(m(n)-1)-1}, \\sigma_{m(m(n)-1)-1}).\n\\end{align*}\nLetting $n \\to +\\infty$ and noting that $m(n) \\geq [n-N]_{+}$, we get $ \\lim_{n \\to +\\infty}\\|w_{m(n)-1} \\| = 0$. \nIf $c >0 $, we also have $\\lim_{n \\to +\\infty}\\|x_{m(n)}-x_{m(n)-1}\\|~=~0$. If $c=0$ and $d >0$, then $\\lim_{n \\to +\\infty}\\|\\nabla f(u_{m(n)-1}) \\| = 0$, which implies that \n\\begin{equation*}\n |x_{m(n)} - x_{m(n)-1}\\| =\\|-\\lambda_{m(n)-1}\\nabla f(u_{m(n)-1}) + \\mu w_{m(n)-1} \\| \\to 0 \\text{~as~} n \\to +\\infty   \n\\end{equation*}\nIf $c=d=0$, then $\\lambda_{\\min} =0$ since  $d >0$ if $\\lambda_{\\min} >0$ and $c=0$. By Step~\\ref{step:main}, for all $n \\in \\N$, $\\lambda_n = \\gamma_n =0$, and so $\\lim_{n \\to +\\infty}\\|x_{m(n)}-x_{m(n)-1}\\| =\\lim_{n \\to +\\infty}\\mu \\|w_{m(n)-1} \\| =0$. Therefore, in any cases,\n\\begin{equation}\\label{eq:indu1}\n\\lim_{n \\to +\\infty}\\|x_{m(n)}-x_{m(n)-1}\\| = 0 \\text{~and~} \\lim_{n \\to +\\infty}\\|w_{m(n)-1} \\| = 0.\n\\end{equation}\nBy the definitions of $\\rho_n$ and $\\sigma_n$, for all $n \\in \\N$,\n$0\\leq \\rho_n \\leq  \\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\bar\\gamma}{\\underline{\\nu}} + \\frac{ L_{\\max}\\bar\\beta +\\bar\\gamma}{2 }$ and $ 0\\leq  \\sigma_n \\leq\\frac{\\mu \\alpha_{\\max}}{2 \\underline{\\nu}}.$ Combining with \\eqref{eq:limV} and \\eqref{eq:indu1}, we derive that\n\\begin{align*}\nf(x_{m(n)})&=V(x_{m(n)}, x_{m(n)-1}, w_{m(n)-1},\\rho_{m(n)-1}, \\sigma_{m(n)-1}) \\notag\\\\\n&\\quad - \\rho_{m(n)-1}\\|x_{m(n)}-x_{m(n)-1}\\|^2 - \\sigma_{m(n)-1} \\| w_{m(n)-1}\\|^2\n\\end{align*}\nalso tends to $\\zeta$ as $n \\to +\\infty$. Since $(x_n)_{n\\in \\N}$ is bounded and $\\lim_{n \\to +\\infty}\\|x_{m(n)}-x_{m(n)-1}\\| = 0$, Lemma~\\ref{l:limit} implies that \n\\begin{equation}\\label{eq:indu2}\n\\lim_{n \\to +\\infty}f(x_{m(n)-1})=\\lim_{n \\to +\\infty}f(x_{m(n)}) = \\zeta.\n\\end{equation}\nNow, we prove by induction that for all $k \\in [1, m(n)]$,\n\\begin{equation}\\label{eq:indu}\n\\lim_{n \\to +\\infty}\\|x_{m(n)-k+1}-x_{m(n)-k}\\|=0 , \\lim_{n \\to +\\infty} \\|w_{m(n)-k}\\|=0, \\text{~and~} \\lim_{n \\to +\\infty} f(x_{m(n)-k}) = \\zeta.\n\\end{equation}\nIt follows from \\eqref{eq:indu1} and \\eqref{eq:indu2} that \\eqref{eq:indu} holds for $k=1$. Suppose that \\eqref{eq:indu} holds for $k=k_0$. Replacing $n$ by $ m(n)-k_0-1:= n_m$ in \\eqref{eq:lsx} yields\n\\begin{align*}\n&V(x_{m_n+1}, x_{m_n}, w_{m_n},\\rho_{m_n}, \\sigma_{m_n}) +c\\|x_{m_n+1}-x_{m_n}\\|^2 +d \\|\\nabla f(u_{m_n})\\|^2+\\delta\\| w_{m_n}\\| \\notag\\\\\n&\\leq V(x_{m(m_n)}, x_{m(m_n)-1}, w_{m(m_n)-1},\\rho_{m(m_n)-1},\\sigma_{m(m_n)-1}).\n\\end{align*}\nUsing the definition of $V$, we have\n\\begin{align*}\n&(c+ \\rho_{m_n})\\|x_{m_n+1}-x_{m_n}\\|^2  +(\\delta + \\sigma_{m_n})\\| w_{m_n}\\| +d \\|\\nabla f(u_{m_n})\\|^2 \\notag\\\\\n&\\leq V(x_{m(m_n)}, x_{m(m_n)-1}, w_{m(m_n)-1},\\rho_{m(m_n)-1},\\sigma_{m(m_n)-1}) - f(x_{m_n+1}).\n\\end{align*}\nSimilar to the proof of \\eqref{eq:indu1}, by the induction hypothesis (\\eqref{eq:indu} holds with $k=k_0$) and \\eqref{eq:limV}, we obtain that \\eqref{eq:indu} holds for $k=k_0+1$, which completes the induction.\n\nNext, let $n \\geq N+1$. By the definition of $m(n)$, we have $n -N \\leq m(n) \\leq n$, and so $k_n := m(n) -(n -N) +1 \\in [1, m(n)]$. Then $n-N = m(n) -k_n +1$ and by \\eqref{eq:indu}, \n\\begin{align*}\n&\\lim_{n \\to +\\infty}\\|x_n-x_{n-1}\\| = \\lim_{n \\to +\\infty}\\|x_{m(n)-k_n +1}-x_{m(n)-k_n}\\| =0, \\\\\n&\\lim_{n \\to +\\infty}\\|w_{n}\\|=\\lim_{n \\to +\\infty}\\|w_{n-N-1}\\| =\\lim_{n \\to +\\infty}\\|w_{m(n)-k_n}\\|= 0, \\text{~and}\\\\\n&\\lim_{n \\to +\\infty} f(x_{n}) =\\lim_{n \\to +\\infty} f(x_{n-N-1}) = \\lim_{n \\to +\\infty} f(x_{m(n)-k_n }) = \\zeta.\n\\end{align*}\nCombining this with the definition of $V$ and the boundedness of $\\rho_n$ and $\\sigma_n$, we deduce that\n\\begin{equation}\\label{eq:limitf}\n\\lim_{n \\to +\\infty}V(x_{n}, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}) = \\lim_{n \\to +\\infty}f(x_n) = \\zeta.\n\\end{equation}\nTake an arbitrary $\\bar x \\in \\Omega$. Then there exists a sequence $(x_{k_n})_{n \\in \\N}$ such that $x_{k_n} \\to \\bar x$ as $n \\to +\\infty$. Since $f$ is continuous, we have $\\lim_{n \\to +\\infty}f(x_{k_n}) =f(\\bar x)$. This, together with \\eqref{eq:limitf}, yields that $f(x_n) \\to f(\\bar x)$ and $V(x_{n}, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}) \\to f(\\bar x)$ as $n \\to +\\infty$.\n\n\\ref{t:subseq_simplify}: Since $\\lambda_{n} \\nabla f(u_n) = v_n- x_{n+1} + \\mu w_n = -(x_{n+1} -x_n) + \\gamma_n (x_n -x_{n-1}) + \\mu w_n$, we have that\n\\begin{align*}\n\\lambda_{\\min} \\|\\nabla f(u_n)\\|\\leq \\lambda_n \\|\\nabla f(u_n)\\| \n% &\\leq \\|x_{n+1} -x_n\\| +\\gamma_n\\|x_n -x_{n-1}\\| + \\mu\\|w_n\\| \\notag\\\\\n\\leq \\|x_{n+1} -x_n\\| +\\bar\\gamma\\|x_n -x_{n-1}\\| + \\mu \\|w_n\\|.\n\\end{align*}\nCombining with \\ref{t:subseq_limit} yields $\\lim_{n \\to +\\infty} \\|\\nabla f(u_n)\\|=0$. Since $\\nabla f$ is $L$-Lipschitz continuous, it holds that\n\\begin{equation*}\n0\\leq \\left|\\|\\nabla f(u_n) \\|-\\|\\nabla f(x_n)\\|\\right| \\leq \\|\\nabla f(u_n) -\\nabla f(x_n)\\| \\leq L\\|u_n -x_n\\| = L\\beta_n\\|x_n -x_{n-1}\\|\n\\end{equation*}\nwhich implies that $\\lim_{n \\to +\\infty} \\|\\nabla f(x_n)\\| =0$, and so $\\lim_{n \\to +\\infty} \\|\\nabla f(\\bar x)\\| =0$ any $\\bar x \\in \\Omega$. \n\nIt follows from $\\lim_{n \\to +\\infty}\\|\\nabla f(u_n)\\| =0$ and $\\lim_{n \\to +\\infty}\\|w_n\\| =0$ that there exists $N_0 \\in \\mathbb{N}$ such that, for all $n \\geq N_0$, $\\kappa_{\\max} \\|\\nabla f(u_n)\\| +\\mu \\bar\\alpha \\|w_{n-1}\\| \\leq \\delta$. Combining this with Lemma~\\ref{l:descent}\\ref{l:descent_wn}, we have for all $n \\geq N_0$, $w_n =0$, which yields $ x_{n+1} -v_n +\\lambda_n \\nabla f(u_n)=0$, where $u_n = x_n +\\beta_n(x_n-x_{n-1})$ and $v_n = x_n +\\gamma_n(x_n -x_{n-1})$.\n\\end{proof}","sha256_utf8":"efaf062344bac8b5d31496767499872195f4c29327648d9c13ffc0634fa3d0da"},{"start":62813,"end":64933,"start_line":796,"end_line":819,"raw_tex":"\\begin{lemma}[Abstract convergence under KL property]\n\\label{l:KL}\nLet $\\mathcal{H}$ and $\\mathcal{K}$ be two finite-dimensional real Hilbert\nspaces. Let $h\\colon \\mathcal{K}\\to \\R$ be a continuous function, $(x_n)_{n\\in \\mathbb{N}}$ and $(z_n)_{n\\in \\mathbb{N}}$ be sequences in $\\mathcal{H}$ and $\\mathcal{K}$, respectively, $(\\Delta_n)_{n\\in \\mathbb{N}}$ be a sequence of nonnegative real numbers, and $I$ be a finite set of integer indices with the smallest index not exceeding $1$. Set $\\Delta_k =0$ for $k <0$ and consider the following conditions:\n\\begin{enumerate}\n\\renewcommand\\theenumi{(\\alph{enumi})}\n\\renewcommand{\\labelenumi}{\\rm (\\alph{enumi})}\n\\item\\label{a:decrease}\nThere exists a positive real number $\\alpha$ such that, for all $n \\in \\N$, $h(z_{n+1}) + \\alpha \\Delta_n^2 \\leq  h(z_n)$.\n\\item\\label{a:error}\nThere exist nonnegative real numbers $\\beta_i$, $i\\in I$ such that, for all $n \\in \\N$, $\\dist(0, \\partial h(z_n)) \\leq \\sum_{i\\in I} \\beta_i\\Delta_{n-i}$.\n\\item\\label{a:distance}\nThere exists an integer $j$ such that, for all $n \\in \\N$, $\\|x_{n+1} -x_n\\| \\leq  \\Delta_{n+j}$.\n\\end{enumerate}\nSuppose that \\ref{a:decrease} and \\ref{a:error} hold, that the sequence $(z_n)_{n \\in \\N}$ is bounded with $\\Omega$ being the set of cluster points, and that $h$ satisfies the KL property at any point of $\\Omega$. Then the following hold:\n\\begin{enumerate}\n\\item\nFor all $\\bar z \\in \\Omega$, $0 \\in \\partial h(\\bar z)$ and $h(z_n)\\downarrow \\bar h := h(\\overline{z})$ as $n\\to +\\infty$.\n\\item \nIf \\ref{a:distance} holds, then $\\sum_{n=0}^{+\\infty} \\|x_{n+1} -x_n\\| < +\\infty$ and the sequence $(x_n)_{n \\in \\N}$ converges to a point $\\bar x \\in \\mathcal{H}$.\n\\item \nSuppose further that $h$ satisfies the KL property at any point of $\\Omega$ with exponent $\\alpha \\leq 1/2$. Then there exist $\\Gamma_1 \\in \\R_{++}$ and $\\zeta \\in (0,1)$ such that, for all $n \\in \\N$, $0 \\leq h(z_n) - \\bar h \\leq \\Gamma_1 \\zeta^{n}$. Moreover, if \\ref{a:distance} holds, then there exists $\\Gamma_2 \\in \\R_{++}$ such that, for all $n \\in \\N$, $\\|x_n -\\bar x\\| \\leq \\Gamma_2 \\zeta^{n/2}.$\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"dc144a158bd57eb75535f1758f311cafda401d9b89675e1802d0a866e3cc15fc"},{"start":64934,"end":65770,"start_line":820,"end_line":822,"raw_tex":"\\begin{proof}\nIn view of \\cite[Lemma~5.1 and Theorem~5.2]{BDL21}, we only need to check that (I) there  exist a subsequence $(z_{k_n})_{n \\in \\N}$ and $\\tilde{z}$ such that $z_{k_n} \\to \\tilde{z}$ and $h(z_{k_n}) \\to h(\\tilde{z})$ as $n \\to +\\infty$; and (II) $h$ is constant on $\\Omega$. The first one is due to the continuity of $h$ and the boundedness of $(z_{n})_{n \\in \\N}$. We now check (II). It follows from \\ref{a:decrease} that $(h(z_n))_{n \\in \\N}$ is nonincreasing, and hence convergent due to (I). Take an arbitrary $\\bar z$ in $\\Omega$. Then, there exists a subsequence $(z_{k_n})_{n \\in \\N}$ such that $z_{k_n} \\to z$ as $n \\to +\\infty$. Since $h$ is continuous, we obtain that $h(\\bar z) =\\lim_{n \\to +\\infty} h(z_{k_n}) =\\lim_{n\\to +\\infty} h(z_n)$, and so $h$ is constant on $\\Omega$. The proof is complete.\n\\end{proof}","sha256_utf8":"ba8fdcbdc0a1bc06e7e1113632e92988d0ba38bfab8df9448ddba0517cfbad82"},{"start":67319,"end":73061,"start_line":834,"end_line":905,"raw_tex":"\\begin{proof}\n\\ref{ADFGe:i}: According to Theorem~\\ref{t:subseq}\\ref{t:subseq_simplify}, there exists $N_0$ such that for all $n \\geq N_0$, $w_{n} =0$ and $x_{n+1}=v_{n}-\\lambda_n \\nabla f(u_n)$. Let $n \\geq N_0+1$. \nWe first see that there exists $\\underline{c} >0$ such that\n\\begin{equation}\\label{eq:KL1}\nV(x_{n+1}, x_n,0,\\rho_n, \\sigma_n)+ \\underline{c} \\|x_{n+1}-x_n\\|^2  \\leq V(x_n, x_{n-1},0,\\rho_{n-1},\\sigma_{n-1}).\n\\end{equation}\nIndeed, since $N=0$, it follows from \\eqref{eq:linesearch} that\n\\begin{equation}\\label{eq:KL1'}\nV(x_{n+1}, x_n,0,\\rho_n, \\sigma_n)+ c\\|x_{n+1}-x_n\\|^2  + d \\|\\nabla f(u_n)\\|^2 \\leq V(x_n, x_{n-1},0,\\rho_{n-1},\\sigma_{n-1}).\n\\end{equation}\nBy Step~\\ref{step:init}, $d >0$ if $\\lambda_{\\min} >0$ and $c=0$. Thus, we must have $c >0$ or $d>0$ since $\\lambda_{\\min} >0$. If $c>0$, then \\eqref{eq:KL1} holds with $\\underline{c}:= c$. If $c=0$ and $d >0$, then by Step~\\ref{step:main}, for all $n \\in \\N$, $\\gamma_n =0$. Thus $x_{n+1} =x_n -\\lambda_n \\nabla f(u_n)$, which implies that $\\|\\nabla f(u_n)\\| = \\frac{1}{\\lambda_n}\\|x_{n+1} -x_n\\| \\geq \\frac{1}{\\lambda_{\\max}}\\|x_{n+1} -x_n\\| $. This together with \\eqref{eq:KL1'} implies that \\eqref{eq:KL1} is satisfied with $\\underline{c} := d/\\lambda_{\\max}^2$.\n\nSet $\\mathbf z_n = (x_{n}, x_{n-1}, \\rho_{n-1})$. Noting that $g(x,y,s)=V(x,y,0,s,t)$, we have from \\eqref{eq:linesearch} that\n\\begin{equation}\\label{eq:KL_1}\ng(\\mathbf z_{n+1})+ \\underline{c} \\|x_{n+1}-x_n\\|^2  \\leq g(\\mathbf z_n).\n\\end{equation}\nOn the other hand,\n\\begin{equation*}\n\\nabla g(\\mathbf z_{n}) =(2\\rho_{n-1}(x_{n} -x_{n-1}) + \\nabla f(x_{n}), -2\\rho_{n-1}(x_{n} -x_{n-1}) ,\\|x_{n} -x_{n-1}\\|^2 ).\n\\end{equation*}\nUsing the definition of $\\rho_n$ and the fact that $\\gamma_n \\leq \\bar\\gamma$, $\\nu_n \\geq \\underline{\\nu}$, $L_n \\leq L_{\\max}$, we have\n\\begin{equation*}\n\\rho_n = \\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\gamma_n}{\\nu_n} + \\frac{ L_n\\beta_n +\\gamma_n}{2 }\\leq \\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\bar\\gamma}{\\underline{\\nu}} + \\frac{ L_{\\max}\\bar\\beta +\\bar\\gamma}{2 }=\\bar\\rho.\n\\end{equation*}\nTherefore, by using the sum extension of the natural norm to the product space,\n\\begin{align}\\label{eq:d1}\n\\|\\nabla g(\\mathbf z_{n})) \\|&\\leq \\|2\\rho_{n-1}(x_{n} -x_{n-1}) +\\nabla f(x_{n})\\| +\\|2\\rho_{n-1}(x_{n} -x_{n-1})\\| +\\|x_{n} -x_{n-1}\\|^2 \\notag\\\\\n&\\leq 4 \\bar\\rho\\|x_{n} -x_{n-1}\\| +\\|\\nabla f(x_{n})\\| +\\|x_{n} -x_{n-1}\\|^2.\n\\end{align}\nSince $\\nabla f$ is $L$-Lipschitz continuous and $\\beta_n \\leq \\bar\\beta$,\n\\begin{equation*}\n\\|\\nabla f(u_n) -\\nabla f(x_n)\\| \\leq L \\|u_n -x_n\\| = L\\|\\beta_n(x_n -x_{n-1})\\| \\leq L \\bar\\beta\\|x_n -x_{n-1}\\|.\n\\end{equation*}\nAs $\\lambda_n \\nabla f(u_n) = v_n - x_{n+1} = -(x_{n+1} -x_n) + \\gamma_n (x_n - x_{n-1})$, it follows that\n\\begin{align}\\label{eq:d2}\n\\|\\nabla f(x_n)\\|& \\leq \\|\\nabla f(u_n)\\| + \\|\\nabla f(u_n) -\\nabla f(x_n)\\|\\notag\\\\\n% & \\leq \\frac{1}{\\lambda_n}\\|x_{n+1}-v_n\\| + L \\bar\\beta\\|x_n -x_{n-1}\\| \\notag\\\\\n&\\leq \\frac{1}{\\lambda_{\\min}} (\\|x_{n+1}-x_n\\| + \\bar\\gamma\\|x_n-x_{n-1}\\|)  + L \\bar\\beta\\|x_n -x_{n-1}\\|  \\notag\\\\\n& =  \\frac{1}{\\lambda_{\\min}}\\|x_{n+1}-x_n\\|  + \\left( \\frac{\\bar\\gamma}{\\lambda_{\\min}} +  L \\bar\\beta\\right)\\|x_n - x_{n-1}\\|.\n\\end{align}\nAccording to Theorem~\\ref{t:subseq}\\ref{t:subseq_limit}, $\\lim_{n \\to +\\infty}\\|x_{n}-x_{n-1}\\| =0$, so there exists $N_1 \\geq N_0 +1$ such that, for all $n \\geq N_1$, $\\|x_{n}-x_{n-1}\\| \\leq 1$, which implies that  $\\|x_{n}-x_{n-1}\\|^2 \\leq \\|x_{n}-x_{n-1}\\|$. Combining this with \\eqref{eq:d1} and \\eqref{eq:d2}, there exists $C$ such that, for all $n \\geq N_1$,\n\\begin{equation} \\label{eq:KL2}\n\\| \\nabla g(\\mathbf z_{n}) \\| \\leq C(\\|x_{n+1}-x_{n}\\|+\\|x_{n} -x_{n-1}\\|).\n\\end{equation}\nMoreover, $(\\mathbf z_n)_{n \\in \\N}$ is bounded due to the boundedness of $(x_n)_{n \\in \\N}$ and $(\\rho_n)_{n \\in \\N}$. Combining this with \\eqref{eq:KL_1} and \\eqref{eq:KL2}, we get the conclusion due to Lemma~\\ref{l:KL}. \n\n\\ref{ADFGe:ii}: As $\\nabla f$ is $L$-Lipschitz continuous, for all $ n \\in \\N$,\n\\begin{align*}\n&f(x_{n+1}) -f(x_n) -\\frac{L}{2}\\|x_{n+1}-x_n\\|^2 \\notag\\\\\n&\\leq \\langle \\nabla f(x_n), x_{n+1} -x_n\\rangle  \\notag\\\\\n& = \\langle\\nabla f(x_n) -\\nabla f(u_n), x_{n+1} -x_n\\rangle + \\langle \\nabla f(u_n), x_{n+1} -x_n\\rangle  \\notag\\\\\n& \\leq L\\|x_n -u_n\\|\\|x_{n+1} -x_n\\| + \\frac{1}{\\lambda_n}\\langle v_n -x_{n+1}, x_{n+1} -x_n \\rangle \\notag\\\\\n& = L \\beta_n \\|x_n -x_{n-1}\\|\\|x_{n+1} -x_n\\|- \\frac{1}{\\lambda_n}\\| x_{n+1} -x_{n}\\|^2 + \\frac{\\gamma_n}{\\lambda_n}\\langle x_n -x_{n-1}, x_{n+1} -x_n \\rangle.\n\\end{align*}\nNoting that $\\langle x_n -x_{n-1}, x_{n+1} -x_n \\rangle \\leq \\frac{1}{2}\\left( \\|x_n -x_{n-1}\\|^2 + \\|x_{n+1} -x_n\\|^2 \\right)$, we derive that\n\\begin{equation*}\nf(x_{n+1}) -f(x_n) +\\left(\\frac{1}{\\lambda_n} - \\frac{L \\beta_n}{2} -\\frac{\\gamma_n}{2\\lambda_n}-\\frac{L}{2} \\right)\\|x_{n+1}-x_n\\|^2 \\leq (\\frac{L \\beta_n}{2} +\\frac{\\gamma_n}{2\\lambda_n})\\|x_n - x_{n-1}\\|^2.\n\\end{equation*}\nUsing the definition of $h$, we have\n\\begin{equation}\\label{eq:cvKL2}\nh(x_{n+1}, x_n) + \\hat c \\|x_{n+1}-x_n\\|^2 \\leq h(x_n,x_{n-1}),\n\\end{equation}\nwhere $\\hat c:= \\frac{1}{\\lambda_{\\max} }- L \\bar\\beta -\\frac{\\bar\\gamma}{\\lambda_{\\min}}-\\frac{L}{2} >0$. On the other hand,\n\\begin{equation*}\n\\nabla h(x_n, x_{n-1}) = (\\nabla f(x_{n}) + (L \\bar\\beta +\\frac{\\bar\\gamma}{ \\lambda_{\\min} })(x_n -x_{n-1}),  -(L \\bar\\beta +\\frac{\\bar\\gamma}{ \\lambda_{\\min} })(x_n -x_{n-1})).\n\\end{equation*}\nCombining this with \\eqref{eq:d2}, we obtain that\n\\begin{equation*}\n\\|\\nabla h(x_n, x_{n-1})\\| \\leq  \\frac{1}{\\lambda_{\\min}}\\|x_{n+1}-x_n\\|  + 3\\left( \\frac{\\bar\\gamma}{\\lambda_{\\min}} +  L \\bar\\beta\\right)\\|x_n - x_{n-1}\\|,\n\\end{equation*}\nwhich together with \\eqref{eq:cvKL2} and Lemma~\\ref{l:KL} completes the proof. \n\\end{proof}","sha256_utf8":"56c2bb39f8d6fa0baa8e99da9032805c199644f0d6e508e000016e5887b2302d"}],"unit_separator":"\n\n"},"nodes":[{"id":"S001","semantic_type":"problem_setting","kind":"problem_setting","title":"希尔伯特空间优化问题","source_units":[{"start":6709,"end":6992,"start_line":188,"end_line":191,"raw_tex":"Throughout this paper, $\\mathcal{H}$ is a real Hilbert space equipped with the scalar product $\\langle \\cdot, \\cdot\\rangle$ and the associated norm $\\|\\cdot\\|$. We consider the nonconvex optimization problem\n\\begin{equation}\\label{eq:pb}\n\\min_{x \\in \\mathcal{H}} f(x),\n\\end{equation}","sha256_utf8":"cc091db26d2b71e7fecce9930c31b75c428b3f834d6067b0db968a81802f25c8"}]},{"id":"S002","semantic_type":"definition","kind":"definition","title":"基本记号与次微分","source_units":[{"start":16127,"end":18232,"start_line":261,"end_line":275,"raw_tex":"We denote the set of nonnegative integers by $\\mathbb{N}$, the set of real numbers by $\\mathbb{R}$, the set\nof nonnegative real numbers by $\\mathbb{R}_+$, and the set of the positive real numbers by $\\mathbb{R}_{++}$. For $x\\in \\mathcal{H}$ and $\\delta \\in \\mathbb{R}_+$, the closed ball in $\\mathcal{H}$ centered at $x$ with radius $\\delta$ is $B(x; \\delta) = \\{y \\in \\mathcal{H}: \\|x - y\\| \\leq \\delta\\}$. For a subset $C$ of $\\mathcal{H}$, its interior is $\\inte C =\\{x\\in C: \\exists \\delta \\in \\mathbb{R}_{++},\\ B(x; \\delta) \\subseteq C\\}$.\n\nGiven $f\\colon \\mathcal{H} \\to (-\\infty, +\\infty]$, its \\emph{domain} is denoted by $\\dom f =\\{x \\in \\mathcal{H} \\colon f(x)  < +\\infty\\}$ and its \\emph{epigraph} by $\\epi f =\\{(x,\\xi) \\colon f(x) \\leq \\xi\\}$. We say that $f$ is \\emph{proper} if $\\dom f \\neq \\varnothing$, \\emph{lower semicontinuous} if $\\epi f$ is a closed set, and \\emph{convex} if $\\epi f$ is a convex set. The function $f$ is said to be \\emph{$\\alpha$-weakly convex} ($\\alpha \\in \\mathbb{R}_+$) if $f + \\frac{\\alpha}{2}\\| \\cdot\\|^2$ is convex.\n\nLet $f\\colon \\mathcal{H} \\to (-\\infty, +\\infty]$ be proper and let $x \\in \\dom f$. The \\emph{Fr\\'echet subdifferential} of $f$ at $x$ is defined as\n\\begin{equation*}\n\\hat{\\partial} f(x) = \\left \\{u \\in \\mathcal{H} \\colon \\liminf_{y \\to x}\\frac{f(y) -f(x) -\\langle u, y-x \\rangle}{\\|y-x\\|} \\geq 0\\right \\}\n\\end{equation*}\nand the \\emph{limiting subdifferential} of $f$ at $x$ is defined as\n\\begin{equation*}\n\\partial f(x) = \\{u \\in \\mathcal{H} \\colon \\exists x_n \\to x, f(x_n) \\to f(x) \\text{~and~} \\exists u_n \\in \\hat{\\partial} f(x_n), u_n \\to u \\text{~as~} n \\to +\\infty\\}.\n\\end{equation*}\nFor $x \\notin  \\dom f$, one takes $\\hat{\\partial} f(x)=\\varnothing$ and $\\partial f(x) = \\varnothing$. The domain of $\\partial f$ is $\\dom \\partial f = \\{ x \\in \\mathcal{H}: \\partial f(x) \\neq \\varnothing\\}$. In the case when $f$ is convex, both Fr\\'echet and limiting subdifferentials coincide with the \\emph{convex subdifferential}, that is,\n $\\partial f(x) =\\{u \\in \\mathcal{H}: \\forall y \\in \\mathcal{H}, \\langle u, y-x\\rangle \\leq f(y) -f(x)\\}.$\n","sha256_utf8":"65b54d47d390f92b1c66b7db9b627056635b89d6f73e134bcadac1ea3a921655"}]},{"id":"S003","semantic_type":"definition","kind":"definition","title":"莫罗包络与近端算子","source_units":[{"start":18232,"end":18856,"start_line":276,"end_line":284,"raw_tex":"Let $\\xi \\in \\R_{++}$. The \\emph{Moreau envelope} of $f$ is the mapping $f_{\\xi} \\colon \\mathcal{H} \\to \\left(-\\infty, +\\infty\\right]$ given by\n\\begin{equation*}\nf_{\\xi}(x) = \\inf_{y \\in \\mathcal{H}} \\left(f(y) + \\frac{1}{2\\xi}\\|x-y\\|^2\\right)\n\\end{equation*} \nand the \\emph{proximal operator} of $f$ is the set-valued mapping $\\prox_{\\xi f} \\colon  \\mathcal{H} \\rightrightarrows \\mathcal{H}$ given by\n\\begin{equation*}\n\\prox_{\\xi f}(x) =\\argmin_{y \\in \\mathcal{H}}  \\left(f(y) + \\frac{1}{2\\xi}\\|x-y\\|^2\\right).\n\\end{equation*}\nIn a common abuse of notation, we consider $\\prox_{\\xi f}(x)$ as a point if it is a singleton. \n","sha256_utf8":"c7ae114ab66c99fc1a6d6a9111d17e49d1d81c76952d9dceb59839cbc399fc99"}]},{"id":"S004","semantic_type":"definition","kind":"definition","title":"类干摩擦性质","source_units":[{"start":19294,"end":19841,"start_line":291,"end_line":294,"raw_tex":"\\begin{definition}[Dry-like friction property]\n\\label{d:DLF}\nWe say that a proper function $\\phi\\colon \\mathcal{H} \\to \\left(-\\infty, +\\infty\\right]$ satisfies the \\emph{dry-like friction property with constant $r \\in \\mathbb{R}_{++}$} if for all $x \\in \\dom \\partial \\phi$, $\\inf_{u \\in \\partial \\phi(x)} \\langle u, x\\rangle \\geq r \\|x\\|$. The function $\\phi$ is said to satisfy the \\emph{dry-like friction property} or called a \\emph{dry-like friction function} if it satisfies dry-like friction property with some constant $r$.\n\\end{definition}","sha256_utf8":"67b9ac53a347e240c5c1f633db671a858f11a269526c5366cb8302fd548aaa65"}]},{"id":"S005","semantic_type":"assumption","kind":"assumption","title":"目标函数光滑性","source_units":[{"start":24218,"end":24754,"start_line":363,"end_line":363,"raw_tex":"In this section, we assume that $f: \\mathcal{H} \\to \\R$ is a differentiable function whose gradient is $L$-Lipschitz continuous. Inspired by the works using dry friction in \\cite{AA20,AA21, AA22, AAL21, ABC21} and the works using nonmonotone line search in \\cite{Dai02,  GZLHY13,GLL86, WNF09,  Yan21, ZH04} to enlarge the choices of the parameters, we propose the AGDNL for solving problem \\eqref{eq:pb} with the potential function $V \\colon \\mathcal{H} \\times \\mathcal{H} \\times \\mathcal{H} \\times \\R_+ \\times \\R_+ \\to \\R $ defined by\n","sha256_utf8":"beca2113f00c813276767ea8643d6bd81c42e94909f871cfab1c06d3f1af423e"}]},{"id":"S006","semantic_type":"definition","kind":"definition","title":"势函数定义","source_units":[{"start":24754,"end":24852,"start_line":364,"end_line":366,"raw_tex":"\\begin{align}\\label{eq:V}\nV(x,y,w,\\rho,\\sigma)= f(x) +\\rho \\|x -y\\|^2 +\\sigma \\|w\\|^2.\n\\end{align}","sha256_utf8":"87c74365f72e2641a09aa4ecd6490d4d59abcf0aa65d90f89748bef87d2f8993"}]},{"id":"S007","semantic_type":"algorithm","kind":"algorithm","title":"AGDNL算法与参数","source_units":[{"start":25323,"end":29631,"start_line":373,"end_line":421,"raw_tex":"\\begin{algo}[AGDNL]\n\\label{algo:AGDNL}\n\\step{}\\label{step:init}\nLet $\\phi\\colon \\mathcal{H}\\to \\left(-\\infty, +\\infty\\right]$ satisfy the dry-like friction property with constant $r$. \nLet $u_{-1}=x_{-1} =x_0 \\in \\mathcal{H}$. \nLet $\\mu \\in \\R_{++}$, $\\bar\\alpha =\\alpha_{-1} \\in (0, \\frac{1}{\\mu})$,  $\\bar\\beta =\\beta_{-1} \\in \\R_{+}$, and $\\bar\\gamma =\\gamma_{-1} \\in \\R_{+}$. \nLet $L_{\\min} \\in (0, L]$, and $L_{\\max} \\in [L, +\\infty)$. Let $\\lambda_{\\min} \\in [0,\\frac{1}{L_{\\max}})$, $\\lambda_{\\max} \\in [\\lambda_{\\min}, +\\infty)$, $\\kappa_{\\min} \\in (0, \\bar\\kappa]$, where $\\bar\\kappa = \\frac{2(1-\\mu \\bar\\alpha)}{\\mu L_{\\max}} -\\frac{2(1-\\mu \\bar\\alpha)\\lambda_{\\min}}{\\mu}$, and $\\kappa_{\\max} \\in [\\kappa_{\\min}, +\\infty)$. Set \n\\begin{equation*}\n\\bar c =\\frac{2(1-\\mu \\bar \\alpha) - \\mu^2 L_{\\max} \\underline{\\nu}}{2 \\mu^2 \\underline{\\nu}} \n\\text{~~and~~} \\bar d =\\frac{(1-\\mu \\bar\\alpha)\\lambda_{\\min}^2 + \\lambda_{\\min} \\mu \\kappa_{\\min}}{\\mu^2 \\underline{\\nu}},    \n\\end{equation*}\nwhere $\\underline{\\nu}=\\frac{2 \\lambda_{\\min}(1 -\\mu \\bar\\alpha)}{\\mu^2}+ \\frac{\\kappa_{\\min}}{\\mu}$. Let $c \\in [0, \\bar c ]$ and $ d \\in [0, \\bar d]$ such that $d >0$ if $\\lambda_{\\min} >0$ and $c=0$. Let $L_{-1} \\in [L_{\\min}, L_{\\max}]$, $\\lambda_{-1} \\in [\\lambda_{\\min}, \\lambda_{\\max}]$, $\\kappa_{-1} \\in [\\kappa_{\\min}, \\kappa_{\\max}]$, and set $\\rho_{-1} = \\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\gamma_{-1}}{\\nu_{-1}} + \\frac{L_{-1}\\beta_{-1} +\\gamma_{-1}}{2}$ and $\\sigma_{-1} =\\frac{\\mu \\alpha_{-1}}{2\\nu_{-1}}$, where $\\nu_{-1} = \\frac{2\\lambda_{-1}(1 -\\mu \\bar\\alpha)}{\\mu^2} +\\frac{\\kappa_{-1}}{\\mu}$. \nSet also $w_{-1} =\\frac{\\lambda_{-1} \\nabla f(u_{-1})}{\\mu}$. Let $\\delta \\in \\R_{++}$ and $\\ell_1, \\ell_2, \\ell_3 \\in (0,1)$. \nLet $N \\in \\N$ and set $n=0$.\n%$d \\in (0,\\frac{(1- \\mu \\alpha)\\lambda_{\\min}^2}{\\mu^2 \\ell L_{\\max}} )$ . \n\n\\step{}\\label{step:main}\nLet $e_n \\in \\mathcal{H}$, $\\theta_n \\in \\R$, and $\\tau_n\\in \\mathbb{R}_+$ such that $r\\tau_n \\geq |\\theta_n| +\\|e_n\\| +\\delta$. \nChoose $\\tilde{L}_n \\in [L_{\\min}, L_{\\max}] $, $\\tilde{\\alpha}_n \\in (0, \\bar\\alpha]$, and $\\tilde{\\kappa}_n \\in [\\kappa_{\\min}, \\kappa_{\\max}]$. Choose $\\tilde{\\beta}_n \\in [0, \\bar\\beta]$ such that $\\tilde{\\beta}_n  = 0$ if $c = \\bar c$. %$\\kappa_{\\min} =\\bar \\kappa$. \nChoose $\\tilde{\\gamma}_n \\in [0, \\bar\\gamma]$ such that $\\tilde{\\gamma}_n =0$ if $d=\\bar d$ or $c=\\bar c$ or $c=0$.\nChoose $\\tilde{\\lambda}_n \\in [\\lambda_{\\min}, \\lambda_{\\max}]$ such that $\\tilde{\\lambda}_n  =0$ if $\\lambda_{\\min} =0$. Set $\\beta_n =\\tilde{\\beta}_n$, $\\gamma_n =\\tilde{\\gamma}_n$, $L_n =\\tilde{L}_n$, $\\alpha_n =\\tilde{\\alpha}_n$, $\\lambda_n =\\tilde{\\lambda}_n$, and $\\kappa_n = \\tilde{\\kappa}_n$.\n\\begin{enumerate}[leftmargin=2em]\n\\item\\label{step:main_comp}\nCompute \n\\begin{equation*}\n\\small\n\\begin{cases}\nu_n = x_n + \\beta_n(x_n - x_{n-1}),\\\\ \n%z_n &= x_n + \\nu_n(x_n - x_{n-1}),\\\\\nv_n = x_n + \\gamma_n (x_n -x_{n-1}),\\\\\nw_n \\in \\prox_{\\tau_n \\phi}\\left(\\alpha_n(x_n -v_{n-1}+\\lambda_{n-1}\\nabla f(u_{n-1}))-\\theta_n \\frac{\\nabla f(u_n)}{\\|\\nabla f(u_n)\\|} -\\kappa_n \\nabla f(u_n)+ e_n\\right ),\\\\\nz_{n+1} = v_n- \\lambda_n\\nabla f(u_n) +\\mu w_n.\n\\end{cases}\n\\end{equation*}\n\\item \n\\label{step:main_line}\nSet $\\rho_n = \\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\gamma_n}{\\nu_n} + \\frac{ L_n\\beta_n +\\gamma_n}{2 }$ and $\\sigma_n =\\frac{\\mu \\alpha_n}{2 \\nu_n}$, where $\\nu_n = \\frac{2 \\lambda_n(1 -\\mu \\bar\\alpha)}{\\mu^2}+ \\frac{\\kappa_n }{\\mu}$. If\n\\begin{align}\\label{eq:linesearch}\nV(z_{n+1}, x_n, w_n,\\rho_n, \\sigma_n) +c \\|z_{n+1}-x_n\\|^2 + d \\|\\nabla f(u_n)\\|^2 +\\frac{\\delta}{\\underline{\\nu}}\\| w_n\\|\\notag\\\\\n\\leq \\max_{[n-N]_+ \\leq k \\leq n}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1}),\n\\end{align}\nthen go to Step~\\ref{step:last}.\n\\item\\label{step:main_mult}\nSet $L_n \\leftarrow \\min\\{L_n/\\ell_1, L_{\\max}\\}$, $ \\beta_n \\leftarrow \\ell_{2}\\beta_n $,  $ \\gamma_n \\leftarrow \\ell_{2}\\gamma_n $, $\\alpha_n \\leftarrow \\ell_{2}\\alpha_n $, $\\lambda_n \\leftarrow \\max\\{\\ell_3 \\lambda_n, \\lambda_{\\min}\\}$, $\\kappa_n \\leftarrow \\max\\{\\ell_3 \\kappa_n, \\kappa_{\\min}\\}$, and go to Step~\\ref{step:main}\\ref{step:main_comp}. \n\\end{enumerate}\n\n\\step{}\\label{step:last}\nSet $x_{n+1} \\leftarrow z_{n+1}$. If a termination criterion is not met, then let $n\\leftarrow n+1$ and go to Step~\\ref{step:main}.\n\\end{algo}","sha256_utf8":"a8cabf2652dba5f8774e30d572a9838216a595ad46069d6c84c5074190867a82"}]},{"id":"S008","semantic_type":"algorithm","kind":"algorithm","title":"AGNL简化算法","source_units":[{"start":44602,"end":44797,"start_line":592,"end_line":598,"raw_tex":"\\begin{equation}\\label{AGNL}\n\\begin{cases}\nu_n &= x_n + \\beta_n(x_n - x_{n-1}),\\\\ \nv_n & = x_n + \\gamma_n (x_n -x_{n-1}),\\\\\nz_{n+1}\\!\\!\\! &= v_n- \\lambda_n\\nabla f(u_n)\n\\end{cases}\n\\end{equation}","sha256_utf8":"34bf2da9693fce588956e204b199236c24edef0cd3cd77398534afacb5d0b8e6"},{"start":44836,"end":45048,"start_line":600,"end_line":604,"raw_tex":"\\begin{align}\\label{AGNL_ls}\nV(z_{n+1}, x_n, 0,\\rho_n, \\sigma_n) +c \\|z_{n+1}-x_n\\|^2 + d \\|\\nabla f(u_n)\\|^2\n\\notag\\\\\n\\leq \\max_{[n-N]_+ \\leq k \\leq n}V(x_k, x_{k-1}, 0,\\rho_{k-1}, \\sigma_{k-1}).    \n\\end{align}","sha256_utf8":"cce79a2a3ede2edb3e500b868897e0598cef9fab432b93428e5515f07a5c4aeb"},{"start":45049,"end":45145,"start_line":605,"end_line":605,"raw_tex":"We call this version as the \\emph{fast gradient algorithm with nonmonotone line search} (AGNL).\n","sha256_utf8":"66cddcdaf9ed3dd1826f7f163f0e54e67c41ac29f06c4344c5330098c98281f9"}]},{"id":"S009","semantic_type":"notation","kind":"notation","title":"窗口极大值索引","source_units":[{"start":45146,"end":45325,"start_line":607,"end_line":607,"raw_tex":"For each $n \\in \\N$, set $m(n)\\in \\argmax_{[n-N]_+ \\leq k \\leq n}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1})$. The following technical lemma will be of use in our analysis.\n","sha256_utf8":"f4bd67b909c36934cc207133929f352fbbcdd5337da3260956be038bc303d4a5"}]},{"id":"S010","semantic_type":"definition","kind":"definition","title":"KL性质与指数","source_units":[{"start":61103,"end":62812,"start_line":784,"end_line":794,"raw_tex":"We now recall the Kurdyka--{\\L}ojasiewicz (KL) property \\cite{Kur98, Loj63} which plays an important role in our convergence analysis. Let $h: \\mathcal{H} \\to (-\\infty, +\\infty]$ be a proper lower semicontinuous function. We say that $h$ satisfies the \\emph{KL property} at $\\bar x \\in \\dom \\partial h$ if there exist $\\varepsilon \\in \\mathbb{R}_{++}$, $\\nu \\in (0, +\\infty]$, and a continuous concave function $\\varphi: [0, \\nu) \\to \\R_+$ such that $\\varphi(0) =0$ and $\\varphi$ is continuously differentiable on  $(0, \\nu)$ with $\\varphi' >0$, and for all $x \\in B(\\bar x; \\varepsilon)$ with $h(\\bar x) < h(x) < h (\\bar x) + \\nu$, one has\n\\begin{equation*}\n\\varphi'(h(x)- h(\\bar x))\\dist(0, \\partial h(x)) \\geq 1.\n\\end{equation*}\nIf $h$ satisfies the KL property at each point in $\\dom \\partial h$, then $h$ is called a \\emph{KL function}.\n\nThe function $h$ is said to satisfy the \\emph{KL property at $\\bar x$ with exponent $\\theta$} if it satisfies the KL property at $\\bar x \\in \\dom \\partial h$ with corresponding function $\\varphi(s) = \\gamma s^{1-\\theta}$ for some $\\gamma \\in \\R_{++}$ and $\\theta \\in [0,1)$, i.e., there exist $c \\in \\mathbb{R}_{++}$, $\\varepsilon \\in \\mathbb{R}_{++}$, and $\\nu \\in (0, +\\infty]$ such that\n\\begin{equation*}\n\\dist(0, \\partial h(x)) \\geq c (h(x) -h(\\bar x))^{\\theta}\n\\end{equation*}\nwhenever $x \\in B(\\bar x; \\varepsilon)$ with $h(\\bar x) < h(x) < h(\\bar x) + \\nu$. If $h$ is a KL function and has the same exponent $\\theta$ at any $\\bar x \\in \\dom \\partial h$, then $h$ is called a \\emph{KL function with exponent $\\theta$}. Some typical KL functions include strongly convex functions, real analytic functions, and semialgebraic functions, see \\cite{ABRS10, Loj63}.\n","sha256_utf8":"a573c549fcce842d073ec04b90b0ed42f9998341c04f9bb004563bc3e54de0bf"}]},{"id":"S011","semantic_type":"theorem","kind":"theorem","title":"全序列收敛与线性速率","source_units":[{"start":65773,"end":67318,"start_line":825,"end_line":833,"raw_tex":"\\begin{theorem}[Sequential and linear convergence]\n\\label{t:fullseq}\nSuppose that $\\mathcal{H}$ is finite dimensional, that $f$ is a differentiable function whose gradient is $L$-Lipschitz continuous, and that $f$ is bounded below. Let $(x_n)_{n\\in \\mathbb{N}}$ be the sequence generated by AGDNL algorithm with $\\lambda_{\\min} >0$. Suppose further that the set $\\{x \\in \\mathcal{H}: f(x) \\leq f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2\\}$ is bounded, and that one of the following holds\n\\begin{enumerate}\n\\item \\label{ADFGe:i}  $N=0$ and $ g(x,y,\\rho):= f(x)+ \\rho  \\|x -y\\|^2 $ satisfies the KL property at $(\\bar x, \\bar x, \\rho)$ for all $\\bar x \\in \\mathcal{H}$ and $\\rho \\in [0, \\bar\\rho]$, where $\\bar\\rho: =\\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\bar\\gamma}{\\underline{\\nu}} + \\frac{ L_{\\max}\\bar\\beta +\\bar\\gamma}{2 }$.\n\\item \\label{ADFGe:ii} $\\frac{1}{\\lambda_{\\max} }- L \\bar\\beta -\\frac{\\bar\\gamma}{\\lambda_{\\min}}-\\frac{L}{2} >0$ and $h(x,y) =f(x) +(\\frac{L \\bar\\beta}{2} +\\frac{\\bar\\gamma}{2 \\lambda_{\\min} }) \\|x-y\\|^2$ satisfies the KL property at $(\\bar x, \\bar x)$  for all $\\bar x \\in \\mathcal{H}$.\n\\end{enumerate}\nThen $\\sum_{k=0}^{+\\infty} \\|x_{n+1} -x_n\\| < +\\infty$ and $(x_n)_{n\\in \\N}$ converges to a critical point $x^*$ of $f$. Moreover, if the KL property in \\ref{ADFGe:i} or \\ref{ADFGe:ii} is satisfied with exponent $\\theta \\leq 1/2$, then there exist $\\Gamma \\in \\R_{++}$ and $\\zeta \\in (0,1)$ such that, for all $n \\in \\N$, $\\|x_n -x^*\\| \\leq \\Gamma{\\zeta}^{n/2}$ and $|f(x_n) -f(x^*) |\\leq \\Gamma{\\zeta}^n$.\n\\end{theorem}","sha256_utf8":"f2cdbf737f13ed8c367bb81c8730a2eb70eeb4f0aa481c2a9b7a1db4e15ea0d4"}]},{"id":"S012","semantic_type":"proof","kind":"proof","title":"主定理完整证明","source_units":[{"start":67319,"end":73061,"start_line":834,"end_line":905,"raw_tex":"\\begin{proof}\n\\ref{ADFGe:i}: According to Theorem~\\ref{t:subseq}\\ref{t:subseq_simplify}, there exists $N_0$ such that for all $n \\geq N_0$, $w_{n} =0$ and $x_{n+1}=v_{n}-\\lambda_n \\nabla f(u_n)$. Let $n \\geq N_0+1$. \nWe first see that there exists $\\underline{c} >0$ such that\n\\begin{equation}\\label{eq:KL1}\nV(x_{n+1}, x_n,0,\\rho_n, \\sigma_n)+ \\underline{c} \\|x_{n+1}-x_n\\|^2  \\leq V(x_n, x_{n-1},0,\\rho_{n-1},\\sigma_{n-1}).\n\\end{equation}\nIndeed, since $N=0$, it follows from \\eqref{eq:linesearch} that\n\\begin{equation}\\label{eq:KL1'}\nV(x_{n+1}, x_n,0,\\rho_n, \\sigma_n)+ c\\|x_{n+1}-x_n\\|^2  + d \\|\\nabla f(u_n)\\|^2 \\leq V(x_n, x_{n-1},0,\\rho_{n-1},\\sigma_{n-1}).\n\\end{equation}\nBy Step~\\ref{step:init}, $d >0$ if $\\lambda_{\\min} >0$ and $c=0$. Thus, we must have $c >0$ or $d>0$ since $\\lambda_{\\min} >0$. If $c>0$, then \\eqref{eq:KL1} holds with $\\underline{c}:= c$. If $c=0$ and $d >0$, then by Step~\\ref{step:main}, for all $n \\in \\N$, $\\gamma_n =0$. Thus $x_{n+1} =x_n -\\lambda_n \\nabla f(u_n)$, which implies that $\\|\\nabla f(u_n)\\| = \\frac{1}{\\lambda_n}\\|x_{n+1} -x_n\\| \\geq \\frac{1}{\\lambda_{\\max}}\\|x_{n+1} -x_n\\| $. This together with \\eqref{eq:KL1'} implies that \\eqref{eq:KL1} is satisfied with $\\underline{c} := d/\\lambda_{\\max}^2$.\n\nSet $\\mathbf z_n = (x_{n}, x_{n-1}, \\rho_{n-1})$. Noting that $g(x,y,s)=V(x,y,0,s,t)$, we have from \\eqref{eq:linesearch} that\n\\begin{equation}\\label{eq:KL_1}\ng(\\mathbf z_{n+1})+ \\underline{c} \\|x_{n+1}-x_n\\|^2  \\leq g(\\mathbf z_n).\n\\end{equation}\nOn the other hand,\n\\begin{equation*}\n\\nabla g(\\mathbf z_{n}) =(2\\rho_{n-1}(x_{n} -x_{n-1}) + \\nabla f(x_{n}), -2\\rho_{n-1}(x_{n} -x_{n-1}) ,\\|x_{n} -x_{n-1}\\|^2 ).\n\\end{equation*}\nUsing the definition of $\\rho_n$ and the fact that $\\gamma_n \\leq \\bar\\gamma$, $\\nu_n \\geq \\underline{\\nu}$, $L_n \\leq L_{\\max}$, we have\n\\begin{equation*}\n\\rho_n = \\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\gamma_n}{\\nu_n} + \\frac{ L_n\\beta_n +\\gamma_n}{2 }\\leq \\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\bar\\gamma}{\\underline{\\nu}} + \\frac{ L_{\\max}\\bar\\beta +\\bar\\gamma}{2 }=\\bar\\rho.\n\\end{equation*}\nTherefore, by using the sum extension of the natural norm to the product space,\n\\begin{align}\\label{eq:d1}\n\\|\\nabla g(\\mathbf z_{n})) \\|&\\leq \\|2\\rho_{n-1}(x_{n} -x_{n-1}) +\\nabla f(x_{n})\\| +\\|2\\rho_{n-1}(x_{n} -x_{n-1})\\| +\\|x_{n} -x_{n-1}\\|^2 \\notag\\\\\n&\\leq 4 \\bar\\rho\\|x_{n} -x_{n-1}\\| +\\|\\nabla f(x_{n})\\| +\\|x_{n} -x_{n-1}\\|^2.\n\\end{align}\nSince $\\nabla f$ is $L$-Lipschitz continuous and $\\beta_n \\leq \\bar\\beta$,\n\\begin{equation*}\n\\|\\nabla f(u_n) -\\nabla f(x_n)\\| \\leq L \\|u_n -x_n\\| = L\\|\\beta_n(x_n -x_{n-1})\\| \\leq L \\bar\\beta\\|x_n -x_{n-1}\\|.\n\\end{equation*}\nAs $\\lambda_n \\nabla f(u_n) = v_n - x_{n+1} = -(x_{n+1} -x_n) + \\gamma_n (x_n - x_{n-1})$, it follows that\n\\begin{align}\\label{eq:d2}\n\\|\\nabla f(x_n)\\|& \\leq \\|\\nabla f(u_n)\\| + \\|\\nabla f(u_n) -\\nabla f(x_n)\\|\\notag\\\\\n% & \\leq \\frac{1}{\\lambda_n}\\|x_{n+1}-v_n\\| + L \\bar\\beta\\|x_n -x_{n-1}\\| \\notag\\\\\n&\\leq \\frac{1}{\\lambda_{\\min}} (\\|x_{n+1}-x_n\\| + \\bar\\gamma\\|x_n-x_{n-1}\\|)  + L \\bar\\beta\\|x_n -x_{n-1}\\|  \\notag\\\\\n& =  \\frac{1}{\\lambda_{\\min}}\\|x_{n+1}-x_n\\|  + \\left( \\frac{\\bar\\gamma}{\\lambda_{\\min}} +  L \\bar\\beta\\right)\\|x_n - x_{n-1}\\|.\n\\end{align}\nAccording to Theorem~\\ref{t:subseq}\\ref{t:subseq_limit}, $\\lim_{n \\to +\\infty}\\|x_{n}-x_{n-1}\\| =0$, so there exists $N_1 \\geq N_0 +1$ such that, for all $n \\geq N_1$, $\\|x_{n}-x_{n-1}\\| \\leq 1$, which implies that  $\\|x_{n}-x_{n-1}\\|^2 \\leq \\|x_{n}-x_{n-1}\\|$. Combining this with \\eqref{eq:d1} and \\eqref{eq:d2}, there exists $C$ such that, for all $n \\geq N_1$,\n\\begin{equation} \\label{eq:KL2}\n\\| \\nabla g(\\mathbf z_{n}) \\| \\leq C(\\|x_{n+1}-x_{n}\\|+\\|x_{n} -x_{n-1}\\|).\n\\end{equation}\nMoreover, $(\\mathbf z_n)_{n \\in \\N}$ is bounded due to the boundedness of $(x_n)_{n \\in \\N}$ and $(\\rho_n)_{n \\in \\N}$. Combining this with \\eqref{eq:KL_1} and \\eqref{eq:KL2}, we get the conclusion due to Lemma~\\ref{l:KL}. \n\n\\ref{ADFGe:ii}: As $\\nabla f$ is $L$-Lipschitz continuous, for all $ n \\in \\N$,\n\\begin{align*}\n&f(x_{n+1}) -f(x_n) -\\frac{L}{2}\\|x_{n+1}-x_n\\|^2 \\notag\\\\\n&\\leq \\langle \\nabla f(x_n), x_{n+1} -x_n\\rangle  \\notag\\\\\n& = \\langle\\nabla f(x_n) -\\nabla f(u_n), x_{n+1} -x_n\\rangle + \\langle \\nabla f(u_n), x_{n+1} -x_n\\rangle  \\notag\\\\\n& \\leq L\\|x_n -u_n\\|\\|x_{n+1} -x_n\\| + \\frac{1}{\\lambda_n}\\langle v_n -x_{n+1}, x_{n+1} -x_n \\rangle \\notag\\\\\n& = L \\beta_n \\|x_n -x_{n-1}\\|\\|x_{n+1} -x_n\\|- \\frac{1}{\\lambda_n}\\| x_{n+1} -x_{n}\\|^2 + \\frac{\\gamma_n}{\\lambda_n}\\langle x_n -x_{n-1}, x_{n+1} -x_n \\rangle.\n\\end{align*}\nNoting that $\\langle x_n -x_{n-1}, x_{n+1} -x_n \\rangle \\leq \\frac{1}{2}\\left( \\|x_n -x_{n-1}\\|^2 + \\|x_{n+1} -x_n\\|^2 \\right)$, we derive that\n\\begin{equation*}\nf(x_{n+1}) -f(x_n) +\\left(\\frac{1}{\\lambda_n} - \\frac{L \\beta_n}{2} -\\frac{\\gamma_n}{2\\lambda_n}-\\frac{L}{2} \\right)\\|x_{n+1}-x_n\\|^2 \\leq (\\frac{L \\beta_n}{2} +\\frac{\\gamma_n}{2\\lambda_n})\\|x_n - x_{n-1}\\|^2.\n\\end{equation*}\nUsing the definition of $h$, we have\n\\begin{equation}\\label{eq:cvKL2}\nh(x_{n+1}, x_n) + \\hat c \\|x_{n+1}-x_n\\|^2 \\leq h(x_n,x_{n-1}),\n\\end{equation}\nwhere $\\hat c:= \\frac{1}{\\lambda_{\\max} }- L \\bar\\beta -\\frac{\\bar\\gamma}{\\lambda_{\\min}}-\\frac{L}{2} >0$. On the other hand,\n\\begin{equation*}\n\\nabla h(x_n, x_{n-1}) = (\\nabla f(x_{n}) + (L \\bar\\beta +\\frac{\\bar\\gamma}{ \\lambda_{\\min} })(x_n -x_{n-1}),  -(L \\bar\\beta +\\frac{\\bar\\gamma}{ \\lambda_{\\min} })(x_n -x_{n-1})).\n\\end{equation*}\nCombining this with \\eqref{eq:d2}, we obtain that\n\\begin{equation*}\n\\|\\nabla h(x_n, x_{n-1})\\| \\leq  \\frac{1}{\\lambda_{\\min}}\\|x_{n+1}-x_n\\|  + 3\\left( \\frac{\\bar\\gamma}{\\lambda_{\\min}} +  L \\bar\\beta\\right)\\|x_n - x_{n-1}\\|,\n\\end{equation*}\nwhich together with \\eqref{eq:cvKL2} and Lemma~\\ref{l:KL} completes the proof. \n\\end{proof}","sha256_utf8":"56c2bb39f8d6fa0baa8e99da9032805c199644f0d6e508e000016e5887b2302d"}]},{"id":"S013","semantic_type":"assumption","kind":"assumption","title":"主定理共同假设","source_units":[{"start":65842,"end":66251,"start_line":827,"end_line":827,"raw_tex":"Suppose that $\\mathcal{H}$ is finite dimensional, that $f$ is a differentiable function whose gradient is $L$-Lipschitz continuous, and that $f$ is bounded below. Let $(x_n)_{n\\in \\mathbb{N}}$ be the sequence generated by AGDNL algorithm with $\\lambda_{\\min} >0$. Suppose further that the set $\\{x \\in \\mathcal{H}: f(x) \\leq f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2\\}$ is bounded, and that one of the following holds\n","sha256_utf8":"48d9028a3f73e6cd98c99933b18760be5fe0f535d034ca604aa4d63980d346e0"}]},{"id":"S014","semantic_type":"assumption","kind":"assumption","title":"单调搜索与可变势函数假设","source_units":[{"start":66269,"end":66593,"start_line":829,"end_line":829,"raw_tex":"\\item \\label{ADFGe:i}  $N=0$ and $ g(x,y,\\rho):= f(x)+ \\rho  \\|x -y\\|^2 $ satisfies the KL property at $(\\bar x, \\bar x, \\rho)$ for all $\\bar x \\in \\mathcal{H}$ and $\\rho \\in [0, \\bar\\rho]$, where $\\bar\\rho: =\\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\bar\\gamma}{\\underline{\\nu}} + \\frac{ L_{\\max}\\bar\\beta +\\bar\\gamma}{2 }$.\n","sha256_utf8":"0484f4cb819498102d26881f62d4c5d8d90c26e6824b70e389056f1d0151e180"}]},{"id":"S015","semantic_type":"assumption","kind":"assumption","title":"参数间隙与固定势函数假设","source_units":[{"start":66593,"end":66882,"start_line":830,"end_line":830,"raw_tex":"\\item \\label{ADFGe:ii} $\\frac{1}{\\lambda_{\\max} }- L \\bar\\beta -\\frac{\\bar\\gamma}{\\lambda_{\\min}}-\\frac{L}{2} >0$ and $h(x,y) =f(x) +(\\frac{L \\bar\\beta}{2} +\\frac{\\bar\\gamma}{2 \\lambda_{\\min} }) \\|x-y\\|^2$ satisfies the KL property at $(\\bar x, \\bar x)$  for all $\\bar x \\in \\mathcal{H}$.\n","sha256_utf8":"d838e3d894a014b8340c66d918240b812a6a86fd2070f853cfccbf6f0f164c5b"}]},{"id":"S016","semantic_type":"lemma","kind":"lemma","title":"近端点摩擦不等式","source_units":[{"start":23449,"end":23728,"start_line":352,"end_line":354,"raw_tex":"\\begin{lemma}\\label{l:dry}\nSuppose that $\\phi\\colon \\mathcal{H}\\to \\left(-\\infty, +\\infty\\right]$ satisfies the dry-like friction property with constant $r$. Let $y \\in \\mathcal{H}$ and $w \\in \\prox_{\\tau \\phi}(y)$. Then $\\langle w-y, w\\rangle  + r\\tau \\|w\\| \\leq 0.$\n\\end{lemma}","sha256_utf8":"12d1390a7b6f9fd03b4c1cfbfc685e0bbbf59d1957822c318dc4d807a41af762"}]},{"id":"S017","semantic_type":"proof","kind":"proof","title":"摩擦不等式完整证明","source_units":[{"start":23729,"end":24140,"start_line":355,"end_line":357,"raw_tex":"\\begin{proof}\nSince $w \\in \\prox_{\\tau \\phi}(y) = \\argmin_{x \\in \\mathcal{H}} \\left(\\phi(x) +\\frac{1}{2 \\tau}\\|x-y\\|^2\\right)$, we have $0 \\in \\partial \\phi(w) + \\frac{1}{\\tau}(w-y)$, which yields $\\frac{1}{\\tau}(y-w) \\in \\partial \\phi(w)$, and so $w \\in \\dom \\partial \\phi$. By the dry-like friction property of $\\phi$, $\\frac{1}{\\tau}\\langle y -w, w\\rangle \\geq r\\|w\\|$, which completes the proof.\n\\end{proof}","sha256_utf8":"ec06408ee31b688bbbdc6212e7e40082beafcaccb57b85a157ef6f8fc6563aaf"}]},{"id":"S018","semantic_type":"proof_step","kind":"proof_step","title":"近端最优性导出摩擦估计","source_units":[{"start":23743,"end":24129,"start_line":356,"end_line":356,"raw_tex":"Since $w \\in \\prox_{\\tau \\phi}(y) = \\argmin_{x \\in \\mathcal{H}} \\left(\\phi(x) +\\frac{1}{2 \\tau}\\|x-y\\|^2\\right)$, we have $0 \\in \\partial \\phi(w) + \\frac{1}{\\tau}(w-y)$, which yields $\\frac{1}{\\tau}(y-w) \\in \\partial \\phi(w)$, and so $w \\in \\dom \\partial \\phi$. By the dry-like friction property of $\\phi$, $\\frac{1}{\\tau}\\langle y -w, w\\rangle \\geq r\\|w\\|$, which completes the proof.\n","sha256_utf8":"e9cfac231dee2e430bef34f921fa7affeb65637b614a124f4691aa8b0179ae8e"}]},{"id":"S019","semantic_type":"lemma","kind":"lemma","title":"摩擦项与势函数下降估计","source_units":[{"start":31797,"end":33154,"start_line":463,"end_line":483,"raw_tex":"\\begin{lemma}\n% [Descent property of the potential function]\n\\label{l:descent}\nSuppose that $f$ is a differentiable function whose gradient is $L$-Lipschitz continuous. Under the setting of AGDNL algorithm, for all $n \\in \\N$, the following hold:\n\\begin{enumerate}\n\\item\\label{l:descent_wn}\n$\\|w_n\\|^2 - \\mu \\alpha_n \\langle w_{n-1}, w_n\\rangle  +\\kappa_n \\langle \\nabla f(u_n), w_n\\rangle  +\\delta \\| w_n\\| \\leq 0$. Consequently, $\\|w_n\\|^2 + (\\delta - k_{\\max} \\|\\nabla f(u_n)\\| - \\mu  \\bar \\alpha\\|w_{n-1}\\|) \\|w_n\\| \\leq 0$. \n\\item\\label{l:descent_Vn}\nIf $L_n \\geq L$, $L_n \\beta_n \\leq L_{n-1} \\beta_{n-1}$, $\\gamma_n \\leq \\gamma_{n-1}$, $\\frac{\\gamma_n}{\\nu_n} \\leq \\frac{\\gamma_{n-1}}{\\nu_{n-1}}$, and $\\frac{\\alpha_n}{\\nu_n} \\leq \\frac{\\alpha_{n-1}}{\\nu_{n-1}}$, then\n\\begin{align*}%\\label{eq:KL1}\n&V(z_{n+1}, x_n, w_n,\\rho_n, \\sigma_n) + \\frac{c_n}{\\nu_n}\\|z_{n+1} -x_n||^2 +  \\frac{d_n}{\\nu_n}\\|\\nabla f(u_n)\\|^2 + \\frac{\\delta}{\\nu_n} \\| w_n\\| \\\\\n&\\leq V(x_n, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}),\n\\end{align*}  \nwhere\n\\begin{align}\nc_n &= \\frac{(1 -\\mu \\bar\\alpha) (1 -2\\gamma_n )}{\\mu^2} - \\frac{(2 L_n\\beta_n +\\gamma_n + L_n)\\nu_n }{2} \\label{d:cn}\\\\\n\\text{and~~}\nd_{n} &= \\frac{2(1- \\mu \\bar\\alpha)(\\lambda_n^2 -\\gamma_n\\lambda_n)-\\mu\\gamma_n\\kappa_n+2\\mu\\lambda_n \\kappa_n}{2\\mu^2} \\label{d:dn}.\n\\end{align}\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"3eac6115b0ea06a96f911c2f731f2c4fd3b01eb2ed345e81fc922d975d805421"}]},{"id":"S020","semantic_type":"proof","kind":"proof","title":"下降引理完整证明","source_units":[{"start":33259,"end":39560,"start_line":485,"end_line":553,"raw_tex":"\\begin{proof}\n\\ref{l:descent_wn}:\nSet $y_n = \\alpha_n(x_n -v_{n-1}+\\lambda_n\\nabla f(u_{n-1}))-\\theta_n \\frac{\\nabla f(u_n)}{\\|\\nabla f(u_n)\\|}-\\kappa_n \\nabla f(u_n)+ e_n$. By the update of $x_n$, we have $\\mu w_{n-1}= x_n - v_{n-1} +\\lambda_{n-1} \\nabla f(u_{n-1})$, and so\n\\begin{equation*}\ny_n = \\alpha_n\\mu w_{n-1} -\\theta_n \\frac{\\nabla f(u_n)}{\\|\\nabla f(u_n)\\|} -\\kappa_n \\nabla f(u_n)+ e_n.\n\\end{equation*}\nBy Step~\\ref{step:main}\\ref{step:main_comp}, $w_n \\in \\prox_{\\tau_n \\phi}(y_n)$. Since $\\phi$ satisfies the dry-like friction property with constant $r$, we have from Lemma~\\ref{l:dry} that $\\langle w_n-y_n, w_n\\rangle  + r\\tau_n \\|w_n\\| \\leq 0$, which yields\n\\begin{equation*}\n\\|w_n\\|^2 - \\alpha_n \\mu\\langle w_{n-1}, w_n\\rangle  + \\kappa_n \\langle \\nabla f(u_n), w_n\\rangle  + (r\\tau_n - |\\theta_n| - \\|e_n\\|)\\| w_n\\| \\leq 0.    \n\\end{equation*}\nCombining this with $r\\tau_n \\geq |\\theta_n| +\\|e_n\\| +\\delta$, we get the first conclusion, which implies the second one.\n\n\\ref{l:descent_Vn}: Set $\\Delta_n = x_n - x_{n-1}$. It follows from the update of $z_{n+1}$ that\n\\begin{equation*}\n\\mu w_n=z_{n+1} -v_n +\\lambda_n \\nabla f(u_n) \n=(z_{n+1} -x_n)-\\gamma_n \\Delta_n +\\lambda_n \\nabla f(u_n),    \n\\end{equation*}\nand so\n\\begin{align}\\label{eq:57}\n\\mu^2\\|w_n\\|^2&=  \\|z_{n+1} -x_n\\|^2 +\\gamma_n^2 \\|\\Delta_n\\|^2 +\\lambda_n^2 \\|\\nabla f(u_n)\\|^2 - 2\\gamma_n \\langle z_{n+1} -x_n,  \\Delta_n\\rangle \\notag\\\\\n & \\quad +2  \\lambda_n\\langle z_{n+1} -x_n, \\nabla f(u_n)\\rangle -2 \\gamma_n\\lambda_n\\langle  \\Delta_n, \\nabla f(u_n) \\rangle \\notag\\\\\n &\\geq  \\|z_{n+1} -x_n\\|^2 +\\gamma_n^2 \\|\\Delta_n\\|^2 +\\lambda_n^2 \\|\\nabla f(u_n)\\|^2 - \\gamma_n (\\|z_{n+1}-x_n\\|^2 + \\|\\Delta_n\\|^2)  \\notag\\\\\n  & \\quad +2 \\lambda_n\\langle z_{n+1} -x_n, \\nabla f(u_n)\\rangle - \\gamma_n\\lambda_n(\\|\\Delta_n\\|^2 + \\|\\nabla f(u_n)\\|^2) \\notag\\\\\n  &=  (1-\\gamma_n)\\|z_{n+1} -x_n\\|^2 +(\\gamma_n^2-\\gamma_n -\\gamma_n\\lambda_n) \\|\\Delta_n\\|^2 + (\\lambda_n^2 -\\gamma_n\\lambda_n) \\|\\nabla f(u_n)\\|^2 \\notag\\\\\n   & \\quad +2 \\lambda_n\\langle z_{n+1} -x_n, \\nabla f(u_n)\\rangle.\n\\end{align}\nSince $\\nabla f$ is $L$-Lipschitz continuous, $L_n \\geq L$, and $u_n = x_n + \\beta_n(x_n - x_{n-1})$, we have\n\\begin{align}\\label{eq:es2}\n\\langle \\nabla f(u_n), z_{n+1} -x_n \\rangle &= \\langle \\nabla f(u_n) - \\nabla f(x_n), z_{n+1} - x_{n} \\rangle + \\langle \\nabla f(x_n), z_{n+1} -x_n \\rangle \\notag\\\\\n& \\geq -L_n\\|u_n -x_n\\| \\|z_{n+1} -x_n\\| + f(z_{n+1}) -f(x_n) -\\frac{L_n}{2}\\|z_{n+1} - x_n\\|^2 \\notag\\\\\n% & \\geq -\\frac{L_n\\beta_n}{2}(\\|\\Delta_n\\|^2 + \\|z_{n+1}-x_n\\|^2) + f(z_{n+1}) -f(x_n) -\\frac{L_n}{2}\\|z_{n+1} -x_n\\|^2\\notag\\\\\n& \\geq  -\\frac{L_n\\beta_n}{2}\\|\\Delta_n\\|^2 -\\frac{L_n\\beta_n +L_n}{2}\\|z_{n+1} -x_n\\|^2 + f(z_{n+1}) -f(x_n).\n\\end{align}\nCombining this with \\eqref{eq:57}, we derive that\n\\begin{align}\\label{eq:es1}\n &\\|w_n\\|^2 - \\mu \\alpha_n \\langle w_{n-1}, w_n\\rangle+\\kappa_n \\langle \\nabla f(u_n), w_n\\rangle \\notag\\\\\n&\\geq \\|w_n\\|^2 - \\frac{\\mu \\alpha_n}{2}(\\|w_n\\|^2 +\\|w_{n-1}\\|^2)+\\kappa_n \\langle \\nabla f(u_n), w_n\\rangle \\notag\\\\\n& \\geq  -\\frac{\\mu \\alpha_{n}}{2} \\|w_{n-1}\\|^2 +\\frac{\\mu \\alpha_n}{2} \\|w_n\\|^2 +(1-\\mu \\bar\\alpha)\\|w_n\\|^2\\notag\\\\\n&\\quad+\\frac{\\kappa_n }{\\mu} \\langle \\nabla f(u_n), z_{n+1} -x_n \\rangle -\\frac{\\gamma_n\\kappa_n }{\\mu} \\langle \\nabla f(u_n), \\Delta_n \\rangle +\\frac{\\lambda_n \\kappa_n }{\\mu}\\|\\nabla f(u_n)\\|^2\\notag\\\\\n& \\geq  -\\frac{\\mu \\alpha_{n}}{2} \\|w_{n-1}\\|^2  +\\frac{\\mu \\alpha_n}{2} \\|w_n\\|^2 \\notag\\\\\n& \\quad +\\frac{1-\\mu \\bar\\alpha }{\\mu^2}\\left( (1-\\gamma_n)\\|z_{n+1} -x_n\\|^2 +(\\gamma_n^2-\\gamma_n -\\gamma_n\\lambda_n) \\|\\Delta_n\\|^2 + (\\lambda_n^2 -\\gamma_n\\lambda_n) \\|\\nabla f(u_n)\\|^2  \\right)\\notag\\\\\n  & \\quad+ \\frac{2\\lambda_n(1-\\mu \\bar\\alpha)}{\\mu^2}\\langle z_{n+1} -x_n, \\nabla f(u_n)\\rangle +\\frac{\\kappa_n}{\\mu} \\langle \\nabla f(u_n), z_{n+1} -x_n \\rangle  \\notag\\\\\n  &\\quad -\\frac{\\gamma_n \\kappa_n }{2\\mu} (\\|\\nabla f(u_n)\\|^2 +\\|\\Delta_n \\|^2) +\\frac{\\lambda_n\\kappa_n }{\\mu}\\|\\nabla f(u_n)\\|^2\\notag\\\\\n&\\geq -\\frac{\\mu \\alpha_{n}}{2} \\|w_{n-1}\\|^2 +\\frac{\\mu \\alpha_n}{2} \\|w_n\\|^2 +\\bar c_n\\|z_{n+1}-x_n\\|^2 -\\tilde c_n\\|\\Delta_n\\|^2+ d_n\\|\\nabla f(u_n)\\|^2\\notag\\\\\n&\\quad +\\left(\\frac{2\\lambda_n(1-\\mu \\bar\\alpha)}{\\mu^2} +\\frac{\\kappa_n }{\\mu}\\right)( f(z_{n+1}) -f(x_n)), \n\\end{align}\nwhere the last inequality follows from \\eqref{eq:es2}, $d_n$ is defined in \\eqref{d:dn} and\n\\begin{align}\n\\bar c_n &=\\frac{(1-\\mu \\bar\\alpha) (1-\\gamma_n)}{\\mu^2} -\\frac{L_n\\beta_n + L_n}{2 }  \\left(\\frac{2\\lambda_n(1-\\mu \\bar\\alpha)}{\\mu^2}+ \\frac{\\kappa_n}{\\mu}\\right), \\notag\\\\\n\\tilde c_n  &=\\frac{(1 -\\mu \\bar\\alpha)\\left(\\gamma_n +\\gamma_n\\lambda_n+L_n\\lambda_n \\beta_n \\right)  + \\frac{\\mu \\kappa_n}{2}(L_n\\beta_n +\\gamma_n)}{\\mu^2} \\notag\\\\\n&=\\frac{(1 -\\mu \\bar\\alpha)\\gamma_n}{\\mu^2} + \\frac{ L_n\\beta_n +\\gamma_n}{2 } \\left(\\frac{2\\lambda_n(1-\\mu \\bar\\alpha)}{\\mu^2}+ \\frac{\\kappa_n}{\\mu}\\right) =\\rho_n \\nu_n.\\label{d:eta} \n\\end{align}\nIt follows from \\ref{l:descent_wn}, \\eqref{eq:es1}, and the definition of $\\nu_n$ that\n\\begin{align*}\n&\\frac{\\mu \\alpha_n}{2} \\|w_n\\|^2 -\\frac{\\mu \\alpha_n}{2}\\|w_{n-1}\\|^2  +  \\nu_n(f(z_{n+1}) -f(x_n))+ \\bar c_n \\|z_{n+1} -x_n||^2  - \\rho_n \\nu_n\\|\\Delta_{n}\\|^2 \\notag\\\\\n& + d_n\\|\\nabla f(u_n)\\|^2 + \\delta \\| w_n\\|  \\leq 0,\n\\end{align*}\nwhich implies that\n\\begin{align}\\label{eq:ADFGe_4}\n &f(z_{n+1})+ \\rho_n \\|z_{n+1} -x_n||^2 + \\frac{\\mu \\alpha_n}{2\\nu_n} \\|w_n\\|^2 + \\frac{c_n}{\\nu_n}\\|z_{n+1} -x_n||^2 +  \\frac{d_n}{\\nu_n}\\|\\nabla f(u_n)\\|^2 \\notag\\\\\n &\\hspace{3cm} + \\frac{\\delta}{\\nu_n} \\| w_n\\| \\leq f(x_n)+ \\rho_n \\|x_n -x_{n-1}||^2 + \\frac{\\mu \\alpha_n}{2 \\nu_n} \\|w_{n-1}\\|^2.\n  \\end{align}  \nHere, we note that\n\\begin{align*}\n\\bar c_n-\\rho_n \\nu_n&= \\frac{(1 -\\mu \\bar\\alpha) (1 -2\\gamma_n )}{\\mu^2} - \\frac{2 L_n\\beta_n +\\gamma_n + L_n}{2} \\left(\\frac{2\\lambda_n(1-\\mu \\bar\\alpha)}{\\mu^2}+ \\frac{\\kappa_n}{\\mu}\\right) \\\\\n&= \\frac{(1 -\\mu \\bar\\alpha) (1 -2\\gamma_n )}{\\mu^2} - \\frac{(2 L_n\\beta_n +\\gamma_n + L_n)\\nu_n}{2}  =c_n.\n\\end{align*}\nSince $L_n \\beta_n \\leq L_{n-1}\\beta_{n-1}$, $\\gamma_n \\leq \\gamma_{n-1}$, and $\\frac{\\gamma_n}{\\nu_n} \\leq \\frac{\\gamma_{n-1}}{\\nu_{n-1}}$, we have $\\rho_n \\leq \\rho_{n-1}$. As $\\frac{\\alpha_n}{\\nu_n} \\leq \\frac{\\alpha_{n-1}}{\\nu_{n-1}}$, using \\eqref{eq:ADFGe_4} and the definition of $V$ in \\eqref{eq:V}, we get the conclusion. \n\\end{proof}","sha256_utf8":"8553a8406e3dfcf98b78ea1125bfe592a6dfa3df3267b1d5b5a000a34092ee6d"}]},{"id":"S021","semantic_type":"proof_step","kind":"proof_step","title":"控制近端摩擦项","source_units":[{"start":33273,"end":34246,"start_line":486,"end_line":495,"raw_tex":"\\ref{l:descent_wn}:\nSet $y_n = \\alpha_n(x_n -v_{n-1}+\\lambda_n\\nabla f(u_{n-1}))-\\theta_n \\frac{\\nabla f(u_n)}{\\|\\nabla f(u_n)\\|}-\\kappa_n \\nabla f(u_n)+ e_n$. By the update of $x_n$, we have $\\mu w_{n-1}= x_n - v_{n-1} +\\lambda_{n-1} \\nabla f(u_{n-1})$, and so\n\\begin{equation*}\ny_n = \\alpha_n\\mu w_{n-1} -\\theta_n \\frac{\\nabla f(u_n)}{\\|\\nabla f(u_n)\\|} -\\kappa_n \\nabla f(u_n)+ e_n.\n\\end{equation*}\nBy Step~\\ref{step:main}\\ref{step:main_comp}, $w_n \\in \\prox_{\\tau_n \\phi}(y_n)$. Since $\\phi$ satisfies the dry-like friction property with constant $r$, we have from Lemma~\\ref{l:dry} that $\\langle w_n-y_n, w_n\\rangle  + r\\tau_n \\|w_n\\| \\leq 0$, which yields\n\\begin{equation*}\n\\|w_n\\|^2 - \\alpha_n \\mu\\langle w_{n-1}, w_n\\rangle  + \\kappa_n \\langle \\nabla f(u_n), w_n\\rangle  + (r\\tau_n - |\\theta_n| - \\|e_n\\|)\\| w_n\\| \\leq 0.    \n\\end{equation*}\nCombining this with $r\\tau_n \\geq |\\theta_n| +\\|e_n\\| +\\delta$, we get the first conclusion, which implies the second one.\n","sha256_utf8":"375c32464fadf54b7f8bdbcfdfc9abc022a190c4317dc02c916736dbe8dbe5bd"}]},{"id":"S022","semantic_type":"proof_step","kind":"proof_step","title":"展开更新方向范数","source_units":[{"start":34247,"end":35329,"start_line":497,"end_line":510,"raw_tex":"\\ref{l:descent_Vn}: Set $\\Delta_n = x_n - x_{n-1}$. It follows from the update of $z_{n+1}$ that\n\\begin{equation*}\n\\mu w_n=z_{n+1} -v_n +\\lambda_n \\nabla f(u_n) \n=(z_{n+1} -x_n)-\\gamma_n \\Delta_n +\\lambda_n \\nabla f(u_n),    \n\\end{equation*}\nand so\n\\begin{align}\\label{eq:57}\n\\mu^2\\|w_n\\|^2&=  \\|z_{n+1} -x_n\\|^2 +\\gamma_n^2 \\|\\Delta_n\\|^2 +\\lambda_n^2 \\|\\nabla f(u_n)\\|^2 - 2\\gamma_n \\langle z_{n+1} -x_n,  \\Delta_n\\rangle \\notag\\\\\n & \\quad +2  \\lambda_n\\langle z_{n+1} -x_n, \\nabla f(u_n)\\rangle -2 \\gamma_n\\lambda_n\\langle  \\Delta_n, \\nabla f(u_n) \\rangle \\notag\\\\\n &\\geq  \\|z_{n+1} -x_n\\|^2 +\\gamma_n^2 \\|\\Delta_n\\|^2 +\\lambda_n^2 \\|\\nabla f(u_n)\\|^2 - \\gamma_n (\\|z_{n+1}-x_n\\|^2 + \\|\\Delta_n\\|^2)  \\notag\\\\\n  & \\quad +2 \\lambda_n\\langle z_{n+1} -x_n, \\nabla f(u_n)\\rangle - \\gamma_n\\lambda_n(\\|\\Delta_n\\|^2 + \\|\\nabla f(u_n)\\|^2) \\notag\\\\\n  &=  (1-\\gamma_n)\\|z_{n+1} -x_n\\|^2 +(\\gamma_n^2-\\gamma_n -\\gamma_n\\lambda_n) \\|\\Delta_n\\|^2 + (\\lambda_n^2 -\\gamma_n\\lambda_n) \\|\\nabla f(u_n)\\|^2 \\notag\\\\\n   & \\quad +2 \\lambda_n\\langle z_{n+1} -x_n, \\nabla f(u_n)\\rangle.\n\\end{align}\n","sha256_utf8":"d0bf162120d52414deffef947064a26344d99af484db0753b2180766d2e0cc06"}]},{"id":"S023","semantic_type":"proof_step","kind":"proof_step","title":"梯度内积控制函数变化","source_units":[{"start":35329,"end":35989,"start_line":511,"end_line":517,"raw_tex":"Since $\\nabla f$ is $L$-Lipschitz continuous, $L_n \\geq L$, and $u_n = x_n + \\beta_n(x_n - x_{n-1})$, we have\n\\begin{align}\\label{eq:es2}\n\\langle \\nabla f(u_n), z_{n+1} -x_n \\rangle &= \\langle \\nabla f(u_n) - \\nabla f(x_n), z_{n+1} - x_{n} \\rangle + \\langle \\nabla f(x_n), z_{n+1} -x_n \\rangle \\notag\\\\\n& \\geq -L_n\\|u_n -x_n\\| \\|z_{n+1} -x_n\\| + f(z_{n+1}) -f(x_n) -\\frac{L_n}{2}\\|z_{n+1} - x_n\\|^2 \\notag\\\\\n% & \\geq -\\frac{L_n\\beta_n}{2}(\\|\\Delta_n\\|^2 + \\|z_{n+1}-x_n\\|^2) + f(z_{n+1}) -f(x_n) -\\frac{L_n}{2}\\|z_{n+1} -x_n\\|^2\\notag\\\\\n& \\geq  -\\frac{L_n\\beta_n}{2}\\|\\Delta_n\\|^2 -\\frac{L_n\\beta_n +L_n}{2}\\|z_{n+1} -x_n\\|^2 + f(z_{n+1}) -f(x_n).\n\\end{align}\n","sha256_utf8":"722b0a38431e33e35558c01276e933d7bfb6708f65fa8fc10a0f84e6b3e88e50"}]},{"id":"S024","semantic_type":"proof_step","kind":"proof_step","title":"合并估计并识别系数","source_units":[{"start":35989,"end":38160,"start_line":518,"end_line":536,"raw_tex":"Combining this with \\eqref{eq:57}, we derive that\n\\begin{align}\\label{eq:es1}\n &\\|w_n\\|^2 - \\mu \\alpha_n \\langle w_{n-1}, w_n\\rangle+\\kappa_n \\langle \\nabla f(u_n), w_n\\rangle \\notag\\\\\n&\\geq \\|w_n\\|^2 - \\frac{\\mu \\alpha_n}{2}(\\|w_n\\|^2 +\\|w_{n-1}\\|^2)+\\kappa_n \\langle \\nabla f(u_n), w_n\\rangle \\notag\\\\\n& \\geq  -\\frac{\\mu \\alpha_{n}}{2} \\|w_{n-1}\\|^2 +\\frac{\\mu \\alpha_n}{2} \\|w_n\\|^2 +(1-\\mu \\bar\\alpha)\\|w_n\\|^2\\notag\\\\\n&\\quad+\\frac{\\kappa_n }{\\mu} \\langle \\nabla f(u_n), z_{n+1} -x_n \\rangle -\\frac{\\gamma_n\\kappa_n }{\\mu} \\langle \\nabla f(u_n), \\Delta_n \\rangle +\\frac{\\lambda_n \\kappa_n }{\\mu}\\|\\nabla f(u_n)\\|^2\\notag\\\\\n& \\geq  -\\frac{\\mu \\alpha_{n}}{2} \\|w_{n-1}\\|^2  +\\frac{\\mu \\alpha_n}{2} \\|w_n\\|^2 \\notag\\\\\n& \\quad +\\frac{1-\\mu \\bar\\alpha }{\\mu^2}\\left( (1-\\gamma_n)\\|z_{n+1} -x_n\\|^2 +(\\gamma_n^2-\\gamma_n -\\gamma_n\\lambda_n) \\|\\Delta_n\\|^2 + (\\lambda_n^2 -\\gamma_n\\lambda_n) \\|\\nabla f(u_n)\\|^2  \\right)\\notag\\\\\n  & \\quad+ \\frac{2\\lambda_n(1-\\mu \\bar\\alpha)}{\\mu^2}\\langle z_{n+1} -x_n, \\nabla f(u_n)\\rangle +\\frac{\\kappa_n}{\\mu} \\langle \\nabla f(u_n), z_{n+1} -x_n \\rangle  \\notag\\\\\n  &\\quad -\\frac{\\gamma_n \\kappa_n }{2\\mu} (\\|\\nabla f(u_n)\\|^2 +\\|\\Delta_n \\|^2) +\\frac{\\lambda_n\\kappa_n }{\\mu}\\|\\nabla f(u_n)\\|^2\\notag\\\\\n&\\geq -\\frac{\\mu \\alpha_{n}}{2} \\|w_{n-1}\\|^2 +\\frac{\\mu \\alpha_n}{2} \\|w_n\\|^2 +\\bar c_n\\|z_{n+1}-x_n\\|^2 -\\tilde c_n\\|\\Delta_n\\|^2+ d_n\\|\\nabla f(u_n)\\|^2\\notag\\\\\n&\\quad +\\left(\\frac{2\\lambda_n(1-\\mu \\bar\\alpha)}{\\mu^2} +\\frac{\\kappa_n }{\\mu}\\right)( f(z_{n+1}) -f(x_n)), \n\\end{align}\nwhere the last inequality follows from \\eqref{eq:es2}, $d_n$ is defined in \\eqref{d:dn} and\n\\begin{align}\n\\bar c_n &=\\frac{(1-\\mu \\bar\\alpha) (1-\\gamma_n)}{\\mu^2} -\\frac{L_n\\beta_n + L_n}{2 }  \\left(\\frac{2\\lambda_n(1-\\mu \\bar\\alpha)}{\\mu^2}+ \\frac{\\kappa_n}{\\mu}\\right), \\notag\\\\\n\\tilde c_n  &=\\frac{(1 -\\mu \\bar\\alpha)\\left(\\gamma_n +\\gamma_n\\lambda_n+L_n\\lambda_n \\beta_n \\right)  + \\frac{\\mu \\kappa_n}{2}(L_n\\beta_n +\\gamma_n)}{\\mu^2} \\notag\\\\\n&=\\frac{(1 -\\mu \\bar\\alpha)\\gamma_n}{\\mu^2} + \\frac{ L_n\\beta_n +\\gamma_n}{2 } \\left(\\frac{2\\lambda_n(1-\\mu \\bar\\alpha)}{\\mu^2}+ \\frac{\\kappa_n}{\\mu}\\right) =\\rho_n \\nu_n.\\label{d:eta} \n\\end{align}\n","sha256_utf8":"5f81df4aa45b032cac68548e582f71de5c02ff82b80da214bbcf66da16eda8f0"}]},{"id":"S025","semantic_type":"proof_step","kind":"proof_step","title":"建立加权下降不等式","source_units":[{"start":38160,"end":38866,"start_line":537,"end_line":546,"raw_tex":"It follows from \\ref{l:descent_wn}, \\eqref{eq:es1}, and the definition of $\\nu_n$ that\n\\begin{align*}\n&\\frac{\\mu \\alpha_n}{2} \\|w_n\\|^2 -\\frac{\\mu \\alpha_n}{2}\\|w_{n-1}\\|^2  +  \\nu_n(f(z_{n+1}) -f(x_n))+ \\bar c_n \\|z_{n+1} -x_n||^2  - \\rho_n \\nu_n\\|\\Delta_{n}\\|^2 \\notag\\\\\n& + d_n\\|\\nabla f(u_n)\\|^2 + \\delta \\| w_n\\|  \\leq 0,\n\\end{align*}\nwhich implies that\n\\begin{align}\\label{eq:ADFGe_4}\n &f(z_{n+1})+ \\rho_n \\|z_{n+1} -x_n||^2 + \\frac{\\mu \\alpha_n}{2\\nu_n} \\|w_n\\|^2 + \\frac{c_n}{\\nu_n}\\|z_{n+1} -x_n||^2 +  \\frac{d_n}{\\nu_n}\\|\\nabla f(u_n)\\|^2 \\notag\\\\\n &\\hspace{3cm} + \\frac{\\delta}{\\nu_n} \\| w_n\\| \\leq f(x_n)+ \\rho_n \\|x_n -x_{n-1}||^2 + \\frac{\\mu \\alpha_n}{2 \\nu_n} \\|w_{n-1}\\|^2.\n  \\end{align}  \n","sha256_utf8":"5f934f55b751cfb1e01be4f94757cb50810c301859a79cf8bac09282a25fef25"}]},{"id":"S026","semantic_type":"proof_step","kind":"proof_step","title":"利用参数单调性完成下降","source_units":[{"start":38866,"end":39549,"start_line":547,"end_line":552,"raw_tex":"Here, we note that\n\\begin{align*}\n\\bar c_n-\\rho_n \\nu_n&= \\frac{(1 -\\mu \\bar\\alpha) (1 -2\\gamma_n )}{\\mu^2} - \\frac{2 L_n\\beta_n +\\gamma_n + L_n}{2} \\left(\\frac{2\\lambda_n(1-\\mu \\bar\\alpha)}{\\mu^2}+ \\frac{\\kappa_n}{\\mu}\\right) \\\\\n&= \\frac{(1 -\\mu \\bar\\alpha) (1 -2\\gamma_n )}{\\mu^2} - \\frac{(2 L_n\\beta_n +\\gamma_n + L_n)\\nu_n}{2}  =c_n.\n\\end{align*}\nSince $L_n \\beta_n \\leq L_{n-1}\\beta_{n-1}$, $\\gamma_n \\leq \\gamma_{n-1}$, and $\\frac{\\gamma_n}{\\nu_n} \\leq \\frac{\\gamma_{n-1}}{\\nu_{n-1}}$, we have $\\rho_n \\leq \\rho_{n-1}$. As $\\frac{\\alpha_n}{\\nu_n} \\leq \\frac{\\alpha_{n-1}}{\\nu_{n-1}}$, using \\eqref{eq:ADFGe_4} and the definition of $V$ in \\eqref{eq:V}, we get the conclusion. \n","sha256_utf8":"55bc7ad2011acd2f0f9fa158bff833b15c718c606ef1f33fabd62bdc8b173b50"}]},{"id":"S027","semantic_type":"lemma","kind":"lemma","title":"线搜索有限终止","source_units":[{"start":39909,"end":40220,"start_line":558,"end_line":560,"raw_tex":"\\begin{lemma}[Well-definedness of the line search criterion]\nSuppose that $f$ is a differentiable function whose gradient is $L$-Lipschitz continuous. Then, for all $n \\in \\N$, the line search criterion \\eqref{eq:linesearch} in AGDNL algorithm is satisfied after a finite number of inner iterations.\n\\end{lemma}","sha256_utf8":"5916578fefe836cfb18959e9519d5fd10dff7eea141998ad24fce4d0d916f568"}]},{"id":"S028","semantic_type":"proof","kind":"proof","title":"线搜索终止完整证明","source_units":[{"start":40221,"end":44414,"start_line":561,"end_line":588,"raw_tex":"\\begin{proof}\nFix $n \\in \\mathbb{N}$. Since $\\ell_1 \\in (0, 1)$ and $\\tilde{L}_n > L_{\\min}$, there exists $k_0$ such that $\\ell_1^{-k_0}\\tilde L_n\\geq L_{\\max}$. It also can be seen that there exists $k_1$ such that $\\ell_3^{k_1} \\tilde{\\lambda}_n \\leq \\lambda_{\\min}$ and $\\ell_3^{k_1} \\tilde{\\kappa}_n \\leq \\kappa_{\\min}$ since $\\ell_3 \\in (0,1)$. Suppose that the line search criterion \\eqref{eq:linesearch} is not satisfied after $\\max\\{k_0, k_1\\}$ inner iterations. At the $k$-th inner iteration with $k >\\max\\{k_0, k_1\\}$, by Step~\\ref{step:main}\\ref{step:main_mult},\n\\begin{equation}\\label{eq:kth}\nL_n = L_{\\max}, \\quad \\gamma_n =\\ell_{2}^{k} \\tilde{\\gamma}_n, \\quad \\alpha_n =\\ell_{2}^{k} \\tilde{\\alpha}_n, \\quad \\beta_n =\\ell_{2}^{k} \\tilde{\\beta}_n, \\quad \\lambda_n = \\lambda_{\\min}, \\text{~and~} \\kappa_n =\\kappa_{\\min}.\n\\end{equation}\nThen $\\nu_n = \\frac{2 \\lambda_{\\min}(1 -\\mu \\bar \\alpha)}{\\mu^2}+ \\frac{\\kappa_{\\min} }{\\mu} = \\underline{\\nu}$, and so $\\frac{\\gamma_n}{\\nu_n} =\\frac{\\ell_2^k\\tilde{\\gamma}_n}{\\nu_n} =\\frac{\\ell_2^k\\tilde{\\gamma}_n}{\\underline{\\nu}}$ and $\\frac{\\alpha_n}{\\nu_n} =\\frac{\\ell_2^k\\tilde{\\alpha}_n}{\\nu_n} =\\frac{\\ell_2^k \\tilde{\\alpha}_n}{\\underline{\\nu}}$. Since $0<\\ell_2 <1$, there exists $k_2 >\\max\\{k_0, k_1\\}$ such that after $k_2$ inner iterations, $\\gamma_n \\leq \\gamma_{n-1}$, $L_n \\beta_n = L_{\\max} \\ell_{2}^{k} \\tilde{\\beta}_n  \\leq L_{n-1} \\beta_{n-1}$,  $\\frac{\\gamma_n}{\\nu_n} \\leq \\frac{\\gamma_{n-1}}{\\nu_{n-1}}$, and $\\frac{\\alpha_n}{\\nu_n} \\leq \\frac{\\alpha_{n-1}}{\\nu_{n-1}}$.\nUsing Lemma~\\ref{l:descent} and noting that $\\nu_n =\\underline{\\nu}$, we get\n\\begin{align}\n\\label{eq:lsV}\n&V(z_{n+1}, x_n, w_n,\\rho_n, \\sigma_n) + \\frac{c_n}{\\underline{\\nu}}\\|z_{n+1} -x_n||^2 +  \\frac{d_n}{\\underline{\\nu}}\\|\\nabla f(u_n)\\|^2 + \\frac{\\delta}{\\underline{\\nu}} \\| w_n\\| \\notag \\\\\n&\\leq V(x_n, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}).\n\\end{align}\nIt follows from \\eqref{eq:kth} and the definition of $c_n$ in \\eqref{d:cn} that\n\\begin{align}\\label{eq:cn}\nc_n&= \\frac{(1 -\\mu \\bar\\alpha) (1 -2\\ell_{2}^{k} \\tilde{\\gamma}_n)}{\\mu^2} - \\frac{( 2 \\ell_{2}^{k}L_{\\max} \\tilde{\\beta}_n +\\ell_{2}^{k}\\tilde{\\gamma}_n + L_{\\max})\\underline{\\nu}}{2} \\notag\\\\\n&= \\frac{ 1-\\mu \\bar\\alpha}{\\mu^2}- \\frac{L_{\\max} \\underline{\\nu}}{2}-\\ell_2^k \\left[ \\frac{2\\tilde{\\gamma}_n (1-\\mu \\bar\\alpha)}{\\mu^2}  + \\frac{2L_{\\max}\\tilde{\\beta}_n \\underline{\\nu} +\\tilde{\\gamma}_n\\underline{\\nu}}{2} \\right]\\notag\\\\\n&=\\bar c \\underline{\\nu} -\\ell_2^k \\left[ \\frac{2\\tilde{\\gamma}_n (1-\\mu \\bar\\alpha)}{\\mu^2}  + \\frac{2L_{\\max}\\tilde{\\beta}_n \\underline{\\nu} +\\tilde{\\gamma}_n\\underline{\\nu}}{2} \\right].\n\\end{align}\nSince $c \\leq \\bar c=\\frac{2(1-\\mu \\bar \\alpha) - \\mu^2 L_{\\max} \\underline{\\nu}}{2 \\mu^2 \\underline{\\nu}}$, we consider two cases: $c <\\bar c$ and $c = \\bar c$. If $c <\\bar c$, using \\eqref{eq:cn} and the fact that $0< \\ell_{2} <1$, there exists $k_3$ such that after $k_3$ inner iterations, $c_n \\geq c\\underline{\\nu}$. If $c = \\bar c$, by the choice of $\\gamma_n$, $\\beta_n$ in Step~\\ref{step:init} and Step~\\ref{step:main}, for all $n \\in \\N$, $\\tilde \\gamma_n =0$ and $\\tilde \\beta_n =0$, which together with \\eqref{eq:cn} yields $c_n \\geq c\\underline{\\nu}$.\n\nNow, we consider $d_n$. Using definition of $d_n$ and \\eqref{eq:kth}, we have \n\\begin{align*}\nd_{n}&= \\frac{2(1- \\mu \\bar\\alpha)(\\lambda_n^2 -\\ell_{2}^k \\tilde{\\gamma}_n\\lambda_n)-\\mu \\ell_{2}^{k}\\tilde{\\gamma}_n\\kappa_n+2\\lambda_n \\mu \\kappa_n }{2\\mu^2}\\notag\\\\\n& = \\frac{(1-\\mu \\bar\\alpha)\\lambda_{\\min}^2 + \\lambda_{\\min} \\mu \\kappa_{\\min}}{\\mu^2} - \\ell_2^k\\frac{\\tilde{\\gamma}_n(2(1 -\\mu \\bar\\alpha)\\lambda_{\\min} +\\mu\\kappa_{\\min})}{2\\mu^2} \\notag\\\\\n& =\\bar d \\underline{\\nu} - \\ell_2^k\\frac{\\tilde{\\gamma}_n(2(1 -\\mu \\bar\\alpha)\\lambda_{\\min} +\\mu\\kappa_{\\min})}{2\\mu^2}.\n\\end{align*}\nBy Step~\\ref{step:init} and Step~\\ref{step:main}, $d \\in [0, \\bar d]$ and if $d =\\bar d$, then $\\tilde{\\gamma}_n =0$. Since $0<\\ell_2 <1$, there exists $k_4$ such that after $k_4$ inner iterations, $d_{n} \\geq d \\underline{\\nu}$. Hence, in view of \\eqref{eq:lsV}, after at most $\\max\\{ k_2,k_3, k_4\\}$ inner iterations, the line search criterion \\eqref{eq:linesearch} holds.\n\\end{proof}","sha256_utf8":"437ef9c7f30839e6c8d75008083820abdb9f5f9fe201630bd274b9f6f38b2f2c"}]},{"id":"S029","semantic_type":"proof_step","kind":"proof_step","title":"内循环参数达到下降条件","source_units":[{"start":40235,"end":41763,"start_line":562,"end_line":566,"raw_tex":"Fix $n \\in \\mathbb{N}$. Since $\\ell_1 \\in (0, 1)$ and $\\tilde{L}_n > L_{\\min}$, there exists $k_0$ such that $\\ell_1^{-k_0}\\tilde L_n\\geq L_{\\max}$. It also can be seen that there exists $k_1$ such that $\\ell_3^{k_1} \\tilde{\\lambda}_n \\leq \\lambda_{\\min}$ and $\\ell_3^{k_1} \\tilde{\\kappa}_n \\leq \\kappa_{\\min}$ since $\\ell_3 \\in (0,1)$. Suppose that the line search criterion \\eqref{eq:linesearch} is not satisfied after $\\max\\{k_0, k_1\\}$ inner iterations. At the $k$-th inner iteration with $k >\\max\\{k_0, k_1\\}$, by Step~\\ref{step:main}\\ref{step:main_mult},\n\\begin{equation}\\label{eq:kth}\nL_n = L_{\\max}, \\quad \\gamma_n =\\ell_{2}^{k} \\tilde{\\gamma}_n, \\quad \\alpha_n =\\ell_{2}^{k} \\tilde{\\alpha}_n, \\quad \\beta_n =\\ell_{2}^{k} \\tilde{\\beta}_n, \\quad \\lambda_n = \\lambda_{\\min}, \\text{~and~} \\kappa_n =\\kappa_{\\min}.\n\\end{equation}\nThen $\\nu_n = \\frac{2 \\lambda_{\\min}(1 -\\mu \\bar \\alpha)}{\\mu^2}+ \\frac{\\kappa_{\\min} }{\\mu} = \\underline{\\nu}$, and so $\\frac{\\gamma_n}{\\nu_n} =\\frac{\\ell_2^k\\tilde{\\gamma}_n}{\\nu_n} =\\frac{\\ell_2^k\\tilde{\\gamma}_n}{\\underline{\\nu}}$ and $\\frac{\\alpha_n}{\\nu_n} =\\frac{\\ell_2^k\\tilde{\\alpha}_n}{\\nu_n} =\\frac{\\ell_2^k \\tilde{\\alpha}_n}{\\underline{\\nu}}$. Since $0<\\ell_2 <1$, there exists $k_2 >\\max\\{k_0, k_1\\}$ such that after $k_2$ inner iterations, $\\gamma_n \\leq \\gamma_{n-1}$, $L_n \\beta_n = L_{\\max} \\ell_{2}^{k} \\tilde{\\beta}_n  \\leq L_{n-1} \\beta_{n-1}$,  $\\frac{\\gamma_n}{\\nu_n} \\leq \\frac{\\gamma_{n-1}}{\\nu_{n-1}}$, and $\\frac{\\alpha_n}{\\nu_n} \\leq \\frac{\\alpha_{n-1}}{\\nu_{n-1}}$.\n","sha256_utf8":"ffe21d4b50084cf3fdab5d2b6dae7cb14326b11273fa8a0cdc1e61bc437141b8"}]},{"id":"S030","semantic_type":"proof_step","kind":"proof_step","title":"应用势函数下降引理","source_units":[{"start":41763,"end":42127,"start_line":567,"end_line":572,"raw_tex":"Using Lemma~\\ref{l:descent} and noting that $\\nu_n =\\underline{\\nu}$, we get\n\\begin{align}\n\\label{eq:lsV}\n&V(z_{n+1}, x_n, w_n,\\rho_n, \\sigma_n) + \\frac{c_n}{\\underline{\\nu}}\\|z_{n+1} -x_n||^2 +  \\frac{d_n}{\\underline{\\nu}}\\|\\nabla f(u_n)\\|^2 + \\frac{\\delta}{\\underline{\\nu}} \\| w_n\\| \\notag \\\\\n&\\leq V(x_n, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}).\n\\end{align}\n","sha256_utf8":"4682d2ec635d1d1a7628d29eec7ec9f962c7c05e6a741ee9728dc9dc0980b12e"}]},{"id":"S031","semantic_type":"proof_step","kind":"proof_step","title":"验证位移下降系数","source_units":[{"start":42127,"end":43435,"start_line":573,"end_line":579,"raw_tex":"It follows from \\eqref{eq:kth} and the definition of $c_n$ in \\eqref{d:cn} that\n\\begin{align}\\label{eq:cn}\nc_n&= \\frac{(1 -\\mu \\bar\\alpha) (1 -2\\ell_{2}^{k} \\tilde{\\gamma}_n)}{\\mu^2} - \\frac{( 2 \\ell_{2}^{k}L_{\\max} \\tilde{\\beta}_n +\\ell_{2}^{k}\\tilde{\\gamma}_n + L_{\\max})\\underline{\\nu}}{2} \\notag\\\\\n&= \\frac{ 1-\\mu \\bar\\alpha}{\\mu^2}- \\frac{L_{\\max} \\underline{\\nu}}{2}-\\ell_2^k \\left[ \\frac{2\\tilde{\\gamma}_n (1-\\mu \\bar\\alpha)}{\\mu^2}  + \\frac{2L_{\\max}\\tilde{\\beta}_n \\underline{\\nu} +\\tilde{\\gamma}_n\\underline{\\nu}}{2} \\right]\\notag\\\\\n&=\\bar c \\underline{\\nu} -\\ell_2^k \\left[ \\frac{2\\tilde{\\gamma}_n (1-\\mu \\bar\\alpha)}{\\mu^2}  + \\frac{2L_{\\max}\\tilde{\\beta}_n \\underline{\\nu} +\\tilde{\\gamma}_n\\underline{\\nu}}{2} \\right].\n\\end{align}\nSince $c \\leq \\bar c=\\frac{2(1-\\mu \\bar \\alpha) - \\mu^2 L_{\\max} \\underline{\\nu}}{2 \\mu^2 \\underline{\\nu}}$, we consider two cases: $c <\\bar c$ and $c = \\bar c$. If $c <\\bar c$, using \\eqref{eq:cn} and the fact that $0< \\ell_{2} <1$, there exists $k_3$ such that after $k_3$ inner iterations, $c_n \\geq c\\underline{\\nu}$. If $c = \\bar c$, by the choice of $\\gamma_n$, $\\beta_n$ in Step~\\ref{step:init} and Step~\\ref{step:main}, for all $n \\in \\N$, $\\tilde \\gamma_n =0$ and $\\tilde \\beta_n =0$, which together with \\eqref{eq:cn} yields $c_n \\geq c\\underline{\\nu}$.\n","sha256_utf8":"a490211f4f3843c6e3dbedee7c5cd5ceee65408052eeaa7d13651686bbe28138"}]},{"id":"S032","semantic_type":"proof_step","kind":"proof_step","title":"验证梯度系数并终止搜索","source_units":[{"start":43436,"end":44403,"start_line":581,"end_line":587,"raw_tex":"Now, we consider $d_n$. Using definition of $d_n$ and \\eqref{eq:kth}, we have \n\\begin{align*}\nd_{n}&= \\frac{2(1- \\mu \\bar\\alpha)(\\lambda_n^2 -\\ell_{2}^k \\tilde{\\gamma}_n\\lambda_n)-\\mu \\ell_{2}^{k}\\tilde{\\gamma}_n\\kappa_n+2\\lambda_n \\mu \\kappa_n }{2\\mu^2}\\notag\\\\\n& = \\frac{(1-\\mu \\bar\\alpha)\\lambda_{\\min}^2 + \\lambda_{\\min} \\mu \\kappa_{\\min}}{\\mu^2} - \\ell_2^k\\frac{\\tilde{\\gamma}_n(2(1 -\\mu \\bar\\alpha)\\lambda_{\\min} +\\mu\\kappa_{\\min})}{2\\mu^2} \\notag\\\\\n& =\\bar d \\underline{\\nu} - \\ell_2^k\\frac{\\tilde{\\gamma}_n(2(1 -\\mu \\bar\\alpha)\\lambda_{\\min} +\\mu\\kappa_{\\min})}{2\\mu^2}.\n\\end{align*}\nBy Step~\\ref{step:init} and Step~\\ref{step:main}, $d \\in [0, \\bar d]$ and if $d =\\bar d$, then $\\tilde{\\gamma}_n =0$. Since $0<\\ell_2 <1$, there exists $k_4$ such that after $k_4$ inner iterations, $d_{n} \\geq d \\underline{\\nu}$. Hence, in view of \\eqref{eq:lsV}, after at most $\\max\\{ k_2,k_3, k_4\\}$ inner iterations, the line search criterion \\eqref{eq:linesearch} holds.\n","sha256_utf8":"73fd3365bec1e5fb488421b421621331438e41480e3bef161021f705fd262a8a"}]},{"id":"S033","semantic_type":"lemma","kind":"lemma","title":"邻近序列函数值差消失","source_units":[{"start":45326,"end":45647,"start_line":609,"end_line":612,"raw_tex":"\\begin{lemma}\n\\label{l:limit}\nLet $f \\colon \\mathcal{H} \\to \\R$ be a continuous differentiable function. Let $(x_n)_{n \\in \\N}$ and $(u_n)_{n \\in \\N}$ be sequences in $\\mathcal{H}$ such that $(x_n)_{n \\in \\N}$ is bounded and $x_n -u_n \\to 0$ as $n \\to +\\infty$. Then $f(x_n) -f(u_n) \\to 0$ as $n \\to +\\infty$.\n\\end{lemma}","sha256_utf8":"4063f1061928d5689ac57117208bb2ab422fe497f2f0082826407fba3cffc49b"}]},{"id":"S034","semantic_type":"proof","kind":"proof","title":"函数值差引理完整证明","source_units":[{"start":45648,"end":46190,"start_line":613,"end_line":615,"raw_tex":"\\begin{proof}\nBy the mean value theorem, $f(x_n) -f(u_n) = \\langle \\nabla f(z_n), x_n -u_n \\rangle$, where $z_n = x_n -\\zeta_n (x_n -u_n)$ for some $\\zeta_n \\in [0,1]$. Then $(z_n)_{n \\in \\N}$ is bounded due to the assumptions on $(x_n)_{n \\in \\N}$ and $(u_n)_{n \\in \\N}$. This together with the continuity of $\\nabla f$ implies the existence of $M$ such that, for all $n \\in \\N$, $\\|\\nabla f(z_n)\\| \\leq M$, which yields $|f(x_n) -f(u_n) | \\leq M\\|x_n -u_n\\|$. Since $x_n -u_n \\to 0$ as $n \\to +\\infty$, we obtain the conclusion.\n\\end{proof}","sha256_utf8":"4e8a577b9f9b5901ad197425fe4777892321892c56dd7e4f5b2be7b30d128f85"}]},{"id":"S035","semantic_type":"proof_step","kind":"proof_step","title":"均值定理估计函数值差","source_units":[{"start":45662,"end":46179,"start_line":614,"end_line":614,"raw_tex":"By the mean value theorem, $f(x_n) -f(u_n) = \\langle \\nabla f(z_n), x_n -u_n \\rangle$, where $z_n = x_n -\\zeta_n (x_n -u_n)$ for some $\\zeta_n \\in [0,1]$. Then $(z_n)_{n \\in \\N}$ is bounded due to the assumptions on $(x_n)_{n \\in \\N}$ and $(u_n)_{n \\in \\N}$. This together with the continuity of $\\nabla f$ implies the existence of $M$ such that, for all $n \\in \\N$, $\\|\\nabla f(z_n)\\| \\leq M$, which yields $|f(x_n) -f(u_n) | \\leq M\\|x_n -u_n\\|$. Since $x_n -u_n \\to 0$ as $n \\to +\\infty$, we obtain the conclusion.\n","sha256_utf8":"51720bf182384820e145d31123db649e90283f6b2e922b6d05448af954bb17b0"}]},{"id":"S036","semantic_type":"theorem","kind":"theorem","title":"子序列收敛与有限结构简化","source_units":[{"start":46193,"end":47319,"start_line":618,"end_line":629,"raw_tex":"\\begin{theorem}[Subsequential convergence and structural simplification]\n\\label{t:subseq}\nLet $(x_n)_{n\\in \\mathbb{N}}$ be the sequence generated by AGDNL algorithm and $\\Omega$ be the set of strong cluster points of $(x_n)_{n \\in \\N}$. Suppose that $f$ is a differentiable function whose gradient is $L$-Lipschitz continuous, that $f$ is bounded below, and that the set $\\{x \\in \\mathcal{H}: f(x) \\leq f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2\\}$ is bounded. Then the following hold:\n\\begin{enumerate}\n\\item\\label{t:subseq_bounded}\n$(x_n)_{n \\in \\N}$ is bounded.\n\\item\\label{t:subseq_limit} \nAs $ n \\to +\\infty$, $\\|x_{n+1}-x_n\\| \\to 0$,  $\\|w_n\\| \\to 0$, $f(x_n) \\to \\zeta$, and \\\\ $V(x_{n}, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}) \\to \\zeta$ for some $\\zeta \\in \\mathbb{R}$. Moreover, for all $\\bar x \\in \\Omega$, $ f(\\bar x) =\\zeta$.  \n\\item\\label{t:subseq_simplify} \nSuppose that $\\lambda_{\\min} >0$. Then $\\lim_{n \\to +\\infty } \\nabla f(x_n) =0$ and for all $\\bar x \\in \\Omega$, $\\nabla f(\\bar x) =0$. Moreover, from some $N_0 \\in \\N$ onwards, $w_n =0$ and AGDNL algorithm becomes AGNL algorithm.\n\\end{enumerate}\n\\end{theorem}","sha256_utf8":"a5b3400b53c79d6ae844f44f1cd66a4ea12617e044199ffee6488ebcfe45befd"}]},{"id":"S037","semantic_type":"proof","kind":"proof","title":"子序列定理完整证明","source_units":[{"start":47320,"end":55603,"start_line":630,"end_line":727,"raw_tex":"\\begin{proof}\n\\ref{t:subseq_bounded}: First, we prove by induction that, for all $n \\in \\N$,\n\\begin{equation}\\label{eq:bx1}\nV(x_{n}, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}) \\leq f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2.\n\\end{equation}\nSince $x_{-1} =x_0$, we have $V(x_{0}, x_{-1}, w_{-1},\\rho_{-1},\\sigma_{-1}) = f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2$, so \\eqref{eq:bx1} holds for $n=0$. Suppose that \\eqref{eq:bx1} holds for all $n \\leq n_0$. %We have $\\sigma_{-1} =0$ since $\\alpha_{-1} =0$.\nFrom \\eqref{eq:linesearch}, we obtain that\n\\begin{align*}\nV(x_{n_0+1}, x_{n_0}, w_{n_0},\\rho_{n_0}, \\sigma_{n_0}) &\\leq \\max_{[n_0-N]_+ \\leq k \\leq n_0}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1})\\notag\\\\\n&\\leq   f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2,\n\\end{align*}\nand hence \\eqref{eq:bx1} is satisfied for $n=n_0+1$, which completes the induction. Next, it follows from \\eqref{eq:bx1} and the definition of $V$ that $f(x_n) \\leq  f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2$. The conclusion now follows from the boundedness of the set $\\{x \\in \\mathcal{H}: f(x) \\leq f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2\\}$.\n\n\\ref{t:subseq_limit}: It follows from the line search criterion \\eqref{eq:linesearch} that\n\\begin{align}\\label{eq:lsx}\n&V(x_{n+1}, x_n, w_n,\\rho_n, \\sigma_n) +c\\|x_{n+1}-x_n\\|^2  + d \\|\\nabla f(u_n)\\|^2+ \\frac{\\delta}{\\underline{\\nu}}\\| w_n\\|\\notag\\\\ &\\leq \\max_{[n-N]_+ \\leq k \\leq n}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1})\\notag\\\\\n&= V(x_{m(n)}, x_{m(n)-1}, w_{m(n)-1},\\rho_{m(n)-1}, \\sigma_{m(n)-1}).\n\\end{align}\nTherefore,\n\\begin{align*}\n&V(x_{m(n+1)}, x_{m(n)}, w_{m(n)},\\rho_{m(n)}, \\sigma_{m(n)}) \\notag\\\\\n&= \\max_{[n+1-N]_+ \\leq k \\leq n+1}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1})\\notag\\\\\n&=\\max \\left\\{V(x_{n+1}, x_n, w_n,\\rho_n, \\sigma_n), \\max_{[n+1-N]_+ \\leq k \\leq n}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1}) \\right\\}\\notag\\\\\n& \\leq  \\max_{[n-N]_+ \\leq k \\leq n}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1}) \\notag\\\\\n&=V(x_{m(n)}, x_{m(n)-1}, w_{m(n)-1},\\rho_{m(n)-1}, \\sigma_{m(n)-1}),\n\\end{align*}\nwhich implies that the sequence $(V(x_{m(n+1)}, x_{m(n)}, w_{m(n)},\\rho_{m(n)}, \\sigma_{m(n)}))_{n\\in \\N}$ is nonincreasing. On the other hand, this sequence is bounded below due to the bounded below property of $f$ and the definition of $V$. Thus, there exists $\\zeta$ such that\n\\begin{equation}\\label{eq:limV}\n\\zeta =\\lim_{n \\to +\\infty}V(x_{m(n)}, x_{m(n)-1}, w_{m(n)-1},\\rho_{m(n)-1}, \\sigma_{m(n)-1}).\n\\end{equation} \n%\\ref{t:subseq_limit}\nIn \\eqref{eq:lsx}, replacing $n$ by $m(n)-1$ yields\n\\begin{align*}\n&V(x_{m(n)}, x_{m(n)-1}, w_{m(n)-1},\\rho_{m(n)-1}, \\sigma_{m(n)-1}) +c\\|x_{m(n)}-x_{m(n)-1}\\|^2 \\notag\\\\ \n&+d \\|\\nabla f(u_{m(n)-1})\\|^2 +\\frac{\\delta}{\\underline{\\nu}}\\| w_{m(n)-1}\\| \\notag\\\\\n&\\leq V(x_{m(m(n)-1)}, x_{m(m(n)-1)-1}, w_{m(m(n)-1)-1},\\rho_{m(m(n)-1)-1}, \\sigma_{m(m(n)-1)-1}).\n\\end{align*}\nLetting $n \\to +\\infty$ and noting that $m(n) \\geq [n-N]_{+}$, we get $ \\lim_{n \\to +\\infty}\\|w_{m(n)-1} \\| = 0$. \nIf $c >0 $, we also have $\\lim_{n \\to +\\infty}\\|x_{m(n)}-x_{m(n)-1}\\|~=~0$. If $c=0$ and $d >0$, then $\\lim_{n \\to +\\infty}\\|\\nabla f(u_{m(n)-1}) \\| = 0$, which implies that \n\\begin{equation*}\n |x_{m(n)} - x_{m(n)-1}\\| =\\|-\\lambda_{m(n)-1}\\nabla f(u_{m(n)-1}) + \\mu w_{m(n)-1} \\| \\to 0 \\text{~as~} n \\to +\\infty   \n\\end{equation*}\nIf $c=d=0$, then $\\lambda_{\\min} =0$ since  $d >0$ if $\\lambda_{\\min} >0$ and $c=0$. By Step~\\ref{step:main}, for all $n \\in \\N$, $\\lambda_n = \\gamma_n =0$, and so $\\lim_{n \\to +\\infty}\\|x_{m(n)}-x_{m(n)-1}\\| =\\lim_{n \\to +\\infty}\\mu \\|w_{m(n)-1} \\| =0$. Therefore, in any cases,\n\\begin{equation}\\label{eq:indu1}\n\\lim_{n \\to +\\infty}\\|x_{m(n)}-x_{m(n)-1}\\| = 0 \\text{~and~} \\lim_{n \\to +\\infty}\\|w_{m(n)-1} \\| = 0.\n\\end{equation}\nBy the definitions of $\\rho_n$ and $\\sigma_n$, for all $n \\in \\N$,\n$0\\leq \\rho_n \\leq  \\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\bar\\gamma}{\\underline{\\nu}} + \\frac{ L_{\\max}\\bar\\beta +\\bar\\gamma}{2 }$ and $ 0\\leq  \\sigma_n \\leq\\frac{\\mu \\alpha_{\\max}}{2 \\underline{\\nu}}.$ Combining with \\eqref{eq:limV} and \\eqref{eq:indu1}, we derive that\n\\begin{align*}\nf(x_{m(n)})&=V(x_{m(n)}, x_{m(n)-1}, w_{m(n)-1},\\rho_{m(n)-1}, \\sigma_{m(n)-1}) \\notag\\\\\n&\\quad - \\rho_{m(n)-1}\\|x_{m(n)}-x_{m(n)-1}\\|^2 - \\sigma_{m(n)-1} \\| w_{m(n)-1}\\|^2\n\\end{align*}\nalso tends to $\\zeta$ as $n \\to +\\infty$. Since $(x_n)_{n\\in \\N}$ is bounded and $\\lim_{n \\to +\\infty}\\|x_{m(n)}-x_{m(n)-1}\\| = 0$, Lemma~\\ref{l:limit} implies that \n\\begin{equation}\\label{eq:indu2}\n\\lim_{n \\to +\\infty}f(x_{m(n)-1})=\\lim_{n \\to +\\infty}f(x_{m(n)}) = \\zeta.\n\\end{equation}\nNow, we prove by induction that for all $k \\in [1, m(n)]$,\n\\begin{equation}\\label{eq:indu}\n\\lim_{n \\to +\\infty}\\|x_{m(n)-k+1}-x_{m(n)-k}\\|=0 , \\lim_{n \\to +\\infty} \\|w_{m(n)-k}\\|=0, \\text{~and~} \\lim_{n \\to +\\infty} f(x_{m(n)-k}) = \\zeta.\n\\end{equation}\nIt follows from \\eqref{eq:indu1} and \\eqref{eq:indu2} that \\eqref{eq:indu} holds for $k=1$. Suppose that \\eqref{eq:indu} holds for $k=k_0$. Replacing $n$ by $ m(n)-k_0-1:= n_m$ in \\eqref{eq:lsx} yields\n\\begin{align*}\n&V(x_{m_n+1}, x_{m_n}, w_{m_n},\\rho_{m_n}, \\sigma_{m_n}) +c\\|x_{m_n+1}-x_{m_n}\\|^2 +d \\|\\nabla f(u_{m_n})\\|^2+\\delta\\| w_{m_n}\\| \\notag\\\\\n&\\leq V(x_{m(m_n)}, x_{m(m_n)-1}, w_{m(m_n)-1},\\rho_{m(m_n)-1},\\sigma_{m(m_n)-1}).\n\\end{align*}\nUsing the definition of $V$, we have\n\\begin{align*}\n&(c+ \\rho_{m_n})\\|x_{m_n+1}-x_{m_n}\\|^2  +(\\delta + \\sigma_{m_n})\\| w_{m_n}\\| +d \\|\\nabla f(u_{m_n})\\|^2 \\notag\\\\\n&\\leq V(x_{m(m_n)}, x_{m(m_n)-1}, w_{m(m_n)-1},\\rho_{m(m_n)-1},\\sigma_{m(m_n)-1}) - f(x_{m_n+1}).\n\\end{align*}\nSimilar to the proof of \\eqref{eq:indu1}, by the induction hypothesis (\\eqref{eq:indu} holds with $k=k_0$) and \\eqref{eq:limV}, we obtain that \\eqref{eq:indu} holds for $k=k_0+1$, which completes the induction.\n\nNext, let $n \\geq N+1$. By the definition of $m(n)$, we have $n -N \\leq m(n) \\leq n$, and so $k_n := m(n) -(n -N) +1 \\in [1, m(n)]$. Then $n-N = m(n) -k_n +1$ and by \\eqref{eq:indu}, \n\\begin{align*}\n&\\lim_{n \\to +\\infty}\\|x_n-x_{n-1}\\| = \\lim_{n \\to +\\infty}\\|x_{m(n)-k_n +1}-x_{m(n)-k_n}\\| =0, \\\\\n&\\lim_{n \\to +\\infty}\\|w_{n}\\|=\\lim_{n \\to +\\infty}\\|w_{n-N-1}\\| =\\lim_{n \\to +\\infty}\\|w_{m(n)-k_n}\\|= 0, \\text{~and}\\\\\n&\\lim_{n \\to +\\infty} f(x_{n}) =\\lim_{n \\to +\\infty} f(x_{n-N-1}) = \\lim_{n \\to +\\infty} f(x_{m(n)-k_n }) = \\zeta.\n\\end{align*}\nCombining this with the definition of $V$ and the boundedness of $\\rho_n$ and $\\sigma_n$, we deduce that\n\\begin{equation}\\label{eq:limitf}\n\\lim_{n \\to +\\infty}V(x_{n}, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}) = \\lim_{n \\to +\\infty}f(x_n) = \\zeta.\n\\end{equation}\nTake an arbitrary $\\bar x \\in \\Omega$. Then there exists a sequence $(x_{k_n})_{n \\in \\N}$ such that $x_{k_n} \\to \\bar x$ as $n \\to +\\infty$. Since $f$ is continuous, we have $\\lim_{n \\to +\\infty}f(x_{k_n}) =f(\\bar x)$. This, together with \\eqref{eq:limitf}, yields that $f(x_n) \\to f(\\bar x)$ and $V(x_{n}, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}) \\to f(\\bar x)$ as $n \\to +\\infty$.\n\n\\ref{t:subseq_simplify}: Since $\\lambda_{n} \\nabla f(u_n) = v_n- x_{n+1} + \\mu w_n = -(x_{n+1} -x_n) + \\gamma_n (x_n -x_{n-1}) + \\mu w_n$, we have that\n\\begin{align*}\n\\lambda_{\\min} \\|\\nabla f(u_n)\\|\\leq \\lambda_n \\|\\nabla f(u_n)\\| \n% &\\leq \\|x_{n+1} -x_n\\| +\\gamma_n\\|x_n -x_{n-1}\\| + \\mu\\|w_n\\| \\notag\\\\\n\\leq \\|x_{n+1} -x_n\\| +\\bar\\gamma\\|x_n -x_{n-1}\\| + \\mu \\|w_n\\|.\n\\end{align*}\nCombining with \\ref{t:subseq_limit} yields $\\lim_{n \\to +\\infty} \\|\\nabla f(u_n)\\|=0$. Since $\\nabla f$ is $L$-Lipschitz continuous, it holds that\n\\begin{equation*}\n0\\leq \\left|\\|\\nabla f(u_n) \\|-\\|\\nabla f(x_n)\\|\\right| \\leq \\|\\nabla f(u_n) -\\nabla f(x_n)\\| \\leq L\\|u_n -x_n\\| = L\\beta_n\\|x_n -x_{n-1}\\|\n\\end{equation*}\nwhich implies that $\\lim_{n \\to +\\infty} \\|\\nabla f(x_n)\\| =0$, and so $\\lim_{n \\to +\\infty} \\|\\nabla f(\\bar x)\\| =0$ any $\\bar x \\in \\Omega$. \n\nIt follows from $\\lim_{n \\to +\\infty}\\|\\nabla f(u_n)\\| =0$ and $\\lim_{n \\to +\\infty}\\|w_n\\| =0$ that there exists $N_0 \\in \\mathbb{N}$ such that, for all $n \\geq N_0$, $\\kappa_{\\max} \\|\\nabla f(u_n)\\| +\\mu \\bar\\alpha \\|w_{n-1}\\| \\leq \\delta$. Combining this with Lemma~\\ref{l:descent}\\ref{l:descent_wn}, we have for all $n \\geq N_0$, $w_n =0$, which yields $ x_{n+1} -v_n +\\lambda_n \\nabla f(u_n)=0$, where $u_n = x_n +\\beta_n(x_n-x_{n-1})$ and $v_n = x_n +\\gamma_n(x_n -x_{n-1})$.\n\\end{proof}","sha256_utf8":"efaf062344bac8b5d31496767499872195f4c29327648d9c13ffc0634fa3d0da"}]},{"id":"S038","semantic_type":"assumption","kind":"assumption","title":"有界子水平集等假设","source_units":[{"start":46283,"end":46669,"start_line":620,"end_line":620,"raw_tex":"Let $(x_n)_{n\\in \\mathbb{N}}$ be the sequence generated by AGDNL algorithm and $\\Omega$ be the set of strong cluster points of $(x_n)_{n \\in \\N}$. Suppose that $f$ is a differentiable function whose gradient is $L$-Lipschitz continuous, that $f$ is bounded below, and that the set $\\{x \\in \\mathcal{H}: f(x) \\leq f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2\\}$ is bounded. Then the following hold:\n","sha256_utf8":"d116ac1484917939224047e74d4eca3e9b793d27c039cedd69c4e8661af2efa4"}]},{"id":"S039","semantic_type":"proof_step","kind":"proof_step","title":"归纳证明迭代有界","source_units":[{"start":47334,"end":48399,"start_line":631,"end_line":641,"raw_tex":"\\ref{t:subseq_bounded}: First, we prove by induction that, for all $n \\in \\N$,\n\\begin{equation}\\label{eq:bx1}\nV(x_{n}, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}) \\leq f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2.\n\\end{equation}\nSince $x_{-1} =x_0$, we have $V(x_{0}, x_{-1}, w_{-1},\\rho_{-1},\\sigma_{-1}) = f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2$, so \\eqref{eq:bx1} holds for $n=0$. Suppose that \\eqref{eq:bx1} holds for all $n \\leq n_0$. %We have $\\sigma_{-1} =0$ since $\\alpha_{-1} =0$.\nFrom \\eqref{eq:linesearch}, we obtain that\n\\begin{align*}\nV(x_{n_0+1}, x_{n_0}, w_{n_0},\\rho_{n_0}, \\sigma_{n_0}) &\\leq \\max_{[n_0-N]_+ \\leq k \\leq n_0}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1})\\notag\\\\\n&\\leq   f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2,\n\\end{align*}\nand hence \\eqref{eq:bx1} is satisfied for $n=n_0+1$, which completes the induction. Next, it follows from \\eqref{eq:bx1} and the definition of $V$ that $f(x_n) \\leq  f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2$. The conclusion now follows from the boundedness of the set $\\{x \\in \\mathcal{H}: f(x) \\leq f(x_0) + \\sigma_{-1}\\|w_{-1}\\|^2\\}$.\n","sha256_utf8":"2798642ccf855cf238341ddc41753b42dfad8ccbcf65618bb37beec5ff1b79a8"}]},{"id":"S040","semantic_type":"proof_step","kind":"proof_step","title":"窗口势函数单调收敛","source_units":[{"start":48400,"end":49767,"start_line":643,"end_line":659,"raw_tex":"\\ref{t:subseq_limit}: It follows from the line search criterion \\eqref{eq:linesearch} that\n\\begin{align}\\label{eq:lsx}\n&V(x_{n+1}, x_n, w_n,\\rho_n, \\sigma_n) +c\\|x_{n+1}-x_n\\|^2  + d \\|\\nabla f(u_n)\\|^2+ \\frac{\\delta}{\\underline{\\nu}}\\| w_n\\|\\notag\\\\ &\\leq \\max_{[n-N]_+ \\leq k \\leq n}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1})\\notag\\\\\n&= V(x_{m(n)}, x_{m(n)-1}, w_{m(n)-1},\\rho_{m(n)-1}, \\sigma_{m(n)-1}).\n\\end{align}\nTherefore,\n\\begin{align*}\n&V(x_{m(n+1)}, x_{m(n)}, w_{m(n)},\\rho_{m(n)}, \\sigma_{m(n)}) \\notag\\\\\n&= \\max_{[n+1-N]_+ \\leq k \\leq n+1}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1})\\notag\\\\\n&=\\max \\left\\{V(x_{n+1}, x_n, w_n,\\rho_n, \\sigma_n), \\max_{[n+1-N]_+ \\leq k \\leq n}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1}) \\right\\}\\notag\\\\\n& \\leq  \\max_{[n-N]_+ \\leq k \\leq n}V(x_k, x_{k-1}, w_{k-1},\\rho_{k-1}, \\sigma_{k-1}) \\notag\\\\\n&=V(x_{m(n)}, x_{m(n)-1}, w_{m(n)-1},\\rho_{m(n)-1}, \\sigma_{m(n)-1}),\n\\end{align*}\nwhich implies that the sequence $(V(x_{m(n+1)}, x_{m(n)}, w_{m(n)},\\rho_{m(n)}, \\sigma_{m(n)}))_{n\\in \\N}$ is nonincreasing. On the other hand, this sequence is bounded below due to the bounded below property of $f$ and the definition of $V$. Thus, there exists $\\zeta$ such that\n\\begin{equation}\\label{eq:limV}\n\\zeta =\\lim_{n \\to +\\infty}V(x_{m(n)}, x_{m(n)-1}, w_{m(n)-1},\\rho_{m(n)-1}, \\sigma_{m(n)-1}).\n\\end{equation} \n","sha256_utf8":"c5880f7821d73e6611c96524a8ced6a1aada76804f1c87d20614e9afffffd840"}]},{"id":"S041","semantic_type":"proof_step","kind":"proof_step","title":"极大索引处位移与摩擦消失","source_units":[{"start":49789,"end":51037,"start_line":661,"end_line":675,"raw_tex":"In \\eqref{eq:lsx}, replacing $n$ by $m(n)-1$ yields\n\\begin{align*}\n&V(x_{m(n)}, x_{m(n)-1}, w_{m(n)-1},\\rho_{m(n)-1}, \\sigma_{m(n)-1}) +c\\|x_{m(n)}-x_{m(n)-1}\\|^2 \\notag\\\\ \n&+d \\|\\nabla f(u_{m(n)-1})\\|^2 +\\frac{\\delta}{\\underline{\\nu}}\\| w_{m(n)-1}\\| \\notag\\\\\n&\\leq V(x_{m(m(n)-1)}, x_{m(m(n)-1)-1}, w_{m(m(n)-1)-1},\\rho_{m(m(n)-1)-1}, \\sigma_{m(m(n)-1)-1}).\n\\end{align*}\nLetting $n \\to +\\infty$ and noting that $m(n) \\geq [n-N]_{+}$, we get $ \\lim_{n \\to +\\infty}\\|w_{m(n)-1} \\| = 0$. \nIf $c >0 $, we also have $\\lim_{n \\to +\\infty}\\|x_{m(n)}-x_{m(n)-1}\\|~=~0$. If $c=0$ and $d >0$, then $\\lim_{n \\to +\\infty}\\|\\nabla f(u_{m(n)-1}) \\| = 0$, which implies that \n\\begin{equation*}\n |x_{m(n)} - x_{m(n)-1}\\| =\\|-\\lambda_{m(n)-1}\\nabla f(u_{m(n)-1}) + \\mu w_{m(n)-1} \\| \\to 0 \\text{~as~} n \\to +\\infty   \n\\end{equation*}\nIf $c=d=0$, then $\\lambda_{\\min} =0$ since  $d >0$ if $\\lambda_{\\min} >0$ and $c=0$. By Step~\\ref{step:main}, for all $n \\in \\N$, $\\lambda_n = \\gamma_n =0$, and so $\\lim_{n \\to +\\infty}\\|x_{m(n)}-x_{m(n)-1}\\| =\\lim_{n \\to +\\infty}\\mu \\|w_{m(n)-1} \\| =0$. Therefore, in any cases,\n\\begin{equation}\\label{eq:indu1}\n\\lim_{n \\to +\\infty}\\|x_{m(n)}-x_{m(n)-1}\\| = 0 \\text{~and~} \\lim_{n \\to +\\infty}\\|w_{m(n)-1} \\| = 0.\n\\end{equation}\n","sha256_utf8":"93c6355ff3c9950f2b60472940e9f49b531847f51701bd16fe95f9fd88a68c16"}]},{"id":"S042","semantic_type":"proof_step","kind":"proof_step","title":"传递相邻索引的函数值极限","source_units":[{"start":51037,"end":51868,"start_line":676,"end_line":685,"raw_tex":"By the definitions of $\\rho_n$ and $\\sigma_n$, for all $n \\in \\N$,\n$0\\leq \\rho_n \\leq  \\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\bar\\gamma}{\\underline{\\nu}} + \\frac{ L_{\\max}\\bar\\beta +\\bar\\gamma}{2 }$ and $ 0\\leq  \\sigma_n \\leq\\frac{\\mu \\alpha_{\\max}}{2 \\underline{\\nu}}.$ Combining with \\eqref{eq:limV} and \\eqref{eq:indu1}, we derive that\n\\begin{align*}\nf(x_{m(n)})&=V(x_{m(n)}, x_{m(n)-1}, w_{m(n)-1},\\rho_{m(n)-1}, \\sigma_{m(n)-1}) \\notag\\\\\n&\\quad - \\rho_{m(n)-1}\\|x_{m(n)}-x_{m(n)-1}\\|^2 - \\sigma_{m(n)-1} \\| w_{m(n)-1}\\|^2\n\\end{align*}\nalso tends to $\\zeta$ as $n \\to +\\infty$. Since $(x_n)_{n\\in \\N}$ is bounded and $\\lim_{n \\to +\\infty}\\|x_{m(n)}-x_{m(n)-1}\\| = 0$, Lemma~\\ref{l:limit} implies that \n\\begin{equation}\\label{eq:indu2}\n\\lim_{n \\to +\\infty}f(x_{m(n)-1})=\\lim_{n \\to +\\infty}f(x_{m(n)}) = \\zeta.\n\\end{equation}\n","sha256_utf8":"2aa95af38bf6cf83a24fe7869833d097da54601709dd9a582313082f1dd5df4a"}]},{"id":"S043","semantic_type":"proof_step","kind":"proof_step","title":"向前序索引归纳传递极限","source_units":[{"start":51868,"end":53061,"start_line":686,"end_line":700,"raw_tex":"Now, we prove by induction that for all $k \\in [1, m(n)]$,\n\\begin{equation}\\label{eq:indu}\n\\lim_{n \\to +\\infty}\\|x_{m(n)-k+1}-x_{m(n)-k}\\|=0 , \\lim_{n \\to +\\infty} \\|w_{m(n)-k}\\|=0, \\text{~and~} \\lim_{n \\to +\\infty} f(x_{m(n)-k}) = \\zeta.\n\\end{equation}\nIt follows from \\eqref{eq:indu1} and \\eqref{eq:indu2} that \\eqref{eq:indu} holds for $k=1$. Suppose that \\eqref{eq:indu} holds for $k=k_0$. Replacing $n$ by $ m(n)-k_0-1:= n_m$ in \\eqref{eq:lsx} yields\n\\begin{align*}\n&V(x_{m_n+1}, x_{m_n}, w_{m_n},\\rho_{m_n}, \\sigma_{m_n}) +c\\|x_{m_n+1}-x_{m_n}\\|^2 +d \\|\\nabla f(u_{m_n})\\|^2+\\delta\\| w_{m_n}\\| \\notag\\\\\n&\\leq V(x_{m(m_n)}, x_{m(m_n)-1}, w_{m(m_n)-1},\\rho_{m(m_n)-1},\\sigma_{m(m_n)-1}).\n\\end{align*}\nUsing the definition of $V$, we have\n\\begin{align*}\n&(c+ \\rho_{m_n})\\|x_{m_n+1}-x_{m_n}\\|^2  +(\\delta + \\sigma_{m_n})\\| w_{m_n}\\| +d \\|\\nabla f(u_{m_n})\\|^2 \\notag\\\\\n&\\leq V(x_{m(m_n)}, x_{m(m_n)-1}, w_{m(m_n)-1},\\rho_{m(m_n)-1},\\sigma_{m(m_n)-1}) - f(x_{m_n+1}).\n\\end{align*}\nSimilar to the proof of \\eqref{eq:indu1}, by the induction hypothesis (\\eqref{eq:indu} holds with $k=k_0$) and \\eqref{eq:limV}, we obtain that \\eqref{eq:indu} holds for $k=k_0+1$, which completes the induction.\n","sha256_utf8":"305eab3c0c8d477607d0a4dfc1c252b5a295df70f4cc10224188cf637c97e4d2"}]},{"id":"S044","semantic_type":"proof_step","kind":"proof_step","title":"推出全序列渐近规律","source_units":[{"start":53062,"end":53873,"start_line":702,"end_line":711,"raw_tex":"Next, let $n \\geq N+1$. By the definition of $m(n)$, we have $n -N \\leq m(n) \\leq n$, and so $k_n := m(n) -(n -N) +1 \\in [1, m(n)]$. Then $n-N = m(n) -k_n +1$ and by \\eqref{eq:indu}, \n\\begin{align*}\n&\\lim_{n \\to +\\infty}\\|x_n-x_{n-1}\\| = \\lim_{n \\to +\\infty}\\|x_{m(n)-k_n +1}-x_{m(n)-k_n}\\| =0, \\\\\n&\\lim_{n \\to +\\infty}\\|w_{n}\\|=\\lim_{n \\to +\\infty}\\|w_{n-N-1}\\| =\\lim_{n \\to +\\infty}\\|w_{m(n)-k_n}\\|= 0, \\text{~and}\\\\\n&\\lim_{n \\to +\\infty} f(x_{n}) =\\lim_{n \\to +\\infty} f(x_{n-N-1}) = \\lim_{n \\to +\\infty} f(x_{m(n)-k_n }) = \\zeta.\n\\end{align*}\nCombining this with the definition of $V$ and the boundedness of $\\rho_n$ and $\\sigma_n$, we deduce that\n\\begin{equation}\\label{eq:limitf}\n\\lim_{n \\to +\\infty}V(x_{n}, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}) = \\lim_{n \\to +\\infty}f(x_n) = \\zeta.\n\\end{equation}\n","sha256_utf8":"8c213a37f170414fbbf0d9b23ec8709d3eca0cd4b44d952aa02c63062f07ece7"}]},{"id":"S045","semantic_type":"proof_step","kind":"proof_step","title":"识别聚点共同函数值","source_units":[{"start":53873,"end":54259,"start_line":712,"end_line":712,"raw_tex":"Take an arbitrary $\\bar x \\in \\Omega$. Then there exists a sequence $(x_{k_n})_{n \\in \\N}$ such that $x_{k_n} \\to \\bar x$ as $n \\to +\\infty$. Since $f$ is continuous, we have $\\lim_{n \\to +\\infty}f(x_{k_n}) =f(\\bar x)$. This, together with \\eqref{eq:limitf}, yields that $f(x_n) \\to f(\\bar x)$ and $V(x_{n}, x_{n-1}, w_{n-1},\\rho_{n-1}, \\sigma_{n-1}) \\to f(\\bar x)$ as $n \\to +\\infty$.\n","sha256_utf8":"b7f9727c6afad9bf183e0735c61f1fb1b55a03731c60b95fb40d5b4597a69f79"}]},{"id":"S046","semantic_type":"proof_step","kind":"proof_step","title":"证明梯度消失与聚点临界性","source_units":[{"start":54260,"end":55109,"start_line":714,"end_line":724,"raw_tex":"\\ref{t:subseq_simplify}: Since $\\lambda_{n} \\nabla f(u_n) = v_n- x_{n+1} + \\mu w_n = -(x_{n+1} -x_n) + \\gamma_n (x_n -x_{n-1}) + \\mu w_n$, we have that\n\\begin{align*}\n\\lambda_{\\min} \\|\\nabla f(u_n)\\|\\leq \\lambda_n \\|\\nabla f(u_n)\\| \n% &\\leq \\|x_{n+1} -x_n\\| +\\gamma_n\\|x_n -x_{n-1}\\| + \\mu\\|w_n\\| \\notag\\\\\n\\leq \\|x_{n+1} -x_n\\| +\\bar\\gamma\\|x_n -x_{n-1}\\| + \\mu \\|w_n\\|.\n\\end{align*}\nCombining with \\ref{t:subseq_limit} yields $\\lim_{n \\to +\\infty} \\|\\nabla f(u_n)\\|=0$. Since $\\nabla f$ is $L$-Lipschitz continuous, it holds that\n\\begin{equation*}\n0\\leq \\left|\\|\\nabla f(u_n) \\|-\\|\\nabla f(x_n)\\|\\right| \\leq \\|\\nabla f(u_n) -\\nabla f(x_n)\\| \\leq L\\|u_n -x_n\\| = L\\beta_n\\|x_n -x_{n-1}\\|\n\\end{equation*}\nwhich implies that $\\lim_{n \\to +\\infty} \\|\\nabla f(x_n)\\| =0$, and so $\\lim_{n \\to +\\infty} \\|\\nabla f(\\bar x)\\| =0$ any $\\bar x \\in \\Omega$. \n","sha256_utf8":"8f64d41fddb7df570f276325da1ef3b724b7adf0bbb18029a91df0d5ca4c1f91"}]},{"id":"S047","semantic_type":"proof_step","kind":"proof_step","title":"摩擦项有限消失并简化算法","source_units":[{"start":55110,"end":55592,"start_line":726,"end_line":726,"raw_tex":"It follows from $\\lim_{n \\to +\\infty}\\|\\nabla f(u_n)\\| =0$ and $\\lim_{n \\to +\\infty}\\|w_n\\| =0$ that there exists $N_0 \\in \\mathbb{N}$ such that, for all $n \\geq N_0$, $\\kappa_{\\max} \\|\\nabla f(u_n)\\| +\\mu \\bar\\alpha \\|w_{n-1}\\| \\leq \\delta$. Combining this with Lemma~\\ref{l:descent}\\ref{l:descent_wn}, we have for all $n \\geq N_0$, $w_n =0$, which yields $ x_{n+1} -v_n +\\lambda_n \\nabla f(u_n)=0$, where $u_n = x_n +\\beta_n(x_n-x_{n-1})$ and $v_n = x_n +\\gamma_n(x_n -x_{n-1})$.\n","sha256_utf8":"651567dc169e487b544f17fc67fa02c1c9efc7f49ba55a7a007eb08745f9b9c5"}]},{"id":"S048","semantic_type":"lemma","kind":"lemma","title":"抽象KL收敛与速率引理","source_units":[{"start":62813,"end":64933,"start_line":796,"end_line":819,"raw_tex":"\\begin{lemma}[Abstract convergence under KL property]\n\\label{l:KL}\nLet $\\mathcal{H}$ and $\\mathcal{K}$ be two finite-dimensional real Hilbert\nspaces. Let $h\\colon \\mathcal{K}\\to \\R$ be a continuous function, $(x_n)_{n\\in \\mathbb{N}}$ and $(z_n)_{n\\in \\mathbb{N}}$ be sequences in $\\mathcal{H}$ and $\\mathcal{K}$, respectively, $(\\Delta_n)_{n\\in \\mathbb{N}}$ be a sequence of nonnegative real numbers, and $I$ be a finite set of integer indices with the smallest index not exceeding $1$. Set $\\Delta_k =0$ for $k <0$ and consider the following conditions:\n\\begin{enumerate}\n\\renewcommand\\theenumi{(\\alph{enumi})}\n\\renewcommand{\\labelenumi}{\\rm (\\alph{enumi})}\n\\item\\label{a:decrease}\nThere exists a positive real number $\\alpha$ such that, for all $n \\in \\N$, $h(z_{n+1}) + \\alpha \\Delta_n^2 \\leq  h(z_n)$.\n\\item\\label{a:error}\nThere exist nonnegative real numbers $\\beta_i$, $i\\in I$ such that, for all $n \\in \\N$, $\\dist(0, \\partial h(z_n)) \\leq \\sum_{i\\in I} \\beta_i\\Delta_{n-i}$.\n\\item\\label{a:distance}\nThere exists an integer $j$ such that, for all $n \\in \\N$, $\\|x_{n+1} -x_n\\| \\leq  \\Delta_{n+j}$.\n\\end{enumerate}\nSuppose that \\ref{a:decrease} and \\ref{a:error} hold, that the sequence $(z_n)_{n \\in \\N}$ is bounded with $\\Omega$ being the set of cluster points, and that $h$ satisfies the KL property at any point of $\\Omega$. Then the following hold:\n\\begin{enumerate}\n\\item\nFor all $\\bar z \\in \\Omega$, $0 \\in \\partial h(\\bar z)$ and $h(z_n)\\downarrow \\bar h := h(\\overline{z})$ as $n\\to +\\infty$.\n\\item \nIf \\ref{a:distance} holds, then $\\sum_{n=0}^{+\\infty} \\|x_{n+1} -x_n\\| < +\\infty$ and the sequence $(x_n)_{n \\in \\N}$ converges to a point $\\bar x \\in \\mathcal{H}$.\n\\item \nSuppose further that $h$ satisfies the KL property at any point of $\\Omega$ with exponent $\\alpha \\leq 1/2$. Then there exist $\\Gamma_1 \\in \\R_{++}$ and $\\zeta \\in (0,1)$ such that, for all $n \\in \\N$, $0 \\leq h(z_n) - \\bar h \\leq \\Gamma_1 \\zeta^{n}$. Moreover, if \\ref{a:distance} holds, then there exists $\\Gamma_2 \\in \\R_{++}$ such that, for all $n \\in \\N$, $\\|x_n -\\bar x\\| \\leq \\Gamma_2 \\zeta^{n/2}.$\n\\end{enumerate}\n\\end{lemma}","sha256_utf8":"dc144a158bd57eb75535f1758f311cafda401d9b89675e1802d0a866e3cc15fc"}]},{"id":"S049","semantic_type":"proof","kind":"proof","title":"抽象KL引理完整证明","source_units":[{"start":64934,"end":65770,"start_line":820,"end_line":822,"raw_tex":"\\begin{proof}\nIn view of \\cite[Lemma~5.1 and Theorem~5.2]{BDL21}, we only need to check that (I) there  exist a subsequence $(z_{k_n})_{n \\in \\N}$ and $\\tilde{z}$ such that $z_{k_n} \\to \\tilde{z}$ and $h(z_{k_n}) \\to h(\\tilde{z})$ as $n \\to +\\infty$; and (II) $h$ is constant on $\\Omega$. The first one is due to the continuity of $h$ and the boundedness of $(z_{n})_{n \\in \\N}$. We now check (II). It follows from \\ref{a:decrease} that $(h(z_n))_{n \\in \\N}$ is nonincreasing, and hence convergent due to (I). Take an arbitrary $\\bar z$ in $\\Omega$. Then, there exists a subsequence $(z_{k_n})_{n \\in \\N}$ such that $z_{k_n} \\to z$ as $n \\to +\\infty$. Since $h$ is continuous, we obtain that $h(\\bar z) =\\lim_{n \\to +\\infty} h(z_{k_n}) =\\lim_{n\\to +\\infty} h(z_n)$, and so $h$ is constant on $\\Omega$. The proof is complete.\n\\end{proof}","sha256_utf8":"ba8fdcbdc0a1bc06e7e1113632e92988d0ba38bfab8df9448ddba0517cfbad82"}]},{"id":"S050","semantic_type":"proof_step","kind":"proof_step","title":"核实引用定理的聚点条件","source_units":[{"start":64948,"end":65759,"start_line":821,"end_line":821,"raw_tex":"In view of \\cite[Lemma~5.1 and Theorem~5.2]{BDL21}, we only need to check that (I) there  exist a subsequence $(z_{k_n})_{n \\in \\N}$ and $\\tilde{z}$ such that $z_{k_n} \\to \\tilde{z}$ and $h(z_{k_n}) \\to h(\\tilde{z})$ as $n \\to +\\infty$; and (II) $h$ is constant on $\\Omega$. The first one is due to the continuity of $h$ and the boundedness of $(z_{n})_{n \\in \\N}$. We now check (II). It follows from \\ref{a:decrease} that $(h(z_n))_{n \\in \\N}$ is nonincreasing, and hence convergent due to (I). Take an arbitrary $\\bar z$ in $\\Omega$. Then, there exists a subsequence $(z_{k_n})_{n \\in \\N}$ such that $z_{k_n} \\to z$ as $n \\to +\\infty$. Since $h$ is continuous, we obtain that $h(\\bar z) =\\lim_{n \\to +\\infty} h(z_{k_n}) =\\lim_{n\\to +\\infty} h(z_n)$, and so $h$ is constant on $\\Omega$. The proof is complete.\n","sha256_utf8":"5bd507ee457fef4a59299357e916cf5fb9ece0502dba3511f531c5b7e5a55d84"}]},{"id":"S051","semantic_type":"assumption","kind":"assumption","title":"抽象KL引理的输入条件","source_units":[{"start":62880,"end":64173,"start_line":798,"end_line":810,"raw_tex":"Let $\\mathcal{H}$ and $\\mathcal{K}$ be two finite-dimensional real Hilbert\nspaces. Let $h\\colon \\mathcal{K}\\to \\R$ be a continuous function, $(x_n)_{n\\in \\mathbb{N}}$ and $(z_n)_{n\\in \\mathbb{N}}$ be sequences in $\\mathcal{H}$ and $\\mathcal{K}$, respectively, $(\\Delta_n)_{n\\in \\mathbb{N}}$ be a sequence of nonnegative real numbers, and $I$ be a finite set of integer indices with the smallest index not exceeding $1$. Set $\\Delta_k =0$ for $k <0$ and consider the following conditions:\n\\begin{enumerate}\n\\renewcommand\\theenumi{(\\alph{enumi})}\n\\renewcommand{\\labelenumi}{\\rm (\\alph{enumi})}\n\\item\\label{a:decrease}\nThere exists a positive real number $\\alpha$ such that, for all $n \\in \\N$, $h(z_{n+1}) + \\alpha \\Delta_n^2 \\leq  h(z_n)$.\n\\item\\label{a:error}\nThere exist nonnegative real numbers $\\beta_i$, $i\\in I$ such that, for all $n \\in \\N$, $\\dist(0, \\partial h(z_n)) \\leq \\sum_{i\\in I} \\beta_i\\Delta_{n-i}$.\n\\item\\label{a:distance}\nThere exists an integer $j$ such that, for all $n \\in \\N$, $\\|x_{n+1} -x_n\\| \\leq  \\Delta_{n+j}$.\n\\end{enumerate}\nSuppose that \\ref{a:decrease} and \\ref{a:error} hold, that the sequence $(z_n)_{n \\in \\N}$ is bounded with $\\Omega$ being the set of cluster points, and that $h$ satisfies the KL property at any point of $\\Omega$. Then the following hold:\n","sha256_utf8":"748f237812190a138c2ec56eda1319048ae97953973cc77e2847f2aa894a242d"}]},{"id":"S052","semantic_type":"proof_step","kind":"proof_step","title":"进入无摩擦尾序列","source_units":[{"start":67333,"end":67536,"start_line":835,"end_line":835,"raw_tex":"\\ref{ADFGe:i}: According to Theorem~\\ref{t:subseq}\\ref{t:subseq_simplify}, there exists $N_0$ such that for all $n \\geq N_0$, $w_{n} =0$ and $x_{n+1}=v_{n}-\\lambda_n \\nabla f(u_n)$. Let $n \\geq N_0+1$. \n","sha256_utf8":"faba89361d65bf970c0ad5d02602db5c18569ac9aec2c2fb5245cab19e68d39d"}]},{"id":"S053","semantic_type":"proof_step","kind":"proof_step","title":"建立严格位移下降","source_units":[{"start":67536,"end":68565,"start_line":836,"end_line":844,"raw_tex":"We first see that there exists $\\underline{c} >0$ such that\n\\begin{equation}\\label{eq:KL1}\nV(x_{n+1}, x_n,0,\\rho_n, \\sigma_n)+ \\underline{c} \\|x_{n+1}-x_n\\|^2  \\leq V(x_n, x_{n-1},0,\\rho_{n-1},\\sigma_{n-1}).\n\\end{equation}\nIndeed, since $N=0$, it follows from \\eqref{eq:linesearch} that\n\\begin{equation}\\label{eq:KL1'}\nV(x_{n+1}, x_n,0,\\rho_n, \\sigma_n)+ c\\|x_{n+1}-x_n\\|^2  + d \\|\\nabla f(u_n)\\|^2 \\leq V(x_n, x_{n-1},0,\\rho_{n-1},\\sigma_{n-1}).\n\\end{equation}\nBy Step~\\ref{step:init}, $d >0$ if $\\lambda_{\\min} >0$ and $c=0$. Thus, we must have $c >0$ or $d>0$ since $\\lambda_{\\min} >0$. If $c>0$, then \\eqref{eq:KL1} holds with $\\underline{c}:= c$. If $c=0$ and $d >0$, then by Step~\\ref{step:main}, for all $n \\in \\N$, $\\gamma_n =0$. Thus $x_{n+1} =x_n -\\lambda_n \\nabla f(u_n)$, which implies that $\\|\\nabla f(u_n)\\| = \\frac{1}{\\lambda_n}\\|x_{n+1} -x_n\\| \\geq \\frac{1}{\\lambda_{\\max}}\\|x_{n+1} -x_n\\| $. This together with \\eqref{eq:KL1'} implies that \\eqref{eq:KL1} is satisfied with $\\underline{c} := d/\\lambda_{\\max}^2$.\n","sha256_utf8":"47a04dcb17cd9060b45a88f2bc5a4a94af80da0dfdd5341c55982896416fb680"}]},{"id":"S054","semantic_type":"proof_step","kind":"proof_step","title":"构造可变参数扩展序列","source_units":[{"start":68566,"end":68814,"start_line":846,"end_line":849,"raw_tex":"Set $\\mathbf z_n = (x_{n}, x_{n-1}, \\rho_{n-1})$. Noting that $g(x,y,s)=V(x,y,0,s,t)$, we have from \\eqref{eq:linesearch} that\n\\begin{equation}\\label{eq:KL_1}\ng(\\mathbf z_{n+1})+ \\underline{c} \\|x_{n+1}-x_n\\|^2  \\leq g(\\mathbf z_n).\n\\end{equation}\n","sha256_utf8":"41378c74ee3676c014edd45da1ef68ad8112d7cfc3a0b5783a52539c58093328"}]},{"id":"S055","semantic_type":"proof_step","kind":"proof_step","title":"计算扩展势函数梯度","source_units":[{"start":68814,"end":68994,"start_line":850,"end_line":853,"raw_tex":"On the other hand,\n\\begin{equation*}\n\\nabla g(\\mathbf z_{n}) =(2\\rho_{n-1}(x_{n} -x_{n-1}) + \\nabla f(x_{n}), -2\\rho_{n-1}(x_{n} -x_{n-1}) ,\\|x_{n} -x_{n-1}\\|^2 ).\n\\end{equation*}\n","sha256_utf8":"e8aab9d920da9ce5faffc978fcb340801ead3e68068b86b2be15d6a5cffba448"}]},{"id":"S056","semantic_type":"proof_step","kind":"proof_step","title":"统一控制势函数参数","source_units":[{"start":68994,"end":69394,"start_line":854,"end_line":857,"raw_tex":"Using the definition of $\\rho_n$ and the fact that $\\gamma_n \\leq \\bar\\gamma$, $\\nu_n \\geq \\underline{\\nu}$, $L_n \\leq L_{\\max}$, we have\n\\begin{equation*}\n\\rho_n = \\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\gamma_n}{\\nu_n} + \\frac{ L_n\\beta_n +\\gamma_n}{2 }\\leq \\frac{1}{\\mu^2}(1 -\\mu \\bar\\alpha)\\frac{\\bar\\gamma}{\\underline{\\nu}} + \\frac{ L_{\\max}\\bar\\beta +\\bar\\gamma}{2 }=\\bar\\rho.\n\\end{equation*}\n","sha256_utf8":"6b12a99610ddc934c1d32175ab1d5a5119f2e189e167265c2b63e6c5165c4695"}]},{"id":"S057","semantic_type":"proof_step","kind":"proof_step","title":"估计扩展势函数梯度","source_units":[{"start":69394,"end":69740,"start_line":858,"end_line":862,"raw_tex":"Therefore, by using the sum extension of the natural norm to the product space,\n\\begin{align}\\label{eq:d1}\n\\|\\nabla g(\\mathbf z_{n})) \\|&\\leq \\|2\\rho_{n-1}(x_{n} -x_{n-1}) +\\nabla f(x_{n})\\| +\\|2\\rho_{n-1}(x_{n} -x_{n-1})\\| +\\|x_{n} -x_{n-1}\\|^2 \\notag\\\\\n&\\leq 4 \\bar\\rho\\|x_{n} -x_{n-1}\\| +\\|\\nabla f(x_{n})\\| +\\|x_{n} -x_{n-1}\\|^2.\n\\end{align}\n","sha256_utf8":"6d7e99d65beee0f7821944e2a8b75c0c79c2ef9ad42bcb73dc018dfd2fd53582"}]},{"id":"S058","semantic_type":"proof_step","kind":"proof_step","title":"用相邻位移控制目标梯度","source_units":[{"start":69740,"end":70526,"start_line":863,"end_line":873,"raw_tex":"Since $\\nabla f$ is $L$-Lipschitz continuous and $\\beta_n \\leq \\bar\\beta$,\n\\begin{equation*}\n\\|\\nabla f(u_n) -\\nabla f(x_n)\\| \\leq L \\|u_n -x_n\\| = L\\|\\beta_n(x_n -x_{n-1})\\| \\leq L \\bar\\beta\\|x_n -x_{n-1}\\|.\n\\end{equation*}\nAs $\\lambda_n \\nabla f(u_n) = v_n - x_{n+1} = -(x_{n+1} -x_n) + \\gamma_n (x_n - x_{n-1})$, it follows that\n\\begin{align}\\label{eq:d2}\n\\|\\nabla f(x_n)\\|& \\leq \\|\\nabla f(u_n)\\| + \\|\\nabla f(u_n) -\\nabla f(x_n)\\|\\notag\\\\\n% & \\leq \\frac{1}{\\lambda_n}\\|x_{n+1}-v_n\\| + L \\bar\\beta\\|x_n -x_{n-1}\\| \\notag\\\\\n&\\leq \\frac{1}{\\lambda_{\\min}} (\\|x_{n+1}-x_n\\| + \\bar\\gamma\\|x_n-x_{n-1}\\|)  + L \\bar\\beta\\|x_n -x_{n-1}\\|  \\notag\\\\\n& =  \\frac{1}{\\lambda_{\\min}}\\|x_{n+1}-x_n\\|  + \\left( \\frac{\\bar\\gamma}{\\lambda_{\\min}} +  L \\bar\\beta\\right)\\|x_n - x_{n-1}\\|.\n\\end{align}\n","sha256_utf8":"5eee18614678ed6014e1c369f7a677c4b118beb55d1fc9be48092e83561edec4"}]},{"id":"S059","semantic_type":"proof_step","kind":"proof_step","title":"核实相对误差并应用KL引理","source_units":[{"start":70526,"end":71238,"start_line":874,"end_line":878,"raw_tex":"According to Theorem~\\ref{t:subseq}\\ref{t:subseq_limit}, $\\lim_{n \\to +\\infty}\\|x_{n}-x_{n-1}\\| =0$, so there exists $N_1 \\geq N_0 +1$ such that, for all $n \\geq N_1$, $\\|x_{n}-x_{n-1}\\| \\leq 1$, which implies that  $\\|x_{n}-x_{n-1}\\|^2 \\leq \\|x_{n}-x_{n-1}\\|$. Combining this with \\eqref{eq:d1} and \\eqref{eq:d2}, there exists $C$ such that, for all $n \\geq N_1$,\n\\begin{equation} \\label{eq:KL2}\n\\| \\nabla g(\\mathbf z_{n}) \\| \\leq C(\\|x_{n+1}-x_{n}\\|+\\|x_{n} -x_{n-1}\\|).\n\\end{equation}\nMoreover, $(\\mathbf z_n)_{n \\in \\N}$ is bounded due to the boundedness of $(x_n)_{n \\in \\N}$ and $(\\rho_n)_{n \\in \\N}$. Combining this with \\eqref{eq:KL_1} and \\eqref{eq:KL2}, we get the conclusion due to Lemma~\\ref{l:KL}. \n","sha256_utf8":"334392314075eb01732827e1dfefec7f554859140849979e00dd7579f0183761"}]},{"id":"S060","semantic_type":"proof_step","kind":"proof_step","title":"估计固定势函数的函数变化","source_units":[{"start":71239,"end":71852,"start_line":880,"end_line":887,"raw_tex":"\\ref{ADFGe:ii}: As $\\nabla f$ is $L$-Lipschitz continuous, for all $ n \\in \\N$,\n\\begin{align*}\n&f(x_{n+1}) -f(x_n) -\\frac{L}{2}\\|x_{n+1}-x_n\\|^2 \\notag\\\\\n&\\leq \\langle \\nabla f(x_n), x_{n+1} -x_n\\rangle  \\notag\\\\\n& = \\langle\\nabla f(x_n) -\\nabla f(u_n), x_{n+1} -x_n\\rangle + \\langle \\nabla f(u_n), x_{n+1} -x_n\\rangle  \\notag\\\\\n& \\leq L\\|x_n -u_n\\|\\|x_{n+1} -x_n\\| + \\frac{1}{\\lambda_n}\\langle v_n -x_{n+1}, x_{n+1} -x_n \\rangle \\notag\\\\\n& = L \\beta_n \\|x_n -x_{n-1}\\|\\|x_{n+1} -x_n\\|- \\frac{1}{\\lambda_n}\\| x_{n+1} -x_{n}\\|^2 + \\frac{\\gamma_n}{\\lambda_n}\\langle x_n -x_{n-1}, x_{n+1} -x_n \\rangle.\n\\end{align*}\n","sha256_utf8":"22a5fa0b0878f06bd616d10993da1b11ed9cb9232e76502cab4f7687491b941e"}]},{"id":"S061","semantic_type":"proof_step","kind":"proof_step","title":"分离相邻位移的二次项","source_units":[{"start":71852,"end":72240,"start_line":888,"end_line":891,"raw_tex":"Noting that $\\langle x_n -x_{n-1}, x_{n+1} -x_n \\rangle \\leq \\frac{1}{2}\\left( \\|x_n -x_{n-1}\\|^2 + \\|x_{n+1} -x_n\\|^2 \\right)$, we derive that\n\\begin{equation*}\nf(x_{n+1}) -f(x_n) +\\left(\\frac{1}{\\lambda_n} - \\frac{L \\beta_n}{2} -\\frac{\\gamma_n}{2\\lambda_n}-\\frac{L}{2} \\right)\\|x_{n+1}-x_n\\|^2 \\leq (\\frac{L \\beta_n}{2} +\\frac{\\gamma_n}{2\\lambda_n})\\|x_n - x_{n-1}\\|^2.\n\\end{equation*}\n","sha256_utf8":"7e7dbca17b9900b0eb39ea287414ac391f34a2b772a1402b90bd73017861a00e"}]},{"id":"S062","semantic_type":"proof_step","kind":"proof_step","title":"验证固定势函数的KL收敛条件","source_units":[{"start":72240,"end":73050,"start_line":892,"end_line":904,"raw_tex":"Using the definition of $h$, we have\n\\begin{equation}\\label{eq:cvKL2}\nh(x_{n+1}, x_n) + \\hat c \\|x_{n+1}-x_n\\|^2 \\leq h(x_n,x_{n-1}),\n\\end{equation}\nwhere $\\hat c:= \\frac{1}{\\lambda_{\\max} }- L \\bar\\beta -\\frac{\\bar\\gamma}{\\lambda_{\\min}}-\\frac{L}{2} >0$. On the other hand,\n\\begin{equation*}\n\\nabla h(x_n, x_{n-1}) = (\\nabla f(x_{n}) + (L \\bar\\beta +\\frac{\\bar\\gamma}{ \\lambda_{\\min} })(x_n -x_{n-1}),  -(L \\bar\\beta +\\frac{\\bar\\gamma}{ \\lambda_{\\min} })(x_n -x_{n-1})).\n\\end{equation*}\nCombining this with \\eqref{eq:d2}, we obtain that\n\\begin{equation*}\n\\|\\nabla h(x_n, x_{n-1})\\| \\leq  \\frac{1}{\\lambda_{\\min}}\\|x_{n+1}-x_n\\|  + 3\\left( \\frac{\\bar\\gamma}{\\lambda_{\\min}} +  L \\bar\\beta\\right)\\|x_n - x_{n-1}\\|,\n\\end{equation*}\nwhich together with \\eqref{eq:cvKL2} and Lemma~\\ref{l:KL} completes the proof. \n","sha256_utf8":"a55bea723d73fd7f1d489e2abb425c8b0228eb2dc4b3510aabb84ab2a1a73a08"}]},{"id":"S063","semantic_type":"external_fact","kind":"external_fact","title":"引用的抽象KL收敛定理","source_units":[],"declared_external":true,"declaration_text":["BDL21, Lemma 5.1 and Theorem 5.2 supply the cited abstract KL convergence theorem: sufficient decrease h(z_{n+1})+a Delta_n^2 <= h(z_n), a relative-error bound dist(0,partial h(z_n)) <= sum_{i in I} b_i Delta_{n-i} with finite I, the required convergent-value subsequence, and constancy of h on the bounded sequence's cluster set yield critical cluster points and convergence of merit values. With the stated control of iterate displacements they yield finite length and iterate convergence; KL exponent at most 1/2 yields geometric merit-value decay and the corresponding geometric iterate rate."]},{"id":"S064","semantic_type":"external_fact","kind":"external_fact","title":"次微分最优性与光滑扰动法则","source_units":[],"declared_external":true,"declaration_text":["At a local minimizer w of a proper function F, 0 belongs to its limiting subdifferential. For a continuously differentiable q, partial(F+q)(w)=partial F(w)+nabla q(w). Thus w in Prox_{tau phi}(y), tau>0, implies (y-w)/tau in partial phi(w)."]},{"id":"S065","semantic_type":"external_fact","kind":"external_fact","title":"光滑函数下降与余项估计","source_units":[],"declared_external":true,"declaration_text":["If nabla f is L-Lipschitz, then |f(y)-f(x)-<nabla f(x),y-x>| <= (L/2)||y-x||^2. In particular, f(y) <= f(x)+<nabla f(x),y-x>+(L/2)||y-x||^2; at a critical point x*, |f(y)-f(x*)| <= (L/2)||y-x*||^2."]},{"id":"S066","semantic_type":"external_fact","kind":"external_fact","title":"线段上的均值定理","source_units":[],"declared_external":true,"declaration_text":["For a continuously differentiable real-valued function on a Hilbert space and points x,u, there is a point z on their segment such that f(x)-f(u)=<nabla f(z),x-u>."]},{"id":"S067","semantic_type":"external_fact","kind":"external_fact","title":"有限维紧性与连续性","source_units":[],"declared_external":true,"declaration_text":["In a finite-dimensional real Hilbert space, every bounded sequence has a convergent subsequence, closed bounded sets are compact, and a continuous function is bounded on every compact set."]}],"dependency_graph":{"direction_convention":"supporter_to_supported","main_result_id":"S011","edges":[{"source_id":"S001","target_id":"S007","kind":"supports"},{"source_id":"S002","target_id":"S003","kind":"supports"},{"source_id":"S002","target_id":"S004","kind":"supports"},{"source_id":"S002","target_id":"S010","kind":"supports"},{"source_id":"S003","target_id":"S018","kind":"supports"},{"source_id":"S004","target_id":"S018","kind":"supports"},{"source_id":"S064","target_id":"S018","kind":"supports"},{"source_id":"S018","target_id":"S017","kind":"supports"},{"source_id":"S017","target_id":"S016","kind":"supports"},{"source_id":"S016","target_id":"S021","kind":"supports"},{"source_id":"S007","target_id":"S021","kind":"supports"},{"source_id":"S007","target_id":"S022","kind":"supports"},{"source_id":"S005","target_id":"S023","kind":"supports"},{"source_id":"S007","target_id":"S023","kind":"supports"},{"source_id":"S065","target_id":"S023","kind":"supports"},{"source_id":"S022","target_id":"S024","kind":"supports"},{"source_id":"S023","target_id":"S024","kind":"supports"},{"source_id":"S007","target_id":"S024","kind":"supports"},{"source_id":"S021","target_id":"S025","kind":"supports"},{"source_id":"S024","target_id":"S025","kind":"supports"},{"source_id":"S025","target_id":"S026","kind":"supports"},{"source_id":"S006","target_id":"S026","kind":"supports"},{"source_id":"S007","target_id":"S026","kind":"supports"},{"source_id":"S021","target_id":"S020","kind":"supports"},{"source_id":"S022","target_id":"S020","kind":"supports"},{"source_id":"S023","target_id":"S020","kind":"supports"},{"source_id":"S024","target_id":"S020","kind":"supports"},{"source_id":"S025","target_id":"S020","kind":"supports"},{"source_id":"S026","target_id":"S020","kind":"supports"},{"source_id":"S020","target_id":"S019","kind":"supports"},{"source_id":"S007","target_id":"S029","kind":"supports"},{"source_id":"S029","target_id":"S030","kind":"supports"},{"source_id":"S019","target_id":"S030","kind":"supports"},{"source_id":"S029","target_id":"S031","kind":"supports"},{"source_id":"S019","target_id":"S031","kind":"supports"},{"source_id":"S007","target_id":"S031","kind":"supports"},{"source_id":"S029","target_id":"S032","kind":"supports"},{"source_id":"S030","target_id":"S032","kind":"supports"},{"source_id":"S031","target_id":"S032","kind":"supports"},{"source_id":"S019","target_id":"S032","kind":"supports"},{"source_id":"S007","target_id":"S032","kind":"supports"},{"source_id":"S029","target_id":"S028","kind":"supports"},{"source_id":"S030","target_id":"S028","kind":"supports"},{"source_id":"S031","target_id":"S028","kind":"supports"},{"source_id":"S032","target_id":"S028","kind":"supports"},{"source_id":"S028","target_id":"S027","kind":"supports"},{"source_id":"S027","target_id":"S039","kind":"supports"},{"source_id":"S066","target_id":"S035","kind":"supports"},{"source_id":"S035","target_id":"S034","kind":"supports"},{"source_id":"S034","target_id":"S033","kind":"supports"},{"source_id":"S038","target_id":"S039","kind":"supports"},{"source_id":"S006","target_id":"S039","kind":"supports"},{"source_id":"S007","target_id":"S039","kind":"supports"},{"source_id":"S038","target_id":"S040","kind":"supports"},{"source_id":"S006","target_id":"S040","kind":"supports"},{"source_id":"S007","target_id":"S040","kind":"supports"},{"source_id":"S009","target_id":"S040","kind":"supports"},{"source_id":"S040","target_id":"S041","kind":"supports"},{"source_id":"S007","target_id":"S041","kind":"supports"},{"source_id":"S039","target_id":"S042","kind":"supports"},{"source_id":"S040","target_id":"S042","kind":"supports"},{"source_id":"S041","target_id":"S042","kind":"supports"},{"source_id":"S033","target_id":"S042","kind":"supports"},{"source_id":"S006","target_id":"S042","kind":"supports"},{"source_id":"S007","target_id":"S042","kind":"supports"},{"source_id":"S040","target_id":"S043","kind":"supports"},{"source_id":"S041","target_id":"S043","kind":"supports"},{"source_id":"S042","target_id":"S043","kind":"supports"},{"source_id":"S006","target_id":"S043","kind":"supports"},{"source_id":"S007","target_id":"S043","kind":"supports"},{"source_id":"S009","target_id":"S043","kind":"supports"},{"source_id":"S043","target_id":"S044","kind":"supports"},{"source_id":"S042","target_id":"S044","kind":"supports"},{"source_id":"S006","target_id":"S044","kind":"supports"},{"source_id":"S009","target_id":"S044","kind":"supports"},{"source_id":"S044","target_id":"S045","kind":"supports"},{"source_id":"S038","target_id":"S045","kind":"supports"},{"source_id":"S044","target_id":"S046","kind":"supports"},{"source_id":"S007","target_id":"S046","kind":"supports"},{"source_id":"S013","target_id":"S046","kind":"supports"},{"source_id":"S044","target_id":"S047","kind":"supports"},{"source_id":"S046","target_id":"S047","kind":"supports"},{"source_id":"S019","target_id":"S047","kind":"supports"},{"source_id":"S007","target_id":"S047","kind":"supports"},{"source_id":"S008","target_id":"S047","kind":"supports"},{"source_id":"S039","target_id":"S037","kind":"supports"},{"source_id":"S040","target_id":"S037","kind":"supports"},{"source_id":"S041","target_id":"S037","kind":"supports"},{"source_id":"S042","target_id":"S037","kind":"supports"},{"source_id":"S043","target_id":"S037","kind":"supports"},{"source_id":"S044","target_id":"S037","kind":"supports"},{"source_id":"S045","target_id":"S037","kind":"supports"},{"source_id":"S046","target_id":"S037","kind":"supports"},{"source_id":"S047","target_id":"S037","kind":"supports"},{"source_id":"S037","target_id":"S036","kind":"supports"},{"source_id":"S010","target_id":"S050","kind":"supports"},{"source_id":"S051","target_id":"S050","kind":"supports"},{"source_id":"S063","target_id":"S050","kind":"supports"},{"source_id":"S067","target_id":"S050","kind":"supports"},{"source_id":"S050","target_id":"S049","kind":"supports"},{"source_id":"S049","target_id":"S048","kind":"supports"},{"source_id":"S013","target_id":"S052","kind":"supports"},{"source_id":"S036","target_id":"S052","kind":"supports"},{"source_id":"S008","target_id":"S052","kind":"supports"},{"source_id":"S052","target_id":"S053","kind":"supports"},{"source_id":"S014","target_id":"S053","kind":"supports"},{"source_id":"S007","target_id":"S053","kind":"supports"},{"source_id":"S053","target_id":"S054","kind":"supports"},{"source_id":"S006","target_id":"S054","kind":"supports"},{"source_id":"S014","target_id":"S054","kind":"supports"},{"source_id":"S014","target_id":"S055","kind":"supports"},{"source_id":"S007","target_id":"S056","kind":"supports"},{"source_id":"S055","target_id":"S057","kind":"supports"},{"source_id":"S056","target_id":"S057","kind":"supports"},{"source_id":"S052","target_id":"S058","kind":"supports"},{"source_id":"S007","target_id":"S058","kind":"supports"},{"source_id":"S013","target_id":"S058","kind":"supports"},{"source_id":"S054","target_id":"S059","kind":"supports"},{"source_id":"S056","target_id":"S059","kind":"supports"},{"source_id":"S057","target_id":"S059","kind":"supports"},{"source_id":"S058","target_id":"S059","kind":"supports"},{"source_id":"S036","target_id":"S059","kind":"supports"},{"source_id":"S048","target_id":"S059","kind":"supports"},{"source_id":"S010","target_id":"S059","kind":"supports"},{"source_id":"S013","target_id":"S059","kind":"supports"},{"source_id":"S014","target_id":"S059","kind":"supports"},{"source_id":"S065","target_id":"S059","kind":"supports"},{"source_id":"S052","target_id":"S060","kind":"supports"},{"source_id":"S007","target_id":"S060","kind":"supports"},{"source_id":"S013","target_id":"S060","kind":"supports"},{"source_id":"S065","target_id":"S060","kind":"supports"},{"source_id":"S060","target_id":"S061","kind":"supports"},{"source_id":"S061","target_id":"S062","kind":"supports"},{"source_id":"S058","target_id":"S062","kind":"supports"},{"source_id":"S036","target_id":"S062","kind":"supports"},{"source_id":"S048","target_id":"S062","kind":"supports"},{"source_id":"S010","target_id":"S062","kind":"supports"},{"source_id":"S013","target_id":"S062","kind":"supports"},{"source_id":"S015","target_id":"S062","kind":"supports"},{"source_id":"S065","target_id":"S062","kind":"supports"},{"source_id":"S052","target_id":"S012","kind":"supports"},{"source_id":"S053","target_id":"S012","kind":"supports"},{"source_id":"S054","target_id":"S012","kind":"supports"},{"source_id":"S055","target_id":"S012","kind":"supports"},{"source_id":"S056","target_id":"S012","kind":"supports"},{"source_id":"S057","target_id":"S012","kind":"supports"},{"source_id":"S058","target_id":"S012","kind":"supports"},{"source_id":"S059","target_id":"S012","kind":"supports"},{"source_id":"S060","target_id":"S012","kind":"supports"},{"source_id":"S061","target_id":"S012","kind":"supports"},{"source_id":"S062","target_id":"S012","kind":"supports"},{"source_id":"S012","target_id":"S011","kind":"supports"}]}}